mbox series

[00/25] Add Command Duration Limits support

Message ID 20221208105947.2399894-1-niklas.cassel@wdc.com (mailing list archive)
Headers show
Series Add Command Duration Limits support | expand

Message

Niklas Cassel Dec. 8, 2022, 10:59 a.m. UTC 
Hello,

This series adds support for Command Duration Limits.
The series is based on linux-next tag: next-20221205
The series can also be found in git:
https://github.com/floatious/linux/commits/cdl-v1


=================
CDL in ATA / SCSI
=================
Command Duration Limits is defined in:
T13 ATA Command Set - 5 (ACS-5) and
T10 SCSI Primary Commands - 6 (SPC-6) respectively
(a simpler version of CDL is defined in T10 SPC-5).

CDL defines Duration Limits Descriptors (DLD).
7 DLDs for read commands and 7 DLDs for write commands.
Simply put, a DLD contains a limit and a policy.

A command can specify that a certain limit should be applied by setting
the DLD index field (3 bits, so 0-7) in the command itself.

The DLD index points to one of the 7 DLDs.
DLD index 0 means no descriptor, so no limit.
DLD index 1-7 means DLD 1-7.

A DLD can have a few different policies, but the two major ones are:
-Policy 0xF (abort), command will be completed with command aborted error
(ATA) or status CHECK CONDITION (SCSI), with sense data indicating that
the command timed out.
-Policy 0xD (complete-unavailable), command will be completed without
error (ATA) or status GOOD (SCSI), with sense data indicating that the
command timed out. Note that the command will not have transferred any
data to/from the device when the command timed out, even though the
command returned success.

Regardless of the CDL policy, in case of a CDL timeout, the I/O will
result in a -ETIME error to user-space.

The DLDs are defined in the CDL log page(s) and are readable and writable.
For convenience, the kernel provides a sysfs interface for reading the
descriptors. If a user really wants to change the descriptors, they can do
so using a user-space application that sends passthrough commands,
one such application is cdl-tools:
https://github.com/westerndigitalcorporation/cdl-tools
DAMIEN: need to change the settings on the repo so that it public


==============================
How to use CDL from user-space
==============================
Since CDL is mutually exclusive with NCQ priority
(see ncq_prio_enable and sas_ncq_prio_enable in
Documentation/ABI/testing/sysfs-block-device),
CDL has to be enabled using:
echo 1 > /sys/block/$bdev/device/duration_limits/enable

In order for user-space to be able to select a specific DLD for an I/O,
we have decided to reuse the I/O priority API.

This means that we introduce a new priority class (IOPRIO_CLASS_DL).
When using this class, the existing I/O priority levels (0-7) directly
indicates the DLD index to use.

By reusing the I/O priority API, the user can both define DLD to use
per AIO (io_uring sqe->ioprio or libaio iocb->aio_reqprio) or per-thread
(ioprio_set()).


=======
Testing
=======
With the following fio patch that simply adds the new priority class:
https://github.com/westerndigitalcorporation/cdl-tools/blob/main/patches/fio-3.29-and-newer/0001-os-linux-Add-IORPIO_CLASS_DL-definition.patch

CDL can be tested using fio, e.g.:
fio --cmdprio_percentage=10 --cmdprio_class=4 --cmdprio=DLD_index

A simple way to test is to use a DLD with a very short duration limit,
and send large writes. Regardless of the CDL policy, in case of a CDL
timeout, the I/O will result in a -ETIME error to user-space.

In case of using a SATA drive, you might want to disable the write-cache:
sudo hdparm -W 0 /dev/$bdev


We have tested this patch series using:
-real hardware
-the following QEMU implementation:
https://github.com/floatious/qemu/tree/cdl


===================
Further information
===================
For further information about CDL, see Damien's slides:

Presented at SDC 2021:
https://www.snia.org/sites/default/files/SDC/2021/pdfs/SNIA-SDC21-LeMoal-Be-On-Time-command-duration-limits-Feature-Support-in%20Linux.pdf

Presented at Lund Linux Con 2022:
https://drive.google.com/file/d/1I6ChFc0h4JY9qZdO1bY5oCAdYCSZVqWw/view?usp=sharing


Kind regards,
Niklas & Damien

Damien Le Moal (14):
  ata: libata: simplify qc_fill_rtf port operation interface
  ata: libata-scsi: improve ata_scsiop_maint_in()
  scsi: support retrieving sub-pages of mode pages
  scsi: support service action in scsi_report_opcode()
  block: introduce duration-limits priority class
  block: introduce BLK_STS_DURATION_LIMIT
  ata: libata: detect support for command duration limits
  ata: libata-scsi: handle CDL bits in ata_scsiop_maint_in()
  ata: libata-scsi: add support for CDL pages mode sense
  ata: libata: add ATA feature control sub-page translation
  ata: libata: set read/write commands CDL index
  scsi: sd: detect support for command duration limits
  scsi: sd: set read/write commands CDL index
  Documentation: sysfs-block-device: document command duration limits

Niklas Cassel (11):
  ata: scsi: rename flag ATA_QCFLAG_FAILED to ATA_QCFLAG_EH
  ata: libata: move NCQ related ATA_DFLAGs
  ata: libata: fix broken NCQ command status handling
  ata: libata: respect successfully completed commands during errors
  ata: libata: allow ata_scsi_set_sense() to not set CHECK_CONDITION
  ata: libata: allow ata_eh_request_sense() to not set CHECK_CONDITION
  ata: libata-scsi: do not overwrite SCSI ML and status bytes
  scsi: core: allow libata to complete successful commands via EH
  scsi: move get_scsi_ml_byte() to scsi_priv.h
  scsi: sd: handle read/write CDL timeout failures
  ata: libata: handle completion of CDL commands using policy 0xD

 Documentation/ABI/testing/sysfs-block-device | 143 +++
 block/bfq-iosched.c                          |  10 +
 block/blk-core.c                             |   3 +
 block/blk-ioprio.c                           |   3 +
 block/ioprio.c                               |   3 +-
 block/mq-deadline.c                          |   1 +
 drivers/ata/acard-ahci.c                     |   8 +-
 drivers/ata/libahci.c                        | 171 +++-
 drivers/ata/libata-core.c                    | 219 ++++-
 drivers/ata/libata-eh.c                      | 139 ++-
 drivers/ata/libata-sata.c                    | 111 ++-
 drivers/ata/libata-scsi.c                    | 414 +++++++--
 drivers/ata/libata-sff.c                     |  10 +-
 drivers/ata/libata-trace.c                   |   2 +-
 drivers/ata/libata.h                         |   6 +-
 drivers/ata/sata_fsl.c                       |   5 +-
 drivers/ata/sata_inic162x.c                  |  14 +-
 drivers/ata/sata_promise.c                   |   2 +-
 drivers/ata/sata_sil24.c                     |   7 +-
 drivers/ata/sata_sx4.c                       |   2 +-
 drivers/scsi/Makefile                        |   2 +-
 drivers/scsi/ipr.c                           |  11 +-
 drivers/scsi/libsas/sas_ata.c                |  11 +-
 drivers/scsi/scsi.c                          |  28 +-
 drivers/scsi/scsi_error.c                    |  49 +-
 drivers/scsi/scsi_lib.c                      |  13 +-
 drivers/scsi/scsi_priv.h                     |   6 +
 drivers/scsi/scsi_transport_sas.c            |   2 +-
 drivers/scsi/sd.c                            |  37 +-
 drivers/scsi/sd.h                            |  71 ++
 drivers/scsi/sd_cdl.c                        | 894 +++++++++++++++++++
 drivers/scsi/sr.c                            |   2 +-
 include/linux/ata.h                          |  11 +-
 include/linux/blk_types.h                    |   6 +
 include/linux/ioprio.h                       |   2 +-
 include/linux/libata.h                       |  44 +-
 include/scsi/scsi_cmnd.h                     |   5 +
 include/scsi/scsi_device.h                   |   8 +-
 include/uapi/linux/ioprio.h                  |   7 +
 39 files changed, 2225 insertions(+), 257 deletions(-)
 create mode 100644 drivers/scsi/sd_cdl.c

Comments

Chaitanya Kulkarni Dec. 8, 2022, 6:18 p.m. UTC | #1 
> Kind regards,
> Niklas & Damien
> 
> Damien Le Moal (14):
>    ata: libata: simplify qc_fill_rtf port operation interface
>    ata: libata-scsi: improve ata_scsiop_maint_in()
>    scsi: support retrieving sub-pages of mode pages
>    scsi: support service action in scsi_report_opcode()
>    block: introduce duration-limits priority class
>    block: introduce BLK_STS_DURATION_LIMIT
>    ata: libata: detect support for command duration limits
>    ata: libata-scsi: handle CDL bits in ata_scsiop_maint_in()
>    ata: libata-scsi: add support for CDL pages mode sense
>    ata: libata: add ATA feature control sub-page translation
>    ata: libata: set read/write commands CDL index
>    scsi: sd: detect support for command duration limits
>    scsi: sd: set read/write commands CDL index
>    Documentation: sysfs-block-device: document command duration limits
> 
> Niklas Cassel (11):
>    ata: scsi: rename flag ATA_QCFLAG_FAILED to ATA_QCFLAG_EH
>    ata: libata: move NCQ related ATA_DFLAGs
>    ata: libata: fix broken NCQ command status handling
>    ata: libata: respect successfully completed commands during errors
>    ata: libata: allow ata_scsi_set_sense() to not set CHECK_CONDITION
>    ata: libata: allow ata_eh_request_sense() to not set CHECK_CONDITION
>    ata: libata-scsi: do not overwrite SCSI ML and status bytes
>    scsi: core: allow libata to complete successful commands via EH
>    scsi: move get_scsi_ml_byte() to scsi_priv.h
>    scsi: sd: handle read/write CDL timeout failures
>    ata: libata: handle completion of CDL commands using policy 0xD
> 

Out of 25 patches linux-block mailing list only got 3,
was this on purpose ? see this and [1] :-

https://marc.info/?l=linux-block&w=2&r=1&s=Command+Duration+limit&q=b

-ck

[1]

   7. 2022-12-08  [1] [PATCH 1/4] sbitmap: remove unnecessary 
calculation o linux-blo Kemeng Shi
   8. 2022-12-08  [1] [PATCH 0/4] A few cleanup patches for sbitmap 
     linux-blo Kemeng Shi
   9. 2022-12-08  [1] [PATCH 2/4] sbitmap: remove redundant check in 
__sbit linux-blo Kemeng Shi
  10. 2022-12-08  [1] [PATCH 3/4] sbitmap: rewrite 
sbitmap_find_bit_in_inde linux-blo Kemeng Shi
  11. 2022-12-08  [1] [PATCH 4/4] sbitmap: add sbitmap_find_bit to 
remove r linux-blo Kemeng Shi
  12. 2022-12-08  [3] Re: [PATCH 3/9] bfq: Split shared queues on move 
betw linux-blo Yu Kuai
* 13. 2022-12-08  [1] [PATCH 15/25] block: introduce 
BLK_STS_DURATION_LIMIT linux-blo Niklas Cassel*
* 14. 2022-12-08  [1] [PATCH 14/25] block: introduce duration-limits 
priori linux-blo Niklas Cassel*
* 15. 2022-12-08  [1] [PATCH 00/25] Add Command Duration Limits support 
     linux-blo Niklas Cassel*
  16. 2022-12-08  [1] [PATCH V9 8/8] block, bfq: balance I/O injection 
amon linux-blo Paolo Valente
  17. 2022-12-08  [1] [PATCH V9 7/8] block, bfq: inject I/O to 
underutilize linux-blo Paolo Valente
  18. 2022-12-08  [1] [PATCH V9 6/8] block, bfq: retrieve independent 
acces linux-blo Paolo Valente
Damien Le Moal Dec. 9, 2022, 12:29 a.m. UTC | #2 
On 12/9/22 03:18, Chaitanya Kulkarni wrote:
> 
>> Kind regards,
>> Niklas & Damien
>>
>> Damien Le Moal (14):
>>    ata: libata: simplify qc_fill_rtf port operation interface
>>    ata: libata-scsi: improve ata_scsiop_maint_in()
>>    scsi: support retrieving sub-pages of mode pages
>>    scsi: support service action in scsi_report_opcode()
>>    block: introduce duration-limits priority class
>>    block: introduce BLK_STS_DURATION_LIMIT
>>    ata: libata: detect support for command duration limits
>>    ata: libata-scsi: handle CDL bits in ata_scsiop_maint_in()
>>    ata: libata-scsi: add support for CDL pages mode sense
>>    ata: libata: add ATA feature control sub-page translation
>>    ata: libata: set read/write commands CDL index
>>    scsi: sd: detect support for command duration limits
>>    scsi: sd: set read/write commands CDL index
>>    Documentation: sysfs-block-device: document command duration limits
>>
>> Niklas Cassel (11):
>>    ata: scsi: rename flag ATA_QCFLAG_FAILED to ATA_QCFLAG_EH
>>    ata: libata: move NCQ related ATA_DFLAGs
>>    ata: libata: fix broken NCQ command status handling
>>    ata: libata: respect successfully completed commands during errors
>>    ata: libata: allow ata_scsi_set_sense() to not set CHECK_CONDITION
>>    ata: libata: allow ata_eh_request_sense() to not set CHECK_CONDITION
>>    ata: libata-scsi: do not overwrite SCSI ML and status bytes
>>    scsi: core: allow libata to complete successful commands via EH
>>    scsi: move get_scsi_ml_byte() to scsi_priv.h
>>    scsi: sd: handle read/write CDL timeout failures
>>    ata: libata: handle completion of CDL commands using policy 0xD
>>
> 
> Out of 25 patches linux-block mailing list only got 3,
> was this on purpose ? see this and [1] :-

Not sure how Niklas sent the series.

Niklas,

For the next rev (we will need one to at least rebase on 6.2-rc1 I think),
please make sure to send all patches to all lists/maintainers.