mbox series

[v4,00/11] Zoned block device support improvements

Message ID 20181012100850.23316-1-damien.lemoal@wdc.com (mailing list archive)
Headers show
Series Zoned block device support improvements | expand

Message

Damien Le Moal Oct. 12, 2018, 10:08 a.m. UTC
This series improves zoned block device support (reduce overhead) and
introduces many simplifications to the code (overall, there are more deletions
than insertions).

In more details:
* Patches 1 to 3 are SCSI side (sd driver) cleanups and improvements reducing
  the overhead of report zones command execution during disk scan and
  revalidation.
* Patches 4 to 9 improve the useability and user API of zoned block devices.
* Patch 10 is the main part of this series. This patch replaces the
  REQ_OP_ZONE_REPORT BIO/request operation for executing report zones commands
  with a block device file operation, removing the need for the command reply
  payload in-place rewriting in the BIO buffer. This leads to major
  simplification of the code in many places.
* Patch 11 further simplifies the code of low level drivers by providing a
  generic implementation of zoned block device request queue zone bitmaps
  initialization and revalidation.

Please consider the addition of these patches in 4.20.
Comments are as always welcome.

Changes from v3:
* Changed the interface of sd_zbc_check_zones() in patch 3 to return an int and
  added an argument to return the zone size to avoid potential type overflow
  with the new int return type.

Changes from v2:
* Reworked patch 9 to preserve the declaration of struct request_queue nr_zones
  field being conditional on CONFIG_BLK_DEV_ZONED

Changes from v1:
* Addressed Christoph's and Bart's comments
* Fixed several compilation errors with zoned block device support disabled
* Rebased on latest rc including the most recent dm patches

Christoph Hellwig (1):
  block: add a report_zones method

Damien Le Moal (10):
  scsi: sd_zbc: Rearrange code
  scsi: sd_zbc: Reduce boot device scan and revalidate time
  scsi: sd_zbc: Fix sd_zbc_check_zones() error checks
  block: Introduce blkdev_nr_zones() helper
  block: Limit allocation of zone descriptors for report zones
  block: Introduce BLKGETZONESZ ioctl
  block: Introduce BLKGETNRZONES ioctl
  block: Improve zone reset execution
  block: Expose queue nr_zones in sysfs
  block: Introduce blk_revalidate_disk_zones()

 block/blk-core.c               |   1 -
 block/blk-lib.c                |  13 +-
 block/blk-mq-debugfs.c         |   1 -
 block/blk-sysfs.c              |  13 +
 block/blk-zoned.c              | 359 ++++++++++++++---------
 block/blk.h                    |   8 +
 block/ioctl.c                  |   4 +
 drivers/block/null_blk.h       |  11 +-
 drivers/block/null_blk_main.c  |  30 +-
 drivers/block/null_blk_zoned.c |  57 +---
 drivers/md/dm-flakey.c         |  30 +-
 drivers/md/dm-linear.c         |  35 ++-
 drivers/md/dm-table.c          |  10 +
 drivers/md/dm-zoned-target.c   |   3 +-
 drivers/md/dm.c                | 169 ++++++-----
 drivers/scsi/sd.c              |  15 +-
 drivers/scsi/sd.h              |  15 +-
 drivers/scsi/sd_zbc.c          | 501 +++++++++------------------------
 include/linux/blk_types.h      |   2 -
 include/linux/blkdev.h         |  30 +-
 include/linux/device-mapper.h  |  12 +-
 include/trace/events/f2fs.h    |   1 -
 include/uapi/linux/blkzoned.h  |   3 +
 23 files changed, 602 insertions(+), 721 deletions(-)

Comments

Jens Axboe Oct. 13, 2018, 10:43 p.m. UTC | #1
On 10/12/18 4:08 AM, Damien Le Moal wrote:
> This series improves zoned block device support (reduce overhead) and
> introduces many simplifications to the code (overall, there are more deletions
> than insertions).
> 
> In more details:
> * Patches 1 to 3 are SCSI side (sd driver) cleanups and improvements reducing
>   the overhead of report zones command execution during disk scan and
>   revalidation.
> * Patches 4 to 9 improve the useability and user API of zoned block devices.
> * Patch 10 is the main part of this series. This patch replaces the
>   REQ_OP_ZONE_REPORT BIO/request operation for executing report zones commands
>   with a block device file operation, removing the need for the command reply
>   payload in-place rewriting in the BIO buffer. This leads to major
>   simplification of the code in many places.
> * Patch 11 further simplifies the code of low level drivers by providing a
>   generic implementation of zoned block device request queue zone bitmaps
>   initialization and revalidation.
> 
> Please consider the addition of these patches in 4.20.
> Comments are as always welcome.

How do we want to funnel this series? 1-3 look separate, so perhaps they
should go through the scsi tree. Then I can take the rest. Or do some
of the later ones depend on 1-3 being in, in terms of applying cleanly?
I can also take all of them, looks like only #3 needs a SCSI ack.
Damien Le Moal Oct. 15, 2018, 12:45 a.m. UTC | #2
Jens,

On 2018/10/14 7:43, Jens Axboe wrote:
> On 10/12/18 4:08 AM, Damien Le Moal wrote:
>> This series improves zoned block device support (reduce overhead) and
>> introduces many simplifications to the code (overall, there are more deletions
>> than insertions).
>>
>> In more details:
>> * Patches 1 to 3 are SCSI side (sd driver) cleanups and improvements reducing
>>   the overhead of report zones command execution during disk scan and
>>   revalidation.
>> * Patches 4 to 9 improve the useability and user API of zoned block devices.
>> * Patch 10 is the main part of this series. This patch replaces the
>>   REQ_OP_ZONE_REPORT BIO/request operation for executing report zones commands
>>   with a block device file operation, removing the need for the command reply
>>   payload in-place rewriting in the BIO buffer. This leads to major
>>   simplification of the code in many places.
>> * Patch 11 further simplifies the code of low level drivers by providing a
>>   generic implementation of zoned block device request queue zone bitmaps
>>   initialization and revalidation.
>>
>> Please consider the addition of these patches in 4.20.
>> Comments are as always welcome.
> 
> How do we want to funnel this series? 1-3 look separate, so perhaps they
> should go through the scsi tree. Then I can take the rest. Or do some
> of the later ones depend on 1-3 being in, in terms of applying cleanly?
> I can also take all of them, looks like only #3 needs a SCSI ack.

Patch 10 and 11 will not apply cleanly for the scsi part without 1-3 in first.
1-3, 10 and 11 would need an ack/review from Martin (SCSI).
And 10-11 also probably need an ack/review from Mike for the DM changes.

Best regards.
Jens Axboe Oct. 16, 2018, 2:34 a.m. UTC | #3
On 10/14/18 6:45 PM, Damien Le Moal wrote:
> Jens,
> 
> On 2018/10/14 7:43, Jens Axboe wrote:
>> On 10/12/18 4:08 AM, Damien Le Moal wrote:
>>> This series improves zoned block device support (reduce overhead) and
>>> introduces many simplifications to the code (overall, there are more deletions
>>> than insertions).
>>>
>>> In more details:
>>> * Patches 1 to 3 are SCSI side (sd driver) cleanups and improvements reducing
>>>   the overhead of report zones command execution during disk scan and
>>>   revalidation.
>>> * Patches 4 to 9 improve the useability and user API of zoned block devices.
>>> * Patch 10 is the main part of this series. This patch replaces the
>>>   REQ_OP_ZONE_REPORT BIO/request operation for executing report zones commands
>>>   with a block device file operation, removing the need for the command reply
>>>   payload in-place rewriting in the BIO buffer. This leads to major
>>>   simplification of the code in many places.
>>> * Patch 11 further simplifies the code of low level drivers by providing a
>>>   generic implementation of zoned block device request queue zone bitmaps
>>>   initialization and revalidation.
>>>
>>> Please consider the addition of these patches in 4.20.
>>> Comments are as always welcome.
>>
>> How do we want to funnel this series? 1-3 look separate, so perhaps
>> they should go through the scsi tree. Then I can take the rest. Or do
>> some of the later ones depend on 1-3 being in, in terms of applying
>> cleanly?  I can also take all of them, looks like only #3 needs a
>> SCSI ack.
> 
> Patch 10 and 11 will not apply cleanly for the scsi part without 1-3
> in first.  1-3, 10 and 11 would need an ack/review from Martin (SCSI).
> And 10-11 also probably need an ack/review from Mike for the DM
> changes.

Probably want to start pinging people, if you're targeting 4.20 with
this...
Damien Le Moal Oct. 16, 2018, 3:43 a.m. UTC | #4
Martin, Mike,

On 2018/10/16 11:34, Jens Axboe wrote:
> On 10/14/18 6:45 PM, Damien Le Moal wrote:
>> Jens,
>>
>> On 2018/10/14 7:43, Jens Axboe wrote:
>>> On 10/12/18 4:08 AM, Damien Le Moal wrote:
>>>> This series improves zoned block device support (reduce overhead) and
>>>> introduces many simplifications to the code (overall, there are more deletions
>>>> than insertions).
>>>>
>>>> In more details:
>>>> * Patches 1 to 3 are SCSI side (sd driver) cleanups and improvements reducing
>>>>   the overhead of report zones command execution during disk scan and
>>>>   revalidation.
>>>> * Patches 4 to 9 improve the useability and user API of zoned block devices.
>>>> * Patch 10 is the main part of this series. This patch replaces the
>>>>   REQ_OP_ZONE_REPORT BIO/request operation for executing report zones commands
>>>>   with a block device file operation, removing the need for the command reply
>>>>   payload in-place rewriting in the BIO buffer. This leads to major
>>>>   simplification of the code in many places.
>>>> * Patch 11 further simplifies the code of low level drivers by providing a
>>>>   generic implementation of zoned block device request queue zone bitmaps
>>>>   initialization and revalidation.
>>>>
>>>> Please consider the addition of these patches in 4.20.
>>>> Comments are as always welcome.
>>>
>>> How do we want to funnel this series? 1-3 look separate, so perhaps
>>> they should go through the scsi tree. Then I can take the rest. Or do
>>> some of the later ones depend on 1-3 being in, in terms of applying
>>> cleanly?  I can also take all of them, looks like only #3 needs a
>>> SCSI ack.
>>
>> Patch 10 and 11 will not apply cleanly for the scsi part without 1-3
>> in first.  1-3, 10 and 11 would need an ack/review from Martin (SCSI).
>> And 10-11 also probably need an ack/review from Mike for the DM
>> changes.
> 
> Probably want to start pinging people, if you're targeting 4.20 with
> this...

Could you please review or ack the parts relevant to SCSI and DM of this
series ?

For DM, changes are in patches 10 and 11
For SCSI, patches 1-3, 10 and 11 introduce changes.

Thank you.

Best regards.
Damien Le Moal Oct. 18, 2018, 7:57 a.m. UTC | #5
On 2018/10/16 11:34, Jens Axboe wrote:
> On 10/14/18 6:45 PM, Damien Le Moal wrote:
>> Jens,
>>
>> On 2018/10/14 7:43, Jens Axboe wrote:
>>> On 10/12/18 4:08 AM, Damien Le Moal wrote:
>>>> This series improves zoned block device support (reduce overhead) and
>>>> introduces many simplifications to the code (overall, there are more deletions
>>>> than insertions).
>>>>
>>>> In more details:
>>>> * Patches 1 to 3 are SCSI side (sd driver) cleanups and improvements reducing
>>>>   the overhead of report zones command execution during disk scan and
>>>>   revalidation.
>>>> * Patches 4 to 9 improve the useability and user API of zoned block devices.
>>>> * Patch 10 is the main part of this series. This patch replaces the
>>>>   REQ_OP_ZONE_REPORT BIO/request operation for executing report zones commands
>>>>   with a block device file operation, removing the need for the command reply
>>>>   payload in-place rewriting in the BIO buffer. This leads to major
>>>>   simplification of the code in many places.
>>>> * Patch 11 further simplifies the code of low level drivers by providing a
>>>>   generic implementation of zoned block device request queue zone bitmaps
>>>>   initialization and revalidation.
>>>>
>>>> Please consider the addition of these patches in 4.20.
>>>> Comments are as always welcome.
>>>
>>> How do we want to funnel this series? 1-3 look separate, so perhaps
>>> they should go through the scsi tree. Then I can take the rest. Or do
>>> some of the later ones depend on 1-3 being in, in terms of applying
>>> cleanly?  I can also take all of them, looks like only #3 needs a
>>> SCSI ack.
>>
>> Patch 10 and 11 will not apply cleanly for the scsi part without 1-3
>> in first.  1-3, 10 and 11 would need an ack/review from Martin (SCSI).
>> And 10-11 also probably need an ack/review from Mike for the DM
>> changes.
> 
> Probably want to start pinging people, if you're targeting 4.20 with
> this...

Hi Jens,

Martin and Mike acked/reviewed the SCSI and DM parts respectively (thanks !).
Do you think you can take this series for 4.20 ?

Thanks.
Jens Axboe Oct. 23, 2018, 3:59 p.m. UTC | #6
On 10/12/18 4:08 AM, Damien Le Moal wrote:
> This series improves zoned block device support (reduce overhead) and
> introduces many simplifications to the code (overall, there are more deletions
> than insertions).
> 
> In more details:
> * Patches 1 to 3 are SCSI side (sd driver) cleanups and improvements reducing
>   the overhead of report zones command execution during disk scan and
>   revalidation.
> * Patches 4 to 9 improve the useability and user API of zoned block devices.
> * Patch 10 is the main part of this series. This patch replaces the
>   REQ_OP_ZONE_REPORT BIO/request operation for executing report zones commands
>   with a block device file operation, removing the need for the command reply
>   payload in-place rewriting in the BIO buffer. This leads to major
>   simplification of the code in many places.
> * Patch 11 further simplifies the code of low level drivers by providing a
>   generic implementation of zoned block device reuest queue zone bitmaps
>   initialization and revalidation.

I've applied this, but I have two complaints:

1) Two had to be hand applied, it wasn't against the block tree.
2) The ordering of the signed-off-by. Someone told me that this is
   patchwork, but I absolutely hate it. SOB should go last, not before
   the reviewed-by. I fixed that up too.
Martin K. Petersen Oct. 24, 2018, 2:26 a.m. UTC | #7
Jens,

> 2) The ordering of the signed-off-by. Someone told me that this is
>    patchwork, but I absolutely hate it. SOB should go last, not before
>    the reviewed-by. I fixed that up too.

You keep mentioning this, but I don't recall ever seeing anything to
that effect. The rest of the kernel appears to be either arbitrary
ordering or favoring author SoB as the first tag.
Damien Le Moal Oct. 24, 2018, 8:04 a.m. UTC | #8
Jens,

On 2018/10/23 16:59, Jens Axboe wrote:
> On 10/12/18 4:08 AM, Damien Le Moal wrote:
>> This series improves zoned block device support (reduce overhead) and
>> introduces many simplifications to the code (overall, there are more deletions
>> than insertions).
>>
>> In more details:
>> * Patches 1 to 3 are SCSI side (sd driver) cleanups and improvements reducing
>>   the overhead of report zones command execution during disk scan and
>>   revalidation.
>> * Patches 4 to 9 improve the useability and user API of zoned block devices.
>> * Patch 10 is the main part of this series. This patch replaces the
>>   REQ_OP_ZONE_REPORT BIO/request operation for executing report zones commands
>>   with a block device file operation, removing the need for the command reply
>>   payload in-place rewriting in the BIO buffer. This leads to major
>>   simplification of the code in many places.
>> * Patch 11 further simplifies the code of low level drivers by providing a
>>   generic implementation of zoned block device reuest queue zone bitmaps
>>   initialization and revalidation.
> 
> I've applied this, but I have two complaints:
> 
> 1) Two had to be hand applied, it wasn't against the block tree.

My apologies for that. I had to rebase against the latest RC because of a
conflict with late fixes in DM. I should have checked more carefully against the
block tree after that. Sorry about this, I will be more careful going forward.

> 2) The ordering of the signed-off-by. Someone told me that this is
>    patchwork, but I absolutely hate it. SOB should go last, not before
>    the reviewed-by. I fixed that up too.

Thank you. I think this was me rather than patchwork when I a manually added the
reviewed-by tags with new versions of the series. I will remember this and tag
in you preferred order.

Thank you.

Best regards.
Mike Snitzer Oct. 24, 2018, 3:03 p.m. UTC | #9
On Tue, Oct 23 2018 at 10:26pm -0400,
Martin K. Petersen <martin.petersen@oracle.com> wrote:

> 
> Jens,
> 
> > 2) The ordering of the signed-off-by. Someone told me that this is
> >    patchwork, but I absolutely hate it. SOB should go last, not before
> >    the reviewed-by. I fixed that up too.
> 
> You keep mentioning this, but I don't recall ever seeing anything to
> that effect. The rest of the kernel appears to be either arbitrary
> ordering or favoring author SoB as the first tag.

I've always felt the proper order is how Jens likes it too (all dm
commits from me follow that order).

Mike
Martin K. Petersen Oct. 24, 2018, 3:37 p.m. UTC | #10
Mike,

>> You keep mentioning this, but I don't recall ever seeing anything to
>> that effect. The rest of the kernel appears to be either arbitrary
>> ordering or favoring author SoB as the first tag.
>
> I've always felt the proper order is how Jens likes it too (all dm
> commits from me follow that order).

That's fine, I don't have any particular preference. And I don't have
any issue with you guys sticking to a certain ordering in your
respective subsystems. I occasionally shuffle tags when I commit things
in SCSI too.

I just think it should be properly documented if there is a preferred
way to order things...
Bart Van Assche Oct. 24, 2018, 4:04 p.m. UTC | #11
On Wed, 2018-10-24 at 11:37 -0400, Martin K. Petersen wrote:
> Mike,
> 
> > > You keep mentioning this, but I don't recall ever seeing anything to
> > > that effect. The rest of the kernel appears to be either arbitrary
> > > ordering or favoring author SoB as the first tag.
> > 
> > I've always felt the proper order is how Jens likes it too (all dm
> > commits from me follow that order).
> 
> That's fine, I don't have any particular preference. And I don't have
> any issue with you guys sticking to a certain ordering in your
> respective subsystems. I occasionally shuffle tags when I commit things
> in SCSI too.
> 
> I just think it should be properly documented if there is a preferred
> way to order things...

When I tried to look up documentation for this I couldn't find anything under
the Documentation directory. Maybe it's there but I didn't look carefully
enough. All I could find on the web is e-mails from Linus in which he explains
that the order of Signed-off-by's should match the chain of authorship.

Bart.
Jens Axboe Oct. 25, 2018, 2:30 p.m. UTC | #12
On 10/24/18 10:04 AM, Bart Van Assche wrote:
> On Wed, 2018-10-24 at 11:37 -0400, Martin K. Petersen wrote:
>> Mike,
>>
>>>> You keep mentioning this, but I don't recall ever seeing anything to
>>>> that effect. The rest of the kernel appears to be either arbitrary
>>>> ordering or favoring author SoB as the first tag.
>>>
>>> I've always felt the proper order is how Jens likes it too (all dm
>>> commits from me follow that order).
>>
>> That's fine, I don't have any particular preference. And I don't have
>> any issue with you guys sticking to a certain ordering in your
>> respective subsystems. I occasionally shuffle tags when I commit things
>> in SCSI too.
>>
>> I just think it should be properly documented if there is a preferred
>> way to order things...
> 
> When I tried to look up documentation for this I couldn't find anything under
> the Documentation directory. Maybe it's there but I didn't look carefully
> enough. All I could find on the web is e-mails from Linus in which he explains
> that the order of Signed-off-by's should match the chain of authorship.

I don't think there's any documentation on it, none that I've seen.
Haven't looked for it, though.

It's more of a "it always looked like that", until we got patchwork messing
things up. And then people see that, and do the same. It's a bit
frustrating. I like to be able to see the SOB chain in a patch, and if
it's intermingled with other things, it's much harder to read. At least
for me.

I'll continue fixing these up, but I do hope that at least the regulars
on the block side use the proper formatting.