| Message ID | 20170531123706.20885-4-ming.lei@redhat.com (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
On Wed, 2017-05-31 at 20:37 +0800, Ming Lei wrote: > diff --git a/drivers/scsi/scsi_lib.c b/drivers/scsi/scsi_lib.c > index 99e16ac479e3..ffcf05765e2b 100644 > --- a/drivers/scsi/scsi_lib.c > +++ b/drivers/scsi/scsi_lib.c > @@ -3031,7 +3031,10 @@ scsi_internal_device_unblock(struct scsi_device *sdev, > return -EINVAL; > > if (q->mq_ops) { > - blk_mq_start_stopped_hw_queues(q, false); > + if (blk_queue_quiesced(q)) > + blk_mq_unquiesce_queue(q); > + else > + blk_mq_start_stopped_hw_queues(q, false); > } else { > spin_lock_irqsave(q->queue_lock, flags); > blk_start_queue(q); As I commented on v2, this change is really wrong. All what's needed here is a call to blk_mq_unquiesce_queue() and nothing else. Adding a call to blk_mq_start_stopped_hw_queues() is wrong because it makes it impossible to use the STOPPED flag in the SCSI core to make the block layer core stop calling .queue_rq() if a SCSI LLD returns "busy". Bart.
On Wed, May 31, 2017 at 03:21:41PM +0000, Bart Van Assche wrote: > On Wed, 2017-05-31 at 20:37 +0800, Ming Lei wrote: > > diff --git a/drivers/scsi/scsi_lib.c b/drivers/scsi/scsi_lib.c > > index 99e16ac479e3..ffcf05765e2b 100644 > > --- a/drivers/scsi/scsi_lib.c > > +++ b/drivers/scsi/scsi_lib.c > > @@ -3031,7 +3031,10 @@ scsi_internal_device_unblock(struct scsi_device *sdev, > > return -EINVAL; > > > > if (q->mq_ops) { > > - blk_mq_start_stopped_hw_queues(q, false); > > + if (blk_queue_quiesced(q)) > > + blk_mq_unquiesce_queue(q); > > + else > > + blk_mq_start_stopped_hw_queues(q, false); > > } else { > > spin_lock_irqsave(q->queue_lock, flags); > > blk_start_queue(q); > > As I commented on v2, this change is really wrong. All what's needed here is > a call to blk_mq_unquiesce_queue() and nothing else. Adding a call to > blk_mq_start_stopped_hw_queues() is wrong because it makes it impossible to > use the STOPPED flag in the SCSI core to make the block layer core stop calling > .queue_rq() if a SCSI LLD returns "busy". I am not sure if I understand your idea, could you explain a bit why it is wrong? Let's see the function of scsi_internal_device_block(): if (q->mq_ops) { if (wait) blk_mq_quiesce_queue(q); else blk_mq_stop_hw_queues(q); } So the queue may be put into quiesced if 'wait' is true, or it is stopped if 'wait' is false. And this patch just makes the two SCSI APIs symmetrical. Since we will not stop queue in blk_mq_quiesce_queue() later, I have to unquiese one queue only if it is quiesced. So suppose the queue is put into stopped in scsi_internal_device_block(), do we expect not to restart it in scsi_internal_device_unblock() via blk_mq_unquiesce_queue()? Thanks, Ming
On Thu, 2017-06-01 at 08:54 +0800, Ming Lei wrote: > On Wed, May 31, 2017 at 03:21:41PM +0000, Bart Van Assche wrote: > > On Wed, 2017-05-31 at 20:37 +0800, Ming Lei wrote: > > > diff --git a/drivers/scsi/scsi_lib.c b/drivers/scsi/scsi_lib.c > > > index 99e16ac479e3..ffcf05765e2b 100644 > > > --- a/drivers/scsi/scsi_lib.c > > > +++ b/drivers/scsi/scsi_lib.c > > > @@ -3031,7 +3031,10 @@ scsi_internal_device_unblock(struct scsi_device *sdev, > > > return -EINVAL; > > > > > > if (q->mq_ops) { > > > - blk_mq_start_stopped_hw_queues(q, false); > > > + if (blk_queue_quiesced(q)) > > > + blk_mq_unquiesce_queue(q); > > > + else > > > + blk_mq_start_stopped_hw_queues(q, false); > > > } else { > > > spin_lock_irqsave(q->queue_lock, flags); > > > blk_start_queue(q); > > > > As I commented on v2, this change is really wrong. All what's needed here is > > a call to blk_mq_unquiesce_queue() and nothing else. Adding a call to > > blk_mq_start_stopped_hw_queues() is wrong because it makes it impossible to > > use the STOPPED flag in the SCSI core to make the block layer core stop calling > > .queue_rq() if a SCSI LLD returns "busy". > > I am not sure if I understand your idea, could you explain a bit why it is wrong? > > Let's see the function of scsi_internal_device_block(): > > if (q->mq_ops) { > if (wait) > blk_mq_quiesce_queue(q); > else > blk_mq_stop_hw_queues(q); > } > > So the queue may be put into quiesced if 'wait' is true, or it is > stopped if 'wait' is false. > > And this patch just makes the two SCSI APIs symmetrical. > > Since we will not stop queue in blk_mq_quiesce_queue() later, > I have to unquiese one queue only if it is quiesced. > > So suppose the queue is put into stopped in scsi_internal_device_block(), > do we expect not to restart it in scsi_internal_device_unblock() via > blk_mq_unquiesce_queue()? Hello Ming, My opinion is that scsi_internal_device_block() and scsi_internal_device_unblock() should be changed as follows for the scsi-mq code path: * scsi_internal_device_block() should call blk_mq_quiesce_queue(q) if the "wait" argument is true and should set the QUEUE_FLAG_QUIESCED flag if the "wait" argument is false. * scsi_internal_device_unblock() should call blk_mq_unquiesce_queue(). I am aware it sounds weird to set the QUEUE_FLAG_QUIESCED flag without waiting until ongoing .queue_rq() calls have finished. The only driver that triggers that code path is the mpt3sas driver. I think it's unfortunate that that driver has ever been allowed to call scsi_internal_device_block() because it's the only driver that calls that function from a context where sleeping is not allowed. No matter whether the scsi_internal_device_block() call from the mpt3sas driver sets the QUEUE_FLAG_QUIESCED or the BLK_MQ_S_STOPPED flag, I don't think that will have the effect the authors of this driver intended. Unfortunately I'm not familiar enough with the mpt3sas driver to fix that driver myself. Note: these changes conflict with a patch SCSI patch series I started working on about two months ago. See also https://www.spinics.net/lists/linux-scsi/msg109103.html. Bart.
On Thu, Jun 01, 2017 at 11:09:00PM +0000, Bart Van Assche wrote: > On Thu, 2017-06-01 at 08:54 +0800, Ming Lei wrote: > > On Wed, May 31, 2017 at 03:21:41PM +0000, Bart Van Assche wrote: > > > On Wed, 2017-05-31 at 20:37 +0800, Ming Lei wrote: > > > > diff --git a/drivers/scsi/scsi_lib.c b/drivers/scsi/scsi_lib.c > > > > index 99e16ac479e3..ffcf05765e2b 100644 > > > > --- a/drivers/scsi/scsi_lib.c > > > > +++ b/drivers/scsi/scsi_lib.c > > > > @@ -3031,7 +3031,10 @@ scsi_internal_device_unblock(struct scsi_device *sdev, > > > > return -EINVAL; > > > > > > > > if (q->mq_ops) { > > > > - blk_mq_start_stopped_hw_queues(q, false); > > > > + if (blk_queue_quiesced(q)) > > > > + blk_mq_unquiesce_queue(q); > > > > + else > > > > + blk_mq_start_stopped_hw_queues(q, false); > > > > } else { > > > > spin_lock_irqsave(q->queue_lock, flags); > > > > blk_start_queue(q); > > > > > > As I commented on v2, this change is really wrong. All what's needed here is > > > a call to blk_mq_unquiesce_queue() and nothing else. Adding a call to > > > blk_mq_start_stopped_hw_queues() is wrong because it makes it impossible to > > > use the STOPPED flag in the SCSI core to make the block layer core stop calling > > > .queue_rq() if a SCSI LLD returns "busy". > > > > I am not sure if I understand your idea, could you explain a bit why it is wrong? > > > > Let's see the function of scsi_internal_device_block(): > > > > if (q->mq_ops) { > > if (wait) > > blk_mq_quiesce_queue(q); > > else > > blk_mq_stop_hw_queues(q); > > } > > > > So the queue may be put into quiesced if 'wait' is true, or it is > > stopped if 'wait' is false. > > > > And this patch just makes the two SCSI APIs symmetrical. > > > > Since we will not stop queue in blk_mq_quiesce_queue() later, > > I have to unquiese one queue only if it is quiesced. > > > > So suppose the queue is put into stopped in scsi_internal_device_block(), > > do we expect not to restart it in scsi_internal_device_unblock() via > > blk_mq_unquiesce_queue()? > > Hello Ming, > > My opinion is that scsi_internal_device_block() and scsi_internal_device_unblock() > should be changed as follows for the scsi-mq code path: > * scsi_internal_device_block() should call blk_mq_quiesce_queue(q) if the "wait" > argument is true and should set the QUEUE_FLAG_QUIESCED flag if the "wait" > argument is false. > * scsi_internal_device_unblock() should call blk_mq_unquiesce_queue(). > > I am aware it sounds weird to set the QUEUE_FLAG_QUIESCED flag without waiting > until ongoing .queue_rq() calls have finished. The only driver that triggers This way may break the contract of blk_mq_quiesce_queue() and is really weird. And the above change shouldn't be related with this patchset, so better to do whatever you want once this patch is merged. This patchset won't break current SCSI uses of blk_mq_quiesce_queue() and won't cause regression, and I setup one iSCSI target yesterday and it works just fine, how about just keeping this patch as it is? Once it is merged, you can figure out one perfect solution, but the contract of blk_mq_quiesce_queue() still shouldn't be broken. Thanks, Ming
On Fri, 2017-06-02 at 10:00 +0800, Ming Lei wrote: > On Thu, Jun 01, 2017 at 11:09:00PM +0000, Bart Van Assche wrote: > > My opinion is that scsi_internal_device_block() and scsi_internal_device_unblock() > > should be changed as follows for the scsi-mq code path: > > * scsi_internal_device_block() should call blk_mq_quiesce_queue(q) if the "wait" > > argument is true and should set the QUEUE_FLAG_QUIESCED flag if the "wait" > > argument is false. > > * scsi_internal_device_unblock() should call blk_mq_unquiesce_queue(). > > > > I am aware it sounds weird to set the QUEUE_FLAG_QUIESCED flag without waiting > > until ongoing .queue_rq() calls have finished. The only driver that triggers > > This way may break the contract of blk_mq_quiesce_queue() and is really weird. > And the above change shouldn't be related with this patchset, so better > to do whatever you want once this patch is merged. No, what I proposed doesn't break the contract of blk_mq_quiesce_queue(). The contract of that function is that if it is called that all pending .queue_rq() calls have finished and no new .queue_rq() calls will occur until blk_mq_unquiesce_queue() is called. Since the wait == false path does not call blk_mq_quiesce_queue() that contract does not apply. > This patchset won't break current SCSI uses of blk_mq_quiesce_queue() and > won't cause regression, and I setup one iSCSI target yesterday and it works > just fine, how about just keeping this patch as it is? As I explained in a previous e-mail, the iSCSI initiator driver does not trigger the wait == false path so your test did not trigger that code path. Something else I explained in a previous e-mail is that your patch makes it impossible to use the STOPPED flag in the SCSI core what it was originally intended for, namely preventing that the block layer keeps trying to queue requests while the block driver knows that it will have to return "BUSY". This is why I asked you to modify the SCSI integration of your patch. Bart.
diff --git a/drivers/md/dm-rq.c b/drivers/md/dm-rq.c index b639fa7246ee..ea4029de077f 100644 --- a/drivers/md/dm-rq.c +++ b/drivers/md/dm-rq.c @@ -71,7 +71,7 @@ static void dm_old_start_queue(struct request_queue *q) static void dm_mq_start_queue(struct request_queue *q) { - blk_mq_start_stopped_hw_queues(q, true); + blk_mq_unquiesce_queue(q); blk_mq_kick_requeue_list(q); } diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c index a60926410438..c3f189e54d10 100644 --- a/drivers/nvme/host/core.c +++ b/drivers/nvme/host/core.c @@ -2526,7 +2526,7 @@ void nvme_start_queues(struct nvme_ctrl *ctrl) mutex_lock(&ctrl->namespaces_mutex); list_for_each_entry(ns, &ctrl->namespaces, list) { - blk_mq_start_stopped_hw_queues(ns->queue, true); + blk_mq_unquiesce_queue(ns->queue); blk_mq_kick_requeue_list(ns->queue); } mutex_unlock(&ctrl->namespaces_mutex); diff --git a/drivers/scsi/scsi_lib.c b/drivers/scsi/scsi_lib.c index 99e16ac479e3..ffcf05765e2b 100644 --- a/drivers/scsi/scsi_lib.c +++ b/drivers/scsi/scsi_lib.c @@ -3031,7 +3031,10 @@ scsi_internal_device_unblock(struct scsi_device *sdev, return -EINVAL; if (q->mq_ops) { - blk_mq_start_stopped_hw_queues(q, false); + if (blk_queue_quiesced(q)) + blk_mq_unquiesce_queue(q); + else + blk_mq_start_stopped_hw_queues(q, false); } else { spin_lock_irqsave(q->queue_lock, flags); blk_start_queue(q);
blk_mq_unquiesce_queue() is used for unquiescing the queue explicitly, so replace blk_mq_start_stopped_hw_queues() with it. Cc: linux-nvme@lists.infradead.org Cc: linux-scsi@vger.kernel.org Cc: dm-devel@redhat.com Signed-off-by: Ming Lei <ming.lei@redhat.com> --- drivers/md/dm-rq.c | 2 +- drivers/nvme/host/core.c | 2 +- drivers/scsi/scsi_lib.c | 5 ++++- 3 files changed, 6 insertions(+), 3 deletions(-)