From patchwork Thu Jan 19 14:35:24 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 9525993 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 393B2601AE for ; Thu, 19 Jan 2017 14:35:36 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 2A1BC28520 for ; Thu, 19 Jan 2017 14:35:36 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 1EEB22852D; Thu, 19 Jan 2017 14:35:36 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.3 required=2.0 tests=BAYES_00,DKIM_SIGNED, RCVD_IN_DNSWL_HI, RCVD_IN_SORBS_SPAM, T_DKIM_INVALID autolearn=unavailable version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id EB69E28520 for ; Thu, 19 Jan 2017 14:35:34 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753072AbdASOfd (ORCPT ); Thu, 19 Jan 2017 09:35:33 -0500 Received: from mail-pf0-f181.google.com ([209.85.192.181]:35889 "EHLO mail-pf0-f181.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753059AbdASOfc (ORCPT ); Thu, 19 Jan 2017 09:35:32 -0500 Received: by mail-pf0-f181.google.com with SMTP id 189so14063613pfu.3 for ; Thu, 19 Jan 2017 06:35:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20150623.gappssmtp.com; s=20150623; h=subject:to:references:cc:from:message-id:date:user-agent :mime-version:in-reply-to:content-transfer-encoding; bh=rbTrhvSo5FLov/Wc73nRh3vNes/5X5qtCDX9zbMnYmM=; b=Wc1hxyt0p7VekTMe3/6AMG4kBJeiIuvTPb4iGeDcJbT+dzKXy09WvKf39/I3wLBbrA U+hPmv6kh57ozX/eNU7Exn4VjM+MHGLzxGj2NlI6SmRjOdgVtfwYPnGFD8R4V+QPwRdE kYnw/sQo6Dr402EQH2s072lmWJCLnEVvtCklUyHp82Ya7RNrKJ8BjWHnLenlwohC7nmp ddNLuRcDkkp/PNxd//xM7mW9eK6xww28nmN7AslufuAhgky7rB2Ibij7fylt+8lT8+3b 98Ny0irhTVC/Li4j5++/aaU1mDL12wUVdDRodx7A5GSp7TzNenRVHdlhlJip7A6SttQH jQcg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:references:cc:from:message-id:date :user-agent:mime-version:in-reply-to:content-transfer-encoding; bh=rbTrhvSo5FLov/Wc73nRh3vNes/5X5qtCDX9zbMnYmM=; b=gO6KYLV+47IJJ6l0vFJhKMh3CpGxHdThuPpljcx+7hO/lrA9Tb0Y5pnpPpt49i+9Qk rH+tJmTQ15VbT7fMXlTR/PDBVAWyLz8mciXEZqPe52basmzdWX3hIvMdMNnY9wI0jyCN rsudDzLHRW/qTQUn/QdmkjNVr/2chQHINeGVOgPtKje20WMXx6jSKJRuLSq1S1hasD1o 61RSLU39+/Li2H6TeCiNLkjo1xPBY2mwtn8KYI2EnLdJtetsl9GomxTuqV/OFiXlypxS mROhU+myy9jmmIVEzAH8R/Vdeiz/3Z6MDj66c+wcMIqsuhA4AtMrOGwImVqT6h0dOwbf XmAg== X-Gm-Message-State: AIkVDXLYvz/HT8qD0f+2C9VkFTGbQHvFOdCmUIaSC/JK2+AvM5VTEvRlYJQBbMUTl/YLbQ== X-Received: by 10.84.232.70 with SMTP id f6mr13697785pln.113.1484836526269; Thu, 19 Jan 2017 06:35:26 -0800 (PST) Received: from [192.168.102.217] (50-233-49-230-static.hfc.comcastbusiness.net. [50.233.49.230]) by smtp.gmail.com with ESMTPSA id i86sm9379549pfj.87.2017.01.19.06.35.25 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 19 Jan 2017 06:35:25 -0800 (PST) Subject: Re: requeue failure with blk-mq-sched To: Hannes Reinecke References: <8b7436c0-91c4-0b70-071c-364abe1f24f5@kernel.dk> <9b90e296-7e39-34a5-c18e-df27a5674417@suse.de> Cc: "linux-block@vger.kernel.org" , "linux-scsi@vger.kernel.org" , Christoph Hellwig From: Jens Axboe Message-ID: Date: Thu, 19 Jan 2017 06:35:24 -0800 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Thunderbird/45.5.1 MIME-Version: 1.0 In-Reply-To: <9b90e296-7e39-34a5-c18e-df27a5674417@suse.de> Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP On 01/19/2017 06:24 AM, Hannes Reinecke wrote: > On 01/19/2017 03:09 PM, Jens Axboe wrote: >> On 01/19/2017 04:27 AM, Hannes Reinecke wrote: >>> Hi Jens, >>> >>> upon further testing with your blk-mq-sched branch I hit a queue stall >>> during requeing: >>> >>> [ 202.340959] sd 3:0:4:1: tag#473 Send: scmd 0xffff880422e7a440 >>> [ 202.340962] sd 3:0:4:1: tag#473 CDB: Test Unit Ready 00 00 00 00 00 00 >>> [ 202.341161] sd 3:0:4:1: tag#473 Done: ADD_TO_MLQUEUE Result: >>> hostbyte=DID_OK driverbyte=DRIVER_OK >>> [ 202.341164] sd 3:0:4:1: tag#473 CDB: Test Unit Ready 00 00 00 00 00 00 >>> [ 202.341167] sd 3:0:4:1: tag#473 Sense Key : Unit Attention [current] >>> [ 202.341171] sd 3:0:4:1: tag#473 Add. Sense: Power on, reset, or bus >>> device reset occurred >>> [ 202.341173] sd 3:0:4:1: tag#473 scsi host busy 1 failed 0 >>> [ 202.341176] sd 3:0:4:1: tag#473 Inserting command ffff880422e7a440 >>> into mlqueue >>> >>> ... and that is the last ever heard of that device. >>> The 'device_busy' count remains at '1' and no further commands will be >>> sent to the device. >> >> If device_busy is at 1, then it should have a command pending. Where did >> you log this - it would be bandy if you attached whatever debug patch >> you put in, so we can see where the printks are coming from. If we get a >> BUSY with nothing pending, the driver should be ensuring that the queue >> gets run again later through blk_mq_delay_queue(), for instance. >> >> When the device is stuck, does it restart if you send it IO? >> > Meanwhile I've found it. > > Problem is that scsi_queue_rq() will not stop the queue when hitting a > busy condition before sending commands down to the driver, but still > calls blk_mq_delay_queue(): > > switch (ret) { > case BLK_MQ_RQ_QUEUE_BUSY: > if (atomic_read(&sdev->device_busy) == 0 && > !scsi_device_blocked(sdev)) > blk_mq_delay_queue(hctx, SCSI_QUEUE_DELAY); > break; > > As the queue isn't stopped blk_mq_delay_queue() won't do anything, > so queue_rq() will never be called. > I've send a patch to linux-scsi. > > BTW: Is it a hard requirement that the queue has to be stopped when > returning BLK_MQ_RQ_QUEUE_BUSY? No, but currently it is apparently a hard requirement that the queue be stopped when you call delay. Which does make sense, since there's little point in doing the delay if the queue is run anyway. > The comments indicate as such, but none of the drivers do so... > Also, blk_mq_delay_queue() is a bit odd, in that it'll only start > stopped hardware queues. I would at least document that the queue has to > be stopped when calling that. > Better still, can't we have blk_mq_delay_queue start the queues > unconditionally? I think so, it doesn't make sense to have blk_mq_delay_queue() NOT stop the queue, yet its own work handler requires it to be set to actually run. diff --git a/block/blk-mq.c b/block/blk-mq.c index fa1f8619bfe7..739a29208a63 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -1161,8 +1161,8 @@ static void blk_mq_delay_work_fn(struct work_struct *work) hctx = container_of(work, struct blk_mq_hw_ctx, delay_work.work); - if (test_and_clear_bit(BLK_MQ_S_STOPPED, &hctx->state)) - __blk_mq_run_hw_queue(hctx); + clear_bit(BLK_MQ_S_STOPPED, &hctx->state); + __blk_mq_run_hw_queue(hctx); } void blk_mq_delay_queue(struct blk_mq_hw_ctx *hctx, unsigned long msecs)