From patchwork Wed Jan 10 18:04:28 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bart Van Assche X-Patchwork-Id: 10155777 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id B57AF602D8 for ; Wed, 10 Jan 2018 18:04:32 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id AACE020499 for ; Wed, 10 Jan 2018 18:04:32 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 9D4AF20856; Wed, 10 Jan 2018 18:04:32 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.8 required=2.0 tests=BAYES_00,DKIM_SIGNED, RCVD_IN_DNSWL_HI,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 2AFD328522 for ; Wed, 10 Jan 2018 18:04:31 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751891AbeAJSEa (ORCPT ); Wed, 10 Jan 2018 13:04:30 -0500 Received: from esa6.hgst.iphmx.com ([216.71.154.45]:32474 "EHLO esa6.hgst.iphmx.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751831AbeAJSE3 (ORCPT ); Wed, 10 Jan 2018 13:04:29 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=wdc.com; i=@wdc.com; q=dns/txt; s=dkim.wdc.com; t=1515607469; x=1547143469; h=from:to:cc:subject:date:message-id; bh=vI9N3Zoo7J8AXnquMty2HL2EOD0Cn+ubPuxrIgh2XlY=; b=eVzThbxgRj0Yej6MIUr5V3IfJR+66Bteov3yBFFbu7+jlCn2R01FW2Ba CBBJui5XnZOzIynqz3QeLV8DA+64UWbdee92TvxmB+PqhOI5k43NJ55pO idUSc5GdPgpmSd6EvcnAb5HdSs/zCrL7BksHfgxO38rmxrILtXQiMDSrJ a90Rj3DzunM8Tm5z7dqx8HGdZkmLsXMx4Zo23G35Rk0ubv7X+mL8+GO0L RqjdfQSMcwpyZRbH6xBLEw2FCmlAZc4bSTfuL6TIidcsQCbtyqIO0aKvz dmWx3gPYMqEE7afhzPX5dR63TCVF8qRsGWMQyWRtmfH5Y6fgsyroOZAQ2 g==; X-IronPort-AV: E=Sophos;i="5.46,341,1511798400"; d="scan'208";a="68906944" Received: from h199-255-45-15.hgst.com (HELO uls-op-cesaep02.wdc.com) ([199.255.45.15]) by ob1.hgst.iphmx.com with ESMTP; 11 Jan 2018 02:04:29 +0800 Received: from uls-op-cesaip02.wdc.com ([10.248.3.37]) by uls-op-cesaep02.wdc.com with ESMTP; 10 Jan 2018 10:00:00 -0800 Received: from thinkpad-bart.sdcorp.global.sandisk.com (HELO thinkpad-bart.int.fusionio.com) ([10.11.171.236]) by uls-op-cesaip02.wdc.com with ESMTP; 10 Jan 2018 10:04:29 -0800 From: Bart Van Assche To: Jens Axboe Cc: linux-block@vger.kernel.org, Christoph Hellwig , Bart Van Assche , Omar Sandoval , Hannes Reinecke , Johannes Thumshirn Subject: [PATCH] blk-mq: Rework blk_mq_mark_tag_wait() Date: Wed, 10 Jan 2018 10:04:28 -0800 Message-Id: <20180110180428.22587-1-bart.vanassche@wdc.com> X-Mailer: git-send-email 2.15.1 Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Use prepare_to_wait() and finish_wait() instead of open-coding these functions. Reduce the number of if-statements to make blk_mq_mark_tag_wait() easier to read. Both add_wait_queue() and blk_mq_dispatch_wake() protect wait queue manipulations with the wait queue lock. Hence also protect the !list_empty(&wait->entry) test with the wait queue lock instead of the hctx lock. Note: a side effect of this patch is that the task state is changed for shared queues before and after the blk_mq_get_driver_tag(). Since blk_mq_dispatch_wake() ignores the task state these task state changes do not affect which task gets woken up. See also commit f906a6a0f426 ("blk-mq: improve tag waiting setup for non-shared tags"). Signed-off-by: Bart Van Assche Cc: Christoph Hellwig Cc: Omar Sandoval Cc: Hannes Reinecke Cc: Johannes Thumshirn --- block/blk-mq.c | 49 +++++++++++++++++-------------------------------- 1 file changed, 17 insertions(+), 32 deletions(-) diff --git a/block/blk-mq.c b/block/blk-mq.c index 61bdbf45c3be..29f140b4dbf7 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -1121,50 +1121,35 @@ static bool blk_mq_mark_tag_wait(struct blk_mq_hw_ctx **hctx, if (!shared_tags) { if (!test_bit(BLK_MQ_S_SCHED_RESTART, &this_hctx->state)) set_bit(BLK_MQ_S_SCHED_RESTART, &this_hctx->state); + ret = blk_mq_get_driver_tag(rq, hctx, false); + /* + * Don't clear RESTART here, someone else could have set it. + * At most this will cost an extra queue run. + */ } else { wait = &this_hctx->dispatch_wait; if (!list_empty_careful(&wait->entry)) return false; - spin_lock(&this_hctx->lock); - if (!list_empty(&wait->entry)) { - spin_unlock(&this_hctx->lock); - return false; - } - ws = bt_wait_ptr(&this_hctx->tags->bitmap_tags, this_hctx); - add_wait_queue(&ws->wait, wait); - } - - /* - * It's possible that a tag was freed in the window between the - * allocation failure and adding the hardware queue to the wait - * queue. - */ - ret = blk_mq_get_driver_tag(rq, hctx, false); + prepare_to_wait(&ws->wait, wait, TASK_UNINTERRUPTIBLE); - if (!shared_tags) { /* - * Don't clear RESTART here, someone else could have set it. - * At most this will cost an extra queue run. + * It's possible that a tag was freed in the window between the + * allocation failure and adding the hardware queue to the wait + * queue. */ - return ret; - } else { - if (!ret) { - spin_unlock(&this_hctx->lock); - return false; - } - + ret = blk_mq_get_driver_tag(rq, hctx, false); /* - * We got a tag, remove ourselves from the wait queue to ensure - * someone else gets the wakeup. + * If we got a tag, remove ourselves from the wait queue to + * ensure someone else gets the wakeup. */ - spin_lock_irq(&ws->wait.lock); - list_del_init(&wait->entry); - spin_unlock_irq(&ws->wait.lock); - spin_unlock(&this_hctx->lock); - return true; + if (ret) + finish_wait(&ws->wait, wait); + else + __set_current_state(TASK_RUNNING); } + return ret; } bool blk_mq_dispatch_rq_list(struct request_queue *q, struct list_head *list,