From patchwork Tue Jan 16 18:30:24 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bart Van Assche X-Patchwork-Id: 10167891 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 7574D601E7 for ; Tue, 16 Jan 2018 18:31:10 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6DC80201F3 for ; Tue, 16 Jan 2018 18:31:10 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 5E0A12674B; Tue, 16 Jan 2018 18:31:10 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.8 required=2.0 tests=BAYES_00,DKIM_SIGNED, RCVD_IN_DNSWL_HI,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 99467201F3 for ; Tue, 16 Jan 2018 18:31:09 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751070AbeAPSbH (ORCPT ); Tue, 16 Jan 2018 13:31:07 -0500 Received: from esa1.hgst.iphmx.com ([68.232.141.245]:54538 "EHLO esa1.hgst.iphmx.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751892AbeAPSaa (ORCPT ); Tue, 16 Jan 2018 13:30:30 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=wdc.com; i=@wdc.com; q=dns/txt; s=dkim.wdc.com; t=1516127430; x=1547663430; h=from:to:cc:subject:date:message-id; bh=W3S8gO+RtQQ9/MjMCvT/gew+OH7AFOXMIb2hDWfO6PA=; b=TpEjB9Qg/8/xzqzGKz1G6nPDxjQWQ3ZoyT/z+gxraThoCqUMHj9BusNE x5ZQcFiEswEI7ayUUwXM3Uk1bO3p5ZR1yKf3LN5CozaIvzUkIPubW8b51 KGCArT5Dy5kVUWmG3q52S7nfhJHFScP3hxQdOgItbUU8o1gAFurJ77mB0 tV0NnldaIQ7FkGwvvnlo6GD//lp8BPHmWDgQHAdGQwUphKyGi6g5yb8zM 4TpCB1gTIYsA6KwG1QrPVftcH9PhcjDyK46ffXPThnTJg0+01nDkJ8met KwvzaQsLNfBhUvuV7WsnSScWov8g7Qp0lzuzbHalih71/1xINaBgWPwy3 A==; X-IronPort-AV: E=Sophos;i="5.46,369,1511798400"; d="scan'208";a="172011877" Received: from h199-255-45-14.hgst.com (HELO uls-op-cesaep01.wdc.com) ([199.255.45.14]) by ob1.hgst.iphmx.com with ESMTP; 17 Jan 2018 02:30:25 +0800 Received: from uls-op-cesaip02.wdc.com ([10.248.3.37]) by uls-op-cesaep01.wdc.com with ESMTP; 16 Jan 2018 10:26:07 -0800 Received: from thinkpad-bart.sdcorp.global.sandisk.com (HELO thinkpad-bart.int.fusionio.com) ([10.11.171.236]) by uls-op-cesaip02.wdc.com with ESMTP; 16 Jan 2018 10:30:25 -0800 From: Bart Van Assche To: Jens Axboe Cc: linux-block@vger.kernel.org, Christoph Hellwig , Bart Van Assche , Omar Sandoval , Hannes Reinecke , Johannes Thumshirn Subject: [PATCH] blk-mq: Fix a race condition in blk_mq_mark_tag_wait() Date: Tue, 16 Jan 2018 10:30:24 -0800 Message-Id: <20180116183024.29225-1-bart.vanassche@wdc.com> X-Mailer: git-send-email 2.15.1 Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Because the hctx lock is not held around the only blk_mq_tag_wakeup_all() call in the block layer, the wait queue entry removal in blk_mq_dispatch_wake() is protected by the wait queue lock only. Since the hctx->dispatch_wait entry can occur on any of the SBQ_WAIT_QUEUES, the wait queue presence check, adding .dispatch_wait to a wait queue and removing the wait queue entry must all be protected by both the hctx lock and the wait queue lock. Signed-off-by: Bart Van Assche Cc: Christoph Hellwig Cc: Omar Sandoval Cc: Hannes Reinecke Cc: Johannes Thumshirn --- block/blk-mq.c | 31 ++++++++++++++++++------------- 1 file changed, 18 insertions(+), 13 deletions(-) diff --git a/block/blk-mq.c b/block/blk-mq.c index edb1291a42c5..6ca85052e63b 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -1109,7 +1109,7 @@ static bool blk_mq_mark_tag_wait(struct blk_mq_hw_ctx **hctx, struct blk_mq_hw_ctx *this_hctx = *hctx; struct sbq_wait_state *ws; wait_queue_entry_t *wait; - bool ret; + bool ret = false; if (!(this_hctx->flags & BLK_MQ_F_TAG_SHARED)) { if (!test_bit(BLK_MQ_S_SCHED_RESTART, &this_hctx->state)) @@ -1130,14 +1130,20 @@ static bool blk_mq_mark_tag_wait(struct blk_mq_hw_ctx **hctx, if (!list_empty_careful(&wait->entry)) return false; + ws = bt_wait_ptr(&this_hctx->tags->bitmap_tags, this_hctx); + + /* + * Since hctx->dispatch_wait can already be on any of the + * SBQ_WAIT_QUEUES number of wait queues, serialize the check and + * add_wait_queue() calls below with this_hctx->lock. + */ spin_lock(&this_hctx->lock); - if (!list_empty(&wait->entry)) { - spin_unlock(&this_hctx->lock); - return false; - } + spin_lock_irq(&ws->wait.lock); + if (!list_empty(&wait->entry)) + goto unlock; - ws = bt_wait_ptr(&this_hctx->tags->bitmap_tags, this_hctx); - add_wait_queue(&ws->wait, wait); + wait->flags &= ~WQ_FLAG_EXCLUSIVE; + __add_wait_queue(&ws->wait, wait); /* * It's possible that a tag was freed in the window between the @@ -1145,21 +1151,20 @@ static bool blk_mq_mark_tag_wait(struct blk_mq_hw_ctx **hctx, * queue. */ ret = blk_mq_get_driver_tag(rq, hctx, false); - if (!ret) { - spin_unlock(&this_hctx->lock); - return false; - } + if (!ret) + goto unlock; /* * We got a tag, remove ourselves from the wait queue to ensure * someone else gets the wakeup. */ - spin_lock_irq(&ws->wait.lock); list_del_init(&wait->entry); + +unlock: spin_unlock_irq(&ws->wait.lock); spin_unlock(&this_hctx->lock); - return true; + return ret; } bool blk_mq_dispatch_rq_list(struct request_queue *q, struct list_head *list,