From patchwork Tue Nov 26 17:56:54 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bart Van Assche X-Patchwork-Id: 11262933 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B3C52138C for ; Tue, 26 Nov 2019 17:57:08 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 9AAA220684 for ; Tue, 26 Nov 2019 17:57:08 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727126AbfKZR5I (ORCPT ); Tue, 26 Nov 2019 12:57:08 -0500 Received: from mail-pl1-f196.google.com ([209.85.214.196]:41186 "EHLO mail-pl1-f196.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726101AbfKZR5H (ORCPT ); Tue, 26 Nov 2019 12:57:07 -0500 Received: by mail-pl1-f196.google.com with SMTP id t8so8409561plr.8 for ; Tue, 26 Nov 2019 09:57:07 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=9tHZCi3NoBcxdk0wnNtr+GI/gJFOUJ2God40o2P9nxY=; b=Pn2j94X14kUHr7kz3npkpQT3RVLV4arObUq+B2K3dBpqoMPtuGAOXwcbUgDdXd+X9H p/u8qYsOuq81YD129YxypvkwsM0bIwRjdf1+FzFWB5v/nUWafx/yKCa7rheYqGekBYOh WcaFB4FNN5HV+e25YENP8bvYn9nZwNAqSijRih/LO+1stFIU+KwlC3/JcvbeOnA1HvYp ASRJ2zOxxGZ4q7Ud3QVZ7YSk2QvZtXP8NAgxZ25bfG7C04GHYQoZBFoeR0C8pWPdIjmR fQLjdoSdjLC+zTF+GLDY7UIPjYOXkgWlMa/YpQQDz3FrsBu1SFoaxyv3hAEWPiNuQSK2 QWQQ== X-Gm-Message-State: APjAAAVUBOfERMoVWZe6vbXKAJoVIE9R36Er1Twmtle1XXZL81m2iSM9 X1vI8Sh6ELxIbPbWNYIy4UU= X-Google-Smtp-Source: APXvYqwdaAqhai3oa+tBNYnXVvVVhObp940ZPbC4DfaGMu6WZOk6ZAsPKSdpUJ3IKSQeQ1u+40WRmA== X-Received: by 2002:a17:90a:195e:: with SMTP id 30mr425932pjh.60.1574791027311; Tue, 26 Nov 2019 09:57:07 -0800 (PST) Received: from desktop-bart.svl.corp.google.com ([2620:15c:2cd:202:4308:52a3:24b6:2c60]) by smtp.gmail.com with ESMTPSA id 82sm13178715pfa.115.2019.11.26.09.57.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 26 Nov 2019 09:57:06 -0800 (PST) From: Bart Van Assche To: Jens Axboe Cc: linux-block@vger.kernel.org, Christoph Hellwig , Bart Van Assche , John Garry , Christoph Hellwig , Ming Lei , Hannes Reinecke Subject: [PATCH 1/3] blk-mq: Remove some unused function arguments Date: Tue, 26 Nov 2019 09:56:54 -0800 Message-Id: <20191126175656.67638-2-bvanassche@acm.org> X-Mailer: git-send-email 2.24.0.432.g9d3f5f5b63-goog In-Reply-To: <20191126175656.67638-1-bvanassche@acm.org> References: <20191126175656.67638-1-bvanassche@acm.org> MIME-Version: 1.0 Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org From: John Garry The hardware queue argument in blk_mq_put_tag(), blk_mq_poll_nsecs(), and blk_mq_poll_hybrid_sleep() is unused, so remove it. Cc: Christoph Hellwig Cc: Ming Lei Cc: Hannes Reinecke Cc: John Garry Signed-off-by: John Garry [ bvanassche: edited patch description ] Signed-off-by: Bart Van Assche --- block/blk-mq-tag.c | 4 ++-- block/blk-mq-tag.h | 3 +-- block/blk-mq.c | 10 ++++------ block/blk-mq.h | 2 +- 4 files changed, 8 insertions(+), 11 deletions(-) diff --git a/block/blk-mq-tag.c b/block/blk-mq-tag.c index fbacde454718..586c9d6e904a 100644 --- a/block/blk-mq-tag.c +++ b/block/blk-mq-tag.c @@ -183,8 +183,8 @@ unsigned int blk_mq_get_tag(struct blk_mq_alloc_data *data) return tag + tag_offset; } -void blk_mq_put_tag(struct blk_mq_hw_ctx *hctx, struct blk_mq_tags *tags, - struct blk_mq_ctx *ctx, unsigned int tag) +void blk_mq_put_tag(struct blk_mq_tags *tags, struct blk_mq_ctx *ctx, + unsigned int tag) { if (!blk_mq_tag_is_reserved(tags, tag)) { const int real_tag = tag - tags->nr_reserved_tags; diff --git a/block/blk-mq-tag.h b/block/blk-mq-tag.h index 15bc74acb57e..d0c10d043891 100644 --- a/block/blk-mq-tag.h +++ b/block/blk-mq-tag.h @@ -26,8 +26,7 @@ extern struct blk_mq_tags *blk_mq_init_tags(unsigned int nr_tags, unsigned int r extern void blk_mq_free_tags(struct blk_mq_tags *tags); extern unsigned int blk_mq_get_tag(struct blk_mq_alloc_data *data); -extern void blk_mq_put_tag(struct blk_mq_hw_ctx *hctx, struct blk_mq_tags *tags, - struct blk_mq_ctx *ctx, unsigned int tag); +extern void blk_mq_put_tag(struct blk_mq_tags *tags, struct blk_mq_ctx *ctx, unsigned int tag); extern int blk_mq_tag_update_depth(struct blk_mq_hw_ctx *hctx, struct blk_mq_tags **tags, unsigned int depth, bool can_grow); diff --git a/block/blk-mq.c b/block/blk-mq.c index 323c9cb28066..fec4b82ff91c 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -477,9 +477,9 @@ static void __blk_mq_free_request(struct request *rq) blk_pm_mark_last_busy(rq); rq->mq_hctx = NULL; if (rq->tag != -1) - blk_mq_put_tag(hctx, hctx->tags, ctx, rq->tag); + blk_mq_put_tag(hctx->tags, ctx, rq->tag); if (sched_tag != -1) - blk_mq_put_tag(hctx, hctx->sched_tags, ctx, sched_tag); + blk_mq_put_tag(hctx->sched_tags, ctx, sched_tag); blk_mq_sched_restart(hctx); blk_queue_exit(q); } @@ -3340,7 +3340,6 @@ static void blk_mq_poll_stats_fn(struct blk_stat_callback *cb) } static unsigned long blk_mq_poll_nsecs(struct request_queue *q, - struct blk_mq_hw_ctx *hctx, struct request *rq) { unsigned long ret = 0; @@ -3373,7 +3372,6 @@ static unsigned long blk_mq_poll_nsecs(struct request_queue *q, } static bool blk_mq_poll_hybrid_sleep(struct request_queue *q, - struct blk_mq_hw_ctx *hctx, struct request *rq) { struct hrtimer_sleeper hs; @@ -3393,7 +3391,7 @@ static bool blk_mq_poll_hybrid_sleep(struct request_queue *q, if (q->poll_nsec > 0) nsecs = q->poll_nsec; else - nsecs = blk_mq_poll_nsecs(q, hctx, rq); + nsecs = blk_mq_poll_nsecs(q, rq); if (!nsecs) return false; @@ -3448,7 +3446,7 @@ static bool blk_mq_poll_hybrid(struct request_queue *q, return false; } - return blk_mq_poll_hybrid_sleep(q, hctx, rq); + return blk_mq_poll_hybrid_sleep(q, rq); } /** diff --git a/block/blk-mq.h b/block/blk-mq.h index eaaca8fc1c28..ba37cd64894c 100644 --- a/block/blk-mq.h +++ b/block/blk-mq.h @@ -199,7 +199,7 @@ static inline bool blk_mq_get_dispatch_budget(struct blk_mq_hw_ctx *hctx) static inline void __blk_mq_put_driver_tag(struct blk_mq_hw_ctx *hctx, struct request *rq) { - blk_mq_put_tag(hctx, hctx->tags, rq->mq_ctx, rq->tag); + blk_mq_put_tag(hctx->tags, rq->mq_ctx, rq->tag); rq->tag = -1; if (rq->rq_flags & RQF_MQ_INFLIGHT) { From patchwork Tue Nov 26 17:56:55 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bart Van Assche X-Patchwork-Id: 11262935 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 8198A138C for ; Tue, 26 Nov 2019 17:57:10 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 65E2420684 for ; Tue, 26 Nov 2019 17:57:10 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727165AbfKZR5J (ORCPT ); Tue, 26 Nov 2019 12:57:09 -0500 Received: from mail-pf1-f195.google.com ([209.85.210.195]:42861 "EHLO mail-pf1-f195.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727154AbfKZR5J (ORCPT ); Tue, 26 Nov 2019 12:57:09 -0500 Received: by mail-pf1-f195.google.com with SMTP id s5so9534079pfh.9 for ; Tue, 26 Nov 2019 09:57:08 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=oVAGa5wHGZVQTdnJfY1Fz6cSSmZXI39aRTwBcdpUujA=; b=S+rwZpC/ckr7Riva4s1rXDkzDY/JyVpJzw5xFQYCri7RhpW4i6wZ/odcFO/+fiPPwt R7ttjc9AK2+2pgG8BQQUCZsvqbY5dRXD3nw4UXuLVq7pD1yv/U3kPlWENOGqIqpJQEvq sFet7SCxgJVW/g9iggJwqpKHiUTvAbHh3FcbQVXsB7HjPHQ3rFO4s1/4AWUYzjvEXzqC tJJHl3qaUzRaS4ti7DomtrTM9kfGiNQI8BzszriJYYMahMAesUQVQEzIW0SfWEI95BMC 1o4jdB4Bjcr8DThp+Eestlpyi0xDTXWuAOzHxZnAhcXFlZVvMVK5F7sdl8uMEdaJ3tUC hAOg== X-Gm-Message-State: APjAAAVhA+pAzYU754GBfAcqEwZlugwxbHeTK0rMXw30KWnZuoTVN9Xg 6L9MoW84ktLO1rsaXfYfZlAlwPMV X-Google-Smtp-Source: APXvYqyiV0VDoZ5mpAefkcKB3/kgIKmua3DFH/oI2busNc2GeeUIBRLigF9vCnqISqkjBj5vyw82wg== X-Received: by 2002:a62:e914:: with SMTP id j20mr43546380pfh.245.1574791028508; Tue, 26 Nov 2019 09:57:08 -0800 (PST) Received: from desktop-bart.svl.corp.google.com ([2620:15c:2cd:202:4308:52a3:24b6:2c60]) by smtp.gmail.com with ESMTPSA id 82sm13178715pfa.115.2019.11.26.09.57.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 26 Nov 2019 09:57:07 -0800 (PST) From: Bart Van Assche To: Jens Axboe Cc: linux-block@vger.kernel.org, Christoph Hellwig , Bart Van Assche , Christoph Hellwig , Ming Lei , Hannes Reinecke , John Garry Subject: [PATCH 2/3] blk-mq: Move the TAG_ACTIVE and SCHED_RESTART flags from hctx into blk_mq_tags Date: Tue, 26 Nov 2019 09:56:55 -0800 Message-Id: <20191126175656.67638-3-bvanassche@acm.org> X-Mailer: git-send-email 2.24.0.432.g9d3f5f5b63-goog In-Reply-To: <20191126175656.67638-1-bvanassche@acm.org> References: <20191126175656.67638-1-bvanassche@acm.org> MIME-Version: 1.0 Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org If each hardware queue has its own tag set it's fine to manage these flags per hardware queue. Since the next patch will share tag sets across hardware queues, move these flags into blk_mq_tags. This patch does not change any functionality. Cc: Christoph Hellwig Cc: Ming Lei Cc: Hannes Reinecke Cc: John Garry Signed-off-by: Bart Van Assche --- block/blk-mq-debugfs.c | 2 -- block/blk-mq-sched.c | 8 ++++---- block/blk-mq-sched.h | 2 +- block/blk-mq-tag.c | 8 ++++---- block/blk-mq-tag.h | 10 ++++++++++ include/linux/blk-mq.h | 2 -- 6 files changed, 19 insertions(+), 13 deletions(-) diff --git a/block/blk-mq-debugfs.c b/block/blk-mq-debugfs.c index b3f2ba483992..3678e95ec947 100644 --- a/block/blk-mq-debugfs.c +++ b/block/blk-mq-debugfs.c @@ -211,8 +211,6 @@ static const struct blk_mq_debugfs_attr blk_mq_debugfs_queue_attrs[] = { #define HCTX_STATE_NAME(name) [BLK_MQ_S_##name] = #name static const char *const hctx_state_name[] = { HCTX_STATE_NAME(STOPPED), - HCTX_STATE_NAME(TAG_ACTIVE), - HCTX_STATE_NAME(SCHED_RESTART), }; #undef HCTX_STATE_NAME diff --git a/block/blk-mq-sched.c b/block/blk-mq-sched.c index ca22afd47b3d..7d98b6513148 100644 --- a/block/blk-mq-sched.c +++ b/block/blk-mq-sched.c @@ -64,18 +64,18 @@ void blk_mq_sched_assign_ioc(struct request *rq) */ void blk_mq_sched_mark_restart_hctx(struct blk_mq_hw_ctx *hctx) { - if (test_bit(BLK_MQ_S_SCHED_RESTART, &hctx->state)) + if (test_bit(BLK_MQ_T_SCHED_RESTART, &hctx->tags->state)) return; - set_bit(BLK_MQ_S_SCHED_RESTART, &hctx->state); + set_bit(BLK_MQ_T_SCHED_RESTART, &hctx->tags->state); } EXPORT_SYMBOL_GPL(blk_mq_sched_mark_restart_hctx); void blk_mq_sched_restart(struct blk_mq_hw_ctx *hctx) { - if (!test_bit(BLK_MQ_S_SCHED_RESTART, &hctx->state)) + if (!test_bit(BLK_MQ_T_SCHED_RESTART, &hctx->tags->state)) return; - clear_bit(BLK_MQ_S_SCHED_RESTART, &hctx->state); + clear_bit(BLK_MQ_T_SCHED_RESTART, &hctx->tags->state); blk_mq_run_hw_queue(hctx, true); } diff --git a/block/blk-mq-sched.h b/block/blk-mq-sched.h index 126021fc3a11..15174a646468 100644 --- a/block/blk-mq-sched.h +++ b/block/blk-mq-sched.h @@ -82,7 +82,7 @@ static inline bool blk_mq_sched_has_work(struct blk_mq_hw_ctx *hctx) static inline bool blk_mq_sched_needs_restart(struct blk_mq_hw_ctx *hctx) { - return test_bit(BLK_MQ_S_SCHED_RESTART, &hctx->state); + return test_bit(BLK_MQ_T_SCHED_RESTART, &hctx->tags->state); } #endif diff --git a/block/blk-mq-tag.c b/block/blk-mq-tag.c index 586c9d6e904a..a60e1b4a8158 100644 --- a/block/blk-mq-tag.c +++ b/block/blk-mq-tag.c @@ -23,8 +23,8 @@ */ bool __blk_mq_tag_busy(struct blk_mq_hw_ctx *hctx) { - if (!test_bit(BLK_MQ_S_TAG_ACTIVE, &hctx->state) && - !test_and_set_bit(BLK_MQ_S_TAG_ACTIVE, &hctx->state)) + if (!test_bit(BLK_MQ_T_ACTIVE, &hctx->tags->state) && + !test_and_set_bit(BLK_MQ_T_ACTIVE, &hctx->tags->state)) atomic_inc(&hctx->tags->active_queues); return true; @@ -48,7 +48,7 @@ void __blk_mq_tag_idle(struct blk_mq_hw_ctx *hctx) { struct blk_mq_tags *tags = hctx->tags; - if (!test_and_clear_bit(BLK_MQ_S_TAG_ACTIVE, &hctx->state)) + if (!test_and_clear_bit(BLK_MQ_T_ACTIVE, &hctx->tags->state)) return; atomic_dec(&tags->active_queues); @@ -67,7 +67,7 @@ static inline bool hctx_may_queue(struct blk_mq_hw_ctx *hctx, if (!hctx || !(hctx->flags & BLK_MQ_F_TAG_SHARED)) return true; - if (!test_bit(BLK_MQ_S_TAG_ACTIVE, &hctx->state)) + if (!test_bit(BLK_MQ_T_ACTIVE, &hctx->tags->state)) return true; /* diff --git a/block/blk-mq-tag.h b/block/blk-mq-tag.h index d0c10d043891..f75fa936b090 100644 --- a/block/blk-mq-tag.h +++ b/block/blk-mq-tag.h @@ -4,6 +4,11 @@ #include "blk-mq.h" +enum { + BLK_MQ_T_ACTIVE = 1, + BLK_MQ_T_SCHED_RESTART = 2, +}; + /* * Tag address space map. */ @@ -11,6 +16,11 @@ struct blk_mq_tags { unsigned int nr_tags; unsigned int nr_reserved_tags; + /** + * @state: BLK_MQ_T_* flags. Defines the state of the hw + * queue (active, scheduled to restart). + */ + unsigned long state; atomic_t active_queues; struct sbitmap_queue bitmap_tags; diff --git a/include/linux/blk-mq.h b/include/linux/blk-mq.h index 11cfd6470b1a..522631d108af 100644 --- a/include/linux/blk-mq.h +++ b/include/linux/blk-mq.h @@ -394,8 +394,6 @@ enum { BLK_MQ_F_ALLOC_POLICY_BITS = 1, BLK_MQ_S_STOPPED = 0, - BLK_MQ_S_TAG_ACTIVE = 1, - BLK_MQ_S_SCHED_RESTART = 2, BLK_MQ_MAX_DEPTH = 10240, From patchwork Tue Nov 26 17:56:56 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bart Van Assche X-Patchwork-Id: 11262937 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 02603930 for ; Tue, 26 Nov 2019 17:57:12 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id D3A3F20835 for ; Tue, 26 Nov 2019 17:57:11 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727154AbfKZR5L (ORCPT ); Tue, 26 Nov 2019 12:57:11 -0500 Received: from mail-pj1-f68.google.com ([209.85.216.68]:45003 "EHLO mail-pj1-f68.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725895AbfKZR5L (ORCPT ); Tue, 26 Nov 2019 12:57:11 -0500 Received: by mail-pj1-f68.google.com with SMTP id w8so8620759pjh.11 for ; Tue, 26 Nov 2019 09:57:10 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=6sYX/PuzZ4W11V+bVaVB6YqBrthJITB36bhwtzYqCfs=; b=Vp2J4rlSwSNf6ERqmFO79UDoWjrpxfFSjbZSZFUxb3YQsX/mO2dy7anOataAwF/ZWJ NwAGk0DOuZB3vk0Je5qeLL+LchAMU/zPrQWeKsZygf0mfIIjeUZHCioF3xOgfteSF13v xQ8hsdzEmB7WkblqgRMqXMhJ7h+wjtG9KdvBO0fg3sL1smZVhfUBYMY8GVWP+67wQZFY xpPS3bsF3u7Vl1yJaALuS8HO8ObJUK/sjsZLRczS6Hk+vtHbEjGDn51uZxLJYyF/0/+A mqVpQatqn9UCnw5pcWL0gXElXMS4BUMa4sy0X2lWo8hQHnIRCDvbW9R6UxmDtz2AvacK YuBg== X-Gm-Message-State: APjAAAV+cDEoQnH4AUctjhcnAFvs9IXxbRKsfBilsPonC8trCRci6LdF kFY9lyxCxFQv062t863IGehOrSOH X-Google-Smtp-Source: APXvYqwAdK3aCN1wCdtVq0MtB1PeDVcZsNUpCP8HuLehn2x3QRFpHHYrm49HbKp07az6EmgpUT9J2g== X-Received: by 2002:a17:90a:a616:: with SMTP id c22mr429274pjq.46.1574791029692; Tue, 26 Nov 2019 09:57:09 -0800 (PST) Received: from desktop-bart.svl.corp.google.com ([2620:15c:2cd:202:4308:52a3:24b6:2c60]) by smtp.gmail.com with ESMTPSA id 82sm13178715pfa.115.2019.11.26.09.57.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 26 Nov 2019 09:57:08 -0800 (PST) From: Bart Van Assche To: Jens Axboe Cc: linux-block@vger.kernel.org, Christoph Hellwig , Bart Van Assche , Christoph Hellwig , Ming Lei , Hannes Reinecke , John Garry Subject: [PATCH 3/3] block: Add support for sharing tags across hardware queues Date: Tue, 26 Nov 2019 09:56:56 -0800 Message-Id: <20191126175656.67638-4-bvanassche@acm.org> X-Mailer: git-send-email 2.24.0.432.g9d3f5f5b63-goog In-Reply-To: <20191126175656.67638-1-bvanassche@acm.org> References: <20191126175656.67638-1-bvanassche@acm.org> MIME-Version: 1.0 Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org Add a boolean member 'share_tags' in struct blk_mq_tag_set. If that member variable is set, make all hctx->tags[] pointers identical. Implement the necessary changes in the functions that allocate, free and resize tag sets. Modify blk_mq_tagset_busy_iter() such that it continues to call the callback function once per request. Modify blk_mq_queue_tag_busy_iter() such that the callback function is only called with the correct hctx as first argument. Modify the debugfs code such that it keeps showing only matching tags per hctx. This patch has been tested by running blktests on top of a kernel that includes the following change to enable shared tags for all block drivers except the NVMe drivers: diff --git a/block/blk-mq.c b/block/blk-mq.c @@ -3037,6 +3037,10 @@ int blk_mq_alloc_tag_set(struct blk_mq_tag_set *set) BUILD_BUG_ON(BLK_MQ_MAX_DEPTH > 1 << BLK_MQ_UNIQUE_TAG_BITS); + /* Test code: enable tag sharing for all block drivers except NVMe */ + if (!set->ops->poll) + set->share_tags = true; + if (!set->nr_hw_queues) return -EINVAL; if (!set->queue_depth) Cc: Christoph Hellwig Cc: Ming Lei Cc: Hannes Reinecke Cc: John Garry Signed-off-by: Bart Van Assche --- block/blk-mq-debugfs.c | 40 ++++++++++++++++++++++++++++++++++++++-- block/blk-mq-tag.c | 7 +++++-- block/blk-mq.c | 28 +++++++++++++++++++++------- include/linux/blk-mq.h | 8 ++++++-- 4 files changed, 70 insertions(+), 13 deletions(-) diff --git a/block/blk-mq-debugfs.c b/block/blk-mq-debugfs.c index 3678e95ec947..653e80ede3bd 100644 --- a/block/blk-mq-debugfs.c +++ b/block/blk-mq-debugfs.c @@ -472,20 +472,56 @@ static int hctx_tags_show(void *data, struct seq_file *m) return res; } +struct hctx_sb_data { + struct sbitmap *sb; /* output bitmap */ + struct blk_mq_hw_ctx *hctx; /* input hctx */ +}; + +static bool hctx_filter_fn(struct blk_mq_hw_ctx *hctx, struct request *req, + void *priv, bool reserved) +{ + struct hctx_sb_data *hctx_sb_data = priv; + + if (hctx == hctx_sb_data->hctx) + sbitmap_set_bit(hctx_sb_data->sb, req->tag); + return true; +} + +static void hctx_filter_sb(struct sbitmap *sb, struct blk_mq_hw_ctx *hctx) +{ + struct hctx_sb_data hctx_sb_data = { .sb = sb, .hctx = hctx }; + + blk_mq_queue_tag_busy_iter(hctx->queue, hctx_filter_fn, &hctx_sb_data); +} + static int hctx_tags_bitmap_show(void *data, struct seq_file *m) { struct blk_mq_hw_ctx *hctx = data; struct request_queue *q = hctx->queue; + struct sbitmap sb, *hctx_sb; int res; + if (!hctx->tags) + return 0; + hctx_sb = &hctx->tags->bitmap_tags.sb; + res = sbitmap_init_node(&sb, hctx_sb->depth, hctx_sb->shift, GFP_KERNEL, + NUMA_NO_NODE); + if (res) + return res; + res = mutex_lock_interruptible(&q->sysfs_lock); if (res) goto out; - if (hctx->tags) - sbitmap_bitmap_show(&hctx->tags->bitmap_tags.sb, m); + /* + * If tags are shared across hardware queues, hctx_sb contains tags + * for multiple hardware queues. Filter the tags for 'hctx' into 'sb'. + */ + hctx_filter_sb(&sb, hctx); mutex_unlock(&q->sysfs_lock); + sbitmap_bitmap_show(&sb, m); out: + sbitmap_free(&sb); return res; } diff --git a/block/blk-mq-tag.c b/block/blk-mq-tag.c index a60e1b4a8158..770fe2324230 100644 --- a/block/blk-mq-tag.c +++ b/block/blk-mq-tag.c @@ -220,7 +220,7 @@ static bool bt_iter(struct sbitmap *bitmap, unsigned int bitnr, void *data) * We can hit rq == NULL here, because the tagging functions * test and set the bit before assigning ->rqs[]. */ - if (rq && rq->q == hctx->queue) + if (rq && rq->q == hctx->queue && rq->mq_hctx == hctx) return iter_data->fn(hctx, rq, iter_data->data, reserved); return true; } @@ -341,8 +341,11 @@ void blk_mq_tagset_busy_iter(struct blk_mq_tag_set *tagset, int i; for (i = 0; i < tagset->nr_hw_queues; i++) { - if (tagset->tags && tagset->tags[i]) + if (tagset->tags && tagset->tags[i]) { blk_mq_all_tag_busy_iter(tagset->tags[i], fn, priv); + if (tagset->share_tags) + break; + } } } EXPORT_SYMBOL(blk_mq_tagset_busy_iter); diff --git a/block/blk-mq.c b/block/blk-mq.c index fec4b82ff91c..fa4cfc4b7e7c 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -2404,6 +2404,12 @@ static bool __blk_mq_alloc_rq_map(struct blk_mq_tag_set *set, int hctx_idx) { int ret = 0; + if (hctx_idx > 0 && set->share_tags) { + WARN_ON_ONCE(!set->tags[0]); + set->tags[hctx_idx] = set->tags[0]; + return 0; + } + set->tags[hctx_idx] = blk_mq_alloc_rq_map(set, hctx_idx, set->queue_depth, set->reserved_tags); if (!set->tags[hctx_idx]) @@ -2423,8 +2429,10 @@ static void blk_mq_free_map_and_requests(struct blk_mq_tag_set *set, unsigned int hctx_idx) { if (set->tags && set->tags[hctx_idx]) { - blk_mq_free_rqs(set, set->tags[hctx_idx], hctx_idx); - blk_mq_free_rq_map(set->tags[hctx_idx]); + if (hctx_idx == 0 || !set->share_tags) { + blk_mq_free_rqs(set, set->tags[hctx_idx], hctx_idx); + blk_mq_free_rq_map(set->tags[hctx_idx]); + } set->tags[hctx_idx] = NULL; } } @@ -2568,7 +2576,7 @@ static void blk_mq_del_queue_tag_set(struct request_queue *q) mutex_lock(&set->tag_list_lock); list_del_rcu(&q->tag_set_list); - if (list_is_singular(&set->tag_list)) { + if (list_is_singular(&set->tag_list) && !set->share_tags) { /* just transitioned to unshared */ set->flags &= ~BLK_MQ_F_TAG_SHARED; /* update existing queue */ @@ -2586,7 +2594,7 @@ static void blk_mq_add_queue_tag_set(struct blk_mq_tag_set *set, /* * Check to see if we're transitioning to shared (from 1 to 2 queues). */ - if (!list_empty(&set->tag_list) && + if ((!list_empty(&set->tag_list) || set->share_tags) && !(set->flags & BLK_MQ_F_TAG_SHARED)) { set->flags |= BLK_MQ_F_TAG_SHARED; /* update existing queue */ @@ -2911,15 +2919,21 @@ static int __blk_mq_alloc_rq_maps(struct blk_mq_tag_set *set) { int i; - for (i = 0; i < set->nr_hw_queues; i++) - if (!__blk_mq_alloc_rq_map(set, i)) + for (i = 0; i < set->nr_hw_queues; i++) { + if (i > 0 && set->share_tags) { + set->tags[i] = set->tags[0]; + } else if (!__blk_mq_alloc_rq_map(set, i)) goto out_unwind; + } return 0; out_unwind: - while (--i >= 0) + while (--i >= 0) { + if (i > 0 && set->share_tags) + continue; blk_mq_free_rq_map(set->tags[i]); + } return -ENOMEM; } diff --git a/include/linux/blk-mq.h b/include/linux/blk-mq.h index 522631d108af..dd5517476314 100644 --- a/include/linux/blk-mq.h +++ b/include/linux/blk-mq.h @@ -224,10 +224,13 @@ enum hctx_type { * @numa_node: NUMA node the storage adapter has been connected to. * @timeout: Request processing timeout in jiffies. * @flags: Zero or more BLK_MQ_F_* flags. + * @share_tags: Whether or not to share one tag set across hardware queues. * @driver_data: Pointer to data owned by the block driver that created this * tag set. - * @tags: Tag sets. One tag set per hardware queue. Has @nr_hw_queues - * elements. + * @tags: Array of tag set pointers. Has @nr_hw_queues elements. If + * share_tags has not been set, all tag set pointers are + * different. If share_tags has been set, all tag_set pointers + * are identical. * @tag_list_lock: Serializes tag_list accesses. * @tag_list: List of the request queues that use this tag set. See also * request_queue.tag_set_list. @@ -243,6 +246,7 @@ struct blk_mq_tag_set { int numa_node; unsigned int timeout; unsigned int flags; + bool share_tags; void *driver_data; struct blk_mq_tags **tags;