From patchwork Fri Jul 28 20:14:47 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Keith Busch X-Patchwork-Id: 13332629 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E206BEB64DD for ; Fri, 28 Jul 2023 20:15:49 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229488AbjG1UPt (ORCPT ); Fri, 28 Jul 2023 16:15:49 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35260 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233061AbjG1UPs (ORCPT ); Fri, 28 Jul 2023 16:15:48 -0400 Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com [67.231.145.42]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8D46C423B for ; Fri, 28 Jul 2023 13:15:46 -0700 (PDT) Received: from pps.filterd (m0109333.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 36SJUEeB010680 for ; Fri, 28 Jul 2023 13:15:46 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=meta.com; h=from : to : cc : subject : date : message-id : mime-version : content-transfer-encoding : content-type; s=s2048-2021-q4; bh=HdLTZx3ywuV58VYqToINJa02yzTgr3TXTJwla58Xm/4=; b=AeVpb8S/iPtsZsbESzhMlmiY4YHyMmYmUnnCzXTJZpAsMgwkYcpQ6o7cZ10n6LH+6uKX JNqe5/IhXJ5/FnpRZKGCHvENGlo0yM3FtWbJfUChf0cXlBtlWKftEzkR3Dp5al1ivS4Y Cu7CdrEqGSZItpKoCMjDLK7NdkTC8GeVHN/bS0fiTaANdFkQdRUBCzqbVFJnUggERfqx N+dPJsJpGK4yZuwoLlK24wing7buEPukmm/98PILpUHwM+J8/nOcPINYSwYgr0oT/NUC 7wEKFCnXoTu0Iy6epGDAQ0SuAmDGZK4iJYIctcY7w7nFuKtv4ZRVXcl0+Np6vH6idoj5 wQ== Received: from mail.thefacebook.com ([163.114.132.120]) by mx0a-00082601.pphosted.com (PPS) with ESMTPS id 3s49g0wxfj-2 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Fri, 28 Jul 2023 13:15:46 -0700 Received: from twshared10975.09.ash9.facebook.com (2620:10d:c085:208::f) by mail.thefacebook.com (2620:10d:c085:21d::5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.27; Fri, 28 Jul 2023 13:15:44 -0700 Received: by devbig007.nao1.facebook.com (Postfix, from userid 544533) id B3C801C47B2B5; Fri, 28 Jul 2023 13:14:50 -0700 (PDT) From: Keith Busch To: , , , CC: Keith Busch Subject: [PATCH 1/3] io_uring: split req init from submit Date: Fri, 28 Jul 2023 13:14:47 -0700 Message-ID: <20230728201449.3350962-1-kbusch@meta.com> X-Mailer: git-send-email 2.34.1 MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-ORIG-GUID: tRTx517z3bdh_Svg9bQC56FITLQpdaU8 X-Proofpoint-GUID: tRTx517z3bdh_Svg9bQC56FITLQpdaU8 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.254,Aquarius:18.0.957,Hydra:6.0.591,FMLib:17.11.176.26 definitions=2023-07-27_10,2023-07-26_01,2023-05-22_02 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org From: Keith Busch Split the req initialization and link handling from the submit. This simplifies the submit path since everything that can fail is separate from it, and makes it easier to create batched submissions later. Signed-off-by: Keith Busch --- io_uring/io_uring.c | 66 +++++++++++++++++++++++++-------------------- 1 file changed, 37 insertions(+), 29 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index d585171560ce5..818b2d1661c5e 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -2279,18 +2279,20 @@ static __cold int io_submit_fail_init(const struct io_uring_sqe *sqe, return 0; } -static inline int io_submit_sqe(struct io_ring_ctx *ctx, struct io_kiocb *req, - const struct io_uring_sqe *sqe) - __must_hold(&ctx->uring_lock) +static inline void io_submit_sqe(struct io_kiocb *req) { - struct io_submit_link *link = &ctx->submit_state.link; - int ret; + trace_io_uring_submit_req(req); - ret = io_init_req(ctx, req, sqe); - if (unlikely(ret)) - return io_submit_fail_init(sqe, req, ret); + if (unlikely(req->flags & (REQ_F_FORCE_ASYNC | REQ_F_FAIL))) + io_queue_sqe_fallback(req); + else + io_queue_sqe(req); +} - trace_io_uring_submit_req(req); +static int io_setup_link(struct io_submit_link *link, struct io_kiocb **orig) +{ + struct io_kiocb *req = *orig; + int ret; /* * If we already have a head request, queue this one for async @@ -2300,35 +2302,28 @@ static inline int io_submit_sqe(struct io_ring_ctx *ctx, struct io_kiocb *req, * conditions are true (normal request), then just queue it. */ if (unlikely(link->head)) { + *orig = NULL; + ret = io_req_prep_async(req); if (unlikely(ret)) - return io_submit_fail_init(sqe, req, ret); + return ret; trace_io_uring_link(req, link->head); link->last->link = req; link->last = req; - if (req->flags & IO_REQ_LINK_FLAGS) return 0; + /* last request of the link, flush it */ - req = link->head; + *orig = link->head; link->head = NULL; - if (req->flags & (REQ_F_FORCE_ASYNC | REQ_F_FAIL)) - goto fallback; - - } else if (unlikely(req->flags & (IO_REQ_LINK_FLAGS | - REQ_F_FORCE_ASYNC | REQ_F_FAIL))) { - if (req->flags & IO_REQ_LINK_FLAGS) { - link->head = req; - link->last = req; - } else { -fallback: - io_queue_sqe_fallback(req); - } - return 0; + } else if (unlikely(req->flags & IO_REQ_LINK_FLAGS)) { + link->head = req; + link->last = req; + *orig = NULL; + return 0; } - io_queue_sqe(req); return 0; } @@ -2412,9 +2407,10 @@ static bool io_get_sqe(struct io_ring_ctx *ctx, const struct io_uring_sqe **sqe) int io_submit_sqes(struct io_ring_ctx *ctx, unsigned int nr) __must_hold(&ctx->uring_lock) { + struct io_submit_link *link = &ctx->submit_state.link; unsigned int entries = io_sqring_entries(ctx); unsigned int left; - int ret; + int ret, err; if (unlikely(!entries)) return 0; @@ -2434,12 +2430,24 @@ int io_submit_sqes(struct io_ring_ctx *ctx, unsigned int nr) break; } + err = io_init_req(ctx, req, sqe); + if (unlikely(err)) + goto error; + + err = io_setup_link(link, &req); + if (unlikely(err)) + goto error; + + if (likely(req)) + io_submit_sqe(req); + continue; +error: /* * Continue submitting even for sqe failure if the * ring was setup with IORING_SETUP_SUBMIT_ALL */ - if (unlikely(io_submit_sqe(ctx, req, sqe)) && - !(ctx->flags & IORING_SETUP_SUBMIT_ALL)) { + err = io_submit_fail_init(sqe, req, err); + if (err && !(ctx->flags & IORING_SETUP_SUBMIT_ALL)) { left--; break; } From patchwork Fri Jul 28 20:14:48 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Keith Busch X-Patchwork-Id: 13332639 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B0B58C0015E for ; Fri, 28 Jul 2023 20:20:14 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231468AbjG1UUN (ORCPT ); Fri, 28 Jul 2023 16:20:13 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37944 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230381AbjG1UUK (ORCPT ); Fri, 28 Jul 2023 16:20:10 -0400 Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com [67.231.145.42]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 477DE4695 for ; Fri, 28 Jul 2023 13:19:45 -0700 (PDT) Received: from pps.filterd (m0044012.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 36SK148b005583 for ; Fri, 28 Jul 2023 13:19:13 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=meta.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=s2048-2021-q4; bh=kJWrtTkStC6cxrlTIyl+TkH/zRRo0Y8FfDLQItMN1pw=; b=S6aFzlxqTpjTDSlxeMmrrDOTp/L3GhXwphRMBIU2ZxJwYUA/YeV93tBH99lFqzB1dbun acVYJt+rldJSYhO45l5G3njly3e9yjaxW1IalSJNDOwalwsS3qkhRLI+9MKgKZUB2CNx wVJ6djLSf1W2CvI3Y/hp6ZZdDk+Y+BmaEpTwmcKmSg+iEevwmXWghQcMoVzIT7O7Y8RV 0wURBHveBthaO1vk64Wy4jifboyfPeVUIOpxzo4xjkk7e8jm8sYOtTa0RIMmrVNP9hpG SuvoURbBolZacf5CAShlbn5N1yiQRtwo0tUQ7/hcMBIsoPSjNprU7r+RMRN6kp1SovCN PQ== Received: from maileast.thefacebook.com ([163.114.130.16]) by mx0a-00082601.pphosted.com (PPS) with ESMTPS id 3s3v48n11j-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Fri, 28 Jul 2023 13:19:13 -0700 Received: from twshared19625.39.frc1.facebook.com (2620:10d:c0a8:1b::30) by mail.thefacebook.com (2620:10d:c0a8:82::d) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.27; Fri, 28 Jul 2023 13:19:11 -0700 Received: by devbig007.nao1.facebook.com (Postfix, from userid 544533) id 1C32C1C47B5D5; Fri, 28 Jul 2023 13:15:57 -0700 (PDT) From: Keith Busch To: , , , CC: Keith Busch Subject: [PATCH 2/3] io_uring: split req prep and submit loops Date: Fri, 28 Jul 2023 13:14:48 -0700 Message-ID: <20230728201449.3350962-2-kbusch@meta.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230728201449.3350962-1-kbusch@meta.com> References: <20230728201449.3350962-1-kbusch@meta.com> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-GUID: 5il7Z9WEoZJ9fGsefnA23Gsb3v2wQBEV X-Proofpoint-ORIG-GUID: 5il7Z9WEoZJ9fGsefnA23Gsb3v2wQBEV X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.254,Aquarius:18.0.957,Hydra:6.0.591,FMLib:17.11.176.26 definitions=2023-07-27_10,2023-07-26_01,2023-05-22_02 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org From: Keith Busch Do all the prep work up front, then dispatch all synchronous requests at once. This will make it easier to count batches for plugging. Signed-off-by: Keith Busch --- io_uring/io_uring.c | 26 ++++++++++++++++++-------- io_uring/slist.h | 4 ++++ 2 files changed, 22 insertions(+), 8 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 818b2d1661c5e..5434aef0a8ef7 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -1082,6 +1082,7 @@ static void io_preinit_req(struct io_kiocb *req, struct io_ring_ctx *ctx) req->ctx = ctx; req->link = NULL; req->async_data = NULL; + req->comp_list.next = NULL; /* not necessary, but safer to zero */ req->cqe.res = 0; } @@ -2282,11 +2283,7 @@ static __cold int io_submit_fail_init(const struct io_uring_sqe *sqe, static inline void io_submit_sqe(struct io_kiocb *req) { trace_io_uring_submit_req(req); - - if (unlikely(req->flags & (REQ_F_FORCE_ASYNC | REQ_F_FAIL))) - io_queue_sqe_fallback(req); - else - io_queue_sqe(req); + io_queue_sqe(req); } static int io_setup_link(struct io_submit_link *link, struct io_kiocb **orig) @@ -2409,6 +2406,9 @@ int io_submit_sqes(struct io_ring_ctx *ctx, unsigned int nr) { struct io_submit_link *link = &ctx->submit_state.link; unsigned int entries = io_sqring_entries(ctx); + struct io_wq_work_node *pos, *next; + struct io_wq_work_list req_list; + struct io_kiocb *req; unsigned int left; int ret, err; @@ -2419,6 +2419,7 @@ int io_submit_sqes(struct io_ring_ctx *ctx, unsigned int nr) io_get_task_refs(left); io_submit_state_start(&ctx->submit_state, left); + INIT_WQ_LIST(&req_list); do { const struct io_uring_sqe *sqe; struct io_kiocb *req; @@ -2437,9 +2438,12 @@ int io_submit_sqes(struct io_ring_ctx *ctx, unsigned int nr) err = io_setup_link(link, &req); if (unlikely(err)) goto error; - - if (likely(req)) - io_submit_sqe(req); + else if (unlikely(!req)) + continue; + else if (unlikely(req->flags & (REQ_F_FORCE_ASYNC | REQ_F_FAIL))) + io_queue_sqe_fallback(req); + else + wq_list_add_tail(&req->comp_list, &req_list); continue; error: /* @@ -2453,6 +2457,12 @@ int io_submit_sqes(struct io_ring_ctx *ctx, unsigned int nr) } } while (--left); + wq_list_for_each_safe(pos, next, &req_list) { + req = container_of(pos, struct io_kiocb, comp_list); + req->comp_list.next = NULL; + io_submit_sqe(req); + } + if (unlikely(left)) { ret -= left; /* try again if it submitted nothing and can't allocate a req */ diff --git a/io_uring/slist.h b/io_uring/slist.h index 0eb194817242e..93fbb715111ca 100644 --- a/io_uring/slist.h +++ b/io_uring/slist.h @@ -12,6 +12,10 @@ #define wq_list_for_each_resume(pos, prv) \ for (; pos; prv = pos, pos = (pos)->next) +#define wq_list_for_each_safe(pos, n, head) \ + for (pos = (head)->first, n = pos ? pos->next : NULL; \ + pos; pos = n, n = pos ? pos->next : NULL) + #define wq_list_empty(list) (READ_ONCE((list)->first) == NULL) #define INIT_WQ_LIST(list) do { \ From patchwork Fri Jul 28 20:14:49 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Keith Busch X-Patchwork-Id: 13332640 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 05559C41513 for ; Fri, 28 Jul 2023 20:20:16 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231880AbjG1UUO (ORCPT ); Fri, 28 Jul 2023 16:20:14 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37930 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229818AbjG1UUM (ORCPT ); Fri, 28 Jul 2023 16:20:12 -0400 Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com [67.231.145.42]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 075CE46A0 for ; Fri, 28 Jul 2023 13:19:46 -0700 (PDT) Received: from pps.filterd (m0044012.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 36SK148e005583 for ; Fri, 28 Jul 2023 13:19:15 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=meta.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=s2048-2021-q4; bh=IvrblMJCemFaRJq0nFN/OA8NPGjjsNq02SzUiRrfWdo=; b=aI5s+PKMuW5zPnyumeVh62RVi5/qlCVUr011SVek8NB7AMAh5Z5VHwZLLDAUCAFpO3CC 02m/BRnV8LEDJXWibJin8vkhiVwf866F6dotB3xmHir9nmxlZHZKDDsre3BFo0yR2WhT rze6h4biwDm4epWzlI2qTPv2V3IULTLMrp85l0G2dQV/8DBXWL/AeiiSL1mKEqUa/c7X lEqkHma4p8+bYQ1xkrx2NO45hsK6tGpv+1yHTmoa/Rg3I4QiG97U80j6he2HLnK4RWTv HxDQgTJTDSRJUOeYkBUVj1JkhfFbCoSQh6tt43WMORm5ZKsSOBVoY86W1hY3F9kAMJT2 rQ== Received: from maileast.thefacebook.com ([163.114.130.16]) by mx0a-00082601.pphosted.com (PPS) with ESMTPS id 3s3v48n11j-4 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Fri, 28 Jul 2023 13:19:14 -0700 Received: from twshared19625.39.frc1.facebook.com (2620:10d:c0a8:1c::1b) by mail.thefacebook.com (2620:10d:c0a8:82::d) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.27; Fri, 28 Jul 2023 13:19:11 -0700 Received: by devbig007.nao1.facebook.com (Postfix, from userid 544533) id 584B31C47B5E8; Fri, 28 Jul 2023 13:16:20 -0700 (PDT) From: Keith Busch To: , , , CC: Keith Busch Subject: [PATCH 3/3] io_uring: set plug tags for same file Date: Fri, 28 Jul 2023 13:14:49 -0700 Message-ID: <20230728201449.3350962-3-kbusch@meta.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230728201449.3350962-1-kbusch@meta.com> References: <20230728201449.3350962-1-kbusch@meta.com> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-GUID: 6KnQd3D0gScbynAOVBX2CgLgeQ55FLWX X-Proofpoint-ORIG-GUID: 6KnQd3D0gScbynAOVBX2CgLgeQ55FLWX X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.254,Aquarius:18.0.957,Hydra:6.0.591,FMLib:17.11.176.26 definitions=2023-07-27_10,2023-07-26_01,2023-05-22_02 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org From: Keith Busch io_uring tries to optimize allocating tags by hinting to the plug how many it expects to need for a batch instead of allocating each tag individually. But io_uring submission queueus may have a mix of many devices for io, so the number of io's counted may be overestimated. This can lead to allocating too many tags, which adds overhead to finding that many contiguous tags, freeing up the ones we didn't use, and may starve out other users that can actually use them. When starting a new batch of uring commands, count only commands that match the file descriptor of the first seen for this optimization. This avoids have to call the unlikely "blk_mq_free_plug_rqs()" at the end of a submission when multiple devices are used in a batch. Signed-off-by: Keith Busch --- block/blk-core.c | 49 +++++++++++++++------------------- block/blk-mq.c | 6 +++-- include/linux/io_uring_types.h | 1 + io_uring/io_uring.c | 19 ++++++++----- 4 files changed, 40 insertions(+), 35 deletions(-) diff --git a/block/blk-core.c b/block/blk-core.c index 90de50082146a..72523a983c419 100644 --- a/block/blk-core.c +++ b/block/blk-core.c @@ -1043,32 +1043,6 @@ int kblockd_mod_delayed_work_on(int cpu, struct delayed_work *dwork, } EXPORT_SYMBOL(kblockd_mod_delayed_work_on); -void blk_start_plug_nr_ios(struct blk_plug *plug, unsigned short nr_ios) -{ - struct task_struct *tsk = current; - - /* - * If this is a nested plug, don't actually assign it. - */ - if (tsk->plug) - return; - - plug->mq_list = NULL; - plug->cached_rq = NULL; - plug->nr_ios = min_t(unsigned short, nr_ios, BLK_MAX_REQUEST_COUNT); - plug->rq_count = 0; - plug->multiple_queues = false; - plug->has_elevator = false; - plug->nowait = false; - INIT_LIST_HEAD(&plug->cb_list); - - /* - * Store ordering should not be needed here, since a potential - * preempt will imply a full memory barrier - */ - tsk->plug = plug; -} - /** * blk_start_plug - initialize blk_plug and track it inside the task_struct * @plug: The &struct blk_plug that needs to be initialized @@ -1094,7 +1068,28 @@ void blk_start_plug_nr_ios(struct blk_plug *plug, unsigned short nr_ios) */ void blk_start_plug(struct blk_plug *plug) { - blk_start_plug_nr_ios(plug, 1); + struct task_struct *tsk = current; + + /* + * If this is a nested plug, don't actually assign it. + */ + if (tsk->plug) + return; + + plug->mq_list = NULL; + plug->cached_rq = NULL; + plug->nr_ios = 1; + plug->rq_count = 0; + plug->multiple_queues = false; + plug->has_elevator = false; + plug->nowait = false; + INIT_LIST_HEAD(&plug->cb_list); + + /* + * Store ordering should not be needed here, since a potential + * preempt will imply a full memory barrier + */ + tsk->plug = plug; } EXPORT_SYMBOL(blk_start_plug); diff --git a/block/blk-mq.c b/block/blk-mq.c index f14b8669ac69f..1e18ccd7d1376 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -524,7 +524,8 @@ static struct request *blk_mq_rq_cache_fill(struct request_queue *q, .q = q, .flags = flags, .cmd_flags = opf, - .nr_tags = plug->nr_ios, + .nr_tags = min_t(unsigned int, plug->nr_ios, + BLK_MAX_REQUEST_COUNT), .cached_rq = &plug->cached_rq, }; struct request *rq; @@ -2867,7 +2868,8 @@ static struct request *blk_mq_get_new_requests(struct request_queue *q, rq_qos_throttle(q, bio); if (plug) { - data.nr_tags = plug->nr_ios; + data.nr_tags = min_t(unsigned int, plug->nr_ios, + BLK_MAX_REQUEST_COUNT); plug->nr_ios = 1; data.cached_rq = &plug->cached_rq; } diff --git a/include/linux/io_uring_types.h b/include/linux/io_uring_types.h index 598553877fc25..6d922e7749989 100644 --- a/include/linux/io_uring_types.h +++ b/include/linux/io_uring_types.h @@ -175,6 +175,7 @@ struct io_submit_state { bool need_plug; unsigned short submit_nr; unsigned int cqes_count; + int fd; struct blk_plug plug; struct io_uring_cqe cqes[16]; }; diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 5434aef0a8ef7..379e41b53efde 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -2209,18 +2209,25 @@ static int io_init_req(struct io_ring_ctx *ctx, struct io_kiocb *req, return -EINVAL; if (def->needs_file) { - struct io_submit_state *state = &ctx->submit_state; - req->cqe.fd = READ_ONCE(sqe->fd); /* * Plug now if we have more than 2 IO left after this, and the * target is potentially a read/write to block based storage. */ - if (state->need_plug && def->plug) { - state->plug_started = true; - state->need_plug = false; - blk_start_plug_nr_ios(&state->plug, state->submit_nr); + if (def->plug) { + struct io_submit_state *state = &ctx->submit_state; + + if (state->need_plug) { + state->plug_started = true; + state->need_plug = false; + state->fd = req->cqe.fd; + blk_start_plug(&state->plug); + } else if (state->plug_started && + state->fd == req->cqe.fd && + !state->link.head) { + state->plug.nr_ios++; + } } }