From patchwork Thu Feb 9 23:01:42 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Roesch X-Patchwork-Id: 13135134 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 936BDC6379F for ; Thu, 9 Feb 2023 23:02:28 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229964AbjBIXC1 (ORCPT ); Thu, 9 Feb 2023 18:02:27 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55722 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230385AbjBIXCY (ORCPT ); Thu, 9 Feb 2023 18:02:24 -0500 Received: from 66-220-144-178.mail-mxout.facebook.com (66-220-144-178.mail-mxout.facebook.com [66.220.144.178]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7374E5EBDD for ; Thu, 9 Feb 2023 15:02:17 -0800 (PST) Received: by dev0134.prn3.facebook.com (Postfix, from userid 425415) id BBA906AB6777; Thu, 9 Feb 2023 15:02:01 -0800 (PST) From: Stefan Roesch To: io-uring@vger.kernel.org, kernel-team@fb.com Cc: shr@devkernel.io, axboe@kernel.dk, ammarfaizi2@gnuweeb.org, Olivier Langlois Subject: [PATCH v8 5/7] io-uring: add sqpoll support for napi busy poll Date: Thu, 9 Feb 2023 15:01:42 -0800 Message-Id: <20230209230144.465620-6-shr@devkernel.io> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20230209230144.465620-1-shr@devkernel.io> References: <20230209230144.465620-1-shr@devkernel.io> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org This adds the sqpoll support to the io-uring napi. Signed-off-by: Stefan Roesch Suggested-by: Olivier Langlois --- io_uring/napi.c | 25 +++++++++++++++++++++++++ io_uring/napi.h | 2 ++ io_uring/sqpoll.c | 4 ++++ 3 files changed, 31 insertions(+) diff --git a/io_uring/napi.c b/io_uring/napi.c index c9e2afae382d..038957b46a0e 100644 --- a/io_uring/napi.c +++ b/io_uring/napi.c @@ -278,4 +278,29 @@ void io_napi_end_busy_loop(struct io_ring_ctx *ctx, struct io_wait_queue *iowq, io_napi_merge_lists(ctx, napi_list); } +/* + * io_napi_sqpoll_busy_poll() - busy poll loop for sqpoll + * @ctx: pointer to io-uring context structure + * @napi_list: pointer to head of napi list + * + * Splice of the napi list and execute the napi busy poll loop. + */ +int io_napi_sqpoll_busy_poll(struct io_ring_ctx *ctx, struct list_head *napi_list) +{ + int ret = 0; + + spin_lock(&ctx->napi_lock); + list_splice_init(&ctx->napi_list, napi_list); + spin_unlock(&ctx->napi_lock); + + if (!list_empty(napi_list) && + READ_ONCE(ctx->napi_busy_poll_to) > 0 && + io_napi_busy_loop(napi_list, ctx->napi_prefer_busy_poll)) { + io_napi_merge_lists(ctx, napi_list); + ret = 1; + } + + return ret; +} + #endif diff --git a/io_uring/napi.h b/io_uring/napi.h index 0672592cfb79..23a6df32805f 100644 --- a/io_uring/napi.h +++ b/io_uring/napi.h @@ -23,6 +23,7 @@ void io_napi_adjust_busy_loop_timeout(struct io_ring_ctx *ctx, struct timespec64 *ts); void io_napi_end_busy_loop(struct io_ring_ctx *ctx, struct io_wait_queue *iowq, struct list_head *napi_list); +int io_napi_sqpoll_busy_poll(struct io_ring_ctx *ctx, struct list_head *napi_list); #else @@ -43,6 +44,7 @@ static inline void io_napi_add(struct io_kiocb *req) #define io_napi_setup_busy_loop(ctx, iowq, napi_list) do {} while (0) #define io_napi_adjust_busy_loop_timeout(ctx, iowq, napi_list, ts) do {} while (0) #define io_napi_end_busy_loop(ctx, iowq, napi_list) do {} while (0) +#define io_napi_sqpoll_busy_poll(ctx, napi_list) (0) #endif diff --git a/io_uring/sqpoll.c b/io_uring/sqpoll.c index 0119d3f1a556..90fdbd87434a 100644 --- a/io_uring/sqpoll.c +++ b/io_uring/sqpoll.c @@ -15,6 +15,7 @@ #include #include "io_uring.h" +#include "napi.h" #include "sqpoll.h" #define IORING_SQPOLL_CAP_ENTRIES_VALUE 8 @@ -168,6 +169,7 @@ static int __io_sq_thread(struct io_ring_ctx *ctx, bool cap_entries) { unsigned int to_submit; int ret = 0; + NAPI_LIST_HEAD(local_napi_list); to_submit = io_sqring_entries(ctx); /* if we're handling multiple rings, cap submit size for fairness */ @@ -193,6 +195,8 @@ static int __io_sq_thread(struct io_ring_ctx *ctx, bool cap_entries) ret = io_submit_sqes(ctx, to_submit); mutex_unlock(&ctx->uring_lock); + ret += io_napi_sqpoll_busy_poll(ctx, &local_napi_list); + if (to_submit && wq_has_sleeper(&ctx->sqo_sq_wait)) wake_up(&ctx->sqo_sq_wait); if (creds)