From patchwork Mon Jun 5 21:20:04 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Roesch X-Patchwork-Id: 13268057 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1283EC7EE2F for ; Mon, 5 Jun 2023 21:20:40 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231407AbjFEVUi (ORCPT ); Mon, 5 Jun 2023 17:20:38 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40624 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229698AbjFEVUh (ORCPT ); Mon, 5 Jun 2023 17:20:37 -0400 Received: from 66-220-144-179.mail-mxout.facebook.com (66-220-144-179.mail-mxout.facebook.com [66.220.144.179]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 733D7FD for ; Mon, 5 Jun 2023 14:20:35 -0700 (PDT) Received: by devbig1114.prn1.facebook.com (Postfix, from userid 425415) id 3295769921A9; Mon, 5 Jun 2023 14:20:22 -0700 (PDT) From: Stefan Roesch To: io-uring@vger.kernel.org, kernel-team@fb.com Cc: shr@devkernel.io, axboe@kernel.dk, ammarfaizi2@gnuweeb.org, netdev@vger.kernel.org, kuba@kernel.org, olivier@trillion01.com Subject: [PATCH v14 3/8] net: split off _napi_busy_loop() Date: Mon, 5 Jun 2023 14:20:04 -0700 Message-Id: <20230605212009.1992313-4-shr@devkernel.io> X-Mailer: git-send-email 2.39.1 In-Reply-To: <20230605212009.1992313-1-shr@devkernel.io> References: <20230605212009.1992313-1-shr@devkernel.io> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org This splits off the function _napi_busy_loop from the commonality of napi_busy_loop() and napi_busy_loop_rcu(). Signed-off-by: Stefan Roesch --- net/core/dev.c | 66 ++++++++++++++++++++++++-------------------------- 1 file changed, 31 insertions(+), 35 deletions(-) diff --git a/net/core/dev.c b/net/core/dev.c index fcd4a6a70646..2a2d9bae1eb4 100644 --- a/net/core/dev.c +++ b/net/core/dev.c @@ -6214,39 +6214,10 @@ static inline void __napi_busy_poll(struct napi_busy_poll_ctx *ctx, local_bh_enable(); } -/* - * Warning: can exit without calling need_resched(). - */ -void napi_busy_loop_rcu(unsigned int napi_id, - bool (*loop_end)(void *, unsigned long), - void *loop_end_arg, bool prefer_busy_poll, u16 budget) -{ - unsigned long start_time = loop_end ? busy_loop_current_time() : 0; - struct napi_busy_poll_ctx ctx = {}; - - ctx.napi = napi_by_id(napi_id); - if (!ctx.napi) - return; - - preempt_disable(); - for (;;) { - __napi_busy_poll(&ctx, prefer_busy_poll, budget); - - if (!loop_end || loop_end(loop_end_arg, start_time)) - break; - if (unlikely(need_resched())) - break; - - cpu_relax(); - } - if (ctx.napi_poll) - busy_poll_stop(ctx.napi, ctx.have_poll_lock, prefer_busy_poll, budget); - preempt_enable(); -} - -void napi_busy_loop(unsigned int napi_id, +static void _napi_busy_loop(unsigned int napi_id, bool (*loop_end)(void *, unsigned long), - void *loop_end_arg, bool prefer_busy_poll, u16 budget) + void *loop_end_arg, bool prefer_busy_poll, u16 budget, + bool rcu) { unsigned long start_time = loop_end ? busy_loop_current_time() : 0; struct napi_busy_poll_ctx ctx = {}; @@ -6254,7 +6225,8 @@ void napi_busy_loop(unsigned int napi_id, restart: ctx.napi_poll = NULL; - rcu_read_lock(); + if (rcu) + rcu_read_lock(); ctx.napi = napi_by_id(napi_id); if (!ctx.napi) @@ -6268,8 +6240,12 @@ void napi_busy_loop(unsigned int napi_id, break; if (unlikely(need_resched())) { + if (rcu) + break; + if (ctx.napi_poll) - busy_poll_stop(ctx.napi, ctx.have_poll_lock, prefer_busy_poll, budget); + busy_poll_stop(ctx.napi, ctx.have_poll_lock, + prefer_busy_poll, budget); preempt_enable(); rcu_read_unlock(); cond_resched(); @@ -6283,7 +6259,27 @@ void napi_busy_loop(unsigned int napi_id, busy_poll_stop(ctx.napi, ctx.have_poll_lock, prefer_busy_poll, budget); preempt_enable(); out: - rcu_read_unlock(); + if (rcu) + rcu_read_unlock(); +} + +/* + * Warning: can exit without calling need_resched(). + */ +void napi_busy_loop_rcu(unsigned int napi_id, + bool (*loop_end)(void *, unsigned long), + void *loop_end_arg, bool prefer_busy_poll, u16 budget) +{ + _napi_busy_loop(napi_id, loop_end, loop_end_arg, prefer_busy_poll, + budget, true); +} + +void napi_busy_loop(unsigned int napi_id, + bool (*loop_end)(void *, unsigned long), + void *loop_end_arg, bool prefer_busy_poll, u16 budget) +{ + _napi_busy_loop(napi_id, loop_end, loop_end_arg, prefer_busy_poll, + budget, false); } EXPORT_SYMBOL(napi_busy_loop);