From patchwork Mon May 16 04:24:53 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Dumazet X-Patchwork-Id: 12850257 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id EE4E3C433EF for ; Mon, 16 May 2022 04:25:15 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234988AbiEPEZM (ORCPT ); Mon, 16 May 2022 00:25:12 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38786 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233621AbiEPEZD (ORCPT ); Mon, 16 May 2022 00:25:03 -0400 Received: from mail-pf1-x433.google.com (mail-pf1-x433.google.com [IPv6:2607:f8b0:4864:20::433]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A1AD018374 for ; Sun, 15 May 2022 21:25:02 -0700 (PDT) Received: by mail-pf1-x433.google.com with SMTP id j6so12943059pfe.13 for ; Sun, 15 May 2022 21:25:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=nhTvmdEK7HHrLLgEc3IpFQmJ1GuDt+FHbBOgrYC2DyA=; b=PQ8All6mHHDAkmrB3gxaMW7sILpS35wMqRRv4JA5Q2ynn2ixEevUUcrj3L9VUpaAy2 SqlVZYkK9ZO2kTJCX6pAbPx8glNaRiMQQRRoQ+UMUka7VdRrEKZgFt3bmR3lsN947ZNE 84jxHdNbwXAzEMMCZec23sPJK5FIQ/yqUwEVYQfKlgxaL6qhmgbTgmbmK7MHn/H3wQyv c/3zL4yTiAJWd5PNVlBnRUBAumAaJ3UG8p6BroPytBR9rpGHiRS9YvpktggHJGUe2ciY P36XXGYMQ2no3Dzn3Uu/RelxQiwLIpFIrtoApqZWSXWRFfoK0KYBaEEomtN89DhWz431 bybQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=nhTvmdEK7HHrLLgEc3IpFQmJ1GuDt+FHbBOgrYC2DyA=; b=IdSn/rdtDkWNRAEkfWfU/479oaTRuNFrpSwjWairX0x3xhx2/DiCdNOSJvEijbnv99 0NYYdEOvuToZ8ni0SoByKKVHo/perG/iwsmLzU7EF3jPki/KKeCCnR1qLzS9I2e9vo4k SzwpzIvAjSxmO7zsImdZmvhHn9kqVfB1eTxKZj5J15ngGDMtHT51tJTkWIAbdakAKygg oglRFLEcwzA9SOSK8/8cgl4uPH04ogpByMozjZeOEEzKRCFQzmM+k2cAt7gRPs2Ptk1V PATRfWb8G0HaB9IKdEOIlmU4dt7NUoBwodYdHvs4L1G6IjlfjYRg/+A3XRj9VTflnoKG 89mg== X-Gm-Message-State: AOAM531QxthB7DT20BkTEj45fJM4A2n6Z9GzA6g09BQZ/kJ3a4I4UoA/ oybBZOPuqvjyMlQmwe/lV6k= X-Google-Smtp-Source: ABdhPJyhAAGq0FT2g3iL8unNUN595Lin36Q4bkpXDqxneeFR+pOUgkG3zw9TK/DA6UOpn2ghm6m0Wg== X-Received: by 2002:a65:48c5:0:b0:3c5:fe30:75dd with SMTP id o5-20020a6548c5000000b003c5fe3075ddmr13517049pgs.269.1652675102198; Sun, 15 May 2022 21:25:02 -0700 (PDT) Received: from edumazet1.svl.corp.google.com ([2620:15c:2c4:201:983e:a432:c95c:71c2]) by smtp.gmail.com with ESMTPSA id w16-20020a634910000000b003f27adead72sm308403pga.90.2022.05.15.21.25.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 15 May 2022 21:25:01 -0700 (PDT) From: Eric Dumazet To: "David S . Miller" , Jakub Kicinski , Paolo Abeni Cc: netdev , Eric Dumazet , Eric Dumazet Subject: [PATCH net-next 1/4] net: fix possible race in skb_attempt_defer_free() Date: Sun, 15 May 2022 21:24:53 -0700 Message-Id: <20220516042456.3014395-2-eric.dumazet@gmail.com> X-Mailer: git-send-email 2.36.0.550.gb090851708-goog In-Reply-To: <20220516042456.3014395-1-eric.dumazet@gmail.com> References: <20220516042456.3014395-1-eric.dumazet@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: kuba@kernel.org From: Eric Dumazet A cpu can observe sd->defer_count reaching 128, and call smp_call_function_single_async() Problem is that the remote CPU can clear sd->defer_count before the IPI is run/acknowledged. Other cpus can queue more packets and also decide to call smp_call_function_single_async() while the pending IPI was not yet delivered. This is a common issue with smp_call_function_single_async(). Callers must ensure correct synchronization and serialization. I triggered this issue while experimenting smaller threshold. Performing the call to smp_call_function_single_async() under sd->defer_lock protection did not solve the problem. Commit 5a18ceca6350 ("smp: Allow smp_call_function_single_async() to insert locked csd") replaced an informative WARN_ON_ONCE() with a return of -EBUSY, which is often ignored. Test of CSD_FLAG_LOCK presence is racy anyway. Fixes: 68822bdf76f1 ("net: generalize skb freeing deferral to per-cpu lists") Signed-off-by: Eric Dumazet --- include/linux/netdevice.h | 1 + net/core/dev.c | 7 +++++-- net/core/skbuff.c | 5 ++--- 3 files changed, 8 insertions(+), 5 deletions(-) diff --git a/include/linux/netdevice.h b/include/linux/netdevice.h index 536321691c725ebd311088f4654dd04b9abbaaef..89699a299ba1b0b544b7abb782708d827abe55ac 100644 --- a/include/linux/netdevice.h +++ b/include/linux/netdevice.h @@ -3127,6 +3127,7 @@ struct softnet_data { /* Another possibly contended cache line */ spinlock_t defer_lock ____cacheline_aligned_in_smp; int defer_count; + int defer_ipi_scheduled; struct sk_buff *defer_list; call_single_data_t defer_csd; }; diff --git a/net/core/dev.c b/net/core/dev.c index a601da3b4a7c800801f763f097f00f3a3b591107..d708f95356e0b03a61e8211adcf6d272dfa322b5 100644 --- a/net/core/dev.c +++ b/net/core/dev.c @@ -4581,9 +4581,12 @@ static void rps_trigger_softirq(void *data) #endif /* CONFIG_RPS */ /* Called from hardirq (IPI) context */ -static void trigger_rx_softirq(void *data __always_unused) +static void trigger_rx_softirq(void *data) { + struct softnet_data *sd = data; + __raise_softirq_irqoff(NET_RX_SOFTIRQ); + smp_store_release(&sd->defer_ipi_scheduled, 0); } /* @@ -11381,7 +11384,7 @@ static int __init net_dev_init(void) INIT_CSD(&sd->csd, rps_trigger_softirq, sd); sd->cpu = i; #endif - INIT_CSD(&sd->defer_csd, trigger_rx_softirq, NULL); + INIT_CSD(&sd->defer_csd, trigger_rx_softirq, sd); spin_lock_init(&sd->defer_lock); init_gro_hash(&sd->backlog); diff --git a/net/core/skbuff.c b/net/core/skbuff.c index bd16e158b3668f496a9d2c8e8b6f3433a326314c..1e2180682f2e94c45e3f26059af6d18be2d9f9d3 100644 --- a/net/core/skbuff.c +++ b/net/core/skbuff.c @@ -6514,8 +6514,7 @@ void skb_attempt_defer_free(struct sk_buff *skb) sd->defer_count++; /* kick every time queue length reaches 128. - * This should avoid blocking in smp_call_function_single_async(). - * This condition should hardly be bit under normal conditions, + * This condition should hardly be hit under normal conditions, * unless cpu suddenly stopped to receive NIC interrupts. */ kick = sd->defer_count == 128; @@ -6525,6 +6524,6 @@ void skb_attempt_defer_free(struct sk_buff *skb) /* Make sure to trigger NET_RX_SOFTIRQ on the remote CPU * if we are unlucky enough (this seems very unlikely). */ - if (unlikely(kick)) + if (unlikely(kick) && !cmpxchg(&sd->defer_ipi_scheduled, 0, 1)) smp_call_function_single_async(cpu, &sd->defer_csd); } From patchwork Mon May 16 04:24:54 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Dumazet X-Patchwork-Id: 12850258 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B814EC433F5 for ; Mon, 16 May 2022 04:25:17 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235027AbiEPEZP (ORCPT ); Mon, 16 May 2022 00:25:15 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38834 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231135AbiEPEZF (ORCPT ); Mon, 16 May 2022 00:25:05 -0400 Received: from mail-pj1-x102f.google.com (mail-pj1-x102f.google.com [IPv6:2607:f8b0:4864:20::102f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2CB2C1839D for ; Sun, 15 May 2022 21:25:04 -0700 (PDT) Received: by mail-pj1-x102f.google.com with SMTP id n10so13316361pjh.5 for ; Sun, 15 May 2022 21:25:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=j1na0dbzzw4voQbMWC5mAjFj7mR/JqvFVI07U9sF25w=; b=ERuzkeSF/O3LNVloKe6bT9MNG9O69IcAOejN0UbCFG2f8z4umfb/z4Mzcqyy72niXA vF3m64x1SEEkHixuwBf5/0tXl8sy2BH0id5XJ8pQfFXcCZujeim0Lx5sT4taxCORsyf2 ojdmxrZAF7kWVCcJAMxtflpvIpUEAiVXpIMO6xfx62ZfE0FL/dU7nD9DcFYktOmHK2XM GbST5Cr16jNo7CfWK9nq3y7ZUt21yxZVjZEXesLCIBQFodwbpSksJw1KysDB+LhznQYU aqZ1U8E3AfgdXCKWf+YjSViO4wz34Ho0i8pipw/clTngpB5TXsoEu5DbWIjx6DqAE+Gz bQpg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=j1na0dbzzw4voQbMWC5mAjFj7mR/JqvFVI07U9sF25w=; b=nGo8DBMSEI0yCODoit/LpOemcTHY/Lf41pzZXSXROHjf7B52hpF8o8VJ7TKS65TKPk AVSiWaUjMBCmgbKe9WI22UP9DFJ2rqIZPD7i82kIrNUINlDool9H+vZrV6jpvUNIjXyL 9xk0YksiJb4I/FOnD5+ZV8GnfQDByuwApIxPYtiFlDThWjHLhuu2wLwf5Fuhc02yVHUa OpXaQiZzBNaZzJGw/TWc4EEPE3ff4tnFBJY5DzYcFIOFQB7IuLlmcExooTDw+DDqoY/e HRUPauyXhFuR9zrU3B57/6AzzzlRlo/y9RsGmYgsA506NhcI84UauRpdi+al14Wa4L0h iflQ== X-Gm-Message-State: AOAM532XZvOVteW5JtjjM0FCKNufNEqOw+HMkPIPydkfCR+OlB3CCj6Z PGuxmQ1M0CFiwgWSTGOoNXmYNp1+Dhc= X-Google-Smtp-Source: ABdhPJzqaGU6dqzJOsTyKlRMuQHnFyveU19iZ6NbPrE5byQQBIEMbiUlFkeF8EC/OgpkQZjX2dsKTw== X-Received: by 2002:a17:902:f789:b0:14e:ebbc:264b with SMTP id q9-20020a170902f78900b0014eebbc264bmr15417031pln.169.1652675103625; Sun, 15 May 2022 21:25:03 -0700 (PDT) Received: from edumazet1.svl.corp.google.com ([2620:15c:2c4:201:983e:a432:c95c:71c2]) by smtp.gmail.com with ESMTPSA id w16-20020a634910000000b003f27adead72sm308403pga.90.2022.05.15.21.25.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 15 May 2022 21:25:03 -0700 (PDT) From: Eric Dumazet To: "David S . Miller" , Jakub Kicinski , Paolo Abeni Cc: netdev , Eric Dumazet , Eric Dumazet Subject: [PATCH net-next 2/4] net: use napi_consume_skb() in skb_defer_free_flush() Date: Sun, 15 May 2022 21:24:54 -0700 Message-Id: <20220516042456.3014395-3-eric.dumazet@gmail.com> X-Mailer: git-send-email 2.36.0.550.gb090851708-goog In-Reply-To: <20220516042456.3014395-1-eric.dumazet@gmail.com> References: <20220516042456.3014395-1-eric.dumazet@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: kuba@kernel.org From: Eric Dumazet skb_defer_free_flush() runs from softirq context, we have the opportunity to refill the napi_alloc_cache, and/or use kmem_cache_free_bulk() when this cache is full. Signed-off-by: Eric Dumazet --- net/core/dev.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/net/core/dev.c b/net/core/dev.c index d708f95356e0b03a61e8211adcf6d272dfa322b5..35b6d79b0c51412534dc3b3374b8d797d212f2d8 100644 --- a/net/core/dev.c +++ b/net/core/dev.c @@ -6632,7 +6632,7 @@ static void skb_defer_free_flush(struct softnet_data *sd) while (skb != NULL) { next = skb->next; - __kfree_skb(skb); + napi_consume_skb(skb, 1); skb = next; } } From patchwork Mon May 16 04:24:55 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Dumazet X-Patchwork-Id: 12850260 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 18590C433F5 for ; Mon, 16 May 2022 04:25:22 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236227AbiEPEZU (ORCPT ); Mon, 16 May 2022 00:25:20 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38862 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233863AbiEPEZG (ORCPT ); Mon, 16 May 2022 00:25:06 -0400 Received: from mail-pj1-x1029.google.com (mail-pj1-x1029.google.com [IPv6:2607:f8b0:4864:20::1029]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7451E18379 for ; Sun, 15 May 2022 21:25:05 -0700 (PDT) Received: by mail-pj1-x1029.google.com with SMTP id pt3-20020a17090b3d0300b001df448c8d79so2091510pjb.5 for ; Sun, 15 May 2022 21:25:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=zxxIu40tB7xyc4/4mwu0tXbbd4pQ0H0wSYiSFSUKaAg=; b=lErUHXRyfN+xmKA70nfUjnw6XiaRPSwszzQz3NC2ec6TgoeBrI4RbsEsBCkUZU3JKP YVjRDJwF/OBWFU7QegkP+iQzytfN8j/22wr6H8qDYfHeW2EGo2kMqqCODiWBlYjaFvpr o7awtXvZSlV/7+xXgUkZCtXwfxXBVaW6kiL4QGGGka6qjoFFz94aDP+911VeNJjx7J+Q nJB8H70NPJttcnXBKKalSRDCkR0jZFHG3wBgBEVP6KBSMozeFKvNUybOwmc8oWs5Q7RO tRUxVkMShZs2fryRaRbJBxwnofKNYSRUT3914LnNBZwo7niOPivwaysAeRtIU4euLiEn UNnA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=zxxIu40tB7xyc4/4mwu0tXbbd4pQ0H0wSYiSFSUKaAg=; b=osmKVhyefSiSt2mOasCf9aH40Jci1LRhv0LaQHwiOKrUHnR2RZ+7XLkQo2kIdIeX95 Q4jDUNkNRvS7xA/+Hy8RsnhaAgpOpkLy8e8qeCGPYl8ZWfgvNCajPcsCxkE3r9knSxPd obel8YOpSBIwhVSIxxQ/Wxp+2G9AJ+hWi41Vc47Q+ZfSMMb444Zjru4ZzQ57QhfmxZcA IUch8RsZH4lklGV4vUj+YgfXEIF8vWAyh+dF/GxVTeHGaBoSc1sk8D1ij7ZVDDrgtDEq kU7bJThv43n27asqmD1qxCLhhTBPvVgT2/JogpYJmLcb3Kx5jZiyGpxXu0XWVVfhRKnz HnLg== X-Gm-Message-State: AOAM532t7ASMvE5e7ximwmsRpzg3my0lnyeU8oCOKM5bYaTg6NJoXHqf EE8bjNysFPKDsNqyqPgPr94= X-Google-Smtp-Source: ABdhPJyPX/5Qtp+yrJ813PTNCTQ7WUL4UcjoN36TV6T95lpvr3dsbZOKU/BqYs3cvYFmZ/RqKWUDKw== X-Received: by 2002:a17:902:dac4:b0:15f:2a36:22ad with SMTP id q4-20020a170902dac400b0015f2a3622admr16049842plx.94.1652675105019; Sun, 15 May 2022 21:25:05 -0700 (PDT) Received: from edumazet1.svl.corp.google.com ([2620:15c:2c4:201:983e:a432:c95c:71c2]) by smtp.gmail.com with ESMTPSA id w16-20020a634910000000b003f27adead72sm308403pga.90.2022.05.15.21.25.04 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 15 May 2022 21:25:04 -0700 (PDT) From: Eric Dumazet To: "David S . Miller" , Jakub Kicinski , Paolo Abeni Cc: netdev , Eric Dumazet , Eric Dumazet Subject: [PATCH net-next 3/4] net: add skb_defer_max sysctl Date: Sun, 15 May 2022 21:24:55 -0700 Message-Id: <20220516042456.3014395-4-eric.dumazet@gmail.com> X-Mailer: git-send-email 2.36.0.550.gb090851708-goog In-Reply-To: <20220516042456.3014395-1-eric.dumazet@gmail.com> References: <20220516042456.3014395-1-eric.dumazet@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: kuba@kernel.org From: Eric Dumazet commit 68822bdf76f1 ("net: generalize skb freeing deferral to per-cpu lists") added another per-cpu cache of skbs. It was expected to be small, and an IPI was forced whenever the list reached 128 skbs. We might need to be able to control more precisely queue capacity and added latency. An IPI is generated whenever queue reaches half capacity. Default value of the new limit is 64. Signed-off-by: Eric Dumazet --- Documentation/admin-guide/sysctl/net.rst | 8 ++++++++ net/core/dev.c | 1 + net/core/dev.h | 2 +- net/core/skbuff.c | 15 +++++++++------ net/core/sysctl_net_core.c | 8 ++++++++ 5 files changed, 27 insertions(+), 7 deletions(-) diff --git a/Documentation/admin-guide/sysctl/net.rst b/Documentation/admin-guide/sysctl/net.rst index f86b5e1623c6922b070fd7c62e83271ee9aee46c..fa4dcdb283cf8937df8414906b57949cc7a3c2bc 100644 --- a/Documentation/admin-guide/sysctl/net.rst +++ b/Documentation/admin-guide/sysctl/net.rst @@ -322,6 +322,14 @@ a leaked reference faster. A larger value may be useful to prevent false warnings on slow/loaded systems. Default value is 10, minimum 1, maximum 3600. +skb_defer_max +------------- + +Max size (in skbs) of the per-cpu list of skbs being freed +by the cpu which allocated them. Used by TCP stack so far. + +Default: 64 + optmem_max ---------- diff --git a/net/core/dev.c b/net/core/dev.c index 35b6d79b0c51412534dc3b3374b8d797d212f2d8..ac22fedfeaf72dc0d46f4793bbd9b2d5dd301730 100644 --- a/net/core/dev.c +++ b/net/core/dev.c @@ -4329,6 +4329,7 @@ int netdev_max_backlog __read_mostly = 1000; EXPORT_SYMBOL(netdev_max_backlog); int netdev_tstamp_prequeue __read_mostly = 1; +unsigned int sysctl_skb_defer_max __read_mostly = 64; int netdev_budget __read_mostly = 300; /* Must be at least 2 jiffes to guarantee 1 jiffy timeout */ unsigned int __read_mostly netdev_budget_usecs = 2 * USEC_PER_SEC / HZ; diff --git a/net/core/dev.h b/net/core/dev.h index 328b37af90ba9465d83c833dffd18547ddef4028..cbb8a925175a257f8ce2a27eebb02e03041eebb8 100644 --- a/net/core/dev.h +++ b/net/core/dev.h @@ -39,7 +39,7 @@ void dev_addr_check(struct net_device *dev); /* sysctls not referred to from outside net/core/ */ extern int netdev_budget; extern unsigned int netdev_budget_usecs; - +extern unsigned int sysctl_skb_defer_max; extern int netdev_tstamp_prequeue; extern int netdev_unregister_timeout_secs; extern int weight_p; diff --git a/net/core/skbuff.c b/net/core/skbuff.c index 1e2180682f2e94c45e3f26059af6d18be2d9f9d3..ecb8fe7045a2f9c080cd0299ff7c0c1ea88d996b 100644 --- a/net/core/skbuff.c +++ b/net/core/skbuff.c @@ -80,6 +80,7 @@ #include #include +#include "dev.h" #include "sock_destructor.h" struct kmem_cache *skbuff_head_cache __ro_after_init; @@ -6494,16 +6495,21 @@ void skb_attempt_defer_free(struct sk_buff *skb) int cpu = skb->alloc_cpu; struct softnet_data *sd; unsigned long flags; + unsigned int defer_max; bool kick; if (WARN_ON_ONCE(cpu >= nr_cpu_ids) || !cpu_online(cpu) || cpu == raw_smp_processor_id()) { - __kfree_skb(skb); +nodefer: __kfree_skb(skb); return; } sd = &per_cpu(softnet_data, cpu); + defer_max = READ_ONCE(sysctl_skb_defer_max); + if (READ_ONCE(sd->defer_count) >= defer_max) + goto nodefer; + /* We do not send an IPI or any signal. * Remote cpu will eventually call skb_defer_free_flush() */ @@ -6513,11 +6519,8 @@ void skb_attempt_defer_free(struct sk_buff *skb) WRITE_ONCE(sd->defer_list, skb); sd->defer_count++; - /* kick every time queue length reaches 128. - * This condition should hardly be hit under normal conditions, - * unless cpu suddenly stopped to receive NIC interrupts. - */ - kick = sd->defer_count == 128; + /* Send an IPI every time queue reaches half capacity. */ + kick = sd->defer_count == (defer_max >> 1); spin_unlock_irqrestore(&sd->defer_lock, flags); diff --git a/net/core/sysctl_net_core.c b/net/core/sysctl_net_core.c index 195ca5c2877125bf420128d3c6465ac216f459e5..ca8d38325e1e1d7775d61893fab94ff9499ef5f8 100644 --- a/net/core/sysctl_net_core.c +++ b/net/core/sysctl_net_core.c @@ -578,6 +578,14 @@ static struct ctl_table net_core_table[] = { .extra1 = SYSCTL_ONE, .extra2 = &int_3600, }, + { + .procname = "skb_defer_max", + .data = &sysctl_skb_defer_max, + .maxlen = sizeof(unsigned int), + .mode = 0644, + .proc_handler = proc_dointvec_minmax, + .extra1 = SYSCTL_ZERO, + }, { } }; From patchwork Mon May 16 04:24:56 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Dumazet X-Patchwork-Id: 12850259 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3A7D8C433EF for ; Mon, 16 May 2022 04:25:21 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235127AbiEPEZR (ORCPT ); Mon, 16 May 2022 00:25:17 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38906 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234226AbiEPEZI (ORCPT ); Mon, 16 May 2022 00:25:08 -0400 Received: from mail-pf1-x42c.google.com (mail-pf1-x42c.google.com [IPv6:2607:f8b0:4864:20::42c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E57AB183B0 for ; Sun, 15 May 2022 21:25:06 -0700 (PDT) Received: by mail-pf1-x42c.google.com with SMTP id v11so12976760pff.6 for ; Sun, 15 May 2022 21:25:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=4bDXGPqNCgyo9mP5knI0TyDmWOF8dGUajlMnVc+U4g4=; b=bouKb/gu2bbCkHZHuHopuKf9I/KU9jPt9ntnIQH5i9Xe/KtxT5rOs+29IciTwIn0Gq s6ciJ37jJETgwIjb+7PTZ7bKUIE3Pq9ruAPjP+XjDRQ9lEcH9kuN+NH9jNe4XLVGd00K 5DTCq/2O4KL0O7LYyoBd/qmdcsWaGgCdEWQDzkFqvuesfVmStinPJf3LpWgRqapB7skJ 1XCh2Famdh45TKOPi7fW2SnI2Fx1YXxH15Jum8ysAiunup0mE7Yc5J3U+CUdnLoX/uYg KnOktOv54AHPkp2IzYN9mv99kgkWleQ4ZG4FnPpUzBLlbTedFfQV8WR57Tw29b4g3y9D HKoA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=4bDXGPqNCgyo9mP5knI0TyDmWOF8dGUajlMnVc+U4g4=; b=IX1zpZl3ZQc3q/vOV5MojgdFIiMG/5zwGGjZZ5XVdThnA8ltywCfQBEmHcRIv2XIqg 7Ib3oLLY7Wxp1WejSx/7raVe2u0idyOXiV1xGXWXhyV6XHuWu4VMRgZzD0KC0KYvZUFk BGlJM2NRo3uWvDPsmjVFFa9IOwcRS0d8YJtpuxlofzF2KE+W0TQkTMMjLsuhPQzNd+y8 +vHRaJqh/HV//GWjmE9NsljeMy1Xe3T+RMpdYewBrABV9XGEaqzBcZuSyVeclWKeqUrJ Jl/G8hWqgRxP7m7zmjGZHEdBSojWokwFu0A2yCCRnWn1LaJzutoazFlHuJ8lrGESClwo /K3g== X-Gm-Message-State: AOAM530Op7/2NRyoBndP/mWy7xqhuYKYFLNrNUwRs7fd4+jwPdDdnmGw bP20xWFyjWBCg6NzXcWC4nw= X-Google-Smtp-Source: ABdhPJyD18422OxobYz3WW/pHQ1oOS8QOK8PvD3H3RiCbNNyN6gIsCPONra2gJWofUGlKumb2JPy3g== X-Received: by 2002:a05:6a00:2444:b0:4fd:db81:cbdd with SMTP id d4-20020a056a00244400b004fddb81cbddmr15956040pfj.32.1652675106426; Sun, 15 May 2022 21:25:06 -0700 (PDT) Received: from edumazet1.svl.corp.google.com ([2620:15c:2c4:201:983e:a432:c95c:71c2]) by smtp.gmail.com with ESMTPSA id w16-20020a634910000000b003f27adead72sm308403pga.90.2022.05.15.21.25.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 15 May 2022 21:25:06 -0700 (PDT) From: Eric Dumazet To: "David S . Miller" , Jakub Kicinski , Paolo Abeni Cc: netdev , Eric Dumazet , Eric Dumazet Subject: [PATCH net-next 4/4] net: call skb_defer_free_flush() before each napi_poll() Date: Sun, 15 May 2022 21:24:56 -0700 Message-Id: <20220516042456.3014395-5-eric.dumazet@gmail.com> X-Mailer: git-send-email 2.36.0.550.gb090851708-goog In-Reply-To: <20220516042456.3014395-1-eric.dumazet@gmail.com> References: <20220516042456.3014395-1-eric.dumazet@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: kuba@kernel.org From: Eric Dumazet skb_defer_free_flush() can consume cpu cycles, it seems better to call it in the inner loop: - Potentially frees page/skb that will be reallocated while hot. - Account for the cpu cycles in the @time_limit determination. - Keep softnet_data.defer_count small to reduce chances for skb_attempt_defer_free() to send an IPI. Signed-off-by: Eric Dumazet --- net/core/dev.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/net/core/dev.c b/net/core/dev.c index ac22fedfeaf72dc0d46f4793bbd9b2d5dd301730..aabb695e25f35402bbc85602e1bb9c86d2fa209e 100644 --- a/net/core/dev.c +++ b/net/core/dev.c @@ -6654,6 +6654,8 @@ static __latent_entropy void net_rx_action(struct softirq_action *h) for (;;) { struct napi_struct *n; + skb_defer_free_flush(sd); + if (list_empty(&list)) { if (!sd_has_rps_ipi_waiting(sd) && list_empty(&repoll)) goto end; @@ -6683,8 +6685,7 @@ static __latent_entropy void net_rx_action(struct softirq_action *h) __raise_softirq_irqoff(NET_RX_SOFTIRQ); net_rps_action_and_irq_enable(sd); -end: - skb_defer_free_flush(sd); +end:; } struct netdev_adjacent {