From patchwork Thu Mar 28 17:03:02 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Dumazet X-Patchwork-Id: 13609220 X-Patchwork-Delegate: kuba@kernel.org Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8FC8AED8 for ; Thu, 28 Mar 2024 17:03:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711645395; cv=none; b=lnAkaWFvkDmQ6Co4VL6PI0kFOBDu/JC5wOlewj3WDKWsfDrCtrMlzqtQ19rGGNvOQ/Ez2vYov/v9G2DvEPqOP8/q6i2HOQGTi6crQ8yR5N5WDVJwh+KmXhQnXFtKfBfqqVg+ek+EL68bHTt05XKuVuHn1qON0hEMZrzGbosyvVk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711645395; c=relaxed/simple; bh=lFeGl7dZQs/ZceANoJRiP1nMhVUnouVH1jtKb+pAUk8=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=P2RgCzoTrJddRlvCbsJk2+MSf1FUgo/JNNLZgl1pmrlaM96HqLEcxpFOMRRzqisx9pQs5TnmT8JlAzSIIx60Rfyg/u2QzQ1XAmYgj72x3mGjx2sFq+B9gd+GJW9mDlcYv9GfV0hDXBPIcQUmFGmvAwvsLStCKwJ8WYejQOx76+I= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--edumazet.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=RxglwRYq; arc=none smtp.client-ip=209.85.219.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--edumazet.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="RxglwRYq" Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-dc743cc50a6so1421378276.2 for ; Thu, 28 Mar 2024 10:03:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1711645392; x=1712250192; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=Jply39cwIs/FhQ97Ad44YD+TULK2Sf78Gyu9nOqDg5M=; b=RxglwRYqMRSA2np36Q3xwH1x4/f7c4W1e2KudJVrpAO0XtG3aH4wiJipnMXHr+HPYx PwL9HP8vHooJ+HDR+hbpAfpH3nh57QgEZUNHu5tQLhzLK63Jcx695ba73R6/LFd01kCZ CLn1jRvDBTemcHATa20geMUU6kec2zoszYd43+1gFDfgL1j+QqWal9u0Pd+AKUzYIBFJ iBBNEB/k3oz25VKELfszgbu8advQVzSlFnFSOUud0xmT7IX3B9PzHToCLgQCow2TgOq5 cOHTm8vw+52ur6ONSEPLepP/2xgk/6ul/iQCVXrsCnnUpT8tIQBL2iVqrrjnV3LOKDt2 isYw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711645392; x=1712250192; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Jply39cwIs/FhQ97Ad44YD+TULK2Sf78Gyu9nOqDg5M=; b=QZ7RcV2RfQ8G6mHL4UpogF5CqRImmXtZXDxyf7Vb/2fFW7Dxq5xDcf+k4NrdpT++xv u69xI9K2pTltkduP+2Jro9H1HXyJXIstOn5fK0u04T08EVMNySUQutGHxz+tzBaWr/pE jTfONqi8PGvUt+GiQyn8sJGjS6XrY7touSChTqDpUu1GPrCRJL70v9vw3QwYgTyKEDvE 0LoLMU6PIRwaNcR0/SSoy+9COgbXKJiI+dpe7DAYNdUssmG35Od1pAaHt5XvPXJsfLh+ BBUoLiGXy+Y0spyLoggPLmkbXfUQ5ybQ3LDJb5PJ0wLU8+okY5W40axGT7nR+Zm22/2B atQQ== X-Gm-Message-State: AOJu0YxKkXbT7NZp3CHdRA/ZZmADakVfSpzXMckJ6NA9ZhzMgbLuubap +95WJ0OCapBJ5oHHQAo/KmyM5KDjKdx7Wr2tvx93k2oHzTb3FEMh7zZXoJir1bLB4G/G/hCUj1u EwYAyWzGKaA== X-Google-Smtp-Source: AGHT+IG/P0JQvxEUyyUSbD8dr02ctAEomlSHiHMsKdqbDOB308hqdlTiFM7oSI1/wIeYAcxV4WtQY5hXAFIjJw== X-Received: from edumazet1.c.googlers.com ([fda3:e722:ac3:cc00:2b:7d90:c0a8:395a]) (user=edumazet job=sendgmr) by 2002:a05:6902:2413:b0:dc2:5273:53f9 with SMTP id dr19-20020a056902241300b00dc2527353f9mr249276ybb.1.1711645392697; Thu, 28 Mar 2024 10:03:12 -0700 (PDT) Date: Thu, 28 Mar 2024 17:03:02 +0000 In-Reply-To: <20240328170309.2172584-1-edumazet@google.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240328170309.2172584-1-edumazet@google.com> X-Mailer: git-send-email 2.44.0.478.gd926399ef9-goog Message-ID: <20240328170309.2172584-2-edumazet@google.com> Subject: [PATCH net-next 1/8] net: move kick_defer_list_purge() to net/core/dev.h From: Eric Dumazet To: "David S . Miller" , Jakub Kicinski , Paolo Abeni Cc: netdev@vger.kernel.org, eric.dumazet@gmail.com, Eric Dumazet X-Patchwork-Delegate: kuba@kernel.org kick_defer_list_purge() is defined in net/core/dev.c and used from net/core/skubff.c Because we need softnet_data, include from net/core/dev.h Signed-off-by: Eric Dumazet --- include/linux/netdevice.h | 1 - net/core/dev.h | 6 +++--- 2 files changed, 3 insertions(+), 4 deletions(-) diff --git a/include/linux/netdevice.h b/include/linux/netdevice.h index e41d30ebaca61e48a2ceb43edf777fa8b9859ef2..cb37817d6382c29117afd8ce54db6dba94f8c930 100644 --- a/include/linux/netdevice.h +++ b/include/linux/netdevice.h @@ -3287,7 +3287,6 @@ static inline void dev_xmit_recursion_dec(void) __this_cpu_dec(softnet_data.xmit.recursion); } -void kick_defer_list_purge(struct softnet_data *sd, unsigned int cpu); void __netif_schedule(struct Qdisc *q); void netif_schedule_queue(struct netdev_queue *txq); diff --git a/net/core/dev.h b/net/core/dev.h index 2bcaf8eee50c179db2ca59880521b9be6ecd45c8..9d0f8b4587f81f4c12487f1783d8ba5cc49fc1d6 100644 --- a/net/core/dev.h +++ b/net/core/dev.h @@ -4,11 +4,9 @@ #include #include +#include struct net; -struct net_device; -struct netdev_bpf; -struct netdev_phys_item_id; struct netlink_ext_ack; struct cpumask; @@ -150,4 +148,6 @@ static inline void xdp_do_check_flushed(struct napi_struct *napi) { } #endif struct napi_struct *napi_by_id(unsigned int napi_id); +void kick_defer_list_purge(struct softnet_data *sd, unsigned int cpu); + #endif From patchwork Thu Mar 28 17:03:03 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Dumazet X-Patchwork-Id: 13609221 X-Patchwork-Delegate: kuba@kernel.org Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 064004436D for ; Thu, 28 Mar 2024 17:03:14 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711645396; cv=none; b=lZixxEl8daCH135BfWKJclcO8ccdNYK086J+YbBTGNiVNGPb20XKd78eCXzkoTIgsp7mURTlVVSokEszX62EnPM6/3jQWYJscTyrGIHV0QmLPAA4OoSoxw1vQrusuM1SaFnzcKEiVJNno4kb/j9FIPav9NzVifn47SUtStqLikQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711645396; c=relaxed/simple; bh=u6CKHjdRZ3Q3jFE+0JJbKpQLd3hkwh+kwA+sSojO8Vs=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=USFPSvIVqZSX6PPMsFkp8vNwtPPkOuGp5/aSZriB7I3x4LaPCEdnz4qzOMrfpeo4/zLoCJijwmq4jbT5KYfcn87mkOdAKZ/2B2OySN7hf5fM1jzUNXZVICc4T8EhHK4BSjVsr3IpPfYHoaMxJfaeazKsnTBYknIl24iLA2+GPtI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--edumazet.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=NtY2nOw0; arc=none smtp.client-ip=209.85.219.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--edumazet.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="NtY2nOw0" Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-dc6dbdcfd39so1891297276.2 for ; Thu, 28 Mar 2024 10:03:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1711645394; x=1712250194; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=zkkkR4bJVGdAmtjki7FbOrCs0xkI3kc/cNsgAoiErHk=; b=NtY2nOw0nmHqIEXkS52pIfHOt8/hJX7x9xF2SY2rGqq7HgIjRlIMxvHTWhwZ8/QAog SCkQ3op82Ke3DNvwyEPxYcgK+ERNWks3yPp/9B5DGeX+2JXIcUy4O7vD7Wxi4P9+0qvt L1/hHlfjXjJE58P3ZrFrxXD8atSYHf0GF6xgopYESYtvDbY54CAczdYvB2twabm9n3yS det8caSOTXXM9b5ruwFA+erDVgC5ODcOIERLfk1GEmww5+OI6Z+4pzSXsn1Xy5YApwZF 2heFr11Ef1ljIgFXxhn/K7D39Q7LgvOpI/+yycPZDGkrAGinuMuh7kq6Y/By37AxwhY/ IaRw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711645394; x=1712250194; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=zkkkR4bJVGdAmtjki7FbOrCs0xkI3kc/cNsgAoiErHk=; b=X5D0rY8Hu/iiZZNSKPObZdHn9t+8TGPDXW3sT7xHYI/Xgh8q1T64d/pJ2ytZSm0OUG W2dp9frqwVCEzTVkcKHiMgjijnSrVmQOyoFndHoOSQOCr9oK7xQOHS2xKdyR1CZf7gA7 c6U4JAUY5I8TlrFGSTS5hsj94Qz4F1rqOtV+i/yxiDFRysQ08+CIF/NhdwFv0M8GOj9U QG1tJgoDHSWJi1869McvCxNWS7oJNNyXQsZIcCzP9Qv2edKErnee+nRm2q2OPhUwlADz 2DdafUTOt0mjuGkssgVeTkLEn1fmNbrLms8a4Wu/w+AvDT1Gs1UM06KZRvdkIaW7GVQr EyGg== X-Gm-Message-State: AOJu0YyhFvCMlLVRP/FxwBDGE/WaFug4fJ6Jg7ix0NnQXq0bshfmj3kR Nq7HnQ0OajD43W8z50ixKMSjh8kp/3iiZa91VrGwvvh+fTcJ98TUNDHpGrEsMM4E/t3ChB5jtZ5 M299DNnNUbg== X-Google-Smtp-Source: AGHT+IEkDMdQwB28sjXqbQO6DPsBKrPbAJ86d1jrLX7OZqcPrx6CURW1BBpKVVOSGf8tTKhq7AJ/4pFN5ifMvQ== X-Received: from edumazet1.c.googlers.com ([fda3:e722:ac3:cc00:2b:7d90:c0a8:395a]) (user=edumazet job=sendgmr) by 2002:a05:6902:110c:b0:dcc:f01f:65e1 with SMTP id o12-20020a056902110c00b00dccf01f65e1mr1089559ybu.8.1711645394049; Thu, 28 Mar 2024 10:03:14 -0700 (PDT) Date: Thu, 28 Mar 2024 17:03:03 +0000 In-Reply-To: <20240328170309.2172584-1-edumazet@google.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240328170309.2172584-1-edumazet@google.com> X-Mailer: git-send-email 2.44.0.478.gd926399ef9-goog Message-ID: <20240328170309.2172584-3-edumazet@google.com> Subject: [PATCH net-next 2/8] net: move dev_xmit_recursion() helpers to net/core/dev.h From: Eric Dumazet To: "David S . Miller" , Jakub Kicinski , Paolo Abeni Cc: netdev@vger.kernel.org, eric.dumazet@gmail.com, Eric Dumazet X-Patchwork-Delegate: kuba@kernel.org Move dev_xmit_recursion() and friends to net/core/dev.h They are only used from net/core/dev.c and net/core/filter.c. Signed-off-by: Eric Dumazet --- include/linux/netdevice.h | 17 ----------------- net/core/dev.h | 17 +++++++++++++++++ 2 files changed, 17 insertions(+), 17 deletions(-) diff --git a/include/linux/netdevice.h b/include/linux/netdevice.h index cb37817d6382c29117afd8ce54db6dba94f8c930..70775021cc269e0983f538619115237b0067d408 100644 --- a/include/linux/netdevice.h +++ b/include/linux/netdevice.h @@ -3270,23 +3270,6 @@ static inline int dev_recursion_level(void) return this_cpu_read(softnet_data.xmit.recursion); } -#define XMIT_RECURSION_LIMIT 8 -static inline bool dev_xmit_recursion(void) -{ - return unlikely(__this_cpu_read(softnet_data.xmit.recursion) > - XMIT_RECURSION_LIMIT); -} - -static inline void dev_xmit_recursion_inc(void) -{ - __this_cpu_inc(softnet_data.xmit.recursion); -} - -static inline void dev_xmit_recursion_dec(void) -{ - __this_cpu_dec(softnet_data.xmit.recursion); -} - void __netif_schedule(struct Qdisc *q); void netif_schedule_queue(struct netdev_queue *txq); diff --git a/net/core/dev.h b/net/core/dev.h index 9d0f8b4587f81f4c12487f1783d8ba5cc49fc1d6..8572d2c8dc4adce75c98868c888363e6a32e0f52 100644 --- a/net/core/dev.h +++ b/net/core/dev.h @@ -150,4 +150,21 @@ static inline void xdp_do_check_flushed(struct napi_struct *napi) { } struct napi_struct *napi_by_id(unsigned int napi_id); void kick_defer_list_purge(struct softnet_data *sd, unsigned int cpu); +#define XMIT_RECURSION_LIMIT 8 +static inline bool dev_xmit_recursion(void) +{ + return unlikely(__this_cpu_read(softnet_data.xmit.recursion) > + XMIT_RECURSION_LIMIT); +} + +static inline void dev_xmit_recursion_inc(void) +{ + __this_cpu_inc(softnet_data.xmit.recursion); +} + +static inline void dev_xmit_recursion_dec(void) +{ + __this_cpu_dec(softnet_data.xmit.recursion); +} + #endif From patchwork Thu Mar 28 17:03:04 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Dumazet X-Patchwork-Id: 13609222 X-Patchwork-Delegate: kuba@kernel.org Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 973035A78E for ; Thu, 28 Mar 2024 17:03:16 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711645398; cv=none; b=cCGdZsbAvajmSOL1T++vWkp30kzykgolV/uw0RKCHkbZxFzyJmmUXldn0hcr7EEUWQqJ4GcgPh2niZt74ue6X9XRO2H6RhZ//6HBgYsVffDt4eU2FxrQ+QMZNAVbSzqiM9ybm265aNwrvKrJ1sl3oSij7iPCYFq20GKDnfeOVc4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711645398; c=relaxed/simple; bh=F4gefdTj70muIJN2Zn7X1B8TXmBV5hqnEjBHGBIqff8=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=n0Oed3D1A8lc6/AN1ZEtoB6IXSGJA0URnDG7+XjFnJY9OVVwG0RhhI9jfJnaChASFD51CKlLEyVjjKevZaCps6AFAPwbpkEesYjxzFaUkWRebbVHJvZSRleditXcsAo9t6sfnRX5Unbg0GQ2X0aPQBnhblGQuSEV3JzOlfCETC8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--edumazet.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=bvbTVzn5; arc=none smtp.client-ip=209.85.219.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--edumazet.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="bvbTVzn5" Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-dd169dd4183so1379617276.3 for ; Thu, 28 Mar 2024 10:03:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1711645395; x=1712250195; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=FtfH02Ug/jt3Vtvx2bBwAZ1F+VYum2JtqqevVN5MyMc=; b=bvbTVzn5lpS1821HnOAn8qLAoFXvXlx0n0Qu8qASV7ZL1Bvw5HBjK71tioP15LJZce 0S+XjG+7xUOZgGmRxU3cKcFC46mQ3nKsX86ZfOi6j0eRk1pnZUBUFofAoizF7up8gIPT wIIjCgsVLWd0VYw89VxBZVAvYX71i1O1KDP3iM0Vdj5xiAvRXv2ObLimhNOPnXBD6pTC oFoJcSuKPVY4XUk86DN+U5URs4bw7daokt9oc6N6wUhF6R08syVP8g7h/napKFlFPcOa AUlKadmhNVdvba60tXeldB07MdI+349TJjkaxxS9pv4q6rotkhBxl/p6W/1oqf/2Dhuc EO/w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711645395; x=1712250195; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=FtfH02Ug/jt3Vtvx2bBwAZ1F+VYum2JtqqevVN5MyMc=; b=ulkyzdidUBug2wxr1AcVsmurfTc+alzYjayUWBvqw/kUf6MotOSWLHU5zClidRmkS9 C7R79oDFVkFC0iksftBwLRJ9mcDupRYs80ft5kBObFHe/h4nyajxRC3kbacttue0rDXs 83DUUHObkA23B3aUztNtZJt8QHt5LKBu9TLpnlcRyFipF2sfBnoNJ92vJ1zL5Tjy4llN Z9zftTtOpm7zw7PkvTGdIUqH5k/wYHMFzz/jlBSM+/cCWGUj6kFepdVPSKRFI8j8RAzO QWWwAeOf1XjNregjU9kZJIJoBibe7VlPtwZrANMx5THq1V0I+iEeqmMbZtGlxvDz5nHJ LCyg== X-Gm-Message-State: AOJu0Yy9qImlIvnZGuJAdeVMzk6O9h+DvJ573ZggSxKrYkqeZj6QrocX oH+Grjn4qHp60MjccEsXqG1NFNhRPMIwlHf9qFC+DohQHHCUukCX6nHbAr0SAp5344ZFb1CVYhT woE/nFCo9+Q== X-Google-Smtp-Source: AGHT+IH+8zFiZOYmBZcmX4Q9j60Xdg9XImP4FW/41zvE6T/HOTeWE3/uBt0aUrcda2YyGzUocDfaX0OEueuBlQ== X-Received: from edumazet1.c.googlers.com ([fda3:e722:ac3:cc00:2b:7d90:c0a8:395a]) (user=edumazet job=sendgmr) by 2002:a05:6902:1502:b0:dda:c565:1025 with SMTP id q2-20020a056902150200b00ddac5651025mr252047ybu.2.1711645395779; Thu, 28 Mar 2024 10:03:15 -0700 (PDT) Date: Thu, 28 Mar 2024 17:03:04 +0000 In-Reply-To: <20240328170309.2172584-1-edumazet@google.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240328170309.2172584-1-edumazet@google.com> X-Mailer: git-send-email 2.44.0.478.gd926399ef9-goog Message-ID: <20240328170309.2172584-4-edumazet@google.com> Subject: [PATCH net-next 3/8] net: enqueue_to_backlog() change vs not running device From: Eric Dumazet To: "David S . Miller" , Jakub Kicinski , Paolo Abeni Cc: netdev@vger.kernel.org, eric.dumazet@gmail.com, Eric Dumazet X-Patchwork-Delegate: kuba@kernel.org If the device attached to the packet given to enqueue_to_backlog() is not running, we drop the packet. But we accidentally increase sd->dropped, giving false signals to admins: sd->dropped should be reserved to cpu backlog pressure, not to temporary glitches at device dismantles. While we are at it, perform the netif_running() test before we get the rps lock, and use REASON_DEV_READY drop reason instead of NOT_SPECIFIED. Signed-off-by: Eric Dumazet --- net/core/dev.c | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/net/core/dev.c b/net/core/dev.c index 5d36a634f468ffdeaca598c3dd033fe06d240bd0..af7a34b0a7d6683c6ffb21dd3388ed678473d95e 100644 --- a/net/core/dev.c +++ b/net/core/dev.c @@ -4791,12 +4791,13 @@ static int enqueue_to_backlog(struct sk_buff *skb, int cpu, unsigned long flags; unsigned int qlen; - reason = SKB_DROP_REASON_NOT_SPECIFIED; + reason = SKB_DROP_REASON_DEV_READY; + if (!netif_running(skb->dev)) + goto bad_dev; + sd = &per_cpu(softnet_data, cpu); backlog_lock_irq_save(sd, &flags); - if (!netif_running(skb->dev)) - goto drop; qlen = skb_queue_len(&sd->input_pkt_queue); if (qlen <= READ_ONCE(net_hotdata.max_backlog) && !skb_flow_limit(skb, qlen)) { @@ -4817,10 +4818,10 @@ static int enqueue_to_backlog(struct sk_buff *skb, int cpu, } reason = SKB_DROP_REASON_CPU_BACKLOG; -drop: sd->dropped++; backlog_unlock_irq_restore(sd, &flags); +bad_dev: dev_core_stats_rx_dropped_inc(skb->dev); kfree_skb_reason(skb, reason); return NET_RX_DROP; From patchwork Thu Mar 28 17:03:05 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Dumazet X-Patchwork-Id: 13609223 X-Patchwork-Delegate: kuba@kernel.org Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 113F8131BA3 for ; Thu, 28 Mar 2024 17:03:17 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711645399; cv=none; b=IemnSaljglBFV0uCEJA+1gDbI7NJIy2eP5mI02i+jNkvZ+xUzhGgtYRk1RjTss9FwyixDA5Wr4Gz2+FfwFocby9jdXeTUS6YwMBtDQoxcpkrjmhh5sDibtsDSzz6C10BSaoJwfqCtcUd+8H7r4OYkVJBPxnYMukL0dxF/4Rsdkg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711645399; c=relaxed/simple; bh=dN/v+E0kL8WG56Mv6N2bzCQJg+7TLsKE8PqoMrU1wOs=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=FH6IIeja1+p3J+csPgpBrYvUXm2K3IJUEycGvWbhYMHmf2vkZYGIsF5DUU2nE7uhaedcsBvtKmegy1mACoBia/OIi7JNGQB8gsxFhQIbLnGessjglfGhFJFprZmp5K4Obj1BWFkGi/5mTKYiBqyXaNf9+fRtZDzpeFrhJEeR/VM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--edumazet.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=gOxJmiUC; arc=none smtp.client-ip=209.85.128.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--edumazet.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="gOxJmiUC" Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-60a54004e9fso21966967b3.3 for ; Thu, 28 Mar 2024 10:03:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1711645397; x=1712250197; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=rg1OdI5e7r+RvUDxZBQUJCmu93WtBEz8eFjqLMQtw/s=; b=gOxJmiUCuBO2hh0GC93Hi7fGru8z+Qm57jdyKKq9NCu3C1A4jiSQ8k6er3kx0hTqID fCA4ZpgGTL/N1jp+ixsGEas5gB8O37x75A4tb5NDAI6WdiY1vyOn1e6OHzbA8tm04To1 wloloD+uUrNqc4BCBSdrU0P5iDPFP6HNBGnWIy26jCSgblxLqQyzKfIayK1e+R2Syrpc Gwtr/QhvnvUZQfPgfsF0+bORDVMIRw+mwPld3QHf/LaNHMOHs8GclyAYtWC5ZIojilVv P5Tu61keRyGSkzw5+GjMxojLhbB2rNujFpQGop3yvQ7h7gS77PSilK5x9u3+6NbjgqcJ 6WEg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711645397; x=1712250197; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=rg1OdI5e7r+RvUDxZBQUJCmu93WtBEz8eFjqLMQtw/s=; b=HGkbhhj2ke7cGuxLBBuJ+jutL6Lavlhx9A3CSJCX0pwBC2YFm6+Bg10k0nXefu8pZi qQYzKJuZEUvZEtNA6oMgfWOL1wL8eCCXszPmUEvYn+GQCh5v0XdJEwjuqe5915LSpnFs EU433QGQ7KnL4GPNkUrGSfPGKpQ1ScukNLhtWniTlaXMwk/lUqaqBACoRWzcvtL4O3CK Fe0lCp8aG8ZYm3vmwn/GKl8Edy7NmXf1XtZUCovfoJUxBQ4PGTQI5gW93px3EtrbGG7j /OW0Kt+K3jib62W20TDy+Gv6tX+JZ528W3LgTshazKF8tChS5/gYt6FcZaU9zYWq6H8b mM2A== X-Gm-Message-State: AOJu0Yz9bwovV75iCkqeLY8xdDhPsD9vxv6dOgw3u+jAxtiP/+6XbcoA z/me2cdSjxS+BYQ5J0+Z32PDBf4VlWGcofoAlqsA5eZeQxoFNzWbkYbbqY4txu2p3kTtekjOjX8 Jh35EL6X0ag== X-Google-Smtp-Source: AGHT+IF2+6girXyoZaFPLNw3BWl6MtVk49zlLOSIu82lkK8jgxX7qHHaqDCEGz0PHoNvFGso/t8yAx3w8HMrCg== X-Received: from edumazet1.c.googlers.com ([fda3:e722:ac3:cc00:2b:7d90:c0a8:395a]) (user=edumazet job=sendgmr) by 2002:a81:6dd0:0:b0:611:34aa:ea6e with SMTP id i199-20020a816dd0000000b0061134aaea6emr4392ywc.1.1711645397135; Thu, 28 Mar 2024 10:03:17 -0700 (PDT) Date: Thu, 28 Mar 2024 17:03:05 +0000 In-Reply-To: <20240328170309.2172584-1-edumazet@google.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240328170309.2172584-1-edumazet@google.com> X-Mailer: git-send-email 2.44.0.478.gd926399ef9-goog Message-ID: <20240328170309.2172584-5-edumazet@google.com> Subject: [PATCH net-next 4/8] net: make softnet_data.dropped an atomic_t From: Eric Dumazet To: "David S . Miller" , Jakub Kicinski , Paolo Abeni Cc: netdev@vger.kernel.org, eric.dumazet@gmail.com, Eric Dumazet X-Patchwork-Delegate: kuba@kernel.org If under extreme cpu backlog pressure enqueue_to_backlog() has to drop a packet, it could do this without dirtying a cache line and potentially slowing down the target cpu. Move sd->dropped into a separate cache line, and make it atomic. In non pressure mode, this field is not touched, no need to consume valuable space in a hot cache line. Signed-off-by: Eric Dumazet --- include/linux/netdevice.h | 3 ++- net/core/dev.c | 13 +++++++++---- net/core/net-procfs.c | 3 ++- 3 files changed, 13 insertions(+), 6 deletions(-) diff --git a/include/linux/netdevice.h b/include/linux/netdevice.h index 70775021cc269e0983f538619115237b0067d408..1c31cd2691d32064613836141fbdeeebc831b21f 100644 --- a/include/linux/netdevice.h +++ b/include/linux/netdevice.h @@ -3236,10 +3236,11 @@ struct softnet_data { unsigned int input_queue_tail; #endif unsigned int received_rps; - unsigned int dropped; struct sk_buff_head input_pkt_queue; struct napi_struct backlog; + atomic_t dropped ____cacheline_aligned_in_smp; + /* Another possibly contended cache line */ spinlock_t defer_lock ____cacheline_aligned_in_smp; int defer_count; diff --git a/net/core/dev.c b/net/core/dev.c index af7a34b0a7d6683c6ffb21dd3388ed678473d95e..be2392ba56bc57bed456e2748b332d4971c83a4e 100644 --- a/net/core/dev.c +++ b/net/core/dev.c @@ -4790,17 +4790,22 @@ static int enqueue_to_backlog(struct sk_buff *skb, int cpu, struct softnet_data *sd; unsigned long flags; unsigned int qlen; + int max_backlog; reason = SKB_DROP_REASON_DEV_READY; if (!netif_running(skb->dev)) goto bad_dev; + reason = SKB_DROP_REASON_CPU_BACKLOG; sd = &per_cpu(softnet_data, cpu); + qlen = skb_queue_len_lockless(&sd->input_pkt_queue); + max_backlog = READ_ONCE(net_hotdata.max_backlog); + if (unlikely(qlen > max_backlog)) + goto cpu_backlog_drop; backlog_lock_irq_save(sd, &flags); qlen = skb_queue_len(&sd->input_pkt_queue); - if (qlen <= READ_ONCE(net_hotdata.max_backlog) && - !skb_flow_limit(skb, qlen)) { + if (qlen <= max_backlog && !skb_flow_limit(skb, qlen)) { if (qlen) { enqueue: __skb_queue_tail(&sd->input_pkt_queue, skb); @@ -4816,11 +4821,11 @@ static int enqueue_to_backlog(struct sk_buff *skb, int cpu, napi_schedule_rps(sd); goto enqueue; } - reason = SKB_DROP_REASON_CPU_BACKLOG; - sd->dropped++; backlog_unlock_irq_restore(sd, &flags); +cpu_backlog_drop: + atomic_inc(&sd->dropped); bad_dev: dev_core_stats_rx_dropped_inc(skb->dev); kfree_skb_reason(skb, reason); diff --git a/net/core/net-procfs.c b/net/core/net-procfs.c index a97eceb84e61ec347ad132ff0f22c8ce82f12e90..fa6d3969734a6ec154c3444d1b25ee93edfc5588 100644 --- a/net/core/net-procfs.c +++ b/net/core/net-procfs.c @@ -144,7 +144,8 @@ static int softnet_seq_show(struct seq_file *seq, void *v) seq_printf(seq, "%08x %08x %08x %08x %08x %08x %08x %08x %08x %08x %08x %08x %08x " "%08x %08x\n", - sd->processed, sd->dropped, sd->time_squeeze, 0, + sd->processed, atomic_read(&sd->dropped), + sd->time_squeeze, 0, 0, 0, 0, 0, /* was fastroute */ 0, /* was cpu_collision */ sd->received_rps, flow_limit_count, From patchwork Thu Mar 28 17:03:06 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Dumazet X-Patchwork-Id: 13609224 X-Patchwork-Delegate: kuba@kernel.org Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3D5AA3BBC3 for ; Thu, 28 Mar 2024 17:03:18 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711645400; cv=none; b=ujAVry1CtFhKq2W6SfStXXbfVYaJeuvur7y/xdw0QpA0GWpJUXK2+C+VH9o5U686AhsVqZDT8zvinYDmYidZYhjK/ahBK6hsxuN9lMJpUh1RYTpyVJ5iG9CHef/tbiB+0MdLUHD18kBuaVBA8A7EGhbtpl91e+rzABDh0G1Bh1A= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711645400; c=relaxed/simple; bh=a5vZkEq5DezRtLOpRv6IVjtsS4YsnwOaL3X+e14DSBo=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=UXbY5HOegI0btudfNVF2z2biSgxTyoCuqzD0htsGs3BLH5U8xZ4DAAPba4z675YDFOCPM/+XMR8O3XxB332bxWnkKrQh1Bwte6pBJ33wFoCeXQWJ84JNR33lUyFmgZZUtakLHnpqEb3CkHjE4qntBUx2ZU/QYl2gQR5Em1EX8/Y= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--edumazet.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=xfgA/28H; arc=none smtp.client-ip=209.85.219.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--edumazet.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="xfgA/28H" Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-dc743cc50a6so1421488276.2 for ; Thu, 28 Mar 2024 10:03:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1711645398; x=1712250198; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=uhxv1Ra6YkEoXOpZkudRcI7LgeFiDY3fBalWZj0ho38=; b=xfgA/28HjItDtn9QOudKxF8mfVhD4GU9FTHLjTm6tnZGZO4htphJIYvDp3ISbDZN4F jMjtPzwhzPYceGN6lEuKJlg9ZqDBiIY2xW7vVzqSofiGdYQs8Da340a7vGlY+P6G/Jit AlTAXE2JrglVgeflM8S8JoCyDf5d/p7jdwzaQY/Zm0k38YahcPfVYoqbBcewPLBwXJm5 JDSyofZhQN19VJLYEgHpPGK0t75gaBwhA/T7ffBtHZdz66LEi0F3LeVCi3K9ASi+gq7J C9NOhWuY0AvOAKkCSV37lGYIRNeeMiDwxQDmmvuAC+9MdBRM6O99tlvKE5h9LipMbpeN 64/g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711645398; x=1712250198; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=uhxv1Ra6YkEoXOpZkudRcI7LgeFiDY3fBalWZj0ho38=; b=S58xJpfDU6nVdGcdwcVoZHXfJ40ewP+qsFMz4gzaOn/8O3pq3sAFQAmWut8JBXMOxy 8tZsfqo2fWS5JedvxuUA6ctgZogVsdGiyG8x86mrOAa+ckiJI8VKWzn8Xai/WvvAtyME eraJypxfs9yHzKLkCrDnwl2uL6hMcrcISJ9G7QPMMdxm3nn09L5v6PP5DwSS/wuIgDe/ BCk2Yv927v0oLFGp79NIS8K0I2xZCLYQNJWLZKK/DE/9XwSubL7LEHgBRTiO8UwUiHJd wpSxc1tlAjuun1k5toP6k1olm1xoxcDf22aS/L2Y250HCmY2bcLwumI7i4N1ECmBIl+F VKVg== X-Gm-Message-State: AOJu0YyHOhDUPo1YTr71Khhg9KBH8uk0Aniwl2BxLHorHJR//z32Ml29 sJZI2UPCPNE+O39D3EG4tpspMK6PqrULsRo00uqS5C080vwIBqJTbxFitnovr2VuHs9sJFLjnF2 4nfQ4hydsdg== X-Google-Smtp-Source: AGHT+IGKBdCTZx/gyFlLk9bokGh571tTVNzKSnkODppmlHg4A4lprmaymYJNOpTmAFDpbcq4bcD9sgzEog4BXw== X-Received: from edumazet1.c.googlers.com ([fda3:e722:ac3:cc00:2b:7d90:c0a8:395a]) (user=edumazet job=sendgmr) by 2002:a05:6902:1b06:b0:dcc:94b7:a7a3 with SMTP id eh6-20020a0569021b0600b00dcc94b7a7a3mr250092ybb.12.1711645398283; Thu, 28 Mar 2024 10:03:18 -0700 (PDT) Date: Thu, 28 Mar 2024 17:03:06 +0000 In-Reply-To: <20240328170309.2172584-1-edumazet@google.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240328170309.2172584-1-edumazet@google.com> X-Mailer: git-send-email 2.44.0.478.gd926399ef9-goog Message-ID: <20240328170309.2172584-6-edumazet@google.com> Subject: [PATCH net-next 5/8] net: enqueue_to_backlog() cleanup From: Eric Dumazet To: "David S . Miller" , Jakub Kicinski , Paolo Abeni Cc: netdev@vger.kernel.org, eric.dumazet@gmail.com, Eric Dumazet X-Patchwork-Delegate: kuba@kernel.org We can remove a goto and a label by reversing a condition. Signed-off-by: Eric Dumazet --- net/core/dev.c | 24 +++++++++++------------- 1 file changed, 11 insertions(+), 13 deletions(-) diff --git a/net/core/dev.c b/net/core/dev.c index be2392ba56bc57bed456e2748b332d4971c83a4e..4e52745f23412bac6d3ff1b9f4d9f2ce4a2eb666 100644 --- a/net/core/dev.c +++ b/net/core/dev.c @@ -4806,20 +4806,18 @@ static int enqueue_to_backlog(struct sk_buff *skb, int cpu, backlog_lock_irq_save(sd, &flags); qlen = skb_queue_len(&sd->input_pkt_queue); if (qlen <= max_backlog && !skb_flow_limit(skb, qlen)) { - if (qlen) { -enqueue: - __skb_queue_tail(&sd->input_pkt_queue, skb); - input_queue_tail_incr_save(sd, qtail); - backlog_unlock_irq_restore(sd, &flags); - return NET_RX_SUCCESS; + if (!qlen) { + /* Schedule NAPI for backlog device. We can use + * non atomic operation as we own the queue lock. + */ + if (!__test_and_set_bit(NAPI_STATE_SCHED, + &sd->backlog.state)) + napi_schedule_rps(sd); } - - /* Schedule NAPI for backlog device - * We can use non atomic operation since we own the queue lock - */ - if (!__test_and_set_bit(NAPI_STATE_SCHED, &sd->backlog.state)) - napi_schedule_rps(sd); - goto enqueue; + __skb_queue_tail(&sd->input_pkt_queue, skb); + input_queue_tail_incr_save(sd, qtail); + backlog_unlock_irq_restore(sd, &flags); + return NET_RX_SUCCESS; } backlog_unlock_irq_restore(sd, &flags); From patchwork Thu Mar 28 17:03:07 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Dumazet X-Patchwork-Id: 13609225 X-Patchwork-Delegate: kuba@kernel.org Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 228C81DDF5 for ; Thu, 28 Mar 2024 17:03:23 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711645405; cv=none; b=iiZe0hhP5L8KVbXirRjfIQKieom+xZE27PIrxX+AgD3mVZRo2NU59JNjOebvhLhzvQZX9ujY17F55iKwXHKTpUu9Rn+Y05YxpKsFccnIgrsunAoejQjKqvI9whl0r8JlYp9AruP9ulC1TjQwtnHlO0etiwuuTtOxRMjPMFXgBMU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711645405; c=relaxed/simple; bh=q3Isl5smKGqMI33IT/alUBOXFmQZgBphrYGm25auXXk=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=QozjSNrIdQmXvEAQwQEEi6XTbmJQ5ffCPCDYjWC0SQsCRuYLUdIOinro4T/Y3gfQQXYFyymBEkko/zLrW/CqLt15GRuoNAgP9N7d9pQ/MEHA2z3brtaIj0V5RjmOEayd0nHQlCtCZuI7aFw7/TJm7zxQGbfzunpzXjNoaJUt5qc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--edumazet.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=ugSTvUtg; arc=none smtp.client-ip=209.85.219.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--edumazet.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="ugSTvUtg" Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-dc6b2682870so1770976276.0 for ; Thu, 28 Mar 2024 10:03:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1711645403; x=1712250203; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=TCHluyXzMsZacgqMrIjjaDOAIpWQewSa765EqrHSqdc=; b=ugSTvUtg1JMSmZyo6GI7Wq1ZYS6fyBfXpsQerNYunPoHfOQ12yV8uB53jghH+tv52o FutiUDjhGvh/RXQVsoGWvGdxXzwvkN/CFPCVUs6AJJQkG9DzzMYk5XMhfwRWusFsPogT 469l65Lfle1gwXRmD0rNh+ii2z4o4m3tzQtQ482nc1jLIUenwqXyaviiEEoXQjYLc7+y Hl0AvLVYq5I6+zlRAI3dlg5w8ltzMzlFDPN4K3XuO0+qS6NH2Nae15b+3gVIoW9n7qUG 9uzpn0tWaJi/A3E3/TWaEde/T4n829g/UQQCD1RoRMdsexS4I4/ZSc6oIzMpEM+n/ZOq 43vw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711645403; x=1712250203; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=TCHluyXzMsZacgqMrIjjaDOAIpWQewSa765EqrHSqdc=; b=pRsaZN8MPHHCg2E1lacaJ8RDGHUj4MGbI6Lp3EwBWh+PAZe9y5teSZOP6jy28fAd0u hal1YvCAN0xv7T3k1bfmgTCgh60eHxK50ISOAalHS+aFTDgc2H7r690ZlXse8XQhM83B DTpC6nZjeiUWP9Iljo9eICdITzBKZChytpqa+MMHmrMJ0+9PzwFlOLcEfIsAngQ6xca3 xho+fycH00Uzxudsdho4V9cTMq63I2aWAaafI9xP8IqiRnTGRGEunnOo+tDGYo6XwqLn u9QrqY+reRkJQNFSFKA/2IVyNCreIUt6N0/YFdrL7DYJuTpVLnGSvcT4QN35pQLMy3pS lDGA== X-Gm-Message-State: AOJu0Yzh8Y+FKT5WLQ4QB5qiTH7fNqc9GUGikMyGOC9EfMLVD1gcyCnx FGVNDt1Ia6N/z7kGCfHMBsQYpK/h1sMRsTPFjP/bJmMbRReDWXp3FiMwQjwclf075k4WuupHCf0 uiPFaR0YfXw== X-Google-Smtp-Source: AGHT+IHnj4V0QmcHIMpo4nSvXg1Kt87vodACERO8zjLmA8o2QtYXxQAa8fulNzRk+S7qc3BZl6gZBP50vpnCNA== X-Received: from edumazet1.c.googlers.com ([fda3:e722:ac3:cc00:2b:7d90:c0a8:395a]) (user=edumazet job=sendgmr) by 2002:a05:6902:102e:b0:dd9:4ad1:a1f7 with SMTP id x14-20020a056902102e00b00dd94ad1a1f7mr1030646ybt.9.1711645403142; Thu, 28 Mar 2024 10:03:23 -0700 (PDT) Date: Thu, 28 Mar 2024 17:03:07 +0000 In-Reply-To: <20240328170309.2172584-1-edumazet@google.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240328170309.2172584-1-edumazet@google.com> X-Mailer: git-send-email 2.44.0.478.gd926399ef9-goog Message-ID: <20240328170309.2172584-7-edumazet@google.com> Subject: [PATCH net-next 6/8] net: rps: change input_queue_tail_incr_save() From: Eric Dumazet To: "David S . Miller" , Jakub Kicinski , Paolo Abeni Cc: netdev@vger.kernel.org, eric.dumazet@gmail.com, Eric Dumazet X-Patchwork-Delegate: kuba@kernel.org input_queue_tail_incr_save() is incrementing the sd queue_tail and save it in the flow last_qtail. Two issues here : - no lock protects the write on last_qtail, we should use appropriate annotations. - We can perform this write after releasing the per-cpu backlog lock, to decrease this lock hold duration (move away the cache line miss) Also move input_queue_head_incr() and rps helpers to include/net/rps.h, while adding rps_ prefix to better reflect their role. Signed-off-by: Eric Dumazet --- include/linux/netdevice.h | 15 --------------- include/net/rps.h | 23 +++++++++++++++++++++++ net/core/dev.c | 20 ++++++++++++-------- 3 files changed, 35 insertions(+), 23 deletions(-) diff --git a/include/linux/netdevice.h b/include/linux/netdevice.h index 1c31cd2691d32064613836141fbdeeebc831b21f..14f19cc2616452d7e6afbbaa52f8ad3e61a419e9 100644 --- a/include/linux/netdevice.h +++ b/include/linux/netdevice.h @@ -3249,21 +3249,6 @@ struct softnet_data { call_single_data_t defer_csd; }; -static inline void input_queue_head_incr(struct softnet_data *sd) -{ -#ifdef CONFIG_RPS - sd->input_queue_head++; -#endif -} - -static inline void input_queue_tail_incr_save(struct softnet_data *sd, - unsigned int *qtail) -{ -#ifdef CONFIG_RPS - *qtail = ++sd->input_queue_tail; -#endif -} - DECLARE_PER_CPU_ALIGNED(struct softnet_data, softnet_data); static inline int dev_recursion_level(void) diff --git a/include/net/rps.h b/include/net/rps.h index 7660243e905b92651a41292e04caf72c5f12f26e..c13f829b8556fda63e76544c332f2c089f0d6ea4 100644 --- a/include/net/rps.h +++ b/include/net/rps.h @@ -35,6 +35,29 @@ struct rps_dev_flow { }; #define RPS_NO_FILTER 0xffff +static inline u32 rps_input_queue_tail_incr(struct softnet_data *sd) +{ +#ifdef CONFIG_RPS + return ++sd->input_queue_tail; +#else + return 0; +#endif +} + +static inline void rps_input_queue_tail_save(u32 *dest, u32 tail) +{ +#ifdef CONFIG_RPS + WRITE_ONCE(*dest, tail); +#endif +} + +static inline void rps_input_queue_head_incr(struct softnet_data *sd) +{ +#ifdef CONFIG_RPS + sd->input_queue_head++; +#endif +} + /* * The rps_dev_flow_table structure contains a table of flow mappings. */ diff --git a/net/core/dev.c b/net/core/dev.c index 4e52745f23412bac6d3ff1b9f4d9f2ce4a2eb666..1fe7c6b10793d45a03461ee581d240d2442f9e17 100644 --- a/net/core/dev.c +++ b/net/core/dev.c @@ -4601,7 +4601,7 @@ static int get_rps_cpu(struct net_device *dev, struct sk_buff *skb, if (unlikely(tcpu != next_cpu) && (tcpu >= nr_cpu_ids || !cpu_online(tcpu) || ((int)(per_cpu(softnet_data, tcpu).input_queue_head - - rflow->last_qtail)) >= 0)) { + READ_ONCE(rflow->last_qtail))) >= 0)) { tcpu = next_cpu; rflow = set_rps_cpu(dev, skb, rflow, next_cpu); } @@ -4656,7 +4656,7 @@ bool rps_may_expire_flow(struct net_device *dev, u16 rxq_index, cpu = READ_ONCE(rflow->cpu); if (rflow->filter == filter_id && cpu < nr_cpu_ids && ((int)(per_cpu(softnet_data, cpu).input_queue_head - - rflow->last_qtail) < + READ_ONCE(rflow->last_qtail)) < (int)(10 * flow_table->mask))) expire = false; } @@ -4791,6 +4791,7 @@ static int enqueue_to_backlog(struct sk_buff *skb, int cpu, unsigned long flags; unsigned int qlen; int max_backlog; + u32 tail; reason = SKB_DROP_REASON_DEV_READY; if (!netif_running(skb->dev)) @@ -4815,8 +4816,11 @@ static int enqueue_to_backlog(struct sk_buff *skb, int cpu, napi_schedule_rps(sd); } __skb_queue_tail(&sd->input_pkt_queue, skb); - input_queue_tail_incr_save(sd, qtail); + tail = rps_input_queue_tail_incr(sd); backlog_unlock_irq_restore(sd, &flags); + + /* save the tail outside of the critical section */ + rps_input_queue_tail_save(qtail, tail); return NET_RX_SUCCESS; } @@ -5894,7 +5898,7 @@ static void flush_backlog(struct work_struct *work) if (skb->dev->reg_state == NETREG_UNREGISTERING) { __skb_unlink(skb, &sd->input_pkt_queue); dev_kfree_skb_irq(skb); - input_queue_head_incr(sd); + rps_input_queue_head_incr(sd); } } backlog_unlock_irq_enable(sd); @@ -5903,7 +5907,7 @@ static void flush_backlog(struct work_struct *work) if (skb->dev->reg_state == NETREG_UNREGISTERING) { __skb_unlink(skb, &sd->process_queue); kfree_skb(skb); - input_queue_head_incr(sd); + rps_input_queue_head_incr(sd); } } local_bh_enable(); @@ -6031,7 +6035,7 @@ static int process_backlog(struct napi_struct *napi, int quota) rcu_read_lock(); __netif_receive_skb(skb); rcu_read_unlock(); - input_queue_head_incr(sd); + rps_input_queue_head_incr(sd); if (++work >= quota) return work; @@ -11445,11 +11449,11 @@ static int dev_cpu_dead(unsigned int oldcpu) /* Process offline CPU's input_pkt_queue */ while ((skb = __skb_dequeue(&oldsd->process_queue))) { netif_rx(skb); - input_queue_head_incr(oldsd); + rps_input_queue_head_incr(oldsd); } while ((skb = skb_dequeue(&oldsd->input_pkt_queue))) { netif_rx(skb); - input_queue_head_incr(oldsd); + rps_input_queue_head_incr(oldsd); } return 0; From patchwork Thu Mar 28 17:03:08 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Dumazet X-Patchwork-Id: 13609226 X-Patchwork-Delegate: kuba@kernel.org Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9EF1939FCF for ; Thu, 28 Mar 2024 17:03:25 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711645407; cv=none; b=X4UhTapvpYW30GUegs28EG5H1jnzoYahu092IxXsH7a5Z0WcXtA4Wfk2s2Kmmgw9P4KKG0bE6LYxTEttl4N5F8Ux2bRRl2kqMgYshhrqLffCRURkoIexPRkE4sInHq/APj3AOKhXMNDfOn8o7J/yr8cBH54vVCaQnTP68u267h4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711645407; c=relaxed/simple; bh=VWLua73p7ZUAjdHrayoP/lX15Em4rPKgzz1/a9x1KxQ=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=SVRyS7VzszEWRzROMIYHzpEu0v4KRdMeP5k5iMyGO5jalHdPqhh5eQTLscAfSibDGU2aC7OZsrH3I8PQAVBVejLm6s0BQvYBdwMpZykEUXm7f1dEgMJlrpqKRxQp1fBm7Pr/vmfD+FLIV2jN/pZyyaaRW877tFrDkdXqDDs5E0U= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--edumazet.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=tHmPJZUC; arc=none smtp.client-ip=209.85.219.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--edumazet.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="tHmPJZUC" Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-dcbee93a3e1so1734847276.3 for ; Thu, 28 Mar 2024 10:03:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1711645404; x=1712250204; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=DyO3NL4DZ5+zyEmGrUmGhqOJCsHXA1HlVzHyeVmkchI=; b=tHmPJZUCToedZH0Khr2fAir1EVZdmg1TKq27+KLpk5ZLSTaQfmvRPaiRyMGES4y1rh YB9Tsnyz/W+i8cyNJTVrhL4VRJnoETtIUzTTx/Ge1wMKjIzii2YvGkH3a0DCRA2DBWPg i91BrwR2zxNpSYdik/d8MR2nzfGQOEWOaHMYgjuzpF/fPciQzsrBJCOdepOH2m0PCye6 NuH2Yw863EwNpW4nbuoEqSGRWESiZQVuqU74N02/DZJ/73WnQPww8InKq1yZfYGwXO/n nTXNcekTdlnSaX6fxILl3qQJjQcRcOoCLrMalGriKdciDPn+6fJbQrVIjnJfDHTQURUn XTPg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711645404; x=1712250204; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=DyO3NL4DZ5+zyEmGrUmGhqOJCsHXA1HlVzHyeVmkchI=; b=eBDqcVK3A7pNtkUSquIJkMQ6hSWHAuT6vTwwSopFyYkcFD8/ZBJWraK0BhgOqS4Zpd sCpOjv8tGfrGD3hnEsqCXOWHF1yTGVIOwjfaV9U4WUb7I099tYBqXtIeZmj2G9+hcD1C XhwdCeg8dmHFq9r5zF28B4PhbWRkJzFftmecnZjChvHfSTWL5W2qH+8JAyTndRzoFzkD AbMn8/d2piMfl43lX+SIGXv0095URlMxFaSkVox2AQuL0jJtGaur5Ch862WF0m9eOwhN +I0GrPUMRa+UOsHIhf0tG5aWa+ZwT2yFZ31KY6puyYyqWuVG+VB/K2j9zHt+dEoK/52v J8Kw== X-Gm-Message-State: AOJu0YzeQ56qysJsC9OWI3rlwkHYK74A5e4uItpSJYuwHySg1nXCTZHb FbmLN0OeF/zpp0lDkBZ0S/MzGGA6fwzkb2TRCS5n+MXCRIrNarxsmV9NRG2S4NuV1nq99MKv8vn hZVhLN9bjTA== X-Google-Smtp-Source: AGHT+IEcAU6On1FebhvDzRMHxKgc0gG9W0lAfWMgqvnygBHzEBD0mgjfLuzwj379NGUbm10JUxpCruxTKGqE3g== X-Received: from edumazet1.c.googlers.com ([fda3:e722:ac3:cc00:2b:7d90:c0a8:395a]) (user=edumazet job=sendgmr) by 2002:a05:6902:2292:b0:ddd:7581:1234 with SMTP id dn18-20020a056902229200b00ddd75811234mr207996ybb.11.1711645404695; Thu, 28 Mar 2024 10:03:24 -0700 (PDT) Date: Thu, 28 Mar 2024 17:03:08 +0000 In-Reply-To: <20240328170309.2172584-1-edumazet@google.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240328170309.2172584-1-edumazet@google.com> X-Mailer: git-send-email 2.44.0.478.gd926399ef9-goog Message-ID: <20240328170309.2172584-8-edumazet@google.com> Subject: [PATCH net-next 7/8] net: rps: add rps_input_queue_head_add() helper From: Eric Dumazet To: "David S . Miller" , Jakub Kicinski , Paolo Abeni Cc: netdev@vger.kernel.org, eric.dumazet@gmail.com, Eric Dumazet X-Patchwork-Delegate: kuba@kernel.org process_backlog() can batch increments of sd->input_queue_head, saving some memory bandwidth. Also add READ_ONCE()/WRITE_ONCE() annotations around sd->input_queue_head accesses. Signed-off-by: Eric Dumazet --- include/net/rps.h | 9 +++++++-- net/core/dev.c | 13 ++++++++----- 2 files changed, 15 insertions(+), 7 deletions(-) diff --git a/include/net/rps.h b/include/net/rps.h index c13f829b8556fda63e76544c332f2c089f0d6ea4..135427bc6fcd29b9dad92a671c9a9f4efc975dec 100644 --- a/include/net/rps.h +++ b/include/net/rps.h @@ -51,13 +51,18 @@ static inline void rps_input_queue_tail_save(u32 *dest, u32 tail) #endif } -static inline void rps_input_queue_head_incr(struct softnet_data *sd) +static inline void rps_input_queue_head_add(struct softnet_data *sd, int val) { #ifdef CONFIG_RPS - sd->input_queue_head++; + WRITE_ONCE(sd->input_queue_head, sd->input_queue_head + val); #endif } +static inline void rps_input_queue_head_incr(struct softnet_data *sd) +{ + rps_input_queue_head_add(sd, 1); +} + /* * The rps_dev_flow_table structure contains a table of flow mappings. */ diff --git a/net/core/dev.c b/net/core/dev.c index 1fe7c6b10793d45a03461ee581d240d2442f9e17..59e7fc30e8f03880340bfbeda0fa9e9ac757a168 100644 --- a/net/core/dev.c +++ b/net/core/dev.c @@ -4518,7 +4518,7 @@ set_rps_cpu(struct net_device *dev, struct sk_buff *skb, out: #endif rflow->last_qtail = - per_cpu(softnet_data, next_cpu).input_queue_head; + READ_ONCE(per_cpu(softnet_data, next_cpu).input_queue_head); } rflow->cpu = next_cpu; @@ -4600,7 +4600,7 @@ static int get_rps_cpu(struct net_device *dev, struct sk_buff *skb, */ if (unlikely(tcpu != next_cpu) && (tcpu >= nr_cpu_ids || !cpu_online(tcpu) || - ((int)(per_cpu(softnet_data, tcpu).input_queue_head - + ((int)(READ_ONCE(per_cpu(softnet_data, tcpu).input_queue_head) - READ_ONCE(rflow->last_qtail))) >= 0)) { tcpu = next_cpu; rflow = set_rps_cpu(dev, skb, rflow, next_cpu); @@ -4655,7 +4655,7 @@ bool rps_may_expire_flow(struct net_device *dev, u16 rxq_index, rflow = &flow_table->flows[flow_id]; cpu = READ_ONCE(rflow->cpu); if (rflow->filter == filter_id && cpu < nr_cpu_ids && - ((int)(per_cpu(softnet_data, cpu).input_queue_head - + ((int)(READ_ONCE(per_cpu(softnet_data, cpu).input_queue_head) - READ_ONCE(rflow->last_qtail)) < (int)(10 * flow_table->mask))) expire = false; @@ -6035,9 +6035,10 @@ static int process_backlog(struct napi_struct *napi, int quota) rcu_read_lock(); __netif_receive_skb(skb); rcu_read_unlock(); - rps_input_queue_head_incr(sd); - if (++work >= quota) + if (++work >= quota) { + rps_input_queue_head_add(sd, work); return work; + } } @@ -6060,6 +6061,8 @@ static int process_backlog(struct napi_struct *napi, int quota) backlog_unlock_irq_enable(sd); } + if (work) + rps_input_queue_head_add(sd, work); return work; } From patchwork Thu Mar 28 17:03:09 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Dumazet X-Patchwork-Id: 13609227 X-Patchwork-Delegate: kuba@kernel.org Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A92C31327FF for ; Thu, 28 Mar 2024 17:03:27 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711645409; cv=none; b=sit2Fg+dKO1h63+DVDAKBUFfCqpPHvJYvKwHsxKlOYX4FFMyNP5MJYa888F9QNAr6ZEB80GVtGEYs7GZ3nlHgUO6ePzy95LLKYDzmOmHC5lAG4mZYA+PPOk62LKJIIaYvsQQgz+eohmjTwkza1pg7ZDfoCd+W2ptxT/LmtilPW4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711645409; c=relaxed/simple; bh=cDQ06vC2AO2hdTXvL2nVcFw/pVw+7ML0rIty1NNvoeY=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=LHT+kbe63y9StQwflP3rycBTrMv5NdZdqWaJWEOJLRJ+A5dnZkYRGHuUSjKgjdeCiJ4IrE00t9Mqu2r5EDQiXQacOZNIKW5FfNzrRSBP8J0kf7fJMlAoOP4qrBg41gwZh2Lst/VbZvfxZtvR1CJpQ1W+hyozJ96qQ2lna8bcDWk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--edumazet.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=Mo8XnrxT; arc=none smtp.client-ip=209.85.219.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--edumazet.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="Mo8XnrxT" Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-dbf618042daso1644776276.0 for ; Thu, 28 Mar 2024 10:03:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1711645406; x=1712250206; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=P4Rf3n0ThGg2QZAaciZ/MGFx+ZUjVEIBegGBLhJw3P8=; b=Mo8XnrxTutPEuyDHw4HvrkJUF2u9wCB0LNXSY7HxVn+FWo68kkKav6TVCqVL0hTe9l ntkqnP1Gg7biSlYp4N6mV7+2ArhObXI37aO++GvTPjVyKT/SkGThsS2gtFnmcQspj6zF ADri6GzHQ5scSFCfB8znRSRRLHyyeallr009GQXPHW3xS18nxIRCvCwAwYap+HcQsTZC YGclO2DuRWBmPFS95AM7yaVSpCs3LN49Imki5FaUZN6909kl7a/q01yy17Lo2VMk+33Q kg5dNn0786of2u34dfteFZx/MHXxBkX2afrtGmWZdGIOd2Sk8T32gL9O+2w54ylL45C6 SlZA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711645406; x=1712250206; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=P4Rf3n0ThGg2QZAaciZ/MGFx+ZUjVEIBegGBLhJw3P8=; b=LE6z0XK97LrhxNm+hxPjcbVDfTCmYML0aMKnYi2+tHVxs3gAkdv3MtWw089yJEMelL 1H+GWIGoJNvWhJGFYQ/RXjL0DvuAErndnYXRiNCqUmmS+46yGM4khfRSkSuiXM/3kHSG i1JxqeHiyMyWgPqNeRsJdqa3t9z4D/F8X3smgwbYvOj2pabuGGS4fyLe0EqbYIVkf/uK 6uiE6TWrFZsVCBtj8hPdT43zxp6z3Csm2FwCooZ9DFxJdxIyEhO2QUXlUF1ONt2XMcKA D1hU6ZJJt1Q8ekArcnacJ6xHyQWJBvfygknQu1tV1z702wqMx1kUsRdDNo5KQdIfQi1n IHUQ== X-Gm-Message-State: AOJu0YwwJ6q/7uWKOwKx5Zj6xKudoCQImzy4q/tgpXGFwuzLcVaQU98e Bq98SYyIfRxxvs6gGztQPIERFZXoAoSwd9ceiK47CuUX6jds6N0T3d7QE/1TSAiRgQTb3GzqFiy 1RmwPuvQclw== X-Google-Smtp-Source: AGHT+IE8RUsLi5NEWaCQpBxCzHAtR+Tt/Xs83paI2PEaSb0MWJvXoJgJAFNVO+Mxu01+BA4dDmGcfec5aB2qRw== X-Received: from edumazet1.c.googlers.com ([fda3:e722:ac3:cc00:2b:7d90:c0a8:395a]) (user=edumazet job=sendgmr) by 2002:a05:6902:2413:b0:dc2:5273:53f9 with SMTP id dr19-20020a056902241300b00dc2527353f9mr249416ybb.1.1711645406759; Thu, 28 Mar 2024 10:03:26 -0700 (PDT) Date: Thu, 28 Mar 2024 17:03:09 +0000 In-Reply-To: <20240328170309.2172584-1-edumazet@google.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240328170309.2172584-1-edumazet@google.com> X-Mailer: git-send-email 2.44.0.478.gd926399ef9-goog Message-ID: <20240328170309.2172584-9-edumazet@google.com> Subject: [PATCH net-next 8/8] net: rps: move received_rps field to a better location From: Eric Dumazet To: "David S . Miller" , Jakub Kicinski , Paolo Abeni Cc: netdev@vger.kernel.org, eric.dumazet@gmail.com, Eric Dumazet X-Patchwork-Delegate: kuba@kernel.org Commit 14d898f3c1b3 ("dev: Move received_rps counter next to RPS members in softnet data") was unfortunate: received_rps is dirtied by a cpu and never read by other cpus in fast path. Its presence in the hot RPS cache line (shared by many cpus) is hurting RPS/RFS performance. Signed-off-by: Eric Dumazet --- include/linux/netdevice.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/include/linux/netdevice.h b/include/linux/netdevice.h index 14f19cc2616452d7e6afbbaa52f8ad3e61a419e9..274d8db48b4858c70b43ea4628544e924ba6a263 100644 --- a/include/linux/netdevice.h +++ b/include/linux/netdevice.h @@ -3203,6 +3203,7 @@ struct softnet_data { struct softnet_data *rps_ipi_list; #endif + unsigned int received_rps; bool in_net_rx_action; bool in_napi_threaded_poll; @@ -3235,7 +3236,6 @@ struct softnet_data { unsigned int cpu; unsigned int input_queue_tail; #endif - unsigned int received_rps; struct sk_buff_head input_pkt_queue; struct napi_struct backlog;