From patchwork Wed Mar 6 16:00:14 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Dumazet X-Patchwork-Id: 13584249 X-Patchwork-Delegate: kuba@kernel.org Received: from mail-qk1-f201.google.com (mail-qk1-f201.google.com [209.85.222.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B09A41361DA for ; Wed, 6 Mar 2024 16:00:36 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.222.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709740838; cv=none; b=i/Ab8sdCqY33bOyTCrF3UxaD7SYliyA/jaLEJLIPwp+rhL3B5g8R/JMFwePogodT8hgBkqcZS7NU8uF7pXug3cKfodiGS9AZ06B+gHDSEHkTDIiOuIAsJTk9R7KYxGUkKy0KSIOEJbpowXQq8Eya2a59S2Dxat3jUBFZ4QdRoZs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709740838; c=relaxed/simple; bh=b8tGS5Bg1ZQGioOPXDGL09P1xNFDTIXeGmKAUoGqJpc=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=T4cPsSU9iBpoDjREX+dy5cmtAoYI7LFM2EKSZOMW3g2ZCRkG2oYf0L0xgLmW2//evU4WvWscy3ibvGU4M7kODDlhCptc7zrLVyhpVCxmNPiqYRJA2j/7MD9gh1ExvK1k3rslMJwLA3Bzg3XxRN//FAzXIp7eg0G7047ENTmwUO8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--edumazet.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=LQIir/m1; arc=none smtp.client-ip=209.85.222.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--edumazet.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="LQIir/m1" Received: by mail-qk1-f201.google.com with SMTP id af79cd13be357-783350c4584so290131485a.1 for ; Wed, 06 Mar 2024 08:00:36 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1709740835; x=1710345635; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=NV810Z4fsrtVPUGOMX/5wV9XHl3oeUEGwp+Vxn8EwaU=; b=LQIir/m1CNvqp65BF14Pij+vDxnngq+wbf9P0Zymbn7qhEMf8wbh5lNfp1e0pPDN9i cRXQjmAFcJAmDnGW38xBYC5u7306njtU77jSoVWyFi1yJzhJIlSgC5Rj8cU5s5G0nX3/ vbFaVZnzIgKfSrONtnU0EDGFmkBY52KEkIoUcu86LPWgj/K/crkK3hwaO86dNZweqopH +9wD8l4YTAbLwyXBXk9ZG4ZrXpjea86Z4zUw9Yz5KpqkPtqkXyc/fGf6VNxnZTtWtkgb gAD12lzIJQXKwxeDh1wZCjGyiB81r860nIISzBUB8b1k2ICoGVpYafieMcbgNfpa22w7 w2Uw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709740835; x=1710345635; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=NV810Z4fsrtVPUGOMX/5wV9XHl3oeUEGwp+Vxn8EwaU=; b=bhTroAQiCPDk07TBubu6DI6IgzwqOSYgeTXsghgRYEHRpYPZ+nJErRgdcWw1+pgYyx 1nSQoxurZqDT+2o78mz65In+CAynqTwrqduaRbBZHfgHr7fXVLywcoVVYsA492eiaOHj EZtvbBO4ewQyifsY1wfrOkWlUw297eOQGfwcmY39MPPyURU2TTGzly2YzhbWlK+wMTO0 Am84KSjSKajuFNgqdebTO1fGD+3GWeKCtFCSoAbPbIojV/5mlzTHs6738xlAYHqGP/i+ kQMbgZzf5hqAItiQ0SXLUrpsxbfQAtFpjV9zbozvgxjncIVLOiwixbsnZ/4i9ZoW+oy2 WvOg== X-Forwarded-Encrypted: i=1; AJvYcCVAx0QfcUtnbA9zdMwMq+zFj6sLgMqsbpb6TKIcxL6o/J8aRq+4kH/c3ua+vsTMdk0ZTqUoss9Jtz6SqE37wGCzkNVRD/fi X-Gm-Message-State: AOJu0YxG7lwJ59IXCC+Pms1NfsMoluNROyUn/vW4tXS/NOwNhl0eeLFF apMgn8ma4rPo5f7BtVmWatRPUqQOcc3p1r1w2r0qr9OqZb9X+cWmi5Snw0KzqWPhYmKP/60DKas m/3DhS0IIGA== X-Google-Smtp-Source: AGHT+IFJ8HhpzGe8FncW+AjgyndWpMJKHdp6hB+NQWsEat+S3uMe07jDrTk5ahb7MlnxMuRTGRUXRaKd+94Ohw== X-Received: from edumazet1.c.googlers.com ([fda3:e722:ac3:cc00:2b:7d90:c0a8:395a]) (user=edumazet job=sendgmr) by 2002:a05:620a:3906:b0:788:33f3:fef1 with SMTP id qr6-20020a05620a390600b0078833f3fef1mr162710qkn.11.1709740835671; Wed, 06 Mar 2024 08:00:35 -0800 (PST) Date: Wed, 6 Mar 2024 16:00:14 +0000 In-Reply-To: <20240306160031.874438-1-edumazet@google.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240306160031.874438-1-edumazet@google.com> X-Mailer: git-send-email 2.44.0.278.ge034bb2e1d-goog Message-ID: <20240306160031.874438-2-edumazet@google.com> Subject: [PATCH v2 net-next 01/18] net: introduce struct net_hotdata From: Eric Dumazet To: "David S . Miller" , Jakub Kicinski , Paolo Abeni Cc: David Ahern , Willem de Bruijn , Soheil Hassas Yeganeh , Neal Cardwell , netdev@vger.kernel.org, eric.dumazet@gmail.com, Eric Dumazet X-Patchwork-Delegate: kuba@kernel.org Instead of spreading networking critical fields all over the places, add a custom net_hotdata structure so that we can precisely control its layout. In this first patch, move : - gro_normal_batch used in rx (GRO stack) - offload_base used in rx and tx (GRO and TSO stacks) Signed-off-by: Eric Dumazet Acked-by: Soheil Hassas Yeganeh Reviewed-by: David Ahern --- include/linux/netdevice.h | 1 - include/net/gro.h | 5 ++--- include/net/hotdata.h | 15 +++++++++++++++ net/core/Makefile | 1 + net/core/gro.c | 15 ++++++--------- net/core/gso.c | 4 ++-- net/core/hotdata.c | 9 +++++++++ net/core/sysctl_net_core.c | 3 ++- 8 files changed, 37 insertions(+), 16 deletions(-) create mode 100644 include/net/hotdata.h create mode 100644 net/core/hotdata.c diff --git a/include/linux/netdevice.h b/include/linux/netdevice.h index c41019f3417948d09ae9a50b57b856be1dc8ae42..15ce809e0541078bff7a48b8d7cb2cf2c1ac8a93 100644 --- a/include/linux/netdevice.h +++ b/include/linux/netdevice.h @@ -4790,7 +4790,6 @@ void dev_get_tstats64(struct net_device *dev, struct rtnl_link_stats64 *s); extern int netdev_max_backlog; extern int dev_rx_weight; extern int dev_tx_weight; -extern int gro_normal_batch; enum { NESTED_SYNC_IMM_BIT, diff --git a/include/net/gro.h b/include/net/gro.h index 2b58671a65492bf3f9dabf1e7a2d985cee007e11..d6fc8fbd37302338fc09ab01fead899002c5833f 100644 --- a/include/net/gro.h +++ b/include/net/gro.h @@ -9,6 +9,7 @@ #include #include #include +#include struct napi_gro_cb { union { @@ -446,7 +447,7 @@ static inline void gro_normal_one(struct napi_struct *napi, struct sk_buff *skb, { list_add_tail(&skb->list, &napi->rx_list); napi->rx_count += segs; - if (napi->rx_count >= READ_ONCE(gro_normal_batch)) + if (napi->rx_count >= READ_ONCE(net_hotdata.gro_normal_batch)) gro_normal_list(napi); } @@ -493,6 +494,4 @@ static inline void inet6_get_iif_sdif(const struct sk_buff *skb, int *iif, int * #endif } -extern struct list_head offload_base; - #endif /* _NET_IPV6_GRO_H */ diff --git a/include/net/hotdata.h b/include/net/hotdata.h new file mode 100644 index 0000000000000000000000000000000000000000..6ed32e4e34aa3bdc6e860f5a8a6cab69c36c7fad --- /dev/null +++ b/include/net/hotdata.h @@ -0,0 +1,15 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ +#ifndef _NET_HOTDATA_H +#define _NET_HOTDATA_H + +#include + +/* Read mostly data used in network fast paths. */ +struct net_hotdata { + struct list_head offload_base; + int gro_normal_batch; +}; + +extern struct net_hotdata net_hotdata; + +#endif /* _NET_HOTDATA_H */ diff --git a/net/core/Makefile b/net/core/Makefile index 821aec06abf1460d3504de4b6b66a328bba748d8..6e6548011fae570e345717e43eb3c1a6133571c7 100644 --- a/net/core/Makefile +++ b/net/core/Makefile @@ -18,6 +18,7 @@ obj-y += dev.o dev_addr_lists.o dst.o netevent.o \ obj-$(CONFIG_NETDEV_ADDR_LIST_TEST) += dev_addr_lists_test.o obj-y += net-sysfs.o +obj-y += hotdata.o obj-$(CONFIG_PAGE_POOL) += page_pool.o page_pool_user.o obj-$(CONFIG_PROC_FS) += net-procfs.o obj-$(CONFIG_NET_PKTGEN) += pktgen.o diff --git a/net/core/gro.c b/net/core/gro.c index 6a0edbd826a17573b51c5f71e20ff0c09364fc21..ee30d4f0c03876e78795397d1c495881a2c9e80f 100644 --- a/net/core/gro.c +++ b/net/core/gro.c @@ -10,9 +10,6 @@ #define GRO_MAX_HEAD (MAX_HEADER + 128) static DEFINE_SPINLOCK(offload_lock); -struct list_head offload_base __read_mostly = LIST_HEAD_INIT(offload_base); -/* Maximum number of GRO_NORMAL skbs to batch up for list-RX */ -int gro_normal_batch __read_mostly = 8; /** * dev_add_offload - register offload handlers @@ -31,7 +28,7 @@ void dev_add_offload(struct packet_offload *po) struct packet_offload *elem; spin_lock(&offload_lock); - list_for_each_entry(elem, &offload_base, list) { + list_for_each_entry(elem, &net_hotdata.offload_base, list) { if (po->priority < elem->priority) break; } @@ -55,7 +52,7 @@ EXPORT_SYMBOL(dev_add_offload); */ static void __dev_remove_offload(struct packet_offload *po) { - struct list_head *head = &offload_base; + struct list_head *head = &net_hotdata.offload_base; struct packet_offload *po1; spin_lock(&offload_lock); @@ -235,9 +232,9 @@ int skb_gro_receive(struct sk_buff *p, struct sk_buff *skb) static void napi_gro_complete(struct napi_struct *napi, struct sk_buff *skb) { + struct list_head *head = &net_hotdata.offload_base; struct packet_offload *ptype; __be16 type = skb->protocol; - struct list_head *head = &offload_base; int err = -ENOENT; BUILD_BUG_ON(sizeof(struct napi_gro_cb) > sizeof(skb->cb)); @@ -444,7 +441,7 @@ static enum gro_result dev_gro_receive(struct napi_struct *napi, struct sk_buff { u32 bucket = skb_get_hash_raw(skb) & (GRO_HASH_BUCKETS - 1); struct gro_list *gro_list = &napi->gro_hash[bucket]; - struct list_head *head = &offload_base; + struct list_head *head = &net_hotdata.offload_base; struct packet_offload *ptype; __be16 type = skb->protocol; struct sk_buff *pp = NULL; @@ -550,7 +547,7 @@ static enum gro_result dev_gro_receive(struct napi_struct *napi, struct sk_buff struct packet_offload *gro_find_receive_by_type(__be16 type) { - struct list_head *offload_head = &offload_base; + struct list_head *offload_head = &net_hotdata.offload_base; struct packet_offload *ptype; list_for_each_entry_rcu(ptype, offload_head, list) { @@ -564,7 +561,7 @@ EXPORT_SYMBOL(gro_find_receive_by_type); struct packet_offload *gro_find_complete_by_type(__be16 type) { - struct list_head *offload_head = &offload_base; + struct list_head *offload_head = &net_hotdata.offload_base; struct packet_offload *ptype; list_for_each_entry_rcu(ptype, offload_head, list) { diff --git a/net/core/gso.c b/net/core/gso.c index 9e1803bfc9c6cac2fe7054661f8995909a6c28d9..bcd156372f4df080f83cc45fc96df1789125a8ae 100644 --- a/net/core/gso.c +++ b/net/core/gso.c @@ -17,7 +17,7 @@ struct sk_buff *skb_eth_gso_segment(struct sk_buff *skb, struct packet_offload *ptype; rcu_read_lock(); - list_for_each_entry_rcu(ptype, &offload_base, list) { + list_for_each_entry_rcu(ptype, &net_hotdata.offload_base, list) { if (ptype->type == type && ptype->callbacks.gso_segment) { segs = ptype->callbacks.gso_segment(skb, features); break; @@ -48,7 +48,7 @@ struct sk_buff *skb_mac_gso_segment(struct sk_buff *skb, __skb_pull(skb, vlan_depth); rcu_read_lock(); - list_for_each_entry_rcu(ptype, &offload_base, list) { + list_for_each_entry_rcu(ptype, &net_hotdata.offload_base, list) { if (ptype->type == type && ptype->callbacks.gso_segment) { segs = ptype->callbacks.gso_segment(skb, features); break; diff --git a/net/core/hotdata.c b/net/core/hotdata.c new file mode 100644 index 0000000000000000000000000000000000000000..abb8ad19d59acc0d7d6e1b06f4506afa42bde44b --- /dev/null +++ b/net/core/hotdata.c @@ -0,0 +1,9 @@ +// SPDX-License-Identifier: GPL-2.0-or-later +#include +#include +#include + +struct net_hotdata net_hotdata __cacheline_aligned = { + .offload_base = LIST_HEAD_INIT(net_hotdata.offload_base), + .gro_normal_batch = 8, +}; diff --git a/net/core/sysctl_net_core.c b/net/core/sysctl_net_core.c index 986f15e5d6c41250c8b9099fc1d2883112e77ffb..0eb1242eabbe0d3ea58886b1db409c9d991ac672 100644 --- a/net/core/sysctl_net_core.c +++ b/net/core/sysctl_net_core.c @@ -23,6 +23,7 @@ #include #include #include +#include #include "dev.h" @@ -632,7 +633,7 @@ static struct ctl_table net_core_table[] = { }, { .procname = "gro_normal_batch", - .data = &gro_normal_batch, + .data = &net_hotdata.gro_normal_batch, .maxlen = sizeof(unsigned int), .mode = 0644, .proc_handler = proc_dointvec_minmax, From patchwork Wed Mar 6 16:00:15 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Dumazet X-Patchwork-Id: 13584250 X-Patchwork-Delegate: kuba@kernel.org Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B04DC13665D for ; Wed, 6 Mar 2024 16:00:42 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709740844; cv=none; b=dWuLrKgzi5gx1oN/BaCSKuYKaR8fOriR8haxq7Qi+5Petf/XTt2xBcurOPA8hdcL6z+D1hSGkuAIEjVa+X7rid7PrFl5vViSsrrmQWeYIw+vv8jRYmsRGsLJRpJik+Xo+4oF2b5RHW7cjjRLoj/oSvuTsC/zK5uVrOC3WQQ9MyI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709740844; c=relaxed/simple; bh=kMWm8kH4b+ITapk4/NpQ8GbNj7rfzxEY18zU3EY8ZOI=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=sXT9L2CwAJtwCzqb2OuMjZfhkFgB10lh01dpdrx+AMYB0qBG7P2f8mlCxeYItNl5amt8G4WYRe5pgmo5eciQV5CDerNIMdCIZNWUTwUWqAnC95cHEyDbf1WUID8fDvQk9zXWgbIHy0Md+wh51sYmOxBpvSmiXEhK8JwcR1oGTi8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--edumazet.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=CTzoZ48R; arc=none smtp.client-ip=209.85.128.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--edumazet.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="CTzoZ48R" Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-609a8fc232bso52423137b3.0 for ; Wed, 06 Mar 2024 08:00:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1709740841; x=1710345641; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=+qh3Zyb/w9TvxBiWKj2uKBnLvw6V7yRjmepRKjM7p4M=; b=CTzoZ48RmYVptcJ+bl6HMQuFmcrF2SooaWmYI3jyrCOBpo5BfTmjO7Op1JstFMEJfc aVmUJrgF4Qjef9c1aCUC2LPLw5cWnuDkuDblvtc2M9nB87jx6FIICM+BMWfWH+Z19OAv wP2G9hAWI7htH6lr3lwfi73+sj2N6GOTpoGCF5KXRVEhT5d/gNMtnOkHGQJ4horXDUsa vtsOMl1M/sHdAs1+eEGdSAjelmx92KhbhRGcvfwaf7LcMXsXXozpLPb3UV0OW9jn8YEg YSms/YNir6BgPdGlvr345nlBS/dpmzDJy/lOSoZF+AdUmIepIoJ7W4PVB6bjmD2vSwCC 5Gdg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709740841; x=1710345641; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=+qh3Zyb/w9TvxBiWKj2uKBnLvw6V7yRjmepRKjM7p4M=; b=BoAN3ExZioeoNejE+7QFb+dzHS2HJ7wkjRcXtJxqm2acQKq+2Sp5D1Bp564mL8wntn Lf0u2mSKY7TA2wC7BoclAM3LdmMCWNA7U4kcFP8svTZ1x8NIyYwb+32NAflt1tFqjkCR FfmWkfdGn0hzXrEddozvGkqTdMuUlOcuA+q89Wd1H9sweasrZfBJu+nqowBakEYNadIz ywfs8StS5KOkHMKAIBcc5NRZ7LlOfijjRgkBjq9CyLNwH8G2pUkdso08tR+loMJ5hknG obSFvYm8bqhgkOSuzjqjiWEGD+ycGybkg3L/BVa7jsnq3Oyy5f0dZYNpdX5t2Y4sBs7O 83XQ== X-Forwarded-Encrypted: i=1; AJvYcCXo6kKPcSAtVyVcHLGTUN+G9nIOdRJPwde/Ia64ES2HKsdPtZ41HUCPJdmnVJvke3tT4lBtUpsaRs8v0lvmLhk0ACvySzyU X-Gm-Message-State: AOJu0YzyFg7EDUbPgZtE3WPJpAgMBbDpQDDBNt9OpIb0T7sesD8KYy0v 23WQUcSEDoWz9MWhbiziZLX650TePEle0ARXK4ACzMjkfkio46t/GgcmDVmeUHgCOU+0QWUFOQJ ccZHzIV9HRw== X-Google-Smtp-Source: AGHT+IHu67eXwLlEmc6YSNh1k8oQMoU41Pfbk34iphMBVBGm91WrNVj+bQSrDpf1USRutGGAPwQNFU+EC2fjdQ== X-Received: from edumazet1.c.googlers.com ([fda3:e722:ac3:cc00:2b:7d90:c0a8:395a]) (user=edumazet job=sendgmr) by 2002:a05:6902:1148:b0:dc6:b813:5813 with SMTP id p8-20020a056902114800b00dc6b8135813mr501255ybu.9.1709740841603; Wed, 06 Mar 2024 08:00:41 -0800 (PST) Date: Wed, 6 Mar 2024 16:00:15 +0000 In-Reply-To: <20240306160031.874438-1-edumazet@google.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240306160031.874438-1-edumazet@google.com> X-Mailer: git-send-email 2.44.0.278.ge034bb2e1d-goog Message-ID: <20240306160031.874438-3-edumazet@google.com> Subject: [PATCH v2 net-next 02/18] net: move netdev_budget and netdev_budget to net_hotdata From: Eric Dumazet To: "David S . Miller" , Jakub Kicinski , Paolo Abeni Cc: David Ahern , Willem de Bruijn , Soheil Hassas Yeganeh , Neal Cardwell , netdev@vger.kernel.org, eric.dumazet@gmail.com, Eric Dumazet X-Patchwork-Delegate: kuba@kernel.org netdev_budget and netdev_budget are used in rx path (net_rx_action()) Move them into net_hotdata for better cache locality. Signed-off-by: Eric Dumazet Acked-by: Soheil Hassas Yeganeh Reviewed-by: David Ahern --- include/net/hotdata.h | 2 ++ net/core/dev.c | 7 ++----- net/core/dev.h | 2 -- net/core/hotdata.c | 6 ++++++ net/core/sysctl_net_core.c | 4 ++-- 5 files changed, 12 insertions(+), 9 deletions(-) diff --git a/include/net/hotdata.h b/include/net/hotdata.h index 6ed32e4e34aa3bdc6e860f5a8a6cab69c36c7fad..72170223385ebe65cce42f762b3686c072291d36 100644 --- a/include/net/hotdata.h +++ b/include/net/hotdata.h @@ -8,6 +8,8 @@ struct net_hotdata { struct list_head offload_base; int gro_normal_batch; + int netdev_budget; + int netdev_budget_usecs; }; extern struct net_hotdata net_hotdata; diff --git a/net/core/dev.c b/net/core/dev.c index fe054cbd41e92cbca87f1c0640c6ebe4fb6b2d86..0102a1810e7b148f465b87886b743e3d12c0e578 100644 --- a/net/core/dev.c +++ b/net/core/dev.c @@ -4410,9 +4410,6 @@ EXPORT_SYMBOL(netdev_max_backlog); int netdev_tstamp_prequeue __read_mostly = 1; unsigned int sysctl_skb_defer_max __read_mostly = 64; -int netdev_budget __read_mostly = 300; -/* Must be at least 2 jiffes to guarantee 1 jiffy timeout */ -unsigned int __read_mostly netdev_budget_usecs = 2 * USEC_PER_SEC / HZ; int weight_p __read_mostly = 64; /* old backlog weight */ int dev_weight_rx_bias __read_mostly = 1; /* bias for backlog weight */ int dev_weight_tx_bias __read_mostly = 1; /* bias for output_queue quota */ @@ -6790,8 +6787,8 @@ static __latent_entropy void net_rx_action(struct softirq_action *h) { struct softnet_data *sd = this_cpu_ptr(&softnet_data); unsigned long time_limit = jiffies + - usecs_to_jiffies(READ_ONCE(netdev_budget_usecs)); - int budget = READ_ONCE(netdev_budget); + usecs_to_jiffies(READ_ONCE(net_hotdata.netdev_budget_usecs)); + int budget = READ_ONCE(net_hotdata.netdev_budget); LIST_HEAD(list); LIST_HEAD(repoll); diff --git a/net/core/dev.h b/net/core/dev.h index 45892267848d7a35a09aea95f04cfd9b72204d3c..9a6170530850c78508f9234ec82b174f4bf5a4a3 100644 --- a/net/core/dev.h +++ b/net/core/dev.h @@ -38,8 +38,6 @@ int dev_addr_init(struct net_device *dev); void dev_addr_check(struct net_device *dev); /* sysctls not referred to from outside net/core/ */ -extern int netdev_budget; -extern unsigned int netdev_budget_usecs; extern unsigned int sysctl_skb_defer_max; extern int netdev_tstamp_prequeue; extern int netdev_unregister_timeout_secs; diff --git a/net/core/hotdata.c b/net/core/hotdata.c index abb8ad19d59acc0d7d6e1b06f4506afa42bde44b..907d69120397dfb8d5a901912b72580fe256c762 100644 --- a/net/core/hotdata.c +++ b/net/core/hotdata.c @@ -1,9 +1,15 @@ // SPDX-License-Identifier: GPL-2.0-or-later #include #include +#include #include + struct net_hotdata net_hotdata __cacheline_aligned = { .offload_base = LIST_HEAD_INIT(net_hotdata.offload_base), .gro_normal_batch = 8, + + .netdev_budget = 300, + /* Must be at least 2 jiffes to guarantee 1 jiffy timeout */ + .netdev_budget_usecs = 2 * USEC_PER_SEC / HZ, }; diff --git a/net/core/sysctl_net_core.c b/net/core/sysctl_net_core.c index 0eb1242eabbe0d3ea58886b1db409c9d991ac672..a9c2d798b219506da75a5d0a30d490ff4011d668 100644 --- a/net/core/sysctl_net_core.c +++ b/net/core/sysctl_net_core.c @@ -577,7 +577,7 @@ static struct ctl_table net_core_table[] = { #endif { .procname = "netdev_budget", - .data = &netdev_budget, + .data = &net_hotdata.netdev_budget, .maxlen = sizeof(int), .mode = 0644, .proc_handler = proc_dointvec @@ -600,7 +600,7 @@ static struct ctl_table net_core_table[] = { }, { .procname = "netdev_budget_usecs", - .data = &netdev_budget_usecs, + .data = &net_hotdata.netdev_budget_usecs, .maxlen = sizeof(unsigned int), .mode = 0644, .proc_handler = proc_dointvec_minmax, From patchwork Wed Mar 6 16:00:16 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Dumazet X-Patchwork-Id: 13584251 X-Patchwork-Delegate: kuba@kernel.org Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C8FD413667B for ; Wed, 6 Mar 2024 16:00:44 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709740846; cv=none; b=iOkfSevPrClMKvfvmJlhIVHy2Iqw7bcVOIZv6jsE8fbyjEcwxDxgEvJz+UecKPpvMcbQnnbaQ1VHPqdrBsGDLaxJnz1RpdE1Lg2S8tJKT2itpv2o1E6xWnbHDKtDRwOR/Poluz6LeCAyokOZTrXV7C5o56nkxFuy+YyYVmzxvXI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709740846; c=relaxed/simple; bh=f4wxbe4KgsVU2UnfrwyI6uMEOA7LUdOMPKRsbXWElNU=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=cUA658yPo+xL4gYzW/Ss0SVH6BkMtfuxVc7Iu/tItr3y5DorfUf+m8aGIW+jNzmL8lVzr87IFqOlEXtfyJYuZD8h5VmJPK3Mr2xK7ELy+grF2azDu5uW/tFA/1F/56w7rJ3WT06J1rAbqPfcxLBB1hkLCdKITX2zz46f1/vG3A4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--edumazet.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=e41lbpT7; arc=none smtp.client-ip=209.85.128.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--edumazet.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="e41lbpT7" Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-6096493f3d3so94987357b3.2 for ; Wed, 06 Mar 2024 08:00:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1709740844; x=1710345644; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=1rVDJdLTVx9j+9qoaD0w8tRyN1ITErVjhB+hHzPdz9c=; b=e41lbpT7w2T02buNAMB5yE93PFx3NhciNzqmWJPAeZA8ihKrEUvgMQY55BSW1thra9 R9LC0R6mAXVnrowkaGO1uAcigAL8b6djBvX0CGchSYZfPGhiviLdNQu6ujPXEkstj7Yl giJvjzLGSeSPkwuo6UyXMSjTiDsZ4wpdXlY/hB0ksQk2KVpmFCVf1hAHr0tbqLVJDGlU eLYHmcO1MFwocAiUAXrwQh21l4mQDv8x/e99Xcj8iu3HMRA9thppyGeyf+XdcX6XbD03 NdKTqGp85+twAX0NS/V2wuosyjLb0skg90gXR93/olXRtv/kx9o3EJXej3kDnRwkpY6t ABBg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709740844; x=1710345644; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=1rVDJdLTVx9j+9qoaD0w8tRyN1ITErVjhB+hHzPdz9c=; b=rxzpaukaFnHC6xue7o/AtyWhVRyz3k4okKSBxmTB+v1I8a310wpvM5OA9Z6IPDxPeL 2MaSiZMt2Q3C0Ze3T3qLXzfC9+Tizthj03RBKNgSYuvXyaxAOIfk6ZmRy9TkeQFfclq7 6wW49SQk/RwghKNMlDesPv0QQU+480x7jxoiRclISoDjqdk76C9++lE4kHnAP9Jto/wj G212B7oWGjQEJ1PUzmV2V620QyGWX4GDB+AgJg5PXYeWtXzZknV0ctAvgWWkdLEzL1tO saJxwaaP7SHSuIPX0QIle27P6n1vDHbUHiEZrlKBkE3ecuKGmUJugrgxF3md+pD2wEUL rVew== X-Forwarded-Encrypted: i=1; AJvYcCUTCCm/xbhpFSRX6VWsYhtpTNI4qTLDKKgvyx+soRzXZfwNd79KYGtYj+pLU0OHV5PjyndpX+EaPps5PoMkUmf4f+TBTgy9 X-Gm-Message-State: AOJu0Yz/FbIQjsXLB6zi7uY8w9XnAQVrdtl0SEt+S0NB8ah9BXtAyute g9TliefuiRzZiVUjwgohiODtpeo//zd2Y+e80SGdlRb+kpoB9bD6fooktskERyIO7eMIZj7zpWz R1CxE2nJsoQ== X-Google-Smtp-Source: AGHT+IFEZd7XhA13yNbj1WGFW5PB/X2aA+9VFVBGeNwF/4bYof801w2hRM4KDDs2unpJpLoIAI0TMfKuhzZzDg== X-Received: from edumazet1.c.googlers.com ([fda3:e722:ac3:cc00:2b:7d90:c0a8:395a]) (user=edumazet job=sendgmr) by 2002:a05:690c:82:b0:609:78d7:4e9 with SMTP id be2-20020a05690c008200b0060978d704e9mr3210275ywb.6.1709740843847; Wed, 06 Mar 2024 08:00:43 -0800 (PST) Date: Wed, 6 Mar 2024 16:00:16 +0000 In-Reply-To: <20240306160031.874438-1-edumazet@google.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240306160031.874438-1-edumazet@google.com> X-Mailer: git-send-email 2.44.0.278.ge034bb2e1d-goog Message-ID: <20240306160031.874438-4-edumazet@google.com> Subject: [PATCH v2 net-next 03/18] net: move netdev_tstamp_prequeue into net_hotdata From: Eric Dumazet To: "David S . Miller" , Jakub Kicinski , Paolo Abeni Cc: David Ahern , Willem de Bruijn , Soheil Hassas Yeganeh , Neal Cardwell , netdev@vger.kernel.org, eric.dumazet@gmail.com, Eric Dumazet X-Patchwork-Delegate: kuba@kernel.org netdev_tstamp_prequeue is used in rx path. Move it to net_hotdata for better cache locality. Signed-off-by: Eric Dumazet Acked-by: Soheil Hassas Yeganeh Reviewed-by: David Ahern --- include/net/hotdata.h | 1 + net/core/dev.c | 10 +++++----- net/core/dev.h | 1 - net/core/hotdata.c | 2 ++ net/core/sysctl_net_core.c | 2 +- 5 files changed, 9 insertions(+), 7 deletions(-) diff --git a/include/net/hotdata.h b/include/net/hotdata.h index 72170223385ebe65cce42f762b3686c072291d36..149e56528537d8ed3365e46d6dc96e39c73a733a 100644 --- a/include/net/hotdata.h +++ b/include/net/hotdata.h @@ -10,6 +10,7 @@ struct net_hotdata { int gro_normal_batch; int netdev_budget; int netdev_budget_usecs; + int tstamp_prequeue; }; extern struct net_hotdata net_hotdata; diff --git a/net/core/dev.c b/net/core/dev.c index 0102a1810e7b148f465b87886b743e3d12c0e578..53ebdb55e8b7c0a6522eb3fdbb7bdd00948eb9a5 100644 --- a/net/core/dev.c +++ b/net/core/dev.c @@ -4408,7 +4408,6 @@ EXPORT_SYMBOL(__dev_direct_xmit); int netdev_max_backlog __read_mostly = 1000; EXPORT_SYMBOL(netdev_max_backlog); -int netdev_tstamp_prequeue __read_mostly = 1; unsigned int sysctl_skb_defer_max __read_mostly = 64; int weight_p __read_mostly = 64; /* old backlog weight */ int dev_weight_rx_bias __read_mostly = 1; /* bias for backlog weight */ @@ -5052,7 +5051,7 @@ static int netif_rx_internal(struct sk_buff *skb) { int ret; - net_timestamp_check(READ_ONCE(netdev_tstamp_prequeue), skb); + net_timestamp_check(READ_ONCE(net_hotdata.tstamp_prequeue), skb); trace_netif_rx(skb); @@ -5344,7 +5343,7 @@ static int __netif_receive_skb_core(struct sk_buff **pskb, bool pfmemalloc, int ret = NET_RX_DROP; __be16 type; - net_timestamp_check(!READ_ONCE(netdev_tstamp_prequeue), skb); + net_timestamp_check(!READ_ONCE(net_hotdata.tstamp_prequeue), skb); trace_netif_receive_skb(skb); @@ -5728,7 +5727,7 @@ static int netif_receive_skb_internal(struct sk_buff *skb) { int ret; - net_timestamp_check(READ_ONCE(netdev_tstamp_prequeue), skb); + net_timestamp_check(READ_ONCE(net_hotdata.tstamp_prequeue), skb); if (skb_defer_rx_timestamp(skb)) return NET_RX_SUCCESS; @@ -5758,7 +5757,8 @@ void netif_receive_skb_list_internal(struct list_head *head) INIT_LIST_HEAD(&sublist); list_for_each_entry_safe(skb, next, head, list) { - net_timestamp_check(READ_ONCE(netdev_tstamp_prequeue), skb); + net_timestamp_check(READ_ONCE(net_hotdata.tstamp_prequeue), + skb); skb_list_del_init(skb); if (!skb_defer_rx_timestamp(skb)) list_add_tail(&skb->list, &sublist); diff --git a/net/core/dev.h b/net/core/dev.h index 9a6170530850c78508f9234ec82b174f4bf5a4a3..2bcaf8eee50c179db2ca59880521b9be6ecd45c8 100644 --- a/net/core/dev.h +++ b/net/core/dev.h @@ -39,7 +39,6 @@ void dev_addr_check(struct net_device *dev); /* sysctls not referred to from outside net/core/ */ extern unsigned int sysctl_skb_defer_max; -extern int netdev_tstamp_prequeue; extern int netdev_unregister_timeout_secs; extern int weight_p; extern int dev_weight_rx_bias; diff --git a/net/core/hotdata.c b/net/core/hotdata.c index 907d69120397dfb8d5a901912b72580fe256c762..087c4c84987df07f11a87112a778a5cac608a654 100644 --- a/net/core/hotdata.c +++ b/net/core/hotdata.c @@ -12,4 +12,6 @@ struct net_hotdata net_hotdata __cacheline_aligned = { .netdev_budget = 300, /* Must be at least 2 jiffes to guarantee 1 jiffy timeout */ .netdev_budget_usecs = 2 * USEC_PER_SEC / HZ, + + .tstamp_prequeue = 1, }; diff --git a/net/core/sysctl_net_core.c b/net/core/sysctl_net_core.c index a9c2d798b219506da75a5d0a30d490ff4011d668..bddd07da099886f2747f2ac4ba39a482b0f4231d 100644 --- a/net/core/sysctl_net_core.c +++ b/net/core/sysctl_net_core.c @@ -499,7 +499,7 @@ static struct ctl_table net_core_table[] = { #endif { .procname = "netdev_tstamp_prequeue", - .data = &netdev_tstamp_prequeue, + .data = &net_hotdata.tstamp_prequeue, .maxlen = sizeof(int), .mode = 0644, .proc_handler = proc_dointvec From patchwork Wed Mar 6 16:00:17 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Dumazet X-Patchwork-Id: 13584252 X-Patchwork-Delegate: kuba@kernel.org Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9AF80137936 for ; Wed, 6 Mar 2024 16:00:46 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709740848; cv=none; b=ugGhunLUDbpRsacOcbPl79mkyg0Fm7detqFSSF457iwWyzDponp75Ptd0NIrpnZrdGpFOoVbtpWOlo1tJpQkl7hpM0DK/Fn+M+q6KV2rcXD5NG+ZEcKcmPbEWPjJXifOau62bDFzTW3I/+jg4I9oYk7LlPEBgQyPGBVfZX9f5XY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709740848; c=relaxed/simple; bh=CCrywqwkFo9FhkcQxA38b6+rPZ9hdCuIDojQ7g2AubM=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=azkwIg5xZlAiFNZo0McBYgQ0WizThipVpfqckBNuq+As+4uTa4G62NPUpxXpWBDTHlh18oDQnYQlR4GRLAgly3R513pX/vAAF0T2gbpRo34wJJK405iHKdHdNCko9ouUhRa+yKSBFMtbqt6BnQlJ/O4+gu/zB/nNpyfJQ5vkmJs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--edumazet.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=4OIrcfnR; arc=none smtp.client-ip=209.85.219.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--edumazet.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="4OIrcfnR" Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-dcc4563611cso10823012276.3 for ; Wed, 06 Mar 2024 08:00:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1709740845; x=1710345645; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=NcKOs0+vx6Ph3LwiTlyCoVW8OOGNMT4JjMZZ2TR+HNY=; b=4OIrcfnRyIbHTgWXcuzvJ56p7Fa8tr5YgSccmA8ARoONf6rUAkDxfObG2qE5JxE0P0 0bxl+4r8nr67BZBdQCN/gU1yZZb9zSHws7ip7VYs4BSJOjoR2UhtJ3O/PhZl0PirhfqP NsKU2k8Lms0Ug3uTvIr4ChC4DfedvugLC9z9cw03dXyTwcdHq+0nXet7n96AFmmbBtyZ DmDs8C1oPfNxohsvcLyMK7rmRUwM2n9YeSkajK+WxfBBXUyomaBwECM/hTxewaySl/fD 6MvZhrINfpMoiWpj0UJx0coPQjQzsafhoHrXPa1zqWLHrue7gP+yHk3JtrBwd2ma+jm9 fkbQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709740845; x=1710345645; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=NcKOs0+vx6Ph3LwiTlyCoVW8OOGNMT4JjMZZ2TR+HNY=; b=JgFEwDIY8q+K6ipkDxNTkT/7WczR09TjYz0RAYCCWOZ+RJqApp4iEMS5VjaF5w35eU 7PMwTjhdREoh20+vZHhI7zVFWIvZnHj5wnsKxph8kNOvFnGraRxQP9iy+5Rl34a3RRsK WMapPK3ftU386clvssQuVxLjdt8sGpk8WVAmBwGzPaLgI/+jAP/xsdQzOPQFaDdqpbxN cMgN7VzCCyOovk3LnFip6TKBVcVDEDz2iB0o64Z0TXSqvYGx3cPN9Dt/pNBI4+L/P/jR 4b8LTteFDXMXNfbJlI6uptGrpUbr/pUJeLDEFTOCkE0tmmxWlgjhXgPBB6mbWjUZ8yiw lRKA== X-Forwarded-Encrypted: i=1; AJvYcCUS1uPHCTUZ/KeQZm1+UROK5wiGU9HO2odhYIcRt+aPmPxkSI64Blq/oLlSr9mw8EaqHpqZkgln1qNCf0fAlMrGOW9G4RX4 X-Gm-Message-State: AOJu0YymWo2CouMXgA4Nf9GR/dnmqCNgKzyh7Co8yqDWCnt6iB24iIt7 HfJ/VCzH0kfiMcd2KrzqmCE2bic4tWdAkEwqNYhR5hI0wBfaIe3GLBIlRGgoEXAWhUjgfFROdc2 qWom1ncHqIw== X-Google-Smtp-Source: AGHT+IEIEvyk2Awqp0vgH+nC4Ko4ssDtG34y3yN8hwAV5n/BPNtW5IJRHVXALdOqB5uSeARECOH+tS3drJcYoQ== X-Received: from edumazet1.c.googlers.com ([fda3:e722:ac3:cc00:2b:7d90:c0a8:395a]) (user=edumazet job=sendgmr) by 2002:a05:6902:120a:b0:dc2:398d:a671 with SMTP id s10-20020a056902120a00b00dc2398da671mr3922272ybu.10.1709740845772; Wed, 06 Mar 2024 08:00:45 -0800 (PST) Date: Wed, 6 Mar 2024 16:00:17 +0000 In-Reply-To: <20240306160031.874438-1-edumazet@google.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240306160031.874438-1-edumazet@google.com> X-Mailer: git-send-email 2.44.0.278.ge034bb2e1d-goog Message-ID: <20240306160031.874438-5-edumazet@google.com> Subject: [PATCH v2 net-next 04/18] net: move ptype_all into net_hotdata From: Eric Dumazet To: "David S . Miller" , Jakub Kicinski , Paolo Abeni Cc: David Ahern , Willem de Bruijn , Soheil Hassas Yeganeh , Neal Cardwell , netdev@vger.kernel.org, eric.dumazet@gmail.com, Eric Dumazet X-Patchwork-Delegate: kuba@kernel.org ptype_all is used in rx/tx fast paths. Move it to net_hotdata for better cache locality. Signed-off-by: Eric Dumazet Acked-by: Soheil Hassas Yeganeh Reviewed-by: David Ahern --- include/linux/netdevice.h | 1 - include/net/hotdata.h | 1 + net/core/dev.c | 16 +++++++--------- net/core/hotdata.c | 1 + net/core/net-procfs.c | 7 ++++--- 5 files changed, 13 insertions(+), 13 deletions(-) diff --git a/include/linux/netdevice.h b/include/linux/netdevice.h index 15ce809e0541078bff7a48b8d7cb2cf2c1ac8a93..e2e7d5a7ef34de165cd293eb71800e1e6b450432 100644 --- a/include/linux/netdevice.h +++ b/include/linux/netdevice.h @@ -5300,7 +5300,6 @@ static inline const char *netdev_reg_state(const struct net_device *dev) #define PTYPE_HASH_SIZE (16) #define PTYPE_HASH_MASK (PTYPE_HASH_SIZE - 1) -extern struct list_head ptype_all __read_mostly; extern struct list_head ptype_base[PTYPE_HASH_SIZE] __read_mostly; extern struct net_device *blackhole_netdev; diff --git a/include/net/hotdata.h b/include/net/hotdata.h index 149e56528537d8ed3365e46d6dc96e39c73a733a..d462cb8f16bad459b439c566274c01a0f18a95d0 100644 --- a/include/net/hotdata.h +++ b/include/net/hotdata.h @@ -7,6 +7,7 @@ /* Read mostly data used in network fast paths. */ struct net_hotdata { struct list_head offload_base; + struct list_head ptype_all; int gro_normal_batch; int netdev_budget; int netdev_budget_usecs; diff --git a/net/core/dev.c b/net/core/dev.c index 53ebdb55e8b7c0a6522eb3fdbb7bdd00948eb9a5..119b7004a8e51ea6785c60e558988d369eec8935 100644 --- a/net/core/dev.c +++ b/net/core/dev.c @@ -161,7 +161,6 @@ static DEFINE_SPINLOCK(ptype_lock); struct list_head ptype_base[PTYPE_HASH_SIZE] __read_mostly; -struct list_head ptype_all __read_mostly; /* Taps */ static int netif_rx_internal(struct sk_buff *skb); static int call_netdevice_notifiers_extack(unsigned long val, @@ -540,7 +539,7 @@ static inline void netdev_set_addr_lockdep_class(struct net_device *dev) static inline struct list_head *ptype_head(const struct packet_type *pt) { if (pt->type == htons(ETH_P_ALL)) - return pt->dev ? &pt->dev->ptype_all : &ptype_all; + return pt->dev ? &pt->dev->ptype_all : &net_hotdata.ptype_all; else return pt->dev ? &pt->dev->ptype_specific : &ptype_base[ntohs(pt->type) & PTYPE_HASH_MASK]; @@ -2226,7 +2225,8 @@ static inline bool skb_loop_sk(struct packet_type *ptype, struct sk_buff *skb) */ bool dev_nit_active(struct net_device *dev) { - return !list_empty(&ptype_all) || !list_empty(&dev->ptype_all); + return !list_empty(&net_hotdata.ptype_all) || + !list_empty(&dev->ptype_all); } EXPORT_SYMBOL_GPL(dev_nit_active); @@ -2237,10 +2237,9 @@ EXPORT_SYMBOL_GPL(dev_nit_active); void dev_queue_xmit_nit(struct sk_buff *skb, struct net_device *dev) { - struct packet_type *ptype; + struct list_head *ptype_list = &net_hotdata.ptype_all; + struct packet_type *ptype, *pt_prev = NULL; struct sk_buff *skb2 = NULL; - struct packet_type *pt_prev = NULL; - struct list_head *ptype_list = &ptype_all; rcu_read_lock(); again: @@ -2286,7 +2285,7 @@ void dev_queue_xmit_nit(struct sk_buff *skb, struct net_device *dev) pt_prev = ptype; } - if (ptype_list == &ptype_all) { + if (ptype_list == &net_hotdata.ptype_all) { ptype_list = &dev->ptype_all; goto again; } @@ -5387,7 +5386,7 @@ static int __netif_receive_skb_core(struct sk_buff **pskb, bool pfmemalloc, if (pfmemalloc) goto skip_taps; - list_for_each_entry_rcu(ptype, &ptype_all, list) { + list_for_each_entry_rcu(ptype, &net_hotdata.ptype_all, list) { if (pt_prev) ret = deliver_skb(skb, pt_prev, orig_dev); pt_prev = ptype; @@ -11771,7 +11770,6 @@ static int __init net_dev_init(void) if (netdev_kobject_init()) goto out; - INIT_LIST_HEAD(&ptype_all); for (i = 0; i < PTYPE_HASH_SIZE; i++) INIT_LIST_HEAD(&ptype_base[i]); diff --git a/net/core/hotdata.c b/net/core/hotdata.c index 087c4c84987df07f11a87112a778a5cac608a654..29fcfe89fd9a697120f826dbe2fd36a1617581a1 100644 --- a/net/core/hotdata.c +++ b/net/core/hotdata.c @@ -7,6 +7,7 @@ struct net_hotdata net_hotdata __cacheline_aligned = { .offload_base = LIST_HEAD_INIT(net_hotdata.offload_base), + .ptype_all = LIST_HEAD_INIT(net_hotdata.ptype_all), .gro_normal_batch = 8, .netdev_budget = 300, diff --git a/net/core/net-procfs.c b/net/core/net-procfs.c index 2e4e96d30ee1a7a51e49587378aab47aed1290da..a97eceb84e61ec347ad132ff0f22c8ce82f12e90 100644 --- a/net/core/net-procfs.c +++ b/net/core/net-procfs.c @@ -3,6 +3,7 @@ #include #include #include +#include #include "dev.h" @@ -183,7 +184,7 @@ static void *ptype_get_idx(struct seq_file *seq, loff_t pos) } } - list_for_each_entry_rcu(pt, &ptype_all, list) { + list_for_each_entry_rcu(pt, &net_hotdata.ptype_all, list) { if (i == pos) return pt; ++i; @@ -231,13 +232,13 @@ static void *ptype_seq_next(struct seq_file *seq, void *v, loff_t *pos) } } - nxt = ptype_all.next; + nxt = net_hotdata.ptype_all.next; goto ptype_all; } if (pt->type == htons(ETH_P_ALL)) { ptype_all: - if (nxt != &ptype_all) + if (nxt != &net_hotdata.ptype_all) goto found; hash = 0; nxt = ptype_base[0].next; From patchwork Wed Mar 6 16:00:18 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Dumazet X-Patchwork-Id: 13584253 X-Patchwork-Delegate: kuba@kernel.org Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 94483137936 for ; Wed, 6 Mar 2024 16:00:49 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709740851; cv=none; b=rhPn5KxFoESv2Th53USBe0tKlBbhzny4JYlSyJJw3sFus8Xyoo/opcyBqtYU35qk3/BmodLKG5dau5le/Irsg3t0CgiTJIvnI3bBZXMJriiJeTyl7zvmG88RXM1co+A+o8ErL6j7x1Ni+X2Pkr0NMpVnApbO6LMGLJZcPR1orZM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709740851; c=relaxed/simple; bh=0a9W4HIDQVY0vwNk7wHx78HQ9ooqOP6SWSqIAUjxM5M=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=N6oXf+/qy9pl1FFBsLEhtHQQflzfF2D9IKYodFbxaBQrMFVg33FlvQbymdXQiDj9E4+XEg15UrJDbJx6DrOnK2J2sVtGKN6MUSB2yRKfp8QTHd2NCkEJ1UpE0EBxtXX3Nrdu2uvuq4AVpzAHVmYvlt1YvPiPf87y30205bnBIyU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--edumazet.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=WVfS/ITE; arc=none smtp.client-ip=209.85.219.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--edumazet.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="WVfS/ITE" Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-dcc05887ee9so9860872276.1 for ; Wed, 06 Mar 2024 08:00:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1709740848; x=1710345648; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=K3vewczgsKGiW/OBDkqgpG4XxHTBMfPPcUpL4NtIU7M=; b=WVfS/ITEJg35wmzuUKpYNE3Xvvr0admlmcI0k9FlypKRdIcTjGReeWXIbj7j3PcGt6 +rTetx9ED/cqibURz4202w9hLhBC6QumtfJ6zEF8A/jKK5/vLmzrx/zcbi19VhsZh0Sl x8Zhx9BozO/4uhd3D6B1F9O5J5L7hihN317+jCQ+pPAJSoPrtmfXxgviFp672NQdrQ+F NoTHp3dyliePUrBRBxroc1+JENiHne8NBXM9LytsMgkBFGheVkP7OphIPYC0Aghpd8xG Yb0TP/wGiq1PYj+sq/7E+v/2s4Iw+rRoHLlvkll7fbcLOVSOE4G1q7yc5vb7DmSkTJEj 9Djw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709740848; x=1710345648; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=K3vewczgsKGiW/OBDkqgpG4XxHTBMfPPcUpL4NtIU7M=; b=bDmwSauggEp68PHGyvbNIcSC3DMEn08iV06yGLFzo7nV+T+G+14+uxHA5gEELzZ+73 pLe7uljjre/a3XLc2Iw0TfUi+NA91jxxD7J2g+G4581Jg8fU4yCVCMdd7LDME88HcHK8 x66LQbW9Wjixuq68FE7tTU2eVNQNfSg7OSNTfvhBxZ8pAhlex726oTD9KEO6VzhGjogv LbRJ0k26wjAfbC/PH0LHVrzGpnrgty9fRje4fQk23uXM7gmFnjOdNpsq58jQ0KAE//HB uKVztEN56rT+qSbdjC1DbMkmb03xiWkGXUL6ZqUrNIngmwhLnIAuOmyS+GuOoznQObKp JaPw== X-Forwarded-Encrypted: i=1; AJvYcCXUDNkaA6ZKsgWAlxO1MxCo+VgYc47t6clyMyN62vzmNfTExvnC9iQzra18FXMObX4s/+iaQWG9AoLdGQBkhXLUo5+c24IL X-Gm-Message-State: AOJu0Yzy7Whsgb68PQj/Cff88qQp2osvaMAWnCDe8UEIESiqOjzRxHYA 7YNVRPU/Qa225rrtR9YvCOZvMVTzCtbIYWjuPHhONdaRcq79Tx1bQGEGLy6T7mC3Vv8wYB9k+Sr 12ZIEtKz0aA== X-Google-Smtp-Source: AGHT+IFQk0FQnc9Ze52HzuVowAOpTr0e89rNio2F2DSOHCjrjGshJwDbTfc+irbrLH++K7cewJuqIdGnZ2shNg== X-Received: from edumazet1.c.googlers.com ([fda3:e722:ac3:cc00:2b:7d90:c0a8:395a]) (user=edumazet job=sendgmr) by 2002:a05:6902:1009:b0:dbe:387d:a8ef with SMTP id w9-20020a056902100900b00dbe387da8efmr504777ybt.1.1709740847981; Wed, 06 Mar 2024 08:00:47 -0800 (PST) Date: Wed, 6 Mar 2024 16:00:18 +0000 In-Reply-To: <20240306160031.874438-1-edumazet@google.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240306160031.874438-1-edumazet@google.com> X-Mailer: git-send-email 2.44.0.278.ge034bb2e1d-goog Message-ID: <20240306160031.874438-6-edumazet@google.com> Subject: [PATCH v2 net-next 05/18] net: move netdev_max_backlog to net_hotdata From: Eric Dumazet To: "David S . Miller" , Jakub Kicinski , Paolo Abeni Cc: David Ahern , Willem de Bruijn , Soheil Hassas Yeganeh , Neal Cardwell , netdev@vger.kernel.org, eric.dumazet@gmail.com, Eric Dumazet X-Patchwork-Delegate: kuba@kernel.org netdev_max_backlog is used in rx fat path. Move it to net_hodata for better cache locality. Signed-off-by: Eric Dumazet Acked-by: Soheil Hassas Yeganeh Reviewed-by: David Ahern --- include/linux/netdevice.h | 1 - include/net/hotdata.h | 1 + net/core/dev.c | 8 +++----- net/core/gro_cells.c | 3 ++- net/core/hotdata.c | 2 ++ net/core/sysctl_net_core.c | 2 +- net/xfrm/espintcp.c | 4 +++- net/xfrm/xfrm_input.c | 3 ++- 8 files changed, 14 insertions(+), 10 deletions(-) diff --git a/include/linux/netdevice.h b/include/linux/netdevice.h index e2e7d5a7ef34de165cd293eb71800e1e6b450432..044d6f5b2ace3e2decd4296e01c8d3e200c6c7dc 100644 --- a/include/linux/netdevice.h +++ b/include/linux/netdevice.h @@ -4787,7 +4787,6 @@ void dev_fetch_sw_netstats(struct rtnl_link_stats64 *s, const struct pcpu_sw_netstats __percpu *netstats); void dev_get_tstats64(struct net_device *dev, struct rtnl_link_stats64 *s); -extern int netdev_max_backlog; extern int dev_rx_weight; extern int dev_tx_weight; diff --git a/include/net/hotdata.h b/include/net/hotdata.h index d462cb8f16bad459b439c566274c01a0f18a95d0..dc50b200a94b6b935cd79d8e0406a61209fdc68e 100644 --- a/include/net/hotdata.h +++ b/include/net/hotdata.h @@ -12,6 +12,7 @@ struct net_hotdata { int netdev_budget; int netdev_budget_usecs; int tstamp_prequeue; + int max_backlog; }; extern struct net_hotdata net_hotdata; diff --git a/net/core/dev.c b/net/core/dev.c index 119b7004a8e51ea6785c60e558988d369eec8935..1b112c4db983c2d7cd280bc8c2ebc621ea3c6145 100644 --- a/net/core/dev.c +++ b/net/core/dev.c @@ -4404,9 +4404,6 @@ EXPORT_SYMBOL(__dev_direct_xmit); * Receiver routines *************************************************************************/ -int netdev_max_backlog __read_mostly = 1000; -EXPORT_SYMBOL(netdev_max_backlog); - unsigned int sysctl_skb_defer_max __read_mostly = 64; int weight_p __read_mostly = 64; /* old backlog weight */ int dev_weight_rx_bias __read_mostly = 1; /* bias for backlog weight */ @@ -4713,7 +4710,7 @@ static bool skb_flow_limit(struct sk_buff *skb, unsigned int qlen) struct softnet_data *sd; unsigned int old_flow, new_flow; - if (qlen < (READ_ONCE(netdev_max_backlog) >> 1)) + if (qlen < (READ_ONCE(net_hotdata.max_backlog) >> 1)) return false; sd = this_cpu_ptr(&softnet_data); @@ -4761,7 +4758,8 @@ static int enqueue_to_backlog(struct sk_buff *skb, int cpu, if (!netif_running(skb->dev)) goto drop; qlen = skb_queue_len(&sd->input_pkt_queue); - if (qlen <= READ_ONCE(netdev_max_backlog) && !skb_flow_limit(skb, qlen)) { + if (qlen <= READ_ONCE(net_hotdata.max_backlog) && + !skb_flow_limit(skb, qlen)) { if (qlen) { enqueue: __skb_queue_tail(&sd->input_pkt_queue, skb); diff --git a/net/core/gro_cells.c b/net/core/gro_cells.c index ed5ec5de47f670753924bd0c72db1e3ceb9b9e7a..ff8e5b64bf6b76451a69e3eae132b593c60ee204 100644 --- a/net/core/gro_cells.c +++ b/net/core/gro_cells.c @@ -3,6 +3,7 @@ #include #include #include +#include struct gro_cell { struct sk_buff_head napi_skbs; @@ -26,7 +27,7 @@ int gro_cells_receive(struct gro_cells *gcells, struct sk_buff *skb) cell = this_cpu_ptr(gcells->cells); - if (skb_queue_len(&cell->napi_skbs) > READ_ONCE(netdev_max_backlog)) { + if (skb_queue_len(&cell->napi_skbs) > READ_ONCE(net_hotdata.max_backlog)) { drop: dev_core_stats_rx_dropped_inc(dev); kfree_skb(skb); diff --git a/net/core/hotdata.c b/net/core/hotdata.c index 29fcfe89fd9a697120f826dbe2fd36a1617581a1..35ed5a83ecc7ebda513fe4fafc596e053f0252c5 100644 --- a/net/core/hotdata.c +++ b/net/core/hotdata.c @@ -15,4 +15,6 @@ struct net_hotdata net_hotdata __cacheline_aligned = { .netdev_budget_usecs = 2 * USEC_PER_SEC / HZ, .tstamp_prequeue = 1, + .max_backlog = 1000, }; +EXPORT_SYMBOL(net_hotdata); diff --git a/net/core/sysctl_net_core.c b/net/core/sysctl_net_core.c index bddd07da099886f2747f2ac4ba39a482b0f4231d..8eaeeb289914258f90cf940e906d5c6be0cc0cd6 100644 --- a/net/core/sysctl_net_core.c +++ b/net/core/sysctl_net_core.c @@ -440,7 +440,7 @@ static struct ctl_table net_core_table[] = { }, { .procname = "netdev_max_backlog", - .data = &netdev_max_backlog, + .data = &net_hotdata.max_backlog, .maxlen = sizeof(int), .mode = 0644, .proc_handler = proc_dointvec diff --git a/net/xfrm/espintcp.c b/net/xfrm/espintcp.c index d3b3f9e720b3b6c2a4ea89df4257a564100b2c4b..fe82e2d073006e5ab1b03868c851147c0422d26d 100644 --- a/net/xfrm/espintcp.c +++ b/net/xfrm/espintcp.c @@ -10,6 +10,7 @@ #if IS_ENABLED(CONFIG_IPV6) #include #endif +#include static void handle_nonesp(struct espintcp_ctx *ctx, struct sk_buff *skb, struct sock *sk) @@ -169,7 +170,8 @@ int espintcp_queue_out(struct sock *sk, struct sk_buff *skb) { struct espintcp_ctx *ctx = espintcp_getctx(sk); - if (skb_queue_len(&ctx->out_queue) >= READ_ONCE(netdev_max_backlog)) + if (skb_queue_len(&ctx->out_queue) >= + READ_ONCE(net_hotdata.max_backlog)) return -ENOBUFS; __skb_queue_tail(&ctx->out_queue, skb); diff --git a/net/xfrm/xfrm_input.c b/net/xfrm/xfrm_input.c index bd4ce21d76d7551d8f0a4a11f4b75705a7f634c5..161f535c8b9495b01f6d9689e14c40e5c0885968 100644 --- a/net/xfrm/xfrm_input.c +++ b/net/xfrm/xfrm_input.c @@ -21,6 +21,7 @@ #include #include #include +#include #include "xfrm_inout.h" @@ -764,7 +765,7 @@ int xfrm_trans_queue_net(struct net *net, struct sk_buff *skb, trans = this_cpu_ptr(&xfrm_trans_tasklet); - if (skb_queue_len(&trans->queue) >= READ_ONCE(netdev_max_backlog)) + if (skb_queue_len(&trans->queue) >= READ_ONCE(net_hotdata.max_backlog)) return -ENOBUFS; BUILD_BUG_ON(sizeof(struct xfrm_trans_cb) > sizeof(skb->cb)); From patchwork Wed Mar 6 16:00:19 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Dumazet X-Patchwork-Id: 13584254 X-Patchwork-Delegate: kuba@kernel.org Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6D994136998 for ; Wed, 6 Mar 2024 16:00:51 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709740852; cv=none; b=WusJf6Iub+7aMMBI6bVE2/EqdvwVMwVT/xu56cAXM6d7LzxwONcEROoVuk47v9Xl4X+d+QmTofUghHCFzsuKLwQV3AHSC55Qtq24X7fvIB6seGtF8s08cRWDyiRuvSJ3dbjrOODI6I1ofrk1G1WnzuCWj20QntTxxmHsz2+TjQo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709740852; c=relaxed/simple; bh=MgQ7bW4+E3Jxaoa83pBNP1QjdJSImfMpuDEUph0mqO4=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=N3yr9OzW4418NrSiGWR4hobNxZO87cYBz2suzjE1Vm4FV/GcVS/BbSPfiTXgrsfhTbkq9La+PwMv6Y6fLa9Bo1Gsag2Pu4Q6F1FbQcsYrDGQp9pYIBywf7p23NK+rl/9K7C47fJTx6LPGEi6F8gRNnlgdqAyzJXnQCnXmciKcbo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--edumazet.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=M01EJlU4; arc=none smtp.client-ip=209.85.219.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--edumazet.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="M01EJlU4" Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-dccc49ef73eso12045239276.2 for ; Wed, 06 Mar 2024 08:00:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1709740850; x=1710345650; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=WLpVDXMV6KhDn8Nv4o048oZYVIssSOAdchwz6e7oWZ8=; b=M01EJlU4tv5SP6TpFbxdMV+1V+xpO5akHJMUeXEDc4cUnObnuKAS+kq5bhGIHB52sg XTTZaJgE0tadvpgDnOxntJ0z7be2z1uBOUz5InAsj3gerOU4FgzZp2Faq/7C23bx6llp UULP+3KM6pl8U0XJ2GqIgmPuAGnqj8mtQK5RKqeBpBxMYeAn3ulPiHU7a/1lzQW0vPpn aPtka0BPMUbwrSLSnR7cEoSXffWwl8CDrVWPwlwvHI2IYIu9tNOeLAX/vPisGJAYgF87 1BOieMZglfgEItAF7ne+PKwGAlBU97WieQtICVzPU0pr1mUdKncMM56lSERhzk5p15CP C6eQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709740850; x=1710345650; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=WLpVDXMV6KhDn8Nv4o048oZYVIssSOAdchwz6e7oWZ8=; b=FKjULk8j1yx9IF7pA7k9wIJpbmHCnm39ejJu/Pl7BY8LFwPzCGUG5IieHl14M+VUg6 sIMeE+fAnqSGp2IN7AbhMn52fsHGcyIERSSjHGCWBzypg81pmFv7jPxYKqjhCtM1/b71 Jl+j2aESw7cixqoDEkNooKXCmW0q4gEC83+HWfjdeMynWDilqX93xmfSHqIvHK0uRsCL flvGfHZrKPPggkB/4MbgWodvz3mZzXI8UY3HmnKqnRAEAzLsAWuhhDH7lwMpWFxrlW2T aY5BNm5p5Y+6JkfudwBf1SHDOFpK5DoZ3qtowiEFLp5cQhrcoQWyI0WGWIVM3YYlrqwS D39g== X-Forwarded-Encrypted: i=1; AJvYcCXkc5QE88LPMiFA6fYjMCba6HQogCach2UC0HpYJBUNCX8U1fO/+10VuF+jWK7lmjai6UAfx60SuaM2VCDD50oNNh7Q6kUk X-Gm-Message-State: AOJu0YzJR1RbOV0oAT0Ce7hxP53wyA7LCcToR0n+APm4mKZ/xLWSnoSY 6kKMjsurY8sLNqjktDR08N2U7xSP7+V8JN6AJIgdoOvC8WPOG9thZNeMZb1yA570u7zMj6K7Wlg gZEym4gd0Vg== X-Google-Smtp-Source: AGHT+IGNs/PstXwFgoaPvgLhkFIkV3YgK1l/zuuLccrJ1SFL4jnauHdDA6HNCl94MTwf7a6UIeGZ5/rWaKlyEw== X-Received: from edumazet1.c.googlers.com ([fda3:e722:ac3:cc00:2b:7d90:c0a8:395a]) (user=edumazet job=sendgmr) by 2002:a05:6902:f0d:b0:dcd:4286:4498 with SMTP id et13-20020a0569020f0d00b00dcd42864498mr531916ybb.6.1709740850487; Wed, 06 Mar 2024 08:00:50 -0800 (PST) Date: Wed, 6 Mar 2024 16:00:19 +0000 In-Reply-To: <20240306160031.874438-1-edumazet@google.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240306160031.874438-1-edumazet@google.com> X-Mailer: git-send-email 2.44.0.278.ge034bb2e1d-goog Message-ID: <20240306160031.874438-7-edumazet@google.com> Subject: [PATCH v2 net-next 06/18] net: move ip_packet_offload and ipv6_packet_offload to net_hotdata From: Eric Dumazet To: "David S . Miller" , Jakub Kicinski , Paolo Abeni Cc: David Ahern , Willem de Bruijn , Soheil Hassas Yeganeh , Neal Cardwell , netdev@vger.kernel.org, eric.dumazet@gmail.com, Eric Dumazet X-Patchwork-Delegate: kuba@kernel.org These structures are used in GRO and GSO paths. v2: ipv6_packet_offload definition depends on CONFIG_INET Signed-off-by: Eric Dumazet Acked-by: Soheil Hassas Yeganeh Reviewed-by: David Ahern --- include/net/hotdata.h | 5 +++++ net/ipv4/af_inet.c | 18 +++++++++--------- net/ipv6/ip6_offload.c | 18 +++++++++--------- 3 files changed, 23 insertions(+), 18 deletions(-) diff --git a/include/net/hotdata.h b/include/net/hotdata.h index dc50b200a94b6b935cd79d8e0406a61209fdc68e..d8ce20d3215d3cf13a23fa81619a1c52627f2913 100644 --- a/include/net/hotdata.h +++ b/include/net/hotdata.h @@ -3,9 +3,14 @@ #define _NET_HOTDATA_H #include +#include /* Read mostly data used in network fast paths. */ struct net_hotdata { +#if IS_ENABLED(CONFIG_INET) + struct packet_offload ip_packet_offload; + struct packet_offload ipv6_packet_offload; +#endif struct list_head offload_base; struct list_head ptype_all; int gro_normal_batch; diff --git a/net/ipv4/af_inet.c b/net/ipv4/af_inet.c index 5daebdcbca326aa1fc042e1e1ff1e82a18bd283d..08dda6955562ea6b89e02b8299b03ab52b342f27 100644 --- a/net/ipv4/af_inet.c +++ b/net/ipv4/af_inet.c @@ -1904,14 +1904,6 @@ static int ipv4_proc_init(void); * IP protocol layer initialiser */ -static struct packet_offload ip_packet_offload __read_mostly = { - .type = cpu_to_be16(ETH_P_IP), - .callbacks = { - .gso_segment = inet_gso_segment, - .gro_receive = inet_gro_receive, - .gro_complete = inet_gro_complete, - }, -}; static const struct net_offload ipip_offload = { .callbacks = { @@ -1938,7 +1930,15 @@ static int __init ipv4_offload_init(void) if (ipip_offload_init() < 0) pr_crit("%s: Cannot add IPIP protocol offload\n", __func__); - dev_add_offload(&ip_packet_offload); + net_hotdata.ip_packet_offload = (struct packet_offload) { + .type = cpu_to_be16(ETH_P_IP), + .callbacks = { + .gso_segment = inet_gso_segment, + .gro_receive = inet_gro_receive, + .gro_complete = inet_gro_complete, + }, + }; + dev_add_offload(&net_hotdata.ip_packet_offload); return 0; } diff --git a/net/ipv6/ip6_offload.c b/net/ipv6/ip6_offload.c index cca64c7809bee9a0360cbfab6a645d3f8d2ffea3..b41e35af69ea2835aa47d6ca01d9b109d4092462 100644 --- a/net/ipv6/ip6_offload.c +++ b/net/ipv6/ip6_offload.c @@ -419,14 +419,6 @@ static int ip4ip6_gro_complete(struct sk_buff *skb, int nhoff) return inet_gro_complete(skb, nhoff); } -static struct packet_offload ipv6_packet_offload __read_mostly = { - .type = cpu_to_be16(ETH_P_IPV6), - .callbacks = { - .gso_segment = ipv6_gso_segment, - .gro_receive = ipv6_gro_receive, - .gro_complete = ipv6_gro_complete, - }, -}; static struct sk_buff *sit_gso_segment(struct sk_buff *skb, netdev_features_t features) @@ -486,7 +478,15 @@ static int __init ipv6_offload_init(void) if (ipv6_exthdrs_offload_init() < 0) pr_crit("%s: Cannot add EXTHDRS protocol offload\n", __func__); - dev_add_offload(&ipv6_packet_offload); + net_hotdata.ipv6_packet_offload = (struct packet_offload) { + .type = cpu_to_be16(ETH_P_IPV6), + .callbacks = { + .gso_segment = ipv6_gso_segment, + .gro_receive = ipv6_gro_receive, + .gro_complete = ipv6_gro_complete, + }, + }; + dev_add_offload(&net_hotdata.ipv6_packet_offload); inet_add_offload(&sit_offload, IPPROTO_IPV6); inet6_add_offload(&ip6ip6_offload, IPPROTO_IPV6); From patchwork Wed Mar 6 16:00:20 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Dumazet X-Patchwork-Id: 13584255 X-Patchwork-Delegate: kuba@kernel.org Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 11C3213958B for ; Wed, 6 Mar 2024 16:00:52 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709740854; cv=none; b=cew00Rec6KFlSpnT7mLLzJ32B5/ka7FEOLGkO3fPcodvvvJPkIopl8+CtGxCEHQpdsyljiQ5oBwZ+mgvIXW6B4exFOIxDBM9nTKeD/507oKQNvZpFDs/1Qn9LecGdobPNQoCUeZWiyYJ+ACzR6ioeCUO/ej8hnuMBhZmvYzexcA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709740854; c=relaxed/simple; bh=w/U3P7HNxanK+riwfp1V2MV0wWbB1aJEPUSWyHW4dcs=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=jFvWKpiZd4XJl47Lau3xO5a13HPsiDM+ywTnd9AkGAtJ4HMvh2zsH6+4rZ1ErQocCSNJ+BF9kAVEsDTaTlK37t60qajWv++dvvzK/Z5tVFcAkJNpsJ4RQ5H4E4ph7nFcDrr5x6Eb5octOw8rqSuQa73gKZCt/5oU5hU24TJHjAE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--edumazet.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=mM70KzMw; arc=none smtp.client-ip=209.85.219.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--edumazet.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="mM70KzMw" Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-dc74ac7d015so8533966276.0 for ; Wed, 06 Mar 2024 08:00:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1709740852; x=1710345652; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=TTG84diJudvJTDGXkuZrR7YIFKVcaeJvr+RXcEnoJTM=; b=mM70KzMwMCc0C/+fK9Rxd4S2i0wjfUdMkmUT7gPKXq0v5COLZtGVcG9PVLFoY+ayXb MHacAaGj0PbeeFoFT9UEr8kB8f9Vd+s54cd8B8F2tKt42/5aYzfx1jDK6RhTe3I12/z4 2El+uvyDotS85BsR3O99oRV1OhN4aQRMyjDTp5hhK4iwgvUr5GNFdFdijHtx0qfRBAa7 SoNmIo32HLxU2jX7V3ZCXptFXT28Cpo2Bfb9PCm+cfJsDz9TC7tBGvGGDSBvXWOy/28V uj0BFwrexW0EqlfazyE/ZZTGI0hQSlMCul9tzuMcpfqcC8q+i2wKeRs5JNwd0zaFSr/Q 01YA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709740852; x=1710345652; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=TTG84diJudvJTDGXkuZrR7YIFKVcaeJvr+RXcEnoJTM=; b=bPvK+09oJal4dhFS6IG6N9K4FumODQPBz24ScEbB4B7eFlPWU+X1XLme5lklkmW0JS y2bu5j+QKNTjfVrWtvuFjy2uMxqo23nZMuGjmqxlmOWuWY4Fo+laMVONUmvTU6THZccp lPXRk7mf5B2n0QXogyxqsvTZyyo352LaySkOunRs59NFDCPp6uQO60c9kcj5nGPFIpjm 32jGQJ5tim6mX3HTbU3gUIyqEr63+H+LQJ+XkGbHSmyVNj0nk7cAylEMwi8QMyrq1KHY gNifGFh1R9bD8iuCFRQWJ6IhEsMarOSJBnzxkEQ1fAflPkMe0DGfyi6+Wx9U7R+beAJl oZ6w== X-Forwarded-Encrypted: i=1; AJvYcCWlegv/jHicCx5ZHWCnK5EsfmBVAenUoQbVysTW1dITdJw36Raicosia2yge9Kc6HzzHjNCNX0BbABmkxKb6+O35ku91cO9 X-Gm-Message-State: AOJu0YxuSOcVG00vX7JWCr5ZBKdTvDEu+Frh+Pl0jLmGZts0zM/lb02G lnqQuwgRyOnPjZeX/4KBnxpBa3Zdcpdb/7QeNVDFuGWGfQc0R+Dvo3qS94iY1VCZ0mbND/ZQRDX WpjiqgPZdLw== X-Google-Smtp-Source: AGHT+IH8mEjNOcCjJDuOYlyzram7GEY3P9EBfZSanCbuiID+kKXNkvDoSlkfI5DtFcp0Fuj3TpLApVjg6lnySw== X-Received: from edumazet1.c.googlers.com ([fda3:e722:ac3:cc00:2b:7d90:c0a8:395a]) (user=edumazet job=sendgmr) by 2002:a05:6902:1549:b0:dc6:e647:3fae with SMTP id r9-20020a056902154900b00dc6e6473faemr653390ybu.2.1709740852064; Wed, 06 Mar 2024 08:00:52 -0800 (PST) Date: Wed, 6 Mar 2024 16:00:20 +0000 In-Reply-To: <20240306160031.874438-1-edumazet@google.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240306160031.874438-1-edumazet@google.com> X-Mailer: git-send-email 2.44.0.278.ge034bb2e1d-goog Message-ID: <20240306160031.874438-8-edumazet@google.com> Subject: [PATCH v2 net-next 07/18] net: move tcpv4_offload and tcpv6_offload to net_hotdata From: Eric Dumazet To: "David S . Miller" , Jakub Kicinski , Paolo Abeni Cc: David Ahern , Willem de Bruijn , Soheil Hassas Yeganeh , Neal Cardwell , netdev@vger.kernel.org, eric.dumazet@gmail.com, Eric Dumazet X-Patchwork-Delegate: kuba@kernel.org These are used in TCP fast paths. Move them into net_hotdata for better cache locality. v2: tcpv6_offload definition depends on CONFIG_INET Signed-off-by: Eric Dumazet Acked-by: Soheil Hassas Yeganeh Reviewed-by: David Ahern --- include/net/hotdata.h | 3 +++ net/ipv4/tcp_offload.c | 17 ++++++++--------- net/ipv6/tcpv6_offload.c | 16 ++++++++-------- 3 files changed, 19 insertions(+), 17 deletions(-) diff --git a/include/net/hotdata.h b/include/net/hotdata.h index d8ce20d3215d3cf13a23fa81619a1c52627f2913..d86d02f156fc350d508531d23b4fe06b2a27aca2 100644 --- a/include/net/hotdata.h +++ b/include/net/hotdata.h @@ -4,12 +4,15 @@ #include #include +#include /* Read mostly data used in network fast paths. */ struct net_hotdata { #if IS_ENABLED(CONFIG_INET) struct packet_offload ip_packet_offload; + struct net_offload tcpv4_offload; struct packet_offload ipv6_packet_offload; + struct net_offload tcpv6_offload; #endif struct list_head offload_base; struct list_head ptype_all; diff --git a/net/ipv4/tcp_offload.c b/net/ipv4/tcp_offload.c index b955ab3b236d965a38054efa004fe12f03074c70..ebe4722bb0204433936e69724879779141288789 100644 --- a/net/ipv4/tcp_offload.c +++ b/net/ipv4/tcp_offload.c @@ -345,15 +345,14 @@ INDIRECT_CALLABLE_SCOPE int tcp4_gro_complete(struct sk_buff *skb, int thoff) return 0; } -static const struct net_offload tcpv4_offload = { - .callbacks = { - .gso_segment = tcp4_gso_segment, - .gro_receive = tcp4_gro_receive, - .gro_complete = tcp4_gro_complete, - }, -}; - int __init tcpv4_offload_init(void) { - return inet_add_offload(&tcpv4_offload, IPPROTO_TCP); + net_hotdata.tcpv4_offload = (struct net_offload) { + .callbacks = { + .gso_segment = tcp4_gso_segment, + .gro_receive = tcp4_gro_receive, + .gro_complete = tcp4_gro_complete, + }, + }; + return inet_add_offload(&net_hotdata.tcpv4_offload, IPPROTO_TCP); } diff --git a/net/ipv6/tcpv6_offload.c b/net/ipv6/tcpv6_offload.c index bf0c957e4b5eaaabc0ac3a7e55c7de6608cec156..4b07d1e6c952957419924c83c5d3ac0e9d0b2565 100644 --- a/net/ipv6/tcpv6_offload.c +++ b/net/ipv6/tcpv6_offload.c @@ -66,15 +66,15 @@ static struct sk_buff *tcp6_gso_segment(struct sk_buff *skb, return tcp_gso_segment(skb, features); } -static const struct net_offload tcpv6_offload = { - .callbacks = { - .gso_segment = tcp6_gso_segment, - .gro_receive = tcp6_gro_receive, - .gro_complete = tcp6_gro_complete, - }, -}; int __init tcpv6_offload_init(void) { - return inet6_add_offload(&tcpv6_offload, IPPROTO_TCP); + net_hotdata.tcpv6_offload = (struct net_offload) { + .callbacks = { + .gso_segment = tcp6_gso_segment, + .gro_receive = tcp6_gro_receive, + .gro_complete = tcp6_gro_complete, + }, + }; + return inet6_add_offload(&net_hotdata.tcpv6_offload, IPPROTO_TCP); } From patchwork Wed Mar 6 16:00:21 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Dumazet X-Patchwork-Id: 13584256 X-Patchwork-Delegate: kuba@kernel.org Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A5E4813AA5E for ; Wed, 6 Mar 2024 16:00:54 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709740856; cv=none; b=dDPC7+vtoRuE0ftigL1Pg3S1RU1h9pMhWkxx8F2ANd9Tdbz11NdGNMd3RMpU3a4w13BSxw0V6NYtVTmHCzwbzZEnU7S4j5hLOJoRGvfmAzV4vWWsX3Womz8G4nNtHXjhqeGHY5zqoWFeUaQatO6LgCZBkQEwuxRQBG01R0+Pe7M= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709740856; c=relaxed/simple; bh=UpOECQ3EsXx1llJf2/AC24w2jXjh4gJW+k6maZ5Axp4=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=CvdeFL4DuuAwgBdgTfXerYuvCA/3W+ac3mit4II1O85Z1Tpg3x3liZNq8hCadLILHIPiYlexOnjT8M3gJzSkhteKExcZfEz+m9Puz7DakkfLR7xJEOCLm5bKm6erpwhHBiZdsTwkncBeuplOExsBOtUCRlTOu0ZbsGD7/OHx1ZM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--edumazet.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=vhZl1x3t; arc=none smtp.client-ip=209.85.219.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--edumazet.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="vhZl1x3t" Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-dbe9e13775aso1878317276.1 for ; Wed, 06 Mar 2024 08:00:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1709740853; x=1710345653; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=Bzq8LB/+DmQPB8YTErAS3EzxbMbhKqJ8XJlWs7RBP3w=; b=vhZl1x3t/ISO+qQPm1JrsMaLh+nEzu+pVXm1F9Q8FtHdjwiGS21aVnzlZb7ueWU9pJ fjejxMZlQ9wyYkn5dOBtFjPq29C/sacQpaNPbqA7xYHx5pNl3tvX7SN6rpUkinTvqgA8 PHsjP5S0WtR/d8B+vczWxrp7+9DoF1WopBrfSpMkGsvoae7sTMu/5TTVPI9ocW3lM2Tz Rjvbvr1UzCtR0mf7jLAQRfTK98x/0C9BgDJWG1hiKwfGgnYZvwSop23z4sQnXBqcq/5v TSu4EZQhFsBxtqyl/SxIZ8RwDBTEMyXTUUVFhYbPna90MzcK8Vaso+752cFAcM47114B QdpA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709740853; x=1710345653; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Bzq8LB/+DmQPB8YTErAS3EzxbMbhKqJ8XJlWs7RBP3w=; b=qDrZqOzVc6PqvDY29RSZD5lmij5oaBJ8dY0WOub7luWUlTtrAHQ4cb0Ki00LWaYutr pu47Qz3DYI0F+OPHXnwggBNTM/tEGCRv4uKGKzeAIuPVHTlcbXnAyZMNu4Zc+XuIyYlp XCl2QRFVtobEH38im4CemMscpayTrnmIM7zk4r6b8FoIskPFp1mzpYlFAbFtc+xcMxHf ByUWoj+Dda1K679rQcNOdWaPNP0UwEeY8stx+M19I85HjBt4p3f8flhJ95nIK+zS+fC5 rMd7C2KLUoYMlpljPxZjWLaI2i2TAC8SI2+uxNzvXth/Sgfskr5XDh+u9mNlS991v1yz 3OvQ== X-Forwarded-Encrypted: i=1; AJvYcCUKKP8niJUVmA123l9/gg0xqGJwP5zHExbmG67AtanhEQEuh9cXCiRKkj46gZnE1TXZ2z//HwU1sMa0ExADH+yuAzYRtufQ X-Gm-Message-State: AOJu0YzvJFPTSdE9qZCtbE/2LjlJtAQiFwI8gG7KWkFNOR+ouDmEm5+N QcNzkhDrew7nvgtaTWXxltN/wsTz/u7k8LwabySJKkP4cwLVDCp0BN5l7uuLlt/ECqHnxWbx/KI AXMZVVMh2dg== X-Google-Smtp-Source: AGHT+IGLXIFvoPvNLsXQqZoE+I9VDNHceFpYv8P+8C/18+bmievlR+3wlusQp78rMB6lL+i/XpbTSDsFgLxmMQ== X-Received: from edumazet1.c.googlers.com ([fda3:e722:ac3:cc00:2b:7d90:c0a8:395a]) (user=edumazet job=sendgmr) by 2002:a25:df54:0:b0:dc6:53c3:bcbd with SMTP id w81-20020a25df54000000b00dc653c3bcbdmr3793723ybg.7.1709740853720; Wed, 06 Mar 2024 08:00:53 -0800 (PST) Date: Wed, 6 Mar 2024 16:00:21 +0000 In-Reply-To: <20240306160031.874438-1-edumazet@google.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240306160031.874438-1-edumazet@google.com> X-Mailer: git-send-email 2.44.0.278.ge034bb2e1d-goog Message-ID: <20240306160031.874438-9-edumazet@google.com> Subject: [PATCH v2 net-next 08/18] net: move dev_tx_weight to net_hotdata From: Eric Dumazet To: "David S . Miller" , Jakub Kicinski , Paolo Abeni Cc: David Ahern , Willem de Bruijn , Soheil Hassas Yeganeh , Neal Cardwell , netdev@vger.kernel.org, eric.dumazet@gmail.com, Eric Dumazet X-Patchwork-Delegate: kuba@kernel.org dev_tx_weight is used in tx fast path. Move it to net_hotdata for better cache locality. Signed-off-by: Eric Dumazet Acked-by: Soheil Hassas Yeganeh Reviewed-by: David Ahern --- include/linux/netdevice.h | 1 - include/net/hotdata.h | 1 + net/core/dev.c | 1 - net/core/hotdata.c | 1 + net/core/sysctl_net_core.c | 2 +- net/sched/sch_generic.c | 3 ++- 6 files changed, 5 insertions(+), 4 deletions(-) diff --git a/include/linux/netdevice.h b/include/linux/netdevice.h index 044d6f5b2ace3e2decd4296e01c8d3e200c6c7dc..c2a735edc44be95fbd4bbd1e234d883582bfde10 100644 --- a/include/linux/netdevice.h +++ b/include/linux/netdevice.h @@ -4788,7 +4788,6 @@ void dev_fetch_sw_netstats(struct rtnl_link_stats64 *s, void dev_get_tstats64(struct net_device *dev, struct rtnl_link_stats64 *s); extern int dev_rx_weight; -extern int dev_tx_weight; enum { NESTED_SYNC_IMM_BIT, diff --git a/include/net/hotdata.h b/include/net/hotdata.h index d86d02f156fc350d508531d23b4fe06b2a27aca2..ffea9cc263e50e4eb69f4f60e2f10c256d9b633f 100644 --- a/include/net/hotdata.h +++ b/include/net/hotdata.h @@ -21,6 +21,7 @@ struct net_hotdata { int netdev_budget_usecs; int tstamp_prequeue; int max_backlog; + int dev_tx_weight; }; extern struct net_hotdata net_hotdata; diff --git a/net/core/dev.c b/net/core/dev.c index 1b112c4db983c2d7cd280bc8c2ebc621ea3c6145..3f8d451f42fb9ba621f1ea19e784c7f0be0bf2d8 100644 --- a/net/core/dev.c +++ b/net/core/dev.c @@ -4409,7 +4409,6 @@ int weight_p __read_mostly = 64; /* old backlog weight */ int dev_weight_rx_bias __read_mostly = 1; /* bias for backlog weight */ int dev_weight_tx_bias __read_mostly = 1; /* bias for output_queue quota */ int dev_rx_weight __read_mostly = 64; -int dev_tx_weight __read_mostly = 64; /* Called with irq disabled */ static inline void ____napi_schedule(struct softnet_data *sd, diff --git a/net/core/hotdata.c b/net/core/hotdata.c index 35ed5a83ecc7ebda513fe4fafc596e053f0252c5..ec8c3b48e8fea57491c5870055cffb44c779db44 100644 --- a/net/core/hotdata.c +++ b/net/core/hotdata.c @@ -16,5 +16,6 @@ struct net_hotdata net_hotdata __cacheline_aligned = { .tstamp_prequeue = 1, .max_backlog = 1000, + .dev_tx_weight = 64, }; EXPORT_SYMBOL(net_hotdata); diff --git a/net/core/sysctl_net_core.c b/net/core/sysctl_net_core.c index 8eaeeb289914258f90cf940e906d5c6be0cc0cd6..a30016a8660e09db89b3153e4103c185a800a2ef 100644 --- a/net/core/sysctl_net_core.c +++ b/net/core/sysctl_net_core.c @@ -302,7 +302,7 @@ static int proc_do_dev_weight(struct ctl_table *table, int write, if (!ret && write) { weight = READ_ONCE(weight_p); WRITE_ONCE(dev_rx_weight, weight * dev_weight_rx_bias); - WRITE_ONCE(dev_tx_weight, weight * dev_weight_tx_bias); + WRITE_ONCE(net_hotdata.dev_tx_weight, weight * dev_weight_tx_bias); } mutex_unlock(&dev_weight_mutex); diff --git a/net/sched/sch_generic.c b/net/sched/sch_generic.c index 9b3e9262040b6ef6516752c558c8997bf4054123..ff5336493777507242320d7e9214c637663f0734 100644 --- a/net/sched/sch_generic.c +++ b/net/sched/sch_generic.c @@ -27,6 +27,7 @@ #include #include #include +#include #include #include #include @@ -409,7 +410,7 @@ static inline bool qdisc_restart(struct Qdisc *q, int *packets) void __qdisc_run(struct Qdisc *q) { - int quota = READ_ONCE(dev_tx_weight); + int quota = READ_ONCE(net_hotdata.dev_tx_weight); int packets; while (qdisc_restart(q, &packets)) { From patchwork Wed Mar 6 16:00:22 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Dumazet X-Patchwork-Id: 13584257 X-Patchwork-Delegate: kuba@kernel.org Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6774D13BAD0 for ; Wed, 6 Mar 2024 16:00:56 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709740857; cv=none; b=TqGWlvp4vP9jt0gYduy1tfb7T245svF+30ldoAVQPxxHjfjAtq8osyi1KbWg0P0dXihXvoDDn11qGtwyz0FEj6xZKvOj8OCIvLItSt6WQulJoJb5hd6kkq7btc43X38D5jkulUeTgyQwTv4+bTpKh5VZM/sttLj+syTa3Wu8lMg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709740857; c=relaxed/simple; bh=KXKN08tlstmfKE8ZXAi8jqIBurtQLBC6PZjviSAZzAU=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=cevaUkYwsnueAdlcJVw3DqrdmczKslkIRbj4wkwezG/TJBCNPYAIFvNDU/MldT6LUJvY9dx7NjNEInN5xcXyuN6q+X2f/SGjcPVGdkFRRytHiMAClvRRd/SCwDO5QayfAJSBGl+nEO3VCvP8DkGtTpx9hLKpOQhB1iTPRXkrW20= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--edumazet.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=Pv5Vbu1l; arc=none smtp.client-ip=209.85.128.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--edumazet.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="Pv5Vbu1l" Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-609c800881fso36294117b3.0 for ; Wed, 06 Mar 2024 08:00:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1709740855; x=1710345655; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=Caqqa5nak6NSJOsavnSGkyHAkf1ZcxluNQW5G1xrl1Y=; b=Pv5Vbu1lPvSckOZvpkeOcYhK0Cs3uGS17UR3HvCEQ3axJVKeAmruUCGGIjTt3A4I8v 9XQ79465GSCyRp/PwdzQO+R8u8L6DWtuZkAHC1DMqEUomwob9Wc8SmJxP5O5CwUDgMNc CJ50MZ0KtEXObrnqb4kPIl+KljRIGT1hnp8GTmy64hgkntTIVmKrDPNkIWcPSe8pV9AB EsoTZJ/LlKSzAbsG56tHaAOJY1pMRoyO/hV08oQnTLzOYxWtVifyJFvBPP4TTMEIanNp mx8Hg0R+wIKcqjF7Aq4V5yVNLLbWvkmrHNHFnzr0R/k4doAG33OlO5LHfmicjOK6tSuN i23g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709740855; x=1710345655; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Caqqa5nak6NSJOsavnSGkyHAkf1ZcxluNQW5G1xrl1Y=; b=YcDjGhssku2Sk4u4d6KszRPqhgn+zfnY7DtVEKsCagtel5CeuKHbPuGP89SFbHSjbv EoBrRU+wEdmZdUM8En2/4C5/ZraNJjZ7js+4WqKCi0Q6QXZ9/Ap1N+xymZvDv0+32Gk0 uiF03t2Dt962gHpFPxW1qUjUlvgfmiV4ZfzBrzcUO51hT9UUhxcmaPI7eXJrrGPdvnt0 va6hzFN7q0kj6JL3eAudWxu3irnN9mbbcruAAteLi10iiwdIBN2tQPXMswAPDMb0mGA1 ZpFnLsJz+SbdcbES2c+8JmXg3dDONcnv0F3+bFj9ZWPu7s/5UrFYDUWcD67K+8QYKskS EjPg== X-Forwarded-Encrypted: i=1; AJvYcCVxBU0TylT+sAKEw5gs/53K96O/UonwFpKH8Snw1aQSJmj1vzDPxBJtBSb6jZ+BoqpB1JHTslnRJ8AEVmOuW95C5gLOH1uW X-Gm-Message-State: AOJu0Yy+FFnw2OUO2XroIfgbXZ3dxYb/8FqEpJo355yJY1Z9nz8J/GYc ymBXOXNX3/NTzQOItooySg8nMl8QjLvj4pybiLW3Y0sXaTnm2v1KuiT9R1MX1AshOrn3s79kKYL azRWJKYT3tw== X-Google-Smtp-Source: AGHT+IHL+ocLTe8ycY2hDxbZOo82Ej2Pn4w1Z0YW1EvqVGj5s5PO3Kos+6aT8XaSCfYxLc5EpOmdGXLxPZrE8A== X-Received: from edumazet1.c.googlers.com ([fda3:e722:ac3:cc00:2b:7d90:c0a8:395a]) (user=edumazet job=sendgmr) by 2002:a81:7945:0:b0:609:22ea:f95e with SMTP id u66-20020a817945000000b0060922eaf95emr4147243ywc.4.1709740855500; Wed, 06 Mar 2024 08:00:55 -0800 (PST) Date: Wed, 6 Mar 2024 16:00:22 +0000 In-Reply-To: <20240306160031.874438-1-edumazet@google.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240306160031.874438-1-edumazet@google.com> X-Mailer: git-send-email 2.44.0.278.ge034bb2e1d-goog Message-ID: <20240306160031.874438-10-edumazet@google.com> Subject: [PATCH v2 net-next 09/18] net: move dev_rx_weight to net_hotdata From: Eric Dumazet To: "David S . Miller" , Jakub Kicinski , Paolo Abeni Cc: David Ahern , Willem de Bruijn , Soheil Hassas Yeganeh , Neal Cardwell , netdev@vger.kernel.org, eric.dumazet@gmail.com, Eric Dumazet X-Patchwork-Delegate: kuba@kernel.org dev_rx_weight is read from process_backlog(). Move it to net_hotdata for better cache locality. Signed-off-by: Eric Dumazet Acked-by: Soheil Hassas Yeganeh Reviewed-by: David Ahern --- include/linux/netdevice.h | 2 -- include/net/hotdata.h | 1 + net/core/dev.c | 3 +-- net/core/hotdata.c | 1 + net/core/sysctl_net_core.c | 2 +- 5 files changed, 4 insertions(+), 5 deletions(-) diff --git a/include/linux/netdevice.h b/include/linux/netdevice.h index c2a735edc44be95fbd4bbd1e234d883582bfde10..d3d4d1052ecb0bc704e9bb40fffbea85cfb6ab03 100644 --- a/include/linux/netdevice.h +++ b/include/linux/netdevice.h @@ -4787,8 +4787,6 @@ void dev_fetch_sw_netstats(struct rtnl_link_stats64 *s, const struct pcpu_sw_netstats __percpu *netstats); void dev_get_tstats64(struct net_device *dev, struct rtnl_link_stats64 *s); -extern int dev_rx_weight; - enum { NESTED_SYNC_IMM_BIT, NESTED_SYNC_TODO_BIT, diff --git a/include/net/hotdata.h b/include/net/hotdata.h index ffea9cc263e50e4eb69f4f60e2f10c256d9b633f..e6595ed2c3be527fb90959984aca78349147f2eb 100644 --- a/include/net/hotdata.h +++ b/include/net/hotdata.h @@ -22,6 +22,7 @@ struct net_hotdata { int tstamp_prequeue; int max_backlog; int dev_tx_weight; + int dev_rx_weight; }; extern struct net_hotdata net_hotdata; diff --git a/net/core/dev.c b/net/core/dev.c index 3f8d451f42fb9ba621f1ea19e784c7f0be0bf2d8..26676dbd6c8594371a4d30e0dd95d72342e226ca 100644 --- a/net/core/dev.c +++ b/net/core/dev.c @@ -4408,7 +4408,6 @@ unsigned int sysctl_skb_defer_max __read_mostly = 64; int weight_p __read_mostly = 64; /* old backlog weight */ int dev_weight_rx_bias __read_mostly = 1; /* bias for backlog weight */ int dev_weight_tx_bias __read_mostly = 1; /* bias for output_queue quota */ -int dev_rx_weight __read_mostly = 64; /* Called with irq disabled */ static inline void ____napi_schedule(struct softnet_data *sd, @@ -5978,7 +5977,7 @@ static int process_backlog(struct napi_struct *napi, int quota) net_rps_action_and_irq_enable(sd); } - napi->weight = READ_ONCE(dev_rx_weight); + napi->weight = READ_ONCE(net_hotdata.dev_rx_weight); while (again) { struct sk_buff *skb; diff --git a/net/core/hotdata.c b/net/core/hotdata.c index ec8c3b48e8fea57491c5870055cffb44c779db44..c8a7a451c18a383d091e413a510d84d163473f2f 100644 --- a/net/core/hotdata.c +++ b/net/core/hotdata.c @@ -17,5 +17,6 @@ struct net_hotdata net_hotdata __cacheline_aligned = { .tstamp_prequeue = 1, .max_backlog = 1000, .dev_tx_weight = 64, + .dev_rx_weight = 64, }; EXPORT_SYMBOL(net_hotdata); diff --git a/net/core/sysctl_net_core.c b/net/core/sysctl_net_core.c index a30016a8660e09db89b3153e4103c185a800a2ef..8a4c698dad9c97636ca9cebfad925d4220e98f2a 100644 --- a/net/core/sysctl_net_core.c +++ b/net/core/sysctl_net_core.c @@ -301,7 +301,7 @@ static int proc_do_dev_weight(struct ctl_table *table, int write, ret = proc_dointvec(table, write, buffer, lenp, ppos); if (!ret && write) { weight = READ_ONCE(weight_p); - WRITE_ONCE(dev_rx_weight, weight * dev_weight_rx_bias); + WRITE_ONCE(net_hotdata.dev_rx_weight, weight * dev_weight_rx_bias); WRITE_ONCE(net_hotdata.dev_tx_weight, weight * dev_weight_tx_bias); } mutex_unlock(&dev_weight_mutex); From patchwork Wed Mar 6 16:00:23 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Dumazet X-Patchwork-Id: 13584258 X-Patchwork-Delegate: kuba@kernel.org Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D720113AA5E for ; Wed, 6 Mar 2024 16:00:57 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709740859; cv=none; b=J3rODU3Jm44okoFuj7ioyNx4irQUvhKMEjdwVCYZXmsX5fkkcJu4bEouKPdZYc8UJx46ozfPiSBCpRuYGrpBUfy4kQHXMCtN5k+60hPySqv7J+9g4vz4CQs1v/K9N7XOf8P5FLYOeA/SoV1gp3AAARphhUAizjBAwTU4KOOYozM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709740859; c=relaxed/simple; bh=ZT9ucg+z3Pm2yuSdNcBQYsor7Yn5tf9Ef9uiRrCkM5E=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=Ewm8893KK5TcynboXC2m41WnLJi7Ecy8+Vwygmb3J5I3N7OivveRTzALVUVex2Sgqiotlcmkhx0cY5xcqFL8GMKDwYSvb7UvlLGtKpfW6EFMhoZGZ/mE6iI4tgcZ/smkCO9yZiYdWEAYIw5e/SrkPCNsRVnoXtDaPyVtp+GEbhQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--edumazet.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=kDe+gHNT; arc=none smtp.client-ip=209.85.219.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--edumazet.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="kDe+gHNT" Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-dc6ceade361so11693460276.0 for ; Wed, 06 Mar 2024 08:00:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1709740857; x=1710345657; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=la3gaDwD6szAZz3or1Lh83rg/7bNzc7493RHTfseZ/g=; b=kDe+gHNTYt3evKeekkg5dr3TBzeQ1uRmb5SIq3p3jyShghW96MWrQwylmPAOrZNcv6 2LSY6bCFLgwY62SEn4TARJLd959bSUoS+jeqLvDf1vwGuOBInAPk/uzdV3pDmFsIKZj+ /qSOR5KaG9d1u7rsmLalvCYGA4ugbvJk7/pQurr0UegXDsebHXT4FgNOVc23ntZ42DL1 KO9SHjZUp/NWpVKXK3OupgOgi9VqDXqg7AHDVSMTR/F4oa/HOraKIMeGbf41PHR0MT2U ei4f7LoWcdn5jLfssd9O343X5foKEeaXPdakZOMIrauQYWQRzpjhiBzHHc+4/nYFPZBN 79cw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709740857; x=1710345657; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=la3gaDwD6szAZz3or1Lh83rg/7bNzc7493RHTfseZ/g=; b=YpnVnTyafhP40iDE3tYFamoaPXSQjV1HhOnMCUeOrIC6/TCsYmwbcddWCMKksQMWQL 8+M0eXP0jyWP7IPM+0AyDDqOKqYbYFjEdXmlda6//r+hRGAM09U9kYOUuNzLqk/+7UGX W6jsgsj6shlK3bO6HcJ4V3afg3bCRp2mHAxN8dQ0yOzg5tllTRcrNjgeBSRv0sJGPmE0 HiZBPhNRP1BOvw20EtCjc2tPA3CkS1pFfDwPrniJ3FrZn8n+ra2wGSaGianQKydvRP5N iasHVZJlcXqgrH6cdEcur9ot8T+ZDYDDKC+MuGHJyvB3YGAQGLwBvGemtvabCnLvTfrW DKNw== X-Forwarded-Encrypted: i=1; AJvYcCXVaZKu+BAfL7gdkOLcmj8kSH5IYjfhBTkPObBDmj0zPyZDGg5/qdPeG5bcvUUd1iVHPdfA659gwbazV7q8M907/p+umdPP X-Gm-Message-State: AOJu0Ywa7sTUugOzMPPX03hAnWYFq2vxIPud4z6WXs59baO1SELQRVYv jLR+hx55ysrISqrIHKvItCT4/zyeySzG+C6WpWthJj7M/zXoCMCbIoHaB6QRVRzxGlPbZG5ssqf LpAomCVq3nw== X-Google-Smtp-Source: AGHT+IH5r/6sfUZH0CHXmIGs39/gtXUAisYbK72a0bNlu21SgY6lEtkDhh9jcO7gmHeilOtKY2RdvJMMLj6/Hw== X-Received: from edumazet1.c.googlers.com ([fda3:e722:ac3:cc00:2b:7d90:c0a8:395a]) (user=edumazet job=sendgmr) by 2002:a25:ce82:0:b0:dc8:5e26:f4d7 with SMTP id x124-20020a25ce82000000b00dc85e26f4d7mr3912226ybe.13.1709740857140; Wed, 06 Mar 2024 08:00:57 -0800 (PST) Date: Wed, 6 Mar 2024 16:00:23 +0000 In-Reply-To: <20240306160031.874438-1-edumazet@google.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240306160031.874438-1-edumazet@google.com> X-Mailer: git-send-email 2.44.0.278.ge034bb2e1d-goog Message-ID: <20240306160031.874438-11-edumazet@google.com> Subject: [PATCH v2 net-next 10/18] net: move skbuff_cache(s) to net_hotdata From: Eric Dumazet To: "David S . Miller" , Jakub Kicinski , Paolo Abeni Cc: David Ahern , Willem de Bruijn , Soheil Hassas Yeganeh , Neal Cardwell , netdev@vger.kernel.org, eric.dumazet@gmail.com, Eric Dumazet X-Patchwork-Delegate: kuba@kernel.org skbuff_cache, skbuff_fclone_cache and skb_small_head_cache are used in rx/tx fast paths. Move them to net_hotdata for better cache locality. Signed-off-by: Eric Dumazet Acked-by: Soheil Hassas Yeganeh Reviewed-by: David Ahern --- include/linux/skbuff.h | 1 - include/net/hotdata.h | 3 +++ kernel/bpf/cpumap.c | 4 +++- net/bpf/test_run.c | 4 +++- net/core/skbuff.c | 44 +++++++++++++++++++----------------------- net/core/xdp.c | 5 +++-- 6 files changed, 32 insertions(+), 29 deletions(-) diff --git a/include/linux/skbuff.h b/include/linux/skbuff.h index 3013355b63f5f29acfcb0331bfd1c3308aba034d..d0508f90bed50ca66850b29383033e11985dad34 100644 --- a/include/linux/skbuff.h +++ b/include/linux/skbuff.h @@ -1271,7 +1271,6 @@ static inline void consume_skb(struct sk_buff *skb) void __consume_stateless_skb(struct sk_buff *skb); void __kfree_skb(struct sk_buff *skb); -extern struct kmem_cache *skbuff_cache; void kfree_skb_partial(struct sk_buff *skb, bool head_stolen); bool skb_try_coalesce(struct sk_buff *to, struct sk_buff *from, diff --git a/include/net/hotdata.h b/include/net/hotdata.h index e6595ed2c3be527fb90959984aca78349147f2eb..a8f7e5e826fb749965573a5bcf13825a973c16ae 100644 --- a/include/net/hotdata.h +++ b/include/net/hotdata.h @@ -16,6 +16,9 @@ struct net_hotdata { #endif struct list_head offload_base; struct list_head ptype_all; + struct kmem_cache *skbuff_cache; + struct kmem_cache *skbuff_fclone_cache; + struct kmem_cache *skb_small_head_cache; int gro_normal_batch; int netdev_budget; int netdev_budget_usecs; diff --git a/kernel/bpf/cpumap.c b/kernel/bpf/cpumap.c index 8a0bb80fe48a344964e4029fec5e895ee512babf..33f4246cdf0de8232bfbba6a4300f158db3ca9d5 100644 --- a/kernel/bpf/cpumap.c +++ b/kernel/bpf/cpumap.c @@ -24,6 +24,7 @@ #include #include #include +#include #include #include @@ -326,7 +327,8 @@ static int cpu_map_kthread_run(void *data) /* Support running another XDP prog on this CPU */ nframes = cpu_map_bpf_prog_run(rcpu, frames, xdp_n, &stats, &list); if (nframes) { - m = kmem_cache_alloc_bulk(skbuff_cache, gfp, nframes, skbs); + m = kmem_cache_alloc_bulk(net_hotdata.skbuff_cache, + gfp, nframes, skbs); if (unlikely(m == 0)) { for (i = 0; i < nframes; i++) skbs[i] = NULL; /* effect: xdp_return_frame */ diff --git a/net/bpf/test_run.c b/net/bpf/test_run.c index 5535f9adc6589d028138026ebff24286098dc46d..61efeadaff8db0529a62d074f441e2a7c35eaa9e 100644 --- a/net/bpf/test_run.c +++ b/net/bpf/test_run.c @@ -12,6 +12,7 @@ #include #include #include +#include #include #include #include @@ -254,7 +255,8 @@ static int xdp_recv_frames(struct xdp_frame **frames, int nframes, int i, n; LIST_HEAD(list); - n = kmem_cache_alloc_bulk(skbuff_cache, gfp, nframes, (void **)skbs); + n = kmem_cache_alloc_bulk(net_hotdata.skbuff_cache, gfp, nframes, + (void **)skbs); if (unlikely(n == 0)) { for (i = 0; i < nframes; i++) xdp_return_frame(frames[i]); diff --git a/net/core/skbuff.c b/net/core/skbuff.c index 43d7fc150acc9263760162c3f5778fa0a646bcc4..766219011aeaf5782df9d624696d273ef6c1577c 100644 --- a/net/core/skbuff.c +++ b/net/core/skbuff.c @@ -69,6 +69,7 @@ #include #include #include +#include #include #include #include @@ -88,15 +89,10 @@ #include "dev.h" #include "sock_destructor.h" -struct kmem_cache *skbuff_cache __ro_after_init; -static struct kmem_cache *skbuff_fclone_cache __ro_after_init; #ifdef CONFIG_SKB_EXTENSIONS static struct kmem_cache *skbuff_ext_cache __ro_after_init; #endif - -static struct kmem_cache *skb_small_head_cache __ro_after_init; - #define SKB_SMALL_HEAD_SIZE SKB_HEAD_ALIGN(MAX_TCP_HEADER) /* We want SKB_SMALL_HEAD_CACHE_SIZE to not be a power of two. @@ -349,7 +345,7 @@ static struct sk_buff *napi_skb_cache_get(void) struct sk_buff *skb; if (unlikely(!nc->skb_count)) { - nc->skb_count = kmem_cache_alloc_bulk(skbuff_cache, + nc->skb_count = kmem_cache_alloc_bulk(net_hotdata.skbuff_cache, GFP_ATOMIC, NAPI_SKB_CACHE_BULK, nc->skb_cache); @@ -358,7 +354,7 @@ static struct sk_buff *napi_skb_cache_get(void) } skb = nc->skb_cache[--nc->skb_count]; - kasan_mempool_unpoison_object(skb, kmem_cache_size(skbuff_cache)); + kasan_mempool_unpoison_object(skb, kmem_cache_size(net_hotdata.skbuff_cache)); return skb; } @@ -416,7 +412,7 @@ struct sk_buff *slab_build_skb(void *data) struct sk_buff *skb; unsigned int size; - skb = kmem_cache_alloc(skbuff_cache, GFP_ATOMIC); + skb = kmem_cache_alloc(net_hotdata.skbuff_cache, GFP_ATOMIC); if (unlikely(!skb)) return NULL; @@ -467,7 +463,7 @@ struct sk_buff *__build_skb(void *data, unsigned int frag_size) { struct sk_buff *skb; - skb = kmem_cache_alloc(skbuff_cache, GFP_ATOMIC); + skb = kmem_cache_alloc(net_hotdata.skbuff_cache, GFP_ATOMIC); if (unlikely(!skb)) return NULL; @@ -578,7 +574,7 @@ static void *kmalloc_reserve(unsigned int *size, gfp_t flags, int node, obj_size = SKB_HEAD_ALIGN(*size); if (obj_size <= SKB_SMALL_HEAD_CACHE_SIZE && !(flags & KMALLOC_NOT_NORMAL_BITS)) { - obj = kmem_cache_alloc_node(skb_small_head_cache, + obj = kmem_cache_alloc_node(net_hotdata.skb_small_head_cache, flags | __GFP_NOMEMALLOC | __GFP_NOWARN, node); *size = SKB_SMALL_HEAD_CACHE_SIZE; @@ -586,7 +582,7 @@ static void *kmalloc_reserve(unsigned int *size, gfp_t flags, int node, goto out; /* Try again but now we are using pfmemalloc reserves */ ret_pfmemalloc = true; - obj = kmem_cache_alloc_node(skb_small_head_cache, flags, node); + obj = kmem_cache_alloc_node(net_hotdata.skb_small_head_cache, flags, node); goto out; } @@ -649,7 +645,7 @@ struct sk_buff *__alloc_skb(unsigned int size, gfp_t gfp_mask, u8 *data; cache = (flags & SKB_ALLOC_FCLONE) - ? skbuff_fclone_cache : skbuff_cache; + ? net_hotdata.skbuff_fclone_cache : net_hotdata.skbuff_cache; if (sk_memalloc_socks() && (flags & SKB_ALLOC_RX)) gfp_mask |= __GFP_MEMALLOC; @@ -1095,7 +1091,7 @@ static int skb_pp_frag_ref(struct sk_buff *skb) static void skb_kfree_head(void *head, unsigned int end_offset) { if (end_offset == SKB_SMALL_HEAD_HEADROOM) - kmem_cache_free(skb_small_head_cache, head); + kmem_cache_free(net_hotdata.skb_small_head_cache, head); else kfree(head); } @@ -1162,7 +1158,7 @@ static void kfree_skbmem(struct sk_buff *skb) switch (skb->fclone) { case SKB_FCLONE_UNAVAILABLE: - kmem_cache_free(skbuff_cache, skb); + kmem_cache_free(net_hotdata.skbuff_cache, skb); return; case SKB_FCLONE_ORIG: @@ -1183,7 +1179,7 @@ static void kfree_skbmem(struct sk_buff *skb) if (!refcount_dec_and_test(&fclones->fclone_ref)) return; fastpath: - kmem_cache_free(skbuff_fclone_cache, fclones); + kmem_cache_free(net_hotdata.skbuff_fclone_cache, fclones); } void skb_release_head_state(struct sk_buff *skb) @@ -1280,7 +1276,7 @@ static void kfree_skb_add_bulk(struct sk_buff *skb, sa->skb_array[sa->skb_count++] = skb; if (unlikely(sa->skb_count == KFREE_SKB_BULK_SIZE)) { - kmem_cache_free_bulk(skbuff_cache, KFREE_SKB_BULK_SIZE, + kmem_cache_free_bulk(net_hotdata.skbuff_cache, KFREE_SKB_BULK_SIZE, sa->skb_array); sa->skb_count = 0; } @@ -1305,7 +1301,7 @@ kfree_skb_list_reason(struct sk_buff *segs, enum skb_drop_reason reason) } if (sa.skb_count) - kmem_cache_free_bulk(skbuff_cache, sa.skb_count, sa.skb_array); + kmem_cache_free_bulk(net_hotdata.skbuff_cache, sa.skb_count, sa.skb_array); } EXPORT_SYMBOL(kfree_skb_list_reason); @@ -1467,9 +1463,9 @@ static void napi_skb_cache_put(struct sk_buff *skb) if (unlikely(nc->skb_count == NAPI_SKB_CACHE_SIZE)) { for (i = NAPI_SKB_CACHE_HALF; i < NAPI_SKB_CACHE_SIZE; i++) kasan_mempool_unpoison_object(nc->skb_cache[i], - kmem_cache_size(skbuff_cache)); + kmem_cache_size(net_hotdata.skbuff_cache)); - kmem_cache_free_bulk(skbuff_cache, NAPI_SKB_CACHE_HALF, + kmem_cache_free_bulk(net_hotdata.skbuff_cache, NAPI_SKB_CACHE_HALF, nc->skb_cache + NAPI_SKB_CACHE_HALF); nc->skb_count = NAPI_SKB_CACHE_HALF; } @@ -2066,7 +2062,7 @@ struct sk_buff *skb_clone(struct sk_buff *skb, gfp_t gfp_mask) if (skb_pfmemalloc(skb)) gfp_mask |= __GFP_MEMALLOC; - n = kmem_cache_alloc(skbuff_cache, gfp_mask); + n = kmem_cache_alloc(net_hotdata.skbuff_cache, gfp_mask); if (!n) return NULL; @@ -5005,7 +5001,7 @@ static void skb_extensions_init(void) {} void __init skb_init(void) { - skbuff_cache = kmem_cache_create_usercopy("skbuff_head_cache", + net_hotdata.skbuff_cache = kmem_cache_create_usercopy("skbuff_head_cache", sizeof(struct sk_buff), 0, SLAB_HWCACHE_ALIGN|SLAB_PANIC| @@ -5013,7 +5009,7 @@ void __init skb_init(void) offsetof(struct sk_buff, cb), sizeof_field(struct sk_buff, cb), NULL); - skbuff_fclone_cache = kmem_cache_create("skbuff_fclone_cache", + net_hotdata.skbuff_fclone_cache = kmem_cache_create("skbuff_fclone_cache", sizeof(struct sk_buff_fclones), 0, SLAB_HWCACHE_ALIGN|SLAB_PANIC, @@ -5022,7 +5018,7 @@ void __init skb_init(void) * struct skb_shared_info is located at the end of skb->head, * and should not be copied to/from user. */ - skb_small_head_cache = kmem_cache_create_usercopy("skbuff_small_head", + net_hotdata.skb_small_head_cache = kmem_cache_create_usercopy("skbuff_small_head", SKB_SMALL_HEAD_CACHE_SIZE, 0, SLAB_HWCACHE_ALIGN | SLAB_PANIC, @@ -5895,7 +5891,7 @@ void kfree_skb_partial(struct sk_buff *skb, bool head_stolen) { if (head_stolen) { skb_release_head_state(skb); - kmem_cache_free(skbuff_cache, skb); + kmem_cache_free(net_hotdata.skbuff_cache, skb); } else { __kfree_skb(skb); } diff --git a/net/core/xdp.c b/net/core/xdp.c index 0e3709a29175baf3ee90fdc9cc2d97d861da11a3..41693154e426f78656cc18fc97ca7e82e648270b 100644 --- a/net/core/xdp.c +++ b/net/core/xdp.c @@ -16,6 +16,7 @@ #include #include +#include #include #include /* struct xdp_mem_allocator */ #include @@ -589,7 +590,7 @@ EXPORT_SYMBOL_GPL(xdp_warn); int xdp_alloc_skb_bulk(void **skbs, int n_skb, gfp_t gfp) { - n_skb = kmem_cache_alloc_bulk(skbuff_cache, gfp, n_skb, skbs); + n_skb = kmem_cache_alloc_bulk(net_hotdata.skbuff_cache, gfp, n_skb, skbs); if (unlikely(!n_skb)) return -ENOMEM; @@ -658,7 +659,7 @@ struct sk_buff *xdp_build_skb_from_frame(struct xdp_frame *xdpf, { struct sk_buff *skb; - skb = kmem_cache_alloc(skbuff_cache, GFP_ATOMIC); + skb = kmem_cache_alloc(net_hotdata.skbuff_cache, GFP_ATOMIC); if (unlikely(!skb)) return NULL; From patchwork Wed Mar 6 16:00:24 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Dumazet X-Patchwork-Id: 13584259 X-Patchwork-Delegate: kuba@kernel.org Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E753813BAF1 for ; Wed, 6 Mar 2024 16:00:59 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709740861; cv=none; b=mC45e+pJrc3ejwx4IY6aMrLzhJiIfr5pb+Ro6jwpMB+xBtDNoqMFNd0oAQoNUtpemBbwemh7hRWXPGCQRRDg7TxO/oiwxspQ/oiZXZgiZ8yxKgBitqtWypO45F6LoWM7Y/JVCLvY1K2Sj9yhY+F6Q7DS2Sd3zjcYT3XdA9U/9lQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709740861; c=relaxed/simple; bh=7Zc5yfv2/wmmgPY9sB4q+JazwB9o4N2yz/Dr0gPHVv4=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=MPYujCUQletmKaDVLzYNkbjcBF60CUvOz7q/d3vm3e/9NjSSV7Qx/xN8jk1MVPIZLGp39Xu/JXzNU5a45Hqhq8nOf3EqgMD0RncuZ4QVskxMYsK8Mb7bGkhGornq+P+DpyCFdGbSGxNjfJlOrXdwLrTpNMHTMp/7nEiruGQ3nCw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--edumazet.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=Q7SLnKSb; arc=none smtp.client-ip=209.85.128.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--edumazet.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="Q7SLnKSb" Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-6047fed0132so105807267b3.1 for ; Wed, 06 Mar 2024 08:00:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1709740859; x=1710345659; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=kNlWSDSik5lU/4gINrhMuZfSkjaQaOOEbDG3yphfEDE=; b=Q7SLnKSbndOnwnuaDK2IHaQa7cXxXCQKyth0Lydhz85TyHGK3pbnV518j/OdHMBUIq YMxzx0q5B6tYz94VEKMDAVzAhScmi5mZrb6YcKEn4n53U1YbZvipUCCt1EfQpQPEmCCK SHNK8SgiPGtKGB2CN/G6U3hmbcftr6aBg5aGTAwn0FiIv8IQv4YzPssD79VSD4NAsX2/ oAY7qigj38CbkilD0eXZkkbi2ryIyrwjV+6xM9Pk+k+8SwuarJTN5JXvMMihMY60XxG1 hdJT4R4PRSzBzYEwWtYsgBd1ZSeJWYHZIFUFQ1ekH3gQXAFWjmmzqkRoxpjjXBSKisVJ LH8g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709740859; x=1710345659; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=kNlWSDSik5lU/4gINrhMuZfSkjaQaOOEbDG3yphfEDE=; b=qrH0dUu/l+7lOi7Ba/hUA2IG9qCWNKl2MXAMByyQ/WEDFPB3Gs5TtZLI76AdNH9i9W jtV7tSgRLh1ZgXYPft9NNAQ8/g3sxnSnlq8hdYG1DBO8VgdXl7fKIUgtLjQUoSU/+WPI kZQINHpHxA7yX+5C1lwKh59xdDC79ZkKZs/J1t8rV1zKhvVoJUrTH7vpuB/UoaGzd0oU eIYeOgjSggqPHmx22dJfXY1+oHfQwYafkQGK6le58gH9uuQR4nnx1qO1Z+MkgFWaSCwB d8PDnBVpskrwX8adv7L46NlZvr3ZCZU2igXLIxMOGDfgYv3YrU2d7qQQsFG0+ysiZ8KF eEgg== X-Forwarded-Encrypted: i=1; AJvYcCXKnWEHmrCiVi2v9MkW4Z8Xf014i86TpcOPpwgMO6x/B1DHTaD+MBKVLERFomviyHddqAk5IYNonmhfRyeCMW0fqxdifpys X-Gm-Message-State: AOJu0YxRK4gLUnPUNGa31+B0OHU+QCqVocABKbFc7ctzbFdR8xV/hiUF 84ypoU0kI0mQojK9QdpBw1kHFrPdQG9e7ig/b8RrrShIJlqOKveqsarT+8iPrVmw8wRv/q/w5lx 5eN7Mt8Xk3Q== X-Google-Smtp-Source: AGHT+IF6TIS0obR2vkHg66SBGzY49ZzRQAfo5G73OZDyg7+ddJZ028clMiosmkgBSnzarFfrIp35ANFzUCp13A== X-Received: from edumazet1.c.googlers.com ([fda3:e722:ac3:cc00:2b:7d90:c0a8:395a]) (user=edumazet job=sendgmr) by 2002:a25:c794:0:b0:dcb:c2c0:b319 with SMTP id w142-20020a25c794000000b00dcbc2c0b319mr645844ybe.9.1709740859013; Wed, 06 Mar 2024 08:00:59 -0800 (PST) Date: Wed, 6 Mar 2024 16:00:24 +0000 In-Reply-To: <20240306160031.874438-1-edumazet@google.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240306160031.874438-1-edumazet@google.com> X-Mailer: git-send-email 2.44.0.278.ge034bb2e1d-goog Message-ID: <20240306160031.874438-12-edumazet@google.com> Subject: [PATCH v2 net-next 11/18] udp: move udpv4_offload and udpv6_offload to net_hotdata From: Eric Dumazet To: "David S . Miller" , Jakub Kicinski , Paolo Abeni Cc: David Ahern , Willem de Bruijn , Soheil Hassas Yeganeh , Neal Cardwell , netdev@vger.kernel.org, eric.dumazet@gmail.com, Eric Dumazet X-Patchwork-Delegate: kuba@kernel.org These structures are used in GRO and GSO paths. Move them to net_hodata for better cache locality. v2: udpv6_offload definition depends on CONFIG_INET=y Signed-off-by: Eric Dumazet Acked-by: Soheil Hassas Yeganeh Reviewed-by: David Ahern --- include/net/hotdata.h | 2 ++ net/ipv4/udp_offload.c | 17 ++++++++--------- net/ipv6/udp_offload.c | 21 ++++++++++----------- 3 files changed, 20 insertions(+), 20 deletions(-) diff --git a/include/net/hotdata.h b/include/net/hotdata.h index a8f7e5e826fb749965573a5bcf13825a973c16ae..daeee8ce8084e885984716113779750ca7b72147 100644 --- a/include/net/hotdata.h +++ b/include/net/hotdata.h @@ -11,8 +11,10 @@ struct net_hotdata { #if IS_ENABLED(CONFIG_INET) struct packet_offload ip_packet_offload; struct net_offload tcpv4_offload; + struct net_offload udpv4_offload; struct packet_offload ipv6_packet_offload; struct net_offload tcpv6_offload; + struct net_offload udpv6_offload; #endif struct list_head offload_base; struct list_head ptype_all; diff --git a/net/ipv4/udp_offload.c b/net/ipv4/udp_offload.c index 6c95d28d0c4a7e56d587a986113b3711f8de964c..b9880743765c6c24c28bea095f16f0cf091664ce 100644 --- a/net/ipv4/udp_offload.c +++ b/net/ipv4/udp_offload.c @@ -737,15 +737,14 @@ INDIRECT_CALLABLE_SCOPE int udp4_gro_complete(struct sk_buff *skb, int nhoff) return udp_gro_complete(skb, nhoff, udp4_lib_lookup_skb); } -static const struct net_offload udpv4_offload = { - .callbacks = { - .gso_segment = udp4_ufo_fragment, - .gro_receive = udp4_gro_receive, - .gro_complete = udp4_gro_complete, - }, -}; - int __init udpv4_offload_init(void) { - return inet_add_offload(&udpv4_offload, IPPROTO_UDP); + net_hotdata.udpv4_offload = (struct net_offload) { + .callbacks = { + .gso_segment = udp4_ufo_fragment, + .gro_receive = udp4_gro_receive, + .gro_complete = udp4_gro_complete, + }, + }; + return inet_add_offload(&net_hotdata.udpv4_offload, IPPROTO_UDP); } diff --git a/net/ipv6/udp_offload.c b/net/ipv6/udp_offload.c index 6b95ba241ebe2af7e5f2760d8a9c1d78f08579c5..312bcaeea96fb78ac488124cf7795aa834392c64 100644 --- a/net/ipv6/udp_offload.c +++ b/net/ipv6/udp_offload.c @@ -192,20 +192,19 @@ INDIRECT_CALLABLE_SCOPE int udp6_gro_complete(struct sk_buff *skb, int nhoff) return udp_gro_complete(skb, nhoff, udp6_lib_lookup_skb); } -static const struct net_offload udpv6_offload = { - .callbacks = { - .gso_segment = udp6_ufo_fragment, - .gro_receive = udp6_gro_receive, - .gro_complete = udp6_gro_complete, - }, -}; - -int udpv6_offload_init(void) +int __init udpv6_offload_init(void) { - return inet6_add_offload(&udpv6_offload, IPPROTO_UDP); + net_hotdata.udpv6_offload = (struct net_offload) { + .callbacks = { + .gso_segment = udp6_ufo_fragment, + .gro_receive = udp6_gro_receive, + .gro_complete = udp6_gro_complete, + }, + }; + return inet6_add_offload(&net_hotdata.udpv6_offload, IPPROTO_UDP); } int udpv6_offload_exit(void) { - return inet6_del_offload(&udpv6_offload, IPPROTO_UDP); + return inet6_del_offload(&net_hotdata.udpv6_offload, IPPROTO_UDP); } From patchwork Wed Mar 6 16:00:25 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Dumazet X-Patchwork-Id: 13584260 X-Patchwork-Delegate: kuba@kernel.org Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8848D13BAFC for ; Wed, 6 Mar 2024 16:01:01 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709740863; cv=none; b=HLA903ix7S815bVuDETFNooq//ljXjj52cad6PhTPRKT5jqBxIQEKdRJnI07QCErDulsOAvq65pLWmKPCCrq5rX9vS6FQGHh4LLnVkMfq8JXhEC2mRM7lRmFc1SKD/eEkHVHoUxN0HS6NdJrRLmOvjcN9vBBjvyAu+SQIUcX568= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709740863; c=relaxed/simple; bh=SxvSc5S4M2aTghYlZe1x6o2vP5OkMe3Bg91v4hjdbW0=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=GZizrvt9kLOxPWWss5vOS/R0eHA2F97gj0KAjkkk1Sx8lyEUf9/vx736qWLsOfoUFzNgnLJHskIcRer7s9FKqIUzUPFMhFttwGQr8dXJWmx2adRZRMPbpct+U9g972yTcGWtgFDe8mUtllC+BWIFUZbG57BOdvi7xQ57hD54qAE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--edumazet.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=YtZ1LN15; arc=none smtp.client-ip=209.85.219.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--edumazet.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="YtZ1LN15" Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-dc647f65573so12321107276.2 for ; Wed, 06 Mar 2024 08:01:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1709740860; x=1710345660; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=VD7Qrzoy5OwZBAl3bKy1hcgCDrneG/RPcheXFXFrsPg=; b=YtZ1LN15zFS35Oytib0a2s8VYqqayKsyvRosdbyfGnnP3eRrR6vX6axa1iIcpkXfrw 6Owai7NrfR0jqDKK5gCswBqUwLSEKx0LYIWDw+op5z2tA+W1CAi1IaUWjOSy9FYQ2Ngh NWBEJ+30WQlRt4kNsunVnVD0a7aKKirqDjx6tHEyhZXRLPL4EuWGbTre/CtdBYjsjQ6l 3RgqRQtgXAwuHl5saMtBGXPJc6ES3GMITJMQ5IRiGEAv+9fnikUw4upO4kLTZjqq9cy/ 4pr1gbzQ/81Vev8aPPCAoLx94m8Userrg8pikPbv+TqlnNK6blh1bnqXJ8sSurqSCW/8 fIGA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709740860; x=1710345660; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=VD7Qrzoy5OwZBAl3bKy1hcgCDrneG/RPcheXFXFrsPg=; b=PnLnW2e/7wWEJif1NkyWS+vhLxZKBr65NfktIRdt8HM2iXQLcsaFDFZ9vRzj4snjFJ Ta22GgoDX5pZ1dKPikRrxs7ddzwqTw8BTWEpipCAfr4wpMXx+ixJ1iAZFvn6ADrWvJTq eYL84T5eSuV8PBaj+fTHAeu3iLrDAsD3lg60glxn2xM0bPZtQ+I27/IUAgUPL9IwD1Bh XPScCwFoUmnfYyb6jjSrDmZLXgCX0bwloyzpJPDH4zWVRKDpPy/DApaLr9+x/em2VFym nYsVmcav/SoxcDazjm+81ak5mN8PlStRhzNIw8TCypgwurPtAcGliBXNcbeHMQSJFrVw KNRQ== X-Forwarded-Encrypted: i=1; AJvYcCV3bikyQ6XMmwQ9oJlf211MBLArNJay5aPZzWyh5TAUcXoU+yCU1H9hMMAI0H0UZ2cEyQ6Y/Pelf0mYy/iWetKvN09Mzfkg X-Gm-Message-State: AOJu0Yw792CwUgXFgb4jnnkKqu9sZRHZseDB89kT27eH1Oe89n18dD6g 1G65T0SKXvk+BKV/A129uirXmOE54Q4dnBlVIR0kp7U/NNJAezcvwJKMZWrXdHgUsms1ceiivRO rjAC/tPJjJA== X-Google-Smtp-Source: AGHT+IF1CCUvyz603godRcCYOaZh6QOV1bRfIkwXgljeJO1mYaYlvB1HCjKULTZe+V7nKsTpuTgZKLALuZtrtQ== X-Received: from edumazet1.c.googlers.com ([fda3:e722:ac3:cc00:2b:7d90:c0a8:395a]) (user=edumazet job=sendgmr) by 2002:a05:6902:154d:b0:dcf:411d:67b9 with SMTP id r13-20020a056902154d00b00dcf411d67b9mr3899565ybu.5.1709740860714; Wed, 06 Mar 2024 08:01:00 -0800 (PST) Date: Wed, 6 Mar 2024 16:00:25 +0000 In-Reply-To: <20240306160031.874438-1-edumazet@google.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240306160031.874438-1-edumazet@google.com> X-Mailer: git-send-email 2.44.0.278.ge034bb2e1d-goog Message-ID: <20240306160031.874438-13-edumazet@google.com> Subject: [PATCH v2 net-next 12/18] ipv6: move tcpv6_protocol and udpv6_protocol to net_hotdata From: Eric Dumazet To: "David S . Miller" , Jakub Kicinski , Paolo Abeni Cc: David Ahern , Willem de Bruijn , Soheil Hassas Yeganeh , Neal Cardwell , netdev@vger.kernel.org, eric.dumazet@gmail.com, Eric Dumazet X-Patchwork-Delegate: kuba@kernel.org These structures are read in rx path, move them to net_hotdata for better cache locality. Signed-off-by: Eric Dumazet Acked-by: Soheil Hassas Yeganeh Reviewed-by: David Ahern --- include/net/hotdata.h | 4 ++++ net/ipv6/tcp_ipv6.c | 17 +++++++++-------- net/ipv6/udp.c | 16 ++++++++-------- 3 files changed, 21 insertions(+), 16 deletions(-) diff --git a/include/net/hotdata.h b/include/net/hotdata.h index daeee8ce8084e885984716113779750ca7b72147..03d758d25c02864b00e0a557603b64dc9d749b9c 100644 --- a/include/net/hotdata.h +++ b/include/net/hotdata.h @@ -14,6 +14,10 @@ struct net_hotdata { struct net_offload udpv4_offload; struct packet_offload ipv6_packet_offload; struct net_offload tcpv6_offload; +#if IS_ENABLED(CONFIG_IPV6) + struct inet6_protocol tcpv6_protocol; + struct inet6_protocol udpv6_protocol; +#endif struct net_offload udpv6_offload; #endif struct list_head offload_base; diff --git a/net/ipv6/tcp_ipv6.c b/net/ipv6/tcp_ipv6.c index f677f0fa51968d00c3571d55ae7850742387f2d1..3f4cba49e9ee6520987993dcea082e6065b4688b 100644 --- a/net/ipv6/tcp_ipv6.c +++ b/net/ipv6/tcp_ipv6.c @@ -58,6 +58,7 @@ #include #include #include +#include #include #include @@ -2367,11 +2368,6 @@ struct proto tcpv6_prot = { }; EXPORT_SYMBOL_GPL(tcpv6_prot); -static const struct inet6_protocol tcpv6_protocol = { - .handler = tcp_v6_rcv, - .err_handler = tcp_v6_err, - .flags = INET6_PROTO_NOPOLICY|INET6_PROTO_FINAL, -}; static struct inet_protosw tcpv6_protosw = { .type = SOCK_STREAM, @@ -2408,7 +2404,12 @@ int __init tcpv6_init(void) { int ret; - ret = inet6_add_protocol(&tcpv6_protocol, IPPROTO_TCP); + net_hotdata.tcpv6_protocol = (struct inet6_protocol) { + .handler = tcp_v6_rcv, + .err_handler = tcp_v6_err, + .flags = INET6_PROTO_NOPOLICY | INET6_PROTO_FINAL, + }; + ret = inet6_add_protocol(&net_hotdata.tcpv6_protocol, IPPROTO_TCP); if (ret) goto out; @@ -2433,7 +2434,7 @@ int __init tcpv6_init(void) out_tcpv6_protosw: inet6_unregister_protosw(&tcpv6_protosw); out_tcpv6_protocol: - inet6_del_protocol(&tcpv6_protocol, IPPROTO_TCP); + inet6_del_protocol(&net_hotdata.tcpv6_protocol, IPPROTO_TCP); goto out; } @@ -2441,5 +2442,5 @@ void tcpv6_exit(void) { unregister_pernet_subsys(&tcpv6_net_ops); inet6_unregister_protosw(&tcpv6_protosw); - inet6_del_protocol(&tcpv6_protocol, IPPROTO_TCP); + inet6_del_protocol(&net_hotdata.tcpv6_protocol, IPPROTO_TCP); } diff --git a/net/ipv6/udp.c b/net/ipv6/udp.c index 3f2249b4cd5f6a594dd9768e29f20f0d9a57faed..97d86909aabb6588d0bba901f6df1f23a4f2e561 100644 --- a/net/ipv6/udp.c +++ b/net/ipv6/udp.c @@ -1702,11 +1702,6 @@ int udpv6_getsockopt(struct sock *sk, int level, int optname, return ipv6_getsockopt(sk, level, optname, optval, optlen); } -static const struct inet6_protocol udpv6_protocol = { - .handler = udpv6_rcv, - .err_handler = udpv6_err, - .flags = INET6_PROTO_NOPOLICY|INET6_PROTO_FINAL, -}; /* ------------------------------------------------------------------------ */ #ifdef CONFIG_PROC_FS @@ -1803,7 +1798,12 @@ int __init udpv6_init(void) { int ret; - ret = inet6_add_protocol(&udpv6_protocol, IPPROTO_UDP); + net_hotdata.udpv6_protocol = (struct inet6_protocol) { + .handler = udpv6_rcv, + .err_handler = udpv6_err, + .flags = INET6_PROTO_NOPOLICY | INET6_PROTO_FINAL, + }; + ret = inet6_add_protocol(&net_hotdata.udpv6_protocol, IPPROTO_UDP); if (ret) goto out; @@ -1814,12 +1814,12 @@ int __init udpv6_init(void) return ret; out_udpv6_protocol: - inet6_del_protocol(&udpv6_protocol, IPPROTO_UDP); + inet6_del_protocol(&net_hotdata.udpv6_protocol, IPPROTO_UDP); goto out; } void udpv6_exit(void) { inet6_unregister_protosw(&udpv6_protosw); - inet6_del_protocol(&udpv6_protocol, IPPROTO_UDP); + inet6_del_protocol(&net_hotdata.udpv6_protocol, IPPROTO_UDP); } From patchwork Wed Mar 6 16:00:26 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Dumazet X-Patchwork-Id: 13584261 X-Patchwork-Delegate: kuba@kernel.org Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C7D6413C9D4 for ; Wed, 6 Mar 2024 16:01:03 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709740865; cv=none; b=ejWoriKfl+lo8LIdkZE4a0keu9H2Sz1KbbOc6H+8IX1inf9ri5oBKE5H+bxLppExWZRicHYekNWYhIbgt2DV2V03dpgGx+bADK9E/HvjSCs+cuGifUiLjiTiqnk0pcOo9Icb3xIZonXxd3/sh/gR8kIvUc9ntEGlAOX9D5ggha4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709740865; c=relaxed/simple; bh=Q/XwijER6R+qYN0p3vjy3asKZUPyJXODtTyK+3FMed8=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=Vs2tcAXzc4Ddu9n2fVj4m1UlTXN+Z6Ck7PB68SSH0rD4bHR1zBdqtyi3iG3R1kTq+ZDEeWyUCf+/17WEG+g98Lxsu2s4Z6HjXB8HR+UsJJCxwa63D8mB3OuQYdrsNsXNU4uuJsg7LHbn2KyJfn8s4t23hu0ChdjgwCYXNCnG/WQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--edumazet.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=v+B1p/0T; arc=none smtp.client-ip=209.85.219.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--edumazet.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="v+B1p/0T" Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-dc6b26783b4so8729549276.0 for ; Wed, 06 Mar 2024 08:01:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1709740863; x=1710345663; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=l2Hgk8n9p4LCBmuuj6kS57w+CBqwKZ1lZeYV+GA9Ymk=; b=v+B1p/0TWSOsfn9ITqgXPAO+dCUKEmJqbFes6hyE20lTA520FlKyh+J7szZaTqv4LY KRgcLR5lumEV9cCpUp6cw+Iovi25IeWu0rUB6yO7gXVzaQWrtpq2VbgXnpSiIhdx0q9C zUYoCRBiDtqJ4JIb7x0XqCr4m+2e9WtfPdrPHkJwCrKX85lidPOuDrTs+ptDJibFnyWg 93X2tUbdOpdsDL9I72CQuJgy7eCS9GORlSH0HSr+r+SGW3TyUAMfPd/SHnIW7jnuGd8+ 3fOauaL2RZOtcqLPPOSNR7nh39tV0NoBsMaz3LcqK/eO4wZxHeE9OmED4hvMwqJ8ophi Fqtg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709740863; x=1710345663; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=l2Hgk8n9p4LCBmuuj6kS57w+CBqwKZ1lZeYV+GA9Ymk=; b=Ek6FT0JExDLWreLIsmm75Knih+8NcjszDUmPxxQ15u2FCIYFoET4cnyDQT58ZzolVN 4TdJTP/dRIHTB7coDiFP1rlozZ2f/ws9iBXO7hHVkiFJpICLGCRK9uGRuv+KTvBdt9Ml Nj12FwtZIMkIl2tpYhLnLQbd0GSWuzX9jPMt3kVsJwxAaa+Ezj5IKjtaKEifSLk3aP2B INWywUZRMn6H4UAEa9Fsjp5YF9GlbB8f1x5kCzUu0xEoHvjt/TTl/K55X6AtyvzRTiFj FEoD+RudVSy/X/niJvY5MoHIZ08sDKYTJvPgLkVm2h7aLqQqMQcntEYdO4pcgYqlseO+ lW8g== X-Forwarded-Encrypted: i=1; AJvYcCXcYy0Vhf3dvjdYTozdp0yx2EpXCeTIzSD0HWvBUfYokIFenBbtpmlUIOo9U/DUPF0SQ8ynDvjHN5A+qgPqQreXw+KnW5yo X-Gm-Message-State: AOJu0Ywf4zSs2pxTey5HNP9WcdoTFwHa8Tv7RMXo2FCuIwuSepmouwOs oP937rkb9kKEDLAeG86IADtSnEKUV4pDBeiB1xXOV/AQSDUbBRbyPE6E/j6rEJ3vdFFols9wAj7 +bumw0VANyg== X-Google-Smtp-Source: AGHT+IHJvE9nu91L/pxhHEbqxOImZdVvS7nvzs7hXXkOYhLmeBH5bxxqceQtLH7XvYLPITzotwZXRvq5IwUebQ== X-Received: from edumazet1.c.googlers.com ([fda3:e722:ac3:cc00:2b:7d90:c0a8:395a]) (user=edumazet job=sendgmr) by 2002:a05:6902:1148:b0:dc6:b813:5813 with SMTP id p8-20020a056902114800b00dc6b8135813mr501260ybu.9.1709740863022; Wed, 06 Mar 2024 08:01:03 -0800 (PST) Date: Wed, 6 Mar 2024 16:00:26 +0000 In-Reply-To: <20240306160031.874438-1-edumazet@google.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240306160031.874438-1-edumazet@google.com> X-Mailer: git-send-email 2.44.0.278.ge034bb2e1d-goog Message-ID: <20240306160031.874438-14-edumazet@google.com> Subject: [PATCH v2 net-next 13/18] inet: move tcp_protocol and udp_protocol to net_hotdata From: Eric Dumazet To: "David S . Miller" , Jakub Kicinski , Paolo Abeni Cc: David Ahern , Willem de Bruijn , Soheil Hassas Yeganeh , Neal Cardwell , netdev@vger.kernel.org, eric.dumazet@gmail.com, Eric Dumazet X-Patchwork-Delegate: kuba@kernel.org These structures are read in rx path, move them to net_hotdata for better cache locality. Signed-off-by: Eric Dumazet Acked-by: Soheil Hassas Yeganeh Reviewed-by: David Ahern --- include/net/hotdata.h | 2 ++ net/ipv4/af_inet.c | 30 +++++++++++++++--------------- 2 files changed, 17 insertions(+), 15 deletions(-) diff --git a/include/net/hotdata.h b/include/net/hotdata.h index 03d758d25c02864b00e0a557603b64dc9d749b9c..87215f7ac200f2ba34de9b52841bc0c9e4849857 100644 --- a/include/net/hotdata.h +++ b/include/net/hotdata.h @@ -11,7 +11,9 @@ struct net_hotdata { #if IS_ENABLED(CONFIG_INET) struct packet_offload ip_packet_offload; struct net_offload tcpv4_offload; + struct net_protocol tcp_protocol; struct net_offload udpv4_offload; + struct net_protocol udp_protocol; struct packet_offload ipv6_packet_offload; struct net_offload tcpv6_offload; #if IS_ENABLED(CONFIG_IPV6) diff --git a/net/ipv4/af_inet.c b/net/ipv4/af_inet.c index 08dda6955562ea6b89e02b8299b03ab52b342f27..6f1cfd176e7b84f23d8a5e505bf8e13b2b755f06 100644 --- a/net/ipv4/af_inet.c +++ b/net/ipv4/af_inet.c @@ -1751,19 +1751,6 @@ static const struct net_protocol igmp_protocol = { }; #endif -static const struct net_protocol tcp_protocol = { - .handler = tcp_v4_rcv, - .err_handler = tcp_v4_err, - .no_policy = 1, - .icmp_strict_tag_validation = 1, -}; - -static const struct net_protocol udp_protocol = { - .handler = udp_rcv, - .err_handler = udp_err, - .no_policy = 1, -}; - static const struct net_protocol icmp_protocol = { .handler = icmp_rcv, .err_handler = icmp_err, @@ -1992,9 +1979,22 @@ static int __init inet_init(void) if (inet_add_protocol(&icmp_protocol, IPPROTO_ICMP) < 0) pr_crit("%s: Cannot add ICMP protocol\n", __func__); - if (inet_add_protocol(&udp_protocol, IPPROTO_UDP) < 0) + + net_hotdata.udp_protocol = (struct net_protocol) { + .handler = udp_rcv, + .err_handler = udp_err, + .no_policy = 1, + }; + if (inet_add_protocol(&net_hotdata.udp_protocol, IPPROTO_UDP) < 0) pr_crit("%s: Cannot add UDP protocol\n", __func__); - if (inet_add_protocol(&tcp_protocol, IPPROTO_TCP) < 0) + + net_hotdata.tcp_protocol = (struct net_protocol) { + .handler = tcp_v4_rcv, + .err_handler = tcp_v4_err, + .no_policy = 1, + .icmp_strict_tag_validation = 1, + }; + if (inet_add_protocol(&net_hotdata.tcp_protocol, IPPROTO_TCP) < 0) pr_crit("%s: Cannot add TCP protocol\n", __func__); #ifdef CONFIG_IP_MULTICAST if (inet_add_protocol(&igmp_protocol, IPPROTO_IGMP) < 0) From patchwork Wed Mar 6 16:00:27 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Dumazet X-Patchwork-Id: 13584262 X-Patchwork-Delegate: kuba@kernel.org Received: from mail-qt1-f202.google.com (mail-qt1-f202.google.com [209.85.160.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BCECD13D2E8 for ; Wed, 6 Mar 2024 16:01:05 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.160.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709740867; cv=none; b=MmAY4b0RQtXpgor7zr5WA5CsdjZSbyPPN1+agI99rfr5u3ZilR8dw0tiZxhwg52kJAvmA9jL03jPxvNWoEYaSKpqAfbis0AxSJOpjAIR2r0pxKVRiRX4O3YvswCeoz1b/LQkivDg+S3s0sk9XXyvfEaD5UBaUgiIXgtOK8A+/JM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709740867; c=relaxed/simple; bh=2opeJepDAvcGIAcNgC0yTFwPLR2lWRo0m74ZuzjGz90=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=rVWq+Kek3YmIEIieSK8AEUmo6A/Ivi/8q3LLVWbazhuGy6Rxa916FrIqv/5x1TQJCaOFeJRRCkK690F5UDYEgNoED8FQdukt6HbdEgACuRJ6Q47LVKWrPdeLRIuYqlq+R6RnYQbkM9XMPi1bsAb4XpqZzYrz1nKYwuRqVofu0yU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--edumazet.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=UltMZEt7; arc=none smtp.client-ip=209.85.160.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--edumazet.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="UltMZEt7" Received: by mail-qt1-f202.google.com with SMTP id d75a77b69052e-42edd97c9e6so43291701cf.1 for ; Wed, 06 Mar 2024 08:01:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1709740865; x=1710345665; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=j6QGdYIRZ1txU93levxcNkXzKVT3WDaykDhB0OxJjos=; b=UltMZEt7jSLgvipEDBJma5OE1qU2T98VP4H2/2bUFJKT3nLki5cOVRRY+p3XzoCztz UMlzKVHTuxisPyzpxqb0OCN3ZEn9rnRcDLxlfFngTtB265mqkkGBRWfleMsW7UQj0o9P 1ObMz8rrPvYmvoJ4C1J14oArjakbYviWxJ2h0AQBDyhA4sjV29MhyNTrMMoIEv9mJofQ LXX+OOkq7nAP/qpl3K9UrKQ/EYyYSmuUs5XnB5BuAMrw2wuJ8DGRZONHA0bW6UpN/ixk LSN9+4U3WqmrunLC28kpp3zYSbXB8MI7JG1otQ96OrES3Qt0+ztHwNeFKCU5jMsomcNV OT5g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709740865; x=1710345665; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=j6QGdYIRZ1txU93levxcNkXzKVT3WDaykDhB0OxJjos=; b=DMffpMOGrLynV7zm7Cu4lmFyFJ9HStat/BxTzR47TncNsvDHkxybTcal0SodBJngaS mdqdC0c0BNcbMkH3fy8bSOMipfybQdA+MYEheYQNtvdGj3Dj7b8uHUCc1p/LxnYjetko bTFNR+QfLtAwEHAGxMVx9SUUE825bpIUPmNEvn8UCBRTRIkAea7SHFHLB1CQmedpPaAr MvUUvmycrYzqN47uzygWZjTdRuVT8AUk7FJCOhEUBDV5VmDN6Ykbidft/49kso+rmK82 AEVXTgN7XiOQZW0DUiTqfNRQcOp+oXMU1uqEdXQ56McEtF1aGmHWpoh0gTVyQD7gXGS5 bXlQ== X-Forwarded-Encrypted: i=1; AJvYcCWlKg2HelEM2A7o+k+tVs/R2ysH2V/gcEufu0qDqy2Zsl036Km6g4f8wTRIt3pA27he+ZGZLW794u+QC9SWZDCuHbyDHT5N X-Gm-Message-State: AOJu0YzqbwXSYBTbjQWqkkkBKZ6lwnHv7/72dYMGrQPg2wjlS6CvorSY 2XL4sh1mDP8sPG9SPuBrkp9yigims6z1DxX09gcl5fOSm6QKcEFRcQfpMI9YLDKt3Onwm5z5hsA jS4tKT5ApLA== X-Google-Smtp-Source: AGHT+IGDhIwrizAJYK/rH1j4Y0BOrFqAe/Pku5kY9rF96ILeVxHeVyNuOrxRZKohTtbPuLjPre5WFOWUDeaGbQ== X-Received: from edumazet1.c.googlers.com ([fda3:e722:ac3:cc00:2b:7d90:c0a8:395a]) (user=edumazet job=sendgmr) by 2002:a05:6214:f22:b0:68f:2fe5:d4bf with SMTP id iw2-20020a0562140f2200b0068f2fe5d4bfmr317188qvb.13.1709740864721; Wed, 06 Mar 2024 08:01:04 -0800 (PST) Date: Wed, 6 Mar 2024 16:00:27 +0000 In-Reply-To: <20240306160031.874438-1-edumazet@google.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240306160031.874438-1-edumazet@google.com> X-Mailer: git-send-email 2.44.0.278.ge034bb2e1d-goog Message-ID: <20240306160031.874438-15-edumazet@google.com> Subject: [PATCH v2 net-next 14/18] inet: move inet_ehash_secret and udp_ehash_secret into net_hotdata From: Eric Dumazet To: "David S . Miller" , Jakub Kicinski , Paolo Abeni Cc: David Ahern , Willem de Bruijn , Soheil Hassas Yeganeh , Neal Cardwell , netdev@vger.kernel.org, eric.dumazet@gmail.com, Eric Dumazet X-Patchwork-Delegate: kuba@kernel.org "struct net_protocol" has a 32bit hole in 32bit arches. Use it to store the 32bit secret used by UDP and TCP, to increase cache locality in rx path. Signed-off-by: Eric Dumazet Acked-by: Soheil Hassas Yeganeh Reviewed-by: David Ahern --- include/net/hotdata.h | 3 +++ include/net/protocol.h | 1 + net/ipv4/inet_hashtables.c | 3 +-- net/ipv4/udp.c | 2 -- 4 files changed, 5 insertions(+), 4 deletions(-) diff --git a/include/net/hotdata.h b/include/net/hotdata.h index 87215f7ac200f2ba34de9b52841bc0c9e4849857..7a210ea6899c6af4e0424201cd265755ef460078 100644 --- a/include/net/hotdata.h +++ b/include/net/hotdata.h @@ -36,6 +36,9 @@ struct net_hotdata { int dev_rx_weight; }; +#define inet_ehash_secret net_hotdata.tcp_protocol.secret +#define udp_ehash_secret net_hotdata.udp_protocol.secret + extern struct net_hotdata net_hotdata; #endif /* _NET_HOTDATA_H */ diff --git a/include/net/protocol.h b/include/net/protocol.h index 6aef8cb11cc8c409e5f7a2519f5e747be584c8d5..3ff26e66735cec98b08eadb1c3f129e011923cb0 100644 --- a/include/net/protocol.h +++ b/include/net/protocol.h @@ -46,6 +46,7 @@ struct net_protocol { * socket lookup? */ icmp_strict_tag_validation:1; + u32 secret; }; #if IS_ENABLED(CONFIG_IPV6) diff --git a/net/ipv4/inet_hashtables.c b/net/ipv4/inet_hashtables.c index 308ff34002ea6b5e0620004f65ffd833087afbc1..7498af3201647fd937bf8177f04c200bea178a79 100644 --- a/net/ipv4/inet_hashtables.c +++ b/net/ipv4/inet_hashtables.c @@ -24,6 +24,7 @@ #include #endif #include +#include #include #include #include @@ -32,8 +33,6 @@ u32 inet_ehashfn(const struct net *net, const __be32 laddr, const __u16 lport, const __be32 faddr, const __be16 fport) { - static u32 inet_ehash_secret __read_mostly; - net_get_random_once(&inet_ehash_secret, sizeof(inet_ehash_secret)); return __inet_ehashfn(laddr, lport, faddr, fport, diff --git a/net/ipv4/udp.c b/net/ipv4/udp.c index a8acea17b4e5344d022ae8f8eb674d1a36f8035a..2beabf5b2d8628f1fed69a0212c57bd3cd638483 100644 --- a/net/ipv4/udp.c +++ b/net/ipv4/udp.c @@ -411,8 +411,6 @@ INDIRECT_CALLABLE_SCOPE u32 udp_ehashfn(const struct net *net, const __be32 laddr, const __u16 lport, const __be32 faddr, const __be16 fport) { - static u32 udp_ehash_secret __read_mostly; - net_get_random_once(&udp_ehash_secret, sizeof(udp_ehash_secret)); return __inet_ehashfn(laddr, lport, faddr, fport, From patchwork Wed Mar 6 16:00:28 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Dumazet X-Patchwork-Id: 13584263 X-Patchwork-Delegate: kuba@kernel.org Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9DF1713D31A for ; Wed, 6 Mar 2024 16:01:07 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709740869; cv=none; b=nvnkTRyR83FHwrc/MwBZ4Ow9HmbZG4Z1A0U1II29Btrm2hH8wFv+Oj0DM8UZENrnWrw5Vt3rxwXp4eX7D0RXPFIcnSv3X9cQov38Tw+GGdk8LeiiUrOvNhoB36H93ISuBJ2H54ox60E/G4dV8BibWZ7a4RimVNLDiA47fYtLcqQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709740869; c=relaxed/simple; bh=/eW15AbMiemK9oqz4ilGYw0a/HW/ZN6orQzNDLi+j2Y=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=XK6KRqoT4rF3zPvJ9+sECEm5PSifUnOAarJbUTIr+wJQYY/2/9A/rxdYUM6TNH0CyUrbpFkYPRMwuYqctBB3NyIoVAD31ZtkLzSqyKbRDR641xAAgXxBOOMerzFTvpE4PuwIuKISDvyYuHXgzho5aWlSxHqSKJjx7qufmYZI8VY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--edumazet.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=eIhtizzP; arc=none smtp.client-ip=209.85.219.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--edumazet.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="eIhtizzP" Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-dc64e0fc7c8so10474628276.2 for ; Wed, 06 Mar 2024 08:01:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1709740867; x=1710345667; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=p2ukU5wCmt4sKFPMRc+fc8OauvFPCsq5nLlkABZEOnE=; b=eIhtizzPeh70wE0/5Q8OL5DBHvRRZuqQiraTmlSO4yEU50gvIw+o5Jm0JRJ7AwNo/g oBVW+0268t60kasJ3GZmrEq8eEt5QlmXGUO+NZoE1mfgVgob7tusCRxw8Zglg89cGCZq AOixz7AO0pDks5cHvRo6B9AkHuY4HjU4C2X/2Ur/CeJrv7J+3X3UEoZIa0wcmloE/t2V cEsk7fgsXkCQHVUA6W/UH+zH69vll4nnW3j2LDD9wbp8jzQF5dGekzkKz+HARRDisQN2 60vHPTgVoz9VGlb/NuMkp4k3k5JBiDoTjK4QNyq0SGSZUCt99bSHlp28y92zA6AwHHhG JXEg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709740867; x=1710345667; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=p2ukU5wCmt4sKFPMRc+fc8OauvFPCsq5nLlkABZEOnE=; b=jWKBtG6HdfGChlEf3GYlc68Xd3T1HLZC4kdRiBTVjJjTVOF7RqGX1ZjX9thNAgcfbh 9e1d4BVbTk/a7WO+AYcRBERfA68gd2cxg2tcn3bW6AS9YddvxIwQ8IS+ZFeTnToNgx3s qsklmbFVLFOZeF0yTFPRBSfjSKOfvlQvlM2FracVyJuhozFJ9/BisAlJc3qGqBK29Y5i awL98uxj9WgsjBKjvzxpIBtce0TQfnxKlMOA8CR0RYVnHvNGXWESeNI3ggdjF2i6PTmp n+L3mQstEwYM6WzoTVfImB5nHsVQrt21EvC2ZuFI/oHEs+j1LBD0A1InArKP7y0FhHd9 DmhA== X-Forwarded-Encrypted: i=1; AJvYcCV+RFIOnXF+m+JlTUOdUQlPDqwTKwXSrqY1J2BuG80Uh1Lb4pGiUqBo5NcMwqRj8VcE3Px0nYZOSEyOikFX/gckMEtu0Ep5 X-Gm-Message-State: AOJu0YybU1vkarR12X48a0TdkPuNtSehbIIrfJW1zK5SyAPln1laNpoH 8nFd+qpoIP9IpFXXnJb8INgPouCaMjOYE7vMvH5SyNAbXh29q2ObdbnibkE3N8glueOBwpq0/hl 9BMa5gLUTlA== X-Google-Smtp-Source: AGHT+IFfgR5Q1V5r56+jQkxCYyYWXwrOOZfCPaoNQs5240C00aL+3LW1Iq+7rt8o8E8JPfE2KCubLeugHje2BA== X-Received: from edumazet1.c.googlers.com ([fda3:e722:ac3:cc00:2b:7d90:c0a8:395a]) (user=edumazet job=sendgmr) by 2002:a05:6902:2206:b0:dc2:26f6:fbc8 with SMTP id dm6-20020a056902220600b00dc226f6fbc8mr649839ybb.7.1709740866795; Wed, 06 Mar 2024 08:01:06 -0800 (PST) Date: Wed, 6 Mar 2024 16:00:28 +0000 In-Reply-To: <20240306160031.874438-1-edumazet@google.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240306160031.874438-1-edumazet@google.com> X-Mailer: git-send-email 2.44.0.278.ge034bb2e1d-goog Message-ID: <20240306160031.874438-16-edumazet@google.com> Subject: [PATCH v2 net-next 15/18] ipv6: move inet6_ehash_secret and udp6_ehash_secret into net_hotdata From: Eric Dumazet To: "David S . Miller" , Jakub Kicinski , Paolo Abeni Cc: David Ahern , Willem de Bruijn , Soheil Hassas Yeganeh , Neal Cardwell , netdev@vger.kernel.org, eric.dumazet@gmail.com, Eric Dumazet X-Patchwork-Delegate: kuba@kernel.org "struct inet6_protocol" has a 32bit hole in 32bit arches. Use it to store the 32bit secret used by UDP and TCP, to increase cache locality in rx path. Signed-off-by: Eric Dumazet Acked-by: Soheil Hassas Yeganeh Reviewed-by: David Ahern --- include/net/hotdata.h | 2 ++ include/net/protocol.h | 1 + net/ipv6/inet6_hashtables.c | 2 +- net/ipv6/udp.c | 1 - 4 files changed, 4 insertions(+), 2 deletions(-) diff --git a/include/net/hotdata.h b/include/net/hotdata.h index 7a210ea6899c6af4e0424201cd265755ef460078..6d5cd967183a59271da1a27edfee1bc6bf155c1b 100644 --- a/include/net/hotdata.h +++ b/include/net/hotdata.h @@ -38,6 +38,8 @@ struct net_hotdata { #define inet_ehash_secret net_hotdata.tcp_protocol.secret #define udp_ehash_secret net_hotdata.udp_protocol.secret +#define inet6_ehash_secret net_hotdata.tcpv6_protocol.secret +#define udp6_ehash_secret net_hotdata.udpv6_protocol.secret extern struct net_hotdata net_hotdata; diff --git a/include/net/protocol.h b/include/net/protocol.h index 3ff26e66735cec98b08eadb1c3f129e011923cb0..213649d2ab098edc4eb9f44a5403540887c2b8ef 100644 --- a/include/net/protocol.h +++ b/include/net/protocol.h @@ -60,6 +60,7 @@ struct inet6_protocol { __be32 info); unsigned int flags; /* INET6_PROTO_xxx */ + u32 secret; }; #define INET6_PROTO_NOPOLICY 0x1 diff --git a/net/ipv6/inet6_hashtables.c b/net/ipv6/inet6_hashtables.c index b0e8d278e8a9b794d0001efdc0f43716f9a34f8f..0fee97f3166cf8326b3b714ac6bde48ca5188cec 100644 --- a/net/ipv6/inet6_hashtables.c +++ b/net/ipv6/inet6_hashtables.c @@ -14,6 +14,7 @@ #include #include +#include #include #include #include @@ -25,7 +26,6 @@ u32 inet6_ehashfn(const struct net *net, const struct in6_addr *laddr, const u16 lport, const struct in6_addr *faddr, const __be16 fport) { - static u32 inet6_ehash_secret __read_mostly; static u32 ipv6_hash_secret __read_mostly; u32 lhash, fhash; diff --git a/net/ipv6/udp.c b/net/ipv6/udp.c index 97d86909aabb6588d0bba901f6df1f23a4f2e561..1e1c67a51675e8534a953a6d4d63211388d95ca9 100644 --- a/net/ipv6/udp.c +++ b/net/ipv6/udp.c @@ -79,7 +79,6 @@ u32 udp6_ehashfn(const struct net *net, const struct in6_addr *faddr, const __be16 fport) { - static u32 udp6_ehash_secret __read_mostly; static u32 udp_ipv6_hash_secret __read_mostly; u32 lhash, fhash; From patchwork Wed Mar 6 16:00:29 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Dumazet X-Patchwork-Id: 13584264 X-Patchwork-Delegate: kuba@kernel.org Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CA58913DBA8 for ; Wed, 6 Mar 2024 16:01:09 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709740871; cv=none; b=rMPMfwOzBbHyeT1hrxJ+tvTDuN/hCO3lOxS5a5JfMUGExlTf4Zcox49/DwIWj6gja1nH6dF+2gfr01z/Slriyx84a9prRqxkKzQQgDY84n6DchGPXmPgXtBfWREoMLGWxXVToeAQqxSTV9vuIaPQ828WJyhXiV+rcluOO/Tfbf4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709740871; c=relaxed/simple; bh=bJ9aOpH+BaxXE35lQxVH4r4D5Eph3o4WWJ9g3/xoPEM=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=BsmGoJYk96dq40EHkxs4RyHZRpHO7jTRRdNMpYYyqyxZswRBMrmeV4kn49AWvZ0l1vc8rx3KmwXm9PmoEeX06/MytfsOP4FxQl7lXEWLnkMvASzREDOUti+jY4soog4+JsJSJE3BcRME5kOjzlXvFacSmBFnPT8/4zUvpvgUkxY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--edumazet.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=W1oUnGUy; arc=none smtp.client-ip=209.85.128.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--edumazet.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="W1oUnGUy" Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-609e2b87a6bso7428017b3.3 for ; Wed, 06 Mar 2024 08:01:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1709740869; x=1710345669; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=2R25cjMr9AGnx6ecTXh/eQZ9+fsMiqc+w/JGSBoEK8o=; b=W1oUnGUyAGjJbxkxwnf2Ii7uZ7pv2iRNa6y331LZjMOguQlj4qdCTFU+YElBcu8UNY yOh8BpWj5LQRqlqdv3zwJMu1ec6vpIo5JHoUAAaxCjq6O6SCFFGKv65Ss5TeQb15o/Px gPygoeHDY0WPyF0rKY6AVmSKVyPR3CcnXjpmmPXMG2LvnlOluLtHMHFDgY3flEc/rP6C KFscjL3kVFA7PZfgDdYJQZIxZfoJZhrndGnE+yP/AllLV8AD6MkhWAIM3iW8IuyEVycc rwWBz0SKfDlsL1xYvLBzyTbwwoYjFHnFnvuaqaqsCXV5KKr60ZpdkiWZPy2ltgeTdA7C AFqQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709740869; x=1710345669; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=2R25cjMr9AGnx6ecTXh/eQZ9+fsMiqc+w/JGSBoEK8o=; b=G1q4wxwY2JmwH1MlY+ep8nJLjgktFCJHAukcLTOAx7JRmQjWMRslTGabgGMI1fjtzN 1jSFl3BiYBzDtWQlyEADIB/7flVBcLPJVbTT3B7MSNMWqV3aMQtLbr6P8u8qvZrvaa2r vSKX/0XMJGsQicNnO+7kUH4Rnxgni3JfGD3cHa29G/KKVMrQYA84hiOdiI6WurCktaRF 1ow688g1NnocqPWDvj76pxmYZjuvu2u92knbzumJWrctqCPnvqNadw87NbABWv9vGyKd Sy9rxtOmd6W1aYzPDUHohVz59Pr0tZ2UfClt3K22TeUXOA7zI0U0b9z/A8/iss2KOIrK Vcpw== X-Forwarded-Encrypted: i=1; AJvYcCWaHP9KQ8/1xRKuYnH64t9wY6lpDg/xrUDoUT3d7nZXACeXpehBty0XZIwikOga4L8gwAIeIwJFwGEKMY3DRPVLKP6qV5Jq X-Gm-Message-State: AOJu0YylzekhitO3eitwfTa0Wsgp4t4iJcsilEY+8sa9N+ioCxu1lDg9 oLJxYokTSMDJ1eqGG+EbNalj+9476g0pGuqHwi202s3ebsQXmYFYm3xGVp1qUvRWpiFKFe4VRT1 95EeT+QYLdg== X-Google-Smtp-Source: AGHT+IGuH/+1sm8gr5pH6TpPwdyBiLAhBdAhs0+TbaxqCAinOgnH+MLmHW+UYQXq106TUoc8dPAXIVL5NDIyaQ== X-Received: from edumazet1.c.googlers.com ([fda3:e722:ac3:cc00:2b:7d90:c0a8:395a]) (user=edumazet job=sendgmr) by 2002:a81:7908:0:b0:609:2cab:1bd7 with SMTP id u8-20020a817908000000b006092cab1bd7mr4361436ywc.1.1709740868862; Wed, 06 Mar 2024 08:01:08 -0800 (PST) Date: Wed, 6 Mar 2024 16:00:29 +0000 In-Reply-To: <20240306160031.874438-1-edumazet@google.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240306160031.874438-1-edumazet@google.com> X-Mailer: git-send-email 2.44.0.278.ge034bb2e1d-goog Message-ID: <20240306160031.874438-17-edumazet@google.com> Subject: [PATCH v2 net-next 16/18] ipv6: move tcp_ipv6_hash_secret and udp_ipv6_hash_secret to net_hotdata From: Eric Dumazet To: "David S . Miller" , Jakub Kicinski , Paolo Abeni Cc: David Ahern , Willem de Bruijn , Soheil Hassas Yeganeh , Neal Cardwell , netdev@vger.kernel.org, eric.dumazet@gmail.com, Eric Dumazet X-Patchwork-Delegate: kuba@kernel.org Use a 32bit hole in "struct net_offload" to store the remaining 32bit secrets used by TCPv6 and UDPv6. Signed-off-by: Eric Dumazet Acked-by: Soheil Hassas Yeganeh Reviewed-by: David Ahern --- include/net/hotdata.h | 2 ++ include/net/protocol.h | 1 + net/ipv6/inet6_hashtables.c | 6 ++---- net/ipv6/udp.c | 2 -- 4 files changed, 5 insertions(+), 6 deletions(-) diff --git a/include/net/hotdata.h b/include/net/hotdata.h index 6d5cd967183a59271da1a27edfee1bc6bf155c1b..b0b847585f7e62245cee81a56b5a252051e07834 100644 --- a/include/net/hotdata.h +++ b/include/net/hotdata.h @@ -39,7 +39,9 @@ struct net_hotdata { #define inet_ehash_secret net_hotdata.tcp_protocol.secret #define udp_ehash_secret net_hotdata.udp_protocol.secret #define inet6_ehash_secret net_hotdata.tcpv6_protocol.secret +#define tcp_ipv6_hash_secret net_hotdata.tcpv6_offload.secret #define udp6_ehash_secret net_hotdata.udpv6_protocol.secret +#define udp_ipv6_hash_secret net_hotdata.udpv6_offload.secret extern struct net_hotdata net_hotdata; diff --git a/include/net/protocol.h b/include/net/protocol.h index 213649d2ab098edc4eb9f44a5403540887c2b8ef..b2499f88f8f8199de28555d0cbd6b4523325205b 100644 --- a/include/net/protocol.h +++ b/include/net/protocol.h @@ -70,6 +70,7 @@ struct inet6_protocol { struct net_offload { struct offload_callbacks callbacks; unsigned int flags; /* Flags used by IPv6 for now */ + u32 secret; }; /* This should be set for any extension header which is compatible with GSO. */ #define INET6_PROTO_GSO_EXTHDR 0x1 diff --git a/net/ipv6/inet6_hashtables.c b/net/ipv6/inet6_hashtables.c index 0fee97f3166cf8326b3b714ac6bde48ca5188cec..2e81383b663b71b95719a295fd9629f1193e4225 100644 --- a/net/ipv6/inet6_hashtables.c +++ b/net/ipv6/inet6_hashtables.c @@ -26,15 +26,13 @@ u32 inet6_ehashfn(const struct net *net, const struct in6_addr *laddr, const u16 lport, const struct in6_addr *faddr, const __be16 fport) { - static u32 ipv6_hash_secret __read_mostly; - u32 lhash, fhash; net_get_random_once(&inet6_ehash_secret, sizeof(inet6_ehash_secret)); - net_get_random_once(&ipv6_hash_secret, sizeof(ipv6_hash_secret)); + net_get_random_once(&tcp_ipv6_hash_secret, sizeof(tcp_ipv6_hash_secret)); lhash = (__force u32)laddr->s6_addr32[3]; - fhash = __ipv6_addr_jhash(faddr, ipv6_hash_secret); + fhash = __ipv6_addr_jhash(faddr, tcp_ipv6_hash_secret); return __inet6_ehashfn(lhash, lport, fhash, fport, inet6_ehash_secret + net_hash_mix(net)); diff --git a/net/ipv6/udp.c b/net/ipv6/udp.c index 1e1c67a51675e8534a953a6d4d63211388d95ca9..80ad8f436b179d7279cbbd5108c3494a9773c0d8 100644 --- a/net/ipv6/udp.c +++ b/net/ipv6/udp.c @@ -79,8 +79,6 @@ u32 udp6_ehashfn(const struct net *net, const struct in6_addr *faddr, const __be16 fport) { - static u32 udp_ipv6_hash_secret __read_mostly; - u32 lhash, fhash; net_get_random_once(&udp6_ehash_secret, From patchwork Wed Mar 6 16:00:30 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Dumazet X-Patchwork-Id: 13584265 X-Patchwork-Delegate: kuba@kernel.org Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B6ADA13DBB6 for ; Wed, 6 Mar 2024 16:01:11 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709740873; cv=none; b=Ei11B2rysmVKrZxRWbl2fG00hH1ErqNGcFEnUwRRjuSfcbTEfeRHwm42RuVkfxZG4ciRs/eezlAfu6lnfwIAr9QaGPFIF+fkwrsCrP5zcRjMW9dis7icsnNwByZ7Vk7a9FUofeFRdlHL42XjiuDrkBFbw/4JsKbjtHED+w5y8U4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709740873; c=relaxed/simple; bh=oqqNL4Z1T6Z6WkpJ00qFLExCGHgsk544z8bLD5203nQ=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=gsWk7O7yFWslQSaQ1lusdSU/rDRj/ske5hR0fHPx1fWjMcbKXc9g+X8wye7e/Y5Yo2x/vbdnRSHQUyjiCNb+Ikt6oCAnxphJJYO0tKLjfqR3qqe+916oE8SVivPsdSy6UYcriqEChkhsheVmD0jRMGkAHihbFUjLnta/dQ1qSaA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--edumazet.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=LhfELiFo; arc=none smtp.client-ip=209.85.219.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--edumazet.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="LhfELiFo" Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-dc6b267bf11so8015068276.2 for ; Wed, 06 Mar 2024 08:01:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1709740870; x=1710345670; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=LnhY5lGlh/lAm5l28CdAlmLkBgaxlmQl8oMh/F4izdc=; b=LhfELiFoeniyiQwxEwsXQzkFSSv07UTgrt+GKxg4PGOfEMHwpgDlcHgs4moskaLN6a PPmrJDWgs0jKB2fnIx9U5v3eouLcMw1aV0c0Il5WWNOwPrTsCuzymvGCwVMGR2GXaQ1x 7lNHU+I1tytSRWBPosdDZFGjIdbqEuh6AQaDOqhIuyVkIIpKWJWhC6gSMJKQCA3zXfet Q5FnUeXuiIgv/JPA8UEQqQJmRa33v2M6YvKi75t6s5qXtkoUed092vkCcMIiQ/4JSwpP w7eEvehPtil/5MO0FRP9ZsVLqyovSMc8ZNAYzbtn47M5tHMbwTmqECHDcok+0CgVjLxs SSAw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709740870; x=1710345670; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=LnhY5lGlh/lAm5l28CdAlmLkBgaxlmQl8oMh/F4izdc=; b=LXt6LyqZlkHBIoPw+Ji1mqPzBI716qJZJd4cLMoeZltiny3VqG/oohDLX9l3pfJLxD +kpGxLU32o6kaYlba2mq0HiTG1+NJ+RkRRT5zhUpQysl3NLxOtofndO0ZuSyDI/xWhXb Voyt5t4RBK3Tyd2rGIezJWrKwGKINGI57qW5t8lZnKPURlhsaI3FagajiyLtvxjvEvbk yusrumIINS0mFeoKrKTJKpSXI3kY0Y7Sn+aKDaVj7mlyiwD/64k2If5FxJ5grPc00Cwv qQ+8VrRREN83HxEAx26Q1q/x68SaNa04TN9G+wgAL6hYZf3FQSmz7liPaBfwxkFiHLXg QkJA== X-Forwarded-Encrypted: i=1; AJvYcCXoHnBF1wYE/juZQaRkuAPHhc/Lm/d61DoR7CoIuNZT8v1vvy9WaH1wyL5Sz9HRtaDGRm+wx+/HlTOwIsR9yIdbswJpaRiy X-Gm-Message-State: AOJu0Yza6OBUfhFChl/VH3t1Qo3K+Q7o3tbYrOEwqY8YZHdyJ55PWBqG HHR6TuYaSuV5d8BH0U4QhUy5jPuqm+O8NScIy1UuSbAYD6u+WxWQDqxOmYJ6Rb3QRa63DCG7x4O gHXj89Q7TvQ== X-Google-Smtp-Source: AGHT+IEFmbNmVwkKKb2Iohq0f7Anpk/IY+rmFSWKLPFKKPNcKb63D2CEA7SfZLbeWnwIluEhsAZS9GWOnCL8Qw== X-Received: from edumazet1.c.googlers.com ([fda3:e722:ac3:cc00:2b:7d90:c0a8:395a]) (user=edumazet job=sendgmr) by 2002:a05:6902:f02:b0:dcc:53c6:1133 with SMTP id et2-20020a0569020f0200b00dcc53c61133mr498358ybb.13.1709740870717; Wed, 06 Mar 2024 08:01:10 -0800 (PST) Date: Wed, 6 Mar 2024 16:00:30 +0000 In-Reply-To: <20240306160031.874438-1-edumazet@google.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240306160031.874438-1-edumazet@google.com> X-Mailer: git-send-email 2.44.0.278.ge034bb2e1d-goog Message-ID: <20240306160031.874438-18-edumazet@google.com> Subject: [PATCH v2 net-next 17/18] net: introduce include/net/rps.h From: Eric Dumazet To: "David S . Miller" , Jakub Kicinski , Paolo Abeni Cc: David Ahern , Willem de Bruijn , Soheil Hassas Yeganeh , Neal Cardwell , netdev@vger.kernel.org, eric.dumazet@gmail.com, Eric Dumazet X-Patchwork-Delegate: kuba@kernel.org Move RPS related structures and helpers from include/linux/netdevice.h and include/net/sock.h to a new include file. Signed-off-by: Eric Dumazet Acked-by: Soheil Hassas Yeganeh Reviewed-by: David Ahern --- drivers/net/ethernet/intel/ice/ice_arfs.c | 1 + .../net/ethernet/mellanox/mlx4/en_netdev.c | 1 + .../net/ethernet/mellanox/mlx5/core/en_arfs.c | 1 + drivers/net/ethernet/sfc/rx_common.c | 1 + drivers/net/ethernet/sfc/siena/rx_common.c | 1 + drivers/net/tun.c | 1 + include/linux/netdevice.h | 82 ----------- include/net/rps.h | 127 ++++++++++++++++++ include/net/sock.h | 35 ----- net/core/dev.c | 1 + net/core/net-sysfs.c | 1 + net/core/sysctl_net_core.c | 1 + net/ipv4/af_inet.c | 1 + net/ipv4/tcp.c | 1 + net/ipv6/af_inet6.c | 1 + net/sctp/socket.c | 1 + 16 files changed, 140 insertions(+), 117 deletions(-) create mode 100644 include/net/rps.h diff --git a/drivers/net/ethernet/intel/ice/ice_arfs.c b/drivers/net/ethernet/intel/ice/ice_arfs.c index cca0e753f38ff92be24f8e8fc8963e2bf2416cfa..7cee365cc7d167865511c8b0dab08520814d4ec1 100644 --- a/drivers/net/ethernet/intel/ice/ice_arfs.c +++ b/drivers/net/ethernet/intel/ice/ice_arfs.c @@ -2,6 +2,7 @@ /* Copyright (C) 2018-2020, Intel Corporation. */ #include "ice.h" +#include /** * ice_is_arfs_active - helper to check is aRFS is active diff --git a/drivers/net/ethernet/mellanox/mlx4/en_netdev.c b/drivers/net/ethernet/mellanox/mlx4/en_netdev.c index d7da62cda821f70af530f95d23b63afd64423219..5d3fde63b273922f52f2466c7b339f3c9908e705 100644 --- a/drivers/net/ethernet/mellanox/mlx4/en_netdev.c +++ b/drivers/net/ethernet/mellanox/mlx4/en_netdev.c @@ -42,6 +42,7 @@ #include #include #include +#include #include #include diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_arfs.c b/drivers/net/ethernet/mellanox/mlx5/core/en_arfs.c index e66f486faafe1a6b0cfc75f0f11b2e957b040842..c7f542d0b8f08c635a6fad868a364a8f5f91ba8c 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en_arfs.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_arfs.c @@ -34,6 +34,7 @@ #include #include #include +#include #include "en.h" #define ARFS_HASH_SHIFT BITS_PER_BYTE diff --git a/drivers/net/ethernet/sfc/rx_common.c b/drivers/net/ethernet/sfc/rx_common.c index fac227d372db4c832a52a62144ff9c6c89995335..dcd901eccfc8f1f9c68ee391de548a2c1602ab94 100644 --- a/drivers/net/ethernet/sfc/rx_common.c +++ b/drivers/net/ethernet/sfc/rx_common.c @@ -11,6 +11,7 @@ #include "net_driver.h" #include #include +#include #include "efx.h" #include "nic.h" #include "rx_common.h" diff --git a/drivers/net/ethernet/sfc/siena/rx_common.c b/drivers/net/ethernet/sfc/siena/rx_common.c index 4579f43484c3675963de1ad8c7753a200c680fe1..219fb358a646d399dbd3c66cc34039f70bbd7341 100644 --- a/drivers/net/ethernet/sfc/siena/rx_common.c +++ b/drivers/net/ethernet/sfc/siena/rx_common.c @@ -11,6 +11,7 @@ #include "net_driver.h" #include #include +#include #include "efx.h" #include "nic.h" #include "rx_common.h" diff --git a/drivers/net/tun.c b/drivers/net/tun.c index 8d258e263f548da9332afeb2f8a1bd6b07f72db1..0b3f21cba552f27221a2c7fbe8147feb34e69724 100644 --- a/drivers/net/tun.c +++ b/drivers/net/tun.c @@ -78,6 +78,7 @@ #include #include #include +#include #include #include diff --git a/include/linux/netdevice.h b/include/linux/netdevice.h index d3d4d1052ecb0bc704e9bb40fffbea85cfb6ab03..aaacf86df56ab480485b34b19fecebfd00c34c59 100644 --- a/include/linux/netdevice.h +++ b/include/linux/netdevice.h @@ -227,12 +227,6 @@ struct net_device_core_stats { #include #include -#ifdef CONFIG_RPS -#include -extern struct static_key_false rps_needed; -extern struct static_key_false rfs_needed; -#endif - struct neighbour; struct neigh_parms; struct sk_buff; @@ -732,86 +726,10 @@ static inline void netdev_queue_numa_node_write(struct netdev_queue *q, int node #endif } -#ifdef CONFIG_RPS -/* - * This structure holds an RPS map which can be of variable length. The - * map is an array of CPUs. - */ -struct rps_map { - unsigned int len; - struct rcu_head rcu; - u16 cpus[]; -}; -#define RPS_MAP_SIZE(_num) (sizeof(struct rps_map) + ((_num) * sizeof(u16))) - -/* - * The rps_dev_flow structure contains the mapping of a flow to a CPU, the - * tail pointer for that CPU's input queue at the time of last enqueue, and - * a hardware filter index. - */ -struct rps_dev_flow { - u16 cpu; - u16 filter; - unsigned int last_qtail; -}; -#define RPS_NO_FILTER 0xffff - -/* - * The rps_dev_flow_table structure contains a table of flow mappings. - */ -struct rps_dev_flow_table { - unsigned int mask; - struct rcu_head rcu; - struct rps_dev_flow flows[]; -}; -#define RPS_DEV_FLOW_TABLE_SIZE(_num) (sizeof(struct rps_dev_flow_table) + \ - ((_num) * sizeof(struct rps_dev_flow))) - -/* - * The rps_sock_flow_table contains mappings of flows to the last CPU - * on which they were processed by the application (set in recvmsg). - * Each entry is a 32bit value. Upper part is the high-order bits - * of flow hash, lower part is CPU number. - * rps_cpu_mask is used to partition the space, depending on number of - * possible CPUs : rps_cpu_mask = roundup_pow_of_two(nr_cpu_ids) - 1 - * For example, if 64 CPUs are possible, rps_cpu_mask = 0x3f, - * meaning we use 32-6=26 bits for the hash. - */ -struct rps_sock_flow_table { - u32 mask; - - u32 ents[] ____cacheline_aligned_in_smp; -}; -#define RPS_SOCK_FLOW_TABLE_SIZE(_num) (offsetof(struct rps_sock_flow_table, ents[_num])) - -#define RPS_NO_CPU 0xffff - -extern u32 rps_cpu_mask; -extern struct rps_sock_flow_table __rcu *rps_sock_flow_table; - -static inline void rps_record_sock_flow(struct rps_sock_flow_table *table, - u32 hash) -{ - if (table && hash) { - unsigned int index = hash & table->mask; - u32 val = hash & ~rps_cpu_mask; - - /* We only give a hint, preemption can change CPU under us */ - val |= raw_smp_processor_id(); - - /* The following WRITE_ONCE() is paired with the READ_ONCE() - * here, and another one in get_rps_cpu(). - */ - if (READ_ONCE(table->ents[index]) != val) - WRITE_ONCE(table->ents[index], val); - } -} - #ifdef CONFIG_RFS_ACCEL bool rps_may_expire_flow(struct net_device *dev, u16 rxq_index, u32 flow_id, u16 filter_id); #endif -#endif /* CONFIG_RPS */ /* XPS map type and offset of the xps map within net_device->xps_maps[]. */ enum xps_map_type { diff --git a/include/net/rps.h b/include/net/rps.h new file mode 100644 index 0000000000000000000000000000000000000000..6081d817d2458b7b34036d87fbdef3fa6dc914ea --- /dev/null +++ b/include/net/rps.h @@ -0,0 +1,127 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ +#ifndef _NET_RPS_H +#define _NET_RPS_H + +#include +#include +#include + +#ifdef CONFIG_RPS + +extern struct static_key_false rps_needed; +extern struct static_key_false rfs_needed; + +/* + * This structure holds an RPS map which can be of variable length. The + * map is an array of CPUs. + */ +struct rps_map { + unsigned int len; + struct rcu_head rcu; + u16 cpus[]; +}; +#define RPS_MAP_SIZE(_num) (sizeof(struct rps_map) + ((_num) * sizeof(u16))) + +/* + * The rps_dev_flow structure contains the mapping of a flow to a CPU, the + * tail pointer for that CPU's input queue at the time of last enqueue, and + * a hardware filter index. + */ +struct rps_dev_flow { + u16 cpu; + u16 filter; + unsigned int last_qtail; +}; +#define RPS_NO_FILTER 0xffff + +/* + * The rps_dev_flow_table structure contains a table of flow mappings. + */ +struct rps_dev_flow_table { + unsigned int mask; + struct rcu_head rcu; + struct rps_dev_flow flows[]; +}; +#define RPS_DEV_FLOW_TABLE_SIZE(_num) (sizeof(struct rps_dev_flow_table) + \ + ((_num) * sizeof(struct rps_dev_flow))) + +/* + * The rps_sock_flow_table contains mappings of flows to the last CPU + * on which they were processed by the application (set in recvmsg). + * Each entry is a 32bit value. Upper part is the high-order bits + * of flow hash, lower part is CPU number. + * rps_cpu_mask is used to partition the space, depending on number of + * possible CPUs : rps_cpu_mask = roundup_pow_of_two(nr_cpu_ids) - 1 + * For example, if 64 CPUs are possible, rps_cpu_mask = 0x3f, + * meaning we use 32-6=26 bits for the hash. + */ +struct rps_sock_flow_table { + u32 mask; + + u32 ents[] ____cacheline_aligned_in_smp; +}; +#define RPS_SOCK_FLOW_TABLE_SIZE(_num) (offsetof(struct rps_sock_flow_table, ents[_num])) + +#define RPS_NO_CPU 0xffff + +extern u32 rps_cpu_mask; +extern struct rps_sock_flow_table __rcu *rps_sock_flow_table; + +static inline void rps_record_sock_flow(struct rps_sock_flow_table *table, + u32 hash) +{ + unsigned int index = hash & table->mask; + u32 val = hash & ~rps_cpu_mask; + + /* We only give a hint, preemption can change CPU under us */ + val |= raw_smp_processor_id(); + + /* The following WRITE_ONCE() is paired with the READ_ONCE() + * here, and another one in get_rps_cpu(). + */ + if (READ_ONCE(table->ents[index]) != val) + WRITE_ONCE(table->ents[index], val); +} + +#endif /* CONFIG_RPS */ + +static inline void sock_rps_record_flow_hash(__u32 hash) +{ +#ifdef CONFIG_RPS + struct rps_sock_flow_table *sock_flow_table; + + if (!hash) + return; + rcu_read_lock(); + sock_flow_table = rcu_dereference(rps_sock_flow_table); + if (sock_flow_table) + rps_record_sock_flow(sock_flow_table, hash); + rcu_read_unlock(); +#endif +} + +static inline void sock_rps_record_flow(const struct sock *sk) +{ +#ifdef CONFIG_RPS + if (static_branch_unlikely(&rfs_needed)) { + /* Reading sk->sk_rxhash might incur an expensive cache line + * miss. + * + * TCP_ESTABLISHED does cover almost all states where RFS + * might be useful, and is cheaper [1] than testing : + * IPv4: inet_sk(sk)->inet_daddr + * IPv6: ipv6_addr_any(&sk->sk_v6_daddr) + * OR an additional socket flag + * [1] : sk_state and sk_prot are in the same cache line. + */ + if (sk->sk_state == TCP_ESTABLISHED) { + /* This READ_ONCE() is paired with the WRITE_ONCE() + * from sock_rps_save_rxhash() and sock_rps_reset_rxhash(). + */ + sock_rps_record_flow_hash(READ_ONCE(sk->sk_rxhash)); + } + } +#endif +} + +#endif /* _NET_RPS_H */ diff --git a/include/net/sock.h b/include/net/sock.h index 09a0cde8bf52286167f3b9e9b32f632f3d1b5487..b5e00702acc1f037df7eb8ad085d00e0b18079a8 100644 --- a/include/net/sock.h +++ b/include/net/sock.h @@ -1117,41 +1117,6 @@ static inline void sk_incoming_cpu_update(struct sock *sk) WRITE_ONCE(sk->sk_incoming_cpu, cpu); } -static inline void sock_rps_record_flow_hash(__u32 hash) -{ -#ifdef CONFIG_RPS - struct rps_sock_flow_table *sock_flow_table; - - rcu_read_lock(); - sock_flow_table = rcu_dereference(rps_sock_flow_table); - rps_record_sock_flow(sock_flow_table, hash); - rcu_read_unlock(); -#endif -} - -static inline void sock_rps_record_flow(const struct sock *sk) -{ -#ifdef CONFIG_RPS - if (static_branch_unlikely(&rfs_needed)) { - /* Reading sk->sk_rxhash might incur an expensive cache line - * miss. - * - * TCP_ESTABLISHED does cover almost all states where RFS - * might be useful, and is cheaper [1] than testing : - * IPv4: inet_sk(sk)->inet_daddr - * IPv6: ipv6_addr_any(&sk->sk_v6_daddr) - * OR an additional socket flag - * [1] : sk_state and sk_prot are in the same cache line. - */ - if (sk->sk_state == TCP_ESTABLISHED) { - /* This READ_ONCE() is paired with the WRITE_ONCE() - * from sock_rps_save_rxhash() and sock_rps_reset_rxhash(). - */ - sock_rps_record_flow_hash(READ_ONCE(sk->sk_rxhash)); - } - } -#endif -} static inline void sock_rps_save_rxhash(struct sock *sk, const struct sk_buff *skb) diff --git a/net/core/dev.c b/net/core/dev.c index 26676dbd6c8594371a4d30e0dd95d72342e226ca..e9f24a31ae121f713e6ef5a530a65218bbb457e8 100644 --- a/net/core/dev.c +++ b/net/core/dev.c @@ -155,6 +155,7 @@ #include #include #include +#include #include "dev.h" #include "net-sysfs.h" diff --git a/net/core/net-sysfs.c b/net/core/net-sysfs.c index af238026ac3c6d81267bf0b53a7b240ee9ba32b1..5560083774b1ad4e2612637990dfa3f80f750c83 100644 --- a/net/core/net-sysfs.c +++ b/net/core/net-sysfs.c @@ -24,6 +24,7 @@ #include #include #include +#include #include "dev.h" #include "net-sysfs.h" diff --git a/net/core/sysctl_net_core.c b/net/core/sysctl_net_core.c index 8a4c698dad9c97636ca9cebfad925d4220e98f2a..4b93e27404e83a5b3afaa23ebd18cf55b1fdc6e8 100644 --- a/net/core/sysctl_net_core.c +++ b/net/core/sysctl_net_core.c @@ -24,6 +24,7 @@ #include #include #include +#include #include "dev.h" diff --git a/net/ipv4/af_inet.c b/net/ipv4/af_inet.c index 6f1cfd176e7b84f23d8a5e505bf8e13b2b755f06..55bd72997b31063b7baad350fdcff40e938aecb8 100644 --- a/net/ipv4/af_inet.c +++ b/net/ipv4/af_inet.c @@ -119,6 +119,7 @@ #endif #include #include +#include #include diff --git a/net/ipv4/tcp.c b/net/ipv4/tcp.c index 7e1b848398d04f2da2a91c3af97b1e2e3895b8ee..c5b83875411aee6037d0afd060eb9d3e16f8b6b0 100644 --- a/net/ipv4/tcp.c +++ b/net/ipv4/tcp.c @@ -279,6 +279,7 @@ #include #include #include +#include /* Track pending CMSGs. */ enum { diff --git a/net/ipv6/af_inet6.c b/net/ipv6/af_inet6.c index b90d46533cdcc1ffb61ca483e6f67ab358ede55c..8041dc181bd42e5e1af2d9e9fe6af50057e5b58f 100644 --- a/net/ipv6/af_inet6.c +++ b/net/ipv6/af_inet6.c @@ -64,6 +64,7 @@ #include #include #include +#include #include #include diff --git a/net/sctp/socket.c b/net/sctp/socket.c index 6b9fcdb0952a0fe599ae5d1d1cc6fa9557a3a3bc..c67679a41044fc8e801d175b235249f2c8b99dc0 100644 --- a/net/sctp/socket.c +++ b/net/sctp/socket.c @@ -67,6 +67,7 @@ #include #include #include +#include /* Forward declarations for internal helper functions. */ static bool sctp_writeable(const struct sock *sk); From patchwork Wed Mar 6 16:00:31 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Dumazet X-Patchwork-Id: 13584266 X-Patchwork-Delegate: kuba@kernel.org Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 68A5E13E7CD for ; Wed, 6 Mar 2024 16:01:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709740875; cv=none; b=EFve8HZUPUUNS++XNGPFqGHHI0hiFB3eX0MYLekQfE+TzBh0AMeoIveBWPeIa8fDcXjUkpjMvH6RQ8u6gm2hUWmzb45BCCbTCqauiBwzWh5CmdPrfKza1Fp1B9CYL2YfztXfPOUyMlj03SGVST1isfe8fk1SH5rAPVZa+Xa/xZo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709740875; c=relaxed/simple; bh=mixqPoC7wBm3Hfn9Y8bfZrYiNcXavH1UGM4b4KcTeN4=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=ldD0h5f7GOAElt1l5fVZeJKjArz6cM5eW3XHp/AhVQz063gUUgWv/5KQbqq3qIlMxT9YpheJuprRfn32M2bY4DKowpHYb6CkE6CemgcWZPkXfkmHf1TLbXihp7XQg0R3oo+O6vi0lcWBZSEeFdzU97C1fmUtQee1ZS8msEliwJk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--edumazet.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=yvNqna4a; arc=none smtp.client-ip=209.85.219.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--edumazet.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="yvNqna4a" Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-dc6b5d1899eso1384724276.0 for ; Wed, 06 Mar 2024 08:01:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1709740872; x=1710345672; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=yZqZtL36XlZ0L93IPccEZyr0UkHGWH0yynvNuk+9H1s=; b=yvNqna4aW5Qo+QP9mNqvUfbVLZyzH9RPDUcu6CH3LirtZdMBhDGSXZ06Zk7IO86nzL x+XgeNKar54U3eeCKDMCr5AFFtEZ+XHGnOmMUozT0pbJuYlpI3FQdRiRvY1K+cvwViX9 2fq60Hvh9V7qfMI49oT70udAtLhL3/aan5ngfrFIOwMHp8TrRpt8sKLKNcdNvkhzq4dL 1fiyZNpXiHWkaQk2E746jsWOhP9k6A0w2nonBSKLgimog1FEjwpMjZyJ+LSjUhNqcIp+ GPWjPg3cQ6nAGT5oT5LxPvipUPy+vxH87qnq6MEeM9L425iZCjGU2k7uYPesA5gHSarA 7Y8w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709740872; x=1710345672; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=yZqZtL36XlZ0L93IPccEZyr0UkHGWH0yynvNuk+9H1s=; b=jwrfOfnceuUDIyJklxiqEMrfJP92nfkKT/RYBrB/0a//SYmHNtwD9394CcVr1Tfsdz LeRJQXr5mbgQQ11wiGio69Ixo0+sDo6hvA7nBbJeYz5u6BRIS+7yyxrt4Xj5ntkv0cY7 IsYgD8p4MV0djl9YwfjoH817NIJ3qn75VJgaxw8EgGjQNd2gcJFIEvkyKt+8TbDAc+OD FDzCnTWxqUt+DrvUi7SyGn4F8XsWoQwwHih5OjElkzfEfpui00hb6hsBZ7KVcv7FlUuZ /uFTtvM3zuyvfle7cX7wK8NPMkYEVh7Yuw47OIKSFKq35ZtHDS6ZMBOIEB/hd2GQlkkO iBWw== X-Forwarded-Encrypted: i=1; AJvYcCVAQwh8YMRcVSq2xerL6hr+gmXmK7CpyM8/JEvBQGxU0rxC+jF8Qp3OiEn+39OdkDw4jOgX69LiobIS8cTGsyEmWWY+/wtt X-Gm-Message-State: AOJu0Yz+NnJz7VsFluLqXaYtLLIfBowBbmOLdaeWlVJWrJ95KBDS4eR3 iABImlXpe+BYuOlF1gHMc8otaNjCApFm4laDTvI89xZms/K4SqSzf+oQzQLmekeeWKVMwR/6G/K ndywYNSMK+Q== X-Google-Smtp-Source: AGHT+IF0Q/aIuW1PYjs+Eu4RDg3Qvm/kVHGsiW5z21Ud90L5/sSOHv1QGWf/8uqbqtWhcYsdZu+uCya3eoyI/g== X-Received: from edumazet1.c.googlers.com ([fda3:e722:ac3:cc00:2b:7d90:c0a8:395a]) (user=edumazet job=sendgmr) by 2002:a5b:54c:0:b0:dc7:5925:92d2 with SMTP id r12-20020a5b054c000000b00dc7592592d2mr1157339ybp.1.1709740872542; Wed, 06 Mar 2024 08:01:12 -0800 (PST) Date: Wed, 6 Mar 2024 16:00:31 +0000 In-Reply-To: <20240306160031.874438-1-edumazet@google.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240306160031.874438-1-edumazet@google.com> X-Mailer: git-send-email 2.44.0.278.ge034bb2e1d-goog Message-ID: <20240306160031.874438-19-edumazet@google.com> Subject: [PATCH v2 net-next 18/18] net: move rps_sock_flow_table to net_hotdata From: Eric Dumazet To: "David S . Miller" , Jakub Kicinski , Paolo Abeni Cc: David Ahern , Willem de Bruijn , Soheil Hassas Yeganeh , Neal Cardwell , netdev@vger.kernel.org, eric.dumazet@gmail.com, Eric Dumazet X-Patchwork-Delegate: kuba@kernel.org rps_sock_flow_table and rps_cpu_mask are used in fast path. Move them to net_hotdata for better cache locality. Signed-off-by: Eric Dumazet Acked-by: Soheil Hassas Yeganeh Reviewed-by: David Ahern --- include/net/hotdata.h | 4 ++++ include/net/rps.h | 8 +++----- net/core/dev.c | 12 +++--------- net/core/sysctl_net_core.c | 9 ++++++--- 4 files changed, 16 insertions(+), 17 deletions(-) diff --git a/include/net/hotdata.h b/include/net/hotdata.h index b0b847585f7e62245cee81a56b5a252051e07834..003667a1efd6b63fc0f0d7cadd2c8472281331b0 100644 --- a/include/net/hotdata.h +++ b/include/net/hotdata.h @@ -27,6 +27,10 @@ struct net_hotdata { struct kmem_cache *skbuff_cache; struct kmem_cache *skbuff_fclone_cache; struct kmem_cache *skb_small_head_cache; +#ifdef CONFIG_RPS + struct rps_sock_flow_table __rcu *rps_sock_flow_table; + u32 rps_cpu_mask; +#endif int gro_normal_batch; int netdev_budget; int netdev_budget_usecs; diff --git a/include/net/rps.h b/include/net/rps.h index 6081d817d2458b7b34036d87fbdef3fa6dc914ea..7660243e905b92651a41292e04caf72c5f12f26e 100644 --- a/include/net/rps.h +++ b/include/net/rps.h @@ -5,6 +5,7 @@ #include #include #include +#include #ifdef CONFIG_RPS @@ -64,14 +65,11 @@ struct rps_sock_flow_table { #define RPS_NO_CPU 0xffff -extern u32 rps_cpu_mask; -extern struct rps_sock_flow_table __rcu *rps_sock_flow_table; - static inline void rps_record_sock_flow(struct rps_sock_flow_table *table, u32 hash) { unsigned int index = hash & table->mask; - u32 val = hash & ~rps_cpu_mask; + u32 val = hash & ~net_hotdata.rps_cpu_mask; /* We only give a hint, preemption can change CPU under us */ val |= raw_smp_processor_id(); @@ -93,7 +91,7 @@ static inline void sock_rps_record_flow_hash(__u32 hash) if (!hash) return; rcu_read_lock(); - sock_flow_table = rcu_dereference(rps_sock_flow_table); + sock_flow_table = rcu_dereference(net_hotdata.rps_sock_flow_table); if (sock_flow_table) rps_record_sock_flow(sock_flow_table, hash); rcu_read_unlock(); diff --git a/net/core/dev.c b/net/core/dev.c index e9f24a31ae121f713e6ef5a530a65218bbb457e8..9f54d8e61ec0f4a4ad1824e333f89ad08c5ec431 100644 --- a/net/core/dev.c +++ b/net/core/dev.c @@ -4450,12 +4450,6 @@ static inline void ____napi_schedule(struct softnet_data *sd, #ifdef CONFIG_RPS -/* One global table that all flow-based protocols share. */ -struct rps_sock_flow_table __rcu *rps_sock_flow_table __read_mostly; -EXPORT_SYMBOL(rps_sock_flow_table); -u32 rps_cpu_mask __read_mostly; -EXPORT_SYMBOL(rps_cpu_mask); - struct static_key_false rps_needed __read_mostly; EXPORT_SYMBOL(rps_needed); struct static_key_false rfs_needed __read_mostly; @@ -4547,7 +4541,7 @@ static int get_rps_cpu(struct net_device *dev, struct sk_buff *skb, if (!hash) goto done; - sock_flow_table = rcu_dereference(rps_sock_flow_table); + sock_flow_table = rcu_dereference(net_hotdata.rps_sock_flow_table); if (flow_table && sock_flow_table) { struct rps_dev_flow *rflow; u32 next_cpu; @@ -4557,10 +4551,10 @@ static int get_rps_cpu(struct net_device *dev, struct sk_buff *skb, * This READ_ONCE() pairs with WRITE_ONCE() from rps_record_sock_flow(). */ ident = READ_ONCE(sock_flow_table->ents[hash & sock_flow_table->mask]); - if ((ident ^ hash) & ~rps_cpu_mask) + if ((ident ^ hash) & ~net_hotdata.rps_cpu_mask) goto try_rps; - next_cpu = ident & rps_cpu_mask; + next_cpu = ident & net_hotdata.rps_cpu_mask; /* OK, now we know there is a match, * we can look at the local (per receive queue) flow table diff --git a/net/core/sysctl_net_core.c b/net/core/sysctl_net_core.c index 4b93e27404e83a5b3afaa23ebd18cf55b1fdc6e8..6973dda3abda63e0924efa4b6b7026786e8bfb4f 100644 --- a/net/core/sysctl_net_core.c +++ b/net/core/sysctl_net_core.c @@ -140,7 +140,8 @@ static int rps_sock_flow_sysctl(struct ctl_table *table, int write, mutex_lock(&sock_flow_mutex); - orig_sock_table = rcu_dereference_protected(rps_sock_flow_table, + orig_sock_table = rcu_dereference_protected( + net_hotdata.rps_sock_flow_table, lockdep_is_held(&sock_flow_mutex)); size = orig_size = orig_sock_table ? orig_sock_table->mask + 1 : 0; @@ -161,7 +162,8 @@ static int rps_sock_flow_sysctl(struct ctl_table *table, int write, mutex_unlock(&sock_flow_mutex); return -ENOMEM; } - rps_cpu_mask = roundup_pow_of_two(nr_cpu_ids) - 1; + net_hotdata.rps_cpu_mask = + roundup_pow_of_two(nr_cpu_ids) - 1; sock_table->mask = size - 1; } else sock_table = orig_sock_table; @@ -172,7 +174,8 @@ static int rps_sock_flow_sysctl(struct ctl_table *table, int write, sock_table = NULL; if (sock_table != orig_sock_table) { - rcu_assign_pointer(rps_sock_flow_table, sock_table); + rcu_assign_pointer(net_hotdata.rps_sock_flow_table, + sock_table); if (sock_table) { static_branch_inc(&rps_needed); static_branch_inc(&rfs_needed);