From patchwork Mon Mar 25 20:47:34 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Asbj=C3=B8rn_Sloth_T=C3=B8nnesen?= X-Patchwork-Id: 13602960 X-Patchwork-Delegate: kuba@kernel.org Received: from mail1.fiberby.net (mail1.fiberby.net [193.104.135.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4B2F55D723; Mon, 25 Mar 2024 20:54:07 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=193.104.135.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711400052; cv=none; b=Zp0s4N7P99Q14FT93ocxWEZWi8Vfkvr9R7E2IgyFZtZJmUeohM5TMvk/8buJfdj5bp31RcLV8B6OObHpVK9GVBkQ0fucFBSQVgc32FC2oIaDlvbcfFZvJFPbrB9/umY8PDvcONSbrWkdXekvxlTOzSCYNi2qw/x2fUsAgOe2QN8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711400052; c=relaxed/simple; bh=hZuTiHOZSNOnV/RnssrMjJTe2eOUX3VXDysAnQilAAc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=LmNiGdMX1UT1FtEmEUIQnnTtC7+OTTKcTQORt9NID7Z/mb3DecAHvxB/VdSPAjoTiJhG+ldd6Z6zuc5falI9qnLGSe1PffJUlHj7cVkQcp3odcXjQVczNG9ebEzHZQpPzTMtbQvMP6bDmbwf2BJUGBDqxMbOJLqSUWsCYGXFfo4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=fiberby.net; spf=pass smtp.mailfrom=fiberby.net; dkim=pass (2048-bit key) header.d=fiberby.net header.i=@fiberby.net header.b=EiEFkAC9; arc=none smtp.client-ip=193.104.135.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=fiberby.net Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=fiberby.net Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=fiberby.net header.i=@fiberby.net header.b="EiEFkAC9" Received: from x201s (193-104-135-243.ip4.fiberby.net [193.104.135.243]) by mail1.fiberby.net (Postfix) with ESMTPSA id 5EF666030B; Mon, 25 Mar 2024 20:48:34 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=fiberby.net; s=202008; t=1711399726; bh=hZuTiHOZSNOnV/RnssrMjJTe2eOUX3VXDysAnQilAAc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=EiEFkAC9JMQRewKe1gebFBf1DFr8ZhxyPvG9Gla/eJebzEFSKItg2mhOIO/y9FDOr emuoZL3RaN8zuI1wmZWMO+TRK+LJpQzTEj2GucbRf2k0ghT4rNAaCrplPj6eYEIQRJ 8q89Jq+OdSM+3rWswmtDGa8ChcZiu7oH9Htpd9ZUlbNU8ETTE39hFAQ2hlt8KpWlCz zQg5fn86y/+tZ6ekGEaoWJs2jlf/sfzAdA2ibLNX//jhuW1C3uoe6ZlCkUO57j8phZ XJcmtjXbPt8ohByShadkmcnXmn3ZzgB7+wM4utfEv7fgt9Cp0m0Z97pgdlUVcnpPIH S5tUv7AzEJttQ== Received: by x201s (Postfix, from userid 1000) id 6DD8F204AFD; Mon, 25 Mar 2024 20:47:50 +0000 (UTC) From: =?utf-8?q?Asbj=C3=B8rn_Sloth_T=C3=B8nnesen?= To: Jamal Hadi Salim , Cong Wang , Jiri Pirko Cc: =?utf-8?q?Asbj=C3=B8rn_Sloth_T=C3=B8nnesen?= , Daniel Borkmann , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Vlad Buslov , Marcelo Ricardo Leitner , netdev@vger.kernel.org, linux-kernel@vger.kernel.org, llu@fiberby.dk, Jiri Pirko Subject: [PATCH net-next v4 1/3] net: sched: cls_api: add skip_sw counter Date: Mon, 25 Mar 2024 20:47:34 +0000 Message-ID: <20240325204740.1393349-2-ast@fiberby.net> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240325204740.1393349-1-ast@fiberby.net> References: <20240325204740.1393349-1-ast@fiberby.net> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: kuba@kernel.org Maintain a count of skip_sw filters. This counter is protected by the cb_lock, and is updated at the same time as offloadcnt. Signed-off-by: Asbjørn Sloth Tønnesen Reviewed-by: Jiri Pirko Reviewed-by: Simon Horman Reviewed-by: Marcelo Ricardo Leitner --- include/net/sch_generic.h | 1 + net/sched/cls_api.c | 4 ++++ 2 files changed, 5 insertions(+) diff --git a/include/net/sch_generic.h b/include/net/sch_generic.h index cefe0c4bdae3..120a4ca6ec9b 100644 --- a/include/net/sch_generic.h +++ b/include/net/sch_generic.h @@ -471,6 +471,7 @@ struct tcf_block { struct flow_block flow_block; struct list_head owner_list; bool keep_dst; + atomic_t skipswcnt; /* Number of skip_sw filters */ atomic_t offloadcnt; /* Number of oddloaded filters */ unsigned int nooffloaddevcnt; /* Number of devs unable to do offload */ unsigned int lockeddevcnt; /* Number of devs that require rtnl lock. */ diff --git a/net/sched/cls_api.c b/net/sched/cls_api.c index ca5676b2668e..397c3d29659c 100644 --- a/net/sched/cls_api.c +++ b/net/sched/cls_api.c @@ -3483,6 +3483,8 @@ static void tcf_block_offload_inc(struct tcf_block *block, u32 *flags) if (*flags & TCA_CLS_FLAGS_IN_HW) return; *flags |= TCA_CLS_FLAGS_IN_HW; + if (tc_skip_sw(*flags)) + atomic_inc(&block->skipswcnt); atomic_inc(&block->offloadcnt); } @@ -3491,6 +3493,8 @@ static void tcf_block_offload_dec(struct tcf_block *block, u32 *flags) if (!(*flags & TCA_CLS_FLAGS_IN_HW)) return; *flags &= ~TCA_CLS_FLAGS_IN_HW; + if (tc_skip_sw(*flags)) + atomic_dec(&block->skipswcnt); atomic_dec(&block->offloadcnt); } From patchwork Mon Mar 25 20:47:35 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Asbj=C3=B8rn_Sloth_T=C3=B8nnesen?= X-Patchwork-Id: 13602958 X-Patchwork-Delegate: kuba@kernel.org Received: from mail1.fiberby.net (mail1.fiberby.net [193.104.135.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DB9AA5C613; Mon, 25 Mar 2024 20:54:06 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=193.104.135.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711400050; cv=none; b=A3uatDzxX1FAf6r2I6Pgx2z7PGzBef8dZHVm7FpyEXUl+vmjcmFAsMhZDWTDX5GvSAQ4WMYVF39x1czx425mSeh3Pt9d5ZWYiY8yqsY4dPk/tO6FpHT9NeiH5gyhBGB8TAzuq2Z5WSaQNdLSt3kJCvXMkj/aGwPw0n31EXWOySo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711400050; c=relaxed/simple; bh=UsuadOPE5XUQNgscef/NyQQnyL0c3yDFZvzX9R4Pwe8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=o3XjHF669geeudnvbuALWxkvIzzuQu3e735BEGHpfQ7KnbeL4TRcnOqMh7ZIVkehSzAxxxTuvm5h5THZkY9swWuFrPoLDwhhPktgiwt5Zqf3uCmdi2L/8vtS/k7AlaDnvzuYrMXuplakK+bs6n4ZXBC1ZaD78WhCwsRGPDxH0lY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=fiberby.net; spf=pass smtp.mailfrom=fiberby.net; dkim=pass (2048-bit key) header.d=fiberby.net header.i=@fiberby.net header.b=CL+8RpkF; arc=none smtp.client-ip=193.104.135.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=fiberby.net Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=fiberby.net Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=fiberby.net header.i=@fiberby.net header.b="CL+8RpkF" Received: from x201s (193-104-135-243.ip4.fiberby.net [193.104.135.243]) by mail1.fiberby.net (Postfix) with ESMTPSA id 5C71D6030A; Mon, 25 Mar 2024 20:48:34 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=fiberby.net; s=202008; t=1711399726; bh=UsuadOPE5XUQNgscef/NyQQnyL0c3yDFZvzX9R4Pwe8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=CL+8RpkFaNKXpH6sJVL24f37GOWJ5scHuI+tTnaxOOoWQZrF42pm8EzjKbt4LAuzW iH3mVd3X+lPa/P/qt1K0TK32q9OMzxX7A6MFXR9nX5Q+042UswnNLwcbFgDursA8L8 Jp6il8ivBKqZ8r9G8G66Y+H0IhpeRZVQNvWo+uw269/04YqCU6iGo4bFc6u0ptNYFd 2kadXIv3B9x3n5iL11H47jfpzrt0hzC34uFDDAGjDBfN6T0WNDt59tuKrBANeNA/xe WV8FsEDVHWLe7kpYWwAvDxL/DYWaWE1AbsAbv24327Gmc0rFyEIIMvbBqn4ep7ljYL 7sPgy1gTwKHLg== Received: by x201s (Postfix, from userid 1000) id C7CB8204BA9; Mon, 25 Mar 2024 20:47:51 +0000 (UTC) From: =?utf-8?q?Asbj=C3=B8rn_Sloth_T=C3=B8nnesen?= To: Jamal Hadi Salim , Cong Wang , Jiri Pirko Cc: =?utf-8?q?Asbj=C3=B8rn_Sloth_T=C3=B8nnesen?= , Daniel Borkmann , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Vlad Buslov , Marcelo Ricardo Leitner , netdev@vger.kernel.org, linux-kernel@vger.kernel.org, llu@fiberby.dk Subject: [PATCH net-next v4 2/3] net: sched: cls_api: add filter counter Date: Mon, 25 Mar 2024 20:47:35 +0000 Message-ID: <20240325204740.1393349-3-ast@fiberby.net> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240325204740.1393349-1-ast@fiberby.net> References: <20240325204740.1393349-1-ast@fiberby.net> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: kuba@kernel.org Maintain a count of filters per block. Counter updates are protected by cb_lock, which is also used to protect the offload counters. Signed-off-by: Asbjørn Sloth Tønnesen Reviewed-by: Simon Horman Reviewed-by: Marcelo Ricardo Leitner --- include/net/sch_generic.h | 2 ++ net/sched/cls_api.c | 19 +++++++++++++++++++ 2 files changed, 21 insertions(+) diff --git a/include/net/sch_generic.h b/include/net/sch_generic.h index 120a4ca6ec9b..eb3872c22fcd 100644 --- a/include/net/sch_generic.h +++ b/include/net/sch_generic.h @@ -422,6 +422,7 @@ struct tcf_proto { */ spinlock_t lock; bool deleting; + bool counted; refcount_t refcnt; struct rcu_head rcu; struct hlist_node destroy_ht_node; @@ -471,6 +472,7 @@ struct tcf_block { struct flow_block flow_block; struct list_head owner_list; bool keep_dst; + atomic_t filtercnt; /* Number of filters */ atomic_t skipswcnt; /* Number of skip_sw filters */ atomic_t offloadcnt; /* Number of oddloaded filters */ unsigned int nooffloaddevcnt; /* Number of devs unable to do offload */ diff --git a/net/sched/cls_api.c b/net/sched/cls_api.c index 397c3d29659c..304a46ab0e0b 100644 --- a/net/sched/cls_api.c +++ b/net/sched/cls_api.c @@ -410,12 +410,30 @@ static void tcf_proto_get(struct tcf_proto *tp) refcount_inc(&tp->refcnt); } +static void tcf_block_filter_cnt_update(struct tcf_block *block, bool *counted, bool add) +{ + lockdep_assert_not_held(&block->cb_lock); + + down_write(&block->cb_lock); + if (*counted != add) { + if (add) { + atomic_inc(&block->filtercnt); + *counted = true; + } else { + atomic_dec(&block->filtercnt); + *counted = false; + } + } + up_write(&block->cb_lock); +} + static void tcf_chain_put(struct tcf_chain *chain); static void tcf_proto_destroy(struct tcf_proto *tp, bool rtnl_held, bool sig_destroy, struct netlink_ext_ack *extack) { tp->ops->destroy(tp, rtnl_held, extack); + tcf_block_filter_cnt_update(tp->chain->block, &tp->counted, false); if (sig_destroy) tcf_proto_signal_destroyed(tp->chain, tp); tcf_chain_put(tp->chain); @@ -2364,6 +2382,7 @@ static int tc_new_tfilter(struct sk_buff *skb, struct nlmsghdr *n, err = tp->ops->change(net, skb, tp, cl, t->tcm_handle, tca, &fh, flags, extack); if (err == 0) { + tcf_block_filter_cnt_update(block, &tp->counted, true); tfilter_notify(net, skb, n, tp, block, q, parent, fh, RTM_NEWTFILTER, false, rtnl_held, extack); tfilter_put(tp, fh); From patchwork Mon Mar 25 20:47:36 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Asbj=C3=B8rn_Sloth_T=C3=B8nnesen?= X-Patchwork-Id: 13602957 X-Patchwork-Delegate: kuba@kernel.org Received: from mail1.fiberby.net (mail1.fiberby.net [193.104.135.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DB9705C612; Mon, 25 Mar 2024 20:54:06 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=193.104.135.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711400050; cv=none; b=FRJ5RBTZiBDkwICWVGezEclpLHXOTU72TmenR3oiLW0hpzYwCEoLB0YkVo8S5IrVcdITKQav4YUASWbA07Daq+vX3U6UnRAlqPfqco6xwlTza1/2D+m+qfmykfggBwWn6tIl9rd13NsPFl72Z4lZK73/trfh+e0M8+cM6Vcejd8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711400050; c=relaxed/simple; bh=WMyHr/I//z+M/NWcBi3s3WXQw2/fQcwA/ZQ2ZXqHJMQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=WcJi0zaqySMlPY6EADBuMVbvy0Jt15JP1i2fMNWQ3h+iodg1Df4VZA5jOSEl+xLG4QaY4+FUK6vOtWs+rEJOwLbRT7SYsw4xkbUemY8oXncGj5bxo+mo6TtryEkE58wGfHIO0DUhZAHVOzbsK3X3AssnRnoT1qVwI/Dwrzl6iLI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=fiberby.net; spf=pass smtp.mailfrom=fiberby.net; dkim=pass (2048-bit key) header.d=fiberby.net header.i=@fiberby.net header.b=FTaqahHM; arc=none smtp.client-ip=193.104.135.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=fiberby.net Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=fiberby.net Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=fiberby.net header.i=@fiberby.net header.b="FTaqahHM" Received: from x201s (193-104-135-243.ip4.fiberby.net [193.104.135.243]) by mail1.fiberby.net (Postfix) with ESMTPSA id 59A14600D7; Mon, 25 Mar 2024 20:48:34 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=fiberby.net; s=202008; t=1711399726; bh=WMyHr/I//z+M/NWcBi3s3WXQw2/fQcwA/ZQ2ZXqHJMQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=FTaqahHMTSG/MyUQ2viKiLFXCx8yaOAyDWNXklBJTFUGNgXRxbPO4kbX97Sqn27Ax oQ0SH8agpr+LHBZtgD/wgaCG/HvUcV316RAdTRsaKSdNhFnnDWcj77DBA2v+7/+C20 WxT0+cQET03LIVOQM2BvBwxwlxqAubPTbAn+WmDcEcaMEMjdSAM8+KYUN/w5MwNhw7 YtK8Wm2S++cpinwOXP0oq/oDVviyvu8qHkO8J4KEpMopoUGhGoHelE56+zVIfCcF/X 6OjaPsdtBYLgkdwWgo16t9IEOud1Pt8x7Ej+++iLTTPWRpyu7cgCIaTtEkLxQWjDqS +1tHe7MqQbLMQ== Received: by x201s (Postfix, from userid 1000) id A26C6204BFA; Mon, 25 Mar 2024 20:47:52 +0000 (UTC) From: =?utf-8?q?Asbj=C3=B8rn_Sloth_T=C3=B8nnesen?= To: Jamal Hadi Salim , Cong Wang , Jiri Pirko Cc: =?utf-8?q?Asbj=C3=B8rn_Sloth_T=C3=B8nnesen?= , Daniel Borkmann , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Vlad Buslov , Marcelo Ricardo Leitner , netdev@vger.kernel.org, linux-kernel@vger.kernel.org, llu@fiberby.dk Subject: [PATCH net-next v4 3/3] net: sched: make skip_sw actually skip software Date: Mon, 25 Mar 2024 20:47:36 +0000 Message-ID: <20240325204740.1393349-4-ast@fiberby.net> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240325204740.1393349-1-ast@fiberby.net> References: <20240325204740.1393349-1-ast@fiberby.net> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: kuba@kernel.org TC filters come in 3 variants: - no flag (try to process in hardware, but fallback to software)) - skip_hw (do not process filter by hardware) - skip_sw (do not process filter by software) However skip_sw is implemented so that the skip_sw flag can first be checked, after it has been matched. IMHO it's common when using skip_sw, to use it on all rules. So if all filters in a block is skip_sw filters, then we can bail early, we can thus avoid having to match the filters, just to check for the skip_sw flag. This patch adds a bypass, for when only TC skip_sw rules are used. The bypass is guarded by a static key, to avoid harming other workloads. There are 3 ways that a packet from a skip_sw ruleset, can end up in the kernel path. Although the send packets to a non-existent chain way is only improved a few percents, then I believe it's worth optimizing the trap and fall-though use-cases. +----------------------------+--------+--------+--------+ | Test description | Pre- | Post- | Rel. | | | kpps | kpps | chg. | +----------------------------+--------+--------+--------+ | basic forwarding + notrack | 3589.3 | 3587.9 | 1.00x | | switch to eswitch mode | 3081.8 | 3094.7 | 1.00x | | add ingress qdisc | 3042.9 | 3063.6 | 1.01x | | tc forward in hw / skip_sw |37024.7 |37028.4 | 1.00x | | tc forward in sw / skip_hw | 3245.0 | 3245.3 | 1.00x | +----------------------------+--------+--------+--------+ | tests with only skip_sw rules below: | +----------------------------+--------+--------+--------+ | 1 non-matching rule | 2694.7 | 3058.7 | 1.14x | | 1 n-m rule, match trap | 2611.2 | 3323.1 | 1.27x | | 1 n-m rule, goto non-chain | 2886.8 | 2945.9 | 1.02x | | 5 non-matching rules | 1958.2 | 3061.3 | 1.56x | | 5 n-m rules, match trap | 1911.9 | 3327.0 | 1.74x | | 5 n-m rules, goto non-chain| 2883.1 | 2947.5 | 1.02x | | 10 non-matching rules | 1466.3 | 3062.8 | 2.09x | | 10 n-m rules, match trap | 1444.3 | 3317.9 | 2.30x | | 10 n-m rules,goto non-chain| 2883.1 | 2939.5 | 1.02x | | 25 non-matching rules | 838.5 | 3058.9 | 3.65x | | 25 n-m rules, match trap | 824.5 | 3323.0 | 4.03x | | 25 n-m rules,goto non-chain| 2875.8 | 2944.7 | 1.02x | | 50 non-matching rules | 488.1 | 3054.7 | 6.26x | | 50 n-m rules, match trap | 484.9 | 3318.5 | 6.84x | | 50 n-m rules,goto non-chain| 2884.1 | 2939.7 | 1.02x | +----------------------------+--------+--------+--------+ perf top (25 n-m skip_sw rules - pre patch): 20.39% [kernel] [k] __skb_flow_dissect 16.43% [kernel] [k] rhashtable_jhash2 10.58% [kernel] [k] fl_classify 10.23% [kernel] [k] fl_mask_lookup 4.79% [kernel] [k] memset_orig 2.58% [kernel] [k] tcf_classify 1.47% [kernel] [k] __x86_indirect_thunk_rax 1.42% [kernel] [k] __dev_queue_xmit 1.36% [kernel] [k] nft_do_chain 1.21% [kernel] [k] __rcu_read_lock perf top (25 n-m skip_sw rules - post patch): 5.12% [kernel] [k] __dev_queue_xmit 4.77% [kernel] [k] nft_do_chain 3.65% [kernel] [k] dev_gro_receive 3.41% [kernel] [k] check_preemption_disabled 3.14% [kernel] [k] mlx5e_skb_from_cqe_mpwrq_nonlinear 2.88% [kernel] [k] __netif_receive_skb_core.constprop.0 2.49% [kernel] [k] mlx5e_xmit 2.15% [kernel] [k] ip_forward 1.95% [kernel] [k] mlx5e_tc_restore_tunnel 1.92% [kernel] [k] vlan_gro_receive Test setup: DUT: Intel Xeon D-1518 (2.20GHz) w/ Nvidia/Mellanox ConnectX-6 Dx 2x100G Data rate measured on switch (Extreme X690), and DUT connected as a router on a stick, with pktgen and pktsink as VLANs. Pktgen-dpdk was in range 36.6-37.7 Mpps 64B packets across all tests. Full test data at https://files.fiberby.net/ast/2024/tc_skip_sw/v2_tests/ Signed-off-by: Asbjørn Sloth Tønnesen Reviewed-by: Simon Horman Reviewed-by: Marcelo Ricardo Leitner --- include/net/pkt_cls.h | 9 +++++++++ include/net/sch_generic.h | 1 + net/core/dev.c | 10 ++++++++++ net/sched/cls_api.c | 18 ++++++++++++++++++ 4 files changed, 38 insertions(+) diff --git a/include/net/pkt_cls.h b/include/net/pkt_cls.h index a4ee43f493bb..41297bd38dff 100644 --- a/include/net/pkt_cls.h +++ b/include/net/pkt_cls.h @@ -74,6 +74,15 @@ static inline bool tcf_block_non_null_shared(struct tcf_block *block) return block && block->index; } +#ifdef CONFIG_NET_CLS_ACT +DECLARE_STATIC_KEY_FALSE(tcf_bypass_check_needed_key); + +static inline bool tcf_block_bypass_sw(struct tcf_block *block) +{ + return block && block->bypass_wanted; +} +#endif + static inline struct Qdisc *tcf_block_q(struct tcf_block *block) { WARN_ON(tcf_block_shared(block)); diff --git a/include/net/sch_generic.h b/include/net/sch_generic.h index eb3872c22fcd..76db6be16083 100644 --- a/include/net/sch_generic.h +++ b/include/net/sch_generic.h @@ -472,6 +472,7 @@ struct tcf_block { struct flow_block flow_block; struct list_head owner_list; bool keep_dst; + bool bypass_wanted; atomic_t filtercnt; /* Number of filters */ atomic_t skipswcnt; /* Number of skip_sw filters */ atomic_t offloadcnt; /* Number of oddloaded filters */ diff --git a/net/core/dev.c b/net/core/dev.c index 9a67003e49db..53f36991ea8e 100644 --- a/net/core/dev.c +++ b/net/core/dev.c @@ -2057,6 +2057,11 @@ void net_dec_egress_queue(void) EXPORT_SYMBOL_GPL(net_dec_egress_queue); #endif +#ifdef CONFIG_NET_CLS_ACT +DEFINE_STATIC_KEY_FALSE(tcf_bypass_check_needed_key); +EXPORT_SYMBOL(tcf_bypass_check_needed_key); +#endif + DEFINE_STATIC_KEY_FALSE(netstamp_needed_key); EXPORT_SYMBOL(netstamp_needed_key); #ifdef CONFIG_JUMP_LABEL @@ -3911,6 +3916,11 @@ static int tc_run(struct tcx_entry *entry, struct sk_buff *skb, if (!miniq) return ret; + if (static_branch_unlikely(&tcf_bypass_check_needed_key)) { + if (tcf_block_bypass_sw(miniq->block)) + return ret; + } + tc_skb_cb(skb)->mru = 0; tc_skb_cb(skb)->post_ct = false; tcf_set_drop_reason(skb, *drop_reason); diff --git a/net/sched/cls_api.c b/net/sched/cls_api.c index 304a46ab0e0b..db0653993632 100644 --- a/net/sched/cls_api.c +++ b/net/sched/cls_api.c @@ -410,6 +410,23 @@ static void tcf_proto_get(struct tcf_proto *tp) refcount_inc(&tp->refcnt); } +static void tcf_maintain_bypass(struct tcf_block *block) +{ + int filtercnt = atomic_read(&block->filtercnt); + int skipswcnt = atomic_read(&block->skipswcnt); + bool bypass_wanted = filtercnt > 0 && filtercnt == skipswcnt; + + if (bypass_wanted != block->bypass_wanted) { +#ifdef CONFIG_NET_CLS_ACT + if (bypass_wanted) + static_branch_inc(&tcf_bypass_check_needed_key); + else + static_branch_dec(&tcf_bypass_check_needed_key); +#endif + block->bypass_wanted = bypass_wanted; + } +} + static void tcf_block_filter_cnt_update(struct tcf_block *block, bool *counted, bool add) { lockdep_assert_not_held(&block->cb_lock); @@ -424,6 +441,7 @@ static void tcf_block_filter_cnt_update(struct tcf_block *block, bool *counted, *counted = false; } } + tcf_maintain_bypass(block); up_write(&block->cb_lock); }