From patchwork Wed Jul 28 16:23:59 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Paolo Abeni X-Patchwork-Id: 12406325 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A01B9C4338F for ; Wed, 28 Jul 2021 16:24:39 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 8024E60F9B for ; Wed, 28 Jul 2021 16:24:39 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230041AbhG1QYj (ORCPT ); Wed, 28 Jul 2021 12:24:39 -0400 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:36310 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229977AbhG1QYi (ORCPT ); Wed, 28 Jul 2021 12:24:38 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1627489476; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=03zyxcpyFQ3z/dlAQUsY54cVAItX7uVxV1tMy1518Os=; b=Ntw+YE+nr6PeXxy1pnxUg0pJRvHLJn1jtVdjhOvAAy7LA+L5qjqofEeB/SDulIIZ0G+XO9 6NET5uztcDyEVI1h8WphlwF0wWFtRK6IA9MDqyoYJ1Scgajz9hA1u5XxVYYGTFDl18EHJf oiigqWgZv5A7Nw0RchlOmSwmTwENT0s= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-553-qKtxpsmGNO2D8YMYB0hhWg-1; Wed, 28 Jul 2021 12:24:34 -0400 X-MC-Unique: qKtxpsmGNO2D8YMYB0hhWg-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 999031006706; Wed, 28 Jul 2021 16:24:33 +0000 (UTC) Received: from gerbillo.redhat.com (ovpn-113-169.ams2.redhat.com [10.36.113.169]) by smtp.corp.redhat.com (Postfix) with ESMTP id 822EA5C1B4; Wed, 28 Jul 2021 16:24:32 +0000 (UTC) From: Paolo Abeni To: netdev@vger.kernel.org Cc: "David S. Miller" , Jakub Kicinski , Eric Dumazet Subject: [PATCH net-next 1/6] sk_buff: introduce 'slow_gro' flags Date: Wed, 28 Jul 2021 18:23:59 +0200 Message-Id: <075a3b28299ad4a0b00c28ff5886b9b4fe475392.1627405778.git.pabeni@redhat.com> In-Reply-To: References: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: kuba@kernel.org The new flag tracks if any state field is set, so that GRO requires 'unusual'/slow prepare steps. Set such flag when a ct entry is attached to the skb, and never clear it. The new bit uses an existing hole into the sk_buff struct RFC -> v1: - use a single state bit, never clear it - avoid moving the _nfct field Signed-off-by: Paolo Abeni --- include/linux/skbuff.h | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/include/linux/skbuff.h b/include/linux/skbuff.h index f19190820e63..3ff18300d210 100644 --- a/include/linux/skbuff.h +++ b/include/linux/skbuff.h @@ -689,6 +689,7 @@ typedef unsigned char *sk_buff_data_t; * CHECKSUM_UNNECESSARY (max 3) * @dst_pending_confirm: need to confirm neighbour * @decrypted: Decrypted SKB + * @slow_gro: state present at GRO time, slower prepare step required * @napi_id: id of the NAPI struct this skb came from * @sender_cpu: (aka @napi_id) source CPU in XPS * @secmark: security marking @@ -870,6 +871,7 @@ struct sk_buff { #ifdef CONFIG_TLS_DEVICE __u8 decrypted:1; #endif + __u8 slow_gro:1; #ifdef CONFIG_NET_SCHED __u16 tc_index; /* traffic control index */ @@ -4216,6 +4218,7 @@ static inline unsigned long skb_get_nfct(const struct sk_buff *skb) static inline void skb_set_nfct(struct sk_buff *skb, unsigned long nfct) { #if IS_ENABLED(CONFIG_NF_CONNTRACK) + skb->slow_gro |= !!nfct; skb->_nfct = nfct; #endif } @@ -4375,6 +4378,7 @@ static inline void nf_copy(struct sk_buff *dst, const struct sk_buff *src) #if defined(CONFIG_NF_CONNTRACK) || defined(CONFIG_NF_CONNTRACK_MODULE) nf_conntrack_put(skb_nfct(dst)); #endif + dst->slow_gro = src->slow_gro; __nf_copy(dst, src, true); } From patchwork Wed Jul 28 16:24:00 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Paolo Abeni X-Patchwork-Id: 12406327 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 44184C4338F for ; Wed, 28 Jul 2021 16:24:43 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 217986101B for ; Wed, 28 Jul 2021 16:24:43 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230301AbhG1QYn (ORCPT ); Wed, 28 Jul 2021 12:24:43 -0400 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:56847 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230050AbhG1QYk (ORCPT ); Wed, 28 Jul 2021 12:24:40 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1627489477; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=jOo2JsCLb9TBbS6ldjkssK+2QlGTN2jf+nwfruktFaU=; b=bnG9SPGCGmiy616P131X4fwnWHmZYMBRbQRSfI7/smuiCem2+147lryUEb/Mj/IPTUlw1m uBnZtUA7lM3Rp7bual8KgYq7SmD7Ck3xpBZhjMxADE8v0+zoXIZIp8CYfrP6d4l/RExWUi uh5ZMCqCBo1609i2o+mh5rV5T7zJXXc= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-457-URRO55bgMdya8ack9nACJA-1; Wed, 28 Jul 2021 12:24:36 -0400 X-MC-Unique: URRO55bgMdya8ack9nACJA-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 37A5618C8C03; Wed, 28 Jul 2021 16:24:35 +0000 (UTC) Received: from gerbillo.redhat.com (ovpn-113-169.ams2.redhat.com [10.36.113.169]) by smtp.corp.redhat.com (Postfix) with ESMTP id EE30B5C1B4; Wed, 28 Jul 2021 16:24:33 +0000 (UTC) From: Paolo Abeni To: netdev@vger.kernel.org Cc: "David S. Miller" , Jakub Kicinski , Eric Dumazet Subject: [PATCH net-next 2/6] sk_buff: track dst status in slow_gro Date: Wed, 28 Jul 2021 18:24:00 +0200 Message-Id: In-Reply-To: References: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: kuba@kernel.org Similar to the previous patch, but covering the dst field: the slow_gro flag is additionally set when a dst is attached to the skb RFC -> v1: - use the existing flag instead of adding a new one Signed-off-by: Paolo Abeni --- include/linux/skbuff.h | 2 ++ include/net/dst.h | 2 ++ 2 files changed, 4 insertions(+) diff --git a/include/linux/skbuff.h b/include/linux/skbuff.h index 3ff18300d210..b1e5bbfcc926 100644 --- a/include/linux/skbuff.h +++ b/include/linux/skbuff.h @@ -992,6 +992,7 @@ static inline struct dst_entry *skb_dst(const struct sk_buff *skb) */ static inline void skb_dst_set(struct sk_buff *skb, struct dst_entry *dst) { + skb->slow_gro |= !!dst; skb->_skb_refdst = (unsigned long)dst; } @@ -1008,6 +1009,7 @@ static inline void skb_dst_set(struct sk_buff *skb, struct dst_entry *dst) static inline void skb_dst_set_noref(struct sk_buff *skb, struct dst_entry *dst) { WARN_ON(!rcu_read_lock_held() && !rcu_read_lock_bh_held()); + skb->slow_gro = !!dst; skb->_skb_refdst = (unsigned long)dst | SKB_DST_NOREF; } diff --git a/include/net/dst.h b/include/net/dst.h index 75b1e734e9c2..a057319aabef 100644 --- a/include/net/dst.h +++ b/include/net/dst.h @@ -277,6 +277,7 @@ static inline void skb_dst_drop(struct sk_buff *skb) static inline void __skb_dst_copy(struct sk_buff *nskb, unsigned long refdst) { + nskb->slow_gro |= !!refdst; nskb->_skb_refdst = refdst; if (!(nskb->_skb_refdst & SKB_DST_NOREF)) dst_clone(skb_dst(nskb)); @@ -316,6 +317,7 @@ static inline bool skb_dst_force(struct sk_buff *skb) dst = NULL; skb->_skb_refdst = (unsigned long)dst; + skb->slow_gro |= !!dst; } return skb->_skb_refdst != 0UL; From patchwork Wed Jul 28 16:24:01 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Paolo Abeni X-Patchwork-Id: 12406329 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D9905C432BE for ; Wed, 28 Jul 2021 16:24:44 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id BF4A361037 for ; Wed, 28 Jul 2021 16:24:44 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230262AbhG1QYo (ORCPT ); Wed, 28 Jul 2021 12:24:44 -0400 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:49484 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230213AbhG1QYm (ORCPT ); Wed, 28 Jul 2021 12:24:42 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1627489479; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=+TeIpvJ1BAVpAfhn5njvcI5dr+DIMci7+nWCkodmJl0=; b=TRvQwgdpcwjvV66gFi/mqddXw+/14hYAUNBpOgBgPJykjcEH37vcX2Oh1JhLCMHcMn8u5b OsVxM+5tE66Dc82MfUQcJbTcF7mPXZBH6Q+J5qu6qoKpr3ygqxC0R91N1101Eg8ab7Zg1F KmV051tkyJDfIDrMXakCHznvzEFotGA= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-27-6MuXRCX1P56kbxlUR6GgXw-1; Wed, 28 Jul 2021 12:24:37 -0400 X-MC-Unique: 6MuXRCX1P56kbxlUR6GgXw-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id A3E39760C8; Wed, 28 Jul 2021 16:24:36 +0000 (UTC) Received: from gerbillo.redhat.com (ovpn-113-169.ams2.redhat.com [10.36.113.169]) by smtp.corp.redhat.com (Postfix) with ESMTP id 8BAD35C1B4; Wed, 28 Jul 2021 16:24:35 +0000 (UTC) From: Paolo Abeni To: netdev@vger.kernel.org Cc: "David S. Miller" , Jakub Kicinski , Eric Dumazet Subject: [PATCH net-next 3/6] sk_buff: track extension status in slow_gro Date: Wed, 28 Jul 2021 18:24:01 +0200 Message-Id: In-Reply-To: References: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: kuba@kernel.org Similar to the previous one, but tracking the active_extensions field status. Signed-off-by: Paolo Abeni --- net/core/skbuff.c | 1 + 1 file changed, 1 insertion(+) diff --git a/net/core/skbuff.c b/net/core/skbuff.c index 517ee2c36425..a990e11c393c 100644 --- a/net/core/skbuff.c +++ b/net/core/skbuff.c @@ -6455,6 +6455,7 @@ void *skb_ext_add(struct sk_buff *skb, enum skb_ext_id id) new->chunks = newlen; new->offset[id] = newoff; set_active: + skb->slow_gro = 1; skb->extensions = new; skb->active_extensions |= 1 << id; return skb_ext_get_ptr(new, id); From patchwork Wed Jul 28 16:24:02 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Paolo Abeni X-Patchwork-Id: 12406331 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id CADC9C432BE for ; Wed, 28 Jul 2021 16:24:47 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id AD1A860F23 for ; Wed, 28 Jul 2021 16:24:47 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230286AbhG1QYs (ORCPT ); Wed, 28 Jul 2021 12:24:48 -0400 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:31982 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230288AbhG1QYn (ORCPT ); Wed, 28 Jul 2021 12:24:43 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1627489480; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=dG7SShBa+93pJWuOsLAMXbghStF9LVom0np5u1WWtm4=; b=F+hoSFtr3l0mTK4n7qVzdOXUT0HvM7EKHIoxGbG7SGGRVEQ4jO/UcH9XYqnTlzt9q0wHdS OXXt2F9iv4SvZXOU5L4LGyVc/5Mw1wWDfEF3FnikC5tngvvpKvmOkCVaJvDCZMTs0FuR+I ++u9WtSbEkIwaogrZXHAJccwqcpnPus= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-147-bSi7T0MVNX6UXVi9BS0wUw-1; Wed, 28 Jul 2021 12:24:39 -0400 X-MC-Unique: bSi7T0MVNX6UXVi9BS0wUw-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 356D018C8C00; Wed, 28 Jul 2021 16:24:38 +0000 (UTC) Received: from gerbillo.redhat.com (ovpn-113-169.ams2.redhat.com [10.36.113.169]) by smtp.corp.redhat.com (Postfix) with ESMTP id 033965C1B4; Wed, 28 Jul 2021 16:24:36 +0000 (UTC) From: Paolo Abeni To: netdev@vger.kernel.org Cc: "David S. Miller" , Jakub Kicinski , Eric Dumazet Subject: [PATCH net-next 4/6] net: optimize GRO for the common case. Date: Wed, 28 Jul 2021 18:24:02 +0200 Message-Id: <4f4b2aa4021b31c3c398d55628064e186a7ec50d.1627405778.git.pabeni@redhat.com> In-Reply-To: References: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: kuba@kernel.org After the previous patches, at GRO time, skb->slow_gro is usually 0, unless the packets comes from some H/W offload slowpath or tunnel. We can optimize the GRO code assuming !skb->slow_gro is likely. This remove multiple conditionals in the most common path, at the price of an additional one when we hit the above "slow-paths". Signed-off-by: Paolo Abeni --- net/core/dev.c | 30 ++++++++++++++++++++++-------- net/core/skbuff.c | 9 ++++++--- 2 files changed, 28 insertions(+), 11 deletions(-) diff --git a/net/core/dev.c b/net/core/dev.c index fb5d12a3d52d..19565f7497ee 100644 --- a/net/core/dev.c +++ b/net/core/dev.c @@ -6014,7 +6014,6 @@ static void gro_list_prepare(const struct list_head *head, diffs |= skb_vlan_tag_present(p) ^ skb_vlan_tag_present(skb); if (skb_vlan_tag_present(p)) diffs |= skb_vlan_tag_get(p) ^ skb_vlan_tag_get(skb); - diffs |= skb_metadata_dst_cmp(p, skb); diffs |= skb_metadata_differs(p, skb); if (maclen == ETH_HLEN) diffs |= compare_ether_header(skb_mac_header(p), @@ -6024,17 +6023,29 @@ static void gro_list_prepare(const struct list_head *head, skb_mac_header(skb), maclen); - diffs |= skb_get_nfct(p) ^ skb_get_nfct(skb); + /* in most common scenarions _state is 0 + * otherwise we are already on some slower paths + * either skip all the infrequent tests altogether or + * avoid trying too hard to skip each of them individually + */ + if (!diffs && unlikely(skb->slow_gro | p->slow_gro)) { +#if IS_ENABLED(CONFIG_SKB_EXTENSIONS) && IS_ENABLED(CONFIG_NET_TC_SKB_EXT) + struct tc_skb_ext *skb_ext; + struct tc_skb_ext *p_ext; +#endif + + diffs |= skb_metadata_dst_cmp(p, skb); + diffs |= skb_get_nfct(p) ^ skb_get_nfct(skb); + #if IS_ENABLED(CONFIG_SKB_EXTENSIONS) && IS_ENABLED(CONFIG_NET_TC_SKB_EXT) - if (!diffs) { - struct tc_skb_ext *skb_ext = skb_ext_find(skb, TC_SKB_EXT); - struct tc_skb_ext *p_ext = skb_ext_find(p, TC_SKB_EXT); + skb_ext = skb_ext_find(skb, TC_SKB_EXT); + p_ext = skb_ext_find(p, TC_SKB_EXT); diffs |= (!!p_ext) ^ (!!skb_ext); if (!diffs && unlikely(skb_ext)) diffs |= p_ext->chain ^ skb_ext->chain; - } #endif + } NAPI_GRO_CB(p)->same_flow = !diffs; } @@ -6299,8 +6310,11 @@ static void napi_reuse_skb(struct napi_struct *napi, struct sk_buff *skb) skb->encapsulation = 0; skb_shinfo(skb)->gso_type = 0; skb->truesize = SKB_TRUESIZE(skb_end_offset(skb)); - skb_ext_reset(skb); - nf_reset_ct(skb); + if (unlikely(skb->slow_gro)) { + skb_ext_reset(skb); + nf_reset_ct(skb); + skb->slow_gro = 0; + } napi->skb = skb; } diff --git a/net/core/skbuff.c b/net/core/skbuff.c index a990e11c393c..8231cbddb6ed 100644 --- a/net/core/skbuff.c +++ b/net/core/skbuff.c @@ -954,9 +954,12 @@ void __kfree_skb_defer(struct sk_buff *skb) void napi_skb_free_stolen_head(struct sk_buff *skb) { - nf_reset_ct(skb); - skb_dst_drop(skb); - skb_ext_put(skb); + if (unlikely(skb->slow_gro)) { + nf_reset_ct(skb); + skb_dst_drop(skb); + skb_ext_put(skb); + skb->slow_gro = 0; + } napi_skb_cache_put(skb); } From patchwork Wed Jul 28 16:24:03 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Paolo Abeni X-Patchwork-Id: 12406333 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 787F1C4338F for ; Wed, 28 Jul 2021 16:24:52 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 61F2A60F9B for ; Wed, 28 Jul 2021 16:24:52 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230420AbhG1QYw (ORCPT ); Wed, 28 Jul 2021 12:24:52 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:40892 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229537AbhG1QYo (ORCPT ); Wed, 28 Jul 2021 12:24:44 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1627489482; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ODP3Lmh29eQiV6enDZGG45Jje9Il0RtrlXidMbt/qVI=; b=eEIzndCK8sy3s7BF8S6HWucFz2msSDp0SAX584QKsqUazDc4RqQ8A1iql3WEyWNqtZNtlv q8FSYJvP51fu78TNHgAHuWzP97ANOoFHhhFRJ3Nkmf5RQIg0azKTcOXOh7C3VbzYqlgr4V sVr0W+tzKpDYZ79q2ojEudGxZxeGM+8= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-366-hi2_Xf2UMhCvPGIWnkYzGA-1; Wed, 28 Jul 2021 12:24:40 -0400 X-MC-Unique: hi2_Xf2UMhCvPGIWnkYzGA-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id A018A1084F40; Wed, 28 Jul 2021 16:24:39 +0000 (UTC) Received: from gerbillo.redhat.com (ovpn-113-169.ams2.redhat.com [10.36.113.169]) by smtp.corp.redhat.com (Postfix) with ESMTP id 88D6E5C1B4; Wed, 28 Jul 2021 16:24:38 +0000 (UTC) From: Paolo Abeni To: netdev@vger.kernel.org Cc: "David S. Miller" , Jakub Kicinski , Eric Dumazet Subject: [PATCH net-next 5/6] skbuff: allow 'slow_gro' for skb carring sock reference Date: Wed, 28 Jul 2021 18:24:03 +0200 Message-Id: <4864c71049779d924c1cf9d2cd8a86b064f8fd17.1627405778.git.pabeni@redhat.com> In-Reply-To: References: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: kuba@kernel.org This change leverages the infrastructure introduced by the previous patches to allow soft devices passing to the GRO engine owned skbs without impacting the fast-path. It's up to the GRO caller ensuring the slow_gro bit validity before invoking the GRO engine. The new helper skb_prepare_for_gro() is introduced for that goal. On slow_gro, skbs are aggregated only with equal sk. Additionally, skb truesize on GRO recycle and free is correctly updated so that sk wmem is not changed by the GRO processing. rfc-> v1: - fixed bad truesize on dev_gro_receive NAPI_FREE - use the existing state bit Signed-off-by: Paolo Abeni --- include/net/sock.h | 9 +++++++++ net/core/dev.c | 2 ++ net/core/skbuff.c | 17 +++++++++++++---- 3 files changed, 24 insertions(+), 4 deletions(-) diff --git a/include/net/sock.h b/include/net/sock.h index f23cb259b0e2..ff1be7e7e90b 100644 --- a/include/net/sock.h +++ b/include/net/sock.h @@ -2249,6 +2249,15 @@ static inline __must_check bool skb_set_owner_sk_safe(struct sk_buff *skb, struc return false; } +static inline void skb_prepare_for_gro(struct sk_buff *skb) +{ + if (skb->destructor != sock_wfree) { + skb_orphan(skb); + return; + } + skb->slow_gro = 1; +} + void sk_reset_timer(struct sock *sk, struct timer_list *timer, unsigned long expires); diff --git a/net/core/dev.c b/net/core/dev.c index 19565f7497ee..dcc87fcd64ba 100644 --- a/net/core/dev.c +++ b/net/core/dev.c @@ -6034,6 +6034,7 @@ static void gro_list_prepare(const struct list_head *head, struct tc_skb_ext *p_ext; #endif + diffs |= p->sk != skb->sk; diffs |= skb_metadata_dst_cmp(p, skb); diffs |= skb_get_nfct(p) ^ skb_get_nfct(skb); @@ -6311,6 +6312,7 @@ static void napi_reuse_skb(struct napi_struct *napi, struct sk_buff *skb) skb_shinfo(skb)->gso_type = 0; skb->truesize = SKB_TRUESIZE(skb_end_offset(skb)); if (unlikely(skb->slow_gro)) { + skb_orphan(skb); skb_ext_reset(skb); nf_reset_ct(skb); skb->slow_gro = 0; diff --git a/net/core/skbuff.c b/net/core/skbuff.c index 8231cbddb6ed..9510cb0807bc 100644 --- a/net/core/skbuff.c +++ b/net/core/skbuff.c @@ -958,6 +958,7 @@ void napi_skb_free_stolen_head(struct sk_buff *skb) nf_reset_ct(skb); skb_dst_drop(skb); skb_ext_put(skb); + skb_orphan(skb); skb->slow_gro = 0; } napi_skb_cache_put(skb); @@ -3898,6 +3899,9 @@ int skb_gro_receive_list(struct sk_buff *p, struct sk_buff *skb) NAPI_GRO_CB(p)->last = skb; NAPI_GRO_CB(p)->count++; p->data_len += skb->len; + + /* sk owenrship - if any - completely transferred to the aggregated packet */ + skb->destructor = NULL; p->truesize += skb->truesize; p->len += skb->len; @@ -4265,6 +4269,7 @@ int skb_gro_receive(struct sk_buff *p, struct sk_buff *skb) unsigned int headlen = skb_headlen(skb); unsigned int len = skb_gro_len(skb); unsigned int delta_truesize; + unsigned int new_truesize; struct sk_buff *lp; if (unlikely(p->len + len >= 65536 || NAPI_GRO_CB(skb)->flush)) @@ -4296,10 +4301,10 @@ int skb_gro_receive(struct sk_buff *p, struct sk_buff *skb) skb_frag_size_sub(frag, offset); /* all fragments truesize : remove (head size + sk_buff) */ - delta_truesize = skb->truesize - - SKB_TRUESIZE(skb_end_offset(skb)); + new_truesize = SKB_TRUESIZE(skb_end_offset(skb)); + delta_truesize = skb->truesize - new_truesize; - skb->truesize -= skb->data_len; + skb->truesize = new_truesize; skb->len -= skb->data_len; skb->data_len = 0; @@ -4328,12 +4333,16 @@ int skb_gro_receive(struct sk_buff *p, struct sk_buff *skb) memcpy(frag + 1, skbinfo->frags, sizeof(*frag) * skbinfo->nr_frags); /* We dont need to clear skbinfo->nr_frags here */ - delta_truesize = skb->truesize - SKB_DATA_ALIGN(sizeof(struct sk_buff)); + new_truesize = SKB_TRUESIZE(sizeof(struct sk_buff)); + delta_truesize = skb->truesize - new_truesize; + skb->truesize = new_truesize; NAPI_GRO_CB(skb)->free = NAPI_GRO_FREE_STOLEN_HEAD; goto done; } merge: + /* sk owenrship - if any - completely transferred to the aggregated packet */ + skb->destructor = NULL; delta_truesize = skb->truesize; if (offset > headlen) { unsigned int eat = offset - headlen; From patchwork Wed Jul 28 16:24:04 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Paolo Abeni X-Patchwork-Id: 12406335 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id F1097C432BE for ; Wed, 28 Jul 2021 16:24:53 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id DB1C960F23 for ; Wed, 28 Jul 2021 16:24:53 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230437AbhG1QYy (ORCPT ); Wed, 28 Jul 2021 12:24:54 -0400 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:27607 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230283AbhG1QYp (ORCPT ); Wed, 28 Jul 2021 12:24:45 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1627489483; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=9zrJJBSp28F7+S8SVJdlDpI62YOFgpVTiDbprhSEFuw=; b=UyFpJI9sUcpRacMg88L2VaxU+zHPaySfdnr4gkVtD/q+lwtDlzyS2/J2Mc5sBAxZs2y3LD g1+/YL+ynk74buUol1IWxzpspo3XJkapsh+JL5NVeycgdjz96pobTqa4qy1QFKolw9Auci onZZ8ikvsrg3lHWrn6bUELQELdtwqy0= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-357-kZ1aZCJrMAe3gikLlWSn-Q-1; Wed, 28 Jul 2021 12:24:42 -0400 X-MC-Unique: kZ1aZCJrMAe3gikLlWSn-Q-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 186A110066E5; Wed, 28 Jul 2021 16:24:41 +0000 (UTC) Received: from gerbillo.redhat.com (ovpn-113-169.ams2.redhat.com [10.36.113.169]) by smtp.corp.redhat.com (Postfix) with ESMTP id 004EE5C1B4; Wed, 28 Jul 2021 16:24:39 +0000 (UTC) From: Paolo Abeni To: netdev@vger.kernel.org Cc: "David S. Miller" , Jakub Kicinski , Eric Dumazet Subject: [PATCH net-next 6/6] veth: use skb_prepare_for_gro() Date: Wed, 28 Jul 2021 18:24:04 +0200 Message-Id: In-Reply-To: References: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: kuba@kernel.org Leveraging the previous patch we can now avoid orphaning the skb in the veth gro path, allowing correct backpressure. Signed-off-by: Paolo Abeni --- drivers/net/veth.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drivers/net/veth.c b/drivers/net/veth.c index 381670c08ba7..50eb43e5bf45 100644 --- a/drivers/net/veth.c +++ b/drivers/net/veth.c @@ -713,7 +713,7 @@ static struct sk_buff *veth_xdp_rcv_skb(struct veth_rq *rq, int mac_len, delta, off; struct xdp_buff xdp; - skb_orphan_partial(skb); + skb_prepare_for_gro(skb); rcu_read_lock(); xdp_prog = rcu_dereference(rq->xdp_prog);