From patchwork Fri Apr 26 06:51:35 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Felix Fietkau X-Patchwork-Id: 13644155 X-Patchwork-Delegate: kuba@kernel.org Received: from nbd.name (nbd.name [46.4.11.11]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B37CE13A3ED; Fri, 26 Apr 2024 06:51:51 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=46.4.11.11 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1714114313; cv=none; b=mGn0tmoPak/MlHUL2pQxjBKuQrvIirku4/kf3V0XLou/bA4OR3SzUmxYhpOhQ+HLQl30wb9TGejsLu1wgGbPivQoJOc7AX9ESfBjm/rMhsYr7cA7EFeQBFDyIFKX5GyH8S/yctydQ215pmjrJT5v/VAVzrqyl+H6HbvTiSMy8G8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1714114313; c=relaxed/simple; bh=IXZq04hiu+Pn+sF7BKtV+tVHTOSfOBndsfo4YDuDbeo=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=fF7TQ95yJ+dF8gE8RSaln8khpKx290Ldc7/QJMc8Y8mGrHO7VGzV3G3n/S0KBpWjQgK98TmlKf6IZW7haMKyOH8ULpm8DWsHu9tqVmdHxdUDn6itrJcuHAfc3yjV4xnNUXo4XYnC557Zl1VsP2uB0WWYD+O9z/pr8VXa0pCNtxg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=nbd.name; spf=none smtp.mailfrom=nbd.name; dkim=pass (1024-bit key) header.d=nbd.name header.i=@nbd.name header.b=nDehk99w; arc=none smtp.client-ip=46.4.11.11 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=nbd.name Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=nbd.name Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=nbd.name header.i=@nbd.name header.b="nDehk99w" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=nbd.name; s=20160729; h=Content-Transfer-Encoding:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Sender:Reply-To:Content-Type:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:List-Subscribe: List-Post:List-Owner:List-Archive; bh=3H6dBRGk+BHU+y8mRcNlZ506UX4DS949WCdH9l5DVuI=; b=nDehk99we7gXeh0IqH5E7Qi4u+ yhJCwnI4Q8HCR3USL2X3QHIU4ccbv08REv1p3ck9EWsqhFDMpnQLEILCiUmtp4B2bzCEYAwambnmx L8qjEUg1jJG1Yy9aUcArPUM+ko5afZ0iuQD2ELGQTaVTczYzhU0wIiXhqu5HY5tLpk2o=; Received: from p54ae9c93.dip0.t-ipconnect.de ([84.174.156.147] helo=localhost.localdomain) by ds12 with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256 (Exim 4.96) (envelope-from ) id 1s0FQv-007ltt-1N; Fri, 26 Apr 2024 08:51:45 +0200 From: Felix Fietkau To: netdev@vger.kernel.org, "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , David Ahern Cc: willemdebruijn.kernel@gmail.com, linux-kernel@vger.kernel.org Subject: [PATCH v3 net-next v3 1/6] net: move skb_gro_receive_list from udp to core Date: Fri, 26 Apr 2024 08:51:35 +0200 Message-ID: <20240426065143.4667-2-nbd@nbd.name> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240426065143.4667-1-nbd@nbd.name> References: <20240426065143.4667-1-nbd@nbd.name> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: kuba@kernel.org This helper function will be used for TCP fraglist GRO support Signed-off-by: Felix Fietkau Reviewed-by: Eric Dumazet --- include/net/gro.h | 1 + net/core/gro.c | 27 +++++++++++++++++++++++++++ net/ipv4/udp_offload.c | 27 --------------------------- 3 files changed, 28 insertions(+), 27 deletions(-) diff --git a/include/net/gro.h b/include/net/gro.h index 50f1e403dbbb..ca8e4b3de044 100644 --- a/include/net/gro.h +++ b/include/net/gro.h @@ -429,6 +429,7 @@ static inline __wsum ip6_gro_compute_pseudo(const struct sk_buff *skb, } int skb_gro_receive(struct sk_buff *p, struct sk_buff *skb); +int skb_gro_receive_list(struct sk_buff *p, struct sk_buff *skb); /* Pass the currently batched GRO_NORMAL SKBs up to the stack. */ static inline void gro_normal_list(struct napi_struct *napi) diff --git a/net/core/gro.c b/net/core/gro.c index 2459ab697f7f..268c6c826d09 100644 --- a/net/core/gro.c +++ b/net/core/gro.c @@ -231,6 +231,33 @@ int skb_gro_receive(struct sk_buff *p, struct sk_buff *skb) return 0; } +int skb_gro_receive_list(struct sk_buff *p, struct sk_buff *skb) +{ + if (unlikely(p->len + skb->len >= 65536)) + return -E2BIG; + + if (NAPI_GRO_CB(p)->last == p) + skb_shinfo(p)->frag_list = skb; + else + NAPI_GRO_CB(p)->last->next = skb; + + skb_pull(skb, skb_gro_offset(skb)); + + NAPI_GRO_CB(p)->last = skb; + NAPI_GRO_CB(p)->count++; + p->data_len += skb->len; + + /* sk ownership - if any - completely transferred to the aggregated packet */ + skb->destructor = NULL; + skb->sk = NULL; + p->truesize += skb->truesize; + p->len += skb->len; + + NAPI_GRO_CB(skb)->same_flow = 1; + + return 0; +} + static void napi_gro_complete(struct napi_struct *napi, struct sk_buff *skb) { diff --git a/net/ipv4/udp_offload.c b/net/ipv4/udp_offload.c index 3498dd1d0694..a3cd546a1aea 100644 --- a/net/ipv4/udp_offload.c +++ b/net/ipv4/udp_offload.c @@ -433,33 +433,6 @@ static struct sk_buff *udp4_ufo_fragment(struct sk_buff *skb, return segs; } -static int skb_gro_receive_list(struct sk_buff *p, struct sk_buff *skb) -{ - if (unlikely(p->len + skb->len >= 65536)) - return -E2BIG; - - if (NAPI_GRO_CB(p)->last == p) - skb_shinfo(p)->frag_list = skb; - else - NAPI_GRO_CB(p)->last->next = skb; - - skb_pull(skb, skb_gro_offset(skb)); - - NAPI_GRO_CB(p)->last = skb; - NAPI_GRO_CB(p)->count++; - p->data_len += skb->len; - - /* sk ownership - if any - completely transferred to the aggregated packet */ - skb->destructor = NULL; - skb->sk = NULL; - p->truesize += skb->truesize; - p->len += skb->len; - - NAPI_GRO_CB(skb)->same_flow = 1; - - return 0; -} - #define UDP_GRO_CNT_MAX 64 static struct sk_buff *udp_gro_receive_segment(struct list_head *head, From patchwork Fri Apr 26 06:51:36 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Felix Fietkau X-Patchwork-Id: 13644157 X-Patchwork-Delegate: kuba@kernel.org Received: from nbd.name (nbd.name [46.4.11.11]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id AFA9E13AA2F; Fri, 26 Apr 2024 06:51:53 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=46.4.11.11 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1714114315; cv=none; b=mmhoB1xg9YCQXorUEozgxHq0XXd0Hv2S3rttLSknZM6yf+fotsrABqd9HzZYoNWoluI5MNDzqytpLPXYOpuPvZJ5wSfzyOx970XP1CtMRHxkNbCJE7+vh0jGM259gRG/AcWeGbT+37HSUDkVx4OdIlDBTBRLxZ3ydKPwwDsph+w= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1714114315; c=relaxed/simple; bh=PyxgIB0SMMBIlfZ94obTA8c5rSzdnpxywtXW67Fqyys=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Z1mM8cNiTXsaPRWycxvTEMofImXzQmHL3hlUOu/zIx1uTIxOBiww64lG0qQRAqDllisHpdLlc4YqlvfkEGW/KnETu9HMSfQRvFNTeQzw3fAng9CUF5I7HhYcdmsMWvtarqu3laQe5+5r9U1giofIrWB4MurHt0nUzJI36KC/uL4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=nbd.name; spf=none smtp.mailfrom=nbd.name; dkim=pass (1024-bit key) header.d=nbd.name header.i=@nbd.name header.b=OzvdfUSq; arc=none smtp.client-ip=46.4.11.11 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=nbd.name Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=nbd.name Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=nbd.name header.i=@nbd.name header.b="OzvdfUSq" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=nbd.name; s=20160729; h=Content-Transfer-Encoding:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Sender:Reply-To:Content-Type:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:List-Subscribe: List-Post:List-Owner:List-Archive; bh=vxlaBzsWrAM9sVKYpZz1fvr0Z7qBXoZWBNzp0ai3Mm4=; b=OzvdfUSqt4a8hd+/h2YDgFCZUN 1+5pRiboqISbkBWbb0SWfBgg+tWsw9AR6Yp9HhyUo6CSPPxnCcZGYcR9hmACz1eTsHj9lnTpkvLjv hLTfunP3Njiajy7HLjtI9e4LY/paBFKsd69G50sTVobbnXOYLgHKAURjuMEK9Pcy9OLU=; Received: from p54ae9c93.dip0.t-ipconnect.de ([84.174.156.147] helo=localhost.localdomain) by ds12 with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256 (Exim 4.96) (envelope-from ) id 1s0FQw-007ltt-0Z; Fri, 26 Apr 2024 08:51:46 +0200 From: Felix Fietkau To: netdev@vger.kernel.org, Eric Dumazet , "David S. Miller" , David Ahern , Jakub Kicinski , Paolo Abeni Cc: willemdebruijn.kernel@gmail.com, linux-kernel@vger.kernel.org Subject: [PATCH v3 net-next v3 2/6] net: add support for segmenting TCP fraglist GSO packets Date: Fri, 26 Apr 2024 08:51:36 +0200 Message-ID: <20240426065143.4667-3-nbd@nbd.name> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240426065143.4667-1-nbd@nbd.name> References: <20240426065143.4667-1-nbd@nbd.name> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: kuba@kernel.org Preparation for adding TCP fraglist GRO support. It expects packets to be combined in a similar way as UDP fraglist GSO packets. For IPv4 packets, NAT is handled in the same way as UDP fraglist GSO. Signed-off-by: Felix Fietkau --- net/ipv4/tcp_offload.c | 65 ++++++++++++++++++++++++++++++++++++++++ net/ipv6/tcpv6_offload.c | 3 ++ 2 files changed, 68 insertions(+) diff --git a/net/ipv4/tcp_offload.c b/net/ipv4/tcp_offload.c index fab0973f995b..c493e95e09a5 100644 --- a/net/ipv4/tcp_offload.c +++ b/net/ipv4/tcp_offload.c @@ -28,6 +28,68 @@ static void tcp_gso_tstamp(struct sk_buff *skb, unsigned int ts_seq, } } +static void __tcpv4_gso_segment_csum(struct sk_buff *seg, + __be32 *oldip, __be32 *newip, + __be16 *oldport, __be16 *newport) +{ + struct tcphdr *th; + struct iphdr *iph; + + if (*oldip == *newip && *oldport == *newport) + return; + + th = tcp_hdr(seg); + iph = ip_hdr(seg); + + inet_proto_csum_replace4(&th->check, seg, *oldip, *newip, true); + inet_proto_csum_replace2(&th->check, seg, *oldport, *newport, false); + *oldport = *newport; + + csum_replace4(&iph->check, *oldip, *newip); + *oldip = *newip; +} + +static struct sk_buff *__tcpv4_gso_segment_list_csum(struct sk_buff *segs) +{ + struct sk_buff *seg; + struct tcphdr *th, *th2; + struct iphdr *iph, *iph2; + + seg = segs; + th = tcp_hdr(seg); + iph = ip_hdr(seg); + th2 = tcp_hdr(seg->next); + iph2 = ip_hdr(seg->next); + + if (!(*(u32 *)&th->source ^ *(u32 *)&th2->source) && + iph->daddr == iph2->daddr && iph->saddr == iph2->saddr) + return segs; + + while ((seg = seg->next)) { + th2 = tcp_hdr(seg); + iph2 = ip_hdr(seg); + + __tcpv4_gso_segment_csum(seg, + &iph2->saddr, &iph->saddr, + &th2->source, &th->source); + __tcpv4_gso_segment_csum(seg, + &iph2->daddr, &iph->daddr, + &th2->dest, &th->dest); + } + + return segs; +} + +static struct sk_buff *__tcp4_gso_segment_list(struct sk_buff *skb, + netdev_features_t features) +{ + skb = skb_segment_list(skb, features, skb_mac_header_len(skb)); + if (IS_ERR(skb)) + return skb; + + return __tcpv4_gso_segment_list_csum(skb); +} + static struct sk_buff *tcp4_gso_segment(struct sk_buff *skb, netdev_features_t features) { @@ -37,6 +99,9 @@ static struct sk_buff *tcp4_gso_segment(struct sk_buff *skb, if (!pskb_may_pull(skb, sizeof(struct tcphdr))) return ERR_PTR(-EINVAL); + if (skb_shinfo(skb)->gso_type & SKB_GSO_FRAGLIST) + return __tcp4_gso_segment_list(skb, features); + if (unlikely(skb->ip_summed != CHECKSUM_PARTIAL)) { const struct iphdr *iph = ip_hdr(skb); struct tcphdr *th = tcp_hdr(skb); diff --git a/net/ipv6/tcpv6_offload.c b/net/ipv6/tcpv6_offload.c index 4b07d1e6c952..b3b8e1f6b92a 100644 --- a/net/ipv6/tcpv6_offload.c +++ b/net/ipv6/tcpv6_offload.c @@ -51,6 +51,9 @@ static struct sk_buff *tcp6_gso_segment(struct sk_buff *skb, if (!pskb_may_pull(skb, sizeof(*th))) return ERR_PTR(-EINVAL); + if (skb_shinfo(skb)->gso_type & SKB_GSO_FRAGLIST) + return skb_segment_list(skb, features, skb_mac_header_len(skb)); + if (unlikely(skb->ip_summed != CHECKSUM_PARTIAL)) { const struct ipv6hdr *ipv6h = ipv6_hdr(skb); struct tcphdr *th = tcp_hdr(skb); From patchwork Fri Apr 26 06:51:37 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Felix Fietkau X-Patchwork-Id: 13644156 X-Patchwork-Delegate: kuba@kernel.org Received: from nbd.name (nbd.name [46.4.11.11]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8BD8A13AA2B; Fri, 26 Apr 2024 06:51:53 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=46.4.11.11 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1714114315; cv=none; b=NnWhySunLUC9g3EKdSunHeuWdZoDk5afpPzJw2iLeb/DhaiNBHwgXCYh4rClmXCfDT5rYHXYyu8sTq0iiF3bOrIieFITlEzt89AxxtugfEa8hOMUAYtWTE8ZGvapSUU/aBAcfxR7XMkIfTvnf5e08c/YksUGjaz96GRj6r3wjhg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1714114315; c=relaxed/simple; bh=gC+fCg5cz12fZc/PuhyH/jEzDll18Gv/XQabXEaQBoE=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=atgTjMxYe0fZ8oEXNtfcMVaDQqLBmPFyFQ9M8ubtxkW5YfGnsC49iIpLQjqdrViQwKG0kOPFs9bLPu5a81XaYcqbpGGdAf4sx7ukGEBV121bzQnYJdtcrIjvqKvg7rC08jcpRoQxApOmCgBY7BMcWHS0Xj9cAcKCrEFFO+Ugcps= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=nbd.name; spf=none smtp.mailfrom=nbd.name; dkim=pass (1024-bit key) header.d=nbd.name header.i=@nbd.name header.b=q7Fc99Mp; arc=none smtp.client-ip=46.4.11.11 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=nbd.name Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=nbd.name Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=nbd.name header.i=@nbd.name header.b="q7Fc99Mp" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=nbd.name; s=20160729; h=Content-Transfer-Encoding:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Sender:Reply-To:Content-Type:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:List-Subscribe: List-Post:List-Owner:List-Archive; bh=0KhOWxd59asjecYDzN7/r5pbN5b2cJi5bmFWlIpb04M=; b=q7Fc99MpiF1Eqzcxlv+7LPwVr4 1mlW+CtuXagRp/h01wEtsYcYu3REBcYw4ctAADBf6QpWaK0tF+krFiCaNjCBvB1a3lj28VU7WGR6T R9Pmd53J5crqcLTz8RDSzv36Q4xylHvnOQoXMxN6aoXiEI6/4r4fjq7ih3dtoaX7h+2g=; Received: from p54ae9c93.dip0.t-ipconnect.de ([84.174.156.147] helo=localhost.localdomain) by ds12 with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256 (Exim 4.96) (envelope-from ) id 1s0FQw-007ltt-2w; Fri, 26 Apr 2024 08:51:46 +0200 From: Felix Fietkau To: netdev@vger.kernel.org, Eric Dumazet , "David S. Miller" , David Ahern , Jakub Kicinski , Paolo Abeni Cc: willemdebruijn.kernel@gmail.com, linux-kernel@vger.kernel.org Subject: [PATCH v3 net-next v3 3/6] net: add code for TCP fraglist GRO Date: Fri, 26 Apr 2024 08:51:37 +0200 Message-ID: <20240426065143.4667-4-nbd@nbd.name> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240426065143.4667-1-nbd@nbd.name> References: <20240426065143.4667-1-nbd@nbd.name> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: kuba@kernel.org This implements fraglist GRO similar to how it's handled in UDP, however no functional changes are added yet. The next change adds a heuristic for using fraglist GRO instead of regular GRO. Signed-off-by: Felix Fietkau --- net/ipv4/tcp_offload.c | 22 ++++++++++++++++++++++ net/ipv6/tcpv6_offload.c | 9 +++++++++ 2 files changed, 31 insertions(+) diff --git a/net/ipv4/tcp_offload.c b/net/ipv4/tcp_offload.c index c493e95e09a5..ffd6b7a4163a 100644 --- a/net/ipv4/tcp_offload.c +++ b/net/ipv4/tcp_offload.c @@ -332,6 +332,19 @@ struct sk_buff *tcp_gro_receive(struct list_head *head, struct sk_buff *skb) flush |= (ntohl(th2->seq) + skb_gro_len(p)) ^ ntohl(th->seq); flush |= skb_cmp_decrypted(p, skb); + if (NAPI_GRO_CB(p)->is_flist) { + flush |= (__force int)(flags ^ tcp_flag_word(th2)); + flush |= skb->ip_summed != p->ip_summed; + flush |= skb->csum_level != p->csum_level; + flush |= !pskb_may_pull(skb, skb_gro_offset(skb)); + flush |= NAPI_GRO_CB(p)->count >= 64; + + if (flush || skb_gro_receive_list(p, skb)) + mss = 1; + + goto out_check_final; + } + if (flush || skb_gro_receive(p, skb)) { mss = 1; goto out_check_final; @@ -398,6 +411,15 @@ INDIRECT_CALLABLE_SCOPE int tcp4_gro_complete(struct sk_buff *skb, int thoff) const struct iphdr *iph = ip_hdr(skb); struct tcphdr *th = tcp_hdr(skb); + if (NAPI_GRO_CB(skb)->is_flist) { + skb_shinfo(skb)->gso_type |= SKB_GSO_FRAGLIST | SKB_GSO_TCPV4; + skb_shinfo(skb)->gso_segs = NAPI_GRO_CB(skb)->count; + + __skb_incr_checksum_unnecessary(skb); + + return 0; + } + th->check = ~tcp_v4_check(skb->len - thoff, iph->saddr, iph->daddr, 0); diff --git a/net/ipv6/tcpv6_offload.c b/net/ipv6/tcpv6_offload.c index b3b8e1f6b92a..c97d55cf036f 100644 --- a/net/ipv6/tcpv6_offload.c +++ b/net/ipv6/tcpv6_offload.c @@ -32,6 +32,15 @@ INDIRECT_CALLABLE_SCOPE int tcp6_gro_complete(struct sk_buff *skb, int thoff) const struct ipv6hdr *iph = ipv6_hdr(skb); struct tcphdr *th = tcp_hdr(skb); + if (NAPI_GRO_CB(skb)->is_flist) { + skb_shinfo(skb)->gso_type |= SKB_GSO_FRAGLIST | SKB_GSO_TCPV6; + skb_shinfo(skb)->gso_segs = NAPI_GRO_CB(skb)->count; + + __skb_incr_checksum_unnecessary(skb); + + return 0; + } + th->check = ~tcp_v6_check(skb->len - thoff, &iph->saddr, &iph->daddr, 0); skb_shinfo(skb)->gso_type |= SKB_GSO_TCPV6; From patchwork Fri Apr 26 06:51:38 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Felix Fietkau X-Patchwork-Id: 13644159 X-Patchwork-Delegate: kuba@kernel.org Received: from nbd.name (nbd.name [46.4.11.11]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8AAAA13AA3E; Fri, 26 Apr 2024 06:51:54 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=46.4.11.11 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1714114316; cv=none; b=pBRKh+V7PD6L5Asi0Jo3UZIVfxjJucCKXsgjpm8G6s0qu+zcqdoRSeYiNthfbKBKBCMNF7mI7z5YyjYs6Ufx3JeoXqCqhirLH81pRo7wD/fMQmF5rDNHh9YEjvf5Ztq5SSn4ZYVaS0akU5iG6Z5aIiFcSzOEDy9fgmLCTge6VL4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1714114316; c=relaxed/simple; bh=VuEbKNV4XwlPXs+jXQPrrZF3c45OJOnoayT18xqgRjw=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=MhcvpEVI0WGWc78RWNvJ25he/E3DLH6Ml17dv20YZczNJZ1O/s0OqjZdx32zlRfYAsVdTFV7woT+NN28Jp78LeWfFCGIPvp0Ipy6h74hxPqulquFxk6zBtzsAwZPj7/u9WPRz5ZLFqAsd/gMVMlAQyKtPSfVKrxW7P8Q2tGrLS0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=nbd.name; spf=none smtp.mailfrom=nbd.name; dkim=pass (1024-bit key) header.d=nbd.name header.i=@nbd.name header.b=JSGPig9Z; arc=none smtp.client-ip=46.4.11.11 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=nbd.name Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=nbd.name Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=nbd.name header.i=@nbd.name header.b="JSGPig9Z" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=nbd.name; s=20160729; h=Content-Transfer-Encoding:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Sender:Reply-To:Content-Type:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:List-Subscribe: List-Post:List-Owner:List-Archive; bh=kuFarNp67I2ntE4ihdHxeLO9vh/dXbyzjFjq3+OVjSY=; b=JSGPig9Za8M2F+Ki534JH/2Xjb qywuUMMyy3vPK6r1pEKJglddhqWa638b13fJ3F3Y7ToMYaaCieGAu3Qhmsr1cgXDTvvGKGIu0kzgC m7RHPW+AmE1rwikCEOTFIrAAzjsR6xhW/OPZY2ml0C9GSOOCo4TEAI9GCsG9XRwQuZTI=; Received: from p54ae9c93.dip0.t-ipconnect.de ([84.174.156.147] helo=localhost.localdomain) by ds12 with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256 (Exim 4.96) (envelope-from ) id 1s0FQx-007ltt-24; Fri, 26 Apr 2024 08:51:47 +0200 From: Felix Fietkau To: netdev@vger.kernel.org, Eric Dumazet , "David S. Miller" , Jakub Kicinski , Paolo Abeni , David Ahern Cc: willemdebruijn.kernel@gmail.com, linux-kernel@vger.kernel.org Subject: [PATCH v3 net-next v3 4/6] net: create tcp_gro_lookup helper function Date: Fri, 26 Apr 2024 08:51:38 +0200 Message-ID: <20240426065143.4667-5-nbd@nbd.name> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240426065143.4667-1-nbd@nbd.name> References: <20240426065143.4667-1-nbd@nbd.name> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: kuba@kernel.org This pulls the flow port matching out of tcp_gro_receive, so that it can be reused for the next change, which adds the TCP fraglist GRO heuristic. Signed-off-by: Felix Fietkau Reviewed-by: Eric Dumazet --- include/net/tcp.h | 1 + net/ipv4/tcp_offload.c | 41 +++++++++++++++++++++++++---------------- 2 files changed, 26 insertions(+), 16 deletions(-) diff --git a/include/net/tcp.h b/include/net/tcp.h index ffc9371fe9de..a3f09aa44487 100644 --- a/include/net/tcp.h +++ b/include/net/tcp.h @@ -2198,6 +2198,7 @@ void tcp_v4_destroy_sock(struct sock *sk); struct sk_buff *tcp_gso_segment(struct sk_buff *skb, netdev_features_t features); +struct sk_buff *tcp_gro_lookup(struct list_head *head, struct tcphdr *th); struct sk_buff *tcp_gro_receive(struct list_head *head, struct sk_buff *skb); INDIRECT_CALLABLE_DECLARE(int tcp4_gro_complete(struct sk_buff *skb, int thoff)); INDIRECT_CALLABLE_DECLARE(struct sk_buff *tcp4_gro_receive(struct list_head *head, struct sk_buff *skb)); diff --git a/net/ipv4/tcp_offload.c b/net/ipv4/tcp_offload.c index ffd6b7a4163a..3ee5a1b21976 100644 --- a/net/ipv4/tcp_offload.c +++ b/net/ipv4/tcp_offload.c @@ -243,6 +243,27 @@ struct sk_buff *tcp_gso_segment(struct sk_buff *skb, return segs; } +struct sk_buff *tcp_gro_lookup(struct list_head *head, struct tcphdr *th) +{ + struct tcphdr *th2; + struct sk_buff *p; + + list_for_each_entry(p, head, list) { + if (!NAPI_GRO_CB(p)->same_flow) + continue; + + th2 = tcp_hdr(p); + if (*(u32 *)&th->source ^ *(u32 *)&th2->source) { + NAPI_GRO_CB(p)->same_flow = 0; + continue; + } + + return p; + } + + return NULL; +} + struct sk_buff *tcp_gro_receive(struct list_head *head, struct sk_buff *skb) { struct sk_buff *pp = NULL; @@ -280,24 +301,12 @@ struct sk_buff *tcp_gro_receive(struct list_head *head, struct sk_buff *skb) len = skb_gro_len(skb); flags = tcp_flag_word(th); - list_for_each_entry(p, head, list) { - if (!NAPI_GRO_CB(p)->same_flow) - continue; - - th2 = tcp_hdr(p); - - if (*(u32 *)&th->source ^ *(u32 *)&th2->source) { - NAPI_GRO_CB(p)->same_flow = 0; - continue; - } - - goto found; - } - p = NULL; - goto out_check_final; + p = tcp_gro_lookup(head, th); + if (!p) + goto out_check_final; -found: /* Include the IP ID check below from the inner most IP hdr */ + th2 = tcp_hdr(p); flush = NAPI_GRO_CB(p)->flush; flush |= (__force int)(flags & TCP_FLAG_CWR); flush |= (__force int)((flags ^ tcp_flag_word(th2)) & From patchwork Fri Apr 26 06:51:39 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Felix Fietkau X-Patchwork-Id: 13644158 X-Patchwork-Delegate: kuba@kernel.org Received: from nbd.name (nbd.name [46.4.11.11]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4EF5E13AA42; Fri, 26 Apr 2024 06:51:54 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=46.4.11.11 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1714114315; cv=none; b=KIAeE7d5wUAHX+GGnb2/dNCF2pHQ6HCWT1Iwwjp7J1gz/yC169rXuy5YpUUJAW3Os++Ee+Al55/bJhX/sI039nykCrxeQ+8bWrONvpkntoXycVuPXAQXGyRv0EBlvApP/RsvaSYu5UklnmLDv6dbKTh+/il/ctsS6W1AtfgO2jM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1714114315; c=relaxed/simple; bh=ceSTfBzAhi8Ctp5vddNFEmKNI61lSCkNIu3TusNCR+I=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=qfmFBxtq7FPf4HTo2vadkkBhj2CD2nSpVujavR2EEf9WbJOxDChYSJW9nHFzKK49rDwNZ0Dzrz+ZG4hF/LqTRWREVqcuiF5d2aYWcNuUC5LK1lzNzReXBTk6Kl2szM1A/vEFPFZpG3wfKT+VLFLmqLJx6V/A8g8w2jrD84Vj2I8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=nbd.name; spf=none smtp.mailfrom=nbd.name; dkim=pass (1024-bit key) header.d=nbd.name header.i=@nbd.name header.b=Rw4kYQsa; arc=none smtp.client-ip=46.4.11.11 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=nbd.name Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=nbd.name Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=nbd.name header.i=@nbd.name header.b="Rw4kYQsa" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=nbd.name; s=20160729; h=Content-Transfer-Encoding:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Sender:Reply-To:Content-Type:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:List-Subscribe: List-Post:List-Owner:List-Archive; bh=zM82BVrp9dKSJQz4YNDsH67gi3zq4Tz1t20Ldkmr8Tc=; b=Rw4kYQsaZYQoXQXEw/wzTRocxX PS4+T/O2X5KA5LIEn9fJF7w2t4nt1wr4UKfSkvT4gcwfpVLld8JdHD09a3Dc8NqRF1nNAhUAVfrzW MAim6ZNbDjpE/bAeq/Ab2ALxltfUKlZBOvLFlmJVMwjdjCBU6JMi3hJ4cWz+4YsZu8rc=; Received: from p54ae9c93.dip0.t-ipconnect.de ([84.174.156.147] helo=localhost.localdomain) by ds12 with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256 (Exim 4.96) (envelope-from ) id 1s0FQy-007ltt-24; Fri, 26 Apr 2024 08:51:48 +0200 From: Felix Fietkau To: netdev@vger.kernel.org, Eric Dumazet , "David S. Miller" , Jakub Kicinski , Paolo Abeni , David Ahern Cc: willemdebruijn.kernel@gmail.com, linux-kernel@vger.kernel.org Subject: [PATCH v3 net-next v3 5/6] net: create tcp_gro_header_pull helper function Date: Fri, 26 Apr 2024 08:51:39 +0200 Message-ID: <20240426065143.4667-6-nbd@nbd.name> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240426065143.4667-1-nbd@nbd.name> References: <20240426065143.4667-1-nbd@nbd.name> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: kuba@kernel.org Pull the code out of tcp_gro_receive in order to access the tcp header from tcp4/6_gro_receive. Signed-off-by: Felix Fietkau Reviewed-by: Eric Dumazet --- include/net/tcp.h | 4 ++- net/ipv4/tcp_offload.c | 55 +++++++++++++++++++++++++--------------- net/ipv6/tcpv6_offload.c | 18 +++++++++---- 3 files changed, 50 insertions(+), 27 deletions(-) diff --git a/include/net/tcp.h b/include/net/tcp.h index a3f09aa44487..52371be181a2 100644 --- a/include/net/tcp.h +++ b/include/net/tcp.h @@ -2198,8 +2198,10 @@ void tcp_v4_destroy_sock(struct sock *sk); struct sk_buff *tcp_gso_segment(struct sk_buff *skb, netdev_features_t features); +struct tcphdr *tcp_gro_pull_header(struct sk_buff *skb); struct sk_buff *tcp_gro_lookup(struct list_head *head, struct tcphdr *th); -struct sk_buff *tcp_gro_receive(struct list_head *head, struct sk_buff *skb); +struct sk_buff *tcp_gro_receive(struct list_head *head, struct sk_buff *skb, + struct tcphdr *th); INDIRECT_CALLABLE_DECLARE(int tcp4_gro_complete(struct sk_buff *skb, int thoff)); INDIRECT_CALLABLE_DECLARE(struct sk_buff *tcp4_gro_receive(struct list_head *head, struct sk_buff *skb)); INDIRECT_CALLABLE_DECLARE(int tcp6_gro_complete(struct sk_buff *skb, int thoff)); diff --git a/net/ipv4/tcp_offload.c b/net/ipv4/tcp_offload.c index 3ee5a1b21976..ee5403760775 100644 --- a/net/ipv4/tcp_offload.c +++ b/net/ipv4/tcp_offload.c @@ -264,40 +264,46 @@ struct sk_buff *tcp_gro_lookup(struct list_head *head, struct tcphdr *th) return NULL; } -struct sk_buff *tcp_gro_receive(struct list_head *head, struct sk_buff *skb) +struct tcphdr *tcp_gro_pull_header(struct sk_buff *skb) { - struct sk_buff *pp = NULL; - struct sk_buff *p; + unsigned int thlen, hlen, off; struct tcphdr *th; - struct tcphdr *th2; - unsigned int len; - unsigned int thlen; - __be32 flags; - unsigned int mss = 1; - unsigned int hlen; - unsigned int off; - int flush = 1; - int i; off = skb_gro_offset(skb); hlen = off + sizeof(*th); th = skb_gro_header(skb, hlen, off); if (unlikely(!th)) - goto out; + return NULL; thlen = th->doff * 4; if (thlen < sizeof(*th)) - goto out; + return NULL; hlen = off + thlen; if (!skb_gro_may_pull(skb, hlen)) { th = skb_gro_header_slow(skb, hlen, off); if (unlikely(!th)) - goto out; + return NULL; } skb_gro_pull(skb, thlen); + return th; +} + +struct sk_buff *tcp_gro_receive(struct list_head *head, struct sk_buff *skb, + struct tcphdr *th) +{ + unsigned int thlen = th->doff * 4; + struct sk_buff *pp = NULL; + struct sk_buff *p; + struct tcphdr *th2; + unsigned int len; + __be32 flags; + unsigned int mss = 1; + int flush = 1; + int i; + len = skb_gro_len(skb); flags = tcp_flag_word(th); @@ -375,7 +381,6 @@ struct sk_buff *tcp_gro_receive(struct list_head *head, struct sk_buff *skb) if (p && (!NAPI_GRO_CB(skb)->same_flow || flush)) pp = p; -out: NAPI_GRO_CB(skb)->flush |= (flush != 0); return pp; @@ -404,15 +409,23 @@ EXPORT_SYMBOL(tcp_gro_complete); INDIRECT_CALLABLE_SCOPE struct sk_buff *tcp4_gro_receive(struct list_head *head, struct sk_buff *skb) { + struct tcphdr *th; + /* Don't bother verifying checksum if we're going to flush anyway. */ if (!NAPI_GRO_CB(skb)->flush && skb_gro_checksum_validate(skb, IPPROTO_TCP, - inet_gro_compute_pseudo)) { - NAPI_GRO_CB(skb)->flush = 1; - return NULL; - } + inet_gro_compute_pseudo)) + goto flush; + + th = tcp_gro_pull_header(skb); + if (!th) + goto flush; - return tcp_gro_receive(head, skb); + return tcp_gro_receive(head, skb, th); + +flush: + NAPI_GRO_CB(skb)->flush = 1; + return NULL; } INDIRECT_CALLABLE_SCOPE int tcp4_gro_complete(struct sk_buff *skb, int thoff) diff --git a/net/ipv6/tcpv6_offload.c b/net/ipv6/tcpv6_offload.c index c97d55cf036f..c01ace2e9ff0 100644 --- a/net/ipv6/tcpv6_offload.c +++ b/net/ipv6/tcpv6_offload.c @@ -16,15 +16,23 @@ INDIRECT_CALLABLE_SCOPE struct sk_buff *tcp6_gro_receive(struct list_head *head, struct sk_buff *skb) { + struct tcphdr *th; + /* Don't bother verifying checksum if we're going to flush anyway. */ if (!NAPI_GRO_CB(skb)->flush && skb_gro_checksum_validate(skb, IPPROTO_TCP, - ip6_gro_compute_pseudo)) { - NAPI_GRO_CB(skb)->flush = 1; - return NULL; - } + ip6_gro_compute_pseudo)) + goto flush; + + th = tcp_gro_pull_header(skb); + if (!th) + goto flush; + + return tcp_gro_receive(head, skb, th); - return tcp_gro_receive(head, skb); +flush: + NAPI_GRO_CB(skb)->flush = 1; + return NULL; } INDIRECT_CALLABLE_SCOPE int tcp6_gro_complete(struct sk_buff *skb, int thoff) From patchwork Fri Apr 26 06:51:40 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Felix Fietkau X-Patchwork-Id: 13644160 X-Patchwork-Delegate: kuba@kernel.org Received: from nbd.name (nbd.name [46.4.11.11]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id ACF1113AD13; Fri, 26 Apr 2024 06:51:55 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=46.4.11.11 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1714114317; cv=none; b=cLH6waj+7KSyUZbnKAXrnlDGoQvsfb+YGF4qAW9lKOPPs10t4dXZAunOF7FRqGkRyJAtUtA40e3q4o+8aF0AKAsWXYAIDnpv2RSOUyybDoLTzElRFXDs6nnxUaHPmrzPf55D/eAjJOHb5nnaBqkOtrSoJ3B/vCgEbW6P18zF9cg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1714114317; c=relaxed/simple; bh=u6sxceRCUEfa6tDoPeaxsGfysUqRnA8WVB4Ljbei6H4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=AW0kIoxxBHWzAcd/1KITD+CGM5L/E2woqTHXTxI7NS43Xj7zNObLJngmSxxeTd5xCP/9HUwcZ3isGvcGpuMLKY3aNRlGY5SQqJqrbYldYGMjVyzvtEZBdf530UduhjbfePh2RIvuvYG7odkDa04XqbG1PQTecMTXgvu/gsIE5fw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=nbd.name; spf=none smtp.mailfrom=nbd.name; dkim=pass (1024-bit key) header.d=nbd.name header.i=@nbd.name header.b=s662oJ+o; arc=none smtp.client-ip=46.4.11.11 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=nbd.name Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=nbd.name Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=nbd.name header.i=@nbd.name header.b="s662oJ+o" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=nbd.name; s=20160729; h=Content-Transfer-Encoding:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Sender:Reply-To:Content-Type:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:List-Subscribe: List-Post:List-Owner:List-Archive; bh=9DeZ9jS+7vxRnrb2c4sFQl5rrUE4Xdcx5KZfOfzctuw=; b=s662oJ+obki5MsoDyXPzToe27l wfduu9dsEepCqFRIvFIrvfswibO4+EkxwjnQUwVugx3euqA3fuYf3q9AIf7hoQVgo3rScbL+yZ/30 f8/o2ltC9u0fRrBUl8H1TfTOlUJWyVg8kWOeDtTnz5RQNCyLmLFOm1YR2RFgbw3I1YHM=; Received: from p54ae9c93.dip0.t-ipconnect.de ([84.174.156.147] helo=localhost.localdomain) by ds12 with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256 (Exim 4.96) (envelope-from ) id 1s0FQz-007ltt-1N; Fri, 26 Apr 2024 08:51:49 +0200 From: Felix Fietkau To: netdev@vger.kernel.org, Eric Dumazet , "David S. Miller" , David Ahern , Jakub Kicinski , Paolo Abeni Cc: willemdebruijn.kernel@gmail.com, linux-kernel@vger.kernel.org Subject: [PATCH v3 net-next v3 6/6] net: add heuristic for enabling TCP fraglist GRO Date: Fri, 26 Apr 2024 08:51:40 +0200 Message-ID: <20240426065143.4667-7-nbd@nbd.name> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240426065143.4667-1-nbd@nbd.name> References: <20240426065143.4667-1-nbd@nbd.name> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: kuba@kernel.org When forwarding TCP after GRO, software segmentation is very expensive, especially when the checksum needs to be recalculated. One case where that's currently unavoidable is when routing packets over PPPoE. Performance improves significantly when using fraglist GRO implemented in the same way as for UDP. When NETIF_F_GRO_FRAGLIST is enabled, perform a lookup for an established socket in the same netns as the receiving device. While this may not cover all relevant use cases in multi-netns configurations, it should be good enough for most configurations that need this. Here's a measurement of running 2 TCP streams through a MediaTek MT7622 device (2-core Cortex-A53), which runs NAT with flow offload enabled from one ethernet port to PPPoE on another ethernet port + cake qdisc set to 1Gbps. rx-gro-list off: 630 Mbit/s, CPU 35% idle rx-gro-list on: 770 Mbit/s, CPU 40% idle Signe-off-by: Felix Fietkau --- net/ipv4/tcp_offload.c | 30 ++++++++++++++++++++++++++++++ net/ipv6/tcpv6_offload.c | 33 +++++++++++++++++++++++++++++++++ 2 files changed, 63 insertions(+) diff --git a/net/ipv4/tcp_offload.c b/net/ipv4/tcp_offload.c index ee5403760775..2ae83f4394dc 100644 --- a/net/ipv4/tcp_offload.c +++ b/net/ipv4/tcp_offload.c @@ -406,6 +406,34 @@ void tcp_gro_complete(struct sk_buff *skb) } EXPORT_SYMBOL(tcp_gro_complete); +static void tcp4_check_fraglist_gro(struct list_head *head, struct sk_buff *skb, + struct tcphdr *th) +{ + const struct iphdr *iph = skb_gro_network_header(skb); + struct net *net = dev_net(skb->dev); + struct sk_buff *p; + struct sock *sk; + int iif, sdif; + + if (!(skb->dev->features & NETIF_F_GRO_FRAGLIST)) + return; + + p = tcp_gro_lookup(head, th); + if (p) { + NAPI_GRO_CB(skb)->is_flist = NAPI_GRO_CB(p)->is_flist; + return; + } + + inet_get_iif_sdif(skb, &iif, &sdif); + sk = __inet_lookup_established(net, net->ipv4.tcp_death_row.hashinfo, + iph->saddr, th->source, + iph->daddr, ntohs(th->dest), + iif, sdif); + NAPI_GRO_CB(skb)->is_flist = !sk; + if (sk) + sock_put(sk); +} + INDIRECT_CALLABLE_SCOPE struct sk_buff *tcp4_gro_receive(struct list_head *head, struct sk_buff *skb) { @@ -421,6 +449,8 @@ struct sk_buff *tcp4_gro_receive(struct list_head *head, struct sk_buff *skb) if (!th) goto flush; + tcp4_check_fraglist_gro(head, skb, th); + return tcp_gro_receive(head, skb, th); flush: diff --git a/net/ipv6/tcpv6_offload.c b/net/ipv6/tcpv6_offload.c index c01ace2e9ff0..1ab45cca3936 100644 --- a/net/ipv6/tcpv6_offload.c +++ b/net/ipv6/tcpv6_offload.c @@ -7,12 +7,43 @@ */ #include #include +#include #include #include #include #include #include "ip6_offload.h" +static void tcp6_check_fraglist_gro(struct list_head *head, struct sk_buff *skb, + struct tcphdr *th) +{ +#if IS_ENABLED(CONFIG_IPV6) + const struct ipv6hdr *hdr = skb_gro_network_header(skb); + struct net *net = dev_net(skb->dev); + struct sk_buff *p; + struct sock *sk; + int iif, sdif; + + if (!(skb->dev->features & NETIF_F_GRO_FRAGLIST)) + return; + + p = tcp_gro_lookup(head, th); + if (p) { + NAPI_GRO_CB(skb)->is_flist = NAPI_GRO_CB(p)->is_flist; + return; + } + + inet6_get_iif_sdif(skb, &iif, &sdif); + sk = __inet6_lookup_established(net, net->ipv4.tcp_death_row.hashinfo, + &hdr->saddr, th->source, + &hdr->daddr, ntohs(th->dest), + iif, sdif); + NAPI_GRO_CB(skb)->is_flist = !sk; + if (sk) + sock_put(sk); +#endif /* IS_ENABLED(CONFIG_IPV6) */ +} + INDIRECT_CALLABLE_SCOPE struct sk_buff *tcp6_gro_receive(struct list_head *head, struct sk_buff *skb) { @@ -28,6 +59,8 @@ struct sk_buff *tcp6_gro_receive(struct list_head *head, struct sk_buff *skb) if (!th) goto flush; + tcp6_check_fraglist_gro(head, skb, th); + return tcp_gro_receive(head, skb, th); flush: