From patchwork Tue Feb 16 14:34:00 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Lobakin X-Patchwork-Id: 12090079 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 34707C433E0 for ; Tue, 16 Feb 2021 14:35:34 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 0C64C64DF5 for ; Tue, 16 Feb 2021 14:35:34 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230360AbhBPOe5 (ORCPT ); Tue, 16 Feb 2021 09:34:57 -0500 Received: from mail-40134.protonmail.ch ([185.70.40.134]:55821 "EHLO mail-40134.protonmail.ch" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230336AbhBPOex (ORCPT ); Tue, 16 Feb 2021 09:34:53 -0500 Date: Tue, 16 Feb 2021 14:34:00 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pm.me; s=protonmail; t=1613486050; bh=lYDCa6iLqFT+6be7jVyPBd42LZlNQ5lJltfvJUcfGWA=; h=Date:To:From:Cc:Reply-To:Subject:In-Reply-To:References:From; b=GKBYwAoLlfpGLfTCo0twTluBuP3DI2gbTrilQvDFjOv16JAno+bxYQ4NAV1Y7JUF8 c/K3uIhRwqOb7dD6HMQpWl56GWfagGV16ifMXY6CtNtyXeDMuvEHxyjOIkGouILrb7 khMABDkJUrxc7isbmT0AfbvGbE/btW4uNArWaRu43JpLPaw39/fo0FoOlwj0dKQSan Lk3WNWG4CJJJLHKGhwnet3g3bty9/ZET++PLuVY6BW4mDXbfTYHp683UohjPXGLf9g uYTF91pOY60vtVpKSs9WYmC7m53Z9A+oG6TjqqGSln3IGPOT1QvGmLIyXNuAOYZdEq J+PBIDGRqezKw== To: Magnus Karlsson , =?utf-8?b?QmrDtnJuIFTDtnBl?= =?utf-8?b?bA==?= From: Alexander Lobakin Cc: "Michael S. Tsirkin" , Jason Wang , "David S. Miller" , Jakub Kicinski , Jonathan Lemon , Alexei Starovoitov , Daniel Borkmann , Jesper Dangaard Brouer , John Fastabend , Andrii Nakryiko , Martin KaFai Lau , Song Liu , Yonghong Song , KP Singh , Paolo Abeni , Eric Dumazet , Xuan Zhuo , Dust Li , Alexander Lobakin , virtualization@lists.linux-foundation.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org, bpf@vger.kernel.org Reply-To: Alexander Lobakin Subject: [PATCH v5 bpf-next 1/6] netdev_priv_flags: add missing IFF_PHONY_HEADROOM self-definition Message-ID: <20210216143333.5861-2-alobakin@pm.me> In-Reply-To: <20210216143333.5861-1-alobakin@pm.me> References: <20210216143333.5861-1-alobakin@pm.me> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net This is harmless for now, but comes fatal for the subsequent patch. Fixes: 871b642adebe3 ("netdev: introduce ndo_set_rx_headroom") Signed-off-by: Alexander Lobakin --- include/linux/netdevice.h | 1 + 1 file changed, 1 insertion(+) diff --git a/include/linux/netdevice.h b/include/linux/netdevice.h index b9bcbfde7849..b895973390ee 100644 --- a/include/linux/netdevice.h +++ b/include/linux/netdevice.h @@ -1584,6 +1584,7 @@ enum netdev_priv_flags { #define IFF_L3MDEV_SLAVE IFF_L3MDEV_SLAVE #define IFF_TEAM IFF_TEAM #define IFF_RXFH_CONFIGURED IFF_RXFH_CONFIGURED +#define IFF_PHONY_HEADROOM IFF_PHONY_HEADROOM #define IFF_MACSEC IFF_MACSEC #define IFF_NO_RX_HANDLER IFF_NO_RX_HANDLER #define IFF_FAILOVER IFF_FAILOVER From patchwork Tue Feb 16 14:34:16 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Lobakin X-Patchwork-Id: 12090077 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,UPPERCASE_50_75, URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 52286C433DB for ; Tue, 16 Feb 2021 14:35:34 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 220A564E04 for ; Tue, 16 Feb 2021 14:35:34 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230227AbhBPOfN (ORCPT ); Tue, 16 Feb 2021 09:35:13 -0500 Received: from mail-40134.protonmail.ch ([185.70.40.134]:34050 "EHLO mail-40134.protonmail.ch" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230362AbhBPOfF (ORCPT ); Tue, 16 Feb 2021 09:35:05 -0500 Date: Tue, 16 Feb 2021 14:34:16 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pm.me; s=protonmail; t=1613486061; bh=F79GxxLFgGar0uDA252Y0utA5OPZJ+zP9u/t7cIgBJg=; h=Date:To:From:Cc:Reply-To:Subject:In-Reply-To:References:From; b=eEMOhQ21Ih63jKmwgdrqH1NtMqjuglv/W9LhnJEinSZ0dPZFYSG/ovDV5+oLVPLmP IG9j65SkR2g28LhuC/+9EL/ydrPau2j9cKzAWGCblPK6nObfu53yJxFSSf7i/7cmbZ EBgINC+pO7KgOrqE9scTOz6xK5hflJfklQ/Btcm4qhn/nGQxtRwdMj9ZW3jYlLjF2X 0nzQAFeXtDHBC02GgAO/knfwCo5v+e4WceiB6S45p2/gBXC2CSCiX0IP1qlSKYbpOr R6fLaE3A/x1HZsrk/yqfhnoBVjHbUgBGRIfB6sDeFKIRxDZt0KH/Dm0zMv4VbLlf6X NGxRtnLIZDddw== To: Magnus Karlsson , =?utf-8?b?QmrDtnJuIFTDtnBl?= =?utf-8?b?bA==?= From: Alexander Lobakin Cc: "Michael S. Tsirkin" , Jason Wang , "David S. Miller" , Jakub Kicinski , Jonathan Lemon , Alexei Starovoitov , Daniel Borkmann , Jesper Dangaard Brouer , John Fastabend , Andrii Nakryiko , Martin KaFai Lau , Song Liu , Yonghong Song , KP Singh , Paolo Abeni , Eric Dumazet , Xuan Zhuo , Dust Li , Alexander Lobakin , virtualization@lists.linux-foundation.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org, bpf@vger.kernel.org, kernel test robot Reply-To: Alexander Lobakin Subject: [PATCH v5 bpf-next 2/6] netdevice: check for net_device::priv_flags bitfield overflow Message-ID: <20210216143333.5861-3-alobakin@pm.me> In-Reply-To: <20210216143333.5861-1-alobakin@pm.me> References: <20210216143333.5861-1-alobakin@pm.me> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net We almost ran out of unsigned int bitwidth. Define priv flags and check for potential overflow in the fashion of netdev_features_t. Defined this way, priv_flags can be easily expanded later with just changing its typedef. Signed-off-by: Alexander Lobakin Reported-by: kernel test robot # Inverted assert condition --- include/linux/netdevice.h | 135 ++++++++++++++++++++------------------ 1 file changed, 72 insertions(+), 63 deletions(-) diff --git a/include/linux/netdevice.h b/include/linux/netdevice.h index b895973390ee..0a9b2b31f411 100644 --- a/include/linux/netdevice.h +++ b/include/linux/netdevice.h @@ -1527,70 +1527,79 @@ struct net_device_ops { * @IFF_LIVE_RENAME_OK: rename is allowed while device is up and running */ enum netdev_priv_flags { - IFF_802_1Q_VLAN = 1<<0, - IFF_EBRIDGE = 1<<1, - IFF_BONDING = 1<<2, - IFF_ISATAP = 1<<3, - IFF_WAN_HDLC = 1<<4, - IFF_XMIT_DST_RELEASE = 1<<5, - IFF_DONT_BRIDGE = 1<<6, - IFF_DISABLE_NETPOLL = 1<<7, - IFF_MACVLAN_PORT = 1<<8, - IFF_BRIDGE_PORT = 1<<9, - IFF_OVS_DATAPATH = 1<<10, - IFF_TX_SKB_SHARING = 1<<11, - IFF_UNICAST_FLT = 1<<12, - IFF_TEAM_PORT = 1<<13, - IFF_SUPP_NOFCS = 1<<14, - IFF_LIVE_ADDR_CHANGE = 1<<15, - IFF_MACVLAN = 1<<16, - IFF_XMIT_DST_RELEASE_PERM = 1<<17, - IFF_L3MDEV_MASTER = 1<<18, - IFF_NO_QUEUE = 1<<19, - IFF_OPENVSWITCH = 1<<20, - IFF_L3MDEV_SLAVE = 1<<21, - IFF_TEAM = 1<<22, - IFF_RXFH_CONFIGURED = 1<<23, - IFF_PHONY_HEADROOM = 1<<24, - IFF_MACSEC = 1<<25, - IFF_NO_RX_HANDLER = 1<<26, - IFF_FAILOVER = 1<<27, - IFF_FAILOVER_SLAVE = 1<<28, - IFF_L3MDEV_RX_HANDLER = 1<<29, - IFF_LIVE_RENAME_OK = 1<<30, + IFF_802_1Q_VLAN_BIT, + IFF_EBRIDGE_BIT, + IFF_BONDING_BIT, + IFF_ISATAP_BIT, + IFF_WAN_HDLC_BIT, + IFF_XMIT_DST_RELEASE_BIT, + IFF_DONT_BRIDGE_BIT, + IFF_DISABLE_NETPOLL_BIT, + IFF_MACVLAN_PORT_BIT, + IFF_BRIDGE_PORT_BIT, + IFF_OVS_DATAPATH_BIT, + IFF_TX_SKB_SHARING_BIT, + IFF_UNICAST_FLT_BIT, + IFF_TEAM_PORT_BIT, + IFF_SUPP_NOFCS_BIT, + IFF_LIVE_ADDR_CHANGE_BIT, + IFF_MACVLAN_BIT, + IFF_XMIT_DST_RELEASE_PERM_BIT, + IFF_L3MDEV_MASTER_BIT, + IFF_NO_QUEUE_BIT, + IFF_OPENVSWITCH_BIT, + IFF_L3MDEV_SLAVE_BIT, + IFF_TEAM_BIT, + IFF_RXFH_CONFIGURED_BIT, + IFF_PHONY_HEADROOM_BIT, + IFF_MACSEC_BIT, + IFF_NO_RX_HANDLER_BIT, + IFF_FAILOVER_BIT, + IFF_FAILOVER_SLAVE_BIT, + IFF_L3MDEV_RX_HANDLER_BIT, + IFF_LIVE_RENAME_OK_BIT, + + NETDEV_PRIV_FLAG_COUNT, }; -#define IFF_802_1Q_VLAN IFF_802_1Q_VLAN -#define IFF_EBRIDGE IFF_EBRIDGE -#define IFF_BONDING IFF_BONDING -#define IFF_ISATAP IFF_ISATAP -#define IFF_WAN_HDLC IFF_WAN_HDLC -#define IFF_XMIT_DST_RELEASE IFF_XMIT_DST_RELEASE -#define IFF_DONT_BRIDGE IFF_DONT_BRIDGE -#define IFF_DISABLE_NETPOLL IFF_DISABLE_NETPOLL -#define IFF_MACVLAN_PORT IFF_MACVLAN_PORT -#define IFF_BRIDGE_PORT IFF_BRIDGE_PORT -#define IFF_OVS_DATAPATH IFF_OVS_DATAPATH -#define IFF_TX_SKB_SHARING IFF_TX_SKB_SHARING -#define IFF_UNICAST_FLT IFF_UNICAST_FLT -#define IFF_TEAM_PORT IFF_TEAM_PORT -#define IFF_SUPP_NOFCS IFF_SUPP_NOFCS -#define IFF_LIVE_ADDR_CHANGE IFF_LIVE_ADDR_CHANGE -#define IFF_MACVLAN IFF_MACVLAN -#define IFF_XMIT_DST_RELEASE_PERM IFF_XMIT_DST_RELEASE_PERM -#define IFF_L3MDEV_MASTER IFF_L3MDEV_MASTER -#define IFF_NO_QUEUE IFF_NO_QUEUE -#define IFF_OPENVSWITCH IFF_OPENVSWITCH -#define IFF_L3MDEV_SLAVE IFF_L3MDEV_SLAVE -#define IFF_TEAM IFF_TEAM -#define IFF_RXFH_CONFIGURED IFF_RXFH_CONFIGURED -#define IFF_PHONY_HEADROOM IFF_PHONY_HEADROOM -#define IFF_MACSEC IFF_MACSEC -#define IFF_NO_RX_HANDLER IFF_NO_RX_HANDLER -#define IFF_FAILOVER IFF_FAILOVER -#define IFF_FAILOVER_SLAVE IFF_FAILOVER_SLAVE -#define IFF_L3MDEV_RX_HANDLER IFF_L3MDEV_RX_HANDLER -#define IFF_LIVE_RENAME_OK IFF_LIVE_RENAME_OK +typedef u32 netdev_priv_flags_t; +static_assert(sizeof(netdev_priv_flags_t) * BITS_PER_BYTE >= + NETDEV_PRIV_FLAG_COUNT); + +#define __IFF_BIT(bit) ((netdev_priv_flags_t)1 << (bit)) +#define __IFF(name) __IFF_BIT(IFF_##name##_BIT) + +#define IFF_802_1Q_VLAN __IFF(802_1Q_VLAN) +#define IFF_EBRIDGE __IFF(EBRIDGE) +#define IFF_BONDING __IFF(BONDING) +#define IFF_ISATAP __IFF(ISATAP) +#define IFF_WAN_HDLC __IFF(WAN_HDLC) +#define IFF_XMIT_DST_RELEASE __IFF(XMIT_DST_RELEASE) +#define IFF_DONT_BRIDGE __IFF(DONT_BRIDGE) +#define IFF_DISABLE_NETPOLL __IFF(DISABLE_NETPOLL) +#define IFF_MACVLAN_PORT __IFF(MACVLAN_PORT) +#define IFF_BRIDGE_PORT __IFF(BRIDGE_PORT) +#define IFF_OVS_DATAPATH __IFF(OVS_DATAPATH) +#define IFF_TX_SKB_SHARING __IFF(TX_SKB_SHARING) +#define IFF_UNICAST_FLT __IFF(UNICAST_FLT) +#define IFF_TEAM_PORT __IFF(TEAM_PORT) +#define IFF_SUPP_NOFCS __IFF(SUPP_NOFCS) +#define IFF_LIVE_ADDR_CHANGE __IFF(LIVE_ADDR_CHANGE) +#define IFF_MACVLAN __IFF(MACVLAN) +#define IFF_XMIT_DST_RELEASE_PERM __IFF(XMIT_DST_RELEASE_PERM) +#define IFF_L3MDEV_MASTER __IFF(L3MDEV_MASTER) +#define IFF_NO_QUEUE __IFF(NO_QUEUE) +#define IFF_OPENVSWITCH __IFF(OPENVSWITCH) +#define IFF_L3MDEV_SLAVE __IFF(L3MDEV_SLAVE) +#define IFF_TEAM __IFF(TEAM) +#define IFF_RXFH_CONFIGURED __IFF(RXFH_CONFIGURED) +#define IFF_PHONY_HEADROOM __IFF(PHONY_HEADROOM) +#define IFF_MACSEC __IFF(MACSEC) +#define IFF_NO_RX_HANDLER __IFF(NO_RX_HANDLER) +#define IFF_FAILOVER __IFF(FAILOVER) +#define IFF_FAILOVER_SLAVE __IFF(FAILOVER_SLAVE) +#define IFF_L3MDEV_RX_HANDLER __IFF(L3MDEV_RX_HANDLER) +#define IFF_LIVE_RENAME_OK __IFF(LIVE_RENAME_OK) /** * struct net_device - The DEVICE structure. @@ -1925,7 +1934,7 @@ struct net_device { const struct header_ops *header_ops; unsigned int flags; - unsigned int priv_flags; + netdev_priv_flags_t priv_flags; unsigned short gflags; unsigned short padded; From patchwork Tue Feb 16 14:34:27 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Lobakin X-Patchwork-Id: 12090081 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A9961C433E9 for ; Tue, 16 Feb 2021 14:35:34 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 7A8EF64DF5 for ; Tue, 16 Feb 2021 14:35:34 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230280AbhBPOfX (ORCPT ); Tue, 16 Feb 2021 09:35:23 -0500 Received: from mail2.protonmail.ch ([185.70.40.22]:16883 "EHLO mail2.protonmail.ch" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230228AbhBPOfR (ORCPT ); Tue, 16 Feb 2021 09:35:17 -0500 Date: Tue, 16 Feb 2021 14:34:27 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pm.me; s=protonmail; t=1613486072; bh=QDosC33QktE2waBddw6zce1oGZOsZXv08mklFMTqals=; h=Date:To:From:Cc:Reply-To:Subject:In-Reply-To:References:From; b=RltsaRItpmNwK93bKkNLvPXbxVDZOJMkDltQakklX5Gcp1z8Jg2TKu3tB4X7iF/e3 gC0rLfPxXhveQQI0KUwZ+jqm0tK25Doh3tTdbjy5uq/qJpm4w6Oj4Z/FSi4W+733um mXdAiO71p0LxB+tt+FfCwdFP9CQGLeAEYOCYz5tLfAKE+wHK99cjRrLOgnUTo0gxrX tUT0w5yOPsZanGgpAYTRudqqrYN1KYsGYBKBfKhz20GCVRq1rrh6//IBtYCs9A4UnD Bsmoplwc7fT/K05M9uFYud3sDj+/cXY5UyBetdrcyTt0JL6zkTATfHSHzSPcitXR1j CTtG26muO43EQ== To: Magnus Karlsson , =?utf-8?b?QmrDtnJuIFTDtnBl?= =?utf-8?b?bA==?= From: Alexander Lobakin Cc: "Michael S. Tsirkin" , Jason Wang , "David S. Miller" , Jakub Kicinski , Jonathan Lemon , Alexei Starovoitov , Daniel Borkmann , Jesper Dangaard Brouer , John Fastabend , Andrii Nakryiko , Martin KaFai Lau , Song Liu , Yonghong Song , KP Singh , Paolo Abeni , Eric Dumazet , Xuan Zhuo , Dust Li , Alexander Lobakin , virtualization@lists.linux-foundation.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org, bpf@vger.kernel.org Reply-To: Alexander Lobakin Subject: [PATCH v5 bpf-next 3/6] net: add priv_flags for allow tx skb without linear Message-ID: <20210216143333.5861-4-alobakin@pm.me> In-Reply-To: <20210216143333.5861-1-alobakin@pm.me> References: <20210216143333.5861-1-alobakin@pm.me> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net From: Xuan Zhuo In some cases, we hope to construct skb directly based on the existing memory without copying data. In this case, the page will be placed directly in the skb, and the linear space of skb is empty. But unfortunately, many the network card does not support this operation. For example Mellanox Technologies MT27710 Family [ConnectX-4 Lx] will get the following error message: mlx5_core 0000:3b:00.1 eth1: Error cqe on cqn 0x817, ci 0x8, qn 0x1dbb, opcode 0xd, syndrome 0x1, vendor syndrome 0x68 00000000: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00000010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00000020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00000030: 00 00 00 00 60 10 68 01 0a 00 1d bb 00 0f 9f d2 WQE DUMP: WQ size 1024 WQ cur size 0, WQE index 0xf, len: 64 00000000: 00 00 0f 0a 00 1d bb 03 00 00 00 08 00 00 00 00 00000010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00000020: 00 00 00 2b 00 08 00 00 00 00 00 05 9e e3 08 00 00000030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 mlx5_core 0000:3b:00.1 eth1: ERR CQE on SQ: 0x1dbb So a priv_flag is added here to indicate whether the network card supports this feature. Signed-off-by: Xuan Zhuo Suggested-by: Alexander Lobakin [ alobakin: give a new flag more detailed description ] Signed-off-by: Alexander Lobakin --- include/linux/netdevice.h | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/include/linux/netdevice.h b/include/linux/netdevice.h index 0a9b2b31f411..ecaf67efab5b 100644 --- a/include/linux/netdevice.h +++ b/include/linux/netdevice.h @@ -1525,6 +1525,8 @@ struct net_device_ops { * @IFF_FAILOVER_SLAVE: device is lower dev of a failover master device * @IFF_L3MDEV_RX_HANDLER: only invoke the rx handler of L3 master device * @IFF_LIVE_RENAME_OK: rename is allowed while device is up and running + * @IFF_TX_SKB_NO_LINEAR: device/driver is capable of xmitting frames with + * skb_headlen(skb) == 0 (data starts from frag0) */ enum netdev_priv_flags { IFF_802_1Q_VLAN_BIT, @@ -1558,6 +1560,7 @@ enum netdev_priv_flags { IFF_FAILOVER_SLAVE_BIT, IFF_L3MDEV_RX_HANDLER_BIT, IFF_LIVE_RENAME_OK_BIT, + IFF_TX_SKB_NO_LINEAR_BIT, NETDEV_PRIV_FLAG_COUNT, }; @@ -1600,6 +1603,7 @@ static_assert(sizeof(netdev_priv_flags_t) * BITS_PER_BYTE >= #define IFF_FAILOVER_SLAVE __IFF(FAILOVER_SLAVE) #define IFF_L3MDEV_RX_HANDLER __IFF(L3MDEV_RX_HANDLER) #define IFF_LIVE_RENAME_OK __IFF(LIVE_RENAME_OK) +#define IFF_TX_SKB_NO_LINEAR __IFF(TX_SKB_NO_LINEAR) /** * struct net_device - The DEVICE structure. From patchwork Tue Feb 16 14:34:39 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Lobakin X-Patchwork-Id: 12090083 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 818D9C433DB for ; Tue, 16 Feb 2021 14:36:09 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 46E2864E07 for ; Tue, 16 Feb 2021 14:36:09 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230378AbhBPOfg (ORCPT ); Tue, 16 Feb 2021 09:35:36 -0500 Received: from mail-40133.protonmail.ch ([185.70.40.133]:55457 "EHLO mail-40133.protonmail.ch" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230295AbhBPOfb (ORCPT ); Tue, 16 Feb 2021 09:35:31 -0500 Date: Tue, 16 Feb 2021 14:34:39 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pm.me; s=protonmail; t=1613486083; bh=lgg1wNkMT8XQOATCoVN+lIKDRAuSPLux4jRMr6UVN+I=; h=Date:To:From:Cc:Reply-To:Subject:In-Reply-To:References:From; b=n3+ZZewhVF2gv5WzYKrGyZ+FQUQzcddVc+ZixijiqEj+LEY0b4hinVo/gu/NhpJk7 zU1ytXnLhMiX5Tz1y9wIb5PsYIla6Can2VQF0f5dFvx402pJ0qghGQY3JYGSR/eYJf HYkzCdeknwMSK9qEd7YqZAcVuaS9vZU/4GRlriSxVV/z+TYUswblA4y/QBM9A6iq43 nUFRZg1XYSya/me4EPkjUX4xj/n4McwQYrLztj+Eb47NTZwaN8yVI4Yk0fXtdHG3PW mYTGLZBD7x7ytCiVEOrwA1cB/MS2Hwn2mYfOX60y3rIpbyCweP71Mz2ZMe7FN/Jkd2 O7HIwMR/z0zpg== To: Magnus Karlsson , =?utf-8?b?QmrDtnJuIFTDtnBl?= =?utf-8?b?bA==?= From: Alexander Lobakin Cc: "Michael S. Tsirkin" , Jason Wang , "David S. Miller" , Jakub Kicinski , Jonathan Lemon , Alexei Starovoitov , Daniel Borkmann , Jesper Dangaard Brouer , John Fastabend , Andrii Nakryiko , Martin KaFai Lau , Song Liu , Yonghong Song , KP Singh , Paolo Abeni , Eric Dumazet , Xuan Zhuo , Dust Li , Alexander Lobakin , virtualization@lists.linux-foundation.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org, bpf@vger.kernel.org Reply-To: Alexander Lobakin Subject: [PATCH v5 bpf-next 4/6] virtio-net: support IFF_TX_SKB_NO_LINEAR Message-ID: <20210216143333.5861-5-alobakin@pm.me> In-Reply-To: <20210216143333.5861-1-alobakin@pm.me> References: <20210216143333.5861-1-alobakin@pm.me> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net From: Xuan Zhuo Virtio net supports the case where the skb linear space is empty, so add priv_flags. Signed-off-by: Xuan Zhuo Acked-by: Michael S. Tsirkin Signed-off-by: Alexander Lobakin --- drivers/net/virtio_net.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c index ba8e63792549..f2ff6c3906c1 100644 --- a/drivers/net/virtio_net.c +++ b/drivers/net/virtio_net.c @@ -2972,7 +2972,8 @@ static int virtnet_probe(struct virtio_device *vdev) return -ENOMEM; /* Set up network device as normal. */ - dev->priv_flags |= IFF_UNICAST_FLT | IFF_LIVE_ADDR_CHANGE; + dev->priv_flags |= IFF_UNICAST_FLT | IFF_LIVE_ADDR_CHANGE | + IFF_TX_SKB_NO_LINEAR; dev->netdev_ops = &virtnet_netdev; dev->features = NETIF_F_HIGHDMA; From patchwork Tue Feb 16 14:34:50 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Lobakin X-Patchwork-Id: 12090085 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 77D1FC433E0 for ; Tue, 16 Feb 2021 14:36:28 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 484E964E04 for ; Tue, 16 Feb 2021 14:36:28 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230365AbhBPOfu (ORCPT ); Tue, 16 Feb 2021 09:35:50 -0500 Received: from mail-40131.protonmail.ch ([185.70.40.131]:21031 "EHLO mail-40131.protonmail.ch" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230377AbhBPOfh (ORCPT ); Tue, 16 Feb 2021 09:35:37 -0500 Date: Tue, 16 Feb 2021 14:34:50 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pm.me; s=protonmail; t=1613486093; bh=rD+msxxQv7QdHAN2CCbl57zhJ/ALAb4aGQXqzumABFM=; h=Date:To:From:Cc:Reply-To:Subject:In-Reply-To:References:From; b=lwFOldjOtdJbiUeGmu4i3LwrK/27aYjWFVXLsFuyrSJQ+aEkv5t22hN+txliNdheg 1ScVEmhEHIhoCNsXwPTiZXdU4mE77sNlMitqEcn40WD+1eZebCou+t60LvlGgQgSsd 900RRKjH4sPTg3D3sh7l5kBvgoBQWtjgReTsTQJQK8mC7Boz0CdHBNDwnafcpYnty2 hdvI2YYagPQFLrASqmybaqYQaUmA9p391u2QsNSWvJ3Hm9rhkJ2i2gN6a7jngs37Fa hOEr6PIotUBB3ui4SGecaZp5XucV5PFpmu4vQVgYMLrqAt9X39wYkRfsPQqb7Gulml kOWzGC30f7xgA== To: Magnus Karlsson , =?utf-8?b?QmrDtnJuIFTDtnBl?= =?utf-8?b?bA==?= From: Alexander Lobakin Cc: "Michael S. Tsirkin" , Jason Wang , "David S. Miller" , Jakub Kicinski , Jonathan Lemon , Alexei Starovoitov , Daniel Borkmann , Jesper Dangaard Brouer , John Fastabend , Andrii Nakryiko , Martin KaFai Lau , Song Liu , Yonghong Song , KP Singh , Paolo Abeni , Eric Dumazet , Xuan Zhuo , Dust Li , Alexander Lobakin , virtualization@lists.linux-foundation.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org, bpf@vger.kernel.org Reply-To: Alexander Lobakin Subject: [PATCH v5 bpf-next 5/6] xsk: respect device's headroom and tailroom on generic xmit path Message-ID: <20210216143333.5861-6-alobakin@pm.me> In-Reply-To: <20210216143333.5861-1-alobakin@pm.me> References: <20210216143333.5861-1-alobakin@pm.me> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net xsk_generic_xmit() allocates a new skb and then queues it for xmitting. The size of new skb's headroom is desc->len, so it comes to the driver/device with no reserved headroom and/or tailroom. Lots of drivers need some headroom (and sometimes tailroom) to prepend (and/or append) some headers or data, e.g. CPU tags, device-specific headers/descriptors (LSO, TLS etc.), and if case of no available space skb_cow_head() will reallocate the skb. Reallocations are unwanted on fast-path, especially when it comes to XDP, so generic XSK xmit should reserve the spaces declared in dev->needed_headroom and dev->needed tailroom to avoid them. Note on max(NET_SKB_PAD, L1_CACHE_ALIGN(dev->needed_headroom)): Usually, output functions reserve LL_RESERVED_SPACE(dev), which consists of dev->hard_header_len + dev->needed_headroom, aligned by 16. However, on XSK xmit hard header is already here in the chunk, so hard_header_len is not needed. But it'd still be better to align data up to cacheline, while reserving no less than driver requests for headroom. NET_SKB_PAD here is to double-insure there will be no reallocations even when the driver advertises no needed_headroom, but in fact need it (not so rare case). Fixes: 35fcde7f8deb ("xsk: support for Tx") Signed-off-by: Alexander Lobakin Acked-by: Magnus Karlsson --- net/xdp/xsk.c | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/net/xdp/xsk.c b/net/xdp/xsk.c index 4faabd1ecfd1..143979ea4165 100644 --- a/net/xdp/xsk.c +++ b/net/xdp/xsk.c @@ -454,12 +454,16 @@ static int xsk_generic_xmit(struct sock *sk) struct sk_buff *skb; unsigned long flags; int err = 0; + u32 hr, tr; mutex_lock(&xs->mutex); if (xs->queue_id >= xs->dev->real_num_tx_queues) goto out; + hr = max(NET_SKB_PAD, L1_CACHE_ALIGN(xs->dev->needed_headroom)); + tr = xs->dev->needed_tailroom; + while (xskq_cons_peek_desc(xs->tx, &desc, xs->pool)) { char *buffer; u64 addr; @@ -471,11 +475,13 @@ static int xsk_generic_xmit(struct sock *sk) } len = desc.len; - skb = sock_alloc_send_skb(sk, len, 1, &err); + skb = sock_alloc_send_skb(sk, hr + len + tr, 1, &err); if (unlikely(!skb)) goto out; + skb_reserve(skb, hr); skb_put(skb, len); + addr = desc.addr; buffer = xsk_buff_raw_get_data(xs->pool, addr); err = skb_store_bits(skb, 0, buffer, len); From patchwork Tue Feb 16 14:35:02 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Lobakin X-Patchwork-Id: 12090087 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8F1EFC433E9 for ; Tue, 16 Feb 2021 14:36:28 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 5EA5764E08 for ; Tue, 16 Feb 2021 14:36:28 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230393AbhBPOgI (ORCPT ); Tue, 16 Feb 2021 09:36:08 -0500 Received: from mail2.protonmail.ch ([185.70.40.22]:58141 "EHLO mail2.protonmail.ch" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230383AbhBPOfs (ORCPT ); Tue, 16 Feb 2021 09:35:48 -0500 Date: Tue, 16 Feb 2021 14:35:02 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pm.me; s=protonmail; t=1613486103; bh=PUPLXjpwpkpzqk95FTsck/1hnkmAyZsE+RsTIuWPKnY=; h=Date:To:From:Cc:Reply-To:Subject:In-Reply-To:References:From; b=X0KO8Ega8wHesK00ZIwrSYc9iOEoawjMUnAsj2OQVTRXTg3nbl7hhAH6ud7otDGE8 lzGC2dkmHnpVwn//C+dT4B2Zl1vM9UqccR2ffC974eWvwuE6V0jzpbkXWuTv7sFaB5 /6fIs0IUSb4UhfwKdcR7oLjdACyv5uz4/nIwLeV6HkNQcVPZt+oLVnjCRypUqvlBIz Xn1ogvueiV4Jc0IgofIyv9uy+2lwXhlKxWZMf5QKtdT8GF+zlzzLfnqw7Qzj4rGs/Y rn2npD2AON2S6AvmLgH/054gOofGUgEvEoB8sH+VVE/VG2VWzARUHORXmigLHY37S9 oIGeQRjXPqXsg== To: Magnus Karlsson , =?utf-8?b?QmrDtnJuIFTDtnBl?= =?utf-8?b?bA==?= From: Alexander Lobakin Cc: "Michael S. Tsirkin" , Jason Wang , "David S. Miller" , Jakub Kicinski , Jonathan Lemon , Alexei Starovoitov , Daniel Borkmann , Jesper Dangaard Brouer , John Fastabend , Andrii Nakryiko , Martin KaFai Lau , Song Liu , Yonghong Song , KP Singh , Paolo Abeni , Eric Dumazet , Xuan Zhuo , Dust Li , Alexander Lobakin , virtualization@lists.linux-foundation.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org, bpf@vger.kernel.org Reply-To: Alexander Lobakin Subject: [PATCH v5 bpf-next 6/6] xsk: build skb by page (aka generic zerocopy xmit) Message-ID: <20210216143333.5861-7-alobakin@pm.me> In-Reply-To: <20210216143333.5861-1-alobakin@pm.me> References: <20210216143333.5861-1-alobakin@pm.me> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net From: Xuan Zhuo This patch is used to construct skb based on page to save memory copy overhead. This function is implemented based on IFF_TX_SKB_NO_LINEAR. Only the network card priv_flags supports IFF_TX_SKB_NO_LINEAR will use page to directly construct skb. If this feature is not supported, it is still necessary to copy data to construct skb. ---------------- Performance Testing ------------ The test environment is Aliyun ECS server. Test cmd: ``` xdpsock -i eth0 -t -S -s ``` Test result data: size 64 512 1024 1500 copy 1916747 1775988 1600203 1440054 page 1974058 1953655 1945463 1904478 percent 3.0% 10.0% 21.58% 32.3% Signed-off-by: Xuan Zhuo Reviewed-by: Dust Li [ alobakin: - expand subject to make it clearer; - improve skb->truesize calculation; - reserve some headroom in skb for drivers; - tailroom is not needed as skb is non-linear ] Signed-off-by: Alexander Lobakin Acked-by: Magnus Karlsson --- net/xdp/xsk.c | 119 ++++++++++++++++++++++++++++++++++++++++---------- 1 file changed, 95 insertions(+), 24 deletions(-) diff --git a/net/xdp/xsk.c b/net/xdp/xsk.c index 143979ea4165..ff7bd06e1241 100644 --- a/net/xdp/xsk.c +++ b/net/xdp/xsk.c @@ -445,6 +445,96 @@ static void xsk_destruct_skb(struct sk_buff *skb) sock_wfree(skb); } +static struct sk_buff *xsk_build_skb_zerocopy(struct xdp_sock *xs, + struct xdp_desc *desc) +{ + struct xsk_buff_pool *pool = xs->pool; + u32 hr, len, offset, copy, copied; + struct sk_buff *skb; + struct page *page; + void *buffer; + int err, i; + u64 addr; + + hr = max(NET_SKB_PAD, L1_CACHE_ALIGN(xs->dev->needed_headroom)); + + skb = sock_alloc_send_skb(&xs->sk, hr, 1, &err); + if (unlikely(!skb)) + return ERR_PTR(err); + + skb_reserve(skb, hr); + + addr = desc->addr; + len = desc->len; + + buffer = xsk_buff_raw_get_data(pool, addr); + offset = offset_in_page(buffer); + addr = buffer - pool->addrs; + + for (copied = 0, i = 0; copied < len; i++) { + page = pool->umem->pgs[addr >> PAGE_SHIFT]; + get_page(page); + + copy = min_t(u32, PAGE_SIZE - offset, len - copied); + skb_fill_page_desc(skb, i, page, offset, copy); + + copied += copy; + addr += copy; + offset = 0; + } + + skb->len += len; + skb->data_len += len; + skb->truesize += pool->unaligned ? len : pool->chunk_size; + + refcount_add(skb->truesize, &xs->sk.sk_wmem_alloc); + + return skb; +} + +static struct sk_buff *xsk_build_skb(struct xdp_sock *xs, + struct xdp_desc *desc) +{ + struct net_device *dev = xs->dev; + struct sk_buff *skb; + + if (dev->priv_flags & IFF_TX_SKB_NO_LINEAR) { + skb = xsk_build_skb_zerocopy(xs, desc); + if (IS_ERR(skb)) + return skb; + } else { + u32 hr, tr, len; + void *buffer; + int err; + + hr = max(NET_SKB_PAD, L1_CACHE_ALIGN(dev->needed_headroom)); + tr = dev->needed_tailroom; + len = desc->len; + + skb = sock_alloc_send_skb(&xs->sk, hr + len + tr, 1, &err); + if (unlikely(!skb)) + return ERR_PTR(err); + + skb_reserve(skb, hr); + skb_put(skb, len); + + buffer = xsk_buff_raw_get_data(xs->pool, desc->addr); + err = skb_store_bits(skb, 0, buffer, len); + if (unlikely(err)) { + kfree_skb(skb); + return ERR_PTR(err); + } + } + + skb->dev = dev; + skb->priority = xs->sk.sk_priority; + skb->mark = xs->sk.sk_mark; + skb_shinfo(skb)->destructor_arg = (void *)(long)desc->addr; + skb->destructor = xsk_destruct_skb; + + return skb; +} + static int xsk_generic_xmit(struct sock *sk) { struct xdp_sock *xs = xdp_sk(sk); @@ -454,56 +544,37 @@ static int xsk_generic_xmit(struct sock *sk) struct sk_buff *skb; unsigned long flags; int err = 0; - u32 hr, tr; mutex_lock(&xs->mutex); if (xs->queue_id >= xs->dev->real_num_tx_queues) goto out; - hr = max(NET_SKB_PAD, L1_CACHE_ALIGN(xs->dev->needed_headroom)); - tr = xs->dev->needed_tailroom; - while (xskq_cons_peek_desc(xs->tx, &desc, xs->pool)) { - char *buffer; - u64 addr; - u32 len; - if (max_batch-- == 0) { err = -EAGAIN; goto out; } - len = desc.len; - skb = sock_alloc_send_skb(sk, hr + len + tr, 1, &err); - if (unlikely(!skb)) + skb = xsk_build_skb(xs, &desc); + if (IS_ERR(skb)) { + err = PTR_ERR(skb); goto out; + } - skb_reserve(skb, hr); - skb_put(skb, len); - - addr = desc.addr; - buffer = xsk_buff_raw_get_data(xs->pool, addr); - err = skb_store_bits(skb, 0, buffer, len); /* This is the backpressure mechanism for the Tx path. * Reserve space in the completion queue and only proceed * if there is space in it. This avoids having to implement * any buffering in the Tx path. */ spin_lock_irqsave(&xs->pool->cq_lock, flags); - if (unlikely(err) || xskq_prod_reserve(xs->pool->cq)) { + if (xskq_prod_reserve(xs->pool->cq)) { spin_unlock_irqrestore(&xs->pool->cq_lock, flags); kfree_skb(skb); goto out; } spin_unlock_irqrestore(&xs->pool->cq_lock, flags); - skb->dev = xs->dev; - skb->priority = sk->sk_priority; - skb->mark = sk->sk_mark; - skb_shinfo(skb)->destructor_arg = (void *)(long)desc.addr; - skb->destructor = xsk_destruct_skb; - err = __dev_direct_xmit(skb, xs->queue_id); if (err == NETDEV_TX_BUSY) { /* Tell user-space to retry the send */