From patchwork Thu Feb 13 01:06:32 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jakub Kicinski X-Patchwork-Id: 13972649 X-Patchwork-Delegate: kuba@kernel.org Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 68A307081A for ; Thu, 13 Feb 2025 01:06:42 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739408802; cv=none; b=gLHq3SN+yqtZkTO8wGJ1Oik/bCIZNG07KfavuT2t/9ih7IzKOLUAGFfCtydW1M9Cnemzd5vmhZoa36J/wOuPI+uNrtrHwHPyNZXdUC0ELd6IpnoCvWDHZxZ+Qm9Z5I4FYnsmHwemv80wimSk8Yw2apql6zWb7ATNBw2sXmX5KJU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739408802; c=relaxed/simple; bh=rImjUt0qc0DeiY9g5lZl4Ra6VLDP0VtNaaWhl2uBUg8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=SV7HXAVyq1U+eQGJ0cYsVGwHauiUaaB6kTspUNpnG12ypHVcKAVkMcU370BCt0lGnmy6TnpZ8BZToPkww2Pi0dVZ5gUpX5M5jv+RI+ZfaFeKGlKLw+P7K26KftYHpBc+rg76G+t+2SyzQPIFACsooZydMrdkkMHYh6bhu+4WJwc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=G9IDLehn; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="G9IDLehn" Received: by smtp.kernel.org (Postfix) with ESMTPSA id A07E2C4CEE5; Thu, 13 Feb 2025 01:06:41 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1739408801; bh=rImjUt0qc0DeiY9g5lZl4Ra6VLDP0VtNaaWhl2uBUg8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=G9IDLehnyz6iuxXe8aRwiihkVgLjMY6g+y0cUGpjWhTJak7IRI88oXM039p8ZeFqN aFG8BQjSgZqXIL2E+toBcPJq86resrc7kNzH7XqG6tj9nIAoj9usqAerEfEGCSDEdk 40ZATOXbrxJ66BmgU+buQAc3hmgWfWcioVgUfHe2a+duJfB7S9xy8PHGx975ICl6ch 1EvTBzzNKtyvaC0ZRTz9VlVKu2rqtk9LY51Hnct84XyUGzVn5U2un1cBSdYj8huhSl jfL9tlsSiLM2u1UAjE/qTuXlIvcSbsyc1qhYfa/giHYv0J5C43fj88yytJzcTYMLeb qUaiwhTXAYQjg== From: Jakub Kicinski To: davem@davemloft.net Cc: tariqt@nvidia.com, idosch@idosch.org, hawk@kernel.org, netdev@vger.kernel.org, edumazet@google.com, pabeni@redhat.com, andrew+netdev@lunn.ch, horms@kernel.org, Jakub Kicinski Subject: [PATCH net-next v3 1/4] eth: mlx4: create a page pool for Rx Date: Wed, 12 Feb 2025 17:06:32 -0800 Message-ID: <20250213010635.1354034-2-kuba@kernel.org> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250213010635.1354034-1-kuba@kernel.org> References: <20250213010635.1354034-1-kuba@kernel.org> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: kuba@kernel.org Create a pool per rx queue. Subsequent patches will make use of it. Move fcs_del to a hole to make space for the pointer. Per common "wisdom" base the page pool size on the ring size. Note that the page pool cache size is in full pages, so just round up the effective buffer size to pages. Signed-off-by: Jakub Kicinski Reviewed-by: Tariq Toukan --- v3: - use priv->rx_skb_size for effective buffer size - use priv->dma_dir for DMA mapping direction, instead of always BIDIR v2: https://lore.kernel.org/20250211192141.619024-2-kuba@kernel.org - update pp.pool_size v1: https://lore.kernel.org/20250205031213.358973-2-kuba@kernel.org --- drivers/net/ethernet/mellanox/mlx4/mlx4_en.h | 3 ++- drivers/net/ethernet/mellanox/mlx4/en_rx.c | 24 +++++++++++++++++++- 2 files changed, 25 insertions(+), 2 deletions(-) diff --git a/drivers/net/ethernet/mellanox/mlx4/mlx4_en.h b/drivers/net/ethernet/mellanox/mlx4/mlx4_en.h index 28b70dcc652e..29f48e63081b 100644 --- a/drivers/net/ethernet/mellanox/mlx4/mlx4_en.h +++ b/drivers/net/ethernet/mellanox/mlx4/mlx4_en.h @@ -335,10 +335,11 @@ struct mlx4_en_rx_ring { u16 stride; u16 log_stride; u16 cqn; /* index of port CQ associated with this ring */ + u8 fcs_del; u32 prod; u32 cons; u32 buf_size; - u8 fcs_del; + struct page_pool *pp; void *buf; void *rx_info; struct bpf_prog __rcu *xdp_prog; diff --git a/drivers/net/ethernet/mellanox/mlx4/en_rx.c b/drivers/net/ethernet/mellanox/mlx4/en_rx.c index 15c57e9517e9..a8c0cf5d0d08 100644 --- a/drivers/net/ethernet/mellanox/mlx4/en_rx.c +++ b/drivers/net/ethernet/mellanox/mlx4/en_rx.c @@ -48,6 +48,7 @@ #if IS_ENABLED(CONFIG_IPV6) #include #endif +#include #include "mlx4_en.h" @@ -268,6 +269,7 @@ int mlx4_en_create_rx_ring(struct mlx4_en_priv *priv, u32 size, u16 stride, int node, int queue_index) { struct mlx4_en_dev *mdev = priv->mdev; + struct page_pool_params pp = {}; struct mlx4_en_rx_ring *ring; int err = -ENOMEM; int tmp; @@ -286,9 +288,26 @@ int mlx4_en_create_rx_ring(struct mlx4_en_priv *priv, ring->log_stride = ffs(ring->stride) - 1; ring->buf_size = ring->size * ring->stride + TXBB_SIZE; - if (xdp_rxq_info_reg(&ring->xdp_rxq, priv->dev, queue_index, 0) < 0) + pp.flags = PP_FLAG_DMA_MAP; + pp.pool_size = size * DIV_ROUND_UP(priv->rx_skb_size, PAGE_SIZE); + pp.nid = node; + pp.napi = &priv->rx_cq[queue_index]->napi; + pp.netdev = priv->dev; + pp.dev = &mdev->dev->persist->pdev->dev; + pp.dma_dir = priv->dma_dir; + + ring->pp = page_pool_create(&pp); + if (!ring->pp) goto err_ring; + if (xdp_rxq_info_reg(&ring->xdp_rxq, priv->dev, queue_index, 0) < 0) + goto err_pp; + + err = xdp_rxq_info_reg_mem_model(&ring->xdp_rxq, MEM_TYPE_PAGE_POOL, + ring->pp); + if (err) + goto err_xdp_info; + tmp = size * roundup_pow_of_two(MLX4_EN_MAX_RX_FRAGS * sizeof(struct mlx4_en_rx_alloc)); ring->rx_info = kvzalloc_node(tmp, GFP_KERNEL, node); @@ -319,6 +338,8 @@ int mlx4_en_create_rx_ring(struct mlx4_en_priv *priv, ring->rx_info = NULL; err_xdp_info: xdp_rxq_info_unreg(&ring->xdp_rxq); +err_pp: + page_pool_destroy(ring->pp); err_ring: kfree(ring); *pring = NULL; @@ -445,6 +466,7 @@ void mlx4_en_destroy_rx_ring(struct mlx4_en_priv *priv, xdp_rxq_info_unreg(&ring->xdp_rxq); mlx4_free_hwq_res(mdev->dev, &ring->wqres, size * stride + TXBB_SIZE); kvfree(ring->rx_info); + page_pool_destroy(ring->pp); ring->rx_info = NULL; kfree(ring); *pring = NULL; From patchwork Thu Feb 13 01:06:33 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jakub Kicinski X-Patchwork-Id: 13972650 X-Patchwork-Delegate: kuba@kernel.org Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 093C484D29 for ; Thu, 13 Feb 2025 01:06:42 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739408803; cv=none; b=Gws4oPYGqPBqg8K+GDLGMEZOtobVvAMM7MFe3AMxZ/HYG0j02xkasvMmzEB64RCMNQSv2CmqJVRlfBPD4I6zat/2kSKmMiQy6K8RQriz5v8JQXlESZQcOgCHpJ3zZAWAUJt64yloqb/Jh10e0LqMqPbjVf8jPyuHrThoa29Ruwo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739408803; c=relaxed/simple; bh=BnfEtkuSRakxY27SXHJr8MvG2/eU/7W7dexHODJa/pQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=HkECT9eG+BMJQWNWegyvq6scQGw9pG3wjz6yyj1x4hVcwJCybOr7AmkZQADkH01X0FvkUF3RqJ/fLyPrUR/cAtMRRdTCxePJ+gU9zh+msqP46niogJkikFA/q4Fm5X2DBwrY5Xjo0LwcZS0sFH5MYsTUAWqpZMFPyzC3R9FVjWg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=sLbuk7c3; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="sLbuk7c3" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 152D3C4CEEA; Thu, 13 Feb 2025 01:06:42 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1739408802; bh=BnfEtkuSRakxY27SXHJr8MvG2/eU/7W7dexHODJa/pQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=sLbuk7c3enOn5VvueHigHRg/KYeHdtXSAnXGipfIC1Mwsqs0/G6wIis5wt34bBVeE zaFvHy5uPDNQ6Vcs7REbgeVNypdFcBwD5RZO8Cmahfbg/hJMIIUY8yodpNTN6dLW+d 6F/bySgVEsXTHN6jiw1drfl/Gna3jcARKqWaHXBkfoLyO23GQPMOMnh13yeHGPsU8K Qk434RSYRnRcECC3owcY0S3o9x2Zng5gtIJz8qtow+PuWEyT91oDRXrbZIb+bbq8QY kD/JBT9e2mlX2uGR0zTKitIMXSrSyqvYb/hQDL92vB6zgxgtWXNdQvZxlyF/1VUpFK VwBf5TqLcrvrA== From: Jakub Kicinski To: davem@davemloft.net Cc: tariqt@nvidia.com, idosch@idosch.org, hawk@kernel.org, netdev@vger.kernel.org, edumazet@google.com, pabeni@redhat.com, andrew+netdev@lunn.ch, horms@kernel.org, Jakub Kicinski Subject: [PATCH net-next v3 2/4] eth: mlx4: don't try to complete XDP frames in netpoll Date: Wed, 12 Feb 2025 17:06:33 -0800 Message-ID: <20250213010635.1354034-3-kuba@kernel.org> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250213010635.1354034-1-kuba@kernel.org> References: <20250213010635.1354034-1-kuba@kernel.org> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: kuba@kernel.org mlx4 doesn't support ndo_xdp_xmit / XDP_REDIRECT and wasn't using page pool until now, so it could run XDP completions in netpoll (NAPI budget == 0) just fine. Page pool has calling context requirements, make sure we don't try to call it from what is potentially HW IRQ context. Signed-off-by: Jakub Kicinski Reviewed-by: Tariq Toukan --- v2: - clarify the commit msg v1: https://lore.kernel.org/20250205031213.358973-3-kuba@kernel.org --- drivers/net/ethernet/mellanox/mlx4/en_tx.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/drivers/net/ethernet/mellanox/mlx4/en_tx.c b/drivers/net/ethernet/mellanox/mlx4/en_tx.c index 1ddb11cb25f9..6e077d202827 100644 --- a/drivers/net/ethernet/mellanox/mlx4/en_tx.c +++ b/drivers/net/ethernet/mellanox/mlx4/en_tx.c @@ -450,6 +450,8 @@ int mlx4_en_process_tx_cq(struct net_device *dev, if (unlikely(!priv->port_up)) return 0; + if (unlikely(!napi_budget) && cq->type == TX_XDP) + return 0; netdev_txq_bql_complete_prefetchw(ring->tx_queue); From patchwork Thu Feb 13 01:06:34 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jakub Kicinski X-Patchwork-Id: 13972651 X-Patchwork-Delegate: kuba@kernel.org Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9B1376FC3 for ; Thu, 13 Feb 2025 01:06:43 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739408803; cv=none; b=fTXE7ICmsY8bjynkhTc7Tl2/9bgIxhDpO9oOtahPdUZuGHJgShQ/jxxQLUdasuixyp8VUwsR2MUCLJPaV14OjUQ5g9vB7uAhK9YN6Zkp4DcTrScRZ7BB0b26Ruac100p2xF3XiyNrABpFv6kk66WK/Q1kM4wfF4GyVkdkGswQ2Y= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739408803; c=relaxed/simple; bh=LZjsCcV2JzkGy236sxg2zqz123cUvoPszuJJ1DdKNbc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=ZMagMk0hDT8Pupl/v9xsAUoGpWD+uxnXLZxqoz7QV3BNJpwz2qqJtnBq7F54AJfhKOWS6VNnJSqWhi1Fl5tdqPzcAKsBKy5deq+0VjOKTkv/XJ5dpOawXymbl5c+V3BHZ8xTF6vOUTypKqvLP1qChWGa7tfzRDqwMlWCSqFUt7o= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=BXXOJNbt; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="BXXOJNbt" Received: by smtp.kernel.org (Postfix) with ESMTPSA id B41A2C4CEE8; Thu, 13 Feb 2025 01:06:42 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1739408803; bh=LZjsCcV2JzkGy236sxg2zqz123cUvoPszuJJ1DdKNbc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=BXXOJNbtVIWhj9cAEYfqhe6gmBYuOjqAKIVQVPPWfPBPJkM6cwqBJglzLW7+cTU9O 3wCCYN4krUSgXQqJmW8g0km+Dc2TyNar4OdwkD3nt4nlZgKdpve3hIku+7CbTNJ2v/ FeDN3liqImUqdXoGxp/KWNGbmPY6K+BjA1PEhy+i8wsvLjhgTmaf6i9CPoOZHHWQr5 zQzzy2iFjsuXAX9kLgvViLCJSiJYiMhTbFO73vomnwWDp2Zk02H4Lw31GLw1TBxZrw sPQPzaVBM5wam3eVB1xhfOa7pQnQade9WpXuxn/GegW2Ra+wUu55hPdGSg63EoVofr 9fglRMA7sPbgQ== From: Jakub Kicinski To: davem@davemloft.net Cc: tariqt@nvidia.com, idosch@idosch.org, hawk@kernel.org, netdev@vger.kernel.org, edumazet@google.com, pabeni@redhat.com, andrew+netdev@lunn.ch, horms@kernel.org, Jakub Kicinski Subject: [PATCH net-next v3 3/4] eth: mlx4: remove the local XDP fast-recycling ring Date: Wed, 12 Feb 2025 17:06:34 -0800 Message-ID: <20250213010635.1354034-4-kuba@kernel.org> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250213010635.1354034-1-kuba@kernel.org> References: <20250213010635.1354034-1-kuba@kernel.org> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: kuba@kernel.org It will be replaced with page pool's built-in recycling. Signed-off-by: Jakub Kicinski Reviewed-by: Tariq Toukan --- v2: no change --- drivers/net/ethernet/mellanox/mlx4/mlx4_en.h | 11 ------ drivers/net/ethernet/mellanox/mlx4/en_rx.c | 40 -------------------- drivers/net/ethernet/mellanox/mlx4/en_tx.c | 11 +----- 3 files changed, 2 insertions(+), 60 deletions(-) diff --git a/drivers/net/ethernet/mellanox/mlx4/mlx4_en.h b/drivers/net/ethernet/mellanox/mlx4/mlx4_en.h index 29f48e63081b..97311c98569f 100644 --- a/drivers/net/ethernet/mellanox/mlx4/mlx4_en.h +++ b/drivers/net/ethernet/mellanox/mlx4/mlx4_en.h @@ -253,14 +253,6 @@ struct mlx4_en_rx_alloc { #define MLX4_EN_CACHE_SIZE (2 * NAPI_POLL_WEIGHT) -struct mlx4_en_page_cache { - u32 index; - struct { - struct page *page; - dma_addr_t dma; - } buf[MLX4_EN_CACHE_SIZE]; -}; - enum { MLX4_EN_TX_RING_STATE_RECOVERING, }; @@ -343,7 +335,6 @@ struct mlx4_en_rx_ring { void *buf; void *rx_info; struct bpf_prog __rcu *xdp_prog; - struct mlx4_en_page_cache page_cache; unsigned long bytes; unsigned long packets; unsigned long csum_ok; @@ -708,8 +699,6 @@ netdev_tx_t mlx4_en_xmit_frame(struct mlx4_en_rx_ring *rx_ring, struct mlx4_en_priv *priv, unsigned int length, int tx_ind, bool *doorbell_pending); void mlx4_en_xmit_doorbell(struct mlx4_en_tx_ring *ring); -bool mlx4_en_rx_recycle(struct mlx4_en_rx_ring *ring, - struct mlx4_en_rx_alloc *frame); int mlx4_en_create_tx_ring(struct mlx4_en_priv *priv, struct mlx4_en_tx_ring **pring, diff --git a/drivers/net/ethernet/mellanox/mlx4/en_rx.c b/drivers/net/ethernet/mellanox/mlx4/en_rx.c index a8c0cf5d0d08..d2cfbf2e38d9 100644 --- a/drivers/net/ethernet/mellanox/mlx4/en_rx.c +++ b/drivers/net/ethernet/mellanox/mlx4/en_rx.c @@ -142,18 +142,6 @@ static int mlx4_en_prepare_rx_desc(struct mlx4_en_priv *priv, (index << ring->log_stride); struct mlx4_en_rx_alloc *frags = ring->rx_info + (index << priv->log_rx_info); - if (likely(ring->page_cache.index > 0)) { - /* XDP uses a single page per frame */ - if (!frags->page) { - ring->page_cache.index--; - frags->page = ring->page_cache.buf[ring->page_cache.index].page; - frags->dma = ring->page_cache.buf[ring->page_cache.index].dma; - } - frags->page_offset = XDP_PACKET_HEADROOM; - rx_desc->data[0].addr = cpu_to_be64(frags->dma + - XDP_PACKET_HEADROOM); - return 0; - } return mlx4_en_alloc_frags(priv, ring, rx_desc, frags, gfp); } @@ -430,26 +418,6 @@ void mlx4_en_recover_from_oom(struct mlx4_en_priv *priv) } } -/* When the rx ring is running in page-per-packet mode, a released frame can go - * directly into a small cache, to avoid unmapping or touching the page - * allocator. In bpf prog performance scenarios, buffers are either forwarded - * or dropped, never converted to skbs, so every page can come directly from - * this cache when it is sized to be a multiple of the napi budget. - */ -bool mlx4_en_rx_recycle(struct mlx4_en_rx_ring *ring, - struct mlx4_en_rx_alloc *frame) -{ - struct mlx4_en_page_cache *cache = &ring->page_cache; - - if (cache->index >= MLX4_EN_CACHE_SIZE) - return false; - - cache->buf[cache->index].page = frame->page; - cache->buf[cache->index].dma = frame->dma; - cache->index++; - return true; -} - void mlx4_en_destroy_rx_ring(struct mlx4_en_priv *priv, struct mlx4_en_rx_ring **pring, u32 size, u16 stride) @@ -475,14 +443,6 @@ void mlx4_en_destroy_rx_ring(struct mlx4_en_priv *priv, void mlx4_en_deactivate_rx_ring(struct mlx4_en_priv *priv, struct mlx4_en_rx_ring *ring) { - int i; - - for (i = 0; i < ring->page_cache.index; i++) { - dma_unmap_page(priv->ddev, ring->page_cache.buf[i].dma, - PAGE_SIZE, priv->dma_dir); - put_page(ring->page_cache.buf[i].page); - } - ring->page_cache.index = 0; mlx4_en_free_rx_buf(priv, ring); if (ring->stride <= TXBB_SIZE) ring->buf -= TXBB_SIZE; diff --git a/drivers/net/ethernet/mellanox/mlx4/en_tx.c b/drivers/net/ethernet/mellanox/mlx4/en_tx.c index 6e077d202827..fe1378a689a1 100644 --- a/drivers/net/ethernet/mellanox/mlx4/en_tx.c +++ b/drivers/net/ethernet/mellanox/mlx4/en_tx.c @@ -350,16 +350,9 @@ u32 mlx4_en_recycle_tx_desc(struct mlx4_en_priv *priv, int napi_mode) { struct mlx4_en_tx_info *tx_info = &ring->tx_info[index]; - struct mlx4_en_rx_alloc frame = { - .page = tx_info->page, - .dma = tx_info->map0_dma, - }; - if (!napi_mode || !mlx4_en_rx_recycle(ring->recycle_ring, &frame)) { - dma_unmap_page(priv->ddev, tx_info->map0_dma, - PAGE_SIZE, priv->dma_dir); - put_page(tx_info->page); - } + dma_unmap_page(priv->ddev, tx_info->map0_dma, PAGE_SIZE, priv->dma_dir); + put_page(tx_info->page); return tx_info->nr_txbb; } From patchwork Thu Feb 13 01:06:35 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jakub Kicinski X-Patchwork-Id: 13972652 X-Patchwork-Delegate: kuba@kernel.org Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id EBC1D13774D for ; Thu, 13 Feb 2025 01:06:43 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739408804; cv=none; b=KheHvni3cLKG/NQMRnMN/f/xbgnYarmO1jsLBDE3UWWs2n1GesQMiiWJ0J3OnIaZwYVUvGoReCj28wRhs4R8fInlnzcJqRAw3dwbCJ9xf66jGtdaKHkbbABG91g+DcGIjJ+TinFCtoAyBH1gto5OVLWEDKU8oVh3vwqQKm7Ntoo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739408804; c=relaxed/simple; bh=hPQsxe/XjfjZ1RKv8rQ60pCX3IxmNrI8aJbj7oiPDGc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=X6ZMx1BYqSySumzYS/uFIAg0X9bMMbhcLflZ0xRSsevyuZsDpS76OQ2TG1zKdGeVKXc96ir6TPeT2+J7NjYlI7IAN0n2B+9YIPlPfUVt31sPCbPLAb01812CS5tw/TOVKU12AXEHyv/ACL4W1MdrirUzRfK/6lms56oItQeH/Hc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=aZkXYCGY; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="aZkXYCGY" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 342B1C4AF0B; Thu, 13 Feb 2025 01:06:43 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1739408803; bh=hPQsxe/XjfjZ1RKv8rQ60pCX3IxmNrI8aJbj7oiPDGc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=aZkXYCGYS7hRXJybuxYM3aXBcJlM4yMNJvgZ0MufHesBmjOjikEQYibbZNpvBaQG4 G4SROOeHo2aRuq9oQ2gjbR/XmkBMn21OLvLOAeyM7DqIaZKFfn1JiBskYqGaMcbLKh uyqyIj3TyaTtMLyAIQOKPN2/XOg9wlglOCUvPhatA+9tviAq6WGUhQC7/MqsDPhjV5 uLL3hPHaZHwyg4ghUTgT4PfGZjH9hT5WoPM2GNwAsd/u1fvXjrPOzLkxeL5qBPTlsb Fz6RwsLr4+OZdL3OTr6YuICB7INBQxNVzCSkFMBJivQZ6sBGFqKtrxR8ub+QYkzO06 f11oq/8sTnohA== From: Jakub Kicinski To: davem@davemloft.net Cc: tariqt@nvidia.com, idosch@idosch.org, hawk@kernel.org, netdev@vger.kernel.org, edumazet@google.com, pabeni@redhat.com, andrew+netdev@lunn.ch, horms@kernel.org, Jakub Kicinski Subject: [PATCH net-next v3 4/4] eth: mlx4: use the page pool for Rx buffers Date: Wed, 12 Feb 2025 17:06:35 -0800 Message-ID: <20250213010635.1354034-5-kuba@kernel.org> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250213010635.1354034-1-kuba@kernel.org> References: <20250213010635.1354034-1-kuba@kernel.org> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: kuba@kernel.org Simple conversion to page pool. Preserve the current fragmentation logic / page splitting. Each page starts with a single frag reference, and then we bump that when attaching to skbs. This can likely be optimized further. Signed-off-by: Jakub Kicinski Reviewed-by: Tariq Toukan --- v2: - remove unnecessary .max_len setting v1: https://lore.kernel.org/20250205031213.358973-5-kuba@kernel.org --- drivers/net/ethernet/mellanox/mlx4/mlx4_en.h | 1 - drivers/net/ethernet/mellanox/mlx4/en_rx.c | 55 +++++++------------- drivers/net/ethernet/mellanox/mlx4/en_tx.c | 8 +-- 3 files changed, 25 insertions(+), 39 deletions(-) diff --git a/drivers/net/ethernet/mellanox/mlx4/mlx4_en.h b/drivers/net/ethernet/mellanox/mlx4/mlx4_en.h index 97311c98569f..ad0d91a75184 100644 --- a/drivers/net/ethernet/mellanox/mlx4/mlx4_en.h +++ b/drivers/net/ethernet/mellanox/mlx4/mlx4_en.h @@ -247,7 +247,6 @@ struct mlx4_en_tx_desc { struct mlx4_en_rx_alloc { struct page *page; - dma_addr_t dma; u32 page_offset; }; diff --git a/drivers/net/ethernet/mellanox/mlx4/en_rx.c b/drivers/net/ethernet/mellanox/mlx4/en_rx.c index d2cfbf2e38d9..b33285d755b9 100644 --- a/drivers/net/ethernet/mellanox/mlx4/en_rx.c +++ b/drivers/net/ethernet/mellanox/mlx4/en_rx.c @@ -52,57 +52,39 @@ #include "mlx4_en.h" -static int mlx4_alloc_page(struct mlx4_en_priv *priv, - struct mlx4_en_rx_alloc *frag, - gfp_t gfp) -{ - struct page *page; - dma_addr_t dma; - - page = alloc_page(gfp); - if (unlikely(!page)) - return -ENOMEM; - dma = dma_map_page(priv->ddev, page, 0, PAGE_SIZE, priv->dma_dir); - if (unlikely(dma_mapping_error(priv->ddev, dma))) { - __free_page(page); - return -ENOMEM; - } - frag->page = page; - frag->dma = dma; - frag->page_offset = priv->rx_headroom; - return 0; -} - static int mlx4_en_alloc_frags(struct mlx4_en_priv *priv, struct mlx4_en_rx_ring *ring, struct mlx4_en_rx_desc *rx_desc, struct mlx4_en_rx_alloc *frags, gfp_t gfp) { + dma_addr_t dma; int i; for (i = 0; i < priv->num_frags; i++, frags++) { if (!frags->page) { - if (mlx4_alloc_page(priv, frags, gfp)) { + frags->page = page_pool_alloc_pages(ring->pp, gfp); + if (!frags->page) { ring->alloc_fail++; return -ENOMEM; } + page_pool_fragment_page(frags->page, 1); + frags->page_offset = priv->rx_headroom; + ring->rx_alloc_pages++; } - rx_desc->data[i].addr = cpu_to_be64(frags->dma + - frags->page_offset); + dma = page_pool_get_dma_addr(frags->page); + rx_desc->data[i].addr = cpu_to_be64(dma + frags->page_offset); } return 0; } static void mlx4_en_free_frag(const struct mlx4_en_priv *priv, + struct mlx4_en_rx_ring *ring, struct mlx4_en_rx_alloc *frag) { - if (frag->page) { - dma_unmap_page(priv->ddev, frag->dma, - PAGE_SIZE, priv->dma_dir); - __free_page(frag->page); - } + if (frag->page) + page_pool_put_full_page(ring->pp, frag->page, false); /* We need to clear all fields, otherwise a change of priv->log_rx_info * could lead to see garbage later in frag->page. */ @@ -167,7 +149,7 @@ static void mlx4_en_free_rx_desc(const struct mlx4_en_priv *priv, frags = ring->rx_info + (index << priv->log_rx_info); for (nr = 0; nr < priv->num_frags; nr++) { en_dbg(DRV, priv, "Freeing fragment:%d\n", nr); - mlx4_en_free_frag(priv, frags + nr); + mlx4_en_free_frag(priv, ring, frags + nr); } } @@ -469,7 +451,7 @@ static int mlx4_en_complete_rx_desc(struct mlx4_en_priv *priv, if (unlikely(!page)) goto fail; - dma = frags->dma; + dma = page_pool_get_dma_addr(page); dma_sync_single_range_for_cpu(priv->ddev, dma, frags->page_offset, frag_size, priv->dma_dir); @@ -480,6 +462,7 @@ static int mlx4_en_complete_rx_desc(struct mlx4_en_priv *priv, if (frag_info->frag_stride == PAGE_SIZE / 2) { frags->page_offset ^= PAGE_SIZE / 2; release = page_count(page) != 1 || + atomic_long_read(&page->pp_ref_count) != 1 || page_is_pfmemalloc(page) || page_to_nid(page) != numa_mem_id(); } else if (!priv->rx_headroom) { @@ -493,10 +476,9 @@ static int mlx4_en_complete_rx_desc(struct mlx4_en_priv *priv, release = frags->page_offset + frag_info->frag_size > PAGE_SIZE; } if (release) { - dma_unmap_page(priv->ddev, dma, PAGE_SIZE, priv->dma_dir); frags->page = NULL; } else { - page_ref_inc(page); + page_pool_ref_page(page); } nr++; @@ -766,7 +748,8 @@ int mlx4_en_process_rx_cq(struct net_device *dev, struct mlx4_en_cq *cq, int bud /* Get pointer to first fragment since we haven't * skb yet and cast it to ethhdr struct */ - dma = frags[0].dma + frags[0].page_offset; + dma = page_pool_get_dma_addr(frags[0].page); + dma += frags[0].page_offset; dma_sync_single_for_cpu(priv->ddev, dma, sizeof(*ethh), DMA_FROM_DEVICE); @@ -805,7 +788,8 @@ int mlx4_en_process_rx_cq(struct net_device *dev, struct mlx4_en_cq *cq, int bud void *orig_data; u32 act; - dma = frags[0].dma + frags[0].page_offset; + dma = page_pool_get_dma_addr(frags[0].page); + dma += frags[0].page_offset; dma_sync_single_for_cpu(priv->ddev, dma, priv->frag_info[0].frag_size, DMA_FROM_DEVICE); @@ -868,6 +852,7 @@ int mlx4_en_process_rx_cq(struct net_device *dev, struct mlx4_en_cq *cq, int bud skb = napi_get_frags(&cq->napi); if (unlikely(!skb)) goto next; + skb_mark_for_recycle(skb); if (unlikely(ring->hwtstamp_rx_filter == HWTSTAMP_FILTER_ALL)) { u64 timestamp = mlx4_en_get_cqe_ts(cqe); diff --git a/drivers/net/ethernet/mellanox/mlx4/en_tx.c b/drivers/net/ethernet/mellanox/mlx4/en_tx.c index fe1378a689a1..87f35bcbeff8 100644 --- a/drivers/net/ethernet/mellanox/mlx4/en_tx.c +++ b/drivers/net/ethernet/mellanox/mlx4/en_tx.c @@ -44,6 +44,7 @@ #include #include #include +#include #include "mlx4_en.h" @@ -350,9 +351,10 @@ u32 mlx4_en_recycle_tx_desc(struct mlx4_en_priv *priv, int napi_mode) { struct mlx4_en_tx_info *tx_info = &ring->tx_info[index]; + struct page_pool *pool = ring->recycle_ring->pp; - dma_unmap_page(priv->ddev, tx_info->map0_dma, PAGE_SIZE, priv->dma_dir); - put_page(tx_info->page); + /* Note that napi_mode = 0 means ndo_close() path, not budget = 0 */ + page_pool_put_full_page(pool, tx_info->page, !!napi_mode); return tx_info->nr_txbb; } @@ -1189,7 +1191,7 @@ netdev_tx_t mlx4_en_xmit_frame(struct mlx4_en_rx_ring *rx_ring, tx_desc = ring->buf + (index << LOG_TXBB_SIZE); data = &tx_desc->data; - dma = frame->dma; + dma = page_pool_get_dma_addr(frame->page); tx_info->page = frame->page; frame->page = NULL;