From patchwork Tue Oct 27 19:04:07 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Lorenzo Bianconi X-Patchwork-Id: 11861597 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.1 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,INCLUDES_PATCH,MAILING_LIST_MULTI, SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5DAF9C55179 for ; Tue, 27 Oct 2020 19:04:24 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 0D0FD21D7B for ; Tue, 27 Oct 2020 19:04:24 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1603825464; bh=3kPUBW3Z65xw9KZDG7WCBMPjdP8XVy7PKBgHAtt0ups=; h=From:To:Cc:Subject:Date:In-Reply-To:References:List-ID:From; b=e2C9boQNYh/nO5tI9vt1ZG690WvspvGhyie2PCUgN/o4Z/Aadul1FLATtJKS69DAi AJy/bCTdIquKEDE+8ohX/tF8DdrPZmHcmEW/M+fk6iBprwMcB+KJWU6ZyoCSYmwiq4 a0CQhgUiNGKfxg/RfF3I2YWuqmljPPn/bgbr/hkU= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1829286AbgJ0TEW (ORCPT ); Tue, 27 Oct 2020 15:04:22 -0400 Received: from mail.kernel.org ([198.145.29.99]:41828 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2502069AbgJ0TEV (ORCPT ); Tue, 27 Oct 2020 15:04:21 -0400 Received: from lore-desk.redhat.com (unknown [151.66.125.178]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 541CE21D41; Tue, 27 Oct 2020 19:04:19 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1603825460; bh=3kPUBW3Z65xw9KZDG7WCBMPjdP8XVy7PKBgHAtt0ups=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=anNXQyp+K7sbVEqrQoatkq0eqR2BqW1yrLzKml8alSfaHPOyVL7S8o7ntxZLabIRh JOaV7g0kRY7ZLbFVoZdH2JeJ/EXvD5h4jJk/2hTYUUQG57uUrPDfksSD6Mam8nJiHa 2xTL7irgtM1uMN/O6prF+n3O5vdSfhhMiwrlPVtE= From: Lorenzo Bianconi To: netdev@vger.kernel.org Cc: bpf@vger.kernel.org, lorenzo.bianconi@redhat.com, davem@davemloft.net, kuba@kernel.org, brouer@redhat.com, ilias.apalodimas@linaro.org Subject: [PATCH net-next 1/4] net: xdp: introduce bulking for xdp tx return path Date: Tue, 27 Oct 2020 20:04:07 +0100 Message-Id: <7495b5ac96b0fd2bf5ab79b12e01bf0ee0fff803.1603824486.git.lorenzo@kernel.org> X-Mailer: git-send-email 2.26.2 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org X-Patchwork-Delegate: kuba@kernel.org Introduce bulking capability in xdp tx return path (XDP_REDIRECT). xdp_return_frame is usually run inside the driver NAPI tx completion loop so it is possible batch it. Current implementation considers only page_pool memory model. Convert mvneta driver to xdp_return_frame_bulk APIs. Suggested-by: Jesper Dangaard Brouer Signed-off-by: Lorenzo Bianconi --- drivers/net/ethernet/marvell/mvneta.c | 5 ++- include/net/xdp.h | 9 +++++ net/core/xdp.c | 51 +++++++++++++++++++++++++++ 3 files changed, 64 insertions(+), 1 deletion(-) diff --git a/drivers/net/ethernet/marvell/mvneta.c b/drivers/net/ethernet/marvell/mvneta.c index 54b0bf574c05..43ab8a73900e 100644 --- a/drivers/net/ethernet/marvell/mvneta.c +++ b/drivers/net/ethernet/marvell/mvneta.c @@ -1834,8 +1834,10 @@ static void mvneta_txq_bufs_free(struct mvneta_port *pp, struct netdev_queue *nq, bool napi) { unsigned int bytes_compl = 0, pkts_compl = 0; + struct xdp_frame_bulk bq; int i; + bq.xa = NULL; for (i = 0; i < num; i++) { struct mvneta_tx_buf *buf = &txq->buf[txq->txq_get_index]; struct mvneta_tx_desc *tx_desc = txq->descs + @@ -1857,9 +1859,10 @@ static void mvneta_txq_bufs_free(struct mvneta_port *pp, if (napi && buf->type == MVNETA_TYPE_XDP_TX) xdp_return_frame_rx_napi(buf->xdpf); else - xdp_return_frame(buf->xdpf); + xdp_return_frame_bulk(buf->xdpf, &bq); } } + xdp_flush_frame_bulk(&bq); netdev_tx_completed_queue(nq, pkts_compl, bytes_compl); } diff --git a/include/net/xdp.h b/include/net/xdp.h index 3814fb631d52..9567110845ef 100644 --- a/include/net/xdp.h +++ b/include/net/xdp.h @@ -104,6 +104,12 @@ struct xdp_frame { struct net_device *dev_rx; /* used by cpumap */ }; +#define XDP_BULK_QUEUE_SIZE 16 +struct xdp_frame_bulk { + void *q[XDP_BULK_QUEUE_SIZE]; + int count; + void *xa; +}; static inline struct skb_shared_info * xdp_get_shared_info_from_frame(struct xdp_frame *frame) @@ -194,6 +200,9 @@ struct xdp_frame *xdp_convert_buff_to_frame(struct xdp_buff *xdp) void xdp_return_frame(struct xdp_frame *xdpf); void xdp_return_frame_rx_napi(struct xdp_frame *xdpf); void xdp_return_buff(struct xdp_buff *xdp); +void xdp_flush_frame_bulk(struct xdp_frame_bulk *bq); +void xdp_return_frame_bulk(struct xdp_frame *xdpf, + struct xdp_frame_bulk *bq); /* When sending xdp_frame into the network stack, then there is no * return point callback, which is needed to release e.g. DMA-mapping diff --git a/net/core/xdp.c b/net/core/xdp.c index 48aba933a5a8..93eabd789246 100644 --- a/net/core/xdp.c +++ b/net/core/xdp.c @@ -380,6 +380,57 @@ void xdp_return_frame_rx_napi(struct xdp_frame *xdpf) } EXPORT_SYMBOL_GPL(xdp_return_frame_rx_napi); +void xdp_flush_frame_bulk(struct xdp_frame_bulk *bq) +{ + struct xdp_mem_allocator *xa = bq->xa; + int i; + + if (unlikely(!xa)) + return; + + for (i = 0; i < bq->count; i++) { + struct page *page = virt_to_head_page(bq->q[i]); + + page_pool_put_full_page(xa->page_pool, page, false); + } + bq->count = 0; +} +EXPORT_SYMBOL_GPL(xdp_flush_frame_bulk); + +void xdp_return_frame_bulk(struct xdp_frame *xdpf, + struct xdp_frame_bulk *bq) +{ + struct xdp_mem_info *mem = &xdpf->mem; + struct xdp_mem_allocator *xa, *nxa; + + if (mem->type != MEM_TYPE_PAGE_POOL) { + __xdp_return(xdpf->data, &xdpf->mem, false); + return; + } + + rcu_read_lock(); + + xa = bq->xa; + if (unlikely(!xa || mem->id != xa->mem.id)) { + nxa = rhashtable_lookup(mem_id_ht, &mem->id, mem_id_rht_params); + if (unlikely(!xa)) { + bq->count = 0; + bq->xa = nxa; + xa = nxa; + } + } + + if (mem->id != xa->mem.id || bq->count == XDP_BULK_QUEUE_SIZE) + xdp_flush_frame_bulk(bq); + + bq->q[bq->count++] = xdpf->data; + if (mem->id != xa->mem.id) + bq->xa = nxa; + + rcu_read_unlock(); +} +EXPORT_SYMBOL_GPL(xdp_return_frame_bulk); + void xdp_return_buff(struct xdp_buff *xdp) { __xdp_return(xdp->data, &xdp->rxq->mem, true); From patchwork Tue Oct 27 19:04:08 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Lorenzo Bianconi X-Patchwork-Id: 11861601 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.1 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,INCLUDES_PATCH,MAILING_LIST_MULTI, SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 31F6BC388F9 for ; Tue, 27 Oct 2020 19:04:41 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id E2B6721527 for ; Tue, 27 Oct 2020 19:04:40 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1603825481; bh=lSoZLpecrlVBCYUDmnUAWFbOwM83846UqIsq0uKlo/A=; h=From:To:Cc:Subject:Date:In-Reply-To:References:List-ID:From; b=kS9Bzj26dBXDduYvu2Cy8yNS4ct9T1XYfYU6zBBwd7oRbHiLpeL3nlqscSikPc1ER 3BhfgLL/nvIvdrh5J+7prR5yvLzZUI+FkCSSSU4OHxVUKWHfieW+gPmRCrnEoJPELf HH0LL/n94d+kvoqU5HeeoVhgml+duUTL7pFRDQIM= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2502069AbgJ0TEZ (ORCPT ); Tue, 27 Oct 2020 15:04:25 -0400 Received: from mail.kernel.org ([198.145.29.99]:41894 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1829306AbgJ0TEY (ORCPT ); Tue, 27 Oct 2020 15:04:24 -0400 Received: from lore-desk.redhat.com (unknown [151.66.125.178]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 5AFA621D24; Tue, 27 Oct 2020 19:04:22 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1603825463; bh=lSoZLpecrlVBCYUDmnUAWFbOwM83846UqIsq0uKlo/A=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=L5o3yOdmmGukNXT4OHCf9Jn2J/eqzSOi1W9OxjlJvUDJdhb4N2CWoWcWBZ0O0ZAQn /83bymjfinaZJka/HxPPAHK7qzESi1EupIz0LX5+0KdBMJZ2wedyOjR1XV1E8Jud63 gkQ/nPgdVN2j92r9yEuujzJyG+7uSlUvd8CGvmWc= From: Lorenzo Bianconi To: netdev@vger.kernel.org Cc: bpf@vger.kernel.org, lorenzo.bianconi@redhat.com, davem@davemloft.net, kuba@kernel.org, brouer@redhat.com, ilias.apalodimas@linaro.org Subject: [PATCH net-next 2/4] net: page_pool: add bulk support for ptr_ring Date: Tue, 27 Oct 2020 20:04:08 +0100 Message-Id: X-Mailer: git-send-email 2.26.2 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org X-Patchwork-Delegate: kuba@kernel.org Introduce the capability to batch page_pool ptr_ring refill since it is usually run inside the driver NAPI tx completion loop. Suggested-by: Jesper Dangaard Brouer Signed-off-by: Lorenzo Bianconi --- include/net/page_pool.h | 26 ++++++++++++++++++++++++++ net/core/page_pool.c | 33 +++++++++++++++++++++++++++++++++ net/core/xdp.c | 9 ++------- 3 files changed, 61 insertions(+), 7 deletions(-) diff --git a/include/net/page_pool.h b/include/net/page_pool.h index 81d7773f96cd..b5b195305346 100644 --- a/include/net/page_pool.h +++ b/include/net/page_pool.h @@ -152,6 +152,8 @@ struct page_pool *page_pool_create(const struct page_pool_params *params); void page_pool_destroy(struct page_pool *pool); void page_pool_use_xdp_mem(struct page_pool *pool, void (*disconnect)(void *)); void page_pool_release_page(struct page_pool *pool, struct page *page); +void page_pool_put_page_bulk(struct page_pool *pool, void **data, + int count); #else static inline void page_pool_destroy(struct page_pool *pool) { @@ -165,6 +167,11 @@ static inline void page_pool_release_page(struct page_pool *pool, struct page *page) { } + +static inline void page_pool_put_page_bulk(struct page_pool *pool, void **data, + int count) +{ +} #endif void page_pool_put_page(struct page_pool *pool, struct page *page, @@ -215,4 +222,23 @@ static inline void page_pool_nid_changed(struct page_pool *pool, int new_nid) if (unlikely(pool->p.nid != new_nid)) page_pool_update_nid(pool, new_nid); } + +static inline void page_pool_ring_lock(struct page_pool *pool) + __acquires(&pool->ring.producer_lock) +{ + if (in_serving_softirq()) + spin_lock(&pool->ring.producer_lock); + else + spin_lock_bh(&pool->ring.producer_lock); +} + +static inline void page_pool_ring_unlock(struct page_pool *pool) + __releases(&pool->ring.producer_lock) +{ + if (in_serving_softirq()) + spin_unlock(&pool->ring.producer_lock); + else + spin_unlock_bh(&pool->ring.producer_lock); +} + #endif /* _NET_PAGE_POOL_H */ diff --git a/net/core/page_pool.c b/net/core/page_pool.c index ef98372facf6..84fb21f8865e 100644 --- a/net/core/page_pool.c +++ b/net/core/page_pool.c @@ -11,6 +11,8 @@ #include #include +#include + #include #include #include @@ -408,6 +410,37 @@ void page_pool_put_page(struct page_pool *pool, struct page *page, } EXPORT_SYMBOL(page_pool_put_page); +void page_pool_put_page_bulk(struct page_pool *pool, void **data, + int count) +{ + struct page *page_ring[XDP_BULK_QUEUE_SIZE]; + int i, len = 0; + + for (i = 0; i < count; i++) { + struct page *page = virt_to_head_page(data[i]); + + if (unlikely(page_ref_count(page) != 1 || + !pool_page_reusable(pool, page))) { + page_pool_release_page(pool, page); + put_page(page); + continue; + } + + if (pool->p.flags & PP_FLAG_DMA_SYNC_DEV) + page_pool_dma_sync_for_device(pool, page, -1); + + page_ring[len++] = page; + } + + page_pool_ring_lock(pool); + for (i = 0; i < len; i++) { + if (__ptr_ring_produce(&pool->ring, page_ring[i])) + page_pool_return_page(pool, page_ring[i]); + } + page_pool_ring_unlock(pool); +} +EXPORT_SYMBOL(page_pool_put_page_bulk); + static void page_pool_empty_ring(struct page_pool *pool) { struct page *page; diff --git a/net/core/xdp.c b/net/core/xdp.c index 93eabd789246..9f9a8d14df38 100644 --- a/net/core/xdp.c +++ b/net/core/xdp.c @@ -383,16 +383,11 @@ EXPORT_SYMBOL_GPL(xdp_return_frame_rx_napi); void xdp_flush_frame_bulk(struct xdp_frame_bulk *bq) { struct xdp_mem_allocator *xa = bq->xa; - int i; - if (unlikely(!xa)) + if (unlikely(!xa || !bq->count)) return; - for (i = 0; i < bq->count; i++) { - struct page *page = virt_to_head_page(bq->q[i]); - - page_pool_put_full_page(xa->page_pool, page, false); - } + page_pool_put_page_bulk(xa->page_pool, bq->q, bq->count); bq->count = 0; } EXPORT_SYMBOL_GPL(xdp_flush_frame_bulk); From patchwork Tue Oct 27 19:04:09 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Lorenzo Bianconi X-Patchwork-Id: 11861603 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.1 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,INCLUDES_PATCH,MAILING_LIST_MULTI, SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id CCD24C4363A for ; Tue, 27 Oct 2020 19:04:28 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 8720B21D24 for ; Tue, 27 Oct 2020 19:04:28 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1603825468; bh=CEqASvpURooVyTo2fuJKqWW0pHRASjrmZUlThXYDT4w=; h=From:To:Cc:Subject:Date:In-Reply-To:References:List-ID:From; b=Q4jOQenx3A7jBNisvoYLNXM/o+Kom8aniHyJojAEVN0s0Dh4LQNz/7mXcc16oSqA1 Vk/CPg8BuueklQGPH9KWPJfjgeZrgB4HT72e3Idr3T6YXDAi1gteeXN2aV+NZ93wRL MSbGIS9ABBdt5f0eGoXTFSL2XdhSAh/Tvpj1Nd7g= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1829315AbgJ0TE1 (ORCPT ); Tue, 27 Oct 2020 15:04:27 -0400 Received: from mail.kernel.org ([198.145.29.99]:41924 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1829312AbgJ0TE0 (ORCPT ); Tue, 27 Oct 2020 15:04:26 -0400 Received: from lore-desk.redhat.com (unknown [151.66.125.178]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id D51A0206D4; Tue, 27 Oct 2020 19:04:24 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1603825466; bh=CEqASvpURooVyTo2fuJKqWW0pHRASjrmZUlThXYDT4w=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=q4DW2tArOkAsyjerl+wTBg9RkhnSGoAE2t5pymuWCI//0u9DCw2YuHcUSXKvGK3Eo NkjG+UAn7UrAvf4inii5AUHb07XecTye1VQIJR6uTUtowk7fW9oyPgl0kwh6hTcQ/K 1hSpQd58KOjQoby3389e5oHAa3RN3hy+vxPvwKxM= From: Lorenzo Bianconi To: netdev@vger.kernel.org Cc: bpf@vger.kernel.org, lorenzo.bianconi@redhat.com, davem@davemloft.net, kuba@kernel.org, brouer@redhat.com, ilias.apalodimas@linaro.org Subject: [PATCH net-next 3/4] net: mvpp2: add xdp tx return bulking support Date: Tue, 27 Oct 2020 20:04:09 +0100 Message-Id: X-Mailer: git-send-email 2.26.2 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org X-Patchwork-Delegate: kuba@kernel.org Convert mvpp2 driver to xdp_return_frame_bulk APIs. XDP_REDIRECT (upstream codepath): 1.79Mpps XDP_REDIRECT (upstream codepath + bulking APIs): 1.93Mpps Tested-by: Matteo Croce Signed-off-by: Lorenzo Bianconi --- drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c b/drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c index f6616c8933ca..04f24d1d72ab 100644 --- a/drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c +++ b/drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c @@ -2440,8 +2440,10 @@ static void mvpp2_txq_bufs_free(struct mvpp2_port *port, struct mvpp2_tx_queue *txq, struct mvpp2_txq_pcpu *txq_pcpu, int num) { + struct xdp_frame_bulk bq; int i; + bq.xa = NULL; for (i = 0; i < num; i++) { struct mvpp2_txq_pcpu_buf *tx_buf = txq_pcpu->buffs + txq_pcpu->txq_get_index; @@ -2454,10 +2456,11 @@ static void mvpp2_txq_bufs_free(struct mvpp2_port *port, dev_kfree_skb_any(tx_buf->skb); else if (tx_buf->type == MVPP2_TYPE_XDP_TX || tx_buf->type == MVPP2_TYPE_XDP_NDO) - xdp_return_frame(tx_buf->xdpf); + xdp_return_frame_bulk(tx_buf->xdpf, &bq); mvpp2_txq_inc_get(txq_pcpu); } + xdp_flush_frame_bulk(&bq); } static inline struct mvpp2_rx_queue *mvpp2_get_rx_queue(struct mvpp2_port *port, From patchwork Tue Oct 27 19:04:10 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Lorenzo Bianconi X-Patchwork-Id: 11861599 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.1 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,INCLUDES_PATCH,MAILING_LIST_MULTI, SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 06DBFC4363A for ; Tue, 27 Oct 2020 19:04:36 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id A433D21527 for ; Tue, 27 Oct 2020 19:04:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1603825475; bh=2T02Rnj9JSqlSyqTOWwETG3CNvYOciUI7owaOFmtoLQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:List-ID:From; b=KuYabLuUx47iQ93fHjKRQi1bY+/vj2u8EEE0zsahHM22npYeT6CPhbiX0cPz0k9Ca Y0ifkHtolXzvOOJSnYW7Og1gRyReijsNYVAGYInGOD+pQ6oEWEpoyN86U2IK/RVdei D+iTw8nODdkvaI7+m88orSmp2i3Ewwza/OCxcfJo= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2502334AbgJ0TEd (ORCPT ); Tue, 27 Oct 2020 15:04:33 -0400 Received: from mail.kernel.org ([198.145.29.99]:41938 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1829318AbgJ0TE3 (ORCPT ); Tue, 27 Oct 2020 15:04:29 -0400 Received: from lore-desk.redhat.com (unknown [151.66.125.178]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 3999621556; Tue, 27 Oct 2020 19:04:27 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1603825468; bh=2T02Rnj9JSqlSyqTOWwETG3CNvYOciUI7owaOFmtoLQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=rEtW5g+IUThlIqRGApCs0iXqcJCjOcxYseqMI5kv9F9H59QkeeX0za/LJmNrrAgez nO6Vr0oIspau/fYc9MrhD5qjpT73H88aujwxlHA+FeNiSLNgjKgSoYOOfe8IEVEaje 15xHExSANIUvx+ojW577MxlOBU4uI/NkQP8usbx8= From: Lorenzo Bianconi To: netdev@vger.kernel.org Cc: bpf@vger.kernel.org, lorenzo.bianconi@redhat.com, davem@davemloft.net, kuba@kernel.org, brouer@redhat.com, ilias.apalodimas@linaro.org Subject: [PATCH net-next 4/4] net: mlx5: add xdp tx return bulking support Date: Tue, 27 Oct 2020 20:04:10 +0100 Message-Id: <3fb334388ac7af755e1f03abb76a0a6335a90ff6.1603824486.git.lorenzo@kernel.org> X-Mailer: git-send-email 2.26.2 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org X-Patchwork-Delegate: kuba@kernel.org Convert mlx5 driver to xdp_return_frame_bulk APIs. XDP_REDIRECT (upstream codepath): 8.5Mpps XDP_REDIRECT (upstream codepath + bulking APIs): 10.1Mpps Signed-off-by: Lorenzo Bianconi --- drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c b/drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c index ae90d533a350..5fdfbf390d5c 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c @@ -369,8 +369,10 @@ static void mlx5e_free_xdpsq_desc(struct mlx5e_xdpsq *sq, bool recycle) { struct mlx5e_xdp_info_fifo *xdpi_fifo = &sq->db.xdpi_fifo; + struct xdp_frame_bulk bq; u16 i; + bq.xa = NULL; for (i = 0; i < wi->num_pkts; i++) { struct mlx5e_xdp_info xdpi = mlx5e_xdpi_fifo_pop(xdpi_fifo); @@ -379,7 +381,7 @@ static void mlx5e_free_xdpsq_desc(struct mlx5e_xdpsq *sq, /* XDP_TX from the XSK RQ and XDP_REDIRECT */ dma_unmap_single(sq->pdev, xdpi.frame.dma_addr, xdpi.frame.xdpf->len, DMA_TO_DEVICE); - xdp_return_frame(xdpi.frame.xdpf); + xdp_return_frame_bulk(xdpi.frame.xdpf, &bq); break; case MLX5E_XDP_XMIT_MODE_PAGE: /* XDP_TX from the regular RQ */ @@ -393,6 +395,7 @@ static void mlx5e_free_xdpsq_desc(struct mlx5e_xdpsq *sq, WARN_ON_ONCE(true); } } + xdp_flush_frame_bulk(&bq); } bool mlx5e_poll_xdpsq_cq(struct mlx5e_cq *cq)