From patchwork Thu Dec 2 00:02:22 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Toke_H=C3=B8iland-J=C3=B8rgensen?= X-Patchwork-Id: 12651351 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C8CDEC433F5 for ; Thu, 2 Dec 2021 00:04:03 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1353989AbhLBAHU (ORCPT ); Wed, 1 Dec 2021 19:07:20 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]:27717 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1344175AbhLBAHR (ORCPT ); Wed, 1 Dec 2021 19:07:17 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1638403436; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=U/VVqJ2rvS3+utXonMTYrBEeM8oEcGBPKY47J7o2598=; b=XX+GAkY+VwEhfrfduVBKBk6jW1cawTvExATVkTuUe3/ayEosE8PS+NzCoc2Q4QvSf1Ri37 9wJfjfQVgKyhtTXm3sPcuCL9NyIaFnSpDpkToG1wbCOPiF6gEkDV9JHLAQj8ADJM/gNN5Y xbKz/7vghGKeOAq7hoG3FFaXLIIgBeU= Received: from mail-ed1-f69.google.com (mail-ed1-f69.google.com [209.85.208.69]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-585-4S4UNNNvMy-ZdJ1ueW7-mA-1; Wed, 01 Dec 2021 19:03:55 -0500 X-MC-Unique: 4S4UNNNvMy-ZdJ1ueW7-mA-1 Received: by mail-ed1-f69.google.com with SMTP id y9-20020aa7c249000000b003e7bf7a1579so21804283edo.5 for ; Wed, 01 Dec 2021 16:03:54 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=U/VVqJ2rvS3+utXonMTYrBEeM8oEcGBPKY47J7o2598=; b=G8S9lKij/Y0qqfJQlU41y5TgW4sCK2lwB6slu/OjLLC3VlBG5kBMpTv2w+Vhh+iRrS FNeZJbT3VIQmw7Xm32x8CV9fCOiMh4/B1L1iaeVDXGlB4GPArjSkMaxrVSWFRx1C6452 ZRqs79RUdiFrLBlnIeH9dpWBPV7CCzr3aT3+FnxEFUZabPAkOBi12/Cw7Q2xJS1hntgg NVrs+a/YJJNc4/v221fo+2cxh8gHoVoX1nKqwqAy0Kg0IsKvjVK2JP4uIZQ3o1gYb63v /1myUQvuMHmSXMZoK4FN3qajulqmvFIsgsUPPnjOIDN3cQhhKBiH2hRPC2GSjZSegwel ySyg== X-Gm-Message-State: AOAM530jut1WhDOCYM2gDSRLUI9UIv4zsqhic10MDjriVnG2e0/qxHg1 los6wdjFzDXDT8y9kupCo6B4LV0F8nzVFZiQ+931UEx0ik49TSGG3N2kAU2Xom4nTehrP/fyUdO SjL5aEEN9Fm0DPz7Y X-Received: by 2002:a50:e102:: with SMTP id h2mr12566893edl.298.1638403433879; Wed, 01 Dec 2021 16:03:53 -0800 (PST) X-Google-Smtp-Source: ABdhPJyZmHop5Ge6qSzyuP7uAvHY/UWbBwr7FnWuIOUcaT9S7FEQFAGTzcI+yRyeuOzgpn3YScDHQA== X-Received: by 2002:a50:e102:: with SMTP id h2mr12566858edl.298.1638403433621; Wed, 01 Dec 2021 16:03:53 -0800 (PST) Received: from alrua-x1.borgediget.toke.dk ([45.145.92.2]) by smtp.gmail.com with ESMTPSA id g1sm621476eje.105.2021.12.01.16.03.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 01 Dec 2021 16:03:53 -0800 (PST) Received: by alrua-x1.borgediget.toke.dk (Postfix, from userid 1000) id ABD431802A0; Thu, 2 Dec 2021 01:03:52 +0100 (CET) From: =?utf-8?q?Toke_H=C3=B8iland-J=C3=B8rgensen?= To: Jesper Dangaard Brouer , Ilias Apalodimas , "David S. Miller" , Jakub Kicinski , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Martin KaFai Lau , Song Liu , Yonghong Song , John Fastabend , KP Singh Cc: =?utf-8?q?Toke_H=C3=B8iland-J=C3=B8rgensen?= , netdev@vger.kernel.org, bpf@vger.kernel.org Subject: [PATCH bpf-next 1/8] page_pool: Add callback to init pages when they are allocated Date: Thu, 2 Dec 2021 01:02:22 +0100 Message-Id: <20211202000232.380824-2-toke@redhat.com> X-Mailer: git-send-email 2.34.0 In-Reply-To: <20211202000232.380824-1-toke@redhat.com> References: <20211202000232.380824-1-toke@redhat.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net Add a new callback function to page_pool that, if set, will be called every time a new page is allocated. This will be used from bpf_test_run() to initialise the page data with the data provided by userspace when running XDP programs with redirect turned on. Signed-off-by: Toke Høiland-Jørgensen Acked-by: John Fastabend --- include/net/page_pool.h | 2 ++ net/core/page_pool.c | 2 ++ 2 files changed, 4 insertions(+) diff --git a/include/net/page_pool.h b/include/net/page_pool.h index 3855f069627f..a71201854c41 100644 --- a/include/net/page_pool.h +++ b/include/net/page_pool.h @@ -80,6 +80,8 @@ struct page_pool_params { enum dma_data_direction dma_dir; /* DMA mapping direction */ unsigned int max_len; /* max DMA sync memory size */ unsigned int offset; /* DMA addr offset */ + void (*init_callback)(struct page *page, void *arg); + void *init_arg; }; struct page_pool { diff --git a/net/core/page_pool.c b/net/core/page_pool.c index 9b60e4301a44..fb5a90b9d574 100644 --- a/net/core/page_pool.c +++ b/net/core/page_pool.c @@ -219,6 +219,8 @@ static void page_pool_set_pp_info(struct page_pool *pool, { page->pp = pool; page->pp_magic |= PP_SIGNATURE; + if (unlikely(pool->p.init_callback)) + pool->p.init_callback(page, pool->p.init_arg); } static void page_pool_clear_pp_info(struct page *page) From patchwork Thu Dec 2 00:02:23 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Toke_H=C3=B8iland-J=C3=B8rgensen?= X-Patchwork-Id: 12651367 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 61744C433EF for ; Thu, 2 Dec 2021 00:04:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1354016AbhLBAH4 (ORCPT ); Wed, 1 Dec 2021 19:07:56 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:26083 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1354044AbhLBAHW (ORCPT ); Wed, 1 Dec 2021 19:07:22 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1638403441; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Ds6QTVDpGNxBUDXmv5+fit0E/Kglxb7++rnaRLkBHbc=; b=KPAUVwgP4Q/PfFYyStlO/hwY7JWK+TSBUMhO4zFik4h24jRpapv0y0ZP52h3U9ePJj+OqO AoCKUcu26tem1xv7u0FPCiBOTEPRQsuSwr/E+KQV8QGmI2PLQDKAnPBfO/c6whuSIgFZmO 0U9jyKGGTeckeXTdG4JHk9MPLe0okaY= Received: from mail-ed1-f71.google.com (mail-ed1-f71.google.com [209.85.208.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-496-DxidAPJ4OB2JO0zJE0lwXw-1; Wed, 01 Dec 2021 19:04:00 -0500 X-MC-Unique: DxidAPJ4OB2JO0zJE0lwXw-1 Received: by mail-ed1-f71.google.com with SMTP id v10-20020aa7d9ca000000b003e7bed57968so21785401eds.23 for ; Wed, 01 Dec 2021 16:03:59 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Ds6QTVDpGNxBUDXmv5+fit0E/Kglxb7++rnaRLkBHbc=; b=yYyc0MfWMWOvXzCw2TgfbEf+ZpdjHA/hwSAcJSKSBBBRT7NM2O8uBvgsZxQyXeg9iN T7Fg9XTEanA7cXC52uV+5CmkoNhGAUhlz41mHJPiNuXzBPfjR/dm5l8co5J0R9UfEKAU dLj+mDESsr5CPBXMllfjd36E93cRWnIb4FKGQXv0az2Lc1XBQ0PJhH1rRSWljne0DYPG Khfe7Xtiy0xkveKx+fhQAB818D0YNYYCJ3hfiD+DD3w5p7edzusDFtZzj+Zj/mdW5UFs 021GT7eRCDy1C+Q1etQxcV71mYjwenPZYsSe5ljuZn3Hl0pq/67Fo4FROXotkBjyW7Yf T3hw== X-Gm-Message-State: AOAM531BaryzUkiKVyPU+wvf++Mr/CYgi9JJOPL5ww0XMNWCXwR7M+j2 w0myOFpYXqY78sh4wyrMlP/igKi3xRknFc+k+omUwINH5ghpLRTZLsNGnL+1T4OzjiXZY/taI9G W9r48Br/Rrbl6r6Rm X-Received: by 2002:a17:906:3b54:: with SMTP id h20mr10892759ejf.468.1638403437206; Wed, 01 Dec 2021 16:03:57 -0800 (PST) X-Google-Smtp-Source: ABdhPJyoufdJJICk1VnB2bgN091I/pO0LQ6Z0sm63QuEmHyms076/9FNiAEOYKSR6KhAIhKoEjjgUg== X-Received: by 2002:a17:906:3b54:: with SMTP id h20mr10892689ejf.468.1638403436491; Wed, 01 Dec 2021 16:03:56 -0800 (PST) Received: from alrua-x1.borgediget.toke.dk ([2a0c:4d80:42:443::2]) by smtp.gmail.com with ESMTPSA id q7sm796702edr.9.2021.12.01.16.03.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 01 Dec 2021 16:03:56 -0800 (PST) Received: by alrua-x1.borgediget.toke.dk (Postfix, from userid 1000) id 8C1FB1802A0; Thu, 2 Dec 2021 01:03:54 +0100 (CET) From: =?utf-8?q?Toke_H=C3=B8iland-J=C3=B8rgensen?= To: Jesper Dangaard Brouer , Ilias Apalodimas , "David S. Miller" , Jakub Kicinski , Alexei Starovoitov , Daniel Borkmann , John Fastabend , Andrii Nakryiko , Martin KaFai Lau , Song Liu , Yonghong Song , KP Singh Cc: =?utf-8?q?Toke_H=C3=B8iland-J=C3=B8rgensen?= , netdev@vger.kernel.org, bpf@vger.kernel.org Subject: [PATCH bpf-next 2/8] page_pool: Store the XDP mem id Date: Thu, 2 Dec 2021 01:02:23 +0100 Message-Id: <20211202000232.380824-3-toke@redhat.com> X-Mailer: git-send-email 2.34.0 In-Reply-To: <20211202000232.380824-1-toke@redhat.com> References: <20211202000232.380824-1-toke@redhat.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net Store the XDP mem ID inside the page_pool struct so it can be retrieved later for use in bpf_prog_run(). Signed-off-by: Toke Høiland-Jørgensen --- include/net/page_pool.h | 9 +++++++-- net/core/page_pool.c | 4 +++- net/core/xdp.c | 2 +- 3 files changed, 11 insertions(+), 4 deletions(-) diff --git a/include/net/page_pool.h b/include/net/page_pool.h index a71201854c41..6bc0409c4ffd 100644 --- a/include/net/page_pool.h +++ b/include/net/page_pool.h @@ -96,6 +96,7 @@ struct page_pool { unsigned int frag_offset; struct page *frag_page; long frag_users; + u32 xdp_mem_id; /* * Data structure for allocation side @@ -170,9 +171,12 @@ bool page_pool_return_skb_page(struct page *page); struct page_pool *page_pool_create(const struct page_pool_params *params); +struct xdp_mem_info; + #ifdef CONFIG_PAGE_POOL void page_pool_destroy(struct page_pool *pool); -void page_pool_use_xdp_mem(struct page_pool *pool, void (*disconnect)(void *)); +void page_pool_use_xdp_mem(struct page_pool *pool, void (*disconnect)(void *), + struct xdp_mem_info *mem); void page_pool_release_page(struct page_pool *pool, struct page *page); void page_pool_put_page_bulk(struct page_pool *pool, void **data, int count); @@ -182,7 +186,8 @@ static inline void page_pool_destroy(struct page_pool *pool) } static inline void page_pool_use_xdp_mem(struct page_pool *pool, - void (*disconnect)(void *)) + void (*disconnect)(void *), + struct xdp_mem_info *mem) { } static inline void page_pool_release_page(struct page_pool *pool, diff --git a/net/core/page_pool.c b/net/core/page_pool.c index fb5a90b9d574..2605467251f1 100644 --- a/net/core/page_pool.c +++ b/net/core/page_pool.c @@ -695,10 +695,12 @@ static void page_pool_release_retry(struct work_struct *wq) schedule_delayed_work(&pool->release_dw, DEFER_TIME); } -void page_pool_use_xdp_mem(struct page_pool *pool, void (*disconnect)(void *)) +void page_pool_use_xdp_mem(struct page_pool *pool, void (*disconnect)(void *), + struct xdp_mem_info *mem) { refcount_inc(&pool->user_cnt); pool->disconnect = disconnect; + pool->xdp_mem_id = mem->id; } void page_pool_destroy(struct page_pool *pool) diff --git a/net/core/xdp.c b/net/core/xdp.c index 5ddc29f29bad..143388c6d9dd 100644 --- a/net/core/xdp.c +++ b/net/core/xdp.c @@ -318,7 +318,7 @@ int xdp_rxq_info_reg_mem_model(struct xdp_rxq_info *xdp_rxq, } if (type == MEM_TYPE_PAGE_POOL) - page_pool_use_xdp_mem(allocator, mem_allocator_disconnect); + page_pool_use_xdp_mem(allocator, mem_allocator_disconnect, mem); mutex_unlock(&mem_id_lock); From patchwork Thu Dec 2 00:02:24 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Toke_H=C3=B8iland-J=C3=B8rgensen?= X-Patchwork-Id: 12651355 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D711CC433EF for ; Thu, 2 Dec 2021 00:04:14 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1354033AbhLBAHe (ORCPT ); Wed, 1 Dec 2021 19:07:34 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]:50647 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1354032AbhLBAHW (ORCPT ); Wed, 1 Dec 2021 19:07:22 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1638403440; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=xDXI2dfRgWUXP7V439OsHy/fTRVufq6CwTbb4xhL8Gk=; b=Po3SWBSNwINeUuOoxXesRm1KGsP190LaxCUeXBYdo1qveqRQoXlI5wrmu+Nj391p5H/j4S FjwQ+i6p6CDgCvtcvJQv5wf1HIVS/pYf+qU/fbt9f0GOi9+lChdNGBvnOYj2A4+jQ75t0z D8hd/Z7z0c4cgrP7etnfI6XBEXeDWgI= Received: from mail-ed1-f70.google.com (mail-ed1-f70.google.com [209.85.208.70]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-171-l6UfRk4nOsiNjoY_oWja2Q-1; Wed, 01 Dec 2021 19:03:59 -0500 X-MC-Unique: l6UfRk4nOsiNjoY_oWja2Q-1 Received: by mail-ed1-f70.google.com with SMTP id v10-20020aa7d9ca000000b003e7bed57968so21785376eds.23 for ; Wed, 01 Dec 2021 16:03:59 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=xDXI2dfRgWUXP7V439OsHy/fTRVufq6CwTbb4xhL8Gk=; b=SXAq/0urTAOazq/OFM/Bs2qAmn2nJYAmOESegGqG/od/Sx3uciVSpxC5hXM9lBZ8DW fJRlwVQp4KJJ/urH0A06l2WTXUMrmzeHmsN4YHNRH8UrylMGlXtH/cSIL+bsKl+kb2Eg G/dhQ1C0L4pj5U+kiakGDvxF1X5+A5pqAxKDc1iTSnGbFHGnN5NNiMJu5/vCbBkc+822 R3Iq/mUuveOEBTSCKtMNDHws0Sc/8GRi0Aooci5z+ZlRGUjS3YDsI7yNlYrMuCjMGT3M 8vT10JonB4dw80bFBh/CKlVq12dN7WS1YD//HKTEPONaJ2KfOt+anKtmvsYF3W3nzpRL b1rg== X-Gm-Message-State: AOAM530vjUSfPCkfR50ubIzt5Qq45N3cHso/jKo40+oWNDRzxecGlsPb CPcjUScXZS+RBk1DFxG5HkeAoZXRzldjiiy3F6+uGA7G07fWRIz7WhXeAP6m1mjkAhovarfaGer CkIkq8VEkVwQF3z3n X-Received: by 2002:a17:906:3085:: with SMTP id 5mr10705857ejv.365.1638403437914; Wed, 01 Dec 2021 16:03:57 -0800 (PST) X-Google-Smtp-Source: ABdhPJxEEM3I9IYJgFZKYnyZ+to/tY9V079Mc7mCeD/f4tsRiwkPVmnLg07hOTYahOEa4Roy/ipWLw== X-Received: by 2002:a17:906:3085:: with SMTP id 5mr10705810ejv.365.1638403437569; Wed, 01 Dec 2021 16:03:57 -0800 (PST) Received: from alrua-x1.borgediget.toke.dk ([45.145.92.2]) by smtp.gmail.com with ESMTPSA id ne33sm692644ejc.6.2021.12.01.16.03.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 01 Dec 2021 16:03:57 -0800 (PST) Received: by alrua-x1.borgediget.toke.dk (Postfix, from userid 1000) id 95C1C1802A0; Thu, 2 Dec 2021 01:03:56 +0100 (CET) From: =?utf-8?q?Toke_H=C3=B8iland-J=C3=B8rgensen?= To: Alexei Starovoitov , Daniel Borkmann , "David S. Miller" , Jakub Kicinski , Jesper Dangaard Brouer , John Fastabend , Andrii Nakryiko , Martin KaFai Lau , Song Liu , Yonghong Song , KP Singh Cc: =?utf-8?q?Toke_H=C3=B8iland-J=C3=B8rgensen?= , netdev@vger.kernel.org, bpf@vger.kernel.org Subject: [PATCH bpf-next 3/8] xdp: Allow registering memory model without rxq reference Date: Thu, 2 Dec 2021 01:02:24 +0100 Message-Id: <20211202000232.380824-4-toke@redhat.com> X-Mailer: git-send-email 2.34.0 In-Reply-To: <20211202000232.380824-1-toke@redhat.com> References: <20211202000232.380824-1-toke@redhat.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net The functions that register an XDP memory model take a struct xdp_rxq as parameter, but the RXQ is not actually used for anything other than pulling out the struct xdp_mem_info that it embeds. So refactor the register functions and export variants that just take a pointer to the xdp_mem_info. This is in preparation for enabling XDP_REDIRECT in bpf_prog_run(), using a page_pool instance that is not connected to any network device. Signed-off-by: Toke Høiland-Jørgensen --- include/net/xdp.h | 3 ++ net/core/xdp.c | 92 +++++++++++++++++++++++++++++++---------------- 2 files changed, 65 insertions(+), 30 deletions(-) diff --git a/include/net/xdp.h b/include/net/xdp.h index 447f9b1578f3..8f0812e4996d 100644 --- a/include/net/xdp.h +++ b/include/net/xdp.h @@ -260,6 +260,9 @@ bool xdp_rxq_info_is_reg(struct xdp_rxq_info *xdp_rxq); int xdp_rxq_info_reg_mem_model(struct xdp_rxq_info *xdp_rxq, enum xdp_mem_type type, void *allocator); void xdp_rxq_info_unreg_mem_model(struct xdp_rxq_info *xdp_rxq); +int xdp_reg_mem_model(struct xdp_mem_info *mem, + enum xdp_mem_type type, void *allocator); +void xdp_unreg_mem_model(struct xdp_mem_info *mem); /* Drivers not supporting XDP metadata can use this helper, which * rejects any room expansion for metadata as a result. diff --git a/net/core/xdp.c b/net/core/xdp.c index 143388c6d9dd..2901bb7004cc 100644 --- a/net/core/xdp.c +++ b/net/core/xdp.c @@ -110,20 +110,15 @@ static void mem_allocator_disconnect(void *allocator) mutex_unlock(&mem_id_lock); } -void xdp_rxq_info_unreg_mem_model(struct xdp_rxq_info *xdp_rxq) +void xdp_unreg_mem_model(struct xdp_mem_info *mem) { struct xdp_mem_allocator *xa; - int type = xdp_rxq->mem.type; - int id = xdp_rxq->mem.id; + int type = mem->type; + int id = mem->id; /* Reset mem info to defaults */ - xdp_rxq->mem.id = 0; - xdp_rxq->mem.type = 0; - - if (xdp_rxq->reg_state != REG_STATE_REGISTERED) { - WARN(1, "Missing register, driver bug"); - return; - } + mem->id = 0; + mem->type = 0; if (id == 0) return; @@ -135,6 +130,17 @@ void xdp_rxq_info_unreg_mem_model(struct xdp_rxq_info *xdp_rxq) rcu_read_unlock(); } } +EXPORT_SYMBOL_GPL(xdp_unreg_mem_model); + +void xdp_rxq_info_unreg_mem_model(struct xdp_rxq_info *xdp_rxq) +{ + if (xdp_rxq->reg_state != REG_STATE_REGISTERED) { + WARN(1, "Missing register, driver bug"); + return; + } + + xdp_unreg_mem_model(&xdp_rxq->mem); +} EXPORT_SYMBOL_GPL(xdp_rxq_info_unreg_mem_model); void xdp_rxq_info_unreg(struct xdp_rxq_info *xdp_rxq) @@ -259,28 +265,24 @@ static bool __is_supported_mem_type(enum xdp_mem_type type) return true; } -int xdp_rxq_info_reg_mem_model(struct xdp_rxq_info *xdp_rxq, - enum xdp_mem_type type, void *allocator) +static struct xdp_mem_allocator *__xdp_reg_mem_model(struct xdp_mem_info *mem, + enum xdp_mem_type type, + void *allocator) { struct xdp_mem_allocator *xdp_alloc; gfp_t gfp = GFP_KERNEL; int id, errno, ret; void *ptr; - if (xdp_rxq->reg_state != REG_STATE_REGISTERED) { - WARN(1, "Missing register, driver bug"); - return -EFAULT; - } - if (!__is_supported_mem_type(type)) - return -EOPNOTSUPP; + return ERR_PTR(-EOPNOTSUPP); - xdp_rxq->mem.type = type; + mem->type = type; if (!allocator) { if (type == MEM_TYPE_PAGE_POOL) - return -EINVAL; /* Setup time check page_pool req */ - return 0; + return ERR_PTR(-EINVAL); /* Setup time check page_pool req */ + return NULL; } /* Delay init of rhashtable to save memory if feature isn't used */ @@ -290,13 +292,13 @@ int xdp_rxq_info_reg_mem_model(struct xdp_rxq_info *xdp_rxq, mutex_unlock(&mem_id_lock); if (ret < 0) { WARN_ON(1); - return ret; + return ERR_PTR(ret); } } xdp_alloc = kzalloc(sizeof(*xdp_alloc), gfp); if (!xdp_alloc) - return -ENOMEM; + return ERR_PTR(-ENOMEM); mutex_lock(&mem_id_lock); id = __mem_id_cyclic_get(gfp); @@ -304,15 +306,15 @@ int xdp_rxq_info_reg_mem_model(struct xdp_rxq_info *xdp_rxq, errno = id; goto err; } - xdp_rxq->mem.id = id; - xdp_alloc->mem = xdp_rxq->mem; + mem->id = id; + xdp_alloc->mem = *mem; xdp_alloc->allocator = allocator; /* Insert allocator into ID lookup table */ ptr = rhashtable_insert_slow(mem_id_ht, &id, &xdp_alloc->node); if (IS_ERR(ptr)) { - ida_simple_remove(&mem_id_pool, xdp_rxq->mem.id); - xdp_rxq->mem.id = 0; + ida_simple_remove(&mem_id_pool, mem->id); + mem->id = 0; errno = PTR_ERR(ptr); goto err; } @@ -322,13 +324,43 @@ int xdp_rxq_info_reg_mem_model(struct xdp_rxq_info *xdp_rxq, mutex_unlock(&mem_id_lock); - trace_mem_connect(xdp_alloc, xdp_rxq); - return 0; + return xdp_alloc; err: mutex_unlock(&mem_id_lock); kfree(xdp_alloc); - return errno; + return ERR_PTR(errno); +} + +int xdp_reg_mem_model(struct xdp_mem_info *mem, + enum xdp_mem_type type, void *allocator) +{ + struct xdp_mem_allocator *xdp_alloc; + + xdp_alloc = __xdp_reg_mem_model(mem, type, allocator); + if (IS_ERR(xdp_alloc)) + return PTR_ERR(xdp_alloc); + return 0; +} +EXPORT_SYMBOL_GPL(xdp_reg_mem_model); + +int xdp_rxq_info_reg_mem_model(struct xdp_rxq_info *xdp_rxq, + enum xdp_mem_type type, void *allocator) +{ + struct xdp_mem_allocator *xdp_alloc; + + if (xdp_rxq->reg_state != REG_STATE_REGISTERED) { + WARN(1, "Missing register, driver bug"); + return -EFAULT; + } + + xdp_alloc = __xdp_reg_mem_model(&xdp_rxq->mem, type, allocator); + if (IS_ERR(xdp_alloc)) + return PTR_ERR(xdp_alloc); + + trace_mem_connect(xdp_alloc, xdp_rxq); + return 0; } + EXPORT_SYMBOL_GPL(xdp_rxq_info_reg_mem_model); /* XDP RX runs under NAPI protection, and in different delivery error From patchwork Thu Dec 2 00:02:25 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Toke_H=C3=B8iland-J=C3=B8rgensen?= X-Patchwork-Id: 12651357 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7C8B7C433FE for ; Thu, 2 Dec 2021 00:04:26 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1354054AbhLBAHq (ORCPT ); Wed, 1 Dec 2021 19:07:46 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]:33040 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1353972AbhLBAHc (ORCPT ); Wed, 1 Dec 2021 19:07:32 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1638403446; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=06j+7gHempL9IdsXmq8SYxnXXYon7FIwMCRzsyscV9A=; b=Z5O3dji6OGSEBC/J+YS8qQaU/P5E1sgO9IN8XNc9SMcFXSZb0eFgNYRVKWv8dXrGWGGb+D vkQMBt1gUHzoUgVpNT5omFAKNnM8VOc3Y5BJkFXJy12YKc0eBl1pM1AwAYu1BALDtxqfkw rgLWWrThQwlUYXkhe57Edd6TVJcn5dw= Received: from mail-ed1-f71.google.com (mail-ed1-f71.google.com [209.85.208.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-431-cQkb4j_7M4Sj7T1xZnd2xw-1; Wed, 01 Dec 2021 19:04:05 -0500 X-MC-Unique: cQkb4j_7M4Sj7T1xZnd2xw-1 Received: by mail-ed1-f71.google.com with SMTP id v10-20020aa7d9ca000000b003e7bed57968so21785537eds.23 for ; Wed, 01 Dec 2021 16:04:05 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=06j+7gHempL9IdsXmq8SYxnXXYon7FIwMCRzsyscV9A=; b=oNyiG12UaiZx30a1b/vTQDzoWefsg2p2y4JD+8KldFVVDh2rVLVKE/H52C2wQbWJ86 D0xoZqxEr+AVH7KceA5dxfzEA3pQQN+jS0IGvXsdzszBYcszgb03v8FoiRxeQrgq2rOC 8PiTUN5TBPm/qm0SWegEz1buXxN7fDxie4roMY7tx89b9qT5V/7lgyfPIflfmRlmpK02 ZmoHKRRrwsUg+/YB8MKI5stZ3m6UA3rlt9S35nyLk3GFxHLmrSY7EEOOVwU0OWGmud6x TTL5+38kiJHyQxEsgQy/BlzO036w53CrlJcXcOMlvDtyRdgSpTkLvXvb0TElY+Y0xUOu SYnA== X-Gm-Message-State: AOAM532AhVNXyGXPPRoYyWzdSEku1g9aKhEm7WmlnT88CE2Qn3z5bQju hbbxurjQ6sShUOTHKRjyGb3FIQ1I4F26gSc5NGUeUrf1zMEnzveHeOBvxocmrKQ1fX9SEuoodmu BmwqfXFRSOvsiR8cT X-Received: by 2002:a17:907:98f6:: with SMTP id ke22mr10821023ejc.500.1638403442845; Wed, 01 Dec 2021 16:04:02 -0800 (PST) X-Google-Smtp-Source: ABdhPJypRXYfDUMPmfyC6INng9VHJzWhJNLcJcTPchSayKiWEbCjbfpmDg8y84uQUhPFeoqDYWOmIA== X-Received: by 2002:a17:907:98f6:: with SMTP id ke22mr10820911ejc.500.1638403441890; Wed, 01 Dec 2021 16:04:01 -0800 (PST) Received: from alrua-x1.borgediget.toke.dk ([2a0c:4d80:42:443::2]) by smtp.gmail.com with ESMTPSA id c8sm744776edu.60.2021.12.01.16.04.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 01 Dec 2021 16:04:01 -0800 (PST) Received: by alrua-x1.borgediget.toke.dk (Postfix, from userid 1000) id 5190A1802A0; Thu, 2 Dec 2021 01:03:59 +0100 (CET) From: =?utf-8?q?Toke_H=C3=B8iland-J=C3=B8rgensen?= To: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Martin KaFai Lau , Song Liu , Yonghong Song , John Fastabend , KP Singh , "David S. Miller" , Jakub Kicinski , Jesper Dangaard Brouer Cc: =?utf-8?q?Toke_H=C3=B8iland-J=C3=B8rgensen?= , netdev@vger.kernel.org, bpf@vger.kernel.org Subject: [PATCH bpf-next 4/8] xdp: Move conversion to xdp_frame out of map functions Date: Thu, 2 Dec 2021 01:02:25 +0100 Message-Id: <20211202000232.380824-5-toke@redhat.com> X-Mailer: git-send-email 2.34.0 In-Reply-To: <20211202000232.380824-1-toke@redhat.com> References: <20211202000232.380824-1-toke@redhat.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net All map redirect functions except XSK maps convert xdp_buff to xdp_frame before enqueueing it. So move this conversion of out the map functions and into xdp_do_redirect(). This removes a bit of duplicated code, but more importantly it makes it possible to support caller-allocated xdp_frame structures, which will be added in a subsequent commit. Signed-off-by: Toke Høiland-Jørgensen --- include/linux/bpf.h | 20 ++++++++++---------- kernel/bpf/cpumap.c | 8 +------- kernel/bpf/devmap.c | 32 +++++++++++--------------------- net/core/filter.c | 23 +++++++++++++++++------ 4 files changed, 39 insertions(+), 44 deletions(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index cad0829710be..cd756294446f 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -1621,17 +1621,17 @@ void bpf_patch_call_args(struct bpf_insn *insn, u32 stack_depth); struct btf *bpf_get_btf_vmlinux(void); /* Map specifics */ -struct xdp_buff; +struct xdp_frame; struct sk_buff; struct bpf_dtab_netdev; struct bpf_cpu_map_entry; void __dev_flush(void); -int dev_xdp_enqueue(struct net_device *dev, struct xdp_buff *xdp, +int dev_xdp_enqueue(struct net_device *dev, struct xdp_frame *xdpf, struct net_device *dev_rx); -int dev_map_enqueue(struct bpf_dtab_netdev *dst, struct xdp_buff *xdp, +int dev_map_enqueue(struct bpf_dtab_netdev *dst, struct xdp_frame *xdpf, struct net_device *dev_rx); -int dev_map_enqueue_multi(struct xdp_buff *xdp, struct net_device *dev_rx, +int dev_map_enqueue_multi(struct xdp_frame *xdpf, struct net_device *dev_rx, struct bpf_map *map, bool exclude_ingress); int dev_map_generic_redirect(struct bpf_dtab_netdev *dst, struct sk_buff *skb, struct bpf_prog *xdp_prog); @@ -1640,7 +1640,7 @@ int dev_map_redirect_multi(struct net_device *dev, struct sk_buff *skb, bool exclude_ingress); void __cpu_map_flush(void); -int cpu_map_enqueue(struct bpf_cpu_map_entry *rcpu, struct xdp_buff *xdp, +int cpu_map_enqueue(struct bpf_cpu_map_entry *rcpu, struct xdp_frame *xdpf, struct net_device *dev_rx); int cpu_map_generic_redirect(struct bpf_cpu_map_entry *rcpu, struct sk_buff *skb); @@ -1810,26 +1810,26 @@ static inline void __dev_flush(void) { } -struct xdp_buff; +struct xdp_frame; struct bpf_dtab_netdev; struct bpf_cpu_map_entry; static inline -int dev_xdp_enqueue(struct net_device *dev, struct xdp_buff *xdp, +int dev_xdp_enqueue(struct net_device *dev, struct xdp_frame *xdpf, struct net_device *dev_rx) { return 0; } static inline -int dev_map_enqueue(struct bpf_dtab_netdev *dst, struct xdp_buff *xdp, +int dev_map_enqueue(struct bpf_dtab_netdev *dst, struct xdp_frame *xdpf, struct net_device *dev_rx) { return 0; } static inline -int dev_map_enqueue_multi(struct xdp_buff *xdp, struct net_device *dev_rx, +int dev_map_enqueue_multi(struct xdp_frame *xdpf, struct net_device *dev_rx, struct bpf_map *map, bool exclude_ingress) { return 0; @@ -1857,7 +1857,7 @@ static inline void __cpu_map_flush(void) } static inline int cpu_map_enqueue(struct bpf_cpu_map_entry *rcpu, - struct xdp_buff *xdp, + struct xdp_frame *xdpf, struct net_device *dev_rx) { return 0; diff --git a/kernel/bpf/cpumap.c b/kernel/bpf/cpumap.c index 585b2b77ccc4..12798b2c68d9 100644 --- a/kernel/bpf/cpumap.c +++ b/kernel/bpf/cpumap.c @@ -746,15 +746,9 @@ static void bq_enqueue(struct bpf_cpu_map_entry *rcpu, struct xdp_frame *xdpf) list_add(&bq->flush_node, flush_list); } -int cpu_map_enqueue(struct bpf_cpu_map_entry *rcpu, struct xdp_buff *xdp, +int cpu_map_enqueue(struct bpf_cpu_map_entry *rcpu, struct xdp_frame *xdpf, struct net_device *dev_rx) { - struct xdp_frame *xdpf; - - xdpf = xdp_convert_buff_to_frame(xdp); - if (unlikely(!xdpf)) - return -EOVERFLOW; - /* Info needed when constructing SKB on remote CPU */ xdpf->dev_rx = dev_rx; diff --git a/kernel/bpf/devmap.c b/kernel/bpf/devmap.c index f02d04540c0c..f29f439fac76 100644 --- a/kernel/bpf/devmap.c +++ b/kernel/bpf/devmap.c @@ -467,24 +467,19 @@ static void bq_enqueue(struct net_device *dev, struct xdp_frame *xdpf, bq->q[bq->count++] = xdpf; } -static inline int __xdp_enqueue(struct net_device *dev, struct xdp_buff *xdp, +static inline int __xdp_enqueue(struct net_device *dev, struct xdp_frame *xdpf, struct net_device *dev_rx, struct bpf_prog *xdp_prog) { - struct xdp_frame *xdpf; int err; if (!dev->netdev_ops->ndo_xdp_xmit) return -EOPNOTSUPP; - err = xdp_ok_fwd_dev(dev, xdp->data_end - xdp->data); + err = xdp_ok_fwd_dev(dev, xdpf->len); if (unlikely(err)) return err; - xdpf = xdp_convert_buff_to_frame(xdp); - if (unlikely(!xdpf)) - return -EOVERFLOW; - bq_enqueue(dev, xdpf, dev_rx, xdp_prog); return 0; } @@ -520,27 +515,27 @@ static u32 dev_map_bpf_prog_run_skb(struct sk_buff *skb, struct bpf_dtab_netdev return act; } -int dev_xdp_enqueue(struct net_device *dev, struct xdp_buff *xdp, +int dev_xdp_enqueue(struct net_device *dev, struct xdp_frame *xdpf, struct net_device *dev_rx) { - return __xdp_enqueue(dev, xdp, dev_rx, NULL); + return __xdp_enqueue(dev, xdpf, dev_rx, NULL); } -int dev_map_enqueue(struct bpf_dtab_netdev *dst, struct xdp_buff *xdp, +int dev_map_enqueue(struct bpf_dtab_netdev *dst, struct xdp_frame *xdpf, struct net_device *dev_rx) { struct net_device *dev = dst->dev; - return __xdp_enqueue(dev, xdp, dev_rx, dst->xdp_prog); + return __xdp_enqueue(dev, xdpf, dev_rx, dst->xdp_prog); } -static bool is_valid_dst(struct bpf_dtab_netdev *obj, struct xdp_buff *xdp) +static bool is_valid_dst(struct bpf_dtab_netdev *obj, struct xdp_frame *xdpf) { if (!obj || !obj->dev->netdev_ops->ndo_xdp_xmit) return false; - if (xdp_ok_fwd_dev(obj->dev, xdp->data_end - xdp->data)) + if (xdp_ok_fwd_dev(obj->dev, xdpf->len)) return false; return true; @@ -586,14 +581,13 @@ static int get_upper_ifindexes(struct net_device *dev, int *indexes) return n; } -int dev_map_enqueue_multi(struct xdp_buff *xdp, struct net_device *dev_rx, +int dev_map_enqueue_multi(struct xdp_frame *xdpf, struct net_device *dev_rx, struct bpf_map *map, bool exclude_ingress) { struct bpf_dtab *dtab = container_of(map, struct bpf_dtab, map); struct bpf_dtab_netdev *dst, *last_dst = NULL; int excluded_devices[1+MAX_NEST_DEV]; struct hlist_head *head; - struct xdp_frame *xdpf; int num_excluded = 0; unsigned int i; int err; @@ -603,15 +597,11 @@ int dev_map_enqueue_multi(struct xdp_buff *xdp, struct net_device *dev_rx, excluded_devices[num_excluded++] = dev_rx->ifindex; } - xdpf = xdp_convert_buff_to_frame(xdp); - if (unlikely(!xdpf)) - return -EOVERFLOW; - if (map->map_type == BPF_MAP_TYPE_DEVMAP) { for (i = 0; i < map->max_entries; i++) { dst = rcu_dereference_check(dtab->netdev_map[i], rcu_read_lock_bh_held()); - if (!is_valid_dst(dst, xdp)) + if (!is_valid_dst(dst, xdpf)) continue; if (is_ifindex_excluded(excluded_devices, num_excluded, dst->dev->ifindex)) @@ -634,7 +624,7 @@ int dev_map_enqueue_multi(struct xdp_buff *xdp, struct net_device *dev_rx, head = dev_map_index_hash(dtab, i); hlist_for_each_entry_rcu(dst, head, index_hlist, lockdep_is_held(&dtab->index_lock)) { - if (!is_valid_dst(dst, xdp)) + if (!is_valid_dst(dst, xdpf)) continue; if (is_ifindex_excluded(excluded_devices, num_excluded, diff --git a/net/core/filter.c b/net/core/filter.c index fe27c91e3758..1e86130a913a 100644 --- a/net/core/filter.c +++ b/net/core/filter.c @@ -3964,12 +3964,24 @@ int xdp_do_redirect(struct net_device *dev, struct xdp_buff *xdp, enum bpf_map_type map_type = ri->map_type; void *fwd = ri->tgt_value; u32 map_id = ri->map_id; + struct xdp_frame *xdpf; struct bpf_map *map; int err; ri->map_id = 0; /* Valid map id idr range: [1,INT_MAX[ */ ri->map_type = BPF_MAP_TYPE_UNSPEC; + if (map_type == BPF_MAP_TYPE_XSKMAP) { + err = __xsk_map_redirect(fwd, xdp); + goto out; + } + + xdpf = xdp_convert_buff_to_frame(xdp); + if (unlikely(!xdpf)) { + err = -EOVERFLOW; + goto err; + } + switch (map_type) { case BPF_MAP_TYPE_DEVMAP: fallthrough; @@ -3977,17 +3989,15 @@ int xdp_do_redirect(struct net_device *dev, struct xdp_buff *xdp, map = READ_ONCE(ri->map); if (unlikely(map)) { WRITE_ONCE(ri->map, NULL); - err = dev_map_enqueue_multi(xdp, dev, map, + err = dev_map_enqueue_multi(xdpf, dev, map, ri->flags & BPF_F_EXCLUDE_INGRESS); } else { - err = dev_map_enqueue(fwd, xdp, dev); + err = dev_map_enqueue(fwd, xdpf, dev); } break; case BPF_MAP_TYPE_CPUMAP: - err = cpu_map_enqueue(fwd, xdp, dev); + err = cpu_map_enqueue(fwd, xdpf, dev); break; - case BPF_MAP_TYPE_XSKMAP: - err = __xsk_map_redirect(fwd, xdp); break; case BPF_MAP_TYPE_UNSPEC: if (map_id == INT_MAX) { @@ -3996,7 +4006,7 @@ int xdp_do_redirect(struct net_device *dev, struct xdp_buff *xdp, err = -EINVAL; break; } - err = dev_xdp_enqueue(fwd, xdp, dev); + err = dev_xdp_enqueue(fwd, xdpf, dev); break; } fallthrough; @@ -4004,6 +4014,7 @@ int xdp_do_redirect(struct net_device *dev, struct xdp_buff *xdp, err = -EBADRQC; } +out: if (unlikely(err)) goto err; From patchwork Thu Dec 2 00:02:26 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Toke_H=C3=B8iland-J=C3=B8rgensen?= X-Patchwork-Id: 12651359 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3A468C433EF for ; Thu, 2 Dec 2021 00:04:28 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1354022AbhLBAHr (ORCPT ); Wed, 1 Dec 2021 19:07:47 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]:35384 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229665AbhLBAHc (ORCPT ); Wed, 1 Dec 2021 19:07:32 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1638403445; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=2W/44qFKQfDTqcBT48p2Uxe3qzLn64OCXa3a4zfm4NU=; b=cd90q82GGK2DRuzC1vpEyq2JgCZvGR3umY3YstBva6aP/uF4xLpZz2C3IBnhCmHz5dR+Im qnBO8R00VSRRtZ3w8x72QcF0k9SqGwNEUpNNGh5D0j39LtxPd6ZOUCCFTJK9LZPd7Ohug+ kWXi1Us8nYoU/0P5r7dQscwxnjnb9Ls= Received: from mail-ed1-f72.google.com (mail-ed1-f72.google.com [209.85.208.72]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-497-_G5zjUniO2Kf-gEo6zb56g-1; Wed, 01 Dec 2021 19:04:04 -0500 X-MC-Unique: _G5zjUniO2Kf-gEo6zb56g-1 Received: by mail-ed1-f72.google.com with SMTP id v10-20020aa7d9ca000000b003e7bed57968so21785505eds.23 for ; Wed, 01 Dec 2021 16:04:04 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=2W/44qFKQfDTqcBT48p2Uxe3qzLn64OCXa3a4zfm4NU=; b=wivO4GCa3VqDkIUFCAWaDd4nqCFZ/Bu1vf5+O+C8DJbe1FJ87AXllCNmrA4Ta/llqM TQXgm84U/8eVDtt/gwIG2fshDOUuB+HfhghvS0qAihMbetAKsBg6ToJUAroLB5VjQE1P px0r8IFH9bho0a2ZqZ8JEQVyS4MkQfbcb0cCHHbupGglJoo3SWnlgBYrtCDAJEXMMFID tYGm10ht7zxou4b73hkug9hETYi4Sl5uenCDWJX/+aYijPcTX7CV3G6YAIja7dZ9axu0 j0VIRErgcNaG22LSVRSTeOe0rRYQ8KPxGO9/pqBxsCABmM74Y2CEkOlsIJJK8uvll2xT 2BcA== X-Gm-Message-State: AOAM531aqnuApyHMQsZY8sYAalxqhx9qERK617v7+ZJVbbeJ4Rhp4of6 Kp3JSKdZ3vZMMYu2DwZmPVM4SoTTMd35mlbhzvb6FMOj3HbQoZDzZABPu4fJLiwhiILI38xXl1+ 3OYXamc7xUK3c8fp9 X-Received: by 2002:aa7:cd99:: with SMTP id x25mr13243262edv.249.1638403442821; Wed, 01 Dec 2021 16:04:02 -0800 (PST) X-Google-Smtp-Source: ABdhPJwaREVZ/SRhsyoyTktsxFg17mRvv7Dgk0J4972XycW57VidbMbXnUBgWi8xHahFE5RWUaaZjA== X-Received: by 2002:aa7:cd99:: with SMTP id x25mr13243161edv.249.1638403442070; Wed, 01 Dec 2021 16:04:02 -0800 (PST) Received: from alrua-x1.borgediget.toke.dk ([2a0c:4d80:42:443::2]) by smtp.gmail.com with ESMTPSA id w5sm802906edc.58.2021.12.01.16.04.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 01 Dec 2021 16:04:01 -0800 (PST) Received: by alrua-x1.borgediget.toke.dk (Postfix, from userid 1000) id B11C31802A2; Thu, 2 Dec 2021 01:04:00 +0100 (CET) From: =?utf-8?q?Toke_H=C3=B8iland-J=C3=B8rgensen?= To: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Martin KaFai Lau , Song Liu , Yonghong Song , John Fastabend , KP Singh , "David S. Miller" , Jakub Kicinski , Jesper Dangaard Brouer Cc: =?utf-8?q?Toke_H=C3=B8iland-J=C3=B8rgensen?= , netdev@vger.kernel.org, bpf@vger.kernel.org Subject: [PATCH bpf-next 5/8] xdp: add xdp_do_redirect_frame() for pre-computed xdp_frames Date: Thu, 2 Dec 2021 01:02:26 +0100 Message-Id: <20211202000232.380824-6-toke@redhat.com> X-Mailer: git-send-email 2.34.0 In-Reply-To: <20211202000232.380824-1-toke@redhat.com> References: <20211202000232.380824-1-toke@redhat.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net Add an xdp_do_redirect_frame() variant which supports pre-computed xdp_frame structures. This will be used in bpf_prog_run() to avoid having to write to the xdp_frame structure when the XDP program doesn't modify the frame boundaries. Signed-off-by: Toke Høiland-Jørgensen --- include/linux/filter.h | 4 ++++ net/core/filter.c | 28 +++++++++++++++++++++------- 2 files changed, 25 insertions(+), 7 deletions(-) diff --git a/include/linux/filter.h b/include/linux/filter.h index b6a216eb217a..845452c83e0f 100644 --- a/include/linux/filter.h +++ b/include/linux/filter.h @@ -1022,6 +1022,10 @@ int xdp_do_generic_redirect(struct net_device *dev, struct sk_buff *skb, int xdp_do_redirect(struct net_device *dev, struct xdp_buff *xdp, struct bpf_prog *prog); +int xdp_do_redirect_frame(struct net_device *dev, + struct xdp_buff *xdp, + struct xdp_frame *xdpf, + struct bpf_prog *prog); void xdp_do_flush(void); /* The xdp_do_flush_map() helper has been renamed to drop the _map suffix, as diff --git a/net/core/filter.c b/net/core/filter.c index 1e86130a913a..d8fe74cc8b66 100644 --- a/net/core/filter.c +++ b/net/core/filter.c @@ -3957,14 +3957,13 @@ u32 xdp_master_redirect(struct xdp_buff *xdp) } EXPORT_SYMBOL_GPL(xdp_master_redirect); -int xdp_do_redirect(struct net_device *dev, struct xdp_buff *xdp, - struct bpf_prog *xdp_prog) +static int __xdp_do_redirect(struct net_device *dev, struct xdp_buff *xdp, + struct xdp_frame *xdpf, struct bpf_prog *xdp_prog) { struct bpf_redirect_info *ri = this_cpu_ptr(&bpf_redirect_info); enum bpf_map_type map_type = ri->map_type; void *fwd = ri->tgt_value; u32 map_id = ri->map_id; - struct xdp_frame *xdpf; struct bpf_map *map; int err; @@ -3976,10 +3975,12 @@ int xdp_do_redirect(struct net_device *dev, struct xdp_buff *xdp, goto out; } - xdpf = xdp_convert_buff_to_frame(xdp); - if (unlikely(!xdpf)) { - err = -EOVERFLOW; - goto err; + if (!xdpf) { + xdpf = xdp_convert_buff_to_frame(xdp); + if (unlikely(!xdpf)) { + err = -EOVERFLOW; + goto err; + } } switch (map_type) { @@ -4024,8 +4025,21 @@ int xdp_do_redirect(struct net_device *dev, struct xdp_buff *xdp, _trace_xdp_redirect_map_err(dev, xdp_prog, fwd, map_type, map_id, ri->tgt_index, err); return err; } + +int xdp_do_redirect(struct net_device *dev, struct xdp_buff *xdp, + struct bpf_prog *xdp_prog) +{ + return __xdp_do_redirect(dev, xdp, NULL, xdp_prog); +} EXPORT_SYMBOL_GPL(xdp_do_redirect); +int xdp_do_redirect_frame(struct net_device *dev, struct xdp_buff *xdp, + struct xdp_frame *xdpf, struct bpf_prog *xdp_prog) +{ + return __xdp_do_redirect(dev, xdp, xdpf, xdp_prog); +} +EXPORT_SYMBOL_GPL(xdp_do_redirect_frame); + static int xdp_do_generic_redirect_map(struct net_device *dev, struct sk_buff *skb, struct xdp_buff *xdp, From patchwork Thu Dec 2 00:02:27 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Toke_H=C3=B8iland-J=C3=B8rgensen?= X-Patchwork-Id: 12651363 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 56E33C433F5 for ; Thu, 2 Dec 2021 00:04:34 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1354039AbhLBAHx (ORCPT ); Wed, 1 Dec 2021 19:07:53 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:25450 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1354095AbhLBAHc (ORCPT ); Wed, 1 Dec 2021 19:07:32 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1638403447; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=zezzgPFkHKYrn5hcvxo/GoPv2/ANQG50DLyMlYQ5oxw=; b=d9hwNcBTj8cnABASgOgi3ftvzGMMp57f6O/BZCQHzVxfD1TYToA5slzPLcvlWBfaTJK34Y hmOTm3pia2+3rCpAjuU/OFtlGxm/PoCFuEbJHXIEsnB/FIpImbfVriAE9BLMUHqrYrHfb/ nm0uptgp1+u9OtbbF5ktXzOCgFDyy3s= Received: from mail-ed1-f69.google.com (mail-ed1-f69.google.com [209.85.208.69]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-604-nMtx_oKTOS2hLwsYrSLATw-1; Wed, 01 Dec 2021 19:04:06 -0500 X-MC-Unique: nMtx_oKTOS2hLwsYrSLATw-1 Received: by mail-ed1-f69.google.com with SMTP id n11-20020aa7c68b000000b003e7d68e9874so21847158edq.8 for ; Wed, 01 Dec 2021 16:04:06 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=zezzgPFkHKYrn5hcvxo/GoPv2/ANQG50DLyMlYQ5oxw=; b=Z09lXCkXigKpj5z+ahgkR6p8uXnw3X/P5bcXa7c+P++cI9NThvcxO5UJHasdaa4l32 NwPQjJIFYQ2sUjsYtBVC90RuBOeV4N0QKBgNa8XRg6dYtKjrfeDIcV8sL0o4ZkjPuyeD MTTFR/xAH4G1CLlrXXcT1TSKQEMYZfxptciCbvinY2FLlkb8IkN0lWNYluGpsovj7OOX 7ampC5gozYLz0WUMGtEoDnJuCJhFbNxuWyg70kCSvTleEyWW99p15rKyb3q624H9nwqE d9vTyRd76I2Pwvl2vCFVGlBk1E/OeH7vlarHwOw8FQdX1tjm6wRKm7K8TxlCAlEMbsEk f6ag== X-Gm-Message-State: AOAM532a9RlBMsgwaviIVL9pgpgbf8ygscFiQOSl+WNVEAoxpJnzF65U HmdrVXR9n/tjvp1s+4lFwXG3a9zU7/lL7XotLhkY2S7HPd3QP9xR8jnHjjdXvrhCKiA3sjMFtOQ DzJ5SZaikEL6gofZN X-Received: by 2002:a17:906:18b2:: with SMTP id c18mr11353687ejf.403.1638403444715; Wed, 01 Dec 2021 16:04:04 -0800 (PST) X-Google-Smtp-Source: ABdhPJwaBguj69/qdIYhhUG03dgzmOum2yKTR/sCzidi5spbgIkaoN8xIyBEFlmaF0QAEekJAt/J3A== X-Received: by 2002:a17:906:18b2:: with SMTP id c18mr11353536ejf.403.1638403443397; Wed, 01 Dec 2021 16:04:03 -0800 (PST) Received: from alrua-x1.borgediget.toke.dk ([2a0c:4d80:42:443::2]) by smtp.gmail.com with ESMTPSA id e15sm761517edq.46.2021.12.01.16.04.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 01 Dec 2021 16:04:02 -0800 (PST) Received: by alrua-x1.borgediget.toke.dk (Postfix, from userid 1000) id 700A51802A0; Thu, 2 Dec 2021 01:04:02 +0100 (CET) From: =?utf-8?q?Toke_H=C3=B8iland-J=C3=B8rgensen?= To: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Martin KaFai Lau , Song Liu , Yonghong Song , John Fastabend , KP Singh , "David S. Miller" , Jakub Kicinski , Jesper Dangaard Brouer Cc: =?utf-8?q?Toke_H=C3=B8iland-J=C3=B8rgensen?= , netdev@vger.kernel.org, bpf@vger.kernel.org Subject: [PATCH bpf-next 6/8] bpf: Add XDP_REDIRECT support to XDP for bpf_prog_run() Date: Thu, 2 Dec 2021 01:02:27 +0100 Message-Id: <20211202000232.380824-7-toke@redhat.com> X-Mailer: git-send-email 2.34.0 In-Reply-To: <20211202000232.380824-1-toke@redhat.com> References: <20211202000232.380824-1-toke@redhat.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net This adds support for doing real redirects when an XDP program returns XDP_REDIRECT in bpf_prog_run(). To achieve this, we create a page pool instance while setting up the test run, and feed pages from that into the XDP program. The setup cost of this is amortised over the number of repetitions specified by userspace. To support performance testing use case, we further optimise the setup step so that all pages in the pool are pre-initialised with the packet data, and pre-computed context and xdp_frame objects stored at the start of each page. This makes it possible to entirely avoid touching the page content on each XDP program invocation, and enables sending up to 11.5 Mpps/core on my test box. Because the data pages are recycled by the page pool, and the test runner doesn't re-initialise them for each run, subsequent invocations of the XDP program will see the packet data in the state it was after the last time it ran on that particular page. This means that an XDP program that modifies the packet before redirecting it has to be careful about which assumptions it makes about the packet content, but that is only an issue for the most naively written programs. Previous uses of bpf_prog_run() for XDP returned the modified packet data and return code to userspace, which is a different semantic then this new redirect mode. For this reason, the caller has to set the new BPF_F_TEST_XDP_DO_REDIRECT flag when calling bpf_prog_run() to opt in to the different semantics. Enabling this flag is only allowed if not setting ctx_out and data_out in the test specification, since it means frames will be redirected somewhere else, so they can't be returned. Signed-off-by: Toke Høiland-Jørgensen --- include/uapi/linux/bpf.h | 2 + kernel/bpf/Kconfig | 1 + net/bpf/test_run.c | 197 +++++++++++++++++++++++++++++++-- tools/include/uapi/linux/bpf.h | 2 + 4 files changed, 190 insertions(+), 12 deletions(-) diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h index 211b43afd0fb..4797763ef8a4 100644 --- a/include/uapi/linux/bpf.h +++ b/include/uapi/linux/bpf.h @@ -1225,6 +1225,8 @@ enum { /* If set, run the test on the cpu specified by bpf_attr.test.cpu */ #define BPF_F_TEST_RUN_ON_CPU (1U << 0) +/* If set, support performing redirection of XDP frames */ +#define BPF_F_TEST_XDP_DO_REDIRECT (1U << 1) /* type for BPF_ENABLE_STATS */ enum bpf_stats_type { diff --git a/kernel/bpf/Kconfig b/kernel/bpf/Kconfig index d24d518ddd63..c8c920020d11 100644 --- a/kernel/bpf/Kconfig +++ b/kernel/bpf/Kconfig @@ -30,6 +30,7 @@ config BPF_SYSCALL select TASKS_TRACE_RCU select BINARY_PRINTF select NET_SOCK_MSG if NET + select PAGE_POOL if NET default n help Enable the bpf() system call that allows to manipulate BPF programs diff --git a/net/bpf/test_run.c b/net/bpf/test_run.c index 46dd95755967..77326b6cf8ca 100644 --- a/net/bpf/test_run.c +++ b/net/bpf/test_run.c @@ -14,6 +14,7 @@ #include #include #include +#include #include #include #include @@ -23,19 +24,34 @@ #include struct bpf_test_timer { - enum { NO_PREEMPT, NO_MIGRATE } mode; + enum { NO_PREEMPT, NO_MIGRATE, XDP } mode; u32 i; u64 time_start, time_spent; + struct { + struct xdp_buff *orig_ctx; + struct xdp_rxq_info rxq; + struct page_pool *pp; + u16 frame_cnt; + } xdp; }; static void bpf_test_timer_enter(struct bpf_test_timer *t) __acquires(rcu) { rcu_read_lock(); - if (t->mode == NO_PREEMPT) + switch (t->mode) { + case NO_PREEMPT: preempt_disable(); - else + break; + case XDP: + migrate_disable(); + xdp_set_return_frame_no_direct(); + t->xdp.frame_cnt = 0; + break; + case NO_MIGRATE: migrate_disable(); + break; + } t->time_start = ktime_get_ns(); } @@ -45,10 +61,18 @@ static void bpf_test_timer_leave(struct bpf_test_timer *t) { t->time_start = 0; - if (t->mode == NO_PREEMPT) + switch (t->mode) { + case NO_PREEMPT: preempt_enable(); - else + break; + case XDP: + xdp_do_flush(); + xdp_clear_return_frame_no_direct(); + fallthrough; + case NO_MIGRATE: migrate_enable(); + break; + } rcu_read_unlock(); } @@ -87,13 +111,141 @@ static bool bpf_test_timer_continue(struct bpf_test_timer *t, u32 repeat, int *e return false; } +static struct xdp_buff *ctx_from_page(struct page *page) +{ + /* we put an xdp_buff context at the start of the page so we can reuse + * it without having to write to it for every packet + */ + void *data = phys_to_virt(page_to_phys(page)); + + prefetch(data); + return data; +} + +#define TEST_XDP_FRAME_SIZE (PAGE_SIZE - sizeof(struct xdp_buff) \ + - sizeof(struct xdp_frame) \ + - sizeof(struct skb_shared_info)) + +static void bpf_test_run_xdp_init_page(struct page *page, void *arg) +{ + struct xdp_buff *new_ctx, *orig_ctx; + u32 headroom = XDP_PACKET_HEADROOM; + struct bpf_test_timer *t = arg; + struct xdp_frame *frm; + size_t frm_len; + void *data; + + orig_ctx = t->xdp.orig_ctx; + frm_len = orig_ctx->data_end - orig_ctx->data_meta; + + new_ctx = ctx_from_page(page); + frm = (void *)(new_ctx + 1); + data = (void *)(frm + 1); + memcpy(data + headroom, orig_ctx->data_meta, frm_len); + + xdp_init_buff(new_ctx, TEST_XDP_FRAME_SIZE, &t->xdp.rxq); + xdp_prepare_buff(new_ctx, data, headroom, frm_len, true); + + xdp_update_frame_from_buff(new_ctx, frm); + frm->mem = new_ctx->rxq->mem; +} + +static int bpf_test_run_xdp_setup(struct bpf_test_timer *t, struct xdp_buff *orig_ctx) +{ + struct xdp_mem_info mem = {}; + struct page_pool *pp; + int err; + struct page_pool_params pp_params = { + .order = 0, + .flags = 0, + .pool_size = NAPI_POLL_WEIGHT * 2, + .nid = NUMA_NO_NODE, + .max_len = TEST_XDP_FRAME_SIZE, + .init_callback = bpf_test_run_xdp_init_page, + .init_arg = t, + }; + + pp = page_pool_create(&pp_params); + if (IS_ERR(pp)) + return PTR_ERR(pp); + + /* will copy 'mem->id' into pp->xdp_mem_id */ + err = xdp_reg_mem_model(&mem, MEM_TYPE_PAGE_POOL, pp); + if (err) { + page_pool_destroy(pp); + return err; + } + t->xdp.pp = pp; + + /* We create a 'fake' RXQ referencing the original dev, but with an + * xdp_mem_info pointing to our page_pool + */ + xdp_rxq_info_reg(&t->xdp.rxq, orig_ctx->rxq->dev, 0, 0); + t->xdp.rxq.mem.type = MEM_TYPE_PAGE_POOL; + t->xdp.rxq.mem.id = pp->xdp_mem_id; + t->xdp.orig_ctx = orig_ctx; + + return 0; +} + +static void bpf_test_run_xdp_teardown(struct bpf_test_timer *t) +{ + struct xdp_mem_info mem = { + .id = t->xdp.pp->xdp_mem_id, + .type = MEM_TYPE_PAGE_POOL, + }; + xdp_unreg_mem_model(&mem); +} + +static int bpf_test_run_xdp_redirect(struct bpf_test_timer *t, + struct bpf_prog *prog, struct xdp_buff *orig_ctx) +{ + void *data, *data_end, *data_meta; + struct xdp_frame *frm; + struct xdp_buff *ctx; + struct page *page; + int ret, err = 0; + + page = page_pool_dev_alloc_pages(t->xdp.pp); + if (!page) + return -ENOMEM; + + ctx = ctx_from_page(page); + data = ctx->data; + data_meta = ctx->data_meta; + data_end = ctx->data_end; + + ret = bpf_prog_run_xdp(prog, ctx); + if (ret == XDP_REDIRECT) { + frm = (struct xdp_frame *)(ctx + 1); + /* if program changed pkt bounds we need to update the xdp_frame */ + if (unlikely(data != ctx->data || + data_meta != ctx->data_meta || + data_end != ctx->data_end)) + xdp_update_frame_from_buff(ctx, frm); + + err = xdp_do_redirect_frame(ctx->rxq->dev, ctx, frm, prog); + if (err) + ret = err; + } + if (ret != XDP_REDIRECT) + xdp_return_buff(ctx); + + if (++t->xdp.frame_cnt >= NAPI_POLL_WEIGHT) { + xdp_do_flush(); + t->xdp.frame_cnt = 0; + } + + return ret; +} + static int bpf_test_run(struct bpf_prog *prog, void *ctx, u32 repeat, - u32 *retval, u32 *time, bool xdp) + u32 *retval, u32 *time, bool xdp, bool xdp_redirect) { struct bpf_prog_array_item item = {.prog = prog}; struct bpf_run_ctx *old_ctx; struct bpf_cg_run_ctx run_ctx; - struct bpf_test_timer t = { NO_MIGRATE }; + struct bpf_test_timer t = { .mode = (xdp && xdp_redirect) ? XDP : NO_MIGRATE }; enum bpf_cgroup_storage_type stype; int ret; @@ -110,14 +262,26 @@ static int bpf_test_run(struct bpf_prog *prog, void *ctx, u32 repeat, if (!repeat) repeat = 1; + if (t.mode == XDP) { + ret = bpf_test_run_xdp_setup(&t, ctx); + if (ret) + return ret; + } + bpf_test_timer_enter(&t); old_ctx = bpf_set_run_ctx(&run_ctx.run_ctx); do { run_ctx.prog_item = &item; - if (xdp) + if (xdp && xdp_redirect) { + ret = bpf_test_run_xdp_redirect(&t, prog, ctx); + if (unlikely(ret < 0)) + break; + *retval = ret; + } else if (xdp) { *retval = bpf_prog_run_xdp(prog, ctx); - else + } else { *retval = bpf_prog_run(prog, ctx); + } } while (bpf_test_timer_continue(&t, repeat, &ret, time)); bpf_reset_run_ctx(old_ctx); bpf_test_timer_leave(&t); @@ -125,6 +289,9 @@ static int bpf_test_run(struct bpf_prog *prog, void *ctx, u32 repeat, for_each_cgroup_storage_type(stype) bpf_cgroup_storage_free(item.cgroup_storage[stype]); + if (t.mode == XDP) + bpf_test_run_xdp_teardown(&t); + return ret; } @@ -663,7 +830,7 @@ int bpf_prog_test_run_skb(struct bpf_prog *prog, const union bpf_attr *kattr, ret = convert___skb_to_skb(skb, ctx); if (ret) goto out; - ret = bpf_test_run(prog, skb, repeat, &retval, &duration, false); + ret = bpf_test_run(prog, skb, repeat, &retval, &duration, false, false); if (ret) goto out; if (!is_l2) { @@ -757,6 +924,7 @@ static void xdp_convert_buff_to_md(struct xdp_buff *xdp, struct xdp_md *xdp_md) int bpf_prog_test_run_xdp(struct bpf_prog *prog, const union bpf_attr *kattr, union bpf_attr __user *uattr) { + bool do_redirect = (kattr->test.flags & BPF_F_TEST_XDP_DO_REDIRECT); u32 tailroom = SKB_DATA_ALIGN(sizeof(struct skb_shared_info)); u32 headroom = XDP_PACKET_HEADROOM; u32 size = kattr->test.data_size_in; @@ -773,6 +941,9 @@ int bpf_prog_test_run_xdp(struct bpf_prog *prog, const union bpf_attr *kattr, prog->expected_attach_type == BPF_XDP_CPUMAP) return -EINVAL; + if (kattr->test.flags & ~BPF_F_TEST_XDP_DO_REDIRECT) + return -EINVAL; + ctx = bpf_ctx_init(kattr, sizeof(struct xdp_md)); if (IS_ERR(ctx)) return PTR_ERR(ctx); @@ -781,7 +952,8 @@ int bpf_prog_test_run_xdp(struct bpf_prog *prog, const union bpf_attr *kattr, /* There can't be user provided data before the meta data */ if (ctx->data_meta || ctx->data_end != size || ctx->data > ctx->data_end || - unlikely(xdp_metalen_invalid(ctx->data))) + unlikely(xdp_metalen_invalid(ctx->data)) || + (do_redirect && (kattr->test.data_out || kattr->test.ctx_out))) goto free_ctx; /* Meta data is allocated from the headroom */ headroom -= ctx->data; @@ -807,7 +979,8 @@ int bpf_prog_test_run_xdp(struct bpf_prog *prog, const union bpf_attr *kattr, if (repeat > 1) bpf_prog_change_xdp(NULL, prog); - ret = bpf_test_run(prog, &xdp, repeat, &retval, &duration, true); + ret = bpf_test_run(prog, &xdp, repeat, &retval, &duration, + true, do_redirect); /* We convert the xdp_buff back to an xdp_md before checking the return * code so the reference count of any held netdevice will be decremented * even if the test run failed. diff --git a/tools/include/uapi/linux/bpf.h b/tools/include/uapi/linux/bpf.h index 211b43afd0fb..4797763ef8a4 100644 --- a/tools/include/uapi/linux/bpf.h +++ b/tools/include/uapi/linux/bpf.h @@ -1225,6 +1225,8 @@ enum { /* If set, run the test on the cpu specified by bpf_attr.test.cpu */ #define BPF_F_TEST_RUN_ON_CPU (1U << 0) +/* If set, support performing redirection of XDP frames */ +#define BPF_F_TEST_XDP_DO_REDIRECT (1U << 1) /* type for BPF_ENABLE_STATS */ enum bpf_stats_type { From patchwork Thu Dec 2 00:02:28 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Toke_H=C3=B8iland-J=C3=B8rgensen?= X-Patchwork-Id: 12651361 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A7FD9C433EF for ; Thu, 2 Dec 2021 00:04:32 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1353968AbhLBAHt (ORCPT ); Wed, 1 Dec 2021 19:07:49 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]:23173 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1354093AbhLBAHc (ORCPT ); Wed, 1 Dec 2021 19:07:32 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1638403447; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=HeYVFn3C/kHLoloeDuy+DYyyuZygKIHXuCGMV5j00so=; b=dsZhx2YBoSsLcfeXi/O565XNTZxvUKTck3dsV2e4ok1DziSYUldq21viULmssHnMV4HRVL wQYYlIhzWRwkqOJ261RqbeiNNZoYDVUHVQr7cHKkHi75KE8AMGmuV9Q8UswNY2o2DndRcC 65s674n7JjKkC3/D0xaNsEiYCfiTjIU= Received: from mail-ed1-f71.google.com (mail-ed1-f71.google.com [209.85.208.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-149-Ww4ItmQtN_WCjyI1B0_OCQ-1; Wed, 01 Dec 2021 19:04:06 -0500 X-MC-Unique: Ww4ItmQtN_WCjyI1B0_OCQ-1 Received: by mail-ed1-f71.google.com with SMTP id a3-20020a05640213c300b003e7d12bb925so21948971edx.9 for ; Wed, 01 Dec 2021 16:04:06 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=HeYVFn3C/kHLoloeDuy+DYyyuZygKIHXuCGMV5j00so=; b=SyZsbBEDO0VypV48y2dBloXVEFiGxoUa+xARELdTVFt4Bx1svIFYsHghgpLhevVaVT 9bbja85n0aONIz4EccofssBOtfg7PYyRVxwekfToWHSN9sP/5jXXHNfUI5GTUnS3gwsD 9GoGZguBvRC09bes7JVecG0B3b/EMH2qG6rI2mbQdFHha9NQA7niczXTS0jwVAolWPCa 0Gycp2otiA9CJjYSUEcpKP3V5jJgtR/cmKSU1jbLD7XZlkLoEGpzTCHwyT6/UURiFMrZ v0P+u0Qr9QX9VUelG3qjmvMSlqxbehZrYmot+mKj76wQzZ3mtE5DTgK1W5wNH7Us/PR6 tYgw== X-Gm-Message-State: AOAM531O95UlmnZ8tiZlqYtKH3nZLEe+oXef57HoTDBaVm8eFaFaq1/z dd63M6h96aq+c1EA28KFnU2O07UfK0f+HnYWxuVVsZSiZ44LqeyjPTHcUhj13O+k9o827C8NRyt 1nS/9v1OPUkPy9Fab X-Received: by 2002:a05:6402:3551:: with SMTP id f17mr12489395edd.129.1638403445484; Wed, 01 Dec 2021 16:04:05 -0800 (PST) X-Google-Smtp-Source: ABdhPJyeJBPV+unSKdbOtAmXJoB0/EgM0fv5RUBXQLvMLIVG5oNj32Uxu53MfmcDbzk7sEbsJk/8Mw== X-Received: by 2002:a05:6402:3551:: with SMTP id f17mr12489354edd.129.1638403445214; Wed, 01 Dec 2021 16:04:05 -0800 (PST) Received: from alrua-x1.borgediget.toke.dk ([45.145.92.2]) by smtp.gmail.com with ESMTPSA id m25sm739413edj.80.2021.12.01.16.04.04 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 01 Dec 2021 16:04:04 -0800 (PST) Received: by alrua-x1.borgediget.toke.dk (Postfix, from userid 1000) id 126401802A0; Thu, 2 Dec 2021 01:04:04 +0100 (CET) From: =?utf-8?q?Toke_H=C3=B8iland-J=C3=B8rgensen?= To: Alexei Starovoitov , Daniel Borkmann , "David S. Miller" , Jakub Kicinski , Jesper Dangaard Brouer , John Fastabend , Andrii Nakryiko , Martin KaFai Lau , Song Liu , Yonghong Song , KP Singh Cc: =?utf-8?q?Toke_H=C3=B8iland-J=C3=B8rgensen?= , Shuah Khan , netdev@vger.kernel.org, bpf@vger.kernel.org Subject: [PATCH bpf-next 7/8] selftests/bpf: Add selftest for XDP_REDIRECT in bpf_prog_run() Date: Thu, 2 Dec 2021 01:02:28 +0100 Message-Id: <20211202000232.380824-8-toke@redhat.com> X-Mailer: git-send-email 2.34.0 In-Reply-To: <20211202000232.380824-1-toke@redhat.com> References: <20211202000232.380824-1-toke@redhat.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net This adds a selftest for the XDP_REDIRECT facility in bpf_prog_run, that redirects packets into a veth and counts them using an XDP program on the other side of the veth pair. Signed-off-by: Toke Høiland-Jørgensen --- .../bpf/prog_tests/xdp_do_redirect.c | 74 +++++++++++++++++++ .../bpf/progs/test_xdp_do_redirect.c | 34 +++++++++ 2 files changed, 108 insertions(+) create mode 100644 tools/testing/selftests/bpf/prog_tests/xdp_do_redirect.c create mode 100644 tools/testing/selftests/bpf/progs/test_xdp_do_redirect.c diff --git a/tools/testing/selftests/bpf/prog_tests/xdp_do_redirect.c b/tools/testing/selftests/bpf/prog_tests/xdp_do_redirect.c new file mode 100644 index 000000000000..c2effcf076a6 --- /dev/null +++ b/tools/testing/selftests/bpf/prog_tests/xdp_do_redirect.c @@ -0,0 +1,74 @@ +// SPDX-License-Identifier: GPL-2.0 +#include +#include +#include +#include "test_xdp_do_redirect.skel.h" + +#define SYS(fmt, ...) \ + ({ \ + char cmd[1024]; \ + snprintf(cmd, sizeof(cmd), fmt, ##__VA_ARGS__); \ + if (!ASSERT_OK(system(cmd), cmd)) \ + goto fail; \ + }) + +#define NUM_PKTS 10 +void test_xdp_do_redirect(void) +{ + struct test_xdp_do_redirect *skel = NULL; + struct ipv6_packet data = pkt_v6; + struct xdp_md ctx_in = { .data_end = sizeof(data) }; + __u8 dst_mac[ETH_ALEN] = {0x00, 0x11, 0x22, 0x33, 0x44, 0x55}; + __u8 src_mac[ETH_ALEN] = {0x66, 0x77, 0x88, 0x99, 0xaa, 0xbb}; + DECLARE_LIBBPF_OPTS(bpf_test_run_opts, opts, + .data_in = &data, + .data_size_in = sizeof(data), + .ctx_in = &ctx_in, + .ctx_size_in = sizeof(ctx_in), + .flags = BPF_F_TEST_XDP_DO_REDIRECT, + .repeat = NUM_PKTS, + ); + int err, prog_fd, ifindex_src, ifindex_dst; + struct bpf_link *link; + + memcpy(data.eth.h_dest, dst_mac, ETH_ALEN); + memcpy(data.eth.h_source, src_mac, ETH_ALEN); + + skel = test_xdp_do_redirect__open(); + if (!ASSERT_OK_PTR(skel, "skel")) + return; + + SYS("ip link add veth_src type veth peer name veth_dst"); + SYS("ip link set dev veth_src up"); + SYS("ip link set dev veth_dst up"); + + ifindex_src = if_nametoindex("veth_src"); + ifindex_dst = if_nametoindex("veth_dst"); + if (!ASSERT_NEQ(ifindex_src, 0, "ifindex_src") || + !ASSERT_NEQ(ifindex_dst, 0, "ifindex_dst")) + goto fail; + + memcpy(skel->rodata->expect_dst, dst_mac, ETH_ALEN); + skel->rodata->ifindex_out = ifindex_src; + + if (!ASSERT_OK(test_xdp_do_redirect__load(skel), "load")) + goto fail; + + link = bpf_program__attach_xdp(skel->progs.xdp_count_pkts, ifindex_dst); + if (!ASSERT_OK_PTR(link, "prog_attach")) + goto fail; + skel->links.xdp_count_pkts = link; + + prog_fd = bpf_program__fd(skel->progs.xdp_redirect_notouch); + err = bpf_prog_test_run_opts(prog_fd, &opts); + if (!ASSERT_OK(err, "prog_run")) + goto fail; + + /* wait for the packets to be flushed */ + kern_sync_rcu(); + + ASSERT_EQ(skel->bss->pkts_seen, NUM_PKTS, "pkt_count"); +fail: + system("ip link del dev veth_src"); + test_xdp_do_redirect__destroy(skel); +} diff --git a/tools/testing/selftests/bpf/progs/test_xdp_do_redirect.c b/tools/testing/selftests/bpf/progs/test_xdp_do_redirect.c new file mode 100644 index 000000000000..254ebf523f37 --- /dev/null +++ b/tools/testing/selftests/bpf/progs/test_xdp_do_redirect.c @@ -0,0 +1,34 @@ +// SPDX-License-Identifier: GPL-2.0 +#include +#include + +#define ETH_ALEN 6 +const volatile int ifindex_out; +const volatile __u8 expect_dst[ETH_ALEN]; +volatile int pkts_seen = 0; + +SEC("xdp") +int xdp_redirect_notouch(struct xdp_md *xdp) +{ + return bpf_redirect(ifindex_out, 0); +} + +SEC("xdp") +int xdp_count_pkts(struct xdp_md *xdp) +{ + void *data = (void *)(long)xdp->data; + void *data_end = (void *)(long)xdp->data_end; + struct ethhdr *eth = data; + int i; + + if (eth + 1 > data_end) + return XDP_ABORTED; + + for (i = 0; i < ETH_ALEN; i++) + if (expect_dst[i] != eth->h_dest[i]) + return XDP_ABORTED; + pkts_seen++; + return XDP_DROP; +} + +char _license[] SEC("license") = "GPL"; From patchwork Thu Dec 2 00:02:29 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Toke_H=C3=B8iland-J=C3=B8rgensen?= X-Patchwork-Id: 12651365 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 65E51C43217 for ; Thu, 2 Dec 2021 00:04:35 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1354057AbhLBAHz (ORCPT ); Wed, 1 Dec 2021 19:07:55 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]:25052 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1354023AbhLBAHc (ORCPT ); Wed, 1 Dec 2021 19:07:32 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1638403450; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=YPdQ7z4wlVhHQ2g+/gW20ulcqekuyZB8qGRvq9OYQdI=; b=NouWIafPobuLuI7xdAQOMFusb+ZlRtXiQZpm21oNCWRnicWI3woGnIYXvGho7VGqHrX3bK APCv0klzXLL6TeeElEtUB8N1+TnW5CdnpKhYlW3aZaUQvYJT+qNZ/jwUlDXx5PzhRObjl0 CJCWeH0QaAfRBFjF0A7ksJXaFmktJug= Received: from mail-ed1-f71.google.com (mail-ed1-f71.google.com [209.85.208.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-411-xY5V3COTOS248xaV3Hp9GQ-1; Wed, 01 Dec 2021 19:04:09 -0500 X-MC-Unique: xY5V3COTOS248xaV3Hp9GQ-1 Received: by mail-ed1-f71.google.com with SMTP id s12-20020a50ab0c000000b003efdf5a226fso15666570edc.10 for ; Wed, 01 Dec 2021 16:04:09 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=YPdQ7z4wlVhHQ2g+/gW20ulcqekuyZB8qGRvq9OYQdI=; b=Io2gbwYUvifzTVXjq+N9XApobOdk+pQ4hg+/AGLfj0wdY4yEEKdnDM1VFY5oSV06qR wzQKvc/Fm5BHO9BoLdUelFg37zdNE+rn6RRcm8lRwA/6ALvXDHBVibP2A0pD94o7whK8 sh/tBH+A+l8ubvdHsABPriF2RItH+GI6vID9F6uy+2TvGLIougnudy7gtY3z/XE3WvuL xLUbW/sVza813pOFrid1ybvi+TmW3woqkaw9cPIJa2hYtCfydAIbT3xj+poTbxJw9sHg 2Vy/PAObwq0PU4kJXaqmur7AJbtw1XSkS0Sa8sDQXjyGBBTvYT2XZ1b5ndw9QXqmbcNx 2yow== X-Gm-Message-State: AOAM5333+w/Bq2cWK3o/GrLkDGh4g94aU3KiA+Tz/Y/STyg/p+LoGlQT f86caH04B/wBAbU+XQfN8JRl/yAL+zHmVu46B5CfL/Ne2x5HwMvC4ZU+hdgfpYkAkrps+0xQzoV mFidpu2DpKMgppXFZ X-Received: by 2002:aa7:cd99:: with SMTP id x25mr13243982edv.249.1638403448275; Wed, 01 Dec 2021 16:04:08 -0800 (PST) X-Google-Smtp-Source: ABdhPJymhyEwn2KhzKzzUxYmHCQhjUgs3Ur/Ae7xZOxJUNyJpumHf2JT6Wya8r22eftTIeGdXiUZCQ== X-Received: by 2002:aa7:cd99:: with SMTP id x25mr13243923edv.249.1638403447904; Wed, 01 Dec 2021 16:04:07 -0800 (PST) Received: from alrua-x1.borgediget.toke.dk ([45.145.92.2]) by smtp.gmail.com with ESMTPSA id ar2sm664891ejc.20.2021.12.01.16.04.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 01 Dec 2021 16:04:06 -0800 (PST) Received: by alrua-x1.borgediget.toke.dk (Postfix, from userid 1000) id EEF931802A0; Thu, 2 Dec 2021 01:04:05 +0100 (CET) From: =?utf-8?q?Toke_H=C3=B8iland-J=C3=B8rgensen?= To: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Martin KaFai Lau , Song Liu , Yonghong Song , John Fastabend , KP Singh , "David S. Miller" , Jakub Kicinski , Jesper Dangaard Brouer Cc: =?utf-8?q?Toke_H=C3=B8iland-J=C3=B8rgensen?= , netdev@vger.kernel.org, bpf@vger.kernel.org Subject: [PATCH bpf-next 8/8] samples/bpf: Add xdp_trafficgen sample Date: Thu, 2 Dec 2021 01:02:29 +0100 Message-Id: <20211202000232.380824-9-toke@redhat.com> X-Mailer: git-send-email 2.34.0 In-Reply-To: <20211202000232.380824-1-toke@redhat.com> References: <20211202000232.380824-1-toke@redhat.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net This adds an XDP-based traffic generator sample which uses the DO_REDIRECT flag of bpf_prog_run(). It works by building the initial packet in userspace and passing it to the kernel where an XDP program redirects the packet to the target interface. The traffic generator supports two modes of operation: one that just sends copies of the same packet as fast as it can without touching the packet data at all, and one that rewrites the destination port number of each packet, making the generated traffic span a range of port numbers. The dynamic mode is included to demonstrate how the bpf_prog_run() facility enables building a completely programmable packet generator using XDP. Using the dynamic mode has about a 10% overhead compared to the static mode, because the latter completely avoids touching the page data. Signed-off-by: Toke Høiland-Jørgensen --- samples/bpf/.gitignore | 1 + samples/bpf/Makefile | 4 + samples/bpf/xdp_redirect.bpf.c | 34 +++ samples/bpf/xdp_trafficgen_user.c | 444 ++++++++++++++++++++++++++++++ 4 files changed, 483 insertions(+) create mode 100644 samples/bpf/xdp_trafficgen_user.c diff --git a/samples/bpf/.gitignore b/samples/bpf/.gitignore index 0e7bfdbff80a..935672cbdd80 100644 --- a/samples/bpf/.gitignore +++ b/samples/bpf/.gitignore @@ -49,6 +49,7 @@ xdp_redirect_map_multi xdp_router_ipv4 xdp_rxq_info xdp_sample_pkts +xdp_trafficgen xdp_tx_iptunnel xdpsock xdpsock_ctrl_proc diff --git a/samples/bpf/Makefile b/samples/bpf/Makefile index a886dff1ba89..590a09c8e7a5 100644 --- a/samples/bpf/Makefile +++ b/samples/bpf/Makefile @@ -58,6 +58,7 @@ tprogs-y += xdp_redirect_cpu tprogs-y += xdp_redirect_map_multi tprogs-y += xdp_redirect_map tprogs-y += xdp_redirect +tprogs-y += xdp_trafficgen tprogs-y += xdp_monitor # Libbpf dependencies @@ -123,6 +124,7 @@ xdp_redirect_map_multi-objs := xdp_redirect_map_multi_user.o $(XDP_SAMPLE) xdp_redirect_cpu-objs := xdp_redirect_cpu_user.o $(XDP_SAMPLE) xdp_redirect_map-objs := xdp_redirect_map_user.o $(XDP_SAMPLE) xdp_redirect-objs := xdp_redirect_user.o $(XDP_SAMPLE) +xdp_trafficgen-objs := xdp_trafficgen_user.o $(XDP_SAMPLE) xdp_monitor-objs := xdp_monitor_user.o $(XDP_SAMPLE) # Tell kbuild to always build the programs @@ -221,6 +223,7 @@ TPROGLDLIBS_map_perf_test += -lrt TPROGLDLIBS_test_overhead += -lrt TPROGLDLIBS_xdpsock += -pthread -lcap TPROGLDLIBS_xsk_fwd += -pthread +TPROGLDLIBS_xdp_trafficgen += -pthread # Allows pointing LLC/CLANG to a LLVM backend with bpf support, redefine on cmdline: # make M=samples/bpf LLC=~/git/llvm-project/llvm/build/bin/llc CLANG=~/git/llvm-project/llvm/build/bin/clang @@ -336,6 +339,7 @@ $(obj)/xdp_redirect_cpu_user.o: $(obj)/xdp_redirect_cpu.skel.h $(obj)/xdp_redirect_map_multi_user.o: $(obj)/xdp_redirect_map_multi.skel.h $(obj)/xdp_redirect_map_user.o: $(obj)/xdp_redirect_map.skel.h $(obj)/xdp_redirect_user.o: $(obj)/xdp_redirect.skel.h +$(obj)/xdp_trafficgen_user.o: $(obj)/xdp_redirect.skel.h $(obj)/xdp_monitor_user.o: $(obj)/xdp_monitor.skel.h $(obj)/tracex5_kern.o: $(obj)/syscall_nrs.h diff --git a/samples/bpf/xdp_redirect.bpf.c b/samples/bpf/xdp_redirect.bpf.c index 7c02bacfe96b..a09c6f576b79 100644 --- a/samples/bpf/xdp_redirect.bpf.c +++ b/samples/bpf/xdp_redirect.bpf.c @@ -39,6 +39,40 @@ int xdp_redirect_prog(struct xdp_md *ctx) return bpf_redirect(ifindex_out, 0); } +SEC("xdp") +int xdp_redirect_notouch(struct xdp_md *ctx) +{ + return bpf_redirect(ifindex_out, 0); +} + +const volatile __u16 port_start; +const volatile __u16 port_range; +volatile __u16 next_port = 0; + +SEC("xdp") +int xdp_redirect_update_port(struct xdp_md *ctx) +{ + void *data_end = (void *)(long)ctx->data_end; + void *data = (void *)(long)ctx->data; + __u16 cur_port, cksum_diff; + struct udphdr *hdr; + + hdr = data + (sizeof(struct ethhdr) + sizeof(struct ipv6hdr)); + if (hdr + 1 > data_end) + return XDP_ABORTED; + + cur_port = bpf_ntohs(hdr->dest); + cksum_diff = next_port - cur_port; + if (cksum_diff) { + hdr->check = bpf_htons(~(~bpf_ntohs(hdr->check) + cksum_diff)); + hdr->dest = bpf_htons(next_port); + } + if (next_port++ >= port_start + port_range - 1) + next_port = port_start; + + return bpf_redirect(ifindex_out, 0); +} + /* Redirect require an XDP bpf_prog loaded on the TX device */ SEC("xdp") int xdp_redirect_dummy_prog(struct xdp_md *ctx) diff --git a/samples/bpf/xdp_trafficgen_user.c b/samples/bpf/xdp_trafficgen_user.c new file mode 100644 index 000000000000..92e09ee385c7 --- /dev/null +++ b/samples/bpf/xdp_trafficgen_user.c @@ -0,0 +1,444 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* Copyright (c) 2021 Toke Høiland-Jørgensen + */ +static const char *__doc__ = +"XDP trafficgen tool, using bpf_redirect helper\n" +"Usage: xdp_trafficgen [options] _OUT\n"; + +#define _GNU_SOURCE +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include "bpf_util.h" +#include "xdp_sample_user.h" +#include "xdp_redirect.skel.h" + +static int mask = SAMPLE_REDIRECT_ERR_CNT | + SAMPLE_EXCEPTION_CNT | SAMPLE_DEVMAP_XMIT_CNT_MULTI; + +DEFINE_SAMPLE_INIT(xdp_redirect); + +static const struct option long_options[] = { + {"dst-mac", required_argument, NULL, 'm' }, + {"src-mac", required_argument, NULL, 'M' }, + {"dst-ip", required_argument, NULL, 'a' }, + {"src-ip", required_argument, NULL, 'A' }, + {"dst-port", required_argument, NULL, 'p' }, + {"src-port", required_argument, NULL, 'P' }, + {"dynamic-ports", required_argument, NULL, 'd' }, + {"help", no_argument, NULL, 'h' }, + {"stats", no_argument, NULL, 's' }, + {"interval", required_argument, NULL, 'i' }, + {"n-pkts", required_argument, NULL, 'n' }, + {"threads", required_argument, NULL, 't' }, + {"verbose", no_argument, NULL, 'v' }, + {} +}; + +static int sample_res; +static bool sample_exited; + +static void *run_samples(void *arg) +{ + unsigned long *interval = arg; + + sample_res = sample_run(*interval, NULL, NULL); + sample_exited = true; + return NULL; +} + +struct ipv6_packet { + struct ethhdr eth; + struct ipv6hdr iph; + struct udphdr udp; + __u8 payload[64 - sizeof(struct udphdr) + - sizeof(struct ethhdr) - sizeof(struct ipv6hdr)]; +} __packed; +static struct ipv6_packet pkt_v6 = { + .eth.h_proto = __bpf_constant_htons(ETH_P_IPV6), + .iph.version = 6, + .iph.nexthdr = IPPROTO_UDP, + .iph.payload_len = bpf_htons(sizeof(struct ipv6_packet) + - offsetof(struct ipv6_packet, udp)), + .iph.hop_limit = 1, + .iph.saddr.s6_addr16 = {bpf_htons(0xfe80), 0, 0, 0, 0, 0, 0, bpf_htons(1)}, + .iph.daddr.s6_addr16 = {bpf_htons(0xfe80), 0, 0, 0, 0, 0, 0, bpf_htons(2)}, + .udp.source = bpf_htons(1), + .udp.dest = bpf_htons(1), + .udp.len = bpf_htons(sizeof(struct ipv6_packet) + - offsetof(struct ipv6_packet, udp)), +}; + +struct thread_config { + void *pkt; + size_t pkt_size; + __u32 cpu_core_id; + __u32 num_pkts; + int prog_fd; +}; + +struct config { + __be64 src_mac; + __be64 dst_mac; + struct in6_addr src_ip; + struct in6_addr dst_ip; + __be16 src_port; + __be16 dst_port; + int ifindex; + char ifname[IFNAMSIZ]; +}; + +/* Get the mac address of the interface given interface name */ +static __be64 getmac(char *iface) +{ + struct ifreq ifr; + __be64 mac = 0; + int fd, i; + + fd = socket(AF_INET, SOCK_DGRAM, 0); + if (fd < 0) + return -1; + ifr.ifr_addr.sa_family = AF_INET; + memcpy(&ifr.ifr_name, iface, IFNAMSIZ); + if (ioctl(fd, SIOCGIFHWADDR, &ifr) < 0) { + fprintf(stderr, "ioctl failed leaving....\n"); + close(fd); + return -1; + } + for (i = 0; i < 6 ; i++) + *((__u8 *)&mac + i) = (__u8)ifr.ifr_hwaddr.sa_data[i]; + close(fd); + return mac; +} + +static void *run_traffic(void *arg) +{ + const struct thread_config *cfg = arg; + struct xdp_md ctx_in = { + .data_end = cfg->pkt_size, + }; + DECLARE_LIBBPF_OPTS(bpf_test_run_opts, opts, + .data_in = cfg->pkt, + .data_size_in = cfg->pkt_size, + .ctx_in = &ctx_in, + .ctx_size_in = sizeof(ctx_in), + .repeat = cfg->num_pkts ?: 1 << 24, + .flags = BPF_F_TEST_XDP_DO_REDIRECT, + ); + __u64 iterations = 0; + cpu_set_t cpu_cores; + int err; + + CPU_ZERO(&cpu_cores); + CPU_SET(cfg->cpu_core_id, &cpu_cores); + pthread_setaffinity_np(pthread_self(), sizeof(cpu_set_t), &cpu_cores); + do { + err = bpf_prog_test_run_opts(cfg->prog_fd, &opts); + if (err) { + printf("bpf_prog_test_run ret %d errno %d\n", err, errno); + break; + } + iterations += opts.repeat; + } while (!sample_exited && (!cfg->num_pkts || cfg->num_pkts < iterations)); + return NULL; +} + +static __be16 calc_udp_cksum(const struct ipv6_packet *pkt) +{ + __u32 chksum = pkt->iph.nexthdr + bpf_ntohs(pkt->iph.payload_len); + int i; + + for (i = 0; i < 8; i++) { + chksum += bpf_ntohs(pkt->iph.saddr.s6_addr16[i]); + chksum += bpf_ntohs(pkt->iph.daddr.s6_addr16[i]); + } + chksum += bpf_ntohs(pkt->udp.source); + chksum += bpf_ntohs(pkt->udp.dest); + chksum += bpf_ntohs(pkt->udp.len); + + while (chksum >> 16) + chksum = (chksum & 0xFFFF) + (chksum >> 16); + return bpf_htons(~chksum); +} + +static int prepare_pkt(struct config *cfg) +{ + __be64 src_mac = cfg->src_mac; + struct in6_addr nulladdr = {}; + int i; + + if (!src_mac) { + src_mac = getmac(cfg->ifname); + if (src_mac == -1) + return -1; + } + for (i = 0; i < 6 ; i++) { + pkt_v6.eth.h_source[i] = *((__u8 *)&src_mac + i); + if (cfg->dst_mac) + pkt_v6.eth.h_dest[i] = *((__u8 *)&cfg->dst_mac + i); + } + if (memcmp(&cfg->src_ip, &nulladdr, sizeof(nulladdr))) + pkt_v6.iph.saddr = cfg->src_ip; + if (memcmp(&cfg->dst_ip, &nulladdr, sizeof(nulladdr))) + pkt_v6.iph.daddr = cfg->dst_ip; + if (cfg->src_port) + pkt_v6.udp.source = cfg->src_port; + if (cfg->dst_port) + pkt_v6.udp.dest = cfg->dst_port; + pkt_v6.udp.check = calc_udp_cksum(&pkt_v6); + return 0; +} + +int main(int argc, char **argv) +{ + unsigned long interval = 2, threads = 1, dynports = 0; + __u64 num_pkts = 0; + pthread_t sample_thread, *runner_threads = NULL; + struct thread_config *t = NULL, tcfg = { + .pkt = &pkt_v6, + .pkt_size = sizeof(pkt_v6), + }; + int ret = EXIT_FAIL_OPTION; + struct xdp_redirect *skel; + struct config cfg = {}; + bool error = true; + int opt, i, err; + + while ((opt = getopt_long(argc, argv, "a:A:d:hi:m:M:n:p:P:t:vs", + long_options, NULL)) != -1) { + switch (opt) { + case 'a': + if (!inet_pton(AF_INET6, optarg, &cfg.dst_ip)) { + fprintf(stderr, "Invalid IPv6 address: %s\n", optarg); + return -1; + } + break; + case 'A': + if (!inet_pton(AF_INET6, optarg, &cfg.src_ip)) { + fprintf(stderr, "Invalid IPv6 address: %s\n", optarg); + return -1; + } + break; + case 'd': + dynports = strtoul(optarg, NULL, 0); + if (dynports < 2 || dynports >= 65535) { + fprintf(stderr, "Dynamic port range must be >1 and < 65535\n"); + return -1; + } + break; + case 'i': + interval = strtoul(optarg, NULL, 0); + if (interval < 1 || interval == ULONG_MAX) { + fprintf(stderr, "Need non-zero interval\n"); + return -1; + } + break; + case 't': + threads = strtoul(optarg, NULL, 0); + if (threads < 1 || threads == ULONG_MAX) { + fprintf(stderr, "Need at least 1 thread\n"); + return -1; + } + break; + case 'm': + case 'M': + struct ether_addr *a; + + a = ether_aton(optarg); + if (!a) { + fprintf(stderr, "Invalid MAC: %s\n", optarg); + return -1; + } + if (opt == 'm') + memcpy(&cfg.dst_mac, a, sizeof(*a)); + else + memcpy(&cfg.src_mac, a, sizeof(*a)); + break; + case 'n': + num_pkts = strtoull(optarg, NULL, 0); + if (num_pkts >= 1ULL << 32) { + fprintf(stderr, "Can send up to 2^32-1 pkts or infinite (0)\n"); + return -1; + } + tcfg.num_pkts = num_pkts; + break; + case 'p': + case 'P': + unsigned long p; + + p = strtoul(optarg, NULL, 0); + if (!p || p > 0xFFFF) { + fprintf(stderr, "Invalid port: %s\n", optarg); + return -1; + } + if (opt == 'p') + cfg.dst_port = bpf_htons(p); + else + cfg.src_port = bpf_htons(p); + break; + case 'v': + sample_switch_mode(); + break; + case 's': + mask |= SAMPLE_REDIRECT_CNT; + break; + case 'h': + error = false; + default: + sample_usage(argv, long_options, __doc__, mask, error); + return ret; + } + } + + if (argc <= optind) { + sample_usage(argv, long_options, __doc__, mask, true); + return ret; + } + + cfg.ifindex = if_nametoindex(argv[optind]); + if (!cfg.ifindex) + cfg.ifindex = strtoul(argv[optind], NULL, 0); + + if (!cfg.ifindex) { + fprintf(stderr, "Bad interface index or name\n"); + sample_usage(argv, long_options, __doc__, mask, true); + goto end; + } + + if (!if_indextoname(cfg.ifindex, cfg.ifname)) { + fprintf(stderr, "Failed to if_indextoname for %d: %s\n", cfg.ifindex, + strerror(errno)); + goto end; + } + + err = prepare_pkt(&cfg); + if (err) + goto end; + + if (dynports) { + if (!cfg.dst_port) { + fprintf(stderr, "Must specify dst port when using dynamic port range\n"); + goto end; + } + + if (dynports + bpf_ntohs(cfg.dst_port) - 1 > 65535) { + fprintf(stderr, "Dynamic port range must end <= 65535\n"); + goto end; + } + } + + skel = xdp_redirect__open(); + if (!skel) { + fprintf(stderr, "Failed to xdp_redirect__open: %s\n", strerror(errno)); + ret = EXIT_FAIL_BPF; + goto end; + } + + ret = sample_init_pre_load(skel); + if (ret < 0) { + fprintf(stderr, "Failed to sample_init_pre_load: %s\n", strerror(-ret)); + ret = EXIT_FAIL_BPF; + goto end_destroy; + } + + skel->rodata->to_match[0] = cfg.ifindex; + skel->rodata->ifindex_out = cfg.ifindex; + skel->rodata->port_start = bpf_ntohs(cfg.dst_port); + skel->rodata->port_range = dynports; + skel->bss->next_port = bpf_ntohs(cfg.dst_port); + + ret = xdp_redirect__load(skel); + if (ret < 0) { + fprintf(stderr, "Failed to xdp_redirect__load: %s\n", strerror(errno)); + ret = EXIT_FAIL_BPF; + goto end_destroy; + } + + if (dynports) + tcfg.prog_fd = bpf_program__fd(skel->progs.xdp_redirect_update_port); + else + tcfg.prog_fd = bpf_program__fd(skel->progs.xdp_redirect_notouch); + + ret = sample_init(skel, mask); + if (ret < 0) { + fprintf(stderr, "Failed to initialize sample: %s\n", strerror(-ret)); + ret = EXIT_FAIL; + goto end_destroy; + } + + ret = EXIT_FAIL; + + runner_threads = calloc(sizeof(pthread_t), threads); + if (!runner_threads) { + fprintf(stderr, "Couldn't allocate memory\n"); + goto end_destroy; + } + t = calloc(sizeof(struct thread_config), threads); + if (!t) { + fprintf(stderr, "Couldn't allocate memory\n"); + goto end_destroy; + } + + printf("Transmitting on %s (ifindex %d; driver %s)\n", + cfg.ifname, cfg.ifindex, get_driver_name(cfg.ifindex)); + + sample_exited = false; + ret = pthread_create(&sample_thread, NULL, run_samples, &interval); + if (ret < 0) { + fprintf(stderr, "Failed to create sample thread: %s\n", strerror(-ret)); + goto end_destroy; + } + sleep(1); + for (i = 0; i < threads; i++) { + memcpy(&t[i], &tcfg, sizeof(tcfg)); + tcfg.cpu_core_id++; + + ret = pthread_create(&runner_threads[i], NULL, run_traffic, &t[i]); + if (ret < 0) { + fprintf(stderr, "Failed to create traffic thread: %s\n", strerror(-ret)); + ret = EXIT_FAIL; + goto end_cancel; + } + } + pthread_join(sample_thread, NULL); + for (i = 0; i < 0; i++) + pthread_join(runner_threads[i], NULL); + ret = sample_res; + goto end_destroy; + +end_cancel: + pthread_cancel(sample_thread); + for (i = 0; i < 0; i++) + pthread_cancel(runner_threads[i]); +end_destroy: + xdp_redirect__destroy(skel); + free(runner_threads); + free(t); +end: + sample_exit(ret); +}