From patchwork Sat Aug 31 00:43:05 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mina Almasry X-Patchwork-Id: 13785792 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C01CDCA101A for ; Sat, 31 Aug 2024 00:43:36 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 24F118D0012; Fri, 30 Aug 2024 20:43:36 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1FFAC8D0002; Fri, 30 Aug 2024 20:43:36 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 053958D0012; Fri, 30 Aug 2024 20:43:36 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id CADF78D0002 for ; Fri, 30 Aug 2024 20:43:35 -0400 (EDT) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id DA4911407A6 for ; Sat, 31 Aug 2024 00:43:34 +0000 (UTC) X-FDA: 82510692348.11.73BD517 Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf02.hostedemail.com (Postfix) with ESMTP id 0494480003 for ; Sat, 31 Aug 2024 00:43:32 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=BBMQgvhm; spf=pass (imf02.hostedemail.com: domain of 3M2fSZgsKCIgmxym43Ayuzms00sxq.o0yxuz69-yyw7mow.03s@flex--almasrymina.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3M2fSZgsKCIgmxym43Ayuzms00sxq.o0yxuz69-yyw7mow.03s@flex--almasrymina.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1725064991; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Bt8VQbg/IyDrLaYuaLsX/ErEComlcrcLp21qg9SiU6M=; b=pS8aRGXDa3V+NQxGMfd/+M96iDPADkrX+yh1ePGyc8mbAMk/q8gmwoRNpZehVaGJ/fllSN VupWFka18AzB6AW06Fx9/6jY6iLugYBsbHm13uouAiWM0QUf/AGC58rgCaoyeKLFCOK78v F+qx2+EcGY94+2hCeO82SvpE7cQyT6Q= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=BBMQgvhm; spf=pass (imf02.hostedemail.com: domain of 3M2fSZgsKCIgmxym43Ayuzms00sxq.o0yxuz69-yyw7mow.03s@flex--almasrymina.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3M2fSZgsKCIgmxym43Ayuzms00sxq.o0yxuz69-yyw7mow.03s@flex--almasrymina.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1725064991; a=rsa-sha256; cv=none; b=gUe1Abj0n6Ee05JMceBeOay+1o6tboeRH57+Q2Jq+QVjMTzie7XM2VyBJ+eyBf5UY2aGBf MWEtC8wxed2ku1ELOLVDusnxXhJCaVBkTlOb49BnAoBbvfqOcfZEcKQWrqElEiWGxSXI5D o7TNtmIPK9DmG0db160o+W5+zakbYYA= Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-6d2c2cf7473so46637617b3.0 for ; Fri, 30 Aug 2024 17:43:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1725065012; x=1725669812; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=Bt8VQbg/IyDrLaYuaLsX/ErEComlcrcLp21qg9SiU6M=; b=BBMQgvhmxAHKonKXQprNOBdc51DHIMOrzNOI2IyqM1Elje1xTOkxQGjkT4oHNzPsr8 cMuOF3yjoOH9JKEvuVxoUtLriXY/t51dDgdIg3YmrpMr+GR7WWpUcrQzoQ9/vlMA8r5r mx6KGnoouDhDBivtCGyy9Vj6KfbHRq6nntCpaQOlTegLAlvfckdItB12j4x290PdbejM e96I4MxYnLa2f1BA7zul2QtktOV6INj37sznhruvSITx5RgZLBxNrGNfSIbqAHbNAiDL FrJ1h8/oFR+rqxDWWSXrP1o6GowUhT22oyYfVdcrBu2NFu3+OFbcNnzTC8YYfLPjrHng pi8Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725065012; x=1725669812; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Bt8VQbg/IyDrLaYuaLsX/ErEComlcrcLp21qg9SiU6M=; b=dFXhk4SvsGpTtTMJKIDUPyr8GFdN9G5IzbwAJscfRgAWka032MElwVuenEd+j16IP7 xiXzmnGGIv4aMSwpKBkpZBorrXSHh/mIr2FwUQWpNOtUwlKC/6jDqKzH0qXUqudA1/m1 NquazTN0nAClNaSbnPs9pcwVCQq5dNMAX3ECRpO1NxonDSBJIbFBL/oc4l8H+DYKZXwr jQbX5EsODYJsjbQq/4O3oaq+WjoJA7Cq/hh4bbOq+IbWa8o57WoF1SwK+fyO3gzu6f6t EXU8CW42eDfq7oNIezJOVri9MKRgwULdTTYxCxvYb+gfsk2YRKGTlcPDVaTSfR+r5y3/ PIiA== X-Forwarded-Encrypted: i=1; AJvYcCWik8zaZ+NhXLBOEmBfZXtwcfTcovARw3eYpRkQOy/P0ZZxfw6FxHoQy/GLdiDE6bo9j1uMBe3kBg==@kvack.org X-Gm-Message-State: AOJu0YyGdwZ9d2z1388toKMKi8Si3+On0mo53HK2mwH3pqm3CMN0AUlM dGYaIj7ZucewqT0I2pCXduKl1pDUW37ccUZQKY8jWHujwImNZ2UBsGWo/EeRO9Gpch3JPGOv5jB HvmHneiHXAHhkwbKCkU4UAQ== X-Google-Smtp-Source: AGHT+IH0v+Lk/7bRc7Yo/TbUA0Vy9P+A0jzGq8M8Es2kI0GBN9TtAqXkY082vvzQu+k3SufM4DFvbN30Ws6wBWCRxA== X-Received: from almasrymina.c.googlers.com ([fda3:e722:ac3:cc00:20:ed76:c0a8:4bc5]) (user=almasrymina job=sendgmr) by 2002:a5b:407:0:b0:e0b:f93:fe8c with SMTP id 3f1490d57ef6-e1a79ce6cd9mr202342276.0.1725065011607; Fri, 30 Aug 2024 17:43:31 -0700 (PDT) Date: Sat, 31 Aug 2024 00:43:05 +0000 In-Reply-To: <20240831004313.3713467-1-almasrymina@google.com> Mime-Version: 1.0 References: <20240831004313.3713467-1-almasrymina@google.com> X-Mailer: git-send-email 2.46.0.469.g59c65b2a67-goog Message-ID: <20240831004313.3713467-6-almasrymina@google.com> Subject: [PATCH net-next v24 05/13] page_pool: devmem support From: Mina Almasry To: netdev@vger.kernel.org, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-alpha@vger.kernel.org, linux-mips@vger.kernel.org, linux-parisc@vger.kernel.org, sparclinux@vger.kernel.org, linux-trace-kernel@vger.kernel.org, linux-arch@vger.kernel.org, bpf@vger.kernel.org, linux-kselftest@vger.kernel.org, linux-media@vger.kernel.org, dri-devel@lists.freedesktop.org Cc: Mina Almasry , Donald Hunter , Jakub Kicinski , "David S. Miller" , Eric Dumazet , Paolo Abeni , Jonathan Corbet , Richard Henderson , Ivan Kokshaysky , Matt Turner , Thomas Bogendoerfer , "James E.J. Bottomley" , Helge Deller , Andreas Larsson , Jesper Dangaard Brouer , Ilias Apalodimas , Steven Rostedt , Masami Hiramatsu , Mathieu Desnoyers , Arnd Bergmann , Steffen Klassert , Herbert Xu , David Ahern , Willem de Bruijn , " =?utf-8?b?QmrDtnJu?= =?utf-8?b?IFTDtnBlbA==?= " , Magnus Karlsson , Maciej Fijalkowski , Jonathan Lemon , Shuah Khan , Alexei Starovoitov , Daniel Borkmann , John Fastabend , Sumit Semwal , " =?utf-8?q?Christian_K=C3=B6nig?= " , Pavel Begunkov , David Wei , Jason Gunthorpe , Yunsheng Lin , Shailend Chand , Harshitha Ramamurthy , Shakeel Butt , Jeroen de Borst , Praveen Kaligineedi , Bagas Sanjaya , Christoph Hellwig , Nikolay Aleksandrov , Taehee Yoo , linux-mm@kvack.org, Matthew Wilcox X-Rspam-User: X-Stat-Signature: fwa6ytrsy34m1ib3csg34r87gw6ohhgf X-Rspamd-Queue-Id: 0494480003 X-Rspamd-Server: rspam11 X-HE-Tag: 1725065012-398160 X-HE-Meta: U2FsdGVkX18vcApogCnOzqF1Hv/aAVIFxkOAJM5yWsdNwSHax5LXDtb9NCA+tRZ2vuSsKAQVpRzeJ6q0c+6F3ZBa8IVD1EJz9j8dmkG2Sj9alvnvmTzOH/UqB9qiD866KooACygpPz7ga3FhVn/SRyPqX526pkergQAQue4LfiM4LZwN66HjMRyqvEpphMlMD58uI34JmAYt9hYFQoqtdwHE6FHqvksLW5gQ1fO8dqvh5gSnikoCAzdA02+od0sfEHxnBHSy3qy3pfX3f7s7Xa2YTubuRRj0DdL2QR3kPnef/6DjiIXNSli819aRWLkrTwrIbdHN3QJGxQ/M9PYCwQ0SNOQftP00oXn20eVZ2UfCOKYaM2MUFEcUojtc65OY6ex8u0lRzsQLk4kunuELj9+8NOUJD1cowT3VeAKKPi/s/UBMNTep0fWW/LbHqWZ00gBf4DTuguE/MtaDlHI4ktAR3+6SX/CtD6VIVHgCvayW0M4U6fnCJA+kFYoLQH0lFsQdNCQkIpu3AVa8UPBukP4RGKR4tpY2yXgw1Umjt9HXv/Du9/nVN7gW657C7mzuZzQZFPrwBLb5r5hg5R+ssOnBs1tme1X7eIcAPvcrTtBYFd34Ixk29Cqf8bFKu0FAXRBLvJH0mc+5LeqNfP1av9RCYUiSsEbpU7FSbMwzSq8veoBC3dAYrFVCXlfORYFJgF+xR9E+KCoEHoZj8CxtHy+HqqNgSH464IL8Mqg65p1G4GQ98ptc8taYNG88zkOUQLdSa1X3uVF/QI7v4k61cwxaGwVjzHQnlAgs2pnXwuh4Rb5/DJbxW8IjlnatbQxMGVtIsj0k5SZl3zrKm1I+xsYdBRr6ZhowKknDnAbCfeqjh2WPXSS/1aOYST7HuJ2s+Ub+3Ll1t/yaCO0ErXGShkZSWIUq91fixEgDgWBV6WvBoCxs+CUdyEzrkqSr2Lnsqv/ZkMV9ZkpeKlZJfQw ZPkJfq4T rBKcl8+8L7C1yADELnbJRGlA63zC7cpLdFNdUYyEd8j1c8iAyAxyUEQzZT1jNZMlDk2AdoSneuC5BR7I5AQvOlmsF1x7qaB9P+7u7L010+5KC6cj2bFwaYN8k9mLk91VaQKH6+nDGmSpJZRKWP9E0Lsw9Sq4pZKGBAmofLzYg24WtboyDODTTene8sQGi3w0ch1qovK4mRGHtpIjBjKnBOpu3CtxuLRsAoQFQ5JA14xmMAnjat5nnq6yr67V9VFi5XdYqBaqeCkPaHrG97HVCrfhiMSw28fiLiAQNotiObz6Bk6OCKAz0trl0yiEOxUuaONdc26T9xiXqHMLqaPW2hIvEXlIRiEZU47ur5oI5e/mjFgQZZhYqNO/yTVDAyHQtnMW0NY0+fMLtjxlW/liXmEGDwwhHWCidc4lqnkOub4PO8/Q1R1gV253hWmV9KjJKqQW7Be7Ha+QvbLrrbwl4XqTSb7/jR15kqy92olSDcspjS8xjM6Ag3LKF9+2dlgcUwXEZnCXX+M/6njqE303HH7ghaqd2Wy3jHPk0JDsh7w/d7Dzl6Tv0ms5ah4X2ALt0vtGyLsfEg5myp+6pQM+k6H3qOGyxvDST7s7TIS0tgM6LUBUXxwG3N+01i9rxkAqPDrpeCV92sBEjt9tHorjWe25v92L964Sp9fhRzFv3b4JlnQWTjaQ0xT/q8+25vJuv4Zff3DgbuLks/wU8WMdfasy++CbosBQ4AZM8QUUyyHrEaZFhGu9JHX2AJqomgXLFIJ/26HnIgprKmvpVO0zNwT32W4NtNq7QwcGQ5YJbpeyMb7N7eyeBgV++shmeW3FNFIRYtGwg6ktMDzbAxico7cI3sxGS/SbDmYnGdkEpxKKMH4M= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Convert netmem to be a union of struct page and struct netmem. Overload the LSB of struct netmem* to indicate that it's a net_iov, otherwise it's a page. Currently these entries in struct page are rented by the page_pool and used exclusively by the net stack: struct { unsigned long pp_magic; struct page_pool *pp; unsigned long _pp_mapping_pad; unsigned long dma_addr; atomic_long_t pp_ref_count; }; Mirror these (and only these) entries into struct net_iov and implement netmem helpers that can access these common fields regardless of whether the underlying type is page or net_iov. Implement checks for net_iov in netmem helpers which delegate to mm APIs, to ensure net_iov are never passed to the mm stack. Signed-off-by: Mina Almasry Reviewed-by: Pavel Begunkov Acked-by: Jakub Kicinski --- v23: - Fix comment on netmem_is_perf_nid (Jakub) v19: - Move page_pool_set_dma_addr(_netmem) to page_pool_priv.h - Don't reset niov dma_addr on allocation/free. Instead, it's set once when the binding happens and it never changes (Jakub) v17: - Rename netmem_to_pfn to netmem_pfn_trace (Jakub) - Move some low level netmem helpers to netmem_priv.h (Jakub). v13: - Move NET_IOV dependent changes to this patch. - Fixed comment (Pavel) - Applied Reviewed-by from Pavel. v9: https://lore.kernel.org/netdev/20240403002053.2376017-8-almasrymina@google.com/ - Remove CONFIG checks in netmem_is_net_iov() (Pavel/David/Jens) v7: - Remove static_branch_unlikely from netmem_to_net_iov(). We're getting better results from the fast path in bench_page_pool_simple tests without the static_branch_unlikely, and the addition of static_branch_unlikely doesn't improve performance of devmem TCP. Additionally only check netmem_to_net_iov() if CONFIG_DMA_SHARED_BUFFER is enabled, otherwise dmabuf net_iovs cannot exist anyway. net-next base: 8 cycle fast path. with static_branch_unlikely: 10 cycle fast path. without static_branch_unlikely: 9 cycle fast path. CONFIG_DMA_SHARED_BUFFER disabled: 8 cycle fast path as baseline. Performance of devmem TCP is at 95% line rate is regardless of static_branch_unlikely or not. v6: - Rebased on top of the merged netmem_ref type. - Rebased on top of the merged skb_pp_frag_ref() changes. v5: - Use netmem instead of page* with LSB set. - Use pp_ref_count for refcounting net_iov. - Removed many of the custom checks for netmem. v1: - Disable fragmentation support for iov properly. - fix napi_pp_put_page() path (Yunsheng). - Use pp_frag_count for devmem refcounting. Cc: linux-mm@kvack.org Cc: Matthew Wilcox --- include/net/netmem.h | 122 +++++++++++++++++++++++++++++-- include/net/page_pool/helpers.h | 39 ++-------- include/trace/events/page_pool.h | 12 +-- net/core/devmem.c | 8 ++ net/core/netmem_priv.h | 31 ++++++++ net/core/page_pool.c | 25 ++++--- net/core/page_pool_priv.h | 26 +++++++ net/core/skbuff.c | 23 +++--- 8 files changed, 217 insertions(+), 69 deletions(-) create mode 100644 net/core/netmem_priv.h diff --git a/include/net/netmem.h b/include/net/netmem.h index 0fbc0999091a..ac6c7945117b 100644 --- a/include/net/netmem.h +++ b/include/net/netmem.h @@ -9,13 +9,51 @@ #define _NET_NETMEM_H #include +#include /* net_iov */ +DECLARE_STATIC_KEY_FALSE(page_pool_mem_providers); + +/* We overload the LSB of the struct page pointer to indicate whether it's + * a page or net_iov. + */ +#define NET_IOV 0x01UL + struct net_iov { + unsigned long __unused_padding; + unsigned long pp_magic; + struct page_pool *pp; struct dmabuf_genpool_chunk_owner *owner; + unsigned long dma_addr; + atomic_long_t pp_ref_count; }; +/* These fields in struct page are used by the page_pool and net stack: + * + * struct { + * unsigned long pp_magic; + * struct page_pool *pp; + * unsigned long _pp_mapping_pad; + * unsigned long dma_addr; + * atomic_long_t pp_ref_count; + * }; + * + * We mirror the page_pool fields here so the page_pool can access these fields + * without worrying whether the underlying fields belong to a page or net_iov. + * + * The non-net stack fields of struct page are private to the mm stack and must + * never be mirrored to net_iov. + */ +#define NET_IOV_ASSERT_OFFSET(pg, iov) \ + static_assert(offsetof(struct page, pg) == \ + offsetof(struct net_iov, iov)) +NET_IOV_ASSERT_OFFSET(pp_magic, pp_magic); +NET_IOV_ASSERT_OFFSET(pp, pp); +NET_IOV_ASSERT_OFFSET(dma_addr, dma_addr); +NET_IOV_ASSERT_OFFSET(pp_ref_count, pp_ref_count); +#undef NET_IOV_ASSERT_OFFSET + static inline struct dmabuf_genpool_chunk_owner * net_iov_owner(const struct net_iov *niov) { @@ -46,20 +84,37 @@ net_iov_binding(const struct net_iov *niov) */ typedef unsigned long __bitwise netmem_ref; +static inline bool netmem_is_net_iov(const netmem_ref netmem) +{ + return (__force unsigned long)netmem & NET_IOV; +} + /* This conversion fails (returns NULL) if the netmem_ref is not struct page * backed. - * - * Currently struct page is the only possible netmem, and this helper never - * fails. */ static inline struct page *netmem_to_page(netmem_ref netmem) { + if (WARN_ON_ONCE(netmem_is_net_iov(netmem))) + return NULL; + return (__force struct page *)netmem; } -/* Converting from page to netmem is always safe, because a page can always be - * a netmem. - */ +static inline struct net_iov *netmem_to_net_iov(netmem_ref netmem) +{ + if (netmem_is_net_iov(netmem)) + return (struct net_iov *)((__force unsigned long)netmem & + ~NET_IOV); + + DEBUG_NET_WARN_ON_ONCE(true); + return NULL; +} + +static inline netmem_ref net_iov_to_netmem(struct net_iov *niov) +{ + return (__force netmem_ref)((unsigned long)niov | NET_IOV); +} + static inline netmem_ref page_to_netmem(struct page *page) { return (__force netmem_ref)page; @@ -67,17 +122,70 @@ static inline netmem_ref page_to_netmem(struct page *page) static inline int netmem_ref_count(netmem_ref netmem) { + /* The non-pp refcount of net_iov is always 1. On net_iov, we only + * support pp refcounting which uses the pp_ref_count field. + */ + if (netmem_is_net_iov(netmem)) + return 1; + return page_ref_count(netmem_to_page(netmem)); } -static inline unsigned long netmem_to_pfn(netmem_ref netmem) +static inline unsigned long netmem_pfn_trace(netmem_ref netmem) { + if (netmem_is_net_iov(netmem)) + return 0; + return page_to_pfn(netmem_to_page(netmem)); } +static inline struct net_iov *__netmem_clear_lsb(netmem_ref netmem) +{ + return (struct net_iov *)((__force unsigned long)netmem & ~NET_IOV); +} + +static inline struct page_pool *netmem_get_pp(netmem_ref netmem) +{ + return __netmem_clear_lsb(netmem)->pp; +} + +static inline atomic_long_t *netmem_get_pp_ref_count_ref(netmem_ref netmem) +{ + return &__netmem_clear_lsb(netmem)->pp_ref_count; +} + +static inline bool netmem_is_pref_nid(netmem_ref netmem, int pref_nid) +{ + /* NUMA node preference only makes sense if we're allocating + * system memory. Memory providers (which give us net_iovs) + * choose for us. + */ + if (netmem_is_net_iov(netmem)) + return true; + + return page_to_nid(netmem_to_page(netmem)) == pref_nid; +} + static inline netmem_ref netmem_compound_head(netmem_ref netmem) { + /* niov are never compounded */ + if (netmem_is_net_iov(netmem)) + return netmem; + return page_to_netmem(compound_head(netmem_to_page(netmem))); } +static inline void *netmem_address(netmem_ref netmem) +{ + if (netmem_is_net_iov(netmem)) + return NULL; + + return page_address(netmem_to_page(netmem)); +} + +static inline unsigned long netmem_get_dma_addr(netmem_ref netmem) +{ + return __netmem_clear_lsb(netmem)->dma_addr; +} + #endif /* _NET_NETMEM_H */ diff --git a/include/net/page_pool/helpers.h b/include/net/page_pool/helpers.h index 2b43a893c619..793e6fd78bc5 100644 --- a/include/net/page_pool/helpers.h +++ b/include/net/page_pool/helpers.h @@ -216,7 +216,7 @@ page_pool_get_dma_dir(const struct page_pool *pool) static inline void page_pool_fragment_netmem(netmem_ref netmem, long nr) { - atomic_long_set(&netmem_to_page(netmem)->pp_ref_count, nr); + atomic_long_set(netmem_get_pp_ref_count_ref(netmem), nr); } /** @@ -244,7 +244,7 @@ static inline void page_pool_fragment_page(struct page *page, long nr) static inline long page_pool_unref_netmem(netmem_ref netmem, long nr) { - struct page *page = netmem_to_page(netmem); + atomic_long_t *pp_ref_count = netmem_get_pp_ref_count_ref(netmem); long ret; /* If nr == pp_ref_count then we have cleared all remaining @@ -261,19 +261,19 @@ static inline long page_pool_unref_netmem(netmem_ref netmem, long nr) * initially, and only overwrite it when the page is partitioned into * more than one piece. */ - if (atomic_long_read(&page->pp_ref_count) == nr) { + if (atomic_long_read(pp_ref_count) == nr) { /* As we have ensured nr is always one for constant case using * the BUILD_BUG_ON(), only need to handle the non-constant case * here for pp_ref_count draining, which is a rare case. */ BUILD_BUG_ON(__builtin_constant_p(nr) && nr != 1); if (!__builtin_constant_p(nr)) - atomic_long_set(&page->pp_ref_count, 1); + atomic_long_set(pp_ref_count, 1); return 0; } - ret = atomic_long_sub_return(nr, &page->pp_ref_count); + ret = atomic_long_sub_return(nr, pp_ref_count); WARN_ON(ret < 0); /* We are the last user here too, reset pp_ref_count back to 1 to @@ -282,7 +282,7 @@ static inline long page_pool_unref_netmem(netmem_ref netmem, long nr) * page_pool_unref_page() currently. */ if (unlikely(!ret)) - atomic_long_set(&page->pp_ref_count, 1); + atomic_long_set(pp_ref_count, 1); return ret; } @@ -401,9 +401,7 @@ static inline void page_pool_free_va(struct page_pool *pool, void *va, static inline dma_addr_t page_pool_get_dma_addr_netmem(netmem_ref netmem) { - struct page *page = netmem_to_page(netmem); - - dma_addr_t ret = page->dma_addr; + dma_addr_t ret = netmem_get_dma_addr(netmem); if (PAGE_POOL_32BIT_ARCH_WITH_64BIT_DMA) ret <<= PAGE_SHIFT; @@ -423,24 +421,6 @@ static inline dma_addr_t page_pool_get_dma_addr(const struct page *page) return page_pool_get_dma_addr_netmem(page_to_netmem((struct page *)page)); } -static inline bool page_pool_set_dma_addr_netmem(netmem_ref netmem, - dma_addr_t addr) -{ - struct page *page = netmem_to_page(netmem); - - if (PAGE_POOL_32BIT_ARCH_WITH_64BIT_DMA) { - page->dma_addr = addr >> PAGE_SHIFT; - - /* We assume page alignment to shave off bottom bits, - * if this "compression" doesn't work we need to drop. - */ - return addr != (dma_addr_t)page->dma_addr << PAGE_SHIFT; - } - - page->dma_addr = addr; - return false; -} - /** * page_pool_dma_sync_for_cpu - sync Rx page for CPU after it's written by HW * @pool: &page_pool the @page belongs to @@ -463,11 +443,6 @@ static inline void page_pool_dma_sync_for_cpu(const struct page_pool *pool, page_pool_get_dma_dir(pool)); } -static inline bool page_pool_set_dma_addr(struct page *page, dma_addr_t addr) -{ - return page_pool_set_dma_addr_netmem(page_to_netmem(page), addr); -} - static inline bool page_pool_put(struct page_pool *pool) { return refcount_dec_and_test(&pool->user_cnt); diff --git a/include/trace/events/page_pool.h b/include/trace/events/page_pool.h index 543e54e432a1..31825ed30032 100644 --- a/include/trace/events/page_pool.h +++ b/include/trace/events/page_pool.h @@ -57,12 +57,12 @@ TRACE_EVENT(page_pool_state_release, __entry->pool = pool; __entry->netmem = (__force unsigned long)netmem; __entry->release = release; - __entry->pfn = netmem_to_pfn(netmem); + __entry->pfn = netmem_pfn_trace(netmem); ), - TP_printk("page_pool=%p netmem=%p pfn=0x%lx release=%u", + TP_printk("page_pool=%p netmem=%p is_net_iov=%lu pfn=0x%lx release=%u", __entry->pool, (void *)__entry->netmem, - __entry->pfn, __entry->release) + __entry->netmem & NET_IOV, __entry->pfn, __entry->release) ); TRACE_EVENT(page_pool_state_hold, @@ -83,12 +83,12 @@ TRACE_EVENT(page_pool_state_hold, __entry->pool = pool; __entry->netmem = (__force unsigned long)netmem; __entry->hold = hold; - __entry->pfn = netmem_to_pfn(netmem); + __entry->pfn = netmem_pfn_trace(netmem); ), - TP_printk("page_pool=%p netmem=%p pfn=0x%lx hold=%u", + TP_printk("page_pool=%p netmem=%p is_net_iov=%lu, pfn=0x%lx hold=%u", __entry->pool, (void *)__entry->netmem, - __entry->pfn, __entry->hold) + __entry->netmem & NET_IOV, __entry->pfn, __entry->hold) ); TRACE_EVENT(page_pool_update_nid, diff --git a/net/core/devmem.c b/net/core/devmem.c index ad222d108d37..727e5ee39f30 100644 --- a/net/core/devmem.c +++ b/net/core/devmem.c @@ -18,6 +18,8 @@ #include #include +#include "page_pool_priv.h" + /* Device memory support */ /* Protected by rtnl_lock() */ @@ -81,6 +83,10 @@ net_devmem_alloc_dmabuf(struct net_devmem_dmabuf_binding *binding) index = offset / PAGE_SIZE; niov = &owner->niovs[index]; + niov->pp_magic = 0; + niov->pp = NULL; + atomic_long_set(&niov->pp_ref_count, 0); + return niov; } @@ -268,6 +274,8 @@ net_devmem_bind_dmabuf(struct net_device *dev, unsigned int dmabuf_fd, for (i = 0; i < owner->num_niovs; i++) { niov = &owner->niovs[i]; niov->owner = owner; + page_pool_set_dma_addr_netmem(net_iov_to_netmem(niov), + net_devmem_get_dma_addr(niov)); } virtual += len; diff --git a/net/core/netmem_priv.h b/net/core/netmem_priv.h new file mode 100644 index 000000000000..7eadb8393e00 --- /dev/null +++ b/net/core/netmem_priv.h @@ -0,0 +1,31 @@ +/* SPDX-License-Identifier: GPL-2.0 */ + +#ifndef __NETMEM_PRIV_H +#define __NETMEM_PRIV_H + +static inline unsigned long netmem_get_pp_magic(netmem_ref netmem) +{ + return __netmem_clear_lsb(netmem)->pp_magic; +} + +static inline void netmem_or_pp_magic(netmem_ref netmem, unsigned long pp_magic) +{ + __netmem_clear_lsb(netmem)->pp_magic |= pp_magic; +} + +static inline void netmem_clear_pp_magic(netmem_ref netmem) +{ + __netmem_clear_lsb(netmem)->pp_magic = 0; +} + +static inline void netmem_set_pp(netmem_ref netmem, struct page_pool *pool) +{ + __netmem_clear_lsb(netmem)->pp = pool; +} + +static inline void netmem_set_dma_addr(netmem_ref netmem, + unsigned long dma_addr) +{ + __netmem_clear_lsb(netmem)->dma_addr = dma_addr; +} +#endif diff --git a/net/core/page_pool.c b/net/core/page_pool.c index 2abe6e919224..52659db2d765 100644 --- a/net/core/page_pool.c +++ b/net/core/page_pool.c @@ -24,8 +24,11 @@ #include +#include "netmem_priv.h" #include "page_pool_priv.h" +DEFINE_STATIC_KEY_FALSE(page_pool_mem_providers); + #define DEFER_TIME (msecs_to_jiffies(1000)) #define DEFER_WARN_INTERVAL (60 * HZ) @@ -358,7 +361,7 @@ static noinline netmem_ref page_pool_refill_alloc_cache(struct page_pool *pool) if (unlikely(!netmem)) break; - if (likely(page_to_nid(netmem_to_page(netmem)) == pref_nid)) { + if (likely(netmem_is_pref_nid(netmem, pref_nid))) { pool->alloc.cache[pool->alloc.count++] = netmem; } else { /* NUMA mismatch; @@ -454,10 +457,8 @@ static bool page_pool_dma_map(struct page_pool *pool, netmem_ref netmem) static void page_pool_set_pp_info(struct page_pool *pool, netmem_ref netmem) { - struct page *page = netmem_to_page(netmem); - - page->pp = pool; - page->pp_magic |= PP_SIGNATURE; + netmem_set_pp(netmem, pool); + netmem_or_pp_magic(netmem, PP_SIGNATURE); /* Ensuring all pages have been split into one fragment initially: * page_pool_set_pp_info() is only called once for every page when it @@ -472,10 +473,8 @@ static void page_pool_set_pp_info(struct page_pool *pool, netmem_ref netmem) static void page_pool_clear_pp_info(netmem_ref netmem) { - struct page *page = netmem_to_page(netmem); - - page->pp_magic = 0; - page->pp = NULL; + netmem_clear_pp_magic(netmem); + netmem_set_pp(netmem, NULL); } static struct page *__page_pool_alloc_page_order(struct page_pool *pool, @@ -692,8 +691,9 @@ static bool page_pool_recycle_in_cache(netmem_ref netmem, static bool __page_pool_page_can_be_recycled(netmem_ref netmem) { - return page_ref_count(netmem_to_page(netmem)) == 1 && - !page_is_pfmemalloc(netmem_to_page(netmem)); + return netmem_is_net_iov(netmem) || + (page_ref_count(netmem_to_page(netmem)) == 1 && + !page_is_pfmemalloc(netmem_to_page(netmem))); } /* If the page refcnt == 1, this will try to recycle the page. @@ -728,6 +728,7 @@ __page_pool_put_page(struct page_pool *pool, netmem_ref netmem, /* Page found as candidate for recycling */ return netmem; } + /* Fallback/non-XDP mode: API user have elevated refcnt. * * Many drivers split up the page into fragments, and some @@ -949,7 +950,7 @@ static void page_pool_empty_ring(struct page_pool *pool) /* Empty recycle ring */ while ((netmem = (__force netmem_ref)ptr_ring_consume_bh(&pool->ring))) { /* Verify the refcnt invariant of cached pages */ - if (!(page_ref_count(netmem_to_page(netmem)) == 1)) + if (!(netmem_ref_count(netmem) == 1)) pr_crit("%s() page_pool refcnt %d violation\n", __func__, netmem_ref_count(netmem)); diff --git a/net/core/page_pool_priv.h b/net/core/page_pool_priv.h index 90665d40f1eb..2142caeddb7c 100644 --- a/net/core/page_pool_priv.h +++ b/net/core/page_pool_priv.h @@ -3,10 +3,36 @@ #ifndef __PAGE_POOL_PRIV_H #define __PAGE_POOL_PRIV_H +#include + +#include "netmem_priv.h" + s32 page_pool_inflight(const struct page_pool *pool, bool strict); int page_pool_list(struct page_pool *pool); void page_pool_detached(struct page_pool *pool); void page_pool_unlist(struct page_pool *pool); +static inline bool page_pool_set_dma_addr_netmem(netmem_ref netmem, + dma_addr_t addr) +{ + if (PAGE_POOL_32BIT_ARCH_WITH_64BIT_DMA) { + netmem_set_dma_addr(netmem, addr >> PAGE_SHIFT); + + /* We assume page alignment to shave off bottom bits, + * if this "compression" doesn't work we need to drop. + */ + return addr != (dma_addr_t)netmem_get_dma_addr(netmem) + << PAGE_SHIFT; + } + + netmem_set_dma_addr(netmem, addr); + return false; +} + +static inline bool page_pool_set_dma_addr(struct page *page, dma_addr_t addr) +{ + return page_pool_set_dma_addr_netmem(page_to_netmem(page), addr); +} + #endif diff --git a/net/core/skbuff.c b/net/core/skbuff.c index a52638363ea5..d9634ab342cc 100644 --- a/net/core/skbuff.c +++ b/net/core/skbuff.c @@ -88,6 +88,7 @@ #include #include "dev.h" +#include "netmem_priv.h" #include "sock_destructor.h" #ifdef CONFIG_SKB_EXTENSIONS @@ -920,9 +921,9 @@ static void skb_clone_fraglist(struct sk_buff *skb) skb_get(list); } -static bool is_pp_page(struct page *page) +static bool is_pp_netmem(netmem_ref netmem) { - return (page->pp_magic & ~0x3UL) == PP_SIGNATURE; + return (netmem_get_pp_magic(netmem) & ~0x3UL) == PP_SIGNATURE; } int skb_pp_cow_data(struct page_pool *pool, struct sk_buff **pskb, @@ -1020,9 +1021,7 @@ EXPORT_SYMBOL(skb_cow_data_for_xdp); #if IS_ENABLED(CONFIG_PAGE_POOL) bool napi_pp_put_page(netmem_ref netmem) { - struct page *page = netmem_to_page(netmem); - - page = compound_head(page); + netmem = netmem_compound_head(netmem); /* page->pp_magic is OR'ed with PP_SIGNATURE after the allocation * in order to preserve any existing bits, such as bit 0 for the @@ -1031,10 +1030,10 @@ bool napi_pp_put_page(netmem_ref netmem) * and page_is_pfmemalloc() is checked in __page_pool_put_page() * to avoid recycling the pfmemalloc page. */ - if (unlikely(!is_pp_page(page))) + if (unlikely(!is_pp_netmem(netmem))) return false; - page_pool_put_full_netmem(page->pp, page_to_netmem(page), false); + page_pool_put_full_netmem(netmem_get_pp(netmem), netmem, false); return true; } @@ -1061,7 +1060,7 @@ static bool skb_pp_recycle(struct sk_buff *skb, void *data) static int skb_pp_frag_ref(struct sk_buff *skb) { struct skb_shared_info *shinfo; - struct page *head_page; + netmem_ref head_netmem; int i; if (!skb->pp_recycle) @@ -1070,11 +1069,11 @@ static int skb_pp_frag_ref(struct sk_buff *skb) shinfo = skb_shinfo(skb); for (i = 0; i < shinfo->nr_frags; i++) { - head_page = compound_head(skb_frag_page(&shinfo->frags[i])); - if (likely(is_pp_page(head_page))) - page_pool_ref_page(head_page); + head_netmem = netmem_compound_head(shinfo->frags[i].netmem); + if (likely(is_pp_netmem(head_netmem))) + page_pool_ref_netmem(head_netmem); else - page_ref_inc(head_page); + page_ref_inc(netmem_to_page(head_netmem)); } return 0; }