From patchwork Mon Apr 7 16:53:28 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Toke_H=C3=B8iland-J=C3=B8rgensen?= X-Patchwork-Id: 14041336 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id EC399C36010 for ; Mon, 7 Apr 2025 16:55:28 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B49696B0008; Mon, 7 Apr 2025 12:55:27 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id AABF86B000A; Mon, 7 Apr 2025 12:55:27 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 92F606B000D; Mon, 7 Apr 2025 12:55:27 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 6DBB56B0008 for ; Mon, 7 Apr 2025 12:55:27 -0400 (EDT) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id D5FD01CC24E for ; Mon, 7 Apr 2025 16:55:27 +0000 (UTC) X-FDA: 83307848694.13.60D4652 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf13.hostedemail.com (Postfix) with ESMTP id 5270F2000A for ; Mon, 7 Apr 2025 16:55:25 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=cCvCSyWh; spf=pass (imf13.hostedemail.com: domain of toke@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=toke@redhat.com; dmarc=pass (policy=quarantine) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1744044925; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=/uOjWDhXK5MlXVzffQg4sX2fJ7S1/v80HTUaI6iK6sg=; b=AJ872WUiL23hr6dX2g9qalRJ2cixxiUdrEFaAdep643r5/0bh25zXOL0eO0WasXlSkDyRd f7zrj7FDbjaUwozSVhE4+BZS3Zx1jWvE3UJQDBAfCQz49Xl4PLLYay/w+lxs0xkXBZfsuw +4I2c5j/MOEdvK3c11DqJdx3VDpO1xg= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1744044925; a=rsa-sha256; cv=none; b=HVgsoieRLOncmv/8t6nQVC32kcAWpoJd+UGRjuAojzQn0mS5mqfOT4cS1VaVz9dDj1GAGW v7G0hOTMbOWLWvcylBLWQ9FVtT6kEEzJpFFCCsiyAZn7YoPPeNdc70jKuAwLWdi0wr0ZPy LL0dlRXUjmXnKI9ajLnKDfw/5HGfBS8= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=cCvCSyWh; spf=pass (imf13.hostedemail.com: domain of toke@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=toke@redhat.com; dmarc=pass (policy=quarantine) header.from=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1744044924; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=/uOjWDhXK5MlXVzffQg4sX2fJ7S1/v80HTUaI6iK6sg=; b=cCvCSyWhivi41j/V2Fk0Yd52s+OVutXYMX0FHuce3nS5XeVpZM73mSfg0L0ZHeIKaep98X gtv61wYkrkTwU2CBtDt1a57YYFWSieEdfx5JeZOhVv0Qjd8jJT4wXGv1Bn/5oHsYhNYeME N8DIm5OSTq9su+OV1JU5P6bQNRY1h/4= Received: from mail-lj1-f197.google.com (mail-lj1-f197.google.com [209.85.208.197]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-414-7EXsHogwMji1ef-ndWWo2w-1; Mon, 07 Apr 2025 12:55:23 -0400 X-MC-Unique: 7EXsHogwMji1ef-ndWWo2w-1 X-Mimecast-MFC-AGG-ID: 7EXsHogwMji1ef-ndWWo2w_1744044922 Received: by mail-lj1-f197.google.com with SMTP id 38308e7fff4ca-30c07698ebfso23246691fa.0 for ; Mon, 07 Apr 2025 09:55:23 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1744044922; x=1744649722; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:subject:date:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=/uOjWDhXK5MlXVzffQg4sX2fJ7S1/v80HTUaI6iK6sg=; b=KGuNse4nY/TYj+dhTqqRT9VKioA1BXZ5r/GHgFNuemT2RuP0cDST048MRiNOE02ID4 qHpHTuh5GxApNPQS8Ar9Z0o1i5VkZ+UIbTc8miaX4EhawJ7+wq75K9Nmvv69oUgnfSBP wguN67BfwJ8PYnW0nE822mj2dMkrHLOAgA75e9PkQ8/ZiZuM4GQRDevOQhFGulgIQhkj Jx1Nd/L3cBdf2uLniJBB2eVjbDmGICBJeHFwon/JDgSo9LTmnIyKWoFqZK4vN1pLX8MH 0G7bsjJgVc55DOxld3E2i/bfMZECCnZALOSqjA6ecKOGLig023ygm98/O1C2IZy9zBGR jlKg== X-Forwarded-Encrypted: i=1; AJvYcCWJqj33J1PmkXwNLVUkUqlsURkOMGGQjEKF3vJHs3oGp7+EFFvzcEOyBfV0n7+2BmmdlAzJieiywA==@kvack.org X-Gm-Message-State: AOJu0YydK+AuO6Txq1Y7AIitI9PN1c34Ov+fRM3xNRibN3QmnfXuaaD5 ll2oGM5QXNyhveO7G4ReOWh/cT0g0WXHsHjIPtq4uWp20ifm7zyk5yZGYNyay3Hv24d2f6TYlhe wwzgJixw8D6lxcwgqUy+IezVG7bNwully3X5uN5QOEmncgzwe X-Gm-Gg: ASbGncv2EvUIGu8vJqg6c2bsbslpDcQKKbeKH6JgXOAiIVTfH/x2dnEcyURxRIod6Ir j/hLRUJRVRNTL3hiVYDhK9IXU6EH3cNjlmPvdS5T6dYzJS+3xsHtvsDbCVqKzCU7dBJFGRHd36V xHmxkq8S1zHSTcCKgD3ha4C52GV/4VYCAHGy27kgErZC5AEEx7cy/j9JPK6ndGCwyKzSQuT4+O5 XXo0FmhrDPhFCYpLS0BoLiMyj4ZQQfy7dSHfWHtE1xtF4QJFwU5JaX5ToEAb+zQVhnZoIiFbFko XL4S45/oniWf X-Received: by 2002:a05:651c:988:b0:30c:15b:1268 with SMTP id 38308e7fff4ca-30f0bf2dd00mr33421961fa.15.1744044921590; Mon, 07 Apr 2025 09:55:21 -0700 (PDT) X-Google-Smtp-Source: AGHT+IFYWY2hL2juUcMMBXHQzWMHhnwy4fPLxZdp5hY7xQD602JPS3wKfPW5WzTGnq7lmui6fdhOTw== X-Received: by 2002:a05:651c:988:b0:30c:15b:1268 with SMTP id 38308e7fff4ca-30f0bf2dd00mr33421681fa.15.1744044921179; Mon, 07 Apr 2025 09:55:21 -0700 (PDT) Received: from alrua-x1.borgediget.toke.dk ([2a0c:4d80:42:443::2]) by smtp.gmail.com with ESMTPSA id 38308e7fff4ca-30f0313f430sm16057451fa.36.2025.04.07.09.55.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 07 Apr 2025 09:55:20 -0700 (PDT) Received: by alrua-x1.borgediget.toke.dk (Postfix, from userid 1000) id 4E31619918DA; Mon, 07 Apr 2025 18:55:19 +0200 (CEST) From: =?utf-8?q?Toke_H=C3=B8iland-J=C3=B8rgensen?= Date: Mon, 07 Apr 2025 18:53:28 +0200 Subject: [PATCH net-next v8 1/2] page_pool: Move pp_magic check into helper functions MIME-Version: 1.0 Message-Id: <20250407-page-pool-track-dma-v8-1-da9500d4ba21@redhat.com> References: <20250407-page-pool-track-dma-v8-0-da9500d4ba21@redhat.com> In-Reply-To: <20250407-page-pool-track-dma-v8-0-da9500d4ba21@redhat.com> To: "David S. Miller" , Jakub Kicinski , Jesper Dangaard Brouer , Saeed Mahameed , Leon Romanovsky , Tariq Toukan , Andrew Lunn , Eric Dumazet , Paolo Abeni , Ilias Apalodimas , Simon Horman , Andrew Morton , Mina Almasry , Yonglong Liu , Yunsheng Lin , Pavel Begunkov , Matthew Wilcox Cc: netdev@vger.kernel.org, bpf@vger.kernel.org, linux-rdma@vger.kernel.org, linux-mm@kvack.org, =?utf-8?q?Toke_H=C3=B8iland-J=C3=B8rgensen?= X-Mailer: b4 0.14.2 X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: RM8PxutRsrOf8lsJTKe-xVEgqkrb9B_JpcuwTa2eVIw_1744044922 X-Mimecast-Originator: redhat.com X-Rspam-User: X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 5270F2000A X-Stat-Signature: 7rhuu9xqomuddkew8xjiofreqf8xhged X-HE-Tag: 1744044925-251829 X-HE-Meta: U2FsdGVkX18nYjX9GTzTr5dihunjpPGxHbuKpNkSptmdke/wNer2AhPdrNmpOMCoqVSuvZFHI2/L8fpIMjKrz43Al2vFRZsb93ibfYGDJaKTMNv+D6jTP8boebrA+kUVAtU+jK5brh842qu3NkpkZMZC8XYvfme5+eqllieg7vw1XC74i3PtTyXkrcxPMwWwXTWu4PdViJW1H6RVfC0UfawJhySCp4/je8voGbuPR5B2dUWq9wPptwVE4HgaPuLSLAot/yW77hEUU/RrIiKnBQVrkc6vJzYijGBZCP83HMDihoX3TAf6jS/N8RPvzHRvVI3/alyKmWGg0tfBaLhWhKf0JmSGI8g4di/XZ258kt2+kvqwVRHjfJqlHR7bnHGPYZgaelRi7LQo/4u4k4WZ9Nr2InYHOQSAm9KyjfAp5Xg3t7QKniYTvqBhKQjyv6Y67tG+tD7N3qUCcBxpltvRuw3xt5bT+wphsP6vzZG4o6T0Pslo5T3jvxYE1ZqPmUlyL6cIRpRi13Nb0qOq2sdpNqoZVqW60uID+6R+uCwIVrizlIfzTyte6OQwAEQPJgiOoNbgwqbwEKMNMdOuvnE+ZFntlS1HutFSX8aMhS5ZHO41HDFzczm8uUWMMaOCByH3SHGge4CA0ml8nmi+A+3wPgGSvX8/SXhnOeb8tZ7NTSR1FOb1iVeLGoNd/bpDajnb7gwMLa36ml1Th3k6MPgJwMi3gjijBC0Vo3eLxwMtJCgl42VCVhu8UO16tc6kZab/agWxLlxRriA2+JjGEsGWLHtjA+PtgzT7dtCRutqBJhQeh6zn9+EUqbn4rrJ8kK/tzP1j6gy6Bj1b3ELbIJB4mfzyjcjO0BUigLWkNSqGLcPeH/CwdK1Ty75QklzaLPy63LarFkKF/Pk/XNdO39L1wtupH9U7Coq70FFNmnd373zcgaigY97HCLRUJnVgWfPj9YtMF7RSOO+W32ZKzaC 7LEJLi20 QPEvgLuuHdjRMeRAfcEPU1qjY7pBqMdNB7QKcKwxD6/3F5Yq55SleVV1SjpxnSC1TPj8ptYLjJ0SG2VOlj4GCtB7SeCHJnU8dKBRmoEo/iBBoVLVSgNHD6LDEjwvhcANbTKgwe5USl7t/VCmgQjNMhM2YAzW5xAoWX5ftCSdr89nqYvd3VQn32WWQweUNfbMPTVcl3tmU35x2IdhPKGSP/h54SjSUvmo8wXFm7nf7xmlGgX+EIp3tDwfe9agD+fq9To156i3CYG2+Ami6TxRjS9/tV9THZ2Azxvd2lZSGu5Nje8KylyDJXAjcr0jCVOR/qAi2YakOVcspEmMVsfEO6FtqlevoDZGdOQKigU4eLnBOa928tUyJvIa0fZCwiZmljYeUMr7VkPDAv7gCDFgDQxNeULahFlQCdvE1mNCggxu3CghTFHMmI2tl/9jfCKJC/4bDmmet49Yrcrao4FMa8YIajTpPlJlMCevlR1c9+4RULS20K5mLjkGTAWKaniTU+pj6rzPpx58GCzxPRiF7P/EVlo5LvvhuDeXslHGuBSQF7hdf8uHA8xcF4r8NuubnODjk+b6qLm/ROYuMXiNXzHE3EBjDlU2PefByU9dqd/2FPHN44ANhOTOG5I5+R6L0cwkn30HSNgvr0H7u5D27AsJjMXNQ5KJlQmFdP8NbkjZRnd5PCGuFBgMN2uh+bsGwUTECJdnRWF/Ea0qa31Z8YmPNIsZXvPqb1os7 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Since we are about to stash some more information into the pp_magic field, let's move the magic signature checks into a pair of helper functions so it can be changed in one place. Reviewed-by: Mina Almasry Tested-by: Yonglong Liu Acked-by: Jesper Dangaard Brouer Reviewed-by: Ilias Apalodimas Signed-off-by: Toke Høiland-Jørgensen --- drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c | 4 ++-- include/linux/mm.h | 21 +++++++++++++++++++++ include/net/page_pool/types.h | 1 + mm/page_alloc.c | 8 ++------ net/core/netmem_priv.h | 5 +++++ net/core/skbuff.c | 16 ++-------------- net/core/xdp.c | 4 ++-- 7 files changed, 35 insertions(+), 24 deletions(-) diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c b/drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c index f803e1c93590068d3a7829b0683be4af019266d1..5ce1b463b7a8dd7969e391618658d66f6e836cc1 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c @@ -707,8 +707,8 @@ static void mlx5e_free_xdpsq_desc(struct mlx5e_xdpsq *sq, xdpi = mlx5e_xdpi_fifo_pop(xdpi_fifo); page = xdpi.page.page; - /* No need to check ((page->pp_magic & ~0x3UL) == PP_SIGNATURE) - * as we know this is a page_pool page. + /* No need to check page_pool_page_is_pp() as we + * know this is a page_pool page. */ page_pool_recycle_direct(page->pp, page); } while (++n < num); diff --git a/include/linux/mm.h b/include/linux/mm.h index b7f13f087954bdccfe1e263d39a59bfd1d738ab6..6f9ef1634f75701ae0be146add1ea2c11beb6e48 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -4248,4 +4248,25 @@ int arch_lock_shadow_stack_status(struct task_struct *t, unsigned long status); #define VM_SEALED_SYSMAP VM_NONE #endif +/* Mask used for checking in page_pool_page_is_pp() below. page->pp_magic is + * OR'ed with PP_SIGNATURE after the allocation in order to preserve bit 0 for + * the head page of compound page and bit 1 for pfmemalloc page. + * page_is_pfmemalloc() is checked in __page_pool_put_page() to avoid recycling + * the pfmemalloc page. + */ +#define PP_MAGIC_MASK ~0x3UL + +#ifdef CONFIG_PAGE_POOL +static inline bool page_pool_page_is_pp(struct page *page) +{ + return (page->pp_magic & PP_MAGIC_MASK) == PP_SIGNATURE; +} +#else +static inline bool page_pool_page_is_pp(struct page *page) +{ + return false; +} +#endif + + #endif /* _LINUX_MM_H */ diff --git a/include/net/page_pool/types.h b/include/net/page_pool/types.h index 36eb57d73abc6cfc601e700ca08be20fb8281055..31e6c5c6724b1cffbf5ad2535b3eaee5dec54d9d 100644 --- a/include/net/page_pool/types.h +++ b/include/net/page_pool/types.h @@ -264,6 +264,7 @@ void page_pool_destroy(struct page_pool *pool); void page_pool_use_xdp_mem(struct page_pool *pool, void (*disconnect)(void *), const struct xdp_mem_info *mem); void page_pool_put_netmem_bulk(netmem_ref *data, u32 count); + #else static inline void page_pool_destroy(struct page_pool *pool) { diff --git a/mm/page_alloc.c b/mm/page_alloc.c index fd6b865cb1abfbd3d2ebd67cdaa5f86d92a62e14..a18340b3221835bc81a4db058e5b655575ef665c 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -897,9 +897,7 @@ static inline bool page_expected_state(struct page *page, #ifdef CONFIG_MEMCG page->memcg_data | #endif -#ifdef CONFIG_PAGE_POOL - ((page->pp_magic & ~0x3UL) == PP_SIGNATURE) | -#endif + page_pool_page_is_pp(page) | (page->flags & check_flags))) return false; @@ -926,10 +924,8 @@ static const char *page_bad_reason(struct page *page, unsigned long flags) if (unlikely(page->memcg_data)) bad_reason = "page still charged to cgroup"; #endif -#ifdef CONFIG_PAGE_POOL - if (unlikely((page->pp_magic & ~0x3UL) == PP_SIGNATURE)) + if (unlikely(page_pool_page_is_pp(page))) bad_reason = "page_pool leak"; -#endif return bad_reason; } diff --git a/net/core/netmem_priv.h b/net/core/netmem_priv.h index 7eadb8393e002fd1cc2cef8a313d2ea7df76f301..f33162fd281c23e109273ba09950c5d0a2829bc9 100644 --- a/net/core/netmem_priv.h +++ b/net/core/netmem_priv.h @@ -18,6 +18,11 @@ static inline void netmem_clear_pp_magic(netmem_ref netmem) __netmem_clear_lsb(netmem)->pp_magic = 0; } +static inline bool netmem_is_pp(netmem_ref netmem) +{ + return (netmem_get_pp_magic(netmem) & PP_MAGIC_MASK) == PP_SIGNATURE; +} + static inline void netmem_set_pp(netmem_ref netmem, struct page_pool *pool) { __netmem_clear_lsb(netmem)->pp = pool; diff --git a/net/core/skbuff.c b/net/core/skbuff.c index 6cbf77bc61fce74c934628fd74b3a2cb7809e464..74a2d886a35b518d55b6d3cafcb8442212f9beee 100644 --- a/net/core/skbuff.c +++ b/net/core/skbuff.c @@ -893,11 +893,6 @@ static void skb_clone_fraglist(struct sk_buff *skb) skb_get(list); } -static bool is_pp_netmem(netmem_ref netmem) -{ - return (netmem_get_pp_magic(netmem) & ~0x3UL) == PP_SIGNATURE; -} - int skb_pp_cow_data(struct page_pool *pool, struct sk_buff **pskb, unsigned int headroom) { @@ -995,14 +990,7 @@ bool napi_pp_put_page(netmem_ref netmem) { netmem = netmem_compound_head(netmem); - /* page->pp_magic is OR'ed with PP_SIGNATURE after the allocation - * in order to preserve any existing bits, such as bit 0 for the - * head page of compound page and bit 1 for pfmemalloc page, so - * mask those bits for freeing side when doing below checking, - * and page_is_pfmemalloc() is checked in __page_pool_put_page() - * to avoid recycling the pfmemalloc page. - */ - if (unlikely(!is_pp_netmem(netmem))) + if (unlikely(!netmem_is_pp(netmem))) return false; page_pool_put_full_netmem(netmem_get_pp(netmem), netmem, false); @@ -1042,7 +1030,7 @@ static int skb_pp_frag_ref(struct sk_buff *skb) for (i = 0; i < shinfo->nr_frags; i++) { head_netmem = netmem_compound_head(shinfo->frags[i].netmem); - if (likely(is_pp_netmem(head_netmem))) + if (likely(netmem_is_pp(head_netmem))) page_pool_ref_netmem(head_netmem); else page_ref_inc(netmem_to_page(head_netmem)); diff --git a/net/core/xdp.c b/net/core/xdp.c index f86eedad586a77eb63a96a85aa6d068d3e94f077..0ba73943c6eed873b3d1c681b3b9a802b590f2d9 100644 --- a/net/core/xdp.c +++ b/net/core/xdp.c @@ -437,8 +437,8 @@ void __xdp_return(netmem_ref netmem, enum xdp_mem_type mem_type, netmem = netmem_compound_head(netmem); if (napi_direct && xdp_return_frame_no_direct()) napi_direct = false; - /* No need to check ((page->pp_magic & ~0x3UL) == PP_SIGNATURE) - * as mem->type knows this a page_pool page + /* No need to check netmem_is_pp() as mem->type knows this a + * page_pool page */ page_pool_put_full_netmem(netmem_get_pp(netmem), netmem, napi_direct);