From patchwork Fri Mar 28 12:19:08 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Toke_H=C3=B8iland-J=C3=B8rgensen?= X-Patchwork-Id: 14031876 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1352AC36013 for ; Fri, 28 Mar 2025 12:19:29 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 04D0C280138; Fri, 28 Mar 2025 08:19:26 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id DE70D28013B; Fri, 28 Mar 2025 08:19:25 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B9D83280138; Fri, 28 Mar 2025 08:19:25 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 8E685280139 for ; Fri, 28 Mar 2025 08:19:25 -0400 (EDT) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 56D1F12108A for ; Fri, 28 Mar 2025 12:19:26 +0000 (UTC) X-FDA: 83270865132.08.3D72F90 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf25.hostedemail.com (Postfix) with ESMTP id C5FD2A0008 for ; Fri, 28 Mar 2025 12:19:23 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=iN3RsIT5; dmarc=pass (policy=quarantine) header.from=redhat.com; spf=pass (imf25.hostedemail.com: domain of toke@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=toke@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1743164364; a=rsa-sha256; cv=none; b=ZIcyYLCsAKXRumHxgLlGHJ/NYtMHP7RqKlS0V5YVJdpV7ECJPpfAW68LXaFonba4s/C4XD Xc/yvZkpfgV85e7aO6mPKoplSBewYenJ4Pi85IudE11z4CV/N/u64uvG7Pxes9M2lVM517 w24p0u+80+x7YPDcoDMLUrUCc4xNdcE= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=iN3RsIT5; dmarc=pass (policy=quarantine) header.from=redhat.com; spf=pass (imf25.hostedemail.com: domain of toke@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=toke@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1743164364; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=DD6/BKzSTAkniyKROx0SYrtrdsrFKedUxdUETO2d2g0=; b=ttdy02Sqi9dmF1cNjaDSSOS69Tbr34C+2lwDs5f4Q7CjNJE+YmzqGuy2SIYw0VK14rXDtY OUC25z26o0LC62Ugoj8PG9ePWXFMEMwPBCu7qqL4cVbrzyQJbesXJ8N+IxCC4E+HPQ1qRX M+Gy5i+oLcEE9NFCqeb1wTrKJA7wg9Q= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1743164363; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=DD6/BKzSTAkniyKROx0SYrtrdsrFKedUxdUETO2d2g0=; b=iN3RsIT5c0YHwGDxj1+1WXAFfTbkeW7nNyr+IfGG+LljXiDugcpuKcK37y0NY46YNT3Ae5 jId1MJtUQLaTduoY3oetqY/4aRpbiq9n0Xz9rAKvP3pRZLaH3cHOnGmv8ZWLeq1wKQNddW o7HA607vKu4TbhyJUNa2F+O285DiHiU= Received: from mail-ed1-f72.google.com (mail-ed1-f72.google.com [209.85.208.72]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-286-r_OY83h9OQu_BCy68buM8w-1; Fri, 28 Mar 2025 08:19:22 -0400 X-MC-Unique: r_OY83h9OQu_BCy68buM8w-1 X-Mimecast-MFC-AGG-ID: r_OY83h9OQu_BCy68buM8w_1743164361 Received: by mail-ed1-f72.google.com with SMTP id 4fb4d7f45d1cf-5e5d8a28de7so2354777a12.2 for ; Fri, 28 Mar 2025 05:19:21 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1743164361; x=1743769161; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:subject:date:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=DD6/BKzSTAkniyKROx0SYrtrdsrFKedUxdUETO2d2g0=; b=suWNXRZMLA4lY1gGqhU4MIMXw6lW0M6ucNXS4+78XG+cHtjtEePeC6hzSV/fLdhpjn gydb/SjQCZLDJ7YMVUl0Dv/1St2tZki9tUkMSHc+rp9M03IFfFOmJXocp28pXCUNOS3B bKRIQ1+U2DNQVpdZm6i/FTgxs6zz1JN9mhIE7O3SYG+pAHCW2/atspQIehxci8Cyn57X EydaCqzI4lBL+1tuDSC52fpx6pcaI4Z0MbpKJjpFPiyufPm3FZu+5RMXwSHI4RQVHqZU SGBkStM6PzCLptmjwGNbqY3dowImpUJngwN3MMVKUUrb4/WvzxyPdfX01V7O8oqRONON I9rQ== X-Forwarded-Encrypted: i=1; AJvYcCWqeHtiCzG3Hkp7l9Me8vZ2OcG5U+H04SWFY83pWBoZ3690B4z6UeTOlex28UlwztslLKK9X6Tv0Q==@kvack.org X-Gm-Message-State: AOJu0YyiFXawSTb0HTQ6WSL2vVrKAtbJsyMcOz3SUksmQaR5Y4KRzpW0 AoDSIDBXTCSAZPUmdfG9yLmdN86qZLtzf2xmcGYyFFSdsKhxX+0RvwXLEgauoTQIAha8Cy4xUGt 96eqvVxOvngFJv08TmZw/7vE4nf/E3sxJ+qMb1fvM493i8AL4 X-Gm-Gg: ASbGncs04wug6WudCposFnRD2DI9FYdrxomlyvfsWKAQ1GFg+ahv4af1Zj5x2BbptCK V37Eztaubh6S3cJGL+ij8gEy+/Ohhqdbwbg7VFBCDYBUvP5iNrr7VMbQgaKH8gZPSrpSy0/CLAE DOZ40anaQ4tqkpiClk69NOnoB8bKrQTT5mYLxcVi8xU/Ew+/3JXY/kluKLTa13FAAS74viz1W5z 5mXhkKK0nuhOa1xhrTqoEXyhGP6nlLGJ7ACxTrdWAd1qmV8aKUuYpYDyoszf0VgXr5Uik/98ayu Uw6Cqm7c7PCr X-Received: by 2002:a17:907:f213:b0:ac7:150e:8013 with SMTP id a640c23a62f3a-ac7150e8725mr465946066b.15.1743164360607; Fri, 28 Mar 2025 05:19:20 -0700 (PDT) X-Google-Smtp-Source: AGHT+IH2sH2HU5ArIiKzX+0LiYkdqwF13nh1l17P2mIqzHoh6BBNKHd8I1cIquZFx0QCVq5kjgQpAg== X-Received: by 2002:a17:907:f213:b0:ac7:150e:8013 with SMTP id a640c23a62f3a-ac7150e8725mr465941466b.15.1743164360121; Fri, 28 Mar 2025 05:19:20 -0700 (PDT) Received: from alrua-x1.borgediget.toke.dk ([2a0c:4d80:42:443::2]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-ac71922bb65sm152046166b.34.2025.03.28.05.19.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 28 Mar 2025 05:19:18 -0700 (PDT) Received: by alrua-x1.borgediget.toke.dk (Postfix, from userid 1000) id 8A09B18FCDC5; Fri, 28 Mar 2025 13:19:17 +0100 (CET) From: =?utf-8?q?Toke_H=C3=B8iland-J=C3=B8rgensen?= Date: Fri, 28 Mar 2025 13:19:08 +0100 Subject: [PATCH net-next v5 1/2] page_pool: Move pp_magic check into helper functions MIME-Version: 1.0 Message-Id: <20250328-page-pool-track-dma-v5-1-55002af683ad@redhat.com> References: <20250328-page-pool-track-dma-v5-0-55002af683ad@redhat.com> In-Reply-To: <20250328-page-pool-track-dma-v5-0-55002af683ad@redhat.com> To: "David S. Miller" , Jakub Kicinski , Jesper Dangaard Brouer , Saeed Mahameed , Leon Romanovsky , Tariq Toukan , Andrew Lunn , Eric Dumazet , Paolo Abeni , Ilias Apalodimas , Simon Horman , Andrew Morton , Mina Almasry , Yonglong Liu , Yunsheng Lin , Pavel Begunkov , Matthew Wilcox Cc: netdev@vger.kernel.org, bpf@vger.kernel.org, linux-rdma@vger.kernel.org, linux-mm@kvack.org, =?utf-8?q?Toke_H=C3=B8iland-J=C3=B8rgensen?= X-Mailer: b4 0.14.2 X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: 7Yit8QotoFdHi8BQ_ncr-r4gMz5Umgax62atc-fq0p4_1743164361 X-Mimecast-Originator: redhat.com X-Rspam-User: X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: C5FD2A0008 X-Stat-Signature: 7cfznap8iy436zmrwfc9165thoc5rfyg X-HE-Tag: 1743164363-852177 X-HE-Meta: U2FsdGVkX1/qxmX+a+svqskSx22QjzqT5cpeEl8wIVbxvYQUN/ftg5fbPGeYgnLV5qw2EuoqIONLbMoSZzSLSDn0jRC/h7w4WS8RwXOhJkreQrhgTGzzIhDhi+Sxbin/4stme34gt2umnfrt2R0bz4fA2V0ChdH4fG1sRGNBoUxZiK09iB4N9+kBywdeV8cAf/y+JuvTuxITV1MxKA+6A/hTtmnjdpagOjyurmgfeshWE6bsx5yc6KKeFWV/c7q0+nd7TMzVSeKFJH+DtL+NuoBN7/Z5C5xjvmaDWMGZs9TXa1j5J2MRTzbFwtpZ62DHWRHewQ8boZ2rqadLESCCmfkke24pJQbiDQNBNTA0Y+/WoIwAucDTZMP+4L9bXveLD7+MNGaXtlmcqRYebg9XDIfeRb9oQeXMpUo6iRFvOh9qVJ5ofwBLTKY0Ss0+k/GvXhFKebVQDRPrXuoqHbWKhnAvl4icp8veErv1ks/9inHfPoUiKGY7rLkZ/AqSjZvFJkWJ+aArL0Pk/aMsLE6GWNloqse7RO2VkZN5bMFw8HokfIf6oRe8K5fdWUxVmAK3GZgtEePqzM1/l0a/7xUIAM995X7dpIlZIeckdgDba61zaqnhMAtaqIDFOapwTMy4qd0lv5aMKpxQwH9UE9KPcNIoNV7Ax+4WXxS7tJYkCfZ4KU7kQxdhUNNBZngBNE1VenzyS5ZYDxlcm9dFqUS6YiGatQmiBl3k/GB8YNijBk0Fm0vc3WtyciELvT2MtVuQ6hTpClzBAcm3IjnMayKbPfE4qkdCVRu6kTzPKKYNxtx5j+1txVpKZVd9GHi/w7yDJM7yDWFTD/Oy5Z2Uw2cwuLv+f0st8iXu5AzsiSDIWeP7yKI9xCBjUTxv9KPIFK+5JZl20tRhfLv2qaWxOOsbJge9JYfCmAphJqB0ivt5VIXOn7N6A7YKU+6SEhMqj2kTTgr+//a3M/OJmlwCdLe UbQF/5hF sxRE6fzz7+XwEQhBEFGbrVnwVr19rP6MRWWwcNjdP6y+2GiGiNk30PiIAI4RghFHUh+ex9gIjZbemUpwT3mDubdMv6qhmakUaewycCls9B/USqiPfTQ7+89l8+OoFIbrnaX6ygAPKB0PZhyJVQha2pTkYaWb1A6DVR+j9st31UgZnBPKKGZn6aEs/NCEuMmMXqrKKALFjkq88su/AQXy4kWh+aQs5gVZY7NI5eqsDaCnmJB2R/SHNgzi3AIbSFAi4QRyxQ6PBge5hO0EWcXQLuxh95XccWZ93PAKoNWXQzvLrlvzVOt0jSFYjNWlY5kdApQoQ5ZhyVHf9iKXz2xLOlnAVDn3QIvjTSJavIwqTDq3gxTFasSBp5Qpr+aaoL3OSVYd5BizZz0jVrQEGGFALCxJwTTwcdmS9GzQED1eHEZwX1rS15R6jpaH3p3CCFjhQ2aPnQHVpVnP/+7cLC7GacMOwyE8zvDzeABOBh7q5+AN//oVh8Tn8m/GEEYR5qy1a/ANcZqYWx0OHv8RvvKuRUtXE6OvLWM+qrLDJ9h2L6mSyLBsa7IUfh1fNi8SU4vMHHxp+wPFOBwdSs6Th8r3QzX4O/ndobpDNf74bJivW56ukxcWul/9APA8XsxB4I4XugxyM8BEHNS0NUdCCDUyGAB9jB7NoQ2xMCUrtLFZ36U5/HXsOr1m2dw8ZE6QCylqOIRCCSSx3kHB2b5hLmyzsrrrlxbZBzLq+tZxo X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Since we are about to stash some more information into the pp_magic field, let's move the magic signature checks into a pair of helper functions so it can be changed in one place. Reviewed-by: Mina Almasry Tested-by: Yonglong Liu Acked-by: Jesper Dangaard Brouer Reviewed-by: Ilias Apalodimas Signed-off-by: Toke Høiland-Jørgensen --- drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c | 4 ++-- include/net/page_pool/types.h | 18 ++++++++++++++++++ mm/page_alloc.c | 9 +++------ net/core/netmem_priv.h | 5 +++++ net/core/skbuff.c | 16 ++-------------- net/core/xdp.c | 4 ++-- 6 files changed, 32 insertions(+), 24 deletions(-) diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c b/drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c index f803e1c93590068d3a7829b0683be4af019266d1..5ce1b463b7a8dd7969e391618658d66f6e836cc1 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c @@ -707,8 +707,8 @@ static void mlx5e_free_xdpsq_desc(struct mlx5e_xdpsq *sq, xdpi = mlx5e_xdpi_fifo_pop(xdpi_fifo); page = xdpi.page.page; - /* No need to check ((page->pp_magic & ~0x3UL) == PP_SIGNATURE) - * as we know this is a page_pool page. + /* No need to check page_pool_page_is_pp() as we + * know this is a page_pool page. */ page_pool_recycle_direct(page->pp, page); } while (++n < num); diff --git a/include/net/page_pool/types.h b/include/net/page_pool/types.h index 36eb57d73abc6cfc601e700ca08be20fb8281055..df0d3c1608929605224feb26173135ff37951ef8 100644 --- a/include/net/page_pool/types.h +++ b/include/net/page_pool/types.h @@ -54,6 +54,14 @@ struct pp_alloc_cache { netmem_ref cache[PP_ALLOC_CACHE_SIZE]; }; +/* Mask used for checking in page_pool_page_is_pp() below. page->pp_magic is + * OR'ed with PP_SIGNATURE after the allocation in order to preserve bit 0 for + * the head page of compound page and bit 1 for pfmemalloc page. + * page_is_pfmemalloc() is checked in __page_pool_put_page() to avoid recycling + * the pfmemalloc page. + */ +#define PP_MAGIC_MASK ~0x3UL + /** * struct page_pool_params - page pool parameters * @fast: params accessed frequently on hotpath @@ -264,6 +272,11 @@ void page_pool_destroy(struct page_pool *pool); void page_pool_use_xdp_mem(struct page_pool *pool, void (*disconnect)(void *), const struct xdp_mem_info *mem); void page_pool_put_netmem_bulk(netmem_ref *data, u32 count); + +static inline bool page_pool_page_is_pp(struct page *page) +{ + return (page->pp_magic & PP_MAGIC_MASK) == PP_SIGNATURE; +} #else static inline void page_pool_destroy(struct page_pool *pool) { @@ -278,6 +291,11 @@ static inline void page_pool_use_xdp_mem(struct page_pool *pool, static inline void page_pool_put_netmem_bulk(netmem_ref *data, u32 count) { } + +static inline bool page_pool_page_is_pp(struct page *page) +{ + return false; +} #endif void page_pool_put_unrefed_netmem(struct page_pool *pool, netmem_ref netmem, diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 542d25f77be80304b731411ffd29b276ee13be0c..3535ee76afe946cbb042ecbce603bdbedc9233b9 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -55,6 +55,7 @@ #include #include #include +#include #include #include "internal.h" #include "shuffle.h" @@ -872,9 +873,7 @@ static inline bool page_expected_state(struct page *page, #ifdef CONFIG_MEMCG page->memcg_data | #endif -#ifdef CONFIG_PAGE_POOL - ((page->pp_magic & ~0x3UL) == PP_SIGNATURE) | -#endif + page_pool_page_is_pp(page) | (page->flags & check_flags))) return false; @@ -901,10 +900,8 @@ static const char *page_bad_reason(struct page *page, unsigned long flags) if (unlikely(page->memcg_data)) bad_reason = "page still charged to cgroup"; #endif -#ifdef CONFIG_PAGE_POOL - if (unlikely((page->pp_magic & ~0x3UL) == PP_SIGNATURE)) + if (unlikely(page_pool_page_is_pp(page))) bad_reason = "page_pool leak"; -#endif return bad_reason; } diff --git a/net/core/netmem_priv.h b/net/core/netmem_priv.h index 7eadb8393e002fd1cc2cef8a313d2ea7df76f301..f33162fd281c23e109273ba09950c5d0a2829bc9 100644 --- a/net/core/netmem_priv.h +++ b/net/core/netmem_priv.h @@ -18,6 +18,11 @@ static inline void netmem_clear_pp_magic(netmem_ref netmem) __netmem_clear_lsb(netmem)->pp_magic = 0; } +static inline bool netmem_is_pp(netmem_ref netmem) +{ + return (netmem_get_pp_magic(netmem) & PP_MAGIC_MASK) == PP_SIGNATURE; +} + static inline void netmem_set_pp(netmem_ref netmem, struct page_pool *pool) { __netmem_clear_lsb(netmem)->pp = pool; diff --git a/net/core/skbuff.c b/net/core/skbuff.c index 6cbf77bc61fce74c934628fd74b3a2cb7809e464..74a2d886a35b518d55b6d3cafcb8442212f9beee 100644 --- a/net/core/skbuff.c +++ b/net/core/skbuff.c @@ -893,11 +893,6 @@ static void skb_clone_fraglist(struct sk_buff *skb) skb_get(list); } -static bool is_pp_netmem(netmem_ref netmem) -{ - return (netmem_get_pp_magic(netmem) & ~0x3UL) == PP_SIGNATURE; -} - int skb_pp_cow_data(struct page_pool *pool, struct sk_buff **pskb, unsigned int headroom) { @@ -995,14 +990,7 @@ bool napi_pp_put_page(netmem_ref netmem) { netmem = netmem_compound_head(netmem); - /* page->pp_magic is OR'ed with PP_SIGNATURE after the allocation - * in order to preserve any existing bits, such as bit 0 for the - * head page of compound page and bit 1 for pfmemalloc page, so - * mask those bits for freeing side when doing below checking, - * and page_is_pfmemalloc() is checked in __page_pool_put_page() - * to avoid recycling the pfmemalloc page. - */ - if (unlikely(!is_pp_netmem(netmem))) + if (unlikely(!netmem_is_pp(netmem))) return false; page_pool_put_full_netmem(netmem_get_pp(netmem), netmem, false); @@ -1042,7 +1030,7 @@ static int skb_pp_frag_ref(struct sk_buff *skb) for (i = 0; i < shinfo->nr_frags; i++) { head_netmem = netmem_compound_head(shinfo->frags[i].netmem); - if (likely(is_pp_netmem(head_netmem))) + if (likely(netmem_is_pp(head_netmem))) page_pool_ref_netmem(head_netmem); else page_ref_inc(netmem_to_page(head_netmem)); diff --git a/net/core/xdp.c b/net/core/xdp.c index f86eedad586a77eb63a96a85aa6d068d3e94f077..0ba73943c6eed873b3d1c681b3b9a802b590f2d9 100644 --- a/net/core/xdp.c +++ b/net/core/xdp.c @@ -437,8 +437,8 @@ void __xdp_return(netmem_ref netmem, enum xdp_mem_type mem_type, netmem = netmem_compound_head(netmem); if (napi_direct && xdp_return_frame_no_direct()) napi_direct = false; - /* No need to check ((page->pp_magic & ~0x3UL) == PP_SIGNATURE) - * as mem->type knows this a page_pool page + /* No need to check netmem_is_pp() as mem->type knows this a + * page_pool page */ page_pool_put_full_netmem(netmem_get_pp(netmem), netmem, napi_direct);