From patchwork Wed Mar 26 08:18:38 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Toke_H=C3=B8iland-J=C3=B8rgensen?= X-Patchwork-Id: 14029862 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 68B9FC36010 for ; Wed, 26 Mar 2025 08:19:04 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1BE2B280063; Wed, 26 Mar 2025 04:19:02 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 146E9280056; Wed, 26 Mar 2025 04:19:02 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EDB36280063; Wed, 26 Mar 2025 04:19:01 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id CCA35280056 for ; Wed, 26 Mar 2025 04:19:01 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 038751A0E4E for ; Wed, 26 Mar 2025 08:19:02 +0000 (UTC) X-FDA: 83263001766.06.4E4E76F Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf17.hostedemail.com (Postfix) with ESMTP id 85C9940006 for ; Wed, 26 Mar 2025 08:19:00 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=NPWA1fba; spf=pass (imf17.hostedemail.com: domain of toke@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=toke@redhat.com; dmarc=pass (policy=quarantine) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1742977140; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=wHU3gd5R+PcFSPJyd+8lTgr7QvXzHN6SGWRXjaUKaHA=; b=qHolZXiiZTKvwDQobgBysPK91FUWrASS2rHVLBFyR1o/1tHtU3Ro8MgOhlwEDEKLLZUeb6 kyNYuiGKMhYiYeCbXZMowlh5ixq+fnZAmEfykYcN6ydeo5vJdnJCSp8FnnV1y5my4kypNE PsOTDW2jri1pAcBOdcgcyax3QKHt2lg= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=NPWA1fba; spf=pass (imf17.hostedemail.com: domain of toke@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=toke@redhat.com; dmarc=pass (policy=quarantine) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1742977140; a=rsa-sha256; cv=none; b=PCwsgoRyGTf+QSCpy2KdLSAWwlVM9NPBwfEi28nnCrsQhrpg5R7ldEF6LoLUcWuay895zi 512iIivUQMIRVpf4UFXdAAXDGTp6vLlVjkE8UDc6kiBoEtoMK1SOmz5adjc7jrYdB+smdk 4n57aymTNTC2u/fG39g4C5EgFy1vk9o= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1742977139; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=wHU3gd5R+PcFSPJyd+8lTgr7QvXzHN6SGWRXjaUKaHA=; b=NPWA1fbaTHm8PBhNTk7W+JR11NPQkiiqpXv0+uYSvitaictl7a8hlxFPxsgkMNdzjuBrXE huGZvUi88nyXUveYlX+yhR6brriZxjUNzus8AXlh7esCZkx/hYSQu+g0QKVpYYwUndcgaP zbjuPlS/rDjczi+i3thpxZdWk1TTYnE= Received: from mail-ej1-f70.google.com (mail-ej1-f70.google.com [209.85.218.70]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-685-JmS_cLx_N8CwtL8ms6S8SQ-1; Wed, 26 Mar 2025 04:18:58 -0400 X-MC-Unique: JmS_cLx_N8CwtL8ms6S8SQ-1 X-Mimecast-MFC-AGG-ID: JmS_cLx_N8CwtL8ms6S8SQ_1742977137 Received: by mail-ej1-f70.google.com with SMTP id a640c23a62f3a-abec83a498cso522333866b.1 for ; Wed, 26 Mar 2025 01:18:58 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1742977137; x=1743581937; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:subject:date:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=wHU3gd5R+PcFSPJyd+8lTgr7QvXzHN6SGWRXjaUKaHA=; b=dUL6cqrWxka3v8fnI6WStQP5TI8G80enez6H6P/lPip5wib7rFlnxiNTPqJbbjfLDf MpnSyWy8YZI3z/JZknFfrFzDbld6MSrxFi2GvXZOi6/iWQKkYiEKBxqngdP+W15vR7ni fAU5neTrez79n8DFMvuPMTl7786Q4RwG4f0jlKk4J4tzJPbDea2ALDxYUpuzLtBYFBrx ppVxo9ah2JTSTd91JsVbYDBORe2UqO2YiiCZT1qRmR/1A3ktgmtK4F7fMN19f14TfQbk iE5Al2ZsXlkmpWisprGW7JqAS1i390Z70KeewrEGbV12YqgNy5K6UzSk1WYfyYyDtyKE 5P3w== X-Forwarded-Encrypted: i=1; AJvYcCXQotlPQiAuETRcF+kGzxXK4LsR4PfPoUDnzNGWcc+nYmvMmZ2uSGucf/5FBqwPh5oHHzED9iaobw==@kvack.org X-Gm-Message-State: AOJu0YwECUjD9qWwA4pQiFxEk+wojp42kLjPzWUXdgHKV+MymrTfFkd9 6ug8i6KN/w7Ss3PdCtibHlNFvqZkuvBH7KHz6fQbmWY1sCafL8IaM0/9boC9BAxgt/p9c8YvMA+ SV5oiuiLk1WsvY1quCSo5xzlwNxw09lJSu/Fr8nhNf2ll5xUy X-Gm-Gg: ASbGncvL5Vq8bkj9lipgFJRnVS7LLbcbxsdWe+qJaL/kGUKI2CzT56dCrxxdnqujPuR mwT076fJCVIBJQ8uxQbg/qv6zswsmXr0YFUlpd8MwRiunmMDcefNDOiNaPEoNDEedGyX83ABiFe EZ/bJQs1HHhylssqWPRG6FGt02Q4BykXMesV+KZuTkrS6TQ891YtcT82novbtUE7RQEJ47xd0HF nkSHZ523Uqq24bDN74o1hx7rXOk5TSAwwAWjuEoB/D6Awc/NLs17v5qZZ8ks8/1tA0KrypFt/Cr HD5aZUxo8+bLTHNBQcJrzcUYKy/0IW1FrgMNQGM2 X-Received: by 2002:a17:907:6ea6:b0:ac6:b731:d327 with SMTP id a640c23a62f3a-ac6b731d3dcmr641047766b.4.1742977136838; Wed, 26 Mar 2025 01:18:56 -0700 (PDT) X-Google-Smtp-Source: AGHT+IGwezWzI1MgFdP9DAT+/2zNUhExClr8nnWD8c6ObA/s1wOXjNoqUjslh6fgHhj93GwDZJxa9g== X-Received: by 2002:a17:907:6ea6:b0:ac6:b731:d327 with SMTP id a640c23a62f3a-ac6b731d3dcmr641044566b.4.1742977136301; Wed, 26 Mar 2025 01:18:56 -0700 (PDT) Received: from alrua-x1.borgediget.toke.dk ([45.145.92.2]) by smtp.gmail.com with ESMTPSA id 4fb4d7f45d1cf-5ebcd0c74b9sm8933274a12.62.2025.03.26.01.18.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 26 Mar 2025 01:18:55 -0700 (PDT) Received: by alrua-x1.borgediget.toke.dk (Postfix, from userid 1000) id 50AE618FC9CC; Wed, 26 Mar 2025 09:18:54 +0100 (CET) From: =?utf-8?q?Toke_H=C3=B8iland-J=C3=B8rgensen?= Date: Wed, 26 Mar 2025 09:18:38 +0100 Subject: [PATCH net-next v3 1/3] page_pool: Move pp_magic check into helper functions MIME-Version: 1.0 Message-Id: <20250326-page-pool-track-dma-v3-1-8e464016e0ac@redhat.com> References: <20250326-page-pool-track-dma-v3-0-8e464016e0ac@redhat.com> In-Reply-To: <20250326-page-pool-track-dma-v3-0-8e464016e0ac@redhat.com> To: "David S. Miller" , Jakub Kicinski , Jesper Dangaard Brouer , Saeed Mahameed , Leon Romanovsky , Tariq Toukan , Andrew Lunn , Eric Dumazet , Paolo Abeni , Ilias Apalodimas , Simon Horman , Andrew Morton , Mina Almasry , Yonglong Liu , Yunsheng Lin , Pavel Begunkov , Matthew Wilcox Cc: netdev@vger.kernel.org, bpf@vger.kernel.org, linux-rdma@vger.kernel.org, linux-mm@kvack.org, =?utf-8?q?Toke_H=C3=B8iland-J=C3=B8rgensen?= X-Mailer: b4 0.14.2 X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: MgrBWVIevGR5ifzZxoiccQG-GPWUcIcOQeNiCGIwgsE_1742977137 X-Mimecast-Originator: redhat.com X-Rspam-User: X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 85C9940006 X-Stat-Signature: pu56ssxrez7ue3wsob7m4p987w3n9ak9 X-HE-Tag: 1742977140-553315 X-HE-Meta: U2FsdGVkX1+6J50vofMmOtP+Wirsu4Ti9Xc2cn/Ka4/ZQZFzOE8GVdZ3u+HVyutXHEfhsS71rlcwCvJfZ5xwpvNLeFUQQE0B7wouOPB0l6XBGw4YAS36k2CfR4L8Swb6slGAKHiCOvu5gGc5o5YkPHW5JvDCKs3s/ZCX6cEI/Zy3hG4XBIBGCoGb2mNoJI/pNmjslpLBrq823CNNnUtkZbPCqVON6A+A3eKyd0zdyfJGXIyeS9Ot/SqttywmDcy8Xj3w5BzFfiyaA9+0g+v7U/7kL0IMzA/C3tz0huvWRzBhceDOGWAd3Md9pyCljrRPiv9RYRuKSKzhIjEhMOly5WXrx5lxLteR9F6QSwHTDxKovJkgiyJl7FYvtq+DScAwSh5eDbtTM/gPkOJHgapis2k9e7LHJ9rZd7p2y14TS0pm3wwmo0FEWZ7w93kxA4omIewvMbd3fI73B9qtTEzL9N+g9ef4dMt+vQxIZc891rB1+KgAh1t2UzCRECiNNKSKK+2fsM2ur9VrtbE7XgA2rrgHeOvtwTTVuIC6WQ3MVxYqIEfNlXv9ufaLeyy/rZGtb/yHcc/RFfU8XlWacmrhysnWPpS7wGkiaU8tntuHtDmFaXSKx0Ie23Oj2wY93C5Zl0vxlU1W7rtrg1Fi/v9HQxIEXtc/MyYhmS9bvPN/AABIOe046MZLq23U6UavFJ8LKWOEaGl+7nyWvmPGlCdrOC1YagBxXTKkzf8OcPabph6B+QyIgnwBNlJTDJsJ0RjWMN/3zKU8ObR7W8D25UHJXjn1atc+Fuio0xLVIUN8iSmGCCcGtQkiK0z2os0Mpo5YzUVOrfqdyQB3Kj6qHnIk/1vKaJqBNa3AyXvJ6T7s4wi0rYjhDUr4pWF2QjH4kdqUQ4a1B/Cdd1nWiKBAVwNG/bEgRHJ+h4yUOhm6JxOqALV34WjOy+Nrr5am9bVxMUdlYkibfjhcFqC2M+op1Bb EueZPm4N i4/vS8OpTL3yeGuoEB16bTLUUD6zxfQfIQOPd91mf3RhKRUpwV9Xbd4N77LbXz/lam1yuwv379bWnDH5fYphIJuHltXr5tp4SEaagpPcS6v2uQRXfC0OTeHIYiLJqPXyWhtIGQoihlAjKA/v5u0lmHRaNYD+neY+rLBu+WMIEIqxWZ/DKPRyId4PFmV5f1KoCJ+0X/PCkVjhddhYN4DCI01rBrvEsxrFoT+yZSTxVrkCwfDjJuwb074lXl738x7g3YLnPsiVtrkoKKbrUMh6wXyisuqtaQtaNrwCIQV3nl4+RKjzP7ehMavUwDkCunvS7rGZc69UxEIzU3Gln0V88MFgi2AC5+MfIFgKmmoPYT8C+K2YotsI8eQNbZGEJCvVTjbNAKBZC4A1dAfTDuZUkMxUurAwuKX8M3j1YvZjCjwCRlTMv97pWAINPMXTmC6NLo0m3Nk/AYxz+chQFRx7Y1KPGKEAAGnubt8U1H5GqdFZy+GBTW92r15bIbkO7M4LkMdAII/PQ6zTCo2G9Zz7RTSwJGUjOnFGErCfIYS1kVGAt+lcwYcuOe854HJskdTkl6AOdu1aBDKfaf6JZ/1N3By7EG7c0onzDzc70OSvq8wX5dzSnLQNEFtyylwRQGeOwx9tL9sqEQhfXDOo= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Since we are about to stash some more information into the pp_magic field, let's move the magic signature checks into a pair of helper functions so it can be changed in one place. Reviewed-by: Mina Almasry Tested-by: Yonglong Liu Signed-off-by: Toke Høiland-Jørgensen Acked-by: Jesper Dangaard Brouer Reviewed-by: Ilias Apalodimas --- drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c | 4 ++-- include/net/page_pool/types.h | 18 ++++++++++++++++++ mm/page_alloc.c | 9 +++------ net/core/netmem_priv.h | 5 +++++ net/core/skbuff.c | 16 ++-------------- net/core/xdp.c | 4 ++-- 6 files changed, 32 insertions(+), 24 deletions(-) diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c b/drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c index 6f3094a479e1ec61854bb48a6a0c812167487173..70c6f0b2abb921778c98fbd428594ebd7986a302 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c @@ -706,8 +706,8 @@ static void mlx5e_free_xdpsq_desc(struct mlx5e_xdpsq *sq, xdpi = mlx5e_xdpi_fifo_pop(xdpi_fifo); page = xdpi.page.page; - /* No need to check ((page->pp_magic & ~0x3UL) == PP_SIGNATURE) - * as we know this is a page_pool page. + /* No need to check page_pool_page_is_pp() as we + * know this is a page_pool page. */ page_pool_recycle_direct(page->pp, page); } while (++n < num); diff --git a/include/net/page_pool/types.h b/include/net/page_pool/types.h index 36eb57d73abc6cfc601e700ca08be20fb8281055..df0d3c1608929605224feb26173135ff37951ef8 100644 --- a/include/net/page_pool/types.h +++ b/include/net/page_pool/types.h @@ -54,6 +54,14 @@ struct pp_alloc_cache { netmem_ref cache[PP_ALLOC_CACHE_SIZE]; }; +/* Mask used for checking in page_pool_page_is_pp() below. page->pp_magic is + * OR'ed with PP_SIGNATURE after the allocation in order to preserve bit 0 for + * the head page of compound page and bit 1 for pfmemalloc page. + * page_is_pfmemalloc() is checked in __page_pool_put_page() to avoid recycling + * the pfmemalloc page. + */ +#define PP_MAGIC_MASK ~0x3UL + /** * struct page_pool_params - page pool parameters * @fast: params accessed frequently on hotpath @@ -264,6 +272,11 @@ void page_pool_destroy(struct page_pool *pool); void page_pool_use_xdp_mem(struct page_pool *pool, void (*disconnect)(void *), const struct xdp_mem_info *mem); void page_pool_put_netmem_bulk(netmem_ref *data, u32 count); + +static inline bool page_pool_page_is_pp(struct page *page) +{ + return (page->pp_magic & PP_MAGIC_MASK) == PP_SIGNATURE; +} #else static inline void page_pool_destroy(struct page_pool *pool) { @@ -278,6 +291,11 @@ static inline void page_pool_use_xdp_mem(struct page_pool *pool, static inline void page_pool_put_netmem_bulk(netmem_ref *data, u32 count) { } + +static inline bool page_pool_page_is_pp(struct page *page) +{ + return false; +} #endif void page_pool_put_unrefed_netmem(struct page_pool *pool, netmem_ref netmem, diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 542d25f77be80304b731411ffd29b276ee13be0c..3535ee76afe946cbb042ecbce603bdbedc9233b9 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -55,6 +55,7 @@ #include #include #include +#include #include #include "internal.h" #include "shuffle.h" @@ -872,9 +873,7 @@ static inline bool page_expected_state(struct page *page, #ifdef CONFIG_MEMCG page->memcg_data | #endif -#ifdef CONFIG_PAGE_POOL - ((page->pp_magic & ~0x3UL) == PP_SIGNATURE) | -#endif + page_pool_page_is_pp(page) | (page->flags & check_flags))) return false; @@ -901,10 +900,8 @@ static const char *page_bad_reason(struct page *page, unsigned long flags) if (unlikely(page->memcg_data)) bad_reason = "page still charged to cgroup"; #endif -#ifdef CONFIG_PAGE_POOL - if (unlikely((page->pp_magic & ~0x3UL) == PP_SIGNATURE)) + if (unlikely(page_pool_page_is_pp(page))) bad_reason = "page_pool leak"; -#endif return bad_reason; } diff --git a/net/core/netmem_priv.h b/net/core/netmem_priv.h index 7eadb8393e002fd1cc2cef8a313d2ea7df76f301..f33162fd281c23e109273ba09950c5d0a2829bc9 100644 --- a/net/core/netmem_priv.h +++ b/net/core/netmem_priv.h @@ -18,6 +18,11 @@ static inline void netmem_clear_pp_magic(netmem_ref netmem) __netmem_clear_lsb(netmem)->pp_magic = 0; } +static inline bool netmem_is_pp(netmem_ref netmem) +{ + return (netmem_get_pp_magic(netmem) & PP_MAGIC_MASK) == PP_SIGNATURE; +} + static inline void netmem_set_pp(netmem_ref netmem, struct page_pool *pool) { __netmem_clear_lsb(netmem)->pp = pool; diff --git a/net/core/skbuff.c b/net/core/skbuff.c index ab8acb737b93299f503e5c298b87e18edd59d555..a64d777488e403d5fdef83ae42ae9e4924c1a0dc 100644 --- a/net/core/skbuff.c +++ b/net/core/skbuff.c @@ -893,11 +893,6 @@ static void skb_clone_fraglist(struct sk_buff *skb) skb_get(list); } -static bool is_pp_netmem(netmem_ref netmem) -{ - return (netmem_get_pp_magic(netmem) & ~0x3UL) == PP_SIGNATURE; -} - int skb_pp_cow_data(struct page_pool *pool, struct sk_buff **pskb, unsigned int headroom) { @@ -995,14 +990,7 @@ bool napi_pp_put_page(netmem_ref netmem) { netmem = netmem_compound_head(netmem); - /* page->pp_magic is OR'ed with PP_SIGNATURE after the allocation - * in order to preserve any existing bits, such as bit 0 for the - * head page of compound page and bit 1 for pfmemalloc page, so - * mask those bits for freeing side when doing below checking, - * and page_is_pfmemalloc() is checked in __page_pool_put_page() - * to avoid recycling the pfmemalloc page. - */ - if (unlikely(!is_pp_netmem(netmem))) + if (unlikely(!netmem_is_pp(netmem))) return false; page_pool_put_full_netmem(netmem_get_pp(netmem), netmem, false); @@ -1042,7 +1030,7 @@ static int skb_pp_frag_ref(struct sk_buff *skb) for (i = 0; i < shinfo->nr_frags; i++) { head_netmem = netmem_compound_head(shinfo->frags[i].netmem); - if (likely(is_pp_netmem(head_netmem))) + if (likely(netmem_is_pp(head_netmem))) page_pool_ref_netmem(head_netmem); else page_ref_inc(netmem_to_page(head_netmem)); diff --git a/net/core/xdp.c b/net/core/xdp.c index f86eedad586a77eb63a96a85aa6d068d3e94f077..0ba73943c6eed873b3d1c681b3b9a802b590f2d9 100644 --- a/net/core/xdp.c +++ b/net/core/xdp.c @@ -437,8 +437,8 @@ void __xdp_return(netmem_ref netmem, enum xdp_mem_type mem_type, netmem = netmem_compound_head(netmem); if (napi_direct && xdp_return_frame_no_direct()) napi_direct = false; - /* No need to check ((page->pp_magic & ~0x3UL) == PP_SIGNATURE) - * as mem->type knows this a page_pool page + /* No need to check netmem_is_pp() as mem->type knows this a + * page_pool page */ page_pool_put_full_netmem(netmem_get_pp(netmem), netmem, napi_direct); From patchwork Wed Mar 26 08:18:39 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Toke_H=C3=B8iland-J=C3=B8rgensen?= X-Patchwork-Id: 14029863 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8E54AC36008 for ; Wed, 26 Mar 2025 08:19:06 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 66DAC280064; Wed, 26 Mar 2025 04:19:04 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 5F22B280056; Wed, 26 Mar 2025 04:19:04 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 44597280064; Wed, 26 Mar 2025 04:19:04 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 1B7BF280056 for ; Wed, 26 Mar 2025 04:19:04 -0400 (EDT) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 7CB5016196F for ; Wed, 26 Mar 2025 08:19:05 +0000 (UTC) X-FDA: 83263001850.26.0C4D2E6 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf10.hostedemail.com (Postfix) with ESMTP id 3191FC000D for ; Wed, 26 Mar 2025 08:19:03 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Pk2Em1ci; dmarc=pass (policy=quarantine) header.from=redhat.com; spf=pass (imf10.hostedemail.com: domain of toke@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=toke@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1742977143; a=rsa-sha256; cv=none; b=7IpwczxP1EUlb1DSoHzfRW5bpY/OeZc7C+sSSVs6P4sqFCdqDdngFOJ89+aznAH6AVufWB lbkIx/8GAsCKhip4Ysh6j8+r+UzeCd+lW/azDPqzAKv7QPmllSxihvDYTLke8hJ2TpiawW H4rGIbAdBNUp+5GX0NoPD5az9Abr7NE= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Pk2Em1ci; dmarc=pass (policy=quarantine) header.from=redhat.com; spf=pass (imf10.hostedemail.com: domain of toke@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=toke@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1742977143; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=pOTdxZD/uqiBEKSTfpgx9RU52PQylymJns53moyygZc=; b=TFd4ebOqwv6pkSegZrspSe2ybE/SNM9oGeUo1v5BPI2oKUzr+NiQdD2lRjVX4v9XHdnnKz LBKkyXCSoMfsNqCKwpgL0DtwY8LAoo9XlZJQs1rhTtTDBnJmdcQzVTLgGuDeftAzI60b/x aWiuMnuWobXknml+vXdr+YN8M2+PYeQ= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1742977142; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=pOTdxZD/uqiBEKSTfpgx9RU52PQylymJns53moyygZc=; b=Pk2Em1cimMDkpGzjnLgEyQZDBsJf3peZ7Pmz6ojyoQaTVrNtHK/qzN1FOI160jEmK/rOqD j1RV/+vF7Jjbdn2Vf2n2s2UPGCWKtX4FaAmCkg18v83YqasSPCaAZr7148bwKcsSmeDYmw vL1Xvz31nzufmKubMY84vQ91SYArrqo= Received: from mail-ed1-f69.google.com (mail-ed1-f69.google.com [209.85.208.69]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-628-2gTLhc-tO2S4uLR93kP9jg-1; Wed, 26 Mar 2025 04:18:59 -0400 X-MC-Unique: 2gTLhc-tO2S4uLR93kP9jg-1 X-Mimecast-MFC-AGG-ID: 2gTLhc-tO2S4uLR93kP9jg_1742977138 Received: by mail-ed1-f69.google.com with SMTP id 4fb4d7f45d1cf-5e6c14c0a95so4858949a12.0 for ; Wed, 26 Mar 2025 01:18:58 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1742977138; x=1743581938; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:subject:date:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=pOTdxZD/uqiBEKSTfpgx9RU52PQylymJns53moyygZc=; b=P2j3UuBdky27TP3BuOjGP9uU17qa8JFfTUcN53mb9Zmt9AwB9+btAX2+AAGZQ/hRbe SpiXAGNZVDWxVR0SVKM6UsyVFKrcZ3aZIhaDCBsRgrVLUz+8/fuF1po8jTY0BrMIbTny ZlCV+sOtVpE2+wweRFIq4sspw5PTtwwfgHKn8fLTTfytN4RpNjs20wC2Kg3MKH3VD8HK 9P5dYevgBwVum9mN3JA/l0Mp5FBaBvO8AxB64kqTLouarnTUMTUOTtU/Z67ILu+W0Qwa KsiQNRZtbweBhfmpXuR55fpYin5UbZLK4ki4bP3lLDouTeGlTnjJ97Vsr2RQ+kcxfuPz HULA== X-Forwarded-Encrypted: i=1; AJvYcCWowG2U7DO7ohMNPBnT9DK4qYyARUI56IYU1SBF/sO+8saudpNvA0a4W34kFKRj/6dn32soUcpJlg==@kvack.org X-Gm-Message-State: AOJu0Yye+JZHn/Nry2AXqNCoQqCRDvJQTHQ+KdrXQf/3WgtC7IvaJZjD GLNfYNFbKWS7EOfuJ/JbfA2fpSlnpQjIadhPiZ9lgpLNJmUea2KXfivNjQR7eNKgtJgQQ5Q1zIz FGYMrrxhO+sFw073q4Y6DgucF+1htexWBQn2IPazPSeiGPXDF X-Gm-Gg: ASbGncvMBw8Nr7WbaD5NejfgH7XVba96uo0bQD4DPJW06bkTOu5fmhuGzvrZ9a5D+KU q8Rdj/IJtXGKrms9NVkR7pZSDLTWS4PZI8P7pgkID2uQsgWUpfPF8AJLx63EFIbd/p1e6jTFN5t VXoRY27Hlvh67nngiDSXO4jGGEzm3fCLbq4X8Vwr0OjXpEEZ7PmOoRTh1zxRkQRd5ccTVD7V6kj sXNNtSIfRZMhxYiJ1kdMCmW6/1r8zbOhzwZvSnypntTQ/nPAi4z5PsLoqNvP4sQjp/oHMoeWhji DtuqTVE5w7n4jh1jo5sAToXkH9U5jVCRsIBCggTQ X-Received: by 2002:a05:6402:2345:b0:5ed:1909:d422 with SMTP id 4fb4d7f45d1cf-5ed1909d5d6mr4532643a12.2.1742977137790; Wed, 26 Mar 2025 01:18:57 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHbdeo79z8qV2b7AKSc5yQktiHryMqsZHBcfBd1BlUWJgmAJ39Ce+nU2q92PUFekHQpyPfq7Q== X-Received: by 2002:a05:6402:2345:b0:5ed:1909:d422 with SMTP id 4fb4d7f45d1cf-5ed1909d5d6mr4532610a12.2.1742977137314; Wed, 26 Mar 2025 01:18:57 -0700 (PDT) Received: from alrua-x1.borgediget.toke.dk ([45.145.92.2]) by smtp.gmail.com with ESMTPSA id 4fb4d7f45d1cf-5ebccfae189sm8870097a12.37.2025.03.26.01.18.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 26 Mar 2025 01:18:56 -0700 (PDT) Received: by alrua-x1.borgediget.toke.dk (Postfix, from userid 1000) id 5291A18FC9CE; Wed, 26 Mar 2025 09:18:54 +0100 (CET) From: =?utf-8?q?Toke_H=C3=B8iland-J=C3=B8rgensen?= Date: Wed, 26 Mar 2025 09:18:39 +0100 Subject: [PATCH net-next v3 2/3] page_pool: Turn dma_sync into a full-width bool field MIME-Version: 1.0 Message-Id: <20250326-page-pool-track-dma-v3-2-8e464016e0ac@redhat.com> References: <20250326-page-pool-track-dma-v3-0-8e464016e0ac@redhat.com> In-Reply-To: <20250326-page-pool-track-dma-v3-0-8e464016e0ac@redhat.com> To: "David S. Miller" , Jakub Kicinski , Jesper Dangaard Brouer , Saeed Mahameed , Leon Romanovsky , Tariq Toukan , Andrew Lunn , Eric Dumazet , Paolo Abeni , Ilias Apalodimas , Simon Horman , Andrew Morton , Mina Almasry , Yonglong Liu , Yunsheng Lin , Pavel Begunkov , Matthew Wilcox Cc: netdev@vger.kernel.org, bpf@vger.kernel.org, linux-rdma@vger.kernel.org, linux-mm@kvack.org, =?utf-8?q?Toke_H=C3=B8iland-J=C3=B8rgensen?= X-Mailer: b4 0.14.2 X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: su-YSxIbKat03HG0nn_T3T1RlZ_GTIhQzunZF_et21o_1742977138 X-Mimecast-Originator: redhat.com X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 3191FC000D X-Stat-Signature: 9epn81xyef94ri3s9ogjnrdfrqzrr91u X-Rspam-User: X-HE-Tag: 1742977143-389096 X-HE-Meta: U2FsdGVkX18V6PH7tkhRtuIM8htFj/SXQ58XrlZJG3nuCyHcf2Gn3YykA6H9w8oMT6EWD2cqty9Aekge+udraOlMAPhutO0wU7cEeULfMucFO265moRJEHTbmYnYUEnDrtc2juvTVpErRzm9Bjm2yJ9h/M2OUMX9XtMr+Pcsl8PkvYA8hPWYDsOqVOo3MBtnveI35yQkpid7H/D4vBjF87PX8vsUedQayp++GD6UH1jDgkZInklyj2vJuJqSFf6+4oCMLf6uyggfh7hiEtppA5Cv5gz8ou68gEdJz1dVReEq0ZkKEWIgRUtGQKRL9Ia9JBr5QBAOw3mT2W6gHvu5p2l6ezbygbISvE065mF0td/PLfDSW/pDGzB3oeR4oTTnRiU728SB89Jud+j6nAX67zYhtSlUdnn6qyH+vinkrIif4bOJsvdR5XZxxZxF4chH7scz6ypYEBl/gxxI8c65yh4CDgqHMlNXr2C47j6ngka44EC94mOUGHt2rRzU/aZ+ZCW7hokzALtNGejLdX83Z6Paakd+/ae/clbDS0ppOwF2NJu+d4rsEYdTvNRd/eX8q149c2ScRSd2nhOr1V0GguN/d9MpU47mLjFKIpfzT/otw4N1w1JwhB2aRlVQErIbCzZ5wpIHjxwFLW46Q3uUtpvZ0UUgOYyWJfCS8I+JSnPeQO/F0EfeT2vjO/kE3thzpoPA1G21yZeutdFqNhxe8bXnUlJvkk/BPe1pJjrewk94Kl34MnNQYEST1uBy9fu2WoJd67pCXW76fe7SXKeMfb3xJVW/koCJ5LRte3Hj50hmiE67NlW/Vzej/05BAbcwtJWJp4yebEnGL6jUIh0eryx7Su+4OQ659LHfNeXC1sNoOHDU22xWxHv5HhocGAtZRMLBdx6hXLQ7oaFSUSGC1gnqmeDR7bM56mzgW4GWBoYZzRoNbk8mk7xx2KVG4WyDMD80gh/7CSTZwEWfDbu ind2zWlv xT+F9LKsIsxAhcrK4Q6MGJdGOMVM6wO103LmsEg/hPCPZmpZ9imlc/y25uyFsQvj867dw/hMlScH0G6qndYFbz8CPwbeEjEKR9XzXYdl88eM3uMUd3mTk6iXYWmN7UtK53LnlrU2FwOISk2ioNr+U4jhoyFjGsJtuDyLsI8MuqheTgpy5NJhlZlmJJr7lZb3nmgxx9Fc6VN/2lSfUscbCpA9nAh7+fPSsBMCpNt2ME5YsA33V/lOGHlAVfZ7XXqtfuQ30IH9t3yHq+gS6SsjHwDx2Hq/xaGTeFgxEl0t4qX3FNNnBe6/duUOTLpXHTfLuEPWQTLK3Lnvy+XSWIQFXamKeRyv1Oes/jQ9mSUSr/+1yudv4sO30gtSBIvRJZkk1dZ02pNatwGkrHe4SFCuhFq5wEcwgDhf+DzdBbPK1GHAWaFA8CrYTHxAW9YBLx3r/3iVi8SnXnaDFLqr8p8GaLE8x6jCkvtuuiHEsxrfhUNBC1HFynZ4EVuZU2MsqiFsD8moGxN7JPHwFBgFEZUHhhDas6bvwb2kO7J5o24iIzJVcoRcl9o83KTdjB2rww++5VnfBXeoz8wXYuAJgpTy2q6BH5D7dzzC+rh+yoS+A+ZKH+iElLn6nazjRdH9YNXAsSrMvadxqnVdJdMg= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Change the single-bit boolean for dma_sync into a full-width bool, so we can read it as volatile with READ_ONCE(). A subsequent patch will add writing with WRITE_ONCE() on teardown. Reviewed-by: Mina Almasry Tested-by: Yonglong Liu Signed-off-by: Toke Høiland-Jørgensen Acked-by: Jesper Dangaard Brouer --- include/net/page_pool/types.h | 6 +++--- net/core/page_pool.c | 2 +- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/include/net/page_pool/types.h b/include/net/page_pool/types.h index df0d3c1608929605224feb26173135ff37951ef8..d6c93150384fbc4579bb0d0afb357ebb26c564a3 100644 --- a/include/net/page_pool/types.h +++ b/include/net/page_pool/types.h @@ -173,10 +173,10 @@ struct page_pool { int cpuid; u32 pages_state_hold_cnt; - bool has_init_callback:1; /* slow::init_callback is set */ + bool dma_sync; /* Perform DMA sync for device */ + bool dma_sync_for_cpu:1; /* Perform DMA sync for cpu */ bool dma_map:1; /* Perform DMA mapping */ - bool dma_sync:1; /* Perform DMA sync for device */ - bool dma_sync_for_cpu:1; /* Perform DMA sync for cpu */ + bool has_init_callback:1; /* slow::init_callback is set */ #ifdef CONFIG_PAGE_POOL_STATS bool system:1; /* This is a global percpu pool */ #endif diff --git a/net/core/page_pool.c b/net/core/page_pool.c index acef1fcd8ddcfd1853a6f2055c1f1820ab248e8d..fb32768a97765aacc7f1103bfee38000c988b0de 100644 --- a/net/core/page_pool.c +++ b/net/core/page_pool.c @@ -466,7 +466,7 @@ page_pool_dma_sync_for_device(const struct page_pool *pool, netmem_ref netmem, u32 dma_sync_size) { - if (pool->dma_sync && dma_dev_need_sync(pool->p.dev)) + if (READ_ONCE(pool->dma_sync) && dma_dev_need_sync(pool->p.dev)) __page_pool_dma_sync_for_device(pool, netmem, dma_sync_size); } From patchwork Wed Mar 26 08:18:40 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Toke_H=C3=B8iland-J=C3=B8rgensen?= X-Patchwork-Id: 14029865 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A99DDC3600E for ; Wed, 26 Mar 2025 08:19:11 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A4A14280066; Wed, 26 Mar 2025 04:19:07 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 95DB1280056; Wed, 26 Mar 2025 04:19:07 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7165F280066; Wed, 26 Mar 2025 04:19:07 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 46567280056 for ; Wed, 26 Mar 2025 04:19:07 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id A5030C1EC6 for ; Wed, 26 Mar 2025 08:19:08 +0000 (UTC) X-FDA: 83263001976.12.BD47105 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf22.hostedemail.com (Postfix) with ESMTP id 8B413C0006 for ; Wed, 26 Mar 2025 08:19:06 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=OkIHxxZQ; spf=pass (imf22.hostedemail.com: domain of toke@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=toke@redhat.com; dmarc=pass (policy=quarantine) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1742977146; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=L3o8kOx0A3a9gpCq5fYDk6BqU9uXGkmyjhzH6v4re0o=; b=jhd2SuhP738hTqcr97Mm0LxWgCch+qupTodffBqYwSJ/Njj9fdaCpFVa/qN7gG2t+WZh/N nN/AKkrXcUjBdrNwwui0IOniWJapsqIo0pFMZfGtA+69yX+Cymob1YXw++UyWaDCA416n3 6JLvF8lmWNrvLaPNuXeZIOljKggzzkg= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1742977146; a=rsa-sha256; cv=none; b=i7XowyJUSAxssJv2JjqDLMmnjzgSD0XX9rA8wXveb2Emnvj5HFuGQlRxyg1svajLsU9aRE RaW0SQVf8206tkZXqzzG2lJvjRwrTgxyeT2yhnWuzQe9ioimDOh8dIQezbONh6PqbPgXG3 2cJ+Y741iILuxHznstItZxt4HAM/00Y= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=OkIHxxZQ; spf=pass (imf22.hostedemail.com: domain of toke@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=toke@redhat.com; dmarc=pass (policy=quarantine) header.from=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1742977145; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=L3o8kOx0A3a9gpCq5fYDk6BqU9uXGkmyjhzH6v4re0o=; b=OkIHxxZQPnYeyDa0nzn/26GeKbBgvLRVs5yo4uNwk5JSTluIN6tU0lFpI/FUC/yl6z/EBw H8N2mnpjQQiBWIe299qZwVZ8KvNeUY3tICu83R7B9Rvy/EU3h2ale3YHLOoQk3E5LzLlsm wcHfKhoXjm8/fsGjz14vYs2scdNo3rI= Received: from mail-ed1-f72.google.com (mail-ed1-f72.google.com [209.85.208.72]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-62-OEmHOtmiP-6ybNbpxS6Ljw-1; Wed, 26 Mar 2025 04:19:04 -0400 X-MC-Unique: OEmHOtmiP-6ybNbpxS6Ljw-1 X-Mimecast-MFC-AGG-ID: OEmHOtmiP-6ybNbpxS6Ljw_1742977143 Received: by mail-ed1-f72.google.com with SMTP id 4fb4d7f45d1cf-5e5eb195f01so5805299a12.1 for ; Wed, 26 Mar 2025 01:19:04 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1742977143; x=1743581943; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:subject:date:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=L3o8kOx0A3a9gpCq5fYDk6BqU9uXGkmyjhzH6v4re0o=; b=UL2dcHwI0Js4rHJuZ2KkobOUviVfZwsqzwHqDCANahhNRNkVvtYPnugzeDICuuLWCK YyYJ+kVAnEpe4jsNowcucwiSEGoCBhN1EYoB20IkR8ubSzS8ZYMxJQg73T2nY6BM5pEP ptKNUsJawiWTsoU3a3Oq1z1ufYMgrPA+2x4AyE7TEB4jeuNB31Z5FeZGYmpo9okXS36D t5vjmqrm1JMZXWv1Xjgu/j6P8HlpuPJJeoZKPmxflFnwv06UuJcKpMTSWl2uJ8Sjtl6Z Zf984TgaQu2LtsIXtcTXrs5bZC9PN+QoSovap2pO4BTqKkhYuYdJbfX+mGDqFx/GS8FP JRzA== X-Forwarded-Encrypted: i=1; AJvYcCXaC5plx4Hoe7E1Uh+J5Jbn46k78febWYThbL2EFXHPuO9dOVZJtiBVaaHiMivyK+tAyFUCXMr1kQ==@kvack.org X-Gm-Message-State: AOJu0YyuBKvw+JfYdHYW7LNUVppOmj08/EdaIznE3YrYo1449IRSakRW FpJ6GK2HULBeDIHYhHiwZEdUsK2YCUhOKUjsFrqFTGekJe0U5QyTAOssUFqiYiPhJB21sH9hQaL Xe1y00s2NN1SoU1yV+GDtGhHJFB3N4R1SgjDMsD6GhiZPEm16 X-Gm-Gg: ASbGnctFv8pa6jB6qu461B72s0rLt1xCqYMaewhU6hOGRNWJSk8mkTi1qWFf0DFeRQ1 gq2ltz4Vx90uohyqVbrsGpnCD0atBNHxigcO0Q77vQouudyJCjnyXyfDkbE9u28MXhDvNpvF+kT 35s76T+byxVe7Zqn8Sn7T18erZUCkDvUw1PuRL2zOts6EJUDo9B6qeaiLkncTHstQPSHoasoQUt vZ9NuJ2WAIFbqvIZkgbtCvoCiXcgCEHvIpQnviqa/Q2z3xSVVfV3JVJsZKmzJJKt24GAFqPNZUI dACch9p92cmZo8MvJ+mvnVnmj4gk5VKGxXbRbSDz X-Received: by 2002:a17:906:d7cb:b0:ac4:5fd:6e29 with SMTP id a640c23a62f3a-ac405fd6eaamr1484667266b.26.1742977142658; Wed, 26 Mar 2025 01:19:02 -0700 (PDT) X-Google-Smtp-Source: AGHT+IFKE+mfhhLv2lQjs/xlsgXqislY1jyDT4Dp3H+d9g7vxrFzqat7h23i/Fa4g4X1nbBbVSrbqA== X-Received: by 2002:a17:906:d7cb:b0:ac4:5fd:6e29 with SMTP id a640c23a62f3a-ac405fd6eaamr1484664166b.26.1742977142073; Wed, 26 Mar 2025 01:19:02 -0700 (PDT) Received: from alrua-x1.borgediget.toke.dk ([45.145.92.2]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-ac3efbd3d3esm983376066b.130.2025.03.26.01.19.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 26 Mar 2025 01:19:01 -0700 (PDT) Received: by alrua-x1.borgediget.toke.dk (Postfix, from userid 1000) id 54C1F18FC9D0; Wed, 26 Mar 2025 09:18:54 +0100 (CET) From: =?utf-8?q?Toke_H=C3=B8iland-J=C3=B8rgensen?= Date: Wed, 26 Mar 2025 09:18:40 +0100 Subject: [PATCH net-next v3 3/3] page_pool: Track DMA-mapped pages and unmap them when destroying the pool MIME-Version: 1.0 Message-Id: <20250326-page-pool-track-dma-v3-3-8e464016e0ac@redhat.com> References: <20250326-page-pool-track-dma-v3-0-8e464016e0ac@redhat.com> In-Reply-To: <20250326-page-pool-track-dma-v3-0-8e464016e0ac@redhat.com> To: "David S. Miller" , Jakub Kicinski , Jesper Dangaard Brouer , Saeed Mahameed , Leon Romanovsky , Tariq Toukan , Andrew Lunn , Eric Dumazet , Paolo Abeni , Ilias Apalodimas , Simon Horman , Andrew Morton , Mina Almasry , Yonglong Liu , Yunsheng Lin , Pavel Begunkov , Matthew Wilcox Cc: netdev@vger.kernel.org, bpf@vger.kernel.org, linux-rdma@vger.kernel.org, linux-mm@kvack.org, =?utf-8?q?Toke_H=C3=B8iland-J=C3=B8rgensen?= , Qiuling Ren , Yuying Ma X-Mailer: b4 0.14.2 X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: 0zvfGezq-0jms-9al2IyyF5U6jpS_lXJFiqn1ADQlSw_1742977143 X-Mimecast-Originator: redhat.com X-Stat-Signature: 6quzicuydaehgpr57gbqhnsh5dwphjn8 X-Rspam-User: X-Rspamd-Queue-Id: 8B413C0006 X-Rspamd-Server: rspam08 X-HE-Tag: 1742977146-445544 X-HE-Meta: U2FsdGVkX19cu/9jt/z4ItWUPqN7zPLbIvKlFNRw2RZm3rMwS4YmtEZcLE30g2d2DJVd9Nlb2WrujaVpk9PzObiFxs/PHs+1GDPwufZ1khoZgBcsiAEqMf57r+o4Lr71nes9PgeRZ/t/JXhOCkTrw1HyKv/Zw9/84hnOPKGz2PxI3YvG2VhqeyQZruH8DLZZrXv9Yycfc98ct4eG9Fv0dpsusxlySJ5bh2iNqigw5/xlAxP36xM50AyI7FVmTUvZEiiLoeRK5NRWtoAmt8Vzhi3hff5kjtk/GYUIns+cVyPenkqeuQE1WVw1Vufz1NcpOAnV3HJgr6tB/u9uf4iUBrE6uaAbN7RyH3FTJEcrjSgdhdbqkMVzldoMECCwxnVIBqq0LZTs46GalfppX8WAFpLLvdCLgS36uMegpOU5zTZWKkFG2hGeTN+cDMjd5vYqelMYJJs2xPRT80PBER3ek2d5+K3EEd/wD8vOCRYjwzeVTR5m6LVzY3A9reqaN4LCrcogve8lyD3R6z6AAqO73xYsQCHXqc3noTPWcNHO5fsi+e+7D8PFlywHRVVYrBL8otNcZqBC5pQTVdAjuSq0I0YyA60SXvfhBK7iYNEQO1un0FHI3IrMFi126qreDQUS3nG8+E/XhEdyG0xCNpjJeKSew+Yd9QWAxrzP/M61Sr1Omy/PWEqOmXCB3EJR+6JPh0A14FRIoaxWCQ+OSzCEafxNqrmfA+aDaxrL19SeHfLb8Tutr2HEK1ilwQ1DaV9QyUvnVGZDxjdtiJndESrz2RQ5mxXQGbVgA7I+mfYJhpuR46Edul+3I00Fakz0mfZN8+9OQMWynNo03IkdtmdxjcmIamhcHx4gYAITZDcK5qKp6bMMPuehc8Eznwd2CCwRdUz6O2JYsjaVrudTQqJYAgZqUljUP2JzSSs1mamGGEYUboS+psnFPkGlaB4IVCETjhKQlvZPkt4cjLxHcOv 1MT1ivmf +Au5NMp8Az9BRLlUE0Ejp2STUCVre+x/hE+wa8RhpJcOjrff0/caboq2knxYlVQtRi0i4y4L2DDhlLwjzPFarQp0FsvFKy/aC0LCQpCOCjD1tzOx+xoPz4P1aFvjHWugNHoS8 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: When enabling DMA mapping in page_pool, pages are kept DMA mapped until they are released from the pool, to avoid the overhead of re-mapping the pages every time they are used. This causes resource leaks and/or crashes when there are pages still outstanding while the device is torn down, because page_pool will attempt an unmap through a non-existent DMA device on the subsequent page return. To fix this, implement a simple tracking of outstanding DMA-mapped pages in page pool using an xarray. This was first suggested by Mina[0], and turns out to be fairly straight forward: We simply store pointers to pages directly in the xarray with xa_alloc() when they are first DMA mapped, and remove them from the array on unmap. Then, when a page pool is torn down, it can simply walk the xarray and unmap all pages still present there before returning, which also allows us to get rid of the get/put_device() calls in page_pool. Using xa_cmpxchg(), no additional synchronisation is needed, as a page will only ever be unmapped once. To avoid having to walk the entire xarray on unmap to find the page reference, we stash the ID assigned by xa_alloc() into the page structure itself, using the upper bits of the pp_magic field. This requires a couple of defines to avoid conflicting with the POINTER_POISON_DELTA define, but this is all evaluated at compile-time, so does not affect run-time performance. The bitmap calculations in this patch gives the following number of bits for different architectures: - 23 bits on 32-bit architectures - 21 bits on PPC64 (because of the definition of ILLEGAL_POINTER_VALUE) - 32 bits on other 64-bit architectures Stashing a value into the unused bits of pp_magic does have the effect that it can make the value stored there lie outside the unmappable range (as governed by the mmap_min_addr sysctl), for architectures that don't define ILLEGAL_POINTER_VALUE. This means that if one of the pointers that is aliased to the pp_magic field (such as page->lru.next) is dereferenced while the page is owned by page_pool, that could lead to a dereference into userspace, which is a security concern. The risk of this is mitigated by the fact that (a) we always clear pp_magic before releasing a page from page_pool, and (b) this would need a use-after-free bug for struct page, which can have many other risks since page->lru.next is used as a generic list pointer in multiple places in the kernel. As such, with this patch we take the position that this risk is negligible in practice. For more discussion, see[1]. Since all the tracking added in this patch is performed on DMA map/unmap, no additional code is needed in the fast path, meaning the performance overhead of this tracking is negligible there. A micro-benchmark shows that the total overhead of the tracking itself is about 400 ns (39 cycles(tsc) 395.218 ns; sum for both map and unmap[2]). Since this cost is only paid on DMA map and unmap, it seems like an acceptable cost to fix the late unmap issue. Further optimisation can narrow the cases where this cost is paid (for instance by eliding the tracking when DMA map/unmap is a no-op). The extra memory needed to track the pages is neatly encapsulated inside xarray, which uses the 'struct xa_node' structure to track items. This structure is 576 bytes long, with slots for 64 items, meaning that a full node occurs only 9 bytes of overhead per slot it tracks (in practice, it probably won't be this efficient, but in any case it should be an acceptable overhead). [0] https://lore.kernel.org/all/CAHS8izPg7B5DwKfSuzz-iOop_YRbk3Sd6Y4rX7KBG9DcVJcyWg@mail.gmail.com/ [1] https://lore.kernel.org/r/20250320023202.GA25514@openwall.com [2] https://lore.kernel.org/r/ae07144c-9295-4c9d-a400-153bb689fe9e@huawei.com Reported-by: Yonglong Liu Closes: https://lore.kernel.org/r/8743264a-9700-4227-a556-5f931c720211@huawei.com Fixes: ff7d6b27f894 ("page_pool: refurbish version of page_pool code") Suggested-by: Mina Almasry Reviewed-by: Mina Almasry Reviewed-by: Jesper Dangaard Brouer Tested-by: Jesper Dangaard Brouer Tested-by: Qiuling Ren Tested-by: Yuying Ma Tested-by: Yonglong Liu Signed-off-by: Toke Høiland-Jørgensen --- include/linux/poison.h | 4 +++ include/net/page_pool/types.h | 49 +++++++++++++++++++++++--- net/core/netmem_priv.h | 28 ++++++++++++++- net/core/page_pool.c | 81 ++++++++++++++++++++++++++++++++++++------- 4 files changed, 145 insertions(+), 17 deletions(-) diff --git a/include/linux/poison.h b/include/linux/poison.h index 331a9a996fa8746626afa63ea462b85ca3e5938b..5351efd710d5e21cc341f7bb533b1aeea4a0808a 100644 --- a/include/linux/poison.h +++ b/include/linux/poison.h @@ -70,6 +70,10 @@ #define KEY_DESTROY 0xbd /********** net/core/page_pool.c **********/ +/* + * page_pool uses additional free bits within this value to store data, see the + * definition of PP_DMA_INDEX_MASK in include/net/page_pool/types.h + */ #define PP_SIGNATURE (0x40 + POISON_POINTER_DELTA) /********** net/core/skbuff.c **********/ diff --git a/include/net/page_pool/types.h b/include/net/page_pool/types.h index d6c93150384fbc4579bb0d0afb357ebb26c564a3..cdc06fb67642c104b071998920bd88125fbefd43 100644 --- a/include/net/page_pool/types.h +++ b/include/net/page_pool/types.h @@ -6,6 +6,7 @@ #include #include #include +#include #include #define PP_FLAG_DMA_MAP BIT(0) /* Should page_pool do the DMA @@ -54,13 +55,51 @@ struct pp_alloc_cache { netmem_ref cache[PP_ALLOC_CACHE_SIZE]; }; +/* + * DMA mapping IDs + * + * When DMA-mapping a page, we allocate an ID (from an xarray) and stash this in + * the upper bits of page->pp_magic. We always want to be able to unambiguously + * identify page pool pages (using page_pool_page_is_pp()). Non-PP pages can + * have arbitrary kernel pointers stored in the same field as pp_magic (since it + * overlaps with page->lru.next), so we must ensure that we cannot mistake a + * valid kernel pointer with any of the values we write into this field. + * + * On architectures that set POISON_POINTER_DELTA, this is already ensured, + * since this value becomes part of PP_SIGNATURE; meaning we can just use the + * space between the PP_SIGNATURE value (without POISON_POINTER_DELTA), and the + * lowest bits of POISON_POINTER_DELTA. On arches where POISON_POINTER_DELTA is + * 0, we make sure that we leave the two topmost bits empty, as that guarantees + * we won't mistake a valid kernel pointer for a value we set, regardless of the + * VMSPLIT setting. + * + * Altogether, this means that the number of bits available is constrained by + * the size of an unsigned long (at the upper end, subtracting two bits per the + * above), and the definition of PP_SIGNATURE (with or without + * POISON_POINTER_DELTA). + */ +#define PP_DMA_INDEX_SHIFT (1 + __fls(PP_SIGNATURE - POISON_POINTER_DELTA)) +#if POISON_POINTER_DELTA > 0 +/* PP_SIGNATURE includes POISON_POINTER_DELTA, so limit the size of the DMA + * index to not overlap with that if set + */ +#define PP_DMA_INDEX_BITS MIN(32, __ffs(POISON_POINTER_DELTA) - PP_DMA_INDEX_SHIFT) +#else +/* Always leave out the topmost two; see above. */ +#define PP_DMA_INDEX_BITS MIN(32, BITS_PER_LONG - PP_DMA_INDEX_SHIFT - 2) +#endif + +#define PP_DMA_INDEX_MASK GENMASK(PP_DMA_INDEX_BITS + PP_DMA_INDEX_SHIFT - 1, \ + PP_DMA_INDEX_SHIFT) +#define PP_DMA_INDEX_LIMIT XA_LIMIT(1, BIT(PP_DMA_INDEX_BITS) - 1) + /* Mask used for checking in page_pool_page_is_pp() below. page->pp_magic is * OR'ed with PP_SIGNATURE after the allocation in order to preserve bit 0 for - * the head page of compound page and bit 1 for pfmemalloc page. - * page_is_pfmemalloc() is checked in __page_pool_put_page() to avoid recycling - * the pfmemalloc page. + * the head page of compound page and bit 1 for pfmemalloc page, as well as the + * bits used for the DMA index. page_is_pfmemalloc() is checked in + * __page_pool_put_page() to avoid recycling the pfmemalloc page. */ -#define PP_MAGIC_MASK ~0x3UL +#define PP_MAGIC_MASK ~(PP_DMA_INDEX_MASK | 0x3UL) /** * struct page_pool_params - page pool parameters @@ -229,6 +268,8 @@ struct page_pool { void *mp_priv; const struct memory_provider_ops *mp_ops; + struct xarray dma_mapped; + #ifdef CONFIG_PAGE_POOL_STATS /* recycle stats are per-cpu to avoid locking */ struct page_pool_recycle_stats __percpu *recycle_stats; diff --git a/net/core/netmem_priv.h b/net/core/netmem_priv.h index f33162fd281c23e109273ba09950c5d0a2829bc9..cd95394399b40c3604934ba7898eeeeacb8aee99 100644 --- a/net/core/netmem_priv.h +++ b/net/core/netmem_priv.h @@ -5,7 +5,7 @@ static inline unsigned long netmem_get_pp_magic(netmem_ref netmem) { - return __netmem_clear_lsb(netmem)->pp_magic; + return __netmem_clear_lsb(netmem)->pp_magic & ~PP_DMA_INDEX_MASK; } static inline void netmem_or_pp_magic(netmem_ref netmem, unsigned long pp_magic) @@ -15,6 +15,8 @@ static inline void netmem_or_pp_magic(netmem_ref netmem, unsigned long pp_magic) static inline void netmem_clear_pp_magic(netmem_ref netmem) { + WARN_ON_ONCE(__netmem_clear_lsb(netmem)->pp_magic & PP_DMA_INDEX_MASK); + __netmem_clear_lsb(netmem)->pp_magic = 0; } @@ -33,4 +35,28 @@ static inline void netmem_set_dma_addr(netmem_ref netmem, { __netmem_clear_lsb(netmem)->dma_addr = dma_addr; } + +static inline unsigned long netmem_get_dma_index(netmem_ref netmem) +{ + unsigned long magic; + + if (WARN_ON_ONCE(netmem_is_net_iov(netmem))) + return 0; + + magic = __netmem_clear_lsb(netmem)->pp_magic; + + return (magic & PP_DMA_INDEX_MASK) >> PP_DMA_INDEX_SHIFT; +} + +static inline void netmem_set_dma_index(netmem_ref netmem, + unsigned long id) +{ + unsigned long magic; + + if (WARN_ON_ONCE(netmem_is_net_iov(netmem))) + return; + + magic = netmem_get_pp_magic(netmem) | (id << PP_DMA_INDEX_SHIFT); + __netmem_clear_lsb(netmem)->pp_magic = magic; +} #endif diff --git a/net/core/page_pool.c b/net/core/page_pool.c index fb32768a97765aacc7f1103bfee38000c988b0de..5eb132f7f2193bda76121cc10a24dac4993b2c21 100644 --- a/net/core/page_pool.c +++ b/net/core/page_pool.c @@ -226,6 +226,8 @@ static int page_pool_init(struct page_pool *pool, return -EINVAL; pool->dma_map = true; + + xa_init_flags(&pool->dma_mapped, XA_FLAGS_ALLOC1); } if (pool->slow.flags & PP_FLAG_DMA_SYNC_DEV) { @@ -275,9 +277,6 @@ static int page_pool_init(struct page_pool *pool, /* Driver calling page_pool_create() also call page_pool_destroy() */ refcount_set(&pool->user_cnt, 1); - if (pool->dma_map) - get_device(pool->p.dev); - if (pool->slow.flags & PP_FLAG_ALLOW_UNREADABLE_NETMEM) { /* We rely on rtnl_lock()ing to make sure netdev_rx_queue * configuration doesn't change while we're initializing @@ -325,7 +324,7 @@ static void page_pool_uninit(struct page_pool *pool) ptr_ring_cleanup(&pool->ring, NULL); if (pool->dma_map) - put_device(pool->p.dev); + xa_destroy(&pool->dma_mapped); #ifdef CONFIG_PAGE_POOL_STATS if (!pool->system) @@ -466,13 +465,20 @@ page_pool_dma_sync_for_device(const struct page_pool *pool, netmem_ref netmem, u32 dma_sync_size) { - if (READ_ONCE(pool->dma_sync) && dma_dev_need_sync(pool->p.dev)) - __page_pool_dma_sync_for_device(pool, netmem, dma_sync_size); + if (dma_dev_need_sync(pool->p.dev)) { + rcu_read_lock(); + if (READ_ONCE(pool->dma_sync)) + __page_pool_dma_sync_for_device(pool, netmem, + dma_sync_size); + rcu_read_unlock(); + } } -static bool page_pool_dma_map(struct page_pool *pool, netmem_ref netmem) +static bool page_pool_dma_map(struct page_pool *pool, netmem_ref netmem, gfp_t gfp) { dma_addr_t dma; + int err; + u32 id; /* Setup DMA mapping: use 'struct page' area for storing DMA-addr * since dma_addr_t can be either 32 or 64 bits and does not always fit @@ -486,15 +492,28 @@ static bool page_pool_dma_map(struct page_pool *pool, netmem_ref netmem) if (dma_mapping_error(pool->p.dev, dma)) return false; - if (page_pool_set_dma_addr_netmem(netmem, dma)) + if (in_softirq()) + err = xa_alloc(&pool->dma_mapped, &id, netmem_to_page(netmem), + PP_DMA_INDEX_LIMIT, gfp); + else + err = xa_alloc_bh(&pool->dma_mapped, &id, netmem_to_page(netmem), + PP_DMA_INDEX_LIMIT, gfp); + if (err) { + WARN_ONCE(1, "couldn't track DMA mapping, please report to netdev@"); goto unmap_failed; + } + if (page_pool_set_dma_addr_netmem(netmem, dma)) { + WARN_ONCE(1, "unexpected DMA address, please report to netdev@"); + goto unmap_failed; + } + + netmem_set_dma_index(netmem, id); page_pool_dma_sync_for_device(pool, netmem, pool->p.max_len); return true; unmap_failed: - WARN_ONCE(1, "unexpected DMA address, please report to netdev@"); dma_unmap_page_attrs(pool->p.dev, dma, PAGE_SIZE << pool->p.order, pool->p.dma_dir, DMA_ATTR_SKIP_CPU_SYNC | DMA_ATTR_WEAK_ORDERING); @@ -511,7 +530,7 @@ static struct page *__page_pool_alloc_page_order(struct page_pool *pool, if (unlikely(!page)) return NULL; - if (pool->dma_map && unlikely(!page_pool_dma_map(pool, page_to_netmem(page)))) { + if (pool->dma_map && unlikely(!page_pool_dma_map(pool, page_to_netmem(page), gfp))) { put_page(page); return NULL; } @@ -557,7 +576,7 @@ static noinline netmem_ref __page_pool_alloc_pages_slow(struct page_pool *pool, */ for (i = 0; i < nr_pages; i++) { netmem = pool->alloc.cache[i]; - if (dma_map && unlikely(!page_pool_dma_map(pool, netmem))) { + if (dma_map && unlikely(!page_pool_dma_map(pool, netmem, gfp))) { put_page(netmem_to_page(netmem)); continue; } @@ -659,6 +678,8 @@ void page_pool_clear_pp_info(netmem_ref netmem) static __always_inline void __page_pool_release_page_dma(struct page_pool *pool, netmem_ref netmem) { + struct page *old, *page = netmem_to_page(netmem); + unsigned long id; dma_addr_t dma; if (!pool->dma_map) @@ -667,6 +688,17 @@ static __always_inline void __page_pool_release_page_dma(struct page_pool *pool, */ return; + id = netmem_get_dma_index(netmem); + if (!id) + return; + + if (in_softirq()) + old = xa_cmpxchg(&pool->dma_mapped, id, page, NULL, 0); + else + old = xa_cmpxchg_bh(&pool->dma_mapped, id, page, NULL, 0); + if (old != page) + return; + dma = page_pool_get_dma_addr_netmem(netmem); /* When page is unmapped, it cannot be returned to our pool */ @@ -674,6 +706,7 @@ static __always_inline void __page_pool_release_page_dma(struct page_pool *pool, PAGE_SIZE << pool->p.order, pool->p.dma_dir, DMA_ATTR_SKIP_CPU_SYNC | DMA_ATTR_WEAK_ORDERING); page_pool_set_dma_addr_netmem(netmem, 0); + netmem_set_dma_index(netmem, 0); } /* Disconnects a page (from a page_pool). API users can have a need @@ -1083,8 +1116,32 @@ static void page_pool_empty_alloc_cache_once(struct page_pool *pool) static void page_pool_scrub(struct page_pool *pool) { + unsigned long id; + void *ptr; + page_pool_empty_alloc_cache_once(pool); - pool->destroy_cnt++; + if (!pool->destroy_cnt++ && pool->dma_map) { + if (pool->dma_sync) { + /* paired with READ_ONCE in + * page_pool_dma_sync_for_device() and + * __page_pool_dma_sync_for_cpu() + */ + WRITE_ONCE(pool->dma_sync, false); + + /* Make sure all concurrent returns that may see the old + * value of dma_sync (and thus perform a sync) have + * finished before doing the unmapping below. Skip the + * wait if the device doesn't actually need syncing, or + * if there are no outstanding mapped pages. + */ + if (dma_dev_need_sync(pool->p.dev) && + !xa_empty(&pool->dma_mapped)) + synchronize_net(); + } + + xa_for_each(&pool->dma_mapped, id, ptr) + __page_pool_release_page_dma(pool, page_to_netmem(ptr)); + } /* No more consumers should exist, but producers could still * be in-flight.