From patchwork Tue Oct 1 07:58:53 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yunsheng Lin X-Patchwork-Id: 13895545 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 10F8CE7716D for ; Thu, 5 Dec 2024 15:21:25 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1BAC96B00C7; Thu, 5 Dec 2024 10:19:16 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id EA8A96B00DF; Thu, 5 Dec 2024 10:19:12 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D94856B00B5; Thu, 5 Dec 2024 10:19:10 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id E279B280036 for ; Tue, 1 Oct 2024 04:00:07 -0400 (EDT) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 87B021C544F for ; Tue, 1 Oct 2024 08:00:07 +0000 (UTC) X-FDA: 82624285254.26.C67D406 Received: from mail-pg1-f195.google.com (mail-pg1-f195.google.com [209.85.215.195]) by imf13.hostedemail.com (Postfix) with ESMTP id 77E362001B for ; Tue, 1 Oct 2024 08:00:05 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=iGHVivh4; spf=pass (imf13.hostedemail.com: domain of yunshenglin0825@gmail.com designates 209.85.215.195 as permitted sender) smtp.mailfrom=yunshenglin0825@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1727769478; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=r1RucxwqQRpOPX6vOYgsd3mSR4ltbKTf5O9GaoA3/H0=; b=L0jFZjTHi5i2fnQvYHpYhV6hMo9praRPldIDCKonvY7Cc4mHDUHR0q8dhUg7AcezNSFF8V gIvcUuiZzjoO+cPQP89Kh4nFsu0lNcoLz5y+e4aiQM54CmhI2w+o0nIjuAb0Iuf2gVeJ00 mGi7JUusrHCH10rLUo76qNA4m05pBi4= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1727769478; a=rsa-sha256; cv=none; b=24PmKbkTTauX0Igh7mss+aoJFqRNurQBaFB0cgNiIXRi6yGimjYB0zFSira0UQ6Nf5hOS4 WVdzaeClqFK8aPZCQimm9oPZcSXSwCb4nDrKwTSw+eFgM2cxzYsUMwp4fqHFbfQBKEcC1Q csIrDEe2ETIET2IbOZssW1h4zphFckI= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=iGHVivh4; spf=pass (imf13.hostedemail.com: domain of yunshenglin0825@gmail.com designates 209.85.215.195 as permitted sender) smtp.mailfrom=yunshenglin0825@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-pg1-f195.google.com with SMTP id 41be03b00d2f7-7c1324be8easo4690822a12.1 for ; Tue, 01 Oct 2024 01:00:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1727769604; x=1728374404; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=r1RucxwqQRpOPX6vOYgsd3mSR4ltbKTf5O9GaoA3/H0=; b=iGHVivh4HBGXNcRqdvc+YWi5u3nekEzQdZj75N7mkTry5lhz0k6Ulil2+cFZ6uiMKV 1gm7IIBwXZJKwUt0G2lKY6NzOTvSl7qDFBEkQP2bTRIMBMwYAP1jCojYh1zd4U8gMopy fTL1zSBee8OgQ0cEbGluhgG1kbCwkAxqDdDBa2+IBmJLHhtlPyjHbbAsL1CWmpiWvjBx 1AdgMRM3ic2SB+ZAAowZp7Df0zDMzYj5fV1KdFTVZWCR++Be9X3KyCM8gAqXo90GJAqb a3cESj5APiTZLwvexV04JXIEIPDxG8Jitos7K1OMIYSFQfzDKixe/gD00JiFeZVBX17+ oNOg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1727769604; x=1728374404; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=r1RucxwqQRpOPX6vOYgsd3mSR4ltbKTf5O9GaoA3/H0=; b=XGMgdvYaGc0dMoEs4W5FwQXpsRPD8q1T9LL/6uIvQUG7WJWMiTUmozX2jZfeDk+O2T ZEvjXgEipe+NlXl4UaR9BIBZ0M2Uy3zuj+AucyXqbnWbvINwzoar1XnHVQhL+Zmazza5 HwS5NwNB1msAhb1u08IycOREGDg2m1pl1yk2sUstuOuzX2Nfb8j1AZmOXqlHDQoXvntV cKnElLxHyvME1xF4XDDbAbnldSUR9xvz6h5CNM7NR/BeRFNd771blyK+DFh92mYuFvoC K44uXywjqj5ciDjWMAc0lwH/srwu1d9HyRM4YLysV8D69ZEnar64H5rSCeYp1gwf2YrW JIvA== X-Forwarded-Encrypted: i=1; AJvYcCWrEsUkAqD3bA2KxPaCw/6ejgwaIZfKgWyshEEqAflxOMIKHSJ3nUf7DgeT7G6s2G2gJxPEfmhVKA==@kvack.org X-Gm-Message-State: AOJu0Yz7ItNH2qTWiHMc7TFWTKT3Mllc+flVgXEcBjSzreeH1gMXLy9b XnsemmKxORecFpYJ01wvIIZzm0WXYHIfjiROO+u5mHQWLkwc/1XI X-Google-Smtp-Source: AGHT+IFPxcxt0MFdf7tMGPm+w589CsajkXhlQaj5n+1Gi7AVAld6EP944pYZHk1LqVuIZirkwu8ymQ== X-Received: by 2002:a17:90a:e7d1:b0:2e0:728d:fb3b with SMTP id 98e67ed59e1d1-2e15a3431afmr3591602a91.18.1727769604111; Tue, 01 Oct 2024 01:00:04 -0700 (PDT) Received: from yunshenglin-MS-7549.. ([2409:8a55:301b:e120:88bd:a0fb:c6d6:c4a2]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-2e06e16d6d2sm13168168a91.2.2024.10.01.01.00.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 01 Oct 2024 01:00:03 -0700 (PDT) From: Yunsheng Lin X-Google-Original-From: Yunsheng Lin To: davem@davemloft.net, kuba@kernel.org, pabeni@redhat.com Cc: netdev@vger.kernel.org, linux-kernel@vger.kernel.org, Yunsheng Lin , Alexander Duyck , Andrew Morton , linux-mm@kvack.org Subject: [PATCH net-next v19 10/14] mm: page_frag: introduce prepare/probe/commit API Date: Tue, 1 Oct 2024 15:58:53 +0800 Message-Id: <20241001075858.48936-11-linyunsheng@huawei.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20241001075858.48936-1-linyunsheng@huawei.com> References: <20241001075858.48936-1-linyunsheng@huawei.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 77E362001B X-Stat-Signature: a3ze9gmrg66r9owu7kx5tn53idwp3g1b X-HE-Tag: 1727769605-272100 X-HE-Meta: U2FsdGVkX181cEW39VYHEsFWSrDcwnO2WsNRuSaANKtDRKwFYjDz4EtEOthCtdpird89sL3JC0JKI5Vo99Gko1iKb6MEWkPEJl3iHhcUNe9kqnlcn++2tClzkC2biU7eFhJQ27+ca2y/WSP0+5PyXIkn8L0Qin+Kmm8VF9tg9cXJ4cG/YMXqnWA/OUILjDLMu99b8ifKzeWovJR7U5e5vAtCfOcgmizBtlOsFzxx7GBFmHRs0o6hSm5V6yft4ouWpLbrwA8mgfeqPCoffcNMG0DK8lnX1HIgWpg6CPdTLtaa4Vcaup+7aP+LO+kCe/x7doFPamQC2vrIY+sEpVinsA7RwDmSzagpQ9G7KyN2xTL+VY8RuUjA13Zx/o9EwBAl7/OGjtNRXTPxYMXowSJ96o4+7tR7yxDi2YvoHxEVs97qi+IhzeqSZ8utKOD8jpK/F1PlochyMTGg0h9FacMiMBCZK2FlTJaDP3aluB8o22NkJxNPcoI6Kpm5FjX/EN9Zw48up+CPijlPCt1pe3aDZ3bbR51655fEVr7zbY7Np+pTLq+C+6oqZhnEwh3Tr6TUSCCwCQWv2HqRg0Ku26TZFoV2JobdEJjUM/SkzvSu5EkJtLcmydE5dB8nnTxe9/cA4aiGnipEuzdgcS0wkGHgvGG94FmIapu7m7yF0hnXCxk1QjVrL0m0xKP2GBIDyYrsKnXt+QYFYnIW0ORLSjASrzFWNHs8Q2rA0gOORINHEhbm/rZMracnXk1GLInQ88JydGaJ1ImHNcWJGIE7vxl8XtL+8gVOSVVKtL3YLl4b/L9ofLu2P3nJwecHaKFROnL11zj4VLfGrVHgRsdVp7RzuY/hFNDReqM6n+EbWCnmXLQj+ommcIZswsvlaubwW1mYXGtZFqXFXI1jM0DS8/gBsIVludCeJ9x3toe3w//TckEE+JYyklu5ai4kbT+u6FAh5626TeRXX5qIrBugJtL howwd49i DarhFLBt1tTpU/2ZwIAc3br6Xmmp+XepjZeoPsecTccdl/P9CI499BeaO3QnvI0slxQOvdJMjrnsXkIdG+mthL6yXmN2Q7Is212vQ8CNikIbM6KMnjgoe8QyV3DhE45ybR4B0/0iF4G0bxEcwW+6TRNG3+7YXe6W5CEArMCibZynLj5owjzzMF5Y8oNzWS6a7tdEAcsnZUuS4rpw8+Z0JKEgQSL/KLOPCM2ne9cNH1ry4t8FiSt2Jipc9i4CRcxYI9b8tXzbaNE279YKb8ZOpc84tYBgErCQl9Sh+pZ1aIiEGCTT9fN1789V7KOz0w5DEECI6Rfj+xnlSOPNao/Q2dI3Fvxs/AlqTixYPK71UMgkdeI1PATJ2USrYzwDgxpCt0RI9Xsw9PrUqSL75dc5TLZ5yiQFRj781AjKBFPenIXv5swjfJHcAlT3TakUC2GPo0NuQO4gwETm96hxQc9D7g6F6YNOdV5jZnR+aNhhzt4Ajd1a6Ux3m6sDY18FYbJNvmbZPBDqMiGwh/ZI6LkxH4O9rbdy1sQ5UIowUQ1LnfEhxs4UJbWMN0YaBfjwNjUrIgwigXjuplWyQrHJfJ5z0JvO5UWn1OwhdlXpH X-Bogosity: Ham, tests=bogofilter, spamicity=0.000004, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: There are many use cases that need minimum memory in order for forward progress, but more performant if more memory is available or need to probe the cache info to use any memory available for frag caoleasing reason. Currently skb_page_frag_refill() API is used to solve the above use cases, but caller needs to know about the internal detail and access the data field of 'struct page_frag' to meet the requirement of the above use cases and its implementation is similar to the one in mm subsystem. To unify those two page_frag implementations, introduce a prepare API to ensure minimum memory is satisfied and return how much the actual memory is available to the caller and a probe API to report the current available memory to caller without doing cache refilling. The caller needs to either call the commit API to report how much memory it actually uses, or not do so if deciding to not use any memory. CC: Alexander Duyck Signed-off-by: Yunsheng Lin --- include/linux/page_frag_cache.h | 135 ++++++++++++++++++++++++++++++++ mm/page_frag_cache.c | 21 +++++ 2 files changed, 156 insertions(+) diff --git a/include/linux/page_frag_cache.h b/include/linux/page_frag_cache.h index b634e1338741..4e9018051956 100644 --- a/include/linux/page_frag_cache.h +++ b/include/linux/page_frag_cache.h @@ -40,6 +40,11 @@ static inline bool page_frag_cache_is_pfmemalloc(struct page_frag_cache *nc) return page_frag_encoded_page_pfmemalloc(nc->encoded_page); } +static inline unsigned int page_frag_cache_page_offset(const struct page_frag_cache *nc) +{ + return nc->offset; +} + void page_frag_cache_drain(struct page_frag_cache *nc); void __page_frag_cache_drain(struct page *page, unsigned int count); void *__page_frag_cache_prepare(struct page_frag_cache *nc, unsigned int fragsz, @@ -48,6 +53,10 @@ void *__page_frag_cache_prepare(struct page_frag_cache *nc, unsigned int fragsz, unsigned int __page_frag_cache_commit_noref(struct page_frag_cache *nc, struct page_frag *pfrag, unsigned int used_sz); +void *__page_frag_alloc_refill_probe_align(struct page_frag_cache *nc, + unsigned int fragsz, + struct page_frag *pfrag, + unsigned int align_mask); static inline unsigned int __page_frag_cache_commit(struct page_frag_cache *nc, struct page_frag *pfrag, @@ -90,6 +99,132 @@ static inline void *page_frag_alloc(struct page_frag_cache *nc, return __page_frag_alloc_align(nc, fragsz, gfp_mask, ~0u); } +static inline bool __page_frag_refill_align(struct page_frag_cache *nc, + unsigned int fragsz, + struct page_frag *pfrag, + gfp_t gfp_mask, + unsigned int align_mask) +{ + if (unlikely(!__page_frag_cache_prepare(nc, fragsz, pfrag, gfp_mask, + align_mask))) + return false; + + __page_frag_cache_commit(nc, pfrag, fragsz); + return true; +} + +static inline bool page_frag_refill_align(struct page_frag_cache *nc, + unsigned int fragsz, + struct page_frag *pfrag, + gfp_t gfp_mask, unsigned int align) +{ + WARN_ON_ONCE(!is_power_of_2(align)); + return __page_frag_refill_align(nc, fragsz, pfrag, gfp_mask, -align); +} + +static inline bool page_frag_refill(struct page_frag_cache *nc, + unsigned int fragsz, + struct page_frag *pfrag, gfp_t gfp_mask) +{ + return __page_frag_refill_align(nc, fragsz, pfrag, gfp_mask, ~0u); +} + +static inline bool __page_frag_refill_prepare_align(struct page_frag_cache *nc, + unsigned int fragsz, + struct page_frag *pfrag, + gfp_t gfp_mask, + unsigned int align_mask) +{ + return !!__page_frag_cache_prepare(nc, fragsz, pfrag, gfp_mask, + align_mask); +} + +static inline bool page_frag_refill_prepare_align(struct page_frag_cache *nc, + unsigned int fragsz, + struct page_frag *pfrag, + gfp_t gfp_mask, + unsigned int align) +{ + WARN_ON_ONCE(!is_power_of_2(align)); + return __page_frag_refill_prepare_align(nc, fragsz, pfrag, gfp_mask, + -align); +} + +static inline bool page_frag_refill_prepare(struct page_frag_cache *nc, + unsigned int fragsz, + struct page_frag *pfrag, + gfp_t gfp_mask) +{ + return __page_frag_refill_prepare_align(nc, fragsz, pfrag, gfp_mask, + ~0u); +} + +static inline void *__page_frag_alloc_refill_prepare_align(struct page_frag_cache *nc, + unsigned int fragsz, + struct page_frag *pfrag, + gfp_t gfp_mask, + unsigned int align_mask) +{ + return __page_frag_cache_prepare(nc, fragsz, pfrag, gfp_mask, align_mask); +} + +static inline void *page_frag_alloc_refill_prepare_align(struct page_frag_cache *nc, + unsigned int fragsz, + struct page_frag *pfrag, + gfp_t gfp_mask, + unsigned int align) +{ + WARN_ON_ONCE(!is_power_of_2(align)); + return __page_frag_alloc_refill_prepare_align(nc, fragsz, pfrag, + gfp_mask, -align); +} + +static inline void *page_frag_alloc_refill_prepare(struct page_frag_cache *nc, + unsigned int fragsz, + struct page_frag *pfrag, + gfp_t gfp_mask) +{ + return __page_frag_alloc_refill_prepare_align(nc, fragsz, pfrag, + gfp_mask, ~0u); +} + +static inline void *page_frag_alloc_refill_probe(struct page_frag_cache *nc, + unsigned int fragsz, + struct page_frag *pfrag) +{ + return __page_frag_alloc_refill_probe_align(nc, fragsz, pfrag, ~0u); +} + +static inline bool page_frag_refill_probe(struct page_frag_cache *nc, + unsigned int fragsz, + struct page_frag *pfrag) +{ + return !!page_frag_alloc_refill_probe(nc, fragsz, pfrag); +} + +static inline void page_frag_commit(struct page_frag_cache *nc, + struct page_frag *pfrag, + unsigned int used_sz) +{ + __page_frag_cache_commit(nc, pfrag, used_sz); +} + +static inline void page_frag_commit_noref(struct page_frag_cache *nc, + struct page_frag *pfrag, + unsigned int used_sz) +{ + __page_frag_cache_commit_noref(nc, pfrag, used_sz); +} + +static inline void page_frag_alloc_abort(struct page_frag_cache *nc, + unsigned int fragsz) +{ + VM_BUG_ON(fragsz > nc->offset); + + nc->pagecnt_bias++; + nc->offset -= fragsz; +} + void page_frag_free(void *addr); #endif diff --git a/mm/page_frag_cache.c b/mm/page_frag_cache.c index a5448b44068a..c052c77a96eb 100644 --- a/mm/page_frag_cache.c +++ b/mm/page_frag_cache.c @@ -117,6 +117,27 @@ unsigned int __page_frag_cache_commit_noref(struct page_frag_cache *nc, } EXPORT_SYMBOL(__page_frag_cache_commit_noref); +void *__page_frag_alloc_refill_probe_align(struct page_frag_cache *nc, + unsigned int fragsz, + struct page_frag *pfrag, + unsigned int align_mask) +{ + unsigned long encoded_page = nc->encoded_page; + unsigned int size, offset; + + size = page_frag_cache_page_size(encoded_page); + offset = __ALIGN_KERNEL_MASK(nc->offset, ~align_mask); + if (unlikely(!encoded_page || offset + fragsz > size)) + return NULL; + + pfrag->page = page_frag_encoded_page_ptr(encoded_page); + pfrag->size = size - offset; + pfrag->offset = offset; + + return page_frag_encoded_page_address(encoded_page) + offset; +} +EXPORT_SYMBOL(__page_frag_alloc_refill_probe_align); + void *__page_frag_cache_prepare(struct page_frag_cache *nc, unsigned int fragsz, struct page_frag *pfrag, gfp_t gfp_mask, unsigned int align_mask)