From patchwork Mon Apr 15 13:19:37 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yunsheng Lin X-Patchwork-Id: 13630035 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B5A8EC4345F for ; Mon, 15 Apr 2024 13:22:26 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 251216B00A7; Mon, 15 Apr 2024 09:22:25 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 201616B00A6; Mon, 15 Apr 2024 09:22:25 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0CB876B00A7; Mon, 15 Apr 2024 09:22:24 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id D92316B00A5 for ; Mon, 15 Apr 2024 09:22:24 -0400 (EDT) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id A51DF804A6 for ; Mon, 15 Apr 2024 13:22:24 +0000 (UTC) X-FDA: 82011830208.23.DC6559F Received: from szxga06-in.huawei.com (szxga06-in.huawei.com [45.249.212.32]) by imf26.hostedemail.com (Postfix) with ESMTP id 2E2CE140017 for ; Mon, 15 Apr 2024 13:22:21 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=none; dmarc=pass (policy=quarantine) header.from=huawei.com; spf=pass (imf26.hostedemail.com: domain of linyunsheng@huawei.com designates 45.249.212.32 as permitted sender) smtp.mailfrom=linyunsheng@huawei.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1713187342; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=tg8ub9r3BJQYC3dUFyIgL4Vkq+m+TACZ2+T1X/aYvvw=; b=qXveBQyJb4kU/Z90YADfStey/7hG8d5gs6usxPhZXWRMG9wOHgBKHQ5Qbf0ixbv8xTPzS2 rV+h1PM6cCf/Srlbks55D0EzN7E5/YzfgEL4FLjhwgMNA0ACyt/56OGunVe+PyNkUewEjv 8Jz8USQzg6OQOmQMuemROVrZS6r38FM= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=none; dmarc=pass (policy=quarantine) header.from=huawei.com; spf=pass (imf26.hostedemail.com: domain of linyunsheng@huawei.com designates 45.249.212.32 as permitted sender) smtp.mailfrom=linyunsheng@huawei.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1713187342; a=rsa-sha256; cv=none; b=IYGCxET1JTR/Z8scjFAfyqwWKg8DpUQlXqmUzx6cNAHrSTE2z7vvbzM7J899tctVKG0XDF PZYSEgi1Z7Wp/Vm9e9Wa365RMyF5RJ9aVoK6hu3PrZzMz3FPz/w7pN7/KDg5d9ippuRkxf UYaSCC2o5VC7o9ABa5z30wNOu1s7NMU= Received: from mail.maildlp.com (unknown [172.19.163.17]) by szxga06-in.huawei.com (SkyGuard) with ESMTP id 4VJ79F6CnJz1ws3y; Mon, 15 Apr 2024 21:21:21 +0800 (CST) Received: from dggpemm500005.china.huawei.com (unknown [7.185.36.74]) by mail.maildlp.com (Postfix) with ESMTPS id 8DB7B1A0172; Mon, 15 Apr 2024 21:22:19 +0800 (CST) Received: from localhost.localdomain (10.69.192.56) by dggpemm500005.china.huawei.com (7.185.36.74) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.35; Mon, 15 Apr 2024 21:22:19 +0800 From: Yunsheng Lin To: , , CC: , , Yunsheng Lin , Alexander Duyck , Andrew Morton , Subject: [PATCH net-next v2 12/15] mm: page_frag: introduce prepare/commit API for page_frag Date: Mon, 15 Apr 2024 21:19:37 +0800 Message-ID: <20240415131941.51153-13-linyunsheng@huawei.com> X-Mailer: git-send-email 2.33.0 In-Reply-To: <20240415131941.51153-1-linyunsheng@huawei.com> References: <20240415131941.51153-1-linyunsheng@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.69.192.56] X-ClientProxiedBy: dggems706-chm.china.huawei.com (10.3.19.183) To dggpemm500005.china.huawei.com (7.185.36.74) X-Rspamd-Queue-Id: 2E2CE140017 X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: crmo67ubojnz3hrw87zdjyxtcbxuek36 X-HE-Tag: 1713187341-911376 X-HE-Meta: U2FsdGVkX18SKPXtL3Fa3lCqww/cEWwtmVsZ2Y8W3C4nQFp11ZomHUbEopmkT1O3q6jnKEKhSTpXD/L3gvi6kCcJHeLLrzNR7aCxY7E1ezuPvnkFrE5cUEDFhG/QkWuocgy9EqRT6Z5tnmL2y2qo5VmlZbBxjVxHE/ZkHn2lltBXVrkyh6OiG/p+jlJKSEanK5VbP6aa4pfYxDzwmJrLgEVQ9pe3oQLT0IP2DJcBkb30UjbKu0BcdvToIEVWcyil7ews973xD4Rg0288Gvyuc5nwCFzhVLymGfppAUMw1yKsSpETfnSvOinzBpwAPyr84JPSnkZ3ZmsSyDdrzJJyqKd4IQgQnhvTHmH9xLvnCPvuFkASQA4BHuaGcCuMrcug6wFLVVkvHz35pFPsTR4WfAJrScn48jWZjAfewmY1hBQ4uja3UwpSPIja31rp/EULaa5lePrK3X1JZ3tGPcmEZXeXFkuPsQ2LYXvroRIhhrX25MTCYYSFfGVHweWTGMRJswnKhJg6WvyYdu2Hv/17VFTEVmdcRtDtPy83oF8uKlHp57C/uzM/dg7pdevkYopTes1OTo5rTbu2ZPlyeZs0Ee4Om3pe7AwFNM7VH4pU5nsKPASigiyPvU5g+x8KStaBAiuxC8+1hKfDs0VLdG4gHiW+y8wW7u1zwoxGNYblyNunUaOtN/m+kKnoFJX3bIfxjB1vdzyvXByBh+gIplGC6CK6Bs+yztlrhT7G5ZkwqIYwPIlc9APbW0b6gRVqIKBhErTzHjnXBar3aIc6MGi4p3PpyLq8zTXf+6jPd1y39jlNNh4FklPuj3CgfFgpcn1zCBiKq9J+jiSjMhbi6Nd8bU/0xhKBw+4mozZgrDJkLf3pf/akKpka2yKUwJvq+56rd0nmHSFXWokz/UWz9FjX9azv9FPymbEzgIMPsktViaVaqqcJJ8JGkvmFuzcx8hNTroI4GW9/o88SqJ2OUL3 T8hdbzbO smUXasDZagPC75VRIDscQPZU9Z9mFEM88W1b3uijkAh28/htqkDQGbWipAZQL6hBULHByqnQuGnoYo2GCOzgBap48I8obbIdU7tuBBkSHXBYoKheFOHJZfNFG1LU/y9cwvwIVB89+/rv13ulIe4Q8gW2JGwEOn8kTVPCiETVwACzbNYoizorntgG4NZxhC3LruHkrtKSEAGKsZ2l46xbyEH5UAxBurO4mztN5SheymYmJceladbHNdT6zfTgkWbLPWIwml6notveY2P+hvi+Wo8Cp7Wk6WQdFx6rgLHomOdchdw/8zvyV+on6t/rzgJ6s5N2z X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: There are many use cases that need minimum memory in order for forward progressing, but can do better if there is more memory available. Currently skb_page_frag_refill() API is used to solve the above usecases, as mentioned in [1], its implementation is similar to the one in mm subsystem. To unify those two page_frag implementations, introduce a prepare API to ensure minimum memory is satisfied and return how much the actual memory is available to the caller. And the caller can decide how much memory to use by calling commit API, or not calling the commit API if deciding to not use any memory. Note it seems hard to decide which header files for caling virt_to_page() in the inline helper, so macro is used instead of inline helper to avoid dealing with that. 1. https://lore.kernel.org/all/20240228093013.8263-1-linyunsheng@huawei.com/ Signed-off-by: Yunsheng Lin --- include/linux/page_frag_cache.h | 141 +++++++++++++++++++++++++++++++- mm/page_frag_cache.c | 13 ++- 2 files changed, 144 insertions(+), 10 deletions(-) diff --git a/include/linux/page_frag_cache.h b/include/linux/page_frag_cache.h index a97a1ac017d6..28185969cd2c 100644 --- a/include/linux/page_frag_cache.h +++ b/include/linux/page_frag_cache.h @@ -43,8 +43,25 @@ static inline bool page_frag_cache_is_pfmemalloc(struct page_frag_cache *nc) void page_frag_cache_drain(struct page_frag_cache *nc); void __page_frag_cache_drain(struct page *page, unsigned int count); -void *page_frag_alloc_va(struct page_frag_cache *nc, unsigned int fragsz, - gfp_t gfp_mask); +void *page_frag_cache_refill(struct page_frag_cache *nc, unsigned int fragsz, + gfp_t gfp_mask); + +static inline void *page_frag_alloc_va(struct page_frag_cache *nc, + unsigned int fragsz, gfp_t gfp_mask) +{ + unsigned int offset; + void *va; + + va = page_frag_cache_refill(nc, fragsz, gfp_mask); + if (unlikely(!va)) + return NULL; + + offset = nc->offset; + nc->pagecnt_bias--; + nc->offset = offset + fragsz; + + return va + offset; +} static inline void *__page_frag_alloc_va_align(struct page_frag_cache *nc, unsigned int fragsz, @@ -69,6 +86,126 @@ static inline void *page_frag_alloc_va_align(struct page_frag_cache *nc, return __page_frag_alloc_va_align(nc, fragsz, gfp_mask, align); } +static inline void *page_frag_alloc_va_prepare(struct page_frag_cache *nc, + unsigned int *offset, + unsigned int *size, + gfp_t gfp_mask) +{ + void *va; + + va = page_frag_cache_refill(nc, *size, gfp_mask); + if (unlikely(!va)) + return NULL; + + *offset = nc->offset; + +#if (PAGE_SIZE < PAGE_FRAG_CACHE_MAX_SIZE) + *size = nc->size_mask - *offset + 1; +#else + *size = PAGE_SIZE - *offset; +#endif + + return va + *offset; +} + +static inline void *page_frag_alloc_va_prepare_align(struct page_frag_cache *nc, + unsigned int *offset, + unsigned int *size, + unsigned int align, + gfp_t gfp_mask) +{ + WARN_ON_ONCE(!is_power_of_2(align) || align >= PAGE_SIZE || + *size < sizeof(unsigned int)); + + *offset = nc->offset; + nc->offset = ALIGN(*offset, align); + return page_frag_alloc_va_prepare(nc, offset, size, gfp_mask); +} + +static inline void *__page_frag_alloc_pg_prepare(struct page_frag_cache *nc, + unsigned int *offset, + unsigned int *size, + gfp_t gfp_mask) +{ + void *va; + + va = page_frag_cache_refill(nc, *size, gfp_mask); + if (unlikely(!va)) + return NULL; + + *offset = nc->offset; + +#if (PAGE_SIZE < PAGE_FRAG_CACHE_MAX_SIZE) + *size = nc->size_mask - *offset + 1; +#else + *size = PAGE_SIZE - *offset; +#endif + + return va; +} + +#define page_frag_alloc_pg_prepare(nc, offset, size, gfp) \ +({ \ + struct page *__page = NULL; \ + void *__va; \ + \ + __va = __page_frag_alloc_pg_prepare(nc, offset, size, gfp); \ + if (likely(__va)) \ + __page = virt_to_page(__va); \ + \ + __page; \ +}) + +static inline void *__page_frag_alloc_prepare(struct page_frag_cache *nc, + unsigned int *offset, + unsigned int *size, + void **va, gfp_t gfp_mask) +{ + void *nc_va; + + nc_va = page_frag_cache_refill(nc, *size, gfp_mask); + if (unlikely(!nc_va)) + return NULL; + + *offset = nc->offset; + *va = nc_va + *offset; + +#if (PAGE_SIZE < PAGE_FRAG_CACHE_MAX_SIZE) + *size = nc->size_mask - *offset + 1; +#else + *size = PAGE_SIZE - *offset; +#endif + + return nc_va; +} + +#define page_frag_alloc_prepare(nc, offset, size, va, gfp) \ +({ \ + struct page *__page = NULL; \ + void *__va; \ + \ + __va = __page_frag_alloc_prepare(nc, offset, size, va, gfp); \ + if (likely(__va)) \ + __page = virt_to_page(__va); \ + \ + __page; \ +}) + +static inline void page_frag_alloc_commit(struct page_frag_cache *nc, + unsigned int offset, + unsigned int size) +{ + nc->pagecnt_bias--; + nc->offset = offset + size; +} + +static inline void page_frag_alloc_commit_noref(struct page_frag_cache *nc, + unsigned int offset, + unsigned int size) +{ + nc->offset = offset + size; +} + void page_frag_free_va(void *addr); #endif diff --git a/mm/page_frag_cache.c b/mm/page_frag_cache.c index 5f7f96c88163..8774cb07e630 100644 --- a/mm/page_frag_cache.c +++ b/mm/page_frag_cache.c @@ -79,8 +79,8 @@ void __page_frag_cache_drain(struct page *page, unsigned int count) } EXPORT_SYMBOL(__page_frag_cache_drain); -void *page_frag_alloc_va(struct page_frag_cache *nc, unsigned int fragsz, - gfp_t gfp_mask) +void *page_frag_cache_refill(struct page_frag_cache *nc, unsigned int fragsz, + gfp_t gfp_mask) { unsigned long size_mask; unsigned int offset; @@ -118,7 +118,7 @@ void *page_frag_alloc_va(struct page_frag_cache *nc, unsigned int fragsz, set_page_count(page, size_mask); nc->pagecnt_bias |= size_mask; - offset = 0; + nc->offset = 0; if (unlikely(fragsz > (size_mask + 1))) { /* * The caller is trying to allocate a fragment @@ -133,12 +133,9 @@ void *page_frag_alloc_va(struct page_frag_cache *nc, unsigned int fragsz, } } - nc->pagecnt_bias--; - nc->offset = offset + fragsz; - - return va + offset; + return va; } -EXPORT_SYMBOL(page_frag_alloc_va); +EXPORT_SYMBOL(page_frag_cache_refill); /* * Frees a page fragment allocated out of either a compound or order 0 page.