From patchwork Mon Oct 14 10:58:11 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13834672 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id EA213D1A443 for ; Mon, 14 Oct 2024 10:59:41 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 83DEC6B008A; Mon, 14 Oct 2024 06:59:41 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 7C7866B008C; Mon, 14 Oct 2024 06:59:41 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 690106B0092; Mon, 14 Oct 2024 06:59:41 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 4D5476B008A for ; Mon, 14 Oct 2024 06:59:41 -0400 (EDT) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 5C5A38075C for ; Mon, 14 Oct 2024 10:59:34 +0000 (UTC) X-FDA: 82671911910.13.79D0A95 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf17.hostedemail.com (Postfix) with ESMTP id 60FC240010 for ; Mon, 14 Oct 2024 10:59:34 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=none; spf=pass (imf17.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1728903391; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=kYeTTVqmvopusG+SkQLqUutllx+fKzmiPfJpsd8f/kQ=; b=kRFgBwa1uksiX+zMwFszoVmrtbLNmPmwim+9QMSdsea1H29WSSctPcr/5fKvkK2ZiazW6o AB8GCg2dJK/igUPe/Kxa4t5WmOKcq1CWDW8fJsYqOTr7ZgvIDN7vSx5kmEPtHDq5oCNWjB ec2qtBrlVZOF4ilE/NNa3FgaKFeGvt8= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=none; spf=pass (imf17.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1728903391; a=rsa-sha256; cv=none; b=3RaWhNKSzxLfPSv6R+ZA7FvS2JWC2ZEMAjVZY9GLbg+saRQIGxfCFoFQWcwKxAbUd8JdDf 5oS8l/u4hzUzkxkn0t3DVgS/qaZZ77/JuG67pS6OWyxXhf7PcKMt9Sx8JLWewgcuv3gYL0 r8rHGqk/rpxMae8m8x3IzqvVcBuUyvE= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 405271688; Mon, 14 Oct 2024 04:00:08 -0700 (PDT) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.27]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 403163F51B; Mon, 14 Oct 2024 03:59:36 -0700 (PDT) From: Ryan Roberts To: Andrew Morton , Anshuman Khandual , Ard Biesheuvel , Catalin Marinas , David Hildenbrand , Greg Marsden , Ivan Ivanov , Kalesh Singh , Marc Zyngier , Mark Rutland , Matthias Brugger , Miroslav Benes , Will Deacon Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [RFC PATCH v1 04/57] mm/page_alloc: Make page_frag_cache boot-time page size compatible Date: Mon, 14 Oct 2024 11:58:11 +0100 Message-ID: <20241014105912.3207374-4-ryan.roberts@arm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20241014105912.3207374-1-ryan.roberts@arm.com> References: <20241014105514.3206191-1-ryan.roberts@arm.com> <20241014105912.3207374-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 60FC240010 X-Stat-Signature: eu519u4fxnqkzpk9sr1j4zwba5bc4p6h X-Rspam-User: X-HE-Tag: 1728903574-993617 X-HE-Meta: U2FsdGVkX1+JqLx+wMzBeRkiDlq7cXGYBk9VrNRDAZMabR4QQf1dxvvb7rtszRbPDlDWkb4KXZ56lhKokmTk9W7sa8AKHixe0+yWvqU07uJ7ACc2ojWZjTuNS4rLyhVja/PaKE+TUMVxByp1mrRcYq4pleMpJFbkrXUX51ODSAJCQBeZRdyYMLL9tbun9xBiZz9AWygjeCIdq8kdfZy4A4MvMqsWPVwX6kl2qmE+CqAgOIeZVXfjqNI6Ap4Yb3+pXfnbCoPvYNIVcM3Y24ZxIUWuqT70lyRHxywgomr4XT2PhnAGj4Utp4BEPSV2ieJd2PU+r2xkyLkFm/IucQFFFgpPSLC+cGOr2kx0cVtR2nfo5J1z9NlUTMQ3Se21VquPtzYUMb30rO31+3/9rSCwzd2Knjev0hjhzFQTkwvr+zLRj0Jl+Jdh9m6tfVrD1HvnmPI+Pd1WZWa5IdV/UdbaW0S/9AdYVSspPtHUfjnzpjkc7zaZvqfiFG9M8qVvOblvYnlBo2M85uDrHc3c/pb53JmkLwd22vpjKdRN5przfBNruEux0ZQ6sC1ibv0y/EYLiF9lvQMkgogXAx74XnQUsiiQK5n8KleNzf1bRS7Kldc41Z/wLI3Elia67iX0m4YaxkF1E/l55PGbtxoQKueedJ/5qhtW2WQ8KzEY1aYcOgacbfWg4tN6y23V4NlA4GpKQ85je2aNo4d/Tff0qnM1rgtpLtTuBrZmDGi3Q27mY9lVrDoWjlDi5vOh+nNRmdMcavjR428nxgIxXN8YysA5qsBHxVsiKHIrbzLZzgLXouNczhfGA1oT8Tb3en7rUA/xML0a1a0lExbCSpNVjjE0UhIAulYDaLHuBx3JLJcHuNj/LSv2uvof3IP0ELyf4eU4Z8HkicJiEEFHd3NYWE5VLVk1AOSc/Ce2zbxDlM6lW6kTWnl1av24LUS8yz/Oc4EDlrpUYJopeuqinQMZXU2 VUn+18ob 30o+0fEUK3Py61RZmUkgf5ibT+5igoGVJZwquBj22KsSqIWkzX+m1Bjir6g6XC9TiOCCFuaXHiLsYLw316tunZ8pV12mMLif6bVO5noglMoTffpa4IzhMLcB7GvJ0kt4VWu46aq7DWW3ZVSSz2E42kBbnp6jA04OwN5pZZcnxxdx9+1BsVIWT7DLQFsTjq/e5VXerVIi9Otn6TlcFrtLApykF3cc3LGKSCkTU X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: "struct page_frag_cache" has some optimizations that depend on page size. Let's refactor it a bit so that those optimizations can be determined at run-time for the case where page size is a boot-time parameter. For compile-time page size, the compiler should dead code strip and the result is very similar to before. One wrinkle is that we don't know if we need the size member until runtime. So remove the ifdeffery and always define offset as u32 (needed if PAGE_SIZE is >= 64K) and size as u16 (only used when PAGE_SIZE <= 32K). We move the members around a bit so that the overall size of the struct remains the same; 24 bytes for 64-bit and 16 bytes on 32 bit. Signed-off-by: Ryan Roberts page_alloc Acked-by: Vlastimil Babka --- ***NOTE*** Any confused maintainers may want to read the cover note here for context: https://lore.kernel.org/all/20241014105514.3206191-1-ryan.roberts@arm.com/ include/linux/mm_types.h | 13 ++++++------- mm/page_alloc.c | 31 ++++++++++++++++++------------- 2 files changed, 24 insertions(+), 20 deletions(-) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 4854249792545..0844ed7cfaa53 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -544,16 +544,15 @@ static inline void *folio_get_private(struct folio *folio) struct page_frag_cache { void * va; -#if (PAGE_SIZE < PAGE_FRAG_CACHE_MAX_SIZE) - __u16 offset; - __u16 size; -#else - __u32 offset; -#endif /* we maintain a pagecount bias, so that we dont dirty cache line * containing page->_refcount every time we allocate a fragment. */ - unsigned int pagecnt_bias; + unsigned int pagecnt_bias; + __u32 offset; + /* size only used when PAGE_SIZE < PAGE_FRAG_CACHE_MAX_SIZE, in which + * case PAGE_FRAG_CACHE_MAX_SIZE is 32K and 16 bits is sufficient. + */ + __u16 size; bool pfmemalloc; }; diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 91ace8ca97e21..8678103b1b396 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -4822,13 +4822,18 @@ static struct page *__page_frag_cache_refill(struct page_frag_cache *nc, struct page *page = NULL; gfp_t gfp = gfp_mask; -#if (PAGE_SIZE < PAGE_FRAG_CACHE_MAX_SIZE) - gfp_mask = (gfp_mask & ~__GFP_DIRECT_RECLAIM) | __GFP_COMP | - __GFP_NOWARN | __GFP_NORETRY | __GFP_NOMEMALLOC; - page = alloc_pages_node(NUMA_NO_NODE, gfp_mask, - PAGE_FRAG_CACHE_MAX_ORDER); - nc->size = page ? PAGE_FRAG_CACHE_MAX_SIZE : PAGE_SIZE; -#endif + if (PAGE_SIZE < PAGE_FRAG_CACHE_MAX_SIZE) { + gfp_mask = (gfp_mask & ~__GFP_DIRECT_RECLAIM) | __GFP_COMP | + __GFP_NOWARN | __GFP_NORETRY | __GFP_NOMEMALLOC; + page = alloc_pages_node(NUMA_NO_NODE, gfp_mask, + PAGE_FRAG_CACHE_MAX_ORDER); + /* + * Cast to silence warning due to 16-bit nc->size. Not real + * because PAGE_SIZE only less than PAGE_FRAG_CACHE_MAX_SIZE + * when PAGE_FRAG_CACHE_MAX_SIZE is 32K. + */ + nc->size = (__u16)(page ? PAGE_FRAG_CACHE_MAX_SIZE : PAGE_SIZE); + } if (unlikely(!page)) page = alloc_pages_node(NUMA_NO_NODE, gfp, 0); @@ -4870,10 +4875,10 @@ void *__page_frag_alloc_align(struct page_frag_cache *nc, if (!page) return NULL; -#if (PAGE_SIZE < PAGE_FRAG_CACHE_MAX_SIZE) /* if size can vary use size else just use PAGE_SIZE */ - size = nc->size; -#endif + if (PAGE_SIZE < PAGE_FRAG_CACHE_MAX_SIZE) + size = nc->size; + /* Even if we own the page, we do not use atomic_set(). * This would break get_page_unless_zero() users. */ @@ -4897,10 +4902,10 @@ void *__page_frag_alloc_align(struct page_frag_cache *nc, goto refill; } -#if (PAGE_SIZE < PAGE_FRAG_CACHE_MAX_SIZE) /* if size can vary use size else just use PAGE_SIZE */ - size = nc->size; -#endif + if (PAGE_SIZE < PAGE_FRAG_CACHE_MAX_SIZE) + size = nc->size; + /* OK, page count is 0, we can safely set it */ set_page_count(page, PAGE_FRAG_CACHE_MAX_SIZE + 1);