From patchwork Mon Oct 14 10:58:11 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13834805 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id CCBA7D16246 for ; Mon, 14 Oct 2024 11:29:17 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=kYeTTVqmvopusG+SkQLqUutllx+fKzmiPfJpsd8f/kQ=; b=DrITu5yuKHaGjbK4BK29zXSfJW OESD/tNgHEPTTMUQxQpuUgIVHiuxtVA0Cnvzs2vvm2jtwIDI3KUlTiCutfqzdHs8skDNfk0MjS1HE gMfknCPqu/xDIgJDInpI8fvQTgQBaHRMq2f+KGdBkFxxnaOuYUX8+qApPfh55ABC1Ra/Q8Vjl19qb 5byaeN4SuWCfM0zk4rdJNTPafrrbg/5UNmc2hYeqxnsTIPXI4GIwYw9Y0k0rJ08TpJYkWm3s5/azy 1NBOR6hJ5TO7XyTdL1r681LYejHopMa7kPpCMetC3ZflmRi1pirFJN8G+FSsTvElN577Ta/U7MQ2x 3DJ/0mmg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1t0JG5-00000004vfg-3hgg; Mon, 14 Oct 2024 11:29:05 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1t0Inb-00000004nQ8-04g3 for linux-arm-kernel@lists.infradead.org; Mon, 14 Oct 2024 10:59:41 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 405271688; Mon, 14 Oct 2024 04:00:08 -0700 (PDT) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.27]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 403163F51B; Mon, 14 Oct 2024 03:59:36 -0700 (PDT) From: Ryan Roberts To: Andrew Morton , Anshuman Khandual , Ard Biesheuvel , Catalin Marinas , David Hildenbrand , Greg Marsden , Ivan Ivanov , Kalesh Singh , Marc Zyngier , Mark Rutland , Matthias Brugger , Miroslav Benes , Will Deacon Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [RFC PATCH v1 04/57] mm/page_alloc: Make page_frag_cache boot-time page size compatible Date: Mon, 14 Oct 2024 11:58:11 +0100 Message-ID: <20241014105912.3207374-4-ryan.roberts@arm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20241014105912.3207374-1-ryan.roberts@arm.com> References: <20241014105514.3206191-1-ryan.roberts@arm.com> <20241014105912.3207374-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241014_035939_286634_E3D4692B X-CRM114-Status: GOOD ( 16.95 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org "struct page_frag_cache" has some optimizations that depend on page size. Let's refactor it a bit so that those optimizations can be determined at run-time for the case where page size is a boot-time parameter. For compile-time page size, the compiler should dead code strip and the result is very similar to before. One wrinkle is that we don't know if we need the size member until runtime. So remove the ifdeffery and always define offset as u32 (needed if PAGE_SIZE is >= 64K) and size as u16 (only used when PAGE_SIZE <= 32K). We move the members around a bit so that the overall size of the struct remains the same; 24 bytes for 64-bit and 16 bytes on 32 bit. Signed-off-by: Ryan Roberts page_alloc Acked-by: Vlastimil Babka --- ***NOTE*** Any confused maintainers may want to read the cover note here for context: https://lore.kernel.org/all/20241014105514.3206191-1-ryan.roberts@arm.com/ include/linux/mm_types.h | 13 ++++++------- mm/page_alloc.c | 31 ++++++++++++++++++------------- 2 files changed, 24 insertions(+), 20 deletions(-) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 4854249792545..0844ed7cfaa53 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -544,16 +544,15 @@ static inline void *folio_get_private(struct folio *folio) struct page_frag_cache { void * va; -#if (PAGE_SIZE < PAGE_FRAG_CACHE_MAX_SIZE) - __u16 offset; - __u16 size; -#else - __u32 offset; -#endif /* we maintain a pagecount bias, so that we dont dirty cache line * containing page->_refcount every time we allocate a fragment. */ - unsigned int pagecnt_bias; + unsigned int pagecnt_bias; + __u32 offset; + /* size only used when PAGE_SIZE < PAGE_FRAG_CACHE_MAX_SIZE, in which + * case PAGE_FRAG_CACHE_MAX_SIZE is 32K and 16 bits is sufficient. + */ + __u16 size; bool pfmemalloc; }; diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 91ace8ca97e21..8678103b1b396 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -4822,13 +4822,18 @@ static struct page *__page_frag_cache_refill(struct page_frag_cache *nc, struct page *page = NULL; gfp_t gfp = gfp_mask; -#if (PAGE_SIZE < PAGE_FRAG_CACHE_MAX_SIZE) - gfp_mask = (gfp_mask & ~__GFP_DIRECT_RECLAIM) | __GFP_COMP | - __GFP_NOWARN | __GFP_NORETRY | __GFP_NOMEMALLOC; - page = alloc_pages_node(NUMA_NO_NODE, gfp_mask, - PAGE_FRAG_CACHE_MAX_ORDER); - nc->size = page ? PAGE_FRAG_CACHE_MAX_SIZE : PAGE_SIZE; -#endif + if (PAGE_SIZE < PAGE_FRAG_CACHE_MAX_SIZE) { + gfp_mask = (gfp_mask & ~__GFP_DIRECT_RECLAIM) | __GFP_COMP | + __GFP_NOWARN | __GFP_NORETRY | __GFP_NOMEMALLOC; + page = alloc_pages_node(NUMA_NO_NODE, gfp_mask, + PAGE_FRAG_CACHE_MAX_ORDER); + /* + * Cast to silence warning due to 16-bit nc->size. Not real + * because PAGE_SIZE only less than PAGE_FRAG_CACHE_MAX_SIZE + * when PAGE_FRAG_CACHE_MAX_SIZE is 32K. + */ + nc->size = (__u16)(page ? PAGE_FRAG_CACHE_MAX_SIZE : PAGE_SIZE); + } if (unlikely(!page)) page = alloc_pages_node(NUMA_NO_NODE, gfp, 0); @@ -4870,10 +4875,10 @@ void *__page_frag_alloc_align(struct page_frag_cache *nc, if (!page) return NULL; -#if (PAGE_SIZE < PAGE_FRAG_CACHE_MAX_SIZE) /* if size can vary use size else just use PAGE_SIZE */ - size = nc->size; -#endif + if (PAGE_SIZE < PAGE_FRAG_CACHE_MAX_SIZE) + size = nc->size; + /* Even if we own the page, we do not use atomic_set(). * This would break get_page_unless_zero() users. */ @@ -4897,10 +4902,10 @@ void *__page_frag_alloc_align(struct page_frag_cache *nc, goto refill; } -#if (PAGE_SIZE < PAGE_FRAG_CACHE_MAX_SIZE) /* if size can vary use size else just use PAGE_SIZE */ - size = nc->size; -#endif + if (PAGE_SIZE < PAGE_FRAG_CACHE_MAX_SIZE) + size = nc->size; + /* OK, page count is 0, we can safely set it */ set_page_count(page, PAGE_FRAG_CACHE_MAX_SIZE + 1);