From patchwork Mon Oct 14 10:58:09 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13834803 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 1EECCD16245 for ; Mon, 14 Oct 2024 11:26:30 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=O6/Jx7UMSGM5GsZJBCvRZxEmHJTJyTPnGhAVFa7W4Fs=; b=eq0hQ0/HaEbZ0VZxp23lYxjqQo rkEayx9zGw+0PiXqieh8zqG980oZ7mKyOK4VIU3qt9HRdiNi4tzXNs1nksnPUNXMATSYvQDRliKav pCbVRmK7fzSPRxrnc8ftLa5q4fwfEwpEfBBv94AiFdmNfH40Jv+2Uqje8JcNoDRqZI6HVH56qS0Zh mIsSBHxifRRhj6g9EpBnfSPns7NZJZh9wM0O+ZZ34PpjKoSdagIrtyu9bC5BBbkfWtJ5XEgkO5xKQ sXhAGb7D6WO5ycJxvr+RJF6IAEOcvLgfZ4rFuIin5Iy9K08g0j6eaMYOrvmW/39r6Nj0AOOopqB3k OqUNumXw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1t0JDN-00000004vEa-0gJN; Mon, 14 Oct 2024 11:26:17 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1t0InV-00000004nN0-0yw7 for linux-arm-kernel@lists.infradead.org; Mon, 14 Oct 2024 10:59:35 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 443921684; Mon, 14 Oct 2024 04:00:02 -0700 (PDT) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.27]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 9E5CD3F51B; Mon, 14 Oct 2024 03:59:29 -0700 (PDT) From: Ryan Roberts To: Andrew Morton , Anshuman Khandual , Ard Biesheuvel , Arnd Bergmann , Catalin Marinas , Christoph Lameter , David Hildenbrand , Dennis Zhou , Greg Marsden , Ivan Ivanov , Kalesh Singh , Marc Zyngier , Mark Rutland , Matthias Brugger , Miroslav Benes , Tejun Heo , Will Deacon Cc: Ryan Roberts , linux-arch@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [RFC PATCH v1 02/57] vmlinux: Align to PAGE_SIZE_MAX Date: Mon, 14 Oct 2024 11:58:09 +0100 Message-ID: <20241014105912.3207374-2-ryan.roberts@arm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20241014105912.3207374-1-ryan.roberts@arm.com> References: <20241014105514.3206191-1-ryan.roberts@arm.com> <20241014105912.3207374-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241014_035933_396368_A532CBF9 X-CRM114-Status: GOOD ( 14.20 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Increase alignment of structures requiring at least PAGE_SIZE alignment to PAGE_SIZE_MAX. For compile-time PAGE_SIZE, PAGE_SIZE_MAX == PAGE_SIZE so there is no change. For boot-time PAGE_SIZE, PAGE_SIZE_MAX is the largest selectable page size. Signed-off-by: Ryan Roberts --- ***NOTE*** Any confused maintainers may want to read the cover note here for context: https://lore.kernel.org/all/20241014105514.3206191-1-ryan.roberts@arm.com/ include/asm-generic/vmlinux.lds.h | 32 +++++++++++++++---------------- include/linux/linkage.h | 4 ++-- include/linux/percpu-defs.h | 4 ++-- 3 files changed, 20 insertions(+), 20 deletions(-) diff --git a/include/asm-generic/vmlinux.lds.h b/include/asm-generic/vmlinux.lds.h index 1ae44793132a8..5727f883001bb 100644 --- a/include/asm-generic/vmlinux.lds.h +++ b/include/asm-generic/vmlinux.lds.h @@ -13,7 +13,7 @@ * . = START; * __init_begin = .; * HEAD_TEXT_SECTION - * INIT_TEXT_SECTION(PAGE_SIZE) + * INIT_TEXT_SECTION(PAGE_SIZE_MAX) * INIT_DATA_SECTION(...) * PERCPU_SECTION(CACHELINE_SIZE) * __init_end = .; @@ -23,7 +23,7 @@ * _etext = .; * * _sdata = .; - * RO_DATA(PAGE_SIZE) + * RO_DATA(PAGE_SIZE_MAX) * RW_DATA(...) * _edata = .; * @@ -371,10 +371,10 @@ * Data section helpers */ #define NOSAVE_DATA \ - . = ALIGN(PAGE_SIZE); \ + . = ALIGN(PAGE_SIZE_MAX); \ __nosave_begin = .; \ *(.data..nosave) \ - . = ALIGN(PAGE_SIZE); \ + . = ALIGN(PAGE_SIZE_MAX); \ __nosave_end = .; #define PAGE_ALIGNED_DATA(page_align) \ @@ -733,9 +733,9 @@ . = ALIGN(bss_align); \ .bss : AT(ADDR(.bss) - LOAD_OFFSET) { \ BSS_FIRST_SECTIONS \ - . = ALIGN(PAGE_SIZE); \ + . = ALIGN(PAGE_SIZE_MAX); \ *(.bss..page_aligned) \ - . = ALIGN(PAGE_SIZE); \ + . = ALIGN(PAGE_SIZE_MAX); \ *(.dynbss) \ *(BSS_MAIN) \ *(COMMON) \ @@ -950,9 +950,9 @@ */ #ifdef CONFIG_AMD_MEM_ENCRYPT #define PERCPU_DECRYPTED_SECTION \ - . = ALIGN(PAGE_SIZE); \ + . = ALIGN(PAGE_SIZE_MAX); \ *(.data..percpu..decrypted) \ - . = ALIGN(PAGE_SIZE); + . = ALIGN(PAGE_SIZE_MAX); #else #define PERCPU_DECRYPTED_SECTION #endif @@ -1030,7 +1030,7 @@ #define PERCPU_INPUT(cacheline) \ __per_cpu_start = .; \ *(.data..percpu..first) \ - . = ALIGN(PAGE_SIZE); \ + . = ALIGN(PAGE_SIZE_MAX); \ *(.data..percpu..page_aligned) \ . = ALIGN(cacheline); \ *(.data..percpu..read_mostly) \ @@ -1075,16 +1075,16 @@ * PERCPU_SECTION - define output section for percpu area, simple version * @cacheline: cacheline size * - * Align to PAGE_SIZE and outputs output section for percpu area. This + * Align to PAGE_SIZE_MAX and outputs output section for percpu area. This * macro doesn't manipulate @vaddr or @phdr and __per_cpu_load and * __per_cpu_start will be identical. * - * This macro is equivalent to ALIGN(PAGE_SIZE); PERCPU_VADDR(@cacheline,,) + * This macro is equivalent to ALIGN(PAGE_SIZE_MAX); PERCPU_VADDR(@cacheline,,) * except that __per_cpu_load is defined as a relative symbol against * .data..percpu which is required for relocatable x86_32 configuration. */ #define PERCPU_SECTION(cacheline) \ - . = ALIGN(PAGE_SIZE); \ + . = ALIGN(PAGE_SIZE_MAX); \ .data..percpu : AT(ADDR(.data..percpu) - LOAD_OFFSET) { \ __per_cpu_load = .; \ PERCPU_INPUT(cacheline) \ @@ -1102,15 +1102,15 @@ * All sections are combined in a single .data section. * The sections following CONSTRUCTORS are arranged so their * typical alignment matches. - * A cacheline is typical/always less than a PAGE_SIZE so + * A cacheline is typical/always less than a PAGE_SIZE_MAX so * the sections that has this restriction (or similar) - * is located before the ones requiring PAGE_SIZE alignment. - * NOSAVE_DATA starts and ends with a PAGE_SIZE alignment which + * is located before the ones requiring PAGE_SIZE_MAX alignment. + * NOSAVE_DATA starts and ends with a PAGE_SIZE_MAX alignment which * matches the requirement of PAGE_ALIGNED_DATA. * * use 0 as page_align if page_aligned data is not used */ #define RW_DATA(cacheline, pagealigned, inittask) \ - . = ALIGN(PAGE_SIZE); \ + . = ALIGN(PAGE_SIZE_MAX); \ .data : AT(ADDR(.data) - LOAD_OFFSET) { \ INIT_TASK_DATA(inittask) \ NOSAVE_DATA \ diff --git a/include/linux/linkage.h b/include/linux/linkage.h index 5c8865bb59d91..68aa9775fce51 100644 --- a/include/linux/linkage.h +++ b/include/linux/linkage.h @@ -36,8 +36,8 @@ __stringify(name)) #endif -#define __page_aligned_data __section(".data..page_aligned") __aligned(PAGE_SIZE) -#define __page_aligned_bss __section(".bss..page_aligned") __aligned(PAGE_SIZE) +#define __page_aligned_data __section(".data..page_aligned") __aligned(PAGE_SIZE_MAX) +#define __page_aligned_bss __section(".bss..page_aligned") __aligned(PAGE_SIZE_MAX) /* * For assembly routines. diff --git a/include/linux/percpu-defs.h b/include/linux/percpu-defs.h index 8efce7414fad6..89c7f430015ba 100644 --- a/include/linux/percpu-defs.h +++ b/include/linux/percpu-defs.h @@ -156,11 +156,11 @@ */ #define DECLARE_PER_CPU_PAGE_ALIGNED(type, name) \ DECLARE_PER_CPU_SECTION(type, name, "..page_aligned") \ - __aligned(PAGE_SIZE) + __aligned(PAGE_SIZE_MAX) #define DEFINE_PER_CPU_PAGE_ALIGNED(type, name) \ DEFINE_PER_CPU_SECTION(type, name, "..page_aligned") \ - __aligned(PAGE_SIZE) + __aligned(PAGE_SIZE_MAX) /* * Declaration/definition used for per-CPU variables that must be read mostly. From patchwork Mon Oct 14 10:58:10 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13834804 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id C9DFED16243 for ; Mon, 14 Oct 2024 11:27:54 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=VpicSstlstrg5d6N29OH0psgAbIQb3i7oiCjbYwYKQQ=; b=2keFX9hMA8sIIVW81i7hlUgUyX 1CJQctaYjqadJ0Vr++IteWmJp3O8fBsLqLFIavnCC75k7VD0qJhVH59GonHgzQW+DoRWhGY26iuer 8sTywHbRBDJdWxnWXgDWrCv0GUrFDb4SbsITZX1TBkNxBotUYcGuWe15JmylqzPOMHQmkEWPOflFM e8E5V4sgryelK21ppzy6JxvBWurFVoAmDfWqqTFy9sc6Jg9Y/avI65OaePvkhLhtIWe1RZ9t7obz2 tL/IGwzUblOWj1vUdT/Bz7cp+oowsUYSD2GUmlRLcmUnlyKMFIsIVcQQky4qHw6z0b0oG/D+N1HMM nX6ubPoA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1t0JEk-00000004vUF-1flD; Mon, 14 Oct 2024 11:27:42 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1t0InY-00000004nOM-1at9 for linux-arm-kernel@lists.infradead.org; Mon, 14 Oct 2024 10:59:39 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 8FB151424; Mon, 14 Oct 2024 04:00:05 -0700 (PDT) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.27]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id E9C283F51B; Mon, 14 Oct 2024 03:59:32 -0700 (PDT) From: Ryan Roberts To: Andrew Morton , Anshuman Khandual , Ard Biesheuvel , Catalin Marinas , David Hildenbrand , Greg Marsden , Ivan Ivanov , Johannes Weiner , Kalesh Singh , Marc Zyngier , Mark Rutland , Matthias Brugger , Michal Hocko , Miroslav Benes , Roman Gushchin , Shakeel Butt , Will Deacon Cc: Ryan Roberts , cgroups@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [RFC PATCH v1 03/57] mm/memcontrol: Fix seq_buf size to save memory when PAGE_SIZE is large Date: Mon, 14 Oct 2024 11:58:10 +0100 Message-ID: <20241014105912.3207374-3-ryan.roberts@arm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20241014105912.3207374-1-ryan.roberts@arm.com> References: <20241014105514.3206191-1-ryan.roberts@arm.com> <20241014105912.3207374-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241014_035937_080818_541181A8 X-CRM114-Status: GOOD ( 15.52 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Previously the seq_buf used for accumulating the memory.stat output was sized at PAGE_SIZE. But the amount of output is invariant to PAGE_SIZE; If 4K is enough on a 4K page system, then it should also be enough on a 64K page system, so we can save 60K om the static buffer used in mem_cgroup_print_oom_meminfo(). Let's make it so. This also has the beneficial side effect of removing a place in the code that assumed PAGE_SIZE is a compile-time constant. So this helps our quest towards supporting boot-time page size selection. Signed-off-by: Ryan Roberts Acked-by: Johannes Weiner Acked-by: Shakeel Butt Acked-by: Michal Hocko Acked-by: Roman Gushchin --- ***NOTE*** Any confused maintainers may want to read the cover note here for context: https://lore.kernel.org/all/20241014105514.3206191-1-ryan.roberts@arm.com/ mm/memcontrol.c | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index d563fb515766b..c5f9195f76c65 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -95,6 +95,7 @@ static DECLARE_WAIT_QUEUE_HEAD(memcg_cgwb_frn_waitq); #define THRESHOLDS_EVENTS_TARGET 128 #define SOFTLIMIT_EVENTS_TARGET 1024 +#define SEQ_BUF_SIZE SZ_4K static inline bool task_is_dying(void) { @@ -1519,7 +1520,7 @@ void mem_cgroup_print_oom_context(struct mem_cgroup *memcg, struct task_struct * void mem_cgroup_print_oom_meminfo(struct mem_cgroup *memcg) { /* Use static buffer, for the caller is holding oom_lock. */ - static char buf[PAGE_SIZE]; + static char buf[SEQ_BUF_SIZE]; struct seq_buf s; lockdep_assert_held(&oom_lock); @@ -1545,7 +1546,7 @@ void mem_cgroup_print_oom_meminfo(struct mem_cgroup *memcg) pr_info("Memory cgroup stats for "); pr_cont_cgroup_path(memcg->css.cgroup); pr_cont(":"); - seq_buf_init(&s, buf, sizeof(buf)); + seq_buf_init(&s, buf, SEQ_BUF_SIZE); memory_stat_format(memcg, &s); seq_buf_do_printk(&s, KERN_INFO); } @@ -4158,12 +4159,12 @@ static int memory_events_local_show(struct seq_file *m, void *v) int memory_stat_show(struct seq_file *m, void *v) { struct mem_cgroup *memcg = mem_cgroup_from_seq(m); - char *buf = kmalloc(PAGE_SIZE, GFP_KERNEL); + char *buf = kmalloc(SEQ_BUF_SIZE, GFP_KERNEL); struct seq_buf s; if (!buf) return -ENOMEM; - seq_buf_init(&s, buf, PAGE_SIZE); + seq_buf_init(&s, buf, SEQ_BUF_SIZE); memory_stat_format(memcg, &s); seq_puts(m, buf); kfree(buf); From patchwork Mon Oct 14 10:58:11 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13834805 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id CCBA7D16246 for ; Mon, 14 Oct 2024 11:29:17 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=kYeTTVqmvopusG+SkQLqUutllx+fKzmiPfJpsd8f/kQ=; b=DrITu5yuKHaGjbK4BK29zXSfJW OESD/tNgHEPTTMUQxQpuUgIVHiuxtVA0Cnvzs2vvm2jtwIDI3KUlTiCutfqzdHs8skDNfk0MjS1HE gMfknCPqu/xDIgJDInpI8fvQTgQBaHRMq2f+KGdBkFxxnaOuYUX8+qApPfh55ABC1Ra/Q8Vjl19qb 5byaeN4SuWCfM0zk4rdJNTPafrrbg/5UNmc2hYeqxnsTIPXI4GIwYw9Y0k0rJ08TpJYkWm3s5/azy 1NBOR6hJ5TO7XyTdL1r681LYejHopMa7kPpCMetC3ZflmRi1pirFJN8G+FSsTvElN577Ta/U7MQ2x 3DJ/0mmg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1t0JG5-00000004vfg-3hgg; Mon, 14 Oct 2024 11:29:05 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1t0Inb-00000004nQ8-04g3 for linux-arm-kernel@lists.infradead.org; Mon, 14 Oct 2024 10:59:41 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 405271688; Mon, 14 Oct 2024 04:00:08 -0700 (PDT) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.27]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 403163F51B; Mon, 14 Oct 2024 03:59:36 -0700 (PDT) From: Ryan Roberts To: Andrew Morton , Anshuman Khandual , Ard Biesheuvel , Catalin Marinas , David Hildenbrand , Greg Marsden , Ivan Ivanov , Kalesh Singh , Marc Zyngier , Mark Rutland , Matthias Brugger , Miroslav Benes , Will Deacon Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [RFC PATCH v1 04/57] mm/page_alloc: Make page_frag_cache boot-time page size compatible Date: Mon, 14 Oct 2024 11:58:11 +0100 Message-ID: <20241014105912.3207374-4-ryan.roberts@arm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20241014105912.3207374-1-ryan.roberts@arm.com> References: <20241014105514.3206191-1-ryan.roberts@arm.com> <20241014105912.3207374-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241014_035939_286634_E3D4692B X-CRM114-Status: GOOD ( 16.95 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org "struct page_frag_cache" has some optimizations that depend on page size. Let's refactor it a bit so that those optimizations can be determined at run-time for the case where page size is a boot-time parameter. For compile-time page size, the compiler should dead code strip and the result is very similar to before. One wrinkle is that we don't know if we need the size member until runtime. So remove the ifdeffery and always define offset as u32 (needed if PAGE_SIZE is >= 64K) and size as u16 (only used when PAGE_SIZE <= 32K). We move the members around a bit so that the overall size of the struct remains the same; 24 bytes for 64-bit and 16 bytes on 32 bit. Signed-off-by: Ryan Roberts page_alloc Acked-by: Vlastimil Babka --- ***NOTE*** Any confused maintainers may want to read the cover note here for context: https://lore.kernel.org/all/20241014105514.3206191-1-ryan.roberts@arm.com/ include/linux/mm_types.h | 13 ++++++------- mm/page_alloc.c | 31 ++++++++++++++++++------------- 2 files changed, 24 insertions(+), 20 deletions(-) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 4854249792545..0844ed7cfaa53 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -544,16 +544,15 @@ static inline void *folio_get_private(struct folio *folio) struct page_frag_cache { void * va; -#if (PAGE_SIZE < PAGE_FRAG_CACHE_MAX_SIZE) - __u16 offset; - __u16 size; -#else - __u32 offset; -#endif /* we maintain a pagecount bias, so that we dont dirty cache line * containing page->_refcount every time we allocate a fragment. */ - unsigned int pagecnt_bias; + unsigned int pagecnt_bias; + __u32 offset; + /* size only used when PAGE_SIZE < PAGE_FRAG_CACHE_MAX_SIZE, in which + * case PAGE_FRAG_CACHE_MAX_SIZE is 32K and 16 bits is sufficient. + */ + __u16 size; bool pfmemalloc; }; diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 91ace8ca97e21..8678103b1b396 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -4822,13 +4822,18 @@ static struct page *__page_frag_cache_refill(struct page_frag_cache *nc, struct page *page = NULL; gfp_t gfp = gfp_mask; -#if (PAGE_SIZE < PAGE_FRAG_CACHE_MAX_SIZE) - gfp_mask = (gfp_mask & ~__GFP_DIRECT_RECLAIM) | __GFP_COMP | - __GFP_NOWARN | __GFP_NORETRY | __GFP_NOMEMALLOC; - page = alloc_pages_node(NUMA_NO_NODE, gfp_mask, - PAGE_FRAG_CACHE_MAX_ORDER); - nc->size = page ? PAGE_FRAG_CACHE_MAX_SIZE : PAGE_SIZE; -#endif + if (PAGE_SIZE < PAGE_FRAG_CACHE_MAX_SIZE) { + gfp_mask = (gfp_mask & ~__GFP_DIRECT_RECLAIM) | __GFP_COMP | + __GFP_NOWARN | __GFP_NORETRY | __GFP_NOMEMALLOC; + page = alloc_pages_node(NUMA_NO_NODE, gfp_mask, + PAGE_FRAG_CACHE_MAX_ORDER); + /* + * Cast to silence warning due to 16-bit nc->size. Not real + * because PAGE_SIZE only less than PAGE_FRAG_CACHE_MAX_SIZE + * when PAGE_FRAG_CACHE_MAX_SIZE is 32K. + */ + nc->size = (__u16)(page ? PAGE_FRAG_CACHE_MAX_SIZE : PAGE_SIZE); + } if (unlikely(!page)) page = alloc_pages_node(NUMA_NO_NODE, gfp, 0); @@ -4870,10 +4875,10 @@ void *__page_frag_alloc_align(struct page_frag_cache *nc, if (!page) return NULL; -#if (PAGE_SIZE < PAGE_FRAG_CACHE_MAX_SIZE) /* if size can vary use size else just use PAGE_SIZE */ - size = nc->size; -#endif + if (PAGE_SIZE < PAGE_FRAG_CACHE_MAX_SIZE) + size = nc->size; + /* Even if we own the page, we do not use atomic_set(). * This would break get_page_unless_zero() users. */ @@ -4897,10 +4902,10 @@ void *__page_frag_alloc_align(struct page_frag_cache *nc, goto refill; } -#if (PAGE_SIZE < PAGE_FRAG_CACHE_MAX_SIZE) /* if size can vary use size else just use PAGE_SIZE */ - size = nc->size; -#endif + if (PAGE_SIZE < PAGE_FRAG_CACHE_MAX_SIZE) + size = nc->size; + /* OK, page count is 0, we can safely set it */ set_page_count(page, PAGE_FRAG_CACHE_MAX_SIZE + 1); From patchwork Mon Oct 14 10:58:12 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13834806 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id CF5EAD16243 for ; Mon, 14 Oct 2024 11:30:49 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=qPt13oBtsQSK+UGOOEWOc/EwkFI9q79bukNoe+ZJFuA=; b=FszNW5xcqvVlb6y4siwlYimYoD K08yfPgLGss9nz3zPSgLXWU44l8YoqQpCgguYA6Nro3WbC2umHqrUqpzTtsZgrPomMq/wWzPcrA+I vJ+WUJLtWqQdGukzJ6724qGYEjTVINCeIysPl0Boj/g9zvW07lelay/C3q5Fzo4YwtsGvUBif0kwg hWGIzr7VIW2KwInCAOtVGvHGHyA7zL96UVzqOVSXlxfn5r89PZnD5OumytJ5bqawOYQKEdgF9mqmG Xff7t1F8ijB49lG4ONZymtWMzRhcdVIN8KqzthG/aJopd9x8fNhn8sUJRcVPfkh+eqeA9FS8eysoj sopzihfg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1t0JHR-00000004vsY-48IM; Mon, 14 Oct 2024 11:30:29 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1t0Ine-00000004nRv-2Z8f for linux-arm-kernel@lists.infradead.org; Mon, 14 Oct 2024 10:59:44 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id E3E091684; Mon, 14 Oct 2024 04:00:10 -0700 (PDT) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.27]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id E47583F51B; Mon, 14 Oct 2024 03:59:38 -0700 (PDT) From: Ryan Roberts To: Andrew Morton , Anshuman Khandual , Ard Biesheuvel , Catalin Marinas , David Hildenbrand , Greg Marsden , Ivan Ivanov , Kalesh Singh , Marc Zyngier , Mark Rutland , Matthias Brugger , Miroslav Benes , Will Deacon Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [RFC PATCH v1 05/57] mm: Avoid split pmd ptl if pmd level is run-time folded Date: Mon, 14 Oct 2024 11:58:12 +0100 Message-ID: <20241014105912.3207374-5-ryan.roberts@arm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20241014105912.3207374-1-ryan.roberts@arm.com> References: <20241014105514.3206191-1-ryan.roberts@arm.com> <20241014105912.3207374-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241014_035942_857655_CD690FAB X-CRM114-Status: GOOD ( 19.78 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org If there are only 2 levels of translation, the first level (pgd) may not be an entire page and so does not have a ptdesc backing it (this may be true on arm64 depending on the VA size and page size). Even if it is an entire page and does therefore have an entire ptdesc, pagetable_pmd_ctor() won't be called for the ptdesc (since it's a pgd not pmd table) and so the per-ptdec ptl fields won't be initialised. To date this has been fine; the arch knows at compile time if it needs to fold the pmd level and in this case does not select CONFIG_ARCH_ENABLE_SPLIT_PMD_PTLOCK. However, if the number of levels are not known at compile time (as is the case for boot-time page size selection), we want to be able to choose at boot whether to use split pmd ptls in the pmd's ptdesc or simply fall back to the lock in the mm_struct. So let's make that change; when CONFIG_ARCH_ENABLE_SPLIT_PMD_PTLOCK is selected, determine if it should be used at run-time based on mm_pmd_folded(). This sets us up for arm64 to support boot-time page size selection. Signed-off-by: Ryan Roberts --- ***NOTE*** Any confused maintainers may want to read the cover note here for context: https://lore.kernel.org/all/20241014105514.3206191-1-ryan.roberts@arm.com/ include/linux/mm.h | 15 ++++++++++++++- include/linux/mm_types.h | 2 +- kernel/fork.c | 4 ++-- 3 files changed, 17 insertions(+), 4 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 1470736017168..09a840517c23a 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -3037,6 +3037,8 @@ static inline struct ptdesc *pmd_ptdesc(pmd_t *pmd) static inline spinlock_t *pmd_lockptr(struct mm_struct *mm, pmd_t *pmd) { + if (mm_pmd_folded(mm)) + return &mm->page_table_lock; return ptlock_ptr(pmd_ptdesc(pmd)); } @@ -3056,7 +3058,18 @@ static inline void pmd_ptlock_free(struct ptdesc *ptdesc) ptlock_free(ptdesc); } -#define pmd_huge_pte(mm, pmd) (pmd_ptdesc(pmd)->pmd_huge_pte) +static inline pgtable_t *__pmd_huge_pte(struct mm_struct *mm, pmd_t *pmd) +{ +#ifdef CONFIG_TRANSPARENT_HUGEPAGE + if (mm_pmd_folded(mm)) + return &mm->pmd_huge_pte; + return &pmd_ptdesc(pmd)->pmd_huge_pte; +#else + return NULL; +#endif +} + +#define pmd_huge_pte(mm, pmd) (*__pmd_huge_pte(mm, pmd)) #else diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 0844ed7cfaa53..87dc6de7b7baf 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -946,7 +946,7 @@ struct mm_struct { #ifdef CONFIG_MMU_NOTIFIER struct mmu_notifier_subscriptions *notifier_subscriptions; #endif -#if defined(CONFIG_TRANSPARENT_HUGEPAGE) && !USE_SPLIT_PMD_PTLOCKS +#if defined(CONFIG_TRANSPARENT_HUGEPAGE) pgtable_t pmd_huge_pte; /* protected by page_table_lock */ #endif #ifdef CONFIG_NUMA_BALANCING diff --git a/kernel/fork.c b/kernel/fork.c index cc760491f2012..ea472566d4fcc 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -832,7 +832,7 @@ static void check_mm(struct mm_struct *mm) pr_alert("BUG: non-zero pgtables_bytes on freeing mm: %ld\n", mm_pgtables_bytes(mm)); -#if defined(CONFIG_TRANSPARENT_HUGEPAGE) && !USE_SPLIT_PMD_PTLOCKS +#if defined(CONFIG_TRANSPARENT_HUGEPAGE) VM_BUG_ON_MM(mm->pmd_huge_pte, mm); #endif } @@ -1276,7 +1276,7 @@ static struct mm_struct *mm_init(struct mm_struct *mm, struct task_struct *p, RCU_INIT_POINTER(mm->exe_file, NULL); mmu_notifier_subscriptions_init(mm); init_tlb_flush_pending(mm); -#if defined(CONFIG_TRANSPARENT_HUGEPAGE) && !USE_SPLIT_PMD_PTLOCKS +#if defined(CONFIG_TRANSPARENT_HUGEPAGE) mm->pmd_huge_pte = NULL; #endif mm_init_uprobes_state(mm); From patchwork Mon Oct 14 10:58:13 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13834807 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id D7614D16247 for ; Mon, 14 Oct 2024 11:32:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=cG9KRB9iAdom5fH60FL/Vpswjxk6D87RpKBF0kA0w7Q=; b=FYKkiLBMvrP8mcFjVP7S4F0rZj ViCLeLFVH3zbkN8r8JZ6c3zM0/J1YeyCZrZ5t/PD1SLora7KiEIW6M2T0gYVP+1SYuDc4XJiq585u gGcdLIe9SwVPkXjnvp05pzf8mzDQdSAqFRjjaQD7uf4rG4rowxQ5QwILRQ9kCmAWTkZNVhIK0ZiF4 59KFLiVOgms8c4GAQR1RpRjobBNGBzPRi0/Q6NwTCun+phgRtihOlkoGIxarouZ3o6lZ/J1YwAKP1 ROuzeBLQbSLuh0wjBRSjNIwYE+l7RNeraxA+Y9ZuP1sf9IUbQolhIILBH+YSliT6EUyouiV6ljf+6 WgZtN+Qw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1t0JIn-00000004w32-3fWZ; Mon, 14 Oct 2024 11:31:53 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1t0Ini-00000004nUB-16Qe; Mon, 14 Oct 2024 10:59:49 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 7DC7F1424; Mon, 14 Oct 2024 04:00:15 -0700 (PDT) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.27]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 979593F51B; Mon, 14 Oct 2024 03:59:41 -0700 (PDT) From: Ryan Roberts To: Andrew Morton , Anshuman Khandual , Ard Biesheuvel , Catalin Marinas , Christoph Lameter , David Hildenbrand , David Rientjes , Greg Marsden , Ivan Ivanov , Johannes Weiner , Joonsoo Kim , Kalesh Singh , Marc Zyngier , Mark Rutland , Matthias Brugger , Michal Hocko , Miquel Raynal , Miroslav Benes , Pekka Enberg , Richard Weinberger , Shakeel Butt , Vignesh Raghavendra , Vlastimil Babka , Will Deacon Cc: Ryan Roberts , cgroups@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-mtd@lists.infradead.org Subject: [RFC PATCH v1 06/57] mm: Remove PAGE_SIZE compile-time constant assumption Date: Mon, 14 Oct 2024 11:58:13 +0100 Message-ID: <20241014105912.3207374-6-ryan.roberts@arm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20241014105912.3207374-1-ryan.roberts@arm.com> References: <20241014105514.3206191-1-ryan.roberts@arm.com> <20241014105912.3207374-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241014_035946_449607_16743C9E X-CRM114-Status: GOOD ( 25.35 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org To prepare for supporting boot-time page size selection, refactor code to remove assumptions about PAGE_SIZE being compile-time constant. Code intended to be equivalent when compile-time page size is active. Refactor "struct vmap_block" to use a flexible array for used_mmap since VMAP_BBMAP_BITS is not a compile time constant for the boot-time page size case. Update various BUILD_BUG_ON() instances to check against appropriate page size limit. Re-define "union swap_header" so that it's no longer exactly page-sized. Instead define a flexible "magic" array with a define which tells the offset to where the magic signature begins. Consider page size limit in some CPP condditionals. Wrap global variables that are initialized with PAGE_SIZE derived values using DEFINE_GLOBAL_PAGE_SIZE_VAR() so their initialization can be deferred for boot-time page size builds. Signed-off-by: Ryan Roberts --- ***NOTE*** Any confused maintainers may want to read the cover note here for context: https://lore.kernel.org/all/20241014105514.3206191-1-ryan.roberts@arm.com/ drivers/mtd/mtdswap.c | 4 ++-- include/linux/mm.h | 2 +- include/linux/mm_types_task.h | 2 +- include/linux/mmzone.h | 3 ++- include/linux/slab.h | 7 ++++--- include/linux/swap.h | 17 ++++++++++++----- include/linux/swapops.h | 6 +++++- mm/memcontrol.c | 2 +- mm/memory.c | 4 ++-- mm/mmap.c | 2 +- mm/page-writeback.c | 2 +- mm/slub.c | 2 +- mm/sparse.c | 2 +- mm/swapfile.c | 2 +- mm/vmalloc.c | 7 ++++--- 15 files changed, 39 insertions(+), 25 deletions(-) diff --git a/drivers/mtd/mtdswap.c b/drivers/mtd/mtdswap.c index 680366616da24..7412a32708114 100644 --- a/drivers/mtd/mtdswap.c +++ b/drivers/mtd/mtdswap.c @@ -1062,13 +1062,13 @@ static int mtdswap_auto_header(struct mtdswap_dev *d, char *buf) { union swap_header *hd = (union swap_header *)(buf); - memset(buf, 0, PAGE_SIZE - 10); + memset(buf, 0, SWAP_HEADER_MAGIC); hd->info.version = 1; hd->info.last_page = d->mbd_dev->size - 1; hd->info.nr_badpages = 0; - memcpy(buf + PAGE_SIZE - 10, "SWAPSPACE2", 10); + memcpy(buf + SWAP_HEADER_MAGIC, "SWAPSPACE2", 10); return 0; } diff --git a/include/linux/mm.h b/include/linux/mm.h index 09a840517c23a..49c2078354e6e 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2927,7 +2927,7 @@ static inline spinlock_t *pte_lockptr(struct mm_struct *mm, pmd_t *pmd) static inline spinlock_t *ptep_lockptr(struct mm_struct *mm, pte_t *pte) { BUILD_BUG_ON(IS_ENABLED(CONFIG_HIGHPTE)); - BUILD_BUG_ON(MAX_PTRS_PER_PTE * sizeof(pte_t) > PAGE_SIZE); + BUILD_BUG_ON(MAX_PTRS_PER_PTE * sizeof(pte_t) > PAGE_SIZE_MAX); return ptlock_ptr(virt_to_ptdesc(pte)); } diff --git a/include/linux/mm_types_task.h b/include/linux/mm_types_task.h index a2f6179b672b8..c356897d5f41c 100644 --- a/include/linux/mm_types_task.h +++ b/include/linux/mm_types_task.h @@ -37,7 +37,7 @@ struct page; struct page_frag { struct page *page; -#if (BITS_PER_LONG > 32) || (PAGE_SIZE >= 65536) +#if (BITS_PER_LONG > 32) || (PAGE_SIZE_MAX >= 65536) __u32 offset; __u32 size; #else diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index 1dc6248feb832..cd58034b82c81 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -1744,6 +1744,7 @@ static inline bool movable_only_nodes(nodemask_t *nodes) */ #define PA_SECTION_SHIFT (SECTION_SIZE_BITS) #define PFN_SECTION_SHIFT (SECTION_SIZE_BITS - PAGE_SHIFT) +#define PFN_SECTION_SHIFT_MIN (SECTION_SIZE_BITS - PAGE_SHIFT_MAX) #define NR_MEM_SECTIONS (1UL << SECTIONS_SHIFT) @@ -1753,7 +1754,7 @@ static inline bool movable_only_nodes(nodemask_t *nodes) #define SECTION_BLOCKFLAGS_BITS \ ((1UL << (PFN_SECTION_SHIFT - pageblock_order)) * NR_PAGEBLOCK_BITS) -#if (MAX_PAGE_ORDER + PAGE_SHIFT) > SECTION_SIZE_BITS +#if (MAX_PAGE_ORDER + PAGE_SHIFT_MAX) > SECTION_SIZE_BITS #error Allocator MAX_PAGE_ORDER exceeds SECTION_SIZE #endif diff --git a/include/linux/slab.h b/include/linux/slab.h index eb2bf46291576..11c6ff3a12579 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -347,7 +347,7 @@ static inline unsigned int arch_slab_minalign(void) */ #define __assume_kmalloc_alignment __assume_aligned(ARCH_KMALLOC_MINALIGN) #define __assume_slab_alignment __assume_aligned(ARCH_SLAB_MINALIGN) -#define __assume_page_alignment __assume_aligned(PAGE_SIZE) +#define __assume_page_alignment __assume_aligned(PAGE_SIZE_MIN) /* * Kmalloc array related definitions @@ -358,6 +358,7 @@ static inline unsigned int arch_slab_minalign(void) * (PAGE_SIZE*2). Larger requests are passed to the page allocator. */ #define KMALLOC_SHIFT_HIGH (PAGE_SHIFT + 1) +#define KMALLOC_SHIFT_HIGH_MAX (PAGE_SHIFT_MAX + 1) #define KMALLOC_SHIFT_MAX (MAX_PAGE_ORDER + PAGE_SHIFT) #ifndef KMALLOC_SHIFT_LOW #define KMALLOC_SHIFT_LOW 3 @@ -426,7 +427,7 @@ enum kmalloc_cache_type { NR_KMALLOC_TYPES }; -typedef struct kmem_cache * kmem_buckets[KMALLOC_SHIFT_HIGH + 1]; +typedef struct kmem_cache * kmem_buckets[KMALLOC_SHIFT_HIGH_MAX + 1]; extern kmem_buckets kmalloc_caches[NR_KMALLOC_TYPES]; @@ -524,7 +525,7 @@ static __always_inline unsigned int __kmalloc_index(size_t size, /* Will never be reached. Needed because the compiler may complain */ return -1; } -static_assert(PAGE_SHIFT <= 20); +static_assert(PAGE_SHIFT_MAX <= 20); #define kmalloc_index(s) __kmalloc_index(s, true) #include diff --git a/include/linux/swap.h b/include/linux/swap.h index ba7ea95d1c57a..e85df0332979f 100644 --- a/include/linux/swap.h +++ b/include/linux/swap.h @@ -132,10 +132,17 @@ static inline int current_is_kswapd(void) * bootbits... */ union swap_header { - struct { - char reserved[PAGE_SIZE - 10]; - char magic[10]; /* SWAP-SPACE or SWAPSPACE2 */ - } magic; + /* + * Exists conceptually, but since PAGE_SIZE may not be known at compile + * time, we must access through pointer arithmetic at run time. + * + * struct { + * char reserved[PAGE_SIZE - 10]; + * char magic[10]; SWAP-SPACE or SWAPSPACE2 + * } magic; + */ +#define SWAP_HEADER_MAGIC (PAGE_SIZE - 10) + char magic[1]; struct { char bootbits[1024]; /* Space for disklabel etc. */ __u32 version; @@ -201,7 +208,7 @@ struct swap_extent { * Max bad pages in the new format.. */ #define MAX_SWAP_BADPAGES \ - ((offsetof(union swap_header, magic.magic) - \ + ((SWAP_HEADER_MAGIC - \ offsetof(union swap_header, info.badpages)) / sizeof(int)) enum { diff --git a/include/linux/swapops.h b/include/linux/swapops.h index cb468e418ea11..890fe6a3e6702 100644 --- a/include/linux/swapops.h +++ b/include/linux/swapops.h @@ -34,10 +34,14 @@ */ #ifdef MAX_PHYSMEM_BITS #define SWP_PFN_BITS (MAX_PHYSMEM_BITS - PAGE_SHIFT) +#define SWP_PFN_BITS_MAX (MAX_PHYSMEM_BITS - PAGE_SHIFT_MIN) #else /* MAX_PHYSMEM_BITS */ #define SWP_PFN_BITS min_t(int, \ sizeof(phys_addr_t) * 8 - PAGE_SHIFT, \ SWP_TYPE_SHIFT) +#define SWP_PFN_BITS_MAX min_t(int, \ + sizeof(phys_addr_t) * 8 - PAGE_SHIFT_MIN, \ + SWP_TYPE_SHIFT) #endif /* MAX_PHYSMEM_BITS */ #define SWP_PFN_MASK (BIT(SWP_PFN_BITS) - 1) @@ -519,7 +523,7 @@ static inline struct folio *pfn_swap_entry_folio(swp_entry_t entry) static inline bool is_pfn_swap_entry(swp_entry_t entry) { /* Make sure the swp offset can always store the needed fields */ - BUILD_BUG_ON(SWP_TYPE_SHIFT < SWP_PFN_BITS); + BUILD_BUG_ON(SWP_TYPE_SHIFT < SWP_PFN_BITS_MAX); return is_migration_entry(entry) || is_device_private_entry(entry) || is_device_exclusive_entry(entry) || is_hwpoison_entry(entry); diff --git a/mm/memcontrol.c b/mm/memcontrol.c index c5f9195f76c65..4b17bec566fbd 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -4881,7 +4881,7 @@ static int __init mem_cgroup_init(void) * to work fine, we should make sure that the overfill threshold can't * exceed S32_MAX / PAGE_SIZE. */ - BUILD_BUG_ON(MEMCG_CHARGE_BATCH > S32_MAX / PAGE_SIZE); + BUILD_BUG_ON(MEMCG_CHARGE_BATCH > S32_MAX / PAGE_SIZE_MIN); cpuhp_setup_state_nocalls(CPUHP_MM_MEMCQ_DEAD, "mm/memctrl:dead", NULL, memcg_hotplug_cpu_dead); diff --git a/mm/memory.c b/mm/memory.c index ebfc9768f801a..14b5ef6870486 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -4949,8 +4949,8 @@ vm_fault_t finish_fault(struct vm_fault *vmf) return ret; } -static unsigned long fault_around_pages __read_mostly = - 65536 >> PAGE_SHIFT; +static __DEFINE_GLOBAL_PAGE_SIZE_VAR(unsigned long, fault_around_pages, + __read_mostly, 65536 >> PAGE_SHIFT); #ifdef CONFIG_DEBUG_FS static int fault_around_bytes_get(void *data, u64 *val) diff --git a/mm/mmap.c b/mm/mmap.c index d0dfc85b209bb..d9642aba07ac4 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -2279,7 +2279,7 @@ int expand_downwards(struct vm_area_struct *vma, unsigned long address) } /* enforced gap between the expanding stack and other mappings. */ -unsigned long stack_guard_gap = 256UL< PFN_SECTION_SHIFT); + BUILD_BUG_ON(SECTION_MAP_LAST_BIT > PFN_SECTION_SHIFT_MIN); BUG_ON(coded_mem_map & ~SECTION_MAP_MASK); return coded_mem_map; } diff --git a/mm/swapfile.c b/mm/swapfile.c index 38bdc439651ac..6311a1cc7e46b 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -2931,7 +2931,7 @@ static unsigned long read_swap_header(struct swap_info_struct *p, unsigned long swapfilepages; unsigned long last_page; - if (memcmp("SWAPSPACE2", swap_header->magic.magic, 10)) { + if (memcmp("SWAPSPACE2", &swap_header->magic[SWAP_HEADER_MAGIC], 10)) { pr_err("Unable to find swap-space signature\n"); return 0; } diff --git a/mm/vmalloc.c b/mm/vmalloc.c index a0df1e2e155a8..b4fbba204603c 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -2497,12 +2497,12 @@ struct vmap_block { spinlock_t lock; struct vmap_area *va; unsigned long free, dirty; - DECLARE_BITMAP(used_map, VMAP_BBMAP_BITS); unsigned long dirty_min, dirty_max; /*< dirty range */ struct list_head free_list; struct rcu_head rcu_head; struct list_head purge; unsigned int cpu; + unsigned long used_map[]; }; /* Queue of free and dirty vmap blocks, for allocation and flushing purposes */ @@ -2600,11 +2600,12 @@ static void *new_vmap_block(unsigned int order, gfp_t gfp_mask) unsigned long vb_idx; int node, err; void *vaddr; + size_t size; node = numa_node_id(); - vb = kmalloc_node(sizeof(struct vmap_block), - gfp_mask & GFP_RECLAIM_MASK, node); + size = struct_size(vb, used_map, BITS_TO_LONGS(VMAP_BBMAP_BITS)); + vb = kmalloc_node(size, gfp_mask & GFP_RECLAIM_MASK, node); if (unlikely(!vb)) return ERR_PTR(-ENOMEM); From patchwork Mon Oct 14 10:58:14 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13834957 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id D1B83D1625C for ; Mon, 14 Oct 2024 12:45:33 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=hXo3oIPvtAqxU/+rzneWWG3oIuEw/H2AD2/H28YBHxs=; b=m9M/jOvmmWbdB29bmifzu3CJZL 89kmChDO9raK+wJ2CdPXtzx+k7IxdofUYU/cB9Nz1CzrhLMbn6rdC9k8VSIbKif87ehZ3//qZJkkk 8PDaCiTe3lqyYdQNO2rpFngzK/qbj3YbIf6IoaGxF6z74KgXECzNyvrbvoyQGneVMu4/zZgRcIGwA fnqsaQBDTUd3vCST9y7R54DztTvDBmL73Up8wcYsQyj/ob6NG5kwljGdSShBFYnMcmIUYrtch9iaZ Nqli8AjbBo1nYxTGdhLAvmw+wZGHDaHfqDpifx97xCz68RNBOmS9t+gQ6RPRDRLCY7HyCBeVXzPf5 D+gs8fjA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1t0KRt-000000057dY-42I9; Mon, 14 Oct 2024 12:45:21 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1t0Inl-00000004nWL-3oCc for linux-arm-kernel@lists.infradead.org; Mon, 14 Oct 2024 10:59:54 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 130DB1684; Mon, 14 Oct 2024 04:00:19 -0700 (PDT) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.27]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 2FB0B3F51B; Mon, 14 Oct 2024 03:59:46 -0700 (PDT) From: Ryan Roberts To: "Theodore Ts'o" , Alexander Viro , Andreas Dilger , Andrew Morton , Anshuman Khandual , Ard Biesheuvel , Catalin Marinas , Christian Brauner , David Hildenbrand , Greg Marsden , Ivan Ivanov , Kalesh Singh , Marc Zyngier , Mark Rutland , Matthias Brugger , Miroslav Benes , OGAWA Hirofumi , Will Deacon Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, linux-ext4@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [RFC PATCH v1 07/57] fs: Introduce MAX_BUF_PER_PAGE_SIZE_MAX for array sizing Date: Mon, 14 Oct 2024 11:58:14 +0100 Message-ID: <20241014105912.3207374-7-ryan.roberts@arm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20241014105912.3207374-1-ryan.roberts@arm.com> References: <20241014105514.3206191-1-ryan.roberts@arm.com> <20241014105912.3207374-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241014_035950_206757_9C749222 X-CRM114-Status: GOOD ( 14.53 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org To prepare for supporting boot-time page size selection, refactor code to remove assumptions about PAGE_SIZE being compile-time constant. Code intended to be equivalent when compile-time page size is active. Code that previously defined arrays with MAX_BUF_PER_PAGE will no longer work with boot-time page selection because PAGE_SIZE is not known at compile-time. Introduce MAX_BUF_PER_PAGE_SIZE_MAX for this purpose, which is the requirement in the limit when PAGE_SIZE_MAX is the selected page size. Signed-off-by: Ryan Roberts --- ***NOTE*** Any confused maintainers may want to read the cover note here for context: https://lore.kernel.org/all/20241014105514.3206191-1-ryan.roberts@arm.com/ fs/buffer.c | 2 +- fs/ext4/move_extent.c | 2 +- fs/ext4/readpage.c | 2 +- fs/fat/dir.c | 4 ++-- fs/fat/fatent.c | 4 ++-- include/linux/buffer_head.h | 1 + 6 files changed, 8 insertions(+), 7 deletions(-) diff --git a/fs/buffer.c b/fs/buffer.c index e55ad471c5306..f00542ad43a5c 100644 --- a/fs/buffer.c +++ b/fs/buffer.c @@ -2371,7 +2371,7 @@ int block_read_full_folio(struct folio *folio, get_block_t *get_block) { struct inode *inode = folio->mapping->host; sector_t iblock, lblock; - struct buffer_head *bh, *head, *arr[MAX_BUF_PER_PAGE]; + struct buffer_head *bh, *head, *arr[MAX_BUF_PER_PAGE_SIZE_MAX]; size_t blocksize; int nr, i; int fully_mapped = 1; diff --git a/fs/ext4/move_extent.c b/fs/ext4/move_extent.c index 204f53b236229..68304426c6f45 100644 --- a/fs/ext4/move_extent.c +++ b/fs/ext4/move_extent.c @@ -172,7 +172,7 @@ mext_page_mkuptodate(struct folio *folio, unsigned from, unsigned to) { struct inode *inode = folio->mapping->host; sector_t block; - struct buffer_head *bh, *head, *arr[MAX_BUF_PER_PAGE]; + struct buffer_head *bh, *head, *arr[MAX_BUF_PER_PAGE_SIZE_MAX]; unsigned int blocksize, block_start, block_end; int i, err, nr = 0, partial = 0; BUG_ON(!folio_test_locked(folio)); diff --git a/fs/ext4/readpage.c b/fs/ext4/readpage.c index 8494492582abe..5808d85096aeb 100644 --- a/fs/ext4/readpage.c +++ b/fs/ext4/readpage.c @@ -221,7 +221,7 @@ int ext4_mpage_readpages(struct inode *inode, sector_t block_in_file; sector_t last_block; sector_t last_block_in_file; - sector_t blocks[MAX_BUF_PER_PAGE]; + sector_t blocks[MAX_BUF_PER_PAGE_SIZE_MAX]; unsigned page_block; struct block_device *bdev = inode->i_sb->s_bdev; int length; diff --git a/fs/fat/dir.c b/fs/fat/dir.c index acbec5bdd5210..f3e96ecf21c92 100644 --- a/fs/fat/dir.c +++ b/fs/fat/dir.c @@ -1146,7 +1146,7 @@ int fat_alloc_new_dir(struct inode *dir, struct timespec64 *ts) { struct super_block *sb = dir->i_sb; struct msdos_sb_info *sbi = MSDOS_SB(sb); - struct buffer_head *bhs[MAX_BUF_PER_PAGE]; + struct buffer_head *bhs[MAX_BUF_PER_PAGE_SIZE_MAX]; struct msdos_dir_entry *de; sector_t blknr; __le16 date, time; @@ -1213,7 +1213,7 @@ static int fat_add_new_entries(struct inode *dir, void *slots, int nr_slots, { struct super_block *sb = dir->i_sb; struct msdos_sb_info *sbi = MSDOS_SB(sb); - struct buffer_head *bhs[MAX_BUF_PER_PAGE]; + struct buffer_head *bhs[MAX_BUF_PER_PAGE_SIZE_MAX]; sector_t blknr, start_blknr, last_blknr; unsigned long size, copy; int err, i, n, offset, cluster[2]; diff --git a/fs/fat/fatent.c b/fs/fat/fatent.c index 1db348f8f887a..322cf5b8e5590 100644 --- a/fs/fat/fatent.c +++ b/fs/fat/fatent.c @@ -469,7 +469,7 @@ int fat_alloc_clusters(struct inode *inode, int *cluster, int nr_cluster) struct msdos_sb_info *sbi = MSDOS_SB(sb); const struct fatent_operations *ops = sbi->fatent_ops; struct fat_entry fatent, prev_ent; - struct buffer_head *bhs[MAX_BUF_PER_PAGE]; + struct buffer_head *bhs[MAX_BUF_PER_PAGE_SIZE_MAX]; int i, count, err, nr_bhs, idx_clus; BUG_ON(nr_cluster > (MAX_BUF_PER_PAGE / 2)); /* fixed limit */ @@ -557,7 +557,7 @@ int fat_free_clusters(struct inode *inode, int cluster) struct msdos_sb_info *sbi = MSDOS_SB(sb); const struct fatent_operations *ops = sbi->fatent_ops; struct fat_entry fatent; - struct buffer_head *bhs[MAX_BUF_PER_PAGE]; + struct buffer_head *bhs[MAX_BUF_PER_PAGE_SIZE_MAX]; int i, err, nr_bhs; int first_cl = cluster, dirty_fsinfo = 0; diff --git a/include/linux/buffer_head.h b/include/linux/buffer_head.h index 14acf1bbe0ce6..5dff4837b76cd 100644 --- a/include/linux/buffer_head.h +++ b/include/linux/buffer_head.h @@ -41,6 +41,7 @@ enum bh_state_bits { }; #define MAX_BUF_PER_PAGE (PAGE_SIZE / 512) +#define MAX_BUF_PER_PAGE_SIZE_MAX (PAGE_SIZE_MAX / 512) struct page; struct buffer_head; From patchwork Mon Oct 14 10:58:15 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13834822 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 2DDAFD16243 for ; Mon, 14 Oct 2024 11:34:56 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=dafiuhHFBOL0KU3UgDvSqFL+BWETMThHwzJyb2+CQGw=; b=wwJi7+6F7xL9j71Civah62LT1e pKqUPqvq6sA4sx+myt+SBujMTjKSEm/uGbasLXthoIvwiuSER6RbrejChuaWGcQePuEW51kAc5C6A xDlf0ODRCymYuy0w2Md8yX2Q84Nb9iL6U6BQ7gvRAa9UMkLgB3cj7OT8Akw2xG4JTXbnZG8hWZZf4 6WTLxmk+sZ2Cv7uj5sAjAegOtZ60StYwIA2RwWsN9lCggGDlxG9gZFCuGSjrPY3rNpY05RaOqjLdY S2AzRczi7SFoxOw/Uan/5/Ufroagum8z1Hr2PUqlh7QIyxDdPb8wpziKv4ccEUb1OqkfCUAHz9yQy 1kjWIACA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1t0JLV-00000004wUh-2O6a; Mon, 14 Oct 2024 11:34:41 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1t0Inp-00000004nYc-0QAd for linux-arm-kernel@lists.infradead.org; Mon, 14 Oct 2024 10:59:55 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 1FE2A1688; Mon, 14 Oct 2024 04:00:22 -0700 (PDT) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.27]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id B86113F51B; Mon, 14 Oct 2024 03:59:49 -0700 (PDT) From: Ryan Roberts To: Alexander Viro , Andrew Morton , Anshuman Khandual , Ard Biesheuvel , Catalin Marinas , Christian Brauner , David Hildenbrand , Greg Marsden , Ivan Ivanov , Kalesh Singh , Marc Zyngier , Mark Rutland , Matthias Brugger , Miroslav Benes , Will Deacon Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [RFC PATCH v1 08/57] fs: Remove PAGE_SIZE compile-time constant assumption Date: Mon, 14 Oct 2024 11:58:15 +0100 Message-ID: <20241014105912.3207374-8-ryan.roberts@arm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20241014105912.3207374-1-ryan.roberts@arm.com> References: <20241014105514.3206191-1-ryan.roberts@arm.com> <20241014105912.3207374-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241014_035953_639266_7908E898 X-CRM114-Status: GOOD ( 16.46 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org To prepare for supporting boot-time page size selection, refactor code to remove assumptions about PAGE_SIZE being compile-time constant. Code intended to be equivalent when compile-time page size is active. In binfmt_elf, convert CPP conditional to C ternary operator; this will be folded to the same code by the compiler when in compile-time page size mode, but will also work for runtime evaluation in boot-time page size mode. In coredump, modify __dump_skip() to emit zeros in blocks of PAGE_SIZE_MIN. This resolves to the previous PAGE_SIZE for compile-time page size, but that doesn't work for boot-time page size. PAGE_SIZE_MIN is preferred here over PAGE_SIZE_MAX to save memory. Wrap global variables that are initialized with PAGE_SIZE derived values using DEFINE_GLOBAL_PAGE_SIZE_VAR() so their initialization can be deferred for boot-time page size builds. Signed-off-by: Ryan Roberts --- ***NOTE*** Any confused maintainers may want to read the cover note here for context: https://lore.kernel.org/all/20241014105514.3206191-1-ryan.roberts@arm.com/ fs/binfmt_elf.c | 11 ++++------- fs/coredump.c | 8 ++++---- 2 files changed, 8 insertions(+), 11 deletions(-) diff --git a/fs/binfmt_elf.c b/fs/binfmt_elf.c index 19fa49cd9907f..e439d36c43c7e 100644 --- a/fs/binfmt_elf.c +++ b/fs/binfmt_elf.c @@ -84,11 +84,8 @@ static int elf_core_dump(struct coredump_params *cprm); #define elf_core_dump NULL #endif -#if ELF_EXEC_PAGESIZE > PAGE_SIZE -#define ELF_MIN_ALIGN ELF_EXEC_PAGESIZE -#else -#define ELF_MIN_ALIGN PAGE_SIZE -#endif +#define ELF_MIN_ALIGN \ + (ELF_EXEC_PAGESIZE > PAGE_SIZE ? ELF_EXEC_PAGESIZE : PAGE_SIZE) #ifndef ELF_CORE_EFLAGS #define ELF_CORE_EFLAGS 0 @@ -98,7 +95,7 @@ static int elf_core_dump(struct coredump_params *cprm); #define ELF_PAGEOFFSET(_v) ((_v) & (ELF_MIN_ALIGN-1)) #define ELF_PAGEALIGN(_v) (((_v) + ELF_MIN_ALIGN - 1) & ~(ELF_MIN_ALIGN - 1)) -static struct linux_binfmt elf_format = { +static DEFINE_GLOBAL_PAGE_SIZE_VAR(struct linux_binfmt, elf_format, { .module = THIS_MODULE, .load_binary = load_elf_binary, .load_shlib = load_elf_library, @@ -106,7 +103,7 @@ static struct linux_binfmt elf_format = { .core_dump = elf_core_dump, .min_coredump = ELF_EXEC_PAGESIZE, #endif -}; +}); #define BAD_ADDR(x) (unlikely((unsigned long)(x) >= TASK_SIZE)) diff --git a/fs/coredump.c b/fs/coredump.c index 7f12ff6ad1d3e..203f2a158246e 100644 --- a/fs/coredump.c +++ b/fs/coredump.c @@ -825,7 +825,7 @@ static int __dump_emit(struct coredump_params *cprm, const void *addr, int nr) static int __dump_skip(struct coredump_params *cprm, size_t nr) { - static char zeroes[PAGE_SIZE]; + static char zeroes[PAGE_SIZE_MIN]; struct file *file = cprm->file; if (file->f_mode & FMODE_LSEEK) { if (dump_interrupted() || @@ -834,10 +834,10 @@ static int __dump_skip(struct coredump_params *cprm, size_t nr) cprm->pos += nr; return 1; } else { - while (nr > PAGE_SIZE) { - if (!__dump_emit(cprm, zeroes, PAGE_SIZE)) + while (nr > PAGE_SIZE_MIN) { + if (!__dump_emit(cprm, zeroes, PAGE_SIZE_MIN)) return 0; - nr -= PAGE_SIZE; + nr -= PAGE_SIZE_MIN; } return __dump_emit(cprm, zeroes, nr); } From patchwork Mon Oct 14 10:58:16 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13834823 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 2E541D16247 for ; Mon, 14 Oct 2024 11:36:23 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=HQvPCF1x2W8nC78duvmTyHGJn39OjXD+rSjYpslxs/Y=; b=uOcJ2PfQEF5B7UvMPfhWZML1QQ i8f2IQV87bzlCPo6dx7GA95MaO8l4Nnr0qep8wmdyg8g86wUPYnvaGae5QmJBK3Vqee6Ptix5vxb9 vtMkQJp0jC3wB+5YxXSV3c+5w8dWiHHfim3h1yKdvpb49yukMec0oG8LzRSe51zCKeUi2VUcuObiV yNhL2uDPHIv9tRdzuaKMXrza2xHAREXoUI7NqzF/x/FDGKDldKKvwhFZysqZzGPxPNGq8oVWHhz6l sdX4xGSkEgCkkYnXtXanRyIYG6u5PI1qB221sfqukpjXJNV283wclk0NlDGDRkD0vRvT6/2Ew5q7Y Xpx9GkbQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1t0JMw-00000004wpp-0YT9; Mon, 14 Oct 2024 11:36:10 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1t0Inr-00000004naz-3zmS for linux-arm-kernel@lists.infradead.org; Mon, 14 Oct 2024 10:59:58 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 2D6B51424; Mon, 14 Oct 2024 04:00:25 -0700 (PDT) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.27]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id C5CEC3F51B; Mon, 14 Oct 2024 03:59:52 -0700 (PDT) From: Ryan Roberts To: Andrew Morton , Anna Schumaker , Anshuman Khandual , Ard Biesheuvel , Catalin Marinas , David Hildenbrand , Greg Marsden , Ivan Ivanov , Kalesh Singh , Marc Zyngier , Mark Rutland , Matthias Brugger , Miroslav Benes , Trond Myklebust , Will Deacon Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-nfs@vger.kernel.org Subject: [RFC PATCH v1 09/57] fs/nfs: Remove PAGE_SIZE compile-time constant assumption Date: Mon, 14 Oct 2024 11:58:16 +0100 Message-ID: <20241014105912.3207374-9-ryan.roberts@arm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20241014105912.3207374-1-ryan.roberts@arm.com> References: <20241014105514.3206191-1-ryan.roberts@arm.com> <20241014105912.3207374-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241014_035956_221380_F70852AF X-CRM114-Status: GOOD ( 12.30 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org To prepare for supporting boot-time page size selection, refactor code to remove assumptions about PAGE_SIZE being compile-time constant. Code intended to be equivalent when compile-time page size is active. Calculation of NFS4ACL_MAXPAGES and NFS4XATTR_MAXPAGES are modified to give max pages when page size is at the minimum. BUILD_BUG_ON() is modified to test against the min page size, which implicitly also applies to all other page sizes. Signed-off-by: Ryan Roberts --- ***NOTE*** Any confused maintainers may want to read the cover note here for context: https://lore.kernel.org/all/20241014105514.3206191-1-ryan.roberts@arm.com/ fs/nfs/nfs42proc.c | 2 +- fs/nfs/nfs42xattr.c | 2 +- fs/nfs/nfs4proc.c | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/fs/nfs/nfs42proc.c b/fs/nfs/nfs42proc.c index 28704f924612c..c600574105c63 100644 --- a/fs/nfs/nfs42proc.c +++ b/fs/nfs/nfs42proc.c @@ -1161,7 +1161,7 @@ int nfs42_proc_clone(struct file *src_f, struct file *dst_f, return err; } -#define NFS4XATTR_MAXPAGES DIV_ROUND_UP(XATTR_SIZE_MAX, PAGE_SIZE) +#define NFS4XATTR_MAXPAGES DIV_ROUND_UP(XATTR_SIZE_MAX, PAGE_SIZE_MIN) static int _nfs42_proc_removexattr(struct inode *inode, const char *name) { diff --git a/fs/nfs/nfs42xattr.c b/fs/nfs/nfs42xattr.c index b6e3d8f77b910..734177eb44889 100644 --- a/fs/nfs/nfs42xattr.c +++ b/fs/nfs/nfs42xattr.c @@ -183,7 +183,7 @@ nfs4_xattr_alloc_entry(const char *name, const void *value, uint32_t flags; BUILD_BUG_ON(sizeof(struct nfs4_xattr_entry) + - XATTR_NAME_MAX + 1 > PAGE_SIZE); + XATTR_NAME_MAX + 1 > PAGE_SIZE_MIN); alloclen = sizeof(struct nfs4_xattr_entry); if (name != NULL) { diff --git a/fs/nfs/nfs4proc.c b/fs/nfs/nfs4proc.c index b8ffbe52ba15a..3c3622f46d3e0 100644 --- a/fs/nfs/nfs4proc.c +++ b/fs/nfs/nfs4proc.c @@ -5928,7 +5928,7 @@ static bool nfs4_server_supports_acls(const struct nfs_server *server, * it's OK to put sizeof(void) * (XATTR_SIZE_MAX/PAGE_SIZE) bytes on * the stack. */ -#define NFS4ACL_MAXPAGES DIV_ROUND_UP(XATTR_SIZE_MAX, PAGE_SIZE) +#define NFS4ACL_MAXPAGES DIV_ROUND_UP(XATTR_SIZE_MAX, PAGE_SIZE_MIN) int nfs4_buf_to_pages_noslab(const void *buf, size_t buflen, struct page **pages) From patchwork Mon Oct 14 10:58:17 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13834825 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id DDC4CD16247 for ; Mon, 14 Oct 2024 11:37:53 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=4YDRe8auf3W9gPGvw4mfS6WcApteYPOWnCg7H4VxM6g=; b=zDl8I9OInZe60zt2190FRK/m9W fR7ZshUGrFjRAFXXvQX01jYG/GK1EPHd7yLPkGhdb2sgmF2YLeU5DUHx2U2ntPen3RAnUdHAJPoIC DrmpZ+uARajsSoWLzJ2IRKXShQiqDh1PI+ccsATD3HezHBUjT+BSrX9u+MNCSEs0xHyFqy658V4wP wF1QXyTX5QBdRVPMLWl4fUw9J8ICRVykBesitd8m0FwM53piMuc7HWBk9HrqX7Ki58iBX06+pU+5D GAd+W4kpWa/jtaLj0BTwhQQzp+FL3EA/VU10/V8qa03pfAirMGisKw+nh2pzPYd0qdr15XzEw2iGA q6/Ck3LA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1t0JOR-00000004xKD-0Jl1; Mon, 14 Oct 2024 11:37:43 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1t0Inv-00000004nco-0UUP for linux-arm-kernel@lists.infradead.org; Mon, 14 Oct 2024 11:00:00 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 432CD1684; Mon, 14 Oct 2024 04:00:28 -0700 (PDT) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.27]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id D33E63F51B; Mon, 14 Oct 2024 03:59:55 -0700 (PDT) From: Ryan Roberts To: "Theodore Ts'o" , Andreas Dilger , Andrew Morton , Anshuman Khandual , Ard Biesheuvel , Catalin Marinas , David Hildenbrand , Greg Marsden , Ivan Ivanov , Kalesh Singh , Marc Zyngier , Mark Rutland , Matthias Brugger , Miroslav Benes , Will Deacon Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, linux-ext4@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [RFC PATCH v1 10/57] fs/ext4: Remove PAGE_SIZE compile-time constant assumption Date: Mon, 14 Oct 2024 11:58:17 +0100 Message-ID: <20241014105912.3207374-10-ryan.roberts@arm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20241014105912.3207374-1-ryan.roberts@arm.com> References: <20241014105514.3206191-1-ryan.roberts@arm.com> <20241014105912.3207374-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241014_035959_318943_93D63E8C X-CRM114-Status: GOOD ( 12.06 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org To prepare for supporting boot-time page size selection, refactor code to remove assumptions about PAGE_SIZE being compile-time constant. Code intended to be equivalent when compile-time page size is active. Convert CPP PAGE_SIZE conditionals to C if/else. For compile-time page size, the compiler will strip the dead part, and for boot-time page size, the condition will be evaluated at run time. Signed-off-by: Ryan Roberts --- ***NOTE*** Any confused maintainers may want to read the cover note here for context: https://lore.kernel.org/all/20241014105514.3206191-1-ryan.roberts@arm.com/ fs/ext4/ext4.h | 36 ++++++++++++++++++------------------ 1 file changed, 18 insertions(+), 18 deletions(-) diff --git a/fs/ext4/ext4.h b/fs/ext4/ext4.h index 08acd152261ed..1a6dbd925024a 100644 --- a/fs/ext4/ext4.h +++ b/fs/ext4/ext4.h @@ -2415,31 +2415,31 @@ ext4_rec_len_from_disk(__le16 dlen, unsigned blocksize) { unsigned len = le16_to_cpu(dlen); -#if (PAGE_SIZE >= 65536) - if (len == EXT4_MAX_REC_LEN || len == 0) - return blocksize; - return (len & 65532) | ((len & 3) << 16); -#else - return len; -#endif + if (PAGE_SIZE >= 65536) { + if (len == EXT4_MAX_REC_LEN || len == 0) + return blocksize; + return (len & 65532) | ((len & 3) << 16); + } else { + return len; + } } static inline __le16 ext4_rec_len_to_disk(unsigned len, unsigned blocksize) { BUG_ON((len > blocksize) || (blocksize > (1 << 18)) || (len & 3)); -#if (PAGE_SIZE >= 65536) - if (len < 65536) + if (PAGE_SIZE >= 65536) { + if (len < 65536) + return cpu_to_le16(len); + if (len == blocksize) { + if (blocksize == 65536) + return cpu_to_le16(EXT4_MAX_REC_LEN); + else + return cpu_to_le16(0); + } + return cpu_to_le16((len & 65532) | ((len >> 16) & 3)); + } else { return cpu_to_le16(len); - if (len == blocksize) { - if (blocksize == 65536) - return cpu_to_le16(EXT4_MAX_REC_LEN); - else - return cpu_to_le16(0); } - return cpu_to_le16((len & 65532) | ((len >> 16) & 3)); -#else - return cpu_to_le16(len); -#endif } /* From patchwork Mon Oct 14 10:58:18 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13834826 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 5437CD16247 for ; Mon, 14 Oct 2024 11:39:22 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=B2YC/X3UwnXWGexJl2uM83krn8GDdx7g25hAIge8UU4=; b=Bx3Y2FcK8s4xuYSE/zzE2oyuaW 6uppWCfTEym+vr+sA3xWht+XVbV3BuxIRPsEgi9wRiOB945xN3+mNkWF/cduWY5rbV9Z70Ecw1KI6 VHwxIpYPYPaZsCdC/FGiXkUwQbmzQWoMLNOMewnMM6as5TEQeK96bC0x0MVWXbnA1Fg9bzI0HpbgT H7EmvCXWe/SUlob9eKlLsaodXE7rgqvLku+OrG4SZr9sX8g5Yblgt2hAh1Nhxe4psoy8bPix25ncf YEj/anMqBLXiBtfFFcW1OPd1oKzHaVRSeclIiOZURM55fNJVOuhC1hOPICx0bvQHveWRVnp5nSKcO jc3nmWTQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1t0JPq-00000004xey-0Wqz; Mon, 14 Oct 2024 11:39:10 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1t0Iny-00000004neQ-2ohZ for linux-arm-kernel@lists.infradead.org; Mon, 14 Oct 2024 11:00:04 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id E3B391424; Mon, 14 Oct 2024 04:00:31 -0700 (PDT) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.27]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id E14783F51B; Mon, 14 Oct 2024 03:59:58 -0700 (PDT) From: Ryan Roberts To: Andrew Morton , Andrey Ryabinin , Anshuman Khandual , Ard Biesheuvel , Arnd Bergmann , Catalin Marinas , David Hildenbrand , Greg Marsden , Ingo Molnar , Ivan Ivanov , Juri Lelli , Kalesh Singh , Marc Zyngier , Mark Rutland , Matthias Brugger , Miroslav Benes , Peter Zijlstra , Vincent Guittot , Will Deacon Cc: Ryan Roberts , kasan-dev@googlegroups.com, linux-arch@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [RFC PATCH v1 11/57] fork: Permit boot-time THREAD_SIZE determination Date: Mon, 14 Oct 2024 11:58:18 +0100 Message-ID: <20241014105912.3207374-11-ryan.roberts@arm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20241014105912.3207374-1-ryan.roberts@arm.com> References: <20241014105514.3206191-1-ryan.roberts@arm.com> <20241014105912.3207374-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241014_040002_829989_5EC556F0 X-CRM114-Status: GOOD ( 23.07 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org THREAD_SIZE defines the size of a kernel thread stack. To date, it has been set at compile-time. However, when using vmap stacks, the size must be a multiple of PAGE_SIZE, and given we are in the process of supporting boot-time page size, we must also do the same for THREAD_SIZE. The alternative would be to define THREAD_SIZE for the largest supported page size, but this would waste memory when using a smaller page size. For example, arm64 requires THREAD_SIZE to be 16K, but when using 64K pages and a vmap stack, we must increase the size to 64K. If we required 64K when 4K or 16K page size was in use, we would waste 48K per kernel thread. So let's refactor to allow THREAD_SIZE to not be a compile-time constant. THREAD_SIZE_MAX (and THREAD_ALIGN_MAX) are introduced to manage the limits, as is done for PAGE_SIZE. When THREAD_SIZE is a compile-time constant, behaviour and code size should be equivalent. Signed-off-by: Ryan Roberts Acked-by: Vlastimil Babka --- ***NOTE*** Any confused maintainers may want to read the cover note here for context: https://lore.kernel.org/all/20241014105514.3206191-1-ryan.roberts@arm.com/ include/asm-generic/vmlinux.lds.h | 6 ++- include/linux/sched.h | 4 +- include/linux/thread_info.h | 10 ++++- init/main.c | 2 +- kernel/fork.c | 67 +++++++++++-------------------- mm/kasan/report.c | 3 +- 6 files changed, 42 insertions(+), 50 deletions(-) diff --git a/include/asm-generic/vmlinux.lds.h b/include/asm-generic/vmlinux.lds.h index 5727f883001bb..f19bab7a2e8f9 100644 --- a/include/asm-generic/vmlinux.lds.h +++ b/include/asm-generic/vmlinux.lds.h @@ -56,6 +56,10 @@ #define LOAD_OFFSET 0 #endif +#ifndef THREAD_SIZE_MAX +#define THREAD_SIZE_MAX THREAD_SIZE +#endif + /* * Only some architectures want to have the .notes segment visible in * a separate PT_NOTE ELF Program Header. When this happens, it needs @@ -398,7 +402,7 @@ init_stack = .; \ KEEP(*(.data..init_task)) \ KEEP(*(.data..init_thread_info)) \ - . = __start_init_stack + THREAD_SIZE; \ + . = __start_init_stack + THREAD_SIZE_MAX; \ __end_init_stack = .; #define JUMP_TABLE_DATA \ diff --git a/include/linux/sched.h b/include/linux/sched.h index f8d150343d42d..3de4f655ee492 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -1863,14 +1863,14 @@ union thread_union { #ifndef CONFIG_THREAD_INFO_IN_TASK struct thread_info thread_info; #endif - unsigned long stack[THREAD_SIZE/sizeof(long)]; + unsigned long stack[THREAD_SIZE_MAX/sizeof(long)]; }; #ifndef CONFIG_THREAD_INFO_IN_TASK extern struct thread_info init_thread_info; #endif -extern unsigned long init_stack[THREAD_SIZE / sizeof(unsigned long)]; +extern unsigned long init_stack[THREAD_SIZE_MAX / sizeof(unsigned long)]; #ifdef CONFIG_THREAD_INFO_IN_TASK # define task_thread_info(task) (&(task)->thread_info) diff --git a/include/linux/thread_info.h b/include/linux/thread_info.h index 9ea0b28068f49..a7ccc448cd298 100644 --- a/include/linux/thread_info.h +++ b/include/linux/thread_info.h @@ -74,7 +74,15 @@ static inline long set_restart_fn(struct restart_block *restart, } #ifndef THREAD_ALIGN -#define THREAD_ALIGN THREAD_SIZE +#define THREAD_ALIGN THREAD_SIZE +#endif + +#ifndef THREAD_SIZE_MAX +#define THREAD_SIZE_MAX THREAD_SIZE +#endif + +#ifndef THREAD_ALIGN_MAX +#define THREAD_ALIGN_MAX max(THREAD_ALIGN, THREAD_SIZE_MAX) #endif #define THREADINFO_GFP (GFP_KERNEL_ACCOUNT | __GFP_ZERO) diff --git a/init/main.c b/init/main.c index ba1515eb20b9d..4dc28115fdf57 100644 --- a/init/main.c +++ b/init/main.c @@ -797,7 +797,7 @@ void __init __weak smp_prepare_boot_cpu(void) { } -# if THREAD_SIZE >= PAGE_SIZE +#ifdef CONFIG_VMAP_STACK void __init __weak thread_stack_cache_init(void) { } diff --git a/kernel/fork.c b/kernel/fork.c index ea472566d4fcc..cbc3e73f9b501 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -184,13 +184,7 @@ static inline void free_task_struct(struct task_struct *tsk) kmem_cache_free(task_struct_cachep, tsk); } -/* - * Allocate pages if THREAD_SIZE is >= PAGE_SIZE, otherwise use a - * kmemcache based allocator. - */ -# if THREAD_SIZE >= PAGE_SIZE || defined(CONFIG_VMAP_STACK) - -# ifdef CONFIG_VMAP_STACK +#ifdef CONFIG_VMAP_STACK /* * vmalloc() is a bit slow, and calling vfree() enough times will force a TLB * flush. Try to minimize the number of calls by caching stacks. @@ -343,46 +337,21 @@ static void free_thread_stack(struct task_struct *tsk) tsk->stack_vm_area = NULL; } -# else /* !CONFIG_VMAP_STACK */ +#else /* !CONFIG_VMAP_STACK */ -static void thread_stack_free_rcu(struct rcu_head *rh) -{ - __free_pages(virt_to_page(rh), THREAD_SIZE_ORDER); -} - -static void thread_stack_delayed_free(struct task_struct *tsk) -{ - struct rcu_head *rh = tsk->stack; - - call_rcu(rh, thread_stack_free_rcu); -} - -static int alloc_thread_stack_node(struct task_struct *tsk, int node) -{ - struct page *page = alloc_pages_node(node, THREADINFO_GFP, - THREAD_SIZE_ORDER); - - if (likely(page)) { - tsk->stack = kasan_reset_tag(page_address(page)); - return 0; - } - return -ENOMEM; -} - -static void free_thread_stack(struct task_struct *tsk) -{ - thread_stack_delayed_free(tsk); - tsk->stack = NULL; -} - -# endif /* CONFIG_VMAP_STACK */ -# else /* !(THREAD_SIZE >= PAGE_SIZE || defined(CONFIG_VMAP_STACK)) */ +/* + * Allocate pages if THREAD_SIZE is >= PAGE_SIZE, otherwise use a + * kmemcache based allocator. + */ static struct kmem_cache *thread_stack_cache; static void thread_stack_free_rcu(struct rcu_head *rh) { - kmem_cache_free(thread_stack_cache, rh); + if (THREAD_SIZE >= PAGE_SIZE) + __free_pages(virt_to_page(rh), THREAD_SIZE_ORDER); + else + kmem_cache_free(thread_stack_cache, rh); } static void thread_stack_delayed_free(struct task_struct *tsk) @@ -395,7 +364,16 @@ static void thread_stack_delayed_free(struct task_struct *tsk) static int alloc_thread_stack_node(struct task_struct *tsk, int node) { unsigned long *stack; - stack = kmem_cache_alloc_node(thread_stack_cache, THREADINFO_GFP, node); + struct page *page; + + if (THREAD_SIZE >= PAGE_SIZE) { + page = alloc_pages_node(node, THREADINFO_GFP, THREAD_SIZE_ORDER); + stack = likely(page) ? page_address(page) : NULL; + } else { + stack = kmem_cache_alloc_node(thread_stack_cache, + THREADINFO_GFP, node); + } + stack = kasan_reset_tag(stack); tsk->stack = stack; return stack ? 0 : -ENOMEM; @@ -409,13 +387,16 @@ static void free_thread_stack(struct task_struct *tsk) void thread_stack_cache_init(void) { + if (THREAD_SIZE >= PAGE_SIZE) + return; + thread_stack_cache = kmem_cache_create_usercopy("thread_stack", THREAD_SIZE, THREAD_SIZE, 0, 0, THREAD_SIZE, NULL); BUG_ON(thread_stack_cache == NULL); } -# endif /* THREAD_SIZE >= PAGE_SIZE || defined(CONFIG_VMAP_STACK) */ +#endif /* CONFIG_VMAP_STACK */ /* SLAB cache for signal_struct structures (tsk->signal) */ static struct kmem_cache *signal_cachep; diff --git a/mm/kasan/report.c b/mm/kasan/report.c index b48c768acc84d..57c877852dbc6 100644 --- a/mm/kasan/report.c +++ b/mm/kasan/report.c @@ -365,8 +365,7 @@ static inline bool kernel_or_module_addr(const void *addr) static inline bool init_task_stack_addr(const void *addr) { return addr >= (void *)&init_thread_union.stack && - (addr <= (void *)&init_thread_union.stack + - sizeof(init_thread_union.stack)); + (addr <= (void *)&init_thread_union.stack + THREAD_SIZE); } static void print_address_description(void *addr, u8 tag, From patchwork Mon Oct 14 10:58:19 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13834827 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 35EDED16248 for ; Mon, 14 Oct 2024 11:40:47 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=4UO2yeUO5mzgxVnueQx6h7oEw8kU90IKNX4PaGU1b4Y=; b=zyN+JR/F3HyrCLA/WDeYwkPZ2o rnxbxPvbR248IC7M3juon/Et2oOQopxSzVdqrONCTXSwf79n5WtZNWV3DUHhzcu6pOMEJtLRGhU3g JQB+pIYoxxGrw8zBu74qhNzK+Q1oTBowFd+2A5L9GDBFNJKEgkyoaeg0HriKSt8BYZXPQ1xa2oce6 6UUFkwJpRN20jgbH5obNROfpYopZgBsL00Lpi5P6HzkizJkrsOfLqtVvyvnYnLOhsSsTKM8d/q0E6 sbprmbYeHCy3yis5WdJ6HxoVr7jfSV3MB7BlT0jWnuDMxeQy/b0KWhcW5Y6P0sajjJj8d8AZYUE+g K1VM5Crw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1t0JRD-00000004xr0-2LXA; Mon, 14 Oct 2024 11:40:35 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1t0Io2-00000004nfK-05Un for linux-arm-kernel@lists.infradead.org; Mon, 14 Oct 2024 11:00:07 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 3AD141684; Mon, 14 Oct 2024 04:00:35 -0700 (PDT) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.27]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 94D563F51B; Mon, 14 Oct 2024 04:00:02 -0700 (PDT) From: Ryan Roberts To: =?utf-8?q?Michal_Koutn=C3=BD?= , Andrew Morton , Anshuman Khandual , Ard Biesheuvel , Catalin Marinas , David Hildenbrand , Greg Marsden , Ivan Ivanov , Johannes Weiner , Kalesh Singh , Marc Zyngier , Mark Rutland , Matthias Brugger , Miroslav Benes , Tejun Heo , Will Deacon , Zefan Li Cc: Ryan Roberts , cgroups@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [RFC PATCH v1 12/57] cgroup: Remove PAGE_SIZE compile-time constant assumption Date: Mon, 14 Oct 2024 11:58:19 +0100 Message-ID: <20241014105912.3207374-12-ryan.roberts@arm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20241014105912.3207374-1-ryan.roberts@arm.com> References: <20241014105514.3206191-1-ryan.roberts@arm.com> <20241014105912.3207374-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241014_040006_131244_C192745A X-CRM114-Status: GOOD ( 10.48 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org To prepare for supporting boot-time page size selection, refactor code to remove assumptions about PAGE_SIZE being compile-time constant. Code intended to be equivalent when compile-time page size is active. Wrap global variables that are initialized with PAGE_SIZE derived values using DEFINE_GLOBAL_PAGE_SIZE_VAR() so their initialization can be deferred for boot-time page size builds. Signed-off-by: Ryan Roberts --- ***NOTE*** Any confused maintainers may want to read the cover note here for context: https://lore.kernel.org/all/20241014105514.3206191-1-ryan.roberts@arm.com/ kernel/cgroup/cgroup.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/kernel/cgroup/cgroup.c b/kernel/cgroup/cgroup.c index c8e4b62b436a4..1e9c96210821d 100644 --- a/kernel/cgroup/cgroup.c +++ b/kernel/cgroup/cgroup.c @@ -4176,16 +4176,16 @@ static int cgroup_seqfile_show(struct seq_file *m, void *arg) return 0; } -static struct kernfs_ops cgroup_kf_single_ops = { +static DEFINE_GLOBAL_PAGE_SIZE_VAR(struct kernfs_ops, cgroup_kf_single_ops, { .atomic_write_len = PAGE_SIZE, .open = cgroup_file_open, .release = cgroup_file_release, .write = cgroup_file_write, .poll = cgroup_file_poll, .seq_show = cgroup_seqfile_show, -}; +}); -static struct kernfs_ops cgroup_kf_ops = { +static DEFINE_GLOBAL_PAGE_SIZE_VAR(struct kernfs_ops, cgroup_kf_ops, { .atomic_write_len = PAGE_SIZE, .open = cgroup_file_open, .release = cgroup_file_release, @@ -4195,7 +4195,7 @@ static struct kernfs_ops cgroup_kf_ops = { .seq_next = cgroup_seqfile_next, .seq_stop = cgroup_seqfile_stop, .seq_show = cgroup_seqfile_show, -}; +}); static void cgroup_file_notify_timer(struct timer_list *timer) { From patchwork Mon Oct 14 10:58:20 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13834828 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id DBE11D16248 for ; Mon, 14 Oct 2024 11:42:08 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=TNFFxQG6M4Wg0iZv/cwkxpYlIfMOJbdGyBVBjoAfo3E=; b=NGiYusCsUE+FdfzWIbWX9A7+2B TO3GKRsarellWRRfArYnSUv54x3QtFtsTrtZDk6+6JdfGZ5ES9WCpot072v8eoJQiqoztAs7ngP6/ 8BgalAbebIlc+ogindPx3LGSmRETobZlu4L9OPH9ppxxNGMCsMJaVdST35Y0LDna8Ep11knWfNeyi s7f+WqUrr4JHAnrkJ5Y/W56E/bu3XezDHcVITKghFvseaWclu3qb0aIUUXfH2GvpxuWnWNpQTZp0K zk0G8fLVdoZdcbKTRa62ll1dxtAwCffXNcUfA8z1ytGF02UoOo/EajXYF6kqh53omplnDT/IcsMZ5 zZkSoLPA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1t0JSY-00000004xz3-2Ree; Mon, 14 Oct 2024 11:41:58 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1t0Io5-00000004ngP-04Rz for linux-arm-kernel@lists.infradead.org; Mon, 14 Oct 2024 11:00:11 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 478EC1688; Mon, 14 Oct 2024 04:00:38 -0700 (PDT) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.27]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id DF8813F51B; Mon, 14 Oct 2024 04:00:05 -0700 (PDT) From: Ryan Roberts To: Alexei Starovoitov , Andrew Morton , Anshuman Khandual , Ard Biesheuvel , Catalin Marinas , Daniel Borkmann , David Hildenbrand , Greg Marsden , Ivan Ivanov , Kalesh Singh , Marc Zyngier , Mark Rutland , Matthias Brugger , Miroslav Benes , Will Deacon Cc: Ryan Roberts , bpf@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [RFC PATCH v1 13/57] bpf: Remove PAGE_SIZE compile-time constant assumption Date: Mon, 14 Oct 2024 11:58:20 +0100 Message-ID: <20241014105912.3207374-13-ryan.roberts@arm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20241014105912.3207374-1-ryan.roberts@arm.com> References: <20241014105514.3206191-1-ryan.roberts@arm.com> <20241014105912.3207374-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241014_040009_173758_D5FD897B X-CRM114-Status: GOOD ( 23.31 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org To prepare for supporting boot-time page size selection, refactor code to remove assumptions about PAGE_SIZE being compile-time constant. Code intended to be equivalent when compile-time page size is active. Refactor "struct bpf_ringbuf" so that consumer_pos, producer_pos, pending_pos and data are no longer embedded at (static) page offsets within the struct. This can't work for boot-time page size because the page size isn't known at compile-time. Instead, only define the meta data in the struct, along with pointers to those values. At "struct bpf_ringbuf" allocation time, the extra pages are allocated at the end and the pointers are initialized to point to the correct locations. Additionally, only expose the __PAGE_SIZE enum to BTF for compile-time page size builds. We don't know the page size at compile-time for boot-time builds. NOTE: This may need some extra thought; perhaps __PAGE_SIZE should be exposed as 0 in this case? And/or perhaps __PAGE_SIZE_MIN/__PAGE_SIZE_MAX should be exposed? And there would need to be a runtime mechanism for querying the page size (e.g. getpagesize()). Signed-off-by: Ryan Roberts --- ***NOTE*** Any confused maintainers may want to read the cover note here for context: https://lore.kernel.org/all/20241014105514.3206191-1-ryan.roberts@arm.com/ kernel/bpf/core.c | 9 ++++++-- kernel/bpf/ringbuf.c | 54 ++++++++++++++++++++++++-------------------- 2 files changed, 37 insertions(+), 26 deletions(-) diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c index 7ee62e38faf0e..485875aa78e63 100644 --- a/kernel/bpf/core.c +++ b/kernel/bpf/core.c @@ -89,10 +89,15 @@ void *bpf_internal_load_pointer_neg_helper(const struct sk_buff *skb, int k, uns return NULL; } -/* tell bpf programs that include vmlinux.h kernel's PAGE_SIZE */ +/* + * tell bpf programs that include vmlinux.h kernel's PAGE_SIZE. We can only do + * this for compile-time PAGE_SIZE builds. + */ +#if PAGE_SIZE_MIN == PAGE_SIZE_MAX enum page_size_enum { __PAGE_SIZE = PAGE_SIZE }; +#endif struct bpf_prog *bpf_prog_alloc_no_stats(unsigned int size, gfp_t gfp_extra_flags) { @@ -100,7 +105,7 @@ struct bpf_prog *bpf_prog_alloc_no_stats(unsigned int size, gfp_t gfp_extra_flag struct bpf_prog_aux *aux; struct bpf_prog *fp; - size = round_up(size, __PAGE_SIZE); + size = round_up(size, PAGE_SIZE); fp = __vmalloc(size, gfp_flags); if (fp == NULL) return NULL; diff --git a/kernel/bpf/ringbuf.c b/kernel/bpf/ringbuf.c index e20b90c361316..8e4093ddbc638 100644 --- a/kernel/bpf/ringbuf.c +++ b/kernel/bpf/ringbuf.c @@ -14,9 +14,9 @@ #define RINGBUF_CREATE_FLAG_MASK (BPF_F_NUMA_NODE) -/* non-mmap()'able part of bpf_ringbuf (everything up to consumer page) */ +/* non-mmap()'able part of bpf_ringbuf (everything defined in struct) */ #define RINGBUF_PGOFF \ - (offsetof(struct bpf_ringbuf, consumer_pos) >> PAGE_SHIFT) + (PAGE_ALIGN(sizeof(struct bpf_ringbuf)) >> PAGE_SHIFT) /* consumer page and producer page */ #define RINGBUF_POS_PAGES 2 #define RINGBUF_NR_META_PAGES (RINGBUF_PGOFF + RINGBUF_POS_PAGES) @@ -69,10 +69,10 @@ struct bpf_ringbuf { * validate each sample to ensure that they're correctly formatted, and * fully contained within the ring buffer. */ - unsigned long consumer_pos __aligned(PAGE_SIZE); - unsigned long producer_pos __aligned(PAGE_SIZE); - unsigned long pending_pos; - char data[] __aligned(PAGE_SIZE); + unsigned long *consumer_pos; + unsigned long *producer_pos; + unsigned long *pending_pos; + char *data; }; struct bpf_ringbuf_map { @@ -134,9 +134,15 @@ static struct bpf_ringbuf *bpf_ringbuf_area_alloc(size_t data_sz, int numa_node) rb = vmap(pages, nr_meta_pages + 2 * nr_data_pages, VM_MAP | VM_USERMAP, PAGE_KERNEL); if (rb) { + void *base = rb; + kmemleak_not_leak(pages); rb->pages = pages; rb->nr_pages = nr_pages; + rb->consumer_pos = (unsigned long *)(base + PAGE_SIZE * RINGBUF_PGOFF); + rb->producer_pos = (unsigned long *)(base + PAGE_SIZE * (RINGBUF_PGOFF + 1)); + rb->pending_pos = rb->producer_pos + 1; + rb->data = base + PAGE_SIZE * nr_meta_pages; return rb; } @@ -179,9 +185,9 @@ static struct bpf_ringbuf *bpf_ringbuf_alloc(size_t data_sz, int numa_node) init_irq_work(&rb->work, bpf_ringbuf_notify); rb->mask = data_sz - 1; - rb->consumer_pos = 0; - rb->producer_pos = 0; - rb->pending_pos = 0; + *rb->consumer_pos = 0; + *rb->producer_pos = 0; + *rb->pending_pos = 0; return rb; } @@ -300,8 +306,8 @@ static unsigned long ringbuf_avail_data_sz(struct bpf_ringbuf *rb) { unsigned long cons_pos, prod_pos; - cons_pos = smp_load_acquire(&rb->consumer_pos); - prod_pos = smp_load_acquire(&rb->producer_pos); + cons_pos = smp_load_acquire(rb->consumer_pos); + prod_pos = smp_load_acquire(rb->producer_pos); return prod_pos - cons_pos; } @@ -418,7 +424,7 @@ static void *__bpf_ringbuf_reserve(struct bpf_ringbuf *rb, u64 size) if (len > ringbuf_total_data_sz(rb)) return NULL; - cons_pos = smp_load_acquire(&rb->consumer_pos); + cons_pos = smp_load_acquire(rb->consumer_pos); if (in_nmi()) { if (!spin_trylock_irqsave(&rb->spinlock, flags)) @@ -427,8 +433,8 @@ static void *__bpf_ringbuf_reserve(struct bpf_ringbuf *rb, u64 size) spin_lock_irqsave(&rb->spinlock, flags); } - pend_pos = rb->pending_pos; - prod_pos = rb->producer_pos; + pend_pos = *rb->pending_pos; + prod_pos = *rb->producer_pos; new_prod_pos = prod_pos + len; while (pend_pos < prod_pos) { @@ -440,7 +446,7 @@ static void *__bpf_ringbuf_reserve(struct bpf_ringbuf *rb, u64 size) tmp_size = round_up(tmp_size + BPF_RINGBUF_HDR_SZ, 8); pend_pos += tmp_size; } - rb->pending_pos = pend_pos; + *rb->pending_pos = pend_pos; /* check for out of ringbuf space: * - by ensuring producer position doesn't advance more than @@ -460,7 +466,7 @@ static void *__bpf_ringbuf_reserve(struct bpf_ringbuf *rb, u64 size) hdr->pg_off = pg_off; /* pairs with consumer's smp_load_acquire() */ - smp_store_release(&rb->producer_pos, new_prod_pos); + smp_store_release(rb->producer_pos, new_prod_pos); spin_unlock_irqrestore(&rb->spinlock, flags); @@ -506,7 +512,7 @@ static void bpf_ringbuf_commit(void *sample, u64 flags, bool discard) * new data availability */ rec_pos = (void *)hdr - (void *)rb->data; - cons_pos = smp_load_acquire(&rb->consumer_pos) & rb->mask; + cons_pos = smp_load_acquire(rb->consumer_pos) & rb->mask; if (flags & BPF_RB_FORCE_WAKEUP) irq_work_queue(&rb->work); @@ -580,9 +586,9 @@ BPF_CALL_2(bpf_ringbuf_query, struct bpf_map *, map, u64, flags) case BPF_RB_RING_SIZE: return ringbuf_total_data_sz(rb); case BPF_RB_CONS_POS: - return smp_load_acquire(&rb->consumer_pos); + return smp_load_acquire(rb->consumer_pos); case BPF_RB_PROD_POS: - return smp_load_acquire(&rb->producer_pos); + return smp_load_acquire(rb->producer_pos); default: return 0; } @@ -680,12 +686,12 @@ static int __bpf_user_ringbuf_peek(struct bpf_ringbuf *rb, void **sample, u32 *s u64 cons_pos, prod_pos; /* Synchronizes with smp_store_release() in user-space producer. */ - prod_pos = smp_load_acquire(&rb->producer_pos); + prod_pos = smp_load_acquire(rb->producer_pos); if (prod_pos % 8) return -EINVAL; /* Synchronizes with smp_store_release() in __bpf_user_ringbuf_sample_release() */ - cons_pos = smp_load_acquire(&rb->consumer_pos); + cons_pos = smp_load_acquire(rb->consumer_pos); if (cons_pos >= prod_pos) return -ENODATA; @@ -715,7 +721,7 @@ static int __bpf_user_ringbuf_peek(struct bpf_ringbuf *rb, void **sample, u32 *s * Update the consumer pos, and return -EAGAIN so the caller * knows to skip this sample and try to read the next one. */ - smp_store_release(&rb->consumer_pos, cons_pos + total_len); + smp_store_release(rb->consumer_pos, cons_pos + total_len); return -EAGAIN; } @@ -737,9 +743,9 @@ static void __bpf_user_ringbuf_sample_release(struct bpf_ringbuf *rb, size_t siz * prevents another task from writing to consumer_pos after it was read * by this task with smp_load_acquire() in __bpf_user_ringbuf_peek(). */ - consumer_pos = rb->consumer_pos; + consumer_pos = *rb->consumer_pos; /* Synchronizes with smp_load_acquire() in user-space producer. */ - smp_store_release(&rb->consumer_pos, consumer_pos + rounded_size); + smp_store_release(rb->consumer_pos, consumer_pos + rounded_size); } BPF_CALL_4(bpf_user_ringbuf_drain, struct bpf_map *, map, From patchwork Mon Oct 14 10:58:21 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13834841 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id D1789D16248 for ; Mon, 14 Oct 2024 11:43:34 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=S61ImBLZUPZT81QYOovWzbBNwho35ypnuFAVdu44m+A=; b=olsdAvbOdlCVjxL1DPLW4zJp29 sTT65iRs7EUSNCUju1XruiWb92EzG1gEbCxAWut+6mJs+6mxEJ5MhzNoyXqZEykYDGbCVWBVevNpP lAOdap+K3DouekM9Cd59swnwrWtrEVC6wooWUtlzQU5qKQ2Qu+uKF2u4dj73HSyCdsaMHZVj+g6qg M1xDbcJssfKNFAm97Wd3GSxAJ8aa4vu3Vgi/4M12eX6TQKm06O7U2jgrPfssQiUpWcEN6+davh2FI CzkfJtarqttqO0GAep7Jd3EmbGd+rXhqzV9z+hfpazWPGjb/8HY1p6OtvSWoP7Cu5QLfvjnPq5Aae neC0r7Nw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1t0JTt-00000004y9f-32Ai; Mon, 14 Oct 2024 11:43:21 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1t0Io7-00000004niP-3WUT for linux-arm-kernel@lists.infradead.org; Mon, 14 Oct 2024 11:00:15 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 15DE8168F; Mon, 14 Oct 2024 04:00:41 -0700 (PDT) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.27]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id EC2633F51B; Mon, 14 Oct 2024 04:00:08 -0700 (PDT) From: Ryan Roberts To: Andrew Morton , Anshuman Khandual , Ard Biesheuvel , Catalin Marinas , David Hildenbrand , Greg Marsden , Ivan Ivanov , Kalesh Singh , Marc Zyngier , Mark Rutland , Matthias Brugger , Miroslav Benes , Will Deacon Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-pm@vger.kernel.org Subject: [RFC PATCH v1 14/57] pm/hibernate: Remove PAGE_SIZE compile-time constant assumption Date: Mon, 14 Oct 2024 11:58:21 +0100 Message-ID: <20241014105912.3207374-14-ryan.roberts@arm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20241014105912.3207374-1-ryan.roberts@arm.com> References: <20241014105514.3206191-1-ryan.roberts@arm.com> <20241014105912.3207374-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241014_040012_024252_DF4FE82C X-CRM114-Status: GOOD ( 24.63 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org To prepare for supporting boot-time page size selection, refactor code to remove assumptions about PAGE_SIZE being compile-time constant. Code intended to be equivalent when compile-time page size is active. "struct linked_page", "struct swap_map_page" and "struct swsusp_header" were all previously sized to be exactly PAGE_SIZE. Refactor those structures to remove the padding, then superimpose them on a page at runtime. "struct cmp_data" and "struct dec_data" previously contained embedded "unc" and "cmp" arrays, who's sizes were derived from PAGE_SIZE. We can't use flexible array approach here since there are 2 arrays in the structure, so convert to pointers and define an allocator and deallocator for each struct. Signed-off-by: Ryan Roberts --- ***NOTE*** Any confused maintainers may want to read the cover note here for context: https://lore.kernel.org/all/20241014105514.3206191-1-ryan.roberts@arm.com/ kernel/power/power.h | 2 +- kernel/power/snapshot.c | 2 +- kernel/power/swap.c | 129 +++++++++++++++++++++++++++++++++------- 3 files changed, 108 insertions(+), 25 deletions(-) diff --git a/kernel/power/power.h b/kernel/power/power.h index de0e6b1077f23..74af2eb8d48a4 100644 --- a/kernel/power/power.h +++ b/kernel/power/power.h @@ -16,7 +16,7 @@ struct swsusp_info { unsigned long image_pages; unsigned long pages; unsigned long size; -} __aligned(PAGE_SIZE); +} __aligned(PAGE_SIZE_MAX); #ifdef CONFIG_HIBERNATION /* kernel/power/snapshot.c */ diff --git a/kernel/power/snapshot.c b/kernel/power/snapshot.c index 405eddbda4fc5..144e92f786e35 100644 --- a/kernel/power/snapshot.c +++ b/kernel/power/snapshot.c @@ -155,7 +155,7 @@ struct pbe *restore_pblist; struct linked_page { struct linked_page *next; - char data[LINKED_PAGE_DATA_SIZE]; + char data[]; } __packed; /* diff --git a/kernel/power/swap.c b/kernel/power/swap.c index 82b884b67152f..ffd4c864acfa2 100644 --- a/kernel/power/swap.c +++ b/kernel/power/swap.c @@ -59,6 +59,7 @@ static bool clean_pages_on_decompress; */ #define MAP_PAGE_ENTRIES (PAGE_SIZE / sizeof(sector_t) - 1) +#define NEXT_SWAP_INDEX MAP_PAGE_ENTRIES /* * Number of free pages that are not high. @@ -78,8 +79,11 @@ static inline unsigned long reqd_free_pages(void) } struct swap_map_page { - sector_t entries[MAP_PAGE_ENTRIES]; - sector_t next_swap; + /* + * A PAGE_SIZE structure with (PAGE_SIZE / sizeof(sector_t)) entries. + * The last entry, [NEXT_SWAP_INDEX], is `.next_swap`. + */ + sector_t entries[1]; }; struct swap_map_page_list { @@ -103,8 +107,6 @@ struct swap_map_handle { }; struct swsusp_header { - char reserved[PAGE_SIZE - 20 - sizeof(sector_t) - sizeof(int) - - sizeof(u32) - sizeof(u32)]; u32 hw_sig; u32 crc32; sector_t image; @@ -113,6 +115,7 @@ struct swsusp_header { char sig[10]; } __packed; +static char *swsusp_header_pg; static struct swsusp_header *swsusp_header; /* @@ -315,7 +318,7 @@ static int mark_swapfiles(struct swap_map_handle *handle, unsigned int flags) { int error; - hib_submit_io(REQ_OP_READ, swsusp_resume_block, swsusp_header, NULL); + hib_submit_io(REQ_OP_READ, swsusp_resume_block, swsusp_header_pg, NULL); if (!memcmp("SWAP-SPACE",swsusp_header->sig, 10) || !memcmp("SWAPSPACE2",swsusp_header->sig, 10)) { memcpy(swsusp_header->orig_sig,swsusp_header->sig, 10); @@ -329,7 +332,7 @@ static int mark_swapfiles(struct swap_map_handle *handle, unsigned int flags) if (flags & SF_CRC32_MODE) swsusp_header->crc32 = handle->crc32; error = hib_submit_io(REQ_OP_WRITE | REQ_SYNC, - swsusp_resume_block, swsusp_header, NULL); + swsusp_resume_block, swsusp_header_pg, NULL); } else { pr_err("Swap header not found!\n"); error = -ENODEV; @@ -466,7 +469,7 @@ static int swap_write_page(struct swap_map_handle *handle, void *buf, offset = alloc_swapdev_block(root_swap); if (!offset) return -ENOSPC; - handle->cur->next_swap = offset; + handle->cur->entries[NEXT_SWAP_INDEX] = offset; error = write_page(handle->cur, handle->cur_swap, hb); if (error) goto out; @@ -643,8 +646,8 @@ struct cmp_data { wait_queue_head_t done; /* compression done */ size_t unc_len; /* uncompressed length */ size_t cmp_len; /* compressed length */ - unsigned char unc[UNC_SIZE]; /* uncompressed buffer */ - unsigned char cmp[CMP_SIZE]; /* compressed buffer */ + unsigned char *unc; /* uncompressed buffer */ + unsigned char *cmp; /* compressed buffer */ }; /* Indicates the image size after compression */ @@ -683,6 +686,45 @@ static int compress_threadfn(void *data) return 0; } +static void free_cmp_data(struct cmp_data *data, unsigned nr_threads) +{ + int i; + + if (!data) + return; + + for (i = 0; i < nr_threads; i++) { + vfree(data[i].unc); + vfree(data[i].cmp); + } + + vfree(data); +} + +static struct cmp_data *alloc_cmp_data(unsigned nr_threads) +{ + struct cmp_data *data = NULL; + int i = -1; + + data = vzalloc(array_size(nr_threads, sizeof(*data))); + if (!data) + goto fail; + + for (i = 0; i < nr_threads; i++) { + data[i].unc = vzalloc(UNC_SIZE); + if (!data[i].unc) + goto fail; + data[i].cmp = vzalloc(CMP_SIZE); + if (!data[i].cmp) + goto fail; + } + + return data; +fail: + free_cmp_data(data, nr_threads); + return NULL; +} + /** * save_compressed_image - Save the suspend image data after compression. * @handle: Swap map handle to use for saving the image. @@ -724,7 +766,7 @@ static int save_compressed_image(struct swap_map_handle *handle, goto out_clean; } - data = vzalloc(array_size(nr_threads, sizeof(*data))); + data = alloc_cmp_data(nr_threads); if (!data) { pr_err("Failed to allocate %s data\n", hib_comp_algo); ret = -ENOMEM; @@ -902,7 +944,7 @@ static int save_compressed_image(struct swap_map_handle *handle, if (data[thr].cc) crypto_free_comp(data[thr].cc); } - vfree(data); + free_cmp_data(data, nr_threads); } if (page) free_page((unsigned long)page); @@ -1036,7 +1078,7 @@ static int get_swap_reader(struct swap_map_handle *handle, release_swap_reader(handle); return error; } - offset = tmp->map->next_swap; + offset = tmp->map->entries[NEXT_SWAP_INDEX]; } handle->k = 0; handle->cur = handle->maps->map; @@ -1150,8 +1192,8 @@ struct dec_data { wait_queue_head_t done; /* decompression done */ size_t unc_len; /* uncompressed length */ size_t cmp_len; /* compressed length */ - unsigned char unc[UNC_SIZE]; /* uncompressed buffer */ - unsigned char cmp[CMP_SIZE]; /* compressed buffer */ + unsigned char *unc; /* uncompressed buffer */ + unsigned char *cmp; /* compressed buffer */ }; /* @@ -1189,6 +1231,45 @@ static int decompress_threadfn(void *data) return 0; } +static void free_dec_data(struct dec_data *data, unsigned nr_threads) +{ + int i; + + if (!data) + return; + + for (i = 0; i < nr_threads; i++) { + vfree(data[i].unc); + vfree(data[i].cmp); + } + + vfree(data); +} + +static struct dec_data *alloc_dec_data(unsigned nr_threads) +{ + struct dec_data *data = NULL; + int i = -1; + + data = vzalloc(array_size(nr_threads, sizeof(*data))); + if (!data) + goto fail; + + for (i = 0; i < nr_threads; i++) { + data[i].unc = vzalloc(UNC_SIZE); + if (!data[i].unc) + goto fail; + data[i].cmp = vzalloc(CMP_SIZE); + if (!data[i].cmp) + goto fail; + } + + return data; +fail: + free_dec_data(data, nr_threads); + return NULL; +} + /** * load_compressed_image - Load compressed image data and decompress it. * @handle: Swap map handle to use for loading data. @@ -1231,7 +1312,7 @@ static int load_compressed_image(struct swap_map_handle *handle, goto out_clean; } - data = vzalloc(array_size(nr_threads, sizeof(*data))); + data = alloc_dec_data(nr_threads); if (!data) { pr_err("Failed to allocate %s data\n", hib_comp_algo); ret = -ENOMEM; @@ -1510,7 +1591,7 @@ static int load_compressed_image(struct swap_map_handle *handle, if (data[thr].cc) crypto_free_comp(data[thr].cc); } - vfree(data); + free_dec_data(data, nr_threads); } vfree(page); @@ -1569,9 +1650,9 @@ int swsusp_check(bool exclusive) hib_resume_bdev_file = bdev_file_open_by_dev(swsusp_resume_device, BLK_OPEN_READ, holder, NULL); if (!IS_ERR(hib_resume_bdev_file)) { - clear_page(swsusp_header); + clear_page(swsusp_header_pg); error = hib_submit_io(REQ_OP_READ, swsusp_resume_block, - swsusp_header, NULL); + swsusp_header_pg, NULL); if (error) goto put; @@ -1581,7 +1662,7 @@ int swsusp_check(bool exclusive) /* Reset swap signature now */ error = hib_submit_io(REQ_OP_WRITE | REQ_SYNC, swsusp_resume_block, - swsusp_header, NULL); + swsusp_header_pg, NULL); } else { error = -EINVAL; } @@ -1631,12 +1712,12 @@ int swsusp_unmark(void) int error; hib_submit_io(REQ_OP_READ, swsusp_resume_block, - swsusp_header, NULL); + swsusp_header_pg, NULL); if (!memcmp(HIBERNATE_SIG,swsusp_header->sig, 10)) { memcpy(swsusp_header->sig,swsusp_header->orig_sig, 10); error = hib_submit_io(REQ_OP_WRITE | REQ_SYNC, swsusp_resume_block, - swsusp_header, NULL); + swsusp_header_pg, NULL); } else { pr_err("Cannot find swsusp signature!\n"); error = -ENODEV; @@ -1653,9 +1734,11 @@ int swsusp_unmark(void) static int __init swsusp_header_init(void) { - swsusp_header = (struct swsusp_header*) __get_free_page(GFP_KERNEL); - if (!swsusp_header) + swsusp_header_pg = (char *)__get_free_page(GFP_KERNEL); + if (!swsusp_header_pg) panic("Could not allocate memory for swsusp_header\n"); + swsusp_header = (struct swsusp_header *)(swsusp_header_pg + + PAGE_SIZE - sizeof(struct swsusp_header)); return 0; } From patchwork Mon Oct 14 10:58:22 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13834842 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id D995ED1624B for ; Mon, 14 Oct 2024 11:44:56 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=mddGb9Xpz4LiyvZj0cyiEir/DtiqGjM7+BRPU5mZ/qY=; b=izRLPdlYUEHbwgcOabHPX4B4Yk D9d7v6e8d2yVj5ax7Ss8QDKMq7iuJtGX2dH+2PGvJ6gfqEzwmT2+KOOi6tRwnMskrKpdWIKJ2NrJx 3J33Cw42mFww3S1VBUyc7xq+N+lD8I+L8vgFu+Wj82OJJhj5lBcZ/fJR65uSiBh4JYPeD6u18YBMf 1WGYOszZvsLOmchSxCxHDJA0MMm53NWksmb82LkTcODJaV6LLF4Pn3Jbkq2xVMmtGucSEAnVhnWT/ eKcx7LUrwyTMZr2MZylAXTgzUXOjQXrGQqGmu/bcJjnVRDkQTHugUQZcyMtgXRrtIdw/sLx7nVsNC wUXaeWWQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1t0JVG-00000004yN4-0YrH; Mon, 14 Oct 2024 11:44:46 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1t0IoA-00000004njm-2KLJ for linux-arm-kernel@lists.infradead.org; Mon, 14 Oct 2024 11:00:16 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id BA5741691; Mon, 14 Oct 2024 04:00:43 -0700 (PDT) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.27]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id BB0D53F51B; Mon, 14 Oct 2024 04:00:11 -0700 (PDT) From: Ryan Roberts To: Andrew Morton , Anshuman Khandual , Ard Biesheuvel , Catalin Marinas , David Hildenbrand , Greg Marsden , Ivan Ivanov , Kalesh Singh , Marc Zyngier , Mark Rutland , Matthias Brugger , Miroslav Benes , Will Deacon Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [RFC PATCH v1 15/57] stackdepot: Remove PAGE_SIZE compile-time constant assumption Date: Mon, 14 Oct 2024 11:58:22 +0100 Message-ID: <20241014105912.3207374-15-ryan.roberts@arm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20241014105912.3207374-1-ryan.roberts@arm.com> References: <20241014105514.3206191-1-ryan.roberts@arm.com> <20241014105912.3207374-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241014_040014_818323_31CEFF0F X-CRM114-Status: GOOD ( 17.05 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org To prepare for supporting boot-time page size selection, refactor code to remove assumptions about PAGE_SIZE being compile-time constant. Code intended to be equivalent when compile-time page size is active. "union handle_parts" previously calculated the number of bits required for its pool index and offset members based on PAGE_SHIFT. This is problematic for boot-time page size builds because the actual page size isn't known until boot-time. We could use PAGE_SHIFT_MAX in calculating the worst case offset bits, but bits would be wasted that could be used for pool index when PAGE_SIZE is set smaller than MAX, the end result being that stack depot can address less memory than it should. To avoid needing to dynamically define the offset and index bit widths, let's instead fix the pool size and derive the order at runtime based on the PAGE_SIZE. This means that the fields' widths can remain static, with the down side being slightly increased risk of failing to allocate the large folio. This only affects boot-time page size builds. compile-time page size builds will still always allocate order-2 folios. Additionally, wrap global variables that are initialized with PAGE_SIZE derived values using DEFINE_GLOBAL_PAGE_SIZE_VAR() so their initialization can be deferred for boot-time page size builds. Signed-off-by: Ryan Roberts Acked-by: Vlastimil Babka --- ***NOTE*** Any confused maintainers may want to read the cover note here for context: https://lore.kernel.org/all/20241014105514.3206191-1-ryan.roberts@arm.com/ include/linux/stackdepot.h | 6 +++--- lib/stackdepot.c | 6 +++--- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/include/linux/stackdepot.h b/include/linux/stackdepot.h index e9ec32fb97d4a..ac877a4e90406 100644 --- a/include/linux/stackdepot.h +++ b/include/linux/stackdepot.h @@ -32,10 +32,10 @@ typedef u32 depot_stack_handle_t; #define DEPOT_HANDLE_BITS (sizeof(depot_stack_handle_t) * 8) -#define DEPOT_POOL_ORDER 2 /* Pool size order, 4 pages */ -#define DEPOT_POOL_SIZE (1LL << (PAGE_SHIFT + DEPOT_POOL_ORDER)) +#define DEPOT_POOL_ORDER 2 /* Pool size order, 4 pages of PAGE_SIZE_MAX */ +#define DEPOT_POOL_SIZE (1LL << (PAGE_SHIFT_MAX + DEPOT_POOL_ORDER)) #define DEPOT_STACK_ALIGN 4 -#define DEPOT_OFFSET_BITS (DEPOT_POOL_ORDER + PAGE_SHIFT - DEPOT_STACK_ALIGN) +#define DEPOT_OFFSET_BITS (DEPOT_POOL_ORDER + PAGE_SHIFT_MAX - DEPOT_STACK_ALIGN) #define DEPOT_POOL_INDEX_BITS (DEPOT_HANDLE_BITS - DEPOT_OFFSET_BITS - \ STACK_DEPOT_EXTRA_BITS) diff --git a/lib/stackdepot.c b/lib/stackdepot.c index 5ed34cc963fc3..974351f0e9e3c 100644 --- a/lib/stackdepot.c +++ b/lib/stackdepot.c @@ -68,7 +68,7 @@ static void *new_pool; /* Number of pools in stack_pools. */ static int pools_num; /* Offset to the unused space in the currently used pool. */ -static size_t pool_offset = DEPOT_POOL_SIZE; +static DEFINE_GLOBAL_PAGE_SIZE_VAR(size_t, pool_offset, DEPOT_POOL_SIZE); /* Freelist of stack records within stack_pools. */ static LIST_HEAD(free_stacks); /* The lock must be held when performing pool or freelist modifications. */ @@ -625,7 +625,7 @@ depot_stack_handle_t stack_depot_save_flags(unsigned long *entries, */ if (unlikely(can_alloc && !READ_ONCE(new_pool))) { page = alloc_pages(gfp_nested_mask(alloc_flags), - DEPOT_POOL_ORDER); + get_order(DEPOT_POOL_SIZE)); if (page) prealloc = page_address(page); } @@ -663,7 +663,7 @@ depot_stack_handle_t stack_depot_save_flags(unsigned long *entries, exit: if (prealloc) { /* Stack depot didn't use this memory, free it. */ - free_pages((unsigned long)prealloc, DEPOT_POOL_ORDER); + free_pages((unsigned long)prealloc, get_order(DEPOT_POOL_SIZE)); } if (found) handle = found->handle.handle; From patchwork Mon Oct 14 10:58:23 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13834843 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 91C6FD1624B for ; Mon, 14 Oct 2024 11:46:23 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=9GjRzzDOoiNCAX5UvCNi+d9DYFyCOtMmFOQ33Q8ONIw=; b=X2dKe2LTZBzs8bs9+cvrU6Pq+6 M9StMBUORvPoU8DPA1NLNzz53IDUyYtt6xS2E1J9DrCEW2UEC5A7MS/NWpAGreFi2rFv6M10nrYlw hnVGf8xFGPZetDIZynyxTUxLtRSsW9fVW7uK4wpVUsIUIzNYZ9Flw5SSnJR61QflguCMXs765NJFn +U3mA2ri64KdcK8/msIPTHVWVJHDet7CzFi82k/c6uqOZqeTfmHDRNiL3vxAgefwEbeUwyguDDenz PrIXlC1SI4qMWtRNJqPDfWIgk5pVbneCCgEUNK11aGu1dsuVRMtPFi8QyWiZjWY3j/oJQcRAJCnQK 4T2qwSdw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1t0JWd-00000004yeN-1A5O; Mon, 14 Oct 2024 11:46:11 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1t0IoD-00000004nku-1D3M for linux-arm-kernel@lists.infradead.org; Mon, 14 Oct 2024 11:00:18 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 89DBA168F; Mon, 14 Oct 2024 04:00:46 -0700 (PDT) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.27]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 6B5403F51B; Mon, 14 Oct 2024 04:00:14 -0700 (PDT) From: Ryan Roberts To: Andrew Morton , Anshuman Khandual , Ard Biesheuvel , Catalin Marinas , David Hildenbrand , Greg Marsden , Ivan Ivanov , Kalesh Singh , Marc Zyngier , Mark Rutland , Matthias Brugger , Miroslav Benes , Will Deacon Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-perf-users@vger.kernel.org Subject: [RFC PATCH v1 16/57] perf: Remove PAGE_SIZE compile-time constant assumption Date: Mon, 14 Oct 2024 11:58:23 +0100 Message-ID: <20241014105912.3207374-16-ryan.roberts@arm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20241014105912.3207374-1-ryan.roberts@arm.com> References: <20241014105514.3206191-1-ryan.roberts@arm.com> <20241014105912.3207374-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241014_040017_406726_2609C2BB X-CRM114-Status: GOOD ( 12.40 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org To prepare for supporting boot-time page size selection, refactor code to remove assumptions about PAGE_SIZE being compile-time constant. Code intended to be equivalent when compile-time page size is active. Refactor a BUILD_BUG_ON() so that we test against the limit; _format is invariant to page size so testing it is no bigger than the minimum supported size is sufficient. Wrap global variables that are initialized with PAGE_SIZE derived values using DEFINE_GLOBAL_PAGE_SIZE_VAR() so their initialization can be deferred for boot-time page size builds. Signed-off-by: Ryan Roberts --- ***NOTE*** Any confused maintainers may want to read the cover note here for context: https://lore.kernel.org/all/20241014105514.3206191-1-ryan.roberts@arm.com/ include/linux/perf_event.h | 2 +- kernel/events/core.c | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/include/linux/perf_event.h b/include/linux/perf_event.h index 1a8942277ddad..b7972155f93eb 100644 --- a/include/linux/perf_event.h +++ b/include/linux/perf_event.h @@ -1872,7 +1872,7 @@ _name##_show(struct device *dev, \ struct device_attribute *attr, \ char *page) \ { \ - BUILD_BUG_ON(sizeof(_format) >= PAGE_SIZE); \ + BUILD_BUG_ON(sizeof(_format) >= PAGE_SIZE_MIN); \ return sprintf(page, _format "\n"); \ } \ diff --git a/kernel/events/core.c b/kernel/events/core.c index 8a6c6bbcd658a..81149663ab7d8 100644 --- a/kernel/events/core.c +++ b/kernel/events/core.c @@ -419,7 +419,7 @@ static struct kmem_cache *perf_event_cache; int sysctl_perf_event_paranoid __read_mostly = 2; /* Minimum for 512 kiB + 1 user control page */ -int sysctl_perf_event_mlock __read_mostly = 512 + (PAGE_SIZE / 1024); /* 'free' kiB per user */ +__DEFINE_GLOBAL_PAGE_SIZE_VAR(int, sysctl_perf_event_mlock, __read_mostly, 512 + (PAGE_SIZE / 1024)); /* 'free' kiB per user */ /* * max perf event sample rate From patchwork Mon Oct 14 10:58:24 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13834844 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 2C068D1624B for ; Mon, 14 Oct 2024 11:47:48 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=SCVxAiH0ahzES1Ifh6ZjiVheU6y/DFb6NZERTs5N+W4=; b=oxLBD9+CBRpLuiT7MYDgptc9w6 NgqzScwqztC9TPgk1GOe9Scj6xIadwVtaFJowXdkoFu2oUnIS78/fkpSmN0PO7J+w3Ae95QldquNS BpdAOo8aOaqonkSt695DCXVfKMSWRUgVIXs3v8AoFYKBguKIIvyzIIM3v1lzuaOd7JiWY8nTYaxoZ EX4uN56OFK2o6Tg8pSnbTD6O4fSlcGWIFHeBX2utFyQ+iAoVskub0SL+pKQPyQclpM7//8TSzRBqS G/q8ZRmtVzvM0SM5tJCilLzBRNz9MRtDeHXwd/1wLd0AjkciVJlkZIDEPx1++V7iTIFjfwVPBp6G4 BfAKBP9Q==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1t0JY0-00000004ysh-2Yjt; Mon, 14 Oct 2024 11:47:36 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1t0IoG-00000004nmH-1jaf for linux-arm-kernel@lists.infradead.org; Mon, 14 Oct 2024 11:00:22 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 58C151684; Mon, 14 Oct 2024 04:00:49 -0700 (PDT) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.27]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 3A6AC3F51B; Mon, 14 Oct 2024 04:00:17 -0700 (PDT) From: Ryan Roberts To: Andrew Morton , Anshuman Khandual , Ard Biesheuvel , Catalin Marinas , David Hildenbrand , Greg Marsden , Ivan Ivanov , Kalesh Singh , Marc Zyngier , Mark Rutland , Matthias Brugger , Miroslav Benes , Will Deacon Cc: Ryan Roberts , kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [RFC PATCH v1 17/57] kvm: Remove PAGE_SIZE compile-time constant assumption Date: Mon, 14 Oct 2024 11:58:24 +0100 Message-ID: <20241014105912.3207374-17-ryan.roberts@arm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20241014105912.3207374-1-ryan.roberts@arm.com> References: <20241014105514.3206191-1-ryan.roberts@arm.com> <20241014105912.3207374-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241014_040020_760268_AF9691D3 X-CRM114-Status: GOOD ( 11.22 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org To prepare for supporting boot-time page size selection, refactor code to remove assumptions about PAGE_SIZE being compile-time constant. Code intended to be equivalent when compile-time page size is active. Modify BUILD_BUG_ON() to compare with page size limit. Signed-off-by: Ryan Roberts Reviewed-by: Sean Christopherson --- ***NOTE*** Any confused maintainers may want to read the cover note here for context: https://lore.kernel.org/all/20241014105514.3206191-1-ryan.roberts@arm.com/ virt/kvm/kvm_main.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c index cb2b78e92910f..6c862bc41a672 100644 --- a/virt/kvm/kvm_main.c +++ b/virt/kvm/kvm_main.c @@ -4244,7 +4244,7 @@ static int kvm_vm_ioctl_create_vcpu(struct kvm *kvm, unsigned long id) goto vcpu_decrement; } - BUILD_BUG_ON(sizeof(struct kvm_run) > PAGE_SIZE); + BUILD_BUG_ON(sizeof(struct kvm_run) > PAGE_SIZE_MIN); page = alloc_page(GFP_KERNEL_ACCOUNT | __GFP_ZERO); if (!page) { r = -ENOMEM; From patchwork Mon Oct 14 10:58:25 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13834846 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 56E4FD1624B for ; Mon, 14 Oct 2024 11:49:14 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=WYYsCtIDe3FfCiqGTFtQtqmkCpDu+2cX7LKMS9zkbAc=; b=3HI8m/tAeLIVpZvtfyBVM43Rak slq2iudAmMDL8fyyK9kVm2l58mr/HAxzOKjA54XtPLJSfwuFBfQFFjL2hoFuaRQ8Hv28pzUeJcr9v CFgZt0+C/3+shhll98XpRA8BSNeaJntQopx/0zprIw1EMi53ruk52vnssCYYyhqPp5Vywi5TJqSUc R3PVDu+9RSnMcG0SGsnrHNqz6HrRoYl648Iy5RM96ch2Phr6NMCIKM3rBDYo7o7PgMm8twfzw+E2x TXmFApTkZ346fHNtnqYSGl/9mPerundiVcwlctNSca8lPuFeJZrFjvw2L7xVsz8n7O4SewD8qR4Lz SsaVBMTw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1t0JZM-00000004z0E-1P5q; Mon, 14 Oct 2024 11:49:00 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1t0IoJ-00000004no7-0igD for linux-arm-kernel@lists.infradead.org; Mon, 14 Oct 2024 11:00:25 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 65FA91688; Mon, 14 Oct 2024 04:00:52 -0700 (PDT) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.27]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 09E9C3F51B; Mon, 14 Oct 2024 04:00:19 -0700 (PDT) From: Ryan Roberts To: Andrew Morton , Anshuman Khandual , Ard Biesheuvel , Catalin Marinas , David Hildenbrand , Greg Marsden , Ivan Ivanov , Kalesh Singh , Marc Zyngier , Mark Rutland , Masami Hiramatsu , Matthias Brugger , Miroslav Benes , Steven Rostedt , Will Deacon Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-trace-kernel@vger.kernel.org Subject: [RFC PATCH v1 18/57] trace: Remove PAGE_SIZE compile-time constant assumption Date: Mon, 14 Oct 2024 11:58:25 +0100 Message-ID: <20241014105912.3207374-18-ryan.roberts@arm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20241014105912.3207374-1-ryan.roberts@arm.com> References: <20241014105514.3206191-1-ryan.roberts@arm.com> <20241014105912.3207374-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241014_040023_357344_C876D1BC X-CRM114-Status: GOOD ( 14.31 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org To prepare for supporting boot-time page size selection, refactor code to remove assumptions about PAGE_SIZE being compile-time constant. Code intended to be equivalent when compile-time page size is active. Convert BUILD_BUG_ON() BUG_ON() since the argument depends on PAGE_SIZE and its not trivial to test against a page size limit. Redefine FTRACE_KSTACK_ENTRIES so that "struct ftrace_stacks" is always sized at 32K for 64-bit and 16K for 32-bit. It was previously defined in terms of PAGE_SIZE (and worked out at the quoted sizes for a 4K page size). But for 64K pages, the size expanded to 512K. Given the ftrace stacks should be invariant to page size, this seemed like a waste. As a side effect, it removes the PAGE_SIZE compile-time constant assumption from this code. Signed-off-by: Ryan Roberts --- ***NOTE*** Any confused maintainers may want to read the cover note here for context: https://lore.kernel.org/all/20241014105514.3206191-1-ryan.roberts@arm.com/ kernel/trace/fgraph.c | 2 +- kernel/trace/trace.c | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/kernel/trace/fgraph.c b/kernel/trace/fgraph.c index d7d4fb403f6f0..47aa5c8d8090e 100644 --- a/kernel/trace/fgraph.c +++ b/kernel/trace/fgraph.c @@ -534,7 +534,7 @@ ftrace_push_return_trace(unsigned long ret, unsigned long func, if (!current->ret_stack) return -EBUSY; - BUILD_BUG_ON(SHADOW_STACK_SIZE % sizeof(long)); + BUG_ON(SHADOW_STACK_SIZE % sizeof(long)); /* Set val to "reserved" with the delta to the new fgraph frame */ val = (FGRAPH_TYPE_RESERVED << FGRAPH_TYPE_SHIFT) | FGRAPH_FRAME_OFFSET; diff --git a/kernel/trace/trace.c b/kernel/trace/trace.c index c3b2c7dfadef1..0f2ec3d30579f 100644 --- a/kernel/trace/trace.c +++ b/kernel/trace/trace.c @@ -2887,7 +2887,7 @@ trace_function(struct trace_array *tr, unsigned long ip, unsigned long /* Allow 4 levels of nesting: normal, softirq, irq, NMI */ #define FTRACE_KSTACK_NESTING 4 -#define FTRACE_KSTACK_ENTRIES (PAGE_SIZE / FTRACE_KSTACK_NESTING) +#define FTRACE_KSTACK_ENTRIES (SZ_4K / FTRACE_KSTACK_NESTING) struct ftrace_stack { unsigned long calls[FTRACE_KSTACK_ENTRIES]; From patchwork Mon Oct 14 10:58:26 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13834847 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 807BBD1624B for ; Mon, 14 Oct 2024 11:50:36 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=P0Pjm4UAl0+EiWZfuspcZJKa6yxJiEhvXEi1XSb1a9E=; b=YQz4nE7ceDVBejoPpBey8ivlD9 hV99TpC45lRkNZZd6BUnjXiO/SXYIoFrb0bHkoq3/7Xz9IoIE6lWEcRGT7n907HcLZrS+d88EjAwz LRH0Nfka9DPYiRBJUFfzEVIib6sMHrfdwpovSsQJ3PRG4imxl7sTZbCGS/RZqPLVFjvQ7L9wNJXTs d+K8AdFef6vSLdMUAV0xt+dQLGmKGnAe0rM10vqidHzRlUg2Lbx97USgog0nTYqIy99TEc+Cqla4r aU8n2kzijneTfuS4N/l2zoNArTqHGkBfUFQ+UZIEnKVfuCpkqGYVk+/oMmO7aXjnrt6iaZSBXHKrs 5z5onSYA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1t0Jaj-00000004zGu-2tHh; Mon, 14 Oct 2024 11:50:25 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1t0IoM-00000004npi-06qx; Mon, 14 Oct 2024 11:00:27 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 5428F1684; Mon, 14 Oct 2024 04:00:55 -0700 (PDT) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.27]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 16F953F51B; Mon, 14 Oct 2024 04:00:22 -0700 (PDT) From: Ryan Roberts To: Andrew Morton , Anshuman Khandual , Ard Biesheuvel , Baoquan He , Catalin Marinas , David Hildenbrand , Greg Marsden , Ivan Ivanov , Kalesh Singh , Marc Zyngier , Mark Rutland , Matthias Brugger , Miroslav Benes , Will Deacon Cc: Ryan Roberts , kexec@lists.infradead.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [RFC PATCH v1 19/57] crash: Remove PAGE_SIZE compile-time constant assumption Date: Mon, 14 Oct 2024 11:58:26 +0100 Message-ID: <20241014105912.3207374-19-ryan.roberts@arm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20241014105912.3207374-1-ryan.roberts@arm.com> References: <20241014105514.3206191-1-ryan.roberts@arm.com> <20241014105912.3207374-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241014_040026_127702_2457ED95 X-CRM114-Status: GOOD ( 10.64 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org To prepare for supporting boot-time page size selection, refactor code to remove assumptions about PAGE_SIZE being compile-time constant. Code intended to be equivalent when compile-time page size is active. Updated BUILD_BUG_ON() to test against limit. Signed-off-by: Ryan Roberts Acked-by: Baoquan He --- ***NOTE*** Any confused maintainers may want to read the cover note here for context: https://lore.kernel.org/all/20241014105514.3206191-1-ryan.roberts@arm.com/ kernel/crash_core.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/kernel/crash_core.c b/kernel/crash_core.c index 63cf89393c6eb..978c600a47ac8 100644 --- a/kernel/crash_core.c +++ b/kernel/crash_core.c @@ -465,7 +465,7 @@ static int __init crash_notes_memory_init(void) * Break compile if size is bigger than PAGE_SIZE since crash_notes * definitely will be in 2 pages with that. */ - BUILD_BUG_ON(size > PAGE_SIZE); + BUILD_BUG_ON(size > PAGE_SIZE_MIN); crash_notes = __alloc_percpu(size, align); if (!crash_notes) { From patchwork Mon Oct 14 10:58:27 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13834848 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id DE5AAD1624A for ; Mon, 14 Oct 2024 11:52:02 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=zTmw5JCDPKyesvf4SHt2c/Iv1pPV+8Di4FJdKvNluCo=; b=C/V5Diub2FERF7KROrKs6LBEsO nWr9+zSsZrbckn2p79mzTlE2wpwEKie1sDVkjPkj9pYOxdOn1C0LU4gW7qCqOJ52PZOoOctRgy1Of 7yiU2xEqeLx+p9SyULNcYOI5yrgp/kuTHV81dRuo85UVtn8XrUlq0df/k/e5YIUxeWBmDmtaE10tD eCfsCjaBg7X5A8lOTbxsOe9XIiNXHJr86Axj+TwRcTC7icvzhI9S5ZtMQZewQYmozvsYnd8kNVZNt pNfeJibYW7up+bltIHYzoxb0rlIQReOLfo+6dA8xRf23C8z72UuwM4i8zBw3K0Jz47Ynl/QsEs7/Z DOWXA40g==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1t0Jc6-00000004zUp-04Na; Mon, 14 Oct 2024 11:51:50 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1t0IoP-00000004nr0-0hX8 for linux-arm-kernel@lists.infradead.org; Mon, 14 Oct 2024 11:00:30 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 61C921688; Mon, 14 Oct 2024 04:00:58 -0700 (PDT) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.27]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 0597C3F51B; Mon, 14 Oct 2024 04:00:25 -0700 (PDT) From: Ryan Roberts To: "David S. Miller" , Andrew Morton , Anshuman Khandual , Ard Biesheuvel , Catalin Marinas , David Hildenbrand , Greg Marsden , Herbert Xu , Ivan Ivanov , Kalesh Singh , Marc Zyngier , Mark Rutland , Matthias Brugger , Miroslav Benes , Will Deacon Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, linux-crypto@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [RFC PATCH v1 20/57] crypto: Remove PAGE_SIZE compile-time constant assumption Date: Mon, 14 Oct 2024 11:58:27 +0100 Message-ID: <20241014105912.3207374-20-ryan.roberts@arm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20241014105912.3207374-1-ryan.roberts@arm.com> References: <20241014105514.3206191-1-ryan.roberts@arm.com> <20241014105912.3207374-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241014_040029_271127_475FF057 X-CRM114-Status: GOOD ( 10.95 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org To prepare for supporting boot-time page size selection, refactor code to remove assumptions about PAGE_SIZE being compile-time constant. Code intended to be equivalent when compile-time page size is active. Updated BUILD_BUG_ON() to test against limit. Signed-off-by: Ryan Roberts Acked-by: Herbert Xu --- ***NOTE*** Any confused maintainers may want to read the cover note here for context: https://lore.kernel.org/all/20241014105514.3206191-1-ryan.roberts@arm.com/ crypto/lskcipher.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/crypto/lskcipher.c b/crypto/lskcipher.c index cdb4897c63e6f..2b84cefba7cd1 100644 --- a/crypto/lskcipher.c +++ b/crypto/lskcipher.c @@ -79,8 +79,8 @@ static int crypto_lskcipher_crypt_unaligned( u8 *tiv; u8 *p; - BUILD_BUG_ON(MAX_CIPHER_BLOCKSIZE > PAGE_SIZE || - MAX_CIPHER_ALIGNMASK >= PAGE_SIZE); + BUILD_BUG_ON(MAX_CIPHER_BLOCKSIZE > PAGE_SIZE_MIN || + MAX_CIPHER_ALIGNMASK >= PAGE_SIZE_MIN); tiv = kmalloc(PAGE_SIZE, GFP_ATOMIC); if (!tiv) From patchwork Mon Oct 14 10:58:28 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13834851 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 9524FD1624A for ; Mon, 14 Oct 2024 11:53:26 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=WjYxq47rZAOyOAKetOXqZvgmx001+cTD5LxEFEwtYek=; b=k2uP6vA7nQElobNnEQcGMrlZ3F T5I6uvJrRMUD4kCAD40YxfbwSy+qZ+PK5uJEA8zxHufmm2GK20ri2wqlpruxRZaJThDwY0lOAe2yd Iip5bSfAB8Kb/S6LZ9nx5VF4y2cjrl1vH71KyuLPcgtRgDWbaKtwxqyXw/cnGs7tHrmLfMfScWFEz QqyaP4Ofz3OPvSaaKmLS509oExc6zUKq2oC8+8D6mYBjn2HumFuRbquU6Zs2P+6G/zyxhhD85QPRj rIZYStirRfnUeaNGtKH2cUajopHsF3gIywi4B/dCTNgOsVlwqwor6CKtfB3Sap+ZRE/bqHNwXtxYE 8qu9LxPw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1t0JdS-00000004zjj-0G22; Mon, 14 Oct 2024 11:53:14 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1t0IoS-00000004nrx-0tnY for linux-arm-kernel@lists.infradead.org; Mon, 14 Oct 2024 11:00:33 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 6EB86168F; Mon, 14 Oct 2024 04:01:01 -0700 (PDT) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.27]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 1323A3F51B; Mon, 14 Oct 2024 04:00:28 -0700 (PDT) From: Ryan Roberts To: Andrew Morton , Anna Schumaker , Anshuman Khandual , Ard Biesheuvel , Catalin Marinas , David Hildenbrand , Greg Marsden , Ivan Ivanov , Kalesh Singh , Marc Zyngier , Mark Rutland , Matthias Brugger , Miroslav Benes , Trond Myklebust , Will Deacon Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-nfs@vger.kernel.org Subject: [RFC PATCH v1 21/57] sunrpc: Remove PAGE_SIZE compile-time constant assumption Date: Mon, 14 Oct 2024 11:58:28 +0100 Message-ID: <20241014105912.3207374-21-ryan.roberts@arm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20241014105912.3207374-1-ryan.roberts@arm.com> References: <20241014105514.3206191-1-ryan.roberts@arm.com> <20241014105912.3207374-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241014_040032_371192_919C71D2 X-CRM114-Status: GOOD ( 14.63 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org To prepare for supporting boot-time page size selection, refactor code to remove assumptions about PAGE_SIZE being compile-time constant. Code intended to be equivalent when compile-time page size is active. Updated array sizes in various structs to contain enough entries for the smallest supported page size. Signed-off-by: Ryan Roberts --- ***NOTE*** Any confused maintainers may want to read the cover note here for context: https://lore.kernel.org/all/20241014105514.3206191-1-ryan.roberts@arm.com/ include/linux/sunrpc/svc.h | 8 +++++--- include/linux/sunrpc/svc_rdma.h | 4 ++-- include/linux/sunrpc/svcsock.h | 2 +- 3 files changed, 8 insertions(+), 6 deletions(-) diff --git a/include/linux/sunrpc/svc.h b/include/linux/sunrpc/svc.h index a7d0406b9ef59..dda44018b8f36 100644 --- a/include/linux/sunrpc/svc.h +++ b/include/linux/sunrpc/svc.h @@ -160,6 +160,8 @@ extern u32 svc_max_payload(const struct svc_rqst *rqstp); */ #define RPCSVC_MAXPAGES ((RPCSVC_MAXPAYLOAD+PAGE_SIZE-1)/PAGE_SIZE \ + 2 + 1) +#define RPCSVC_MAXPAGES_MAX ((RPCSVC_MAXPAYLOAD+PAGE_SIZE_MIN-1)/PAGE_SIZE_MIN \ + + 2 + 1) /* * The context of a single thread, including the request currently being @@ -190,14 +192,14 @@ struct svc_rqst { struct xdr_stream rq_res_stream; struct page *rq_scratch_page; struct xdr_buf rq_res; - struct page *rq_pages[RPCSVC_MAXPAGES + 1]; + struct page *rq_pages[RPCSVC_MAXPAGES_MAX + 1]; struct page * *rq_respages; /* points into rq_pages */ struct page * *rq_next_page; /* next reply page to use */ struct page * *rq_page_end; /* one past the last page */ struct folio_batch rq_fbatch; - struct kvec rq_vec[RPCSVC_MAXPAGES]; /* generally useful.. */ - struct bio_vec rq_bvec[RPCSVC_MAXPAGES]; + struct kvec rq_vec[RPCSVC_MAXPAGES_MAX]; /* generally useful.. */ + struct bio_vec rq_bvec[RPCSVC_MAXPAGES_MAX]; __be32 rq_xid; /* transmission id */ u32 rq_prog; /* program number */ diff --git a/include/linux/sunrpc/svc_rdma.h b/include/linux/sunrpc/svc_rdma.h index d33bab33099ab..7c6441e8d6f7a 100644 --- a/include/linux/sunrpc/svc_rdma.h +++ b/include/linux/sunrpc/svc_rdma.h @@ -200,7 +200,7 @@ struct svc_rdma_recv_ctxt { struct svc_rdma_pcl rc_reply_pcl; unsigned int rc_page_count; - struct page *rc_pages[RPCSVC_MAXPAGES]; + struct page *rc_pages[RPCSVC_MAXPAGES_MAX]; }; /* @@ -242,7 +242,7 @@ struct svc_rdma_send_ctxt { void *sc_xprt_buf; int sc_page_count; int sc_cur_sge_no; - struct page *sc_pages[RPCSVC_MAXPAGES]; + struct page *sc_pages[RPCSVC_MAXPAGES_MAX]; struct ib_sge sc_sges[]; }; diff --git a/include/linux/sunrpc/svcsock.h b/include/linux/sunrpc/svcsock.h index 7c78ec6356b92..6c6bcc82685a3 100644 --- a/include/linux/sunrpc/svcsock.h +++ b/include/linux/sunrpc/svcsock.h @@ -40,7 +40,7 @@ struct svc_sock { struct completion sk_handshake_done; - struct page * sk_pages[RPCSVC_MAXPAGES]; /* received data */ + struct page * sk_pages[RPCSVC_MAXPAGES_MAX]; /* received data */ }; static inline u32 svc_sock_reclen(struct svc_sock *svsk) From patchwork Mon Oct 14 10:58:29 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13834852 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 0DCF1D1624A for ; Mon, 14 Oct 2024 11:54:52 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=wEHuKb5rtMoKCIErGijWSzrMjkfF9oh6ylp1H4/kTdU=; b=0ON/QD5vM2ogWaViIaFJis9cjW MdJzUKHJ4VOvHp3kR2iYRn05KxvLUYBYANOZ2jyAYC1uzjKRyBvegESetsZhPUCMLUPQ8D6BTiY5C e7sKQhs19+dAqQc2n1UzBxhzR1xcVRvt40QPFQluKCT1tGcml4waCNMpT2kzzolg2r0L+SS7aVuRq FzqVp3cwZ6x77RoHqJPL9nNvyvfkgtVE8tiN9MRXHs/ViMmNrADbqW/NyV97hIsnxlhJK0fdc3BqQ rXL1JxF8Fm3QXJqb6SbOHm+MG+aKUT6X82SztnV5/6eguyNSB+yYfyuT3bcC4gDLdqZgQsaKqT1C3 QQtr54FA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1t0Jep-00000004zxp-2EdC; Mon, 14 Oct 2024 11:54:39 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1t0IoV-00000004nu5-14Gy for linux-arm-kernel@lists.infradead.org; Mon, 14 Oct 2024 11:00:36 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 7C133168F; Mon, 14 Oct 2024 04:01:04 -0700 (PDT) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.27]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 201A53F51B; Mon, 14 Oct 2024 04:00:32 -0700 (PDT) From: Ryan Roberts To: Andrew Morton , Anshuman Khandual , Ard Biesheuvel , Catalin Marinas , David Hildenbrand , Greg Marsden , Ivan Ivanov , Jaroslav Kysela , Kalesh Singh , Marc Zyngier , Mark Rutland , Matthias Brugger , Miroslav Benes , Takashi Iwai , Will Deacon Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-sound@vger.kernel.org Subject: [RFC PATCH v1 22/57] sound: Remove PAGE_SIZE compile-time constant assumption Date: Mon, 14 Oct 2024 11:58:29 +0100 Message-ID: <20241014105912.3207374-22-ryan.roberts@arm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20241014105912.3207374-1-ryan.roberts@arm.com> References: <20241014105514.3206191-1-ryan.roberts@arm.com> <20241014105912.3207374-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241014_040035_369517_9036B14E X-CRM114-Status: GOOD ( 11.58 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org To prepare for supporting boot-time page size selection, refactor code to remove assumptions about PAGE_SIZE being compile-time constant. Code intended to be equivalent when compile-time page size is active. Wrap global variables that are initialized with PAGE_SIZE derived values using DEFINE_GLOBAL_PAGE_SIZE_VAR() so their initialization can be deferred for boot-time page size builds. Signed-off-by: Ryan Roberts --- ***NOTE*** Any confused maintainers may want to read the cover note here for context: https://lore.kernel.org/all/20241014105514.3206191-1-ryan.roberts@arm.com/ sound/soc/soc-utils.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/sound/soc/soc-utils.c b/sound/soc/soc-utils.c index 303823dc45d7a..74e1bc1087de4 100644 --- a/sound/soc/soc-utils.c +++ b/sound/soc/soc-utils.c @@ -98,7 +98,7 @@ int snd_soc_tdm_params_to_bclk(const struct snd_pcm_hw_params *params, } EXPORT_SYMBOL_GPL(snd_soc_tdm_params_to_bclk); -static const struct snd_pcm_hardware dummy_dma_hardware = { +static DEFINE_GLOBAL_PAGE_SIZE_VAR_CONST(struct snd_pcm_hardware, dummy_dma_hardware, { /* Random values to keep userspace happy when checking constraints */ .info = SNDRV_PCM_INFO_INTERLEAVED | SNDRV_PCM_INFO_BLOCK_TRANSFER, @@ -107,7 +107,7 @@ static const struct snd_pcm_hardware dummy_dma_hardware = { .period_bytes_max = PAGE_SIZE*2, .periods_min = 2, .periods_max = 128, -}; +}); static const struct snd_soc_component_driver dummy_platform; From patchwork Mon Oct 14 10:58:30 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13834853 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 8D3ACD1624E for ; Mon, 14 Oct 2024 11:56:20 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=LsJL+IRqe/3ZGlQHKLYmptDwM4z2+Mr5za/KqOhCeys=; b=tJh8Eo9IKaGRmouaDnbsIIUOFl HzEGZ6MiFcQDMZas9d81l4jhKtJjxx4CUSt7FZu3tEhagJYTEU4nwYicM+xrmD6i2ZlaNGTM3cHbi TAWc5VCkSBNgnGoVR3Cgp8eDhJqYPJwny0wzGXqVjMLm3Gq/9+lQWPet4FUIuHD/0J7MrfqW+gBp8 djIdXakaVriMSxbF8YuzbXIZrXGVJ4jsyEsSKU9wU/vNrEtkayhFk8EMxY7vMyMMoTSKhyadjX31H tuyBxJUOhgybl9Aokpy/6ecLvG8b9AYI4AT5XkfByNXCs5HEyq1lmaM0wCT9RhOdIF16jnlF/I34V UrhNn1jQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1t0JgG-000000050Fi-0qjw; Mon, 14 Oct 2024 11:56:08 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1t0IoY-00000004nvb-3rxU for linux-arm-kernel@lists.infradead.org; Mon, 14 Oct 2024 11:00:40 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 306CE1691; Mon, 14 Oct 2024 04:01:08 -0700 (PDT) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.27]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 2DF153F51B; Mon, 14 Oct 2024 04:00:35 -0700 (PDT) From: Ryan Roberts To: "David S. Miller" , Andrew Morton , Anna Schumaker , Anshuman Khandual , Ard Biesheuvel , Catalin Marinas , David Hildenbrand , Eric Dumazet , Greg Marsden , Ivan Ivanov , Jakub Kicinski , Kalesh Singh , Marc Zyngier , Mark Rutland , Matthias Brugger , Miroslav Benes , Paolo Abeni , Trond Myklebust , Will Deacon Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-nfs@vger.kernel.org, netdev@vger.kernel.org Subject: [RFC PATCH v1 23/57] net: Remove PAGE_SIZE compile-time constant assumption Date: Mon, 14 Oct 2024 11:58:30 +0100 Message-ID: <20241014105912.3207374-23-ryan.roberts@arm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20241014105912.3207374-1-ryan.roberts@arm.com> References: <20241014105514.3206191-1-ryan.roberts@arm.com> <20241014105912.3207374-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241014_040039_072229_3EE1B665 X-CRM114-Status: GOOD ( 16.64 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org To prepare for supporting boot-time page size selection, refactor code to remove assumptions about PAGE_SIZE being compile-time constant. Code intended to be equivalent when compile-time page size is active. Define NLMSG_GOODSIZE using min() instead of ifdeffery. This will now evaluate to a compile-time constant for compile-time page size, but evaluate at run-time when using boot-time page size. Rework NAPI small page frag infrastructure so that for boot-time page size it is compiled in if 4K page size is in the possible range, but defer deciding to use it to run time when the page size is known. No change for compile-time page size case. Resize cache_defer_hash[] array for PAGE_SIZE_MAX. Convert a complex BUILD_BUG_ON() to runtime BUG_ON(). Wrap global variables that are initialized with PAGE_SIZE derived values using DEFINE_GLOBAL_PAGE_SIZE_VAR() so their initialization can be deferred for boot-time page size builds. Signed-off-by: Ryan Roberts --- ***NOTE*** Any confused maintainers may want to read the cover note here for context: https://lore.kernel.org/all/20241014105514.3206191-1-ryan.roberts@arm.com/ include/linux/netlink.h | 6 +----- net/core/hotdata.c | 4 ++-- net/core/skbuff.c | 4 ++-- net/core/sysctl_net_core.c | 2 +- net/sunrpc/cache.c | 3 ++- net/unix/af_unix.c | 2 +- 6 files changed, 9 insertions(+), 12 deletions(-) diff --git a/include/linux/netlink.h b/include/linux/netlink.h index b332c2048c755..ffa1e94111f89 100644 --- a/include/linux/netlink.h +++ b/include/linux/netlink.h @@ -267,11 +267,7 @@ netlink_skb_clone(struct sk_buff *skb, gfp_t gfp_mask) * use enormous buffer sizes on recvmsg() calls just to avoid * MSG_TRUNC when PAGE_SIZE is very large. */ -#if PAGE_SIZE < 8192UL -#define NLMSG_GOODSIZE SKB_WITH_OVERHEAD(PAGE_SIZE) -#else -#define NLMSG_GOODSIZE SKB_WITH_OVERHEAD(8192UL) -#endif +#define NLMSG_GOODSIZE SKB_WITH_OVERHEAD(min(PAGE_SIZE, 8192UL)) #define NLMSG_DEFAULT_SIZE (NLMSG_GOODSIZE - NLMSG_HDRLEN) diff --git a/net/core/hotdata.c b/net/core/hotdata.c index d0aaaaa556f22..e1f30e87ba6e9 100644 --- a/net/core/hotdata.c +++ b/net/core/hotdata.c @@ -5,7 +5,7 @@ #include #include -struct net_hotdata net_hotdata __cacheline_aligned = { +__DEFINE_GLOBAL_PAGE_SIZE_VAR(struct net_hotdata, net_hotdata, __cacheline_aligned, { .offload_base = LIST_HEAD_INIT(net_hotdata.offload_base), .ptype_all = LIST_HEAD_INIT(net_hotdata.ptype_all), .gro_normal_batch = 8, @@ -21,5 +21,5 @@ struct net_hotdata net_hotdata __cacheline_aligned = { .sysctl_max_skb_frags = MAX_SKB_FRAGS, .sysctl_skb_defer_max = 64, .sysctl_mem_pcpu_rsv = SK_MEMORY_PCPU_RESERVE -}; +}); EXPORT_SYMBOL(net_hotdata); diff --git a/net/core/skbuff.c b/net/core/skbuff.c index 83f8cd8aa2d16..b6c8eee0cc74b 100644 --- a/net/core/skbuff.c +++ b/net/core/skbuff.c @@ -219,9 +219,9 @@ static void skb_under_panic(struct sk_buff *skb, unsigned int sz, void *addr) #define NAPI_SKB_CACHE_BULK 16 #define NAPI_SKB_CACHE_HALF (NAPI_SKB_CACHE_SIZE / 2) -#if PAGE_SIZE == SZ_4K +#if PAGE_SIZE_MIN <= SZ_4K && SZ_4K <= PAGE_SIZE_MAX -#define NAPI_HAS_SMALL_PAGE_FRAG 1 +#define NAPI_HAS_SMALL_PAGE_FRAG (PAGE_SIZE == SZ_4K) #define NAPI_SMALL_PAGE_PFMEMALLOC(nc) ((nc).pfmemalloc) /* specialized page frag allocator using a single order 0 page diff --git a/net/core/sysctl_net_core.c b/net/core/sysctl_net_core.c index 86a2476678c48..a7a2eb7581bd1 100644 --- a/net/core/sysctl_net_core.c +++ b/net/core/sysctl_net_core.c @@ -33,7 +33,7 @@ static int int_3600 = 3600; static int min_sndbuf = SOCK_MIN_SNDBUF; static int min_rcvbuf = SOCK_MIN_RCVBUF; static int max_skb_frags = MAX_SKB_FRAGS; -static int min_mem_pcpu_rsv = SK_MEMORY_PCPU_RESERVE; +static DEFINE_GLOBAL_PAGE_SIZE_VAR(int, min_mem_pcpu_rsv, SK_MEMORY_PCPU_RESERVE); static int net_msg_warn; /* Unused, but still a sysctl */ diff --git a/net/sunrpc/cache.c b/net/sunrpc/cache.c index 95ff747061046..4e682c0cd7586 100644 --- a/net/sunrpc/cache.c +++ b/net/sunrpc/cache.c @@ -573,13 +573,14 @@ EXPORT_SYMBOL_GPL(cache_purge); */ #define DFR_HASHSIZE (PAGE_SIZE/sizeof(struct list_head)) +#define DFR_HASHSIZE_MAX (PAGE_SIZE_MAX/sizeof(struct list_head)) #define DFR_HASH(item) ((((long)item)>>4 ^ (((long)item)>>13)) % DFR_HASHSIZE) #define DFR_MAX 300 /* ??? */ static DEFINE_SPINLOCK(cache_defer_lock); static LIST_HEAD(cache_defer_list); -static struct hlist_head cache_defer_hash[DFR_HASHSIZE]; +static struct hlist_head cache_defer_hash[DFR_HASHSIZE_MAX]; static int cache_defer_cnt; static void __unhash_deferred_req(struct cache_deferred_req *dreq) diff --git a/net/unix/af_unix.c b/net/unix/af_unix.c index 0be0dcb07f7b6..1cf9f583358af 100644 --- a/net/unix/af_unix.c +++ b/net/unix/af_unix.c @@ -2024,7 +2024,7 @@ static int unix_dgram_sendmsg(struct socket *sock, struct msghdr *msg, MAX_SKB_FRAGS * PAGE_SIZE); data_len = PAGE_ALIGN(data_len); - BUILD_BUG_ON(SKB_MAX_ALLOC < PAGE_SIZE); + BUG_ON(SKB_MAX_ALLOC < PAGE_SIZE); } skb = sock_alloc_send_pskb(sk, len - data_len, data_len, From patchwork Mon Oct 14 10:58:31 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13834854 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 98642D1624E for ; Mon, 14 Oct 2024 11:57:45 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=2nn418B1fZ6U0bhmNO3CvxvB05kOVNrCxzyBbdDTXlE=; b=LqTu2DW95GtQS9oUqR7F/nM3VF Px3HXYtM9s+xa6ErNwUDxsny+EhS5fpo/Eo5vRlp626baezVV7HvOB4khoGdbKtclZOAylMo44hzH 18I1vsUN/o0XBtxvEBIvqvgPF3CUlYDz3/ppI1tEt7BIuhzHxVVw0UP9oI3AkpTTUzcEEZqOsYoLL 0yJij1I76oaSs+Um824rQEzue8bh4IeuhVLiqFIRWEZG/oq2Fb6SPcFidqnTClo1HDjC7B/JbW5XV 1Xw4l2SsQsLtEIfNqJ3bAXfOQLyvCi7p7Zz6kEh8PosxYgc57t73Cx5JxO6AMx0LysaHuJJB4CmM7 tkotyX0g==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1t0Jhc-000000050Pv-0jAZ; Mon, 14 Oct 2024 11:57:32 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1t0Ioc-00000004nxF-2OSV for linux-arm-kernel@lists.infradead.org; Mon, 14 Oct 2024 11:00:45 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id B723A168F; Mon, 14 Oct 2024 04:01:11 -0700 (PDT) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.27]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id D51813F51B; Mon, 14 Oct 2024 04:00:38 -0700 (PDT) From: Ryan Roberts To: "David S. Miller" , Andrew Morton , Anshuman Khandual , Ard Biesheuvel , Catalin Marinas , David Hildenbrand , Eric Dumazet , Greg Marsden , Ivan Ivanov , Jakub Kicinski , Kalesh Singh , Marc Zyngier , Mark Rutland , Matthias Brugger , Miroslav Benes , Paolo Abeni , Wei Fang , Will Deacon Cc: Ryan Roberts , imx@lists.linux.dev, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, netdev@vger.kernel.org Subject: [RFC PATCH v1 24/57] net: fec: Remove PAGE_SIZE compile-time constant assumption Date: Mon, 14 Oct 2024 11:58:31 +0100 Message-ID: <20241014105912.3207374-24-ryan.roberts@arm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20241014105912.3207374-1-ryan.roberts@arm.com> References: <20241014105514.3206191-1-ryan.roberts@arm.com> <20241014105912.3207374-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241014_040042_712401_6659A0C1 X-CRM114-Status: GOOD ( 14.15 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org To prepare for supporting boot-time page size selection, refactor code to remove assumptions about PAGE_SIZE being compile-time constant. Code intended to be equivalent when compile-time page size is active. Refactored "struct fec_enet_priv_rx_q" to use a flexible array member for "rx_skb_info", since its length depends on PAGE_SIZE. Signed-off-by: Ryan Roberts --- ***NOTE*** Any confused maintainers may want to read the cover note here for context: https://lore.kernel.org/all/20241014105514.3206191-1-ryan.roberts@arm.com/ drivers/net/ethernet/freescale/fec.h | 3 ++- drivers/net/ethernet/freescale/fec_main.c | 5 +++-- 2 files changed, 5 insertions(+), 3 deletions(-) diff --git a/drivers/net/ethernet/freescale/fec.h b/drivers/net/ethernet/freescale/fec.h index a19cb2a786fd2..afc8b3f360555 100644 --- a/drivers/net/ethernet/freescale/fec.h +++ b/drivers/net/ethernet/freescale/fec.h @@ -571,7 +571,6 @@ struct fec_enet_priv_tx_q { struct fec_enet_priv_rx_q { struct bufdesc_prop bd; - struct fec_enet_priv_txrx_info rx_skb_info[RX_RING_SIZE]; /* page_pool */ struct page_pool *page_pool; @@ -580,6 +579,8 @@ struct fec_enet_priv_rx_q { /* rx queue number, in the range 0-7 */ u8 id; + + struct fec_enet_priv_txrx_info rx_skb_info[]; }; struct fec_stop_mode_gpr { diff --git a/drivers/net/ethernet/freescale/fec_main.c b/drivers/net/ethernet/freescale/fec_main.c index a923cb95cdc62..b9214c12d537e 100644 --- a/drivers/net/ethernet/freescale/fec_main.c +++ b/drivers/net/ethernet/freescale/fec_main.c @@ -3339,6 +3339,8 @@ static int fec_enet_alloc_queue(struct net_device *ndev) int i; int ret = 0; struct fec_enet_priv_tx_q *txq; + size_t rxq_sz = struct_size(fep->rx_queue[0], rx_skb_info, RX_RING_SIZE); + for (i = 0; i < fep->num_tx_queues; i++) { txq = kzalloc(sizeof(*txq), GFP_KERNEL); @@ -3364,8 +3366,7 @@ static int fec_enet_alloc_queue(struct net_device *ndev) } for (i = 0; i < fep->num_rx_queues; i++) { - fep->rx_queue[i] = kzalloc(sizeof(*fep->rx_queue[i]), - GFP_KERNEL); + fep->rx_queue[i] = kzalloc(rxq_sz, GFP_KERNEL); if (!fep->rx_queue[i]) { ret = -ENOMEM; goto alloc_failed; From patchwork Mon Oct 14 10:58:32 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13834855 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 65ADDD1624D for ; Mon, 14 Oct 2024 11:59:09 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=PSUHqjoRyP5Y2BnZxqoD+IML1c1ukTJ/pQox4PTBxLQ=; b=kvjQRoj+LNnHHXqSN3qEoh686/ WChSVUxJpcNiT7b0T32YsEbzHsdXSV4gXVu8KlrjkJZ61rAy7b5YU5+2/OAq5qCdCny53qx3i9S5D 4/IWjP+hCu3/VGL5oPRgFA48WbRM/uhYJgfQqJBaWl1Gi0iIMPzG5XoKRoEUnyS+0yDL9YU67asTp s8SRWTIj7qTZ/zZFWGz0T0oA4KcaHkhEa/+KsHWdoEwcliIK0gy0jalXRFCptLlztHFqHFNliFsZB DZS0EHktA//IIXBFUt3JgGpnfyJPgS4DSmzVy9RAjUi6XGux2XzNJZ5v9coBeG4sGPzqlu7BSW9PW +YaOsBZg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1t0Jiz-000000050gb-0bkp; Mon, 14 Oct 2024 11:58:57 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1t0Iog-00000004nzN-0MQ3 for linux-arm-kernel@lists.infradead.org; Mon, 14 Oct 2024 11:00:48 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 2C1911692; Mon, 14 Oct 2024 04:01:15 -0700 (PDT) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.27]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 68E4F3F51B; Mon, 14 Oct 2024 04:00:42 -0700 (PDT) From: Ryan Roberts To: "David S. Miller" , Andrew Morton , Anshuman Khandual , Ard Biesheuvel , Catalin Marinas , David Hildenbrand , Eric Dumazet , Greg Marsden , Ivan Ivanov , Jakub Kicinski , Kalesh Singh , Marc Zyngier , Marcin Wojtas , Mark Rutland , Matthias Brugger , Miroslav Benes , Paolo Abeni , Will Deacon Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, netdev@vger.kernel.org Subject: [RFC PATCH v1 25/57] net: marvell: Remove PAGE_SIZE compile-time constant assumption Date: Mon, 14 Oct 2024 11:58:32 +0100 Message-ID: <20241014105912.3207374-25-ryan.roberts@arm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20241014105912.3207374-1-ryan.roberts@arm.com> References: <20241014105514.3206191-1-ryan.roberts@arm.com> <20241014105912.3207374-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241014_040046_289607_EE48A01A X-CRM114-Status: GOOD ( 14.53 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org To prepare for supporting boot-time page size selection, refactor code to remove assumptions about PAGE_SIZE being compile-time constant. Code intended to be equivalent when compile-time page size is active. Updated sky2 "struct rx_ring_info" member frag_addr[] to contain enough entries for the smallest supported page size. Updated mvneta "struct mvneta_tx_queue" members tso_hdrs[] and tso_hdrs_phys[] to contain enough entries for the smallest supported page size. Signed-off-by: Ryan Roberts --- ***NOTE*** Any confused maintainers may want to read the cover note here for context: https://lore.kernel.org/all/20241014105514.3206191-1-ryan.roberts@arm.com/ drivers/net/ethernet/marvell/mvneta.c | 9 ++++++--- drivers/net/ethernet/marvell/sky2.h | 2 +- 2 files changed, 7 insertions(+), 4 deletions(-) diff --git a/drivers/net/ethernet/marvell/mvneta.c b/drivers/net/ethernet/marvell/mvneta.c index 41894834fb53c..f3ac371d8f3a7 100644 --- a/drivers/net/ethernet/marvell/mvneta.c +++ b/drivers/net/ethernet/marvell/mvneta.c @@ -346,12 +346,15 @@ /* The size of a TSO header page */ #define MVNETA_TSO_PAGE_SIZE (2 * PAGE_SIZE) +#define MVNETA_TSO_PAGE_SIZE_MIN (2 * PAGE_SIZE_MIN) /* Number of TSO headers per page. This should be a power of 2 */ #define MVNETA_TSO_PER_PAGE (MVNETA_TSO_PAGE_SIZE / TSO_HEADER_SIZE) +#define MVNETA_TSO_PER_PAGE_MIN (MVNETA_TSO_PAGE_SIZE_MIN / TSO_HEADER_SIZE) /* Maximum number of TSO header pages */ #define MVNETA_MAX_TSO_PAGES (MVNETA_MAX_TXD / MVNETA_TSO_PER_PAGE) +#define MVNETA_MAX_TSO_PAGES_MAX (MVNETA_MAX_TXD / MVNETA_TSO_PER_PAGE_MIN) /* descriptor aligned size */ #define MVNETA_DESC_ALIGNED_SIZE 32 @@ -696,10 +699,10 @@ struct mvneta_tx_queue { int next_desc_to_proc; /* DMA buffers for TSO headers */ - char *tso_hdrs[MVNETA_MAX_TSO_PAGES]; + char *tso_hdrs[MVNETA_MAX_TSO_PAGES_MAX]; /* DMA address of TSO headers */ - dma_addr_t tso_hdrs_phys[MVNETA_MAX_TSO_PAGES]; + dma_addr_t tso_hdrs_phys[MVNETA_MAX_TSO_PAGES_MAX]; /* Affinity mask for CPUs*/ cpumask_t affinity_mask; @@ -5895,7 +5898,7 @@ static int __init mvneta_driver_init(void) { int ret; - BUILD_BUG_ON_NOT_POWER_OF_2(MVNETA_TSO_PER_PAGE); + BUILD_BUG_ON_NOT_POWER_OF_2(MVNETA_TSO_PER_PAGE_MIN); ret = cpuhp_setup_state_multi(CPUHP_AP_ONLINE_DYN, "net/mvneta:online", mvneta_cpu_online, diff --git a/drivers/net/ethernet/marvell/sky2.h b/drivers/net/ethernet/marvell/sky2.h index 8d0bacf4e49cc..8ee73ae087dfc 100644 --- a/drivers/net/ethernet/marvell/sky2.h +++ b/drivers/net/ethernet/marvell/sky2.h @@ -2195,7 +2195,7 @@ struct rx_ring_info { struct sk_buff *skb; dma_addr_t data_addr; DEFINE_DMA_UNMAP_LEN(data_size); - dma_addr_t frag_addr[ETH_JUMBO_MTU >> PAGE_SHIFT ?: 1]; + dma_addr_t frag_addr[ETH_JUMBO_MTU >> PAGE_SHIFT_MIN ?: 1]; }; enum flow_control { From patchwork Mon Oct 14 10:58:33 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13834856 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 0CE3DD1624E for ; Mon, 14 Oct 2024 12:00:36 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=e7pcQwcyCDB+vF+TDiljSktz52PqeG01ydw3eqf01m8=; b=xMF9zqTAkfdeDlH1HpgJe6qj2f Q9q+TjGkuKK2H7pz+c4o5I9uvCEwPHrreW7L3HZjUsP2iXnyRSXlBoDPWwa+TD58+Q+jTiHBDrheY bj/PU74DYqN3HHLoUMIuzpalieZBTsUm4bl4eVfPp9oeWeCsPu2QS+OyWl2Ls1rWMQpXPXH10Ot7H MvG60N5koxyerS2Bt6a2xGU4F9TW8v9+NqKv4zCzOBdgzB+b3TI5sDZZU4qtDX2saVOMtr9lm415R AarKYJrD/38MrbHoc70Bp7PzMoyv0FhATpO2ceSYqA+/F7sWOf6VF0686b2IbIhdkKW1fT9TR178/ mzqRlVhw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1t0JkL-000000050pu-1ZJj; Mon, 14 Oct 2024 12:00:21 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1t0Ioj-00000004o1B-2Uuu for linux-arm-kernel@lists.infradead.org; Mon, 14 Oct 2024 11:00:51 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id D37AC169C; Mon, 14 Oct 2024 04:01:18 -0700 (PDT) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.27]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id D25203F51B; Mon, 14 Oct 2024 04:00:45 -0700 (PDT) From: Ryan Roberts To: "David S. Miller" , Andrew Morton , Anshuman Khandual , Ard Biesheuvel , Catalin Marinas , David Hildenbrand , Eric Dumazet , Greg Marsden , Ivan Ivanov , Jakub Kicinski , Jijie Shao , Kalesh Singh , Marc Zyngier , Mark Rutland , Matthias Brugger , Miroslav Benes , Paolo Abeni , Salil Mehta , Will Deacon , Yisen Zhuang Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, netdev@vger.kernel.org Subject: [RFC PATCH v1 26/57] net: hns3: Remove PAGE_SIZE compile-time constant assumption Date: Mon, 14 Oct 2024 11:58:33 +0100 Message-ID: <20241014105912.3207374-26-ryan.roberts@arm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20241014105912.3207374-1-ryan.roberts@arm.com> References: <20241014105514.3206191-1-ryan.roberts@arm.com> <20241014105912.3207374-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241014_040049_704493_E13429AB X-CRM114-Status: GOOD ( 11.42 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org To prepare for supporting boot-time page size selection, refactor code to remove assumptions about PAGE_SIZE being compile-time constant. Code intended to be equivalent when compile-time page size is active. Convert a CPP conditional to a C conditional. The compiler will dead code strip when doing a compile-time page size build, for the same end effect. But this will also work with boot-time page size builds. Signed-off-by: Ryan Roberts --- ***NOTE*** Any confused maintainers may want to read the cover note here for context: https://lore.kernel.org/all/20241014105514.3206191-1-ryan.roberts@arm.com/ drivers/net/ethernet/hisilicon/hns3/hns3_enet.h | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/drivers/net/ethernet/hisilicon/hns3/hns3_enet.h b/drivers/net/ethernet/hisilicon/hns3/hns3_enet.h index d36c4ed16d8dd..5e675721b7364 100644 --- a/drivers/net/ethernet/hisilicon/hns3/hns3_enet.h +++ b/drivers/net/ethernet/hisilicon/hns3/hns3_enet.h @@ -681,10 +681,8 @@ static inline bool hns3_nic_resetting(struct net_device *netdev) static inline unsigned int hns3_page_order(struct hns3_enet_ring *ring) { -#if (PAGE_SIZE < 8192) - if (ring->buf_size > (PAGE_SIZE / 2)) + if (PAGE_SIZE < 8192 && ring->buf_size > (PAGE_SIZE / 2)) return 1; -#endif return 0; } From patchwork Mon Oct 14 10:58:34 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13834857 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 1F7E9D1624F for ; Mon, 14 Oct 2024 12:01:56 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=cjEGbvB+Gk2WgjQzVrxDcv3CTXI9xiLVm+8stz+1Wsc=; b=Y4DabAk9/cfjz5CayyvVUkWdwV 1jJbwTmagA26KQYFkUzD3Ml/W1tqxe6j+iN+nX8Sgj5cAmByqXYR/Z0FMEnlK47vzL9ar6KqLcNAj B4zS7WRHw23BavhZAYh2mcG0lGr3y1iv0ZqPy3v+u1A3SQzwUEwuUp/fiiMyV3SYmrKjPUy2f1RQg i4ZsIrAcjLVMCByAI+USiXjjwdM/COeB6hDzrmXTmY7mKZ1DFTVB0WItZ9c+obd+82Mf/EOrh7cqm A1++1thFf6Kj92wHNedMTE0tCGcTaSgiMkxzzwNCdese1eAWA3DDiQuAt5yPdA0HcIespahbB+0Mj w2INatlA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1t0Jlh-000000050z4-1wxN; Mon, 14 Oct 2024 12:01:45 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1t0Ion-00000004o35-0TiI for linux-arm-kernel@lists.infradead.org; Mon, 14 Oct 2024 11:00:54 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 47DC9169E; Mon, 14 Oct 2024 04:01:22 -0700 (PDT) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.27]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 84C1A3F51B; Mon, 14 Oct 2024 04:00:49 -0700 (PDT) From: Ryan Roberts To: "David S. Miller" , Andrew Morton , Anshuman Khandual , Ard Biesheuvel , Catalin Marinas , David Hildenbrand , Eric Dumazet , Greg Marsden , Ivan Ivanov , Jakub Kicinski , Kalesh Singh , Marc Zyngier , Mark Rutland , Matthias Brugger , Miroslav Benes , Paolo Abeni , Will Deacon Cc: Ryan Roberts , intel-wired-lan@lists.osuosl.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, netdev@vger.kernel.org Subject: [RFC PATCH v1 27/57] net: e1000: Remove PAGE_SIZE compile-time constant assumption Date: Mon, 14 Oct 2024 11:58:34 +0100 Message-ID: <20241014105912.3207374-27-ryan.roberts@arm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20241014105912.3207374-1-ryan.roberts@arm.com> References: <20241014105514.3206191-1-ryan.roberts@arm.com> <20241014105912.3207374-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241014_040053_292467_869BC6C0 X-CRM114-Status: GOOD ( 10.93 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org To prepare for supporting boot-time page size selection, refactor code to remove assumptions about PAGE_SIZE being compile-time constant. Code intended to be equivalent when compile-time page size is active. Convert CPP conditionals to C conditionals. The compiler will dead code strip when doing a compile-time page size build, for the same end effect. But this will also work with boot-time page size builds. Signed-off-by: Ryan Roberts --- ***NOTE*** Any confused maintainers may want to read the cover note here for context: https://lore.kernel.org/all/20241014105514.3206191-1-ryan.roberts@arm.com/ drivers/net/ethernet/intel/e1000/e1000_main.c | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/drivers/net/ethernet/intel/e1000/e1000_main.c b/drivers/net/ethernet/intel/e1000/e1000_main.c index ab7ae418d2948..cc14788f5bb04 100644 --- a/drivers/net/ethernet/intel/e1000/e1000_main.c +++ b/drivers/net/ethernet/intel/e1000/e1000_main.c @@ -3553,12 +3553,10 @@ static int e1000_change_mtu(struct net_device *netdev, int new_mtu) if (max_frame <= E1000_RXBUFFER_2048) adapter->rx_buffer_len = E1000_RXBUFFER_2048; - else -#if (PAGE_SIZE >= E1000_RXBUFFER_16384) + else if (PAGE_SIZE >= E1000_RXBUFFER_16384) adapter->rx_buffer_len = E1000_RXBUFFER_16384; -#elif (PAGE_SIZE >= E1000_RXBUFFER_4096) + else if (PAGE_SIZE >= E1000_RXBUFFER_4096) adapter->rx_buffer_len = PAGE_SIZE; -#endif /* adjust allocation if LPE protects us, and we aren't using SBP */ if (!hw->tbi_compatibility_on && From patchwork Mon Oct 14 10:58:35 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13834858 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 99BC1D1624E for ; Mon, 14 Oct 2024 12:03:25 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=I1DzdKxvBfT38zdagRbaKNLNUfAtWjrd/PCrV+EF/6Q=; b=sdn8OTCCKIVXMLL7dkoW9TnxYk OhOnZmUMOYHnRPiUGcrDoH5wjhqE3hrN+mHv9X8jkFn7wqTY2z05E6b283hCn/nXzflYwowJRXIz+ J91unb/2I7HtRrK4APHGNQELtxv4zk63fQcQtz9x7wD5VuLL/GPPZA/Itx9aPbpptoGSZVFNaInHJ pO5ntm+OqmzmOstsKK1sXJ3BqxCjrXLhfHJSLGO0F/jf8+bdN7r+bjIogmkKC9WR2Iucuw1OkYdw7 dg5SjDanmuIfpOJi9tqh8W/A+mCmpTN+/dnjNknzIGmwm9dbwfKQJL9acW+azpk5qAXA5L5yeZUDY RzlyJ6ig==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1t0Jn5-000000051Bd-0Ta7; Mon, 14 Oct 2024 12:03:11 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1t0Ioq-00000004o4R-1x5Y for linux-arm-kernel@lists.infradead.org; Mon, 14 Oct 2024 11:00:57 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id B102016F2; Mon, 14 Oct 2024 04:01:25 -0700 (PDT) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.27]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id EDCA03F51B; Mon, 14 Oct 2024 04:00:52 -0700 (PDT) From: Ryan Roberts To: "David S. Miller" , Andrew Morton , Anshuman Khandual , Ard Biesheuvel , Catalin Marinas , David Hildenbrand , Eric Dumazet , Greg Marsden , Ivan Ivanov , Jakub Kicinski , Kalesh Singh , Marc Zyngier , Mark Rutland , Matthias Brugger , Miroslav Benes , Paolo Abeni , Will Deacon Cc: Ryan Roberts , intel-wired-lan@lists.osuosl.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, netdev@vger.kernel.org Subject: [RFC PATCH v1 28/57] net: igbvf: Remove PAGE_SIZE compile-time constant assumption Date: Mon, 14 Oct 2024 11:58:35 +0100 Message-ID: <20241014105912.3207374-28-ryan.roberts@arm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20241014105912.3207374-1-ryan.roberts@arm.com> References: <20241014105514.3206191-1-ryan.roberts@arm.com> <20241014105912.3207374-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241014_040056_574492_441D13A8 X-CRM114-Status: GOOD ( 11.11 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org To prepare for supporting boot-time page size selection, refactor code to remove assumptions about PAGE_SIZE being compile-time constant. Code intended to be equivalent when compile-time page size is active. Convert CPP conditionals to C conditionals. The compiler will dead code strip when doing a compile-time page size build, for the same end effect. But this will also work with boot-time page size builds. Signed-off-by: Ryan Roberts --- ***NOTE*** Any confused maintainers may want to read the cover note here for context: https://lore.kernel.org/all/20241014105514.3206191-1-ryan.roberts@arm.com/ drivers/net/ethernet/intel/igbvf/netdev.c | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/drivers/net/ethernet/intel/igbvf/netdev.c b/drivers/net/ethernet/intel/igbvf/netdev.c index 925d7286a8ee4..2e11d999168de 100644 --- a/drivers/net/ethernet/intel/igbvf/netdev.c +++ b/drivers/net/ethernet/intel/igbvf/netdev.c @@ -2419,12 +2419,10 @@ static int igbvf_change_mtu(struct net_device *netdev, int new_mtu) adapter->rx_buffer_len = 1024; else if (max_frame <= 2048) adapter->rx_buffer_len = 2048; - else -#if (PAGE_SIZE / 2) > 16384 + else if ((PAGE_SIZE / 2) > 16384) adapter->rx_buffer_len = 16384; -#else + else adapter->rx_buffer_len = PAGE_SIZE / 2; -#endif /* adjust allocation if LPE protects us, and we aren't using SBP */ if ((max_frame == ETH_FRAME_LEN + ETH_FCS_LEN) || From patchwork Mon Oct 14 10:58:36 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13834859 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 21FAFD1624E for ; Mon, 14 Oct 2024 12:04:48 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=r9lKqnlxiymaFGc73YACDsyodHkCfWFElNljnle06Lc=; b=eE4npaG+3j0/L448pwdYhdSMMI NRcM8ihQQgj5WGDlUbfdpAjGVA/4pLnUqg565/jOtHgnfYU4Yd/pXNkSzUPy3ur/g/D5MiL3llMze oibNu9Hm7KIR0kQ12jBv+XlqRKwKy48Jrt+8swh3exrsdJgiYsoix9AbXCAa/HJCDMb/mQOnmWR4t caHOzzTCYdZsjqm8EA/rghUns+TBmz9u1ILY01jUfFCTw6dfJuQMf394pdz98x6DYYmRsEnkT6lNW Ov6raJyFCF3c3Nnr5686+WGV1439lNRyCnQXhe11HH+yi8YS656k6Ich37bfB7H7wB/eEyLaOAID6 THrSAwJw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1t0JoR-000000051Nb-18cH; Mon, 14 Oct 2024 12:04:35 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1t0Iou-00000004o6i-065b for linux-arm-kernel@lists.infradead.org; Mon, 14 Oct 2024 11:01:05 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 45F9016A3; Mon, 14 Oct 2024 04:01:29 -0700 (PDT) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.27]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 635553F51B; Mon, 14 Oct 2024 04:00:56 -0700 (PDT) From: Ryan Roberts To: "David S. Miller" , Andrew Morton , Anshuman Khandual , Ard Biesheuvel , Catalin Marinas , David Hildenbrand , Eric Dumazet , Greg Marsden , Ivan Ivanov , Jakub Kicinski , Kalesh Singh , Marc Zyngier , Mark Rutland , Matthias Brugger , Miroslav Benes , Paolo Abeni , Will Deacon Cc: Ryan Roberts , bpf@vger.kernel.org, intel-wired-lan@lists.osuosl.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, netdev@vger.kernel.org Subject: [RFC PATCH v1 29/57] net: igb: Remove PAGE_SIZE compile-time constant assumption Date: Mon, 14 Oct 2024 11:58:36 +0100 Message-ID: <20241014105912.3207374-29-ryan.roberts@arm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20241014105912.3207374-1-ryan.roberts@arm.com> References: <20241014105514.3206191-1-ryan.roberts@arm.com> <20241014105912.3207374-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241014_040100_177850_BA51C6A8 X-CRM114-Status: GOOD ( 24.36 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org To prepare for supporting boot-time page size selection, refactor code to remove assumptions about PAGE_SIZE being compile-time constant. Code intended to be equivalent when compile-time page size is active. Convert CPP conditionals to C conditionals. The compiler will dead code strip when doing a compile-time page size build, for the same end effect. But this will also work with boot-time page size builds. Signed-off-by: Ryan Roberts --- ***NOTE*** Any confused maintainers may want to read the cover note here for context: https://lore.kernel.org/all/20241014105514.3206191-1-ryan.roberts@arm.com/ drivers/net/ethernet/intel/igb/igb.h | 25 ++-- drivers/net/ethernet/intel/igb/igb_main.c | 149 +++++++++++----------- 2 files changed, 82 insertions(+), 92 deletions(-) diff --git a/drivers/net/ethernet/intel/igb/igb.h b/drivers/net/ethernet/intel/igb/igb.h index 3c2dc7bdebb50..04aeebcd363b3 100644 --- a/drivers/net/ethernet/intel/igb/igb.h +++ b/drivers/net/ethernet/intel/igb/igb.h @@ -158,7 +158,6 @@ struct vf_mac_filter { * up negative. In these cases we should fall back to the 3K * buffers. */ -#if (PAGE_SIZE < 8192) #define IGB_MAX_FRAME_BUILD_SKB (IGB_RXBUFFER_1536 - NET_IP_ALIGN) #define IGB_2K_TOO_SMALL_WITH_PADDING \ ((NET_SKB_PAD + IGB_TS_HDR_LEN + IGB_RXBUFFER_1536) > SKB_WITH_OVERHEAD(IGB_RXBUFFER_2048)) @@ -177,6 +176,9 @@ static inline int igb_skb_pad(void) { int rx_buf_len; + if (PAGE_SIZE >= 8192) + return NET_SKB_PAD + NET_IP_ALIGN; + /* If a 2K buffer cannot handle a standard Ethernet frame then * optimize padding for a 3K buffer instead of a 1.5K buffer. * @@ -196,9 +198,6 @@ static inline int igb_skb_pad(void) } #define IGB_SKB_PAD igb_skb_pad() -#else -#define IGB_SKB_PAD (NET_SKB_PAD + NET_IP_ALIGN) -#endif /* How many Rx Buffers do we bundle into one write to the hardware ? */ #define IGB_RX_BUFFER_WRITE 16 /* Must be power of 2 */ @@ -280,7 +279,7 @@ struct igb_tx_buffer { struct igb_rx_buffer { dma_addr_t dma; struct page *page; -#if (BITS_PER_LONG > 32) || (PAGE_SIZE >= 65536) +#if (BITS_PER_LONG > 32) || (PAGE_SIZE_MAX >= 65536) __u32 page_offset; #else __u16 page_offset; @@ -403,22 +402,20 @@ enum e1000_ring_flags_t { static inline unsigned int igb_rx_bufsz(struct igb_ring *ring) { -#if (PAGE_SIZE < 8192) - if (ring_uses_large_buffer(ring)) - return IGB_RXBUFFER_3072; + if (PAGE_SIZE < 8192) { + if (ring_uses_large_buffer(ring)) + return IGB_RXBUFFER_3072; - if (ring_uses_build_skb(ring)) - return IGB_MAX_FRAME_BUILD_SKB; -#endif + if (ring_uses_build_skb(ring)) + return IGB_MAX_FRAME_BUILD_SKB; + } return IGB_RXBUFFER_2048; } static inline unsigned int igb_rx_pg_order(struct igb_ring *ring) { -#if (PAGE_SIZE < 8192) - if (ring_uses_large_buffer(ring)) + if (PAGE_SIZE < 8192 && ring_uses_large_buffer(ring)) return 1; -#endif return 0; } diff --git a/drivers/net/ethernet/intel/igb/igb_main.c b/drivers/net/ethernet/intel/igb/igb_main.c index 1ef4cb871452a..4f2c53dece1a2 100644 --- a/drivers/net/ethernet/intel/igb/igb_main.c +++ b/drivers/net/ethernet/intel/igb/igb_main.c @@ -4797,9 +4797,7 @@ void igb_configure_rx_ring(struct igb_adapter *adapter, static void igb_set_rx_buffer_len(struct igb_adapter *adapter, struct igb_ring *rx_ring) { -#if (PAGE_SIZE < 8192) struct e1000_hw *hw = &adapter->hw; -#endif /* set build_skb and buffer size flags */ clear_ring_build_skb_enabled(rx_ring); @@ -4810,12 +4808,11 @@ static void igb_set_rx_buffer_len(struct igb_adapter *adapter, set_ring_build_skb_enabled(rx_ring); -#if (PAGE_SIZE < 8192) - if (adapter->max_frame_size > IGB_MAX_FRAME_BUILD_SKB || + if (PAGE_SIZE < 8192 && + (adapter->max_frame_size > IGB_MAX_FRAME_BUILD_SKB || IGB_2K_TOO_SMALL_WITH_PADDING || - rd32(E1000_RCTL) & E1000_RCTL_SBP) + rd32(E1000_RCTL) & E1000_RCTL_SBP)) set_ring_uses_large_buffer(rx_ring); -#endif } /** @@ -5314,12 +5311,10 @@ static void igb_set_rx_mode(struct net_device *netdev) E1000_RCTL_VFE); wr32(E1000_RCTL, rctl); -#if (PAGE_SIZE < 8192) - if (!adapter->vfs_allocated_count) { + if (PAGE_SIZE < 8192 && !adapter->vfs_allocated_count) { if (adapter->max_frame_size <= IGB_MAX_FRAME_BUILD_SKB) rlpml = IGB_MAX_FRAME_BUILD_SKB; } -#endif wr32(E1000_RLPML, rlpml); /* In order to support SR-IOV and eventually VMDq it is necessary to set @@ -5338,11 +5333,10 @@ static void igb_set_rx_mode(struct net_device *netdev) /* enable Rx jumbo frames, restrict as needed to support build_skb */ vmolr &= ~E1000_VMOLR_RLPML_MASK; -#if (PAGE_SIZE < 8192) - if (adapter->max_frame_size <= IGB_MAX_FRAME_BUILD_SKB) + if (PAGE_SIZE < 8192 && + adapter->max_frame_size <= IGB_MAX_FRAME_BUILD_SKB) vmolr |= IGB_MAX_FRAME_BUILD_SKB; else -#endif vmolr |= MAX_JUMBO_FRAME_SIZE; vmolr |= E1000_VMOLR_LPE; @@ -8435,17 +8429,17 @@ static bool igb_can_reuse_rx_page(struct igb_rx_buffer *rx_buffer, if (!dev_page_is_reusable(page)) return false; -#if (PAGE_SIZE < 8192) - /* if we are only owner of page we can reuse it */ - if (unlikely((rx_buf_pgcnt - pagecnt_bias) > 1)) - return false; -#else + if (PAGE_SIZE < 8192) { + /* if we are only owner of page we can reuse it */ + if (unlikely((rx_buf_pgcnt - pagecnt_bias) > 1)) + return false; + } else { #define IGB_LAST_OFFSET \ (SKB_WITH_OVERHEAD(PAGE_SIZE) - IGB_RXBUFFER_2048) - if (rx_buffer->page_offset > IGB_LAST_OFFSET) - return false; -#endif + if (rx_buffer->page_offset > IGB_LAST_OFFSET) + return false; + } /* If we have drained the page fragment pool we need to update * the pagecnt_bias and page count so that we fully restock the @@ -8473,20 +8467,22 @@ static void igb_add_rx_frag(struct igb_ring *rx_ring, struct sk_buff *skb, unsigned int size) { -#if (PAGE_SIZE < 8192) - unsigned int truesize = igb_rx_pg_size(rx_ring) / 2; -#else - unsigned int truesize = ring_uses_build_skb(rx_ring) ? + unsigned int truesize; + + if (PAGE_SIZE < 8192) + truesize = igb_rx_pg_size(rx_ring) / 2; + else + truesize = ring_uses_build_skb(rx_ring) ? SKB_DATA_ALIGN(IGB_SKB_PAD + size) : SKB_DATA_ALIGN(size); -#endif + skb_add_rx_frag(skb, skb_shinfo(skb)->nr_frags, rx_buffer->page, rx_buffer->page_offset, size, truesize); -#if (PAGE_SIZE < 8192) - rx_buffer->page_offset ^= truesize; -#else - rx_buffer->page_offset += truesize; -#endif + + if (PAGE_SIZE < 8192) + rx_buffer->page_offset ^= truesize; + else + rx_buffer->page_offset += truesize; } static struct sk_buff *igb_construct_skb(struct igb_ring *rx_ring, @@ -8494,16 +8490,16 @@ static struct sk_buff *igb_construct_skb(struct igb_ring *rx_ring, struct xdp_buff *xdp, ktime_t timestamp) { -#if (PAGE_SIZE < 8192) - unsigned int truesize = igb_rx_pg_size(rx_ring) / 2; -#else - unsigned int truesize = SKB_DATA_ALIGN(xdp->data_end - - xdp->data_hard_start); -#endif unsigned int size = xdp->data_end - xdp->data; + unsigned int truesize; unsigned int headlen; struct sk_buff *skb; + if (PAGE_SIZE < 8192) + truesize = igb_rx_pg_size(rx_ring) / 2; + else + truesize = SKB_DATA_ALIGN(xdp->data_end - xdp->data_hard_start); + /* prefetch first cache line of first page */ net_prefetch(xdp->data); @@ -8529,11 +8525,10 @@ static struct sk_buff *igb_construct_skb(struct igb_ring *rx_ring, skb_add_rx_frag(skb, 0, rx_buffer->page, (xdp->data + headlen) - page_address(rx_buffer->page), size, truesize); -#if (PAGE_SIZE < 8192) - rx_buffer->page_offset ^= truesize; -#else - rx_buffer->page_offset += truesize; -#endif + if (PAGE_SIZE < 8192) + rx_buffer->page_offset ^= truesize; + else + rx_buffer->page_offset += truesize; } else { rx_buffer->pagecnt_bias++; } @@ -8546,16 +8541,17 @@ static struct sk_buff *igb_build_skb(struct igb_ring *rx_ring, struct xdp_buff *xdp, ktime_t timestamp) { -#if (PAGE_SIZE < 8192) - unsigned int truesize = igb_rx_pg_size(rx_ring) / 2; -#else - unsigned int truesize = SKB_DATA_ALIGN(sizeof(struct skb_shared_info)) + - SKB_DATA_ALIGN(xdp->data_end - - xdp->data_hard_start); -#endif unsigned int metasize = xdp->data - xdp->data_meta; + unsigned int truesize; struct sk_buff *skb; + if (PAGE_SIZE < 8192) + truesize = igb_rx_pg_size(rx_ring) / 2; + else + truesize = SKB_DATA_ALIGN(sizeof(struct skb_shared_info)) + + SKB_DATA_ALIGN(xdp->data_end - + xdp->data_hard_start); + /* prefetch first cache line of first page */ net_prefetch(xdp->data_meta); @@ -8575,11 +8571,10 @@ static struct sk_buff *igb_build_skb(struct igb_ring *rx_ring, skb_hwtstamps(skb)->hwtstamp = timestamp; /* update buffer offset */ -#if (PAGE_SIZE < 8192) - rx_buffer->page_offset ^= truesize; -#else - rx_buffer->page_offset += truesize; -#endif + if (PAGE_SIZE < 8192) + rx_buffer->page_offset ^= truesize; + else + rx_buffer->page_offset += truesize; return skb; } @@ -8634,14 +8629,14 @@ static unsigned int igb_rx_frame_truesize(struct igb_ring *rx_ring, { unsigned int truesize; -#if (PAGE_SIZE < 8192) - truesize = igb_rx_pg_size(rx_ring) / 2; /* Must be power-of-2 */ -#else - truesize = ring_uses_build_skb(rx_ring) ? - SKB_DATA_ALIGN(IGB_SKB_PAD + size) + - SKB_DATA_ALIGN(sizeof(struct skb_shared_info)) : - SKB_DATA_ALIGN(size); -#endif + if (PAGE_SIZE < 8192) + truesize = igb_rx_pg_size(rx_ring) / 2; /* Must be power-of-2 */ + else + truesize = ring_uses_build_skb(rx_ring) ? + SKB_DATA_ALIGN(IGB_SKB_PAD + size) + + SKB_DATA_ALIGN(sizeof(struct skb_shared_info)) : + SKB_DATA_ALIGN(size); + return truesize; } @@ -8650,11 +8645,11 @@ static void igb_rx_buffer_flip(struct igb_ring *rx_ring, unsigned int size) { unsigned int truesize = igb_rx_frame_truesize(rx_ring, size); -#if (PAGE_SIZE < 8192) - rx_buffer->page_offset ^= truesize; -#else - rx_buffer->page_offset += truesize; -#endif + + if (PAGE_SIZE < 8192) + rx_buffer->page_offset ^= truesize; + else + rx_buffer->page_offset += truesize; } static inline void igb_rx_checksum(struct igb_ring *ring, @@ -8825,12 +8820,12 @@ static struct igb_rx_buffer *igb_get_rx_buffer(struct igb_ring *rx_ring, struct igb_rx_buffer *rx_buffer; rx_buffer = &rx_ring->rx_buffer_info[rx_ring->next_to_clean]; - *rx_buf_pgcnt = -#if (PAGE_SIZE < 8192) - page_count(rx_buffer->page); -#else - 0; -#endif + + if (PAGE_SIZE < 8192) + *rx_buf_pgcnt = page_count(rx_buffer->page); + else + *rx_buf_pgcnt = 0; + prefetchw(rx_buffer->page); /* we are reusing so sync this buffer for CPU use */ @@ -8881,9 +8876,8 @@ static int igb_clean_rx_irq(struct igb_q_vector *q_vector, const int budget) int rx_buf_pgcnt; /* Frame size depend on rx_ring setup when PAGE_SIZE=4K */ -#if (PAGE_SIZE < 8192) - frame_sz = igb_rx_frame_truesize(rx_ring, 0); -#endif + if (PAGE_SIZE < 8192) + frame_sz = igb_rx_frame_truesize(rx_ring, 0); xdp_init_buff(&xdp, frame_sz, &rx_ring->xdp_rxq); while (likely(total_packets < budget)) { @@ -8932,10 +8926,9 @@ static int igb_clean_rx_irq(struct igb_q_vector *q_vector, const int budget) xdp_prepare_buff(&xdp, hard_start, offset, size, true); xdp_buff_clear_frags_flag(&xdp); -#if (PAGE_SIZE > 4096) /* At larger PAGE_SIZE, frame_sz depend on len size */ - xdp.frame_sz = igb_rx_frame_truesize(rx_ring, size); -#endif + if (PAGE_SIZE > 4096) + xdp.frame_sz = igb_rx_frame_truesize(rx_ring, size); skb = igb_run_xdp(adapter, rx_ring, &xdp); } From patchwork Mon Oct 14 10:58:37 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13834860 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 14509D1624F for ; Mon, 14 Oct 2024 12:06:15 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=S0gIad1IKQmWm9pceFZyemJEemJ3RuiyqEiPFLVA8hg=; b=ktVphqcC6B5u4+eLEuZWVJvLbj n5BM566xfvf3xR4jug2rhQfA2OcVyqJloBwc/l0qSuHgTYtwoFDIyl59usvehAiQ5HwHrPF4P+d0C kgi59qZTpCvgctnWWB7KCLC5ShoW0AF8sfqDeQTwnocICRMdGx4hSSnHcZyEXMIyS7YpWL8knsWmX Gg1iq4DkMPFK+McR7YB07e9wwSR0CbJ3i2ytDUD1QO3r7nfmtlu5f9XyioCJi+33CSrxrfi9IENPu TR61XTvSIaRQhjbGAp7ntE3abyNozvBXFIXkSfQtOnwifw5iklL4oVXUFKNWv1xHpZshmakfWIE2H 2cX6OqSA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1t0Jpp-000000051dn-2zA4; Mon, 14 Oct 2024 12:06:01 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1t0Iow-00000004o80-3Dmc for linux-arm-kernel@lists.infradead.org; Mon, 14 Oct 2024 11:01:06 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 1454116F3; Mon, 14 Oct 2024 04:01:32 -0700 (PDT) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.27]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id EAD463F51B; Mon, 14 Oct 2024 04:00:59 -0700 (PDT) From: Ryan Roberts To: Andrew Morton , Anshuman Khandual , Ard Biesheuvel , Catalin Marinas , David Hildenbrand , Greg Marsden , Ivan Ivanov , Kalesh Singh , Marc Zyngier , Mark Rutland , Matthias Brugger , Miroslav Benes , Will Deacon , Yury Norov Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [RFC PATCH v1 30/57] drivers/base: Remove PAGE_SIZE compile-time constant assumption Date: Mon, 14 Oct 2024 11:58:37 +0100 Message-ID: <20241014105912.3207374-30-ryan.roberts@arm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20241014105912.3207374-1-ryan.roberts@arm.com> References: <20241014105514.3206191-1-ryan.roberts@arm.com> <20241014105912.3207374-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241014_040105_036108_6892937F X-CRM114-Status: GOOD ( 12.49 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org To prepare for supporting boot-time page size selection, refactor code to remove assumptions about PAGE_SIZE being compile-time constant. Code intended to be equivalent when compile-time page size is active. Update BUILD_BUG_ON() to test against page size limits. CPUMAP_FILE_MAX_BYTES and CPULIST_FILE_MAX_BYTES are both defined relative to PAGE_SIZE, so when these values are assigned to global variables via BIN_ATTR_RO(), let's wrap them with DEFINE_GLOBAL_PAGE_SIZE_VAR() so that their assignment can be deferred until boot-time. Signed-off-by: Ryan Roberts --- ***NOTE*** Any confused maintainers may want to read the cover note here for context: https://lore.kernel.org/all/20241014105514.3206191-1-ryan.roberts@arm.com/ drivers/base/node.c | 6 +++--- drivers/base/topology.c | 32 ++++++++++++++++---------------- include/linux/cpumask.h | 5 +++++ 3 files changed, 24 insertions(+), 19 deletions(-) diff --git a/drivers/base/node.c b/drivers/base/node.c index eb72580288e62..30e6549e4c438 100644 --- a/drivers/base/node.c +++ b/drivers/base/node.c @@ -45,7 +45,7 @@ static inline ssize_t cpumap_read(struct file *file, struct kobject *kobj, return n; } -static BIN_ATTR_RO(cpumap, CPUMAP_FILE_MAX_BYTES); +static CPU_FILE_BIN_ATTR_RO(cpumap, CPUMAP_FILE_MAX_BYTES); static inline ssize_t cpulist_read(struct file *file, struct kobject *kobj, struct bin_attribute *attr, char *buf, @@ -66,7 +66,7 @@ static inline ssize_t cpulist_read(struct file *file, struct kobject *kobj, return n; } -static BIN_ATTR_RO(cpulist, CPULIST_FILE_MAX_BYTES); +static CPU_FILE_BIN_ATTR_RO(cpulist, CPULIST_FILE_MAX_BYTES); /** * struct node_access_nodes - Access class device to hold user visible @@ -558,7 +558,7 @@ static ssize_t node_read_distance(struct device *dev, * buf is currently PAGE_SIZE in length and each node needs 4 chars * at the most (distance + space or newline). */ - BUILD_BUG_ON(MAX_NUMNODES * 4 > PAGE_SIZE); + BUILD_BUG_ON(MAX_NUMNODES * 4 > PAGE_SIZE_MIN); for_each_online_node(i) { len += sysfs_emit_at(buf, len, "%s%d", diff --git a/drivers/base/topology.c b/drivers/base/topology.c index 89f98be5c5b99..bdbdbefd95b15 100644 --- a/drivers/base/topology.c +++ b/drivers/base/topology.c @@ -62,47 +62,47 @@ define_id_show_func(ppin, "0x%llx"); static DEVICE_ATTR_ADMIN_RO(ppin); define_siblings_read_func(thread_siblings, sibling_cpumask); -static BIN_ATTR_RO(thread_siblings, CPUMAP_FILE_MAX_BYTES); -static BIN_ATTR_RO(thread_siblings_list, CPULIST_FILE_MAX_BYTES); +static CPU_FILE_BIN_ATTR_RO(thread_siblings, CPUMAP_FILE_MAX_BYTES); +static CPU_FILE_BIN_ATTR_RO(thread_siblings_list, CPULIST_FILE_MAX_BYTES); define_siblings_read_func(core_cpus, sibling_cpumask); -static BIN_ATTR_RO(core_cpus, CPUMAP_FILE_MAX_BYTES); -static BIN_ATTR_RO(core_cpus_list, CPULIST_FILE_MAX_BYTES); +static CPU_FILE_BIN_ATTR_RO(core_cpus, CPUMAP_FILE_MAX_BYTES); +static CPU_FILE_BIN_ATTR_RO(core_cpus_list, CPULIST_FILE_MAX_BYTES); define_siblings_read_func(core_siblings, core_cpumask); -static BIN_ATTR_RO(core_siblings, CPUMAP_FILE_MAX_BYTES); -static BIN_ATTR_RO(core_siblings_list, CPULIST_FILE_MAX_BYTES); +static CPU_FILE_BIN_ATTR_RO(core_siblings, CPUMAP_FILE_MAX_BYTES); +static CPU_FILE_BIN_ATTR_RO(core_siblings_list, CPULIST_FILE_MAX_BYTES); #ifdef TOPOLOGY_CLUSTER_SYSFS define_siblings_read_func(cluster_cpus, cluster_cpumask); -static BIN_ATTR_RO(cluster_cpus, CPUMAP_FILE_MAX_BYTES); -static BIN_ATTR_RO(cluster_cpus_list, CPULIST_FILE_MAX_BYTES); +static CPU_FILE_BIN_ATTR_RO(cluster_cpus, CPUMAP_FILE_MAX_BYTES); +static CPU_FILE_BIN_ATTR_RO(cluster_cpus_list, CPULIST_FILE_MAX_BYTES); #endif #ifdef TOPOLOGY_DIE_SYSFS define_siblings_read_func(die_cpus, die_cpumask); -static BIN_ATTR_RO(die_cpus, CPUMAP_FILE_MAX_BYTES); -static BIN_ATTR_RO(die_cpus_list, CPULIST_FILE_MAX_BYTES); +static CPU_FILE_BIN_ATTR_RO(die_cpus, CPUMAP_FILE_MAX_BYTES); +static CPU_FILE_BIN_ATTR_RO(die_cpus_list, CPULIST_FILE_MAX_BYTES); #endif define_siblings_read_func(package_cpus, core_cpumask); -static BIN_ATTR_RO(package_cpus, CPUMAP_FILE_MAX_BYTES); -static BIN_ATTR_RO(package_cpus_list, CPULIST_FILE_MAX_BYTES); +static CPU_FILE_BIN_ATTR_RO(package_cpus, CPUMAP_FILE_MAX_BYTES); +static CPU_FILE_BIN_ATTR_RO(package_cpus_list, CPULIST_FILE_MAX_BYTES); #ifdef TOPOLOGY_BOOK_SYSFS define_id_show_func(book_id, "%d"); static DEVICE_ATTR_RO(book_id); define_siblings_read_func(book_siblings, book_cpumask); -static BIN_ATTR_RO(book_siblings, CPUMAP_FILE_MAX_BYTES); -static BIN_ATTR_RO(book_siblings_list, CPULIST_FILE_MAX_BYTES); +static CPU_FILE_BIN_ATTR_RO(book_siblings, CPUMAP_FILE_MAX_BYTES); +static CPU_FILE_BIN_ATTR_RO(book_siblings_list, CPULIST_FILE_MAX_BYTES); #endif #ifdef TOPOLOGY_DRAWER_SYSFS define_id_show_func(drawer_id, "%d"); static DEVICE_ATTR_RO(drawer_id); define_siblings_read_func(drawer_siblings, drawer_cpumask); -static BIN_ATTR_RO(drawer_siblings, CPUMAP_FILE_MAX_BYTES); -static BIN_ATTR_RO(drawer_siblings_list, CPULIST_FILE_MAX_BYTES); +static CPU_FILE_BIN_ATTR_RO(drawer_siblings, CPUMAP_FILE_MAX_BYTES); +static CPU_FILE_BIN_ATTR_RO(drawer_siblings_list, CPULIST_FILE_MAX_BYTES); #endif static struct bin_attribute *bin_attrs[] = { diff --git a/include/linux/cpumask.h b/include/linux/cpumask.h index 53158de44b837..f654b4198abc2 100644 --- a/include/linux/cpumask.h +++ b/include/linux/cpumask.h @@ -1292,4 +1292,9 @@ cpumap_print_list_to_buf(char *buf, const struct cpumask *mask, ? (NR_CPUS * 9)/32 - 1 : PAGE_SIZE) #define CPULIST_FILE_MAX_BYTES (((NR_CPUS * 7)/2 > PAGE_SIZE) ? (NR_CPUS * 7)/2 : PAGE_SIZE) +#define CPU_FILE_BIN_ATTR_RO(_name, _size) \ + DEFINE_GLOBAL_PAGE_SIZE_VAR(struct bin_attribute, \ + bin_attr_##_name, \ + __BIN_ATTR_RO(_name, _size)) + #endif /* __LINUX_CPUMASK_H */ From patchwork Mon Oct 14 10:58:38 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13834862 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 33400D1624F for ; Mon, 14 Oct 2024 12:07:33 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=xJWf5Towi95SWggcq2L3OEdCfoka1zUWhvozEl9HTkI=; b=08Jp/dqnQI2PNt4/LkkM4UOuaC MowcFfREe79Zk+z8SNkOB1QeV2b/LqdFlDaXVHsS1qHSEJRyiAYZKGXOvnP5cUFzEoVrQ066drFeo hstGZ6xxNwzE3e0nwciug4AB9ME+Cn2Nsy3qumuUp84WZZLqYjTCAE0w65MOJ/iIZfEcChLLYY8em YRD/eO2h4PQxWJa5XTC722I8ANNGWrKAydFl5sGke85x1E0SSclTLt76vrTaZo1l0oVWK4bfIiU9l A5wlwvKCq3cms4WUTnpbnc8wJ3ceR6W9l5yZE6dG/craBz3vrLQDsPPZthaOo8jxI36EjpZOQvHJ/ sgmSbrpw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1t0JrA-000000051kl-2OKk; Mon, 14 Oct 2024 12:07:24 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1t0Ioz-00000004o8q-3406 for linux-arm-kernel@lists.infradead.org; Mon, 14 Oct 2024 11:01:07 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id D88E716F8; Mon, 14 Oct 2024 04:01:34 -0700 (PDT) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.27]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id B95D73F51B; Mon, 14 Oct 2024 04:01:02 -0700 (PDT) From: Ryan Roberts To: Andrew Morton , Anshuman Khandual , Ard Biesheuvel , Catalin Marinas , David Hildenbrand , Greg Marsden , Ivan Ivanov , Kalesh Singh , Marc Zyngier , Mark Rutland , Matthias Brugger , Miroslav Benes , Will Deacon Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, linux-edac@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [RFC PATCH v1 31/57] edac: Remove PAGE_SIZE compile-time constant assumption Date: Mon, 14 Oct 2024 11:58:38 +0100 Message-ID: <20241014105912.3207374-31-ryan.roberts@arm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20241014105912.3207374-1-ryan.roberts@arm.com> References: <20241014105514.3206191-1-ryan.roberts@arm.com> <20241014105912.3207374-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241014_040106_025076_4390EDB3 X-CRM114-Status: GOOD ( 11.15 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org To prepare for supporting boot-time page size selection, refactor code to remove assumptions about PAGE_SIZE being compile-time constant. Code intended to be equivalent when compile-time page size is active. Convert PAGES_TO_MiB() and MiB_TO_PAGES() to use the ternary operator so that they continue to work with boot-time page size; Boot-time page size can't be used with CPP because it's value is not known at compile time. For compile-time page size builds, the compiler will dead code strip for the same result. Signed-off-by: Ryan Roberts --- ***NOTE*** Any confused maintainers may want to read the cover note here for context: https://lore.kernel.org/all/20241014105514.3206191-1-ryan.roberts@arm.com/ drivers/edac/edac_mc.h | 13 ++++++------- 1 file changed, 6 insertions(+), 7 deletions(-) diff --git a/drivers/edac/edac_mc.h b/drivers/edac/edac_mc.h index 881b00eadf7a5..22132ee86e953 100644 --- a/drivers/edac/edac_mc.h +++ b/drivers/edac/edac_mc.h @@ -37,13 +37,12 @@ #include #include -#if PAGE_SHIFT < 20 -#define PAGES_TO_MiB(pages) ((pages) >> (20 - PAGE_SHIFT)) -#define MiB_TO_PAGES(mb) ((mb) << (20 - PAGE_SHIFT)) -#else /* PAGE_SHIFT > 20 */ -#define PAGES_TO_MiB(pages) ((pages) << (PAGE_SHIFT - 20)) -#define MiB_TO_PAGES(mb) ((mb) >> (PAGE_SHIFT - 20)) -#endif +#define PAGES_TO_MiB(pages) (PAGE_SHIFT < 20 ? \ + ((pages) >> (20 - PAGE_SHIFT)) :\ + ((pages) << (PAGE_SHIFT - 20))) +#define MiB_TO_PAGES(mb) (PAGE_SHIFT < 20 ? \ + ((mb) << (20 - PAGE_SHIFT)) : \ + ((mb) >> (PAGE_SHIFT - 20))) #define edac_printk(level, prefix, fmt, arg...) \ printk(level "EDAC " prefix ": " fmt, ##arg) From patchwork Mon Oct 14 10:58:39 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13834863 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 8AE03D1624E for ; Mon, 14 Oct 2024 12:09:02 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=NJ0XCsdELuRevj5tw1yfqEQTo4IwlrxkOoKIOsf10K4=; b=JD1EwoELZI61JKgVK+V6Kx338z 5D76cfrMKQc4xTEIfPFq/oVbtwCzhk2YqUCO3ZR84zehIsEcI3XsmX8MP8UiS925ZtTAkZzhaMBCo 2E0zB97OsaEJ4i+Y1QuvyAjoMxAIACKcFsDQ5e2uaq+AxgRhBymdRA/9IEMw9R6jO+YBzwcb1AN8j so8rSiZ/fk0Ajw4k9lboZ7vG4q6o56J4ZFARsUaj2Bg34IYwFZpsiss/wY6ydch6t49LfqJPvBA1X uDpfFqCayGNkZDOBDUOUeK/57DX+0if4bG6fx1McUhs3XE65jUl65TXiS42l6Mb2E5EO308FOvzZq 3tfovMLQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1t0JsX-000000051ua-39sI; Mon, 14 Oct 2024 12:08:49 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1t0Ip2-00000004oAy-2RUE for linux-arm-kernel@lists.infradead.org; Mon, 14 Oct 2024 11:01:10 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id C7AA116F8; Mon, 14 Oct 2024 04:01:37 -0700 (PDT) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.27]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 8A33C3F51B; Mon, 14 Oct 2024 04:01:05 -0700 (PDT) From: Ryan Roberts To: Andrew Morton , Anshuman Khandual , Ard Biesheuvel , Catalin Marinas , David Hildenbrand , Greg Marsden , Ivan Ivanov , Jens Wiklander , Kalesh Singh , Marc Zyngier , Mark Rutland , Matthias Brugger , Miroslav Benes , Will Deacon Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, op-tee@lists.trustedfirmware.org Subject: [RFC PATCH v1 32/57] optee: Remove PAGE_SIZE compile-time constant assumption Date: Mon, 14 Oct 2024 11:58:39 +0100 Message-ID: <20241014105912.3207374-32-ryan.roberts@arm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20241014105912.3207374-1-ryan.roberts@arm.com> References: <20241014105514.3206191-1-ryan.roberts@arm.com> <20241014105912.3207374-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241014_040108_756472_74D28045 X-CRM114-Status: GOOD ( 12.97 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org To prepare for supporting boot-time page size selection, refactor code to remove assumptions about PAGE_SIZE being compile-time constant. Code intended to be equivalent when compile-time page size is active. Updated BUILD_BUG_ON() to test against limit. Refactored "struct optee_shm_arg_entry" to use a flexible array member for "map", since its length depends on PAGE_SIZE. Signed-off-by: Ryan Roberts --- ***NOTE*** Any confused maintainers may want to read the cover note here for context: https://lore.kernel.org/all/20241014105514.3206191-1-ryan.roberts@arm.com/ drivers/tee/optee/call.c | 7 +++++-- drivers/tee/optee/smc_abi.c | 2 +- 2 files changed, 6 insertions(+), 3 deletions(-) diff --git a/drivers/tee/optee/call.c b/drivers/tee/optee/call.c index 16eb953e14bb6..41bd7ace6606e 100644 --- a/drivers/tee/optee/call.c +++ b/drivers/tee/optee/call.c @@ -36,7 +36,7 @@ struct optee_shm_arg_entry { struct list_head list_node; struct tee_shm *shm; - DECLARE_BITMAP(map, MAX_ARG_COUNT_PER_ENTRY); + unsigned long map[]; }; void optee_cq_init(struct optee_call_queue *cq, int thread_count) @@ -271,6 +271,7 @@ struct optee_msg_arg *optee_get_msg_arg(struct tee_context *ctx, struct optee_shm_arg_entry *entry; struct optee_msg_arg *ma; size_t args_per_entry; + size_t entry_sz; u_long bit; u_int offs; void *res; @@ -293,7 +294,9 @@ struct optee_msg_arg *optee_get_msg_arg(struct tee_context *ctx, /* * No entry was found, let's allocate a new. */ - entry = kzalloc(sizeof(*entry), GFP_KERNEL); + entry_sz = struct_size(entry, map, + BITS_TO_LONGS(MAX_ARG_COUNT_PER_ENTRY)); + entry = kzalloc(entry_sz, GFP_KERNEL); if (!entry) { res = ERR_PTR(-ENOMEM); goto out; diff --git a/drivers/tee/optee/smc_abi.c b/drivers/tee/optee/smc_abi.c index 844285d4f03c1..005689380d848 100644 --- a/drivers/tee/optee/smc_abi.c +++ b/drivers/tee/optee/smc_abi.c @@ -418,7 +418,7 @@ static void optee_fill_pages_list(u64 *dst, struct page **pages, int num_pages, * code heavily relies on this assumption, so it is better be * safe than sorry. */ - BUILD_BUG_ON(PAGE_SIZE < OPTEE_MSG_NONCONTIG_PAGE_SIZE); + BUILD_BUG_ON(PAGE_SIZE_MIN < OPTEE_MSG_NONCONTIG_PAGE_SIZE); pages_data = (void *)dst; /* From patchwork Mon Oct 14 10:58:40 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13834864 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id E836BD1624F for ; Mon, 14 Oct 2024 12:10:28 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=X8VEpkJ0okgdpGRrvWSbQdQmtub8XCGEswPb3bOfN9k=; b=Tx+Db6NmLop9qI/M4DIUsebUZ/ 8qgvGM72y0DH/BzBw7z2CkVelIOxu0Lzztjtp29JV+WjzXZGu5P4Kx4wJGjB4qg5uOO/gDu5RSno7 xkuwHU7sovXDR8iowT0Dsj+6vwo/+y+kIz8n3X9yIjpPJZ8eLPN5VuzSkxNLpgriNoqI040McL+YN 2Tt/Gyt+LmlpEmmeakV73XfXww5c/O52+CzNGtITAl0CsXHWXySJTfi+9/eIM3SbmZ/AbFW7IQluL Frv0I7Sz+kvNhSa0wB0skrZy+WFvMunNEppJrfZz/fL7LBJ80NKo/ebu37ExTMne1Xy/E1JBzJWTx 8yZX1Xpw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1t0Jtx-000000052Bk-0Lqz; Mon, 14 Oct 2024 12:10:17 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1t0Ip5-00000004oCo-25py for linux-arm-kernel@lists.infradead.org; Mon, 14 Oct 2024 11:01:12 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id B62B016F8; Mon, 14 Oct 2024 04:01:40 -0700 (PDT) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.27]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 78EA33F51B; Mon, 14 Oct 2024 04:01:08 -0700 (PDT) From: Ryan Roberts To: "Jason A. Donenfeld" , "Theodore Ts'o" , Andrew Morton , Anshuman Khandual , Ard Biesheuvel , Catalin Marinas , David Hildenbrand , Greg Marsden , Ivan Ivanov , Kalesh Singh , Marc Zyngier , Mark Rutland , Matthias Brugger , Miroslav Benes , Will Deacon Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [RFC PATCH v1 33/57] random: Remove PAGE_SIZE compile-time constant assumption Date: Mon, 14 Oct 2024 11:58:40 +0100 Message-ID: <20241014105912.3207374-33-ryan.roberts@arm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20241014105912.3207374-1-ryan.roberts@arm.com> References: <20241014105514.3206191-1-ryan.roberts@arm.com> <20241014105912.3207374-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241014_040111_602127_0343693B X-CRM114-Status: GOOD ( 10.88 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org To prepare for supporting boot-time page size selection, refactor code to remove assumptions about PAGE_SIZE being compile-time constant. Code intended to be equivalent when compile-time page size is active. Update BUILD_BUG_ON()s to test against page size limits. Signed-off-by: Ryan Roberts --- ***NOTE*** Any confused maintainers may want to read the cover note here for context: https://lore.kernel.org/all/20241014105514.3206191-1-ryan.roberts@arm.com/ drivers/char/random.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/drivers/char/random.c b/drivers/char/random.c index 87fe61295ea1f..49d6c4ef16df4 100644 --- a/drivers/char/random.c +++ b/drivers/char/random.c @@ -466,7 +466,7 @@ static ssize_t get_random_bytes_user(struct iov_iter *iter) if (!iov_iter_count(iter) || copied != sizeof(block)) break; - BUILD_BUG_ON(PAGE_SIZE % sizeof(block) != 0); + BUILD_BUG_ON(PAGE_SIZE_MIN % sizeof(block) != 0); if (ret % PAGE_SIZE == 0) { if (signal_pending(current)) break; @@ -1428,7 +1428,7 @@ static ssize_t write_pool_user(struct iov_iter *iter) if (!iov_iter_count(iter) || copied != sizeof(block)) break; - BUILD_BUG_ON(PAGE_SIZE % sizeof(block) != 0); + BUILD_BUG_ON(PAGE_SIZE_MIN % sizeof(block) != 0); if (ret % PAGE_SIZE == 0) { if (signal_pending(current)) break; From patchwork Mon Oct 14 10:58:41 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13834865 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 3BB85D1624F for ; Mon, 14 Oct 2024 12:11:56 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=1NGA+tE4zfya25VBkRDEA5yZy6epmiLwve34sECMDkQ=; b=CCgwoveMm2LNGp5+5j2Bs5WJsa pFSRuDN0chSfqRoY1sg2BFHq75mzStdbtMqhJDa/zsRHAeOVoY12DHGwVU/Z8bfFggbA2hs4737di weYYhb7CZo6jZM9zqv9BhrYfwPhvUATk2NmltE7aq5Nmrq+KL8lxtbOBt0Gb+NIMGkSf/83h6P8aG LdZAwQ3AwEVq24s6g5TXeQXOM/huJdJruGjrY3oPpinUc6mpnBKD8snNI0RSKkQTmruK/BZWaZdSd UKfkZIo1SqmHyXwl6S/2V9VNhHTTpcp7b6gPuVJLbev3eZlFpI4IKonKA02OpxU8DT4IBjxINAc66 llFv+91A==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1t0JvK-000000052R7-0p9J; Mon, 14 Oct 2024 12:11:42 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1t0Ip8-00000004oE1-2DJT for linux-arm-kernel@lists.infradead.org; Mon, 14 Oct 2024 11:01:16 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id C49D916F8; Mon, 14 Oct 2024 04:01:43 -0700 (PDT) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.27]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 679B23F51B; Mon, 14 Oct 2024 04:01:11 -0700 (PDT) From: Ryan Roberts To: Andrew Morton , Anshuman Khandual , Ard Biesheuvel , Catalin Marinas , Damien Le Moal , David Hildenbrand , Greg Marsden , Ivan Ivanov , Kalesh Singh , Marc Zyngier , Mark Rutland , Matthias Brugger , Miroslav Benes , Niklas Cassel , Will Deacon Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, linux-ide@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [RFC PATCH v1 34/57] sata_sil24: Remove PAGE_SIZE compile-time constant assumption Date: Mon, 14 Oct 2024 11:58:41 +0100 Message-ID: <20241014105912.3207374-34-ryan.roberts@arm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20241014105912.3207374-1-ryan.roberts@arm.com> References: <20241014105514.3206191-1-ryan.roberts@arm.com> <20241014105912.3207374-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241014_040114_705770_AE3D4B9B X-CRM114-Status: GOOD ( 17.88 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org To prepare for supporting boot-time page size selection, refactor code to remove assumptions about PAGE_SIZE being compile-time constant. Code intended to be equivalent when compile-time page size is active. Convert "struct sil24_ata_block" and "struct sil24_atapi_block" to use a flexible array member for their sge[] array. The previous static size of SIL24_MAX_SGE depends on PAGE_SIZE so doesn't work for boot-time page size. Wrap global variables that are initialized with PAGE_SIZE derived values using DEFINE_GLOBAL_PAGE_SIZE_VAR() so their initialization can be deferred for boot-time page size builds. Signed-off-by: Ryan Roberts --- ***NOTE*** Any confused maintainers may want to read the cover note here for context: https://lore.kernel.org/all/20241014105514.3206191-1-ryan.roberts@arm.com/ drivers/ata/sata_sil24.c | 46 +++++++++++++++++++--------------------- 1 file changed, 22 insertions(+), 24 deletions(-) diff --git a/drivers/ata/sata_sil24.c b/drivers/ata/sata_sil24.c index 72c03cbdaff43..85c6382976626 100644 --- a/drivers/ata/sata_sil24.c +++ b/drivers/ata/sata_sil24.c @@ -42,26 +42,25 @@ struct sil24_sge { __le32 flags; }; +/* + * sil24 fetches in chunks of 64bytes. The first block + * contains the PRB and two SGEs. From the second block, it's + * consisted of four SGEs and called SGT. Calculate the + * number of SGTs that fit into one page. + */ +#define SIL24_PRB_SZ (sizeof(struct sil24_prb) + 2 * sizeof(struct sil24_sge)) +#define SIL24_MAX_SGT ((PAGE_SIZE - SIL24_PRB_SZ) / (4 * sizeof(struct sil24_sge))) + +/* + * This will give us one unused SGEs for ATA. This extra SGE + * will be used to store CDB for ATAPI devices. + */ +#define SIL24_MAX_SGE (4 * SIL24_MAX_SGT + 1) enum { SIL24_HOST_BAR = 0, SIL24_PORT_BAR = 2, - /* sil24 fetches in chunks of 64bytes. The first block - * contains the PRB and two SGEs. From the second block, it's - * consisted of four SGEs and called SGT. Calculate the - * number of SGTs that fit into one page. - */ - SIL24_PRB_SZ = sizeof(struct sil24_prb) - + 2 * sizeof(struct sil24_sge), - SIL24_MAX_SGT = (PAGE_SIZE - SIL24_PRB_SZ) - / (4 * sizeof(struct sil24_sge)), - - /* This will give us one unused SGEs for ATA. This extra SGE - * will be used to store CDB for ATAPI devices. - */ - SIL24_MAX_SGE = 4 * SIL24_MAX_SGT + 1, - /* * Global controller registers (128 bytes @ BAR0) */ @@ -244,13 +243,13 @@ enum { struct sil24_ata_block { struct sil24_prb prb; - struct sil24_sge sge[SIL24_MAX_SGE]; + struct sil24_sge sge[]; }; struct sil24_atapi_block { struct sil24_prb prb; u8 cdb[16]; - struct sil24_sge sge[SIL24_MAX_SGE]; + struct sil24_sge sge[]; }; union sil24_cmd_block { @@ -373,7 +372,7 @@ static struct pci_driver sil24_pci_driver = { #endif }; -static const struct scsi_host_template sil24_sht = { +static DEFINE_GLOBAL_PAGE_SIZE_VAR_CONST(struct scsi_host_template, sil24_sht, { __ATA_BASE_SHT(DRV_NAME), .can_queue = SIL24_MAX_CMDS, .sg_tablesize = SIL24_MAX_SGE, @@ -382,7 +381,7 @@ static const struct scsi_host_template sil24_sht = { .sdev_groups = ata_ncq_sdev_groups, .change_queue_depth = ata_scsi_change_queue_depth, .device_configure = ata_scsi_device_configure -}; +}); static struct ata_port_operations sil24_ops = { .inherits = &sata_pmp_port_ops, @@ -1193,7 +1192,7 @@ static int sil24_port_start(struct ata_port *ap) struct device *dev = ap->host->dev; struct sil24_port_priv *pp; union sil24_cmd_block *cb; - size_t cb_size = sizeof(*cb) * SIL24_MAX_CMDS; + size_t cb_size = PAGE_SIZE * SIL24_MAX_CMDS; dma_addr_t cb_dma; pp = devm_kzalloc(dev, sizeof(*pp), GFP_KERNEL); @@ -1258,7 +1257,6 @@ static void sil24_init_controller(struct ata_host *host) static int sil24_init_one(struct pci_dev *pdev, const struct pci_device_id *ent) { - extern int __MARKER__sil24_cmd_block_is_sized_wrongly; struct ata_port_info pi = sil24_port_info[ent->driver_data]; const struct ata_port_info *ppi[] = { &pi, NULL }; void __iomem * const *iomap; @@ -1266,9 +1264,9 @@ static int sil24_init_one(struct pci_dev *pdev, const struct pci_device_id *ent) int rc; u32 tmp; - /* cause link error if sil24_cmd_block is sized wrongly */ - if (sizeof(union sil24_cmd_block) != PAGE_SIZE) - __MARKER__sil24_cmd_block_is_sized_wrongly = 1; + /* union sil24_cmd_block must be PAGE_SIZE */ + BUG_ON(struct_size_t(struct sil24_atapi_block, sge, SIL24_MAX_SGE) != PAGE_SIZE); + BUG_ON(struct_size_t(struct sil24_ata_block, sge, SIL24_MAX_SGE) > PAGE_SIZE); ata_print_version_once(&pdev->dev, DRV_VERSION); From patchwork Mon Oct 14 10:58:42 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13834870 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 3C97BD1624E for ; Mon, 14 Oct 2024 12:13:16 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=5o+gDL2mN3xapmdN2m55fCy14ysHc8+6na6uUa6irXM=; b=gqxRvuKo5MyKQZBGnEWsfu4Qpx /KjU9U/LfPOrLWOVnQJ6P7+KVT4SP8TBtILpSBCEXU0HKnYvyqQ20OIKhh8LUmW2ykPRBm2zfdIVC n/2LiOuIZIET8BpjXnXfkPopKhlgrKRnlSn1bwT/BARfFn8s31VP2WFkPIXx0Za+1cYLQnhNHl41u blL6cemZfTIzgebuQShcOLXRZ+ZSPdw7jH98IFxsan7TaerJIf1r0qA0yZlFLD9W6QIM5bGm10frH BgILTSSTEeE20k9Q+gLE/4rTLFQuOZyb3TVv3WqTQIh544UbCzLjx2ODF91w+PJlchXx41YJoArs2 GRLSDq5Q==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1t0Jwg-000000052gc-3rvP; Mon, 14 Oct 2024 12:13:06 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1t0IpC-00000004oGb-1tVf for linux-arm-kernel@lists.infradead.org; Mon, 14 Oct 2024 11:01:20 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 997AD16F8; Mon, 14 Oct 2024 04:01:47 -0700 (PDT) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.27]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 769A53F51B; Mon, 14 Oct 2024 04:01:14 -0700 (PDT) From: Ryan Roberts To: "Michael S. Tsirkin" , Andrew Morton , Anshuman Khandual , Ard Biesheuvel , Catalin Marinas , David Hildenbrand , Dominique Martinet , Eric Van Hensbergen , Greg Marsden , Ivan Ivanov , Jason Wang , Jens Axboe , Kalesh Singh , Latchesar Ionkov , Marc Zyngier , Mark Rutland , Matthias Brugger , Miroslav Benes , Will Deacon Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, v9fs@lists.linux.dev, virtualization@lists.linux.dev Subject: [RFC PATCH v1 35/57] virtio: Remove PAGE_SIZE compile-time constant assumption Date: Mon, 14 Oct 2024 11:58:42 +0100 Message-ID: <20241014105912.3207374-35-ryan.roberts@arm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20241014105912.3207374-1-ryan.roberts@arm.com> References: <20241014105514.3206191-1-ryan.roberts@arm.com> <20241014105912.3207374-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241014_040118_602762_E2FAE656 X-CRM114-Status: GOOD ( 15.84 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org To prepare for supporting boot-time page size selection, refactor code to remove assumptions about PAGE_SIZE being compile-time constant. Code intended to be equivalent when compile-time page size is active. Updated multiple BUILD_BUG_ON() instances to test against page size limits. Wrap global variables that are initialized with PAGE_SIZE derived values using DEFINE_GLOBAL_PAGE_SIZE_VAR() so their initialization can be deferred for boot-time page size builds. Signed-off-by: Ryan Roberts --- ***NOTE*** Any confused maintainers may want to read the cover note here for context: https://lore.kernel.org/all/20241014105514.3206191-1-ryan.roberts@arm.com/ drivers/block/virtio_blk.c | 2 +- drivers/virtio/virtio_balloon.c | 10 ++++++---- net/9p/trans_virtio.c | 4 ++-- 3 files changed, 9 insertions(+), 7 deletions(-) diff --git a/drivers/block/virtio_blk.c b/drivers/block/virtio_blk.c index 194417abc1053..8a8960b609bc9 100644 --- a/drivers/block/virtio_blk.c +++ b/drivers/block/virtio_blk.c @@ -899,7 +899,7 @@ static ssize_t serial_show(struct device *dev, int err; /* sysfs gives us a PAGE_SIZE buffer */ - BUILD_BUG_ON(PAGE_SIZE < VIRTIO_BLK_ID_BYTES); + BUILD_BUG_ON(PAGE_SIZE_MIN < VIRTIO_BLK_ID_BYTES); buf[VIRTIO_BLK_ID_BYTES] = '\0'; err = virtblk_get_id(disk, buf); diff --git a/drivers/virtio/virtio_balloon.c b/drivers/virtio/virtio_balloon.c index 54469277ca303..3818d894bd212 100644 --- a/drivers/virtio/virtio_balloon.c +++ b/drivers/virtio/virtio_balloon.c @@ -25,6 +25,7 @@ * page units. */ #define VIRTIO_BALLOON_PAGES_PER_PAGE (unsigned int)(PAGE_SIZE >> VIRTIO_BALLOON_PFN_SHIFT) +#define VIRTIO_BALLOON_PAGES_PER_PAGE_MAX (unsigned int)(PAGE_SIZE_MAX >> VIRTIO_BALLOON_PFN_SHIFT) #define VIRTIO_BALLOON_ARRAY_PFNS_MAX 256 /* Maximum number of (4k) pages to deflate on OOM notifications. */ #define VIRTIO_BALLOON_OOM_NR_PAGES 256 @@ -138,7 +139,7 @@ static u32 page_to_balloon_pfn(struct page *page) { unsigned long pfn = page_to_pfn(page); - BUILD_BUG_ON(PAGE_SHIFT < VIRTIO_BALLOON_PFN_SHIFT); + BUILD_BUG_ON(PAGE_SHIFT_MIN < VIRTIO_BALLOON_PFN_SHIFT); /* Convert pfn from Linux page size to balloon page size. */ return pfn * VIRTIO_BALLOON_PAGES_PER_PAGE; } @@ -228,7 +229,7 @@ static void set_page_pfns(struct virtio_balloon *vb, { unsigned int i; - BUILD_BUG_ON(VIRTIO_BALLOON_PAGES_PER_PAGE > VIRTIO_BALLOON_ARRAY_PFNS_MAX); + BUILD_BUG_ON(VIRTIO_BALLOON_PAGES_PER_PAGE_MAX > VIRTIO_BALLOON_ARRAY_PFNS_MAX); /* * Set balloon pfns pointing at this page. @@ -1042,8 +1043,9 @@ static int virtballoon_probe(struct virtio_device *vdev) * host's base page size. However, it needs more work to report * that value. The hard-coded order would be fine currently. */ -#if defined(CONFIG_ARM64) && defined(CONFIG_ARM64_64K_PAGES) - vb->pr_dev_info.order = 5; +#if defined(CONFIG_ARM64) + if (PAGE_SIZE == SZ_64K) + vb->pr_dev_info.order = 5; #endif err = page_reporting_register(&vb->pr_dev_info); diff --git a/net/9p/trans_virtio.c b/net/9p/trans_virtio.c index 0b8086f58ad55..25b8253011cec 100644 --- a/net/9p/trans_virtio.c +++ b/net/9p/trans_virtio.c @@ -786,7 +786,7 @@ static struct virtio_driver p9_virtio_drv = { .remove = p9_virtio_remove, }; -static struct p9_trans_module p9_virtio_trans = { +static DEFINE_GLOBAL_PAGE_SIZE_VAR(struct p9_trans_module, p9_virtio_trans, { .name = "virtio", .create = p9_virtio_create, .close = p9_virtio_close, @@ -804,7 +804,7 @@ static struct p9_trans_module p9_virtio_trans = { .pooled_rbuffers = false, .def = 1, .owner = THIS_MODULE, -}; +}); /* The standard init function */ static int __init p9_virtio_init(void) From patchwork Mon Oct 14 10:58:43 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13834871 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id D2ABFD1624E for ; Mon, 14 Oct 2024 12:14:42 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=SCs5zLvTJKb41Hb5mZyIydGLU4/ZfA/FgaJwPe9tmC8=; b=tmq0kxGrNPrtH1kNEZRPOcu2kH dqZ48DWi5zzhlHx93Rm1N33qgFc0Gexrvdsp+qBdDHMchd2xuvrw7HwD8NSzngeUIkrEUfeMm9zj2 YKSJ6NtZWdJfPhNMHRwOa6GcDVBOZ0KvLwL5mI56Ah02BZwUnSRsXZyo3SFwChWdSAQWPiUWFvzj8 lTFzYCZiR+stq2IceWleUZTFrb4t+yD6YzEhdLpRsuJWwy+QbZbio9HW81u23v6/qjpW9TU3BDX/p 6JO0mcvKNM1sQxOr+tcQHFQr/axOud6A1cBDgSmruaBYu/kQWeXuLZ7+YTIzrc8yfvKl1GzaBly4K N6DwpyWw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1t0Jy2-000000052rv-0xTy; Mon, 14 Oct 2024 12:14:30 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1t0IpF-00000004oIj-0epK for linux-arm-kernel@lists.infradead.org; Mon, 14 Oct 2024 11:01:23 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 6816B1713; Mon, 14 Oct 2024 04:01:50 -0700 (PDT) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.27]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 4980A3F51B; Mon, 14 Oct 2024 04:01:18 -0700 (PDT) From: Ryan Roberts To: Andrew Morton , Anshuman Khandual , Ard Biesheuvel , Catalin Marinas , David Hildenbrand , Greg Marsden , Ivan Ivanov , Kalesh Singh , Marc Zyngier , Mark Rutland , Matthias Brugger , Miroslav Benes , Will Deacon Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, xen-devel@lists.xenproject.org Subject: [RFC PATCH v1 36/57] xen: Remove PAGE_SIZE compile-time constant assumption Date: Mon, 14 Oct 2024 11:58:43 +0100 Message-ID: <20241014105912.3207374-36-ryan.roberts@arm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20241014105912.3207374-1-ryan.roberts@arm.com> References: <20241014105514.3206191-1-ryan.roberts@arm.com> <20241014105912.3207374-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241014_040121_359297_83CAFFAA X-CRM114-Status: GOOD ( 18.67 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org To prepare for supporting boot-time page size selection, refactor code to remove assumptions about PAGE_SIZE being compile-time constant. Code intended to be equivalent when compile-time page size is active. Allocate enough "frame_list" static storage in the balloon driver for the maximum supported page size. Although continue to use only the first PAGE_SIZE of the buffer at run-time to maintain existing behaviour. Refactor xen_biovec_phys_mergeable() to convert ifdeffery to c if/else. For compile-time page size, the compiler will choose one branch and strip the dead one. For boot-time, it can be evaluated at run time. Refactor a BUILD_BUG_ON to evaluate the limit (when the minimum supported page size is selected at boot-time). Reserve enough storage for max page size in "struct remap_data" and "struct xenbus_map_node". Signed-off-by: Ryan Roberts Reviewed-by: Stefano Stabellini --- ***NOTE*** Any confused maintainers may want to read the cover note here for context: https://lore.kernel.org/all/20241014105514.3206191-1-ryan.roberts@arm.com/ drivers/xen/balloon.c | 11 ++++++----- drivers/xen/biomerge.c | 12 ++++++------ drivers/xen/privcmd.c | 2 +- drivers/xen/xenbus/xenbus_client.c | 5 +++-- drivers/xen/xlate_mmu.c | 6 +++--- include/xen/page.h | 2 ++ 6 files changed, 21 insertions(+), 17 deletions(-) diff --git a/drivers/xen/balloon.c b/drivers/xen/balloon.c index 528395133b4f8..0ed5f6453af0e 100644 --- a/drivers/xen/balloon.c +++ b/drivers/xen/balloon.c @@ -131,7 +131,8 @@ struct balloon_stats balloon_stats; EXPORT_SYMBOL_GPL(balloon_stats); /* We increase/decrease in batches which fit in a page */ -static xen_pfn_t frame_list[PAGE_SIZE / sizeof(xen_pfn_t)]; +static xen_pfn_t frame_list[PAGE_SIZE_MAX / sizeof(xen_pfn_t)]; +#define FRAME_LIST_NR_ENTRIES (PAGE_SIZE / sizeof(xen_pfn_t)) /* List of ballooned pages, threaded through the mem_map array. */ @@ -389,8 +390,8 @@ static enum bp_state increase_reservation(unsigned long nr_pages) unsigned long i; struct page *page; - if (nr_pages > ARRAY_SIZE(frame_list)) - nr_pages = ARRAY_SIZE(frame_list); + if (nr_pages > FRAME_LIST_NR_ENTRIES) + nr_pages = FRAME_LIST_NR_ENTRIES; page = list_first_entry_or_null(&ballooned_pages, struct page, lru); for (i = 0; i < nr_pages; i++) { @@ -434,8 +435,8 @@ static enum bp_state decrease_reservation(unsigned long nr_pages, gfp_t gfp) int ret; LIST_HEAD(pages); - if (nr_pages > ARRAY_SIZE(frame_list)) - nr_pages = ARRAY_SIZE(frame_list); + if (nr_pages > FRAME_LIST_NR_ENTRIES) + nr_pages = FRAME_LIST_NR_ENTRIES; for (i = 0; i < nr_pages; i++) { page = alloc_page(gfp); diff --git a/drivers/xen/biomerge.c b/drivers/xen/biomerge.c index 05a286d24f148..28f0887e40026 100644 --- a/drivers/xen/biomerge.c +++ b/drivers/xen/biomerge.c @@ -8,16 +8,16 @@ bool xen_biovec_phys_mergeable(const struct bio_vec *vec1, const struct page *page) { -#if XEN_PAGE_SIZE == PAGE_SIZE - unsigned long bfn1 = pfn_to_bfn(page_to_pfn(vec1->bv_page)); - unsigned long bfn2 = pfn_to_bfn(page_to_pfn(page)); + if (XEN_PAGE_SIZE == PAGE_SIZE) { + unsigned long bfn1 = pfn_to_bfn(page_to_pfn(vec1->bv_page)); + unsigned long bfn2 = pfn_to_bfn(page_to_pfn(page)); + + return bfn1 + PFN_DOWN(vec1->bv_offset + vec1->bv_len) == bfn2; + } - return bfn1 + PFN_DOWN(vec1->bv_offset + vec1->bv_len) == bfn2; -#else /* * XXX: Add support for merging bio_vec when using different page * size in Xen and Linux. */ return false; -#endif } diff --git a/drivers/xen/privcmd.c b/drivers/xen/privcmd.c index 9563650dfbafc..847f7b806caf7 100644 --- a/drivers/xen/privcmd.c +++ b/drivers/xen/privcmd.c @@ -557,7 +557,7 @@ static long privcmd_ioctl_mmap_batch( state.global_error = 0; state.version = version; - BUILD_BUG_ON(((PAGE_SIZE / sizeof(xen_pfn_t)) % XEN_PFN_PER_PAGE) != 0); + BUILD_BUG_ON(((PAGE_SIZE_MIN / sizeof(xen_pfn_t)) % XEN_PFN_PER_PAGE_MAX) != 0); /* mmap_batch_fn guarantees ret == 0 */ BUG_ON(traverse_pages_block(m.num, sizeof(xen_pfn_t), &pagelist, mmap_batch_fn, &state)); diff --git a/drivers/xen/xenbus/xenbus_client.c b/drivers/xen/xenbus/xenbus_client.c index 51b3124b0d56c..99bde836c10c4 100644 --- a/drivers/xen/xenbus/xenbus_client.c +++ b/drivers/xen/xenbus/xenbus_client.c @@ -49,9 +49,10 @@ #include "xenbus.h" -#define XENBUS_PAGES(_grants) (DIV_ROUND_UP(_grants, XEN_PFN_PER_PAGE)) +#define XENBUS_PAGES(_grants) (DIV_ROUND_UP(_grants, XEN_PFN_PER_PAGE)) +#define XENBUS_PAGES_MAX(_grants) (DIV_ROUND_UP(_grants, XEN_PFN_PER_PAGE_MIN)) -#define XENBUS_MAX_RING_PAGES (XENBUS_PAGES(XENBUS_MAX_RING_GRANTS)) +#define XENBUS_MAX_RING_PAGES (XENBUS_PAGES_MAX(XENBUS_MAX_RING_GRANTS)) struct xenbus_map_node { struct list_head next; diff --git a/drivers/xen/xlate_mmu.c b/drivers/xen/xlate_mmu.c index f17c4c03db30c..a757c801a7542 100644 --- a/drivers/xen/xlate_mmu.c +++ b/drivers/xen/xlate_mmu.c @@ -74,9 +74,9 @@ struct remap_data { int mapped; /* Hypercall parameters */ - int h_errs[XEN_PFN_PER_PAGE]; - xen_ulong_t h_idxs[XEN_PFN_PER_PAGE]; - xen_pfn_t h_gpfns[XEN_PFN_PER_PAGE]; + int h_errs[XEN_PFN_PER_PAGE_MAX]; + xen_ulong_t h_idxs[XEN_PFN_PER_PAGE_MAX]; + xen_pfn_t h_gpfns[XEN_PFN_PER_PAGE_MAX]; int h_iter; /* Iterator */ }; diff --git a/include/xen/page.h b/include/xen/page.h index 285677b42943a..86683a30038a3 100644 --- a/include/xen/page.h +++ b/include/xen/page.h @@ -21,6 +21,8 @@ ((page_to_pfn(page)) << (PAGE_SHIFT - XEN_PAGE_SHIFT)) #define XEN_PFN_PER_PAGE (PAGE_SIZE / XEN_PAGE_SIZE) +#define XEN_PFN_PER_PAGE_MIN (PAGE_SIZE_MIN / XEN_PAGE_SIZE) +#define XEN_PFN_PER_PAGE_MAX (PAGE_SIZE_MAX / XEN_PAGE_SIZE) #define XEN_PFN_DOWN(x) ((x) >> XEN_PAGE_SHIFT) #define XEN_PFN_UP(x) (((x) + XEN_PAGE_SIZE-1) >> XEN_PAGE_SHIFT) From patchwork Mon Oct 14 10:58:44 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13834872 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 6FCD2D16250 for ; Mon, 14 Oct 2024 12:16:10 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=sUJHwqHFRx+14j6h+df8VkAiqppEg2Zp0LAwcqaTc7A=; b=G9L4IEDLWYl1cfxz2kyOxepXWI JuzrRl+ypfoHmzcTl5ygZ+PIo2p+Wr2jeiIe/I5PwoCDfg/hxxO90r5KtA7zwkNTaQTtZ5VQ6S7Il Fqt20DXloDiWZPb27MUQEApkMycwujo/6UaI8Kj8Vz/zdi1o3bbosbMZN4+JV9ItrT9VXw7vG0Uul boYgcSluXrAXwdePI2cXa+zwgJJRub5FxfukQmzTMAH3ltAl5yDb3m6m6QZ4N2k1RqiE9IKHtTZ4s JN5hHY+Eh32y2thCfz7uZA7sYpAzxLqFP6z0ch0N6jshPUHteaqAzorYBtzIrX4sI4o+iULwffAto QZC1iXrw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1t0JzT-000000053C8-2J8w; Mon, 14 Oct 2024 12:15:59 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1t0IpH-00000004oK1-3M46 for linux-arm-kernel@lists.infradead.org; Mon, 14 Oct 2024 11:01:25 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 1861F1756; Mon, 14 Oct 2024 04:01:53 -0700 (PDT) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.27]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 18CCC3F51B; Mon, 14 Oct 2024 04:01:20 -0700 (PDT) From: Ryan Roberts To: Andrew Morton , Anshuman Khandual , Ard Biesheuvel , Catalin Marinas , David Hildenbrand , Greg Marsden , Ivan Ivanov , Kalesh Singh , Marc Zyngier , Mark Rutland , Matthias Brugger , Miroslav Benes , Will Deacon Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [RFC PATCH v1 37/57] arm64: Fix macros to work in C code in addition to the linker script Date: Mon, 14 Oct 2024 11:58:44 +0100 Message-ID: <20241014105912.3207374-37-ryan.roberts@arm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20241014105912.3207374-1-ryan.roberts@arm.com> References: <20241014105514.3206191-1-ryan.roberts@arm.com> <20241014105912.3207374-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241014_040124_000796_2695B370 X-CRM114-Status: GOOD ( 15.77 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Previously INIT_DIR_SIZE and INIT_IDMAP_DIR_SIZE used _end for the end address of the kernel image. In the linker script context, this resolves to an integer that refers to the link va of the end of the kernel image. But in C code it resolves to a pointer to the end of the image as placed in memory. So there are 2 problems; because its a pointer, we can't do arithmetic on it. And because the image may be in a different location in memory than the one it was linked at, it is not correct to find the image size by subtracting KIMAGE_VADDR. So introduce KIMAGE_VADDR_END, which always represents the link va of the end of the kernel image as an integer. Signed-off-by: Ryan Roberts --- ***NOTE*** Any confused maintainers may want to read the cover note here for context: https://lore.kernel.org/all/20241014105514.3206191-1-ryan.roberts@arm.com/ arch/arm64/include/asm/kernel-pgtable.h | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/arch/arm64/include/asm/kernel-pgtable.h b/arch/arm64/include/asm/kernel-pgtable.h index bf05a77873a49..1722b9217d47d 100644 --- a/arch/arm64/include/asm/kernel-pgtable.h +++ b/arch/arm64/include/asm/kernel-pgtable.h @@ -35,6 +35,8 @@ #define IDMAP_LEVELS ARM64_HW_PGTABLE_LEVELS(IDMAP_VA_BITS) #define IDMAP_ROOT_LEVEL (4 - IDMAP_LEVELS) +#define KIMAGE_VADDR_END (_AT(u64, _end) - _AT(u64, _text) + KIMAGE_VADDR) + /* * A relocatable kernel may execute from an address that differs from the one at * which it was linked. In the worst case, its runtime placement may intersect @@ -56,10 +58,10 @@ + EARLY_LEVEL(3, (lvls), (vstart), (vend), add) /* each entry needs a next level page table */ \ + EARLY_LEVEL(2, (lvls), (vstart), (vend), add) /* each entry needs a next level page table */ \ + EARLY_LEVEL(1, (lvls), (vstart), (vend), add))/* each entry needs a next level page table */ -#define INIT_DIR_SIZE (PAGE_SIZE * (EARLY_PAGES(SWAPPER_PGTABLE_LEVELS, KIMAGE_VADDR, _end, EXTRA_PAGE) \ +#define INIT_DIR_SIZE (PAGE_SIZE * (EARLY_PAGES(SWAPPER_PGTABLE_LEVELS, KIMAGE_VADDR, KIMAGE_VADDR_END, EXTRA_PAGE) \ + EARLY_SEGMENT_EXTRA_PAGES)) -#define INIT_IDMAP_DIR_PAGES (EARLY_PAGES(INIT_IDMAP_PGTABLE_LEVELS, KIMAGE_VADDR, _end, 1)) +#define INIT_IDMAP_DIR_PAGES (EARLY_PAGES(INIT_IDMAP_PGTABLE_LEVELS, KIMAGE_VADDR, KIMAGE_VADDR_END, 1)) #define INIT_IDMAP_DIR_SIZE ((INIT_IDMAP_DIR_PAGES + EARLY_IDMAP_EXTRA_PAGES) * PAGE_SIZE) #define INIT_IDMAP_FDT_PAGES (EARLY_PAGES(INIT_IDMAP_PGTABLE_LEVELS, 0UL, UL(MAX_FDT_SIZE), 1) - 1) From patchwork Mon Oct 14 10:58:45 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13834873 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 4F580D16251 for ; Mon, 14 Oct 2024 12:17:40 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=Vs2AuMehXjNKtnTB/Gm3C8UdYqhpeS4EkHa/BZQq/AA=; b=lLZzLo97PrxmpcolEDRmcZiJMb mLSYgpCFOT/FoVRSsDFGL1zLEizwx1qsti8N/fAeUrldu4jBoKrMmUWgqjDl4RslmWLEgCnidNniY xiH6Lk6ZBetMufOrtgb7hPGZaeWRdMaC8CfG3iKP5wC3oKd+eZqwsW0cqFiscP+gBbMYYKMf54kFP 6sFstwZieqa5gSq0PYwNL1MGqo6Xutjal1OqINPhDEBee++dapRcMUmnLKKxTe+3ddf1mJ7zKYdkZ xTFzN+HM7iHl8S2G5RDJSqvfdTC7hkxrYLD9oYCM5mL0+YLQexZiYtUywHxNUKIM1PU1/+BdCbWUo wy5EtYrw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1t0K0t-000000053PV-3o7V; Mon, 14 Oct 2024 12:17:27 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1t0IpK-00000004oLV-1wnW for linux-arm-kernel@lists.infradead.org; Mon, 14 Oct 2024 11:01:28 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id BE0851713; Mon, 14 Oct 2024 04:01:55 -0700 (PDT) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.27]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id BE5983F51B; Mon, 14 Oct 2024 04:01:23 -0700 (PDT) From: Ryan Roberts To: Andrew Morton , Anshuman Khandual , Ard Biesheuvel , Catalin Marinas , David Hildenbrand , Greg Marsden , Ivan Ivanov , Kalesh Singh , Marc Zyngier , Mark Rutland , Matthias Brugger , Miroslav Benes , Will Deacon Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [RFC PATCH v1 38/57] arm64: Track early pgtable allocation limit Date: Mon, 14 Oct 2024 11:58:45 +0100 Message-ID: <20241014105912.3207374-38-ryan.roberts@arm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20241014105912.3207374-1-ryan.roberts@arm.com> References: <20241014105514.3206191-1-ryan.roberts@arm.com> <20241014105912.3207374-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241014_040126_626388_3521892E X-CRM114-Status: GOOD ( 21.32 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Early pgtables (e.g. init_idmap_pg_dir, init_pg_dir, etc) are allocated from statically defined memory blocks within the kernel image that are sized for the calculated worst case requirements. Let's make the allocator aware of the block's limit so that it can detect any overflow. This boils down to passing the limit of the memory block to map_range() so let's add it as a parameter. If an overflow is detected, report the error to __early_cpu_boot_status and park the cpu. Signed-off-by: Ryan Roberts --- ***NOTE*** Any confused maintainers may want to read the cover note here for context: https://lore.kernel.org/all/20241014105514.3206191-1-ryan.roberts@arm.com/ arch/arm64/include/asm/smp.h | 1 + arch/arm64/kernel/head.S | 3 +- arch/arm64/kernel/image-vars.h | 3 ++ arch/arm64/kernel/pi/map_kernel.c | 35 +++++++++++--------- arch/arm64/kernel/pi/map_range.c | 54 +++++++++++++++++++++++++------ arch/arm64/kernel/pi/pi.h | 4 +-- arch/arm64/mm/mmu.c | 14 +++++--- 7 files changed, 81 insertions(+), 33 deletions(-) diff --git a/arch/arm64/include/asm/smp.h b/arch/arm64/include/asm/smp.h index 2510eec026f7e..86edc5f8c9673 100644 --- a/arch/arm64/include/asm/smp.h +++ b/arch/arm64/include/asm/smp.h @@ -22,6 +22,7 @@ #define CPU_STUCK_REASON_52_BIT_VA (UL(1) << CPU_STUCK_REASON_SHIFT) #define CPU_STUCK_REASON_NO_GRAN (UL(2) << CPU_STUCK_REASON_SHIFT) +#define CPU_STUCK_REASON_NO_PGTABLE_MEM (UL(3) << CPU_STUCK_REASON_SHIFT) #ifndef __ASSEMBLY__ diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S index cb68adcabe078..7e17a71fd9e4b 100644 --- a/arch/arm64/kernel/head.S +++ b/arch/arm64/kernel/head.S @@ -89,7 +89,8 @@ SYM_CODE_START(primary_entry) mov sp, x1 mov x29, xzr adrp x0, init_idmap_pg_dir - mov x1, xzr + adrp x1, init_idmap_pg_end + mov x2, xzr bl __pi_create_init_idmap /* diff --git a/arch/arm64/kernel/image-vars.h b/arch/arm64/kernel/image-vars.h index 8f5422ed1b758..a168f3337446f 100644 --- a/arch/arm64/kernel/image-vars.h +++ b/arch/arm64/kernel/image-vars.h @@ -52,6 +52,7 @@ PROVIDE(__pi_cavium_erratum_27456_cpus = cavium_erratum_27456_cpus); #endif PROVIDE(__pi__ctype = _ctype); PROVIDE(__pi_memstart_offset_seed = memstart_offset_seed); +PROVIDE(__pi___early_cpu_boot_status = __early_cpu_boot_status); PROVIDE(__pi_init_idmap_pg_dir = init_idmap_pg_dir); PROVIDE(__pi_init_idmap_pg_end = init_idmap_pg_end); @@ -68,6 +69,8 @@ PROVIDE(__pi___inittext_end = __inittext_end); PROVIDE(__pi___initdata_begin = __initdata_begin); PROVIDE(__pi___initdata_end = __initdata_end); PROVIDE(__pi__data = _data); +PROVIDE(__pi___mmuoff_data_start = __mmuoff_data_start); +PROVIDE(__pi___mmuoff_data_end = __mmuoff_data_end); PROVIDE(__pi___bss_start = __bss_start); PROVIDE(__pi__end = _end); diff --git a/arch/arm64/kernel/pi/map_kernel.c b/arch/arm64/kernel/pi/map_kernel.c index f374a3e5a5fe1..dcf9233ccfff2 100644 --- a/arch/arm64/kernel/pi/map_kernel.c +++ b/arch/arm64/kernel/pi/map_kernel.c @@ -20,11 +20,11 @@ extern const u8 __eh_frame_start[], __eh_frame_end[]; extern void idmap_cpu_replace_ttbr1(void *pgdir); -static void __init map_segment(pgd_t *pg_dir, u64 *pgd, u64 va_offset, - void *start, void *end, pgprot_t prot, - bool may_use_cont, int root_level) +static void __init map_segment(pgd_t *pg_dir, u64 *pgd, u64 limit, + u64 va_offset, void *start, void *end, + pgprot_t prot, bool may_use_cont, int root_level) { - map_range(pgd, ((u64)start + va_offset) & ~PAGE_OFFSET, + map_range(pgd, limit, ((u64)start + va_offset) & ~PAGE_OFFSET, ((u64)end + va_offset) & ~PAGE_OFFSET, (u64)start, prot, root_level, (pte_t *)pg_dir, may_use_cont, 0); } @@ -32,7 +32,7 @@ static void __init map_segment(pgd_t *pg_dir, u64 *pgd, u64 va_offset, static void __init unmap_segment(pgd_t *pg_dir, u64 va_offset, void *start, void *end, int root_level) { - map_segment(pg_dir, NULL, va_offset, start, end, __pgprot(0), + map_segment(pg_dir, NULL, 0, va_offset, start, end, __pgprot(0), false, root_level); } @@ -41,6 +41,7 @@ static void __init map_kernel(u64 kaslr_offset, u64 va_offset, int root_level) bool enable_scs = IS_ENABLED(CONFIG_UNWIND_PATCH_PAC_INTO_SCS); bool twopass = IS_ENABLED(CONFIG_RELOCATABLE); u64 pgdp = (u64)init_pg_dir + PAGE_SIZE; + u64 limit = (u64)init_pg_end; pgprot_t text_prot = PAGE_KERNEL_ROX; pgprot_t data_prot = PAGE_KERNEL; pgprot_t prot; @@ -78,16 +79,16 @@ static void __init map_kernel(u64 kaslr_offset, u64 va_offset, int root_level) twopass |= enable_scs; prot = twopass ? data_prot : text_prot; - map_segment(init_pg_dir, &pgdp, va_offset, _stext, _etext, prot, + map_segment(init_pg_dir, &pgdp, limit, va_offset, _stext, _etext, prot, !twopass, root_level); - map_segment(init_pg_dir, &pgdp, va_offset, __start_rodata, + map_segment(init_pg_dir, &pgdp, limit, va_offset, __start_rodata, __inittext_begin, data_prot, false, root_level); - map_segment(init_pg_dir, &pgdp, va_offset, __inittext_begin, + map_segment(init_pg_dir, &pgdp, limit, va_offset, __inittext_begin, __inittext_end, prot, false, root_level); - map_segment(init_pg_dir, &pgdp, va_offset, __initdata_begin, + map_segment(init_pg_dir, &pgdp, limit, va_offset, __initdata_begin, __initdata_end, data_prot, false, root_level); - map_segment(init_pg_dir, &pgdp, va_offset, _data, _end, data_prot, - true, root_level); + map_segment(init_pg_dir, &pgdp, limit, va_offset, _data, _end, + data_prot, true, root_level); dsb(ishst); idmap_cpu_replace_ttbr1(init_pg_dir); @@ -120,9 +121,9 @@ static void __init map_kernel(u64 kaslr_offset, u64 va_offset, int root_level) * Remap these segments with different permissions * No new page table allocations should be needed */ - map_segment(init_pg_dir, NULL, va_offset, _stext, _etext, + map_segment(init_pg_dir, NULL, 0, va_offset, _stext, _etext, text_prot, true, root_level); - map_segment(init_pg_dir, NULL, va_offset, __inittext_begin, + map_segment(init_pg_dir, NULL, 0, va_offset, __inittext_begin, __inittext_end, text_prot, false, root_level); } @@ -164,7 +165,7 @@ static void __init remap_idmap_for_lpa2(void) * LPA2 compatible fashion, and update the initial ID map while running * from that. */ - create_init_idmap(init_pg_dir, mask); + create_init_idmap(init_pg_dir, init_pg_end, mask); dsb(ishst); set_ttbr0_for_lpa2((u64)init_pg_dir); @@ -175,7 +176,7 @@ static void __init remap_idmap_for_lpa2(void) memset(init_idmap_pg_dir, 0, (u64)init_idmap_pg_end - (u64)init_idmap_pg_dir); - create_init_idmap(init_idmap_pg_dir, mask); + create_init_idmap(init_idmap_pg_dir, init_idmap_pg_end, mask); dsb(ishst); /* switch back to the updated initial ID map */ @@ -188,6 +189,7 @@ static void __init remap_idmap_for_lpa2(void) static void __init map_fdt(u64 fdt) { static u8 ptes[INIT_IDMAP_FDT_SIZE] __initdata __aligned(PAGE_SIZE); + u64 limit = (u64)&ptes[INIT_IDMAP_FDT_SIZE]; u64 efdt = fdt + MAX_FDT_SIZE; u64 ptep = (u64)ptes; @@ -195,7 +197,8 @@ static void __init map_fdt(u64 fdt) * Map up to MAX_FDT_SIZE bytes, but avoid overlap with * the kernel image. */ - map_range(&ptep, fdt, (u64)_text > fdt ? min((u64)_text, efdt) : efdt, + map_range(&ptep, limit, fdt, + (u64)_text > fdt ? min((u64)_text, efdt) : efdt, fdt, PAGE_KERNEL, IDMAP_ROOT_LEVEL, (pte_t *)init_idmap_pg_dir, false, 0); dsb(ishst); diff --git a/arch/arm64/kernel/pi/map_range.c b/arch/arm64/kernel/pi/map_range.c index 5410b2cac5907..f0024d9b1d921 100644 --- a/arch/arm64/kernel/pi/map_range.c +++ b/arch/arm64/kernel/pi/map_range.c @@ -11,11 +11,36 @@ #include "pi.h" +static void __init mmuoff_data_clean(void) +{ + bool cache_ena = !!(read_sysreg(sctlr_el1) & SCTLR_ELx_C); + + if (cache_ena) + dcache_clean_poc((unsigned long)__mmuoff_data_start, + (unsigned long)__mmuoff_data_end); + else + dcache_inval_poc((unsigned long)__mmuoff_data_start, + (unsigned long)__mmuoff_data_end); +} + +static void __init report_cpu_stuck(long val) +{ + val |= CPU_STUCK_IN_KERNEL; + WRITE_ONCE(__early_cpu_boot_status, val); + + /* Ensure the visibility of the status update */ + dsb(ishst); + mmuoff_data_clean(); + + cpu_park_loop(); +} + /** * map_range - Map a contiguous range of physical pages into virtual memory * * @pte: Address of physical pointer to array of pages to * allocate page tables from + * @limit: Physical address of end of page allocation array * @start: Virtual address of the start of the range * @end: Virtual address of the end of the range (exclusive) * @pa: Physical address of the start of the range @@ -26,8 +51,9 @@ * @va_offset: Offset between a physical page and its current mapping * in the VA space */ -void __init map_range(u64 *pte, u64 start, u64 end, u64 pa, pgprot_t prot, - int level, pte_t *tbl, bool may_use_cont, u64 va_offset) +void __init map_range(u64 *pte, u64 limit, u64 start, u64 end, u64 pa, + pgprot_t prot, int level, pte_t *tbl, bool may_use_cont, + u64 va_offset) { u64 cmask = (level == 3) ? CONT_PTE_SIZE - 1 : U64_MAX; u64 protval = pgprot_val(prot) & ~PTE_TYPE_MASK; @@ -56,11 +82,18 @@ void __init map_range(u64 *pte, u64 start, u64 end, u64 pa, pgprot_t prot, * table mapping if necessary and recurse. */ if (pte_none(*tbl)) { + u64 size = PTRS_PER_PTE * sizeof(pte_t); + + if (*pte + size > limit) { + report_cpu_stuck( + CPU_STUCK_REASON_NO_PGTABLE_MEM); + } + *tbl = __pte(__phys_to_pte_val(*pte) | PMD_TYPE_TABLE | PMD_TABLE_UXN); - *pte += PTRS_PER_PTE * sizeof(pte_t); + *pte += size; } - map_range(pte, start, next, pa, prot, level + 1, + map_range(pte, limit, start, next, pa, prot, level + 1, (pte_t *)(__pte_to_phys(*tbl) + va_offset), may_use_cont, va_offset); } else { @@ -87,7 +120,8 @@ void __init map_range(u64 *pte, u64 start, u64 end, u64 pa, pgprot_t prot, } } -asmlinkage u64 __init create_init_idmap(pgd_t *pg_dir, pteval_t clrmask) +asmlinkage u64 __init create_init_idmap(pgd_t *pg_dir, pgd_t *pg_end, + pteval_t clrmask) { u64 ptep = (u64)pg_dir + PAGE_SIZE; pgprot_t text_prot = PAGE_KERNEL_ROX; @@ -96,10 +130,12 @@ asmlinkage u64 __init create_init_idmap(pgd_t *pg_dir, pteval_t clrmask) pgprot_val(text_prot) &= ~clrmask; pgprot_val(data_prot) &= ~clrmask; - map_range(&ptep, (u64)_stext, (u64)__initdata_begin, (u64)_stext, - text_prot, IDMAP_ROOT_LEVEL, (pte_t *)pg_dir, false, 0); - map_range(&ptep, (u64)__initdata_begin, (u64)_end, (u64)__initdata_begin, - data_prot, IDMAP_ROOT_LEVEL, (pte_t *)pg_dir, false, 0); + map_range(&ptep, (u64)pg_end, (u64)_stext, (u64)__initdata_begin, + (u64)_stext, text_prot, IDMAP_ROOT_LEVEL, (pte_t *)pg_dir, + false, 0); + map_range(&ptep, (u64)pg_end, (u64)__initdata_begin, (u64)_end, + (u64)__initdata_begin, data_prot, IDMAP_ROOT_LEVEL, + (pte_t *)pg_dir, false, 0); return ptep; } diff --git a/arch/arm64/kernel/pi/pi.h b/arch/arm64/kernel/pi/pi.h index c91e5e965cd39..20fe0941cb8ee 100644 --- a/arch/arm64/kernel/pi/pi.h +++ b/arch/arm64/kernel/pi/pi.h @@ -28,9 +28,9 @@ u64 kaslr_early_init(void *fdt, int chosen); void relocate_kernel(u64 offset); int scs_patch(const u8 eh_frame[], int size); -void map_range(u64 *pgd, u64 start, u64 end, u64 pa, pgprot_t prot, +void map_range(u64 *pgd, u64 limit, u64 start, u64 end, u64 pa, pgprot_t prot, int level, pte_t *tbl, bool may_use_cont, u64 va_offset); asmlinkage void early_map_kernel(u64 boot_status, void *fdt); -asmlinkage u64 create_init_idmap(pgd_t *pgd, pteval_t clrmask); +asmlinkage u64 create_init_idmap(pgd_t *pgd, pgd_t *pg_end, pteval_t clrmask); diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index 353ea5dc32b85..969348a2e93c9 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -773,8 +773,9 @@ static void __init declare_kernel_vmas(void) declare_vma(&vmlinux_seg[4], _data, _end, 0); } -void __pi_map_range(u64 *pgd, u64 start, u64 end, u64 pa, pgprot_t prot, - int level, pte_t *tbl, bool may_use_cont, u64 va_offset); +void __pi_map_range(u64 *pgd, u64 limit, u64 start, u64 end, u64 pa, + pgprot_t prot, int level, pte_t *tbl, bool may_use_cont, + u64 va_offset); static u8 idmap_ptes[IDMAP_LEVELS - 1][PAGE_SIZE] __aligned(PAGE_SIZE) __ro_after_init, kpti_ptes[IDMAP_LEVELS - 1][PAGE_SIZE] __aligned(PAGE_SIZE) __ro_after_init; @@ -784,8 +785,9 @@ static void __init create_idmap(void) u64 start = __pa_symbol(__idmap_text_start); u64 end = __pa_symbol(__idmap_text_end); u64 ptep = __pa_symbol(idmap_ptes); + u64 limit = __pa_symbol(&idmap_ptes[IDMAP_LEVELS - 1][0]); - __pi_map_range(&ptep, start, end, start, PAGE_KERNEL_ROX, + __pi_map_range(&ptep, limit, start, end, start, PAGE_KERNEL_ROX, IDMAP_ROOT_LEVEL, (pte_t *)idmap_pg_dir, false, __phys_to_virt(ptep) - ptep); @@ -798,8 +800,10 @@ static void __init create_idmap(void) * of its synchronization flag in the ID map. */ ptep = __pa_symbol(kpti_ptes); - __pi_map_range(&ptep, pa, pa + sizeof(u32), pa, PAGE_KERNEL, - IDMAP_ROOT_LEVEL, (pte_t *)idmap_pg_dir, false, + limit = __pa_symbol(&kpti_ptes[IDMAP_LEVELS - 1][0]); + __pi_map_range(&ptep, limit, pa, pa + sizeof(u32), pa, + PAGE_KERNEL, IDMAP_ROOT_LEVEL, + (pte_t *)idmap_pg_dir, false, __phys_to_virt(ptep) - ptep); } } From patchwork Mon Oct 14 10:58:46 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13834900 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id C3822D16252 for ; Mon, 14 Oct 2024 12:19:01 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=11sxvXvLRPO0HsHmpSNAXFOkl+67WniK18hQuyFjNNA=; b=f3NEMAW0ulIVee6cT2t/rKB6zT aH0mdraZmrst7ren804doG0/sBQXYCppDGk5KQ4kfwVWV4o5wQC4oaLdAxbinth7eibS68XK+IUc6 MrzMbUgrr2avdElGnkG0ue2MaQg5REMwM9V/CR+bPzG4LDFExliVTSt7exVdD7MCLbDwHk7CcePUw RkVDU2tMCIpbiOrUwRVkyyTAPYSlAKa0lcFKDIaDpS3+SVUH2sGlggoH2jCryelhlQzt/xEbKaUxK JmnY6UqBbRWZBEzYny5oSknb39X5FC7Mj047AqNYM6eBSfHsPg2ec9C7ndoRKht5MNMTWnDLsucSC HpfqZCoA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1t0K2G-000000053cl-12YJ; Mon, 14 Oct 2024 12:18:52 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1t0IpN-00000004oMq-1vcd for linux-arm-kernel@lists.infradead.org; Mon, 14 Oct 2024 11:01:31 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 6F1811756; Mon, 14 Oct 2024 04:01:58 -0700 (PDT) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.27]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 6E88C3F51B; Mon, 14 Oct 2024 04:01:26 -0700 (PDT) From: Ryan Roberts To: Andrew Morton , Anshuman Khandual , Ard Biesheuvel , Catalin Marinas , David Hildenbrand , Greg Marsden , Ivan Ivanov , Kalesh Singh , Marc Zyngier , Mark Rutland , Matthias Brugger , Miroslav Benes , Will Deacon Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [RFC PATCH v1 39/57] arm64: Introduce macros required for boot-time page selection Date: Mon, 14 Oct 2024 11:58:46 +0100 Message-ID: <20241014105912.3207374-39-ryan.roberts@arm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20241014105912.3207374-1-ryan.roberts@arm.com> References: <20241014105514.3206191-1-ryan.roberts@arm.com> <20241014105912.3207374-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241014_040129_670714_99E4AB2B X-CRM114-Status: GOOD ( 15.27 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org This minmal set of macros will allow boot-time page selection support to be added to the arm64 arch code incrementally over the following set of patches. The definitions in pgtable-geometry.h are for compile-time page size currently, but they will be modified in future to support boot-time page size. Signed-off-by: Ryan Roberts --- ***NOTE*** Any confused maintainers may want to read the cover note here for context: https://lore.kernel.org/all/20241014105514.3206191-1-ryan.roberts@arm.com/ arch/arm64/include/asm/page-def.h | 5 ++-- arch/arm64/include/asm/pgtable-geometry.h | 28 +++++++++++++++++++++++ arch/arm64/include/asm/pgtable-hwdef.h | 16 ++++++++----- 3 files changed, 40 insertions(+), 9 deletions(-) create mode 100644 arch/arm64/include/asm/pgtable-geometry.h diff --git a/arch/arm64/include/asm/page-def.h b/arch/arm64/include/asm/page-def.h index d69971cf49cd2..b99dee0112463 100644 --- a/arch/arm64/include/asm/page-def.h +++ b/arch/arm64/include/asm/page-def.h @@ -9,12 +9,11 @@ #define __ASM_PAGE_DEF_H #include +#include /* PAGE_SHIFT determines the page size */ -#define PAGE_SHIFT CONFIG_PAGE_SHIFT +#define PAGE_SHIFT ptg_page_shift #define PAGE_SIZE (_AC(1, UL) << PAGE_SHIFT) #define PAGE_MASK (~(PAGE_SIZE-1)) -#include - #endif /* __ASM_PAGE_DEF_H */ diff --git a/arch/arm64/include/asm/pgtable-geometry.h b/arch/arm64/include/asm/pgtable-geometry.h new file mode 100644 index 0000000000000..62fe125909c08 --- /dev/null +++ b/arch/arm64/include/asm/pgtable-geometry.h @@ -0,0 +1,28 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +#ifndef ASM_PGTABLE_GEOMETRY_H +#define ASM_PGTABLE_GEOMETRY_H + +#define ARM64_PAGE_SHIFT_4K 12 +#define ARM64_PAGE_SHIFT_16K 14 +#define ARM64_PAGE_SHIFT_64K 16 + +#define PAGE_SHIFT_MIN CONFIG_PAGE_SHIFT +#define PAGE_SIZE_MIN (_AC(1, UL) << PAGE_SHIFT_MIN) +#define PAGE_MASK_MIN (~(PAGE_SIZE_MIN-1)) + +#define PAGE_SHIFT_MAX CONFIG_PAGE_SHIFT +#define PAGE_SIZE_MAX (_AC(1, UL) << PAGE_SHIFT_MAX) +#define PAGE_MASK_MAX (~(PAGE_SIZE_MAX-1)) + +#include + +#define ptg_page_shift CONFIG_PAGE_SHIFT +#define ptg_pmd_shift ARM64_HW_PGTABLE_LEVEL_SHIFT(2) +#define ptg_pud_shift ARM64_HW_PGTABLE_LEVEL_SHIFT(1) +#define ptg_p4d_shift ARM64_HW_PGTABLE_LEVEL_SHIFT(0) +#define ptg_pgdir_shift ARM64_HW_PGTABLE_LEVEL_SHIFT(4 - CONFIG_PGTABLE_LEVELS) +#define ptg_cont_pte_shift (CONFIG_ARM64_CONT_PTE_SHIFT + PAGE_SHIFT) +#define ptg_cont_pmd_shift (CONFIG_ARM64_CONT_PMD_SHIFT + PMD_SHIFT) +#define ptg_pgtable_levels CONFIG_PGTABLE_LEVELS + +#endif /* ASM_PGTABLE_GEOMETRY_H */ diff --git a/arch/arm64/include/asm/pgtable-hwdef.h b/arch/arm64/include/asm/pgtable-hwdef.h index 1f60aa1bc750c..54a9153f56bc5 100644 --- a/arch/arm64/include/asm/pgtable-hwdef.h +++ b/arch/arm64/include/asm/pgtable-hwdef.h @@ -41,39 +41,43 @@ #define ARM64_HW_PGTABLE_LEVEL_SHIFT(n) ((PAGE_SHIFT - 3) * (4 - (n)) + 3) #define PTRS_PER_PTE (1 << (PAGE_SHIFT - 3)) +#define MAX_PTRS_PER_PTE (1 << (PAGE_SHIFT_MAX - 3)) /* * PMD_SHIFT determines the size a level 2 page table entry can map. */ #if CONFIG_PGTABLE_LEVELS > 2 -#define PMD_SHIFT ARM64_HW_PGTABLE_LEVEL_SHIFT(2) +#define PMD_SHIFT ptg_pmd_shift #define PMD_SIZE (_AC(1, UL) << PMD_SHIFT) #define PMD_MASK (~(PMD_SIZE-1)) #define PTRS_PER_PMD (1 << (PAGE_SHIFT - 3)) +#define MAX_PTRS_PER_PMD (1 << (PAGE_SHIFT_MAX - 3)) #endif /* * PUD_SHIFT determines the size a level 1 page table entry can map. */ #if CONFIG_PGTABLE_LEVELS > 3 -#define PUD_SHIFT ARM64_HW_PGTABLE_LEVEL_SHIFT(1) +#define PUD_SHIFT ptg_pud_shift #define PUD_SIZE (_AC(1, UL) << PUD_SHIFT) #define PUD_MASK (~(PUD_SIZE-1)) #define PTRS_PER_PUD (1 << (PAGE_SHIFT - 3)) +#define MAX_PTRS_PER_PUD (1 << (PAGE_SHIFT_MAX - 3)) #endif #if CONFIG_PGTABLE_LEVELS > 4 -#define P4D_SHIFT ARM64_HW_PGTABLE_LEVEL_SHIFT(0) +#define P4D_SHIFT ptg_p4d_shift #define P4D_SIZE (_AC(1, UL) << P4D_SHIFT) #define P4D_MASK (~(P4D_SIZE-1)) #define PTRS_PER_P4D (1 << (PAGE_SHIFT - 3)) +#define MAX_PTRS_PER_P4D (1 << (PAGE_SHIFT_MAX - 3)) #endif /* * PGDIR_SHIFT determines the size a top-level page table entry can map * (depending on the configuration, this level can be -1, 0, 1 or 2). */ -#define PGDIR_SHIFT ARM64_HW_PGTABLE_LEVEL_SHIFT(4 - CONFIG_PGTABLE_LEVELS) +#define PGDIR_SHIFT ptg_pgdir_shift #define PGDIR_SIZE (_AC(1, UL) << PGDIR_SHIFT) #define PGDIR_MASK (~(PGDIR_SIZE-1)) #define PTRS_PER_PGD (1 << (VA_BITS - PGDIR_SHIFT)) @@ -81,12 +85,12 @@ /* * Contiguous page definitions. */ -#define CONT_PTE_SHIFT (CONFIG_ARM64_CONT_PTE_SHIFT + PAGE_SHIFT) +#define CONT_PTE_SHIFT ptg_cont_pte_shift #define CONT_PTES (1 << (CONT_PTE_SHIFT - PAGE_SHIFT)) #define CONT_PTE_SIZE (CONT_PTES * PAGE_SIZE) #define CONT_PTE_MASK (~(CONT_PTE_SIZE - 1)) -#define CONT_PMD_SHIFT (CONFIG_ARM64_CONT_PMD_SHIFT + PMD_SHIFT) +#define CONT_PMD_SHIFT ptg_cont_pmd_shift #define CONT_PMDS (1 << (CONT_PMD_SHIFT - PMD_SHIFT)) #define CONT_PMD_SIZE (CONT_PMDS * PMD_SIZE) #define CONT_PMD_MASK (~(CONT_PMD_SIZE - 1)) From patchwork Mon Oct 14 10:58:47 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13834901 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 00B58D16251 for ; Mon, 14 Oct 2024 12:20:29 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=v2NcdhGv+sWZMGvU1TlMDdXozrRt5nJhcZPuO+Evnng=; b=LUJ5DjoQHh54fYa9U50nWikSTR zeGkuCNbYaEaBXvq4DgKpj0w0n/7kdSD5F7eiMVL+/vJGTXGOTPIRjpwCxNtVUO3ly7XRR0ghzCVA 2Y9KdfbsFlRQu8mYOW9ozM8C/oPeVzzFeoH29MQQw5kMXkVc7v/fPEz8RL8wXOq5wXk0CXZNK40EP AScDLoe972vJ/1JIPWlatYWedgyITo5YazN1iL6Qf/OYduET7kwMkZnQo2I7aFt6oCkg1VdlnTaAL LwQirsz+P+GPc4etyhy1kXvK+B6uQW0e0JjB1ly1qyC+fZQrfi3KSz2fIP6f4++InTX6kBsZbLffh ojAexjPQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1t0K3d-000000053nF-0w4U; Mon, 14 Oct 2024 12:20:17 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1t0IpP-00000004oOW-3RtN for linux-arm-kernel@lists.infradead.org; Mon, 14 Oct 2024 11:01:37 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 1FFE01713; Mon, 14 Oct 2024 04:02:01 -0700 (PDT) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.27]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 204203F51B; Mon, 14 Oct 2024 04:01:29 -0700 (PDT) From: Ryan Roberts To: Andrew Morton , Anshuman Khandual , Ard Biesheuvel , Catalin Marinas , David Hildenbrand , Greg Marsden , Ivan Ivanov , Kalesh Singh , Marc Zyngier , Mark Rutland , Matthias Brugger , Miroslav Benes , Will Deacon Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [RFC PATCH v1 40/57] arm64: Refactor early pgtable size calculation macros Date: Mon, 14 Oct 2024 11:58:47 +0100 Message-ID: <20241014105912.3207374-40-ryan.roberts@arm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20241014105912.3207374-1-ryan.roberts@arm.com> References: <20241014105514.3206191-1-ryan.roberts@arm.com> <20241014105912.3207374-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241014_040132_053128_E4AE8DF6 X-CRM114-Status: GOOD ( 21.45 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org The various early idmaps and init/swapper pgtables are constructed using static storage, the size of which is obviously calculated at compile-time based on the selected page size. But in the near future, boot-time page size builds will need to statically allocate enough storage for the worst case, depending on which page size is selected. Therefore, refactor the macros that determine the storage requirement to take a page_shift parameter, then perform the calculation for each page size we are compiling with support for and take the max. For compile-time page size builds, the end result is exactly the same because there is only 1 page size we support. For boot-time page size builds we end up with the worst case required size. Signed-off-by: Ryan Roberts --- ***NOTE*** Any confused maintainers may want to read the cover note here for context: https://lore.kernel.org/all/20241014105514.3206191-1-ryan.roberts@arm.com/ arch/arm64/include/asm/kernel-pgtable.h | 148 +++++++++++++++++------- arch/arm64/include/asm/pgtable-hwdef.h | 6 +- arch/arm64/kernel/pi/map_kernel.c | 6 +- arch/arm64/kernel/pi/map_range.c | 8 +- arch/arm64/kernel/vmlinux.lds.S | 4 +- arch/arm64/mm/mmu.c | 13 +-- 6 files changed, 124 insertions(+), 61 deletions(-) diff --git a/arch/arm64/include/asm/kernel-pgtable.h b/arch/arm64/include/asm/kernel-pgtable.h index 1722b9217d47d..facdf273d4cda 100644 --- a/arch/arm64/include/asm/kernel-pgtable.h +++ b/arch/arm64/include/asm/kernel-pgtable.h @@ -12,28 +12,38 @@ #include #include +#define PGTABLE_LEVELS(page_shift, va_bits) \ + __ARM64_HW_PGTABLE_LEVELS(page_shift, va_bits) +#define PGTABLE_LEVEL_SHIFT(page_shift, n) \ + __ARM64_HW_PGTABLE_LEVEL_SHIFT(page_shift, n) +#define PGTABLE_LEVEL_SIZE(page_shift, n) \ + (UL(1) << PGTABLE_LEVEL_SHIFT(page_shift, n)) + /* * The physical and virtual addresses of the start of the kernel image are * equal modulo 2 MiB (per the arm64 booting.txt requirements). Hence we can * use section mapping with 4K (section size = 2M) but not with 16K (section * size = 32M) or 64K (section size = 512M). */ -#if defined(PMD_SIZE) && PMD_SIZE <= MIN_KIMG_ALIGN -#define SWAPPER_BLOCK_SHIFT PMD_SHIFT -#define SWAPPER_SKIP_LEVEL 1 -#else -#define SWAPPER_BLOCK_SHIFT PAGE_SHIFT -#define SWAPPER_SKIP_LEVEL 0 -#endif -#define SWAPPER_BLOCK_SIZE (UL(1) << SWAPPER_BLOCK_SHIFT) -#define SWAPPER_TABLE_SHIFT (SWAPPER_BLOCK_SHIFT + PAGE_SHIFT - 3) - -#define SWAPPER_PGTABLE_LEVELS (CONFIG_PGTABLE_LEVELS - SWAPPER_SKIP_LEVEL) -#define INIT_IDMAP_PGTABLE_LEVELS (IDMAP_LEVELS - SWAPPER_SKIP_LEVEL) - -#define IDMAP_VA_BITS 48 -#define IDMAP_LEVELS ARM64_HW_PGTABLE_LEVELS(IDMAP_VA_BITS) -#define IDMAP_ROOT_LEVEL (4 - IDMAP_LEVELS) +#define SWAPPER_BLOCK_SHIFT(page_shift) \ + ((PGTABLE_LEVEL_SIZE(page_shift, 2) <= MIN_KIMG_ALIGN) ? \ + PGTABLE_LEVEL_SHIFT(page_shift, 2) : (page_shift)) + +#define SWAPPER_SKIP_LEVEL(page_shift) \ + ((PGTABLE_LEVEL_SIZE(page_shift, 2) <= MIN_KIMG_ALIGN) ? 1 : 0) + +#define SWAPPER_BLOCK_SIZE(page_shift) \ + (UL(1) << SWAPPER_BLOCK_SHIFT(page_shift)) + +#define SWAPPER_PGTABLE_LEVELS(page_shift) \ + (PGTABLE_LEVELS(page_shift, VA_BITS) - SWAPPER_SKIP_LEVEL(page_shift)) + +#define INIT_IDMAP_PGTABLE_LEVELS(page_shift) \ + (IDMAP_LEVELS(page_shift) - SWAPPER_SKIP_LEVEL(page_shift)) + +#define IDMAP_VA_BITS 48 +#define IDMAP_LEVELS(page_shift) PGTABLE_LEVELS(page_shift, IDMAP_VA_BITS) +#define IDMAP_ROOT_LEVEL(page_shift) (4 - IDMAP_LEVELS(page_shift)) #define KIMAGE_VADDR_END (_AT(u64, _end) - _AT(u64, _text) + KIMAGE_VADDR) @@ -43,47 +53,99 @@ * with two adjacent PGDIR entries, which means that an additional page table * may be needed at each subordinate level. */ -#define EXTRA_PAGE __is_defined(CONFIG_RELOCATABLE) +#define EXTRA_PAGE __is_defined(CONFIG_RELOCATABLE) -#define SPAN_NR_ENTRIES(vstart, vend, shift) \ +#define SPAN_NR_ENTRIES(vstart, vend, shift) \ ((((vend) - 1) >> (shift)) - ((vstart) >> (shift)) + 1) -#define EARLY_ENTRIES(vstart, vend, shift, add) \ +#define EARLY_ENTRIES(vstart, vend, shift, add) \ (SPAN_NR_ENTRIES(vstart, vend, shift) + (add)) -#define EARLY_LEVEL(lvl, lvls, vstart, vend, add) \ - (lvls > lvl ? EARLY_ENTRIES(vstart, vend, SWAPPER_BLOCK_SHIFT + lvl * (PAGE_SHIFT - 3), add) : 0) - -#define EARLY_PAGES(lvls, vstart, vend, add) (1 /* PGDIR page */ \ - + EARLY_LEVEL(3, (lvls), (vstart), (vend), add) /* each entry needs a next level page table */ \ - + EARLY_LEVEL(2, (lvls), (vstart), (vend), add) /* each entry needs a next level page table */ \ - + EARLY_LEVEL(1, (lvls), (vstart), (vend), add))/* each entry needs a next level page table */ -#define INIT_DIR_SIZE (PAGE_SIZE * (EARLY_PAGES(SWAPPER_PGTABLE_LEVELS, KIMAGE_VADDR, KIMAGE_VADDR_END, EXTRA_PAGE) \ - + EARLY_SEGMENT_EXTRA_PAGES)) - -#define INIT_IDMAP_DIR_PAGES (EARLY_PAGES(INIT_IDMAP_PGTABLE_LEVELS, KIMAGE_VADDR, KIMAGE_VADDR_END, 1)) -#define INIT_IDMAP_DIR_SIZE ((INIT_IDMAP_DIR_PAGES + EARLY_IDMAP_EXTRA_PAGES) * PAGE_SIZE) +#define EARLY_LEVEL(page_shift, lvl, lvls, vstart, vend, add) \ + (lvls > lvl ? EARLY_ENTRIES(vstart, vend, \ + SWAPPER_BLOCK_SHIFT(page_shift) + lvl * ((page_shift) - 3), \ + add) : 0) -#define INIT_IDMAP_FDT_PAGES (EARLY_PAGES(INIT_IDMAP_PGTABLE_LEVELS, 0UL, UL(MAX_FDT_SIZE), 1) - 1) -#define INIT_IDMAP_FDT_SIZE ((INIT_IDMAP_FDT_PAGES + EARLY_IDMAP_EXTRA_FDT_PAGES) * PAGE_SIZE) +#define EARLY_PAGES(page_shift, lvls, vstart, vend, add) (1 /* PGDIR */ \ + + EARLY_LEVEL((page_shift), 3, (lvls), (vstart), (vend), add) \ + + EARLY_LEVEL((page_shift), 2, (lvls), (vstart), (vend), add) \ + + EARLY_LEVEL((page_shift), 1, (lvls), (vstart), (vend), add)) /* The number of segments in the kernel image (text, rodata, inittext, initdata, data+bss) */ -#define KERNEL_SEGMENT_COUNT 5 +#define KERNEL_SEGMENT_COUNT 5 -#if SWAPPER_BLOCK_SIZE > SEGMENT_ALIGN -#define EARLY_SEGMENT_EXTRA_PAGES (KERNEL_SEGMENT_COUNT + 1) /* * The initial ID map consists of the kernel image, mapped as two separate * segments, and may appear misaligned wrt the swapper block size. This means * we need 3 additional pages. The DT could straddle a swapper block boundary, * so it may need 2. */ -#define EARLY_IDMAP_EXTRA_PAGES 3 -#define EARLY_IDMAP_EXTRA_FDT_PAGES 2 -#else -#define EARLY_SEGMENT_EXTRA_PAGES 0 -#define EARLY_IDMAP_EXTRA_PAGES 0 -#define EARLY_IDMAP_EXTRA_FDT_PAGES 0 -#endif +#define EARLY_SEGMENT_EXTRA_PAGES(page_shift) \ + ((SWAPPER_BLOCK_SIZE(page_shift) > SEGMENT_ALIGN) ? \ + (KERNEL_SEGMENT_COUNT + 1) : 0) + +#define EARLY_IDMAP_EXTRA_PAGES(page_shift) \ + ((SWAPPER_BLOCK_SIZE(page_shift) > SEGMENT_ALIGN) ? 3 : 0) + +#define EARLY_IDMAP_EXTRA_FDT_PAGES(page_shift) \ + ((SWAPPER_BLOCK_SIZE(page_shift) > SEGMENT_ALIGN) ? 2 : 0) + +#define INIT_DIR_PAGES(page_shift) \ + (EARLY_PAGES((page_shift), SWAPPER_PGTABLE_LEVELS(page_shift), \ + KIMAGE_VADDR, KIMAGE_VADDR_END, EXTRA_PAGE)) + +#define INIT_DIR_SIZE(page_shift) \ + ((INIT_DIR_PAGES(page_shift) + \ + EARLY_SEGMENT_EXTRA_PAGES(page_shift)) * (UL(1) << (page_shift))) + +#define INIT_IDMAP_DIR_PAGES(page_shift) \ + (EARLY_PAGES((page_shift), \ + INIT_IDMAP_PGTABLE_LEVELS(page_shift), \ + KIMAGE_VADDR, KIMAGE_VADDR_END, 1)) + +#define INIT_IDMAP_DIR_SIZE(page_shift) \ + ((INIT_IDMAP_DIR_PAGES(page_shift) + \ + EARLY_IDMAP_EXTRA_PAGES(page_shift)) * (UL(1) << (page_shift))) + +#define INIT_IDMAP_FDT_PAGES(page_shift) \ + (EARLY_PAGES((page_shift), \ + INIT_IDMAP_PGTABLE_LEVELS(page_shift), \ + UL(0), UL(MAX_FDT_SIZE), 1) - 1) + +#define INIT_IDMAP_FDT_SIZE(page_shift) \ + ((INIT_IDMAP_FDT_PAGES(page_shift) + \ + EARLY_IDMAP_EXTRA_FDT_PAGES(page_shift)) * (UL(1) << (page_shift))) + +#define VAL_IF_HAVE_PGSZ(val, page_shift) \ + ((page_shift) >= PAGE_SHIFT_MIN && \ + (page_shift) <= PAGE_SHIFT_MAX ? (val) : 0) + +#define MAX_IF_HAVE_PGSZ(val4k, val16k, val64k) \ + MAX(VAL_IF_HAVE_PGSZ((val4k), ARM64_PAGE_SHIFT_4K), MAX( \ + VAL_IF_HAVE_PGSZ((val16k), ARM64_PAGE_SHIFT_16K), \ + VAL_IF_HAVE_PGSZ((val64k), ARM64_PAGE_SHIFT_64K))) + +#define IDMAP_LEVELS_MAX \ + MAX_IF_HAVE_PGSZ(IDMAP_LEVELS(ARM64_PAGE_SHIFT_4K), \ + IDMAP_LEVELS(ARM64_PAGE_SHIFT_16K), \ + IDMAP_LEVELS(ARM64_PAGE_SHIFT_64K)) + +#define __INIT_DIR_SIZE_MAX \ + MAX_IF_HAVE_PGSZ(INIT_DIR_SIZE(ARM64_PAGE_SHIFT_4K), \ + INIT_DIR_SIZE(ARM64_PAGE_SHIFT_16K), \ + INIT_DIR_SIZE(ARM64_PAGE_SHIFT_64K)) + +#define INIT_DIR_SIZE_MAX \ + MAX(__INIT_DIR_SIZE_MAX, INIT_IDMAP_DIR_SIZE_MAX) + +#define INIT_IDMAP_DIR_SIZE_MAX \ + MAX_IF_HAVE_PGSZ(INIT_IDMAP_DIR_SIZE(ARM64_PAGE_SHIFT_4K), \ + INIT_IDMAP_DIR_SIZE(ARM64_PAGE_SHIFT_16K), \ + INIT_IDMAP_DIR_SIZE(ARM64_PAGE_SHIFT_64K)) + +#define INIT_IDMAP_FDT_SIZE_MAX \ + MAX_IF_HAVE_PGSZ(INIT_IDMAP_FDT_SIZE(ARM64_PAGE_SHIFT_4K), \ + INIT_IDMAP_FDT_SIZE(ARM64_PAGE_SHIFT_16K), \ + INIT_IDMAP_FDT_SIZE(ARM64_PAGE_SHIFT_64K)) #endif /* __ASM_KERNEL_PGTABLE_H */ diff --git a/arch/arm64/include/asm/pgtable-hwdef.h b/arch/arm64/include/asm/pgtable-hwdef.h index 54a9153f56bc5..ca8bcbc1fe220 100644 --- a/arch/arm64/include/asm/pgtable-hwdef.h +++ b/arch/arm64/include/asm/pgtable-hwdef.h @@ -23,7 +23,8 @@ * * which gets simplified as : */ -#define ARM64_HW_PGTABLE_LEVELS(va_bits) (((va_bits) - 4) / (PAGE_SHIFT - 3)) +#define __ARM64_HW_PGTABLE_LEVELS(page_shift, va_bits) (((va_bits) - 4) / ((page_shift) - 3)) +#define ARM64_HW_PGTABLE_LEVELS(va_bits) __ARM64_HW_PGTABLE_LEVELS(PAGE_SHIFT, va_bits) /* * Size mapped by an entry at level n ( -1 <= n <= 3) @@ -38,7 +39,8 @@ * Rearranging it a bit we get : * (4 - n) * (PAGE_SHIFT - 3) + 3 */ -#define ARM64_HW_PGTABLE_LEVEL_SHIFT(n) ((PAGE_SHIFT - 3) * (4 - (n)) + 3) +#define __ARM64_HW_PGTABLE_LEVEL_SHIFT(page_shift, n) (((page_shift) - 3) * (4 - (n)) + 3) +#define ARM64_HW_PGTABLE_LEVEL_SHIFT(n) __ARM64_HW_PGTABLE_LEVEL_SHIFT(PAGE_SHIFT, n) #define PTRS_PER_PTE (1 << (PAGE_SHIFT - 3)) #define MAX_PTRS_PER_PTE (1 << (PAGE_SHIFT_MAX - 3)) diff --git a/arch/arm64/kernel/pi/map_kernel.c b/arch/arm64/kernel/pi/map_kernel.c index dcf9233ccfff2..a53fc225d2d0d 100644 --- a/arch/arm64/kernel/pi/map_kernel.c +++ b/arch/arm64/kernel/pi/map_kernel.c @@ -188,8 +188,8 @@ static void __init remap_idmap_for_lpa2(void) static void __init map_fdt(u64 fdt) { - static u8 ptes[INIT_IDMAP_FDT_SIZE] __initdata __aligned(PAGE_SIZE); - u64 limit = (u64)&ptes[INIT_IDMAP_FDT_SIZE]; + static u8 ptes[INIT_IDMAP_FDT_SIZE_MAX] __initdata __aligned(PAGE_SIZE); + u64 limit = (u64)&ptes[INIT_IDMAP_FDT_SIZE_MAX]; u64 efdt = fdt + MAX_FDT_SIZE; u64 ptep = (u64)ptes; @@ -199,7 +199,7 @@ static void __init map_fdt(u64 fdt) */ map_range(&ptep, limit, fdt, (u64)_text > fdt ? min((u64)_text, efdt) : efdt, - fdt, PAGE_KERNEL, IDMAP_ROOT_LEVEL, + fdt, PAGE_KERNEL, IDMAP_ROOT_LEVEL(PAGE_SHIFT), (pte_t *)init_idmap_pg_dir, false, 0); dsb(ishst); } diff --git a/arch/arm64/kernel/pi/map_range.c b/arch/arm64/kernel/pi/map_range.c index f0024d9b1d921..b62d2e3135f81 100644 --- a/arch/arm64/kernel/pi/map_range.c +++ b/arch/arm64/kernel/pi/map_range.c @@ -131,11 +131,11 @@ asmlinkage u64 __init create_init_idmap(pgd_t *pg_dir, pgd_t *pg_end, pgprot_val(data_prot) &= ~clrmask; map_range(&ptep, (u64)pg_end, (u64)_stext, (u64)__initdata_begin, - (u64)_stext, text_prot, IDMAP_ROOT_LEVEL, (pte_t *)pg_dir, - false, 0); + (u64)_stext, text_prot, + IDMAP_ROOT_LEVEL(PAGE_SHIFT), (pte_t *)pg_dir, false, 0); map_range(&ptep, (u64)pg_end, (u64)__initdata_begin, (u64)_end, - (u64)__initdata_begin, data_prot, IDMAP_ROOT_LEVEL, - (pte_t *)pg_dir, false, 0); + (u64)__initdata_begin, data_prot, + IDMAP_ROOT_LEVEL(PAGE_SHIFT), (pte_t *)pg_dir, false, 0); return ptep; } diff --git a/arch/arm64/kernel/vmlinux.lds.S b/arch/arm64/kernel/vmlinux.lds.S index 55a8e310ea12c..7f3f6d709ae73 100644 --- a/arch/arm64/kernel/vmlinux.lds.S +++ b/arch/arm64/kernel/vmlinux.lds.S @@ -249,7 +249,7 @@ SECTIONS __initdata_begin = .; init_idmap_pg_dir = .; - . += INIT_IDMAP_DIR_SIZE; + . += INIT_IDMAP_DIR_SIZE_MAX; init_idmap_pg_end = .; .init.data : { @@ -319,7 +319,7 @@ SECTIONS . = ALIGN(PAGE_SIZE); init_pg_dir = .; - . += INIT_DIR_SIZE; + . += INIT_DIR_SIZE_MAX; init_pg_end = .; /* end of zero-init region */ diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index 969348a2e93c9..d4d30eaefb4cd 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -777,19 +777,19 @@ void __pi_map_range(u64 *pgd, u64 limit, u64 start, u64 end, u64 pa, pgprot_t prot, int level, pte_t *tbl, bool may_use_cont, u64 va_offset); -static u8 idmap_ptes[IDMAP_LEVELS - 1][PAGE_SIZE] __aligned(PAGE_SIZE) __ro_after_init, - kpti_ptes[IDMAP_LEVELS - 1][PAGE_SIZE] __aligned(PAGE_SIZE) __ro_after_init; +static u8 idmap_ptes[IDMAP_LEVELS_MAX - 1][PAGE_SIZE] __aligned(PAGE_SIZE) __ro_after_init, + kpti_ptes[IDMAP_LEVELS_MAX - 1][PAGE_SIZE] __aligned(PAGE_SIZE) __ro_after_init; static void __init create_idmap(void) { u64 start = __pa_symbol(__idmap_text_start); u64 end = __pa_symbol(__idmap_text_end); u64 ptep = __pa_symbol(idmap_ptes); - u64 limit = __pa_symbol(&idmap_ptes[IDMAP_LEVELS - 1][0]); + u64 limit = __pa_symbol(&idmap_ptes[IDMAP_LEVELS_MAX - 1][0]); __pi_map_range(&ptep, limit, start, end, start, PAGE_KERNEL_ROX, - IDMAP_ROOT_LEVEL, (pte_t *)idmap_pg_dir, false, - __phys_to_virt(ptep) - ptep); + IDMAP_ROOT_LEVEL(PAGE_SHIFT), (pte_t *)idmap_pg_dir, + false, __phys_to_virt(ptep) - ptep); if (IS_ENABLED(CONFIG_UNMAP_KERNEL_AT_EL0) && !arm64_use_ng_mappings) { extern u32 __idmap_kpti_flag; @@ -800,9 +800,8 @@ static void __init create_idmap(void) * of its synchronization flag in the ID map. */ ptep = __pa_symbol(kpti_ptes); - limit = __pa_symbol(&kpti_ptes[IDMAP_LEVELS - 1][0]); __pi_map_range(&ptep, limit, pa, pa + sizeof(u32), pa, - PAGE_KERNEL, IDMAP_ROOT_LEVEL, + PAGE_KERNEL, IDMAP_ROOT_LEVEL(PAGE_SHIFT), (pte_t *)idmap_pg_dir, false, __phys_to_virt(ptep) - ptep); } From patchwork Mon Oct 14 10:58:48 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13834902 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 086D0D16251 for ; Mon, 14 Oct 2024 12:21:53 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=q5VF7i0WQs1N+NdAn0MqfjtRBJLE8glwn7/XULIeBsQ=; b=iObsKSt+BcyJxGgmTfZCvMdFFa TsewVsjG6UyAlzELdCnUmRuR/IMptOjDnsnu823Q564MfKOqa/CGJ9YnoTpWLg/nNztT+p3KYUk75 yhPmBnf6c/8gFlvBEB0rkIT3Vi9idutkqGe0SiSHEbmUQUDsnVVA7Zz/YS9QSCNsvdA/cE6l6r+yj TQyGRQOy78Sly3A508FYtWqU4bhhnoss9azPXfLG3HJdhFBygoghysP+R3CqvINw7PmTTFAXS4n4a 1/dNOZDbTtEQZk6KHVry2XHdFAqobRCu/1fhPG0Z8w0fTflGflVxZk0+2hhnAascJ5JSIryc48sCh ECKM2BMw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1t0K4z-0000000541j-1ZCD; Mon, 14 Oct 2024 12:21:41 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1t0IpS-00000004oQZ-28Ge for linux-arm-kernel@lists.infradead.org; Mon, 14 Oct 2024 11:01:39 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id C45931758; Mon, 14 Oct 2024 04:02:03 -0700 (PDT) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.27]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id C54893F51B; Mon, 14 Oct 2024 04:01:31 -0700 (PDT) From: Ryan Roberts To: Andrew Morton , Anshuman Khandual , Ard Biesheuvel , Catalin Marinas , David Hildenbrand , Greg Marsden , Ivan Ivanov , Kalesh Singh , Marc Zyngier , Mark Rutland , Matthias Brugger , Miroslav Benes , Will Deacon Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [RFC PATCH v1 41/57] arm64: Pass desired page size on command line Date: Mon, 14 Oct 2024 11:58:48 +0100 Message-ID: <20241014105912.3207374-41-ryan.roberts@arm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20241014105912.3207374-1-ryan.roberts@arm.com> References: <20241014105514.3206191-1-ryan.roberts@arm.com> <20241014105912.3207374-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241014_040134_712310_A7D1D8D9 X-CRM114-Status: GOOD ( 14.72 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Allow user to pass desired page size via command line as either "arm64.pagesize=4k", "arm64.pagesize=16k", or "arm64.pagesize=64k". The specified value is stored in the SW_FEATURE register as an encoded page shift in a 4 bit field. We only allow setting the page size override if the requested size is supported by the HW and is within the compile-time [PAGE_SIZE_MIN, PAGE_SIZE_MAX] range. This second condition means that overrides get ignored when we have a compile-time page size (because PAGE_SIZE_MIN == PAGE_SIZE_MAX). Signed-off-by: Ryan Roberts --- ***NOTE*** Any confused maintainers may want to read the cover note here for context: https://lore.kernel.org/all/20241014105514.3206191-1-ryan.roberts@arm.com/ arch/arm64/include/asm/cpufeature.h | 11 ++++++++ arch/arm64/kernel/pi/idreg-override.c | 36 +++++++++++++++++++++++++++ 2 files changed, 47 insertions(+) diff --git a/arch/arm64/include/asm/cpufeature.h b/arch/arm64/include/asm/cpufeature.h index 5584342672715..4edbb586810d7 100644 --- a/arch/arm64/include/asm/cpufeature.h +++ b/arch/arm64/include/asm/cpufeature.h @@ -18,6 +18,7 @@ #define ARM64_SW_FEATURE_OVERRIDE_NOKASLR 0 #define ARM64_SW_FEATURE_OVERRIDE_HVHE 4 #define ARM64_SW_FEATURE_OVERRIDE_RODATA_OFF 8 +#define ARM64_SW_FEATURE_OVERRIDE_PAGESHIFT 12 #ifndef __ASSEMBLY__ @@ -963,6 +964,16 @@ static inline bool arm64_test_sw_feature_override(int feat) &arm64_sw_feature_override); } +static inline int arm64_pageshift_cmdline(void) +{ + int val; + + val = arm64_apply_feature_override(0, + ARM64_SW_FEATURE_OVERRIDE_PAGESHIFT, + 4, &arm64_sw_feature_override); + return val ? val * 2 + 10 : 0; +} + static inline bool kaslr_disabled_cmdline(void) { return arm64_test_sw_feature_override(ARM64_SW_FEATURE_OVERRIDE_NOKASLR); diff --git a/arch/arm64/kernel/pi/idreg-override.c b/arch/arm64/kernel/pi/idreg-override.c index 29d4b6244a6f6..5a38bdb231bc8 100644 --- a/arch/arm64/kernel/pi/idreg-override.c +++ b/arch/arm64/kernel/pi/idreg-override.c @@ -183,6 +183,38 @@ static bool __init hvhe_filter(u64 val) ID_AA64MMFR1_EL1_VH_SHIFT)); } +static bool __init pageshift_filter(u64 val) +{ + u64 mmfr0 = read_sysreg_s(SYS_ID_AA64MMFR0_EL1); + u32 tgran64 = SYS_FIELD_GET(ID_AA64MMFR0_EL1, TGRAN64, mmfr0); + u32 tgran16 = SYS_FIELD_GET(ID_AA64MMFR0_EL1, TGRAN16, mmfr0); + u32 tgran4 = SYS_FIELD_GET(ID_AA64MMFR0_EL1, TGRAN4, mmfr0); + + /* pageshift is stored compressed in 4 bit field. */ + if (val) + val = val * 2 + 10; + + if (val < PAGE_SHIFT_MIN || val > PAGE_SHIFT_MAX) + return false; + + if (val == ARM64_PAGE_SHIFT_64K && + tgran64 >= ID_AA64MMFR0_EL1_TGRAN64_SUPPORTED_MIN && + tgran64 <= ID_AA64MMFR0_EL1_TGRAN64_SUPPORTED_MAX) + return true; + + if (val == ARM64_PAGE_SHIFT_16K && + tgran16 >= ID_AA64MMFR0_EL1_TGRAN16_SUPPORTED_MIN && + tgran16 <= ID_AA64MMFR0_EL1_TGRAN16_SUPPORTED_MAX) + return true; + + if (val == ARM64_PAGE_SHIFT_4K && + tgran4 >= ID_AA64MMFR0_EL1_TGRAN4_SUPPORTED_MIN && + tgran4 <= ID_AA64MMFR0_EL1_TGRAN4_SUPPORTED_MAX) + return true; + + return false; +} + static const struct ftr_set_desc sw_features __prel64_initconst = { .name = "arm64_sw", .override = &arm64_sw_feature_override, @@ -190,6 +222,7 @@ static const struct ftr_set_desc sw_features __prel64_initconst = { FIELD("nokaslr", ARM64_SW_FEATURE_OVERRIDE_NOKASLR, NULL), FIELD("hvhe", ARM64_SW_FEATURE_OVERRIDE_HVHE, hvhe_filter), FIELD("rodataoff", ARM64_SW_FEATURE_OVERRIDE_RODATA_OFF, NULL), + FIELD("pageshift", ARM64_SW_FEATURE_OVERRIDE_PAGESHIFT, pageshift_filter), {} }, }; @@ -225,6 +258,9 @@ static const struct { { "rodata=off", "arm64_sw.rodataoff=1" }, { "arm64.nolva", "id_aa64mmfr2.varange=0" }, { "arm64.no32bit_el0", "id_aa64pfr0.el0=1" }, + { "arm64.pagesize=4k", "arm64_sw.pageshift=1" }, + { "arm64.pagesize=16k", "arm64_sw.pageshift=2" }, + { "arm64.pagesize=64k", "arm64_sw.pageshift=3" }, }; static int __init parse_hexdigit(const char *p, u64 *v) From patchwork Mon Oct 14 10:58:49 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13834906 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 20297D16252 for ; Mon, 14 Oct 2024 12:24:46 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=MTB1pReV6clp7TUE7p0HuOwj9vCbr05wShDzVX3O4iE=; b=JGjiAp5zkCRsffO0SDYxVz8lD3 n7gSYAGZY/Fa4u1RQpCiw8blvbVIJFO5MJG30qUDNHGK6GMp3BlNCsC47QPlIxVzsTPLQGT1M74yQ P0tnv4LW8AqFftdBhw895+VKXv7u24mux/PNPJH6zFF8Tzr+8zQIsQX98vKQsKb2mtysmq5R22O8L 7D5xY+WmBrigjpFks0qhJIxgXCu0D6D9kHzEVSBhZq1jhptRwriNjtyXc0ilEqKC7KCk8EEqXNWG3 /yh75+8Y5O8LguPArFVuODihT7UubviC6XBhCov+x3fKENqqIqtUT/W9UUSK8rNGerQaTIe818/HU vx+9mMuQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1t0K7j-000000054X4-0GTk; Mon, 14 Oct 2024 12:24:31 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1t0IpV-00000004oRR-0lfc for linux-arm-kernel@lists.infradead.org; Mon, 14 Oct 2024 11:01:44 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 7674D175A; Mon, 14 Oct 2024 04:02:06 -0700 (PDT) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.27]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 760F73F51B; Mon, 14 Oct 2024 04:01:34 -0700 (PDT) From: Ryan Roberts To: Andrew Morton , Anshuman Khandual , Ard Biesheuvel , Catalin Marinas , David Hildenbrand , Greg Marsden , Ivan Ivanov , Kalesh Singh , Marc Zyngier , Mark Rutland , Matthias Brugger , Miroslav Benes , Will Deacon Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [RFC PATCH v1 42/57] arm64: Divorce early init from PAGE_SIZE Date: Mon, 14 Oct 2024 11:58:49 +0100 Message-ID: <20241014105912.3207374-42-ryan.roberts@arm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20241014105912.3207374-1-ryan.roberts@arm.com> References: <20241014105514.3206191-1-ryan.roberts@arm.com> <20241014105912.3207374-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241014_040137_523384_A67EC78F X-CRM114-Status: GOOD ( 33.69 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Refactor all the early code between entrypoint and start_kernel() so that it does not rely on compile-time configuration of page size. For now, that code chooses to use the compile-time page size as its boot-time selected page size, but this will change in the near future to allow the page size to be specified on the command line. An initial page size is selected by probe_init_idmap_page_shift(), which is used for the init_idmap_pg_dir. This selects the largest page size that the HW supports and which is also in the range of page sizes allowed by the compile-time config. For now, the allowed range only covers the compile-time selected page size, but in future boot-time page size builds, the range will be expanded. Once the mmu is enabled, we access the command line in the device tree as before, which allows us to determine the page size requested by the user, filtered by the same allowed compile-time range of page sizes - still just the compile-time selected page size for now. If no acceptable page size was specified on the command line, we fall back to the page size selected in probe_init_idmap_page_shift(). We then do a dance to repaint init_idmap_pg_dir for the final page size and for LPA2, if in use. Finally with that installed, we can continue booting the kernel. For all of this to work, we must replace previous compile-time decisions with run-time decisions. For the most part, we can do this by looking at tcr_el1.tg0 to determine the installed page size. These run-time decisions are not in hot code paths so I don't anticipate any performance regressions as a result and therefore I prefer the simplicity of using the run-time approach even for builds that specify a compile-time page size. Of course to be able to actually change the early page size, the static storage for the page tables need to be sized for the maximum requirement. Currently they are still sized for the compile-time page size. Signed-off-by: Ryan Roberts --- ***NOTE*** Any confused maintainers may want to read the cover note here for context: https://lore.kernel.org/all/20241014105514.3206191-1-ryan.roberts@arm.com/ arch/arm64/include/asm/assembler.h | 50 +++++++- arch/arm64/include/asm/cpufeature.h | 33 +++-- arch/arm64/include/asm/pgtable-prot.h | 2 +- arch/arm64/kernel/head.S | 40 ++++-- arch/arm64/kernel/pi/idreg-override.c | 32 +++-- arch/arm64/kernel/pi/map_kernel.c | 87 +++++++++---- arch/arm64/kernel/pi/map_range.c | 171 ++++++++++++++++++++++---- arch/arm64/kernel/pi/pi.h | 61 ++++++++- arch/arm64/mm/proc.S | 21 ++-- 9 files changed, 405 insertions(+), 92 deletions(-) diff --git a/arch/arm64/include/asm/assembler.h b/arch/arm64/include/asm/assembler.h index bc0b0d75acef7..77c2d707adb1a 100644 --- a/arch/arm64/include/asm/assembler.h +++ b/arch/arm64/include/asm/assembler.h @@ -568,6 +568,14 @@ alternative_endif mrs \rd, sp_el0 .endm +/* + * Retrieve and return tcr_el1.tg0 in \tg0. + */ + .macro get_tg0, tg0 + mrs \tg0, tcr_el1 + and \tg0, \tg0, #TCR_TG0_MASK + .endm + /* * If the kernel is built for 52-bit virtual addressing but the hardware only * supports 48 bits, we cannot program the pgdir address into TTBR1 directly, @@ -584,12 +592,16 @@ alternative_endif * ttbr: Value of ttbr to set, modified. */ .macro offset_ttbr1, ttbr, tmp -#if defined(CONFIG_ARM64_VA_BITS_52) && !defined(CONFIG_ARM64_LPA2) +#if defined(CONFIG_ARM64_VA_BITS_52) + get_tg0 \tmp + cmp \tmp, #TCR_TG0_64K + b.ne .Ldone\@ mrs \tmp, tcr_el1 and \tmp, \tmp, #TCR_T1SZ_MASK cmp \tmp, #TCR_T1SZ(VA_BITS_MIN) orr \tmp, \ttbr, #TTBR1_BADDR_4852_OFFSET csel \ttbr, \tmp, \ttbr, eq +.Ldone\@: #endif .endm @@ -863,4 +875,40 @@ alternative_cb ARM64_ALWAYS_SYSTEM, spectre_bhb_patch_clearbhb alternative_cb_end #endif /* CONFIG_MITIGATE_SPECTRE_BRANCH_HISTORY */ .endm + +/* + * Given \tg0, populates \val with one of the 3 passed in values, corresponding + * to the page size advertised by \tg0. + */ + .macro value_for_page_size, val, tg0, val4k, val16k, val64k +.Lsz_64k\@: + cmp \tg0, #TCR_TG0_64K + b.ne .Lsz_16k\@ + mov \val, #\val64k + b .Ldone\@ +.Lsz_16k\@: + cmp \tg0, #TCR_TG0_16K + b.ne .Lsz_4k\@ + mov \val, #\val16k + b .Ldone\@ +.Lsz_4k\@: + mov \val, #\val4k +.Ldone\@: + .endm + + .macro tgran_shift, val, tg0 + value_for_page_size \val, \tg0, ID_AA64MMFR0_EL1_TGRAN4_SHIFT, ID_AA64MMFR0_EL1_TGRAN16_SHIFT, ID_AA64MMFR0_EL1_TGRAN64_SHIFT + .endm + + .macro tgran_min, val, tg0 + value_for_page_size \val, \tg0, ID_AA64MMFR0_EL1_TGRAN4_SUPPORTED_MIN, ID_AA64MMFR0_EL1_TGRAN16_SUPPORTED_MIN, ID_AA64MMFR0_EL1_TGRAN64_SUPPORTED_MIN + .endm + + .macro tgran_max, val, tg0 + value_for_page_size \val, \tg0, ID_AA64MMFR0_EL1_TGRAN4_SUPPORTED_MAX, ID_AA64MMFR0_EL1_TGRAN16_SUPPORTED_MAX, ID_AA64MMFR0_EL1_TGRAN64_SUPPORTED_MAX + .endm + + .macro tgran_lpa2, val, tg0 + value_for_page_size \val, \tg0, ID_AA64MMFR0_EL1_TGRAN4_52_BIT, ID_AA64MMFR0_EL1_TGRAN16_52_BIT, -1 + .endm #endif /* __ASM_ASSEMBLER_H */ diff --git a/arch/arm64/include/asm/cpufeature.h b/arch/arm64/include/asm/cpufeature.h index 4edbb586810d7..2c22cfdc04bc7 100644 --- a/arch/arm64/include/asm/cpufeature.h +++ b/arch/arm64/include/asm/cpufeature.h @@ -26,6 +26,7 @@ #include #include #include +#include /* * CPU feature register tracking @@ -1014,10 +1015,13 @@ static inline bool cpu_has_pac(void) &id_aa64isar2_override); } -static inline bool cpu_has_lva(void) +static inline bool cpu_has_lva(u64 page_size) { u64 mmfr2; + if (page_size != SZ_64K) + return false; + mmfr2 = read_sysreg_s(SYS_ID_AA64MMFR2_EL1); mmfr2 &= ~id_aa64mmfr2_override.mask; mmfr2 |= id_aa64mmfr2_override.val; @@ -1025,22 +1029,31 @@ static inline bool cpu_has_lva(void) ID_AA64MMFR2_EL1_VARange_SHIFT); } -static inline bool cpu_has_lpa2(void) +static inline bool cpu_has_lpa2(u64 page_size) { -#ifdef CONFIG_ARM64_LPA2 u64 mmfr0; int feat; + int shift; + int minval; + + switch (page_size) { + case SZ_4K: + shift = ID_AA64MMFR0_EL1_TGRAN4_SHIFT; + minval = ID_AA64MMFR0_EL1_TGRAN4_52_BIT; + break; + case SZ_16K: + shift = ID_AA64MMFR0_EL1_TGRAN16_SHIFT; + minval = ID_AA64MMFR0_EL1_TGRAN16_52_BIT; + break; + default: + return false; + } mmfr0 = read_sysreg(id_aa64mmfr0_el1); mmfr0 &= ~id_aa64mmfr0_override.mask; mmfr0 |= id_aa64mmfr0_override.val; - feat = cpuid_feature_extract_signed_field(mmfr0, - ID_AA64MMFR0_EL1_TGRAN_SHIFT); - - return feat >= ID_AA64MMFR0_EL1_TGRAN_LPA2; -#else - return false; -#endif + feat = cpuid_feature_extract_signed_field(mmfr0, shift); + return feat >= minval; } #endif /* __ASSEMBLY__ */ diff --git a/arch/arm64/include/asm/pgtable-prot.h b/arch/arm64/include/asm/pgtable-prot.h index b11cfb9fdd379..f8ebf424ca016 100644 --- a/arch/arm64/include/asm/pgtable-prot.h +++ b/arch/arm64/include/asm/pgtable-prot.h @@ -74,7 +74,7 @@ extern bool arm64_use_ng_mappings; #define PTE_MAYBE_NG (arm64_use_ng_mappings ? PTE_NG : 0) #define PMD_MAYBE_NG (arm64_use_ng_mappings ? PMD_SECT_NG : 0) -#ifndef CONFIG_ARM64_LPA2 +#ifndef CONFIG_ARM64_VA_BITS_52 #define lpa2_is_enabled() false #define PTE_MAYBE_SHARED PTE_SHARED #define PMD_MAYBE_SHARED PMD_SECT_S diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S index 7e17a71fd9e4b..761b7f5633e15 100644 --- a/arch/arm64/kernel/head.S +++ b/arch/arm64/kernel/head.S @@ -88,6 +88,8 @@ SYM_CODE_START(primary_entry) adrp x1, early_init_stack mov sp, x1 mov x29, xzr + bl __pi_probe_init_idmap_page_shift + mov x3, x0 adrp x0, init_idmap_pg_dir adrp x1, init_idmap_pg_end mov x2, xzr @@ -471,11 +473,16 @@ SYM_FUNC_END(set_cpu_boot_mode_flag) */ .section ".idmap.text","a" SYM_FUNC_START(__enable_mmu) + get_tg0 x3 + tgran_shift x4, x3 + tgran_min x5, x3 + tgran_max x6, x3 mrs x3, ID_AA64MMFR0_EL1 - ubfx x3, x3, #ID_AA64MMFR0_EL1_TGRAN_SHIFT, 4 - cmp x3, #ID_AA64MMFR0_EL1_TGRAN_SUPPORTED_MIN + lsr x3, x3, x4 + ubfx x3, x3, #0, 4 + cmp x3, x5 b.lt __no_granule_support - cmp x3, #ID_AA64MMFR0_EL1_TGRAN_SUPPORTED_MAX + cmp x3, x6 b.gt __no_granule_support phys_to_ttbr x2, x2 msr ttbr0_el1, x2 // load TTBR0 @@ -488,17 +495,32 @@ SYM_FUNC_END(__enable_mmu) #ifdef CONFIG_ARM64_VA_BITS_52 SYM_FUNC_START(__cpu_secondary_check52bitva) -#ifndef CONFIG_ARM64_LPA2 + /* + * tcr_el1 is not yet loaded (there is a chicken-and-egg problem) so we + * can't figure out LPA2 vs LVA from that. But this is only called for + * secondary cpus so tcr_boot_fields has already been populated by the + * primary cpu. So grab that and rely on the DS bit to tell us if we are + * LPA2 or LVA. + */ + adr_l x1, __pi_tcr_boot_fields + ldr x0, [x1] + and x1, x0, #TCR_DS + cbnz x1, .Llpa2 +.Llva: mrs_s x0, SYS_ID_AA64MMFR2_EL1 and x0, x0, ID_AA64MMFR2_EL1_VARange_MASK cbnz x0, 2f -#else + b .Lfail +.Llpa2: + and x0, x0, #TCR_TG0_MASK + tgran_shift x1, x0 + tgran_lpa2 x2, x0 mrs x0, id_aa64mmfr0_el1 - sbfx x0, x0, #ID_AA64MMFR0_EL1_TGRAN_SHIFT, 4 - cmp x0, #ID_AA64MMFR0_EL1_TGRAN_LPA2 + lsr x0, x0, x1 + sbfx x0, x0, #0, 4 + cmp x0, x2 b.ge 2f -#endif - +.Lfail: update_early_cpu_boot_status \ CPU_STUCK_IN_KERNEL | CPU_STUCK_REASON_52_BIT_VA, x0, x1 1: wfe diff --git a/arch/arm64/kernel/pi/idreg-override.c b/arch/arm64/kernel/pi/idreg-override.c index 5a38bdb231bc8..0685c0a3255e2 100644 --- a/arch/arm64/kernel/pi/idreg-override.c +++ b/arch/arm64/kernel/pi/idreg-override.c @@ -62,20 +62,34 @@ static const struct ftr_set_desc mmfr1 __prel64_initconst = { static bool __init mmfr2_varange_filter(u64 val) { - int __maybe_unused feat; + u64 mmfr0; + int feat; + int shift; + int minval; if (val) return false; -#ifdef CONFIG_ARM64_LPA2 - feat = cpuid_feature_extract_signed_field(read_sysreg(id_aa64mmfr0_el1), - ID_AA64MMFR0_EL1_TGRAN_SHIFT); - if (feat >= ID_AA64MMFR0_EL1_TGRAN_LPA2) { - id_aa64mmfr0_override.val |= - (ID_AA64MMFR0_EL1_TGRAN_LPA2 - 1) << ID_AA64MMFR0_EL1_TGRAN_SHIFT; - id_aa64mmfr0_override.mask |= 0xfU << ID_AA64MMFR0_EL1_TGRAN_SHIFT; + mmfr0 = read_sysreg(id_aa64mmfr0_el1); + + /* Remove LPA2 support for 4K granule. */ + shift = ID_AA64MMFR0_EL1_TGRAN4_SHIFT; + minval = ID_AA64MMFR0_EL1_TGRAN4_52_BIT; + feat = cpuid_feature_extract_signed_field(mmfr0, shift); + if (feat >= minval) { + id_aa64mmfr0_override.val |= (minval - 1) << shift; + id_aa64mmfr0_override.mask |= 0xfU << shift; + } + + /* Remove LPA2 support for 16K granule. */ + shift = ID_AA64MMFR0_EL1_TGRAN16_SHIFT; + minval = ID_AA64MMFR0_EL1_TGRAN16_52_BIT; + feat = cpuid_feature_extract_signed_field(mmfr0, shift); + if (feat >= minval) { + id_aa64mmfr0_override.val |= (minval - 1) << shift; + id_aa64mmfr0_override.mask |= 0xfU << shift; } -#endif + return true; } diff --git a/arch/arm64/kernel/pi/map_kernel.c b/arch/arm64/kernel/pi/map_kernel.c index a53fc225d2d0d..7a62d4238449d 100644 --- a/arch/arm64/kernel/pi/map_kernel.c +++ b/arch/arm64/kernel/pi/map_kernel.c @@ -133,10 +133,18 @@ static void __init map_kernel(u64 kaslr_offset, u64 va_offset, int root_level) idmap_cpu_replace_ttbr1(swapper_pg_dir); } -static void noinline __section(".idmap.text") set_ttbr0_for_lpa2(u64 ttbr) +static void noinline __section(".idmap.text") set_ttbr0(u64 ttbr, bool use_lpa2, + int page_shift) { u64 sctlr = read_sysreg(sctlr_el1); - u64 tcr = read_sysreg(tcr_el1) | TCR_DS; + u64 tcr = read_sysreg(tcr_el1); + u64 boot_fields = early_page_shift_to_tcr_tgx(page_shift); + + if (use_lpa2) + boot_fields |= TCR_DS; + + tcr &= ~(TCR_TG0_MASK | TCR_TG1_MASK | TCR_DS); + tcr |= boot_fields; asm(" msr sctlr_el1, %0 ;" " isb ;" @@ -149,57 +157,66 @@ static void noinline __section(".idmap.text") set_ttbr0_for_lpa2(u64 ttbr) " msr sctlr_el1, %3 ;" " isb ;" :: "r"(sctlr & ~SCTLR_ELx_M), "r"(ttbr), "r"(tcr), "r"(sctlr)); + + /* Stash this for __cpu_setup to configure secondary cpus' tcr_el1. */ + set_tcr_boot_fields(boot_fields); } -static void __init remap_idmap_for_lpa2(void) +static void __init remap_idmap(bool use_lpa2, int page_shift) { /* clear the bits that change meaning once LPA2 is turned on */ - pteval_t mask = PTE_SHARED; + pteval_t mask = use_lpa2 ? PTE_SHARED : 0; /* - * We have to clear bits [9:8] in all block or page descriptors in the - * initial ID map, as otherwise they will be (mis)interpreted as + * For LPA2, We have to clear bits [9:8] in all block or page descriptors + * in the initial ID map, as otherwise they will be (mis)interpreted as * physical address bits once we flick the LPA2 switch (TCR.DS). Since * we cannot manipulate live descriptors in that way without creating * potential TLB conflicts, let's create another temporary ID map in a * LPA2 compatible fashion, and update the initial ID map while running * from that. */ - create_init_idmap(init_pg_dir, init_pg_end, mask); + create_init_idmap(init_pg_dir, init_pg_end, mask, page_shift); dsb(ishst); - set_ttbr0_for_lpa2((u64)init_pg_dir); + set_ttbr0((u64)init_pg_dir, use_lpa2, page_shift); /* - * Recreate the initial ID map with the same granularity as before. - * Don't bother with the FDT, we no longer need it after this. + * Recreate the initial ID map with new page size and, if LPA2 is in + * use, bits [9:8] cleared. */ memset(init_idmap_pg_dir, 0, (u64)init_idmap_pg_end - (u64)init_idmap_pg_dir); - create_init_idmap(init_idmap_pg_dir, init_idmap_pg_end, mask); + create_init_idmap(init_idmap_pg_dir, init_idmap_pg_end, mask, page_shift); dsb(ishst); /* switch back to the updated initial ID map */ - set_ttbr0_for_lpa2((u64)init_idmap_pg_dir); + set_ttbr0((u64)init_idmap_pg_dir, use_lpa2, page_shift); /* wipe the temporary ID map from memory */ memset(init_pg_dir, 0, (u64)init_pg_end - (u64)init_pg_dir); } -static void __init map_fdt(u64 fdt) +static void __init map_fdt(u64 fdt, int page_shift) { static u8 ptes[INIT_IDMAP_FDT_SIZE_MAX] __initdata __aligned(PAGE_SIZE); + static bool first_time __initdata = true; u64 limit = (u64)&ptes[INIT_IDMAP_FDT_SIZE_MAX]; u64 efdt = fdt + MAX_FDT_SIZE; u64 ptep = (u64)ptes; + if (!first_time) { + memset(ptes, 0, sizeof(ptes)); + first_time = false; + } + /* * Map up to MAX_FDT_SIZE bytes, but avoid overlap with * the kernel image. */ map_range(&ptep, limit, fdt, (u64)_text > fdt ? min((u64)_text, efdt) : efdt, - fdt, PAGE_KERNEL, IDMAP_ROOT_LEVEL(PAGE_SHIFT), + fdt, PAGE_KERNEL, IDMAP_ROOT_LEVEL(page_shift), (pte_t *)init_idmap_pg_dir, false, 0); dsb(ishst); } @@ -207,13 +224,16 @@ static void __init map_fdt(u64 fdt) asmlinkage void __init early_map_kernel(u64 boot_status, void *fdt) { static char const chosen_str[] __initconst = "/chosen"; + u64 early_page_shift = early_tcr_tg0_to_page_shift(); u64 va_base, pa_base = (u64)&_text; u64 kaslr_offset = pa_base % MIN_KIMG_ALIGN; - int root_level = 4 - CONFIG_PGTABLE_LEVELS; int va_bits = VA_BITS; + bool use_lpa2 = false; + int root_level; + int page_shift; int chosen; - map_fdt((u64)fdt); + map_fdt((u64)fdt, early_page_shift); /* Clear BSS and the initial page tables */ memset(__bss_start, 0, (u64)init_pg_end - (u64)__bss_start); @@ -222,16 +242,37 @@ asmlinkage void __init early_map_kernel(u64 boot_status, void *fdt) chosen = fdt_path_offset(fdt, chosen_str); init_feature_override(boot_status, fdt, chosen); - if (IS_ENABLED(CONFIG_ARM64_64K_PAGES) && !cpu_has_lva()) { - va_bits = VA_BITS_MIN; - } else if (IS_ENABLED(CONFIG_ARM64_LPA2) && !cpu_has_lpa2()) { - va_bits = VA_BITS_MIN; - root_level++; + /* Get page_shift from cmdline, falling back to early_page_shift. */ + page_shift = arm64_pageshift_cmdline(); + if (!page_shift) + page_shift = early_page_shift; + + if (va_bits > 48) { + u64 page_size = early_page_size(page_shift); + + if (page_size == SZ_64K) { + if (!cpu_has_lva(page_size)) + va_bits = VA_BITS_MIN; + } else { + use_lpa2 = cpu_has_lpa2(page_size); + if (!use_lpa2) + va_bits = VA_BITS_MIN; + } } if (va_bits > VA_BITS_MIN) sysreg_clear_set(tcr_el1, TCR_T1SZ_MASK, TCR_T1SZ(va_bits)); + /* + * This will update tg0/tg1 in tcr for the final page size. After this, + * PAGE_SIZE and friends can be used safely. kaslr_early_init(), below, + * is the first such user. + */ + if (use_lpa2 || page_shift != early_page_shift) { + remap_idmap(use_lpa2, page_shift); + map_fdt((u64)fdt, page_shift); + } + /* * The virtual KASLR displacement modulo 2MiB is decided by the * physical placement of the image, as otherwise, we might not be able @@ -248,9 +289,7 @@ asmlinkage void __init early_map_kernel(u64 boot_status, void *fdt) kaslr_offset |= kaslr_seed & ~(MIN_KIMG_ALIGN - 1); } - if (IS_ENABLED(CONFIG_ARM64_LPA2) && va_bits > VA_BITS_MIN) - remap_idmap_for_lpa2(); - va_base = KIMAGE_VADDR + kaslr_offset; + root_level = 4 - PGTABLE_LEVELS(page_shift, va_bits); map_kernel(kaslr_offset, va_base - pa_base, root_level); } diff --git a/arch/arm64/kernel/pi/map_range.c b/arch/arm64/kernel/pi/map_range.c index b62d2e3135f81..be5470a969a47 100644 --- a/arch/arm64/kernel/pi/map_range.c +++ b/arch/arm64/kernel/pi/map_range.c @@ -11,6 +11,34 @@ #include "pi.h" +static inline u64 __init pte_get_oa(pte_t pte, int page_shift, bool oa52bit) +{ + pteval_t pv = pte_val(pte); + +#ifdef CONFIG_ARM64_PA_BITS_52 + if (oa52bit) { + if (early_page_size(page_shift) == SZ_64K) + return (pv & GENMASK(47, 16)) | + ((pv & GENMASK(15, 12)) << 36); + return (pv & GENMASK(49, 12)) | ((pv & GENMASK(9, 8)) << 42); + } +#endif + return pv & GENMASK(47, 12); +} + +static inline u64 __init pte_prep_oa(u64 oa, int page_shift, bool oa52bit) +{ +#ifdef CONFIG_ARM64_PA_BITS_52 + if (oa52bit) { + if (early_page_size(page_shift) == SZ_64K) + return (oa & GENMASK(47, 16)) | + ((oa >> 36) & GENMASK(15, 12)); + return (oa & GENMASK(49, 12)) | ((oa >> 42) & GENMASK(9, 8)); + } +#endif + return oa; +} + static void __init mmuoff_data_clean(void) { bool cache_ena = !!(read_sysreg(sctlr_el1) & SCTLR_ELx_C); @@ -35,8 +63,19 @@ static void __init report_cpu_stuck(long val) cpu_park_loop(); } +u64 __section(".mmuoff.data.read") tcr_boot_fields; + +void __init set_tcr_boot_fields(u64 val) +{ + WRITE_ONCE(tcr_boot_fields, val); + + /* Ensure the visibility of the new value */ + dsb(ishst); + mmuoff_data_clean(); +} + /** - * map_range - Map a contiguous range of physical pages into virtual memory + * __map_range - Map a contiguous range of physical pages into virtual memory * * @pte: Address of physical pointer to array of pages to * allocate page tables from @@ -50,21 +89,28 @@ static void __init report_cpu_stuck(long val) * @may_use_cont: Whether the use of the contiguous attribute is allowed * @va_offset: Offset between a physical page and its current mapping * in the VA space + * @page_shift: Page size (as a shift) to create page table for + * @oa52bit: Whether to store output addresses in 52-bit format */ -void __init map_range(u64 *pte, u64 limit, u64 start, u64 end, u64 pa, - pgprot_t prot, int level, pte_t *tbl, bool may_use_cont, - u64 va_offset) +static void __init __map_range(u64 *pte, u64 limit, u64 start, u64 end, u64 pa, + pgprot_t prot, int level, pte_t *tbl, + bool may_use_cont, u64 va_offset, int page_shift, + bool oa52bit) { - u64 cmask = (level == 3) ? CONT_PTE_SIZE - 1 : U64_MAX; + const u64 page_size = early_page_size(page_shift); + const u64 page_mask = early_page_mask(page_shift); + const u64 cont_pte_size = early_cont_pte_size(page_shift); + const u64 ptrs_per_pte = early_ptrs_per_pte(page_shift); + u64 cmask = (level == 3) ? cont_pte_size - 1 : U64_MAX; u64 protval = pgprot_val(prot) & ~PTE_TYPE_MASK; - int lshift = (3 - level) * (PAGE_SHIFT - 3); - u64 lmask = (PAGE_SIZE << lshift) - 1; + int lshift = (3 - level) * (page_shift - 3); + u64 lmask = (page_size << lshift) - 1; - start &= PAGE_MASK; - pa &= PAGE_MASK; + start &= page_mask; + pa &= page_mask; /* Advance tbl to the entry that covers start */ - tbl += (start >> (lshift + PAGE_SHIFT)) % PTRS_PER_PTE; + tbl += (start >> (lshift + page_shift)) % ptrs_per_pte; /* * Set the right block/page bits for this level unless we are @@ -74,7 +120,7 @@ void __init map_range(u64 *pte, u64 limit, u64 start, u64 end, u64 pa, protval |= (level < 3) ? PMD_TYPE_SECT : PTE_TYPE_PAGE; while (start < end) { - u64 next = min((start | lmask) + 1, PAGE_ALIGN(end)); + u64 next = min((start | lmask) + 1, ALIGN(end, page_size)); if (level < 3 && (start | next | pa) & lmask) { /* @@ -82,20 +128,20 @@ void __init map_range(u64 *pte, u64 limit, u64 start, u64 end, u64 pa, * table mapping if necessary and recurse. */ if (pte_none(*tbl)) { - u64 size = PTRS_PER_PTE * sizeof(pte_t); + u64 size = ptrs_per_pte * sizeof(pte_t); if (*pte + size > limit) { report_cpu_stuck( CPU_STUCK_REASON_NO_PGTABLE_MEM); } - *tbl = __pte(__phys_to_pte_val(*pte) | + *tbl = __pte(pte_prep_oa(*pte, page_shift, oa52bit) | PMD_TYPE_TABLE | PMD_TABLE_UXN); *pte += size; } - map_range(pte, limit, start, next, pa, prot, level + 1, - (pte_t *)(__pte_to_phys(*tbl) + va_offset), - may_use_cont, va_offset); + __map_range(pte, limit, start, next, pa, prot, level + 1, + (pte_t *)(pte_get_oa(*tbl, page_shift, oa52bit) + va_offset), + may_use_cont, va_offset, page_shift, oa52bit); } else { /* * Start a contiguous range if start and pa are @@ -112,7 +158,8 @@ void __init map_range(u64 *pte, u64 limit, u64 start, u64 end, u64 pa, protval &= ~PTE_CONT; /* Put down a block or page mapping */ - *tbl = __pte(__phys_to_pte_val(pa) | protval); + *tbl = __pte(pte_prep_oa(pa, page_shift, oa52bit) | + protval); } pa += next - start; start = next; @@ -120,22 +167,96 @@ void __init map_range(u64 *pte, u64 limit, u64 start, u64 end, u64 pa, } } +/** + * map_range - Map a contiguous range of physical pages into virtual memory + * + * As per __map_range(), except it uses the page_shift and oa52bit of the + * currently tcr-installed granule size instead of passing explicitly. + */ +void __init map_range(u64 *pte, u64 limit, u64 start, u64 end, u64 pa, + pgprot_t prot, int level, pte_t *tbl, bool may_use_cont, + u64 va_offset) +{ + int page_shift = early_tcr_tg0_to_page_shift(); + bool oa52bit = false; + +#ifdef CONFIG_ARM64_PA_BITS_52 + /* + * We can safely assume 52bit for 64K pages because if it turns out to + * be 48bit, its still safe to treat [51:48] as address bits because + * they are 0. + */ + if (early_page_size(page_shift) == SZ_64K) + oa52bit = true; + /* + * For 4K and 16K, on the other hand, those bits are used for something + * else when LPA2 is not explicitly enabled. Deliberately not using + * read_tcr() since it is marked pure, and at this point, the tcr is not + * yet stable. + */ + else if (read_sysreg(tcr_el1) & TCR_DS) + oa52bit = true; +#endif + + __map_range(pte, limit, start, end, pa, prot, level, tbl, may_use_cont, + va_offset, page_shift, oa52bit); +} + +asmlinkage u64 __init probe_init_idmap_page_shift(void) +{ + u64 mmfr0 = read_sysreg_s(SYS_ID_AA64MMFR0_EL1); + u32 tgran64 = SYS_FIELD_GET(ID_AA64MMFR0_EL1, TGRAN64, mmfr0); + u32 tgran16 = SYS_FIELD_GET(ID_AA64MMFR0_EL1, TGRAN16, mmfr0); + u32 tgran4 = SYS_FIELD_GET(ID_AA64MMFR0_EL1, TGRAN4, mmfr0); + u64 page_shift; + + /* + * Select the largest page size supported by the HW, which is also + * allowed by the compilation config. + */ + if (ARM64_PAGE_SHIFT_64K >= PAGE_SHIFT_MIN && + ARM64_PAGE_SHIFT_64K <= PAGE_SHIFT_MAX && + tgran64 >= ID_AA64MMFR0_EL1_TGRAN64_SUPPORTED_MIN && + tgran64 <= ID_AA64MMFR0_EL1_TGRAN64_SUPPORTED_MAX) + page_shift = ARM64_PAGE_SHIFT_64K; + else if (ARM64_PAGE_SHIFT_16K >= PAGE_SHIFT_MIN && + ARM64_PAGE_SHIFT_16K <= PAGE_SHIFT_MAX && + tgran16 >= ID_AA64MMFR0_EL1_TGRAN16_SUPPORTED_MIN && + tgran16 <= ID_AA64MMFR0_EL1_TGRAN16_SUPPORTED_MAX) + page_shift = ARM64_PAGE_SHIFT_16K; + else if (ARM64_PAGE_SHIFT_4K >= PAGE_SHIFT_MIN && + ARM64_PAGE_SHIFT_4K <= PAGE_SHIFT_MAX && + tgran4 >= ID_AA64MMFR0_EL1_TGRAN4_SUPPORTED_MIN && + tgran4 <= ID_AA64MMFR0_EL1_TGRAN4_SUPPORTED_MAX) + page_shift = ARM64_PAGE_SHIFT_4K; + else + report_cpu_stuck(CPU_STUCK_REASON_NO_GRAN); + + /* Stash this for __cpu_setup to configure primary cpu's tcr_el1. */ + set_tcr_boot_fields(early_page_shift_to_tcr_tgx(page_shift)); + + return page_shift; +} + asmlinkage u64 __init create_init_idmap(pgd_t *pg_dir, pgd_t *pg_end, - pteval_t clrmask) + pteval_t clrmask, int page_shift) { - u64 ptep = (u64)pg_dir + PAGE_SIZE; + const u64 page_size = early_page_size(page_shift); + u64 ptep = (u64)pg_dir + page_size; pgprot_t text_prot = PAGE_KERNEL_ROX; pgprot_t data_prot = PAGE_KERNEL; pgprot_val(text_prot) &= ~clrmask; pgprot_val(data_prot) &= ~clrmask; - map_range(&ptep, (u64)pg_end, (u64)_stext, (u64)__initdata_begin, - (u64)_stext, text_prot, - IDMAP_ROOT_LEVEL(PAGE_SHIFT), (pte_t *)pg_dir, false, 0); - map_range(&ptep, (u64)pg_end, (u64)__initdata_begin, (u64)_end, - (u64)__initdata_begin, data_prot, - IDMAP_ROOT_LEVEL(PAGE_SHIFT), (pte_t *)pg_dir, false, 0); + __map_range(&ptep, (u64)pg_end, (u64)_stext, (u64)__initdata_begin, + (u64)_stext, text_prot, + IDMAP_ROOT_LEVEL(page_shift), (pte_t *)pg_dir, false, 0, + page_shift, false); + __map_range(&ptep, (u64)pg_end, (u64)__initdata_begin, (u64)_end, + (u64)__initdata_begin, data_prot, + IDMAP_ROOT_LEVEL(page_shift), (pte_t *)pg_dir, false, 0, + page_shift, false); return ptep; } diff --git a/arch/arm64/kernel/pi/pi.h b/arch/arm64/kernel/pi/pi.h index 20fe0941cb8ee..15c14d0aa6c63 100644 --- a/arch/arm64/kernel/pi/pi.h +++ b/arch/arm64/kernel/pi/pi.h @@ -23,6 +23,62 @@ extern bool dynamic_scs_is_enabled; extern pgd_t init_idmap_pg_dir[], init_idmap_pg_end[]; +static inline u64 early_page_size(int page_shift) +{ + return 1UL << page_shift; +} + +static inline u64 early_page_mask(int page_shift) +{ + return ~(early_page_size(page_shift) - 1); +} + +static inline u64 early_cont_pte_size(int page_shift) +{ + switch (page_shift) { + case 16: /* 64K */ + case 14: /* 16K */ + return SZ_2M; + default: /* 12 4K */ + return SZ_64K; + } +} + +static inline u64 early_ptrs_per_pte(int page_shift) +{ + return 1UL << (page_shift - 3); +} + +static inline int early_tcr_tg0_to_page_shift(void) +{ + /* + * Deliberately not using read_tcr() since it is marked pure, and at + * this point, the tcr is not yet stable. + */ + u64 tg0 = read_sysreg(tcr_el1) & TCR_TG0_MASK; + + switch (tg0) { + case TCR_TG0_64K: + return 16; + case TCR_TG0_16K: + return 14; + default: /* TCR_TG0_4K */ + return 12; + } +} + +static inline u64 early_page_shift_to_tcr_tgx(int page_shift) +{ + switch (early_page_size(page_shift)) { + case SZ_64K: + return TCR_TG0_64K | TCR_TG1_64K; + case SZ_16K: + return TCR_TG0_16K | TCR_TG1_16K; + default: + return TCR_TG0_4K | TCR_TG1_4K; + } +} + void init_feature_override(u64 boot_status, const void *fdt, int chosen); u64 kaslr_early_init(void *fdt, int chosen); void relocate_kernel(u64 offset); @@ -33,4 +89,7 @@ void map_range(u64 *pgd, u64 limit, u64 start, u64 end, u64 pa, pgprot_t prot, asmlinkage void early_map_kernel(u64 boot_status, void *fdt); -asmlinkage u64 create_init_idmap(pgd_t *pgd, pgd_t *pg_end, pteval_t clrmask); +void set_tcr_boot_fields(u64 val); +asmlinkage u64 probe_init_idmap_page_shift(void); +asmlinkage u64 create_init_idmap(pgd_t *pgd, pgd_t *pg_end, pteval_t clrmask, + int page_shift); diff --git a/arch/arm64/mm/proc.S b/arch/arm64/mm/proc.S index f4bc6c5bac062..ab5aa84923524 100644 --- a/arch/arm64/mm/proc.S +++ b/arch/arm64/mm/proc.S @@ -22,14 +22,6 @@ #include #include -#ifdef CONFIG_ARM64_64K_PAGES -#define TCR_TG_FLAGS TCR_TG0_64K | TCR_TG1_64K -#elif defined(CONFIG_ARM64_16K_PAGES) -#define TCR_TG_FLAGS TCR_TG0_16K | TCR_TG1_16K -#else /* CONFIG_ARM64_4K_PAGES */ -#define TCR_TG_FLAGS TCR_TG0_4K | TCR_TG1_4K -#endif - #ifdef CONFIG_RANDOMIZE_BASE #define TCR_KASLR_FLAGS TCR_NFD1 #else @@ -469,18 +461,23 @@ SYM_FUNC_START(__cpu_setup) tcr .req x16 mov_q mair, MAIR_EL1_SET mov_q tcr, TCR_T0SZ(IDMAP_VA_BITS) | TCR_T1SZ(VA_BITS_MIN) | TCR_CACHE_FLAGS | \ - TCR_SMP_FLAGS | TCR_TG_FLAGS | TCR_KASLR_FLAGS | TCR_ASID16 | \ + TCR_SMP_FLAGS | TCR_KASLR_FLAGS | TCR_ASID16 | \ TCR_TBI0 | TCR_A1 | TCR_KASAN_SW_FLAGS | TCR_MTE_FLAGS + /* + * Insert the boot-time determined fields (TG0, TG1 and DS), which are + * cached in the tcr_boot_fields variable. + */ + adr_l x2, __pi_tcr_boot_fields + ldr x3, [x2] + orr tcr, tcr, x3 + tcr_clear_errata_bits tcr, x9, x5 #ifdef CONFIG_ARM64_VA_BITS_52 mov x9, #64 - VA_BITS alternative_if ARM64_HAS_VA52 tcr_set_t1sz tcr, x9 -#ifdef CONFIG_ARM64_LPA2 - orr tcr, tcr, #TCR_DS -#endif alternative_else_nop_endif #endif From patchwork Mon Oct 14 10:58:50 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13834905 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 61E7ED16252 for ; Mon, 14 Oct 2024 12:23:19 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=r9v3S3hMjuiCiaBUIv6jmo6gDiOvZ9WOws2YRMMwzI8=; b=XJ4U6oPiiIjOC0fywyfb5q0oWX YOd0RvebNjBiZucPNiuKqzFR3YLXFlwM5K1DDLorsZk45QehBCG2CCIH1j5pxinpbQ10QCpV4ry6U BcTpAuwC9P9QfVZt5AH45mRAhvvY2LWlIrDMofvrz0IeFSxIl1rbyLhHIaE3RRy7AAXrIim3RWP9S iod9X+HOHDIv2gNvLtmX6wdezTMoR0CtQN8nI+wHQK9esLZTXD9Lf1peYSBpU+48bCw9Sdm96hDr9 ywlK8j060At95qyFtXOeb4/C9AdByaHwmwI4A91iunXu6Ne9coHlO8vqotXurkyltOqIZboFE8NSj IDb66jbQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1t0K6M-000000054Im-0eeY; Mon, 14 Oct 2024 12:23:06 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1t0IpY-00000004oTX-2pAB for linux-arm-kernel@lists.infradead.org; Mon, 14 Oct 2024 11:01:43 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id C1414175D; Mon, 14 Oct 2024 04:02:09 -0700 (PDT) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.27]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 2811C3F51B; Mon, 14 Oct 2024 04:01:37 -0700 (PDT) From: Ryan Roberts To: Andrew Morton , Andrey Ryabinin , Anshuman Khandual , Ard Biesheuvel , Catalin Marinas , David Hildenbrand , Greg Marsden , Ivan Ivanov , Kalesh Singh , Marc Zyngier , Mark Rutland , Matthias Brugger , Miroslav Benes , Oliver Upton , Thomas Gleixner , Will Deacon Cc: Ryan Roberts , kasan-dev@googlegroups.com, kvmarm@lists.linux.dev, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [RFC PATCH v1 43/57] arm64: Clean up simple cases of CONFIG_ARM64_*K_PAGES Date: Mon, 14 Oct 2024 11:58:50 +0100 Message-ID: <20241014105912.3207374-43-ryan.roberts@arm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20241014105912.3207374-1-ryan.roberts@arm.com> References: <20241014105514.3206191-1-ryan.roberts@arm.com> <20241014105912.3207374-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241014_040140_983517_D78D41E4 X-CRM114-Status: GOOD ( 24.21 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org There are a number of places that define macros conditionally depending on which of the CONFIG_ARM64_*K_PAGES macros are defined. But in preparation for supporting boot-time page size selection, we will no longer be able to make these decisions at compile time. So let's refactor the code to check the size of PAGE_SIZE using the ternary operator. This approach will still resolve to compile-time constants when configured for a compile-time page size, but it will also work when we turn PAGE_SIZE into a run-time value. Additionally, IS_ENABLED(CONFIG_ARM64_*K_PAGES) instances are also converted to test the size of PAGE_SIZE. Additionally modify ARM64_HAS_VA52 capability detection to use a custom match function, which chooses which feature register and field to check based on PAGE_SIZE. The compiler will eliminate the other page sizes when selecting a compile time page size, but will also now cope with seting page size at boot time. Signed-off-by: Ryan Roberts --- ***NOTE*** Any confused maintainers may want to read the cover note here for context: https://lore.kernel.org/all/20241014105514.3206191-1-ryan.roberts@arm.com/ arch/arm64/include/asm/kvm_arm.h | 21 ++++------- arch/arm64/include/asm/kvm_pgtable.h | 6 +--- arch/arm64/include/asm/memory.h | 7 ++-- arch/arm64/include/asm/processor.h | 10 +++--- arch/arm64/include/asm/sparsemem.h | 11 ++---- arch/arm64/include/asm/sysreg.h | 54 ++++++++++++++++++---------- arch/arm64/kernel/cpufeature.c | 43 +++++++++++++--------- arch/arm64/mm/fixmap.c | 2 +- arch/arm64/mm/init.c | 20 +++++------ arch/arm64/mm/kasan_init.c | 8 ++--- arch/arm64/mm/mmu.c | 2 +- drivers/irqchip/irq-gic-v3-its.c | 2 +- 12 files changed, 94 insertions(+), 92 deletions(-) diff --git a/arch/arm64/include/asm/kvm_arm.h b/arch/arm64/include/asm/kvm_arm.h index d81cc746e0ebd..08155dc17ad17 100644 --- a/arch/arm64/include/asm/kvm_arm.h +++ b/arch/arm64/include/asm/kvm_arm.h @@ -189,22 +189,13 @@ * Entry_Level = 4 - Number_of_levels. * */ -#ifdef CONFIG_ARM64_64K_PAGES +#define VTCR_EL2_TGRAN \ + (PAGE_SIZE == SZ_64K ? \ + VTCR_EL2_TG0_64K : \ + (PAGE_SIZE == SZ_16K ? VTCR_EL2_TG0_16K : VTCR_EL2_TG0_4K)) -#define VTCR_EL2_TGRAN VTCR_EL2_TG0_64K -#define VTCR_EL2_TGRAN_SL0_BASE 3UL - -#elif defined(CONFIG_ARM64_16K_PAGES) - -#define VTCR_EL2_TGRAN VTCR_EL2_TG0_16K -#define VTCR_EL2_TGRAN_SL0_BASE 3UL - -#else /* 4K */ - -#define VTCR_EL2_TGRAN VTCR_EL2_TG0_4K -#define VTCR_EL2_TGRAN_SL0_BASE 2UL - -#endif +#define VTCR_EL2_TGRAN_SL0_BASE \ + (PAGE_SIZE == SZ_64K ? 3UL : (PAGE_SIZE == SZ_16K ? 3UL : 2UL)) #define VTCR_EL2_LVLS_TO_SL0(levels) \ ((VTCR_EL2_TGRAN_SL0_BASE - (4 - (levels))) << VTCR_EL2_SL0_SHIFT) diff --git a/arch/arm64/include/asm/kvm_pgtable.h b/arch/arm64/include/asm/kvm_pgtable.h index 19278dfe79782..796614bf59e78 100644 --- a/arch/arm64/include/asm/kvm_pgtable.h +++ b/arch/arm64/include/asm/kvm_pgtable.h @@ -20,11 +20,7 @@ * - 16K (level 2): 32MB * - 64K (level 2): 512MB */ -#ifdef CONFIG_ARM64_4K_PAGES -#define KVM_PGTABLE_MIN_BLOCK_LEVEL 1 -#else -#define KVM_PGTABLE_MIN_BLOCK_LEVEL 2 -#endif +#define KVM_PGTABLE_MIN_BLOCK_LEVEL (PAGE_SIZE == SZ_4K ? 1 : 2) #define kvm_lpa2_is_enabled() system_supports_lpa2() diff --git a/arch/arm64/include/asm/memory.h b/arch/arm64/include/asm/memory.h index 54fb014eba058..6aa97fa22dc30 100644 --- a/arch/arm64/include/asm/memory.h +++ b/arch/arm64/include/asm/memory.h @@ -188,11 +188,8 @@ #define MT_S2_FWB_NORMAL_NC 5 #define MT_S2_FWB_DEVICE_nGnRE 1 -#ifdef CONFIG_ARM64_4K_PAGES -#define IOREMAP_MAX_ORDER (PUD_SHIFT) -#else -#define IOREMAP_MAX_ORDER (PMD_SHIFT) -#endif +#define IOREMAP_MAX_ORDER \ + (PAGE_SIZE == SZ_4K ? PUD_SHIFT : PMD_SHIFT) /* * Open-coded (swapper_pg_dir - reserved_pg_dir) as this cannot be calculated diff --git a/arch/arm64/include/asm/processor.h b/arch/arm64/include/asm/processor.h index f77371232d8c6..444694a4e6733 100644 --- a/arch/arm64/include/asm/processor.h +++ b/arch/arm64/include/asm/processor.h @@ -55,15 +55,15 @@ #define TASK_SIZE_MAX (UL(1) << VA_BITS) #ifdef CONFIG_COMPAT -#if defined(CONFIG_ARM64_64K_PAGES) && defined(CONFIG_KUSER_HELPERS) +#if defined(CONFIG_KUSER_HELPERS) /* - * With CONFIG_ARM64_64K_PAGES enabled, the last page is occupied - * by the compat vectors page. + * With 64K pages in use, the last page is occupied by the compat vectors page. */ -#define TASK_SIZE_32 UL(0x100000000) +#define TASK_SIZE_32 \ + (PAGE_SIZE == SZ_64K ? UL(0x100000000) : (UL(0x100000000) - PAGE_SIZE)) #else #define TASK_SIZE_32 (UL(0x100000000) - PAGE_SIZE) -#endif /* CONFIG_ARM64_64K_PAGES */ +#endif /* CONFIG_KUSER_HELPERS */ #define TASK_SIZE (test_thread_flag(TIF_32BIT) ? \ TASK_SIZE_32 : TASK_SIZE_64) #define TASK_SIZE_OF(tsk) (test_tsk_thread_flag(tsk, TIF_32BIT) ? \ diff --git a/arch/arm64/include/asm/sparsemem.h b/arch/arm64/include/asm/sparsemem.h index 8a8acc220371c..a05fdd54014f7 100644 --- a/arch/arm64/include/asm/sparsemem.h +++ b/arch/arm64/include/asm/sparsemem.h @@ -11,19 +11,12 @@ * Section size must be at least 512MB for 64K base * page size config. Otherwise it will be less than * MAX_PAGE_ORDER and the build process will fail. - */ -#ifdef CONFIG_ARM64_64K_PAGES -#define SECTION_SIZE_BITS 29 - -#else - -/* + * * Section size must be at least 128MB for 4K base * page size config. Otherwise PMD based huge page * entries could not be created for vmemmap mappings. * 16K follows 4K for simplicity. */ -#define SECTION_SIZE_BITS 27 -#endif /* CONFIG_ARM64_64K_PAGES */ +#define SECTION_SIZE_BITS (PAGE_SIZE == SZ_64K ? 29 : 27) #endif diff --git a/arch/arm64/include/asm/sysreg.h b/arch/arm64/include/asm/sysreg.h index 4a9ea103817e8..cbcf861bbf2a6 100644 --- a/arch/arm64/include/asm/sysreg.h +++ b/arch/arm64/include/asm/sysreg.h @@ -10,10 +10,12 @@ #define __ASM_SYSREG_H #include +#include #include #include #include +#include /* * ARMv8 ARM reserves the following encoding for system registers: @@ -913,24 +915,40 @@ #define ID_AA64MMFR0_EL1_PARANGE_MAX ID_AA64MMFR0_EL1_PARANGE_48 #endif -#if defined(CONFIG_ARM64_4K_PAGES) -#define ID_AA64MMFR0_EL1_TGRAN_SHIFT ID_AA64MMFR0_EL1_TGRAN4_SHIFT -#define ID_AA64MMFR0_EL1_TGRAN_LPA2 ID_AA64MMFR0_EL1_TGRAN4_52_BIT -#define ID_AA64MMFR0_EL1_TGRAN_SUPPORTED_MIN ID_AA64MMFR0_EL1_TGRAN4_SUPPORTED_MIN -#define ID_AA64MMFR0_EL1_TGRAN_SUPPORTED_MAX ID_AA64MMFR0_EL1_TGRAN4_SUPPORTED_MAX -#define ID_AA64MMFR0_EL1_TGRAN_2_SHIFT ID_AA64MMFR0_EL1_TGRAN4_2_SHIFT -#elif defined(CONFIG_ARM64_16K_PAGES) -#define ID_AA64MMFR0_EL1_TGRAN_SHIFT ID_AA64MMFR0_EL1_TGRAN16_SHIFT -#define ID_AA64MMFR0_EL1_TGRAN_LPA2 ID_AA64MMFR0_EL1_TGRAN16_52_BIT -#define ID_AA64MMFR0_EL1_TGRAN_SUPPORTED_MIN ID_AA64MMFR0_EL1_TGRAN16_SUPPORTED_MIN -#define ID_AA64MMFR0_EL1_TGRAN_SUPPORTED_MAX ID_AA64MMFR0_EL1_TGRAN16_SUPPORTED_MAX -#define ID_AA64MMFR0_EL1_TGRAN_2_SHIFT ID_AA64MMFR0_EL1_TGRAN16_2_SHIFT -#elif defined(CONFIG_ARM64_64K_PAGES) -#define ID_AA64MMFR0_EL1_TGRAN_SHIFT ID_AA64MMFR0_EL1_TGRAN64_SHIFT -#define ID_AA64MMFR0_EL1_TGRAN_SUPPORTED_MIN ID_AA64MMFR0_EL1_TGRAN64_SUPPORTED_MIN -#define ID_AA64MMFR0_EL1_TGRAN_SUPPORTED_MAX ID_AA64MMFR0_EL1_TGRAN64_SUPPORTED_MAX -#define ID_AA64MMFR0_EL1_TGRAN_2_SHIFT ID_AA64MMFR0_EL1_TGRAN64_2_SHIFT -#endif +#define ID_AA64MMFR0_EL1_TGRAN_SHIFT \ + (PAGE_SIZE == SZ_4K ? \ + ID_AA64MMFR0_EL1_TGRAN4_SHIFT : \ + (PAGE_SIZE == SZ_16K ? \ + ID_AA64MMFR0_EL1_TGRAN16_SHIFT : \ + ID_AA64MMFR0_EL1_TGRAN64_SHIFT)) + +#define ID_AA64MMFR0_EL1_TGRAN_LPA2 \ + (PAGE_SIZE == SZ_4K ? \ + ID_AA64MMFR0_EL1_TGRAN4_52_BIT : \ + (PAGE_SIZE == SZ_16K ? \ + ID_AA64MMFR0_EL1_TGRAN16_52_BIT : \ + -1)) + +#define ID_AA64MMFR0_EL1_TGRAN_SUPPORTED_MIN \ + (PAGE_SIZE == SZ_4K ? \ + ID_AA64MMFR0_EL1_TGRAN4_SUPPORTED_MIN : \ + (PAGE_SIZE == SZ_16K ? \ + ID_AA64MMFR0_EL1_TGRAN16_SUPPORTED_MIN : \ + ID_AA64MMFR0_EL1_TGRAN64_SUPPORTED_MIN)) + +#define ID_AA64MMFR0_EL1_TGRAN_SUPPORTED_MAX \ + (PAGE_SIZE == SZ_4K ? \ + ID_AA64MMFR0_EL1_TGRAN4_SUPPORTED_MAX : \ + (PAGE_SIZE == SZ_16K ? \ + ID_AA64MMFR0_EL1_TGRAN16_SUPPORTED_MAX : \ + ID_AA64MMFR0_EL1_TGRAN64_SUPPORTED_MAX)) + +#define ID_AA64MMFR0_EL1_TGRAN_2_SHIFT \ + (PAGE_SIZE == SZ_4K ? \ + ID_AA64MMFR0_EL1_TGRAN4_2_SHIFT : \ + (PAGE_SIZE == SZ_16K ? \ + ID_AA64MMFR0_EL1_TGRAN16_2_SHIFT : \ + ID_AA64MMFR0_EL1_TGRAN64_2_SHIFT)) #define CPACR_EL1_FPEN_EL1EN (BIT(20)) /* enable EL1 access */ #define CPACR_EL1_FPEN_EL0EN (BIT(21)) /* enable EL0 access, if EL1EN set */ diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c index 646ecd3069fdd..7705c9c0e7142 100644 --- a/arch/arm64/kernel/cpufeature.c +++ b/arch/arm64/kernel/cpufeature.c @@ -1831,11 +1831,13 @@ static bool has_nv1(const struct arm64_cpu_capabilities *entry, int scope) is_midr_in_range_list(read_cpuid_id(), nv1_ni_list))); } -#if defined(ID_AA64MMFR0_EL1_TGRAN_LPA2) && defined(ID_AA64MMFR0_EL1_TGRAN_2_SUPPORTED_LPA2) static bool has_lpa2_at_stage1(u64 mmfr0) { unsigned int tgran; + if (PAGE_SIZE == SZ_64K) + return false; + tgran = cpuid_feature_extract_unsigned_field(mmfr0, ID_AA64MMFR0_EL1_TGRAN_SHIFT); return tgran == ID_AA64MMFR0_EL1_TGRAN_LPA2; @@ -1845,6 +1847,9 @@ static bool has_lpa2_at_stage2(u64 mmfr0) { unsigned int tgran; + if (PAGE_SIZE == SZ_64K) + return false; + tgran = cpuid_feature_extract_unsigned_field(mmfr0, ID_AA64MMFR0_EL1_TGRAN_2_SHIFT); return tgran == ID_AA64MMFR0_EL1_TGRAN_2_SUPPORTED_LPA2; @@ -1857,10 +1862,26 @@ static bool has_lpa2(const struct arm64_cpu_capabilities *entry, int scope) mmfr0 = read_sanitised_ftr_reg(SYS_ID_AA64MMFR0_EL1); return has_lpa2_at_stage1(mmfr0) && has_lpa2_at_stage2(mmfr0); } -#else -static bool has_lpa2(const struct arm64_cpu_capabilities *entry, int scope) + +#ifdef CONFIG_ARM64_VA_BITS_52 +static bool has_va52(const struct arm64_cpu_capabilities *entry, int scope) { - return false; + const struct arm64_cpu_capabilities entry_64k = { + ARM64_CPUID_FIELDS(ID_AA64MMFR2_EL1, VARange, 52) + }; + const struct arm64_cpu_capabilities entry_16k = { + ARM64_CPUID_FIELDS(ID_AA64MMFR0_EL1, TGRAN16, 52_BIT) + }; + const struct arm64_cpu_capabilities entry_4k = { + ARM64_CPUID_FIELDS(ID_AA64MMFR0_EL1, TGRAN4, 52_BIT) + }; + + if (PAGE_SIZE == SZ_64K) + return has_cpuid_feature(&entry_64k, scope); + else if (PAGE_SIZE == SZ_16K) + return has_cpuid_feature(&entry_16k, scope); + else + return has_cpuid_feature(&entry_4k, scope); } #endif @@ -2847,20 +2868,10 @@ static const struct arm64_cpu_capabilities arm64_features[] = { }, #ifdef CONFIG_ARM64_VA_BITS_52 { + .desc = "52-bit Virtual Addressing", .capability = ARM64_HAS_VA52, .type = ARM64_CPUCAP_BOOT_CPU_FEATURE, - .matches = has_cpuid_feature, -#ifdef CONFIG_ARM64_64K_PAGES - .desc = "52-bit Virtual Addressing (LVA)", - ARM64_CPUID_FIELDS(ID_AA64MMFR2_EL1, VARange, 52) -#else - .desc = "52-bit Virtual Addressing (LPA2)", -#ifdef CONFIG_ARM64_4K_PAGES - ARM64_CPUID_FIELDS(ID_AA64MMFR0_EL1, TGRAN4, 52_BIT) -#else - ARM64_CPUID_FIELDS(ID_AA64MMFR0_EL1, TGRAN16, 52_BIT) -#endif -#endif + .matches = has_va52, }, #endif { diff --git a/arch/arm64/mm/fixmap.c b/arch/arm64/mm/fixmap.c index de1e09d986ad2..15ce3253ad359 100644 --- a/arch/arm64/mm/fixmap.c +++ b/arch/arm64/mm/fixmap.c @@ -82,7 +82,7 @@ static void __init early_fixmap_init_pud(p4d_t *p4dp, unsigned long addr, * share the top level pgd entry, which should only happen on * 16k/4 levels configurations. */ - BUG_ON(!IS_ENABLED(CONFIG_ARM64_16K_PAGES)); + BUG_ON(PAGE_SIZE != SZ_16K); } if (p4d_none(p4d)) diff --git a/arch/arm64/mm/init.c b/arch/arm64/mm/init.c index 9b5ab6818f7f3..42eb246949072 100644 --- a/arch/arm64/mm/init.c +++ b/arch/arm64/mm/init.c @@ -73,13 +73,10 @@ phys_addr_t __ro_after_init arm64_dma_phys_limit; * (64k granule), or a multiple that can be mapped using contiguous bits * in the page tables: 32 * PMD_SIZE (16k granule) */ -#if defined(CONFIG_ARM64_4K_PAGES) -#define ARM64_MEMSTART_SHIFT PUD_SHIFT -#elif defined(CONFIG_ARM64_16K_PAGES) -#define ARM64_MEMSTART_SHIFT CONT_PMD_SHIFT -#else -#define ARM64_MEMSTART_SHIFT PMD_SHIFT -#endif +#define ARM64_MEMSTART_SHIFT \ + (PAGE_SIZE == SZ_4K ? \ + PUD_SHIFT : \ + (PAGE_SIZE == SZ_16K ? CONT_PMD_SHIFT : PMD_SHIFT)) /* * sparsemem vmemmap imposes an additional requirement on the alignment of @@ -87,11 +84,10 @@ phys_addr_t __ro_after_init arm64_dma_phys_limit; * has a direct correspondence, and needs to appear sufficiently aligned * in the virtual address space. */ -#if ARM64_MEMSTART_SHIFT < SECTION_SIZE_BITS -#define ARM64_MEMSTART_ALIGN (1UL << SECTION_SIZE_BITS) -#else -#define ARM64_MEMSTART_ALIGN (1UL << ARM64_MEMSTART_SHIFT) -#endif +#define ARM64_MEMSTART_ALIGN \ + (ARM64_MEMSTART_SHIFT < SECTION_SIZE_BITS ? \ + (1UL << SECTION_SIZE_BITS) : \ + (1UL << ARM64_MEMSTART_SHIFT)) static void __init arch_reserve_crashkernel(void) { diff --git a/arch/arm64/mm/kasan_init.c b/arch/arm64/mm/kasan_init.c index b65a29440a0c9..9af897fb3c432 100644 --- a/arch/arm64/mm/kasan_init.c +++ b/arch/arm64/mm/kasan_init.c @@ -178,10 +178,10 @@ static void __init kasan_pgd_populate(unsigned long addr, unsigned long end, } while (pgdp++, addr = next, addr != end); } -#if defined(CONFIG_ARM64_64K_PAGES) || CONFIG_PGTABLE_LEVELS > 4 +#if CONFIG_PGTABLE_LEVELS > 4 #define SHADOW_ALIGN P4D_SIZE #else -#define SHADOW_ALIGN PUD_SIZE +#define SHADOW_ALIGN (PAGE_SIZE == SZ_64K ? P4D_SIZE : PUD_SIZE) #endif /* @@ -243,8 +243,8 @@ static int __init root_level_idx(u64 addr) * not implemented. This means we need to index the table as usual, * instead of masking off bits based on vabits_actual. */ - u64 vabits = IS_ENABLED(CONFIG_ARM64_64K_PAGES) ? VA_BITS - : vabits_actual; + u64 vabits = PAGE_SIZE == SZ_64K ? VA_BITS + : vabits_actual; int shift = (ARM64_HW_PGTABLE_LEVELS(vabits) - 1) * (PAGE_SHIFT - 3); return (addr & ~_PAGE_OFFSET(vabits)) >> (shift + PAGE_SHIFT); diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index d4d30eaefb4cd..a528787c1e550 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -1179,7 +1179,7 @@ int __meminit vmemmap_populate(unsigned long start, unsigned long end, int node, { WARN_ON((start < VMEMMAP_START) || (end > VMEMMAP_END)); - if (!IS_ENABLED(CONFIG_ARM64_4K_PAGES)) + if (PAGE_SIZE != SZ_4K) return vmemmap_populate_basepages(start, end, node, altmap); else return vmemmap_populate_hugepages(start, end, node, altmap); diff --git a/drivers/irqchip/irq-gic-v3-its.c b/drivers/irqchip/irq-gic-v3-its.c index fdec478ba5e70..b745579b4b9f3 100644 --- a/drivers/irqchip/irq-gic-v3-its.c +++ b/drivers/irqchip/irq-gic-v3-its.c @@ -2323,7 +2323,7 @@ static int its_setup_baser(struct its_node *its, struct its_baser *baser, baser_phys = virt_to_phys(base); /* Check if the physical address of the memory is above 48bits */ - if (IS_ENABLED(CONFIG_ARM64_64K_PAGES) && (baser_phys >> 48)) { + if (PAGE_SIZE == SZ_64K && (baser_phys >> 48)) { /* 52bit PA is supported only when PageSize=64K */ if (psz != SZ_64K) { From patchwork Mon Oct 14 10:58:51 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13834907 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 7FFE8D16252 for ; Mon, 14 Oct 2024 12:26:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=XkvwSRpxq/ReVb2KZsrYsZWT5AXEytyt77PllAZm++A=; b=quLcCdyP6eHegaarfAIQTNyuH8 gBc74rzusk64WxdJSxS+Y09Pe6e1EJ3KCom7IkMbJZTQXMDkASYW4PSxfs3I8GqFd+ul3Rs2YiB4E o++2LhZjIEzBpiHj45TqD0BiaOSx11MwLThqXC97KH3d7vYV9O8gZBQPEyiEcfkWRXCzUnKo1nARt OquwRB0KZC7E6bWpT+9iayG+bf5GPXr9r4qiKAgF+B79pl/AmTN/Qxk64Jezv11dcxyfLhbKlmSoy 9e69YyHclegiO6FJB8sg4tSlLHWQKFrV7xdIHTaaCs8gSMMlICbiiheFLTPnb8Aek68u+2yPVbPyh KsT6hjIg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1t0K94-000000054dJ-2IBT; Mon, 14 Oct 2024 12:25:54 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1t0Ipb-00000004oVN-1nXQ for linux-arm-kernel@lists.infradead.org; Mon, 14 Oct 2024 11:01:47 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id AEC241763; Mon, 14 Oct 2024 04:02:12 -0700 (PDT) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.27]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 722EB3F51B; Mon, 14 Oct 2024 04:01:40 -0700 (PDT) From: Ryan Roberts To: Andrew Morton , Anshuman Khandual , Ard Biesheuvel , Catalin Marinas , David Hildenbrand , Greg Marsden , Ivan Ivanov , Kalesh Singh , Marc Zyngier , Mark Rutland , Matthias Brugger , Miroslav Benes , Oliver Upton , Will Deacon Cc: Ryan Roberts , kvmarm@lists.linux.dev, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [RFC PATCH v1 44/57] arm64: Align sections to PAGE_SIZE_MAX Date: Mon, 14 Oct 2024 11:58:51 +0100 Message-ID: <20241014105912.3207374-44-ryan.roberts@arm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20241014105912.3207374-1-ryan.roberts@arm.com> References: <20241014105514.3206191-1-ryan.roberts@arm.com> <20241014105912.3207374-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241014_040143_632337_0A454B30 X-CRM114-Status: GOOD ( 17.85 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Increase alignment of sections in nvhe hyp, vdso and final vmlinux image from PAGE_SIZE to PAGE_SIZE_MAX. For compile-time PAGE_SIZE, PAGE_SIZE_MAX == PAGE_SIZE so there is no change. For boot-time PAGE_SIZE, PAGE_SIZE_MAX is the largest selectable page size. For a boot-time page size build, image size is comparable to a 64K page size compile-time build. In future, it may be desirable to optimize run-time memory consumption by freeing unused padding pages when the boot-time selected page size is less than PAGE_SIZE_MAX. Signed-off-by: Ryan Roberts --- ***NOTE*** Any confused maintainers may want to read the cover note here for context: https://lore.kernel.org/all/20241014105514.3206191-1-ryan.roberts@arm.com/ arch/arm64/include/asm/memory.h | 4 +-- arch/arm64/kernel/vdso-wrap.S | 4 +-- arch/arm64/kernel/vdso.c | 7 +++--- arch/arm64/kernel/vdso/vdso.lds.S | 4 +-- arch/arm64/kernel/vdso32-wrap.S | 4 +-- arch/arm64/kernel/vdso32/vdso.lds.S | 4 +-- arch/arm64/kernel/vmlinux.lds.S | 38 ++++++++++++++--------------- arch/arm64/kvm/hyp/nvhe/hyp.lds.S | 2 +- 8 files changed, 34 insertions(+), 33 deletions(-) diff --git a/arch/arm64/include/asm/memory.h b/arch/arm64/include/asm/memory.h index 6aa97fa22dc30..5393a859183f7 100644 --- a/arch/arm64/include/asm/memory.h +++ b/arch/arm64/include/asm/memory.h @@ -195,13 +195,13 @@ * Open-coded (swapper_pg_dir - reserved_pg_dir) as this cannot be calculated * until link time. */ -#define RESERVED_SWAPPER_OFFSET (PAGE_SIZE) +#define RESERVED_SWAPPER_OFFSET (PAGE_SIZE_MAX) /* * Open-coded (swapper_pg_dir - tramp_pg_dir) as this cannot be calculated * until link time. */ -#define TRAMP_SWAPPER_OFFSET (2 * PAGE_SIZE) +#define TRAMP_SWAPPER_OFFSET (2 * PAGE_SIZE_MAX) #ifndef __ASSEMBLY__ diff --git a/arch/arm64/kernel/vdso-wrap.S b/arch/arm64/kernel/vdso-wrap.S index c4b1990bf2be0..79fa77628199b 100644 --- a/arch/arm64/kernel/vdso-wrap.S +++ b/arch/arm64/kernel/vdso-wrap.S @@ -13,10 +13,10 @@ .globl vdso_start, vdso_end .section .rodata - .balign PAGE_SIZE + .balign PAGE_SIZE_MAX vdso_start: .incbin "arch/arm64/kernel/vdso/vdso.so" - .balign PAGE_SIZE + .balign PAGE_SIZE_MAX vdso_end: .previous diff --git a/arch/arm64/kernel/vdso.c b/arch/arm64/kernel/vdso.c index 89b6e78400023..1efe98909a2e0 100644 --- a/arch/arm64/kernel/vdso.c +++ b/arch/arm64/kernel/vdso.c @@ -195,7 +195,7 @@ static int __setup_additional_pages(enum vdso_abi abi, vdso_text_len = vdso_info[abi].vdso_pages << PAGE_SHIFT; /* Be sure to map the data page */ - vdso_mapping_len = vdso_text_len + VVAR_NR_PAGES * PAGE_SIZE; + vdso_mapping_len = vdso_text_len + VVAR_NR_PAGES * PAGE_SIZE_MAX; vdso_base = get_unmapped_area(NULL, 0, vdso_mapping_len, 0, 0); if (IS_ERR_VALUE(vdso_base)) { @@ -203,7 +203,8 @@ static int __setup_additional_pages(enum vdso_abi abi, goto up_fail; } - ret = _install_special_mapping(mm, vdso_base, VVAR_NR_PAGES * PAGE_SIZE, + ret = _install_special_mapping(mm, vdso_base, + VVAR_NR_PAGES * PAGE_SIZE_MAX, VM_READ|VM_MAYREAD|VM_PFNMAP, vdso_info[abi].dm); if (IS_ERR(ret)) @@ -212,7 +213,7 @@ static int __setup_additional_pages(enum vdso_abi abi, if (system_supports_bti_kernel()) gp_flags = VM_ARM64_BTI; - vdso_base += VVAR_NR_PAGES * PAGE_SIZE; + vdso_base += VVAR_NR_PAGES * PAGE_SIZE_MAX; mm->context.vdso = (void *)vdso_base; ret = _install_special_mapping(mm, vdso_base, vdso_text_len, VM_READ|VM_EXEC|gp_flags| diff --git a/arch/arm64/kernel/vdso/vdso.lds.S b/arch/arm64/kernel/vdso/vdso.lds.S index 45354f2ddf706..f7d1537a689e8 100644 --- a/arch/arm64/kernel/vdso/vdso.lds.S +++ b/arch/arm64/kernel/vdso/vdso.lds.S @@ -18,9 +18,9 @@ OUTPUT_ARCH(aarch64) SECTIONS { - PROVIDE(_vdso_data = . - __VVAR_PAGES * PAGE_SIZE); + PROVIDE(_vdso_data = . - __VVAR_PAGES * PAGE_SIZE_MAX); #ifdef CONFIG_TIME_NS - PROVIDE(_timens_data = _vdso_data + PAGE_SIZE); + PROVIDE(_timens_data = _vdso_data + PAGE_SIZE_MAX); #endif . = VDSO_LBASE + SIZEOF_HEADERS; diff --git a/arch/arm64/kernel/vdso32-wrap.S b/arch/arm64/kernel/vdso32-wrap.S index e72ac7bc4c04f..1c6069d6c457e 100644 --- a/arch/arm64/kernel/vdso32-wrap.S +++ b/arch/arm64/kernel/vdso32-wrap.S @@ -10,10 +10,10 @@ .globl vdso32_start, vdso32_end .section .rodata - .balign PAGE_SIZE + .balign PAGE_SIZE_MAX vdso32_start: .incbin "arch/arm64/kernel/vdso32/vdso.so" - .balign PAGE_SIZE + .balign PAGE_SIZE_MAX vdso32_end: .previous diff --git a/arch/arm64/kernel/vdso32/vdso.lds.S b/arch/arm64/kernel/vdso32/vdso.lds.S index 8d95d7d35057d..c46d18a69d1ce 100644 --- a/arch/arm64/kernel/vdso32/vdso.lds.S +++ b/arch/arm64/kernel/vdso32/vdso.lds.S @@ -18,9 +18,9 @@ OUTPUT_ARCH(arm) SECTIONS { - PROVIDE_HIDDEN(_vdso_data = . - __VVAR_PAGES * PAGE_SIZE); + PROVIDE_HIDDEN(_vdso_data = . - __VVAR_PAGES * PAGE_SIZE_MAX); #ifdef CONFIG_TIME_NS - PROVIDE_HIDDEN(_timens_data = _vdso_data + PAGE_SIZE); + PROVIDE_HIDDEN(_timens_data = _vdso_data + PAGE_SIZE_MAX); #endif . = VDSO_LBASE + SIZEOF_HEADERS; diff --git a/arch/arm64/kernel/vmlinux.lds.S b/arch/arm64/kernel/vmlinux.lds.S index 7f3f6d709ae73..1ef6dea13b57c 100644 --- a/arch/arm64/kernel/vmlinux.lds.S +++ b/arch/arm64/kernel/vmlinux.lds.S @@ -15,16 +15,16 @@ #define HYPERVISOR_DATA_SECTIONS \ HYP_SECTION_NAME(.rodata) : { \ - . = ALIGN(PAGE_SIZE); \ + . = ALIGN(PAGE_SIZE_MAX); \ __hyp_rodata_start = .; \ *(HYP_SECTION_NAME(.data..ro_after_init)) \ *(HYP_SECTION_NAME(.rodata)) \ - . = ALIGN(PAGE_SIZE); \ + . = ALIGN(PAGE_SIZE_MAX); \ __hyp_rodata_end = .; \ } #define HYPERVISOR_PERCPU_SECTION \ - . = ALIGN(PAGE_SIZE); \ + . = ALIGN(PAGE_SIZE_MAX); \ HYP_SECTION_NAME(.data..percpu) : { \ *(HYP_SECTION_NAME(.data..percpu)) \ } @@ -39,7 +39,7 @@ #define BSS_FIRST_SECTIONS \ __hyp_bss_start = .; \ *(HYP_SECTION_NAME(.bss)) \ - . = ALIGN(PAGE_SIZE); \ + . = ALIGN(PAGE_SIZE_MAX); \ __hyp_bss_end = .; /* @@ -48,7 +48,7 @@ * between them, which can in some cases cause the linker to misalign them. To * work around the issue, force a page alignment for __bss_start. */ -#define SBSS_ALIGN PAGE_SIZE +#define SBSS_ALIGN PAGE_SIZE_MAX #else /* CONFIG_KVM */ #define HYPERVISOR_EXTABLE #define HYPERVISOR_DATA_SECTIONS @@ -75,14 +75,14 @@ ENTRY(_text) jiffies = jiffies_64; #define HYPERVISOR_TEXT \ - . = ALIGN(PAGE_SIZE); \ + . = ALIGN(PAGE_SIZE_MAX); \ __hyp_idmap_text_start = .; \ *(.hyp.idmap.text) \ __hyp_idmap_text_end = .; \ __hyp_text_start = .; \ *(.hyp.text) \ HYPERVISOR_EXTABLE \ - . = ALIGN(PAGE_SIZE); \ + . = ALIGN(PAGE_SIZE_MAX); \ __hyp_text_end = .; #define IDMAP_TEXT \ @@ -113,11 +113,11 @@ jiffies = jiffies_64; #ifdef CONFIG_UNMAP_KERNEL_AT_EL0 #define TRAMP_TEXT \ - . = ALIGN(PAGE_SIZE); \ + . = ALIGN(PAGE_SIZE_MAX); \ __entry_tramp_text_start = .; \ *(.entry.tramp.text) \ - . = ALIGN(PAGE_SIZE); \ __entry_tramp_text_end = .; \ + . = ALIGN(PAGE_SIZE_MAX); \ *(.entry.tramp.rodata) #else #define TRAMP_TEXT @@ -187,7 +187,7 @@ SECTIONS _etext = .; /* End of text section */ /* everything from this point to __init_begin will be marked RO NX */ - RO_DATA(PAGE_SIZE) + RO_DATA(PAGE_SIZE_MAX) HYPERVISOR_DATA_SECTIONS @@ -206,22 +206,22 @@ SECTIONS HIBERNATE_TEXT KEXEC_TEXT IDMAP_TEXT - . = ALIGN(PAGE_SIZE); + . = ALIGN(PAGE_SIZE_MAX); } idmap_pg_dir = .; - . += PAGE_SIZE; + . += PAGE_SIZE_MAX; #ifdef CONFIG_UNMAP_KERNEL_AT_EL0 tramp_pg_dir = .; - . += PAGE_SIZE; + . += PAGE_SIZE_MAX; #endif reserved_pg_dir = .; - . += PAGE_SIZE; + . += PAGE_SIZE_MAX; swapper_pg_dir = .; - . += PAGE_SIZE; + . += PAGE_SIZE_MAX; . = ALIGN(SEGMENT_ALIGN); __init_begin = .; @@ -290,7 +290,7 @@ SECTIONS _data = .; _sdata = .; - RW_DATA(L1_CACHE_BYTES, PAGE_SIZE, THREAD_ALIGN) + RW_DATA(L1_CACHE_BYTES, PAGE_SIZE_MAX, THREAD_ALIGN) /* * Data written with the MMU off but read with the MMU on requires @@ -317,7 +317,7 @@ SECTIONS /* start of zero-init region */ BSS_SECTION(SBSS_ALIGN, 0, 0) - . = ALIGN(PAGE_SIZE); + . = ALIGN(PAGE_SIZE_MAX); init_pg_dir = .; . += INIT_DIR_SIZE_MAX; init_pg_end = .; @@ -356,7 +356,7 @@ SECTIONS * former is page-aligned, but the latter may not be with 16K or 64K pages, so * it should also not cross a page boundary. */ -ASSERT(__hyp_idmap_text_end - __hyp_idmap_text_start <= PAGE_SIZE, +ASSERT(__hyp_idmap_text_end - __hyp_idmap_text_start <= SZ_4K, "HYP init code too big") ASSERT(__idmap_text_end - (__idmap_text_start & ~(SZ_4K - 1)) <= SZ_4K, "ID map text too big or misaligned") @@ -367,7 +367,7 @@ ASSERT(__hibernate_exit_text_start == swsusp_arch_suspend_exit, "Hibernate exit text does not start with swsusp_arch_suspend_exit") #endif #ifdef CONFIG_UNMAP_KERNEL_AT_EL0 -ASSERT((__entry_tramp_text_end - __entry_tramp_text_start) <= 3*PAGE_SIZE, +ASSERT((__entry_tramp_text_end - __entry_tramp_text_start) <= 3 * SZ_4K, "Entry trampoline text too big") #endif #ifdef CONFIG_KVM diff --git a/arch/arm64/kvm/hyp/nvhe/hyp.lds.S b/arch/arm64/kvm/hyp/nvhe/hyp.lds.S index f4562f417d3fc..74c7c21626270 100644 --- a/arch/arm64/kvm/hyp/nvhe/hyp.lds.S +++ b/arch/arm64/kvm/hyp/nvhe/hyp.lds.S @@ -21,7 +21,7 @@ SECTIONS { * .hyp..data..percpu needs to be page aligned to maintain the same * alignment for when linking into vmlinux. */ - . = ALIGN(PAGE_SIZE); + . = ALIGN(PAGE_SIZE_MAX); BEGIN_HYP_SECTION(.data..percpu) PERCPU_INPUT(L1_CACHE_BYTES) END_HYP_SECTION From patchwork Mon Oct 14 10:58:52 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13834921 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id D208ED16252 for ; Mon, 14 Oct 2024 12:27:31 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=OACUuu7EkK/bVx0oPSFpoAc1oPx/wt8iZOtMNIFSzjQ=; b=pPgtLORaWSyr4cSjoQUaY+m02o s+J+lsloOjQa+2Hl1yMe8bk4c2OefxjlYiqh1q98Fr4f5Tk8JbhDgHKDvlLuQXV75doa9Ktonmgqs wU0SvqKEIq5pfnX3GS91lHRSOvrtW66J/gFe2vW+7DP2kQT7NwMnDFtiwmYTaRH0+L2p0CigwwrnB 0IRap2zg4ytUbH394iwzq9q+OvwUfKX74UAH6fjb7kQJA3x+ZIejowqP/JcQh+Rp+lf1Qo84rVsSi e8BiKDnng2+c3etKV/rrW/hh+W1mBkC3cuXgWb3x9Tg3fDRpVx5dG350+pyHsGiytyOHFwMzkHSML PwcT9SmA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1t0KAR-000000054o8-0Bd6; Mon, 14 Oct 2024 12:27:19 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1t0Ipe-00000004oWr-0afB for linux-arm-kernel@lists.infradead.org; Mon, 14 Oct 2024 11:01:48 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 5E6BA1764; Mon, 14 Oct 2024 04:02:15 -0700 (PDT) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.27]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 5FBE13F51B; Mon, 14 Oct 2024 04:01:43 -0700 (PDT) From: Ryan Roberts To: Andrew Morton , Anshuman Khandual , Ard Biesheuvel , Catalin Marinas , David Hildenbrand , Greg Marsden , Ivan Ivanov , Kalesh Singh , Marc Zyngier , Mark Rutland , Matthias Brugger , Miroslav Benes , Will Deacon Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [RFC PATCH v1 45/57] arm64: Rework trampoline rodata mapping Date: Mon, 14 Oct 2024 11:58:52 +0100 Message-ID: <20241014105912.3207374-45-ryan.roberts@arm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20241014105912.3207374-1-ryan.roberts@arm.com> References: <20241014105514.3206191-1-ryan.roberts@arm.com> <20241014105912.3207374-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241014_040146_366337_C1B1B19E X-CRM114-Status: GOOD ( 19.45 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Now that the trampoline rodata is aligned to the next PAGE_SIZE_MAX boundary after the end of the trampoline text, the code that maps it in the fixmap is incorrect, because it still assumes the rodata is in the next page immediately after the text. Of course it still works for now with compile-time page size but for boot-time page size when selecting a page size less than PAGE_SIZE_MAX, it will fail. So let's fix that by allocating sufficient fixmap slots to cover the extra alignment padding in the worst case (PAGE_SIZE == PAGE_SIZE_MIN) and explicitly mapping the rodata to the slot offset correctly from the text. Signed-off-by: Ryan Roberts --- ***NOTE*** Any confused maintainers may want to read the cover note here for context: https://lore.kernel.org/all/20241014105514.3206191-1-ryan.roberts@arm.com/ arch/arm64/include/asm/fixmap.h | 16 +++++++++++----- arch/arm64/include/asm/sections.h | 1 + arch/arm64/kernel/vmlinux.lds.S | 4 +++- arch/arm64/mm/mmu.c | 22 ++++++++++++++-------- 4 files changed, 29 insertions(+), 14 deletions(-) diff --git a/arch/arm64/include/asm/fixmap.h b/arch/arm64/include/asm/fixmap.h index 87e307804b99c..9a496d54dfe6e 100644 --- a/arch/arm64/include/asm/fixmap.h +++ b/arch/arm64/include/asm/fixmap.h @@ -59,13 +59,19 @@ enum fixed_addresses { #endif /* CONFIG_ACPI_APEI_GHES */ #ifdef CONFIG_UNMAP_KERNEL_AT_EL0 +#define TRAMP_TEXT_SIZE (PAGE_SIZE_MIN * 3) #ifdef CONFIG_RELOCATABLE - FIX_ENTRY_TRAMP_TEXT4, /* one extra slot for the data page */ +#define TRAMP_DATA_SIZE PAGE_SIZE_MIN +#define TRAMP_PAD_SIZE (PAGE_SIZE_MAX - PAGE_SIZE_MIN) +#else +#define TRAMP_DATA_SIZE 0 +#define TRAMP_PAD_SIZE 0 #endif - FIX_ENTRY_TRAMP_TEXT3, - FIX_ENTRY_TRAMP_TEXT2, - FIX_ENTRY_TRAMP_TEXT1, -#define TRAMP_VALIAS (__fix_to_virt(FIX_ENTRY_TRAMP_TEXT1)) +#define TRAMP_SIZE (TRAMP_TEXT_SIZE + TRAMP_DATA_SIZE + TRAMP_PAD_SIZE) + FIX_ENTRY_TRAMP_END, + FIX_ENTRY_TRAMP_BEGIN = FIX_ENTRY_TRAMP_END + + DIV_ROUND_UP(TRAMP_SIZE, PAGE_SIZE_MIN) - 1, +#define TRAMP_VALIAS (__fix_to_virt(FIX_ENTRY_TRAMP_BEGIN)) #endif /* CONFIG_UNMAP_KERNEL_AT_EL0 */ __end_of_permanent_fixed_addresses, diff --git a/arch/arm64/include/asm/sections.h b/arch/arm64/include/asm/sections.h index 40971ac1303f9..252ec58963093 100644 --- a/arch/arm64/include/asm/sections.h +++ b/arch/arm64/include/asm/sections.h @@ -21,6 +21,7 @@ extern char __exittext_begin[], __exittext_end[]; extern char __irqentry_text_start[], __irqentry_text_end[]; extern char __mmuoff_data_start[], __mmuoff_data_end[]; extern char __entry_tramp_text_start[], __entry_tramp_text_end[]; +extern char __entry_tramp_rodata_start[], __entry_tramp_rodata_end[]; extern char __relocate_new_kernel_start[], __relocate_new_kernel_end[]; static inline size_t entry_tramp_text_size(void) diff --git a/arch/arm64/kernel/vmlinux.lds.S b/arch/arm64/kernel/vmlinux.lds.S index 1ef6dea13b57c..09fcc234c0f77 100644 --- a/arch/arm64/kernel/vmlinux.lds.S +++ b/arch/arm64/kernel/vmlinux.lds.S @@ -118,7 +118,9 @@ jiffies = jiffies_64; *(.entry.tramp.text) \ __entry_tramp_text_end = .; \ . = ALIGN(PAGE_SIZE_MAX); \ - *(.entry.tramp.rodata) + __entry_tramp_rodata_start = .; \ + *(.entry.tramp.rodata) \ + __entry_tramp_rodata_end = .; #else #define TRAMP_TEXT #endif diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index a528787c1e550..84df9f278d24d 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -734,25 +734,31 @@ static int __init map_entry_trampoline(void) return 0; pgprot_t prot = kernel_exec_prot(); - phys_addr_t pa_start = __pa_symbol(__entry_tramp_text_start); + phys_addr_t pa_text = __pa_symbol(__entry_tramp_text_start); + phys_addr_t pa_data = __pa_symbol(__entry_tramp_rodata_start); + int slot = FIX_ENTRY_TRAMP_BEGIN; /* The trampoline is always mapped and can therefore be global */ pgprot_val(prot) &= ~PTE_NG; /* Map only the text into the trampoline page table */ memset(tramp_pg_dir, 0, PGD_SIZE); - __create_pgd_mapping(tramp_pg_dir, pa_start, TRAMP_VALIAS, + __create_pgd_mapping(tramp_pg_dir, pa_text, TRAMP_VALIAS, entry_tramp_text_size(), prot, __pgd_pgtable_alloc, NO_BLOCK_MAPPINGS); /* Map both the text and data into the kernel page table */ - for (i = 0; i < DIV_ROUND_UP(entry_tramp_text_size(), PAGE_SIZE); i++) - __set_fixmap(FIX_ENTRY_TRAMP_TEXT1 - i, - pa_start + i * PAGE_SIZE, prot); + for (i = 0; i < DIV_ROUND_UP(entry_tramp_text_size(), PAGE_SIZE); i++) { + __set_fixmap(slot, pa_text, prot); + pa_text += PAGE_SIZE; + slot--; + } - if (IS_ENABLED(CONFIG_RELOCATABLE)) - __set_fixmap(FIX_ENTRY_TRAMP_TEXT1 - i, - pa_start + i * PAGE_SIZE, PAGE_KERNEL_RO); + if (IS_ENABLED(CONFIG_RELOCATABLE)) { + slot -= (pa_data - pa_text) / PAGE_SIZE; + VM_BUG_ON(slot < FIX_ENTRY_TRAMP_END); + __set_fixmap(slot, pa_data, PAGE_KERNEL_RO); + } return 0; } From patchwork Mon Oct 14 10:58:53 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13834922 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 13C71D16256 for ; Mon, 14 Oct 2024 12:28:57 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=Pem7/e2IY8K92Dd4GqzgmCKk29IF+hVKC5zPWZhX1eg=; b=WVwVaTTL+2STvSQ2EydJaA1a+V 244hZDwBcghe0JPQ/YsijkfHOPxkrsHsk/fwpFbQCnhfTmNJcPOiyPVUEsDG8TKIwQd44JLHQjteO o1oABRwa49uRfEBKfq+hwTvKyAGY6oD3ASN2LPACIva5Tqtyo4VRD60rdh3J4KkMbO0tPuxsLS+zb H1ytGDi5ORoYklBttYup6KEfdr4NcpUk6guVtQ0DZU4Dz4WOvKq9ecYkE1rxnVS1jlcRZpi4nMG6S RuiIv1BeoY0FkUmeGO5hVIWoc5SuoiJ4Hd8dtKruvNB65HaSpJ/DKsd89EIuFBrR8NQBYdIyjZ8pd urvkP3TQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1t0KBm-000000054zm-3IVK; Mon, 14 Oct 2024 12:28:42 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1t0Ipg-00000004oZ1-3DHe for linux-arm-kernel@lists.infradead.org; Mon, 14 Oct 2024 11:01:50 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 0E2C6176A; Mon, 14 Oct 2024 04:02:18 -0700 (PDT) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.27]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 0FACB3F51B; Mon, 14 Oct 2024 04:01:45 -0700 (PDT) From: Ryan Roberts To: Andrew Morton , Anshuman Khandual , Ard Biesheuvel , Catalin Marinas , David Hildenbrand , Greg Marsden , Ivan Ivanov , Kalesh Singh , Marc Zyngier , Mark Rutland , Matthias Brugger , Miroslav Benes , Will Deacon Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [RFC PATCH v1 46/57] arm64: Generalize fixmap for boot-time page size Date: Mon, 14 Oct 2024 11:58:53 +0100 Message-ID: <20241014105912.3207374-46-ryan.roberts@arm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20241014105912.3207374-1-ryan.roberts@arm.com> References: <20241014105514.3206191-1-ryan.roberts@arm.com> <20241014105912.3207374-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241014_040148_919059_C131A498 X-CRM114-Status: GOOD ( 14.54 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Some fixmap fixed address slots previously depended on PAGE_SIZE (i.e. to determine how many slots were required to cover a given size). Since we require the fixed address slots to be compile-time constant, let's work out the worst case number of required slots when page size is PAGE_SIZE_MIN instead. Additionally, let's determine the worst-case number of PTE tables we require and statically allocate enough memory. For compile-time page size builds, the end result is the same as it was previously. Signed-off-by: Ryan Roberts --- ***NOTE*** Any confused maintainers may want to read the cover note here for context: https://lore.kernel.org/all/20241014105514.3206191-1-ryan.roberts@arm.com/ arch/arm64/include/asm/fixmap.h | 12 ++++++++---- arch/arm64/mm/fixmap.c | 34 ++++++++++++++++++++++----------- 2 files changed, 31 insertions(+), 15 deletions(-) diff --git a/arch/arm64/include/asm/fixmap.h b/arch/arm64/include/asm/fixmap.h index 9a496d54dfe6e..c73fd3c1334ff 100644 --- a/arch/arm64/include/asm/fixmap.h +++ b/arch/arm64/include/asm/fixmap.h @@ -43,7 +43,7 @@ enum fixed_addresses { * whether it crosses any page boundary. */ FIX_FDT_END, - FIX_FDT = FIX_FDT_END + DIV_ROUND_UP(MAX_FDT_SIZE, PAGE_SIZE) + 1, + FIX_FDT = FIX_FDT_END + DIV_ROUND_UP(MAX_FDT_SIZE, PAGE_SIZE_MIN) + 1, FIX_EARLYCON_MEM_BASE, FIX_TEXT_POKE0, @@ -79,7 +79,7 @@ enum fixed_addresses { * Temporary boot-time mappings, used by early_ioremap(), * before ioremap() is functional. */ -#define NR_FIX_BTMAPS (SZ_256K / PAGE_SIZE) +#define NR_FIX_BTMAPS (SZ_256K / PAGE_SIZE_MIN) #define FIX_BTMAPS_SLOTS 7 #define TOTAL_FIX_BTMAPS (NR_FIX_BTMAPS * FIX_BTMAPS_SLOTS) @@ -101,8 +101,12 @@ enum fixed_addresses { #define FIXADDR_SIZE (__end_of_permanent_fixed_addresses << PAGE_SHIFT) #define FIXADDR_START (FIXADDR_TOP - FIXADDR_SIZE) -#define FIXADDR_TOT_SIZE (__end_of_fixed_addresses << PAGE_SHIFT) -#define FIXADDR_TOT_START (FIXADDR_TOP - FIXADDR_TOT_SIZE) +#define __FIXADDR_TOT_SIZE(page_shift) \ + (__end_of_fixed_addresses << (page_shift)) +#define __FIXADDR_TOT_START(page_shift) \ + (FIXADDR_TOP - __FIXADDR_TOT_SIZE(page_shift)) +#define FIXADDR_TOT_SIZE __FIXADDR_TOT_SIZE(PAGE_SHIFT) +#define FIXADDR_TOT_START __FIXADDR_TOT_START(PAGE_SHIFT) #define FIXMAP_PAGE_IO __pgprot(PROT_DEVICE_nGnRE) diff --git a/arch/arm64/mm/fixmap.c b/arch/arm64/mm/fixmap.c index 15ce3253ad359..a0dcf2375ccb4 100644 --- a/arch/arm64/mm/fixmap.c +++ b/arch/arm64/mm/fixmap.c @@ -17,27 +17,39 @@ #include /* ensure that the fixmap region does not grow down into the PCI I/O region */ -static_assert(FIXADDR_TOT_START > PCI_IO_END); +static_assert(__FIXADDR_TOT_START(PAGE_SHIFT_MAX) > PCI_IO_END); -#define NR_BM_PTE_TABLES \ - SPAN_NR_ENTRIES(FIXADDR_TOT_START, FIXADDR_TOP, PMD_SHIFT) -#define NR_BM_PMD_TABLES \ - SPAN_NR_ENTRIES(FIXADDR_TOT_START, FIXADDR_TOP, PUD_SHIFT) +#define FIXMAP_LEVEL(page_shift, lvl, vstart, vend) \ + SPAN_NR_ENTRIES(vstart, vend, PGTABLE_LEVEL_SHIFT(page_shift, lvl)) -static_assert(NR_BM_PMD_TABLES == 1); +#define FIXMAP_PAGES(page_shift, level) \ + FIXMAP_LEVEL(page_shift, level, \ + __FIXADDR_TOT_START(page_shift), FIXADDR_TOP) + +#define FIXMAP_SIZE(page_shift, level) \ + (FIXMAP_PAGES(page_shift, level) * (UL(1) << (page_shift))) + +#define FIXMAP_PTE_SIZE_MAX \ + MAX_IF_HAVE_PGSZ(FIXMAP_SIZE(ARM64_PAGE_SHIFT_4K, 2), \ + FIXMAP_SIZE(ARM64_PAGE_SHIFT_16K, 2), \ + FIXMAP_SIZE(ARM64_PAGE_SHIFT_64K, 2)) + +static_assert(FIXMAP_PAGES(ARM64_PAGE_SHIFT_4K, 1) == 1); +static_assert(FIXMAP_PAGES(ARM64_PAGE_SHIFT_16K, 1) == 1); +static_assert(FIXMAP_PAGES(ARM64_PAGE_SHIFT_64K, 1) == 1); #define __BM_TABLE_IDX(addr, shift) \ (((addr) >> (shift)) - (FIXADDR_TOT_START >> (shift))) #define BM_PTE_TABLE_IDX(addr) __BM_TABLE_IDX(addr, PMD_SHIFT) -static pte_t bm_pte[NR_BM_PTE_TABLES][PTRS_PER_PTE] __page_aligned_bss; -static pmd_t bm_pmd[PTRS_PER_PMD] __page_aligned_bss __maybe_unused; -static pud_t bm_pud[PTRS_PER_PUD] __page_aligned_bss __maybe_unused; +static pte_t bm_pte[FIXMAP_PTE_SIZE_MAX / sizeof(pte_t)] __page_aligned_bss; +static pmd_t bm_pmd[MAX_PTRS_PER_PMD] __page_aligned_bss __maybe_unused; +static pud_t bm_pud[MAX_PTRS_PER_PUD] __page_aligned_bss __maybe_unused; static inline pte_t *fixmap_pte(unsigned long addr) { - return &bm_pte[BM_PTE_TABLE_IDX(addr)][pte_index(addr)]; + return &bm_pte[BM_PTE_TABLE_IDX(addr) * PTRS_PER_PTE + pte_index(addr)]; } static void __init early_fixmap_init_pte(pmd_t *pmdp, unsigned long addr) @@ -46,7 +58,7 @@ static void __init early_fixmap_init_pte(pmd_t *pmdp, unsigned long addr) pte_t *ptep; if (pmd_none(pmd)) { - ptep = bm_pte[BM_PTE_TABLE_IDX(addr)]; + ptep = &bm_pte[BM_PTE_TABLE_IDX(addr) * PTRS_PER_PTE]; __pmd_populate(pmdp, __pa_symbol(ptep), PMD_TYPE_TABLE); } } From patchwork Mon Oct 14 10:58:54 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13834923 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 097A3D16256 for ; Mon, 14 Oct 2024 12:30:20 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=YPpXHP2mLKgaz5VTCYQnV2Bzp3U6nRyN8VK8UrnJsws=; b=4N9oCP+5ysKbpx3HPMk3BT4DAW aHo3FOBCWvObatJWeUBRh7Jpj4tDzOiSuAEmHJOrEGsIZ3J0Lb7kdm/aCyOG8lAuvvijlcxOzYhtA shNp+1MaB0FJE01J8mbhuDgL3oLcnO68Qz1/grhrDM4wDtuQd2mu9BNl55xA8KAOCnYeDq8+fqekp tKDS2dqWw/zd8OyVhEa20V+8IIo41f7Sf/ZDqwBidfGuQTx0LajpFZk4IoL/79X+i+A1dBh+tGiTs phse/7L1J7Ix3KWanYKq9u3/rnuI3JMwhIBA1kK5e7hmhpSFH9g4jlaOdXlDUCHB+3Yn/YbloPeW4 cojLyu+Q==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1t0KD9-000000055FS-3k9R; Mon, 14 Oct 2024 12:30:07 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1t0Ipj-00000004oaH-2PCr for linux-arm-kernel@lists.infradead.org; Mon, 14 Oct 2024 11:01:53 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id B1DC6176B; Mon, 14 Oct 2024 04:02:20 -0700 (PDT) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.27]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id B31C53F51B; Mon, 14 Oct 2024 04:01:48 -0700 (PDT) From: Ryan Roberts To: Andrew Morton , Anshuman Khandual , Ard Biesheuvel , Catalin Marinas , David Hildenbrand , Greg Marsden , Ivan Ivanov , Kalesh Singh , Marc Zyngier , Mark Rutland , Matthias Brugger , Miroslav Benes , Will Deacon Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [RFC PATCH v1 47/57] arm64: Statically allocate and align for worst-case page size Date: Mon, 14 Oct 2024 11:58:54 +0100 Message-ID: <20241014105912.3207374-47-ryan.roberts@arm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20241014105912.3207374-1-ryan.roberts@arm.com> References: <20241014105514.3206191-1-ryan.roberts@arm.com> <20241014105912.3207374-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241014_040151_727676_1098DA2D X-CRM114-Status: GOOD ( 12.18 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Increase the size and alignment of the zero page and various static buffers used for page tables to PAGE_SIZE_MAX. This resolves to the same thing for compile-time page size builds. For boot-time builds, we may in future consider freeing unused pages at runtime when the selected page size is less than MAX. Signed-off-by: Ryan Roberts --- ***NOTE*** Any confused maintainers may want to read the cover note here for context: https://lore.kernel.org/all/20241014105514.3206191-1-ryan.roberts@arm.com/ arch/arm64/include/asm/pgtable.h | 2 +- arch/arm64/kernel/pi/map_kernel.c | 2 +- arch/arm64/mm/mmu.c | 6 +++--- 3 files changed, 5 insertions(+), 5 deletions(-) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 7a4f5604be3f7..fd47f70a42396 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -61,7 +61,7 @@ * ZERO_PAGE is a global shared page that is always zero: used * for zero-mapped memory areas etc.. */ -extern unsigned long empty_zero_page[PAGE_SIZE / sizeof(unsigned long)]; +extern unsigned long empty_zero_page[PAGE_SIZE_MAX / sizeof(unsigned long)]; #define ZERO_PAGE(vaddr) phys_to_page(__pa_symbol(empty_zero_page)) #define pte_ERROR(e) \ diff --git a/arch/arm64/kernel/pi/map_kernel.c b/arch/arm64/kernel/pi/map_kernel.c index 7a62d4238449d..deb8cd50b0b0c 100644 --- a/arch/arm64/kernel/pi/map_kernel.c +++ b/arch/arm64/kernel/pi/map_kernel.c @@ -199,7 +199,7 @@ static void __init remap_idmap(bool use_lpa2, int page_shift) static void __init map_fdt(u64 fdt, int page_shift) { - static u8 ptes[INIT_IDMAP_FDT_SIZE_MAX] __initdata __aligned(PAGE_SIZE); + static u8 ptes[INIT_IDMAP_FDT_SIZE_MAX] __initdata __aligned(PAGE_SIZE_MAX); static bool first_time __initdata = true; u64 limit = (u64)&ptes[INIT_IDMAP_FDT_SIZE_MAX]; u64 efdt = fdt + MAX_FDT_SIZE; diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index 84df9f278d24d..b4cd3b6a73c22 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -62,7 +62,7 @@ long __section(".mmuoff.data.write") __early_cpu_boot_status; * Empty_zero_page is a special page that is used for zero-initialized data * and COW. */ -unsigned long empty_zero_page[PAGE_SIZE / sizeof(unsigned long)] __page_aligned_bss; +unsigned long empty_zero_page[PAGE_SIZE_MAX / sizeof(unsigned long)] __page_aligned_bss; EXPORT_SYMBOL(empty_zero_page); static DEFINE_SPINLOCK(swapper_pgdir_lock); @@ -783,8 +783,8 @@ void __pi_map_range(u64 *pgd, u64 limit, u64 start, u64 end, u64 pa, pgprot_t prot, int level, pte_t *tbl, bool may_use_cont, u64 va_offset); -static u8 idmap_ptes[IDMAP_LEVELS_MAX - 1][PAGE_SIZE] __aligned(PAGE_SIZE) __ro_after_init, - kpti_ptes[IDMAP_LEVELS_MAX - 1][PAGE_SIZE] __aligned(PAGE_SIZE) __ro_after_init; +static u8 idmap_ptes[IDMAP_LEVELS_MAX - 1][PAGE_SIZE_MAX] __aligned(PAGE_SIZE_MAX) __ro_after_init, + kpti_ptes[IDMAP_LEVELS_MAX - 1][PAGE_SIZE_MAX] __aligned(PAGE_SIZE_MAX) __ro_after_init; static void __init create_idmap(void) { From patchwork Mon Oct 14 10:58:55 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13834924 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 2A619D16256 for ; Mon, 14 Oct 2024 12:31:46 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=1Pv7dW4G5U3czwTGJgTczogiNPWDVKQoCd/L9alqD4s=; b=cTpnYKdt7sU4/q3+W/MfO8vuVg 1hF7vO5NdhLf/MMt5T2Yo/Jp4YJDB4XqAyOIswesQdw5b/DKUfgCmgij5SwiBkLmk8bCnslOqMJhD QodT3QJcJxguEpaZdY5eRQnuA5J9rqboBiS/8jCoWMBAYn3sFB9nLhYP9n7oIH2l6+ofQcjD268Lh WvsclfBNrBh4DvB7Ra42aZkV9xpGJBHRJSMfcHBQ+U3hxZqqUvm0CNFN0EMjHTV+9cgQ7jhJEHFsm qxJdxRTwQEXd36fftWdFAXT97cREu4coVkgVMYdvH2UySj8IwoUdpk3IkPCmpz4Ye8Ixuxfzo3fTa FhKKmolQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1t0KEX-000000055ao-2ux6; Mon, 14 Oct 2024 12:31:33 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1t0Ipm-00000004oc2-1eFQ for linux-arm-kernel@lists.infradead.org; Mon, 14 Oct 2024 11:01:55 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id A0064176C; Mon, 14 Oct 2024 04:02:23 -0700 (PDT) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.27]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 62FEC3F51B; Mon, 14 Oct 2024 04:01:51 -0700 (PDT) From: Ryan Roberts To: Andrew Morton , Anshuman Khandual , Ard Biesheuvel , Catalin Marinas , David Hildenbrand , Greg Marsden , Ivan Ivanov , Kalesh Singh , Marc Zyngier , Mark Rutland , Matthias Brugger , Miroslav Benes , Oliver Upton , Will Deacon Cc: Ryan Roberts , kvmarm@lists.linux.dev, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [RFC PATCH v1 48/57] arm64: Convert switch to if for non-const comparison values Date: Mon, 14 Oct 2024 11:58:55 +0100 Message-ID: <20241014105912.3207374-48-ryan.roberts@arm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20241014105912.3207374-1-ryan.roberts@arm.com> References: <20241014105514.3206191-1-ryan.roberts@arm.com> <20241014105912.3207374-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241014_040154_580194_A8BF12C6 X-CRM114-Status: GOOD ( 12.63 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org When we enable boot-time page size, some macros are no longer compile-time constants. Where these macros are used as cases in switch statements, the switch statements no longer compile. Let's convert these to if/else blocks, which can handle the runtime values. Signed-off-by: Ryan Roberts --- ***NOTE*** Any confused maintainers may want to read the cover note here for context: https://lore.kernel.org/all/20241014105514.3206191-1-ryan.roberts@arm.com/ arch/arm64/kvm/mmu.c | 32 +++++++++++++++----------------- arch/arm64/mm/hugetlbpage.c | 34 +++++++++++----------------------- 2 files changed, 26 insertions(+), 40 deletions(-) diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c index a509b63bd4dd5..248a2d7ad6dbb 100644 --- a/arch/arm64/kvm/mmu.c +++ b/arch/arm64/kvm/mmu.c @@ -1487,29 +1487,27 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa, vma_shift = get_vma_page_shift(vma, hva); } - switch (vma_shift) { #ifndef __PAGETABLE_PMD_FOLDED - case PUD_SHIFT: - if (fault_supports_stage2_huge_mapping(memslot, hva, PUD_SIZE)) - break; - fallthrough; + if (vma_shift == PUD_SHIFT) { + if (!fault_supports_stage2_huge_mapping(memslot, hva, PUD_SIZE)) + vma_shift = PMD_SHIFT; + } #endif - case CONT_PMD_SHIFT: + if (vma_shift == CONT_PMD_SHIFT) { vma_shift = PMD_SHIFT; - fallthrough; - case PMD_SHIFT: - if (fault_supports_stage2_huge_mapping(memslot, hva, PMD_SIZE)) - break; - fallthrough; - case CONT_PTE_SHIFT: + } + if (vma_shift == PMD_SHIFT) { + if (!fault_supports_stage2_huge_mapping(memslot, hva, PMD_SIZE)) + vma_shift = PAGE_SHIFT; + } + if (vma_shift == CONT_PTE_SHIFT) { vma_shift = PAGE_SHIFT; force_pte = true; - fallthrough; - case PAGE_SHIFT: - break; - default: - WARN_ONCE(1, "Unknown vma_shift %d", vma_shift); } + if (vma_shift != PUD_SHIFT && + vma_shift != PMD_SHIFT && + vma_shift != PAGE_SHIFT) + WARN_ONCE(1, "Unknown vma_shift %d", vma_shift); vma_pagesize = 1UL << vma_shift; diff --git a/arch/arm64/mm/hugetlbpage.c b/arch/arm64/mm/hugetlbpage.c index 5f1e2103888b7..bc98c20655bba 100644 --- a/arch/arm64/mm/hugetlbpage.c +++ b/arch/arm64/mm/hugetlbpage.c @@ -51,16 +51,12 @@ void __init arm64_hugetlb_cma_reserve(void) static bool __hugetlb_valid_size(unsigned long size) { - switch (size) { #ifndef __PAGETABLE_PMD_FOLDED - case PUD_SIZE: + if (size == PUD_SIZE) return pud_sect_supported(); #endif - case CONT_PMD_SIZE: - case PMD_SIZE: - case CONT_PTE_SIZE: + if (size == CONT_PMD_SIZE || size == PMD_SIZE || size == CONT_PTE_SIZE) return true; - } return false; } @@ -104,24 +100,20 @@ static inline int num_contig_ptes(unsigned long size, size_t *pgsize) *pgsize = size; - switch (size) { #ifndef __PAGETABLE_PMD_FOLDED - case PUD_SIZE: + if (size == PUD_SIZE) { if (pud_sect_supported()) contig_ptes = 1; - break; + } else #endif - case PMD_SIZE: + if (size == PMD_SIZE) { contig_ptes = 1; - break; - case CONT_PMD_SIZE: + } else if (size == CONT_PMD_SIZE) { *pgsize = PMD_SIZE; contig_ptes = CONT_PMDS; - break; - case CONT_PTE_SIZE: + } else if (size == CONT_PTE_SIZE) { *pgsize = PAGE_SIZE; contig_ptes = CONT_PTES; - break; } return contig_ptes; @@ -339,20 +331,16 @@ unsigned long hugetlb_mask_last_page(struct hstate *h) { unsigned long hp_size = huge_page_size(h); - switch (hp_size) { #ifndef __PAGETABLE_PMD_FOLDED - case PUD_SIZE: + if (hp_size == PUD_SIZE) return PGDIR_SIZE - PUD_SIZE; #endif - case CONT_PMD_SIZE: + if (hp_size == CONT_PMD_SIZE) return PUD_SIZE - CONT_PMD_SIZE; - case PMD_SIZE: + if (hp_size == PMD_SIZE) return PUD_SIZE - PMD_SIZE; - case CONT_PTE_SIZE: + if (hp_size == CONT_PTE_SIZE) return PMD_SIZE - CONT_PTE_SIZE; - default: - break; - } return 0UL; } From patchwork Mon Oct 14 10:58:56 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13834931 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 2EEAFD16256 for ; Mon, 14 Oct 2024 12:33:08 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=97p+R0dspaRUKJqMA/Y3wP7vnE/E8FcpBV0RwE1dMZY=; b=wVyXAcBKuAl/0RcdokGwjs5IS+ uYUXaocKhiT4wiRObMQf/Wan32br/l+++rUDLCx1tBAC3gQ9QJRJ/VWf0xqzgAMmVSD4P4j6C9YnX VwsFLWsRznRz4L+ksGbOq0a+xuAoMidl+GmQLA3VrXBiTNciCn1uH36ISVgEggMVtu3qOMc88g0lD viwb2aGPw9XpDwIoIz1Hyn/rWPFRiGN/xtZuuzQXkAE+JIV2RMA3jLRCCaNFCaNrJw/AvQWrNvPtG 6aWElz6mhQ7KmWjLxVdaK/38ICMxscYDu7YYYfIzrva7z60iPg2ZO+FrahTwX+D0WQOdZGmp88SJV nlIGwsrg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1t0KFt-000000055oo-3CPd; Mon, 14 Oct 2024 12:32:57 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1t0Ipp-00000004od8-06R9 for linux-arm-kernel@lists.infradead.org; Mon, 14 Oct 2024 11:01:58 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 4FFED19F0; Mon, 14 Oct 2024 04:02:26 -0700 (PDT) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.27]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 512493F51B; Mon, 14 Oct 2024 04:01:54 -0700 (PDT) From: Ryan Roberts To: Andrew Morton , Anshuman Khandual , Ard Biesheuvel , Catalin Marinas , David Hildenbrand , Greg Marsden , Ivan Ivanov , Kalesh Singh , Marc Zyngier , Mark Rutland , Matthias Brugger , Miroslav Benes , Will Deacon Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [RFC PATCH v1 49/57] arm64: Convert BUILD_BUG_ON to VM_BUG_ON Date: Mon, 14 Oct 2024 11:58:56 +0100 Message-ID: <20241014105912.3207374-49-ryan.roberts@arm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20241014105912.3207374-1-ryan.roberts@arm.com> References: <20241014105514.3206191-1-ryan.roberts@arm.com> <20241014105912.3207374-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241014_040157_177235_639781C5 X-CRM114-Status: GOOD ( 14.28 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org There are some build bug checks that will no longer compile for boot-time page size because the values they are testing are no longer compile-time constants. Resolve these by converting them to VM_BUG_ON, which will perform a runtime check. Signed-off-by: Ryan Roberts --- ***NOTE*** Any confused maintainers may want to read the cover note here for context: https://lore.kernel.org/all/20241014105514.3206191-1-ryan.roberts@arm.com/ arch/arm64/mm/init.c | 6 +++--- arch/arm64/mm/mmu.c | 4 ++-- arch/arm64/mm/pgd.c | 2 +- 3 files changed, 6 insertions(+), 6 deletions(-) diff --git a/arch/arm64/mm/init.c b/arch/arm64/mm/init.c index 42eb246949072..4d24034418b39 100644 --- a/arch/arm64/mm/init.c +++ b/arch/arm64/mm/init.c @@ -388,15 +388,15 @@ void __init mem_init(void) * detected at build time already. */ #ifdef CONFIG_COMPAT - BUILD_BUG_ON(TASK_SIZE_32 > DEFAULT_MAP_WINDOW_64); + VM_BUG_ON(TASK_SIZE_32 > DEFAULT_MAP_WINDOW_64); #endif /* * Selected page table levels should match when derived from * scratch using the virtual address range and page size. */ - BUILD_BUG_ON(ARM64_HW_PGTABLE_LEVELS(CONFIG_ARM64_VA_BITS) != - CONFIG_PGTABLE_LEVELS); + VM_BUG_ON(ARM64_HW_PGTABLE_LEVELS(CONFIG_ARM64_VA_BITS) != + CONFIG_PGTABLE_LEVELS); if (PAGE_SIZE >= 16384 && get_num_physpages() <= 128) { extern int sysctl_overcommit_memory; diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index b4cd3b6a73c22..ad7fd3fda705a 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -639,8 +639,8 @@ static void __init map_mem(pgd_t *pgdp) * entire reduced VA space is covered by a single pgd_t which will have * been populated without the PXNTable attribute by the time we get here.) */ - BUILD_BUG_ON(pgd_index(direct_map_end - 1) == pgd_index(direct_map_end) && - pgd_index(_PAGE_OFFSET(VA_BITS_MIN)) != PTRS_PER_PGD - 1); + VM_BUG_ON(pgd_index(direct_map_end - 1) == pgd_index(direct_map_end) && + pgd_index(_PAGE_OFFSET(VA_BITS_MIN)) != PTRS_PER_PGD - 1); early_kfence_pool = arm64_kfence_alloc_pool(); diff --git a/arch/arm64/mm/pgd.c b/arch/arm64/mm/pgd.c index 0c501cabc2384..4b106510358b1 100644 --- a/arch/arm64/mm/pgd.c +++ b/arch/arm64/mm/pgd.c @@ -56,7 +56,7 @@ void __init pgtable_cache_init(void) * With 52-bit physical addresses, the architecture requires the * top-level table to be aligned to at least 64 bytes. */ - BUILD_BUG_ON(PGD_SIZE < 64); + VM_BUG_ON(PGD_SIZE < 64); #endif /* From patchwork Mon Oct 14 10:58:57 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13834932 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 14A86D16258 for ; Mon, 14 Oct 2024 12:34:33 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=1XPkM6wVUNtvQ6sRxRh3jqob0EiliLUCc+oDCpziCPI=; b=o1D6XhR5mA6WWxMGooOZwquKq/ fbUqeXipCfdQDRn8dq0Wk+t0npm35IUSjxu5IzRT/tExuWIqvUZnzWD2E5OuDGLaPurFVIErzAfb6 BQv5TGVQLON9W1KO0vOezckWoGKYEHS8GK2ng6jNgssI3xrLbLSMO9NrOPOE/J2mtPDJ4fsxHfWot A8Y8MLNA2HE/imWkuDmJ8yagvywL6b7/RkevZItRs6mAwj1MNLlP7sGmX7gz4d3oq4LnOTe3B2FQ2 P4HekFSx0f1rzZuO9e7ztWPsvXxNdGM168DVlX+RXK3EM3En8GBz+5hAdNy9MlF5fpLnXP+EkQhqA gASGTfnA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1t0KHF-000000055x1-1Wht; Mon, 14 Oct 2024 12:34:21 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1t0Ipr-00000004ofE-38mQ for linux-arm-kernel@lists.infradead.org; Mon, 14 Oct 2024 11:02:02 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 0495C1A00; Mon, 14 Oct 2024 04:02:29 -0700 (PDT) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.27]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 0154B3F51B; Mon, 14 Oct 2024 04:01:56 -0700 (PDT) From: Ryan Roberts To: Andrew Morton , Anshuman Khandual , Ard Biesheuvel , Catalin Marinas , David Hildenbrand , Greg Marsden , Ivan Ivanov , Kalesh Singh , Marc Zyngier , Mark Rutland , Matthias Brugger , Miroslav Benes , Will Deacon Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [RFC PATCH v1 50/57] arm64: Remove PAGE_SZ asm-offset Date: Mon, 14 Oct 2024 11:58:57 +0100 Message-ID: <20241014105912.3207374-50-ryan.roberts@arm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20241014105912.3207374-1-ryan.roberts@arm.com> References: <20241014105514.3206191-1-ryan.roberts@arm.com> <20241014105912.3207374-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241014_040159_917303_D5F23E57 X-CRM114-Status: UNSURE ( 9.26 ) X-CRM114-Notice: Please train this message. X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org PAGE_SZ is not used anywhere and for boot-time builds, where it is not a compile-time constant, we don't know the value anyway. So let's remove it. Signed-off-by: Ryan Roberts --- ***NOTE*** Any confused maintainers may want to read the cover note here for context: https://lore.kernel.org/all/20241014105514.3206191-1-ryan.roberts@arm.com/ arch/arm64/kernel/asm-offsets.c | 2 -- 1 file changed, 2 deletions(-) diff --git a/arch/arm64/kernel/asm-offsets.c b/arch/arm64/kernel/asm-offsets.c index 27de1dddb0abe..f32b8d7f00b2a 100644 --- a/arch/arm64/kernel/asm-offsets.c +++ b/arch/arm64/kernel/asm-offsets.c @@ -111,8 +111,6 @@ int main(void) BLANK(); DEFINE(VM_EXEC, VM_EXEC); BLANK(); - DEFINE(PAGE_SZ, PAGE_SIZE); - BLANK(); DEFINE(DMA_TO_DEVICE, DMA_TO_DEVICE); DEFINE(DMA_FROM_DEVICE, DMA_FROM_DEVICE); BLANK(); From patchwork Mon Oct 14 10:58:58 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13834933 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id C5515D16258 for ; Mon, 14 Oct 2024 12:35:58 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=VfAcTUi91ptQaK/CGe1CeXA9aaPE8rozFIbYUTlYxXU=; b=hi/zZWyBJSfhstyKYdMyZTwlsL 5bLX8suflNTvmnYAhu0di2Gbs+P149bAusbBWCueKpGQwLBa5EVZeamMzTJOw4gZz3JRh3aYHRtJJ BCYnrk5FaFDeZvIrIs3YPvDplxTbkzdmAg90IbpopDzmf1EarTF7IfG9jdAyEJabGHN6EwCjEy1Zh P849ETInyJSEwcabNXobyPyMqd8GDhqEA2NO2LjEpl6JeD3sLNvI95URPtl8rMfoIYj8T8pKu8OuM dZiHxTUiuiOdch2LT7/Q2hvkGmZ7HZfKR4nF9ectZXYzvduVw9rqbPGWp7sWU4ST8IAWi2En9j9xY AZHeCaow==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1t0KIe-000000056H5-3Kl4; Mon, 14 Oct 2024 12:35:48 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1t0Ipu-00000004ogb-2JYN for linux-arm-kernel@lists.infradead.org; Mon, 14 Oct 2024 11:02:04 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id A49A51A25; Mon, 14 Oct 2024 04:02:31 -0700 (PDT) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.27]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id A5B033F51B; Mon, 14 Oct 2024 04:01:59 -0700 (PDT) From: Ryan Roberts To: Andrew Morton , Anshuman Khandual , Ard Biesheuvel , Catalin Marinas , David Hildenbrand , Greg Marsden , Ivan Ivanov , Kalesh Singh , Marc Zyngier , Mark Rutland , Matthias Brugger , Miroslav Benes , Will Deacon Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [RFC PATCH v1 51/57] arm64: Introduce cpu features for page sizes Date: Mon, 14 Oct 2024 11:58:58 +0100 Message-ID: <20241014105912.3207374-51-ryan.roberts@arm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20241014105912.3207374-1-ryan.roberts@arm.com> References: <20241014105514.3206191-1-ryan.roberts@arm.com> <20241014105912.3207374-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241014_040202_656580_2AAC26DC X-CRM114-Status: GOOD ( 13.20 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Introduce one boot cpu feature per page size, ARM64_USE_PAGE_SIZE_*K. These will enable use of the alternatives framework to patch code based on the page size selected at boot. Additionally, this provides a neat way to figure out what page size is in use during boot from the kernel log. Signed-off-by: Ryan Roberts --- ***NOTE*** Any confused maintainers may want to read the cover note here for context: https://lore.kernel.org/all/20241014105514.3206191-1-ryan.roberts@arm.com/ arch/arm64/kernel/cpufeature.c | 25 +++++++++++++++++++++++++ arch/arm64/tools/cpucaps | 3 +++ 2 files changed, 28 insertions(+) diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c index 7705c9c0e7142..e5618423bb99d 100644 --- a/arch/arm64/kernel/cpufeature.c +++ b/arch/arm64/kernel/cpufeature.c @@ -1831,6 +1831,13 @@ static bool has_nv1(const struct arm64_cpu_capabilities *entry, int scope) is_midr_in_range_list(read_cpuid_id(), nv1_ni_list))); } +static bool use_page_size(const struct arm64_cpu_capabilities *entry, int scope) +{ + return (entry->capability == ARM64_USE_PAGE_SIZE_4K && PAGE_SIZE == SZ_4K) || + (entry->capability == ARM64_USE_PAGE_SIZE_16K && PAGE_SIZE == SZ_16K) || + (entry->capability == ARM64_USE_PAGE_SIZE_64K && PAGE_SIZE == SZ_64K); +} + static bool has_lpa2_at_stage1(u64 mmfr0) { unsigned int tgran; @@ -2881,6 +2888,24 @@ static const struct arm64_cpu_capabilities arm64_features[] = { .matches = has_nv1, ARM64_CPUID_FIELDS_NEG(ID_AA64MMFR4_EL1, E2H0, NI_NV1) }, + { + .desc = "4K page size", + .capability = ARM64_USE_PAGE_SIZE_4K, + .type = ARM64_CPUCAP_BOOT_CPU_FEATURE, + .matches = use_page_size, + }, + { + .desc = "16K page size", + .capability = ARM64_USE_PAGE_SIZE_16K, + .type = ARM64_CPUCAP_BOOT_CPU_FEATURE, + .matches = use_page_size, + }, + { + .desc = "64K page size", + .capability = ARM64_USE_PAGE_SIZE_64K, + .type = ARM64_CPUCAP_BOOT_CPU_FEATURE, + .matches = use_page_size, + }, {}, }; diff --git a/arch/arm64/tools/cpucaps b/arch/arm64/tools/cpucaps index ac3429d892b9a..5cb75675303c6 100644 --- a/arch/arm64/tools/cpucaps +++ b/arch/arm64/tools/cpucaps @@ -71,6 +71,9 @@ SPECTRE_BHB SSBS SVE UNMAP_KERNEL_AT_EL0 +USE_PAGE_SIZE_4K +USE_PAGE_SIZE_16K +USE_PAGE_SIZE_64K WORKAROUND_834220 WORKAROUND_843419 WORKAROUND_845719 From patchwork Mon Oct 14 10:58:59 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13834934 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 6502AD16258 for ; Mon, 14 Oct 2024 12:37:26 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=CYEBM3jqbad+bofavVbJlOnhYtw41610magjm74TJ1s=; b=TVtw781geZ0t1ylNpx0+/TzR13 HLuVQELVp+rDas22Zq783Iht12RRie78gEVhcjwXiF7qjOAnEz5AVFEBqFtoBOKZs2IHp6IxugCFT +t0R5V7MWmpUHxBBuNzfJC95eqR/jpTLswf7vE7Xzl4Uqo59cI8ib0eJs0PvOrxDiNCzea+zRoPEO 0d+YNVUvI0i29t0JSUk0eC+2nPdd0ZlPUeUAL6fvokTW9h8jXGNtV7Rw9Jbg/k9+qFdqz3ue25I4z s4ffk+mPVLGW98hHBF9+6Lhs7xnnW6JelK4Num5DuepeHgTbznCTQR9AQowqz7UvGHHffmCTv9BzX frq8KpxQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1t0KK0-000000056QJ-0RtN; Mon, 14 Oct 2024 12:37:12 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1t0Ipx-00000004ohe-1EHv for linux-arm-kernel@lists.infradead.org; Mon, 14 Oct 2024 11:02:07 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 926581A2D; Mon, 14 Oct 2024 04:02:34 -0700 (PDT) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.27]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 55EA53F51B; Mon, 14 Oct 2024 04:02:02 -0700 (PDT) From: Ryan Roberts To: Andrew Morton , Anshuman Khandual , Ard Biesheuvel , Catalin Marinas , David Hildenbrand , Greg Marsden , Ivan Ivanov , Kalesh Singh , Marc Zyngier , Mark Rutland , Matthias Brugger , Miroslav Benes , Oliver Upton , Will Deacon Cc: Ryan Roberts , kvmarm@lists.linux.dev, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [RFC PATCH v1 52/57] arm64: Remove PAGE_SIZE from assembly code Date: Mon, 14 Oct 2024 11:58:59 +0100 Message-ID: <20241014105912.3207374-52-ryan.roberts@arm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20241014105912.3207374-1-ryan.roberts@arm.com> References: <20241014105514.3206191-1-ryan.roberts@arm.com> <20241014105912.3207374-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241014_040205_456076_5907DF78 X-CRM114-Status: GOOD ( 20.64 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Remove usage of PAGE_SHIFT, PAGE_SIZE and PAGE_MASK macros from assembly code since these are no longer compile-time constants when boot-time page size is in use. For the most part, they are replaced with run-time lookups based on the value of TG0. This is done outside of loops so while there is a cost of a few extra instructions, performance should not be impacted. However, invalid_host_el2_vect requires that the page shift be an immediate since it has no registers to spare. So for this, let's use alternatives patching. This code is guarranteed not to run until after patching is complete. __pi_copy_page has no registers to spare to hold the page size, and we want to avoid having to reload it on every iteration of the loop. Since I couldn't provably conclude that the function is not called prior to alternatives patching, I opted to make a copy of the function for each page size and branch to the right one at the start. Signed-off-by: Ryan Roberts --- ***NOTE*** Any confused maintainers may want to read the cover note here for context: https://lore.kernel.org/all/20241014105514.3206191-1-ryan.roberts@arm.com/ arch/arm64/include/asm/assembler.h | 18 +++++++++++++--- arch/arm64/kernel/hibernate-asm.S | 6 ++++-- arch/arm64/kernel/relocate_kernel.S | 10 ++++++--- arch/arm64/kvm/hyp/nvhe/host.S | 10 ++++++++- arch/arm64/lib/clear_page.S | 7 ++++-- arch/arm64/lib/copy_page.S | 33 +++++++++++++++++++++-------- arch/arm64/lib/mte.S | 27 +++++++++++++++++------ 7 files changed, 85 insertions(+), 26 deletions(-) diff --git a/arch/arm64/include/asm/assembler.h b/arch/arm64/include/asm/assembler.h index 77c2d707adb1a..6424fd6be1cbe 100644 --- a/arch/arm64/include/asm/assembler.h +++ b/arch/arm64/include/asm/assembler.h @@ -495,9 +495,11 @@ alternative_endif .Lskip_\@: .endm /* - * copy_page - copy src to dest using temp registers t1-t8 + * copy_page - copy src to dest using temp registers t1-t9 */ - .macro copy_page dest:req src:req t1:req t2:req t3:req t4:req t5:req t6:req t7:req t8:req + .macro copy_page dest:req src:req t1:req t2:req t3:req t4:req t5:req t6:req t7:req t8:req t9:req + get_page_size \t9 + sub \t9, \t9, #1 // (PAGE_SIZE - 1) in \t9 9998: ldp \t1, \t2, [\src] ldp \t3, \t4, [\src, #16] ldp \t5, \t6, [\src, #32] @@ -508,7 +510,7 @@ alternative_endif stnp \t5, \t6, [\dest, #32] stnp \t7, \t8, [\dest, #48] add \dest, \dest, #64 - tst \src, #(PAGE_SIZE - 1) + tst \src, \t9 b.ne 9998b .endm @@ -911,4 +913,14 @@ alternative_cb_end .macro tgran_lpa2, val, tg0 value_for_page_size \val, \tg0, ID_AA64MMFR0_EL1_TGRAN4_52_BIT, ID_AA64MMFR0_EL1_TGRAN16_52_BIT, -1 .endm + + .macro get_page_size, val + get_tg0 \val + value_for_page_size \val, \val, SZ_4K, SZ_16K, SZ_64K + .endm + + .macro get_page_mask, val + get_tg0 \val + value_for_page_size \val, \val, (~(SZ_4K-1)), (~(SZ_16K-1)), (~(SZ_64K-1)) + .endm #endif /* __ASM_ASSEMBLER_H */ diff --git a/arch/arm64/kernel/hibernate-asm.S b/arch/arm64/kernel/hibernate-asm.S index 0e1d9c3c6a933..375b2fcf82e84 100644 --- a/arch/arm64/kernel/hibernate-asm.S +++ b/arch/arm64/kernel/hibernate-asm.S @@ -57,6 +57,8 @@ SYM_CODE_START(swsusp_arch_suspend_exit) mov x24, x4 mov x25, x5 + get_page_size x12 + /* walk the restore_pblist and use copy_page() to over-write memory */ mov x19, x3 @@ -64,9 +66,9 @@ SYM_CODE_START(swsusp_arch_suspend_exit) mov x0, x10 ldr x1, [x19, #HIBERN_PBE_ADDR] - copy_page x0, x1, x2, x3, x4, x5, x6, x7, x8, x9 + copy_page x0, x1, x2, x3, x4, x5, x6, x7, x8, x9, x11 - add x1, x10, #PAGE_SIZE + add x1, x10, x12 /* Clean the copied page to PoU - based on caches_clean_inval_pou() */ raw_dcache_line_size x2, x3 sub x3, x2, #1 diff --git a/arch/arm64/kernel/relocate_kernel.S b/arch/arm64/kernel/relocate_kernel.S index 413f899e4ac63..bc4f37fba6c74 100644 --- a/arch/arm64/kernel/relocate_kernel.S +++ b/arch/arm64/kernel/relocate_kernel.S @@ -46,6 +46,10 @@ SYM_CODE_START(arm64_relocate_new_kernel) ldr x27, [x0, #KIMAGE_ARCH_EL2_VECTORS] ldr x26, [x0, #KIMAGE_ARCH_DTB_MEM] + /* Grab page size values. */ + get_page_size x10 /* x10 = PAGE_SIZE */ + get_page_mask x11 /* x11 = PAGE_MASK */ + /* Setup the list loop variables. */ ldr x18, [x0, #KIMAGE_ARCH_ZERO_PAGE] /* x18 = zero page for BBM */ ldr x17, [x0, #KIMAGE_ARCH_TTBR1] /* x17 = linear map copy */ @@ -54,7 +58,7 @@ SYM_CODE_START(arm64_relocate_new_kernel) raw_dcache_line_size x15, x1 /* x15 = dcache line size */ break_before_make_ttbr_switch x18, x17, x1, x2 /* set linear map */ .Lloop: - and x12, x16, PAGE_MASK /* x12 = addr */ + and x12, x16, x11 /* x12 = addr */ sub x12, x12, x22 /* Convert x12 to virt */ /* Test the entry flags. */ .Ltest_source: @@ -62,8 +66,8 @@ SYM_CODE_START(arm64_relocate_new_kernel) /* Invalidate dest page to PoC. */ mov x19, x13 - copy_page x13, x12, x1, x2, x3, x4, x5, x6, x7, x8 - add x1, x19, #PAGE_SIZE + copy_page x13, x12, x1, x2, x3, x4, x5, x6, x7, x8, x9 + add x1, x19, x10 dcache_by_myline_op civac, sy, x19, x1, x15, x20 b .Lnext .Ltest_indirection: diff --git a/arch/arm64/kvm/hyp/nvhe/host.S b/arch/arm64/kvm/hyp/nvhe/host.S index 3d610fc51f4d3..2b0d583fcf1af 100644 --- a/arch/arm64/kvm/hyp/nvhe/host.S +++ b/arch/arm64/kvm/hyp/nvhe/host.S @@ -193,7 +193,15 @@ SYM_FUNC_END(__host_hvc) */ add sp, sp, x0 // sp' = sp + x0 sub x0, sp, x0 // x0' = sp' - x0 = (sp + x0) - x0 = sp - tbz x0, #PAGE_SHIFT, .L__hyp_sp_overflow\@ +alternative_if ARM64_USE_PAGE_SIZE_4K + tbz x0, #ARM64_PAGE_SHIFT_4K, .L__hyp_sp_overflow\@ +alternative_else_nop_endif +alternative_if ARM64_USE_PAGE_SIZE_16K + tbz x0, #ARM64_PAGE_SHIFT_16K, .L__hyp_sp_overflow\@ +alternative_else_nop_endif +alternative_if ARM64_USE_PAGE_SIZE_64K + tbz x0, #ARM64_PAGE_SHIFT_64K, .L__hyp_sp_overflow\@ +alternative_else_nop_endif sub x0, sp, x0 // x0'' = sp' - x0' = (sp + x0) - sp = x0 sub sp, sp, x0 // sp'' = sp' - x0 = (sp + x0) - x0 = sp diff --git a/arch/arm64/lib/clear_page.S b/arch/arm64/lib/clear_page.S index ebde40e7fa2b2..b6f2cb8d704cc 100644 --- a/arch/arm64/lib/clear_page.S +++ b/arch/arm64/lib/clear_page.S @@ -15,6 +15,9 @@ * x0 - dest */ SYM_FUNC_START(__pi_clear_page) + get_page_size x3 + sub x3, x3, #1 /* (PAGE_SIZE - 1) in x3 */ + mrs x1, dczid_el0 tbnz x1, #4, 2f /* Branch if DC ZVA is prohibited */ and w1, w1, #0xf @@ -23,7 +26,7 @@ SYM_FUNC_START(__pi_clear_page) 1: dc zva, x0 add x0, x0, x1 - tst x0, #(PAGE_SIZE - 1) + tst x0, x3 b.ne 1b ret @@ -32,7 +35,7 @@ SYM_FUNC_START(__pi_clear_page) stnp xzr, xzr, [x0, #32] stnp xzr, xzr, [x0, #48] add x0, x0, #64 - tst x0, #(PAGE_SIZE - 1) + tst x0, x3 b.ne 2b ret SYM_FUNC_END(__pi_clear_page) diff --git a/arch/arm64/lib/copy_page.S b/arch/arm64/lib/copy_page.S index 6a56d7cf309da..6c19b03ab4d69 100644 --- a/arch/arm64/lib/copy_page.S +++ b/arch/arm64/lib/copy_page.S @@ -10,14 +10,7 @@ #include #include -/* - * Copy a page from src to dest (both are page aligned) - * - * Parameters: - * x0 - dest - * x1 - src - */ -SYM_FUNC_START(__pi_copy_page) + .macro copy_page_body, page_size ldp x2, x3, [x1] ldp x4, x5, [x1, #16] ldp x6, x7, [x1, #32] @@ -30,7 +23,7 @@ SYM_FUNC_START(__pi_copy_page) add x0, x0, #256 add x1, x1, #128 1: - tst x0, #(PAGE_SIZE - 1) + tst x0, #(\page_size - 1) stnp x2, x3, [x0, #-256] ldp x2, x3, [x1] @@ -62,7 +55,29 @@ SYM_FUNC_START(__pi_copy_page) stnp x12, x13, [x0, #80 - 256] stnp x14, x15, [x0, #96 - 256] stnp x16, x17, [x0, #112 - 256] + .endm +/* + * Copy a page from src to dest (both are page aligned) + * + * Parameters: + * x0 - dest + * x1 - src + */ +SYM_FUNC_START(__pi_copy_page) + get_tg0 x2 +.Lsz_64k: + cmp x2, #TCR_TG0_64K + b.ne .Lsz_16k + copy_page_body SZ_64K + ret +.Lsz_16k: + cmp x2, #TCR_TG0_16K + b.ne .Lsz_4k + copy_page_body SZ_16K + ret +.Lsz_4k: + copy_page_body SZ_4K ret SYM_FUNC_END(__pi_copy_page) SYM_FUNC_ALIAS(copy_page, __pi_copy_page) diff --git a/arch/arm64/lib/mte.S b/arch/arm64/lib/mte.S index 5018ac03b6bf3..b4f6f5be0ec79 100644 --- a/arch/arm64/lib/mte.S +++ b/arch/arm64/lib/mte.S @@ -28,10 +28,13 @@ * x0 - address of the page to be cleared */ SYM_FUNC_START(mte_clear_page_tags) + get_page_size x3 + sub x3, x3, #1 // (PAGE_SIZE - 1) in x3 + multitag_transfer_size x1, x2 1: stgm xzr, [x0] add x0, x0, x1 - tst x0, #(PAGE_SIZE - 1) + tst x0, x3 b.ne 1b ret SYM_FUNC_END(mte_clear_page_tags) @@ -43,6 +46,9 @@ SYM_FUNC_END(mte_clear_page_tags) * x0 - address to the beginning of the page */ SYM_FUNC_START(mte_zero_clear_page_tags) + get_page_size x3 + sub x3, x3, #1 // (PAGE_SIZE - 1) in x3 + and x0, x0, #(1 << MTE_TAG_SHIFT) - 1 // clear the tag mrs x1, dczid_el0 tbnz x1, #4, 2f // Branch if DC GZVA is prohibited @@ -52,12 +58,12 @@ SYM_FUNC_START(mte_zero_clear_page_tags) 1: dc gzva, x0 add x0, x0, x1 - tst x0, #(PAGE_SIZE - 1) + tst x0, x3 b.ne 1b ret 2: stz2g x0, [x0], #(MTE_GRANULE_SIZE * 2) - tst x0, #(PAGE_SIZE - 1) + tst x0, x3 b.ne 2b ret SYM_FUNC_END(mte_zero_clear_page_tags) @@ -68,6 +74,9 @@ SYM_FUNC_END(mte_zero_clear_page_tags) * x1 - address of the source page */ SYM_FUNC_START(mte_copy_page_tags) + get_page_size x7 + sub x7, x7, #1 // (PAGE_SIZE - 1) in x7 + mov x2, x0 mov x3, x1 multitag_transfer_size x5, x6 @@ -75,7 +84,7 @@ SYM_FUNC_START(mte_copy_page_tags) stgm x4, [x2] add x2, x2, x5 add x3, x3, x5 - tst x2, #(PAGE_SIZE - 1) + tst x2, x7 b.ne 1b ret SYM_FUNC_END(mte_copy_page_tags) @@ -137,6 +146,9 @@ SYM_FUNC_END(mte_copy_tags_to_user) * x1 - tag storage, MTE_PAGE_TAG_STORAGE bytes */ SYM_FUNC_START(mte_save_page_tags) + get_page_size x3 + sub x3, x3, #1 // (PAGE_SIZE - 1) in x3 + multitag_transfer_size x7, x5 1: mov x2, #0 @@ -149,7 +161,7 @@ SYM_FUNC_START(mte_save_page_tags) str x2, [x1], #8 - tst x0, #(PAGE_SIZE - 1) + tst x0, x3 b.ne 1b ret @@ -161,6 +173,9 @@ SYM_FUNC_END(mte_save_page_tags) * x1 - tag storage, MTE_PAGE_TAG_STORAGE bytes */ SYM_FUNC_START(mte_restore_page_tags) + get_page_size x3 + sub x3, x3, #1 // (PAGE_SIZE - 1) in x3 + multitag_transfer_size x7, x5 1: ldr x2, [x1], #8 @@ -170,7 +185,7 @@ SYM_FUNC_START(mte_restore_page_tags) tst x0, #0xFF b.ne 2b - tst x0, #(PAGE_SIZE - 1) + tst x0, x3 b.ne 1b ret From patchwork Mon Oct 14 10:59:00 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13834935 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id B2274D16258 for ; Mon, 14 Oct 2024 12:38:48 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=/eQV3kabbUGp6wsnBtNJdYTniQk07KbfuLXN2MYutAQ=; b=z+ftkkg3YIC+6+WOfxtHq7IUPo M78xYFHXxBj2QPaxxgqVveIz5rORin3KAI11SegYEk6BvYq3Ld5KY0IkIN0qN/iDvxhOSVW3qLWQC PmlCpK5sLhoxzqZcbeO5PuVPC51mkkIrINFfzdxmxX3n5mA5kiutPr1gfnq2H1CIkFWL2DM70mo08 bzMb+vLwEiz7VYd17sd8x6TnkdjpnbQnXmWAq4P1EU62vjVH2JYeLnhPCRhqvqJjycMrhotEgIgnj RRTKzGR957BpDU6rX+vIruUZTHJjRJLBXotvKO3l1rPXq0CPVV3/nQ+Eo6PXHPuTZSzeAFQsn4dDn vTCDXeqQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1t0KLL-000000056XO-1683; Mon, 14 Oct 2024 12:38:35 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1t0Iq1-00000004ok6-1b6B for linux-arm-kernel@lists.infradead.org; Mon, 14 Oct 2024 11:02:12 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 08F6A1713; Mon, 14 Oct 2024 04:02:38 -0700 (PDT) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.27]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 44CBF3F51B; Mon, 14 Oct 2024 04:02:05 -0700 (PDT) From: Ryan Roberts To: "Aneesh Kumar K.V" , Andrew Morton , Anshuman Khandual , Ard Biesheuvel , Catalin Marinas , David Hildenbrand , Greg Marsden , Ivan Ivanov , Kalesh Singh , Marc Zyngier , Mark Rutland , Matthias Brugger , Miroslav Benes , Nick Piggin , Oliver Upton , Peter Zijlstra , Will Deacon Cc: Ryan Roberts , kvmarm@lists.linux.dev, linux-arch@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [RFC PATCH v1 53/57] arm64: Runtime-fold pmd level Date: Mon, 14 Oct 2024 11:59:00 +0100 Message-ID: <20241014105912.3207374-53-ryan.roberts@arm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20241014105912.3207374-1-ryan.roberts@arm.com> References: <20241014105514.3206191-1-ryan.roberts@arm.com> <20241014105912.3207374-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241014_040209_804942_30830764 X-CRM114-Status: GOOD ( 23.02 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org For a given VA size, the number of levels of lookup depends on the page size. With boot-time page size selection, we therefore don't know how many levels of lookup we require until boot time. So we need to runtime-fold some levels of lookup. We already have code to runtime-fold p4d and pud levels; that exists for LPA2 fallback paths and can be repurposed for our needs. But pmd level also needs to support runtime folding; for example, 16K/36-bit and 64K/42-bit configs require only 2 levels. So let's add the required code. However, note that until we actually add the boot-time page size config, pgtable_l3_enabled() simply returns the compile-time determined answer. Signed-off-by: Ryan Roberts --- ***NOTE*** Any confused maintainers may want to read the cover note here for context: https://lore.kernel.org/all/20241014105514.3206191-1-ryan.roberts@arm.com/ arch/arm64/include/asm/pgalloc.h | 16 +++- arch/arm64/include/asm/pgtable.h | 123 +++++++++++++++++++++++-------- arch/arm64/include/asm/tlb.h | 3 + arch/arm64/kernel/cpufeature.c | 4 +- arch/arm64/kvm/mmu.c | 9 +-- arch/arm64/mm/fixmap.c | 2 +- arch/arm64/mm/hugetlbpage.c | 16 ++-- arch/arm64/mm/init.c | 2 +- arch/arm64/mm/mmu.c | 2 +- arch/arm64/mm/ptdump.c | 3 +- 10 files changed, 126 insertions(+), 54 deletions(-) diff --git a/arch/arm64/include/asm/pgalloc.h b/arch/arm64/include/asm/pgalloc.h index 8ff5f2a2579e4..51cc2f32931d2 100644 --- a/arch/arm64/include/asm/pgalloc.h +++ b/arch/arm64/include/asm/pgalloc.h @@ -15,6 +15,7 @@ #define __HAVE_ARCH_PGD_FREE #define __HAVE_ARCH_PUD_FREE +#define __HAVE_ARCH_PMD_FREE #include #define PGD_SIZE (PTRS_PER_PGD * sizeof(pgd_t)) @@ -23,7 +24,8 @@ static inline void __pud_populate(pud_t *pudp, phys_addr_t pmdp, pudval_t prot) { - set_pud(pudp, __pud(__phys_to_pud_val(pmdp) | prot)); + if (pgtable_l3_enabled()) + set_pud(pudp, __pud(__phys_to_pud_val(pmdp) | prot)); } static inline void pud_populate(struct mm_struct *mm, pud_t *pudp, pmd_t *pmdp) @@ -33,6 +35,18 @@ static inline void pud_populate(struct mm_struct *mm, pud_t *pudp, pmd_t *pmdp) pudval |= (mm == &init_mm) ? PUD_TABLE_UXN : PUD_TABLE_PXN; __pud_populate(pudp, __pa(pmdp), pudval); } + +static inline void pmd_free(struct mm_struct *mm, pmd_t *pmd) +{ + struct ptdesc *ptdesc = virt_to_ptdesc(pmd); + + if (!pgtable_l3_enabled()) + return; + + BUG_ON((unsigned long)pmd & (PAGE_SIZE-1)); + pagetable_pmd_dtor(ptdesc); + pagetable_free(ptdesc); +} #else static inline void __pud_populate(pud_t *pudp, phys_addr_t pmdp, pudval_t prot) { diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index fd47f70a42396..8ead41da715b0 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -672,15 +672,21 @@ extern pgprot_t phys_mem_access_prot(struct file *file, unsigned long pfn, #define pmd_leaf_size(pmd) (pmd_cont(pmd) ? CONT_PMD_SIZE : PMD_SIZE) #define pte_leaf_size(pte) (pte_cont(pte) ? CONT_PTE_SIZE : PAGE_SIZE) -#if defined(CONFIG_ARM64_64K_PAGES) || CONFIG_PGTABLE_LEVELS < 3 -static inline bool pud_sect(pud_t pud) { return false; } -static inline bool pud_table(pud_t pud) { return true; } -#else -#define pud_sect(pud) ((pud_val(pud) & PUD_TYPE_MASK) == \ - PUD_TYPE_SECT) -#define pud_table(pud) ((pud_val(pud) & PUD_TYPE_MASK) == \ - PUD_TYPE_TABLE) -#endif +static inline bool pgtable_l3_enabled(void); + +static inline bool pud_sect(pud_t pud) +{ + if (PAGE_SIZE == SZ_64K || !pgtable_l3_enabled()) + return false; + return (pud_val(pud) & PUD_TYPE_MASK) == PUD_TYPE_SECT; +} + +static inline bool pud_table(pud_t pud) +{ + if (PAGE_SIZE == SZ_64K || !pgtable_l3_enabled()) + return true; + return (pud_val(pud) & PUD_TYPE_MASK) == PUD_TYPE_TABLE; +} extern pgd_t init_pg_dir[]; extern pgd_t init_pg_end[]; @@ -699,12 +705,10 @@ static inline bool in_swapper_pgdir(void *addr) static inline void set_pmd(pmd_t *pmdp, pmd_t pmd) { -#ifdef __PAGETABLE_PMD_FOLDED - if (in_swapper_pgdir(pmdp)) { + if (!pgtable_l3_enabled() && in_swapper_pgdir(pmdp)) { set_swapper_pgd((pgd_t *)pmdp, __pgd(pmd_val(pmd))); return; } -#endif /* __PAGETABLE_PMD_FOLDED */ WRITE_ONCE(*pmdp, pmd); @@ -749,20 +753,27 @@ static inline unsigned long pmd_page_vaddr(pmd_t pmd) #if CONFIG_PGTABLE_LEVELS > 2 +static __always_inline bool pgtable_l3_enabled(void) +{ + return true; +} + +static inline bool mm_pmd_folded(const struct mm_struct *mm) +{ + return !pgtable_l3_enabled(); +} +#define mm_pmd_folded mm_pmd_folded + #define pmd_ERROR(e) \ pr_err("%s:%d: bad pmd %016llx.\n", __FILE__, __LINE__, pmd_val(e)) -#define pud_none(pud) (!pud_val(pud)) -#define pud_bad(pud) (!pud_table(pud)) -#define pud_present(pud) pte_present(pud_pte(pud)) -#ifndef __PAGETABLE_PMD_FOLDED -#define pud_leaf(pud) (pud_present(pud) && !pud_table(pud)) -#else -#define pud_leaf(pud) false -#endif -#define pud_valid(pud) pte_valid(pud_pte(pud)) -#define pud_user(pud) pte_user(pud_pte(pud)) -#define pud_user_exec(pud) pte_user_exec(pud_pte(pud)) +#define pud_none(pud) (pgtable_l3_enabled() && !pud_val(pud)) +#define pud_bad(pud) (pgtable_l3_enabled() && !pud_table(pud)) +#define pud_present(pud) (!pgtable_l3_enabled() || pte_present(pud_pte(pud))) +#define pud_leaf(pud) (pgtable_l3_enabled() && pte_present(pud_pte(pud)) && !pud_table(pud)) +#define pud_valid(pud) (pgtable_l3_enabled() && pte_valid(pud_pte(pud))) +#define pud_user(pud) (pgtable_l3_enabled() && pte_user(pud_pte(pud))) +#define pud_user_exec(pud) (pgtable_l3_enabled() && pte_user_exec(pud_pte(pud))) static inline bool pgtable_l4_enabled(void); @@ -783,7 +794,8 @@ static inline void set_pud(pud_t *pudp, pud_t pud) static inline void pud_clear(pud_t *pudp) { - set_pud(pudp, __pud(0)); + if (pgtable_l3_enabled()) + set_pud(pudp, __pud(0)); } static inline phys_addr_t pud_page_paddr(pud_t pud) @@ -791,25 +803,74 @@ static inline phys_addr_t pud_page_paddr(pud_t pud) return __pud_to_phys(pud); } +#define pmd_index(addr) (((addr) >> PMD_SHIFT) & (PTRS_PER_PMD - 1)) + +static inline pmd_t *pud_to_folded_pmd(pud_t *pudp, unsigned long addr) +{ + return (pmd_t *)pudp; +} + static inline pmd_t *pud_pgtable(pud_t pud) { return (pmd_t *)__va(pud_page_paddr(pud)); } -/* Find an entry in the second-level page table. */ -#define pmd_offset_phys(dir, addr) (pud_page_paddr(READ_ONCE(*(dir))) + pmd_index(addr) * sizeof(pmd_t)) +static inline phys_addr_t pmd_offset_phys(pud_t *pudp, unsigned long addr) +{ + BUG_ON(!pgtable_l3_enabled()); + + return pud_page_paddr(READ_ONCE(*pudp)) + pmd_index(addr) * sizeof(pmd_t); +} + +static inline pmd_t *pmd_offset_lockless(pud_t *pudp, pud_t pud, + unsigned long addr) +{ + if (!pgtable_l3_enabled()) + return pud_to_folded_pmd(pudp, addr); + return (pmd_t *)__va(pud_page_paddr(pud)) + pmd_index(addr); +} +#define pmd_offset_lockless pmd_offset_lockless -#define pmd_set_fixmap(addr) ((pmd_t *)set_fixmap_offset(FIX_PMD, addr)) -#define pmd_set_fixmap_offset(pud, addr) pmd_set_fixmap(pmd_offset_phys(pud, addr)) -#define pmd_clear_fixmap() clear_fixmap(FIX_PMD) +static inline pmd_t *pmd_offset(pud_t *pudp, unsigned long addr) +{ + return pmd_offset_lockless(pudp, READ_ONCE(*pudp), addr); +} +#define pmd_offset pmd_offset -#define pud_page(pud) phys_to_page(__pud_to_phys(pud)) +static inline pmd_t *pmd_set_fixmap(unsigned long addr) +{ + if (!pgtable_l3_enabled()) + return NULL; + return (pmd_t *)set_fixmap_offset(FIX_PMD, addr); +} + +static inline pmd_t *pmd_set_fixmap_offset(pud_t *pudp, unsigned long addr) +{ + if (!pgtable_l3_enabled()) + return pud_to_folded_pmd(pudp, addr); + return pmd_set_fixmap(pmd_offset_phys(pudp, addr)); +} + +static inline void pmd_clear_fixmap(void) +{ + if (pgtable_l3_enabled()) + clear_fixmap(FIX_PMD); +} /* use ONLY for statically allocated translation tables */ -#define pmd_offset_kimg(dir,addr) ((pmd_t *)__phys_to_kimg(pmd_offset_phys((dir), (addr)))) +static inline pmd_t *pmd_offset_kimg(pud_t *pudp, u64 addr) +{ + if (!pgtable_l3_enabled()) + return pud_to_folded_pmd(pudp, addr); + return (pmd_t *)__phys_to_kimg(pmd_offset_phys(pudp, addr)); +} + +#define pud_page(pud) phys_to_page(__pud_to_phys(pud)) #else +static inline bool pgtable_l3_enabled(void) { return false; } + #define pud_valid(pud) false #define pud_page_paddr(pud) ({ BUILD_BUG(); 0; }) #define pud_user_exec(pud) pud_user(pud) /* Always 0 with folding */ diff --git a/arch/arm64/include/asm/tlb.h b/arch/arm64/include/asm/tlb.h index a947c6e784ed2..527630f0803c6 100644 --- a/arch/arm64/include/asm/tlb.h +++ b/arch/arm64/include/asm/tlb.h @@ -92,6 +92,9 @@ static inline void __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmdp, { struct ptdesc *ptdesc = virt_to_ptdesc(pmdp); + if (!pgtable_l3_enabled()) + return; + pagetable_pmd_dtor(ptdesc); tlb_remove_ptdesc(tlb, ptdesc); } diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c index e5618423bb99d..663cc76569a27 100644 --- a/arch/arm64/kernel/cpufeature.c +++ b/arch/arm64/kernel/cpufeature.c @@ -1923,8 +1923,10 @@ static int __init __kpti_install_ng_mappings(void *__unused) if (levels == 5 && !pgtable_l5_enabled()) levels = 4; - else if (levels == 4 && !pgtable_l4_enabled()) + if (levels == 4 && !pgtable_l4_enabled()) levels = 3; + if (levels == 3 && !pgtable_l3_enabled()) + levels = 2; remap_fn = (void *)__pa_symbol(idmap_kpti_install_ng_mappings); diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c index 248a2d7ad6dbb..146ecdaaaf647 100644 --- a/arch/arm64/kvm/mmu.c +++ b/arch/arm64/kvm/mmu.c @@ -1370,12 +1370,11 @@ static int get_vma_page_shift(struct vm_area_struct *vma, unsigned long hva) pa = (vma->vm_pgoff << PAGE_SHIFT) + (hva - vma->vm_start); -#ifndef __PAGETABLE_PMD_FOLDED - if ((hva & (PUD_SIZE - 1)) == (pa & (PUD_SIZE - 1)) && + if (pgtable_l3_enabled() && + (hva & (PUD_SIZE - 1)) == (pa & (PUD_SIZE - 1)) && ALIGN_DOWN(hva, PUD_SIZE) >= vma->vm_start && ALIGN(hva, PUD_SIZE) <= vma->vm_end) return PUD_SHIFT; -#endif if ((hva & (PMD_SIZE - 1)) == (pa & (PMD_SIZE - 1)) && ALIGN_DOWN(hva, PMD_SIZE) >= vma->vm_start && @@ -1487,12 +1486,10 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa, vma_shift = get_vma_page_shift(vma, hva); } -#ifndef __PAGETABLE_PMD_FOLDED - if (vma_shift == PUD_SHIFT) { + if (pgtable_l3_enabled() && vma_shift == PUD_SHIFT) { if (!fault_supports_stage2_huge_mapping(memslot, hva, PUD_SIZE)) vma_shift = PMD_SHIFT; } -#endif if (vma_shift == CONT_PMD_SHIFT) { vma_shift = PMD_SHIFT; } diff --git a/arch/arm64/mm/fixmap.c b/arch/arm64/mm/fixmap.c index a0dcf2375ccb4..f2c6678046a96 100644 --- a/arch/arm64/mm/fixmap.c +++ b/arch/arm64/mm/fixmap.c @@ -87,7 +87,7 @@ static void __init early_fixmap_init_pud(p4d_t *p4dp, unsigned long addr, p4d_t p4d = READ_ONCE(*p4dp); pud_t *pudp; - if (CONFIG_PGTABLE_LEVELS > 3 && !p4d_none(p4d) && + if (ptg_pgtable_levels > 3 && !p4d_none(p4d) && p4d_page_paddr(p4d) != __pa_symbol(bm_pud)) { /* * We only end up here if the kernel mapping and the fixmap diff --git a/arch/arm64/mm/hugetlbpage.c b/arch/arm64/mm/hugetlbpage.c index bc98c20655bba..2add0839179e3 100644 --- a/arch/arm64/mm/hugetlbpage.c +++ b/arch/arm64/mm/hugetlbpage.c @@ -51,10 +51,9 @@ void __init arm64_hugetlb_cma_reserve(void) static bool __hugetlb_valid_size(unsigned long size) { -#ifndef __PAGETABLE_PMD_FOLDED - if (size == PUD_SIZE) + if (pgtable_l3_enabled() && size == PUD_SIZE) return pud_sect_supported(); -#endif + if (size == CONT_PMD_SIZE || size == PMD_SIZE || size == CONT_PTE_SIZE) return true; @@ -100,13 +99,10 @@ static inline int num_contig_ptes(unsigned long size, size_t *pgsize) *pgsize = size; -#ifndef __PAGETABLE_PMD_FOLDED - if (size == PUD_SIZE) { + if (pgtable_l3_enabled() && size == PUD_SIZE) { if (pud_sect_supported()) contig_ptes = 1; - } else -#endif - if (size == PMD_SIZE) { + } else if (size == PMD_SIZE) { contig_ptes = 1; } else if (size == CONT_PMD_SIZE) { *pgsize = PMD_SIZE; @@ -331,10 +327,8 @@ unsigned long hugetlb_mask_last_page(struct hstate *h) { unsigned long hp_size = huge_page_size(h); -#ifndef __PAGETABLE_PMD_FOLDED - if (hp_size == PUD_SIZE) + if (pgtable_l3_enabled() && hp_size == PUD_SIZE) return PGDIR_SIZE - PUD_SIZE; -#endif if (hp_size == CONT_PMD_SIZE) return PUD_SIZE - CONT_PMD_SIZE; if (hp_size == PMD_SIZE) diff --git a/arch/arm64/mm/init.c b/arch/arm64/mm/init.c index 4d24034418b39..62587104f30d8 100644 --- a/arch/arm64/mm/init.c +++ b/arch/arm64/mm/init.c @@ -396,7 +396,7 @@ void __init mem_init(void) * scratch using the virtual address range and page size. */ VM_BUG_ON(ARM64_HW_PGTABLE_LEVELS(CONFIG_ARM64_VA_BITS) != - CONFIG_PGTABLE_LEVELS); + ptg_pgtable_levels); if (PAGE_SIZE >= 16384 && get_num_physpages() <= 128) { extern int sysctl_overcommit_memory; diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index ad7fd3fda705a..b78a341cd9e70 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -1046,7 +1046,7 @@ static void free_empty_pmd_table(pud_t *pudp, unsigned long addr, free_empty_pte_table(pmdp, addr, next, floor, ceiling); } while (addr = next, addr < end); - if (CONFIG_PGTABLE_LEVELS <= 2) + if (!pgtable_l3_enabled()) return; if (!pgtable_range_aligned(start, end, floor, ceiling, PUD_MASK)) diff --git a/arch/arm64/mm/ptdump.c b/arch/arm64/mm/ptdump.c index 6986827e0d645..045a4188afc10 100644 --- a/arch/arm64/mm/ptdump.c +++ b/arch/arm64/mm/ptdump.c @@ -230,7 +230,8 @@ static void note_page(struct ptdump_state *pt_st, unsigned long addr, int level, /* check if the current level has been folded dynamically */ if ((level == 1 && mm_p4d_folded(st->mm)) || - (level == 2 && mm_pud_folded(st->mm))) + (level == 2 && mm_pud_folded(st->mm)) || + (level == 3 && mm_pmd_folded(st->mm))) level = 0; if (level >= 0) From patchwork Mon Oct 14 10:59:01 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13834936 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id CFBB9D1625A for ; Mon, 14 Oct 2024 12:40:14 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=24J2s1s+JtIBsBU/tXNcd36ly+a41o9WkxedPu4yJ84=; b=pw9uQIABfCnmukB5GM36MN6lZR tRfx/mPXcEKManWRUKVcZee1+ED8Oga/QeKS7zt9Svle82rACvcBiWS6YoGZojJ9aRWNAuKVqfup7 93EyfFDeCEWWnBRCzi4QCWf3Btx9PGprHww23wz5RD7Yn6xTOYebccU4IvwYTaU0pb0cl6aQiECaH mV8cqiZF3GAX3z5pMQsgHS6oEhaGAOInIexhDf3PhpDA5sggOdAiq0RvY740n29p+Twt4A7RK0jpe xY8+hYZ6dqpMrZbt3TcU9lOsLsDbZzNV3zQdpzivtCiqn3FthQsXdY8WHymSlwaFdviEn1r/BqMe/ zdjK2yNQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1t0KMj-000000056nM-0zPA; Mon, 14 Oct 2024 12:40:01 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1t0Iq3-00000004olV-26CS for linux-arm-kernel@lists.infradead.org; Mon, 14 Oct 2024 11:02:14 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id AD6AF1763; Mon, 14 Oct 2024 04:02:40 -0700 (PDT) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.27]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id AE3BE3F51B; Mon, 14 Oct 2024 04:02:08 -0700 (PDT) From: Ryan Roberts To: Andrew Morton , Anshuman Khandual , Ard Biesheuvel , Catalin Marinas , David Hildenbrand , Greg Marsden , Ivan Ivanov , Kalesh Singh , Marc Zyngier , Mark Rutland , Matthias Brugger , Miroslav Benes , Will Deacon Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [RFC PATCH v1 54/57] arm64: Support runtime folding in idmap_kpti_install_ng_mappings Date: Mon, 14 Oct 2024 11:59:01 +0100 Message-ID: <20241014105912.3207374-54-ryan.roberts@arm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20241014105912.3207374-1-ryan.roberts@arm.com> References: <20241014105514.3206191-1-ryan.roberts@arm.com> <20241014105912.3207374-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241014_040211_763591_C6913D46 X-CRM114-Status: GOOD ( 15.38 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org TODO: Signed-off-by: Ryan Roberts --- ***NOTE*** Any confused maintainers may want to read the cover note here for context: https://lore.kernel.org/all/20241014105514.3206191-1-ryan.roberts@arm.com/ arch/arm64/include/asm/assembler.h | 5 ++ arch/arm64/kernel/cpufeature.c | 21 +++++- arch/arm64/mm/proc.S | 107 ++++++++++++++++++++++------- 3 files changed, 108 insertions(+), 25 deletions(-) diff --git a/arch/arm64/include/asm/assembler.h b/arch/arm64/include/asm/assembler.h index 6424fd6be1cbe..0cfa7c3efd214 100644 --- a/arch/arm64/include/asm/assembler.h +++ b/arch/arm64/include/asm/assembler.h @@ -919,6 +919,11 @@ alternative_cb_end value_for_page_size \val, \val, SZ_4K, SZ_16K, SZ_64K .endm + .macro get_page_shift, val + get_tg0 \val + value_for_page_size \val, \val, ARM64_PAGE_SHIFT_4K, ARM64_PAGE_SHIFT_16K, ARM64_PAGE_SHIFT_64K + .endm + .macro get_page_mask, val get_tg0 \val value_for_page_size \val, \val, (~(SZ_4K-1)), (~(SZ_16K-1)), (~(SZ_64K-1)) diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c index 663cc76569a27..ee94de556d3f0 100644 --- a/arch/arm64/kernel/cpufeature.c +++ b/arch/arm64/kernel/cpufeature.c @@ -1908,11 +1908,27 @@ static phys_addr_t __init kpti_ng_pgd_alloc(int shift) return kpti_ng_temp_alloc; } +struct install_ng_pgtable_geometry { + unsigned long ptrs_per_pte; + unsigned long ptrs_per_pmd; + unsigned long ptrs_per_pud; + unsigned long ptrs_per_p4d; + unsigned long ptrs_per_pgd; +}; + static int __init __kpti_install_ng_mappings(void *__unused) { - typedef void (kpti_remap_fn)(int, int, phys_addr_t, unsigned long); + typedef void (kpti_remap_fn)(int, int, phys_addr_t, unsigned long, + struct install_ng_pgtable_geometry *); extern kpti_remap_fn idmap_kpti_install_ng_mappings; kpti_remap_fn *remap_fn; + struct install_ng_pgtable_geometry geometry = { + .ptrs_per_pte = PTRS_PER_PTE, + .ptrs_per_pmd = PTRS_PER_PMD, + .ptrs_per_pud = PTRS_PER_PUD, + .ptrs_per_p4d = PTRS_PER_P4D, + .ptrs_per_pgd = PTRS_PER_PGD, + }; int cpu = smp_processor_id(); int levels = CONFIG_PGTABLE_LEVELS; @@ -1957,7 +1973,8 @@ static int __init __kpti_install_ng_mappings(void *__unused) } cpu_install_idmap(); - remap_fn(cpu, num_online_cpus(), kpti_ng_temp_pgd_pa, KPTI_NG_TEMP_VA); + remap_fn(cpu, num_online_cpus(), kpti_ng_temp_pgd_pa, KPTI_NG_TEMP_VA, + &geometry); cpu_uninstall_idmap(); if (!cpu) { diff --git a/arch/arm64/mm/proc.S b/arch/arm64/mm/proc.S index ab5aa84923524..11bf6ba6dac33 100644 --- a/arch/arm64/mm/proc.S +++ b/arch/arm64/mm/proc.S @@ -190,7 +190,7 @@ SYM_FUNC_ALIAS(__pi_idmap_cpu_replace_ttbr1, idmap_cpu_replace_ttbr1) .pushsection ".idmap.text", "a" .macro pte_to_phys, phys, pte - and \phys, \pte, #PTE_ADDR_LOW + and \phys, \pte, pte_addr_low #ifdef CONFIG_ARM64_PA_BITS_52 and \pte, \pte, #PTE_ADDR_HIGH orr \phys, \phys, \pte, lsl #PTE_ADDR_HIGH_SHIFT @@ -198,7 +198,8 @@ SYM_FUNC_ALIAS(__pi_idmap_cpu_replace_ttbr1, idmap_cpu_replace_ttbr1) .endm .macro kpti_mk_tbl_ng, type, num_entries - add end_\type\()p, cur_\type\()p, #\num_entries * 8 + lsl scratch, \num_entries, #3 + add end_\type\()p, cur_\type\()p, scratch .Ldo_\type: ldr \type, [cur_\type\()p], #8 // Load the entry and advance tbz \type, #0, .Lnext_\type // Skip invalid and @@ -220,14 +221,18 @@ SYM_FUNC_ALIAS(__pi_idmap_cpu_replace_ttbr1, idmap_cpu_replace_ttbr1) .macro kpti_map_pgtbl, type, level str xzr, [temp_pte, #8 * (\level + 2)] // break before make dsb nshst - add pte, temp_pte, #PAGE_SIZE * (\level + 2) + mov scratch, #(\level + 2) + mul scratch, scratch, page_size + add pte, temp_pte, scratch lsr pte, pte, #12 tlbi vaae1, pte dsb nsh isb phys_to_pte pte, cur_\type\()p - add cur_\type\()p, temp_pte, #PAGE_SIZE * (\level + 2) + mov scratch, #(\level + 2) + mul scratch, scratch, page_size + add cur_\type\()p, temp_pte, scratch orr pte, pte, pte_flags str pte, [temp_pte, #8 * (\level + 2)] dsb nshst @@ -235,7 +240,8 @@ SYM_FUNC_ALIAS(__pi_idmap_cpu_replace_ttbr1, idmap_cpu_replace_ttbr1) /* * void __kpti_install_ng_mappings(int cpu, int num_secondaries, phys_addr_t temp_pgd, - * unsigned long temp_pte_va) + * unsigned long temp_pte_va, + * struct install_ng_pgtable_geometry *geometry) * * Called exactly once from stop_machine context by each CPU found during boot. */ @@ -251,6 +257,8 @@ SYM_TYPED_FUNC_START(idmap_kpti_install_ng_mappings) temp_pgd_phys .req x2 swapper_ttb .req x3 flag_ptr .req x4 + geometry .req x4 + scratch .req x4 cur_pgdp .req x5 end_pgdp .req x6 pgd .req x7 @@ -264,18 +272,45 @@ SYM_TYPED_FUNC_START(idmap_kpti_install_ng_mappings) valid .req x17 cur_p4dp .req x19 end_p4dp .req x20 - - mov x5, x3 // preserve temp_pte arg - mrs swapper_ttb, ttbr1_el1 - adr_l flag_ptr, __idmap_kpti_flag + page_size .req x21 + ptrs_per_pte .req x22 + ptrs_per_pmd .req x23 + ptrs_per_pud .req x24 + ptrs_per_p4d .req x25 + ptrs_per_pgd .req x26 + pte_addr_low .req x27 cbnz cpu, __idmap_kpti_secondary -#if CONFIG_PGTABLE_LEVELS > 4 - stp x29, x30, [sp, #-32]! + /* Preserve callee-saved registers */ + stp x19, x20, [sp, #-96]! + stp x21, x22, [sp, #80] + stp x23, x24, [sp, #64] + stp x25, x26, [sp, #48] + stp x27, x28, [sp, #32] + stp x29, x30, [sp, #16] mov x29, sp - stp x19, x20, [sp, #16] -#endif + + /* Load pgtable geometry parameters */ + get_page_size page_size + ldr ptrs_per_pte, [geometry, #0] + ldr ptrs_per_pmd, [geometry, #8] + ldr ptrs_per_pud, [geometry, #16] + ldr ptrs_per_p4d, [geometry, #24] + ldr ptrs_per_pgd, [geometry, #32] + + /* Precalculate pte_addr_low mask */ + get_page_shift x0 + mov pte_addr_low, #50 + sub pte_addr_low, pte_addr_low, x0 + mov scratch, #1 + lsl pte_addr_low, scratch, pte_addr_low + sub pte_addr_low, pte_addr_low, #1 + lsl pte_addr_low, pte_addr_low, x0 + + mov temp_pte, x3 + mrs swapper_ttb, ttbr1_el1 + adr_l flag_ptr, __idmap_kpti_flag /* We're the boot CPU. Wait for the others to catch up */ sevl @@ -290,7 +325,6 @@ SYM_TYPED_FUNC_START(idmap_kpti_install_ng_mappings) msr ttbr1_el1, temp_pgd_phys isb - mov temp_pte, x5 mov_q pte_flags, KPTI_NG_PTE_FLAGS /* Everybody is enjoying the idmap, so we can rewrite swapper. */ @@ -320,7 +354,7 @@ alternative_else_nop_endif /* PGD */ adrp cur_pgdp, swapper_pg_dir kpti_map_pgtbl pgd, -1 - kpti_mk_tbl_ng pgd, PTRS_PER_PGD + kpti_mk_tbl_ng pgd, ptrs_per_pgd /* Ensure all the updated entries are visible to secondary CPUs */ dsb ishst @@ -331,21 +365,33 @@ alternative_else_nop_endif isb /* Set the flag to zero to indicate that we're all done */ + adr_l flag_ptr, __idmap_kpti_flag str wzr, [flag_ptr] -#if CONFIG_PGTABLE_LEVELS > 4 - ldp x19, x20, [sp, #16] - ldp x29, x30, [sp], #32 -#endif + + /* Restore callee-saved registers */ + ldp x29, x30, [sp, #16] + ldp x27, x28, [sp, #32] + ldp x25, x26, [sp, #48] + ldp x23, x24, [sp, #64] + ldp x21, x22, [sp, #80] + ldp x19, x20, [sp], #96 + ret .Lderef_pgd: /* P4D */ .if CONFIG_PGTABLE_LEVELS > 4 p4d .req x30 + cmp ptrs_per_p4d, #1 + b.eq .Lfold_p4d pte_to_phys cur_p4dp, pgd kpti_map_pgtbl p4d, 0 - kpti_mk_tbl_ng p4d, PTRS_PER_P4D + kpti_mk_tbl_ng p4d, ptrs_per_p4d b .Lnext_pgd +.Lfold_p4d: + mov p4d, pgd // fold to next level + mov cur_p4dp, end_p4dp // must be equal to terminate loop + b .Lderef_p4d .else /* CONFIG_PGTABLE_LEVELS <= 4 */ p4d .req pgd .set .Lnext_p4d, .Lnext_pgd @@ -355,10 +401,16 @@ alternative_else_nop_endif /* PUD */ .if CONFIG_PGTABLE_LEVELS > 3 pud .req x10 + cmp ptrs_per_pud, #1 + b.eq .Lfold_pud pte_to_phys cur_pudp, p4d kpti_map_pgtbl pud, 1 - kpti_mk_tbl_ng pud, PTRS_PER_PUD + kpti_mk_tbl_ng pud, ptrs_per_pud b .Lnext_p4d +.Lfold_pud: + mov pud, p4d // fold to next level + mov cur_pudp, end_pudp // must be equal to terminate loop + b .Lderef_pud .else /* CONFIG_PGTABLE_LEVELS <= 3 */ pud .req pgd .set .Lnext_pud, .Lnext_pgd @@ -368,10 +420,16 @@ alternative_else_nop_endif /* PMD */ .if CONFIG_PGTABLE_LEVELS > 2 pmd .req x13 + cmp ptrs_per_pmd, #1 + b.eq .Lfold_pmd pte_to_phys cur_pmdp, pud kpti_map_pgtbl pmd, 2 - kpti_mk_tbl_ng pmd, PTRS_PER_PMD + kpti_mk_tbl_ng pmd, ptrs_per_pmd b .Lnext_pud +.Lfold_pmd: + mov pmd, pud // fold to next level + mov cur_pmdp, end_pmdp // must be equal to terminate loop + b .Lderef_pmd .else /* CONFIG_PGTABLE_LEVELS <= 2 */ pmd .req pgd .set .Lnext_pmd, .Lnext_pgd @@ -381,7 +439,7 @@ alternative_else_nop_endif /* PTE */ pte_to_phys cur_ptep, pmd kpti_map_pgtbl pte, 3 - kpti_mk_tbl_ng pte, PTRS_PER_PTE + kpti_mk_tbl_ng pte, ptrs_per_pte b .Lnext_pmd .unreq cpu @@ -408,6 +466,9 @@ alternative_else_nop_endif /* Secondary CPUs end up here */ __idmap_kpti_secondary: + mrs swapper_ttb, ttbr1_el1 + adr_l flag_ptr, __idmap_kpti_flag + /* Uninstall swapper before surgery begins */ __idmap_cpu_set_reserved_ttbr1 x16, x17 From patchwork Mon Oct 14 10:59:02 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13834937 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 3AC64D1625C for ; Mon, 14 Oct 2024 12:41:38 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=BkgcTQ3nJpwwleh27xwrJdMVBeha9+gRlJOOjRb7rV4=; b=yf66iO3kpCq09v7IfQBbgHTKPi BWmPYNZeBTcZUDI64yc9WoXntmrr1QCv1aDRR5+ZrNU1g/Mz7JUY+k7pq1fi8i8VeVf1uredLJt0p snLFfyLcKdXcPV19421OPjrQH4VJ+FPS16kXcnVNhAxW4RB+W0VPex/K698PMtqjGVfOtDT+WoBxk i4JYBV0F+Mi9Y/wrxnUC7WFsZ8+m64Pto6AqDS/8mYwobUvG2zGlb3tQWoLp5IrCoZ+K9bVEqgKxV qLehl3jxy6EL6kKx+9pcm7pfb2oncUBOBN07RYlLaEDLI40sagK8aes21Fo48PiaUQtjDlzaKk9Z5 cy/z8PUQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1t0KO6-000000056yQ-1W71; Mon, 14 Oct 2024 12:41:26 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1t0Iq6-00000004onG-0kTV for linux-arm-kernel@lists.infradead.org; Mon, 14 Oct 2024 11:02:16 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 5E32F1756; Mon, 14 Oct 2024 04:02:43 -0700 (PDT) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.27]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 5E9263F51B; Mon, 14 Oct 2024 04:02:11 -0700 (PDT) From: Ryan Roberts To: Andrew Morton , Anshuman Khandual , Ard Biesheuvel , Catalin Marinas , David Hildenbrand , Greg Marsden , Ivan Ivanov , Kalesh Singh , Marc Zyngier , Mark Rutland , Matthias Brugger , Miroslav Benes , Will Deacon Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [RFC PATCH v1 55/57] arm64: TRAMP_VALIAS is no longer compile-time constant Date: Mon, 14 Oct 2024 11:59:02 +0100 Message-ID: <20241014105912.3207374-55-ryan.roberts@arm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20241014105912.3207374-1-ryan.roberts@arm.com> References: <20241014105514.3206191-1-ryan.roberts@arm.com> <20241014105912.3207374-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241014_040214_353659_CF94FA63 X-CRM114-Status: GOOD ( 11.44 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org When boot-time page size is in operation, TRAMP_VALIAS is no longer a compile-time constant, because the VA of a fixmap slot depends upon PAGE_SIZE. Let's handle this by instead exporting the slot index, FIX_ENTRY_TRAMP_BEGIN,to assembly, then do the TRAMP_VALIAS calculation per page size and use alternatives to decide which variant to activate. Note that for the tramp_map_kernel case, we are one instruction short of space in the vector to have NOPs for all 3 page size variants. So we do if/else for 16K/64K and branch around it for the 4K case. This saves 2 instructions. Signed-off-by: Ryan Roberts --- ***NOTE*** Any confused maintainers may want to read the cover note here for context: https://lore.kernel.org/all/20241014105514.3206191-1-ryan.roberts@arm.com/ arch/arm64/kernel/asm-offsets.c | 2 +- arch/arm64/kernel/entry.S | 50 ++++++++++++++++++++++++++------- 2 files changed, 41 insertions(+), 11 deletions(-) diff --git a/arch/arm64/kernel/asm-offsets.c b/arch/arm64/kernel/asm-offsets.c index f32b8d7f00b2a..c45fa3e281884 100644 --- a/arch/arm64/kernel/asm-offsets.c +++ b/arch/arm64/kernel/asm-offsets.c @@ -172,7 +172,7 @@ int main(void) DEFINE(ARM64_FTR_SYSVAL, offsetof(struct arm64_ftr_reg, sys_val)); BLANK(); #ifdef CONFIG_UNMAP_KERNEL_AT_EL0 - DEFINE(TRAMP_VALIAS, TRAMP_VALIAS); + DEFINE(FIX_ENTRY_TRAMP_BEGIN, FIX_ENTRY_TRAMP_BEGIN); #endif #ifdef CONFIG_ARM_SDE_INTERFACE DEFINE(SDEI_EVENT_INTREGS, offsetof(struct sdei_registered_event, interrupted_regs)); diff --git a/arch/arm64/kernel/entry.S b/arch/arm64/kernel/entry.S index 7ef0e127b149f..ba47dc8672c04 100644 --- a/arch/arm64/kernel/entry.S +++ b/arch/arm64/kernel/entry.S @@ -101,11 +101,27 @@ .org .Lventry_start\@ + 128 // Did we overflow the ventry slot? .endm +#define TRAMP_VALIAS(page_shift) (FIXADDR_TOP - (FIX_ENTRY_TRAMP_BEGIN << (page_shift))) + .macro tramp_alias, dst, sym - .set .Lalias\@, TRAMP_VALIAS + \sym - .entry.tramp.text - movz \dst, :abs_g2_s:.Lalias\@ - movk \dst, :abs_g1_nc:.Lalias\@ - movk \dst, :abs_g0_nc:.Lalias\@ +alternative_if ARM64_USE_PAGE_SIZE_4K + .set .Lalias4k\@, TRAMP_VALIAS(ARM64_PAGE_SHIFT_4K) + \sym - .entry.tramp.text + movz \dst, :abs_g2_s:.Lalias4k\@ + movk \dst, :abs_g1_nc:.Lalias4k\@ + movk \dst, :abs_g0_nc:.Lalias4k\@ +alternative_else_nop_endif +alternative_if ARM64_USE_PAGE_SIZE_16K + .set .Lalias16k\@, TRAMP_VALIAS(ARM64_PAGE_SHIFT_16K) + \sym - .entry.tramp.text + movz \dst, :abs_g2_s:.Lalias16k\@ + movk \dst, :abs_g1_nc:.Lalias16k\@ + movk \dst, :abs_g0_nc:.Lalias16k\@ +alternative_else_nop_endif +alternative_if ARM64_USE_PAGE_SIZE_64K + .set .Lalias64k\@, TRAMP_VALIAS(ARM64_PAGE_SHIFT_64K) + \sym - .entry.tramp.text + movz \dst, :abs_g2_s:.Lalias64k\@ + movk \dst, :abs_g1_nc:.Lalias64k\@ + movk \dst, :abs_g0_nc:.Lalias64k\@ +alternative_else_nop_endif .endm /* @@ -627,16 +643,30 @@ SYM_CODE_END(ret_to_user) bic \tmp, \tmp, #USER_ASID_FLAG msr ttbr1_el1, \tmp #ifdef CONFIG_QCOM_FALKOR_ERRATUM_1003 -alternative_if ARM64_WORKAROUND_QCOM_FALKOR_E1003 +alternative_if_not ARM64_WORKAROUND_QCOM_FALKOR_E1003 + b .Lskip_falkor_e1003\@ +alternative_else_nop_endif /* ASID already in \tmp[63:48] */ - movk \tmp, #:abs_g2_nc:(TRAMP_VALIAS >> 12) - movk \tmp, #:abs_g1_nc:(TRAMP_VALIAS >> 12) - /* 2MB boundary containing the vectors, so we nobble the walk cache */ - movk \tmp, #:abs_g0_nc:((TRAMP_VALIAS & ~(SZ_2M - 1)) >> 12) +alternative_if ARM64_USE_PAGE_SIZE_4K + movk \tmp, #:abs_g2_nc:(TRAMP_VALIAS(ARM64_PAGE_SHIFT_4K) >> 12) + movk \tmp, #:abs_g1_nc:(TRAMP_VALIAS(ARM64_PAGE_SHIFT_4K) >> 12) + movk \tmp, #:abs_g0_nc:((TRAMP_VALIAS(ARM64_PAGE_SHIFT_4K) & ~(SZ_2M - 1)) >> 12) + b .Lfinish_falkor_e1003\@ +alternative_else_nop_endif +alternative_if ARM64_USE_PAGE_SIZE_16K + movk \tmp, #:abs_g2_nc:(TRAMP_VALIAS(ARM64_PAGE_SHIFT_16K) >> 12) + movk \tmp, #:abs_g1_nc:(TRAMP_VALIAS(ARM64_PAGE_SHIFT_16K) >> 12) + movk \tmp, #:abs_g0_nc:((TRAMP_VALIAS(ARM64_PAGE_SHIFT_16K) & ~(SZ_2M - 1)) >> 12) +alternative_else /* ARM64_USE_PAGE_SIZE_64K */ + movk \tmp, #:abs_g2_nc:(TRAMP_VALIAS(ARM64_PAGE_SHIFT_64K) >> 12) + movk \tmp, #:abs_g1_nc:(TRAMP_VALIAS(ARM64_PAGE_SHIFT_64K) >> 12) + movk \tmp, #:abs_g0_nc:((TRAMP_VALIAS(ARM64_PAGE_SHIFT_64K) & ~(SZ_2M - 1)) >> 12) +alternative_endif +.Lfinish_falkor_e1003\@: isb tlbi vae1, \tmp dsb nsh -alternative_else_nop_endif +.Lskip_falkor_e1003\@: #endif /* CONFIG_QCOM_FALKOR_ERRATUM_1003 */ .endm From patchwork Mon Oct 14 10:59:03 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13834955 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 5C29FD1625B for ; Mon, 14 Oct 2024 12:43:05 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=cPtmn6bEPN9e8rdSq6k9AO9DyT36CNVCz9b2AR0mIuI=; b=PLau5Oz58e9YbLAjrFU2F0Cm2J B3nViBZFoIixRuJPliSMbpwDeXd3MKe+EzNdmW8MGcMurdFB6EJZMhQ49m+jxU1g5pxzp5Wtg7rFK PTLv3iD2xqchnTd+4mZhCbF3nzxaFbbhQKYS5YS3QURyPblleqAuJQdW+b7a58RyKh+juPUzdpkVI vNksOGrWlXatTBXK1LotYKku9Dcqo1i5Pk9zA+hRPYsZ03rQ564UmqmFvSNA6lMVIvbBRW6yow0Qw RjsTVMYKYO/eI3XErmtTc5XWt5LMqEkYeITWtwhMfk26Ji2dNMjzEsQ4/jcmigynfIj6aUmh8Zu0r 14JBSHXw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1t0KPR-000000057AP-3weW; Mon, 14 Oct 2024 12:42:49 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1t0Iq9-00000004ooV-0qTA for linux-arm-kernel@lists.infradead.org; Mon, 14 Oct 2024 11:02:20 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 6C0D71756; Mon, 14 Oct 2024 04:02:46 -0700 (PDT) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.27]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 0FB423F51B; Mon, 14 Oct 2024 04:02:13 -0700 (PDT) From: Ryan Roberts To: Andrew Morton , Anshuman Khandual , Ard Biesheuvel , Catalin Marinas , David Hildenbrand , Greg Marsden , Ivan Ivanov , Kalesh Singh , Marc Zyngier , Mark Rutland , Matthias Brugger , Miroslav Benes , Oliver Upton , Will Deacon Cc: Ryan Roberts , kvmarm@lists.linux.dev, linux-arm-kernel@lists.infradead.org, linux-efi@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [RFC PATCH v1 56/57] arm64: Determine THREAD_SIZE at boot-time Date: Mon, 14 Oct 2024 11:59:03 +0100 Message-ID: <20241014105912.3207374-56-ryan.roberts@arm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20241014105912.3207374-1-ryan.roberts@arm.com> References: <20241014105514.3206191-1-ryan.roberts@arm.com> <20241014105912.3207374-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241014_040217_384440_AAFE1539 X-CRM114-Status: GOOD ( 17.23 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Since THREAD_SIZE depends on PAGE_SIZE when stacks are vmapped, we must defer the decision on THREAD_SIZE until we have selected PAGE_SIZE at boot. The one wrinkle is entry.S's requirement to have THREAD_SHIFT as an immediate in order to check that the stack has not overflowed without clobbering any registers, early in the exception handler. Solve this by patching alternatives. During early boot, all 3 options are NOPs until the alternative is patched in. So we forgo overflow checking until boot-cpu patching is complete. Signed-off-by: Ryan Roberts --- ***NOTE*** Any confused maintainers may want to read the cover note here for context: https://lore.kernel.org/all/20241014105514.3206191-1-ryan.roberts@arm.com/ arch/arm64/include/asm/assembler.h | 5 +++ arch/arm64/include/asm/efi.h | 2 +- arch/arm64/include/asm/memory.h | 51 +++++++++++++++++++++++++----- arch/arm64/kernel/efi.c | 2 +- arch/arm64/kernel/entry.S | 10 +++++- arch/arm64/kernel/head.S | 3 +- arch/arm64/kernel/vmlinux.lds.S | 4 +-- arch/arm64/kvm/hyp/nvhe/hyp.lds.S | 2 +- 8 files changed, 64 insertions(+), 15 deletions(-) diff --git a/arch/arm64/include/asm/assembler.h b/arch/arm64/include/asm/assembler.h index 0cfa7c3efd214..745328e7768b7 100644 --- a/arch/arm64/include/asm/assembler.h +++ b/arch/arm64/include/asm/assembler.h @@ -928,4 +928,9 @@ alternative_cb_end get_tg0 \val value_for_page_size \val, \val, (~(SZ_4K-1)), (~(SZ_16K-1)), (~(SZ_64K-1)) .endm + + .macro get_task_size, val + get_tg0 \val + value_for_page_size \val, \val, (1 << THREAD_SHIFT_4K), (1 << THREAD_SHIFT_16K), (1 << THREAD_SHIFT_64K) + .endm #endif /* __ASM_ASSEMBLER_H */ diff --git a/arch/arm64/include/asm/efi.h b/arch/arm64/include/asm/efi.h index bcd5622aa0968..913f599c14e40 100644 --- a/arch/arm64/include/asm/efi.h +++ b/arch/arm64/include/asm/efi.h @@ -68,7 +68,7 @@ void arch_efi_call_virt_teardown(void); * kernel need greater alignment than we require the segments to be padded to. */ #define EFI_KIMG_ALIGN \ - (SEGMENT_ALIGN > THREAD_ALIGN ? SEGMENT_ALIGN : THREAD_ALIGN) + (SEGMENT_ALIGN > THREAD_ALIGN_MAX ? SEGMENT_ALIGN : THREAD_ALIGN_MAX) /* * On arm64, we have to ensure that the initrd ends up in the linear region, diff --git a/arch/arm64/include/asm/memory.h b/arch/arm64/include/asm/memory.h index 5393a859183f7..e28f5700ef022 100644 --- a/arch/arm64/include/asm/memory.h +++ b/arch/arm64/include/asm/memory.h @@ -110,23 +110,56 @@ #define PAGE_END (_PAGE_END(VA_BITS_MIN)) #endif /* CONFIG_KASAN */ -#define MIN_THREAD_SHIFT (14 + KASAN_THREAD_SHIFT) +#define IDEAL_THREAD_SHIFT (14 + KASAN_THREAD_SHIFT) /* * VMAP'd stacks are allocated at page granularity, so we must ensure that such * stacks are a multiple of page size. */ -#if defined(CONFIG_VMAP_STACK) && (MIN_THREAD_SHIFT < PAGE_SHIFT) -#define THREAD_SHIFT PAGE_SHIFT + +#if defined(CONFIG_VMAP_STACK) +#define THREAD_SHIFT \ + (IDEAL_THREAD_SHIFT < PAGE_SHIFT ? PAGE_SHIFT : IDEAL_THREAD_SHIFT) +#if (IDEAL_THREAD_SHIFT < PAGE_SHIFT_MIN) +#define THREAD_SHIFT_MIN PAGE_SHIFT_MIN #else -#define THREAD_SHIFT MIN_THREAD_SHIFT +#define THREAD_SHIFT_MIN IDEAL_THREAD_SHIFT #endif - -#if THREAD_SHIFT >= PAGE_SHIFT -#define THREAD_SIZE_ORDER (THREAD_SHIFT - PAGE_SHIFT) +#if (IDEAL_THREAD_SHIFT < PAGE_SHIFT_MAX) +#define THREAD_SHIFT_MAX PAGE_SHIFT_MAX +#else +#define THREAD_SHIFT_MAX IDEAL_THREAD_SHIFT +#endif +#if (IDEAL_THREAD_SHIFT < ARM64_PAGE_SHIFT_4K) +#define THREAD_SHIFT_4K ARM64_PAGE_SHIFT_4K +#else +#define THREAD_SHIFT_4K IDEAL_THREAD_SHIFT +#endif +#if (IDEAL_THREAD_SHIFT < ARM64_PAGE_SHIFT_16K) +#define THREAD_SHIFT_16K ARM64_PAGE_SHIFT_16K +#else +#define THREAD_SHIFT_16K IDEAL_THREAD_SHIFT +#endif +#if (IDEAL_THREAD_SHIFT < ARM64_PAGE_SHIFT_64K) +#define THREAD_SHIFT_64K ARM64_PAGE_SHIFT_64K +#else +#define THREAD_SHIFT_64K IDEAL_THREAD_SHIFT #endif +#else +#define THREAD_SHIFT IDEAL_THREAD_SHIFT +#define THREAD_SHIFT_MIN IDEAL_THREAD_SHIFT +#define THREAD_SHIFT_MAX IDEAL_THREAD_SHIFT +#define THREAD_SHIFT_4K IDEAL_THREAD_SHIFT +#define THREAD_SHIFT_16K IDEAL_THREAD_SHIFT +#define THREAD_SHIFT_64K IDEAL_THREAD_SHIFT +#endif + +#define THREAD_SIZE_ORDER \ + (PAGE_SHIFT < THREAD_SHIFT ? THREAD_SHIFT - PAGE_SHIFT : 0) #define THREAD_SIZE (UL(1) << THREAD_SHIFT) +#define THREAD_SIZE_MIN (UL(1) << THREAD_SHIFT_MIN) +#define THREAD_SIZE_MAX (UL(1) << THREAD_SHIFT_MAX) /* * By aligning VMAP'd stacks to 2 * THREAD_SIZE, we can detect overflow by @@ -135,11 +168,13 @@ */ #ifdef CONFIG_VMAP_STACK #define THREAD_ALIGN (2 * THREAD_SIZE) +#define THREAD_ALIGN_MAX (2 * THREAD_SIZE_MAX) #else #define THREAD_ALIGN THREAD_SIZE +#define THREAD_ALIGN_MAX THREAD_SIZE_MAX #endif -#define IRQ_STACK_SIZE THREAD_SIZE +#define IRQ_STACK_SIZE THREAD_SIZE_MIN #define OVERFLOW_STACK_SIZE SZ_4K diff --git a/arch/arm64/kernel/efi.c b/arch/arm64/kernel/efi.c index 712718aed5dd9..ebc44b7e83199 100644 --- a/arch/arm64/kernel/efi.c +++ b/arch/arm64/kernel/efi.c @@ -197,7 +197,7 @@ bool efi_runtime_fixup_exception(struct pt_regs *regs, const char *msg) } /* EFI requires 8 KiB of stack space for runtime services */ -static_assert(THREAD_SIZE >= SZ_8K); +static_assert(THREAD_SIZE_MIN >= SZ_8K); static int __init arm64_efi_rt_init(void) { diff --git a/arch/arm64/kernel/entry.S b/arch/arm64/kernel/entry.S index ba47dc8672c04..1ab65e406b62e 100644 --- a/arch/arm64/kernel/entry.S +++ b/arch/arm64/kernel/entry.S @@ -62,7 +62,15 @@ */ add sp, sp, x0 // sp' = sp + x0 sub x0, sp, x0 // x0' = sp' - x0 = (sp + x0) - x0 = sp - tbnz x0, #THREAD_SHIFT, 0f +alternative_if ARM64_USE_PAGE_SIZE_4K + tbnz x0, #THREAD_SHIFT_4K, 0f +alternative_else_nop_endif +alternative_if ARM64_USE_PAGE_SIZE_16K + tbnz x0, #THREAD_SHIFT_16K, 0f +alternative_else_nop_endif +alternative_if ARM64_USE_PAGE_SIZE_64K + tbnz x0, #THREAD_SHIFT_64K, 0f +alternative_else_nop_endif sub x0, sp, x0 // x0'' = sp' - x0' = (sp + x0) - sp = x0 sub sp, sp, x0 // sp'' = sp' - x0 = (sp + x0) - x0 = sp b el\el\ht\()_\regsize\()_\label diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S index 761b7f5633e15..2530ee5cee548 100644 --- a/arch/arm64/kernel/head.S +++ b/arch/arm64/kernel/head.S @@ -198,7 +198,8 @@ SYM_CODE_END(preserve_boot_args) msr sp_el0, \tsk ldr \tmp1, [\tsk, #TSK_STACK] - add sp, \tmp1, #THREAD_SIZE + get_task_size \tmp2 + add sp, \tmp1, \tmp2 sub sp, sp, #PT_REGS_SIZE stp xzr, xzr, [sp, #S_STACKFRAME] diff --git a/arch/arm64/kernel/vmlinux.lds.S b/arch/arm64/kernel/vmlinux.lds.S index 09fcc234c0f77..937900a458a89 100644 --- a/arch/arm64/kernel/vmlinux.lds.S +++ b/arch/arm64/kernel/vmlinux.lds.S @@ -60,11 +60,11 @@ #define RO_EXCEPTION_TABLE_ALIGN 4 #define RUNTIME_DISCARD_EXIT +#include #include #include #include #include -#include #include #include "image.h" @@ -292,7 +292,7 @@ SECTIONS _data = .; _sdata = .; - RW_DATA(L1_CACHE_BYTES, PAGE_SIZE_MAX, THREAD_ALIGN) + RW_DATA(L1_CACHE_BYTES, PAGE_SIZE_MAX, THREAD_ALIGN_MAX) /* * Data written with the MMU off but read with the MMU on requires diff --git a/arch/arm64/kvm/hyp/nvhe/hyp.lds.S b/arch/arm64/kvm/hyp/nvhe/hyp.lds.S index 74c7c21626270..fe1fbfa8f8f05 100644 --- a/arch/arm64/kvm/hyp/nvhe/hyp.lds.S +++ b/arch/arm64/kvm/hyp/nvhe/hyp.lds.S @@ -7,9 +7,9 @@ */ #include +#include #include #include -#include SECTIONS { HYP_SECTION(.idmap.text) From patchwork Mon Oct 14 10:59:04 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13834956 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id C7053D1625B for ; Mon, 14 Oct 2024 12:44:27 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=xQxbzyYZeDib8MIhoxkIiml0NfyISTXv5vXI0QGhods=; b=lOLnrKTu41yTv/od6GjjRyS5ye c0IgqUfm9wlBXU2q0/zLULSsjfsp8VmKmhuc7O/yEkBfLJZoPCvQ1E6wBETDUKC52m12yPSao3CB4 ua6Mq7e/ZhCbfFqW1cKfvcBxEK3Cmbd9/EYHBJjLETTfyT8q4FOG7QG2abXKwxm6QqmvJXU47IuNw gcmsVoa9eweuc03uGjBu/YtJXBENNx0Uowi8p3xKDLrfvJv4r3xfgRJ2g46mIZajNoFtN/E8o9t2N AuoJDbi+K1tGaFYOxKL16BDBYJA4g5sSxggB/pUUu0rkXB6PsIc5bOKK+xdp88gJxbV7nR7+1uy9Q q6DaOmKg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1t0KQo-000000057PV-32DF; Mon, 14 Oct 2024 12:44:15 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1t0IqC-00000004oq6-13Cj for linux-arm-kernel@lists.infradead.org; Mon, 14 Oct 2024 11:02:23 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 7ACB21758; Mon, 14 Oct 2024 04:02:49 -0700 (PDT) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.27]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 1E2F93F51B; Mon, 14 Oct 2024 04:02:16 -0700 (PDT) From: Ryan Roberts To: Andrew Morton , Anshuman Khandual , Ard Biesheuvel , Catalin Marinas , David Hildenbrand , Greg Marsden , Ivan Ivanov , Kalesh Singh , Marc Zyngier , Mark Rutland , Matthias Brugger , Miroslav Benes , Oliver Upton , Will Deacon Cc: Ryan Roberts , kvmarm@lists.linux.dev, linux-arm-kernel@lists.infradead.org, linux-efi@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [RFC PATCH v1 57/57] arm64: Enable boot-time page size selection Date: Mon, 14 Oct 2024 11:59:04 +0100 Message-ID: <20241014105912.3207374-57-ryan.roberts@arm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20241014105912.3207374-1-ryan.roberts@arm.com> References: <20241014105514.3206191-1-ryan.roberts@arm.com> <20241014105912.3207374-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241014_040220_434337_A8B76E1F X-CRM114-Status: GOOD ( 23.00 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Introduce a new Kconfig, ARM64_BOOT_TIME_PAGE_SIZE, which can be selected instead of a page size. When selected, the resulting kernel's page size can be configured at boot via the command line. For now, boot-time page size kernels are limited to 48-bit VA, since more work is required to support LPA2. Additionally MMAP_RND_BITS and SECTION_SIZE_BITS are configured for the worst case (64K pages). Future work could be implemented to be able to configure these at boot time for optimial page size-specific values. Signed-off-by: Ryan Roberts --- ***NOTE*** Any confused maintainers may want to read the cover note here for context: https://lore.kernel.org/all/20241014105514.3206191-1-ryan.roberts@arm.com/ arch/arm64/Kconfig | 26 ++++++++++--- arch/arm64/include/asm/kvm_hyp.h | 11 ++++++ arch/arm64/include/asm/pgtable-geometry.h | 22 ++++++++++- arch/arm64/include/asm/pgtable-hwdef.h | 6 +-- arch/arm64/include/asm/pgtable.h | 10 ++++- arch/arm64/include/asm/sparsemem.h | 4 ++ arch/arm64/kernel/image-vars.h | 11 ++++++ arch/arm64/kernel/image.h | 4 ++ arch/arm64/kernel/pi/map_kernel.c | 45 ++++++++++++++++++++++ arch/arm64/kvm/arm.c | 10 +++++ arch/arm64/kvm/hyp/nvhe/Makefile | 1 + arch/arm64/kvm/hyp/nvhe/pgtable-geometry.c | 16 ++++++++ arch/arm64/mm/Makefile | 1 + arch/arm64/mm/pgd.c | 10 +++-- arch/arm64/mm/pgtable-geometry.c | 24 ++++++++++++ drivers/firmware/efi/libstub/arm64.c | 3 +- 16 files changed, 187 insertions(+), 17 deletions(-) create mode 100644 arch/arm64/kvm/hyp/nvhe/pgtable-geometry.c create mode 100644 arch/arm64/mm/pgtable-geometry.c diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index a2f8ff354ca67..573d308741169 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -121,6 +121,7 @@ config ARM64 select BUILDTIME_TABLE_SORT select CLONE_BACKWARDS select COMMON_CLK + select CONSTRUCTORS if ARM64_BOOT_TIME_PAGE_SIZE select CPU_PM if (SUSPEND || CPU_IDLE) select CPUMASK_OFFSTACK if NR_CPUS > 256 select CRC32 @@ -284,18 +285,20 @@ config MMU config ARM64_CONT_PTE_SHIFT int + depends on !ARM64_BOOT_TIME_PAGE_SIZE default 5 if PAGE_SIZE_64KB default 7 if PAGE_SIZE_16KB default 4 config ARM64_CONT_PMD_SHIFT int + depends on !ARM64_BOOT_TIME_PAGE_SIZE default 5 if PAGE_SIZE_64KB default 5 if PAGE_SIZE_16KB default 4 config ARCH_MMAP_RND_BITS_MIN - default 14 if PAGE_SIZE_64KB + default 14 if ARM64_64K_PAGES || ARM64_BOOT_TIME_PAGE_SIZE default 16 if PAGE_SIZE_16KB default 18 @@ -306,15 +309,15 @@ config ARCH_MMAP_RND_BITS_MAX default 24 if ARM64_VA_BITS=39 default 27 if ARM64_VA_BITS=42 default 30 if ARM64_VA_BITS=47 - default 29 if ARM64_VA_BITS=48 && ARM64_64K_PAGES + default 29 if ARM64_VA_BITS=48 && (ARM64_64K_PAGES || ARM64_BOOT_TIME_PAGE_SIZE) default 31 if ARM64_VA_BITS=48 && ARM64_16K_PAGES default 33 if ARM64_VA_BITS=48 - default 14 if ARM64_64K_PAGES + default 14 if ARM64_64K_PAGES || ARM64_BOOT_TIME_PAGE_SIZE default 16 if ARM64_16K_PAGES default 18 config ARCH_MMAP_RND_COMPAT_BITS_MIN - default 7 if ARM64_64K_PAGES + default 7 if ARM64_64K_PAGES || ARM64_BOOT_TIME_PAGE_SIZE default 9 if ARM64_16K_PAGES default 11 @@ -362,6 +365,7 @@ config FIX_EARLYCON_MEM config PGTABLE_LEVELS int + default 4 if ARM64_BOOT_TIME_PAGE_SIZE # Advertise max supported levels default 2 if ARM64_16K_PAGES && ARM64_VA_BITS_36 default 2 if ARM64_64K_PAGES && ARM64_VA_BITS_42 default 3 if ARM64_64K_PAGES && (ARM64_VA_BITS_48 || ARM64_VA_BITS_52) @@ -1316,6 +1320,14 @@ config ARM64_64K_PAGES look-up. AArch32 emulation requires applications compiled with 64K aligned segments. +config ARM64_BOOT_TIME_PAGE_SIZE + bool "Boot-time selection" + select HAVE_PAGE_SIZE_64KB # Advertise largest page size to core + help + Select desired page size (4KB, 16KB or 64KB) at boot-time via the + kernel command line option "arm64.pagesize=4k", "arm64.pagesize=16k" + or "arm64.pagesize=64k". + endchoice choice @@ -1348,6 +1360,7 @@ config ARM64_VA_BITS_48 config ARM64_VA_BITS_52 bool "52-bit" depends on ARM64_PAN || !ARM64_SW_TTBR0_PAN + depends on !ARM64_BOOT_TIME_PAGE_SIZE help Enable 52-bit virtual addressing for userspace when explicitly requested via a hint to mmap(). The kernel will also use 52-bit @@ -1588,9 +1601,10 @@ config XEN # 4K | 27 | 12 | 15 | 10 | # 16K | 27 | 14 | 13 | 11 | # 64K | 29 | 16 | 13 | 13 | +# BOOT| 29 | 16 (max) | 13 | 13 | config ARCH_FORCE_MAX_ORDER int - default "13" if ARM64_64K_PAGES + default "13" if ARM64_64K_PAGES || ARM64_BOOT_TIME_PAGE_SIZE default "11" if ARM64_16K_PAGES default "10" help @@ -1663,7 +1677,7 @@ config ARM64_TAGGED_ADDR_ABI menuconfig COMPAT bool "Kernel support for 32-bit EL0" - depends on ARM64_4K_PAGES || EXPERT + depends on ARM64_4K_PAGES || ARM64_BOOT_TIME_PAGE_SIZE || EXPERT select HAVE_UID16 select OLD_SIGSUSPEND3 select COMPAT_OLD_SIGACTION diff --git a/arch/arm64/include/asm/kvm_hyp.h b/arch/arm64/include/asm/kvm_hyp.h index c838309e4ec47..9397a14642afa 100644 --- a/arch/arm64/include/asm/kvm_hyp.h +++ b/arch/arm64/include/asm/kvm_hyp.h @@ -145,4 +145,15 @@ extern unsigned long kvm_nvhe_sym(__icache_flags); extern unsigned int kvm_nvhe_sym(kvm_arm_vmid_bits); extern unsigned int kvm_nvhe_sym(kvm_host_sve_max_vl); +#ifdef CONFIG_ARM64_BOOT_TIME_PAGE_SIZE +extern int kvm_nvhe_sym(ptg_page_shift); +extern int kvm_nvhe_sym(ptg_pmd_shift); +extern int kvm_nvhe_sym(ptg_pud_shift); +extern int kvm_nvhe_sym(ptg_p4d_shift); +extern int kvm_nvhe_sym(ptg_pgdir_shift); +extern int kvm_nvhe_sym(ptg_cont_pte_shift); +extern int kvm_nvhe_sym(ptg_cont_pmd_shift); +extern int kvm_nvhe_sym(ptg_pgtable_levels); +#endif /* CONFIG_ARM64_BOOT_TIME_PAGE_SIZE */ + #endif /* __ARM64_KVM_HYP_H__ */ diff --git a/arch/arm64/include/asm/pgtable-geometry.h b/arch/arm64/include/asm/pgtable-geometry.h index 62fe125909c08..18a8c8d499ecc 100644 --- a/arch/arm64/include/asm/pgtable-geometry.h +++ b/arch/arm64/include/asm/pgtable-geometry.h @@ -6,16 +6,33 @@ #define ARM64_PAGE_SHIFT_16K 14 #define ARM64_PAGE_SHIFT_64K 16 +#ifdef CONFIG_ARM64_BOOT_TIME_PAGE_SIZE +#define PAGE_SHIFT_MIN ARM64_PAGE_SHIFT_4K +#define PAGE_SHIFT_MAX ARM64_PAGE_SHIFT_64K +#else /* CONFIG_ARM64_BOOT_TIME_PAGE_SIZE */ #define PAGE_SHIFT_MIN CONFIG_PAGE_SHIFT +#define PAGE_SHIFT_MAX CONFIG_PAGE_SHIFT +#endif /* CONFIG_ARM64_BOOT_TIME_PAGE_SIZE */ + #define PAGE_SIZE_MIN (_AC(1, UL) << PAGE_SHIFT_MIN) #define PAGE_MASK_MIN (~(PAGE_SIZE_MIN-1)) - -#define PAGE_SHIFT_MAX CONFIG_PAGE_SHIFT #define PAGE_SIZE_MAX (_AC(1, UL) << PAGE_SHIFT_MAX) #define PAGE_MASK_MAX (~(PAGE_SIZE_MAX-1)) #include +#ifdef CONFIG_ARM64_BOOT_TIME_PAGE_SIZE +#ifndef __ASSEMBLY__ +extern int ptg_page_shift; +extern int ptg_pmd_shift; +extern int ptg_pud_shift; +extern int ptg_p4d_shift; +extern int ptg_pgdir_shift; +extern int ptg_cont_pte_shift; +extern int ptg_cont_pmd_shift; +extern int ptg_pgtable_levels; +#endif /* __ASSEMBLY__ */ +#else /* CONFIG_ARM64_BOOT_TIME_PAGE_SIZE */ #define ptg_page_shift CONFIG_PAGE_SHIFT #define ptg_pmd_shift ARM64_HW_PGTABLE_LEVEL_SHIFT(2) #define ptg_pud_shift ARM64_HW_PGTABLE_LEVEL_SHIFT(1) @@ -24,5 +41,6 @@ #define ptg_cont_pte_shift (CONFIG_ARM64_CONT_PTE_SHIFT + PAGE_SHIFT) #define ptg_cont_pmd_shift (CONFIG_ARM64_CONT_PMD_SHIFT + PMD_SHIFT) #define ptg_pgtable_levels CONFIG_PGTABLE_LEVELS +#endif /* CONFIG_ARM64_BOOT_TIME_PAGE_SIZE */ #endif /* ASM_PGTABLE_GEOMETRY_H */ diff --git a/arch/arm64/include/asm/pgtable-hwdef.h b/arch/arm64/include/asm/pgtable-hwdef.h index ca8bcbc1fe220..da5404617acbf 100644 --- a/arch/arm64/include/asm/pgtable-hwdef.h +++ b/arch/arm64/include/asm/pgtable-hwdef.h @@ -52,7 +52,7 @@ #define PMD_SHIFT ptg_pmd_shift #define PMD_SIZE (_AC(1, UL) << PMD_SHIFT) #define PMD_MASK (~(PMD_SIZE-1)) -#define PTRS_PER_PMD (1 << (PAGE_SHIFT - 3)) +#define PTRS_PER_PMD (ptg_pgtable_levels > 2 ? (1 << (PAGE_SHIFT - 3)) : 1) #define MAX_PTRS_PER_PMD (1 << (PAGE_SHIFT_MAX - 3)) #endif @@ -63,7 +63,7 @@ #define PUD_SHIFT ptg_pud_shift #define PUD_SIZE (_AC(1, UL) << PUD_SHIFT) #define PUD_MASK (~(PUD_SIZE-1)) -#define PTRS_PER_PUD (1 << (PAGE_SHIFT - 3)) +#define PTRS_PER_PUD (ptg_pgtable_levels > 3 ? (1 << (PAGE_SHIFT - 3)) : 1) #define MAX_PTRS_PER_PUD (1 << (PAGE_SHIFT_MAX - 3)) #endif @@ -71,7 +71,7 @@ #define P4D_SHIFT ptg_p4d_shift #define P4D_SIZE (_AC(1, UL) << P4D_SHIFT) #define P4D_MASK (~(P4D_SIZE-1)) -#define PTRS_PER_P4D (1 << (PAGE_SHIFT - 3)) +#define PTRS_PER_P4D (ptg_pgtable_levels > 4 ? (1 << (PAGE_SHIFT - 3)) : 1) #define MAX_PTRS_PER_P4D (1 << (PAGE_SHIFT_MAX - 3)) #endif diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 8ead41da715b0..ad9f75f5cc29a 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -755,7 +755,7 @@ static inline unsigned long pmd_page_vaddr(pmd_t pmd) static __always_inline bool pgtable_l3_enabled(void) { - return true; + return ptg_pgtable_levels > 2; } static inline bool mm_pmd_folded(const struct mm_struct *mm) @@ -888,6 +888,8 @@ static inline bool pgtable_l3_enabled(void) { return false; } static __always_inline bool pgtable_l4_enabled(void) { + if (IS_ENABLED(CONFIG_ARM64_BOOT_TIME_PAGE_SIZE)) + return ptg_pgtable_levels > 3; if (CONFIG_PGTABLE_LEVELS > 4 || !IS_ENABLED(CONFIG_ARM64_LPA2)) return true; if (!alternative_has_cap_likely(ARM64_ALWAYS_BOOT)) @@ -935,6 +937,8 @@ static inline phys_addr_t p4d_page_paddr(p4d_t p4d) static inline pud_t *p4d_to_folded_pud(p4d_t *p4dp, unsigned long addr) { + if (IS_ENABLED(CONFIG_ARM64_BOOT_TIME_PAGE_SIZE)) + return (pud_t *)p4dp; return (pud_t *)PTR_ALIGN_DOWN(p4dp, PAGE_SIZE) + pud_index(addr); } @@ -1014,6 +1018,8 @@ static inline bool pgtable_l4_enabled(void) { return false; } static __always_inline bool pgtable_l5_enabled(void) { + if (IS_ENABLED(CONFIG_ARM64_BOOT_TIME_PAGE_SIZE)) + return ptg_pgtable_levels > 4; if (!alternative_has_cap_likely(ARM64_ALWAYS_BOOT)) return vabits_actual == VA_BITS; return alternative_has_cap_unlikely(ARM64_HAS_VA52); @@ -1059,6 +1065,8 @@ static inline phys_addr_t pgd_page_paddr(pgd_t pgd) static inline p4d_t *pgd_to_folded_p4d(pgd_t *pgdp, unsigned long addr) { + if (IS_ENABLED(CONFIG_ARM64_BOOT_TIME_PAGE_SIZE)) + return (p4d_t *)pgdp; return (p4d_t *)PTR_ALIGN_DOWN(pgdp, PAGE_SIZE) + p4d_index(addr); } diff --git a/arch/arm64/include/asm/sparsemem.h b/arch/arm64/include/asm/sparsemem.h index a05fdd54014f7..2daf1263ba638 100644 --- a/arch/arm64/include/asm/sparsemem.h +++ b/arch/arm64/include/asm/sparsemem.h @@ -17,6 +17,10 @@ * entries could not be created for vmemmap mappings. * 16K follows 4K for simplicity. */ +#ifdef CONFIG_ARM64_BOOT_TIME_PAGE_SIZE +#define SECTION_SIZE_BITS 29 +#else #define SECTION_SIZE_BITS (PAGE_SIZE == SZ_64K ? 29 : 27) +#endif #endif diff --git a/arch/arm64/kernel/image-vars.h b/arch/arm64/kernel/image-vars.h index a168f3337446f..9968320f83bc4 100644 --- a/arch/arm64/kernel/image-vars.h +++ b/arch/arm64/kernel/image-vars.h @@ -36,6 +36,17 @@ PROVIDE(__pi___memcpy = __pi_memcpy); PROVIDE(__pi___memmove = __pi_memmove); PROVIDE(__pi___memset = __pi_memset); +#ifdef CONFIG_ARM64_BOOT_TIME_PAGE_SIZE +PROVIDE(__pi_ptg_page_shift = ptg_page_shift); +PROVIDE(__pi_ptg_pmd_shift = ptg_pmd_shift); +PROVIDE(__pi_ptg_pud_shift = ptg_pud_shift); +PROVIDE(__pi_ptg_p4d_shift = ptg_p4d_shift); +PROVIDE(__pi_ptg_pgdir_shift = ptg_pgdir_shift); +PROVIDE(__pi_ptg_cont_pte_shift = ptg_cont_pte_shift); +PROVIDE(__pi_ptg_cont_pmd_shift = ptg_cont_pmd_shift); +PROVIDE(__pi_ptg_pgtable_levels = ptg_pgtable_levels); +#endif /* CONFIG_ARM64_BOOT_TIME_PAGE_SIZE */ + PROVIDE(__pi_id_aa64isar1_override = id_aa64isar1_override); PROVIDE(__pi_id_aa64isar2_override = id_aa64isar2_override); PROVIDE(__pi_id_aa64mmfr0_override = id_aa64mmfr0_override); diff --git a/arch/arm64/kernel/image.h b/arch/arm64/kernel/image.h index 7bc3ba8979019..01502fc3b891b 100644 --- a/arch/arm64/kernel/image.h +++ b/arch/arm64/kernel/image.h @@ -47,7 +47,11 @@ #define __HEAD_FLAG_BE ARM64_IMAGE_FLAG_LE #endif +#ifdef CONFIG_ARM64_BOOT_TIME_PAGE_SIZE +#define __HEAD_FLAG_PAGE_SIZE 0 +#else #define __HEAD_FLAG_PAGE_SIZE ((PAGE_SHIFT - 10) / 2) +#endif #define __HEAD_FLAG_PHYS_BASE 1 diff --git a/arch/arm64/kernel/pi/map_kernel.c b/arch/arm64/kernel/pi/map_kernel.c index deb8cd50b0b0c..22b3c70e04f9c 100644 --- a/arch/arm64/kernel/pi/map_kernel.c +++ b/arch/arm64/kernel/pi/map_kernel.c @@ -221,6 +221,49 @@ static void __init map_fdt(u64 fdt, int page_shift) dsb(ishst); } +#ifdef CONFIG_ARM64_BOOT_TIME_PAGE_SIZE +static void __init ptg_init(int page_shift) +{ + ptg_pgtable_levels = + __ARM64_HW_PGTABLE_LEVELS(page_shift, CONFIG_ARM64_VA_BITS); + + ptg_pgdir_shift = __ARM64_HW_PGTABLE_LEVEL_SHIFT(page_shift, + 4 - ptg_pgtable_levels); + + ptg_p4d_shift = ptg_pgtable_levels >= 5 ? + __ARM64_HW_PGTABLE_LEVEL_SHIFT(page_shift, 0) : + ptg_pgdir_shift; + + ptg_pud_shift = ptg_pgtable_levels >= 4 ? + __ARM64_HW_PGTABLE_LEVEL_SHIFT(page_shift, 1) : + ptg_pgdir_shift; + + ptg_pmd_shift = ptg_pgtable_levels >= 3 ? + __ARM64_HW_PGTABLE_LEVEL_SHIFT(page_shift, 2) : + ptg_pgdir_shift; + + ptg_page_shift = page_shift; + + switch (page_shift) { + case ARM64_PAGE_SHIFT_64K: + ptg_cont_pte_shift = ptg_page_shift + 5; + ptg_cont_pmd_shift = ptg_pmd_shift + 5; + break; + case ARM64_PAGE_SHIFT_16K: + ptg_cont_pte_shift = ptg_page_shift + 7; + ptg_cont_pmd_shift = ptg_pmd_shift + 5; + break; + default: /* ARM64_PAGE_SHIFT_4K */ + ptg_cont_pte_shift = ptg_page_shift + 4; + ptg_cont_pmd_shift = ptg_pmd_shift + 4; + } +} +#else /* CONFIG_ARM64_BOOT_TIME_PAGE_SIZE */ +static inline void ptg_init(int page_shift) +{ +} +#endif /* CONFIG_ARM64_BOOT_TIME_PAGE_SIZE */ + asmlinkage void __init early_map_kernel(u64 boot_status, void *fdt) { static char const chosen_str[] __initconst = "/chosen"; @@ -247,6 +290,8 @@ asmlinkage void __init early_map_kernel(u64 boot_status, void *fdt) if (!page_shift) page_shift = early_page_shift; + ptg_init(page_shift); + if (va_bits > 48) { u64 page_size = early_page_size(page_shift); diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c index 9bef7638342ef..c835a50b8b768 100644 --- a/arch/arm64/kvm/arm.c +++ b/arch/arm64/kvm/arm.c @@ -2424,6 +2424,16 @@ static void kvm_hyp_init_symbols(void) kvm_nvhe_sym(id_aa64smfr0_el1_sys_val) = read_sanitised_ftr_reg(SYS_ID_AA64SMFR0_EL1); kvm_nvhe_sym(__icache_flags) = __icache_flags; kvm_nvhe_sym(kvm_arm_vmid_bits) = kvm_arm_vmid_bits; +#ifdef CONFIG_ARM64_BOOT_TIME_PAGE_SIZE + kvm_nvhe_sym(ptg_page_shift) = ptg_page_shift; + kvm_nvhe_sym(ptg_pmd_shift) = ptg_pmd_shift; + kvm_nvhe_sym(ptg_pud_shift) = ptg_pud_shift; + kvm_nvhe_sym(ptg_p4d_shift) = ptg_p4d_shift; + kvm_nvhe_sym(ptg_pgdir_shift) = ptg_pgdir_shift; + kvm_nvhe_sym(ptg_cont_pte_shift) = ptg_cont_pte_shift; + kvm_nvhe_sym(ptg_cont_pmd_shift) = ptg_cont_pmd_shift; + kvm_nvhe_sym(ptg_pgtable_levels) = ptg_pgtable_levels; +#endif /* CONFIG_ARM64_BOOT_TIME_PAGE_SIZE */ } static int __init kvm_hyp_init_protection(u32 hyp_va_bits) diff --git a/arch/arm64/kvm/hyp/nvhe/Makefile b/arch/arm64/kvm/hyp/nvhe/Makefile index b43426a493df5..a8fcbb84c7996 100644 --- a/arch/arm64/kvm/hyp/nvhe/Makefile +++ b/arch/arm64/kvm/hyp/nvhe/Makefile @@ -27,6 +27,7 @@ hyp-obj-y := timer-sr.o sysreg-sr.o debug-sr.o switch.o tlb.o hyp-init.o host.o cache.o setup.o mm.o mem_protect.o sys_regs.o pkvm.o stacktrace.o ffa.o hyp-obj-y += ../vgic-v3-sr.o ../aarch32.o ../vgic-v2-cpuif-proxy.o ../entry.o \ ../fpsimd.o ../hyp-entry.o ../exception.o ../pgtable.o +hyp-obj-$(CONFIG_ARM64_BOOT_TIME_PAGE_SIZE) += pgtable-geometry.o hyp-obj-$(CONFIG_LIST_HARDENED) += list_debug.o hyp-obj-y += $(lib-objs) diff --git a/arch/arm64/kvm/hyp/nvhe/pgtable-geometry.c b/arch/arm64/kvm/hyp/nvhe/pgtable-geometry.c new file mode 100644 index 0000000000000..17f807450a31a --- /dev/null +++ b/arch/arm64/kvm/hyp/nvhe/pgtable-geometry.c @@ -0,0 +1,16 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright (C) 2024 ARM Ltd. + */ + +#include +#include + +int ptg_page_shift __ro_after_init; +int ptg_pmd_shift __ro_after_init; +int ptg_pud_shift __ro_after_init; +int ptg_p4d_shift __ro_after_init; +int ptg_pgdir_shift __ro_after_init; +int ptg_cont_pte_shift __ro_after_init; +int ptg_cont_pmd_shift __ro_after_init; +int ptg_pgtable_levels __ro_after_init; diff --git a/arch/arm64/mm/Makefile b/arch/arm64/mm/Makefile index 60454256945b8..2ba30d06b35fe 100644 --- a/arch/arm64/mm/Makefile +++ b/arch/arm64/mm/Makefile @@ -3,6 +3,7 @@ obj-y := dma-mapping.o extable.o fault.o init.o \ cache.o copypage.o flush.o \ ioremap.o mmap.o pgd.o mmu.o \ context.o proc.o pageattr.o fixmap.o +obj-$(CONFIG_ARM64_BOOT_TIME_PAGE_SIZE) += pgtable-geometry.o obj-$(CONFIG_ARM64_CONTPTE) += contpte.o obj-$(CONFIG_HUGETLB_PAGE) += hugetlbpage.o obj-$(CONFIG_PTDUMP_CORE) += ptdump.o diff --git a/arch/arm64/mm/pgd.c b/arch/arm64/mm/pgd.c index 4b106510358b1..c052d0dcb0c69 100644 --- a/arch/arm64/mm/pgd.c +++ b/arch/arm64/mm/pgd.c @@ -21,10 +21,12 @@ static bool pgdir_is_page_size(void) { if (PGD_SIZE == PAGE_SIZE) return true; - if (CONFIG_PGTABLE_LEVELS == 4) - return !pgtable_l4_enabled(); - if (CONFIG_PGTABLE_LEVELS == 5) - return !pgtable_l5_enabled(); + if (!IS_ENABLED(CONFIG_ARM64_BOOT_TIME_PAGE_SIZE)) { + if (CONFIG_PGTABLE_LEVELS == 4) + return !pgtable_l4_enabled(); + if (CONFIG_PGTABLE_LEVELS == 5) + return !pgtable_l5_enabled(); + } return false; } diff --git a/arch/arm64/mm/pgtable-geometry.c b/arch/arm64/mm/pgtable-geometry.c new file mode 100644 index 0000000000000..ba50637f1e9d0 --- /dev/null +++ b/arch/arm64/mm/pgtable-geometry.c @@ -0,0 +1,24 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright (C) 2024 ARM Ltd. + */ + +#include +#include + +/* + * TODO: These should be __ro_after_init, but we need to write to them from the + * pi code where they are mapped in the early page table as read-only. + * __ro_after_init doesn't become writable until later when the swapper pgtable + * is fully set up. We should update the early page table to map __ro_after_init + * as read-write. + */ + +int ptg_page_shift __read_mostly; +int ptg_pmd_shift __read_mostly; +int ptg_pud_shift __read_mostly; +int ptg_p4d_shift __read_mostly; +int ptg_pgdir_shift __read_mostly; +int ptg_cont_pte_shift __read_mostly; +int ptg_cont_pmd_shift __read_mostly; +int ptg_pgtable_levels __read_mostly; diff --git a/drivers/firmware/efi/libstub/arm64.c b/drivers/firmware/efi/libstub/arm64.c index e57cd3de0a00f..8db9dba7d5423 100644 --- a/drivers/firmware/efi/libstub/arm64.c +++ b/drivers/firmware/efi/libstub/arm64.c @@ -68,7 +68,8 @@ efi_status_t check_platform_features(void) efi_novamap = true; /* UEFI mandates support for 4 KB granularity, no need to check */ - if (IS_ENABLED(CONFIG_ARM64_4K_PAGES)) + if (IS_ENABLED(CONFIG_ARM64_4K_PAGES) || + IS_ENABLED(CONFIG_ARM64_BOOT_TIME_PAGE_SIZE)) return EFI_SUCCESS; tg = (read_cpuid(ID_AA64MMFR0_EL1) >> ID_AA64MMFR0_EL1_TGRAN_SHIFT) & 0xf;