From patchwork Mon Oct 14 10:59:03 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13834724 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B2C73D1A45A for ; Mon, 14 Oct 2024 11:02:21 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id AB1AA6B00ED; Mon, 14 Oct 2024 07:02:19 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9C6B76B00EF; Mon, 14 Oct 2024 07:02:19 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 864B46B00F0; Mon, 14 Oct 2024 07:02:19 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 66E066B00ED for ; Mon, 14 Oct 2024 07:02:19 -0400 (EDT) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 97E7F80DB5 for ; Mon, 14 Oct 2024 11:02:12 +0000 (UTC) X-FDA: 82671918714.29.D281452 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf06.hostedemail.com (Postfix) with ESMTP id 070D8180012 for ; Mon, 14 Oct 2024 11:02:12 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf06.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1728903691; a=rsa-sha256; cv=none; b=ptULGsFQWPVcJXpxFZl17huKO5cmX7UuOz2DH9vfkudtdJsfrG7I2TLdduzJibUQHs7ziN Wq8yJIuxET7omRSmfxbscqeVmKhAeMH9UVYFIMsGVWVbqYozSsZ7JtU5sd6QggakF/842X BRHBQJf2YG8wzCca6GFgIyi9ZjX+n88= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf06.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1728903691; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=cPtmn6bEPN9e8rdSq6k9AO9DyT36CNVCz9b2AR0mIuI=; b=Of91oKTfsa1uzFtbYRWasH6yNVbkaAQuKbH41W2J412cEwDS74FkOReMjM/Flbx07rjgcS k6ioPbzk0nbObDZe5ikhQdVG4Idn5XvZL5aR0B01z/5jOqIrGDkPIGKxUcXz1yMT1oDCfS XqtSszoj7CuGcPuKvktXG94mq5ao+t8= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 6C0D71756; Mon, 14 Oct 2024 04:02:46 -0700 (PDT) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.27]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 0FB423F51B; Mon, 14 Oct 2024 04:02:13 -0700 (PDT) From: Ryan Roberts To: Andrew Morton , Anshuman Khandual , Ard Biesheuvel , Catalin Marinas , David Hildenbrand , Greg Marsden , Ivan Ivanov , Kalesh Singh , Marc Zyngier , Mark Rutland , Matthias Brugger , Miroslav Benes , Oliver Upton , Will Deacon Cc: Ryan Roberts , kvmarm@lists.linux.dev, linux-arm-kernel@lists.infradead.org, linux-efi@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [RFC PATCH v1 56/57] arm64: Determine THREAD_SIZE at boot-time Date: Mon, 14 Oct 2024 11:59:03 +0100 Message-ID: <20241014105912.3207374-56-ryan.roberts@arm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20241014105912.3207374-1-ryan.roberts@arm.com> References: <20241014105514.3206191-1-ryan.roberts@arm.com> <20241014105912.3207374-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Queue-Id: 070D8180012 X-Rspamd-Server: rspam01 X-Stat-Signature: 8s8egkzz6y7dc4kx6m9o7mxt79uoc4d4 X-HE-Tag: 1728903732-707447 X-HE-Meta: U2FsdGVkX1+/PBKnUuOgNe3s5RGIVqn8E5M/bvqBhNO4fx/j0PR7C9y3UWrOKmKu2J2HKxCHv3ARN1ydEiONiEk1WT5rozNiaJFlgpLeXnSXEiJYNtaPQNwC3mMFRqxHFdFJ4fSxobH6jBDwGkVO2sYzCZ/rip/BA8N022PoD7rPj7YstK/1Mt0SO0uqXf3ib2b1S/wFL25mYZ4oMxFRX3EpKgWBdR4SMdIYHmvYHar/QJ/6Z2owug15aRhs1aCPoyDVRKECFPtWy9xJYKK2kbBt4f+xZWLZJezU8cazYN7wPLkq7cU08s4Oh5fCl3dTOHo9s9e8xCTqmkAVb6aYeW1FOxXs0mImd7AOTvgfTeuKOSjf1xs/NfiJoi9kOh1zCxU4rH8Fj5Y/tAxCr5w1LrZPVTFfuKA55VF7sgrkKH5O7mUEf/bdcvpobaDZT47+InJKoUR36CuCbTC8WZeESKrSSNHrj5Z0wCX+1a/txF/jeY4C7S2j1xZJ49TQ5N3+KX9fRzIYYLHSpcbbDQwd/Fuyj1qbwze9+nMs/8Y/HFecBIDRWChHrqn0cJb2Cu0XOdXVfHcPk5p0hmgXRxWbfAK6I1I7NILDOqbqoThOqCKVCQJWLpKS8O9UwR7eolIuXaY3t5zNuUPCoF86gYHcsorl0VpttWSdAYHDfX1EaceAcan9eBuqLqQZZUtOo3X8LJkOFe3q8QdkNcTN7sSnuVFyh0M3eFhlEOXKQU+aHI9x5Wv3AHqCCXsvEyK35l+TL3pMmkyv7dOTSKOu9A1fu58WCajTBPiZDTH8oDmcp4Y7eyD8FNEihmnwzg6+nsUZns8DiHdRu1ZQfqJDa0NGqoKoQcX0Nb5bFWjLLKbXF5NlPP5peIEIYbotxcygyIvXGOphkL1S6AWmp8M5vYm3UzdPG18N/WYtu+CxhVr93ObRap6eNimf8/Ae9wFAVfosT/H1HbQUEho8FSft0va XJ0nDr1E SQUqkc0nZmS0A+4bZR2LDxOBKbJyDj9lelNaW6DWTJmu/prklf2EAlACurkUf0byypBK/icVS1TxHM5vav+M9kTdCCO7DVYi/nO0DSlfgtVmt8kAVG9U8pgmpEkxwKi6mtc7/AnGcqE/O5XNIkA8jXmoL/3t9Zbyp+Ned9dAmOw4CWIhT+3OAFBHyHgmYkskpvDs+VwwgbdaMGIv1+YcOEg+C9vVIbzIc3KCS X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Since THREAD_SIZE depends on PAGE_SIZE when stacks are vmapped, we must defer the decision on THREAD_SIZE until we have selected PAGE_SIZE at boot. The one wrinkle is entry.S's requirement to have THREAD_SHIFT as an immediate in order to check that the stack has not overflowed without clobbering any registers, early in the exception handler. Solve this by patching alternatives. During early boot, all 3 options are NOPs until the alternative is patched in. So we forgo overflow checking until boot-cpu patching is complete. Signed-off-by: Ryan Roberts --- ***NOTE*** Any confused maintainers may want to read the cover note here for context: https://lore.kernel.org/all/20241014105514.3206191-1-ryan.roberts@arm.com/ arch/arm64/include/asm/assembler.h | 5 +++ arch/arm64/include/asm/efi.h | 2 +- arch/arm64/include/asm/memory.h | 51 +++++++++++++++++++++++++----- arch/arm64/kernel/efi.c | 2 +- arch/arm64/kernel/entry.S | 10 +++++- arch/arm64/kernel/head.S | 3 +- arch/arm64/kernel/vmlinux.lds.S | 4 +-- arch/arm64/kvm/hyp/nvhe/hyp.lds.S | 2 +- 8 files changed, 64 insertions(+), 15 deletions(-) diff --git a/arch/arm64/include/asm/assembler.h b/arch/arm64/include/asm/assembler.h index 0cfa7c3efd214..745328e7768b7 100644 --- a/arch/arm64/include/asm/assembler.h +++ b/arch/arm64/include/asm/assembler.h @@ -928,4 +928,9 @@ alternative_cb_end get_tg0 \val value_for_page_size \val, \val, (~(SZ_4K-1)), (~(SZ_16K-1)), (~(SZ_64K-1)) .endm + + .macro get_task_size, val + get_tg0 \val + value_for_page_size \val, \val, (1 << THREAD_SHIFT_4K), (1 << THREAD_SHIFT_16K), (1 << THREAD_SHIFT_64K) + .endm #endif /* __ASM_ASSEMBLER_H */ diff --git a/arch/arm64/include/asm/efi.h b/arch/arm64/include/asm/efi.h index bcd5622aa0968..913f599c14e40 100644 --- a/arch/arm64/include/asm/efi.h +++ b/arch/arm64/include/asm/efi.h @@ -68,7 +68,7 @@ void arch_efi_call_virt_teardown(void); * kernel need greater alignment than we require the segments to be padded to. */ #define EFI_KIMG_ALIGN \ - (SEGMENT_ALIGN > THREAD_ALIGN ? SEGMENT_ALIGN : THREAD_ALIGN) + (SEGMENT_ALIGN > THREAD_ALIGN_MAX ? SEGMENT_ALIGN : THREAD_ALIGN_MAX) /* * On arm64, we have to ensure that the initrd ends up in the linear region, diff --git a/arch/arm64/include/asm/memory.h b/arch/arm64/include/asm/memory.h index 5393a859183f7..e28f5700ef022 100644 --- a/arch/arm64/include/asm/memory.h +++ b/arch/arm64/include/asm/memory.h @@ -110,23 +110,56 @@ #define PAGE_END (_PAGE_END(VA_BITS_MIN)) #endif /* CONFIG_KASAN */ -#define MIN_THREAD_SHIFT (14 + KASAN_THREAD_SHIFT) +#define IDEAL_THREAD_SHIFT (14 + KASAN_THREAD_SHIFT) /* * VMAP'd stacks are allocated at page granularity, so we must ensure that such * stacks are a multiple of page size. */ -#if defined(CONFIG_VMAP_STACK) && (MIN_THREAD_SHIFT < PAGE_SHIFT) -#define THREAD_SHIFT PAGE_SHIFT + +#if defined(CONFIG_VMAP_STACK) +#define THREAD_SHIFT \ + (IDEAL_THREAD_SHIFT < PAGE_SHIFT ? PAGE_SHIFT : IDEAL_THREAD_SHIFT) +#if (IDEAL_THREAD_SHIFT < PAGE_SHIFT_MIN) +#define THREAD_SHIFT_MIN PAGE_SHIFT_MIN #else -#define THREAD_SHIFT MIN_THREAD_SHIFT +#define THREAD_SHIFT_MIN IDEAL_THREAD_SHIFT #endif - -#if THREAD_SHIFT >= PAGE_SHIFT -#define THREAD_SIZE_ORDER (THREAD_SHIFT - PAGE_SHIFT) +#if (IDEAL_THREAD_SHIFT < PAGE_SHIFT_MAX) +#define THREAD_SHIFT_MAX PAGE_SHIFT_MAX +#else +#define THREAD_SHIFT_MAX IDEAL_THREAD_SHIFT +#endif +#if (IDEAL_THREAD_SHIFT < ARM64_PAGE_SHIFT_4K) +#define THREAD_SHIFT_4K ARM64_PAGE_SHIFT_4K +#else +#define THREAD_SHIFT_4K IDEAL_THREAD_SHIFT +#endif +#if (IDEAL_THREAD_SHIFT < ARM64_PAGE_SHIFT_16K) +#define THREAD_SHIFT_16K ARM64_PAGE_SHIFT_16K +#else +#define THREAD_SHIFT_16K IDEAL_THREAD_SHIFT +#endif +#if (IDEAL_THREAD_SHIFT < ARM64_PAGE_SHIFT_64K) +#define THREAD_SHIFT_64K ARM64_PAGE_SHIFT_64K +#else +#define THREAD_SHIFT_64K IDEAL_THREAD_SHIFT #endif +#else +#define THREAD_SHIFT IDEAL_THREAD_SHIFT +#define THREAD_SHIFT_MIN IDEAL_THREAD_SHIFT +#define THREAD_SHIFT_MAX IDEAL_THREAD_SHIFT +#define THREAD_SHIFT_4K IDEAL_THREAD_SHIFT +#define THREAD_SHIFT_16K IDEAL_THREAD_SHIFT +#define THREAD_SHIFT_64K IDEAL_THREAD_SHIFT +#endif + +#define THREAD_SIZE_ORDER \ + (PAGE_SHIFT < THREAD_SHIFT ? THREAD_SHIFT - PAGE_SHIFT : 0) #define THREAD_SIZE (UL(1) << THREAD_SHIFT) +#define THREAD_SIZE_MIN (UL(1) << THREAD_SHIFT_MIN) +#define THREAD_SIZE_MAX (UL(1) << THREAD_SHIFT_MAX) /* * By aligning VMAP'd stacks to 2 * THREAD_SIZE, we can detect overflow by @@ -135,11 +168,13 @@ */ #ifdef CONFIG_VMAP_STACK #define THREAD_ALIGN (2 * THREAD_SIZE) +#define THREAD_ALIGN_MAX (2 * THREAD_SIZE_MAX) #else #define THREAD_ALIGN THREAD_SIZE +#define THREAD_ALIGN_MAX THREAD_SIZE_MAX #endif -#define IRQ_STACK_SIZE THREAD_SIZE +#define IRQ_STACK_SIZE THREAD_SIZE_MIN #define OVERFLOW_STACK_SIZE SZ_4K diff --git a/arch/arm64/kernel/efi.c b/arch/arm64/kernel/efi.c index 712718aed5dd9..ebc44b7e83199 100644 --- a/arch/arm64/kernel/efi.c +++ b/arch/arm64/kernel/efi.c @@ -197,7 +197,7 @@ bool efi_runtime_fixup_exception(struct pt_regs *regs, const char *msg) } /* EFI requires 8 KiB of stack space for runtime services */ -static_assert(THREAD_SIZE >= SZ_8K); +static_assert(THREAD_SIZE_MIN >= SZ_8K); static int __init arm64_efi_rt_init(void) { diff --git a/arch/arm64/kernel/entry.S b/arch/arm64/kernel/entry.S index ba47dc8672c04..1ab65e406b62e 100644 --- a/arch/arm64/kernel/entry.S +++ b/arch/arm64/kernel/entry.S @@ -62,7 +62,15 @@ */ add sp, sp, x0 // sp' = sp + x0 sub x0, sp, x0 // x0' = sp' - x0 = (sp + x0) - x0 = sp - tbnz x0, #THREAD_SHIFT, 0f +alternative_if ARM64_USE_PAGE_SIZE_4K + tbnz x0, #THREAD_SHIFT_4K, 0f +alternative_else_nop_endif +alternative_if ARM64_USE_PAGE_SIZE_16K + tbnz x0, #THREAD_SHIFT_16K, 0f +alternative_else_nop_endif +alternative_if ARM64_USE_PAGE_SIZE_64K + tbnz x0, #THREAD_SHIFT_64K, 0f +alternative_else_nop_endif sub x0, sp, x0 // x0'' = sp' - x0' = (sp + x0) - sp = x0 sub sp, sp, x0 // sp'' = sp' - x0 = (sp + x0) - x0 = sp b el\el\ht\()_\regsize\()_\label diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S index 761b7f5633e15..2530ee5cee548 100644 --- a/arch/arm64/kernel/head.S +++ b/arch/arm64/kernel/head.S @@ -198,7 +198,8 @@ SYM_CODE_END(preserve_boot_args) msr sp_el0, \tsk ldr \tmp1, [\tsk, #TSK_STACK] - add sp, \tmp1, #THREAD_SIZE + get_task_size \tmp2 + add sp, \tmp1, \tmp2 sub sp, sp, #PT_REGS_SIZE stp xzr, xzr, [sp, #S_STACKFRAME] diff --git a/arch/arm64/kernel/vmlinux.lds.S b/arch/arm64/kernel/vmlinux.lds.S index 09fcc234c0f77..937900a458a89 100644 --- a/arch/arm64/kernel/vmlinux.lds.S +++ b/arch/arm64/kernel/vmlinux.lds.S @@ -60,11 +60,11 @@ #define RO_EXCEPTION_TABLE_ALIGN 4 #define RUNTIME_DISCARD_EXIT +#include #include #include #include #include -#include #include #include "image.h" @@ -292,7 +292,7 @@ SECTIONS _data = .; _sdata = .; - RW_DATA(L1_CACHE_BYTES, PAGE_SIZE_MAX, THREAD_ALIGN) + RW_DATA(L1_CACHE_BYTES, PAGE_SIZE_MAX, THREAD_ALIGN_MAX) /* * Data written with the MMU off but read with the MMU on requires diff --git a/arch/arm64/kvm/hyp/nvhe/hyp.lds.S b/arch/arm64/kvm/hyp/nvhe/hyp.lds.S index 74c7c21626270..fe1fbfa8f8f05 100644 --- a/arch/arm64/kvm/hyp/nvhe/hyp.lds.S +++ b/arch/arm64/kvm/hyp/nvhe/hyp.lds.S @@ -7,9 +7,9 @@ */ #include +#include #include #include -#include SECTIONS { HYP_SECTION(.idmap.text)