From patchwork Wed Feb 14 12:29:11 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13556416 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id B3B80C48BC1 for ; Wed, 14 Feb 2024 12:32:22 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=WdzdNCPsWLDeW/rDWjlpaQzxmKTChTbwRQnufqmrrys=; b=kpbpCYj+a0vxFBGIbQLL/5oPhb pN5hYrgVUW2/5qL2fOKrnm0iUbLnm0lSttuvIn+Pb9I45GCO+jNF3Jhy36NwRLd2Mi/GhOyRFLAJy +oylmR8ui9g5gnyslvty393hbToWo9Gw0GGjxMRA/L3zs0eydZ5JwEXQBBfoKjYEGnMP2pD8wGOH2 uHwCjgduKEqTAUFaGU/7co15pdHgXj5NSnk6gqtpaUS0966QxAPXxnOH5U17uHmRRYoYDMGwsupU+ /frH7nvg1qGWOYR8eA6ODpXN5kYOyzRuai4B8pP0EkJ5rwrR8AcJ65AVjhxVlH6jGfkaInRcqXlY/ d/gju10Q==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1raEQr-0000000CpPg-2lVz; Wed, 14 Feb 2024 12:32:09 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1raEPT-0000000CopK-0q9M for linux-arm-kernel@bombadil.infradead.org; Wed, 14 Feb 2024 12:30:43 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=nK12yNXLs+EEiFX+OJdo6kPMOjaXyo97bMa/fX0wvqY=; b=lKT8YArxAwy2j7bEVjQzCYjvuA F8j1rJjhwOXdYFXt9Wgt5FVmupYP3XsSA/1FU4NVq0EjWfOqzHIcp3Mq0kGS0pga/ocf2f6eF9QvF OX/UvpElXZOmfoHkRrKUnsGvp/3lR77qRLY1SWp1SgCuXBkC0MiZ8WRUS69xLUk3jxnDSU1PmT+RJ kZYJG3K8Ioz1vPTTcjz8iOL7mh2Rtw+SP1ctgLDrbmAcf20GPfw8sT4xZl6vtbOs4dVKKmxexy0Ww xCz0dXBFvSc6k0LGPuBNrZOyAIC3us9w6drFLdH24Duj4IT5raxuGxyBz1eDbHfvDEs1LmMvtU9wb R8xhotOQ==; Received: from mail-wr1-x44a.google.com ([2a00:1450:4864:20::44a]) by desiato.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1raEPO-0000000GE3H-0IHV for linux-arm-kernel@lists.infradead.org; Wed, 14 Feb 2024 12:30:42 +0000 Received: by mail-wr1-x44a.google.com with SMTP id ffacd0b85a97d-33b2238ceceso2436400f8f.3 for ; Wed, 14 Feb 2024 04:30:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1707913836; x=1708518636; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=nK12yNXLs+EEiFX+OJdo6kPMOjaXyo97bMa/fX0wvqY=; b=pskp+jwjtlA/X5x6kV1uyhtfijznUeM1mxGwljKSoB6ToCleu95XydTTAga9/VCjVh 3Lvv9L7FZnP//yBnDir3/AmbcjjX8AbVJuuiBRtvVEy3IcfvBDx9Df+1oC4EuxWMbl7F wjNg9cuvnLAych4lcMs8JG3g0SRCb7MiArYusVv2gwwsQPR+VrOYP1Js18Wzgk64njxD I8BTL+l3K/cJL2r/bRlNCDqKViJt5pXwt/eac7TDWL5lJxDDa8pxFd6lnxZMtqU5oDOA eR9mWJY9Wrx0wEmkBSRpKkeXWCo/P33QAwkNLQWtZf9+BAMdS6jxsp+Qw9X2FksdZJme q25g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707913836; x=1708518636; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=nK12yNXLs+EEiFX+OJdo6kPMOjaXyo97bMa/fX0wvqY=; b=Z2+2ROTn/h3Iwk5iZ1HXArRNupre7we4XMZCt9oU/GBf7ky3EzijeHepYEn0yXLhho +Th0soG7ntnmma+xDQ9IoiSAiszw6RHZ8OZMojrzQ61OaCcGlZ0Etg0/1TeZWej1WGGu swiy9ezxWxf1RR9LIWqFIhZjpYm52NEZhWI8IwwOmnK+p+fpW64+4yG1PNIwem6bVIOk kvLDZjvr3/zD0/n6PinYb1MVAc+hav3Gp9aGsLy1DmGG6La4Fh4pk7lXozgYr94hBmCk V1M7n1Xvwmebfl4d8gVyJs+dBa94h9xep1NLQ01ABfauenA5BV7TjZSlRQ8xMJ1IuFwU CqIg== X-Gm-Message-State: AOJu0Yy6fGHlHlH+2NXYEWnYcqqMIuUK1+FMIXHmfeqsB1agutd8M2Pn by7jN6nGfcxdXoXWuwaL33r36ZNUniMJuWYI7WUqk38kezWXogGuCUKhXsbvKrHcoSBpuxhLzpJ FK78qI1uAYiFxIfz6xyix/r9F5uiMCUyp1dHh9Ucg/bl4wLKCpnaFL9PpcMAMZMaOhjpes4ORxj ntwzoICKtuFlbgarCFl06oczguET3uVA73ptRFOfC2 X-Google-Smtp-Source: AGHT+IFfLt4aWkGenSOZNiWfsjzkWuu6pvBPIAJJnVYxAcgJOyeUdSy7327DOxGWleIUAYlH6Gskx5tn X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a5d:6e51:0:b0:33c:e36a:5324 with SMTP id j17-20020a5d6e51000000b0033ce36a5324mr3165wrz.2.1707913835550; Wed, 14 Feb 2024 04:30:35 -0800 (PST) Date: Wed, 14 Feb 2024 13:29:11 +0100 In-Reply-To: <20240214122845.2033971-45-ardb+git@google.com> Mime-Version: 1.0 References: <20240214122845.2033971-45-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=10210; i=ardb@kernel.org; h=from:subject; bh=xvmS+exhEE1+bzLfyG/gprvQGJEXnuuOxi51VB8dpZ4=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIfXMJsmH1c8ObNieL87opbV/B9sa68lC90+Gbm6fsmxD/ jHLtGn5HaUsDGIcDLJiiiwCs/++23l6olSt8yxZmDmsTCBDGLg4BWAi224yMnyZdJF71oqe1gau v2KR7jejjr5tPvTpluJPzTpW58W2rB2MDF1nGRuXpT5t2/E2urcg18ngmODX4CYviSTbbwFZAU0 TuQA= X-Mailer: git-send-email 2.43.0.687.g38aa6559b0-goog Message-ID: <20240214122845.2033971-70-ardb+git@google.com> Subject: [PATCH v8 25/43] arm64: mm: Handle LVA support as a CPU feature From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240214_123038_608220_500CC59E X-CRM114-Status: GOOD ( 24.81 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel Currently, we detect CPU support for 52-bit virtual addressing (LVA) extremely early, before creating the kernel page tables or enabling the MMU. We cannot override the feature this early, and so large virtual addressing is always enabled on CPUs that implement support for it if the software support for it was enabled at build time. It also means we rely on non-trivial code in asm to deal with this feature. Given that both the ID map and the TTBR1 mapping of the kernel image are guaranteed to be 48-bit addressable, it is not actually necessary to enable support this early, and instead, we can model it as a CPU feature. That way, we can rely on code patching to get the correct TCR.T1SZ values programmed on secondary boot and resume from suspend. On the primary boot path, we simply enable the MMU with 48-bit virtual addressing initially, and update TCR.T1SZ if LVA is supported from C code, right before creating the kernel mapping. Given that TTBR1 still points to reserved_pg_dir at this point, updating TCR.T1SZ should be safe without the need for explicit TLB maintenance. Since this gets rid of all accesses to the vabits_actual variable from asm code that occurred before TCR.T1SZ had been programmed, we no longer have a need for this variable, and we can replace it with a C expression that produces the correct value directly, based on the value of TCR.T1SZ. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/cpufeature.h | 9 ++++++ arch/arm64/include/asm/memory.h | 13 ++++++++- arch/arm64/kernel/cpufeature.c | 13 +++++++++ arch/arm64/kernel/head.S | 29 +++++--------------- arch/arm64/kernel/image-vars.h | 1 - arch/arm64/kernel/pi/map_kernel.c | 3 ++ arch/arm64/kernel/sleep.S | 3 -- arch/arm64/mm/mmu.c | 5 ---- arch/arm64/mm/proc.S | 9 +++--- arch/arm64/tools/cpucaps | 1 + 10 files changed, 49 insertions(+), 37 deletions(-) diff --git a/arch/arm64/include/asm/cpufeature.h b/arch/arm64/include/asm/cpufeature.h index e3edae1825f3..4f4dc5496ee3 100644 --- a/arch/arm64/include/asm/cpufeature.h +++ b/arch/arm64/include/asm/cpufeature.h @@ -995,6 +995,15 @@ static inline bool cpu_has_pac(void) &id_aa64isar2_override); } +static inline bool cpu_has_lva(void) +{ + u64 mmfr2; + + mmfr2 = read_sysreg_s(SYS_ID_AA64MMFR2_EL1); + return cpuid_feature_extract_unsigned_field(mmfr2, + ID_AA64MMFR2_EL1_VARange_SHIFT); +} + #endif /* __ASSEMBLY__ */ #endif diff --git a/arch/arm64/include/asm/memory.h b/arch/arm64/include/asm/memory.h index 60904a6c4b42..9680d7444b3b 100644 --- a/arch/arm64/include/asm/memory.h +++ b/arch/arm64/include/asm/memory.h @@ -209,9 +209,20 @@ #include #include #include +#include + +static inline u64 __pure read_tcr(void) +{ + u64 tcr; + + // read_sysreg() uses asm volatile, so avoid it here + asm("mrs %0, tcr_el1" : "=r"(tcr)); + return tcr; +} #if VA_BITS > 48 -extern u64 vabits_actual; +// For reasons of #include hell, we can't use TCR_T1SZ_OFFSET/TCR_T1SZ_MASK here +#define vabits_actual (64 - ((read_tcr() >> 16) & 63)) #else #define vabits_actual ((u64)VA_BITS) #endif diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c index 7064cf13f226..8eb8c7f7b317 100644 --- a/arch/arm64/kernel/cpufeature.c +++ b/arch/arm64/kernel/cpufeature.c @@ -2692,6 +2692,19 @@ static const struct arm64_cpu_capabilities arm64_features[] = { .type = ARM64_CPUCAP_SYSTEM_FEATURE, .matches = has_lpa2, }, +#ifdef CONFIG_ARM64_VA_BITS_52 + { + .desc = "52-bit Virtual Addressing (LVA)", + .capability = ARM64_HAS_VA52, + .type = ARM64_CPUCAP_BOOT_CPU_FEATURE, + .sys_reg = SYS_ID_AA64MMFR2_EL1, + .sign = FTR_UNSIGNED, + .field_width = 4, + .field_pos = ID_AA64MMFR2_EL1_VARange_SHIFT, + .matches = has_cpuid_feature, + .min_field_value = ID_AA64MMFR2_EL1_VARange_52, + }, +#endif {}, }; diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S index 545b5d8976f4..e25351addfd0 100644 --- a/arch/arm64/kernel/head.S +++ b/arch/arm64/kernel/head.S @@ -80,7 +80,6 @@ * x19 primary_entry() .. start_kernel() whether we entered with the MMU on * x20 primary_entry() .. __primary_switch() CPU boot mode * x21 primary_entry() .. start_kernel() FDT pointer passed at boot in x0 - * x25 primary_entry() .. start_kernel() supported VA size */ SYM_CODE_START(primary_entry) bl record_mmu_state @@ -125,14 +124,6 @@ SYM_CODE_START(primary_entry) * On return, the CPU will be ready for the MMU to be turned on and * the TCR will have been set. */ -#if VA_BITS > 48 - mrs_s x0, SYS_ID_AA64MMFR2_EL1 - tst x0, ID_AA64MMFR2_EL1_VARange_MASK - mov x0, #VA_BITS - mov x25, #VA_BITS_MIN - csel x25, x25, x0, eq - mov x0, x25 -#endif bl __cpu_setup // initialise processor b __primary_switch SYM_CODE_END(primary_entry) @@ -242,11 +233,6 @@ SYM_FUNC_START_LOCAL(__primary_switched) mov x0, x20 bl set_cpu_boot_mode_flag -#if VA_BITS > 48 - adr_l x8, vabits_actual // Set this early so KASAN early init - str x25, [x8] // ... observes the correct value - dc civac, x8 // Make visible to booting secondaries -#endif #if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS) bl kasan_early_init #endif @@ -376,10 +362,13 @@ SYM_FUNC_START_LOCAL(secondary_startup) * Common entry point for secondary CPUs. */ mov x20, x0 // preserve boot mode + +#ifdef CONFIG_ARM64_VA_BITS_52 +alternative_if ARM64_HAS_VA52 bl __cpu_secondary_check52bitva -#if VA_BITS > 48 - ldr_l x0, vabits_actual +alternative_else_nop_endif #endif + bl __cpu_setup // initialise processor adrp x1, swapper_pg_dir adrp x2, idmap_pg_dir @@ -482,12 +471,8 @@ SYM_FUNC_START(__enable_mmu) ret SYM_FUNC_END(__enable_mmu) +#ifdef CONFIG_ARM64_VA_BITS_52 SYM_FUNC_START(__cpu_secondary_check52bitva) -#if VA_BITS > 48 - ldr_l x0, vabits_actual - cmp x0, #52 - b.ne 2f - mrs_s x0, SYS_ID_AA64MMFR2_EL1 and x0, x0, ID_AA64MMFR2_EL1_VARange_MASK cbnz x0, 2f @@ -498,9 +483,9 @@ SYM_FUNC_START(__cpu_secondary_check52bitva) wfi b 1b -#endif 2: ret SYM_FUNC_END(__cpu_secondary_check52bitva) +#endif SYM_FUNC_START_LOCAL(__no_granule_support) /* Indicate that this CPU can't boot and is stuck in the kernel */ diff --git a/arch/arm64/kernel/image-vars.h b/arch/arm64/kernel/image-vars.h index e140c5bda90b..2b9d702abe0f 100644 --- a/arch/arm64/kernel/image-vars.h +++ b/arch/arm64/kernel/image-vars.h @@ -36,7 +36,6 @@ PROVIDE(__pi___memcpy = __pi_memcpy); PROVIDE(__pi___memmove = __pi_memmove); PROVIDE(__pi___memset = __pi_memset); -PROVIDE(__pi_vabits_actual = vabits_actual); PROVIDE(__pi_id_aa64isar1_override = id_aa64isar1_override); PROVIDE(__pi_id_aa64isar2_override = id_aa64isar2_override); PROVIDE(__pi_id_aa64mmfr1_override = id_aa64mmfr1_override); diff --git a/arch/arm64/kernel/pi/map_kernel.c b/arch/arm64/kernel/pi/map_kernel.c index 4b76a007a50d..1853825aa29d 100644 --- a/arch/arm64/kernel/pi/map_kernel.c +++ b/arch/arm64/kernel/pi/map_kernel.c @@ -165,6 +165,9 @@ asmlinkage void __init early_map_kernel(u64 boot_status, void *fdt) chosen = fdt_path_offset(fdt, chosen_str); init_feature_override(boot_status, fdt, chosen); + if (VA_BITS > VA_BITS_MIN && cpu_has_lva()) + sysreg_clear_set(tcr_el1, TCR_T1SZ_MASK, TCR_T1SZ(VA_BITS)); + /* * The virtual KASLR displacement modulo 2MiB is decided by the * physical placement of the image, as otherwise, we might not be able diff --git a/arch/arm64/kernel/sleep.S b/arch/arm64/kernel/sleep.S index 2aa5129d8253..f093cdf71be1 100644 --- a/arch/arm64/kernel/sleep.S +++ b/arch/arm64/kernel/sleep.S @@ -102,9 +102,6 @@ SYM_CODE_START(cpu_resume) mov x0, xzr bl init_kernel_el mov x19, x0 // preserve boot mode -#if VA_BITS > 48 - ldr_l x0, vabits_actual -#endif bl __cpu_setup /* enable the MMU early - so we can access sleep_save_stash by va */ adrp x1, swapper_pg_dir diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index a3d23da92d87..ba00d0205447 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -45,11 +45,6 @@ #define NO_CONT_MAPPINGS BIT(1) #define NO_EXEC_MAPPINGS BIT(2) /* assumes FEAT_HPDS is not used */ -#if VA_BITS > 48 -u64 vabits_actual __ro_after_init = VA_BITS_MIN; -EXPORT_SYMBOL(vabits_actual); -#endif - u64 kimage_voffset __ro_after_init; EXPORT_SYMBOL(kimage_voffset); diff --git a/arch/arm64/mm/proc.S b/arch/arm64/mm/proc.S index 55c366dbda8f..d104ddab26a4 100644 --- a/arch/arm64/mm/proc.S +++ b/arch/arm64/mm/proc.S @@ -397,8 +397,6 @@ SYM_FUNC_END(idmap_kpti_install_ng_mappings) * * Initialise the processor for turning the MMU on. * - * Input: - * x0 - actual number of VA bits (ignored unless VA_BITS > 48) * Output: * Return in x0 the value of the SCTLR_EL1 register. */ @@ -422,16 +420,17 @@ SYM_FUNC_START(__cpu_setup) mair .req x17 tcr .req x16 mov_q mair, MAIR_EL1_SET - mov_q tcr, TCR_T0SZ(IDMAP_VA_BITS) | TCR_T1SZ(VA_BITS) | TCR_CACHE_FLAGS | \ + mov_q tcr, TCR_T0SZ(IDMAP_VA_BITS) | TCR_T1SZ(VA_BITS_MIN) | TCR_CACHE_FLAGS | \ TCR_SMP_FLAGS | TCR_TG_FLAGS | TCR_KASLR_FLAGS | TCR_ASID16 | \ TCR_TBI0 | TCR_A1 | TCR_KASAN_SW_FLAGS | TCR_MTE_FLAGS tcr_clear_errata_bits tcr, x9, x5 #ifdef CONFIG_ARM64_VA_BITS_52 - sub x9, xzr, x0 - add x9, x9, #64 + mov x9, #64 - VA_BITS +alternative_if ARM64_HAS_VA52 tcr_set_t1sz tcr, x9 +alternative_else_nop_endif #endif /* diff --git a/arch/arm64/tools/cpucaps b/arch/arm64/tools/cpucaps index b912b1409fc0..b370d808b3ec 100644 --- a/arch/arm64/tools/cpucaps +++ b/arch/arm64/tools/cpucaps @@ -50,6 +50,7 @@ HAS_STAGE2_FWB HAS_TCR2 HAS_TIDCP1 HAS_TLB_RANGE +HAS_VA52 HAS_VIRT_HOST_EXTN HAS_WFXT HW_DBM