From patchwork Wed Feb 14 12:29:12 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13556421 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id D7815C48BC1 for ; Wed, 14 Feb 2024 12:32:25 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=RCNEgOH0+ePlFzddtC3zPUdsKD3PGUrVtU25Yo4n49k=; b=AXYR1GRh5xV3UMrq5+GEyfXixK RBBe/bEfrkNXmhXZgGCn3e17dn/HIB0DbvdLH0JhuozXNOjxW4XXOmpi76Qo65YFfF3ZvT8AJVF5f jWCAUCBWH18IAA67TG0eY7wBy/+35QSP4NFfXswFhEd+V25FCg2cwg87uAZas09wgSI1yB0yDGay3 Ix5YmbAKhDN2OrEkPR/0ENgNUXlYLcQRjikodIswo3st4QKVnFCCkOSpKhtw4FP7EVQTRwqZVu5uQ 0lyPcXIZTnft/UOiN73EjJke1VQ0mXm7l8YNvK/g5bDmmhUc8Dra6iKBiHsCMZCUd8lrW7bw6SGfE yz0ZvRqg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1raEQu-0000000CpRL-0rDI; Wed, 14 Feb 2024 12:32:12 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1raEPU-0000000Coqj-3O0S for linux-arm-kernel@bombadil.infradead.org; Wed, 14 Feb 2024 12:30:44 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=nuKS8y64zOxLGqqXlUa7rXS63dgERVoqcE7CokOkmPU=; b=TQWng2/pR59tLK66LLxtJka/lZ gtSjniN1uYcuwx9JGiYzyO4LLb5tjia2rjfR9oAu7kKbo38TvN70ij6y6gepTlwq0rgTYDEoYzAwj J2yGHsTVcxB6te/ypH7lLxolGup+RbyojjihX2io51KxwI4gUKLxnh9WgLyUSHAfT6ocSzemFWO3L 4LUoDzd5DxzTapXVuMA+q0hhZu/xpDYpt8dU2GCac5c/w3j8JofhYLlhPoS8UqkZRytTIlUrEX6pb XdSejLa3QrrrziWn1kpgl/9cC2T9jL6H+J8I0r9M7IDVQWGkYa8AgR8UXpUkYGCzBqoeo/T5dvNZS fbWhxPIA==; Received: from mail-wm1-x349.google.com ([2a00:1450:4864:20::349]) by desiato.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1raEPP-0000000GE4S-1Jce for linux-arm-kernel@lists.infradead.org; Wed, 14 Feb 2024 12:30:43 +0000 Received: by mail-wm1-x349.google.com with SMTP id 5b1f17b1804b1-40fd153b5c3so3279875e9.1 for ; Wed, 14 Feb 2024 04:30:39 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1707913838; x=1708518638; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=nuKS8y64zOxLGqqXlUa7rXS63dgERVoqcE7CokOkmPU=; b=IFnJfWcCXi1i4dCUawPZVxv7Tl38889yXcpU4QdkHlMlEPBjo0EI3vVdBphqVqkbQk wH5gWcqhgYADDl6uWTuPa317t+Eu0Q9IdG0RFWN55Zg+KF12fYsA5ZA4VGAk1omMXb61 kqfMU2L2WpOuFkgOdDxi64iNP48Eif7TiLkkMh16PC1yIVlHG1N9if4GlV9disUqJjrr 5AtXrpCgM8aqhTmOv5Q2wJ/flnFxJUAcx7D/L31WLtc/rABtP8Dn73pxWY6llllOMDhR C3YBhEkrhQ4b7+CEZp2PxJTXYN+WtOC78yFwK3XAzJGQRzdZNJftF12FAO4gbd3sUVsL SVmQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707913838; x=1708518638; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=nuKS8y64zOxLGqqXlUa7rXS63dgERVoqcE7CokOkmPU=; b=P7uJdhum2SUTjyawEHfXkeiX4/HdcIAJ0NRKDQ0GEB/9k/gxs20WwSE+GWxGbofqaW 6nXcEmrqlrv2apxsBIteruhEh2KQ1a5hwIhPMnHQWZYTrtDKaO4toL8bnzLtxFY1cnhh wvP7FKTbTq8mV3yk1gRNlw6bQahoW/eQ/Kkfmd38PuRZejLbqf1j7qSFA9Ft1dm30VQi QOOVjyBdCUx0ow0BGEzdAdYbAMXobKy1s4nXldHT6ipk01KElyWaP/f02OPbDKNh457E pgFQlYC+gpyJDB9nBhz7iDcnGmegJOJYjUr8AQRrbJprQQ5Km2DyD8PsI05m2iu3/bDU jGeQ== X-Gm-Message-State: AOJu0YxNAwUBNZpDsrTfdWAfXYbZBXOoiu/4bsSC70W675lMxwlZWVa4 g9gJAl1rZrItryndNZk5bxmLeNRDjEWEyVzwLHV3rOCLpv2dMBZ9z77PjjuIRe32JX2h/j4XIFP 8e/9fSk7ZnWVxVFofhWLSHaefiNmaB0taNFaBRgyJ8MPSexIm2PNZJRnUM6L+HvtWOpOqYFwdtW kDg7lRH/4gzPbFrFfjvUYJ3Pm/ZbepFTpH45bsNtce X-Google-Smtp-Source: AGHT+IFbTt880XqJiblCj703Fo9/VEONF5xI3o9W5xQLt4umZWh01WOdIGMJF1jUMIuCpCFWzukz+gnq X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:600c:3ba0:b0:411:c45a:3926 with SMTP id n32-20020a05600c3ba000b00411c45a3926mr44468wms.1.1707913837929; Wed, 14 Feb 2024 04:30:37 -0800 (PST) Date: Wed, 14 Feb 2024 13:29:12 +0100 In-Reply-To: <20240214122845.2033971-45-ardb+git@google.com> Mime-Version: 1.0 References: <20240214122845.2033971-45-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=6997; i=ardb@kernel.org; h=from:subject; bh=JARDPVhZMz1ESavwKrqfgoYfE1bGtajfQzEL+oG6cGE=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIfXMJqnL2zcoNv06EyzOEqS9+/j1vrPMPJnT5v+vssjcL vqx7HhCRykLgxgHg6yYIovA7L/vdp6eKFXrPEsWZg4rE8gQBi5OAZhIyWZGhnvVFw1+Mp2eNWWd /DXvDtWqS97v4kQSEi7vfqY76auaMj/D/zjVsAZh3a2LE300AjRtOK3XFd/LOBho+Lu05vmF9Ce e3AA= X-Mailer: git-send-email 2.43.0.687.g38aa6559b0-goog Message-ID: <20240214122845.2033971-71-ardb+git@google.com> Subject: [PATCH v8 26/43] arm64: mm: Add feature override support for LVA From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240214_123039_684782_B39AC1BB X-CRM114-Status: GOOD ( 21.90 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel Add support for overriding the VARange field of the MMFR2 CPU ID register. This permits the associated LVA feature to be overridden early enough for the boot code that creates the kernel mapping to take it into account. Given that LPA2 implies LVA, disabling the latter should disable the former as well. So override the ID_AA64MMFR0.TGran field of the current page size as well if it advertises support for 52-bit addressing. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/assembler.h | 17 ++++++----- arch/arm64/include/asm/cpufeature.h | 4 +++ arch/arm64/kernel/cpufeature.c | 8 +++-- arch/arm64/kernel/image-vars.h | 2 ++ arch/arm64/kernel/pi/idreg-override.c | 31 ++++++++++++++++++++ 5 files changed, 53 insertions(+), 9 deletions(-) diff --git a/arch/arm64/include/asm/assembler.h b/arch/arm64/include/asm/assembler.h index 6a467c694039..68a99b116256 100644 --- a/arch/arm64/include/asm/assembler.h +++ b/arch/arm64/include/asm/assembler.h @@ -576,18 +576,21 @@ alternative_endif .endm /* - * Offset ttbr1 to allow for 48-bit kernel VAs set with 52-bit PTRS_PER_PGD. + * If the kernel is built for 52-bit virtual addressing but the hardware only + * supports 48 bits, we cannot program the pgdir address into TTBR1 directly, + * but we have to add an offset so that the TTBR1 address corresponds with the + * pgdir entry that covers the lowest 48-bit addressable VA. + * * orr is used as it can cover the immediate value (and is idempotent). - * In future this may be nop'ed out when dealing with 52-bit kernel VAs. * ttbr: Value of ttbr to set, modified. */ .macro offset_ttbr1, ttbr, tmp #ifdef CONFIG_ARM64_VA_BITS_52 - mrs_s \tmp, SYS_ID_AA64MMFR2_EL1 - and \tmp, \tmp, #(0xf << ID_AA64MMFR2_EL1_VARange_SHIFT) - cbnz \tmp, .Lskipoffs_\@ - orr \ttbr, \ttbr, #TTBR1_BADDR_4852_OFFSET -.Lskipoffs_\@ : + mrs \tmp, tcr_el1 + and \tmp, \tmp, #TCR_T1SZ_MASK + cmp \tmp, #TCR_T1SZ(VA_BITS_MIN) + orr \tmp, \ttbr, #TTBR1_BADDR_4852_OFFSET + csel \ttbr, \tmp, \ttbr, eq #endif .endm diff --git a/arch/arm64/include/asm/cpufeature.h b/arch/arm64/include/asm/cpufeature.h index 4f4dc5496ee3..a2ac31aecdd9 100644 --- a/arch/arm64/include/asm/cpufeature.h +++ b/arch/arm64/include/asm/cpufeature.h @@ -906,7 +906,9 @@ static inline unsigned int get_vmid_bits(u64 mmfr1) s64 arm64_ftr_safe_value(const struct arm64_ftr_bits *ftrp, s64 new, s64 cur); struct arm64_ftr_reg *get_arm64_ftr_reg(u32 sys_id); +extern struct arm64_ftr_override id_aa64mmfr0_override; extern struct arm64_ftr_override id_aa64mmfr1_override; +extern struct arm64_ftr_override id_aa64mmfr2_override; extern struct arm64_ftr_override id_aa64pfr0_override; extern struct arm64_ftr_override id_aa64pfr1_override; extern struct arm64_ftr_override id_aa64zfr0_override; @@ -1000,6 +1002,8 @@ static inline bool cpu_has_lva(void) u64 mmfr2; mmfr2 = read_sysreg_s(SYS_ID_AA64MMFR2_EL1); + mmfr2 &= ~id_aa64mmfr2_override.mask; + mmfr2 |= id_aa64mmfr2_override.val; return cpuid_feature_extract_unsigned_field(mmfr2, ID_AA64MMFR2_EL1_VARange_SHIFT); } diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c index 8eb8c7f7b317..ed9670d8360c 100644 --- a/arch/arm64/kernel/cpufeature.c +++ b/arch/arm64/kernel/cpufeature.c @@ -655,7 +655,9 @@ static const struct arm64_ftr_bits ftr_raz[] = { #define ARM64_FTR_REG(id, table) \ __ARM64_FTR_REG_OVERRIDE(#id, id, table, &no_override) +struct arm64_ftr_override id_aa64mmfr0_override; struct arm64_ftr_override id_aa64mmfr1_override; +struct arm64_ftr_override id_aa64mmfr2_override; struct arm64_ftr_override id_aa64pfr0_override; struct arm64_ftr_override id_aa64pfr1_override; struct arm64_ftr_override id_aa64zfr0_override; @@ -719,10 +721,12 @@ static const struct __ftr_reg_entry { &id_aa64isar2_override), /* Op1 = 0, CRn = 0, CRm = 7 */ - ARM64_FTR_REG(SYS_ID_AA64MMFR0_EL1, ftr_id_aa64mmfr0), + ARM64_FTR_REG_OVERRIDE(SYS_ID_AA64MMFR0_EL1, ftr_id_aa64mmfr0, + &id_aa64mmfr0_override), ARM64_FTR_REG_OVERRIDE(SYS_ID_AA64MMFR1_EL1, ftr_id_aa64mmfr1, &id_aa64mmfr1_override), - ARM64_FTR_REG(SYS_ID_AA64MMFR2_EL1, ftr_id_aa64mmfr2), + ARM64_FTR_REG_OVERRIDE(SYS_ID_AA64MMFR2_EL1, ftr_id_aa64mmfr2, + &id_aa64mmfr2_override), ARM64_FTR_REG(SYS_ID_AA64MMFR3_EL1, ftr_id_aa64mmfr3), /* Op1 = 1, CRn = 0, CRm = 0 */ diff --git a/arch/arm64/kernel/image-vars.h b/arch/arm64/kernel/image-vars.h index 2b9d702abe0f..ff81f809a240 100644 --- a/arch/arm64/kernel/image-vars.h +++ b/arch/arm64/kernel/image-vars.h @@ -38,7 +38,9 @@ PROVIDE(__pi___memset = __pi_memset); PROVIDE(__pi_id_aa64isar1_override = id_aa64isar1_override); PROVIDE(__pi_id_aa64isar2_override = id_aa64isar2_override); +PROVIDE(__pi_id_aa64mmfr0_override = id_aa64mmfr0_override); PROVIDE(__pi_id_aa64mmfr1_override = id_aa64mmfr1_override); +PROVIDE(__pi_id_aa64mmfr2_override = id_aa64mmfr2_override); PROVIDE(__pi_id_aa64pfr0_override = id_aa64pfr0_override); PROVIDE(__pi_id_aa64pfr1_override = id_aa64pfr1_override); PROVIDE(__pi_id_aa64smfr0_override = id_aa64smfr0_override); diff --git a/arch/arm64/kernel/pi/idreg-override.c b/arch/arm64/kernel/pi/idreg-override.c index 1884bd936c0d..aad399796e81 100644 --- a/arch/arm64/kernel/pi/idreg-override.c +++ b/arch/arm64/kernel/pi/idreg-override.c @@ -59,6 +59,35 @@ static const struct ftr_set_desc mmfr1 __prel64_initconst = { }, }; + +static bool __init mmfr2_varange_filter(u64 val) +{ + int __maybe_unused feat; + + if (val) + return false; + +#ifdef CONFIG_ARM64_LPA2 + feat = cpuid_feature_extract_signed_field(read_sysreg(id_aa64mmfr0_el1), + ID_AA64MMFR0_EL1_TGRAN_SHIFT); + if (feat >= ID_AA64MMFR0_EL1_TGRAN_LPA2) { + id_aa64mmfr0_override.val |= + (ID_AA64MMFR0_EL1_TGRAN_LPA2 - 1) << ID_AA64MMFR0_EL1_TGRAN_SHIFT; + id_aa64mmfr0_override.mask |= 0xfU << ID_AA64MMFR0_EL1_TGRAN_SHIFT; + } +#endif + return true; +} + +static const struct ftr_set_desc mmfr2 __prel64_initconst = { + .name = "id_aa64mmfr2", + .override = &id_aa64mmfr2_override, + .fields = { + FIELD("varange", ID_AA64MMFR2_EL1_VARange_SHIFT, mmfr2_varange_filter), + {} + }, +}; + static bool __init pfr0_sve_filter(u64 val) { /* @@ -167,6 +196,7 @@ static const struct ftr_set_desc sw_features __prel64_initconst = { static const PREL64(const struct ftr_set_desc, reg) regs[] __prel64_initconst = { { &mmfr1 }, + { &mmfr2 }, { &pfr0 }, { &pfr1 }, { &isar1 }, @@ -192,6 +222,7 @@ static const struct { { "arm64.nomte", "id_aa64pfr1.mte=0" }, { "nokaslr", "arm64_sw.nokaslr=1" }, { "rodata=off", "arm64_sw.rodataoff=1" }, + { "arm64.nolva", "id_aa64mmfr2.varange=0" }, }; static int __init parse_hexdigit(const char *p, u64 *v)