From patchwork Wed Feb 14 12:29:18 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13556547 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 2B0CBC48BC1 for ; Wed, 14 Feb 2024 13:46:01 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=7DKATZ8jRW44o5yMhr2CfRlxPvpkbAbBWOrYBEJ/2i0=; b=uz8WiCzVjpu19mhrKM/TfLkgvL aL9meHlJKeEW1q5jwixuyhPO3jUapDOy3t1RaOSXb94QEWSxnw+nsAv3ossKqY4E1gj3HcETigYCR hwWIfXMAjPCG2Obg+/SHl2v38xMzLu4a2TmKvua1Qcv/UapqKTonrbI326SykYELjZHWQ5lnfiUDk onEYRHATXj49a1E5BumgNWa0RHL1B0xJLzqR7loLF7t25z0HAfZrcn1bH56KgGL4mHDVwOYpXQbCO QJCO2SEfZX2WICXp/YPYy9vzP+RFQKfj/DhlQztXlDsNSq7IgNjfZfpuJPynZDc6+mY7rdLMdpK91 oOhC4BvA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1raFa7-0000000D3gL-2YdA; Wed, 14 Feb 2024 13:45:47 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1raEPj-0000000Cp0s-1nbH for linux-arm-kernel@bombadil.infradead.org; Wed, 14 Feb 2024 12:30:59 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=9okug4N0q8cmBvmDgQedNs6ipr2VdqxXnTFEpYyzSg4=; b=gdcxRF2+QEJwzPXd2Vkg9nsyeS hbU0uQAUzgHfculOfB64KVMfH+Ey5QMexJC2ujOwq9Y9mySqfk2qGtRoMQ67mW7xHrxoDqZnJ81KI a1oQThMLct7mbC7wm3NZzqEQD6TfN5B5i2QbTwZQLyG4lxsOShnrk5sprygAT6fMdbG5Qu9Jly4V8 mmRPRei8jknsUfcmQok8Ybv7m+i1oLF7Byp0Z5A6QWYzOKH7yqs69LdI11AcrSQrLmH6okSdNAWhL m9+JwGM+7lY7fWLYmQgnZRjUMTQ252nxB8sZoBIcj6JHqEOJVqacJ2pEVBfl+YkM6PiIh/JkF0h0e s43mNV+Q==; Received: from mail-yb1-xb49.google.com ([2607:f8b0:4864:20::b49]) by desiato.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1raEPf-0000000GEAI-0qo2 for linux-arm-kernel@lists.infradead.org; Wed, 14 Feb 2024 12:30:58 +0000 Received: by mail-yb1-xb49.google.com with SMTP id 3f1490d57ef6-dc64f63d768so10024783276.2 for ; Wed, 14 Feb 2024 04:30:53 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1707913852; x=1708518652; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=9okug4N0q8cmBvmDgQedNs6ipr2VdqxXnTFEpYyzSg4=; b=lsxhwZGK6t2hTHzCbZWyyGykNJaGwoDJKER2zEfItrjR6z1ciRvcSZ05qXDZQR3yXo PTpO5RwpAq7zhrKuh7zB/awfBLq/YbmU6THHvShTd+O1N3c4ojWFX47/JQrFWszqI/mX flo5Zw3ZR5vvD5Lai/Hs4GYrIlVfjde0QPH5m8et13zu/f/ZSjAZ1ilKPLS7UtxgmTpC AFkPklWAb8tWYjapBMugnvoDQ4vM6NcerpzU6yO8ZwOwxhsLep+C/qLVsLMjBxfqo5il Sxo4l3l3C+yY0SvrzsjbCTRsud/9hglQkERjNqEZic2TRf5KqDD8IhRgPYNaRqfSUFuN DE+g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707913852; x=1708518652; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=9okug4N0q8cmBvmDgQedNs6ipr2VdqxXnTFEpYyzSg4=; b=gDfa19JdhjjK9uLkpvIKDsHC0JkpPg0rSm0W8kACsIo0AnqbLwnJf2eEk65EygOOMu tVtI4KfiiyV4ILrM3/auNjoGGy7mBx0To1gk4DfmV4ZxKLPY16P7L1sCTkQo02fixJaK vUYGG55oMwsKSoc41dnmHlUP46xuM2FwlSYgZIln65GbwAxBvPdUM5y3H0KUEmioZ8r6 Ng3/pMggEJ5M6ozOLwBLfkEHMZj8qIUSZcU65shLNlW6DySGrYxwtiFbj0XPogN8ng+p IPWqPZ1c8VDEm6nDZzgegze27/Tb90PBP0wZKr7XKF0QCNg6gcYF5aHjKxnyezg6jRiD Qjow== X-Gm-Message-State: AOJu0Yzij4Twq+0JnvqjaHRbYQvj6S+ewmINbD3VOulRF5kgv7UQbK8X shrA29CrpgMs0B007njGeV6q2PyRIQCko+ReWXXYCpUn02j8m8ckZfzNnv0GijELrzyTadq3RVL ze/aasF85Q4KY63vZ3nYK5iKX86AZZFY0mBTTmkFie9btOuOCNHV84l0vWtAHLHXE1LZTlg5I6E YL1lBvpFvcJDF6UgAS+9bhjDPfQJdaRwlC0ENIeqp6 X-Google-Smtp-Source: AGHT+IH0pHk0qDJNqIAsD1Z9tfR+Tnf3soZlRZtdM6+9lyqtv8A9QQO6bqv2p5NegceReOFlqzR5+9z7 X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a25:ad0c:0:b0:dcd:875:4c40 with SMTP id y12-20020a25ad0c000000b00dcd08754c40mr490235ybi.10.1707913852140; Wed, 14 Feb 2024 04:30:52 -0800 (PST) Date: Wed, 14 Feb 2024 13:29:18 +0100 In-Reply-To: <20240214122845.2033971-45-ardb+git@google.com> Mime-Version: 1.0 References: <20240214122845.2033971-45-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=6529; i=ardb@kernel.org; h=from:subject; bh=7PfwX9kdlqeh9VOV1E3/xjyJ7s9l8UtTK1JPzHL11YQ=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIfXMJoWWbw+qfkZOqSpjX8UWIy4meDqVP+bJgWkHpr942 Dkhmy+7o5SFQYyDQVZMkUVg9t93O09PlKp1niULM4eVCWQIAxenAEzE7yvDf//FFmX8XYlT9X4f 3V8eUW9ms2DW9o6d3ge//+a8ySd1cBrDX6Fj/teTCmf26P5Tq5VU2LNWzeR4bGbD/Csx54vaMva rsgIA X-Mailer: git-send-email 2.43.0.687.g38aa6559b0-goog Message-ID: <20240214122845.2033971-77-ardb+git@google.com> Subject: [PATCH v8 32/43] arm64: mm: add LPA2 and 5 level paging support to G-to-nG conversion From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240214_123055_988584_99D62A83 X-CRM114-Status: GOOD ( 20.88 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel Add support for 5 level paging in the G-to-nG routine that creates its own temporary page tables to traverse the swapper page tables. Also add support for running the 5 level configuration with the top level folded at runtime, to support CPUs that do not implement the LPA2 extension. While at it, wire up the level skipping logic so it will also trigger on 4 level configurations with LPA2 enabled at build time but not active at runtime, as we'll fall back to 3 level paging in that case. Signed-off-by: Ard Biesheuvel --- arch/arm64/kernel/cpufeature.c | 9 ++- arch/arm64/mm/proc.S | 70 +++++++++++++++++--- 2 files changed, 66 insertions(+), 13 deletions(-) diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c index ed9670d8360c..bc5e4e569864 100644 --- a/arch/arm64/kernel/cpufeature.c +++ b/arch/arm64/kernel/cpufeature.c @@ -1765,6 +1765,9 @@ static int __init __kpti_install_ng_mappings(void *__unused) pgd_t *kpti_ng_temp_pgd; u64 alloc = 0; + if (levels == 5 && !pgtable_l5_enabled()) + levels = 4; + remap_fn = (void *)__pa_symbol(idmap_kpti_install_ng_mappings); if (!cpu) { @@ -1778,9 +1781,9 @@ static int __init __kpti_install_ng_mappings(void *__unused) // // The physical pages are laid out as follows: // - // +--------+-/-------+-/------ +-\\--------+ - // : PTE[] : | PMD[] : | PUD[] : || PGD[] : - // +--------+-\-------+-\------ +-//--------+ + // +--------+-/-------+-/------ +-/------ +-\\\--------+ + // : PTE[] : | PMD[] : | PUD[] : | P4D[] : ||| PGD[] : + // +--------+-\-------+-\------ +-\------ +-///--------+ // ^ // The first page is mapped into this hierarchy at a PMD_SHIFT // aligned virtual address, so that we can manipulate the PTE diff --git a/arch/arm64/mm/proc.S b/arch/arm64/mm/proc.S index d03434b7bca5..fa0d7c63f8d2 100644 --- a/arch/arm64/mm/proc.S +++ b/arch/arm64/mm/proc.S @@ -216,16 +216,15 @@ SYM_FUNC_ALIAS(__pi_idmap_cpu_replace_ttbr1, idmap_cpu_replace_ttbr1) .macro kpti_mk_tbl_ng, type, num_entries add end_\type\()p, cur_\type\()p, #\num_entries * 8 .Ldo_\type: - ldr \type, [cur_\type\()p] // Load the entry + ldr \type, [cur_\type\()p], #8 // Load the entry and advance tbz \type, #0, .Lnext_\type // Skip invalid and tbnz \type, #11, .Lnext_\type // non-global entries orr \type, \type, #PTE_NG // Same bit for blocks and pages - str \type, [cur_\type\()p] // Update the entry + str \type, [cur_\type\()p, #-8] // Update the entry .ifnc \type, pte tbnz \type, #1, .Lderef_\type .endif .Lnext_\type: - add cur_\type\()p, cur_\type\()p, #8 cmp cur_\type\()p, end_\type\()p b.ne .Ldo_\type .endm @@ -235,18 +234,18 @@ SYM_FUNC_ALIAS(__pi_idmap_cpu_replace_ttbr1, idmap_cpu_replace_ttbr1) * fixmap slot associated with the current level. */ .macro kpti_map_pgtbl, type, level - str xzr, [temp_pte, #8 * (\level + 1)] // break before make + str xzr, [temp_pte, #8 * (\level + 2)] // break before make dsb nshst - add pte, temp_pte, #PAGE_SIZE * (\level + 1) + add pte, temp_pte, #PAGE_SIZE * (\level + 2) lsr pte, pte, #12 tlbi vaae1, pte dsb nsh isb phys_to_pte pte, cur_\type\()p - add cur_\type\()p, temp_pte, #PAGE_SIZE * (\level + 1) + add cur_\type\()p, temp_pte, #PAGE_SIZE * (\level + 2) orr pte, pte, pte_flags - str pte, [temp_pte, #8 * (\level + 1)] + str pte, [temp_pte, #8 * (\level + 2)] dsb nshst .endm @@ -279,6 +278,8 @@ SYM_TYPED_FUNC_START(idmap_kpti_install_ng_mappings) end_ptep .req x15 pte .req x16 valid .req x17 + cur_p4dp .req x19 + end_p4dp .req x20 mov x5, x3 // preserve temp_pte arg mrs swapper_ttb, ttbr1_el1 @@ -286,6 +287,12 @@ SYM_TYPED_FUNC_START(idmap_kpti_install_ng_mappings) cbnz cpu, __idmap_kpti_secondary +#if CONFIG_PGTABLE_LEVELS > 4 + stp x29, x30, [sp, #-32]! + mov x29, sp + stp x19, x20, [sp, #16] +#endif + /* We're the boot CPU. Wait for the others to catch up */ sevl 1: wfe @@ -303,9 +310,32 @@ SYM_TYPED_FUNC_START(idmap_kpti_install_ng_mappings) mov_q pte_flags, KPTI_NG_PTE_FLAGS /* Everybody is enjoying the idmap, so we can rewrite swapper. */ + +#ifdef CONFIG_ARM64_LPA2 + /* + * If LPA2 support is configured, but 52-bit virtual addressing is not + * enabled at runtime, we will fall back to one level of paging less, + * and so we have to walk swapper_pg_dir as if we dereferenced its + * address from a PGD level entry, and terminate the PGD level loop + * right after. + */ + adrp pgd, swapper_pg_dir // walk &swapper_pg_dir at the next level + mov cur_pgdp, end_pgdp // must be equal to terminate the PGD loop +alternative_if_not ARM64_HAS_VA52 + b .Lderef_pgd // skip to the next level +alternative_else_nop_endif + /* + * LPA2 based 52-bit virtual addressing requires 52-bit physical + * addressing to be enabled as well. In this case, the shareability + * bits are repurposed as physical address bits, and should not be + * set in pte_flags. + */ + bic pte_flags, pte_flags, #PTE_SHARED +#endif + /* PGD */ adrp cur_pgdp, swapper_pg_dir - kpti_map_pgtbl pgd, 0 + kpti_map_pgtbl pgd, -1 kpti_mk_tbl_ng pgd, PTRS_PER_PGD /* Ensure all the updated entries are visible to secondary CPUs */ @@ -318,16 +348,33 @@ SYM_TYPED_FUNC_START(idmap_kpti_install_ng_mappings) /* Set the flag to zero to indicate that we're all done */ str wzr, [flag_ptr] +#if CONFIG_PGTABLE_LEVELS > 4 + ldp x19, x20, [sp, #16] + ldp x29, x30, [sp], #32 +#endif ret .Lderef_pgd: + /* P4D */ + .if CONFIG_PGTABLE_LEVELS > 4 + p4d .req x30 + pte_to_phys cur_p4dp, pgd + kpti_map_pgtbl p4d, 0 + kpti_mk_tbl_ng p4d, PTRS_PER_P4D + b .Lnext_pgd + .else /* CONFIG_PGTABLE_LEVELS <= 4 */ + p4d .req pgd + .set .Lnext_p4d, .Lnext_pgd + .endif + +.Lderef_p4d: /* PUD */ .if CONFIG_PGTABLE_LEVELS > 3 pud .req x10 - pte_to_phys cur_pudp, pgd + pte_to_phys cur_pudp, p4d kpti_map_pgtbl pud, 1 kpti_mk_tbl_ng pud, PTRS_PER_PUD - b .Lnext_pgd + b .Lnext_p4d .else /* CONFIG_PGTABLE_LEVELS <= 3 */ pud .req pgd .set .Lnext_pud, .Lnext_pgd @@ -371,6 +418,9 @@ SYM_TYPED_FUNC_START(idmap_kpti_install_ng_mappings) .unreq end_ptep .unreq pte .unreq valid + .unreq cur_p4dp + .unreq end_p4dp + .unreq p4d /* Secondary CPUs end up here */ __idmap_kpti_secondary: