From patchwork Wed Feb 14 12:29:05 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13556414 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 3AAA6C48BC3 for ; Wed, 14 Feb 2024 12:32:18 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=PDHvYjIsHEfeCLwFLPtkXxTZuYETwCa1ECn3oS0Pjjs=; b=x0YKGrLiDkQNHOwIRnG50t13Td qgjoKf3J33zPyRUZ0LVxHLzblom+rt8IjuOo7EXL3pJgpqITiy1tNkG9A5JlKU4AtsyKPxnfI9RCf dCxelC3C+G7nhP5IjQ7GWOlBhbZtvtu9+7IKfoQjPNFjIY8b329ZUfPg89y9uNODx3Y7q+7Bgq52K tV2Ej+SeQsdt5htneKpPx4IbeQonzyYeiw9cQomAmBkOXwmGnIYz8rRdmevOXW84MHIJK4L7PAZy7 kYoBxMhl6EQ747z1Bq93xP2vSUxYoTbPg9m5YWYtL/TZIAIj2NqK7dhn/dEZFoHmq+pWZgdZ1wxjS JUSI1ATg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1raEQm-0000000CpN6-2w1j; Wed, 14 Feb 2024 12:32:04 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1raEPL-0000000CoiD-15f2 for linux-arm-kernel@bombadil.infradead.org; Wed, 14 Feb 2024 12:30:35 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=VvnB0e9tOYh+zNHys77cOZeL4hzuiQZQpjfGT9yM30s=; b=lEXbZVFufVZRqwSRd1FOWrHi0r QHsG7W4hR91lOT/iNMlRCBRoylcit0HDiLEXUhiVR0DvtkWLRkoEXz+G6jN2QBqKnRqNOxQP9eC45 VLIP6Pt1JZhxTenbbOutmv9JtzbHOhZHwD8LUYd/R4l6fFIiK6Dpd+hqWxZDuNe7iIkJNmbue+jUw V+ZGIGM22CWMY/in1ssmxWYqB/Rd/YE9BN0Y4nFLuV2IyYTpOaWcfuZmSc1BCEte6EIFnkeEivjnG sOtq2xGYM6WbyOkzxhYAkRRsZOqqg9/pWIF2wCPkn6rsSUBy1tWIjv7FOBjEo/AiMyVJ7dh4wdEL9 +JMzO63w==; Received: from mail-wm1-x349.google.com ([2a00:1450:4864:20::349]) by desiato.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1raEPD-0000000GDzv-0WJU for linux-arm-kernel@lists.infradead.org; Wed, 14 Feb 2024 12:30:33 +0000 Received: by mail-wm1-x349.google.com with SMTP id 5b1f17b1804b1-411a8c8d9b2so13131515e9.0 for ; Wed, 14 Feb 2024 04:30:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1707913822; x=1708518622; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=VvnB0e9tOYh+zNHys77cOZeL4hzuiQZQpjfGT9yM30s=; b=vHH+tdqY+aHl1hsc0mYvTMtohoe1FhRCyJ1mas5bvrGCWEmKSF6MlBqTKby84idLnn 61iqx0UIkv0MFgiZY3o/tfDMIi+gOMvZrxhg1z1VqpZP5CMKub/8ENmCqYlB8TKQEXGr feIXU64UeMfQmN71SggkqZjO3Kdd9E3YtlmGJm/1wv4GA67PqqVEC3k5RcdT9kM2paMA eClz/yG+GrJDqKQiekLKnAf0CJcYlp9QluuWXc6I/ndwrB1OH/DXsmNzLSji+zRlNebf jsMSKy0CR7L7rOHXVZAdLBz3m+K1op7hghJmM9dX2tzw73dsDNbSiQyORCXzQvCr+VJW I/HA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707913822; x=1708518622; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=VvnB0e9tOYh+zNHys77cOZeL4hzuiQZQpjfGT9yM30s=; b=N1YUVTFuNdcGmVzURwDBgePU1EsfoiFWr9wNXJxV/xIj5gWMZMMmtqqwXejJmidTpG g90zeN5RqWuqidvMmQVp/0e/katW1+b7K8PSy5rU/Ej0TH/SWTJraS9vJjLbEl34Z19D iTyVk+ENsWEn9YOiE5sHvMpbZYEX3RMSFejMSEtHtDbqK13d6QU3mXb06K1tn9z/NLSj YZWmmWzJBFFGLThazoplZwDuaYHDNMDWkakTFxvmaWAStekVb+5UcqgAWVknvAQc2cw5 /uSxECNNHRCdnxqYcI4xw9EgG8EMHNLLJkPTgNAMMUTdkv5G1wOcx4p5MDw//QLrLz4j bwtw== X-Gm-Message-State: AOJu0YzIO9cFKsEJeX0CnN7obJtMcstZEhTol6kiar5YjCdzdaRzNeJr qefGApARKpicU7uasdUzqioNZvlUN2BOcXnda5ns/LO3l5fuzD3DsiRp/f2difeki00VZnDxVeL T7pI2/OUF7lPqrQqb6Qw/IoR+1XHlhBxgVa8ws16iJ5uxmh0XJYvtymywop0jG0wmsjuhClwYwM Ih29Z5tthoEylPQNiHuLGZP2Ubb3/IVef0dFk2F7A0 X-Google-Smtp-Source: AGHT+IEQ4DMW3korYTIzzOGk0EMaA7cc/vwsJlBA+E1ZOuSN2Owp8FEj7KvA57fRsXB+pvmsmkkcYxTo X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:6000:1f96:b0:33c:deec:4ad7 with SMTP id bw22-20020a0560001f9600b0033cdeec4ad7mr7560wrb.7.1707913822232; Wed, 14 Feb 2024 04:30:22 -0800 (PST) Date: Wed, 14 Feb 2024 13:29:05 +0100 In-Reply-To: <20240214122845.2033971-45-ardb+git@google.com> Mime-Version: 1.0 References: <20240214122845.2033971-45-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=7010; i=ardb@kernel.org; h=from:subject; bh=K6dlk+khRRcpqbypZSsft3meiNcDy9eMVYLffSD3B8c=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIfXMJqHu/GNz4rYyzpDjO/BFz4nrbFgdU0yPgs2OVTuEp z2X4XzUUcrCIMbBICumyCIw+++7nacnStU6z5KFmcPKBDKEgYtTACaS9Z3hr5iVcJ/4yhr1OQlx xxTu7m/SXXp9UtHNliP/+uP8cz+d38Hwv5K9Mlf415l7rKzxp3cfzH5gsfmYsOuLhXkiqrsOSuZ uYgQA X-Mailer: git-send-email 2.43.0.687.g38aa6559b0-goog Message-ID: <20240214122845.2033971-64-ardb+git@google.com> Subject: [PATCH v8 19/43] arm64: mm: Use 48-bit virtual addressing for the permanent ID map From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240214_123027_835891_FFF94BF3 X-CRM114-Status: GOOD ( 24.55 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel Even though we support loading kernels anywhere in 48-bit addressable physical memory, we create the ID maps based on the number of levels that we happened to configure for the kernel VA and user VA spaces. The reason for this is that the PGD/PUD/PMD based classification of translation levels, along with the associated folding when the number of levels is less than 5, does not permit creating a page table hierarchy of a set number of levels. This means that, for instance, on 39-bit VA kernels we need to configure an additional level above PGD level on the fly, and 36-bit VA kernels still only support 47-bit virtual addressing with this trick applied. Now that we have a separate helper to populate page table hierarchies that does not define the levels in terms of PUDS/PMDS/etc at all, let's reuse it to create the permanent ID map with a fixed VA size of 48 bits. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/kernel-pgtable.h | 3 ++ arch/arm64/kernel/head.S | 5 +++ arch/arm64/kvm/mmu.c | 15 +++------ arch/arm64/mm/mmu.c | 32 +++++++++++--------- arch/arm64/mm/proc.S | 9 ++---- 5 files changed, 32 insertions(+), 32 deletions(-) diff --git a/arch/arm64/include/asm/kernel-pgtable.h b/arch/arm64/include/asm/kernel-pgtable.h index 0631604995ee..742a4b2778f7 100644 --- a/arch/arm64/include/asm/kernel-pgtable.h +++ b/arch/arm64/include/asm/kernel-pgtable.h @@ -35,6 +35,9 @@ #define SWAPPER_PGTABLE_LEVELS (CONFIG_PGTABLE_LEVELS) #endif +#define IDMAP_VA_BITS 48 +#define IDMAP_LEVELS ARM64_HW_PGTABLE_LEVELS(IDMAP_VA_BITS) +#define IDMAP_ROOT_LEVEL (4 - IDMAP_LEVELS) /* * A relocatable kernel may execute from an address that differs from the one at diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S index ffacce7b5a02..a1c29d64e875 100644 --- a/arch/arm64/kernel/head.S +++ b/arch/arm64/kernel/head.S @@ -729,6 +729,11 @@ SYM_FUNC_START_LOCAL(__no_granule_support) SYM_FUNC_END(__no_granule_support) SYM_FUNC_START_LOCAL(__primary_switch) + mrs x1, tcr_el1 + mov x2, #64 - VA_BITS + tcr_set_t0sz x1, x2 + msr tcr_el1, x1 + adrp x1, reserved_pg_dir adrp x2, init_idmap_pg_dir bl __enable_mmu diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c index d14504821b79..6fa9e816df40 100644 --- a/arch/arm64/kvm/mmu.c +++ b/arch/arm64/kvm/mmu.c @@ -1874,16 +1874,9 @@ int __init kvm_mmu_init(u32 *hyp_va_bits) BUG_ON((hyp_idmap_start ^ (hyp_idmap_end - 1)) & PAGE_MASK); /* - * The ID map may be configured to use an extended virtual address - * range. This is only the case if system RAM is out of range for the - * currently configured page size and VA_BITS_MIN, in which case we will - * also need the extended virtual range for the HYP ID map, or we won't - * be able to enable the EL2 MMU. - * - * However, in some cases the ID map may be configured for fewer than - * the number of VA bits used by the regular kernel stage 1. This - * happens when VA_BITS=52 and the kernel image is placed in PA space - * below 48 bits. + * The ID map is always configured for 48 bits of translation, which + * may be fewer than the number of VA bits used by the regular kernel + * stage 1, when VA_BITS=52. * * At EL2, there is only one TTBR register, and we can't switch between * translation tables *and* update TCR_EL2.T0SZ at the same time. Bottom @@ -1894,7 +1887,7 @@ int __init kvm_mmu_init(u32 *hyp_va_bits) * 1 VA bits to assure that the hypervisor can both ID map its code page * and map any kernel memory. */ - idmap_bits = 64 - ((idmap_t0sz & TCR_T0SZ_MASK) >> TCR_T0SZ_OFFSET); + idmap_bits = IDMAP_VA_BITS; kernel_bits = vabits_actual; *hyp_va_bits = max(idmap_bits, kernel_bits); diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index f9332eea318f..a991f195592b 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -757,22 +757,21 @@ static void __init map_kernel(pgd_t *pgdp) kasan_copy_shadow(pgdp); } +void __pi_map_range(u64 *pgd, u64 start, u64 end, u64 pa, pgprot_t prot, + int level, pte_t *tbl, bool may_use_cont, u64 va_offset); + +static u8 idmap_ptes[IDMAP_LEVELS - 1][PAGE_SIZE] __aligned(PAGE_SIZE) __ro_after_init, + kpti_ptes[IDMAP_LEVELS - 1][PAGE_SIZE] __aligned(PAGE_SIZE) __ro_after_init; + static void __init create_idmap(void) { u64 start = __pa_symbol(__idmap_text_start); - u64 size = __pa_symbol(__idmap_text_end) - start; - pgd_t *pgd = idmap_pg_dir; - u64 pgd_phys; - - /* check if we need an additional level of translation */ - if (VA_BITS < 48 && idmap_t0sz < (64 - VA_BITS_MIN)) { - pgd_phys = early_pgtable_alloc(PAGE_SHIFT); - set_pgd(&idmap_pg_dir[start >> VA_BITS], - __pgd(pgd_phys | P4D_TYPE_TABLE)); - pgd = __va(pgd_phys); - } - __create_pgd_mapping(pgd, start, start, size, PAGE_KERNEL_ROX, - early_pgtable_alloc, 0); + u64 end = __pa_symbol(__idmap_text_end); + u64 ptep = __pa_symbol(idmap_ptes); + + __pi_map_range(&ptep, start, end, start, PAGE_KERNEL_ROX, + IDMAP_ROOT_LEVEL, (pte_t *)idmap_pg_dir, false, + __phys_to_virt(ptep) - ptep); if (IS_ENABLED(CONFIG_UNMAP_KERNEL_AT_EL0)) { extern u32 __idmap_kpti_flag; @@ -782,8 +781,10 @@ static void __init create_idmap(void) * The KPTI G-to-nG conversion code needs a read-write mapping * of its synchronization flag in the ID map. */ - __create_pgd_mapping(pgd, pa, pa, sizeof(u32), PAGE_KERNEL, - early_pgtable_alloc, 0); + ptep = __pa_symbol(kpti_ptes); + __pi_map_range(&ptep, pa, pa + sizeof(u32), pa, PAGE_KERNEL, + IDMAP_ROOT_LEVEL, (pte_t *)idmap_pg_dir, false, + __phys_to_virt(ptep) - ptep); } } @@ -808,6 +809,7 @@ void __init paging_init(void) memblock_allow_resize(); create_idmap(); + idmap_t0sz = TCR_T0SZ(IDMAP_VA_BITS); } #ifdef CONFIG_MEMORY_HOTPLUG diff --git a/arch/arm64/mm/proc.S b/arch/arm64/mm/proc.S index 7c1bdaf25408..47ede52bb900 100644 --- a/arch/arm64/mm/proc.S +++ b/arch/arm64/mm/proc.S @@ -421,9 +421,9 @@ SYM_FUNC_START(__cpu_setup) mair .req x17 tcr .req x16 mov_q mair, MAIR_EL1_SET - mov_q tcr, TCR_TxSZ(VA_BITS) | TCR_CACHE_FLAGS | TCR_SMP_FLAGS | \ - TCR_TG_FLAGS | TCR_KASLR_FLAGS | TCR_ASID16 | \ - TCR_TBI0 | TCR_A1 | TCR_KASAN_SW_FLAGS | TCR_MTE_FLAGS + mov_q tcr, TCR_T0SZ(IDMAP_VA_BITS) | TCR_T1SZ(VA_BITS) | TCR_CACHE_FLAGS | \ + TCR_SMP_FLAGS | TCR_TG_FLAGS | TCR_KASLR_FLAGS | TCR_ASID16 | \ + TCR_TBI0 | TCR_A1 | TCR_KASAN_SW_FLAGS | TCR_MTE_FLAGS tcr_clear_errata_bits tcr, x9, x5 @@ -431,10 +431,7 @@ SYM_FUNC_START(__cpu_setup) sub x9, xzr, x0 add x9, x9, #64 tcr_set_t1sz tcr, x9 -#else - idmap_get_t0sz x9 #endif - tcr_set_t0sz tcr, x9 /* * Set the IPS bits in TCR_EL1.