From patchwork Fri Jun 24 15:06:36 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 12894745 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id D5A8DC43334 for ; Fri, 24 Jun 2022 15:09:11 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=Zvl7u0IKdtJvlxdJPYrK1Zp+6aw8WHROzbUzEQcE0Do=; b=lw9Jh1wW5xqIZB d89FHM3y9sqpPdpBq7XFFMG/rdU1v8hCM64DStMI0ci2v+okgAUXnXVOLsfz2S2cLfAQBDFjCEHy+ sCjewh7bm2QEDAyVCIgGRhUCsjTYs8iPk9LT+e9xFNCa2LKzHOw0tac2gK4C97sU6fFd1cL7G7atP whI5EMbZBg8L/tA7nmp2fy0ASJxZ+z28qUa77Ou0pAxMaxEzsWJlPyOynj8A8Jm9nRNK/mO+iWNuu GdLERbOUbPqW/7qZifHObY9RQCCcs6W2lz1UTiiS+H56o0aLWD6+1nJVkoVmj/8+P2UyhyEVqzqzM GfjFwKsovoGwaZw8huxQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1o4kuh-002idU-QV; Fri, 24 Jun 2022 15:08:03 +0000 Received: from ams.source.kernel.org ([145.40.68.75]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1o4ku5-002iNq-Sl for linux-arm-kernel@lists.infradead.org; Fri, 24 Jun 2022 15:07:27 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 9032CB826EE; Fri, 24 Jun 2022 15:07:24 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 758B9C341C0; Fri, 24 Jun 2022 15:07:21 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1656083243; bh=uR6DlXNQLjSiEeBqNN7df4CkoP+TjlQK6KQcfX40dhM=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Wv0G/HRB0HnRjHLqEkI5v8jMNilmKDKd648RlMFV7o5RhAPLNEappqCVLXtgb/EKr Jp5HdGJPZgO0+iCmOj01eL7kQVM44HvlZrJzzVqJVoNlCde3GpXhVXm1xlHbilrSx1 qU12DLbxWWz8+xuIluO58WtrdzIP8L9U+WyMcqIvX7KpZyAa49QCS+Tvh5MwXnUu+t pukghuOwIMaEeZA04uvUmFA2h1ZNs/iv5mhMlixdI/g0zs30A+wmelKy0jVqwKwPV4 dQ2lF6Z+55zDdcQwmX13Sblm6fELN1P/y6+K1z6IFyoomxuF1b9BegWM0wL3j+IahN EzgxXCUEQ/D3w== From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Marc Zyngier , Will Deacon , Mark Rutland , Kees Cook , Catalin Marinas , Mark Brown , Anshuman Khandual Subject: [PATCH v5 06/21] arm64: head: switch to map_memory macro for the extended ID map Date: Fri, 24 Jun 2022 17:06:36 +0200 Message-Id: <20220624150651.1358849-7-ardb@kernel.org> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220624150651.1358849-1-ardb@kernel.org> References: <20220624150651.1358849-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=6360; h=from:subject; bh=uR6DlXNQLjSiEeBqNN7df4CkoP+TjlQK6KQcfX40dhM=; b=owEB7QES/pANAwAKAcNPIjmS2Y8kAcsmYgBitdLwdCRi/VuhL5KvoeTsPU+1VRO0P/nLOKhrzExp 7FJ3dg6JAbMEAAEKAB0WIQT72WJ8QGnJQhU3VynDTyI5ktmPJAUCYrXS8AAKCRDDTyI5ktmPJIu3C/ 9owHbJh90PiZN0+IwTjk8S3G92aDe2IBuawngzN1iKIuIcl6oeKJFOdJVc6uIFTGGU2xMQD21PY/jE WLKpGGktRodd7j2pHszCaKUVslSXbyEA/f9rs99HyQ552VzsvQ1UlgZSRneX3mOY1pIh1W0aMDmSX8 XnnrzZbzB3exUxIhZ+CHt27Tcs8C3CczeLYLkSHqpHooChX51Xk4TTS8nc5hb0S4Og6rHizucDY7+3 wn+R2QwakBKiXyoLBvMnsL52gEMZmPqxf9BqDm9diaedvNRwMvR4HXd3HXQ3ft2u36LA1xYL1KdcfV 28BvRx+57vnXpb39CI/16SoVYN9Aw/yn9CpG8izzb0rHS3a75WLXGuMtRNGHHBVUYS8RIGPXyfSSqU kaVcrqdrxnCNs0NEfTtYnjcOejz4Ye/d0LMLfjNXDAqQQVi0bhOv1bLgZfQo4WFcOP0NvRfBpUPOne ctYWPlQXtPK/Pf1ljk85te2SKnIQuC+tcaNjPj5rEgFUg= X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220624_080726_262713_8A4E605B X-CRM114-Status: GOOD ( 22.91 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org In a future patch, we will start using an ID map that covers the entire image, rather than a single page. This means that we need to deal with the pathological case of an extended ID map where the kernel image does not fit neatly inside a single entry at the root level, which means we will need to create additional table entries and map additional pages for page tables. The existing map_memory macro already takes care of most of that, so let's just extend it to deal with this case as well. While at it, drop the conditional branch on the value of T0SZ: we don't set the variable anymore in the entry code, and so we can just let the map_memory macro deal with the case where the output address exceeds VA_BITS. Signed-off-by: Ard Biesheuvel --- arch/arm64/kernel/head.S | 76 ++++++++++---------- 1 file changed, 37 insertions(+), 39 deletions(-) diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S index 9fdde2f9cc0f..eb54c0289c8a 100644 --- a/arch/arm64/kernel/head.S +++ b/arch/arm64/kernel/head.S @@ -122,29 +122,6 @@ SYM_CODE_START_LOCAL(preserve_boot_args) b dcache_inval_poc // tail call SYM_CODE_END(preserve_boot_args) -/* - * Macro to create a table entry to the next page. - * - * tbl: page table address - * virt: virtual address - * shift: #imm page table shift - * ptrs: #imm pointers per table page - * - * Preserves: virt - * Corrupts: ptrs, tmp1, tmp2 - * Returns: tbl -> next level table page address - */ - .macro create_table_entry, tbl, virt, shift, ptrs, tmp1, tmp2 - add \tmp1, \tbl, #PAGE_SIZE - phys_to_pte \tmp2, \tmp1 - orr \tmp2, \tmp2, #PMD_TYPE_TABLE // address of next table and entry type - lsr \tmp1, \virt, #\shift - sub \ptrs, \ptrs, #1 - and \tmp1, \tmp1, \ptrs // table index - str \tmp2, [\tbl, \tmp1, lsl #3] - add \tbl, \tbl, #PAGE_SIZE // next level table page - .endm - /* * Macro to populate page table entries, these entries can be pointers to the next level * or last level entries pointing to physical memory. @@ -209,15 +186,27 @@ SYM_CODE_END(preserve_boot_args) * phys: physical address corresponding to vstart - physical memory is contiguous * order: #imm 2log(number of entries in PGD table) * + * If extra_shift is set, an extra level will be populated if the end address does + * not fit in 'extra_shift' bits. This assumes vend is in the TTBR0 range. + * * Temporaries: istart, iend, tmp, count, sv - these need to be different registers * Preserves: vstart, flags * Corrupts: tbl, rtbl, vend, istart, iend, tmp, count, sv */ - .macro map_memory, tbl, rtbl, vstart, vend, flags, phys, order, istart, iend, tmp, count, sv + .macro map_memory, tbl, rtbl, vstart, vend, flags, phys, order, istart, iend, tmp, count, sv, extra_shift sub \vend, \vend, #1 add \rtbl, \tbl, #PAGE_SIZE mov \count, #0 + .ifnb \extra_shift + tst \vend, #~((1 << (\extra_shift)) - 1) + b.eq .L_\@ + compute_indices \vstart, \vend, #\extra_shift, #(PAGE_SHIFT - 3), \istart, \iend, \count + mov \sv, \rtbl + populate_entries \tbl, \rtbl, \istart, \iend, #PMD_TYPE_TABLE, #PAGE_SIZE, \tmp + mov \tbl, \sv + .endif +.L_\@: compute_indices \vstart, \vend, #PGDIR_SHIFT, #\order, \istart, \iend, \count mov \sv, \rtbl populate_entries \tbl, \rtbl, \istart, \iend, #PMD_TYPE_TABLE, #PAGE_SIZE, \tmp @@ -284,20 +273,32 @@ SYM_FUNC_START_LOCAL(__create_page_tables) adrp x3, __idmap_text_start // __pa(__idmap_text_start) /* - * VA_BITS may be too small to allow for an ID mapping to be created - * that covers system RAM if that is located sufficiently high in the - * physical address space. So for the ID map, use an extended virtual - * range in that case, and configure an additional translation level - * if needed. + * The ID map carries a 1:1 mapping of the physical address range + * covered by the loaded image, which could be anywhere in DRAM. This + * means that the required size of the VA (== PA) space is decided at + * boot time, and could be more than the configured size of the VA + * space for ordinary kernel and user space mappings. + * + * There are three cases to consider here: + * - 39 <= VA_BITS < 48, and the ID map needs up to 48 VA bits to cover + * the placement of the image. In this case, we configure one extra + * level of translation on the fly for the ID map only. (This case + * also covers 42-bit VA/52-bit PA on 64k pages). + * + * - VA_BITS == 48, and the ID map needs more than 48 VA bits. This can + * only happen when using 64k pages, in which case we need to extend + * the root level table rather than add a level. Note that we can + * treat this case as 'always extended' as long as we take care not + * to program an unsupported T0SZ value into the TCR register. + * + * - Combinations that would require two additional levels of + * translation are not supported, e.g., VA_BITS==36 on 16k pages, or + * VA_BITS==39/4k pages with 5-level paging, where the input address + * requires more than 47 or 48 bits, respectively. */ - idmap_get_t0sz x5 - cmp x5, TCR_T0SZ(VA_BITS_MIN) // default T0SZ small enough? - b.ge 1f // .. then skip VA range extension - #if (VA_BITS < 48) #define IDMAP_PGD_ORDER (VA_BITS - PGDIR_SHIFT) #define EXTRA_SHIFT (PGDIR_SHIFT + PAGE_SHIFT - 3) -#define EXTRA_PTRS (1 << (PHYS_MASK_SHIFT - EXTRA_SHIFT)) /* * If VA_BITS < 48, we have to configure an additional table level. @@ -309,20 +310,17 @@ SYM_FUNC_START_LOCAL(__create_page_tables) #if VA_BITS != EXTRA_SHIFT #error "Mismatch between VA_BITS and page size/number of translation levels" #endif - - mov x2, EXTRA_PTRS - create_table_entry x0, x3, EXTRA_SHIFT, x2, x5, x6 #else #define IDMAP_PGD_ORDER (PHYS_MASK_SHIFT - PGDIR_SHIFT) +#define EXTRA_SHIFT /* * If VA_BITS == 48, we don't have to configure an additional * translation level, but the top-level table has more entries. */ #endif -1: adr_l x6, __idmap_text_end // __pa(__idmap_text_end) - map_memory x0, x1, x3, x6, x7, x3, IDMAP_PGD_ORDER, x10, x11, x12, x13, x14 + map_memory x0, x1, x3, x6, x7, x3, IDMAP_PGD_ORDER, x10, x11, x12, x13, x14, EXTRA_SHIFT /* * Map the kernel image (starting with PHYS_OFFSET).