From patchwork Thu Apr 21 14:03:38 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 12821690 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 6690DC433EF for ; Thu, 21 Apr 2022 14:05:16 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=hcn8AQ9rfQc9w/n2GREtdvUXaxpYQZAPbdG5TRAtxlM=; b=QrCWFGN9irQxcb 5DPs4Tk9AFNZ3O0N3cb4kzk97EVMT84DhT9beRlGNkayZQpxZ/UN9IxlYmdEbrE74yPYuj31gMsu3 6AEkLvyPWZQFU78EZ8hG6sJpO2CG58NR48BvPvVahalDGcEKBryQgGvQEEDVV0yqVZjpm4LXhLVQ4 gpHpcjH7ypAiHlk5wjN3ZUHvqSWyVKlz39wRTR8ZJuHlEkwLblv78NAAco6i4YQTa4mfGuuhHT8i6 Ikw6us2gcv7hREfOJZ9NgeICWGzDU94EeKVtajoSAL76YgMDb8caaYTVkAjPXF3dtBpGXAche08KM mMyIpbccMj29K3Hxw+1w==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1nhXPg-00Dfaj-VB; Thu, 21 Apr 2022 14:04:05 +0000 Received: from dfw.source.kernel.org ([2604:1380:4641:c500::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1nhXPW-00DfWq-TU for linux-arm-kernel@lists.infradead.org; Thu, 21 Apr 2022 14:03:56 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 56D3361D60; Thu, 21 Apr 2022 14:03:53 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 31D8AC385AC; Thu, 21 Apr 2022 14:03:51 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1650549832; bh=UVCjV3GWPRpvAoW7QHeTFw1uLq+FlqrS+VoKJHF8vFE=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Ny4/6YL7j82GLH6kg4G2OeinTiEofLR+2DSIl6S24Vmr+X2pnHM0/mPipx4QjA396 /jr/oGe/M2hIFsr28G3WkpAtinN2fZhC2erRWwqSHycCiEFXa4fXzY1TLVlU872EgN ZhGFZAuqBrTD3hNuDDlWrzbIBW1ltK4zHK2ikJ1qoATHa0z/egu9MNZRD5GofJOlaK TbLVSQZaBTD6rxh48hZ3Qg7zBVbKpw0lrby809hfgUdoW/eFgeOpPtO9yU4y5V83EY EtQHnzSJ0sLGXNdFdk1cuLZbERUwsTMworY15hnuC/F/C0CzMFhj8yU8EQhAQFFcCM +JkSeixfFLFgA== From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , will@kernel.org, maz@kernel.org, mark.rutland@arm.com, catalin.marinas@arm.com, keescook@chromium.org Subject: [PATCH v3 1/2] arm64: kpti-ng: simplify page table traversal logic Date: Thu, 21 Apr 2022 16:03:38 +0200 Message-Id: <20220421140339.1329019-2-ardb@kernel.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220421140339.1329019-1-ardb@kernel.org> References: <20220421140339.1329019-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=5060; h=from:subject; bh=UVCjV3GWPRpvAoW7QHeTFw1uLq+FlqrS+VoKJHF8vFE=; b=owEB7QES/pANAwAKAcNPIjmS2Y8kAcsmYgBiYWQ5RTCFNSemZJKJrxvSDDt11TmzKHiEPKD0qC4F UQuh3TOJAbMEAAEKAB0WIQT72WJ8QGnJQhU3VynDTyI5ktmPJAUCYmFkOQAKCRDDTyI5ktmPJCj9C/ 9ibqz+CSIz1bDHyj9YFGgcB3kSjg0l4u+a9WQAIPa1GYIg5NvpAuibvaOGy4LZHiq3PNWz3cEj/U6j 5GuC8WN08u21jWg0M0AX3jnAQDhCfrfKXeTbBpFiQOzIcz+PTWyDEdr0zMgdMVJZ32KZY81UVcEdjn f6+X+0cqHkvlFLOAylvDhvaC37ea+M+zmRjc3CJEISeLJRdveEAQBYA8zMjYTGLlUSk6j2TwcvZ74S +J11hnaI2tgveRuclozhwdIwLsPc15gm0hbFV6u9plhiwDlsWA3Zjwjri1jGzG837wFS/aIDY+HpCa u8Wt5sE0ijXpG4RjsrcTsyvLbJ9IniT9SQ5/TN7+K80BmbXYn3d6+tgMGSxARWcpcL0j2WUtFD15uh VG9HQal1+h+goLItn53cus6n0kBW/YxguzEGvUBUjIgwTO9PooxAqbxisPRYeMxWheMUTOzHxzBdlb KmkekCEbgEm6MmciSoiuWmexK0VV1BGodLghmWW1AcNXc= X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220421_070355_106461_976FDD77 X-CRM114-Status: GOOD ( 15.43 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Simplify the KPTI G-to-nG asm helper code by: - pulling the 'table bit' test into the get/put macros so we can combine them and incorporate the entire loop; - moving the 'table bit' test after the update of bit #11 so we no longer need separate next_xxx and skip_xxx labels; - redefining the pmd/pud register aliases and the next_pmd/next_pud labels instead of branching to them if the number of configured page table levels is less than 3 or 4, respectively; - folding the descriptor pointer increment into the LDR instructions. No functional change intended, except for the fact that we now descend into a next level table after setting bit #11 on its descriptor but this should make no difference in practice. While at it, switch to .L prefixed local labels so they don't clutter up the symbol tables, kallsyms, etc, and clean up the indentation for legibility. Signed-off-by: Ard Biesheuvel Reviewed-by: Mark Rutland Tested-by: Mark Rutland --- arch/arm64/mm/proc.S | 97 +++++++------------- 1 file changed, 34 insertions(+), 63 deletions(-) diff --git a/arch/arm64/mm/proc.S b/arch/arm64/mm/proc.S index 50bbed947bec..5619c00f8cd4 100644 --- a/arch/arm64/mm/proc.S +++ b/arch/arm64/mm/proc.S @@ -202,19 +202,24 @@ SYM_FUNC_END(idmap_cpu_replace_ttbr1) #ifdef CONFIG_UNMAP_KERNEL_AT_EL0 .pushsection ".idmap.text", "awx" - .macro __idmap_kpti_get_pgtable_ent, type + .macro kpti_mk_tbl_ng, type, num_entries + add end_\type\()p, cur_\type\()p, #\num_entries * 8 +.Ldo_\type: dc cvac, cur_\()\type\()p // Ensure any existing dirty dmb sy // lines are written back before - ldr \type, [cur_\()\type\()p] // loading the entry - tbz \type, #0, skip_\()\type // Skip invalid and - tbnz \type, #11, skip_\()\type // non-global entries - .endm - - .macro __idmap_kpti_put_pgtable_ent_ng, type + ldr \type, [cur_\type\()p], #8 // loading the entry + tbz \type, #0, .Lnext_\type // Skip invalid and + tbnz \type, #11, .Lnext_\type // non-global entries orr \type, \type, #PTE_NG // Same bit for blocks and pages - str \type, [cur_\()\type\()p] // Update the entry and ensure + str \type, [cur_\type\()p, #-8] // Update the entry and ensure dmb sy // that it is visible to all dc civac, cur_\()\type\()p // CPUs. + .ifnc \type, pte + tbnz \type, #1, .Lderef_\type + .endif +.Lnext_\type: + cmp cur_\type\()p, end_\type\()p + b.ne .Ldo_\type .endm /* @@ -235,10 +240,8 @@ SYM_FUNC_START(idmap_kpti_install_ng_mappings) pgd .req x7 cur_pudp .req x8 end_pudp .req x9 - pud .req x10 cur_pmdp .req x11 end_pmdp .req x12 - pmd .req x13 cur_ptep .req x14 end_ptep .req x15 pte .req x16 @@ -265,16 +268,8 @@ SYM_FUNC_START(idmap_kpti_install_ng_mappings) /* Everybody is enjoying the idmap, so we can rewrite swapper. */ /* PGD */ - mov cur_pgdp, swapper_pa - add end_pgdp, cur_pgdp, #(PTRS_PER_PGD * 8) -do_pgd: __idmap_kpti_get_pgtable_ent pgd - tbnz pgd, #1, walk_puds -next_pgd: - __idmap_kpti_put_pgtable_ent_ng pgd -skip_pgd: - add cur_pgdp, cur_pgdp, #8 - cmp cur_pgdp, end_pgdp - b.ne do_pgd + mov cur_pgdp, swapper_pa + kpti_mk_tbl_ng pgd, PTRS_PER_PGD /* Publish the updated tables and nuke all the TLBs */ dsb sy @@ -291,59 +286,35 @@ skip_pgd: str wzr, [flag_ptr] ret +.Lderef_pgd: /* PUD */ -walk_puds: - .if CONFIG_PGTABLE_LEVELS > 3 + .if CONFIG_PGTABLE_LEVELS > 3 + pud .req x10 pte_to_phys cur_pudp, pgd - add end_pudp, cur_pudp, #(PTRS_PER_PUD * 8) -do_pud: __idmap_kpti_get_pgtable_ent pud - tbnz pud, #1, walk_pmds -next_pud: - __idmap_kpti_put_pgtable_ent_ng pud -skip_pud: - add cur_pudp, cur_pudp, 8 - cmp cur_pudp, end_pudp - b.ne do_pud - b next_pgd - .else /* CONFIG_PGTABLE_LEVELS <= 3 */ - mov pud, pgd - b walk_pmds -next_pud: - b next_pgd + kpti_mk_tbl_ng pud, PTRS_PER_PUD + b .Lnext_pgd + .else /* CONFIG_PGTABLE_LEVELS <= 3 */ + pud .req pgd + .set .Lnext_pud, .Lnext_pgd .endif +.Lderef_pud: /* PMD */ -walk_pmds: - .if CONFIG_PGTABLE_LEVELS > 2 + .if CONFIG_PGTABLE_LEVELS > 2 + pmd .req x13 pte_to_phys cur_pmdp, pud - add end_pmdp, cur_pmdp, #(PTRS_PER_PMD * 8) -do_pmd: __idmap_kpti_get_pgtable_ent pmd - tbnz pmd, #1, walk_ptes -next_pmd: - __idmap_kpti_put_pgtable_ent_ng pmd -skip_pmd: - add cur_pmdp, cur_pmdp, #8 - cmp cur_pmdp, end_pmdp - b.ne do_pmd - b next_pud - .else /* CONFIG_PGTABLE_LEVELS <= 2 */ - mov pmd, pud - b walk_ptes -next_pmd: - b next_pud + kpti_mk_tbl_ng pmd, PTRS_PER_PMD + b .Lnext_pud + .else /* CONFIG_PGTABLE_LEVELS <= 2 */ + pmd .req pgd + .set .Lnext_pmd, .Lnext_pgd .endif +.Lderef_pmd: /* PTE */ -walk_ptes: pte_to_phys cur_ptep, pmd - add end_ptep, cur_ptep, #(PTRS_PER_PTE * 8) -do_pte: __idmap_kpti_get_pgtable_ent pte - __idmap_kpti_put_pgtable_ent_ng pte -skip_pte: - add cur_ptep, cur_ptep, #8 - cmp cur_ptep, end_ptep - b.ne do_pte - b next_pmd + kpti_mk_tbl_ng pte, PTRS_PER_PTE + b .Lnext_pmd .unreq cpu .unreq num_cpus