From patchwork Thu Jul 27 15:30:00 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Russell King (Oracle)" X-Patchwork-Id: 13330311 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id AFEA3C04A6A for ; Thu, 27 Jul 2023 15:32:52 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Date:Message-Id:MIME-Version:Subject:Cc :To:From:References:In-Reply-To:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=DcIvbA9MQqfNh2uBeCSiOYWKjh3So5/KfOmXXR02ZdE=; b=IO2y7/zBVa7ZiB LIKWTG6UQuB0ekfn9Ba4FXLLuHIhRG7JVxlrKL2g0r5r9inkCSr9uQ7SvWUEbQcXwvaZHv/T1+597 jtwywxmC9KHDgCK3fTs5oTMgbR4PB0rKkKHEPpLDmFd/fWmYCIxShWLWYe61pLVf+af5UbZElpcSf bEWb+LnG2vl3JKUi54Hcc5qetamuHsqEpOKF8OQP+bWpqa1co71DasmBtHN0la+uMyczJtH3lsOlo 7LWG0QiNXItDoXr6cLW+BpfGSXLoYc04rCcReA0boPwlNHuMVVr6Aau64S6k3mpSG1H1Y+xrYaFQA OBzj6tm51vpSyJ1iUX3g==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qP2yb-00FwCT-1s; Thu, 27 Jul 2023 15:32:29 +0000 Received: from [2001:4d48:ad52:32c8:5054:ff:fe00:142] (helo=pandora.armlinux.org.uk) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qP2yD-00Fu7X-0g for linux-arm-kernel@lists.infradead.org; Thu, 27 Jul 2023 15:32:07 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=armlinux.org.uk; s=pandora-2019; h=Date:Sender:Message-Id:Content-Type: Content-Transfer-Encoding:MIME-Version:Subject:Cc:To:From:References: In-Reply-To:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help: List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=cXdDaf/oSNMNxEYmBBFd5azlzfsaSz/S5gpI4Qrctiw=; b=IU/mLeDfyy9zKp9mj5HCMGC4Wf jJsRdRtB116UaGoZ3HF6iXj81zT2ESY1c0PSuQpascfqwlufI0ZoLbvdCDynY4rhJfo+7BFxLp4+X VsirhGsb4/wwqzcqdpY8Rn3AcVRAkzn54fz2wSbaapjLhfHoiXvTS3Aug9Gwt/EqJqN9t7oBoKkzt kkVDWgI/8dvjvTF/D9hS9zS47OKI7uWoO3ewjvsN1aTaRDxT10reDG7/JdJT0V+a1W6mWXDaykM9B sS35dKYT8hqLKVdMuFixxMzN4JYgFLFW0KOfwQpHZrvXgjm8COfPjDmWue+wThbPS6u/73EaaZwe1 djp1JPhA==; Received: from e0022681537dd.dyn.armlinux.org.uk ([fd8f:7570:feb6:1:222:68ff:fe15:37dd]:33488 helo=rmk-PC.armlinux.org.uk) by pandora.armlinux.org.uk with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.96) (envelope-from ) id 1qP2wC-00062G-0h; Thu, 27 Jul 2023 16:30:00 +0100 Received: from rmk by rmk-PC.armlinux.org.uk with local (Exim 4.94.2) (envelope-from ) id 1qP2wC-001CWc-DF; Thu, 27 Jul 2023 16:30:00 +0100 In-Reply-To: References: From: "Russell King (Oracle)" To: Catalin Marinas , Jonathan Corbet , Will Deacon Cc: linux-arm-kernel@lists.infradead.org, linux-doc@vger.kernel.org Subject: [PATCH v2 01/16] arm64: provide cpu_replace_ttbr1_phys() MIME-Version: 1.0 Content-Disposition: inline Message-Id: Date: Thu, 27 Jul 2023 16:30:00 +0100 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230727_083205_282197_F555102A X-CRM114-Status: GOOD ( 13.78 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Provide a version of cpu_replace_ttbr1_phys() which operates using a physical address rather than the virtual address of the page tables. Signed-off-by: Russell King (Oracle) --- arch/arm64/include/asm/mmu_context.h | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/arch/arm64/include/asm/mmu_context.h b/arch/arm64/include/asm/mmu_context.h index a6fb325424e7..42085b2b1d92 100644 --- a/arch/arm64/include/asm/mmu_context.h +++ b/arch/arm64/include/asm/mmu_context.h @@ -152,7 +152,7 @@ static inline void cpu_install_ttbr0(phys_addr_t ttbr0, unsigned long t0sz) * Atomically replaces the active TTBR1_EL1 PGD with a new VA-compatible PGD, * avoiding the possibility of conflicting TLB entries being allocated. */ -static inline void cpu_replace_ttbr1(pgd_t *pgdp, pgd_t *idmap) +static inline void cpu_replace_ttbr1_phys(phys_addr_t pgd_phys, pgd_t *idmap) { typedef void (ttbr_replace_func)(phys_addr_t); extern ttbr_replace_func idmap_cpu_replace_ttbr1; @@ -160,9 +160,10 @@ static inline void cpu_replace_ttbr1(pgd_t *pgdp, pgd_t *idmap) unsigned long daif; /* phys_to_ttbr() zeros lower 2 bits of ttbr with 52-bit PA */ - phys_addr_t ttbr1 = phys_to_ttbr(virt_to_phys(pgdp)); + phys_addr_t ttbr1 = phys_to_ttbr(pgd_phys); - if (system_supports_cnp() && !WARN_ON(pgdp != lm_alias(swapper_pg_dir))) { + if (system_supports_cnp() && + !WARN_ON(pgd_phys != virt_to_phys(lm_alias(swapper_pg_dir)))) { /* * cpu_replace_ttbr1() is used when there's a boot CPU * up (i.e. cpufeature framework is not up yet) and @@ -189,6 +190,11 @@ static inline void cpu_replace_ttbr1(pgd_t *pgdp, pgd_t *idmap) cpu_uninstall_idmap(); } +static inline void __nocfi cpu_replace_ttbr1(pgd_t *pgdp, pgd_t *idmap) +{ + cpu_replace_ttbr1_phys(virt_to_phys(pgdp), idmap); +} + /* * It would be nice to return ASIDs back to the allocator, but unfortunately * that introduces a race with a generation rollover where we could erroneously From patchwork Thu Jul 27 15:30:05 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Russell King (Oracle)" X-Patchwork-Id: 13330294 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id F30A9C001E0 for ; Thu, 27 Jul 2023 15:30:37 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Date:Message-Id:MIME-Version:Subject:Cc :To:From:References:In-Reply-To:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=MXVyWFrXVZFUhrfx7HaSH/0B/mouho0PO4Hl984+bsM=; b=UtDyCf73YiJydp yC28o8wdbvmWkXkc1IkDHBH1HqfMBJtn3M8lMwkGeSs5x9EdqzB363APsC12DyW+Ez7HssSVahgVP 6GwRSkIOlptAC5k2VEMgFvkt3UTfw7qrxsGFf3AurralFdJkSf+PEQpMaZe1l00Y39XXHelPWIXUY CT/TIrjL15c/HsHaUpWNM08CLNql3gc0vjNcKjS4FMUP6pqAAG2Ly5ESMkBeVBa6zjHFMma7ygW5I DavGXpo5U8UvABYMiyrcBpy2mQWxKclB9jFS36Oit4/f84AJX6QvzQ/vBdgq1MAHVJLJVn+C49Sj0 V7anJkXqbpk0cE0J7buA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qP2wP-00FuGN-2H; Thu, 27 Jul 2023 15:30:13 +0000 Received: from [2001:4d48:ad52:32c8:5054:ff:fe00:142] (helo=pandora.armlinux.org.uk) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qP2wL-00FuAy-2c for linux-arm-kernel@lists.infradead.org; Thu, 27 Jul 2023 15:30:12 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=armlinux.org.uk; s=pandora-2019; h=Date:Sender:Message-Id:Content-Type: Content-Transfer-Encoding:MIME-Version:Subject:Cc:To:From:References: In-Reply-To:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help: List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=+0TeAiTNMzXFe/AzbLWZ/AnvsbUiDG4Bzu+mfiD7tNE=; b=aConhkK+g/sEV4UZbndv/kyXuB aW32ChzzcA2tg0chWBedtLobmHALSUVIRV9XVFOmers4/tFZvOHkLw37dZfPmUlIHu+kXNTLCykbn 625/hBsxL3EDkg+kEDFHfKWlv1DkM5/4n64H9dsx/eDmcxe/qnBl0ti/Ki212400OvhexGMCYUjdi rLHWvB4dqG5kyqwXuiZ/RnAoVpibjRc0tDEGS9uf8AwrI82oaLwjlaPt69bQNG91eDQ5vh8Vk7qVg 4zLHMrelZFTf8qtlV/Ex0GFite8/nmHQCr9BKG5wvVP/FvdgAcRPwXOhjR7aJ+JEixpouBW82sF/L lI7SwnZg==; Received: from e0022681537dd.dyn.armlinux.org.uk ([fd8f:7570:feb6:1:222:68ff:fe15:37dd]:40498 helo=rmk-PC.armlinux.org.uk) by pandora.armlinux.org.uk with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.96) (envelope-from ) id 1qP2wH-00062Q-1K; Thu, 27 Jul 2023 16:30:05 +0100 Received: from rmk by rmk-PC.armlinux.org.uk with local (Exim 4.94.2) (envelope-from ) id 1qP2wH-001CWz-Hs; Thu, 27 Jul 2023 16:30:05 +0100 In-Reply-To: References: From: "Russell King (Oracle)" To: Catalin Marinas , Jonathan Corbet , Will Deacon Cc: linux-arm-kernel@lists.infradead.org, linux-doc@vger.kernel.org Subject: [PATCH v2 02/16] arm64: make clean_dcache_range_nopatch() visible MIME-Version: 1.0 Content-Disposition: inline Message-Id: Date: Thu, 27 Jul 2023 16:30:05 +0100 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230727_083010_142305_442DA334 X-CRM114-Status: GOOD ( 12.99 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org When we hook into the kernel text patching code, we will need to call clean_dcache_range_nopatch() to ensure that the patching of the replicated kernel text is properly visible to other CPUs. Make this function available to the replication code. Signed-off-by: Russell King (Oracle) --- arch/arm64/include/asm/cacheflush.h | 2 ++ arch/arm64/kernel/alternative.c | 2 +- 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/arch/arm64/include/asm/cacheflush.h b/arch/arm64/include/asm/cacheflush.h index 37185e978aeb..ac9ad56d5212 100644 --- a/arch/arm64/include/asm/cacheflush.h +++ b/arch/arm64/include/asm/cacheflush.h @@ -104,6 +104,8 @@ static inline void flush_icache_range(unsigned long start, unsigned long end) } #define flush_icache_range flush_icache_range +void clean_dcache_range_nopatch(u64 start, u64 end); + /* * Copy user data from/to a page which is mapped into a different * processes address space. Really, we want to allow our "user diff --git a/arch/arm64/kernel/alternative.c b/arch/arm64/kernel/alternative.c index 8ff6610af496..ea3f4104771d 100644 --- a/arch/arm64/kernel/alternative.c +++ b/arch/arm64/kernel/alternative.c @@ -121,7 +121,7 @@ static noinstr void patch_alternative(struct alt_instr *alt, * accidentally call into the cache.S code, which is patched by us at * runtime. */ -static noinstr void clean_dcache_range_nopatch(u64 start, u64 end) +noinstr void clean_dcache_range_nopatch(u64 start, u64 end) { u64 cur, d_size, ctr_el0; From patchwork Thu Jul 27 15:30:10 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Russell King (Oracle)" X-Patchwork-Id: 13330295 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 617C7C001DC for ; Thu, 27 Jul 2023 15:30:46 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Date:Message-Id:MIME-Version:Subject:Cc :To:From:References:In-Reply-To:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=+1iKyMI7YNpUOZEv8OQYPn5dEgKsb3dGo7lguCVqHBo=; b=ogHSh0ErY8ZeNb TLf48bKV5HjlIrHdOPHsHocDMWwa9TG2qSMMk9qFXvMYSTbjLz3B3JEg/cFv1cN87B2z6yd8tSnpD OgVnmKqa23XO5EMW2sMFYPzPVLclXKjlXHGCjaYw8yOaAQ0lnc3AT6VNsV06mo27mGLFF3K3qtLYz igE3EEnPKlmziKljs3tM/rTc54PvGJxuvipA5DQeoOmOWDJtAE7ArCxhvxRx7LGgp+A8o9UnHOsL/ Lduj7OcHeEDITqm2lvMgasdKg/vwktxcavPrSdvvsqa1aDpLU10MzcJqzCeqA+6oim9IjuPtvsIvY V/A9/cHwxcJG4gZHf+qA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qP2wa-00FuP6-2B; Thu, 27 Jul 2023 15:30:24 +0000 Received: from [2001:4d48:ad52:32c8:5054:ff:fe00:142] (helo=pandora.armlinux.org.uk) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qP2wP-00FuEl-0A for linux-arm-kernel@lists.infradead.org; Thu, 27 Jul 2023 15:30:15 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=armlinux.org.uk; s=pandora-2019; h=Date:Sender:Message-Id:Content-Type: Content-Transfer-Encoding:MIME-Version:Subject:Cc:To:From:References: In-Reply-To:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help: List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=uvTxkvmHBcqFK8eJQas9KOF2RbssOQGRGFH22MwP0ec=; b=GaZTOqVmO7ohus88Hy+YEOoBFj 9VunljIlwIt1LNgu4WKn13WGbHNe12yKwNcLfqQmAyMPbBVA+7+2tslFvyN+OSaIwPDQwnFCtfIfp bXP+APuoBokPw/mPTYaEo/ARE0yWMNcq/1wRnbknM6gpBjsUlB/wM70Q1DgYwua2nccQ1kAiOMUgB b5+J1SfvLUqQP9Vu+jMPiLPXQR0k6gA/XjuRUParFIagUbuU+shuOo6q0rzWpOegCEDWgVKf83EHv mJVXxao8BMqXknSfcGiFwHGg5u8X93RPp4opPq3NCOKtdgWqruV30FhH/S+FfZHKL4PQQDrJSQstJ xYDN0mTg==; Received: from e0022681537dd.dyn.armlinux.org.uk ([fd8f:7570:feb6:1:222:68ff:fe15:37dd]:40506 helo=rmk-PC.armlinux.org.uk) by pandora.armlinux.org.uk with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.96) (envelope-from ) id 1qP2wM-00062b-1n; Thu, 27 Jul 2023 16:30:10 +0100 Received: from rmk by rmk-PC.armlinux.org.uk with local (Exim 4.94.2) (envelope-from ) id 1qP2wM-001CX8-Mk; Thu, 27 Jul 2023 16:30:10 +0100 In-Reply-To: References: From: "Russell King (Oracle)" To: Catalin Marinas , Jonathan Corbet , Will Deacon Cc: linux-arm-kernel@lists.infradead.org, linux-doc@vger.kernel.org Subject: [PATCH v2 03/16] arm64: place kernel in its own L0 page table entry MIME-Version: 1.0 Content-Disposition: inline Message-Id: Date: Thu, 27 Jul 2023 16:30:10 +0100 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230727_083013_103034_05F0AEED X-CRM114-Status: GOOD ( 19.54 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Kernel text replication needs to maintain separate per-node page tables for the kernel text. In order to do this without affecting other kernel memory mappings, placing the kernel such that it does not share a L0 page table entry with any other mapping is desirable. Prior to this commit, the layout without KASLR was: +----------+ | vmalloc | +----------+ | Kernel | +----------+ MODULES_END, VMALLOC_START, KIMAGE_VADDR = | Modules | MODULES_VADDR + MODULES_VSIZE +----------+ MODULES_VADDR = _PAGE_END(VA_BITS_MIN) | VA space | +----------+ 0 This becomes: +----------+ | vmalloc | +----------+ VMALLOC_START = MODULES_END + PGDIR_SIZE | Kernel | +----------+ MODULES_END, KIMAGE_VADDR = _PAGE_END(VA_BITS_MIN) + | Modules | max(PGDIR_SIZE, MODULES_VSIZE) +----------+ MODULES_VADDR = MODULES_END - MODULES_VSIZE | VA space | +----------+ 0 This assumes MODULES_VSIZE (128M) <= PGDIR_SIZE. One side effect of this change is that KIMAGE_VADDR's definition now includes PGDIR_SIZE (to leave room for the modules) but this is not defined when asm/memory.h is included. This means KIMAGE_VADDR can not be used in inline functions within this file, so we convert kaslr_offset() and kaslr_enabled() to be macros instead. Signed-off-by: Russell King (Oracle) --- arch/arm64/include/asm/memory.h | 28 +++++++++++++++++++++------- arch/arm64/include/asm/pgtable.h | 2 +- arch/arm64/kernel/kaslr.c | 1 + 3 files changed, 23 insertions(+), 8 deletions(-) diff --git a/arch/arm64/include/asm/memory.h b/arch/arm64/include/asm/memory.h index fde4186cc387..9410ec4e4207 100644 --- a/arch/arm64/include/asm/memory.h +++ b/arch/arm64/include/asm/memory.h @@ -43,9 +43,26 @@ #define VA_BITS (CONFIG_ARM64_VA_BITS) #define _PAGE_OFFSET(va) (-(UL(1) << (va))) #define PAGE_OFFSET (_PAGE_OFFSET(VA_BITS)) -#define KIMAGE_VADDR (MODULES_END) -#define MODULES_END (MODULES_VADDR + MODULES_VSIZE) -#define MODULES_VADDR (_PAGE_END(VA_BITS_MIN)) + +/* + * Setting KIMAGE_VADDR has got a lot harder, ideally we'd like to use + * min(PGDIR_SIZE, MODULES_VSIZE) but this can't work because this is used + * both in assembly as C, where it causes problems. min_t() solves the + * C problems but can't be used in assembly. + * CONFIG_ARM64_4K_PAGES, PGDIR_SIZE is 2M, 1G, 512G + * CONFIG_ARM64_16K_PAGES, PGDIR_SIZE is 32M, 64G or 128T + * CONFIG_ARM64_64K_PAGES, PGDIR_SIZE is 512M or 4T + */ +#if (CONFIG_ARM64_4K_PAGES && CONFIG_PGTABLE_LEVELS < 4) || \ + (CONFIG_ARM64_16K_PAGES && CONFIG_PGTABLE_LEVELS < 3) || \ + (CONFIG_ARM64_64K_PAGES && CONFIG_PGTABLE_LEVELS < 2) +#define KIMAGE_OFFSET MODULES_VSIZE +#else +#define KIMAGE_OFFSET PGDIR_SIZE +#endif +#define KIMAGE_VADDR (_PAGE_END(VA_BITS_MIN) + KIMAGE_OFFSET) +#define MODULES_END (KIMAGE_VADDR) +#define MODULES_VADDR (MODULES_END - MODULES_VSIZE) #define MODULES_VSIZE (SZ_2G) #define VMEMMAP_START (-(UL(1) << (VA_BITS - VMEMMAP_SHIFT))) #define VMEMMAP_END (VMEMMAP_START + VMEMMAP_SIZE) @@ -199,10 +216,7 @@ extern u64 kimage_vaddr; /* the offset between the kernel virtual and physical mappings */ extern u64 kimage_voffset; -static inline unsigned long kaslr_offset(void) -{ - return kimage_vaddr - KIMAGE_VADDR; -} +#define kaslr_offset() ((unsigned long)(kimage_vaddr - KIMAGE_VADDR)) #ifdef CONFIG_RANDOMIZE_BASE void kaslr_init(void); diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 0bd18de9fd97..cb526e69299d 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -21,7 +21,7 @@ * VMALLOC_END: extends to the available space below vmemmap, PCI I/O space * and fixed mappings */ -#define VMALLOC_START (MODULES_END) +#define VMALLOC_START (MODULES_END + PGDIR_SIZE) #define VMALLOC_END (VMEMMAP_START - SZ_256M) #define vmemmap ((struct page *)VMEMMAP_START - (memstart_addr >> PAGE_SHIFT)) diff --git a/arch/arm64/kernel/kaslr.c b/arch/arm64/kernel/kaslr.c index 94a269cd1f07..6ffea2ce1a11 100644 --- a/arch/arm64/kernel/kaslr.c +++ b/arch/arm64/kernel/kaslr.c @@ -9,6 +9,7 @@ #include #include +#include u16 __initdata memstart_offset_seed; From patchwork Thu Jul 27 15:30:15 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Russell King (Oracle)" X-Patchwork-Id: 13330296 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 195B2C001E0 for ; Thu, 27 Jul 2023 15:30:52 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Date:Message-Id:MIME-Version:Subject:Cc :To:From:References:In-Reply-To:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=xKLw/FHXaEyvJFLKRDrV4MELJ3U+EDW77wg4FbO5Y4Q=; b=dqLujLpTZKarf5 4rAjoLs9c6k8fJqKZjyPE9qgqbTfdxMvpO3b2EiMntJ03k//GkjYkl0FiPLmDVxvYRbUpJE/WvZRD S1X/Lge5Va9eYXL8wpBXGOJeEEeH9/nOTPR2v8J8T280tJ5p1bNVaOk6OfJmyVCG/5gSiF162A2UA 2VMWO0CIkmJe+pI/CYAcN+P3e6eYEDWGMfnIC2aqEH8fCsH+cDG3kEDl76uiUwvqjG1ICRiNIVX42 hvsg22DsVGGuM2F7apsiqeyFlfb2X9wUeq5HHAE3Kci9mNuXSj9pXLV5kFoYjmy0bU49BDBnEQCh2 gHZqoz0voR88bZupeCEg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qP2wb-00FuPk-19; Thu, 27 Jul 2023 15:30:25 +0000 Received: from [2001:4d48:ad52:32c8:5054:ff:fe00:142] (helo=pandora.armlinux.org.uk) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qP2wV-00FuJC-04 for linux-arm-kernel@lists.infradead.org; Thu, 27 Jul 2023 15:30:21 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=armlinux.org.uk; s=pandora-2019; h=Date:Sender:Message-Id:Content-Type: Content-Transfer-Encoding:MIME-Version:Subject:Cc:To:From:References: In-Reply-To:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help: List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=0W65xWM7Ob4dljZQ3ePDRFaG7Y0QsYhjSpbvt2L/B28=; b=a9ntwDIYuJlV5/RFq2KFwonAZ6 CNjzyu/7oJYgMvb/e49MK+k3UYihAZuRj/T1S5+YfAKbvcTZevMbbU4HreCpOlEbE06jAGtLvZ+hs CBisprgeaXHLt0PjocTpo4YTkfhiASSDmbLaOPvv769n2Zq0IoRdxJQE0BJNRkuMKrQiMiuTjPq9V DzC4PDZVCUh0Lturn3xTm7sLTIYn9d9jtoUbH7I7NkEl4pGYEi5VeRZfP3TmKnmfH8ENJO/x3BNi8 yictd2h0ORdUN7G9UxHWNeDugUP8z6rWxLr/LSNYRXoydCv7IfWvg2nwQG4E8S005Hf/a2hpr3g3u x3fBBYfA==; Received: from e0022681537dd.dyn.armlinux.org.uk ([fd8f:7570:feb6:1:222:68ff:fe15:37dd]:49430 helo=rmk-PC.armlinux.org.uk) by pandora.armlinux.org.uk with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.96) (envelope-from ) id 1qP2wR-00062l-2E; Thu, 27 Jul 2023 16:30:15 +0100 Received: from rmk by rmk-PC.armlinux.org.uk with local (Exim 4.94.2) (envelope-from ) id 1qP2wR-001CXE-Ro; Thu, 27 Jul 2023 16:30:15 +0100 In-Reply-To: References: From: "Russell King (Oracle)" To: Catalin Marinas , Jonathan Corbet , Will Deacon Cc: linux-arm-kernel@lists.infradead.org, linux-doc@vger.kernel.org Subject: [PATCH v2 04/16] arm64: text replication: add init function MIME-Version: 1.0 Content-Disposition: inline Message-Id: Date: Thu, 27 Jul 2023 16:30:15 +0100 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230727_083019_064549_4111A367 X-CRM114-Status: GOOD ( 15.55 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org A simple patch that adds an empty function for kernel text replication initialisation and hooks it into the initialisation path. Signed-off-by: Russell King (Oracle) --- arch/arm64/include/asm/ktext.h | 20 ++++++++++++++++++++ arch/arm64/mm/Makefile | 2 ++ arch/arm64/mm/init.c | 3 +++ arch/arm64/mm/ktext.c | 8 ++++++++ 4 files changed, 33 insertions(+) create mode 100644 arch/arm64/include/asm/ktext.h create mode 100644 arch/arm64/mm/ktext.c diff --git a/arch/arm64/include/asm/ktext.h b/arch/arm64/include/asm/ktext.h new file mode 100644 index 000000000000..1a5f7452a3bf --- /dev/null +++ b/arch/arm64/include/asm/ktext.h @@ -0,0 +1,20 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* + * Copyright (C) 2022, Oracle and/or its affiliates. + */ +#ifndef ASM_KTEXT_H +#define ASM_KTEXT_H + +#ifdef CONFIG_REPLICATE_KTEXT + +void ktext_replication_init(void); + +#else + +static inline void ktext_replication_init(void) +{ +} + +#endif + +#endif diff --git a/arch/arm64/mm/Makefile b/arch/arm64/mm/Makefile index dbd1bc95967d..41e705027c57 100644 --- a/arch/arm64/mm/Makefile +++ b/arch/arm64/mm/Makefile @@ -14,3 +14,5 @@ KASAN_SANITIZE_physaddr.o += n obj-$(CONFIG_KASAN) += kasan_init.o KASAN_SANITIZE_kasan_init.o := n + +obj-$(CONFIG_REPLICATE_KTEXT) += ktext.o diff --git a/arch/arm64/mm/init.c b/arch/arm64/mm/init.c index d31c3a9290c5..df90738fd6bb 100644 --- a/arch/arm64/mm/init.c +++ b/arch/arm64/mm/init.c @@ -36,6 +36,7 @@ #include #include #include +#include #include #include #include @@ -425,6 +426,8 @@ void __init bootmem_init(void) arch_numa_init(); + ktext_replication_init(); + /* * must be done after arch_numa_init() which calls numa_init() to * initialize node_online_map that gets used in hugetlb_cma_reserve() diff --git a/arch/arm64/mm/ktext.c b/arch/arm64/mm/ktext.c new file mode 100644 index 000000000000..3a8d37c9abc4 --- /dev/null +++ b/arch/arm64/mm/ktext.c @@ -0,0 +1,8 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright (C) 2022, Oracle and/or its affiliates. + */ + +void __init ktext_replication_init(void) +{ +} From patchwork Thu Jul 27 15:30:20 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Russell King (Oracle)" X-Patchwork-Id: 13330298 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id AE794C001DC for ; Thu, 27 Jul 2023 15:31:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Date:Message-Id:MIME-Version:Subject:Cc :To:From:References:In-Reply-To:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=AU1Sd5wzrFWxr+3Zv+LuOVQROZYurxd8H3YEK+p9/Zo=; b=QX0Tl8NjvskflS jyETzzCOzOhG9MzyBhZdTnFyFRUjP25GCBJaCY8J7vz6sN96MGmT69+5h92NzkSOvCWsEHv9k9Eux lxCfE1umrHocDHVmvoLd6TTMFK2SYoD1LPABChGUbm8igxhmJbEeyeHxrR4JciZAsTi56EL2b5Nqj m/ur4SBaLtDfJogCn8Og1vPwZ+DZ0PKUlB4AUb7AJlnEHayw6S5BDBbQxPyBv7bi0N8CUt5UXZeqV 2SYVE4BuEcWwJz3ul7fZIlHQkEeR0IktZhnAltjPjmarVLK357ZOg4GEZ5Bl9Yo49nXem3M/ihEiY 4OEngLZw8wbdIlRVgCaQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qP2wr-00FudU-2r; Thu, 27 Jul 2023 15:30:41 +0000 Received: from [2001:4d48:ad52:32c8:5054:ff:fe00:142] (helo=pandora.armlinux.org.uk) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qP2wZ-00FuNG-1n for linux-arm-kernel@lists.infradead.org; Thu, 27 Jul 2023 15:30:25 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=armlinux.org.uk; s=pandora-2019; h=Date:Sender:Message-Id:Content-Type: Content-Transfer-Encoding:MIME-Version:Subject:Cc:To:From:References: In-Reply-To:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help: List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=nyj2l8PY3qAzzBlei2fGIlbkY3GLY8pUVEJtHA0LwRE=; b=bVgJGDhGTIg5HdLjMLDsEtxgmv m3DvR2CYMo8YKaU+1N089+ohCZTsM9XponAqRuB5zRKUw+nvNie0fjBreSNKyyGDYVrkHgfk2fA+p 0dRJxVeniunsHnyiGtCUc+Y9qHqTOYqGRBa3z1bGZPxWprHv8qtaOOFZOUfb2vkm6WafgAvJYnTIG QX/dGIM8FYiWJTFDVhy/pmnTaAIKXjxeOsHPMUUVrkXptOfaXdny0a4J8gzMrSReHggcFYDhv8Xo4 4RYutcI8bwGwR+HHl35qIgAImZbjkM2pjS26YrTIqtktPa4AF84a8ce97paGka0wJ0BBryO5aFHD6 7lxHSueg==; Received: from e0022681537dd.dyn.armlinux.org.uk ([fd8f:7570:feb6:1:222:68ff:fe15:37dd]:49446 helo=rmk-PC.armlinux.org.uk) by pandora.armlinux.org.uk with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.96) (envelope-from ) id 1qP2wW-00062v-2i; Thu, 27 Jul 2023 16:30:20 +0100 Received: from rmk by rmk-PC.armlinux.org.uk with local (Exim 4.94.2) (envelope-from ) id 1qP2wW-001CXK-WB; Thu, 27 Jul 2023 16:30:21 +0100 In-Reply-To: References: From: "Russell King (Oracle)" To: Catalin Marinas , Jonathan Corbet , Will Deacon Cc: linux-arm-kernel@lists.infradead.org, linux-doc@vger.kernel.org Subject: [PATCH v2 05/16] arm64: text replication: add sanity checks MIME-Version: 1.0 Content-Disposition: inline Message-Id: Date: Thu, 27 Jul 2023 16:30:20 +0100 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230727_083023_611212_ECC6F099 X-CRM114-Status: GOOD ( 11.37 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org The kernel text and modules must be in separate L0 page table entries. Signed-off-by: Russell King (Oracle) --- arch/arm64/mm/ktext.c | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) diff --git a/arch/arm64/mm/ktext.c b/arch/arm64/mm/ktext.c index 3a8d37c9abc4..901f159c65e6 100644 --- a/arch/arm64/mm/ktext.c +++ b/arch/arm64/mm/ktext.c @@ -3,6 +3,27 @@ * Copyright (C) 2022, Oracle and/or its affiliates. */ +#include +#include + +#include +#include + void __init ktext_replication_init(void) { + int kidx = pgd_index((phys_addr_t)KERNEL_START); + + /* + * If we've messed up and the kernel shares a L0 entry with the + * module or vmalloc area, then don't even attempt to use text + * replication. + */ + if (pgd_index(MODULES_VADDR) == kidx) { + pr_warn("Kernel is located in the same L0 index as modules - text replication disabled\n"); + return; + } + if (pgd_index(VMALLOC_START) == kidx) { + pr_warn("Kernel is located in the same L0 index as vmalloc - text replication disabled\n"); + return; + } } From patchwork Thu Jul 27 15:30:26 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Russell King (Oracle)" X-Patchwork-Id: 13330299 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 97B77C001E0 for ; Thu, 27 Jul 2023 15:31:08 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Date:Message-Id:MIME-Version:Subject:Cc :To:From:References:In-Reply-To:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=WVzP3yr1So6jbzqKFLjITmKXfTkGTV6vKUAAiQOsXdM=; b=Vlkg1bJcuypCGT UNP4KZt23iNdsRfNHsqlWjTgn0siIrgEJjKfdKfziQyQzDnGZxmRYWtKM7Z0kN8lN6iW2SgJO1Eps IdxvaeLevhydxt9YdrV4lRiQ/qdEgVNHtA08RD3dCSNfdBrksgOju9uEuoYrNnv5EvVZXkdZBRLfW UY+f0TUaD3QQ0HBE9J97iBF3B625qtPm+qhl8KvtKkl+/umiNzVjQs+C7z7rNhiqDwwE9dHz+xoGZ cwSeyWTjkHgYkBy7X8DpRAl3gui1xkd5vaNpTR+QpGurt0QVJsarIKH3NjoOQyKoOCRL2X8O08CXk IteDIpHnYY+/i2wENY8w==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qP2ws-00FueF-1s; Thu, 27 Jul 2023 15:30:42 +0000 Received: from [2001:4d48:ad52:32c8:5054:ff:fe00:142] (helo=pandora.armlinux.org.uk) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qP2we-00FuRW-2U for linux-arm-kernel@lists.infradead.org; Thu, 27 Jul 2023 15:30:30 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=armlinux.org.uk; s=pandora-2019; h=Date:Sender:Message-Id:Content-Type: Content-Transfer-Encoding:MIME-Version:Subject:Cc:To:From:References: In-Reply-To:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help: List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=kS6q0fvuI7d1aTJBBBQTVQRRjCIadUHAhNRNeZ4W2zs=; b=WRD0SkBZpN+XFnicJDX3+ZDD6L PFrIl2UBjYeskUrLB7Elnt51CIuMs0qtyzpmdXQ7ZnhTYBYZFlZk0gz01Nj2YkjLWOyJTDwuBf5+I CfjjIBllw0f5LV+O0h/VvFlClEc4wCy7HKEh0mginR7S6cZtjxMr612xCxSbQE7Ni6804AzgrJedP i2YgYobQwk1m9lu/xgn3GjE2I/oJ6yYGpG7P0W2vOsV9XXmrkzvLJI28OtQ/MC6SywIh3Dm/qhDYG b4X/gma9LThFGQwJ9nnXQwpra+X21kEqUt/fA+sRi9b/qC5xC+eDpNkYh8gA6pdl3Sl/mc9rmd5Ya MIYvQ+Aw==; Received: from e0022681537dd.dyn.armlinux.org.uk ([fd8f:7570:feb6:1:222:68ff:fe15:37dd]:45314 helo=rmk-PC.armlinux.org.uk) by pandora.armlinux.org.uk with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.96) (envelope-from ) id 1qP2wb-000639-37; Thu, 27 Jul 2023 16:30:26 +0100 Received: from rmk by rmk-PC.armlinux.org.uk with local (Exim 4.94.2) (envelope-from ) id 1qP2wc-001CXQ-4h; Thu, 27 Jul 2023 16:30:26 +0100 In-Reply-To: References: From: "Russell King (Oracle)" To: Catalin Marinas , Jonathan Corbet , Will Deacon Cc: linux-arm-kernel@lists.infradead.org, linux-doc@vger.kernel.org Subject: [PATCH v2 06/16] arm64: text replication: copy initial kernel text MIME-Version: 1.0 Content-Disposition: inline Message-Id: Date: Thu, 27 Jul 2023 16:30:26 +0100 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230727_083028_817884_8041395D X-CRM114-Status: GOOD ( 14.80 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Allocate memory on the appropriate node for the per-node copies of the kernel text, and copy the kernel text to that memory. Clean and invalidate the caches to the point of unification so that the copied text is correctly visible to the target node. Signed-off-by: Russell King (Oracle) --- arch/arm64/mm/ktext.c | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) diff --git a/arch/arm64/mm/ktext.c b/arch/arm64/mm/ktext.c index 901f159c65e6..4c803b89fcfe 100644 --- a/arch/arm64/mm/ktext.c +++ b/arch/arm64/mm/ktext.c @@ -4,14 +4,23 @@ */ #include +#include +#include #include +#include +#include #include #include +static void *kernel_texts[MAX_NUMNODES]; + +/* Allocate memory for the replicated kernel texts. */ void __init ktext_replication_init(void) { + size_t size = _etext - _stext; int kidx = pgd_index((phys_addr_t)KERNEL_START); + int nid; /* * If we've messed up and the kernel shares a L0 entry with the @@ -26,4 +35,16 @@ void __init ktext_replication_init(void) pr_warn("Kernel is located in the same L0 index as vmalloc - text replication disabled\n"); return; } + + for_each_node(nid) { + /* Nothing to do for node 0 */ + if (!nid) + continue; + + /* Allocate and copy initial kernel text for this node */ + kernel_texts[nid] = memblock_alloc_node(size, PAGE_SIZE, nid); + memcpy(kernel_texts[nid], _stext, size); + caches_clean_inval_pou((u64)kernel_texts[nid], + (u64)kernel_texts[nid] + size); + } } From patchwork Thu Jul 27 15:30:31 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Russell King (Oracle)" X-Patchwork-Id: 13330297 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id AEFB2C04A6A for ; Thu, 27 Jul 2023 15:31:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Date:Message-Id:MIME-Version:Subject:Cc :To:From:References:In-Reply-To:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=n2sXbvEScMj+e+MRRAGZuiwvzNK3+fNhWE9JEuM/zNo=; b=pDyNwlUSKC7bzo jqtHQ/9strsLTFl1J2eXcTFzrNmNsBzFivYmBxxLEI+GIeA8s4A+4cMR1HANWZFBVVIoQx8HLZPs6 DWozD1jcezn/u7FX1vg2ufBtAQCC7RVqZh0iN8/p/kx60VS9pi1mV/yap2LaBmATmisBkUWRNpghE zzL37yEzG1sNCf5eG8To5Q69JGDmhtu9m/wAEAqjTbSzClOhX/cF57gna5QxpdpVv+OmbXRdmNcmX 1D/X80Qw7+CIOE2fvHTyIjAi10X25LwlEFUJEc+SnaYuHPsXWASh7vW+OxHfP378GC44i+1gbCmCB H9rupmhLprkmER+vM3Xg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qP2wt-00FufC-1S; Thu, 27 Jul 2023 15:30:43 +0000 Received: from [2001:4d48:ad52:32c8:5054:ff:fe00:142] (helo=pandora.armlinux.org.uk) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qP2wj-00FuVK-2Q for linux-arm-kernel@lists.infradead.org; Thu, 27 Jul 2023 15:30:35 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=armlinux.org.uk; s=pandora-2019; h=Date:Sender:Message-Id:Content-Type: Content-Transfer-Encoding:MIME-Version:Subject:Cc:To:From:References: In-Reply-To:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help: List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=H67bVYoLhWNzChiULPPa2CQcWOVHZM/oIAaPLDi7i9o=; b=zD9bi4qUtGRJwQel6yFqA04Ryc 07ybJhD87QsbkPjjVvVlZkwyt+bDa34pxDlWNjyucitEMCHiVceCeqNUdaAQ6a5AAD25/zq/Gp6qn EE8LzKFEgBoVJCCrbn2J3bVml0sGl5TAcgLnA450dfdSZU+sNkt9qc3Fv7OcuOSn+Ini57joQ4HyQ x+vYU8BKOt0291qFiH9XWRaiFQ6An9K40OD0rx0raJIbb7dAiyIcHUM5MVIuC6d8fRZLQaeRHgebq VxGRtBSxgzLu0+VxyNjGhU9N3Rd/2Eco0LqBCULG8ZxoXS/PbQ/WnFqgjXE0JeT81JzZ19Pof1BXk Cg7/swEQ==; Received: from e0022681537dd.dyn.armlinux.org.uk ([fd8f:7570:feb6:1:222:68ff:fe15:37dd]:45326 helo=rmk-PC.armlinux.org.uk) by pandora.armlinux.org.uk with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.96) (envelope-from ) id 1qP2wh-00063K-0V; Thu, 27 Jul 2023 16:30:31 +0100 Received: from rmk by rmk-PC.armlinux.org.uk with local (Exim 4.94.2) (envelope-from ) id 1qP2wh-001CXX-9L; Thu, 27 Jul 2023 16:30:31 +0100 In-Reply-To: References: From: "Russell King (Oracle)" To: Catalin Marinas , Jonathan Corbet , Will Deacon Cc: linux-arm-kernel@lists.infradead.org, linux-doc@vger.kernel.org Subject: [PATCH v2 07/16] arm64: text replication: add node text patching MIME-Version: 1.0 Content-Disposition: inline Message-Id: Date: Thu, 27 Jul 2023 16:30:31 +0100 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230727_083033_805266_2054E76A X-CRM114-Status: GOOD ( 17.92 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Add support for text patching on our replicated texts. Signed-off-by: Russell King (Oracle) --- arch/arm64/include/asm/ktext.h | 12 +++++++ arch/arm64/kernel/alternative.c | 2 ++ arch/arm64/kernel/patching.c | 7 +++- arch/arm64/mm/ktext.c | 58 +++++++++++++++++++++++++++++++++ 4 files changed, 78 insertions(+), 1 deletion(-) diff --git a/arch/arm64/include/asm/ktext.h b/arch/arm64/include/asm/ktext.h index 1a5f7452a3bf..289e11289c06 100644 --- a/arch/arm64/include/asm/ktext.h +++ b/arch/arm64/include/asm/ktext.h @@ -5,9 +5,13 @@ #ifndef ASM_KTEXT_H #define ASM_KTEXT_H +#include + #ifdef CONFIG_REPLICATE_KTEXT void ktext_replication_init(void); +void __kprobes ktext_replication_patch(u32 *tp, __le32 insn); +void ktext_replication_patch_alternative(__le32 *src, int nr_inst); #else @@ -15,6 +19,14 @@ static inline void ktext_replication_init(void) { } +static inline void __kprobes ktext_replication_patch(u32 *tp, __le32 insn) +{ +} + +static inline void ktext_replication_patch_alternative(__le32 *src, int nr_inst) +{ +} + #endif #endif diff --git a/arch/arm64/kernel/alternative.c b/arch/arm64/kernel/alternative.c index ea3f4104771d..6f17e2b4e1c3 100644 --- a/arch/arm64/kernel/alternative.c +++ b/arch/arm64/kernel/alternative.c @@ -15,6 +15,7 @@ #include #include #include +#include #include #include #include @@ -174,6 +175,7 @@ static void __apply_alternatives(const struct alt_region *region, alt_cb(alt, origptr, updptr, nr_inst); if (!is_module) { + ktext_replication_patch_alternative(updptr, nr_inst); clean_dcache_range_nopatch((u64)origptr, (u64)(origptr + nr_inst)); } diff --git a/arch/arm64/kernel/patching.c b/arch/arm64/kernel/patching.c index b4835f6d594b..627fff6ddda2 100644 --- a/arch/arm64/kernel/patching.c +++ b/arch/arm64/kernel/patching.c @@ -10,6 +10,7 @@ #include #include #include +#include #include #include @@ -115,9 +116,13 @@ int __kprobes aarch64_insn_patch_text_nosync(void *addr, u32 insn) return -EINVAL; ret = aarch64_insn_write(tp, insn); - if (ret == 0) + if (ret == 0) { + /* Also patch the other nodes */ + ktext_replication_patch(tp, cpu_to_le32(insn)); + caches_clean_inval_pou((uintptr_t)tp, (uintptr_t)tp + AARCH64_INSN_SIZE); + } return ret; } diff --git a/arch/arm64/mm/ktext.c b/arch/arm64/mm/ktext.c index 4c803b89fcfe..04b5ceddae4e 100644 --- a/arch/arm64/mm/ktext.c +++ b/arch/arm64/mm/ktext.c @@ -3,8 +3,10 @@ * Copyright (C) 2022, Oracle and/or its affiliates. */ +#include #include #include +#include #include #include #include @@ -15,6 +17,62 @@ static void *kernel_texts[MAX_NUMNODES]; +void __kprobes ktext_replication_patch(u32 *tp, __le32 insn) +{ + unsigned long offset; + int nid, this_nid; + __le32 *p; + + if (!is_kernel_text((unsigned long)tp)) + return; + + offset = (unsigned long)tp - (unsigned long)_stext; + + this_nid = numa_node_id(); + if (this_nid) { + /* The cache maintenance by aarch64_insn_patch_text_nosync() + * will occur on this node. We need it to occur on node 0. + */ + p = (void *)lm_alias(_stext) + offset; + caches_clean_inval_pou((u64)p, (u64)p + AARCH64_INSN_SIZE); + } + + for_each_node(nid) { + if (!kernel_texts[nid]) + continue; + + p = kernel_texts[nid] + offset; + WRITE_ONCE(*p, insn); + caches_clean_inval_pou((u64)p, (u64)p + AARCH64_INSN_SIZE); + } +} + +/* Copy the patched alternative from the node0 image to the other + * modes. src is the node 0 linear-mapping address. + */ +void ktext_replication_patch_alternative(__le32 *src, int nr_inst) +{ + unsigned long offset; + size_t size; + int nid; + __le32 *p; + + offset = (unsigned long)src - (unsigned long)lm_alias(_stext); + if (offset >= _etext - _stext) + return; + + size = AARCH64_INSN_SIZE * nr_inst; + + for_each_node(nid) { + if (!kernel_texts[nid]) + continue; + + p = kernel_texts[nid] + offset; + memcpy(p, src, size); + clean_dcache_range_nopatch((u64)p, (u64)p + size); + } +} + /* Allocate memory for the replicated kernel texts. */ void __init ktext_replication_init(void) { From patchwork Thu Jul 27 15:30:36 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Russell King (Oracle)" X-Patchwork-Id: 13330300 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id AB8A6C04A6A for ; Thu, 27 Jul 2023 15:31:11 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Date:Message-Id:MIME-Version:Subject:Cc :To:From:References:In-Reply-To:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=o73xWTRLwu8NDXy/UL/4y6So10PTCL9iG5AAtP5HIs0=; b=xKuonsH7Pk6X1D eDlQQ/0odB/8eTWeC/6IN9dBCGhBJySWgX65dxDojM2Ji78nr9Daul0aadCJmjWKrYzan0oBpIEI0 b3BpcLPF+q4AYkTUOl8o7YH65PgFjsEoLRSYU7dlpwwWsSXfqfjxc8g4XWaow5uTcL4uoo7iUFai8 BPwoXg4nDfD0Doi1agzSUObYkXwZR02LqQ/WFOGqXaxeBCBhl0lwZyX0H+B/cOgG+OdZWUf8qzgFM YOzKXgaGQPym/tUp+jt6z9RDa54/VMiU1S51d5UhaxLXfD7rOVl7oVz68dicg1RV0fQs3HOXMDMfo zE8mWn8+16S+5/XMh/Ww==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qP2wu-00FugD-12; Thu, 27 Jul 2023 15:30:44 +0000 Received: from [2001:4d48:ad52:32c8:5054:ff:fe00:142] (helo=pandora.armlinux.org.uk) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qP2wo-00FuZP-2k for linux-arm-kernel@lists.infradead.org; Thu, 27 Jul 2023 15:30:40 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=armlinux.org.uk; s=pandora-2019; h=Date:Sender:Message-Id:Content-Type: Content-Transfer-Encoding:MIME-Version:Subject:Cc:To:From:References: In-Reply-To:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help: List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=IOBpLtQM2CaS7C/Ec+fNgQfrrd1kDqArSAg2kFIx4jc=; b=SwqvptSyYvmu105/RME6qd+8Wt c6TggVh2DC9x6UZzY47UvebgDy5Dfk33d/sT68snQM65KsuWW/lx3x716PHLiGsy1i2zoNqf930e8 KbdrpXioGSMpmjWlh5aJL3FZNuu5XBf1u9VxKo06zX68DxqB8y36wC8hgAgTe+exet2hINICNwfAy FDFWE02TIQfpM/H+YVVtWWfCF5LfgCscoC4k2egF7lWe6GRv6IgpUHizlI+phnR3CwxSvcBShpI7r VBQ0PnImZ7ySSS8qWcoc47N4H4Mg8fH45g2mZPcVUeFxw+421LIEubIpWbqp864eK6alN4+taf4TF Y4Nj1ejA==; Received: from e0022681537dd.dyn.armlinux.org.uk ([fd8f:7570:feb6:1:222:68ff:fe15:37dd]:43352 helo=rmk-PC.armlinux.org.uk) by pandora.armlinux.org.uk with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.96) (envelope-from ) id 1qP2wm-00063e-0x; Thu, 27 Jul 2023 16:30:36 +0100 Received: from rmk by rmk-PC.armlinux.org.uk with local (Exim 4.94.2) (envelope-from ) id 1qP2wm-001CXd-Ep; Thu, 27 Jul 2023 16:30:36 +0100 In-Reply-To: References: From: "Russell King (Oracle)" To: Catalin Marinas , Jonathan Corbet , Will Deacon Cc: linux-arm-kernel@lists.infradead.org, linux-doc@vger.kernel.org Subject: [PATCH v2 08/16] arm64: text replication: add node 0 page table definitions MIME-Version: 1.0 Content-Disposition: inline Message-Id: Date: Thu, 27 Jul 2023 16:30:36 +0100 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230727_083038_908233_1D39E87C X-CRM114-Status: GOOD ( 11.39 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Add a struct definition for the level zero page table group (the optional trampoline page tables, reserved page tables, and swapper page tables). Add a symbol and extern declaration for the node 0 page table group. Add an array of pointers to per-node page tables, which will default to using the node 0 page table group. Signed-off-by: Russell King (Oracle) --- arch/arm64/include/asm/pgtable.h | 14 ++++++++++++++ arch/arm64/kernel/vmlinux.lds.S | 3 +++ arch/arm64/mm/ktext.c | 4 ++++ 3 files changed, 21 insertions(+) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index cb526e69299d..1e72067d1e9e 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -615,6 +615,20 @@ extern pgd_t idmap_pg_dir[PTRS_PER_PGD]; extern pgd_t tramp_pg_dir[PTRS_PER_PGD]; extern pgd_t reserved_pg_dir[PTRS_PER_PGD]; +struct pgtables { +#ifdef CONFIG_UNMAP_KERNEL_AT_EL0 + pgd_t tramp_pg_dir[PTRS_PER_PGD]; +#endif + pgd_t reserved_pg_dir[PTRS_PER_PGD]; + pgd_t swapper_pg_dir[PTRS_PER_PGD]; +}; + +extern struct pgtables pgtable_node0; + +#ifdef CONFIG_REPLICATE_KTEXT +extern struct pgtables *pgtables[MAX_NUMNODES]; +#endif + extern void set_swapper_pgd(pgd_t *pgdp, pgd_t pgd); static inline bool in_swapper_pgdir(void *addr) diff --git a/arch/arm64/kernel/vmlinux.lds.S b/arch/arm64/kernel/vmlinux.lds.S index 3cd7e76cc562..d3c7ed76adbf 100644 --- a/arch/arm64/kernel/vmlinux.lds.S +++ b/arch/arm64/kernel/vmlinux.lds.S @@ -212,6 +212,9 @@ SECTIONS idmap_pg_dir = .; . += PAGE_SIZE; + /* pgtable struct - covers the tramp, reserved and swapper pgdirs */ + pgtable_node0 = .; + #ifdef CONFIG_UNMAP_KERNEL_AT_EL0 tramp_pg_dir = .; . += PAGE_SIZE; diff --git a/arch/arm64/mm/ktext.c b/arch/arm64/mm/ktext.c index 04b5ceddae4e..48d7943d6907 100644 --- a/arch/arm64/mm/ktext.c +++ b/arch/arm64/mm/ktext.c @@ -15,6 +15,10 @@ #include #include +struct pgtables *pgtables[MAX_NUMNODES] = { + [0 ... MAX_NUMNODES - 1] = &pgtable_node0, +}; + static void *kernel_texts[MAX_NUMNODES]; void __kprobes ktext_replication_patch(u32 *tp, __le32 insn) From patchwork Thu Jul 27 15:30:41 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Russell King (Oracle)" X-Patchwork-Id: 13330301 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 4A259C001DC for ; Thu, 27 Jul 2023 15:31:40 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Date:Message-Id:MIME-Version:Subject:Cc :To:From:References:In-Reply-To:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=IKnnOAsJ+Tmnv1F8pcpyRnjKGJjEpl9Hsn6BJMvdjbM=; b=c6+BbdPxOB14yA qLwgjDEE+xPhMe7xKfMPgrjTRL6CVraA0S3w2dBlsFqeqX+2sZsaDYRWsTr77g2It86epUmgJG4iM km8DqGHUVwUmwrLqt2a0CN8wj5WPzAxfE7KH9ZzeTg9k0tcRsUP+UfkNicdd3qGoRZ6YFLyBEksNp HERTSRL/edq4+OwrRh7FM8MT3rrPXEwNYvEW8JeqPWZ9kqSo748wRZelmtDpvxvRJiZmZucvRZbrJ v/OB+qoQWnZSQMNHds/OFAJKj5Yo8vomWbvv4luR12304wrGxhH/raMDeNvaqUBKohUUZoAB6zJHh MXg/tznh9vJY9fY+DFgA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qP2xU-00FvCm-1F; Thu, 27 Jul 2023 15:31:20 +0000 Received: from [2001:4d48:ad52:32c8:5054:ff:fe00:142] (helo=pandora.armlinux.org.uk) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qP2wu-00FueA-0A for linux-arm-kernel@lists.infradead.org; Thu, 27 Jul 2023 15:30:46 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=armlinux.org.uk; s=pandora-2019; h=Date:Sender:Message-Id:Content-Type: Content-Transfer-Encoding:MIME-Version:Subject:Cc:To:From:References: In-Reply-To:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help: List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=H/bsnJ3nxuaPYk8WVBKA64ofddky6JIAr3Z9JZdkWE0=; b=rwwK7hWh69mXMulm6Lyj8ENCmS 3V8Hhm048cnQAyM0EYGXpM+vH+/EImP5q3DHuua+f2mXftSTXq6rrKQckRqYwCqI2BpZwCFFLyLJK Z8g4q2xFCA5NP+szKiWfr6ixU7KoWdys+BY7vK38sRYhHPTTi/++OGXTO3JVJsVKbO7hG2Yqz/KFa hJZXBBYvGrWbNk8VUQQtP58jGtX9P9fRPfiLFdqszNd0mDXcNOHM39UbgNVVgage7OtVbpt7wQmi6 ayRnL9pGOAKOlzZU5wl+IKoBqAD7vpEi6KBNzbqhbkyzox+npuD8dJlsH0L1YbFXYM5jgX6o421+w m+RVuO1g==; Received: from e0022681537dd.dyn.armlinux.org.uk ([fd8f:7570:feb6:1:222:68ff:fe15:37dd]:43366 helo=rmk-PC.armlinux.org.uk) by pandora.armlinux.org.uk with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.96) (envelope-from ) id 1qP2wr-000642-1O; Thu, 27 Jul 2023 16:30:41 +0100 Received: from rmk by rmk-PC.armlinux.org.uk with local (Exim 4.94.2) (envelope-from ) id 1qP2wr-001CXk-JZ; Thu, 27 Jul 2023 16:30:41 +0100 In-Reply-To: References: From: "Russell King (Oracle)" To: Catalin Marinas , Jonathan Corbet , Will Deacon Cc: linux-arm-kernel@lists.infradead.org, linux-doc@vger.kernel.org Subject: [PATCH v2 09/16] arm64: text replication: add swapper page directory helpers MIME-Version: 1.0 Content-Disposition: inline Message-Id: Date: Thu, 27 Jul 2023 16:30:41 +0100 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230727_083044_295326_6C1E9789 X-CRM114-Status: GOOD ( 12.78 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Add a series of helpers for the swapper page directories - a set which return those for the calling CPU, and those which take the NUMA node number. Signed-off-by: Russell King (Oracle) --- arch/arm64/include/asm/pgtable.h | 19 +++++++++++++++++++ arch/arm64/kernel/hibernate.c | 2 +- arch/arm64/mm/ktext.c | 20 ++++++++++++++++++++ 3 files changed, 40 insertions(+), 1 deletion(-) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 1e72067d1e9e..5cfff64e4944 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -627,6 +627,25 @@ extern struct pgtables pgtable_node0; #ifdef CONFIG_REPLICATE_KTEXT extern struct pgtables *pgtables[MAX_NUMNODES]; + +pgd_t *swapper_pg_dir_node(void); +phys_addr_t __swapper_pg_dir_node_phys(int nid); +phys_addr_t swapper_pg_dir_node_phys(void); +#else +static inline pgd_t *swapper_pg_dir_node(void) +{ + return swapper_pg_dir; +} + +static inline phys_addr_t __swapper_pg_dir_node_phys(int nid) +{ + return __pa_symbol(swapper_pg_dir); +} + +static inline phys_addr_t swapper_pg_dir_node_phys(void) +{ + return __pa_symbol(swapper_pg_dir); +} #endif extern void set_swapper_pgd(pgd_t *pgdp, pgd_t pgd); diff --git a/arch/arm64/kernel/hibernate.c b/arch/arm64/kernel/hibernate.c index 02870beb271e..be69515da802 100644 --- a/arch/arm64/kernel/hibernate.c +++ b/arch/arm64/kernel/hibernate.c @@ -113,7 +113,7 @@ int arch_hibernation_header_save(void *addr, unsigned int max_size) return -EOVERFLOW; arch_hdr_invariants(&hdr->invariants); - hdr->ttbr1_el1 = __pa_symbol(swapper_pg_dir); + hdr->ttbr1_el1 = swapper_pg_dir_node_phys(); hdr->reenter_kernel = _cpu_resume; /* We can't use __hyp_get_vectors() because kvm may still be loaded */ diff --git a/arch/arm64/mm/ktext.c b/arch/arm64/mm/ktext.c index 48d7943d6907..7b9a1f1b12a1 100644 --- a/arch/arm64/mm/ktext.c +++ b/arch/arm64/mm/ktext.c @@ -21,6 +21,26 @@ struct pgtables *pgtables[MAX_NUMNODES] = { static void *kernel_texts[MAX_NUMNODES]; +static pgd_t *__swapper_pg_dir_node(int nid) +{ + return pgtables[nid]->swapper_pg_dir; +} + +pgd_t *swapper_pg_dir_node(void) +{ + return __swapper_pg_dir_node(numa_node_id()); +} + +phys_addr_t __swapper_pg_dir_node_phys(int nid) +{ + return __pa(__swapper_pg_dir_node(nid)); +} + +phys_addr_t swapper_pg_dir_node_phys(void) +{ + return __swapper_pg_dir_node_phys(numa_node_id()); +} + void __kprobes ktext_replication_patch(u32 *tp, __le32 insn) { unsigned long offset; From patchwork Thu Jul 27 15:30:46 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Russell King (Oracle)" X-Patchwork-Id: 13330303 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 44CBCC001E0 for ; Thu, 27 Jul 2023 15:31:44 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Date:Message-Id:MIME-Version:Subject:Cc :To:From:References:In-Reply-To:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=8KaOdjV/rPvXidbaYjd9cMfwpmvuHlZGIHPO0u6OFpo=; b=dRY22WDaDrMXWv H8cGLU2RvV12Qk+Z09RIM+sGmF4l0kKTcL1k/MGyDht/l0ApLbGjV1BdWoNK09bo84xBcG38O7M4q +7qFWj+csYKQTR4HCCf8h67zgqzFNLABHmWt04IwWHySi5z4OciUiE8QY0V1g7voRki0i3M2XyKb2 ztBE6ZvnCum610x2CjyMu3Pym7jl6Mjl3EkDk9X/mS7fmKIqi9XlzMdu07jQLK8QY//B6I8LChSqS aNZ/azKi8Syd6y0n9t5SeECq4/5icedRvoW8UdSq96vD+w8ecXCAxUKnwHzfZl7H+MNMvCHYqT0Ib sAE9gsEMp19oznj24ZNw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qP2xV-00FvDW-09; Thu, 27 Jul 2023 15:31:21 +0000 Received: from [2001:4d48:ad52:32c8:5054:ff:fe00:142] (helo=pandora.armlinux.org.uk) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qP2wz-00FujW-0Y for linux-arm-kernel@lists.infradead.org; Thu, 27 Jul 2023 15:30:51 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=armlinux.org.uk; s=pandora-2019; h=Date:Sender:Message-Id:Content-Type: Content-Transfer-Encoding:MIME-Version:Subject:Cc:To:From:References: In-Reply-To:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help: List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=5/5o8z1eC9/ZLfhQ33mBBlzJO88MoASTulgcAEXGg/Q=; b=131mzP6vxZJ11CFOKHSuwSzHwH vGZrh9R4lkQJ5PJhTtnsErrNnKoS6WAtVnVeObVjnfjGRcWVJ1kWIHF9iDG0sl1XXQ63Us1L6v07r COXK1tHtr/P9IZi1po2sSO9ONF1TCASz6qCScIWSelQEfqVK96szZzjb5NdAzlsLyf/mUJsM+o28E 0LAQ9GHi6kvSuiGLnchF+lJDb/d3/AOkFggyDtbIjWObbGsqwrEdF5Ebj3Wn6SuSSf1xVJKF4bNET tIzQ9vTg+/JPpvLJUKVRqNy9X5cTFY4lcv2FHhh7PcQikjxReATb4Td1/plEeqoR+rj/kv9MQIoK8 CBjp2uOQ==; Received: from e0022681537dd.dyn.armlinux.org.uk ([fd8f:7570:feb6:1:222:68ff:fe15:37dd]:52420 helo=rmk-PC.armlinux.org.uk) by pandora.armlinux.org.uk with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.96) (envelope-from ) id 1qP2ww-00064C-1x; Thu, 27 Jul 2023 16:30:46 +0100 Received: from rmk by rmk-PC.armlinux.org.uk with local (Exim 4.94.2) (envelope-from ) id 1qP2ww-001CXq-O9; Thu, 27 Jul 2023 16:30:46 +0100 In-Reply-To: References: From: "Russell King (Oracle)" To: Catalin Marinas , Jonathan Corbet , Will Deacon Cc: linux-arm-kernel@lists.infradead.org, linux-doc@vger.kernel.org Subject: [PATCH v2 10/16] arm64: text replication: create per-node kernel page tables MIME-Version: 1.0 Content-Disposition: inline Message-Id: Date: Thu, 27 Jul 2023 16:30:46 +0100 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230727_083049_311228_659944F3 X-CRM114-Status: GOOD ( 21.25 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Allocate the level 0 page tables for the per-node kernel text replication, but copy all level 0 table entries from the NUMA node 0 table. Therefore, for the time being, each node's level 0 page tables will contain identical entries, and thus other nodes will continue to use the node 0 kernel text. Since the level 0 page tables can be updated at runtime to add entries for vmalloc and module space, propagate these updates to the other swapper page tables. The exception is if we see an update for the level 0 entry which points to the kernel mapping. We also need to setup a copy of the trampoline page tables as well, as the assembly code relies on the two page tables being a fixed offset apart. Signed-off-by: Russell King (Oracle) --- arch/arm64/include/asm/ktext.h | 12 ++++++++++ arch/arm64/mm/ktext.c | 42 +++++++++++++++++++++++++++++++++- arch/arm64/mm/mmu.c | 5 ++++ 3 files changed, 58 insertions(+), 1 deletion(-) diff --git a/arch/arm64/include/asm/ktext.h b/arch/arm64/include/asm/ktext.h index 289e11289c06..386f9812d3c1 100644 --- a/arch/arm64/include/asm/ktext.h +++ b/arch/arm64/include/asm/ktext.h @@ -7,11 +7,15 @@ #include +#include + #ifdef CONFIG_REPLICATE_KTEXT void ktext_replication_init(void); void __kprobes ktext_replication_patch(u32 *tp, __le32 insn); void ktext_replication_patch_alternative(__le32 *src, int nr_inst); +void ktext_replication_set_swapper_pgd(pgd_t *pgdp, pgd_t pgd); +void ktext_replication_init_tramp(void); #else @@ -27,6 +31,14 @@ static inline void ktext_replication_patch_alternative(__le32 *src, int nr_inst) { } +static inline void ktext_replication_set_swapper_pgd(pgd_t *pgdp, pgd_t pgd) +{ +} + +static inline void ktext_replication_init_tramp(void) +{ +} + #endif #endif diff --git a/arch/arm64/mm/ktext.c b/arch/arm64/mm/ktext.c index 7b9a1f1b12a1..9efd21eb3308 100644 --- a/arch/arm64/mm/ktext.c +++ b/arch/arm64/mm/ktext.c @@ -14,6 +14,7 @@ #include #include #include +#include struct pgtables *pgtables[MAX_NUMNODES] = { [0 ... MAX_NUMNODES - 1] = &pgtable_node0, @@ -97,7 +98,7 @@ void ktext_replication_patch_alternative(__le32 *src, int nr_inst) } } -/* Allocate memory for the replicated kernel texts. */ +/* Allocate page tables and memory for the replicated kernel texts. */ void __init ktext_replication_init(void) { size_t size = _etext - _stext; @@ -128,5 +129,44 @@ void __init ktext_replication_init(void) memcpy(kernel_texts[nid], _stext, size); caches_clean_inval_pou((u64)kernel_texts[nid], (u64)kernel_texts[nid] + size); + + /* Allocate the pagetables for this node */ + pgtables[nid] = memblock_alloc_node(sizeof(*pgtables[0]), + PGD_SIZE, nid); + + /* Copy initial swapper page directory */ + memcpy(pgtables[nid]->swapper_pg_dir, swapper_pg_dir, PGD_SIZE); + } +} + +void ktext_replication_set_swapper_pgd(pgd_t *pgdp, pgd_t pgd) +{ + unsigned long idx = pgdp - swapper_pg_dir; + int nid; + + if (WARN_ON_ONCE(idx >= PTRS_PER_PGD) || + WARN_ON_ONCE(idx == pgd_index((phys_addr_t)KERNEL_START))) + return; + + for_each_node(nid) { + if (pgtables[nid]->swapper_pg_dir == swapper_pg_dir) + continue; + + WRITE_ONCE(pgtables[nid]->swapper_pg_dir[idx], pgd); + } +} + +#ifdef CONFIG_UNMAP_KERNEL_AT_EL0 +void __init ktext_replication_init_tramp(void) +{ + int nid; + + for_each_node(nid) { + /* Nothing to do for node 0 */ + if (pgtables[nid]->tramp_pg_dir == tramp_pg_dir) + continue; + + memcpy(pgtables[nid]->tramp_pg_dir, tramp_pg_dir, PGD_SIZE); } } +#endif diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index 95d360805f8a..4ffa2650afd6 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -31,6 +31,7 @@ #include #include #include +#include #include #include #include @@ -81,6 +82,7 @@ void set_swapper_pgd(pgd_t *pgdp, pgd_t pgd) pgd_t *fixmap_pgdp; spin_lock(&swapper_pgdir_lock); + ktext_replication_set_swapper_pgd(pgdp, pgd); fixmap_pgdp = pgd_set_fixmap(__pa_symbol(pgdp)); WRITE_ONCE(*fixmap_pgdp, pgd); /* @@ -694,6 +696,9 @@ static int __init map_entry_trampoline(void) __set_fixmap(FIX_ENTRY_TRAMP_TEXT1 - i, pa_start + i * PAGE_SIZE, PAGE_KERNEL_RO); + /* Copy trampoline page tables to other numa nodes */ + ktext_replication_init_tramp(); + return 0; } core_initcall(map_entry_trampoline); From patchwork Thu Jul 27 15:30:51 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Russell King (Oracle)" X-Patchwork-Id: 13330302 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 05374C04A6A for ; Thu, 27 Jul 2023 15:31:43 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Date:Message-Id:MIME-Version:Subject:Cc :To:From:References:In-Reply-To:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=WELG4silK7ZK5ddWKsv2cGtgOMULInLN4DZjfCNWRfc=; b=Oubz4Zi/KGQTt1 LBtQ+LT82WLayepFX6gfrsoD0eJMpmLJZ3zi3Ca5GKlcJJjQ94AMhDLIK7yMdC36qFXsjq453+Rm4 hnFFUxxF3inZaTk9yqL3m52GbWFnTm5TTQULBj7Xx3Nzu+WPsFITEbFdpPmILSCXZf2HkhaJnz84C UIbAw9uNmiIWX/5VaQy+87H0pztF3GSN39PT0IU54a92J4ozzxRd/ezu5rwlkj7b18kIt3P0Hqmah OrubQeO7qCRyjEVMDv44TqJkJQWAx0gq7xXRhyCOvoL1aS90p9wIh/sDsEHaN8u0J9KMMCVOm+DQO BJiT7f1u2sVcHdN6ZpdQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qP2xW-00FvEd-0Z; Thu, 27 Jul 2023 15:31:22 +0000 Received: from [2001:4d48:ad52:32c8:5054:ff:fe00:142] (helo=pandora.armlinux.org.uk) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qP2x5-00FupT-1J for linux-arm-kernel@lists.infradead.org; Thu, 27 Jul 2023 15:30:57 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=armlinux.org.uk; s=pandora-2019; h=Date:Sender:Message-Id:Content-Type: Content-Transfer-Encoding:MIME-Version:Subject:Cc:To:From:References: In-Reply-To:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help: List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=LMAC+TyBqcZ8o2wxvL/7UyQ8+VQxNVXfpmzbUUbW3Sg=; b=BD//IFNEu5KwStdwtZYJooTbnI r32kfuCuWqa5iKteqx7dgQhaAAIR74sla3HXFOsZ89/KU2XmCzxveqKb+TTs9ra2td1TW3NP/ziH2 Sb4lk2SVGGZ3IP8NYIC1/P3oLik60Fw6wLn7o+cfTLW54YLAwDj5zy2DA4gmu88kNby8apgMuaAb9 ts9QL9KPwwicVcf72v9ITArJ2/biLxC4MKFsxgQzPD/3ZItFzkduxZkFWtRCkx51OLvwHeEz76viq rysnxUBLFU0PJEkGUtPVUT9qEwhLNGLyrzRalx8tY3lFBvb8/CHeE7XrO5UYlVIiuaCOK4EMDhYKq ACOUFMxg==; Received: from e0022681537dd.dyn.armlinux.org.uk ([fd8f:7570:feb6:1:222:68ff:fe15:37dd]:52432 helo=rmk-PC.armlinux.org.uk) by pandora.armlinux.org.uk with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.96) (envelope-from ) id 1qP2x1-00064T-30; Thu, 27 Jul 2023 16:30:51 +0100 Received: from rmk by rmk-PC.armlinux.org.uk with local (Exim 4.94.2) (envelope-from ) id 1qP2x1-001CXw-Sz; Thu, 27 Jul 2023 16:30:51 +0100 In-Reply-To: References: From: "Russell King (Oracle)" To: Catalin Marinas , Jonathan Corbet , Will Deacon Cc: linux-arm-kernel@lists.infradead.org, linux-doc@vger.kernel.org Subject: [PATCH v2 11/16] arm64: text replication: boot secondary CPUs with appropriate TTBR1 MIME-Version: 1.0 Content-Disposition: inline Message-Id: Date: Thu, 27 Jul 2023 16:30:51 +0100 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230727_083055_546186_0915D418 X-CRM114-Status: GOOD ( 12.00 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Arrange for secondary CPUs to boot with TTBR1 pointing at the appropriate per-node copy of the kernel page tables for the CPUs NUMA node. Signed-off-by: Russell King (Oracle) --- arch/arm64/include/asm/smp.h | 1 + arch/arm64/kernel/asm-offsets.c | 1 + arch/arm64/kernel/head.S | 3 ++- arch/arm64/kernel/smp.c | 3 +++ 4 files changed, 7 insertions(+), 1 deletion(-) diff --git a/arch/arm64/include/asm/smp.h b/arch/arm64/include/asm/smp.h index 9b31e6d0da17..9635e473969b 100644 --- a/arch/arm64/include/asm/smp.h +++ b/arch/arm64/include/asm/smp.h @@ -79,6 +79,7 @@ asmlinkage void secondary_start_kernel(void); struct secondary_data { struct task_struct *task; long status; + phys_addr_t ttbr1; }; extern struct secondary_data secondary_data; diff --git a/arch/arm64/kernel/asm-offsets.c b/arch/arm64/kernel/asm-offsets.c index 5ff1942b04fc..ce9d265bc099 100644 --- a/arch/arm64/kernel/asm-offsets.c +++ b/arch/arm64/kernel/asm-offsets.c @@ -121,6 +121,7 @@ int main(void) DEFINE(IRQ_CPUSTAT_SOFTIRQ_PENDING, offsetof(irq_cpustat_t, __softirq_pending)); BLANK(); DEFINE(CPU_BOOT_TASK, offsetof(struct secondary_data, task)); + DEFINE(CPU_BOOT_TTBR1, offsetof(struct secondary_data, ttbr1)); BLANK(); DEFINE(FTR_OVR_VAL_OFFSET, offsetof(struct arm64_ftr_override, val)); DEFINE(FTR_OVR_MASK_OFFSET, offsetof(struct arm64_ftr_override, mask)); diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S index 757a0de07f91..52643080a2c2 100644 --- a/arch/arm64/kernel/head.S +++ b/arch/arm64/kernel/head.S @@ -648,7 +648,8 @@ SYM_FUNC_START_LOCAL(secondary_startup) ldr_l x0, vabits_actual #endif bl __cpu_setup // initialise processor - adrp x1, swapper_pg_dir + adr_l x1, secondary_data + ldr x1, [x1, #CPU_BOOT_TTBR1] adrp x2, idmap_pg_dir bl __enable_mmu ldr x8, =__secondary_switched diff --git a/arch/arm64/kernel/smp.c b/arch/arm64/kernel/smp.c index edd63894d61e..95b649eb5999 100644 --- a/arch/arm64/kernel/smp.c +++ b/arch/arm64/kernel/smp.c @@ -119,6 +119,9 @@ int __cpu_up(unsigned int cpu, struct task_struct *idle) * page tables. */ secondary_data.task = idle; + secondary_data.ttbr1 = __swapper_pg_dir_node_phys(cpu_to_node(cpu)); + dcache_clean_poc((uintptr_t)&secondary_data, + (uintptr_t)&secondary_data + sizeof(secondary_data)); update_cpu_boot_status(CPU_MMU_OFF); /* Now bring the CPU into our world */ From patchwork Thu Jul 27 15:30:57 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Russell King (Oracle)" X-Patchwork-Id: 13330304 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id A89DAC001DC for ; Thu, 27 Jul 2023 15:31:47 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Date:Message-Id:MIME-Version:Subject:Cc :To:From:References:In-Reply-To:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=JLUJIgTrWyoIFUoYZ+6PsO6ixscjCjWt9MC0JN59CjY=; b=CAcNd28xw4t5wp J3RQCqDqLjaq1jmRssFeH4B4HN45Ba9mNwhbNWwEEvP3mOCrYUwA3K3heY8pnMgqLjr1+wj61ghMI j2HSaHCnGE6cWt1bsAsVl9CTQd1ArjIvrw/0kifWOKYRPOwRerD+O8bWIV2DIoFYIEKCEq9WM4fna Ygpwoo+XuX5QIayB7f/eg4pYp5nll3p1Ixz98uakPNa0WFH2mcWNc8F4YYotMcnv/dqWomt0idJSY hzISmxB4tNO9cuBeE1ijJD3wHZQ5+7myd7YZoyQmv3sg6dicNuSn0SdIIRKcORJVh3UMPYlflP+7n sYuPl3mzJhrU2QX1qlfg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qP2xX-00FvFV-0h; Thu, 27 Jul 2023 15:31:23 +0000 Received: from [2001:4d48:ad52:32c8:5054:ff:fe00:142] (helo=pandora.armlinux.org.uk) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qP2x8-00FutH-2y for linux-arm-kernel@lists.infradead.org; Thu, 27 Jul 2023 15:31:00 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=armlinux.org.uk; s=pandora-2019; h=Date:Sender:Message-Id:Content-Type: Content-Transfer-Encoding:MIME-Version:Subject:Cc:To:From:References: In-Reply-To:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help: List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=H7jN31HbcucbQvEkESj2M5pcxN2Y1iQH1UiJJb3dzkI=; b=C0nJUk4e1E7Cfk+YzAR1sG1omY dCKBODVPrGQAods11C+Fii9AY2SwTOmnOGxyltWbonSRLsLmPn9ZJEBlubymq3zCX5WyUou8htC5O JOwilloNSwKRWdIiBKR7GGNqEolh5n4Se/ccadxMgo39i424ymduoYi1CGVMVAcj4xRub6DmgSUTe 4DdUjayyVrhzUGBRcfr5XxG3dxKtMvKCmdQgcTI77qtGiaOv04fXpBMlUVQHPu6PiSx5OlQpHSDyg PUMEjnjBX0xzOB9nATBOe+XJ5m3/HQdyulbhRqoZ7ctVOxoU2fUW2r0dzZbHaxeoSDqLin59XMcH0 VElyRq/Q==; Received: from e0022681537dd.dyn.armlinux.org.uk ([fd8f:7570:feb6:1:222:68ff:fe15:37dd]:43914 helo=rmk-PC.armlinux.org.uk) by pandora.armlinux.org.uk with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.96) (envelope-from ) id 1qP2x6-00064p-2y; Thu, 27 Jul 2023 16:30:56 +0100 Received: from rmk by rmk-PC.armlinux.org.uk with local (Exim 4.94.2) (envelope-from ) id 1qP2x7-001CY3-16; Thu, 27 Jul 2023 16:30:57 +0100 In-Reply-To: References: From: "Russell King (Oracle)" To: Catalin Marinas , Jonathan Corbet , Will Deacon Cc: linux-arm-kernel@lists.infradead.org, linux-doc@vger.kernel.org Subject: [PATCH v2 12/16] arm64: text replication: update cnp support MIME-Version: 1.0 Content-Disposition: inline Message-Id: Date: Thu, 27 Jul 2023 16:30:57 +0100 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230727_083059_018330_F84CBA83 X-CRM114-Status: GOOD ( 13.50 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Add changes for CNP (Common Not Private) support of kernel text replication. Although text replication has only been tested on dual-socket Ampere A1 systems, provided the different NUMA nodes are not part of the same inner shareable domain, CNP should not be a problem. Signed-off-by: Russell King (Oracle) --- arch/arm64/include/asm/mmu_context.h | 2 +- arch/arm64/kernel/cpufeature.c | 2 +- arch/arm64/kernel/suspend.c | 3 ++- 3 files changed, 4 insertions(+), 3 deletions(-) diff --git a/arch/arm64/include/asm/mmu_context.h b/arch/arm64/include/asm/mmu_context.h index 42085b2b1d92..36f6b1b65ae5 100644 --- a/arch/arm64/include/asm/mmu_context.h +++ b/arch/arm64/include/asm/mmu_context.h @@ -163,7 +163,7 @@ static inline void cpu_replace_ttbr1_phys(phys_addr_t pgd_phys, pgd_t *idmap) phys_addr_t ttbr1 = phys_to_ttbr(pgd_phys); if (system_supports_cnp() && - !WARN_ON(pgd_phys != virt_to_phys(lm_alias(swapper_pg_dir)))) { + !WARN_ON(pgd_phys != swapper_pg_dir_node_phys())) { /* * cpu_replace_ttbr1() is used when there's a boot CPU * up (i.e. cpufeature framework is not up yet) and diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c index f9d456fe132d..e7dc7ec849ab 100644 --- a/arch/arm64/kernel/cpufeature.c +++ b/arch/arm64/kernel/cpufeature.c @@ -3417,7 +3417,7 @@ subsys_initcall_sync(init_32bit_el0_mask); static void __maybe_unused cpu_enable_cnp(struct arm64_cpu_capabilities const *cap) { - cpu_replace_ttbr1(lm_alias(swapper_pg_dir), idmap_pg_dir); + cpu_replace_ttbr1_phys(swapper_pg_dir_node_phys(), idmap_pg_dir); } /* diff --git a/arch/arm64/kernel/suspend.c b/arch/arm64/kernel/suspend.c index 0fbdf5fe64d8..49fa80bafd6d 100644 --- a/arch/arm64/kernel/suspend.c +++ b/arch/arm64/kernel/suspend.c @@ -55,7 +55,8 @@ void notrace __cpu_suspend_exit(void) /* Restore CnP bit in TTBR1_EL1 */ if (system_supports_cnp()) - cpu_replace_ttbr1(lm_alias(swapper_pg_dir), idmap_pg_dir); + cpu_replace_ttbr1_phys(swapper_pg_dir_node_phys(), + idmap_pg_dir); /* * PSTATE was not saved over suspend/resume, re-enable any detected From patchwork Thu Jul 27 15:31:02 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Russell King (Oracle)" X-Patchwork-Id: 13330307 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 05313C001DC for ; Thu, 27 Jul 2023 15:31:57 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Date:Message-Id:MIME-Version:Subject:Cc :To:From:References:In-Reply-To:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=YxFg0gLAmAVqFDVQs9KbEGQZcQzdk8pzWyJ+hQ9aSNU=; b=TS+i74ZE+c2qgA 1zQKO+4NfyEedH0dHVMxSnzr1Ohhdlk91wNNn+uRpRlNSnBHlz/nQLl7gK3P0ygewWzWxidavfdIx gEQ6yJDNjEU051lgKOgxyFQN8yduIZi1sM7XHiVC1S64wywKzVsnNuWz1CQPBtsrzOeB8Bzt0pDaE tzJNYogPvE9Qxx0TiVmmqdhn3RB3G4HuKjqM18Xn5Jszr903m7234zvs6WNUVSjw5mCvWSfCaBSkQ 3pmt0VYTFS8e2gTSHMAGZtPzXlcnhcD0WPppHYU4ebKnf1cxw6eJGXBeRgezSNHmJlEGFxsq1QRlv 2i+QFKew/Hf1L+2RKMjA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qP2xY-00FvGZ-07; Thu, 27 Jul 2023 15:31:24 +0000 Received: from [2001:4d48:ad52:32c8:5054:ff:fe00:142] (helo=pandora.armlinux.org.uk) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qP2xE-00Fuxe-2J for linux-arm-kernel@lists.infradead.org; Thu, 27 Jul 2023 15:31:06 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=armlinux.org.uk; s=pandora-2019; h=Date:Sender:Message-Id:Content-Type: Content-Transfer-Encoding:MIME-Version:Subject:Cc:To:From:References: In-Reply-To:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help: List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=81UKMmKbHgaP5z+oUfAHrIg9FFPRK2ZpDNlYVjo02BU=; b=ZIR+mSWXty2dSW7m4//S8LY+49 KWFbm22igOPOliabVO/i4Hu5QB6vrKNnawQzehsxcsgdneiHvCqQCzaXQB/GJn+B8sm9IEA3SC/SQ 5ya3ioSlbTV2NvfQ0g9zv2nG7dz12+fEqG/pC1E0RQAnl9yBDCyVepp6fdYQ0YLUnFxPVrhGOW9uo 72aK+IkKvWbp2nIZHg1bMuF3rT+XT7dK0F1WXvF4/evnHLV4fg0D60E1J7eYNYC1JnhHDlLBY0BBr fo/TNkN6wrjJLwfYsfle4LtdUtkFCw//kmrx3Q8a5cFPodB/az2gGz/Kbhjc7eoNmwjNyU49Zk8p1 mD1N6TbA==; Received: from e0022681537dd.dyn.armlinux.org.uk ([fd8f:7570:feb6:1:222:68ff:fe15:37dd]:43926 helo=rmk-PC.armlinux.org.uk) by pandora.armlinux.org.uk with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.96) (envelope-from ) id 1qP2xC-000650-0R; Thu, 27 Jul 2023 16:31:02 +0100 Received: from rmk by rmk-PC.armlinux.org.uk with local (Exim 4.94.2) (envelope-from ) id 1qP2xC-001CYA-7r; Thu, 27 Jul 2023 16:31:02 +0100 In-Reply-To: References: From: "Russell King (Oracle)" To: Catalin Marinas , Jonathan Corbet , Will Deacon Cc: linux-arm-kernel@lists.infradead.org, linux-doc@vger.kernel.org Subject: [PATCH v2 13/16] arm64: text replication: setup page tables for copied kernel MIME-Version: 1.0 Content-Disposition: inline Message-Id: Date: Thu, 27 Jul 2023 16:31:02 +0100 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230727_083104_800146_6B66DE25 X-CRM114-Status: GOOD ( 14.88 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Setup page table entries in each non-boot NUMA node page table to point at each node's own copy of the kernel text. This switches each node to use its own unique copy of the kernel text. Signed-off-by: Russell King (Oracle) --- arch/arm64/include/asm/ktext.h | 1 + arch/arm64/mm/ktext.c | 8 +++++ arch/arm64/mm/mmu.c | 53 ++++++++++++++++++++++++++++------ 3 files changed, 53 insertions(+), 9 deletions(-) diff --git a/arch/arm64/include/asm/ktext.h b/arch/arm64/include/asm/ktext.h index 386f9812d3c1..6ece59ca90a2 100644 --- a/arch/arm64/include/asm/ktext.h +++ b/arch/arm64/include/asm/ktext.h @@ -16,6 +16,7 @@ void __kprobes ktext_replication_patch(u32 *tp, __le32 insn); void ktext_replication_patch_alternative(__le32 *src, int nr_inst); void ktext_replication_set_swapper_pgd(pgd_t *pgdp, pgd_t pgd); void ktext_replication_init_tramp(void); +void create_kernel_nid_map(pgd_t *pgdp, void *ktext); #else diff --git a/arch/arm64/mm/ktext.c b/arch/arm64/mm/ktext.c index 9efd21eb3308..6692759e78a8 100644 --- a/arch/arm64/mm/ktext.c +++ b/arch/arm64/mm/ktext.c @@ -136,6 +136,14 @@ void __init ktext_replication_init(void) /* Copy initial swapper page directory */ memcpy(pgtables[nid]->swapper_pg_dir, swapper_pg_dir, PGD_SIZE); + + /* Clear the kernel mapping */ + memset(&pgtables[nid]->swapper_pg_dir[kidx], 0, + sizeof(pgtables[nid]->swapper_pg_dir[kidx])); + + /* Create kernel mapping pointing at our local copy */ + create_kernel_nid_map(pgtables[nid]->swapper_pg_dir, + kernel_texts[nid]); } } diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index 4ffa2650afd6..fb9c476605d1 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -640,6 +640,16 @@ void mark_rodata_ro(void) debug_checkwx(); } +static void __init create_kernel_mapping(pgd_t *pgdp, phys_addr_t pa_start, + void *va_start, void *va_end, + pgprot_t prot, int flags) +{ + size_t size = va_end - va_start; + + __create_pgd_mapping(pgdp, pa_start, (unsigned long)va_start, size, + prot, early_pgtable_alloc, flags); +} + static void __init map_kernel_segment(pgd_t *pgdp, void *va_start, void *va_end, pgprot_t prot, struct vm_struct *vma, int flags, unsigned long vm_flags) @@ -650,8 +660,7 @@ static void __init map_kernel_segment(pgd_t *pgdp, void *va_start, void *va_end, BUG_ON(!PAGE_ALIGNED(pa_start)); BUG_ON(!PAGE_ALIGNED(size)); - __create_pgd_mapping(pgdp, pa_start, (unsigned long)va_start, size, prot, - early_pgtable_alloc, flags); + create_kernel_mapping(pgdp, pa_start, va_start, va_end, prot, flags); if (!(vm_flags & VM_NO_GUARD)) size += PAGE_SIZE; @@ -720,14 +729,8 @@ static bool arm64_early_this_cpu_has_bti(void) ID_AA64PFR1_EL1_BT_SHIFT); } -/* - * Create fine-grained mappings for the kernel. - */ -static void __init map_kernel(pgd_t *pgdp) +static pgprot_t __init kernel_text_pgprot(void) { - static struct vm_struct vmlinux_text, vmlinux_rodata, vmlinux_inittext, - vmlinux_initdata, vmlinux_data; - /* * External debuggers may need to write directly to the text * mapping to install SW breakpoints. Allow this (only) when @@ -743,6 +746,38 @@ static void __init map_kernel(pgd_t *pgdp) if (arm64_early_this_cpu_has_bti()) text_prot = __pgprot_modify(text_prot, PTE_GP, PTE_GP); + return text_prot; +} + +#ifdef CONFIG_REPLICATE_KTEXT +void __init create_kernel_nid_map(pgd_t *pgdp, void *ktext) +{ + pgprot_t text_prot = kernel_text_pgprot(); + + create_kernel_mapping(pgdp, __pa(ktext), _stext, _etext, text_prot, 0); + create_kernel_mapping(pgdp, __pa_symbol(__start_rodata), + __start_rodata, __inittext_begin, + PAGE_KERNEL, NO_CONT_MAPPINGS); + create_kernel_mapping(pgdp, __pa_symbol(__inittext_begin), + __inittext_begin, __inittext_end, + text_prot, 0); + create_kernel_mapping(pgdp, __pa_symbol(__initdata_begin), + __initdata_begin, __initdata_end, + PAGE_KERNEL, 0); + create_kernel_mapping(pgdp, __pa_symbol(_data), _data, _end, + PAGE_KERNEL, 0); +} +#endif + +/* + * Create fine-grained mappings for the kernel. + */ +static void __init map_kernel(pgd_t *pgdp) +{ + static struct vm_struct vmlinux_text, vmlinux_rodata, vmlinux_inittext, + vmlinux_initdata, vmlinux_data; + pgprot_t text_prot = kernel_text_pgprot(); + /* * Only rodata will be remapped with different permissions later on, * all other segments are allowed to use contiguous mappings. From patchwork Thu Jul 27 15:31:07 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Russell King (Oracle)" X-Patchwork-Id: 13330305 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id A65DAC04A94 for ; Thu, 27 Jul 2023 15:31:50 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Date:Message-Id:MIME-Version:Subject:Cc :To:From:References:In-Reply-To:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=Xv+hg8b9+HbTmubZiKfavqzl5Efv0og707la6pQpr5E=; b=A4PHZ7BhJc02bP N+DhRhogRsXOS1mWHMKSV0qWpBhMegl3jaXHw3fal60NdCmm+9sjS+PwH57+bT081W1+AKM4z8+SU ZLkffSOMrI02188/vGBHQxqhNKrW8DO+3VWQKySE1BHkJl7E44oR0HrgglqBsLyerakmy+FUkxtlZ Fr5QEqXa4roLUzjZygVD9erj/VlplHRpUofqXgRMppK8woVFL+1kB+H4GgXfRcbs0nHQ2CZi5kf6t Vjf0tcprqxHkeFl6hT6OF7tN7CoQO1tRAecIla2BXSaXCqxdvigUfpIOzuk0JthJTgCS9HtZ8RWmm TDBX74+WxZJgQtZzvgpA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qP2xZ-00FvHv-0A; Thu, 27 Jul 2023 15:31:25 +0000 Received: from [2001:4d48:ad52:32c8:5054:ff:fe00:142] (helo=pandora.armlinux.org.uk) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qP2xK-00Fv2g-0F for linux-arm-kernel@lists.infradead.org; Thu, 27 Jul 2023 15:31:12 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=armlinux.org.uk; s=pandora-2019; h=Date:Sender:Message-Id:Content-Type: Content-Transfer-Encoding:MIME-Version:Subject:Cc:To:From:References: In-Reply-To:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help: List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=6VoLgh+3RZU4lx5aCym9VAv5S9Kh8dawR5c+i8YES6g=; b=zaDUc+O16qOuwjxhuiboWKNF5K /0qzALt2nfIpRoewqBV5uewUwXfGUVbUgfWFKV0PfIeLNUMY2EqbvsgUSKC6ahSedunFJqm2XH9m5 tsB9gWNj7wa0rSF1AP0E47AVdwZUzv4BIpiil1jlaee/NNXr476c3tJVYJhSZAWlM6f31DKTw+quT D7XdoLUKqB6xGPWoc4AejxR+zGTr717mjYv4OxbWqxMEohUCNqg3t6KLC64VaejHXTian/ZlNNTvu ovCCfN2kH7RSVSpuOFqo1EleIvqiJHxRwYK7dQop6MmKbfF/iG0aLgwYYNjnCyDtWygm71s37UBpx 6NZxkDkQ==; Received: from e0022681537dd.dyn.armlinux.org.uk ([fd8f:7570:feb6:1:222:68ff:fe15:37dd]:48392 helo=rmk-PC.armlinux.org.uk) by pandora.armlinux.org.uk with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.96) (envelope-from ) id 1qP2xH-00065A-1K; Thu, 27 Jul 2023 16:31:07 +0100 Received: from rmk by rmk-PC.armlinux.org.uk with local (Exim 4.94.2) (envelope-from ) id 1qP2xH-001CYG-Fb; Thu, 27 Jul 2023 16:31:07 +0100 In-Reply-To: References: From: "Russell King (Oracle)" To: Catalin Marinas , Jonathan Corbet , Will Deacon Cc: linux-arm-kernel@lists.infradead.org, linux-doc@vger.kernel.org Subject: [PATCH v2 14/16] arm64: text replication: include most of read-only data as well MIME-Version: 1.0 Content-Disposition: inline Message-Id: Date: Thu, 27 Jul 2023 16:31:07 +0100 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230727_083110_123384_FD1CF046 X-CRM114-Status: GOOD ( 15.58 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Include as much of the read-only data in the replication as we can without needing to move away from the generic RO_DATA() macro in the linker script. Unfortunately, the read-only data section is immedaitely followed by the read-only after init data with no page alignment, which means we can't have separate mappings for the read-only data section and everything else. Changing that would mean replacing the generic RO_DATA() macro which increases the maintenance burden. however, this is likely not worth the effort as the majority of read-only data will be covered. Signed-off-by: Russell King (Oracle) --- arch/arm64/mm/ktext.c | 2 +- arch/arm64/mm/mmu.c | 21 ++++++++++++++++++--- 2 files changed, 19 insertions(+), 4 deletions(-) diff --git a/arch/arm64/mm/ktext.c b/arch/arm64/mm/ktext.c index 6692759e78a8..6265a2db449b 100644 --- a/arch/arm64/mm/ktext.c +++ b/arch/arm64/mm/ktext.c @@ -101,7 +101,7 @@ void ktext_replication_patch_alternative(__le32 *src, int nr_inst) /* Allocate page tables and memory for the replicated kernel texts. */ void __init ktext_replication_init(void) { - size_t size = _etext - _stext; + size_t size = __end_rodata - _stext; int kidx = pgd_index((phys_addr_t)KERNEL_START); int nid; diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index fb9c476605d1..bf674bdaf336 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -752,11 +752,26 @@ static pgprot_t __init kernel_text_pgprot(void) #ifdef CONFIG_REPLICATE_KTEXT void __init create_kernel_nid_map(pgd_t *pgdp, void *ktext) { + phys_addr_t pa_ktext; + size_t ro_offset; + void *ro_end; pgprot_t text_prot = kernel_text_pgprot(); - create_kernel_mapping(pgdp, __pa(ktext), _stext, _etext, text_prot, 0); - create_kernel_mapping(pgdp, __pa_symbol(__start_rodata), - __start_rodata, __inittext_begin, + pa_ktext = __pa(ktext); + ro_offset = __pa_symbol(__start_rodata) - __pa_symbol(_stext); + /* + * We must not cover the read-only data after init, since this + * is written to during boot, and thus must be shared between + * the NUMA nodes. + */ + ro_end = PTR_ALIGN_DOWN((void *)__start_ro_after_init, PAGE_SIZE); + + create_kernel_mapping(pgdp, pa_ktext, _stext, _etext, text_prot, 0); + create_kernel_mapping(pgdp, pa_ktext + ro_offset, + __start_rodata, ro_end, + PAGE_KERNEL, NO_CONT_MAPPINGS); + create_kernel_mapping(pgdp, __pa_symbol(ro_end), + ro_end, __inittext_begin, PAGE_KERNEL, NO_CONT_MAPPINGS); create_kernel_mapping(pgdp, __pa_symbol(__inittext_begin), __inittext_begin, __inittext_end, From patchwork Thu Jul 27 15:31:12 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Russell King (Oracle)" X-Patchwork-Id: 13330306 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 77F69C001DC for ; Thu, 27 Jul 2023 15:31:53 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Date:Message-Id:MIME-Version:Subject:Cc :To:From:References:In-Reply-To:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=rjITNHlt53WAeZ3INDJ1Kc9upMSY32fUxhXRSTRx6+s=; b=ytkJd+lnsNvgC8 uNWBSgjsY7K8VhL9+U2HnObgqpuUQlCtRAs5kwLjjtHqOTuhaxLwZ9KpErWTje7dJteikcaaKgj80 /PFJVZeiyQbUvwO8XdLDioCvXe8vCFy2TxUUoxdnt9sWR/9vGFdLE39+Vpxq73QcLabBlxlOifsTZ PT0huKl4nuujmNM43QbVRm/w50fkvKXgIRX3TiBdfS5WRU/aXNHf++kvBdceml/2pHEHnNfbtzrhO Je5I80/Z1OZgYPgK9o3egT8FIJ5gkM/IeVaee/SzpJkoZIo6KuHIOd4G2a4RyB/cVZqURZ39oxwgg a4HY5wi/hRkh61/2o3MQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qP2xa-00FvJT-1n; Thu, 27 Jul 2023 15:31:26 +0000 Received: from [2001:4d48:ad52:32c8:5054:ff:fe00:142] (helo=pandora.armlinux.org.uk) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qP2xQ-00Fv85-0F for linux-arm-kernel@lists.infradead.org; Thu, 27 Jul 2023 15:31:18 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=armlinux.org.uk; s=pandora-2019; h=Date:Sender:Message-Id:Content-Type: Content-Transfer-Encoding:MIME-Version:Subject:Cc:To:From:References: In-Reply-To:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help: List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=w3WzY4I9PxfqVxgI2q1t7Icd6Vj1ik9I7ne5m3k+cAI=; b=hK09aIoxoOrzdtLHG39/OOSptn Lc0WX0s+CRjPQnAcGAQ8hOgN0Z2Akvhtxrh9WHA9TmjcSMZV6w+Mi6YgBNYWTiPRLwAg02juWgbGY 1EMVKvzKZbzHQQg/qnTua5LOC054uv3mdENvkbPn8LUtW7biwQIukrSbeNq393OlF+IMI3i1FvYWq jUX25jPuSXc2e2WAqho0x518WbphFq8QDcxDXP2F4mkfgjGRQ3UWBlX50SdZ0T8vzIJ0pQ4jYt1v8 9WA4Fbs2EicKuMPAKURj6Ta5kD+nv+yPRRY6bzkDPhFduNuedS5MwYqDqavoreUaALV90eqmFqPFA gIhqSc1A==; Received: from e0022681537dd.dyn.armlinux.org.uk ([fd8f:7570:feb6:1:222:68ff:fe15:37dd]:48400 helo=rmk-PC.armlinux.org.uk) by pandora.armlinux.org.uk with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.96) (envelope-from ) id 1qP2xM-00065K-21; Thu, 27 Jul 2023 16:31:13 +0100 Received: from rmk by rmk-PC.armlinux.org.uk with local (Exim 4.94.2) (envelope-from ) id 1qP2xM-001CYM-Mq; Thu, 27 Jul 2023 16:31:12 +0100 In-Reply-To: References: From: "Russell King (Oracle)" To: Catalin Marinas , Jonathan Corbet , Will Deacon Cc: linux-arm-kernel@lists.infradead.org, linux-doc@vger.kernel.org Subject: [PATCH v2 15/16] arm64: text replication: early kernel option to enable replication MIME-Version: 1.0 Content-Disposition: inline Message-Id: Date: Thu, 27 Jul 2023 16:31:12 +0100 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230727_083116_254835_2EFD8AF5 X-CRM114-Status: GOOD ( 14.26 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Provide an early kernel option "ktext=" which allows the kernel text replication to be enabled. This takes a boolean argument. The way this has been implemented means that we take all the same paths through the kernel at runtime whether kernel text replication has been enabled or not; this allows the performance effects of the code changes to be evaluated separately from the act of running with replicating the kernel text. Signed-off-by: Russell King (Oracle) --- .../admin-guide/kernel-parameters.txt | 5 +++++ arch/arm64/mm/ktext.c | 18 ++++++++++++++++++ 2 files changed, 23 insertions(+) diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt index a1457995fd41..ceb351a6ff2a 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -2494,6 +2494,11 @@ 0: force disabled 1: force enabled + ktext= [ARM64] Control kernel text replication on NUMA + machines. Default: disabled. + 0: disable kernel text replication + 1: enable kernel text replication + kunit.enable= [KUNIT] Enable executing KUnit tests. Requires CONFIG_KUNIT to be set to be fully enabled. The default value can be overridden via diff --git a/arch/arm64/mm/ktext.c b/arch/arm64/mm/ktext.c index 6265a2db449b..f5580ae14ca4 100644 --- a/arch/arm64/mm/ktext.c +++ b/arch/arm64/mm/ktext.c @@ -98,6 +98,21 @@ void ktext_replication_patch_alternative(__le32 *src, int nr_inst) } } +static bool ktext_enabled; + +static int __init parse_ktext(char *str) +{ + bool enabled; + int ret = strtobool(str, &enabled); + + if (ret) + return ret; + + ktext_enabled = enabled; + return 0; +} +early_param("ktext", parse_ktext); + /* Allocate page tables and memory for the replicated kernel texts. */ void __init ktext_replication_init(void) { @@ -119,6 +134,9 @@ void __init ktext_replication_init(void) return; } + if (!ktext_enabled) + return; + for_each_node(nid) { /* Nothing to do for node 0 */ if (!nid) From patchwork Thu Jul 27 15:31:17 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Russell King (Oracle)" X-Patchwork-Id: 13330310 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 37707C001DC for ; Thu, 27 Jul 2023 15:32:52 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Date:Message-Id:MIME-Version:Subject:Cc :To:From:References:In-Reply-To:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=ijdLL4Y8qfgTuiEVA6R9OFwwRQO0/g+xEMYv7U8vqM8=; b=3xyWU4DtTk0b1K BgGUdsCXg21Lr6Kgg2rP49zf2gaL5sr0CI9PNqXjqZ3QmDlK2ohfdtcPRIgsAjiBLlVwR+TX/QRAp nZgJekYvXOBuLcnUZBMi9Zb1GVS8nCXVxx0cIVew4MeNe9uo5HSPIworMUI1budoPIi4veiT+f/3O +RD9qeX/s285g8pMunQuMBH7NYCtJQuQmsRhHyAzPxrdPWH2lrDfBwhku8+7QWB9cIr3OB0zYAl7u LrlVIik2NRSaFjAR/ohHwV2wYDqTzHSvLS2fPhXx0+rLU3riXSQPklGwNS55okzoVfaQIhYpwHojV Zmf1AdO2c41a/1BaRUnA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qP2yZ-00FwAc-1w; Thu, 27 Jul 2023 15:32:28 +0000 Received: from [2001:4d48:ad52:32c8:5054:ff:fe00:142] (helo=pandora.armlinux.org.uk) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qP2xT-00FvBS-1o for linux-arm-kernel@lists.infradead.org; Thu, 27 Jul 2023 15:31:21 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=armlinux.org.uk; s=pandora-2019; h=Date:Sender:Message-Id:Content-Type: Content-Transfer-Encoding:MIME-Version:Subject:Cc:To:From:References: In-Reply-To:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help: List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=O8C+AYuO0MXUP+rSaG3vZziyS5K9WEK8DFQRI0HoQTM=; b=QpNqHg33/nSRYUppQOova0LMuW y5Y/SAexjVbIBofZdKr9YlJVhmIfWHc9qFk2ymCGCtCuuPSDAPb98qON9aLA4LDBXiEmoEjc8lH2G jU8UWvrHzVjpkF93e8h127BcTIu2wR8C6duoyN/2uh9hZFQQiDZeyMhBqx89R8qzYrBK7UoOW0aLa MFoDfEdqsxqjShNm/9F5gDQIW+2n1M1attR0h5h5puiGaFA0ngx0pB8gpuE7nmftFdmfGIZtEvSK7 gIQR0E82idk55OaCnlAVqUjsgcqewidj/hVjVCpHQZo+8L7pwC+b9Z3D5GnPi9RBJ6cPFO+lE8I+z FzA6uI1A==; Received: from e0022681537dd.dyn.armlinux.org.uk ([fd8f:7570:feb6:1:222:68ff:fe15:37dd]:40264 helo=rmk-PC.armlinux.org.uk) by pandora.armlinux.org.uk with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.96) (envelope-from ) id 1qP2xR-00065U-2R; Thu, 27 Jul 2023 16:31:17 +0100 Received: from rmk by rmk-PC.armlinux.org.uk with local (Exim 4.94.2) (envelope-from ) id 1qP2xR-001CYS-Ti; Thu, 27 Jul 2023 16:31:17 +0100 In-Reply-To: References: From: "Russell King (Oracle)" To: Catalin Marinas , Jonathan Corbet , Will Deacon Cc: linux-arm-kernel@lists.infradead.org, linux-doc@vger.kernel.org Subject: [PATCH v2 16/16] arm64: text replication: add Kconfig MIME-Version: 1.0 Content-Disposition: inline Message-Id: Date: Thu, 27 Jul 2023 16:31:17 +0100 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230727_083119_619477_F70BCC5B X-CRM114-Status: GOOD ( 12.78 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Add the Kconfig symbol for kernel text replication. This unfortunately requires KASAN and kernel text randomisation options to be disabled at the moment. Signed-off-by: Russell King (Oracle) --- arch/arm64/Kconfig | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 7856c3a3e35a..53143324c392 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -160,7 +160,7 @@ config ARM64 select HAVE_ARCH_HUGE_VMAP select HAVE_ARCH_JUMP_LABEL select HAVE_ARCH_JUMP_LABEL_RELATIVE - select HAVE_ARCH_KASAN if !(ARM64_16K_PAGES && ARM64_VA_BITS_48) + select HAVE_ARCH_KASAN if !(ARM64_16K_PAGES && ARM64_VA_BITS_48 && !REPLICATE_KTEXT) select HAVE_ARCH_KASAN_VMALLOC if HAVE_ARCH_KASAN select HAVE_ARCH_KASAN_SW_TAGS if HAVE_ARCH_KASAN select HAVE_ARCH_KASAN_HW_TAGS if (HAVE_ARCH_KASAN && ARM64_MTE) @@ -1426,6 +1426,13 @@ config NODES_SHIFT Specify the maximum number of NUMA Nodes available on the target system. Increases memory reserved to accommodate various tables. +config REPLICATE_KTEXT + bool "Replicate kernel text across numa nodes" + depends on NUMA + help + Say Y here to enable replicating the kernel text across multiple + nodes in a NUMA cluster. This trades memory for speed. + source "kernel/Kconfig.hz" config ARCH_SPARSEMEM_ENABLE @@ -2173,6 +2180,7 @@ config RELOCATABLE config RANDOMIZE_BASE bool "Randomize the address of the kernel image" + depends on !REPLICATE_KTEXT select RELOCATABLE help Randomizes the virtual address at which the kernel image is