From patchwork Tue May 30 14:04:35 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Russell King (Oracle)" X-Patchwork-Id: 13260141 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 9410DC77B7A for ; Tue, 30 May 2023 14:05:10 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Date:Message-Id:MIME-Version:Subject:Cc :To:From:References:In-Reply-To:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=elBeR1GYpNk2khJcXOY0hYALvYs5zePP1+W4P7KMPa8=; b=q5DMYVIWyqiX2N 1U6WEuysFgtUXK3lRUULl5X5+Rhp+if+8CuHAzK3tMFdMZbPdObQRshRtUc+rpxM4eJbSN1wmN5gS qOJK+R3rsq/nTeXlNetvK5z1sH/WD/xHQxPnuzBXKlKe7vEwckzfafJUfi/55YSasS0s672phEloo BnVNWckPkU78B4HW/j9mcSuWtaGk+NvXAS0OCWnzhDwRLblgVk2LnugdbAgTSnMzB443YTKpDC5ay 5zh5MJt7RVBfKG9N8qtyUNWVdM46oOj21QrC5a2cfGgEpjTsBkvmHnLG9CphuAtut6WPl2Wuzl/oW Q1e5k+U/6JWLoJT6soQg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1q3zxx-00E99d-16; Tue, 30 May 2023 14:04:49 +0000 Received: from pandora.armlinux.org.uk ([2001:4d48:ad52:32c8:5054:ff:fe00:142]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1q3zxm-00E94o-37 for linux-arm-kernel@lists.infradead.org; Tue, 30 May 2023 14:04:40 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=armlinux.org.uk; s=pandora-2019; h=Date:Sender:Message-Id:Content-Type: Content-Transfer-Encoding:MIME-Version:Subject:Cc:To:From:References: In-Reply-To:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help: List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=ec+Kz+6HYn6Bv7Pf4i2WLVnJVgRzIVT2Uy7ccItK/SY=; b=ZGtLVJfQg8epRb1ukMdOtU6sOt Ec1I6qZt361VTaYDUdtYIZZR8LqqgBPTvnDkJgn8rjt+pDVjsiOOKSKPQmeD/sWQVR7GFGFdMQcJh wKhXPPS/6DUJbpwb7RCdGoO7H+yXviDLC7cf00WhxmpVIjTdiPVPd7FSYJ+Q5mlKCRJ67gxUR/0IB GlAJePHloFzkDvYbSNAEYK+DoQId7oXsTWvNQxSyqG9M9v+1IlDw/2fKy+rzQt9lLWrppuXJ8IWqB B2XmtwiNHExwW3kJY6aLMOzgMIWGBUFssnY+DWZuQl+Se0iroezOWCJkwDLanjQfk180SZ5DAlzXs wz78a34Q==; Received: from e0022681537dd.dyn.armlinux.org.uk ([fd8f:7570:feb6:1:222:68ff:fe15:37dd]:48830 helo=rmk-PC.armlinux.org.uk) by pandora.armlinux.org.uk with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1q3zxj-0002pm-QH; Tue, 30 May 2023 15:04:35 +0100 Received: from rmk by rmk-PC.armlinux.org.uk with local (Exim 4.94.2) (envelope-from ) id 1q3zxj-00A5Jm-5k; Tue, 30 May 2023 15:04:35 +0100 In-Reply-To: References: From: "Russell King (Oracle)" To: Catalin Marinas , Jonathan Corbet , Will Deacon Cc: linux-arm-kernel@lists.infradead.org, linux-doc@vger.kernel.org Subject: [PATCH RFC 01/17] arm64: consolidate rox page protection logic MIME-Version: 1.0 Content-Disposition: inline Message-Id: Date: Tue, 30 May 2023 15:04:35 +0100 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230530_070439_003298_8AFDAE70 X-CRM114-Status: GOOD ( 12.08 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Consolidate the arm64 decision making for the page protections used for executable pages, used by both the trampoline code and the kernel text mapping code. Signed-off-by: Russell King (Oracle) --- arch/arm64/mm/mmu.c | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index af6bc8403ee4..4829abe017e9 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -663,12 +663,17 @@ static void __init map_kernel_segment(pgd_t *pgdp, void *va_start, void *va_end, vm_area_add_early(vma); } +static pgprot_t kernel_exec_prot(void) +{ + return rodata_enabled ? PAGE_KERNEL_ROX : PAGE_KERNEL_EXEC; +} + #ifdef CONFIG_UNMAP_KERNEL_AT_EL0 static int __init map_entry_trampoline(void) { int i; - pgprot_t prot = rodata_enabled ? PAGE_KERNEL_ROX : PAGE_KERNEL_EXEC; + pgprot_t prot = kernel_exec_prot(); phys_addr_t pa_start = __pa_symbol(__entry_tramp_text_start); /* The trampoline is always mapped and can therefore be global */ @@ -723,7 +728,7 @@ static void __init map_kernel(pgd_t *pgdp) * mapping to install SW breakpoints. Allow this (only) when * explicitly requested with rodata=off. */ - pgprot_t text_prot = rodata_enabled ? PAGE_KERNEL_ROX : PAGE_KERNEL_EXEC; + pgprot_t text_prot = kernel_exec_prot(); /* * If we have a CPU that supports BTI and a kernel built for From patchwork Tue May 30 14:04:40 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Russell King (Oracle)" X-Patchwork-Id: 13260143 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id A3AEFC7EE23 for ; Tue, 30 May 2023 14:05:14 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Date:Message-Id:MIME-Version:Subject:Cc :To:From:References:In-Reply-To:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=TllqyCjpS1ZmtehQDPCJkWg0vbfSprIx+6vtGwHjIVw=; b=ZWtWOJJWQWC2Fj 6a77nd6i+J+dBJDdVU4nGSces/fKiPyPO4EBLl1LjbaXy1Qa6onOnZHReR6oRikO4fLTHfBH9YwNi 6iCAAnOXf2wrIKkUPur2FQ28QgDVgbSqE/MNTD6LyBI2YNoNTkAGJ8Z83kx1qh4qgr0Y0H5XfiZQn Kq0U+wihxjYxGqsuNOWR0S8Y6RKzfh38+fSMvpOVOQzFfe3fp0h68Zuh+xIc+3TZZMzIdQzeX0wh8 ORQpgJZk3n1qdLg9VhteOO7S0oFDrd1YWw+gnyoW9WGdycV3CgZJtcSsf51AOnaJ0WDBojgRjJ0eR f2m5sgP+hIS0S0iN4dQg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1q3zxx-00E99t-2l; Tue, 30 May 2023 14:04:49 +0000 Received: from pandora.armlinux.org.uk ([2001:4d48:ad52:32c8:5054:ff:fe00:142]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1q3zxq-00E96x-0h for linux-arm-kernel@lists.infradead.org; Tue, 30 May 2023 14:04:43 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=armlinux.org.uk; s=pandora-2019; h=Date:Sender:Message-Id:Content-Type: Content-Transfer-Encoding:MIME-Version:Subject:Cc:To:From:References: In-Reply-To:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help: List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=ViMJ9zzDgjHP4eU7T/dKOV8UztyaowPjfrn2rFyO8M0=; b=mVvgb3RckzKwUTqA1nfK2mRhje z0ghxwtw9PP3bVenM+VRFaM531Ter4eLNHI6HVD2lEgHnW7b5NqO9l3Bc3y1YLT7Q2SufT1zOufJg 29qIUJyU4ddNhzGuiMhjzZmOld8RhGynhIG1TDjzNc5WPZ2TDnohuuyGjxVQR/Ca1IO5hLrre5PoT c4X8VR8G4MbiCRGCOWwAtku/kD8FEdmnslnV+WqAXeiKW5Rbxj42SZlB0m12wu5brKVUwXQToI5Ce TEr8kPMNvGKT+eegCuJg9YDWxnyXpzkzv8oTAPNOAZk0ae/roSBRYJXGAsIpIiLxCQkr6XU2+QcMk vgF4nCjw==; Received: from e0022681537dd.dyn.armlinux.org.uk ([fd8f:7570:feb6:1:222:68ff:fe15:37dd]:48834 helo=rmk-PC.armlinux.org.uk) by pandora.armlinux.org.uk with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1q3zxo-0002py-RT; Tue, 30 May 2023 15:04:40 +0100 Received: from rmk by rmk-PC.armlinux.org.uk with local (Exim 4.94.2) (envelope-from ) id 1q3zxo-00A5Js-9E; Tue, 30 May 2023 15:04:40 +0100 In-Reply-To: References: From: "Russell King (Oracle)" To: Catalin Marinas , Jonathan Corbet , Will Deacon Cc: linux-arm-kernel@lists.infradead.org, linux-doc@vger.kernel.org Subject: [PATCH RFC 02/17] arm64: place kernel in its own L0 page table entry MIME-Version: 1.0 Content-Disposition: inline Message-Id: Date: Tue, 30 May 2023 15:04:40 +0100 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230530_070442_262477_D1C4E4A2 X-CRM114-Status: GOOD ( 19.39 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Kernel text replication needs to maintain separate per-node page tables for the kernel text. In order to do this without affecting other kernel memory mappings, placing the kernel such that it does not share a L0 page table entry with any other mapping is desirable. Prior to this commit, the layout without KASLR was: +----------+ | vmalloc | +----------+ | Kernel | +----------+ MODULES_END, VMALLOC_START, KIMAGE_VADDR = | Modules | MODULES_VADDR + MODULES_VSIZE +----------+ MODULES_VADDR = _PAGE_END(VA_BITS_MIN) | VA space | +----------+ 0 This becomes: +----------+ | vmalloc | +----------+ VMALLOC_START = MODULES_END + PGDIR_SIZE | Kernel | +----------+ MODULES_END, KIMAGE_VADDR = _PAGE_END(VA_BITS_MIN) + PGDIR_SIZE | Modules | +----------+ MODULES_VADDR = MODULES_END - MODULES_VSIZE | VA space | +----------+ 0 This assumes MODULES_VSIZE (128M) <= PGDIR_SIZE. One side effect of this change is that KIMAGE_VADDR's definition now includes PGDIR_SIZE (to leave room for the modules) but this is not defined when asm/memory.h is included. This means KIMAGE_VADDR can not be used in inline functions within this file, so we convert kaslr_offset() and kaslr_enabled() to be macros instead. Signed-off-by: Russell King (Oracle) --- arch/arm64/include/asm/memory.h | 26 ++++++++++---------------- arch/arm64/include/asm/pgtable.h | 2 +- arch/arm64/mm/mmu.c | 3 ++- 3 files changed, 13 insertions(+), 18 deletions(-) diff --git a/arch/arm64/include/asm/memory.h b/arch/arm64/include/asm/memory.h index c735afdf639b..089f556b7387 100644 --- a/arch/arm64/include/asm/memory.h +++ b/arch/arm64/include/asm/memory.h @@ -43,9 +43,9 @@ #define VA_BITS (CONFIG_ARM64_VA_BITS) #define _PAGE_OFFSET(va) (-(UL(1) << (va))) #define PAGE_OFFSET (_PAGE_OFFSET(VA_BITS)) -#define KIMAGE_VADDR (MODULES_END) -#define MODULES_END (MODULES_VADDR + MODULES_VSIZE) -#define MODULES_VADDR (_PAGE_END(VA_BITS_MIN)) +#define KIMAGE_VADDR (_PAGE_END(VA_BITS_MIN) + PGDIR_SIZE) +#define MODULES_END (KIMAGE_VADDR) +#define MODULES_VADDR (MODULES_END - MODULES_VSIZE) #define MODULES_VSIZE (SZ_128M) #define VMEMMAP_START (-(UL(1) << (VA_BITS - VMEMMAP_SHIFT))) #define VMEMMAP_END (VMEMMAP_START + VMEMMAP_SIZE) @@ -199,20 +199,14 @@ extern u64 kimage_vaddr; /* the offset between the kernel virtual and physical mappings */ extern u64 kimage_voffset; -static inline unsigned long kaslr_offset(void) -{ - return kimage_vaddr - KIMAGE_VADDR; -} +#define kaslr_offset() ((unsigned long)(kimage_vaddr - KIMAGE_VADDR)) -static inline bool kaslr_enabled(void) -{ - /* - * The KASLR offset modulo MIN_KIMG_ALIGN is taken from the physical - * placement of the image rather than from the seed, so a displacement - * of less than MIN_KIMG_ALIGN means that no seed was provided. - */ - return kaslr_offset() >= MIN_KIMG_ALIGN; -} +/* + * The KASLR offset modulo MIN_KIMG_ALIGN is taken from the physical + * placement of the image rather than from the seed, so a displacement + * of less than MIN_KIMG_ALIGN means that no seed was provided. + */ +#define kaslr_enabled() (kaslr_offset() >= MIN_KIMG_ALIGN) /* * Allow all memory at the discovery stage. We will clip it later. diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 0bd18de9fd97..cb526e69299d 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -21,7 +21,7 @@ * VMALLOC_END: extends to the available space below vmemmap, PCI I/O space * and fixed mappings */ -#define VMALLOC_START (MODULES_END) +#define VMALLOC_START (MODULES_END + PGDIR_SIZE) #define VMALLOC_END (VMEMMAP_START - SZ_256M) #define vmemmap ((struct page *)VMEMMAP_START - (memstart_addr >> PAGE_SHIFT)) diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index 4829abe017e9..baf74d0c43c9 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -478,7 +478,8 @@ void __init create_pgd_mapping(struct mm_struct *mm, phys_addr_t phys, static void update_mapping_prot(phys_addr_t phys, unsigned long virt, phys_addr_t size, pgprot_t prot) { - if ((virt >= PAGE_END) && (virt < VMALLOC_START)) { + if ((virt >= PAGE_END) && (virt < VMALLOC_START) && + !is_kernel(virt)) { pr_warn("BUG: not updating mapping for %pa at 0x%016lx - outside kernel range\n", &phys, virt); return; From patchwork Tue May 30 14:04:45 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Russell King (Oracle)" X-Patchwork-Id: 13260142 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id B4586C77B73 for ; Tue, 30 May 2023 14:05:11 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Date:Message-Id:MIME-Version:Subject:Cc :To:From:References:In-Reply-To:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=JgGbZcGBgovdLrnPfoIK3Ko7ppkwNepGO3qiiD+Sf0g=; b=G8i+BafKV0cChR l+4tGXELfKfWGcS1ji2FP6dvH4Gi6OLrtEKFCiTZjbFeD9esOq1VJ0QdWeWklg/IIPiV+bTEfeZph iZNMV6mWiErJDZR5POgw3fjj7tC3J6GRpBzCqPnQEm6PqklsbjjSE+SG8swEsy/SxD92fEkhd6m1k Pcf0eFdvjdc7dyrlEZmspSX0/vsLU1r5cf8K0om9hKYnI469Pe6zmo2AXVyc6D2NQCpqZfkTHUmum E34xZjaXqmByphATy2+R0BS+krKjUinM6WcVaZQ2QW2L3HM/6gtYLngvjI+f0IduyxKdV5NQeEvvc suVrlDTYGM1PXUg48LnQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1q3zxy-00E9AS-1Y; Tue, 30 May 2023 14:04:50 +0000 Received: from pandora.armlinux.org.uk ([2001:4d48:ad52:32c8:5054:ff:fe00:142]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1q3zxv-00E98s-0y for linux-arm-kernel@lists.infradead.org; Tue, 30 May 2023 14:04:48 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=armlinux.org.uk; s=pandora-2019; h=Date:Sender:Message-Id:Content-Type: Content-Transfer-Encoding:MIME-Version:Subject:Cc:To:From:References: In-Reply-To:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help: List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=q8TUQt8YnxjlNGCvUsFaTTf2K7LNZ0RBLteg63TX/40=; b=fxnUJ+jpvl1w6yNYZrgBe7FvMd osL8PBkY7rkKKwaGUoU/V04vXCp+spzH2atkY9KzwuQ7uTCnTfpBDQtOykIa6jqrlTj2/u26f2zS0 DViXmiVmmIy2rKwK4aUSUyjbmVp9Ob+2lNJia1dgatsm7UCpqurivOUyaJ1KkM1mAl/COG11DcEnm djQlVZEarSwSjRWl4s1E2PbfvVgzOTeesu/ao8TJszIsBeB9samBvhwHfg2EDN6dZ+P3tcfAy2hBJ L7wr6S7edKoYHt6UNm2+aUAO6urVMUckTPP+Omn24HCsnuJ1C+zRdp8J1NKewU8+xcT+DD0D8Hubu puIGra+Q==; Received: from e0022681537dd.dyn.armlinux.org.uk ([fd8f:7570:feb6:1:222:68ff:fe15:37dd]:50816 helo=rmk-PC.armlinux.org.uk) by pandora.armlinux.org.uk with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1q3zxt-0002q8-V1; Tue, 30 May 2023 15:04:45 +0100 Received: from rmk by rmk-PC.armlinux.org.uk with local (Exim 4.94.2) (envelope-from ) id 1q3zxt-00A5Jz-CF; Tue, 30 May 2023 15:04:45 +0100 In-Reply-To: References: From: "Russell King (Oracle)" To: Catalin Marinas , Jonathan Corbet , Will Deacon Cc: linux-arm-kernel@lists.infradead.org, linux-doc@vger.kernel.org Subject: [PATCH RFC 03/17] arm64: provide cpu_replace_ttbr1_phys() MIME-Version: 1.0 Content-Disposition: inline Message-Id: Date: Tue, 30 May 2023 15:04:45 +0100 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230530_070447_335865_0CE96691 X-CRM114-Status: GOOD ( 13.53 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Provide a version of cpu_replace_ttbr1_phys() which operates using a physical address rather than the virtual address of the page tables. Signed-off-by: Russell King (Oracle) --- arch/arm64/include/asm/mmu_context.h | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/arch/arm64/include/asm/mmu_context.h b/arch/arm64/include/asm/mmu_context.h index 56911691bef0..4aa6afc6f935 100644 --- a/arch/arm64/include/asm/mmu_context.h +++ b/arch/arm64/include/asm/mmu_context.h @@ -148,7 +148,7 @@ static inline void cpu_install_ttbr0(phys_addr_t ttbr0, unsigned long t0sz) * Atomically replaces the active TTBR1_EL1 PGD with a new VA-compatible PGD, * avoiding the possibility of conflicting TLB entries being allocated. */ -static inline void cpu_replace_ttbr1(pgd_t *pgdp, pgd_t *idmap) +static inline void cpu_replace_ttbr1_phys(phys_addr_t pgd_phys, pgd_t *idmap) { typedef void (ttbr_replace_func)(phys_addr_t); extern ttbr_replace_func idmap_cpu_replace_ttbr1; @@ -156,9 +156,10 @@ static inline void cpu_replace_ttbr1(pgd_t *pgdp, pgd_t *idmap) unsigned long daif; /* phys_to_ttbr() zeros lower 2 bits of ttbr with 52-bit PA */ - phys_addr_t ttbr1 = phys_to_ttbr(virt_to_phys(pgdp)); + phys_addr_t ttbr1 = phys_to_ttbr(pgd_phys); - if (system_supports_cnp() && !WARN_ON(pgdp != lm_alias(swapper_pg_dir))) { + if (system_supports_cnp() && + !WARN_ON(pgd_phys != virt_to_phys(lm_alias(swapper_pg_dir)))) { /* * cpu_replace_ttbr1() is used when there's a boot CPU * up (i.e. cpufeature framework is not up yet) and @@ -185,6 +186,11 @@ static inline void cpu_replace_ttbr1(pgd_t *pgdp, pgd_t *idmap) cpu_uninstall_idmap(); } +static inline void __nocfi cpu_replace_ttbr1(pgd_t *pgdp, pgd_t *idmap) +{ + cpu_replace_ttbr1_phys(virt_to_phys(pgdp), idmap); +} + /* * It would be nice to return ASIDs back to the allocator, but unfortunately * that introduces a race with a generation rollover where we could erroneously From patchwork Tue May 30 14:04:50 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Russell King (Oracle)" X-Patchwork-Id: 13260145 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 23C0BC77B7A for ; Tue, 30 May 2023 14:05:39 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Date:Message-Id:MIME-Version:Subject:Cc :To:From:References:In-Reply-To:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=yU8H6Itvs9SFbbSD2yxx/QDh6OEnO/KXHImUri2/A2g=; b=2Hzs5CNxIPR2J6 pK2xGtzsKc30vlSVPnQ2AeB74GpfJDqzONrXyZ01I+hMg1gazsg9AKLtMlk2dc4qh77uqmRAlOOhJ jSAD0BqYgtA3XZXQnb8hzb1L7N+/QN8Ow29g6U1VP3vsvnpKgPHImopCqVx++BCtzf+vSXPPPipLm /VZUVre9kPFN3VR5VTikahAdkOzgP7h44YOF2n9EyA97NTyORbjUxq/SvJggOjZWTScWS11/XNqVm YfN437WbIVaqmOc4K9iWEh06miuVDUBZeSWc8gLLY9CTQ8694xgKLH8Fs8wZP1HH9BnooDAdV59+P DCjjkNHdN2lw90iacpwg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1q3zyM-00E9Jf-1D; Tue, 30 May 2023 14:05:14 +0000 Received: from pandora.armlinux.org.uk ([2001:4d48:ad52:32c8:5054:ff:fe00:142]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1q3zy0-00E9BG-1E for linux-arm-kernel@lists.infradead.org; Tue, 30 May 2023 14:04:53 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=armlinux.org.uk; s=pandora-2019; h=Date:Sender:Message-Id:Content-Type: Content-Transfer-Encoding:MIME-Version:Subject:Cc:To:From:References: In-Reply-To:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help: List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=1EnzjEtF1WYDn+TvglRvqTszcMqj93hsj/tUGVubobk=; b=iYWbHOiKblNGqcX2ob/VU4iK47 bXycuM9Y1G4L1mnuZjhe5FNdcZpUsdYtA9rdStSvIqyQYG7nTK/saBzT+dWRLLqHtsUmtvBHmG2ey OdIFzLqvY+gQ2o6eddRD1ZcYhmge+22JTmY81k+kN86TKMTUmCM8BSSRNsUMPUeBeru2ebcNq8z/M oQJOdGe/tbIJTeI4z4RmK2YhScMSJNEhIC7ePMzHANwqh/6lBE7AEExJoVuqVJTmQdymC/RF9K3+g wAJ9P0iEujyKbpcoVzTjkZUjj0gR878PCdeY2ry6uDCmrspTdFYy5TZvHnLS5FUDc1eNBMMhNtZUb jhMdxrLQ==; Received: from e0022681537dd.dyn.armlinux.org.uk ([fd8f:7570:feb6:1:222:68ff:fe15:37dd]:50828 helo=rmk-PC.armlinux.org.uk) by pandora.armlinux.org.uk with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1q3zxz-0002qI-22; Tue, 30 May 2023 15:04:51 +0100 Received: from rmk by rmk-PC.armlinux.org.uk with local (Exim 4.94.2) (envelope-from ) id 1q3zxy-00A5KE-Ft; Tue, 30 May 2023 15:04:50 +0100 In-Reply-To: References: From: "Russell King (Oracle)" To: Catalin Marinas , Jonathan Corbet , Will Deacon Cc: linux-arm-kernel@lists.infradead.org, linux-doc@vger.kernel.org Subject: [PATCH RFC 04/17] arm64: make clean_dcache_range_nopatch() visible MIME-Version: 1.0 Content-Disposition: inline Message-Id: Date: Tue, 30 May 2023 15:04:50 +0100 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230530_070452_421332_CB467B07 X-CRM114-Status: GOOD ( 12.75 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org When we hook into the kernel text patching code, we will need to call clean_dcache_range_nopatch() to ensure that the patching of the replicated kernel text is properly visible to other CPUs. Make this function available to the replication code. Signed-off-by: Russell King (Oracle) --- arch/arm64/include/asm/cacheflush.h | 2 ++ arch/arm64/kernel/alternative.c | 2 +- 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/arch/arm64/include/asm/cacheflush.h b/arch/arm64/include/asm/cacheflush.h index 37185e978aeb..ac9ad56d5212 100644 --- a/arch/arm64/include/asm/cacheflush.h +++ b/arch/arm64/include/asm/cacheflush.h @@ -104,6 +104,8 @@ static inline void flush_icache_range(unsigned long start, unsigned long end) } #define flush_icache_range flush_icache_range +void clean_dcache_range_nopatch(u64 start, u64 end); + /* * Copy user data from/to a page which is mapped into a different * processes address space. Really, we want to allow our "user diff --git a/arch/arm64/kernel/alternative.c b/arch/arm64/kernel/alternative.c index d32d4ed5519b..df9a73458a85 100644 --- a/arch/arm64/kernel/alternative.c +++ b/arch/arm64/kernel/alternative.c @@ -121,7 +121,7 @@ static noinstr void patch_alternative(struct alt_instr *alt, * accidentally call into the cache.S code, which is patched by us at * runtime. */ -static void clean_dcache_range_nopatch(u64 start, u64 end) +void clean_dcache_range_nopatch(u64 start, u64 end) { u64 cur, d_size, ctr_el0; From patchwork Tue May 30 14:04:55 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Russell King (Oracle)" X-Patchwork-Id: 13260147 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 2CBA2C7EE23 for ; Tue, 30 May 2023 14:05:41 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Date:Message-Id:MIME-Version:Subject:Cc :To:From:References:In-Reply-To:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=+JDMeEFedXBIT5Yent2pr/XrPDEEN+os2DDRnn2Mvp0=; b=D+3b7dKZzOogWY frHO8/MIuZF3X+9njcdQmEuj9mV59r0GMsrfsO4cZFh52gv0GHGHsTWhDOXJJjZAdsfRIT4eZjxEK AOx5J0HDjMlUefC8KHPlEA9PvuAT3D43+jUuGz3eIQGZ5g9JcR0yDm21liV+I1dbiO4JFLy6Qx87A 6pbBAdcE2FBQ5h4RE0l6IvYIYxtn/XhMnLfUZHE+66nknaUqHB1wF8dEgnclSEE/sFs9nI1AqeFNy /7GcjXG/K4qT7ncEUkw/FbStwWiWe5wdNjPpDELyFQZISygQUzzCzrD6Zr09G65v3JtffMSCIXRjq iwDSuqHMGsDoZGDE+kQA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1q3zyM-00E9K5-2y; Tue, 30 May 2023 14:05:14 +0000 Received: from pandora.armlinux.org.uk ([2001:4d48:ad52:32c8:5054:ff:fe00:142]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1q3zy6-00E9EE-0X for linux-arm-kernel@lists.infradead.org; Tue, 30 May 2023 14:04:59 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=armlinux.org.uk; s=pandora-2019; h=Date:Sender:Message-Id:Content-Type: Content-Transfer-Encoding:MIME-Version:Subject:Cc:To:From:References: In-Reply-To:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help: List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=tKmg1wvbWm3Sut3X0el13kLY6MXHU/o3mbUv4cgLoEE=; b=Sgcy602VpQu91gcslBmraZ86HK 6BOF7iIqExdi/Wawa7+JOiWK8dU5qxa2TR7QlXS37GD/XhpI3mr1ZmGD3ZwlcfDBG/lpTAails8dw hf3ofAk4zHH10zCyNNyXqN9EQfPEbjUYdtMeUZv+xEyVr4ZUfdR/uWmwGuW+LTHod/D4hCz/2sKcs XNRUx8mUvh8uxwkYLQh3uhnsM9Ug05Yd3KiZEMQF9aaHOE5frfGUbt09y05HnNzVCVpkfpb1//Gzc T9rsrC+qML/DNV6Cg2Y1z101LXYXLWziUTzWqUqzj+XP0xzWreXWFwX0+hGyhOA3c0duaEOb/p7F7 tyOQnnVA==; Received: from e0022681537dd.dyn.armlinux.org.uk ([fd8f:7570:feb6:1:222:68ff:fe15:37dd]:59738 helo=rmk-PC.armlinux.org.uk) by pandora.armlinux.org.uk with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1q3zy4-0002qS-B6; Tue, 30 May 2023 15:04:56 +0100 Received: from rmk by rmk-PC.armlinux.org.uk with local (Exim 4.94.2) (envelope-from ) id 1q3zy3-00A5KM-JA; Tue, 30 May 2023 15:04:55 +0100 In-Reply-To: References: From: "Russell King (Oracle)" To: Catalin Marinas , Jonathan Corbet , Will Deacon Cc: linux-arm-kernel@lists.infradead.org, linux-doc@vger.kernel.org Subject: [PATCH RFC 05/17] arm64: text replication: add init function MIME-Version: 1.0 Content-Disposition: inline Message-Id: Date: Tue, 30 May 2023 15:04:55 +0100 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230530_070458_198771_353BD1AE X-CRM114-Status: GOOD ( 15.40 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org A simple patch that adds an empty function for kernel text replication initialisation and hooks it into the initialisation path. Signed-off-by: Russell King (Oracle) --- arch/arm64/include/asm/ktext.h | 20 ++++++++++++++++++++ arch/arm64/mm/Makefile | 2 ++ arch/arm64/mm/init.c | 3 +++ arch/arm64/mm/ktext.c | 8 ++++++++ 4 files changed, 33 insertions(+) create mode 100644 arch/arm64/include/asm/ktext.h create mode 100644 arch/arm64/mm/ktext.c diff --git a/arch/arm64/include/asm/ktext.h b/arch/arm64/include/asm/ktext.h new file mode 100644 index 000000000000..1a5f7452a3bf --- /dev/null +++ b/arch/arm64/include/asm/ktext.h @@ -0,0 +1,20 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* + * Copyright (C) 2022, Oracle and/or its affiliates. + */ +#ifndef ASM_KTEXT_H +#define ASM_KTEXT_H + +#ifdef CONFIG_REPLICATE_KTEXT + +void ktext_replication_init(void); + +#else + +static inline void ktext_replication_init(void) +{ +} + +#endif + +#endif diff --git a/arch/arm64/mm/Makefile b/arch/arm64/mm/Makefile index dbd1bc95967d..41e705027c57 100644 --- a/arch/arm64/mm/Makefile +++ b/arch/arm64/mm/Makefile @@ -14,3 +14,5 @@ KASAN_SANITIZE_physaddr.o += n obj-$(CONFIG_KASAN) += kasan_init.o KASAN_SANITIZE_kasan_init.o := n + +obj-$(CONFIG_REPLICATE_KTEXT) += ktext.o diff --git a/arch/arm64/mm/init.c b/arch/arm64/mm/init.c index 66e70ca47680..a0e4f2d93ee6 100644 --- a/arch/arm64/mm/init.c +++ b/arch/arm64/mm/init.c @@ -36,6 +36,7 @@ #include #include #include +#include #include #include #include @@ -401,6 +402,8 @@ void __init bootmem_init(void) arch_numa_init(); + ktext_replication_init(); + /* * must be done after arch_numa_init() which calls numa_init() to * initialize node_online_map that gets used in hugetlb_cma_reserve() diff --git a/arch/arm64/mm/ktext.c b/arch/arm64/mm/ktext.c new file mode 100644 index 000000000000..3a8d37c9abc4 --- /dev/null +++ b/arch/arm64/mm/ktext.c @@ -0,0 +1,8 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright (C) 2022, Oracle and/or its affiliates. + */ + +void __init ktext_replication_init(void) +{ +} From patchwork Tue May 30 14:05:00 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Russell King (Oracle)" X-Patchwork-Id: 13260144 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 8C978C77B7A for ; Tue, 30 May 2023 14:05:36 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Date:Message-Id:MIME-Version:Subject:Cc :To:From:References:In-Reply-To:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=AU1Sd5wzrFWxr+3Zv+LuOVQROZYurxd8H3YEK+p9/Zo=; b=lT8z0TqJWBHWta f8IdSW8/XMG2PLsecnywc5UJ/W/+MQLztQIFeIwuhbCvrJDVoqnbUeYjGRIKkD8FeqNGpNq425KR1 N61E8IyYdQjetRtspdF72Ooq/amYIu0NzQjlMjW3otyOQTEsOspOIYR4DS7McYLq5bi1YR4LabwOP b+5iCI7WVB00Ts0NW+OQVaYYg7mU+HdRNKxrUCEONMmZld4Z08xVHZuewgk3gPiCuVL6sc0Jo3XKp gTSzMHFkj/D1s3rt5h2qcSQGfzrXnOrXyxKptJXB1UGo9tnuuvrV1uG/BgKVj1eKET9GRFZCDI4Xh UfepOBlPJOULFKDN6gaQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1q3zyN-00E9Kd-1T; Tue, 30 May 2023 14:05:15 +0000 Received: from pandora.armlinux.org.uk ([2001:4d48:ad52:32c8:5054:ff:fe00:142]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1q3zyA-00E9Fp-2f for linux-arm-kernel@lists.infradead.org; Tue, 30 May 2023 14:05:03 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=armlinux.org.uk; s=pandora-2019; h=Date:Sender:Message-Id:Content-Type: Content-Transfer-Encoding:MIME-Version:Subject:Cc:To:From:References: In-Reply-To:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help: List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=nyj2l8PY3qAzzBlei2fGIlbkY3GLY8pUVEJtHA0LwRE=; b=W7nmtbT408cEZTFXKzvM8r4hhL u0ofMJxHijW1UPdwrPW2t9bCDCfpownf0sr8OcasGxJdNzSELGxkMbB+UzjHr7rwnxdPViDUgsK/H Dh+WI8BEHEzhlX0+3EFo33Aj2AZCJlblF9m6NYDHAO7N/FEaNCYnTUMaoasejEk1krgqqB/uCGaKJ /azzEDxtBoK6UVfjBXa1433K4ISkBy/A2xJN/BCkFCWI1BQtCxOQAPbqatZweAmWRFZL3BCK9DipL aCp+jngwUFmj8FCPywcAV/ieTo20SJCbJI1bt0RLf0Ntvgxu1Xl3P3SLtqU/FX2E7LZTZfi6wCEhE Eom/9Ppw==; Received: from e0022681537dd.dyn.armlinux.org.uk ([fd8f:7570:feb6:1:222:68ff:fe15:37dd]:59742 helo=rmk-PC.armlinux.org.uk) by pandora.armlinux.org.uk with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1q3zy9-0002qw-GV; Tue, 30 May 2023 15:05:01 +0100 Received: from rmk by rmk-PC.armlinux.org.uk with local (Exim 4.94.2) (envelope-from ) id 1q3zy8-00A5KY-SD; Tue, 30 May 2023 15:05:00 +0100 In-Reply-To: References: From: "Russell King (Oracle)" To: Catalin Marinas , Jonathan Corbet , Will Deacon Cc: linux-arm-kernel@lists.infradead.org, linux-doc@vger.kernel.org Subject: [PATCH RFC 06/17] arm64: text replication: add sanity checks MIME-Version: 1.0 Content-Disposition: inline Message-Id: Date: Tue, 30 May 2023 15:05:00 +0100 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230530_070502_871395_10E0861D X-CRM114-Status: GOOD ( 11.13 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org The kernel text and modules must be in separate L0 page table entries. Signed-off-by: Russell King (Oracle) --- arch/arm64/mm/ktext.c | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) diff --git a/arch/arm64/mm/ktext.c b/arch/arm64/mm/ktext.c index 3a8d37c9abc4..901f159c65e6 100644 --- a/arch/arm64/mm/ktext.c +++ b/arch/arm64/mm/ktext.c @@ -3,6 +3,27 @@ * Copyright (C) 2022, Oracle and/or its affiliates. */ +#include +#include + +#include +#include + void __init ktext_replication_init(void) { + int kidx = pgd_index((phys_addr_t)KERNEL_START); + + /* + * If we've messed up and the kernel shares a L0 entry with the + * module or vmalloc area, then don't even attempt to use text + * replication. + */ + if (pgd_index(MODULES_VADDR) == kidx) { + pr_warn("Kernel is located in the same L0 index as modules - text replication disabled\n"); + return; + } + if (pgd_index(VMALLOC_START) == kidx) { + pr_warn("Kernel is located in the same L0 index as vmalloc - text replication disabled\n"); + return; + } } From patchwork Tue May 30 14:05:05 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Russell King (Oracle)" X-Patchwork-Id: 13260148 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 0C50BC77B7A for ; Tue, 30 May 2023 14:05:44 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Date:Message-Id:MIME-Version:Subject:Cc :To:From:References:In-Reply-To:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=WVzP3yr1So6jbzqKFLjITmKXfTkGTV6vKUAAiQOsXdM=; b=1ZI0UW3FftOPir ODMNJTc5JGdSU7tOJpzCCXopqaj7t1TE1hmnhdyiFFrjxLYKZGxbhNZG7p/GFADBps5ldlW9eOdj6 X3QmXawV9Mt5g8gSXChC6DEul9kY4YQBF49MH/mUftq7peenEU+iUb8tr67eISlZ4lfGU87dGrcIW OG7l7Vs2R/bN8mTXeDqJlLGJFs8+o1WxNG6eNIme+lcZAPAwnXPGAioUQ9/YSIlKJBMNRRViI6H9F gO+GQViOOMhMWhkK6OAnhK/SMM3gmPjWwYeJd2qSfmCoaqUSBPz5/IVT73eZmH96CJMYcOWAQTqIv UYYu3Fhfhh/43+t9vVXg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1q3zyO-00E9L8-02; Tue, 30 May 2023 14:05:16 +0000 Received: from pandora.armlinux.org.uk ([2001:4d48:ad52:32c8:5054:ff:fe00:142]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1q3zyF-00E9HT-2m for linux-arm-kernel@lists.infradead.org; Tue, 30 May 2023 14:05:09 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=armlinux.org.uk; s=pandora-2019; h=Date:Sender:Message-Id:Content-Type: Content-Transfer-Encoding:MIME-Version:Subject:Cc:To:From:References: In-Reply-To:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help: List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=kS6q0fvuI7d1aTJBBBQTVQRRjCIadUHAhNRNeZ4W2zs=; b=DSgQPv4FC+r+dieFeCN46fvkig tBs0AzAlBQU/QEhDQO/8tp/aP+rwE72PQ1TBiZvJ6h5YhZjhugRuDPIOHI7dyKUc46DUU/0hASyn7 16CMxOlDmCDIzTbWVrkRZLWpUGDVeSkGnxxDScI0FQC8naUDFCm/EsrSPBwqX0QH7TC89fyYA0tq+ w29fdhD2rxHCb+nZP6O8qQr4xdC05WL9NQwu7KBUCDtHrZgtiBXleFBm8Jj8iVUsrIc7Dd2meAvEo 3yaMhIcY62XNOmM/LVycylPpIIynH6kBtwmtc1phU+Rd4vJUZnkFiW2DbAX5cHPZHpftcIbb5Rjcb EwSPn8gw==; Received: from e0022681537dd.dyn.armlinux.org.uk ([fd8f:7570:feb6:1:222:68ff:fe15:37dd]:34648 helo=rmk-PC.armlinux.org.uk) by pandora.armlinux.org.uk with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1q3zyE-0002r7-Hm; Tue, 30 May 2023 15:05:06 +0100 Received: from rmk by rmk-PC.armlinux.org.uk with local (Exim 4.94.2) (envelope-from ) id 1q3zyD-00A5Ki-VV; Tue, 30 May 2023 15:05:06 +0100 In-Reply-To: References: From: "Russell King (Oracle)" To: Catalin Marinas , Jonathan Corbet , Will Deacon Cc: linux-arm-kernel@lists.infradead.org, linux-doc@vger.kernel.org Subject: [PATCH RFC 07/17] arm64: text replication: copy initial kernel text MIME-Version: 1.0 Content-Disposition: inline Message-Id: Date: Tue, 30 May 2023 15:05:05 +0100 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230530_070507_899557_3C7D667F X-CRM114-Status: GOOD ( 14.55 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Allocate memory on the appropriate node for the per-node copies of the kernel text, and copy the kernel text to that memory. Clean and invalidate the caches to the point of unification so that the copied text is correctly visible to the target node. Signed-off-by: Russell King (Oracle) --- arch/arm64/mm/ktext.c | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) diff --git a/arch/arm64/mm/ktext.c b/arch/arm64/mm/ktext.c index 901f159c65e6..4c803b89fcfe 100644 --- a/arch/arm64/mm/ktext.c +++ b/arch/arm64/mm/ktext.c @@ -4,14 +4,23 @@ */ #include +#include +#include #include +#include +#include #include #include +static void *kernel_texts[MAX_NUMNODES]; + +/* Allocate memory for the replicated kernel texts. */ void __init ktext_replication_init(void) { + size_t size = _etext - _stext; int kidx = pgd_index((phys_addr_t)KERNEL_START); + int nid; /* * If we've messed up and the kernel shares a L0 entry with the @@ -26,4 +35,16 @@ void __init ktext_replication_init(void) pr_warn("Kernel is located in the same L0 index as vmalloc - text replication disabled\n"); return; } + + for_each_node(nid) { + /* Nothing to do for node 0 */ + if (!nid) + continue; + + /* Allocate and copy initial kernel text for this node */ + kernel_texts[nid] = memblock_alloc_node(size, PAGE_SIZE, nid); + memcpy(kernel_texts[nid], _stext, size); + caches_clean_inval_pou((u64)kernel_texts[nid], + (u64)kernel_texts[nid] + size); + } } From patchwork Tue May 30 14:05:11 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Russell King (Oracle)" X-Patchwork-Id: 13260146 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 7AC23C77B73 for ; Tue, 30 May 2023 14:05:39 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Date:Message-Id:MIME-Version:Subject:Cc :To:From:References:In-Reply-To:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=5svQcYoVvayjP21u7uX1tLJy6rOXTvadYEEPT1XArik=; b=2vEsvGDxaqBZgy v5cfob9Qug/pihyATTyund8od5L9SCeUkDQw0ssjXIW5o+E2v7hUC3fnkDSNOn3QaaRngUkfIaQrd 9p2lJ8f6hVZh6mbxjMQdxUcjzrfMJFjMeNDenE/CSDIe+Lcvd51I8zmdGz0PBH8Gih4JZdjekE/Qj RHkN4xEErua2PFhces6XtJaoWOHaGGJTrR4SwrhUGmTSd/xM5mqbmc1IqZbL8aDAMrCm7s/iLax4+ +KEOLCBNx77215PPHQ+TXka5VGJsncndgP7ICSk5YqDy18i3hoqU76r7wSjOkVdRetK26JQAkfLjj O5fWWdk2GHi7aJTqLS7g==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1q3zyP-00E9Lz-0n; Tue, 30 May 2023 14:05:17 +0000 Received: from pandora.armlinux.org.uk ([2001:4d48:ad52:32c8:5054:ff:fe00:142]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1q3zyL-00E9Ip-03 for linux-arm-kernel@lists.infradead.org; Tue, 30 May 2023 14:05:14 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=armlinux.org.uk; s=pandora-2019; h=Date:Sender:Message-Id:Content-Type: Content-Transfer-Encoding:MIME-Version:Subject:Cc:To:From:References: In-Reply-To:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help: List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=AfVF8WKknrmFxQThJnNdxhv6A8Sxo2qqKBgvh/eDYqM=; b=KsYn4wY2z6nF+NCXFfg/6IlsbO ew46ZPNDy36Mfa8crNqECJZh1DdbHVeCmqNCf2UsBYcUewFqAv6/pXhM9ZITOR2oM8+FCt688TNwu o6G3YvBGy+u2TUHWHWw7bw3HX5muL8JKVkdJpLnWCRx2ExVdyo9ghcetUZp3yTDJVK2ZaXPNNsAux QSey23N5jJEhf1t3u3vGYMvz6uAi1DIm0MDjNhLSbk1h9MYVf6pLi6jN8ZiN7WVkMpK/FJZRMQZK1 x0qSC7FpjDAPfYCPgiF4b4TPLBE//LQGFnvXvoSm25zn+Gi+DrBvrVUQqSFCzE5bhUPB/H1YGxonF XGt8jGow==; Received: from e0022681537dd.dyn.armlinux.org.uk ([fd8f:7570:feb6:1:222:68ff:fe15:37dd]:34652 helo=rmk-PC.armlinux.org.uk) by pandora.armlinux.org.uk with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1q3zyJ-0002rI-L7; Tue, 30 May 2023 15:05:11 +0100 Received: from rmk by rmk-PC.armlinux.org.uk with local (Exim 4.94.2) (envelope-from ) id 1q3zyJ-00A5Kq-2X; Tue, 30 May 2023 15:05:11 +0100 In-Reply-To: References: From: "Russell King (Oracle)" To: Catalin Marinas , Jonathan Corbet , Will Deacon Cc: linux-arm-kernel@lists.infradead.org, linux-doc@vger.kernel.org Subject: [PATCH RFC 08/17] arm64: text replication: add node text patching MIME-Version: 1.0 Content-Disposition: inline Message-Id: Date: Tue, 30 May 2023 15:05:11 +0100 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230530_070513_056150_AAFD3A36 X-CRM114-Status: GOOD ( 17.67 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Add support for text patching on our replicated texts. Signed-off-by: Russell King (Oracle) --- arch/arm64/include/asm/ktext.h | 12 +++++++ arch/arm64/kernel/alternative.c | 2 ++ arch/arm64/kernel/patching.c | 7 +++- arch/arm64/mm/ktext.c | 58 +++++++++++++++++++++++++++++++++ 4 files changed, 78 insertions(+), 1 deletion(-) diff --git a/arch/arm64/include/asm/ktext.h b/arch/arm64/include/asm/ktext.h index 1a5f7452a3bf..289e11289c06 100644 --- a/arch/arm64/include/asm/ktext.h +++ b/arch/arm64/include/asm/ktext.h @@ -5,9 +5,13 @@ #ifndef ASM_KTEXT_H #define ASM_KTEXT_H +#include + #ifdef CONFIG_REPLICATE_KTEXT void ktext_replication_init(void); +void __kprobes ktext_replication_patch(u32 *tp, __le32 insn); +void ktext_replication_patch_alternative(__le32 *src, int nr_inst); #else @@ -15,6 +19,14 @@ static inline void ktext_replication_init(void) { } +static inline void __kprobes ktext_replication_patch(u32 *tp, __le32 insn) +{ +} + +static inline void ktext_replication_patch_alternative(__le32 *src, int nr_inst) +{ +} + #endif #endif diff --git a/arch/arm64/kernel/alternative.c b/arch/arm64/kernel/alternative.c index df9a73458a85..6a897d1dda76 100644 --- a/arch/arm64/kernel/alternative.c +++ b/arch/arm64/kernel/alternative.c @@ -15,6 +15,7 @@ #include #include #include +#include #include #include #include @@ -174,6 +175,7 @@ static void __apply_alternatives(const struct alt_region *region, alt_cb(alt, origptr, updptr, nr_inst); if (!is_module) { + ktext_replication_patch_alternative(updptr, nr_inst); clean_dcache_range_nopatch((u64)origptr, (u64)(origptr + nr_inst)); } diff --git a/arch/arm64/kernel/patching.c b/arch/arm64/kernel/patching.c index b4835f6d594b..627fff6ddda2 100644 --- a/arch/arm64/kernel/patching.c +++ b/arch/arm64/kernel/patching.c @@ -10,6 +10,7 @@ #include #include #include +#include #include #include @@ -115,9 +116,13 @@ int __kprobes aarch64_insn_patch_text_nosync(void *addr, u32 insn) return -EINVAL; ret = aarch64_insn_write(tp, insn); - if (ret == 0) + if (ret == 0) { + /* Also patch the other nodes */ + ktext_replication_patch(tp, cpu_to_le32(insn)); + caches_clean_inval_pou((uintptr_t)tp, (uintptr_t)tp + AARCH64_INSN_SIZE); + } return ret; } diff --git a/arch/arm64/mm/ktext.c b/arch/arm64/mm/ktext.c index 4c803b89fcfe..80120b5fd29f 100644 --- a/arch/arm64/mm/ktext.c +++ b/arch/arm64/mm/ktext.c @@ -3,8 +3,10 @@ * Copyright (C) 2022, Oracle and/or its affiliates. */ +#include #include #include +#include #include #include #include @@ -15,6 +17,62 @@ static void *kernel_texts[MAX_NUMNODES]; +void __kprobes ktext_replication_patch(u32 *tp, __le32 insn) +{ + unsigned long offset; + int nid, this_nid; + __le32 *p; + + if (!is_kernel_text((unsigned long)tp)) + return; + + offset = (unsigned long)tp - (unsigned long)_stext; + + this_nid = numa_node_id(); + if (this_nid) { + /* The cache maintenance by aarch64_insn_patch_text_nosync() + * will occur on this node. We need it to occur on node 0. + */ + p = (void *)lm_alias(_stext) + offset; + caches_clean_inval_pou((u64)p, (u64)p + AARCH64_INSN_SIZE); + } + + for_each_node(nid) { + if (!kernel_texts[nid]) + continue; + + p = kernel_texts[nid] + offset; + WRITE_ONCE(*p, insn); + caches_clean_inval_pou((u64)p, (u64)p + AARCH64_INSN_SIZE); + } +} + +/* Copy the patched alternative from the node0 image to the other + * modes. src is the node 0 linear-mapping address. + */ +void ktext_replication_patch_alternative(__le32 *src, int nr_inst) +{ + unsigned long offset; + size_t size; + int nid; + __le32 *p; + + offset = (unsigned long)src - (unsigned long)lm_alias(_stext); + if (WARN_ON_ONCE(offset >= _etext - _stext)) + return; + + size = AARCH64_INSN_SIZE * nr_inst; + + for_each_node(nid) { + if (!kernel_texts[nid]) + continue; + + p = kernel_texts[nid] + offset; + memcpy(p, src, size); + clean_dcache_range_nopatch((u64)p, (u64)p + size); + } +} + /* Allocate memory for the replicated kernel texts. */ void __init ktext_replication_init(void) { From patchwork Tue May 30 14:05:16 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Russell King (Oracle)" X-Patchwork-Id: 13260152 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id EF1A2C7EE31 for ; Tue, 30 May 2023 14:06:21 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Date:Message-Id:MIME-Version:Subject:Cc :To:From:References:In-Reply-To:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=57Yuc2njJvwJ7/MF1EZRoPNdi2T7hKI5b4pMQfe7nw4=; b=EmTEVXcytfGryF /ztifEFVXV23BoVICMyTp1FQ7WLU17uoCK+sTrghBeniGIs9Y928i/1IPzlNtQ3tddZA5YTZ5YHok Hyhbk9OAn3/2xyvHJJvJZ+KchYb5d2BptnhTmwDuNKjyKsoiJunZfhxT+ODPFZBgyVfjB/x267pnA hO185+Q1cuGDloepxIkG7khhlQEdcWhllMEBEBJtUqRAKyLYHd9oeZL8IdPhqXbRQePr48hCB2HEs OG8sT8m+5b49O6vJKEokEcdQ99pxNwj6aGt8dQGeig2TX/jRJon1nPTMPVtadX3h16HdFUzbMfCYq wD6gnvb0o3pr3CSmTSKQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1q3zz2-00E9ex-0W; Tue, 30 May 2023 14:05:56 +0000 Received: from pandora.armlinux.org.uk ([2001:4d48:ad52:32c8:5054:ff:fe00:142]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1q3zyQ-00E9M9-0T for linux-arm-kernel@lists.infradead.org; Tue, 30 May 2023 14:05:19 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=armlinux.org.uk; s=pandora-2019; h=Date:Sender:Message-Id:Content-Type: Content-Transfer-Encoding:MIME-Version:Subject:Cc:To:From:References: In-Reply-To:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help: List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=oXqFasuCpSAoOt6n7DWf+f+JsPewVatr28dIEL6imc4=; b=heK4WxHNpz8UDoxcu3VrZbIJbG s2jY/W0+7whvgUpps5quKdEyZw6t4+97sVhXvaE6ibpYgQccSi476pDHpAEF++nAFVMQ2gJN4eU1e QqexwGSK3h/yOw0GwdFlb1Dy2O5yoLJaVzqOitGLJFB/nr7WAF4hi70PwVdMdMRYpZKMhKa7wW/rl G4XaE8HO3be3G9P+oeLK6PxE+TzHeUFpxN0AfcDJGdJ5ZVjOhw35VvJEVDUAS8k5Kz9j0L0QFnWTg tEHtizxAhgYL27klRRq1w8Jvy6TkVjCsfhY41QIbSeERM5TJMtqbQ0YBiJnk6XiTV/02UJhZMnGka NjabiIcg==; Received: from e0022681537dd.dyn.armlinux.org.uk ([fd8f:7570:feb6:1:222:68ff:fe15:37dd]:48808 helo=rmk-PC.armlinux.org.uk) by pandora.armlinux.org.uk with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1q3zyO-0002rS-PY; Tue, 30 May 2023 15:05:16 +0100 Received: from rmk by rmk-PC.armlinux.org.uk with local (Exim 4.94.2) (envelope-from ) id 1q3zyO-00A5L2-5y; Tue, 30 May 2023 15:05:16 +0100 In-Reply-To: References: From: "Russell King (Oracle)" To: Catalin Marinas , Jonathan Corbet , Will Deacon Cc: linux-arm-kernel@lists.infradead.org, linux-doc@vger.kernel.org Subject: [PATCH RFC 09/17] arm64: text replication: add node 0 page table definitions MIME-Version: 1.0 Content-Disposition: inline Message-Id: Date: Tue, 30 May 2023 15:05:16 +0100 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230530_070518_184542_BC3B2760 X-CRM114-Status: GOOD ( 11.35 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Add a struct definition for the level zero page table group (the optional trampoline page tables, reserved page tables, and swapper page tables). Add a symbol and extern declaration for the node 0 page table group. Add an array of pointers to per-node page tables, which will default to using the node 0 page table group. Signed-off-by: Russell King (Oracle) --- arch/arm64/include/asm/pgtable.h | 14 ++++++++++++++ arch/arm64/kernel/vmlinux.lds.S | 3 +++ arch/arm64/mm/ktext.c | 4 ++++ 3 files changed, 21 insertions(+) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index cb526e69299d..1e72067d1e9e 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -615,6 +615,20 @@ extern pgd_t idmap_pg_dir[PTRS_PER_PGD]; extern pgd_t tramp_pg_dir[PTRS_PER_PGD]; extern pgd_t reserved_pg_dir[PTRS_PER_PGD]; +struct pgtables { +#ifdef CONFIG_UNMAP_KERNEL_AT_EL0 + pgd_t tramp_pg_dir[PTRS_PER_PGD]; +#endif + pgd_t reserved_pg_dir[PTRS_PER_PGD]; + pgd_t swapper_pg_dir[PTRS_PER_PGD]; +}; + +extern struct pgtables pgtable_node0; + +#ifdef CONFIG_REPLICATE_KTEXT +extern struct pgtables *pgtables[MAX_NUMNODES]; +#endif + extern void set_swapper_pgd(pgd_t *pgdp, pgd_t pgd); static inline bool in_swapper_pgdir(void *addr) diff --git a/arch/arm64/kernel/vmlinux.lds.S b/arch/arm64/kernel/vmlinux.lds.S index 3cd7e76cc562..d3c7ed76adbf 100644 --- a/arch/arm64/kernel/vmlinux.lds.S +++ b/arch/arm64/kernel/vmlinux.lds.S @@ -212,6 +212,9 @@ SECTIONS idmap_pg_dir = .; . += PAGE_SIZE; + /* pgtable struct - covers the tramp, reserved and swapper pgdirs */ + pgtable_node0 = .; + #ifdef CONFIG_UNMAP_KERNEL_AT_EL0 tramp_pg_dir = .; . += PAGE_SIZE; diff --git a/arch/arm64/mm/ktext.c b/arch/arm64/mm/ktext.c index 80120b5fd29f..85fc97877d75 100644 --- a/arch/arm64/mm/ktext.c +++ b/arch/arm64/mm/ktext.c @@ -15,6 +15,10 @@ #include #include +struct pgtables *pgtables[MAX_NUMNODES] = { + [0 ... MAX_NUMNODES - 1] = &pgtable_node0, +}; + static void *kernel_texts[MAX_NUMNODES]; void __kprobes ktext_replication_patch(u32 *tp, __le32 insn) From patchwork Tue May 30 14:05:21 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Russell King (Oracle)" X-Patchwork-Id: 13260150 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 2256DC7EE2C for ; Tue, 30 May 2023 14:06:21 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Date:Message-Id:MIME-Version:Subject:Cc :To:From:References:In-Reply-To:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=fFrALfB9/FvDW080kvu1ZAYuz0gNSLx5bMTttsaOcn8=; b=keSHzkOn0x59VI FIZ7wc2zICbPpt8wENZlj+BE8gZ+WntsOOxVJ3zGp6XJBZ6SGliZ55B6T1DGgI4mpvqy79yMTzIxZ G/dSW19CHb6Tz+LOM41VMEWCHSE5PViXGiorhgXguNKvQH2tnL7EORrUBrxfFYzdRR/lnDRpwjTiK FiaBzgBzvg6SdL9I9etF2kIXTCnKDYntzgcDS7Sc5ir/zmYM3t1QZRGu/utVW1CqBPl+dBqfL9VJk trja1ZWRw+0duhjREdC5OglHLrU4LoYEgIXs2jL5d9juzSc0UBaIKO8tEbw3mJIJeLZi5/DNXCzzz 9Zg3SGcu7mE4d7Ipy1dw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1q3zz2-00E9fM-2X; Tue, 30 May 2023 14:05:56 +0000 Received: from pandora.armlinux.org.uk ([2001:4d48:ad52:32c8:5054:ff:fe00:142]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1q3zyV-00E9Pl-18 for linux-arm-kernel@lists.infradead.org; Tue, 30 May 2023 14:05:25 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=armlinux.org.uk; s=pandora-2019; h=Date:Sender:Message-Id:Content-Type: Content-Transfer-Encoding:MIME-Version:Subject:Cc:To:From:References: In-Reply-To:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help: List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=nBBrF3BbR5Lwl8Cc/QvChlKOe5ANaMj83t0gXJYwyIY=; b=oQ5cbCl+2ssMheOt0eV7lHvx6M dxxhnEXbZLp54vSQvkkdmNayzD2ZdDqYEi+j40AJGCzdcpK6k0HSV09c7BMAYsbZ1EbGC+gYpeyNa Zyt7am/soPWFwXgQLdnyFYrPOC5N2U+dJmIwKTv0MWyMkJfsVpHst1jgl+XuWSwe98fQ+Big+sIJy Ecf2sWV1+2w4/U+X0BZnqqHLaV7r3WqnT5EViQ74qgW1ord8v7UJQuW90GiNZ2tcfBa0+9/k+Djwm gufjuMo5Y4yxyT/ob+tuFBHAMCtLl+1dq2iQk+B2wera0WoVCGbZ0obIXxdEIMETFv1YOKZiVE91S YI5xLVQA==; Received: from e0022681537dd.dyn.armlinux.org.uk ([fd8f:7570:feb6:1:222:68ff:fe15:37dd]:48812 helo=rmk-PC.armlinux.org.uk) by pandora.armlinux.org.uk with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1q3zyT-0002rc-TN; Tue, 30 May 2023 15:05:21 +0100 Received: from rmk by rmk-PC.armlinux.org.uk with local (Exim 4.94.2) (envelope-from ) id 1q3zyT-00A5LC-9X; Tue, 30 May 2023 15:05:21 +0100 In-Reply-To: References: From: "Russell King (Oracle)" To: Catalin Marinas , Jonathan Corbet , Will Deacon Cc: linux-arm-kernel@lists.infradead.org, linux-doc@vger.kernel.org Subject: [PATCH RFC 10/17] arm64: text replication: add swapper page directory helpers MIME-Version: 1.0 Content-Disposition: inline Message-Id: Date: Tue, 30 May 2023 15:05:21 +0100 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230530_070523_714227_C9FBC416 X-CRM114-Status: GOOD ( 12.53 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Add a series of helpers for the swapper page directories - a set which return those for the calling CPU, and those which take the NUMA node number. Signed-off-by: Russell King (Oracle) --- arch/arm64/include/asm/pgtable.h | 19 +++++++++++++++++++ arch/arm64/kernel/hibernate.c | 2 +- arch/arm64/mm/ktext.c | 20 ++++++++++++++++++++ 3 files changed, 40 insertions(+), 1 deletion(-) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 1e72067d1e9e..5cfff64e4944 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -627,6 +627,25 @@ extern struct pgtables pgtable_node0; #ifdef CONFIG_REPLICATE_KTEXT extern struct pgtables *pgtables[MAX_NUMNODES]; + +pgd_t *swapper_pg_dir_node(void); +phys_addr_t __swapper_pg_dir_node_phys(int nid); +phys_addr_t swapper_pg_dir_node_phys(void); +#else +static inline pgd_t *swapper_pg_dir_node(void) +{ + return swapper_pg_dir; +} + +static inline phys_addr_t __swapper_pg_dir_node_phys(int nid) +{ + return __pa_symbol(swapper_pg_dir); +} + +static inline phys_addr_t swapper_pg_dir_node_phys(void) +{ + return __pa_symbol(swapper_pg_dir); +} #endif extern void set_swapper_pgd(pgd_t *pgdp, pgd_t pgd); diff --git a/arch/arm64/kernel/hibernate.c b/arch/arm64/kernel/hibernate.c index 788597a6b6a2..2236119bf16d 100644 --- a/arch/arm64/kernel/hibernate.c +++ b/arch/arm64/kernel/hibernate.c @@ -114,7 +114,7 @@ int arch_hibernation_header_save(void *addr, unsigned int max_size) return -EOVERFLOW; arch_hdr_invariants(&hdr->invariants); - hdr->ttbr1_el1 = __pa_symbol(swapper_pg_dir); + hdr->ttbr1_el1 = swapper_pg_dir_node_phys(); hdr->reenter_kernel = _cpu_resume; /* We can't use __hyp_get_vectors() because kvm may still be loaded */ diff --git a/arch/arm64/mm/ktext.c b/arch/arm64/mm/ktext.c index 85fc97877d75..ac5754972a09 100644 --- a/arch/arm64/mm/ktext.c +++ b/arch/arm64/mm/ktext.c @@ -21,6 +21,26 @@ struct pgtables *pgtables[MAX_NUMNODES] = { static void *kernel_texts[MAX_NUMNODES]; +static pgd_t *__swapper_pg_dir_node(int nid) +{ + return pgtables[nid]->swapper_pg_dir; +} + +pgd_t *swapper_pg_dir_node(void) +{ + return __swapper_pg_dir_node(numa_node_id()); +} + +phys_addr_t __swapper_pg_dir_node_phys(int nid) +{ + return __pa(__swapper_pg_dir_node(nid)); +} + +phys_addr_t swapper_pg_dir_node_phys(void) +{ + return __swapper_pg_dir_node_phys(numa_node_id()); +} + void __kprobes ktext_replication_patch(u32 *tp, __le32 insn) { unsigned long offset; From patchwork Tue May 30 14:05:26 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Russell King (Oracle)" X-Patchwork-Id: 13260153 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 831A2C7EE32 for ; Tue, 30 May 2023 14:06:22 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Date:Message-Id:MIME-Version:Subject:Cc :To:From:References:In-Reply-To:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=23YFCT3/bCr+N67+JxXg3agaVfe3J7DV/JerpDIM09s=; b=R7sWz1+LKwugQ7 xKSA/pz2sjo10h7mcWODFKr82swHGlhfTakUQC9XX8gjPWpfg6mKWcyySkiEFeEqEtSj314b99eAb pmF+a3UsdDxAJj1AV+QfUSWtzLkWP5p/Ud/yaNNXgsXiNh8+EO/Fki9WgtxFR+tn14JSFg0AY+URG IiEGxE6XrsJba25+r/Skq7dXAAg0xarvUySddHpr9qqjk/Iwc0UTa+xA02sbWpBJzQBK2d4a9i+OP rIrCWYTygnTxTuM3jpOXpIVW5oZOsaXJGStS3QFfwX1d9IJAziNVf/nneT3TUWIaoYDgx35Lt+lnN liBTyt3RJGQJHgwZZL7g==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1q3zz4-00E9gL-0J; Tue, 30 May 2023 14:05:58 +0000 Received: from pandora.armlinux.org.uk ([2001:4d48:ad52:32c8:5054:ff:fe00:142]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1q3zya-00E9SR-1J for linux-arm-kernel@lists.infradead.org; Tue, 30 May 2023 14:05:29 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=armlinux.org.uk; s=pandora-2019; h=Date:Sender:Message-Id:Content-Type: Content-Transfer-Encoding:MIME-Version:Subject:Cc:To:From:References: In-Reply-To:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help: List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=Db19HrZuPZw4+5KBCaIs6xv1CNqiis+vz1vgIyiDv9Y=; b=NNbN4IwrkkbXaXTdMY8G7qTzaV u+KnUt6mA8epo4YWGZkuLv6nhEmi2KFc9yP+Qk14+VJn5afMoO3RKQjA8BBJ2WUiuDWcQixSboU/o P8Ud3/Gb1rOVr8s0eGnjRZV5c988VG9wPQh6FLmURwiWr6bJW6SBBPwhAhgOkLU1WBxIVTPpEdtIB yGOeq9G02fgZsRQJrwWeb8fXwwKL6ItSiuAiB1ep+JcwSqd9pQ6D6Q0uAVPaXI4zKWoXA91nOO9a6 A2mDLsbvYSe2APLuigrrpUyKVC0pz85aIGCRiJqWgxevlGXJCLYkk2J9cLpPkDAVCENXENePSeyyQ ogG5ZLCQ==; Received: from e0022681537dd.dyn.armlinux.org.uk ([fd8f:7570:feb6:1:222:68ff:fe15:37dd]:47486 helo=rmk-PC.armlinux.org.uk) by pandora.armlinux.org.uk with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1q3zyZ-0002sA-0T; Tue, 30 May 2023 15:05:27 +0100 Received: from rmk by rmk-PC.armlinux.org.uk with local (Exim 4.94.2) (envelope-from ) id 1q3zyY-00A5LM-D8; Tue, 30 May 2023 15:05:26 +0100 In-Reply-To: References: From: "Russell King (Oracle)" To: Catalin Marinas , Jonathan Corbet , Will Deacon Cc: linux-arm-kernel@lists.infradead.org, linux-doc@vger.kernel.org Subject: [PATCH RFC 11/17] arm64: text replication: create per-node kernel page tables MIME-Version: 1.0 Content-Disposition: inline Message-Id: Date: Tue, 30 May 2023 15:05:26 +0100 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230530_070528_470518_790F727A X-CRM114-Status: GOOD ( 20.90 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Allocate the level 0 page tables for the per-node kernel text replication, but copy all level 0 table entries from the NUMA node 0 table. Therefore, for the time being, each node's level 0 page tables will contain identical entries, and thus other nodes will continue to use the node 0 kernel text. Since the level 0 page tables can be updated at runtime to add entries for vmalloc and module space, propagate these updates to the other swapper page tables. The exception is if we see an update for the level 0 entry which points to the kernel mapping. We also need to setup a copy of the trampoline page tables as well, as the assembly code relies on the two page tables being a fixed offset apart. Signed-off-by: Russell King (Oracle) --- arch/arm64/include/asm/ktext.h | 12 ++++++++++ arch/arm64/mm/ktext.c | 42 +++++++++++++++++++++++++++++++++- arch/arm64/mm/mmu.c | 5 ++++ 3 files changed, 58 insertions(+), 1 deletion(-) diff --git a/arch/arm64/include/asm/ktext.h b/arch/arm64/include/asm/ktext.h index 289e11289c06..386f9812d3c1 100644 --- a/arch/arm64/include/asm/ktext.h +++ b/arch/arm64/include/asm/ktext.h @@ -7,11 +7,15 @@ #include +#include + #ifdef CONFIG_REPLICATE_KTEXT void ktext_replication_init(void); void __kprobes ktext_replication_patch(u32 *tp, __le32 insn); void ktext_replication_patch_alternative(__le32 *src, int nr_inst); +void ktext_replication_set_swapper_pgd(pgd_t *pgdp, pgd_t pgd); +void ktext_replication_init_tramp(void); #else @@ -27,6 +31,14 @@ static inline void ktext_replication_patch_alternative(__le32 *src, int nr_inst) { } +static inline void ktext_replication_set_swapper_pgd(pgd_t *pgdp, pgd_t pgd) +{ +} + +static inline void ktext_replication_init_tramp(void) +{ +} + #endif #endif diff --git a/arch/arm64/mm/ktext.c b/arch/arm64/mm/ktext.c index ac5754972a09..290012d2bd03 100644 --- a/arch/arm64/mm/ktext.c +++ b/arch/arm64/mm/ktext.c @@ -14,6 +14,7 @@ #include #include #include +#include struct pgtables *pgtables[MAX_NUMNODES] = { [0 ... MAX_NUMNODES - 1] = &pgtable_node0, @@ -97,7 +98,7 @@ void ktext_replication_patch_alternative(__le32 *src, int nr_inst) } } -/* Allocate memory for the replicated kernel texts. */ +/* Allocate page tables and memory for the replicated kernel texts. */ void __init ktext_replication_init(void) { size_t size = _etext - _stext; @@ -128,5 +129,44 @@ void __init ktext_replication_init(void) memcpy(kernel_texts[nid], _stext, size); caches_clean_inval_pou((u64)kernel_texts[nid], (u64)kernel_texts[nid] + size); + + /* Allocate the pagetables for this node */ + pgtables[nid] = memblock_alloc_node(sizeof(*pgtables[0]), + PGD_SIZE, nid); + + /* Copy initial swapper page directory */ + memcpy(pgtables[nid]->swapper_pg_dir, swapper_pg_dir, PGD_SIZE); + } +} + +void ktext_replication_set_swapper_pgd(pgd_t *pgdp, pgd_t pgd) +{ + unsigned long idx = pgdp - swapper_pg_dir; + int nid; + + if (WARN_ON_ONCE(idx >= PTRS_PER_PGD) || + WARN_ON_ONCE(idx == pgd_index((phys_addr_t)KERNEL_START))) + return; + + for_each_node(nid) { + if (pgtables[nid]->swapper_pg_dir == swapper_pg_dir) + continue; + + WRITE_ONCE(pgtables[nid]->swapper_pg_dir[idx], pgd); + } +} + +#ifdef CONFIG_UNMAP_KERNEL_AT_EL0 +void __init ktext_replication_init_tramp(void) +{ + int nid; + + for_each_node(nid) { + /* Nothing to do for node 0 */ + if (pgtables[nid]->tramp_pg_dir == tramp_pg_dir) + continue; + + memcpy(pgtables[nid]->tramp_pg_dir, tramp_pg_dir, PGD_SIZE); } } +#endif diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index baf74d0c43c9..12fc3b1116e6 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -31,6 +31,7 @@ #include #include #include +#include #include #include #include @@ -81,6 +82,7 @@ void set_swapper_pgd(pgd_t *pgdp, pgd_t pgd) pgd_t *fixmap_pgdp; spin_lock(&swapper_pgdir_lock); + ktext_replication_set_swapper_pgd(pgdp, pgd); fixmap_pgdp = pgd_set_fixmap(__pa_symbol(pgdp)); WRITE_ONCE(*fixmap_pgdp, pgd); /* @@ -695,6 +697,9 @@ static int __init map_entry_trampoline(void) __set_fixmap(FIX_ENTRY_TRAMP_TEXT1 - i, pa_start + i * PAGE_SIZE, PAGE_KERNEL_RO); + /* Copy trampoline page tables to other numa nodes */ + ktext_replication_init_tramp(); + return 0; } core_initcall(map_entry_trampoline); From patchwork Tue May 30 14:05:31 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Russell King (Oracle)" X-Patchwork-Id: 13260149 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 873E3C7EE23 for ; Tue, 30 May 2023 14:06:20 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Date:Message-Id:MIME-Version:Subject:Cc :To:From:References:In-Reply-To:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=gGbre8h8wnWfU3ZW9OCZ5e/hz2hHSZ0PcIksgoWZHnQ=; b=j5h9SARrkfhDhz 3K1hs22yxPu1WzjDm5mtfWD91za6Xc82kESJ4DrIBG/uJHAteP1NiQ5IbaUmrCw9eaBLDotQON/DK YlIxxVv+gxXFR3YqJgDtf/ovEeme1s8ak3a0BEVpXrmhbCxT+aiSsKvNfK1v/HSl89R+o/9iOXnVw Sv+VHkXcXRSs9tMxbsngX0ndcUvB7p8dpYlzkW09mXJObjHEZZwfA9p5dWSmhXgdDUTI5PmNMEXDC 0EBUqVtq/lt81G68tN3JB6g6WmKxx2/2DembSWXnhllffDO71NsrWc1masQZdaYetT0bU54nLMDU9 //4LljeFTIiVu3youLUA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1q3zz4-00E9h9-2s; Tue, 30 May 2023 14:05:58 +0000 Received: from pandora.armlinux.org.uk ([2001:4d48:ad52:32c8:5054:ff:fe00:142]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1q3zyg-00E9VI-0L for linux-arm-kernel@lists.infradead.org; Tue, 30 May 2023 14:05:35 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=armlinux.org.uk; s=pandora-2019; h=Date:Sender:Message-Id:Content-Type: Content-Transfer-Encoding:MIME-Version:Subject:Cc:To:From:References: In-Reply-To:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help: List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=35iZ/Ua8mvKVmClKBiJfRXLH3cOhYz9V553i5knsgVc=; b=KHCSuVDS8Gfeua7sdHLXHZGPRu iaeB/VaTYkzw+WBiyTdM7zs/FA+oe+MT3xiE61Lyp63ObPyTQYSc4r3DUvamVBKCSN/PHsCxecW6R 7a43lfESGQO85ohcQj8Nctwtb9wdXoApxuRSn70r54f4xf/KSeZPLdnOM3KHhazNwR2oeeJo13oSP kNNLnFVHW3RcQsWbqvcTbV//Pn8R1eM5tx+vRIVDcE6P6ZPJ7xdHDMNy+fkGWT3jEH33/wAy7BqoY 3lkZemmoZOMsKmxMREhVUtPDul8X0eelm/o/GNFiZ2pcWyx+mNlQ602keyKK2x9yin/EEfI+3j5kI Jvj8JqFA==; Received: from e0022681537dd.dyn.armlinux.org.uk ([fd8f:7570:feb6:1:222:68ff:fe15:37dd]:47490 helo=rmk-PC.armlinux.org.uk) by pandora.armlinux.org.uk with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1q3zye-0002sL-3y; Tue, 30 May 2023 15:05:32 +0100 Received: from rmk by rmk-PC.armlinux.org.uk with local (Exim 4.94.2) (envelope-from ) id 1q3zyd-00A5LW-Gr; Tue, 30 May 2023 15:05:31 +0100 In-Reply-To: References: From: "Russell King (Oracle)" To: Catalin Marinas , Jonathan Corbet , Will Deacon Cc: linux-arm-kernel@lists.infradead.org, linux-doc@vger.kernel.org Subject: [PATCH RFC 12/17] arm64: text replication: boot secondary CPUs with appropriate TTBR1 MIME-Version: 1.0 Content-Disposition: inline Message-Id: Date: Tue, 30 May 2023 15:05:31 +0100 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230530_070534_142539_51FECDD8 X-CRM114-Status: GOOD ( 11.75 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Arrange for secondary CPUs to boot with TTBR1 pointing at the appropriate per-node copy of the kernel page tables for the CPUs NUMA node. Signed-off-by: Russell King (Oracle) --- arch/arm64/include/asm/smp.h | 1 + arch/arm64/kernel/asm-offsets.c | 1 + arch/arm64/kernel/head.S | 3 ++- arch/arm64/kernel/smp.c | 3 +++ 4 files changed, 7 insertions(+), 1 deletion(-) diff --git a/arch/arm64/include/asm/smp.h b/arch/arm64/include/asm/smp.h index f2d26235bfb4..9a4246141599 100644 --- a/arch/arm64/include/asm/smp.h +++ b/arch/arm64/include/asm/smp.h @@ -79,6 +79,7 @@ asmlinkage void secondary_start_kernel(void); struct secondary_data { struct task_struct *task; long status; + phys_addr_t ttbr1; }; extern struct secondary_data secondary_data; diff --git a/arch/arm64/kernel/asm-offsets.c b/arch/arm64/kernel/asm-offsets.c index 0996094b0d22..f3b8cf661de2 100644 --- a/arch/arm64/kernel/asm-offsets.c +++ b/arch/arm64/kernel/asm-offsets.c @@ -121,6 +121,7 @@ int main(void) DEFINE(IRQ_CPUSTAT_SOFTIRQ_PENDING, offsetof(irq_cpustat_t, __softirq_pending)); BLANK(); DEFINE(CPU_BOOT_TASK, offsetof(struct secondary_data, task)); + DEFINE(CPU_BOOT_TTBR1, offsetof(struct secondary_data, ttbr1)); BLANK(); DEFINE(FTR_OVR_VAL_OFFSET, offsetof(struct arm64_ftr_override, val)); DEFINE(FTR_OVR_MASK_OFFSET, offsetof(struct arm64_ftr_override, mask)); diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S index e92caebff46a..e66ee578f755 100644 --- a/arch/arm64/kernel/head.S +++ b/arch/arm64/kernel/head.S @@ -646,7 +646,8 @@ SYM_FUNC_START_LOCAL(secondary_startup) ldr_l x0, vabits_actual #endif bl __cpu_setup // initialise processor - adrp x1, swapper_pg_dir + adr_l x1, secondary_data + ldr x1, [x1, #CPU_BOOT_TTBR1] adrp x2, idmap_pg_dir bl __enable_mmu ldr x8, =__secondary_switched diff --git a/arch/arm64/kernel/smp.c b/arch/arm64/kernel/smp.c index d00d4cbb31b1..8b4cd4924abf 100644 --- a/arch/arm64/kernel/smp.c +++ b/arch/arm64/kernel/smp.c @@ -119,6 +119,9 @@ int __cpu_up(unsigned int cpu, struct task_struct *idle) * page tables. */ secondary_data.task = idle; + secondary_data.ttbr1 = __swapper_pg_dir_node_phys(cpu_to_node(cpu)); + dcache_clean_poc((uintptr_t)&secondary_data, + (uintptr_t)&secondary_data + sizeof(secondary_data)); update_cpu_boot_status(CPU_MMU_OFF); /* Now bring the CPU into our world */ From patchwork Tue May 30 14:05:36 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Russell King (Oracle)" X-Patchwork-Id: 13260155 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 0534EC77B7A for ; Tue, 30 May 2023 14:06:24 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Date:Message-Id:MIME-Version:Subject:Cc :To:From:References:In-Reply-To:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=AsN6fshesEJP4ZQ8m3XXn2kLsohzrJ0epYNs2YitoYI=; b=L7Nbz0nujn1Krf /D1tgp7R1fgaZ5K12GBxBKKPF9KRwyBEuo6cSY13BT8R0A7mFWzC+8jwUG1n/oSa+eEmuypQQ4hLW rfVGPo9nIM8SwyDx2Lr7/wyxtzd07fJsTivhZBYev/sITTs0jT1rciE6OZ57BJRBzegZHU6WNMsSB 7gXN8R6m/LbqITMfzLJpZqvRSNCfiRPCVEfxPJpQ2bBNLbLSzR9xKrf2LUp9jOZUFDqZdWScdwyGK J6+kML0gk3wvIvT5KlinacfP8eCRwIVmZlkoy6ovJG15ifDkPAFKV8v7MA8Day9n1kqGRGH+j3V9K LgkrAqFrGgQNO6FJLmNQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1q3zz6-00E9iY-02; Tue, 30 May 2023 14:06:00 +0000 Received: from pandora.armlinux.org.uk ([2001:4d48:ad52:32c8:5054:ff:fe00:142]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1q3zyk-00E9Xb-1n for linux-arm-kernel@lists.infradead.org; Tue, 30 May 2023 14:05:39 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=armlinux.org.uk; s=pandora-2019; h=Date:Sender:Message-Id:Content-Type: Content-Transfer-Encoding:MIME-Version:Subject:Cc:To:From:References: In-Reply-To:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help: List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=ygnjEx4v2MVO+CzvSSoDe24Bab7+pDTKqVVz0AOs2qk=; b=1BpMFUySZ4yqxhdlJ29sA4+s9l GF3ArZWXyOK2dfbEN10nrVpCEluzqroG9fCPR9COYdCPsjTl2ChU4IVhFIfmOEEX37+W3AnV6hrXo 6o7oVOQKGLS2wwWbXpSrbRQUNq/nTkbtJUi5hpkXNom97w8ijiCNKrw15gjXsMmn0BJKaTmsuF8BA r2jnDrj/ik5MtU0EXaUs0fZ4GPhI829CGvnCnJIJASZcPMEQ+jpoDqVMKFaU2hd0G8uDH8U1BXREb BpACSYTFTA+Xn1yCfwAuQh+Kayf67B4tfTX5yUfybpKYPSUhrNrqAKoMai2nwmw0lwSVgtNvVnf+P D0sFsCbQ==; Received: from e0022681537dd.dyn.armlinux.org.uk ([fd8f:7570:feb6:1:222:68ff:fe15:37dd]:48782 helo=rmk-PC.armlinux.org.uk) by pandora.armlinux.org.uk with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1q3zyj-0002sV-71; Tue, 30 May 2023 15:05:37 +0100 Received: from rmk by rmk-PC.armlinux.org.uk with local (Exim 4.94.2) (envelope-from ) id 1q3zyi-00A5Lg-Kq; Tue, 30 May 2023 15:05:36 +0100 In-Reply-To: References: From: "Russell King (Oracle)" To: Catalin Marinas , Jonathan Corbet , Will Deacon Cc: linux-arm-kernel@lists.infradead.org, linux-doc@vger.kernel.org Subject: [PATCH RFC 13/17] arm64: text replication: update cnp support MIME-Version: 1.0 Content-Disposition: inline Message-Id: Date: Tue, 30 May 2023 15:05:36 +0100 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230530_070538_595287_256F3C4E X-CRM114-Status: GOOD ( 13.18 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Add changes for CNP (Common Not Private) support of kernel text replication. Although text replication has only been tested on dual-socket Ampere A1 systems, provided the different NUMA nodes are not part of the same inner shareable domain, CNP should not be a problem. Signed-off-by: Russell King (Oracle) --- arch/arm64/include/asm/mmu_context.h | 2 +- arch/arm64/kernel/cpufeature.c | 2 +- arch/arm64/kernel/suspend.c | 3 ++- 3 files changed, 4 insertions(+), 3 deletions(-) diff --git a/arch/arm64/include/asm/mmu_context.h b/arch/arm64/include/asm/mmu_context.h index 4aa6afc6f935..c0e13dc73fd7 100644 --- a/arch/arm64/include/asm/mmu_context.h +++ b/arch/arm64/include/asm/mmu_context.h @@ -159,7 +159,7 @@ static inline void cpu_replace_ttbr1_phys(phys_addr_t pgd_phys, pgd_t *idmap) phys_addr_t ttbr1 = phys_to_ttbr(pgd_phys); if (system_supports_cnp() && - !WARN_ON(pgd_phys != virt_to_phys(lm_alias(swapper_pg_dir)))) { + !WARN_ON(pgd_phys != swapper_pg_dir_node_phys())) { /* * cpu_replace_ttbr1() is used when there's a boot CPU * up (i.e. cpufeature framework is not up yet) and diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c index 7d7128c65161..fdb93b5b7d8e 100644 --- a/arch/arm64/kernel/cpufeature.c +++ b/arch/arm64/kernel/cpufeature.c @@ -3341,7 +3341,7 @@ subsys_initcall_sync(init_32bit_el0_mask); static void __maybe_unused cpu_enable_cnp(struct arm64_cpu_capabilities const *cap) { - cpu_replace_ttbr1(lm_alias(swapper_pg_dir), idmap_pg_dir); + cpu_replace_ttbr1_phys(swapper_pg_dir_node_phys(), idmap_pg_dir); } /* diff --git a/arch/arm64/kernel/suspend.c b/arch/arm64/kernel/suspend.c index 0fbdf5fe64d8..49fa80bafd6d 100644 --- a/arch/arm64/kernel/suspend.c +++ b/arch/arm64/kernel/suspend.c @@ -55,7 +55,8 @@ void notrace __cpu_suspend_exit(void) /* Restore CnP bit in TTBR1_EL1 */ if (system_supports_cnp()) - cpu_replace_ttbr1(lm_alias(swapper_pg_dir), idmap_pg_dir); + cpu_replace_ttbr1_phys(swapper_pg_dir_node_phys(), + idmap_pg_dir); /* * PSTATE was not saved over suspend/resume, re-enable any detected From patchwork Tue May 30 14:05:41 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Russell King (Oracle)" X-Patchwork-Id: 13260151 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 0CB82C77B7A for ; Tue, 30 May 2023 14:06:21 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Date:Message-Id:MIME-Version:Subject:Cc :To:From:References:In-Reply-To:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=wkyFbY4lI4WhdIvcocS/SdktFVYgDGJYHDLkqtVdB5k=; b=QVbh4x7rOJgbud JRRaNR4zQ4OH+I3hpmMRCM2Hb0cAtP5ogH3wBn2DCadE3fZ1EIqRvlTOMcmW8ZWiaPzaZ8YCpBfRp mB3NykDp2+ZTPQJMw/WT2KPQceYjbiRGVonm4FLan6WJt2m/gc9JreAWTXBjkCQXMMtSZw4yScpZv 71zG5IaLa/glm9iTD57MbdlQbtxxUSuqKIc1SYHZ3ZSDMwGrmCPESmbDN7jk/YtScKmFZZFFaRgh8 G6An5RzO1h6C6exlLzHXDjOna6nnkiNIA5rYGeJyukKBPB2z3rQOhrg66DVO5pu3Wd+Yt+HXBe0Tm r+x9AA1wWMWrzkTxBR8Q==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1q3zz6-00E9jC-2N; Tue, 30 May 2023 14:06:00 +0000 Received: from pandora.armlinux.org.uk ([2001:4d48:ad52:32c8:5054:ff:fe00:142]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1q3zyp-00E9aJ-1w for linux-arm-kernel@lists.infradead.org; Tue, 30 May 2023 14:05:45 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=armlinux.org.uk; s=pandora-2019; h=Date:Sender:Message-Id:Content-Type: Content-Transfer-Encoding:MIME-Version:Subject:Cc:To:From:References: In-Reply-To:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help: List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=uWU3PWV3b+t6w/ma7OF0J+w5sjCRGf6skcM5pn98FwY=; b=aDKKQnRQ8+ZJpWlj2ZqsjqiADf /eqLsyw0VAjIrrvfsSfqNhXVtyzVkcgTmzKnTeV7htfhB3M3UiDOgsv7Bxil36tDWmBu7FptwGgzX mhGcHVX0U3xfZliQq+dDDSzLsN/9CX+pZG6bvGSblVf2p1oDY0RNnIvOpRazl3TzpUT2QW9rT+9JP 7gIZaqybUSL56uMlaz7qLHIzGKRS5OYd19z1ErR28qiNTvI0KOfcnCSlUP3G31haGQmvUe0N72gFZ uR7UNp458aMem9GAL+6YrZkGvKbKlUvGHyHjM8LIkT0WBVgQNHPN1n0W50utfdgH/g2oU9lw1h/mi YI1aMYnQ==; Received: from e0022681537dd.dyn.armlinux.org.uk ([fd8f:7570:feb6:1:222:68ff:fe15:37dd]:48796 helo=rmk-PC.armlinux.org.uk) by pandora.armlinux.org.uk with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1q3zyo-0002sf-9c; Tue, 30 May 2023 15:05:42 +0100 Received: from rmk by rmk-PC.armlinux.org.uk with local (Exim 4.94.2) (envelope-from ) id 1q3zyn-00A5Lo-Nm; Tue, 30 May 2023 15:05:41 +0100 In-Reply-To: References: From: "Russell King (Oracle)" To: Catalin Marinas , Jonathan Corbet , Will Deacon Cc: linux-arm-kernel@lists.infradead.org, linux-doc@vger.kernel.org Subject: [PATCH RFC 14/17] arm64: text replication: setup page tables for copied kernel MIME-Version: 1.0 Content-Disposition: inline Message-Id: Date: Tue, 30 May 2023 15:05:41 +0100 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230530_070543_640390_3D395518 X-CRM114-Status: GOOD ( 14.63 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Setup page table entries in each non-boot NUMA node page table to point at each node's own copy of the kernel text. This switches each node to use its own unique copy of the kernel text. Signed-off-by: Russell King (Oracle) --- arch/arm64/include/asm/ktext.h | 1 + arch/arm64/mm/ktext.c | 8 +++++ arch/arm64/mm/mmu.c | 53 ++++++++++++++++++++++++++++------ 3 files changed, 53 insertions(+), 9 deletions(-) diff --git a/arch/arm64/include/asm/ktext.h b/arch/arm64/include/asm/ktext.h index 386f9812d3c1..6ece59ca90a2 100644 --- a/arch/arm64/include/asm/ktext.h +++ b/arch/arm64/include/asm/ktext.h @@ -16,6 +16,7 @@ void __kprobes ktext_replication_patch(u32 *tp, __le32 insn); void ktext_replication_patch_alternative(__le32 *src, int nr_inst); void ktext_replication_set_swapper_pgd(pgd_t *pgdp, pgd_t pgd); void ktext_replication_init_tramp(void); +void create_kernel_nid_map(pgd_t *pgdp, void *ktext); #else diff --git a/arch/arm64/mm/ktext.c b/arch/arm64/mm/ktext.c index 290012d2bd03..11eba88fdd49 100644 --- a/arch/arm64/mm/ktext.c +++ b/arch/arm64/mm/ktext.c @@ -136,6 +136,14 @@ void __init ktext_replication_init(void) /* Copy initial swapper page directory */ memcpy(pgtables[nid]->swapper_pg_dir, swapper_pg_dir, PGD_SIZE); + + /* Clear the kernel mapping */ + memset(&pgtables[nid]->swapper_pg_dir[kidx], 0, + sizeof(pgtables[nid]->swapper_pg_dir[kidx])); + + /* Create kernel mapping pointing at our local copy */ + create_kernel_nid_map(pgtables[nid]->swapper_pg_dir, + kernel_texts[nid]); } } diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index 12fc3b1116e6..2ba5cdfa28ce 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -641,6 +641,16 @@ void mark_rodata_ro(void) debug_checkwx(); } +static void __init create_kernel_mapping(pgd_t *pgdp, phys_addr_t pa_start, + void *va_start, void *va_end, + pgprot_t prot, int flags) +{ + size_t size = va_end - va_start; + + __create_pgd_mapping(pgdp, pa_start, (unsigned long)va_start, size, + prot, early_pgtable_alloc, flags); +} + static void __init map_kernel_segment(pgd_t *pgdp, void *va_start, void *va_end, pgprot_t prot, struct vm_struct *vma, int flags, unsigned long vm_flags) @@ -651,8 +661,7 @@ static void __init map_kernel_segment(pgd_t *pgdp, void *va_start, void *va_end, BUG_ON(!PAGE_ALIGNED(pa_start)); BUG_ON(!PAGE_ALIGNED(size)); - __create_pgd_mapping(pgdp, pa_start, (unsigned long)va_start, size, prot, - early_pgtable_alloc, flags); + create_kernel_mapping(pgdp, pa_start, va_start, va_end, prot, flags); if (!(vm_flags & VM_NO_GUARD)) size += PAGE_SIZE; @@ -721,14 +730,8 @@ static bool arm64_early_this_cpu_has_bti(void) ID_AA64PFR1_EL1_BT_SHIFT); } -/* - * Create fine-grained mappings for the kernel. - */ -static void __init map_kernel(pgd_t *pgdp) +static pgprot_t __init kernel_text_pgprot(void) { - static struct vm_struct vmlinux_text, vmlinux_rodata, vmlinux_inittext, - vmlinux_initdata, vmlinux_data; - /* * External debuggers may need to write directly to the text * mapping to install SW breakpoints. Allow this (only) when @@ -744,6 +747,38 @@ static void __init map_kernel(pgd_t *pgdp) if (arm64_early_this_cpu_has_bti()) text_prot = __pgprot_modify(text_prot, PTE_GP, PTE_GP); + return text_prot; +} + +#ifdef CONFIG_REPLICATE_KTEXT +void __init create_kernel_nid_map(pgd_t *pgdp, void *ktext) +{ + pgprot_t text_prot = kernel_text_pgprot(); + + create_kernel_mapping(pgdp, __pa(ktext), _stext, _etext, text_prot, 0); + create_kernel_mapping(pgdp, __pa_symbol(__start_rodata), + __start_rodata, __inittext_begin, + PAGE_KERNEL, NO_CONT_MAPPINGS); + create_kernel_mapping(pgdp, __pa_symbol(__inittext_begin), + __inittext_begin, __inittext_end, + text_prot, 0); + create_kernel_mapping(pgdp, __pa_symbol(__initdata_begin), + __initdata_begin, __initdata_end, + PAGE_KERNEL, 0); + create_kernel_mapping(pgdp, __pa_symbol(_data), _data, _end, + PAGE_KERNEL, 0); +} +#endif + +/* + * Create fine-grained mappings for the kernel. + */ +static void __init map_kernel(pgd_t *pgdp) +{ + static struct vm_struct vmlinux_text, vmlinux_rodata, vmlinux_inittext, + vmlinux_initdata, vmlinux_data; + pgprot_t text_prot = kernel_text_pgprot(); + /* * Only rodata will be remapped with different permissions later on, * all other segments are allowed to use contiguous mappings. From patchwork Tue May 30 14:05:46 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Russell King (Oracle)" X-Patchwork-Id: 13260156 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 2E4C3C7EE23 for ; Tue, 30 May 2023 14:06:27 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Date:Message-Id:MIME-Version:Subject:Cc :To:From:References:In-Reply-To:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=HcE+XS3zfIodd4DQeeiYFVRcs4PdFF8+ypelPoA4r5w=; b=26svy38CD2axxl yeIY4JYhtKUFofsCv+2iQtMT1o4URZSJCvQ5o63TmSiE/K1tu6unjF4OeU7PISOuEtTt6kDQYNBF1 tpNoONgkaZLuIdp0pn1jmcjyr32siIIudloMbfBLEe168Pdm/sJcNVUgo91SoQvJzf8SXS74dPMaa Bc7ga1Ja9GcRoHVRIUGYNx5HXxi15hv/5JY4YaDbNlUSYCaYh2lYLFZJvCRdQwaDO/Wyk5XHWJjtY yjF5SnjIEaIXcqNkLJoFLbZ983oBk1f0dZ2Wx4Qg8ebn2TmbvmcGs+mEAYGCfUX1hdN4fKSGjaani r0/cU/bSkJXn9tAlrIOg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1q3zz7-00E9jy-1f; Tue, 30 May 2023 14:06:01 +0000 Received: from pandora.armlinux.org.uk ([2001:4d48:ad52:32c8:5054:ff:fe00:142]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1q3zyu-00E9cE-2X for linux-arm-kernel@lists.infradead.org; Tue, 30 May 2023 14:05:50 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=armlinux.org.uk; s=pandora-2019; h=Date:Sender:Message-Id:Content-Type: Content-Transfer-Encoding:MIME-Version:Subject:Cc:To:From:References: In-Reply-To:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help: List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=j73sK6spBSnIr1eNE6vGhJALNwM1f6fBM4G1hUkLttA=; b=unJutXRiCGHwGOdig5Y2+g+y4u cvzjOTL9NH/I5W6AWOt955Qzq8ypMcoEx4IXpoklX9piKIJXI6LbZ03RHWCiE2R4GkpSEoo0W1h3X UV6yhD8S78OkfEkLWm58o6rVV0tuoiIlztYe9bOGGcbwoJQ8kBxOA1dNIFx/bW/aUNEz9kxrRG+UM LK9fD/qX1I2S2JD/sPuXC2wl/kb7zzWFItH1hQrSkg+SEXmZ0Mu2NQlrN6D6NcO0cVZ48Kz3mU6Ee zinw5X2ub0Kz7CCdpxLBZhvC44T04cNSVHXR//ZYCBrBLE5qNQQbXhRzaft+0QorcFK8nNEUBSJnJ BjFMPWWw==; Received: from e0022681537dd.dyn.armlinux.org.uk ([fd8f:7570:feb6:1:222:68ff:fe15:37dd]:48868 helo=rmk-PC.armlinux.org.uk) by pandora.armlinux.org.uk with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1q3zyt-0002sp-CG; Tue, 30 May 2023 15:05:47 +0100 Received: from rmk by rmk-PC.armlinux.org.uk with local (Exim 4.94.2) (envelope-from ) id 1q3zys-00A5Lw-QO; Tue, 30 May 2023 15:05:46 +0100 In-Reply-To: References: From: "Russell King (Oracle)" To: Catalin Marinas , Jonathan Corbet , Will Deacon Cc: linux-arm-kernel@lists.infradead.org, linux-doc@vger.kernel.org Subject: [PATCH RFC 15/17] arm64: text replication: include most of read-only data as well MIME-Version: 1.0 Content-Disposition: inline Message-Id: Date: Tue, 30 May 2023 15:05:46 +0100 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230530_070548_823110_DE066C48 X-CRM114-Status: GOOD ( 15.24 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Include as much of the read-only data in the replication as we can without needing to move away from the generic RO_DATA() macro in the linker script. Unfortunately, the read-only data section is immedaitely followed by the read-only after init data with no page alignment, which means we can't have separate mappings for the read-only data section and everything else. Changing that would mean replacing the generic RO_DATA() macro which increases the maintenance burden. however, this is likely not worth the effort as the majority of read-only data will be covered. Signed-off-by: Russell King (Oracle) --- arch/arm64/mm/ktext.c | 2 +- arch/arm64/mm/mmu.c | 21 ++++++++++++++++++--- 2 files changed, 19 insertions(+), 4 deletions(-) diff --git a/arch/arm64/mm/ktext.c b/arch/arm64/mm/ktext.c index 11eba88fdd49..f64a649f06a4 100644 --- a/arch/arm64/mm/ktext.c +++ b/arch/arm64/mm/ktext.c @@ -101,7 +101,7 @@ void ktext_replication_patch_alternative(__le32 *src, int nr_inst) /* Allocate page tables and memory for the replicated kernel texts. */ void __init ktext_replication_init(void) { - size_t size = _etext - _stext; + size_t size = __end_rodata - _stext; int kidx = pgd_index((phys_addr_t)KERNEL_START); int nid; diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index 2ba5cdfa28ce..5e4cb28f3e5f 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -753,11 +753,26 @@ static pgprot_t __init kernel_text_pgprot(void) #ifdef CONFIG_REPLICATE_KTEXT void __init create_kernel_nid_map(pgd_t *pgdp, void *ktext) { + phys_addr_t pa_ktext; + size_t ro_offset; + void *ro_end; pgprot_t text_prot = kernel_text_pgprot(); - create_kernel_mapping(pgdp, __pa(ktext), _stext, _etext, text_prot, 0); - create_kernel_mapping(pgdp, __pa_symbol(__start_rodata), - __start_rodata, __inittext_begin, + pa_ktext = __pa(ktext); + ro_offset = __pa_symbol(__start_rodata) - __pa_symbol(_stext); + /* + * We must not cover the read-only data after init, since this + * is written to during boot, and thus must be shared between + * the NUMA nodes. + */ + ro_end = PTR_ALIGN_DOWN((void *)__start_ro_after_init, PAGE_SIZE); + + create_kernel_mapping(pgdp, pa_ktext, _stext, _etext, text_prot, 0); + create_kernel_mapping(pgdp, pa_ktext + ro_offset, + __start_rodata, ro_end, + PAGE_KERNEL, NO_CONT_MAPPINGS); + create_kernel_mapping(pgdp, __pa_symbol(ro_end), + ro_end, __inittext_begin, PAGE_KERNEL, NO_CONT_MAPPINGS); create_kernel_mapping(pgdp, __pa_symbol(__inittext_begin), __inittext_begin, __inittext_end, From patchwork Tue May 30 14:05:51 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Russell King (Oracle)" X-Patchwork-Id: 13260154 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id C9ED7C77B73 for ; Tue, 30 May 2023 14:06:22 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Date:Message-Id:MIME-Version:Subject:Cc :To:From:References:In-Reply-To:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=Oykcwwho2s2oHfb/rr47cZ6gqOU8HTf8BLNaxzaHq5w=; b=QG+VNfPlM+rJOI Rov6R+9qXKVUYVfcCDM4ZGxgOssc9EI/5Mq+UM28zPm0EmJofggQE54vaIWuxEbNF9OvHFwncz3XO EfLNr7lSStokFpHSnKkj0LcGQvBOcxmD4sxFp9lVBT6cEfkxWezPc4RE9KH0+YbMcoTa1vsDBpbbC XQimFhtTLo4pL4nblos/jcEXh3S5yPedmilVcoU1PVmi8s3WUQ7VrvKYG4a5eg2G2CJnK6qavotp1 W1dBM0N79E3ot65vWjQBx426DwCF0ltjqegaZRSl0EiP09JjWlu5eVKezetR+n8TeQF3limV1w0l6 1S2GmzOMGshY/QAorq7g==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1q3zz8-00E9kt-0m; Tue, 30 May 2023 14:06:02 +0000 Received: from pandora.armlinux.org.uk ([2001:4d48:ad52:32c8:5054:ff:fe00:142]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1q3zyz-00E9e3-2e for linux-arm-kernel@lists.infradead.org; Tue, 30 May 2023 14:05:55 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=armlinux.org.uk; s=pandora-2019; h=Date:Sender:Message-Id:Content-Type: Content-Transfer-Encoding:MIME-Version:Subject:Cc:To:From:References: In-Reply-To:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help: List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=d/FJ8eAsQuybHTDjCAtIMygFgqbgutZK8wvxGecc0vQ=; b=vXKqqJoAV6hypA+7bUWY/j34MR tq8rAU0EAjd502zkD/WmKjArkbsPbRC5dg694MFyguKR1P8VpTCTFs7XDsGvXN+ANfDCXjD1aqCAf +K8VKyTcoV8AcF6WAxG6x4eoFetIIa4kGzlJwYzFx/YSMYu86pd6pnVOFRJbUZPDQxTCueUv1TUK5 Rfkply6JpjF22IsYVt8PCXNXNJWLkPSId9Gq5VUfHZiiyUo/Gdw4YdstfpJNuLGRxWPtRWSgRfS13 ugMvTQsaz/9T48pDCrCXSxuz5FZXkqCvMXqtu0CXxVchuQJ1QDcMrDjkKFOkJYKKojsmHRaMT5DvK BrCAOpjA==; Received: from e0022681537dd.dyn.armlinux.org.uk ([fd8f:7570:feb6:1:222:68ff:fe15:37dd]:48884 helo=rmk-PC.armlinux.org.uk) by pandora.armlinux.org.uk with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1q3zyy-0002sz-GJ; Tue, 30 May 2023 15:05:52 +0100 Received: from rmk by rmk-PC.armlinux.org.uk with local (Exim 4.94.2) (envelope-from ) id 1q3zyx-00A5M2-TK; Tue, 30 May 2023 15:05:51 +0100 In-Reply-To: References: From: "Russell King (Oracle)" To: Catalin Marinas , Jonathan Corbet , Will Deacon Cc: linux-arm-kernel@lists.infradead.org, linux-doc@vger.kernel.org Subject: [PATCH RFC 16/17] arm64: text replication: early kernel option to enable replication MIME-Version: 1.0 Content-Disposition: inline Message-Id: Date: Tue, 30 May 2023 15:05:51 +0100 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230530_070554_243100_18A8634E X-CRM114-Status: GOOD ( 14.02 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Provide an early kernel option "ktext=" which allows the kernel text replication to be enabled. This takes a boolean argument. The way this has been implemented means that we take all the same paths through the kernel at runtime whether kernel text replication has been enabled or not; this allows the performance effects of the code changes to be evaluated separately from the act of running with replicating the kernel text. Signed-off-by: Russell King (Oracle) --- .../admin-guide/kernel-parameters.txt | 5 +++++ arch/arm64/mm/ktext.c | 18 ++++++++++++++++++ 2 files changed, 23 insertions(+) diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt index 9e5bab29685f..684bd004816e 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -2488,6 +2488,11 @@ 0: force disabled 1: force enabled + ktext= [ARM64] Control kernel text replication on NUMA + machines. Default: disabled. + 0: disable kernel text replication + 1: enable kernel text replication + kunit.enable= [KUNIT] Enable executing KUnit tests. Requires CONFIG_KUNIT to be set to be fully enabled. The default value can be overridden via diff --git a/arch/arm64/mm/ktext.c b/arch/arm64/mm/ktext.c index f64a649f06a4..ec2d474bbd2c 100644 --- a/arch/arm64/mm/ktext.c +++ b/arch/arm64/mm/ktext.c @@ -98,6 +98,21 @@ void ktext_replication_patch_alternative(__le32 *src, int nr_inst) } } +static bool ktext_enabled; + +static int __init parse_ktext(char *str) +{ + bool enabled; + int ret = strtobool(str, &enabled); + + if (ret) + return ret; + + ktext_enabled = enabled; + return 0; +} +early_param("ktext", parse_ktext); + /* Allocate page tables and memory for the replicated kernel texts. */ void __init ktext_replication_init(void) { @@ -119,6 +134,9 @@ void __init ktext_replication_init(void) return; } + if (!ktext_enabled) + return; + for_each_node(nid) { /* Nothing to do for node 0 */ if (!nid) From patchwork Tue May 30 14:05:57 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Russell King (Oracle)" X-Patchwork-Id: 13260157 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 18A51C77B73 for ; Tue, 30 May 2023 14:07:30 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Date:Message-Id:MIME-Version:Subject:Cc :To:From:References:In-Reply-To:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=Svi/XvEjUqJ9EGNKa6imy5lTrPkGiHbgmOxGXlhmOGU=; b=W1yTvRyjMToJ8s P7mgiXFYfLCr4l4sSJn4i3vXBcxYH2Up19HPVqBAiQGT+l9Actj478GBq/aGTPypgB0aJhsj43A7Q unudoIklvRx59FIvJJN3u63RJVf3pYmG47QRMsdtdGoIh7Mkmcm/2LtOeKeE4JexFkPsbxdJwEHXO vhN5itOCJo2FkSdoZCcQMGIw3wgjsQ5LP8WVHWFRRj33CkGSDpU95Qs/hjZRJlH6IZidGhcB+ZFWu ceQG0mz0clY697tNg23Qx9XnS0qCIASw0bXhpb4mBvjIW7zSreyFPTXDKaMTOc5pFvixaEV07HBvT wS6QXoyhHEs260AWxJhA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1q400C-00EAJl-0O; Tue, 30 May 2023 14:07:08 +0000 Received: from pandora.armlinux.org.uk ([2001:4d48:ad52:32c8:5054:ff:fe00:142]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1q3zz5-00E9ga-02 for linux-arm-kernel@lists.infradead.org; Tue, 30 May 2023 14:06:00 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=armlinux.org.uk; s=pandora-2019; h=Date:Sender:Message-Id:Content-Type: Content-Transfer-Encoding:MIME-Version:Subject:Cc:To:From:References: In-Reply-To:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help: List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=OfjrN2mAxyr/FD8uaYqIzLo8HjYikj9c1IOeRy2rdnM=; b=k7VHRF2fVnIQdmm1SN3wDo45Ci jdCwnzNmejTs9iyU2AB+hxqJ8UcV2dXOBDmrQxxoWkjDWLPY0zPXXMKt98lp9bCDml+/JCQIW0YRe alV+MH+UXZxil2+ZuCu2yil7Aip+5FYDHCUc7toTol7xgXL1w2N0JD7fM1eHIaL3L92dC3lEoS/Gi VRkUb9WF1ag6b7gGkb3EZfxZFLI5SXrvvkVv9kZO+F7gib6jxB1Sza3LDjCCgfZrrLcP0+ZVyyvif Hx04qEXBNPj0aWSx6J5/MclbsBlsCWrJb29RJKtIFGIvAqr5WjWfTKiPnfpwNhAz9QUMqA4k9hojo 6GzN3r3Q==; Received: from e0022681537dd.dyn.armlinux.org.uk ([fd8f:7570:feb6:1:222:68ff:fe15:37dd]:35060 helo=rmk-PC.armlinux.org.uk) by pandora.armlinux.org.uk with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1q3zz3-0002t9-Ip; Tue, 30 May 2023 15:05:57 +0100 Received: from rmk by rmk-PC.armlinux.org.uk with local (Exim 4.94.2) (envelope-from ) id 1q3zz3-00A5M8-0M; Tue, 30 May 2023 15:05:57 +0100 In-Reply-To: References: From: "Russell King (Oracle)" To: Catalin Marinas , Jonathan Corbet , Will Deacon Cc: linux-arm-kernel@lists.infradead.org, linux-doc@vger.kernel.org Subject: [PATCH RFC 17/17] arm64: text replication: add Kconfig MIME-Version: 1.0 Content-Disposition: inline Message-Id: Date: Tue, 30 May 2023 15:05:57 +0100 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230530_070559_048489_ABB992E9 X-CRM114-Status: GOOD ( 12.17 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Add the Kconfig symbol for kernel text replication. This unfortunately requires KASAN and kernel text randomisation options to be disabled at the moment. Signed-off-by: Russell King (Oracle) --- arch/arm64/Kconfig | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index b1201d25a8a4..e1120841e26e 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -159,7 +159,7 @@ config ARM64 select HAVE_ARCH_HUGE_VMAP select HAVE_ARCH_JUMP_LABEL select HAVE_ARCH_JUMP_LABEL_RELATIVE - select HAVE_ARCH_KASAN if !(ARM64_16K_PAGES && ARM64_VA_BITS_48) + select HAVE_ARCH_KASAN if !(ARM64_16K_PAGES && ARM64_VA_BITS_48 && !REPLICATE_KTEXT) select HAVE_ARCH_KASAN_VMALLOC if HAVE_ARCH_KASAN select HAVE_ARCH_KASAN_SW_TAGS if HAVE_ARCH_KASAN select HAVE_ARCH_KASAN_HW_TAGS if (HAVE_ARCH_KASAN && ARM64_MTE) @@ -1400,6 +1400,13 @@ config NODES_SHIFT Specify the maximum number of NUMA Nodes available on the target system. Increases memory reserved to accommodate various tables. +config REPLICATE_KTEXT + bool "Replicate kernel text across numa nodes" + depends on NUMA + help + Say Y here to enable replicating the kernel text across multiple + nodes in a NUMA cluster. This trades memory for speed. + source "kernel/Kconfig.hz" config ARCH_SPARSEMEM_ENABLE @@ -2167,6 +2174,7 @@ config RELOCATABLE config RANDOMIZE_BASE bool "Randomize the address of the kernel image" + depends on !REPLICATE_KTEXT select ARM64_MODULE_PLTS if MODULES select RELOCATABLE help