From patchwork Tue May 30 14:05:41 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Russell King (Oracle)" X-Patchwork-Id: 13260151 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 0CB82C77B7A for ; Tue, 30 May 2023 14:06:21 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Date:Message-Id:MIME-Version:Subject:Cc :To:From:References:In-Reply-To:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=wkyFbY4lI4WhdIvcocS/SdktFVYgDGJYHDLkqtVdB5k=; b=QVbh4x7rOJgbud JRRaNR4zQ4OH+I3hpmMRCM2Hb0cAtP5ogH3wBn2DCadE3fZ1EIqRvlTOMcmW8ZWiaPzaZ8YCpBfRp mB3NykDp2+ZTPQJMw/WT2KPQceYjbiRGVonm4FLan6WJt2m/gc9JreAWTXBjkCQXMMtSZw4yScpZv 71zG5IaLa/glm9iTD57MbdlQbtxxUSuqKIc1SYHZ3ZSDMwGrmCPESmbDN7jk/YtScKmFZZFFaRgh8 G6An5RzO1h6C6exlLzHXDjOna6nnkiNIA5rYGeJyukKBPB2z3rQOhrg66DVO5pu3Wd+Yt+HXBe0Tm r+x9AA1wWMWrzkTxBR8Q==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1q3zz6-00E9jC-2N; Tue, 30 May 2023 14:06:00 +0000 Received: from pandora.armlinux.org.uk ([2001:4d48:ad52:32c8:5054:ff:fe00:142]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1q3zyp-00E9aJ-1w for linux-arm-kernel@lists.infradead.org; Tue, 30 May 2023 14:05:45 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=armlinux.org.uk; s=pandora-2019; h=Date:Sender:Message-Id:Content-Type: Content-Transfer-Encoding:MIME-Version:Subject:Cc:To:From:References: In-Reply-To:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help: List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=uWU3PWV3b+t6w/ma7OF0J+w5sjCRGf6skcM5pn98FwY=; b=aDKKQnRQ8+ZJpWlj2ZqsjqiADf /eqLsyw0VAjIrrvfsSfqNhXVtyzVkcgTmzKnTeV7htfhB3M3UiDOgsv7Bxil36tDWmBu7FptwGgzX mhGcHVX0U3xfZliQq+dDDSzLsN/9CX+pZG6bvGSblVf2p1oDY0RNnIvOpRazl3TzpUT2QW9rT+9JP 7gIZaqybUSL56uMlaz7qLHIzGKRS5OYd19z1ErR28qiNTvI0KOfcnCSlUP3G31haGQmvUe0N72gFZ uR7UNp458aMem9GAL+6YrZkGvKbKlUvGHyHjM8LIkT0WBVgQNHPN1n0W50utfdgH/g2oU9lw1h/mi YI1aMYnQ==; Received: from e0022681537dd.dyn.armlinux.org.uk ([fd8f:7570:feb6:1:222:68ff:fe15:37dd]:48796 helo=rmk-PC.armlinux.org.uk) by pandora.armlinux.org.uk with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1q3zyo-0002sf-9c; Tue, 30 May 2023 15:05:42 +0100 Received: from rmk by rmk-PC.armlinux.org.uk with local (Exim 4.94.2) (envelope-from ) id 1q3zyn-00A5Lo-Nm; Tue, 30 May 2023 15:05:41 +0100 In-Reply-To: References: From: "Russell King (Oracle)" To: Catalin Marinas , Jonathan Corbet , Will Deacon Cc: linux-arm-kernel@lists.infradead.org, linux-doc@vger.kernel.org Subject: [PATCH RFC 14/17] arm64: text replication: setup page tables for copied kernel MIME-Version: 1.0 Content-Disposition: inline Message-Id: Date: Tue, 30 May 2023 15:05:41 +0100 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230530_070543_640390_3D395518 X-CRM114-Status: GOOD ( 14.63 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Setup page table entries in each non-boot NUMA node page table to point at each node's own copy of the kernel text. This switches each node to use its own unique copy of the kernel text. Signed-off-by: Russell King (Oracle) --- arch/arm64/include/asm/ktext.h | 1 + arch/arm64/mm/ktext.c | 8 +++++ arch/arm64/mm/mmu.c | 53 ++++++++++++++++++++++++++++------ 3 files changed, 53 insertions(+), 9 deletions(-) diff --git a/arch/arm64/include/asm/ktext.h b/arch/arm64/include/asm/ktext.h index 386f9812d3c1..6ece59ca90a2 100644 --- a/arch/arm64/include/asm/ktext.h +++ b/arch/arm64/include/asm/ktext.h @@ -16,6 +16,7 @@ void __kprobes ktext_replication_patch(u32 *tp, __le32 insn); void ktext_replication_patch_alternative(__le32 *src, int nr_inst); void ktext_replication_set_swapper_pgd(pgd_t *pgdp, pgd_t pgd); void ktext_replication_init_tramp(void); +void create_kernel_nid_map(pgd_t *pgdp, void *ktext); #else diff --git a/arch/arm64/mm/ktext.c b/arch/arm64/mm/ktext.c index 290012d2bd03..11eba88fdd49 100644 --- a/arch/arm64/mm/ktext.c +++ b/arch/arm64/mm/ktext.c @@ -136,6 +136,14 @@ void __init ktext_replication_init(void) /* Copy initial swapper page directory */ memcpy(pgtables[nid]->swapper_pg_dir, swapper_pg_dir, PGD_SIZE); + + /* Clear the kernel mapping */ + memset(&pgtables[nid]->swapper_pg_dir[kidx], 0, + sizeof(pgtables[nid]->swapper_pg_dir[kidx])); + + /* Create kernel mapping pointing at our local copy */ + create_kernel_nid_map(pgtables[nid]->swapper_pg_dir, + kernel_texts[nid]); } } diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index 12fc3b1116e6..2ba5cdfa28ce 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -641,6 +641,16 @@ void mark_rodata_ro(void) debug_checkwx(); } +static void __init create_kernel_mapping(pgd_t *pgdp, phys_addr_t pa_start, + void *va_start, void *va_end, + pgprot_t prot, int flags) +{ + size_t size = va_end - va_start; + + __create_pgd_mapping(pgdp, pa_start, (unsigned long)va_start, size, + prot, early_pgtable_alloc, flags); +} + static void __init map_kernel_segment(pgd_t *pgdp, void *va_start, void *va_end, pgprot_t prot, struct vm_struct *vma, int flags, unsigned long vm_flags) @@ -651,8 +661,7 @@ static void __init map_kernel_segment(pgd_t *pgdp, void *va_start, void *va_end, BUG_ON(!PAGE_ALIGNED(pa_start)); BUG_ON(!PAGE_ALIGNED(size)); - __create_pgd_mapping(pgdp, pa_start, (unsigned long)va_start, size, prot, - early_pgtable_alloc, flags); + create_kernel_mapping(pgdp, pa_start, va_start, va_end, prot, flags); if (!(vm_flags & VM_NO_GUARD)) size += PAGE_SIZE; @@ -721,14 +730,8 @@ static bool arm64_early_this_cpu_has_bti(void) ID_AA64PFR1_EL1_BT_SHIFT); } -/* - * Create fine-grained mappings for the kernel. - */ -static void __init map_kernel(pgd_t *pgdp) +static pgprot_t __init kernel_text_pgprot(void) { - static struct vm_struct vmlinux_text, vmlinux_rodata, vmlinux_inittext, - vmlinux_initdata, vmlinux_data; - /* * External debuggers may need to write directly to the text * mapping to install SW breakpoints. Allow this (only) when @@ -744,6 +747,38 @@ static void __init map_kernel(pgd_t *pgdp) if (arm64_early_this_cpu_has_bti()) text_prot = __pgprot_modify(text_prot, PTE_GP, PTE_GP); + return text_prot; +} + +#ifdef CONFIG_REPLICATE_KTEXT +void __init create_kernel_nid_map(pgd_t *pgdp, void *ktext) +{ + pgprot_t text_prot = kernel_text_pgprot(); + + create_kernel_mapping(pgdp, __pa(ktext), _stext, _etext, text_prot, 0); + create_kernel_mapping(pgdp, __pa_symbol(__start_rodata), + __start_rodata, __inittext_begin, + PAGE_KERNEL, NO_CONT_MAPPINGS); + create_kernel_mapping(pgdp, __pa_symbol(__inittext_begin), + __inittext_begin, __inittext_end, + text_prot, 0); + create_kernel_mapping(pgdp, __pa_symbol(__initdata_begin), + __initdata_begin, __initdata_end, + PAGE_KERNEL, 0); + create_kernel_mapping(pgdp, __pa_symbol(_data), _data, _end, + PAGE_KERNEL, 0); +} +#endif + +/* + * Create fine-grained mappings for the kernel. + */ +static void __init map_kernel(pgd_t *pgdp) +{ + static struct vm_struct vmlinux_text, vmlinux_rodata, vmlinux_inittext, + vmlinux_initdata, vmlinux_data; + pgprot_t text_prot = kernel_text_pgprot(); + /* * Only rodata will be remapped with different permissions later on, * all other segments are allowed to use contiguous mappings.