From patchwork Wed Feb 14 12:28:52 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13556408 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id F14DFC48BC1 for ; Wed, 14 Feb 2024 12:30:32 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=oXbXwgMMWmy3SzExzne7NTgntMhA7xhWk6m9L+fpx+Y=; b=P+omMZ2V+Dks8lkFTnWfEd7SXQ 6AOARZZZr7OcMKyxntaEJbtADuBxweG0dhCQUurkJFKNpafXlVHwU/+bwA6TJzCmFiym7ZXj1Tt7c re1lxifbdpBHCfkTJyLJlSHszsMBuuyJ0ojUhQK9pPzbRNJwUk1D/CyecfThC/auyH5TQVl7P6su4 YVJGLunOn7TkF/Vc6qccP1Cge/Iw7hzSsjlhfLX5gYfSrS4WTz5gHs5LVrV+omdbUNUZq3MZ3sUQz RN1HFHLHvKG3qDdo2Jf1eO3En1aunrC4jXFRygklKfbqL+0KSroICVHfOxbrOl16P6FJUfCGJCGrm ouAUEVHA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1raEP7-0000000CoYE-2npJ; Wed, 14 Feb 2024 12:30:21 +0000 Received: from mail-yb1-xb49.google.com ([2607:f8b0:4864:20::b49]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1raEOg-0000000CoGp-0iWY for linux-arm-kernel@lists.infradead.org; Wed, 14 Feb 2024 12:29:55 +0000 Received: by mail-yb1-xb49.google.com with SMTP id 3f1490d57ef6-dc64b659a9cso9348196276.3 for ; Wed, 14 Feb 2024 04:29:53 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1707913792; x=1708518592; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=ssYO1q2/J7UmM6YLohBW2jBC2yELEfuEpqD7FWGV1jE=; b=byJEGW9YUp8YR9Kz62qAgNfv4kHUHwL2kJ8Cp3ZeoNz9QpbyeQBWUU4DjB+tV5HJTB zegwmYM1IvTBuWA1ifrw4pZRQ2FW+cXXNoge3nj0tsTvi3xxVd2zcuYyEjy4uC663AHr gqQqWmddg/EpxxKdt9NnK9Ofxl9ILvtpw4Wo/DuuIAgsvY8jZbj+ssjqV37kwgTegJ3k LNOdayzJOAf4FEPxHffefx+Y+76o1ARERL0gXcEyE9mDcGv//ckp8NXu2xPW22p6sXUh yOiRChiI0eF4usdr4r9JRLDfySDcHha3WomPCtwsL87qAmMc0L+6WVzw2UgrfzLXv4yK 2mcQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707913792; x=1708518592; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=ssYO1q2/J7UmM6YLohBW2jBC2yELEfuEpqD7FWGV1jE=; b=nqRB5/jqHNaT2MRts8LzDAvAAtMkVUnvzFghghGpnCGE5gXbweYaCFqvv430G8Ik/v rbSas0gii0mtQBrJ9GDmTpeHhAjQBeRJfPKHGPAbJ4KLtS3EkGFRAn1Q+fYgtCN9gTl/ 1o5uLKMotJ84qL4UJQuvqVHHk/wj59VsDFnfcndp98nolENz9G3DiQX5qpPqQT8G5WG4 5u/Nvhrw26vOrtOmRvl7l5rIJgvKdY5IQ3GQUK36AyjDMGqs3lwreK/Lp3LLG0GFxHUJ Ek0gqyMG22DW/YQYIVqxFgQZL7shBnOenhJVB2uU3tzFV3AJRJ1SBQULnZMnCkQq+cW4 1Ryw== X-Gm-Message-State: AOJu0YyM3sGcP6krWjVOCOSmmQTNLOflaocn4jvbhbYuC47nZa+CNWEJ IZl0gnEXOgc/b62KZPFRkTHLtsX+MlQT17CSPH3gL6CmnkM79QqO8pg0DgjnTar7uK9cP6aeS5J zBheIu/l5aNDIgynKyMPwNDidbHBsFgSdLNZ4F5rNMJo3UF+Px7KHTG917lU+zKnlEmOVir4/hN Mr8o1ssBBGBXB5cDjXiGp/GNWC1P5ZkPU+ouyYYWji X-Google-Smtp-Source: AGHT+IHk/+Pp1Q+ziAYKMLa2tIErVRB7Fy9TvlM/htup92KPDYP/d6bQe9suWerVbcwNLeio9MwXQ5dt X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:6902:1027:b0:dc2:4ab7:3d89 with SMTP id x7-20020a056902102700b00dc24ab73d89mr567410ybt.1.1707913792387; Wed, 14 Feb 2024 04:29:52 -0800 (PST) Date: Wed, 14 Feb 2024 13:28:52 +0100 In-Reply-To: <20240214122845.2033971-45-ardb+git@google.com> Mime-Version: 1.0 References: <20240214122845.2033971-45-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=3389; i=ardb@kernel.org; h=from:subject; bh=FuEpWiYD4ARO9/4KLGaHk3hLzIqaeMWlfGeGxZwdczc=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIfXMJpYs2/0NFXx3GXf+ERBdMP1pfZ/mQYuJr4tzMxlfO Sv2PX/eUcrCIMbBICumyCIw+++7nacnStU6z5KFmcPKBDKEgYtTACZyeiLDf2/pndI7nTovpFuf uvRjVl/33c+6pj7JD043PbLe9OOx9xZGhic9N6/3ON09VpRRzrr/pkbnDssXZ+d6VBbZqcSWHm7 +wQwA X-Mailer: git-send-email 2.43.0.687.g38aa6559b0-goog Message-ID: <20240214122845.2033971-51-ardb+git@google.com> Subject: [PATCH v8 06/43] arm64: head: Clear BSS and the kernel page tables in one go From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240214_042954_346823_14554C46 X-CRM114-Status: GOOD ( 15.49 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel We will move the CPU feature overrides into BSS in a subsequent patch, and this requires that BSS is zeroed before the feature override detection code runs. So let's map BSS read-write in the ID map, and zero it via this mapping. Since the kernel page tables are right next to it, and also zeroed via the ID map, let's drop the separate clear_page_tables() function, and just zero everything in one go. Signed-off-by: Ard Biesheuvel --- arch/arm64/kernel/head.S | 33 +++++++------------- arch/arm64/kernel/vmlinux.lds.S | 3 ++ 2 files changed, 14 insertions(+), 22 deletions(-) diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S index ca5e5fbefcd3..2af518161f3a 100644 --- a/arch/arm64/kernel/head.S +++ b/arch/arm64/kernel/head.S @@ -177,17 +177,6 @@ SYM_CODE_START_LOCAL(preserve_boot_args) ret SYM_CODE_END(preserve_boot_args) -SYM_FUNC_START_LOCAL(clear_page_tables) - /* - * Clear the init page tables. - */ - adrp x0, init_pg_dir - adrp x1, init_pg_end - sub x2, x1, x0 - mov x1, xzr - b __pi_memset // tail call -SYM_FUNC_END(clear_page_tables) - /* * Macro to populate page table entries, these entries can be pointers to the next level * or last level entries pointing to physical memory. @@ -386,9 +375,9 @@ SYM_FUNC_START_LOCAL(create_idmap) map_memory x0, x1, x3, x6, x7, x3, IDMAP_PGD_ORDER, x10, x11, x12, x13, x14, EXTRA_SHIFT - /* Remap the kernel page tables r/w in the ID map */ + /* Remap BSS and the kernel page tables r/w in the ID map */ adrp x1, _text - adrp x2, init_pg_dir + adrp x2, __bss_start adrp x3, _end bic x4, x2, #SWAPPER_BLOCK_SIZE - 1 mov_q x5, SWAPPER_RW_MMUFLAGS @@ -489,14 +478,6 @@ SYM_FUNC_START_LOCAL(__primary_switched) mov x0, x20 bl set_cpu_boot_mode_flag - // Clear BSS - adr_l x0, __bss_start - mov x1, xzr - adr_l x2, __bss_stop - sub x2, x2, x0 - bl __pi_memset - dsb ishst // Make zero page visible to PTW - #if VA_BITS > 48 adr_l x8, vabits_actual // Set this early so KASAN early init str x25, [x8] // ... observes the correct value @@ -782,6 +763,15 @@ SYM_FUNC_START_LOCAL(__primary_switch) adrp x1, reserved_pg_dir adrp x2, init_idmap_pg_dir bl __enable_mmu + + // Clear BSS + adrp x0, __bss_start + mov x1, xzr + adrp x2, init_pg_end + sub x2, x2, x0 + bl __pi_memset + dsb ishst // Make zero page visible to PTW + #ifdef CONFIG_RELOCATABLE adrp x23, KERNEL_START and x23, x23, MIN_KIMG_ALIGN - 1 @@ -796,7 +786,6 @@ SYM_FUNC_START_LOCAL(__primary_switch) orr x23, x23, x0 // record kernel offset #endif #endif - bl clear_page_tables bl create_kernel_mapping adrp x1, init_pg_dir diff --git a/arch/arm64/kernel/vmlinux.lds.S b/arch/arm64/kernel/vmlinux.lds.S index 8dd5dda66f7c..8a3c6aacc355 100644 --- a/arch/arm64/kernel/vmlinux.lds.S +++ b/arch/arm64/kernel/vmlinux.lds.S @@ -311,12 +311,15 @@ SECTIONS __pecoff_data_rawsize = ABSOLUTE(. - __initdata_begin); _edata = .; + /* start of zero-init region */ BSS_SECTION(SBSS_ALIGN, 0, 0) . = ALIGN(PAGE_SIZE); init_pg_dir = .; . += INIT_DIR_SIZE; init_pg_end = .; + /* end of zero-init region */ + #ifdef CONFIG_RELOCATABLE . += SZ_4K; /* stack for the early relocation code */ early_init_stack = .;