From patchwork Tue Jan 23 14:53:28 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13527548 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 69853C47DDB for ; Tue, 23 Jan 2024 14:57:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=7HeIjaWK3zsiBGorH4z9vg+Gt72XnHuyfTK/kpcym6M=; b=acx/yxvdILdm9wpDvhnCKypCaL DHXJjnA36VpzXMaOVODv2sJLcMAu5D5z7OGBfF4zlQxiN+dJDl1mgmJEleJfsh7LJVmGfpqkty+P0 nYwqXcm17+lT5gqrZvh4tvaTzv4OksqPsUpr/ebvsXff+ogoPE+IFXqQe9MJZyGV74VNiAaGAndCe Jun3boFtYCiuCEczPPcoxltkXnl13AJA0nEXTfLHcQWjew67xav6vsa4UTQyS+lebNtZVH6ET0JTq CV3qdmC1UdPZbCoJ062eqcGs0/3NgO4GCfXxjyocSHiq5sOYT01MZ7lHC3/YIHBU8mHh1/PqMAgWN QfcvYuDA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rSICU-00GufC-3B; Tue, 23 Jan 2024 14:56:30 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rSIBd-00GuF5-1g for linux-arm-kernel@bombadil.infradead.org; Tue, 23 Jan 2024 14:55:37 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=H1iOnsZsuoVNv+q6thaoOxA5ijothGdYrW/92/IhhbM=; b=klzFvMvloVKInN6PRE6IXmFdk1 guPlfjRZIdsrFoEJ7OVWDBGTtLXyUhOqIElp8V3PlRekUcnrrIROgREcgUTLG/7IwOWHtEx6HLWv4 xWqDJv0dbn0UvPZTC5e65LuT+hRBjXuqZMS6J3PCdthyUtCtLsiv3N8vjA3hQpN/cYhtcj34964XK 1M8oSdocQY224qEvzm41U40X8gzGud6LYDX+xy/j9l5FfvBnWy0tyZEyfzoqwO7ArayKOCVZO+YyF xJxMtMCB6obXIW4ggPj6v4zyM3Z+dlC1cpB5bODDP2/YmhsixFAaova0Kgdpbk8VBknx3fMUY1Izn x5VIvy6Q==; Received: from mail-yb1-xb49.google.com ([2607:f8b0:4864:20::b49]) by desiato.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rSIBY-00000003vS1-1d3d for linux-arm-kernel@lists.infradead.org; Tue, 23 Jan 2024 14:55:36 +0000 Received: by mail-yb1-xb49.google.com with SMTP id 3f1490d57ef6-dc251408f56so5313198276.3 for ; Tue, 23 Jan 2024 06:55:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1706021730; x=1706626530; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=H1iOnsZsuoVNv+q6thaoOxA5ijothGdYrW/92/IhhbM=; b=1v71dm9Gwy3Lvcwe/tA29FwciEfiOF5np5Nl6BeriJDOCGK2kS+zXpG6mWz7qPN69t vnH1Bzz70j4asH5x+bc6FNvN6PY42RJ4HCK8QaGccYZx5lV3CMni8+dTQG8qK0MA5dph OlAJN94EUmd+efx6/SfPvvPAJHKc3PuIDNuU/5mWTRcFtRTWxbFqKDNHLwZk79c+M6GH GViOaLBArl0lgCyw4tCvaGQpFQiW5mIFmFFev7nUnu3AEeZNzfKJDTPtSPySQ3zDiKKW LWfQT382k+uv8CqOxe/CKlx6Nu3MjuU2aUJme6dbAhjp2QtmHRzsOLYnyE11PDqIKRxC tyNg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706021730; x=1706626530; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=H1iOnsZsuoVNv+q6thaoOxA5ijothGdYrW/92/IhhbM=; b=OQQdRmZ+vwhjTRB68pRfPksJImYlz9WGUn0vgAV9PIIcioBg182jnp6mytpUdj4fsa 7a4OIAte8pzloCQhQt2JE9BbsnZfW5FnWtGzlG3lxr1Ho3cBarKBiesHltOTilA2jcRG fiGPZxMGVipPkjyLDUS40dANfpfZDJ7i9IMeD6SzcMG6O/EykiZtJSPzSD3Gbliq/O+Y SW1fgx+6J3rXUWx2tY9DrYEeSw14lFLBwUqcCv1dLzHTQLaESCuUEYZHx7g3wR6he14k Mol/IavndAUDM4lndUVjIfBVfymgD1C10xlpsIFRcohsBV8A9tpNd5QzCraNYrDA31jw AQ7g== X-Gm-Message-State: AOJu0YzafLt4ovnAatec+xn8Oa7MJmqquEPOIg+y8PV6qovy8WdM79jj RVhdkMKSXXWCsaFj/6S098HbTwOL+jf5PWa6fz5QZXwFlhYg3OhMXq6xBjE/JbyjhSbjBZTEl5R He05kFK99C65QaVO/ErvgGrLaOm+fJB186aWUTu+8h2+zSL0km252llPdqKdtZxLUq/KyZdNdbq XKF62AcIe9OEXgVbeSbcZPyspRe9mEW1fR5yDmBR6c X-Google-Smtp-Source: AGHT+IERPnzVnCsElNfP/GC8z5MnQFydrEu/u0Y5dNDPlmnJIDED6IiLwNKJKezGPOk8IkgtIJ55H+yr X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:6902:1b92:b0:dc2:2e43:29ac with SMTP id ei18-20020a0569021b9200b00dc22e4329acmr329121ybb.12.1706021729334; Tue, 23 Jan 2024 06:55:29 -0800 (PST) Date: Tue, 23 Jan 2024 15:53:28 +0100 In-Reply-To: <20240123145258.1462979-52-ardb+git@google.com> Mime-Version: 1.0 References: <20240123145258.1462979-52-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=2122; i=ardb@kernel.org; h=from:subject; bh=fCt9XeVduRrl1YkF83ISOqj+J+Pif8rApwfg9HcHkuw=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIXX9pVe71r7WPBd9y3DC/NglKa/NCq2m8CeafqmzeXaPf /Hc0MzVHaUsDGIcDLJiiiwCs/++23l6olSt8yxZmDmsTCBDGLg4BWAir64yMhxa7vlerv5+5fl9 /yW/RfEc3W5StPlhDbPzrenzKx6sblvB8Iez2iPzYNiWR4sXNvv05aVUNFvNXLM82Xb1/oLL7Ew tR9kA X-Mailer: git-send-email 2.43.0.429.g432eaa2c6b-goog Message-ID: <20240123145258.1462979-81-ardb+git@google.com> Subject: [PATCH v7 29/50] arm64: mm: avoid fixmap for early swapper_pg_dir updates From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240123_145532_675095_09729B51 X-CRM114-Status: GOOD ( 14.25 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel Early in the boot, when .rodata is still writable, we can poke swapper_pg_dir entries directly, and there is no need to go through the fixmap. After a future patch, we will enter the kernel with swapper_pg_dir already active, and early swapper_pg_dir updates for creating the fixmap page table hierarchy itself cannot go through the fixmap for obvious reaons. So let's keep track of whether rodata is writable, and update the descriptor directly in that case. As the same reasoning applies to early KASAN init, make the function noinstr as well. Signed-off-by: Ard Biesheuvel --- arch/arm64/mm/mmu.c | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-) diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index 14a62c773201..9758f7e3f4b6 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -55,6 +55,8 @@ EXPORT_SYMBOL(kimage_voffset); u32 __boot_cpu_mode[] = { BOOT_CPU_MODE_EL2, BOOT_CPU_MODE_EL1 }; +static bool rodata_is_rw __ro_after_init = true; + /* * The booting CPU updates the failed status @__early_cpu_boot_status, * with MMU turned off. @@ -71,10 +73,21 @@ EXPORT_SYMBOL(empty_zero_page); static DEFINE_SPINLOCK(swapper_pgdir_lock); static DEFINE_MUTEX(fixmap_lock); -void set_swapper_pgd(pgd_t *pgdp, pgd_t pgd) +void noinstr set_swapper_pgd(pgd_t *pgdp, pgd_t pgd) { pgd_t *fixmap_pgdp; + /* + * Don't bother with the fixmap if swapper_pg_dir is still mapped + * writable in the kernel mapping. + */ + if (rodata_is_rw) { + WRITE_ONCE(*pgdp, pgd); + dsb(ishst); + isb(); + return; + } + spin_lock(&swapper_pgdir_lock); fixmap_pgdp = pgd_set_fixmap(__pa_symbol(pgdp)); WRITE_ONCE(*fixmap_pgdp, pgd); @@ -628,6 +641,7 @@ void mark_rodata_ro(void) * to cover NOTES and EXCEPTION_TABLE. */ section_size = (unsigned long)__init_begin - (unsigned long)__start_rodata; + WRITE_ONCE(rodata_is_rw, false); update_mapping_prot(__pa_symbol(__start_rodata), (unsigned long)__start_rodata, section_size, PAGE_KERNEL_RO);