From patchwork Wed Feb 14 12:29:08 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13556419 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 815BEC48BC4 for ; Wed, 14 Feb 2024 12:32:24 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=bkGYdxFdPgvleGB3uYGyEDhImt+Y4F+pwlf69qtdWfQ=; b=yYLxd288OT7XnJYW3PPZ1+7uIm r3mulQ06FTR/ZffiQD5X19+D3xdY3ZAQ3uWlFoewG7vYn7u2H7O+YzlXpUxMYyYH1Bbwo5bo4v1uN DfAcxfpsTaZLTCh/FJv+7hLNy95VSy9PHzdjUeHTUhAl9D4l8Ws4MvAscHjhSdGZMS2bxqRcLbG6J jXByIp7s4Qc3g7omUxdXFTD0frrxtC/4MFqWLNHz3NBYuovu1IziYkildzASkZQMqCPLRecTxDoG3 5nayu+BuDr3RxieECDhDm4GG+pOa4pEd17h0w1GAwxTy0aGhGyTshgLSR/H/0Z/uAzf9wqyZ2incq 4KFcxQ/Q==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1raEQp-0000000CpOv-3dMp; Wed, 14 Feb 2024 12:32:07 +0000 Received: from casper.infradead.org ([2001:8b0:10b:1236::1]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1raEPS-0000000Cooi-1pj6 for linux-arm-kernel@bombadil.infradead.org; Wed, 14 Feb 2024 12:30:42 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=7fAeJkvIpRC6qpF8jpEwHWSgHc+l/OrvPGb3S3qTTzU=; b=lQdtGMcj38+vZUzDvetibyd8IE FroBP5Z4PLyOTpUN9qFdCDbBgvVhMF/mGtEtSB0O53pN7wmePKTcvLlfuD+yFbfbAQYNVJrnzZwGq o15ZxFkC4eYObwdz4ObZfKMuipTvXmDNFyRKL1rwKowhHWVf8KvZSSAsHsgUHM2cNSEQ4Q3P04DG+ tzSUmrWMlj+wrcDAIh2y4fpesZ/Uap04zL2tab0DF7zihuknqlqwddPKQp+rPpyqULdN9cDMJVZ24 hdMteKoDwmHHS+G1f9HpbPcgmGVWvjgTvWqjcqD/q0lV0yb8AjQCXqwi60Y16dk0HndHDAOTsptuJ 86SE482Q==; Received: from mail-wm1-x34a.google.com ([2a00:1450:4864:20::34a]) by casper.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1raEPM-0000000Ga0R-49rO for linux-arm-kernel@lists.infradead.org; Wed, 14 Feb 2024 12:30:41 +0000 Received: by mail-wm1-x34a.google.com with SMTP id 5b1f17b1804b1-40e4478a3afso3594385e9.1 for ; Wed, 14 Feb 2024 04:30:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1707913829; x=1708518629; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=7fAeJkvIpRC6qpF8jpEwHWSgHc+l/OrvPGb3S3qTTzU=; b=stjX60P18l57zDegmgPTfJKhgcWs5zNZLGH1VO5hIPTcHsHKV3JC6Bc8ug2alOOzax 7JR4gbN0WgGfMJnMBDPzkX64WumKGPHbZqoreJitEvhym5JA24xYuYb227J4J/BeLVyD 38DadUu7Xf3O5Lx0mfeaJCfd1kfz304DYHKpNp8sepDUhv1VyqsWf93pWCI57AL3jOmV Tp8wVu0qEw69JlxL0CaaQdkA4SDIK4UH7w5REmxy7dxLDsnLZy8rwrF2nSTqf+NT1mhk ARElIf8wfzONGhu3NIYtXn3w9rMuXMtj8GYcomVOaywkF9Qo9kJaeh094aYXjtEACpX1 p+3g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707913829; x=1708518629; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=7fAeJkvIpRC6qpF8jpEwHWSgHc+l/OrvPGb3S3qTTzU=; b=TlXDKHBYFaUYrE1k96do5gdRBSmbv61Wk9lfbPnfbno7Tyd66G+i7AEe/82Wzp0WXj YlHFiZ4z39Xaab8GlX7sns/dEz+hq134jqjrejmn4xXpRhF/HmM9PZfI/Hw4GK9LgT6a PTiMHaXBa3KDA1RwW7OGMFaKhjqzb4kafCZ6vGzHuBmF3tQZOTprtPogezL35THXzJJD 3Q/owr4Mj1w2+M1kCPHZiOnhKJmp40o/Dl8T2jTb5tTH02KJdH/Ngu+DXGZZHPE63mjA mIVppFMpPYzZ9IM/yGMRhCmqaXYGgGZpURt1hyMSvFEVT5C9lvhVMkFGgt7xX/MCbeyT Y46w== X-Gm-Message-State: AOJu0Yw/GQw8ESEyovXwH3rp0fyuh9fNgjkjk6jU7a+TN8J8CQzA1s/e YA0fAnfTs8ski5UJkr/eSLpWKeM0dv8+OaibyF0cCgU5/l4zDCm2GkgHL/t7fXwalR5EaunSqvE 2xw9Wzz8XLE3zqBhVd17Jq8R18duSN+QwcK/eKRLl1Ili8szvrGcaCpYlLcIcYRw5m5HkPfiLUY UWrIu0Ay1v6ASX339a1q+IMfhz5SHKvIQribMIJHFf X-Google-Smtp-Source: AGHT+IFy13d/M5X47bBSjYv8vV7p8AI1yVfLeCMQZ9FtFnHUL3nzCb5XNG9nPkxav365BYjf+XXVmkqU X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:600c:519b:b0:411:bc74:195f with SMTP id fa27-20020a05600c519b00b00411bc74195fmr46895wmb.2.1707913829095; Wed, 14 Feb 2024 04:30:29 -0800 (PST) Date: Wed, 14 Feb 2024 13:29:08 +0100 In-Reply-To: <20240214122845.2033971-45-ardb+git@google.com> Mime-Version: 1.0 References: <20240214122845.2033971-45-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=2122; i=ardb@kernel.org; h=from:subject; bh=qFRKJ7gXmCU+cZPFl5JJUliUFkG3MloRwbrG4prZL4U=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIfXMJrGjd+f5nAzlSHyYXjLjZK1KzpzOplhO7bVPokLVJ y8NP9baUcrCIMbBICumyCIw+++7nacnStU6z5KFmcPKBDKEgYtTACbyah/D/6xln58HrOi5Gpcv c+dlOK/kpGcVGrxvOeuehQj+t69eFMTIsFaXlVl/0mnlIuUj6vZzrifvMnjXztN30rpn1/v0Lul 6fgA= X-Mailer: git-send-email 2.43.0.687.g38aa6559b0-goog Message-ID: <20240214122845.2033971-67-ardb+git@google.com> Subject: [PATCH v8 22/43] arm64: mm: avoid fixmap for early swapper_pg_dir updates From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240214_123038_461891_E16AF6AA X-CRM114-Status: GOOD ( 14.13 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel Early in the boot, when .rodata is still writable, we can poke swapper_pg_dir entries directly, and there is no need to go through the fixmap. After a future patch, we will enter the kernel with swapper_pg_dir already active, and early swapper_pg_dir updates for creating the fixmap page table hierarchy itself cannot go through the fixmap for obvious reaons. So let's keep track of whether rodata is writable, and update the descriptor directly in that case. As the same reasoning applies to early KASAN init, make the function noinstr as well. Signed-off-by: Ard Biesheuvel --- arch/arm64/mm/mmu.c | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-) diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index 14a62c773201..9758f7e3f4b6 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -55,6 +55,8 @@ EXPORT_SYMBOL(kimage_voffset); u32 __boot_cpu_mode[] = { BOOT_CPU_MODE_EL2, BOOT_CPU_MODE_EL1 }; +static bool rodata_is_rw __ro_after_init = true; + /* * The booting CPU updates the failed status @__early_cpu_boot_status, * with MMU turned off. @@ -71,10 +73,21 @@ EXPORT_SYMBOL(empty_zero_page); static DEFINE_SPINLOCK(swapper_pgdir_lock); static DEFINE_MUTEX(fixmap_lock); -void set_swapper_pgd(pgd_t *pgdp, pgd_t pgd) +void noinstr set_swapper_pgd(pgd_t *pgdp, pgd_t pgd) { pgd_t *fixmap_pgdp; + /* + * Don't bother with the fixmap if swapper_pg_dir is still mapped + * writable in the kernel mapping. + */ + if (rodata_is_rw) { + WRITE_ONCE(*pgdp, pgd); + dsb(ishst); + isb(); + return; + } + spin_lock(&swapper_pgdir_lock); fixmap_pgdp = pgd_set_fixmap(__pa_symbol(pgdp)); WRITE_ONCE(*fixmap_pgdp, pgd); @@ -628,6 +641,7 @@ void mark_rodata_ro(void) * to cover NOTES and EXCEPTION_TABLE. */ section_size = (unsigned long)__init_begin - (unsigned long)__start_rodata; + WRITE_ONCE(rodata_is_rw, false); update_mapping_prot(__pa_symbol(__start_rodata), (unsigned long)__start_rodata, section_size, PAGE_KERNEL_RO);