From patchwork Fri Jul 1 13:04:41 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 12903285 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 97152C433EF for ; Fri, 1 Jul 2022 13:08:47 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=QmXSEaZjJomTYTvPXCfjQY5SCUQ59OljU5YKwMwJQ8A=; b=gs692noJ2MKCyw DQ7q7x1vbznzv87R+gqAFlq02s8U7aLfyVrovtTi/Dv+wB3X09xS3sV/OJB7UqTmNCgYLHhA3ReSu nSgRAt7CZh5TeBMFY2aNCMqrk2XTPve8EJia1VoMZoUrxZ+T/4Kh2+RkzFhVWzFHabIJy5Wu7DPLX bULBy5DMcPKCrgYl2oPcaIpvqbP4oQlx0nOl9IMHAqG7UbuZicSY56IUUBw4B8EAG/tdFDHhw6IZQ 4z+dZaPgr02Ji0DC8Zn04sYVxrzpIvg9WQuWAknF4lmTw2UoTR2VMAuG9+/Ajn/JjOseHjWyTwT2I MDdG//zJogLgZCq3KvIw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1o7GMz-004wpA-09; Fri, 01 Jul 2022 13:07:37 +0000 Received: from ams.source.kernel.org ([2604:1380:4601:e00::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1o7GKf-004vmI-La for linux-arm-kernel@lists.infradead.org; Fri, 01 Jul 2022 13:05:15 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 40C7FB8302F; Fri, 1 Jul 2022 13:05:12 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 06FFDC341C8; Fri, 1 Jul 2022 13:05:08 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1656680710; bh=1fzmg/3OD9g7R0gjW+Y/xHZpBn6aEnC6uCRxWRxEE5g=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=tM0D/zzwdXjv9lOWn6XA3GcN64l/oZCTYXDQ0Sb13v/Q1eEDFT3qNVaIpIN3j+iXS Q+ieZrZaIZvZ1gbHuHvIN1h6/jjXpxS8DR2Lgiw2P5zAy389n5KVecEJkTHGiEgv5K z0vsYNqdwuW5ABCqbj7TITvnrlKmOT34GPVw1E8D/KhQ3OEJwKl30wYYzsWk48VMZy OW9IyY8RjRgamnueOFC1WuphvhCvohgu2FhmDCZOd3hfU9b0CToVJ3g9OhrMmGWMOn 8wDKK+7iKesoQDaUtucZZiwG/WBCBmnyuOT9bTYBarfPo/+y6aXoJpPEl2xWoKdHHV DttsFr38Aq6aQ== From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Marc Zyngier , Will Deacon , Mark Rutland , Kees Cook , Catalin Marinas , Mark Brown , Anshuman Khandual Subject: [PATCH v6 6/9] arm64: mm: avoid fixmap for early swapper_pg_dir updates Date: Fri, 1 Jul 2022 15:04:41 +0200 Message-Id: <20220701130444.2945106-7-ardb@kernel.org> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220701130444.2945106-1-ardb@kernel.org> References: <20220701130444.2945106-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=2138; h=from:subject; bh=1fzmg/3OD9g7R0gjW+Y/xHZpBn6aEnC6uCRxWRxEE5g=; b=owEB7QES/pANAwAKAcNPIjmS2Y8kAcsmYgBivvDmxtPFw3NvugDCx2PtZN0DIuvG46vy3JQP2pcX URahXtWJAbMEAAEKAB0WIQT72WJ8QGnJQhU3VynDTyI5ktmPJAUCYr7w5gAKCRDDTyI5ktmPJB2xC/ 97lW6vTZIY7HduhCgaiccdbDqE+AnQyGqzuX/DqKuwdVTFZzUq/Tjnu8AB4lgwwNOyC/ZYCuUWT9Hj updr+UJgYyJC7GE5w3NOXBIFWhakbgLIU+I7c+ptKRW5sKuq3Iyr55ajWPHICFtmaGdPcgTJceD3T+ WANSFaNURXDaCw0P/GPwJbLjyPqvAItmy0/pDFg/O8R31oUl6e7aaGsotOIzaVKF2UcrbDOC32t8kY W+HevnNDiEs8YtnYlQd0hR1QyoZOm3BPP8doIzj4IFIIF06jjC55fmo++QB5HGY965HgZCbACX8Ib4 eCrnNB4lDzNha5HxsQR3cPNVS4XKEOS01ahFDPVQQCb89uX+LbSub4C+TUktz/Vp33ebP0QJKlrpXA eovxYNIAL4U2IVIOoItHM+qQ1xJqRhQMCR2/5/iH9HryNnp7pwFV2UziKiBdrdhFHqIeoPOyzx7ZgK vxPadTfwcZImH49MsBT7MjbHxd2yICFgpUsAT/lHtdv1g= X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220701_060514_101268_85542C43 X-CRM114-Status: GOOD ( 15.92 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Early in the boot, when .rodata is still writable, we can poke swapper_pg_dir entries directly, and there is no need to go through the fixmap. After a future patch, we will enter the kernel with swapper_pg_dir already active, and early swapper_pg_dir updates for creating the fixmap page table hierarchy itself cannot go through the fixmap for obvious reaons. So let's keep track of whether rodata is writable, and update the descriptor directly in that case. As the same reasoning applies to early KASAN init, make the function noinstr as well. Signed-off-by: Ard Biesheuvel --- arch/arm64/mm/mmu.c | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-) diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index a723bd2cfc27..8965f621e2c3 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -58,6 +58,8 @@ EXPORT_SYMBOL(kimage_voffset); u32 __boot_cpu_mode[] = { BOOT_CPU_MODE_EL2, BOOT_CPU_MODE_EL1 }; +static bool rodata_is_rw __ro_after_init = true; + /* * The booting CPU updates the failed status @__early_cpu_boot_status, * with MMU turned off. @@ -78,10 +80,21 @@ static pud_t bm_pud[PTRS_PER_PUD] __page_aligned_bss __maybe_unused; static DEFINE_SPINLOCK(swapper_pgdir_lock); static DEFINE_MUTEX(fixmap_lock); -void set_swapper_pgd(pgd_t *pgdp, pgd_t pgd) +void noinstr set_swapper_pgd(pgd_t *pgdp, pgd_t pgd) { pgd_t *fixmap_pgdp; + /* + * Don't bother with the fixmap if swapper_pg_dir is still mapped + * writable in the kernel mapping. + */ + if (rodata_is_rw) { + WRITE_ONCE(*pgdp, pgd); + dsb(ishst); + isb(); + return; + } + spin_lock(&swapper_pgdir_lock); fixmap_pgdp = pgd_set_fixmap(__pa_symbol(pgdp)); WRITE_ONCE(*fixmap_pgdp, pgd); @@ -613,6 +626,7 @@ void mark_rodata_ro(void) * to cover NOTES and EXCEPTION_TABLE. */ section_size = (unsigned long)__init_begin - (unsigned long)__start_rodata; + WRITE_ONCE(rodata_is_rw, false); update_mapping_prot(__pa_symbol(__start_rodata), (unsigned long)__start_rodata, section_size, PAGE_KERNEL_RO);