From patchwork Tue Jan 23 14:53:00 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13527532 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 09E01C47258 for ; Tue, 23 Jan 2024 14:55:05 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=aCnzWgOV2HiESHNd5tI32wxgxKXeykzxZ9k+pVJSwk4=; b=YOCDXE7iS0p72qBCcmnaTz/swq IYQtytqu2uM6+xxNKv40Hw+ZdkLqO3NMVFCtIducfJoMf8FkwhsmYkZ5AS1Necq4P1I06AMT9tr5a nUGRIusoJq4ztQ85VQQ8DkYlB6D6RKqrrsI4NNykKyaifezb9zcLfTYM3yWfmi+rC+wKlAe7sGzq5 TfIstJ2M3DXkW8ZxdXGbNBbfYZJVl3DUL+GYVAjiv33j0oBd2zELHex2lwvb70HPixRgmyJajWFUa ITDMjJtq1pDhe0eCT+sHSKn9vPSzZKmRs1Sk51DxHX7unVbXhesOsa928t8oMRM6Z95Qky3tbE8T/ AP+/IfCQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rSIAb-00Gtco-2Q; Tue, 23 Jan 2024 14:54:33 +0000 Received: from mail-wr1-x449.google.com ([2a00:1450:4864:20::449]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rSIAT-00Gtae-2S for linux-arm-kernel@lists.infradead.org; Tue, 23 Jan 2024 14:54:27 +0000 Received: by mail-wr1-x449.google.com with SMTP id ffacd0b85a97d-337c6828d84so2908578f8f.0 for ; Tue, 23 Jan 2024 06:54:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1706021663; x=1706626463; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=Vdvnoqxfp/ZbCpL2VnMdxQFjE+HexSFT1TWTjcV9QMc=; b=tRA6Vy5VW2U4GARrCU9KeabFIPqrpyzXyR5J88X7ScGf0aufOpKQzmuUSVAEwoApay Va5qTBGmzGLr8TvE+WEfTtOtHWKtbnbP30KtyeKul/RM0HhTvlWjf81jLUahXglUs9cH ZBGUxXQarOPQaErX//vRyUKBzDVj17Hj6qMMpwesAUTKZKZ8epNS+beguPlztDu17FEd 3rXkpDi0n+PNKph7W+E36CIKC2G4JfMe0qutFIn5GUOCvqoMKCNF75BGcuXe66TEiD5B TJL6Fkj6+Neg9fyS7NxWnC2et+sp8iYtatEkt6VIEGsjtIOhBkQjTyy2/9w7v0xNlGAG 6YxQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706021663; x=1706626463; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Vdvnoqxfp/ZbCpL2VnMdxQFjE+HexSFT1TWTjcV9QMc=; b=ROxL5axH21FHRytZil8Z6Hy5325+k3D3SGl8Q2BJAqwyRtRn48EuNxR4cMvihBx8Gs xYrVKMDqCVtmNPG+vYHFVT6thKILzt+D+0mMjKnFcV4lr2Xh8DY5HbmdHaLQdYhV+77O XxuvCg87hSAiaPZbFYQdjF1cEU7D60VXz6oFcUf+eTCKgBBI6jm3ZJQKFz3cI4ixJB4v 75d19iFCeisropPedUg9PS9KsAuRRb0F0lCy55xd4vxhubClFGwV0CwrzL3J0mtwRQVC J0HHXJ+A1GRqmeESWvTne+UJalUlNDQVenWSS2JOdkDmLQCutwnPxiq5w+oIpYq4Iv7P biYg== X-Gm-Message-State: AOJu0Yx1H7+Zk0Rm62Y9oqAJkVyoWt8ET61RK6rMpM9r2CpK4+HJp4Is Qm5JsEUnHSXqTr2U0X3GGtq8CJZvs8T7aRBZfiry1pCgL9w5QgmCsIgwgftXefmuRZPvZjVw0Tt MRZeB3mAkylgOCAJ0FEUjOC3Q2xephr/NswTl5A75p1wsxeOoysxT4dac2RJWApbl7AzkgeezbE jgyY+43IY50AmJFwRss0e8ngmL4DXTeSY8v6aYUA5k X-Google-Smtp-Source: AGHT+IH+MviibXpXMjl+I6q/TSW6u6BfALavFqoq6Vite1SjI0UhodhtAc+lgjUrv7nrloBDp7nC5D4x X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a5d:4251:0:b0:337:8fa2:7a2d with SMTP id s17-20020a5d4251000000b003378fa27a2dmr21293wrr.7.1706021663127; Tue, 23 Jan 2024 06:54:23 -0800 (PST) Date: Tue, 23 Jan 2024 15:53:00 +0100 In-Reply-To: <20240123145258.1462979-52-ardb+git@google.com> Mime-Version: 1.0 References: <20240123145258.1462979-52-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=2007; i=ardb@kernel.org; h=from:subject; bh=dVRNZix1VDPavfiMLhCdLyphPas/Cz+Fk40XocV1I7o=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIXX9pTM7fx1zuMQWev2So2KI14aIo8uviU3U+fb1iIfG6 Z0aNcnhHaUsDGIcDLJiiiwCs/++23l6olSt8yxZmDmsTCBDGLg4BWAikX8Z/nste/DH2Xx29mtd ySU2zpKMcfs+FBzduaxcLGi+Yca/qVEM/2yO7IqyuaW6d26Pcrj3/ZK+V4k+l00Lj702LjhiMs2 HjRUA X-Mailer: git-send-email 2.43.0.429.g432eaa2c6b-goog Message-ID: <20240123145258.1462979-53-ardb+git@google.com> Subject: [PATCH v7 01/50] arm64: mm: Move PCI I/O emulation region above the vmemmap region From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240123_065425_799446_76A8F3D0 X-CRM114-Status: GOOD ( 12.90 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel Move the PCI I/O region above the vmemmap region in the kernel's VA space. This will permit us to reclaim the lower part of the vmemmap region for vmalloc/vmap allocations when running a 52-bit VA capable build on a 48-bit VA capable system. Also, given that PCI_IO_START is derived from VMEMMAP_END, use that symbolic constant directly in ptdump rather than deriving it from VMEMMAP_START and VMEMMAP_SIZE, as those definitions will change in subsequent patches. Acked-by: Mark Rutland Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/memory.h | 4 ++-- arch/arm64/mm/ptdump.c | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/arch/arm64/include/asm/memory.h b/arch/arm64/include/asm/memory.h index d82305ab420f..b04fa8a8a262 100644 --- a/arch/arm64/include/asm/memory.h +++ b/arch/arm64/include/asm/memory.h @@ -49,8 +49,8 @@ #define MODULES_VSIZE (SZ_2G) #define VMEMMAP_START (-(UL(1) << (VA_BITS - VMEMMAP_SHIFT))) #define VMEMMAP_END (VMEMMAP_START + VMEMMAP_SIZE) -#define PCI_IO_END (VMEMMAP_START - SZ_8M) -#define PCI_IO_START (PCI_IO_END - PCI_IO_SIZE) +#define PCI_IO_START (VMEMMAP_END + SZ_8M) +#define PCI_IO_END (PCI_IO_START + PCI_IO_SIZE) #define FIXADDR_TOP (VMEMMAP_START - SZ_32M) #if VA_BITS > 48 diff --git a/arch/arm64/mm/ptdump.c b/arch/arm64/mm/ptdump.c index e305b6593c4e..46acb2a24da0 100644 --- a/arch/arm64/mm/ptdump.c +++ b/arch/arm64/mm/ptdump.c @@ -47,10 +47,10 @@ static struct addr_marker address_markers[] = { { VMALLOC_END, "vmalloc() end" }, { FIXADDR_TOT_START, "Fixmap start" }, { FIXADDR_TOP, "Fixmap end" }, + { VMEMMAP_START, "vmemmap start" }, + { VMEMMAP_END, "vmemmap end" }, { PCI_IO_START, "PCI I/O start" }, { PCI_IO_END, "PCI I/O end" }, - { VMEMMAP_START, "vmemmap start" }, - { VMEMMAP_START + VMEMMAP_SIZE, "vmemmap end" }, { -1, NULL }, }; From patchwork Tue Jan 23 14:53:01 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13527530 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id C2B00C47258 for ; Tue, 23 Jan 2024 14:54:59 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=+YsccQ1/z2uDca++Ruf30o0mLgZpkDBQfuu2ceSEJYA=; b=arH3KoxjQAaAC4dYk1Fgvsw/Ga PyGbJy6SDeInark0+8HB8i4S4H0rhXwLTgMIL40Gq84c1Wr+u6JXFGcj4IxtsFYypGf0wXOzynZyC 68TdAp8/EkrarrQfukDzex0j173JIan4O86OrgCy2Y/7f0Y7wdlIXfqOrGQpNt8TeC5JjYZ8qtBAe pE5gAKBLxpv03n5DPm6p50xXaOiHBDCaGIfhRS4pXjbalVKIrhNB9Vjtf291//SxMDhLluxymWTm/ lfT0lK1OO7thC7Ge3LPyDSDkR7P1j3jSvX/O6OwuQqFxBM3ZyX/DLGrS+SPX6kybLWtxfFt3giDYT JIle+4Lw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rSIAc-00GtdF-1H; Tue, 23 Jan 2024 14:54:34 +0000 Received: from mail-yw1-x1149.google.com ([2607:f8b0:4864:20::1149]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rSIAV-00Gtb0-2S for linux-arm-kernel@lists.infradead.org; Tue, 23 Jan 2024 14:54:29 +0000 Received: by mail-yw1-x1149.google.com with SMTP id 00721157ae682-5e73bd9079eso72048237b3.2 for ; Tue, 23 Jan 2024 06:54:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1706021665; x=1706626465; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=NPxGZ40dkByGWudN3BtDjpXgJcnI+pEUOr8DDOXiZlc=; b=PUOaQ6P/x0zArKZGOPoapWbTy2wjygNnRLOWpqqbH30u+b5ZzKeljRwuLgdD0jwbf+ vTOCAcM2CDYj8PnS2lEp9H/d5ShJRBfXxxlfD3Wc2VwgJA1+Hoo9SAtHSihv7U7LPeym TFGJdG3JQR53sz7jUsCJtyoq1n9CAL6Cc/kDduZnKfMzgrNMYD/Am0RZ8BOBgouXuecN L0Kr/D14BcKfaP0+6f8QgBouVmX4f6bBrDOT4Jgwwtu+Y+H0Z5wmogaYVbtfYDg3+28S Axzb1VqQNy7K8tOQQEPCS5g1LMxi04G1sWpIVR4qSXl3WuAbmzC5S8EDhCG6GZsu06q6 N9rQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706021665; x=1706626465; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=NPxGZ40dkByGWudN3BtDjpXgJcnI+pEUOr8DDOXiZlc=; b=kVyfAklJkwk2/5kBmhlGoAQUkaz6etfCCNvgKA3RggjmGTZdCk2WnLyqI4dEoUKYwy tJHn+XpZYYIQDJtWr4cyDmAADX/isdXTbwfqXhA6Szwwp/jO04DwmEubvGIBwh6+leCU BLQrKGfDHmGytRYafCpR9g31iRaHaov6c7jAbd4ABX2/9AB2eEWzFyxO5maQaaz9bSt5 y/vR5H4ItRWnfoomtVQFgHxGDh/F6Y+fJUaNdiR/KHx5Xe/AQniUb0AQ//fafsrBM4BG zeqZOlIjV8bPzr2QJcIvddbsPUdPATgjdHBjcUmlNrUQRYjI5XYucjHAOV5NNkOcMNcp E8yQ== X-Gm-Message-State: AOJu0Yzy5kdICDxhW0pCglJve/QbUnvjhsyVM6WWQROj/vcjqIO2fqdy PxyN7Fq/g8kzYubRa+SjNw3NNqBXc31HB48ngVHuRPBQ0700PsVhqJNXj+YrbwrpS758inH1e+O e8r4xg6AlMAki7Qk2qfJfdjj6jw9Y4OTqtsVRoeBz6BrZjCX2gBRzHUPLtpDsX6H1EqblLDzGl4 VmQXhSRYf/E59kt8zhFBVK/fXZchQLGhkhfO0aYZwQ X-Google-Smtp-Source: AGHT+IGpHWru25yldacDYV6MVHekADw1NaRTtTKSQXcptCgpKuNXEt4vYPiia3cA+AYeV2PMvFOcfhrI X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:6902:2412:b0:dc2:4cf5:9ed0 with SMTP id dr18-20020a056902241200b00dc24cf59ed0mr2737746ybb.5.1706021665497; Tue, 23 Jan 2024 06:54:25 -0800 (PST) Date: Tue, 23 Jan 2024 15:53:01 +0100 In-Reply-To: <20240123145258.1462979-52-ardb+git@google.com> Mime-Version: 1.0 References: <20240123145258.1462979-52-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=3092; i=ardb@kernel.org; h=from:subject; bh=gYLVr8321g17hWMeRcnC+K3579d3jinccJ0uP287Skg=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIXX9pbP6PR/37FEXPH3sos/eRQtyGzZlRV69v3HjlTYDz 5n5KudqO0pZGMQ4GGTFFFkEZv99t/P0RKla51myMHNYmUCGMHBxCsBEhL8y/E/4K7PyRXekzLXp 9XO4yidE/2xVZMxufugcZtS89HhpejXDP91chU0cTkWt03Z+cQ/m3SiSUaowI///RfE67t6CDZn PGQE= X-Mailer: git-send-email 2.43.0.429.g432eaa2c6b-goog Message-ID: <20240123145258.1462979-54-ardb+git@google.com> Subject: [PATCH v7 02/50] arm64: mm: Move fixmap region above vmemmap region From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240123_065427_796354_A5B6532D X-CRM114-Status: GOOD ( 17.34 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel Move the fixmap region above the vmemmap region, so that the start of the vmemmap delineates the end of the region available for vmalloc and vmap allocations and the randomized placement of the kernel and modules. In a subsequent patch, we will take advantage of this to reclaim most of the vmemmap area when running a 52-bit VA capable build with 52-bit virtual addressing disabled at runtime. Note that the existing guard region of 256 MiB covers the fixmap and PCI I/O regions as well, so we can reduce it 8 MiB, which is what we use in other places too. Acked-by: Mark Rutland Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/memory.h | 2 +- arch/arm64/include/asm/pgtable.h | 2 +- arch/arm64/mm/fixmap.c | 3 +++ arch/arm64/mm/ptdump.c | 4 ++-- 4 files changed, 7 insertions(+), 4 deletions(-) diff --git a/arch/arm64/include/asm/memory.h b/arch/arm64/include/asm/memory.h index b04fa8a8a262..f3be3ea74138 100644 --- a/arch/arm64/include/asm/memory.h +++ b/arch/arm64/include/asm/memory.h @@ -51,7 +51,7 @@ #define VMEMMAP_END (VMEMMAP_START + VMEMMAP_SIZE) #define PCI_IO_START (VMEMMAP_END + SZ_8M) #define PCI_IO_END (PCI_IO_START + PCI_IO_SIZE) -#define FIXADDR_TOP (VMEMMAP_START - SZ_32M) +#define FIXADDR_TOP (-UL(SZ_8M)) #if VA_BITS > 48 #define VA_BITS_MIN (48) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 79ce70fbb751..2aa2e3c961d7 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -22,7 +22,7 @@ * and fixed mappings */ #define VMALLOC_START (MODULES_END) -#define VMALLOC_END (VMEMMAP_START - SZ_256M) +#define VMALLOC_END (VMEMMAP_START - SZ_8M) #define vmemmap ((struct page *)VMEMMAP_START - (memstart_addr >> PAGE_SHIFT)) diff --git a/arch/arm64/mm/fixmap.c b/arch/arm64/mm/fixmap.c index c0a3301203bd..6fc17b2e1714 100644 --- a/arch/arm64/mm/fixmap.c +++ b/arch/arm64/mm/fixmap.c @@ -16,6 +16,9 @@ #include #include +/* ensure that the fixmap region does not grow down into the PCI I/O region */ +static_assert(FIXADDR_TOT_START > PCI_IO_END); + #define NR_BM_PTE_TABLES \ SPAN_NR_ENTRIES(FIXADDR_TOT_START, FIXADDR_TOP, PMD_SHIFT) #define NR_BM_PMD_TABLES \ diff --git a/arch/arm64/mm/ptdump.c b/arch/arm64/mm/ptdump.c index 46acb2a24da0..a929b5a321db 100644 --- a/arch/arm64/mm/ptdump.c +++ b/arch/arm64/mm/ptdump.c @@ -45,12 +45,12 @@ static struct addr_marker address_markers[] = { { MODULES_END, "Modules end" }, { VMALLOC_START, "vmalloc() area" }, { VMALLOC_END, "vmalloc() end" }, - { FIXADDR_TOT_START, "Fixmap start" }, - { FIXADDR_TOP, "Fixmap end" }, { VMEMMAP_START, "vmemmap start" }, { VMEMMAP_END, "vmemmap end" }, { PCI_IO_START, "PCI I/O start" }, { PCI_IO_END, "PCI I/O end" }, + { FIXADDR_TOT_START, "Fixmap start" }, + { FIXADDR_TOP, "Fixmap end" }, { -1, NULL }, }; From patchwork Tue Jan 23 14:53:02 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13527533 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 807F6C47DDB for ; Tue, 23 Jan 2024 14:55:05 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=1EfxZBKQsssTpoVRy/0EEMJcf9n2t96V74ro84S7Efg=; b=sL9sZvVLfLqDkb5wcdmkluo69F AgWPqnV9L3zNddLqV9yqznFzbAqhMoF3zIWrWxRYjebRAY6O8sfXzkYQ0db8yXsx/0mK9IEsVWrQ5 zHHMVGlPzrLBx0pmwtTWM76e5/0/aOD0FWkQJAGdBKFQeYu4bItw/MEWc9zhvTFRs2jqOIR85Xc6g 18sshXtnNWFvkQc9Db9yd8spx6h/L4lXYZvy+uvnrM66I5q1iYpJnRNQK4dOO38MAXs+06bzpCidd G5z6hYnSXkcUeFqSJMhWR/q16vgm+bWoNb86FPr4UtKbBDW1vJ/PelpnZgiTh3d+SDYzPUnYvlwKE L41ELqMg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rSIAd-00Gtdw-0A; Tue, 23 Jan 2024 14:54:35 +0000 Received: from mail-yb1-xb4a.google.com ([2607:f8b0:4864:20::b4a]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rSIAY-00Gtbf-1m for linux-arm-kernel@lists.infradead.org; Tue, 23 Jan 2024 14:54:31 +0000 Received: by mail-yb1-xb4a.google.com with SMTP id 3f1490d57ef6-dbe9dacc912so5374435276.2 for ; Tue, 23 Jan 2024 06:54:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1706021668; x=1706626468; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=7KUbNPLy/oDuU7/UNppO3TqIU9rfMMujD0oXCa/9bK8=; b=f6O5NY0CVsVrjxiUVV/dnGS4Id9WT3Wl5e8X2RBwDMi4OoMfGLAE0GcPPvjx+P+92+ L4Q3KbSw9X+u7RkVZRCvLZi0utatNZSyXAeWet4q+m+8YCgM7gMlTe7IkhPXb+tfUjmc /txTZkBf6uoEUZQYr+gRHk9PnbAqR3lMAPdvJOBCR9WbzeF3fbZGH50Kot9ze7qEfWLd 1Zuspei24/rH8DoQoWiz+cPWblDJW/dhwKB6kvamCMfNv7GXU6GAn7yC7ouYpt/8yvMr H/5h0WansGJfoGmyCbMgZHgeelt/cH1s5h9mcaGL1sNZjJMjY++4YyixUlHp8IxW/Lmt dcfQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706021668; x=1706626468; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=7KUbNPLy/oDuU7/UNppO3TqIU9rfMMujD0oXCa/9bK8=; b=pPfnkKz+nJdXIM+UUCexkIkcVON+cCuKAkmrPZ1vQDaWVhc5ox3Vyokw3bJOXhh+uI KM3ZSITb1tMVRZ43VoOx/xhyFP+ZPHZ9z7gaysV5o4bw3bY0Dmtu0F0QvlndbdtE2IxP LkkpoXviqvCIqvGjgi38Cuy3Ijt0V5aqssyfXC0rVpeiPvI3ycmxqF1rFZO0hT5Ytyg+ +nJdTunAY8/bXrKI8axitxFtWPDUZOP0y3ZPKjlfjOmqGKY4N4hBec5u0BdOUU6kzftE 95cEfJvn4A3+h3efCF78p52PyEbpotY7HU+jNw15aK5GKmWqiKEuFr6ruDFN+Rx5CKgr bSZQ== X-Gm-Message-State: AOJu0YzzcpSaZpslJQGcmHmn4Wrn36nDyzvF/eXdWSu2ZctaFUd6mxCf vE5b49L4q7TzJVfVt1qAU2EDHEcF0sHCOZYQO6cWgcX47fMFloIKz/K2MN8ROKWb39NpndM4YF1 zWk+QCux3le9921JKOgmNtpFOZYAstYzPFcQtwr5e+vjgtcL/TWZK0KJ33ahkJwe019fAL/SzZQ KRfZ1q8CD2eMOO1vLnqqk2Yp6Vq42Lvlfy5erZYjoL X-Google-Smtp-Source: AGHT+IE/dczq0iAnwBVVnnuG4tlwkNAVV6Mje9vg0lAPTQYwcSf1FA0VgazlOG0BQZrQ8eeAiskkmuXD X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:6902:1808:b0:dc2:3426:c9ee with SMTP id cf8-20020a056902180800b00dc23426c9eemr351083ybb.11.1706021668044; Tue, 23 Jan 2024 06:54:28 -0800 (PST) Date: Tue, 23 Jan 2024 15:53:02 +0100 In-Reply-To: <20240123145258.1462979-52-ardb+git@google.com> Mime-Version: 1.0 References: <20240123145258.1462979-52-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=3343; i=ardb@kernel.org; h=from:subject; bh=RwMtCDsH/GKJzHHMqvbJUk+zkQfhIRA3ItctTiVgwKI=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIXX9pXNRZ98KvVp432Ddh0yOtLbLs5SmXN11z4Dt9LEFj LvtnfdqdJSyMIhxMMiKKbIIzP77bufpiVK1zrNkYeawMoEMYeDiFICJBCox/I9k2u9fws++42bZ ZefHp1/yih1zzEsLvFhhpGfXsSQrvZfhr4T3EqnpFm8CpVND785XX3Lc8GtqwAnnMP2mV8ea6xb 4sAEA X-Mailer: git-send-email 2.43.0.429.g432eaa2c6b-goog Message-ID: <20240123145258.1462979-55-ardb+git@google.com> Subject: [PATCH v7 03/50] arm64: ptdump: Allow all region boundaries to be defined at boot time From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240123_065430_596109_8752ABB9 X-CRM114-Status: GOOD ( 14.44 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel Rework the way the address_markers array is populated so that we can tolerate values that are not compile time constants generally, rather than keeping track manually of the array indexes in question, and poking new values into them manually. This will be needed for VMALLOC_END, which will cease to be a compile time constant after a subsequent patch. Acked-by: Mark Rutland Signed-off-by: Ard Biesheuvel --- arch/arm64/mm/ptdump.c | 54 ++++++++------------ 1 file changed, 22 insertions(+), 32 deletions(-) diff --git a/arch/arm64/mm/ptdump.c b/arch/arm64/mm/ptdump.c index a929b5a321db..66ccb8d6997e 100644 --- a/arch/arm64/mm/ptdump.c +++ b/arch/arm64/mm/ptdump.c @@ -26,34 +26,6 @@ #include -enum address_markers_idx { - PAGE_OFFSET_NR = 0, - PAGE_END_NR, -#if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS) - KASAN_START_NR, -#endif -}; - -static struct addr_marker address_markers[] = { - { PAGE_OFFSET, "Linear Mapping start" }, - { 0 /* PAGE_END */, "Linear Mapping end" }, -#if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS) - { 0 /* KASAN_SHADOW_START */, "Kasan shadow start" }, - { KASAN_SHADOW_END, "Kasan shadow end" }, -#endif - { MODULES_VADDR, "Modules start" }, - { MODULES_END, "Modules end" }, - { VMALLOC_START, "vmalloc() area" }, - { VMALLOC_END, "vmalloc() end" }, - { VMEMMAP_START, "vmemmap start" }, - { VMEMMAP_END, "vmemmap end" }, - { PCI_IO_START, "PCI I/O start" }, - { PCI_IO_END, "PCI I/O end" }, - { FIXADDR_TOT_START, "Fixmap start" }, - { FIXADDR_TOP, "Fixmap end" }, - { -1, NULL }, -}; - #define pt_dump_seq_printf(m, fmt, args...) \ ({ \ if (m) \ @@ -339,9 +311,8 @@ static void __init ptdump_initialize(void) pg_level[i].mask |= pg_level[i].bits[j].mask; } -static struct ptdump_info kernel_ptdump_info = { +static struct ptdump_info kernel_ptdump_info __ro_after_init = { .mm = &init_mm, - .markers = address_markers, .base_addr = PAGE_OFFSET, }; @@ -375,10 +346,29 @@ void ptdump_check_wx(void) static int __init ptdump_init(void) { - address_markers[PAGE_END_NR].start_address = PAGE_END; + struct addr_marker m[] = { + { PAGE_OFFSET, "Linear Mapping start" }, + { PAGE_END, "Linear Mapping end" }, #if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS) - address_markers[KASAN_START_NR].start_address = KASAN_SHADOW_START; + { KASAN_SHADOW_START, "Kasan shadow start" }, + { KASAN_SHADOW_END, "Kasan shadow end" }, #endif + { MODULES_VADDR, "Modules start" }, + { MODULES_END, "Modules end" }, + { VMALLOC_START, "vmalloc() area" }, + { VMALLOC_END, "vmalloc() end" }, + { VMEMMAP_START, "vmemmap start" }, + { VMEMMAP_END, "vmemmap end" }, + { PCI_IO_START, "PCI I/O start" }, + { PCI_IO_END, "PCI I/O end" }, + { FIXADDR_TOT_START, "Fixmap start" }, + { FIXADDR_TOP, "Fixmap end" }, + { -1, NULL }, + }; + static struct addr_marker address_markers[ARRAY_SIZE(m)] __ro_after_init; + + kernel_ptdump_info.markers = memcpy(address_markers, m, sizeof(m)); + ptdump_initialize(); ptdump_debugfs_register(&kernel_ptdump_info, "kernel_page_tables"); return 0; From patchwork Tue Jan 23 14:53:03 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13527536 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 89217C47258 for ; Tue, 23 Jan 2024 14:55:20 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=3BQrT5O1I9IlkjXbUrGR7NcWKmiqripMfyHhdoGumVk=; b=GkkXRk5e2WxnibXpCHr3DfMWG+ grdHy/Kg23v3xJIM5OOeryTfFyUmAT13VXz0uo3Q00Mm0vw4vZAxpgEwCLm5BnjTyqzDRaCz7+kFp rYb6G8Co+cVezKtP61T/JpVSQdR+EQ9Ccv15cL5NOfRIugEHi7Ro/cOlg5q49pm10FwF6LXlaxmA2 FwUxmh7h0Xr236S9zQ2KYbuA2Km16wzqTz7vQh1isXyoX6bVVhASn78fYr2kXpuWpuDfrs1VHn1Tp iTABR2b3iJ2u/07sbaUthv5ut5QFhLnTSVJKE4BXHBTF1XqiE2zAfsQEOYHWoPfbCjEWVUEClMap4 U04Fguqw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rSIAr-00Gtjy-2O; Tue, 23 Jan 2024 14:54:49 +0000 Received: from mail-yw1-x1149.google.com ([2607:f8b0:4864:20::1149]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rSIAa-00GtcI-0S for linux-arm-kernel@lists.infradead.org; Tue, 23 Jan 2024 14:54:33 +0000 Received: by mail-yw1-x1149.google.com with SMTP id 00721157ae682-5ff908a0247so48500497b3.2 for ; Tue, 23 Jan 2024 06:54:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1706021671; x=1706626471; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=YqZpebP/619MZl4FrpiPrqRKlomSscik+QpqTvG+E/0=; b=TjBfEGYIDACo4700Q2oGalXiV0MIchsOHERR2MvviYOqeRses0pWbqM0M65QP+Huhm RNjXoXbzf+1MCvgU9MLEOpxWGI2jEnFLtyoDHY2xjqyYgBrjlsgZnii3cefNnX6flR7+ DMVAdOP5ZA/3G3+oSnvWr9OC6q8Wb1AUCEKtSPiyZ67opEsefhR86lY2C5gSz9rnIoVw ZoE6W55GsDrzOL9ITc2IrvYCtQC0xict1aNz7hrJtAmS8cS5L8n7ba6oUJWISnoH02GL rud9b88NaGQl19ibbzY3HcjBJma2lAcV/pAY9KmoyQqvqP+x6NDLdPAXVL9BswZDh0b+ D1dg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706021671; x=1706626471; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=YqZpebP/619MZl4FrpiPrqRKlomSscik+QpqTvG+E/0=; b=K6afgKS/tcT8qb8G3v7OIPyOEEV5mK95SHMWfdwcrDvW/QQNZiAVqW1JSUowyrjaCV ecsWYy1XziLFAMROKT2Ud8aVYOoC88spRF9b5vm1X+goX6miGpFoAoF9vSjpv4ycVH76 XKQUJRZlSRdvsATjHfM5TkxQj8JAVN1HwXH5PAQphx2jCobpgh5h1jqAoBjNh3oKXrLe zQStGB27f9KgQRWVgZ/22jGaB4q9jZW5YpyaFF5AXbcP1tWV2xgxaH6ljIC66MiQygie ldWLRNEIajoF6QJNOthl7JltmwsgUgNyeZmWHW9SvzJMjJX2/ZJCnB06s1e0i7lU1B6/ pWjQ== X-Gm-Message-State: AOJu0YwPh37Yx/ijKZCcv6jt1vcYw5rW5oWYj+CqtcXJaMLKerzhMTiN +rF/yktWaL9ln8FEcQfqE660lupI2yWJ8TRDXpH7zegJv2BVqluEyYDF/87chNJO1jREgGu6sHd bpNaCSgcM4Oss22QJ8LKqAOpDME9P/LCpVMXQ3AYnMmvJyJbdylv17qx7zbph2pYSw9McMMRglM m3v1cliUHxqdb2w452+p4MSxSfab9zwAhL3f33HPLA X-Google-Smtp-Source: AGHT+IGZ7LHsAR1y9kL/k7v453+mxgypoThLExj21IPccAxuGuflr8aAOfNVo/wUhdYtMOiipGovAijW X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a25:d08f:0:b0:dc2:25fd:eff1 with SMTP id h137-20020a25d08f000000b00dc225fdeff1mr318865ybg.4.1706021670456; Tue, 23 Jan 2024 06:54:30 -0800 (PST) Date: Tue, 23 Jan 2024 15:53:03 +0100 In-Reply-To: <20240123145258.1462979-52-ardb+git@google.com> Mime-Version: 1.0 References: <20240123145258.1462979-52-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=1413; i=ardb@kernel.org; h=from:subject; bh=hVe/Y/PUF55gLJVsLW/mBqvWk0Rwo/FbI6Wtlis7ICs=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIXX9pfPdQYou37Xv/fk3i9tn4qV1zdma9//nsW9XjfV6v HJTn5BhRykLgxgHg6yYIovA7L/vdp6eKFXrPEsWZg4rE8gQBi5OAZjIrreMDJO8F67sfMLf3cbf ciZgQd+FW+1Pez9LrX5WqZ4jU+sbwsbwh1/zWWNH24TigAWvD31JXFAVbJyuyBt7XqtT/xOv1lx PJgA= X-Mailer: git-send-email 2.43.0.429.g432eaa2c6b-goog Message-ID: <20240123145258.1462979-56-ardb+git@google.com> Subject: [PATCH v7 04/50] arm64: ptdump: Discover start of vmemmap region at runtime From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240123_065432_176658_75A73310 X-CRM114-Status: GOOD ( 14.14 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel We will soon reclaim the part of the vmemmap region that covers VA space that is not addressable by the hardware. To avoid confusion, ensure that the 'vmemmap start' marker points at the start of the region that is actually being used for the struct page array, rather than the start of the region we set aside for it at build time. Acked-by: Mark Rutland Signed-off-by: Ard Biesheuvel --- arch/arm64/mm/ptdump.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/arch/arm64/mm/ptdump.c b/arch/arm64/mm/ptdump.c index 66ccb8d6997e..5f0849528ccf 100644 --- a/arch/arm64/mm/ptdump.c +++ b/arch/arm64/mm/ptdump.c @@ -346,6 +346,8 @@ void ptdump_check_wx(void) static int __init ptdump_init(void) { + u64 page_offset = _PAGE_OFFSET(vabits_actual); + u64 vmemmap_start = (u64)virt_to_page((void *)page_offset); struct addr_marker m[] = { { PAGE_OFFSET, "Linear Mapping start" }, { PAGE_END, "Linear Mapping end" }, @@ -357,7 +359,7 @@ static int __init ptdump_init(void) { MODULES_END, "Modules end" }, { VMALLOC_START, "vmalloc() area" }, { VMALLOC_END, "vmalloc() end" }, - { VMEMMAP_START, "vmemmap start" }, + { vmemmap_start, "vmemmap start" }, { VMEMMAP_END, "vmemmap end" }, { PCI_IO_START, "PCI I/O start" }, { PCI_IO_END, "PCI I/O end" }, From patchwork Tue Jan 23 14:53:04 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13527534 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 72355C47DDB for ; Tue, 23 Jan 2024 14:55:15 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=Ux5FEuSCInVXsv/7DieWf/EFj04GKwnvbIIgnHa30sI=; b=kDHYeNosRHYdMAlTVyRT8cUNqZ cP9pc0z1uMHlPXR6kbrMVMAyotKNvblvyIldsXyDWe3fci0Y0qd3+/Vu3xmZETYny4e+5qFVdHlIx 6RUvKB99onh0HgO/mIKaOnPD8VCPTn15ygCFNVZPGNIRTtheCsEVEpDhBHcYPfI4p2wUX7q/LYxqK pOax0aaeLPx1EmuHYYbtyD88eEYTRxcOxlWQH4H1xjmCcFSY0U6aP6apCE8GxM1qQO2BgXAW2tcfd aMDkwpNZOTS4WvgdfId38UOsBO6KVFXRwXRjHNsI9ofOn98kxpjM/ErTdj84GCcCCmFiHom7KLzWi nbvr/Hdg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rSIAs-00GtkP-1c; Tue, 23 Jan 2024 14:54:50 +0000 Received: from mail-yb1-xb4a.google.com ([2607:f8b0:4864:20::b4a]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rSIAc-00Gtci-1R for linux-arm-kernel@lists.infradead.org; Tue, 23 Jan 2024 14:54:35 +0000 Received: by mail-yb1-xb4a.google.com with SMTP id 3f1490d57ef6-dc221fad8c7so6311940276.3 for ; Tue, 23 Jan 2024 06:54:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1706021673; x=1706626473; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=jA+DKoZHst8Q9IMSbPhxUf/d1q0YMDko04eruQNM4yk=; b=pFPQeQpEVWKWoAXUuI1KW2fhxrIovUKTbY50hOetwkxYLNZddS7GHebm5XNMLGjBD2 mT7YfyewwSlVvfNU5Q7TIm1MIBhVm1W3ZvFPfMY4G8OW+WtdBMglmFmHaPl587QQgV9e p3wTwoa2PVUyeXAYf4OVDE8Q6RoEb4ylZhvo2NQ9CsJh1eLW8VzeZS1zicTV3MW19uPH u0/StuQz1mByT0QMeG1GIYe4HNijmn1tqb5L2R0rvaNqmnUMY1echSOC+5M7G4T9vtZ5 ZtJszQ1+3uyzjvW+fjgoK+PCK9ffnhwSKIPNdFraaVRBtWxPdsi1SkurA8/OU6auatgP 8Gnw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706021673; x=1706626473; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=jA+DKoZHst8Q9IMSbPhxUf/d1q0YMDko04eruQNM4yk=; b=VVPFhJP7gKIeNZlBvLghxLRJSwVpC/H/qs/KIoWWPLWcv1SzAKknvAZVa0HXgSor36 ZH8bV/9swTjAaNmzTaIYmjpwEbaQrnQBdTAIclVDHIij3Hb2zaZDRkLax8K6dtG62ZKz +s+WIV7UE8lUp6YTtzwfbt4E2DXRPhutbiGRvr8sXt55kAJpkGxZzrKl1yPDw9xLDjL9 8hHErA0zUpM/2NBnG/QF2H4rmMYop5wKyJShUo7M0J4f3KV+xalxdl9Y6uIOEZdkaRn1 L3xYmdPLmyrCNOOXEQ48hZn8q5lBlLrmSQ9yPmrzAEW8oUHzPdOuTlFGmdKdUGHdCP+6 ZPbw== X-Gm-Message-State: AOJu0YzIZu72+IgUsPOEGcqbePwhYUt7fn2GUUgx+k+xN1h9rgbPzoPJ F9MhCgho3vA2y/GKHeywjvzyZ32evmUzW9j1vz3nEo5uJXrb+ju3XugH1AIKsB0KZJzydsOg7Gk GvzMy8Bak1asrHuKB4YNzs9dVPfBnreQn3yDoF009z3xMZ/XH326vQ9GkOHNeNVyHlWB9hzZTNK 29Cn/vwPwAYFz+mZAxQb2KH4CwIJZTuG+we0oYgZBW X-Google-Smtp-Source: AGHT+IECY0+JYPv7S/1IlpDt42Vxp0vrygBO4p6W3UbYe1m/cIW5GDmRTwR983UrSByQn61uha0Wid3r X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:6902:1005:b0:dc2:5525:f6b with SMTP id w5-20020a056902100500b00dc255250f6bmr2741798ybt.7.1706021673082; Tue, 23 Jan 2024 06:54:33 -0800 (PST) Date: Tue, 23 Jan 2024 15:53:04 +0100 In-Reply-To: <20240123145258.1462979-52-ardb+git@google.com> Mime-Version: 1.0 References: <20240123145258.1462979-52-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=2503; i=ardb@kernel.org; h=from:subject; bh=yh9XctYC8E98u+//lXEWXJdFuG3EvzjK+wj6zwGC/yg=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIXX9pQsG+5NSdv+8eF7/Z2C6yu7+jc+qdsWnZDk1vi+cc rrY3Gt6RykLgxgHg6yYIovA7L/vdp6eKFXrPEsWZg4rE8gQBi5OAZiIci8jw4nN+nMWdPxfMXH3 zpxTvEsjJt6+vdhO9PRnLda5ZlNrzfcy/LNOFpZe/Ec1YRtnrO7BuZVb5wjJbtLcHSnxtGTNveW Fe7gA X-Mailer: git-send-email 2.43.0.429.g432eaa2c6b-goog Message-ID: <20240123145258.1462979-57-ardb+git@google.com> Subject: [PATCH v7 05/50] arm64: vmemmap: Avoid base2 order of struct page size to dimension region From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240123_065434_497825_B5FA4748 X-CRM114-Status: GOOD ( 17.07 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel The placement and size of the vmemmap region in the kernel virtual address space is currently derived from the base2 order of the size of a struct page. This makes for nicely aligned constants with lots of leading 0xf and trailing 0x0 digits, but given that the actual struct pages are indexed as an ordinary array, this resulting region is severely overdimensioned when the size of a struct page is just over a power of 2. This doesn't matter today, but once we enable 52-bit virtual addressing for 4k pages configurations, the vmemmap region may take up almost half of the upper VA region with the current struct page upper bound at 64 bytes. And once we enable KMSAN or other features that push the size of a struct page over 64 bytes, we will run out of VMALLOC space entirely. So instead, let's derive the region size from the actual size of a struct page, and place the entire region 1 GB from the top of the VA space, where it still doesn't share any lower level translation table entries with the fixmap. Acked-by: Mark Rutland Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/memory.h | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/arch/arm64/include/asm/memory.h b/arch/arm64/include/asm/memory.h index f3be3ea74138..60904a6c4b42 100644 --- a/arch/arm64/include/asm/memory.h +++ b/arch/arm64/include/asm/memory.h @@ -30,8 +30,8 @@ * keep a constant PAGE_OFFSET and "fallback" to using the higher end * of the VMEMMAP where 52-bit support is not available in hardware. */ -#define VMEMMAP_SHIFT (PAGE_SHIFT - STRUCT_PAGE_MAX_SHIFT) -#define VMEMMAP_SIZE ((_PAGE_END(VA_BITS_MIN) - PAGE_OFFSET) >> VMEMMAP_SHIFT) +#define VMEMMAP_RANGE (_PAGE_END(VA_BITS_MIN) - PAGE_OFFSET) +#define VMEMMAP_SIZE ((VMEMMAP_RANGE >> PAGE_SHIFT) * sizeof(struct page)) /* * PAGE_OFFSET - the virtual address of the start of the linear map, at the @@ -47,8 +47,8 @@ #define MODULES_END (MODULES_VADDR + MODULES_VSIZE) #define MODULES_VADDR (_PAGE_END(VA_BITS_MIN)) #define MODULES_VSIZE (SZ_2G) -#define VMEMMAP_START (-(UL(1) << (VA_BITS - VMEMMAP_SHIFT))) -#define VMEMMAP_END (VMEMMAP_START + VMEMMAP_SIZE) +#define VMEMMAP_START (VMEMMAP_END - VMEMMAP_SIZE) +#define VMEMMAP_END (-UL(SZ_1G)) #define PCI_IO_START (VMEMMAP_END + SZ_8M) #define PCI_IO_END (PCI_IO_START + PCI_IO_SIZE) #define FIXADDR_TOP (-UL(SZ_8M)) From patchwork Tue Jan 23 14:53:05 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13527538 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id B8100C47DDB for ; Tue, 23 Jan 2024 14:55:23 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=WdI8k8hKDivzltfr1ccwLnVa+5zkxzIH+q4rfbp2tI8=; b=PD2+T7JyG2P2cP0g9PjWjabNOJ QSbSDoyIVSzyMg6Q5S0lXBPGcz3F5DmLmC9pf380hFppp2kLYNBFLEg1Oblq8g9YVU8sE+elCb9BC gtWlKUccrVtf3y3Wp9ZDKtcZG1noVeT8mOV73lW3mJL2zGQFvoV/4tsQOSP7exwell9f78RYYXhyT 8q9GwNegTVKurvFDQ2q7EFLXec3dAGrOo+93DVfwGvN2kbdjk5SvxLEnZGZC1Xdpndm+TclU05h3Q u8Qarzr97gTzMGhMZ/PmfzVdRbiJMMF4RWH+WbaX+sYkCcF1ntC1nPdbjWu+K8Ygwlj4arADTzN92 gKVAzP2g==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rSIAt-00Gtl5-2D; Tue, 23 Jan 2024 14:54:51 +0000 Received: from mail-wm1-x349.google.com ([2a00:1450:4864:20::349]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rSIAf-00Gtef-2R for linux-arm-kernel@lists.infradead.org; Tue, 23 Jan 2024 14:54:39 +0000 Received: by mail-wm1-x349.google.com with SMTP id 5b1f17b1804b1-40e412c1e65so38557125e9.2 for ; Tue, 23 Jan 2024 06:54:36 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1706021675; x=1706626475; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=5NlMdZIB9itIIAJFmR4aXI2QYcCN1XpZu8tjp+LjBQw=; b=qI7g39O8FVNLXGaAxedSF4uoyC3w+JYT2bUaUn9v1uyM0BrpytSNoE/4xZ/W7aCLco Fa5ruvtDIqacFFQsWdk5XeIRvcH0fbooACLYVV3cWMAPFSHoHrW4GPT0D/bXugt0Q4a7 1eKIauJ0iYG+HMte7KNwaMgMxwAZp2RJK0R6J13+XpEhA1eJtX9Wn+tSJOHsryYY2HUN BxDqxaT3SYPOa24bzyz5853oc6AWYCjQr2zDgGt84lrgvmwcQfPSzgUzjZFYxVZOu8Ef aJOq4Qqk2jvwdCrmpiknnRGHECSP2yLAmtPXPsuMZ/7rVlepdR52QLKrKqOE80uV4lkW pxlg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706021675; x=1706626475; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=5NlMdZIB9itIIAJFmR4aXI2QYcCN1XpZu8tjp+LjBQw=; b=ZHNTrSZEXIx7IInq3bYCTbGJZoZZpX/SxkgTcE0yVU1B3DA3k2pGAvNLkvq0Omy0Ho iOyup63bqJL1tqbcNI5EAL4K4EXsd6gOf6oqrMdgIOsqmY9GJb44ezk3JT+b52X5/8w2 G0L/roPisBqjCzY0aGE29Baflys5WaToo2DjHzSO0clf21vegLIJ4qqhLio799M8UDo0 79Pr/+clOdJWJyLUSvXNS8v9N+br2tF3qCyGToarDLnqfJN3nVU/Ew8OWvbUUwTIGskp MfQ/SPefPS5u+uioBnGE8beXNnYRAdIlWEV+ae4EpLLNqiVsI/ptgjSu5huOTgD1M63Q Lt0w== X-Gm-Message-State: AOJu0Yzl0UBYCB3Ono8cgKmFHI4vAbo5irregMGP+262lNJ4ZmI9snki LIZSmK0NLpbtch59jqdiix73CfccnD/4exqubJuzn98n0BNRRXdX6hihfkygDH6IxbxW4oXdWbI eSNCliqWj+YNSCEdJtotRRu79RfKO9wK1zG/7kNYnJ+KDPHZcmj0PIKrQGTpqCtxdONCtHOKAjA 4qACuZx8xGZv715p473T3XpWGUuBW1v68TtgVCWFeu X-Google-Smtp-Source: AGHT+IE0mz02C+Ivqu8wGhKB2y2ML4VtaCHrQb1w9vnhEFZpeYhuM5KdysIxEKgjg0bqqNIKz/CAJzIj X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:600c:519b:b0:40d:5b86:5fee with SMTP id fa27-20020a05600c519b00b0040d5b865feemr11315wmb.3.1706021675261; Tue, 23 Jan 2024 06:54:35 -0800 (PST) Date: Tue, 23 Jan 2024 15:53:05 +0100 In-Reply-To: <20240123145258.1462979-52-ardb+git@google.com> Mime-Version: 1.0 References: <20240123145258.1462979-52-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=1837; i=ardb@kernel.org; h=from:subject; bh=/dnFr10N9WRRkLMCwbVhnPYzrkgM/mTgegkcttUfcHM=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIXX9pYv/7p+tbYr5zr3D8ZDlnZQOg4nrO/78TWD/efzy5 bOB0W0tHaUsDGIcDLJiiiwCs/++23l6olSt8yxZmDmsTCBDGLg4BWAivxoY/sro2705dl3y0UWT QOGKVXN87/U47yx78neVZnfYo3uK/7IZGXbYZ2UJZXHkivevZthd4nkg7f7S4E1p369v8e7/M/F zHg8A X-Mailer: git-send-email 2.43.0.429.g432eaa2c6b-goog Message-ID: <20240123145258.1462979-58-ardb+git@google.com> Subject: [PATCH v7 06/50] arm64: mm: Reclaim unused vmemmap region for vmalloc use From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240123_065437_794023_9C110483 X-CRM114-Status: GOOD ( 14.92 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel The vmemmap array is statically sized based on the maximum supported size of the virtual address space, but it is located inside the upper VA region, which is statically sized based on the *minimum* supported size of the VA space. This doesn't matter much when using 64k pages, which is the only configuration that currently supports 52-bit virtual addressing. However, upcoming LPA2 support will change this picture somewhat, as in that case, the vmemmap array will take up more than 25% of the upper VA region when using 4k pages. Given that most of this space is never used when running on a system that does not support 52-bit virtual addressing, let's reclaim the unused vmemmap area in that case. Acked-by: Mark Rutland Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/pgtable.h | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 2aa2e3c961d7..522c21348ae8 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -18,11 +18,15 @@ * VMALLOC range. * * VMALLOC_START: beginning of the kernel vmalloc space - * VMALLOC_END: extends to the available space below vmemmap, PCI I/O space - * and fixed mappings + * VMALLOC_END: extends to the available space below vmemmap */ #define VMALLOC_START (MODULES_END) +#if VA_BITS == VA_BITS_MIN #define VMALLOC_END (VMEMMAP_START - SZ_8M) +#else +#define VMEMMAP_UNUSED_NPAGES ((_PAGE_OFFSET(vabits_actual) - PAGE_OFFSET) >> PAGE_SHIFT) +#define VMALLOC_END (VMEMMAP_START + VMEMMAP_UNUSED_NPAGES * sizeof(struct page) - SZ_8M) +#endif #define vmemmap ((struct page *)VMEMMAP_START - (memstart_addr >> PAGE_SHIFT)) From patchwork Tue Jan 23 14:53:06 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13527535 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 1002DC47DDB for ; Tue, 23 Jan 2024 14:55:18 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=idHjmJTOHePxpUY8nG2b7Ql+E+uSddcqLP8Mc66EN5I=; b=Tmn6cRFpZTYPmRfHkYuu7FrtsG lmrk0gLzqp3ObWtBweJAvMD5QP/O1hxp3j3q8dNe6uLyh3cIs1hOtWcYWuj8lckY+ei7zmmXvUJUO T27J9vnu3+MuKLXfkHF0KDlpF6Xlw9nVFTA8Q0ShrSjQhDwGs0i67jwh16NYVbsqWr2o5/PoNs4XP 0/03QlFAH40UgHDNo6w7AE8R69R4rJjQcKLl6pMBJCXPQg+u3kzzhI4MZdMs/VOn7EXDzrJHGtuBD nlo4Rk1a1UDtqPOGP0Hudw/PPw69LtX/DHNpnbT9p/NL19MVq/M+4753E82Z7uRF6viU+gaQ5ZQqZ 2T2YLfWQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rSIAu-00GtmW-2c; Tue, 23 Jan 2024 14:54:52 +0000 Received: from mail-yb1-xb4a.google.com ([2607:f8b0:4864:20::b4a]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rSIAg-00GtfY-2U for linux-arm-kernel@lists.infradead.org; Tue, 23 Jan 2024 14:54:40 +0000 Received: by mail-yb1-xb4a.google.com with SMTP id 3f1490d57ef6-dc253fee264so6518774276.2 for ; Tue, 23 Jan 2024 06:54:38 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1706021677; x=1706626477; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=62asg65Pu52oGNnRn/JupEiRgpOjTOLJtlcIrLV4Xuw=; b=kx/DDRSlv9nEwqgQltvbzVgAJ+cdYd5UEcWdgdtPWWPa1qTx1Ayaluv0R9dNgU+92P 8WY63475YQEtUO8pb7qDIntkPVsqFbyq6YihFzOUcnpq//i360Cb1GHjm70I3sP+QWt6 cQ0TxZy+Zf4zHtC05+OgaiJSaFP/s7GRaIFCTYwafBGaiPDPiqfax7OqTpfWwKtLEDE+ BrNxZ3jux7ohamTiiApXWmVApCc4Md6U/zPvh/N5Kx6m8wJ2vAJj8rBR59FwjJ0HgH6t lq2CvuqgZ2WKLD9tzoEdnfN0T4uWTPX0jW6A1Y150ItjMOYqYDKimSOmeDfBHCIJIerl WwZA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706021677; x=1706626477; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=62asg65Pu52oGNnRn/JupEiRgpOjTOLJtlcIrLV4Xuw=; b=rZ/LSTUZDhcyLcfFmVm5ULKk0uncwR45LGaEELoiCB0RysW53+68BrwHZlBuD9Nizi aI4YqHszLSc8iPbDMAIvCWmug6gIAPWkR+pzW6+kx2HKqVjvlfKR+Vds/yJrfPv1KUJr GiN/Mc04nknd7LNilM8byrori3Rh80iZpx35jzIaafHW7gWqiRc7ox53Xs5Uh8pbPGWK uBSy0KI6RAQCyutGbCl4G7h8At/RuYxWEyJC8cB2g+sLcjWvTxT1aCYf+MSgiGyuVyoS xMrkFRPCq8KZHRXM0+T65EY3oGPBXuMkHdKcSYpBonZvF+FCKxX/9s7sul+IRw+xgj26 jHVw== X-Gm-Message-State: AOJu0Yz4chzj96H1ay1gu9iehOkcVERLUxA/I+3ageZJ+F9QBpruuhop VNDo/CnSyYz/1YRcIyHKNgRrNG2nO8ExePyjjF+wqu8+cW9DFukDTUcyx27ObeLlwKyKVG3RoXc +G//d9bwBfNVGtGOcvuzmh9JQ+lgZ2zdqCOlo0KdJfLxZgqeOX46oSZbvyegfum5SkAUdu6ZA+f SnIS03/0+GaBXpGWelPXDiPue2Wy3aCjzi5TS78q6N X-Google-Smtp-Source: AGHT+IGXlNZxzKmWFxzplS7FfMzuf3DZ3E6vXsGh729PWZU+KsnfhlTxLRUQtXx/+SspGy83loRYLkiM X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a25:8209:0:b0:dc2:2ae2:6ca with SMTP id q9-20020a258209000000b00dc22ae206camr2886215ybk.5.1706021677656; Tue, 23 Jan 2024 06:54:37 -0800 (PST) Date: Tue, 23 Jan 2024 15:53:06 +0100 In-Reply-To: <20240123145258.1462979-52-ardb+git@google.com> Mime-Version: 1.0 References: <20240123145258.1462979-52-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=2810; i=ardb@kernel.org; h=from:subject; bh=0rBiGe78OB3uG2sV1+OHG9RVJcuVmafaaV1uQQAQPqk=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIXX9pUuBzo5a/p+ZFbW+7fu40IdZr+7ZpyOlJctuiGiGs jaEz+DtKGVhEONgkBVTZBGY/ffdztMTpWqdZ8nCzGFlAhnCwMUpABO5bsnwP9gzuHDinIt2J5XY Wzcv2vxSfcmhpNQfE7jfNrou2LV9miAjw3GlRYKR13R2n9C9f/Om2qpSkyalDGuV+nsiC76qLPP /xwoA X-Mailer: git-send-email 2.43.0.429.g432eaa2c6b-goog Message-ID: <20240123145258.1462979-59-ardb+git@google.com> Subject: [PATCH v7 07/50] arm64: kaslr: Adjust randomization range dynamically From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240123_065438_810196_D395B1B2 X-CRM114-Status: GOOD ( 15.91 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel Currently, we base the KASLR randomization range on a rough estimate of the available space in the upper VA region: the lower 1/4th has the module region and the upper 1/4th has the fixmap, vmemmap and PCI I/O ranges, and so we pick a random location in the remaining space in the middle. Once we enable support for 5-level paging with 4k pages, this no longer works: the vmemmap region, being dimensioned to cover a 52-bit linear region, takes up so much space in the upper VA region (the size of which is based on a 48-bit VA space for compatibility with non-LVA hardware) that the region above the vmalloc region takes up more than a quarter of the available space. So instead of a heuristic, let's derive the randomization range from the actual boundaries of the vmalloc region. Acked-by: Mark Rutland Signed-off-by: Ard Biesheuvel --- arch/arm64/kernel/image-vars.h | 2 ++ arch/arm64/kernel/pi/kaslr_early.c | 11 ++++++----- 2 files changed, 8 insertions(+), 5 deletions(-) diff --git a/arch/arm64/kernel/image-vars.h b/arch/arm64/kernel/image-vars.h index 5e4dc72ab1bd..e931ce078a00 100644 --- a/arch/arm64/kernel/image-vars.h +++ b/arch/arm64/kernel/image-vars.h @@ -36,6 +36,8 @@ PROVIDE(__pi___memcpy = __pi_memcpy); PROVIDE(__pi___memmove = __pi_memmove); PROVIDE(__pi___memset = __pi_memset); +PROVIDE(__pi_vabits_actual = vabits_actual); + #ifdef CONFIG_KVM /* diff --git a/arch/arm64/kernel/pi/kaslr_early.c b/arch/arm64/kernel/pi/kaslr_early.c index 17bff6e399e4..b9e0bb4bc6a9 100644 --- a/arch/arm64/kernel/pi/kaslr_early.c +++ b/arch/arm64/kernel/pi/kaslr_early.c @@ -14,6 +14,7 @@ #include #include +#include /* taken from lib/string.c */ static char *__strstr(const char *s1, const char *s2) @@ -87,7 +88,7 @@ static u64 get_kaslr_seed(void *fdt) asmlinkage u64 kaslr_early_init(void *fdt) { - u64 seed; + u64 seed, range; if (is_kaslr_disabled_cmdline(fdt)) return 0; @@ -102,9 +103,9 @@ asmlinkage u64 kaslr_early_init(void *fdt) /* * OK, so we are proceeding with KASLR enabled. Calculate a suitable * kernel image offset from the seed. Let's place the kernel in the - * middle half of the VMALLOC area (VA_BITS_MIN - 2), and stay clear of - * the lower and upper quarters to avoid colliding with other - * allocations. + * 'middle' half of the VMALLOC area, and stay clear of the lower and + * upper quarters to avoid colliding with other allocations. */ - return BIT(VA_BITS_MIN - 3) + (seed & GENMASK(VA_BITS_MIN - 3, 0)); + range = (VMALLOC_END - KIMAGE_VADDR) / 2; + return range / 2 + (((__uint128_t)range * seed) >> 64); } From patchwork Tue Jan 23 14:53:07 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13527539 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 9F0BCC47258 for ; Tue, 23 Jan 2024 14:55:24 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=xaXeomi/WX6hRyTOAwTUjJJWjFGOpsvJnVXVBqxHqhU=; b=ep1b8tQjRKgp30cCF/ZHWqo6/x 1SJM5X4t8WOQsviZbmXFoBhikp/74+ieVIq4JD1MdgP3ge19TzAvr3BWH6FGh6qcoOImyGblCms2w COVDJ/ahFvZxklA22imAK++/dYQGndpnTFe5wvZBO/TO1sNfachyOpN1oI/Ct2tFoMcsak4x2YoT+ 4ofu6Uz41+pYuTpxQo5XOsFZ/zpFSWJuenyEUM8T6l3irlWKmNoKFyVKif1XYCur5RU0KvuZ2SiDz xWFJScl9LB0nYeYyci1D87gcbyCjsOkENv/RopvUCFziA5qYMBNSvCbB/LARPLLLZ4OEDG8ARXejF 8yCC/pfA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rSIAv-00GtnO-2o; Tue, 23 Jan 2024 14:54:53 +0000 Received: from mail-yw1-x1149.google.com ([2607:f8b0:4864:20::1149]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rSIAj-00GtgT-1A for linux-arm-kernel@lists.infradead.org; Tue, 23 Jan 2024 14:54:43 +0000 Received: by mail-yw1-x1149.google.com with SMTP id 00721157ae682-5fc6463b0edso72293587b3.0 for ; Tue, 23 Jan 2024 06:54:40 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1706021680; x=1706626480; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=uHapg+q84iY+9DVDrmY/Rzm5S7AajVfJPXOm3N8uiK0=; b=lP8ZahdiA7AZz7DOf9kIJnuGCkpG+dSzmQVlMA3G4h4tU7YkPDavv02NGcEhyYa6GL GdbGXOsi1MEfNOpEzotpX7Jpq9FemYUVFygi/LEjawk/WW9pKtWuMg4d2yM6uXQxkd1Q Ox4bb+01Ow4SWv+cUKMkfHAwEjQaZXVRinJu4Fm+bh+VpMywvyUtA3hqeL5x8SK4dDB3 DDYhwlQeiA185ayeakS51dwpU0ZBQwaJiHzleeYTAd3yh7OEnSqsvYKzncUmOVDfYF++ qV13NXhtS8rKhQ6BoOso4fuPGxPGOD5Oum9a63lQHqc7+AoNg3COe9t0Ze3MCsfv8O4U 70Yw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706021680; x=1706626480; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=uHapg+q84iY+9DVDrmY/Rzm5S7AajVfJPXOm3N8uiK0=; b=rn5cVMhgQA4ArCmXR+K2JzUCrTKrfuK3mx/Cvr/rN65LBV1837geRMc67lNYpSCal3 xiS3U5I68dlr6rBZZ1GwW9ka9D8Zp/1++fI5M8ZLbXYHlpa7/ypsIQidFcEm32ikOtz7 H9Tth8BXFA1mIoEMdvcAWcOctwyTXvfZIdT0AfUtPzYIKdfTmua7fJ/0zL2YHOzXs9fS Udw3dmH3R7iUXYg6TTGPp31zhGhokXyKCm2QtA2wk+1sKPTv0gvS5QtW/c+gf/S3vjcN dageHJXY6HYMXW3t6ETrtZ+NYGRD1zI4I0Fx82GFXeJY5QPHguLOfFeXdBXd6703CS+W z1qg== X-Gm-Message-State: AOJu0Yy1eCeiIuV56uQTD+dzWSAyzjQxt9D6Uu8nUk2fiZe7zCyDewPn cEQe3+i3K5M9D5n9kuZ6qod8QZT5EO/1JdOqkt336CYGvVA3OwZS6g+rjd46zDfQ3For1KlNOoY znRFztxNgFBnTKJlyFa4a7ebwNAWsIhH7w0g0DMp6aq68AZVJ/ux9l5iuFxt8l+wL1dKM5EktjP qUMFTRM3SsMjr/C/k9FIx7OK/mHqqFqgg1D6iF7Ccu X-Google-Smtp-Source: AGHT+IFlm2oI/U4DYho9OPIhPb71qMfc98Ar1kdveeP7X/hjNivrXRQsFZveDGjggZAcU9+w8/93fxkP X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a25:ce92:0:b0:dc2:65e2:58f3 with SMTP id x140-20020a25ce92000000b00dc265e258f3mr350474ybe.7.1706021680150; Tue, 23 Jan 2024 06:54:40 -0800 (PST) Date: Tue, 23 Jan 2024 15:53:07 +0100 In-Reply-To: <20240123145258.1462979-52-ardb+git@google.com> Mime-Version: 1.0 References: <20240123145258.1462979-52-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=6618; i=ardb@kernel.org; h=from:subject; bh=IdH78ZaX1BFe8iHl3LhquM/a8D/A7UQlQ1jqPN1t198=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIXX9pcuyH++tO/b38gbRPWf5Hz3z/6b16fS8pZUH+D/7O 105vXbnt45SFgYxDgZZMUUWgdl/3+08PVGq1nmWLMwcViaQIQxcnAIwkYXvGf6HmLXZ86v0ztYM bF5wZ/7x5pI+V8ZLji1zfupKpHBO6JzO8L+ac5vC1qX3T1YzBJ5x+m/Il8cpz/ax78Xf8nsz0xQ 2qfECAA== X-Mailer: git-send-email 2.43.0.429.g432eaa2c6b-goog Message-ID: <20240123145258.1462979-60-ardb+git@google.com> Subject: [PATCH v7 08/50] arm64: kernel: Manage absolute relocations in code built under pi/ From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240123_065441_399490_4A407EA5 X-CRM114-Status: GOOD ( 24.16 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel The mini C runtime runs before relocations are processed, and so it cannot rely on statically initialized pointer variables. Add a check to ensure that such code does not get introduced by accident, by going over the relocations in each object, identifying the ones that operate on data sections that are part of the executable image, and raising an error if any relocations of type R_AARCH64_ABS64 exist. Note that such relocations are permitted in other places (e.g., debug sections) and will never occur in compiler generated code sections when using the small code model, so only check sections that have SHF_ALLOC set and SHF_EXECINSTR cleared. To accommodate cases where statically initialized symbol references are unavoidable, introduce a special case for ELF input data sections that have ".rodata.prel64" in their names, and in these cases, instead of rejecting any encountered ABS64 relocations, convert them into PREL64 relocations, which don't require any runtime fixups. Note that the code in question must still be modified to deal with this, as it needs to convert the 64-bit signed offsets into absolute addresses before use. Signed-off-by: Ard Biesheuvel --- arch/arm64/kernel/pi/Makefile | 9 +- arch/arm64/kernel/pi/pi.h | 18 +++ arch/arm64/kernel/pi/relacheck.c | 130 ++++++++++++++++++++ 3 files changed, 155 insertions(+), 2 deletions(-) diff --git a/arch/arm64/kernel/pi/Makefile b/arch/arm64/kernel/pi/Makefile index c844a0546d7f..bc32a431fe35 100644 --- a/arch/arm64/kernel/pi/Makefile +++ b/arch/arm64/kernel/pi/Makefile @@ -22,11 +22,16 @@ KCSAN_SANITIZE := n UBSAN_SANITIZE := n KCOV_INSTRUMENT := n +hostprogs := relacheck + +quiet_cmd_piobjcopy = $(quiet_cmd_objcopy) + cmd_piobjcopy = $(cmd_objcopy) && $(obj)/relacheck $(@) $(<) + $(obj)/%.pi.o: OBJCOPYFLAGS := --prefix-symbols=__pi_ \ --remove-section=.note.gnu.property \ --prefix-alloc-sections=.init -$(obj)/%.pi.o: $(obj)/%.o FORCE - $(call if_changed,objcopy) +$(obj)/%.pi.o: $(obj)/%.o $(obj)/relacheck FORCE + $(call if_changed,piobjcopy) $(obj)/lib-%.o: $(srctree)/lib/%.c FORCE $(call if_changed_rule,cc_o_c) diff --git a/arch/arm64/kernel/pi/pi.h b/arch/arm64/kernel/pi/pi.h new file mode 100644 index 000000000000..7c2d9bbf0ff9 --- /dev/null +++ b/arch/arm64/kernel/pi/pi.h @@ -0,0 +1,18 @@ +// SPDX-License-Identifier: GPL-2.0-only +// Copyright 2023 Google LLC +// Author: Ard Biesheuvel + +#define __prel64_initconst __section(".init.rodata.prel64") + +#define PREL64(type, name) union { type *name; prel64_t name ## _prel; } + +#define prel64_pointer(__d) (typeof(__d))prel64_to_pointer(&__d##_prel) + +typedef volatile signed long prel64_t; + +static inline void *prel64_to_pointer(const prel64_t *offset) +{ + if (!*offset) + return NULL; + return (void *)offset + *offset; +} diff --git a/arch/arm64/kernel/pi/relacheck.c b/arch/arm64/kernel/pi/relacheck.c new file mode 100644 index 000000000000..b0cd4d0d275b --- /dev/null +++ b/arch/arm64/kernel/pi/relacheck.c @@ -0,0 +1,130 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright (C) 2023 - Google LLC + * Author: Ard Biesheuvel + */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#if __BYTE_ORDER__ == __ORDER_LITTLE_ENDIAN__ +#define HOST_ORDER ELFDATA2LSB +#elif __BYTE_ORDER__ == __ORDER_BIG_ENDIAN__ +#define HOST_ORDER ELFDATA2MSB +#endif + +static Elf64_Ehdr *ehdr; +static Elf64_Shdr *shdr; +static const char *strtab; +static bool swap; + +static uint64_t swab_elfxword(uint64_t val) +{ + return swap ? __builtin_bswap64(val) : val; +} + +static uint32_t swab_elfword(uint32_t val) +{ + return swap ? __builtin_bswap32(val) : val; +} + +static uint16_t swab_elfhword(uint16_t val) +{ + return swap ? __builtin_bswap16(val) : val; +} + +int main(int argc, char *argv[]) +{ + struct stat stat; + int fd, ret; + + if (argc < 3) { + fprintf(stderr, "file arguments missing\n"); + exit(EXIT_FAILURE); + } + + fd = open(argv[1], O_RDWR); + if (fd < 0) { + fprintf(stderr, "failed to open %s\n", argv[1]); + exit(EXIT_FAILURE); + } + + ret = fstat(fd, &stat); + if (ret < 0) { + fprintf(stderr, "failed to stat() %s\n", argv[1]); + exit(EXIT_FAILURE); + } + + ehdr = mmap(0, stat.st_size, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0); + if (ehdr == MAP_FAILED) { + fprintf(stderr, "failed to mmap() %s\n", argv[1]); + exit(EXIT_FAILURE); + } + + swap = ehdr->e_ident[EI_DATA] != HOST_ORDER; + shdr = (void *)ehdr + swab_elfxword(ehdr->e_shoff); + strtab = (void *)ehdr + + swab_elfxword(shdr[swab_elfhword(ehdr->e_shstrndx)].sh_offset); + + for (int i = 0; i < swab_elfhword(ehdr->e_shnum); i++) { + unsigned long info, flags; + bool prel64 = false; + Elf64_Rela *rela; + int numrela; + + if (swab_elfword(shdr[i].sh_type) != SHT_RELA) + continue; + + /* only consider RELA sections operating on data */ + info = swab_elfword(shdr[i].sh_info); + flags = swab_elfxword(shdr[info].sh_flags); + if ((flags & (SHF_ALLOC | SHF_EXECINSTR)) != SHF_ALLOC) + continue; + + /* + * We generally don't permit ABS64 relocations in the code that + * runs before relocation processing occurs. If statically + * initialized absolute symbol references are unavoidable, they + * may be emitted into a *.rodata.prel64 section and they will + * be converted to place-relative 64-bit references. This + * requires special handling in the referring code. + */ + if (strstr(strtab + swab_elfword(shdr[info].sh_name), + ".rodata.prel64")) { + prel64 = true; + } + + rela = (void *)ehdr + swab_elfxword(shdr[i].sh_offset); + numrela = swab_elfxword(shdr[i].sh_size) / sizeof(*rela); + + for (int j = 0; j < numrela; j++) { + uint64_t info = swab_elfxword(rela[j].r_info); + + if (ELF64_R_TYPE(info) != R_AARCH64_ABS64) + continue; + + if (prel64) { + /* convert ABS64 into PREL64 */ + info ^= R_AARCH64_ABS64 ^ R_AARCH64_PREL64; + rela[j].r_info = swab_elfxword(info); + } else { + fprintf(stderr, + "Unexpected absolute relocations detected in %s\n", + argv[2]); + close(fd); + unlink(argv[1]); + exit(EXIT_FAILURE); + } + } + } + close(fd); + return 0; +} From patchwork Tue Jan 23 14:53:08 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13527537 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id C846DC47DDF for ; Tue, 23 Jan 2024 14:55:21 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=HIJMhfIF8ruJN5x4e5a/t1xakwdnrj4BT6Re2+spWYw=; b=Nm2274OtYBozUx2tXn7zSVV6/v Go/5cbSRJCbx3k8XaRUeHovkf+Wn0a+Bq8xZyOPIlnBv2eol2EKEyUgiOwLv4wPmmuGbOptmhxVf4 A5RKh2cmTFFtn6/3NYzDOQjl8r0+KUEJhXcjYV9An6zYoifgV6GTX5lPaewS/qGsGPzGXfQStgqNt /Y4IZNfWFtBnUMezSrsWBPX4US2YRLxMnO87qM5+BmfxuoOU4leM6LaI9fipq8SomKZFL0jCjRJOA +STMYWWoTypAwRi2FS+BDkwKeZnlBTwmyRU5wHvySbyg0HGTUNqrFiaPy4eIVzIYETKY4ZUCIe0QP pRyP/fiQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rSIAx-00GtoZ-1Z; Tue, 23 Jan 2024 14:54:55 +0000 Received: from mail-yb1-xb49.google.com ([2607:f8b0:4864:20::b49]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rSIAl-00Gthb-2W for linux-arm-kernel@lists.infradead.org; Tue, 23 Jan 2024 14:54:45 +0000 Received: by mail-yb1-xb49.google.com with SMTP id 3f1490d57ef6-dbdb46770d7so6224489276.1 for ; Tue, 23 Jan 2024 06:54:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1706021682; x=1706626482; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=IrdsYyXQe3FCauN3JAkyHZXj9TK2DtbhA8RCtm550mE=; b=RHxT2ZBHI5CCqlUkyYOOvnnn8LlNoT1ZKIq0iz0sX6TitOf0A1s5GKR1S3ozmA0Tx6 heKu8Sj+5QYTLGJuMm3QvD4rEaWxEuRAduWVx67XBHmwidrlS4S9zoHOjPWSFjlnl6x1 79lPbYqPet8uZMbxM18jNoj/60SujhEg4t/9io8PxrmC+EmlY4E+0/JXfJpcG6DviBhy RbJM7lPgKaDIdKFsPWGLGMqewvUNTRaz6b4VpGncBrJIetKTANNhbEACQTKF8reMfOCM JXem5XyoI5lR5rjFvLo5J+eC8Z9u43l74J2efhDLSPiZkOd/FyO5KUxaGNoHV8aps/hE /ehw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706021682; x=1706626482; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=IrdsYyXQe3FCauN3JAkyHZXj9TK2DtbhA8RCtm550mE=; b=qOeQuqPfkxFR/VPlNjfrWZCClvOLtxjGxd+TjjL07UEDWB8Gpy7mxvEm2zk2apoCQe Ab2MWD95ocUQk3441uYy9sSPqNP+nIvJA7JTwHIqHdcXnFXzn3jtBZHcqS13f6Dm30UI bOI0lkFB57utlx/3tpUyqvIuRrxG3evVpWHFTUIhR1Go61E7kAVjB8y7xW/JLcC3yqTX 618vVH60cMuzN8fuGEDdvPAe9eR2AsrOFycN8zIPao/uSlrtXmO5+JS7GTT5cRPkHU5j vRB95jSj1UOIDNdJMP3tDqPsyi9mi5jQ9OVbcfKiZmaOu2tVCqO6/dvN7OsgNSDqg/RY 1E2A== X-Gm-Message-State: AOJu0YxolPMCJQ2PtIOYVi+DIoJ/1ZeQENmazkTtgSzf6bM/cQ7EgJM2 I2ZnsmsHU/lYIK/ZOCX1paMuaZ/XdpQIwLVG7EMTWrG9wsYxz3fUjMjU9emPiCZa3ck5six+/Ms UqQ+Ofo0TMeDgutOhcSIwTvYPzPheijt9dwUcJTTTcRHKUrkfVg/MoiBy0ztZ94lxnV3YZJb27G Mk4wiNBD58e5dlXF8DytTJF7aT7ZXx+0z2UulrZD/e X-Google-Smtp-Source: AGHT+IGqQG6n2fHa07Moal/lRW0FVccLcmouerMa4NYqIIjgo5e+u7eoIF14o+GpAKNg6qomKESP6zRk X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:6902:1b12:b0:db5:4766:e363 with SMTP id eh18-20020a0569021b1200b00db54766e363mr2616012ybb.6.1706021682630; Tue, 23 Jan 2024 06:54:42 -0800 (PST) Date: Tue, 23 Jan 2024 15:53:08 +0100 In-Reply-To: <20240123145258.1462979-52-ardb+git@google.com> Mime-Version: 1.0 References: <20240123145258.1462979-52-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=3211; i=ardb@kernel.org; h=from:subject; bh=iN+q1zt4hvM4Hv4pBU7ECixAWT79q1GdhLb+jhHxr5I=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIXX9pSt907U41ezjzLvXaAVOZ78ZwpN48vIxr7T/HzbJC 16WeVrbUcrCIMbBICumyCIw+++7nacnStU6z5KFmcPKBDKEgYtTACZy4AfDX6l/Jkf23epamfnw 1zbWi7d8i9Sdon8JzbD7q9lTGzc1X4+R4dSeM2ejo6/+kCqYK3rh3NTW68+elEt2HS1+ofL9yBS lFFYA X-Mailer: git-send-email 2.43.0.429.g432eaa2c6b-goog Message-ID: <20240123145258.1462979-61-ardb+git@google.com> Subject: [PATCH v7 09/50] arm64: kernel: Don't rely on objcopy to make code under pi/ __init From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240123_065443_821805_F7DE782F X-CRM114-Status: GOOD ( 12.97 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel We will add some code under pi/ that contains global variables that should not end up in __initdata, as they will not be writable via the initial ID map. So only rely on objcopy for making the libfdt code __init, and use explicit annotations for the rest. Signed-off-by: Ard Biesheuvel --- arch/arm64/kernel/pi/Makefile | 6 ++++-- arch/arm64/kernel/pi/kaslr_early.c | 16 +++++++++------- 2 files changed, 13 insertions(+), 9 deletions(-) diff --git a/arch/arm64/kernel/pi/Makefile b/arch/arm64/kernel/pi/Makefile index bc32a431fe35..2bbe866417d4 100644 --- a/arch/arm64/kernel/pi/Makefile +++ b/arch/arm64/kernel/pi/Makefile @@ -28,11 +28,13 @@ quiet_cmd_piobjcopy = $(quiet_cmd_objcopy) cmd_piobjcopy = $(cmd_objcopy) && $(obj)/relacheck $(@) $(<) $(obj)/%.pi.o: OBJCOPYFLAGS := --prefix-symbols=__pi_ \ - --remove-section=.note.gnu.property \ - --prefix-alloc-sections=.init + --remove-section=.note.gnu.property $(obj)/%.pi.o: $(obj)/%.o $(obj)/relacheck FORCE $(call if_changed,piobjcopy) +# ensure that all the lib- code ends up as __init code and data +$(obj)/lib-%.pi.o: OBJCOPYFLAGS += --prefix-alloc-sections=.init + $(obj)/lib-%.o: $(srctree)/lib/%.c FORCE $(call if_changed_rule,cc_o_c) diff --git a/arch/arm64/kernel/pi/kaslr_early.c b/arch/arm64/kernel/pi/kaslr_early.c index b9e0bb4bc6a9..167081b30a15 100644 --- a/arch/arm64/kernel/pi/kaslr_early.c +++ b/arch/arm64/kernel/pi/kaslr_early.c @@ -17,7 +17,7 @@ #include /* taken from lib/string.c */ -static char *__strstr(const char *s1, const char *s2) +static char *__init __strstr(const char *s1, const char *s2) { size_t l1, l2; @@ -33,7 +33,7 @@ static char *__strstr(const char *s1, const char *s2) } return NULL; } -static bool cmdline_contains_nokaslr(const u8 *cmdline) +static bool __init cmdline_contains_nokaslr(const u8 *cmdline) { const u8 *str; @@ -41,7 +41,7 @@ static bool cmdline_contains_nokaslr(const u8 *cmdline) return str == cmdline || (str > cmdline && *(str - 1) == ' '); } -static bool is_kaslr_disabled_cmdline(void *fdt) +static bool __init is_kaslr_disabled_cmdline(void *fdt) { if (!IS_ENABLED(CONFIG_CMDLINE_FORCE)) { int node; @@ -67,17 +67,19 @@ static bool is_kaslr_disabled_cmdline(void *fdt) return cmdline_contains_nokaslr(CONFIG_CMDLINE); } -static u64 get_kaslr_seed(void *fdt) +static u64 __init get_kaslr_seed(void *fdt) { + static char const chosen_str[] __initconst = "chosen"; + static char const seed_str[] __initconst = "kaslr-seed"; int node, len; fdt64_t *prop; u64 ret; - node = fdt_path_offset(fdt, "/chosen"); + node = fdt_path_offset(fdt, chosen_str); if (node < 0) return 0; - prop = fdt_getprop_w(fdt, node, "kaslr-seed", &len); + prop = fdt_getprop_w(fdt, node, seed_str, &len); if (!prop || len != sizeof(u64)) return 0; @@ -86,7 +88,7 @@ static u64 get_kaslr_seed(void *fdt) return ret; } -asmlinkage u64 kaslr_early_init(void *fdt) +asmlinkage u64 __init kaslr_early_init(void *fdt) { u64 seed, range; From patchwork Tue Jan 23 14:53:09 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13527540 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 9CEB9C47DDB for ; Tue, 23 Jan 2024 14:55:53 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=oKyJWKvr6rTITLaVhFvnkp2hJTslZxdvBq1ezbm4gs8=; b=gnvkM7z6VnlsUku2Vf2hBJrmpv QH03rLILUIF1Zu/pjcTfEm6LznmBLULKsIiTQYPb8MChp8AMwLX9/0WAg7L5BPWol3vAPUeXrhPGs uLTPk1teuUslf6VyXPcjx42jdLtBlTsMCJeSXWzFyn1Qy9wPmbgetlKCkIA+8veZlu9iJPBfhSgaP gT7M4bpbr/HKM+fJSCJ9MfxX+SGVK6iy4w4uzWkYEI6ozS9Itxgb3FhDDtBdT1Q8Gx0XuR0ZphIr4 1gzpFSezx9zJiZvbvEYwv4NKPobDMX/r6jx67jeTrzCbETtxfQlsqHBGnOwdbVMhVgANw3BYaEEaH u7QI+glg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rSIBQ-00Gu7r-1Z; Tue, 23 Jan 2024 14:55:24 +0000 Received: from mail-wm1-x34a.google.com ([2a00:1450:4864:20::34a]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rSIAp-00GtiV-0L for linux-arm-kernel@lists.infradead.org; Tue, 23 Jan 2024 14:54:49 +0000 Received: by mail-wm1-x34a.google.com with SMTP id 5b1f17b1804b1-40e439092a0so21140725e9.0 for ; Tue, 23 Jan 2024 06:54:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1706021685; x=1706626485; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=YRSTRS4jCk6d9EWkjEv3aLYLOZsc3QM2VaALgc6acdA=; b=0t2xkoSFBkWZ3eal1NiHxUUdpQv0O0IzGlyWdvNCYnTqGU+h1NBT+tT2pM6SKlwLVm Hz+MIX4piZxf/7jrXkWKp7Qbl3/jQkrIeFOVerI1Yj64qWBs6jcoUX43qtANzRIJyRpY 7vAxAVPHuiAKorJ4VbL4Set2mAsp9pLJk/D96L4p54/DbJ2zppszGdOIdGOXpo3PDL1N SJm5TrZFB7YAdO7NHWLQH4aGiiPqDhQe/bpt/rbWvryPCmtjZe1CmrS3dopEyqDNfF85 gH8/r0pN5OmTqB0ZVB1cZinPYiC96KOxGTTAxsg19lLxYnMPjslaV+gef9pyiZU7CzCf GEBA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706021685; x=1706626485; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=YRSTRS4jCk6d9EWkjEv3aLYLOZsc3QM2VaALgc6acdA=; b=vVh7HkRCIDHS+Y4OJG7+pcpB8el7xiW12jdV8mFfMuwjpNaN7dingCCkS340KOyepa 3dO0BLec9V3ooZKR/fDUGu9fMM3MrA0+vg+dDjDt0Cz2nWzZeK+O4b2zSaXNvwWO7LqZ XErT6U89YGopzza61+zgmJJvc0ZQKQtNB4v0qQkKpepVcvdaGdFEFWb1PnhZXaIfw6In MAhHQpQi6Gy9hN6zp49h+BUrjOy5OLuDsH71ODHssBU13vXRap4hKeGmTxhKNd+LYL3q PeYnUwpsV2wTaVt4wCeVf06w8JVBMA95Nkhu5kVnYAErEWzO7gWS8HgtPsn3iKpLBlp+ xI+g== X-Gm-Message-State: AOJu0YyAzXcoc/P3wgva0LDkXusJ49phtLOzUkIDfVxSrfYQHue4o6Fv zi408YmsVKGCmLGYzG4T0lh+jETJ60GHBHYSoTbilwPi8DJAReLHciJ5bnKGX+bKQdpUJ6bxrl/ o2ReOdDNikqkqSIpEyRMFGuDRIIoyEhcUcsdFkoAeda3D9xDGPqjmE8rf4XUPhkm7iCZS3XUEGX lhGtQmEyy8KBUUl/ysTCnKs8UqEv6F1anbfyFx2ueu X-Google-Smtp-Source: AGHT+IHZicYlO2vhkmNBI5iO10IX0olIl/UutRdrABhJmSs4lMJO0w5OlOgtIF/LZcDdMXHEjiv9IwAS X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:600c:3d95:b0:40e:c031:34a9 with SMTP id bi21-20020a05600c3d9500b0040ec03134a9mr2899wmb.2.1706021684715; Tue, 23 Jan 2024 06:54:44 -0800 (PST) Date: Tue, 23 Jan 2024 15:53:09 +0100 In-Reply-To: <20240123145258.1462979-52-ardb+git@google.com> Mime-Version: 1.0 References: <20240123145258.1462979-52-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=10134; i=ardb@kernel.org; h=from:subject; bh=sPl5yZoKAAQREswBBhLHjj8luRVvQ7sRFvQ8bmtBjOU=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIXX9pasT/7Bvv1+xytpZL3SJaHLrai13QfX150qFT277u ic6u0a/o5SFQYyDQVZMkUVg9t93O09PlKp1niULM4eVCWQIAxenAExk6kaG3+yrZ1yo/1LD91vJ 6+rlbR5/NYQ3R2nwmu74VLDFfovL7whGhos+H9bWa4fxCwXta77iHrOrctOOh5u1uRmurcuzPsd 8lw8A X-Mailer: git-send-email 2.43.0.429.g432eaa2c6b-goog Message-ID: <20240123145258.1462979-62-ardb+git@google.com> Subject: [PATCH v7 10/50] arm64: head: move relocation handling to C code From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240123_065447_153713_34AE4EE3 X-CRM114-Status: GOOD ( 27.63 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel Now that we have a mini C runtime before the kernel mapping is up, we can move the non-trivial relocation processing code out of head.S and reimplement it in C. Signed-off-by: Ard Biesheuvel --- arch/arm64/kernel/Makefile | 3 +- arch/arm64/kernel/head.S | 104 ++------------------ arch/arm64/kernel/pi/Makefile | 5 +- arch/arm64/kernel/pi/relocate.c | 62 ++++++++++++ arch/arm64/kernel/vmlinux.lds.S | 12 ++- 5 files changed, 82 insertions(+), 104 deletions(-) diff --git a/arch/arm64/kernel/Makefile b/arch/arm64/kernel/Makefile index e5d03a7039b4..84db82cf6ba8 100644 --- a/arch/arm64/kernel/Makefile +++ b/arch/arm64/kernel/Makefile @@ -57,7 +57,8 @@ obj-$(CONFIG_ACPI) += acpi.o obj-$(CONFIG_ACPI_NUMA) += acpi_numa.o obj-$(CONFIG_ARM64_ACPI_PARKING_PROTOCOL) += acpi_parking_protocol.o obj-$(CONFIG_PARAVIRT) += paravirt.o -obj-$(CONFIG_RANDOMIZE_BASE) += kaslr.o pi/ +obj-$(CONFIG_RELOCATABLE) += pi/ +obj-$(CONFIG_RANDOMIZE_BASE) += kaslr.o obj-$(CONFIG_HIBERNATION) += hibernate.o hibernate-asm.o obj-$(CONFIG_ELF_CORE) += elfcore.o obj-$(CONFIG_KEXEC_CORE) += machine_kexec.o relocate_kernel.o \ diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S index cab7f91949d8..a8fa64fc30d7 100644 --- a/arch/arm64/kernel/head.S +++ b/arch/arm64/kernel/head.S @@ -81,7 +81,7 @@ * x20 primary_entry() .. __primary_switch() CPU boot mode * x21 primary_entry() .. start_kernel() FDT pointer passed at boot in x0 * x22 create_idmap() .. start_kernel() ID map VA of the DT blob - * x23 primary_entry() .. start_kernel() physical misalignment/KASLR offset + * x23 __primary_switch() physical misalignment/KASLR offset * x24 __primary_switch() linear map KASLR seed * x25 primary_entry() .. start_kernel() supported VA size * x28 create_idmap() callee preserved temp register @@ -389,7 +389,7 @@ SYM_FUNC_START_LOCAL(create_idmap) /* Remap the kernel page tables r/w in the ID map */ adrp x1, _text adrp x2, init_pg_dir - adrp x3, init_pg_end + adrp x3, _end bic x4, x2, #SWAPPER_BLOCK_SIZE - 1 mov_q x5, SWAPPER_RW_MMUFLAGS mov x6, #SWAPPER_BLOCK_SHIFT @@ -779,97 +779,6 @@ SYM_FUNC_START_LOCAL(__no_granule_support) b 1b SYM_FUNC_END(__no_granule_support) -#ifdef CONFIG_RELOCATABLE -SYM_FUNC_START_LOCAL(__relocate_kernel) - /* - * Iterate over each entry in the relocation table, and apply the - * relocations in place. - */ - adr_l x9, __rela_start - adr_l x10, __rela_end - mov_q x11, KIMAGE_VADDR // default virtual offset - add x11, x11, x23 // actual virtual offset - -0: cmp x9, x10 - b.hs 1f - ldp x12, x13, [x9], #24 - ldr x14, [x9, #-8] - cmp w13, #R_AARCH64_RELATIVE - b.ne 0b - add x14, x14, x23 // relocate - str x14, [x12, x23] - b 0b - -1: -#ifdef CONFIG_RELR - /* - * Apply RELR relocations. - * - * RELR is a compressed format for storing relative relocations. The - * encoded sequence of entries looks like: - * [ AAAAAAAA BBBBBBB1 BBBBBBB1 ... AAAAAAAA BBBBBB1 ... ] - * - * i.e. start with an address, followed by any number of bitmaps. The - * address entry encodes 1 relocation. The subsequent bitmap entries - * encode up to 63 relocations each, at subsequent offsets following - * the last address entry. - * - * The bitmap entries must have 1 in the least significant bit. The - * assumption here is that an address cannot have 1 in lsb. Odd - * addresses are not supported. Any odd addresses are stored in the RELA - * section, which is handled above. - * - * Excluding the least significant bit in the bitmap, each non-zero - * bit in the bitmap represents a relocation to be applied to - * a corresponding machine word that follows the base address - * word. The second least significant bit represents the machine - * word immediately following the initial address, and each bit - * that follows represents the next word, in linear order. As such, - * a single bitmap can encode up to 63 relocations in a 64-bit object. - * - * In this implementation we store the address of the next RELR table - * entry in x9, the address being relocated by the current address or - * bitmap entry in x13 and the address being relocated by the current - * bit in x14. - */ - adr_l x9, __relr_start - adr_l x10, __relr_end - -2: cmp x9, x10 - b.hs 7f - ldr x11, [x9], #8 - tbnz x11, #0, 3f // branch to handle bitmaps - add x13, x11, x23 - ldr x12, [x13] // relocate address entry - add x12, x12, x23 - str x12, [x13], #8 // adjust to start of bitmap - b 2b - -3: mov x14, x13 -4: lsr x11, x11, #1 - cbz x11, 6f - tbz x11, #0, 5f // skip bit if not set - ldr x12, [x14] // relocate bit - add x12, x12, x23 - str x12, [x14] - -5: add x14, x14, #8 // move to next bit's address - b 4b - -6: /* - * Move to the next bitmap's address. 8 is the word size, and 63 is the - * number of significant bits in a bitmap entry. - */ - add x13, x13, #(8 * 63) - b 2b - -7: -#endif - ret - -SYM_FUNC_END(__relocate_kernel) -#endif - SYM_FUNC_START_LOCAL(__primary_switch) adrp x1, reserved_pg_dir adrp x2, init_idmap_pg_dir @@ -877,11 +786,11 @@ SYM_FUNC_START_LOCAL(__primary_switch) #ifdef CONFIG_RELOCATABLE adrp x23, KERNEL_START and x23, x23, MIN_KIMG_ALIGN - 1 -#ifdef CONFIG_RANDOMIZE_BASE - mov x0, x22 - adrp x1, init_pg_end + adrp x1, early_init_stack mov sp, x1 mov x29, xzr +#ifdef CONFIG_RANDOMIZE_BASE + mov x0, x22 bl __pi_kaslr_early_init and x24, x0, #SZ_2M - 1 // capture memstart offset seed bic x0, x0, #SZ_2M - 1 @@ -894,7 +803,8 @@ SYM_FUNC_START_LOCAL(__primary_switch) adrp x1, init_pg_dir load_ttbr1 x1, x1, x2 #ifdef CONFIG_RELOCATABLE - bl __relocate_kernel + mov x0, x23 + bl __pi_relocate_kernel #endif ldr x8, =__primary_switched adrp x0, KERNEL_START // __pa(KERNEL_START) diff --git a/arch/arm64/kernel/pi/Makefile b/arch/arm64/kernel/pi/Makefile index 2bbe866417d4..d084c1dcf416 100644 --- a/arch/arm64/kernel/pi/Makefile +++ b/arch/arm64/kernel/pi/Makefile @@ -38,5 +38,6 @@ $(obj)/lib-%.pi.o: OBJCOPYFLAGS += --prefix-alloc-sections=.init $(obj)/lib-%.o: $(srctree)/lib/%.c FORCE $(call if_changed_rule,cc_o_c) -obj-y := kaslr_early.pi.o lib-fdt.pi.o lib-fdt_ro.pi.o -extra-y := $(patsubst %.pi.o,%.o,$(obj-y)) +obj-y := relocate.pi.o +obj-$(CONFIG_RANDOMIZE_BASE) += kaslr_early.pi.o lib-fdt.pi.o lib-fdt_ro.pi.o +extra-y := $(patsubst %.pi.o,%.o,$(obj-y)) diff --git a/arch/arm64/kernel/pi/relocate.c b/arch/arm64/kernel/pi/relocate.c new file mode 100644 index 000000000000..1853408ea76b --- /dev/null +++ b/arch/arm64/kernel/pi/relocate.c @@ -0,0 +1,62 @@ +// SPDX-License-Identifier: GPL-2.0-only +// Copyright 2023 Google LLC +// Authors: Ard Biesheuvel +// Peter Collingbourne + +#include +#include +#include + +extern const Elf64_Rela rela_start[], rela_end[]; +extern const u64 relr_start[], relr_end[]; + +void __init relocate_kernel(u64 offset) +{ + u64 *place = NULL; + + for (const Elf64_Rela *rela = rela_start; rela < rela_end; rela++) { + if (ELF64_R_TYPE(rela->r_info) != R_AARCH64_RELATIVE) + continue; + *(u64 *)(rela->r_offset + offset) = rela->r_addend + offset; + } + + if (!IS_ENABLED(CONFIG_RELR) || !offset) + return; + + /* + * Apply RELR relocations. + * + * RELR is a compressed format for storing relative relocations. The + * encoded sequence of entries looks like: + * [ AAAAAAAA BBBBBBB1 BBBBBBB1 ... AAAAAAAA BBBBBB1 ... ] + * + * i.e. start with an address, followed by any number of bitmaps. The + * address entry encodes 1 relocation. The subsequent bitmap entries + * encode up to 63 relocations each, at subsequent offsets following + * the last address entry. + * + * The bitmap entries must have 1 in the least significant bit. The + * assumption here is that an address cannot have 1 in lsb. Odd + * addresses are not supported. Any odd addresses are stored in the + * RELA section, which is handled above. + * + * With the exception of the least significant bit, each bit in the + * bitmap corresponds with a machine word that follows the base address + * word, and the bit value indicates whether or not a relocation needs + * to be applied to it. The second least significant bit represents the + * machine word immediately following the initial address, and each bit + * that follows represents the next word, in linear order. As such, a + * single bitmap can encode up to 63 relocations in a 64-bit object. + */ + for (const u64 *relr = relr_start; relr < relr_end; relr++) { + if ((*relr & 1) == 0) { + place = (u64 *)(*relr + offset); + *place++ += offset; + } else { + for (u64 *p = place, r = *relr >> 1; r; p++, r >>= 1) + if (r & 1) + *p += offset; + place += 63; + } + } +} diff --git a/arch/arm64/kernel/vmlinux.lds.S b/arch/arm64/kernel/vmlinux.lds.S index 3cd7e76cc562..8dd5dda66f7c 100644 --- a/arch/arm64/kernel/vmlinux.lds.S +++ b/arch/arm64/kernel/vmlinux.lds.S @@ -270,15 +270,15 @@ SECTIONS HYPERVISOR_RELOC_SECTION .rela.dyn : ALIGN(8) { - __rela_start = .; + __pi_rela_start = .; *(.rela .rela*) - __rela_end = .; + __pi_rela_end = .; } .relr.dyn : ALIGN(8) { - __relr_start = .; + __pi_relr_start = .; *(.relr.dyn) - __relr_end = .; + __pi_relr_end = .; } . = ALIGN(SEGMENT_ALIGN); @@ -317,6 +317,10 @@ SECTIONS init_pg_dir = .; . += INIT_DIR_SIZE; init_pg_end = .; +#ifdef CONFIG_RELOCATABLE + . += SZ_4K; /* stack for the early relocation code */ + early_init_stack = .; +#endif . = ALIGN(SEGMENT_ALIGN); __pecoff_data_size = ABSOLUTE(. - __initdata_begin); From patchwork Tue Jan 23 14:53:10 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13527718 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 3FA3CC47258 for ; Tue, 23 Jan 2024 16:01:00 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=HRqhzlevYNdvjTvnX8jZxSntVkCSCri3fWhqSr262OU=; b=puiqPFs6GjZLkJDtJTOSTqB0MW l14FmXA0f8bxt2DzPr9nyCBupAuOiYAsPE3x/FU7UVXLoxJ0nk0goc/w/jPgbAk6FZ0VVg8NG0/WN lz91WvmUnYIbA5CCvDjtVD4POuGmDBoGd9Zmc4Rx8jpIa/11VwmO3CWqWzZe6ImO+AC65HJ2Z3KP5 aWiGlqIKFaRHFMaCfarw/nBCBzAlS8UPQOoD6vyHKFx7YD5zbF3BwdomRv2b1e1g/hhPNbMwFDTY5 0aZodIPA0XvKHN/BxxMOrbsLjilCnyGnbTySlcMhnR2aUIcwymvcdSx8gAgguw0I4hiHNcptR2Zs1 S9LHdEeg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rSJCQ-00HAqT-0N; Tue, 23 Jan 2024 16:00:30 +0000 Received: from mail-yb1-xb4a.google.com ([2607:f8b0:4864:20::b4a]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rSIAq-00Gtj7-0X for linux-arm-kernel@lists.infradead.org; Tue, 23 Jan 2024 14:54:50 +0000 Received: by mail-yb1-xb4a.google.com with SMTP id 3f1490d57ef6-dc223d528cfso6626754276.0 for ; Tue, 23 Jan 2024 06:54:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1706021687; x=1706626487; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=12uRlojCIuYdKr6dT1QfLTLFhbPXNVqaaRvgxwtReCo=; b=YFSNAblgu08pPOEvxRNGjJHXcFBhGc+PF3Sjb757JcpIiob55TXtbdZvodrE7q3WAg 3UFIRzPdHBvc8eQYbWLbRyHvQL2eKtrhXmb9FoEaAq6Znt/cmm4fIJmDgYvpjr2bZ/b6 5Dh6UxSKXsPR19supr9aTKgCb+LbhagXUEtXOuIrAt3z47XuYcFV8VpeD62JKUD2Qs3X seOH4mwgP/tdHSxMnIjSeP+zcBwWiNm58eXOb1h8fYhor+KzZaOiBoaeDiQaEZENnJTR T46Fv85yfieMp/BOn1ojW/jqCHudEzkLgyV44cdZ1d4o50Jp6vnCUpbn5CdVqlEsjw7C sXkQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706021687; x=1706626487; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=12uRlojCIuYdKr6dT1QfLTLFhbPXNVqaaRvgxwtReCo=; b=Ju4dDwzoXf2jE9Wh3qtSYHBmpGr+3NR8OTQZ40gINXAjubehpvwNFIVss/0A5L5PSM x1m9wYer7aDWHW3SMe6WHnRtsA5wHhM6Uu3NS6aAQ9b8YFLLIxuwImCZ/znqgoxjMn+Z f+vZ4NyVK1I670CLxRQ6Oij1+92JzSo2pUn8PpQKqNW8hnhFAASxpJ00p0l/SSD0Cqe3 7Ql8mCn8epRhn6yClQIqwl1cOKLSb6H0ql2FNJP3FoMVNOE3/5TgOTwdrJPTE3zBVS2R ZevgDJtN2RUnfyb4QPdQ+MjmCVNd8XE/4bRQEqoKb++CJvr9xYndZAIh63cmn/VIQQPX Eq5g== X-Gm-Message-State: AOJu0YxE5BKphfkOLhFMSNyC5xtXGJdVC9gSkgZ1sESNN+6hGRJ54a5u 85NPgw/7fTQD6qIuorFtXIvz759oheMSag77eHXO9t6V6qzR1/4YcFrQnQaQiE+1XUNg/L99jYM ooOZsAvQOaJTeA95qJpgEs+mwBtjXuYYKRVkneEJKfODlFEldAiUyHd3dQNoY7WRD9U2UL9wkRE 28+TqeZALMTMh2k/5oyXewiOzEypaNrVgqa7oiQfLk X-Google-Smtp-Source: AGHT+IEqSj+e4CWddv0Zv31thGIYPIQZTrseG42Gi+gq+ZgFTNKjV+1+58JZzZpdt5ns36g+XvN5VaW5 X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:6902:a8c:b0:dc2:65e9:6a35 with SMTP id cd12-20020a0569020a8c00b00dc265e96a35mr2594822ybb.4.1706021687204; Tue, 23 Jan 2024 06:54:47 -0800 (PST) Date: Tue, 23 Jan 2024 15:53:10 +0100 In-Reply-To: <20240123145258.1462979-52-ardb+git@google.com> Mime-Version: 1.0 References: <20240123145258.1462979-52-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=6698; i=ardb@kernel.org; h=from:subject; bh=5dZBKPpsAfolZRpZ7v54Gy8vlDdggwgRtcVwD8tvCcA=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIXX9pWveyvfuvb9e6Gs17/W8B8+6pS93/zbx23PM/L2qk nyVdo55RykLgxgHg6yYIovA7L/vdp6eKFXrPEsWZg4rE8gQBi5OAZjIisOMDGd9k/LXBtwI46jb 875aPrL6ZPD/266PJlxaKGmVnhXpp8XIcLktI/CinUhjy/tpwhysB8rz2KO2+gQLParu7/tUO3s JIwA= X-Mailer: git-send-email 2.43.0.429.g432eaa2c6b-goog Message-ID: <20240123145258.1462979-63-ardb+git@google.com> Subject: [PATCH v7 11/50] arm64: idreg-override: Move to early mini C runtime From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240123_065448_229312_04AD4371 X-CRM114-Status: GOOD ( 17.33 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel We will want to parse the ID register overrides even earlier, so that we can take them into account before creating the kernel mapping. So migrate the code and make it work in the context of the early C runtime. We will move the invocation to an earlier stage in a subsequent patch. Signed-off-by: Ard Biesheuvel --- arch/arm64/kernel/Makefile | 4 +-- arch/arm64/kernel/head.S | 5 ++- arch/arm64/kernel/image-vars.h | 9 +++++ arch/arm64/kernel/pi/Makefile | 5 +-- arch/arm64/kernel/{ => pi}/idreg-override.c | 35 +++++++++----------- 5 files changed, 30 insertions(+), 28 deletions(-) diff --git a/arch/arm64/kernel/Makefile b/arch/arm64/kernel/Makefile index 84db82cf6ba8..649efbbf0225 100644 --- a/arch/arm64/kernel/Makefile +++ b/arch/arm64/kernel/Makefile @@ -33,8 +33,7 @@ obj-y := debug-monitors.o entry.o irq.o fpsimd.o \ return_address.o cpuinfo.o cpu_errata.o \ cpufeature.o alternative.o cacheinfo.o \ smp.o smp_spin_table.o topology.o smccc-call.o \ - syscall.o proton-pack.o idreg-override.o idle.o \ - patching.o + syscall.o proton-pack.o idle.o patching.o pi/ obj-$(CONFIG_COMPAT) += sys32.o signal32.o \ sys_compat.o @@ -57,7 +56,6 @@ obj-$(CONFIG_ACPI) += acpi.o obj-$(CONFIG_ACPI_NUMA) += acpi_numa.o obj-$(CONFIG_ARM64_ACPI_PARKING_PROTOCOL) += acpi_parking_protocol.o obj-$(CONFIG_PARAVIRT) += paravirt.o -obj-$(CONFIG_RELOCATABLE) += pi/ obj-$(CONFIG_RANDOMIZE_BASE) += kaslr.o obj-$(CONFIG_HIBERNATION) += hibernate.o hibernate-asm.o obj-$(CONFIG_ELF_CORE) += elfcore.o diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S index a8fa64fc30d7..ca5e5fbefcd3 100644 --- a/arch/arm64/kernel/head.S +++ b/arch/arm64/kernel/head.S @@ -510,10 +510,9 @@ SYM_FUNC_START_LOCAL(__primary_switched) #if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS) bl kasan_early_init #endif - mov x0, x21 // pass FDT address in x0 - bl early_fdt_map // Try mapping the FDT early mov x0, x20 // pass the full boot status - bl init_feature_override // Parse cpu feature overrides + mov x1, x22 // pass the low FDT mapping + bl __pi_init_feature_override // Parse cpu feature overrides #ifdef CONFIG_UNWIND_PATCH_PAC_INTO_SCS bl scs_patch_vmlinux #endif diff --git a/arch/arm64/kernel/image-vars.h b/arch/arm64/kernel/image-vars.h index e931ce078a00..eacc3d167733 100644 --- a/arch/arm64/kernel/image-vars.h +++ b/arch/arm64/kernel/image-vars.h @@ -37,6 +37,15 @@ PROVIDE(__pi___memmove = __pi_memmove); PROVIDE(__pi___memset = __pi_memset); PROVIDE(__pi_vabits_actual = vabits_actual); +PROVIDE(__pi_id_aa64isar1_override = id_aa64isar1_override); +PROVIDE(__pi_id_aa64isar2_override = id_aa64isar2_override); +PROVIDE(__pi_id_aa64mmfr1_override = id_aa64mmfr1_override); +PROVIDE(__pi_id_aa64pfr0_override = id_aa64pfr0_override); +PROVIDE(__pi_id_aa64pfr1_override = id_aa64pfr1_override); +PROVIDE(__pi_id_aa64smfr0_override = id_aa64smfr0_override); +PROVIDE(__pi_id_aa64zfr0_override = id_aa64zfr0_override); +PROVIDE(__pi_arm64_sw_feature_override = arm64_sw_feature_override); +PROVIDE(__pi__ctype = _ctype); #ifdef CONFIG_KVM diff --git a/arch/arm64/kernel/pi/Makefile b/arch/arm64/kernel/pi/Makefile index d084c1dcf416..7f6dfce893c3 100644 --- a/arch/arm64/kernel/pi/Makefile +++ b/arch/arm64/kernel/pi/Makefile @@ -38,6 +38,7 @@ $(obj)/lib-%.pi.o: OBJCOPYFLAGS += --prefix-alloc-sections=.init $(obj)/lib-%.o: $(srctree)/lib/%.c FORCE $(call if_changed_rule,cc_o_c) -obj-y := relocate.pi.o -obj-$(CONFIG_RANDOMIZE_BASE) += kaslr_early.pi.o lib-fdt.pi.o lib-fdt_ro.pi.o +obj-y := idreg-override.pi.o lib-fdt.pi.o lib-fdt_ro.pi.o +obj-$(CONFIG_RELOCATABLE) += relocate.pi.o +obj-$(CONFIG_RANDOMIZE_BASE) += kaslr_early.pi.o extra-y := $(patsubst %.pi.o,%.o,$(obj-y)) diff --git a/arch/arm64/kernel/idreg-override.c b/arch/arm64/kernel/pi/idreg-override.c similarity index 93% rename from arch/arm64/kernel/idreg-override.c rename to arch/arm64/kernel/pi/idreg-override.c index e30fd9e32ef3..f9e05c10faab 100644 --- a/arch/arm64/kernel/idreg-override.c +++ b/arch/arm64/kernel/pi/idreg-override.c @@ -14,6 +14,8 @@ #include #include +#include "pi.h" + #define FTR_DESC_NAME_LEN 20 #define FTR_DESC_FIELD_LEN 10 #define FTR_ALIAS_NAME_LEN 30 @@ -21,15 +23,6 @@ static u64 __boot_status __initdata; -// temporary __prel64 related definitions -// to be removed when this code is moved under pi/ - -#define __prel64_initconst __initconst - -#define PREL64(type, name) union { type *name; } - -#define prel64_pointer(__d) (__d) - typedef bool filter_t(u64 val); struct ftr_set_desc { @@ -313,16 +306,11 @@ static __init void __parse_cmdline(const char *cmdline, bool parse_aliases) } while (1); } -static __init const u8 *get_bootargs_cmdline(void) +static __init const u8 *get_bootargs_cmdline(const void *fdt) { const u8 *prop; - void *fdt; int node; - fdt = get_early_fdt_ptr(); - if (!fdt) - return NULL; - node = fdt_path_offset(fdt, "/chosen"); if (node < 0) return NULL; @@ -334,9 +322,9 @@ static __init const u8 *get_bootargs_cmdline(void) return strlen(prop) ? prop : NULL; } -static __init void parse_cmdline(void) +static __init void parse_cmdline(const void *fdt) { - const u8 *prop = get_bootargs_cmdline(); + const u8 *prop = get_bootargs_cmdline(fdt); if (IS_ENABLED(CONFIG_CMDLINE_FORCE) || !prop) __parse_cmdline(CONFIG_CMDLINE, true); @@ -346,9 +334,9 @@ static __init void parse_cmdline(void) } /* Keep checkers quiet */ -void init_feature_override(u64 boot_status); +void init_feature_override(u64 boot_status, const void *fdt); -asmlinkage void __init init_feature_override(u64 boot_status) +asmlinkage void __init init_feature_override(u64 boot_status, const void *fdt) { struct arm64_ftr_override *override; const struct ftr_set_desc *reg; @@ -364,7 +352,7 @@ asmlinkage void __init init_feature_override(u64 boot_status) __boot_status = boot_status; - parse_cmdline(); + parse_cmdline(fdt); for (i = 0; i < ARRAY_SIZE(regs); i++) { reg = prel64_pointer(regs[i].reg); @@ -373,3 +361,10 @@ asmlinkage void __init init_feature_override(u64 boot_status) (unsigned long)(override + 1)); } } + +char * __init skip_spaces(const char *str) +{ + while (isspace(*str)) + ++str; + return (char *)str; +} From patchwork Tue Jan 23 14:53:11 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13527720 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 404A7C47258 for ; Tue, 23 Jan 2024 16:01:08 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=4L4ZRiJBQBCp30TIMpIrqZAHyE5Bm97Lk1GYZeQFLo0=; b=FzrfUqhL3mxJ7OBuSsrebp5Nf+ sEyYUJM0yAxvd2X5b4pp+pgIPOFYEGyqNbsjjtoVTFzqfg6IpGfm9PxXmMNqA43+BSWlPhBkx7ESg k99LasIvY/GuxZdaGm4OK/jId3yxRXIyZy8MFluYlnCX0QyNVE4+a/DLtgHgTis9U0o2qbxTqkelu qx2ZwOOvveAUYfAAeTrG1RajjxzWYCLZSglaoAvteoZa3amorvoJ6HX9Axa9qjtnRO2/TDKKx6hAb 5Py85ClgJSQtb8+2uIyfiPZ5ZoWvkTnOld6ac2n0WHy6RCkeWQHfkR6Zk0scBzgny+TMemW+7Imze VxdsAkCg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rSJCQ-00HAr0-2G; Tue, 23 Jan 2024 16:00:30 +0000 Received: from mail-yb1-xb4a.google.com ([2607:f8b0:4864:20::b4a]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rSIAt-00GtkN-1S for linux-arm-kernel@lists.infradead.org; Tue, 23 Jan 2024 14:54:53 +0000 Received: by mail-yb1-xb4a.google.com with SMTP id 3f1490d57ef6-dc26605c3aaso5338952276.0 for ; Tue, 23 Jan 2024 06:54:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1706021690; x=1706626490; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=ZizGP2MgvhoQ6GFFWc8x2nUKPwFTo5h0Cxlo9Nyx284=; b=wR0EI2FwMOi1jXeXvG5TY5mBOuKjMZXQpCNLuc5N9Y/o6sWRhIgS7n9gS+pgZOP+98 sFh8GIuFWSPSfV4mH2RAiCKntede4aFF2iTWUx6wRxbLekZZUnMTLx1x/fKyXyaj98Q9 WE3hCBWBNuva//AKOOeQHo5x/G2tI7IfPmZvxKTfRWMdyXSoa1gmjxmjcCjRKk0M6a/a MgmIAdXkJmyxjsQV0GIedh1hxLUH+RXvj440fPpdaqBamibqA0dSOKtVz9rrRhkWUuXk 6v0oXvrlffmAKog1lfU1ZHMIletTWSdb9y0tz0Xmf1oLnW82DXLJQbef90YXc7fkwOaa L1sA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706021690; x=1706626490; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=ZizGP2MgvhoQ6GFFWc8x2nUKPwFTo5h0Cxlo9Nyx284=; b=VRC9ro7MyJm1m0OSqLcrHugKnDdd8NgBKFBzyIGlWDMIaV6Dxvh72Ecszbd7XDub9d kDLk0jo+KQceWgYOPRCH8QwfShEQmmuz1B3ngZ3PvlYGms4l7DoT6cPWWegNTiTo3/vD zOPLLUH745Svq30vbOqNQq9mtzPPe/Qnui1mk5pY6Cbxicaj5erZlrExr/Rwpj2b6bvx v8EF4asig7jwEXaTW9j49FoiIVSc6KUx4hk1K0VlWOW+wv3jU7lfwxhmvdP++T68s+r4 pxllhOOOvZT8/uTu2+dEk4gre5FjU1dYWS21wGaRGIKSZ2MiZqMR6GILFawnRPNegEKH A9Vg== X-Gm-Message-State: AOJu0Yy8MqwwJp4jAK66qvR0/Q8fhoBMoQZeCn5JR+oCsXszxRtoARk6 WrMRCY+yy+/N9SGc/hCjy+ss2Z9B7TyV9LaFIARObVcEVefhJMaRDLtlCFP7j6gpnGSslV+uRQp LCtu7H7Z1Tu1DWM6lap/Hw4fRrbsRLmY9DvnGiAiBPVqawBYsrXCYhjanB8oait7v3uZ9uKXrOA a/tlA9WJIDo202nM5AppcAnEcCT9Lo5C+EWjmtpSwN X-Google-Smtp-Source: AGHT+IE0uvuaUsVQxjUQDtmgptLh7UNuQNkD7eLEQPC9XQQylgN6XFDDugX+J/uW9rI+t1jWex4Lq1jT X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:6902:1a47:b0:dc2:65f1:57e0 with SMTP id cy7-20020a0569021a4700b00dc265f157e0mr417534ybb.8.1706021689392; Tue, 23 Jan 2024 06:54:49 -0800 (PST) Date: Tue, 23 Jan 2024 15:53:11 +0100 In-Reply-To: <20240123145258.1462979-52-ardb+git@google.com> Mime-Version: 1.0 References: <20240123145258.1462979-52-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=1354; i=ardb@kernel.org; h=from:subject; bh=Uq8t5laFy5sAfHDuHsWo+xKThP4usVNXxMFxvR3LmsA=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIXX9pRsXe79kexh+/COZq91jtU6uYVe7bqLujPmuOhvPe 7AUcMp3lLIwiHEwyIopsgjM/vtu5+mJUrXOs2Rh5rAygQxh4OIUgInsrGD4yVjBf6n+oTm3wOZ/ v3aY2pfsSGhRcWCwVSqdo+a21vPqPYa/MtIehg4P1wreD1UIP/zh0Z3W0B61e9v9e1iX/KmN/ba TCwA= X-Mailer: git-send-email 2.43.0.429.g432eaa2c6b-goog Message-ID: <20240123145258.1462979-64-ardb+git@google.com> Subject: [PATCH v7 12/50] arm64: kernel: Remove early fdt remap code From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240123_065451_559162_68FEB445 X-CRM114-Status: GOOD ( 10.32 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel The early FDT remap code is no longer used so let's drop it. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/setup.h | 3 --- arch/arm64/kernel/setup.c | 15 --------------- 2 files changed, 18 deletions(-) diff --git a/arch/arm64/include/asm/setup.h b/arch/arm64/include/asm/setup.h index 2e4d7da74fb8..ba269a7a3201 100644 --- a/arch/arm64/include/asm/setup.h +++ b/arch/arm64/include/asm/setup.h @@ -7,9 +7,6 @@ #include -void *get_early_fdt_ptr(void); -void early_fdt_map(u64 dt_phys); - /* * These two variables are used in the head.S file. */ diff --git a/arch/arm64/kernel/setup.c b/arch/arm64/kernel/setup.c index 42c690bb2d60..97d2143669cf 100644 --- a/arch/arm64/kernel/setup.c +++ b/arch/arm64/kernel/setup.c @@ -166,21 +166,6 @@ static void __init smp_build_mpidr_hash(void) pr_warn("Large number of MPIDR hash buckets detected\n"); } -static void *early_fdt_ptr __initdata; - -void __init *get_early_fdt_ptr(void) -{ - return early_fdt_ptr; -} - -asmlinkage void __init early_fdt_map(u64 dt_phys) -{ - int fdt_size; - - early_fixmap_init(); - early_fdt_ptr = fixmap_remap_fdt(dt_phys, &fdt_size, PAGE_KERNEL); -} - static void __init setup_machine_fdt(phys_addr_t dt_phys) { int size; From patchwork Tue Jan 23 14:53:12 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13527722 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 00755C47DDF for ; Tue, 23 Jan 2024 16:01:08 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=N2xbvML27RG2Z3raVSmHZUiduCtNnLKb2DJ8JYr8z+4=; b=2JYlvAbK0CgMXslJak9qPMqlV/ yybAYEDJQcdh4RFcVQxEYJVhi6xdYEkzBVTNMiW+UzNkVc1PbiRF//PlEFwLX33OEJ8OYQ4HHoNcf TRTQkEz9EoMwAvvf9ZNEFjGfa0DfS5PVjhqSjU9YxoQrS/fC3R31aRJnHl9M37bQ96mUyzJDzCM7V OSh19QlUHC+ApIxs3efk8TBFNEtKJa6+Kq1RZ95qpO/kf9UzRUpHRvn1BkZp4uvkxzxGiIYSKNavl ffMYlUuFQ0x/BcwbiZycEXEWAhp4hUIKR0DuC5N0DphnhK7LPpGqnh+BTvRtkd7m9IxnKGbZlnbsR 4kDsDItg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rSJCR-00HArF-0i; Tue, 23 Jan 2024 16:00:31 +0000 Received: from mail-yb1-xb49.google.com ([2607:f8b0:4864:20::b49]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rSIAv-00Gtm5-13 for linux-arm-kernel@lists.infradead.org; Tue, 23 Jan 2024 14:55:02 +0000 Received: by mail-yb1-xb49.google.com with SMTP id 3f1490d57ef6-dc26605c3aaso5339006276.0 for ; Tue, 23 Jan 2024 06:54:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1706021692; x=1706626492; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=ZXAZZP0VPvaQerv7KCHEBp12Yb5fRWF0OZVW+VVhVrM=; b=tHYQ/bQte5p4JEfs0S/HLqQkoTJQdJX2Rn/8qQY4KpGL9B1dHmMv+EN0WlooSw9efY 3PPbqQ4cHoNd5ly/jnzsKiieHYm0kyT6Jm0//iRaFyN5BdJaWVQVkK8/nw4UloSYnDmr mz+pgW1PfQlU7P2R4wh+HJseAk+teOIwHMajc9NaEYlbRxnGWV28nGDwpqq6cSJW/Ki9 IjSf3PYnKq5BpXH3k6uDI6e1IR30saB6TtQrUlkvEqdQw7locT+D6YqJnXx69MYunZ8Q PcI0HHwFKsaWRJM/Ka0IAxJwe2wKq3ulo8uT4yZmzS/nw0l5FwTY8+NJmIvCrJuT/lV/ ihvg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706021692; x=1706626492; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=ZXAZZP0VPvaQerv7KCHEBp12Yb5fRWF0OZVW+VVhVrM=; b=RT2Ancm9owJZsVHVJN3vle/HLu5jNSIAExOWgIlSKGFft29OUYlmpGGirjO/S5DPW/ sHavL2rbOmnsPZhM2v4M3lZl2eaQBbBLU+IFanEKuR2UnhmKpkeR66EknxKvSEeiJYC3 ZHu2UhO9YuOiA3ENO1L5GRat8rBvuI8nnaMBSWF6DM1vH2pKWBFbuO+hqiKbdDOd5CQg 6vBmIEQtIUTyWdTH7hlGiuYyB7F5Px7yP+g8n06W4WZpYpSJdi6FfmCMT2O5/1G8fNd5 /Zz9Pn3wrcBp0xdm1an3fDiUrFZwY2ciq0X3sMbrMx1UxB+m/+gF4K7aWTcsAcTWL0xf UeQw== X-Gm-Message-State: AOJu0YxRuL9eA2c+NVlw5HWXYZUPXPm5iKC7yFEtdEd58orejZeP17F/ GwCxKHkaUv+0/j1kVi/3i+doSJ/PW+5B8UqXH2xNB00Jmy7lQPOVek3wm7b7aNkLs7Y4jPWGnF6 Vv50LwbQNehIyWTdBcjOPTI0mz5UB7vTt4YFc58qZZmohcKTsULqOILYxSHccfoHkuDIGVDgPYu 2MOpBCMHV7QFiroCCEtBXi3sr3MmxlqjAte6j0eggn X-Google-Smtp-Source: AGHT+IF4w5lRpARpQGQ27GqnkxdobUxYqxFQSiQrx++5yB5uplTXFLUaNWRtE0oTohZTnhEn+UIYl6Rn X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:6902:1a47:b0:dc2:65f1:57e0 with SMTP id cy7-20020a0569021a4700b00dc265f157e0mr417536ybb.8.1706021691770; Tue, 23 Jan 2024 06:54:51 -0800 (PST) Date: Tue, 23 Jan 2024 15:53:12 +0100 In-Reply-To: <20240123145258.1462979-52-ardb+git@google.com> Mime-Version: 1.0 References: <20240123145258.1462979-52-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=3389; i=ardb@kernel.org; h=from:subject; bh=/tshOW2txPHrwVcGFKbXruO3ZYyYRxtzskBM1EnORU0=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIXX9pZtyx4rVRLZUGeQGXLSU+1DlanP+2b4ey2MPNl//+ n6e0ifWjlIWBjEOBlkxRRaB2X/f7Tw9UarWeZYszBxWJpAhDFycAjCRZbUM/93Tl8wMf+Pq/vRa W9S/xbJs8r1NTd7vrPL0mLjyLe8y7mJkOJIo9Z2pI9c44dzkW5NfFdso3Djw5FqrhOOTmMw5s/T /swMA X-Mailer: git-send-email 2.43.0.429.g432eaa2c6b-goog Message-ID: <20240123145258.1462979-65-ardb+git@google.com> Subject: [PATCH v7 13/50] arm64: head: Clear BSS and the kernel page tables in one go From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240123_065453_389607_706936E4 X-CRM114-Status: GOOD ( 15.57 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel We will move the CPU feature overrides into BSS in a subsequent patch, and this requires that BSS is zeroed before the feature override detection code runs. So let's map BSS read-write in the ID map, and zero it via this mapping. Since the kernel page tables are right next to it, and also zeroed via the ID map, let's drop the separate clear_page_tables() function, and just zero everything in one go. Signed-off-by: Ard Biesheuvel --- arch/arm64/kernel/head.S | 33 +++++++------------- arch/arm64/kernel/vmlinux.lds.S | 3 ++ 2 files changed, 14 insertions(+), 22 deletions(-) diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S index ca5e5fbefcd3..2af518161f3a 100644 --- a/arch/arm64/kernel/head.S +++ b/arch/arm64/kernel/head.S @@ -177,17 +177,6 @@ SYM_CODE_START_LOCAL(preserve_boot_args) ret SYM_CODE_END(preserve_boot_args) -SYM_FUNC_START_LOCAL(clear_page_tables) - /* - * Clear the init page tables. - */ - adrp x0, init_pg_dir - adrp x1, init_pg_end - sub x2, x1, x0 - mov x1, xzr - b __pi_memset // tail call -SYM_FUNC_END(clear_page_tables) - /* * Macro to populate page table entries, these entries can be pointers to the next level * or last level entries pointing to physical memory. @@ -386,9 +375,9 @@ SYM_FUNC_START_LOCAL(create_idmap) map_memory x0, x1, x3, x6, x7, x3, IDMAP_PGD_ORDER, x10, x11, x12, x13, x14, EXTRA_SHIFT - /* Remap the kernel page tables r/w in the ID map */ + /* Remap BSS and the kernel page tables r/w in the ID map */ adrp x1, _text - adrp x2, init_pg_dir + adrp x2, __bss_start adrp x3, _end bic x4, x2, #SWAPPER_BLOCK_SIZE - 1 mov_q x5, SWAPPER_RW_MMUFLAGS @@ -489,14 +478,6 @@ SYM_FUNC_START_LOCAL(__primary_switched) mov x0, x20 bl set_cpu_boot_mode_flag - // Clear BSS - adr_l x0, __bss_start - mov x1, xzr - adr_l x2, __bss_stop - sub x2, x2, x0 - bl __pi_memset - dsb ishst // Make zero page visible to PTW - #if VA_BITS > 48 adr_l x8, vabits_actual // Set this early so KASAN early init str x25, [x8] // ... observes the correct value @@ -782,6 +763,15 @@ SYM_FUNC_START_LOCAL(__primary_switch) adrp x1, reserved_pg_dir adrp x2, init_idmap_pg_dir bl __enable_mmu + + // Clear BSS + adrp x0, __bss_start + mov x1, xzr + adrp x2, init_pg_end + sub x2, x2, x0 + bl __pi_memset + dsb ishst // Make zero page visible to PTW + #ifdef CONFIG_RELOCATABLE adrp x23, KERNEL_START and x23, x23, MIN_KIMG_ALIGN - 1 @@ -796,7 +786,6 @@ SYM_FUNC_START_LOCAL(__primary_switch) orr x23, x23, x0 // record kernel offset #endif #endif - bl clear_page_tables bl create_kernel_mapping adrp x1, init_pg_dir diff --git a/arch/arm64/kernel/vmlinux.lds.S b/arch/arm64/kernel/vmlinux.lds.S index 8dd5dda66f7c..8a3c6aacc355 100644 --- a/arch/arm64/kernel/vmlinux.lds.S +++ b/arch/arm64/kernel/vmlinux.lds.S @@ -311,12 +311,15 @@ SECTIONS __pecoff_data_rawsize = ABSOLUTE(. - __initdata_begin); _edata = .; + /* start of zero-init region */ BSS_SECTION(SBSS_ALIGN, 0, 0) . = ALIGN(PAGE_SIZE); init_pg_dir = .; . += INIT_DIR_SIZE; init_pg_end = .; + /* end of zero-init region */ + #ifdef CONFIG_RELOCATABLE . += SZ_4K; /* stack for the early relocation code */ early_init_stack = .; From patchwork Tue Jan 23 14:53:13 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13527719 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 59BACC47DDB for ; Tue, 23 Jan 2024 16:01:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=Vi6Z/LqZGKoLQyPnxflJ7ny9SFCJU4xcMnV0ngmwdxc=; b=KyNtSAM9s4IWkMGIneTYkqAbN8 wzZ2b4ZQnowKAKy+jZRwCrCyuw+vGBYuQel+Epmslw6m3qPYafFsg2yWehrN1Frp9JWCDK/bQ1E7D oMNK1B0XMZJg+wg/qhRbIPdIGHpcAK+CmOnz8CyjoaBJoP5IQdgYUPk9dBNo5YXMvEdphuwZGG+Ab 3xijfmDJSbhlMdrDg8gdocAVZ9P9FZQkFjfDs+P/5lCuu2DaXLClpcWYv+oBUxHlVaKZbizzSv2pd lf3ZhcoEEdfyJyoIjlktV6KWqnOg6gKmuiprsLUNXhdlgDIeN4AcVrqUaea+v0uYDPSlGp4aSoppG Eh29D58g==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rSJCR-00HArV-2T; Tue, 23 Jan 2024 16:00:31 +0000 Received: from mail-yb1-xb4a.google.com ([2607:f8b0:4864:20::b4a]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rSIAx-00GtoI-2C for linux-arm-kernel@lists.infradead.org; Tue, 23 Jan 2024 14:55:03 +0000 Received: by mail-yb1-xb4a.google.com with SMTP id 3f1490d57ef6-dc37b0ac33cso689948276.2 for ; Tue, 23 Jan 2024 06:54:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1706021694; x=1706626494; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=FEN215+vp7ZnhP1/JiILPedrshyx5I3UlpHwXzSi3Fg=; b=aVp79voBy4NAqEM/rjrPyDOVT0HSH9sr3ZXI9HbU4g7JeKv8C+etaUwUrenpf8ipiG Em5pFHjhmxUMv65HVOFPtCsnuztC3N5cPkT2OuY04QQpA8LrSkkUDuRMpf8qE/uuQrlb OGVtS8ZVcopw93etybX4z33Vs/KsMC38hii3szPp6GYj+yIgY3BN2v2+OPmy51BmG/3N 3410gspVLEM+NobGrXi1YBhyl9c3Nb+gPtxq3367MiEkjtoyzz+5LxQHW6jCiXgXDVxU xo/QILBEJeMzLhREwL7aFs0Tp6/U0eqA29GO1oizK35bG2+355lQbbKzNOnMKeVkdqFl a76A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706021694; x=1706626494; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=FEN215+vp7ZnhP1/JiILPedrshyx5I3UlpHwXzSi3Fg=; b=se5AZ/RspCZZnD/bwXpK/BA4bwj+r+tyTWnS6q4WFk+rUvxTXll8zltcW8ZXf7p+MH GEtGPea/Lxfo3A8FJ06LERVvE5ZRA6CHz2YAPHW53SvrzcI1BOC+lGH0Ps9VLsEv1yuO HU1O+TiU6dKbcFQBGieXhGveIpILVhgZRJ9LWZNNjb9EiCwovE6ej0+d46sXbCGndQeX jKHsVFgKaPyyA5X4f4v+yL6hjWUrvdjlDALg56x6cO31ypKnJkhq/YzjEW9T5daOdIKd lqLBCwHztrMgfnD9SYde9cYPt6dNksSM/Fr57vAQ/PevF3IO6TJacfvoUQcCF9WRu0pF P6kQ== X-Gm-Message-State: AOJu0YxTzBhGkl84wN/u8FLAS2vot3Tr55WVXaX+yQW0sFoM+IVMAjCe VoV90cZELS1hLQD9LWVyCDs+ogO5Az1vJql7qORtjLgeJeM4BSo8wea/5TMJ7UYWRBTEH+TKR/R fTs4Yy1DJMk5TKKkg87ut6L5DkpKGsbBZRwGadwbZSm1adbln8TZgJ33Y/ciumUnLEIS8M+z9Yc ixPaskVCTNZB1316ItjN7+SDNi6+psL3qp7R3TzKfx X-Google-Smtp-Source: AGHT+IFNtVhCMIW9rYikMxFwQj2q6WDjYB7+XsZ0kQ+CUTbns4xEiPhEBU39VNL+yorqQ6tEBpC/ictO X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a25:d385:0:b0:db5:3ce2:7977 with SMTP id e127-20020a25d385000000b00db53ce27977mr2680947ybf.7.1706021694321; Tue, 23 Jan 2024 06:54:54 -0800 (PST) Date: Tue, 23 Jan 2024 15:53:13 +0100 In-Reply-To: <20240123145258.1462979-52-ardb+git@google.com> Mime-Version: 1.0 References: <20240123145258.1462979-52-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=1677; i=ardb@kernel.org; h=from:subject; bh=YJlMcHy0Yl0/2FBKrYcsDu4CFFijocVGx6QsSYsz73g=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIXX9pVsKi/q1mdK3TZzUnlJbZSA8uSefv/PJLUYm8YtzL 0vcbv/XUcrCIMbBICumyCIw+++7nacnStU6z5KFmcPKBDKEgYtTACYytZXhr4xtypF4N73Pauck /BkkhCyXhK+KnhVRJVEuuPNSV4xRGSPDnClHA/vEttU9eqO62f7Vml8v5+5k1J/8xHN/kydPxte L3AA= X-Mailer: git-send-email 2.43.0.429.g432eaa2c6b-goog Message-ID: <20240123145258.1462979-66-ardb+git@google.com> Subject: [PATCH v7 14/50] arm64: Move feature overrides into the BSS section From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240123_065455_776174_5F0DE025 X-CRM114-Status: GOOD ( 11.38 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel In order to allow the CPU feature override detection code to run even earlier, move the feature override global variables into BSS, which is the only part of the static kernel image that is mapped read-write in the initial ID map. Signed-off-by: Ard Biesheuvel --- arch/arm64/kernel/cpufeature.c | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c index 8d1a634a403e..a99ad79adee2 100644 --- a/arch/arm64/kernel/cpufeature.c +++ b/arch/arm64/kernel/cpufeature.c @@ -655,13 +655,13 @@ static const struct arm64_ftr_bits ftr_raz[] = { #define ARM64_FTR_REG(id, table) \ __ARM64_FTR_REG_OVERRIDE(#id, id, table, &no_override) -struct arm64_ftr_override __ro_after_init id_aa64mmfr1_override; -struct arm64_ftr_override __ro_after_init id_aa64pfr0_override; -struct arm64_ftr_override __ro_after_init id_aa64pfr1_override; -struct arm64_ftr_override __ro_after_init id_aa64zfr0_override; -struct arm64_ftr_override __ro_after_init id_aa64smfr0_override; -struct arm64_ftr_override __ro_after_init id_aa64isar1_override; -struct arm64_ftr_override __ro_after_init id_aa64isar2_override; +struct arm64_ftr_override id_aa64mmfr1_override; +struct arm64_ftr_override id_aa64pfr0_override; +struct arm64_ftr_override id_aa64pfr1_override; +struct arm64_ftr_override id_aa64zfr0_override; +struct arm64_ftr_override id_aa64smfr0_override; +struct arm64_ftr_override id_aa64isar1_override; +struct arm64_ftr_override id_aa64isar2_override; struct arm64_ftr_override arm64_sw_feature_override; From patchwork Tue Jan 23 14:53:14 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13527726 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 86E8BC48285 for ; Tue, 23 Jan 2024 16:01:11 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=2L32XDF0TXcaXAzChe0UqYGqfMDc6frzH5p09MFa7FI=; b=hJdDR9iX4LAChQYzRj+zauvVR2 C9W3yUqFqdQh804RqICffwnjUtMgHZ/IKlMQOuk3IS7tjMqdgaWixBLa1Tr4PJfipKIsAme5WLxmh VfrHDy+lB7aehO0KxV1uPhXq+udTsymwkHfwb6tTcOh4atFBwYrU8HAmYkIg0l0sDIoDtBLrEkd2I xVXzRbfehq1NZTAYF7j3QoDXJCFSTG7pQUrtcerS6QaQ3LZ9pvXl9sh9bMo3XSwdzwcPKEPv2gDj1 63Sk7UTjDSiG2HO9VilcDsUpSBaIBAiIGsXs+TiXLFhOyMiSEnxKMaIHYDnGJ8CrmvsGin+zPb6Ta 2/42qXvQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rSJCS-00HArq-14; Tue, 23 Jan 2024 16:00:32 +0000 Received: from mail-yw1-x114a.google.com ([2607:f8b0:4864:20::114a]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rSIB3-00Gtq4-0g for linux-arm-kernel@lists.infradead.org; Tue, 23 Jan 2024 14:55:10 +0000 Received: by mail-yw1-x114a.google.com with SMTP id 00721157ae682-5ffa2bb4810so34490947b3.0 for ; Tue, 23 Jan 2024 06:54:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1706021697; x=1706626497; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=ODYIdDM+nqe1w4y1lwTl7vo5ShPL7ZusEVAukESBcmA=; b=kuPHehNLMElNELpyFaE0czZLAM7eqAgM/f8HcOkTLFctEzWz01mT1Ocvee+uK/pJqS KmsPPLCecbAOVDlBjD7q2IZnQBO7oNF8lFHTLpE70m/zzchIMOe9u7u4rd1PtSu6VWBq NOSygg0DuMBIYqGPB+Kj5VAYThOeNCDauWf8w8PmpeHnhZK4nWAjqCwDoN1d4/fJcu8g LoXo41AeBU39GZFVcdFr8CcEMvV+ocNMja2ggn4gNuDwhrN4XmhEprW0a1CQcMC1Ochv VTGDtY2ptr2wxXiMDAJWv79O7IrsOr4jKvfIwAYhl9l9Wk75RTjkVzv3W+Obw6SmJrW2 lVXw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706021697; x=1706626497; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=ODYIdDM+nqe1w4y1lwTl7vo5ShPL7ZusEVAukESBcmA=; b=So2wQpKAD8wfPfkH7f8E/yPqPds4PiIWC+Rq+fqC1woFyQfiNcaSymERYtn0S4+8Bj lwQlmGTBq9TZrhcl8a1Iem6iWAAwjqiuRuWM8Zm14c5toSDVe67fH/z9XXrFkXofw9zE w5taMtZ5mJmziq6kkaLFbuNS3vhcY+NUxObwjVfM9t9Fc/0pwh9B5lSCPj28zxOllnBm IfVQUPUuR4vAhvjDwyLjD7JCTrwVYJfPJzr9tgXk008M1PBdhd0fiA+vjTJybjKvVk/+ 4xP3vCptLiG303Q+jhw5VOQdVx7TnaSInx3ctt5lDqjIApUcClxvuMmzPkNxNpLkr/BP A5ow== X-Gm-Message-State: AOJu0YzDszWsEBjHYYum8DrNIBHIkft8+G3Y27WpSV5FXbZIrCXOI+av xUWnwWEIxY/lNY8E37gK4kcQmqyhhSd0RsMXxthg4aZ158I8OVMqvc3QJiIsuogtBum6I4MYbR4 4/qw56ic8l3SF1k/uNM7njneqa/gX8cNL58vvtI6l21W3SZtUY9nX1u02TdVPf1V6Aw5vFo9qVo casHC6ARsVJ3R/yonW8M01oIMJDu3RuGPaoOdIpXsm X-Google-Smtp-Source: AGHT+IG4BFKauWXrRjQFf+n3/FITwowBhf9WAUIL4UiM8G8m0V2+3TFjLue7igRROnHRqjyEZFgYF7pF X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a81:4cc6:0:b0:5ff:5389:526c with SMTP id z189-20020a814cc6000000b005ff5389526cmr2111331ywa.2.1706021696914; Tue, 23 Jan 2024 06:54:56 -0800 (PST) Date: Tue, 23 Jan 2024 15:53:14 +0100 In-Reply-To: <20240123145258.1462979-52-ardb+git@google.com> Mime-Version: 1.0 References: <20240123145258.1462979-52-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=2825; i=ardb@kernel.org; h=from:subject; bh=ytqy22hIensXZScckAkMf5yP4E4Kr8RYIQMTUB1HVx4=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIXX9pdt8fqVGeSz8XYLLWyfGa4cs335GKFZeRVDKZ2m3j f2hGouOUhYGMQ4GWTFFFoHZf9/tPD1RqtZ5lizMHFYmkCEMXJwCMJHXuxgZHojypmg+n3Re/nJG zTxp7h6+2nNXb7MVZdt1xpvwLT2hyPA/ht1Q60zpXGMf4+3J5pmT9sq1qF+dseXfO4GggBdPdl7 gBAA= X-Mailer: git-send-email 2.43.0.429.g432eaa2c6b-goog Message-ID: <20240123145258.1462979-67-ardb+git@google.com> Subject: [PATCH v7 15/50] arm64: head: Run feature override detection before mapping the kernel From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240123_065501_258504_6AE1BB99 X-CRM114-Status: GOOD ( 14.16 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel To permit the feature overrides to be taken into account before the KASLR init code runs and the kernel mapping is created, move the detection code to an earlier stage in the boot. In a subsequent patch, this will be taken advantage of by merging the preliminary and permanent mappings of the kernel text and data into a single one that gets created and relocated before start_kernel() is called. Signed-off-by: Ard Biesheuvel --- arch/arm64/kernel/head.S | 17 +++++++++-------- arch/arm64/kernel/vmlinux.lds.S | 4 +--- 2 files changed, 10 insertions(+), 11 deletions(-) diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S index 2af518161f3a..865ecc1f8255 100644 --- a/arch/arm64/kernel/head.S +++ b/arch/arm64/kernel/head.S @@ -375,9 +375,9 @@ SYM_FUNC_START_LOCAL(create_idmap) map_memory x0, x1, x3, x6, x7, x3, IDMAP_PGD_ORDER, x10, x11, x12, x13, x14, EXTRA_SHIFT - /* Remap BSS and the kernel page tables r/w in the ID map */ + /* Remap [.init].data, BSS and the kernel page tables r/w in the ID map */ adrp x1, _text - adrp x2, __bss_start + adrp x2, __initdata_begin adrp x3, _end bic x4, x2, #SWAPPER_BLOCK_SIZE - 1 mov_q x5, SWAPPER_RW_MMUFLAGS @@ -491,9 +491,6 @@ SYM_FUNC_START_LOCAL(__primary_switched) #if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS) bl kasan_early_init #endif - mov x0, x20 // pass the full boot status - mov x1, x22 // pass the low FDT mapping - bl __pi_init_feature_override // Parse cpu feature overrides #ifdef CONFIG_UNWIND_PATCH_PAC_INTO_SCS bl scs_patch_vmlinux #endif @@ -772,12 +769,16 @@ SYM_FUNC_START_LOCAL(__primary_switch) bl __pi_memset dsb ishst // Make zero page visible to PTW -#ifdef CONFIG_RELOCATABLE - adrp x23, KERNEL_START - and x23, x23, MIN_KIMG_ALIGN - 1 adrp x1, early_init_stack mov sp, x1 mov x29, xzr + mov x0, x20 // pass the full boot status + mov x1, x22 // pass the low FDT mapping + bl __pi_init_feature_override // Parse cpu feature overrides + +#ifdef CONFIG_RELOCATABLE + adrp x23, KERNEL_START + and x23, x23, MIN_KIMG_ALIGN - 1 #ifdef CONFIG_RANDOMIZE_BASE mov x0, x22 bl __pi_kaslr_early_init diff --git a/arch/arm64/kernel/vmlinux.lds.S b/arch/arm64/kernel/vmlinux.lds.S index 8a3c6aacc355..3afb4223a5e8 100644 --- a/arch/arm64/kernel/vmlinux.lds.S +++ b/arch/arm64/kernel/vmlinux.lds.S @@ -320,10 +320,8 @@ SECTIONS init_pg_end = .; /* end of zero-init region */ -#ifdef CONFIG_RELOCATABLE - . += SZ_4K; /* stack for the early relocation code */ + . += SZ_4K; /* stack for the early C runtime */ early_init_stack = .; -#endif . = ALIGN(SEGMENT_ALIGN); __pecoff_data_size = ABSOLUTE(. - __initdata_begin); From patchwork Tue Jan 23 14:53:15 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13527727 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id BE914C48286 for ; Tue, 23 Jan 2024 16:01:11 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=XpwzuVPX3oi15vSXpkpZJcVSJ+w1wPCLwONMI4xfoWo=; b=TMvwngG+lda1JstMwtdbjvp620 0MHpkF12naGlJmaejVRoDTc4KsPftDrel2k2IRMVpZNpDd9Kp54b1Q9rOjMkznT5l5OGXaLRgZzus XY2pxIR6S5k8q5HFH59YpTqoyIM3rz9ms4HQTAZa7zT7rwFOm0a1zal1z6AVDDHQ6QZTbRwSA8LPs smWIf+YKLX7bOWA9NBn4ipHFbLm0iXxGEtAEzLICorL9jalFsoq8qd5NAitPnnRe+u9fQ9fIP71Z3 fBTLb3cyZ8wS24Z7oIXjIxSg2JJsl362sIkLj+vX7k2uy85WD8kgR0kT7OpQQJuLdD4s0Tqd0TTlT bNSlQLbA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rSJCS-00HAsI-2t; Tue, 23 Jan 2024 16:00:32 +0000 Received: from mail-yb1-xb49.google.com ([2607:f8b0:4864:20::b49]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rSIB3-00GtrH-0u for linux-arm-kernel@lists.infradead.org; Tue, 23 Jan 2024 14:55:11 +0000 Received: by mail-yb1-xb49.google.com with SMTP id 3f1490d57ef6-d9a541b720aso6420336276.0 for ; Tue, 23 Jan 2024 06:55:00 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1706021699; x=1706626499; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=GaLAWlN5DivvCQOJGrqV10GR0f1TaXebcsmZDzLGDnQ=; b=zSNaDVVIR1pAAA6g3KhWEVrIhKkQrgvGeqmHZeE1VlfiuCvFABcjgTtgoYqGkKy0h3 QHBubON9eHRyDHk6VMpGhf5kq9bpLn+OmTbRpCYrXDjCg+qiUVtYlFzyumg/PKrS+ar5 Ztdp8ohOEKHT/ZBhBR9LT1wqKLedsD90vVtbg81fgL0kmUqA4cnJ5YqeYq0PqKF5bsa5 FWfezjqqEQLd/3OFWfzqVzTzBMEmaNVv1TmEhG97tOU4HJpdrjuH5hJRDpnPaX6XuEdF lHPes+yZqK9L1aGVgK6VqqTuaGp+ix2LQN9TJV3e23DIh9AAUGfqHAWgGar7+k1Iiid+ pIHA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706021699; x=1706626499; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=GaLAWlN5DivvCQOJGrqV10GR0f1TaXebcsmZDzLGDnQ=; b=xLQYi7ibsmrHXwadYNA/A+bBrkE7YetpxG+AECmelF42kRYBYhkRgthvkdbfnVTiTK GJ1nI3sJLqzJcKM3Ni6wdYeEgNFuLymTPxP3Ccb3iaL1W8HDepUWy1tFQX272Onbyldw MlogG98cHYDPnmJHkjp4J8Ej6gy4IMm3ih+1Fr7f/uVnnXB9e+48TTltytaYY+GHtwK0 1E81h1hnoiLQzNIWO2eiuJ6wbvYFaep3RzYybHTvyENeXoA6ce0ut79fj1eDbwylJun+ O6TVH6OHqYILz1iNyAGA3O44AvdYXmzD9Av/GyRVwjnphLQkZDCiIVWg+2YHEbfllS1U KvRg== X-Gm-Message-State: AOJu0YyYHtA2uVKPZq08EAo/G5ALadyKvquMxEk9AZF3C5OpWbd/7cHU /dD+zdDofU94Uc4VCzIfMupM7AoQsQvlP2xp19stFw6y9aeWVA6u4bY54bsx7BoxCy1CNewVdCe ZsE2AlsFmgycNGwKWKazAtvEuzXTDofloq0ONhKXSeORVekankVbTgJcJINrhH4h20/7xaHhZvu hosJpyou9loImeQ130VOrhHwf6IHTTPsDAcjha7Zx5 X-Google-Smtp-Source: AGHT+IFP8Kkvfy3Bdhst6ihfDdcsd6AiSYh9UkeK+tgBwgbZKItVPaaKK7gPuv4vy9Z9s5vbKZWZy8Ih X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a25:ced1:0:b0:dc2:2d2c:962a with SMTP id x200-20020a25ced1000000b00dc22d2c962amr2679318ybe.8.1706021699465; Tue, 23 Jan 2024 06:54:59 -0800 (PST) Date: Tue, 23 Jan 2024 15:53:15 +0100 In-Reply-To: <20240123145258.1462979-52-ardb+git@google.com> Mime-Version: 1.0 References: <20240123145258.1462979-52-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=6725; i=ardb@kernel.org; h=from:subject; bh=sWn2XWjamlMLiq34+D/jKWGP43072AjuH39eDPU3JFY=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIXX9pTvv1cMiNd6amUr/WDBBaOKZw7IBvQ6aVVPcT4fk3 IoW+LWjo5SFQYyDQVZMkUVg9t93O09PlKp1niULM4eVCWQIAxenAEzEUIGR4eVmnqpC3i4OKe2V Xk9/71MJf69lr1pe9qvh68+NUcbvtjD8L1vN87pu4VeNGu+itnW7naRXnbm6ZAarQJGXw6azyz+ s5wMA X-Mailer: git-send-email 2.43.0.429.g432eaa2c6b-goog Message-ID: <20240123145258.1462979-68-ardb+git@google.com> Subject: [PATCH v7 16/50] arm64: head: move dynamic shadow call stack patching into early C runtime From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240123_065501_335196_C3216213 X-CRM114-Status: GOOD ( 19.96 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel Once we update the early kernel mapping code to only map the kernel once with the right permissions, we can no longer perform code patching via this mapping. So move this code to an earlier stage of the boot, right after applying the relocations. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/scs.h | 4 +-- arch/arm64/kernel/Makefile | 8 ------ arch/arm64/kernel/head.S | 8 +++--- arch/arm64/kernel/module.c | 2 +- arch/arm64/kernel/pi/Makefile | 10 +++++--- arch/arm64/kernel/{ => pi}/patch-scs.c | 26 ++++++++++---------- 6 files changed, 27 insertions(+), 31 deletions(-) diff --git a/arch/arm64/include/asm/scs.h b/arch/arm64/include/asm/scs.h index 3fdae5fe3142..eca2ba5a6276 100644 --- a/arch/arm64/include/asm/scs.h +++ b/arch/arm64/include/asm/scs.h @@ -72,8 +72,8 @@ static inline void dynamic_scs_init(void) static inline void dynamic_scs_init(void) {} #endif -int scs_patch(const u8 eh_frame[], int size); -asmlinkage void scs_patch_vmlinux(void); +int __pi_scs_patch(const u8 eh_frame[], int size); +asmlinkage void __pi_scs_patch_vmlinux(void); #endif /* __ASSEMBLY __ */ diff --git a/arch/arm64/kernel/Makefile b/arch/arm64/kernel/Makefile index 649efbbf0225..14b4a179bad3 100644 --- a/arch/arm64/kernel/Makefile +++ b/arch/arm64/kernel/Makefile @@ -71,14 +71,6 @@ obj-$(CONFIG_ARM64_PTR_AUTH) += pointer_auth.o obj-$(CONFIG_ARM64_MTE) += mte.o obj-y += vdso-wrap.o obj-$(CONFIG_COMPAT_VDSO) += vdso32-wrap.o -obj-$(CONFIG_UNWIND_PATCH_PAC_INTO_SCS) += patch-scs.o - -# We need to prevent the SCS patching code from patching itself. Using -# -mbranch-protection=none here to avoid the patchable PAC opcodes from being -# generated triggers an issue with full LTO on Clang, which stops emitting PAC -# instructions altogether. So instead, omit the unwind tables used by the -# patching code, so it will not be able to locate its own PAC instructions. -CFLAGS_patch-scs.o += -fno-asynchronous-unwind-tables -fno-unwind-tables # Force dependency (vdso*-wrap.S includes vdso.so through incbin) $(obj)/vdso-wrap.o: $(obj)/vdso/vdso.so diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S index 865ecc1f8255..b320702032a7 100644 --- a/arch/arm64/kernel/head.S +++ b/arch/arm64/kernel/head.S @@ -490,9 +490,6 @@ SYM_FUNC_START_LOCAL(__primary_switched) #endif #if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS) bl kasan_early_init -#endif -#ifdef CONFIG_UNWIND_PATCH_PAC_INTO_SCS - bl scs_patch_vmlinux #endif mov x0, x20 bl finalise_el2 // Prefer VHE if possible @@ -794,6 +791,11 @@ SYM_FUNC_START_LOCAL(__primary_switch) #ifdef CONFIG_RELOCATABLE mov x0, x23 bl __pi_relocate_kernel +#endif +#ifdef CONFIG_UNWIND_PATCH_PAC_INTO_SCS + ldr x0, =__eh_frame_start + ldr x1, =__eh_frame_end + bl __pi_scs_patch_vmlinux #endif ldr x8, =__primary_switched adrp x0, KERNEL_START // __pa(KERNEL_START) diff --git a/arch/arm64/kernel/module.c b/arch/arm64/kernel/module.c index dd851297596e..47e0be610bb6 100644 --- a/arch/arm64/kernel/module.c +++ b/arch/arm64/kernel/module.c @@ -595,7 +595,7 @@ int module_finalize(const Elf_Ehdr *hdr, if (scs_is_dynamic()) { s = find_section(hdr, sechdrs, ".init.eh_frame"); if (s) - scs_patch((void *)s->sh_addr, s->sh_size); + __pi_scs_patch((void *)s->sh_addr, s->sh_size); } return module_init_ftrace_plt(hdr, sechdrs, me); diff --git a/arch/arm64/kernel/pi/Makefile b/arch/arm64/kernel/pi/Makefile index 7f6dfce893c3..a8b302245f15 100644 --- a/arch/arm64/kernel/pi/Makefile +++ b/arch/arm64/kernel/pi/Makefile @@ -38,7 +38,9 @@ $(obj)/lib-%.pi.o: OBJCOPYFLAGS += --prefix-alloc-sections=.init $(obj)/lib-%.o: $(srctree)/lib/%.c FORCE $(call if_changed_rule,cc_o_c) -obj-y := idreg-override.pi.o lib-fdt.pi.o lib-fdt_ro.pi.o -obj-$(CONFIG_RELOCATABLE) += relocate.pi.o -obj-$(CONFIG_RANDOMIZE_BASE) += kaslr_early.pi.o -extra-y := $(patsubst %.pi.o,%.o,$(obj-y)) +obj-y := idreg-override.pi.o \ + lib-fdt.pi.o lib-fdt_ro.pi.o +obj-$(CONFIG_RELOCATABLE) += relocate.pi.o +obj-$(CONFIG_RANDOMIZE_BASE) += kaslr_early.pi.o +obj-$(CONFIG_UNWIND_PATCH_PAC_INTO_SCS) += patch-scs.pi.o +extra-y := $(patsubst %.pi.o,%.o,$(obj-y)) diff --git a/arch/arm64/kernel/patch-scs.c b/arch/arm64/kernel/pi/patch-scs.c similarity index 91% rename from arch/arm64/kernel/patch-scs.c rename to arch/arm64/kernel/pi/patch-scs.c index a1fe4b4ff591..c65ef40d1e6b 100644 --- a/arch/arm64/kernel/patch-scs.c +++ b/arch/arm64/kernel/pi/patch-scs.c @@ -4,14 +4,11 @@ * Author: Ard Biesheuvel */ -#include #include #include #include -#include #include -#include #include // @@ -81,7 +78,11 @@ static void __always_inline scs_patch_loc(u64 loc) */ return; } - dcache_clean_pou(loc, loc + sizeof(u32)); + if (IS_ENABLED(CONFIG_ARM64_WORKAROUND_CLEAN_CACHE)) + asm("dc civac, %0" :: "r"(loc)); + else + asm(ALTERNATIVE("dc cvau, %0", "nop", ARM64_HAS_CACHE_IDC) + :: "r"(loc)); } /* @@ -128,10 +129,10 @@ struct eh_frame { }; }; -static int noinstr scs_handle_fde_frame(const struct eh_frame *frame, - bool fde_has_augmentation_data, - int code_alignment_factor, - bool dry_run) +static int scs_handle_fde_frame(const struct eh_frame *frame, + bool fde_has_augmentation_data, + int code_alignment_factor, + bool dry_run) { int size = frame->size - offsetof(struct eh_frame, opcodes) + 4; u64 loc = (u64)offset_to_ptr(&frame->initial_loc); @@ -198,14 +199,13 @@ static int noinstr scs_handle_fde_frame(const struct eh_frame *frame, break; default: - pr_err("unhandled opcode: %02x in FDE frame %lx\n", opcode[-1], (uintptr_t)frame); return -ENOEXEC; } } return 0; } -int noinstr scs_patch(const u8 eh_frame[], int size) +int scs_patch(const u8 eh_frame[], int size) { const u8 *p = eh_frame; @@ -251,12 +251,12 @@ int noinstr scs_patch(const u8 eh_frame[], int size) return 0; } -asmlinkage void __init scs_patch_vmlinux(void) +asmlinkage void __init scs_patch_vmlinux(const u8 start[], const u8 end[]) { if (!should_patch_pac_into_scs()) return; - WARN_ON(scs_patch(__eh_frame_start, __eh_frame_end - __eh_frame_start)); - icache_inval_all_pou(); + scs_patch(start, end - start); + asm("ic ialluis"); isb(); } From patchwork Tue Jan 23 14:53:16 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13527724 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 09CA0C48277 for ; Tue, 23 Jan 2024 16:01:11 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=DSQevxT8m3YrmIsF83xN9jTER14uckwI2tVZ3ifl4uk=; b=p11iXTmy77RkMsQJbestrMrpjY lu2Q+RpMTOVgWAqq0RxH49hy7Q9ZwARitUtDo3N2NpWgTL85/AACaezD/ht9y8FohGApCu3trkqs+ w8sXivKYBuVhR/x3WFmGRtlDEoEn2df2k3plc3N+pCPMuiBNGzTN5D9MON3RySlBohVXkSloWu15l L46DAaTwF8AuRIF/fSP4Xu67Dp2oal6ALNHt+6etdSPYQMpSrHTfoGkJ+WAnXXsBn27h9DY5EkOJ6 smBs3lshnGNp/5JQCgVpbvhMUOL8kBjC0GpbI+ljtJGjMXZS005V42+iwIoLv8OkfbR+A6pxA8KJS p94wQI9g==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rSJCT-00HAsc-1R; Tue, 23 Jan 2024 16:00:33 +0000 Received: from mail-wr1-x449.google.com ([2a00:1450:4864:20::449]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rSIB5-00Gtt5-2J for linux-arm-kernel@lists.infradead.org; Tue, 23 Jan 2024 14:55:11 +0000 Received: by mail-wr1-x449.google.com with SMTP id ffacd0b85a97d-33941f7e299so461266f8f.0 for ; Tue, 23 Jan 2024 06:55:02 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1706021702; x=1706626502; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=oIPUbU2klustjscI8Xq9j+nMdsdXjLu5K5MyPJ3m0ro=; b=aEVOTkZs1cJWp+lvvls8wmzif5cuj2UhwPTJrwepgFpPZQt0KqvQpat5dW7pMr3IUH O1IolkfWFMCKGsI7IiYe7m8z0IW0eL+ltRP42pqWOOSWmAG0FZ06i5XkuIMlVYlppduz QV2nxdEl/Xwp1te4tCbIFpeUJ26trY0QJHq+5EH8hMvyNgEHrwAQo0Ppq5QlKo+g/Tq1 GzJfYAJa3UNO+igPD1EILFF7kshNBNtemXH5cJ4Cz0wgQdPntMQfhRSJ3r58Nh/g803v pWftPhgjUkZHv0GIUEU96/zYoVdIt5K9kNBtB407Jg+gsU3eYAlFvyw7AA1DwSMsk1Ft YAmQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706021702; x=1706626502; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=oIPUbU2klustjscI8Xq9j+nMdsdXjLu5K5MyPJ3m0ro=; b=BLhJbGrVf45xIyF5zp73qPvPvUU7pP12iKn1FwpY/989q+4oYt11e+d+lynjfoBIsf IYG1dV4s2KAMJJCGhmk1nhqbgAdpQvir9sqipt796/vXZ5S+fLUN1HN7kGDO2WOMXNcP O7zFhsn3Vrvb5GTAzcEcDr6Xum/vAXeBm7mr01pkY4XcJJ7Q2NJQOOXsFbxYIcfZ2oKU iz6WedKkV79q7YgYriC+lyujrTHaUhjFAnmKtUkSetuWffLp5ZBqRWn+zxL3sQsJUVxv zf32BZUybLc6dGEoSFV/48BZAIzI8dBotbTlgG8HFrYa+M7QQ1m1cv5+D6v2B3Y86yfu QIMA== X-Gm-Message-State: AOJu0YwZa7a8GynlpNbPVdti7TF5u1yyHvVV+hJQH1Ud9yt1UN7yxONI om25oWOyPNVEHJabV3toZbPQceP7UyjD7YPHyCCmjb/JJMHT0x15WmUNn0Du/RQKy1TtW6cTa2H QwoPucI2+lIGelCzGoj3JhLSeKvnsxRLij6nBjKW6XVgYcnsYZZ6tY2EphA7Ez/HsZv7S5OKiAT YDriqyt2rAI98L9TW2/8cqBBq6GE9jr0Yc+addlKWi X-Google-Smtp-Source: AGHT+IE7+K1q3vhrlJEF4nBo7svGNHkHfViDbX5tuKaxhQAoHtyuJy8zRPF9kT5QKcCi8VmFtdN1zk7W X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a5d:6f0a:0:b0:339:3d4f:33d0 with SMTP id ay10-20020a5d6f0a000000b003393d4f33d0mr7246wrb.10.1706021701653; Tue, 23 Jan 2024 06:55:01 -0800 (PST) Date: Tue, 23 Jan 2024 15:53:16 +0100 In-Reply-To: <20240123145258.1462979-52-ardb+git@google.com> Mime-Version: 1.0 References: <20240123145258.1462979-52-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=3269; i=ardb@kernel.org; h=from:subject; bh=jjJW/+rY1cUPTnYpF7LVY6kB/ZlMNlalGuOVDcgSLzY=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIXX9pbsaFauXZSUdfTu3YbZ4UJz39D2JEvYbl1YKGopLb Qn6pXKqo5SFQYyDQVZMkUVg9t93O09PlKp1niULM4eVCWQIAxenAExkjiojw0Sb6ILVzVXzd2ZL vLu2mzPK9JiRmN+pDyKzHER5mUU+vGH4Kymb9/D9hEkn3Jjt+FYs/bxR70HK5mzGvu8R/zunT1f RZAYA X-Mailer: git-send-email 2.43.0.429.g432eaa2c6b-goog Message-ID: <20240123145258.1462979-69-ardb+git@google.com> Subject: [PATCH v7 17/50] arm64: cpufeature: Add helper to test for CPU feature overrides From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240123_065503_804768_6250149C X-CRM114-Status: GOOD ( 18.85 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel Add some helpers to extract and apply feature overrides to the bare idreg values. This involves inspecting the value and mask of the specific field that we are interested in, given that an override value/mask pair might be invalid for one field but valid for another. Then, wire up the new helper for the hVHE test - note that we can drop the sysreg test here, as the override will be invalid when trying to enable hVHE on non-VHE capable hardware. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/cpufeature.h | 39 ++++++++++++++++++++ arch/arm64/kernel/cpufeature.c | 9 +---- 2 files changed, 40 insertions(+), 8 deletions(-) diff --git a/arch/arm64/include/asm/cpufeature.h b/arch/arm64/include/asm/cpufeature.h index 21c824edf8ce..acd8f4949583 100644 --- a/arch/arm64/include/asm/cpufeature.h +++ b/arch/arm64/include/asm/cpufeature.h @@ -915,6 +915,45 @@ extern struct arm64_ftr_override id_aa64isar2_override; extern struct arm64_ftr_override arm64_sw_feature_override; +static inline +u64 arm64_apply_feature_override(u64 val, int feat, int width, + const struct arm64_ftr_override *override) +{ + u64 oval = override->val; + + /* + * When it encounters an invalid override (e.g., an override that + * cannot be honoured due to a missing CPU feature), the early idreg + * override code will set the mask to 0x0 and the value to non-zero for + * the field in question. In order to determine whether the override is + * valid or not for the field we are interested in, we first need to + * disregard bits belonging to other fields. + */ + oval &= GENMASK_ULL(feat + width - 1, feat); + + /* + * The override is valid if all value bits are accounted for in the + * mask. If so, replace the masked bits with the override value. + */ + if (oval == (oval & override->mask)) { + val &= ~override->mask; + val |= oval; + } + + /* Extract the field from the updated value */ + return cpuid_feature_extract_unsigned_field(val, feat); +} + +static inline bool arm64_test_sw_feature_override(int feat) +{ + /* + * Software features are pseudo CPU features that have no underlying + * CPUID system register value to apply the override to. + */ + return arm64_apply_feature_override(0, feat, 4, + &arm64_sw_feature_override); +} + u32 get_kvm_ipa_limit(void); void dump_cpu_features(void); diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c index a99ad79adee2..d0ffb872a31a 100644 --- a/arch/arm64/kernel/cpufeature.c +++ b/arch/arm64/kernel/cpufeature.c @@ -2042,14 +2042,7 @@ static bool has_nested_virt_support(const struct arm64_cpu_capabilities *cap, static bool hvhe_possible(const struct arm64_cpu_capabilities *entry, int __unused) { - u64 val; - - val = read_sysreg(id_aa64mmfr1_el1); - if (!cpuid_feature_extract_unsigned_field(val, ID_AA64MMFR1_EL1_VH_SHIFT)) - return false; - - val = arm64_sw_feature_override.val & arm64_sw_feature_override.mask; - return cpuid_feature_extract_unsigned_field(val, ARM64_SW_FEATURE_OVERRIDE_HVHE); + return arm64_test_sw_feature_override(ARM64_SW_FEATURE_OVERRIDE_HVHE); } #ifdef CONFIG_ARM64_PAN From patchwork Tue Jan 23 14:53:17 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13527721 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id C421BC48260 for ; Tue, 23 Jan 2024 16:01:09 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=812OOAinciTrjIgqjf6dd38hFsLexU31AelH4EIgYOg=; b=RsR9teLNhATR7FjWul2knNSV7u vKYEqgEj0OP3seyZ7K3B4ENOxErvU0nDuyFBmXG0NjsOwjtS7zARnbRYoK9stTW/XvpFidiqbteIF K4tyxRv5dR7HeaASMhzYIhL8xrnKtSORjZ1jEHGBjl6U5qk6KnmzxYtxD1MyE4Mgk7gJrbclSxjW5 OqGHiPYwb1trSxeALcegCcQHaXbtdLl/h45rgFpoBQVeTRNPtJksXPomlZapx+3d3g/Jzx0Ex/kXn sQTwHwDm/QtBrnAa4KVtoQ3Dkhnl4cEwldIIydhwClZ6+4Zw9b/X/KGG8J98vXnhpUAhs2w/+g3wF t9LEKXlg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rSJCU-00HAsz-00; Tue, 23 Jan 2024 16:00:34 +0000 Received: from mail-yb1-xb49.google.com ([2607:f8b0:4864:20::b49]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rSIB7-00GtuP-1Q for linux-arm-kernel@lists.infradead.org; Tue, 23 Jan 2024 14:55:11 +0000 Received: by mail-yb1-xb49.google.com with SMTP id 3f1490d57ef6-dc221fad8c7so6312672276.3 for ; Tue, 23 Jan 2024 06:55:04 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1706021704; x=1706626504; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=g+IjSAC2BSuOeuAquyaNlWu8DV5TXu9d/4vrOJzH/bQ=; b=c8i2xOFfktrwejX6nSIBkqGqUeTRnkCbWSpu749+g5vFsQ/OYeK3KIqvyhIkNvnvMc YVKTu50WveOf8L6Sx8fe4IHxwNtCl3s3qUMFAhscfQVMgMX6UiIGyWBRFoeGNqDkb82S R/gnK4LWsESYHdmjovQAcwnXjQielRueNU6bQwOCBSCJ2FK/erTsLSS0vIWMY735DnOw podMzERNjxPkoznrLRnMIIQjmg3o8cYhRBG/ur4YwHu2d/AizHIyzn8sC9pW1kzFkC4m Uxa7eIdxaicb+LWsQWpZwrzqsQ9NwvTqkn9n8VFqWugq9USxjYj17rerIfmQkcPa3ecq gwGQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706021704; x=1706626504; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=g+IjSAC2BSuOeuAquyaNlWu8DV5TXu9d/4vrOJzH/bQ=; b=LabPsWAKxmJjVyVmImlvoxJVJDX23TDrXiXo99PSZx96tZQwC/jrkME1n3OOIBOtBv 82l3xh45qtzlZ+ITZxxqJNUErK6fFXIk/WDTJh90gKb8oSSYJxqyzZYTojoLMgId5hFP h0e3DauQvVa+fLJJ2KCoAJijGJSGQ01Zk2DxFpLJZ/MLSdc9l33Du4dTQvVPzCh9f2D5 7gtiod9ve6R6pi+iWUXW/MqaiLMP2Wy+a54MGYA07pJzdRvSx3W69ce4qblzfB/D0Uam Np5axiLV37IPB+tePU+GecfMUUQbFN/FJxUnJvvQOQc1TnolWebFUxwam1YrqvKONa4b Wljw== X-Gm-Message-State: AOJu0YxntJMaX4tG3nDH2q/uRfwKc+TYQNigjHf6K8Ck6zOfscSe//jS CGuS2SjvSPUbZvzAorgnB2cseallC8WZ9OOZxwtK+mVBKX1yLn+A7n2wsF4Y4q2dtG4198BzB9n 0ZlVcDoR4PRcgsbhQMKquVpH0SrCVhj6lbjaIqvkRnODwl1v3gOFJFD5Ru2dONQjXRt4KbYgGug L2lubLtlOwrUAMC9K3dO5xfFoSK+EPZB4SBGhbKMjP X-Google-Smtp-Source: AGHT+IFuWrHvZQ/qn4ttHLipfvTc4z2YBFHqgJmC+GNDB2rDiNQF+0TOD0ENAitWVe4l5JbJh28Qb0Os X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:6902:68e:b0:dc2:354a:f191 with SMTP id i14-20020a056902068e00b00dc2354af191mr2712280ybt.10.1706021704112; Tue, 23 Jan 2024 06:55:04 -0800 (PST) Date: Tue, 23 Jan 2024 15:53:17 +0100 In-Reply-To: <20240123145258.1462979-52-ardb+git@google.com> Mime-Version: 1.0 References: <20240123145258.1462979-52-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=3267; i=ardb@kernel.org; h=from:subject; bh=5WW9MiuhQ59pNWgySa0oqX7tr/bTl6pkykpfUXS9Mz8=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIXX9pXtarLMfSX61+7OV23iB95EtXW21nra2vzZ1G/jKt e9YekCoo5SFQYyDQVZMkUVg9t93O09PlKp1niULM4eVCWQIAxenAEwkypbhf4nSZO/s+veu4V3z Wz7nej2oqNH5xqd0f825Pd0vFv8SzmBkeK7M0/FtR1GO8qv1CdOmv7zOYiC0TIelWd219kFQZ2M ZGwA= X-Mailer: git-send-email 2.43.0.429.g432eaa2c6b-goog Message-ID: <20240123145258.1462979-70-ardb+git@google.com> Subject: [PATCH v7 18/50] arm64: kaslr: Use feature override instead of parsing the cmdline again From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240123_065505_501503_5E810E17 X-CRM114-Status: GOOD ( 15.70 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel The early kaslr code open codes the detection of 'nokaslr' on the kernel command line, and this is no longer necessary now that the feature detection code, which also looks for the same string, executes before this code. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/cpufeature.h | 5 ++ arch/arm64/kernel/kaslr.c | 4 +- arch/arm64/kernel/pi/kaslr_early.c | 53 +------------------- 3 files changed, 7 insertions(+), 55 deletions(-) diff --git a/arch/arm64/include/asm/cpufeature.h b/arch/arm64/include/asm/cpufeature.h index acd8f4949583..e309255b7f04 100644 --- a/arch/arm64/include/asm/cpufeature.h +++ b/arch/arm64/include/asm/cpufeature.h @@ -954,6 +954,11 @@ static inline bool arm64_test_sw_feature_override(int feat) &arm64_sw_feature_override); } +static inline bool kaslr_disabled_cmdline(void) +{ + return arm64_test_sw_feature_override(ARM64_SW_FEATURE_OVERRIDE_NOKASLR); +} + u32 get_kvm_ipa_limit(void); void dump_cpu_features(void); diff --git a/arch/arm64/kernel/kaslr.c b/arch/arm64/kernel/kaslr.c index 12c7f3c8ba76..1da3e25f9d9e 100644 --- a/arch/arm64/kernel/kaslr.c +++ b/arch/arm64/kernel/kaslr.c @@ -16,9 +16,7 @@ bool __ro_after_init __kaslr_is_enabled = false; void __init kaslr_init(void) { - if (cpuid_feature_extract_unsigned_field(arm64_sw_feature_override.val & - arm64_sw_feature_override.mask, - ARM64_SW_FEATURE_OVERRIDE_NOKASLR)) { + if (kaslr_disabled_cmdline()) { pr_info("KASLR disabled on command line\n"); return; } diff --git a/arch/arm64/kernel/pi/kaslr_early.c b/arch/arm64/kernel/pi/kaslr_early.c index 167081b30a15..f2305e276ec3 100644 --- a/arch/arm64/kernel/pi/kaslr_early.c +++ b/arch/arm64/kernel/pi/kaslr_early.c @@ -16,57 +16,6 @@ #include #include -/* taken from lib/string.c */ -static char *__init __strstr(const char *s1, const char *s2) -{ - size_t l1, l2; - - l2 = strlen(s2); - if (!l2) - return (char *)s1; - l1 = strlen(s1); - while (l1 >= l2) { - l1--; - if (!memcmp(s1, s2, l2)) - return (char *)s1; - s1++; - } - return NULL; -} -static bool __init cmdline_contains_nokaslr(const u8 *cmdline) -{ - const u8 *str; - - str = __strstr(cmdline, "nokaslr"); - return str == cmdline || (str > cmdline && *(str - 1) == ' '); -} - -static bool __init is_kaslr_disabled_cmdline(void *fdt) -{ - if (!IS_ENABLED(CONFIG_CMDLINE_FORCE)) { - int node; - const u8 *prop; - - node = fdt_path_offset(fdt, "/chosen"); - if (node < 0) - goto out; - - prop = fdt_getprop(fdt, node, "bootargs", NULL); - if (!prop) - goto out; - - if (cmdline_contains_nokaslr(prop)) - return true; - - if (IS_ENABLED(CONFIG_CMDLINE_EXTEND)) - goto out; - - return false; - } -out: - return cmdline_contains_nokaslr(CONFIG_CMDLINE); -} - static u64 __init get_kaslr_seed(void *fdt) { static char const chosen_str[] __initconst = "chosen"; @@ -92,7 +41,7 @@ asmlinkage u64 __init kaslr_early_init(void *fdt) { u64 seed, range; - if (is_kaslr_disabled_cmdline(fdt)) + if (kaslr_disabled_cmdline()) return 0; seed = get_kaslr_seed(fdt); From patchwork Tue Jan 23 14:53:18 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13527542 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 6E68AC47258 for ; Tue, 23 Jan 2024 14:56:59 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=RkwBjAj5VM5v2p1tkuY9bGwz/mB71agn1VBKgyHR5f0=; b=MG6n6lRC9WoVKmxpq/95U66Ab7 vm01X5WyLEKAB3RJVKKj9lXfOZ5CdW8vsGJoVQHI8JlpuxfScWflK2i/GQHfscrRYNy1ioeRAgJbv 41V/0e1plH9z8jYA4yGJ22QZHJAJc/sCwADD98OgTTh9gJ4CBS1t8Rb7Bp0QbDtUksXX48PNtv33Q I03d4ML0evgscD5lqrhoHmolfRyjljY8bCPY3KpCn8AG1tZH/d8mfGGPUo4icuHI1QHX5kDigrjZW 1fOcrYHIWxYyNBfCnxyGQGpmW0ZW1Snu4/qfpI8kcayn68FDIwL50GX9zN7ckibfigUIcdTzzePSm bYO1vTYw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rSICO-00Gub6-02; Tue, 23 Jan 2024 14:56:24 +0000 Received: from mail-wr1-x44a.google.com ([2a00:1450:4864:20::44a]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rSIBG-00Gtvz-32 for linux-arm-kernel@lists.infradead.org; Tue, 23 Jan 2024 14:55:24 +0000 Received: by mail-wr1-x44a.google.com with SMTP id ffacd0b85a97d-33941f7e299so461306f8f.0 for ; Tue, 23 Jan 2024 06:55:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1706021706; x=1706626506; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=u5Ytiqm7DJK16NBOWkHGv3NwmmxkyHPu/nv4smY9SHw=; b=S1gK2fJw55C0Ey/lX72c2MhNRVfzZE5TuYmKZH67WZYzjDwqf/ZkrsKvsJAxOeF0HS hAvFcPOdp3xgGzk831YHOIzCVJeHU3D42OYPy8cxUbhiwZvnMw3TXUXGQqFPelQfFNAZ 2tAma4L/qdavvxGNlfXDxI9wd9GU593/rra4+++qKeLhnLmZQ6tlB/9ucLkHe9PMoX/S X3IfATm79mJ97WSwCMiN5BPVMQpK5DR5zdDBo58us+0vcdf1GzCKzTV4j5kEL+r6SUpn wDLw51ZJeY9nPx9E181K/R8RZ3zprj3wuCrywbsTcI/uRCplvf1RCwyspDi65GPXOIbp jAAQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706021706; x=1706626506; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=u5Ytiqm7DJK16NBOWkHGv3NwmmxkyHPu/nv4smY9SHw=; b=vcP2uyOT2CjNXrQUad+uFyK4rFdHrM2HwX4KhqR6wgtJ+XbxfVKWAaVNeG8Hvjd0pC oyI2+V/BdCczBh1NaLaqs+mbd4Hd4lwt9eATYbwybSK3fN2Ad7uehFfyOaLnGep5qLzw IkXcnDX7eT9Fopc8qXSEsAcyr3Q/WmmsBzAdA+qPkysOoWaRYuUvmdRiYc9ojq4dp0bd gciaXBzlnyP1XLKvz9HI3Gv4mwTMt5P2NxNgZX0qf5+9GQ3RuAjgZiMEDAonOth9++HG c03M1ZiS19W3afurxxJ0MzYUjx6ZNcALb8rVU/8wIWzjfcc0sQ9/IdWmrlJMKebqVsfx 2kmw== X-Gm-Message-State: AOJu0YwhI90oLyluMYZLQwSPe+61Mu1AKA5VNQqUj3QOlcO5xhQPcvtH SKiQsGF4peZf9XLpUsPjnG9ZuwG6pT3txAH0UOA8IjJcA0BQ51VgzPUOOESnJe+xN3PS/OFJmAF GC1dnLtv1S5gjGjFCBU4rA7QfQl1fmYab7MQHkg4avyVma1oBQSvBxzFV74L8E1K/Q40/kq308q aizZWFHA+BNPTNQ61k/+1SLGrPl0TOWi2GHP+bTcuA X-Google-Smtp-Source: AGHT+IEuPt2stG/N6BSal7k5ZGPaDygWtpeGvnjTPB4Fp/0FkaZwBL1P0N3exHpnGuKdEGtCV7inaHJ3 X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:6000:1f0b:b0:339:45ec:a004 with SMTP id bv11-20020a0560001f0b00b0033945eca004mr2349wrb.4.1706021706304; Tue, 23 Jan 2024 06:55:06 -0800 (PST) Date: Tue, 23 Jan 2024 15:53:18 +0100 In-Reply-To: <20240123145258.1462979-52-ardb+git@google.com> Mime-Version: 1.0 References: <20240123145258.1462979-52-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=1646; i=ardb@kernel.org; h=from:subject; bh=xxBqTnYojIEb8ydbpndpvYYfpJ0Y7OfGgXRsNkNYvt0=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIXX9pftJqft1WudwimUua3z9/W34LC2H5Lgv7e+/6otem tXz4JpfRykLgxgHg6yYIovA7L/vdp6eKFXrPEsWZg4rE8gQBi5OAZhIsD8jw4drr+Yd7G83PCnf vqp6Y95HlZMl0jULRGZoJmZGKl+64sPwP9Lkc6J8525WvYNmxXwq5n1Srz7eW/wz9PYVfX4OqTf cPAA= X-Mailer: git-send-email 2.43.0.429.g432eaa2c6b-goog Message-ID: <20240123145258.1462979-71-ardb+git@google.com> Subject: [PATCH v7 19/50] arm64: idreg-override: Create a pseudo feature for rodata=off From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240123_065515_021897_0B836FBE X-CRM114-Status: GOOD ( 13.15 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel Add rodata=off to the set of kernel command line options that is parsed early using the CPU feature override detection code, so we can easily refer to it when creating the kernel mapping. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/cpufeature.h | 1 + arch/arm64/kernel/pi/idreg-override.c | 2 ++ 2 files changed, 3 insertions(+) diff --git a/arch/arm64/include/asm/cpufeature.h b/arch/arm64/include/asm/cpufeature.h index e309255b7f04..03c34242bfc7 100644 --- a/arch/arm64/include/asm/cpufeature.h +++ b/arch/arm64/include/asm/cpufeature.h @@ -17,6 +17,7 @@ #define ARM64_SW_FEATURE_OVERRIDE_NOKASLR 0 #define ARM64_SW_FEATURE_OVERRIDE_HVHE 4 +#define ARM64_SW_FEATURE_OVERRIDE_RODATA_OFF 8 #ifndef __ASSEMBLY__ diff --git a/arch/arm64/kernel/pi/idreg-override.c b/arch/arm64/kernel/pi/idreg-override.c index f9e05c10faab..e4bcabcc6860 100644 --- a/arch/arm64/kernel/pi/idreg-override.c +++ b/arch/arm64/kernel/pi/idreg-override.c @@ -159,6 +159,7 @@ static const struct ftr_set_desc sw_features __prel64_initconst = { .fields = { FIELD("nokaslr", ARM64_SW_FEATURE_OVERRIDE_NOKASLR, NULL), FIELD("hvhe", ARM64_SW_FEATURE_OVERRIDE_HVHE, hvhe_filter), + FIELD("rodataoff", ARM64_SW_FEATURE_OVERRIDE_RODATA_OFF, NULL), {} }, }; @@ -190,6 +191,7 @@ static const struct { { "arm64.nomops", "id_aa64isar2.mops=0" }, { "arm64.nomte", "id_aa64pfr1.mte=0" }, { "nokaslr", "arm64_sw.nokaslr=1" }, + { "rodata=off", "arm64_sw.rodataoff=1" }, }; static int __init parse_hexdigit(const char *p, u64 *v) From patchwork Tue Jan 23 14:53:19 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13527725 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 1C3AFC47258 for ; Tue, 23 Jan 2024 16:01:11 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=Qgknm07jsIAYXLcjSUCdNPgiShxZqiwf2KE+a4a/4+8=; b=lRrJGBOLfj6OAs7BDmp5MYKgHl YUGvjM+kFws0f72QrTzedczcL60Q8pZWHDsfkOX3KH3l68SDxx7V4TumH5lXJl613TIfX50zgPq5V 3d3tYSpr1h9oLlWQMnK5ff2pvCVfVZY9mQ1PrxgIa7Q8xg4Y/X97JFGe4sLduCkvNTBbfcH/xGFJh tRbnqptujEZA7U37Ct7Uha+8TDg29Q//oVKDE2B2GJhg8Xh015IU8RBUPt1ec5spP0DMWZH/X8SYa Gj0D+B142lihLBGIeVw+AGrs47HIEusciCf8uSp6DCHw1LdjXTS+hI6TZRJXLRJCS+NS5YpQzW2Lg XIUIwhaw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rSJCV-00HAu6-0T; Tue, 23 Jan 2024 16:00:35 +0000 Received: from mail-wm1-f73.google.com ([209.85.128.73]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rSIBG-00Gtx6-18 for linux-arm-kernel@lists.infradead.org; Tue, 23 Jan 2024 14:55:21 +0000 Received: by mail-wm1-f73.google.com with SMTP id 5b1f17b1804b1-40e6f3c5730so41432175e9.1 for ; Tue, 23 Jan 2024 06:55:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1706021708; x=1706626508; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=Yp7yk/Cag4dYbspz2hNgNv5OLTeWFwTs8/hp9F7aYHY=; b=bf1xo1ZjXsDOw4ELoP0CcUqBWI3lHnwF2miJzhAiNU7pduagvDnAEDBiVNP4A8lSF+ OUDCQj+Br23ImO7ONga5f7tvAqwA80AoH+JY/eoBCHG4klgRrmpAt8kuB1ZC6pevWpGh Cd2s2x7+a/k99+/uMsS9b9QQQ16SzvEkXJZenzv3m4nHWjE3TaP0Bebj0bsv7/gblAN2 9OYHwmJSZfajXGAg/KpQiQNNGNlkv6GzO801p4Sh/602pYBwgXlLMfVrk0zd9P3toRoS MzDQYmA1eSZUJs3YCGjNLfF1CcHL23idDnbsPt7eja8ddY1H7SIU7pzG5ZhZIZbHIRjK xhIg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706021708; x=1706626508; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Yp7yk/Cag4dYbspz2hNgNv5OLTeWFwTs8/hp9F7aYHY=; b=mxDoQ+RjSMxHo6azbhM4aHyIdouZV09DX/+dckmnCK+4c+LcU5gIrQuG0ERwewNYKO 4IxPc4+yR1CPQVv71fJNbbawwUPMdL0zd94R/glxo1rU/6ulNvfLeQRC+VCueQuxC5R3 JdaYH2AFywF6zAURdO6c6ec7iv/dxYvEe+TN/JoZnJFfO3GfOGZNauvfE+emxulUJm/t 2SJe7odNteCdzw6+ibztlcQCne7JWb+KpzOXTNSBDjiQLjDkC96kSG223Ih9Qn758gR/ jiyYIundgGiFV3t1iJRmjpo5IrruOnl+hocU6KBRMKyqFQhZ0PQGwFcPjtUsTWHRzobE aALw== X-Gm-Message-State: AOJu0YwN3HHElRSXWhbD8tIeOg03FLIxOOfy1qZay0AYCdWkMc2qrGwc W+lRPfODNKpbjqciHNu+jFalP+vNdHwpdhJliVUcF3DOXlJymbV629f7/TqeZV33NOxuLLNyyKm VHT0BzRHTKTWCqb+7ElJv9kZ9S4CDztBoM0W/Xq/irZEjPmo+HEo4DlN//CJUoEhfotC2ECS3u7 dbVTacJpTedEmrW2WZ7nVqqk43X1zcOBE2krQkYIWS X-Google-Smtp-Source: AGHT+IHFVV3f/zpQo6hbo7JSX//vYpTjJ6oNZh6Uae3un0U5ERIBnjMN7Lngk8jj8Wr8GGf3sJ1NM/Fl X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:600c:b8e:b0:40e:bf34:46b8 with SMTP id fl14-20020a05600c0b8e00b0040ebf3446b8mr6413wmb.5.1706021708519; Tue, 23 Jan 2024 06:55:08 -0800 (PST) Date: Tue, 23 Jan 2024 15:53:19 +0100 In-Reply-To: <20240123145258.1462979-52-ardb+git@google.com> Mime-Version: 1.0 References: <20240123145258.1462979-52-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=1676; i=ardb@kernel.org; h=from:subject; bh=Tss2ysfkGPhsFex5mRbRurkjoHKWotU+L9jnR8SQpAE=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIXX9pQePNW7IvTslyGZbpr4xlvcb88Y/v6Ru/rxz80nd6 VNZ7+z/dZSyMIhxMMiKKbIIzP77bufpiVK1zrNkYeawMoEMYeDiFICJ3P3B8N8tnrmgrHVnwvkl 7u9Pz6sUXnCMVUve9pRLafxsp4wPr18w/GQ8c3bNV9+M5xKPcr7lXT395L3Q1XxRRqMwnheyRYs TDrEDAA== X-Mailer: git-send-email 2.43.0.429.g432eaa2c6b-goog Message-ID: <20240123145258.1462979-72-ardb+git@google.com> Subject: [PATCH v7 20/50] arm64: Add helpers to probe local CPU for PAC and BTI support From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240123_065514_426603_74FE3A10 X-CRM114-Status: GOOD ( 12.55 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel Add some helpers that will be used by the early kernel mapping code to check feature support on the local CPU. This permits the early kernel mapping to be created with the right attributes, removing the need for tearing it down and recreating it. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/cpufeature.h | 32 ++++++++++++++++++++ 1 file changed, 32 insertions(+) diff --git a/arch/arm64/include/asm/cpufeature.h b/arch/arm64/include/asm/cpufeature.h index 03c34242bfc7..e3edae1825f3 100644 --- a/arch/arm64/include/asm/cpufeature.h +++ b/arch/arm64/include/asm/cpufeature.h @@ -963,6 +963,38 @@ static inline bool kaslr_disabled_cmdline(void) u32 get_kvm_ipa_limit(void); void dump_cpu_features(void); +static inline bool cpu_has_bti(void) +{ + if (!IS_ENABLED(CONFIG_ARM64_BTI)) + return false; + + return arm64_apply_feature_override(read_cpuid(ID_AA64PFR1_EL1), + ID_AA64PFR1_EL1_BT_SHIFT, 4, + &id_aa64pfr1_override); +} + +static inline bool cpu_has_pac(void) +{ + u64 isar1, isar2; + + if (!IS_ENABLED(CONFIG_ARM64_PTR_AUTH)) + return false; + + isar1 = read_cpuid(ID_AA64ISAR1_EL1); + isar2 = read_cpuid(ID_AA64ISAR2_EL1); + + if (arm64_apply_feature_override(isar1, ID_AA64ISAR1_EL1_APA_SHIFT, 4, + &id_aa64isar1_override)) + return true; + + if (arm64_apply_feature_override(isar1, ID_AA64ISAR1_EL1_API_SHIFT, 4, + &id_aa64isar1_override)) + return true; + + return arm64_apply_feature_override(isar2, ID_AA64ISAR2_EL1_APA3_SHIFT, 4, + &id_aa64isar2_override); +} + #endif /* __ASSEMBLY__ */ #endif From patchwork Tue Jan 23 14:53:20 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13527729 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 785B5C47DDF for ; Tue, 23 Jan 2024 16:01:12 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=mSjMe1/k197A6FLeCq00zWTy1PpxLrhU5FqN4QjOEm0=; b=m0XC6qyQvaSItJUwbWlmFzffgo Zyipw1Y6JRicPH4/jo49UaC1MIxz+8VvFdhPB2vFflM9kdNCvxo0CaMtaKaJZwIdy9qIjVyky59mZ wnHpFHqBUxKGsFgUrviNGhEYCnMWSSJzRJeLoIHpTeOe+xulrPvVfz3JjIvzVcT/xAaADLlwL2Zwq Vvv7U7EqFLW3C6nDQDRSXcN0WDISW2LB4AcIAXM3pAiQzuxDhGqx6hzidBPJLYwFKKs9deY1NZp2J wRRzzbDRrC3hL+S3UDrPJuoodU4XCLPCcW36EWmGRwrdD/YSvUuMrU2q8Nw83br3jZ0d0aDS4TK77 awRUI6lg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rSJCU-00HAtP-1n; Tue, 23 Jan 2024 16:00:34 +0000 Received: from mail-yw1-x1149.google.com ([2607:f8b0:4864:20::1149]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rSIBE-00GtyM-0x for linux-arm-kernel@lists.infradead.org; Tue, 23 Jan 2024 14:55:14 +0000 Received: by mail-yw1-x1149.google.com with SMTP id 00721157ae682-5ffac5f7afeso44952167b3.0 for ; Tue, 23 Jan 2024 06:55:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1706021711; x=1706626511; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=Uy1Vj/NfeSeXiLCQRi32LsAk8kkB+RPPEQasS6jcwuo=; b=DWtzlXBjUbFk/AiM0kY1wnSLUHtczKRsx7Xucjy7wpSRrWHzIJ4J6seI+ycxWeY2GW npiGnyOpQY00uRUf9grUJlZuIqhLH6/CjYdJoXmoKOTprBH76UAuS0+LjfKRqkGX8tZn 3zSDmW7CeXzECrGupe8B+dQ7g35bxBF43wk5ad/XzPUBJpnFPV16GTVnQDYFHivchp+0 qUv4J23yvUhH+kMrFleaslzarjvlDW5dooVDu6iBZoPllxVpuUEOvDLpgJxfE2hvnpIs 5OGAyCePmoxhwW8zQdberRgzIx6aTgvX10MnfocVO5lM5Vxl8UIgYk0ytwUpGJQHrAly yU1A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706021711; x=1706626511; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Uy1Vj/NfeSeXiLCQRi32LsAk8kkB+RPPEQasS6jcwuo=; b=s9qWge+V10kJAeDonsaLXjXyA088/TOzMhHaJ+7cXs124zUo6TdJM2p39ECKXwBUuv p+75FS9/Ck0P4t8UYjGLXxIAOPMuP0MbLLwE64myyk4gyn/oxZMf0BLz5o9AkXgBxXNz NES4qXpNt1igmEX9Y8kawwIXW6VHhMUFxAYxpYIOdSjsmKRgJqSjuXnkd/wxBQBtSPD3 iR8yRiTzqA6pmmg+uNvuZjxeSt9/9fK546VdBbAuDRZFIhGSEHVKS5s2ay1fYLk3jgh6 ThRIYsenW2IpJZXoz3G4jEf25Um2ocDxUEyuaKBRTkDWDhiYCic7aYvNJj7hAB0tewcw sVuQ== X-Gm-Message-State: AOJu0Yw7DidQGOwi2roCfgGhWC6fl/lAftKQ90qbkvGUxxwdFERBrdM2 wa8bsofWpWi2JOwwBNj7Rn7vvR0TYY9+rd1pVCIxWa0v412FDnzaoYAkZycX7kpvVgIi06iUOzP gqxn2qdl0C22Q/cSEu+9Zl8tnR5fUZMuIFHQqgNzi85qaPQI/5WxK9fiUgGVUDUbSwWBWHtT5cP ud1Pooz6ATBsoA9Ew39hJHJMHGPYUs247l6WpgDcfJ X-Google-Smtp-Source: AGHT+IHOMSjLlgyngzm0Ir4FvJIOwJWjvz2KuUtwO3dc09qTTNSHgCBV3fA/lpFkFrLAonR/Nw0ay5IF X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a5b:a42:0:b0:dc2:46c5:649a with SMTP id z2-20020a5b0a42000000b00dc246c5649amr2594671ybq.12.1706021710786; Tue, 23 Jan 2024 06:55:10 -0800 (PST) Date: Tue, 23 Jan 2024 15:53:20 +0100 In-Reply-To: <20240123145258.1462979-52-ardb+git@google.com> Mime-Version: 1.0 References: <20240123145258.1462979-52-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=1834; i=ardb@kernel.org; h=from:subject; bh=RdZd8uoP/KZBVeewE8k4d/d2vzCDE5jaFUp/FdYLmLg=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIXX9pYeshnqv+Zo2GN7ZYXvVZtLh16p2xnLLHNN7b03Ku hSbcTW+o5SFQYyDQVZMkUVg9t93O09PlKp1niULM4eVCWQIAxenAExk2yRGhkOr962uOpNbOcdh yt3IGWsfaun8m3uXIfbgrN8azukqAdsY/nCsvFibeG5etlzSzw3bmzk1HOslWl2n7vOexP7rae1 6cQYA X-Mailer: git-send-email 2.43.0.429.g432eaa2c6b-goog Message-ID: <20240123145258.1462979-73-ardb+git@google.com> Subject: [PATCH v7 21/50] arm64: head: allocate more pages for the kernel mapping From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240123_065512_375849_8866ABB0 X-CRM114-Status: GOOD ( 13.86 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel In preparation for switching to an early kernel mapping routine that maps each segment according to its precise boundaries, and with the correct attributes, let's allocate some extra pages for page tables for the 4k page size configuration. This is necessary because the start and end of each segment may not be aligned to the block size, and so we'll need an extra page table at each segment boundary. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/kernel-pgtable.h | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/arch/arm64/include/asm/kernel-pgtable.h b/arch/arm64/include/asm/kernel-pgtable.h index 83ddb14b95a5..0631604995ee 100644 --- a/arch/arm64/include/asm/kernel-pgtable.h +++ b/arch/arm64/include/asm/kernel-pgtable.h @@ -68,7 +68,7 @@ + EARLY_PGDS((vstart), (vend), add) /* each PGDIR needs a next level page table */ \ + EARLY_PUDS((vstart), (vend), add) /* each PUD needs a next level page table */ \ + EARLY_PMDS((vstart), (vend), add)) /* each PMD needs a next level page table */ -#define INIT_DIR_SIZE (PAGE_SIZE * EARLY_PAGES(KIMAGE_VADDR, _end, EXTRA_PAGE)) +#define INIT_DIR_SIZE (PAGE_SIZE * (EARLY_PAGES(KIMAGE_VADDR, _end, EXTRA_PAGE) + EARLY_SEGMENT_EXTRA_PAGES)) /* the initial ID map may need two extra pages if it needs to be extended */ #if VA_BITS < 48 @@ -89,6 +89,15 @@ #define SWAPPER_TABLE_SHIFT PMD_SHIFT #endif +/* The number of segments in the kernel image (text, rodata, inittext, initdata, data+bss) */ +#define KERNEL_SEGMENT_COUNT 5 + +#if SWAPPER_BLOCK_SIZE > SEGMENT_ALIGN +#define EARLY_SEGMENT_EXTRA_PAGES (KERNEL_SEGMENT_COUNT + 1) +#else +#define EARLY_SEGMENT_EXTRA_PAGES 0 +#endif + /* * Initial memory map attributes. */ From patchwork Tue Jan 23 14:53:21 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13527543 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 860F5C47258 for ; Tue, 23 Jan 2024 14:57:02 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=/6tqMSPehbNfqAt0noVQad9CpHD2ilVtoFnCNyt2K1M=; b=l6BTktV+geoMxAslRrQ5fjI8Uv fCEXdDcJ54mbICoi5Q0tUQXoLxcjXz9TIWh4MQ9uq9ZgLOrFacmWiNstctnQeQ2nVU/OX0ytdpLLw mJ5x95/bTOQKGtEnEupCoFT/xr71NMdzQh60Apr0n+wvOoADLYQonjPWs6+t6Cx4OJ7F+tGEIVhRp ih4Y+6F7gdxho9uWAsXGNUE/cWQX6vsBrNx1Q6MEVoATiny80iIpfv4tk66D5tofbcIQl7ST3q+de GLXCfWqjRQCT9LG670GjA9i4Y77xQX0SPgNktjxGkoiko6KS0QME9KZIfb5WqZ1JPAZfk3lhItGOW P/LwiDXw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rSICT-00Gue9-10; Tue, 23 Jan 2024 14:56:29 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rSIBZ-00GuCt-29 for linux-arm-kernel@bombadil.infradead.org; Tue, 23 Jan 2024 14:55:33 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=GdG6OVFnEQ+dH+HNISsEfMCC9piil90r9HvYsrUMWtA=; b=VfZXW3ef9XDzoIqEqWJHyDwVJJ OhP/2cPyL0UFUZA5b0c+483/B8vpffdeb4Zwr7GBDW0zVGcuZp+oO99fUI0DEzx+JqGiPAIs+9MGy ZiMKbjL2A6bHe/7AOXxwZNN5twl5UAst5NPvGqehHW/2rgE8+vOZXLJ1iLoprbxfNQ4vsfZAvx4hV TDTHhdrmIjLQqQbcSYnJP1K0z6uILZw38UUR/2IYHy0ZofyWetqzj5aYDbz3Ryv4zTzVMuS56nof+ qBjW0Cz9RWBg52OQ0UocJNBjwaVCHIkat6ryV9NCAM+vM1jFalqWFVqT9Bus00aGw5LBBnKyDD9yb ulKBk3zg==; Received: from mail-yw1-x1149.google.com ([2607:f8b0:4864:20::1149]) by desiato.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rSIBR-00000003vON-0ewd for linux-arm-kernel@lists.infradead.org; Tue, 23 Jan 2024 14:55:32 +0000 Received: by mail-yw1-x1149.google.com with SMTP id 00721157ae682-5e6fe91c706so74933557b3.2 for ; Tue, 23 Jan 2024 06:55:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1706021713; x=1706626513; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=GdG6OVFnEQ+dH+HNISsEfMCC9piil90r9HvYsrUMWtA=; b=oSuPcK4THvO79qf6+aQhG2etJsiPDMGBnoNaRcmIcsc0eSLMcRSJLz1cLUYPGiHv5E 6uIenKNzM0EkTj14s8MSPjkVDRenEcOtqUAR49ebp9qNZCK7ICtRpXgjyuj7MMu+ROiO jHfxZLR4GK2FtkNuYOBracZl5Bn0tQ8AJwNA/iPp6PpKJZCCqj+HtCS2t/M4YWtkcRlV IDEcS2Lo2SXsMFWmoU0z59f7IlgTuNJtQ1xCkdeEX9hwMi4zi/vDy3z9ZmLA2uMOxxe+ OI8+4W5zLSzIFck6ueP/a+ilI9KX6FmJ8AzzR5GXPblB1UyilV/sVsnTNfGzAYtmc5yO 7kTw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706021713; x=1706626513; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=GdG6OVFnEQ+dH+HNISsEfMCC9piil90r9HvYsrUMWtA=; b=MhvEA11h4Pp4z9wpdkHPbFXvZICMc+8OmjS+8MtxViXZK14P5BpREpJO56kcsDnpQU 1aCGEBo+7XnmcCcqhuJZXn8WAE8tlQoIz2rx7cLsfEhduLGV3er7yF1keem1PZOiB+zT mveYWj6udbjCwPT5Ds7yz7UKWcffb/Vj2OYiX3bgFt6l8pwNkUbT9TuuZgPc1LxgsYnY Gv81bN+MIm+ATkaj9gYq5W1xb25wnflQiaGPYAuy+om8QJVJEQQKT8Pk6UuDoMIIxK1k 8Zfj5wsaTF1x14sYNbiP03Yde3K5zuPz3bS8P0jwWtYUEXltG4/GZJce/dH13bVxSFa0 1bGQ== X-Gm-Message-State: AOJu0YyND28RzDGFfkcMu4pcgg/x6w11rEhDDg2SbV3hHhqCPvvE1VX5 E4pxMha8LLK2MHIBNMDWwfpLoupIoW4U0D/GKw4PY+cKBeQ+RpHbMgujDl3Cwgb7sgNkb0pOUJ4 xwJiUAs8xymQUx3LLRZrTAH7GIz9Aj9NZOAmaBo6rkpqW8ucQR1LimHbfu/GQMlJDRGaolBAlrf vmXTH5SDZKHrJ43en5qTGFujHuvLS+YAeE9VBmKwHt X-Google-Smtp-Source: AGHT+IFb6HrsrThLPxS5Q4MyDJdhAdoBDbf5JN+4R/F0o8MmciXNnwLzCKVdeszHAGTVn/sgG9UBrj+B X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a81:8487:0:b0:5ff:a62d:e2a with SMTP id u129-20020a818487000000b005ffa62d0e2amr2937895ywf.4.1706021713059; Tue, 23 Jan 2024 06:55:13 -0800 (PST) Date: Tue, 23 Jan 2024 15:53:21 +0100 In-Reply-To: <20240123145258.1462979-52-ardb+git@google.com> Mime-Version: 1.0 References: <20240123145258.1462979-52-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=3177; i=ardb@kernel.org; h=from:subject; bh=msHSAl0RyM+v/A+LmM2ZEJKkw9FXy9rp8/sq2Dz3h4o=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIXX9pUdx0ywmlKa+mSMpWl1Ut+5tpTzLm3CmNrFnjrEz6 7/ZTTfqKGVhEONgkBVTZBGY/ffdztMTpWqdZ8nCzGFlAhnCwMUpABPh42FkmCFw5NvcrabzPWbf 61hv6X5pjsfl1nQGpYcfKqS/ys4KKWRkeMokXjvzZcSr5uq9i4VZSrRkGKu3Vq1iT9S9uG7pUb+ 3zAA= X-Mailer: git-send-email 2.43.0.429.g432eaa2c6b-goog Message-ID: <20240123145258.1462979-74-ardb+git@google.com> Subject: [PATCH v7 22/50] arm64: head: move memstart_offset_seed handling to C code From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240123_145525_728103_1A1C34B3 X-CRM114-Status: GOOD ( 13.86 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel Now that we can set BSS variables from the early code running from the ID map, we can set memstart_offset_seed directly from the C code that derives the value instead of passing it back and forth between C and asm code. Signed-off-by: Ard Biesheuvel --- arch/arm64/kernel/head.S | 7 ------- arch/arm64/kernel/image-vars.h | 1 + arch/arm64/kernel/pi/kaslr_early.c | 4 ++++ 3 files changed, 5 insertions(+), 7 deletions(-) diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S index b320702032a7..aa7766dc64d9 100644 --- a/arch/arm64/kernel/head.S +++ b/arch/arm64/kernel/head.S @@ -82,7 +82,6 @@ * x21 primary_entry() .. start_kernel() FDT pointer passed at boot in x0 * x22 create_idmap() .. start_kernel() ID map VA of the DT blob * x23 __primary_switch() physical misalignment/KASLR offset - * x24 __primary_switch() linear map KASLR seed * x25 primary_entry() .. start_kernel() supported VA size * x28 create_idmap() callee preserved temp register */ @@ -483,11 +482,6 @@ SYM_FUNC_START_LOCAL(__primary_switched) str x25, [x8] // ... observes the correct value dc civac, x8 // Make visible to booting secondaries #endif - -#ifdef CONFIG_RANDOMIZE_BASE - adrp x5, memstart_offset_seed // Save KASLR linear map seed - strh w24, [x5, :lo12:memstart_offset_seed] -#endif #if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS) bl kasan_early_init #endif @@ -779,7 +773,6 @@ SYM_FUNC_START_LOCAL(__primary_switch) #ifdef CONFIG_RANDOMIZE_BASE mov x0, x22 bl __pi_kaslr_early_init - and x24, x0, #SZ_2M - 1 // capture memstart offset seed bic x0, x0, #SZ_2M - 1 orr x23, x23, x0 // record kernel offset #endif diff --git a/arch/arm64/kernel/image-vars.h b/arch/arm64/kernel/image-vars.h index eacc3d167733..8d96052079e8 100644 --- a/arch/arm64/kernel/image-vars.h +++ b/arch/arm64/kernel/image-vars.h @@ -46,6 +46,7 @@ PROVIDE(__pi_id_aa64smfr0_override = id_aa64smfr0_override); PROVIDE(__pi_id_aa64zfr0_override = id_aa64zfr0_override); PROVIDE(__pi_arm64_sw_feature_override = arm64_sw_feature_override); PROVIDE(__pi__ctype = _ctype); +PROVIDE(__pi_memstart_offset_seed = memstart_offset_seed); #ifdef CONFIG_KVM diff --git a/arch/arm64/kernel/pi/kaslr_early.c b/arch/arm64/kernel/pi/kaslr_early.c index f2305e276ec3..eeecee7ffd6f 100644 --- a/arch/arm64/kernel/pi/kaslr_early.c +++ b/arch/arm64/kernel/pi/kaslr_early.c @@ -16,6 +16,8 @@ #include #include +extern u16 memstart_offset_seed; + static u64 __init get_kaslr_seed(void *fdt) { static char const chosen_str[] __initconst = "chosen"; @@ -51,6 +53,8 @@ asmlinkage u64 __init kaslr_early_init(void *fdt) return 0; } + memstart_offset_seed = seed & U16_MAX; + /* * OK, so we are proceeding with KASLR enabled. Calculate a suitable * kernel image offset from the seed. Let's place the kernel in the From patchwork Tue Jan 23 14:53:22 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13527541 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 65F6DC47DDB for ; Tue, 23 Jan 2024 14:56:55 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=25o3HSQRc+zWEfc6JQrDH7MWyTnbL9zkLEKIDswq4RI=; b=2qwCVqedfKoP+45iQnx54n5vMM WNgMp4qyxfygk8sXq8GJJUH/kJIdClNudt+H7LypXDArE+0gEuVLoGF8BG1h6v5UDg0UcraNYmWsM +oEjbKxiVAfFKuUc59Zru96P6LIqzmos6SR6ATGr0sahxymR/8YA9nL6TJZKLZJ/9QfgJgJbKlIkT Sz01unrugslvKU1MHyF5ijN/t80neviZjnnOu8ECZsE4B5H1ZOVeVr+vfpfH/rY7KMXPXW8pvJ3CT zBF25VODSH7jkYHhMP2rkzosh/6flESEc/Is6jMxpJosyWOWnvZCX4bhob5vP3ydLklp1s1RZ6qe5 8vmWyr4g==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rSICO-00GubZ-2g; Tue, 23 Jan 2024 14:56:24 +0000 Received: from mail-wm1-x34a.google.com ([2a00:1450:4864:20::34a]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rSIBJ-00Gu2s-1I for linux-arm-kernel@lists.infradead.org; Tue, 23 Jan 2024 14:55:31 +0000 Received: by mail-wm1-x34a.google.com with SMTP id 5b1f17b1804b1-40e478693bcso24533495e9.1 for ; Tue, 23 Jan 2024 06:55:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1706021715; x=1706626515; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=Ukkq5zTjwFGShePAgonGWwjW49P0qsEhwo2lDCGDIME=; b=kv//iBtzrPoXPmTmvsDGmA2IoDGsNGRQ6sTFeOlBP9rHtMixFddOaKtsVsXhONVWTt KrtL9Sy7N2wP171sq5u36dZaOVw7IeVvgfoPJB5j/VtupZnZoHmkChvy1Jgan5WJKWkf 1Fz6jETe3zGh8AEEuZP3o529awiZL5aEDfGge+wpra+h/gmL0+1vBmKYjjZjprP2uRJY onKGkbNSTjNrDmn39F028WV/QDGohgjYVwGiPqG695pflyIJM/bXJ3usmbe2vPl0P+bG OilxeztM5d5vAxuOPeIc+09U5vUDWJYmO3MPJLjqXYX2KUuKszlMb/6ioEiajtLC52vu Bc6w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706021715; x=1706626515; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Ukkq5zTjwFGShePAgonGWwjW49P0qsEhwo2lDCGDIME=; b=jfF+DHp3YSX/UEB6J3dkPrVwPU6WVMd+8+zZbN1AgTIukh7Bge4r7ELRGsKvueK5B2 5v2ldCIetj8Ij9kmqP9C5LzfIFPcnXH0kdB+NLFDZ1H/MG2Nb9MMNctBkSUZqht545dG zcbZEOBWGXF0kYX3zyaZT6FUXSooVYuUd0ZLXiq2x+uU/r/Xn71PGuOhZOi9GCbR5+wO jb9HmZniOY1QxsLXwvbkCul/sV0+kJXY+9bXOHCqKEhJNjqmL+LzlfaP9X6DgbD2+Fum QR97+ouVTCVO3uyUYq6fwLV6BbT8xarJ6VU+8Ik+21CxPF7kD0eGJZHz7IOyQzxHlAP6 2H4g== X-Gm-Message-State: AOJu0YytqRHBfxxCBMlcpUDjDVI3Cdz+9cJ8Px7dByI12WjhrAderaPP y9t6NIEdB/CLYGK514HxHSXfSfUL17uIkxWdvrIYN+LYBKlugnT+yEnk0vASKsDefYhuYJJuG2G 2PqrbTYsYJ9PRzHKW2Go82vJihtnrJuA4Kj3wCpWnYh5cOw6jad3uUnISsD5IvEGNeI5o/wO+uN S9HzQjO4mw0M61vFen2t2AZDU8h6L+WAb6d5qZo7zV X-Google-Smtp-Source: AGHT+IHGwM6WLchL0EH1pAuT/pcSWcSVupdfU1KXVpzCa1obUlSANj5imp+X5QDnkU6QkK41p9bucSdt X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:600c:b93:b0:40e:9e30:5722 with SMTP id fl19-20020a05600c0b9300b0040e9e305722mr96547wmb.4.1706021715511; Tue, 23 Jan 2024 06:55:15 -0800 (PST) Date: Tue, 23 Jan 2024 15:53:22 +0100 In-Reply-To: <20240123145258.1462979-52-ardb+git@google.com> Mime-Version: 1.0 References: <20240123145258.1462979-52-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=5075; i=ardb@kernel.org; h=from:subject; bh=AZz7FTFLHXTxu+8k1f0CpRqTjjs7igvnlb/1XVxMwRU=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIXX9pcdJAt2dkwqKD//Sijyl8O5Nsc31w7kXT/D5vdusf ONZ0ZfDHaUsDGIcDLJiiiwCs/++23l6olSt8yxZmDmsTCBDGLg4BWAiDuoM/wu5+/3uX+mQijls fuz0yhP7mR5EXPzMmuiwaP1Ps2Us7/gZ/hfJPFrglGcpctvFb4bbs61TP/x6u9iG4UqJi1Tc8uW F69gA X-Mailer: git-send-email 2.43.0.429.g432eaa2c6b-goog Message-ID: <20240123145258.1462979-75-ardb+git@google.com> Subject: [PATCH v7 23/50] arm64: mm: Make kaslr_requires_kpti() a static inline From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240123_065517_486838_818BA30C X-CRM114-Status: GOOD ( 23.79 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel In preparation for moving the first assignment of arm64_use_ng_mappings to an earlier stage in the boot, ensure that kaslr_requires_kpti() is accessible without relying on the core kernel's view on whether or not KASLR is enabled. So make it a static inline, and move the kaslr_enabled() check out of it and into the callers, one of which will disappear in a subsequent patch. Once/when support for the obsolete ThunderX 1 platform is dropped, this check reduces to a E0PD feature check on the local CPU. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/mmu.h | 38 +++++++++++++++++- arch/arm64/kernel/cpufeature.c | 42 +------------------- arch/arm64/kernel/setup.c | 2 +- 3 files changed, 39 insertions(+), 43 deletions(-) diff --git a/arch/arm64/include/asm/mmu.h b/arch/arm64/include/asm/mmu.h index 2fcf51231d6e..d0b8b4b413b6 100644 --- a/arch/arm64/include/asm/mmu.h +++ b/arch/arm64/include/asm/mmu.h @@ -71,7 +71,43 @@ extern void create_pgd_mapping(struct mm_struct *mm, phys_addr_t phys, pgprot_t prot, bool page_mappings_only); extern void *fixmap_remap_fdt(phys_addr_t dt_phys, int *size, pgprot_t prot); extern void mark_linear_text_alias_ro(void); -extern bool kaslr_requires_kpti(void); + +/* + * This check is triggered during the early boot before the cpufeature + * is initialised. Checking the status on the local CPU allows the boot + * CPU to detect the need for non-global mappings and thus avoiding a + * pagetable re-write after all the CPUs are booted. This check will be + * anyway run on individual CPUs, allowing us to get the consistent + * state once the SMP CPUs are up and thus make the switch to non-global + * mappings if required. + */ +static inline bool kaslr_requires_kpti(void) +{ + /* + * E0PD does a similar job to KPTI so can be used instead + * where available. + */ + if (IS_ENABLED(CONFIG_ARM64_E0PD)) { + u64 mmfr2 = read_sysreg_s(SYS_ID_AA64MMFR2_EL1); + if (cpuid_feature_extract_unsigned_field(mmfr2, + ID_AA64MMFR2_EL1_E0PD_SHIFT)) + return false; + } + + /* + * Systems affected by Cavium erratum 24756 are incompatible + * with KPTI. + */ + if (IS_ENABLED(CONFIG_CAVIUM_ERRATUM_27456)) { + extern const struct midr_range cavium_erratum_27456_cpus[]; + + if (is_midr_in_range_list(read_cpuid_id(), + cavium_erratum_27456_cpus)) + return false; + } + + return true; +} #define INIT_MM_CONTEXT(name) \ .pgd = init_pg_dir, diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c index d0ffb872a31a..7064cf13f226 100644 --- a/arch/arm64/kernel/cpufeature.c +++ b/arch/arm64/kernel/cpufeature.c @@ -1620,46 +1620,6 @@ has_useable_cnp(const struct arm64_cpu_capabilities *entry, int scope) return has_cpuid_feature(entry, scope); } -/* - * This check is triggered during the early boot before the cpufeature - * is initialised. Checking the status on the local CPU allows the boot - * CPU to detect the need for non-global mappings and thus avoiding a - * pagetable re-write after all the CPUs are booted. This check will be - * anyway run on individual CPUs, allowing us to get the consistent - * state once the SMP CPUs are up and thus make the switch to non-global - * mappings if required. - */ -bool kaslr_requires_kpti(void) -{ - if (!IS_ENABLED(CONFIG_RANDOMIZE_BASE)) - return false; - - /* - * E0PD does a similar job to KPTI so can be used instead - * where available. - */ - if (IS_ENABLED(CONFIG_ARM64_E0PD)) { - u64 mmfr2 = read_sysreg_s(SYS_ID_AA64MMFR2_EL1); - if (cpuid_feature_extract_unsigned_field(mmfr2, - ID_AA64MMFR2_EL1_E0PD_SHIFT)) - return false; - } - - /* - * Systems affected by Cavium erratum 24756 are incompatible - * with KPTI. - */ - if (IS_ENABLED(CONFIG_CAVIUM_ERRATUM_27456)) { - extern const struct midr_range cavium_erratum_27456_cpus[]; - - if (is_midr_in_range_list(read_cpuid_id(), - cavium_erratum_27456_cpus)) - return false; - } - - return kaslr_enabled(); -} - static bool __meltdown_safe = true; static int __kpti_forced; /* 0: not forced, >0: forced on, <0: forced off */ @@ -1712,7 +1672,7 @@ static bool unmap_kernel_at_el0(const struct arm64_cpu_capabilities *entry, } /* Useful for KASLR robustness */ - if (kaslr_requires_kpti()) { + if (kaslr_enabled() && kaslr_requires_kpti()) { if (!__kpti_forced) { str = "KASLR"; __kpti_forced = 1; diff --git a/arch/arm64/kernel/setup.c b/arch/arm64/kernel/setup.c index 97d2143669cf..0ef45d1927b3 100644 --- a/arch/arm64/kernel/setup.c +++ b/arch/arm64/kernel/setup.c @@ -288,7 +288,7 @@ void __init __no_sanitize_address setup_arch(char **cmdline_p) * mappings from the start, avoiding the cost of rewriting * everything later. */ - arm64_use_ng_mappings = kaslr_requires_kpti(); + arm64_use_ng_mappings = kaslr_enabled() && kaslr_requires_kpti(); early_fixmap_init(); early_ioremap_init(); From patchwork Tue Jan 23 14:53:23 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13527545 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 2B408C47DDB for ; Tue, 23 Jan 2024 14:57:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=brBxvxQbup7tqvTIs542EKJ43WoLPH4aTxbtlRC1nVg=; b=B2eF97161Yv8uqkJr3f+7pUZBW N2uU/btS2gZcVufY9E+onpdaFm/JiEesMyZ9Z37baq7c+NDIV16k4/zkVp5gP1RZ2ErsLojG0OUiv 6rmqRIhLJy2TSd0uESbeNB/Sj/DtkwamkfpHHlIJhE/SvXtR0z9f+vk6dPjUErwkeh4Dqvc/AZw4F ueP9pQtOstiEq9IJMqmObAoXIeDNaORoZ9tKAVZ5Q+0c9wznhCAY5COjLHr6mfNcf1irsS37G4+q/ QHA2cQE0w9UE3R+EbqxqtXRB0/OWNJEO5GJqwbdLlib0NJLVNBgJHOHzXaYrZf+NOVRrnSAhTgdPr npO4fu7Q==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rSICR-00GudH-2a; Tue, 23 Jan 2024 14:56:27 +0000 Received: from mail-wm1-x349.google.com ([2a00:1450:4864:20::349]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rSIBL-00Gu4Q-1g for linux-arm-kernel@lists.infradead.org; Tue, 23 Jan 2024 14:55:33 +0000 Received: by mail-wm1-x349.google.com with SMTP id 5b1f17b1804b1-40e4caa37f5so36938485e9.0 for ; Tue, 23 Jan 2024 06:55:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1706021717; x=1706626517; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=2aHlmoe8TgJ5wk4+0WsKDD2g3W308cUjShrTOsxYTJ0=; b=r5ahGNLhiqcifUU1ZzIv9CYzHd4RdTyyN0pkntkthHw/z8lz/aGrtvqo/2hSdRBnxA uJXGe5187jWtdxEbEDuzwXAWw1Flr6EQ4NLNTT87waIVNgw0NeLulXaFp2aX4QCd0m5m I81YbyPZaVkG9MTZiRmVh2MJ58bSTFvfZLuVL84rKlusgx8DMor41Pv65uToXNTUlkBx bmj0Sp8fyM+t0Ye+foGeNFGr3Sp9J7rf+JsPuKLNYwkqsIJFqt5Sxv3vx4UyA3+W6iCy V4A9IEUA7aMxIKMhFj0jVHExGYF2FMClpnMi6s2Wy8f7uivDzjyzTeA/UUqRXn2giY70 14rQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706021717; x=1706626517; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=2aHlmoe8TgJ5wk4+0WsKDD2g3W308cUjShrTOsxYTJ0=; b=BQr/pSUV9+oAgK+uhlsn7CXXBU66TTZ3wWnTMsTBkhPTaSbUR7QrtCobDUpCCyWMNC b8QDasseiA8cFLa7VH6K0iRxDIjxpOHMUn9kRj52QvfVE+UwvNOkzloD/xGWipl/YjXQ oIQgRjNhyG/lfMqRjRPUVvq6XHifPzNTySbx7Mub6LZLeNwvOhXzTEznehWXT3FJqgbB zE7pi4tm29pOMZpfKYEjzspqfoPwJr1LdFswllkkUe5xjYhdI733eondc6RVFxZZEK77 gYt9KCZNSqyLlGRSbQOBIkgI3L0qsMeTwIHlnSrioKpgTUCU7JC2aFa/ox1sNwlzlB1P drHA== X-Gm-Message-State: AOJu0YwYbgx92tS2FuO3AcW4rmj4bIwed54grYvCqWLKu+HWVQJ3NxqR N2ViYCfWnv/biosopuqTdOoJOl8mL7BcLER2o1mC0XK7+EoeD+8DhOdby9eGM992ATCt9XijKoE mwRtd5Nz3evqIprHrXe/maiHZuQPKoIz710cj2B71US1dqDENoXDHlhWzHqRg95dxbQrVOu9y33 tXfZ2hKuA/UkNpxLBiB0HmCxrOkWIWplGhXBWVR1Z8 X-Google-Smtp-Source: AGHT+IHKM7j5lRYVmQeIN7Y2QSi1A7eDrcamb522EtvN1TpsmT5kNokoF+G/nLNgaN8zZC2US6G6gYoo X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:600c:1e03:b0:40e:b318:bd0f with SMTP id ay3-20020a05600c1e0300b0040eb318bd0fmr17113wmb.2.1706021717637; Tue, 23 Jan 2024 06:55:17 -0800 (PST) Date: Tue, 23 Jan 2024 15:53:23 +0100 In-Reply-To: <20240123145258.1462979-52-ardb+git@google.com> Mime-Version: 1.0 References: <20240123145258.1462979-52-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=3644; i=ardb@kernel.org; h=from:subject; bh=wahRQ463F2BnEgYAw33JvSBo/x55U0C4nIAL5DwX90U=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIXX9paerg/LdLoaeOHKxYdbM+X3yxyZYtzBOeSycp8V4/ 5asb2JfRykLgxgHg6yYIovA7L/vdp6eKFXrPEsWZg4rE8gQBi5OAZgIkxYjw/9Zmj+1kuVLd0/z DMwI2Zq/6fzynJwF+25N3d+8Vm9XrhHDf8c0dV6hBVaSky7ETYqN8H679pef64Gta7ap/ObuWrT pGh8A X-Mailer: git-send-email 2.43.0.429.g432eaa2c6b-goog Message-ID: <20240123145258.1462979-76-ardb+git@google.com> Subject: [PATCH v7 24/50] arm64: mmu: Make __cpu_replace_ttbr1() out of line From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240123_065519_596230_8E4A371E X-CRM114-Status: GOOD ( 16.41 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel __cpu_replace_ttbr1() is a static inline, and so it gets instantiated wherever it is used. This is not really necessary, as it is never called on a hot path. It also has the unfortunate side effect that the symbol idmap_cpu_replace_ttbr1 may never be referenced from kCFI enabled C code, and this means the type id symbol may not exist either. This will result in a build error once we start referring to this symbol from asm code as well. (Note that this problem only occurs when CnP, KAsan and suspend/resume are all disabled in the Kconfig but that is a valid config, if unusual). So let's just move it out of line so all callers will share the same implementation, which will reference idmap_cpu_replace_ttbr1 unconditionally. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/mmu_context.h | 32 +------------------- arch/arm64/mm/mmu.c | 32 ++++++++++++++++++++ 2 files changed, 33 insertions(+), 31 deletions(-) diff --git a/arch/arm64/include/asm/mmu_context.h b/arch/arm64/include/asm/mmu_context.h index 9ce4200508b1..926fbbcecbe0 100644 --- a/arch/arm64/include/asm/mmu_context.h +++ b/arch/arm64/include/asm/mmu_context.h @@ -148,37 +148,7 @@ static inline void cpu_install_ttbr0(phys_addr_t ttbr0, unsigned long t0sz) isb(); } -/* - * Atomically replaces the active TTBR1_EL1 PGD with a new VA-compatible PGD, - * avoiding the possibility of conflicting TLB entries being allocated. - */ -static inline void __cpu_replace_ttbr1(pgd_t *pgdp, pgd_t *idmap, bool cnp) -{ - typedef void (ttbr_replace_func)(phys_addr_t); - extern ttbr_replace_func idmap_cpu_replace_ttbr1; - ttbr_replace_func *replace_phys; - unsigned long daif; - - /* phys_to_ttbr() zeros lower 2 bits of ttbr with 52-bit PA */ - phys_addr_t ttbr1 = phys_to_ttbr(virt_to_phys(pgdp)); - - if (cnp) - ttbr1 |= TTBR_CNP_BIT; - - replace_phys = (void *)__pa_symbol(idmap_cpu_replace_ttbr1); - - __cpu_install_idmap(idmap); - - /* - * We really don't want to take *any* exceptions while TTBR1 is - * in the process of being replaced so mask everything. - */ - daif = local_daif_save(); - replace_phys(ttbr1); - local_daif_restore(daif); - - cpu_uninstall_idmap(); -} +void __cpu_replace_ttbr1(pgd_t *pgdp, pgd_t *idmap, bool cnp); static inline void cpu_enable_swapper_cnp(void) { diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index 1ac7467d34c9..f9332eea318f 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -1486,3 +1486,35 @@ void ptep_modify_prot_commit(struct vm_area_struct *vma, unsigned long addr, pte { set_pte_at(vma->vm_mm, addr, ptep, pte); } + +/* + * Atomically replaces the active TTBR1_EL1 PGD with a new VA-compatible PGD, + * avoiding the possibility of conflicting TLB entries being allocated. + */ +void __cpu_replace_ttbr1(pgd_t *pgdp, pgd_t *idmap, bool cnp) +{ + typedef void (ttbr_replace_func)(phys_addr_t); + extern ttbr_replace_func idmap_cpu_replace_ttbr1; + ttbr_replace_func *replace_phys; + unsigned long daif; + + /* phys_to_ttbr() zeros lower 2 bits of ttbr with 52-bit PA */ + phys_addr_t ttbr1 = phys_to_ttbr(virt_to_phys(pgdp)); + + if (cnp) + ttbr1 |= TTBR_CNP_BIT; + + replace_phys = (void *)__pa_symbol(idmap_cpu_replace_ttbr1); + + __cpu_install_idmap(idmap); + + /* + * We really don't want to take *any* exceptions while TTBR1 is + * in the process of being replaced so mask everything. + */ + daif = local_daif_save(); + replace_phys(ttbr1); + local_daif_restore(daif); + + cpu_uninstall_idmap(); +} From patchwork Tue Jan 23 14:53:24 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13527551 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 0DBD6C47DDF for ; Tue, 23 Jan 2024 14:57:11 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=PzOItEsonoLgKiMz8q1wROpvSXGvdiOGsTi3aht5s0I=; b=Gejn1lbSHUsN3q9fFsJxUhAVTw XKm5U9fIh30HHuqXUnW6ORJ4wNZ1zN4W9w9gCnxgPlakcAfsO+hKScsizSmyAxLsTExj3JIlB2RO0 TPwJpXJyms3TFX56tvXjVq6xWtfYpr8syFsEB/U8oo8FjSnsYZR07oyiETRcFvxa9mFLufLxngubn 8Bq+s9jqm2x+6d8b3hYVzacc365P+Mu+Az4izdqyelPQSkiUbm9eDquo8buFRa1q7cvZIRMdPuzOU YODj3j39/TVdiUUc9PcseIskzxSU1J0GKmgA6LJYDX7YNxYm7nZyXDJYMhZ9l0FKt1xVfERFjlL4b j3jez1Qw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rSICY-00GuhK-0M; Tue, 23 Jan 2024 14:56:34 +0000 Received: from mail-yw1-x1149.google.com ([2607:f8b0:4864:20::1149]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rSIBO-00Gu5n-0B for linux-arm-kernel@lists.infradead.org; Tue, 23 Jan 2024 14:55:37 +0000 Received: by mail-yw1-x1149.google.com with SMTP id 00721157ae682-5ffee6fc6f3so38102547b3.3 for ; Tue, 23 Jan 2024 06:55:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1706021720; x=1706626520; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=IDFGpV18CsrPD2RCPYOrXLH37zOmS9aKNJDDlsPo+7Y=; b=SUtc0y6BaRJMW2FnBcHVsEe5SuiYHlTBd5UTW86FXNu27OIjO3QZ7HEVuNJ0nMy4e+ ci2ShT6tN6/Ip6Cyep5NnEQJTwomV8UR2KItKHV/2CZWlK35d/rJzghGlczkd8PJ0G4C 9ZRvyft3CA4t9T2y7l+rEGQn+c/+yvfvrONrgLthq1MHYMyY484WH5JajfUC9rPsME/x HQQdsjgcA8tF3sXT8Fi2rDL3VBc38CC320RVutU2mWfmwvSVhyywCy9VCkLrr0eacP6l GIrBKyE0u/E1Q5buh5qxGa/kUf5apZuc8VbDSWxogRbrqnyH5x6UCfC5vWrcWOG2osm5 knSw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706021720; x=1706626520; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=IDFGpV18CsrPD2RCPYOrXLH37zOmS9aKNJDDlsPo+7Y=; b=guS1+xLYk3NBHbqAl9Yr1NKHP/bLG15KJAR9gmJszuKm7KHBMUTaSrCoai9RlQa3fs B6uGYjHo25Ust7gvu3zfsKeBz84oPVelL1xX8VqD0ETfDMpoBNPjhD6U1WwokGT3bpcM EILLOBeN8zo4mHmHch8b3/xRHaJGISUStCnyU7A8Xm5vB2mfW1ZAu2US3fBOWNGhHOgq RCLa6OxyBTH5LbTK4rE4qLV0Saa+Nj5zq/qpKX0MYlHSO9bFtPDWgA5ZnEze532P49tW a2MVM5BdcCoEVqs5zima+ATBRV5ifrZzKvgEqEiFRqXoiJhMTXY9l3tGewVNKQwmGE6f zLUw== X-Gm-Message-State: AOJu0Yw/vx7oQlp0MsLEqzZ8JS95iMBqMmAzovtBtRn/5ypGXBgmPNeg iRi5m3WqcRwxef+4tiNHp3qBZXgYS72y659GbQ7fb2NcEh4t+lE5luH1X6TZRdeP1mJkljEMvjr qckUwHGZXFBHegTasgQMhkRLBaTt2Vkyw7aVvCjDGQt99JzospQ3BRJuJm9P3dQHUqirsTMmdk9 Sx+MqwEVdjBQKvz7cXNE59dhSo+yz6T+lrgs/A1p5C X-Google-Smtp-Source: AGHT+IFK79e0gJ+PUD7Wr/oBClLu7VbrdmacVXLTIyb1SaaZvjv3SA6XGHfoOVjXcJc4FaMOwQ22clpg X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a25:d850:0:b0:dc2:4ab7:3d89 with SMTP id p77-20020a25d850000000b00dc24ab73d89mr2692679ybg.1.1706021720299; Tue, 23 Jan 2024 06:55:20 -0800 (PST) Date: Tue, 23 Jan 2024 15:53:24 +0100 In-Reply-To: <20240123145258.1462979-52-ardb+git@google.com> Mime-Version: 1.0 References: <20240123145258.1462979-52-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=23592; i=ardb@kernel.org; h=from:subject; bh=7DVr4/A6kd7VV1+P6o84u0Wu25YY3wgaJQGUnrQMbKA=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIXX9pWeBwleKTeS8SjRX1fhsPbI4sWe1r3/XlMennBzNL 34Ul5foKGVhEONgkBVTZBGY/ffdztMTpWqdZ8nCzGFlAhnCwMUpABPhncfwP/C5R/V50aecZ/d6 hjLpZKqtv/x/jdaZzNjTqUZJH6qlJBgZVvA62nbIRfEbrJ+q2HPtz6EzD1eKNq5e/fCXX4xJ080 cRgA= X-Mailer: git-send-email 2.43.0.429.g432eaa2c6b-goog Message-ID: <20240123145258.1462979-77-ardb+git@google.com> Subject: [PATCH v7 25/50] arm64: head: Move early kernel mapping routines into C code From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240123_065522_141334_C934970F X-CRM114-Status: GOOD ( 25.30 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel The asm version of the kernel mapping code works fine for creating a coarse grained identity map, but for mapping the kernel down to its exact boundaries with the right attributes, it is not suitable. This is why we create a preliminary RWX kernel mapping first, and then rebuild it from scratch later on. So let's reimplement this in C, in a way that will make it unnecessary to create the kernel page tables yet another time in paging_init(). Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/archrandom.h | 2 - arch/arm64/include/asm/scs.h | 32 +--- arch/arm64/kernel/head.S | 52 +------ arch/arm64/kernel/image-vars.h | 19 +++ arch/arm64/kernel/pi/Makefile | 1 + arch/arm64/kernel/pi/idreg-override.c | 22 ++- arch/arm64/kernel/pi/kaslr_early.c | 12 +- arch/arm64/kernel/pi/map_kernel.c | 164 ++++++++++++++++++++ arch/arm64/kernel/pi/map_range.c | 88 +++++++++++ arch/arm64/kernel/pi/patch-scs.c | 16 +- arch/arm64/kernel/pi/pi.h | 14 ++ arch/arm64/kernel/pi/relocate.c | 2 + arch/arm64/kernel/setup.c | 7 - arch/arm64/kernel/vmlinux.lds.S | 4 +- arch/arm64/mm/proc.S | 1 + 15 files changed, 315 insertions(+), 121 deletions(-) diff --git a/arch/arm64/include/asm/archrandom.h b/arch/arm64/include/asm/archrandom.h index ecdb3cfcd0f8..8babfbe31f95 100644 --- a/arch/arm64/include/asm/archrandom.h +++ b/arch/arm64/include/asm/archrandom.h @@ -129,6 +129,4 @@ static inline bool __init __early_cpu_has_rndr(void) return (ftr >> ID_AA64ISAR0_EL1_RNDR_SHIFT) & 0xf; } -u64 kaslr_early_init(void *fdt); - #endif /* _ASM_ARCHRANDOM_H */ diff --git a/arch/arm64/include/asm/scs.h b/arch/arm64/include/asm/scs.h index eca2ba5a6276..2e010ea76be2 100644 --- a/arch/arm64/include/asm/scs.h +++ b/arch/arm64/include/asm/scs.h @@ -33,37 +33,11 @@ #include #ifdef CONFIG_UNWIND_PATCH_PAC_INTO_SCS -static inline bool should_patch_pac_into_scs(void) -{ - u64 reg; - - /* - * We only enable the shadow call stack dynamically if we are running - * on a system that does not implement PAC or BTI. PAC and SCS provide - * roughly the same level of protection, and BTI relies on the PACIASP - * instructions serving as landing pads, preventing us from patching - * those instructions into something else. - */ - reg = read_sysreg_s(SYS_ID_AA64ISAR1_EL1); - if (SYS_FIELD_GET(ID_AA64ISAR1_EL1, APA, reg) | - SYS_FIELD_GET(ID_AA64ISAR1_EL1, API, reg)) - return false; - - reg = read_sysreg_s(SYS_ID_AA64ISAR2_EL1); - if (SYS_FIELD_GET(ID_AA64ISAR2_EL1, APA3, reg)) - return false; - - if (IS_ENABLED(CONFIG_ARM64_BTI_KERNEL)) { - reg = read_sysreg_s(SYS_ID_AA64PFR1_EL1); - if (reg & (0xf << ID_AA64PFR1_EL1_BT_SHIFT)) - return false; - } - return true; -} - static inline void dynamic_scs_init(void) { - if (should_patch_pac_into_scs()) { + extern bool __pi_dynamic_scs_is_enabled; + + if (__pi_dynamic_scs_is_enabled) { pr_info("Enabling dynamic shadow call stack\n"); static_branch_enable(&dynamic_scs_enabled); } diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S index aa7766dc64d9..ffacce7b5a02 100644 --- a/arch/arm64/kernel/head.S +++ b/arch/arm64/kernel/head.S @@ -81,7 +81,6 @@ * x20 primary_entry() .. __primary_switch() CPU boot mode * x21 primary_entry() .. start_kernel() FDT pointer passed at boot in x0 * x22 create_idmap() .. start_kernel() ID map VA of the DT blob - * x23 __primary_switch() physical misalignment/KASLR offset * x25 primary_entry() .. start_kernel() supported VA size * x28 create_idmap() callee preserved temp register */ @@ -408,24 +407,6 @@ SYM_FUNC_START_LOCAL(create_idmap) 0: ret x28 SYM_FUNC_END(create_idmap) -SYM_FUNC_START_LOCAL(create_kernel_mapping) - adrp x0, init_pg_dir - mov_q x5, KIMAGE_VADDR // compile time __va(_text) -#ifdef CONFIG_RELOCATABLE - add x5, x5, x23 // add KASLR displacement -#endif - adrp x6, _end // runtime __pa(_end) - adrp x3, _text // runtime __pa(_text) - sub x6, x6, x3 // _end - _text - add x6, x6, x5 // runtime __va(_end) - mov_q x7, SWAPPER_RW_MMUFLAGS - - map_memory x0, x1, x5, x6, x7, x3, (VA_BITS - PGDIR_SHIFT), x10, x11, x12, x13, x14 - - dsb ishst // sync with page table walker - ret -SYM_FUNC_END(create_kernel_mapping) - /* * Initialize CPU registers with task-specific and cpu-specific context. * @@ -752,44 +733,13 @@ SYM_FUNC_START_LOCAL(__primary_switch) adrp x2, init_idmap_pg_dir bl __enable_mmu - // Clear BSS - adrp x0, __bss_start - mov x1, xzr - adrp x2, init_pg_end - sub x2, x2, x0 - bl __pi_memset - dsb ishst // Make zero page visible to PTW - adrp x1, early_init_stack mov sp, x1 mov x29, xzr mov x0, x20 // pass the full boot status mov x1, x22 // pass the low FDT mapping - bl __pi_init_feature_override // Parse cpu feature overrides - -#ifdef CONFIG_RELOCATABLE - adrp x23, KERNEL_START - and x23, x23, MIN_KIMG_ALIGN - 1 -#ifdef CONFIG_RANDOMIZE_BASE - mov x0, x22 - bl __pi_kaslr_early_init - bic x0, x0, #SZ_2M - 1 - orr x23, x23, x0 // record kernel offset -#endif -#endif - bl create_kernel_mapping + bl __pi_early_map_kernel // Map and relocate the kernel - adrp x1, init_pg_dir - load_ttbr1 x1, x1, x2 -#ifdef CONFIG_RELOCATABLE - mov x0, x23 - bl __pi_relocate_kernel -#endif -#ifdef CONFIG_UNWIND_PATCH_PAC_INTO_SCS - ldr x0, =__eh_frame_start - ldr x1, =__eh_frame_end - bl __pi_scs_patch_vmlinux -#endif ldr x8, =__primary_switched adrp x0, KERNEL_START // __pa(KERNEL_START) br x8 diff --git a/arch/arm64/kernel/image-vars.h b/arch/arm64/kernel/image-vars.h index 8d96052079e8..e566b32f9c22 100644 --- a/arch/arm64/kernel/image-vars.h +++ b/arch/arm64/kernel/image-vars.h @@ -45,9 +45,28 @@ PROVIDE(__pi_id_aa64pfr1_override = id_aa64pfr1_override); PROVIDE(__pi_id_aa64smfr0_override = id_aa64smfr0_override); PROVIDE(__pi_id_aa64zfr0_override = id_aa64zfr0_override); PROVIDE(__pi_arm64_sw_feature_override = arm64_sw_feature_override); +PROVIDE(__pi_arm64_use_ng_mappings = arm64_use_ng_mappings); +#ifdef CONFIG_CAVIUM_ERRATUM_27456 +PROVIDE(__pi_cavium_erratum_27456_cpus = cavium_erratum_27456_cpus); +#endif PROVIDE(__pi__ctype = _ctype); PROVIDE(__pi_memstart_offset_seed = memstart_offset_seed); +PROVIDE(__pi_init_pg_dir = init_pg_dir); +PROVIDE(__pi_init_pg_end = init_pg_end); + +PROVIDE(__pi__text = _text); +PROVIDE(__pi__stext = _stext); +PROVIDE(__pi__etext = _etext); +PROVIDE(__pi___start_rodata = __start_rodata); +PROVIDE(__pi___inittext_begin = __inittext_begin); +PROVIDE(__pi___inittext_end = __inittext_end); +PROVIDE(__pi___initdata_begin = __initdata_begin); +PROVIDE(__pi___initdata_end = __initdata_end); +PROVIDE(__pi__data = _data); +PROVIDE(__pi___bss_start = __bss_start); +PROVIDE(__pi__end = _end); + #ifdef CONFIG_KVM /* diff --git a/arch/arm64/kernel/pi/Makefile b/arch/arm64/kernel/pi/Makefile index a8b302245f15..8c2f80a46b93 100644 --- a/arch/arm64/kernel/pi/Makefile +++ b/arch/arm64/kernel/pi/Makefile @@ -39,6 +39,7 @@ $(obj)/lib-%.o: $(srctree)/lib/%.c FORCE $(call if_changed_rule,cc_o_c) obj-y := idreg-override.pi.o \ + map_kernel.pi.o map_range.pi.o \ lib-fdt.pi.o lib-fdt_ro.pi.o obj-$(CONFIG_RELOCATABLE) += relocate.pi.o obj-$(CONFIG_RANDOMIZE_BASE) += kaslr_early.pi.o diff --git a/arch/arm64/kernel/pi/idreg-override.c b/arch/arm64/kernel/pi/idreg-override.c index e4bcabcc6860..1884bd936c0d 100644 --- a/arch/arm64/kernel/pi/idreg-override.c +++ b/arch/arm64/kernel/pi/idreg-override.c @@ -308,37 +308,35 @@ static __init void __parse_cmdline(const char *cmdline, bool parse_aliases) } while (1); } -static __init const u8 *get_bootargs_cmdline(const void *fdt) +static __init const u8 *get_bootargs_cmdline(const void *fdt, int node) { + static char const bootargs[] __initconst = "bootargs"; const u8 *prop; - int node; - node = fdt_path_offset(fdt, "/chosen"); if (node < 0) return NULL; - prop = fdt_getprop(fdt, node, "bootargs", NULL); + prop = fdt_getprop(fdt, node, bootargs, NULL); if (!prop) return NULL; return strlen(prop) ? prop : NULL; } -static __init void parse_cmdline(const void *fdt) +static __init void parse_cmdline(const void *fdt, int chosen) { - const u8 *prop = get_bootargs_cmdline(fdt); + static char const cmdline[] __initconst = CONFIG_CMDLINE; + const u8 *prop = get_bootargs_cmdline(fdt, chosen); if (IS_ENABLED(CONFIG_CMDLINE_FORCE) || !prop) - __parse_cmdline(CONFIG_CMDLINE, true); + __parse_cmdline(cmdline, true); if (!IS_ENABLED(CONFIG_CMDLINE_FORCE) && prop) __parse_cmdline(prop, true); } -/* Keep checkers quiet */ -void init_feature_override(u64 boot_status, const void *fdt); - -asmlinkage void __init init_feature_override(u64 boot_status, const void *fdt) +void __init init_feature_override(u64 boot_status, const void *fdt, + int chosen) { struct arm64_ftr_override *override; const struct ftr_set_desc *reg; @@ -354,7 +352,7 @@ asmlinkage void __init init_feature_override(u64 boot_status, const void *fdt) __boot_status = boot_status; - parse_cmdline(fdt); + parse_cmdline(fdt, chosen); for (i = 0; i < ARRAY_SIZE(regs); i++) { reg = prel64_pointer(regs[i].reg); diff --git a/arch/arm64/kernel/pi/kaslr_early.c b/arch/arm64/kernel/pi/kaslr_early.c index eeecee7ffd6f..0257b43819db 100644 --- a/arch/arm64/kernel/pi/kaslr_early.c +++ b/arch/arm64/kernel/pi/kaslr_early.c @@ -16,17 +16,17 @@ #include #include +#include "pi.h" + extern u16 memstart_offset_seed; -static u64 __init get_kaslr_seed(void *fdt) +static u64 __init get_kaslr_seed(void *fdt, int node) { - static char const chosen_str[] __initconst = "chosen"; static char const seed_str[] __initconst = "kaslr-seed"; - int node, len; fdt64_t *prop; u64 ret; + int len; - node = fdt_path_offset(fdt, chosen_str); if (node < 0) return 0; @@ -39,14 +39,14 @@ static u64 __init get_kaslr_seed(void *fdt) return ret; } -asmlinkage u64 __init kaslr_early_init(void *fdt) +u64 __init kaslr_early_init(void *fdt, int chosen) { u64 seed, range; if (kaslr_disabled_cmdline()) return 0; - seed = get_kaslr_seed(fdt); + seed = get_kaslr_seed(fdt, chosen); if (!seed) { if (!__early_cpu_has_rndr() || !__arm64_rndr((unsigned long *)&seed)) diff --git a/arch/arm64/kernel/pi/map_kernel.c b/arch/arm64/kernel/pi/map_kernel.c new file mode 100644 index 000000000000..f206373b28b0 --- /dev/null +++ b/arch/arm64/kernel/pi/map_kernel.c @@ -0,0 +1,164 @@ +// SPDX-License-Identifier: GPL-2.0-only +// Copyright 2023 Google LLC +// Author: Ard Biesheuvel + +#include +#include +#include +#include +#include +#include + +#include +#include +#include +#include + +#include "pi.h" + +extern const u8 __eh_frame_start[], __eh_frame_end[]; + +extern void idmap_cpu_replace_ttbr1(void *pgdir); + +static void __init map_segment(pgd_t *pg_dir, u64 *pgd, u64 va_offset, + void *start, void *end, pgprot_t prot, + bool may_use_cont, int root_level) +{ + map_range(pgd, ((u64)start + va_offset) & ~PAGE_OFFSET, + ((u64)end + va_offset) & ~PAGE_OFFSET, (u64)start, + prot, root_level, (pte_t *)pg_dir, may_use_cont, 0); +} + +static void __init unmap_segment(pgd_t *pg_dir, u64 va_offset, void *start, + void *end, int root_level) +{ + map_segment(pg_dir, NULL, va_offset, start, end, __pgprot(0), + false, root_level); +} + +static void __init map_kernel(u64 kaslr_offset, u64 va_offset, int root_level) +{ + bool enable_scs = IS_ENABLED(CONFIG_UNWIND_PATCH_PAC_INTO_SCS); + bool twopass = IS_ENABLED(CONFIG_RELOCATABLE); + u64 pgdp = (u64)init_pg_dir + PAGE_SIZE; + pgprot_t text_prot = PAGE_KERNEL_ROX; + pgprot_t data_prot = PAGE_KERNEL; + pgprot_t prot; + + /* + * External debuggers may need to write directly to the text mapping to + * install SW breakpoints. Allow this (only) when explicitly requested + * with rodata=off. + */ + if (arm64_test_sw_feature_override(ARM64_SW_FEATURE_OVERRIDE_RODATA_OFF)) + text_prot = PAGE_KERNEL_EXEC; + + /* + * We only enable the shadow call stack dynamically if we are running + * on a system that does not implement PAC or BTI. PAC and SCS provide + * roughly the same level of protection, and BTI relies on the PACIASP + * instructions serving as landing pads, preventing us from patching + * those instructions into something else. + */ + if (IS_ENABLED(CONFIG_ARM64_PTR_AUTH_KERNEL) && cpu_has_pac()) + enable_scs = false; + + if (IS_ENABLED(CONFIG_ARM64_BTI_KERNEL) && cpu_has_bti()) { + enable_scs = false; + + /* + * If we have a CPU that supports BTI and a kernel built for + * BTI then mark the kernel executable text as guarded pages + * now so we don't have to rewrite the page tables later. + */ + text_prot = __pgprot_modify(text_prot, PTE_GP, PTE_GP); + } + + /* Map all code read-write on the first pass if needed */ + twopass |= enable_scs; + prot = twopass ? data_prot : text_prot; + + map_segment(init_pg_dir, &pgdp, va_offset, _stext, _etext, prot, + !twopass, root_level); + map_segment(init_pg_dir, &pgdp, va_offset, __start_rodata, + __inittext_begin, data_prot, false, root_level); + map_segment(init_pg_dir, &pgdp, va_offset, __inittext_begin, + __inittext_end, prot, false, root_level); + map_segment(init_pg_dir, &pgdp, va_offset, __initdata_begin, + __initdata_end, data_prot, false, root_level); + map_segment(init_pg_dir, &pgdp, va_offset, _data, _end, data_prot, + true, root_level); + dsb(ishst); + + idmap_cpu_replace_ttbr1(init_pg_dir); + + if (twopass) { + if (IS_ENABLED(CONFIG_RELOCATABLE)) + relocate_kernel(kaslr_offset); + + if (enable_scs) { + scs_patch(__eh_frame_start + va_offset, + __eh_frame_end - __eh_frame_start); + asm("ic ialluis"); + + dynamic_scs_is_enabled = true; + } + + /* + * Unmap the text region before remapping it, to avoid + * potential TLB conflicts when creating the contiguous + * descriptors. + */ + unmap_segment(init_pg_dir, va_offset, _stext, _etext, + root_level); + dsb(ishst); + isb(); + __tlbi(vmalle1); + isb(); + + /* + * Remap these segments with different permissions + * No new page table allocations should be needed + */ + map_segment(init_pg_dir, NULL, va_offset, _stext, _etext, + text_prot, true, root_level); + map_segment(init_pg_dir, NULL, va_offset, __inittext_begin, + __inittext_end, text_prot, false, root_level); + dsb(ishst); + } +} + +asmlinkage void __init early_map_kernel(u64 boot_status, void *fdt) +{ + static char const chosen_str[] __initconst = "/chosen"; + u64 va_base, pa_base = (u64)&_text; + u64 kaslr_offset = pa_base % MIN_KIMG_ALIGN; + int root_level = 4 - CONFIG_PGTABLE_LEVELS; + int chosen; + + /* Clear BSS and the initial page tables */ + memset(__bss_start, 0, (u64)init_pg_end - (u64)__bss_start); + + /* Parse the command line for CPU feature overrides */ + chosen = fdt_path_offset(fdt, chosen_str); + init_feature_override(boot_status, fdt, chosen); + + /* + * The virtual KASLR displacement modulo 2MiB is decided by the + * physical placement of the image, as otherwise, we might not be able + * to create the early kernel mapping using 2 MiB block descriptors. So + * take the low bits of the KASLR offset from the physical address, and + * fill in the high bits from the seed. + */ + if (IS_ENABLED(CONFIG_RANDOMIZE_BASE)) { + u64 kaslr_seed = kaslr_early_init(fdt, chosen); + + if (kaslr_seed && kaslr_requires_kpti()) + arm64_use_ng_mappings = true; + + kaslr_offset |= kaslr_seed & ~(MIN_KIMG_ALIGN - 1); + } + + va_base = KIMAGE_VADDR + kaslr_offset; + map_kernel(kaslr_offset, va_base - pa_base, root_level); +} diff --git a/arch/arm64/kernel/pi/map_range.c b/arch/arm64/kernel/pi/map_range.c new file mode 100644 index 000000000000..c31feda18f47 --- /dev/null +++ b/arch/arm64/kernel/pi/map_range.c @@ -0,0 +1,88 @@ +// SPDX-License-Identifier: GPL-2.0-only +// Copyright 2023 Google LLC +// Author: Ard Biesheuvel + +#include +#include + +#include +#include +#include + +#include "pi.h" + +/** + * map_range - Map a contiguous range of physical pages into virtual memory + * + * @pte: Address of physical pointer to array of pages to + * allocate page tables from + * @start: Virtual address of the start of the range + * @end: Virtual address of the end of the range (exclusive) + * @pa: Physical address of the start of the range + * @prot: Access permissions of the range + * @level: Translation level for the mapping + * @tbl: The level @level page table to create the mappings in + * @may_use_cont: Whether the use of the contiguous attribute is allowed + * @va_offset: Offset between a physical page and its current mapping + * in the VA space + */ +void __init map_range(u64 *pte, u64 start, u64 end, u64 pa, pgprot_t prot, + int level, pte_t *tbl, bool may_use_cont, u64 va_offset) +{ + u64 cmask = (level == 3) ? CONT_PTE_SIZE - 1 : U64_MAX; + u64 protval = pgprot_val(prot) & ~PTE_TYPE_MASK; + int lshift = (3 - level) * (PAGE_SHIFT - 3); + u64 lmask = (PAGE_SIZE << lshift) - 1; + + start &= PAGE_MASK; + pa &= PAGE_MASK; + + /* Advance tbl to the entry that covers start */ + tbl += (start >> (lshift + PAGE_SHIFT)) % PTRS_PER_PTE; + + /* + * Set the right block/page bits for this level unless we are + * clearing the mapping + */ + if (protval) + protval |= (level < 3) ? PMD_TYPE_SECT : PTE_TYPE_PAGE; + + while (start < end) { + u64 next = min((start | lmask) + 1, PAGE_ALIGN(end)); + + if (level < 3 && (start | next | pa) & lmask) { + /* + * This chunk needs a finer grained mapping. Create a + * table mapping if necessary and recurse. + */ + if (pte_none(*tbl)) { + *tbl = __pte(__phys_to_pte_val(*pte) | + PMD_TYPE_TABLE | PMD_TABLE_UXN); + *pte += PTRS_PER_PTE * sizeof(pte_t); + } + map_range(pte, start, next, pa, prot, level + 1, + (pte_t *)(__pte_to_phys(*tbl) + va_offset), + may_use_cont, va_offset); + } else { + /* + * Start a contiguous range if start and pa are + * suitably aligned + */ + if (((start | pa) & cmask) == 0 && may_use_cont) + protval |= PTE_CONT; + + /* + * Clear the contiguous attribute if the remaining + * range does not cover a contiguous block + */ + if ((end & ~cmask) <= start) + protval &= ~PTE_CONT; + + /* Put down a block or page mapping */ + *tbl = __pte(__phys_to_pte_val(pa) | protval); + } + pa += next - start; + start = next; + tbl++; + } +} diff --git a/arch/arm64/kernel/pi/patch-scs.c b/arch/arm64/kernel/pi/patch-scs.c index c65ef40d1e6b..49d8b40e61bc 100644 --- a/arch/arm64/kernel/pi/patch-scs.c +++ b/arch/arm64/kernel/pi/patch-scs.c @@ -11,6 +11,10 @@ #include +#include "pi.h" + +bool dynamic_scs_is_enabled; + // // This minimal DWARF CFI parser is partially based on the code in // arch/arc/kernel/unwind.c, and on the document below: @@ -46,8 +50,6 @@ #define DW_CFA_GNU_negative_offset_extended 0x2f #define DW_CFA_hi_user 0x3f -extern const u8 __eh_frame_start[], __eh_frame_end[]; - enum { PACIASP = 0xd503233f, AUTIASP = 0xd50323bf, @@ -250,13 +252,3 @@ int scs_patch(const u8 eh_frame[], int size) } return 0; } - -asmlinkage void __init scs_patch_vmlinux(const u8 start[], const u8 end[]) -{ - if (!should_patch_pac_into_scs()) - return; - - scs_patch(start, end - start); - asm("ic ialluis"); - isb(); -} diff --git a/arch/arm64/kernel/pi/pi.h b/arch/arm64/kernel/pi/pi.h index 7c2d9bbf0ff9..d307c58e9741 100644 --- a/arch/arm64/kernel/pi/pi.h +++ b/arch/arm64/kernel/pi/pi.h @@ -2,6 +2,8 @@ // Copyright 2023 Google LLC // Author: Ard Biesheuvel +#include + #define __prel64_initconst __section(".init.rodata.prel64") #define PREL64(type, name) union { type *name; prel64_t name ## _prel; } @@ -16,3 +18,15 @@ static inline void *prel64_to_pointer(const prel64_t *offset) return NULL; return (void *)offset + *offset; } + +extern bool dynamic_scs_is_enabled; + +void init_feature_override(u64 boot_status, const void *fdt, int chosen); +u64 kaslr_early_init(void *fdt, int chosen); +void relocate_kernel(u64 offset); +int scs_patch(const u8 eh_frame[], int size); + +void map_range(u64 *pgd, u64 start, u64 end, u64 pa, pgprot_t prot, + int level, pte_t *tbl, bool may_use_cont, u64 va_offset); + +asmlinkage void early_map_kernel(u64 boot_status, void *fdt); diff --git a/arch/arm64/kernel/pi/relocate.c b/arch/arm64/kernel/pi/relocate.c index 1853408ea76b..2407d2696398 100644 --- a/arch/arm64/kernel/pi/relocate.c +++ b/arch/arm64/kernel/pi/relocate.c @@ -7,6 +7,8 @@ #include #include +#include "pi.h" + extern const Elf64_Rela rela_start[], rela_end[]; extern const u64 relr_start[], relr_end[]; diff --git a/arch/arm64/kernel/setup.c b/arch/arm64/kernel/setup.c index 0ef45d1927b3..0ea45b6d0177 100644 --- a/arch/arm64/kernel/setup.c +++ b/arch/arm64/kernel/setup.c @@ -283,13 +283,6 @@ void __init __no_sanitize_address setup_arch(char **cmdline_p) kaslr_init(); - /* - * If know now we are going to need KPTI then use non-global - * mappings from the start, avoiding the cost of rewriting - * everything later. - */ - arm64_use_ng_mappings = kaslr_enabled() && kaslr_requires_kpti(); - early_fixmap_init(); early_ioremap_init(); diff --git a/arch/arm64/kernel/vmlinux.lds.S b/arch/arm64/kernel/vmlinux.lds.S index 3afb4223a5e8..755a22d4f840 100644 --- a/arch/arm64/kernel/vmlinux.lds.S +++ b/arch/arm64/kernel/vmlinux.lds.S @@ -126,9 +126,9 @@ jiffies = jiffies_64; #ifdef CONFIG_UNWIND_TABLES #define UNWIND_DATA_SECTIONS \ .eh_frame : { \ - __eh_frame_start = .; \ + __pi___eh_frame_start = .; \ *(.eh_frame) \ - __eh_frame_end = .; \ + __pi___eh_frame_end = .; \ } #else #define UNWIND_DATA_SECTIONS diff --git a/arch/arm64/mm/proc.S b/arch/arm64/mm/proc.S index f66c37a1610e..7c1bdaf25408 100644 --- a/arch/arm64/mm/proc.S +++ b/arch/arm64/mm/proc.S @@ -195,6 +195,7 @@ SYM_TYPED_FUNC_START(idmap_cpu_replace_ttbr1) ret SYM_FUNC_END(idmap_cpu_replace_ttbr1) +SYM_FUNC_ALIAS(__pi_idmap_cpu_replace_ttbr1, idmap_cpu_replace_ttbr1) .popsection #ifdef CONFIG_UNMAP_KERNEL_AT_EL0 From patchwork Tue Jan 23 14:53:25 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13527547 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id BC1B3C47DDF for ; Tue, 23 Jan 2024 14:57:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=1wl3ABu0CAipi32VOG9J0/coO5UgtQkJ/pUhp6ljhYw=; b=tO7G8l3+9Hkls/0qXXs6924MPn ETbgNnjrfLIFwf9f4IX5/p7Nb8jBMggfTsbzk1FApX+38f3z7gmjWE5X6C7BFlYX+ODkZtQgoP5I0 zC+PtkCDIDv+r0ZSA54PKy70a36uUNPsziSEULG/P8NB8hctegwZ5Jrz+6ZCW3MdxyoUfaW3akrkW MpiZITi/3f6F/X64fjLAJlJEBz8VgQgmvM887Spq4K52JvgYx6sIat8rYMQtl65fcEFR5S3E69OVx rPjrC7losAAsUTX5iFSIE3hIa7DzDPI+iO152AudZDzC6+g50+dqx0kseeXV2UnPwNe65t2mWRuGa 8MTcUUgQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rSICW-00GugG-1e; Tue, 23 Jan 2024 14:56:32 +0000 Received: from mail-yb1-xb49.google.com ([2607:f8b0:4864:20::b49]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rSIBQ-00Gu71-0k for linux-arm-kernel@lists.infradead.org; Tue, 23 Jan 2024 14:55:38 +0000 Received: by mail-yb1-xb49.google.com with SMTP id 3f1490d57ef6-dc26605c273so6756975276.0 for ; Tue, 23 Jan 2024 06:55:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1706021723; x=1706626523; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=GoGUvQmqlGpRYSlH5itbOToFzNuu1DxOGxtHebXlkbg=; b=d+ADVhhCTVYeOeLFcuncmifmq9z7gIYcVIMqFd1duEsQc1wpwEXNs1bqZSxCTI+JGY DAjM87f8s7WenBX3wvf9tQOYdTVqHw0UHhNhEGyYaBEXqjNmOP7KhJMlYyxYOK0Iix0b ne93Fc8e+0WTwFsVKSsYkv4N9j2TSn0uTTbHcOuYleiAkEER4qOaDZ76pNSvQHlgQ9ew mPp5wk4uamqZS4SDrQqOal9iRCDr+Q4MxbXnZXxcvSsnchQSJ/DKpk3665SZbPMUsapl mnsP8Hm9kuanEalwUuUBJS/bPdvFo9yhxjBVxYnUHWm3pyTboJswrwFKK76kC3t8FKVu dLzQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706021723; x=1706626523; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=GoGUvQmqlGpRYSlH5itbOToFzNuu1DxOGxtHebXlkbg=; b=YDRUBVaiBcjRhrW6EXBiv/KxxkcDAJzmAPH7GHYvycE4PDkHqBgQSebNH71gJJUdPN UjNNWkSnUxhY5byO7pJpCenYK3u+QGL2Y6HspovvLOqyzLQqs/8ZRnuKkHwc/R4Qud84 jWmkmW/keHhuxZkd2pbTM+IFoG9RV+JsiSDissf7WvRTNtJaaBaJqPZiS8lPwndZiEmh GE7vIwLclAHcBPcMNb6uO4q2MpQohs98Z3aJrWsS/t4KX/gZ/LpajSHxaPEm0+VWP00q qNUJZ0j45yXGbzlrRpk+48MncrXvlDP0+n3nICAT+Qk6Lx4pY41ae4QPBKzrhrCekmuE 4i6g== X-Gm-Message-State: AOJu0YxePaEBOfsv8HcRh2J+qRo7V52O1PgPxAwXe7u6UY/Ri3KILgGC 3rBm7CxF+Y2f6PpiEpJ5DXZYg6kB+/ilIaUU2dB2eX9P3iI0DwweHOWfpWlfDnKSHBvPM1MEC5A jnjWDUgjlLpElV5eUPOP6sEChSPBcSxbg0KpMxsai+/53DI5N4cKoqO0WQjQ5DZPt9/TFKYtONX SzpbPzcgIb6bfurhLU1sttxRcjLZnvC4ZyI/wozvg8 X-Google-Smtp-Source: AGHT+IFmE7jv0JJzNBwmlf7agrFEgn4H62gp0QGGTCnfBONVfUq4eY1RGIlctp1t4A5tpIUtq5NBQTtj X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:6902:1005:b0:dc2:5525:f6b with SMTP id w5-20020a056902100500b00dc255250f6bmr2742118ybt.7.1706021722838; Tue, 23 Jan 2024 06:55:22 -0800 (PST) Date: Tue, 23 Jan 2024 15:53:25 +0100 In-Reply-To: <20240123145258.1462979-52-ardb+git@google.com> Mime-Version: 1.0 References: <20240123145258.1462979-52-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=7010; i=ardb@kernel.org; h=from:subject; bh=cmLpm3OH5FO7lmpVfZth8QyxByRj1Y7QTqYuQci14J8=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIXX9pedfgvJnnY2W0gvSUe+/+C/kXEgES8zV9ocVldOul TJz9Qp0lLIwiHEwyIopsgjM/vtu5+mJUrXOs2Rh5rAygQxh4OIUgInM1Gf4K3GpvnTKq8zrNcaz T9Q76Lqmduy/k7JlNu/ETjuba/H/JjIyHOlJffh/ye2a9w/N+Tg39f369v3AROvjF3dMeyTCEMo pyAgA X-Mailer: git-send-email 2.43.0.429.g432eaa2c6b-goog Message-ID: <20240123145258.1462979-78-ardb+git@google.com> Subject: [PATCH v7 26/50] arm64: mm: Use 48-bit virtual addressing for the permanent ID map From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240123_065524_282177_A997475B X-CRM114-Status: GOOD ( 24.62 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel Even though we support loading kernels anywhere in 48-bit addressable physical memory, we create the ID maps based on the number of levels that we happened to configure for the kernel VA and user VA spaces. The reason for this is that the PGD/PUD/PMD based classification of translation levels, along with the associated folding when the number of levels is less than 5, does not permit creating a page table hierarchy of a set number of levels. This means that, for instance, on 39-bit VA kernels we need to configure an additional level above PGD level on the fly, and 36-bit VA kernels still only support 47-bit virtual addressing with this trick applied. Now that we have a separate helper to populate page table hierarchies that does not define the levels in terms of PUDS/PMDS/etc at all, let's reuse it to create the permanent ID map with a fixed VA size of 48 bits. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/kernel-pgtable.h | 3 ++ arch/arm64/kernel/head.S | 5 +++ arch/arm64/kvm/mmu.c | 15 +++------ arch/arm64/mm/mmu.c | 32 +++++++++++--------- arch/arm64/mm/proc.S | 9 ++---- 5 files changed, 32 insertions(+), 32 deletions(-) diff --git a/arch/arm64/include/asm/kernel-pgtable.h b/arch/arm64/include/asm/kernel-pgtable.h index 0631604995ee..742a4b2778f7 100644 --- a/arch/arm64/include/asm/kernel-pgtable.h +++ b/arch/arm64/include/asm/kernel-pgtable.h @@ -35,6 +35,9 @@ #define SWAPPER_PGTABLE_LEVELS (CONFIG_PGTABLE_LEVELS) #endif +#define IDMAP_VA_BITS 48 +#define IDMAP_LEVELS ARM64_HW_PGTABLE_LEVELS(IDMAP_VA_BITS) +#define IDMAP_ROOT_LEVEL (4 - IDMAP_LEVELS) /* * A relocatable kernel may execute from an address that differs from the one at diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S index ffacce7b5a02..a1c29d64e875 100644 --- a/arch/arm64/kernel/head.S +++ b/arch/arm64/kernel/head.S @@ -729,6 +729,11 @@ SYM_FUNC_START_LOCAL(__no_granule_support) SYM_FUNC_END(__no_granule_support) SYM_FUNC_START_LOCAL(__primary_switch) + mrs x1, tcr_el1 + mov x2, #64 - VA_BITS + tcr_set_t0sz x1, x2 + msr tcr_el1, x1 + adrp x1, reserved_pg_dir adrp x2, init_idmap_pg_dir bl __enable_mmu diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c index d14504821b79..6fa9e816df40 100644 --- a/arch/arm64/kvm/mmu.c +++ b/arch/arm64/kvm/mmu.c @@ -1874,16 +1874,9 @@ int __init kvm_mmu_init(u32 *hyp_va_bits) BUG_ON((hyp_idmap_start ^ (hyp_idmap_end - 1)) & PAGE_MASK); /* - * The ID map may be configured to use an extended virtual address - * range. This is only the case if system RAM is out of range for the - * currently configured page size and VA_BITS_MIN, in which case we will - * also need the extended virtual range for the HYP ID map, or we won't - * be able to enable the EL2 MMU. - * - * However, in some cases the ID map may be configured for fewer than - * the number of VA bits used by the regular kernel stage 1. This - * happens when VA_BITS=52 and the kernel image is placed in PA space - * below 48 bits. + * The ID map is always configured for 48 bits of translation, which + * may be fewer than the number of VA bits used by the regular kernel + * stage 1, when VA_BITS=52. * * At EL2, there is only one TTBR register, and we can't switch between * translation tables *and* update TCR_EL2.T0SZ at the same time. Bottom @@ -1894,7 +1887,7 @@ int __init kvm_mmu_init(u32 *hyp_va_bits) * 1 VA bits to assure that the hypervisor can both ID map its code page * and map any kernel memory. */ - idmap_bits = 64 - ((idmap_t0sz & TCR_T0SZ_MASK) >> TCR_T0SZ_OFFSET); + idmap_bits = IDMAP_VA_BITS; kernel_bits = vabits_actual; *hyp_va_bits = max(idmap_bits, kernel_bits); diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index f9332eea318f..a991f195592b 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -757,22 +757,21 @@ static void __init map_kernel(pgd_t *pgdp) kasan_copy_shadow(pgdp); } +void __pi_map_range(u64 *pgd, u64 start, u64 end, u64 pa, pgprot_t prot, + int level, pte_t *tbl, bool may_use_cont, u64 va_offset); + +static u8 idmap_ptes[IDMAP_LEVELS - 1][PAGE_SIZE] __aligned(PAGE_SIZE) __ro_after_init, + kpti_ptes[IDMAP_LEVELS - 1][PAGE_SIZE] __aligned(PAGE_SIZE) __ro_after_init; + static void __init create_idmap(void) { u64 start = __pa_symbol(__idmap_text_start); - u64 size = __pa_symbol(__idmap_text_end) - start; - pgd_t *pgd = idmap_pg_dir; - u64 pgd_phys; - - /* check if we need an additional level of translation */ - if (VA_BITS < 48 && idmap_t0sz < (64 - VA_BITS_MIN)) { - pgd_phys = early_pgtable_alloc(PAGE_SHIFT); - set_pgd(&idmap_pg_dir[start >> VA_BITS], - __pgd(pgd_phys | P4D_TYPE_TABLE)); - pgd = __va(pgd_phys); - } - __create_pgd_mapping(pgd, start, start, size, PAGE_KERNEL_ROX, - early_pgtable_alloc, 0); + u64 end = __pa_symbol(__idmap_text_end); + u64 ptep = __pa_symbol(idmap_ptes); + + __pi_map_range(&ptep, start, end, start, PAGE_KERNEL_ROX, + IDMAP_ROOT_LEVEL, (pte_t *)idmap_pg_dir, false, + __phys_to_virt(ptep) - ptep); if (IS_ENABLED(CONFIG_UNMAP_KERNEL_AT_EL0)) { extern u32 __idmap_kpti_flag; @@ -782,8 +781,10 @@ static void __init create_idmap(void) * The KPTI G-to-nG conversion code needs a read-write mapping * of its synchronization flag in the ID map. */ - __create_pgd_mapping(pgd, pa, pa, sizeof(u32), PAGE_KERNEL, - early_pgtable_alloc, 0); + ptep = __pa_symbol(kpti_ptes); + __pi_map_range(&ptep, pa, pa + sizeof(u32), pa, PAGE_KERNEL, + IDMAP_ROOT_LEVEL, (pte_t *)idmap_pg_dir, false, + __phys_to_virt(ptep) - ptep); } } @@ -808,6 +809,7 @@ void __init paging_init(void) memblock_allow_resize(); create_idmap(); + idmap_t0sz = TCR_T0SZ(IDMAP_VA_BITS); } #ifdef CONFIG_MEMORY_HOTPLUG diff --git a/arch/arm64/mm/proc.S b/arch/arm64/mm/proc.S index 7c1bdaf25408..47ede52bb900 100644 --- a/arch/arm64/mm/proc.S +++ b/arch/arm64/mm/proc.S @@ -421,9 +421,9 @@ SYM_FUNC_START(__cpu_setup) mair .req x17 tcr .req x16 mov_q mair, MAIR_EL1_SET - mov_q tcr, TCR_TxSZ(VA_BITS) | TCR_CACHE_FLAGS | TCR_SMP_FLAGS | \ - TCR_TG_FLAGS | TCR_KASLR_FLAGS | TCR_ASID16 | \ - TCR_TBI0 | TCR_A1 | TCR_KASAN_SW_FLAGS | TCR_MTE_FLAGS + mov_q tcr, TCR_T0SZ(IDMAP_VA_BITS) | TCR_T1SZ(VA_BITS) | TCR_CACHE_FLAGS | \ + TCR_SMP_FLAGS | TCR_TG_FLAGS | TCR_KASLR_FLAGS | TCR_ASID16 | \ + TCR_TBI0 | TCR_A1 | TCR_KASAN_SW_FLAGS | TCR_MTE_FLAGS tcr_clear_errata_bits tcr, x9, x5 @@ -431,10 +431,7 @@ SYM_FUNC_START(__cpu_setup) sub x9, xzr, x0 add x9, x9, #64 tcr_set_t1sz tcr, x9 -#else - idmap_get_t0sz x9 #endif - tcr_set_t0sz tcr, x9 /* * Set the IPS bits in TCR_EL1. From patchwork Tue Jan 23 14:53:26 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13527544 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 9E99AC47DDF for ; Tue, 23 Jan 2024 14:57:03 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=mlStx1g+wdLkIuUHVXPME0QCvqUJJi7rYZ+CV00EXVU=; b=yKiiZx3Al7wbxiET62lADWQ6UD y2U89DiyPBoYbZ8snpETEgNl/tF/9TscEiHELZgXV3VD53hZl5sno4gOX2A0MjwlYr5uv4BI7WAVY CpG3mFnnfoNPPGel0xOiP6A7lrXTNwhDDjkORMtqC3ClYkguP7fARIVuGE1jryFPfRa1JB8FQFEp7 fEv+fvMrq9ShvnP4DHloF+24bQ9rptDVpQlPfjqGb7LvphblgcBrIs+0H6QSFtwYKrTsXBytbQVKR Hu78v4U3dkf3/P8Cz/hG4NCkNiKrR4hznAxQMSbR0WCg9H9eQqq6Odzr6Am2rLopOWL4rKscAHU75 KPBuQ6Fw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rSICQ-00GucK-08; Tue, 23 Jan 2024 14:56:26 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rSIBb-00GuDx-15 for linux-arm-kernel@bombadil.infradead.org; Tue, 23 Jan 2024 14:55:35 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=Kn7P6Oq1LFp+DUl5GfvtGAwXOTQGXV6AGlrtT3ROFPE=; b=kmzDiuJkmsEWz7whhg45SI76/a mcos/Wen+zi1igRs7JsTiNscikyjN46qiObpqwsb9Nbkwx/3979btxtRJu1yT6qfG4Ycc/YRwtyfk R4TvQ4VaEnaXDAQkKEQiWqgAuOFcE9EuTxtcRhm1OqrknOokS+XcF3u65dJs9PQtlb9aRKC2uIZZX b+yIhJEcNFCQKcZbY8+Opgbp8uAgo2sN6lyct0SecSPuciSZQl0eBQoBkRsqBVFTPGClXK345/teJ sGBmt+izK1MC0HV8FNW4XAag7EFSPjo3F/QDrHF0smBiFcO/mSXIyaNsjBek7piKMeytriJk+GM7c sKocvdOg==; Received: from mail-wr1-x44a.google.com ([2a00:1450:4864:20::44a]) by desiato.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rSIBT-00000003vQw-288f for linux-arm-kernel@lists.infradead.org; Tue, 23 Jan 2024 14:55:34 +0000 Received: by mail-wr1-x44a.google.com with SMTP id ffacd0b85a97d-3392c9e8001so1157867f8f.0 for ; Tue, 23 Jan 2024 06:55:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1706021725; x=1706626525; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=Kn7P6Oq1LFp+DUl5GfvtGAwXOTQGXV6AGlrtT3ROFPE=; b=TrwN74pHKM+ayAmJsh2xJDuSc2ipP9oWN/283RXDFo2SJnP9IPr5Y9XvjsfqEDbhAs NNaeQwrQlQmrIW+ADjjj1JpBK/efVx6ywqeGzFY6MX28T/3tBIMBNR6nXhKagC/YWmEm RyMZ6q6XHjPFFVs4tduf8wPIPkcrhGw3S1X8aZDa7p9axTaQuj2AUqA2iPOdFH7FogI3 XTWSScNL78onH8XgJqlf7jzw+bhVobljvjDYyHj8Yv9vHJmWpEuL70ahj27AUsjALViA TVPaTK6QKVuhbTv3SwN7goPqVDFwKb2uq1lkUvsXzsZWowXzoGJA2AuHsJ13Yn3dwI/2 4LdA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706021725; x=1706626525; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Kn7P6Oq1LFp+DUl5GfvtGAwXOTQGXV6AGlrtT3ROFPE=; b=Tyy6IINws//Mqphq2pUmnL4I5kwRbjPg6i2Di6knTSHyp/FT82ALJxh5qxzd6hGjUl VsvhwSb3/WYwCLXJD6xDjSnOONJrWWkDv/dX8bVAQsKHFVdguOCXdqqrSE8Sp9dQ1Abp 1+wywQSb3aFMDgXw9bOq1uyw08yA4+eAPqwjOKu1OhfrV7s3VM/fSvHkX9ztAok+L7x8 qTV5LSayqYCZOisoVMXGKo3b7/fXYs0dplOQRMuZsmz0eter50VbKTMVXf0B2hma4tED sjrYTUlmMmDGzRjAOm2K4aRKVAlj1FwZ8bJYUw10Ifpoz4gXeV52K7WMltNQ3ttZmzr5 5BMA== X-Gm-Message-State: AOJu0YwVuLh36aFg1EHHR3v7m8ZJwXlx7RH44Ox8B5dFlr2eqCwPrwoT SSdiNjYORCTqV6DmdNX/7tRtFzOeukuZLOETrTyV3LyKBLB3l0ySZVLDGFvatHIkmFCCTefCiZ2 qLN84Kl50LRoYUvi32iqSmBpxuz8dFoGY5EkCFSDvNYdC4V/fWXO0XPBNlOM2R/5zko9qnBcOme 07cOxqI0lvGbXPuEIUUJr0riW0hh9RaFoo1SxzW3mM X-Google-Smtp-Source: AGHT+IFSGlfzVFO1SEhS+T9cw6KdL5rmEayfzpP52Mr8aUfn3VQFjxlhpXWNDZ60LXWFYCHwM6UiiRDZ X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a5d:6a42:0:b0:337:bce4:7f3b with SMTP id t2-20020a5d6a42000000b00337bce47f3bmr29930wrw.7.1706021724969; Tue, 23 Jan 2024 06:55:24 -0800 (PST) Date: Tue, 23 Jan 2024 15:53:26 +0100 In-Reply-To: <20240123145258.1462979-52-ardb+git@google.com> Mime-Version: 1.0 References: <20240123145258.1462979-52-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=5168; i=ardb@kernel.org; h=from:subject; bh=3d8OYNvEb2VUdK20lgpBOirlo30qq1GDYd9RE4l7dFo=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIXX9pRdBDk0NgScj781KvTKn7b/rlfANCp2CnfuNdRY39 deUTrnfUcrCIMbBICumyCIw+++7nacnStU6z5KFmcPKBDKEgYtTACbyvIyRoU1lsfRGidm/7wqe dzh/6+4pTd4pU9qPCa8q5jugWbQ8ZxfD/6LOvRLbeyYIr1Z4tf2Nf9v1a+wn24uP5yw5y79l+7Z 55/gA X-Mailer: git-send-email 2.43.0.429.g432eaa2c6b-goog Message-ID: <20240123145258.1462979-79-ardb+git@google.com> Subject: [PATCH v7 27/50] arm64: pgtable: Decouple PGDIR size macros from PGD/PUD/PMD levels From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240123_145527_743912_19125897 X-CRM114-Status: GOOD ( 18.96 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel The mapping from PGD/PUD/PMD to levels and shifts is very confusing, given that, due to folding, the shifts may be equal for different levels, if the macros are even #define'd to begin with. In a subsequent patch, we will modify the ID mapping code to decouple the number of levels from the kernel's view of how these types are folded, so prepare for this by reformulating the macros without the use of these types. Instead, use SWAPPER_BLOCK_SHIFT as the base quantity, and derive it from either PAGE_SHIFT or PMD_SHIFT, which -if defined at all- are defined unambiguously for a given page size, regardless of the number of configured levels. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/kernel-pgtable.h | 65 ++++++-------------- 1 file changed, 19 insertions(+), 46 deletions(-) diff --git a/arch/arm64/include/asm/kernel-pgtable.h b/arch/arm64/include/asm/kernel-pgtable.h index 742a4b2778f7..f1fc98a233d5 100644 --- a/arch/arm64/include/asm/kernel-pgtable.h +++ b/arch/arm64/include/asm/kernel-pgtable.h @@ -13,27 +13,22 @@ #include /* - * The linear mapping and the start of memory are both 2M aligned (per - * the arm64 booting.txt requirements). Hence we can use section mapping - * with 4K (section size = 2M) but not with 16K (section size = 32M) or - * 64K (section size = 512M). + * The physical and virtual addresses of the start of the kernel image are + * equal modulo 2 MiB (per the arm64 booting.txt requirements). Hence we can + * use section mapping with 4K (section size = 2M) but not with 16K (section + * size = 32M) or 64K (section size = 512M). */ - -/* - * The idmap and swapper page tables need some space reserved in the kernel - * image. Both require pgd, pud (4 levels only) and pmd tables to (section) - * map the kernel. With the 64K page configuration, swapper and idmap need to - * map to pte level. The swapper also maps the FDT (see __create_page_tables - * for more information). Note that the number of ID map translation levels - * could be increased on the fly if system RAM is out of reach for the default - * VA range, so pages required to map highest possible PA are reserved in all - * cases. - */ -#ifdef CONFIG_ARM64_4K_PAGES -#define SWAPPER_PGTABLE_LEVELS (CONFIG_PGTABLE_LEVELS - 1) +#if defined(PMD_SIZE) && PMD_SIZE <= MIN_KIMG_ALIGN +#define SWAPPER_BLOCK_SHIFT PMD_SHIFT +#define SWAPPER_SKIP_LEVEL 1 #else -#define SWAPPER_PGTABLE_LEVELS (CONFIG_PGTABLE_LEVELS) +#define SWAPPER_BLOCK_SHIFT PAGE_SHIFT +#define SWAPPER_SKIP_LEVEL 0 #endif +#define SWAPPER_BLOCK_SIZE (UL(1) << SWAPPER_BLOCK_SHIFT) +#define SWAPPER_TABLE_SHIFT (SWAPPER_BLOCK_SHIFT + PAGE_SHIFT - 3) + +#define SWAPPER_PGTABLE_LEVELS (CONFIG_PGTABLE_LEVELS - SWAPPER_SKIP_LEVEL) #define IDMAP_VA_BITS 48 #define IDMAP_LEVELS ARM64_HW_PGTABLE_LEVELS(IDMAP_VA_BITS) @@ -53,24 +48,13 @@ #define EARLY_ENTRIES(vstart, vend, shift, add) \ (SPAN_NR_ENTRIES(vstart, vend, shift) + (add)) -#define EARLY_PGDS(vstart, vend, add) (EARLY_ENTRIES(vstart, vend, PGDIR_SHIFT, add)) - -#if SWAPPER_PGTABLE_LEVELS > 3 -#define EARLY_PUDS(vstart, vend, add) (EARLY_ENTRIES(vstart, vend, PUD_SHIFT, add)) -#else -#define EARLY_PUDS(vstart, vend, add) (0) -#endif +#define EARLY_LEVEL(lvl, vstart, vend, add) \ + (SWAPPER_PGTABLE_LEVELS > lvl ? EARLY_ENTRIES(vstart, vend, SWAPPER_BLOCK_SHIFT + lvl * (PAGE_SHIFT - 3), add) : 0) -#if SWAPPER_PGTABLE_LEVELS > 2 -#define EARLY_PMDS(vstart, vend, add) (EARLY_ENTRIES(vstart, vend, SWAPPER_TABLE_SHIFT, add)) -#else -#define EARLY_PMDS(vstart, vend, add) (0) -#endif - -#define EARLY_PAGES(vstart, vend, add) ( 1 /* PGDIR page */ \ - + EARLY_PGDS((vstart), (vend), add) /* each PGDIR needs a next level page table */ \ - + EARLY_PUDS((vstart), (vend), add) /* each PUD needs a next level page table */ \ - + EARLY_PMDS((vstart), (vend), add)) /* each PMD needs a next level page table */ +#define EARLY_PAGES(vstart, vend, add) (1 /* PGDIR page */ \ + + EARLY_LEVEL(3, (vstart), (vend), add) /* each entry needs a next level page table */ \ + + EARLY_LEVEL(2, (vstart), (vend), add) /* each entry needs a next level page table */ \ + + EARLY_LEVEL(1, (vstart), (vend), add))/* each entry needs a next level page table */ #define INIT_DIR_SIZE (PAGE_SIZE * (EARLY_PAGES(KIMAGE_VADDR, _end, EXTRA_PAGE) + EARLY_SEGMENT_EXTRA_PAGES)) /* the initial ID map may need two extra pages if it needs to be extended */ @@ -81,17 +65,6 @@ #endif #define INIT_IDMAP_DIR_PAGES EARLY_PAGES(KIMAGE_VADDR, _end + MAX_FDT_SIZE + SWAPPER_BLOCK_SIZE, 1) -/* Initial memory map size */ -#ifdef CONFIG_ARM64_4K_PAGES -#define SWAPPER_BLOCK_SHIFT PMD_SHIFT -#define SWAPPER_BLOCK_SIZE PMD_SIZE -#define SWAPPER_TABLE_SHIFT PUD_SHIFT -#else -#define SWAPPER_BLOCK_SHIFT PAGE_SHIFT -#define SWAPPER_BLOCK_SIZE PAGE_SIZE -#define SWAPPER_TABLE_SHIFT PMD_SHIFT -#endif - /* The number of segments in the kernel image (text, rodata, inittext, initdata, data+bss) */ #define KERNEL_SEGMENT_COUNT 5 From patchwork Tue Jan 23 14:53:27 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13527546 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 0D4EDC47258 for ; Tue, 23 Jan 2024 14:57:05 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=VM06ROewX0xS5ZbfERb2DjXAnDciB5wiczeZ020WTdg=; b=hmvyxmMN5shpkn98aEoBNuk0th tvR2AGDUJCbxMywWE8o18w7sEcGq1AvTyc44tgde1icfm2tLb+4dTwh9Jha7YQ17qPfk49Ry9fime IBleN1+tVgBFrk8KZ4c/EsD4ZQ7Hy/S53+kc0jkVCyZocVt2J9sRy8uc7eInpgRN4eGxGk/aYOUzS 3qp4Ghg08G3YW4KdUI06uhvaK/6/GjQd1rkM8s85NW5L4PAuQFTrnf4IxcYub6SrKl0G8CZ7ueF0M LmhDykaMfi21OiJanvh4WSMXSCLSbWdyc5MxOfud9ya1yXK86W4NBc9/+bQeJBYPy6u22UfIEUruH chkTeLgw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rSICQ-00Guck-3D; Tue, 23 Jan 2024 14:56:27 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rSIBb-00GuE9-2B for linux-arm-kernel@bombadil.infradead.org; Tue, 23 Jan 2024 14:55:35 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=WwB0zqSPrKg3/NTE6s3VcUbfHoT1YjxwpWMuCs7RKTM=; b=BM9iUUwFLQNINlq7SKttv8pTbG Wl1lUebc9F6w0icth0McxjQS/HTlV1vtDw8i9caPxOfzh/CG0Kd1OmnYM3e99V/n7lDdjtqh78e5B Wrl1bXqcWBwIxORPYCOrMzJcpRLX9YgWWr3rg2ltVnW5IP4ht8E7prABfSh1oqGtz0IBRVEWTERRq QGUQA/5KsZ9yHXyVepMkiK4PrISU9QJBW8i3ytBkXUTgq/i0VehHleHS8NkyXwKsmykYOVIGJbYTF Cty6u5K15qpuImFZJj5Yq2GMMsAz8s4Z1336AZQkUmWc2IxdKsggmYDUl0ZQXd6Di3WDTWrcsw+MS MWDJqXxQ==; Received: from mail-wm1-x34a.google.com ([2a00:1450:4864:20::34a]) by desiato.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rSIBW-00000003vRP-1WMW for linux-arm-kernel@lists.infradead.org; Tue, 23 Jan 2024 14:55:34 +0000 Received: by mail-wm1-x34a.google.com with SMTP id 5b1f17b1804b1-40e5769e2deso41381055e9.2 for ; Tue, 23 Jan 2024 06:55:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1706021727; x=1706626527; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=WwB0zqSPrKg3/NTE6s3VcUbfHoT1YjxwpWMuCs7RKTM=; b=EgNogCymOqSUtNsrEIsRpZTQiHuC4PcxGOKd3ZZZOHeQw9wJqcG/MtYPNL/9WgtYKj 4Dng/yVThYkAevCj6zBy7OFBSUrHH8GAgRTbfBp4BjzRgh30Y7YR/BWzap2GZq2qlrJB 4kPL0u6aG//dzDgdskh4i+iWOGrPoUFtHtohAgkb866st0MQ2vHhjtZRKPukmOQEb2bB Ufgf4iHgo8NMpvgScjD7ZLfPj/fNbSHydsRF4WgaQ3l1HAs6ES6oDyonmW1aZ3D/t/vv PHdg92Zj5VQT4QNGBm7/NxQBLwihC/CVS+VqdP9NK0VY3k9r59/+JupdErKHtV8eukSR ztaQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706021727; x=1706626527; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=WwB0zqSPrKg3/NTE6s3VcUbfHoT1YjxwpWMuCs7RKTM=; b=YW0EzafsRlc7GODRyIb37ZcLOEhfyHKlIdLxm1lPy53OuZDmtGOerCRSC9Zmk48Y/Q 4V9DgbDL1IHTvAoKwbosoruJt3oq9OIZihytd5aD57Rk/It/jV7UdMR3g+skXspk9XGf uhsPgcllARX7JPCYWoV4h7J4ShDZMzDGaoFw/5ZdNTuaiaHkfUlJWRb2PpvHtEJ80V2W h/qSp5D+302hhaEo6UNmeGkjplobgUczDFX5/UUA1rdeesYsAclGI6ob1LlOvFsTfBQ4 ELxhh+hwYWbOYzjcm/ncsYOuIY9tWayEaHWORyLZv6r50x3EeqLQQ+G3Qzzj+EK/CjQn Wa5g== X-Gm-Message-State: AOJu0YwtH+SEyZpACnEXO14YQ+T+t1o7mP4WIZs/c0S41zohYvyVKnlC PR/W7J1QzwXyFP2LjfOfM8yLRY9LyX45td33qGGPaN4G9ZNAeeD246Fk+9euCDej4c3UiWM7Oiq cc3rPM7sSXyNh9y/gK2KRgFgst0cnDUFriDRUr8pySpYPdl9M3Z15d5OglM0MkA7e+pCknWSM5J S1UQOE9OOOm/j/5/cKbAnbYbjD/FsRyKw1Y3D63NWh X-Google-Smtp-Source: AGHT+IFRoR/kC2aAzAVRCmUx+BNiU/+O21M9fwvdNl0vSjwp2IR/1/c0NWz7eNUryif9yCddx03BqDpw X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:600c:6025:b0:40e:c2ba:6716 with SMTP id az37-20020a05600c602500b0040ec2ba6716mr1887wmb.2.1706021727054; Tue, 23 Jan 2024 06:55:27 -0800 (PST) Date: Tue, 23 Jan 2024 15:53:27 +0100 In-Reply-To: <20240123145258.1462979-52-ardb+git@google.com> Mime-Version: 1.0 References: <20240123145258.1462979-52-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=24473; i=ardb@kernel.org; h=from:subject; bh=cNbVNhzJ5mlr6HeTVabif1MqH1oS3fQmZ+kKjiQ63qk=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIXX9pZdMjgsn/Dnop2k4ZZOI/aSKOxVWzvLf31p5HNkpe Cf19oRbHaUsDGIcDLJiiiwCs/++23l6olSt8yxZmDmsTCBDGLg4BWAia48wMrRcOt477cChcp9j ol8MTbXuGN/e+tHhiFbf08yMk/OuWW9gZNitIrVlj8/vi82WK+0kJO9NPV+7xicny0831nB72cH eg5wA X-Mailer: git-send-email 2.43.0.429.g432eaa2c6b-goog Message-ID: <20240123145258.1462979-80-ardb+git@google.com> Subject: [PATCH v7 28/50] arm64: kernel: Create initial ID map from C code From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240123_145530_668590_5F4B7DEE X-CRM114-Status: GOOD ( 32.72 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel The asm code that creates the initial ID map is rather intricate and hard to follow. This is problematic because it makes adding support for things like LPA2 or WXN more difficult than necessary. Also, it is parameterized like the rest of the MM code to run with a configurable number of levels, which is rather pointless, given that all AArch64 CPUs implement support for 48-bit virtual addressing, and that many systems exist with DRAM located outside of the 39-bit addressable range, which is the only smaller VA size that is widely used, and we need additional tricks to make things work in that combination. So let's bite the bullet, and rip out all the asm macros, and fiddly code, and replace it with a C implementation based on the newly added routines for creating the early kernel VA mappings. And while at it, create the initial ID map based on 48-bit virtual addressing as well, regardless of the number of configured levels for the kernel proper. Note that this code may execute with the MMU and caches disabled, and is therefore not permitted to make unaligned accesses. This shouldn't generally happen in any case for the algorithm as implemented, but to be sure, let's pass -mstrict-align to the compiler just in case. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/assembler.h | 14 - arch/arm64/include/asm/kernel-pgtable.h | 50 ++-- arch/arm64/include/asm/mmu_context.h | 6 +- arch/arm64/kernel/head.S | 267 ++------------------ arch/arm64/kernel/image-vars.h | 1 + arch/arm64/kernel/pi/Makefile | 3 + arch/arm64/kernel/pi/map_kernel.c | 18 ++ arch/arm64/kernel/pi/map_range.c | 12 + arch/arm64/kernel/pi/pi.h | 4 + arch/arm64/mm/mmu.c | 5 - arch/arm64/mm/proc.S | 3 +- 11 files changed, 88 insertions(+), 295 deletions(-) diff --git a/arch/arm64/include/asm/assembler.h b/arch/arm64/include/asm/assembler.h index 513787e43329..6a467c694039 100644 --- a/arch/arm64/include/asm/assembler.h +++ b/arch/arm64/include/asm/assembler.h @@ -345,20 +345,6 @@ alternative_cb_end bfi \valreg, \t1sz, #TCR_T1SZ_OFFSET, #TCR_TxSZ_WIDTH .endm -/* - * idmap_get_t0sz - get the T0SZ value needed to cover the ID map - * - * Calculate the maximum allowed value for TCR_EL1.T0SZ so that the - * entire ID map region can be mapped. As T0SZ == (64 - #bits used), - * this number conveniently equals the number of leading zeroes in - * the physical address of _end. - */ - .macro idmap_get_t0sz, reg - adrp \reg, _end - orr \reg, \reg, #(1 << VA_BITS_MIN) - 1 - clz \reg, \reg - .endm - /* * tcr_compute_pa_size - set TCR.(I)PS to the highest supported * ID_AA64MMFR0_EL1.PARange value diff --git a/arch/arm64/include/asm/kernel-pgtable.h b/arch/arm64/include/asm/kernel-pgtable.h index f1fc98a233d5..bf05a77873a4 100644 --- a/arch/arm64/include/asm/kernel-pgtable.h +++ b/arch/arm64/include/asm/kernel-pgtable.h @@ -29,6 +29,7 @@ #define SWAPPER_TABLE_SHIFT (SWAPPER_BLOCK_SHIFT + PAGE_SHIFT - 3) #define SWAPPER_PGTABLE_LEVELS (CONFIG_PGTABLE_LEVELS - SWAPPER_SKIP_LEVEL) +#define INIT_IDMAP_PGTABLE_LEVELS (IDMAP_LEVELS - SWAPPER_SKIP_LEVEL) #define IDMAP_VA_BITS 48 #define IDMAP_LEVELS ARM64_HW_PGTABLE_LEVELS(IDMAP_VA_BITS) @@ -48,44 +49,39 @@ #define EARLY_ENTRIES(vstart, vend, shift, add) \ (SPAN_NR_ENTRIES(vstart, vend, shift) + (add)) -#define EARLY_LEVEL(lvl, vstart, vend, add) \ - (SWAPPER_PGTABLE_LEVELS > lvl ? EARLY_ENTRIES(vstart, vend, SWAPPER_BLOCK_SHIFT + lvl * (PAGE_SHIFT - 3), add) : 0) +#define EARLY_LEVEL(lvl, lvls, vstart, vend, add) \ + (lvls > lvl ? EARLY_ENTRIES(vstart, vend, SWAPPER_BLOCK_SHIFT + lvl * (PAGE_SHIFT - 3), add) : 0) -#define EARLY_PAGES(vstart, vend, add) (1 /* PGDIR page */ \ - + EARLY_LEVEL(3, (vstart), (vend), add) /* each entry needs a next level page table */ \ - + EARLY_LEVEL(2, (vstart), (vend), add) /* each entry needs a next level page table */ \ - + EARLY_LEVEL(1, (vstart), (vend), add))/* each entry needs a next level page table */ -#define INIT_DIR_SIZE (PAGE_SIZE * (EARLY_PAGES(KIMAGE_VADDR, _end, EXTRA_PAGE) + EARLY_SEGMENT_EXTRA_PAGES)) +#define EARLY_PAGES(lvls, vstart, vend, add) (1 /* PGDIR page */ \ + + EARLY_LEVEL(3, (lvls), (vstart), (vend), add) /* each entry needs a next level page table */ \ + + EARLY_LEVEL(2, (lvls), (vstart), (vend), add) /* each entry needs a next level page table */ \ + + EARLY_LEVEL(1, (lvls), (vstart), (vend), add))/* each entry needs a next level page table */ +#define INIT_DIR_SIZE (PAGE_SIZE * (EARLY_PAGES(SWAPPER_PGTABLE_LEVELS, KIMAGE_VADDR, _end, EXTRA_PAGE) \ + + EARLY_SEGMENT_EXTRA_PAGES)) -/* the initial ID map may need two extra pages if it needs to be extended */ -#if VA_BITS < 48 -#define INIT_IDMAP_DIR_SIZE ((INIT_IDMAP_DIR_PAGES + 2) * PAGE_SIZE) -#else -#define INIT_IDMAP_DIR_SIZE (INIT_IDMAP_DIR_PAGES * PAGE_SIZE) -#endif -#define INIT_IDMAP_DIR_PAGES EARLY_PAGES(KIMAGE_VADDR, _end + MAX_FDT_SIZE + SWAPPER_BLOCK_SIZE, 1) +#define INIT_IDMAP_DIR_PAGES (EARLY_PAGES(INIT_IDMAP_PGTABLE_LEVELS, KIMAGE_VADDR, _end, 1)) +#define INIT_IDMAP_DIR_SIZE ((INIT_IDMAP_DIR_PAGES + EARLY_IDMAP_EXTRA_PAGES) * PAGE_SIZE) + +#define INIT_IDMAP_FDT_PAGES (EARLY_PAGES(INIT_IDMAP_PGTABLE_LEVELS, 0UL, UL(MAX_FDT_SIZE), 1) - 1) +#define INIT_IDMAP_FDT_SIZE ((INIT_IDMAP_FDT_PAGES + EARLY_IDMAP_EXTRA_FDT_PAGES) * PAGE_SIZE) /* The number of segments in the kernel image (text, rodata, inittext, initdata, data+bss) */ #define KERNEL_SEGMENT_COUNT 5 #if SWAPPER_BLOCK_SIZE > SEGMENT_ALIGN #define EARLY_SEGMENT_EXTRA_PAGES (KERNEL_SEGMENT_COUNT + 1) -#else -#define EARLY_SEGMENT_EXTRA_PAGES 0 -#endif - /* - * Initial memory map attributes. + * The initial ID map consists of the kernel image, mapped as two separate + * segments, and may appear misaligned wrt the swapper block size. This means + * we need 3 additional pages. The DT could straddle a swapper block boundary, + * so it may need 2. */ -#define SWAPPER_PTE_FLAGS (PTE_TYPE_PAGE | PTE_AF | PTE_SHARED | PTE_UXN) -#define SWAPPER_PMD_FLAGS (PMD_TYPE_SECT | PMD_SECT_AF | PMD_SECT_S | PTE_UXN) - -#ifdef CONFIG_ARM64_4K_PAGES -#define SWAPPER_RW_MMUFLAGS (PMD_ATTRINDX(MT_NORMAL) | SWAPPER_PMD_FLAGS | PTE_WRITE) -#define SWAPPER_RX_MMUFLAGS (SWAPPER_RW_MMUFLAGS | PMD_SECT_RDONLY) +#define EARLY_IDMAP_EXTRA_PAGES 3 +#define EARLY_IDMAP_EXTRA_FDT_PAGES 2 #else -#define SWAPPER_RW_MMUFLAGS (PTE_ATTRINDX(MT_NORMAL) | SWAPPER_PTE_FLAGS | PTE_WRITE) -#define SWAPPER_RX_MMUFLAGS (SWAPPER_RW_MMUFLAGS | PTE_RDONLY) +#define EARLY_SEGMENT_EXTRA_PAGES 0 +#define EARLY_IDMAP_EXTRA_PAGES 0 +#define EARLY_IDMAP_EXTRA_FDT_PAGES 0 #endif #endif /* __ASM_KERNEL_PGTABLE_H */ diff --git a/arch/arm64/include/asm/mmu_context.h b/arch/arm64/include/asm/mmu_context.h index 926fbbcecbe0..a8a89a0f2867 100644 --- a/arch/arm64/include/asm/mmu_context.h +++ b/arch/arm64/include/asm/mmu_context.h @@ -61,11 +61,9 @@ static inline void cpu_switch_mm(pgd_t *pgd, struct mm_struct *mm) } /* - * TCR.T0SZ value to use when the ID map is active. Usually equals - * TCR_T0SZ(VA_BITS), unless system RAM is positioned very high in - * physical memory, in which case it will be smaller. + * TCR.T0SZ value to use when the ID map is active. */ -extern int idmap_t0sz; +#define idmap_t0sz TCR_T0SZ(IDMAP_VA_BITS) /* * Ensure TCR.T0SZ is set to the provided value. diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S index a1c29d64e875..545b5d8976f4 100644 --- a/arch/arm64/kernel/head.S +++ b/arch/arm64/kernel/head.S @@ -80,26 +80,42 @@ * x19 primary_entry() .. start_kernel() whether we entered with the MMU on * x20 primary_entry() .. __primary_switch() CPU boot mode * x21 primary_entry() .. start_kernel() FDT pointer passed at boot in x0 - * x22 create_idmap() .. start_kernel() ID map VA of the DT blob * x25 primary_entry() .. start_kernel() supported VA size - * x28 create_idmap() callee preserved temp register */ SYM_CODE_START(primary_entry) bl record_mmu_state bl preserve_boot_args - bl create_idmap + + adrp x1, early_init_stack + mov sp, x1 + mov x29, xzr + adrp x0, init_idmap_pg_dir + bl __pi_create_init_idmap + + /* + * If the page tables have been populated with non-cacheable + * accesses (MMU disabled), invalidate those tables again to + * remove any speculatively loaded cache lines. + */ + cbnz x19, 0f + dmb sy + mov x1, x0 // end of used region + adrp x0, init_idmap_pg_dir + adr_l x2, dcache_inval_poc + blr x2 + b 1f /* * If we entered with the MMU and caches on, clean the ID mapped part * of the primary boot code to the PoC so we can safely execute it with * the MMU off. */ - cbz x19, 0f - adrp x0, __idmap_text_start +0: adrp x0, __idmap_text_start adr_l x1, __idmap_text_end adr_l x2, dcache_clean_poc blr x2 -0: mov x0, x19 + +1: mov x0, x19 bl init_kernel_el // w0=cpu_boot_mode mov x20, x0 @@ -175,238 +191,6 @@ SYM_CODE_START_LOCAL(preserve_boot_args) ret SYM_CODE_END(preserve_boot_args) -/* - * Macro to populate page table entries, these entries can be pointers to the next level - * or last level entries pointing to physical memory. - * - * tbl: page table address - * rtbl: pointer to page table or physical memory - * index: start index to write - * eindex: end index to write - [index, eindex] written to - * flags: flags for pagetable entry to or in - * inc: increment to rtbl between each entry - * tmp1: temporary variable - * - * Preserves: tbl, eindex, flags, inc - * Corrupts: index, tmp1 - * Returns: rtbl - */ - .macro populate_entries, tbl, rtbl, index, eindex, flags, inc, tmp1 -.Lpe\@: phys_to_pte \tmp1, \rtbl - orr \tmp1, \tmp1, \flags // tmp1 = table entry - str \tmp1, [\tbl, \index, lsl #3] - add \rtbl, \rtbl, \inc // rtbl = pa next level - add \index, \index, #1 - cmp \index, \eindex - b.ls .Lpe\@ - .endm - -/* - * Compute indices of table entries from virtual address range. If multiple entries - * were needed in the previous page table level then the next page table level is assumed - * to be composed of multiple pages. (This effectively scales the end index). - * - * vstart: virtual address of start of range - * vend: virtual address of end of range - we map [vstart, vend] - * shift: shift used to transform virtual address into index - * order: #imm 2log(number of entries in page table) - * istart: index in table corresponding to vstart - * iend: index in table corresponding to vend - * count: On entry: how many extra entries were required in previous level, scales - * our end index. - * On exit: returns how many extra entries required for next page table level - * - * Preserves: vstart, vend - * Returns: istart, iend, count - */ - .macro compute_indices, vstart, vend, shift, order, istart, iend, count - ubfx \istart, \vstart, \shift, \order - ubfx \iend, \vend, \shift, \order - add \iend, \iend, \count, lsl \order - sub \count, \iend, \istart - .endm - -/* - * Map memory for specified virtual address range. Each level of page table needed supports - * multiple entries. If a level requires n entries the next page table level is assumed to be - * formed from n pages. - * - * tbl: location of page table - * rtbl: address to be used for first level page table entry (typically tbl + PAGE_SIZE) - * vstart: virtual address of start of range - * vend: virtual address of end of range - we map [vstart, vend - 1] - * flags: flags to use to map last level entries - * phys: physical address corresponding to vstart - physical memory is contiguous - * order: #imm 2log(number of entries in PGD table) - * - * If extra_shift is set, an extra level will be populated if the end address does - * not fit in 'extra_shift' bits. This assumes vend is in the TTBR0 range. - * - * Temporaries: istart, iend, tmp, count, sv - these need to be different registers - * Preserves: vstart, flags - * Corrupts: tbl, rtbl, vend, istart, iend, tmp, count, sv - */ - .macro map_memory, tbl, rtbl, vstart, vend, flags, phys, order, istart, iend, tmp, count, sv, extra_shift - sub \vend, \vend, #1 - add \rtbl, \tbl, #PAGE_SIZE - mov \count, #0 - - .ifnb \extra_shift - tst \vend, #~((1 << (\extra_shift)) - 1) - b.eq .L_\@ - compute_indices \vstart, \vend, #\extra_shift, #(PAGE_SHIFT - 3), \istart, \iend, \count - mov \sv, \rtbl - populate_entries \tbl, \rtbl, \istart, \iend, #PMD_TYPE_TABLE, #PAGE_SIZE, \tmp - mov \tbl, \sv - .endif -.L_\@: - compute_indices \vstart, \vend, #PGDIR_SHIFT, #\order, \istart, \iend, \count - mov \sv, \rtbl - populate_entries \tbl, \rtbl, \istart, \iend, #PMD_TYPE_TABLE, #PAGE_SIZE, \tmp - mov \tbl, \sv - -#if SWAPPER_PGTABLE_LEVELS > 3 - compute_indices \vstart, \vend, #PUD_SHIFT, #(PAGE_SHIFT - 3), \istart, \iend, \count - mov \sv, \rtbl - populate_entries \tbl, \rtbl, \istart, \iend, #PMD_TYPE_TABLE, #PAGE_SIZE, \tmp - mov \tbl, \sv -#endif - -#if SWAPPER_PGTABLE_LEVELS > 2 - compute_indices \vstart, \vend, #SWAPPER_TABLE_SHIFT, #(PAGE_SHIFT - 3), \istart, \iend, \count - mov \sv, \rtbl - populate_entries \tbl, \rtbl, \istart, \iend, #PMD_TYPE_TABLE, #PAGE_SIZE, \tmp - mov \tbl, \sv -#endif - - compute_indices \vstart, \vend, #SWAPPER_BLOCK_SHIFT, #(PAGE_SHIFT - 3), \istart, \iend, \count - bic \rtbl, \phys, #SWAPPER_BLOCK_SIZE - 1 - populate_entries \tbl, \rtbl, \istart, \iend, \flags, #SWAPPER_BLOCK_SIZE, \tmp - .endm - -/* - * Remap a subregion created with the map_memory macro with modified attributes - * or output address. The entire remapped region must have been covered in the - * invocation of map_memory. - * - * x0: last level table address (returned in first argument to map_memory) - * x1: start VA of the existing mapping - * x2: start VA of the region to update - * x3: end VA of the region to update (exclusive) - * x4: start PA associated with the region to update - * x5: attributes to set on the updated region - * x6: order of the last level mappings - */ -SYM_FUNC_START_LOCAL(remap_region) - sub x3, x3, #1 // make end inclusive - - // Get the index offset for the start of the last level table - lsr x1, x1, x6 - bfi x1, xzr, #0, #PAGE_SHIFT - 3 - - // Derive the start and end indexes into the last level table - // associated with the provided region - lsr x2, x2, x6 - lsr x3, x3, x6 - sub x2, x2, x1 - sub x3, x3, x1 - - mov x1, #1 - lsl x6, x1, x6 // block size at this level - - populate_entries x0, x4, x2, x3, x5, x6, x7 - ret -SYM_FUNC_END(remap_region) - -SYM_FUNC_START_LOCAL(create_idmap) - mov x28, lr - /* - * The ID map carries a 1:1 mapping of the physical address range - * covered by the loaded image, which could be anywhere in DRAM. This - * means that the required size of the VA (== PA) space is decided at - * boot time, and could be more than the configured size of the VA - * space for ordinary kernel and user space mappings. - * - * There are three cases to consider here: - * - 39 <= VA_BITS < 48, and the ID map needs up to 48 VA bits to cover - * the placement of the image. In this case, we configure one extra - * level of translation on the fly for the ID map only. (This case - * also covers 42-bit VA/52-bit PA on 64k pages). - * - * - VA_BITS == 48, and the ID map needs more than 48 VA bits. This can - * only happen when using 64k pages, in which case we need to extend - * the root level table rather than add a level. Note that we can - * treat this case as 'always extended' as long as we take care not - * to program an unsupported T0SZ value into the TCR register. - * - * - Combinations that would require two additional levels of - * translation are not supported, e.g., VA_BITS==36 on 16k pages, or - * VA_BITS==39/4k pages with 5-level paging, where the input address - * requires more than 47 or 48 bits, respectively. - */ -#if (VA_BITS < 48) -#define IDMAP_PGD_ORDER (VA_BITS - PGDIR_SHIFT) -#define EXTRA_SHIFT (PGDIR_SHIFT + PAGE_SHIFT - 3) - - /* - * If VA_BITS < 48, we have to configure an additional table level. - * First, we have to verify our assumption that the current value of - * VA_BITS was chosen such that all translation levels are fully - * utilised, and that lowering T0SZ will always result in an additional - * translation level to be configured. - */ -#if VA_BITS != EXTRA_SHIFT -#error "Mismatch between VA_BITS and page size/number of translation levels" -#endif -#else -#define IDMAP_PGD_ORDER (PHYS_MASK_SHIFT - PGDIR_SHIFT) -#define EXTRA_SHIFT - /* - * If VA_BITS == 48, we don't have to configure an additional - * translation level, but the top-level table has more entries. - */ -#endif - adrp x0, init_idmap_pg_dir - adrp x3, _text - adrp x6, _end + MAX_FDT_SIZE + SWAPPER_BLOCK_SIZE - mov_q x7, SWAPPER_RX_MMUFLAGS - - map_memory x0, x1, x3, x6, x7, x3, IDMAP_PGD_ORDER, x10, x11, x12, x13, x14, EXTRA_SHIFT - - /* Remap [.init].data, BSS and the kernel page tables r/w in the ID map */ - adrp x1, _text - adrp x2, __initdata_begin - adrp x3, _end - bic x4, x2, #SWAPPER_BLOCK_SIZE - 1 - mov_q x5, SWAPPER_RW_MMUFLAGS - mov x6, #SWAPPER_BLOCK_SHIFT - bl remap_region - - /* Remap the FDT after the kernel image */ - adrp x1, _text - adrp x22, _end + SWAPPER_BLOCK_SIZE - bic x2, x22, #SWAPPER_BLOCK_SIZE - 1 - bfi x22, x21, #0, #SWAPPER_BLOCK_SHIFT // remapped FDT address - add x3, x2, #MAX_FDT_SIZE + SWAPPER_BLOCK_SIZE - bic x4, x21, #SWAPPER_BLOCK_SIZE - 1 - mov_q x5, SWAPPER_RW_MMUFLAGS - mov x6, #SWAPPER_BLOCK_SHIFT - bl remap_region - - /* - * Since the page tables have been populated with non-cacheable - * accesses (MMU disabled), invalidate those tables again to - * remove any speculatively loaded cache lines. - */ - cbnz x19, 0f // skip cache invalidation if MMU is on - dmb sy - - adrp x0, init_idmap_pg_dir - adrp x1, init_idmap_pg_end - bl dcache_inval_poc -0: ret x28 -SYM_FUNC_END(create_idmap) - /* * Initialize CPU registers with task-specific and cpu-specific context. * @@ -729,11 +513,6 @@ SYM_FUNC_START_LOCAL(__no_granule_support) SYM_FUNC_END(__no_granule_support) SYM_FUNC_START_LOCAL(__primary_switch) - mrs x1, tcr_el1 - mov x2, #64 - VA_BITS - tcr_set_t0sz x1, x2 - msr tcr_el1, x1 - adrp x1, reserved_pg_dir adrp x2, init_idmap_pg_dir bl __enable_mmu @@ -742,7 +521,7 @@ SYM_FUNC_START_LOCAL(__primary_switch) mov sp, x1 mov x29, xzr mov x0, x20 // pass the full boot status - mov x1, x22 // pass the low FDT mapping + mov x1, x21 // pass the FDT bl __pi_early_map_kernel // Map and relocate the kernel ldr x8, =__primary_switched diff --git a/arch/arm64/kernel/image-vars.h b/arch/arm64/kernel/image-vars.h index e566b32f9c22..941a14c05184 100644 --- a/arch/arm64/kernel/image-vars.h +++ b/arch/arm64/kernel/image-vars.h @@ -52,6 +52,7 @@ PROVIDE(__pi_cavium_erratum_27456_cpus = cavium_erratum_27456_cpus); PROVIDE(__pi__ctype = _ctype); PROVIDE(__pi_memstart_offset_seed = memstart_offset_seed); +PROVIDE(__pi_init_idmap_pg_dir = init_idmap_pg_dir); PROVIDE(__pi_init_pg_dir = init_pg_dir); PROVIDE(__pi_init_pg_end = init_pg_end); diff --git a/arch/arm64/kernel/pi/Makefile b/arch/arm64/kernel/pi/Makefile index 8c2f80a46b93..4393b41f0b71 100644 --- a/arch/arm64/kernel/pi/Makefile +++ b/arch/arm64/kernel/pi/Makefile @@ -11,6 +11,9 @@ KBUILD_CFLAGS := $(subst $(CC_FLAGS_FTRACE),,$(KBUILD_CFLAGS)) -fpie \ -fno-asynchronous-unwind-tables -fno-unwind-tables \ $(call cc-option,-fno-addrsig) +# this code may run with the MMU off so disable unaligned accesses +CFLAGS_map_range.o += -mstrict-align + # remove SCS flags from all objects in this directory KBUILD_CFLAGS := $(filter-out $(CC_FLAGS_SCS), $(KBUILD_CFLAGS)) # disable LTO diff --git a/arch/arm64/kernel/pi/map_kernel.c b/arch/arm64/kernel/pi/map_kernel.c index f206373b28b0..f86e878d366d 100644 --- a/arch/arm64/kernel/pi/map_kernel.c +++ b/arch/arm64/kernel/pi/map_kernel.c @@ -128,6 +128,22 @@ static void __init map_kernel(u64 kaslr_offset, u64 va_offset, int root_level) } } +static void __init map_fdt(u64 fdt) +{ + static u8 ptes[INIT_IDMAP_FDT_SIZE] __initdata __aligned(PAGE_SIZE); + u64 efdt = fdt + MAX_FDT_SIZE; + u64 ptep = (u64)ptes; + + /* + * Map up to MAX_FDT_SIZE bytes, but avoid overlap with + * the kernel image. + */ + map_range(&ptep, fdt, (u64)_text > fdt ? min((u64)_text, efdt) : efdt, + fdt, PAGE_KERNEL, IDMAP_ROOT_LEVEL, + (pte_t *)init_idmap_pg_dir, false, 0); + dsb(ishst); +} + asmlinkage void __init early_map_kernel(u64 boot_status, void *fdt) { static char const chosen_str[] __initconst = "/chosen"; @@ -136,6 +152,8 @@ asmlinkage void __init early_map_kernel(u64 boot_status, void *fdt) int root_level = 4 - CONFIG_PGTABLE_LEVELS; int chosen; + map_fdt((u64)fdt); + /* Clear BSS and the initial page tables */ memset(__bss_start, 0, (u64)init_pg_end - (u64)__bss_start); diff --git a/arch/arm64/kernel/pi/map_range.c b/arch/arm64/kernel/pi/map_range.c index c31feda18f47..79e4f6a2efe1 100644 --- a/arch/arm64/kernel/pi/map_range.c +++ b/arch/arm64/kernel/pi/map_range.c @@ -86,3 +86,15 @@ void __init map_range(u64 *pte, u64 start, u64 end, u64 pa, pgprot_t prot, tbl++; } } + +asmlinkage u64 __init create_init_idmap(pgd_t *pg_dir) +{ + u64 ptep = (u64)pg_dir + PAGE_SIZE; + + map_range(&ptep, (u64)_stext, (u64)__initdata_begin, (u64)_stext, + PAGE_KERNEL_ROX, IDMAP_ROOT_LEVEL, (pte_t *)pg_dir, false, 0); + map_range(&ptep, (u64)__initdata_begin, (u64)_end, (u64)__initdata_begin, + PAGE_KERNEL, IDMAP_ROOT_LEVEL, (pte_t *)pg_dir, false, 0); + + return ptep; +} diff --git a/arch/arm64/kernel/pi/pi.h b/arch/arm64/kernel/pi/pi.h index d307c58e9741..1ea282a5f96a 100644 --- a/arch/arm64/kernel/pi/pi.h +++ b/arch/arm64/kernel/pi/pi.h @@ -21,6 +21,8 @@ static inline void *prel64_to_pointer(const prel64_t *offset) extern bool dynamic_scs_is_enabled; +extern pgd_t init_idmap_pg_dir[]; + void init_feature_override(u64 boot_status, const void *fdt, int chosen); u64 kaslr_early_init(void *fdt, int chosen); void relocate_kernel(u64 offset); @@ -30,3 +32,5 @@ void map_range(u64 *pgd, u64 start, u64 end, u64 pa, pgprot_t prot, int level, pte_t *tbl, bool may_use_cont, u64 va_offset); asmlinkage void early_map_kernel(u64 boot_status, void *fdt); + +asmlinkage u64 create_init_idmap(pgd_t *pgd); diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index a991f195592b..14a62c773201 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -45,8 +45,6 @@ #define NO_CONT_MAPPINGS BIT(1) #define NO_EXEC_MAPPINGS BIT(2) /* assumes FEAT_HPDS is not used */ -int idmap_t0sz __ro_after_init; - #if VA_BITS > 48 u64 vabits_actual __ro_after_init = VA_BITS_MIN; EXPORT_SYMBOL(vabits_actual); @@ -793,8 +791,6 @@ void __init paging_init(void) pgd_t *pgdp = pgd_set_fixmap(__pa_symbol(swapper_pg_dir)); extern pgd_t init_idmap_pg_dir[]; - idmap_t0sz = 63UL - __fls(__pa_symbol(_end) | GENMASK(VA_BITS_MIN - 1, 0)); - map_kernel(pgdp); map_mem(pgdp); @@ -809,7 +805,6 @@ void __init paging_init(void) memblock_allow_resize(); create_idmap(); - idmap_t0sz = TCR_T0SZ(IDMAP_VA_BITS); } #ifdef CONFIG_MEMORY_HOTPLUG diff --git a/arch/arm64/mm/proc.S b/arch/arm64/mm/proc.S index 47ede52bb900..55c366dbda8f 100644 --- a/arch/arm64/mm/proc.S +++ b/arch/arm64/mm/proc.S @@ -200,7 +200,8 @@ SYM_FUNC_ALIAS(__pi_idmap_cpu_replace_ttbr1, idmap_cpu_replace_ttbr1) #ifdef CONFIG_UNMAP_KERNEL_AT_EL0 -#define KPTI_NG_PTE_FLAGS (PTE_ATTRINDX(MT_NORMAL) | SWAPPER_PTE_FLAGS | PTE_WRITE) +#define KPTI_NG_PTE_FLAGS (PTE_ATTRINDX(MT_NORMAL) | PTE_TYPE_PAGE | \ + PTE_AF | PTE_SHARED | PTE_UXN | PTE_WRITE) .pushsection ".idmap.text", "a" From patchwork Tue Jan 23 14:53:28 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13527548 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 69853C47DDB for ; Tue, 23 Jan 2024 14:57:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=7HeIjaWK3zsiBGorH4z9vg+Gt72XnHuyfTK/kpcym6M=; b=acx/yxvdILdm9wpDvhnCKypCaL DHXJjnA36VpzXMaOVODv2sJLcMAu5D5z7OGBfF4zlQxiN+dJDl1mgmJEleJfsh7LJVmGfpqkty+P0 nYwqXcm17+lT5gqrZvh4tvaTzv4OksqPsUpr/ebvsXff+ogoPE+IFXqQe9MJZyGV74VNiAaGAndCe Jun3boFtYCiuCEczPPcoxltkXnl13AJA0nEXTfLHcQWjew67xav6vsa4UTQyS+lebNtZVH6ET0JTq CV3qdmC1UdPZbCoJ062eqcGs0/3NgO4GCfXxjyocSHiq5sOYT01MZ7lHC3/YIHBU8mHh1/PqMAgWN QfcvYuDA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rSICU-00GufC-3B; Tue, 23 Jan 2024 14:56:30 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rSIBd-00GuF5-1g for linux-arm-kernel@bombadil.infradead.org; Tue, 23 Jan 2024 14:55:37 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=H1iOnsZsuoVNv+q6thaoOxA5ijothGdYrW/92/IhhbM=; b=klzFvMvloVKInN6PRE6IXmFdk1 guPlfjRZIdsrFoEJ7OVWDBGTtLXyUhOqIElp8V3PlRekUcnrrIROgREcgUTLG/7IwOWHtEx6HLWv4 xWqDJv0dbn0UvPZTC5e65LuT+hRBjXuqZMS6J3PCdthyUtCtLsiv3N8vjA3hQpN/cYhtcj34964XK 1M8oSdocQY224qEvzm41U40X8gzGud6LYDX+xy/j9l5FfvBnWy0tyZEyfzoqwO7ArayKOCVZO+YyF xJxMtMCB6obXIW4ggPj6v4zyM3Z+dlC1cpB5bODDP2/YmhsixFAaova0Kgdpbk8VBknx3fMUY1Izn x5VIvy6Q==; Received: from mail-yb1-xb49.google.com ([2607:f8b0:4864:20::b49]) by desiato.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rSIBY-00000003vS1-1d3d for linux-arm-kernel@lists.infradead.org; Tue, 23 Jan 2024 14:55:36 +0000 Received: by mail-yb1-xb49.google.com with SMTP id 3f1490d57ef6-dc251408f56so5313198276.3 for ; Tue, 23 Jan 2024 06:55:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1706021730; x=1706626530; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=H1iOnsZsuoVNv+q6thaoOxA5ijothGdYrW/92/IhhbM=; b=1v71dm9Gwy3Lvcwe/tA29FwciEfiOF5np5Nl6BeriJDOCGK2kS+zXpG6mWz7qPN69t vnH1Bzz70j4asH5x+bc6FNvN6PY42RJ4HCK8QaGccYZx5lV3CMni8+dTQG8qK0MA5dph OlAJN94EUmd+efx6/SfPvvPAJHKc3PuIDNuU/5mWTRcFtRTWxbFqKDNHLwZk79c+M6GH GViOaLBArl0lgCyw4tCvaGQpFQiW5mIFmFFev7nUnu3AEeZNzfKJDTPtSPySQ3zDiKKW LWfQT382k+uv8CqOxe/CKlx6Nu3MjuU2aUJme6dbAhjp2QtmHRzsOLYnyE11PDqIKRxC tyNg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706021730; x=1706626530; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=H1iOnsZsuoVNv+q6thaoOxA5ijothGdYrW/92/IhhbM=; b=OQQdRmZ+vwhjTRB68pRfPksJImYlz9WGUn0vgAV9PIIcioBg182jnp6mytpUdj4fsa 7a4OIAte8pzloCQhQt2JE9BbsnZfW5FnWtGzlG3lxr1Ho3cBarKBiesHltOTilA2jcRG fiGPZxMGVipPkjyLDUS40dANfpfZDJ7i9IMeD6SzcMG6O/EykiZtJSPzSD3Gbliq/O+Y SW1fgx+6J3rXUWx2tY9DrYEeSw14lFLBwUqcCv1dLzHTQLaESCuUEYZHx7g3wR6he14k Mol/IavndAUDM4lndUVjIfBVfymgD1C10xlpsIFRcohsBV8A9tpNd5QzCraNYrDA31jw AQ7g== X-Gm-Message-State: AOJu0YzafLt4ovnAatec+xn8Oa7MJmqquEPOIg+y8PV6qovy8WdM79jj RVhdkMKSXXWCsaFj/6S098HbTwOL+jf5PWa6fz5QZXwFlhYg3OhMXq6xBjE/JbyjhSbjBZTEl5R He05kFK99C65QaVO/ErvgGrLaOm+fJB186aWUTu+8h2+zSL0km252llPdqKdtZxLUq/KyZdNdbq XKF62AcIe9OEXgVbeSbcZPyspRe9mEW1fR5yDmBR6c X-Google-Smtp-Source: AGHT+IERPnzVnCsElNfP/GC8z5MnQFydrEu/u0Y5dNDPlmnJIDED6IiLwNKJKezGPOk8IkgtIJ55H+yr X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:6902:1b92:b0:dc2:2e43:29ac with SMTP id ei18-20020a0569021b9200b00dc22e4329acmr329121ybb.12.1706021729334; Tue, 23 Jan 2024 06:55:29 -0800 (PST) Date: Tue, 23 Jan 2024 15:53:28 +0100 In-Reply-To: <20240123145258.1462979-52-ardb+git@google.com> Mime-Version: 1.0 References: <20240123145258.1462979-52-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=2122; i=ardb@kernel.org; h=from:subject; bh=fCt9XeVduRrl1YkF83ISOqj+J+Pif8rApwfg9HcHkuw=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIXX9pVe71r7WPBd9y3DC/NglKa/NCq2m8CeafqmzeXaPf /Hc0MzVHaUsDGIcDLJiiiwCs/++23l6olSt8yxZmDmsTCBDGLg4BWAir64yMhxa7vlerv5+5fl9 /yW/RfEc3W5StPlhDbPzrenzKx6sblvB8Iez2iPzYNiWR4sXNvv05aVUNFvNXLM82Xb1/oLL7Ew tR9kA X-Mailer: git-send-email 2.43.0.429.g432eaa2c6b-goog Message-ID: <20240123145258.1462979-81-ardb+git@google.com> Subject: [PATCH v7 29/50] arm64: mm: avoid fixmap for early swapper_pg_dir updates From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240123_145532_675095_09729B51 X-CRM114-Status: GOOD ( 14.25 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel Early in the boot, when .rodata is still writable, we can poke swapper_pg_dir entries directly, and there is no need to go through the fixmap. After a future patch, we will enter the kernel with swapper_pg_dir already active, and early swapper_pg_dir updates for creating the fixmap page table hierarchy itself cannot go through the fixmap for obvious reaons. So let's keep track of whether rodata is writable, and update the descriptor directly in that case. As the same reasoning applies to early KASAN init, make the function noinstr as well. Signed-off-by: Ard Biesheuvel --- arch/arm64/mm/mmu.c | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-) diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index 14a62c773201..9758f7e3f4b6 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -55,6 +55,8 @@ EXPORT_SYMBOL(kimage_voffset); u32 __boot_cpu_mode[] = { BOOT_CPU_MODE_EL2, BOOT_CPU_MODE_EL1 }; +static bool rodata_is_rw __ro_after_init = true; + /* * The booting CPU updates the failed status @__early_cpu_boot_status, * with MMU turned off. @@ -71,10 +73,21 @@ EXPORT_SYMBOL(empty_zero_page); static DEFINE_SPINLOCK(swapper_pgdir_lock); static DEFINE_MUTEX(fixmap_lock); -void set_swapper_pgd(pgd_t *pgdp, pgd_t pgd) +void noinstr set_swapper_pgd(pgd_t *pgdp, pgd_t pgd) { pgd_t *fixmap_pgdp; + /* + * Don't bother with the fixmap if swapper_pg_dir is still mapped + * writable in the kernel mapping. + */ + if (rodata_is_rw) { + WRITE_ONCE(*pgdp, pgd); + dsb(ishst); + isb(); + return; + } + spin_lock(&swapper_pgdir_lock); fixmap_pgdp = pgd_set_fixmap(__pa_symbol(pgdp)); WRITE_ONCE(*fixmap_pgdp, pgd); @@ -628,6 +641,7 @@ void mark_rodata_ro(void) * to cover NOTES and EXCEPTION_TABLE. */ section_size = (unsigned long)__init_begin - (unsigned long)__start_rodata; + WRITE_ONCE(rodata_is_rw, false); update_mapping_prot(__pa_symbol(__start_rodata), (unsigned long)__start_rodata, section_size, PAGE_KERNEL_RO); From patchwork Tue Jan 23 14:53:29 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13527549 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id CF46EC47258 for ; Tue, 23 Jan 2024 14:57:08 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=a3iXymDypwIDdnX7sqb0QGGzcNd5puSBHvGmTNXjJmw=; b=FEpT7dZcUg+kWPELaAYd5aJHEn ODcC3WEmPDnCdJ21/3Cs/fxfpqKFicr7pvHr/kWELIvIoYTnYnsa74rqRwpFwi8kCUHkfOZwRVoA9 X6w95LOwD4VPNS2YncplfBEZhWsjboJ6xMxqhUxhBNERH2RRE69zCyIiulAsVT9QrdLn3zHK+d3jE 97XPCcgVGBVItFQZSNHWsiNZiqvmv7Z1gxBmEpafItyzKUBsXpNa++IK1ThjBrv+ofmMjPyjPAGA7 NbcBUmxPvAeDpJVWNphbDrplG0NlzugjHcfm+MNXTHEyvbeJQT/UHD9aaPuJIxWUpM3a/+KHza/yu UjUDzAGg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rSICZ-00GuiL-2c; Tue, 23 Jan 2024 14:56:35 +0000 Received: from mail-yw1-x114a.google.com ([2607:f8b0:4864:20::114a]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rSIBZ-00GuCY-0W for linux-arm-kernel@lists.infradead.org; Tue, 23 Jan 2024 14:55:46 +0000 Received: by mail-yw1-x114a.google.com with SMTP id 00721157ae682-6002a655cc1so21237067b3.1 for ; Tue, 23 Jan 2024 06:55:32 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1706021732; x=1706626532; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=xrADwVSQDaO35AsOIW3w4e6mvDqR/rmIhrCJJDnObxI=; b=Z0IIQtgs3c20K/eigTcb9Tn55gvzjsLkZ+bU0YjwF38TGq7lO1aBBRdkJjBi/HLxjK bhDl+v7ye2FX80ynnSha7GqRA345bafCMZrH7Zb7u/09d+Mu5Lzx84p42VZgyxFqAHjA KMzXsZWC1Dw42vuHXKXEiAg474g7Nr/LMmrwaN1wsPC6zqUsE+U8f0ZkjB9S2wnFdLO4 oUIpdStYdvtqYVj+JqXWIcT8vcjqhan8v/eznNIQv/q+85n03cwcLZ7EPAZvMs6hamYl Cd9iRifWV+Al4E6x81uBsvuGsTqrxB/JmUk5loLV5QQfAiljzRSsqeLQRX2OpRoB/NOZ 7OJA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706021732; x=1706626532; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=xrADwVSQDaO35AsOIW3w4e6mvDqR/rmIhrCJJDnObxI=; b=ETggZJv9NP9E6vt3J1aTk+JcuzW5C086MxGFB6YkeCNK1Xl4SgJpAh2km004AGYK5w NU3wl3qnnt6vcQDgcZn/SPwygHJ3rzbPG1aZ76LJCG0pVfdKhujwMQ3xl9LquRSkaU9U ++Sba6HLZB4YPphLbuCcx1Cl5lyXfdcEAljX1S4LO27s171vWdlgHrmiRZ0KYdiFtEO4 KmFiYzIVloBAjYNL2CnPkQ0USvG2nh9X1JQZ4chlcl8a++o+SEJTSyFLc24Yb+4TMpiv nT+KIdXWwAHa3DVVOR648zXpNsZnjrE2Vw5RbND1aQTsjM0UBt4k9+Rna+R4W3sG627h c02w== X-Gm-Message-State: AOJu0Yypnpc7kHQrTvnY8vhixzItdw4Kvj/V7fYfhrIzYkgtCrDjY4Rt Vv/aUog4r0d8yvXBKsQk5o2PBb8BYQcl9nztCG+wijp9C5Bts2dgK7EkeKFqjxh5VxZnsVYkkir VDRGkrGfuKbKMw+h06RenoG5bd2DwTHML+6Ls6DrCF51V2yGbvE4Z7aeFrpziFxiih5Q53kWry2 2FIVZWrlDBgXuvKB9D2XKyiBu3hsYle35ow4XMPVYq X-Google-Smtp-Source: AGHT+IFFCD+AKhW2sBHloW898Ftv2zRTuDo03Zavhc2eTfmFDqmxUOSrcQWlrJpBMKWFF64ZUyIVmJh3 X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a25:b319:0:b0:dc1:f6f0:1708 with SMTP id l25-20020a25b319000000b00dc1f6f01708mr330858ybj.7.1706021731772; Tue, 23 Jan 2024 06:55:31 -0800 (PST) Date: Tue, 23 Jan 2024 15:53:29 +0100 In-Reply-To: <20240123145258.1462979-52-ardb+git@google.com> Mime-Version: 1.0 References: <20240123145258.1462979-52-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=10237; i=ardb@kernel.org; h=from:subject; bh=AY/Q8Z3pyD2/q2OO+41N6wGPMFqKUj5XGBa1JYzUb5M=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIXX9pddN8QKsJ2I5lzEu8317/lbxV73WmwsC+xcH7nE+c UnxWtOPjlIWBjEOBlkxRRaB2X/f7Tw9UarWeZYszBxWJpAhDFycAjCRDRsZGaYmhpXEvV7T+2be vo1KV3ZwsNT7H8096/DX/tUh4YYlvDyMDEtXF6gW3ghLT7glz/BopXV2GotL0qSHYgp+RgzLqre rMgEA X-Mailer: git-send-email 2.43.0.429.g432eaa2c6b-goog Message-ID: <20240123145258.1462979-82-ardb+git@google.com> Subject: [PATCH v7 30/50] arm64: mm: omit redundant remap of kernel image From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240123_065533_208309_D6C18688 X-CRM114-Status: GOOD ( 26.44 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel Now that the early kernel mapping is created with all the right attributes and segment boundaries, there is no longer a need to recreate it and switch to it. This also means we no longer have to copy the kasan shadow or some parts of the fixmap from one set of page tables to the other. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/fixmap.h | 1 - arch/arm64/include/asm/kasan.h | 2 - arch/arm64/include/asm/mmu.h | 2 +- arch/arm64/kernel/image-vars.h | 1 + arch/arm64/kernel/pi/map_kernel.c | 6 +- arch/arm64/mm/fixmap.c | 34 -------- arch/arm64/mm/kasan_init.c | 15 ---- arch/arm64/mm/mmu.c | 85 ++++---------------- 8 files changed, 21 insertions(+), 125 deletions(-) diff --git a/arch/arm64/include/asm/fixmap.h b/arch/arm64/include/asm/fixmap.h index 58c294a96676..8aabd45e9a13 100644 --- a/arch/arm64/include/asm/fixmap.h +++ b/arch/arm64/include/asm/fixmap.h @@ -100,7 +100,6 @@ enum fixed_addresses { #define FIXMAP_PAGE_IO __pgprot(PROT_DEVICE_nGnRE) void __init early_fixmap_init(void); -void __init fixmap_copy(pgd_t *pgdir); #define __early_set_fixmap __set_fixmap diff --git a/arch/arm64/include/asm/kasan.h b/arch/arm64/include/asm/kasan.h index 7eefc525a9df..e1b57c13f8a4 100644 --- a/arch/arm64/include/asm/kasan.h +++ b/arch/arm64/include/asm/kasan.h @@ -17,11 +17,9 @@ asmlinkage void kasan_early_init(void); void kasan_init(void); -void kasan_copy_shadow(pgd_t *pgdir); #else static inline void kasan_init(void) { } -static inline void kasan_copy_shadow(pgd_t *pgdir) { } #endif #endif diff --git a/arch/arm64/include/asm/mmu.h b/arch/arm64/include/asm/mmu.h index d0b8b4b413b6..65977c7783c5 100644 --- a/arch/arm64/include/asm/mmu.h +++ b/arch/arm64/include/asm/mmu.h @@ -110,7 +110,7 @@ static inline bool kaslr_requires_kpti(void) } #define INIT_MM_CONTEXT(name) \ - .pgd = init_pg_dir, + .pgd = swapper_pg_dir, #endif /* !__ASSEMBLY__ */ #endif diff --git a/arch/arm64/kernel/image-vars.h b/arch/arm64/kernel/image-vars.h index 941a14c05184..e140c5bda90b 100644 --- a/arch/arm64/kernel/image-vars.h +++ b/arch/arm64/kernel/image-vars.h @@ -55,6 +55,7 @@ PROVIDE(__pi_memstart_offset_seed = memstart_offset_seed); PROVIDE(__pi_init_idmap_pg_dir = init_idmap_pg_dir); PROVIDE(__pi_init_pg_dir = init_pg_dir); PROVIDE(__pi_init_pg_end = init_pg_end); +PROVIDE(__pi_swapper_pg_dir = swapper_pg_dir); PROVIDE(__pi__text = _text); PROVIDE(__pi__stext = _stext); diff --git a/arch/arm64/kernel/pi/map_kernel.c b/arch/arm64/kernel/pi/map_kernel.c index f86e878d366d..4b76a007a50d 100644 --- a/arch/arm64/kernel/pi/map_kernel.c +++ b/arch/arm64/kernel/pi/map_kernel.c @@ -124,8 +124,12 @@ static void __init map_kernel(u64 kaslr_offset, u64 va_offset, int root_level) text_prot, true, root_level); map_segment(init_pg_dir, NULL, va_offset, __inittext_begin, __inittext_end, text_prot, false, root_level); - dsb(ishst); } + + /* Copy the root page table to its final location */ + memcpy((void *)swapper_pg_dir + va_offset, init_pg_dir, PGD_SIZE); + dsb(ishst); + idmap_cpu_replace_ttbr1(swapper_pg_dir); } static void __init map_fdt(u64 fdt) diff --git a/arch/arm64/mm/fixmap.c b/arch/arm64/mm/fixmap.c index 6fc17b2e1714..9404f282f829 100644 --- a/arch/arm64/mm/fixmap.c +++ b/arch/arm64/mm/fixmap.c @@ -170,37 +170,3 @@ void *__init fixmap_remap_fdt(phys_addr_t dt_phys, int *size, pgprot_t prot) return dt_virt; } - -/* - * Copy the fixmap region into a new pgdir. - */ -void __init fixmap_copy(pgd_t *pgdir) -{ - if (!READ_ONCE(pgd_val(*pgd_offset_pgd(pgdir, FIXADDR_TOT_START)))) { - /* - * The fixmap falls in a separate pgd to the kernel, and doesn't - * live in the carveout for the swapper_pg_dir. We can simply - * re-use the existing dir for the fixmap. - */ - set_pgd(pgd_offset_pgd(pgdir, FIXADDR_TOT_START), - READ_ONCE(*pgd_offset_k(FIXADDR_TOT_START))); - } else if (CONFIG_PGTABLE_LEVELS > 3) { - pgd_t *bm_pgdp; - p4d_t *bm_p4dp; - pud_t *bm_pudp; - /* - * The fixmap shares its top level pgd entry with the kernel - * mapping. This can really only occur when we are running - * with 16k/4 levels, so we can simply reuse the pud level - * entry instead. - */ - BUG_ON(!IS_ENABLED(CONFIG_ARM64_16K_PAGES)); - bm_pgdp = pgd_offset_pgd(pgdir, FIXADDR_TOT_START); - bm_p4dp = p4d_offset(bm_pgdp, FIXADDR_TOT_START); - bm_pudp = pud_set_fixmap_offset(bm_p4dp, FIXADDR_TOT_START); - pud_populate(&init_mm, bm_pudp, lm_alias(bm_pmd)); - pud_clear_fixmap(); - } else { - BUG(); - } -} diff --git a/arch/arm64/mm/kasan_init.c b/arch/arm64/mm/kasan_init.c index 4c7ad574b946..89828ad2bca7 100644 --- a/arch/arm64/mm/kasan_init.c +++ b/arch/arm64/mm/kasan_init.c @@ -189,21 +189,6 @@ static void __init kasan_map_populate(unsigned long start, unsigned long end, kasan_pgd_populate(start & PAGE_MASK, PAGE_ALIGN(end), node, false); } -/* - * Copy the current shadow region into a new pgdir. - */ -void __init kasan_copy_shadow(pgd_t *pgdir) -{ - pgd_t *pgdp, *pgdp_new, *pgdp_end; - - pgdp = pgd_offset_k(KASAN_SHADOW_START); - pgdp_end = pgd_offset_k(KASAN_SHADOW_END); - pgdp_new = pgd_offset_pgd(pgdir, KASAN_SHADOW_START); - do { - set_pgd(pgdp_new, READ_ONCE(*pgdp)); - } while (pgdp++, pgdp_new++, pgdp != pgdp_end); -} - static void __init clear_pgds(unsigned long start, unsigned long end) { diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index 9758f7e3f4b6..3db40b517947 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -648,9 +648,9 @@ void mark_rodata_ro(void) debug_checkwx(); } -static void __init map_kernel_segment(pgd_t *pgdp, void *va_start, void *va_end, - pgprot_t prot, struct vm_struct *vma, - int flags, unsigned long vm_flags) +static void __init declare_vma(struct vm_struct *vma, + void *va_start, void *va_end, + unsigned long vm_flags) { phys_addr_t pa_start = __pa_symbol(va_start); unsigned long size = va_end - va_start; @@ -658,9 +658,6 @@ static void __init map_kernel_segment(pgd_t *pgdp, void *va_start, void *va_end, BUG_ON(!PAGE_ALIGNED(pa_start)); BUG_ON(!PAGE_ALIGNED(size)); - __create_pgd_mapping(pgdp, pa_start, (unsigned long)va_start, size, prot, - early_pgtable_alloc, flags); - if (!(vm_flags & VM_NO_GUARD)) size += PAGE_SIZE; @@ -673,12 +670,12 @@ static void __init map_kernel_segment(pgd_t *pgdp, void *va_start, void *va_end, vm_area_add_early(vma); } +#ifdef CONFIG_UNMAP_KERNEL_AT_EL0 static pgprot_t kernel_exec_prot(void) { return rodata_enabled ? PAGE_KERNEL_ROX : PAGE_KERNEL_EXEC; } -#ifdef CONFIG_UNMAP_KERNEL_AT_EL0 static int __init map_entry_trampoline(void) { int i; @@ -713,60 +710,17 @@ core_initcall(map_entry_trampoline); #endif /* - * Open coded check for BTI, only for use to determine configuration - * for early mappings for before the cpufeature code has run. - */ -static bool arm64_early_this_cpu_has_bti(void) -{ - u64 pfr1; - - if (!IS_ENABLED(CONFIG_ARM64_BTI_KERNEL)) - return false; - - pfr1 = __read_sysreg_by_encoding(SYS_ID_AA64PFR1_EL1); - return cpuid_feature_extract_unsigned_field(pfr1, - ID_AA64PFR1_EL1_BT_SHIFT); -} - -/* - * Create fine-grained mappings for the kernel. + * Declare the VMA areas for the kernel */ -static void __init map_kernel(pgd_t *pgdp) +static void __init declare_kernel_vmas(void) { - static struct vm_struct vmlinux_text, vmlinux_rodata, vmlinux_inittext, - vmlinux_initdata, vmlinux_data; - - /* - * External debuggers may need to write directly to the text - * mapping to install SW breakpoints. Allow this (only) when - * explicitly requested with rodata=off. - */ - pgprot_t text_prot = kernel_exec_prot(); - - /* - * If we have a CPU that supports BTI and a kernel built for - * BTI then mark the kernel executable text as guarded pages - * now so we don't have to rewrite the page tables later. - */ - if (arm64_early_this_cpu_has_bti()) - text_prot = __pgprot_modify(text_prot, PTE_GP, PTE_GP); + static struct vm_struct vmlinux_seg[KERNEL_SEGMENT_COUNT]; - /* - * Only rodata will be remapped with different permissions later on, - * all other segments are allowed to use contiguous mappings. - */ - map_kernel_segment(pgdp, _stext, _etext, text_prot, &vmlinux_text, 0, - VM_NO_GUARD); - map_kernel_segment(pgdp, __start_rodata, __inittext_begin, PAGE_KERNEL, - &vmlinux_rodata, NO_CONT_MAPPINGS, VM_NO_GUARD); - map_kernel_segment(pgdp, __inittext_begin, __inittext_end, text_prot, - &vmlinux_inittext, 0, VM_NO_GUARD); - map_kernel_segment(pgdp, __initdata_begin, __initdata_end, PAGE_KERNEL, - &vmlinux_initdata, 0, VM_NO_GUARD); - map_kernel_segment(pgdp, _data, _end, PAGE_KERNEL, &vmlinux_data, 0, 0); - - fixmap_copy(pgdp); - kasan_copy_shadow(pgdp); + declare_vma(&vmlinux_seg[0], _stext, _etext, VM_NO_GUARD); + declare_vma(&vmlinux_seg[1], __start_rodata, __inittext_begin, VM_NO_GUARD); + declare_vma(&vmlinux_seg[2], __inittext_begin, __inittext_end, VM_NO_GUARD); + declare_vma(&vmlinux_seg[3], __initdata_begin, __initdata_end, VM_NO_GUARD); + declare_vma(&vmlinux_seg[4], _data, _end, 0); } void __pi_map_range(u64 *pgd, u64 start, u64 end, u64 pa, pgprot_t prot, @@ -802,23 +756,12 @@ static void __init create_idmap(void) void __init paging_init(void) { - pgd_t *pgdp = pgd_set_fixmap(__pa_symbol(swapper_pg_dir)); - extern pgd_t init_idmap_pg_dir[]; - - map_kernel(pgdp); - map_mem(pgdp); - - pgd_clear_fixmap(); - - cpu_replace_ttbr1(lm_alias(swapper_pg_dir), init_idmap_pg_dir); - init_mm.pgd = swapper_pg_dir; - - memblock_phys_free(__pa_symbol(init_pg_dir), - __pa_symbol(init_pg_end) - __pa_symbol(init_pg_dir)); + map_mem(swapper_pg_dir); memblock_allow_resize(); create_idmap(); + declare_kernel_vmas(); } #ifdef CONFIG_MEMORY_HOTPLUG From patchwork Tue Jan 23 14:53:30 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13527550 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id E196EC47258 for ; Tue, 23 Jan 2024 14:57:11 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=Lzd5wcoAyYyYPNbrDKC1Ch1kFP2yMrEc8GjeQqoPdgQ=; b=LIXL43seXrGYvFcvmRqA/zgkRD eHYOBC38XHuF9SQ9XQet4rcL9ckCfq4jcce4EegmhKJRWB7Wlk2PpF1ZDTWuXEifJvmN/WFT3Xf2w 2h5Ci3nnmquWbN4d45htusHAgF9yoXy+Rsr7VmV40u7yk++7Xeiqv1B8CkZ/Nuu6X0e+qWNvCCVKJ O+7nipAEIug/jteC1W/jn+IlRANKR4V8zMqHwdhm3GIpAqWcvKKDwBolPuYrxCiGFvEKur55GDZYM y0cv8PT12YlR4DR/9qhnA0fW7FA2cKzr2jaJGf/NwRWqnfO1HaGN2F8Vu20AbkD+gd/kTVYKqgdXG AKGl4sxw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rSICc-00GukR-1m; Tue, 23 Jan 2024 14:56:38 +0000 Received: from casper.infradead.org ([2001:8b0:10b:1236::1]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rSIBo-00GuL9-18 for linux-arm-kernel@bombadil.infradead.org; Tue, 23 Jan 2024 14:55:48 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=U8NkWuOcE5Bvqh+HJIC2fKSUV/GoC1TWDkBkhd+IbL8=; b=B1wXegTEfmXadz0k81Icd5W6S7 OSwC9TMzTbhjDn8VGkZC+7NwLw7Ht9f3ItMiAr55vdP/n13W3TphLMqMOTiE2UFOJGSDwWbmiMaxK 0UZ3WFGf8fHwy0ojoNRoiyuuvfvW20iJZDfoeSZX7WNM7SFXQwnLyZPBet82MrctoDgdKO8j8E+tR FVbWSkcjLQrovt8S488JcEncRS0K4gBf2Rta6nJd7Guh967u6u71hQ9hGFLAg2IPbwgyrC1bMhTLm 8H5wyimCegGhpXMEqaMdrTZSO7EoJYJ4H+VrXUfnstUHbIb5gAIEpqEPaY53a1U6t8rv2t9qvtrbT 9JGMDYmQ==; Received: from mail-wr1-x449.google.com ([2a00:1450:4864:20::449]) by casper.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rSIBi-00000003TnI-3tqN for linux-arm-kernel@lists.infradead.org; Tue, 23 Jan 2024 14:55:47 +0000 Received: by mail-wr1-x449.google.com with SMTP id ffacd0b85a97d-3392c9e8001so1157934f8f.0 for ; Tue, 23 Jan 2024 06:55:35 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1706021734; x=1706626534; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=U8NkWuOcE5Bvqh+HJIC2fKSUV/GoC1TWDkBkhd+IbL8=; b=k9tH5MGccexEAW1vuRjwH4PibFVER2YT8Huh+c4J22+zlEdH3jpOeO+ajSmopk2jWl poJ33FnsNFVUZ43t0Fqdc6in8tYsNbr2uHrU+IYtwDeisw+rHzN7sAyHohrEV6pqV/sq r79sgzA0Qa7q/ZyYA16qbflAFPWhoDPLf2woE7lnVwlUuUvHyUoaEJaMp22Wz0kWUAhz oesdZ9E8Gccm/9mTTIiAnjtTJQBR/9lVpoWnOUyk9Kfa2VXqeyBuIHd6QP3BXsO5bI0Y RqfFhpbzrZVeoxmj4/iwnMHEqlQrBXmc1IJEQkyIk7+EGZ2cgk+4zzLTKmPdGchxF0bs pMjw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706021734; x=1706626534; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=U8NkWuOcE5Bvqh+HJIC2fKSUV/GoC1TWDkBkhd+IbL8=; b=tr8nIIVfImGcEeS/RlD0+5HqHWoxg/7/2b0er0ayuxQ/uqPSjqjcNKMcfU9feAcKnt EScYf4orCbtBgzwf3CU0IVk0ocPqFKiDNJAKDSaUzvcp3vHRxQUIB3DBm9TK7O3Ib3ad o8q1Dc0iQPNwCjjCIszCL5OsCbkOJLSIycThnWE6+ioMIGyoVMHvJ+asiOFlqSUl3h/j p/eS8I9r2On7LxMPyTII3azIpwvYP8nKIljaGwHzNN6L8yG4QlIAeuMtEmvWmD6s9DtQ PS7NXmAtkVpsvgVVvmkbhMoTeVJFVxnCKemy+/y4xe99JF5LFXDSNFcfhdEoZ92RpK7L EJxA== X-Gm-Message-State: AOJu0Ywc4TQJbfki9SAfcLf8p9iB6BuF7nIT9q6RUPfLx7tcub3pJhvn R1UwV118KP9IJnKtQ+9IwlrfX5EB6SvyHXoUZ94tVUpipFbbNp9Gg7B8PwAEdnatQ1SGgboqDO5 t8cngYzAiFVy6LIr9nVhxKaDr3USR/z0M66ocI7YuZB3kPS0Yqxmkw9FoGRVXBlVb4r0zrG9DQv 3gjy9b0b9cW3TQaQhWuGB9fF4/mM1aY0hGUiW0G362 X-Google-Smtp-Source: AGHT+IEnBJQoPuhwcAzimhLs2Jpfo4quMRAFJN0xof852tqNI9aPKgHQETd/q+3FppYz1sVPYJEW+AkZ X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:adf:e7d1:0:b0:337:d361:8cb1 with SMTP id e17-20020adfe7d1000000b00337d3618cb1mr29877wrn.5.1706021734035; Tue, 23 Jan 2024 06:55:34 -0800 (PST) Date: Tue, 23 Jan 2024 15:53:30 +0100 In-Reply-To: <20240123145258.1462979-52-ardb+git@google.com> Mime-Version: 1.0 References: <20240123145258.1462979-52-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=3830; i=ardb@kernel.org; h=from:subject; bh=QtwsFhFXFFl5f0qjSh5EloEd18nAeBQ/5XD49A0gve8=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIXX9pTfc/j1X7j/Jb3W/eWKqf5NIp9nxX8dnvXvwlrd6f ZOWZ7JsRykLgxgHg6yYIovA7L/vdp6eKFXrPEsWZg4rE8gQBi5OAZjI6kZGhm1H6ha9na0366vv HEvrZ6c52FyfZE5aFHT9melhrmcXbDUY/op9/vmxuVyHv45lgzvnXbnVRQW5lRIrL/NF2Uid0Vv lxwIA X-Mailer: git-send-email 2.43.0.429.g432eaa2c6b-goog Message-ID: <20240123145258.1462979-83-ardb+git@google.com> Subject: [PATCH v7 31/50] arm64: Revert "mm: provide idmap pointer to cpu_replace_ttbr1()" From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240123_145544_355387_9477AE3D X-CRM114-Status: GOOD ( 13.47 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel This reverts commit 1682c45b920643c, which is no longer needed now that we create the permanent kernel mapping directly during early boot. This is a RINO (revert in name only) given that some of the code has moved around, but the changes are straight-forward. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/mmu_context.h | 17 ++++++----------- arch/arm64/mm/kasan_init.c | 4 ++-- arch/arm64/mm/mmu.c | 4 ++-- 3 files changed, 10 insertions(+), 15 deletions(-) diff --git a/arch/arm64/include/asm/mmu_context.h b/arch/arm64/include/asm/mmu_context.h index a8a89a0f2867..c768d16b81a4 100644 --- a/arch/arm64/include/asm/mmu_context.h +++ b/arch/arm64/include/asm/mmu_context.h @@ -108,18 +108,13 @@ static inline void cpu_uninstall_idmap(void) cpu_switch_mm(mm->pgd, mm); } -static inline void __cpu_install_idmap(pgd_t *idmap) +static inline void cpu_install_idmap(void) { cpu_set_reserved_ttbr0(); local_flush_tlb_all(); cpu_set_idmap_tcr_t0sz(); - cpu_switch_mm(lm_alias(idmap), &init_mm); -} - -static inline void cpu_install_idmap(void) -{ - __cpu_install_idmap(idmap_pg_dir); + cpu_switch_mm(lm_alias(idmap_pg_dir), &init_mm); } /* @@ -146,21 +141,21 @@ static inline void cpu_install_ttbr0(phys_addr_t ttbr0, unsigned long t0sz) isb(); } -void __cpu_replace_ttbr1(pgd_t *pgdp, pgd_t *idmap, bool cnp); +void __cpu_replace_ttbr1(pgd_t *pgdp, bool cnp); static inline void cpu_enable_swapper_cnp(void) { - __cpu_replace_ttbr1(lm_alias(swapper_pg_dir), idmap_pg_dir, true); + __cpu_replace_ttbr1(lm_alias(swapper_pg_dir), true); } -static inline void cpu_replace_ttbr1(pgd_t *pgdp, pgd_t *idmap) +static inline void cpu_replace_ttbr1(pgd_t *pgdp) { /* * Only for early TTBR1 replacement before cpucaps are finalized and * before we've decided whether to use CNP. */ WARN_ON(system_capabilities_finalized()); - __cpu_replace_ttbr1(pgdp, idmap, false); + __cpu_replace_ttbr1(pgdp, false); } /* diff --git a/arch/arm64/mm/kasan_init.c b/arch/arm64/mm/kasan_init.c index 89828ad2bca7..a86ab99587c9 100644 --- a/arch/arm64/mm/kasan_init.c +++ b/arch/arm64/mm/kasan_init.c @@ -225,7 +225,7 @@ static void __init kasan_init_shadow(void) */ memcpy(tmp_pg_dir, swapper_pg_dir, sizeof(tmp_pg_dir)); dsb(ishst); - cpu_replace_ttbr1(lm_alias(tmp_pg_dir), idmap_pg_dir); + cpu_replace_ttbr1(lm_alias(tmp_pg_dir)); clear_pgds(KASAN_SHADOW_START, KASAN_SHADOW_END); @@ -261,7 +261,7 @@ static void __init kasan_init_shadow(void) PAGE_KERNEL_RO)); memset(kasan_early_shadow_page, KASAN_SHADOW_INIT, PAGE_SIZE); - cpu_replace_ttbr1(lm_alias(swapper_pg_dir), idmap_pg_dir); + cpu_replace_ttbr1(lm_alias(swapper_pg_dir)); } static void __init kasan_init_depth(void) diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index 3db40b517947..a3d23da92d87 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -1445,7 +1445,7 @@ void ptep_modify_prot_commit(struct vm_area_struct *vma, unsigned long addr, pte * Atomically replaces the active TTBR1_EL1 PGD with a new VA-compatible PGD, * avoiding the possibility of conflicting TLB entries being allocated. */ -void __cpu_replace_ttbr1(pgd_t *pgdp, pgd_t *idmap, bool cnp) +void __cpu_replace_ttbr1(pgd_t *pgdp, bool cnp) { typedef void (ttbr_replace_func)(phys_addr_t); extern ttbr_replace_func idmap_cpu_replace_ttbr1; @@ -1460,7 +1460,7 @@ void __cpu_replace_ttbr1(pgd_t *pgdp, pgd_t *idmap, bool cnp) replace_phys = (void *)__pa_symbol(idmap_cpu_replace_ttbr1); - __cpu_install_idmap(idmap); + cpu_install_idmap(); /* * We really don't want to take *any* exceptions while TTBR1 is From patchwork Tue Jan 23 14:53:31 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13527556 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 42519C47258 for ; Tue, 23 Jan 2024 14:57:23 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=ziBULYYnnC3N8kWYvARpv1mkFm/VrLfgF9ccxOYveZw=; b=gG17T/fqteo7tQNN6v0pKkyY6E VBMEXk9apeFdQdqClIsikyqEAlceNwwPiA/WxlUXDBJqgNwHAtp0s2Uk7KktC/NNohq8KroFBSWUA gOOto+Z2N75Ast6S95OPv+1+0fddIrPlqRS+ZBtbWAATAXOclOONWbzcwHUEAud6qkPpzkC1T1N8I 362Yp6n57BO9+11vegBW/yfGu/JW1m11hdu8Z1EqjFLtMiLletRi7IoLMgrMUpJRAhmp8x2TUzegU F1U+wHVsHTQyUuErvmpvR1KF0DjFE9xzIx+vhEP0P0A7Ctosna8cE8wgV/K57g3j0H9ps8gj89kNB AtrXMLsQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rSICh-00GuoM-2q; Tue, 23 Jan 2024 14:56:43 +0000 Received: from mail-yb1-xb49.google.com ([2607:f8b0:4864:20::b49]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rSIBd-00GuF3-1S for linux-arm-kernel@lists.infradead.org; Tue, 23 Jan 2024 14:55:51 +0000 Received: by mail-yb1-xb49.google.com with SMTP id 3f1490d57ef6-dc2358dce6bso4923164276.3 for ; Tue, 23 Jan 2024 06:55:36 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1706021736; x=1706626536; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=QImWzZFOy8I/y7T+t/nVN2b8o1Hn6w10Yc1HMpxmnT4=; b=mjeOyUMN5AOHLxUUhYDRVkdkOog/zccIMHZa/sq8p2RX+NEpYiPTBZWOePVfo/v6Dv xUMAv+mTbbUJ+qyp0jJIX23y8ZmJGyClUiN5WatGnn86BHrPnFiiiP3xgPiGAgsJhyZ0 4941d/DFRvSk2+Vjua84YwxySxuUAlHXQigdmiXnDRuTzyKPPxjHyZC0LIyJTSZNdbx2 4l4u3eFiTB0B10gnsL3f3tSU6rEQdRl/mY6dewcDoKjwVNeyzghwhHYcBvL9niUHMOXg uE92ylefJpEotNl7YVvNqVSt4R67UMnAnXWd2ppFdAjnclM3/Mc1KV745ZCBqeG0PTT8 c89A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706021736; x=1706626536; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=QImWzZFOy8I/y7T+t/nVN2b8o1Hn6w10Yc1HMpxmnT4=; b=IQDnh6VnQybolnzYrBK1SgJ2OEf/k2Ors1U7hGbGkp4Y7P/WEUnuf1z+BAB7b7K77s b+Ebn4GUOssWiAqg1RR/28Fl7VV0wg89pq3dbm9P1gdlIgNZDh9Y/dP3BuOuLaDG/m+I HwjAGFnyLYNTvkKPT/Vnsl20PZ4V6lPym7fEQbNtzcmcGQwtA3letI0ygw29Kmg7Dott MLhxbtQ2IWNR8p+I76HozTB5wAT88J0Fxvq6CWDNTeFbwLzQBCGjFYzHtGRM5QG7o5oI /pXTI6Cltgfvm65qoGnE0nKsxd6E1ZdwUYKk5Ep5odejqSUVBkI+Lf1Cz2AdunrjNA21 4SnA== X-Gm-Message-State: AOJu0YwEa0GnBxUcZV7PE4AbvyFTYdjID5gRq/nsf8OdLwVu30wFgCEj /Gm+anSoDzrhs6YNCZvJ+m0YvmyluLCam+Juo8vUc4J3UsgRGdGilGpNqRVvAuqygk2reh8z3u7 oL+kWiHBH+drosXEHiJuRC+2rYEKpqoBmXlDm07RZpCt87aNRPFQkV/o6VdWO8f/O61DQN09sXg yJgvOHcxIud1d3Q8hBs7W4hijMjh3UEeUWHRxCGH3V X-Google-Smtp-Source: AGHT+IEz69mNnPOlnQiwXhRHekNSZ82zFlMffqDIAx2gvM5aF9vCIIMowbRswjc6I6/06vxqGa/aHwke X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a25:d0ca:0:b0:dc1:f6f0:1c6c with SMTP id h193-20020a25d0ca000000b00dc1f6f01c6cmr417581ybg.9.1706021736256; Tue, 23 Jan 2024 06:55:36 -0800 (PST) Date: Tue, 23 Jan 2024 15:53:31 +0100 In-Reply-To: <20240123145258.1462979-52-ardb+git@google.com> Mime-Version: 1.0 References: <20240123145258.1462979-52-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=10210; i=ardb@kernel.org; h=from:subject; bh=W/f1ZpQ4Q3qjibw3NuUayTfr8TvN9dv4epz/EaNodLg=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIXX9pbctzBynjsv6aD6/rtbE6n5rd03Lx6e5G3Z+CSn5+ PPdL/OCjlIWBjEOBlkxRRaB2X/f7Tw9UarWeZYszBxWJpAhDFycAjCR8D6G/wn5qR7Ht5xedpJB pfL2pmeV2aUbQqM87EJnvLnS+2b3+3SG/9G3ZZI2zKrVfizvp+w80+Sr8CaR0gm23jYXusM99t8 r4QUA X-Mailer: git-send-email 2.43.0.429.g432eaa2c6b-goog Message-ID: <20240123145258.1462979-84-ardb+git@google.com> Subject: [PATCH v7 32/50] arm64: mm: Handle LVA support as a CPU feature From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240123_065537_537280_9982EE85 X-CRM114-Status: GOOD ( 25.08 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel Currently, we detect CPU support for 52-bit virtual addressing (LVA) extremely early, before creating the kernel page tables or enabling the MMU. We cannot override the feature this early, and so large virtual addressing is always enabled on CPUs that implement support for it if the software support for it was enabled at build time. It also means we rely on non-trivial code in asm to deal with this feature. Given that both the ID map and the TTBR1 mapping of the kernel image are guaranteed to be 48-bit addressable, it is not actually necessary to enable support this early, and instead, we can model it as a CPU feature. That way, we can rely on code patching to get the correct TCR.T1SZ values programmed on secondary boot and resume from suspend. On the primary boot path, we simply enable the MMU with 48-bit virtual addressing initially, and update TCR.T1SZ if LVA is supported from C code, right before creating the kernel mapping. Given that TTBR1 still points to reserved_pg_dir at this point, updating TCR.T1SZ should be safe without the need for explicit TLB maintenance. Since this gets rid of all accesses to the vabits_actual variable from asm code that occurred before TCR.T1SZ had been programmed, we no longer have a need for this variable, and we can replace it with a C expression that produces the correct value directly, based on the value of TCR.T1SZ. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/cpufeature.h | 9 ++++++ arch/arm64/include/asm/memory.h | 13 ++++++++- arch/arm64/kernel/cpufeature.c | 13 +++++++++ arch/arm64/kernel/head.S | 29 +++++--------------- arch/arm64/kernel/image-vars.h | 1 - arch/arm64/kernel/pi/map_kernel.c | 3 ++ arch/arm64/kernel/sleep.S | 3 -- arch/arm64/mm/mmu.c | 5 ---- arch/arm64/mm/proc.S | 9 +++--- arch/arm64/tools/cpucaps | 1 + 10 files changed, 49 insertions(+), 37 deletions(-) diff --git a/arch/arm64/include/asm/cpufeature.h b/arch/arm64/include/asm/cpufeature.h index e3edae1825f3..4f4dc5496ee3 100644 --- a/arch/arm64/include/asm/cpufeature.h +++ b/arch/arm64/include/asm/cpufeature.h @@ -995,6 +995,15 @@ static inline bool cpu_has_pac(void) &id_aa64isar2_override); } +static inline bool cpu_has_lva(void) +{ + u64 mmfr2; + + mmfr2 = read_sysreg_s(SYS_ID_AA64MMFR2_EL1); + return cpuid_feature_extract_unsigned_field(mmfr2, + ID_AA64MMFR2_EL1_VARange_SHIFT); +} + #endif /* __ASSEMBLY__ */ #endif diff --git a/arch/arm64/include/asm/memory.h b/arch/arm64/include/asm/memory.h index 60904a6c4b42..9680d7444b3b 100644 --- a/arch/arm64/include/asm/memory.h +++ b/arch/arm64/include/asm/memory.h @@ -209,9 +209,20 @@ #include #include #include +#include + +static inline u64 __pure read_tcr(void) +{ + u64 tcr; + + // read_sysreg() uses asm volatile, so avoid it here + asm("mrs %0, tcr_el1" : "=r"(tcr)); + return tcr; +} #if VA_BITS > 48 -extern u64 vabits_actual; +// For reasons of #include hell, we can't use TCR_T1SZ_OFFSET/TCR_T1SZ_MASK here +#define vabits_actual (64 - ((read_tcr() >> 16) & 63)) #else #define vabits_actual ((u64)VA_BITS) #endif diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c index 7064cf13f226..8eb8c7f7b317 100644 --- a/arch/arm64/kernel/cpufeature.c +++ b/arch/arm64/kernel/cpufeature.c @@ -2692,6 +2692,19 @@ static const struct arm64_cpu_capabilities arm64_features[] = { .type = ARM64_CPUCAP_SYSTEM_FEATURE, .matches = has_lpa2, }, +#ifdef CONFIG_ARM64_VA_BITS_52 + { + .desc = "52-bit Virtual Addressing (LVA)", + .capability = ARM64_HAS_VA52, + .type = ARM64_CPUCAP_BOOT_CPU_FEATURE, + .sys_reg = SYS_ID_AA64MMFR2_EL1, + .sign = FTR_UNSIGNED, + .field_width = 4, + .field_pos = ID_AA64MMFR2_EL1_VARange_SHIFT, + .matches = has_cpuid_feature, + .min_field_value = ID_AA64MMFR2_EL1_VARange_52, + }, +#endif {}, }; diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S index 545b5d8976f4..e25351addfd0 100644 --- a/arch/arm64/kernel/head.S +++ b/arch/arm64/kernel/head.S @@ -80,7 +80,6 @@ * x19 primary_entry() .. start_kernel() whether we entered with the MMU on * x20 primary_entry() .. __primary_switch() CPU boot mode * x21 primary_entry() .. start_kernel() FDT pointer passed at boot in x0 - * x25 primary_entry() .. start_kernel() supported VA size */ SYM_CODE_START(primary_entry) bl record_mmu_state @@ -125,14 +124,6 @@ SYM_CODE_START(primary_entry) * On return, the CPU will be ready for the MMU to be turned on and * the TCR will have been set. */ -#if VA_BITS > 48 - mrs_s x0, SYS_ID_AA64MMFR2_EL1 - tst x0, ID_AA64MMFR2_EL1_VARange_MASK - mov x0, #VA_BITS - mov x25, #VA_BITS_MIN - csel x25, x25, x0, eq - mov x0, x25 -#endif bl __cpu_setup // initialise processor b __primary_switch SYM_CODE_END(primary_entry) @@ -242,11 +233,6 @@ SYM_FUNC_START_LOCAL(__primary_switched) mov x0, x20 bl set_cpu_boot_mode_flag -#if VA_BITS > 48 - adr_l x8, vabits_actual // Set this early so KASAN early init - str x25, [x8] // ... observes the correct value - dc civac, x8 // Make visible to booting secondaries -#endif #if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS) bl kasan_early_init #endif @@ -376,10 +362,13 @@ SYM_FUNC_START_LOCAL(secondary_startup) * Common entry point for secondary CPUs. */ mov x20, x0 // preserve boot mode + +#ifdef CONFIG_ARM64_VA_BITS_52 +alternative_if ARM64_HAS_VA52 bl __cpu_secondary_check52bitva -#if VA_BITS > 48 - ldr_l x0, vabits_actual +alternative_else_nop_endif #endif + bl __cpu_setup // initialise processor adrp x1, swapper_pg_dir adrp x2, idmap_pg_dir @@ -482,12 +471,8 @@ SYM_FUNC_START(__enable_mmu) ret SYM_FUNC_END(__enable_mmu) +#ifdef CONFIG_ARM64_VA_BITS_52 SYM_FUNC_START(__cpu_secondary_check52bitva) -#if VA_BITS > 48 - ldr_l x0, vabits_actual - cmp x0, #52 - b.ne 2f - mrs_s x0, SYS_ID_AA64MMFR2_EL1 and x0, x0, ID_AA64MMFR2_EL1_VARange_MASK cbnz x0, 2f @@ -498,9 +483,9 @@ SYM_FUNC_START(__cpu_secondary_check52bitva) wfi b 1b -#endif 2: ret SYM_FUNC_END(__cpu_secondary_check52bitva) +#endif SYM_FUNC_START_LOCAL(__no_granule_support) /* Indicate that this CPU can't boot and is stuck in the kernel */ diff --git a/arch/arm64/kernel/image-vars.h b/arch/arm64/kernel/image-vars.h index e140c5bda90b..2b9d702abe0f 100644 --- a/arch/arm64/kernel/image-vars.h +++ b/arch/arm64/kernel/image-vars.h @@ -36,7 +36,6 @@ PROVIDE(__pi___memcpy = __pi_memcpy); PROVIDE(__pi___memmove = __pi_memmove); PROVIDE(__pi___memset = __pi_memset); -PROVIDE(__pi_vabits_actual = vabits_actual); PROVIDE(__pi_id_aa64isar1_override = id_aa64isar1_override); PROVIDE(__pi_id_aa64isar2_override = id_aa64isar2_override); PROVIDE(__pi_id_aa64mmfr1_override = id_aa64mmfr1_override); diff --git a/arch/arm64/kernel/pi/map_kernel.c b/arch/arm64/kernel/pi/map_kernel.c index 4b76a007a50d..1853825aa29d 100644 --- a/arch/arm64/kernel/pi/map_kernel.c +++ b/arch/arm64/kernel/pi/map_kernel.c @@ -165,6 +165,9 @@ asmlinkage void __init early_map_kernel(u64 boot_status, void *fdt) chosen = fdt_path_offset(fdt, chosen_str); init_feature_override(boot_status, fdt, chosen); + if (VA_BITS > VA_BITS_MIN && cpu_has_lva()) + sysreg_clear_set(tcr_el1, TCR_T1SZ_MASK, TCR_T1SZ(VA_BITS)); + /* * The virtual KASLR displacement modulo 2MiB is decided by the * physical placement of the image, as otherwise, we might not be able diff --git a/arch/arm64/kernel/sleep.S b/arch/arm64/kernel/sleep.S index 2aa5129d8253..f093cdf71be1 100644 --- a/arch/arm64/kernel/sleep.S +++ b/arch/arm64/kernel/sleep.S @@ -102,9 +102,6 @@ SYM_CODE_START(cpu_resume) mov x0, xzr bl init_kernel_el mov x19, x0 // preserve boot mode -#if VA_BITS > 48 - ldr_l x0, vabits_actual -#endif bl __cpu_setup /* enable the MMU early - so we can access sleep_save_stash by va */ adrp x1, swapper_pg_dir diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index a3d23da92d87..ba00d0205447 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -45,11 +45,6 @@ #define NO_CONT_MAPPINGS BIT(1) #define NO_EXEC_MAPPINGS BIT(2) /* assumes FEAT_HPDS is not used */ -#if VA_BITS > 48 -u64 vabits_actual __ro_after_init = VA_BITS_MIN; -EXPORT_SYMBOL(vabits_actual); -#endif - u64 kimage_voffset __ro_after_init; EXPORT_SYMBOL(kimage_voffset); diff --git a/arch/arm64/mm/proc.S b/arch/arm64/mm/proc.S index 55c366dbda8f..d104ddab26a4 100644 --- a/arch/arm64/mm/proc.S +++ b/arch/arm64/mm/proc.S @@ -397,8 +397,6 @@ SYM_FUNC_END(idmap_kpti_install_ng_mappings) * * Initialise the processor for turning the MMU on. * - * Input: - * x0 - actual number of VA bits (ignored unless VA_BITS > 48) * Output: * Return in x0 the value of the SCTLR_EL1 register. */ @@ -422,16 +420,17 @@ SYM_FUNC_START(__cpu_setup) mair .req x17 tcr .req x16 mov_q mair, MAIR_EL1_SET - mov_q tcr, TCR_T0SZ(IDMAP_VA_BITS) | TCR_T1SZ(VA_BITS) | TCR_CACHE_FLAGS | \ + mov_q tcr, TCR_T0SZ(IDMAP_VA_BITS) | TCR_T1SZ(VA_BITS_MIN) | TCR_CACHE_FLAGS | \ TCR_SMP_FLAGS | TCR_TG_FLAGS | TCR_KASLR_FLAGS | TCR_ASID16 | \ TCR_TBI0 | TCR_A1 | TCR_KASAN_SW_FLAGS | TCR_MTE_FLAGS tcr_clear_errata_bits tcr, x9, x5 #ifdef CONFIG_ARM64_VA_BITS_52 - sub x9, xzr, x0 - add x9, x9, #64 + mov x9, #64 - VA_BITS +alternative_if ARM64_HAS_VA52 tcr_set_t1sz tcr, x9 +alternative_else_nop_endif #endif /* diff --git a/arch/arm64/tools/cpucaps b/arch/arm64/tools/cpucaps index b912b1409fc0..b370d808b3ec 100644 --- a/arch/arm64/tools/cpucaps +++ b/arch/arm64/tools/cpucaps @@ -50,6 +50,7 @@ HAS_STAGE2_FWB HAS_TCR2 HAS_TIDCP1 HAS_TLB_RANGE +HAS_VA52 HAS_VIRT_HOST_EXTN HAS_WFXT HW_DBM From patchwork Tue Jan 23 14:53:32 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13527730 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id CF1F3C47258 for ; Tue, 23 Jan 2024 16:01:13 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=KOLhNhWsOwLdz4JC9VVQf7eA+OLgJjhX6smgtOuhlMo=; b=UStK71UucTAgw2kVtfsOJAityg mSlc1th/0EW6f6sFeSvwUvkNkoWYWI2fXuzFSvkPGx7DWsfO2WrHrlfbwGrMsp95URcbk/tVJL4ja AXBDMkhJAzLFSPGHnp2QJ3kEIdHFUzuz+9bCdq/pfB9pzxIqTXya8or4QAZPEGEFrkTF5Zh6wM4mt 0JI1xeDFk59/6MZQfjAVDSdu35NzqYlXYlP7x65UPiL408GfYUNJhXwt/wRUK581OROG43gg2vs5u jfHHmJuW9a+2xnkoFKEGym/U6qiMVrLmkHHd58WyTbVWPyusRqEG10MXKmRvd67rmI4moccqCjIvn sYlljoUw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rSJCV-00HAuX-2D; Tue, 23 Jan 2024 16:00:35 +0000 Received: from mail-yw1-f201.google.com ([209.85.128.201]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rSIBi-00GuGq-0N for linux-arm-kernel@lists.infradead.org; Tue, 23 Jan 2024 14:55:58 +0000 Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-5ff7eabfd94so64238087b3.3 for ; Tue, 23 Jan 2024 06:55:39 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1706021738; x=1706626538; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=OgqzMMo0YQZO+a6NJtgIIV7c1m+h9plq6R7e39zthig=; b=dBbSNJCzupociiGOuUTvpK71PpCCL04H8mPa36Qx3ej/r9Psd2JhJfkXCViH0SdCRV /6HeoYqlWevValE56UwzgWsVQlIMue900hHy0B/BnEyePYVahuDF7S76pPe3sFmkq4WZ IKJtdtCl67U5C1w3VZ8q8uYyhnVVhf24hs0QUgrEcrHIHwRgH3OL5B81YTJwlPXIXVdE WqPe0SAnhx2Osyn1pNCrf3GF7Bm3Z9FmdhpdSddWsMvXOHqeoXLWUCuDZKGhZ+/E93jo Z6+O9z9AyWnJpLY3K9peCghB6+Vx9bQdenbUTXGJC8gMTYX7UtYDAksia1m7fon3nwNG +oFA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706021738; x=1706626538; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=OgqzMMo0YQZO+a6NJtgIIV7c1m+h9plq6R7e39zthig=; b=NVmHd7GYM9hvqZf9ssDJQtb3f6o3n1Bz1kDlh3mDERdgnMhUSQJCc/slu1RlSD4Aln gV5J5LzSzZdZ7QTZQiW83qyCzZhyfAIqYILM3tMw7r6KhZC8woH8dQmQ0H3o5b7R0eaR UWOkjOVqYuS+wyZcz4XXBeFW4iRssY7wnE5hbBKTQw4uttgr/ONCi1b3xHPYfomBOSKH Fio74LBN+goAefpc/2c1QTAF7XGjux150Nv47vHCpLgWgA+zlp/Y+yGcI3goyjG2QtPo mlMtrDnFOh1TGUCj4H9XkKGnT5NmMQPM5EcqpFYJ+V0I4QvtZX3oY5oQGYfAgAODNovN mcGQ== X-Gm-Message-State: AOJu0YzFHAS1BIT7bK0diistyhC5v3skxIfwlC+EGJQQ5jYqYuKkZ7Ke yOhTb/z+MfjHnccm6EX6I8oGuMp82x68Y7lpKg8+hrZB6pEecBsySZ7fEBE3mE+8hICvXKwQ6ub Tlu4ntIDuCKT/kAW7FPn002DWQ99uoI4E01aAtmlMwCz/SR/KP8aJP2o1exAVLkDOOzqUI1dbcW Yikgiup3XrfjpG3OXapwBtqXPux2pjUkJrERDgZBnu X-Google-Smtp-Source: AGHT+IG6EJZj1nGwAxXWz6GUceBz/Z7FoJqwShrluVL6GXzOOc9Qg9WNISK/HPhrNa0pbPfw8tUPGk8d X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a81:5709:0:b0:5f8:bd56:176f with SMTP id l9-20020a815709000000b005f8bd56176fmr2778872ywb.2.1706021738410; Tue, 23 Jan 2024 06:55:38 -0800 (PST) Date: Tue, 23 Jan 2024 15:53:32 +0100 In-Reply-To: <20240123145258.1462979-52-ardb+git@google.com> Mime-Version: 1.0 References: <20240123145258.1462979-52-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=6997; i=ardb@kernel.org; h=from:subject; bh=13TyWlMjR5z8U4YiJso59gvqkGS2/GpmFRI04ZiVl+M=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIXX9pXdV1w/f/7nydKd5//LtF0/P+nJC3iLITdZ/8o300 C3FmXf+dZSyMIhxMMiKKbIIzP77bufpiVK1zrNkYeawMoEMYeDiFICJaOQwMkyo+f974ke17LNJ aZ8XCvuzSS5zaJaResrB6jmTuV2rPpbhn51OSPmeHLuExJvJSyVXeRaf2PTmYOrlL/v5HlzzmPU njQEA X-Mailer: git-send-email 2.43.0.429.g432eaa2c6b-goog Message-ID: <20240123145258.1462979-85-ardb+git@google.com> Subject: [PATCH v7 33/50] arm64: mm: Add feature override support for LVA From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240123_065542_188997_4A1DCEF8 X-CRM114-Status: GOOD ( 22.07 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel Add support for overriding the VARange field of the MMFR2 CPU ID register. This permits the associated LVA feature to be overridden early enough for the boot code that creates the kernel mapping to take it into account. Given that LPA2 implies LVA, disabling the latter should disable the former as well. So override the ID_AA64MMFR0.TGran field of the current page size as well if it advertises support for 52-bit addressing. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/assembler.h | 17 ++++++----- arch/arm64/include/asm/cpufeature.h | 4 +++ arch/arm64/kernel/cpufeature.c | 8 +++-- arch/arm64/kernel/image-vars.h | 2 ++ arch/arm64/kernel/pi/idreg-override.c | 31 ++++++++++++++++++++ 5 files changed, 53 insertions(+), 9 deletions(-) diff --git a/arch/arm64/include/asm/assembler.h b/arch/arm64/include/asm/assembler.h index 6a467c694039..68a99b116256 100644 --- a/arch/arm64/include/asm/assembler.h +++ b/arch/arm64/include/asm/assembler.h @@ -576,18 +576,21 @@ alternative_endif .endm /* - * Offset ttbr1 to allow for 48-bit kernel VAs set with 52-bit PTRS_PER_PGD. + * If the kernel is built for 52-bit virtual addressing but the hardware only + * supports 48 bits, we cannot program the pgdir address into TTBR1 directly, + * but we have to add an offset so that the TTBR1 address corresponds with the + * pgdir entry that covers the lowest 48-bit addressable VA. + * * orr is used as it can cover the immediate value (and is idempotent). - * In future this may be nop'ed out when dealing with 52-bit kernel VAs. * ttbr: Value of ttbr to set, modified. */ .macro offset_ttbr1, ttbr, tmp #ifdef CONFIG_ARM64_VA_BITS_52 - mrs_s \tmp, SYS_ID_AA64MMFR2_EL1 - and \tmp, \tmp, #(0xf << ID_AA64MMFR2_EL1_VARange_SHIFT) - cbnz \tmp, .Lskipoffs_\@ - orr \ttbr, \ttbr, #TTBR1_BADDR_4852_OFFSET -.Lskipoffs_\@ : + mrs \tmp, tcr_el1 + and \tmp, \tmp, #TCR_T1SZ_MASK + cmp \tmp, #TCR_T1SZ(VA_BITS_MIN) + orr \tmp, \ttbr, #TTBR1_BADDR_4852_OFFSET + csel \ttbr, \tmp, \ttbr, eq #endif .endm diff --git a/arch/arm64/include/asm/cpufeature.h b/arch/arm64/include/asm/cpufeature.h index 4f4dc5496ee3..a2ac31aecdd9 100644 --- a/arch/arm64/include/asm/cpufeature.h +++ b/arch/arm64/include/asm/cpufeature.h @@ -906,7 +906,9 @@ static inline unsigned int get_vmid_bits(u64 mmfr1) s64 arm64_ftr_safe_value(const struct arm64_ftr_bits *ftrp, s64 new, s64 cur); struct arm64_ftr_reg *get_arm64_ftr_reg(u32 sys_id); +extern struct arm64_ftr_override id_aa64mmfr0_override; extern struct arm64_ftr_override id_aa64mmfr1_override; +extern struct arm64_ftr_override id_aa64mmfr2_override; extern struct arm64_ftr_override id_aa64pfr0_override; extern struct arm64_ftr_override id_aa64pfr1_override; extern struct arm64_ftr_override id_aa64zfr0_override; @@ -1000,6 +1002,8 @@ static inline bool cpu_has_lva(void) u64 mmfr2; mmfr2 = read_sysreg_s(SYS_ID_AA64MMFR2_EL1); + mmfr2 &= ~id_aa64mmfr2_override.mask; + mmfr2 |= id_aa64mmfr2_override.val; return cpuid_feature_extract_unsigned_field(mmfr2, ID_AA64MMFR2_EL1_VARange_SHIFT); } diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c index 8eb8c7f7b317..ed9670d8360c 100644 --- a/arch/arm64/kernel/cpufeature.c +++ b/arch/arm64/kernel/cpufeature.c @@ -655,7 +655,9 @@ static const struct arm64_ftr_bits ftr_raz[] = { #define ARM64_FTR_REG(id, table) \ __ARM64_FTR_REG_OVERRIDE(#id, id, table, &no_override) +struct arm64_ftr_override id_aa64mmfr0_override; struct arm64_ftr_override id_aa64mmfr1_override; +struct arm64_ftr_override id_aa64mmfr2_override; struct arm64_ftr_override id_aa64pfr0_override; struct arm64_ftr_override id_aa64pfr1_override; struct arm64_ftr_override id_aa64zfr0_override; @@ -719,10 +721,12 @@ static const struct __ftr_reg_entry { &id_aa64isar2_override), /* Op1 = 0, CRn = 0, CRm = 7 */ - ARM64_FTR_REG(SYS_ID_AA64MMFR0_EL1, ftr_id_aa64mmfr0), + ARM64_FTR_REG_OVERRIDE(SYS_ID_AA64MMFR0_EL1, ftr_id_aa64mmfr0, + &id_aa64mmfr0_override), ARM64_FTR_REG_OVERRIDE(SYS_ID_AA64MMFR1_EL1, ftr_id_aa64mmfr1, &id_aa64mmfr1_override), - ARM64_FTR_REG(SYS_ID_AA64MMFR2_EL1, ftr_id_aa64mmfr2), + ARM64_FTR_REG_OVERRIDE(SYS_ID_AA64MMFR2_EL1, ftr_id_aa64mmfr2, + &id_aa64mmfr2_override), ARM64_FTR_REG(SYS_ID_AA64MMFR3_EL1, ftr_id_aa64mmfr3), /* Op1 = 1, CRn = 0, CRm = 0 */ diff --git a/arch/arm64/kernel/image-vars.h b/arch/arm64/kernel/image-vars.h index 2b9d702abe0f..ff81f809a240 100644 --- a/arch/arm64/kernel/image-vars.h +++ b/arch/arm64/kernel/image-vars.h @@ -38,7 +38,9 @@ PROVIDE(__pi___memset = __pi_memset); PROVIDE(__pi_id_aa64isar1_override = id_aa64isar1_override); PROVIDE(__pi_id_aa64isar2_override = id_aa64isar2_override); +PROVIDE(__pi_id_aa64mmfr0_override = id_aa64mmfr0_override); PROVIDE(__pi_id_aa64mmfr1_override = id_aa64mmfr1_override); +PROVIDE(__pi_id_aa64mmfr2_override = id_aa64mmfr2_override); PROVIDE(__pi_id_aa64pfr0_override = id_aa64pfr0_override); PROVIDE(__pi_id_aa64pfr1_override = id_aa64pfr1_override); PROVIDE(__pi_id_aa64smfr0_override = id_aa64smfr0_override); diff --git a/arch/arm64/kernel/pi/idreg-override.c b/arch/arm64/kernel/pi/idreg-override.c index 1884bd936c0d..aad399796e81 100644 --- a/arch/arm64/kernel/pi/idreg-override.c +++ b/arch/arm64/kernel/pi/idreg-override.c @@ -59,6 +59,35 @@ static const struct ftr_set_desc mmfr1 __prel64_initconst = { }, }; + +static bool __init mmfr2_varange_filter(u64 val) +{ + int __maybe_unused feat; + + if (val) + return false; + +#ifdef CONFIG_ARM64_LPA2 + feat = cpuid_feature_extract_signed_field(read_sysreg(id_aa64mmfr0_el1), + ID_AA64MMFR0_EL1_TGRAN_SHIFT); + if (feat >= ID_AA64MMFR0_EL1_TGRAN_LPA2) { + id_aa64mmfr0_override.val |= + (ID_AA64MMFR0_EL1_TGRAN_LPA2 - 1) << ID_AA64MMFR0_EL1_TGRAN_SHIFT; + id_aa64mmfr0_override.mask |= 0xfU << ID_AA64MMFR0_EL1_TGRAN_SHIFT; + } +#endif + return true; +} + +static const struct ftr_set_desc mmfr2 __prel64_initconst = { + .name = "id_aa64mmfr2", + .override = &id_aa64mmfr2_override, + .fields = { + FIELD("varange", ID_AA64MMFR2_EL1_VARange_SHIFT, mmfr2_varange_filter), + {} + }, +}; + static bool __init pfr0_sve_filter(u64 val) { /* @@ -167,6 +196,7 @@ static const struct ftr_set_desc sw_features __prel64_initconst = { static const PREL64(const struct ftr_set_desc, reg) regs[] __prel64_initconst = { { &mmfr1 }, + { &mmfr2 }, { &pfr0 }, { &pfr1 }, { &isar1 }, @@ -192,6 +222,7 @@ static const struct { { "arm64.nomte", "id_aa64pfr1.mte=0" }, { "nokaslr", "arm64_sw.nokaslr=1" }, { "rodata=off", "arm64_sw.rodataoff=1" }, + { "arm64.nolva", "id_aa64mmfr2.varange=0" }, }; static int __init parse_hexdigit(const char *p, u64 *v) From patchwork Tue Jan 23 14:53:33 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13527552 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 0480DC47258 for ; Tue, 23 Jan 2024 14:57:14 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=dRS7jSCCOfsPa+XYLLsqWbSLwPIWACe19X7Nra0ANmk=; b=EvDRlpPqatwQNv6GXCZ2xoS1aL aj3IsCirS9VRNlG866HdiH8KkS4rXtsceVXCNVri9G+6WRvRbmsrv61b1bZm026u2FV7zoiZ15Wvg 5fer/B20OYayTA1UkpZ1ihZx74pFD867lHX31dHPnxBU1vDNfuL+fUV2Mk4/xwBCWT5uSv+x72k8W taUaq1f9V3ksCWth00e57y3CVHGKmBRXAZboUWeFeNJ+5HSnCWOV8Ia8BuOKl5d4wQvk7EPvgVva9 KcIp443Mo+lgRxK/0UgPUftCZA4/aMgTAEB1zKV9AW/D3XgdRc2gu7A3+UVmrnpKXFkj8J5xR5eia iCQs79Ag==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rSICb-00Guje-1L; Tue, 23 Jan 2024 14:56:37 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rSIBm-00GuJo-2Q for linux-arm-kernel@bombadil.infradead.org; Tue, 23 Jan 2024 14:55:46 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=MHFTHO0gJ07B4N0Yy9EnU2+LDAtL9wfGuZAkfKiS0UQ=; b=j9iDkPZii8kOYilRCZRORFxkrr NHAeR8fiHqrlQdxyswc7PW3k188AtGQvKGL58hMLpBV2gNJcn57Mii79hwnh6s09cZQZzSQN243XO ANOkj6Lc1PRros8FB1kDCje9C12XrvNGjyS/quqouX+NPXFNEmL0aAIJiVt8Xv1aDDtcqHYv1Yfbw d8xwjnRTJGWzDBgOmOnxEyUdqayQyM64+HbZlzeAadchUeHm9NT39gxB3a7n6Di1+hvvwS4nZGgRR vRcW8xC+0co5ilFnRNbfeXI7YKxvDi59LGgC/ASF5ONk7iUYhPS7q1vS7u6sEjBJdFIA1swMwW5NI +EkegoUQ==; Received: from mail-yw1-x1149.google.com ([2607:f8b0:4864:20::1149]) by desiato.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rSIBj-00000003vVZ-0Y5v for linux-arm-kernel@lists.infradead.org; Tue, 23 Jan 2024 14:55:44 +0000 Received: by mail-yw1-x1149.google.com with SMTP id 00721157ae682-5fface2c4afso36099727b3.0 for ; Tue, 23 Jan 2024 06:55:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1706021741; x=1706626541; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=MHFTHO0gJ07B4N0Yy9EnU2+LDAtL9wfGuZAkfKiS0UQ=; b=PHKGDk64vHxIA/B7mpyyJN9DfqzT0DoBCu0xyZjNwBRvkZoB0W1hWFJ0FCIfo+oKUV mPA2aBzI8OvPGBvNUdi4O0EKT9dQAvxNZfoUBuqDWGa95/9h5bGdcUJDVT7/bvRpDCVw YqibioxTYzKTpbvPl6gUIx/fh/wFM0M781OC+NdcZ0LCJZOuFt0axvfXAeBtkEAyu+Ks zYx8QnRTPseRGFmhWrssDU93iPvRJtNwtDgjXLaYNBoEPWauLiiF2g9WUW9tkjH2TtdK hJ+zyswB5N4nEcoEvgQWrLn0+a9pHOhWUsk/MiV5UwV4ODl1bJwW999g8uLDWvOYNJpq Pm/g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706021741; x=1706626541; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=MHFTHO0gJ07B4N0Yy9EnU2+LDAtL9wfGuZAkfKiS0UQ=; b=Bh/wuIwvgZy/6ifh4GfNRD0aRQeKjvJET6ztkZxL9yL+PvAexoXmx90toUCC7oXDmx 9wMQI+AVLbnt40YoyD9dcXB0Y1Xf7nnunI61K1UcLLYhJ8oWRkX7F0itJHRVXTw2ziFz huMX5dl6KUazzrKc8OgdIO91BhJXvs84DHNmzzyqcLEyGAYDrxhwjaquLXKgaB4r1USg wuXhuDi/6FRt5dz7i7eg2l2gjC3TkbFNZakzw8GWC1ePvh82aPai6gCt/WsKCmeaQfo/ uevuENXIyeUBU4Sgj/KMHisy59mTKyAgTMg+jewJnGmXF9rCPKnbE0WLFW1uLoifT2Sg MVMg== X-Gm-Message-State: AOJu0YzUIDzEOVgbxLkPM98+JmDxl6qlZaZD/TfPuBjZ8FiX9W4iXAlH bxL77IQr8sX5kGNaTkLGZyKIOoM8IqefyKdWamxEGcMWYtxZVEifs2dpQV/BlV8IXaw7tupSxct WcSGbxW7tqbRvieFcNkTwidavJRgjF4MZUBaParsI4iW+eTwzRD1a6dO32DsXNn/1uYwqE05XnV 2Btob42wCCFTqfWTxQ44lJ2qDPk8fsfdquTlBVRNx2 X-Google-Smtp-Source: AGHT+IGIfYafj6uMO+fAV5Evx4JGtjtChvVa+yK8yMiI5RFW4oXjnDAmCkAaDyahCnJDdxO1NSxZpVDn X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:6902:2288:b0:dbe:a220:68ee with SMTP id dn8-20020a056902228800b00dbea22068eemr328557ybb.0.1706021740830; Tue, 23 Jan 2024 06:55:40 -0800 (PST) Date: Tue, 23 Jan 2024 15:53:33 +0100 In-Reply-To: <20240123145258.1462979-52-ardb+git@google.com> Mime-Version: 1.0 References: <20240123145258.1462979-52-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=2581; i=ardb@kernel.org; h=from:subject; bh=+DIdhx9FTnpM/JUeo1/oVW1dtczn86Yl6zsEwdhDaz8=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIXX9pfepm48kB/33q2j/sf+Wws6LtUuFvPUT4/7MbMuX2 7XJv2ZfRykLgxgHg6yYIovA7L/vdp6eKFXrPEsWZg4rE8gQBi5OAZjI9d0M/6u7Wn9oc3tlKKg/ 1/OYyXR2f+LcNXLbl2Y68KyK/MCW58TIcL9hbf0z3qDmKCkba6eA5xuCP+2+p2VnUP2m/iiLmUI rOwA= X-Mailer: git-send-email 2.43.0.429.g432eaa2c6b-goog Message-ID: <20240123145258.1462979-86-ardb+git@google.com> Subject: [PATCH v7 34/50] arm64: Avoid #define'ing PTE_MAYBE_NG to 0x0 for asm use From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240123_145543_319305_912D82CD X-CRM114-Status: GOOD ( 16.03 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel The PROT_* macros resolve to expressions that are only valid in C and not in assembler, and so they are only usable from C code. Currently, we make an exception for the permission indirection init code in proc.S, which doesn't care about the bits that are conditionally set, and so we just #define PTE_MAYBE_NG to 0x0 for any assembler file that includes these definitions. This is dodgy because this means that PROT_NORMAL and friends is generally available in asm code, but defined in a way that deviates from the definition that C code will observe, which might lead to hard to diagnose issues down the road. So instead, #define PTE_MAYBE_NG only in the place where the PIE constants are evaluated, and #undef it again right after. This allows us to drop the #define from pgtable-prot.h, and avoid the risk of deviating definitions between asm and C. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/pgtable-prot.h | 4 ---- arch/arm64/mm/proc.S | 13 +++++++++++++ 2 files changed, 13 insertions(+), 4 deletions(-) diff --git a/arch/arm64/include/asm/pgtable-prot.h b/arch/arm64/include/asm/pgtable-prot.h index 483dbfa39c4c..63ced9ccec21 100644 --- a/arch/arm64/include/asm/pgtable-prot.h +++ b/arch/arm64/include/asm/pgtable-prot.h @@ -57,10 +57,6 @@ #define _PAGE_READONLY_EXEC (_PAGE_DEFAULT | PTE_USER | PTE_RDONLY | PTE_NG | PTE_PXN) #define _PAGE_EXECONLY (_PAGE_DEFAULT | PTE_RDONLY | PTE_NG | PTE_PXN) -#ifdef __ASSEMBLY__ -#define PTE_MAYBE_NG 0 -#endif - #ifndef __ASSEMBLY__ #include diff --git a/arch/arm64/mm/proc.S b/arch/arm64/mm/proc.S index d104ddab26a4..6e1b2bc41a9f 100644 --- a/arch/arm64/mm/proc.S +++ b/arch/arm64/mm/proc.S @@ -456,11 +456,24 @@ alternative_else_nop_endif ubfx x1, x1, #ID_AA64MMFR3_EL1_S1PIE_SHIFT, #4 cbz x1, .Lskip_indirection + /* + * The PROT_* macros describing the various memory types may resolve to + * C expressions if they include the PTE_MAYBE_* macros, and so they + * can only be used from C code. The PIE_E* constants below are also + * defined in terms of those macros, but will mask out those + * PTE_MAYBE_* constants, whether they are set or not. So #define them + * as 0x0 here so we can evaluate the PIE_E* constants in asm context. + */ + +#define PTE_MAYBE_NG 0 + mov_q x0, PIE_E0 msr REG_PIRE0_EL1, x0 mov_q x0, PIE_E1 msr REG_PIR_EL1, x0 +#undef PTE_MAYBE_NG + mov x0, TCR2_EL1x_PIE msr REG_TCR2_EL1, x0 From patchwork Tue Jan 23 14:53:34 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13527554 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id EF80BC47258 for ; Tue, 23 Jan 2024 14:57:17 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=p7wFpU8MRkku6ldiW3Kc/R24Tk2Us67u1LFUeMDYgZs=; b=von5g4XIRAS+WyLgFMAVgpMrg4 7fVnznkB3IOOrMQEYIWIcE3wW4bB+4/6riSZANsLU4pnCWCHusAHU4k8kMlv5Hn7+tmHbVQFydaV7 G6VzvE0gQMCp3DsmIDDb+7UqMnIP9zc2FAFh6mQ3aWoDMbdYLhAc8fAxHNrODZxFCIp0E48AettZR f1V2jF1n30DlneGODqoU3TGN59p/mzXxd0b8fiI7WDAGbGSWHsfTy3rbj1+5/yPFwyvNDDqvsKh5O rHN9kSzGQefGwTj3dJHDSuUDbZ+y4aef7zv8gNUHhDTVLT3/evqHZbt83cZ/7T1TgCtzz6kgPXjHL SwUtiKxw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rSICe-00Gulc-0X; Tue, 23 Jan 2024 14:56:40 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rSIBo-00GuLQ-2B for linux-arm-kernel@bombadil.infradead.org; Tue, 23 Jan 2024 14:55:48 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=F3a2hc8n1NEqmp15ROD9lmeVnHS2pOeqn2GhIHm4mdM=; b=ocmv6TNsr7CbRpftIaoLGkvWmM udk6/PY7ndmnBS6QX5ukenVnqlZSX1wllyL8JqKOiQk+pEvMXxHITaAF5Qrb44EByDGFZPnKGqft5 4GRJkixb+UzL6E96grl0AJPN2uzAgmnAdBzdAV3q6bgSLTjna2tcS+y/lPxwiMfl6qt9MqBUvOueL Jm0mYhJt6OeknAYHIfw1bNqzI+Vk2h0HOr6+6PZH6y9/Swsg1pZ/Misk65l02F3gDeTxI9utW6oeW OYn0nk4210hsic4TLeMlu2oQpf1fJSw9Gi1dzY5+Vv5WsvPO95YhVskp6k8H42BGU4Fj7ghNj+csl gBddUhig==; Received: from mail-yw1-x114a.google.com ([2607:f8b0:4864:20::114a]) by desiato.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rSIBl-00000003vW6-265x for linux-arm-kernel@lists.infradead.org; Tue, 23 Jan 2024 14:55:47 +0000 Received: by mail-yw1-x114a.google.com with SMTP id 00721157ae682-5f874219ff9so59508647b3.0 for ; Tue, 23 Jan 2024 06:55:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1706021743; x=1706626543; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=F3a2hc8n1NEqmp15ROD9lmeVnHS2pOeqn2GhIHm4mdM=; b=TjchzgTkS2tlEuYc3JWnYvMGT+2civJtMt+2FuXmmiJEnH0d/6IJPgTH3CWaUK6BmP 9ul2nBETn0VCfVqy5UgSj2B/2t4Vc+sCvmnecv8PSX9o5cqOTrEG/T72+aVO/n6z4jZm 9eL6eZrlJNX/xv3z/z/T45aVL08Ms5LYZbfEYxOwtDYHwK+FZ8/ppxdhiTD9MF3Q9W5A gOSuJLPg72/pmnGRLMlUXSLlo5fQv/cZb2/CxsnM7XqiO+zqzBK0GCBj+QVXhDRKoxH1 FgcD/QYCA0OlSq6IrHriLXw9imxXSCOLG162IGRFrOk4RKGVEmlRMuJaGB2+xrHhVqvB igYA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706021743; x=1706626543; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=F3a2hc8n1NEqmp15ROD9lmeVnHS2pOeqn2GhIHm4mdM=; b=JUSPWJtOwEJ7Pss5G8R5LXmE2lgZdP9ySa7G5X/ONRlDSo3l9t2B+k0RV6lcZd067v Sz0SRvexKRj3jMcAqjGdYqB5w4LgrLMWIiy28wrBJ8W6ejSt+YheX3fTRt17BBKCs6cz Q0bwlCrz3J8p/BsNiFOTTWOX3GgMgInKmrt8aNyqikSbsZT7+RQ32ARnoJBj1IHESCW1 zitOXbf8G1JZ5nL/YBQxbVsNJdUE9j53S7t1xHNS6SI064s5m0jk7kRn6jyd58d9tLW6 f/hz53CZ+UX207PvNcsVh9ILgSj8OaDS2yyPDetowk3qvJs5x8pu/gzZuPj3qwaMU7An 9iYA== X-Gm-Message-State: AOJu0YyX+Nh60qORiBo0F8u3H2BUEntdT+QsV63GnmZOZbidOWBAlnMB AykolWeTcsX/I4oxMQZHQ3K230OQ8IhkSgZvRWugeJQuxe3pZcKdUa1D8AqY4mNPWbBpYN67zBS LqB01jY9h8ud5NM+7HV39Mpm5SXHpIPwoGTDP7ilPCfvH90r37GBEx6xqlBTmx2sw8mvCI13o19 Gw73EO90jPCX2xtC5MKZJB/LBfhHNm5JUP0jt6TB1S X-Google-Smtp-Source: AGHT+IHYKvKWDX7s05YamCtVyxJyTIOWUJzX9xOAqIuqKd42b+llgpmBuQjcXmEBDVvTGI+ifhP6wU5D X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:690c:300c:b0:5ff:83c7:b5a0 with SMTP id ey12-20020a05690c300c00b005ff83c7b5a0mr3135678ywb.5.1706021743058; Tue, 23 Jan 2024 06:55:43 -0800 (PST) Date: Tue, 23 Jan 2024 15:53:34 +0100 In-Reply-To: <20240123145258.1462979-52-ardb+git@google.com> Mime-Version: 1.0 References: <20240123145258.1462979-52-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=8056; i=ardb@kernel.org; h=from:subject; bh=TzOGEPJlOyzNvktAFd7lIPH+fx980ZM7MqoATlpDSOQ=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIXX9pQ/ZUUkBNvv7D104teGLUOT5xxk8VtPMs1z+/Hiyn GWflte/jlIWBjEOBlkxRRaB2X/f7Tw9UarWeZYszBxWJpAhDFycAjCRNTaMDMtePG77s2VxRPA7 hb6vfxYu/tCnHOb9SW/7HpHsrK9+jycxMhyeGPdljWyz6nP1B48Due4m3O6S7ZETzfqr8tJqKrN 3PRMA X-Mailer: git-send-email 2.43.0.429.g432eaa2c6b-goog Message-ID: <20240123145258.1462979-87-ardb+git@google.com> Subject: [PATCH v7 35/50] arm64: Add ESR decoding for exceptions involving translation level -1 From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240123_145545_800887_A4EF27A2 X-CRM114-Status: GOOD ( 17.60 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel The LPA2 feature introduces new FSC values to report abort exceptions related to translation level -1. Define these and wire them up. Reuse the new ESR FSC classification helpers that arrived via the KVM arm64 tree, and update the one for translation faults to check specifically for a translation fault at level -1. (Access flag or permission faults cannot occur at level -1 because they alway involve a descriptor at the superior level so changing those helpers is not needed). Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/esr.h | 13 ++++----- arch/arm64/include/asm/kvm_emulate.h | 10 ++----- arch/arm64/mm/fault.c | 30 +++++++------------- 3 files changed, 18 insertions(+), 35 deletions(-) diff --git a/arch/arm64/include/asm/esr.h b/arch/arm64/include/asm/esr.h index 353fe08546cf..81606bf7d5ac 100644 --- a/arch/arm64/include/asm/esr.h +++ b/arch/arm64/include/asm/esr.h @@ -117,15 +117,9 @@ #define ESR_ELx_FSC_ACCESS (0x08) #define ESR_ELx_FSC_FAULT (0x04) #define ESR_ELx_FSC_PERM (0x0C) -#define ESR_ELx_FSC_SEA_TTW0 (0x14) -#define ESR_ELx_FSC_SEA_TTW1 (0x15) -#define ESR_ELx_FSC_SEA_TTW2 (0x16) -#define ESR_ELx_FSC_SEA_TTW3 (0x17) +#define ESR_ELx_FSC_SEA_TTW(n) (0x14 + (n)) #define ESR_ELx_FSC_SECC (0x18) -#define ESR_ELx_FSC_SECC_TTW0 (0x1c) -#define ESR_ELx_FSC_SECC_TTW1 (0x1d) -#define ESR_ELx_FSC_SECC_TTW2 (0x1e) -#define ESR_ELx_FSC_SECC_TTW3 (0x1f) +#define ESR_ELx_FSC_SECC_TTW(n) (0x1c + (n)) /* ISS field definitions for Data Aborts */ #define ESR_ELx_ISV_SHIFT (24) @@ -394,6 +388,9 @@ static inline bool esr_is_data_abort(unsigned long esr) static inline bool esr_fsc_is_translation_fault(unsigned long esr) { + /* Translation fault, level -1 */ + if ((esr & ESR_ELx_FSC) == 0b101011) + return true; return (esr & ESR_ELx_FSC_TYPE) == ESR_ELx_FSC_FAULT; } diff --git a/arch/arm64/include/asm/kvm_emulate.h b/arch/arm64/include/asm/kvm_emulate.h index b804fe832184..6f5b41c70103 100644 --- a/arch/arm64/include/asm/kvm_emulate.h +++ b/arch/arm64/include/asm/kvm_emulate.h @@ -425,15 +425,9 @@ static __always_inline bool kvm_vcpu_abt_issea(const struct kvm_vcpu *vcpu) { switch (kvm_vcpu_trap_get_fault(vcpu)) { case ESR_ELx_FSC_EXTABT: - case ESR_ELx_FSC_SEA_TTW0: - case ESR_ELx_FSC_SEA_TTW1: - case ESR_ELx_FSC_SEA_TTW2: - case ESR_ELx_FSC_SEA_TTW3: + case ESR_ELx_FSC_SEA_TTW(-1) ... ESR_ELx_FSC_SEA_TTW(3): case ESR_ELx_FSC_SECC: - case ESR_ELx_FSC_SECC_TTW0: - case ESR_ELx_FSC_SECC_TTW1: - case ESR_ELx_FSC_SECC_TTW2: - case ESR_ELx_FSC_SECC_TTW3: + case ESR_ELx_FSC_SECC_TTW(-1) ... ESR_ELx_FSC_SECC_TTW(3): return true; default: return false; diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c index 55f6455a8284..60265ede48fe 100644 --- a/arch/arm64/mm/fault.c +++ b/arch/arm64/mm/fault.c @@ -257,16 +257,14 @@ static bool is_el1_data_abort(unsigned long esr) static inline bool is_el1_permission_fault(unsigned long addr, unsigned long esr, struct pt_regs *regs) { - unsigned long fsc_type = esr & ESR_ELx_FSC_TYPE; - if (!is_el1_data_abort(esr) && !is_el1_instruction_abort(esr)) return false; - if (fsc_type == ESR_ELx_FSC_PERM) + if (esr_fsc_is_permission_fault(esr)) return true; if (is_ttbr0_addr(addr) && system_uses_ttbr0_pan()) - return fsc_type == ESR_ELx_FSC_FAULT && + return esr_fsc_is_translation_fault(esr) && (regs->pstate & PSR_PAN_BIT); return false; @@ -279,8 +277,7 @@ static bool __kprobes is_spurious_el1_translation_fault(unsigned long addr, unsigned long flags; u64 par, dfsc; - if (!is_el1_data_abort(esr) || - (esr & ESR_ELx_FSC_TYPE) != ESR_ELx_FSC_FAULT) + if (!is_el1_data_abort(esr) || !esr_fsc_is_translation_fault(esr)) return false; local_irq_save(flags); @@ -301,7 +298,7 @@ static bool __kprobes is_spurious_el1_translation_fault(unsigned long addr, * treat the translation fault as spurious. */ dfsc = FIELD_GET(SYS_PAR_EL1_FST, par); - return (dfsc & ESR_ELx_FSC_TYPE) != ESR_ELx_FSC_FAULT; + return !esr_fsc_is_translation_fault(dfsc); } static void die_kernel_fault(const char *msg, unsigned long addr, @@ -368,11 +365,6 @@ static bool is_el1_mte_sync_tag_check_fault(unsigned long esr) return false; } -static bool is_translation_fault(unsigned long esr) -{ - return (esr & ESR_ELx_FSC_TYPE) == ESR_ELx_FSC_FAULT; -} - static void __do_kernel_fault(unsigned long addr, unsigned long esr, struct pt_regs *regs) { @@ -405,7 +397,7 @@ static void __do_kernel_fault(unsigned long addr, unsigned long esr, } else if (addr < PAGE_SIZE) { msg = "NULL pointer dereference"; } else { - if (is_translation_fault(esr) && + if (esr_fsc_is_translation_fault(esr) && kfence_handle_page_fault(addr, esr & ESR_ELx_WNR, regs)) return; @@ -782,18 +774,18 @@ static const struct fault_info fault_info[] = { { do_translation_fault, SIGSEGV, SEGV_MAPERR, "level 1 translation fault" }, { do_translation_fault, SIGSEGV, SEGV_MAPERR, "level 2 translation fault" }, { do_translation_fault, SIGSEGV, SEGV_MAPERR, "level 3 translation fault" }, - { do_bad, SIGKILL, SI_KERNEL, "unknown 8" }, + { do_page_fault, SIGSEGV, SEGV_ACCERR, "level 0 access flag fault" }, { do_page_fault, SIGSEGV, SEGV_ACCERR, "level 1 access flag fault" }, { do_page_fault, SIGSEGV, SEGV_ACCERR, "level 2 access flag fault" }, { do_page_fault, SIGSEGV, SEGV_ACCERR, "level 3 access flag fault" }, - { do_bad, SIGKILL, SI_KERNEL, "unknown 12" }, + { do_page_fault, SIGSEGV, SEGV_ACCERR, "level 0 permission fault" }, { do_page_fault, SIGSEGV, SEGV_ACCERR, "level 1 permission fault" }, { do_page_fault, SIGSEGV, SEGV_ACCERR, "level 2 permission fault" }, { do_page_fault, SIGSEGV, SEGV_ACCERR, "level 3 permission fault" }, { do_sea, SIGBUS, BUS_OBJERR, "synchronous external abort" }, { do_tag_check_fault, SIGSEGV, SEGV_MTESERR, "synchronous tag check fault" }, { do_bad, SIGKILL, SI_KERNEL, "unknown 18" }, - { do_bad, SIGKILL, SI_KERNEL, "unknown 19" }, + { do_sea, SIGKILL, SI_KERNEL, "level -1 (translation table walk)" }, { do_sea, SIGKILL, SI_KERNEL, "level 0 (translation table walk)" }, { do_sea, SIGKILL, SI_KERNEL, "level 1 (translation table walk)" }, { do_sea, SIGKILL, SI_KERNEL, "level 2 (translation table walk)" }, @@ -801,7 +793,7 @@ static const struct fault_info fault_info[] = { { do_sea, SIGBUS, BUS_OBJERR, "synchronous parity or ECC error" }, // Reserved when RAS is implemented { do_bad, SIGKILL, SI_KERNEL, "unknown 25" }, { do_bad, SIGKILL, SI_KERNEL, "unknown 26" }, - { do_bad, SIGKILL, SI_KERNEL, "unknown 27" }, + { do_sea, SIGKILL, SI_KERNEL, "level -1 synchronous parity error (translation table walk)" }, // Reserved when RAS is implemented { do_sea, SIGKILL, SI_KERNEL, "level 0 synchronous parity error (translation table walk)" }, // Reserved when RAS is implemented { do_sea, SIGKILL, SI_KERNEL, "level 1 synchronous parity error (translation table walk)" }, // Reserved when RAS is implemented { do_sea, SIGKILL, SI_KERNEL, "level 2 synchronous parity error (translation table walk)" }, // Reserved when RAS is implemented @@ -815,9 +807,9 @@ static const struct fault_info fault_info[] = { { do_bad, SIGKILL, SI_KERNEL, "unknown 38" }, { do_bad, SIGKILL, SI_KERNEL, "unknown 39" }, { do_bad, SIGKILL, SI_KERNEL, "unknown 40" }, - { do_bad, SIGKILL, SI_KERNEL, "unknown 41" }, + { do_bad, SIGKILL, SI_KERNEL, "level -1 address size fault" }, { do_bad, SIGKILL, SI_KERNEL, "unknown 42" }, - { do_bad, SIGKILL, SI_KERNEL, "unknown 43" }, + { do_translation_fault, SIGSEGV, SEGV_MAPERR, "level -1 translation fault" }, { do_bad, SIGKILL, SI_KERNEL, "unknown 44" }, { do_bad, SIGKILL, SI_KERNEL, "unknown 45" }, { do_bad, SIGKILL, SI_KERNEL, "unknown 46" }, From patchwork Tue Jan 23 14:53:35 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13527553 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 433ACC47DDF for ; Tue, 23 Jan 2024 14:57:14 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=ZRoEcxiGpR0qFZdyu4yJFQUNIOq5vBIO6YRM8PnanmY=; b=dZ0tMbpX3mNZtO6BOGsw3gEl3Y H9KUIoVe04wXmgItcFJ4Q8PcrA7FiPmc4cWWEmzYytx/pVgwhSFjXEvLY9Oazdw0wG9oyMKYRx/fQ /GgksNhJ7N2WIx3KnzRxLX+8kbGT8JCsFX0QYo30dxS64P+BOd9xGOTMmR6WO7S/WKVHxdhdvWkYk dVct2EJ0NTFNUFE4cUF+5DZvtGx7Nz6ES9KH9Bd29+/wBUqAmNXWYmXhDNBJl8ZIt0CrAhY6DnK4C fk/tN1srkm04GY8geWgrCW1Fy1hwQQvL6u868BH3g/jepf52g4CG1lpfpY0LQhDRpyg178AnVbzyJ lS+BGF0w==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rSICf-00Gumn-3B; Tue, 23 Jan 2024 14:56:41 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rSIBq-00GuMK-1Y for linux-arm-kernel@bombadil.infradead.org; Tue, 23 Jan 2024 14:55:50 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=Lg92dN/Mg5BeIohbx0CLPVEaI8ExVr2RHnwHR1fA1wc=; b=Z9vsnml6pyjtgZPxmf2VHl9Tyh q0JeF/qq3rZd/jZiiQIHtn273ow1bfxUzzRwOCRXzftKlw0hTnsHfOND8FLjfyxf9PeiC/b7prcrZ WzdQfiygAo8alWohVvUZ+XEl9AJZKWOaffSyxYoXeGAog6jvuWsemnnlH9y9sVMR40odN4neznc4+ 6n5I5SF2clYnfhI9VaGaTl3oYweW3oNKY27QCFatAKS7ecWQ9wmDsGFf/AHSDAcxitWoJ+rFoRYid E/CXGvsv18a16ABoNZ8QiEgCx4zESja5F2diE70LUhDteRpuhKkv06hLA+O+EIfy9jrlvbbc6/Stt TFeTRfTA==; Received: from mail-wr1-x449.google.com ([2a00:1450:4864:20::449]) by desiato.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rSIBn-00000003vWu-0wgI for linux-arm-kernel@lists.infradead.org; Tue, 23 Jan 2024 14:55:49 +0000 Received: by mail-wr1-x449.google.com with SMTP id ffacd0b85a97d-33941f7e299so461640f8f.0 for ; Tue, 23 Jan 2024 06:55:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1706021745; x=1706626545; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=Lg92dN/Mg5BeIohbx0CLPVEaI8ExVr2RHnwHR1fA1wc=; b=Azgq20WUPJlQ0nMGGLvtO1fIQUatszvsA9j4SgV4q0C6BFa3ZIE5MEED7cV2jKhdkE 2MxfiJya4+aMKNYw6h89PgSU4vu7EiIkOMr/5QL8o4Fu8E+Gwjbc5c8EnHy3Mynz+4WX YCA+ZlNyJTciXQAauCzvzF02Z2GDqrbBN++jh4Ln1hKOdkm8x9UxPqglTtBwXJ6hxbRA ESgYxXr+o9OsoZrgstUYZhMBs2bbcKMMZZ7x55JvnGpVdPy8g2ndDuLqEkppEIdjinI4 bDPVoFO5gCX4Ak0zS3dKmDQAjOXW13hytj9aou0wX/acdxfPuBZakSOMDnp0Wf3dV+ZQ OORA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706021745; x=1706626545; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Lg92dN/Mg5BeIohbx0CLPVEaI8ExVr2RHnwHR1fA1wc=; b=s7xmAMkF6dkWntTwo8VzuPmaC/r3GaRmLGFAmowlMGGg4aFRYtLX2rm/JMb24elSkd vpus1REACgL6P59spMmGSCDfSLXJx1KDGUpCwTfmvUVEpvaP76FuNHQ6570MWi8mmwMQ dHY4t9ct8/Kbjku1zkadaWtyQwvu0L/NTAwARp6Wd8s3+RlRQtm7wHxOaZW6uta/dSdl uhmxEEeXTtEErzeJgJZYTLKTZgH7hzb2o+3/LJWGhZkWqoaLU23JW+s9w222iadmlyz2 +QiKpWeZ4+L9Gs0NnR4e4HSkz1yP2ErhXCcStwvs5EhlVz0R+VVqE1vGtXw/I2VyogBA G1xA== X-Gm-Message-State: AOJu0YzPMNydPOJAtrRTBt9lzh/rLmrZDzNa0iRHYws2L3ZTiUUFBmih VusBmwIqZxtV++DBpAjr8BqufdbyWkLf+hyXKjS6GqNVEZ9W5zO2uNAX4Psuwbux0w+x1OCUMkK kODIl5LkDOWFyt14jRsyW44P9kwMruIK3Dh0/fVMBDK0OoIpeTr1KsRsmYq/4gVQX4I0TylXPgZ yvNA1B8aoi4coq503rb2ZweOJn0Vv02rv0TYTHubdv X-Google-Smtp-Source: AGHT+IHIMpVBHxRfh8l7ePsmWtunUKhYFUPAoC/1gLPv8BwfMD/cacdM/Fm9FDTwuy/d9fd0qYHr7JFe X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:6000:1e12:b0:339:2a23:8e69 with SMTP id bj18-20020a0560001e1200b003392a238e69mr16226wrb.5.1706021745273; Tue, 23 Jan 2024 06:55:45 -0800 (PST) Date: Tue, 23 Jan 2024 15:53:35 +0100 In-Reply-To: <20240123145258.1462979-52-ardb+git@google.com> Mime-Version: 1.0 References: <20240123145258.1462979-52-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=3974; i=ardb@kernel.org; h=from:subject; bh=gecEIJDd8zMAkroaF8QQO7Ni/qDxVGJunsWUBre+5z8=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIXX9pU8v768L3mhcxzJ1j7wm897tJ+3z2HRDp7lLegrdP uX8+eKTjlIWBjEOBlkxRRaB2X/f7Tw9UarWeZYszBxWJpAhDFycAjCRFd8YGb5dyuR2KmT7UVIv pWW8/bo1w8qDb9+asYqzaKa9Ucq81c/wP/L/bb6SsPOl0eXM/u6nPNT2CHw59avjlt+GLV5Bjcm cvAA= X-Mailer: git-send-email 2.43.0.429.g432eaa2c6b-goog Message-ID: <20240123145258.1462979-88-ardb+git@google.com> Subject: [PATCH v7 36/50] arm64: mm: Wire up TCR.DS bit to PTE shareability fields From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240123_145547_426977_AE8E0C6D X-CRM114-Status: GOOD ( 15.96 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel When LPA2 is enabled, bits 8 and 9 of page and block descriptors become part of the output address instead of carrying shareability attributes for the region in question. So avoid setting these bits if TCR.DS == 1, which means LPA2 is enabled. Signed-off-by: Ard Biesheuvel --- arch/arm64/Kconfig | 4 ++++ arch/arm64/include/asm/pgtable-hwdef.h | 1 + arch/arm64/include/asm/pgtable-prot.h | 16 ++++++++++++++-- arch/arm64/mm/mmap.c | 4 ++++ arch/arm64/mm/proc.S | 2 ++ 5 files changed, 25 insertions(+), 2 deletions(-) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index aa7c1d435139..8c2c36fffcf5 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -1377,6 +1377,10 @@ config ARM64_PA_BITS default 48 if ARM64_PA_BITS_48 default 52 if ARM64_PA_BITS_52 +config ARM64_LPA2 + def_bool y + depends on ARM64_PA_BITS_52 && !ARM64_64K_PAGES + choice prompt "Endianness" default CPU_LITTLE_ENDIAN diff --git a/arch/arm64/include/asm/pgtable-hwdef.h b/arch/arm64/include/asm/pgtable-hwdef.h index e4944d517c99..b770f98fc0b5 100644 --- a/arch/arm64/include/asm/pgtable-hwdef.h +++ b/arch/arm64/include/asm/pgtable-hwdef.h @@ -284,6 +284,7 @@ #define TCR_E0PD1 (UL(1) << 56) #define TCR_TCMA0 (UL(1) << 57) #define TCR_TCMA1 (UL(1) << 58) +#define TCR_DS (UL(1) << 59) /* * TTBR. diff --git a/arch/arm64/include/asm/pgtable-prot.h b/arch/arm64/include/asm/pgtable-prot.h index 63ced9ccec21..dd9ee67d1d87 100644 --- a/arch/arm64/include/asm/pgtable-prot.h +++ b/arch/arm64/include/asm/pgtable-prot.h @@ -30,8 +30,8 @@ #define _PROT_DEFAULT (PTE_TYPE_PAGE | PTE_AF | PTE_SHARED) #define _PROT_SECT_DEFAULT (PMD_TYPE_SECT | PMD_SECT_AF | PMD_SECT_S) -#define PROT_DEFAULT (_PROT_DEFAULT | PTE_MAYBE_NG) -#define PROT_SECT_DEFAULT (_PROT_SECT_DEFAULT | PMD_MAYBE_NG) +#define PROT_DEFAULT (PTE_TYPE_PAGE | PTE_MAYBE_NG | PTE_MAYBE_SHARED | PTE_AF) +#define PROT_SECT_DEFAULT (PMD_TYPE_SECT | PMD_MAYBE_NG | PMD_MAYBE_SHARED | PMD_SECT_AF) #define PROT_DEVICE_nGnRnE (PROT_DEFAULT | PTE_PXN | PTE_UXN | PTE_WRITE | PTE_ATTRINDX(MT_DEVICE_nGnRnE)) #define PROT_DEVICE_nGnRE (PROT_DEFAULT | PTE_PXN | PTE_UXN | PTE_WRITE | PTE_ATTRINDX(MT_DEVICE_nGnRE)) @@ -67,7 +67,19 @@ extern bool arm64_use_ng_mappings; #define PTE_MAYBE_NG (arm64_use_ng_mappings ? PTE_NG : 0) #define PMD_MAYBE_NG (arm64_use_ng_mappings ? PMD_SECT_NG : 0) +#ifndef CONFIG_ARM64_LPA2 #define lpa2_is_enabled() false +#define PTE_MAYBE_SHARED PTE_SHARED +#define PMD_MAYBE_SHARED PMD_SECT_S +#else +static inline bool __pure lpa2_is_enabled(void) +{ + return read_tcr() & TCR_DS; +} + +#define PTE_MAYBE_SHARED (lpa2_is_enabled() ? 0 : PTE_SHARED) +#define PMD_MAYBE_SHARED (lpa2_is_enabled() ? 0 : PMD_SECT_S) +#endif /* * If we have userspace only BTI we don't want to mark kernel pages diff --git a/arch/arm64/mm/mmap.c b/arch/arm64/mm/mmap.c index 645fe60d000f..642bdf908b22 100644 --- a/arch/arm64/mm/mmap.c +++ b/arch/arm64/mm/mmap.c @@ -73,6 +73,10 @@ static int __init adjust_protection_map(void) protection_map[VM_EXEC | VM_SHARED] = PAGE_EXECONLY; } + if (lpa2_is_enabled()) + for (int i = 0; i < ARRAY_SIZE(protection_map); i++) + pgprot_val(protection_map[i]) &= ~PTE_SHARED; + return 0; } arch_initcall(adjust_protection_map); diff --git a/arch/arm64/mm/proc.S b/arch/arm64/mm/proc.S index 6e1b2bc41a9f..7c46f8cfd6ae 100644 --- a/arch/arm64/mm/proc.S +++ b/arch/arm64/mm/proc.S @@ -466,6 +466,7 @@ alternative_else_nop_endif */ #define PTE_MAYBE_NG 0 +#define PTE_MAYBE_SHARED 0 mov_q x0, PIE_E0 msr REG_PIRE0_EL1, x0 @@ -473,6 +474,7 @@ alternative_else_nop_endif msr REG_PIR_EL1, x0 #undef PTE_MAYBE_NG +#undef PTE_MAYBE_SHARED mov x0, TCR2_EL1x_PIE msr REG_TCR2_EL1, x0 From patchwork Tue Jan 23 14:53:36 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13527555 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id BE831C47258 for ; Tue, 23 Jan 2024 14:57:20 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=VBh6X38Vlu66A5QeuWR4q7zr243smSAyOEj3cpAH3aA=; b=DgHjEDApe69zjpK4I0TXv3EDih LNU9PWyRubEzY4GxFt7+N8YZ5ywFkEZzCsc0PyX18QLEBCpvnr/AnVp+Nu1Nj7zaqiisuYPABMOOc d+HFL3Ky5KDhIumfrCSjTx4M5b9Pdd0lb36unHgKCxTOcJm9i+rKR7zWO8lThVxG+GsUC/fSGMyC7 aQSEwYOcZkX1HgaBD/vlyYe2aDH/LxLM9fHEa9NrnSlazBVDWGebirwEyGVAeV9+R2GRF/zUI/gOy eiHXxfmhlK4vTUNa5tkc1ns4fAVatdlG+urma2so6bYRl4ULtquSM+6hwWe26yaMKLpTg94js2ZNL 4xI9eR5w==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rSICm-00Gurj-2O; Tue, 23 Jan 2024 14:56:48 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rSIBt-00GuO2-2V for linux-arm-kernel@bombadil.infradead.org; Tue, 23 Jan 2024 14:55:53 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=E4wrsZgepcPmsNgQem185zgjGPeGrdho/+cjw/u+q8w=; b=HbGdudrbO62iuksk3/QVBS+/x0 Sb9Uubh0kFTAzj2d8cX1WJix+gOwuvkb0YkBCtNxtQIkUQW317oW6R5apPlsv7D5U/h0KmbYaxcBM VItWzVsqFlJ6mr+yu24OK6ujiW1/dwKs9VLIFmdeX8vNZANkwDdLOYXm2vdWAjbeYLYI/Nab3tenF IftebdquYJQKaUMfytD0UXgsXEoDiS+atjb5O/vB1sVjgDG3lNAYJcvMij+z67LeGTURLbCnKWQd5 p+kaLhYTGXYJAX8Ka+MCEH9O1WiZCMq9vzutiBYnWmelcYqUgGveQYLugcX35jZddlTB8KCVEE7Ek BI4jNWJA==; Received: from mail-yb1-xb4a.google.com ([2607:f8b0:4864:20::b4a]) by desiato.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rSIBp-00000003vXt-3IgG for linux-arm-kernel@lists.infradead.org; Tue, 23 Jan 2024 14:55:51 +0000 Received: by mail-yb1-xb4a.google.com with SMTP id 3f1490d57ef6-dbdb46770d7so6226098276.1 for ; Tue, 23 Jan 2024 06:55:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1706021747; x=1706626547; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=E4wrsZgepcPmsNgQem185zgjGPeGrdho/+cjw/u+q8w=; b=mFisMEDkh3q/TNu815oJukyBOhL1LqvQ4GmD+XBPjXw1hzHY7EDl95Pfeskx/Ac+7h Dg3oHzBwD68AslRoI+MxgbxctKCTZnGUag7KCanCaWmaYoMg5F1VpKGv1gSoDOLc4d2M hwxZM0DjmcSCjDxxOjTFXKa6DsIdYnNjeXBWVmqzpdPVJ2Y7d/RnY3bFAntRHo0AEkoS NzJdhYUkwjPIEifFHdDAos8CZejvaTW2bAlvsAzNLMRpfpXK4dnmhaq1yB6fXGB10IRC hAE8hkAxwEep9eEwocaRjxWawk4EXFNRlIDDE4iMyEqerhFNs0cHYppzO83kz7ddETLY v09Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706021747; x=1706626547; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=E4wrsZgepcPmsNgQem185zgjGPeGrdho/+cjw/u+q8w=; b=q/6Ddg0ItDfs+PwaYK5OoM5dT1K6xhqxmSJ19+5ofnT7t4WHdQ5wS+dwKE4pZgClm0 Mar6296XSqoBZEp/YBzzwNW+izb9oPgD58OnasxvaOzOwHmob0WRUwd+4s8sQFKuSK8v 8HNHzEwqY6J21B1lYQXj8Y009GxdICPCpr3BBwUDPLmaeoRt6SPA0QPcT8lLSq9H1qtz pV5QiGLiMljFdN99nlyljra7fhL/GPI08RAn0U6oExozVRyrID/9V7fxNaXpQfu3Dj4l x6RLszBI0gq0MFFDNF4/yCSwl5mJqMSVtuejyiNRW8dh5TquJkT27SQu/3htRFmBp1bj oTsA== X-Gm-Message-State: AOJu0Yyy5f6mGxDkuCZmGYSseFhBKnESn+pEHGEi7B2V3C60Hq5eFmdk dc2Zefn5ndxh0I7JsmbT2J0h7saSRnp9ofAmxoiB+pp13V3zateW9lBeYwNdUh4htgr0+gfZas4 Vy/130p721yfj6hx7aq+fgaSwwhcNXyu+VXOw35EFkNnqxC24vkRHby2hpHxRWCv1FnAZdK13ZW 2iYaXEZxO8lJQMhmUFZxtRZdFdWcabQSD7UY5oTkqu X-Google-Smtp-Source: AGHT+IFvSuEtlYS8p2iotss0uTEq41hV2hT/c9lhW+lmYKUjt2wpzhhVniQkaSC3dQZ7YpEAUZUHMPWu X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a25:8152:0:b0:dbc:c697:63bd with SMTP id j18-20020a258152000000b00dbcc69763bdmr2613904ybm.0.1706021747661; Tue, 23 Jan 2024 06:55:47 -0800 (PST) Date: Tue, 23 Jan 2024 15:53:36 +0100 In-Reply-To: <20240123145258.1462979-52-ardb+git@google.com> Mime-Version: 1.0 References: <20240123145258.1462979-52-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=4645; i=ardb@kernel.org; h=from:subject; bh=b423qQkbUAHh+y9gBE+yrDBhYvk6YjkRX7q73L9IZXA=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIXX9pc/bay/JTZyh7F8/8794gVqCDNvJw+yTP9598HHCe rbqv9/YOkpZGMQ4GGTFFFkEZv99t/P0RKla51myMHNYmUCGMHBxCsBEBD4z/BVZGmU5X7L5ysn9 Gn8rnQ5YHP3FychVeL90qs5LuZC15QcZ/llOanebuIj9t4/Y4+d6rAsO5NcddTk7qWras4/LPz0 +eZwZAA== X-Mailer: git-send-email 2.43.0.429.g432eaa2c6b-goog Message-ID: <20240123145258.1462979-89-ardb+git@google.com> Subject: [PATCH v7 37/50] arm64: mm: Add LPA2 support to phys<->pte conversion routines From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240123_145549_958241_90DCA0D0 X-CRM114-Status: GOOD ( 15.40 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel In preparation for enabling LPA2 support, introduce the mask values for converting between physical addresses and their representations in a page table descriptor. While at it, move the pte_to_phys asm macro into its only user, so that we can freely modify it to use its input value register as a temp register. For LPA2, the PTE_ADDR_MASK contains two non-adjacent sequences of zero bits, which means it no longer fits into the immediate field of an ordinary ALU instruction. So let's redefine it to include the bits in between as well, and only use it when converting from physical address to PTE representation, where the distinction does not matter. Also update the name accordingly to emphasize this. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/assembler.h | 16 ++-------------- arch/arm64/include/asm/pgtable-hwdef.h | 10 +++++++--- arch/arm64/include/asm/pgtable.h | 5 +++-- arch/arm64/mm/proc.S | 8 ++++++++ 4 files changed, 20 insertions(+), 19 deletions(-) diff --git a/arch/arm64/include/asm/assembler.h b/arch/arm64/include/asm/assembler.h index 68a99b116256..7eedcb36ebe0 100644 --- a/arch/arm64/include/asm/assembler.h +++ b/arch/arm64/include/asm/assembler.h @@ -612,25 +612,13 @@ alternative_endif .macro phys_to_pte, pte, phys #ifdef CONFIG_ARM64_PA_BITS_52 - /* - * We assume \phys is 64K aligned and this is guaranteed by only - * supporting this configuration with 64K pages. - */ - orr \pte, \phys, \phys, lsr #36 - and \pte, \pte, #PTE_ADDR_MASK + orr \pte, \phys, \phys, lsr #PTE_ADDR_HIGH_SHIFT + and \pte, \pte, #PHYS_TO_PTE_ADDR_MASK #else mov \pte, \phys #endif .endm - .macro pte_to_phys, phys, pte - and \phys, \pte, #PTE_ADDR_MASK -#ifdef CONFIG_ARM64_PA_BITS_52 - orr \phys, \phys, \phys, lsl #PTE_ADDR_HIGH_SHIFT - and \phys, \phys, GENMASK_ULL(PHYS_MASK_SHIFT - 1, PAGE_SHIFT) -#endif - .endm - /* * tcr_clear_errata_bits - Clear TCR bits that trigger an errata on this CPU. */ diff --git a/arch/arm64/include/asm/pgtable-hwdef.h b/arch/arm64/include/asm/pgtable-hwdef.h index b770f98fc0b5..4426f48f2ae0 100644 --- a/arch/arm64/include/asm/pgtable-hwdef.h +++ b/arch/arm64/include/asm/pgtable-hwdef.h @@ -155,13 +155,17 @@ #define PTE_PXN (_AT(pteval_t, 1) << 53) /* Privileged XN */ #define PTE_UXN (_AT(pteval_t, 1) << 54) /* User XN */ -#define PTE_ADDR_LOW (((_AT(pteval_t, 1) << (48 - PAGE_SHIFT)) - 1) << PAGE_SHIFT) +#define PTE_ADDR_LOW (((_AT(pteval_t, 1) << (50 - PAGE_SHIFT)) - 1) << PAGE_SHIFT) #ifdef CONFIG_ARM64_PA_BITS_52 +#ifdef CONFIG_ARM64_64K_PAGES #define PTE_ADDR_HIGH (_AT(pteval_t, 0xf) << 12) -#define PTE_ADDR_MASK (PTE_ADDR_LOW | PTE_ADDR_HIGH) #define PTE_ADDR_HIGH_SHIFT 36 +#define PHYS_TO_PTE_ADDR_MASK (PTE_ADDR_LOW | PTE_ADDR_HIGH) #else -#define PTE_ADDR_MASK PTE_ADDR_LOW +#define PTE_ADDR_HIGH (_AT(pteval_t, 0x3) << 8) +#define PTE_ADDR_HIGH_SHIFT 42 +#define PHYS_TO_PTE_ADDR_MASK GENMASK_ULL(49, 8) +#endif #endif /* diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 522c21348ae8..61de7b1516bc 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -80,15 +80,16 @@ extern unsigned long empty_zero_page[PAGE_SIZE / sizeof(unsigned long)]; #ifdef CONFIG_ARM64_PA_BITS_52 static inline phys_addr_t __pte_to_phys(pte_t pte) { + pte_val(pte) &= ~PTE_MAYBE_SHARED; return (pte_val(pte) & PTE_ADDR_LOW) | ((pte_val(pte) & PTE_ADDR_HIGH) << PTE_ADDR_HIGH_SHIFT); } static inline pteval_t __phys_to_pte_val(phys_addr_t phys) { - return (phys | (phys >> PTE_ADDR_HIGH_SHIFT)) & PTE_ADDR_MASK; + return (phys | (phys >> PTE_ADDR_HIGH_SHIFT)) & PHYS_TO_PTE_ADDR_MASK; } #else -#define __pte_to_phys(pte) (pte_val(pte) & PTE_ADDR_MASK) +#define __pte_to_phys(pte) (pte_val(pte) & PTE_ADDR_LOW) #define __phys_to_pte_val(phys) (phys) #endif diff --git a/arch/arm64/mm/proc.S b/arch/arm64/mm/proc.S index 7c46f8cfd6ae..d03434b7bca5 100644 --- a/arch/arm64/mm/proc.S +++ b/arch/arm64/mm/proc.S @@ -205,6 +205,14 @@ SYM_FUNC_ALIAS(__pi_idmap_cpu_replace_ttbr1, idmap_cpu_replace_ttbr1) .pushsection ".idmap.text", "a" + .macro pte_to_phys, phys, pte + and \phys, \pte, #PTE_ADDR_LOW +#ifdef CONFIG_ARM64_PA_BITS_52 + and \pte, \pte, #PTE_ADDR_HIGH + orr \phys, \phys, \pte, lsl #PTE_ADDR_HIGH_SHIFT +#endif + .endm + .macro kpti_mk_tbl_ng, type, num_entries add end_\type\()p, cur_\type\()p, #\num_entries * 8 .Ldo_\type: From patchwork Tue Jan 23 14:53:37 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13527557 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id BF3C5C47DDB for ; Tue, 23 Jan 2024 14:57:28 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=meKRUUnjwbg6YKxtoMybmP64Rh9RnQ0Cc4D9q7Lx4x8=; b=icsPy7Hkr2Ldkahd9XD/ZPZmBM 4MeOQrC5smxzRl8dYlHnu1lJcpxPOKqmZG5T3/nbQ5keCOzbKMy7aCSAAVVqE46089Rhde56ICofz gErUNm3j1zzOnq6ACNEyoZU+om9vQqNjpYnpr4m0Z9iiIIwVetihPsGuhdZj69LpTk+lLdEz6PP1X rUVEEBMmVO5+74gxE+wo6mYxvETMtiDzhJOBksnNzKpLAOHoij9Du+uvSjD6gv/ha8BSPFwcNjAm1 wO4vjBUqYHOE5kyGbKAX3Y3fmngrDX4Fg0d6Rnu23eJc1rSSXibDUqTx9+qjUyDkPb0zlHA9Pu5aM ZusAUkxw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rSICr-00GuwR-2u; Tue, 23 Jan 2024 14:56:53 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rSIBw-00GuPM-09 for linux-arm-kernel@bombadil.infradead.org; Tue, 23 Jan 2024 14:55:56 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=ifBLHe5g1PzHzakETbF1onEeV1H/9CSx5KQOWcCbVXw=; b=Q2gMi7G6QCy74OS2oZAQ4KYJFf uZnI6YVvCxN4RH8u+9+h/8sCLUpYnWlRtLvQ9rXoA/c7GiCz/YRo2hJi8ESHDXHfQc1IdOSqONwbv NaVlWmI+AgJK+n0u/WRqi+aP6DVPTf4UEH4jSCGNHm76KQOnruNPp2wB1pzk6kWmtlxAHyWbkvl+t 3gpjeM344LMhmU5QHhvFmiXn2bjoSBnPSW9NlVti49YhkxE4jbcsX8WLmgpaa5JR48JIB3lGjmBA+ 74OSoSuDkBCpKSLhsW/yqc569h0sAIbrFCtHla6ESar3KWDgaUiOtBUzMokBr68WiVy+baGaQDlT4 EtAlxYpA==; Received: from mail-yb1-xb4a.google.com ([2607:f8b0:4864:20::b4a]) by desiato.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rSIBs-00000003vYN-39kF for linux-arm-kernel@lists.infradead.org; Tue, 23 Jan 2024 14:55:54 +0000 Received: by mail-yb1-xb4a.google.com with SMTP id 3f1490d57ef6-dc37b0ac33cso691432276.2 for ; Tue, 23 Jan 2024 06:55:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1706021750; x=1706626550; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=ifBLHe5g1PzHzakETbF1onEeV1H/9CSx5KQOWcCbVXw=; b=BIWC+AIYFbrrI8/CBllrkdWIPXLI3klfMLu7nkGIK/At7IvjpR7n+1+pyQact69EiL FP4qEcVmyaJwI6/4J36U1Pbl8T6ndc/7bVhlL5/BflOMvVKY93WlL2Yh7siVXCRtsxVQ K6ykLOWj70mpLFXNTU8jTbmu5nzqpTUuX04EAE9cCeneQ/cgDdH2XxpTHpNDf7VmAtFh xkwI0fxhl9bq5PMlkkzCEeIGNNTUjmFWRYwA58a3CuZU3vzv3V7jVOvL/Wn9JTbWQso6 L5SXIMWoj7teSwvrP1WFIhfWW5FNV9zV9Q71baQvzytYQBHWHEieI1rRG+2ocfXvBgt4 MOeA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706021750; x=1706626550; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=ifBLHe5g1PzHzakETbF1onEeV1H/9CSx5KQOWcCbVXw=; b=Sc3G35nShaOexd37uinkwd5Q00g9tIIVSbN2+x1CVWnEJzlelKqGLStPvQOCmwuiN2 ulbL2DyXFbwnuLxYMx065s/GpRgXHhKt+vA0NBQAre9Te46DtGFKgDd1tabVcyBYWj2a ZHxZRH29/oC2yesBIAcmR4ylJmsIsvAb2IGZChcU2moObtlVzdA0Pri72VN7agNHD7D5 JtP7zrcI41RtXrSWfRBVjNffcKZgcPV9pPG3fBtmgE13Y+W8tGxsAPbihfqlFCNlKJFu UUdSVARscAtsPiwH/gB/pCrdHLjeBs1ulf9Uo3tfDHNBb4P6c3hlS4SL7ZzHW81QNoBp I4XQ== X-Gm-Message-State: AOJu0Yx3WOW8W6IhpWd4QIbpoAnzyJiTAnwu3XiORydGd/6/wFnLuw/+ NzQFc6hSMuGBBIYg7IigUBitoN3bEu2Et9d4rBIviSTxawmDnBRqoruPTBIAxg00DSqcpyQtqvP EefcSgk4H8shH7oeXQX706A1+UiF/nx6gph3XtBL/E+1oeP3rZ2n+4nbsofBNJuC5gZACP17fjC tyg2c0+tvNrVtn8bUIvukM0uhpA2IRelU93xAwVkrr X-Google-Smtp-Source: AGHT+IHcK/Vn5uGYZhmW/0KLaPhie6R4r/lFrkvTL+2o3kQh07NhoKzeGMPRZVTQMihY+9yjMCtQp0I9 X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a5b:ac8:0:b0:dc2:294f:42b8 with SMTP id a8-20020a5b0ac8000000b00dc2294f42b8mr2685563ybr.5.1706021750063; Tue, 23 Jan 2024 06:55:50 -0800 (PST) Date: Tue, 23 Jan 2024 15:53:37 +0100 In-Reply-To: <20240123145258.1462979-52-ardb+git@google.com> Mime-Version: 1.0 References: <20240123145258.1462979-52-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=10782; i=ardb@kernel.org; h=from:subject; bh=YMlyMGYyoOoL8M1NPIZCvTI2M0HDfJMuqGOxpqfRhQ4=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIXX9pS/b/aSSuyYvkd2+9t1asw1M5mInuLTd3RxfZxuus N7IqGPfUcrCIMbBICumyCIw+++7nacnStU6z5KFmcPKBDKEgYtTACZSEsbIcF5I1LbGpC8gvlhU cwJ767Lp3Evrkqz2bt/+SY933zHvIIZ/9r51zIevenG92+xfd9X23meH4xfs2Xc7LG65XRXDsvk zAwA= X-Mailer: git-send-email 2.43.0.429.g432eaa2c6b-goog Message-ID: <20240123145258.1462979-90-ardb+git@google.com> Subject: [PATCH v7 38/50] arm64: mm: Add definitions to support 5 levels of paging From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240123_145552_950026_95A3F664 X-CRM114-Status: GOOD ( 23.84 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel Add the required types and descriptor accessors to support 5 levels of paging in the common code. This is one of the prerequisites for supporting 52-bit virtual addressing with 4k pages. Note that this does not cover the code that handles kernel mappings or the fixmap. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/pgalloc.h | 41 ++++++++++ arch/arm64/include/asm/pgtable-hwdef.h | 22 +++++- arch/arm64/include/asm/pgtable-types.h | 6 ++ arch/arm64/include/asm/pgtable.h | 82 +++++++++++++++++++- arch/arm64/mm/mmu.c | 31 +++++++- arch/arm64/mm/pgd.c | 15 +++- 6 files changed, 188 insertions(+), 9 deletions(-) diff --git a/arch/arm64/include/asm/pgalloc.h b/arch/arm64/include/asm/pgalloc.h index 237224484d0f..cae8c648f462 100644 --- a/arch/arm64/include/asm/pgalloc.h +++ b/arch/arm64/include/asm/pgalloc.h @@ -60,6 +60,47 @@ static inline void __p4d_populate(p4d_t *p4dp, phys_addr_t pudp, p4dval_t prot) } #endif /* CONFIG_PGTABLE_LEVELS > 3 */ +#if CONFIG_PGTABLE_LEVELS > 4 + +static inline void __pgd_populate(pgd_t *pgdp, phys_addr_t p4dp, pgdval_t prot) +{ + if (pgtable_l5_enabled()) + set_pgd(pgdp, __pgd(__phys_to_pgd_val(p4dp) | prot)); +} + +static inline void pgd_populate(struct mm_struct *mm, pgd_t *pgdp, p4d_t *p4dp) +{ + pgdval_t pgdval = PGD_TYPE_TABLE; + + pgdval |= (mm == &init_mm) ? PGD_TABLE_UXN : PGD_TABLE_PXN; + __pgd_populate(pgdp, __pa(p4dp), pgdval); +} + +static inline p4d_t *p4d_alloc_one(struct mm_struct *mm, unsigned long addr) +{ + gfp_t gfp = GFP_PGTABLE_USER; + + if (mm == &init_mm) + gfp = GFP_PGTABLE_KERNEL; + return (p4d_t *)get_zeroed_page(gfp); +} + +static inline void p4d_free(struct mm_struct *mm, p4d_t *p4d) +{ + if (!pgtable_l5_enabled()) + return; + BUG_ON((unsigned long)p4d & (PAGE_SIZE-1)); + free_page((unsigned long)p4d); +} + +#define __p4d_free_tlb(tlb, p4d, addr) p4d_free((tlb)->mm, p4d) +#else +static inline void __pgd_populate(pgd_t *pgdp, phys_addr_t p4dp, pgdval_t prot) +{ + BUILD_BUG(); +} +#endif /* CONFIG_PGTABLE_LEVELS > 4 */ + extern pgd_t *pgd_alloc(struct mm_struct *mm); extern void pgd_free(struct mm_struct *mm, pgd_t *pgdp); diff --git a/arch/arm64/include/asm/pgtable-hwdef.h b/arch/arm64/include/asm/pgtable-hwdef.h index 4426f48f2ae0..ef207a0d4f0d 100644 --- a/arch/arm64/include/asm/pgtable-hwdef.h +++ b/arch/arm64/include/asm/pgtable-hwdef.h @@ -26,10 +26,10 @@ #define ARM64_HW_PGTABLE_LEVELS(va_bits) (((va_bits) - 4) / (PAGE_SHIFT - 3)) /* - * Size mapped by an entry at level n ( 0 <= n <= 3) + * Size mapped by an entry at level n ( -1 <= n <= 3) * We map (PAGE_SHIFT - 3) at all translation levels and PAGE_SHIFT bits * in the final page. The maximum number of translation levels supported by - * the architecture is 4. Hence, starting at level n, we have further + * the architecture is 5. Hence, starting at level n, we have further * ((4 - n) - 1) levels of translation excluding the offset within the page. * So, the total number of bits mapped by an entry at level n is : * @@ -62,9 +62,16 @@ #define PTRS_PER_PUD (1 << (PAGE_SHIFT - 3)) #endif +#if CONFIG_PGTABLE_LEVELS > 4 +#define P4D_SHIFT ARM64_HW_PGTABLE_LEVEL_SHIFT(0) +#define P4D_SIZE (_AC(1, UL) << P4D_SHIFT) +#define P4D_MASK (~(P4D_SIZE-1)) +#define PTRS_PER_P4D (1 << (PAGE_SHIFT - 3)) +#endif + /* * PGDIR_SHIFT determines the size a top-level page table entry can map - * (depending on the configuration, this level can be 0, 1 or 2). + * (depending on the configuration, this level can be -1, 0, 1 or 2). */ #define PGDIR_SHIFT ARM64_HW_PGTABLE_LEVEL_SHIFT(4 - CONFIG_PGTABLE_LEVELS) #define PGDIR_SIZE (_AC(1, UL) << PGDIR_SHIFT) @@ -87,6 +94,15 @@ /* * Hardware page table definitions. * + * Level -1 descriptor (PGD). + */ +#define PGD_TYPE_TABLE (_AT(pgdval_t, 3) << 0) +#define PGD_TABLE_BIT (_AT(pgdval_t, 1) << 1) +#define PGD_TYPE_MASK (_AT(pgdval_t, 3) << 0) +#define PGD_TABLE_PXN (_AT(pgdval_t, 1) << 59) +#define PGD_TABLE_UXN (_AT(pgdval_t, 1) << 60) + +/* * Level 0 descriptor (P4D). */ #define P4D_TYPE_TABLE (_AT(p4dval_t, 3) << 0) diff --git a/arch/arm64/include/asm/pgtable-types.h b/arch/arm64/include/asm/pgtable-types.h index b8f158ae2527..6d6d4065b0cb 100644 --- a/arch/arm64/include/asm/pgtable-types.h +++ b/arch/arm64/include/asm/pgtable-types.h @@ -36,6 +36,12 @@ typedef struct { pudval_t pud; } pud_t; #define __pud(x) ((pud_t) { (x) } ) #endif +#if CONFIG_PGTABLE_LEVELS > 4 +typedef struct { p4dval_t p4d; } p4d_t; +#define p4d_val(x) ((x).p4d) +#define __p4d(x) ((p4d_t) { (x) } ) +#endif + typedef struct { pgdval_t pgd; } pgd_t; #define pgd_val(x) ((x).pgd) #define __pgd(x) ((pgd_t) { (x) } ) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 61de7b1516bc..7eb2b933ed3c 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -808,7 +808,6 @@ static inline pud_t *p4d_pgtable(p4d_t p4d) #else #define p4d_page_paddr(p4d) ({ BUILD_BUG(); 0;}) -#define pgd_page_paddr(pgd) ({ BUILD_BUG(); 0;}) /* Match pud_offset folding in */ #define pud_set_fixmap(addr) NULL @@ -819,6 +818,87 @@ static inline pud_t *p4d_pgtable(p4d_t p4d) #endif /* CONFIG_PGTABLE_LEVELS > 3 */ +#if CONFIG_PGTABLE_LEVELS > 4 + +static __always_inline bool pgtable_l5_enabled(void) +{ + if (!alternative_has_cap_likely(ARM64_ALWAYS_BOOT)) + return vabits_actual == VA_BITS; + return alternative_has_cap_unlikely(ARM64_HAS_VA52); +} + +static inline bool mm_p4d_folded(const struct mm_struct *mm) +{ + return !pgtable_l5_enabled(); +} +#define mm_p4d_folded mm_p4d_folded + +#define p4d_ERROR(e) \ + pr_err("%s:%d: bad p4d %016llx.\n", __FILE__, __LINE__, p4d_val(e)) + +#define pgd_none(pgd) (pgtable_l5_enabled() && !pgd_val(pgd)) +#define pgd_bad(pgd) (pgtable_l5_enabled() && !(pgd_val(pgd) & 2)) +#define pgd_present(pgd) (!pgd_none(pgd)) + +static inline void set_pgd(pgd_t *pgdp, pgd_t pgd) +{ + if (in_swapper_pgdir(pgdp)) { + set_swapper_pgd(pgdp, __pgd(pgd_val(pgd))); + return; + } + + WRITE_ONCE(*pgdp, pgd); + dsb(ishst); + isb(); +} + +static inline void pgd_clear(pgd_t *pgdp) +{ + if (pgtable_l5_enabled()) + set_pgd(pgdp, __pgd(0)); +} + +static inline phys_addr_t pgd_page_paddr(pgd_t pgd) +{ + return __pgd_to_phys(pgd); +} + +#define p4d_index(addr) (((addr) >> P4D_SHIFT) & (PTRS_PER_P4D - 1)) + +static inline p4d_t *pgd_to_folded_p4d(pgd_t *pgdp, unsigned long addr) +{ + return (p4d_t *)PTR_ALIGN_DOWN(pgdp, PAGE_SIZE) + p4d_index(addr); +} + +static inline phys_addr_t p4d_offset_phys(pgd_t *pgdp, unsigned long addr) +{ + BUG_ON(!pgtable_l5_enabled()); + + return pgd_page_paddr(READ_ONCE(*pgdp)) + p4d_index(addr) * sizeof(p4d_t); +} + +static inline +p4d_t *p4d_offset_lockless(pgd_t *pgdp, pgd_t pgd, unsigned long addr) +{ + if (!pgtable_l5_enabled()) + return pgd_to_folded_p4d(pgdp, addr); + return (p4d_t *)__va(pgd_page_paddr(pgd)) + p4d_index(addr); +} +#define p4d_offset_lockless p4d_offset_lockless + +static inline p4d_t *p4d_offset(pgd_t *pgdp, unsigned long addr) +{ + return p4d_offset_lockless(pgdp, READ_ONCE(*pgdp), addr); +} + +#define pgd_page(pgd) pfn_to_page(__phys_to_pfn(__pgd_to_phys(pgd))) + +#else + +static inline bool pgtable_l5_enabled(void) { return false; } + +#endif /* CONFIG_PGTABLE_LEVELS > 4 */ + #define pgd_ERROR(e) \ pr_err("%s:%d: bad pgd %016llx.\n", __FILE__, __LINE__, pgd_val(e)) diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index ba00d0205447..d2e9dec38a15 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -1025,7 +1025,7 @@ static void free_empty_pud_table(p4d_t *p4dp, unsigned long addr, if (CONFIG_PGTABLE_LEVELS <= 3) return; - if (!pgtable_range_aligned(start, end, floor, ceiling, PGDIR_MASK)) + if (!pgtable_range_aligned(start, end, floor, ceiling, P4D_MASK)) return; /* @@ -1048,8 +1048,8 @@ static void free_empty_p4d_table(pgd_t *pgdp, unsigned long addr, unsigned long end, unsigned long floor, unsigned long ceiling) { - unsigned long next; p4d_t *p4dp, p4d; + unsigned long i, next, start = addr; do { next = p4d_addr_end(addr, end); @@ -1061,6 +1061,27 @@ static void free_empty_p4d_table(pgd_t *pgdp, unsigned long addr, WARN_ON(!p4d_present(p4d)); free_empty_pud_table(p4dp, addr, next, floor, ceiling); } while (addr = next, addr < end); + + if (!pgtable_l5_enabled()) + return; + + if (!pgtable_range_aligned(start, end, floor, ceiling, PGDIR_MASK)) + return; + + /* + * Check whether we can free the p4d page if the rest of the + * entries are empty. Overlap with other regions have been + * handled by the floor/ceiling check. + */ + p4dp = p4d_offset(pgdp, 0UL); + for (i = 0; i < PTRS_PER_P4D; i++) { + if (!p4d_none(READ_ONCE(p4dp[i]))) + return; + } + + pgd_clear(pgdp); + __flush_tlb_kernel_pgtable(start); + free_hotplug_pgtable_page(virt_to_page(p4dp)); } static void free_empty_tables(unsigned long addr, unsigned long end, @@ -1145,6 +1166,12 @@ int pmd_set_huge(pmd_t *pmdp, phys_addr_t phys, pgprot_t prot) return 1; } +#ifndef __PAGETABLE_P4D_FOLDED +void p4d_clear_huge(p4d_t *p4dp) +{ +} +#endif + int pud_clear_huge(pud_t *pudp) { if (!pud_sect(READ_ONCE(*pudp))) diff --git a/arch/arm64/mm/pgd.c b/arch/arm64/mm/pgd.c index 4a64089e5771..3c4f8a279d2b 100644 --- a/arch/arm64/mm/pgd.c +++ b/arch/arm64/mm/pgd.c @@ -17,11 +17,20 @@ static struct kmem_cache *pgd_cache __ro_after_init; +static bool pgdir_is_page_size(void) +{ + if (PGD_SIZE == PAGE_SIZE) + return true; + if (CONFIG_PGTABLE_LEVELS == 5) + return !pgtable_l5_enabled(); + return false; +} + pgd_t *pgd_alloc(struct mm_struct *mm) { gfp_t gfp = GFP_PGTABLE_USER; - if (PGD_SIZE == PAGE_SIZE) + if (pgdir_is_page_size()) return (pgd_t *)__get_free_page(gfp); else return kmem_cache_alloc(pgd_cache, gfp); @@ -29,7 +38,7 @@ pgd_t *pgd_alloc(struct mm_struct *mm) void pgd_free(struct mm_struct *mm, pgd_t *pgd) { - if (PGD_SIZE == PAGE_SIZE) + if (pgdir_is_page_size()) free_page((unsigned long)pgd); else kmem_cache_free(pgd_cache, pgd); @@ -37,7 +46,7 @@ void pgd_free(struct mm_struct *mm, pgd_t *pgd) void __init pgtable_cache_init(void) { - if (PGD_SIZE == PAGE_SIZE) + if (pgdir_is_page_size()) return; #ifdef CONFIG_ARM64_PA_BITS_52 From patchwork Tue Jan 23 14:53:38 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13527716 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id E8105C47258 for ; Tue, 23 Jan 2024 16:00:38 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=/omRSc5jnA21AiUvqAqWUDQ2oBFE/zt2piXhvNbno0Q=; b=e6hZozFEhJ1jTdLdhGZ2sxZbpQ /+7AVl1WVWIuxYfIaYf1iyIMECtpcVSxGB806PWggwJERxcX34oJPGb1QrzVPrCn8euXK4be9HaAo rnSf+SGBTuc2gEa2ElO8uBhP7smioVPcTeaJiyS5yH6hcj1q5HQNudAJH//nxW0OEAK7RxK/TIUMA ZF8ZZXEOyIDl8gO7NZMsg6bPuH4zWN1u0XUwYfX7myOCOpueZYPSKPnE3eS6p9zjf/S0m1SlOCm52 NwdvS8WvV2U0JV/y+auPXoV2s277bzGlvUxA2wuN8HgCAAQueWC4D4VHWAg3ia994Hp9CtQ9Lg8e1 vaG+Ccww==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rSJCW-00HAuu-0q; Tue, 23 Jan 2024 16:00:36 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rSIBy-00GuQf-0F for linux-arm-kernel@bombadil.infradead.org; Tue, 23 Jan 2024 14:55:58 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=AviN0IFXbzaVb7eLl7lq4OId7ilOzOEDH0/Mv3Ok3Ak=; b=YVwAMisXCHPVcOL4UGcCvaYdDR eN6uVh0AMjfzl0oDFfMBpS+YLC5NyxZiPGAGnF1quUhm+Tv5MJ776iKZZ3AWy2UZpuiKL3YE7/FOW MXxeNEQP0etVawHqryBzfiAeGMm0NGscRR/CqhmmuZZKn7NY7VBNNGAsa9oM0zr6avU6yGYJxzTXW XZ0kBcZQNOgq9Ea2zW6LXIvIVVDQaCaS6mqZw7ZETh2CmeneMxMgQByBivznIHBvjJXci5iNXccFo SVDQsf1py+ldkIGNOJcBOAgKSsAK54jZeERJy8cS4SwidHws1wlZN1jj8W4mR+yhgryPeoK/YGvTR pSTJF0DQ==; Received: from mail-yb1-xb49.google.com ([2607:f8b0:4864:20::b49]) by desiato.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rSIBu-00000003vZ5-2j6r for linux-arm-kernel@lists.infradead.org; Tue, 23 Jan 2024 14:55:56 +0000 Received: by mail-yb1-xb49.google.com with SMTP id 3f1490d57ef6-dc374283d67so1098074276.3 for ; Tue, 23 Jan 2024 06:55:53 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1706021753; x=1706626553; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=AviN0IFXbzaVb7eLl7lq4OId7ilOzOEDH0/Mv3Ok3Ak=; b=UGE6k4shy9dCq3yhSVlCkpkMu1Zp/AKkJXVmAf5b2lPqmjev7A/DUKdjuS4BFTzV3S JIHpq6s4mV2vDQuAtm4AQS7BgkGU+AacFm40p9qA+Qw33/7G99OMrnySCXGu1aCaBeZD UdlXQYuoyJfxX8xjVnpV2pcgaAkcC4hIcDf6ewlvs93L3XJPN2lazCy2gqi5Y2Ps27UF c8LQeupQ84E9U9dJBY2vcmWttZwEdjnluN2O0IaK0X1nE4ysBOXcOXSjpKBuu7cihP+y gMJnpWdHKjwcyryffd5V65v8TJbe/QNB9xob8ujvNF39TGEcGseLFwIyeYpTNqEQHdnL LPSA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706021753; x=1706626553; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=AviN0IFXbzaVb7eLl7lq4OId7ilOzOEDH0/Mv3Ok3Ak=; b=Yyq1HpYM/vsC/XMvzgTF3zYZ5dBJNfqR3AZUqRFWQd8EM84spi/kllhRPbUmTv/kvQ +opFl0pa1YcbZZvpxtkiCJkiHyxgyWaglTrE0nrthTH2m+6xmDUrVF6kS6E+BfUnyVN2 /UDe29+98+H4ih3hmnGLAVq1s0lmPSZr8xdR2Yxmg+PHYkmCSvKIGAk4l8G+ToNMLAY1 WNtEXu1B2w2vXQ45l16rosfwAMp9E4nTDemMfblW+rokErr0nBdSnYLv3QTBUzRZN8LT gX9G83MTkV/A6ryndZqCtHa9DSNhLMyAi4j7MN1i/2ugaNE09Gkqqpb3WVzY6X4G2LFs Lb/A== X-Gm-Message-State: AOJu0Yybmd6nVYFIPDZ0rqaCobDDJXGM9zkwoAcXWx5pXmNfIjSyYvqM PJug3tKfSWKjZzlmr0+ahR+FGXkgWcYcaLpLl+9Y0JKXoAIW/oGqp9oFMSa0POXsV/wW+D7+Tfz 0hihnlypiKLVBw7kH7odMqR9KEnyIsyXyGHqT8mcrkkhpSRsCyF5IcJfOZ2Mw9tDGPi9k70R27x VqH7qU+2q/oWyHDQBqEqMhycOnnS1ebar5kzprMnSk X-Google-Smtp-Source: AGHT+IHOQJwFpfqdkpWfUOkI2kHhnBNX4/vMO4PgBOKo6ZBBF8eEpT+GMVWAEbeoYrxPiuB1Ji/FGqp2 X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a25:4f55:0:b0:dc2:4fff:75e9 with SMTP id d82-20020a254f55000000b00dc24fff75e9mr2809681ybb.8.1706021752733; Tue, 23 Jan 2024 06:55:52 -0800 (PST) Date: Tue, 23 Jan 2024 15:53:38 +0100 In-Reply-To: <20240123145258.1462979-52-ardb+git@google.com> Mime-Version: 1.0 References: <20240123145258.1462979-52-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=6529; i=ardb@kernel.org; h=from:subject; bh=XYJCQgbo4x5GPyU84XxVjSPhiwpdx2vqDizAij74NQA=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIXX9pa8Nfl+6uJOdozYaqP3Q+vGBafOeNa69rrwyb/qP/ jqfYpndUcrCIMbBICumyCIw+++7nacnStU6z5KFmcPKBDKEgYtTACZyV5zhn0XPqZiF+hcPb+qw 3sL2Uzf4ss88691bLyi231z/aPPDDRcZ/uktzea/uNmpJIO5dWLzpyq1ZaujdxaoSV6wyjX1K93 RxAwA X-Mailer: git-send-email 2.43.0.429.g432eaa2c6b-goog Message-ID: <20240123145258.1462979-91-ardb+git@google.com> Subject: [PATCH v7 39/50] arm64: mm: add LPA2 and 5 level paging support to G-to-nG conversion From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240123_145555_069560_A237DE90 X-CRM114-Status: GOOD ( 20.79 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel Add support for 5 level paging in the G-to-nG routine that creates its own temporary page tables to traverse the swapper page tables. Also add support for running the 5 level configuration with the top level folded at runtime, to support CPUs that do not implement the LPA2 extension. While at it, wire up the level skipping logic so it will also trigger on 4 level configurations with LPA2 enabled at build time but not active at runtime, as we'll fall back to 3 level paging in that case. Signed-off-by: Ard Biesheuvel --- arch/arm64/kernel/cpufeature.c | 9 ++- arch/arm64/mm/proc.S | 70 +++++++++++++++++--- 2 files changed, 66 insertions(+), 13 deletions(-) diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c index ed9670d8360c..bc5e4e569864 100644 --- a/arch/arm64/kernel/cpufeature.c +++ b/arch/arm64/kernel/cpufeature.c @@ -1765,6 +1765,9 @@ static int __init __kpti_install_ng_mappings(void *__unused) pgd_t *kpti_ng_temp_pgd; u64 alloc = 0; + if (levels == 5 && !pgtable_l5_enabled()) + levels = 4; + remap_fn = (void *)__pa_symbol(idmap_kpti_install_ng_mappings); if (!cpu) { @@ -1778,9 +1781,9 @@ static int __init __kpti_install_ng_mappings(void *__unused) // // The physical pages are laid out as follows: // - // +--------+-/-------+-/------ +-\\--------+ - // : PTE[] : | PMD[] : | PUD[] : || PGD[] : - // +--------+-\-------+-\------ +-//--------+ + // +--------+-/-------+-/------ +-/------ +-\\\--------+ + // : PTE[] : | PMD[] : | PUD[] : | P4D[] : ||| PGD[] : + // +--------+-\-------+-\------ +-\------ +-///--------+ // ^ // The first page is mapped into this hierarchy at a PMD_SHIFT // aligned virtual address, so that we can manipulate the PTE diff --git a/arch/arm64/mm/proc.S b/arch/arm64/mm/proc.S index d03434b7bca5..fa0d7c63f8d2 100644 --- a/arch/arm64/mm/proc.S +++ b/arch/arm64/mm/proc.S @@ -216,16 +216,15 @@ SYM_FUNC_ALIAS(__pi_idmap_cpu_replace_ttbr1, idmap_cpu_replace_ttbr1) .macro kpti_mk_tbl_ng, type, num_entries add end_\type\()p, cur_\type\()p, #\num_entries * 8 .Ldo_\type: - ldr \type, [cur_\type\()p] // Load the entry + ldr \type, [cur_\type\()p], #8 // Load the entry and advance tbz \type, #0, .Lnext_\type // Skip invalid and tbnz \type, #11, .Lnext_\type // non-global entries orr \type, \type, #PTE_NG // Same bit for blocks and pages - str \type, [cur_\type\()p] // Update the entry + str \type, [cur_\type\()p, #-8] // Update the entry .ifnc \type, pte tbnz \type, #1, .Lderef_\type .endif .Lnext_\type: - add cur_\type\()p, cur_\type\()p, #8 cmp cur_\type\()p, end_\type\()p b.ne .Ldo_\type .endm @@ -235,18 +234,18 @@ SYM_FUNC_ALIAS(__pi_idmap_cpu_replace_ttbr1, idmap_cpu_replace_ttbr1) * fixmap slot associated with the current level. */ .macro kpti_map_pgtbl, type, level - str xzr, [temp_pte, #8 * (\level + 1)] // break before make + str xzr, [temp_pte, #8 * (\level + 2)] // break before make dsb nshst - add pte, temp_pte, #PAGE_SIZE * (\level + 1) + add pte, temp_pte, #PAGE_SIZE * (\level + 2) lsr pte, pte, #12 tlbi vaae1, pte dsb nsh isb phys_to_pte pte, cur_\type\()p - add cur_\type\()p, temp_pte, #PAGE_SIZE * (\level + 1) + add cur_\type\()p, temp_pte, #PAGE_SIZE * (\level + 2) orr pte, pte, pte_flags - str pte, [temp_pte, #8 * (\level + 1)] + str pte, [temp_pte, #8 * (\level + 2)] dsb nshst .endm @@ -279,6 +278,8 @@ SYM_TYPED_FUNC_START(idmap_kpti_install_ng_mappings) end_ptep .req x15 pte .req x16 valid .req x17 + cur_p4dp .req x19 + end_p4dp .req x20 mov x5, x3 // preserve temp_pte arg mrs swapper_ttb, ttbr1_el1 @@ -286,6 +287,12 @@ SYM_TYPED_FUNC_START(idmap_kpti_install_ng_mappings) cbnz cpu, __idmap_kpti_secondary +#if CONFIG_PGTABLE_LEVELS > 4 + stp x29, x30, [sp, #-32]! + mov x29, sp + stp x19, x20, [sp, #16] +#endif + /* We're the boot CPU. Wait for the others to catch up */ sevl 1: wfe @@ -303,9 +310,32 @@ SYM_TYPED_FUNC_START(idmap_kpti_install_ng_mappings) mov_q pte_flags, KPTI_NG_PTE_FLAGS /* Everybody is enjoying the idmap, so we can rewrite swapper. */ + +#ifdef CONFIG_ARM64_LPA2 + /* + * If LPA2 support is configured, but 52-bit virtual addressing is not + * enabled at runtime, we will fall back to one level of paging less, + * and so we have to walk swapper_pg_dir as if we dereferenced its + * address from a PGD level entry, and terminate the PGD level loop + * right after. + */ + adrp pgd, swapper_pg_dir // walk &swapper_pg_dir at the next level + mov cur_pgdp, end_pgdp // must be equal to terminate the PGD loop +alternative_if_not ARM64_HAS_VA52 + b .Lderef_pgd // skip to the next level +alternative_else_nop_endif + /* + * LPA2 based 52-bit virtual addressing requires 52-bit physical + * addressing to be enabled as well. In this case, the shareability + * bits are repurposed as physical address bits, and should not be + * set in pte_flags. + */ + bic pte_flags, pte_flags, #PTE_SHARED +#endif + /* PGD */ adrp cur_pgdp, swapper_pg_dir - kpti_map_pgtbl pgd, 0 + kpti_map_pgtbl pgd, -1 kpti_mk_tbl_ng pgd, PTRS_PER_PGD /* Ensure all the updated entries are visible to secondary CPUs */ @@ -318,16 +348,33 @@ SYM_TYPED_FUNC_START(idmap_kpti_install_ng_mappings) /* Set the flag to zero to indicate that we're all done */ str wzr, [flag_ptr] +#if CONFIG_PGTABLE_LEVELS > 4 + ldp x19, x20, [sp, #16] + ldp x29, x30, [sp], #32 +#endif ret .Lderef_pgd: + /* P4D */ + .if CONFIG_PGTABLE_LEVELS > 4 + p4d .req x30 + pte_to_phys cur_p4dp, pgd + kpti_map_pgtbl p4d, 0 + kpti_mk_tbl_ng p4d, PTRS_PER_P4D + b .Lnext_pgd + .else /* CONFIG_PGTABLE_LEVELS <= 4 */ + p4d .req pgd + .set .Lnext_p4d, .Lnext_pgd + .endif + +.Lderef_p4d: /* PUD */ .if CONFIG_PGTABLE_LEVELS > 3 pud .req x10 - pte_to_phys cur_pudp, pgd + pte_to_phys cur_pudp, p4d kpti_map_pgtbl pud, 1 kpti_mk_tbl_ng pud, PTRS_PER_PUD - b .Lnext_pgd + b .Lnext_p4d .else /* CONFIG_PGTABLE_LEVELS <= 3 */ pud .req pgd .set .Lnext_pud, .Lnext_pgd @@ -371,6 +418,9 @@ SYM_TYPED_FUNC_START(idmap_kpti_install_ng_mappings) .unreq end_ptep .unreq pte .unreq valid + .unreq cur_p4dp + .unreq end_p4dp + .unreq p4d /* Secondary CPUs end up here */ __idmap_kpti_secondary: From patchwork Tue Jan 23 14:53:39 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13527732 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id E249CC47DDB for ; Tue, 23 Jan 2024 16:01:13 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=Z2K5E1w3qOHO0lXduVcb8ood3chhBxdoZewAhlvwAZs=; b=YPkjE1XKG0W3WrdLGy5MhNLeCM Pj5xSSGwpyPNronN5ekzxO4B2fctk8Fjyxp7Iq3Fsfh9O4ClBZENEytmqUc9ODfq/4kaaf1tde23V Ykt7iWGXYcmpXxZEbdhRca6R96YCiinWbE1ha8w8Br3D03xIHiZfRHynTHTLockUSQN6sHMxwNj8Y MY+ya5lfvJW6jA65esTnBJI78+nJv8spnSk5JnUb0klSS3XydaHCjJyTLHEoe5DqEsBcoTSE0kUe0 dh+Cg1+3hzyRc+zsJ1SCirsPRp390Fe7AhHXwnGj2ciq8LK8/FooJCj9PZp8yxtZveWeypCYYQX2P 1aN3C5aw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rSJCW-00HAvE-1j; Tue, 23 Jan 2024 16:00:36 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rSIC0-00GuRs-1B for linux-arm-kernel@bombadil.infradead.org; Tue, 23 Jan 2024 14:56:00 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=+PEVwrGbR/a1nQ2y5lrnpgLahWt/dgDRyJsvHOdehxM=; b=aYeONcpEZtDfcV+IranyQDrYAr uP9w326nkmQ5ALFXJTx6qzira7ThIvI1criui26MmAAsHzExNroGdBVkY/+mZwuBHDGbX05g0Fxkf THzO4qpNBBjEL588RIPTiBIIN9nBE8orPqkOz2fFZvi6s9qnULdFuzGGabkttOdlePFh8u2/OA7tK oTQE/wNnGEsg2pfJQ95zo9306a71vsPLzQo1q++0lhOn0u0cQ58ua7Z1+ltnnjYKIA8uJC8pqhY/j buDhR1Fr80xnjFjnXgbqPiPX4QVYozOp7hjEktqV9+CpfQj+eQQFF1pbD8G4iDMU3NxPu7uq2OK+k IunFu2Lg==; Received: from mail-yb1-xb49.google.com ([2607:f8b0:4864:20::b49]) by desiato.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rSIBw-00000003vZR-3zDf for linux-arm-kernel@lists.infradead.org; Tue, 23 Jan 2024 14:55:59 +0000 Received: by mail-yb1-xb49.google.com with SMTP id 3f1490d57ef6-dc37425c78eso1092331276.2 for ; Tue, 23 Jan 2024 06:55:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1706021755; x=1706626555; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=+PEVwrGbR/a1nQ2y5lrnpgLahWt/dgDRyJsvHOdehxM=; b=khT4NZt6uA47cux4aLAHLxg/0o8qt0omhLRnHFbO6SSFMijRfKZl+xMQspBVjrdEJ3 kRotLFpFx2i2PTNLTUiHTdRadt0ExBBrRG5YIGUaGHeQwOb70TOMnOc4B2JRRrSNqYEO W7mN20iMe9vB8lGCLanGO9TgM/+8bwnrkM/gefzNAieseon/ZE9mx9yjlRrqp6PXVQpG UU2HXn0jtgLe0Kfac3H/0N1xETtG3ZpO1aRDGBxawuBQ0F1KeQQIffmF/jLOlV3hfsyg mckwK/GlIjcT8BX2EKJhXtCT95xCazy8tFGSBlYVWw5eYzj2ZWgxqIV84ELJAMkvzWiB wLEQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706021755; x=1706626555; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=+PEVwrGbR/a1nQ2y5lrnpgLahWt/dgDRyJsvHOdehxM=; b=ovKXX8n82ASgwJe0cfk3F9jl3iuZjF9J7kPmBNFPv6YQ3BDN7m5AmeX4MtrW4l45eq sO2bZcG3eJf6CEZn71P2aVVc48I/EFZbKfVKssupjF8Cq923T1GtpyA49VjaEjcx/TST Zks/QgJd15PIuuW6PUO5h6Wjczl4GjwlJPFJeVyfdTaa9FD7FiKRJ3dl7NnjoVdiPKDx EZnCkLnX5hs2ilX0OVcLXlAZn6Wc7tW7QYZQS5pDqNvcja9/EgRaJBPAp7P2ivO/pxdG H/YwnBn6BB+2YyhJBWxN4Jpg6LJKvEdA4NJyVLKv5DKIWRgvYZWyfxMSl6DN3hXXRN+U PgMw== X-Gm-Message-State: AOJu0YwaY7PUduspL06DLKQiv95U9E4JuFhie6pRC4gr1my+NfTN3dVo uHGU10lPkJJkut0UtK21sdoVz9y8vnGQABkwOWQ2YwWcdK+D8/36t51uaY5Lj4MbuFWw9+E1xj9 SupZUQJgZCim0NytRCwMOcIaZunKtbVZ5pbLCZy6z3AcYjItqcLSlBsO1I2dwikyRyesHiYx9jq bsTOfOYbW4uqFMNTvsEtgGCE5LYme0FWUIpvNCOfz5 X-Google-Smtp-Source: AGHT+IGXbD0DT2EJeLGfV6lTGIoHV5bmLz2vxeBTbV/5AO5gBI8HfMWinbi9kBkeOazNt873TJVc8HYB X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:6902:1b12:b0:db5:4766:e363 with SMTP id eh18-20020a0569021b1200b00db54766e363mr2616454ybb.6.1706021755122; Tue, 23 Jan 2024 06:55:55 -0800 (PST) Date: Tue, 23 Jan 2024 15:53:39 +0100 In-Reply-To: <20240123145258.1462979-52-ardb+git@google.com> Mime-Version: 1.0 References: <20240123145258.1462979-52-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=13234; i=ardb@kernel.org; h=from:subject; bh=qemP2hmP7wLXMj6PGzNKlQdgeVRajsZvtJmUTpUvJvM=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIXX9pW/tH3/8nONZf/5R6/fnMerrZ6lMPHZY51/VueT17 45Wl5x06ChlYRDjYJAVU2QRmP333c7TE6VqnWfJwsxhZQIZwsDFKQATWbeRkeFb6aSPl/sSpb6e DF55yEe2xy2r98Z6odN2Vz90a7LNk1Ri+Cu0er2wheZy3yu1uqq71GcutlnnsLPgukB1zuTZy7u 66tkB X-Mailer: git-send-email 2.43.0.429.g432eaa2c6b-goog Message-ID: <20240123145258.1462979-92-ardb+git@google.com> Subject: [PATCH v7 40/50] arm64: Enable LPA2 at boot if supported by the system From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240123_145557_327365_F11259B8 X-CRM114-Status: GOOD ( 32.43 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel Update the early kernel mapping code to take 52-bit virtual addressing into account based on the LPA2 feature. This is a bit more involved than LVA (which is supported with 64k pages only), given that some page table descriptor bits change meaning in this case. To keep the handling in asm to a minimum, the initial ID map is still created with 48-bit virtual addressing, which implies that the kernel image must be loaded into 48-bit addressable physical memory. This is currently required by the boot protocol, even though we happen to support placement outside of that for LVA/64k based configurations. Enabling LPA2 involves more than setting TCR.T1SZ to a lower value, there is also a DS bit in TCR that needs to be set, and which changes the meaning of bits [9:8] in all page table descriptors. Since we cannot enable DS and every live page table descriptor at the same time, let's pivot through another temporary mapping. This avoids the need to reintroduce manipulations of the page tables with the MMU and caches disabled. To permit the LPA2 feature to be overridden on the kernel command line, which may be necessary to work around silicon errata, or to deal with mismatched features on heterogeneous SoC designs, test for CPU feature overrides first, and only then enable LPA2. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/assembler.h | 8 ++- arch/arm64/include/asm/cpufeature.h | 18 +++++ arch/arm64/include/asm/memory.h | 4 ++ arch/arm64/kernel/head.S | 8 +++ arch/arm64/kernel/image-vars.h | 1 + arch/arm64/kernel/pi/map_kernel.c | 70 +++++++++++++++++++- arch/arm64/kernel/pi/map_range.c | 11 ++- arch/arm64/kernel/pi/pi.h | 4 +- arch/arm64/mm/init.c | 2 +- arch/arm64/mm/mmu.c | 6 +- arch/arm64/mm/proc.S | 3 + 11 files changed, 124 insertions(+), 11 deletions(-) diff --git a/arch/arm64/include/asm/assembler.h b/arch/arm64/include/asm/assembler.h index 7eedcb36ebe0..ce7b95cd6e79 100644 --- a/arch/arm64/include/asm/assembler.h +++ b/arch/arm64/include/asm/assembler.h @@ -581,11 +581,17 @@ alternative_endif * but we have to add an offset so that the TTBR1 address corresponds with the * pgdir entry that covers the lowest 48-bit addressable VA. * + * Note that this trick is only used for LVA/64k pages - LPA2/4k pages uses an + * additional paging level, and on LPA2/16k pages, we would end up with a root + * level table with only 2 entries, which is suboptimal in terms of TLB + * utilization, so there we fall back to 47 bits of translation if LPA2 is not + * supported. + * * orr is used as it can cover the immediate value (and is idempotent). * ttbr: Value of ttbr to set, modified. */ .macro offset_ttbr1, ttbr, tmp -#ifdef CONFIG_ARM64_VA_BITS_52 +#if defined(CONFIG_ARM64_VA_BITS_52) && !defined(CONFIG_ARM64_LPA2) mrs \tmp, tcr_el1 and \tmp, \tmp, #TCR_T1SZ_MASK cmp \tmp, #TCR_T1SZ(VA_BITS_MIN) diff --git a/arch/arm64/include/asm/cpufeature.h b/arch/arm64/include/asm/cpufeature.h index a2ac31aecdd9..a8f97690ce1f 100644 --- a/arch/arm64/include/asm/cpufeature.h +++ b/arch/arm64/include/asm/cpufeature.h @@ -1008,6 +1008,24 @@ static inline bool cpu_has_lva(void) ID_AA64MMFR2_EL1_VARange_SHIFT); } +static inline bool cpu_has_lpa2(void) +{ +#ifdef CONFIG_ARM64_LPA2 + u64 mmfr0; + int feat; + + mmfr0 = read_sysreg(id_aa64mmfr0_el1); + mmfr0 &= ~id_aa64mmfr0_override.mask; + mmfr0 |= id_aa64mmfr0_override.val; + feat = cpuid_feature_extract_signed_field(mmfr0, + ID_AA64MMFR0_EL1_TGRAN_SHIFT); + + return feat >= ID_AA64MMFR0_EL1_TGRAN_LPA2; +#else + return false; +#endif +} + #endif /* __ASSEMBLY__ */ #endif diff --git a/arch/arm64/include/asm/memory.h b/arch/arm64/include/asm/memory.h index 9680d7444b3b..b850b1b91471 100644 --- a/arch/arm64/include/asm/memory.h +++ b/arch/arm64/include/asm/memory.h @@ -54,7 +54,11 @@ #define FIXADDR_TOP (-UL(SZ_8M)) #if VA_BITS > 48 +#ifdef CONFIG_ARM64_16K_PAGES +#define VA_BITS_MIN (47) +#else #define VA_BITS_MIN (48) +#endif #else #define VA_BITS_MIN (VA_BITS) #endif diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S index e25351addfd0..405e9bce8c73 100644 --- a/arch/arm64/kernel/head.S +++ b/arch/arm64/kernel/head.S @@ -89,6 +89,7 @@ SYM_CODE_START(primary_entry) mov sp, x1 mov x29, xzr adrp x0, init_idmap_pg_dir + mov x1, xzr bl __pi_create_init_idmap /* @@ -473,9 +474,16 @@ SYM_FUNC_END(__enable_mmu) #ifdef CONFIG_ARM64_VA_BITS_52 SYM_FUNC_START(__cpu_secondary_check52bitva) +#ifndef CONFIG_ARM64_LPA2 mrs_s x0, SYS_ID_AA64MMFR2_EL1 and x0, x0, ID_AA64MMFR2_EL1_VARange_MASK cbnz x0, 2f +#else + mrs x0, id_aa64mmfr0_el1 + sbfx x0, x0, #ID_AA64MMFR0_EL1_TGRAN_SHIFT, 4 + cmp x0, #ID_AA64MMFR0_EL1_TGRAN_LPA2 + b.ge 2f +#endif update_early_cpu_boot_status \ CPU_STUCK_IN_KERNEL | CPU_STUCK_REASON_52_BIT_VA, x0, x1 diff --git a/arch/arm64/kernel/image-vars.h b/arch/arm64/kernel/image-vars.h index ff81f809a240..ba4f8f7d6a91 100644 --- a/arch/arm64/kernel/image-vars.h +++ b/arch/arm64/kernel/image-vars.h @@ -54,6 +54,7 @@ PROVIDE(__pi__ctype = _ctype); PROVIDE(__pi_memstart_offset_seed = memstart_offset_seed); PROVIDE(__pi_init_idmap_pg_dir = init_idmap_pg_dir); +PROVIDE(__pi_init_idmap_pg_end = init_idmap_pg_end); PROVIDE(__pi_init_pg_dir = init_pg_dir); PROVIDE(__pi_init_pg_end = init_pg_end); PROVIDE(__pi_swapper_pg_dir = swapper_pg_dir); diff --git a/arch/arm64/kernel/pi/map_kernel.c b/arch/arm64/kernel/pi/map_kernel.c index 1853825aa29d..5fa08e13e17e 100644 --- a/arch/arm64/kernel/pi/map_kernel.c +++ b/arch/arm64/kernel/pi/map_kernel.c @@ -127,11 +127,64 @@ static void __init map_kernel(u64 kaslr_offset, u64 va_offset, int root_level) } /* Copy the root page table to its final location */ - memcpy((void *)swapper_pg_dir + va_offset, init_pg_dir, PGD_SIZE); + memcpy((void *)swapper_pg_dir + va_offset, init_pg_dir, PAGE_SIZE); dsb(ishst); idmap_cpu_replace_ttbr1(swapper_pg_dir); } +static void noinline __section(".idmap.text") set_ttbr0_for_lpa2(u64 ttbr) +{ + u64 sctlr = read_sysreg(sctlr_el1); + u64 tcr = read_sysreg(tcr_el1) | TCR_DS; + + asm(" msr sctlr_el1, %0 ;" + " isb ;" + " msr ttbr0_el1, %1 ;" + " msr tcr_el1, %2 ;" + " isb ;" + " tlbi vmalle1 ;" + " dsb nsh ;" + " isb ;" + " msr sctlr_el1, %3 ;" + " isb ;" + :: "r"(sctlr & ~SCTLR_ELx_M), "r"(ttbr), "r"(tcr), "r"(sctlr)); +} + +static void __init remap_idmap_for_lpa2(void) +{ + /* clear the bits that change meaning once LPA2 is turned on */ + pteval_t mask = PTE_SHARED; + + /* + * We have to clear bits [9:8] in all block or page descriptors in the + * initial ID map, as otherwise they will be (mis)interpreted as + * physical address bits once we flick the LPA2 switch (TCR.DS). Since + * we cannot manipulate live descriptors in that way without creating + * potential TLB conflicts, let's create another temporary ID map in a + * LPA2 compatible fashion, and update the initial ID map while running + * from that. + */ + create_init_idmap(init_pg_dir, mask); + dsb(ishst); + set_ttbr0_for_lpa2((u64)init_pg_dir); + + /* + * Recreate the initial ID map with the same granularity as before. + * Don't bother with the FDT, we no longer need it after this. + */ + memset(init_idmap_pg_dir, 0, + (u64)init_idmap_pg_dir - (u64)init_idmap_pg_end); + + create_init_idmap(init_idmap_pg_dir, mask); + dsb(ishst); + + /* switch back to the updated initial ID map */ + set_ttbr0_for_lpa2((u64)init_idmap_pg_dir); + + /* wipe the temporary ID map from memory */ + memset(init_pg_dir, 0, (u64)init_pg_end - (u64)init_pg_dir); +} + static void __init map_fdt(u64 fdt) { static u8 ptes[INIT_IDMAP_FDT_SIZE] __initdata __aligned(PAGE_SIZE); @@ -154,6 +207,7 @@ asmlinkage void __init early_map_kernel(u64 boot_status, void *fdt) u64 va_base, pa_base = (u64)&_text; u64 kaslr_offset = pa_base % MIN_KIMG_ALIGN; int root_level = 4 - CONFIG_PGTABLE_LEVELS; + int va_bits = VA_BITS; int chosen; map_fdt((u64)fdt); @@ -165,8 +219,15 @@ asmlinkage void __init early_map_kernel(u64 boot_status, void *fdt) chosen = fdt_path_offset(fdt, chosen_str); init_feature_override(boot_status, fdt, chosen); - if (VA_BITS > VA_BITS_MIN && cpu_has_lva()) - sysreg_clear_set(tcr_el1, TCR_T1SZ_MASK, TCR_T1SZ(VA_BITS)); + if (IS_ENABLED(CONFIG_ARM64_64K_PAGES) && !cpu_has_lva()) { + va_bits = VA_BITS_MIN; + } else if (IS_ENABLED(CONFIG_ARM64_LPA2) && !cpu_has_lpa2()) { + va_bits = VA_BITS_MIN; + root_level++; + } + + if (va_bits > VA_BITS_MIN) + sysreg_clear_set(tcr_el1, TCR_T1SZ_MASK, TCR_T1SZ(va_bits)); /* * The virtual KASLR displacement modulo 2MiB is decided by the @@ -184,6 +245,9 @@ asmlinkage void __init early_map_kernel(u64 boot_status, void *fdt) kaslr_offset |= kaslr_seed & ~(MIN_KIMG_ALIGN - 1); } + if (IS_ENABLED(CONFIG_ARM64_LPA2) && va_bits > VA_BITS_MIN) + remap_idmap_for_lpa2(); + va_base = KIMAGE_VADDR + kaslr_offset; map_kernel(kaslr_offset, va_base - pa_base, root_level); } diff --git a/arch/arm64/kernel/pi/map_range.c b/arch/arm64/kernel/pi/map_range.c index 79e4f6a2efe1..5410b2cac590 100644 --- a/arch/arm64/kernel/pi/map_range.c +++ b/arch/arm64/kernel/pi/map_range.c @@ -87,14 +87,19 @@ void __init map_range(u64 *pte, u64 start, u64 end, u64 pa, pgprot_t prot, } } -asmlinkage u64 __init create_init_idmap(pgd_t *pg_dir) +asmlinkage u64 __init create_init_idmap(pgd_t *pg_dir, pteval_t clrmask) { u64 ptep = (u64)pg_dir + PAGE_SIZE; + pgprot_t text_prot = PAGE_KERNEL_ROX; + pgprot_t data_prot = PAGE_KERNEL; + + pgprot_val(text_prot) &= ~clrmask; + pgprot_val(data_prot) &= ~clrmask; map_range(&ptep, (u64)_stext, (u64)__initdata_begin, (u64)_stext, - PAGE_KERNEL_ROX, IDMAP_ROOT_LEVEL, (pte_t *)pg_dir, false, 0); + text_prot, IDMAP_ROOT_LEVEL, (pte_t *)pg_dir, false, 0); map_range(&ptep, (u64)__initdata_begin, (u64)_end, (u64)__initdata_begin, - PAGE_KERNEL, IDMAP_ROOT_LEVEL, (pte_t *)pg_dir, false, 0); + data_prot, IDMAP_ROOT_LEVEL, (pte_t *)pg_dir, false, 0); return ptep; } diff --git a/arch/arm64/kernel/pi/pi.h b/arch/arm64/kernel/pi/pi.h index 1ea282a5f96a..c91e5e965cd3 100644 --- a/arch/arm64/kernel/pi/pi.h +++ b/arch/arm64/kernel/pi/pi.h @@ -21,7 +21,7 @@ static inline void *prel64_to_pointer(const prel64_t *offset) extern bool dynamic_scs_is_enabled; -extern pgd_t init_idmap_pg_dir[]; +extern pgd_t init_idmap_pg_dir[], init_idmap_pg_end[]; void init_feature_override(u64 boot_status, const void *fdt, int chosen); u64 kaslr_early_init(void *fdt, int chosen); @@ -33,4 +33,4 @@ void map_range(u64 *pgd, u64 start, u64 end, u64 pa, pgprot_t prot, asmlinkage void early_map_kernel(u64 boot_status, void *fdt); -asmlinkage u64 create_init_idmap(pgd_t *pgd); +asmlinkage u64 create_init_idmap(pgd_t *pgd, pteval_t clrmask); diff --git a/arch/arm64/mm/init.c b/arch/arm64/mm/init.c index 74c1db8ce271..0f427b50fdc3 100644 --- a/arch/arm64/mm/init.c +++ b/arch/arm64/mm/init.c @@ -238,7 +238,7 @@ void __init arm64_memblock_init(void) * physical address of PAGE_OFFSET, we have to *subtract* from it. */ if (IS_ENABLED(CONFIG_ARM64_VA_BITS_52) && (vabits_actual != 52)) - memstart_addr -= _PAGE_OFFSET(48) - _PAGE_OFFSET(52); + memstart_addr -= _PAGE_OFFSET(vabits_actual) - _PAGE_OFFSET(52); /* * Apply the memory limit if it was set. Since the kernel may be loaded diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index d2e9dec38a15..d30ae4d3fdd9 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -582,8 +582,12 @@ static void __init map_mem(pgd_t *pgdp) * entries at any level are being shared between the linear region and * the vmalloc region. Check whether this is true for the PGD level, in * which case it is guaranteed to be true for all other levels as well. + * (Unless we are running with support for LPA2, in which case the + * entire reduced VA space is covered by a single pgd_t which will have + * been populated without the PXNTable attribute by the time we get here.) */ - BUILD_BUG_ON(pgd_index(direct_map_end - 1) == pgd_index(direct_map_end)); + BUILD_BUG_ON(pgd_index(direct_map_end - 1) == pgd_index(direct_map_end) && + pgd_index(_PAGE_OFFSET(VA_BITS_MIN)) != PTRS_PER_PGD - 1); early_kfence_pool = arm64_kfence_alloc_pool(); diff --git a/arch/arm64/mm/proc.S b/arch/arm64/mm/proc.S index fa0d7c63f8d2..9d40f3ffd8d2 100644 --- a/arch/arm64/mm/proc.S +++ b/arch/arm64/mm/proc.S @@ -488,6 +488,9 @@ SYM_FUNC_START(__cpu_setup) mov x9, #64 - VA_BITS alternative_if ARM64_HAS_VA52 tcr_set_t1sz tcr, x9 +#ifdef CONFIG_ARM64_LPA2 + orr tcr, tcr, #TCR_DS +#endif alternative_else_nop_endif #endif From patchwork Tue Jan 23 14:53:40 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13527566 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 4CA1CC47DDB for ; Tue, 23 Jan 2024 14:58:44 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=kKGxAu/23K5GwPJka3yLNyC3K3IhRJTVVqr7BiT+fM8=; b=eW79b/Uoer7RZw0bbW10G5160X GmEEisJOh+J6Td9vKbcKJ4aMuytKFcnESXyWYIxh19hOEolOvA+DMl+l1bzPRKR9Zm3fRkA33+RyL 5HC8qhqY7wTX/YQLwsV4vS2q+qg7+4+bs4n4NBi5GFSg1InWvxiDSrFt8I+uQUZNQxFvfzdNrzKeV ADpP8OJYH52UTizKzT0NFdDXV/MTguco6GURQmj3eCC9f+GeZZqSLj0K8xsgMJIpDq5NW/gfxmSZA kge8v8ijcX+rASBYgTQZbRNcqQ2P0XVBKfffzDup+rAL4y0ZIEl85PANKdCnhEEmbfT0wHU3WT43i SZr5kCWw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rSIEA-00Gvh1-22; Tue, 23 Jan 2024 14:58:14 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rSIC2-00GuT7-2V for linux-arm-kernel@bombadil.infradead.org; Tue, 23 Jan 2024 14:56:02 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=O9qiOJlG6eSRmj+2/Xt1pD4QlhObgBVPxSvpDPcf5G4=; b=MRYnToQhdvjWfBIWumSWVxjaOl eDEuzLoiW/BzHd7yyQiEQ++ktD8go8wyEORGPihZANe/6jxB+qERG+NbBSyOGeaPxfK/LaeEWWBbI FzFLBl7XibvDT0viJ6XjrolzJtROvjJ6whNbixE4wiRQzR6oQ4CepweOWYF/pgtB2f0OHLwSnRaT7 V2wJZxQwJWsdyRh+bK2nXFR7tfUgFhQv6RNVTbbeja8+vq4Vxz0HkOwO4rFN2L7iz9O6E1gIs+ysw uB0R0XdgxmxDlG8UTx4A0pjA6IUGG/crXeZfC37S2h7S+NQadIMQDCHifelZyvpVCZ7BCaLj515qe jS6ptYiA==; Received: from mail-yb1-xb4a.google.com ([2607:f8b0:4864:20::b4a]) by desiato.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rSIBz-00000003vaj-1Qiy for linux-arm-kernel@lists.infradead.org; Tue, 23 Jan 2024 14:56:01 +0000 Received: by mail-yb1-xb4a.google.com with SMTP id 3f1490d57ef6-dc265e7a67cso4805921276.0 for ; Tue, 23 Jan 2024 06:55:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1706021757; x=1706626557; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=O9qiOJlG6eSRmj+2/Xt1pD4QlhObgBVPxSvpDPcf5G4=; b=vq+CuIZQcf1pZUdzZk7cT/P1ncPdH034IiLPNDDlsQuVWVsuKtfdqM/27tT687bFZm fBgyEm3/WT4075ZiG+mUDVUHw++3VHC8z+zeQ8DeexynBWEzuHFyfn/7q1Mzem6G2SAr blakp+wURILvFpsUdMzcHl1tXD9dLaAKRpR0JoJaEFkGQowHRiI8vKGeYO1RINQTtF8J wCHQHzle7rLKgvo+uMYYwH8NAHyVRS4IgzVU7aSiCuV4e4m6IBw55zkJXCov+lbnrKvh bY7e/37Lqt7I4W743gywcORxBjwhWBhCO0aDmSFsFwSBISj2MNfhn+OpIlku2eGPbNVG 6IQw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706021757; x=1706626557; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=O9qiOJlG6eSRmj+2/Xt1pD4QlhObgBVPxSvpDPcf5G4=; b=XPjZVhPr9ulZNyIpdvCZMfGj1oqsz6yFYTBZXqjA01Anauq93PpU2h4Y0FvzThNfbG rQbkigUJsHI9WjGoylKJkGDh5KU6L8BydDNFTnEZKv1HlMVuSpLy8/RGkkmrU3368P8W 2wj8J5ST8hUqaHelHHjHvOXJ/Hl2vaH8Wic/yS6YqbXNX2kwjFLinJl6vqpKkz8NVO9S w9AS2WB21iejBccXL2Vh8ySa8/cXgUXb0gEF+jcypusK6nVlkbbc3YP8JJObFZ0SzMDs lwCFqWTGdFxNYnrWCqsHRtyJoEPtnJo95fLpAkR+Y5ILuoyR76QdBlzs0Jl5xsxDlqOt l/AA== X-Gm-Message-State: AOJu0YxDYmGfBkto72yoYrnhhvAiOWqp/zgJPjqZ/5SAdws3W6cZPYtf uYr+4Soj0DCxXBRghLXm3NJ9xEGFnNdn7glntUag6bdLv6A9JsPOJoq5iavdEnsVcxlj/50biNO uWcEvMutRcI8x19wTrRgHR5xeGYouhRTbrRR+DUcSoe9VUiR5ANTyGQpyKI9iIDK4F5BkZJoY/S WkJRidnXB5y6vzFm9xq4E26pGuToF2UtanmaB6eu+0 X-Google-Smtp-Source: AGHT+IHGvSasga/ZGkYDTl5sF03mWgQ+ZYkap/sC0aPKguF3Vf3FZZhmZA032Mmrj6IaPmbXHSmPEODv X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:6902:352:b0:dc3:696e:ffae with SMTP id e18-20020a056902035200b00dc3696effaemr778325ybs.3.1706021757629; Tue, 23 Jan 2024 06:55:57 -0800 (PST) Date: Tue, 23 Jan 2024 15:53:40 +0100 In-Reply-To: <20240123145258.1462979-52-ardb+git@google.com> Mime-Version: 1.0 References: <20240123145258.1462979-52-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=5970; i=ardb@kernel.org; h=from:subject; bh=Zg78vKr18nGmVnJrGgMQil7+9p7HwMxWJLpooTu47cY=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIXX9pe/aZn+Tn3AmXXkjNEHllWpBGt+vsr3VrbPaj5ko/ rj8U/RyRykLgxgHg6yYIovA7L/vdp6eKFXrPEsWZg4rE8gQBi5OAZjIPjVGhjdp9X/OCHyduaC0 ebq5U4h3fbv37L5yrY0HPppwnZaM2sTwP0Pt7ZQ6BX03jozDdd/k3AO+9X6NWyqd2rSjtr/twt9 NHAA= X-Mailer: git-send-email 2.43.0.429.g432eaa2c6b-goog Message-ID: <20240123145258.1462979-93-ardb+git@google.com> Subject: [PATCH v7 41/50] arm64: mm: Add 5 level paging support to fixmap and swapper handling From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240123_145559_623883_7EDE6282 X-CRM114-Status: GOOD ( 14.92 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel Add support for using 5 levels of paging in the fixmap, as well as in the kernel page table handling code which uses fixmaps internally. This also handles the case where a 5 level build runs on hardware that only supports 4 levels of paging. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/fixmap.h | 1 + arch/arm64/include/asm/pgtable.h | 45 ++++++++++++++++--- arch/arm64/mm/fixmap.c | 2 +- arch/arm64/mm/mmu.c | 47 ++++++++++++++++++-- 4 files changed, 85 insertions(+), 10 deletions(-) diff --git a/arch/arm64/include/asm/fixmap.h b/arch/arm64/include/asm/fixmap.h index 8aabd45e9a13..87e307804b99 100644 --- a/arch/arm64/include/asm/fixmap.h +++ b/arch/arm64/include/asm/fixmap.h @@ -87,6 +87,7 @@ enum fixed_addresses { FIX_PTE, FIX_PMD, FIX_PUD, + FIX_P4D, FIX_PGD, __end_of_fixed_addresses diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 7eb2b933ed3c..3d7fb3cde83d 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -621,12 +621,12 @@ static inline bool pud_table(pud_t pud) { return true; } PUD_TYPE_TABLE) #endif -extern pgd_t init_pg_dir[PTRS_PER_PGD]; +extern pgd_t init_pg_dir[]; extern pgd_t init_pg_end[]; -extern pgd_t swapper_pg_dir[PTRS_PER_PGD]; -extern pgd_t idmap_pg_dir[PTRS_PER_PGD]; -extern pgd_t tramp_pg_dir[PTRS_PER_PGD]; -extern pgd_t reserved_pg_dir[PTRS_PER_PGD]; +extern pgd_t swapper_pg_dir[]; +extern pgd_t idmap_pg_dir[]; +extern pgd_t tramp_pg_dir[]; +extern pgd_t reserved_pg_dir[]; extern void set_swapper_pgd(pgd_t *pgdp, pgd_t pgd); @@ -891,12 +891,47 @@ static inline p4d_t *p4d_offset(pgd_t *pgdp, unsigned long addr) return p4d_offset_lockless(pgdp, READ_ONCE(*pgdp), addr); } +static inline p4d_t *p4d_set_fixmap(unsigned long addr) +{ + if (!pgtable_l5_enabled()) + return NULL; + return (p4d_t *)set_fixmap_offset(FIX_P4D, addr); +} + +static inline p4d_t *p4d_set_fixmap_offset(pgd_t *pgdp, unsigned long addr) +{ + if (!pgtable_l5_enabled()) + return pgd_to_folded_p4d(pgdp, addr); + return p4d_set_fixmap(p4d_offset_phys(pgdp, addr)); +} + +static inline void p4d_clear_fixmap(void) +{ + if (pgtable_l5_enabled()) + clear_fixmap(FIX_P4D); +} + +/* use ONLY for statically allocated translation tables */ +static inline p4d_t *p4d_offset_kimg(pgd_t *pgdp, u64 addr) +{ + if (!pgtable_l5_enabled()) + return pgd_to_folded_p4d(pgdp, addr); + return (p4d_t *)__phys_to_kimg(p4d_offset_phys(pgdp, addr)); +} + #define pgd_page(pgd) pfn_to_page(__phys_to_pfn(__pgd_to_phys(pgd))) #else static inline bool pgtable_l5_enabled(void) { return false; } +/* Match p4d_offset folding in */ +#define p4d_set_fixmap(addr) NULL +#define p4d_set_fixmap_offset(p4dp, addr) ((p4d_t *)p4dp) +#define p4d_clear_fixmap() + +#define p4d_offset_kimg(dir,addr) ((p4d_t *)dir) + #endif /* CONFIG_PGTABLE_LEVELS > 4 */ #define pgd_ERROR(e) \ diff --git a/arch/arm64/mm/fixmap.c b/arch/arm64/mm/fixmap.c index 9404f282f829..d22506e9c7fd 100644 --- a/arch/arm64/mm/fixmap.c +++ b/arch/arm64/mm/fixmap.c @@ -104,7 +104,7 @@ void __init early_fixmap_init(void) unsigned long end = FIXADDR_TOP; pgd_t *pgdp = pgd_offset_k(addr); - p4d_t *p4dp = p4d_offset(pgdp, addr); + p4d_t *p4dp = p4d_offset_kimg(pgdp, addr); early_fixmap_init_pud(p4dp, addr, end); } diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index d30ae4d3fdd9..8e5b3a7c5afd 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -313,15 +313,14 @@ static void alloc_init_cont_pmd(pud_t *pudp, unsigned long addr, } while (addr = next, addr != end); } -static void alloc_init_pud(pgd_t *pgdp, unsigned long addr, unsigned long end, +static void alloc_init_pud(p4d_t *p4dp, unsigned long addr, unsigned long end, phys_addr_t phys, pgprot_t prot, phys_addr_t (*pgtable_alloc)(int), int flags) { unsigned long next; - pud_t *pudp; - p4d_t *p4dp = p4d_offset(pgdp, addr); p4d_t p4d = READ_ONCE(*p4dp); + pud_t *pudp; if (p4d_none(p4d)) { p4dval_t p4dval = P4D_TYPE_TABLE | P4D_TABLE_UXN; @@ -369,6 +368,46 @@ static void alloc_init_pud(pgd_t *pgdp, unsigned long addr, unsigned long end, pud_clear_fixmap(); } +static void alloc_init_p4d(pgd_t *pgdp, unsigned long addr, unsigned long end, + phys_addr_t phys, pgprot_t prot, + phys_addr_t (*pgtable_alloc)(int), + int flags) +{ + unsigned long next; + pgd_t pgd = READ_ONCE(*pgdp); + p4d_t *p4dp; + + if (pgd_none(pgd)) { + pgdval_t pgdval = PGD_TYPE_TABLE | PGD_TABLE_UXN; + phys_addr_t p4d_phys; + + if (flags & NO_EXEC_MAPPINGS) + pgdval |= PGD_TABLE_PXN; + BUG_ON(!pgtable_alloc); + p4d_phys = pgtable_alloc(P4D_SHIFT); + __pgd_populate(pgdp, p4d_phys, pgdval); + pgd = READ_ONCE(*pgdp); + } + BUG_ON(pgd_bad(pgd)); + + p4dp = p4d_set_fixmap_offset(pgdp, addr); + do { + p4d_t old_p4d = READ_ONCE(*p4dp); + + next = p4d_addr_end(addr, end); + + alloc_init_pud(p4dp, addr, next, phys, prot, + pgtable_alloc, flags); + + BUG_ON(p4d_val(old_p4d) != 0 && + p4d_val(old_p4d) != READ_ONCE(p4d_val(*p4dp))); + + phys += next - addr; + } while (p4dp++, addr = next, addr != end); + + p4d_clear_fixmap(); +} + static void __create_pgd_mapping_locked(pgd_t *pgdir, phys_addr_t phys, unsigned long virt, phys_addr_t size, pgprot_t prot, @@ -391,7 +430,7 @@ static void __create_pgd_mapping_locked(pgd_t *pgdir, phys_addr_t phys, do { next = pgd_addr_end(addr, end); - alloc_init_pud(pgdp, addr, next, phys, prot, pgtable_alloc, + alloc_init_p4d(pgdp, addr, next, phys, prot, pgtable_alloc, flags); phys += next - addr; } while (pgdp++, addr = next, addr != end); From patchwork Tue Jan 23 14:53:41 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13527565 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id BB421C47258 for ; Tue, 23 Jan 2024 14:58:42 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=m8Pk4a2EZhF8MaazWIaoOuYIrwuW8GS96MukrfQfMZ8=; b=W74AwL/UY3IpFpndBKgx60LHKL AodMHJr3+ACJwvZ60DKnny6jMPPxxVMLIXFCJdIxdjmPlM81PZTr1bBbH3dsyLhIvepXV9K+PK59o hIf3x4m5mOsjsuH0Bib+rNns3uyxI9HsNkyZXxxvlQ/A4KvKxNc3QPNsk+9v4A3zT6wQ6MM32biUx iptrRKaD2Ryi+W25miiFvC7p/bLGottN/PDkE63Sq437gbi+zR9a+pMPl4/omwjahmy3wEnquB66b j4d0Fb6CChsSHyQN68+zVAXgam1M2G1Zc8zGmEeJ34EnjlKSSSsrwKMCi0+2U0gDkQyR/qeKFqd0f ot4lh2ng==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rSIEC-00GviG-0f; Tue, 23 Jan 2024 14:58:16 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rSIC5-00GuTq-1B for linux-arm-kernel@bombadil.infradead.org; Tue, 23 Jan 2024 14:56:05 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=g1M9FVqnd3WjH7QzJo0FqrIO2ElrqoOear7DtvDB348=; b=c7BQJTCyRSMYVEY15IkHUOUc2j KO777GWHtofl0NfddIfTP39XsKL3L7btcPa6w1ZcMDEN3tcTqERLXPPFXwf272WRdX3CTSaXyss// vxcPvG413Hd79+9xSKY5yd+8rFkrJE8Rr9kfNWnoc6SVGeEb+sJBESE+H8tkFl++5a5TWEnfafJyy GQv+E43AKyx8Iw8/Ls4lvSVHC66QrsmRaN23e3kFRMGO5v+OCUdzwU7PtC2C/DEu2wzxpj8fPS2fb pVf9UUN96GoyX6Vbh4YpsbMqbdJk4H18Nysqm3ANjCCFGkCJVZ3l+UfaPGLKUJucNQVMGrNI0QYGM qBvBrYZw==; Received: from mail-wm1-x34a.google.com ([2a00:1450:4864:20::34a]) by desiato.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rSIC1-00000003vbz-19U5 for linux-arm-kernel@lists.infradead.org; Tue, 23 Jan 2024 14:56:03 +0000 Received: by mail-wm1-x34a.google.com with SMTP id 5b1f17b1804b1-40e5f548313so38617435e9.1 for ; Tue, 23 Jan 2024 06:56:00 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1706021760; x=1706626560; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=g1M9FVqnd3WjH7QzJo0FqrIO2ElrqoOear7DtvDB348=; b=KxPYsTbaeMpAXH0sgSlTb06A+wH2Y7YmyPByrl6PrUMfHFVoUQB1M+Z7vmaIxMyM6v /ziAIUqNDTjKcsbBZfVEQu/C2PpzfBFMkt0aLdz4rSYa8vcxE2LQbNl3S9mwmEgqKE6Z cRqsc3ARLP4wY/xE+PwA0BTH1Bvf4gyO6JmmYUy18MNsZQOZ2Cq3yXPuqUpOv5vqxtm4 xvdJFn1EuJmjBsKuvFEhfISbjSrS+Fj0AXCcO88mZAj4hVnv+mwWrnzhyKuk08Ji7XQj UqejMhEwk8FXEXmewE+aNRP9p2CeB7Yp/nIU9MLcZ+hG3goqUGtbtqcB1POHqhkicUbW ATPA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706021760; x=1706626560; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=g1M9FVqnd3WjH7QzJo0FqrIO2ElrqoOear7DtvDB348=; b=Mocv1h/uKxXXak1P0fyfNLcUoYr1QY+ocZA2eMBCn3RW55X+iOAgfK7+GlH7SOwtBY KwXpDxnuqjAl3QsosM8vkoHflepcDeiWLAeyjBc61qL/0BI8xaZu8VE/gP6dwdeqCOnG ALVhD1DGDRFiyij2yOnq6h4L6jPTxddNgzOgPRrEK4pHsl6dp/UPgBCLo0JtEkQSleco h1pqZsJ2VpK1O0jEZHMj3HZa59BgyPhnkrj9DYnnZuQ3j4e+8xOXkNnyNgTFgkerU9CE MezZOi4R4vGIKMFtGeCKlOKav3Ni5hSE8SmAZ+xFUuFbvO50aPHEORDxvLoGFFhoduBv 17pg== X-Gm-Message-State: AOJu0YyECaoF5iKdF7xvIIPRH19eSJKzHhkIBNBqIjbq123qeTG/5PfA 9tz/NPBz/3XOGWwPhWciRBZKcURrFXqXaTTXKpxXZqn+UFizYu2MfKklYTfRfBwmju9rE4QYIdn GmPerutgPkZI9CFeVaZT23jRtLXkLU3kw7VV45X8HOks6uFemW+i8/o2Cn0yvD/iEfnGkSkcfJD FsuluiZtf/YOC8x3FPZPXBLZrroxODkOLS0Ah/4VI/ X-Google-Smtp-Source: AGHT+IFbhxAjjykXd0tgMaimZc1cgoRlU4UPCF92qYdiADZMvPoFvw6ifkwlYYiweKlH3xkFICuG7vtp X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:600c:5398:b0:40e:b282:7080 with SMTP id hg24-20020a05600c539800b0040eb2827080mr11390wmb.1.1706021759776; Tue, 23 Jan 2024 06:55:59 -0800 (PST) Date: Tue, 23 Jan 2024 15:53:41 +0100 In-Reply-To: <20240123145258.1462979-52-ardb+git@google.com> Mime-Version: 1.0 References: <20240123145258.1462979-52-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=9481; i=ardb@kernel.org; h=from:subject; bh=qHHYXLLvFfWa7vAyqnjwPUIdaKSvf0MywaVNE5eDiWc=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIXX9pR+Ca6d2m56tuKC1OHmxjf0Pd/e3+9atabjtZ298a 8n0yXIJHaUsDGIcDLJiiiwCs/++23l6olSt8yxZmDmsTCBDGLg4BWAiYncZ/qcsiS3jkTu//mLz 12fdMZHTnSZemaWo7CPXLsghs/H/v++MDG+f5bDVHfsf+8BEMGGTxP0t1/xVZR8fTLovH296f3F QCA8A X-Mailer: git-send-email 2.43.0.429.g432eaa2c6b-goog Message-ID: <20240123145258.1462979-94-ardb+git@google.com> Subject: [PATCH v7 42/50] arm64: kasan: Reduce minimum shadow alignment and enable 5 level paging From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240123_145601_690062_E5124C8C X-CRM114-Status: GOOD ( 26.10 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel Allow the KASAN init code to deal with 5 levels of paging, and relax the requirement that the shadow region is aligned to the top level pgd_t size. This is necessary for LPA2 based 52-bit virtual addressing, where the KASAN shadow will never be aligned to the pgd_t size. Allowing this also enables the 16k/48-bit case for KASAN, which is a nice bonus. This involves some hackery to manipulate the root and next level page tables without having to distinguish all the various configurations, including 16k/48-bits (which has a two entry pgd_t level), and LPA2 configurations running with one translation level less on non-LPA2 hardware. Signed-off-by: Ard Biesheuvel --- arch/arm64/Kconfig | 2 +- arch/arm64/mm/kasan_init.c | 148 +++++++++++++++++--- 2 files changed, 130 insertions(+), 20 deletions(-) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 8c2c36fffcf5..9ca3316d6379 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -164,7 +164,7 @@ config ARM64 select HAVE_ARCH_HUGE_VMAP select HAVE_ARCH_JUMP_LABEL select HAVE_ARCH_JUMP_LABEL_RELATIVE - select HAVE_ARCH_KASAN if !(ARM64_16K_PAGES && ARM64_VA_BITS_48) + select HAVE_ARCH_KASAN select HAVE_ARCH_KASAN_VMALLOC if HAVE_ARCH_KASAN select HAVE_ARCH_KASAN_SW_TAGS if HAVE_ARCH_KASAN select HAVE_ARCH_KASAN_HW_TAGS if (HAVE_ARCH_KASAN && ARM64_MTE) diff --git a/arch/arm64/mm/kasan_init.c b/arch/arm64/mm/kasan_init.c index a86ab99587c9..fbddbf9faf19 100644 --- a/arch/arm64/mm/kasan_init.c +++ b/arch/arm64/mm/kasan_init.c @@ -23,7 +23,7 @@ #if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS) -static pgd_t tmp_pg_dir[PTRS_PER_PGD] __initdata __aligned(PGD_SIZE); +static pgd_t tmp_pg_dir[PTRS_PER_PTE] __initdata __aligned(PAGE_SIZE); /* * The p*d_populate functions call virt_to_phys implicitly so they can't be used @@ -99,6 +99,19 @@ static pud_t *__init kasan_pud_offset(p4d_t *p4dp, unsigned long addr, int node, return early ? pud_offset_kimg(p4dp, addr) : pud_offset(p4dp, addr); } +static p4d_t *__init kasan_p4d_offset(pgd_t *pgdp, unsigned long addr, int node, + bool early) +{ + if (pgd_none(READ_ONCE(*pgdp))) { + phys_addr_t p4d_phys = early ? + __pa_symbol(kasan_early_shadow_p4d) + : kasan_alloc_zeroed_page(node); + __pgd_populate(pgdp, p4d_phys, PGD_TYPE_TABLE); + } + + return early ? p4d_offset_kimg(pgdp, addr) : p4d_offset(pgdp, addr); +} + static void __init kasan_pte_populate(pmd_t *pmdp, unsigned long addr, unsigned long end, int node, bool early) { @@ -144,12 +157,12 @@ static void __init kasan_p4d_populate(pgd_t *pgdp, unsigned long addr, unsigned long end, int node, bool early) { unsigned long next; - p4d_t *p4dp = p4d_offset(pgdp, addr); + p4d_t *p4dp = kasan_p4d_offset(pgdp, addr, node, early); do { next = p4d_addr_end(addr, end); kasan_pud_populate(p4dp, addr, next, node, early); - } while (p4dp++, addr = next, addr != end); + } while (p4dp++, addr = next, addr != end && p4d_none(READ_ONCE(*p4dp))); } static void __init kasan_pgd_populate(unsigned long addr, unsigned long end, @@ -165,19 +178,48 @@ static void __init kasan_pgd_populate(unsigned long addr, unsigned long end, } while (pgdp++, addr = next, addr != end); } +#if defined(CONFIG_ARM64_64K_PAGES) || CONFIG_PGTABLE_LEVELS > 4 +#define SHADOW_ALIGN P4D_SIZE +#else +#define SHADOW_ALIGN PUD_SIZE +#endif + +/* + * Return whether 'addr' is aligned to the size covered by a root level + * descriptor. + */ +static bool __init root_level_aligned(u64 addr) +{ + int shift = (ARM64_HW_PGTABLE_LEVELS(vabits_actual) - 1) * (PAGE_SHIFT - 3); + + return (addr % (PAGE_SIZE << shift)) == 0; +} + /* The early shadow maps everything to a single page of zeroes */ asmlinkage void __init kasan_early_init(void) { BUILD_BUG_ON(KASAN_SHADOW_OFFSET != KASAN_SHADOW_END - (1UL << (64 - KASAN_SHADOW_SCALE_SHIFT))); - /* - * We cannot check the actual value of KASAN_SHADOW_START during build, - * as it depends on vabits_actual. As a best-effort approach, check - * potential values calculated based on VA_BITS and VA_BITS_MIN. - */ - BUILD_BUG_ON(!IS_ALIGNED(_KASAN_SHADOW_START(VA_BITS), PGDIR_SIZE)); - BUILD_BUG_ON(!IS_ALIGNED(_KASAN_SHADOW_START(VA_BITS_MIN), PGDIR_SIZE)); - BUILD_BUG_ON(!IS_ALIGNED(KASAN_SHADOW_END, PGDIR_SIZE)); + BUILD_BUG_ON(!IS_ALIGNED(_KASAN_SHADOW_START(VA_BITS), SHADOW_ALIGN)); + BUILD_BUG_ON(!IS_ALIGNED(_KASAN_SHADOW_START(VA_BITS_MIN), SHADOW_ALIGN)); + BUILD_BUG_ON(!IS_ALIGNED(KASAN_SHADOW_END, SHADOW_ALIGN)); + + if (!root_level_aligned(KASAN_SHADOW_START)) { + /* + * The start address is misaligned, and so the next level table + * will be shared with the linear region. This can happen with + * 4 or 5 level paging, so install a generic pte_t[] as the + * next level. This prevents the kasan_pgd_populate call below + * from inserting an entry that refers to the shared KASAN zero + * shadow pud_t[]/p4d_t[], which could end up getting corrupted + * when the linear region is mapped. + */ + static pte_t tbl[PTRS_PER_PTE] __page_aligned_bss; + pgd_t *pgdp = pgd_offset_k(KASAN_SHADOW_START); + + set_pgd(pgdp, __pgd(__pa_symbol(tbl) | PGD_TYPE_TABLE)); + } + kasan_pgd_populate(KASAN_SHADOW_START, KASAN_SHADOW_END, NUMA_NO_NODE, true); } @@ -189,20 +231,75 @@ static void __init kasan_map_populate(unsigned long start, unsigned long end, kasan_pgd_populate(start & PAGE_MASK, PAGE_ALIGN(end), node, false); } -static void __init clear_pgds(unsigned long start, - unsigned long end) +/* + * Return the descriptor index of 'addr' in the root level table + */ +static int __init root_level_idx(u64 addr) { /* - * Remove references to kasan page tables from - * swapper_pg_dir. pgd_clear() can't be used - * here because it's nop on 2,3-level pagetable setups + * On 64k pages, the TTBR1 range root tables are extended for 52-bit + * virtual addressing, and TTBR1 will simply point to the pgd_t entry + * that covers the start of the 48-bit addressable VA space if LVA is + * not implemented. This means we need to index the table as usual, + * instead of masking off bits based on vabits_actual. */ - for (; start < end; start += PGDIR_SIZE) - set_pgd(pgd_offset_k(start), __pgd(0)); + u64 vabits = IS_ENABLED(CONFIG_ARM64_64K_PAGES) ? VA_BITS + : vabits_actual; + int shift = (ARM64_HW_PGTABLE_LEVELS(vabits) - 1) * (PAGE_SHIFT - 3); + + return (addr & ~_PAGE_OFFSET(vabits)) >> (shift + PAGE_SHIFT); +} + +/* + * Clone a next level table from swapper_pg_dir into tmp_pg_dir + */ +static void __init clone_next_level(u64 addr, pgd_t *tmp_pg_dir, pud_t *pud) +{ + int idx = root_level_idx(addr); + pgd_t pgd = READ_ONCE(swapper_pg_dir[idx]); + pud_t *pudp = (pud_t *)__phys_to_kimg(__pgd_to_phys(pgd)); + + memcpy(pud, pudp, PAGE_SIZE); + tmp_pg_dir[idx] = __pgd(__phys_to_pgd_val(__pa_symbol(pud)) | + PUD_TYPE_TABLE); +} + +/* + * Return the descriptor index of 'addr' in the next level table + */ +static int __init next_level_idx(u64 addr) +{ + int shift = (ARM64_HW_PGTABLE_LEVELS(vabits_actual) - 2) * (PAGE_SHIFT - 3); + + return (addr >> (shift + PAGE_SHIFT)) % PTRS_PER_PTE; +} + +/* + * Dereference the table descriptor at 'pgd_idx' and clear the entries from + * 'start' to 'end' (exclusive) from the table. + */ +static void __init clear_next_level(int pgd_idx, int start, int end) +{ + pgd_t pgd = READ_ONCE(swapper_pg_dir[pgd_idx]); + pud_t *pudp = (pud_t *)__phys_to_kimg(__pgd_to_phys(pgd)); + + memset(&pudp[start], 0, (end - start) * sizeof(pud_t)); +} + +static void __init clear_shadow(u64 start, u64 end) +{ + int l = root_level_idx(start), m = root_level_idx(end); + + if (!root_level_aligned(start)) + clear_next_level(l++, next_level_idx(start), PTRS_PER_PTE); + if (!root_level_aligned(end)) + clear_next_level(m, 0, next_level_idx(end)); + memset(&swapper_pg_dir[l], 0, (m - l) * sizeof(pgd_t)); } static void __init kasan_init_shadow(void) { + static pud_t pud[2][PTRS_PER_PUD] __initdata __aligned(PAGE_SIZE); u64 kimg_shadow_start, kimg_shadow_end; u64 mod_shadow_start; u64 vmalloc_shadow_end; @@ -224,10 +321,23 @@ static void __init kasan_init_shadow(void) * setup will be finished. */ memcpy(tmp_pg_dir, swapper_pg_dir, sizeof(tmp_pg_dir)); + + /* + * If the start or end address of the shadow region is not aligned to + * the root level size, we have to allocate a temporary next-level table + * in each case, clone the next level of descriptors, and install the + * table into tmp_pg_dir. Note that with 5 levels of paging, the next + * level will in fact be p4d_t, but that makes no difference in this + * case. + */ + if (!root_level_aligned(KASAN_SHADOW_START)) + clone_next_level(KASAN_SHADOW_START, tmp_pg_dir, pud[0]); + if (!root_level_aligned(KASAN_SHADOW_END)) + clone_next_level(KASAN_SHADOW_END, tmp_pg_dir, pud[1]); dsb(ishst); cpu_replace_ttbr1(lm_alias(tmp_pg_dir)); - clear_pgds(KASAN_SHADOW_START, KASAN_SHADOW_END); + clear_shadow(KASAN_SHADOW_START, KASAN_SHADOW_END); kasan_map_populate(kimg_shadow_start, kimg_shadow_end, early_pfn_to_nid(virt_to_pfn(lm_alias(KERNEL_START)))); From patchwork Tue Jan 23 14:53:42 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13527568 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id D53BDC47258 for ; Tue, 23 Jan 2024 14:58:47 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=yX6+o3ZBSa7AG25yHXONDJjuocj8pBOQDKyHWeyXc90=; b=jmm/v9Me/WrmvDzDDyGJoPLRWA Q9kN5dGLVFG0yj0eUifXaC1mkuuax0r+M34q3G2UHjrlcnc5/dXRtC4zHe/d6SM3QP1wxmrwrHvbw 14g90wVX9GsHx293uCPa/uLcEZW7bKdJYXa5giPd20xDdjYk6EZDsBXMjadeN317bD1KQerxIfw/v itCURhYvEQqFXmuobDA14Ri0AiXmtI5bXC7LMVlJhMs+QEDnoXtPT6bnqMTfI11oB/N+TPcukdLsS IxBtua1k42q1FpfZ2hapiGUHXRiNtugYNyBuGyimncUKQ7cSh8UP/y2nSPKoh8+P8r4ZbRhRhJE9l Yt2Q3emQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rSIED-00GvjK-33; Tue, 23 Jan 2024 14:58:17 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rSIC7-00GuUl-24 for linux-arm-kernel@bombadil.infradead.org; Tue, 23 Jan 2024 14:56:07 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=JyAZn2RcosymXegNM8cU1SacmF3VpXDlQDuPHJoSCoU=; b=ULQ79X28vqZXP3A+Qr0aHzgh58 k/dNefCK8TE9OZhIjSlFskEYg8fQbxnyXUvbZkxL5k7Z00EbwDZdUBTkZ/rIeMhsGsZgISAC1xhHa 3/0LrqBXTsBCUvDS9tf+5vjTo+6yWXUMvksPvathlFLLVC9I3AyPdFMdrP25ESNVgSUsUgLdlvLyT i7c7wvUvYg9vQ23IJD+N187kYAsVKnjcqCSf/RUSC/g8hUI6a1HuwHfbVa/SArc841AdMLyhmeyq+ oyNHRTfQcLeTPlzV1HYZyeW2pS2aX9PSU7JLBQFM3YueQgP6MzUXYOwi0RNI+1qUpNHA1E5QSWn6N L1wQd4+Q==; Received: from mail-wm1-x349.google.com ([2a00:1450:4864:20::349]) by desiato.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rSIC4-00000003vcg-0q1M for linux-arm-kernel@lists.infradead.org; Tue, 23 Jan 2024 14:56:06 +0000 Received: by mail-wm1-x349.google.com with SMTP id 5b1f17b1804b1-40e479a51e4so24591135e9.0 for ; Tue, 23 Jan 2024 06:56:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1706021762; x=1706626562; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=JyAZn2RcosymXegNM8cU1SacmF3VpXDlQDuPHJoSCoU=; b=QMQXBvUJygFCR/Xiz1VUgBJZS7vWJmzD0s3jm55Z78mcdg2UN7ONaXn+ovw3Q557OL HNrv49vEhz5GSS0Vd+sploAiaB5n9NsB7N2RL1aDvBEkElKuf510aunL4nZRSzyoGwTA +vZIypf0VBJOtnYjxA/ctwDBpZEvdTgRCTxwKABc87tl3kSRH4RYfOuTVbJEB9gnnn6h 6eUa1GuMVuiCADQVzuTb2KWjwpZLyqih+G1kY0wQov4qE9l9lAMzmLGpvSFlG1aE6q4i sL+TyjSR0ksLh2l8g0yslw3a63hHyA2/CRUQanhfkIExEqShmK+yVlc96LldzH5frplN Dsww== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706021762; x=1706626562; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=JyAZn2RcosymXegNM8cU1SacmF3VpXDlQDuPHJoSCoU=; b=tj8Ku1KlnsFtoZRzZd0IAfOR2oaf9KJf5H6RyJue6ujB8TleM/xpiCUC7q3Ac7otEr FivIRA0XCGWBeeO4FH3pBymS6niDd+EDq4SwVGkoc/PEMsNwCd2TObmX8PA2yNq7XYpI 4FhUkxbsWb20kfhSOWxmm0yYCpZiwlyG1CWOT4vVDT6SmopeG7r8QBZkcRu8JacMgVjp rA9r4XY7w53CDsEwsa/LkIMiHyRJV8qBnqBB1/8IoNz/NaX56jJ5Ups4l6s5M895ogmF 3Ifd9WclgkjFMa3sHXzU9ucqoONQwMB1xoQk+X+pyE+J/MeWj9EqGu8E3knRhCGI0/V+ w+0w== X-Gm-Message-State: AOJu0Yz1OI5bfC2/C+shDfPUMLFKjKcWhgUth4OQWSv8vIm4pD1hwreO 766Nei/ZHQJLhR89I6eAV/wNCowDz+5DPAA/4diBKdS1iMT4uwSrDXs57mXtqzZo/jB78bsgD5X KIaLfXm9619YhfOTNrNRc3kB7Z4z60SCsvix7PS3ZOrAjackSZnA3664wePPSd8zv4o2yyKQbPg xQ3NKI9Ifybn7r9403tTo7B3T1Q3F5/2RG3azEzFci X-Google-Smtp-Source: AGHT+IHaP4KRFIHZtVCg0qmKDMxyQSKwCzbljxpOG8izkeEJAhf0nBhCg7+WYsNkcuMLPwUFPzIrsUaz X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:600c:1d87:b0:40e:b3e3:46f8 with SMTP id p7-20020a05600c1d8700b0040eb3e346f8mr96873wms.1.1706021761814; Tue, 23 Jan 2024 06:56:01 -0800 (PST) Date: Tue, 23 Jan 2024 15:53:42 +0100 In-Reply-To: <20240123145258.1462979-52-ardb+git@google.com> Mime-Version: 1.0 References: <20240123145258.1462979-52-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=8257; i=ardb@kernel.org; h=from:subject; bh=2CEm4drm1zbpcLt871DXsU1RNMpcVpEvWcjmOHoLJ70=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIXX9pZ9qb5VSFGdNS79m1nHoV1beXe6FDj9tVux6vPd94 fMfy4MWd5SyMIhxMMiKKbIIzP77bufpiVK1zrNkYeawMoEMYeDiFICJiLxjZLjHI8aY7H1fkufc /NWcPR+/MN5vZQ49wjvH1EL0TYbQL1FGhlcyt91zDSeY7ClNkndSnam8u8Jtae6mr1uvLNIy32R yhBUA X-Mailer: git-send-email 2.43.0.429.g432eaa2c6b-goog Message-ID: <20240123145258.1462979-95-ardb+git@google.com> Subject: [PATCH v7 43/50] arm64: mm: Add support for folding PUDs at runtime From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240123_145604_490664_6766A9FB X-CRM114-Status: GOOD ( 19.30 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel In order to support LPA2 on 16k pages in a way that permits non-LPA2 systems to run the same kernel image, we have to be able to fall back to at most 48 bits of virtual addressing. Falling back to 48 bits would result in a level 0 with only 2 entries, which is suboptimal in terms of TLB utilization. So instead, let's fall back to 47 bits in that case. This means we need to be able to fold PUDs dynamically, similar to how we fold P4Ds for 48 bit virtual addressing on LPA2 with 4k pages. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/pgalloc.h | 12 ++- arch/arm64/include/asm/pgtable.h | 87 +++++++++++++++++--- arch/arm64/include/asm/tlb.h | 3 + arch/arm64/kernel/cpufeature.c | 2 + arch/arm64/mm/mmu.c | 2 +- arch/arm64/mm/pgd.c | 2 + 6 files changed, 95 insertions(+), 13 deletions(-) diff --git a/arch/arm64/include/asm/pgalloc.h b/arch/arm64/include/asm/pgalloc.h index cae8c648f462..aeba2cf15a25 100644 --- a/arch/arm64/include/asm/pgalloc.h +++ b/arch/arm64/include/asm/pgalloc.h @@ -14,6 +14,7 @@ #include #define __HAVE_ARCH_PGD_FREE +#define __HAVE_ARCH_PUD_FREE #include #define PGD_SIZE (PTRS_PER_PGD * sizeof(pgd_t)) @@ -43,7 +44,8 @@ static inline void __pud_populate(pud_t *pudp, phys_addr_t pmdp, pudval_t prot) static inline void __p4d_populate(p4d_t *p4dp, phys_addr_t pudp, p4dval_t prot) { - set_p4d(p4dp, __p4d(__phys_to_p4d_val(pudp) | prot)); + if (pgtable_l4_enabled()) + set_p4d(p4dp, __p4d(__phys_to_p4d_val(pudp) | prot)); } static inline void p4d_populate(struct mm_struct *mm, p4d_t *p4dp, pud_t *pudp) @@ -53,6 +55,14 @@ static inline void p4d_populate(struct mm_struct *mm, p4d_t *p4dp, pud_t *pudp) p4dval |= (mm == &init_mm) ? P4D_TABLE_UXN : P4D_TABLE_PXN; __p4d_populate(p4dp, __pa(pudp), p4dval); } + +static inline void pud_free(struct mm_struct *mm, pud_t *pud) +{ + if (!pgtable_l4_enabled()) + return; + BUG_ON((unsigned long)pud & (PAGE_SIZE-1)); + free_page((unsigned long)pud); +} #else static inline void __p4d_populate(p4d_t *p4dp, phys_addr_t pudp, p4dval_t prot) { diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 3d7fb3cde83d..b3c716fa8121 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -759,12 +759,27 @@ static inline pmd_t *pud_pgtable(pud_t pud) #if CONFIG_PGTABLE_LEVELS > 3 +static __always_inline bool pgtable_l4_enabled(void) +{ + if (CONFIG_PGTABLE_LEVELS > 4 || !IS_ENABLED(CONFIG_ARM64_LPA2)) + return true; + if (!alternative_has_cap_likely(ARM64_ALWAYS_BOOT)) + return vabits_actual == VA_BITS; + return alternative_has_cap_unlikely(ARM64_HAS_VA52); +} + +static inline bool mm_pud_folded(const struct mm_struct *mm) +{ + return !pgtable_l4_enabled(); +} +#define mm_pud_folded mm_pud_folded + #define pud_ERROR(e) \ pr_err("%s:%d: bad pud %016llx.\n", __FILE__, __LINE__, pud_val(e)) -#define p4d_none(p4d) (!p4d_val(p4d)) -#define p4d_bad(p4d) (!(p4d_val(p4d) & 2)) -#define p4d_present(p4d) (p4d_val(p4d)) +#define p4d_none(p4d) (pgtable_l4_enabled() && !p4d_val(p4d)) +#define p4d_bad(p4d) (pgtable_l4_enabled() && !(p4d_val(p4d) & 2)) +#define p4d_present(p4d) (!p4d_none(p4d)) static inline void set_p4d(p4d_t *p4dp, p4d_t p4d) { @@ -780,7 +795,8 @@ static inline void set_p4d(p4d_t *p4dp, p4d_t p4d) static inline void p4d_clear(p4d_t *p4dp) { - set_p4d(p4dp, __p4d(0)); + if (pgtable_l4_enabled()) + set_p4d(p4dp, __p4d(0)); } static inline phys_addr_t p4d_page_paddr(p4d_t p4d) @@ -788,25 +804,74 @@ static inline phys_addr_t p4d_page_paddr(p4d_t p4d) return __p4d_to_phys(p4d); } +#define pud_index(addr) (((addr) >> PUD_SHIFT) & (PTRS_PER_PUD - 1)) + +static inline pud_t *p4d_to_folded_pud(p4d_t *p4dp, unsigned long addr) +{ + return (pud_t *)PTR_ALIGN_DOWN(p4dp, PAGE_SIZE) + pud_index(addr); +} + static inline pud_t *p4d_pgtable(p4d_t p4d) { return (pud_t *)__va(p4d_page_paddr(p4d)); } -/* Find an entry in the first-level page table. */ -#define pud_offset_phys(dir, addr) (p4d_page_paddr(READ_ONCE(*(dir))) + pud_index(addr) * sizeof(pud_t)) +static inline phys_addr_t pud_offset_phys(p4d_t *p4dp, unsigned long addr) +{ + BUG_ON(!pgtable_l4_enabled()); -#define pud_set_fixmap(addr) ((pud_t *)set_fixmap_offset(FIX_PUD, addr)) -#define pud_set_fixmap_offset(p4d, addr) pud_set_fixmap(pud_offset_phys(p4d, addr)) -#define pud_clear_fixmap() clear_fixmap(FIX_PUD) + return p4d_page_paddr(READ_ONCE(*p4dp)) + pud_index(addr) * sizeof(pud_t); +} -#define p4d_page(p4d) pfn_to_page(__phys_to_pfn(__p4d_to_phys(p4d))) +static inline +pud_t *pud_offset_lockless(p4d_t *p4dp, p4d_t p4d, unsigned long addr) +{ + if (!pgtable_l4_enabled()) + return p4d_to_folded_pud(p4dp, addr); + return (pud_t *)__va(p4d_page_paddr(p4d)) + pud_index(addr); +} +#define pud_offset_lockless pud_offset_lockless + +static inline pud_t *pud_offset(p4d_t *p4dp, unsigned long addr) +{ + return pud_offset_lockless(p4dp, READ_ONCE(*p4dp), addr); +} +#define pud_offset pud_offset + +static inline pud_t *pud_set_fixmap(unsigned long addr) +{ + if (!pgtable_l4_enabled()) + return NULL; + return (pud_t *)set_fixmap_offset(FIX_PUD, addr); +} + +static inline pud_t *pud_set_fixmap_offset(p4d_t *p4dp, unsigned long addr) +{ + if (!pgtable_l4_enabled()) + return p4d_to_folded_pud(p4dp, addr); + return pud_set_fixmap(pud_offset_phys(p4dp, addr)); +} + +static inline void pud_clear_fixmap(void) +{ + if (pgtable_l4_enabled()) + clear_fixmap(FIX_PUD); +} /* use ONLY for statically allocated translation tables */ -#define pud_offset_kimg(dir,addr) ((pud_t *)__phys_to_kimg(pud_offset_phys((dir), (addr)))) +static inline pud_t *pud_offset_kimg(p4d_t *p4dp, u64 addr) +{ + if (!pgtable_l4_enabled()) + return p4d_to_folded_pud(p4dp, addr); + return (pud_t *)__phys_to_kimg(pud_offset_phys(p4dp, addr)); +} + +#define p4d_page(p4d) pfn_to_page(__phys_to_pfn(__p4d_to_phys(p4d))) #else +static inline bool pgtable_l4_enabled(void) { return false; } + #define p4d_page_paddr(p4d) ({ BUILD_BUG(); 0;}) /* Match pud_offset folding in */ diff --git a/arch/arm64/include/asm/tlb.h b/arch/arm64/include/asm/tlb.h index 0150deb332af..a947c6e784ed 100644 --- a/arch/arm64/include/asm/tlb.h +++ b/arch/arm64/include/asm/tlb.h @@ -103,6 +103,9 @@ static inline void __pud_free_tlb(struct mmu_gather *tlb, pud_t *pudp, { struct ptdesc *ptdesc = virt_to_ptdesc(pudp); + if (!pgtable_l4_enabled()) + return; + pagetable_pud_dtor(ptdesc); tlb_remove_ptdesc(tlb, ptdesc); } diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c index bc5e4e569864..94f035f6c421 100644 --- a/arch/arm64/kernel/cpufeature.c +++ b/arch/arm64/kernel/cpufeature.c @@ -1767,6 +1767,8 @@ static int __init __kpti_install_ng_mappings(void *__unused) if (levels == 5 && !pgtable_l5_enabled()) levels = 4; + else if (levels == 4 && !pgtable_l4_enabled()) + levels = 3; remap_fn = (void *)__pa_symbol(idmap_kpti_install_ng_mappings); diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index 8e5b3a7c5afd..b131ed31a6c8 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -1065,7 +1065,7 @@ static void free_empty_pud_table(p4d_t *p4dp, unsigned long addr, free_empty_pmd_table(pudp, addr, next, floor, ceiling); } while (addr = next, addr < end); - if (CONFIG_PGTABLE_LEVELS <= 3) + if (!pgtable_l4_enabled()) return; if (!pgtable_range_aligned(start, end, floor, ceiling, P4D_MASK)) diff --git a/arch/arm64/mm/pgd.c b/arch/arm64/mm/pgd.c index 3c4f8a279d2b..0c501cabc238 100644 --- a/arch/arm64/mm/pgd.c +++ b/arch/arm64/mm/pgd.c @@ -21,6 +21,8 @@ static bool pgdir_is_page_size(void) { if (PGD_SIZE == PAGE_SIZE) return true; + if (CONFIG_PGTABLE_LEVELS == 4) + return !pgtable_l4_enabled(); if (CONFIG_PGTABLE_LEVELS == 5) return !pgtable_l5_enabled(); return false; From patchwork Tue Jan 23 14:53:43 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13527717 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 8EEB1C47DDB for ; Tue, 23 Jan 2024 16:00:39 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=IEXrLhwOWiRxJ0PkJGqWBLtQVNWCStDo4GFsNe+U0QA=; b=WxKVAhBwX3O+a0UbRcjUjNXODI mV8kKhhfMMzY0ywKqta3WepejMjC3jXP7cg4nUTihohpEd4qE6bDnUqMVsy28WjMf2vSeIjXykLsz G4UyA6sWFTdp7xL6SRE5Euj9OPm045OaGyGd8lfCSHmXsyFVfAgIj0opqMl1Df421dyxQQF1Egqo8 YueSGiR5q6Kj4jJqF+/hXJWCcF9/hutiJBF8m+xXcDnL38eu2uwOVwiUfdMhrHbKMdtwK8aBq7v8k EloC3d/w/Vvb5UgY4d94e6wEPt0LJXSHgrFdg5Mn9H/wmo3szoF3qpSbM9bomsWV6wIFxQlJptZ8P x73F1GlA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rSJCX-00HAvs-0F; Tue, 23 Jan 2024 16:00:37 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rSIC9-00GuVO-0M for linux-arm-kernel@bombadil.infradead.org; Tue, 23 Jan 2024 14:56:09 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=S4Bndw/zTFmap5t3jLATpGIBfwHi768410CCT9rElcA=; b=JW+brnEUw/x6hwW5BPVj4zEUtn CtBaFkWqQg7aJD1FlWzfxEc8G4368HCQK5gtGVH5FNim5C2aZQSd7vgRL3wC2seEW86gkYCotLYSq Z/NQtsReeQIdJmTTVWCaEBw15gFmJQI+/whzxWpgeLa4amSCqihKUMzDYLTtjq0PWRtTKT6wQADwF F4PP9xhgYJ6AAym/owe9ByyM0WtafDY39JyjepShEPYljOoHYXdguVy1NyKYfItZtv+TvvAcA0j7B AYANv15gFklHplA+ltDyjZpQz+YVhXXF8DKC4rhsJfQ/YtL5kWzksHZPmn/gkhWjgiYFto4egDjSM IKolZ84w==; Received: from mail-yb1-xb4a.google.com ([2607:f8b0:4864:20::b4a]) by desiato.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rSIC5-00000003vdg-46pq for linux-arm-kernel@lists.infradead.org; Tue, 23 Jan 2024 14:56:07 +0000 Received: by mail-yb1-xb4a.google.com with SMTP id 3f1490d57ef6-dc2358dce6bso4923742276.3 for ; Tue, 23 Jan 2024 06:56:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1706021764; x=1706626564; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=S4Bndw/zTFmap5t3jLATpGIBfwHi768410CCT9rElcA=; b=gNkoJrogoRmxvnWUDj5O0Q4DRztKXQyIUn+V090HK/W0YxEnYkxJi8BC36/21rf3yy XJo5Dq2C8ArjtLNf8uWCipxMejbK6U5tDObjedYvVG8hklzFSCwXK0afjiD8lJLcXC2D fMFKdhwv030xFrazE3FlBfryS7wRkSrF71hybFWoMJXenjIhvoBioZDfagmTb0kHx0yY adh20yXCgaZ+tKexnhLsphh+A8lBqokY6D8JYMPFRJhpo/+yZIa5fZ2ihJD7/EkmyEML k6wAgdRAF++mJ71ETdbSzubTpUAGsF12bMzL+EevJUMM84wSblg/R3On1/IFq4kv4RyE d1Cg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706021764; x=1706626564; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=S4Bndw/zTFmap5t3jLATpGIBfwHi768410CCT9rElcA=; b=Shub7+30sgUQh9yVoDk7DWjETa7iNMsMKsfg/eBQxw4/SAlZkd8J1+o396p8s8g9Mw 0YI/cM6xu5WdkkgfaVjewqJDpguAr/KEnTRWrQWgzlkptLbcsNP6+toMXGcNJIRFH2t6 2KyuydvEglRMKRuz5jOE8RvOQaqB0sjEUhAk8MK8jSOHQvDoNU1ZzfwDoiWcooAiyITp vLHMzEujvYszRsNHshgMiU2nGqigJOtm8basJLO+dBqAJdAvG2jvmOwJ3if6ZzJ0qmi3 y9JCkqPe8cW/f+oIm6ytGnae8kVbZ9dByB2R1nltOII/OxuDXIbpTAtSz2gIwBPJvK4i IZcQ== X-Gm-Message-State: AOJu0Yzo1pI03058akHuN7EVa+tA+kL4blnfOgPg/Ij5ZYlvunVhx4d8 /mzJyQAmopiomSqU+zCPjxB96/g43+Y4sYt63KBb9WTeMxU03qHPnfMdUzZXZiIz1rTlDcEVsS3 mF82BMvnDbIiejtN3k28Xs3FfND0wfKsITQoPfLgHiCYrp3tUD+Y810pdlW/4zo2+IEJu4uvELH nUIabL+nKJey1jlLye3HrGQvasvkZa8tdLB1w4CzVC X-Google-Smtp-Source: AGHT+IFrku82qxsL8fFzPjVGnOTmZxXcSh6RQnd1Z2fzUpSmrCtT2OR/VHBtPyQ7emppfpzDOIc7RYPN X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:6902:2288:b0:dbe:a220:68ee with SMTP id dn8-20020a056902228800b00dbea22068eemr328562ybb.0.1706021764275; Tue, 23 Jan 2024 06:56:04 -0800 (PST) Date: Tue, 23 Jan 2024 15:53:43 +0100 In-Reply-To: <20240123145258.1462979-52-ardb+git@google.com> Mime-Version: 1.0 References: <20240123145258.1462979-52-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=1822; i=ardb@kernel.org; h=from:subject; bh=GzVcNg2DAplbAwleBFPVIa5YpcJERq8gk5D5Y6f5riQ=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIXX9pV9B5snv3qf+4Xa5WRFZyXkw5saJmK1HFrswKhsaX NcM2pjTUcrCIMbBICumyCIw+++7nacnStU6z5KFmcPKBDKEgYtTACaSr83I0O6fN2fmPvWgKpu5 e/UKBPoOlT9w1n1p5u8517/o99LeREaGlpbvWbIvvHmrG0NTp8nkzU4/HXHyeVTfobBtAf9iP8m zAAA= X-Mailer: git-send-email 2.43.0.429.g432eaa2c6b-goog Message-ID: <20240123145258.1462979-96-ardb+git@google.com> Subject: [PATCH v7 44/50] arm64: ptdump: Disregard unaddressable VA space From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240123_145606_187884_DB6FA0E0 X-CRM114-Status: GOOD ( 16.95 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel Configurations built with support for 52-bit virtual addressing can also run on CPUs that only support 48 bits of VA space, in which case only that part of swapper_pg_dir that represents the 48-bit addressable region is relevant, and everything else is ignored by the hardware. Our software pagetable walker has little in the way of input address validation, and so it will happily start a walk from an address that is not representable by the number of paging levels that are actually active, resulting in lots of bogus output from the page table dumper unless we take care to start at a valid address. So define the start address at runtime based on vabits_actual. Signed-off-by: Ard Biesheuvel --- arch/arm64/mm/ptdump.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/arch/arm64/mm/ptdump.c b/arch/arm64/mm/ptdump.c index 5f0849528ccf..16d0cf1d85c4 100644 --- a/arch/arm64/mm/ptdump.c +++ b/arch/arm64/mm/ptdump.c @@ -313,7 +313,6 @@ static void __init ptdump_initialize(void) static struct ptdump_info kernel_ptdump_info __ro_after_init = { .mm = &init_mm, - .base_addr = PAGE_OFFSET, }; void ptdump_check_wx(void) @@ -329,7 +328,7 @@ void ptdump_check_wx(void) .ptdump = { .note_page = note_page, .range = (struct ptdump_range[]) { - {PAGE_OFFSET, ~0UL}, + {_PAGE_OFFSET(vabits_actual), ~0UL}, {0, 0} } } @@ -370,6 +369,7 @@ static int __init ptdump_init(void) static struct addr_marker address_markers[ARRAY_SIZE(m)] __ro_after_init; kernel_ptdump_info.markers = memcpy(address_markers, m, sizeof(m)); + kernel_ptdump_info.base_addr = page_offset; ptdump_initialize(); ptdump_debugfs_register(&kernel_ptdump_info, "kernel_page_tables"); From patchwork Tue Jan 23 14:53:44 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13527723 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 22718C47DDB for ; Tue, 23 Jan 2024 16:01:10 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=04zfY9IzUlLoxTlQaA9sOSqFOPaE42qQiMDtO89ymAs=; b=LgrnXDa35EoZFfZNX9wjHZ09bI +xZfq0i6uN//CW/n70cvM/253DxevKEL2969f14yfEZEo0wczI9BlMj14L3iyepVmx561FQhEb2wq JYDUSUN9VFFJVh4FRSErYDC6qq7XZg+PCA7Yp0nAg9yVFvxmn8gmjo5c1w2gl6zdZ/iTt1iduSzhB +imnxDVqYD5yFp89qY4gQdIrMxpvUbxS/k0CRUEA7GN+g5jAL32At7ny5syZT1LukFee9FKUumOzv zrZNy2mxVaMePIxKcSnfU05/W7VbUMbZjkDb0d/Zg7lk3HJRbKpVk2kkV1JIs9YNFNHhZG0E/rCrN m2IV4+8A==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rSJCX-00HAwF-19; Tue, 23 Jan 2024 16:00:37 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rSICA-00GuWM-3B for linux-arm-kernel@bombadil.infradead.org; Tue, 23 Jan 2024 14:56:11 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=VMarVLfe4gjFngr6LcZaQg32BKdVjHDZTQiQmj2hzdg=; b=NDrdAF+/43nphl2oyMRTkirJ0q Vb3nD0DEeyhnOGE9a1pFhCh5smKnYJLBw513weNFDDPdA+K3sMJcBpPqXQF6VebB1vQNdfawY8cbL 1kDg3gHXGxK5JqCNM5bJXKw6CANg0Ou7hWNifIT1j5j3KKIBjNeiVU+XZl+hDPQHpi1i8SB+T0z8W NOXDzrKrEZV01ghlY0ZtHD4dEo9NoAgZoUyyY378WNp3QlWZlD36rA4jHqzA3/KnJVQUapsMHQtKx WedCqnmo88jeovLPkXmsyRhhkvHTZ+8lkpV2ia41hXyjb8kX+D6oSNIcs84/VIHr+RiS7pE8o9ooz FW1Lt9zg==; Received: from mail-wr1-x449.google.com ([2a00:1450:4864:20::449]) by desiato.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rSIC8-00000003veK-0BpZ for linux-arm-kernel@lists.infradead.org; Tue, 23 Jan 2024 14:56:09 +0000 Received: by mail-wr1-x449.google.com with SMTP id ffacd0b85a97d-339221f285eso2403612f8f.2 for ; Tue, 23 Jan 2024 06:56:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1706021766; x=1706626566; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=VMarVLfe4gjFngr6LcZaQg32BKdVjHDZTQiQmj2hzdg=; b=KELBQ4YMwBZPldQesqz2qZEvNiwboJx5191tT9hQ7DqI+1EC9/mAI5zjPd0P09OHlL 7XkjICxoFLQWku65SaBkjCzh6Y6PuF6zky9iCyHrV1XWWKxTwxizSVJtksA2t7vzgnJB TJHpmM27zuDx1Pb8QlwaH7QjxTB8P0l94BeDTWJhmdh/5SP3LSTruCzcKutuk636xmOo b7/GrWiwh0jcxOnl0aap+yo7F/7drCCH8OQs6Tr3DAWGjoZF1w95kwAkRSTtDAdFDSuU QukDxd1mfCEqG2ZK2HdBJXr4xPnWx0Xw8htgPVi7gDGcf4EfaacQYCW+YRaApn41pf1w 1GTA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706021766; x=1706626566; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=VMarVLfe4gjFngr6LcZaQg32BKdVjHDZTQiQmj2hzdg=; b=OLPUvASWWD1nx3A12/Zq4prNqjGBBViLgTaMcvBXrpcmJrjyyfoa7DeA9FfRP05C2d RcW2wgaPaHLy3gLPqsaAPxPFKsGHF8/I7dWZ31/fxUiIc7w6xFzntT9j5qNWELBk+hC9 xv5ayebqB+hSJPsGidqRCeg/l2Uckn0eoJdu+tLbnp788aYQwoXvT34PwyU1n+mV+3gU /cx633kCMb1GUjUTBlAA+0vhyXie+B1Ex3VPCDmLprzhCjAHl49oHoYLt1RwKhxpTbrP /H2Av8qAeYdvCtDcoK9z7bKyqrjyFab9cyl3/UNB9HeVpFhexh4J9z0rW22Ro93+kWu0 k1Qg== X-Gm-Message-State: AOJu0YyhbH/hMHanPXU+Zmf2LGEZp8jFuZ5rJbTWgZaj83Ajbjf5pvJy wnZHFWWZyLzu8a5welOSSNWg8H8U6GUknUVjOngv47BDQwMZ+P3VJLh7yT9NKdmPIHV/nm2pDyK xKoh9u0CRF5vefFGwq1Pg+Q6RxeXxxZECae7EHyPEftG9srXpmQKeOq5QAHQI70uuATLUUXoBii 5EmuKRHfGITJb6qymENoYrM2CBHPctsOdIlLcL6Vj8 X-Google-Smtp-Source: AGHT+IFLBxD3pTIgQuEoHpVcE8uAL6jxcMp0MNMk99dpRnOWWPTx7qIZKoi+ZwMJrKBj2+scS20b/tS0 X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a5d:5a83:0:b0:339:4215:7043 with SMTP id bp3-20020a5d5a83000000b0033942157043mr3617wrb.14.1706021766622; Tue, 23 Jan 2024 06:56:06 -0800 (PST) Date: Tue, 23 Jan 2024 15:53:44 +0100 In-Reply-To: <20240123145258.1462979-52-ardb+git@google.com> Mime-Version: 1.0 References: <20240123145258.1462979-52-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=2891; i=ardb@kernel.org; h=from:subject; bh=Uu+Q3dt7LiHKYQlCeNuQ8VnMCmxBdnpZ5P40dLayepg=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIXX9pd8fnsjGvf7UEDahrOMb9+7lOxvWP7oZXc+sFztZI OZRnV91RykLgxgHg6yYIovA7L/vdp6eKFXrPEsWZg4rE8gQBi5OAZjI215Ght3VcyPv7nwXvC/l bPtnzxXZN0Iqst8Vnfy2JPlx2ibJ3lBGhmmr+CuTZUNyNk6Qu3jgzrt6I/fpIZvUGJdImZsq/pt tzwsA X-Mailer: git-send-email 2.43.0.429.g432eaa2c6b-goog Message-ID: <20240123145258.1462979-97-ardb+git@google.com> Subject: [PATCH v7 45/50] arm64: ptdump: Deal with translation levels folded at runtime From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240123_145608_259612_790AED55 X-CRM114-Status: GOOD ( 19.71 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel Currently, the ptdump code deals with folded PMD or PUD levels at build time, by omitting those levels when invoking note_page. IOW, note_page() is never invoked with level == 1 if P4Ds are folded in the build configuration. With the introduction of LPA2 support, we will defer some of these folding decisions to runtime, so let's take care of this by overriding the 'level' argument when this condition triggers. Substituting the PUD or PMD strings for "PGD" when the level in question is folded at build time is no longer necessary, and so the conditional expressions can be simplified. This also makes the indirection of the 'name' field unnecessary, so change that into a char[] array, and make the whole thing __ro_after_init. Note that the mm_p?d_folded() functions currently ignore their mm pointer arguments, but let's wire them up correctly anyway. Signed-off-by: Ard Biesheuvel --- arch/arm64/mm/ptdump.c | 17 ++++++++++++----- 1 file changed, 12 insertions(+), 5 deletions(-) diff --git a/arch/arm64/mm/ptdump.c b/arch/arm64/mm/ptdump.c index 16d0cf1d85c4..5b87f8d623f7 100644 --- a/arch/arm64/mm/ptdump.c +++ b/arch/arm64/mm/ptdump.c @@ -48,6 +48,7 @@ struct pg_state { struct ptdump_state ptdump; struct seq_file *seq; const struct addr_marker *marker; + const struct mm_struct *mm; unsigned long start_address; int level; u64 current_prot; @@ -144,12 +145,12 @@ static const struct prot_bits pte_bits[] = { struct pg_level { const struct prot_bits *bits; - const char *name; - size_t num; + char name[4]; + int num; u64 mask; }; -static struct pg_level pg_level[] = { +static struct pg_level pg_level[] __ro_after_init = { { /* pgd */ .name = "PGD", .bits = pte_bits, @@ -159,11 +160,11 @@ static struct pg_level pg_level[] = { .bits = pte_bits, .num = ARRAY_SIZE(pte_bits), }, { /* pud */ - .name = (CONFIG_PGTABLE_LEVELS > 3) ? "PUD" : "PGD", + .name = "PUD", .bits = pte_bits, .num = ARRAY_SIZE(pte_bits), }, { /* pmd */ - .name = (CONFIG_PGTABLE_LEVELS > 2) ? "PMD" : "PGD", + .name = "PMD", .bits = pte_bits, .num = ARRAY_SIZE(pte_bits), }, { /* pte */ @@ -227,6 +228,11 @@ static void note_page(struct ptdump_state *pt_st, unsigned long addr, int level, static const char units[] = "KMGTPE"; u64 prot = 0; + /* check if the current level has been folded dynamically */ + if ((level == 1 && mm_p4d_folded(st->mm)) || + (level == 2 && mm_pud_folded(st->mm))) + level = 0; + if (level >= 0) prot = val & pg_level[level].mask; @@ -288,6 +294,7 @@ void ptdump_walk(struct seq_file *s, struct ptdump_info *info) st = (struct pg_state){ .seq = s, .marker = info->markers, + .mm = info->mm, .level = -1, .ptdump = { .note_page = note_page, From patchwork Tue Jan 23 14:53:45 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13527731 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 3A70DC48285 for ; Tue, 23 Jan 2024 16:01:14 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=+UW/uc0YtVOLCI0yhe/eM8cLYD9SycUuoyiUcsPEC9o=; b=UyiaCoWpraUoj23MIDqriC3yed hWpnvClp9ZBIxVTesAoleQU4zwwbeA3j/8jG7nzvw9d3dKDoytR7osuKMfgDADkGbEU9YlwEbGvhS zDsCThjWnmAfcuHp9KeFDfm06tDgdPuw8+JnxYFjPP5SvREg9L1Lr3wlLIOy13in03mxX9hrOuncE 8D0tOAOk6eql4vSctxjrsge1Fj9p2EdiRhDbrb86IIFqQCxy21rSMilSgjA4Qd3fgYAEi7KJVXHgP YDmUURTQbQybhRFYaYOmkbqwe5YYnqajQXyd3XUD+xXanGncj60pK6tCwDIvfhwbu1OrOX0KU3+jp dL0B7Qng==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rSJCX-00HAwe-32; Tue, 23 Jan 2024 16:00:37 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rSICC-00GuWr-30 for linux-arm-kernel@bombadil.infradead.org; Tue, 23 Jan 2024 14:56:13 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=H9Gxd+On0iANIztLwIDzI9X4qQe9MQm66Oh2fMvJ5jQ=; b=L6bxkP5zXO7Fcpp0S6wImBTEKm hpRmUxNHpnS8r+gjBpHHpEm7V4Lfsa4tpLlRhlxnQjnF3sMb7V0wjoOYeY1K8CsnPp7TyOaSd8sdF 4VGps1uD4xGNCUj7uYWpd/zBiSiVdEDl5l2gElNcQ6f+LjymyPZwoy32kTvnHELJ3Ue3eNEW47k45 qLLGpZts7A5luP9nMi18ALLWjoy80Rh9dqCEcSuvEDACKfg63aYRjiyRhGdjFrqt1Waie0hheyeTX S7uFzhxyz0xUeC6lBA+MHCOSaCIemBM1ikYLhdMhW3LbsDKSyXLPHNR0+bL7XZSRW9DHklO4QCju3 bbe/Or7Q==; Received: from mail-wr1-x449.google.com ([2a00:1450:4864:20::449]) by desiato.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rSICA-00000003vef-0Bth for linux-arm-kernel@lists.infradead.org; Tue, 23 Jan 2024 14:56:11 +0000 Received: by mail-wr1-x449.google.com with SMTP id ffacd0b85a97d-337d2ccaab0so2568065f8f.0 for ; Tue, 23 Jan 2024 06:56:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1706021768; x=1706626568; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=H9Gxd+On0iANIztLwIDzI9X4qQe9MQm66Oh2fMvJ5jQ=; b=Jdo89uqjMg+WzJx73tMYZ2pMqJbQk1ncVf+ROs6cy7HAaKfNMgSG2pjz4SuFQ6WDr8 6ADbYiX1SiQoG9+ppu0xVOeVkHnt7kOgiACX+HzbhyWnbn+fLK7wGrb3GXJQVGygqxqe z8Fz6vddTJnUE8qcx46hCNquuMFOAhSRYScbomwsMT/mjtDcTySBE1188cHC6wrnn23A rfEjVW7AuKud9QSVel1NzRRQYUkA9wIkpKJFlf8ytZJMHOb9nZYC2jHzgLv/ikv+EuEi SY1WqXyIPfSahhkqEHOH+uVUYn5ja14R4P+TZSH8Np7Ghuzmr8mIZaSZqrYSxSwQQhTd SX3w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706021768; x=1706626568; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=H9Gxd+On0iANIztLwIDzI9X4qQe9MQm66Oh2fMvJ5jQ=; b=cyMzC9c6jjisIr4ooxbJoGGJDE24wTOaiAl+O+UmrdzWX37BnKZOI65uaDIKTDXbGq WhjHZISf+sy+PIMDupmCWCvnmsKyDf9drd9NntbAZGpskEBjOL1NQKrDjrDI7Xnl64rP 5Yd8ZLWvZfLZrb04ZNgtBT4Qx4Qb27NLXOysvD1ouHJNJC5hPyxC3V3RYqex1IeKm7FI /a4AL5C5FJBPniC627C50uzLJhTX54eujc1mPjjJ7NwprykIYycAVxRgcDuDxvnr/Apw /KDXpri/BtRKsvA4+5kO2xMA/Y5wb4igpLqGAtZGvgERQIvDnQRYzTonnFCAV+fBPkUs /cRA== X-Gm-Message-State: AOJu0YwKRVl+tymODIRIDXLmcasVUT0zRnwOAyrajD7mtDb3lDBLnRj8 gDfBIfUSGfnPF+AZDHaA7UnKv2Z4CQFAOwRlDExxqLFJSHO04nMPrsvV+vnqhduxkWUpegDmRkp jQMlVUV37JaoWiI38bW2onlM0tCVnq77fc5KcHwqEAUft3RbWIwPiveYo1naXZJ/glokyhZdxTj FfBc77NsaggOvlUJyZG6hqDbQykGL+J6qg7js8klU4 X-Google-Smtp-Source: AGHT+IHHX+AYUaLtXOIi5/0HLO6+Be9jYq+aSWT7vdliWA451s3bb99aQsyG3ouiaU3oVlcKhJXKSIqG X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:6000:71a:b0:337:dbc2:3584 with SMTP id bs26-20020a056000071a00b00337dbc23584mr17310wrb.0.1706021768693; Tue, 23 Jan 2024 06:56:08 -0800 (PST) Date: Tue, 23 Jan 2024 15:53:45 +0100 In-Reply-To: <20240123145258.1462979-52-ardb+git@google.com> Mime-Version: 1.0 References: <20240123145258.1462979-52-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=5018; i=ardb@kernel.org; h=from:subject; bh=EK/sRPp8OqikWsFrvEepygSmQvpn2J/XAp8HZowH91o=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIXX9pT+TA+wYVNpvR5WvSrw3dTGniMqpFZ6bdy7hES2on pnhXLulo5SFQYyDQVZMkUVg9t93O09PlKp1niULM4eVCWQIAxenAEwkr4/hf8Wxx3fum505/b/8 163O7E9p67YvP3XwRbT+Lx3+Bt/HG0MZ/vA1f9bZzlEidLU28NXeOUef/rZqc2Z3v+mrXzGrT2z WZW4A X-Mailer: git-send-email 2.43.0.429.g432eaa2c6b-goog Message-ID: <20240123145258.1462979-98-ardb+git@google.com> Subject: [PATCH v7 46/50] arm64: Enable 52-bit virtual addressing for 4k and 16k granule configs From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240123_145610_182650_8B00734E X-CRM114-Status: GOOD ( 16.49 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel Update Kconfig to permit 4k and 16k granule configurations to be built with 52-bit virtual addressing, now that all the prerequisites are in place. While at it, update the feature description so it matches on the appropriate feature bits depending on the page size. For simplicity, let's just keep ARM64_HAS_VA52 as the feature name. Note that LPA2 based 52-bit virtual addressing requires 52-bit physical addressing support to be enabled as well, as programming TCR.TxSZ to values below 16 is not allowed unless TCR.DS is set, which is what activates the 52-bit physical addressing support. While supporting the converse (52-bit physical addressing without 52-bit virtual addressing) would be possible in principle, let's keep things simple, by only allowing these features to be enabled at the same time. Signed-off-by: Ard Biesheuvel --- arch/arm64/Kconfig | 17 ++++++++------- arch/arm64/kernel/cpufeature.c | 22 ++++++++++++++++---- 2 files changed, 28 insertions(+), 11 deletions(-) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 9ca3316d6379..eed8fef08a10 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -368,7 +368,9 @@ config PGTABLE_LEVELS default 3 if ARM64_64K_PAGES && (ARM64_VA_BITS_48 || ARM64_VA_BITS_52) default 3 if ARM64_4K_PAGES && ARM64_VA_BITS_39 default 3 if ARM64_16K_PAGES && ARM64_VA_BITS_47 + default 4 if ARM64_16K_PAGES && (ARM64_VA_BITS_48 || ARM64_VA_BITS_52) default 4 if !ARM64_64K_PAGES && ARM64_VA_BITS_48 + default 5 if ARM64_4K_PAGES && ARM64_VA_BITS_52 config ARCH_SUPPORTS_UPROBES def_bool y @@ -396,13 +398,13 @@ config BUILTIN_RETURN_ADDRESS_STRIPS_PAC config KASAN_SHADOW_OFFSET hex depends on KASAN_GENERIC || KASAN_SW_TAGS - default 0xdfff800000000000 if (ARM64_VA_BITS_48 || ARM64_VA_BITS_52) && !KASAN_SW_TAGS - default 0xdfffc00000000000 if ARM64_VA_BITS_47 && !KASAN_SW_TAGS + default 0xdfff800000000000 if (ARM64_VA_BITS_48 || (ARM64_VA_BITS_52 && !ARM64_16K_PAGES)) && !KASAN_SW_TAGS + default 0xdfffc00000000000 if (ARM64_VA_BITS_47 || ARM64_VA_BITS_52) && ARM64_16K_PAGES && !KASAN_SW_TAGS default 0xdffffe0000000000 if ARM64_VA_BITS_42 && !KASAN_SW_TAGS default 0xdfffffc000000000 if ARM64_VA_BITS_39 && !KASAN_SW_TAGS default 0xdffffff800000000 if ARM64_VA_BITS_36 && !KASAN_SW_TAGS - default 0xefff800000000000 if (ARM64_VA_BITS_48 || ARM64_VA_BITS_52) && KASAN_SW_TAGS - default 0xefffc00000000000 if ARM64_VA_BITS_47 && KASAN_SW_TAGS + default 0xefff800000000000 if (ARM64_VA_BITS_48 || (ARM64_VA_BITS_52 && !ARM64_16K_PAGES)) && KASAN_SW_TAGS + default 0xefffc00000000000 if (ARM64_VA_BITS_47 || ARM64_VA_BITS_52) && ARM64_16K_PAGES && KASAN_SW_TAGS default 0xeffffe0000000000 if ARM64_VA_BITS_42 && KASAN_SW_TAGS default 0xefffffc000000000 if ARM64_VA_BITS_39 && KASAN_SW_TAGS default 0xeffffff800000000 if ARM64_VA_BITS_36 && KASAN_SW_TAGS @@ -1310,7 +1312,7 @@ config ARM64_VA_BITS_48 config ARM64_VA_BITS_52 bool "52-bit" - depends on ARM64_64K_PAGES && (ARM64_PAN || !ARM64_SW_TTBR0_PAN) + depends on ARM64_PAN || !ARM64_SW_TTBR0_PAN help Enable 52-bit virtual addressing for userspace when explicitly requested via a hint to mmap(). The kernel will also use 52-bit @@ -1357,10 +1359,11 @@ choice config ARM64_PA_BITS_48 bool "48-bit" + depends on ARM64_64K_PAGES || !ARM64_VA_BITS_52 config ARM64_PA_BITS_52 - bool "52-bit (ARMv8.2)" - depends on ARM64_64K_PAGES + bool "52-bit" + depends on ARM64_64K_PAGES || ARM64_VA_BITS_52 depends on ARM64_PAN || !ARM64_SW_TTBR0_PAN help Enable support for a 52-bit physical address space, introduced as diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c index 94f035f6c421..0be9296e9253 100644 --- a/arch/arm64/kernel/cpufeature.c +++ b/arch/arm64/kernel/cpufeature.c @@ -2703,15 +2703,29 @@ static const struct arm64_cpu_capabilities arm64_features[] = { }, #ifdef CONFIG_ARM64_VA_BITS_52 { - .desc = "52-bit Virtual Addressing (LVA)", .capability = ARM64_HAS_VA52, .type = ARM64_CPUCAP_BOOT_CPU_FEATURE, - .sys_reg = SYS_ID_AA64MMFR2_EL1, - .sign = FTR_UNSIGNED, + .matches = has_cpuid_feature, .field_width = 4, +#ifdef CONFIG_ARM64_64K_PAGES + .desc = "52-bit Virtual Addressing (LVA)", + .sign = FTR_SIGNED, + .sys_reg = SYS_ID_AA64MMFR2_EL1, .field_pos = ID_AA64MMFR2_EL1_VARange_SHIFT, - .matches = has_cpuid_feature, .min_field_value = ID_AA64MMFR2_EL1_VARange_52, +#else + .desc = "52-bit Virtual Addressing (LPA2)", + .sys_reg = SYS_ID_AA64MMFR0_EL1, +#ifdef CONFIG_ARM64_4K_PAGES + .sign = FTR_SIGNED, + .field_pos = ID_AA64MMFR0_EL1_TGRAN4_SHIFT, + .min_field_value = ID_AA64MMFR0_EL1_TGRAN4_52_BIT, +#else + .sign = FTR_UNSIGNED, + .field_pos = ID_AA64MMFR0_EL1_TGRAN16_SHIFT, + .min_field_value = ID_AA64MMFR0_EL1_TGRAN16_52_BIT, +#endif +#endif }, #endif {}, From patchwork Tue Jan 23 14:53:46 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13527733 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 7A59BC47DDF for ; Tue, 23 Jan 2024 16:01:15 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=cTYFzpPtX03wov+xT1JjbLR5284LPcpFpMbPsoeDpjc=; b=HoEVCURF2VMT8yALFwvgazXx+a HajxMXXa4zfLhkQ01K1kIgsO3oLhXw4XWqXiki4QXgBYY60zlbH/O4zKZArrIGvKwhRz7r09dX5yj WVUPrOz5NQEquJdRIVSgKV1aShSrKEfKAZ4EfPKxVDH02dll9ayKM19HXgfe95RAUo2kq/Uyj7096 kSjZyooY92bjqQODzwpLvG7FafNyJFLwVPwga+BAaxU8oyRWQu8ldI73Z5ihnVAw7vDNRiFhu5t2N e38mAOed4VaWsCTCih8/L8nfMVwV/2TDLxGwuXLxq3xRRmqw+LYkduAQxV7vGRdyoRlvKbFHDMlxa N88czEeA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rSJCY-00HAxI-1Y; Tue, 23 Jan 2024 16:00:38 +0000 Received: from mail-wm1-x349.google.com ([2a00:1450:4864:20::349]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rSICD-00GuWg-0l for linux-arm-kernel@lists.infradead.org; Tue, 23 Jan 2024 14:56:24 +0000 Received: by mail-wm1-x349.google.com with SMTP id 5b1f17b1804b1-40eaedb4446so16916555e9.0 for ; Tue, 23 Jan 2024 06:56:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1706021771; x=1706626571; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=qLxpExRJ+KToT+NOe16t25WMLe6A3z/DcvJ2gUMSVJg=; b=3PbaKHsNBeTU9U/fz2kSYzsIPTAJfNdp4iomWwTSAjmmnkJpYKXoNJowvHi1frBq0x l11H4ssKTy2XebxEHMZ7daaoO57nJoK9xJHuJrWilNNMlvig/xtyo9kCqEipUixGP2Bx KbOJ4/BFI48LPE1lxIg2Cw2A4S7lq91WzEEPo6Xg20jEVy484HAj35tgxN0pzvNm2XDn GpeNX05WlI5DA9t+w2IA8iULgH91nuiK5/0Am96feSMH7fuj1ntomTdVrx71wKuIt9Iw tL1I+FShPc4+1Ml0YXCgmveL7FXOziUGihxzu+dst6WhsD4ahb78VhkSYRO1HKty+sKv ZE1A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706021771; x=1706626571; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=qLxpExRJ+KToT+NOe16t25WMLe6A3z/DcvJ2gUMSVJg=; b=WUFYRT4zh6sLTe/X2fiU8Im5N8RR9MefQbySdpWaFRvCWfQIejwNT7nPvM8HF7vDz9 tfIeFdSWydPITdBkFSpcx89a8Dhzl6/6YckZNv1TXTaobs7oEmEb/lKJitCoqqw2sNJ0 F+44ZCA0uZI7o2RVuQJ1Ie/gLvd+8p1lXc+TBI+H24q5N4GEJfG9xwmFNRXNTkR7yfCh qj602p6xzf7EH/uAPewalM10l4PPBpLTmN/mNB+62dCuQQjxizcZ7WJCg3WBTnKx+gxQ OCOVBblwLbohT9iA5hPxAoIOD8azIW3tOUG96QB4OQqcWRpSRYhsf3MQuUBIe3U2+JIV oG0A== X-Gm-Message-State: AOJu0Yztvq7bjJKwTx991JhbPAlz9y9SMOmJ0Kms6Ia4Csw6qFOy+Vbs RUD0dwVQLuTlLjT1+XlH4U29by80IWLxoZS2i5+sfum/sUhf/DyetAazhsF4sU4jZPeakNmkO7d l1JDhvhUeArsTJm65rf26/Pueu3+b1B8oiOicCMSQdXx64XFDPBigMQIPnZXkg1gPJKaaGxyGrh rNDu3d/6RU6HWfsyAYCwi/LETsPisfsscqBXbQAeez X-Google-Smtp-Source: AGHT+IGu2dW93waToFDU97i/TVNVrunLzp4oPh1eEphyj2bO8wknXPfD/B90FSEr5tFX8QCVtZvmHGIu X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:600c:1f16:b0:40e:c364:b0bd with SMTP id bd22-20020a05600c1f1600b0040ec364b0bdmr224wmb.2.1706021770861; Tue, 23 Jan 2024 06:56:10 -0800 (PST) Date: Tue, 23 Jan 2024 15:53:46 +0100 In-Reply-To: <20240123145258.1462979-52-ardb+git@google.com> Mime-Version: 1.0 References: <20240123145258.1462979-52-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=959; i=ardb@kernel.org; h=from:subject; bh=xh1UqpRMLse90QtRwfBPr+8BmLmmyswMHzRinVg9I1I=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIXX9pX85zScPn5zIPOt3d8ENl6u/Gq82rq7vEyo7+FX7R b/4mdu5HaUsDGIcDLJiiiwCs/++23l6olSt8yxZmDmsTCBDGLg4BWAib6Yw/I9abWXGGvXsdLtq M4fIh1P5LIv2NvpPOKjFYlbzIoT/nD4jw5LItltHHASPX9WcOtuPKX/p6QvFN6fstN0WNsewaju TICsA X-Mailer: git-send-email 2.43.0.429.g432eaa2c6b-goog Message-ID: <20240123145258.1462979-99-ardb+git@google.com> Subject: [PATCH v7 47/50] arm64: defconfig: Enable LPA2 support From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240123_065613_295849_DBB00A97 X-CRM114-Status: GOOD ( 12.25 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel We typically enable support in defconfig for all architectural features for which we can detect at runtime if the hardware actually supports them. Now that we have implemented support for LPA2 based 52-bit virtual addressing in a way that should not impact 48-bit operation on non-LPA2 CPU, we can do the same, and enable 52-bit virtual addressing by default. Signed-off-by: Ard Biesheuvel --- arch/arm64/configs/defconfig | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/arch/arm64/configs/defconfig b/arch/arm64/configs/defconfig index e6cf3e5d63c3..c5919a762109 100644 --- a/arch/arm64/configs/defconfig +++ b/arch/arm64/configs/defconfig @@ -76,7 +76,7 @@ CONFIG_ARCH_VEXPRESS=y CONFIG_ARCH_VISCONTI=y CONFIG_ARCH_XGENE=y CONFIG_ARCH_ZYNQMP=y -CONFIG_ARM64_VA_BITS_48=y +CONFIG_ARM64_VA_BITS_52=y CONFIG_SCHED_MC=y CONFIG_SCHED_SMT=y CONFIG_NUMA=y From patchwork Tue Jan 23 14:53:47 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13527728 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 56ECAC48260 for ; Tue, 23 Jan 2024 16:01:12 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=QOtNPJWKYhZqAMOa0wDTZBTm/NC3tZ+dHFl1n2S7Je8=; b=W44cZGO8o33dwqiL8KcIQ/p9Dq 99a6XWEzrcIISETQZJm5gwOVo5tB1Stl6ioLsLzEWorpt0Qbpkse05GMCKVR8wUOrA4KgHyuik/vW yeDMxzrLHoK/MYa02Gq4238TMcsOWyGcaJ7A7l5uHXxfucxubz9/cgHJCPVX3TsESj31geT8lIWkO tsneNFACFsyoiVE7mV+XHoTdfFLO5OpQT1p0KQveb6wmqr8h71NVPXgRJhvzOKh/qPQpqT4zikoss t5ByHTEJyn/yjyZzMlhyR2h67dMpa3pEmuUp4AxZ43Zk6mOjnOqW+O9krEyAB43/6vxd4SYG3q8VO SeTFR/CA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rSJCZ-00HAxr-07; Tue, 23 Jan 2024 16:00:39 +0000 Received: from mail-yw1-x1149.google.com ([2607:f8b0:4864:20::1149]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rSICE-00GuXU-2L for linux-arm-kernel@lists.infradead.org; Tue, 23 Jan 2024 14:56:24 +0000 Received: by mail-yw1-x1149.google.com with SMTP id 00721157ae682-5ff817c96a5so65567247b3.3 for ; Tue, 23 Jan 2024 06:56:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1706021773; x=1706626573; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=6ImYLwyRlzjAOW9Qb8r8VqsBk7uC0lvTL8XrDobDD+k=; b=b3ftRPz6xy3NwK++Q6kyJZIpK4Jiw3cn1LR+g9WwW6NOjaR9FcQVdk+GTE1PP04puH zIyajkCYb/1dQixhOb5CTDDLsAFAfMG7+aQNtBumXzab/MJsgqZODXbzq5LUezfpZ2pS BShw2TPZk8Tcpt7HRFJCpSKra1GziXJnxNu+Kdm09WCdbUB2suHyloB7TLgP1pfDvQAB d0s5CJguWrJWL5cg3PhUdXgC1GDzbMjMpI+ozda1q+IzHs5i1fZGkIQRSIYO/SQm9DjV ERoUezvT0jqZsvB98wT/RcrpkfLPTaL7sF+PcMoibW9kh1zhD+xZXH4znoNW5wC85RTH 41Pg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706021773; x=1706626573; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=6ImYLwyRlzjAOW9Qb8r8VqsBk7uC0lvTL8XrDobDD+k=; b=kBJQ8BT2/FXOvhkEZzeC4SUn7wTvOlWlIuqxHstuf1nuOVSaAhdO/CLvHcpZ6NI+l3 PzAeYDYRKIyf+z4d0FOqs/JwoKtCzyB2ViZufvZ3VHBI9C4awS9212v8K/DeTmYpu2qF Uyg50C80iidCDPjDeaApwWB9Q1/N1/tbO6bLss0VMKWtSzxPcEOAJ9bserPDwc3AagFC GJFw8faP2GY0OzOtXTatWwxiejd9K9Q+HBmbbQ4rCmjXzJQVwAYQVL3H+l0WX6pED/CA l/aB0x6PWKw23SAK4vRjpcRyW3+V/LDsPsuS6QfGmeWDwwFbGw+j2S69qPeVq4UclyYN tdJw== X-Gm-Message-State: AOJu0YygZQaaxvHJ3OcXsvfAU1oJMNoMzXtCNULCsYrfMzqmNeplNN5Y AmcKwWj2Ofr4dHKitIVNPTNdBa5DLPV3+itJu9Hhas+YVsgULD8lT5TkYxfgOGnTUjtSaUokXzc vGotDLA7MrnATiSCGnPcjmn4BUzmJAS61696TkwUc3ZbLEUwuoA7qdJnSU1d/iw1Io4Yh5EFDNH vgRKpslmWBe04PouQTYNigXX55AaCbFkggVuh1swHr X-Google-Smtp-Source: AGHT+IF212qQ+sz7tw0/MhvsgaVjN9xcMHISB/KT+pl81RPWNOnRbXxwX2WalkovnUo0nRSFigMkkWCd X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:6902:1b12:b0:db5:4766:e363 with SMTP id eh18-20020a0569021b1200b00db54766e363mr2616552ybb.6.1706021773273; Tue, 23 Jan 2024 06:56:13 -0800 (PST) Date: Tue, 23 Jan 2024 15:53:47 +0100 In-Reply-To: <20240123145258.1462979-52-ardb+git@google.com> Mime-Version: 1.0 References: <20240123145258.1462979-52-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=1750; i=ardb@kernel.org; h=from:subject; bh=u8l6jWRluTgDlURmlI8+ytPiGtIliUrIw3crfdugoiM=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIXX9pf+uLV0yvu7Fyyz0OOIPTn2aW7On3E3q993FVlvnT ley3H62o5SFQYyDQVZMkUVg9t93O09PlKp1niULM4eVCWQIAxenAEwkrZaRYfbyB8xrX3xVv20m tb5t2pl3ApvFWrcrHTPhejGfoVdkzXuGv9LCaZZHOBKniv1rtP9enJr8uvOY2M5PHxZa3+h5xel xhQkA X-Mailer: git-send-email 2.43.0.429.g432eaa2c6b-goog Message-ID: <20240123145258.1462979-100-ardb+git@google.com> Subject: [PATCH v7 48/50] mm: add arch hook to validate mmap() prot flags From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240123_065614_791216_465BD1A2 X-CRM114-Status: GOOD ( 13.48 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel Add a hook to permit architectures to perform validation on the prot flags passed to mmap(), like arch_validate_prot() does for mprotect(). This will be used by arm64 to reject PROT_WRITE+PROT_EXEC mappings on configurations that run with WXN enabled. Reviewed-by: Kees Cook Signed-off-by: Ard Biesheuvel --- include/linux/mman.h | 15 +++++++++++++++ mm/mmap.c | 3 +++ 2 files changed, 18 insertions(+) diff --git a/include/linux/mman.h b/include/linux/mman.h index 40d94411d492..f5d8c714b4bd 100644 --- a/include/linux/mman.h +++ b/include/linux/mman.h @@ -124,6 +124,21 @@ static inline bool arch_validate_flags(unsigned long flags) #define arch_validate_flags arch_validate_flags #endif +#ifndef arch_validate_mmap_prot +/* + * This is called from mmap(), which ignores unknown prot bits so the default + * is to accept anything. + * + * Returns true if the prot flags are valid + */ +static inline bool arch_validate_mmap_prot(unsigned long prot, + unsigned long addr) +{ + return true; +} +#define arch_validate_mmap_prot arch_validate_mmap_prot +#endif + /* * Optimisation macro. It is equivalent to: * (x & bit1) ? bit2 : 0 diff --git a/mm/mmap.c b/mm/mmap.c index b78e83d351d2..a20871bac792 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -1229,6 +1229,9 @@ unsigned long do_mmap(struct file *file, unsigned long addr, if (!(file && path_noexec(&file->f_path))) prot |= PROT_EXEC; + if (!arch_validate_mmap_prot(prot, addr)) + return -EACCES; + /* force arch specific MAP_FIXED handling in get_unmapped_area */ if (flags & MAP_FIXED_NOREPLACE) flags |= MAP_FIXED; From patchwork Tue Jan 23 14:53:48 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13527734 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id C8B2CC48260 for ; Tue, 23 Jan 2024 16:01:15 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=54RQh1WhrBFxaKvpKCxXIdXYxPsiMHbzdG34gyUDgS0=; b=mIOPlayc1ro2X9afSAUba8pF1o YLAfia12ppWwltbZwpcIYwe8pPAQxu4jeWRCvfLap+ApIzfhvk57iThWSNfojm6lOn7LZEnKqJrc1 NJUng+PHDHo2EeK5AEJf/daRgSwOU8ew9pLpfR7Bnmcgsa8PZvhlpC6mtQ5EerHwC+v/9FbHEujkt Os/F3nr9w67zmJjL5EwD5t2R8V+I7gHhu963Cvo5CDUAbvPW0mHQb7OerO12vvV9iwIBqkTnJDBVp E8HrjLYALT4Lp3U/6mnr5FCwHQGZx3pmuBfFee5T6WF5XpYiR3lUyMYubvAEAabOkIm5dvf6lmjCE 0e/lrNiw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rSJCZ-00HAyR-2D; Tue, 23 Jan 2024 16:00:39 +0000 Received: from mail-yb1-xb4a.google.com ([2607:f8b0:4864:20::b4a]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rSICH-00GuYI-0Y for linux-arm-kernel@lists.infradead.org; Tue, 23 Jan 2024 14:56:27 +0000 Received: by mail-yb1-xb4a.google.com with SMTP id 3f1490d57ef6-dc2470bc0bdso5827200276.1 for ; Tue, 23 Jan 2024 06:56:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1706021775; x=1706626575; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=PwChEXMOhfSSk5UVkQEnEEFGpXxCRJEoqFqOpgFeQ60=; b=OF2uAkPse3S6wV8lRkpPksoCuwlAogN9eYKDe/Cbphtkg+HMtA+uwOJrLgAA9isKRo /qbMPGVqw7yk/GQQRKs0mUVZ1rdCJZor0QvzEWsi7+TXedqxdYGmXXAnOfoMib9Bizlb mK69iVJfdy5QwYUnaRsk8BVp/IIauar/dcybkldjrrpVENQUBAzyj5h/yTbDgaQj3heo ctjAZ/AHsJzUwklp36VfOQKNtxfWiwwdL7xkDLWwTrL0PP7A46lJ0B1I6C4IwnjUhqaL m94ebo4eSuOD5joLFI/A/ny4UG49kwSfY32/cLa3iTC+v1xePJQiHZrVRmkgMpnocABB wU9A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706021775; x=1706626575; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=PwChEXMOhfSSk5UVkQEnEEFGpXxCRJEoqFqOpgFeQ60=; b=SFpo7WtekM9x2Z6b5ZM+aoo02JGY5tMTi4tciFJzZjhs2+l8OSEvcmTQ0K3UQp39lU rkBkKuL3vsyrcjbtmHobdz12atMXDXot3sZqKvJh0PwuPvNTNYL/2twElPrw3Viu+NKM b2iDXZbCOxvCJIJ4pdWs7VlFXLz/3KhlaeAvDI8O8N9GII8HT4R6qEnXmhGIiBKN42QG COiUMbWjJOcr8cpnc6qy6n1ZAOXbfRWbuB/idgrqRst+I0rdJsyfhLwt3sKE4pAQ8ICt PwfAdyZM2J5NA2SS3tM0d5KZ2t60moh5XAu1/2OQE6SvaxaSrpPMvwJtWr6O8Uja6hwT roSA== X-Gm-Message-State: AOJu0YxCEjMFziX6YL7YwB813BOmSYnE9jxI1qpUflcv0D3RG7LkGQgr RvTUi+tdnBM7ZE+AO6DRuHoqa5QyNSTbiHkPeLUYppOEiqbPk013xWBO7wk3N6ut308iFZNBf4B sypcvAe9oTQRg/1VzkLkLDS43y1ZTPKhgzH0erhhJfvW9IyO5huD7ov+9UW+cIVWz87u6olq4pJ 5bOXk+h8aN1L6Ihd/k16LJSF5iExgwFgC+JQJcMnIh X-Google-Smtp-Source: AGHT+IEt9D9QBRzCozy9bjt+4U3VbnrSzzFrcRbxGiCD+UTg/9NXK8/SMKOYblevBDOephA2u6eOfOcO X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:6902:1808:b0:dc2:3426:c9ee with SMTP id cf8-20020a056902180800b00dc23426c9eemr351100ybb.11.1706021775575; Tue, 23 Jan 2024 06:56:15 -0800 (PST) Date: Tue, 23 Jan 2024 15:53:48 +0100 In-Reply-To: <20240123145258.1462979-52-ardb+git@google.com> Mime-Version: 1.0 References: <20240123145258.1462979-52-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=10010; i=ardb@kernel.org; h=from:subject; bh=0KTS8nFXM9jTbRHlVy3vHRUQjdzSj4cuWECIYmEsiZg=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIXX9ZYbrxj2zL9gEbp30uj7kcL7R37eyi2O/nKnSv6X9U nfnucg5HaUsDGIcDLJiiiwCs/++23l6olSt8yxZmDmsTCBDGLg4BWAiiQEMv9neuS5pfL2hrVA8 d19ye9vWl6lXOO5bahUs41RZsnOVhTYjw4v1Ypt+pQfflL6udvhk/jPt+y4JQoz3Jsa55JvPV2l h5QYA X-Mailer: git-send-email 2.43.0.429.g432eaa2c6b-goog Message-ID: <20240123145258.1462979-101-ardb+git@google.com> Subject: [PATCH v7 49/50] arm64: mm: add support for WXN memory translation attribute From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240123_065617_229485_8439B092 X-CRM114-Status: GOOD ( 29.64 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel The AArch64 virtual memory system supports a global WXN control, which can be enabled to make all writable mappings implicitly no-exec. This is a useful hardening feature, as it prevents mistakes in managing page table permissions from being exploited to attack the system. When enabled at EL1, the restrictions apply to both EL1 and EL0. EL1 is completely under our control, and has been cleaned up to allow WXN to be enabled from boot onwards. EL0 is not under our control, but given that widely deployed security features such as selinux or PaX already limit the ability of user space to create mappings that are writable and executable at the same time, the impact of enabling this for EL0 is expected to be limited. (For this reason, common user space libraries that have a legitimate need for manipulating executable code already carry fallbacks such as [0].) If enabled at compile time, the feature can still be disabled at boot if needed, by passing arm64.nowxn on the kernel command line. [0] https://github.com/libffi/libffi/blob/master/src/closures.c#L440 Signed-off-by: Ard Biesheuvel Reviewed-by: Kees Cook --- arch/arm64/Kconfig | 11 ++++++ arch/arm64/include/asm/cpufeature.h | 8 +++++ arch/arm64/include/asm/mman.h | 36 ++++++++++++++++++++ arch/arm64/include/asm/mmu_context.h | 30 +++++++++++++++- arch/arm64/kernel/pi/idreg-override.c | 4 ++- arch/arm64/kernel/pi/map_kernel.c | 23 +++++++++++++ arch/arm64/mm/proc.S | 6 ++++ 7 files changed, 116 insertions(+), 2 deletions(-) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index eed8fef08a10..403fc2707700 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -1610,6 +1610,17 @@ config RODATA_FULL_DEFAULT_ENABLED This requires the linear region to be mapped down to pages, which may adversely affect performance in some cases. +config ARM64_WXN + bool "Enable WXN attribute so all writable mappings are non-exec" + help + Set the WXN bit in the SCTLR system register so that all writable + mappings are treated as if the PXN/UXN bit is set as well. + If this is set to Y, it can still be disabled at runtime by + passing 'arm64.nowxn' on the kernel command line. + + This should only be set if no software needs to be supported that + relies on being able to execute from writable mappings. + config ARM64_SW_TTBR0_PAN bool "Emulate Privileged Access Never using TTBR0_EL1 switching" help diff --git a/arch/arm64/include/asm/cpufeature.h b/arch/arm64/include/asm/cpufeature.h index a8f97690ce1f..ee33b7e52da7 100644 --- a/arch/arm64/include/asm/cpufeature.h +++ b/arch/arm64/include/asm/cpufeature.h @@ -18,6 +18,7 @@ #define ARM64_SW_FEATURE_OVERRIDE_NOKASLR 0 #define ARM64_SW_FEATURE_OVERRIDE_HVHE 4 #define ARM64_SW_FEATURE_OVERRIDE_RODATA_OFF 8 +#define ARM64_SW_FEATURE_OVERRIDE_NOWXN 12 #ifndef __ASSEMBLY__ @@ -962,6 +963,13 @@ static inline bool kaslr_disabled_cmdline(void) return arm64_test_sw_feature_override(ARM64_SW_FEATURE_OVERRIDE_NOKASLR); } +static inline bool arm64_wxn_enabled(void) +{ + if (!IS_ENABLED(CONFIG_ARM64_WXN)) + return false; + return !arm64_test_sw_feature_override(ARM64_SW_FEATURE_OVERRIDE_NOWXN); +} + u32 get_kvm_ipa_limit(void); void dump_cpu_features(void); diff --git a/arch/arm64/include/asm/mman.h b/arch/arm64/include/asm/mman.h index 5966ee4a6154..6d4940342ba7 100644 --- a/arch/arm64/include/asm/mman.h +++ b/arch/arm64/include/asm/mman.h @@ -35,11 +35,40 @@ static inline unsigned long arch_calc_vm_flag_bits(unsigned long flags) } #define arch_calc_vm_flag_bits(flags) arch_calc_vm_flag_bits(flags) +static inline bool arm64_check_wx_prot(unsigned long prot, + struct task_struct *tsk) +{ + /* + * When we are running with SCTLR_ELx.WXN==1, writable mappings are + * implicitly non-executable. This means we should reject such mappings + * when user space attempts to create them using mmap() or mprotect(). + */ + if (arm64_wxn_enabled() && + ((prot & (PROT_WRITE | PROT_EXEC)) == (PROT_WRITE | PROT_EXEC))) { + /* + * User space libraries such as libffi carry elaborate + * heuristics to decide whether it is worth it to even attempt + * to create writable executable mappings, as PaX or selinux + * enabled systems will outright reject it. They will usually + * fall back to something else (e.g., two separate shared + * mmap()s of a temporary file) on failure. + */ + pr_info_ratelimited( + "process %s (%d) attempted to create PROT_WRITE+PROT_EXEC mapping\n", + tsk->comm, tsk->pid); + return false; + } + return true; +} + static inline bool arch_validate_prot(unsigned long prot, unsigned long addr __always_unused) { unsigned long supported = PROT_READ | PROT_WRITE | PROT_EXEC | PROT_SEM; + if (!arm64_check_wx_prot(prot, current)) + return false; + if (system_supports_bti()) supported |= PROT_BTI; @@ -50,6 +79,13 @@ static inline bool arch_validate_prot(unsigned long prot, } #define arch_validate_prot(prot, addr) arch_validate_prot(prot, addr) +static inline bool arch_validate_mmap_prot(unsigned long prot, + unsigned long addr) +{ + return arm64_check_wx_prot(prot, current); +} +#define arch_validate_mmap_prot arch_validate_mmap_prot + static inline bool arch_validate_flags(unsigned long vm_flags) { if (!system_supports_mte()) diff --git a/arch/arm64/include/asm/mmu_context.h b/arch/arm64/include/asm/mmu_context.h index c768d16b81a4..f0fe2d09d139 100644 --- a/arch/arm64/include/asm/mmu_context.h +++ b/arch/arm64/include/asm/mmu_context.h @@ -20,13 +20,41 @@ #include #include #include -#include #include #include #include extern bool rodata_full; +static inline int arch_dup_mmap(struct mm_struct *oldmm, + struct mm_struct *mm) +{ + return 0; +} + +static inline void arch_exit_mmap(struct mm_struct *mm) +{ +} + +static inline void arch_unmap(struct mm_struct *mm, + unsigned long start, unsigned long end) +{ +} + +static inline bool arch_vma_access_permitted(struct vm_area_struct *vma, + bool write, bool execute, bool foreign) +{ + if (IS_ENABLED(CONFIG_ARM64_WXN) && execute && + (vma->vm_flags & (VM_WRITE | VM_EXEC)) == (VM_WRITE | VM_EXEC)) { + pr_warn_ratelimited( + "process %s (%d) attempted to execute from writable memory\n", + current->comm, current->pid); + /* disallow unless the nowxn override is set */ + return !arm64_wxn_enabled(); + } + return true; +} + static inline void contextidr_thread_switch(struct task_struct *next) { if (!IS_ENABLED(CONFIG_PID_IN_CONTEXTIDR)) diff --git a/arch/arm64/kernel/pi/idreg-override.c b/arch/arm64/kernel/pi/idreg-override.c index aad399796e81..bccfee34f62f 100644 --- a/arch/arm64/kernel/pi/idreg-override.c +++ b/arch/arm64/kernel/pi/idreg-override.c @@ -189,6 +189,7 @@ static const struct ftr_set_desc sw_features __prel64_initconst = { FIELD("nokaslr", ARM64_SW_FEATURE_OVERRIDE_NOKASLR, NULL), FIELD("hvhe", ARM64_SW_FEATURE_OVERRIDE_HVHE, hvhe_filter), FIELD("rodataoff", ARM64_SW_FEATURE_OVERRIDE_RODATA_OFF, NULL), + FIELD("nowxn", ARM64_SW_FEATURE_OVERRIDE_NOWXN, NULL), {} }, }; @@ -221,8 +222,9 @@ static const struct { { "arm64.nomops", "id_aa64isar2.mops=0" }, { "arm64.nomte", "id_aa64pfr1.mte=0" }, { "nokaslr", "arm64_sw.nokaslr=1" }, - { "rodata=off", "arm64_sw.rodataoff=1" }, + { "rodata=off", "arm64_sw.rodataoff=1 arm64_sw.nowxn=1" }, { "arm64.nolva", "id_aa64mmfr2.varange=0" }, + { "arm64.nowxn", "arm64_sw.nowxn=1" }, }; static int __init parse_hexdigit(const char *p, u64 *v) diff --git a/arch/arm64/kernel/pi/map_kernel.c b/arch/arm64/kernel/pi/map_kernel.c index 5fa08e13e17e..cac1e1f63c44 100644 --- a/arch/arm64/kernel/pi/map_kernel.c +++ b/arch/arm64/kernel/pi/map_kernel.c @@ -132,6 +132,25 @@ static void __init map_kernel(u64 kaslr_offset, u64 va_offset, int root_level) idmap_cpu_replace_ttbr1(swapper_pg_dir); } +static void noinline __section(".idmap.text") disable_wxn(void) +{ + u64 sctlr = read_sysreg(sctlr_el1) & ~SCTLR_ELx_WXN; + + /* + * We cannot safely clear the WXN bit while the MMU and caches are on, + * so turn the MMU off, flush the TLBs and turn it on again but with + * the WXN bit cleared this time. + */ + asm(" msr sctlr_el1, %0 ;" + " isb ;" + " tlbi vmalle1 ;" + " dsb nsh ;" + " isb ;" + " msr sctlr_el1, %1 ;" + " isb ;" + :: "r"(sctlr & ~SCTLR_ELx_M), "r"(sctlr)); +} + static void noinline __section(".idmap.text") set_ttbr0_for_lpa2(u64 ttbr) { u64 sctlr = read_sysreg(sctlr_el1); @@ -229,6 +248,10 @@ asmlinkage void __init early_map_kernel(u64 boot_status, void *fdt) if (va_bits > VA_BITS_MIN) sysreg_clear_set(tcr_el1, TCR_T1SZ_MASK, TCR_T1SZ(va_bits)); + if (IS_ENABLED(CONFIG_ARM64_WXN) && + arm64_test_sw_feature_override(ARM64_SW_FEATURE_OVERRIDE_NOWXN)) + disable_wxn(); + /* * The virtual KASLR displacement modulo 2MiB is decided by the * physical placement of the image, as otherwise, we might not be able diff --git a/arch/arm64/mm/proc.S b/arch/arm64/mm/proc.S index 9d40f3ffd8d2..bfd2ad896108 100644 --- a/arch/arm64/mm/proc.S +++ b/arch/arm64/mm/proc.S @@ -546,6 +546,12 @@ alternative_else_nop_endif * Prepare SCTLR */ mov_q x0, INIT_SCTLR_EL1_MMU_ON +#ifdef CONFIG_ARM64_WXN + ldr_l x1, arm64_sw_feature_override + FTR_OVR_VAL_OFFSET + tst x1, #0xf << ARM64_SW_FEATURE_OVERRIDE_NOWXN + orr x1, x0, #SCTLR_ELx_WXN + csel x0, x0, x1, ne +#endif ret // return to head.S .unreq mair From patchwork Tue Jan 23 14:53:49 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13527567 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id C1ED6C47DDF for ; Tue, 23 Jan 2024 14:58:45 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=bw5TRtLuT6DbHWroTajIAKCahrYx6Z2lhcZa5gp/42Y=; b=MXnTC9G60gsWgHCQZh+fyT1YYQ 04BZ0dud8mD/0sV2zZV2PstYt1XSSSArYMyMQDgGWiPEVMuvPqztJS0TsY6nPJDdpXxfu9fnRCWFc Yfhc0xEoBAMov1GQeNqhqljmjXpwQzdh4VFpMwPpLeuA3hpaK/Vp1zm3WbNoQxGzfzMLXdh07NqDO q9Tz2DppbPOc+y89I3GZlug1ourDkxhMthdahwiYTBojkUt4WNpLDJ9PCZ7NAgEPPH54W2U4Lmw/K zuTD3CNF80TBTxeGybM+jbIaXXshJ9q1eOdVmtyNHy1X80SZafvz45f07NVX23+007Fys/Koc2DMf gCHtu+UQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rSIEG-00GvlB-31; Tue, 23 Jan 2024 14:58:20 +0000 Received: from mail-wm1-x34a.google.com ([2a00:1450:4864:20::34a]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rSICK-00GuZ7-02 for linux-arm-kernel@lists.infradead.org; Tue, 23 Jan 2024 14:56:29 +0000 Received: by mail-wm1-x34a.google.com with SMTP id 5b1f17b1804b1-40eba1f0aa7so7406835e9.1 for ; Tue, 23 Jan 2024 06:56:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1706021778; x=1706626578; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=YAtMtSzzHj/aOMSOynry5N3eY6xWLMEJNhIxiRZ9cZY=; b=3WBYlA5raDan83m8CZB4oTiq/RB56kBNWfR6p9WzXM2ZMomUf56JzDqWCIY0hE4GoA +NbyLczZJ43lGrFpx6ygfxlG5xUGJQo5fSb8CB2u7oc+hhqqIr/QaJ9t6zPxTRkY3hCp tcsGtEom5Kl9iEHrXcSqV8W6Kb57deo1net9NkWLddEi/rr7SiSZ6UdG3pz2lgoeR0tc ZCvGB+xCMZf9HoSlQXmhQW6nypoOe+ORM6vR+PwIZoHLNhBqWDoEzRPRGZ2l6R/kU4jx aG9s7/tCAwOYQhkjSzcZCLCWSFzZ0vMwkjs7PrikuSq029WlPN1afA8f0+/lJevqHuFz Sg7A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706021778; x=1706626578; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=YAtMtSzzHj/aOMSOynry5N3eY6xWLMEJNhIxiRZ9cZY=; b=SMs30Joal5bs0V8RdT1LDS3dLDS3H7P4C73TIiH01rNzVrdpJ+Mo41JIG4bldU+9/C IqhCwwXO/Zs9QtL/HXaJ/u5OhpLZ7BR9VTpY0mek6AYXfmuujV/zBX5keBW+m5YUzDVp 6HfKealty5d5Ph/3+2xDiYWtsL0pPwFL3tN91JmH0P1KbhbEhmE8TyIloDBOx8PbrtLA ni5ZFl0WgeUWBRnTFr/jRVtPvIm0N+gHluO0tNkcWfMv/KOc3ihEHOApcff/7smFvSuZ FCxbnwOJ86ybLQ719pIzEUO6IBsCTDjCgZ5Nw3vzrZ1tWvxerbg21urSBqoTsimT/pxs lewQ== X-Gm-Message-State: AOJu0YwSJ7bj1Zs7fUYotcZoSouo6zhUg/uchP9VgkPaqDFEeva0mmyr LvUYPxUOGxz+f1+4E8NY6+7XlSaSDu4vPJR3weJJp5JvP7RqTH8KMcwAa4pDQQAPl0CHFJcQIjU QX6SxGdtGCBbsiaSdg7PNXUVR8SiR1D9zYX61fBK1wenbnxKea4EcSB2N0wh51f0nc4xkcH6vy4 Wa8Jw0N4Q4F5+3eeFfqfFA0Bqimuxc+5A7s2GDMMqe X-Google-Smtp-Source: AGHT+IGiRlShTgiobv2CyzoXtIgq0D8ahIT750HaszdPHUtSxMWL5vA6rES5zgduY39jqub1mG2ILawm X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:600c:4f03:b0:40e:9d0e:afb9 with SMTP id l3-20020a05600c4f0300b0040e9d0eafb9mr58564wmq.8.1706021778073; Tue, 23 Jan 2024 06:56:18 -0800 (PST) Date: Tue, 23 Jan 2024 15:53:49 +0100 In-Reply-To: <20240123145258.1462979-52-ardb+git@google.com> Mime-Version: 1.0 References: <20240123145258.1462979-52-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=1389; i=ardb@kernel.org; h=from:subject; bh=Nt8HrrLf++z1DB/ZA+SeVw3ieMM9IjuVCi2o2T++cu0=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIXX9ZUbfMwceXzDbxD/78qTkaV945SzOvllT9SxnpvSKi BtdTG1rO0pZGMQ4GGTFFFkEZv99t/P0RKla51myMHNYmUCGMHBxCsBE2iwY/ud/fcHnvqUyZs/Z m7Mc9dPrZp51zfJI/r+4+eIzN/UFIl8ZGfavWK7HuW7+pEC91ZwR3x++0Vnv55nhOn/+dKO/Z8K fejIBAA== X-Mailer: git-send-email 2.43.0.429.g432eaa2c6b-goog Message-ID: <20240123145258.1462979-102-ardb+git@google.com> Subject: [PATCH v7 50/50] arm64: Set the default CONFIG_ARM64_VA_BITS_52 in Kconfig rather than defconfig From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240123_065620_072453_737BB905 X-CRM114-Status: GOOD ( 11.27 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Catalin Marinas Currently the "Virtual address space size" arch/arm64/Kconfig menu entry sets different defaults for each page size. However, all are overridden by the defconfig to 52-bit. Set the actual default in Kconfig and remove the defconfig line. Signed-off-by: Catalin Marinas --- arch/arm64/Kconfig | 4 +--- arch/arm64/configs/defconfig | 1 - 2 files changed, 1 insertion(+), 4 deletions(-) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 403fc2707700..7761ffc6dbcf 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -1283,9 +1283,7 @@ endchoice choice prompt "Virtual address space size" - default ARM64_VA_BITS_39 if ARM64_4K_PAGES - default ARM64_VA_BITS_47 if ARM64_16K_PAGES - default ARM64_VA_BITS_42 if ARM64_64K_PAGES + default ARM64_VA_BITS_52 help Allows choosing one of multiple possible virtual address space sizes. The level of translation table is determined by diff --git a/arch/arm64/configs/defconfig b/arch/arm64/configs/defconfig index c5919a762109..f086b0624ec8 100644 --- a/arch/arm64/configs/defconfig +++ b/arch/arm64/configs/defconfig @@ -76,7 +76,6 @@ CONFIG_ARCH_VEXPRESS=y CONFIG_ARCH_VISCONTI=y CONFIG_ARCH_XGENE=y CONFIG_ARCH_ZYNQMP=y -CONFIG_ARM64_VA_BITS_52=y CONFIG_SCHED_MC=y CONFIG_SCHED_SMT=y CONFIG_NUMA=y