From patchwork Wed Feb 14 12:28:47 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13556404 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 6BCC6C48BC1 for ; Wed, 14 Feb 2024 12:30:02 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=lqn9+cUFvYRTJyQwxG/rDP055ei9W69+nuzucXwDrJo=; b=WQ64ThP9VstlQhYb2XaRsDm+6G m+OtnLB/u5Z0+kg2r3e25+QHwYiykFfK6WE5fz0iO1pVwbRMMvhh5lf+LI1Y0BZbs8EEAp3w4Ww2q 0yzBoAc8F4A/qRAi50lfrZyjkuDwn/bAard0SU9+Nv1l1T3eMI8uGdkrrNK2r+YScQL2z4Pi8idUs pKzHAEuqEj7JQ6x2Rhft0/Mz5BKyEuha4x16aMfWu80MLxNjupcwCHqa0/V4l+2x4F5zemqzrI8MX G1syCDeKuiRypfpiPw3tqXIx1rZ8kcE6g6mZma/+bJKbKXz+pobCGlShaqJU5ybGcvad8UlS8/3of vfKkNbJQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1raEOe-0000000CoGL-1AAG; Wed, 14 Feb 2024 12:29:52 +0000 Received: from mail-wm1-x34a.google.com ([2a00:1450:4864:20::34a]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1raEOY-0000000CoDc-0IUT for linux-arm-kernel@lists.infradead.org; Wed, 14 Feb 2024 12:29:47 +0000 Received: by mail-wm1-x34a.google.com with SMTP id 5b1f17b1804b1-410dfd0e1caso17539705e9.0 for ; Wed, 14 Feb 2024 04:29:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1707913781; x=1708518581; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=E7ySR0vcU7U/Aj19kq5Li5vJqxJ6gFKIuAubeInPysA=; b=m8yR01nHEZOEe3w2n0e/s0jNeJUgLOXIM/F/8QzGzXmyE8gb6CdOxpz57+YaM8+ff9 CeChn3qHLGuH32It9uURx+gSkjpLyZ1GFGeC3iQXfdGm5F0B4dVKOsiiS3FQjwSeoHNj UazxgdCmYqt4SaLctU/vGfWZ7jLqrj4z0d5Jok4LMc3o5/UzmQz/ywxPaMl5IxEXP7Yv wDEhlRW9Vlaoh/4e0b26Zheb5jlHvoTU5wfEj69Qcz3vnWcrNpiNucY7siLRwMdobC+m JQ08tMRqX861DpybpK/kzmL7eebNrOZJYpVd0Oqu4kxrEItOBMeoJbBnnDo0qxe9sRDV dXzg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707913781; x=1708518581; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=E7ySR0vcU7U/Aj19kq5Li5vJqxJ6gFKIuAubeInPysA=; b=oesE2Wk3c5k6V3l3bxhbaQu6U7z3jPEf/2gQw4ASlKaxLoFFT3AQKNfaAIFY5ih6Nb uhaRZlO/oDijbzpi3z7ZJYQo/YtFD1Dv3bZNVPIZcD8TBYeoLhJi7SIeVTXZc23VJnca CWrA790eG027bNuzoJ5X1AYdndDkdy4eQfISD9vi9lQY6I6tRj0O2sJfODVoBMRAqegv Pp3XzvwDP3yk29uCjBHgJr08K4hyMsNtn+kp/5FxIIH2WFfJZlQ7n4aiG0TeNypBqvog 4FRY40UpZR6HjEHsrLkAaeUxN8nHSiU/9sc8yTrB3S9PNarzkj5aVRWAZ3sYimWyaYFP qDjw== X-Gm-Message-State: AOJu0YyFJu6nVZmZiOYVGLVTbsvs25927cQQis7/djPQ2QrS1+S219zl 3k6PqaRfCKye8DHvHfZxWh1176uT7t+CSaC2MqxV/FMQMAPA6+trQ7jep7SD7lbIWhyUzV8tdSI RZABI+MYQtOkl/qkkQgw/6/53NZA0BfqtxjWG1UbsKkr2tuNHMcprRuF5J6u6MLUyJ6S+S/4vOj pkSKUEFukHfcmHWXQ7qVjCnunZ7nTp2vB6xKccBkt/ X-Google-Smtp-Source: AGHT+IFoQaUzd1JgkYf56DM1ovETFXcNoqK4k6Atizk1cbAgOh5ZrtumDFegxS65tsOSmS9SAKNPctZS X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:600c:54e8:b0:40f:dfef:cd4 with SMTP id jb8-20020a05600c54e800b0040fdfef0cd4mr42680wmb.6.1707913781213; Wed, 14 Feb 2024 04:29:41 -0800 (PST) Date: Wed, 14 Feb 2024 13:28:47 +0100 In-Reply-To: <20240214122845.2033971-45-ardb+git@google.com> Mime-Version: 1.0 References: <20240214122845.2033971-45-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=6618; i=ardb@kernel.org; h=from:subject; bh=qltVUJ0baJKnUmgvQnZvF+l5dl5tPI+hJ6wQoQcDNqs=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIfXMxv9l23yaP7Y45KbOEF1lcaaiKGKz12SROXOKz63Q5 8w6ULGqo5SFQYyDQVZMkUVg9t93O09PlKp1niULM4eVCWQIAxenAEwkrImR4XiSx64jRYEOpgwL 4r9MYy/4YHZUXNyCVdl+ou26e9tzxRkZTjGeWim9OKH41neNZq0m6xtHb5nNuX1HdNnDkFWfebs fMAIA X-Mailer: git-send-email 2.43.0.687.g38aa6559b0-goog Message-ID: <20240214122845.2033971-46-ardb+git@google.com> Subject: [PATCH v8 01/43] arm64: kernel: Manage absolute relocations in code built under pi/ From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240214_042946_145686_7C34CC23 X-CRM114-Status: GOOD ( 24.13 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel The mini C runtime runs before relocations are processed, and so it cannot rely on statically initialized pointer variables. Add a check to ensure that such code does not get introduced by accident, by going over the relocations in each object, identifying the ones that operate on data sections that are part of the executable image, and raising an error if any relocations of type R_AARCH64_ABS64 exist. Note that such relocations are permitted in other places (e.g., debug sections) and will never occur in compiler generated code sections when using the small code model, so only check sections that have SHF_ALLOC set and SHF_EXECINSTR cleared. To accommodate cases where statically initialized symbol references are unavoidable, introduce a special case for ELF input data sections that have ".rodata.prel64" in their names, and in these cases, instead of rejecting any encountered ABS64 relocations, convert them into PREL64 relocations, which don't require any runtime fixups. Note that the code in question must still be modified to deal with this, as it needs to convert the 64-bit signed offsets into absolute addresses before use. Signed-off-by: Ard Biesheuvel --- arch/arm64/kernel/pi/Makefile | 9 +- arch/arm64/kernel/pi/pi.h | 18 +++ arch/arm64/kernel/pi/relacheck.c | 130 ++++++++++++++++++++ 3 files changed, 155 insertions(+), 2 deletions(-) diff --git a/arch/arm64/kernel/pi/Makefile b/arch/arm64/kernel/pi/Makefile index c844a0546d7f..bc32a431fe35 100644 --- a/arch/arm64/kernel/pi/Makefile +++ b/arch/arm64/kernel/pi/Makefile @@ -22,11 +22,16 @@ KCSAN_SANITIZE := n UBSAN_SANITIZE := n KCOV_INSTRUMENT := n +hostprogs := relacheck + +quiet_cmd_piobjcopy = $(quiet_cmd_objcopy) + cmd_piobjcopy = $(cmd_objcopy) && $(obj)/relacheck $(@) $(<) + $(obj)/%.pi.o: OBJCOPYFLAGS := --prefix-symbols=__pi_ \ --remove-section=.note.gnu.property \ --prefix-alloc-sections=.init -$(obj)/%.pi.o: $(obj)/%.o FORCE - $(call if_changed,objcopy) +$(obj)/%.pi.o: $(obj)/%.o $(obj)/relacheck FORCE + $(call if_changed,piobjcopy) $(obj)/lib-%.o: $(srctree)/lib/%.c FORCE $(call if_changed_rule,cc_o_c) diff --git a/arch/arm64/kernel/pi/pi.h b/arch/arm64/kernel/pi/pi.h new file mode 100644 index 000000000000..7c2d9bbf0ff9 --- /dev/null +++ b/arch/arm64/kernel/pi/pi.h @@ -0,0 +1,18 @@ +// SPDX-License-Identifier: GPL-2.0-only +// Copyright 2023 Google LLC +// Author: Ard Biesheuvel + +#define __prel64_initconst __section(".init.rodata.prel64") + +#define PREL64(type, name) union { type *name; prel64_t name ## _prel; } + +#define prel64_pointer(__d) (typeof(__d))prel64_to_pointer(&__d##_prel) + +typedef volatile signed long prel64_t; + +static inline void *prel64_to_pointer(const prel64_t *offset) +{ + if (!*offset) + return NULL; + return (void *)offset + *offset; +} diff --git a/arch/arm64/kernel/pi/relacheck.c b/arch/arm64/kernel/pi/relacheck.c new file mode 100644 index 000000000000..b0cd4d0d275b --- /dev/null +++ b/arch/arm64/kernel/pi/relacheck.c @@ -0,0 +1,130 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright (C) 2023 - Google LLC + * Author: Ard Biesheuvel + */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#if __BYTE_ORDER__ == __ORDER_LITTLE_ENDIAN__ +#define HOST_ORDER ELFDATA2LSB +#elif __BYTE_ORDER__ == __ORDER_BIG_ENDIAN__ +#define HOST_ORDER ELFDATA2MSB +#endif + +static Elf64_Ehdr *ehdr; +static Elf64_Shdr *shdr; +static const char *strtab; +static bool swap; + +static uint64_t swab_elfxword(uint64_t val) +{ + return swap ? __builtin_bswap64(val) : val; +} + +static uint32_t swab_elfword(uint32_t val) +{ + return swap ? __builtin_bswap32(val) : val; +} + +static uint16_t swab_elfhword(uint16_t val) +{ + return swap ? __builtin_bswap16(val) : val; +} + +int main(int argc, char *argv[]) +{ + struct stat stat; + int fd, ret; + + if (argc < 3) { + fprintf(stderr, "file arguments missing\n"); + exit(EXIT_FAILURE); + } + + fd = open(argv[1], O_RDWR); + if (fd < 0) { + fprintf(stderr, "failed to open %s\n", argv[1]); + exit(EXIT_FAILURE); + } + + ret = fstat(fd, &stat); + if (ret < 0) { + fprintf(stderr, "failed to stat() %s\n", argv[1]); + exit(EXIT_FAILURE); + } + + ehdr = mmap(0, stat.st_size, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0); + if (ehdr == MAP_FAILED) { + fprintf(stderr, "failed to mmap() %s\n", argv[1]); + exit(EXIT_FAILURE); + } + + swap = ehdr->e_ident[EI_DATA] != HOST_ORDER; + shdr = (void *)ehdr + swab_elfxword(ehdr->e_shoff); + strtab = (void *)ehdr + + swab_elfxword(shdr[swab_elfhword(ehdr->e_shstrndx)].sh_offset); + + for (int i = 0; i < swab_elfhword(ehdr->e_shnum); i++) { + unsigned long info, flags; + bool prel64 = false; + Elf64_Rela *rela; + int numrela; + + if (swab_elfword(shdr[i].sh_type) != SHT_RELA) + continue; + + /* only consider RELA sections operating on data */ + info = swab_elfword(shdr[i].sh_info); + flags = swab_elfxword(shdr[info].sh_flags); + if ((flags & (SHF_ALLOC | SHF_EXECINSTR)) != SHF_ALLOC) + continue; + + /* + * We generally don't permit ABS64 relocations in the code that + * runs before relocation processing occurs. If statically + * initialized absolute symbol references are unavoidable, they + * may be emitted into a *.rodata.prel64 section and they will + * be converted to place-relative 64-bit references. This + * requires special handling in the referring code. + */ + if (strstr(strtab + swab_elfword(shdr[info].sh_name), + ".rodata.prel64")) { + prel64 = true; + } + + rela = (void *)ehdr + swab_elfxword(shdr[i].sh_offset); + numrela = swab_elfxword(shdr[i].sh_size) / sizeof(*rela); + + for (int j = 0; j < numrela; j++) { + uint64_t info = swab_elfxword(rela[j].r_info); + + if (ELF64_R_TYPE(info) != R_AARCH64_ABS64) + continue; + + if (prel64) { + /* convert ABS64 into PREL64 */ + info ^= R_AARCH64_ABS64 ^ R_AARCH64_PREL64; + rela[j].r_info = swab_elfxword(info); + } else { + fprintf(stderr, + "Unexpected absolute relocations detected in %s\n", + argv[2]); + close(fd); + unlink(argv[1]); + exit(EXIT_FAILURE); + } + } + } + close(fd); + return 0; +} From patchwork Wed Feb 14 12:28:48 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13556405 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id DA456C48BC1 for ; Wed, 14 Feb 2024 12:30:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=g33TRjpATMrD8lKMbrafI2FBwbCvsXE0gxQACk0lmGE=; b=rgzJfiK4Tppmx+/xBt6YJfXv6X 80PrenmR1za8R5T0eDw2mS6GpTokbBLmvdOARla9ubocwhPgyUZ7+1/QD7si6uY2BxHBrs/0rnII+ y0XiQoGBGz4diJprGKWF5ZXS6kl04Dl2yiRtE/aZ6jhS+i0ni0ujV90NNE9OfCxd3zjPK0c/WxRYh 2FdYYLti3/3IHVMLFt+pe2FYRkLJWcG/Y/cSCRkCzr+zDNiT7IFROBg2IZ1RZULkV0ngvspxsYQ9a /db2J3PQcOAUcmT/aRl7Xjp+koEF21zaSDpcJa05l9J5t31/0tiaVq3Wsu6C2ilpQjh/+1r4+mpIN 4AWn7K0w==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1raEOg-0000000CoHc-183R; Wed, 14 Feb 2024 12:29:54 +0000 Received: from mail-yw1-x114a.google.com ([2607:f8b0:4864:20::114a]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1raEOb-0000000CoDh-0s6Q for linux-arm-kernel@lists.infradead.org; Wed, 14 Feb 2024 12:29:50 +0000 Received: by mail-yw1-x114a.google.com with SMTP id 00721157ae682-60753c3fab9so9177687b3.0 for ; Wed, 14 Feb 2024 04:29:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1707913783; x=1708518583; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=9CXAuLDgkE8EWQF/CV7t6ppGf8UwbT1/5PU84QU+uYE=; b=0csPODYbDWsflOMnhYSa9+nQikPHo7Zsts2FbbAWy+tJcwULPw8XxqGDgOCl1h1Lnh iwGu6OGcIjMD3cBIGWq+0cw/ZK73woUWTNx0kbAF4uMToXvcshVBSuYkc13JEkbHWbS4 +ULDIEwm4dtG8Lxba3Gpts/s0UIhDH41Iqf59NvFUHg9SDgwXXo8t88PxrcDDV5275mB H93zxIAxrQIEJEblyQ9tJyCNrBAEZx1MEAyPo/al2W12f9evy5C/tSpi2wh+4XE/Iqya akSAf2XdLfDbAotw9YP0IGSvbljTiNw5g+LcJKNJi77cH3yJGqzF7mMbeF8mt/umtId+ 7TkA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707913783; x=1708518583; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=9CXAuLDgkE8EWQF/CV7t6ppGf8UwbT1/5PU84QU+uYE=; b=j4C1Yi3n9IV4U5ylWXUnR8Cjq3w36i6Ru8uRRniZTImVd8A2jYHLqighta5mSuSDuO U0xh8Qw9mz/u4OAuk9aOrDJF7AsrDpPhLvO10M3DuJrKSOV7KUy7kylr6LPqk1h0++6V tOnepIfH+iLvrkgenvg+JY/aKTrKq5fNH+E2Ychi1plYCPukw5vTEETIvON+j70yI6C1 7uEybZaUa+kUbj/MeODX2PNDhgqOi7q2kLtYDvNUID3g3bTYfDY4G5tVvxtcFJHq3Chz NMoSAds4feIU678a6+oLeb43MNRXmIaLXCbVTonnkI06LWYsexQ5Kpy3TIDzVFsLEH2M lxKQ== X-Gm-Message-State: AOJu0Yw5vewhJyYuT0MmXe6wEHmrdswCKL4p6MJ3RgHFzwiW5lyo46oF HwsE7qIz6MN87/3ZTVtVw6d4+dJrnYKD2JSbCg4pqF+B88OyxEPKl/LcmTgpZ2JJI5xjOcSN3bM YozNrHmysQWb1XIB18iiHPiN1CzHs8NaUPTG91HNFl31CONyfPTkatRGpwsHakdCjinQLEHurOq FYnOQnYOEHJfgiHLgNGhz35bbAnUrvSFvWnUIDmutt X-Google-Smtp-Source: AGHT+IGSL7fmZPv+yVFJ/uwrxJPct0aIUZHbdTtaVWJ15ms+FsnCFKtHJ6Amo/l9SU4rjrbmFGIGJTvl X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a81:6cd8:0:b0:604:982c:3c26 with SMTP id h207-20020a816cd8000000b00604982c3c26mr337239ywc.3.1707913783476; Wed, 14 Feb 2024 04:29:43 -0800 (PST) Date: Wed, 14 Feb 2024 13:28:48 +0100 In-Reply-To: <20240214122845.2033971-45-ardb+git@google.com> Mime-Version: 1.0 References: <20240214122845.2033971-45-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=3211; i=ardb@kernel.org; h=from:subject; bh=GT9VZCUTsUkmFmQ4vnPkMl0fh4kY3gseXy8jPjS7OR4=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIfXMJoakydIMDCVy1+Rycy2ubxO7/nLriRTdd6w5T14Hr 8xYa9jWUcrCIMbBICumyCIw+++7nacnStU6z5KFmcPKBDKEgYtTACai+pGR4ePN2fPbXa535q76 HMnZc9lj8YPcDS/3mEWsMnSMWPYuYxkjQ1/6uQeScwq+dT/9dGSj3Nt9Torvvb4+KVK+/Ol19LW TV/gB X-Mailer: git-send-email 2.43.0.687.g38aa6559b0-goog Message-ID: <20240214122845.2033971-47-ardb+git@google.com> Subject: [PATCH v8 02/43] arm64: kernel: Don't rely on objcopy to make code under pi/ __init From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240214_042949_334300_C33621A8 X-CRM114-Status: GOOD ( 12.93 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel We will add some code under pi/ that contains global variables that should not end up in __initdata, as they will not be writable via the initial ID map. So only rely on objcopy for making the libfdt code __init, and use explicit annotations for the rest. Signed-off-by: Ard Biesheuvel --- arch/arm64/kernel/pi/Makefile | 6 ++++-- arch/arm64/kernel/pi/kaslr_early.c | 16 +++++++++------- 2 files changed, 13 insertions(+), 9 deletions(-) diff --git a/arch/arm64/kernel/pi/Makefile b/arch/arm64/kernel/pi/Makefile index bc32a431fe35..2bbe866417d4 100644 --- a/arch/arm64/kernel/pi/Makefile +++ b/arch/arm64/kernel/pi/Makefile @@ -28,11 +28,13 @@ quiet_cmd_piobjcopy = $(quiet_cmd_objcopy) cmd_piobjcopy = $(cmd_objcopy) && $(obj)/relacheck $(@) $(<) $(obj)/%.pi.o: OBJCOPYFLAGS := --prefix-symbols=__pi_ \ - --remove-section=.note.gnu.property \ - --prefix-alloc-sections=.init + --remove-section=.note.gnu.property $(obj)/%.pi.o: $(obj)/%.o $(obj)/relacheck FORCE $(call if_changed,piobjcopy) +# ensure that all the lib- code ends up as __init code and data +$(obj)/lib-%.pi.o: OBJCOPYFLAGS += --prefix-alloc-sections=.init + $(obj)/lib-%.o: $(srctree)/lib/%.c FORCE $(call if_changed_rule,cc_o_c) diff --git a/arch/arm64/kernel/pi/kaslr_early.c b/arch/arm64/kernel/pi/kaslr_early.c index b9e0bb4bc6a9..167081b30a15 100644 --- a/arch/arm64/kernel/pi/kaslr_early.c +++ b/arch/arm64/kernel/pi/kaslr_early.c @@ -17,7 +17,7 @@ #include /* taken from lib/string.c */ -static char *__strstr(const char *s1, const char *s2) +static char *__init __strstr(const char *s1, const char *s2) { size_t l1, l2; @@ -33,7 +33,7 @@ static char *__strstr(const char *s1, const char *s2) } return NULL; } -static bool cmdline_contains_nokaslr(const u8 *cmdline) +static bool __init cmdline_contains_nokaslr(const u8 *cmdline) { const u8 *str; @@ -41,7 +41,7 @@ static bool cmdline_contains_nokaslr(const u8 *cmdline) return str == cmdline || (str > cmdline && *(str - 1) == ' '); } -static bool is_kaslr_disabled_cmdline(void *fdt) +static bool __init is_kaslr_disabled_cmdline(void *fdt) { if (!IS_ENABLED(CONFIG_CMDLINE_FORCE)) { int node; @@ -67,17 +67,19 @@ static bool is_kaslr_disabled_cmdline(void *fdt) return cmdline_contains_nokaslr(CONFIG_CMDLINE); } -static u64 get_kaslr_seed(void *fdt) +static u64 __init get_kaslr_seed(void *fdt) { + static char const chosen_str[] __initconst = "chosen"; + static char const seed_str[] __initconst = "kaslr-seed"; int node, len; fdt64_t *prop; u64 ret; - node = fdt_path_offset(fdt, "/chosen"); + node = fdt_path_offset(fdt, chosen_str); if (node < 0) return 0; - prop = fdt_getprop_w(fdt, node, "kaslr-seed", &len); + prop = fdt_getprop_w(fdt, node, seed_str, &len); if (!prop || len != sizeof(u64)) return 0; @@ -86,7 +88,7 @@ static u64 get_kaslr_seed(void *fdt) return ret; } -asmlinkage u64 kaslr_early_init(void *fdt) +asmlinkage u64 __init kaslr_early_init(void *fdt) { u64 seed, range; From patchwork Wed Feb 14 12:28:49 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13556406 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 5FBD4C48BC3 for ; Wed, 14 Feb 2024 12:30:10 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=uu1Hef/uXpQVCWCXjD/Ad1HqIvfnKf0z7wW3fLbx+Sk=; b=RwuXB4XM63CfEqcRdMhAVmrFcD xGlwSlKbIUd8nkMPfhZlkHRb929C7ormWf/fPXNkXrOfwk+k8luyWadXmLs1YLhB/tWR6qZt0fEW4 9OFZ4TyhySFBCZHMWniDMIia7JT18GIXp+tlex3vzjmq9ostwDZRQ8CaHFo1BheC+1oGdG9++X1Up /Gg0hZRS8xG3cpcybXWI4OqdgnhBD8dqBS5PPz/v96ojaqwcSDsJRmhGJhWfUe5cAufQ82FwK5gJ/ KLMHu6OxlUOFIN7O8sy2cTVOAUB1BTO/5Hfxc+ArsSa9n+MffW+fI83H78s7pxfpZZXS6pAgtCAdk lDPGDyRA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1raEOf-0000000CoGy-11IE; Wed, 14 Feb 2024 12:29:53 +0000 Received: from mail-wm1-x34a.google.com ([2a00:1450:4864:20::34a]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1raEOZ-0000000CoEM-3D3j for linux-arm-kernel@lists.infradead.org; Wed, 14 Feb 2024 12:29:49 +0000 Received: by mail-wm1-x34a.google.com with SMTP id 5b1f17b1804b1-40e40126031so32585555e9.0 for ; Wed, 14 Feb 2024 04:29:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1707913785; x=1708518585; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=Q9KkKZgjv0mxU25AjfuEP62RMv0zDLE/hSD1Mx39WOY=; b=ZW6PzJPdjRAUnG/+yq9twA+kEUxuw/ec+kJDJh27ewAilwHdeIChNWLIXUPEhVob0J Y60OOyekNdJ78b4jDyXlzQOzx5GlovfFZhkzy/I8gJkP1aSLAeExEhSPadOmVZNm50XE ffOQ00vtI8bb/mZpHrrMz+3F1TRPhhsOGV8jidMWvwtkErABDJkMAlLry1xfPtb040Sj YJZLJ4+I22vGHx+9uxMShgitzOoVyuMMvamoRN/4lNkgsh9dsfmrSJb96y+qaCutu2X/ kdKciOPwS/4QHg8DBEVehk9e6tIO5bPUej74HqDAtNNgLeJw6xUHlOmXnjyWLEjF+a/M BCzw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707913785; x=1708518585; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Q9KkKZgjv0mxU25AjfuEP62RMv0zDLE/hSD1Mx39WOY=; b=KLOUwe/Ams94gz+lDw0iyjWEqPgUbqeZHl/RaD4+hC4Ku/5tHqnpHbEUeRuYb+fWzZ DQLVtlJEns7PcLObcDxBbZRt23KwqBNVsqCXZrpkaXIZWWew595mGYnpbgex2VkEL/m/ tLpnsJZmFjVBl7CeVF0ac6GCWi6+97iSE4b/ojEVOfaCT0k7KSvyvdACKoEiUVl0jLEq NoyNImZucNPsuV8W30+HwH52qLvtYTuEkID7lxEEs3+daUujuxe5IYDYcTJu2PFTPomA utTCGVkEVXz9nS3wCIGLf/NkFBDAqakOeNxpT8jW6N8Z81qlcLLjSSmjGVJi6DsconF2 4dKQ== X-Gm-Message-State: AOJu0YwX0Iyc0xffqsDVlXU29obO96fPIF7oWIZ8GmnDBrdVRMysm0vI CojHr6H24khhkYY5USD/yjTzoRT8ZuldSeRxchxRjmw3sJcVqnidmCBTPAWS+1zmK8RJIMsgR5p tvrZt2S5V5JrjKFfgrI28+kF3d/wnfSMYRKXsKKygrbPk56qODnni1lvINLNcACKVn13Eki55Xb A5uf9QK02fBzu8mAzdVjfahYOfCjd0ODUgUsW7G8Mt X-Google-Smtp-Source: AGHT+IEl62YmWde8ZhtfWck6txFR22dZtnYixSPll8w/ZEwj0RIQjimwEZITV3sI/jVtvcb0ITr7oj1L X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:600c:c0c:b0:410:bf0c:8a93 with SMTP id fm12-20020a05600c0c0c00b00410bf0c8a93mr8761wmb.8.1707913785672; Wed, 14 Feb 2024 04:29:45 -0800 (PST) Date: Wed, 14 Feb 2024 13:28:49 +0100 In-Reply-To: <20240214122845.2033971-45-ardb+git@google.com> Mime-Version: 1.0 References: <20240214122845.2033971-45-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=10134; i=ardb@kernel.org; h=from:subject; bh=Qmq6PpFmNOGieWwbrDGB1mrZhgvbK9glpmhQpaaWj7Q=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIfXMJsbJnCv/bFjyLC1ut3W/aC3nrgy3xv/SZ+PsBKPKh PY+9vnQUcrCIMbBICumyCIw+++7nacnStU6z5KFmcPKBDKEgYtTACaycwEjw/1Ph16Ei+x7cC5Q qKJ1pVHB9OCCIFW3BZ/YRPj/Wx48vYeR4dTEWetficys+3Qj7M8qrzzJo+0nT3ffEH8qufBBwLq T6zkB X-Mailer: git-send-email 2.43.0.687.g38aa6559b0-goog Message-ID: <20240214122845.2033971-48-ardb+git@google.com> Subject: [PATCH v8 03/43] arm64: head: move relocation handling to C code From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240214_042947_888469_F2218157 X-CRM114-Status: GOOD ( 27.59 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel Now that we have a mini C runtime before the kernel mapping is up, we can move the non-trivial relocation processing code out of head.S and reimplement it in C. Signed-off-by: Ard Biesheuvel --- arch/arm64/kernel/Makefile | 3 +- arch/arm64/kernel/head.S | 104 ++------------------ arch/arm64/kernel/pi/Makefile | 5 +- arch/arm64/kernel/pi/relocate.c | 62 ++++++++++++ arch/arm64/kernel/vmlinux.lds.S | 12 ++- 5 files changed, 82 insertions(+), 104 deletions(-) diff --git a/arch/arm64/kernel/Makefile b/arch/arm64/kernel/Makefile index 467cb7117273..78f14084f6d7 100644 --- a/arch/arm64/kernel/Makefile +++ b/arch/arm64/kernel/Makefile @@ -57,7 +57,8 @@ obj-$(CONFIG_ACPI) += acpi.o obj-$(CONFIG_ACPI_NUMA) += acpi_numa.o obj-$(CONFIG_ARM64_ACPI_PARKING_PROTOCOL) += acpi_parking_protocol.o obj-$(CONFIG_PARAVIRT) += paravirt.o -obj-$(CONFIG_RANDOMIZE_BASE) += kaslr.o pi/ +obj-$(CONFIG_RELOCATABLE) += pi/ +obj-$(CONFIG_RANDOMIZE_BASE) += kaslr.o obj-$(CONFIG_HIBERNATION) += hibernate.o hibernate-asm.o obj-$(CONFIG_ELF_CORE) += elfcore.o obj-$(CONFIG_KEXEC_CORE) += machine_kexec.o relocate_kernel.o \ diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S index cab7f91949d8..a8fa64fc30d7 100644 --- a/arch/arm64/kernel/head.S +++ b/arch/arm64/kernel/head.S @@ -81,7 +81,7 @@ * x20 primary_entry() .. __primary_switch() CPU boot mode * x21 primary_entry() .. start_kernel() FDT pointer passed at boot in x0 * x22 create_idmap() .. start_kernel() ID map VA of the DT blob - * x23 primary_entry() .. start_kernel() physical misalignment/KASLR offset + * x23 __primary_switch() physical misalignment/KASLR offset * x24 __primary_switch() linear map KASLR seed * x25 primary_entry() .. start_kernel() supported VA size * x28 create_idmap() callee preserved temp register @@ -389,7 +389,7 @@ SYM_FUNC_START_LOCAL(create_idmap) /* Remap the kernel page tables r/w in the ID map */ adrp x1, _text adrp x2, init_pg_dir - adrp x3, init_pg_end + adrp x3, _end bic x4, x2, #SWAPPER_BLOCK_SIZE - 1 mov_q x5, SWAPPER_RW_MMUFLAGS mov x6, #SWAPPER_BLOCK_SHIFT @@ -779,97 +779,6 @@ SYM_FUNC_START_LOCAL(__no_granule_support) b 1b SYM_FUNC_END(__no_granule_support) -#ifdef CONFIG_RELOCATABLE -SYM_FUNC_START_LOCAL(__relocate_kernel) - /* - * Iterate over each entry in the relocation table, and apply the - * relocations in place. - */ - adr_l x9, __rela_start - adr_l x10, __rela_end - mov_q x11, KIMAGE_VADDR // default virtual offset - add x11, x11, x23 // actual virtual offset - -0: cmp x9, x10 - b.hs 1f - ldp x12, x13, [x9], #24 - ldr x14, [x9, #-8] - cmp w13, #R_AARCH64_RELATIVE - b.ne 0b - add x14, x14, x23 // relocate - str x14, [x12, x23] - b 0b - -1: -#ifdef CONFIG_RELR - /* - * Apply RELR relocations. - * - * RELR is a compressed format for storing relative relocations. The - * encoded sequence of entries looks like: - * [ AAAAAAAA BBBBBBB1 BBBBBBB1 ... AAAAAAAA BBBBBB1 ... ] - * - * i.e. start with an address, followed by any number of bitmaps. The - * address entry encodes 1 relocation. The subsequent bitmap entries - * encode up to 63 relocations each, at subsequent offsets following - * the last address entry. - * - * The bitmap entries must have 1 in the least significant bit. The - * assumption here is that an address cannot have 1 in lsb. Odd - * addresses are not supported. Any odd addresses are stored in the RELA - * section, which is handled above. - * - * Excluding the least significant bit in the bitmap, each non-zero - * bit in the bitmap represents a relocation to be applied to - * a corresponding machine word that follows the base address - * word. The second least significant bit represents the machine - * word immediately following the initial address, and each bit - * that follows represents the next word, in linear order. As such, - * a single bitmap can encode up to 63 relocations in a 64-bit object. - * - * In this implementation we store the address of the next RELR table - * entry in x9, the address being relocated by the current address or - * bitmap entry in x13 and the address being relocated by the current - * bit in x14. - */ - adr_l x9, __relr_start - adr_l x10, __relr_end - -2: cmp x9, x10 - b.hs 7f - ldr x11, [x9], #8 - tbnz x11, #0, 3f // branch to handle bitmaps - add x13, x11, x23 - ldr x12, [x13] // relocate address entry - add x12, x12, x23 - str x12, [x13], #8 // adjust to start of bitmap - b 2b - -3: mov x14, x13 -4: lsr x11, x11, #1 - cbz x11, 6f - tbz x11, #0, 5f // skip bit if not set - ldr x12, [x14] // relocate bit - add x12, x12, x23 - str x12, [x14] - -5: add x14, x14, #8 // move to next bit's address - b 4b - -6: /* - * Move to the next bitmap's address. 8 is the word size, and 63 is the - * number of significant bits in a bitmap entry. - */ - add x13, x13, #(8 * 63) - b 2b - -7: -#endif - ret - -SYM_FUNC_END(__relocate_kernel) -#endif - SYM_FUNC_START_LOCAL(__primary_switch) adrp x1, reserved_pg_dir adrp x2, init_idmap_pg_dir @@ -877,11 +786,11 @@ SYM_FUNC_START_LOCAL(__primary_switch) #ifdef CONFIG_RELOCATABLE adrp x23, KERNEL_START and x23, x23, MIN_KIMG_ALIGN - 1 -#ifdef CONFIG_RANDOMIZE_BASE - mov x0, x22 - adrp x1, init_pg_end + adrp x1, early_init_stack mov sp, x1 mov x29, xzr +#ifdef CONFIG_RANDOMIZE_BASE + mov x0, x22 bl __pi_kaslr_early_init and x24, x0, #SZ_2M - 1 // capture memstart offset seed bic x0, x0, #SZ_2M - 1 @@ -894,7 +803,8 @@ SYM_FUNC_START_LOCAL(__primary_switch) adrp x1, init_pg_dir load_ttbr1 x1, x1, x2 #ifdef CONFIG_RELOCATABLE - bl __relocate_kernel + mov x0, x23 + bl __pi_relocate_kernel #endif ldr x8, =__primary_switched adrp x0, KERNEL_START // __pa(KERNEL_START) diff --git a/arch/arm64/kernel/pi/Makefile b/arch/arm64/kernel/pi/Makefile index 2bbe866417d4..d084c1dcf416 100644 --- a/arch/arm64/kernel/pi/Makefile +++ b/arch/arm64/kernel/pi/Makefile @@ -38,5 +38,6 @@ $(obj)/lib-%.pi.o: OBJCOPYFLAGS += --prefix-alloc-sections=.init $(obj)/lib-%.o: $(srctree)/lib/%.c FORCE $(call if_changed_rule,cc_o_c) -obj-y := kaslr_early.pi.o lib-fdt.pi.o lib-fdt_ro.pi.o -extra-y := $(patsubst %.pi.o,%.o,$(obj-y)) +obj-y := relocate.pi.o +obj-$(CONFIG_RANDOMIZE_BASE) += kaslr_early.pi.o lib-fdt.pi.o lib-fdt_ro.pi.o +extra-y := $(patsubst %.pi.o,%.o,$(obj-y)) diff --git a/arch/arm64/kernel/pi/relocate.c b/arch/arm64/kernel/pi/relocate.c new file mode 100644 index 000000000000..1853408ea76b --- /dev/null +++ b/arch/arm64/kernel/pi/relocate.c @@ -0,0 +1,62 @@ +// SPDX-License-Identifier: GPL-2.0-only +// Copyright 2023 Google LLC +// Authors: Ard Biesheuvel +// Peter Collingbourne + +#include +#include +#include + +extern const Elf64_Rela rela_start[], rela_end[]; +extern const u64 relr_start[], relr_end[]; + +void __init relocate_kernel(u64 offset) +{ + u64 *place = NULL; + + for (const Elf64_Rela *rela = rela_start; rela < rela_end; rela++) { + if (ELF64_R_TYPE(rela->r_info) != R_AARCH64_RELATIVE) + continue; + *(u64 *)(rela->r_offset + offset) = rela->r_addend + offset; + } + + if (!IS_ENABLED(CONFIG_RELR) || !offset) + return; + + /* + * Apply RELR relocations. + * + * RELR is a compressed format for storing relative relocations. The + * encoded sequence of entries looks like: + * [ AAAAAAAA BBBBBBB1 BBBBBBB1 ... AAAAAAAA BBBBBB1 ... ] + * + * i.e. start with an address, followed by any number of bitmaps. The + * address entry encodes 1 relocation. The subsequent bitmap entries + * encode up to 63 relocations each, at subsequent offsets following + * the last address entry. + * + * The bitmap entries must have 1 in the least significant bit. The + * assumption here is that an address cannot have 1 in lsb. Odd + * addresses are not supported. Any odd addresses are stored in the + * RELA section, which is handled above. + * + * With the exception of the least significant bit, each bit in the + * bitmap corresponds with a machine word that follows the base address + * word, and the bit value indicates whether or not a relocation needs + * to be applied to it. The second least significant bit represents the + * machine word immediately following the initial address, and each bit + * that follows represents the next word, in linear order. As such, a + * single bitmap can encode up to 63 relocations in a 64-bit object. + */ + for (const u64 *relr = relr_start; relr < relr_end; relr++) { + if ((*relr & 1) == 0) { + place = (u64 *)(*relr + offset); + *place++ += offset; + } else { + for (u64 *p = place, r = *relr >> 1; r; p++, r >>= 1) + if (r & 1) + *p += offset; + place += 63; + } + } +} diff --git a/arch/arm64/kernel/vmlinux.lds.S b/arch/arm64/kernel/vmlinux.lds.S index 3cd7e76cc562..8dd5dda66f7c 100644 --- a/arch/arm64/kernel/vmlinux.lds.S +++ b/arch/arm64/kernel/vmlinux.lds.S @@ -270,15 +270,15 @@ SECTIONS HYPERVISOR_RELOC_SECTION .rela.dyn : ALIGN(8) { - __rela_start = .; + __pi_rela_start = .; *(.rela .rela*) - __rela_end = .; + __pi_rela_end = .; } .relr.dyn : ALIGN(8) { - __relr_start = .; + __pi_relr_start = .; *(.relr.dyn) - __relr_end = .; + __pi_relr_end = .; } . = ALIGN(SEGMENT_ALIGN); @@ -317,6 +317,10 @@ SECTIONS init_pg_dir = .; . += INIT_DIR_SIZE; init_pg_end = .; +#ifdef CONFIG_RELOCATABLE + . += SZ_4K; /* stack for the early relocation code */ + early_init_stack = .; +#endif . = ALIGN(SEGMENT_ALIGN); __pecoff_data_size = ABSOLUTE(. - __initdata_begin); From patchwork Wed Feb 14 12:28:50 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13556409 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 51A0CC48BC3 for ; Wed, 14 Feb 2024 12:30:33 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=awkaRNZQWCY/gVZn+OCDbIL6B0u+UP4ZIllf1zq/cYo=; b=GnB+vdKpIAYRgi+jrTYKlFlsBG 3UzTZSshCkjtrPqQzy/WC6gdlCr7KXDrapyjkEFQNNXm6zpW6rie73ZcTuQwJ6/XHH+QLF0hnbdbe v7VpuagN13HGgjou6V275fbx4uVyN4CQO1QZ/3EQ1XrakkHwJsI/x6PTn213oE8wdUDWfo24YN2kG K3Qbqk+aZWshR+DWM9UGFLO9hZQsSrTonW12NlMa0hCftxA9FHUeMh7b2TM6HbvWFeeliolxbKLNr l8oNP0q7gX+1kgDe+8xVuwFmG4AMR13ZrQrZ2fitr8jLaQLFGmxsWse0qWCTR5LdWq0rx3rkTQMeW YBwi+xmQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1raEP5-0000000CoXA-2yIB; Wed, 14 Feb 2024 12:30:19 +0000 Received: from mail-wm1-x349.google.com ([2a00:1450:4864:20::349]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1raEOe-0000000CoF9-1VUO for linux-arm-kernel@lists.infradead.org; Wed, 14 Feb 2024 12:29:53 +0000 Received: by mail-wm1-x349.google.com with SMTP id 5b1f17b1804b1-41082621642so25901335e9.1 for ; Wed, 14 Feb 2024 04:29:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1707913787; x=1708518587; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=XG8Zu0ENW1+A4cgiBmVR2896ZmBQV5+bZyVFMfnmZMU=; b=4u8owcMgN/1NScGgi4DXvjKZaPQCRu3hIDnTgUHnwGomq4BpIU2bmoZK6J/NSxAPtA gNHS1lxnM7giD2Vi1rW/mNaKsMT3nvjcsjgBwEY/lcfgRgPAY+819i/WoW01PZOBCyQx sPZEYWhipPqvxYXn/ERJjiUfF/XxE5/o/puROxacNLxVFSxquRx47jX7rIDqO+q7saeK 036mNQhDDmFaaB5ButEO/8wUxSkXj3isq4QCYKDfWeJtUSr7UBsVu2Vc1MrlkvVDGnZk vurVpri6GebVr6AUOjRze7FNfXaZ480ArmHVO7xgTDeEmbGDIyGzUPWhSwTYrxu02bAU DAvg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707913787; x=1708518587; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=XG8Zu0ENW1+A4cgiBmVR2896ZmBQV5+bZyVFMfnmZMU=; b=URKswQWde+sOEunWqoqeyFzhmQhELB6HTxLALm3cEKrMGsdg3ymUYuMSLP78odNdds icOaq9zvz2u0s2WtX6a2yGO5Dpp4KHGSjYu1SQohfzrbwyFnJWzMPNNiOCZcQYPaEs5E jMnqgp2pXb893fIH/ebl1D19tZ6CRVb8Q7TApYC8b8tn9k3bPJPiCLWoxxue7Tie8sBt OUF9TQja7PNotWlSeUS5HyPQ3T9dPFWEdfRiF4bOs/hiJZjFgVoNvkWE25lL33FfABnf IP5H7retA7vf3nIziax39+nr617/k9zsZ7FQoS4FiXb9lnLYO2j3JQAQY2PscuglSiFF SCYQ== X-Gm-Message-State: AOJu0Yxp9EdF+4jmTHES4SM7yyze4kywusV94ePv3RmQ9m9D8i87A23l dnUwLEY3//Myw6+yGrT59SUasR1zHqQhMx91RLEkXlNL8AMvxoS4dq8pFV/ghKzvFOIF/vwIeFb o22/IU9JfEkP0Zz4LDBA8L1xDboSeO78X+wuef8+3yr/Wbzr3yzO3rCcfrLwWJ36OQbQ1gw9K5a sNegL0Svsvtw7rNenYEYsnz4J/rZdCfCP+2XdMuWlI X-Google-Smtp-Source: AGHT+IHfGX/XRAPIJ4y4GwWJRLFpqd5jqi7I+64YCEZt3MSH6xkkAEUn0CfJRseIj2H22LE4eo/G9hQY X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:600c:6028:b0:411:f8e8:43ba with SMTP id az40-20020a05600c602800b00411f8e843bamr2940wmb.3.1707913787806; Wed, 14 Feb 2024 04:29:47 -0800 (PST) Date: Wed, 14 Feb 2024 13:28:50 +0100 In-Reply-To: <20240214122845.2033971-45-ardb+git@google.com> Mime-Version: 1.0 References: <20240214122845.2033971-45-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=6698; i=ardb@kernel.org; h=from:subject; bh=bGDdRGo2eEjJsbNpn6rx20wBfVEFH3nJ4bYP+rkGLYY=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIfXMJiYB/wVsW9RsT/P7xyzYHsKaKnA+MOyCYdO834l/t FfmNC7pKGVhEONgkBVTZBGY/ffdztMTpWqdZ8nCzGFlAhnCwMUpABPR2c/I0Dq51nI9w9sprSYG 4q7OXP+qXs6SrTPPnmd0qK/gYpaQNSPDg3UuYiYMFtbr3WICzM3WvMuYvfWEw2YXaRbZDfE7X9g yAAA= X-Mailer: git-send-email 2.43.0.687.g38aa6559b0-goog Message-ID: <20240214122845.2033971-49-ardb+git@google.com> Subject: [PATCH v8 04/43] arm64: idreg-override: Move to early mini C runtime From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240214_042952_452547_F4EB3604 X-CRM114-Status: GOOD ( 17.28 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel We will want to parse the ID register overrides even earlier, so that we can take them into account before creating the kernel mapping. So migrate the code and make it work in the context of the early C runtime. We will move the invocation to an earlier stage in a subsequent patch. Signed-off-by: Ard Biesheuvel --- arch/arm64/kernel/Makefile | 4 +-- arch/arm64/kernel/head.S | 5 ++- arch/arm64/kernel/image-vars.h | 9 +++++ arch/arm64/kernel/pi/Makefile | 5 +-- arch/arm64/kernel/{ => pi}/idreg-override.c | 35 +++++++++----------- 5 files changed, 30 insertions(+), 28 deletions(-) diff --git a/arch/arm64/kernel/Makefile b/arch/arm64/kernel/Makefile index 78f14084f6d7..4236f1e0fffa 100644 --- a/arch/arm64/kernel/Makefile +++ b/arch/arm64/kernel/Makefile @@ -33,8 +33,7 @@ obj-y := debug-monitors.o entry.o irq.o fpsimd.o \ return_address.o cpuinfo.o cpu_errata.o \ cpufeature.o alternative.o cacheinfo.o \ smp.o smp_spin_table.o topology.o smccc-call.o \ - syscall.o proton-pack.o idreg-override.o idle.o \ - patching.o + syscall.o proton-pack.o idle.o patching.o pi/ obj-$(CONFIG_COMPAT) += sys32.o signal32.o \ sys_compat.o @@ -57,7 +56,6 @@ obj-$(CONFIG_ACPI) += acpi.o obj-$(CONFIG_ACPI_NUMA) += acpi_numa.o obj-$(CONFIG_ARM64_ACPI_PARKING_PROTOCOL) += acpi_parking_protocol.o obj-$(CONFIG_PARAVIRT) += paravirt.o -obj-$(CONFIG_RELOCATABLE) += pi/ obj-$(CONFIG_RANDOMIZE_BASE) += kaslr.o obj-$(CONFIG_HIBERNATION) += hibernate.o hibernate-asm.o obj-$(CONFIG_ELF_CORE) += elfcore.o diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S index a8fa64fc30d7..ca5e5fbefcd3 100644 --- a/arch/arm64/kernel/head.S +++ b/arch/arm64/kernel/head.S @@ -510,10 +510,9 @@ SYM_FUNC_START_LOCAL(__primary_switched) #if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS) bl kasan_early_init #endif - mov x0, x21 // pass FDT address in x0 - bl early_fdt_map // Try mapping the FDT early mov x0, x20 // pass the full boot status - bl init_feature_override // Parse cpu feature overrides + mov x1, x22 // pass the low FDT mapping + bl __pi_init_feature_override // Parse cpu feature overrides #ifdef CONFIG_UNWIND_PATCH_PAC_INTO_SCS bl scs_patch_vmlinux #endif diff --git a/arch/arm64/kernel/image-vars.h b/arch/arm64/kernel/image-vars.h index e931ce078a00..eacc3d167733 100644 --- a/arch/arm64/kernel/image-vars.h +++ b/arch/arm64/kernel/image-vars.h @@ -37,6 +37,15 @@ PROVIDE(__pi___memmove = __pi_memmove); PROVIDE(__pi___memset = __pi_memset); PROVIDE(__pi_vabits_actual = vabits_actual); +PROVIDE(__pi_id_aa64isar1_override = id_aa64isar1_override); +PROVIDE(__pi_id_aa64isar2_override = id_aa64isar2_override); +PROVIDE(__pi_id_aa64mmfr1_override = id_aa64mmfr1_override); +PROVIDE(__pi_id_aa64pfr0_override = id_aa64pfr0_override); +PROVIDE(__pi_id_aa64pfr1_override = id_aa64pfr1_override); +PROVIDE(__pi_id_aa64smfr0_override = id_aa64smfr0_override); +PROVIDE(__pi_id_aa64zfr0_override = id_aa64zfr0_override); +PROVIDE(__pi_arm64_sw_feature_override = arm64_sw_feature_override); +PROVIDE(__pi__ctype = _ctype); #ifdef CONFIG_KVM diff --git a/arch/arm64/kernel/pi/Makefile b/arch/arm64/kernel/pi/Makefile index d084c1dcf416..7f6dfce893c3 100644 --- a/arch/arm64/kernel/pi/Makefile +++ b/arch/arm64/kernel/pi/Makefile @@ -38,6 +38,7 @@ $(obj)/lib-%.pi.o: OBJCOPYFLAGS += --prefix-alloc-sections=.init $(obj)/lib-%.o: $(srctree)/lib/%.c FORCE $(call if_changed_rule,cc_o_c) -obj-y := relocate.pi.o -obj-$(CONFIG_RANDOMIZE_BASE) += kaslr_early.pi.o lib-fdt.pi.o lib-fdt_ro.pi.o +obj-y := idreg-override.pi.o lib-fdt.pi.o lib-fdt_ro.pi.o +obj-$(CONFIG_RELOCATABLE) += relocate.pi.o +obj-$(CONFIG_RANDOMIZE_BASE) += kaslr_early.pi.o extra-y := $(patsubst %.pi.o,%.o,$(obj-y)) diff --git a/arch/arm64/kernel/idreg-override.c b/arch/arm64/kernel/pi/idreg-override.c similarity index 93% rename from arch/arm64/kernel/idreg-override.c rename to arch/arm64/kernel/pi/idreg-override.c index e30fd9e32ef3..f9e05c10faab 100644 --- a/arch/arm64/kernel/idreg-override.c +++ b/arch/arm64/kernel/pi/idreg-override.c @@ -14,6 +14,8 @@ #include #include +#include "pi.h" + #define FTR_DESC_NAME_LEN 20 #define FTR_DESC_FIELD_LEN 10 #define FTR_ALIAS_NAME_LEN 30 @@ -21,15 +23,6 @@ static u64 __boot_status __initdata; -// temporary __prel64 related definitions -// to be removed when this code is moved under pi/ - -#define __prel64_initconst __initconst - -#define PREL64(type, name) union { type *name; } - -#define prel64_pointer(__d) (__d) - typedef bool filter_t(u64 val); struct ftr_set_desc { @@ -313,16 +306,11 @@ static __init void __parse_cmdline(const char *cmdline, bool parse_aliases) } while (1); } -static __init const u8 *get_bootargs_cmdline(void) +static __init const u8 *get_bootargs_cmdline(const void *fdt) { const u8 *prop; - void *fdt; int node; - fdt = get_early_fdt_ptr(); - if (!fdt) - return NULL; - node = fdt_path_offset(fdt, "/chosen"); if (node < 0) return NULL; @@ -334,9 +322,9 @@ static __init const u8 *get_bootargs_cmdline(void) return strlen(prop) ? prop : NULL; } -static __init void parse_cmdline(void) +static __init void parse_cmdline(const void *fdt) { - const u8 *prop = get_bootargs_cmdline(); + const u8 *prop = get_bootargs_cmdline(fdt); if (IS_ENABLED(CONFIG_CMDLINE_FORCE) || !prop) __parse_cmdline(CONFIG_CMDLINE, true); @@ -346,9 +334,9 @@ static __init void parse_cmdline(void) } /* Keep checkers quiet */ -void init_feature_override(u64 boot_status); +void init_feature_override(u64 boot_status, const void *fdt); -asmlinkage void __init init_feature_override(u64 boot_status) +asmlinkage void __init init_feature_override(u64 boot_status, const void *fdt) { struct arm64_ftr_override *override; const struct ftr_set_desc *reg; @@ -364,7 +352,7 @@ asmlinkage void __init init_feature_override(u64 boot_status) __boot_status = boot_status; - parse_cmdline(); + parse_cmdline(fdt); for (i = 0; i < ARRAY_SIZE(regs); i++) { reg = prel64_pointer(regs[i].reg); @@ -373,3 +361,10 @@ asmlinkage void __init init_feature_override(u64 boot_status) (unsigned long)(override + 1)); } } + +char * __init skip_spaces(const char *str) +{ + while (isspace(*str)) + ++str; + return (char *)str; +} From patchwork Wed Feb 14 12:28:51 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13556407 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 6A013C48BC1 for ; Wed, 14 Feb 2024 12:30:28 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=mGibhyvw73QOx+UVxWa0nVDNstLpu6mpANhX+fA7KCU=; b=a4ByhgI03Pf+We09c5/wNJGcC4 rNbeGaOd30rv/L49GCN4s66yFdD9QXu+IdI0THxW9XV5OrBK5IgX6p2dD/hfGUq4+umh262qsn/W9 NVwUaAtMs5yRCVhwszTls2n49WxL/nBqZ0QZLvboxWTbNLnrMs70dzRKMrcUl6zcRY08SOSgCr3Dx jqFIW/qN1yBa6E/D2jlfOzsoUvIZ6hG/zIGo7SsPfPr2/+BuYyHmGqMdtmidAAa+/tP3aSdpSh44D Sa3TVLzqZeatTsO3PB7AT2IBIS/PlGVHg46CI1OzsoHaLFPXWwmZsroT5ajH1c/GOLJedo0irGktI aEZEtCQw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1raEP3-0000000CoW7-3fy5; Wed, 14 Feb 2024 12:30:17 +0000 Received: from mail-yw1-x114a.google.com ([2607:f8b0:4864:20::114a]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1raEOd-0000000CoFf-34dm for linux-arm-kernel@lists.infradead.org; Wed, 14 Feb 2024 12:29:53 +0000 Received: by mail-yw1-x114a.google.com with SMTP id 00721157ae682-6077ec913f0so42256167b3.0 for ; Wed, 14 Feb 2024 04:29:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1707913790; x=1708518590; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=lrF6jre10ANDgtAh0cN9mGDQMz+5gBr0rftW/uSo1S4=; b=ccgo/LMXQMJo1REYa1QpE7eOosjIaM9Gh5JYzc7PZWZQ9FFbEJF2uX+TOCn7AnkA8/ YSR0EbGYH1FFa8LYOK8B0H0XHkzaUOQ5biOJIrkHmpI3+NbI6oU/VuoGIcjKRXcQ6lTc +YqoMH/bxSZPIFkXteCHnjQ+dX574zjJe47FCzl96BfBAjH/OiwPNHGyktYhM2TNsPIq lrOyjb3QUr4xV9DIUWaLCmP7AxFmbKFLEEap7GqiU7m3QB23M5KqoqF4gI9ENUPKc2C6 6GkwVDzQRl4kdMrLIJqRrTpKw+8IGomqOasaGPXt7Pf2AOinQU4PaIDFzgAU0OJ+TBBG aQ7Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707913790; x=1708518590; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=lrF6jre10ANDgtAh0cN9mGDQMz+5gBr0rftW/uSo1S4=; b=gV8aFTqHfJtfsAEJsoLeM5GPWGioI0T4m4OMWPVT5gc5t+KKz4n8wk+ZLNKK6VisfT pd+lLVrwn2qeLiv+XJ4ZxykSAx2FdY6sNk3RNkpBl6iMEuEW2vyadxXwtW9MFQ/rHu8b 4DSI9NDRaZhn8+1F/h+yTVwnxbAEYH4OKUnGq2OeEiymVwQEbnh2hq/aFQEKDvFJ0XJL /tabb8vx2FwoyYj6wqOoC5r4/2GqTCJfy46Pl0L/2CeHrQ8zVSeOGcoDk3+lWn9h3muI 3WluKAYonqTCZqbQj/Y8nxLjhC+AJglpAVNxJcOElzwnd4OeApfE47m9bhkBMwDtZ6IM FEfg== X-Gm-Message-State: AOJu0YxZwcgnzSmGmcSchEPLmzkGE/gJiCxDlJ10vCILHvYIU2pKjctr nM0O6G36PxoZnClcyZOP8u0AXRPq0CdUQFnm/WCjBFAH7S6sXd/6Cffex9fXyJnoOR3mcTSIqMy KG6mx1CVNaEf03CEQhCbXKIvFgYChSmWBPmTB9GtQSl7BUi7crAkkD6rnv0OBF5gmsQwrV9syv8 XnCHaqhAtQEw2HI9ljDSF4jZUidF7mB1zm1VSVEvWS X-Google-Smtp-Source: AGHT+IG3bmzi8HNG4atEWDqtKezPybx05PRkbAdHQlYJGw62PiAPE+Bkenhol8TQmjRAeSFXfRwgQGN2 X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:6902:1007:b0:dc9:5ef8:2b2d with SMTP id w7-20020a056902100700b00dc95ef82b2dmr553900ybt.4.1707913790202; Wed, 14 Feb 2024 04:29:50 -0800 (PST) Date: Wed, 14 Feb 2024 13:28:51 +0100 In-Reply-To: <20240214122845.2033971-45-ardb+git@google.com> Mime-Version: 1.0 References: <20240214122845.2033971-45-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=1354; i=ardb@kernel.org; h=from:subject; bh=ifIREkq0L6sdhnfitGwY8AL4Fq7CqFcNUukzXs7lcG8=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIfXMJmbFsJ9PtAPXLIz/7/N5Ts4Nz84m00kylmvDclX4k qYt46/rKGVhEONgkBVTZBGY/ffdztMTpWqdZ8nCzGFlAhnCwMUpABOJ1GNk2MfhsCjO/i5j/jWe W5z/8piOe21d9m9GW4HSvpt67q/3TWH471Lo3Hd5pXqAfVDU8p+FLFXXCqwSviXePzSpqSWpOr2 MGwA= X-Mailer: git-send-email 2.43.0.687.g38aa6559b0-goog Message-ID: <20240214122845.2033971-50-ardb+git@google.com> Subject: [PATCH v8 05/43] arm64: kernel: Remove early fdt remap code From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240214_042951_804529_0E6A6484 X-CRM114-Status: GOOD ( 10.39 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel The early FDT remap code is no longer used so let's drop it. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/setup.h | 3 --- arch/arm64/kernel/setup.c | 15 --------------- 2 files changed, 18 deletions(-) diff --git a/arch/arm64/include/asm/setup.h b/arch/arm64/include/asm/setup.h index 2e4d7da74fb8..ba269a7a3201 100644 --- a/arch/arm64/include/asm/setup.h +++ b/arch/arm64/include/asm/setup.h @@ -7,9 +7,6 @@ #include -void *get_early_fdt_ptr(void); -void early_fdt_map(u64 dt_phys); - /* * These two variables are used in the head.S file. */ diff --git a/arch/arm64/kernel/setup.c b/arch/arm64/kernel/setup.c index 42c690bb2d60..97d2143669cf 100644 --- a/arch/arm64/kernel/setup.c +++ b/arch/arm64/kernel/setup.c @@ -166,21 +166,6 @@ static void __init smp_build_mpidr_hash(void) pr_warn("Large number of MPIDR hash buckets detected\n"); } -static void *early_fdt_ptr __initdata; - -void __init *get_early_fdt_ptr(void) -{ - return early_fdt_ptr; -} - -asmlinkage void __init early_fdt_map(u64 dt_phys) -{ - int fdt_size; - - early_fixmap_init(); - early_fdt_ptr = fixmap_remap_fdt(dt_phys, &fdt_size, PAGE_KERNEL); -} - static void __init setup_machine_fdt(phys_addr_t dt_phys) { int size; From patchwork Wed Feb 14 12:28:52 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13556408 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id F14DFC48BC1 for ; Wed, 14 Feb 2024 12:30:32 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=oXbXwgMMWmy3SzExzne7NTgntMhA7xhWk6m9L+fpx+Y=; b=P+omMZ2V+Dks8lkFTnWfEd7SXQ 6AOARZZZr7OcMKyxntaEJbtADuBxweG0dhCQUurkJFKNpafXlVHwU/+bwA6TJzCmFiym7ZXj1Tt7c re1lxifbdpBHCfkTJyLJlSHszsMBuuyJ0ojUhQK9pPzbRNJwUk1D/CyecfThC/auyH5TQVl7P6su4 YVJGLunOn7TkF/Vc6qccP1Cge/Iw7hzSsjlhfLX5gYfSrS4WTz5gHs5LVrV+omdbUNUZq3MZ3sUQz RN1HFHLHvKG3qDdo2Jf1eO3En1aunrC4jXFRygklKfbqL+0KSroICVHfOxbrOl16P6FJUfCGJCGrm ouAUEVHA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1raEP7-0000000CoYE-2npJ; Wed, 14 Feb 2024 12:30:21 +0000 Received: from mail-yb1-xb49.google.com ([2607:f8b0:4864:20::b49]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1raEOg-0000000CoGp-0iWY for linux-arm-kernel@lists.infradead.org; Wed, 14 Feb 2024 12:29:55 +0000 Received: by mail-yb1-xb49.google.com with SMTP id 3f1490d57ef6-dc64b659a9cso9348196276.3 for ; Wed, 14 Feb 2024 04:29:53 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1707913792; x=1708518592; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=ssYO1q2/J7UmM6YLohBW2jBC2yELEfuEpqD7FWGV1jE=; b=byJEGW9YUp8YR9Kz62qAgNfv4kHUHwL2kJ8Cp3ZeoNz9QpbyeQBWUU4DjB+tV5HJTB zegwmYM1IvTBuWA1ifrw4pZRQ2FW+cXXNoge3nj0tsTvi3xxVd2zcuYyEjy4uC663AHr gqQqWmddg/EpxxKdt9NnK9Ofxl9ILvtpw4Wo/DuuIAgsvY8jZbj+ssjqV37kwgTegJ3k LNOdayzJOAf4FEPxHffefx+Y+76o1ARERL0gXcEyE9mDcGv//ckp8NXu2xPW22p6sXUh yOiRChiI0eF4usdr4r9JRLDfySDcHha3WomPCtwsL87qAmMc0L+6WVzw2UgrfzLXv4yK 2mcQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707913792; x=1708518592; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=ssYO1q2/J7UmM6YLohBW2jBC2yELEfuEpqD7FWGV1jE=; b=nqRB5/jqHNaT2MRts8LzDAvAAtMkVUnvzFghghGpnCGE5gXbweYaCFqvv430G8Ik/v rbSas0gii0mtQBrJ9GDmTpeHhAjQBeRJfPKHGPAbJ4KLtS3EkGFRAn1Q+fYgtCN9gTl/ 1o5uLKMotJ84qL4UJQuvqVHHk/wj59VsDFnfcndp98nolENz9G3DiQX5qpPqQT8G5WG4 5u/Nvhrw26vOrtOmRvl7l5rIJgvKdY5IQ3GQUK36AyjDMGqs3lwreK/Lp3LLG0GFxHUJ Ek0gqyMG22DW/YQYIVqxFgQZL7shBnOenhJVB2uU3tzFV3AJRJ1SBQULnZMnCkQq+cW4 1Ryw== X-Gm-Message-State: AOJu0YyM3sGcP6krWjVOCOSmmQTNLOflaocn4jvbhbYuC47nZa+CNWEJ IZl0gnEXOgc/b62KZPFRkTHLtsX+MlQT17CSPH3gL6CmnkM79QqO8pg0DgjnTar7uK9cP6aeS5J zBheIu/l5aNDIgynKyMPwNDidbHBsFgSdLNZ4F5rNMJo3UF+Px7KHTG917lU+zKnlEmOVir4/hN Mr8o1ssBBGBXB5cDjXiGp/GNWC1P5ZkPU+ouyYYWji X-Google-Smtp-Source: AGHT+IHk/+Pp1Q+ziAYKMLa2tIErVRB7Fy9TvlM/htup92KPDYP/d6bQe9suWerVbcwNLeio9MwXQ5dt X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:6902:1027:b0:dc2:4ab7:3d89 with SMTP id x7-20020a056902102700b00dc24ab73d89mr567410ybt.1.1707913792387; Wed, 14 Feb 2024 04:29:52 -0800 (PST) Date: Wed, 14 Feb 2024 13:28:52 +0100 In-Reply-To: <20240214122845.2033971-45-ardb+git@google.com> Mime-Version: 1.0 References: <20240214122845.2033971-45-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=3389; i=ardb@kernel.org; h=from:subject; bh=FuEpWiYD4ARO9/4KLGaHk3hLzIqaeMWlfGeGxZwdczc=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIfXMJpYs2/0NFXx3GXf+ERBdMP1pfZ/mQYuJr4tzMxlfO Sv2PX/eUcrCIMbBICumyCIw+++7nacnStU6z5KFmcPKBDKEgYtTACZyeiLDf2/pndI7nTovpFuf uvRjVl/33c+6pj7JD043PbLe9OOx9xZGhic9N6/3ON09VpRRzrr/pkbnDssXZ+d6VBbZqcSWHm7 +wQwA X-Mailer: git-send-email 2.43.0.687.g38aa6559b0-goog Message-ID: <20240214122845.2033971-51-ardb+git@google.com> Subject: [PATCH v8 06/43] arm64: head: Clear BSS and the kernel page tables in one go From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240214_042954_346823_14554C46 X-CRM114-Status: GOOD ( 15.49 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel We will move the CPU feature overrides into BSS in a subsequent patch, and this requires that BSS is zeroed before the feature override detection code runs. So let's map BSS read-write in the ID map, and zero it via this mapping. Since the kernel page tables are right next to it, and also zeroed via the ID map, let's drop the separate clear_page_tables() function, and just zero everything in one go. Signed-off-by: Ard Biesheuvel --- arch/arm64/kernel/head.S | 33 +++++++------------- arch/arm64/kernel/vmlinux.lds.S | 3 ++ 2 files changed, 14 insertions(+), 22 deletions(-) diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S index ca5e5fbefcd3..2af518161f3a 100644 --- a/arch/arm64/kernel/head.S +++ b/arch/arm64/kernel/head.S @@ -177,17 +177,6 @@ SYM_CODE_START_LOCAL(preserve_boot_args) ret SYM_CODE_END(preserve_boot_args) -SYM_FUNC_START_LOCAL(clear_page_tables) - /* - * Clear the init page tables. - */ - adrp x0, init_pg_dir - adrp x1, init_pg_end - sub x2, x1, x0 - mov x1, xzr - b __pi_memset // tail call -SYM_FUNC_END(clear_page_tables) - /* * Macro to populate page table entries, these entries can be pointers to the next level * or last level entries pointing to physical memory. @@ -386,9 +375,9 @@ SYM_FUNC_START_LOCAL(create_idmap) map_memory x0, x1, x3, x6, x7, x3, IDMAP_PGD_ORDER, x10, x11, x12, x13, x14, EXTRA_SHIFT - /* Remap the kernel page tables r/w in the ID map */ + /* Remap BSS and the kernel page tables r/w in the ID map */ adrp x1, _text - adrp x2, init_pg_dir + adrp x2, __bss_start adrp x3, _end bic x4, x2, #SWAPPER_BLOCK_SIZE - 1 mov_q x5, SWAPPER_RW_MMUFLAGS @@ -489,14 +478,6 @@ SYM_FUNC_START_LOCAL(__primary_switched) mov x0, x20 bl set_cpu_boot_mode_flag - // Clear BSS - adr_l x0, __bss_start - mov x1, xzr - adr_l x2, __bss_stop - sub x2, x2, x0 - bl __pi_memset - dsb ishst // Make zero page visible to PTW - #if VA_BITS > 48 adr_l x8, vabits_actual // Set this early so KASAN early init str x25, [x8] // ... observes the correct value @@ -782,6 +763,15 @@ SYM_FUNC_START_LOCAL(__primary_switch) adrp x1, reserved_pg_dir adrp x2, init_idmap_pg_dir bl __enable_mmu + + // Clear BSS + adrp x0, __bss_start + mov x1, xzr + adrp x2, init_pg_end + sub x2, x2, x0 + bl __pi_memset + dsb ishst // Make zero page visible to PTW + #ifdef CONFIG_RELOCATABLE adrp x23, KERNEL_START and x23, x23, MIN_KIMG_ALIGN - 1 @@ -796,7 +786,6 @@ SYM_FUNC_START_LOCAL(__primary_switch) orr x23, x23, x0 // record kernel offset #endif #endif - bl clear_page_tables bl create_kernel_mapping adrp x1, init_pg_dir diff --git a/arch/arm64/kernel/vmlinux.lds.S b/arch/arm64/kernel/vmlinux.lds.S index 8dd5dda66f7c..8a3c6aacc355 100644 --- a/arch/arm64/kernel/vmlinux.lds.S +++ b/arch/arm64/kernel/vmlinux.lds.S @@ -311,12 +311,15 @@ SECTIONS __pecoff_data_rawsize = ABSOLUTE(. - __initdata_begin); _edata = .; + /* start of zero-init region */ BSS_SECTION(SBSS_ALIGN, 0, 0) . = ALIGN(PAGE_SIZE); init_pg_dir = .; . += INIT_DIR_SIZE; init_pg_end = .; + /* end of zero-init region */ + #ifdef CONFIG_RELOCATABLE . += SZ_4K; /* stack for the early relocation code */ early_init_stack = .; From patchwork Wed Feb 14 12:28:53 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13556410 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id C3534C48BC3 for ; Wed, 14 Feb 2024 12:30:36 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=uqUJdJ/AOfppBi0EPA9TmUclxb7tEUBIxFHeQ1oOVn8=; b=A8aAsWX9xk14TndfKyVqLEkdot EhoLnKitM7xvse+E6mNLzFoQ9Vbm7MBGhnQ8ZpBqnZhQHLDNdsHZBy56vvqJnCAU8CoxM9TyVpAi6 sxVvvcykQ7+1M8Pf6MJ7qmzcxEedibWS88a0TVFwRUpWe3StBGY1VS8A8h//ljfmG2WBrnBr/facB 04TCy4IQ1DjJOWfzJ0rnxybnk89MzYH/CJjcA0c+gBYUVa5PgKWXqTu0/uRoexFZqcluxjGX+eKx9 k1O4UiYy11QBT/gbh1k7jd3bl0t/YhiXWx00jJjnigqokadn7wx7Sijf7MyAUOdJ0YnUebd4YWZ8w bwY6xnIA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1raEPB-0000000Coap-2t9d; Wed, 14 Feb 2024 12:30:25 +0000 Received: from mail-yb1-xb4a.google.com ([2607:f8b0:4864:20::b4a]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1raEOk-0000000CoIR-3rgr for linux-arm-kernel@lists.infradead.org; Wed, 14 Feb 2024 12:30:00 +0000 Received: by mail-yb1-xb4a.google.com with SMTP id 3f1490d57ef6-dc6b26845cdso9295015276.3 for ; Wed, 14 Feb 2024 04:29:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1707913795; x=1708518595; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=oz98kvsb+bZg7S2DdRRRX0fkqMKpKmwxnSQc1WD65MI=; b=wzY+gtIkkvHEdlHd8FZ360i9PzDQ7orijv/xYBKzu3v/+yHS50NlwfxJKgcCcAfryg RnhZncdnUXEaJjacNibFNsy8s/8uduv8Icpw0F6HiRKnjE24jHuMeqD2kiETBTi6WruM +zmqipU/V3hY8KD3L1RKotCxehTXANLZPEgqI9wABzIniPStEr0+joQ3bOQz+Car2sQD ivOSzva2izH41tHDgO7QSM0xNGddsWwwbACaOIQTCtwbIDgIUDZ4Wyhhatv88uW5pk17 DjutNRaYc6yoHqaiRCWdRKLAHFxJcKrW73YMPQTLBAVAWI+YyvPILnUAB6vMHUao0KZ4 +ZbQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707913795; x=1708518595; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=oz98kvsb+bZg7S2DdRRRX0fkqMKpKmwxnSQc1WD65MI=; b=D1oJLlcodOPj/aBXgjb5Caroe0D+f1peAnt2TiqsS1MTsSXpSXIttx+l6iL08Fsev7 bq8s/aXIMnh/vN4XAB/33MTEYJcleVapf8QT+R70DT+bId/NxXh//EDdp1C7Ztpcv3Qd yTE2gHX7mvZdcOn33x7LFFcL6ZICcOd23ARgM474C3Pt+A9WbmjBnPlIXr93xppGiuS9 WKAnP2d6Uh1ST/EB5nUX8EZ0Gf0EAQKsiiKH27z0NbPCHO3CF1eR2wUJJ56qUbW+WqTD 9gHzt7jvVBgxjv0kQcdnkduZt4rHDsHoCAGGkB1TDbcsDAdlJRiWvmd5QcZYqAZGjP5Y 9RXw== X-Gm-Message-State: AOJu0YwM6Jg72GR60iLkFQzQ4M0KUC6qEyF8zGa/MIbCto6p6PuzUa6+ 6qi4tRbZAH/g9cJYllM/YIB3BG/YgjDBMnqF/XGkNeC14+l0GHJ/7d/6nhytiFv7othpluF7uIY /BQXh2DrsWMspFqhWarCa+MSfhnxUzR3MLWCRHi9VRPIkkCDjLLhAopNUW5Fj0ILuyt4k967JoY 9uN1UCm0tE+GRN3MsLvN0HOf+lJJfBjdzJLtdjCq52 X-Google-Smtp-Source: AGHT+IE9MPRp/KXEmft3TzxcvHGFVnkFocQCUrGgMY2Tj2F1Mvkg6abWQylA0Y5PrUHc45/wAWLMv1zr X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:6902:1007:b0:dc9:5ef8:2b2d with SMTP id w7-20020a056902100700b00dc95ef82b2dmr553929ybt.4.1707913794574; Wed, 14 Feb 2024 04:29:54 -0800 (PST) Date: Wed, 14 Feb 2024 13:28:53 +0100 In-Reply-To: <20240214122845.2033971-45-ardb+git@google.com> Mime-Version: 1.0 References: <20240214122845.2033971-45-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=1677; i=ardb@kernel.org; h=from:subject; bh=1Gq4KUOpyU86syaISafZ6nXlZoqyz3apMz9awYk9DKI=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIfXMJrY5xtnyE6z7/z3drb4+85bvBEnhcLGk8yoX/iTfT Ene8vJCRykLgxgHg6yYIovA7L/vdp6eKFXrPEsWZg4rE8gQBi5OAZiIcCbDP7P6eczc5o2fvkSl MqlKOk6L6/bYsu9703Ot6XfE9vu8F2L4K1J4yqfgjYzesxmVmWx/3py88v7kS4aSI/ZeiTKiBWk JLAA= X-Mailer: git-send-email 2.43.0.687.g38aa6559b0-goog Message-ID: <20240214122845.2033971-52-ardb+git@google.com> Subject: [PATCH v8 07/43] arm64: Move feature overrides into the BSS section From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240214_042959_018340_C45F65BB X-CRM114-Status: GOOD ( 11.43 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel In order to allow the CPU feature override detection code to run even earlier, move the feature override global variables into BSS, which is the only part of the static kernel image that is mapped read-write in the initial ID map. Signed-off-by: Ard Biesheuvel --- arch/arm64/kernel/cpufeature.c | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c index 8d1a634a403e..a99ad79adee2 100644 --- a/arch/arm64/kernel/cpufeature.c +++ b/arch/arm64/kernel/cpufeature.c @@ -655,13 +655,13 @@ static const struct arm64_ftr_bits ftr_raz[] = { #define ARM64_FTR_REG(id, table) \ __ARM64_FTR_REG_OVERRIDE(#id, id, table, &no_override) -struct arm64_ftr_override __ro_after_init id_aa64mmfr1_override; -struct arm64_ftr_override __ro_after_init id_aa64pfr0_override; -struct arm64_ftr_override __ro_after_init id_aa64pfr1_override; -struct arm64_ftr_override __ro_after_init id_aa64zfr0_override; -struct arm64_ftr_override __ro_after_init id_aa64smfr0_override; -struct arm64_ftr_override __ro_after_init id_aa64isar1_override; -struct arm64_ftr_override __ro_after_init id_aa64isar2_override; +struct arm64_ftr_override id_aa64mmfr1_override; +struct arm64_ftr_override id_aa64pfr0_override; +struct arm64_ftr_override id_aa64pfr1_override; +struct arm64_ftr_override id_aa64zfr0_override; +struct arm64_ftr_override id_aa64smfr0_override; +struct arm64_ftr_override id_aa64isar1_override; +struct arm64_ftr_override id_aa64isar2_override; struct arm64_ftr_override arm64_sw_feature_override; From patchwork Wed Feb 14 12:28:54 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13556411 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 9FD86C48BC1 for ; Wed, 14 Feb 2024 12:30:37 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=lFiUbacse8MAFr8+JycQvcZjVl3BDTVqtwLapN4b91w=; b=zFkBu13rJbbTdvVvQ3GYx6JO9V 4DrdP352ilyKfeXIG+uhwL8SZTI9KRQGofwDZn10xa/08aQ0mXtNb30K4NAeVM9AUkvNVHfIWXBnZ LqbqotmogTcZ7bsXoD8MFXQhO2UJAsPMrK8leVypLq5qlifSnn/pbkRdkn5OZnCny3SNecDVhOjML bZXqAOp7Wo5kDTr+cZNVcUNLEq1ZZSbOlP9RgzrLwzNuVrYOyTRM+jEwsaywTNxVQxTOYIHSvGWr8 6S320Rv8dwBVYsiSrAwCjyZCWWDTUHGSL+0gFd4c+peefRqkLIdPGVqLs+m5kXn2Qn9/eRknGUowQ 5ZQwm5QQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1raEP9-0000000CoZb-28NH; Wed, 14 Feb 2024 12:30:23 +0000 Received: from mail-yb1-xb4a.google.com ([2607:f8b0:4864:20::b4a]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1raEOk-0000000CoJQ-3idv for linux-arm-kernel@lists.infradead.org; Wed, 14 Feb 2024 12:30:00 +0000 Received: by mail-yb1-xb4a.google.com with SMTP id 3f1490d57ef6-dc647f65573so9989109276.2 for ; Wed, 14 Feb 2024 04:29:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1707913797; x=1708518597; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=yWMB8S0sfoJjHCnqUTC85QDDaeeVGIGJzG9zDMb7I1Y=; b=SC3ZeMqX/vMOHeHRgYnvfz/RrP9VMVvfyqHqYDxCLvIgukO2daSqEHyN+5gLnpQGVx NKfviM2uZhWib/YMawUPNG/gFUjU29wGqut1nOic6W3hd7c9LnFjf2wFpUpZirtW9XZj YXim+qUDjt8u2t7ojrxonG61ICJzA+TgnfqtUDo2ikd2BsZXEOYoBVVWQ8KBox62ahap PXSFRVTaV+1Ze/o9piZC7OpAnHCax7UKnwzKiX7Jo9VvT0GhPV7iEfotf0WcKnLxbVdj D/BmZH5CDN7IwdGrZksKsI7r87ttB6XLuPMCYZwqmLAVYzO6BNaMAgLiAKPVd2ysazDS GTWg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707913797; x=1708518597; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=yWMB8S0sfoJjHCnqUTC85QDDaeeVGIGJzG9zDMb7I1Y=; b=Eo2BZ8WJ0RY7tEV4FltfBCFyg0Ty2YEnu5LCnWejSLo1lc8rKQiYgv1g1JCJAuCa4r hcIBFEwlHEj8YGEbAjYzUTUSPwW21/x+VjE9FkjfTi/w1j3mjDCdd8VD8tM8nPwLbAd/ Kt3El2A6xQ8BjYxct2m7RY3kotUwR14AGWarwvv+2kBqHeV7UoluuZIeQxbzESrWiz2d tMSkEEjy7lw7odXjbIbQVv36lndx8X8PyL+lUtXvWs4c//WHLBcp+Kr366NYJKORcJSM alCuVclUBG5y8P3IZUPSBgRZo38mljv7/gYL0kW51cssioNywZG6ld65Ct+7+zs0A8xq jmwQ== X-Gm-Message-State: AOJu0YwBn+iN3aJu2KDkYjjqU5a/nHgW/VpOLF4IaQMBOYbSQ+JoAH22 dvI/nmUh4tBvxiiDUWirpDIvY8HA5kF0jO8zJLgg5lYbiuEJpanDq5RRz9W6sevhD4UPfrb8kHb eVx9OcUM3yTYS/q37n04BY0QBbIYBd8fyxTinmn9KTQ1mO+pdKSRuvtr66j0vdvOIkoNTZNqcir 4mlWIbVY3N1L1l8XxNtNuzQ5Z7pOmnDOLvmJj2lJd7 X-Google-Smtp-Source: AGHT+IGQh5qFkEgzgwrYtxxNTyCKQLL2wB2umqO6tmVVKUrPGHUEbpV/qyDJET8lFgNOI2sgMHgDAnUo X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:6902:70b:b0:dc7:8e30:e2e3 with SMTP id k11-20020a056902070b00b00dc78e30e2e3mr508048ybt.2.1707913797037; Wed, 14 Feb 2024 04:29:57 -0800 (PST) Date: Wed, 14 Feb 2024 13:28:54 +0100 In-Reply-To: <20240214122845.2033971-45-ardb+git@google.com> Mime-Version: 1.0 References: <20240214122845.2033971-45-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=2825; i=ardb@kernel.org; h=from:subject; bh=Ao6ge1QBnZt8HtdmwwtFlOcMbTxjhwll6puetzVkrqw=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIfXMJvan3xfplWss69nR2PxM4VSu0aIwwTZu96if09Y81 JrZ16feUcrCIMbBICumyCIw+++7nacnStU6z5KFmcPKBDKEgYtTACbiNIHhn6rFrW+7C0yz94Ye tFox+cBpyy1z5mUxb+XeFb5G8dafifEM/9Q3XHiy1cIlcEXg2qcNeYrhU09zq4Ts+pVnsGsDC+s eBnYA X-Mailer: git-send-email 2.43.0.687.g38aa6559b0-goog Message-ID: <20240214122845.2033971-53-ardb+git@google.com> Subject: [PATCH v8 08/43] arm64: head: Run feature override detection before mapping the kernel From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240214_042959_056488_27ACD7F0 X-CRM114-Status: GOOD ( 14.12 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel To permit the feature overrides to be taken into account before the KASLR init code runs and the kernel mapping is created, move the detection code to an earlier stage in the boot. In a subsequent patch, this will be taken advantage of by merging the preliminary and permanent mappings of the kernel text and data into a single one that gets created and relocated before start_kernel() is called. Signed-off-by: Ard Biesheuvel --- arch/arm64/kernel/head.S | 17 +++++++++-------- arch/arm64/kernel/vmlinux.lds.S | 4 +--- 2 files changed, 10 insertions(+), 11 deletions(-) diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S index 2af518161f3a..865ecc1f8255 100644 --- a/arch/arm64/kernel/head.S +++ b/arch/arm64/kernel/head.S @@ -375,9 +375,9 @@ SYM_FUNC_START_LOCAL(create_idmap) map_memory x0, x1, x3, x6, x7, x3, IDMAP_PGD_ORDER, x10, x11, x12, x13, x14, EXTRA_SHIFT - /* Remap BSS and the kernel page tables r/w in the ID map */ + /* Remap [.init].data, BSS and the kernel page tables r/w in the ID map */ adrp x1, _text - adrp x2, __bss_start + adrp x2, __initdata_begin adrp x3, _end bic x4, x2, #SWAPPER_BLOCK_SIZE - 1 mov_q x5, SWAPPER_RW_MMUFLAGS @@ -491,9 +491,6 @@ SYM_FUNC_START_LOCAL(__primary_switched) #if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS) bl kasan_early_init #endif - mov x0, x20 // pass the full boot status - mov x1, x22 // pass the low FDT mapping - bl __pi_init_feature_override // Parse cpu feature overrides #ifdef CONFIG_UNWIND_PATCH_PAC_INTO_SCS bl scs_patch_vmlinux #endif @@ -772,12 +769,16 @@ SYM_FUNC_START_LOCAL(__primary_switch) bl __pi_memset dsb ishst // Make zero page visible to PTW -#ifdef CONFIG_RELOCATABLE - adrp x23, KERNEL_START - and x23, x23, MIN_KIMG_ALIGN - 1 adrp x1, early_init_stack mov sp, x1 mov x29, xzr + mov x0, x20 // pass the full boot status + mov x1, x22 // pass the low FDT mapping + bl __pi_init_feature_override // Parse cpu feature overrides + +#ifdef CONFIG_RELOCATABLE + adrp x23, KERNEL_START + and x23, x23, MIN_KIMG_ALIGN - 1 #ifdef CONFIG_RANDOMIZE_BASE mov x0, x22 bl __pi_kaslr_early_init diff --git a/arch/arm64/kernel/vmlinux.lds.S b/arch/arm64/kernel/vmlinux.lds.S index 8a3c6aacc355..3afb4223a5e8 100644 --- a/arch/arm64/kernel/vmlinux.lds.S +++ b/arch/arm64/kernel/vmlinux.lds.S @@ -320,10 +320,8 @@ SECTIONS init_pg_end = .; /* end of zero-init region */ -#ifdef CONFIG_RELOCATABLE - . += SZ_4K; /* stack for the early relocation code */ + . += SZ_4K; /* stack for the early C runtime */ early_init_stack = .; -#endif . = ALIGN(SEGMENT_ALIGN); __pecoff_data_size = ABSOLUTE(. - __initdata_begin); From patchwork Wed Feb 14 12:28:55 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13556412 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 77799C48BC3 for ; Wed, 14 Feb 2024 12:30:45 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=bZcpm2vP19+bF8DNUkb7CXQYmmuBJPzS+29PTgDTl5Y=; b=q0GWdCXY72Aqn+6VW/wXTrOjKT xNzGmiHYHtg7WT2UOikmyIGWme79McHAU/LzCHV7DQKwKeRF6Z/xOkpCI8aHRf1xoIXanZkb1UTc0 HXv8nhXXM69kfx48+1AU871RxBw7l//+PzdLX1zYyC2yk8oIGDLuBXrXJELRVMhuiAmGASrIO/wYI KMOlEC8dDS0gLizDz+Zmdya209DvwZVrXld1tlwU8nVtg8Il3DVfkUbIjWdgXpoIW1SNF9jTq3pCg wZhomO83alvjA6AOltfBQxAPMvAAzycy56vyGv9O9mWyBaupQI/ySO295gCh8suj6vOi+ZbfP1im+ imp6XRCw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1raEPE-0000000Cocs-0jNR; Wed, 14 Feb 2024 12:30:28 +0000 Received: from mail-wr1-x449.google.com ([2a00:1450:4864:20::449]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1raEOp-0000000CoLF-0THl for linux-arm-kernel@lists.infradead.org; Wed, 14 Feb 2024 12:30:09 +0000 Received: by mail-wr1-x449.google.com with SMTP id ffacd0b85a97d-33cee70ef73so195284f8f.1 for ; Wed, 14 Feb 2024 04:30:00 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1707913799; x=1708518599; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=5T+JZlXorLJgdWJ3YLLbdFriHCmLYbGuJvbl7iiPg80=; b=u/4muvm/WJHlTkq0V14fUA4VPRz4+T2mZC7o1b8eI6/pfy3Jy17oSJ1GHhiWg+m6lr OI1azahohEelc0S6iFhnWfmXSCDefL8Ix0gWUnxKqW8AFxLiBJHhNl4yWLgOH9V8Q/tV GnGvD5Q0zYjuVkgiPdLWaJz6Zz4O8P5XB1UzDlrs3DpVPF9XIyds+P5657LC2yYX1Io6 +yLTTsm45ChaLYFLYyoiNkgCU2gvFpehk+52jM4OGoVUBBbg97zRYX5QOswctxeoIQ6U vJQi1SE4RTWzOq2aG0DBKyiahT2ZCiAiCe6wdlEUhzBgPsAAkjh3KOzFpU9Qgp2USzhb qxQQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707913799; x=1708518599; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=5T+JZlXorLJgdWJ3YLLbdFriHCmLYbGuJvbl7iiPg80=; b=LNjhJc1V4vCb8huaOaEbYSOcfBPUQAwBbVHRpt6Ayz1TvaEr/OxKwiz/h0GoKwK7ar QZfwtljPuzEXUii+na7EYTCGbrHhswoWOLbY/bjpZYkez4oIBmZkmSxYuRVjGzM4IqFL QoXl1QxoUDRjbjEOA52FX8zjOqPoKkQGOClAKDaWNBU/HEFy+dmO/fSRdnneMwk+dEre K42sHbfNttnmrZ9OjnFkhkf3yNIGXxeXM0Z4e8DEF31YpjD2qHuEWDVfuPXxpTbFonX8 ZFWEoGsk2+IL0ZNaxX4ZWzs02HDeq196t2lPf0NZUDH6nnYZ9B3+FRvkotGQqWxJH5fm kaww== X-Gm-Message-State: AOJu0YxrhPr5wjs37IN08CUSNvT0IuSKp4ggpCw1x1m3XDq5rDuEM6Rt Hmvb35gvIQScL6YWDmzWxuFzuCxlNtA2owxyOsaHBuN6hruCA9Bvm0wSB05HRS+C3gpyY4PXjZ4 2Gmltjpyi4KhZZ4C4j9d+2ZxIm8vYo4f0DtH7kByFTTnUB54Uc/tuprYZkEwsDj0Q5IDXonv95R BfaKBwwPXHIGdlVylewO8g5KRcDBE2xv882pGDLady X-Google-Smtp-Source: AGHT+IF6+nN0KwInZ+gksSxO019cnqoyiiIyBKUSB5BtKzETRg3zo6CY75xpkNN3HSk6hR3KlvML/M/J X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a5d:6e51:0:b0:33b:8867:9253 with SMTP id j17-20020a5d6e51000000b0033b88679253mr4592wrz.3.1707913799333; Wed, 14 Feb 2024 04:29:59 -0800 (PST) Date: Wed, 14 Feb 2024 13:28:55 +0100 In-Reply-To: <20240214122845.2033971-45-ardb+git@google.com> Mime-Version: 1.0 References: <20240214122845.2033971-45-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=6671; i=ardb@kernel.org; h=from:subject; bh=IzdxyEQm3auU1iEc65FDEPHnvsNyp60IBKqPiNiKPyM=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIfXMJo7tO093/+evKHpb8NnPdXus6HGGSGEFF5srdZ/3V rdUSjt0lLIwiHEwyIopsgjM/vtu5+mJUrXOs2Rh5rAygQxh4OIUgIl8Y2VkmMp4yLu0tpbbJqp/ xuVjoU3VNk8SdLQjtka7r+p9Ynz9OCPDg5q0H6EmemZPl06wc3p/fN9/3elvjm6ZsHbTyhVVnFW 3OQE= X-Mailer: git-send-email 2.43.0.687.g38aa6559b0-goog Message-ID: <20240214122845.2033971-54-ardb+git@google.com> Subject: [PATCH v8 09/43] arm64: head: move dynamic shadow call stack patching into early C runtime From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240214_043003_349034_541C6E10 X-CRM114-Status: GOOD ( 18.94 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel Once we update the early kernel mapping code to only map the kernel once with the right permissions, we can no longer perform code patching via this mapping. So move this code to an earlier stage of the boot, right after applying the relocations. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/scs.h | 4 +-- arch/arm64/kernel/Makefile | 8 ------ arch/arm64/kernel/head.S | 8 +++--- arch/arm64/kernel/module.c | 2 +- arch/arm64/kernel/pi/Makefile | 10 +++++--- arch/arm64/kernel/{ => pi}/patch-scs.c | 26 ++++++++++---------- 6 files changed, 27 insertions(+), 31 deletions(-) diff --git a/arch/arm64/include/asm/scs.h b/arch/arm64/include/asm/scs.h index 3fdae5fe3142..eca2ba5a6276 100644 --- a/arch/arm64/include/asm/scs.h +++ b/arch/arm64/include/asm/scs.h @@ -72,8 +72,8 @@ static inline void dynamic_scs_init(void) static inline void dynamic_scs_init(void) {} #endif -int scs_patch(const u8 eh_frame[], int size); -asmlinkage void scs_patch_vmlinux(void); +int __pi_scs_patch(const u8 eh_frame[], int size); +asmlinkage void __pi_scs_patch_vmlinux(void); #endif /* __ASSEMBLY __ */ diff --git a/arch/arm64/kernel/Makefile b/arch/arm64/kernel/Makefile index 4236f1e0fffa..14b4a179bad3 100644 --- a/arch/arm64/kernel/Makefile +++ b/arch/arm64/kernel/Makefile @@ -71,14 +71,6 @@ obj-$(CONFIG_ARM64_PTR_AUTH) += pointer_auth.o obj-$(CONFIG_ARM64_MTE) += mte.o obj-y += vdso-wrap.o obj-$(CONFIG_COMPAT_VDSO) += vdso32-wrap.o -obj-$(CONFIG_UNWIND_PATCH_PAC_INTO_SCS) += patch-scs.o - -# We need to prevent the SCS patching code from patching itself. Using -# -mbranch-protection=none here to avoid the patchable PAC opcodes from being -# generated triggers an issue with full LTO on Clang, which stops emitting PAC -# instructions altogether. So disable LTO as well for the compilation unit. -CFLAGS_patch-scs.o += -mbranch-protection=none -CFLAGS_REMOVE_patch-scs.o += $(CC_FLAGS_LTO) # Force dependency (vdso*-wrap.S includes vdso.so through incbin) $(obj)/vdso-wrap.o: $(obj)/vdso/vdso.so diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S index 865ecc1f8255..b320702032a7 100644 --- a/arch/arm64/kernel/head.S +++ b/arch/arm64/kernel/head.S @@ -490,9 +490,6 @@ SYM_FUNC_START_LOCAL(__primary_switched) #endif #if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS) bl kasan_early_init -#endif -#ifdef CONFIG_UNWIND_PATCH_PAC_INTO_SCS - bl scs_patch_vmlinux #endif mov x0, x20 bl finalise_el2 // Prefer VHE if possible @@ -794,6 +791,11 @@ SYM_FUNC_START_LOCAL(__primary_switch) #ifdef CONFIG_RELOCATABLE mov x0, x23 bl __pi_relocate_kernel +#endif +#ifdef CONFIG_UNWIND_PATCH_PAC_INTO_SCS + ldr x0, =__eh_frame_start + ldr x1, =__eh_frame_end + bl __pi_scs_patch_vmlinux #endif ldr x8, =__primary_switched adrp x0, KERNEL_START // __pa(KERNEL_START) diff --git a/arch/arm64/kernel/module.c b/arch/arm64/kernel/module.c index dd851297596e..47e0be610bb6 100644 --- a/arch/arm64/kernel/module.c +++ b/arch/arm64/kernel/module.c @@ -595,7 +595,7 @@ int module_finalize(const Elf_Ehdr *hdr, if (scs_is_dynamic()) { s = find_section(hdr, sechdrs, ".init.eh_frame"); if (s) - scs_patch((void *)s->sh_addr, s->sh_size); + __pi_scs_patch((void *)s->sh_addr, s->sh_size); } return module_init_ftrace_plt(hdr, sechdrs, me); diff --git a/arch/arm64/kernel/pi/Makefile b/arch/arm64/kernel/pi/Makefile index 7f6dfce893c3..a8b302245f15 100644 --- a/arch/arm64/kernel/pi/Makefile +++ b/arch/arm64/kernel/pi/Makefile @@ -38,7 +38,9 @@ $(obj)/lib-%.pi.o: OBJCOPYFLAGS += --prefix-alloc-sections=.init $(obj)/lib-%.o: $(srctree)/lib/%.c FORCE $(call if_changed_rule,cc_o_c) -obj-y := idreg-override.pi.o lib-fdt.pi.o lib-fdt_ro.pi.o -obj-$(CONFIG_RELOCATABLE) += relocate.pi.o -obj-$(CONFIG_RANDOMIZE_BASE) += kaslr_early.pi.o -extra-y := $(patsubst %.pi.o,%.o,$(obj-y)) +obj-y := idreg-override.pi.o \ + lib-fdt.pi.o lib-fdt_ro.pi.o +obj-$(CONFIG_RELOCATABLE) += relocate.pi.o +obj-$(CONFIG_RANDOMIZE_BASE) += kaslr_early.pi.o +obj-$(CONFIG_UNWIND_PATCH_PAC_INTO_SCS) += patch-scs.pi.o +extra-y := $(patsubst %.pi.o,%.o,$(obj-y)) diff --git a/arch/arm64/kernel/patch-scs.c b/arch/arm64/kernel/pi/patch-scs.c similarity index 91% rename from arch/arm64/kernel/patch-scs.c rename to arch/arm64/kernel/pi/patch-scs.c index a1fe4b4ff591..c65ef40d1e6b 100644 --- a/arch/arm64/kernel/patch-scs.c +++ b/arch/arm64/kernel/pi/patch-scs.c @@ -4,14 +4,11 @@ * Author: Ard Biesheuvel */ -#include #include #include #include -#include #include -#include #include // @@ -81,7 +78,11 @@ static void __always_inline scs_patch_loc(u64 loc) */ return; } - dcache_clean_pou(loc, loc + sizeof(u32)); + if (IS_ENABLED(CONFIG_ARM64_WORKAROUND_CLEAN_CACHE)) + asm("dc civac, %0" :: "r"(loc)); + else + asm(ALTERNATIVE("dc cvau, %0", "nop", ARM64_HAS_CACHE_IDC) + :: "r"(loc)); } /* @@ -128,10 +129,10 @@ struct eh_frame { }; }; -static int noinstr scs_handle_fde_frame(const struct eh_frame *frame, - bool fde_has_augmentation_data, - int code_alignment_factor, - bool dry_run) +static int scs_handle_fde_frame(const struct eh_frame *frame, + bool fde_has_augmentation_data, + int code_alignment_factor, + bool dry_run) { int size = frame->size - offsetof(struct eh_frame, opcodes) + 4; u64 loc = (u64)offset_to_ptr(&frame->initial_loc); @@ -198,14 +199,13 @@ static int noinstr scs_handle_fde_frame(const struct eh_frame *frame, break; default: - pr_err("unhandled opcode: %02x in FDE frame %lx\n", opcode[-1], (uintptr_t)frame); return -ENOEXEC; } } return 0; } -int noinstr scs_patch(const u8 eh_frame[], int size) +int scs_patch(const u8 eh_frame[], int size) { const u8 *p = eh_frame; @@ -251,12 +251,12 @@ int noinstr scs_patch(const u8 eh_frame[], int size) return 0; } -asmlinkage void __init scs_patch_vmlinux(void) +asmlinkage void __init scs_patch_vmlinux(const u8 start[], const u8 end[]) { if (!should_patch_pac_into_scs()) return; - WARN_ON(scs_patch(__eh_frame_start, __eh_frame_end - __eh_frame_start)); - icache_inval_all_pou(); + scs_patch(start, end - start); + asm("ic ialluis"); isb(); } From patchwork Wed Feb 14 12:28:56 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13556413 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 5C33CC48BC1 for ; Wed, 14 Feb 2024 12:30:46 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=S9vazj975O1MiVa3+5rFhIoV8m92gcLVNRCC4gEgawk=; b=OLaCS9KNN4cSLdlsu0g4s1rsnh GrtgDv7+TPixI3oC6THQighUyKXu1+MYVN97GDhO3auQ82gD3MCOVdsqjneOMkk+mnA+kb/+qb3F6 WpExYzP8rGLx3niYFc3iuWKifsEI8c9LW5ci8Dt4b0Rkoa//udNdG297pRvDr/XeMo4B6wij1sVTz eSRaepAqk9Qdce+c69CWv2+F/ECqBOvkJSlhAczFo8e8zOzeSDC4tfYDClugGCCahf0feqkPzTLqg /Y5V/u6oFOXsveNCmHIJGUXnc8P+j926DZ6vCDYCxgxsz1fSzuUQXWxbROnlwE2ZzRr4HtIVwOveY VDaQ6bjA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1raEPG-0000000Cof2-40Gu; Wed, 14 Feb 2024 12:30:30 +0000 Received: from mail-wm1-x349.google.com ([2a00:1450:4864:20::349]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1raEOp-0000000CoMe-2Ztr for linux-arm-kernel@lists.infradead.org; Wed, 14 Feb 2024 12:30:09 +0000 Received: by mail-wm1-x349.google.com with SMTP id 5b1f17b1804b1-411a8c8d9b2so13129625e9.0 for ; Wed, 14 Feb 2024 04:30:02 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1707913801; x=1708518601; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=C9PSY9iBLTlxzvPzjgB+0iHAKBZWgvNknFblEmIV87Y=; b=njT+TAHEwPBEhGwkgniOoqM5YbifUvHMzYCHQLZkjPOcNIVGsaRrxAIsIEiOwjzmTu 09W+jiX6ObVnjXqzkY2CD8CTOoYdq/tmqKt56AckCDixndHXISG01E0whOvEgiGbFZyI ps5bBevc5KJpPvqbRC2Z3xbJ6l/lU2sp5Ux7n3elRs71SwgGq4dqq9oouaJhlX9PP0YF vFOJdCHQhN+XxgQyXmBLZNr/HmZTv91ioKUlVpMhQdWqhWcS3x6S8eqjLxbs9/X+SiNQ cjNJThErxslVStZcRlct+U9Gej6BJj8syPmEPfI/dQeEYv+i2oVFFxclARlg5Mf/mE03 4exw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707913801; x=1708518601; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=C9PSY9iBLTlxzvPzjgB+0iHAKBZWgvNknFblEmIV87Y=; b=Rz1UygdyK7Aziex3a9HLtdbE6+JY+CWnPanPgKsJJ73eXwBQ3zPl2Q5O0XHU/D35Qw 8T+KQcN901ASVTwGg5M6QdvGcVywggKlfYm1ujqOLDbtBBrORY53IV8NuvbVT68dKk73 r1j4n6Ik+qtEHmjAXU/2sDKJ7r5Kf+Q3220839ehbyg2d2WHXeStjqf75iYg/WiV/8+r t0koBCmIOkYSnm2g2ZMcLG7/UE41xMR8GNsFfpkEMZAcctQta24CX1RKcKhAy3dO3Mf0 eDooqkxGHs1IObE/ikMocePr9vPSI/qJ2jSzJGDkIw0+4DsmYP9jxJO4UjwKBxgtdPx6 tvjQ== X-Gm-Message-State: AOJu0Yz7LOHP6vdB1u2O5sEBg01G1IEt5kUSwVy2Ws/hPNXwj5v458WS dkDsNmZcH9F+A6aPw7jBQaNCA0bfizl4Xk/hXrrVRWg2nFQJ3syvd97kLvDmB6bmRerDx1O4CeD 3kZkv4oE/YJD43D6ffyAFfqcOLkaFktH9MO5iylcVd/YwtT64B5zrJu5nKKYX7suJtdhIKWotRG SNCBlzZUIvxZXkrizuDOQ/KHgSECqTNySe/KZ/euEu X-Google-Smtp-Source: AGHT+IFEBPng4lnEtDmoR++NAyVLnTisbLTDcGyROmyMsDUAtddG4/xEAQrupjYM5reTKEQcxllW8SIM X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:600c:a385:b0:411:ebf7:d0ff with SMTP id hn5-20020a05600ca38500b00411ebf7d0ffmr4767wmb.5.1707913801443; Wed, 14 Feb 2024 04:30:01 -0800 (PST) Date: Wed, 14 Feb 2024 13:28:56 +0100 In-Reply-To: <20240214122845.2033971-45-ardb+git@google.com> Mime-Version: 1.0 References: <20240214122845.2033971-45-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=3269; i=ardb@kernel.org; h=from:subject; bh=NcyALRdi0AS0TLQf1binGGCHuDJV0cdvb+ltyDJJPRg=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIfXMJs4/HvfS5zRx/ghu3vix4H1HdLNblcpXnh1950Lq1 yyX8jzWUcrCIMbBICumyCIw+++7nacnStU6z5KFmcPKBDKEgYtTACby2oHhf8XLR3bx0WafXQK4 ubQZ1ddu06hRPRcZ87+wOkLu70pRQ4a/8nvOK5gsiDTVTXVI33xHpPPvHw2X4+r+JxPZ9q9q8F3 JDgA= X-Mailer: git-send-email 2.43.0.687.g38aa6559b0-goog Message-ID: <20240214122845.2033971-55-ardb+git@google.com> Subject: [PATCH v8 10/43] arm64: cpufeature: Add helper to test for CPU feature overrides From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240214_043003_926870_0D44DD81 X-CRM114-Status: GOOD ( 18.71 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel Add some helpers to extract and apply feature overrides to the bare idreg values. This involves inspecting the value and mask of the specific field that we are interested in, given that an override value/mask pair might be invalid for one field but valid for another. Then, wire up the new helper for the hVHE test - note that we can drop the sysreg test here, as the override will be invalid when trying to enable hVHE on non-VHE capable hardware. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/cpufeature.h | 39 ++++++++++++++++++++ arch/arm64/kernel/cpufeature.c | 9 +---- 2 files changed, 40 insertions(+), 8 deletions(-) diff --git a/arch/arm64/include/asm/cpufeature.h b/arch/arm64/include/asm/cpufeature.h index 21c824edf8ce..acd8f4949583 100644 --- a/arch/arm64/include/asm/cpufeature.h +++ b/arch/arm64/include/asm/cpufeature.h @@ -915,6 +915,45 @@ extern struct arm64_ftr_override id_aa64isar2_override; extern struct arm64_ftr_override arm64_sw_feature_override; +static inline +u64 arm64_apply_feature_override(u64 val, int feat, int width, + const struct arm64_ftr_override *override) +{ + u64 oval = override->val; + + /* + * When it encounters an invalid override (e.g., an override that + * cannot be honoured due to a missing CPU feature), the early idreg + * override code will set the mask to 0x0 and the value to non-zero for + * the field in question. In order to determine whether the override is + * valid or not for the field we are interested in, we first need to + * disregard bits belonging to other fields. + */ + oval &= GENMASK_ULL(feat + width - 1, feat); + + /* + * The override is valid if all value bits are accounted for in the + * mask. If so, replace the masked bits with the override value. + */ + if (oval == (oval & override->mask)) { + val &= ~override->mask; + val |= oval; + } + + /* Extract the field from the updated value */ + return cpuid_feature_extract_unsigned_field(val, feat); +} + +static inline bool arm64_test_sw_feature_override(int feat) +{ + /* + * Software features are pseudo CPU features that have no underlying + * CPUID system register value to apply the override to. + */ + return arm64_apply_feature_override(0, feat, 4, + &arm64_sw_feature_override); +} + u32 get_kvm_ipa_limit(void); void dump_cpu_features(void); diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c index a99ad79adee2..d0ffb872a31a 100644 --- a/arch/arm64/kernel/cpufeature.c +++ b/arch/arm64/kernel/cpufeature.c @@ -2042,14 +2042,7 @@ static bool has_nested_virt_support(const struct arm64_cpu_capabilities *cap, static bool hvhe_possible(const struct arm64_cpu_capabilities *entry, int __unused) { - u64 val; - - val = read_sysreg(id_aa64mmfr1_el1); - if (!cpuid_feature_extract_unsigned_field(val, ID_AA64MMFR1_EL1_VH_SHIFT)) - return false; - - val = arm64_sw_feature_override.val & arm64_sw_feature_override.mask; - return cpuid_feature_extract_unsigned_field(val, ARM64_SW_FEATURE_OVERRIDE_HVHE); + return arm64_test_sw_feature_override(ARM64_SW_FEATURE_OVERRIDE_HVHE); } #ifdef CONFIG_ARM64_PAN From patchwork Wed Feb 14 12:28:57 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13556538 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id C5895C48BC1 for ; Wed, 14 Feb 2024 13:30:54 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=yD1k/nsFevbPvKlxSZu6WGaXyZG8FF0cAGLPXlxa2sY=; b=fc65EbjIn+gbbfCrM/X0NyxwAf yT1sYee4++jN0ulMXVsqxLrFc17GTTQtLHoNTgaYg08EZDeuyf3RBa/8BILx2qPCbSzcjTQ7Nf1DY OdzDLkE7mxtOd1bkhvHQdOO/IKaxGANcaf9Bt2WEgAP74krxvBQXI7vsotN5k6Sl5tXaE5mRGQ7BO SNoAPryI+c+LzCBJCrVFWh26teyaZUrKS6Y5ycSSZP+LMtvlYXDIyhhjDfk+omz96JpZrm1fuYYCw TXa5zPKhHulEZ+e8RIBLtX/PbIuPAB1tOyKeN00FIb+ZOOPdUQzWJ8Dtzv9p1JZdX+Hikqok3RK+8 0hmJNBCg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1raFLZ-0000000D1kt-3iyx; Wed, 14 Feb 2024 13:30:45 +0000 Received: from mail-yw1-x1149.google.com ([2607:f8b0:4864:20::1149]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1raEOs-0000000CoON-3IBC for linux-arm-kernel@lists.infradead.org; Wed, 14 Feb 2024 12:30:11 +0000 Received: by mail-yw1-x1149.google.com with SMTP id 00721157ae682-607ac7dbd61so5952417b3.3 for ; Wed, 14 Feb 2024 04:30:04 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1707913804; x=1708518604; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=VU4Qhmc1/wiyAnLqhp7NyhVRjUKG6HLUYDxCsDuDND8=; b=yzm/eOanzvhEJu6bT8Fr300UDrb5oH51mW/UPvo7f9m/5sl3uULCMOxH9Y8Gi835a7 bK7WxM9rQMw7k0tbhohXrLhNxO2zO+v76oeXRRwb3bbEh7OqEhOLpr4+5y3g4fC2pLZO urIUv/TXqm8MUq44jXufH00Zek4e1fAhCXjzAWB/cWgB954WvysAhHo7UXL9Eanero0c gtSKy9sjG9XBYMVRW7fMF/npCAH4Dh2Gvl0h4Vw0+2H5VP6luY2X0A+xKR5tJrCt2YtR J6Zr2SZJzb03ElJGo2VcXpOf4Rx0rkGSQmrVYr6elfreNSLdwCybM65e3s2x0gnjFZ4+ EIiw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707913804; x=1708518604; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=VU4Qhmc1/wiyAnLqhp7NyhVRjUKG6HLUYDxCsDuDND8=; b=Mh6hVsPm8VWYZtA/nq9r6v80k5Dl7s1xoorDl+TNGEhEgbn5ui6IIlv7I7Zq0eb/+z ZC5tvRQEQEw4N+RPecqyctBtEnXSJbtLmAxr6UsMTT52ZMR+eWwYrCbdwVRd47Xo6roj QhashvB7NTrcPb9gP3iVvmSmSCnl1M0eJG+ff7fPVQX1sO4SkUGva1rHRmtsFx+8JkQN aeaKTomdaFTDh7Ig2Xwcp5XF14t3qz5bUZmA7vlXLd+zOXGwGeh/SM3Oz42Pfld3K1qc RV3iWiN6HYFiX4q4zq3EZM1LVOFOK9thnFcUHlsnfAShBFy/2JzwgrFLeGQIjy220czt M/9A== X-Gm-Message-State: AOJu0YyJ9IlZZv+i7N0yixH9tu5N1lg1PcMlDLjgQTWie5AdfBr94et8 L+oA/RblVy6cf04bsiqcjX8ha/hCvBD1CGrsQjumAIc4seXECBc85OcUd5T/qhhhNL/twwxNWCv mzlVwZtjOjHmpgqQm58ug4hawipA1lobqSK+QjgRKshTMx/qukYpZ2taC7D8p6pi4vynDouAWVQ QUo3xwP+9gEjgbqr8Z6icP+r8OGSwiO8jVVS1ox2Oi X-Google-Smtp-Source: AGHT+IFchcpbZnr9KJQENau2sH/H4wv2Xn6UmZA5R/AdET8Q6CZV2C6Nw3NY4PNIPa9iEc8gYTTWCsaA X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a0d:ea4b:0:b0:604:d53e:4616 with SMTP id t72-20020a0dea4b000000b00604d53e4616mr374033ywe.6.1707913803679; Wed, 14 Feb 2024 04:30:03 -0800 (PST) Date: Wed, 14 Feb 2024 13:28:57 +0100 In-Reply-To: <20240214122845.2033971-45-ardb+git@google.com> Mime-Version: 1.0 References: <20240214122845.2033971-45-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=3267; i=ardb@kernel.org; h=from:subject; bh=Vk2Gqq1aehL+kNtGFfIi4QCIk4Fya8Bx7QIQOwpmnbU=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIfXMJi7JxJkq1S+OlGqfOWUp/K/+CJOF/8x7U7eX2t76u 22+ySONjlIWBjEOBlkxRRaB2X/f7Tw9UarWeZYszBxWJpAhDFycAjARHllGho7/Qu+WvDsaFvzx q/Ypq6n3oxUv1/znbEv6oh+9LnjNCWNGhq3fZBeXHjNnrV2m076tjac2uW5Rk/0ar6eaeWL1mmn N7AA= X-Mailer: git-send-email 2.43.0.687.g38aa6559b0-goog Message-ID: <20240214122845.2033971-56-ardb+git@google.com> Subject: [PATCH v8 11/43] arm64: kaslr: Use feature override instead of parsing the cmdline again From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240214_043007_090911_5E367780 X-CRM114-Status: GOOD ( 15.66 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel The early kaslr code open codes the detection of 'nokaslr' on the kernel command line, and this is no longer necessary now that the feature detection code, which also looks for the same string, executes before this code. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/cpufeature.h | 5 ++ arch/arm64/kernel/kaslr.c | 4 +- arch/arm64/kernel/pi/kaslr_early.c | 53 +------------------- 3 files changed, 7 insertions(+), 55 deletions(-) diff --git a/arch/arm64/include/asm/cpufeature.h b/arch/arm64/include/asm/cpufeature.h index acd8f4949583..e309255b7f04 100644 --- a/arch/arm64/include/asm/cpufeature.h +++ b/arch/arm64/include/asm/cpufeature.h @@ -954,6 +954,11 @@ static inline bool arm64_test_sw_feature_override(int feat) &arm64_sw_feature_override); } +static inline bool kaslr_disabled_cmdline(void) +{ + return arm64_test_sw_feature_override(ARM64_SW_FEATURE_OVERRIDE_NOKASLR); +} + u32 get_kvm_ipa_limit(void); void dump_cpu_features(void); diff --git a/arch/arm64/kernel/kaslr.c b/arch/arm64/kernel/kaslr.c index 12c7f3c8ba76..1da3e25f9d9e 100644 --- a/arch/arm64/kernel/kaslr.c +++ b/arch/arm64/kernel/kaslr.c @@ -16,9 +16,7 @@ bool __ro_after_init __kaslr_is_enabled = false; void __init kaslr_init(void) { - if (cpuid_feature_extract_unsigned_field(arm64_sw_feature_override.val & - arm64_sw_feature_override.mask, - ARM64_SW_FEATURE_OVERRIDE_NOKASLR)) { + if (kaslr_disabled_cmdline()) { pr_info("KASLR disabled on command line\n"); return; } diff --git a/arch/arm64/kernel/pi/kaslr_early.c b/arch/arm64/kernel/pi/kaslr_early.c index 167081b30a15..f2305e276ec3 100644 --- a/arch/arm64/kernel/pi/kaslr_early.c +++ b/arch/arm64/kernel/pi/kaslr_early.c @@ -16,57 +16,6 @@ #include #include -/* taken from lib/string.c */ -static char *__init __strstr(const char *s1, const char *s2) -{ - size_t l1, l2; - - l2 = strlen(s2); - if (!l2) - return (char *)s1; - l1 = strlen(s1); - while (l1 >= l2) { - l1--; - if (!memcmp(s1, s2, l2)) - return (char *)s1; - s1++; - } - return NULL; -} -static bool __init cmdline_contains_nokaslr(const u8 *cmdline) -{ - const u8 *str; - - str = __strstr(cmdline, "nokaslr"); - return str == cmdline || (str > cmdline && *(str - 1) == ' '); -} - -static bool __init is_kaslr_disabled_cmdline(void *fdt) -{ - if (!IS_ENABLED(CONFIG_CMDLINE_FORCE)) { - int node; - const u8 *prop; - - node = fdt_path_offset(fdt, "/chosen"); - if (node < 0) - goto out; - - prop = fdt_getprop(fdt, node, "bootargs", NULL); - if (!prop) - goto out; - - if (cmdline_contains_nokaslr(prop)) - return true; - - if (IS_ENABLED(CONFIG_CMDLINE_EXTEND)) - goto out; - - return false; - } -out: - return cmdline_contains_nokaslr(CONFIG_CMDLINE); -} - static u64 __init get_kaslr_seed(void *fdt) { static char const chosen_str[] __initconst = "chosen"; @@ -92,7 +41,7 @@ asmlinkage u64 __init kaslr_early_init(void *fdt) { u64 seed, range; - if (is_kaslr_disabled_cmdline(fdt)) + if (kaslr_disabled_cmdline()) return 0; seed = get_kaslr_seed(fdt); From patchwork Wed Feb 14 12:28:58 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13556541 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id A7503C48BED for ; Wed, 14 Feb 2024 13:30:57 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=3Ya46n3I+9bbknHm8/nABdlAn7IavufdZIt3i2CYrxQ=; b=aV2lBrLH+DJVaniFRfBzxKKb4P bLX4zqwERmzrVSf28Lca/80y5vrEGvkagnn368eFFlLzznwLr1BlFUtFFPrqc4C+HmwjrbsZk1VHe uC4KJH0sbaZV8GZpljxdyQ5UNiduRAZnmvqqUeEUl/bcfIPXx9HD7MJ3s/EpsV0tQ9fy0pD9dYPBT zbcywcC3Rdhsdgn/hHotaxwsUUaSkFK0JQpsrD4JAsIKu+ZIogxwXjmYc+q04aSBlps4XLvkStvTM FySg2hXblEH3jsycGbu62SQQ3AsY8dHAdO5hQSSF33I9/Vb/gr9JwQjogw68XqcEh+A+JKcSvUXTp p7M7MiNQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1raFLa-0000000D1lc-2Us3; Wed, 14 Feb 2024 13:30:46 +0000 Received: from mail-wm1-x34a.google.com ([2a00:1450:4864:20::34a]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1raEOt-0000000CoPK-41sd for linux-arm-kernel@lists.infradead.org; Wed, 14 Feb 2024 12:30:12 +0000 Received: by mail-wm1-x34a.google.com with SMTP id 5b1f17b1804b1-40fd153b5c3so3275395e9.1 for ; Wed, 14 Feb 2024 04:30:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1707913806; x=1708518606; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=GHienn26NLkDg8GQxTAgXIDcX5IvttP8vOHFNDCy9GM=; b=ORO/YbJXFmHF+M40do/CbXNPYUvQiYfsKfHSxKcgpJoTFkHpUsAmXxGeJWEeSor5+d FwiL51r9rQlNWiPp3Vgu0N9asExyThelclEP/3zd0iPlCi1MFh16m/xFJmP8tISMj9JD +txPYa7jpCkJUJHmSvzpGC/ss9GHhW5TTjKbViPlcGWpe62pUWYGitFj0f8eeEYG1OwV w3U3DOuNQ+o9IjpAkFHBHAXCJmbHG0qUhg1Mmh+/ROnXBrtXLHTmhwBSchFzlhjRCYm9 3VV7RV/cqLhoUJ9moTt2YM68GnDa6HViovF/Bl9WGSa3z87QINlSccWNyTLGyF0VDsF7 qj2g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707913806; x=1708518606; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=GHienn26NLkDg8GQxTAgXIDcX5IvttP8vOHFNDCy9GM=; b=KJCzKxMlHAQn+bDc9Uqx2PIG0TxdvIUoIhu3wm/arYBHDrybum3rasxulkOKKlEBCJ hSQrOddZNfkSv+DYepJ1JTsdKwv07qCv0Ctb2ILivwUmGFCC2oFo2yXtWZSQp5PeFGKT asNrzgTOl4jjHrTBXTt/kRRjmyzg4Vt0vS3s5F68J0Z+0gP408bGC4NFxAXoVNAE8j1b DAlZXMmQ58AgfvLXaRNOkgUdknop8Di3I0kUx3rQ0K5QzxnlQbm4M72ObF3IWSbZm2oB l7ynLZm9dn7e36QxUCZCYFbnpLZdnA+zq9SAcKs7AhBd2Ouj0zMYIIC8d7DSY+vkiJ4h K28Q== X-Gm-Message-State: AOJu0Yxlbj7bT9EtpiJISow8JV8WiYecR/uKdsB2l5z7buWPhfIr3UXu BFDOax4SJIwMIjRijyT5b8ceV0WsBBPNXMXcpc4jwje/FZdMlGc+R/ZwMmAWBXF5EWJMy4bJJbb ok6tDOgkMmlqB/+RYmjRJ6IchpH/8xTqwpRUrF94NAXQvfzf0uHG8lFzS+tSW64yu6iqbz2KjJm itfX9o3g1c3kHNO/hx0s6O/H06EXbsIBtzI0cDNNjE X-Google-Smtp-Source: AGHT+IGzja8vYqa3pb+GDOKNV31TQPqv/BXaWaumHWVSi8COK8lhOMvS8nxPSECapMhTDeoalBiO/D1a X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:600c:3d91:b0:411:e008:d86c with SMTP id bi17-20020a05600c3d9100b00411e008d86cmr46899wmb.4.1707913805862; Wed, 14 Feb 2024 04:30:05 -0800 (PST) Date: Wed, 14 Feb 2024 13:28:58 +0100 In-Reply-To: <20240214122845.2033971-45-ardb+git@google.com> Mime-Version: 1.0 References: <20240214122845.2033971-45-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=1646; i=ardb@kernel.org; h=from:subject; bh=7NJV+O5Gti5YrDiC4EJgS3VtX3sXg64sBlUrZ/2tWI8=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIfXMJu5jl7ddO/qSc++R8FcME1ymHEvn7th+RSBvYXZcd Zr/Sw7HjlIWBjEOBlkxRRaB2X/f7Tw9UarWeZYszBxWJpAhDFycAjCRa4IM/+O+PdNbYn5tQ9u5 ufxWnw/ffjK3uodp5lqOjbrHppZ9i73C8Fdo4+VnlwrL386sKv46W0i5vMdC739Xb0v6/MW9zho 5N7kA X-Mailer: git-send-email 2.43.0.687.g38aa6559b0-goog Message-ID: <20240214122845.2033971-57-ardb+git@google.com> Subject: [PATCH v8 12/43] arm64: idreg-override: Create a pseudo feature for rodata=off From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240214_043008_235130_8E7295EE X-CRM114-Status: GOOD ( 13.12 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel Add rodata=off to the set of kernel command line options that is parsed early using the CPU feature override detection code, so we can easily refer to it when creating the kernel mapping. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/cpufeature.h | 1 + arch/arm64/kernel/pi/idreg-override.c | 2 ++ 2 files changed, 3 insertions(+) diff --git a/arch/arm64/include/asm/cpufeature.h b/arch/arm64/include/asm/cpufeature.h index e309255b7f04..03c34242bfc7 100644 --- a/arch/arm64/include/asm/cpufeature.h +++ b/arch/arm64/include/asm/cpufeature.h @@ -17,6 +17,7 @@ #define ARM64_SW_FEATURE_OVERRIDE_NOKASLR 0 #define ARM64_SW_FEATURE_OVERRIDE_HVHE 4 +#define ARM64_SW_FEATURE_OVERRIDE_RODATA_OFF 8 #ifndef __ASSEMBLY__ diff --git a/arch/arm64/kernel/pi/idreg-override.c b/arch/arm64/kernel/pi/idreg-override.c index f9e05c10faab..e4bcabcc6860 100644 --- a/arch/arm64/kernel/pi/idreg-override.c +++ b/arch/arm64/kernel/pi/idreg-override.c @@ -159,6 +159,7 @@ static const struct ftr_set_desc sw_features __prel64_initconst = { .fields = { FIELD("nokaslr", ARM64_SW_FEATURE_OVERRIDE_NOKASLR, NULL), FIELD("hvhe", ARM64_SW_FEATURE_OVERRIDE_HVHE, hvhe_filter), + FIELD("rodataoff", ARM64_SW_FEATURE_OVERRIDE_RODATA_OFF, NULL), {} }, }; @@ -190,6 +191,7 @@ static const struct { { "arm64.nomops", "id_aa64isar2.mops=0" }, { "arm64.nomte", "id_aa64pfr1.mte=0" }, { "nokaslr", "arm64_sw.nokaslr=1" }, + { "rodata=off", "arm64_sw.rodataoff=1" }, }; static int __init parse_hexdigit(const char *p, u64 *v) From patchwork Wed Feb 14 12:28:59 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13556540 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id AA4C2C48BEE for ; Wed, 14 Feb 2024 13:30:57 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=bFTsK6qs13KIvoKErUQ2aggD3PatFBclPSCOlLwUS7Q=; b=0KIXogTd97HcGRbAOzagdKjLkt Vgt5zB3Yrjctpp7je6kJUprKV+M2bMGrwnEAU/Ba1JlU3TY+8UW2G+QNZ5tMYGg1jxrBgcd/krsS+ 80Up/eu/QW0nW0eFd36zefd916jO1f6rcOVFDIoi4lVlNWLYoogf3Lbm7qe71rwUvJR0BiXMeyA3z iNoae5lWCT+5wazKg35r74eH3sXGep79kvpCbuptX0dNRcayNNSNcLM7tOv2qqSzE4ls8Td0w9ngd iAG0STg+Mvhq59LrhP4FRQfDAOvSEguSL6fnNP6EbPn4z+3CQytPQeWfFzK4h1WB2ynFkXNUia6MP 8DRlrmvQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1raFLb-0000000D1lw-0m9U; Wed, 14 Feb 2024 13:30:47 +0000 Received: from mail-yw1-x1149.google.com ([2607:f8b0:4864:20::1149]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1raEOv-0000000CoQy-3cFE for linux-arm-kernel@lists.infradead.org; Wed, 14 Feb 2024 12:30:13 +0000 Received: by mail-yw1-x1149.google.com with SMTP id 00721157ae682-607ac7dbd61so5953437b3.3 for ; Wed, 14 Feb 2024 04:30:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1707913808; x=1708518608; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=wQr7EbRosa58dp4pneKL9GcaOGkXWnBFM1NcrTS6enM=; b=PcaTWhwnRjPuRnmKOSzlb9L37W/lZRsEPqPhF/icfq1SXJiJrnn1rE7kRedZzoX5yg iByO/ndXmL0Nh3tRXnFSK3meAOFtfij+Ahsx5ecHaS7sVWgGoxQlVIy0UA2lmcc17Lli yPEWIHzWVTKNK0EkpczxXcq6fE/emjX/SsfWt6XyCsZme/FBRkNd+78Ns2AHmhE9uCpg ns2LGEDk7wEnB6Bb8YbHfJTkxC0LrF65HBccuMrNjV9l7NYk9zNsJHd70mU034BJwvV/ h9uu8N7y0I69SQN8wInsD3li7GieBiP5DDxkDes+TgrzJ40a7fxJV6G+gPXb4x1ByYdr 8neg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707913808; x=1708518608; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=wQr7EbRosa58dp4pneKL9GcaOGkXWnBFM1NcrTS6enM=; b=AlJA1WORCc3mguqbmgak4RMnH3eIQrSltHZnTXqg7Vu655So2tpiJ4AGZzc9kLi2Q5 MwLMa+Q8Wf26fD5jBrU4QgeoPeNHGsyxmgFJSSLwPAa9aK6Oer0e/O401pEQtQ5GqVpC gw7xVjgVGKVab1XNhhxefczXpV/CvZlBBsrLxfEmH8d4mv7HZMZ4mciiQkEcS20+e5VP eLsinN2R+yJpwid77AF+/a/+Nt5h/Xwb+QwzJBIj5HWF5LtH5bh2gf3VUFP44bsPn5Eu caiUsv6Waa/jkzDoFrSZZBGrKt6GtafQzn0zsKBFhB+wzHTd/xC+iH5AXjaBO/5U9pfz Dzdw== X-Gm-Message-State: AOJu0YxzRgcgItm+k471RNNojeWlD5Qcczxm8E17lAzDvwOtEpBhwVxw Axf7ED9EULK+qghabvLJ3EnlbZ6t8dkA7vtl0m2PZ4DaiA9Qcqh9efoQkdI7M9pIYNPUcxcSVpi MJJGp2TsZLnkSZnlsZvpCP8x5z5MqixKzDCnXBezY7JJkzw33AprtrocSu7y68hP6o+jjGtyr0B WShQTIXUDVtSdvDs9jCYxusl0tKGxDaYTsQW+IA+np X-Google-Smtp-Source: AGHT+IHfQS9Iqrj2kyhGmJ5mSO7uOD4TZT+0YyPOijnEzex6HZABST7GHhZ0ARcvxWhQKZMEYk9SvpeP X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:6902:4b0:b0:dc7:865b:22c6 with SMTP id r16-20020a05690204b000b00dc7865b22c6mr66609ybs.8.1707913808067; Wed, 14 Feb 2024 04:30:08 -0800 (PST) Date: Wed, 14 Feb 2024 13:28:59 +0100 In-Reply-To: <20240214122845.2033971-45-ardb+git@google.com> Mime-Version: 1.0 References: <20240214122845.2033971-45-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=1676; i=ardb@kernel.org; h=from:subject; bh=5qdPvoVrtwaUhGBvnLf1VQ9UAGDdk5pMw1SfLLPmHgI=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIfXMJp6+5TH/UzZ9mV+nZ5JU2vX0/7Gpn+Qlj/qsz7psv K5R4f29jlIWBjEOBlkxRRaB2X/f7Tw9UarWeZYszBxWJpAhDFycAjCR61GMDIv4TT6v1erKmucy LevKLDVuy7XpNZExjz4Xb1HkMdyyIJuR4fafFv5QVWt37r2Sem9uaFdtdpvTomzWmJT31OqP53Z /JgA= X-Mailer: git-send-email 2.43.0.687.g38aa6559b0-goog Message-ID: <20240214122845.2033971-58-ardb+git@google.com> Subject: [PATCH v8 13/43] arm64: Add helpers to probe local CPU for PAC and BTI support From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240214_043010_187805_488DC594 X-CRM114-Status: GOOD ( 12.42 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel Add some helpers that will be used by the early kernel mapping code to check feature support on the local CPU. This permits the early kernel mapping to be created with the right attributes, removing the need for tearing it down and recreating it. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/cpufeature.h | 32 ++++++++++++++++++++ 1 file changed, 32 insertions(+) diff --git a/arch/arm64/include/asm/cpufeature.h b/arch/arm64/include/asm/cpufeature.h index 03c34242bfc7..e3edae1825f3 100644 --- a/arch/arm64/include/asm/cpufeature.h +++ b/arch/arm64/include/asm/cpufeature.h @@ -963,6 +963,38 @@ static inline bool kaslr_disabled_cmdline(void) u32 get_kvm_ipa_limit(void); void dump_cpu_features(void); +static inline bool cpu_has_bti(void) +{ + if (!IS_ENABLED(CONFIG_ARM64_BTI)) + return false; + + return arm64_apply_feature_override(read_cpuid(ID_AA64PFR1_EL1), + ID_AA64PFR1_EL1_BT_SHIFT, 4, + &id_aa64pfr1_override); +} + +static inline bool cpu_has_pac(void) +{ + u64 isar1, isar2; + + if (!IS_ENABLED(CONFIG_ARM64_PTR_AUTH)) + return false; + + isar1 = read_cpuid(ID_AA64ISAR1_EL1); + isar2 = read_cpuid(ID_AA64ISAR2_EL1); + + if (arm64_apply_feature_override(isar1, ID_AA64ISAR1_EL1_APA_SHIFT, 4, + &id_aa64isar1_override)) + return true; + + if (arm64_apply_feature_override(isar1, ID_AA64ISAR1_EL1_API_SHIFT, 4, + &id_aa64isar1_override)) + return true; + + return arm64_apply_feature_override(isar2, ID_AA64ISAR2_EL1_APA3_SHIFT, 4, + &id_aa64isar2_override); +} + #endif /* __ASSEMBLY__ */ #endif From patchwork Wed Feb 14 12:29:00 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13556539 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id DF362C48BC4 for ; Wed, 14 Feb 2024 13:30:56 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=EDoKJS3bRCRKjAUm6cgAQXT8sjexNrT4+Ye933naiPk=; b=eO8fFM3GrPrpkwezrp3r8GLO7M kwcCgFLUuQHtBsfjhdw9JMXEkRMCdQ/4IJcu4l/TAYwCklLkeYA7svxLlIzQbfrcUjCSjzW+cNS3s b1irwFSaCoc0LQk3RkRB7ZPTwQhT8+oz/60FL74l9qgepRsE4EQUufeRujqzOy1jxZRZgB8lQ6XAD YlIvY5YDtvbKRbJeZI/D9MYGHhEpozRBmhSoioCXB2e0c4zEp0X9PHyXaU+MaJFLv3nA4HfgIynuP cU1zcMynHYrcPLL0KsxHfcnfYC7yn4NH9qpbCvHFPm8Rf1qtquFoVRrvQr1tICg0+Wfg2wd+fOw7B ZmVc1ScQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1raFLb-0000000D1mF-3I1E; Wed, 14 Feb 2024 13:30:47 +0000 Received: from mail-yb1-xb4a.google.com ([2607:f8b0:4864:20::b4a]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1raEOx-0000000CoS2-331m for linux-arm-kernel@lists.infradead.org; Wed, 14 Feb 2024 12:30:16 +0000 Received: by mail-yb1-xb4a.google.com with SMTP id 3f1490d57ef6-dcdc3db67f0so67567276.1 for ; Wed, 14 Feb 2024 04:30:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1707913810; x=1708518610; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=9Rh7J9+kc1jH+ZATGNo3sjM+R+6H3pGkV8VVVSJtZCs=; b=ca0R9biLn/P5AwMhDf9TYuA6YwxcIRZGvzfojtmQlvG1aw6RcNOa1V+Plc7qAdhBKv DqY3AtOHADBPV9ovTEsERpk2xoaJwbf1Dvdet+WakijYca5Upg2e+PLOTacTgktGcbHA oU/wGS8n6kMlHuX1ICUnyo5jn7PII8/NdLW6Rph0cKe5DzeJplz6uL95Nnd1M3ofL/DR F73XDWt+08wcT3w52BtWV3KuF3927RAEVl+ZpxPZFiO6iDjqayM3TIcpottbUqSkpjLR iqHpsdlQ9HaJ08wEhHxFb85s1ftWLxc4e1dtqGWqKoB+a7VfI/EhQ6okLV/Gcomo1efR XYLg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707913810; x=1708518610; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=9Rh7J9+kc1jH+ZATGNo3sjM+R+6H3pGkV8VVVSJtZCs=; b=U8znu2Sm8KQyVxE2SgnKYWYWthEHJnMkGzfFtGdU8u0N2vN6hDwbAlx0kIX8n5VA/T oFpUKKYVQYh2umEzHUg7ExqUiogTLUNyImBVlPLc0cBlj2MZfZTJZPFca1bwUuo0z9Yw w7/llE4o7IAiwzFHdU9hiJU5MaW74iVU2kChsP9a1hektU5f9C3bAHu9Zwdv6OOFXfVj OYRojHA3PdY1QdKmUcvJ62jaKhx6ean5YMaDTCL/2FXZjRt4XCW3GZdXndDQVVADLhjR zxO8c+wQDOJuSftn4o0o1UmDKCINa4kvChovzB3I5M3G0Gq5BqcE6uSsWwyURKGwdNwV axNg== X-Gm-Message-State: AOJu0Yz1FzY9CCrRhjcxoTCNaqes5YuiYNRCVbYXA4Wv0GwIMe4HCOBH 0jAbFvDExCP3rdfVNmWM6KTlr/Y+zle2aKjdDB2TWi62wmVL/GlvpjTycM9R2juEqs50LDvq3fW 6t79Nuv6OwjNh8TObSkRy5p7KhX+Tx6b2C7eZSmNjZVKJDOb9BM54h5LBEuBGM7ij91ctWZqjFh X8FPDHzi31G6ibvCIQ92Ok5Yhjb10B02IuZzI8/GHX X-Google-Smtp-Source: AGHT+IGnfW24f+Yz1lHGMfiACTuuKnWu9SLnEXYKx9gq2mhQ59zByJdlRdcvE+gtHMTJ4hvya2rlVL7N X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:6902:1884:b0:dbd:ee44:8908 with SMTP id cj4-20020a056902188400b00dbdee448908mr475936ybb.0.1707913810466; Wed, 14 Feb 2024 04:30:10 -0800 (PST) Date: Wed, 14 Feb 2024 13:29:00 +0100 In-Reply-To: <20240214122845.2033971-45-ardb+git@google.com> Mime-Version: 1.0 References: <20240214122845.2033971-45-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=1834; i=ardb@kernel.org; h=from:subject; bh=xWYZgHk/D2BgZ7XwkPHxJSDXujQ5g/J/prtVcRWgTdo=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIfXMJt5s17ULPr1OuOcqc9jmI4OjiDZzesi/PV1c2TJJL vFPWao6SlkYxDgYZMUUWQRm/3238/REqVrnWbIwc1iZQIYwcHEKwERKshgZNn5w/XHmrJRWs9+N Y3ucJ/z1c9S3vaeSv0BJ2P8vo/imPkaGSabe/Ed3bLDqttn/Lfm9Yf6j4G2/Oipvps1ZrD2Fo0m NDwA= X-Mailer: git-send-email 2.43.0.687.g38aa6559b0-goog Message-ID: <20240214122845.2033971-59-ardb+git@google.com> Subject: [PATCH v8 14/43] arm64: head: allocate more pages for the kernel mapping From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240214_043011_957077_AF29096F X-CRM114-Status: GOOD ( 13.82 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel In preparation for switching to an early kernel mapping routine that maps each segment according to its precise boundaries, and with the correct attributes, let's allocate some extra pages for page tables for the 4k page size configuration. This is necessary because the start and end of each segment may not be aligned to the block size, and so we'll need an extra page table at each segment boundary. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/kernel-pgtable.h | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/arch/arm64/include/asm/kernel-pgtable.h b/arch/arm64/include/asm/kernel-pgtable.h index 83ddb14b95a5..0631604995ee 100644 --- a/arch/arm64/include/asm/kernel-pgtable.h +++ b/arch/arm64/include/asm/kernel-pgtable.h @@ -68,7 +68,7 @@ + EARLY_PGDS((vstart), (vend), add) /* each PGDIR needs a next level page table */ \ + EARLY_PUDS((vstart), (vend), add) /* each PUD needs a next level page table */ \ + EARLY_PMDS((vstart), (vend), add)) /* each PMD needs a next level page table */ -#define INIT_DIR_SIZE (PAGE_SIZE * EARLY_PAGES(KIMAGE_VADDR, _end, EXTRA_PAGE)) +#define INIT_DIR_SIZE (PAGE_SIZE * (EARLY_PAGES(KIMAGE_VADDR, _end, EXTRA_PAGE) + EARLY_SEGMENT_EXTRA_PAGES)) /* the initial ID map may need two extra pages if it needs to be extended */ #if VA_BITS < 48 @@ -89,6 +89,15 @@ #define SWAPPER_TABLE_SHIFT PMD_SHIFT #endif +/* The number of segments in the kernel image (text, rodata, inittext, initdata, data+bss) */ +#define KERNEL_SEGMENT_COUNT 5 + +#if SWAPPER_BLOCK_SIZE > SEGMENT_ALIGN +#define EARLY_SEGMENT_EXTRA_PAGES (KERNEL_SEGMENT_COUNT + 1) +#else +#define EARLY_SEGMENT_EXTRA_PAGES 0 +#endif + /* * Initial memory map attributes. */ From patchwork Wed Feb 14 12:29:01 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13556544 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id DEDF0C48BC1 for ; Wed, 14 Feb 2024 13:45:56 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=BRHhyoO2AGJwzLEn0unfTNx3qbt8KuYB7iMDfU1Rpf8=; b=PT7a3EBjXEZb5giC34qiyLPgrQ qaFL/7+A/j3DSxkHhHPLRhUgS8mGz+YNxFfzfKUaCZnfjVzihm6XWFGS3/4wOGVoUqxnTSHoFVgIg mTvst2n+fZzQLwyWjJGo58hh/7xKWE/WZZ0+pv/JVSlBFkGGNWQVWFrqJRs6rpiY95VfKFMrEde+v 2Ycb/zAMGcdzc8NOeX6FYf5PYDc4APjpuEi3eIyOQ1eL3+nloXOKnZIHjVf1Gj1wO0siLr2QGWRRx tTxx00P00ETnfFmnGLzUzIzgOfuzzElgngu43lwQNnD4b+HuP9E7/wTZopSrEqfcjBIqwKN+rP6PG QTTWDi4g==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1raFa5-0000000D3fG-3Elk; Wed, 14 Feb 2024 13:45:45 +0000 Received: from mail-wr1-x44a.google.com ([2a00:1450:4864:20::44a]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1raEP1-0000000CoTL-3zJI for linux-arm-kernel@lists.infradead.org; Wed, 14 Feb 2024 12:30:23 +0000 Received: by mail-wr1-x44a.google.com with SMTP id ffacd0b85a97d-33ce2ba4ee6so327722f8f.0 for ; Wed, 14 Feb 2024 04:30:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1707913812; x=1708518612; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=ruvNrUrxPHYFlNpK26kQcD/B9UWA+NHp79ghm3N4TPw=; b=rEpU55nqCzUbSwPCfkQAQ9rH3arbW7h1IugvpglHzLVoV1rH1EwkuwKjEox7aNWFNt nl6BUUWlI/XbQ1M/HlIxPcXVZc3DEOMR7ppmKsh04RMWoKz/1/znQYd8KHRXUKrChJFB L++LKMKfkCpUgNh+S6MENFDN0wdK7cQdQjI9dmV4cHQZH1fRol0ZG1rkeZnVLeBlNm1k defqb3P72UdJpJkuKDwyJf4TrSyEtJDOL0VnyFGlZ3r1/4vZyf4BwEHlbCo/AKDaVlW4 9ijIPv6yxMSfnyrOfqRhhll4XpN/12meP9Np/UssTf3xMozLVyIE8QiTW4s5h+et8frL sepw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707913812; x=1708518612; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=ruvNrUrxPHYFlNpK26kQcD/B9UWA+NHp79ghm3N4TPw=; b=lq6kPQWdtAAlZsolXkq5B2fv80fPIQTHyalef3+LsU1ypl++iy+/Cj+5IUQ/pUAMtt SibOyv7le7bI1LUVJReaJjra8HPRqk4Vt1oXlQ6v31CoUZkNqnVnbahaSOxjw7yzOLKF jKe2UHy5JSHBGgmGM7nyyWi2P7N9gej50HT57wcbRWnAuX4kch3N7klnfa3OvXYhOche cba4MH6XQ/CuzPIRb/iDQeAZ7QSqcIHQewyYZX6GJ9anBgfSDnHC5IFbMglIGpbtff2Q 2vFMmrDC/T8JLCUDBCfATQ1zZQYQ0rVt6dHcEX7cVy9OHNV74+GZffsjNCAYKAN/+xqe 1T3g== X-Gm-Message-State: AOJu0YxOFbStSbDnt7OqvCRdGAg44l10bEQHRX3TjeGNz/GoZcCipB12 YuMgBG2VE93Yvk1ZVcALtS0EqPo4J2sNRDh3qTmFMJ/uNhm1XrzxeGnS9fE2nSre6LkzBpwBkc4 x8KT92yryUevGysMW7eJsEUVRDG1w//LbWjDEX6VyQerXg+xuTbUanfvqKZOR471dWKW2BiAdxH 8odzv7ATwZ9SYlNEYXh9Fcf2xH/NMXcbQ6M4tFVVp8 X-Google-Smtp-Source: AGHT+IHVARgn30CiKknQ//jlLpFzTrA6E49oduOJKYbxNsyPry4qTVXmSI7wnDCwoFeb8VZhPnGsu7zn X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:adf:a45d:0:b0:33c:e728:c88a with SMTP id e29-20020adfa45d000000b0033ce728c88amr3396wra.3.1707913812670; Wed, 14 Feb 2024 04:30:12 -0800 (PST) Date: Wed, 14 Feb 2024 13:29:01 +0100 In-Reply-To: <20240214122845.2033971-45-ardb+git@google.com> Mime-Version: 1.0 References: <20240214122845.2033971-45-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=3177; i=ardb@kernel.org; h=from:subject; bh=ZqGi+ki4hTr1X+TLcrERZLO6aUgrgkKOKJLRLKyMpck=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIfXMJr5HWyT2JRw8dpBjn9Ome9UHjl98JnWLp+jhiheRK 6rat5podJSyMIhxMMiKKbIIzP77bufpiVK1zrNkYeawMoEMYeDiFICJFE1gZJj69tWdqruTbz8y lbevzp2WyR4r0W8n09N4fV9OV3N2tgojw3ZBRsHKlsk38hW14kX2qWaf5/rw3P95Qi3v+x9NOXJ NjAA= X-Mailer: git-send-email 2.43.0.687.g38aa6559b0-goog Message-ID: <20240214122845.2033971-60-ardb+git@google.com> Subject: [PATCH v8 15/43] arm64: head: move memstart_offset_seed handling to C code From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240214_043016_429271_D756AA9A X-CRM114-Status: GOOD ( 13.88 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel Now that we can set BSS variables from the early code running from the ID map, we can set memstart_offset_seed directly from the C code that derives the value instead of passing it back and forth between C and asm code. Signed-off-by: Ard Biesheuvel --- arch/arm64/kernel/head.S | 7 ------- arch/arm64/kernel/image-vars.h | 1 + arch/arm64/kernel/pi/kaslr_early.c | 4 ++++ 3 files changed, 5 insertions(+), 7 deletions(-) diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S index b320702032a7..aa7766dc64d9 100644 --- a/arch/arm64/kernel/head.S +++ b/arch/arm64/kernel/head.S @@ -82,7 +82,6 @@ * x21 primary_entry() .. start_kernel() FDT pointer passed at boot in x0 * x22 create_idmap() .. start_kernel() ID map VA of the DT blob * x23 __primary_switch() physical misalignment/KASLR offset - * x24 __primary_switch() linear map KASLR seed * x25 primary_entry() .. start_kernel() supported VA size * x28 create_idmap() callee preserved temp register */ @@ -483,11 +482,6 @@ SYM_FUNC_START_LOCAL(__primary_switched) str x25, [x8] // ... observes the correct value dc civac, x8 // Make visible to booting secondaries #endif - -#ifdef CONFIG_RANDOMIZE_BASE - adrp x5, memstart_offset_seed // Save KASLR linear map seed - strh w24, [x5, :lo12:memstart_offset_seed] -#endif #if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS) bl kasan_early_init #endif @@ -779,7 +773,6 @@ SYM_FUNC_START_LOCAL(__primary_switch) #ifdef CONFIG_RANDOMIZE_BASE mov x0, x22 bl __pi_kaslr_early_init - and x24, x0, #SZ_2M - 1 // capture memstart offset seed bic x0, x0, #SZ_2M - 1 orr x23, x23, x0 // record kernel offset #endif diff --git a/arch/arm64/kernel/image-vars.h b/arch/arm64/kernel/image-vars.h index eacc3d167733..8d96052079e8 100644 --- a/arch/arm64/kernel/image-vars.h +++ b/arch/arm64/kernel/image-vars.h @@ -46,6 +46,7 @@ PROVIDE(__pi_id_aa64smfr0_override = id_aa64smfr0_override); PROVIDE(__pi_id_aa64zfr0_override = id_aa64zfr0_override); PROVIDE(__pi_arm64_sw_feature_override = arm64_sw_feature_override); PROVIDE(__pi__ctype = _ctype); +PROVIDE(__pi_memstart_offset_seed = memstart_offset_seed); #ifdef CONFIG_KVM diff --git a/arch/arm64/kernel/pi/kaslr_early.c b/arch/arm64/kernel/pi/kaslr_early.c index f2305e276ec3..eeecee7ffd6f 100644 --- a/arch/arm64/kernel/pi/kaslr_early.c +++ b/arch/arm64/kernel/pi/kaslr_early.c @@ -16,6 +16,8 @@ #include #include +extern u16 memstart_offset_seed; + static u64 __init get_kaslr_seed(void *fdt) { static char const chosen_str[] __initconst = "chosen"; @@ -51,6 +53,8 @@ asmlinkage u64 __init kaslr_early_init(void *fdt) return 0; } + memstart_offset_seed = seed & U16_MAX; + /* * OK, so we are proceeding with KASLR enabled. Calculate a suitable * kernel image offset from the seed. Let's place the kernel in the From patchwork Wed Feb 14 12:29:02 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13556417 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id DC74FC48BED for ; Wed, 14 Feb 2024 12:32:24 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=c+qnMNEKAeXH7F5OfjB8Zw13kBoUwgKGDAoDjlicOVQ=; b=K4dcyvyxc80gRWTdKUcewOLFa8 PGqDhWwtdkg+hlkj5uWutfcWv+q3/opg8NZ43I66u4lDzWIz2gzgQSwNYhs4U/qmF6USFxYwQXoAk Z7fa3Fq6m0RENo2HQZ6Cc6ktxYl6G8RfQioiYgp18T4ms1PwfGwasJdJ1+IWx8Ur23MPjENRvCZrr +QcAXyxnSZHjzSnUG+oW0kpNZ9aHZ3fHt089Srrs/5RMZBUAVgO01tFiF/GqVOgU8i2r9Xnjc9mD5 xz/TQFUJovkoAVtk5QjqExcAY4rlwIPza11gAX62j4rFrbESvK80eGqtTAwUltVdrIPoy6+jRS4Ly fzpDFLbw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1raEQp-0000000CpOY-03jI; Wed, 14 Feb 2024 12:32:07 +0000 Received: from mail-wm1-f73.google.com ([209.85.128.73]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1raEP5-0000000CoUw-0CGu for linux-arm-kernel@lists.infradead.org; Wed, 14 Feb 2024 12:30:40 +0000 Received: by mail-wm1-f73.google.com with SMTP id 5b1f17b1804b1-40fd2fe490bso3343015e9.0 for ; Wed, 14 Feb 2024 04:30:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1707913815; x=1708518615; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=TEtQPPcWPY0ALv0OwL+bGysAAwWr707upVDE6b/XzmA=; b=ItItrDhb/FTwmMcMcauGdN2Hl4DHvdD7RkU4qp7gJ28PIslZqWBVfUbBUPf+H950FM 61AByS0aL4cqwCS7CKrsjwBrTaAFjWSaP+C5v710qjIRp/jEnmGR6HreYtkERSVQ9/ww nz/VCly9ytBkvAtD1QWtyV+ykDLlYoPiH22ld9Vgp+5DyVQNLhuQNpuVww+3AHhywRbn piMKmWZleb7Q8nWOghI/k83um4afUFqMQurAoETdIOI/lS9LBBlR40d+hp0kiy9TwsMm Ud6MD3BlcGBiHIcQu0Ar3GPybZEeJrEM6LUTrGqho24T8vCkVZwTEWkSU5ideoBj4uRh RmOA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707913815; x=1708518615; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=TEtQPPcWPY0ALv0OwL+bGysAAwWr707upVDE6b/XzmA=; b=ssERzwdsmmc5+oNmFE+dBKVUFlO86E4n+AtUedXfYCtJ9/77tuytG152H74TRxGzS5 yt8EgQcj4OHyNfdB86m6NCSANtZAvNGdzlRNVbr92ATsGpNP0/bw+uB/hYvsu2D+Nz9x iCbQNufGPPgWTvT5lDksTx3xQdH3al6Lmjy21we9KugILh5RHhFVr2HSUhNhi0uDTfHi X4qTVsnrWdZq8L5WJhyEuVNpQqD0e6ZLRAFRJS65YFXZd1Ka6NbuLD+uElt6L+NWXqwT mjeOEPvlsTs87D2ycUZks7OJTieMkClLdTmwMtYTBkPeqqnDMthSGP860ejwrFO5jUYF Rhsw== X-Gm-Message-State: AOJu0YwiOJ50VUu8URp/58bvgRO6kNlcPB0lE0H0taAoQzbD1CE978Su +Af6YEtIbwjYJ6vQFf6Y5fzED1/bRd1kIlY9Pb3GVUq8H3ODWdLiYaBatG6g4p8Qku7h+YlR6Pv aUhe9nIvSP55lfPmGET5DCOU5gcLVIPWXuoK9GhpefOdGI7o5qVKU8k4aG8NUuDM3aeb05CPd3B qMYNT39FSy0hidKmYMESYaIh9cehDfiYpbaj/VURCA X-Google-Smtp-Source: AGHT+IGKgzBaCx1uc8lpYFEGvXBQGLgG3xISEHWN/WQLPXSiZD1fXfW02wVkPSmx/CQhCW+LWb0FYCnF X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:600c:54e5:b0:410:e4fe:1a1f with SMTP id jb5-20020a05600c54e500b00410e4fe1a1fmr47704wmb.0.1707913814861; Wed, 14 Feb 2024 04:30:14 -0800 (PST) Date: Wed, 14 Feb 2024 13:29:02 +0100 In-Reply-To: <20240214122845.2033971-45-ardb+git@google.com> Mime-Version: 1.0 References: <20240214122845.2033971-45-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=5075; i=ardb@kernel.org; h=from:subject; bh=tdrW6PySIyYhT0Z6b6jbS+n+X05qOd2DwGUk6HZP+Fg=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIfXMJv7jN1aybEhcPy3TMbzs+eHvUqufy5fPPOuv2v3F/ 2xmwZMDHaUsDGIcDLJiiiwCs/++23l6olSt8yxZmDmsTCBDGLg4BWAiKQKMDDPK6hcuuJ8W/+DQ pNsFb/R03m0Wd7y0zC+g56HBchbvaakM/5OuLp87TYJ1aVFi9/LozwZsTFMPTPS9/cZ504JWzts fNjIBAA== X-Mailer: git-send-email 2.43.0.687.g38aa6559b0-goog Message-ID: <20240214122845.2033971-61-ardb+git@google.com> Subject: [PATCH v8 16/43] arm64: mm: Make kaslr_requires_kpti() a static inline From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240214_043019_402006_F7370757 X-CRM114-Status: GOOD ( 23.76 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel In preparation for moving the first assignment of arm64_use_ng_mappings to an earlier stage in the boot, ensure that kaslr_requires_kpti() is accessible without relying on the core kernel's view on whether or not KASLR is enabled. So make it a static inline, and move the kaslr_enabled() check out of it and into the callers, one of which will disappear in a subsequent patch. Once/when support for the obsolete ThunderX 1 platform is dropped, this check reduces to a E0PD feature check on the local CPU. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/mmu.h | 38 +++++++++++++++++- arch/arm64/kernel/cpufeature.c | 42 +------------------- arch/arm64/kernel/setup.c | 2 +- 3 files changed, 39 insertions(+), 43 deletions(-) diff --git a/arch/arm64/include/asm/mmu.h b/arch/arm64/include/asm/mmu.h index 2fcf51231d6e..d0b8b4b413b6 100644 --- a/arch/arm64/include/asm/mmu.h +++ b/arch/arm64/include/asm/mmu.h @@ -71,7 +71,43 @@ extern void create_pgd_mapping(struct mm_struct *mm, phys_addr_t phys, pgprot_t prot, bool page_mappings_only); extern void *fixmap_remap_fdt(phys_addr_t dt_phys, int *size, pgprot_t prot); extern void mark_linear_text_alias_ro(void); -extern bool kaslr_requires_kpti(void); + +/* + * This check is triggered during the early boot before the cpufeature + * is initialised. Checking the status on the local CPU allows the boot + * CPU to detect the need for non-global mappings and thus avoiding a + * pagetable re-write after all the CPUs are booted. This check will be + * anyway run on individual CPUs, allowing us to get the consistent + * state once the SMP CPUs are up and thus make the switch to non-global + * mappings if required. + */ +static inline bool kaslr_requires_kpti(void) +{ + /* + * E0PD does a similar job to KPTI so can be used instead + * where available. + */ + if (IS_ENABLED(CONFIG_ARM64_E0PD)) { + u64 mmfr2 = read_sysreg_s(SYS_ID_AA64MMFR2_EL1); + if (cpuid_feature_extract_unsigned_field(mmfr2, + ID_AA64MMFR2_EL1_E0PD_SHIFT)) + return false; + } + + /* + * Systems affected by Cavium erratum 24756 are incompatible + * with KPTI. + */ + if (IS_ENABLED(CONFIG_CAVIUM_ERRATUM_27456)) { + extern const struct midr_range cavium_erratum_27456_cpus[]; + + if (is_midr_in_range_list(read_cpuid_id(), + cavium_erratum_27456_cpus)) + return false; + } + + return true; +} #define INIT_MM_CONTEXT(name) \ .pgd = init_pg_dir, diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c index d0ffb872a31a..7064cf13f226 100644 --- a/arch/arm64/kernel/cpufeature.c +++ b/arch/arm64/kernel/cpufeature.c @@ -1620,46 +1620,6 @@ has_useable_cnp(const struct arm64_cpu_capabilities *entry, int scope) return has_cpuid_feature(entry, scope); } -/* - * This check is triggered during the early boot before the cpufeature - * is initialised. Checking the status on the local CPU allows the boot - * CPU to detect the need for non-global mappings and thus avoiding a - * pagetable re-write after all the CPUs are booted. This check will be - * anyway run on individual CPUs, allowing us to get the consistent - * state once the SMP CPUs are up and thus make the switch to non-global - * mappings if required. - */ -bool kaslr_requires_kpti(void) -{ - if (!IS_ENABLED(CONFIG_RANDOMIZE_BASE)) - return false; - - /* - * E0PD does a similar job to KPTI so can be used instead - * where available. - */ - if (IS_ENABLED(CONFIG_ARM64_E0PD)) { - u64 mmfr2 = read_sysreg_s(SYS_ID_AA64MMFR2_EL1); - if (cpuid_feature_extract_unsigned_field(mmfr2, - ID_AA64MMFR2_EL1_E0PD_SHIFT)) - return false; - } - - /* - * Systems affected by Cavium erratum 24756 are incompatible - * with KPTI. - */ - if (IS_ENABLED(CONFIG_CAVIUM_ERRATUM_27456)) { - extern const struct midr_range cavium_erratum_27456_cpus[]; - - if (is_midr_in_range_list(read_cpuid_id(), - cavium_erratum_27456_cpus)) - return false; - } - - return kaslr_enabled(); -} - static bool __meltdown_safe = true; static int __kpti_forced; /* 0: not forced, >0: forced on, <0: forced off */ @@ -1712,7 +1672,7 @@ static bool unmap_kernel_at_el0(const struct arm64_cpu_capabilities *entry, } /* Useful for KASLR robustness */ - if (kaslr_requires_kpti()) { + if (kaslr_enabled() && kaslr_requires_kpti()) { if (!__kpti_forced) { str = "KASLR"; __kpti_forced = 1; diff --git a/arch/arm64/kernel/setup.c b/arch/arm64/kernel/setup.c index 97d2143669cf..0ef45d1927b3 100644 --- a/arch/arm64/kernel/setup.c +++ b/arch/arm64/kernel/setup.c @@ -288,7 +288,7 @@ void __init __no_sanitize_address setup_arch(char **cmdline_p) * mappings from the start, avoiding the cost of rewriting * everything later. */ - arm64_use_ng_mappings = kaslr_requires_kpti(); + arm64_use_ng_mappings = kaslr_enabled() && kaslr_requires_kpti(); early_fixmap_init(); early_ioremap_init(); From patchwork Wed Feb 14 12:29:03 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13556420 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 79E4BC48BEE for ; Wed, 14 Feb 2024 12:32:26 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=DgVORo34ERPsEt0hfNH0SP9DbPj+JjPG4g9Q3dWhsXs=; b=dDE7vPyn7daLrGVFwb8qjemn9R 31E4iZG5lIPoTcaD6ymJ6NL35sP2j1aHC+I/sWQf8HlIVhV8JaGxNVUAuAjHcn8xaE+5gkaBObTGX 2L/EzFxuTapnP6P5JdAt8iEAlGhUDWGpE5F4XyTvehZxqGycSYgL3e1JKIiL8eo/wtTTptUL/llT4 OR40SLk+i4/o8SWeuoBoV51xY6mxyq+MGtB9ybwtQZIS7/oJO5+d3rCvltEHWNQSQKI68F/Ff5vN2 19x0tU55ZFf8u4NbL6abU+jbPtpJ131odHfihC51pxk2QZYb+T6/8dWKwTHIzLWMTTyTdZo9tmb7C A6LP7naA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1raEQq-0000000CpPL-2llI; Wed, 14 Feb 2024 12:32:08 +0000 Received: from mail-yw1-x1149.google.com ([2607:f8b0:4864:20::1149]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1raEP5-0000000CoWO-0ogf for linux-arm-kernel@lists.infradead.org; Wed, 14 Feb 2024 12:30:43 +0000 Received: by mail-yw1-x1149.google.com with SMTP id 00721157ae682-6077ec913f0so42263777b3.0 for ; Wed, 14 Feb 2024 04:30:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1707913818; x=1708518618; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=unR/pVecox9ouLHJTW1/MJ7pGY21RIdyDZtkCV8+fVo=; b=GjVY1HSfjG8bUdlyJuGW1JyarUq2MX5nhWrUhCy8/ougoatm9LsE1wd3Vcf/78bMlr Cwg+AOqB8uFTtVdrmqybETmXO35FmfretwBce/N5CPN0ky9mpx5mlAVTQHjjkJQbRb4A Q5RffWKb9xJnqTSoQF2qUmypCI5p+/V7sm1cwD6Y5TEW+OZ3Vung8g2HI9y18fLIaLuZ ruSozPKiPS7bnwxpcShktrvdYpijYc8j0nDH6guAH0vInjti1z0G6CnMmD6XWCPxQxqt yzC+D2O1VlbGha2HstO3xjozN0AhA+wyw+YxgzC4mIGnTgqbe2wAlKfclxVoFfwEkjrt HvpQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707913818; x=1708518618; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=unR/pVecox9ouLHJTW1/MJ7pGY21RIdyDZtkCV8+fVo=; b=GPZkWwDpY2IMsTfcKBBl82ej8ndhwEHyRMtuWPm1MTsU+H9fSS3JIic7bIpjEYAQC1 4l5FM6FH2Qj24k65oBuAYGRGX3A/4XDeArl8syIYGhLNmMiFMCTRGtRrpYtJ7MkJk+dN eOE76HiqcOLJN6wfkIKwBuHTN+4h7+M85k+lG8pktiooRAzlt8bz99laYXdg+yneexZK bfQOnz5QbzlVVSTi1987SOwrAXKP6aWAdjudM9eoMPphUhOb+Ep4svA7RwLw6y6VCs0t nI/Cm4nlivFNvDERQDVaKZR/wz2x7W6+YeimsZEPo//EAyBswEFnL+8eHypCC59LwU/b aEnA== X-Gm-Message-State: AOJu0YwDWU9EThSjguUg0rC2hvc9cE9Vwo9pP96/vAgToB4PA/u/0UbN c+l4J3FnE7p81DEyOIEOl8iUWTrVpATEfIYCpQNGLJ7ySiERGhABXOvOJKnMpGZITU/8NDVenPi GNlH+QNvqJ9NV9e6SL5BenOOw1/CLPgcjz0+wuhl33m8vHID7OdbxhYVTZrYR6VmmG3zLryMA99 nPAbtigF3EvZFSUoPOcqeRkdaEe7UeonhqjKg6YDJ1 X-Google-Smtp-Source: AGHT+IHN3zejx7M9FiR6GcYODwOwKE72XRoZO7fvo1jy4v0X6jmIXBIjGAI/kZeCHLqHeyBp/dD3tBJF X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:6902:1007:b0:dc9:5ef8:2b2d with SMTP id w7-20020a056902100700b00dc95ef82b2dmr554101ybt.4.1707913817612; Wed, 14 Feb 2024 04:30:17 -0800 (PST) Date: Wed, 14 Feb 2024 13:29:03 +0100 In-Reply-To: <20240214122845.2033971-45-ardb+git@google.com> Mime-Version: 1.0 References: <20240214122845.2033971-45-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=3644; i=ardb@kernel.org; h=from:subject; bh=fwxVCFIDdoHP5fUkExYxnLXpMatgoL0Rf/pPEVWwwN0=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIfXMJoFir2w5J/f2/RdWLFXm9TohcrrkalfI1an78t+9f mW9cKd0RykLgxgHg6yYIovA7L/vdp6eKFXrPEsWZg4rE8gQBi5OAZjIDTGGf7qe2+8GVttsPWo4 8dyeep7XX1m+evH+/Xy9i4/LWCPgkjLDXzllu2t/dndeDLQ+nLP5iE3j9Lns3zdHW/82YZ6iviR oIRMA X-Mailer: git-send-email 2.43.0.687.g38aa6559b0-goog Message-ID: <20240214122845.2033971-62-ardb+git@google.com> Subject: [PATCH v8 17/43] arm64: mmu: Make __cpu_replace_ttbr1() out of line From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240214_043019_549592_6DD9727B X-CRM114-Status: GOOD ( 16.37 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel __cpu_replace_ttbr1() is a static inline, and so it gets instantiated wherever it is used. This is not really necessary, as it is never called on a hot path. It also has the unfortunate side effect that the symbol idmap_cpu_replace_ttbr1 may never be referenced from kCFI enabled C code, and this means the type id symbol may not exist either. This will result in a build error once we start referring to this symbol from asm code as well. (Note that this problem only occurs when CnP, KAsan and suspend/resume are all disabled in the Kconfig but that is a valid config, if unusual). So let's just move it out of line so all callers will share the same implementation, which will reference idmap_cpu_replace_ttbr1 unconditionally. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/mmu_context.h | 32 +------------------- arch/arm64/mm/mmu.c | 32 ++++++++++++++++++++ 2 files changed, 33 insertions(+), 31 deletions(-) diff --git a/arch/arm64/include/asm/mmu_context.h b/arch/arm64/include/asm/mmu_context.h index 9ce4200508b1..926fbbcecbe0 100644 --- a/arch/arm64/include/asm/mmu_context.h +++ b/arch/arm64/include/asm/mmu_context.h @@ -148,37 +148,7 @@ static inline void cpu_install_ttbr0(phys_addr_t ttbr0, unsigned long t0sz) isb(); } -/* - * Atomically replaces the active TTBR1_EL1 PGD with a new VA-compatible PGD, - * avoiding the possibility of conflicting TLB entries being allocated. - */ -static inline void __cpu_replace_ttbr1(pgd_t *pgdp, pgd_t *idmap, bool cnp) -{ - typedef void (ttbr_replace_func)(phys_addr_t); - extern ttbr_replace_func idmap_cpu_replace_ttbr1; - ttbr_replace_func *replace_phys; - unsigned long daif; - - /* phys_to_ttbr() zeros lower 2 bits of ttbr with 52-bit PA */ - phys_addr_t ttbr1 = phys_to_ttbr(virt_to_phys(pgdp)); - - if (cnp) - ttbr1 |= TTBR_CNP_BIT; - - replace_phys = (void *)__pa_symbol(idmap_cpu_replace_ttbr1); - - __cpu_install_idmap(idmap); - - /* - * We really don't want to take *any* exceptions while TTBR1 is - * in the process of being replaced so mask everything. - */ - daif = local_daif_save(); - replace_phys(ttbr1); - local_daif_restore(daif); - - cpu_uninstall_idmap(); -} +void __cpu_replace_ttbr1(pgd_t *pgdp, pgd_t *idmap, bool cnp); static inline void cpu_enable_swapper_cnp(void) { diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index 1ac7467d34c9..f9332eea318f 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -1486,3 +1486,35 @@ void ptep_modify_prot_commit(struct vm_area_struct *vma, unsigned long addr, pte { set_pte_at(vma->vm_mm, addr, ptep, pte); } + +/* + * Atomically replaces the active TTBR1_EL1 PGD with a new VA-compatible PGD, + * avoiding the possibility of conflicting TLB entries being allocated. + */ +void __cpu_replace_ttbr1(pgd_t *pgdp, pgd_t *idmap, bool cnp) +{ + typedef void (ttbr_replace_func)(phys_addr_t); + extern ttbr_replace_func idmap_cpu_replace_ttbr1; + ttbr_replace_func *replace_phys; + unsigned long daif; + + /* phys_to_ttbr() zeros lower 2 bits of ttbr with 52-bit PA */ + phys_addr_t ttbr1 = phys_to_ttbr(virt_to_phys(pgdp)); + + if (cnp) + ttbr1 |= TTBR_CNP_BIT; + + replace_phys = (void *)__pa_symbol(idmap_cpu_replace_ttbr1); + + __cpu_install_idmap(idmap); + + /* + * We really don't want to take *any* exceptions while TTBR1 is + * in the process of being replaced so mask everything. + */ + daif = local_daif_save(); + replace_phys(ttbr1); + local_daif_restore(daif); + + cpu_uninstall_idmap(); +} From patchwork Wed Feb 14 12:29:04 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13556418 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id EB954C48BC3 for ; Wed, 14 Feb 2024 12:32:23 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=IqR+Bzq/qVPy8DDFsTQ8gwczXQb7qq52FhYVQ8FRai8=; b=Ue6mWlROtvVhtgpgBVV464o91n czN35Yxu2prNCUewwHcJ3d6S/FRLUzjAp7P2TuLWMbV+sY8ek3BksdR9AeX7+E/MQgUZfIIao8x0s JrKKOV03V+v0i4LErJrS5iJKkp/B5xv1eeDLLVI37M4QPx4g9sFSS4Wq/W8BRye9aUJypPsE3rLMi 7s+YIL/MmTtH12gVEG3aBiQ5r0lX+3uEVemkbA0g6Ye+A6EisZt3V9NbQpkcZEhgrOu0Q9zC6suc4 SYREnIb1FLoUZch6WV5oqgmkAx+/PUncSUU1Z+rEeFZGeXsTrJX9DAkDNherQ2ldiqIw8Cnb39ijJ OsHm7MqQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1raEQn-0000000CpNh-1y4n; Wed, 14 Feb 2024 12:32:05 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1raEPO-0000000ColR-3d15 for linux-arm-kernel@bombadil.infradead.org; Wed, 14 Feb 2024 12:30:39 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=Ap9KFGV5vVBoqKi4TNkizviqA1yQzx1+cTjUdODcUcE=; b=R9OkYc6C/pe9tdpIx6N+SvFBCN Z/UvkbdEhd3HnaGt8rr6E3/Zk7JBWMPzkTdUc7pGUk7SlGCViF2dm2NjPHkqpjvbMUzEQhe1kyE+T NXosWfyO9yzxtkvQ2zPEOs7iRdw3VNxT4+EqXJu/t3JCr+AsQCBZPZrBuaoNrQS/HE/fJXGsrm6tL A4jqmAOX4oR08G7gpLARRXN7mVc2qwp4SHgLpPfKhGrFk0vIM9Ii/2nQHYINxohv+FdasDWZ0NGbq i0/hRoDnDMkSTXATOJVyqP2xeSd3IZyZ2fWFAfwZHNjppqv8WfIz8UruWtWIt+ZdH6aVPdcKtq3DB a6Upysxg==; Received: from mail-wm1-x349.google.com ([2a00:1450:4864:20::349]) by desiato.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1raEPD-0000000GDzt-2Sm4 for linux-arm-kernel@lists.infradead.org; Wed, 14 Feb 2024 12:30:36 +0000 Received: by mail-wm1-x349.google.com with SMTP id 5b1f17b1804b1-410422e8cd1so30847115e9.3 for ; Wed, 14 Feb 2024 04:30:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1707913820; x=1708518620; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=Ap9KFGV5vVBoqKi4TNkizviqA1yQzx1+cTjUdODcUcE=; b=OZJ8SDTFm0ODhCblPbzt6MiNX/HNh5qONrFyBLG3NE5xlamOmQ20TwV2mXLD1MWnPp EuVgFZM4OTZHadCVtQDsOB71Bl+3TSMIN2qXLaf9AqZBRFIE2gkNmrOgOFDzL0nA8LOF TuFiN+4osJ0fxeHo2uYHEf3Tyhfq4s1y3DbbQs9/pUFuOZhSDO/O00ln+ZmK1BmQ9A3z 8k6TO48h/hNf8XFok2Nl+jSi2ha3TkHmHyArt32yhHenx01QsdFQN/koYs9qERldXely Tfswf9giCQwDwMtm8zYumRZ2zY/YJx71GlR/7ftv0gwJ5BFRGMMnoLEt+2mfDdm1HDbM 4erg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707913820; x=1708518620; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Ap9KFGV5vVBoqKi4TNkizviqA1yQzx1+cTjUdODcUcE=; b=bp5CQresPrhjFJxnKGAItMzlm3g6aOO/wfbv0EaF2xNfdpOBhlB7nrDCIxrgGb6giw gd8RAEOrouBB70uUhuJ4Tx7G4dK5dHN18Sm3yRHPOIuaBhNkBT1rcVZNtV03DVnkheig Pp1tdJlXbnbCaelnJwws54zTqJblticz4EB/1JRhqYXSoC7sq02jGibLKUwMyqtz1y4b DQACoPyFxU1k1/JmuZXsyCy4/rMeKZ3jX7d+ld6VoDWFS5aoI1BewYlW9beHit+JfpCr AFVxiE7zipJbLwqA8c/IaSjM7FBSWg5sj3yauZoJb0csv9X/0Mhjj+TM8Yc/8OILKyVX 4elA== X-Gm-Message-State: AOJu0Yw3iJfJJhgo3a34pRXqSl4dGTb0uYp//dYpnmpb6VomXDIVGHVd 2z3XXy0u6tRTQ3QuE7lq4M7CIiTfBPpxWMpjShpv2qpJRQSv4vfRLlVXSTYWilqlXxWmbdgYZof MJSSyJ6SU9SK/HVOpPX0ENVbRA4huN0D4wufcig6TWIWpqSeLdojKqF6gWmuFCLxQdr3hgCZZ6n b1QYj4tu2DzQyr9vZyr3LHzB2NDt7bwKV0uMGPcYM5 X-Google-Smtp-Source: AGHT+IHRm6i4Qtas609CRYwhhiXGwtMu98qmQb3Ya78yYCFeBo3sc044H1zRa2wlBPYDNP7gPckeG7qg X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a7b:c2b2:0:b0:411:d220:878b with SMTP id c18-20020a7bc2b2000000b00411d220878bmr8630wmk.6.1707913819937; Wed, 14 Feb 2024 04:30:19 -0800 (PST) Date: Wed, 14 Feb 2024 13:29:04 +0100 In-Reply-To: <20240214122845.2033971-45-ardb+git@google.com> Mime-Version: 1.0 References: <20240214122845.2033971-45-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=23592; i=ardb@kernel.org; h=from:subject; bh=lgM35lWHgDaI8TIRQNdCADfLMcPuYO1XMJHb2T6Ge2g=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIfXMJkH3sHlnzLzviG9m7jTNFRUQdAg2PPC7bp1znkS+2 cmMWrmOUhYGMQ4GWTFFFoHZf9/tPD1RqtZ5lizMHFYmkCEMXJwCMBGuDEaGWfFB2zfl8rluSpl3 145zt3V82fSbK1NmuH4Mu71w3unTVxh+MWmdLi622nWA/fGqS8saNk9/ycySw9Sx/PGNJetmR+V 84QQA X-Mailer: git-send-email 2.43.0.687.g38aa6559b0-goog Message-ID: <20240214122845.2033971-63-ardb+git@google.com> Subject: [PATCH v8 18/43] arm64: head: Move early kernel mapping routines into C code From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240214_123028_527897_EF171FE5 X-CRM114-Status: GOOD ( 25.22 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel The asm version of the kernel mapping code works fine for creating a coarse grained identity map, but for mapping the kernel down to its exact boundaries with the right attributes, it is not suitable. This is why we create a preliminary RWX kernel mapping first, and then rebuild it from scratch later on. So let's reimplement this in C, in a way that will make it unnecessary to create the kernel page tables yet another time in paging_init(). Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/archrandom.h | 2 - arch/arm64/include/asm/scs.h | 32 +--- arch/arm64/kernel/head.S | 52 +------ arch/arm64/kernel/image-vars.h | 19 +++ arch/arm64/kernel/pi/Makefile | 1 + arch/arm64/kernel/pi/idreg-override.c | 22 ++- arch/arm64/kernel/pi/kaslr_early.c | 12 +- arch/arm64/kernel/pi/map_kernel.c | 164 ++++++++++++++++++++ arch/arm64/kernel/pi/map_range.c | 88 +++++++++++ arch/arm64/kernel/pi/patch-scs.c | 16 +- arch/arm64/kernel/pi/pi.h | 14 ++ arch/arm64/kernel/pi/relocate.c | 2 + arch/arm64/kernel/setup.c | 7 - arch/arm64/kernel/vmlinux.lds.S | 4 +- arch/arm64/mm/proc.S | 1 + 15 files changed, 315 insertions(+), 121 deletions(-) diff --git a/arch/arm64/include/asm/archrandom.h b/arch/arm64/include/asm/archrandom.h index ecdb3cfcd0f8..8babfbe31f95 100644 --- a/arch/arm64/include/asm/archrandom.h +++ b/arch/arm64/include/asm/archrandom.h @@ -129,6 +129,4 @@ static inline bool __init __early_cpu_has_rndr(void) return (ftr >> ID_AA64ISAR0_EL1_RNDR_SHIFT) & 0xf; } -u64 kaslr_early_init(void *fdt); - #endif /* _ASM_ARCHRANDOM_H */ diff --git a/arch/arm64/include/asm/scs.h b/arch/arm64/include/asm/scs.h index eca2ba5a6276..2e010ea76be2 100644 --- a/arch/arm64/include/asm/scs.h +++ b/arch/arm64/include/asm/scs.h @@ -33,37 +33,11 @@ #include #ifdef CONFIG_UNWIND_PATCH_PAC_INTO_SCS -static inline bool should_patch_pac_into_scs(void) -{ - u64 reg; - - /* - * We only enable the shadow call stack dynamically if we are running - * on a system that does not implement PAC or BTI. PAC and SCS provide - * roughly the same level of protection, and BTI relies on the PACIASP - * instructions serving as landing pads, preventing us from patching - * those instructions into something else. - */ - reg = read_sysreg_s(SYS_ID_AA64ISAR1_EL1); - if (SYS_FIELD_GET(ID_AA64ISAR1_EL1, APA, reg) | - SYS_FIELD_GET(ID_AA64ISAR1_EL1, API, reg)) - return false; - - reg = read_sysreg_s(SYS_ID_AA64ISAR2_EL1); - if (SYS_FIELD_GET(ID_AA64ISAR2_EL1, APA3, reg)) - return false; - - if (IS_ENABLED(CONFIG_ARM64_BTI_KERNEL)) { - reg = read_sysreg_s(SYS_ID_AA64PFR1_EL1); - if (reg & (0xf << ID_AA64PFR1_EL1_BT_SHIFT)) - return false; - } - return true; -} - static inline void dynamic_scs_init(void) { - if (should_patch_pac_into_scs()) { + extern bool __pi_dynamic_scs_is_enabled; + + if (__pi_dynamic_scs_is_enabled) { pr_info("Enabling dynamic shadow call stack\n"); static_branch_enable(&dynamic_scs_enabled); } diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S index aa7766dc64d9..ffacce7b5a02 100644 --- a/arch/arm64/kernel/head.S +++ b/arch/arm64/kernel/head.S @@ -81,7 +81,6 @@ * x20 primary_entry() .. __primary_switch() CPU boot mode * x21 primary_entry() .. start_kernel() FDT pointer passed at boot in x0 * x22 create_idmap() .. start_kernel() ID map VA of the DT blob - * x23 __primary_switch() physical misalignment/KASLR offset * x25 primary_entry() .. start_kernel() supported VA size * x28 create_idmap() callee preserved temp register */ @@ -408,24 +407,6 @@ SYM_FUNC_START_LOCAL(create_idmap) 0: ret x28 SYM_FUNC_END(create_idmap) -SYM_FUNC_START_LOCAL(create_kernel_mapping) - adrp x0, init_pg_dir - mov_q x5, KIMAGE_VADDR // compile time __va(_text) -#ifdef CONFIG_RELOCATABLE - add x5, x5, x23 // add KASLR displacement -#endif - adrp x6, _end // runtime __pa(_end) - adrp x3, _text // runtime __pa(_text) - sub x6, x6, x3 // _end - _text - add x6, x6, x5 // runtime __va(_end) - mov_q x7, SWAPPER_RW_MMUFLAGS - - map_memory x0, x1, x5, x6, x7, x3, (VA_BITS - PGDIR_SHIFT), x10, x11, x12, x13, x14 - - dsb ishst // sync with page table walker - ret -SYM_FUNC_END(create_kernel_mapping) - /* * Initialize CPU registers with task-specific and cpu-specific context. * @@ -752,44 +733,13 @@ SYM_FUNC_START_LOCAL(__primary_switch) adrp x2, init_idmap_pg_dir bl __enable_mmu - // Clear BSS - adrp x0, __bss_start - mov x1, xzr - adrp x2, init_pg_end - sub x2, x2, x0 - bl __pi_memset - dsb ishst // Make zero page visible to PTW - adrp x1, early_init_stack mov sp, x1 mov x29, xzr mov x0, x20 // pass the full boot status mov x1, x22 // pass the low FDT mapping - bl __pi_init_feature_override // Parse cpu feature overrides - -#ifdef CONFIG_RELOCATABLE - adrp x23, KERNEL_START - and x23, x23, MIN_KIMG_ALIGN - 1 -#ifdef CONFIG_RANDOMIZE_BASE - mov x0, x22 - bl __pi_kaslr_early_init - bic x0, x0, #SZ_2M - 1 - orr x23, x23, x0 // record kernel offset -#endif -#endif - bl create_kernel_mapping + bl __pi_early_map_kernel // Map and relocate the kernel - adrp x1, init_pg_dir - load_ttbr1 x1, x1, x2 -#ifdef CONFIG_RELOCATABLE - mov x0, x23 - bl __pi_relocate_kernel -#endif -#ifdef CONFIG_UNWIND_PATCH_PAC_INTO_SCS - ldr x0, =__eh_frame_start - ldr x1, =__eh_frame_end - bl __pi_scs_patch_vmlinux -#endif ldr x8, =__primary_switched adrp x0, KERNEL_START // __pa(KERNEL_START) br x8 diff --git a/arch/arm64/kernel/image-vars.h b/arch/arm64/kernel/image-vars.h index 8d96052079e8..e566b32f9c22 100644 --- a/arch/arm64/kernel/image-vars.h +++ b/arch/arm64/kernel/image-vars.h @@ -45,9 +45,28 @@ PROVIDE(__pi_id_aa64pfr1_override = id_aa64pfr1_override); PROVIDE(__pi_id_aa64smfr0_override = id_aa64smfr0_override); PROVIDE(__pi_id_aa64zfr0_override = id_aa64zfr0_override); PROVIDE(__pi_arm64_sw_feature_override = arm64_sw_feature_override); +PROVIDE(__pi_arm64_use_ng_mappings = arm64_use_ng_mappings); +#ifdef CONFIG_CAVIUM_ERRATUM_27456 +PROVIDE(__pi_cavium_erratum_27456_cpus = cavium_erratum_27456_cpus); +#endif PROVIDE(__pi__ctype = _ctype); PROVIDE(__pi_memstart_offset_seed = memstart_offset_seed); +PROVIDE(__pi_init_pg_dir = init_pg_dir); +PROVIDE(__pi_init_pg_end = init_pg_end); + +PROVIDE(__pi__text = _text); +PROVIDE(__pi__stext = _stext); +PROVIDE(__pi__etext = _etext); +PROVIDE(__pi___start_rodata = __start_rodata); +PROVIDE(__pi___inittext_begin = __inittext_begin); +PROVIDE(__pi___inittext_end = __inittext_end); +PROVIDE(__pi___initdata_begin = __initdata_begin); +PROVIDE(__pi___initdata_end = __initdata_end); +PROVIDE(__pi__data = _data); +PROVIDE(__pi___bss_start = __bss_start); +PROVIDE(__pi__end = _end); + #ifdef CONFIG_KVM /* diff --git a/arch/arm64/kernel/pi/Makefile b/arch/arm64/kernel/pi/Makefile index a8b302245f15..8c2f80a46b93 100644 --- a/arch/arm64/kernel/pi/Makefile +++ b/arch/arm64/kernel/pi/Makefile @@ -39,6 +39,7 @@ $(obj)/lib-%.o: $(srctree)/lib/%.c FORCE $(call if_changed_rule,cc_o_c) obj-y := idreg-override.pi.o \ + map_kernel.pi.o map_range.pi.o \ lib-fdt.pi.o lib-fdt_ro.pi.o obj-$(CONFIG_RELOCATABLE) += relocate.pi.o obj-$(CONFIG_RANDOMIZE_BASE) += kaslr_early.pi.o diff --git a/arch/arm64/kernel/pi/idreg-override.c b/arch/arm64/kernel/pi/idreg-override.c index e4bcabcc6860..1884bd936c0d 100644 --- a/arch/arm64/kernel/pi/idreg-override.c +++ b/arch/arm64/kernel/pi/idreg-override.c @@ -308,37 +308,35 @@ static __init void __parse_cmdline(const char *cmdline, bool parse_aliases) } while (1); } -static __init const u8 *get_bootargs_cmdline(const void *fdt) +static __init const u8 *get_bootargs_cmdline(const void *fdt, int node) { + static char const bootargs[] __initconst = "bootargs"; const u8 *prop; - int node; - node = fdt_path_offset(fdt, "/chosen"); if (node < 0) return NULL; - prop = fdt_getprop(fdt, node, "bootargs", NULL); + prop = fdt_getprop(fdt, node, bootargs, NULL); if (!prop) return NULL; return strlen(prop) ? prop : NULL; } -static __init void parse_cmdline(const void *fdt) +static __init void parse_cmdline(const void *fdt, int chosen) { - const u8 *prop = get_bootargs_cmdline(fdt); + static char const cmdline[] __initconst = CONFIG_CMDLINE; + const u8 *prop = get_bootargs_cmdline(fdt, chosen); if (IS_ENABLED(CONFIG_CMDLINE_FORCE) || !prop) - __parse_cmdline(CONFIG_CMDLINE, true); + __parse_cmdline(cmdline, true); if (!IS_ENABLED(CONFIG_CMDLINE_FORCE) && prop) __parse_cmdline(prop, true); } -/* Keep checkers quiet */ -void init_feature_override(u64 boot_status, const void *fdt); - -asmlinkage void __init init_feature_override(u64 boot_status, const void *fdt) +void __init init_feature_override(u64 boot_status, const void *fdt, + int chosen) { struct arm64_ftr_override *override; const struct ftr_set_desc *reg; @@ -354,7 +352,7 @@ asmlinkage void __init init_feature_override(u64 boot_status, const void *fdt) __boot_status = boot_status; - parse_cmdline(fdt); + parse_cmdline(fdt, chosen); for (i = 0; i < ARRAY_SIZE(regs); i++) { reg = prel64_pointer(regs[i].reg); diff --git a/arch/arm64/kernel/pi/kaslr_early.c b/arch/arm64/kernel/pi/kaslr_early.c index eeecee7ffd6f..0257b43819db 100644 --- a/arch/arm64/kernel/pi/kaslr_early.c +++ b/arch/arm64/kernel/pi/kaslr_early.c @@ -16,17 +16,17 @@ #include #include +#include "pi.h" + extern u16 memstart_offset_seed; -static u64 __init get_kaslr_seed(void *fdt) +static u64 __init get_kaslr_seed(void *fdt, int node) { - static char const chosen_str[] __initconst = "chosen"; static char const seed_str[] __initconst = "kaslr-seed"; - int node, len; fdt64_t *prop; u64 ret; + int len; - node = fdt_path_offset(fdt, chosen_str); if (node < 0) return 0; @@ -39,14 +39,14 @@ static u64 __init get_kaslr_seed(void *fdt) return ret; } -asmlinkage u64 __init kaslr_early_init(void *fdt) +u64 __init kaslr_early_init(void *fdt, int chosen) { u64 seed, range; if (kaslr_disabled_cmdline()) return 0; - seed = get_kaslr_seed(fdt); + seed = get_kaslr_seed(fdt, chosen); if (!seed) { if (!__early_cpu_has_rndr() || !__arm64_rndr((unsigned long *)&seed)) diff --git a/arch/arm64/kernel/pi/map_kernel.c b/arch/arm64/kernel/pi/map_kernel.c new file mode 100644 index 000000000000..f206373b28b0 --- /dev/null +++ b/arch/arm64/kernel/pi/map_kernel.c @@ -0,0 +1,164 @@ +// SPDX-License-Identifier: GPL-2.0-only +// Copyright 2023 Google LLC +// Author: Ard Biesheuvel + +#include +#include +#include +#include +#include +#include + +#include +#include +#include +#include + +#include "pi.h" + +extern const u8 __eh_frame_start[], __eh_frame_end[]; + +extern void idmap_cpu_replace_ttbr1(void *pgdir); + +static void __init map_segment(pgd_t *pg_dir, u64 *pgd, u64 va_offset, + void *start, void *end, pgprot_t prot, + bool may_use_cont, int root_level) +{ + map_range(pgd, ((u64)start + va_offset) & ~PAGE_OFFSET, + ((u64)end + va_offset) & ~PAGE_OFFSET, (u64)start, + prot, root_level, (pte_t *)pg_dir, may_use_cont, 0); +} + +static void __init unmap_segment(pgd_t *pg_dir, u64 va_offset, void *start, + void *end, int root_level) +{ + map_segment(pg_dir, NULL, va_offset, start, end, __pgprot(0), + false, root_level); +} + +static void __init map_kernel(u64 kaslr_offset, u64 va_offset, int root_level) +{ + bool enable_scs = IS_ENABLED(CONFIG_UNWIND_PATCH_PAC_INTO_SCS); + bool twopass = IS_ENABLED(CONFIG_RELOCATABLE); + u64 pgdp = (u64)init_pg_dir + PAGE_SIZE; + pgprot_t text_prot = PAGE_KERNEL_ROX; + pgprot_t data_prot = PAGE_KERNEL; + pgprot_t prot; + + /* + * External debuggers may need to write directly to the text mapping to + * install SW breakpoints. Allow this (only) when explicitly requested + * with rodata=off. + */ + if (arm64_test_sw_feature_override(ARM64_SW_FEATURE_OVERRIDE_RODATA_OFF)) + text_prot = PAGE_KERNEL_EXEC; + + /* + * We only enable the shadow call stack dynamically if we are running + * on a system that does not implement PAC or BTI. PAC and SCS provide + * roughly the same level of protection, and BTI relies on the PACIASP + * instructions serving as landing pads, preventing us from patching + * those instructions into something else. + */ + if (IS_ENABLED(CONFIG_ARM64_PTR_AUTH_KERNEL) && cpu_has_pac()) + enable_scs = false; + + if (IS_ENABLED(CONFIG_ARM64_BTI_KERNEL) && cpu_has_bti()) { + enable_scs = false; + + /* + * If we have a CPU that supports BTI and a kernel built for + * BTI then mark the kernel executable text as guarded pages + * now so we don't have to rewrite the page tables later. + */ + text_prot = __pgprot_modify(text_prot, PTE_GP, PTE_GP); + } + + /* Map all code read-write on the first pass if needed */ + twopass |= enable_scs; + prot = twopass ? data_prot : text_prot; + + map_segment(init_pg_dir, &pgdp, va_offset, _stext, _etext, prot, + !twopass, root_level); + map_segment(init_pg_dir, &pgdp, va_offset, __start_rodata, + __inittext_begin, data_prot, false, root_level); + map_segment(init_pg_dir, &pgdp, va_offset, __inittext_begin, + __inittext_end, prot, false, root_level); + map_segment(init_pg_dir, &pgdp, va_offset, __initdata_begin, + __initdata_end, data_prot, false, root_level); + map_segment(init_pg_dir, &pgdp, va_offset, _data, _end, data_prot, + true, root_level); + dsb(ishst); + + idmap_cpu_replace_ttbr1(init_pg_dir); + + if (twopass) { + if (IS_ENABLED(CONFIG_RELOCATABLE)) + relocate_kernel(kaslr_offset); + + if (enable_scs) { + scs_patch(__eh_frame_start + va_offset, + __eh_frame_end - __eh_frame_start); + asm("ic ialluis"); + + dynamic_scs_is_enabled = true; + } + + /* + * Unmap the text region before remapping it, to avoid + * potential TLB conflicts when creating the contiguous + * descriptors. + */ + unmap_segment(init_pg_dir, va_offset, _stext, _etext, + root_level); + dsb(ishst); + isb(); + __tlbi(vmalle1); + isb(); + + /* + * Remap these segments with different permissions + * No new page table allocations should be needed + */ + map_segment(init_pg_dir, NULL, va_offset, _stext, _etext, + text_prot, true, root_level); + map_segment(init_pg_dir, NULL, va_offset, __inittext_begin, + __inittext_end, text_prot, false, root_level); + dsb(ishst); + } +} + +asmlinkage void __init early_map_kernel(u64 boot_status, void *fdt) +{ + static char const chosen_str[] __initconst = "/chosen"; + u64 va_base, pa_base = (u64)&_text; + u64 kaslr_offset = pa_base % MIN_KIMG_ALIGN; + int root_level = 4 - CONFIG_PGTABLE_LEVELS; + int chosen; + + /* Clear BSS and the initial page tables */ + memset(__bss_start, 0, (u64)init_pg_end - (u64)__bss_start); + + /* Parse the command line for CPU feature overrides */ + chosen = fdt_path_offset(fdt, chosen_str); + init_feature_override(boot_status, fdt, chosen); + + /* + * The virtual KASLR displacement modulo 2MiB is decided by the + * physical placement of the image, as otherwise, we might not be able + * to create the early kernel mapping using 2 MiB block descriptors. So + * take the low bits of the KASLR offset from the physical address, and + * fill in the high bits from the seed. + */ + if (IS_ENABLED(CONFIG_RANDOMIZE_BASE)) { + u64 kaslr_seed = kaslr_early_init(fdt, chosen); + + if (kaslr_seed && kaslr_requires_kpti()) + arm64_use_ng_mappings = true; + + kaslr_offset |= kaslr_seed & ~(MIN_KIMG_ALIGN - 1); + } + + va_base = KIMAGE_VADDR + kaslr_offset; + map_kernel(kaslr_offset, va_base - pa_base, root_level); +} diff --git a/arch/arm64/kernel/pi/map_range.c b/arch/arm64/kernel/pi/map_range.c new file mode 100644 index 000000000000..c31feda18f47 --- /dev/null +++ b/arch/arm64/kernel/pi/map_range.c @@ -0,0 +1,88 @@ +// SPDX-License-Identifier: GPL-2.0-only +// Copyright 2023 Google LLC +// Author: Ard Biesheuvel + +#include +#include + +#include +#include +#include + +#include "pi.h" + +/** + * map_range - Map a contiguous range of physical pages into virtual memory + * + * @pte: Address of physical pointer to array of pages to + * allocate page tables from + * @start: Virtual address of the start of the range + * @end: Virtual address of the end of the range (exclusive) + * @pa: Physical address of the start of the range + * @prot: Access permissions of the range + * @level: Translation level for the mapping + * @tbl: The level @level page table to create the mappings in + * @may_use_cont: Whether the use of the contiguous attribute is allowed + * @va_offset: Offset between a physical page and its current mapping + * in the VA space + */ +void __init map_range(u64 *pte, u64 start, u64 end, u64 pa, pgprot_t prot, + int level, pte_t *tbl, bool may_use_cont, u64 va_offset) +{ + u64 cmask = (level == 3) ? CONT_PTE_SIZE - 1 : U64_MAX; + u64 protval = pgprot_val(prot) & ~PTE_TYPE_MASK; + int lshift = (3 - level) * (PAGE_SHIFT - 3); + u64 lmask = (PAGE_SIZE << lshift) - 1; + + start &= PAGE_MASK; + pa &= PAGE_MASK; + + /* Advance tbl to the entry that covers start */ + tbl += (start >> (lshift + PAGE_SHIFT)) % PTRS_PER_PTE; + + /* + * Set the right block/page bits for this level unless we are + * clearing the mapping + */ + if (protval) + protval |= (level < 3) ? PMD_TYPE_SECT : PTE_TYPE_PAGE; + + while (start < end) { + u64 next = min((start | lmask) + 1, PAGE_ALIGN(end)); + + if (level < 3 && (start | next | pa) & lmask) { + /* + * This chunk needs a finer grained mapping. Create a + * table mapping if necessary and recurse. + */ + if (pte_none(*tbl)) { + *tbl = __pte(__phys_to_pte_val(*pte) | + PMD_TYPE_TABLE | PMD_TABLE_UXN); + *pte += PTRS_PER_PTE * sizeof(pte_t); + } + map_range(pte, start, next, pa, prot, level + 1, + (pte_t *)(__pte_to_phys(*tbl) + va_offset), + may_use_cont, va_offset); + } else { + /* + * Start a contiguous range if start and pa are + * suitably aligned + */ + if (((start | pa) & cmask) == 0 && may_use_cont) + protval |= PTE_CONT; + + /* + * Clear the contiguous attribute if the remaining + * range does not cover a contiguous block + */ + if ((end & ~cmask) <= start) + protval &= ~PTE_CONT; + + /* Put down a block or page mapping */ + *tbl = __pte(__phys_to_pte_val(pa) | protval); + } + pa += next - start; + start = next; + tbl++; + } +} diff --git a/arch/arm64/kernel/pi/patch-scs.c b/arch/arm64/kernel/pi/patch-scs.c index c65ef40d1e6b..49d8b40e61bc 100644 --- a/arch/arm64/kernel/pi/patch-scs.c +++ b/arch/arm64/kernel/pi/patch-scs.c @@ -11,6 +11,10 @@ #include +#include "pi.h" + +bool dynamic_scs_is_enabled; + // // This minimal DWARF CFI parser is partially based on the code in // arch/arc/kernel/unwind.c, and on the document below: @@ -46,8 +50,6 @@ #define DW_CFA_GNU_negative_offset_extended 0x2f #define DW_CFA_hi_user 0x3f -extern const u8 __eh_frame_start[], __eh_frame_end[]; - enum { PACIASP = 0xd503233f, AUTIASP = 0xd50323bf, @@ -250,13 +252,3 @@ int scs_patch(const u8 eh_frame[], int size) } return 0; } - -asmlinkage void __init scs_patch_vmlinux(const u8 start[], const u8 end[]) -{ - if (!should_patch_pac_into_scs()) - return; - - scs_patch(start, end - start); - asm("ic ialluis"); - isb(); -} diff --git a/arch/arm64/kernel/pi/pi.h b/arch/arm64/kernel/pi/pi.h index 7c2d9bbf0ff9..d307c58e9741 100644 --- a/arch/arm64/kernel/pi/pi.h +++ b/arch/arm64/kernel/pi/pi.h @@ -2,6 +2,8 @@ // Copyright 2023 Google LLC // Author: Ard Biesheuvel +#include + #define __prel64_initconst __section(".init.rodata.prel64") #define PREL64(type, name) union { type *name; prel64_t name ## _prel; } @@ -16,3 +18,15 @@ static inline void *prel64_to_pointer(const prel64_t *offset) return NULL; return (void *)offset + *offset; } + +extern bool dynamic_scs_is_enabled; + +void init_feature_override(u64 boot_status, const void *fdt, int chosen); +u64 kaslr_early_init(void *fdt, int chosen); +void relocate_kernel(u64 offset); +int scs_patch(const u8 eh_frame[], int size); + +void map_range(u64 *pgd, u64 start, u64 end, u64 pa, pgprot_t prot, + int level, pte_t *tbl, bool may_use_cont, u64 va_offset); + +asmlinkage void early_map_kernel(u64 boot_status, void *fdt); diff --git a/arch/arm64/kernel/pi/relocate.c b/arch/arm64/kernel/pi/relocate.c index 1853408ea76b..2407d2696398 100644 --- a/arch/arm64/kernel/pi/relocate.c +++ b/arch/arm64/kernel/pi/relocate.c @@ -7,6 +7,8 @@ #include #include +#include "pi.h" + extern const Elf64_Rela rela_start[], rela_end[]; extern const u64 relr_start[], relr_end[]; diff --git a/arch/arm64/kernel/setup.c b/arch/arm64/kernel/setup.c index 0ef45d1927b3..0ea45b6d0177 100644 --- a/arch/arm64/kernel/setup.c +++ b/arch/arm64/kernel/setup.c @@ -283,13 +283,6 @@ void __init __no_sanitize_address setup_arch(char **cmdline_p) kaslr_init(); - /* - * If know now we are going to need KPTI then use non-global - * mappings from the start, avoiding the cost of rewriting - * everything later. - */ - arm64_use_ng_mappings = kaslr_enabled() && kaslr_requires_kpti(); - early_fixmap_init(); early_ioremap_init(); diff --git a/arch/arm64/kernel/vmlinux.lds.S b/arch/arm64/kernel/vmlinux.lds.S index 3afb4223a5e8..755a22d4f840 100644 --- a/arch/arm64/kernel/vmlinux.lds.S +++ b/arch/arm64/kernel/vmlinux.lds.S @@ -126,9 +126,9 @@ jiffies = jiffies_64; #ifdef CONFIG_UNWIND_TABLES #define UNWIND_DATA_SECTIONS \ .eh_frame : { \ - __eh_frame_start = .; \ + __pi___eh_frame_start = .; \ *(.eh_frame) \ - __eh_frame_end = .; \ + __pi___eh_frame_end = .; \ } #else #define UNWIND_DATA_SECTIONS diff --git a/arch/arm64/mm/proc.S b/arch/arm64/mm/proc.S index f66c37a1610e..7c1bdaf25408 100644 --- a/arch/arm64/mm/proc.S +++ b/arch/arm64/mm/proc.S @@ -195,6 +195,7 @@ SYM_TYPED_FUNC_START(idmap_cpu_replace_ttbr1) ret SYM_FUNC_END(idmap_cpu_replace_ttbr1) +SYM_FUNC_ALIAS(__pi_idmap_cpu_replace_ttbr1, idmap_cpu_replace_ttbr1) .popsection #ifdef CONFIG_UNMAP_KERNEL_AT_EL0 From patchwork Wed Feb 14 12:29:05 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13556414 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 3AAA6C48BC3 for ; Wed, 14 Feb 2024 12:32:18 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=PDHvYjIsHEfeCLwFLPtkXxTZuYETwCa1ECn3oS0Pjjs=; b=x0YKGrLiDkQNHOwIRnG50t13Td qgjoKf3J33zPyRUZ0LVxHLzblom+rt8IjuOo7EXL3pJgpqITiy1tNkG9A5JlKU4AtsyKPxnfI9RCf dCxelC3C+G7nhP5IjQ7GWOlBhbZtvtu9+7IKfoQjPNFjIY8b329ZUfPg89y9uNODx3Y7q+7Bgq52K tV2Ej+SeQsdt5htneKpPx4IbeQonzyYeiw9cQomAmBkOXwmGnIYz8rRdmevOXW84MHIJK4L7PAZy7 kYoBxMhl6EQ747z1Bq93xP2vSUxYoTbPg9m5YWYtL/TZIAIj2NqK7dhn/dEZFoHmq+pWZgdZ1wxjS JUSI1ATg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1raEQm-0000000CpN6-2w1j; Wed, 14 Feb 2024 12:32:04 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1raEPL-0000000CoiD-15f2 for linux-arm-kernel@bombadil.infradead.org; Wed, 14 Feb 2024 12:30:35 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=VvnB0e9tOYh+zNHys77cOZeL4hzuiQZQpjfGT9yM30s=; b=lEXbZVFufVZRqwSRd1FOWrHi0r QHsG7W4hR91lOT/iNMlRCBRoylcit0HDiLEXUhiVR0DvtkWLRkoEXz+G6jN2QBqKnRqNOxQP9eC45 VLIP6Pt1JZhxTenbbOutmv9JtzbHOhZHwD8LUYd/R4l6fFIiK6Dpd+hqWxZDuNe7iIkJNmbue+jUw V+ZGIGM22CWMY/in1ssmxWYqB/Rd/YE9BN0Y4nFLuV2IyYTpOaWcfuZmSc1BCEte6EIFnkeEivjnG sOtq2xGYM6WbyOkzxhYAkRRsZOqqg9/pWIF2wCPkn6rsSUBy1tWIjv7FOBjEo/AiMyVJ7dh4wdEL9 +JMzO63w==; Received: from mail-wm1-x349.google.com ([2a00:1450:4864:20::349]) by desiato.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1raEPD-0000000GDzv-0WJU for linux-arm-kernel@lists.infradead.org; Wed, 14 Feb 2024 12:30:33 +0000 Received: by mail-wm1-x349.google.com with SMTP id 5b1f17b1804b1-411a8c8d9b2so13131515e9.0 for ; Wed, 14 Feb 2024 04:30:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1707913822; x=1708518622; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=VvnB0e9tOYh+zNHys77cOZeL4hzuiQZQpjfGT9yM30s=; b=vHH+tdqY+aHl1hsc0mYvTMtohoe1FhRCyJ1mas5bvrGCWEmKSF6MlBqTKby84idLnn 61iqx0UIkv0MFgiZY3o/tfDMIi+gOMvZrxhg1z1VqpZP5CMKub/8ENmCqYlB8TKQEXGr feIXU64UeMfQmN71SggkqZjO3Kdd9E3YtlmGJm/1wv4GA67PqqVEC3k5RcdT9kM2paMA eClz/yG+GrJDqKQiekLKnAf0CJcYlp9QluuWXc6I/ndwrB1OH/DXsmNzLSji+zRlNebf jsMSKy0CR7L7rOHXVZAdLBz3m+K1op7hghJmM9dX2tzw73dsDNbSiQyORCXzQvCr+VJW I/HA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707913822; x=1708518622; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=VvnB0e9tOYh+zNHys77cOZeL4hzuiQZQpjfGT9yM30s=; b=N1YUVTFuNdcGmVzURwDBgePU1EsfoiFWr9wNXJxV/xIj5gWMZMMmtqqwXejJmidTpG g90zeN5RqWuqidvMmQVp/0e/katW1+b7K8PSy5rU/Ej0TH/SWTJraS9vJjLbEl34Z19D iTyVk+ENsWEn9YOiE5sHvMpbZYEX3RMSFejMSEtHtDbqK13d6QU3mXb06K1tn9z/NLSj YZWmmWzJBFFGLThazoplZwDuaYHDNMDWkakTFxvmaWAStekVb+5UcqgAWVknvAQc2cw5 /uSxECNNHRCdnxqYcI4xw9EgG8EMHNLLJkPTgNAMMUTdkv5G1wOcx4p5MDw//QLrLz4j bwtw== X-Gm-Message-State: AOJu0YzIO9cFKsEJeX0CnN7obJtMcstZEhTol6kiar5YjCdzdaRzNeJr qefGApARKpicU7uasdUzqioNZvlUN2BOcXnda5ns/LO3l5fuzD3DsiRp/f2difeki00VZnDxVeL T7pI2/OUF7lPqrQqb6Qw/IoR+1XHlhBxgVa8ws16iJ5uxmh0XJYvtymywop0jG0wmsjuhClwYwM Ih29Z5tthoEylPQNiHuLGZP2Ubb3/IVef0dFk2F7A0 X-Google-Smtp-Source: AGHT+IEQ4DMW3korYTIzzOGk0EMaA7cc/vwsJlBA+E1ZOuSN2Owp8FEj7KvA57fRsXB+pvmsmkkcYxTo X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:6000:1f96:b0:33c:deec:4ad7 with SMTP id bw22-20020a0560001f9600b0033cdeec4ad7mr7560wrb.7.1707913822232; Wed, 14 Feb 2024 04:30:22 -0800 (PST) Date: Wed, 14 Feb 2024 13:29:05 +0100 In-Reply-To: <20240214122845.2033971-45-ardb+git@google.com> Mime-Version: 1.0 References: <20240214122845.2033971-45-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=7010; i=ardb@kernel.org; h=from:subject; bh=K6dlk+khRRcpqbypZSsft3meiNcDy9eMVYLffSD3B8c=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIfXMJqHu/GNz4rYyzpDjO/BFz4nrbFgdU0yPgs2OVTuEp z2X4XzUUcrCIMbBICumyCIw+++7nacnStU6z5KFmcPKBDKEgYtTACaS9Z3hr5iVcJ/4yhr1OQlx xxTu7m/SXXp9UtHNliP/+uP8cz+d38Hwv5K9Mlf415l7rKzxp3cfzH5gsfmYsOuLhXkiqrsOSuZ uYgQA X-Mailer: git-send-email 2.43.0.687.g38aa6559b0-goog Message-ID: <20240214122845.2033971-64-ardb+git@google.com> Subject: [PATCH v8 19/43] arm64: mm: Use 48-bit virtual addressing for the permanent ID map From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240214_123027_835891_FFF94BF3 X-CRM114-Status: GOOD ( 24.55 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel Even though we support loading kernels anywhere in 48-bit addressable physical memory, we create the ID maps based on the number of levels that we happened to configure for the kernel VA and user VA spaces. The reason for this is that the PGD/PUD/PMD based classification of translation levels, along with the associated folding when the number of levels is less than 5, does not permit creating a page table hierarchy of a set number of levels. This means that, for instance, on 39-bit VA kernels we need to configure an additional level above PGD level on the fly, and 36-bit VA kernels still only support 47-bit virtual addressing with this trick applied. Now that we have a separate helper to populate page table hierarchies that does not define the levels in terms of PUDS/PMDS/etc at all, let's reuse it to create the permanent ID map with a fixed VA size of 48 bits. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/kernel-pgtable.h | 3 ++ arch/arm64/kernel/head.S | 5 +++ arch/arm64/kvm/mmu.c | 15 +++------ arch/arm64/mm/mmu.c | 32 +++++++++++--------- arch/arm64/mm/proc.S | 9 ++---- 5 files changed, 32 insertions(+), 32 deletions(-) diff --git a/arch/arm64/include/asm/kernel-pgtable.h b/arch/arm64/include/asm/kernel-pgtable.h index 0631604995ee..742a4b2778f7 100644 --- a/arch/arm64/include/asm/kernel-pgtable.h +++ b/arch/arm64/include/asm/kernel-pgtable.h @@ -35,6 +35,9 @@ #define SWAPPER_PGTABLE_LEVELS (CONFIG_PGTABLE_LEVELS) #endif +#define IDMAP_VA_BITS 48 +#define IDMAP_LEVELS ARM64_HW_PGTABLE_LEVELS(IDMAP_VA_BITS) +#define IDMAP_ROOT_LEVEL (4 - IDMAP_LEVELS) /* * A relocatable kernel may execute from an address that differs from the one at diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S index ffacce7b5a02..a1c29d64e875 100644 --- a/arch/arm64/kernel/head.S +++ b/arch/arm64/kernel/head.S @@ -729,6 +729,11 @@ SYM_FUNC_START_LOCAL(__no_granule_support) SYM_FUNC_END(__no_granule_support) SYM_FUNC_START_LOCAL(__primary_switch) + mrs x1, tcr_el1 + mov x2, #64 - VA_BITS + tcr_set_t0sz x1, x2 + msr tcr_el1, x1 + adrp x1, reserved_pg_dir adrp x2, init_idmap_pg_dir bl __enable_mmu diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c index d14504821b79..6fa9e816df40 100644 --- a/arch/arm64/kvm/mmu.c +++ b/arch/arm64/kvm/mmu.c @@ -1874,16 +1874,9 @@ int __init kvm_mmu_init(u32 *hyp_va_bits) BUG_ON((hyp_idmap_start ^ (hyp_idmap_end - 1)) & PAGE_MASK); /* - * The ID map may be configured to use an extended virtual address - * range. This is only the case if system RAM is out of range for the - * currently configured page size and VA_BITS_MIN, in which case we will - * also need the extended virtual range for the HYP ID map, or we won't - * be able to enable the EL2 MMU. - * - * However, in some cases the ID map may be configured for fewer than - * the number of VA bits used by the regular kernel stage 1. This - * happens when VA_BITS=52 and the kernel image is placed in PA space - * below 48 bits. + * The ID map is always configured for 48 bits of translation, which + * may be fewer than the number of VA bits used by the regular kernel + * stage 1, when VA_BITS=52. * * At EL2, there is only one TTBR register, and we can't switch between * translation tables *and* update TCR_EL2.T0SZ at the same time. Bottom @@ -1894,7 +1887,7 @@ int __init kvm_mmu_init(u32 *hyp_va_bits) * 1 VA bits to assure that the hypervisor can both ID map its code page * and map any kernel memory. */ - idmap_bits = 64 - ((idmap_t0sz & TCR_T0SZ_MASK) >> TCR_T0SZ_OFFSET); + idmap_bits = IDMAP_VA_BITS; kernel_bits = vabits_actual; *hyp_va_bits = max(idmap_bits, kernel_bits); diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index f9332eea318f..a991f195592b 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -757,22 +757,21 @@ static void __init map_kernel(pgd_t *pgdp) kasan_copy_shadow(pgdp); } +void __pi_map_range(u64 *pgd, u64 start, u64 end, u64 pa, pgprot_t prot, + int level, pte_t *tbl, bool may_use_cont, u64 va_offset); + +static u8 idmap_ptes[IDMAP_LEVELS - 1][PAGE_SIZE] __aligned(PAGE_SIZE) __ro_after_init, + kpti_ptes[IDMAP_LEVELS - 1][PAGE_SIZE] __aligned(PAGE_SIZE) __ro_after_init; + static void __init create_idmap(void) { u64 start = __pa_symbol(__idmap_text_start); - u64 size = __pa_symbol(__idmap_text_end) - start; - pgd_t *pgd = idmap_pg_dir; - u64 pgd_phys; - - /* check if we need an additional level of translation */ - if (VA_BITS < 48 && idmap_t0sz < (64 - VA_BITS_MIN)) { - pgd_phys = early_pgtable_alloc(PAGE_SHIFT); - set_pgd(&idmap_pg_dir[start >> VA_BITS], - __pgd(pgd_phys | P4D_TYPE_TABLE)); - pgd = __va(pgd_phys); - } - __create_pgd_mapping(pgd, start, start, size, PAGE_KERNEL_ROX, - early_pgtable_alloc, 0); + u64 end = __pa_symbol(__idmap_text_end); + u64 ptep = __pa_symbol(idmap_ptes); + + __pi_map_range(&ptep, start, end, start, PAGE_KERNEL_ROX, + IDMAP_ROOT_LEVEL, (pte_t *)idmap_pg_dir, false, + __phys_to_virt(ptep) - ptep); if (IS_ENABLED(CONFIG_UNMAP_KERNEL_AT_EL0)) { extern u32 __idmap_kpti_flag; @@ -782,8 +781,10 @@ static void __init create_idmap(void) * The KPTI G-to-nG conversion code needs a read-write mapping * of its synchronization flag in the ID map. */ - __create_pgd_mapping(pgd, pa, pa, sizeof(u32), PAGE_KERNEL, - early_pgtable_alloc, 0); + ptep = __pa_symbol(kpti_ptes); + __pi_map_range(&ptep, pa, pa + sizeof(u32), pa, PAGE_KERNEL, + IDMAP_ROOT_LEVEL, (pte_t *)idmap_pg_dir, false, + __phys_to_virt(ptep) - ptep); } } @@ -808,6 +809,7 @@ void __init paging_init(void) memblock_allow_resize(); create_idmap(); + idmap_t0sz = TCR_T0SZ(IDMAP_VA_BITS); } #ifdef CONFIG_MEMORY_HOTPLUG diff --git a/arch/arm64/mm/proc.S b/arch/arm64/mm/proc.S index 7c1bdaf25408..47ede52bb900 100644 --- a/arch/arm64/mm/proc.S +++ b/arch/arm64/mm/proc.S @@ -421,9 +421,9 @@ SYM_FUNC_START(__cpu_setup) mair .req x17 tcr .req x16 mov_q mair, MAIR_EL1_SET - mov_q tcr, TCR_TxSZ(VA_BITS) | TCR_CACHE_FLAGS | TCR_SMP_FLAGS | \ - TCR_TG_FLAGS | TCR_KASLR_FLAGS | TCR_ASID16 | \ - TCR_TBI0 | TCR_A1 | TCR_KASAN_SW_FLAGS | TCR_MTE_FLAGS + mov_q tcr, TCR_T0SZ(IDMAP_VA_BITS) | TCR_T1SZ(VA_BITS) | TCR_CACHE_FLAGS | \ + TCR_SMP_FLAGS | TCR_TG_FLAGS | TCR_KASLR_FLAGS | TCR_ASID16 | \ + TCR_TBI0 | TCR_A1 | TCR_KASAN_SW_FLAGS | TCR_MTE_FLAGS tcr_clear_errata_bits tcr, x9, x5 @@ -431,10 +431,7 @@ SYM_FUNC_START(__cpu_setup) sub x9, xzr, x0 add x9, x9, #64 tcr_set_t1sz tcr, x9 -#else - idmap_get_t0sz x9 #endif - tcr_set_t0sz tcr, x9 /* * Set the IPS bits in TCR_EL1. From patchwork Wed Feb 14 12:29:06 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13556426 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id DAB8AC48BC3 for ; Wed, 14 Feb 2024 12:32:43 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=tTCudcbGEMFtk3xiguU/DN3JC00G1XjoKMND/GWpwi4=; b=yNDFB5tJruO5/gVpDpj99QxhkK VI92Da/RnOjEEp6qH3uqwO29l+ajwsDAOZhtjzsKYJxDeycpjs74PEoFGaUAS8WJGlKbUNslJe+QY 6NonHdxVi1ynNXg/N9iq7Fb8CbOs9BfZrIRsGI1J+z/g35RPwx4CKskHNUdKUwmNPazkSk68ui5gw IvNMWF2SkHx1J69UFSvkWJog5Zl5QVQrc3xtekHO0yczPz3dBjgo+lcvLNlLMRSCip4lz2IlSwwjt VIVsn+DTdXFuxE2nXmBKUakPLXFbDItKm0KNNEoYupFRJcaQuNkHbKqXTba5VpXaFcZhXxZmwEE+u WidDiRVQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1raER8-0000000CpcQ-43Af; Wed, 14 Feb 2024 12:32:27 +0000 Received: from mail-wr1-x449.google.com ([2a00:1450:4864:20::449]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1raEPC-0000000CoaZ-2w4L for linux-arm-kernel@lists.infradead.org; Wed, 14 Feb 2024 12:30:55 +0000 Received: by mail-wr1-x449.google.com with SMTP id ffacd0b85a97d-33cec911abeso223870f8f.1 for ; Wed, 14 Feb 2024 04:30:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1707913824; x=1708518624; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=rHzLXnWMp/qb9CcigctaSxAY3LCDMp8XlWZr+kP58CI=; b=IYHdU/r6RfPS5bMfJ7vC/KEn9J6f1llRglzn3YKbQSvABRe/aCJyRPd+5Py4GNB7Al 313t2CBlw1UWtq08dtiz7yXUdjQ7THHVleQGrzBZq/gzsxHBtcNvQzYrs9jm96KzLsNj H4gM7PzU2JpswD+IchHEW2RCBD7xp1efRhiFhGN4SiiE5Er9mwxvwS3UP3AsZq2If8Ti eFLgp0z81EH6Bp+SH7rL//q95fVmS50sVYb11FN/BkdXVzmayQrdDDlrfWxtwZ5Utr3Y b0vS1HS1qfECfpXzrZJoKJ1nipLzWvupjGUmQu5HR7MDDmomSh4yoBOtBuC/HFuelBBt RHcQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707913824; x=1708518624; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=rHzLXnWMp/qb9CcigctaSxAY3LCDMp8XlWZr+kP58CI=; b=RZueRBLuxQDzeOc3hOnp4lKXrV4zE0C493eW7sJCZVcmFGJ6NRfMOI79jElVS8MnxV RjsPkJyiV4Vas4syb8YXO/DtZEZpK5dda2kWDzVnmPx+r4BDW38yROJDn4EsDfmKxOxp Rl5vSjEfee12836xDK0NoGEQtoDhuLbLNlU6XTFn5KSpUqmgR+rlQTjRgBoRxhXTJM1C qa9DW/JTDK5yjfQBowvrpIeBlD9yAOCQVLN3K/pcX2mnMG01t0XTkJ8qa626BzA3aKVT BKvTtJHHaKNlNLzyxbeKJRYgsJdE4rksi3JwB/XDba4Q3oTG7GlfAhe5hPuhBVYIupnI ZE6w== X-Gm-Message-State: AOJu0YwcspbhlbjEibsm9hZYsFTGNlOaNhAO6h+XFtkOSNG1CE8Hq7uI vKrJ5L/dFJFW9xlTDGaASxMRRxgN7CxOXFuA8HUU7OslaetHcleSBufbCiNrjhqrR4kiD4px7Iq /nK7It1P7y6O6Zc3mcHcn70xouHlyJOvcnJ8Lq9wSwj7z+0cTgEFprO/tgrqmx5lPkf1njQ7UWq Xh9356wm82AH1UudS/zivoQ9evCyWpTqGJRxj0aAW7 X-Google-Smtp-Source: AGHT+IGwIe3TfoPZ9LcNTq3vHt7jrxSYss7fxn7TyTNi5gPyP3/4y4rk9Myje4KgYAFM7hAi7zUiipYo X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:adf:9d92:0:b0:33c:da83:5332 with SMTP id p18-20020adf9d92000000b0033cda835332mr3410wre.0.1707913824354; Wed, 14 Feb 2024 04:30:24 -0800 (PST) Date: Wed, 14 Feb 2024 13:29:06 +0100 In-Reply-To: <20240214122845.2033971-45-ardb+git@google.com> Mime-Version: 1.0 References: <20240214122845.2033971-45-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=5168; i=ardb@kernel.org; h=from:subject; bh=TV/DizsBLERKZQHsn8/eWrcp5wXP/p2+PhQIr/g3L9Y=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIfXMJpHnswt4Ojub/D9v/PjpYafE5hzedY0Wj6pmXKmXm Txz9tWHHaUsDGIcDLJiiiwCs/++23l6olSt8yxZmDmsTCBDGLg4BWAiPTMZ/ieu7kgqYKnuvMqQ wLl14rYEubYLYdLZxdm7y9IfXpRYc5Xhv6OlyvvrKiJrvAVCVcwbtrNNZdbP87zZKqdVeVeUySK AEwA= X-Mailer: git-send-email 2.43.0.687.g38aa6559b0-goog Message-ID: <20240214122845.2033971-65-ardb+git@google.com> Subject: [PATCH v8 20/43] arm64: pgtable: Decouple PGDIR size macros from PGD/PUD/PMD levels From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240214_043027_175738_83E03EDD X-CRM114-Status: GOOD ( 18.98 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel The mapping from PGD/PUD/PMD to levels and shifts is very confusing, given that, due to folding, the shifts may be equal for different levels, if the macros are even #define'd to begin with. In a subsequent patch, we will modify the ID mapping code to decouple the number of levels from the kernel's view of how these types are folded, so prepare for this by reformulating the macros without the use of these types. Instead, use SWAPPER_BLOCK_SHIFT as the base quantity, and derive it from either PAGE_SHIFT or PMD_SHIFT, which -if defined at all- are defined unambiguously for a given page size, regardless of the number of configured levels. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/kernel-pgtable.h | 65 ++++++-------------- 1 file changed, 19 insertions(+), 46 deletions(-) diff --git a/arch/arm64/include/asm/kernel-pgtable.h b/arch/arm64/include/asm/kernel-pgtable.h index 742a4b2778f7..f1fc98a233d5 100644 --- a/arch/arm64/include/asm/kernel-pgtable.h +++ b/arch/arm64/include/asm/kernel-pgtable.h @@ -13,27 +13,22 @@ #include /* - * The linear mapping and the start of memory are both 2M aligned (per - * the arm64 booting.txt requirements). Hence we can use section mapping - * with 4K (section size = 2M) but not with 16K (section size = 32M) or - * 64K (section size = 512M). + * The physical and virtual addresses of the start of the kernel image are + * equal modulo 2 MiB (per the arm64 booting.txt requirements). Hence we can + * use section mapping with 4K (section size = 2M) but not with 16K (section + * size = 32M) or 64K (section size = 512M). */ - -/* - * The idmap and swapper page tables need some space reserved in the kernel - * image. Both require pgd, pud (4 levels only) and pmd tables to (section) - * map the kernel. With the 64K page configuration, swapper and idmap need to - * map to pte level. The swapper also maps the FDT (see __create_page_tables - * for more information). Note that the number of ID map translation levels - * could be increased on the fly if system RAM is out of reach for the default - * VA range, so pages required to map highest possible PA are reserved in all - * cases. - */ -#ifdef CONFIG_ARM64_4K_PAGES -#define SWAPPER_PGTABLE_LEVELS (CONFIG_PGTABLE_LEVELS - 1) +#if defined(PMD_SIZE) && PMD_SIZE <= MIN_KIMG_ALIGN +#define SWAPPER_BLOCK_SHIFT PMD_SHIFT +#define SWAPPER_SKIP_LEVEL 1 #else -#define SWAPPER_PGTABLE_LEVELS (CONFIG_PGTABLE_LEVELS) +#define SWAPPER_BLOCK_SHIFT PAGE_SHIFT +#define SWAPPER_SKIP_LEVEL 0 #endif +#define SWAPPER_BLOCK_SIZE (UL(1) << SWAPPER_BLOCK_SHIFT) +#define SWAPPER_TABLE_SHIFT (SWAPPER_BLOCK_SHIFT + PAGE_SHIFT - 3) + +#define SWAPPER_PGTABLE_LEVELS (CONFIG_PGTABLE_LEVELS - SWAPPER_SKIP_LEVEL) #define IDMAP_VA_BITS 48 #define IDMAP_LEVELS ARM64_HW_PGTABLE_LEVELS(IDMAP_VA_BITS) @@ -53,24 +48,13 @@ #define EARLY_ENTRIES(vstart, vend, shift, add) \ (SPAN_NR_ENTRIES(vstart, vend, shift) + (add)) -#define EARLY_PGDS(vstart, vend, add) (EARLY_ENTRIES(vstart, vend, PGDIR_SHIFT, add)) - -#if SWAPPER_PGTABLE_LEVELS > 3 -#define EARLY_PUDS(vstart, vend, add) (EARLY_ENTRIES(vstart, vend, PUD_SHIFT, add)) -#else -#define EARLY_PUDS(vstart, vend, add) (0) -#endif +#define EARLY_LEVEL(lvl, vstart, vend, add) \ + (SWAPPER_PGTABLE_LEVELS > lvl ? EARLY_ENTRIES(vstart, vend, SWAPPER_BLOCK_SHIFT + lvl * (PAGE_SHIFT - 3), add) : 0) -#if SWAPPER_PGTABLE_LEVELS > 2 -#define EARLY_PMDS(vstart, vend, add) (EARLY_ENTRIES(vstart, vend, SWAPPER_TABLE_SHIFT, add)) -#else -#define EARLY_PMDS(vstart, vend, add) (0) -#endif - -#define EARLY_PAGES(vstart, vend, add) ( 1 /* PGDIR page */ \ - + EARLY_PGDS((vstart), (vend), add) /* each PGDIR needs a next level page table */ \ - + EARLY_PUDS((vstart), (vend), add) /* each PUD needs a next level page table */ \ - + EARLY_PMDS((vstart), (vend), add)) /* each PMD needs a next level page table */ +#define EARLY_PAGES(vstart, vend, add) (1 /* PGDIR page */ \ + + EARLY_LEVEL(3, (vstart), (vend), add) /* each entry needs a next level page table */ \ + + EARLY_LEVEL(2, (vstart), (vend), add) /* each entry needs a next level page table */ \ + + EARLY_LEVEL(1, (vstart), (vend), add))/* each entry needs a next level page table */ #define INIT_DIR_SIZE (PAGE_SIZE * (EARLY_PAGES(KIMAGE_VADDR, _end, EXTRA_PAGE) + EARLY_SEGMENT_EXTRA_PAGES)) /* the initial ID map may need two extra pages if it needs to be extended */ @@ -81,17 +65,6 @@ #endif #define INIT_IDMAP_DIR_PAGES EARLY_PAGES(KIMAGE_VADDR, _end + MAX_FDT_SIZE + SWAPPER_BLOCK_SIZE, 1) -/* Initial memory map size */ -#ifdef CONFIG_ARM64_4K_PAGES -#define SWAPPER_BLOCK_SHIFT PMD_SHIFT -#define SWAPPER_BLOCK_SIZE PMD_SIZE -#define SWAPPER_TABLE_SHIFT PUD_SHIFT -#else -#define SWAPPER_BLOCK_SHIFT PAGE_SHIFT -#define SWAPPER_BLOCK_SIZE PAGE_SIZE -#define SWAPPER_TABLE_SHIFT PMD_SHIFT -#endif - /* The number of segments in the kernel image (text, rodata, inittext, initdata, data+bss) */ #define KERNEL_SEGMENT_COUNT 5 From patchwork Wed Feb 14 12:29:07 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13556423 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id BB1AAC48BC3 for ; Wed, 14 Feb 2024 12:32:30 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=aQdam0MMLPt3BdYpehu3/YjKWdp2kIiXHvriWwSpztU=; b=TEXV5Awq1/vOpr4+4l+PMnZktZ igUEmpipkh1uR9XpHMuswnjIv0XPC+4OV4C9te36w3hASWs3kxdmpKvbjRnHwWs7yxx/GYnZ2zufo KfX9OBgzeh3lpZxqrWRkqi4ZHi4tAc0Kw0qcq9ztBIrPzF92fcHaS0GUFPB0IlU03eEjHMTwPpFcj lbUdI9cVagTSETyc3IIDiT2PB47fe3uQd8IaxH6yg/WG8Dw6L6D7q0fw4hNOkoHqVsZqiT/6h1dCX dnBCxQcqVc8M+wRv5Jb1c1nkWjIvfQDEUZdO61MiensFycvEM0x5qCjZ4Xfn22Tm1FjWgKsGh0gI+ xB7gs4Dg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1raEQs-0000000CpQC-2ABO; Wed, 14 Feb 2024 12:32:10 +0000 Received: from casper.infradead.org ([2001:8b0:10b:1236::1]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1raEPT-0000000Copc-1q9S for linux-arm-kernel@bombadil.infradead.org; Wed, 14 Feb 2024 12:30:43 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=FxAZICnIfSNBC8PlObMJjkTHYwUVY0Xl7GcOhL6fBCg=; b=AdST5P5q6L63R+Iz5TayW2tHZs QyzX+65QJB95eYw1ZWUkTeTycFQNAXzuQAl4wPjaqL4MY1VHNPtm9QpSugPASfLx5pf8Jv5GJ3Q5q eXtsL6IudayxTaz/MGZm3IdTGu+VT6H/XuWD+lUqeU6t0W9CAYdn2Vgzc1/i7imV0wfDTjrJtGVxL x5CNFyKhyYJtgjCa4SlDYSRfJyuM5Bg3wxF3ZokQlouQySxQxAmJ/3elh7mB7SOgqBaamzqlYt2BI LwxIT0CFRznYhgPbP304R3QHf9aO5srkuDbtgpPqD8HxA8qiVFSh0eoHbGWj71yifJCe6Y/7kcEne b8rMEhfg==; Received: from mail-yb1-xb4a.google.com ([2607:f8b0:4864:20::b4a]) by casper.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1raEPN-0000000Ga0L-0lNI for linux-arm-kernel@lists.infradead.org; Wed, 14 Feb 2024 12:30:42 +0000 Received: by mail-yb1-xb4a.google.com with SMTP id 3f1490d57ef6-dcd1779adbeso2783553276.3 for ; Wed, 14 Feb 2024 04:30:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1707913827; x=1708518627; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=FxAZICnIfSNBC8PlObMJjkTHYwUVY0Xl7GcOhL6fBCg=; b=Q44ueilKQ6Nw2M5f9XCtBvnE+k5GTZGDTqFqXAd+2GDjx73GslUsZTAtqo8Xtw1SeH q5wrn6DeEAWoNIHaDwEsGAUIh9RZ2iEWosNK2drAQBJsApzvdJqGLofe4KvdR1MVKy+X OzMMRt+X7F6KyeYUytatTURiVn4+LhUbPFIfOHGhX2jUWaQknyLHjCEk8Y+K9yEM70KX k39RYsJoE2N3NCLEyUqoynPuICx8NPSxxf5DkYEvrMnaddBHmcJYBfFN7nO2f78u90Ap e4YPhERungVP6nIZhLQiywtbCJmcDvFHHMwlwfiL089TfHwLu14bI07PLanCi4rCBUBW 9I0w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707913827; x=1708518627; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=FxAZICnIfSNBC8PlObMJjkTHYwUVY0Xl7GcOhL6fBCg=; b=tGorteFssrB1PXO5jtGuPNLTdrvlCGK1+qZOU0QMJWSLPHonqvv35EbXigl77K+T8w eufPkUQSyc/CU9AN1X68PWhY0sRnD6ANn261NGOJaPE5c9truC0pY5hnWh2EJyGd3VaV zqNr5Upyzo+5QDKILVrUal0AqViU8MNOa+988fTTRXD9loNXqhe+C6ugXaIwA24SLKOW QRL+xsT1Ff4o1jFqaREdGgxWb6toX1M1nNiRD/gPEW+VL1pY+8oynUjlV2WuAXKWJexK 2a98LMHywLps1HGs0MmfMt6wichyTYCpstF+S3r8n1HQIcjRXUTwb52X+244+Gw/yMmn e8Bg== X-Gm-Message-State: AOJu0YzuCNvqkU39LKFOEgYqfdJjLLYkxeHw8vmAf/qNFjlvHJgEOa11 820B2X67O1J/DCuL4nchg3VF3WjPernaZ8A2dgxf28UrzlfgntjUkO+YxajR7uE5CAcgpK64geA zPpSV82ZoU4BaUWhLfr/VVrvkY+R1wiaA7X5S9VRexELIV63CuQjTbnpJzrvs4c/mKOfgOpuOQM Yn3LoDs49sOm53XC2yAEioXSKTNbxmE/94U8H+jswg X-Google-Smtp-Source: AGHT+IEli3kQ/aLUjGcHaMMEfrGEopLiQkCt3ln4GtpaEozSXpaC3MiT27hNiS+DxeeQMwZkQ36cOwAK X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:6902:1021:b0:dc6:fa35:b42 with SMTP id x1-20020a056902102100b00dc6fa350b42mr494555ybt.2.1707913826939; Wed, 14 Feb 2024 04:30:26 -0800 (PST) Date: Wed, 14 Feb 2024 13:29:07 +0100 In-Reply-To: <20240214122845.2033971-45-ardb+git@google.com> Mime-Version: 1.0 References: <20240214122845.2033971-45-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=24473; i=ardb@kernel.org; h=from:subject; bh=YPQjrB3KBTv2qgT742gKCE0U1LhXInoecpMOsvdMBeM=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIfXMJtHcmgu9ryZKBUV9yeDtmtGe4aXw6L9ojE7D7kN2C zY0aPd0lLIwiHEwyIopsgjM/vtu5+mJUrXOs2Rh5rAygQxh4OIUgInwrGdkaD4o3ehWFV3zqn4b 20w/R72qAoZ1q1fMWZ1YfvviQkvrhYwMh8u1xXomFf19+UPhUUSt+bKYzHV2uxQC7r/Vm9QsNC+ bHwA= X-Mailer: git-send-email 2.43.0.687.g38aa6559b0-goog Message-ID: <20240214122845.2033971-66-ardb+git@google.com> Subject: [PATCH v8 21/43] arm64: kernel: Create initial ID map from C code From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240214_123038_659205_EB974513 X-CRM114-Status: GOOD ( 27.92 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel The asm code that creates the initial ID map is rather intricate and hard to follow. This is problematic because it makes adding support for things like LPA2 or WXN more difficult than necessary. Also, it is parameterized like the rest of the MM code to run with a configurable number of levels, which is rather pointless, given that all AArch64 CPUs implement support for 48-bit virtual addressing, and that many systems exist with DRAM located outside of the 39-bit addressable range, which is the only smaller VA size that is widely used, and we need additional tricks to make things work in that combination. So let's bite the bullet, and rip out all the asm macros, and fiddly code, and replace it with a C implementation based on the newly added routines for creating the early kernel VA mappings. And while at it, create the initial ID map based on 48-bit virtual addressing as well, regardless of the number of configured levels for the kernel proper. Note that this code may execute with the MMU and caches disabled, and is therefore not permitted to make unaligned accesses. This shouldn't generally happen in any case for the algorithm as implemented, but to be sure, let's pass -mstrict-align to the compiler just in case. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/assembler.h | 14 - arch/arm64/include/asm/kernel-pgtable.h | 50 ++-- arch/arm64/include/asm/mmu_context.h | 6 +- arch/arm64/kernel/head.S | 267 ++------------------ arch/arm64/kernel/image-vars.h | 1 + arch/arm64/kernel/pi/Makefile | 3 + arch/arm64/kernel/pi/map_kernel.c | 18 ++ arch/arm64/kernel/pi/map_range.c | 12 + arch/arm64/kernel/pi/pi.h | 4 + arch/arm64/mm/mmu.c | 5 - arch/arm64/mm/proc.S | 3 +- 11 files changed, 88 insertions(+), 295 deletions(-) diff --git a/arch/arm64/include/asm/assembler.h b/arch/arm64/include/asm/assembler.h index 513787e43329..6a467c694039 100644 --- a/arch/arm64/include/asm/assembler.h +++ b/arch/arm64/include/asm/assembler.h @@ -345,20 +345,6 @@ alternative_cb_end bfi \valreg, \t1sz, #TCR_T1SZ_OFFSET, #TCR_TxSZ_WIDTH .endm -/* - * idmap_get_t0sz - get the T0SZ value needed to cover the ID map - * - * Calculate the maximum allowed value for TCR_EL1.T0SZ so that the - * entire ID map region can be mapped. As T0SZ == (64 - #bits used), - * this number conveniently equals the number of leading zeroes in - * the physical address of _end. - */ - .macro idmap_get_t0sz, reg - adrp \reg, _end - orr \reg, \reg, #(1 << VA_BITS_MIN) - 1 - clz \reg, \reg - .endm - /* * tcr_compute_pa_size - set TCR.(I)PS to the highest supported * ID_AA64MMFR0_EL1.PARange value diff --git a/arch/arm64/include/asm/kernel-pgtable.h b/arch/arm64/include/asm/kernel-pgtable.h index f1fc98a233d5..bf05a77873a4 100644 --- a/arch/arm64/include/asm/kernel-pgtable.h +++ b/arch/arm64/include/asm/kernel-pgtable.h @@ -29,6 +29,7 @@ #define SWAPPER_TABLE_SHIFT (SWAPPER_BLOCK_SHIFT + PAGE_SHIFT - 3) #define SWAPPER_PGTABLE_LEVELS (CONFIG_PGTABLE_LEVELS - SWAPPER_SKIP_LEVEL) +#define INIT_IDMAP_PGTABLE_LEVELS (IDMAP_LEVELS - SWAPPER_SKIP_LEVEL) #define IDMAP_VA_BITS 48 #define IDMAP_LEVELS ARM64_HW_PGTABLE_LEVELS(IDMAP_VA_BITS) @@ -48,44 +49,39 @@ #define EARLY_ENTRIES(vstart, vend, shift, add) \ (SPAN_NR_ENTRIES(vstart, vend, shift) + (add)) -#define EARLY_LEVEL(lvl, vstart, vend, add) \ - (SWAPPER_PGTABLE_LEVELS > lvl ? EARLY_ENTRIES(vstart, vend, SWAPPER_BLOCK_SHIFT + lvl * (PAGE_SHIFT - 3), add) : 0) +#define EARLY_LEVEL(lvl, lvls, vstart, vend, add) \ + (lvls > lvl ? EARLY_ENTRIES(vstart, vend, SWAPPER_BLOCK_SHIFT + lvl * (PAGE_SHIFT - 3), add) : 0) -#define EARLY_PAGES(vstart, vend, add) (1 /* PGDIR page */ \ - + EARLY_LEVEL(3, (vstart), (vend), add) /* each entry needs a next level page table */ \ - + EARLY_LEVEL(2, (vstart), (vend), add) /* each entry needs a next level page table */ \ - + EARLY_LEVEL(1, (vstart), (vend), add))/* each entry needs a next level page table */ -#define INIT_DIR_SIZE (PAGE_SIZE * (EARLY_PAGES(KIMAGE_VADDR, _end, EXTRA_PAGE) + EARLY_SEGMENT_EXTRA_PAGES)) +#define EARLY_PAGES(lvls, vstart, vend, add) (1 /* PGDIR page */ \ + + EARLY_LEVEL(3, (lvls), (vstart), (vend), add) /* each entry needs a next level page table */ \ + + EARLY_LEVEL(2, (lvls), (vstart), (vend), add) /* each entry needs a next level page table */ \ + + EARLY_LEVEL(1, (lvls), (vstart), (vend), add))/* each entry needs a next level page table */ +#define INIT_DIR_SIZE (PAGE_SIZE * (EARLY_PAGES(SWAPPER_PGTABLE_LEVELS, KIMAGE_VADDR, _end, EXTRA_PAGE) \ + + EARLY_SEGMENT_EXTRA_PAGES)) -/* the initial ID map may need two extra pages if it needs to be extended */ -#if VA_BITS < 48 -#define INIT_IDMAP_DIR_SIZE ((INIT_IDMAP_DIR_PAGES + 2) * PAGE_SIZE) -#else -#define INIT_IDMAP_DIR_SIZE (INIT_IDMAP_DIR_PAGES * PAGE_SIZE) -#endif -#define INIT_IDMAP_DIR_PAGES EARLY_PAGES(KIMAGE_VADDR, _end + MAX_FDT_SIZE + SWAPPER_BLOCK_SIZE, 1) +#define INIT_IDMAP_DIR_PAGES (EARLY_PAGES(INIT_IDMAP_PGTABLE_LEVELS, KIMAGE_VADDR, _end, 1)) +#define INIT_IDMAP_DIR_SIZE ((INIT_IDMAP_DIR_PAGES + EARLY_IDMAP_EXTRA_PAGES) * PAGE_SIZE) + +#define INIT_IDMAP_FDT_PAGES (EARLY_PAGES(INIT_IDMAP_PGTABLE_LEVELS, 0UL, UL(MAX_FDT_SIZE), 1) - 1) +#define INIT_IDMAP_FDT_SIZE ((INIT_IDMAP_FDT_PAGES + EARLY_IDMAP_EXTRA_FDT_PAGES) * PAGE_SIZE) /* The number of segments in the kernel image (text, rodata, inittext, initdata, data+bss) */ #define KERNEL_SEGMENT_COUNT 5 #if SWAPPER_BLOCK_SIZE > SEGMENT_ALIGN #define EARLY_SEGMENT_EXTRA_PAGES (KERNEL_SEGMENT_COUNT + 1) -#else -#define EARLY_SEGMENT_EXTRA_PAGES 0 -#endif - /* - * Initial memory map attributes. + * The initial ID map consists of the kernel image, mapped as two separate + * segments, and may appear misaligned wrt the swapper block size. This means + * we need 3 additional pages. The DT could straddle a swapper block boundary, + * so it may need 2. */ -#define SWAPPER_PTE_FLAGS (PTE_TYPE_PAGE | PTE_AF | PTE_SHARED | PTE_UXN) -#define SWAPPER_PMD_FLAGS (PMD_TYPE_SECT | PMD_SECT_AF | PMD_SECT_S | PTE_UXN) - -#ifdef CONFIG_ARM64_4K_PAGES -#define SWAPPER_RW_MMUFLAGS (PMD_ATTRINDX(MT_NORMAL) | SWAPPER_PMD_FLAGS | PTE_WRITE) -#define SWAPPER_RX_MMUFLAGS (SWAPPER_RW_MMUFLAGS | PMD_SECT_RDONLY) +#define EARLY_IDMAP_EXTRA_PAGES 3 +#define EARLY_IDMAP_EXTRA_FDT_PAGES 2 #else -#define SWAPPER_RW_MMUFLAGS (PTE_ATTRINDX(MT_NORMAL) | SWAPPER_PTE_FLAGS | PTE_WRITE) -#define SWAPPER_RX_MMUFLAGS (SWAPPER_RW_MMUFLAGS | PTE_RDONLY) +#define EARLY_SEGMENT_EXTRA_PAGES 0 +#define EARLY_IDMAP_EXTRA_PAGES 0 +#define EARLY_IDMAP_EXTRA_FDT_PAGES 0 #endif #endif /* __ASM_KERNEL_PGTABLE_H */ diff --git a/arch/arm64/include/asm/mmu_context.h b/arch/arm64/include/asm/mmu_context.h index 926fbbcecbe0..a8a89a0f2867 100644 --- a/arch/arm64/include/asm/mmu_context.h +++ b/arch/arm64/include/asm/mmu_context.h @@ -61,11 +61,9 @@ static inline void cpu_switch_mm(pgd_t *pgd, struct mm_struct *mm) } /* - * TCR.T0SZ value to use when the ID map is active. Usually equals - * TCR_T0SZ(VA_BITS), unless system RAM is positioned very high in - * physical memory, in which case it will be smaller. + * TCR.T0SZ value to use when the ID map is active. */ -extern int idmap_t0sz; +#define idmap_t0sz TCR_T0SZ(IDMAP_VA_BITS) /* * Ensure TCR.T0SZ is set to the provided value. diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S index a1c29d64e875..545b5d8976f4 100644 --- a/arch/arm64/kernel/head.S +++ b/arch/arm64/kernel/head.S @@ -80,26 +80,42 @@ * x19 primary_entry() .. start_kernel() whether we entered with the MMU on * x20 primary_entry() .. __primary_switch() CPU boot mode * x21 primary_entry() .. start_kernel() FDT pointer passed at boot in x0 - * x22 create_idmap() .. start_kernel() ID map VA of the DT blob * x25 primary_entry() .. start_kernel() supported VA size - * x28 create_idmap() callee preserved temp register */ SYM_CODE_START(primary_entry) bl record_mmu_state bl preserve_boot_args - bl create_idmap + + adrp x1, early_init_stack + mov sp, x1 + mov x29, xzr + adrp x0, init_idmap_pg_dir + bl __pi_create_init_idmap + + /* + * If the page tables have been populated with non-cacheable + * accesses (MMU disabled), invalidate those tables again to + * remove any speculatively loaded cache lines. + */ + cbnz x19, 0f + dmb sy + mov x1, x0 // end of used region + adrp x0, init_idmap_pg_dir + adr_l x2, dcache_inval_poc + blr x2 + b 1f /* * If we entered with the MMU and caches on, clean the ID mapped part * of the primary boot code to the PoC so we can safely execute it with * the MMU off. */ - cbz x19, 0f - adrp x0, __idmap_text_start +0: adrp x0, __idmap_text_start adr_l x1, __idmap_text_end adr_l x2, dcache_clean_poc blr x2 -0: mov x0, x19 + +1: mov x0, x19 bl init_kernel_el // w0=cpu_boot_mode mov x20, x0 @@ -175,238 +191,6 @@ SYM_CODE_START_LOCAL(preserve_boot_args) ret SYM_CODE_END(preserve_boot_args) -/* - * Macro to populate page table entries, these entries can be pointers to the next level - * or last level entries pointing to physical memory. - * - * tbl: page table address - * rtbl: pointer to page table or physical memory - * index: start index to write - * eindex: end index to write - [index, eindex] written to - * flags: flags for pagetable entry to or in - * inc: increment to rtbl between each entry - * tmp1: temporary variable - * - * Preserves: tbl, eindex, flags, inc - * Corrupts: index, tmp1 - * Returns: rtbl - */ - .macro populate_entries, tbl, rtbl, index, eindex, flags, inc, tmp1 -.Lpe\@: phys_to_pte \tmp1, \rtbl - orr \tmp1, \tmp1, \flags // tmp1 = table entry - str \tmp1, [\tbl, \index, lsl #3] - add \rtbl, \rtbl, \inc // rtbl = pa next level - add \index, \index, #1 - cmp \index, \eindex - b.ls .Lpe\@ - .endm - -/* - * Compute indices of table entries from virtual address range. If multiple entries - * were needed in the previous page table level then the next page table level is assumed - * to be composed of multiple pages. (This effectively scales the end index). - * - * vstart: virtual address of start of range - * vend: virtual address of end of range - we map [vstart, vend] - * shift: shift used to transform virtual address into index - * order: #imm 2log(number of entries in page table) - * istart: index in table corresponding to vstart - * iend: index in table corresponding to vend - * count: On entry: how many extra entries were required in previous level, scales - * our end index. - * On exit: returns how many extra entries required for next page table level - * - * Preserves: vstart, vend - * Returns: istart, iend, count - */ - .macro compute_indices, vstart, vend, shift, order, istart, iend, count - ubfx \istart, \vstart, \shift, \order - ubfx \iend, \vend, \shift, \order - add \iend, \iend, \count, lsl \order - sub \count, \iend, \istart - .endm - -/* - * Map memory for specified virtual address range. Each level of page table needed supports - * multiple entries. If a level requires n entries the next page table level is assumed to be - * formed from n pages. - * - * tbl: location of page table - * rtbl: address to be used for first level page table entry (typically tbl + PAGE_SIZE) - * vstart: virtual address of start of range - * vend: virtual address of end of range - we map [vstart, vend - 1] - * flags: flags to use to map last level entries - * phys: physical address corresponding to vstart - physical memory is contiguous - * order: #imm 2log(number of entries in PGD table) - * - * If extra_shift is set, an extra level will be populated if the end address does - * not fit in 'extra_shift' bits. This assumes vend is in the TTBR0 range. - * - * Temporaries: istart, iend, tmp, count, sv - these need to be different registers - * Preserves: vstart, flags - * Corrupts: tbl, rtbl, vend, istart, iend, tmp, count, sv - */ - .macro map_memory, tbl, rtbl, vstart, vend, flags, phys, order, istart, iend, tmp, count, sv, extra_shift - sub \vend, \vend, #1 - add \rtbl, \tbl, #PAGE_SIZE - mov \count, #0 - - .ifnb \extra_shift - tst \vend, #~((1 << (\extra_shift)) - 1) - b.eq .L_\@ - compute_indices \vstart, \vend, #\extra_shift, #(PAGE_SHIFT - 3), \istart, \iend, \count - mov \sv, \rtbl - populate_entries \tbl, \rtbl, \istart, \iend, #PMD_TYPE_TABLE, #PAGE_SIZE, \tmp - mov \tbl, \sv - .endif -.L_\@: - compute_indices \vstart, \vend, #PGDIR_SHIFT, #\order, \istart, \iend, \count - mov \sv, \rtbl - populate_entries \tbl, \rtbl, \istart, \iend, #PMD_TYPE_TABLE, #PAGE_SIZE, \tmp - mov \tbl, \sv - -#if SWAPPER_PGTABLE_LEVELS > 3 - compute_indices \vstart, \vend, #PUD_SHIFT, #(PAGE_SHIFT - 3), \istart, \iend, \count - mov \sv, \rtbl - populate_entries \tbl, \rtbl, \istart, \iend, #PMD_TYPE_TABLE, #PAGE_SIZE, \tmp - mov \tbl, \sv -#endif - -#if SWAPPER_PGTABLE_LEVELS > 2 - compute_indices \vstart, \vend, #SWAPPER_TABLE_SHIFT, #(PAGE_SHIFT - 3), \istart, \iend, \count - mov \sv, \rtbl - populate_entries \tbl, \rtbl, \istart, \iend, #PMD_TYPE_TABLE, #PAGE_SIZE, \tmp - mov \tbl, \sv -#endif - - compute_indices \vstart, \vend, #SWAPPER_BLOCK_SHIFT, #(PAGE_SHIFT - 3), \istart, \iend, \count - bic \rtbl, \phys, #SWAPPER_BLOCK_SIZE - 1 - populate_entries \tbl, \rtbl, \istart, \iend, \flags, #SWAPPER_BLOCK_SIZE, \tmp - .endm - -/* - * Remap a subregion created with the map_memory macro with modified attributes - * or output address. The entire remapped region must have been covered in the - * invocation of map_memory. - * - * x0: last level table address (returned in first argument to map_memory) - * x1: start VA of the existing mapping - * x2: start VA of the region to update - * x3: end VA of the region to update (exclusive) - * x4: start PA associated with the region to update - * x5: attributes to set on the updated region - * x6: order of the last level mappings - */ -SYM_FUNC_START_LOCAL(remap_region) - sub x3, x3, #1 // make end inclusive - - // Get the index offset for the start of the last level table - lsr x1, x1, x6 - bfi x1, xzr, #0, #PAGE_SHIFT - 3 - - // Derive the start and end indexes into the last level table - // associated with the provided region - lsr x2, x2, x6 - lsr x3, x3, x6 - sub x2, x2, x1 - sub x3, x3, x1 - - mov x1, #1 - lsl x6, x1, x6 // block size at this level - - populate_entries x0, x4, x2, x3, x5, x6, x7 - ret -SYM_FUNC_END(remap_region) - -SYM_FUNC_START_LOCAL(create_idmap) - mov x28, lr - /* - * The ID map carries a 1:1 mapping of the physical address range - * covered by the loaded image, which could be anywhere in DRAM. This - * means that the required size of the VA (== PA) space is decided at - * boot time, and could be more than the configured size of the VA - * space for ordinary kernel and user space mappings. - * - * There are three cases to consider here: - * - 39 <= VA_BITS < 48, and the ID map needs up to 48 VA bits to cover - * the placement of the image. In this case, we configure one extra - * level of translation on the fly for the ID map only. (This case - * also covers 42-bit VA/52-bit PA on 64k pages). - * - * - VA_BITS == 48, and the ID map needs more than 48 VA bits. This can - * only happen when using 64k pages, in which case we need to extend - * the root level table rather than add a level. Note that we can - * treat this case as 'always extended' as long as we take care not - * to program an unsupported T0SZ value into the TCR register. - * - * - Combinations that would require two additional levels of - * translation are not supported, e.g., VA_BITS==36 on 16k pages, or - * VA_BITS==39/4k pages with 5-level paging, where the input address - * requires more than 47 or 48 bits, respectively. - */ -#if (VA_BITS < 48) -#define IDMAP_PGD_ORDER (VA_BITS - PGDIR_SHIFT) -#define EXTRA_SHIFT (PGDIR_SHIFT + PAGE_SHIFT - 3) - - /* - * If VA_BITS < 48, we have to configure an additional table level. - * First, we have to verify our assumption that the current value of - * VA_BITS was chosen such that all translation levels are fully - * utilised, and that lowering T0SZ will always result in an additional - * translation level to be configured. - */ -#if VA_BITS != EXTRA_SHIFT -#error "Mismatch between VA_BITS and page size/number of translation levels" -#endif -#else -#define IDMAP_PGD_ORDER (PHYS_MASK_SHIFT - PGDIR_SHIFT) -#define EXTRA_SHIFT - /* - * If VA_BITS == 48, we don't have to configure an additional - * translation level, but the top-level table has more entries. - */ -#endif - adrp x0, init_idmap_pg_dir - adrp x3, _text - adrp x6, _end + MAX_FDT_SIZE + SWAPPER_BLOCK_SIZE - mov_q x7, SWAPPER_RX_MMUFLAGS - - map_memory x0, x1, x3, x6, x7, x3, IDMAP_PGD_ORDER, x10, x11, x12, x13, x14, EXTRA_SHIFT - - /* Remap [.init].data, BSS and the kernel page tables r/w in the ID map */ - adrp x1, _text - adrp x2, __initdata_begin - adrp x3, _end - bic x4, x2, #SWAPPER_BLOCK_SIZE - 1 - mov_q x5, SWAPPER_RW_MMUFLAGS - mov x6, #SWAPPER_BLOCK_SHIFT - bl remap_region - - /* Remap the FDT after the kernel image */ - adrp x1, _text - adrp x22, _end + SWAPPER_BLOCK_SIZE - bic x2, x22, #SWAPPER_BLOCK_SIZE - 1 - bfi x22, x21, #0, #SWAPPER_BLOCK_SHIFT // remapped FDT address - add x3, x2, #MAX_FDT_SIZE + SWAPPER_BLOCK_SIZE - bic x4, x21, #SWAPPER_BLOCK_SIZE - 1 - mov_q x5, SWAPPER_RW_MMUFLAGS - mov x6, #SWAPPER_BLOCK_SHIFT - bl remap_region - - /* - * Since the page tables have been populated with non-cacheable - * accesses (MMU disabled), invalidate those tables again to - * remove any speculatively loaded cache lines. - */ - cbnz x19, 0f // skip cache invalidation if MMU is on - dmb sy - - adrp x0, init_idmap_pg_dir - adrp x1, init_idmap_pg_end - bl dcache_inval_poc -0: ret x28 -SYM_FUNC_END(create_idmap) - /* * Initialize CPU registers with task-specific and cpu-specific context. * @@ -729,11 +513,6 @@ SYM_FUNC_START_LOCAL(__no_granule_support) SYM_FUNC_END(__no_granule_support) SYM_FUNC_START_LOCAL(__primary_switch) - mrs x1, tcr_el1 - mov x2, #64 - VA_BITS - tcr_set_t0sz x1, x2 - msr tcr_el1, x1 - adrp x1, reserved_pg_dir adrp x2, init_idmap_pg_dir bl __enable_mmu @@ -742,7 +521,7 @@ SYM_FUNC_START_LOCAL(__primary_switch) mov sp, x1 mov x29, xzr mov x0, x20 // pass the full boot status - mov x1, x22 // pass the low FDT mapping + mov x1, x21 // pass the FDT bl __pi_early_map_kernel // Map and relocate the kernel ldr x8, =__primary_switched diff --git a/arch/arm64/kernel/image-vars.h b/arch/arm64/kernel/image-vars.h index e566b32f9c22..941a14c05184 100644 --- a/arch/arm64/kernel/image-vars.h +++ b/arch/arm64/kernel/image-vars.h @@ -52,6 +52,7 @@ PROVIDE(__pi_cavium_erratum_27456_cpus = cavium_erratum_27456_cpus); PROVIDE(__pi__ctype = _ctype); PROVIDE(__pi_memstart_offset_seed = memstart_offset_seed); +PROVIDE(__pi_init_idmap_pg_dir = init_idmap_pg_dir); PROVIDE(__pi_init_pg_dir = init_pg_dir); PROVIDE(__pi_init_pg_end = init_pg_end); diff --git a/arch/arm64/kernel/pi/Makefile b/arch/arm64/kernel/pi/Makefile index 8c2f80a46b93..4393b41f0b71 100644 --- a/arch/arm64/kernel/pi/Makefile +++ b/arch/arm64/kernel/pi/Makefile @@ -11,6 +11,9 @@ KBUILD_CFLAGS := $(subst $(CC_FLAGS_FTRACE),,$(KBUILD_CFLAGS)) -fpie \ -fno-asynchronous-unwind-tables -fno-unwind-tables \ $(call cc-option,-fno-addrsig) +# this code may run with the MMU off so disable unaligned accesses +CFLAGS_map_range.o += -mstrict-align + # remove SCS flags from all objects in this directory KBUILD_CFLAGS := $(filter-out $(CC_FLAGS_SCS), $(KBUILD_CFLAGS)) # disable LTO diff --git a/arch/arm64/kernel/pi/map_kernel.c b/arch/arm64/kernel/pi/map_kernel.c index f206373b28b0..f86e878d366d 100644 --- a/arch/arm64/kernel/pi/map_kernel.c +++ b/arch/arm64/kernel/pi/map_kernel.c @@ -128,6 +128,22 @@ static void __init map_kernel(u64 kaslr_offset, u64 va_offset, int root_level) } } +static void __init map_fdt(u64 fdt) +{ + static u8 ptes[INIT_IDMAP_FDT_SIZE] __initdata __aligned(PAGE_SIZE); + u64 efdt = fdt + MAX_FDT_SIZE; + u64 ptep = (u64)ptes; + + /* + * Map up to MAX_FDT_SIZE bytes, but avoid overlap with + * the kernel image. + */ + map_range(&ptep, fdt, (u64)_text > fdt ? min((u64)_text, efdt) : efdt, + fdt, PAGE_KERNEL, IDMAP_ROOT_LEVEL, + (pte_t *)init_idmap_pg_dir, false, 0); + dsb(ishst); +} + asmlinkage void __init early_map_kernel(u64 boot_status, void *fdt) { static char const chosen_str[] __initconst = "/chosen"; @@ -136,6 +152,8 @@ asmlinkage void __init early_map_kernel(u64 boot_status, void *fdt) int root_level = 4 - CONFIG_PGTABLE_LEVELS; int chosen; + map_fdt((u64)fdt); + /* Clear BSS and the initial page tables */ memset(__bss_start, 0, (u64)init_pg_end - (u64)__bss_start); diff --git a/arch/arm64/kernel/pi/map_range.c b/arch/arm64/kernel/pi/map_range.c index c31feda18f47..79e4f6a2efe1 100644 --- a/arch/arm64/kernel/pi/map_range.c +++ b/arch/arm64/kernel/pi/map_range.c @@ -86,3 +86,15 @@ void __init map_range(u64 *pte, u64 start, u64 end, u64 pa, pgprot_t prot, tbl++; } } + +asmlinkage u64 __init create_init_idmap(pgd_t *pg_dir) +{ + u64 ptep = (u64)pg_dir + PAGE_SIZE; + + map_range(&ptep, (u64)_stext, (u64)__initdata_begin, (u64)_stext, + PAGE_KERNEL_ROX, IDMAP_ROOT_LEVEL, (pte_t *)pg_dir, false, 0); + map_range(&ptep, (u64)__initdata_begin, (u64)_end, (u64)__initdata_begin, + PAGE_KERNEL, IDMAP_ROOT_LEVEL, (pte_t *)pg_dir, false, 0); + + return ptep; +} diff --git a/arch/arm64/kernel/pi/pi.h b/arch/arm64/kernel/pi/pi.h index d307c58e9741..1ea282a5f96a 100644 --- a/arch/arm64/kernel/pi/pi.h +++ b/arch/arm64/kernel/pi/pi.h @@ -21,6 +21,8 @@ static inline void *prel64_to_pointer(const prel64_t *offset) extern bool dynamic_scs_is_enabled; +extern pgd_t init_idmap_pg_dir[]; + void init_feature_override(u64 boot_status, const void *fdt, int chosen); u64 kaslr_early_init(void *fdt, int chosen); void relocate_kernel(u64 offset); @@ -30,3 +32,5 @@ void map_range(u64 *pgd, u64 start, u64 end, u64 pa, pgprot_t prot, int level, pte_t *tbl, bool may_use_cont, u64 va_offset); asmlinkage void early_map_kernel(u64 boot_status, void *fdt); + +asmlinkage u64 create_init_idmap(pgd_t *pgd); diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index a991f195592b..14a62c773201 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -45,8 +45,6 @@ #define NO_CONT_MAPPINGS BIT(1) #define NO_EXEC_MAPPINGS BIT(2) /* assumes FEAT_HPDS is not used */ -int idmap_t0sz __ro_after_init; - #if VA_BITS > 48 u64 vabits_actual __ro_after_init = VA_BITS_MIN; EXPORT_SYMBOL(vabits_actual); @@ -793,8 +791,6 @@ void __init paging_init(void) pgd_t *pgdp = pgd_set_fixmap(__pa_symbol(swapper_pg_dir)); extern pgd_t init_idmap_pg_dir[]; - idmap_t0sz = 63UL - __fls(__pa_symbol(_end) | GENMASK(VA_BITS_MIN - 1, 0)); - map_kernel(pgdp); map_mem(pgdp); @@ -809,7 +805,6 @@ void __init paging_init(void) memblock_allow_resize(); create_idmap(); - idmap_t0sz = TCR_T0SZ(IDMAP_VA_BITS); } #ifdef CONFIG_MEMORY_HOTPLUG diff --git a/arch/arm64/mm/proc.S b/arch/arm64/mm/proc.S index 47ede52bb900..55c366dbda8f 100644 --- a/arch/arm64/mm/proc.S +++ b/arch/arm64/mm/proc.S @@ -200,7 +200,8 @@ SYM_FUNC_ALIAS(__pi_idmap_cpu_replace_ttbr1, idmap_cpu_replace_ttbr1) #ifdef CONFIG_UNMAP_KERNEL_AT_EL0 -#define KPTI_NG_PTE_FLAGS (PTE_ATTRINDX(MT_NORMAL) | SWAPPER_PTE_FLAGS | PTE_WRITE) +#define KPTI_NG_PTE_FLAGS (PTE_ATTRINDX(MT_NORMAL) | PTE_TYPE_PAGE | \ + PTE_AF | PTE_SHARED | PTE_UXN | PTE_WRITE) .pushsection ".idmap.text", "a" From patchwork Wed Feb 14 12:29:08 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13556419 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 815BEC48BC4 for ; Wed, 14 Feb 2024 12:32:24 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=bkGYdxFdPgvleGB3uYGyEDhImt+Y4F+pwlf69qtdWfQ=; b=yYLxd288OT7XnJYW3PPZ1+7uIm r3mulQ06FTR/ZffiQD5X19+D3xdY3ZAQ3uWlFoewG7vYn7u2H7O+YzlXpUxMYyYH1Bbwo5bo4v1uN DfAcxfpsTaZLTCh/FJv+7hLNy95VSy9PHzdjUeHTUhAl9D4l8Ws4MvAscHjhSdGZMS2bxqRcLbG6J jXByIp7s4Qc3g7omUxdXFTD0frrxtC/4MFqWLNHz3NBYuovu1IziYkildzASkZQMqCPLRecTxDoG3 5nayu+BuDr3RxieECDhDm4GG+pOa4pEd17h0w1GAwxTy0aGhGyTshgLSR/H/0Z/uAzf9wqyZ2incq 4KFcxQ/Q==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1raEQp-0000000CpOv-3dMp; Wed, 14 Feb 2024 12:32:07 +0000 Received: from casper.infradead.org ([2001:8b0:10b:1236::1]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1raEPS-0000000Cooi-1pj6 for linux-arm-kernel@bombadil.infradead.org; Wed, 14 Feb 2024 12:30:42 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=7fAeJkvIpRC6qpF8jpEwHWSgHc+l/OrvPGb3S3qTTzU=; b=lQdtGMcj38+vZUzDvetibyd8IE FroBP5Z4PLyOTpUN9qFdCDbBgvVhMF/mGtEtSB0O53pN7wmePKTcvLlfuD+yFbfbAQYNVJrnzZwGq o15ZxFkC4eYObwdz4ObZfKMuipTvXmDNFyRKL1rwKowhHWVf8KvZSSAsHsgUHM2cNSEQ4Q3P04DG+ tzSUmrWMlj+wrcDAIh2y4fpesZ/Uap04zL2tab0DF7zihuknqlqwddPKQp+rPpyqULdN9cDMJVZ24 hdMteKoDwmHHS+G1f9HpbPcgmGVWvjgTvWqjcqD/q0lV0yb8AjQCXqwi60Y16dk0HndHDAOTsptuJ 86SE482Q==; Received: from mail-wm1-x34a.google.com ([2a00:1450:4864:20::34a]) by casper.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1raEPM-0000000Ga0R-49rO for linux-arm-kernel@lists.infradead.org; Wed, 14 Feb 2024 12:30:41 +0000 Received: by mail-wm1-x34a.google.com with SMTP id 5b1f17b1804b1-40e4478a3afso3594385e9.1 for ; Wed, 14 Feb 2024 04:30:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1707913829; x=1708518629; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=7fAeJkvIpRC6qpF8jpEwHWSgHc+l/OrvPGb3S3qTTzU=; b=stjX60P18l57zDegmgPTfJKhgcWs5zNZLGH1VO5hIPTcHsHKV3JC6Bc8ug2alOOzax 7JR4gbN0WgGfMJnMBDPzkX64WumKGPHbZqoreJitEvhym5JA24xYuYb227J4J/BeLVyD 38DadUu7Xf3O5Lx0mfeaJCfd1kfz304DYHKpNp8sepDUhv1VyqsWf93pWCI57AL3jOmV Tp8wVu0qEw69JlxL0CaaQdkA4SDIK4UH7w5REmxy7dxLDsnLZy8rwrF2nSTqf+NT1mhk ARElIf8wfzONGhu3NIYtXn3w9rMuXMtj8GYcomVOaywkF9Qo9kJaeh094aYXjtEACpX1 p+3g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707913829; x=1708518629; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=7fAeJkvIpRC6qpF8jpEwHWSgHc+l/OrvPGb3S3qTTzU=; b=TlXDKHBYFaUYrE1k96do5gdRBSmbv61Wk9lfbPnfbno7Tyd66G+i7AEe/82Wzp0WXj YlHFiZ4z39Xaab8GlX7sns/dEz+hq134jqjrejmn4xXpRhF/HmM9PZfI/Hw4GK9LgT6a PTiMHaXBa3KDA1RwW7OGMFaKhjqzb4kafCZ6vGzHuBmF3tQZOTprtPogezL35THXzJJD 3Q/owr4Mj1w2+M1kCPHZiOnhKJmp40o/Dl8T2jTb5tTH02KJdH/Ngu+DXGZZHPE63mjA mIVppFMpPYzZ9IM/yGMRhCmqaXYGgGZpURt1hyMSvFEVT5C9lvhVMkFGgt7xX/MCbeyT Y46w== X-Gm-Message-State: AOJu0Yw/GQw8ESEyovXwH3rp0fyuh9fNgjkjk6jU7a+TN8J8CQzA1s/e YA0fAnfTs8ski5UJkr/eSLpWKeM0dv8+OaibyF0cCgU5/l4zDCm2GkgHL/t7fXwalR5EaunSqvE 2xw9Wzz8XLE3zqBhVd17Jq8R18duSN+QwcK/eKRLl1Ili8szvrGcaCpYlLcIcYRw5m5HkPfiLUY UWrIu0Ay1v6ASX339a1q+IMfhz5SHKvIQribMIJHFf X-Google-Smtp-Source: AGHT+IFy13d/M5X47bBSjYv8vV7p8AI1yVfLeCMQZ9FtFnHUL3nzCb5XNG9nPkxav365BYjf+XXVmkqU X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:600c:519b:b0:411:bc74:195f with SMTP id fa27-20020a05600c519b00b00411bc74195fmr46895wmb.2.1707913829095; Wed, 14 Feb 2024 04:30:29 -0800 (PST) Date: Wed, 14 Feb 2024 13:29:08 +0100 In-Reply-To: <20240214122845.2033971-45-ardb+git@google.com> Mime-Version: 1.0 References: <20240214122845.2033971-45-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=2122; i=ardb@kernel.org; h=from:subject; bh=qFRKJ7gXmCU+cZPFl5JJUliUFkG3MloRwbrG4prZL4U=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIfXMJrGjd+f5nAzlSHyYXjLjZK1KzpzOplhO7bVPokLVJ y8NP9baUcrCIMbBICumyCIw+++7nacnStU6z5KFmcPKBDKEgYtTACbyah/D/6xln58HrOi5Gpcv c+dlOK/kpGcVGrxvOeuehQj+t69eFMTIsFaXlVl/0mnlIuUj6vZzrifvMnjXztN30rpn1/v0Lul 6fgA= X-Mailer: git-send-email 2.43.0.687.g38aa6559b0-goog Message-ID: <20240214122845.2033971-67-ardb+git@google.com> Subject: [PATCH v8 22/43] arm64: mm: avoid fixmap for early swapper_pg_dir updates From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240214_123038_461891_E16AF6AA X-CRM114-Status: GOOD ( 14.13 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel Early in the boot, when .rodata is still writable, we can poke swapper_pg_dir entries directly, and there is no need to go through the fixmap. After a future patch, we will enter the kernel with swapper_pg_dir already active, and early swapper_pg_dir updates for creating the fixmap page table hierarchy itself cannot go through the fixmap for obvious reaons. So let's keep track of whether rodata is writable, and update the descriptor directly in that case. As the same reasoning applies to early KASAN init, make the function noinstr as well. Signed-off-by: Ard Biesheuvel --- arch/arm64/mm/mmu.c | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-) diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index 14a62c773201..9758f7e3f4b6 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -55,6 +55,8 @@ EXPORT_SYMBOL(kimage_voffset); u32 __boot_cpu_mode[] = { BOOT_CPU_MODE_EL2, BOOT_CPU_MODE_EL1 }; +static bool rodata_is_rw __ro_after_init = true; + /* * The booting CPU updates the failed status @__early_cpu_boot_status, * with MMU turned off. @@ -71,10 +73,21 @@ EXPORT_SYMBOL(empty_zero_page); static DEFINE_SPINLOCK(swapper_pgdir_lock); static DEFINE_MUTEX(fixmap_lock); -void set_swapper_pgd(pgd_t *pgdp, pgd_t pgd) +void noinstr set_swapper_pgd(pgd_t *pgdp, pgd_t pgd) { pgd_t *fixmap_pgdp; + /* + * Don't bother with the fixmap if swapper_pg_dir is still mapped + * writable in the kernel mapping. + */ + if (rodata_is_rw) { + WRITE_ONCE(*pgdp, pgd); + dsb(ishst); + isb(); + return; + } + spin_lock(&swapper_pgdir_lock); fixmap_pgdp = pgd_set_fixmap(__pa_symbol(pgdp)); WRITE_ONCE(*fixmap_pgdp, pgd); @@ -628,6 +641,7 @@ void mark_rodata_ro(void) * to cover NOTES and EXCEPTION_TABLE. */ section_size = (unsigned long)__init_begin - (unsigned long)__start_rodata; + WRITE_ONCE(rodata_is_rw, false); update_mapping_prot(__pa_symbol(__start_rodata), (unsigned long)__start_rodata, section_size, PAGE_KERNEL_RO); From patchwork Wed Feb 14 12:29:09 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13556415 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 2B8EAC48BC1 for ; Wed, 14 Feb 2024 12:32:19 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=MV+FN3d6MsTjqX66SRjcElvv2+bbj4S7eYNUlqP/pSA=; b=PKKIDBdIAF7r2tzwQqps8usVRP uKb50VNzDof8Sgb8XehPMRHP5WYv1zMfOWyJ45j3ntZOZSbMG1TrDD1KF4efn7AubJywV2QICdTWS y+0s3ck2VkJcT9r+ptpA+LcsUU1wKBZu9zU5vgltgip+baxfaVMFhbAljQ8VChj3RgSOTgSMsDrfX HKXqG2ghwozfNLPRb1atewT5KnAkKeU1B3BiM0sZo9wSV0R31vOw3aTf9fpmvDl7WbS6VmJZppJuA /bLIOX/GxIPuKzeOLFOSXkf2snouH5xZD9fTBWPQnuaZc8Wa1yaIm2zobQk3SpMJdSAcaoICLCvnE CbBqVS+w==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1raEQo-0000000CpO2-0qUS; Wed, 14 Feb 2024 12:32:06 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1raEPQ-0000000ComX-0sZl for linux-arm-kernel@bombadil.infradead.org; Wed, 14 Feb 2024 12:30:40 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=yqvXscl46yUqhXdRTf9wVQwHr+fzxTY61p3J20yzKXU=; b=XggDKcJYWmR+S4u06TVRfsYhAm CeI5TKNnREOKh+hhY3FQUVH1SdasMlA31GzJzr8Dch/O7qSYlv7d23N97E7vxx50lV8g2XoM62fLK ZQfcahxXRLVNOvNbOzX3iFob0LTmKPp0ChmLzEBbREb+c3Zj3/CfFziXlysvRmCA6WgpB1w/+SCMU YV3iqYOtFS+ZFUNH3xnTov/rxKzDhFNcVIBnAS6eOp2Ebna7MmkZKstLt4BGOfF4doT2xFeTjeoqc n1rCPF8MM9INjTT/ph1ocMICJZ3eTC2aCX0Up6cGlxQ7zRRDgriCzfrttMD+G7OXouiy9k0/oKRow FwYf5BJg==; Received: from mail-yw1-x114a.google.com ([2607:f8b0:4864:20::114a]) by desiato.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1raEPK-0000000GE26-04M0 for linux-arm-kernel@lists.infradead.org; Wed, 14 Feb 2024 12:30:38 +0000 Received: by mail-yw1-x114a.google.com with SMTP id 00721157ae682-6077ec913f0so42267247b3.0 for ; Wed, 14 Feb 2024 04:30:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1707913831; x=1708518631; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=yqvXscl46yUqhXdRTf9wVQwHr+fzxTY61p3J20yzKXU=; b=kdgGsE9G7CX2X0zfubPdnW/wEiF0k+K2svs3rU96t84xOLQT+bqXaozTnDIKk7e8a9 CTlNhHRDyi4c13uWZ+SmlqKrESgbmaHiP5JgLMJkt3W+wmNHRM9/RT5DqjuRxW0cl+lQ nuGUTjpDGF0hl6U78e4I7IaZk/vwrjkAl7bbUqgQmuTDEwifaVdwOC/kF9r02Jfz/ru7 2qBRPX18P9upEjbs9BEHLpiqtuv7MVLv7WghkcYf244rT8GYddYK0RkI4baikuul4OzS 54fuC4wUZWKJnfVWS0f0QuX2iUqMU5Ckfz6xo9nlPNXgRnOKbLrDYV4WqR0bqjeUyQaJ JAgw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707913831; x=1708518631; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=yqvXscl46yUqhXdRTf9wVQwHr+fzxTY61p3J20yzKXU=; b=KEi4Kt1lycC5H/KPowfaOFNhLIeymr4QQY0vB3GI9gO3OCd6mjNI+O20h8g6iNn1/2 n5o+4kfQsxM0yLVj2qGNRb4waunbBNpK2onhUZ9grk1pNaj1Oj5JOJ3fGuXufevigf/9 ntnv/skZr+Two8pAYjAY04FcMQttod59S9K+T6f1FaxdFMPvXsMsVt3b1f71Ak1aZ0rT RYeP/o2JF8H/tPD6oTF9v+FDNRwYcH6q80Q7rQrWdAEKslickQYSToRxeXovtVsIxXOU u9ZM0m4fRbEoy7MJKyDsQV4QffPw8keCjNOm2lbab8pxRXYoXwpNF4fXsde5V5OSNYvH +anw== X-Gm-Message-State: AOJu0YyyRoXDx6abVBCrHCypxEgfdO5MP4Cz+nUZ/nYqViCR3ogo54e/ 07z/O1HxerW78belu7hmyOXrkOEY4GdinnqZ64NA955H+ec8PCyBXHMJZHWi142hx78EUmWdS4L DRABly6g8wBP/2CuwoX5e3FujHuCvlLdiDOpTld8DWTe8+wmskGSybtSBDffRJgbCoGGy5y0r69 QGNU7MThdiAKPPWho5sAdd1Z0cY6loOQ2Wcm/yA142 X-Google-Smtp-Source: AGHT+IF1SwWzmOnoPYIQJfyYiX0iTzomVfyLILurb8O/yMFrdi2mo0okYBwEi9brZ/0owoO5m61HorvC X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a0d:d449:0:b0:5ff:5866:bc37 with SMTP id w70-20020a0dd449000000b005ff5866bc37mr481213ywd.3.1707913831443; Wed, 14 Feb 2024 04:30:31 -0800 (PST) Date: Wed, 14 Feb 2024 13:29:09 +0100 In-Reply-To: <20240214122845.2033971-45-ardb+git@google.com> Mime-Version: 1.0 References: <20240214122845.2033971-45-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=10237; i=ardb@kernel.org; h=from:subject; bh=UXzkaevfXSZh6cTWenKefEdvFH1jy7wrm11g5S/rpLI=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIfXMJnGv3X7twXsabJcF73ApPqZtsW3PrazHvXpC3+d1/ vgRNT2jo5SFQYyDQVZMkUVg9t93O09PlKp1niULM4eVCWQIAxenAEykey8jw2nNiLciGe0T7s0+ /kxS1E57Z/xyI9GEhQErI+aqVr8P6GNkeLgsQGxp5AT/FvePxkkvS1d6ODqK9osa33yy49Gmi2k MjAA= X-Mailer: git-send-email 2.43.0.687.g38aa6559b0-goog Message-ID: <20240214122845.2033971-68-ardb+git@google.com> Subject: [PATCH v8 23/43] arm64: mm: omit redundant remap of kernel image From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240214_123034_328671_FDE9E70F X-CRM114-Status: GOOD ( 26.36 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel Now that the early kernel mapping is created with all the right attributes and segment boundaries, there is no longer a need to recreate it and switch to it. This also means we no longer have to copy the kasan shadow or some parts of the fixmap from one set of page tables to the other. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/fixmap.h | 1 - arch/arm64/include/asm/kasan.h | 2 - arch/arm64/include/asm/mmu.h | 2 +- arch/arm64/kernel/image-vars.h | 1 + arch/arm64/kernel/pi/map_kernel.c | 6 +- arch/arm64/mm/fixmap.c | 34 -------- arch/arm64/mm/kasan_init.c | 15 ---- arch/arm64/mm/mmu.c | 85 ++++---------------- 8 files changed, 21 insertions(+), 125 deletions(-) diff --git a/arch/arm64/include/asm/fixmap.h b/arch/arm64/include/asm/fixmap.h index 58c294a96676..8aabd45e9a13 100644 --- a/arch/arm64/include/asm/fixmap.h +++ b/arch/arm64/include/asm/fixmap.h @@ -100,7 +100,6 @@ enum fixed_addresses { #define FIXMAP_PAGE_IO __pgprot(PROT_DEVICE_nGnRE) void __init early_fixmap_init(void); -void __init fixmap_copy(pgd_t *pgdir); #define __early_set_fixmap __set_fixmap diff --git a/arch/arm64/include/asm/kasan.h b/arch/arm64/include/asm/kasan.h index 7eefc525a9df..e1b57c13f8a4 100644 --- a/arch/arm64/include/asm/kasan.h +++ b/arch/arm64/include/asm/kasan.h @@ -17,11 +17,9 @@ asmlinkage void kasan_early_init(void); void kasan_init(void); -void kasan_copy_shadow(pgd_t *pgdir); #else static inline void kasan_init(void) { } -static inline void kasan_copy_shadow(pgd_t *pgdir) { } #endif #endif diff --git a/arch/arm64/include/asm/mmu.h b/arch/arm64/include/asm/mmu.h index d0b8b4b413b6..65977c7783c5 100644 --- a/arch/arm64/include/asm/mmu.h +++ b/arch/arm64/include/asm/mmu.h @@ -110,7 +110,7 @@ static inline bool kaslr_requires_kpti(void) } #define INIT_MM_CONTEXT(name) \ - .pgd = init_pg_dir, + .pgd = swapper_pg_dir, #endif /* !__ASSEMBLY__ */ #endif diff --git a/arch/arm64/kernel/image-vars.h b/arch/arm64/kernel/image-vars.h index 941a14c05184..e140c5bda90b 100644 --- a/arch/arm64/kernel/image-vars.h +++ b/arch/arm64/kernel/image-vars.h @@ -55,6 +55,7 @@ PROVIDE(__pi_memstart_offset_seed = memstart_offset_seed); PROVIDE(__pi_init_idmap_pg_dir = init_idmap_pg_dir); PROVIDE(__pi_init_pg_dir = init_pg_dir); PROVIDE(__pi_init_pg_end = init_pg_end); +PROVIDE(__pi_swapper_pg_dir = swapper_pg_dir); PROVIDE(__pi__text = _text); PROVIDE(__pi__stext = _stext); diff --git a/arch/arm64/kernel/pi/map_kernel.c b/arch/arm64/kernel/pi/map_kernel.c index f86e878d366d..4b76a007a50d 100644 --- a/arch/arm64/kernel/pi/map_kernel.c +++ b/arch/arm64/kernel/pi/map_kernel.c @@ -124,8 +124,12 @@ static void __init map_kernel(u64 kaslr_offset, u64 va_offset, int root_level) text_prot, true, root_level); map_segment(init_pg_dir, NULL, va_offset, __inittext_begin, __inittext_end, text_prot, false, root_level); - dsb(ishst); } + + /* Copy the root page table to its final location */ + memcpy((void *)swapper_pg_dir + va_offset, init_pg_dir, PGD_SIZE); + dsb(ishst); + idmap_cpu_replace_ttbr1(swapper_pg_dir); } static void __init map_fdt(u64 fdt) diff --git a/arch/arm64/mm/fixmap.c b/arch/arm64/mm/fixmap.c index 6fc17b2e1714..9404f282f829 100644 --- a/arch/arm64/mm/fixmap.c +++ b/arch/arm64/mm/fixmap.c @@ -170,37 +170,3 @@ void *__init fixmap_remap_fdt(phys_addr_t dt_phys, int *size, pgprot_t prot) return dt_virt; } - -/* - * Copy the fixmap region into a new pgdir. - */ -void __init fixmap_copy(pgd_t *pgdir) -{ - if (!READ_ONCE(pgd_val(*pgd_offset_pgd(pgdir, FIXADDR_TOT_START)))) { - /* - * The fixmap falls in a separate pgd to the kernel, and doesn't - * live in the carveout for the swapper_pg_dir. We can simply - * re-use the existing dir for the fixmap. - */ - set_pgd(pgd_offset_pgd(pgdir, FIXADDR_TOT_START), - READ_ONCE(*pgd_offset_k(FIXADDR_TOT_START))); - } else if (CONFIG_PGTABLE_LEVELS > 3) { - pgd_t *bm_pgdp; - p4d_t *bm_p4dp; - pud_t *bm_pudp; - /* - * The fixmap shares its top level pgd entry with the kernel - * mapping. This can really only occur when we are running - * with 16k/4 levels, so we can simply reuse the pud level - * entry instead. - */ - BUG_ON(!IS_ENABLED(CONFIG_ARM64_16K_PAGES)); - bm_pgdp = pgd_offset_pgd(pgdir, FIXADDR_TOT_START); - bm_p4dp = p4d_offset(bm_pgdp, FIXADDR_TOT_START); - bm_pudp = pud_set_fixmap_offset(bm_p4dp, FIXADDR_TOT_START); - pud_populate(&init_mm, bm_pudp, lm_alias(bm_pmd)); - pud_clear_fixmap(); - } else { - BUG(); - } -} diff --git a/arch/arm64/mm/kasan_init.c b/arch/arm64/mm/kasan_init.c index 4c7ad574b946..89828ad2bca7 100644 --- a/arch/arm64/mm/kasan_init.c +++ b/arch/arm64/mm/kasan_init.c @@ -189,21 +189,6 @@ static void __init kasan_map_populate(unsigned long start, unsigned long end, kasan_pgd_populate(start & PAGE_MASK, PAGE_ALIGN(end), node, false); } -/* - * Copy the current shadow region into a new pgdir. - */ -void __init kasan_copy_shadow(pgd_t *pgdir) -{ - pgd_t *pgdp, *pgdp_new, *pgdp_end; - - pgdp = pgd_offset_k(KASAN_SHADOW_START); - pgdp_end = pgd_offset_k(KASAN_SHADOW_END); - pgdp_new = pgd_offset_pgd(pgdir, KASAN_SHADOW_START); - do { - set_pgd(pgdp_new, READ_ONCE(*pgdp)); - } while (pgdp++, pgdp_new++, pgdp != pgdp_end); -} - static void __init clear_pgds(unsigned long start, unsigned long end) { diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index 9758f7e3f4b6..3db40b517947 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -648,9 +648,9 @@ void mark_rodata_ro(void) debug_checkwx(); } -static void __init map_kernel_segment(pgd_t *pgdp, void *va_start, void *va_end, - pgprot_t prot, struct vm_struct *vma, - int flags, unsigned long vm_flags) +static void __init declare_vma(struct vm_struct *vma, + void *va_start, void *va_end, + unsigned long vm_flags) { phys_addr_t pa_start = __pa_symbol(va_start); unsigned long size = va_end - va_start; @@ -658,9 +658,6 @@ static void __init map_kernel_segment(pgd_t *pgdp, void *va_start, void *va_end, BUG_ON(!PAGE_ALIGNED(pa_start)); BUG_ON(!PAGE_ALIGNED(size)); - __create_pgd_mapping(pgdp, pa_start, (unsigned long)va_start, size, prot, - early_pgtable_alloc, flags); - if (!(vm_flags & VM_NO_GUARD)) size += PAGE_SIZE; @@ -673,12 +670,12 @@ static void __init map_kernel_segment(pgd_t *pgdp, void *va_start, void *va_end, vm_area_add_early(vma); } +#ifdef CONFIG_UNMAP_KERNEL_AT_EL0 static pgprot_t kernel_exec_prot(void) { return rodata_enabled ? PAGE_KERNEL_ROX : PAGE_KERNEL_EXEC; } -#ifdef CONFIG_UNMAP_KERNEL_AT_EL0 static int __init map_entry_trampoline(void) { int i; @@ -713,60 +710,17 @@ core_initcall(map_entry_trampoline); #endif /* - * Open coded check for BTI, only for use to determine configuration - * for early mappings for before the cpufeature code has run. - */ -static bool arm64_early_this_cpu_has_bti(void) -{ - u64 pfr1; - - if (!IS_ENABLED(CONFIG_ARM64_BTI_KERNEL)) - return false; - - pfr1 = __read_sysreg_by_encoding(SYS_ID_AA64PFR1_EL1); - return cpuid_feature_extract_unsigned_field(pfr1, - ID_AA64PFR1_EL1_BT_SHIFT); -} - -/* - * Create fine-grained mappings for the kernel. + * Declare the VMA areas for the kernel */ -static void __init map_kernel(pgd_t *pgdp) +static void __init declare_kernel_vmas(void) { - static struct vm_struct vmlinux_text, vmlinux_rodata, vmlinux_inittext, - vmlinux_initdata, vmlinux_data; - - /* - * External debuggers may need to write directly to the text - * mapping to install SW breakpoints. Allow this (only) when - * explicitly requested with rodata=off. - */ - pgprot_t text_prot = kernel_exec_prot(); - - /* - * If we have a CPU that supports BTI and a kernel built for - * BTI then mark the kernel executable text as guarded pages - * now so we don't have to rewrite the page tables later. - */ - if (arm64_early_this_cpu_has_bti()) - text_prot = __pgprot_modify(text_prot, PTE_GP, PTE_GP); + static struct vm_struct vmlinux_seg[KERNEL_SEGMENT_COUNT]; - /* - * Only rodata will be remapped with different permissions later on, - * all other segments are allowed to use contiguous mappings. - */ - map_kernel_segment(pgdp, _stext, _etext, text_prot, &vmlinux_text, 0, - VM_NO_GUARD); - map_kernel_segment(pgdp, __start_rodata, __inittext_begin, PAGE_KERNEL, - &vmlinux_rodata, NO_CONT_MAPPINGS, VM_NO_GUARD); - map_kernel_segment(pgdp, __inittext_begin, __inittext_end, text_prot, - &vmlinux_inittext, 0, VM_NO_GUARD); - map_kernel_segment(pgdp, __initdata_begin, __initdata_end, PAGE_KERNEL, - &vmlinux_initdata, 0, VM_NO_GUARD); - map_kernel_segment(pgdp, _data, _end, PAGE_KERNEL, &vmlinux_data, 0, 0); - - fixmap_copy(pgdp); - kasan_copy_shadow(pgdp); + declare_vma(&vmlinux_seg[0], _stext, _etext, VM_NO_GUARD); + declare_vma(&vmlinux_seg[1], __start_rodata, __inittext_begin, VM_NO_GUARD); + declare_vma(&vmlinux_seg[2], __inittext_begin, __inittext_end, VM_NO_GUARD); + declare_vma(&vmlinux_seg[3], __initdata_begin, __initdata_end, VM_NO_GUARD); + declare_vma(&vmlinux_seg[4], _data, _end, 0); } void __pi_map_range(u64 *pgd, u64 start, u64 end, u64 pa, pgprot_t prot, @@ -802,23 +756,12 @@ static void __init create_idmap(void) void __init paging_init(void) { - pgd_t *pgdp = pgd_set_fixmap(__pa_symbol(swapper_pg_dir)); - extern pgd_t init_idmap_pg_dir[]; - - map_kernel(pgdp); - map_mem(pgdp); - - pgd_clear_fixmap(); - - cpu_replace_ttbr1(lm_alias(swapper_pg_dir), init_idmap_pg_dir); - init_mm.pgd = swapper_pg_dir; - - memblock_phys_free(__pa_symbol(init_pg_dir), - __pa_symbol(init_pg_end) - __pa_symbol(init_pg_dir)); + map_mem(swapper_pg_dir); memblock_allow_resize(); create_idmap(); + declare_kernel_vmas(); } #ifdef CONFIG_MEMORY_HOTPLUG From patchwork Wed Feb 14 12:29:10 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13556546 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 947C0C48BED for ; Wed, 14 Feb 2024 13:45:58 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=Mn6gI9Y/MBYGEI4GX8jmj+Yhyi8d95eStEqiwOGdPFY=; b=H4qUkyl1Ddsz4/TScQt6cQZON/ 8X5P0SUsMIabFveWKgQjWAbWsvw7MxPRlkib0lyufjpfRzwizylthzFL/RL0z4CnHKZmUMXmpY+vd 31hWpvBUv9K2Ddj6RtxZkKJLKW86r+/a28945dWl+faId3nzyw0BzesR6EjuVWfncYiqs8ANtLRs5 3/x5j12vDK5o1yUx7/5bN0ca6JBhdhGIgG7xeL9WWSfqkF0WGlY3vuSRRHIavG7i8HI0/ikOQ4uPW GoTJwxmmbSAancdO48r/U1i77Cr/LXMcbemtBt1U0uvYoG//R2thPdhGELMn1dEvilpWBA4MLaYgx 5mbiwsww==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1raFa6-0000000D3fr-3nbb; Wed, 14 Feb 2024 13:45:46 +0000 Received: from mail-wr1-x44a.google.com ([2a00:1450:4864:20::44a]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1raEPL-0000000Coi0-2Ike for linux-arm-kernel@lists.infradead.org; Wed, 14 Feb 2024 12:30:59 +0000 Received: by mail-wr1-x44a.google.com with SMTP id ffacd0b85a97d-33ceca744d9so305844f8f.0 for ; Wed, 14 Feb 2024 04:30:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1707913833; x=1708518633; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=kg1V7uGCx9MXi7hNzcs4diUXOO3ns/zLfMxNWcLXxZM=; b=dLdHKIJiW7qOnxrC9iOyB/l5ZD01YoCs31g1vAmLTocJ18SJ/NJqMAHvFhaeqEwIZT WCQHcoLO4yan50YfZSWE9EnyWI82dSpDfhbkrB38ZpoYT1pFm6ZYCdQT8FPXT6SauL+k SaKFGquqic5CI4OlwNoX9A/AQksEjyw3JTfyFJRr65l7EcvF/ahBK1KhRYjOR+5VGvGp 0z39Rq9mdVWDCRdAbzQQe3brvkTgl8Ao/GtTMXrbWOTdNssIDsU04gK1r8EM7WysVDTN R2KLv7u1IQV9OnndrF1wFH+p027a7I6/YNMTGN4s0RtvBiV5yAISmA5FBvq3CWXo8/Lx fIUg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707913833; x=1708518633; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=kg1V7uGCx9MXi7hNzcs4diUXOO3ns/zLfMxNWcLXxZM=; b=JxWRM+mCO6YuYumkQyRhW+O9CMZ6AnopYe/S+UOtSqd/JfxqAHGPg23oegfILqXvNR 1cdfB9930mqkmz44yDvaqwvcXP76H+10kqPVnv8uZiwVOIug9Z4sZ8YXKRxWtMSR9Z/z tUm5c/0GkR7/6pE6hnogSJIceBxAVSMxGqNY1v9SnCZ7XE9bHAc15xPoCqtRN+mGlQm5 geIFPaz8uVCJMwRzIQ8cMDTEKmu/eryA72VpaKQPjLHpAEIQF9BMezSt6YX0T4j4c8Eu hXaIz4lWWpmhNoIIa5UY64Z9ZYf2A+exCfA2Gd3ICgRITeshMhyi2VQGuePlAAr7IgBS d7uw== X-Gm-Message-State: AOJu0Yz1GXSZtmjuDZNMiRAbXUAx2y3lNWhmdGhZhpuDZzjbkHQtOr07 tIh694RTokFwvNB+ADiccf2CdSSWPhVYRXDA0Vj8RMplqX+f2Ff/3pUyYl5B76Rno6p1zufaa// Zx89fMVRrLJtjBW3pLMgi53q1iPx6zOxX/RWdIyUPe7GevEQ+PmQl/Mwg+OJimJbE5WHmIMi/Qh RL2Cmx7OrNKamcCqCfTILeaOpvBpcz0LnZuJsOCuX6 X-Google-Smtp-Source: AGHT+IE+xENdySUH6V+v2sJslbdXVgrEnUISViK16+tiGTYdTOTraXBn+zrDDtI5WFbcVExkvwyrmHEN X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:6000:71a:b0:33b:6825:a2e0 with SMTP id bs26-20020a056000071a00b0033b6825a2e0mr5384wrb.4.1707913833494; Wed, 14 Feb 2024 04:30:33 -0800 (PST) Date: Wed, 14 Feb 2024 13:29:10 +0100 In-Reply-To: <20240214122845.2033971-45-ardb+git@google.com> Mime-Version: 1.0 References: <20240214122845.2033971-45-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=3830; i=ardb@kernel.org; h=from:subject; bh=Ltx26cIfG6L9BcoBE0qx7Z/3lmDf9RTveKbmNM5D33k=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIfXMJgnHeofIK08k4t21naMOzJhQHCjA3fWTNbq1/MXT7 TNPPE/oKGVhEONgkBVTZBGY/ffdztMTpWqdZ8nCzGFlAhnCwMUpABNJEmNkWK46vyTGseX+OpvC gMML1HobuX4aFJ32kY/XlVY1NXk3nZFhs5HIJO2Oy3mbWNWnap2uyI9IvjfXmP1g3n/da5eYdEI ZAQ== X-Mailer: git-send-email 2.43.0.687.g38aa6559b0-goog Message-ID: <20240214122845.2033971-69-ardb+git@google.com> Subject: [PATCH v8 24/43] arm64: Revert "mm: provide idmap pointer to cpu_replace_ttbr1()" From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240214_043036_354116_4B484426 X-CRM114-Status: GOOD ( 14.85 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel This reverts commit 1682c45b920643c, which is no longer needed now that we create the permanent kernel mapping directly during early boot. This is a RINO (revert in name only) given that some of the code has moved around, but the changes are straight-forward. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/mmu_context.h | 17 ++++++----------- arch/arm64/mm/kasan_init.c | 4 ++-- arch/arm64/mm/mmu.c | 4 ++-- 3 files changed, 10 insertions(+), 15 deletions(-) diff --git a/arch/arm64/include/asm/mmu_context.h b/arch/arm64/include/asm/mmu_context.h index a8a89a0f2867..c768d16b81a4 100644 --- a/arch/arm64/include/asm/mmu_context.h +++ b/arch/arm64/include/asm/mmu_context.h @@ -108,18 +108,13 @@ static inline void cpu_uninstall_idmap(void) cpu_switch_mm(mm->pgd, mm); } -static inline void __cpu_install_idmap(pgd_t *idmap) +static inline void cpu_install_idmap(void) { cpu_set_reserved_ttbr0(); local_flush_tlb_all(); cpu_set_idmap_tcr_t0sz(); - cpu_switch_mm(lm_alias(idmap), &init_mm); -} - -static inline void cpu_install_idmap(void) -{ - __cpu_install_idmap(idmap_pg_dir); + cpu_switch_mm(lm_alias(idmap_pg_dir), &init_mm); } /* @@ -146,21 +141,21 @@ static inline void cpu_install_ttbr0(phys_addr_t ttbr0, unsigned long t0sz) isb(); } -void __cpu_replace_ttbr1(pgd_t *pgdp, pgd_t *idmap, bool cnp); +void __cpu_replace_ttbr1(pgd_t *pgdp, bool cnp); static inline void cpu_enable_swapper_cnp(void) { - __cpu_replace_ttbr1(lm_alias(swapper_pg_dir), idmap_pg_dir, true); + __cpu_replace_ttbr1(lm_alias(swapper_pg_dir), true); } -static inline void cpu_replace_ttbr1(pgd_t *pgdp, pgd_t *idmap) +static inline void cpu_replace_ttbr1(pgd_t *pgdp) { /* * Only for early TTBR1 replacement before cpucaps are finalized and * before we've decided whether to use CNP. */ WARN_ON(system_capabilities_finalized()); - __cpu_replace_ttbr1(pgdp, idmap, false); + __cpu_replace_ttbr1(pgdp, false); } /* diff --git a/arch/arm64/mm/kasan_init.c b/arch/arm64/mm/kasan_init.c index 89828ad2bca7..a86ab99587c9 100644 --- a/arch/arm64/mm/kasan_init.c +++ b/arch/arm64/mm/kasan_init.c @@ -225,7 +225,7 @@ static void __init kasan_init_shadow(void) */ memcpy(tmp_pg_dir, swapper_pg_dir, sizeof(tmp_pg_dir)); dsb(ishst); - cpu_replace_ttbr1(lm_alias(tmp_pg_dir), idmap_pg_dir); + cpu_replace_ttbr1(lm_alias(tmp_pg_dir)); clear_pgds(KASAN_SHADOW_START, KASAN_SHADOW_END); @@ -261,7 +261,7 @@ static void __init kasan_init_shadow(void) PAGE_KERNEL_RO)); memset(kasan_early_shadow_page, KASAN_SHADOW_INIT, PAGE_SIZE); - cpu_replace_ttbr1(lm_alias(swapper_pg_dir), idmap_pg_dir); + cpu_replace_ttbr1(lm_alias(swapper_pg_dir)); } static void __init kasan_init_depth(void) diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index 3db40b517947..a3d23da92d87 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -1445,7 +1445,7 @@ void ptep_modify_prot_commit(struct vm_area_struct *vma, unsigned long addr, pte * Atomically replaces the active TTBR1_EL1 PGD with a new VA-compatible PGD, * avoiding the possibility of conflicting TLB entries being allocated. */ -void __cpu_replace_ttbr1(pgd_t *pgdp, pgd_t *idmap, bool cnp) +void __cpu_replace_ttbr1(pgd_t *pgdp, bool cnp) { typedef void (ttbr_replace_func)(phys_addr_t); extern ttbr_replace_func idmap_cpu_replace_ttbr1; @@ -1460,7 +1460,7 @@ void __cpu_replace_ttbr1(pgd_t *pgdp, pgd_t *idmap, bool cnp) replace_phys = (void *)__pa_symbol(idmap_cpu_replace_ttbr1); - __cpu_install_idmap(idmap); + cpu_install_idmap(); /* * We really don't want to take *any* exceptions while TTBR1 is From patchwork Wed Feb 14 12:29:11 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13556416 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id B3B80C48BC1 for ; Wed, 14 Feb 2024 12:32:22 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=WdzdNCPsWLDeW/rDWjlpaQzxmKTChTbwRQnufqmrrys=; b=kpbpCYj+a0vxFBGIbQLL/5oPhb pN5hYrgVUW2/5qL2fOKrnm0iUbLnm0lSttuvIn+Pb9I45GCO+jNF3Jhy36NwRLd2Mi/GhOyRFLAJy +oylmR8ui9g5gnyslvty393hbToWo9Gw0GGjxMRA/L3zs0eydZ5JwEXQBBfoKjYEGnMP2pD8wGOH2 uHwCjgduKEqTAUFaGU/7co15pdHgXj5NSnk6gqtpaUS0966QxAPXxnOH5U17uHmRRYoYDMGwsupU+ /frH7nvg1qGWOYR8eA6ODpXN5kYOyzRuai4B8pP0EkJ5rwrR8AcJ65AVjhxVlH6jGfkaInRcqXlY/ d/gju10Q==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1raEQr-0000000CpPg-2lVz; Wed, 14 Feb 2024 12:32:09 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1raEPT-0000000CopK-0q9M for linux-arm-kernel@bombadil.infradead.org; Wed, 14 Feb 2024 12:30:43 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=nK12yNXLs+EEiFX+OJdo6kPMOjaXyo97bMa/fX0wvqY=; b=lKT8YArxAwy2j7bEVjQzCYjvuA F8j1rJjhwOXdYFXt9Wgt5FVmupYP3XsSA/1FU4NVq0EjWfOqzHIcp3Mq0kGS0pga/ocf2f6eF9QvF OX/UvpElXZOmfoHkRrKUnsGvp/3lR77qRLY1SWp1SgCuXBkC0MiZ8WRUS69xLUk3jxnDSU1PmT+RJ kZYJG3K8Ioz1vPTTcjz8iOL7mh2Rtw+SP1ctgLDrbmAcf20GPfw8sT4xZl6vtbOs4dVKKmxexy0Ww xCz0dXBFvSc6k0LGPuBNrZOyAIC3us9w6drFLdH24Duj4IT5raxuGxyBz1eDbHfvDEs1LmMvtU9wb R8xhotOQ==; Received: from mail-wr1-x44a.google.com ([2a00:1450:4864:20::44a]) by desiato.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1raEPO-0000000GE3H-0IHV for linux-arm-kernel@lists.infradead.org; Wed, 14 Feb 2024 12:30:42 +0000 Received: by mail-wr1-x44a.google.com with SMTP id ffacd0b85a97d-33b2238ceceso2436400f8f.3 for ; Wed, 14 Feb 2024 04:30:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1707913836; x=1708518636; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=nK12yNXLs+EEiFX+OJdo6kPMOjaXyo97bMa/fX0wvqY=; b=pskp+jwjtlA/X5x6kV1uyhtfijznUeM1mxGwljKSoB6ToCleu95XydTTAga9/VCjVh 3Lvv9L7FZnP//yBnDir3/AmbcjjX8AbVJuuiBRtvVEy3IcfvBDx9Df+1oC4EuxWMbl7F wjNg9cuvnLAych4lcMs8JG3g0SRCb7MiArYusVv2gwwsQPR+VrOYP1Js18Wzgk64njxD I8BTL+l3K/cJL2r/bRlNCDqKViJt5pXwt/eac7TDWL5lJxDDa8pxFd6lnxZMtqU5oDOA eR9mWJY9Wrx0wEmkBSRpKkeXWCo/P33QAwkNLQWtZf9+BAMdS6jxsp+Qw9X2FksdZJme q25g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707913836; x=1708518636; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=nK12yNXLs+EEiFX+OJdo6kPMOjaXyo97bMa/fX0wvqY=; b=Z2+2ROTn/h3Iwk5iZ1HXArRNupre7we4XMZCt9oU/GBf7ky3EzijeHepYEn0yXLhho +Th0soG7ntnmma+xDQ9IoiSAiszw6RHZ8OZMojrzQ61OaCcGlZ0Etg0/1TeZWej1WGGu swiy9ezxWxf1RR9LIWqFIhZjpYm52NEZhWI8IwwOmnK+p+fpW64+4yG1PNIwem6bVIOk kvLDZjvr3/zD0/n6PinYb1MVAc+hav3Gp9aGsLy1DmGG6La4Fh4pk7lXozgYr94hBmCk V1M7n1Xvwmebfl4d8gVyJs+dBa94h9xep1NLQ01ABfauenA5BV7TjZSlRQ8xMJ1IuFwU CqIg== X-Gm-Message-State: AOJu0Yy6fGHlHlH+2NXYEWnYcqqMIuUK1+FMIXHmfeqsB1agutd8M2Pn by7jN6nGfcxdXoXWuwaL33r36ZNUniMJuWYI7WUqk38kezWXogGuCUKhXsbvKrHcoSBpuxhLzpJ FK78qI1uAYiFxIfz6xyix/r9F5uiMCUyp1dHh9Ucg/bl4wLKCpnaFL9PpcMAMZMaOhjpes4ORxj ntwzoICKtuFlbgarCFl06oczguET3uVA73ptRFOfC2 X-Google-Smtp-Source: AGHT+IFfLt4aWkGenSOZNiWfsjzkWuu6pvBPIAJJnVYxAcgJOyeUdSy7327DOxGWleIUAYlH6Gskx5tn X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a5d:6e51:0:b0:33c:e36a:5324 with SMTP id j17-20020a5d6e51000000b0033ce36a5324mr3165wrz.2.1707913835550; Wed, 14 Feb 2024 04:30:35 -0800 (PST) Date: Wed, 14 Feb 2024 13:29:11 +0100 In-Reply-To: <20240214122845.2033971-45-ardb+git@google.com> Mime-Version: 1.0 References: <20240214122845.2033971-45-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=10210; i=ardb@kernel.org; h=from:subject; bh=xvmS+exhEE1+bzLfyG/gprvQGJEXnuuOxi51VB8dpZ4=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIfXMJsmH1c8ObNieL87opbV/B9sa68lC90+Gbm6fsmxD/ jHLtGn5HaUsDGIcDLJiiiwCs/++23l6olSt8yxZmDmsTCBDGLg4BWAi224yMnyZdJF71oqe1gau v2KR7jejjr5tPvTpluJPzTpW58W2rB2MDF1nGRuXpT5t2/E2urcg18ngmODX4CYviSTbbwFZAU0 TuQA= X-Mailer: git-send-email 2.43.0.687.g38aa6559b0-goog Message-ID: <20240214122845.2033971-70-ardb+git@google.com> Subject: [PATCH v8 25/43] arm64: mm: Handle LVA support as a CPU feature From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240214_123038_608220_500CC59E X-CRM114-Status: GOOD ( 24.81 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel Currently, we detect CPU support for 52-bit virtual addressing (LVA) extremely early, before creating the kernel page tables or enabling the MMU. We cannot override the feature this early, and so large virtual addressing is always enabled on CPUs that implement support for it if the software support for it was enabled at build time. It also means we rely on non-trivial code in asm to deal with this feature. Given that both the ID map and the TTBR1 mapping of the kernel image are guaranteed to be 48-bit addressable, it is not actually necessary to enable support this early, and instead, we can model it as a CPU feature. That way, we can rely on code patching to get the correct TCR.T1SZ values programmed on secondary boot and resume from suspend. On the primary boot path, we simply enable the MMU with 48-bit virtual addressing initially, and update TCR.T1SZ if LVA is supported from C code, right before creating the kernel mapping. Given that TTBR1 still points to reserved_pg_dir at this point, updating TCR.T1SZ should be safe without the need for explicit TLB maintenance. Since this gets rid of all accesses to the vabits_actual variable from asm code that occurred before TCR.T1SZ had been programmed, we no longer have a need for this variable, and we can replace it with a C expression that produces the correct value directly, based on the value of TCR.T1SZ. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/cpufeature.h | 9 ++++++ arch/arm64/include/asm/memory.h | 13 ++++++++- arch/arm64/kernel/cpufeature.c | 13 +++++++++ arch/arm64/kernel/head.S | 29 +++++--------------- arch/arm64/kernel/image-vars.h | 1 - arch/arm64/kernel/pi/map_kernel.c | 3 ++ arch/arm64/kernel/sleep.S | 3 -- arch/arm64/mm/mmu.c | 5 ---- arch/arm64/mm/proc.S | 9 +++--- arch/arm64/tools/cpucaps | 1 + 10 files changed, 49 insertions(+), 37 deletions(-) diff --git a/arch/arm64/include/asm/cpufeature.h b/arch/arm64/include/asm/cpufeature.h index e3edae1825f3..4f4dc5496ee3 100644 --- a/arch/arm64/include/asm/cpufeature.h +++ b/arch/arm64/include/asm/cpufeature.h @@ -995,6 +995,15 @@ static inline bool cpu_has_pac(void) &id_aa64isar2_override); } +static inline bool cpu_has_lva(void) +{ + u64 mmfr2; + + mmfr2 = read_sysreg_s(SYS_ID_AA64MMFR2_EL1); + return cpuid_feature_extract_unsigned_field(mmfr2, + ID_AA64MMFR2_EL1_VARange_SHIFT); +} + #endif /* __ASSEMBLY__ */ #endif diff --git a/arch/arm64/include/asm/memory.h b/arch/arm64/include/asm/memory.h index 60904a6c4b42..9680d7444b3b 100644 --- a/arch/arm64/include/asm/memory.h +++ b/arch/arm64/include/asm/memory.h @@ -209,9 +209,20 @@ #include #include #include +#include + +static inline u64 __pure read_tcr(void) +{ + u64 tcr; + + // read_sysreg() uses asm volatile, so avoid it here + asm("mrs %0, tcr_el1" : "=r"(tcr)); + return tcr; +} #if VA_BITS > 48 -extern u64 vabits_actual; +// For reasons of #include hell, we can't use TCR_T1SZ_OFFSET/TCR_T1SZ_MASK here +#define vabits_actual (64 - ((read_tcr() >> 16) & 63)) #else #define vabits_actual ((u64)VA_BITS) #endif diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c index 7064cf13f226..8eb8c7f7b317 100644 --- a/arch/arm64/kernel/cpufeature.c +++ b/arch/arm64/kernel/cpufeature.c @@ -2692,6 +2692,19 @@ static const struct arm64_cpu_capabilities arm64_features[] = { .type = ARM64_CPUCAP_SYSTEM_FEATURE, .matches = has_lpa2, }, +#ifdef CONFIG_ARM64_VA_BITS_52 + { + .desc = "52-bit Virtual Addressing (LVA)", + .capability = ARM64_HAS_VA52, + .type = ARM64_CPUCAP_BOOT_CPU_FEATURE, + .sys_reg = SYS_ID_AA64MMFR2_EL1, + .sign = FTR_UNSIGNED, + .field_width = 4, + .field_pos = ID_AA64MMFR2_EL1_VARange_SHIFT, + .matches = has_cpuid_feature, + .min_field_value = ID_AA64MMFR2_EL1_VARange_52, + }, +#endif {}, }; diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S index 545b5d8976f4..e25351addfd0 100644 --- a/arch/arm64/kernel/head.S +++ b/arch/arm64/kernel/head.S @@ -80,7 +80,6 @@ * x19 primary_entry() .. start_kernel() whether we entered with the MMU on * x20 primary_entry() .. __primary_switch() CPU boot mode * x21 primary_entry() .. start_kernel() FDT pointer passed at boot in x0 - * x25 primary_entry() .. start_kernel() supported VA size */ SYM_CODE_START(primary_entry) bl record_mmu_state @@ -125,14 +124,6 @@ SYM_CODE_START(primary_entry) * On return, the CPU will be ready for the MMU to be turned on and * the TCR will have been set. */ -#if VA_BITS > 48 - mrs_s x0, SYS_ID_AA64MMFR2_EL1 - tst x0, ID_AA64MMFR2_EL1_VARange_MASK - mov x0, #VA_BITS - mov x25, #VA_BITS_MIN - csel x25, x25, x0, eq - mov x0, x25 -#endif bl __cpu_setup // initialise processor b __primary_switch SYM_CODE_END(primary_entry) @@ -242,11 +233,6 @@ SYM_FUNC_START_LOCAL(__primary_switched) mov x0, x20 bl set_cpu_boot_mode_flag -#if VA_BITS > 48 - adr_l x8, vabits_actual // Set this early so KASAN early init - str x25, [x8] // ... observes the correct value - dc civac, x8 // Make visible to booting secondaries -#endif #if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS) bl kasan_early_init #endif @@ -376,10 +362,13 @@ SYM_FUNC_START_LOCAL(secondary_startup) * Common entry point for secondary CPUs. */ mov x20, x0 // preserve boot mode + +#ifdef CONFIG_ARM64_VA_BITS_52 +alternative_if ARM64_HAS_VA52 bl __cpu_secondary_check52bitva -#if VA_BITS > 48 - ldr_l x0, vabits_actual +alternative_else_nop_endif #endif + bl __cpu_setup // initialise processor adrp x1, swapper_pg_dir adrp x2, idmap_pg_dir @@ -482,12 +471,8 @@ SYM_FUNC_START(__enable_mmu) ret SYM_FUNC_END(__enable_mmu) +#ifdef CONFIG_ARM64_VA_BITS_52 SYM_FUNC_START(__cpu_secondary_check52bitva) -#if VA_BITS > 48 - ldr_l x0, vabits_actual - cmp x0, #52 - b.ne 2f - mrs_s x0, SYS_ID_AA64MMFR2_EL1 and x0, x0, ID_AA64MMFR2_EL1_VARange_MASK cbnz x0, 2f @@ -498,9 +483,9 @@ SYM_FUNC_START(__cpu_secondary_check52bitva) wfi b 1b -#endif 2: ret SYM_FUNC_END(__cpu_secondary_check52bitva) +#endif SYM_FUNC_START_LOCAL(__no_granule_support) /* Indicate that this CPU can't boot and is stuck in the kernel */ diff --git a/arch/arm64/kernel/image-vars.h b/arch/arm64/kernel/image-vars.h index e140c5bda90b..2b9d702abe0f 100644 --- a/arch/arm64/kernel/image-vars.h +++ b/arch/arm64/kernel/image-vars.h @@ -36,7 +36,6 @@ PROVIDE(__pi___memcpy = __pi_memcpy); PROVIDE(__pi___memmove = __pi_memmove); PROVIDE(__pi___memset = __pi_memset); -PROVIDE(__pi_vabits_actual = vabits_actual); PROVIDE(__pi_id_aa64isar1_override = id_aa64isar1_override); PROVIDE(__pi_id_aa64isar2_override = id_aa64isar2_override); PROVIDE(__pi_id_aa64mmfr1_override = id_aa64mmfr1_override); diff --git a/arch/arm64/kernel/pi/map_kernel.c b/arch/arm64/kernel/pi/map_kernel.c index 4b76a007a50d..1853825aa29d 100644 --- a/arch/arm64/kernel/pi/map_kernel.c +++ b/arch/arm64/kernel/pi/map_kernel.c @@ -165,6 +165,9 @@ asmlinkage void __init early_map_kernel(u64 boot_status, void *fdt) chosen = fdt_path_offset(fdt, chosen_str); init_feature_override(boot_status, fdt, chosen); + if (VA_BITS > VA_BITS_MIN && cpu_has_lva()) + sysreg_clear_set(tcr_el1, TCR_T1SZ_MASK, TCR_T1SZ(VA_BITS)); + /* * The virtual KASLR displacement modulo 2MiB is decided by the * physical placement of the image, as otherwise, we might not be able diff --git a/arch/arm64/kernel/sleep.S b/arch/arm64/kernel/sleep.S index 2aa5129d8253..f093cdf71be1 100644 --- a/arch/arm64/kernel/sleep.S +++ b/arch/arm64/kernel/sleep.S @@ -102,9 +102,6 @@ SYM_CODE_START(cpu_resume) mov x0, xzr bl init_kernel_el mov x19, x0 // preserve boot mode -#if VA_BITS > 48 - ldr_l x0, vabits_actual -#endif bl __cpu_setup /* enable the MMU early - so we can access sleep_save_stash by va */ adrp x1, swapper_pg_dir diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index a3d23da92d87..ba00d0205447 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -45,11 +45,6 @@ #define NO_CONT_MAPPINGS BIT(1) #define NO_EXEC_MAPPINGS BIT(2) /* assumes FEAT_HPDS is not used */ -#if VA_BITS > 48 -u64 vabits_actual __ro_after_init = VA_BITS_MIN; -EXPORT_SYMBOL(vabits_actual); -#endif - u64 kimage_voffset __ro_after_init; EXPORT_SYMBOL(kimage_voffset); diff --git a/arch/arm64/mm/proc.S b/arch/arm64/mm/proc.S index 55c366dbda8f..d104ddab26a4 100644 --- a/arch/arm64/mm/proc.S +++ b/arch/arm64/mm/proc.S @@ -397,8 +397,6 @@ SYM_FUNC_END(idmap_kpti_install_ng_mappings) * * Initialise the processor for turning the MMU on. * - * Input: - * x0 - actual number of VA bits (ignored unless VA_BITS > 48) * Output: * Return in x0 the value of the SCTLR_EL1 register. */ @@ -422,16 +420,17 @@ SYM_FUNC_START(__cpu_setup) mair .req x17 tcr .req x16 mov_q mair, MAIR_EL1_SET - mov_q tcr, TCR_T0SZ(IDMAP_VA_BITS) | TCR_T1SZ(VA_BITS) | TCR_CACHE_FLAGS | \ + mov_q tcr, TCR_T0SZ(IDMAP_VA_BITS) | TCR_T1SZ(VA_BITS_MIN) | TCR_CACHE_FLAGS | \ TCR_SMP_FLAGS | TCR_TG_FLAGS | TCR_KASLR_FLAGS | TCR_ASID16 | \ TCR_TBI0 | TCR_A1 | TCR_KASAN_SW_FLAGS | TCR_MTE_FLAGS tcr_clear_errata_bits tcr, x9, x5 #ifdef CONFIG_ARM64_VA_BITS_52 - sub x9, xzr, x0 - add x9, x9, #64 + mov x9, #64 - VA_BITS +alternative_if ARM64_HAS_VA52 tcr_set_t1sz tcr, x9 +alternative_else_nop_endif #endif /* diff --git a/arch/arm64/tools/cpucaps b/arch/arm64/tools/cpucaps index b912b1409fc0..b370d808b3ec 100644 --- a/arch/arm64/tools/cpucaps +++ b/arch/arm64/tools/cpucaps @@ -50,6 +50,7 @@ HAS_STAGE2_FWB HAS_TCR2 HAS_TIDCP1 HAS_TLB_RANGE +HAS_VA52 HAS_VIRT_HOST_EXTN HAS_WFXT HW_DBM From patchwork Wed Feb 14 12:29:12 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13556421 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id D7815C48BC1 for ; Wed, 14 Feb 2024 12:32:25 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=RCNEgOH0+ePlFzddtC3zPUdsKD3PGUrVtU25Yo4n49k=; b=AXYR1GRh5xV3UMrq5+GEyfXixK RBBe/bEfrkNXmhXZgGCn3e17dn/HIB0DbvdLH0JhuozXNOjxW4XXOmpi76Qo65YFfF3ZvT8AJVF5f jWCAUCBWH18IAA67TG0eY7wBy/+35QSP4NFfXswFhEd+V25FCg2cwg87uAZas09wgSI1yB0yDGay3 Ix5YmbAKhDN2OrEkPR/0ENgNUXlYLcQRjikodIswo3st4QKVnFCCkOSpKhtw4FP7EVQTRwqZVu5uQ 0lyPcXIZTnft/UOiN73EjJke1VQ0mXm7l8YNvK/g5bDmmhUc8Dra6iKBiHsCMZCUd8lrW7bw6SGfE yz0ZvRqg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1raEQu-0000000CpRL-0rDI; Wed, 14 Feb 2024 12:32:12 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1raEPU-0000000Coqj-3O0S for linux-arm-kernel@bombadil.infradead.org; Wed, 14 Feb 2024 12:30:44 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=nuKS8y64zOxLGqqXlUa7rXS63dgERVoqcE7CokOkmPU=; b=TQWng2/pR59tLK66LLxtJka/lZ gtSjniN1uYcuwx9JGiYzyO4LLb5tjia2rjfR9oAu7kKbo38TvN70ij6y6gepTlwq0rgTYDEoYzAwj J2yGHsTVcxB6te/ypH7lLxolGup+RbyojjihX2io51KxwI4gUKLxnh9WgLyUSHAfT6ocSzemFWO3L 4LUoDzd5DxzTapXVuMA+q0hhZu/xpDYpt8dU2GCac5c/w3j8JofhYLlhPoS8UqkZRytTIlUrEX6pb XdSejLa3QrrrziWn1kpgl/9cC2T9jL6H+J8I0r9M7IDVQWGkYa8AgR8UXpUkYGCzBqoeo/T5dvNZS fbWhxPIA==; Received: from mail-wm1-x349.google.com ([2a00:1450:4864:20::349]) by desiato.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1raEPP-0000000GE4S-1Jce for linux-arm-kernel@lists.infradead.org; Wed, 14 Feb 2024 12:30:43 +0000 Received: by mail-wm1-x349.google.com with SMTP id 5b1f17b1804b1-40fd153b5c3so3279875e9.1 for ; Wed, 14 Feb 2024 04:30:39 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1707913838; x=1708518638; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=nuKS8y64zOxLGqqXlUa7rXS63dgERVoqcE7CokOkmPU=; b=IFnJfWcCXi1i4dCUawPZVxv7Tl38889yXcpU4QdkHlMlEPBjo0EI3vVdBphqVqkbQk wH5gWcqhgYADDl6uWTuPa317t+Eu0Q9IdG0RFWN55Zg+KF12fYsA5ZA4VGAk1omMXb61 kqfMU2L2WpOuFkgOdDxi64iNP48Eif7TiLkkMh16PC1yIVlHG1N9if4GlV9disUqJjrr 5AtXrpCgM8aqhTmOv5Q2wJ/flnFxJUAcx7D/L31WLtc/rABtP8Dn73pxWY6llllOMDhR C3YBhEkrhQ4b7+CEZp2PxJTXYN+WtOC78yFwK3XAzJGQRzdZNJftF12FAO4gbd3sUVsL SVmQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707913838; x=1708518638; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=nuKS8y64zOxLGqqXlUa7rXS63dgERVoqcE7CokOkmPU=; b=P7uJdhum2SUTjyawEHfXkeiX4/HdcIAJ0NRKDQ0GEB/9k/gxs20WwSE+GWxGbofqaW 6nXcEmrqlrv2apxsBIteruhEh2KQ1a5hwIhPMnHQWZYTrtDKaO4toL8bnzLtxFY1cnhh wvP7FKTbTq8mV3yk1gRNlw6bQahoW/eQ/Kkfmd38PuRZejLbqf1j7qSFA9Ft1dm30VQi QOOVjyBdCUx0ow0BGEzdAdYbAMXobKy1s4nXldHT6ipk01KElyWaP/f02OPbDKNh457E pgFQlYC+gpyJDB9nBhz7iDcnGmegJOJYjUr8AQRrbJprQQ5Km2DyD8PsI05m2iu3/bDU jGeQ== X-Gm-Message-State: AOJu0YxNAwUBNZpDsrTfdWAfXYbZBXOoiu/4bsSC70W675lMxwlZWVa4 g9gJAl1rZrItryndNZk5bxmLeNRDjEWEyVzwLHV3rOCLpv2dMBZ9z77PjjuIRe32JX2h/j4XIFP 8e/9fSk7ZnWVxVFofhWLSHaefiNmaB0taNFaBRgyJ8MPSexIm2PNZJRnUM6L+HvtWOpOqYFwdtW kDg7lRH/4gzPbFrFfjvUYJ3Pm/ZbepFTpH45bsNtce X-Google-Smtp-Source: AGHT+IFbTt880XqJiblCj703Fo9/VEONF5xI3o9W5xQLt4umZWh01WOdIGMJF1jUMIuCpCFWzukz+gnq X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:600c:3ba0:b0:411:c45a:3926 with SMTP id n32-20020a05600c3ba000b00411c45a3926mr44468wms.1.1707913837929; Wed, 14 Feb 2024 04:30:37 -0800 (PST) Date: Wed, 14 Feb 2024 13:29:12 +0100 In-Reply-To: <20240214122845.2033971-45-ardb+git@google.com> Mime-Version: 1.0 References: <20240214122845.2033971-45-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=6997; i=ardb@kernel.org; h=from:subject; bh=JARDPVhZMz1ESavwKrqfgoYfE1bGtajfQzEL+oG6cGE=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIfXMJqnL2zcoNv06EyzOEqS9+/j1vrPMPJnT5v+vssjcL vqx7HhCRykLgxgHg6yYIovA7L/vdp6eKFXrPEsWZg4rE8gQBi5OAZhIyWZGhnvVFw1+Mp2eNWWd /DXvDtWqS97v4kQSEi7vfqY76auaMj/D/zjVsAZh3a2LE300AjRtOK3XFd/LOBho+Lu05vmF9Ce e3AA= X-Mailer: git-send-email 2.43.0.687.g38aa6559b0-goog Message-ID: <20240214122845.2033971-71-ardb+git@google.com> Subject: [PATCH v8 26/43] arm64: mm: Add feature override support for LVA From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240214_123039_684782_B39AC1BB X-CRM114-Status: GOOD ( 21.90 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel Add support for overriding the VARange field of the MMFR2 CPU ID register. This permits the associated LVA feature to be overridden early enough for the boot code that creates the kernel mapping to take it into account. Given that LPA2 implies LVA, disabling the latter should disable the former as well. So override the ID_AA64MMFR0.TGran field of the current page size as well if it advertises support for 52-bit addressing. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/assembler.h | 17 ++++++----- arch/arm64/include/asm/cpufeature.h | 4 +++ arch/arm64/kernel/cpufeature.c | 8 +++-- arch/arm64/kernel/image-vars.h | 2 ++ arch/arm64/kernel/pi/idreg-override.c | 31 ++++++++++++++++++++ 5 files changed, 53 insertions(+), 9 deletions(-) diff --git a/arch/arm64/include/asm/assembler.h b/arch/arm64/include/asm/assembler.h index 6a467c694039..68a99b116256 100644 --- a/arch/arm64/include/asm/assembler.h +++ b/arch/arm64/include/asm/assembler.h @@ -576,18 +576,21 @@ alternative_endif .endm /* - * Offset ttbr1 to allow for 48-bit kernel VAs set with 52-bit PTRS_PER_PGD. + * If the kernel is built for 52-bit virtual addressing but the hardware only + * supports 48 bits, we cannot program the pgdir address into TTBR1 directly, + * but we have to add an offset so that the TTBR1 address corresponds with the + * pgdir entry that covers the lowest 48-bit addressable VA. + * * orr is used as it can cover the immediate value (and is idempotent). - * In future this may be nop'ed out when dealing with 52-bit kernel VAs. * ttbr: Value of ttbr to set, modified. */ .macro offset_ttbr1, ttbr, tmp #ifdef CONFIG_ARM64_VA_BITS_52 - mrs_s \tmp, SYS_ID_AA64MMFR2_EL1 - and \tmp, \tmp, #(0xf << ID_AA64MMFR2_EL1_VARange_SHIFT) - cbnz \tmp, .Lskipoffs_\@ - orr \ttbr, \ttbr, #TTBR1_BADDR_4852_OFFSET -.Lskipoffs_\@ : + mrs \tmp, tcr_el1 + and \tmp, \tmp, #TCR_T1SZ_MASK + cmp \tmp, #TCR_T1SZ(VA_BITS_MIN) + orr \tmp, \ttbr, #TTBR1_BADDR_4852_OFFSET + csel \ttbr, \tmp, \ttbr, eq #endif .endm diff --git a/arch/arm64/include/asm/cpufeature.h b/arch/arm64/include/asm/cpufeature.h index 4f4dc5496ee3..a2ac31aecdd9 100644 --- a/arch/arm64/include/asm/cpufeature.h +++ b/arch/arm64/include/asm/cpufeature.h @@ -906,7 +906,9 @@ static inline unsigned int get_vmid_bits(u64 mmfr1) s64 arm64_ftr_safe_value(const struct arm64_ftr_bits *ftrp, s64 new, s64 cur); struct arm64_ftr_reg *get_arm64_ftr_reg(u32 sys_id); +extern struct arm64_ftr_override id_aa64mmfr0_override; extern struct arm64_ftr_override id_aa64mmfr1_override; +extern struct arm64_ftr_override id_aa64mmfr2_override; extern struct arm64_ftr_override id_aa64pfr0_override; extern struct arm64_ftr_override id_aa64pfr1_override; extern struct arm64_ftr_override id_aa64zfr0_override; @@ -1000,6 +1002,8 @@ static inline bool cpu_has_lva(void) u64 mmfr2; mmfr2 = read_sysreg_s(SYS_ID_AA64MMFR2_EL1); + mmfr2 &= ~id_aa64mmfr2_override.mask; + mmfr2 |= id_aa64mmfr2_override.val; return cpuid_feature_extract_unsigned_field(mmfr2, ID_AA64MMFR2_EL1_VARange_SHIFT); } diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c index 8eb8c7f7b317..ed9670d8360c 100644 --- a/arch/arm64/kernel/cpufeature.c +++ b/arch/arm64/kernel/cpufeature.c @@ -655,7 +655,9 @@ static const struct arm64_ftr_bits ftr_raz[] = { #define ARM64_FTR_REG(id, table) \ __ARM64_FTR_REG_OVERRIDE(#id, id, table, &no_override) +struct arm64_ftr_override id_aa64mmfr0_override; struct arm64_ftr_override id_aa64mmfr1_override; +struct arm64_ftr_override id_aa64mmfr2_override; struct arm64_ftr_override id_aa64pfr0_override; struct arm64_ftr_override id_aa64pfr1_override; struct arm64_ftr_override id_aa64zfr0_override; @@ -719,10 +721,12 @@ static const struct __ftr_reg_entry { &id_aa64isar2_override), /* Op1 = 0, CRn = 0, CRm = 7 */ - ARM64_FTR_REG(SYS_ID_AA64MMFR0_EL1, ftr_id_aa64mmfr0), + ARM64_FTR_REG_OVERRIDE(SYS_ID_AA64MMFR0_EL1, ftr_id_aa64mmfr0, + &id_aa64mmfr0_override), ARM64_FTR_REG_OVERRIDE(SYS_ID_AA64MMFR1_EL1, ftr_id_aa64mmfr1, &id_aa64mmfr1_override), - ARM64_FTR_REG(SYS_ID_AA64MMFR2_EL1, ftr_id_aa64mmfr2), + ARM64_FTR_REG_OVERRIDE(SYS_ID_AA64MMFR2_EL1, ftr_id_aa64mmfr2, + &id_aa64mmfr2_override), ARM64_FTR_REG(SYS_ID_AA64MMFR3_EL1, ftr_id_aa64mmfr3), /* Op1 = 1, CRn = 0, CRm = 0 */ diff --git a/arch/arm64/kernel/image-vars.h b/arch/arm64/kernel/image-vars.h index 2b9d702abe0f..ff81f809a240 100644 --- a/arch/arm64/kernel/image-vars.h +++ b/arch/arm64/kernel/image-vars.h @@ -38,7 +38,9 @@ PROVIDE(__pi___memset = __pi_memset); PROVIDE(__pi_id_aa64isar1_override = id_aa64isar1_override); PROVIDE(__pi_id_aa64isar2_override = id_aa64isar2_override); +PROVIDE(__pi_id_aa64mmfr0_override = id_aa64mmfr0_override); PROVIDE(__pi_id_aa64mmfr1_override = id_aa64mmfr1_override); +PROVIDE(__pi_id_aa64mmfr2_override = id_aa64mmfr2_override); PROVIDE(__pi_id_aa64pfr0_override = id_aa64pfr0_override); PROVIDE(__pi_id_aa64pfr1_override = id_aa64pfr1_override); PROVIDE(__pi_id_aa64smfr0_override = id_aa64smfr0_override); diff --git a/arch/arm64/kernel/pi/idreg-override.c b/arch/arm64/kernel/pi/idreg-override.c index 1884bd936c0d..aad399796e81 100644 --- a/arch/arm64/kernel/pi/idreg-override.c +++ b/arch/arm64/kernel/pi/idreg-override.c @@ -59,6 +59,35 @@ static const struct ftr_set_desc mmfr1 __prel64_initconst = { }, }; + +static bool __init mmfr2_varange_filter(u64 val) +{ + int __maybe_unused feat; + + if (val) + return false; + +#ifdef CONFIG_ARM64_LPA2 + feat = cpuid_feature_extract_signed_field(read_sysreg(id_aa64mmfr0_el1), + ID_AA64MMFR0_EL1_TGRAN_SHIFT); + if (feat >= ID_AA64MMFR0_EL1_TGRAN_LPA2) { + id_aa64mmfr0_override.val |= + (ID_AA64MMFR0_EL1_TGRAN_LPA2 - 1) << ID_AA64MMFR0_EL1_TGRAN_SHIFT; + id_aa64mmfr0_override.mask |= 0xfU << ID_AA64MMFR0_EL1_TGRAN_SHIFT; + } +#endif + return true; +} + +static const struct ftr_set_desc mmfr2 __prel64_initconst = { + .name = "id_aa64mmfr2", + .override = &id_aa64mmfr2_override, + .fields = { + FIELD("varange", ID_AA64MMFR2_EL1_VARange_SHIFT, mmfr2_varange_filter), + {} + }, +}; + static bool __init pfr0_sve_filter(u64 val) { /* @@ -167,6 +196,7 @@ static const struct ftr_set_desc sw_features __prel64_initconst = { static const PREL64(const struct ftr_set_desc, reg) regs[] __prel64_initconst = { { &mmfr1 }, + { &mmfr2 }, { &pfr0 }, { &pfr1 }, { &isar1 }, @@ -192,6 +222,7 @@ static const struct { { "arm64.nomte", "id_aa64pfr1.mte=0" }, { "nokaslr", "arm64_sw.nokaslr=1" }, { "rodata=off", "arm64_sw.rodataoff=1" }, + { "arm64.nolva", "id_aa64mmfr2.varange=0" }, }; static int __init parse_hexdigit(const char *p, u64 *v) From patchwork Wed Feb 14 12:29:13 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13556422 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 47777C48BC1 for ; Wed, 14 Feb 2024 12:32:30 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=yY+jS3bFfP8VNDJiNrQgoLcYcqqXWBMRVGs8ovaGcQo=; b=YdRiJGkpI267OhxEVIvR2nQ0ny 7rYismJzjKOqIxTlzllMkOb18GbPEQwg2kqgzwY+ptKpdYNn7+Hb85Pm1ZdlHPkM97s87km+HtIuZ RFZRQyHHEudUAnc673djnNrFMQD2e1Yl+31cAhqUWLLlWyK4zBsvfE9KuWXiz3f+pHPNqDEmlZwNJ AYrIu+pZwb+7whR7hs/Uz4oLbL40KiKYhNINqJ+nrDwzde48BWAtNEAhZwhteJFaoCNqCHQr0JIBV cpk58JEflB3uOR1jFVN9nezAmMLY1MSb4t+HEl4L/HaptajQBpKJxCtWEdbOp5vmxKzNRugYG8QNr 4qKxPhwg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1raEQv-0000000CpST-2beY; Wed, 14 Feb 2024 12:32:13 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1raEPX-0000000CotU-3di8 for linux-arm-kernel@bombadil.infradead.org; Wed, 14 Feb 2024 12:30:48 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=UmoIXXu47giuVJ5/WB15T78TLfAAHMBF5RoN/vaIbRU=; b=lT/+dbJV2hBQytmcWkCuBSC7sd ZsA60tjC94Q8aG3kX4RYtpAEHadeVx0UZY4ewLO5sBNufPMVPDbDHw1te8gBKpFGRvG4d8BsS1D0W nj0KMsWLJRoLJnLABLgjUNHjdNaJKQ4dJSvyTRQ5E4oZ2p8Viqh+AWARYIYyUDEj9Nl+Z86koIsSK YEs6IbG5iwZavdSSyvCpvjN9sBl8875V37zBu7Rcdsy4ODOsF07XbM/Kt3buSOveHv3LLV6aSES6X 8pqgFTuuGWns2gOAGFqAI8EwqNAJ65mRmevwh7W7G6gSvqqIyODLkMW4EI8BRaKhAnRa6qsWPesoM rsGjW2Gg==; Received: from mail-yw1-f201.google.com ([209.85.128.201]) by desiato.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1raEPS-0000000GE5y-2LAq for linux-arm-kernel@lists.infradead.org; Wed, 14 Feb 2024 12:30:46 +0000 Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-607ac7dbd61so5960067b3.3 for ; Wed, 14 Feb 2024 04:30:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1707913840; x=1708518640; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=UmoIXXu47giuVJ5/WB15T78TLfAAHMBF5RoN/vaIbRU=; b=cvXZI5Tm8hloNeHdCHP43G+IX7g0J8Pr3orPsKweymuROeJk9bkVkdlcNWv3hs7044 tamjGWreWYNQhiZeoz05/aPQH+yTJuLWXJRRQu93e91dYDmF3A8cS1I/xHH9rZgnHwJl QvdIgWEn11Ms9JYyresdsfL+2L5GXQUrXwRiyYvXQcT/1JuKmep1ENBFTvucbBGJ/Kw3 iGbiiZ1fQZyYuFeWs+mw25wV8B85pogC1nemMwfNzSaxa7XttRrGQkyAqJANF67LDkru pPc2FyN6qU0cKBTlHZZJjE0i8SaqTIbGssplqklmIeW+biDYNkFAuU/vjnX7yNTWzCk8 oSDA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707913840; x=1708518640; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=UmoIXXu47giuVJ5/WB15T78TLfAAHMBF5RoN/vaIbRU=; b=WcNW6YtvSkR2CuwiyfjkN2e9IBxS5k4Hl8K+bX87vILIFwARl4TPTSSGPGYj+VDTxc t2ekcZ9rdank/TgTXlKPRl4EEW8bYGvpUJE2QzIiDwBoCa+KpFENOKs79N17VcA6pSnn ycRMU70wWRJ31wbuT2uUVv8TcZ0ykyXYhnMIKytTf4O62vqwnFU4oM+AqXXsm2joVJcV pcBak2OC0fmoGrpwT7VEOpVa/x8GgsdG2ORwET5/JbMGOZZk6xdFw28J9GqFMji05ZB0 otbAc2lwxxoh9IvC2eYeLKZmC9B6ExkvjQm/HOJ7jp+LTRktS01LJ6c9wlz6K1utUkyj WLSA== X-Gm-Message-State: AOJu0YzAtcG8DY38llIx9KOaB31lcjBbKyt3IOQxa8ymQIw/3Eq3A0Ti iVmOGxAR2Y6v1CAI/0iJDwtjsT4tNZTB2XD4O/33b1KL0UXp0xNM6JS4THupQBdOdiZSF9k4hkA +kSxSKlfWjw/EI1p8hCifFLs7nJ3RbYdmd1K7p554ywVlH/D19i5FaB5W48RVWtG3Q6GDsKUtzx wuh1gVuoEg18GnqW9qHMFfW8RLHkdRU3sytygkYhtQ X-Google-Smtp-Source: AGHT+IGdBb6dQYvzbp1+SKDT77XsP93KDTljffX9vj7hWypgEQW6cMiWl4W28ll5CeJCQ2rR7lGryyV5 X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a81:4f05:0:b0:607:b9f1:e821 with SMTP id d5-20020a814f05000000b00607b9f1e821mr30281ywb.5.1707913840128; Wed, 14 Feb 2024 04:30:40 -0800 (PST) Date: Wed, 14 Feb 2024 13:29:13 +0100 In-Reply-To: <20240214122845.2033971-45-ardb+git@google.com> Mime-Version: 1.0 References: <20240214122845.2033971-45-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=2581; i=ardb@kernel.org; h=from:subject; bh=nInT/OYTmfVDNGSlyD0iMsc5q3tEV1+nqq5c+zMgpP8=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIfXMJmnXzWEG7Hdn//Odddj+89z6C6aVL7bsy1b67em9s ExxV2Z1RykLgxgHg6yYIovA7L/vdp6eKFXrPEsWZg4rE8gQBi5OAZjINDdGhqcXlJhWFbN092pu 0n3lqGRlnhfYGDBJ3+hF+w2f9D0V7xn+2dRILnm/csakGEP7isnuLv7HZj+UufvgncW2yQFxr2z eswMA X-Mailer: git-send-email 2.43.0.687.g38aa6559b0-goog Message-ID: <20240214122845.2033971-72-ardb+git@google.com> Subject: [PATCH v8 27/43] arm64: Avoid #define'ing PTE_MAYBE_NG to 0x0 for asm use From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240214_123042_797134_B6417926 X-CRM114-Status: GOOD ( 15.90 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel The PROT_* macros resolve to expressions that are only valid in C and not in assembler, and so they are only usable from C code. Currently, we make an exception for the permission indirection init code in proc.S, which doesn't care about the bits that are conditionally set, and so we just #define PTE_MAYBE_NG to 0x0 for any assembler file that includes these definitions. This is dodgy because this means that PROT_NORMAL and friends is generally available in asm code, but defined in a way that deviates from the definition that C code will observe, which might lead to hard to diagnose issues down the road. So instead, #define PTE_MAYBE_NG only in the place where the PIE constants are evaluated, and #undef it again right after. This allows us to drop the #define from pgtable-prot.h, and avoid the risk of deviating definitions between asm and C. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/pgtable-prot.h | 4 ---- arch/arm64/mm/proc.S | 13 +++++++++++++ 2 files changed, 13 insertions(+), 4 deletions(-) diff --git a/arch/arm64/include/asm/pgtable-prot.h b/arch/arm64/include/asm/pgtable-prot.h index 483dbfa39c4c..63ced9ccec21 100644 --- a/arch/arm64/include/asm/pgtable-prot.h +++ b/arch/arm64/include/asm/pgtable-prot.h @@ -57,10 +57,6 @@ #define _PAGE_READONLY_EXEC (_PAGE_DEFAULT | PTE_USER | PTE_RDONLY | PTE_NG | PTE_PXN) #define _PAGE_EXECONLY (_PAGE_DEFAULT | PTE_RDONLY | PTE_NG | PTE_PXN) -#ifdef __ASSEMBLY__ -#define PTE_MAYBE_NG 0 -#endif - #ifndef __ASSEMBLY__ #include diff --git a/arch/arm64/mm/proc.S b/arch/arm64/mm/proc.S index d104ddab26a4..6e1b2bc41a9f 100644 --- a/arch/arm64/mm/proc.S +++ b/arch/arm64/mm/proc.S @@ -456,11 +456,24 @@ alternative_else_nop_endif ubfx x1, x1, #ID_AA64MMFR3_EL1_S1PIE_SHIFT, #4 cbz x1, .Lskip_indirection + /* + * The PROT_* macros describing the various memory types may resolve to + * C expressions if they include the PTE_MAYBE_* macros, and so they + * can only be used from C code. The PIE_E* constants below are also + * defined in terms of those macros, but will mask out those + * PTE_MAYBE_* constants, whether they are set or not. So #define them + * as 0x0 here so we can evaluate the PIE_E* constants in asm context. + */ + +#define PTE_MAYBE_NG 0 + mov_q x0, PIE_E0 msr REG_PIRE0_EL1, x0 mov_q x0, PIE_E1 msr REG_PIR_EL1, x0 +#undef PTE_MAYBE_NG + mov x0, TCR2_EL1x_PIE msr REG_TCR2_EL1, x0 From patchwork Wed Feb 14 12:29:14 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13556424 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 10E18C48BED for ; Wed, 14 Feb 2024 12:32:31 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=/FKJzN4r6vrai+0jXtqr07TZ0S0o6P24/yxjHEYNgUY=; b=XaQg1zwKcQkgZnyHMJd8xK8Y30 thwLeu5HWv9YpH1c7CVbXWOFSr4ORe+LGtxqRcjgw5xlvNd1endgMEst4B9aVzzPzxd3DAiUu4s7i 8IIe6s+F1LCWULKBf7ZGlIknS8HI/2Cy2Yom2mM3NU43Ux54ROnQfH5yYpDAQMFPWppnHlCG1Eq3W F8KK48+trQfjZqVb6BjJpuaIKlOJaNni56Ux9ROQF8TR2MVo6hcmYaowL1RKOYz8QZZ73G/v/5A4G vPHcE4w+5+cRJfFB2OLtX2BDDlPzyunzB4tnb3o3OFYll8rAvT+I8YqQ5qVV1OwIFmNk2xeAUDtMO KUXieV8A==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1raEQx-0000000CpTd-12Yb; Wed, 14 Feb 2024 12:32:15 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1raEPb-0000000Covk-1902 for linux-arm-kernel@bombadil.infradead.org; Wed, 14 Feb 2024 12:30:51 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=aaqe629abfAj9N4lOXbkCwKNsXMbBLkCq9K/J6q4R5I=; b=hGZznmPCuMSU0nzusfH8fOG1qv ydEwEnDgH++wjZb0u4SzjeFBNiawnqcugtKwMuf+3NLagQUBFYj+0CnVih4TUYiD57ArQ3sUTnAty s4Vg26XM45CXficfGkPZE8Djz5cu6not41/KRAzNRFF+uopXMnSsnY/XyFbG8uZhc9vwddp8P49X3 ZfWtiwfVhieOgOqPCVz2E1Z2Dtrjz5Ypiq487AL0RtorS1mJNEr2TNJuMXeFGmSv2d4XVtMmr15B8 dBVp3CD801VPDyRmlurXHgSu/+eWJzOuitn3B0sVqGLHqD0aV3S9TbukahW44ZM+EnTNSCJcmgKR1 EQ883XsA==; Received: from mail-wm1-f73.google.com ([209.85.128.73]) by desiato.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1raEPV-0000000GE6f-1TIe for linux-arm-kernel@lists.infradead.org; Wed, 14 Feb 2024 12:30:49 +0000 Received: by mail-wm1-f73.google.com with SMTP id 5b1f17b1804b1-411bff89ec8so8333435e9.3 for ; Wed, 14 Feb 2024 04:30:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1707913842; x=1708518642; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=aaqe629abfAj9N4lOXbkCwKNsXMbBLkCq9K/J6q4R5I=; b=j1s5tEbpzQna6dTR7rej3ls+rNmGUO0z1I4lY6ORojAU+V8W8Q2FBP46kuIy+lZNYd z9GREQksJ5kI4inQCP+EkQK8redOXuO+cClk4ITqV0aUtvk5Eew9Zbxw66iEfbj+Dvlz fHAk7Paei/a5fAAFiIvZi7iDVo7m/62gmoRrrH02g9luVKQyo0Z+I6slYi5flahq5w5M j6eiM85bwxZZBMrZhddc0TdEV1S951USvoxIRffXA9YZiMpecVR2DPBTtFL5AkpcrJnR eOD+XPTAslpfFolLaEFYvytwKtUN6O7tUf/Zh9UN2nzZzXdqqlEHyVtE0SGBhWlgYuY9 3TGA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707913842; x=1708518642; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=aaqe629abfAj9N4lOXbkCwKNsXMbBLkCq9K/J6q4R5I=; b=VNL+b57HZINpVEx9gXcrXBHqutPzzvW/DZyZWiI19NsP8goqh6UHZqInUbGz6Rvd80 U0794zrIF+pafm5OC+jdWHvSvQt+3GF4kTE3KM3K28rmrrirKtRQ2wm6DWe9DYppdZwM cNwXACfXopscTZr0IwMddJMhX7a8Eu9BbHd+ZeVVGZOAtvf6yaO2l93UcPTF0kCms+iH 0qeQr222sEvJuQS2mJBeZIKwos2XIz9Xr/MEtqfuOLT9b5pPCsreAFay5NybkIq6GL/X MCjuw5gBkPign2cDtHxHXPWMycqF7pci76gVdwDSK7+KvCIbBUezkJmWB3JKEplovdhx gbjw== X-Gm-Message-State: AOJu0YxcB7feKmqKdrDc2/o9TKcb3sA8W2RI01ehomqrk0IUQLBzKLNb OpQuBRyVsZDSNFMi5ju2XOTXV85MJxL6/RZmZIqnBeg3ff16b3Q/1ikpKfhCS2NBXfJy6ZBhOed 1LEZphl0J9ofq5zmxAcG67Mp27DqxiGWSQyKuVB4cEWZ6WtiWxAttX4XK8i2MFT53NYrxab06N7 Xs67JU1yPL8GmbBnYgbSb7Lpvb5M6DHtsRXkxY53mO X-Google-Smtp-Source: AGHT+IFUrxF9oGiPuAOc7Pl4kuQVk1vNFXIMNt0yc0AYAil8xt07yQSP5zb9AB1+BAvrQb+f1PIvx+08 X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:600c:54cd:b0:410:e78f:d7e5 with SMTP id iw13-20020a05600c54cd00b00410e78fd7e5mr34032wmb.6.1707913842282; Wed, 14 Feb 2024 04:30:42 -0800 (PST) Date: Wed, 14 Feb 2024 13:29:14 +0100 In-Reply-To: <20240214122845.2033971-45-ardb+git@google.com> Mime-Version: 1.0 References: <20240214122845.2033971-45-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=8056; i=ardb@kernel.org; h=from:subject; bh=audsUAQQJOswEvBjinBkMxWmo5zv+W6kQf900Wgjcaw=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIfXMJpltnFEqHpaWpSmBW2+nHbm8/w9r42/lafP1pym8t n021TC+o5SFQYyDQVZMkUVg9t93O09PlKp1niULM4eVCWQIAxenAEzkryDD/3qe7A6p8km5Ihrn WWR93hWfjf6ky1Qb+GGj6d2Krc3i7xgZFrMf+SCrq7Vn41O+2Kcqkhc3Njsu6Y3MuF5xp96mWs2 LEQA= X-Mailer: git-send-email 2.43.0.687.g38aa6559b0-goog Message-ID: <20240214122845.2033971-73-ardb+git@google.com> Subject: [PATCH v8 28/43] arm64: Add ESR decoding for exceptions involving translation level -1 From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240214_123045_886887_2B03B20F X-CRM114-Status: GOOD ( 17.49 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel The LPA2 feature introduces new FSC values to report abort exceptions related to translation level -1. Define these and wire them up. Reuse the new ESR FSC classification helpers that arrived via the KVM arm64 tree, and update the one for translation faults to check specifically for a translation fault at level -1. (Access flag or permission faults cannot occur at level -1 because they alway involve a descriptor at the superior level so changing those helpers is not needed). Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/esr.h | 13 ++++----- arch/arm64/include/asm/kvm_emulate.h | 10 ++----- arch/arm64/mm/fault.c | 30 +++++++------------- 3 files changed, 18 insertions(+), 35 deletions(-) diff --git a/arch/arm64/include/asm/esr.h b/arch/arm64/include/asm/esr.h index 353fe08546cf..81606bf7d5ac 100644 --- a/arch/arm64/include/asm/esr.h +++ b/arch/arm64/include/asm/esr.h @@ -117,15 +117,9 @@ #define ESR_ELx_FSC_ACCESS (0x08) #define ESR_ELx_FSC_FAULT (0x04) #define ESR_ELx_FSC_PERM (0x0C) -#define ESR_ELx_FSC_SEA_TTW0 (0x14) -#define ESR_ELx_FSC_SEA_TTW1 (0x15) -#define ESR_ELx_FSC_SEA_TTW2 (0x16) -#define ESR_ELx_FSC_SEA_TTW3 (0x17) +#define ESR_ELx_FSC_SEA_TTW(n) (0x14 + (n)) #define ESR_ELx_FSC_SECC (0x18) -#define ESR_ELx_FSC_SECC_TTW0 (0x1c) -#define ESR_ELx_FSC_SECC_TTW1 (0x1d) -#define ESR_ELx_FSC_SECC_TTW2 (0x1e) -#define ESR_ELx_FSC_SECC_TTW3 (0x1f) +#define ESR_ELx_FSC_SECC_TTW(n) (0x1c + (n)) /* ISS field definitions for Data Aborts */ #define ESR_ELx_ISV_SHIFT (24) @@ -394,6 +388,9 @@ static inline bool esr_is_data_abort(unsigned long esr) static inline bool esr_fsc_is_translation_fault(unsigned long esr) { + /* Translation fault, level -1 */ + if ((esr & ESR_ELx_FSC) == 0b101011) + return true; return (esr & ESR_ELx_FSC_TYPE) == ESR_ELx_FSC_FAULT; } diff --git a/arch/arm64/include/asm/kvm_emulate.h b/arch/arm64/include/asm/kvm_emulate.h index b804fe832184..6f5b41c70103 100644 --- a/arch/arm64/include/asm/kvm_emulate.h +++ b/arch/arm64/include/asm/kvm_emulate.h @@ -425,15 +425,9 @@ static __always_inline bool kvm_vcpu_abt_issea(const struct kvm_vcpu *vcpu) { switch (kvm_vcpu_trap_get_fault(vcpu)) { case ESR_ELx_FSC_EXTABT: - case ESR_ELx_FSC_SEA_TTW0: - case ESR_ELx_FSC_SEA_TTW1: - case ESR_ELx_FSC_SEA_TTW2: - case ESR_ELx_FSC_SEA_TTW3: + case ESR_ELx_FSC_SEA_TTW(-1) ... ESR_ELx_FSC_SEA_TTW(3): case ESR_ELx_FSC_SECC: - case ESR_ELx_FSC_SECC_TTW0: - case ESR_ELx_FSC_SECC_TTW1: - case ESR_ELx_FSC_SECC_TTW2: - case ESR_ELx_FSC_SECC_TTW3: + case ESR_ELx_FSC_SECC_TTW(-1) ... ESR_ELx_FSC_SECC_TTW(3): return true; default: return false; diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c index 55f6455a8284..60265ede48fe 100644 --- a/arch/arm64/mm/fault.c +++ b/arch/arm64/mm/fault.c @@ -257,16 +257,14 @@ static bool is_el1_data_abort(unsigned long esr) static inline bool is_el1_permission_fault(unsigned long addr, unsigned long esr, struct pt_regs *regs) { - unsigned long fsc_type = esr & ESR_ELx_FSC_TYPE; - if (!is_el1_data_abort(esr) && !is_el1_instruction_abort(esr)) return false; - if (fsc_type == ESR_ELx_FSC_PERM) + if (esr_fsc_is_permission_fault(esr)) return true; if (is_ttbr0_addr(addr) && system_uses_ttbr0_pan()) - return fsc_type == ESR_ELx_FSC_FAULT && + return esr_fsc_is_translation_fault(esr) && (regs->pstate & PSR_PAN_BIT); return false; @@ -279,8 +277,7 @@ static bool __kprobes is_spurious_el1_translation_fault(unsigned long addr, unsigned long flags; u64 par, dfsc; - if (!is_el1_data_abort(esr) || - (esr & ESR_ELx_FSC_TYPE) != ESR_ELx_FSC_FAULT) + if (!is_el1_data_abort(esr) || !esr_fsc_is_translation_fault(esr)) return false; local_irq_save(flags); @@ -301,7 +298,7 @@ static bool __kprobes is_spurious_el1_translation_fault(unsigned long addr, * treat the translation fault as spurious. */ dfsc = FIELD_GET(SYS_PAR_EL1_FST, par); - return (dfsc & ESR_ELx_FSC_TYPE) != ESR_ELx_FSC_FAULT; + return !esr_fsc_is_translation_fault(dfsc); } static void die_kernel_fault(const char *msg, unsigned long addr, @@ -368,11 +365,6 @@ static bool is_el1_mte_sync_tag_check_fault(unsigned long esr) return false; } -static bool is_translation_fault(unsigned long esr) -{ - return (esr & ESR_ELx_FSC_TYPE) == ESR_ELx_FSC_FAULT; -} - static void __do_kernel_fault(unsigned long addr, unsigned long esr, struct pt_regs *regs) { @@ -405,7 +397,7 @@ static void __do_kernel_fault(unsigned long addr, unsigned long esr, } else if (addr < PAGE_SIZE) { msg = "NULL pointer dereference"; } else { - if (is_translation_fault(esr) && + if (esr_fsc_is_translation_fault(esr) && kfence_handle_page_fault(addr, esr & ESR_ELx_WNR, regs)) return; @@ -782,18 +774,18 @@ static const struct fault_info fault_info[] = { { do_translation_fault, SIGSEGV, SEGV_MAPERR, "level 1 translation fault" }, { do_translation_fault, SIGSEGV, SEGV_MAPERR, "level 2 translation fault" }, { do_translation_fault, SIGSEGV, SEGV_MAPERR, "level 3 translation fault" }, - { do_bad, SIGKILL, SI_KERNEL, "unknown 8" }, + { do_page_fault, SIGSEGV, SEGV_ACCERR, "level 0 access flag fault" }, { do_page_fault, SIGSEGV, SEGV_ACCERR, "level 1 access flag fault" }, { do_page_fault, SIGSEGV, SEGV_ACCERR, "level 2 access flag fault" }, { do_page_fault, SIGSEGV, SEGV_ACCERR, "level 3 access flag fault" }, - { do_bad, SIGKILL, SI_KERNEL, "unknown 12" }, + { do_page_fault, SIGSEGV, SEGV_ACCERR, "level 0 permission fault" }, { do_page_fault, SIGSEGV, SEGV_ACCERR, "level 1 permission fault" }, { do_page_fault, SIGSEGV, SEGV_ACCERR, "level 2 permission fault" }, { do_page_fault, SIGSEGV, SEGV_ACCERR, "level 3 permission fault" }, { do_sea, SIGBUS, BUS_OBJERR, "synchronous external abort" }, { do_tag_check_fault, SIGSEGV, SEGV_MTESERR, "synchronous tag check fault" }, { do_bad, SIGKILL, SI_KERNEL, "unknown 18" }, - { do_bad, SIGKILL, SI_KERNEL, "unknown 19" }, + { do_sea, SIGKILL, SI_KERNEL, "level -1 (translation table walk)" }, { do_sea, SIGKILL, SI_KERNEL, "level 0 (translation table walk)" }, { do_sea, SIGKILL, SI_KERNEL, "level 1 (translation table walk)" }, { do_sea, SIGKILL, SI_KERNEL, "level 2 (translation table walk)" }, @@ -801,7 +793,7 @@ static const struct fault_info fault_info[] = { { do_sea, SIGBUS, BUS_OBJERR, "synchronous parity or ECC error" }, // Reserved when RAS is implemented { do_bad, SIGKILL, SI_KERNEL, "unknown 25" }, { do_bad, SIGKILL, SI_KERNEL, "unknown 26" }, - { do_bad, SIGKILL, SI_KERNEL, "unknown 27" }, + { do_sea, SIGKILL, SI_KERNEL, "level -1 synchronous parity error (translation table walk)" }, // Reserved when RAS is implemented { do_sea, SIGKILL, SI_KERNEL, "level 0 synchronous parity error (translation table walk)" }, // Reserved when RAS is implemented { do_sea, SIGKILL, SI_KERNEL, "level 1 synchronous parity error (translation table walk)" }, // Reserved when RAS is implemented { do_sea, SIGKILL, SI_KERNEL, "level 2 synchronous parity error (translation table walk)" }, // Reserved when RAS is implemented @@ -815,9 +807,9 @@ static const struct fault_info fault_info[] = { { do_bad, SIGKILL, SI_KERNEL, "unknown 38" }, { do_bad, SIGKILL, SI_KERNEL, "unknown 39" }, { do_bad, SIGKILL, SI_KERNEL, "unknown 40" }, - { do_bad, SIGKILL, SI_KERNEL, "unknown 41" }, + { do_bad, SIGKILL, SI_KERNEL, "level -1 address size fault" }, { do_bad, SIGKILL, SI_KERNEL, "unknown 42" }, - { do_bad, SIGKILL, SI_KERNEL, "unknown 43" }, + { do_translation_fault, SIGSEGV, SEGV_MAPERR, "level -1 translation fault" }, { do_bad, SIGKILL, SI_KERNEL, "unknown 44" }, { do_bad, SIGKILL, SI_KERNEL, "unknown 45" }, { do_bad, SIGKILL, SI_KERNEL, "unknown 46" }, From patchwork Wed Feb 14 12:29:15 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13556425 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 4A5D7C48BC1 for ; Wed, 14 Feb 2024 12:32:38 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=4JuopDHcRwnlISQH5prfVdc1YSTZ8UJ9DBnM0cDtVxM=; b=gHVNiEvxj5osHYxH8MaWsVc+6c yXu7UiAopWUgBoQBvFN/gWTxNOXsnmtzNeO/nxNDjnxPAtKMQr51xChpmHnKO5MZegXfu+Tb5RwMO mBsIYot5tIJUSZ6I9kZo+fQjEnlhIVrreQ8Ge1b0S7VlVC45s7c+lotTYRl7AXNBc8MvThb+Z9wPE /8q/VVxwpMdpr0KPhcfD5pvxWQe2oze/UA40RZR1LBLF96Nrctbgg3L7gXDp/jaCuUWB6wLOszN+H Q9t2P486MGBJZmR5h1z3WsUZfMvG4fZwM9NIw2HHhM7YsYc9AC3V5Gim78ysDjMK8M2fhVGCCqxYk qN7vrN2g==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1raER1-0000000CpWw-2GnK; Wed, 14 Feb 2024 12:32:19 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1raEPb-0000000Cow7-41nz for linux-arm-kernel@bombadil.infradead.org; Wed, 14 Feb 2024 12:30:52 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=HyvjalD/6fvbAX20kRuPTKXh7jyw9GTdF8mbD1bkAPI=; b=lU9Mb3IFcCVUGGmrARVZbvpgvn sEx5GBCGBkUTvX3h9pqZ8zUyjwrXLIqk9SiacAGALjnL6dVngf68jT61WE7F72owfYwivpQUCMM3R g3MBHRhF+4bnvmlDlFdPi8YVigHuA7oWDAXx+cEBFHVlrIJlQWHBpdRN/9GWfCODzJAp06YdQSPpy jOG+nd70qyR9ZKDr/t5jfWyuBNNCFoI+J+aywI1vw+ggUNKJe6olLUYJOaU3esGBevYUICwSLXwub 0bgvI0Tcg6yqKhFSofxuNqR7i8Tnh0NVVeQJ/8D3GV0D6coyLqKr6+A3eceFAJ+0wGLN5mVD37yFA 0m3QE+hg==; Received: from mail-wr1-x44a.google.com ([2a00:1450:4864:20::44a]) by desiato.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1raEPW-0000000GE7j-2KR5 for linux-arm-kernel@lists.infradead.org; Wed, 14 Feb 2024 12:30:50 +0000 Received: by mail-wr1-x44a.google.com with SMTP id ffacd0b85a97d-33cf46a8932so18309f8f.1 for ; Wed, 14 Feb 2024 04:30:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1707913845; x=1708518645; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=HyvjalD/6fvbAX20kRuPTKXh7jyw9GTdF8mbD1bkAPI=; b=Itw9P5w5WK99CidYdIWYD4kQL3HnqzEOVMh0JUQQB0OtgN6Qtmit+lBUFrq+wwWDum B27eit1tdoU+ks2JsnTr30IJ6R/EvhU1nJassIXIiBo56ncIcB0itDlLCPeIKUXZCFut rlGwe+1n3ooOH5SfLUk5O39/nh6IgZJ/NmXobW74yVxyWUY0+p66kAPJSiFZ2uc5584I LggwaMe4EJJqxdrJ4CDW3juQ1FTWpZ1d5JAI4rd37zPcUrzpKI0RIW2xbIcUsbdCEN44 Dsv8YsFiJ8F0crZ6XrRQsXFX5uWQC4nsdbsKvlF187TmmQcoHiZTrxbJC0xQtNv48/w8 VKtw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707913845; x=1708518645; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=HyvjalD/6fvbAX20kRuPTKXh7jyw9GTdF8mbD1bkAPI=; b=ARYi0o42H0YTrtdafSO0AsIUx6uy07RWedMBbAduA0uOVyzBP4azSnP0Pq1LaKR7J0 Un6UOtcAYx68SCfBRtZ6RmqJuaMtuxUh6CK1/KgCDgOT2CJ6URBaVPjhRwQeEtN/uzOG 4U2y+W+2sgo2e6Du3wUjWZU291+u/VlgItuHOpwfYeG+4v7s8Xs6thglz8KPNGvmUXLF +XZ6Hzr6dl1rkB/+vLDdfF/PUy8/Y8BcqHMLeIzotnI6QCpZJrOVLFAaA9XAyk5zZNIV SCf0WxuKFcaSg03DiKXklg3YyLum+7l19qz/mkkP5voVqQjecg+yuSYm2mzhY7tfB+8C kHpA== X-Gm-Message-State: AOJu0YzoU+PbkZwSMVkhBhzAwMF2VhEneRNF6+rvCDIcYT54TxroDBnT D2S28akMywRvenxhR4hszOo8MK3Axp3KZXAWLFZHo9AWQNKLyktt9O/tyedHBQdf25bbj3wexlQ wIhRl/UzXTm/4sbYNtVSA+UWIeVHADpRgg3HaAtmFgzkzmW/fXFtgvDD5ZaBPbz6csyQe9q8H5B 7RBw1EZ5WCLbCSeZuCDqMxlMSpPuBSjjpT+GHfIYw8 X-Google-Smtp-Source: AGHT+IEr4dzBT9iN6w4yezP7LsLZXoX626Bo3jX9QHhZlShcAr1vhCxGQ1PSv1QeE9OYYUVU9D3JyI+n X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:6000:1d89:b0:33c:e2f7:b157 with SMTP id bk9-20020a0560001d8900b0033ce2f7b157mr4361wrb.4.1707913844526; Wed, 14 Feb 2024 04:30:44 -0800 (PST) Date: Wed, 14 Feb 2024 13:29:15 +0100 In-Reply-To: <20240214122845.2033971-45-ardb+git@google.com> Mime-Version: 1.0 References: <20240214122845.2033971-45-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=3974; i=ardb@kernel.org; h=from:subject; bh=k4s2tDBZMYe2OfnHcnfaPU4Fz87tpYu4VFMcOiHTEr8=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIfXMJln726IbOB/K151v6jf5/vloxFeZvUF2S69vVPrwy vtLnkh4RykLgxgHg6yYIovA7L/vdp6eKFXrPEsWZg4rE8gQBi5OAZjI3hhGhkVTfkRktetMPXG5 z22ra7qzbPRErU8Hk1f800+VmbYoewbDP5UvWUzFrrUmpi+OzT1p119T7f/Xym6OdHvQ9YU9GzW EuAA= X-Mailer: git-send-email 2.43.0.687.g38aa6559b0-goog Message-ID: <20240214122845.2033971-74-ardb+git@google.com> Subject: [PATCH v8 29/43] arm64: mm: Wire up TCR.DS bit to PTE shareability fields From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240214_123046_933919_4704EEF6 X-CRM114-Status: GOOD ( 15.84 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel When LPA2 is enabled, bits 8 and 9 of page and block descriptors become part of the output address instead of carrying shareability attributes for the region in question. So avoid setting these bits if TCR.DS == 1, which means LPA2 is enabled. Signed-off-by: Ard Biesheuvel --- arch/arm64/Kconfig | 4 ++++ arch/arm64/include/asm/pgtable-hwdef.h | 1 + arch/arm64/include/asm/pgtable-prot.h | 16 ++++++++++++++-- arch/arm64/mm/mmap.c | 4 ++++ arch/arm64/mm/proc.S | 2 ++ 5 files changed, 25 insertions(+), 2 deletions(-) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index aa7c1d435139..8c2c36fffcf5 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -1377,6 +1377,10 @@ config ARM64_PA_BITS default 48 if ARM64_PA_BITS_48 default 52 if ARM64_PA_BITS_52 +config ARM64_LPA2 + def_bool y + depends on ARM64_PA_BITS_52 && !ARM64_64K_PAGES + choice prompt "Endianness" default CPU_LITTLE_ENDIAN diff --git a/arch/arm64/include/asm/pgtable-hwdef.h b/arch/arm64/include/asm/pgtable-hwdef.h index e4944d517c99..b770f98fc0b5 100644 --- a/arch/arm64/include/asm/pgtable-hwdef.h +++ b/arch/arm64/include/asm/pgtable-hwdef.h @@ -284,6 +284,7 @@ #define TCR_E0PD1 (UL(1) << 56) #define TCR_TCMA0 (UL(1) << 57) #define TCR_TCMA1 (UL(1) << 58) +#define TCR_DS (UL(1) << 59) /* * TTBR. diff --git a/arch/arm64/include/asm/pgtable-prot.h b/arch/arm64/include/asm/pgtable-prot.h index 63ced9ccec21..dd9ee67d1d87 100644 --- a/arch/arm64/include/asm/pgtable-prot.h +++ b/arch/arm64/include/asm/pgtable-prot.h @@ -30,8 +30,8 @@ #define _PROT_DEFAULT (PTE_TYPE_PAGE | PTE_AF | PTE_SHARED) #define _PROT_SECT_DEFAULT (PMD_TYPE_SECT | PMD_SECT_AF | PMD_SECT_S) -#define PROT_DEFAULT (_PROT_DEFAULT | PTE_MAYBE_NG) -#define PROT_SECT_DEFAULT (_PROT_SECT_DEFAULT | PMD_MAYBE_NG) +#define PROT_DEFAULT (PTE_TYPE_PAGE | PTE_MAYBE_NG | PTE_MAYBE_SHARED | PTE_AF) +#define PROT_SECT_DEFAULT (PMD_TYPE_SECT | PMD_MAYBE_NG | PMD_MAYBE_SHARED | PMD_SECT_AF) #define PROT_DEVICE_nGnRnE (PROT_DEFAULT | PTE_PXN | PTE_UXN | PTE_WRITE | PTE_ATTRINDX(MT_DEVICE_nGnRnE)) #define PROT_DEVICE_nGnRE (PROT_DEFAULT | PTE_PXN | PTE_UXN | PTE_WRITE | PTE_ATTRINDX(MT_DEVICE_nGnRE)) @@ -67,7 +67,19 @@ extern bool arm64_use_ng_mappings; #define PTE_MAYBE_NG (arm64_use_ng_mappings ? PTE_NG : 0) #define PMD_MAYBE_NG (arm64_use_ng_mappings ? PMD_SECT_NG : 0) +#ifndef CONFIG_ARM64_LPA2 #define lpa2_is_enabled() false +#define PTE_MAYBE_SHARED PTE_SHARED +#define PMD_MAYBE_SHARED PMD_SECT_S +#else +static inline bool __pure lpa2_is_enabled(void) +{ + return read_tcr() & TCR_DS; +} + +#define PTE_MAYBE_SHARED (lpa2_is_enabled() ? 0 : PTE_SHARED) +#define PMD_MAYBE_SHARED (lpa2_is_enabled() ? 0 : PMD_SECT_S) +#endif /* * If we have userspace only BTI we don't want to mark kernel pages diff --git a/arch/arm64/mm/mmap.c b/arch/arm64/mm/mmap.c index 645fe60d000f..642bdf908b22 100644 --- a/arch/arm64/mm/mmap.c +++ b/arch/arm64/mm/mmap.c @@ -73,6 +73,10 @@ static int __init adjust_protection_map(void) protection_map[VM_EXEC | VM_SHARED] = PAGE_EXECONLY; } + if (lpa2_is_enabled()) + for (int i = 0; i < ARRAY_SIZE(protection_map); i++) + pgprot_val(protection_map[i]) &= ~PTE_SHARED; + return 0; } arch_initcall(adjust_protection_map); diff --git a/arch/arm64/mm/proc.S b/arch/arm64/mm/proc.S index 6e1b2bc41a9f..7c46f8cfd6ae 100644 --- a/arch/arm64/mm/proc.S +++ b/arch/arm64/mm/proc.S @@ -466,6 +466,7 @@ alternative_else_nop_endif */ #define PTE_MAYBE_NG 0 +#define PTE_MAYBE_SHARED 0 mov_q x0, PIE_E0 msr REG_PIRE0_EL1, x0 @@ -473,6 +474,7 @@ alternative_else_nop_endif msr REG_PIR_EL1, x0 #undef PTE_MAYBE_NG +#undef PTE_MAYBE_SHARED mov x0, TCR2_EL1x_PIE msr REG_TCR2_EL1, x0 From patchwork Wed Feb 14 12:29:16 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13556545 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 5D39AC48BC4 for ; Wed, 14 Feb 2024 13:45:58 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=SZDldf1thjDqVtUEB1g2hhQivmpncLv6uobqZIIeh90=; b=lm/EL9kVj3D3evgCnMM5SxTrmc VAjrovuLEB7mun1XVUkVuTsnXo4ePEq0Kb0Aoa0Xixxko5QscAaEz6oPKnXbzVj+nRZPYTeeg0nSq 5Nj1eiVpJPDC22M44g6Q9j3CYfoeAQV7J4jqdLuzM9uIKrTxR1QJTfomLleYkyzzFcutvjuog/Z6D DNoS8c08dVQJvGBtv+T9+idg+3JbDSc8JfogpZFfBYcJqh6t5Q9nh19iH+hL8cGubdI16qpHwDftq DeBuuE0P6kM6b/Sm8cS2rSU4499DR3z2xH/uLrGgeu6fCPG7NPYLdM4wPlxsiZdrVZA3QE5rlfGBQ 5ED0filw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1raFa6-0000000D3fY-1VaL; Wed, 14 Feb 2024 13:45:46 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1raEPg-0000000CozD-3Tp6 for linux-arm-kernel@bombadil.infradead.org; Wed, 14 Feb 2024 12:30:56 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=XS5iesF0HNBEmWWUWvb99bP+C1kfTRanM7cS9JJcyAQ=; b=hm75TOavJOGiifcjv2AR4ZQ4h+ IzA43zBP7pJRWmpyiM4ZfH83gjXXaGYmJRDlFTWU7L4bwYYJ8GjJ89YtJvpg8pbqfiNersiRPRExr kPrVRhy6TtizSniNPhqH4//9TwzRhOOAOqrO4IHb6ZYNlYD4kbp31jr4ImMSeSJmB16Aumiv/Th8U kR+r90ShsHza8iS/Y1Whb6ZafccmHF9OZw+XZUMS50bFPOH92hvw3VqFag+pqnn8ZFDVvEGi6qrgb ln/+4pD+X6zIfTc2cFah2cZImmKHxEfW8Rp3O46JD4AdJGD8wUZgVYvA9c5pz8ynG3uc1Y+pvyh8v T2b2Nrwg==; Received: from mail-yw1-x1149.google.com ([2607:f8b0:4864:20::1149]) by desiato.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1raEPZ-0000000GE8f-14mW for linux-arm-kernel@lists.infradead.org; Wed, 14 Feb 2024 12:30:53 +0000 Received: by mail-yw1-x1149.google.com with SMTP id 00721157ae682-5eba564eb3fso110708327b3.1 for ; Wed, 14 Feb 2024 04:30:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1707913847; x=1708518647; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=XS5iesF0HNBEmWWUWvb99bP+C1kfTRanM7cS9JJcyAQ=; b=lMY7EzYqj9lJ5/KiaASm3s2rj1fifnIwxHA9PeZWYu9XfI7S+nSO7YV3/W5qDEQJgB btZ9LNWlUcxMBp5BIn+YlD3Q/mhKDrwV1NO0tT0ohlew5Am6Qg7ltrhYy2TJcFr+4XJk mG/v0edgj0uYTUcVKbEN22j9HNDTp5yceaNPWSqCShZoPfxKA02KGvEOt+K6uK1JwF9i 4FOB8As/RsMfhaYURUqo6dYfUp7/AOdx+JewaHbPlxlwubtwEB+003b99ovplyB2uTLl OknJY3s+sG6GHR7O4C8rpwBal9VG0nCPCK/wys34utT7NF0bz7vdQg4yS4Tn8jAi2rxK 22qQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707913847; x=1708518647; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=XS5iesF0HNBEmWWUWvb99bP+C1kfTRanM7cS9JJcyAQ=; b=Gc0DDisuiO4IotSiyuZsFLmo3P66i6rLy8bV+AuUs9m+YboPhg1zoMWwFQMtnSCpSn LrBrZtVEStXCvxYX+fQY4Ch10g2dIsETam0G8XXk/bHn7Fv4LWyxRQgfiFczMGGQQCE/ ow0JNlZ2I71EDUO+jvXs5tOvcjiGizFVthNvuxyHGmtJGxqBKsNg2hgl7GbFn4jvq0I7 auRJ0Dk3bkJDdLl4/bwPBX1WFBy4rGWEexGkXjRGml7vkbzNw+HLEtFoIYv5CDxb1bqH QM4XfGQqDsmjWApuNNpCuRa+OnIxKbynvNeouNez/DQ9ZD1/caBFNKkUvtYw/mblCVIs 868w== X-Gm-Message-State: AOJu0Yy2d0pJOeGVEWuk585fi0AIOkCxftUwZn3+yCGBhDUL91hmsppZ HFipZTuHPxmhN2LAbXAV2nt4q1+QVFWep6ZiyIa7Vf4InaJ6Y+Q6d4lY6rInW+MFayTykC/AWyv Z+KB1SPs9gxGwebPtIWNjUZckk9QKb+mi6LrzsB/2/LTBsNxIUShmr+XWgAcYAxqgasbVCKujIF PvJNxF5YjWsKHn2Vmb0gopBoVYuiTsliQaVK/j9I5D X-Google-Smtp-Source: AGHT+IHI8MGiGaICtaaucPLtig05RYHAf3O3BfbZfPiJgfDSXWvX1mEoHpKVYS+ZekatiEXc2POsKGQ5 X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:6902:1826:b0:dcd:25be:aefb with SMTP id cf38-20020a056902182600b00dcd25beaefbmr494352ybb.13.1707913847174; Wed, 14 Feb 2024 04:30:47 -0800 (PST) Date: Wed, 14 Feb 2024 13:29:16 +0100 In-Reply-To: <20240214122845.2033971-45-ardb+git@google.com> Mime-Version: 1.0 References: <20240214122845.2033971-45-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=4645; i=ardb@kernel.org; h=from:subject; bh=bhq1UZsl34o3/V5QqWl176mqS7CVCTze3JDjfKMi/9U=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIfXMJjl9n1nLCmZoCa6X2SlmEX5/ln/P/wk/Jyx3e3wqK Pd87ePyjlIWBjEOBlkxRRaB2X/f7Tw9UarWeZYszBxWJpAhDFycAjARcX5GhiNcUVbSUxZlL+Lc k3fcVKOybKWSWTb3wVOd6qcObv3qzcnw3y0u74V215eVGRXN3Ht/+VlYTdx1v7uw93R5/afTEXZ lfAA= X-Mailer: git-send-email 2.43.0.687.g38aa6559b0-goog Message-ID: <20240214122845.2033971-75-ardb+git@google.com> Subject: [PATCH v8 30/43] arm64: mm: Add LPA2 support to phys<->pte conversion routines From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240214_123049_449157_76A07356 X-CRM114-Status: GOOD ( 15.39 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel In preparation for enabling LPA2 support, introduce the mask values for converting between physical addresses and their representations in a page table descriptor. While at it, move the pte_to_phys asm macro into its only user, so that we can freely modify it to use its input value register as a temp register. For LPA2, the PTE_ADDR_MASK contains two non-adjacent sequences of zero bits, which means it no longer fits into the immediate field of an ordinary ALU instruction. So let's redefine it to include the bits in between as well, and only use it when converting from physical address to PTE representation, where the distinction does not matter. Also update the name accordingly to emphasize this. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/assembler.h | 16 ++-------------- arch/arm64/include/asm/pgtable-hwdef.h | 10 +++++++--- arch/arm64/include/asm/pgtable.h | 5 +++-- arch/arm64/mm/proc.S | 8 ++++++++ 4 files changed, 20 insertions(+), 19 deletions(-) diff --git a/arch/arm64/include/asm/assembler.h b/arch/arm64/include/asm/assembler.h index 68a99b116256..7eedcb36ebe0 100644 --- a/arch/arm64/include/asm/assembler.h +++ b/arch/arm64/include/asm/assembler.h @@ -612,25 +612,13 @@ alternative_endif .macro phys_to_pte, pte, phys #ifdef CONFIG_ARM64_PA_BITS_52 - /* - * We assume \phys is 64K aligned and this is guaranteed by only - * supporting this configuration with 64K pages. - */ - orr \pte, \phys, \phys, lsr #36 - and \pte, \pte, #PTE_ADDR_MASK + orr \pte, \phys, \phys, lsr #PTE_ADDR_HIGH_SHIFT + and \pte, \pte, #PHYS_TO_PTE_ADDR_MASK #else mov \pte, \phys #endif .endm - .macro pte_to_phys, phys, pte - and \phys, \pte, #PTE_ADDR_MASK -#ifdef CONFIG_ARM64_PA_BITS_52 - orr \phys, \phys, \phys, lsl #PTE_ADDR_HIGH_SHIFT - and \phys, \phys, GENMASK_ULL(PHYS_MASK_SHIFT - 1, PAGE_SHIFT) -#endif - .endm - /* * tcr_clear_errata_bits - Clear TCR bits that trigger an errata on this CPU. */ diff --git a/arch/arm64/include/asm/pgtable-hwdef.h b/arch/arm64/include/asm/pgtable-hwdef.h index b770f98fc0b5..4426f48f2ae0 100644 --- a/arch/arm64/include/asm/pgtable-hwdef.h +++ b/arch/arm64/include/asm/pgtable-hwdef.h @@ -155,13 +155,17 @@ #define PTE_PXN (_AT(pteval_t, 1) << 53) /* Privileged XN */ #define PTE_UXN (_AT(pteval_t, 1) << 54) /* User XN */ -#define PTE_ADDR_LOW (((_AT(pteval_t, 1) << (48 - PAGE_SHIFT)) - 1) << PAGE_SHIFT) +#define PTE_ADDR_LOW (((_AT(pteval_t, 1) << (50 - PAGE_SHIFT)) - 1) << PAGE_SHIFT) #ifdef CONFIG_ARM64_PA_BITS_52 +#ifdef CONFIG_ARM64_64K_PAGES #define PTE_ADDR_HIGH (_AT(pteval_t, 0xf) << 12) -#define PTE_ADDR_MASK (PTE_ADDR_LOW | PTE_ADDR_HIGH) #define PTE_ADDR_HIGH_SHIFT 36 +#define PHYS_TO_PTE_ADDR_MASK (PTE_ADDR_LOW | PTE_ADDR_HIGH) #else -#define PTE_ADDR_MASK PTE_ADDR_LOW +#define PTE_ADDR_HIGH (_AT(pteval_t, 0x3) << 8) +#define PTE_ADDR_HIGH_SHIFT 42 +#define PHYS_TO_PTE_ADDR_MASK GENMASK_ULL(49, 8) +#endif #endif /* diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 522c21348ae8..61de7b1516bc 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -80,15 +80,16 @@ extern unsigned long empty_zero_page[PAGE_SIZE / sizeof(unsigned long)]; #ifdef CONFIG_ARM64_PA_BITS_52 static inline phys_addr_t __pte_to_phys(pte_t pte) { + pte_val(pte) &= ~PTE_MAYBE_SHARED; return (pte_val(pte) & PTE_ADDR_LOW) | ((pte_val(pte) & PTE_ADDR_HIGH) << PTE_ADDR_HIGH_SHIFT); } static inline pteval_t __phys_to_pte_val(phys_addr_t phys) { - return (phys | (phys >> PTE_ADDR_HIGH_SHIFT)) & PTE_ADDR_MASK; + return (phys | (phys >> PTE_ADDR_HIGH_SHIFT)) & PHYS_TO_PTE_ADDR_MASK; } #else -#define __pte_to_phys(pte) (pte_val(pte) & PTE_ADDR_MASK) +#define __pte_to_phys(pte) (pte_val(pte) & PTE_ADDR_LOW) #define __phys_to_pte_val(phys) (phys) #endif diff --git a/arch/arm64/mm/proc.S b/arch/arm64/mm/proc.S index 7c46f8cfd6ae..d03434b7bca5 100644 --- a/arch/arm64/mm/proc.S +++ b/arch/arm64/mm/proc.S @@ -205,6 +205,14 @@ SYM_FUNC_ALIAS(__pi_idmap_cpu_replace_ttbr1, idmap_cpu_replace_ttbr1) .pushsection ".idmap.text", "a" + .macro pte_to_phys, phys, pte + and \phys, \pte, #PTE_ADDR_LOW +#ifdef CONFIG_ARM64_PA_BITS_52 + and \pte, \pte, #PTE_ADDR_HIGH + orr \phys, \phys, \pte, lsl #PTE_ADDR_HIGH_SHIFT +#endif + .endm + .macro kpti_mk_tbl_ng, type, num_entries add end_\type\()p, cur_\type\()p, #\num_entries * 8 .Ldo_\type: From patchwork Wed Feb 14 12:29:17 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13556427 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 1C1C1C48BC1 for ; Wed, 14 Feb 2024 12:32:55 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=RsFPkjRBy4Xnzlnlfh+/xiWyoZTvBLKbddY5W6zfDAM=; b=XEMA3pj0GZC5nl2JOIhg+obNJ+ zEgx/XR66w5zhwGcv290P+o23hAkPeVfKpA3vK6UZc/vGUw/AFgn3TsH4O6RHyTYxfmEggqdvHnp+ OegACBTLSahui5tWPGi3aZzpOga5cnxQdTJXsKqKfpvYgHEgHN4eUvRy419ylxvLTC2waYLQ6cIQY F+h2jDcfFzbKNRyBiXlFjh5ZXVl1CEaTe+9VqVcoLXAaEAAw4YNB3dCx7sEPuV5ANo4XUw4OQd7wH 40tm8nWVWBz8kE5xaxv+WquTxfeYtP+zEk9Rw7h5XCVkxN2CUGAa7Ajum9vHK6APmLBdD8NVnPXe/ 9Wh/NpmQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1raERJ-0000000Cpn9-48Ag; Wed, 14 Feb 2024 12:32:38 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1raEPg-0000000CozI-4AUc for linux-arm-kernel@bombadil.infradead.org; Wed, 14 Feb 2024 12:30:57 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=p5mKfqmr3QrChRrscGODzWzPxAKG6aKZKMpL6RrB8mg=; b=EYWN080V8L9kn+UKp2IGGVFnJ0 rLoOiCfay0/rxS/pcD/QTuk0Dxl8RQOIsPuu9MrlkF6jscrmsYF7SCnHUjQeSErb1RI3lz9YUk4Jh QlzYsvcygvysvMiKkSMnjEzj91FkJQKDcE/OWWZPdUSxQvujb8/Pw1kAWX7F9PyfoDcNu1qnQqQWi v/F2fUdRlYwXekGHD3VW/m3hPe3Amf91XlqERtUZomUvl6TaFOw+Hz5UQIqyThn4r8llS9Qdi8nBU 7cyXe0TBJ+PyjTJM5Pgj5q9B2M5dZ5KRzuX7mR339QunMjlQ/QDWyZt/t0ZCEqvv9hnppBXF2IHyr WZErpYSQ==; Received: from mail-yw1-x1149.google.com ([2607:f8b0:4864:20::1149]) by desiato.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1raEPb-0000000GE9P-1Xst for linux-arm-kernel@lists.infradead.org; Wed, 14 Feb 2024 12:30:54 +0000 Received: by mail-yw1-x1149.google.com with SMTP id 00721157ae682-6047a047f58so98939637b3.3 for ; Wed, 14 Feb 2024 04:30:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1707913850; x=1708518650; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=p5mKfqmr3QrChRrscGODzWzPxAKG6aKZKMpL6RrB8mg=; b=tWD24mAbN3p5/Wopn/oNU5PKSvRT/UHCe7K8xFGdln/5htOBp5HtSazaWY3TK1aBx8 7UzNKQv1Msqeujbqma4dNE59tQWvx6P18EYi1gcnFZ6scWuSjpj+lhYr23pBOopJpJyY 3PhnTWUT0mJtfPDkw+AzYVewXZTaZJ/Ytx9bJxqr0CK67XB62GHaXphjnnqxcb9wIOtS C8H2/6jbZTsdspfw4c0HGmHGR9F3iLzUQ+iCdJlnsCqeY6Vs/QUUBU5aawFnhEV5Kv4n Uqtx3o0zg3O3fEx18hhJfNE6sFq/tbg60ejSkOMCq2NAJTCbJ12CPjEA6NEUW6iQ4pGY tNkw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707913850; x=1708518650; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=p5mKfqmr3QrChRrscGODzWzPxAKG6aKZKMpL6RrB8mg=; b=RWqc9AvaMD5uKm+iAStl62B7laEtMOd6m2iYLSYf+ExWk8U8kfbPUBxHTV7FZSwuaH nKckurWVWCTipvirSGNws3t6hf1YGYTJMKVD/+/igwZWEVPvk/o9W0eWwyq24C/NtibQ q6UFPu1Jrv0mrue9BzK2vZVIyPRFSNp+7vCRorLx1I6x+cvz1rAh2X+E4Gc0XXRCjpet WTphfBnxpngj/xQDJ4oEDjtRorQ2cVEjqrrL8cpSQEEzjvvOoqiKv2VCQd18OOzxhJOg u7QNAtGQ6G9CHTmKkvTEdhasifT4jfs6v652glo2Z95BtUaH5+RXZhAxZBN9fOcf0X54 iLCw== X-Gm-Message-State: AOJu0YzfkNWMNi5LEmC1IfOC3Ka4uNyQrCq7JFXk00UdqbuIM283W+bO aBS3TADAIp1Cii++IUO+mAr3NMHWbWx9Tm3NxHzQPCtVHMJ6oDYvTFCwRm+6M5jA3ZHGv7Ghrpn 7Fsv1vbcbrOq6rs98pYiG/zPQmVIxjEfPKmEch+65CNTImuWIgKPfPHXwDkyFogbYDGBIvZJ3Fj q6568NNpVMWLInSsc6MQ4aJLobz9L6MpkL1fcPQpUN X-Google-Smtp-Source: AGHT+IFcjQ4dL9VIeLj1f8rTAHvc3loFfjyC4AY/q8Ej0nB8wWqydHQNxc7zTKRwStJ7JTyZU/rz69Mr X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a25:ae5a:0:b0:dcd:3172:7265 with SMTP id g26-20020a25ae5a000000b00dcd31727265mr500059ybe.8.1707913849752; Wed, 14 Feb 2024 04:30:49 -0800 (PST) Date: Wed, 14 Feb 2024 13:29:17 +0100 In-Reply-To: <20240214122845.2033971-45-ardb+git@google.com> Mime-Version: 1.0 References: <20240214122845.2033971-45-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=10782; i=ardb@kernel.org; h=from:subject; bh=D4WBd0jeyCVybmv3ifbkxZV1UclYSVZBWr8kVHQNCUo=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIfXMJvnuy+u2vmOzEuphlF6cdWxr6LtNmb1bO4QeFiZ37 13/LnZiRykLgxgHg6yYIovA7L/vdp6eKFXrPEsWZg4rE8gQBi5OAZjI+2SG/6Ur8r8zT9Tl7Zuy vHznBH6vHXOra5QnNk3wKRN5EGLAZsnIMFthr0Sgzd4UfaOYL3NeXHmpIflDRvkej/lKAeegOxs SmQE= X-Mailer: git-send-email 2.43.0.687.g38aa6559b0-goog Message-ID: <20240214122845.2033971-76-ardb+git@google.com> Subject: [PATCH v8 31/43] arm64: mm: Add definitions to support 5 levels of paging From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240214_123051_744273_E148A15C X-CRM114-Status: GOOD ( 23.78 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel Add the required types and descriptor accessors to support 5 levels of paging in the common code. This is one of the prerequisites for supporting 52-bit virtual addressing with 4k pages. Note that this does not cover the code that handles kernel mappings or the fixmap. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/pgalloc.h | 41 ++++++++++ arch/arm64/include/asm/pgtable-hwdef.h | 22 +++++- arch/arm64/include/asm/pgtable-types.h | 6 ++ arch/arm64/include/asm/pgtable.h | 82 +++++++++++++++++++- arch/arm64/mm/mmu.c | 31 +++++++- arch/arm64/mm/pgd.c | 15 +++- 6 files changed, 188 insertions(+), 9 deletions(-) diff --git a/arch/arm64/include/asm/pgalloc.h b/arch/arm64/include/asm/pgalloc.h index 237224484d0f..cae8c648f462 100644 --- a/arch/arm64/include/asm/pgalloc.h +++ b/arch/arm64/include/asm/pgalloc.h @@ -60,6 +60,47 @@ static inline void __p4d_populate(p4d_t *p4dp, phys_addr_t pudp, p4dval_t prot) } #endif /* CONFIG_PGTABLE_LEVELS > 3 */ +#if CONFIG_PGTABLE_LEVELS > 4 + +static inline void __pgd_populate(pgd_t *pgdp, phys_addr_t p4dp, pgdval_t prot) +{ + if (pgtable_l5_enabled()) + set_pgd(pgdp, __pgd(__phys_to_pgd_val(p4dp) | prot)); +} + +static inline void pgd_populate(struct mm_struct *mm, pgd_t *pgdp, p4d_t *p4dp) +{ + pgdval_t pgdval = PGD_TYPE_TABLE; + + pgdval |= (mm == &init_mm) ? PGD_TABLE_UXN : PGD_TABLE_PXN; + __pgd_populate(pgdp, __pa(p4dp), pgdval); +} + +static inline p4d_t *p4d_alloc_one(struct mm_struct *mm, unsigned long addr) +{ + gfp_t gfp = GFP_PGTABLE_USER; + + if (mm == &init_mm) + gfp = GFP_PGTABLE_KERNEL; + return (p4d_t *)get_zeroed_page(gfp); +} + +static inline void p4d_free(struct mm_struct *mm, p4d_t *p4d) +{ + if (!pgtable_l5_enabled()) + return; + BUG_ON((unsigned long)p4d & (PAGE_SIZE-1)); + free_page((unsigned long)p4d); +} + +#define __p4d_free_tlb(tlb, p4d, addr) p4d_free((tlb)->mm, p4d) +#else +static inline void __pgd_populate(pgd_t *pgdp, phys_addr_t p4dp, pgdval_t prot) +{ + BUILD_BUG(); +} +#endif /* CONFIG_PGTABLE_LEVELS > 4 */ + extern pgd_t *pgd_alloc(struct mm_struct *mm); extern void pgd_free(struct mm_struct *mm, pgd_t *pgdp); diff --git a/arch/arm64/include/asm/pgtable-hwdef.h b/arch/arm64/include/asm/pgtable-hwdef.h index 4426f48f2ae0..ef207a0d4f0d 100644 --- a/arch/arm64/include/asm/pgtable-hwdef.h +++ b/arch/arm64/include/asm/pgtable-hwdef.h @@ -26,10 +26,10 @@ #define ARM64_HW_PGTABLE_LEVELS(va_bits) (((va_bits) - 4) / (PAGE_SHIFT - 3)) /* - * Size mapped by an entry at level n ( 0 <= n <= 3) + * Size mapped by an entry at level n ( -1 <= n <= 3) * We map (PAGE_SHIFT - 3) at all translation levels and PAGE_SHIFT bits * in the final page. The maximum number of translation levels supported by - * the architecture is 4. Hence, starting at level n, we have further + * the architecture is 5. Hence, starting at level n, we have further * ((4 - n) - 1) levels of translation excluding the offset within the page. * So, the total number of bits mapped by an entry at level n is : * @@ -62,9 +62,16 @@ #define PTRS_PER_PUD (1 << (PAGE_SHIFT - 3)) #endif +#if CONFIG_PGTABLE_LEVELS > 4 +#define P4D_SHIFT ARM64_HW_PGTABLE_LEVEL_SHIFT(0) +#define P4D_SIZE (_AC(1, UL) << P4D_SHIFT) +#define P4D_MASK (~(P4D_SIZE-1)) +#define PTRS_PER_P4D (1 << (PAGE_SHIFT - 3)) +#endif + /* * PGDIR_SHIFT determines the size a top-level page table entry can map - * (depending on the configuration, this level can be 0, 1 or 2). + * (depending on the configuration, this level can be -1, 0, 1 or 2). */ #define PGDIR_SHIFT ARM64_HW_PGTABLE_LEVEL_SHIFT(4 - CONFIG_PGTABLE_LEVELS) #define PGDIR_SIZE (_AC(1, UL) << PGDIR_SHIFT) @@ -87,6 +94,15 @@ /* * Hardware page table definitions. * + * Level -1 descriptor (PGD). + */ +#define PGD_TYPE_TABLE (_AT(pgdval_t, 3) << 0) +#define PGD_TABLE_BIT (_AT(pgdval_t, 1) << 1) +#define PGD_TYPE_MASK (_AT(pgdval_t, 3) << 0) +#define PGD_TABLE_PXN (_AT(pgdval_t, 1) << 59) +#define PGD_TABLE_UXN (_AT(pgdval_t, 1) << 60) + +/* * Level 0 descriptor (P4D). */ #define P4D_TYPE_TABLE (_AT(p4dval_t, 3) << 0) diff --git a/arch/arm64/include/asm/pgtable-types.h b/arch/arm64/include/asm/pgtable-types.h index b8f158ae2527..6d6d4065b0cb 100644 --- a/arch/arm64/include/asm/pgtable-types.h +++ b/arch/arm64/include/asm/pgtable-types.h @@ -36,6 +36,12 @@ typedef struct { pudval_t pud; } pud_t; #define __pud(x) ((pud_t) { (x) } ) #endif +#if CONFIG_PGTABLE_LEVELS > 4 +typedef struct { p4dval_t p4d; } p4d_t; +#define p4d_val(x) ((x).p4d) +#define __p4d(x) ((p4d_t) { (x) } ) +#endif + typedef struct { pgdval_t pgd; } pgd_t; #define pgd_val(x) ((x).pgd) #define __pgd(x) ((pgd_t) { (x) } ) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 61de7b1516bc..7eb2b933ed3c 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -808,7 +808,6 @@ static inline pud_t *p4d_pgtable(p4d_t p4d) #else #define p4d_page_paddr(p4d) ({ BUILD_BUG(); 0;}) -#define pgd_page_paddr(pgd) ({ BUILD_BUG(); 0;}) /* Match pud_offset folding in */ #define pud_set_fixmap(addr) NULL @@ -819,6 +818,87 @@ static inline pud_t *p4d_pgtable(p4d_t p4d) #endif /* CONFIG_PGTABLE_LEVELS > 3 */ +#if CONFIG_PGTABLE_LEVELS > 4 + +static __always_inline bool pgtable_l5_enabled(void) +{ + if (!alternative_has_cap_likely(ARM64_ALWAYS_BOOT)) + return vabits_actual == VA_BITS; + return alternative_has_cap_unlikely(ARM64_HAS_VA52); +} + +static inline bool mm_p4d_folded(const struct mm_struct *mm) +{ + return !pgtable_l5_enabled(); +} +#define mm_p4d_folded mm_p4d_folded + +#define p4d_ERROR(e) \ + pr_err("%s:%d: bad p4d %016llx.\n", __FILE__, __LINE__, p4d_val(e)) + +#define pgd_none(pgd) (pgtable_l5_enabled() && !pgd_val(pgd)) +#define pgd_bad(pgd) (pgtable_l5_enabled() && !(pgd_val(pgd) & 2)) +#define pgd_present(pgd) (!pgd_none(pgd)) + +static inline void set_pgd(pgd_t *pgdp, pgd_t pgd) +{ + if (in_swapper_pgdir(pgdp)) { + set_swapper_pgd(pgdp, __pgd(pgd_val(pgd))); + return; + } + + WRITE_ONCE(*pgdp, pgd); + dsb(ishst); + isb(); +} + +static inline void pgd_clear(pgd_t *pgdp) +{ + if (pgtable_l5_enabled()) + set_pgd(pgdp, __pgd(0)); +} + +static inline phys_addr_t pgd_page_paddr(pgd_t pgd) +{ + return __pgd_to_phys(pgd); +} + +#define p4d_index(addr) (((addr) >> P4D_SHIFT) & (PTRS_PER_P4D - 1)) + +static inline p4d_t *pgd_to_folded_p4d(pgd_t *pgdp, unsigned long addr) +{ + return (p4d_t *)PTR_ALIGN_DOWN(pgdp, PAGE_SIZE) + p4d_index(addr); +} + +static inline phys_addr_t p4d_offset_phys(pgd_t *pgdp, unsigned long addr) +{ + BUG_ON(!pgtable_l5_enabled()); + + return pgd_page_paddr(READ_ONCE(*pgdp)) + p4d_index(addr) * sizeof(p4d_t); +} + +static inline +p4d_t *p4d_offset_lockless(pgd_t *pgdp, pgd_t pgd, unsigned long addr) +{ + if (!pgtable_l5_enabled()) + return pgd_to_folded_p4d(pgdp, addr); + return (p4d_t *)__va(pgd_page_paddr(pgd)) + p4d_index(addr); +} +#define p4d_offset_lockless p4d_offset_lockless + +static inline p4d_t *p4d_offset(pgd_t *pgdp, unsigned long addr) +{ + return p4d_offset_lockless(pgdp, READ_ONCE(*pgdp), addr); +} + +#define pgd_page(pgd) pfn_to_page(__phys_to_pfn(__pgd_to_phys(pgd))) + +#else + +static inline bool pgtable_l5_enabled(void) { return false; } + +#endif /* CONFIG_PGTABLE_LEVELS > 4 */ + #define pgd_ERROR(e) \ pr_err("%s:%d: bad pgd %016llx.\n", __FILE__, __LINE__, pgd_val(e)) diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index ba00d0205447..d2e9dec38a15 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -1025,7 +1025,7 @@ static void free_empty_pud_table(p4d_t *p4dp, unsigned long addr, if (CONFIG_PGTABLE_LEVELS <= 3) return; - if (!pgtable_range_aligned(start, end, floor, ceiling, PGDIR_MASK)) + if (!pgtable_range_aligned(start, end, floor, ceiling, P4D_MASK)) return; /* @@ -1048,8 +1048,8 @@ static void free_empty_p4d_table(pgd_t *pgdp, unsigned long addr, unsigned long end, unsigned long floor, unsigned long ceiling) { - unsigned long next; p4d_t *p4dp, p4d; + unsigned long i, next, start = addr; do { next = p4d_addr_end(addr, end); @@ -1061,6 +1061,27 @@ static void free_empty_p4d_table(pgd_t *pgdp, unsigned long addr, WARN_ON(!p4d_present(p4d)); free_empty_pud_table(p4dp, addr, next, floor, ceiling); } while (addr = next, addr < end); + + if (!pgtable_l5_enabled()) + return; + + if (!pgtable_range_aligned(start, end, floor, ceiling, PGDIR_MASK)) + return; + + /* + * Check whether we can free the p4d page if the rest of the + * entries are empty. Overlap with other regions have been + * handled by the floor/ceiling check. + */ + p4dp = p4d_offset(pgdp, 0UL); + for (i = 0; i < PTRS_PER_P4D; i++) { + if (!p4d_none(READ_ONCE(p4dp[i]))) + return; + } + + pgd_clear(pgdp); + __flush_tlb_kernel_pgtable(start); + free_hotplug_pgtable_page(virt_to_page(p4dp)); } static void free_empty_tables(unsigned long addr, unsigned long end, @@ -1145,6 +1166,12 @@ int pmd_set_huge(pmd_t *pmdp, phys_addr_t phys, pgprot_t prot) return 1; } +#ifndef __PAGETABLE_P4D_FOLDED +void p4d_clear_huge(p4d_t *p4dp) +{ +} +#endif + int pud_clear_huge(pud_t *pudp) { if (!pud_sect(READ_ONCE(*pudp))) diff --git a/arch/arm64/mm/pgd.c b/arch/arm64/mm/pgd.c index 4a64089e5771..3c4f8a279d2b 100644 --- a/arch/arm64/mm/pgd.c +++ b/arch/arm64/mm/pgd.c @@ -17,11 +17,20 @@ static struct kmem_cache *pgd_cache __ro_after_init; +static bool pgdir_is_page_size(void) +{ + if (PGD_SIZE == PAGE_SIZE) + return true; + if (CONFIG_PGTABLE_LEVELS == 5) + return !pgtable_l5_enabled(); + return false; +} + pgd_t *pgd_alloc(struct mm_struct *mm) { gfp_t gfp = GFP_PGTABLE_USER; - if (PGD_SIZE == PAGE_SIZE) + if (pgdir_is_page_size()) return (pgd_t *)__get_free_page(gfp); else return kmem_cache_alloc(pgd_cache, gfp); @@ -29,7 +38,7 @@ pgd_t *pgd_alloc(struct mm_struct *mm) void pgd_free(struct mm_struct *mm, pgd_t *pgd) { - if (PGD_SIZE == PAGE_SIZE) + if (pgdir_is_page_size()) free_page((unsigned long)pgd); else kmem_cache_free(pgd_cache, pgd); @@ -37,7 +46,7 @@ void pgd_free(struct mm_struct *mm, pgd_t *pgd) void __init pgtable_cache_init(void) { - if (PGD_SIZE == PAGE_SIZE) + if (pgdir_is_page_size()) return; #ifdef CONFIG_ARM64_PA_BITS_52 From patchwork Wed Feb 14 12:29:18 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13556547 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 2B0CBC48BC1 for ; Wed, 14 Feb 2024 13:46:01 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=7DKATZ8jRW44o5yMhr2CfRlxPvpkbAbBWOrYBEJ/2i0=; b=uz8WiCzVjpu19mhrKM/TfLkgvL aL9meHlJKeEW1q5jwixuyhPO3jUapDOy3t1RaOSXb94QEWSxnw+nsAv3ossKqY4E1gj3HcETigYCR hwWIfXMAjPCG2Obg+/SHl2v38xMzLu4a2TmKvua1Qcv/UapqKTonrbI326SykYELjZHWQ5lnfiUDk onEYRHATXj49a1E5BumgNWa0RHL1B0xJLzqR7loLF7t25z0HAfZrcn1bH56KgGL4mHDVwOYpXQbCO QJCO2SEfZX2WICXp/YPYy9vzP+RFQKfj/DhlQztXlDsNSq7IgNjfZfpuJPynZDc6+mY7rdLMdpK91 oOhC4BvA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1raFa7-0000000D3gL-2YdA; Wed, 14 Feb 2024 13:45:47 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1raEPj-0000000Cp0s-1nbH for linux-arm-kernel@bombadil.infradead.org; Wed, 14 Feb 2024 12:30:59 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=9okug4N0q8cmBvmDgQedNs6ipr2VdqxXnTFEpYyzSg4=; b=gdcxRF2+QEJwzPXd2Vkg9nsyeS hbU0uQAUzgHfculOfB64KVMfH+Ey5QMexJC2ujOwq9Y9mySqfk2qGtRoMQ67mW7xHrxoDqZnJ81KI a1oQThMLct7mbC7wm3NZzqEQD6TfN5B5i2QbTwZQLyG4lxsOShnrk5sprygAT6fMdbG5Qu9Jly4V8 mmRPRei8jknsUfcmQok8Ybv7m+i1oLF7Byp0Z5A6QWYzOKH7yqs69LdI11AcrSQrLmH6okSdNAWhL m9+JwGM+7lY7fWLYmQgnZRjUMTQ252nxB8sZoBIcj6JHqEOJVqacJ2pEVBfl+YkM6PiIh/JkF0h0e s43mNV+Q==; Received: from mail-yb1-xb49.google.com ([2607:f8b0:4864:20::b49]) by desiato.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1raEPf-0000000GEAI-0qo2 for linux-arm-kernel@lists.infradead.org; Wed, 14 Feb 2024 12:30:58 +0000 Received: by mail-yb1-xb49.google.com with SMTP id 3f1490d57ef6-dc64f63d768so10024783276.2 for ; Wed, 14 Feb 2024 04:30:53 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1707913852; x=1708518652; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=9okug4N0q8cmBvmDgQedNs6ipr2VdqxXnTFEpYyzSg4=; b=lsxhwZGK6t2hTHzCbZWyyGykNJaGwoDJKER2zEfItrjR6z1ciRvcSZ05qXDZQR3yXo PTpO5RwpAq7zhrKuh7zB/awfBLq/YbmU6THHvShTd+O1N3c4ojWFX47/JQrFWszqI/mX flo5Zw3ZR5vvD5Lai/Hs4GYrIlVfjde0QPH5m8et13zu/f/ZSjAZ1ilKPLS7UtxgmTpC AFkPklWAb8tWYjapBMugnvoDQ4vM6NcerpzU6yO8ZwOwxhsLep+C/qLVsLMjBxfqo5il Sxo4l3l3C+yY0SvrzsjbCTRsud/9hglQkERjNqEZic2TRf5KqDD8IhRgPYNaRqfSUFuN DE+g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707913852; x=1708518652; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=9okug4N0q8cmBvmDgQedNs6ipr2VdqxXnTFEpYyzSg4=; b=gDfa19JdhjjK9uLkpvIKDsHC0JkpPg0rSm0W8kACsIo0AnqbLwnJf2eEk65EygOOMu tVtI4KfiiyV4ILrM3/auNjoGGy7mBx0To1gk4DfmV4ZxKLPY16P7L1sCTkQo02fixJaK vUYGG55oMwsKSoc41dnmHlUP46xuM2FwlSYgZIln65GbwAxBvPdUM5y3H0KUEmioZ8r6 Ng3/pMggEJ5M6ozOLwBLfkEHMZj8qIUSZcU65shLNlW6DySGrYxwtiFbj0XPogN8ng+p IPWqPZ1c8VDEm6nDZzgegze27/Tb90PBP0wZKr7XKF0QCNg6gcYF5aHjKxnyezg6jRiD Qjow== X-Gm-Message-State: AOJu0Yzij4Twq+0JnvqjaHRbYQvj6S+ewmINbD3VOulRF5kgv7UQbK8X shrA29CrpgMs0B007njGeV6q2PyRIQCko+ReWXXYCpUn02j8m8ckZfzNnv0GijELrzyTadq3RVL ze/aasF85Q4KY63vZ3nYK5iKX86AZZFY0mBTTmkFie9btOuOCNHV84l0vWtAHLHXE1LZTlg5I6E YL1lBvpFvcJDF6UgAS+9bhjDPfQJdaRwlC0ENIeqp6 X-Google-Smtp-Source: AGHT+IH0pHk0qDJNqIAsD1Z9tfR+Tnf3soZlRZtdM6+9lyqtv8A9QQO6bqv2p5NegceReOFlqzR5+9z7 X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a25:ad0c:0:b0:dcd:875:4c40 with SMTP id y12-20020a25ad0c000000b00dcd08754c40mr490235ybi.10.1707913852140; Wed, 14 Feb 2024 04:30:52 -0800 (PST) Date: Wed, 14 Feb 2024 13:29:18 +0100 In-Reply-To: <20240214122845.2033971-45-ardb+git@google.com> Mime-Version: 1.0 References: <20240214122845.2033971-45-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=6529; i=ardb@kernel.org; h=from:subject; bh=7PfwX9kdlqeh9VOV1E3/xjyJ7s9l8UtTK1JPzHL11YQ=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIfXMJoWWbw+qfkZOqSpjX8UWIy4meDqVP+bJgWkHpr942 Dkhmy+7o5SFQYyDQVZMkUVg9t93O09PlKp1niULM4eVCWQIAxenAEzE7yvDf//FFmX8XYlT9X4f 3V8eUW9ms2DW9o6d3ge//+a8ySd1cBrDX6Fj/teTCmf26P5Tq5VU2LNWzeR4bGbD/Csx54vaMva rsgIA X-Mailer: git-send-email 2.43.0.687.g38aa6559b0-goog Message-ID: <20240214122845.2033971-77-ardb+git@google.com> Subject: [PATCH v8 32/43] arm64: mm: add LPA2 and 5 level paging support to G-to-nG conversion From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240214_123055_988584_99D62A83 X-CRM114-Status: GOOD ( 20.88 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel Add support for 5 level paging in the G-to-nG routine that creates its own temporary page tables to traverse the swapper page tables. Also add support for running the 5 level configuration with the top level folded at runtime, to support CPUs that do not implement the LPA2 extension. While at it, wire up the level skipping logic so it will also trigger on 4 level configurations with LPA2 enabled at build time but not active at runtime, as we'll fall back to 3 level paging in that case. Signed-off-by: Ard Biesheuvel --- arch/arm64/kernel/cpufeature.c | 9 ++- arch/arm64/mm/proc.S | 70 +++++++++++++++++--- 2 files changed, 66 insertions(+), 13 deletions(-) diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c index ed9670d8360c..bc5e4e569864 100644 --- a/arch/arm64/kernel/cpufeature.c +++ b/arch/arm64/kernel/cpufeature.c @@ -1765,6 +1765,9 @@ static int __init __kpti_install_ng_mappings(void *__unused) pgd_t *kpti_ng_temp_pgd; u64 alloc = 0; + if (levels == 5 && !pgtable_l5_enabled()) + levels = 4; + remap_fn = (void *)__pa_symbol(idmap_kpti_install_ng_mappings); if (!cpu) { @@ -1778,9 +1781,9 @@ static int __init __kpti_install_ng_mappings(void *__unused) // // The physical pages are laid out as follows: // - // +--------+-/-------+-/------ +-\\--------+ - // : PTE[] : | PMD[] : | PUD[] : || PGD[] : - // +--------+-\-------+-\------ +-//--------+ + // +--------+-/-------+-/------ +-/------ +-\\\--------+ + // : PTE[] : | PMD[] : | PUD[] : | P4D[] : ||| PGD[] : + // +--------+-\-------+-\------ +-\------ +-///--------+ // ^ // The first page is mapped into this hierarchy at a PMD_SHIFT // aligned virtual address, so that we can manipulate the PTE diff --git a/arch/arm64/mm/proc.S b/arch/arm64/mm/proc.S index d03434b7bca5..fa0d7c63f8d2 100644 --- a/arch/arm64/mm/proc.S +++ b/arch/arm64/mm/proc.S @@ -216,16 +216,15 @@ SYM_FUNC_ALIAS(__pi_idmap_cpu_replace_ttbr1, idmap_cpu_replace_ttbr1) .macro kpti_mk_tbl_ng, type, num_entries add end_\type\()p, cur_\type\()p, #\num_entries * 8 .Ldo_\type: - ldr \type, [cur_\type\()p] // Load the entry + ldr \type, [cur_\type\()p], #8 // Load the entry and advance tbz \type, #0, .Lnext_\type // Skip invalid and tbnz \type, #11, .Lnext_\type // non-global entries orr \type, \type, #PTE_NG // Same bit for blocks and pages - str \type, [cur_\type\()p] // Update the entry + str \type, [cur_\type\()p, #-8] // Update the entry .ifnc \type, pte tbnz \type, #1, .Lderef_\type .endif .Lnext_\type: - add cur_\type\()p, cur_\type\()p, #8 cmp cur_\type\()p, end_\type\()p b.ne .Ldo_\type .endm @@ -235,18 +234,18 @@ SYM_FUNC_ALIAS(__pi_idmap_cpu_replace_ttbr1, idmap_cpu_replace_ttbr1) * fixmap slot associated with the current level. */ .macro kpti_map_pgtbl, type, level - str xzr, [temp_pte, #8 * (\level + 1)] // break before make + str xzr, [temp_pte, #8 * (\level + 2)] // break before make dsb nshst - add pte, temp_pte, #PAGE_SIZE * (\level + 1) + add pte, temp_pte, #PAGE_SIZE * (\level + 2) lsr pte, pte, #12 tlbi vaae1, pte dsb nsh isb phys_to_pte pte, cur_\type\()p - add cur_\type\()p, temp_pte, #PAGE_SIZE * (\level + 1) + add cur_\type\()p, temp_pte, #PAGE_SIZE * (\level + 2) orr pte, pte, pte_flags - str pte, [temp_pte, #8 * (\level + 1)] + str pte, [temp_pte, #8 * (\level + 2)] dsb nshst .endm @@ -279,6 +278,8 @@ SYM_TYPED_FUNC_START(idmap_kpti_install_ng_mappings) end_ptep .req x15 pte .req x16 valid .req x17 + cur_p4dp .req x19 + end_p4dp .req x20 mov x5, x3 // preserve temp_pte arg mrs swapper_ttb, ttbr1_el1 @@ -286,6 +287,12 @@ SYM_TYPED_FUNC_START(idmap_kpti_install_ng_mappings) cbnz cpu, __idmap_kpti_secondary +#if CONFIG_PGTABLE_LEVELS > 4 + stp x29, x30, [sp, #-32]! + mov x29, sp + stp x19, x20, [sp, #16] +#endif + /* We're the boot CPU. Wait for the others to catch up */ sevl 1: wfe @@ -303,9 +310,32 @@ SYM_TYPED_FUNC_START(idmap_kpti_install_ng_mappings) mov_q pte_flags, KPTI_NG_PTE_FLAGS /* Everybody is enjoying the idmap, so we can rewrite swapper. */ + +#ifdef CONFIG_ARM64_LPA2 + /* + * If LPA2 support is configured, but 52-bit virtual addressing is not + * enabled at runtime, we will fall back to one level of paging less, + * and so we have to walk swapper_pg_dir as if we dereferenced its + * address from a PGD level entry, and terminate the PGD level loop + * right after. + */ + adrp pgd, swapper_pg_dir // walk &swapper_pg_dir at the next level + mov cur_pgdp, end_pgdp // must be equal to terminate the PGD loop +alternative_if_not ARM64_HAS_VA52 + b .Lderef_pgd // skip to the next level +alternative_else_nop_endif + /* + * LPA2 based 52-bit virtual addressing requires 52-bit physical + * addressing to be enabled as well. In this case, the shareability + * bits are repurposed as physical address bits, and should not be + * set in pte_flags. + */ + bic pte_flags, pte_flags, #PTE_SHARED +#endif + /* PGD */ adrp cur_pgdp, swapper_pg_dir - kpti_map_pgtbl pgd, 0 + kpti_map_pgtbl pgd, -1 kpti_mk_tbl_ng pgd, PTRS_PER_PGD /* Ensure all the updated entries are visible to secondary CPUs */ @@ -318,16 +348,33 @@ SYM_TYPED_FUNC_START(idmap_kpti_install_ng_mappings) /* Set the flag to zero to indicate that we're all done */ str wzr, [flag_ptr] +#if CONFIG_PGTABLE_LEVELS > 4 + ldp x19, x20, [sp, #16] + ldp x29, x30, [sp], #32 +#endif ret .Lderef_pgd: + /* P4D */ + .if CONFIG_PGTABLE_LEVELS > 4 + p4d .req x30 + pte_to_phys cur_p4dp, pgd + kpti_map_pgtbl p4d, 0 + kpti_mk_tbl_ng p4d, PTRS_PER_P4D + b .Lnext_pgd + .else /* CONFIG_PGTABLE_LEVELS <= 4 */ + p4d .req pgd + .set .Lnext_p4d, .Lnext_pgd + .endif + +.Lderef_p4d: /* PUD */ .if CONFIG_PGTABLE_LEVELS > 3 pud .req x10 - pte_to_phys cur_pudp, pgd + pte_to_phys cur_pudp, p4d kpti_map_pgtbl pud, 1 kpti_mk_tbl_ng pud, PTRS_PER_PUD - b .Lnext_pgd + b .Lnext_p4d .else /* CONFIG_PGTABLE_LEVELS <= 3 */ pud .req pgd .set .Lnext_pud, .Lnext_pgd @@ -371,6 +418,9 @@ SYM_TYPED_FUNC_START(idmap_kpti_install_ng_mappings) .unreq end_ptep .unreq pte .unreq valid + .unreq cur_p4dp + .unreq end_p4dp + .unreq p4d /* Secondary CPUs end up here */ __idmap_kpti_secondary: From patchwork Wed Feb 14 12:29:19 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13556548 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 177A5C48BC4 for ; Wed, 14 Feb 2024 13:46:03 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=jGnqTs9iGTqiQ1Zx331oj3xk7E77m61eUo48aBs0Zzs=; b=j52D805/auDsIjJSj0QrUN7gVA bPgl8953pUnZbg03k5alid75OsoA47PZ21+kohqQ8tHS6f1khe02ERBeKUpCf7s/5yRzpDG4oFjFc R4xWAGm/h7en9O5G0MXnNuQ/ENKyvgjGsJxkaviOly/d6UNgLGXTKN7BC8YMCBOTuHMMXRFKY3n+0 nQsZhQ72ekwnBtu6Lwj8dzFm2sUUw2z8+Qysx7eOKQ3TKeOTXXcb9sBf+58VygdcHToAk2ls+D8Xr ZwX3DZEFk1+5q/Eb6M1ydmbU/q55NgYOVzukji9tZv367T60bs5WHNh2Hr8pUpZvNqg0b+I/ke6P9 QQ4EGfag==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1raFa8-0000000D3gn-1ioI; Wed, 14 Feb 2024 13:45:48 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1raEPm-0000000Cp2x-1HnW for linux-arm-kernel@bombadil.infradead.org; Wed, 14 Feb 2024 12:31:02 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=jJ67b33XjsmYQcxPAqLALCq+Puu9BZcdfXARvUZl9Dg=; b=NYiEbW2hCXPnfVK9MSprAJ/mbI crDUpfwn5Ug0111sR03bI2cHMonoMSD+66JHwebrO+jtwj031z6PIgtVAULinPe5eIuU4iTiqohqR zQWmR6YA3syMZCBDlbDJ99+pj/eSYTzFnHRp9kgPIF1SuAbIXY2sd8YSXhByG1FD5g4/HbymdHaeM TpYoH8LV/CWaTHP9tfC1KXUZcsQgiVP5W2jY1ktZv85+0fMK+DUtE8K9vcmBIAD8ijbLFF4bwAnjk zwDV4WMScljbYScpr2VMqHWW+P+w5rtJpqE7AVWs3uoIvZhs38ZpmUYyx6cXzdcs/NOt6R+h6ZPAd AbrI6u5g==; Received: from mail-wr1-x449.google.com ([2a00:1450:4864:20::449]) by desiato.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1raEPg-0000000GEAa-3L8z for linux-arm-kernel@lists.infradead.org; Wed, 14 Feb 2024 12:31:01 +0000 Received: by mail-wr1-x449.google.com with SMTP id ffacd0b85a97d-33b8837355dso1127136f8f.0 for ; Wed, 14 Feb 2024 04:30:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1707913854; x=1708518654; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=jJ67b33XjsmYQcxPAqLALCq+Puu9BZcdfXARvUZl9Dg=; b=pfRhbWP4msrOrDSZetCP+V1Lzj48iZ9s/1QZBm6ZedvTOGGIPmRMDL35oqBBszqu3d +i7a6QszSa/xXnFxAH/ri6yBIsC3mAo3sYJm3ENOlTdzWik+/bJodGa2qqHt5znn6tje CeF3WPiGy46JrSBWKVfMRoTRQ4oiDQeP+kaVFP6F8auENJF3iwh3FJVDEL+gdQ8BnGgk +/Pxv062VlxHl3GtAJu93lSn/u1x0NAJJss98SB6bIO2TzNL2xUAdfnlUGdIae2cS+E+ eebpCpKoxrwNaaPiQ3T5k2XpbaIWT6fRjLUziSjeJIvNnOiKou/HbSJWeSip3dvnAPBP Pmww== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707913854; x=1708518654; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=jJ67b33XjsmYQcxPAqLALCq+Puu9BZcdfXARvUZl9Dg=; b=lJYCw9RCsZM/P4yXxHFzYHllRH5VnTTBYWm21Vfug33jvaES84oiFWg7RD+NRKnUZP LIarSjqg8bJaWBYbMEGTolKsaL5Lpv0eEqMUPkLvs2gKahSPucBH5aCmSnq84evDsLq6 2aZwO/OLyjVDMXrZSdseXxvOuwlOnWwPv4dnKyuPcuewnb6LTLyBUCB9N2CxnleFQ1Dt zaFM8qYkRRFJBM8sKdPFdYOvRfEr/b/LvLScIhwf58308zbV91pJ/c+IL3l3Q86HnWqG /pGDnPM28I784GRZuICkM57h52IBWHtIb49FBXBKcGCZQ70fNhGYqvX7CYOYog046JU+ tfsw== X-Gm-Message-State: AOJu0YwYyF7dsJgDuC7NTYXyKcnzgqIjHDIBKf0QhxFe248pDdM+qN89 7831DS69DIVG2HUUFjp1dHqkeAfd1rDjJl86vBtiNA0PavDWuIj3net4pG0s/JX8K3qC0G/SI3h 2ehTijw08gqhXBcY1Mf3QPIZf9x2eJ/+HggiwrQpev0q0OOZNYu8cOiq6T1I915oT2xWeSmTWDt +XSLjX8K0W6KN/f/lb6iW1UB0TLEXjVETiO5CDGAVg X-Google-Smtp-Source: AGHT+IGQpBOg+x7TYCxJO6MNDua3HZJmAFhEaq0WWy5AHhB9U8JMT1Yu1tNWWsaxuE711JflmCOW3BQc X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:6000:1c12:b0:33b:1b97:2b08 with SMTP id ba18-20020a0560001c1200b0033b1b972b08mr4845wrb.5.1707913854460; Wed, 14 Feb 2024 04:30:54 -0800 (PST) Date: Wed, 14 Feb 2024 13:29:19 +0100 In-Reply-To: <20240214122845.2033971-45-ardb+git@google.com> Mime-Version: 1.0 References: <20240214122845.2033971-45-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=13234; i=ardb@kernel.org; h=from:subject; bh=yGmRECzqwo0tjft4F8VJoox4CX7TJ8OuatCTpwo3uXc=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIfXMJqWuVa7hzDeWv9V0j0jZpvdhlZN+SWO6ZtMVk6Xd6 19c8TnZUcrCIMbBICumyCIw+++7nacnStU6z5KFmcPKBDKEgYtTACYi7crI0Dtzjc/iou9Lpyod 8m7JdBVo5IzdvDV7n83lNdEbYxhvb2D4Z153+esP4Y+6Jr84pNy43j0+WeGSyn1g2Zy82HYODqW 9vAA= X-Mailer: git-send-email 2.43.0.687.g38aa6559b0-goog Message-ID: <20240214122845.2033971-78-ardb+git@google.com> Subject: [PATCH v8 33/43] arm64: Enable LPA2 at boot if supported by the system From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240214_123057_416668_636C8842 X-CRM114-Status: GOOD ( 32.38 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel Update the early kernel mapping code to take 52-bit virtual addressing into account based on the LPA2 feature. This is a bit more involved than LVA (which is supported with 64k pages only), given that some page table descriptor bits change meaning in this case. To keep the handling in asm to a minimum, the initial ID map is still created with 48-bit virtual addressing, which implies that the kernel image must be loaded into 48-bit addressable physical memory. This is currently required by the boot protocol, even though we happen to support placement outside of that for LVA/64k based configurations. Enabling LPA2 involves more than setting TCR.T1SZ to a lower value, there is also a DS bit in TCR that needs to be set, and which changes the meaning of bits [9:8] in all page table descriptors. Since we cannot enable DS and every live page table descriptor at the same time, let's pivot through another temporary mapping. This avoids the need to reintroduce manipulations of the page tables with the MMU and caches disabled. To permit the LPA2 feature to be overridden on the kernel command line, which may be necessary to work around silicon errata, or to deal with mismatched features on heterogeneous SoC designs, test for CPU feature overrides first, and only then enable LPA2. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/assembler.h | 8 ++- arch/arm64/include/asm/cpufeature.h | 18 +++++ arch/arm64/include/asm/memory.h | 4 ++ arch/arm64/kernel/head.S | 8 +++ arch/arm64/kernel/image-vars.h | 1 + arch/arm64/kernel/pi/map_kernel.c | 70 +++++++++++++++++++- arch/arm64/kernel/pi/map_range.c | 11 ++- arch/arm64/kernel/pi/pi.h | 4 +- arch/arm64/mm/init.c | 2 +- arch/arm64/mm/mmu.c | 6 +- arch/arm64/mm/proc.S | 3 + 11 files changed, 124 insertions(+), 11 deletions(-) diff --git a/arch/arm64/include/asm/assembler.h b/arch/arm64/include/asm/assembler.h index 7eedcb36ebe0..ce7b95cd6e79 100644 --- a/arch/arm64/include/asm/assembler.h +++ b/arch/arm64/include/asm/assembler.h @@ -581,11 +581,17 @@ alternative_endif * but we have to add an offset so that the TTBR1 address corresponds with the * pgdir entry that covers the lowest 48-bit addressable VA. * + * Note that this trick is only used for LVA/64k pages - LPA2/4k pages uses an + * additional paging level, and on LPA2/16k pages, we would end up with a root + * level table with only 2 entries, which is suboptimal in terms of TLB + * utilization, so there we fall back to 47 bits of translation if LPA2 is not + * supported. + * * orr is used as it can cover the immediate value (and is idempotent). * ttbr: Value of ttbr to set, modified. */ .macro offset_ttbr1, ttbr, tmp -#ifdef CONFIG_ARM64_VA_BITS_52 +#if defined(CONFIG_ARM64_VA_BITS_52) && !defined(CONFIG_ARM64_LPA2) mrs \tmp, tcr_el1 and \tmp, \tmp, #TCR_T1SZ_MASK cmp \tmp, #TCR_T1SZ(VA_BITS_MIN) diff --git a/arch/arm64/include/asm/cpufeature.h b/arch/arm64/include/asm/cpufeature.h index a2ac31aecdd9..a8f97690ce1f 100644 --- a/arch/arm64/include/asm/cpufeature.h +++ b/arch/arm64/include/asm/cpufeature.h @@ -1008,6 +1008,24 @@ static inline bool cpu_has_lva(void) ID_AA64MMFR2_EL1_VARange_SHIFT); } +static inline bool cpu_has_lpa2(void) +{ +#ifdef CONFIG_ARM64_LPA2 + u64 mmfr0; + int feat; + + mmfr0 = read_sysreg(id_aa64mmfr0_el1); + mmfr0 &= ~id_aa64mmfr0_override.mask; + mmfr0 |= id_aa64mmfr0_override.val; + feat = cpuid_feature_extract_signed_field(mmfr0, + ID_AA64MMFR0_EL1_TGRAN_SHIFT); + + return feat >= ID_AA64MMFR0_EL1_TGRAN_LPA2; +#else + return false; +#endif +} + #endif /* __ASSEMBLY__ */ #endif diff --git a/arch/arm64/include/asm/memory.h b/arch/arm64/include/asm/memory.h index 9680d7444b3b..b850b1b91471 100644 --- a/arch/arm64/include/asm/memory.h +++ b/arch/arm64/include/asm/memory.h @@ -54,7 +54,11 @@ #define FIXADDR_TOP (-UL(SZ_8M)) #if VA_BITS > 48 +#ifdef CONFIG_ARM64_16K_PAGES +#define VA_BITS_MIN (47) +#else #define VA_BITS_MIN (48) +#endif #else #define VA_BITS_MIN (VA_BITS) #endif diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S index e25351addfd0..405e9bce8c73 100644 --- a/arch/arm64/kernel/head.S +++ b/arch/arm64/kernel/head.S @@ -89,6 +89,7 @@ SYM_CODE_START(primary_entry) mov sp, x1 mov x29, xzr adrp x0, init_idmap_pg_dir + mov x1, xzr bl __pi_create_init_idmap /* @@ -473,9 +474,16 @@ SYM_FUNC_END(__enable_mmu) #ifdef CONFIG_ARM64_VA_BITS_52 SYM_FUNC_START(__cpu_secondary_check52bitva) +#ifndef CONFIG_ARM64_LPA2 mrs_s x0, SYS_ID_AA64MMFR2_EL1 and x0, x0, ID_AA64MMFR2_EL1_VARange_MASK cbnz x0, 2f +#else + mrs x0, id_aa64mmfr0_el1 + sbfx x0, x0, #ID_AA64MMFR0_EL1_TGRAN_SHIFT, 4 + cmp x0, #ID_AA64MMFR0_EL1_TGRAN_LPA2 + b.ge 2f +#endif update_early_cpu_boot_status \ CPU_STUCK_IN_KERNEL | CPU_STUCK_REASON_52_BIT_VA, x0, x1 diff --git a/arch/arm64/kernel/image-vars.h b/arch/arm64/kernel/image-vars.h index ff81f809a240..ba4f8f7d6a91 100644 --- a/arch/arm64/kernel/image-vars.h +++ b/arch/arm64/kernel/image-vars.h @@ -54,6 +54,7 @@ PROVIDE(__pi__ctype = _ctype); PROVIDE(__pi_memstart_offset_seed = memstart_offset_seed); PROVIDE(__pi_init_idmap_pg_dir = init_idmap_pg_dir); +PROVIDE(__pi_init_idmap_pg_end = init_idmap_pg_end); PROVIDE(__pi_init_pg_dir = init_pg_dir); PROVIDE(__pi_init_pg_end = init_pg_end); PROVIDE(__pi_swapper_pg_dir = swapper_pg_dir); diff --git a/arch/arm64/kernel/pi/map_kernel.c b/arch/arm64/kernel/pi/map_kernel.c index 1853825aa29d..5fa08e13e17e 100644 --- a/arch/arm64/kernel/pi/map_kernel.c +++ b/arch/arm64/kernel/pi/map_kernel.c @@ -127,11 +127,64 @@ static void __init map_kernel(u64 kaslr_offset, u64 va_offset, int root_level) } /* Copy the root page table to its final location */ - memcpy((void *)swapper_pg_dir + va_offset, init_pg_dir, PGD_SIZE); + memcpy((void *)swapper_pg_dir + va_offset, init_pg_dir, PAGE_SIZE); dsb(ishst); idmap_cpu_replace_ttbr1(swapper_pg_dir); } +static void noinline __section(".idmap.text") set_ttbr0_for_lpa2(u64 ttbr) +{ + u64 sctlr = read_sysreg(sctlr_el1); + u64 tcr = read_sysreg(tcr_el1) | TCR_DS; + + asm(" msr sctlr_el1, %0 ;" + " isb ;" + " msr ttbr0_el1, %1 ;" + " msr tcr_el1, %2 ;" + " isb ;" + " tlbi vmalle1 ;" + " dsb nsh ;" + " isb ;" + " msr sctlr_el1, %3 ;" + " isb ;" + :: "r"(sctlr & ~SCTLR_ELx_M), "r"(ttbr), "r"(tcr), "r"(sctlr)); +} + +static void __init remap_idmap_for_lpa2(void) +{ + /* clear the bits that change meaning once LPA2 is turned on */ + pteval_t mask = PTE_SHARED; + + /* + * We have to clear bits [9:8] in all block or page descriptors in the + * initial ID map, as otherwise they will be (mis)interpreted as + * physical address bits once we flick the LPA2 switch (TCR.DS). Since + * we cannot manipulate live descriptors in that way without creating + * potential TLB conflicts, let's create another temporary ID map in a + * LPA2 compatible fashion, and update the initial ID map while running + * from that. + */ + create_init_idmap(init_pg_dir, mask); + dsb(ishst); + set_ttbr0_for_lpa2((u64)init_pg_dir); + + /* + * Recreate the initial ID map with the same granularity as before. + * Don't bother with the FDT, we no longer need it after this. + */ + memset(init_idmap_pg_dir, 0, + (u64)init_idmap_pg_dir - (u64)init_idmap_pg_end); + + create_init_idmap(init_idmap_pg_dir, mask); + dsb(ishst); + + /* switch back to the updated initial ID map */ + set_ttbr0_for_lpa2((u64)init_idmap_pg_dir); + + /* wipe the temporary ID map from memory */ + memset(init_pg_dir, 0, (u64)init_pg_end - (u64)init_pg_dir); +} + static void __init map_fdt(u64 fdt) { static u8 ptes[INIT_IDMAP_FDT_SIZE] __initdata __aligned(PAGE_SIZE); @@ -154,6 +207,7 @@ asmlinkage void __init early_map_kernel(u64 boot_status, void *fdt) u64 va_base, pa_base = (u64)&_text; u64 kaslr_offset = pa_base % MIN_KIMG_ALIGN; int root_level = 4 - CONFIG_PGTABLE_LEVELS; + int va_bits = VA_BITS; int chosen; map_fdt((u64)fdt); @@ -165,8 +219,15 @@ asmlinkage void __init early_map_kernel(u64 boot_status, void *fdt) chosen = fdt_path_offset(fdt, chosen_str); init_feature_override(boot_status, fdt, chosen); - if (VA_BITS > VA_BITS_MIN && cpu_has_lva()) - sysreg_clear_set(tcr_el1, TCR_T1SZ_MASK, TCR_T1SZ(VA_BITS)); + if (IS_ENABLED(CONFIG_ARM64_64K_PAGES) && !cpu_has_lva()) { + va_bits = VA_BITS_MIN; + } else if (IS_ENABLED(CONFIG_ARM64_LPA2) && !cpu_has_lpa2()) { + va_bits = VA_BITS_MIN; + root_level++; + } + + if (va_bits > VA_BITS_MIN) + sysreg_clear_set(tcr_el1, TCR_T1SZ_MASK, TCR_T1SZ(va_bits)); /* * The virtual KASLR displacement modulo 2MiB is decided by the @@ -184,6 +245,9 @@ asmlinkage void __init early_map_kernel(u64 boot_status, void *fdt) kaslr_offset |= kaslr_seed & ~(MIN_KIMG_ALIGN - 1); } + if (IS_ENABLED(CONFIG_ARM64_LPA2) && va_bits > VA_BITS_MIN) + remap_idmap_for_lpa2(); + va_base = KIMAGE_VADDR + kaslr_offset; map_kernel(kaslr_offset, va_base - pa_base, root_level); } diff --git a/arch/arm64/kernel/pi/map_range.c b/arch/arm64/kernel/pi/map_range.c index 79e4f6a2efe1..5410b2cac590 100644 --- a/arch/arm64/kernel/pi/map_range.c +++ b/arch/arm64/kernel/pi/map_range.c @@ -87,14 +87,19 @@ void __init map_range(u64 *pte, u64 start, u64 end, u64 pa, pgprot_t prot, } } -asmlinkage u64 __init create_init_idmap(pgd_t *pg_dir) +asmlinkage u64 __init create_init_idmap(pgd_t *pg_dir, pteval_t clrmask) { u64 ptep = (u64)pg_dir + PAGE_SIZE; + pgprot_t text_prot = PAGE_KERNEL_ROX; + pgprot_t data_prot = PAGE_KERNEL; + + pgprot_val(text_prot) &= ~clrmask; + pgprot_val(data_prot) &= ~clrmask; map_range(&ptep, (u64)_stext, (u64)__initdata_begin, (u64)_stext, - PAGE_KERNEL_ROX, IDMAP_ROOT_LEVEL, (pte_t *)pg_dir, false, 0); + text_prot, IDMAP_ROOT_LEVEL, (pte_t *)pg_dir, false, 0); map_range(&ptep, (u64)__initdata_begin, (u64)_end, (u64)__initdata_begin, - PAGE_KERNEL, IDMAP_ROOT_LEVEL, (pte_t *)pg_dir, false, 0); + data_prot, IDMAP_ROOT_LEVEL, (pte_t *)pg_dir, false, 0); return ptep; } diff --git a/arch/arm64/kernel/pi/pi.h b/arch/arm64/kernel/pi/pi.h index 1ea282a5f96a..c91e5e965cd3 100644 --- a/arch/arm64/kernel/pi/pi.h +++ b/arch/arm64/kernel/pi/pi.h @@ -21,7 +21,7 @@ static inline void *prel64_to_pointer(const prel64_t *offset) extern bool dynamic_scs_is_enabled; -extern pgd_t init_idmap_pg_dir[]; +extern pgd_t init_idmap_pg_dir[], init_idmap_pg_end[]; void init_feature_override(u64 boot_status, const void *fdt, int chosen); u64 kaslr_early_init(void *fdt, int chosen); @@ -33,4 +33,4 @@ void map_range(u64 *pgd, u64 start, u64 end, u64 pa, pgprot_t prot, asmlinkage void early_map_kernel(u64 boot_status, void *fdt); -asmlinkage u64 create_init_idmap(pgd_t *pgd); +asmlinkage u64 create_init_idmap(pgd_t *pgd, pteval_t clrmask); diff --git a/arch/arm64/mm/init.c b/arch/arm64/mm/init.c index 74c1db8ce271..0f427b50fdc3 100644 --- a/arch/arm64/mm/init.c +++ b/arch/arm64/mm/init.c @@ -238,7 +238,7 @@ void __init arm64_memblock_init(void) * physical address of PAGE_OFFSET, we have to *subtract* from it. */ if (IS_ENABLED(CONFIG_ARM64_VA_BITS_52) && (vabits_actual != 52)) - memstart_addr -= _PAGE_OFFSET(48) - _PAGE_OFFSET(52); + memstart_addr -= _PAGE_OFFSET(vabits_actual) - _PAGE_OFFSET(52); /* * Apply the memory limit if it was set. Since the kernel may be loaded diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index d2e9dec38a15..d30ae4d3fdd9 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -582,8 +582,12 @@ static void __init map_mem(pgd_t *pgdp) * entries at any level are being shared between the linear region and * the vmalloc region. Check whether this is true for the PGD level, in * which case it is guaranteed to be true for all other levels as well. + * (Unless we are running with support for LPA2, in which case the + * entire reduced VA space is covered by a single pgd_t which will have + * been populated without the PXNTable attribute by the time we get here.) */ - BUILD_BUG_ON(pgd_index(direct_map_end - 1) == pgd_index(direct_map_end)); + BUILD_BUG_ON(pgd_index(direct_map_end - 1) == pgd_index(direct_map_end) && + pgd_index(_PAGE_OFFSET(VA_BITS_MIN)) != PTRS_PER_PGD - 1); early_kfence_pool = arm64_kfence_alloc_pool(); diff --git a/arch/arm64/mm/proc.S b/arch/arm64/mm/proc.S index fa0d7c63f8d2..9d40f3ffd8d2 100644 --- a/arch/arm64/mm/proc.S +++ b/arch/arm64/mm/proc.S @@ -488,6 +488,9 @@ SYM_FUNC_START(__cpu_setup) mov x9, #64 - VA_BITS alternative_if ARM64_HAS_VA52 tcr_set_t1sz tcr, x9 +#ifdef CONFIG_ARM64_LPA2 + orr tcr, tcr, #TCR_DS +#endif alternative_else_nop_endif #endif From patchwork Wed Feb 14 12:29:20 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13556552 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 8994CC48BC3 for ; Wed, 14 Feb 2024 13:46:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=M4R/Svlk5P7z/bNEavLS7JtNoJc1MIgBqO46LyLqCK0=; b=vRwfoJTjNvZQT6CpPi+qLDn+oW fiGX3ClXiAD0C+cZheBpjUsxiRONs/NI08rUR/TDKNxVGNUsNhZCYJU3v8XDXUu9XVfiTqXz3Ba5w UeZ/5WYSAhKC42zxIKqgcBdMjJC29BbVh5+PkyxFuFsXfkcVn/ewZfwOtbj5PNxK1aK9dE0VC/bsz BOLvghxrH9DPzhlq/Z6kuv6EsQ6mPURXcRQS2wxEFFle3UxdSiZUKjzLuaUFiwqzfXmEVAKrIzNPJ 9YIJG98tZP2o0t21LHlD55lA9P+nUA1aDjZBFTNgv/SCIwGgjRHhokbE/Sfb7jd5s7I7PHBHEtzh7 tZHfOkSA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1raFaC-0000000D3jy-0nuO; Wed, 14 Feb 2024 13:45:52 +0000 Received: from mail-wm1-f74.google.com ([209.85.128.74]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1raEPk-0000000Cp0M-0oet for linux-arm-kernel@lists.infradead.org; Wed, 14 Feb 2024 12:31:16 +0000 Received: by mail-wm1-f74.google.com with SMTP id 5b1f17b1804b1-41082621642so25907985e9.1 for ; Wed, 14 Feb 2024 04:30:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1707913857; x=1708518657; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=dHzoJTSpME10fT7efs3c2Z+eBzrf+W2skzONvU/iPLI=; b=JkgR+yddIlxKVfilhecBpMKlcITG+SPqRze0QA39UH6sC23Tje+C4TPpOriEDo8Wm9 FmSV5hyLXSSG29bwVKeAaClEYXAbmsOb49pUzK5lHcYAOgYkMURRacoN8TD4u9L5T9DR RH54WiLfQe4+0P+RDjnm6Sk2GL+8Ab8uIJS3LUV4Lk01t+oiq2lCJ+tLmNs+KQHZDQIE tbqeexDr4LMN84LnQLuoc2T+UAIt91wrX7WxIJgug9WX3vFWSkhfBo35hPYb96LM8o4j fERZ//Z3twBa3iESwFhcNuFU5k9dnGCi6cpXylWJp5YF04rhbSWYwcwUPGhdn1ipGYb9 JNtA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707913857; x=1708518657; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=dHzoJTSpME10fT7efs3c2Z+eBzrf+W2skzONvU/iPLI=; b=qgPnoSVSCUdw/dGp5WnFKqOIKirLkZthAP/ls4SxNppHbH31z43+pdIW9+bAIC2+Qs rcVbnXgwbqNOgq4y+B7hk8upz5Y9/0Ak/3QCIkFc4+sI4czuxonAJM65O9LqYC8i8egP gWYSgFMEimV7a7t0yvKnn5ANcskR4J6wJChtz1CEcE/i/PTkSYI2cu9gIZGKowOx3Hsd tR5L31Zj+hIaR+zASsH7MIL1BXVQ/HRYwRNIAljs004f4C40cs8DvCicx7+i+/PuszjS bj7VJtO0BGLLS53ZRCEEDbB+NQi8VzgpljB2TAYhF3YLuykuJzjzBrb/CRdWvsJ2m+O3 dOsg== X-Gm-Message-State: AOJu0YwyY0GloUTp78UmKkc8Z/rhy4yD9xxnsH3kCBiF0w+ir/eftUxC l7ujC7g9VeyY/+mhwvzbdbvQ2zyPcMsSmzLIWCaO0j1QGBCgQ4nGCn4xtz8H7B9PuZJG8p1m4/K x2/EtRj6eX/Th5cy00WmhDm5W/w4cXrw2JICLs23oIxjo7DVvz7puBlAt2uEyZXjWql7DLGUJco F7nA2C8TRte3qQyEMh7Me9FDNfbNXFXX5J3ZsnAdPv X-Google-Smtp-Source: AGHT+IG7V/6yak0oYBtkkSR8n6QBnoQqvh4yuYpFiRdf0f5mVGuF7g+sHJBBArb+cfltYOGiOjxyWIpu X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:600c:54e3:b0:411:eed4:a396 with SMTP id jb3-20020a05600c54e300b00411eed4a396mr5090wmb.2.1707913856788; Wed, 14 Feb 2024 04:30:56 -0800 (PST) Date: Wed, 14 Feb 2024 13:29:20 +0100 In-Reply-To: <20240214122845.2033971-45-ardb+git@google.com> Mime-Version: 1.0 References: <20240214122845.2033971-45-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=5970; i=ardb@kernel.org; h=from:subject; bh=bM2TkRl5rFMQ7ywOQnXc9Yw6neLXbIQxwtUHp5mgOM0=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIfXMJuV5Ae+m3+2SZemOrj39p2QK54pPIYaJ2yZ//Z+jw 5L7ReZPRykLgxgHg6yYIovA7L/vdp6eKFXrPEsWZg4rE8gQBi5OAZhIbwUjw7muWRaFbMXMfkm6 crULn07RtmYv38y9fvrjdY+E+02v5TD801mTOsE958opNc1N2/3Vfr+Ne+H27WnKZ8EJkvPOuE9 9yAAA X-Mailer: git-send-email 2.43.0.687.g38aa6559b0-goog Message-ID: <20240214122845.2033971-79-ardb+git@google.com> Subject: [PATCH v8 34/43] arm64: mm: Add 5 level paging support to fixmap and swapper handling From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240214_043100_536359_5AB50903 X-CRM114-Status: GOOD ( 15.03 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel Add support for using 5 levels of paging in the fixmap, as well as in the kernel page table handling code which uses fixmaps internally. This also handles the case where a 5 level build runs on hardware that only supports 4 levels of paging. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/fixmap.h | 1 + arch/arm64/include/asm/pgtable.h | 45 ++++++++++++++++--- arch/arm64/mm/fixmap.c | 2 +- arch/arm64/mm/mmu.c | 47 ++++++++++++++++++-- 4 files changed, 85 insertions(+), 10 deletions(-) diff --git a/arch/arm64/include/asm/fixmap.h b/arch/arm64/include/asm/fixmap.h index 8aabd45e9a13..87e307804b99 100644 --- a/arch/arm64/include/asm/fixmap.h +++ b/arch/arm64/include/asm/fixmap.h @@ -87,6 +87,7 @@ enum fixed_addresses { FIX_PTE, FIX_PMD, FIX_PUD, + FIX_P4D, FIX_PGD, __end_of_fixed_addresses diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 7eb2b933ed3c..3d7fb3cde83d 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -621,12 +621,12 @@ static inline bool pud_table(pud_t pud) { return true; } PUD_TYPE_TABLE) #endif -extern pgd_t init_pg_dir[PTRS_PER_PGD]; +extern pgd_t init_pg_dir[]; extern pgd_t init_pg_end[]; -extern pgd_t swapper_pg_dir[PTRS_PER_PGD]; -extern pgd_t idmap_pg_dir[PTRS_PER_PGD]; -extern pgd_t tramp_pg_dir[PTRS_PER_PGD]; -extern pgd_t reserved_pg_dir[PTRS_PER_PGD]; +extern pgd_t swapper_pg_dir[]; +extern pgd_t idmap_pg_dir[]; +extern pgd_t tramp_pg_dir[]; +extern pgd_t reserved_pg_dir[]; extern void set_swapper_pgd(pgd_t *pgdp, pgd_t pgd); @@ -891,12 +891,47 @@ static inline p4d_t *p4d_offset(pgd_t *pgdp, unsigned long addr) return p4d_offset_lockless(pgdp, READ_ONCE(*pgdp), addr); } +static inline p4d_t *p4d_set_fixmap(unsigned long addr) +{ + if (!pgtable_l5_enabled()) + return NULL; + return (p4d_t *)set_fixmap_offset(FIX_P4D, addr); +} + +static inline p4d_t *p4d_set_fixmap_offset(pgd_t *pgdp, unsigned long addr) +{ + if (!pgtable_l5_enabled()) + return pgd_to_folded_p4d(pgdp, addr); + return p4d_set_fixmap(p4d_offset_phys(pgdp, addr)); +} + +static inline void p4d_clear_fixmap(void) +{ + if (pgtable_l5_enabled()) + clear_fixmap(FIX_P4D); +} + +/* use ONLY for statically allocated translation tables */ +static inline p4d_t *p4d_offset_kimg(pgd_t *pgdp, u64 addr) +{ + if (!pgtable_l5_enabled()) + return pgd_to_folded_p4d(pgdp, addr); + return (p4d_t *)__phys_to_kimg(p4d_offset_phys(pgdp, addr)); +} + #define pgd_page(pgd) pfn_to_page(__phys_to_pfn(__pgd_to_phys(pgd))) #else static inline bool pgtable_l5_enabled(void) { return false; } +/* Match p4d_offset folding in */ +#define p4d_set_fixmap(addr) NULL +#define p4d_set_fixmap_offset(p4dp, addr) ((p4d_t *)p4dp) +#define p4d_clear_fixmap() + +#define p4d_offset_kimg(dir,addr) ((p4d_t *)dir) + #endif /* CONFIG_PGTABLE_LEVELS > 4 */ #define pgd_ERROR(e) \ diff --git a/arch/arm64/mm/fixmap.c b/arch/arm64/mm/fixmap.c index 9404f282f829..d22506e9c7fd 100644 --- a/arch/arm64/mm/fixmap.c +++ b/arch/arm64/mm/fixmap.c @@ -104,7 +104,7 @@ void __init early_fixmap_init(void) unsigned long end = FIXADDR_TOP; pgd_t *pgdp = pgd_offset_k(addr); - p4d_t *p4dp = p4d_offset(pgdp, addr); + p4d_t *p4dp = p4d_offset_kimg(pgdp, addr); early_fixmap_init_pud(p4dp, addr, end); } diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index d30ae4d3fdd9..8e5b3a7c5afd 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -313,15 +313,14 @@ static void alloc_init_cont_pmd(pud_t *pudp, unsigned long addr, } while (addr = next, addr != end); } -static void alloc_init_pud(pgd_t *pgdp, unsigned long addr, unsigned long end, +static void alloc_init_pud(p4d_t *p4dp, unsigned long addr, unsigned long end, phys_addr_t phys, pgprot_t prot, phys_addr_t (*pgtable_alloc)(int), int flags) { unsigned long next; - pud_t *pudp; - p4d_t *p4dp = p4d_offset(pgdp, addr); p4d_t p4d = READ_ONCE(*p4dp); + pud_t *pudp; if (p4d_none(p4d)) { p4dval_t p4dval = P4D_TYPE_TABLE | P4D_TABLE_UXN; @@ -369,6 +368,46 @@ static void alloc_init_pud(pgd_t *pgdp, unsigned long addr, unsigned long end, pud_clear_fixmap(); } +static void alloc_init_p4d(pgd_t *pgdp, unsigned long addr, unsigned long end, + phys_addr_t phys, pgprot_t prot, + phys_addr_t (*pgtable_alloc)(int), + int flags) +{ + unsigned long next; + pgd_t pgd = READ_ONCE(*pgdp); + p4d_t *p4dp; + + if (pgd_none(pgd)) { + pgdval_t pgdval = PGD_TYPE_TABLE | PGD_TABLE_UXN; + phys_addr_t p4d_phys; + + if (flags & NO_EXEC_MAPPINGS) + pgdval |= PGD_TABLE_PXN; + BUG_ON(!pgtable_alloc); + p4d_phys = pgtable_alloc(P4D_SHIFT); + __pgd_populate(pgdp, p4d_phys, pgdval); + pgd = READ_ONCE(*pgdp); + } + BUG_ON(pgd_bad(pgd)); + + p4dp = p4d_set_fixmap_offset(pgdp, addr); + do { + p4d_t old_p4d = READ_ONCE(*p4dp); + + next = p4d_addr_end(addr, end); + + alloc_init_pud(p4dp, addr, next, phys, prot, + pgtable_alloc, flags); + + BUG_ON(p4d_val(old_p4d) != 0 && + p4d_val(old_p4d) != READ_ONCE(p4d_val(*p4dp))); + + phys += next - addr; + } while (p4dp++, addr = next, addr != end); + + p4d_clear_fixmap(); +} + static void __create_pgd_mapping_locked(pgd_t *pgdir, phys_addr_t phys, unsigned long virt, phys_addr_t size, pgprot_t prot, @@ -391,7 +430,7 @@ static void __create_pgd_mapping_locked(pgd_t *pgdir, phys_addr_t phys, do { next = pgd_addr_end(addr, end); - alloc_init_pud(pgdp, addr, next, phys, prot, pgtable_alloc, + alloc_init_p4d(pgdp, addr, next, phys, prot, pgtable_alloc, flags); phys += next - addr; } while (pgdp++, addr = next, addr != end); From patchwork Wed Feb 14 12:29:21 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13556553 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 19BAFC48BC4 for ; Wed, 14 Feb 2024 13:46:09 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=8H0N2DdmjwgHC/ePBRXNYd+FjLV5AfF4etxQbiHsYF8=; b=lzLFAHL33O9ks0Cc4Gbz9HoAb9 N8ItL9nwn8RgkQ/zby19cbxrN8sYW9f2dOQCPuz3J+dORGGuI3I/bqnpI9rMl3mElmHg+R6IlXkop TDMMaZe61sp6P4X00tvbV8N0p1cIxpsGkAz6V42GPK4zRiMQQz5jT4eU4liu6dVvOyB1DumuuaCU/ InUutBbeTDnmkO8CdVSySMXtSXtJzYoXna0l82Q5rbxGZ4tkIapVAAecDuJcZ6Ds1KiaqmX9MLGuK kiX1M/RaZjNPWwvZnqGnOSoK4vvh+m+eLqAjtC6G51TA1TFHpL7mNNz38mVQKwqEkWu23U+bX5MAd ETPdZLnA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1raFaD-0000000D3l5-1E9N; Wed, 14 Feb 2024 13:45:53 +0000 Received: from mail-yw1-x1149.google.com ([2607:f8b0:4864:20::1149]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1raEPk-0000000Cp1W-31Kb for linux-arm-kernel@lists.infradead.org; Wed, 14 Feb 2024 12:31:17 +0000 Received: by mail-yw1-x1149.google.com with SMTP id 00721157ae682-6077fa2fa9bso35544827b3.2 for ; Wed, 14 Feb 2024 04:30:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1707913859; x=1708518659; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=SHkiHdKuoLdTUweGoarzhiDrHgHxRwc0G66o05u5kEs=; b=2QuOeSPWdwPHJPQwxWrDLduzXU7Ctr40Tyd8OTQ7ty5v8I3mgVq6G5W+nGaXZs0TSh LgbKZouSfoQJgwuS5eqjj6FS3ujugpQ0MTaBrkgVTqLAhfft1QkQNGVhxE3ivR1SpUzP mT21QiDRkCjnnlJHdBLb6RqxDil09zzOLt1/SaqqHe3Pm7Z5pwzAGA6EkWZGWzKeeoFY DmpS9G/jP3UL9fjG2Kaof28Yggwj3Bdscw5mxdQiVgH5wu1b+xsH0AJmT0PH0HaQ71bl L8/XrG5371vpvkRs8GsQbcCIJFSHeRItbpbIxQPhMVFS9E9uYVbRakJov22cjAruk08o tLlg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707913859; x=1708518659; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=SHkiHdKuoLdTUweGoarzhiDrHgHxRwc0G66o05u5kEs=; b=JF/foNrV2QZnGcY7F6wfg1o0q5EKA7S/fw995Wv2e8EVQCGtds+pm1QBu7IV4E4pNs ILWL2um4zeROw39b6QIX7UHa4q9v4X4dc28CrVtHZSGc/zz31iHa5c6fPG9j2AHFu75f jJStcNCXP0eVscBYruehokc1Ir+vk2ACtlq53QQQshiG7NLvqrKa5DinEXfD/SFtZIVi zFliaSEVbNdkE09OGg9e+YIrexy6CZymm0lLPh+vrfMxf4FdjGbphh0Aw0lo3rzNTv/v NabhrFXSg5K6pgint1sfF9AQbj9zofMuC2Zhd16dbbNvH7lTox2spll78Z5tWkTYRb/s IYJQ== X-Gm-Message-State: AOJu0YyGuz/rIpNN2uQZSdfbxQl6P1+fC9vJd8To3Aj69bZkqcgwrlqj xblNVVqWpeA1mfH2Kt7z6smcam+OpqNWRWrte3LZ4A7ZlglYAJMg5s34vhk2zQHxJ24zOQIOEbH Nh7FqvJKpsJiMQ4N81Oka5R32R9tFrXDKMath5kdmNZZoTSLlbdyL62aux6QFJXdNgXmLdMACud JUfjJK5uVdZPM86hHcTnlWARkRa3CebVIr23Z9Zlww X-Google-Smtp-Source: AGHT+IFFj6iZA10vgKzN1U6tTeg+LOqh6TOwfV+Zr0P14KRg/syIkOrPaPVmvT4LKtwC4FluCwStwZtx X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a81:834c:0:b0:604:8475:313d with SMTP id t73-20020a81834c000000b006048475313dmr383618ywf.4.1707913859253; Wed, 14 Feb 2024 04:30:59 -0800 (PST) Date: Wed, 14 Feb 2024 13:29:21 +0100 In-Reply-To: <20240214122845.2033971-45-ardb+git@google.com> Mime-Version: 1.0 References: <20240214122845.2033971-45-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=9481; i=ardb@kernel.org; h=from:subject; bh=c6pfjN4e8JWsLL8TeQXywLeyfxAtOT/xSfs0zYYoVZk=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIfXMJpWbKRGL98yTbeDSXbWGyYLHTGNN7r7HM8/1bbjh8 zzTZq1LRykLgxgHg6yYIovA7L/vdp6eKFXrPEsWZg4rE8gQBi5OAZjI9ByGv7KS7gE2R2fmmxqf qPe/pfg1W+hYXsjROOGMdau+6FnYPWD4p9/4sbw+bv+q7ABj5eOXX/az/JVSLT/L4hbm3WMZwnu MBwA= X-Mailer: git-send-email 2.43.0.687.g38aa6559b0-goog Message-ID: <20240214122845.2033971-80-ardb+git@google.com> Subject: [PATCH v8 35/43] arm64: kasan: Reduce minimum shadow alignment and enable 5 level paging From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240214_043101_058180_C4B9D4D3 X-CRM114-Status: GOOD ( 26.12 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel Allow the KASAN init code to deal with 5 levels of paging, and relax the requirement that the shadow region is aligned to the top level pgd_t size. This is necessary for LPA2 based 52-bit virtual addressing, where the KASAN shadow will never be aligned to the pgd_t size. Allowing this also enables the 16k/48-bit case for KASAN, which is a nice bonus. This involves some hackery to manipulate the root and next level page tables without having to distinguish all the various configurations, including 16k/48-bits (which has a two entry pgd_t level), and LPA2 configurations running with one translation level less on non-LPA2 hardware. Signed-off-by: Ard Biesheuvel --- arch/arm64/Kconfig | 2 +- arch/arm64/mm/kasan_init.c | 148 +++++++++++++++++--- 2 files changed, 130 insertions(+), 20 deletions(-) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 8c2c36fffcf5..9ca3316d6379 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -164,7 +164,7 @@ config ARM64 select HAVE_ARCH_HUGE_VMAP select HAVE_ARCH_JUMP_LABEL select HAVE_ARCH_JUMP_LABEL_RELATIVE - select HAVE_ARCH_KASAN if !(ARM64_16K_PAGES && ARM64_VA_BITS_48) + select HAVE_ARCH_KASAN select HAVE_ARCH_KASAN_VMALLOC if HAVE_ARCH_KASAN select HAVE_ARCH_KASAN_SW_TAGS if HAVE_ARCH_KASAN select HAVE_ARCH_KASAN_HW_TAGS if (HAVE_ARCH_KASAN && ARM64_MTE) diff --git a/arch/arm64/mm/kasan_init.c b/arch/arm64/mm/kasan_init.c index a86ab99587c9..fbddbf9faf19 100644 --- a/arch/arm64/mm/kasan_init.c +++ b/arch/arm64/mm/kasan_init.c @@ -23,7 +23,7 @@ #if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS) -static pgd_t tmp_pg_dir[PTRS_PER_PGD] __initdata __aligned(PGD_SIZE); +static pgd_t tmp_pg_dir[PTRS_PER_PTE] __initdata __aligned(PAGE_SIZE); /* * The p*d_populate functions call virt_to_phys implicitly so they can't be used @@ -99,6 +99,19 @@ static pud_t *__init kasan_pud_offset(p4d_t *p4dp, unsigned long addr, int node, return early ? pud_offset_kimg(p4dp, addr) : pud_offset(p4dp, addr); } +static p4d_t *__init kasan_p4d_offset(pgd_t *pgdp, unsigned long addr, int node, + bool early) +{ + if (pgd_none(READ_ONCE(*pgdp))) { + phys_addr_t p4d_phys = early ? + __pa_symbol(kasan_early_shadow_p4d) + : kasan_alloc_zeroed_page(node); + __pgd_populate(pgdp, p4d_phys, PGD_TYPE_TABLE); + } + + return early ? p4d_offset_kimg(pgdp, addr) : p4d_offset(pgdp, addr); +} + static void __init kasan_pte_populate(pmd_t *pmdp, unsigned long addr, unsigned long end, int node, bool early) { @@ -144,12 +157,12 @@ static void __init kasan_p4d_populate(pgd_t *pgdp, unsigned long addr, unsigned long end, int node, bool early) { unsigned long next; - p4d_t *p4dp = p4d_offset(pgdp, addr); + p4d_t *p4dp = kasan_p4d_offset(pgdp, addr, node, early); do { next = p4d_addr_end(addr, end); kasan_pud_populate(p4dp, addr, next, node, early); - } while (p4dp++, addr = next, addr != end); + } while (p4dp++, addr = next, addr != end && p4d_none(READ_ONCE(*p4dp))); } static void __init kasan_pgd_populate(unsigned long addr, unsigned long end, @@ -165,19 +178,48 @@ static void __init kasan_pgd_populate(unsigned long addr, unsigned long end, } while (pgdp++, addr = next, addr != end); } +#if defined(CONFIG_ARM64_64K_PAGES) || CONFIG_PGTABLE_LEVELS > 4 +#define SHADOW_ALIGN P4D_SIZE +#else +#define SHADOW_ALIGN PUD_SIZE +#endif + +/* + * Return whether 'addr' is aligned to the size covered by a root level + * descriptor. + */ +static bool __init root_level_aligned(u64 addr) +{ + int shift = (ARM64_HW_PGTABLE_LEVELS(vabits_actual) - 1) * (PAGE_SHIFT - 3); + + return (addr % (PAGE_SIZE << shift)) == 0; +} + /* The early shadow maps everything to a single page of zeroes */ asmlinkage void __init kasan_early_init(void) { BUILD_BUG_ON(KASAN_SHADOW_OFFSET != KASAN_SHADOW_END - (1UL << (64 - KASAN_SHADOW_SCALE_SHIFT))); - /* - * We cannot check the actual value of KASAN_SHADOW_START during build, - * as it depends on vabits_actual. As a best-effort approach, check - * potential values calculated based on VA_BITS and VA_BITS_MIN. - */ - BUILD_BUG_ON(!IS_ALIGNED(_KASAN_SHADOW_START(VA_BITS), PGDIR_SIZE)); - BUILD_BUG_ON(!IS_ALIGNED(_KASAN_SHADOW_START(VA_BITS_MIN), PGDIR_SIZE)); - BUILD_BUG_ON(!IS_ALIGNED(KASAN_SHADOW_END, PGDIR_SIZE)); + BUILD_BUG_ON(!IS_ALIGNED(_KASAN_SHADOW_START(VA_BITS), SHADOW_ALIGN)); + BUILD_BUG_ON(!IS_ALIGNED(_KASAN_SHADOW_START(VA_BITS_MIN), SHADOW_ALIGN)); + BUILD_BUG_ON(!IS_ALIGNED(KASAN_SHADOW_END, SHADOW_ALIGN)); + + if (!root_level_aligned(KASAN_SHADOW_START)) { + /* + * The start address is misaligned, and so the next level table + * will be shared with the linear region. This can happen with + * 4 or 5 level paging, so install a generic pte_t[] as the + * next level. This prevents the kasan_pgd_populate call below + * from inserting an entry that refers to the shared KASAN zero + * shadow pud_t[]/p4d_t[], which could end up getting corrupted + * when the linear region is mapped. + */ + static pte_t tbl[PTRS_PER_PTE] __page_aligned_bss; + pgd_t *pgdp = pgd_offset_k(KASAN_SHADOW_START); + + set_pgd(pgdp, __pgd(__pa_symbol(tbl) | PGD_TYPE_TABLE)); + } + kasan_pgd_populate(KASAN_SHADOW_START, KASAN_SHADOW_END, NUMA_NO_NODE, true); } @@ -189,20 +231,75 @@ static void __init kasan_map_populate(unsigned long start, unsigned long end, kasan_pgd_populate(start & PAGE_MASK, PAGE_ALIGN(end), node, false); } -static void __init clear_pgds(unsigned long start, - unsigned long end) +/* + * Return the descriptor index of 'addr' in the root level table + */ +static int __init root_level_idx(u64 addr) { /* - * Remove references to kasan page tables from - * swapper_pg_dir. pgd_clear() can't be used - * here because it's nop on 2,3-level pagetable setups + * On 64k pages, the TTBR1 range root tables are extended for 52-bit + * virtual addressing, and TTBR1 will simply point to the pgd_t entry + * that covers the start of the 48-bit addressable VA space if LVA is + * not implemented. This means we need to index the table as usual, + * instead of masking off bits based on vabits_actual. */ - for (; start < end; start += PGDIR_SIZE) - set_pgd(pgd_offset_k(start), __pgd(0)); + u64 vabits = IS_ENABLED(CONFIG_ARM64_64K_PAGES) ? VA_BITS + : vabits_actual; + int shift = (ARM64_HW_PGTABLE_LEVELS(vabits) - 1) * (PAGE_SHIFT - 3); + + return (addr & ~_PAGE_OFFSET(vabits)) >> (shift + PAGE_SHIFT); +} + +/* + * Clone a next level table from swapper_pg_dir into tmp_pg_dir + */ +static void __init clone_next_level(u64 addr, pgd_t *tmp_pg_dir, pud_t *pud) +{ + int idx = root_level_idx(addr); + pgd_t pgd = READ_ONCE(swapper_pg_dir[idx]); + pud_t *pudp = (pud_t *)__phys_to_kimg(__pgd_to_phys(pgd)); + + memcpy(pud, pudp, PAGE_SIZE); + tmp_pg_dir[idx] = __pgd(__phys_to_pgd_val(__pa_symbol(pud)) | + PUD_TYPE_TABLE); +} + +/* + * Return the descriptor index of 'addr' in the next level table + */ +static int __init next_level_idx(u64 addr) +{ + int shift = (ARM64_HW_PGTABLE_LEVELS(vabits_actual) - 2) * (PAGE_SHIFT - 3); + + return (addr >> (shift + PAGE_SHIFT)) % PTRS_PER_PTE; +} + +/* + * Dereference the table descriptor at 'pgd_idx' and clear the entries from + * 'start' to 'end' (exclusive) from the table. + */ +static void __init clear_next_level(int pgd_idx, int start, int end) +{ + pgd_t pgd = READ_ONCE(swapper_pg_dir[pgd_idx]); + pud_t *pudp = (pud_t *)__phys_to_kimg(__pgd_to_phys(pgd)); + + memset(&pudp[start], 0, (end - start) * sizeof(pud_t)); +} + +static void __init clear_shadow(u64 start, u64 end) +{ + int l = root_level_idx(start), m = root_level_idx(end); + + if (!root_level_aligned(start)) + clear_next_level(l++, next_level_idx(start), PTRS_PER_PTE); + if (!root_level_aligned(end)) + clear_next_level(m, 0, next_level_idx(end)); + memset(&swapper_pg_dir[l], 0, (m - l) * sizeof(pgd_t)); } static void __init kasan_init_shadow(void) { + static pud_t pud[2][PTRS_PER_PUD] __initdata __aligned(PAGE_SIZE); u64 kimg_shadow_start, kimg_shadow_end; u64 mod_shadow_start; u64 vmalloc_shadow_end; @@ -224,10 +321,23 @@ static void __init kasan_init_shadow(void) * setup will be finished. */ memcpy(tmp_pg_dir, swapper_pg_dir, sizeof(tmp_pg_dir)); + + /* + * If the start or end address of the shadow region is not aligned to + * the root level size, we have to allocate a temporary next-level table + * in each case, clone the next level of descriptors, and install the + * table into tmp_pg_dir. Note that with 5 levels of paging, the next + * level will in fact be p4d_t, but that makes no difference in this + * case. + */ + if (!root_level_aligned(KASAN_SHADOW_START)) + clone_next_level(KASAN_SHADOW_START, tmp_pg_dir, pud[0]); + if (!root_level_aligned(KASAN_SHADOW_END)) + clone_next_level(KASAN_SHADOW_END, tmp_pg_dir, pud[1]); dsb(ishst); cpu_replace_ttbr1(lm_alias(tmp_pg_dir)); - clear_pgds(KASAN_SHADOW_START, KASAN_SHADOW_END); + clear_shadow(KASAN_SHADOW_START, KASAN_SHADOW_END); kasan_map_populate(kimg_shadow_start, kimg_shadow_end, early_pfn_to_nid(virt_to_pfn(lm_alias(KERNEL_START)))); From patchwork Wed Feb 14 12:29:22 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13556551 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 882D6C48BED for ; Wed, 14 Feb 2024 13:46:03 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=tF6VgPqcjI3W2p/tVyMWthScnK1chgn/PkFvtJAFNnA=; b=f7rV8wdCKSIsv493E9Z6VCq4s9 cFw8eLE/LvW1du3RXIE3DWt/vGokGBM5Ol/gHaMeOr2znotGiZMudrK6NH8BeyyRyFMEj1QA0eO4c dBnNDGNx5kUOl7U8frLcudrW9PW8Pog1FrM3m3QDYVvY6hlJ5/TjSIAqmn4FdqRI8VSmYOEBfto2A lakLkezf2b0JS0rk2V70JxS2iLCrUzHqBfIQ9dUZpFcPfJPVFCxXHqBKEIKe0pLSFn3ni4uB0rc7D 2fc4uS8jDW4qtdRZKafsGAzxULqA3QpsSPovVpWlWsA8ezJy2/2qhLkmfRKg8diEoGePK/WwWvG3m 7JgiTJOg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1raFa9-0000000D3hM-0g25; Wed, 14 Feb 2024 13:45:49 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1raEPs-0000000Cp6H-20U4 for linux-arm-kernel@bombadil.infradead.org; Wed, 14 Feb 2024 12:31:08 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=b9eSR6mWgA6dWrwgPoI/pXiPvt00Qu4wnuxEVbZ5pWU=; b=OjXvJ2S7HNCNpt+alarPfdQWIt isA+KgJAx78IMMj6KrO5BI2ykruLtuf6S6nKx6fw3CyqCIrcG5/pYcwwpCbZR6vud04CocAXqOerG BBroid1oPLq+3GxCbYRyPaj3+OzNhsJx5+067uJdSY5x7aj6UaH9L3iJHhCPiWzkus63XbYvVS07l T5osAG89mbYbHXPGsCqAykMEJIm9gwUOXle5kXZvmaPtJS7GETVcv/tOxXbDaYkg1Ge4qbxbXHjmO 2GD9caqiuHkIaKgvT21TP5cnOxRrpn4e9ikzoKKmojI8u3BDJYfv0wviI2lYUGdFBushLjR4PQryy prrcUKFw==; Received: from mail-yb1-xb4a.google.com ([2607:f8b0:4864:20::b4a]) by desiato.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1raEPn-0000000GECY-3IEk for linux-arm-kernel@lists.infradead.org; Wed, 14 Feb 2024 12:31:05 +0000 Received: by mail-yb1-xb4a.google.com with SMTP id 3f1490d57ef6-dc74ac7d015so7787088276.0 for ; Wed, 14 Feb 2024 04:31:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1707913862; x=1708518662; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=b9eSR6mWgA6dWrwgPoI/pXiPvt00Qu4wnuxEVbZ5pWU=; b=OqnZVrY5AycNhXP7/cHzaUKu4ZEK3rha9Aiq0Hf86TBsKoddhtVVdGHRTC/Kca7a8Z /0yYcq6Ittz8i4TA09/1EL01XQUK5U6uGPMybdgevEZ3OCJ6Ljq+6ScMlosDAwNe1hp2 3JQ9eXK10MCJYncVswIQtjx4pZQsQILe18OlIO87/6m1RfLH/4vezdV6BGWum+vWY4kz /orEuZsTHcB8FgC5ja6EEs3PamhIL+tGAkTzg4bo51xZ4T2+87OzZUfJRXkWmNPXYq0s 8el+OIwwcO/2zNDrN5v92lleu2gMUL0WMMOWhTXaU6iboOddUPK1wSLCJgEOCCqSme5z gqKA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707913862; x=1708518662; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=b9eSR6mWgA6dWrwgPoI/pXiPvt00Qu4wnuxEVbZ5pWU=; b=YnQ5TAf98JDjBuptK5NABi9OcMFdR3Ku5XToaFpNMDzyZKXOGE8EtXRPucZ1dyDva4 dWsFO1GEYOV41vAMw6mpCgZX3Ws264ubk2eAdQIhJs0mNZIml8/JIYxLsX8BEvJ8DFi3 x/R1pFzlUEXNu4r6i9gvv410mD8oLl7HgrWg67jNXba0a61tXEzEHe23Ck5uIwwkEjq+ os+NgjwYq+0V5LV/iJguga52SdN2A/3E8MaoHqqhxfJ2ma2N8hh7DduyHFBrNiMObhWt OggUPC/cCF682Q98eR0w3JnL2STqVSW5THqTaCvEv0VJ34LI5JRtTzBgTVuc368QoYgz 3i9A== X-Gm-Message-State: AOJu0YyXWO3vQJIdHBzrcJ3E3Ew7k13SsO7IfcGw3P/a2Ehj1MZAhRs6 KjAlT9X146JZUOZQ5a3e1DqsiVlgNR3mX/BvfP74rB33Zzc4nQz4xt6uM8QbWjCY/aJHx32M6Q8 bg/nYpXgSYmHfp3lxlnOivUxcyMr5ekX3w7jxyL47VEGSNZmWCjzanXAVMl0n2KvO1x7OmjcVy4 KOg2ULU0V0Gn2e3DvP1wXovluWML7IKljPlBYTITbv X-Google-Smtp-Source: AGHT+IGoGSL/6gG2EV3HKjTUaf67Kf7lON/srfkxKCOc2Cdcn6Hb3plBVzYzqxv6YTnY+YZuzlAazOAH X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:6902:1029:b0:dcc:79ab:e522 with SMTP id x9-20020a056902102900b00dcc79abe522mr86188ybt.11.1707913861578; Wed, 14 Feb 2024 04:31:01 -0800 (PST) Date: Wed, 14 Feb 2024 13:29:22 +0100 In-Reply-To: <20240214122845.2033971-45-ardb+git@google.com> Mime-Version: 1.0 References: <20240214122845.2033971-45-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=8257; i=ardb@kernel.org; h=from:subject; bh=aYJ9ZjtfCDEh0CWupupYYvnBr0yd1EUy+k6IgXXJCEE=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIfXMJtUveTerJc8lv+TzUzk5z6MvXvhIN5f3jF1rK5e1P MuK8n7YUcrCIMbBICumyCIw+++7nacnStU6z5KFmcPKBDKEgYtTACbywInhv/+MtmOlu6tM/G03 bit64/rozp99Ju1rFsje7KnXnxpy+g0jw+1vW6M5LXxYZH/+6Z+60Oq9v0RL24KddhKm8bJGmoe mcgIA X-Mailer: git-send-email 2.43.0.687.g38aa6559b0-goog Message-ID: <20240214122845.2033971-81-ardb+git@google.com> Subject: [PATCH v8 36/43] arm64: mm: Add support for folding PUDs at runtime From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240214_123104_109147_1305A797 X-CRM114-Status: GOOD ( 19.27 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel In order to support LPA2 on 16k pages in a way that permits non-LPA2 systems to run the same kernel image, we have to be able to fall back to at most 48 bits of virtual addressing. Falling back to 48 bits would result in a level 0 with only 2 entries, which is suboptimal in terms of TLB utilization. So instead, let's fall back to 47 bits in that case. This means we need to be able to fold PUDs dynamically, similar to how we fold P4Ds for 48 bit virtual addressing on LPA2 with 4k pages. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/pgalloc.h | 12 ++- arch/arm64/include/asm/pgtable.h | 87 +++++++++++++++++--- arch/arm64/include/asm/tlb.h | 3 + arch/arm64/kernel/cpufeature.c | 2 + arch/arm64/mm/mmu.c | 2 +- arch/arm64/mm/pgd.c | 2 + 6 files changed, 95 insertions(+), 13 deletions(-) diff --git a/arch/arm64/include/asm/pgalloc.h b/arch/arm64/include/asm/pgalloc.h index cae8c648f462..aeba2cf15a25 100644 --- a/arch/arm64/include/asm/pgalloc.h +++ b/arch/arm64/include/asm/pgalloc.h @@ -14,6 +14,7 @@ #include #define __HAVE_ARCH_PGD_FREE +#define __HAVE_ARCH_PUD_FREE #include #define PGD_SIZE (PTRS_PER_PGD * sizeof(pgd_t)) @@ -43,7 +44,8 @@ static inline void __pud_populate(pud_t *pudp, phys_addr_t pmdp, pudval_t prot) static inline void __p4d_populate(p4d_t *p4dp, phys_addr_t pudp, p4dval_t prot) { - set_p4d(p4dp, __p4d(__phys_to_p4d_val(pudp) | prot)); + if (pgtable_l4_enabled()) + set_p4d(p4dp, __p4d(__phys_to_p4d_val(pudp) | prot)); } static inline void p4d_populate(struct mm_struct *mm, p4d_t *p4dp, pud_t *pudp) @@ -53,6 +55,14 @@ static inline void p4d_populate(struct mm_struct *mm, p4d_t *p4dp, pud_t *pudp) p4dval |= (mm == &init_mm) ? P4D_TABLE_UXN : P4D_TABLE_PXN; __p4d_populate(p4dp, __pa(pudp), p4dval); } + +static inline void pud_free(struct mm_struct *mm, pud_t *pud) +{ + if (!pgtable_l4_enabled()) + return; + BUG_ON((unsigned long)pud & (PAGE_SIZE-1)); + free_page((unsigned long)pud); +} #else static inline void __p4d_populate(p4d_t *p4dp, phys_addr_t pudp, p4dval_t prot) { diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 3d7fb3cde83d..b3c716fa8121 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -759,12 +759,27 @@ static inline pmd_t *pud_pgtable(pud_t pud) #if CONFIG_PGTABLE_LEVELS > 3 +static __always_inline bool pgtable_l4_enabled(void) +{ + if (CONFIG_PGTABLE_LEVELS > 4 || !IS_ENABLED(CONFIG_ARM64_LPA2)) + return true; + if (!alternative_has_cap_likely(ARM64_ALWAYS_BOOT)) + return vabits_actual == VA_BITS; + return alternative_has_cap_unlikely(ARM64_HAS_VA52); +} + +static inline bool mm_pud_folded(const struct mm_struct *mm) +{ + return !pgtable_l4_enabled(); +} +#define mm_pud_folded mm_pud_folded + #define pud_ERROR(e) \ pr_err("%s:%d: bad pud %016llx.\n", __FILE__, __LINE__, pud_val(e)) -#define p4d_none(p4d) (!p4d_val(p4d)) -#define p4d_bad(p4d) (!(p4d_val(p4d) & 2)) -#define p4d_present(p4d) (p4d_val(p4d)) +#define p4d_none(p4d) (pgtable_l4_enabled() && !p4d_val(p4d)) +#define p4d_bad(p4d) (pgtable_l4_enabled() && !(p4d_val(p4d) & 2)) +#define p4d_present(p4d) (!p4d_none(p4d)) static inline void set_p4d(p4d_t *p4dp, p4d_t p4d) { @@ -780,7 +795,8 @@ static inline void set_p4d(p4d_t *p4dp, p4d_t p4d) static inline void p4d_clear(p4d_t *p4dp) { - set_p4d(p4dp, __p4d(0)); + if (pgtable_l4_enabled()) + set_p4d(p4dp, __p4d(0)); } static inline phys_addr_t p4d_page_paddr(p4d_t p4d) @@ -788,25 +804,74 @@ static inline phys_addr_t p4d_page_paddr(p4d_t p4d) return __p4d_to_phys(p4d); } +#define pud_index(addr) (((addr) >> PUD_SHIFT) & (PTRS_PER_PUD - 1)) + +static inline pud_t *p4d_to_folded_pud(p4d_t *p4dp, unsigned long addr) +{ + return (pud_t *)PTR_ALIGN_DOWN(p4dp, PAGE_SIZE) + pud_index(addr); +} + static inline pud_t *p4d_pgtable(p4d_t p4d) { return (pud_t *)__va(p4d_page_paddr(p4d)); } -/* Find an entry in the first-level page table. */ -#define pud_offset_phys(dir, addr) (p4d_page_paddr(READ_ONCE(*(dir))) + pud_index(addr) * sizeof(pud_t)) +static inline phys_addr_t pud_offset_phys(p4d_t *p4dp, unsigned long addr) +{ + BUG_ON(!pgtable_l4_enabled()); -#define pud_set_fixmap(addr) ((pud_t *)set_fixmap_offset(FIX_PUD, addr)) -#define pud_set_fixmap_offset(p4d, addr) pud_set_fixmap(pud_offset_phys(p4d, addr)) -#define pud_clear_fixmap() clear_fixmap(FIX_PUD) + return p4d_page_paddr(READ_ONCE(*p4dp)) + pud_index(addr) * sizeof(pud_t); +} -#define p4d_page(p4d) pfn_to_page(__phys_to_pfn(__p4d_to_phys(p4d))) +static inline +pud_t *pud_offset_lockless(p4d_t *p4dp, p4d_t p4d, unsigned long addr) +{ + if (!pgtable_l4_enabled()) + return p4d_to_folded_pud(p4dp, addr); + return (pud_t *)__va(p4d_page_paddr(p4d)) + pud_index(addr); +} +#define pud_offset_lockless pud_offset_lockless + +static inline pud_t *pud_offset(p4d_t *p4dp, unsigned long addr) +{ + return pud_offset_lockless(p4dp, READ_ONCE(*p4dp), addr); +} +#define pud_offset pud_offset + +static inline pud_t *pud_set_fixmap(unsigned long addr) +{ + if (!pgtable_l4_enabled()) + return NULL; + return (pud_t *)set_fixmap_offset(FIX_PUD, addr); +} + +static inline pud_t *pud_set_fixmap_offset(p4d_t *p4dp, unsigned long addr) +{ + if (!pgtable_l4_enabled()) + return p4d_to_folded_pud(p4dp, addr); + return pud_set_fixmap(pud_offset_phys(p4dp, addr)); +} + +static inline void pud_clear_fixmap(void) +{ + if (pgtable_l4_enabled()) + clear_fixmap(FIX_PUD); +} /* use ONLY for statically allocated translation tables */ -#define pud_offset_kimg(dir,addr) ((pud_t *)__phys_to_kimg(pud_offset_phys((dir), (addr)))) +static inline pud_t *pud_offset_kimg(p4d_t *p4dp, u64 addr) +{ + if (!pgtable_l4_enabled()) + return p4d_to_folded_pud(p4dp, addr); + return (pud_t *)__phys_to_kimg(pud_offset_phys(p4dp, addr)); +} + +#define p4d_page(p4d) pfn_to_page(__phys_to_pfn(__p4d_to_phys(p4d))) #else +static inline bool pgtable_l4_enabled(void) { return false; } + #define p4d_page_paddr(p4d) ({ BUILD_BUG(); 0;}) /* Match pud_offset folding in */ diff --git a/arch/arm64/include/asm/tlb.h b/arch/arm64/include/asm/tlb.h index 0150deb332af..a947c6e784ed 100644 --- a/arch/arm64/include/asm/tlb.h +++ b/arch/arm64/include/asm/tlb.h @@ -103,6 +103,9 @@ static inline void __pud_free_tlb(struct mmu_gather *tlb, pud_t *pudp, { struct ptdesc *ptdesc = virt_to_ptdesc(pudp); + if (!pgtable_l4_enabled()) + return; + pagetable_pud_dtor(ptdesc); tlb_remove_ptdesc(tlb, ptdesc); } diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c index bc5e4e569864..94f035f6c421 100644 --- a/arch/arm64/kernel/cpufeature.c +++ b/arch/arm64/kernel/cpufeature.c @@ -1767,6 +1767,8 @@ static int __init __kpti_install_ng_mappings(void *__unused) if (levels == 5 && !pgtable_l5_enabled()) levels = 4; + else if (levels == 4 && !pgtable_l4_enabled()) + levels = 3; remap_fn = (void *)__pa_symbol(idmap_kpti_install_ng_mappings); diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index 8e5b3a7c5afd..b131ed31a6c8 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -1065,7 +1065,7 @@ static void free_empty_pud_table(p4d_t *p4dp, unsigned long addr, free_empty_pmd_table(pudp, addr, next, floor, ceiling); } while (addr = next, addr < end); - if (CONFIG_PGTABLE_LEVELS <= 3) + if (!pgtable_l4_enabled()) return; if (!pgtable_range_aligned(start, end, floor, ceiling, P4D_MASK)) diff --git a/arch/arm64/mm/pgd.c b/arch/arm64/mm/pgd.c index 3c4f8a279d2b..0c501cabc238 100644 --- a/arch/arm64/mm/pgd.c +++ b/arch/arm64/mm/pgd.c @@ -21,6 +21,8 @@ static bool pgdir_is_page_size(void) { if (PGD_SIZE == PAGE_SIZE) return true; + if (CONFIG_PGTABLE_LEVELS == 4) + return !pgtable_l4_enabled(); if (CONFIG_PGTABLE_LEVELS == 5) return !pgtable_l5_enabled(); return false; From patchwork Wed Feb 14 12:29:23 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13556549 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 6066BC48BC3 for ; Wed, 14 Feb 2024 13:46:03 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=L+oaVMJj/Bjzy/8SKJAW9+ByBpJnMcxcCEypuGz6SxQ=; b=PRy7mnVdfmd/AO8ZEF0CHAEkAZ weCc31qeCVxc9LzlauJBai+ctAKlW+NP6MkM9KfgCToj7QkKARB3PG0Oo9GuPpfaO+eC28flXSZLp yI7JVBrD30JiYhT1BXRK9SYZnwPTFIPPVeDVTpgZ4fzANJj2wN9/4xmwxIFZ3YUxENnDi6EzOmN4P hKkqd61d64xD1ZH6aIx9YmMILQ4ZVNbZGVl/Zj0GV2DeD9KedZETJoAPQiUm5040YOU+sjsBSNIMq ILBRTYyEp4YM6H4XP0+uMGFGj3kTrFTe5rdL2mDYGeLSfzyg2TFAiCVBThXV6o6YFvJ0xTKwvZRqY yPc78EWg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1raFa9-0000000D3hx-3j7U; Wed, 14 Feb 2024 13:45:49 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1raEPw-0000000Cp89-1d0X for linux-arm-kernel@bombadil.infradead.org; Wed, 14 Feb 2024 12:31:12 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=xB/mYZkbr5+wxmNwrn7xpjisrtVvEFG2X093yBnkuGY=; b=UzPIOiEIaBh8uFItbmOfPdJ0P8 xtZPx8QtnZaJ1ga++4kvtaNWLsX8h8IWaXQx8YbqRj5XhdwRA6nM82BHDkOpEJuNO4Uo8mPMm02mI AKR3QU5FNT7eMQ/cS+zb/LuGCnB8cexBSxzXOcvOini3nb57TZnbrMpVn0Ocu8LIW7sqvWELJUjJq bpD7BZgsc3snfV5PsKKOQ0zwi5g4VmxcLz8AieuIigAEK56+Pwgo5p4o9n8LxDFiWcEjlvdOqxMx9 MtHCTMxgUdbXbvHk/Xr1J3PW8GpBygqPdCuEFyYyH4CEXSC1jQ4Z/WEVUdjcdGHW9XlHkG3hU9fxW sd+rg55Q==; Received: from mail-wm1-x34a.google.com ([2a00:1450:4864:20::34a]) by desiato.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1raEPp-0000000GEDJ-3Yib for linux-arm-kernel@lists.infradead.org; Wed, 14 Feb 2024 12:31:11 +0000 Received: by mail-wm1-x34a.google.com with SMTP id 5b1f17b1804b1-410040e1fcdso31158965e9.0 for ; Wed, 14 Feb 2024 04:31:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1707913864; x=1708518664; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=xB/mYZkbr5+wxmNwrn7xpjisrtVvEFG2X093yBnkuGY=; b=CBnERD09DiGjiXL/r/qzrulmydi1S/hDdeSeEQshnUNU7WvyTde9OLM8g8ZtcwuYRO cV61/lJ9gIzgdWM7Diz4V1vqcIaACDBiovz5MxU6gImnlamZ/n36wU3vzml+FYirdzqW YXFGSc+X23C+LDprxSH2mFoSpA2QPz+2hufDYoQj4nyLFKrwWYGgjxjaW253XU6volVN I4oCOJ3hnX8inhVDT+vM1oduRmsC916vJszuXuJfKtEk77h5BBLRsnBjEffY89b0t7vs csUti47Ps1tpXtryLv7B8K1pNpFAR70ac5jx+k34ylmFnFrJTjpD3gjUgspbk3gaO9ji OqBg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707913864; x=1708518664; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=xB/mYZkbr5+wxmNwrn7xpjisrtVvEFG2X093yBnkuGY=; b=ulabBSPomx1yfo4HAAl+QrX7zF+m87GXTVWaUTCU9vH3w6yoHr/dnEK3z3QkShsON6 i63d6AIxvPKazY8m2RgQniX9KL4n2uTxKxi8olQXZidFODFkSKuwGVIwYb6TSG6X0Un1 E/8BImfNANiB5J3pRg41kMWEh4RxgiI3Fc7/3twIgCfPOmS3tUcRBsmTTT9bQoSHkPDW Hw7OKy0OHCiBpxGYDBbSCnDOdmvZNh2PwH/mP5/CS71MGLGmFcmLpFVXaoFlR19fboUw F37Rwvp3+i7Em6pd8fmsCwYMKXOCZj4XK7O1VQaq+NTjufqqOaxx8eJOTYELE3V49Dik IY0Q== X-Gm-Message-State: AOJu0YyZfI+Bk8wPxyiXlGISKodpbJykQspZMDuM0GrXfOjI36S0zyZf mPXN8A57LKbt9mgWHox6SNJKJO8ovFWYc3CseNWgGt6JoRLlKVD42gP2pKhis9HDJwOaCcQrT8y qS+5GZ816f/vAmzKgtwOHbxlgymhIZoORfLT9Z36fZO8BFMCcJhHUJZfsLBo7IfFIJLkTwkLdNi CLpi4ZAyGBUTja+P6uOzXgUsAwDC40vDK9xKJCI9Z0 X-Google-Smtp-Source: AGHT+IEA6chSQ8en9KQRNq09V0YgH8/aEkA9N6fU2OWZfPnDx3EAYTviMPrrx7eHgjOFy6B7vW1dI3DN X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:600c:a3a1:b0:411:a7e2:3d1 with SMTP id hn33-20020a05600ca3a100b00411a7e203d1mr8913wmb.6.1707913863998; Wed, 14 Feb 2024 04:31:03 -0800 (PST) Date: Wed, 14 Feb 2024 13:29:23 +0100 In-Reply-To: <20240214122845.2033971-45-ardb+git@google.com> Mime-Version: 1.0 References: <20240214122845.2033971-45-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=1822; i=ardb@kernel.org; h=from:subject; bh=Z0ZgGEfpKUtf6qjfEXxNzR8II3DYIwckYLN0dHdtLDQ=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIfXMJjWXdN7DTzk/PGts/GIvspDv3HdnWa7txbdc7DT+H nR0PencUcrCIMbBICumyCIw+++7nacnStU6z5KFmcPKBDKEgYtTACYSrcrIMG/NFlNple0NV7IM OX3bNt0yfZeudM8lMWFuyS3GqqIl3xn+yk+MTXnN8MdV4ouAWdDkZ3flTPbweQa5PvbTkHPObFL gAgA= X-Mailer: git-send-email 2.43.0.687.g38aa6559b0-goog Message-ID: <20240214122845.2033971-82-ardb+git@google.com> Subject: [PATCH v8 37/43] arm64: ptdump: Disregard unaddressable VA space From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240214_123106_180477_75C2F2D1 X-CRM114-Status: GOOD ( 16.84 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel Configurations built with support for 52-bit virtual addressing can also run on CPUs that only support 48 bits of VA space, in which case only that part of swapper_pg_dir that represents the 48-bit addressable region is relevant, and everything else is ignored by the hardware. Our software pagetable walker has little in the way of input address validation, and so it will happily start a walk from an address that is not representable by the number of paging levels that are actually active, resulting in lots of bogus output from the page table dumper unless we take care to start at a valid address. So define the start address at runtime based on vabits_actual. Signed-off-by: Ard Biesheuvel --- arch/arm64/mm/ptdump.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/arch/arm64/mm/ptdump.c b/arch/arm64/mm/ptdump.c index 5f0849528ccf..16d0cf1d85c4 100644 --- a/arch/arm64/mm/ptdump.c +++ b/arch/arm64/mm/ptdump.c @@ -313,7 +313,6 @@ static void __init ptdump_initialize(void) static struct ptdump_info kernel_ptdump_info __ro_after_init = { .mm = &init_mm, - .base_addr = PAGE_OFFSET, }; void ptdump_check_wx(void) @@ -329,7 +328,7 @@ void ptdump_check_wx(void) .ptdump = { .note_page = note_page, .range = (struct ptdump_range[]) { - {PAGE_OFFSET, ~0UL}, + {_PAGE_OFFSET(vabits_actual), ~0UL}, {0, 0} } } @@ -370,6 +369,7 @@ static int __init ptdump_init(void) static struct addr_marker address_markers[ARRAY_SIZE(m)] __ro_after_init; kernel_ptdump_info.markers = memcpy(address_markers, m, sizeof(m)); + kernel_ptdump_info.base_addr = page_offset; ptdump_initialize(); ptdump_debugfs_register(&kernel_ptdump_info, "kernel_page_tables"); From patchwork Wed Feb 14 12:29:24 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13556550 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id E378BC48BEE for ; Wed, 14 Feb 2024 13:46:03 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=Tn5H5NmtcsXRGhQP1Wh5JVUKx9hDeji/qwMI4A+/Hh8=; b=vnnLl8ov5GT+v3JTcJsfdhN7BQ He5b1eLo6Qc6lpKQr4hwHtT+DnyoWcGBAADrBkrv1bKZMvlPhtohpne1DuYdb7uOE7WYogj3Q7Xfo ifc7NthTALI3Ic4trs2wttu8hdCsXmy+FIwXKObeRt9lxgtJiVRn6S7pWBb7w/isfaeE0engwicWZ ahT+z7KmNUyqdH7BBMK9mwA7ysAGMAuXWIMqHTSvDAVtBWSqDet3gzOLy39Fr4d1A5zfK+0UKBNNE Um5PQOXHXH6ZOCjbH7tebkD/mUbLMFzCQGydchfZrzJOWOcY3oDWmLoN3AT5ov/i3OCIfG0QqvV8x uStasvDA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1raFaA-0000000D3ip-46ol; Wed, 14 Feb 2024 13:45:50 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1raEPy-0000000Cp92-1RPu for linux-arm-kernel@bombadil.infradead.org; Wed, 14 Feb 2024 12:31:14 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=nim3c2gTM7nh9rCbkF9XqrGKHEjAQ3AiMPielUv+s5w=; b=WUidQBhnXvLvTgdzPyaVs9mxQ+ k2pVcy5ZZwiKQq5ms0aDyktvSA9+pZfFc70YWA7tubnIXIBC2lWdycZxS28BaXRp7V4ksnTdy2eyi RX8YjCz1qM/YqF/czwHJRD80/7HIe6WPdWD9XEh490hSpE5J15urCVhi+l2sO52JOwN+uJHCd/wdu cw+oVjo4+e1GmgYSdk9qeRnPkGo4FwtZ1ESxiahsvHNpZ/zRhiV9azfH+xB/aDmSheV/WbRTU7xpD Psd+xwDwsR2r1MyzR3SExB4Tl4bIJCSeYiJw+AFU5UHusz/7vvOpyVPOHbQN7RpzPzhmH78D+GvSi v+E4Zbgw==; Received: from mail-yb1-xb4a.google.com ([2607:f8b0:4864:20::b4a]) by desiato.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1raEPs-0000000GEEE-1qb1 for linux-arm-kernel@lists.infradead.org; Wed, 14 Feb 2024 12:31:12 +0000 Received: by mail-yb1-xb4a.google.com with SMTP id 3f1490d57ef6-dc3645a6790so9889499276.0 for ; Wed, 14 Feb 2024 04:31:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1707913866; x=1708518666; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=nim3c2gTM7nh9rCbkF9XqrGKHEjAQ3AiMPielUv+s5w=; b=Vz/dgE+5lJJzzAXABGSf9YhHyqmh/sCf286HGHL95gUXE9E1DwJ7wrtqevK3UdA3yE sQ8QmZpUTANWcgeYQp9gfydiRnef+I+q54oH0JpuxoSHvENZbqb5Vge0U2jC/TjfhQ7P 4KH31BM5MAVEfc5qd6KT6PTnXyjrVbheX+wjKdvmU6gdcL+EPHpGGevwgyG3mt6XuSM7 aK0FqP3OeYV4ruhXGDdXBr8g+6AFfBuMtGduldGDNPCT+fVNjHCvIiipbhagSZRUH45j zMB2gNyqSiSJJlcQcKEcABLKPYydCYw2cNepf52am+nrGwfkR62EQ26mKSGVExdcpMIS NVkw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707913866; x=1708518666; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=nim3c2gTM7nh9rCbkF9XqrGKHEjAQ3AiMPielUv+s5w=; b=Up8IJiNemq1tkGMKi5VuM4DVlRzeRDtOeWWMzfBSvGyLUIJWlBEJUFgDhkGIhLA2s7 n1iHnty8sz/rO4LRxJPiMc/O08+zOd/qaxjSyND6CWSVTgLG5ELBxMJXgie7vjODhT8Q h8h2YD5efRoABFaiV1FD3CLh+ckVbmMQjTjg7LdEOkQQFHVXlvQtvUHW0MLrNOifAEEM EDMvzEo4DVIdPWniId6kkXRaI//t8t8T1kddrc3/+QqM8zcMqDKcWtGAC8bbz2fD3HpQ ZuCzeC8TC/pRdAEk8qxGMYpmjaoUBn7zGP/5RLjzNoRkHHCNOCjoGdZPANbu0VIiS8eu ROJg== X-Gm-Message-State: AOJu0YwAxxOPEzRy0hBUBK+jxvefqKT18RF0en/EKAhk/SmAhKEmkp2o 7z/6ZLAh+zrtF98JRxZHcRjdQs/bFTVjYSlFZlYtSx2QTwoGAEh1MvQfkl3/qsAlNwmO/373mw7 TMLsG8iw5l5ZqxxPlisWhA1NtEgton1v7Bk55F/8ULEbHLSdj9QjmhjFLvcc7R3C6bTsJoVcJb4 DibOhbt0gB6amTcry4dhG5UWHD9f2s208sKRtjCEgq X-Google-Smtp-Source: AGHT+IHB0JANakyodDNczMw88qFBA3HYj/BIyZyWpSgCjgAui1HyVFg6ldJiPwAwd18NMF4sOT92ow3v X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a25:dc02:0:b0:dcb:b9d7:2760 with SMTP id y2-20020a25dc02000000b00dcbb9d72760mr490452ybe.13.1707913866351; Wed, 14 Feb 2024 04:31:06 -0800 (PST) Date: Wed, 14 Feb 2024 13:29:24 +0100 In-Reply-To: <20240214122845.2033971-45-ardb+git@google.com> Mime-Version: 1.0 References: <20240214122845.2033971-45-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=2891; i=ardb@kernel.org; h=from:subject; bh=YjhX0rklZvJzAi6YOz38nWlFpNJR9o+uJt31nCu8kg8=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIfXMJvUfide9H92XLJEsXPOhrbt4szRnyVoZv2OLW+blH cv48TWzo5SFQYyDQVZMkUVg9t93O09PlKp1niULM4eVCWQIAxenAExEQp+R4Yxof4V34T6TrXfW xh/T+P8l4MwhjeaQ0mmLLK/P+ZT67DfDT8bHU5nez/pwKkfTo+nslyWM6+aw7Jz75MT8nY1HD2Z LaDACAA== X-Mailer: git-send-email 2.43.0.687.g38aa6559b0-goog Message-ID: <20240214122845.2033971-83-ardb+git@google.com> Subject: [PATCH v8 38/43] arm64: ptdump: Deal with translation levels folded at runtime From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240214_123108_817433_98DB937B X-CRM114-Status: GOOD ( 19.57 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel Currently, the ptdump code deals with folded PMD or PUD levels at build time, by omitting those levels when invoking note_page. IOW, note_page() is never invoked with level == 1 if P4Ds are folded in the build configuration. With the introduction of LPA2 support, we will defer some of these folding decisions to runtime, so let's take care of this by overriding the 'level' argument when this condition triggers. Substituting the PUD or PMD strings for "PGD" when the level in question is folded at build time is no longer necessary, and so the conditional expressions can be simplified. This also makes the indirection of the 'name' field unnecessary, so change that into a char[] array, and make the whole thing __ro_after_init. Note that the mm_p?d_folded() functions currently ignore their mm pointer arguments, but let's wire them up correctly anyway. Signed-off-by: Ard Biesheuvel --- arch/arm64/mm/ptdump.c | 17 ++++++++++++----- 1 file changed, 12 insertions(+), 5 deletions(-) diff --git a/arch/arm64/mm/ptdump.c b/arch/arm64/mm/ptdump.c index 16d0cf1d85c4..5b87f8d623f7 100644 --- a/arch/arm64/mm/ptdump.c +++ b/arch/arm64/mm/ptdump.c @@ -48,6 +48,7 @@ struct pg_state { struct ptdump_state ptdump; struct seq_file *seq; const struct addr_marker *marker; + const struct mm_struct *mm; unsigned long start_address; int level; u64 current_prot; @@ -144,12 +145,12 @@ static const struct prot_bits pte_bits[] = { struct pg_level { const struct prot_bits *bits; - const char *name; - size_t num; + char name[4]; + int num; u64 mask; }; -static struct pg_level pg_level[] = { +static struct pg_level pg_level[] __ro_after_init = { { /* pgd */ .name = "PGD", .bits = pte_bits, @@ -159,11 +160,11 @@ static struct pg_level pg_level[] = { .bits = pte_bits, .num = ARRAY_SIZE(pte_bits), }, { /* pud */ - .name = (CONFIG_PGTABLE_LEVELS > 3) ? "PUD" : "PGD", + .name = "PUD", .bits = pte_bits, .num = ARRAY_SIZE(pte_bits), }, { /* pmd */ - .name = (CONFIG_PGTABLE_LEVELS > 2) ? "PMD" : "PGD", + .name = "PMD", .bits = pte_bits, .num = ARRAY_SIZE(pte_bits), }, { /* pte */ @@ -227,6 +228,11 @@ static void note_page(struct ptdump_state *pt_st, unsigned long addr, int level, static const char units[] = "KMGTPE"; u64 prot = 0; + /* check if the current level has been folded dynamically */ + if ((level == 1 && mm_p4d_folded(st->mm)) || + (level == 2 && mm_pud_folded(st->mm))) + level = 0; + if (level >= 0) prot = val & pg_level[level].mask; @@ -288,6 +294,7 @@ void ptdump_walk(struct seq_file *s, struct ptdump_info *info) st = (struct pg_state){ .seq = s, .marker = info->markers, + .mm = info->mm, .level = -1, .ptdump = { .note_page = note_page, From patchwork Wed Feb 14 12:29:25 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13556555 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 0CC91C48BEE for ; Wed, 14 Feb 2024 13:46:10 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=MojlkTuoESdXjtRMi0uIikY9+21pnFVI5Yo7TuFCtOI=; b=DZUjv5YbTX2KXNHmlj3JhYzoz3 Fx+Ez2yIu62N8oOAVwxSpvwo+PItXNRKRf457FhGQMRJhesyq2rSfyYCihDcsfJ6unDSJ6Pc4L5VY gcPLVAz9HPDIgGGJQDbJw5LhjXMlcflCmnH+P9pW8l/F5GI1EIYFnzlkmozJyiyzf6bPqKR/LAF9Y 8eWbAuZauCDC03aSCNbvXNsUj00vBIZM7o3vBBZZgMDcHSRF/ytl0Q4vXQXI7TzEytP6W1x+XWgUh vmGjSPG5t09RlqSxHU3LbwG7hH1P0x1k1x2ut1uiDBxNlINPkh4yV3EItWVw0D36ftk/oFjGyYCVH hvo4kkEg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1raFaG-0000000D3nv-1joL; Wed, 14 Feb 2024 13:45:56 +0000 Received: from mail-yw1-x114a.google.com ([2607:f8b0:4864:20::114a]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1raEPu-0000000Cp6u-0Y33 for linux-arm-kernel@lists.infradead.org; Wed, 14 Feb 2024 12:31:24 +0000 Received: by mail-yw1-x114a.google.com with SMTP id 00721157ae682-6047a047f4cso138170147b3.3 for ; Wed, 14 Feb 2024 04:31:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1707913868; x=1708518668; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=zEHrKJHHdGBSEF5kWmEoE5hindUcwiKdKN6yKAgIVcg=; b=eINUh+L5UZ8EN3Q0iEHydQ7SchikV8LrwY0FOKnEZ2F9GzbJOAkzxtGwfVNPKDxzF6 GfCXTmQjiTh9ePsugBBExH9rxKxipZXB/MAOYaDB94/BKpSMiAyOtBY1v539q4yRNsFY 3VHsCAausE77hr0AnKAS9Nz07t93Pcxj7B+ZKnIu4HIr9whjH7PYozAgqOHPmG8Tr2tV BK0JXOL1ayP4jfGYWLs3HQgjYe4+GuW8xS1onilslvrqP8p2qwxLSu/iYPUmSakyPvPC HTTZ2jWbKvrWIcHiFngQP2kEkWg4Bf+7OC+YWB3fh4b4EI/QYvaLNf5Ov2D50HGjWXZL IhNg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707913868; x=1708518668; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=zEHrKJHHdGBSEF5kWmEoE5hindUcwiKdKN6yKAgIVcg=; b=Q/xygZs5hsQidDT+1exCv/P4aUGKJJ3yI+1dXQPu605lDYaAYiFc06tMBITTr+sIKF 1xPx5WDXxDqe6twQ0V8yawYO3F22+iCZvv6BWD8B2+7PJUufySAQrc29qJHmeRR11pxj rq8nEMGGLcvqxx80cRPWBYbl4pp36W22r8jYTHqZbLrp4+yQVS4j+h9sOo60Lm+uIhLu Qgc3QANT/YSzmk0ZKDMik/tW7a+zw+t6OyNxkEvxAyne91waNzFVWtKOnFtupzgdv1G7 lN/bG6J6+RFos9h/0b/XU2ObR2a6oKcJFW4zNBYqh6XuZ5ozjZmCPCwM/kDybyKZu0g1 hDyw== X-Gm-Message-State: AOJu0YyAZ8ml7Kn1JAyfmn8GWArZnLTz7eOpBLSRCXe2q7tapPXdgNjS qz+7Wv+86F+VWnRtHP+PUDwstN1jYFxd+Ub+24ESf8SooLTdJ/t5WSS+DyabIhumNV0llyIdn6t 3h/lT3qKONKlYNdiO85cxKS0iM73Mu9bZqWzSzH0LmaJ+Fv0yS+jj8t6uFDgTIZHaV84qEYGUsW vJYc4GDob9vT6wtuVq7WOHJ4UyFffJ00mGXcad40Eh X-Google-Smtp-Source: AGHT+IHn9H1f2wkIrn2GAVpRVZK/42A8A9Khe1TF7loeMrPf0NK1J/MzkS156JrjX4H66X38UTJdvYL+ X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:6902:72c:b0:dc7:53a0:83ad with SMTP id l12-20020a056902072c00b00dc753a083admr554670ybt.5.1707913868587; Wed, 14 Feb 2024 04:31:08 -0800 (PST) Date: Wed, 14 Feb 2024 13:29:25 +0100 In-Reply-To: <20240214122845.2033971-45-ardb+git@google.com> Mime-Version: 1.0 References: <20240214122845.2033971-45-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=1120; i=ardb@kernel.org; h=from:subject; bh=M9K6TCvADFrSQrq3imVxSRn39wrusa5LSxXwXi7bCXI=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIfXMJo1DDcs80gXO5Mjav5iievB8GhffF72k9jXck0zeu Ehueq/fUcrCIMbBICumyCIw+++7nacnStU6z5KFmcPKBDKEgYtTACYSdpbhn82jsFsyO5S2vppc zT45dx/3LAZn9ptzvRx9Vfjjo3T43Rj+ezgc2v6oq81ooeleQaaTy8yW/u8/3xsWU3Hzcr/fZsY SfgA= X-Mailer: git-send-email 2.43.0.687.g38aa6559b0-goog Message-ID: <20240214122845.2033971-84-ardb+git@google.com> Subject: [PATCH v8 39/43] arm64: kvm: avoid CONFIG_PGTABLE_LEVELS for runtime levels From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook , Oliver Upton X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240214_043110_441802_8B9A597D X-CRM114-Status: GOOD ( 12.81 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel get_user_mapping_size() uses vabits_actual and CONFIG_PGTABLE_LEVELS to provide the starting point for a table walk. This is fine for LVA, as the number of translation levels is the same regardless of whether LVA is enabled. However, with LPA2, this will no longer be the case, so let's derive the number of levels from the number of VA bits directly. Acked-by: Marc Zyngier Acked-by: Oliver Upton Signed-off-by: Ard Biesheuvel --- arch/arm64/kvm/mmu.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c index 6fa9e816df40..cd9456a03e38 100644 --- a/arch/arm64/kvm/mmu.c +++ b/arch/arm64/kvm/mmu.c @@ -805,7 +805,7 @@ static int get_user_mapping_size(struct kvm *kvm, u64 addr) .pgd = (kvm_pteref_t)kvm->mm->pgd, .ia_bits = vabits_actual, .start_level = (KVM_PGTABLE_LAST_LEVEL - - CONFIG_PGTABLE_LEVELS + 1), + ARM64_HW_PGTABLE_LEVELS(pgt.ia_bits) + 1), .mm_ops = &kvm_user_mm_ops, }; unsigned long flags; From patchwork Wed Feb 14 12:29:26 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13556554 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 948F7C48BC1 for ; Wed, 14 Feb 2024 13:46:09 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=qZ0BMK+/RYR8w9DavIT4m71TCmIjfyCP9ILpbHR3tF0=; b=Gncisvp42ddCScZTFrLW32uuZ3 rcc/xoDdjTAZscIuNT5xFW5zF77WN4W9MsXRrHy668GPn8INy2KLIZ2doLrldCIax/SjEChJQxasX 7wrZEVutbGA4pIMcQZOrbv1cd7HHdFtv0qjJP0okx9qhbFK7g0u7z13t8jSHzc/Y8SO9xJE65Ssid HF9uJuE5TVDrqx1CvTOD9DnOfG/bF+jnK4i8ZFZlH32lRbxRMCGMAzogC2pWfW4kMWTp576wAHfwG FIIBPwurT0O6WsONqQgBthg8472mc+EX2PPSlpUMhA3Pn7DTVU+xUi9ncH4NveJ2MWI60bzNS71HS wdeaXcsw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1raFaE-0000000D3lw-0z4Z; Wed, 14 Feb 2024 13:45:54 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1raEQ2-0000000CpAj-1vLZ for linux-arm-kernel@bombadil.infradead.org; Wed, 14 Feb 2024 12:31:19 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=9KPK5FCNE9zxEgfix6ahooLeTfePYk7EUjPQG8oQi2w=; b=gr1uBOkCsb7FhnpVxLRImGW2Uk 7CpIWDHU9H55DtWV3k0b5NfxXe33/I9ScqP8PBbWsDQLkL63RiMt9A4Ee6yk1UjSvFXxjJyjleVDb Nu00PthgFdaordbd1W/SjgsvbCFNtPJslrpL/X5pl7hf0kuWPAem01k7Uij8U4spCnN0VR4yDj2S5 QGY63X3K7tmEgUQieb94cRT0NaecwyKp/Ee/MtPBJdJ98UuBgwzBc1LCjUCEygxsaH6b0kZu6Hyoh Pk6vAFOnmSHUzPhIqdIqq7VfI7RHgkBsq22OvgCDMKl9ZBUfQF+yznUxpnN3ErGXsChvK/qWSxLXw MevdpBOQ==; Received: from mail-wr1-x449.google.com ([2a00:1450:4864:20::449]) by desiato.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1raEPw-0000000GEFp-28wn for linux-arm-kernel@lists.infradead.org; Wed, 14 Feb 2024 12:31:17 +0000 Received: by mail-wr1-x449.google.com with SMTP id ffacd0b85a97d-33b316fcaecso2041994f8f.3 for ; Wed, 14 Feb 2024 04:31:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1707913871; x=1708518671; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=9KPK5FCNE9zxEgfix6ahooLeTfePYk7EUjPQG8oQi2w=; b=CmknjW5ZiOsnhZ7vgzC1z+SIhMFSPf4+uHLAeg2mslpJte8HpZRTmB6666O1rESbkB /uTJwym4vA35q7YdcchyvWtkPabAo8hn3+SG5EQJlZAAnKPctonuwZ6/6y1bBTBojBfx egXxsGTHJyd1JwW3YXdKqIgcvkUj3W5uo4LDXE1hYdu297awWd/jBlIASQXqK/lxADJl VZbmTUnPfr1YEmzmz9j6bXZMkBFtTOwHVvzeIShvtrRoxtqSGqOkxYycR3JMQx3jEwdN EOyuaPeLrVDsyeGWYXj0oW+0s0JbUDJmo/YS4ja6kDx574ezGBaVxJLRfQQC/Ap6vdzL kTNA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707913871; x=1708518671; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=9KPK5FCNE9zxEgfix6ahooLeTfePYk7EUjPQG8oQi2w=; b=doD0Y+Jk7v+1xV3y5iddFC3CNf9sua0MJijrEYae9MjdSSzY3xUsdFUFbBXEc+cCjU 2yV7ifacg8BCj9/Iy9Wl4vyirANyMhoMipqBE0pIdzaHdhmBqeD4Yt4lEiMN1OyCWsK5 /YekTIlZxxd8bmLft92jrcSqvAq/t2RsMudicxJLu35s0nRRe4piZDBAUQCi4afVSngX NI6oV1/BS4jyLFB0L6wXlofRY21HtIYq9XNZkRU4yiKoCKFX695vw3/hcrEK/J31Z2fk 4NoCgj7Em/Cg+hMl0kXGC4CoWJWUIQ0kNO6W8PhvUa6IW4HxT0sXz5AUCmwINJtwNDd7 zM7Q== X-Gm-Message-State: AOJu0YwxbhSWfOCINMU+/ywFgjlrG1BS7N3GW5GTLJvgoOPMoUQf3731 lSk5nRxE96YWcHa0CmFKDEPLIlGqaidezyYQ9pk278reTdq1UDFXlJEpealqs1Db0qR7bpqmxFk CMHyv3USJpbx3t9JQbc3gjSFA6HfRU3k+5zAUi3FMeRDBeL+8CBKuE3IVMqffyCDc8sRclyjUod ysjSbDEER5u+cD/EGAFn7NAvVnGyZ5EjIqyLb55OY7 X-Google-Smtp-Source: AGHT+IF0q5bOQz/3msxwYVGq0tsMlHO2zp4p+XInYuX9wSfRsvMiHwy7UcX6ucJsX/6CRdlrmCjAwni1 X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:6000:a0f:b0:33c:e1db:710e with SMTP id co15-20020a0560000a0f00b0033ce1db710emr7723wrb.8.1707913870961; Wed, 14 Feb 2024 04:31:10 -0800 (PST) Date: Wed, 14 Feb 2024 13:29:26 +0100 In-Reply-To: <20240214122845.2033971-45-ardb+git@google.com> Mime-Version: 1.0 References: <20240214122845.2033971-45-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=5018; i=ardb@kernel.org; h=from:subject; bh=CItFEmQ61JD6u+vUYuXJZMKoPIeXpt5+N3BV4b917tI=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIfXMJs1V3T83/F87c9+lKLdZXfdizs5ntP8eWa4TJXnnz 6++3Yc2d5SyMIhxMMiKKbIIzP77bufpiVK1zrNkYeawMoEMYeDiFICJJNQy/M+0X3tXV2j1lxPv lkcle17V5lNeICnSZ92myevn+2/OGR+G/xGPMk//aju9SH71o/Xzr+/L4twf837uttxti9MES9k WR3ECAA== X-Mailer: git-send-email 2.43.0.687.g38aa6559b0-goog Message-ID: <20240214122845.2033971-85-ardb+git@google.com> Subject: [PATCH v8 40/43] arm64: Enable 52-bit virtual addressing for 4k and 16k granule configs From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240214_123112_786870_9FF830C6 X-CRM114-Status: GOOD ( 16.36 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel Update Kconfig to permit 4k and 16k granule configurations to be built with 52-bit virtual addressing, now that all the prerequisites are in place. While at it, update the feature description so it matches on the appropriate feature bits depending on the page size. For simplicity, let's just keep ARM64_HAS_VA52 as the feature name. Note that LPA2 based 52-bit virtual addressing requires 52-bit physical addressing support to be enabled as well, as programming TCR.TxSZ to values below 16 is not allowed unless TCR.DS is set, which is what activates the 52-bit physical addressing support. While supporting the converse (52-bit physical addressing without 52-bit virtual addressing) would be possible in principle, let's keep things simple, by only allowing these features to be enabled at the same time. Signed-off-by: Ard Biesheuvel --- arch/arm64/Kconfig | 17 ++++++++------- arch/arm64/kernel/cpufeature.c | 22 ++++++++++++++++---- 2 files changed, 28 insertions(+), 11 deletions(-) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 9ca3316d6379..eed8fef08a10 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -368,7 +368,9 @@ config PGTABLE_LEVELS default 3 if ARM64_64K_PAGES && (ARM64_VA_BITS_48 || ARM64_VA_BITS_52) default 3 if ARM64_4K_PAGES && ARM64_VA_BITS_39 default 3 if ARM64_16K_PAGES && ARM64_VA_BITS_47 + default 4 if ARM64_16K_PAGES && (ARM64_VA_BITS_48 || ARM64_VA_BITS_52) default 4 if !ARM64_64K_PAGES && ARM64_VA_BITS_48 + default 5 if ARM64_4K_PAGES && ARM64_VA_BITS_52 config ARCH_SUPPORTS_UPROBES def_bool y @@ -396,13 +398,13 @@ config BUILTIN_RETURN_ADDRESS_STRIPS_PAC config KASAN_SHADOW_OFFSET hex depends on KASAN_GENERIC || KASAN_SW_TAGS - default 0xdfff800000000000 if (ARM64_VA_BITS_48 || ARM64_VA_BITS_52) && !KASAN_SW_TAGS - default 0xdfffc00000000000 if ARM64_VA_BITS_47 && !KASAN_SW_TAGS + default 0xdfff800000000000 if (ARM64_VA_BITS_48 || (ARM64_VA_BITS_52 && !ARM64_16K_PAGES)) && !KASAN_SW_TAGS + default 0xdfffc00000000000 if (ARM64_VA_BITS_47 || ARM64_VA_BITS_52) && ARM64_16K_PAGES && !KASAN_SW_TAGS default 0xdffffe0000000000 if ARM64_VA_BITS_42 && !KASAN_SW_TAGS default 0xdfffffc000000000 if ARM64_VA_BITS_39 && !KASAN_SW_TAGS default 0xdffffff800000000 if ARM64_VA_BITS_36 && !KASAN_SW_TAGS - default 0xefff800000000000 if (ARM64_VA_BITS_48 || ARM64_VA_BITS_52) && KASAN_SW_TAGS - default 0xefffc00000000000 if ARM64_VA_BITS_47 && KASAN_SW_TAGS + default 0xefff800000000000 if (ARM64_VA_BITS_48 || (ARM64_VA_BITS_52 && !ARM64_16K_PAGES)) && KASAN_SW_TAGS + default 0xefffc00000000000 if (ARM64_VA_BITS_47 || ARM64_VA_BITS_52) && ARM64_16K_PAGES && KASAN_SW_TAGS default 0xeffffe0000000000 if ARM64_VA_BITS_42 && KASAN_SW_TAGS default 0xefffffc000000000 if ARM64_VA_BITS_39 && KASAN_SW_TAGS default 0xeffffff800000000 if ARM64_VA_BITS_36 && KASAN_SW_TAGS @@ -1310,7 +1312,7 @@ config ARM64_VA_BITS_48 config ARM64_VA_BITS_52 bool "52-bit" - depends on ARM64_64K_PAGES && (ARM64_PAN || !ARM64_SW_TTBR0_PAN) + depends on ARM64_PAN || !ARM64_SW_TTBR0_PAN help Enable 52-bit virtual addressing for userspace when explicitly requested via a hint to mmap(). The kernel will also use 52-bit @@ -1357,10 +1359,11 @@ choice config ARM64_PA_BITS_48 bool "48-bit" + depends on ARM64_64K_PAGES || !ARM64_VA_BITS_52 config ARM64_PA_BITS_52 - bool "52-bit (ARMv8.2)" - depends on ARM64_64K_PAGES + bool "52-bit" + depends on ARM64_64K_PAGES || ARM64_VA_BITS_52 depends on ARM64_PAN || !ARM64_SW_TTBR0_PAN help Enable support for a 52-bit physical address space, introduced as diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c index 94f035f6c421..0be9296e9253 100644 --- a/arch/arm64/kernel/cpufeature.c +++ b/arch/arm64/kernel/cpufeature.c @@ -2703,15 +2703,29 @@ static const struct arm64_cpu_capabilities arm64_features[] = { }, #ifdef CONFIG_ARM64_VA_BITS_52 { - .desc = "52-bit Virtual Addressing (LVA)", .capability = ARM64_HAS_VA52, .type = ARM64_CPUCAP_BOOT_CPU_FEATURE, - .sys_reg = SYS_ID_AA64MMFR2_EL1, - .sign = FTR_UNSIGNED, + .matches = has_cpuid_feature, .field_width = 4, +#ifdef CONFIG_ARM64_64K_PAGES + .desc = "52-bit Virtual Addressing (LVA)", + .sign = FTR_SIGNED, + .sys_reg = SYS_ID_AA64MMFR2_EL1, .field_pos = ID_AA64MMFR2_EL1_VARange_SHIFT, - .matches = has_cpuid_feature, .min_field_value = ID_AA64MMFR2_EL1_VARange_52, +#else + .desc = "52-bit Virtual Addressing (LPA2)", + .sys_reg = SYS_ID_AA64MMFR0_EL1, +#ifdef CONFIG_ARM64_4K_PAGES + .sign = FTR_SIGNED, + .field_pos = ID_AA64MMFR0_EL1_TGRAN4_SHIFT, + .min_field_value = ID_AA64MMFR0_EL1_TGRAN4_52_BIT, +#else + .sign = FTR_UNSIGNED, + .field_pos = ID_AA64MMFR0_EL1_TGRAN16_SHIFT, + .min_field_value = ID_AA64MMFR0_EL1_TGRAN16_52_BIT, +#endif +#endif }, #endif {}, From patchwork Wed Feb 14 12:29:27 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13556556 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id EF234C48BED for ; Wed, 14 Feb 2024 13:46:09 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=2enxzlDlrDDJx/F606k3xT1h1yorPzHkZbsyP93R/tY=; b=0yFb8s3SIZB9RfCioGIefx8S5w l/Uw6/iLFRqFJPK09BiIzjvz5F2DYD5tdNhLUe1p6bTsl3n3J5yCMqU5ybImYUXpL99xaZc29nvjv 4Ig3vkmpSdTlIzXCBCO4KiVHm9WZr+giZVrY3jfA97ARIpIo0HwU0/yYtanUhSYapJ16fcD/xgqDN OQdlj0Li48hzyGi4S2A4qbXvwGsWqCqiawXr6YduofH2ESgIqNdwhWMbm3vZMuf453P5it8cHRGOT lRVSYxWbK9gnNiMVpX95CtSvuNpIypQdFYmyHAsevzOqMa34C31eoJRx1bNzHlLJDnsEAh/SlZCLw ya0ercVQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1raFaF-0000000D3ms-166U; Wed, 14 Feb 2024 13:45:55 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1raEQ4-0000000CpB8-02Tz for linux-arm-kernel@bombadil.infradead.org; Wed, 14 Feb 2024 12:31:20 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=ClhD0NSsYlTr7bluMBx07QxrI8tCVIrAUfifP//bQl8=; b=ImXd9t15Mvz6s+L1w3qqgsbOub YZh9d6qh6MLSXlGvwt2q+qlOOfd8n9gWHLzzoQsDrSZHawRTBi6mhI8/szFPLoXIVJ66321jBrOOx c3p58fpFKc2ssko/RXyjOebjA6fDKEkslSqDP6EuWqMMIz+ETg8BijH14qOMC7Qh3jISkh3FSbbsn H8kGC3EzTpbvDXLW3KuksWLvKhw2990v/CIiPTfX8ETofJ/wV2B/2MKTJuuyF2D1eUkSGs0t1HY05 bXocc5VZ9i5P9/CgFkztxLPsoNPkWnh/0/ljNlSWjNXprJRTi0BsDVGABgPh1DROxK/2CxwzyT1lx axG9GiGg==; Received: from mail-wm1-x349.google.com ([2a00:1450:4864:20::349]) by desiato.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1raEPz-0000000GEGT-0U8l for linux-arm-kernel@lists.infradead.org; Wed, 14 Feb 2024 12:31:18 +0000 Received: by mail-wm1-x349.google.com with SMTP id 5b1f17b1804b1-41082621642so25910045e9.1 for ; Wed, 14 Feb 2024 04:31:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1707913873; x=1708518673; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=ClhD0NSsYlTr7bluMBx07QxrI8tCVIrAUfifP//bQl8=; b=1FYhyCkmPbzuV7JzmxmGqytUT+2b6Lqx/trvFNyfQP1gQGde7Bd2wVND2RWSLAtqus cb3RrRXOzaqFotCJj+sUk5cNbyJsGGrA68nkOGKCqF6zR18Jxp5L4FuY7QMt7DjIq9XR tQ/XzZA9T6qgWKkMuSn3UsfMbNhlixQHgr7IebY9/I7otF5Hqhbx8xCbijLvo2FaVZER zNIGqhBtluimEyeyRr0rMkmTJbVbmcqXLfPAI6aPsMF7k+Un4u9m0C++YNmvLVAulHTJ G2cZrZlESLPNv+ouGrt/LXxHKb1r3BCXhaIYvk+2fK3G7D3kDHN9YZh9SNNHN2u6ZmS3 esfg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707913873; x=1708518673; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=ClhD0NSsYlTr7bluMBx07QxrI8tCVIrAUfifP//bQl8=; b=Ag0AF+xMtZAiGgdwWG8YHaX1B5GR2j314hCpwKNNKjieZ1vpjdLKT4iIkWcMhdfS50 4emQKGPDDlwVOGcdaHMFpxYdMuMzQbK2PDeSYGdnFQ7c4YqT2Oc9q0XM0FctCprtm9TT sGYJm1IoFTDiQBdYj4jJp+QOGNk1wFwdPwK23ibNa0xWq7hXVZMmXtpe3Hia+erywd2H TgOeWHbuusIm69tl18721ktcWILuWF5XJzlUtlbDlKlZTHb0E9WeZXvSs73AQbaE3jLz HuDoGHD6D0lZBy1sDgvIaihkBYpCO25se5ryYEmdvqkjt05jHyFxSj2i/UR7qVKtTxdp ErHg== X-Gm-Message-State: AOJu0Yz6qSwY0/L8fNJvKe27G37xr3rxqq8H16e8CEvQTF0npcZoTuJi 8JNVS2uNLYI8pzy/tiGUlP6ZZ6U+jYebh9HNXw6upPv6+R8S1d3uZpMNjzxr4SRrNWHS77MmMAa Dfc4ypic4cwpK4mG3yY1U7AIsUnQ58zGiFtgfG+UuTSyoQEe7BnCSytJSp0aWX3kU+DD/nrZN0R 9Q/+i3UQ/x6PV8pt4OHinxTcMaqCatYYXu3/SQvAx8 X-Google-Smtp-Source: AGHT+IHXEdG7RNcKBM/G+uiitJHWqc0uB6b3odQk/nfzrZKZOb4B0Ix8//zcNsVrEZG/bEUuuAPRoDp1 X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:adf:9d86:0:b0:33b:572e:f3e9 with SMTP id p6-20020adf9d86000000b0033b572ef3e9mr4422wre.11.1707913873035; Wed, 14 Feb 2024 04:31:13 -0800 (PST) Date: Wed, 14 Feb 2024 13:29:27 +0100 In-Reply-To: <20240214122845.2033971-45-ardb+git@google.com> Mime-Version: 1.0 References: <20240214122845.2033971-45-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=1817; i=ardb@kernel.org; h=from:subject; bh=wSCVJMsF5sIMH/ZpkCQee+AWR9d+TwJvuhMzvHxCsfU=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIfXMJq2VMpJn74f8mBlxznTf+anfzATbH+SZfgkr5jDeJ 9Uy+Wh0RykLgxgHg6yYIovA7L/vdp6eKFXrPEsWZg4rE8gQBi5OAZiIsysjw5efN4Wmci1QetG2 M4lf0WPKuzf//8avvRAZJri17tF8sf0Mf4XSFjFUfG3xvhOvU5jnebnuue0sLcZX2Y1hd97e11O dyQUA X-Mailer: git-send-email 2.43.0.687.g38aa6559b0-goog Message-ID: <20240214122845.2033971-86-ardb+git@google.com> Subject: [PATCH v8 41/43] arm64: defconfig: Enable LPA2 support From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240214_123115_359669_9869B369 X-CRM114-Status: GOOD ( 12.64 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel We typically enable support in defconfig for all architectural features for which we can detect at runtime if the hardware actually supports them. Now that we have implemented support for LPA2 based 52-bit virtual addressing in a way that should not impact 48-bit operation on non-LPA2 CPU, we can do the same, and enable 52-bit virtual addressing by default. Catalin adds: Currently the "Virtual address space size" arch/arm64/Kconfig menu entry sets different defaults for each page size. However, all are overridden by the defconfig to 48 bits. Set the new default in Kconfig and remove the defconfig line. [ardb: squash follow-up fix from Catalin] Signed-off-by: Ard Biesheuvel --- arch/arm64/Kconfig | 4 +--- arch/arm64/configs/defconfig | 1 - 2 files changed, 1 insertion(+), 4 deletions(-) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index eed8fef08a10..160856de9bbb 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -1283,9 +1283,7 @@ endchoice choice prompt "Virtual address space size" - default ARM64_VA_BITS_39 if ARM64_4K_PAGES - default ARM64_VA_BITS_47 if ARM64_16K_PAGES - default ARM64_VA_BITS_42 if ARM64_64K_PAGES + default ARM64_VA_BITS_52 help Allows choosing one of multiple possible virtual address space sizes. The level of translation table is determined by diff --git a/arch/arm64/configs/defconfig b/arch/arm64/configs/defconfig index e6cf3e5d63c3..f086b0624ec8 100644 --- a/arch/arm64/configs/defconfig +++ b/arch/arm64/configs/defconfig @@ -76,7 +76,6 @@ CONFIG_ARCH_VEXPRESS=y CONFIG_ARCH_VISCONTI=y CONFIG_ARCH_XGENE=y CONFIG_ARCH_ZYNQMP=y -CONFIG_ARM64_VA_BITS_48=y CONFIG_SCHED_MC=y CONFIG_SCHED_SMT=y CONFIG_NUMA=y From patchwork Wed Feb 14 12:29:28 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13556557 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id C3B18C48BC1 for ; Wed, 14 Feb 2024 13:46:13 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=E8O7y9DabZnbMNBDtN5KSVbARFSDbuxYxNMnwl8zgKc=; b=Cwd+94GPDeNeuGSEAB28c2Lh4Y 9bQ2CGYV3UxmlYJo2TZunvOfmdg9HmWh64ksjnXchddc64/aGw3GKoz0jYVt1HWPJ1hoKmmpNHp6A bGRl7vr1fe7B66rhryH5p6JkqIUyufnou82e7i8uoRyB+VgWw/Lh2qagESlwH2ptVA10wT2VCCv1X gDXHsHfmxjMLw6+VNMpcM1crWVS2f+cnP+wu3WtkrN8qFicncAOQK5yvt23CtfeUdyCVwfnoyNEcr 6eZHm2w3PeNOR8QZM+QoLYPNqwzKSIFjpXcKwtygVOTOZ8IB/ieD0XM2jhVv8fF++O3pnkr1a6sUY +WnybBUw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1raFaI-0000000D3q1-3kav; Wed, 14 Feb 2024 13:45:58 +0000 Received: from mail-yw1-x114a.google.com ([2607:f8b0:4864:20::114a]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1raEQ0-0000000Cp9w-3zeC for linux-arm-kernel@lists.infradead.org; Wed, 14 Feb 2024 12:31:29 +0000 Received: by mail-yw1-x114a.google.com with SMTP id 00721157ae682-6077e501cecso36096957b3.2 for ; Wed, 14 Feb 2024 04:31:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1707913875; x=1708518675; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=TB0o3w4wUdMFbdJIxgmVT97gCq8dSmpzSBeokdjbF68=; b=UsVR6UVsXXpclykt2BKf/piuVRdxAXjRW0UKelwlckY+GYEpnWl5Ai4u/EKInqCHIN X4GSqUHuc+ID3kxxvS/OuQMro5MM2FgUfjZUelDKCAiU2B37HlfE+qG90PvxL8014LqY eWLxzCtZ2FQk0DqVHCegS+xp6kmNvkm+vjIKAoS+IqqrO0zaCQJH4gH2+1NC2ItX1reE iwaSDJQTaB4EdZzzqt6kzNP7Xq6rY6yPuU5YFO9/XBtKlFQj8bLx7RvNqS6fCTZsoVpg DixcKQC04eRtatPpq0DXRzQsNhufmz3ssIk3gJKKhOLRHNvcSPB+95Lan7YZMt9k1Vti W/Fg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707913875; x=1708518675; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=TB0o3w4wUdMFbdJIxgmVT97gCq8dSmpzSBeokdjbF68=; b=raFboNmaAOGl21hp860ZJ1NnVuSt7KEJEYkA56JMmpd86uEPBshzzXpvJAymbC9gjF GKqnynY7LcGlq647GTLJHCIPZU4CV5kh/ALzWLnVgUITgF7nTFpDfWPcQ5h2y5Mswef2 V/Wh9C9ZKUQ0yoBK/xqRggtaGMHivNYMNz56Djlbfp+vRZOfuA3V2haFaYbBU74xbiIc 60680BYMMpTdSmXFGjXHxECZM3ez0tPqgLP+rqrIna7qvh4M2Z4y2R9bTWxV4MsRPpoP jTNN/XboXDvTQZwUGaV0tCAQ1iSyxUQRt4QQIU+PI0aST+QGwEAI1ARbm1ACdIAv4ZiN CYcw== X-Gm-Message-State: AOJu0YzGnQjBJfXbqF/qgL0SwVkVd+UJGk7akU0d5HamDcTEEK6R2cxz MZbdCSdGIamncb54YeClW1fKkrzdsJC7zPlL21J8Z+zhtl5Uyq3l+vEP5VPZmPL2Rfsp9OCb4Ib CS9dNnEH0LNITamgw91IbHy7SSXDZuGA+Ja6zxeJNN7uViZculxObyYSKIp3RDmrHatq5i2F9oc KBj080pC2ITbz4D92Kt+YYClejmyaBPQK9zrfiC3eT X-Google-Smtp-Source: AGHT+IF2VE0jRm3VfVnsHLjnA0z07HJPbfI+JkMBEAUN+5QvUp4ZA6LVsMsAGXxVOhuWZUfakfvLkEoY X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a0d:e60c:0:b0:607:8baf:836 with SMTP id p12-20020a0de60c000000b006078baf0836mr544417ywe.4.1707913875616; Wed, 14 Feb 2024 04:31:15 -0800 (PST) Date: Wed, 14 Feb 2024 13:29:28 +0100 In-Reply-To: <20240214122845.2033971-45-ardb+git@google.com> Mime-Version: 1.0 References: <20240214122845.2033971-45-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=1750; i=ardb@kernel.org; h=from:subject; bh=BWZDj4BWSzj9d+Ke8suXqeVvJtUPu6Dg0TXbGvgFUNk=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIfXMJu1IjuOaN433dK9V5oxbO1tlDpu0qqiaW1uPUcJMX of6tX0dpSwMYhwMsmKKLAKz/77beXqiVK3zLFmYOaxMIEMYuDgFYCJ6jxkZDnKqc07P/3GR797E 5E9Sr9876Cj/ZHK3lxN/en6pi4jxLkaG8+0q8plbc1iuWR43SV+4Ivrix6KjKdst78xUOLJs84e 7zAA= X-Mailer: git-send-email 2.43.0.687.g38aa6559b0-goog Message-ID: <20240214122845.2033971-87-ardb+git@google.com> Subject: [PATCH v8 42/43] mm: add arch hook to validate mmap() prot flags From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240214_043117_205407_72545679 X-CRM114-Status: GOOD ( 13.49 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel Add a hook to permit architectures to perform validation on the prot flags passed to mmap(), like arch_validate_prot() does for mprotect(). This will be used by arm64 to reject PROT_WRITE+PROT_EXEC mappings on configurations that run with WXN enabled. Reviewed-by: Kees Cook Signed-off-by: Ard Biesheuvel --- include/linux/mman.h | 15 +++++++++++++++ mm/mmap.c | 3 +++ 2 files changed, 18 insertions(+) diff --git a/include/linux/mman.h b/include/linux/mman.h index dc7048824be8..ec5e7f606e43 100644 --- a/include/linux/mman.h +++ b/include/linux/mman.h @@ -124,6 +124,21 @@ static inline bool arch_validate_flags(unsigned long flags) #define arch_validate_flags arch_validate_flags #endif +#ifndef arch_validate_mmap_prot +/* + * This is called from mmap(), which ignores unknown prot bits so the default + * is to accept anything. + * + * Returns true if the prot flags are valid + */ +static inline bool arch_validate_mmap_prot(unsigned long prot, + unsigned long addr) +{ + return true; +} +#define arch_validate_mmap_prot arch_validate_mmap_prot +#endif + /* * Optimisation macro. It is equivalent to: * (x & bit1) ? bit2 : 0 diff --git a/mm/mmap.c b/mm/mmap.c index d89770eaab6b..977a8c3fd9f5 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -1229,6 +1229,9 @@ unsigned long do_mmap(struct file *file, unsigned long addr, if (!(file && path_noexec(&file->f_path))) prot |= PROT_EXEC; + if (!arch_validate_mmap_prot(prot, addr)) + return -EACCES; + /* force arch specific MAP_FIXED handling in get_unmapped_area */ if (flags & MAP_FIXED_NOREPLACE) flags |= MAP_FIXED; From patchwork Wed Feb 14 12:29:29 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13556558 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 4480EC48BC3 for ; Wed, 14 Feb 2024 13:46:20 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=YU26Jpxc8yEZb2rjs4iOKnnjNDoFSGz+PnWQyyq8ImA=; b=IkOMGtVK++oduaGrqvP7wvtXI1 zEZjwRwch5R6Ia4SmiD/90EjPNz2AW+ihebtwtnfbkbc34od22d/L2ZMq+lv5hQnYEFCvxBxvU/Sz kSkQMbcT6z/PzL3cehAQQ//dLRjX9c7Nyl7QT5hOZWVS4EHJiF3xesimcqYmxpHZXDfFJ2G3WxyJb MRXGV5NH5vI7AKTPHRUfe6EIlWSeVYwbNfuwxe6raNONg81ettZVDcIZPfgwRwBLA1K9q3Evaj1o1 APQrgRzRQYHdDUFdp9zMHXMIQe2ZHiM2x56tw4xYbapdskYtQJYf0zAXzVU3qKQoG97lK9K7EPkKV apHHUuXw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1raFaN-0000000D3uH-2nM3; Wed, 14 Feb 2024 13:46:03 +0000 Received: from mail-yw1-x1149.google.com ([2607:f8b0:4864:20::1149]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1raEQ3-0000000CpAy-3FVV for linux-arm-kernel@lists.infradead.org; Wed, 14 Feb 2024 12:31:31 +0000 Received: by mail-yw1-x1149.google.com with SMTP id 00721157ae682-60799671960so28424397b3.1 for ; Wed, 14 Feb 2024 04:31:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1707913878; x=1708518678; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=vasKrRsPXeZoGbS6WhPdf8Fejy8kg/UFOebkoB6V2g8=; b=y8CBNZQ3xc747Cv9Fza9l/XgRcofUIBs0ol+n6BfU0gdbHGKcymj7KhZ+DPvq0tSSr GcJ3RhRJmNj+nxPjEsY0adtri7edEY/DO5Yq9CKAECfBpBmlbXCTNb+Lhv7lcgupKbVQ 7jatRcnU7oMJKazMM0/fJNLQPefQg43YWFPy5AOIJESpkvnZePHtgfyS0z8Bz0yZmBs8 E8Jcq89XGgyQmC6/VwGIKgOaLFvXyRJm5Xq+QrtZCXRe4uzGUc0NbK2XKUAGlbg+Tywc uES8IDDlQSI6BnQeOWUa3sbuqZMrmN0ngEU7Qtit1YHWIRSPDlwX84dX4gzfSVvmtpil 5uGg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707913878; x=1708518678; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=vasKrRsPXeZoGbS6WhPdf8Fejy8kg/UFOebkoB6V2g8=; b=Cv7iLXI2TbRSBh+5jvRUPsWwAU48wOYIDOsbFZGipMt6QSBsGHwqmjj+9iGpm8Ztg7 XGUq9t0rYchW3DMsxdwIyI9gMj5tlPZWF0LO02wLYt4XTpoT3PJrNAtmrCXfwYdGUaNW FOMj2mRoUKwPyZb7IFdI1+VhhVcIgQSKGhJeBAz5HoEGHeYc2zdIIG1E7deV56m2jaes N3pXnn4EziJYcKw3VskgjWH/wEfmi85gE+Ew0lqC8Eap/JmZV5PyWaFTM91NiIo+dKHq 2jD+AevhJ09Fycyg4z2C7no7ss/z8PJ+qEVvHqBFKIoEJjRPlXPlhNdq5hI/9kf1+O/U Rfew== X-Gm-Message-State: AOJu0YxJ1+k+PfqxxN71YsKyb9BetOnewW2dmf4LBYdqlhMwjrWWJTAW tP179c/8if7LSHuoXRW+ti5+HLtAH+wneSilFrVNtg/PYJZWfVpw/R+qRYrssLy6bh0BB+zL+30 kXLR3MBiVFdzxEeRoLt2nelEZFptj9AN/uZkF05c4lLeXc0r+9CX/3PEny6k7oKhvJVtG4HrJ0H aB3UgaYGmCdZLG9adAm/51uQAO2moZdpxdy+/BXfiR X-Google-Smtp-Source: AGHT+IEJfocHMmX8we5k0JJzmRLnOtm3/g6VHRFHIwcjjswYGapcn6LQijcf/Wxu1w02NHN/QTgZq3Rn X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a0d:d983:0:b0:604:2252:7827 with SMTP id b125-20020a0dd983000000b0060422527827mr477894ywe.10.1707913877986; Wed, 14 Feb 2024 04:31:17 -0800 (PST) Date: Wed, 14 Feb 2024 13:29:29 +0100 In-Reply-To: <20240214122845.2033971-45-ardb+git@google.com> Mime-Version: 1.0 References: <20240214122845.2033971-45-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=10010; i=ardb@kernel.org; h=from:subject; bh=5H3QgZpq3nI/iHtcrMxQqei8Mt0M9XR/N3AvIbIOQHw=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIfXMJp2X9xbw7ohdetaqebWltc+hJezpfpbeO27t5Q3sX JfksS+yo5SFQYyDQVZMkUVg9t93O09PlKp1niULM4eVCWQIAxenAEzk0X+G/4W8l7IZeXVVV2ip apSd9EvO+sK51OGQ6qQXW64K/Vz2K5vhf9DuDC1BX+Us/2/JD0VePu16t9PvZOX3Q/Htk1nKH94 M5QQA X-Mailer: git-send-email 2.43.0.687.g38aa6559b0-goog Message-ID: <20240214122845.2033971-88-ardb+git@google.com> Subject: [PATCH v8 43/43] arm64: mm: add support for WXN memory translation attribute From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240214_043120_028722_A5D96AB0 X-CRM114-Status: GOOD ( 29.40 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel The AArch64 virtual memory system supports a global WXN control, which can be enabled to make all writable mappings implicitly no-exec. This is a useful hardening feature, as it prevents mistakes in managing page table permissions from being exploited to attack the system. When enabled at EL1, the restrictions apply to both EL1 and EL0. EL1 is completely under our control, and has been cleaned up to allow WXN to be enabled from boot onwards. EL0 is not under our control, but given that widely deployed security features such as selinux or PaX already limit the ability of user space to create mappings that are writable and executable at the same time, the impact of enabling this for EL0 is expected to be limited. (For this reason, common user space libraries that have a legitimate need for manipulating executable code already carry fallbacks such as [0].) If enabled at compile time, the feature can still be disabled at boot if needed, by passing arm64.nowxn on the kernel command line. [0] https://github.com/libffi/libffi/blob/master/src/closures.c#L440 Signed-off-by: Ard Biesheuvel Reviewed-by: Kees Cook --- arch/arm64/Kconfig | 11 ++++++ arch/arm64/include/asm/cpufeature.h | 8 +++++ arch/arm64/include/asm/mman.h | 36 ++++++++++++++++++++ arch/arm64/include/asm/mmu_context.h | 30 +++++++++++++++- arch/arm64/kernel/pi/idreg-override.c | 4 ++- arch/arm64/kernel/pi/map_kernel.c | 23 +++++++++++++ arch/arm64/mm/proc.S | 6 ++++ 7 files changed, 116 insertions(+), 2 deletions(-) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 160856de9bbb..7761ffc6dbcf 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -1608,6 +1608,17 @@ config RODATA_FULL_DEFAULT_ENABLED This requires the linear region to be mapped down to pages, which may adversely affect performance in some cases. +config ARM64_WXN + bool "Enable WXN attribute so all writable mappings are non-exec" + help + Set the WXN bit in the SCTLR system register so that all writable + mappings are treated as if the PXN/UXN bit is set as well. + If this is set to Y, it can still be disabled at runtime by + passing 'arm64.nowxn' on the kernel command line. + + This should only be set if no software needs to be supported that + relies on being able to execute from writable mappings. + config ARM64_SW_TTBR0_PAN bool "Emulate Privileged Access Never using TTBR0_EL1 switching" help diff --git a/arch/arm64/include/asm/cpufeature.h b/arch/arm64/include/asm/cpufeature.h index a8f97690ce1f..ee33b7e52da7 100644 --- a/arch/arm64/include/asm/cpufeature.h +++ b/arch/arm64/include/asm/cpufeature.h @@ -18,6 +18,7 @@ #define ARM64_SW_FEATURE_OVERRIDE_NOKASLR 0 #define ARM64_SW_FEATURE_OVERRIDE_HVHE 4 #define ARM64_SW_FEATURE_OVERRIDE_RODATA_OFF 8 +#define ARM64_SW_FEATURE_OVERRIDE_NOWXN 12 #ifndef __ASSEMBLY__ @@ -962,6 +963,13 @@ static inline bool kaslr_disabled_cmdline(void) return arm64_test_sw_feature_override(ARM64_SW_FEATURE_OVERRIDE_NOKASLR); } +static inline bool arm64_wxn_enabled(void) +{ + if (!IS_ENABLED(CONFIG_ARM64_WXN)) + return false; + return !arm64_test_sw_feature_override(ARM64_SW_FEATURE_OVERRIDE_NOWXN); +} + u32 get_kvm_ipa_limit(void); void dump_cpu_features(void); diff --git a/arch/arm64/include/asm/mman.h b/arch/arm64/include/asm/mman.h index 5966ee4a6154..6d4940342ba7 100644 --- a/arch/arm64/include/asm/mman.h +++ b/arch/arm64/include/asm/mman.h @@ -35,11 +35,40 @@ static inline unsigned long arch_calc_vm_flag_bits(unsigned long flags) } #define arch_calc_vm_flag_bits(flags) arch_calc_vm_flag_bits(flags) +static inline bool arm64_check_wx_prot(unsigned long prot, + struct task_struct *tsk) +{ + /* + * When we are running with SCTLR_ELx.WXN==1, writable mappings are + * implicitly non-executable. This means we should reject such mappings + * when user space attempts to create them using mmap() or mprotect(). + */ + if (arm64_wxn_enabled() && + ((prot & (PROT_WRITE | PROT_EXEC)) == (PROT_WRITE | PROT_EXEC))) { + /* + * User space libraries such as libffi carry elaborate + * heuristics to decide whether it is worth it to even attempt + * to create writable executable mappings, as PaX or selinux + * enabled systems will outright reject it. They will usually + * fall back to something else (e.g., two separate shared + * mmap()s of a temporary file) on failure. + */ + pr_info_ratelimited( + "process %s (%d) attempted to create PROT_WRITE+PROT_EXEC mapping\n", + tsk->comm, tsk->pid); + return false; + } + return true; +} + static inline bool arch_validate_prot(unsigned long prot, unsigned long addr __always_unused) { unsigned long supported = PROT_READ | PROT_WRITE | PROT_EXEC | PROT_SEM; + if (!arm64_check_wx_prot(prot, current)) + return false; + if (system_supports_bti()) supported |= PROT_BTI; @@ -50,6 +79,13 @@ static inline bool arch_validate_prot(unsigned long prot, } #define arch_validate_prot(prot, addr) arch_validate_prot(prot, addr) +static inline bool arch_validate_mmap_prot(unsigned long prot, + unsigned long addr) +{ + return arm64_check_wx_prot(prot, current); +} +#define arch_validate_mmap_prot arch_validate_mmap_prot + static inline bool arch_validate_flags(unsigned long vm_flags) { if (!system_supports_mte()) diff --git a/arch/arm64/include/asm/mmu_context.h b/arch/arm64/include/asm/mmu_context.h index c768d16b81a4..f0fe2d09d139 100644 --- a/arch/arm64/include/asm/mmu_context.h +++ b/arch/arm64/include/asm/mmu_context.h @@ -20,13 +20,41 @@ #include #include #include -#include #include #include #include extern bool rodata_full; +static inline int arch_dup_mmap(struct mm_struct *oldmm, + struct mm_struct *mm) +{ + return 0; +} + +static inline void arch_exit_mmap(struct mm_struct *mm) +{ +} + +static inline void arch_unmap(struct mm_struct *mm, + unsigned long start, unsigned long end) +{ +} + +static inline bool arch_vma_access_permitted(struct vm_area_struct *vma, + bool write, bool execute, bool foreign) +{ + if (IS_ENABLED(CONFIG_ARM64_WXN) && execute && + (vma->vm_flags & (VM_WRITE | VM_EXEC)) == (VM_WRITE | VM_EXEC)) { + pr_warn_ratelimited( + "process %s (%d) attempted to execute from writable memory\n", + current->comm, current->pid); + /* disallow unless the nowxn override is set */ + return !arm64_wxn_enabled(); + } + return true; +} + static inline void contextidr_thread_switch(struct task_struct *next) { if (!IS_ENABLED(CONFIG_PID_IN_CONTEXTIDR)) diff --git a/arch/arm64/kernel/pi/idreg-override.c b/arch/arm64/kernel/pi/idreg-override.c index aad399796e81..bccfee34f62f 100644 --- a/arch/arm64/kernel/pi/idreg-override.c +++ b/arch/arm64/kernel/pi/idreg-override.c @@ -189,6 +189,7 @@ static const struct ftr_set_desc sw_features __prel64_initconst = { FIELD("nokaslr", ARM64_SW_FEATURE_OVERRIDE_NOKASLR, NULL), FIELD("hvhe", ARM64_SW_FEATURE_OVERRIDE_HVHE, hvhe_filter), FIELD("rodataoff", ARM64_SW_FEATURE_OVERRIDE_RODATA_OFF, NULL), + FIELD("nowxn", ARM64_SW_FEATURE_OVERRIDE_NOWXN, NULL), {} }, }; @@ -221,8 +222,9 @@ static const struct { { "arm64.nomops", "id_aa64isar2.mops=0" }, { "arm64.nomte", "id_aa64pfr1.mte=0" }, { "nokaslr", "arm64_sw.nokaslr=1" }, - { "rodata=off", "arm64_sw.rodataoff=1" }, + { "rodata=off", "arm64_sw.rodataoff=1 arm64_sw.nowxn=1" }, { "arm64.nolva", "id_aa64mmfr2.varange=0" }, + { "arm64.nowxn", "arm64_sw.nowxn=1" }, }; static int __init parse_hexdigit(const char *p, u64 *v) diff --git a/arch/arm64/kernel/pi/map_kernel.c b/arch/arm64/kernel/pi/map_kernel.c index 5fa08e13e17e..cac1e1f63c44 100644 --- a/arch/arm64/kernel/pi/map_kernel.c +++ b/arch/arm64/kernel/pi/map_kernel.c @@ -132,6 +132,25 @@ static void __init map_kernel(u64 kaslr_offset, u64 va_offset, int root_level) idmap_cpu_replace_ttbr1(swapper_pg_dir); } +static void noinline __section(".idmap.text") disable_wxn(void) +{ + u64 sctlr = read_sysreg(sctlr_el1) & ~SCTLR_ELx_WXN; + + /* + * We cannot safely clear the WXN bit while the MMU and caches are on, + * so turn the MMU off, flush the TLBs and turn it on again but with + * the WXN bit cleared this time. + */ + asm(" msr sctlr_el1, %0 ;" + " isb ;" + " tlbi vmalle1 ;" + " dsb nsh ;" + " isb ;" + " msr sctlr_el1, %1 ;" + " isb ;" + :: "r"(sctlr & ~SCTLR_ELx_M), "r"(sctlr)); +} + static void noinline __section(".idmap.text") set_ttbr0_for_lpa2(u64 ttbr) { u64 sctlr = read_sysreg(sctlr_el1); @@ -229,6 +248,10 @@ asmlinkage void __init early_map_kernel(u64 boot_status, void *fdt) if (va_bits > VA_BITS_MIN) sysreg_clear_set(tcr_el1, TCR_T1SZ_MASK, TCR_T1SZ(va_bits)); + if (IS_ENABLED(CONFIG_ARM64_WXN) && + arm64_test_sw_feature_override(ARM64_SW_FEATURE_OVERRIDE_NOWXN)) + disable_wxn(); + /* * The virtual KASLR displacement modulo 2MiB is decided by the * physical placement of the image, as otherwise, we might not be able diff --git a/arch/arm64/mm/proc.S b/arch/arm64/mm/proc.S index 9d40f3ffd8d2..bfd2ad896108 100644 --- a/arch/arm64/mm/proc.S +++ b/arch/arm64/mm/proc.S @@ -546,6 +546,12 @@ alternative_else_nop_endif * Prepare SCTLR */ mov_q x0, INIT_SCTLR_EL1_MMU_ON +#ifdef CONFIG_ARM64_WXN + ldr_l x1, arm64_sw_feature_override + FTR_OVR_VAL_OFFSET + tst x1, #0xf << ARM64_SW_FEATURE_OVERRIDE_NOWXN + orr x1, x0, #SCTLR_ELx_WXN + csel x0, x0, x1, ne +#endif ret // return to head.S .unreq mair