From patchwork Tue Feb 20 23:04:03 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Linus Walleij X-Patchwork-Id: 13564721 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id C8435C48BC3 for ; Tue, 20 Feb 2024 23:04:42 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:In-Reply-To:References:Message-Id :MIME-Version:Subject:Date:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=5yl5k3CLdX/OphLuqMDysgxOwp9fee/kj8BWlT1Yo8g=; b=zuKQz4TJDlHz09 Oe0PVgQ96xlvaUevWK9sPjCx1iHG8LxuDERGXhoyxGcfjiJEEkmyL1tl80y8bXfPXoEgjBbE+4AnR kl1fAqMzu4DvzT5TxxP7brYBN4z/haCq7H7+CktRMPNjiEmNd/eaMyEvSLl0UuPDp6ascT4iwAnQ2 9JgKbaEhb/UXJMQJ/MAjMCnhy0XfzqewipDSj25dGUuIzpxdRg7ZMH1ySMNHCzTWyPs594ah1mm4l bTBlkP93NrW6CTduvWv7IqNE9b63qJASNdkKU5rNqL09s4rumCajQFx3qC7hVaKypA6op83dvDBfo 6D93wMOawZXuHpiUtHcA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1rcZA9-0000000GRfz-0VfX; Tue, 20 Feb 2024 23:04:33 +0000 Received: from mail-ej1-x62e.google.com ([2a00:1450:4864:20::62e]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rcZ9o-0000000GRSb-1JHD for linux-arm-kernel@lists.infradead.org; Tue, 20 Feb 2024 23:04:14 +0000 Received: by mail-ej1-x62e.google.com with SMTP id a640c23a62f3a-a26ed1e05c7so820572466b.2 for ; Tue, 20 Feb 2024 15:04:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1708470251; x=1709075051; darn=lists.infradead.org; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:subject:date:from:from:to:cc:subject:date:message-id :reply-to; bh=mD9euVRrZMNhwm70wL8mzCGogUqWgq6QZLjSy1aQp90=; b=pA0qdasME1vLChd5kDXlX8s+NmPToP/oe2qh6Cf7ITfIhhCS6Z0dSdjcU/avK+y+OY oq4KNXL9De/MB/LaoRcIJ7guMAbRA+R1aNNgarcNB6j6loF3/Y4l4xnH+T6CQD8DzMVD b3YqK/aPRJO7G1pgSn8JSAeHFpF2zNSZr9uFrJuNtH7hXYBimxGkBVkE+c9mEdiwsZdl kPyHsll88x4WSsXPu67QXZXYMEpQL8cFkNPcys3ssY8TwOybuuMJs2ymmnhfKNRwXaTp 6VQfKaERNbLaNO8HOXoad9Rnnk55xGzmyKpYMzLC7C+rUtOSCVfm8KaCXtkfTuvPGdLm Phmg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708470251; x=1709075051; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:subject:date:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=mD9euVRrZMNhwm70wL8mzCGogUqWgq6QZLjSy1aQp90=; b=V0cwKpVWNNyhUN86UMHqsppjEBY52c5aVniXBJDV4MAz+OVtHC2w4MLnRcEejOd7b5 y5gJLXgZ/P7UVRfzSd+xk0GQdm8AbT+EoeSUG3hGDHkOVkq11JlOhyPYesBZ/a9wFF6p 64CnZjkrPslctuNUWmpZfAdkxaL4JZ95JwwKAQPoaPh5vW0sW09HaRFppnaX5PAAC1y/ RgK9vI0DZAuRWZn6d5wOBlS6h+26bZXrJyyFI6SAAyASApzLTjQKmOwNG/bQitrMfbTb gu9s6+Mgl3TzPvTYVc0F8+dE4g90HKxIvTGqXHO0apnRd1GK06D+LQnu+Jz+qH6Y8jPt N7VQ== X-Gm-Message-State: AOJu0YzWTu0qzeVCeq31k9jXRDloGiyopvQE94fAYtVpzxvy7+P+z/Uf iH16tcFK/7yH9+148BN5DMPV6lF4C7YXegd4xud7jZ0peXliHDywFtbXKXwIedM= X-Google-Smtp-Source: AGHT+IGm/6+7Kdx7+NvAQ7Bmz2IY20QvFR0JwaC8b/Agx2gkSNjxN2xsj5HBauRsy0+aOiAVYuIuIw== X-Received: by 2002:a17:906:1854:b0:a3e:73c8:cd27 with SMTP id w20-20020a170906185400b00a3e73c8cd27mr4927608eje.73.1708470250839; Tue, 20 Feb 2024 15:04:10 -0800 (PST) Received: from [127.0.1.1] ([85.235.12.238]) by smtp.gmail.com with ESMTPSA id y14-20020a170906448e00b00a379ef08ecbsm4377021ejo.74.2024.02.20.15.04.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 20 Feb 2024 15:04:10 -0800 (PST) From: Linus Walleij Date: Wed, 21 Feb 2024 00:04:03 +0100 Subject: [PATCH v2 4/4] ARM: Implement privileged no-access using TTBR0 page table walks disabling MIME-Version: 1.0 Message-Id: <20240221-arm32-lpae-pan-v2-4-991096bba5d8@linaro.org> References: <20240221-arm32-lpae-pan-v2-0-991096bba5d8@linaro.org> In-Reply-To: <20240221-arm32-lpae-pan-v2-0-991096bba5d8@linaro.org> To: Russell King , Ard Biesheuvel , Arnd Bergmann , Stefan Wahren , Kees Cook , Geert Uytterhoeven Cc: linux-arm-kernel@lists.infradead.org, Linus Walleij , Catalin Marinas X-Mailer: b4 0.12.4 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240220_150412_576486_AFE42546 X-CRM114-Status: GOOD ( 30.92 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Catalin Marinas With LPAE enabled, privileged no-access cannot be enforced using CPU domains as such feature is not available. This patch implements PAN by disabling TTBR0 page table walks while in kernel mode. The ARM architecture allows page table walks to be split between TTBR0 and TTBR1. With LPAE enabled, the split is defined by a combination of TTBCR T0SZ and T1SZ bits. Currently, an LPAE-enabled kernel uses TTBR0 for user addresses and TTBR1 for kernel addresses with the VMSPLIT_2G and VMSPLIT_3G configurations. The main advantage for the 3:1 split is that TTBR1 is reduced to 2 levels, so potentially faster TLB refill (though usually the first level entries are already cached in the TLB). The PAN support on LPAE-enabled kernels uses TTBR0 when running in user space or in kernel space during user access routines (TTBCR T0SZ and T1SZ are both 0). When running user accesses are disabled in kernel mode, TTBR0 page table walks are disabled by setting TTBCR.EPD0. TTBR1 is used for kernel accesses (including loadable modules; anything covered by swapper_pg_dir) by reducing the TTBCR.T0SZ to the minimum (2^(32-7) = 32MB). To avoid user accesses potentially hitting stale TLB entries, the ASID is switched to 0 (reserved) by setting TTBCR.A1 and using the ASID value in TTBR1. The difference from a non-PAN kernel is that with the 3:1 memory split, TTBR1 always uses 3 levels of page tables. Signed-off-by: Catalin Marinas Reviewed-by: Kees Cook Signed-off-by: Linus Walleij --- ChangeLog v1->v2: - Make the SVC mode TTBCR a separate field in struct svc_pt_regs as requested by Russell. - Push the MM page fault permission check into a local function and avoid the too generic uaccess_disabled() as requested by Ard. --- arch/arm/Kconfig | 22 ++++++++++++--- arch/arm/include/asm/assembler.h | 1 + arch/arm/include/asm/pgtable-3level-hwdef.h | 9 +++++++ arch/arm/include/asm/ptrace.h | 1 + arch/arm/include/asm/uaccess-asm.h | 42 +++++++++++++++++++++++++++++ arch/arm/include/asm/uaccess.h | 30 +++++++++++++++++++++ arch/arm/kernel/asm-offsets.c | 1 + arch/arm/kernel/suspend.c | 8 ++++++ arch/arm/lib/csumpartialcopyuser.S | 14 ++++++++++ arch/arm/mm/fault.c | 29 ++++++++++++++++++++ 10 files changed, 154 insertions(+), 3 deletions(-) diff --git a/arch/arm/Kconfig b/arch/arm/Kconfig index 0af6709570d1..3d97a15a3e2d 100644 --- a/arch/arm/Kconfig +++ b/arch/arm/Kconfig @@ -1231,9 +1231,9 @@ config HIGHPTE consumed by page tables. Setting this option will allow user-space 2nd level page tables to reside in high memory. -config CPU_SW_DOMAIN_PAN - bool "Enable use of CPU domains to implement privileged no-access" - depends on MMU && !ARM_LPAE +config ARM_PAN + bool "Enable privileged no-access" + depends on MMU default y help Increase kernel security by ensuring that normal kernel accesses @@ -1242,10 +1242,26 @@ config CPU_SW_DOMAIN_PAN by ensuring that magic values (such as LIST_POISON) will always fault when dereferenced. + The implementation uses CPU domains when !CONFIG_ARM_LPAE and + disabling of TTBR0 page table walks with CONFIG_ARM_LPAE. + +config CPU_SW_DOMAIN_PAN + def_bool y + depends on ARM_PAN && !ARM_LPAE + help + Enable use of CPU domains to implement privileged no-access. + CPUs with low-vector mappings use a best-efforts implementation. Their lower 1MB needs to remain accessible for the vectors, but the remainder of userspace will become appropriately inaccessible. +config CPU_TTBR0_PAN + def_bool y + depends on ARM_PAN && ARM_LPAE + help + Enable privileged no-access by disabling TTBR0 page table walks when + running in kernel mode. + config HW_PERF_EVENTS def_bool y depends on ARM_PMU diff --git a/arch/arm/include/asm/assembler.h b/arch/arm/include/asm/assembler.h index aebe2c8f6a68..d33c1e24e00b 100644 --- a/arch/arm/include/asm/assembler.h +++ b/arch/arm/include/asm/assembler.h @@ -21,6 +21,7 @@ #include #include #include +#include #include #include diff --git a/arch/arm/include/asm/pgtable-3level-hwdef.h b/arch/arm/include/asm/pgtable-3level-hwdef.h index 19da7753a0b8..323ad811732e 100644 --- a/arch/arm/include/asm/pgtable-3level-hwdef.h +++ b/arch/arm/include/asm/pgtable-3level-hwdef.h @@ -74,6 +74,7 @@ #define PHYS_MASK_SHIFT (40) #define PHYS_MASK ((1ULL << PHYS_MASK_SHIFT) - 1) +#ifndef CONFIG_CPU_TTBR0_PAN /* * TTBR0/TTBR1 split (PAGE_OFFSET): * 0x40000000: T0SZ = 2, T1SZ = 0 (not used) @@ -93,6 +94,14 @@ #endif #define TTBR1_SIZE (((PAGE_OFFSET >> 30) - 1) << 16) +#else +/* + * With CONFIG_CPU_TTBR0_PAN enabled, TTBR1 is only used during uaccess + * disabled regions when TTBR0 is disabled. + */ +#define TTBR1_OFFSET 0 /* pointing to swapper_pg_dir */ +#define TTBR1_SIZE 0 /* TTBR1 size controlled via TTBCR.T0SZ */ +#endif /* * TTBCR register bits. diff --git a/arch/arm/include/asm/ptrace.h b/arch/arm/include/asm/ptrace.h index 7f44e88d1f25..f064252498c7 100644 --- a/arch/arm/include/asm/ptrace.h +++ b/arch/arm/include/asm/ptrace.h @@ -19,6 +19,7 @@ struct pt_regs { struct svc_pt_regs { struct pt_regs regs; u32 dacr; + u32 ttbcr; }; #define to_svc_pt_regs(r) container_of(r, struct svc_pt_regs, regs) diff --git a/arch/arm/include/asm/uaccess-asm.h b/arch/arm/include/asm/uaccess-asm.h index ea42ba25920f..7c06040af078 100644 --- a/arch/arm/include/asm/uaccess-asm.h +++ b/arch/arm/include/asm/uaccess-asm.h @@ -65,6 +65,37 @@ .endif .endm +#elif defined(CONFIG_CPU_TTBR0_PAN) + + .macro uaccess_disable, tmp, isb=1 + /* + * Disable TTBR0 page table walks (EDP0 = 1), use the reserved ASID + * from TTBR1 (A1 = 1) and enable TTBR1 page table walks for kernel + * addresses by reducing TTBR0 range to 32MB (T0SZ = 7). + */ + mrc p15, 0, \tmp, c2, c0, 2 @ read TTBCR + orr \tmp, \tmp, #TTBCR_EPD0 | TTBCR_T0SZ_MASK + orr \tmp, \tmp, #TTBCR_A1 + mcr p15, 0, \tmp, c2, c0, 2 @ write TTBCR + .if \isb + instr_sync + .endif + .endm + + .macro uaccess_enable, tmp, isb=1 + /* + * Enable TTBR0 page table walks (T0SZ = 0, EDP0 = 0) and ASID from + * TTBR0 (A1 = 0). + */ + mrc p15, 0, \tmp, c2, c0, 2 @ read TTBCR + bic \tmp, \tmp, #TTBCR_EPD0 | TTBCR_T0SZ_MASK + bic \tmp, \tmp, #TTBCR_A1 + mcr p15, 0, \tmp, c2, c0, 2 @ write TTBCR + .if \isb + instr_sync + .endif + .endm + #else .macro uaccess_disable, tmp, isb=1 @@ -79,6 +110,12 @@ #define DACR(x...) x #else #define DACR(x...) +#endif + +#ifdef CONFIG_CPU_TTBR0_PAN +#define PAN(x...) x +#else +#define PAN(x...) #endif /* @@ -94,6 +131,8 @@ .macro uaccess_entry, tsk, tmp0, tmp1, tmp2, disable DACR( mrc p15, 0, \tmp0, c3, c0, 0) DACR( str \tmp0, [sp, #SVC_DACR]) + PAN( mrc p15, 0, \tmp0, c2, c0, 2) + PAN( str \tmp0, [sp, #SVC_TTBCR]) .if \disable && IS_ENABLED(CONFIG_CPU_SW_DOMAIN_PAN) /* kernel=client, user=no access */ mov \tmp2, #DACR_UACCESS_DISABLE @@ -112,8 +151,11 @@ .macro uaccess_exit, tsk, tmp0, tmp1 DACR( ldr \tmp0, [sp, #SVC_DACR]) DACR( mcr p15, 0, \tmp0, c3, c0, 0) + PAN( ldr \tmp0, [sp, #SVC_TTBCR]) + PAN( mcr p15, 0, \tmp0, c2, c0, 2) .endm #undef DACR +#undef PAN #endif /* __ASM_UACCESS_ASM_H__ */ diff --git a/arch/arm/include/asm/uaccess.h b/arch/arm/include/asm/uaccess.h index 9b9234d1bb6a..a4e86c5e03a2 100644 --- a/arch/arm/include/asm/uaccess.h +++ b/arch/arm/include/asm/uaccess.h @@ -14,6 +14,8 @@ #include #include #include +#include +#include #include #include @@ -43,6 +45,28 @@ static __always_inline void uaccess_restore(unsigned int flags) set_domain(flags); } +#elif defined(CONFIG_CPU_TTBR0_PAN) + +static inline unsigned int uaccess_save_and_enable(void) +{ + unsigned int old_ttbcr = cpu_get_ttbcr(); + + /* + * Enable TTBR0 page table walks (T0SZ = 0, EDP0 = 0) and ASID from + * TTBR0 (A1 = 0). + */ + cpu_set_ttbcr(old_ttbcr & ~(TTBCR_A1 | TTBCR_EPD0 | TTBCR_T0SZ_MASK)); + isb(); + + return old_ttbcr; +} + +static inline void uaccess_restore(unsigned int flags) +{ + cpu_set_ttbcr(flags); + isb(); +} + #else static inline unsigned int uaccess_save_and_enable(void) @@ -54,6 +78,12 @@ static inline void uaccess_restore(unsigned int flags) { } +static inline bool uaccess_disabled(struct pt_regs *regs) +{ + /* Without PAN userspace is always available */ + return false; +} + #endif /* diff --git a/arch/arm/kernel/asm-offsets.c b/arch/arm/kernel/asm-offsets.c index 219cbc7e5d13..dd2567ba987f 100644 --- a/arch/arm/kernel/asm-offsets.c +++ b/arch/arm/kernel/asm-offsets.c @@ -83,6 +83,7 @@ int main(void) DEFINE(S_OLD_R0, offsetof(struct pt_regs, ARM_ORIG_r0)); DEFINE(PT_REGS_SIZE, sizeof(struct pt_regs)); DEFINE(SVC_DACR, offsetof(struct svc_pt_regs, dacr)); + DEFINE(SVC_TTBCR, offsetof(struct svc_pt_regs, ttbcr)); DEFINE(SVC_REGS_SIZE, sizeof(struct svc_pt_regs)); BLANK(); DEFINE(SIGFRAME_RC3_OFFSET, offsetof(struct sigframe, retcode[3])); diff --git a/arch/arm/kernel/suspend.c b/arch/arm/kernel/suspend.c index c3ec3861dd07..58a6441b58c4 100644 --- a/arch/arm/kernel/suspend.c +++ b/arch/arm/kernel/suspend.c @@ -12,6 +12,7 @@ #include #include #include +#include extern int __cpu_suspend(unsigned long, int (*)(unsigned long), u32 cpuid); extern void cpu_resume_mmu(void); @@ -26,6 +27,13 @@ int cpu_suspend(unsigned long arg, int (*fn)(unsigned long)) if (!idmap_pgd) return -EINVAL; + /* + * Needed for the MMU disabling/enabing code to be able to run from + * TTBR0 addresses. + */ + if (IS_ENABLED(CONFIG_CPU_TTBR0_PAN)) + uaccess_save_and_enable(); + /* * Function graph tracer state gets incosistent when the kernel * calls functions that never return (aka suspend finishers) hence diff --git a/arch/arm/lib/csumpartialcopyuser.S b/arch/arm/lib/csumpartialcopyuser.S index 04d8d9d741c7..c289bde04743 100644 --- a/arch/arm/lib/csumpartialcopyuser.S +++ b/arch/arm/lib/csumpartialcopyuser.S @@ -27,6 +27,20 @@ ret lr .endm +#elif defined(CONFIG_CPU_TTBR0_PAN) + + .macro save_regs + mrc p15, 0, ip, c2, c0, 2 @ read TTBCR + stmfd sp!, {r1, r2, r4 - r8, ip, lr} + uaccess_enable ip + .endm + + .macro load_regs + ldmfd sp!, {r1, r2, r4 - r8, ip, lr} + mcr p15, 0, ip, c2, c0, 2 @ restore TTBCR + ret lr + .endm + #else .macro save_regs diff --git a/arch/arm/mm/fault.c b/arch/arm/mm/fault.c index e96fb40b9cc3..7d262a819ad1 100644 --- a/arch/arm/mm/fault.c +++ b/arch/arm/mm/fault.c @@ -235,6 +235,27 @@ static inline bool is_permission_fault(unsigned int fsr) return false; } +#ifdef CONFIG_CPU_TTBR0_PAN +static inline bool ttbr0_usermode_access_allowed(struct pt_regs *regs) +{ + struct svc_pt_regs *svcregs; + + /* If we are in user mode: permission granted */ + if (user_mode(regs)) + return true; + + /* uaccess state saved above pt_regs on SVC exception entry */ + svcregs = to_svc_pt_regs(regs); + + return !(svcregs->ttbcr & TTBCR_EPD0); +} +#else +static inline bool ttbr0_usermode_access_allowed(struct pt_regs *regs) +{ + return true; +} +#endif + static int __kprobes do_page_fault(unsigned long addr, unsigned int fsr, struct pt_regs *regs) { @@ -278,6 +299,14 @@ do_page_fault(unsigned long addr, unsigned int fsr, struct pt_regs *regs) perf_sw_event(PERF_COUNT_SW_PAGE_FAULTS, 1, regs, addr); + /* + * Privileged access aborts with CONFIG_CPU_TTBR0_PAN enabled are + * routed via the translation fault mechanism. Check whether uaccess + * is disabled while in kernel mode. + */ + if (!ttbr0_usermode_access_allowed(regs)) + goto no_context; + if (!(flags & FAULT_FLAG_USER)) goto lock_mmap;