From patchwork Sat Feb 19 01:29:44 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Collingbourne X-Patchwork-Id: 12752124 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 5D62AC433F5 for ; Sat, 19 Feb 2022 01:31:19 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Mime-Version: Message-Id:Date:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:In-Reply-To: References:List-Owner; bh=RyTy0IxQJs83NWxwtDKhIRNq+m6jI00T6mIrUUD/DsE=; b=uwh 89ICLtYfOTGl7MZdri9qTuAIlWsjjccerOna6hbycxOU5g3HW6Jk28qdlJRngHMFQjpa5IPUKfFKS HDY2jzGyhK37sU1dkGUxu3z3QwbLlbzzxSDbsdGq6G7YQoEGJYi88MDeYusQwQXMqrum69Fu9y/08 DmJdkjJFL1SzmVqVF2KtUCaPBLc3sxAr0fqgx950uRnin6jXi+nW3fHZJeQcAYqbHr+A87gqnzx20 tBGTYMCFMgHZGRtG35dcN7hvW2iBoyzBz8mQ7ABfYZN1SZBJFXPNVYDwsnz/0uVJ40Bt09kF9Mb6k qSWp70adG5esw1QB/t2QgRVx6ugNwZA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1nLEZc-00G6e2-Fk; Sat, 19 Feb 2022 01:30:08 +0000 Received: from mail-yb1-xb4a.google.com ([2607:f8b0:4864:20::b4a]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1nLEZY-00G6cW-9l for linux-arm-kernel@lists.infradead.org; Sat, 19 Feb 2022 01:30:05 +0000 Received: by mail-yb1-xb4a.google.com with SMTP id a19-20020a25ca13000000b0061db44646b3so15483184ybg.2 for ; Fri, 18 Feb 2022 17:30:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:message-id:mime-version:subject:from:to:cc; bh=LT74mKv2CuVbvl4vyPNFOnmSsHHTWc1G9caXFZDMFU4=; b=BSQKKgJp+12HKmJcDDDpaXjmHzF8isdWVVKq/SwjpuR/clvMo/6E4BB4s/rbVWsUBd YLsqbAJe3FE/5qRoljf6nSaXzvsLteBuzE3gyJXkGFbbAlPIT2DQreFQi36bY9+FiRz1 UbuR2fPmsJg/JRue1608d6/b5nG0yoO8ImkaHLuF50tT3bZfPXHB6p9J5kgH/chx7ILb gnPHhTEtnWrKTD09O7TrXZxVp4t1/SPOjlRVlJHgQTZi8JaqEg/+jdC+J+9Khl/0FBoR 0S3cgto7Z0B07Fi+V5SSgU5LFBwTFa9KyJ893qT4AEQPkpdHw3mTxukviKmDEcp3yS8m TE0A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:message-id:mime-version:subject:from:to:cc; bh=LT74mKv2CuVbvl4vyPNFOnmSsHHTWc1G9caXFZDMFU4=; b=EUJ7KH/CEC2E0bS3TCcaqT4oRofq22jxknPOANpKLn6O4JAcG5vBnFUy5nOHrikUgm YgP4ug9JQrW1qldkjXaTihn51x07QL0ibFi5faXgGfMtwb5MdrH4Wpg2PIsaEVVaV0L8 A+lH8tJSulvSgK9YTOEKRUlMjll69wPpQmWD+7IxGp/p1Vj3DkKsbDzElY0CkysM1vg7 t8ylt2mgGSrNut4KYm2kduvrGlOPDG5Md2GSLVrlR6S34PFS4wyx5JBf9tb8x92K0nWa PlmPK7LS8Pelpfh7ib8c3hegXANb6gTmuaq5iHccTsNl/vCvrTEL+tdi8meBPi13YQ+g 4cig== X-Gm-Message-State: AOAM533mLJ9EIc5xWscF7HtaVIgm1YBaYjtN6D8f1DoxTeUzrU8QQ67a YhtwmWZYUjmZZE4n/2LvKc1LJyc= X-Google-Smtp-Source: ABdhPJxkxlmq25NnWuXThpI8Z0zOfcD4KPBeX+t1AUKrftXlgLsv/SRY27p189/PuYwk9MchPbVRz0I= X-Received: from pcc-desktop.svl.corp.google.com ([2620:15c:2ce:200:4926:4660:7cdf:2d]) (user=pcc job=sendgmr) by 2002:a25:296:0:b0:623:a267:ab94 with SMTP id 144-20020a250296000000b00623a267ab94mr10306408ybc.430.1645234202497; Fri, 18 Feb 2022 17:30:02 -0800 (PST) Date: Fri, 18 Feb 2022 17:29:44 -0800 Message-Id: <20220219012945.894950-1-pcc@google.com> Mime-Version: 1.0 X-Mailer: git-send-email 2.35.1.473.g83b2b277ed-goog Subject: [PATCH v5 1/2] kasan: split kasan_*enabled() functions into a separate header From: Peter Collingbourne To: Catalin Marinas , Vincenzo Frascino , Will Deacon , Andrey Konovalov , Mark Rutland Cc: Peter Collingbourne , Evgenii Stepanov , linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220218_173004_370031_C781464D X-CRM114-Status: GOOD ( 15.47 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org In an upcoming commit we are going to need to call kasan_hw_tags_enabled() from arch/arm64/include/asm/mte.h. This would create a circular dependency between headers if KASAN_GENERIC or KASAN_SW_TAGS is enabled: linux/kasan.h -> linux/pgtable.h -> asm/pgtable.h -> asm/mte.h -> linux/kasan.h. Break the cycle by introducing a new header linux/kasan-enabled.h with the kasan_*enabled() functions that can be included from asm/mte.h. Link: https://linux-review.googlesource.com/id/I5b0d96c6ed0026fc790899e14d42b2fac6ab568e Signed-off-by: Peter Collingbourne Reviewed-by: Andrey Konovalov --- include/linux/kasan-enabled.h | 33 +++++++++++++++++++++++++++++++++ include/linux/kasan.h | 23 +---------------------- 2 files changed, 34 insertions(+), 22 deletions(-) create mode 100644 include/linux/kasan-enabled.h diff --git a/include/linux/kasan-enabled.h b/include/linux/kasan-enabled.h new file mode 100644 index 000000000000..4b6615375022 --- /dev/null +++ b/include/linux/kasan-enabled.h @@ -0,0 +1,33 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _LINUX_KASAN_ENABLED_H +#define _LINUX_KASAN_ENABLED_H + +#ifdef CONFIG_KASAN_HW_TAGS + +DECLARE_STATIC_KEY_FALSE(kasan_flag_enabled); + +static __always_inline bool kasan_enabled(void) +{ + return static_branch_likely(&kasan_flag_enabled); +} + +static inline bool kasan_hw_tags_enabled(void) +{ + return kasan_enabled(); +} + +#else /* CONFIG_KASAN_HW_TAGS */ + +static inline bool kasan_enabled(void) +{ + return IS_ENABLED(CONFIG_KASAN); +} + +static inline bool kasan_hw_tags_enabled(void) +{ + return false; +} + +#endif /* CONFIG_KASAN_HW_TAGS */ + +#endif /* LINUX_KASAN_ENABLED_H */ diff --git a/include/linux/kasan.h b/include/linux/kasan.h index 4a45562d8893..b6a93261c92a 100644 --- a/include/linux/kasan.h +++ b/include/linux/kasan.h @@ -3,6 +3,7 @@ #define _LINUX_KASAN_H #include +#include #include #include #include @@ -83,33 +84,11 @@ static inline void kasan_disable_current(void) {} #ifdef CONFIG_KASAN_HW_TAGS -DECLARE_STATIC_KEY_FALSE(kasan_flag_enabled); - -static __always_inline bool kasan_enabled(void) -{ - return static_branch_likely(&kasan_flag_enabled); -} - -static inline bool kasan_hw_tags_enabled(void) -{ - return kasan_enabled(); -} - void kasan_alloc_pages(struct page *page, unsigned int order, gfp_t flags); void kasan_free_pages(struct page *page, unsigned int order); #else /* CONFIG_KASAN_HW_TAGS */ -static inline bool kasan_enabled(void) -{ - return IS_ENABLED(CONFIG_KASAN); -} - -static inline bool kasan_hw_tags_enabled(void) -{ - return false; -} - static __always_inline void kasan_alloc_pages(struct page *page, unsigned int order, gfp_t flags) { From patchwork Sat Feb 19 01:29:45 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Collingbourne X-Patchwork-Id: 12752125 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 5458DC433EF for ; Sat, 19 Feb 2022 01:31:26 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:References: Mime-Version:Message-Id:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=A4aE7DFGE6jMmHMJkWOKib/MUkJeY+RJr0fGP7QK0ZM=; b=1+X4uUNQe2bpuGUGG78l6k40aC gqmnAbz4DjHozZUbko/S8Of4uCGmylpa49sFO9gEHdkOxbg0xDLgtfeB7ZF5igcEFpRxIn9XFRpd3 YwGrLTltHpnFwm/C1XtKNny4AAGVuJ1gNOvZpNdqm7erfOHZ7tgnNfxbs5WsY3jFnXrrMN41hmbma Nvq3zZX9PvZM1kEdY9hP5hHaCMwn+z3RXMxhl6CsPSilw+GLBevVi9fzL9pkPJd0+9yPFYi1/AQyV MmDRb2+dKK5KZQSXouOPPJnYm+3QMnvaWguHGO10NOJ6Ju2JiX2IsUeoPhwOmz7lkF9xvcRDEbGrh H0qFJnjg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1nLEZn-00G6fp-49; Sat, 19 Feb 2022 01:30:19 +0000 Received: from mail-yw1-x114a.google.com ([2607:f8b0:4864:20::114a]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1nLEZa-00G6dE-Nb for linux-arm-kernel@lists.infradead.org; Sat, 19 Feb 2022 01:30:11 +0000 Received: by mail-yw1-x114a.google.com with SMTP id 00721157ae682-2d07ae1145aso60165377b3.4 for ; Fri, 18 Feb 2022 17:30:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=HAlYyn9KnmbmoD8rPWV1wNqJvLzz5Tpsb0HkUX4HIHw=; b=tb5C0jiJAUc+t0dRPvAiENwb0uKF7ukIlY7RFbbn/6odLTMsWDFKSufoHsC999Rade W78aZyoguLRulqKo74pOISX9hB3N19AHOi2nRoVkqaY/CCuSkal/cr1q981EJna+PUjL Da+lJWVkUQS/H0rYRq4CSE5PoEoxADt1LDLVqG3ik9DGc3ydUFLGGn/KLB7R6rIrd22c +Xiw403EeX2qPxcUJ2xm7XcKGybdMsZJD85wV2hST0NT4Cjhiu67ZNwDC9SgmvdBKTY7 gcGM2y76UC3F3HwWOdkyafrW0vyh8fNtG2icBf7cQMequz8kXq2gCumBbQkqF3QiX5cy W1tw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=HAlYyn9KnmbmoD8rPWV1wNqJvLzz5Tpsb0HkUX4HIHw=; b=PO3nYgW2vFDXq8oW9DeYF835vnBL+8v+lpNczq/zeZU75FMrqkp4kZw0RwZboQdqbQ sK8NOyTAu1FN2iVsBAevhbqXMlZ4rdM/sFj3gElrsk5ISTb7VuOxSmWbHbSlVOxnRu4/ sRu3ligA9e09frXaQrNdKMJkcdaZMHkxs3jeOgwgC8UJHxhZ7LhyD8l1VN+/Y8tDH2k/ lnK37GS/Ruda3JgvD8a51mPQzT5lGcWtO9LAAFeeOHVxDwFOFh/Hibv6EfVEcW+ykGhY dZ7UhJrpgH67XGokavZYVzWiGqVEkvx3Q7AkvoGYOqyvsnG3ZwQMZyv//Jm9qmlrTqUp 5p1w== X-Gm-Message-State: AOAM531wAqScQ/PaNHgU588sSmKCjAits4ET+glSz6l/S+M4ExqaSYHC KX1VOJ65AOCE6BuU3Xu7RS0Ymn4= X-Google-Smtp-Source: ABdhPJyXqqPv1gHzxgxRtD2X8PQLKtXIr+c5HMVFUnLMzRhNSPgnVpsajrw568Zv0/+/eyjZOTU4ppI= X-Received: from pcc-desktop.svl.corp.google.com ([2620:15c:2ce:200:4926:4660:7cdf:2d]) (user=pcc job=sendgmr) by 2002:a25:e90b:0:b0:61d:9552:6f1a with SMTP id n11-20020a25e90b000000b0061d95526f1amr9592310ybd.400.1645234204963; Fri, 18 Feb 2022 17:30:04 -0800 (PST) Date: Fri, 18 Feb 2022 17:29:45 -0800 In-Reply-To: <20220219012945.894950-1-pcc@google.com> Message-Id: <20220219012945.894950-2-pcc@google.com> Mime-Version: 1.0 References: <20220219012945.894950-1-pcc@google.com> X-Mailer: git-send-email 2.35.1.473.g83b2b277ed-goog Subject: [PATCH v5 2/2] arm64: mte: avoid clearing PSTATE.TCO on entry unless necessary From: Peter Collingbourne To: Catalin Marinas , Vincenzo Frascino , Will Deacon , Andrey Konovalov , Mark Rutland Cc: Peter Collingbourne , Evgenii Stepanov , linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220218_173006_801218_3491B236 X-CRM114-Status: GOOD ( 20.77 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On some microarchitectures, clearing PSTATE.TCO is expensive. Clearing TCO is only necessary if in-kernel MTE is enabled, or if MTE is enabled in the userspace process in synchronous (or, soon, asymmetric) mode, because we do not report uaccess faults to userspace in none or asynchronous modes. Therefore, adjust the kernel entry code to clear TCO only if necessary. Because it is now possible to switch to a task in which TCO needs to be clear from a task in which TCO is set, we also need to do the same thing on task switch. Signed-off-by: Peter Collingbourne Link: https://linux-review.googlesource.com/id/I52d82a580bd0500d420be501af2c35fa8c90729e Reviewed-by: Catalin Marinas --- v5: - fix header circular dependency with KASAN_GENERIC/KASAN_SW_TAGS enabled with a dependent patch v4: - some changes suggested by Catalin v3: - switch to a C implementation v2: - do the same thing in cpu_switch_to() arch/arm64/include/asm/mte.h | 22 ++++++++++++++++++++++ arch/arm64/kernel/entry-common.c | 3 +++ arch/arm64/kernel/entry.S | 7 ------- arch/arm64/kernel/mte.c | 3 +++ 4 files changed, 28 insertions(+), 7 deletions(-) diff --git a/arch/arm64/include/asm/mte.h b/arch/arm64/include/asm/mte.h index 075539f5f1c8..adcb937342f1 100644 --- a/arch/arm64/include/asm/mte.h +++ b/arch/arm64/include/asm/mte.h @@ -11,7 +11,9 @@ #ifndef __ASSEMBLY__ #include +#include #include +#include #include #include @@ -86,6 +88,26 @@ static inline int mte_ptrace_copy_tags(struct task_struct *child, #endif /* CONFIG_ARM64_MTE */ +static inline void mte_disable_tco_entry(struct task_struct *task) +{ + if (!system_supports_mte()) + return; + + /* + * Re-enable tag checking (TCO set on exception entry). This is only + * necessary if MTE is enabled in either the kernel or the userspace + * task in synchronous or asymmetric mode (SCTLR_EL1.TCF0 bit 0 is set + * for both). With MTE disabled in the kernel and disabled or + * asynchronous in userspace, tag check faults (including in uaccesses) + * are not reported, therefore there is no need to re-enable checking. + * This is beneficial on microarchitectures where re-enabling TCO is + * expensive. + */ + if (kasan_hw_tags_enabled() || + (task->thread.sctlr_user & (1UL << SCTLR_EL1_TCF0_SHIFT))) + asm volatile(SET_PSTATE_TCO(0)); +} + #ifdef CONFIG_KASAN_HW_TAGS /* Whether the MTE asynchronous mode is enabled. */ DECLARE_STATIC_KEY_FALSE(mte_async_or_asymm_mode); diff --git a/arch/arm64/kernel/entry-common.c b/arch/arm64/kernel/entry-common.c index ef7fcefb96bd..7093b578e325 100644 --- a/arch/arm64/kernel/entry-common.c +++ b/arch/arm64/kernel/entry-common.c @@ -6,6 +6,7 @@ */ #include +#include #include #include #include @@ -56,6 +57,7 @@ static void noinstr enter_from_kernel_mode(struct pt_regs *regs) { __enter_from_kernel_mode(regs); mte_check_tfsr_entry(); + mte_disable_tco_entry(current); } /* @@ -103,6 +105,7 @@ static __always_inline void __enter_from_user_mode(void) CT_WARN_ON(ct_state() != CONTEXT_USER); user_exit_irqoff(); trace_hardirqs_off_finish(); + mte_disable_tco_entry(current); } static __always_inline void enter_from_user_mode(struct pt_regs *regs) diff --git a/arch/arm64/kernel/entry.S b/arch/arm64/kernel/entry.S index 772ec2ecf488..e1013a83d4f0 100644 --- a/arch/arm64/kernel/entry.S +++ b/arch/arm64/kernel/entry.S @@ -308,13 +308,6 @@ alternative_if ARM64_HAS_IRQ_PRIO_MASKING msr_s SYS_ICC_PMR_EL1, x20 alternative_else_nop_endif - /* Re-enable tag checking (TCO set on exception entry) */ -#ifdef CONFIG_ARM64_MTE -alternative_if ARM64_MTE - SET_PSTATE_TCO(0) -alternative_else_nop_endif -#endif - /* * Registers that may be useful after this macro is invoked: * diff --git a/arch/arm64/kernel/mte.c b/arch/arm64/kernel/mte.c index f418ebc65f95..f983795b5eda 100644 --- a/arch/arm64/kernel/mte.c +++ b/arch/arm64/kernel/mte.c @@ -253,6 +253,9 @@ void mte_thread_switch(struct task_struct *next) mte_update_sctlr_user(next); mte_update_gcr_excl(next); + /* TCO may not have been disabled on exception entry for the current task. */ + mte_disable_tco_entry(next); + /* * Check if an async tag exception occurred at EL1. *