From patchwork Tue Jan 25 00:58:50 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Collingbourne X-Patchwork-Id: 12723086 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 0A4F9C433EF for ; Tue, 25 Jan 2022 01:00:15 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Mime-Version: Message-Id:Date:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:In-Reply-To: References:List-Owner; bh=ZQHjAw7Nr5kVrqvenVnUyvufD1C6qZtVxyFM75cG3Ic=; b=kgZ q/lMhQbH3yKtLQcciLDHPR0c5Dec8SJ4javZW5tQLSaOJGZEQEJWmWO9oH909wxA3SMSog5RUxZO6 GZWwGJwhpjeekfL0yIWUCfdz19IDOF2vByQfkUHwFhT0FHCnaQ6daHGieMSJix3N1RZyfjJvQh5MK qBFpigxb/puRnJ0GxSPOLSPREByX6S1bZxJhwZvets0NACKF7Dyr+HSWfpqjVTD4TXJuAmpoSKnk1 6pjmSNrWfm824JvIuPVTd2SzJEjbIJMeWlfKyMLxqfCsMxK/BT5PXxkFQXkhH6X57MbTAerTdrHe3 Xw+nO7iW5nGbsM98uuDfFVA0es9pvKg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1nCAAo-005uaz-In; Tue, 25 Jan 2022 00:59:02 +0000 Received: from mail-yb1-xb4a.google.com ([2607:f8b0:4864:20::b4a]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1nCAAk-005uaG-Do for linux-arm-kernel@lists.infradead.org; Tue, 25 Jan 2022 00:59:00 +0000 Received: by mail-yb1-xb4a.google.com with SMTP id g7-20020a25bdc7000000b00611c616bc76so38401425ybk.5 for ; Mon, 24 Jan 2022 16:58:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:message-id:mime-version:subject:from:to:cc; bh=CR4El33Q6EZdRbiNzsS8Egb0YYxZa3na4CfFf2tHvV4=; b=BSK8Yqs7RTEeCuJ+5lo+r1S6nUyU5/lhv168vG4VEU40CrqK3NuBfsp9ieBGyJILZ1 dxK/JkV2VjKNFpaKrPpc1QGEzBE+VHAZoqJkdXOzjG2bELTYeSi8YSx//+DFOQDr7lwW ds604QWPTW0NlJbfX5M6UV+Hplzq40QYtIdpTgKGIfv7JY/1Aa0X7DKDm+91JYHgNxEm 5xkAPu/CniJsOGYLKIoFi0onZVBH8Vg9hAUcyyHcYAg6XO+Rm00XebU37p8qqN4KVlI1 cNhYdfjExFK11mgqSMV7ylkJ0aHu0Eat4UD5J80iAXPVm3Pr/aKrHhLCIuzjsmbH/S5r 4A0w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:message-id:mime-version:subject:from:to:cc; bh=CR4El33Q6EZdRbiNzsS8Egb0YYxZa3na4CfFf2tHvV4=; b=z60FiMvkUqCI3ui9UcivKPi4S4+czw2dpBq9JoJKlCslwCTOMKBl3nRkN0mjZlrJsE XQL5BJf3T/tCzGzOvs5X0sk1hDfeksi8ODhyp/6xYUUznCBb/ZO90MOfd56meo6pMrpo iQ35AQhgVPQPQmLXAGnvXIiEVQ8yMrWw3q99IN5Jro5vuMc6t4dUv4f5kKRGf/+SxXkG SaPfF7B6mszhqgmWruLFWGAi09fzsUDjk8BBjAxBi2A+lHjyzxi+DKu0nEJCdocJhkSA nGxYs6yTfwt18rXWpROsNZpH11rwFbd1pNyxja2kCoh7fgA4QarF18lbpnMF8dLymsfm Udjw== X-Gm-Message-State: AOAM531gqItpsgXHKDaSpGgvICLLYcUf1tFjSd5Hmn6bb7cQYxQ8Rya5 3g+nICZ6aNPmzDixzCGQ2TOQ4k4= X-Google-Smtp-Source: ABdhPJxOe8i9dUvdOcBwgJL1gRoz1LXPH0NwtbCrsLHg8SQ5J62BAfH5WO/eL1adhFv64564uJCCI1E= X-Received: from pcc-desktop.svl.corp.google.com ([2620:15c:2ce:200:edb5:f28b:d3c2:ad4f]) (user=pcc job=sendgmr) by 2002:a81:3dd8:0:b0:2ca:287c:6d3f with SMTP id 00721157ae682-2ca287c6f83mr2601697b3.484.1643072336569; Mon, 24 Jan 2022 16:58:56 -0800 (PST) Date: Mon, 24 Jan 2022 16:58:50 -0800 Message-Id: <20220125005850.2500784-1-pcc@google.com> Mime-Version: 1.0 X-Mailer: git-send-email 2.35.0.rc0.227.g00780c9af4-goog Subject: [PATCH v4] arm64: mte: avoid clearing PSTATE.TCO on entry unless necessary From: Peter Collingbourne To: Catalin Marinas , Vincenzo Frascino , Will Deacon , Andrey Konovalov , Mark Rutland Cc: Peter Collingbourne , Evgenii Stepanov , linux-arm-kernel@lists.infradead.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220124_165858_497166_065985F1 X-CRM114-Status: GOOD ( 20.63 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On some microarchitectures, clearing PSTATE.TCO is expensive. Clearing TCO is only necessary if in-kernel MTE is enabled, or if MTE is enabled in the userspace process in synchronous (or, soon, asymmetric) mode, because we do not report uaccess faults to userspace in none or asynchronous modes. Therefore, adjust the kernel entry code to clear TCO only if necessary. Because it is now possible to switch to a task in which TCO needs to be clear from a task in which TCO is set, we also need to do the same thing on task switch. Signed-off-by: Peter Collingbourne Link: https://linux-review.googlesource.com/id/I52d82a580bd0500d420be501af2c35fa8c90729e Reviewed-by: Catalin Marinas --- v4: - some changes suggested by Catalin v3: - switch to a C implementation v2: - do the same thing in cpu_switch_to() arch/arm64/include/asm/mte.h | 22 ++++++++++++++++++++++ arch/arm64/kernel/entry-common.c | 3 +++ arch/arm64/kernel/entry.S | 7 ------- arch/arm64/kernel/mte.c | 3 +++ 4 files changed, 28 insertions(+), 7 deletions(-) diff --git a/arch/arm64/include/asm/mte.h b/arch/arm64/include/asm/mte.h index 075539f5f1c8..808349363625 100644 --- a/arch/arm64/include/asm/mte.h +++ b/arch/arm64/include/asm/mte.h @@ -11,7 +11,9 @@ #ifndef __ASSEMBLY__ #include +#include #include +#include #include #include @@ -86,6 +88,26 @@ static inline int mte_ptrace_copy_tags(struct task_struct *child, #endif /* CONFIG_ARM64_MTE */ +static inline void mte_disable_tco_entry(struct task_struct *task) +{ + if (!system_supports_mte()) + return; + + /* + * Re-enable tag checking (TCO set on exception entry). This is only + * necessary if MTE is enabled in either the kernel or the userspace + * task in synchronous or asymmetric mode (SCTLR_EL1.TCF0 bit 0 is set + * for both). With MTE disabled in the kernel and disabled or + * asynchronous in userspace, tag check faults (including in uaccesses) + * are not reported, therefore there is no need to re-enable checking. + * This is beneficial on microarchitectures where re-enabling TCO is + * expensive. + */ + if (kasan_hw_tags_enabled() || + (task->thread.sctlr_user & (1UL << SCTLR_EL1_TCF0_SHIFT))) + asm volatile(SET_PSTATE_TCO(0)); +} + #ifdef CONFIG_KASAN_HW_TAGS /* Whether the MTE asynchronous mode is enabled. */ DECLARE_STATIC_KEY_FALSE(mte_async_or_asymm_mode); diff --git a/arch/arm64/kernel/entry-common.c b/arch/arm64/kernel/entry-common.c index ef7fcefb96bd..7093b578e325 100644 --- a/arch/arm64/kernel/entry-common.c +++ b/arch/arm64/kernel/entry-common.c @@ -6,6 +6,7 @@ */ #include +#include #include #include #include @@ -56,6 +57,7 @@ static void noinstr enter_from_kernel_mode(struct pt_regs *regs) { __enter_from_kernel_mode(regs); mte_check_tfsr_entry(); + mte_disable_tco_entry(current); } /* @@ -103,6 +105,7 @@ static __always_inline void __enter_from_user_mode(void) CT_WARN_ON(ct_state() != CONTEXT_USER); user_exit_irqoff(); trace_hardirqs_off_finish(); + mte_disable_tco_entry(current); } static __always_inline void enter_from_user_mode(struct pt_regs *regs) diff --git a/arch/arm64/kernel/entry.S b/arch/arm64/kernel/entry.S index 772ec2ecf488..e1013a83d4f0 100644 --- a/arch/arm64/kernel/entry.S +++ b/arch/arm64/kernel/entry.S @@ -308,13 +308,6 @@ alternative_if ARM64_HAS_IRQ_PRIO_MASKING msr_s SYS_ICC_PMR_EL1, x20 alternative_else_nop_endif - /* Re-enable tag checking (TCO set on exception entry) */ -#ifdef CONFIG_ARM64_MTE -alternative_if ARM64_MTE - SET_PSTATE_TCO(0) -alternative_else_nop_endif -#endif - /* * Registers that may be useful after this macro is invoked: * diff --git a/arch/arm64/kernel/mte.c b/arch/arm64/kernel/mte.c index f418ebc65f95..f983795b5eda 100644 --- a/arch/arm64/kernel/mte.c +++ b/arch/arm64/kernel/mte.c @@ -253,6 +253,9 @@ void mte_thread_switch(struct task_struct *next) mte_update_sctlr_user(next); mte_update_gcr_excl(next); + /* TCO may not have been disabled on exception entry for the current task. */ + mte_disable_tco_entry(next); + /* * Check if an async tag exception occurred at EL1. *