From patchwork Thu Jun 10 02:12:29 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Collingbourne X-Patchwork-Id: 12311581 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.9 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D13AFC48BCD for ; Thu, 10 Jun 2021 02:17:16 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 9EDFD61407 for ; Thu, 10 Jun 2021 02:17:16 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 9EDFD61407 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Mime-Version: Message-Id:Date:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:In-Reply-To: References:List-Owner; bh=wNQDFTGd5DDXRdL1jHFpBOH32hlBqVFc6szjMBzYnuY=; b=rhE tn8HHdNTCI7Xen53o6uE1h3EnhXL74nJzwzEA1Z6/L15fXl91CNObUjFpg/88wUg3sS3LfC55M3qb RS5g98JBJjvDWEVX1MdZ7ps9spbICK88hZTYedSngioIMod2NfR41aHs2cG+NLViBX+5DsSEOn9YD UcPRSBNredFMI68cZZFlstXqwGaEcQ/IRfnmrF288dE8Dfa0Wp+hU0n5hOGJuwZ7C3WYFOZAzxQTu BFwoPWM/qf40dgqq7YIUBBHJZNRCcyZwNWP1ViGFlVX7MjtlUhVHTINQA+ieltI2lAlroldvqlWUq 8Ehfp0dtdVKf/zpCBhs1rq1XtBWKZ9w==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1lrABd-00GOBy-5k; Thu, 10 Jun 2021 02:12:49 +0000 Received: from mail-qv1-xf49.google.com ([2607:f8b0:4864:20::f49]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1lrABT-00GO9L-2x for linux-arm-kernel@lists.infradead.org; Thu, 10 Jun 2021 02:12:41 +0000 Received: by mail-qv1-xf49.google.com with SMTP id dr11-20020a05621408ebb029021e40008bd5so18519140qvb.0 for ; Wed, 09 Jun 2021 19:12:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:message-id:mime-version:subject:from:to:cc; bh=iEX0mCjNbx2/ICr8W61+CUTenQY9Hlv6jFHVIIOwUTA=; b=dqfUn4IQvw9P+JQ89iQ/yNKdpD4Nij5ohmpa7Z8YzQaoK9YTSmWhgqilHoK8tOK+ay jc82XQnfbPZqpKlVHH/fhJI9Nhza9WPLytpzh1ZBzLRjvYGJxzY6LmEK8tuemaW/VwWz Ccu2XTlA6L+YNuQLZnluZoDDZhDEHEd1IZGxLbkzCYq27fhICfazQ+BDh9DE0AYGYswh pryJbbY8RwIT6Dv9+L8m9bfcC3VCUOQ8ruEtqgH06SVfm1JVrjfcXdnXCH0zEP3jgiQA sNvqzuIXw7yuX/XWHB89s42hLgJK8DIMojWHhIT6Vrb1M00gNJlS8wPUv/75/SrLn1HK hIBg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:message-id:mime-version:subject:from:to:cc; bh=iEX0mCjNbx2/ICr8W61+CUTenQY9Hlv6jFHVIIOwUTA=; b=NpucqsRejL7TcvT+LoRaa5QplvJ79T5sCU0j9uPWKT6FTY3kP2v+CMrk3xVvTmvjmh eiA8ME/dKHhfinLqvnMIAG2p4CzUSVVvbWIVGCQN+sZ8NzRgK1KiYpobCBpc3eaXarax OQzlRyKNBBGNSAjX0RJmPEllg9rcKguiX5q2sxV9Ub57w7m+el86cEhNpltMMhVKul68 XE3ESsEclve9hUIdaC78jng96Zpl3HT3g6P8cKSuM7dpCD1KO8dlRv54iwXvNqQUsjjO FW8PtQE7OCrYKT418Sj+KYcMWXKZiioI5B1pUJH51dqwC8A0HHJlB9WzXB4WZJHpjJwH eq1A== X-Gm-Message-State: AOAM531/CFE9ZAXC5Npk/wco6kL+FxiBV8RVpvIIgsoxBCVyRY4N39Gf goWenmsqxaozOXUa+NfP3AZc9PI= X-Google-Smtp-Source: ABdhPJyzTbvVFuhBp6bA8BnAu7K+eASviSBEbDxCRf2BrSEemOWLL3F6Ynu3L/HMJqq+mJGSQ5sTmHo= X-Received: from pcc-desktop.svl.corp.google.com ([2620:15c:2ce:200:6a72:2c59:5b4b:a139]) (user=pcc job=sendgmr) by 2002:ad4:5561:: with SMTP id w1mr2927368qvy.47.1623291155896; Wed, 09 Jun 2021 19:12:35 -0700 (PDT) Date: Wed, 9 Jun 2021 19:12:29 -0700 Message-Id: <20210610021229.3605241-1-pcc@google.com> Mime-Version: 1.0 X-Mailer: git-send-email 2.32.0.rc1.229.g3e70b5a671-goog Subject: [PATCH v2] arm64: mte: allow async MTE to be upgraded to sync on a per-CPU basis From: Peter Collingbourne To: Catalin Marinas , Vincenzo Frascino , Will Deacon Cc: Peter Collingbourne , Evgenii Stepanov , linux-arm-kernel@lists.infradead.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210609_191239_210357_78A372B8 X-CRM114-Status: GOOD ( 29.85 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On some CPUs the performance of MTE in synchronous mode is the same as that of asynchronous mode. This makes it worthwhile to enable synchronous mode on those CPUs when asynchronous mode is requested, in order to gain the error detection benefits of synchronous mode without the performance downsides. Therefore, make it possible for user programs to opt into upgrading to synchronous mode on those CPUs via a new prctl flag. The flag is orthogonal to the existing TCF modes in order to accommodate upgrading from other TCF modes in the future. The feature is controlled on a per-CPU basis via a new device tree node, but it may also be controlled via sysfs. Signed-off-by: Peter Collingbourne Link: https://linux-review.googlesource.com/id/Id6f95b71fde6e701dd30b5e108126af7286147e8 --- v2: - make it an opt-in behavior - change the format of the device tree node - also allow controlling the feature via sysfs arch/arm64/include/asm/processor.h | 2 + arch/arm64/kernel/mte.c | 70 +++++++++++++++++++++++++++++- arch/arm64/kernel/process.c | 15 ++++++- arch/arm64/kernel/smp.c | 34 +++++++++++++++ include/uapi/linux/prctl.h | 2 + 5 files changed, 120 insertions(+), 3 deletions(-) diff --git a/arch/arm64/include/asm/processor.h b/arch/arm64/include/asm/processor.h index 9df3feeee890..545ef900e7ce 100644 --- a/arch/arm64/include/asm/processor.h +++ b/arch/arm64/include/asm/processor.h @@ -159,6 +159,7 @@ struct thread_struct { #define SCTLR_USER_MASK \ (SCTLR_ELx_ENIA | SCTLR_ELx_ENIB | SCTLR_ELx_ENDA | SCTLR_ELx_ENDB | \ SCTLR_EL1_TCF0_MASK) +#define SCTLR_USER_DYNAMIC_TCF (1ULL << 63) static inline void arch_thread_struct_whitelist(unsigned long *offset, unsigned long *size) @@ -251,6 +252,7 @@ extern void release_thread(struct task_struct *); unsigned long get_wchan(struct task_struct *p); +void update_sctlr_el1(u64 sctlr); void set_task_sctlr_el1(u64 sctlr); /* Thread switching */ diff --git a/arch/arm64/kernel/mte.c b/arch/arm64/kernel/mte.c index 125a10e413e9..f244bc96464c 100644 --- a/arch/arm64/kernel/mte.c +++ b/arch/arm64/kernel/mte.c @@ -4,11 +4,13 @@ */ #include +#include #include #include #include #include #include +#include #include #include #include @@ -26,6 +28,8 @@ u64 gcr_kernel_excl __ro_after_init; static bool report_fault_once = true; +DECLARE_PER_CPU_READ_MOSTLY(u64, mte_upgrade_async); + #ifdef CONFIG_KASAN_HW_TAGS /* Whether the MTE asynchronous mode is enabled. */ DEFINE_STATIC_KEY_FALSE(mte_async_mode); @@ -262,7 +266,8 @@ void mte_suspend_exit(void) long set_mte_ctrl(struct task_struct *task, unsigned long arg) { - u64 sctlr = task->thread.sctlr_user & ~SCTLR_EL1_TCF0_MASK; + u64 sctlr = task->thread.sctlr_user & + ~(SCTLR_EL1_TCF0_MASK | SCTLR_USER_DYNAMIC_TCF); u64 gcr_excl = ~((arg & PR_MTE_TAG_MASK) >> PR_MTE_TAG_SHIFT) & SYS_GCR_EL1_EXCL_MASK; @@ -283,6 +288,9 @@ long set_mte_ctrl(struct task_struct *task, unsigned long arg) return -EINVAL; } + if (arg & PR_MTE_DYNAMIC_TCF) + sctlr |= SCTLR_USER_DYNAMIC_TCF; + if (task != current) { task->thread.sctlr_user = sctlr; task->thread.gcr_user_excl = gcr_excl; @@ -316,6 +324,9 @@ long get_mte_ctrl(struct task_struct *task) break; } + if (task->thread.sctlr_user & SCTLR_USER_DYNAMIC_TCF) + ret |= PR_MTE_DYNAMIC_TCF; + return ret; } @@ -453,3 +464,60 @@ int mte_ptrace_copy_tags(struct task_struct *child, long request, return ret; } + +static ssize_t mte_upgrade_async_show(struct device *dev, + struct device_attribute *attr, char *buf) +{ + u32 val = per_cpu(mte_upgrade_async, dev->id) >> SCTLR_EL1_TCF0_SHIFT; + + return sysfs_emit(buf, "%u\n", val); +} + +static int sync_sctlr(void *arg) +{ + update_sctlr_el1(current->thread.sctlr_user); + return 0; +} + +static ssize_t mte_upgrade_async_store(struct device *dev, + struct device_attribute *attr, + const char *buf, size_t count) +{ + ssize_t ret; + u32 val; + u64 tcf; + + ret = kstrtou32(buf, 0, &val); + if (ret < 0) + return ret; + + tcf = ((u64)val) << SCTLR_EL1_TCF0_SHIFT; + if (tcf != SCTLR_EL1_TCF0_NONE && tcf != SCTLR_EL1_TCF0_SYNC && + tcf != SCTLR_EL1_TCF0_ASYNC) + return -EINVAL; + + device_lock(dev); + per_cpu(mte_upgrade_async, dev->id) = tcf; + + ret = stop_machine(sync_sctlr, 0, cpumask_of(dev->id)); + if (ret == 0) + ret = count; + device_unlock(dev); + + return ret; +} +static DEVICE_ATTR_RW(mte_upgrade_async); + +static void register_mte_upgrade_async_sysctl(void) +{ + unsigned int cpu; + + if (!system_supports_mte()) + return; + + for_each_possible_cpu(cpu) { + device_create_file(get_cpu_device(cpu), + &dev_attr_mte_upgrade_async); + } +} +subsys_initcall(register_mte_upgrade_async_sysctl); diff --git a/arch/arm64/kernel/process.c b/arch/arm64/kernel/process.c index b4bb67f17a2c..27a12c53529d 100644 --- a/arch/arm64/kernel/process.c +++ b/arch/arm64/kernel/process.c @@ -527,8 +527,19 @@ static void erratum_1418040_thread_switch(struct task_struct *prev, write_sysreg(val, cntkctl_el1); } -static void update_sctlr_el1(u64 sctlr) +DECLARE_PER_CPU_READ_MOSTLY(u64, mte_upgrade_async); + +void update_sctlr_el1(u64 sctlr) { + if (sctlr & SCTLR_USER_DYNAMIC_TCF) { + sctlr &= ~SCTLR_USER_DYNAMIC_TCF; + + if ((sctlr & SCTLR_EL1_TCF0_MASK) == SCTLR_EL1_TCF0_ASYNC) { + sctlr &= ~SCTLR_EL1_TCF0_MASK; + sctlr |= __this_cpu_read(mte_upgrade_async); + } + } + /* * EnIA must not be cleared while in the kernel as this is necessary for * in-kernel PAC. It will be cleared on kernel exit if needed. @@ -659,7 +670,7 @@ long set_tagged_addr_ctrl(struct task_struct *task, unsigned long arg) return -EINVAL; if (system_supports_mte()) - valid_mask |= PR_MTE_TCF_MASK | PR_MTE_TAG_MASK; + valid_mask |= PR_MTE_TCF_MASK | PR_MTE_TAG_MASK | PR_MTE_DYNAMIC_TCF; if (arg & ~valid_mask) return -EINVAL; diff --git a/arch/arm64/kernel/smp.c b/arch/arm64/kernel/smp.c index dcd7041b2b07..20c1503c8cdd 100644 --- a/arch/arm64/kernel/smp.c +++ b/arch/arm64/kernel/smp.c @@ -56,6 +56,8 @@ DEFINE_PER_CPU_READ_MOSTLY(int, cpu_number); EXPORT_PER_CPU_SYMBOL(cpu_number); +DEFINE_PER_CPU_READ_MOSTLY(u64, mte_upgrade_async); +EXPORT_PER_CPU_SYMBOL(mte_upgrade_async); /* * as from 2.5, kernels no longer have an init_tasks structure @@ -649,6 +651,27 @@ static void __init acpi_parse_and_init_cpus(void) #define acpi_parse_and_init_cpus(...) do { } while (0) #endif +/* + * Store the default values for per-CPU properties typically read from DT or + * ACPI to per-CPU variables. + */ +static void __init set_default_cpu_properties(int cpu) +{ + per_cpu(mte_upgrade_async, cpu) = SCTLR_EL1_TCF0_ASYNC; +} + +/* + * Read per-CPU properties from the device tree and store them in per-CPU + * variables for efficient access later. + */ +static void __init of_read_cpu_properties(int cpu, struct device_node *dn) +{ + u32 prop; + + if (of_property_read_u32(dn, "mte-upgrade-async", &prop) == 0) + per_cpu(mte_upgrade_async, cpu) = ((u64)prop) << SCTLR_EL1_TCF0_SHIFT; +} + /* * Enumerate the possible CPU set from the device tree and build the * cpu logical map array containing MPIDR values related to logical @@ -774,6 +797,7 @@ void __init smp_prepare_cpus(unsigned int max_cpus) for_each_possible_cpu(cpu) { per_cpu(cpu_number, cpu) = cpu; + set_default_cpu_properties(cpu); if (cpu == smp_processor_id()) continue; @@ -789,6 +813,16 @@ void __init smp_prepare_cpus(unsigned int max_cpus) set_cpu_present(cpu, true); numa_store_cpu_info(cpu); } + + if (acpi_disabled) { + struct device_node *dn; + int cpu = 0; + + for_each_of_cpu_node(dn) { + of_read_cpu_properties(cpu, dn); + cpu++; + } + } } static const char *ipi_types[NR_IPI] __tracepoint_string = { diff --git a/include/uapi/linux/prctl.h b/include/uapi/linux/prctl.h index 18a9f59dc067..4dab44732814 100644 --- a/include/uapi/linux/prctl.h +++ b/include/uapi/linux/prctl.h @@ -242,6 +242,8 @@ struct prctl_mm_map { /* MTE tag inclusion mask */ # define PR_MTE_TAG_SHIFT 3 # define PR_MTE_TAG_MASK (0xffffUL << PR_MTE_TAG_SHIFT) +/* Enable dynamic upgrading of MTE tag check fault mode */ +# define PR_MTE_DYNAMIC_TCF (1UL << 19) /* Control reclaim behavior when allocating memory */ #define PR_SET_IO_FLUSHER 57