From patchwork Fri Jun 11 21:51:01 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Collingbourne X-Patchwork-Id: 12316601 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.9 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id F2D48C48BE0 for ; Fri, 11 Jun 2021 21:52:41 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id B7F3B613C3 for ; Fri, 11 Jun 2021 21:52:41 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org B7F3B613C3 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Mime-Version: Message-Id:Date:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:In-Reply-To: References:List-Owner; bh=1SVxUzsmAtmkPKry0/Z3eNCHnp4Zgk/K48Dbe6oMJ1w=; b=2Rd pCl1h6qFmD0CokWx+3kKo4PVOlgYyoXlaQyBcWwHbXtTyz4VpqJQMRdE0BiN8m78Iy18cpWX+K2Pr e8fozt+q2yWSQSbhwH0xr3bC4EXn8GrG2bGyIxRaSbRqLpiqBhsMrnI/o3rwJy0GKrXnY1rejG02p xJTFPlTuEd+TUT6sgmgQNuF8QkonCGEUYW7Nw5eoRF6JVu7Eu6Z5AXEKfc44D3VbP3jzzjAvZFShb 8M/0lr1Dr7MAf7Dz1vPVmAiBmO72qln7i9x3nXCDtCzLHWu8cbQxGXoWwuAJqjvy4fQpd9PIPPwHH O/U4FeAekU51COhe8RNLBYDtNAKVjZw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1lrp3a-007FKN-MN; Fri, 11 Jun 2021 21:51:14 +0000 Received: from mail-yb1-xb4a.google.com ([2607:f8b0:4864:20::b4a]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1lrp3W-007FIq-Ca for linux-arm-kernel@lists.infradead.org; Fri, 11 Jun 2021 21:51:13 +0000 Received: by mail-yb1-xb4a.google.com with SMTP id x187-20020a25e0c40000b029052a5f0bf9acso6087756ybg.1 for ; Fri, 11 Jun 2021 14:51:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:message-id:mime-version:subject:from:to:cc; bh=CIO736QCpP0vGs3q8qFcvJzcaOHFR5419N8Jx9U65Jw=; b=fgFL358n5jNnoHbQ2DD9xUno2SOM9k9d3MvWLRho5lqGrU9Vp4bqVnT+OGS1SuO4Fg Fbcig1ckyPv/mdFEqALs8/WVgun0oiew/m97S4aXPzfZpOpIFMVClYX1yw6jGtEEO5+/ /qkvUv56zSTdFZc5cnEa2llwrwNqcKSL9n0XXYYE6YtnS38d4402yqIuKe7T5j0qts9L ZuREUisZLMXSej9YUpZO+vxi8YVvPkfAONca6KyN9Rm2wH7EAL5spzdwb1UcoB1f3RDX EDymaOGTG1M2E3OPMYssGkAQnjvyUxZEQSUqvFEJyPOPZ9RknDHDZwamgCKa75ZnZGOv a8yQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:message-id:mime-version:subject:from:to:cc; bh=CIO736QCpP0vGs3q8qFcvJzcaOHFR5419N8Jx9U65Jw=; b=FL07hDEMNQytFIP7DBgLeTgodqmAQicU6/lqyAhNYmMqCdk3UNmcGrQtBow1ordu1K lVYZkBpTPBysOg3rGcb27dIztysr7w3xdO2cyEzAQblywkm4XSCMLAJa1nG4iIN5D2Rr 1BvtaDDADCRiXG7rwuGOSpJfu9wNV608DJhdreRjKPJZm+2eyG3Vg6WbEDZdFUtrFKCR rFuVNFXtYTuC8pXrNg+NVumYOcJVaUvj9SrhDafx8hRuBCNyt4FaFwLqOVEZw1UZIPIt h9v9uyZ79DeokEfmZXO4A7XHVuDF8QT7TlFVFqMfF6WwOtjcveZXDNwhKIq+6VVNMQs0 j9pg== X-Gm-Message-State: AOAM532aHYmZV6jB0zdxQKZ1cZ0xa75UpQGJOvBt8915/dUV6VPgOz8k 3jaCi/6J/pbH/8TFxviZcs9YWNk= X-Google-Smtp-Source: ABdhPJxgawrN5RdnTR2SLF2Ymk3BbsiVBlvTuWbwj8t12Te7B80AcIsJDOH+Xo0tKN+l3J+4fTXdFYE= X-Received: from pcc-desktop.svl.corp.google.com ([2620:15c:2ce:200:4fb0:9978:e530:6b3c]) (user=pcc job=sendgmr) by 2002:a25:1fc6:: with SMTP id f189mr8885562ybf.452.1623448267341; Fri, 11 Jun 2021 14:51:07 -0700 (PDT) Date: Fri, 11 Jun 2021 14:51:01 -0700 Message-Id: <20210611215101.1060663-1-pcc@google.com> Mime-Version: 1.0 X-Mailer: git-send-email 2.32.0.272.g935e593368-goog Subject: [PATCH v3] arm64: mte: allow async MTE to be upgraded to sync on a per-CPU basis From: Peter Collingbourne To: Catalin Marinas , Vincenzo Frascino , Will Deacon Cc: Peter Collingbourne , Evgenii Stepanov , linux-arm-kernel@lists.infradead.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210611_145110_463143_4B15517C X-CRM114-Status: GOOD ( 30.29 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On some CPUs the performance of MTE in synchronous mode is the same as that of asynchronous mode. This makes it worthwhile to enable synchronous mode on those CPUs when asynchronous mode is requested, in order to gain the error detection benefits of synchronous mode without the performance downsides. Therefore, make it possible for user programs to opt into upgrading to synchronous mode on those CPUs via a new prctl flag. The flag is orthogonal to the existing TCF modes in order to accommodate upgrading from other TCF modes in the future. The feature is controlled on a per-CPU basis via sysfs. Signed-off-by: Peter Collingbourne Link: https://linux-review.googlesource.com/id/Id6f95b71fde6e701dd30b5e108126af7286147e8 --- v3: - drop the device tree support - add documentation - add static_assert to ensure no overlap with real HW bits - move per-CPU variable initialization to mte.c - use smp_call_function_single instead of stop_machine v2: - make it an opt-in behavior - change the format of the device tree node - also allow controlling the feature via sysfs .../arm64/memory-tagging-extension.rst | 20 ++++++ arch/arm64/include/asm/processor.h | 8 +++ arch/arm64/kernel/mte.c | 71 ++++++++++++++++++- arch/arm64/kernel/process.c | 15 +++- include/uapi/linux/prctl.h | 2 + 5 files changed, 113 insertions(+), 3 deletions(-) diff --git a/Documentation/arm64/memory-tagging-extension.rst b/Documentation/arm64/memory-tagging-extension.rst index b540178a93f8..2fc145de6530 100644 --- a/Documentation/arm64/memory-tagging-extension.rst +++ b/Documentation/arm64/memory-tagging-extension.rst @@ -120,6 +120,25 @@ in the ``PR_MTE_TAG_MASK`` bit-field. interface provides an include mask. An include mask of ``0`` (exclusion mask ``0xffff``) results in the CPU always generating tag ``0``. +Upgrading to stricter tag checking modes +---------------------------------------- + +On some CPUs the performance of MTE in stricter tag checking modes +is the same as that of less strict tag checking modes. This makes it +worthwhile to enable stricter checks on those CPUs when a less strict +checking mode is requested, in order to gain the error detection +benefits of the stricter checks without the performance downsides. To +opt into upgrading to a stricter checking mode on those CPUs, the user +can set the ``PR_MTE_DYNAMIC_TCF`` flag bit in the ``flags`` argument +to the ``prctl(PR_SET_TAGGED_ADDR_CTRL, flags, 0, 0, 0)`` system call. + +This feature is currently only supported for upgrading from +asynchronous mode. To configure a CPU to upgrade from asynchronous mode +to synchronous mode, a privileged user may write the value ``1`` to +``/sys/devices/system/cpu/cpu/mte_upgrade_async``, and to disable +upgrading they may write the value ``2``. By default the feature is +disabled on all CPUs. + Initial process state --------------------- @@ -128,6 +147,7 @@ On ``execve()``, the new process has the following configuration: - ``PR_TAGGED_ADDR_ENABLE`` set to 0 (disabled) - Tag checking mode set to ``PR_MTE_TCF_NONE`` - ``PR_MTE_TAG_MASK`` set to 0 (all tags excluded) +- ``PR_MTE_DYNAMIC_TCF`` set to 0 (disabled) - ``PSTATE.TCO`` set to 0 - ``PROT_MTE`` not set on any of the initial memory maps diff --git a/arch/arm64/include/asm/processor.h b/arch/arm64/include/asm/processor.h index 9df3feeee890..531f580e859e 100644 --- a/arch/arm64/include/asm/processor.h +++ b/arch/arm64/include/asm/processor.h @@ -159,6 +159,13 @@ struct thread_struct { #define SCTLR_USER_MASK \ (SCTLR_ELx_ENIA | SCTLR_ELx_ENIB | SCTLR_ELx_ENDA | SCTLR_ELx_ENDB | \ SCTLR_EL1_TCF0_MASK) +#define SCTLR_USER_DYNAMIC_TCF (1ULL << 63) + +/* + * Make sure that SCTLR_USER_DYNAMIC_TCF doesn't overlap with any of the real + * per-task hardware bits. + */ +static_assert((SCTLR_USER_MASK & SCTLR_USER_DYNAMIC_TCF) == 0); static inline void arch_thread_struct_whitelist(unsigned long *offset, unsigned long *size) @@ -251,6 +258,7 @@ extern void release_thread(struct task_struct *); unsigned long get_wchan(struct task_struct *p); +void update_sctlr_el1(u64 sctlr); void set_task_sctlr_el1(u64 sctlr); /* Thread switching */ diff --git a/arch/arm64/kernel/mte.c b/arch/arm64/kernel/mte.c index 125a10e413e9..d40bb03dfacc 100644 --- a/arch/arm64/kernel/mte.c +++ b/arch/arm64/kernel/mte.c @@ -4,6 +4,7 @@ */ #include +#include #include #include #include @@ -26,6 +27,9 @@ u64 gcr_kernel_excl __ro_after_init; static bool report_fault_once = true; +DEFINE_PER_CPU_READ_MOSTLY(u64, mte_upgrade_async); +EXPORT_PER_CPU_SYMBOL(mte_upgrade_async); + #ifdef CONFIG_KASAN_HW_TAGS /* Whether the MTE asynchronous mode is enabled. */ DEFINE_STATIC_KEY_FALSE(mte_async_mode); @@ -262,7 +266,8 @@ void mte_suspend_exit(void) long set_mte_ctrl(struct task_struct *task, unsigned long arg) { - u64 sctlr = task->thread.sctlr_user & ~SCTLR_EL1_TCF0_MASK; + u64 sctlr = task->thread.sctlr_user & + ~(SCTLR_EL1_TCF0_MASK | SCTLR_USER_DYNAMIC_TCF); u64 gcr_excl = ~((arg & PR_MTE_TAG_MASK) >> PR_MTE_TAG_SHIFT) & SYS_GCR_EL1_EXCL_MASK; @@ -283,6 +288,9 @@ long set_mte_ctrl(struct task_struct *task, unsigned long arg) return -EINVAL; } + if (arg & PR_MTE_DYNAMIC_TCF) + sctlr |= SCTLR_USER_DYNAMIC_TCF; + if (task != current) { task->thread.sctlr_user = sctlr; task->thread.gcr_user_excl = gcr_excl; @@ -316,6 +324,9 @@ long get_mte_ctrl(struct task_struct *task) break; } + if (task->thread.sctlr_user & SCTLR_USER_DYNAMIC_TCF) + ret |= PR_MTE_DYNAMIC_TCF; + return ret; } @@ -453,3 +464,61 @@ int mte_ptrace_copy_tags(struct task_struct *child, long request, return ret; } + +static ssize_t mte_upgrade_async_show(struct device *dev, + struct device_attribute *attr, char *buf) +{ + u32 val = per_cpu(mte_upgrade_async, dev->id) >> SCTLR_EL1_TCF0_SHIFT; + + return sysfs_emit(buf, "%u\n", val); +} + +static void sync_sctlr(void *arg) +{ + update_sctlr_el1(current->thread.sctlr_user); +} + +static ssize_t mte_upgrade_async_store(struct device *dev, + struct device_attribute *attr, + const char *buf, size_t count) +{ + ssize_t ret; + u32 val; + u64 tcf; + + ret = kstrtou32(buf, 0, &val); + if (ret < 0) + return ret; + + tcf = ((u64)val) << SCTLR_EL1_TCF0_SHIFT; + if (tcf != SCTLR_EL1_TCF0_NONE && tcf != SCTLR_EL1_TCF0_SYNC && + tcf != SCTLR_EL1_TCF0_ASYNC) + return -EINVAL; + + device_lock(dev); + per_cpu(mte_upgrade_async, dev->id) = tcf; + + if (cpu_online(dev->id)) + ret = smp_call_function_single(dev->id, sync_sctlr, NULL, 0); + if (ret == 0) + ret = count; + device_unlock(dev); + + return ret; +} +static DEVICE_ATTR_RW(mte_upgrade_async); + +static void register_mte_upgrade_async_sysctl(void) +{ + unsigned int cpu; + + if (!system_supports_mte()) + return; + + for_each_possible_cpu(cpu) { + per_cpu(mte_upgrade_async, cpu) = SCTLR_EL1_TCF0_ASYNC; + device_create_file(get_cpu_device(cpu), + &dev_attr_mte_upgrade_async); + } +} +subsys_initcall(register_mte_upgrade_async_sysctl); diff --git a/arch/arm64/kernel/process.c b/arch/arm64/kernel/process.c index b4bb67f17a2c..27a12c53529d 100644 --- a/arch/arm64/kernel/process.c +++ b/arch/arm64/kernel/process.c @@ -527,8 +527,19 @@ static void erratum_1418040_thread_switch(struct task_struct *prev, write_sysreg(val, cntkctl_el1); } -static void update_sctlr_el1(u64 sctlr) +DECLARE_PER_CPU_READ_MOSTLY(u64, mte_upgrade_async); + +void update_sctlr_el1(u64 sctlr) { + if (sctlr & SCTLR_USER_DYNAMIC_TCF) { + sctlr &= ~SCTLR_USER_DYNAMIC_TCF; + + if ((sctlr & SCTLR_EL1_TCF0_MASK) == SCTLR_EL1_TCF0_ASYNC) { + sctlr &= ~SCTLR_EL1_TCF0_MASK; + sctlr |= __this_cpu_read(mte_upgrade_async); + } + } + /* * EnIA must not be cleared while in the kernel as this is necessary for * in-kernel PAC. It will be cleared on kernel exit if needed. @@ -659,7 +670,7 @@ long set_tagged_addr_ctrl(struct task_struct *task, unsigned long arg) return -EINVAL; if (system_supports_mte()) - valid_mask |= PR_MTE_TCF_MASK | PR_MTE_TAG_MASK; + valid_mask |= PR_MTE_TCF_MASK | PR_MTE_TAG_MASK | PR_MTE_DYNAMIC_TCF; if (arg & ~valid_mask) return -EINVAL; diff --git a/include/uapi/linux/prctl.h b/include/uapi/linux/prctl.h index 18a9f59dc067..4dab44732814 100644 --- a/include/uapi/linux/prctl.h +++ b/include/uapi/linux/prctl.h @@ -242,6 +242,8 @@ struct prctl_mm_map { /* MTE tag inclusion mask */ # define PR_MTE_TAG_SHIFT 3 # define PR_MTE_TAG_MASK (0xffffUL << PR_MTE_TAG_SHIFT) +/* Enable dynamic upgrading of MTE tag check fault mode */ +# define PR_MTE_DYNAMIC_TCF (1UL << 19) /* Control reclaim behavior when allocating memory */ #define PR_SET_IO_FLUSHER 57