From patchwork Fri Jan 15 12:00:40 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vincenzo Frascino X-Patchwork-Id: 12022357 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.0 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5519EC433E0 for ; Fri, 15 Jan 2021 12:02:42 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 141B423128 for ; Fri, 15 Jan 2021 12:02:42 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 141B423128 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=arm.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Message-Id:Date: Subject:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=injodCgdKyDcm1N9GtXVEkaVuplloyfhckYFaX4BIog=; b=DtZcspQqhE36G0ypoKzdX/QdK UiMSKs0ic53eH80+rV5zI0zXKuF8lQdKe8eEFq6P1l0J/ysmwGb+hwNFpMc/SfQSUa2bmYXI/tn8j 6ewGGYNFDHlwSwlirDe2HRcePeq9IaMHGkV+U0270BGyqFKqzVFKd/kjsPq8+owZhaSsspmxUOEcT TPFJU2LqLAed5y8vYpaz8Ga1GwzmvrZ1rC6UKHGvSrX0Hcmlrz7PyiCR8dyAcXbaHdxLTLsvB8O1O TFYtUvJcpQPCWE++2vxAmYqa5A9IEppP9+8T/sEIRAhMkc29k24afQuQMNPLXJnUYY31jAN0AVyNm b1tgpzXUQ==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1l0Nmt-00050c-A2; Fri, 15 Jan 2021 12:01:07 +0000 Received: from foss.arm.com ([217.140.110.172]) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1l0Nmk-0004y3-24 for linux-arm-kernel@lists.infradead.org; Fri, 15 Jan 2021 12:00:59 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 5777411B3; Fri, 15 Jan 2021 04:00:55 -0800 (PST) Received: from e119884-lin.cambridge.arm.com (e119884-lin.cambridge.arm.com [10.1.196.72]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id ACBD73F70D; Fri, 15 Jan 2021 04:00:53 -0800 (PST) From: Vincenzo Frascino To: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kasan-dev@googlegroups.com Subject: [PATCH v3 1/4] kasan, arm64: Add KASAN light mode Date: Fri, 15 Jan 2021 12:00:40 +0000 Message-Id: <20210115120043.50023-2-vincenzo.frascino@arm.com> X-Mailer: git-send-email 2.30.0 In-Reply-To: <20210115120043.50023-1-vincenzo.frascino@arm.com> References: <20210115120043.50023-1-vincenzo.frascino@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210115_070058_239319_9D60ACCC X-CRM114-Status: GOOD ( 25.05 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Marco Elver , Catalin Marinas , Branislav Rankov , Alexander Potapenko , Evgenii Stepanov , Andrey Konovalov , Andrey Ryabinin , Vincenzo Frascino , Will Deacon , Dmitry Vyukov Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Architectures supported by KASAN HW can provide a light mode of execution. On an MTE enabled arm64 hw for example this can be identified with the asynch mode of execution. In this mode, if a tag check fault occurs, the TFSR_EL1 register is updated asynchronously. The kernel checks the corresponding bits periodically. KASAN requires a specific mode of execution to make use of this hw feature. Add KASAN HW light execution mode. Note: This patch adds the KASAN_ARG_MODE_LIGHT config option and the "light" kernel command line option to enable the described feature. This patch introduces the kasan_def.h header to make easier to propagate the relevant enumerations to the architectural code. Cc: Dmitry Vyukov Cc: Andrey Ryabinin Cc: Alexander Potapenko Cc: Andrey Konovalov Signed-off-by: Vincenzo Frascino --- arch/arm64/include/asm/memory.h | 2 +- arch/arm64/include/asm/mte-kasan.h | 5 +++-- arch/arm64/kernel/mte.c | 2 +- include/linux/kasan.h | 1 + include/linux/kasan_def.h | 10 ++++++++++ mm/kasan/hw_tags.c | 19 ++++++++++++++++++- mm/kasan/kasan.h | 2 +- 7 files changed, 35 insertions(+), 6 deletions(-) create mode 100644 include/linux/kasan_def.h diff --git a/arch/arm64/include/asm/memory.h b/arch/arm64/include/asm/memory.h index 18fce223b67b..3a7c5beb7096 100644 --- a/arch/arm64/include/asm/memory.h +++ b/arch/arm64/include/asm/memory.h @@ -231,7 +231,7 @@ static inline const void *__tag_set(const void *addr, u8 tag) } #ifdef CONFIG_KASAN_HW_TAGS -#define arch_enable_tagging() mte_enable_kernel() +#define arch_enable_tagging(mode) mte_enable_kernel(mode) #define arch_init_tags(max_tag) mte_init_tags(max_tag) #define arch_get_random_tag() mte_get_random_tag() #define arch_get_mem_tag(addr) mte_get_mem_tag(addr) diff --git a/arch/arm64/include/asm/mte-kasan.h b/arch/arm64/include/asm/mte-kasan.h index 26349a4b5e2e..5402f4c8e88d 100644 --- a/arch/arm64/include/asm/mte-kasan.h +++ b/arch/arm64/include/asm/mte-kasan.h @@ -9,6 +9,7 @@ #ifndef __ASSEMBLY__ +#include #include /* @@ -29,7 +30,7 @@ u8 mte_get_mem_tag(void *addr); u8 mte_get_random_tag(void); void *mte_set_mem_tag_range(void *addr, size_t size, u8 tag); -void mte_enable_kernel(void); +void mte_enable_kernel(enum kasan_hw_tags_mode mode); void mte_init_tags(u64 max_tag); #else /* CONFIG_ARM64_MTE */ @@ -52,7 +53,7 @@ static inline void *mte_set_mem_tag_range(void *addr, size_t size, u8 tag) return addr; } -static inline void mte_enable_kernel(void) +static inline void mte_enable_kernel(enum kasan_hw_tags_mode mode) { } diff --git a/arch/arm64/kernel/mte.c b/arch/arm64/kernel/mte.c index dc9ada64feed..53a6d734e29b 100644 --- a/arch/arm64/kernel/mte.c +++ b/arch/arm64/kernel/mte.c @@ -151,7 +151,7 @@ void mte_init_tags(u64 max_tag) write_sysreg_s(SYS_GCR_EL1_RRND | gcr_kernel_excl, SYS_GCR_EL1); } -void mte_enable_kernel(void) +void mte_enable_kernel(enum kasan_hw_tags_mode mode) { /* Enable MTE Sync Mode for EL1. */ sysreg_clear_set(sctlr_el1, SCTLR_ELx_TCF_MASK, SCTLR_ELx_TCF_SYNC); diff --git a/include/linux/kasan.h b/include/linux/kasan.h index 5e0655fb2a6f..026031444217 100644 --- a/include/linux/kasan.h +++ b/include/linux/kasan.h @@ -2,6 +2,7 @@ #ifndef _LINUX_KASAN_H #define _LINUX_KASAN_H +#include #include #include diff --git a/include/linux/kasan_def.h b/include/linux/kasan_def.h new file mode 100644 index 000000000000..0a55400809c9 --- /dev/null +++ b/include/linux/kasan_def.h @@ -0,0 +1,10 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _LINUX_KASAN_DEF_H +#define _LINUX_KASAN_DEF_H + +enum kasan_hw_tags_mode { + KASAN_HW_TAGS_SYNC, + KASAN_HW_TAGS_ASYNC, +}; + +#endif /* _LINUX_KASAN_DEF_H */ diff --git a/mm/kasan/hw_tags.c b/mm/kasan/hw_tags.c index 55bd6f09c70f..6c3b0742f639 100644 --- a/mm/kasan/hw_tags.c +++ b/mm/kasan/hw_tags.c @@ -22,6 +22,7 @@ enum kasan_arg_mode { KASAN_ARG_MODE_DEFAULT, KASAN_ARG_MODE_OFF, + KASAN_ARG_MODE_LIGHT, KASAN_ARG_MODE_PROD, KASAN_ARG_MODE_FULL, }; @@ -60,6 +61,8 @@ static int __init early_kasan_mode(char *arg) if (!strcmp(arg, "off")) kasan_arg_mode = KASAN_ARG_MODE_OFF; + else if (!strcmp(arg, "light")) + kasan_arg_mode = KASAN_ARG_MODE_LIGHT; else if (!strcmp(arg, "prod")) kasan_arg_mode = KASAN_ARG_MODE_PROD; else if (!strcmp(arg, "full")) @@ -105,9 +108,21 @@ static int __init early_kasan_fault(char *arg) } early_param("kasan.fault", early_kasan_fault); +static inline int hw_init_mode(enum kasan_arg_mode mode) +{ + switch (mode) { + case KASAN_ARG_MODE_LIGHT: + return KASAN_HW_TAGS_ASYNC; + default: + return KASAN_HW_TAGS_SYNC; + } +} + /* kasan_init_hw_tags_cpu() is called for each CPU. */ void kasan_init_hw_tags_cpu(void) { + enum kasan_hw_tags_mode hw_mode; + /* * There's no need to check that the hardware is MTE-capable here, * as this function is only called for MTE-capable hardware. @@ -118,7 +133,8 @@ void kasan_init_hw_tags_cpu(void) return; hw_init_tags(KASAN_TAG_MAX); - hw_enable_tagging(); + hw_mode = hw_init_mode(kasan_arg_mode); + hw_enable_tagging(hw_mode); } /* kasan_init_hw_tags() is called once on boot CPU. */ @@ -145,6 +161,7 @@ void __init kasan_init_hw_tags(void) case KASAN_ARG_MODE_OFF: /* If KASAN is disabled, do nothing. */ return; + case KASAN_ARG_MODE_LIGHT: case KASAN_ARG_MODE_PROD: static_branch_enable(&kasan_flag_enabled); break; diff --git a/mm/kasan/kasan.h b/mm/kasan/kasan.h index cc4d9e1d49b1..78c09279327e 100644 --- a/mm/kasan/kasan.h +++ b/mm/kasan/kasan.h @@ -284,7 +284,7 @@ static inline const void *arch_kasan_set_tag(const void *addr, u8 tag) #define arch_set_mem_tag_range(addr, size, tag) ((void *)(addr)) #endif -#define hw_enable_tagging() arch_enable_tagging() +#define hw_enable_tagging(mode) arch_enable_tagging(mode) #define hw_init_tags(max_tag) arch_init_tags(max_tag) #define hw_get_random_tag() arch_get_random_tag() #define hw_get_mem_tag(addr) arch_get_mem_tag(addr) From patchwork Fri Jan 15 12:00:41 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vincenzo Frascino X-Patchwork-Id: 12022353 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.0 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3592DC433E0 for ; Fri, 15 Jan 2021 12:02:35 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id E9F9223772 for ; Fri, 15 Jan 2021 12:02:34 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org E9F9223772 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=arm.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Message-Id:Date: Subject:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=txVAvl/vt+XoJnR80dYTmMWXEkGbwcJn0TGC+DDYWjU=; b=omsNNHqAT7efqt8BtG14tAtCP UsR1VLnqU5+Oaz7Y6Tgndp0VtaQy+CcNHpdfO0/5MGZRTF8s5ZDewtpGpIieMkM4nH7l7dFOSllQn +/rlArreCQDuPCMqep6abaYaOWllsrps+S6U7cbeXrmi6kY71gOgUQWLzDh8yGBRCbYEonimFFr// 6wmu70ZSRdmlkv8SAxECiujP7WTXgowjZO+Cj+NEdpR+Hgo3lRt4SbTXscyhZXFFKB1U2ep+QidQO gwy3UcrX71+kYKcsL/1Hx6Nln/vnU6SKWaVmJF41O8FJTEGzmEZEbMxqP0ZvAejSaKpXLlxj+DS/v U5O2NsHvA==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1l0Nmp-0004zw-EX; Fri, 15 Jan 2021 12:01:04 +0000 Received: from foss.arm.com ([217.140.110.172]) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1l0Nmj-0004xd-PN for linux-arm-kernel@lists.infradead.org; Fri, 15 Jan 2021 12:00:58 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 380D311FB; Fri, 15 Jan 2021 04:00:57 -0800 (PST) Received: from e119884-lin.cambridge.arm.com (e119884-lin.cambridge.arm.com [10.1.196.72]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 8C7413F70D; Fri, 15 Jan 2021 04:00:55 -0800 (PST) From: Vincenzo Frascino To: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kasan-dev@googlegroups.com Subject: [PATCH v3 2/4] arm64: mte: Add asynchronous mode support Date: Fri, 15 Jan 2021 12:00:41 +0000 Message-Id: <20210115120043.50023-3-vincenzo.frascino@arm.com> X-Mailer: git-send-email 2.30.0 In-Reply-To: <20210115120043.50023-1-vincenzo.frascino@arm.com> References: <20210115120043.50023-1-vincenzo.frascino@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210115_070057_878668_1CC6C587 X-CRM114-Status: GOOD ( 14.22 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Marco Elver , Catalin Marinas , Branislav Rankov , Alexander Potapenko , Evgenii Stepanov , Andrey Konovalov , Andrey Ryabinin , Vincenzo Frascino , Will Deacon , Dmitry Vyukov Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org MTE provides an asynchronous mode for detecting tag exceptions. In particular instead of triggering a fault the arm64 core updates a register which is checked by the kernel after the asynchronous tag check fault has occurred. Add support for MTE asynchronous mode. The exception handling mechanism will be added with a future patch. Note: KASAN HW activates async mode via kasan.mode kernel parameter. The default mode is set to synchronous. The code that verifies the status of TFSR_EL1 will be added with a future patch. Cc: Catalin Marinas Cc: Will Deacon Signed-off-by: Vincenzo Frascino --- arch/arm64/kernel/mte.c | 26 ++++++++++++++++++++++++-- 1 file changed, 24 insertions(+), 2 deletions(-) diff --git a/arch/arm64/kernel/mte.c b/arch/arm64/kernel/mte.c index 53a6d734e29b..df7a1ae26d7c 100644 --- a/arch/arm64/kernel/mte.c +++ b/arch/arm64/kernel/mte.c @@ -153,8 +153,30 @@ void mte_init_tags(u64 max_tag) void mte_enable_kernel(enum kasan_hw_tags_mode mode) { - /* Enable MTE Sync Mode for EL1. */ - sysreg_clear_set(sctlr_el1, SCTLR_ELx_TCF_MASK, SCTLR_ELx_TCF_SYNC); + const char *m; + + /* Preset parameter values based on the mode. */ + switch (mode) { + case KASAN_HW_TAGS_ASYNC: + /* Enable MTE Async Mode for EL1. */ + sysreg_clear_set(sctlr_el1, SCTLR_ELx_TCF_MASK, SCTLR_ELx_TCF_ASYNC); + m = "asynchronous"; + break; + case KASAN_HW_TAGS_SYNC: + sysreg_clear_set(sctlr_el1, SCTLR_ELx_TCF_MASK, SCTLR_ELx_TCF_SYNC); + m = "synchronous"; + break; + default: + /* + * kasan mode should be always set hence we should + * not reach this condition. + */ + WARN_ON_ONCE(1); + return; + } + + pr_info_once("MTE: enabled in %s mode at EL1\n", m); + isb(); } From patchwork Fri Jan 15 12:00:42 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vincenzo Frascino X-Patchwork-Id: 12022359 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.0 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D0004C433E6 for ; Fri, 15 Jan 2021 12:02:43 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 9222623128 for ; Fri, 15 Jan 2021 12:02:43 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 9222623128 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=arm.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Message-Id:Date: Subject:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=TuHER3JqTkw+AUd7/mvjLThtt/MpQp7DpcGj6JaFdFk=; b=x6kIrw0/ZPLtzMLo6dPWaUhzO Nceg2WrLiWKhrmOMpiwxci3/eHQAbptEiCEI3r972W+ne6KduqS0lUFoyUSuhGIKFYJ8rya+Pv/8N +FsvSlZ4rSzOen5SrYO+pqcR6t6rwWLCFtAvcSRm1HiKDsF8OHrB3/Wuy9uwPQhX31Ad8KB8BlCMn YEApStLiRO/8LSNJFpa6meRxU8kPoAbJ3kWYz3vQQZU7d75RhIU5PztQRkMIJm+aT+r2h1Vn4XQeP x3A38eEYFJtvZY+YTmAlG9PgGEFnIvxkNrJJsBb7PGu61Sd5cKVgDn1Mw9DapuaBhC1oHFQakWvXf ZTrzc/cIg==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1l0Nmw-000515-64; Fri, 15 Jan 2021 12:01:10 +0000 Received: from foss.arm.com ([217.140.110.172]) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1l0Nml-0004xd-KW for linux-arm-kernel@lists.infradead.org; Fri, 15 Jan 2021 12:01:01 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 17C8C12FC; Fri, 15 Jan 2021 04:00:59 -0800 (PST) Received: from e119884-lin.cambridge.arm.com (e119884-lin.cambridge.arm.com [10.1.196.72]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 6D3303F70D; Fri, 15 Jan 2021 04:00:57 -0800 (PST) From: Vincenzo Frascino To: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kasan-dev@googlegroups.com Subject: [PATCH v3 3/4] arm64: mte: Enable async tag check fault Date: Fri, 15 Jan 2021 12:00:42 +0000 Message-Id: <20210115120043.50023-4-vincenzo.frascino@arm.com> X-Mailer: git-send-email 2.30.0 In-Reply-To: <20210115120043.50023-1-vincenzo.frascino@arm.com> References: <20210115120043.50023-1-vincenzo.frascino@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210115_070059_810846_C03246D8 X-CRM114-Status: GOOD ( 21.36 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Marco Elver , Catalin Marinas , Branislav Rankov , Alexander Potapenko , Evgenii Stepanov , Andrey Konovalov , Andrey Ryabinin , Vincenzo Frascino , Will Deacon , Dmitry Vyukov Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org MTE provides a mode that asynchronously updates the TFSR_EL1 register when a tag check exception is detected. To take advantage of this mode the kernel has to verify the status of the register at: 1. Context switching 2. Return to user/EL0 (Not required in entry from EL0 since the kernel did not run) 3. Kernel entry from EL1 4. Kernel exit to EL1 If the register is non-zero a trace is reported. Add the required features for EL1 detection and reporting. Note: ITFSB bit is set in the SCTLR_EL1 register hence it guaranties that the indirect writes to TFSR_EL1 are synchronized at exception entry to EL1. On the context switch path the synchronization is guarantied by the dsb() in __switch_to(). Cc: Catalin Marinas Cc: Will Deacon Signed-off-by: Vincenzo Frascino --- arch/arm64/include/asm/mte.h | 21 +++++++++++++++++++ arch/arm64/kernel/entry-common.c | 11 ++++++++++ arch/arm64/kernel/mte.c | 35 ++++++++++++++++++++++++++++++++ 3 files changed, 67 insertions(+) diff --git a/arch/arm64/include/asm/mte.h b/arch/arm64/include/asm/mte.h index d02aff9f493d..1a715963d909 100644 --- a/arch/arm64/include/asm/mte.h +++ b/arch/arm64/include/asm/mte.h @@ -92,5 +92,26 @@ static inline void mte_assign_mem_tag_range(void *addr, size_t size) #endif /* CONFIG_ARM64_MTE */ +#ifdef CONFIG_KASAN_HW_TAGS +void mte_check_tfsr_el1_no_sync(void); +static inline void mte_check_tfsr_el1(void) +{ + mte_check_tfsr_el1_no_sync(); + /* + * The asynchronous faults are synch'ed automatically with + * TFSR_EL1 on kernel entry but for exit an explicit dsb() + * is required. + */ + dsb(ish); +} +#else +static inline void mte_check_tfsr_el1_no_sync(void) +{ +} +static inline void mte_check_tfsr_el1(void) +{ +} +#endif /* CONFIG_KASAN_HW_TAGS */ + #endif /* __ASSEMBLY__ */ #endif /* __ASM_MTE_H */ diff --git a/arch/arm64/kernel/entry-common.c b/arch/arm64/kernel/entry-common.c index 5346953e4382..c6dfe8a525b0 100644 --- a/arch/arm64/kernel/entry-common.c +++ b/arch/arm64/kernel/entry-common.c @@ -37,6 +37,8 @@ static void noinstr enter_from_kernel_mode(struct pt_regs *regs) lockdep_hardirqs_off(CALLER_ADDR0); rcu_irq_enter_check_tick(); trace_hardirqs_off_finish(); + + mte_check_tfsr_el1_no_sync(); } /* @@ -47,6 +49,13 @@ static void noinstr exit_to_kernel_mode(struct pt_regs *regs) { lockdep_assert_irqs_disabled(); + /* + * The dsb() in mte_check_tfsr_el1() is required to relate + * the asynchronous tag check fault to the context in which + * it happens. + */ + mte_check_tfsr_el1(); + if (interrupts_enabled(regs)) { if (regs->exit_rcu) { trace_hardirqs_on_prepare(); @@ -243,6 +252,8 @@ asmlinkage void noinstr enter_from_user_mode(void) asmlinkage void noinstr exit_to_user_mode(void) { + mte_check_tfsr_el1(); + trace_hardirqs_on_prepare(); lockdep_hardirqs_on_prepare(CALLER_ADDR0); user_enter_irqoff(); diff --git a/arch/arm64/kernel/mte.c b/arch/arm64/kernel/mte.c index df7a1ae26d7c..6cb92e9d6ad1 100644 --- a/arch/arm64/kernel/mte.c +++ b/arch/arm64/kernel/mte.c @@ -180,6 +180,32 @@ void mte_enable_kernel(enum kasan_hw_tags_mode mode) isb(); } +#ifdef CONFIG_KASAN_HW_TAGS +void mte_check_tfsr_el1_no_sync(void) +{ + u64 tfsr_el1; + + if (!system_supports_mte()) + return; + + tfsr_el1 = read_sysreg_s(SYS_TFSR_EL1); + + /* + * The kernel should never hit the condition TF0 == 1 + * at this point because for the futex code we set + * PSTATE.TCO. + */ + WARN_ON(tfsr_el1 & SYS_TFSR_EL1_TF0); + + if (tfsr_el1 & SYS_TFSR_EL1_TF1) { + write_sysreg_s(0, SYS_TFSR_EL1); + isb(); + + pr_err("MTE: Asynchronous tag exception detected!"); + } +} +#endif + static void update_sctlr_el1_tcf0(u64 tcf0) { /* ISB required for the kernel uaccess routines */ @@ -245,6 +271,15 @@ void mte_thread_switch(struct task_struct *next) /* avoid expensive SCTLR_EL1 accesses if no change */ if (current->thread.sctlr_tcf0 != next->thread.sctlr_tcf0) update_sctlr_el1_tcf0(next->thread.sctlr_tcf0); + + /* + * Check if an async tag exception occurred at EL1. + * + * Note: On the context switch path we rely on the dsb() present + * in __switch_to() to guarantee that the indirect writes to TFSR_EL1 + * are synchronized before this point. + */ + mte_check_tfsr_el1_no_sync(); } void mte_suspend_exit(void) From patchwork Fri Jan 15 12:00:43 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vincenzo Frascino X-Patchwork-Id: 12022361 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.0 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6F2E2C433E0 for ; Fri, 15 Jan 2021 12:02:54 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 2D4C523128 for ; Fri, 15 Jan 2021 12:02:54 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 2D4C523128 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=arm.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Message-Id:Date: Subject:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=nclfLYzMpOUJXrQ659Pg5XxaKP9LsRBD1/OEQSBpfGA=; b=gZ/YJNO7vqGJOdvKjhXwWd/yw /CxyP7gGCHiXc8T2kWZUnBZegdNeF+gx0mPArplEh/M8C70Dy60tFg7/unz8Eusxs8kRq9wPM9tD7 yuTd9SYLqTRxtPhMB9uMYt+8C/QwmiCZW43mJBffERZ7kV9vrXQus34as3vkL1pcJEMdV8/cNeVCU wFn6PiSI0bO5O17HdcCK9w9WcrR3PQR0sDmrvMzgMAvmVFcGlnlKi90wYW5F1U6ezjbLVQ2App+Di Z7a/h7keICfcdOCM37++cvqikksZGk4bVXRvTTtjpXELoJf1ZZX77el2WhSdUAWQMmFQtg5P/KYyh alXgyI4Rg==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1l0Nn0-00051p-4C; Fri, 15 Jan 2021 12:01:14 +0000 Received: from foss.arm.com ([217.140.110.172]) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1l0Nmn-0004xd-Gg for linux-arm-kernel@lists.infradead.org; Fri, 15 Jan 2021 12:01:02 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id EBA281396; Fri, 15 Jan 2021 04:01:00 -0800 (PST) Received: from e119884-lin.cambridge.arm.com (e119884-lin.cambridge.arm.com [10.1.196.72]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 4CC683F70D; Fri, 15 Jan 2021 04:00:59 -0800 (PST) From: Vincenzo Frascino To: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kasan-dev@googlegroups.com Subject: [PATCH v3 4/4] arm64: mte: Optimize mte_assign_mem_tag_range() Date: Fri, 15 Jan 2021 12:00:43 +0000 Message-Id: <20210115120043.50023-5-vincenzo.frascino@arm.com> X-Mailer: git-send-email 2.30.0 In-Reply-To: <20210115120043.50023-1-vincenzo.frascino@arm.com> References: <20210115120043.50023-1-vincenzo.frascino@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210115_070101_671284_48AC53B5 X-CRM114-Status: GOOD ( 12.53 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Marco Elver , Catalin Marinas , Branislav Rankov , Alexander Potapenko , Evgenii Stepanov , Andrey Konovalov , Andrey Ryabinin , Vincenzo Frascino , Will Deacon , Dmitry Vyukov Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org mte_assign_mem_tag_range() is called on production KASAN HW hot paths. It makes sense to optimize it in an attempt to reduce the overhead. Optimize mte_assign_mem_tag_range() based on the indications provided at [1]. [1] https://lore.kernel.org/r/CAAeHK+wCO+J7D1_T89DG+jJrPLk3X9RsGFKxJGd0ZcUFjQT-9Q@mail.gmail.com/ Cc: Catalin Marinas Cc: Will Deacon Signed-off-by: Vincenzo Frascino --- arch/arm64/include/asm/mte.h | 26 +++++++++++++++++++++++++- arch/arm64/lib/mte.S | 15 --------------- 2 files changed, 25 insertions(+), 16 deletions(-) diff --git a/arch/arm64/include/asm/mte.h b/arch/arm64/include/asm/mte.h index 1a715963d909..9730f2b07b79 100644 --- a/arch/arm64/include/asm/mte.h +++ b/arch/arm64/include/asm/mte.h @@ -49,7 +49,31 @@ long get_mte_ctrl(struct task_struct *task); int mte_ptrace_copy_tags(struct task_struct *child, long request, unsigned long addr, unsigned long data); -void mte_assign_mem_tag_range(void *addr, size_t size); +static inline void mte_assign_mem_tag_range(void *addr, size_t size) +{ + u64 _addr = (u64)addr; + u64 _end = _addr + size; + + /* + * This function must be invoked from an MTE enabled context. + * + * Note: The address must be non-NULL and MTE_GRANULE_SIZE aligned and + * size must be non-zero and MTE_GRANULE_SIZE aligned. + */ + do { + /* + * 'asm volatile' is required to prevent the compiler to move + * the statement outside of the loop. + */ + asm volatile(__MTE_PREAMBLE "stg %0, [%0]" + : + : "r" (_addr) + : "memory"); + + _addr += MTE_GRANULE_SIZE; + } while (_addr < _end); +} + #else /* CONFIG_ARM64_MTE */ diff --git a/arch/arm64/lib/mte.S b/arch/arm64/lib/mte.S index 9e1a12e10053..a0a650451510 100644 --- a/arch/arm64/lib/mte.S +++ b/arch/arm64/lib/mte.S @@ -150,18 +150,3 @@ SYM_FUNC_START(mte_restore_page_tags) ret SYM_FUNC_END(mte_restore_page_tags) -/* - * Assign allocation tags for a region of memory based on the pointer tag - * x0 - source pointer - * x1 - size - * - * Note: The address must be non-NULL and MTE_GRANULE_SIZE aligned and - * size must be non-zero and MTE_GRANULE_SIZE aligned. - */ -SYM_FUNC_START(mte_assign_mem_tag_range) -1: stg x0, [x0] - add x0, x0, #MTE_GRANULE_SIZE - subs x1, x1, #MTE_GRANULE_SIZE - b.gt 1b - ret -SYM_FUNC_END(mte_assign_mem_tag_range)