From patchwork Thu Feb 11 15:33:50 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vincenzo Frascino X-Patchwork-Id: 12083545 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.3 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D4311C433E6 for ; Thu, 11 Feb 2021 15:36:02 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 95C8F64E16 for ; Thu, 11 Feb 2021 15:36:02 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 95C8F64E16 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=arm.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Message-Id:Date: Subject:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=JzfbRNertoE4ZgHWwVln+Z4L3zuXivoXhnxmeTEmOs4=; b=KmWoTE6XzRY/AzhwD9eB18WBb sl2kGDul7e7cK8X6q1QJQJsthCsUioTdVlHnSvBRR1rsdmiqbV/FM1HrFsaaXSVHzsm/ijRh2xxg9 VrEEtVsWDULv89/weaYwURCz+IvdsFAwf6zKZ2faCOixtI7yu9X5XZ62pfzrZl/MAjJGnz2lelzDN xqInTG3zryODUZD4TmbBeZwdYPwzDg6+p/sD5foNr6UVZethIl6DbcuN2PkNXG2ew+D6M9uDYLbbs dYhUTFhwsg7rSZnfU2d7V2hsUyKFu71sXZ9SQmYakJonBlzNQ/AsNG4ErAOcnqg8pJallAZEe53o+ WhT4UCEXQ==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1lADzS-0001he-Tx; Thu, 11 Feb 2021 15:34:47 +0000 Received: from foss.arm.com ([217.140.110.172]) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1lADzC-0001cJ-ND for linux-arm-kernel@lists.infradead.org; Thu, 11 Feb 2021 15:34:32 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 078891424; Thu, 11 Feb 2021 07:34:15 -0800 (PST) Received: from e119884-lin.cambridge.arm.com (e119884-lin.cambridge.arm.com [10.1.196.72]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 2182C3F73D; Thu, 11 Feb 2021 07:34:13 -0800 (PST) From: Vincenzo Frascino To: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kasan-dev@googlegroups.com Subject: [PATCH v13 4/7] arm64: mte: Enable TCO in functions that can read beyond buffer limits Date: Thu, 11 Feb 2021 15:33:50 +0000 Message-Id: <20210211153353.29094-5-vincenzo.frascino@arm.com> X-Mailer: git-send-email 2.30.0 In-Reply-To: <20210211153353.29094-1-vincenzo.frascino@arm.com> References: <20210211153353.29094-1-vincenzo.frascino@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210211_103430_921997_199BB1A1 X-CRM114-Status: GOOD ( 16.89 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Marco Elver , Lorenzo Pieralisi , Catalin Marinas , Branislav Rankov , Alexander Potapenko , Evgenii Stepanov , Andrey Konovalov , Andrey Ryabinin , Andrew Morton , Vincenzo Frascino , Will Deacon , Dmitry Vyukov Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org load_unaligned_zeropad() and __get/put_kernel_nofault() functions can read passed some buffer limits which may include some MTE granule with a different tag. When MTE async mode is enable, the load operation crosses the boundaries and the next granule has a different tag the PE sets the TFSR_EL1.TF1 bit as if an asynchronous tag fault is happened. Enable Tag Check Override (TCO) in these functions before the load and disable it afterwards to prevent this to happen. Note: The same condition can be hit in MTE sync mode but we deal with it through the exception handling. In the current implementation, mte_async_mode flag is set only at boot time but in future kasan might acquire some runtime features that that change the mode dynamically, hence we disable it when sync mode is selected for future proof. Cc: Catalin Marinas Cc: Will Deacon Reported-by: Branislav Rankov Tested-by: Branislav Rankov Signed-off-by: Vincenzo Frascino --- arch/arm64/include/asm/uaccess.h | 24 ++++++++++++++++++++++++ arch/arm64/include/asm/word-at-a-time.h | 4 ++++ arch/arm64/kernel/mte.c | 16 ++++++++++++++++ 3 files changed, 44 insertions(+) diff --git a/arch/arm64/include/asm/uaccess.h b/arch/arm64/include/asm/uaccess.h index 0deb88467111..a857f8f82aeb 100644 --- a/arch/arm64/include/asm/uaccess.h +++ b/arch/arm64/include/asm/uaccess.h @@ -188,6 +188,26 @@ static inline void __uaccess_enable_tco(void) ARM64_MTE, CONFIG_KASAN_HW_TAGS)); } +/* Whether the MTE asynchronous mode is enabled. */ +DECLARE_STATIC_KEY_FALSE(mte_async_mode); + +/* + * These functions disable tag checking only if in MTE async mode + * since the sync mode generates exceptions synchronously and the + * nofault or load_unaligned_zeropad can handle them. + */ +static inline void __uaccess_disable_tco_async(void) +{ + if (static_branch_unlikely(&mte_async_mode)) + __uaccess_disable_tco(); +} + +static inline void __uaccess_enable_tco_async(void) +{ + if (static_branch_unlikely(&mte_async_mode)) + __uaccess_enable_tco(); +} + static inline void uaccess_disable_privileged(void) { __uaccess_disable_tco(); @@ -307,8 +327,10 @@ do { \ do { \ int __gkn_err = 0; \ \ + __uaccess_enable_tco_async(); \ __raw_get_mem("ldr", *((type *)(dst)), \ (__force type *)(src), __gkn_err); \ + __uaccess_disable_tco_async(); \ if (unlikely(__gkn_err)) \ goto err_label; \ } while (0) @@ -380,8 +402,10 @@ do { \ do { \ int __pkn_err = 0; \ \ + __uaccess_enable_tco_async(); \ __raw_put_mem("str", *((type *)(src)), \ (__force type *)(dst), __pkn_err); \ + __uaccess_disable_tco_async(); \ if (unlikely(__pkn_err)) \ goto err_label; \ } while(0) diff --git a/arch/arm64/include/asm/word-at-a-time.h b/arch/arm64/include/asm/word-at-a-time.h index 3333950b5909..c62d9fa791aa 100644 --- a/arch/arm64/include/asm/word-at-a-time.h +++ b/arch/arm64/include/asm/word-at-a-time.h @@ -55,6 +55,8 @@ static inline unsigned long load_unaligned_zeropad(const void *addr) { unsigned long ret, offset; + __uaccess_enable_tco_async(); + /* Load word from unaligned pointer addr */ asm( "1: ldr %0, %3\n" @@ -76,6 +78,8 @@ static inline unsigned long load_unaligned_zeropad(const void *addr) : "=&r" (ret), "=&r" (offset) : "r" (addr), "Q" (*(unsigned long *)addr)); + __uaccess_disable_tco_async(); + return ret; } diff --git a/arch/arm64/kernel/mte.c b/arch/arm64/kernel/mte.c index 706b7ab75f31..65ecb86dd886 100644 --- a/arch/arm64/kernel/mte.c +++ b/arch/arm64/kernel/mte.c @@ -26,6 +26,10 @@ u64 gcr_kernel_excl __ro_after_init; static bool report_fault_once = true; +/* Whether the MTE asynchronous mode is enabled. */ +DEFINE_STATIC_KEY_FALSE(mte_async_mode); +EXPORT_SYMBOL_GPL(mte_async_mode); + static void mte_sync_page_tags(struct page *page, pte_t *ptep, bool check_swap) { pte_t old_pte = READ_ONCE(*ptep); @@ -119,12 +123,24 @@ static inline void __mte_enable_kernel(const char *mode, unsigned long tcf) void mte_enable_kernel_sync(void) { __mte_enable_kernel("synchronous", SCTLR_ELx_TCF_SYNC); + + /* + * This function is called on each active smp core at boot + * time, hence we do not need to take cpu_hotplug_lock again. + */ + static_branch_disable_cpuslocked(&mte_async_mode); } EXPORT_SYMBOL_GPL(mte_enable_kernel_sync); void mte_enable_kernel_async(void) { __mte_enable_kernel("asynchronous", SCTLR_ELx_TCF_ASYNC); + + /* + * This function is called on each active smp core at boot + * time, hence we do not need to take cpu_hotplug_lock again. + */ + static_branch_enable_cpuslocked(&mte_async_mode); } EXPORT_SYMBOL_GPL(mte_enable_kernel_async);