From patchwork Sat Mar 6 00:15:50 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrey Konovalov X-Patchwork-Id: 12119625 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C563BC433DB for ; Sat, 6 Mar 2021 00:18:15 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 4E9EB650A5 for ; Sat, 6 Mar 2021 00:18:15 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 4E9EB650A5 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:Cc:To:From:Subject:References:Mime-Version: Message-Id:In-Reply-To:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=alVZorLGeOypz+GtFp9ZtZxnVh3spi8ung8xU/VJiNA=; b=JeIDodbTRm+LvF ub5nsCK77RcWNoesCYXt9WBNn/puUWdIv72nDDd13KMcOy1rWlB7Ig7KfOHFjFPN+DGk+MhaXLakA pXHsiHoChLvagXXT12xQdQhPRy2rvLBJXAkb7utAtP489jM/Kp1MeSxuyx1Eg2op0SfmDHHj1ar3N M+AeMgKd6aboLciPau9kvmKoYWbtVBBxh8S156OW4ZnjZZsX2AcXMGcGfJupdUpKJ1rhft7ZqjSeS 1WVN+ca3VN/YsZLJEdLZxzoHdzjO2Cot76A9MZRfGEslMizOJ1tMHRe9R4VMjp8PT13OZVZTEORhV 3yAsXjDnIhryklZJOJpg==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lIKcH-00HMi7-LJ; Sat, 06 Mar 2021 00:16:23 +0000 Received: from mail-wr1-x44a.google.com ([2a00:1450:4864:20::44a]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lIKby-00HMUL-Li for linux-arm-kernel@lists.infradead.org; Sat, 06 Mar 2021 00:16:05 +0000 Received: by mail-wr1-x44a.google.com with SMTP id z17so1763571wrv.23 for ; Fri, 05 Mar 2021 16:16:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=0Q6hAqH6fyVVw5hCkpEOYFgiWfgnj6qL15/X+pD6J2A=; b=WPOHPdaRjB7t3Uf0U9zDkowioAdUfOwgJC8qaBOHrjvyNnXDOnWr4l9V4GzpLuk5Bg Ksl3qYlvWhrt1kC+QU5cT9sA7BU5QjM2q0pv8YfC6rE1nnfnwOOEfk3mBsr38vL0b6F1 T2C99k8heVmKyzmnkaVB3+e6PNlPe7+ajQtwdlt/wO7l5SDFf/rrsBAgAMEKYIXbStv5 4Z/izW73EurNctB3ElHwEzI3/5F0eg4fYX3kGBqqG3WGNBrQmZTO/gZUvMqGnNhhtNgF 55HiX9/lIJF9axO+ccL5psWisLTMhuWgFRcXHSuJNnxeq992E1UFBMRe+GGwY5O44Ris Invw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=0Q6hAqH6fyVVw5hCkpEOYFgiWfgnj6qL15/X+pD6J2A=; b=qswKW/KXAbhTLGgQ34eyc+AZzYCBYVC3BHrWbpYAs9U3GiHovKyleGTu4E6LyH16An 7uVOe//FLEGzNzS6xEFpIFAb+c+dkTlp9TIxXRZXYZYkfJWEp11ub7S5CMSMKcGglvYb hyUpvZwfiUJekzSkX3R5W4DB8px+9sapV7LBcE+qqVyO3pEUlipKeg0cuiWGalEQtQng 9UVUGs9EgMMgXEaOrXYxGX3fOcpsQGGJ6bZn86FQRaLIZKe8C8KNXdx8CF2yLrjYt5av xiRbQz4vOcywtJGVGGzs9IliUOFg5d6aOzKTFHmxjqxwiwiC97p8sR0MGWQNPHpikqeS 03OA== X-Gm-Message-State: AOAM5336YTMj08Awe117WCAKtXEY31SMdH7QZ3DGVLjDfS9rydU+nFNg 1Id16Zf3QkDX/2TzN1ZcK3KFL2e8kmOs/2dL X-Google-Smtp-Source: ABdhPJwZARCNlNlc3fuoGFNT12NWkkP5eTaFIQ4dlC/WXEzeU00FCHc+LFMzZ92olvfufEE1d4OvMvi6aPGBotXJ X-Received: from andreyknvl3.muc.corp.google.com ([2a00:79e0:15:13:953b:d7cf:2b01:f178]) (user=andreyknvl job=sendgmr) by 2002:a05:600c:2109:: with SMTP id u9mr11025313wml.44.1614989761143; Fri, 05 Mar 2021 16:16:01 -0800 (PST) Date: Sat, 6 Mar 2021 01:15:50 +0100 In-Reply-To: Message-Id: Mime-Version: 1.0 References: X-Mailer: git-send-email 2.30.1.766.gb4fecdf3b7-goog Subject: [PATCH 1/5] arm64: kasan: allow to init memory when setting tags From: Andrey Konovalov To: Alexander Potapenko Cc: Andrew Morton , Catalin Marinas , Will Deacon , Vincenzo Frascino , Dmitry Vyukov , Andrey Ryabinin , Marco Elver , Peter Collingbourne , Evgenii Stepanov , Branislav Rankov , Kevin Brodsky , kasan-dev@googlegroups.com, linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Andrey Konovalov X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210306_001602_938020_5B44AEC3 X-CRM114-Status: GOOD ( 13.87 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org This change adds an argument to mte_set_mem_tag_range() that allows to enable memory initialization when settinh the allocation tags. The implementation uses stzg instruction instead of stg when this argument indicates to initialize memory. Combining setting allocation tags with memory initialization will improve HW_TAGS KASAN performance when init_on_alloc/free is enabled. This change doesn't integrate memory initialization with KASAN, this is done is subsequent patches in this series. Signed-off-by: Andrey Konovalov Acked-by: Marco Elver --- arch/arm64/include/asm/memory.h | 4 ++-- arch/arm64/include/asm/mte-kasan.h | 20 ++++++++++++++------ mm/kasan/kasan.h | 9 +++++---- 3 files changed, 21 insertions(+), 12 deletions(-) diff --git a/arch/arm64/include/asm/memory.h b/arch/arm64/include/asm/memory.h index c759faf7a1ff..f1ba48b4347d 100644 --- a/arch/arm64/include/asm/memory.h +++ b/arch/arm64/include/asm/memory.h @@ -248,8 +248,8 @@ static inline const void *__tag_set(const void *addr, u8 tag) #define arch_init_tags(max_tag) mte_init_tags(max_tag) #define arch_get_random_tag() mte_get_random_tag() #define arch_get_mem_tag(addr) mte_get_mem_tag(addr) -#define arch_set_mem_tag_range(addr, size, tag) \ - mte_set_mem_tag_range((addr), (size), (tag)) +#define arch_set_mem_tag_range(addr, size, tag, init) \ + mte_set_mem_tag_range((addr), (size), (tag), (init)) #endif /* CONFIG_KASAN_HW_TAGS */ /* diff --git a/arch/arm64/include/asm/mte-kasan.h b/arch/arm64/include/asm/mte-kasan.h index 7ab500e2ad17..35fe549f7ea4 100644 --- a/arch/arm64/include/asm/mte-kasan.h +++ b/arch/arm64/include/asm/mte-kasan.h @@ -53,7 +53,8 @@ static inline u8 mte_get_random_tag(void) * Note: The address must be non-NULL and MTE_GRANULE_SIZE aligned and * size must be non-zero and MTE_GRANULE_SIZE aligned. */ -static inline void mte_set_mem_tag_range(void *addr, size_t size, u8 tag) +static inline void mte_set_mem_tag_range(void *addr, size_t size, + u8 tag, bool init) { u64 curr, end; @@ -68,10 +69,16 @@ static inline void mte_set_mem_tag_range(void *addr, size_t size, u8 tag) * 'asm volatile' is required to prevent the compiler to move * the statement outside of the loop. */ - asm volatile(__MTE_PREAMBLE "stg %0, [%0]" - : - : "r" (curr) - : "memory"); + if (init) + asm volatile(__MTE_PREAMBLE "stzg %0, [%0]" + : + : "r" (curr) + : "memory"); + else + asm volatile(__MTE_PREAMBLE "stg %0, [%0]" + : + : "r" (curr) + : "memory"); curr += MTE_GRANULE_SIZE; } while (curr != end); @@ -100,7 +107,8 @@ static inline u8 mte_get_random_tag(void) return 0xFF; } -static inline void mte_set_mem_tag_range(void *addr, size_t size, u8 tag) +static inline void mte_set_mem_tag_range(void *addr, size_t size, + u8 tag, bool init) { } diff --git a/mm/kasan/kasan.h b/mm/kasan/kasan.h index 8c55634d6edd..7fbb32234414 100644 --- a/mm/kasan/kasan.h +++ b/mm/kasan/kasan.h @@ -291,7 +291,7 @@ static inline const void *arch_kasan_set_tag(const void *addr, u8 tag) #define arch_get_mem_tag(addr) (0xFF) #endif #ifndef arch_set_mem_tag_range -#define arch_set_mem_tag_range(addr, size, tag) ((void *)(addr)) +#define arch_set_mem_tag_range(addr, size, tag, init) ((void *)(addr)) #endif #define hw_enable_tagging() arch_enable_tagging() @@ -299,7 +299,8 @@ static inline const void *arch_kasan_set_tag(const void *addr, u8 tag) #define hw_set_tagging_report_once(state) arch_set_tagging_report_once(state) #define hw_get_random_tag() arch_get_random_tag() #define hw_get_mem_tag(addr) arch_get_mem_tag(addr) -#define hw_set_mem_tag_range(addr, size, tag) arch_set_mem_tag_range((addr), (size), (tag)) +#define hw_set_mem_tag_range(addr, size, tag, init) \ + arch_set_mem_tag_range((addr), (size), (tag), (init)) #else /* CONFIG_KASAN_HW_TAGS */ @@ -343,7 +344,7 @@ static inline void kasan_poison(const void *addr, size_t size, u8 value) if (WARN_ON(size & KASAN_GRANULE_MASK)) return; - hw_set_mem_tag_range((void *)addr, size, value); + hw_set_mem_tag_range((void *)addr, size, value, false); } static inline void kasan_unpoison(const void *addr, size_t size) @@ -360,7 +361,7 @@ static inline void kasan_unpoison(const void *addr, size_t size) return; size = round_up(size, KASAN_GRANULE_SIZE); - hw_set_mem_tag_range((void *)addr, size, tag); + hw_set_mem_tag_range((void *)addr, size, tag, false); } static inline bool kasan_byte_accessible(const void *addr)