From patchwork Tue Mar 9 13:24:35 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrey Konovalov X-Patchwork-Id: 12125217 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 94D47C433E0 for ; Tue, 9 Mar 2021 13:24:48 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id EC5C864D8F for ; Tue, 9 Mar 2021 13:24:47 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org EC5C864D8F Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 8944C8D00F0; Tue, 9 Mar 2021 08:24:47 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 86B2A8D007F; Tue, 9 Mar 2021 08:24:47 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6E4EC8D00F0; Tue, 9 Mar 2021 08:24:47 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0028.hostedemail.com [216.40.44.28]) by kanga.kvack.org (Postfix) with ESMTP id 553CF8D007F for ; Tue, 9 Mar 2021 08:24:47 -0500 (EST) Received: from smtpin20.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 173FB180AD81A for ; Tue, 9 Mar 2021 13:24:47 +0000 (UTC) X-FDA: 77900405814.20.1EB61D6 Received: from mail-wr1-f73.google.com (mail-wr1-f73.google.com [209.85.221.73]) by imf26.hostedemail.com (Postfix) with ESMTP id 5E1B940001DE for ; Tue, 9 Mar 2021 13:24:44 +0000 (UTC) Received: by mail-wr1-f73.google.com with SMTP id f3so6410720wrt.14 for ; Tue, 09 Mar 2021 05:24:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=l+UpHCxZ2NwdA6Wfc2QV4OtHQWAx+WJePRBQJIJEiJo=; b=l34afbqAvPQXcjzsfMShqnGlvlF2i0RtNWfxzKU8zpaVbXnzK9SRfNwVdu0eCO9Vr2 29XkE6R0aWdcnI3YWGOvZ2uJmjeDtfV/PQ8AToJTTmWl+YTzA9fOcnXeKkL/gZ0mNicn 3b3maekCEU0R8ffe1IkKR8PlFHB+iC6ng2btK+k1npxcqAEDFLKcWQPaPNdP7d+YENGU 1ddiVRNoqZ6xfvBdEH23T5LMnnF3XA5PD8FqcTHNMnAmH6L285I2x89IER2tN2qX6p09 uQJzSMnhAqgqTiNckRmECktMPneOakeNkGTfrCvMfsQvtJrTXALafVZMXRxp45+XVgX/ TX3Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=l+UpHCxZ2NwdA6Wfc2QV4OtHQWAx+WJePRBQJIJEiJo=; b=l6uNgA9s3MBxznJZPq0zOXwTss6We7P1EvAJKyF5jm4+k03wNeHrs43OcRrrusnJmQ HPXAesvtXB+QF+eoENBusqYxr8GVP1TVp9rmIePa4bpt9D2E3SGaowybhIQiX7yklWFW Uewl00Okap5lihJS6mVTzBfcmetsP8JvtyXIVhO7YPoCfAYMUvCxT9j/FmVSx7aF7U1h 9bHsebH2avf2KiHhYmgUgaYv/IFhErEb/W7c7NaQYO7GHsiWI7uyIg6Q42OZlOuzR8cI xEprDl+tqILmmtwjqjt8P8PRdHoUvLT3zQEtpQjEkKUQkxCXmObz6rgIXzavdRGxE6vG xF9Q== X-Gm-Message-State: AOAM530cNnA62u9/XmJVMewlWiW6ZOTKHpZvP9PTNohYgGZAbJQiw11/ FkKbeh7rNXyKi9cwGtNJjpl9Ah33SduQ4DPY X-Google-Smtp-Source: ABdhPJzeJ3O/vPx4Qw7ptEvZYassi6QjjMCwgCMDLJKe6O3hz3UbTW/Oc5x1nCdywjwdeW0BcIqpezuZkydvIVQY X-Received: from andreyknvl3.muc.corp.google.com ([2a00:79e0:15:13:5802:818:ce92:dfef]) (user=andreyknvl job=sendgmr) by 2002:a5d:6810:: with SMTP id w16mr28447553wru.333.1615296285067; Tue, 09 Mar 2021 05:24:45 -0800 (PST) Date: Tue, 9 Mar 2021 14:24:35 +0100 In-Reply-To: Message-Id: Mime-Version: 1.0 References: X-Mailer: git-send-email 2.30.1.766.gb4fecdf3b7-goog Subject: [PATCH v3 1/5] arm64: kasan: allow to init memory when setting tags From: Andrey Konovalov To: Andrew Morton , Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , Vlastimil Babka , Catalin Marinas Cc: Will Deacon , Vincenzo Frascino , Dmitry Vyukov , Andrey Ryabinin , Alexander Potapenko , Marco Elver , Peter Collingbourne , Evgenii Stepanov , Branislav Rankov , Kevin Brodsky , kasan-dev@googlegroups.com, linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Andrey Konovalov X-Stat-Signature: xrpidnr9awr1nyoex3bug1qo9f4sfmn1 X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 5E1B940001DE Received-SPF: none (flex--andreyknvl.bounces.google.com>: No applicable sender policy available) receiver=imf26; identity=mailfrom; envelope-from="<3HXdHYAoKCDIObRfSmYbjZUccUZS.QcaZWbil-aaYjOQY.cfU@flex--andreyknvl.bounces.google.com>"; helo=mail-wr1-f73.google.com; client-ip=209.85.221.73 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1615296284-530488 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This change adds an argument to mte_set_mem_tag_range() that allows to enable memory initialization when settinh the allocation tags. The implementation uses stzg instruction instead of stg when this argument indicates to initialize memory. Combining setting allocation tags with memory initialization will improve HW_TAGS KASAN performance when init_on_alloc/free is enabled. This change doesn't integrate memory initialization with KASAN, this is done is subsequent patches in this series. Acked-by: Marco Elver Signed-off-by: Andrey Konovalov --- arch/arm64/include/asm/memory.h | 4 +-- arch/arm64/include/asm/mte-kasan.h | 39 +++++++++++++++++++----------- mm/kasan/kasan.h | 9 ++++--- 3 files changed, 32 insertions(+), 20 deletions(-) diff --git a/arch/arm64/include/asm/memory.h b/arch/arm64/include/asm/memory.h index c759faf7a1ff..f1ba48b4347d 100644 --- a/arch/arm64/include/asm/memory.h +++ b/arch/arm64/include/asm/memory.h @@ -248,8 +248,8 @@ static inline const void *__tag_set(const void *addr, u8 tag) #define arch_init_tags(max_tag) mte_init_tags(max_tag) #define arch_get_random_tag() mte_get_random_tag() #define arch_get_mem_tag(addr) mte_get_mem_tag(addr) -#define arch_set_mem_tag_range(addr, size, tag) \ - mte_set_mem_tag_range((addr), (size), (tag)) +#define arch_set_mem_tag_range(addr, size, tag, init) \ + mte_set_mem_tag_range((addr), (size), (tag), (init)) #endif /* CONFIG_KASAN_HW_TAGS */ /* diff --git a/arch/arm64/include/asm/mte-kasan.h b/arch/arm64/include/asm/mte-kasan.h index 7ab500e2ad17..570af3e99296 100644 --- a/arch/arm64/include/asm/mte-kasan.h +++ b/arch/arm64/include/asm/mte-kasan.h @@ -53,7 +53,8 @@ static inline u8 mte_get_random_tag(void) * Note: The address must be non-NULL and MTE_GRANULE_SIZE aligned and * size must be non-zero and MTE_GRANULE_SIZE aligned. */ -static inline void mte_set_mem_tag_range(void *addr, size_t size, u8 tag) +static inline void mte_set_mem_tag_range(void *addr, size_t size, + u8 tag, bool init) { u64 curr, end; @@ -63,18 +64,27 @@ static inline void mte_set_mem_tag_range(void *addr, size_t size, u8 tag) curr = (u64)__tag_set(addr, tag); end = curr + size; - do { - /* - * 'asm volatile' is required to prevent the compiler to move - * the statement outside of the loop. - */ - asm volatile(__MTE_PREAMBLE "stg %0, [%0]" - : - : "r" (curr) - : "memory"); - - curr += MTE_GRANULE_SIZE; - } while (curr != end); + /* + * 'asm volatile' is required to prevent the compiler to move + * the statement outside of the loop. + */ + if (init) { + do { + asm volatile(__MTE_PREAMBLE "stzg %0, [%0]" + : + : "r" (curr) + : "memory"); + curr += MTE_GRANULE_SIZE; + } while (curr != end); + } else { + do { + asm volatile(__MTE_PREAMBLE "stg %0, [%0]" + : + : "r" (curr) + : "memory"); + curr += MTE_GRANULE_SIZE; + } while (curr != end); + } } void mte_enable_kernel(void); @@ -100,7 +110,8 @@ static inline u8 mte_get_random_tag(void) return 0xFF; } -static inline void mte_set_mem_tag_range(void *addr, size_t size, u8 tag) +static inline void mte_set_mem_tag_range(void *addr, size_t size, + u8 tag, bool init) { } diff --git a/mm/kasan/kasan.h b/mm/kasan/kasan.h index 8c55634d6edd..7fbb32234414 100644 --- a/mm/kasan/kasan.h +++ b/mm/kasan/kasan.h @@ -291,7 +291,7 @@ static inline const void *arch_kasan_set_tag(const void *addr, u8 tag) #define arch_get_mem_tag(addr) (0xFF) #endif #ifndef arch_set_mem_tag_range -#define arch_set_mem_tag_range(addr, size, tag) ((void *)(addr)) +#define arch_set_mem_tag_range(addr, size, tag, init) ((void *)(addr)) #endif #define hw_enable_tagging() arch_enable_tagging() @@ -299,7 +299,8 @@ static inline const void *arch_kasan_set_tag(const void *addr, u8 tag) #define hw_set_tagging_report_once(state) arch_set_tagging_report_once(state) #define hw_get_random_tag() arch_get_random_tag() #define hw_get_mem_tag(addr) arch_get_mem_tag(addr) -#define hw_set_mem_tag_range(addr, size, tag) arch_set_mem_tag_range((addr), (size), (tag)) +#define hw_set_mem_tag_range(addr, size, tag, init) \ + arch_set_mem_tag_range((addr), (size), (tag), (init)) #else /* CONFIG_KASAN_HW_TAGS */ @@ -343,7 +344,7 @@ static inline void kasan_poison(const void *addr, size_t size, u8 value) if (WARN_ON(size & KASAN_GRANULE_MASK)) return; - hw_set_mem_tag_range((void *)addr, size, value); + hw_set_mem_tag_range((void *)addr, size, value, false); } static inline void kasan_unpoison(const void *addr, size_t size) @@ -360,7 +361,7 @@ static inline void kasan_unpoison(const void *addr, size_t size) return; size = round_up(size, KASAN_GRANULE_SIZE); - hw_set_mem_tag_range((void *)addr, size, tag); + hw_set_mem_tag_range((void *)addr, size, tag, false); } static inline bool kasan_byte_accessible(const void *addr)