From patchwork Tue Mar 9 13:24:35 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrey Konovalov X-Patchwork-Id: 12125285 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.0 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E6C77C433DB for ; Tue, 9 Mar 2021 13:42:52 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 0CB13600CD for ; Tue, 9 Mar 2021 13:42:52 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 0CB13600CD Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:Cc:To:From:Subject:References:Mime-Version: Message-Id:In-Reply-To:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=pPTgl43ZxWCgtEsYKmMRyYal38ZvNefDMbB8KQDtu54=; b=hq+FHJgFLzafQP YAbD/b6DIjFneOAmJNqznFHuJejDt4T6sWV55WrQFD3xdsKL3SkfFrT0gQezPadscHf5RoQbBMHLh tRoaFdElCJEL3gtdgglpgnCY/1IvQtBicfP8jBa2nAflm2QFMqQWALZz4oIw6ssyBnNs5SXt3CRJk g/U3T7xlizDdsGQUpXmKIeO4qoKBNzQHGA8nbdRfiTVvP2fNCq12XzxoVXhltrnK2iDLsBLH9iUJ3 yjFqiEqyY881Kpj2xhb+SEsSL1FcaMbiezf55qw/Pkai7BfKLH30TKzMJulhqvb4DFGdoCkXQXyKv mSbToWI3CkBpFTx22v0A==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lJcbC-004hcN-SV; Tue, 09 Mar 2021 13:40:35 +0000 Received: from mail-wm1-x34a.google.com ([2a00:1450:4864:20::34a]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lJcLu-004bny-Ml for linux-arm-kernel@lists.infradead.org; Tue, 09 Mar 2021 13:24:52 +0000 Received: by mail-wm1-x34a.google.com with SMTP id c7so1132891wml.8 for ; Tue, 09 Mar 2021 05:24:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=l+UpHCxZ2NwdA6Wfc2QV4OtHQWAx+WJePRBQJIJEiJo=; b=l34afbqAvPQXcjzsfMShqnGlvlF2i0RtNWfxzKU8zpaVbXnzK9SRfNwVdu0eCO9Vr2 29XkE6R0aWdcnI3YWGOvZ2uJmjeDtfV/PQ8AToJTTmWl+YTzA9fOcnXeKkL/gZ0mNicn 3b3maekCEU0R8ffe1IkKR8PlFHB+iC6ng2btK+k1npxcqAEDFLKcWQPaPNdP7d+YENGU 1ddiVRNoqZ6xfvBdEH23T5LMnnF3XA5PD8FqcTHNMnAmH6L285I2x89IER2tN2qX6p09 uQJzSMnhAqgqTiNckRmECktMPneOakeNkGTfrCvMfsQvtJrTXALafVZMXRxp45+XVgX/ TX3Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=l+UpHCxZ2NwdA6Wfc2QV4OtHQWAx+WJePRBQJIJEiJo=; b=fJ/hl0kvI9HPAFvqqHjdKCoKSFpKV4ragu69sRYrzeFPQ4+9V/dZzROxjXJK6FuWji 4s1ywy1PWkfwxCfhLk5ha9BASS4EFa6oDf6cpl2dNNxTNl2txnd9jGzgNkU1N6+6mRek VbjfEU/bH9Uq78CikdmjVugXgDMCclcmVXq770KT4Wd/urae/+FWBdXbhP8tvXlQCUy0 KbXWeHaiyTxqolYK7eQZsuG3WOvCEClVFiwG+kI1ttxDJcI3K3AhVs4WwkHa0rANJGA1 p9LZDiqyhU/iPBHJHlywhRQOyUiBfotQbVxg4Z/KNZl/0sjV8xrVhXk/vTpC101HjY2S 80vg== X-Gm-Message-State: AOAM530gQXfL01nuP5lKLABFgV7krS3+5V4cs/g9kxpH7Cl6sRBxh3It 4tq62TvIoK3DZ8ow1xb3s3fQiOzYRz2iMyhw X-Google-Smtp-Source: ABdhPJzeJ3O/vPx4Qw7ptEvZYassi6QjjMCwgCMDLJKe6O3hz3UbTW/Oc5x1nCdywjwdeW0BcIqpezuZkydvIVQY X-Received: from andreyknvl3.muc.corp.google.com ([2a00:79e0:15:13:5802:818:ce92:dfef]) (user=andreyknvl job=sendgmr) by 2002:a5d:6810:: with SMTP id w16mr28447553wru.333.1615296285067; Tue, 09 Mar 2021 05:24:45 -0800 (PST) Date: Tue, 9 Mar 2021 14:24:35 +0100 In-Reply-To: Message-Id: Mime-Version: 1.0 References: X-Mailer: git-send-email 2.30.1.766.gb4fecdf3b7-goog Subject: [PATCH v3 1/5] arm64: kasan: allow to init memory when setting tags From: Andrey Konovalov To: Andrew Morton , Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , Vlastimil Babka , Catalin Marinas Cc: Will Deacon , Vincenzo Frascino , Dmitry Vyukov , Andrey Ryabinin , Alexander Potapenko , Marco Elver , Peter Collingbourne , Evgenii Stepanov , Branislav Rankov , Kevin Brodsky , kasan-dev@googlegroups.com, linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Andrey Konovalov X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210309_132446_935202_9D29534B X-CRM114-Status: GOOD ( 16.69 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org This change adds an argument to mte_set_mem_tag_range() that allows to enable memory initialization when settinh the allocation tags. The implementation uses stzg instruction instead of stg when this argument indicates to initialize memory. Combining setting allocation tags with memory initialization will improve HW_TAGS KASAN performance when init_on_alloc/free is enabled. This change doesn't integrate memory initialization with KASAN, this is done is subsequent patches in this series. Acked-by: Marco Elver Signed-off-by: Andrey Konovalov --- arch/arm64/include/asm/memory.h | 4 +-- arch/arm64/include/asm/mte-kasan.h | 39 +++++++++++++++++++----------- mm/kasan/kasan.h | 9 ++++--- 3 files changed, 32 insertions(+), 20 deletions(-) diff --git a/arch/arm64/include/asm/memory.h b/arch/arm64/include/asm/memory.h index c759faf7a1ff..f1ba48b4347d 100644 --- a/arch/arm64/include/asm/memory.h +++ b/arch/arm64/include/asm/memory.h @@ -248,8 +248,8 @@ static inline const void *__tag_set(const void *addr, u8 tag) #define arch_init_tags(max_tag) mte_init_tags(max_tag) #define arch_get_random_tag() mte_get_random_tag() #define arch_get_mem_tag(addr) mte_get_mem_tag(addr) -#define arch_set_mem_tag_range(addr, size, tag) \ - mte_set_mem_tag_range((addr), (size), (tag)) +#define arch_set_mem_tag_range(addr, size, tag, init) \ + mte_set_mem_tag_range((addr), (size), (tag), (init)) #endif /* CONFIG_KASAN_HW_TAGS */ /* diff --git a/arch/arm64/include/asm/mte-kasan.h b/arch/arm64/include/asm/mte-kasan.h index 7ab500e2ad17..570af3e99296 100644 --- a/arch/arm64/include/asm/mte-kasan.h +++ b/arch/arm64/include/asm/mte-kasan.h @@ -53,7 +53,8 @@ static inline u8 mte_get_random_tag(void) * Note: The address must be non-NULL and MTE_GRANULE_SIZE aligned and * size must be non-zero and MTE_GRANULE_SIZE aligned. */ -static inline void mte_set_mem_tag_range(void *addr, size_t size, u8 tag) +static inline void mte_set_mem_tag_range(void *addr, size_t size, + u8 tag, bool init) { u64 curr, end; @@ -63,18 +64,27 @@ static inline void mte_set_mem_tag_range(void *addr, size_t size, u8 tag) curr = (u64)__tag_set(addr, tag); end = curr + size; - do { - /* - * 'asm volatile' is required to prevent the compiler to move - * the statement outside of the loop. - */ - asm volatile(__MTE_PREAMBLE "stg %0, [%0]" - : - : "r" (curr) - : "memory"); - - curr += MTE_GRANULE_SIZE; - } while (curr != end); + /* + * 'asm volatile' is required to prevent the compiler to move + * the statement outside of the loop. + */ + if (init) { + do { + asm volatile(__MTE_PREAMBLE "stzg %0, [%0]" + : + : "r" (curr) + : "memory"); + curr += MTE_GRANULE_SIZE; + } while (curr != end); + } else { + do { + asm volatile(__MTE_PREAMBLE "stg %0, [%0]" + : + : "r" (curr) + : "memory"); + curr += MTE_GRANULE_SIZE; + } while (curr != end); + } } void mte_enable_kernel(void); @@ -100,7 +110,8 @@ static inline u8 mte_get_random_tag(void) return 0xFF; } -static inline void mte_set_mem_tag_range(void *addr, size_t size, u8 tag) +static inline void mte_set_mem_tag_range(void *addr, size_t size, + u8 tag, bool init) { } diff --git a/mm/kasan/kasan.h b/mm/kasan/kasan.h index 8c55634d6edd..7fbb32234414 100644 --- a/mm/kasan/kasan.h +++ b/mm/kasan/kasan.h @@ -291,7 +291,7 @@ static inline const void *arch_kasan_set_tag(const void *addr, u8 tag) #define arch_get_mem_tag(addr) (0xFF) #endif #ifndef arch_set_mem_tag_range -#define arch_set_mem_tag_range(addr, size, tag) ((void *)(addr)) +#define arch_set_mem_tag_range(addr, size, tag, init) ((void *)(addr)) #endif #define hw_enable_tagging() arch_enable_tagging() @@ -299,7 +299,8 @@ static inline const void *arch_kasan_set_tag(const void *addr, u8 tag) #define hw_set_tagging_report_once(state) arch_set_tagging_report_once(state) #define hw_get_random_tag() arch_get_random_tag() #define hw_get_mem_tag(addr) arch_get_mem_tag(addr) -#define hw_set_mem_tag_range(addr, size, tag) arch_set_mem_tag_range((addr), (size), (tag)) +#define hw_set_mem_tag_range(addr, size, tag, init) \ + arch_set_mem_tag_range((addr), (size), (tag), (init)) #else /* CONFIG_KASAN_HW_TAGS */ @@ -343,7 +344,7 @@ static inline void kasan_poison(const void *addr, size_t size, u8 value) if (WARN_ON(size & KASAN_GRANULE_MASK)) return; - hw_set_mem_tag_range((void *)addr, size, value); + hw_set_mem_tag_range((void *)addr, size, value, false); } static inline void kasan_unpoison(const void *addr, size_t size) @@ -360,7 +361,7 @@ static inline void kasan_unpoison(const void *addr, size_t size) return; size = round_up(size, KASAN_GRANULE_SIZE); - hw_set_mem_tag_range((void *)addr, size, tag); + hw_set_mem_tag_range((void *)addr, size, tag, false); } static inline bool kasan_byte_accessible(const void *addr)