From patchwork Mon Mar 8 15:55:14 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrey Konovalov X-Patchwork-Id: 12122865 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.0 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E57D8C433DB for ; Mon, 8 Mar 2021 15:57:09 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 74F6065226 for ; Mon, 8 Mar 2021 15:57:09 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 74F6065226 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:Cc:To:From:Subject:References:Mime-Version: Message-Id:In-Reply-To:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=kAf70lhls5DLJnSSr3wiXveJCDnBn+HQllITID6hHVQ=; b=XFSriqMsZCc+zl YEGrvjCiGETz+G9sMXI4WhB4efhUjJ9jUquMGoZ8Bdp+L65Key0wYfgef0YWvGycw3eB4v0gz18N+ JBmmqUmfPyIOzruIZIIdO1nYcgNtz55/jX0MZ7ua5U5OdrzQRgXJ7oqXIDay50EqepGXFJ/7nh1Sd WqNqD27lXSRLT9DtFKrWSFXZBNeWGc9PcFO6NVlmqZlA7UYg1heMM9+wP+9fS/cf8DI3eX+fouyiP X1ADsIpYv8x6zdScFYBGBncx9NVIKMJCUz21WFyzmIHpahCt4mp4ceEE7l7vrnlOfdIfiVUjsh8Ee w45vPh5k8A6dYjzwt03A==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lJIEL-000lvL-Kw; Mon, 08 Mar 2021 15:55:37 +0000 Received: from mail-wr1-x449.google.com ([2a00:1450:4864:20::449]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lJIEA-000lnf-26 for linux-arm-kernel@lists.infradead.org; Mon, 08 Mar 2021 15:55:27 +0000 Received: by mail-wr1-x449.google.com with SMTP id n17so5020130wrq.5 for ; Mon, 08 Mar 2021 07:55:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=/BAO9ttHzCncRHdq2i5eu5LhUmwsdIn4HIy1rtrNRFc=; b=pB4bszXs7eg4iEC/f7lPbOmUyqfwLVIAkusMpoLIcEzhTvci5FuAKX+WtUCMT0OOKe fNuh/Rke0YH3W4gTl9P1Bi9YFkhxw/+VCRqo/qvNOiebJdM5CG2kMHVmSkKceHEs+4aO pQMMacT0AYSHqQnViAb/FuPuxQ8H3Tw5QcR48/5LJ3LUKJxT/vIyx3gZLT7XPdhcc2go 6KKg3H5hlpiLQ4LDYE64cp17vMyJnMFMRHATAjNODGPkv4VuutUzlMMSFk4Yecm53Spd dY39fPO1aWmwJy1YVxzglXlI6hEgtvH+4n+dUmooGMWTRxZE1J0o+esAlVESEOcH3bJU fEmQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=/BAO9ttHzCncRHdq2i5eu5LhUmwsdIn4HIy1rtrNRFc=; b=i6ieg7d0Av4kqZB6iSeawDIvYgDS0mKxE7inc6ZXpdImITfi4LyDeOas/W3QtxNIo4 uHCk2L0czIMmta3jl4lMIJkjTP0VE2v671qAK3+nVLTY/WyqQh1ZFF08PsEbqmZxeekB 7jaxzbOw0SaYiw7DbhAQi+fp9EehCtCgiybKxFsrHCnG96x40Cz8TE8cadct4SYa8Ojm XX3V2YzPMq8dMVmzJJbnR5zFaV/lclRffuDCJvIPM7hV/HWfF4rfTgOJdbwNEH1LId2d LP6YWSrWUlGBEEYPvR3gie+olZPs7TP+0k9FrqnsLIeuoSXTsCO8wb+u9x8WzEy1S/vW BdBQ== X-Gm-Message-State: AOAM531p4kt4zCbDioxcr6riTF70KMTCHhfFXvZ5vDTVLbvhPMkuX7Tk RulompIdydrzLzxZgjtzypNdHW27r5Ll8SG3 X-Google-Smtp-Source: ABdhPJwJMahHz8a6RnID8ilXLKTGZvWBo52nzB+MU0q2lSC6rkBvHBe6DWbJRXKUNkY2lnhyv6ySlCl374OhDgmp X-Received: from andreyknvl3.muc.corp.google.com ([2a00:79e0:15:13:85fb:aac9:69ed:e574]) (user=andreyknvl job=sendgmr) by 2002:a05:600c:35c1:: with SMTP id r1mr22344025wmq.60.1615218924080; Mon, 08 Mar 2021 07:55:24 -0800 (PST) Date: Mon, 8 Mar 2021 16:55:14 +0100 In-Reply-To: Message-Id: <755161094eac5b0fc15273d609c78a459d4d07b9.1615218180.git.andreyknvl@google.com> Mime-Version: 1.0 References: X-Mailer: git-send-email 2.30.1.766.gb4fecdf3b7-goog Subject: [PATCH v2 1/5] arm64: kasan: allow to init memory when setting tags From: Andrey Konovalov To: Catalin Marinas , Vincenzo Frascino , Alexander Potapenko , Marco Elver Cc: Andrew Morton , Will Deacon , Dmitry Vyukov , Andrey Ryabinin , Peter Collingbourne , Evgenii Stepanov , Branislav Rankov , Kevin Brodsky , kasan-dev@googlegroups.com, linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Andrey Konovalov X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210308_155526_211795_BC1EC29D X-CRM114-Status: GOOD ( 14.92 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org This change adds an argument to mte_set_mem_tag_range() that allows to enable memory initialization when settinh the allocation tags. The implementation uses stzg instruction instead of stg when this argument indicates to initialize memory. Combining setting allocation tags with memory initialization will improve HW_TAGS KASAN performance when init_on_alloc/free is enabled. This change doesn't integrate memory initialization with KASAN, this is done is subsequent patches in this series. Acked-by: Marco Elver Signed-off-by: Andrey Konovalov --- arch/arm64/include/asm/memory.h | 4 ++-- arch/arm64/include/asm/mte-kasan.h | 20 ++++++++++++++------ mm/kasan/kasan.h | 9 +++++---- 3 files changed, 21 insertions(+), 12 deletions(-) diff --git a/arch/arm64/include/asm/memory.h b/arch/arm64/include/asm/memory.h index c759faf7a1ff..f1ba48b4347d 100644 --- a/arch/arm64/include/asm/memory.h +++ b/arch/arm64/include/asm/memory.h @@ -248,8 +248,8 @@ static inline const void *__tag_set(const void *addr, u8 tag) #define arch_init_tags(max_tag) mte_init_tags(max_tag) #define arch_get_random_tag() mte_get_random_tag() #define arch_get_mem_tag(addr) mte_get_mem_tag(addr) -#define arch_set_mem_tag_range(addr, size, tag) \ - mte_set_mem_tag_range((addr), (size), (tag)) +#define arch_set_mem_tag_range(addr, size, tag, init) \ + mte_set_mem_tag_range((addr), (size), (tag), (init)) #endif /* CONFIG_KASAN_HW_TAGS */ /* diff --git a/arch/arm64/include/asm/mte-kasan.h b/arch/arm64/include/asm/mte-kasan.h index 7ab500e2ad17..35fe549f7ea4 100644 --- a/arch/arm64/include/asm/mte-kasan.h +++ b/arch/arm64/include/asm/mte-kasan.h @@ -53,7 +53,8 @@ static inline u8 mte_get_random_tag(void) * Note: The address must be non-NULL and MTE_GRANULE_SIZE aligned and * size must be non-zero and MTE_GRANULE_SIZE aligned. */ -static inline void mte_set_mem_tag_range(void *addr, size_t size, u8 tag) +static inline void mte_set_mem_tag_range(void *addr, size_t size, + u8 tag, bool init) { u64 curr, end; @@ -68,10 +69,16 @@ static inline void mte_set_mem_tag_range(void *addr, size_t size, u8 tag) * 'asm volatile' is required to prevent the compiler to move * the statement outside of the loop. */ - asm volatile(__MTE_PREAMBLE "stg %0, [%0]" - : - : "r" (curr) - : "memory"); + if (init) + asm volatile(__MTE_PREAMBLE "stzg %0, [%0]" + : + : "r" (curr) + : "memory"); + else + asm volatile(__MTE_PREAMBLE "stg %0, [%0]" + : + : "r" (curr) + : "memory"); curr += MTE_GRANULE_SIZE; } while (curr != end); @@ -100,7 +107,8 @@ static inline u8 mte_get_random_tag(void) return 0xFF; } -static inline void mte_set_mem_tag_range(void *addr, size_t size, u8 tag) +static inline void mte_set_mem_tag_range(void *addr, size_t size, + u8 tag, bool init) { } diff --git a/mm/kasan/kasan.h b/mm/kasan/kasan.h index 8c55634d6edd..7fbb32234414 100644 --- a/mm/kasan/kasan.h +++ b/mm/kasan/kasan.h @@ -291,7 +291,7 @@ static inline const void *arch_kasan_set_tag(const void *addr, u8 tag) #define arch_get_mem_tag(addr) (0xFF) #endif #ifndef arch_set_mem_tag_range -#define arch_set_mem_tag_range(addr, size, tag) ((void *)(addr)) +#define arch_set_mem_tag_range(addr, size, tag, init) ((void *)(addr)) #endif #define hw_enable_tagging() arch_enable_tagging() @@ -299,7 +299,8 @@ static inline const void *arch_kasan_set_tag(const void *addr, u8 tag) #define hw_set_tagging_report_once(state) arch_set_tagging_report_once(state) #define hw_get_random_tag() arch_get_random_tag() #define hw_get_mem_tag(addr) arch_get_mem_tag(addr) -#define hw_set_mem_tag_range(addr, size, tag) arch_set_mem_tag_range((addr), (size), (tag)) +#define hw_set_mem_tag_range(addr, size, tag, init) \ + arch_set_mem_tag_range((addr), (size), (tag), (init)) #else /* CONFIG_KASAN_HW_TAGS */ @@ -343,7 +344,7 @@ static inline void kasan_poison(const void *addr, size_t size, u8 value) if (WARN_ON(size & KASAN_GRANULE_MASK)) return; - hw_set_mem_tag_range((void *)addr, size, value); + hw_set_mem_tag_range((void *)addr, size, value, false); } static inline void kasan_unpoison(const void *addr, size_t size) @@ -360,7 +361,7 @@ static inline void kasan_unpoison(const void *addr, size_t size) return; size = round_up(size, KASAN_GRANULE_SIZE); - hw_set_mem_tag_range((void *)addr, size, tag); + hw_set_mem_tag_range((void *)addr, size, tag, false); } static inline bool kasan_byte_accessible(const void *addr)