From patchwork Thu Nov 5 00:02:17 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrey Konovalov X-Patchwork-Id: 11883241 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH,MAILING_LIST_MULTI,SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 14536C00A89 for ; Thu, 5 Nov 2020 05:03:04 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 7D3A72151B for ; Thu, 5 Nov 2020 05:03:03 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="Aau9DTJG"; dkim=fail reason="signature verification failed" (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="Ut7YIsAf"; dkim=fail reason="signature verification failed" (2048-bit key) header.d=google.com header.i=@google.com header.b="DwhOeulf" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 7D3A72151B Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:To:From:Subject:References:Mime-Version:Message-Id: In-Reply-To:Date:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=tc3g/72+gnNdu9h8PIDrcA/7Zv02pZgARGIM70HzzPI=; b=Aau9DTJGren83VX8BAqfrPqtn W6icjHd1Od94dVMwK5epgR2KuHNF3E5/OclRJJJ9ryilz+wJr4rP1XTbAu9cfFOsBkrEcBHstilPv NBfDWhG4ShJ8F2+xNozh/apVWUEPcxhoXA8vyFV7LZJeBQdUCVjqpz3ywHTAcVH/x8mjlrygWrbh9 NRpn9gH22DqHILI7Q62AUr2fOw2SHLtTNFwd7lM73LjPEE+/B/tCFGpfxBZPuB+1O+aliLC3sGMYO ldVVTGaKU59aqTeU6Zolh6gsFZYRxJdLJN7zlXR+UYRccO9qEdmoDg3wqXCpu2yeCpkzJ6RSZMu6V Ftus56N3w==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1kaXOm-0002yo-7k; Thu, 05 Nov 2020 05:01:24 +0000 Received: from casper.infradead.org ([2001:8b0:10b:1236::1]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1kaW3r-00016k-TN for linux-arm-kernel@merlin.infradead.org; Thu, 05 Nov 2020 03:35:44 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:Cc:To:From:Subject: References:Mime-Version:Message-Id:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=/zD7WZPyOqitLWvnpsu4/VKPhFmibJDXB/9ojacn4YU=; b=Ut7YIsAfGVisrVSHxS82EXIB/s SNVNuAR8vFlK36F6CBilSXZsGlRxsAuOTol20pAJsX2n2A+X+QMrExqU44zzve4roGMJbyyA9BlHL zGhThHbZPqtDR5tVXDhmXXNk3WGX3sPo+zC8KSwhSZyK4HQ7j6z2n24qSf1gnIvz/8U3982w7Ipcd GwjCgD92iB4IJrDBQfrNh96NPa0rJrnz3NfNl9/VNfMhKuECMlEcityePN7MImrIHi+9xMg3edz7A NbodHFKrH8aFXjudpILvQ+PjS9qt6jUDLjqogN/c03OKoF7EimF2Ep2DzfDsXLM0PgM/loN7PxaX3 jRMLgRNA==; Received: from mail-qk1-x74a.google.com ([2607:f8b0:4864:20::74a]) by casper.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1kaSkt-0000pm-2m for linux-arm-kernel@lists.infradead.org; Thu, 05 Nov 2020 00:03:58 +0000 Received: by mail-qk1-x74a.google.com with SMTP id f126so28167qke.17 for ; Wed, 04 Nov 2020 16:03:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=/zD7WZPyOqitLWvnpsu4/VKPhFmibJDXB/9ojacn4YU=; b=DwhOeulf583kaNb7jVHS5Ju0oa1dB7TrWi/R74xwn9ypCjTjRyYu4S8mpn18QDz9h6 pT0vF/yvBwh0+yLH/hwX3jrJc3VBDxn//Ereo6E9UElkpvh2qx9EdDD8loDgsfhHC8zh /H81ILLovNBrBT7gnCrmEGE3Hhz43hkPqoNJgdV+2pOGCS1BFKJ7i13Zp1VZHe6XFaIp jn4gXLXl+JEcEF3ImyfvsKrHHAE1zL5Sklo2KONJoFsQ6K4WK/fRubaG3V2m+OzSf0fK 4EUGukjbRkVVQM+qlOKiwuaeSn3pwok6hhAoIzm5rWvkIQ7+3EagaNQNqGwDm+x+1tJZ 6coQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=/zD7WZPyOqitLWvnpsu4/VKPhFmibJDXB/9ojacn4YU=; b=iw4V1O7C85gC7SUdYnV1NLTtwqBLGTcGa+lA35LekcjH17WSdmQLxddnBnROH1XAx5 6TaxO/gFNvQieFIf3HYE9tHVDF5a0Ez6z6r7hq0KPXyYVyCrEzufwDq3K5sOo1T83oQi 1fHLCtrr/MkcAHgY4oUwlG0/vvyR+lJ/sZrISlL0DQ1/uLVNiEJ12xMrI3J6ujQH5Zrs GXTK+uaABEhKadss9QPbET2cCgP5nM/Z2ClIr/lXETxDX/x72axp3V5nI9gAJEldVveK IX9hgzfELM4YpLZzvKnhQSAHBaKPBDCkbPmP6J3hEnZk3gY3olUNZFCDRNXwLPUoqUzZ Nx/w== X-Gm-Message-State: AOAM532CUxBStCKdVfLgb6Q69Xm/clxTu8gc/XvidbqFc4MeTiJuoRZX 4hJF4osMrvtSGlqlRBld3ymvrj7H3dWUk0Aj X-Google-Smtp-Source: ABdhPJx8UCj7hDKh8Iv5Gz2Yjdfp1h/FlWdPy/wN7f+g1RtFnkwPWjZFqkJyR10HO5jKvmjyPuN4iXlFK++lmqPj X-Received: from andreyknvl3.muc.corp.google.com ([2a00:79e0:15:13:7220:84ff:fe09:7e9d]) (user=andreyknvl job=sendgmr) by 2002:ad4:45ca:: with SMTP id v10mr546432qvt.36.1604534571983; Wed, 04 Nov 2020 16:02:51 -0800 (PST) Date: Thu, 5 Nov 2020 01:02:17 +0100 In-Reply-To: Message-Id: Mime-Version: 1.0 References: X-Mailer: git-send-email 2.29.1.341.ge80a0c044ae-goog Subject: [PATCH 07/20] kasan: inline kasan_reset_tag for tag-based modes From: Andrey Konovalov To: Catalin Marinas , Will Deacon , Vincenzo Frascino , Dmitry Vyukov , Alexander Potapenko , Marco Elver X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20201105_000355_346122_423F959B X-CRM114-Status: GOOD ( 16.52 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Branislav Rankov , Andrey Konovalov , Kevin Brodsky , linux-kernel@vger.kernel.org, kasan-dev@googlegroups.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, Andrey Ryabinin , Andrew Morton , Evgenii Stepanov Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Using kasan_reset_tag() currently results in a function call. As it's called quite often from the allocator code, this leads to a noticeable slowdown. Move it to include/linux/kasan.h and turn it into a static inline function. Also remove the now unneeded reset_tag() internal KASAN macro and use kasan_reset_tag() instead. Signed-off-by: Andrey Konovalov Link: https://linux-review.googlesource.com/id/I4d2061acfe91d480a75df00b07c22d8494ef14b5 --- include/linux/kasan.h | 5 ++++- mm/kasan/common.c | 6 +++--- mm/kasan/hw_tags.c | 9 ++------- mm/kasan/kasan.h | 4 ---- mm/kasan/report.c | 4 ++-- mm/kasan/report_hw_tags.c | 2 +- mm/kasan/report_sw_tags.c | 4 ++-- mm/kasan/shadow.c | 4 ++-- mm/kasan/sw_tags.c | 9 ++------- 9 files changed, 18 insertions(+), 29 deletions(-) diff --git a/include/linux/kasan.h b/include/linux/kasan.h index 2c37a39b76ed..0211a4ec5d87 100644 --- a/include/linux/kasan.h +++ b/include/linux/kasan.h @@ -191,7 +191,10 @@ static inline void kasan_record_aux_stack(void *ptr) {} #if defined(CONFIG_KASAN_SW_TAGS) || defined(CONFIG_KASAN_HW_TAGS) -void *kasan_reset_tag(const void *addr); +static inline void *kasan_reset_tag(const void *addr) +{ + return (void *)arch_kasan_reset_tag(addr); +} bool kasan_report(unsigned long addr, size_t size, bool is_write, unsigned long ip); diff --git a/mm/kasan/common.c b/mm/kasan/common.c index 9008fc6b0810..a266b90636a1 100644 --- a/mm/kasan/common.c +++ b/mm/kasan/common.c @@ -174,14 +174,14 @@ size_t kasan_metadata_size(struct kmem_cache *cache) struct kasan_alloc_meta *kasan_get_alloc_meta(struct kmem_cache *cache, const void *object) { - return (void *)reset_tag(object) + cache->kasan_info.alloc_meta_offset; + return kasan_reset_tag(object) + cache->kasan_info.alloc_meta_offset; } struct kasan_free_meta *kasan_get_free_meta(struct kmem_cache *cache, const void *object) { BUILD_BUG_ON(sizeof(struct kasan_free_meta) > 32); - return (void *)reset_tag(object) + cache->kasan_info.free_meta_offset; + return kasan_reset_tag(object) + cache->kasan_info.free_meta_offset; } void kasan_poison_slab(struct page *page) @@ -278,7 +278,7 @@ static bool __kasan_slab_free(struct kmem_cache *cache, void *object, tag = get_tag(object); tagged_object = object; - object = reset_tag(object); + object = kasan_reset_tag(object); if (unlikely(nearest_obj(cache, virt_to_head_page(object), object) != object)) { diff --git a/mm/kasan/hw_tags.c b/mm/kasan/hw_tags.c index d858aeb7387f..fe8e6c8e6319 100644 --- a/mm/kasan/hw_tags.c +++ b/mm/kasan/hw_tags.c @@ -26,20 +26,15 @@ void kasan_init_hw_tags(void) pr_info("KernelAddressSanitizer initialized\n"); } -void *kasan_reset_tag(const void *addr) -{ - return reset_tag(addr); -} - void kasan_poison_memory(const void *address, size_t size, u8 value) { - hw_set_mem_tag_range(reset_tag(address), + hw_set_mem_tag_range(kasan_reset_tag(address), round_up(size, KASAN_GRANULE_SIZE), value); } void kasan_unpoison_memory(const void *address, size_t size) { - hw_set_mem_tag_range(reset_tag(address), + hw_set_mem_tag_range(kasan_reset_tag(address), round_up(size, KASAN_GRANULE_SIZE), get_tag(address)); } diff --git a/mm/kasan/kasan.h b/mm/kasan/kasan.h index 5513b4685007..e9c7d061fbe5 100644 --- a/mm/kasan/kasan.h +++ b/mm/kasan/kasan.h @@ -246,15 +246,11 @@ static inline const void *arch_kasan_set_tag(const void *addr, u8 tag) return addr; } #endif -#ifndef arch_kasan_reset_tag -#define arch_kasan_reset_tag(addr) ((void *)(addr)) -#endif #ifndef arch_kasan_get_tag #define arch_kasan_get_tag(addr) 0 #endif #define set_tag(addr, tag) ((void *)arch_kasan_set_tag((addr), (tag))) -#define reset_tag(addr) ((void *)arch_kasan_reset_tag(addr)) #define get_tag(addr) arch_kasan_get_tag(addr) #ifdef CONFIG_KASAN_HW_TAGS diff --git a/mm/kasan/report.c b/mm/kasan/report.c index 0cac53a57c14..25ca66c99e48 100644 --- a/mm/kasan/report.c +++ b/mm/kasan/report.c @@ -328,7 +328,7 @@ void kasan_report_invalid_free(void *object, unsigned long ip) unsigned long flags; u8 tag = get_tag(object); - object = reset_tag(object); + object = kasan_reset_tag(object); #if IS_ENABLED(CONFIG_KUNIT) if (current->kunit_test) @@ -361,7 +361,7 @@ static void __kasan_report(unsigned long addr, size_t size, bool is_write, disable_trace_on_warning(); tagged_addr = (void *)addr; - untagged_addr = reset_tag(tagged_addr); + untagged_addr = kasan_reset_tag(tagged_addr); info.access_addr = tagged_addr; if (addr_has_metadata(untagged_addr)) diff --git a/mm/kasan/report_hw_tags.c b/mm/kasan/report_hw_tags.c index da543eb832cd..57114f0e14d1 100644 --- a/mm/kasan/report_hw_tags.c +++ b/mm/kasan/report_hw_tags.c @@ -22,7 +22,7 @@ const char *get_bug_type(struct kasan_access_info *info) void *find_first_bad_addr(void *addr, size_t size) { - return reset_tag(addr); + return kasan_reset_tag(addr); } void metadata_fetch_row(char *buffer, void *row) diff --git a/mm/kasan/report_sw_tags.c b/mm/kasan/report_sw_tags.c index 317100fd95b9..7604b46239d4 100644 --- a/mm/kasan/report_sw_tags.c +++ b/mm/kasan/report_sw_tags.c @@ -41,7 +41,7 @@ const char *get_bug_type(struct kasan_access_info *info) int i; tag = get_tag(info->access_addr); - addr = reset_tag(info->access_addr); + addr = kasan_reset_tag(info->access_addr); page = kasan_addr_to_page(addr); if (page && PageSlab(page)) { cache = page->slab_cache; @@ -72,7 +72,7 @@ const char *get_bug_type(struct kasan_access_info *info) void *find_first_bad_addr(void *addr, size_t size) { u8 tag = get_tag(addr); - void *p = reset_tag(addr); + void *p = kasan_reset_tag(addr); void *end = p + size; while (p < end && tag == *(u8 *)kasan_mem_to_shadow(p)) diff --git a/mm/kasan/shadow.c b/mm/kasan/shadow.c index 616ac64c4a21..8e4fa9157a0b 100644 --- a/mm/kasan/shadow.c +++ b/mm/kasan/shadow.c @@ -81,7 +81,7 @@ void kasan_poison_memory(const void *address, size_t size, u8 value) * some of the callers (e.g. kasan_poison_object_data) pass tagged * addresses to this function. */ - address = reset_tag(address); + address = kasan_reset_tag(address); shadow_start = kasan_mem_to_shadow(address); shadow_end = kasan_mem_to_shadow(address + size); @@ -98,7 +98,7 @@ void kasan_unpoison_memory(const void *address, size_t size) * some of the callers (e.g. kasan_unpoison_object_data) pass tagged * addresses to this function. */ - address = reset_tag(address); + address = kasan_reset_tag(address); kasan_poison_memory(address, size, tag); diff --git a/mm/kasan/sw_tags.c b/mm/kasan/sw_tags.c index 3bffb489b144..d1af6f6c6d12 100644 --- a/mm/kasan/sw_tags.c +++ b/mm/kasan/sw_tags.c @@ -67,11 +67,6 @@ u8 random_tag(void) return (u8)(state % (KASAN_TAG_MAX + 1)); } -void *kasan_reset_tag(const void *addr) -{ - return reset_tag(addr); -} - bool check_memory_region(unsigned long addr, size_t size, bool write, unsigned long ret_ip) { @@ -107,7 +102,7 @@ bool check_memory_region(unsigned long addr, size_t size, bool write, if (tag == KASAN_TAG_KERNEL) return true; - untagged_addr = reset_tag((const void *)addr); + untagged_addr = kasan_reset_tag((const void *)addr); if (unlikely(untagged_addr < kasan_shadow_to_mem((void *)KASAN_SHADOW_START))) { return !kasan_report(addr, size, write, ret_ip); @@ -126,7 +121,7 @@ bool check_memory_region(unsigned long addr, size_t size, bool write, bool check_invalid_free(void *addr) { u8 tag = get_tag(addr); - u8 shadow_byte = READ_ONCE(*(u8 *)kasan_mem_to_shadow(reset_tag(addr))); + u8 shadow_byte = READ_ONCE(*(u8 *)kasan_mem_to_shadow(kasan_reset_tag(addr))); return (shadow_byte == KASAN_TAG_INVALID) || (tag != KASAN_TAG_KERNEL && tag != shadow_byte);