From patchwork Mon Feb 1 19:43:25 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrey Konovalov X-Patchwork-Id: 12059863 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.1 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 08879C433DB for ; Mon, 1 Feb 2021 19:45:34 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id BE11364EA9 for ; Mon, 1 Feb 2021 19:45:33 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org BE11364EA9 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:To:From:Subject:References:Mime-Version:Message-Id: In-Reply-To:Date:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=Y7XkGYkAmYAcJyJwA2p3VNuv9dF3rq//9TYmRqNOGTA=; b=ofxo4ofIOFNm6/5UlQuvI7I0J a9FX1rEybFuq6UhsAw6IrBpqA1PFEz04DoVZUAvSC4Zmh+ammLM/jS5uruBuHq5PXXLasJEvIzSMg WnVy5cGA5uHIWltVM+qntQ/2xLtLru/yMgwdph6bWs23Ehf8UogrIS7PDFUW7NwImq5jFbAPW0EOJ ktCOvXPcJy9W58xXZJ1el4Bv+6iIp5euZS2ezyiNrn/vx3o680YbMSe3bMP40T6VuKhkI/oCfolNv gSVl8k3KhbZXUqPRVC14UQnhnlSjXs/9sFhuY0gy0jPoHycXDOIOt3kS8V3G6RU//rr2WzJlLvksz No/j96FCw==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1l6f76-0001v8-5w; Mon, 01 Feb 2021 19:43:56 +0000 Received: from mail-wr1-x449.google.com ([2a00:1450:4864:20::449]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1l6f6v-0001od-Ox for linux-arm-kernel@lists.infradead.org; Mon, 01 Feb 2021 19:43:47 +0000 Received: by mail-wr1-x449.google.com with SMTP id m7so340181wro.12 for ; Mon, 01 Feb 2021 11:43:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=xUtTTHPe7DRCgRtwSi+4scq+pNy2/ibyykEGsvZRuk0=; b=e176a67fK1UTibZXT++SH5zHy4zoNchdCjdpLCcct5knc2liGlNDF3Yamqdq+stmzY UD1Pf4fnW56aHo0VWMIdu0X7NSh9tglx3uCnpOggFaqp7pk/zA0jLS91Vw8jJ1oBwAbv ZdfBExGNllbfqfRyNbp+stdcka9VoQV4HuCvyYKhFs4zFM3kQb0iNgSNATnD3q8SQnlS bo94eGEJb9nNfzLhllE6oILm36yWPrzsRKbNhMrwH4R+PLCtzzlDsNrAtHgIeRJqSKJ7 KKt7zUyIWygctX6LyJdzJWWpJMrEjfTNbfEc77Ex501VCB2gl3GBeZo2AsAuv/+hTAsk ImoA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=xUtTTHPe7DRCgRtwSi+4scq+pNy2/ibyykEGsvZRuk0=; b=uXbAnQF2eAbAy3uBFOl70MLy/I7QKcpNcQoGbgs7YyOi9yAxRsZ2hglxNxsf93amlS 92X5p9qAHUxc6BJFdW/5Ioxpd9D8YexfBJZy9bRohb936DrJ4FuX999PLvVY9qIF8DtV rZX8q7FbbCa1AFuz3hV+yZREWxWjrIgOj24vEdv1YhsloOsScc6XYjWSNwnT+l/x8VFO 2Zr7HgXgiP2J6BWuqEAuHwVDWSjxKibMXYY2HnO5kKskU+RYJyFB2lnN2NU7khD0wITB GACS/SvW0tCT5xM6I6Q67yYbAlE1wqxyeWsf7xjdMApZ2sp44MCs3VoBmsaTWSH2xo9f +glA== X-Gm-Message-State: AOAM531TcQ4WNrbVEuvH5xgtm0tpkSnozr/iGLurXk/N7o9g66B+y14L lFTOJkecOdmRSuzCdYFMfgRCh/O3fOkFeL2A X-Google-Smtp-Source: ABdhPJw8tIfQ1xoMxW7zeobkrkFUwn4wKnbjCuC6YkGhqiO0pJ6PYdc4Nfm8BLCkJlGJR9HYDF5LqYLdtDMXC9jx X-Received: from andreyknvl3.muc.corp.google.com ([2a00:79e0:15:13:7220:84ff:fe09:7e9d]) (user=andreyknvl job=sendgmr) by 2002:a1c:8109:: with SMTP id c9mr380514wmd.137.1612208622386; Mon, 01 Feb 2021 11:43:42 -0800 (PST) Date: Mon, 1 Feb 2021 20:43:25 +0100 In-Reply-To: Message-Id: Mime-Version: 1.0 References: X-Mailer: git-send-email 2.30.0.365.g02bc693789-goog Subject: [PATCH 01/12] kasan, mm: don't save alloc stacks twice From: Andrey Konovalov To: Catalin Marinas , Vincenzo Frascino , Dmitry Vyukov , Alexander Potapenko , Marco Elver X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210201_144345_873310_107D958E X-CRM114-Status: GOOD ( 15.99 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Branislav Rankov , Andrey Konovalov , Kevin Brodsky , Will Deacon , linux-kernel@vger.kernel.org, kasan-dev@googlegroups.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, Andrey Ryabinin , Andrew Morton , Peter Collingbourne , Evgenii Stepanov Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Currently KASAN saves allocation stacks in both kasan_slab_alloc() and kasan_kmalloc() annotations. This patch changes KASAN to save allocation stacks for slab objects from kmalloc caches in kasan_kmalloc() only, and stacks for other slab objects in kasan_slab_alloc() only. This change requires ____kasan_kmalloc() knowing whether the object belongs to a kmalloc cache. This is implemented by adding a flag field to the kasan_info structure. That flag is only set for kmalloc caches via a new kasan_cache_create_kmalloc() annotation. Signed-off-by: Andrey Konovalov Reviewed-by: Marco Elver --- include/linux/kasan.h | 9 +++++++++ mm/kasan/common.c | 18 ++++++++++++++---- mm/slab_common.c | 1 + 3 files changed, 24 insertions(+), 4 deletions(-) diff --git a/include/linux/kasan.h b/include/linux/kasan.h index 6d8f3227c264..2d5de4092185 100644 --- a/include/linux/kasan.h +++ b/include/linux/kasan.h @@ -83,6 +83,7 @@ static inline void kasan_disable_current(void) {} struct kasan_cache { int alloc_meta_offset; int free_meta_offset; + bool is_kmalloc; }; #ifdef CONFIG_KASAN_HW_TAGS @@ -143,6 +144,13 @@ static __always_inline void kasan_cache_create(struct kmem_cache *cache, __kasan_cache_create(cache, size, flags); } +void __kasan_cache_create_kmalloc(struct kmem_cache *cache); +static __always_inline void kasan_cache_create_kmalloc(struct kmem_cache *cache) +{ + if (kasan_enabled()) + __kasan_cache_create_kmalloc(cache); +} + size_t __kasan_metadata_size(struct kmem_cache *cache); static __always_inline size_t kasan_metadata_size(struct kmem_cache *cache) { @@ -278,6 +286,7 @@ static inline void kasan_free_pages(struct page *page, unsigned int order) {} static inline void kasan_cache_create(struct kmem_cache *cache, unsigned int *size, slab_flags_t *flags) {} +static inline void kasan_cache_create_kmalloc(struct kmem_cache *cache) {} static inline size_t kasan_metadata_size(struct kmem_cache *cache) { return 0; } static inline void kasan_poison_slab(struct page *page) {} static inline void kasan_unpoison_object_data(struct kmem_cache *cache, diff --git a/mm/kasan/common.c b/mm/kasan/common.c index fe852f3cfa42..374049564ea3 100644 --- a/mm/kasan/common.c +++ b/mm/kasan/common.c @@ -210,6 +210,11 @@ void __kasan_cache_create(struct kmem_cache *cache, unsigned int *size, *size = optimal_size; } +void __kasan_cache_create_kmalloc(struct kmem_cache *cache) +{ + cache->kasan_info.is_kmalloc = true; +} + size_t __kasan_metadata_size(struct kmem_cache *cache) { if (!kasan_stack_collection_enabled()) @@ -394,17 +399,22 @@ void __kasan_slab_free_mempool(void *ptr, unsigned long ip) } } -static void set_alloc_info(struct kmem_cache *cache, void *object, gfp_t flags) +static void set_alloc_info(struct kmem_cache *cache, void *object, + gfp_t flags, bool kmalloc) { struct kasan_alloc_meta *alloc_meta; + /* Don't save alloc info for kmalloc caches in kasan_slab_alloc(). */ + if (cache->kasan_info.is_kmalloc && !kmalloc) + return; + alloc_meta = kasan_get_alloc_meta(cache, object); if (alloc_meta) kasan_set_track(&alloc_meta->alloc_track, flags); } static void *____kasan_kmalloc(struct kmem_cache *cache, const void *object, - size_t size, gfp_t flags, bool keep_tag) + size_t size, gfp_t flags, bool kmalloc) { unsigned long redzone_start; unsigned long redzone_end; @@ -423,7 +433,7 @@ static void *____kasan_kmalloc(struct kmem_cache *cache, const void *object, KASAN_GRANULE_SIZE); redzone_end = round_up((unsigned long)object + cache->object_size, KASAN_GRANULE_SIZE); - tag = assign_tag(cache, object, false, keep_tag); + tag = assign_tag(cache, object, false, kmalloc); /* Tag is ignored in set_tag without CONFIG_KASAN_SW/HW_TAGS */ kasan_unpoison(set_tag(object, tag), size); @@ -431,7 +441,7 @@ static void *____kasan_kmalloc(struct kmem_cache *cache, const void *object, KASAN_KMALLOC_REDZONE); if (kasan_stack_collection_enabled()) - set_alloc_info(cache, (void *)object, flags); + set_alloc_info(cache, (void *)object, flags, kmalloc); return set_tag(object, tag); } diff --git a/mm/slab_common.c b/mm/slab_common.c index 9aa3d2fe4c55..39d1a8ff9bb8 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -647,6 +647,7 @@ struct kmem_cache *__init create_kmalloc_cache(const char *name, panic("Out of memory when creating slab %s\n", name); create_boot_cache(s, name, size, flags, useroffset, usersize); + kasan_cache_create_kmalloc(s); list_add(&s->list, &slab_caches); s->refcount = 1; return s; From patchwork Mon Feb 1 19:43:26 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrey Konovalov X-Patchwork-Id: 12059865 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.1 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D6F3CC433E6 for ; Mon, 1 Feb 2021 19:45:47 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 6B54D64EAB for ; Mon, 1 Feb 2021 19:45:47 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 6B54D64EAB Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:To:From:Subject:References:Mime-Version:Message-Id: In-Reply-To:Date:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=Fv6OCgKoDRn/L7Ka8+etHwPvu4yizq71BykcQORgMAY=; b=DC2RUGnUz+0UZ8tCddHM3ahAc +1J+cHKNrDr0Y49xHcqz5zFvcpjh/hqnPp/VKhl37lEcp6jeO+DFasEsmbqYu/1zmqGm3P1zsTVJH JTi+693SwHQeM2fOy2h66Utnnt/gagwPbV0W99rBCkUGPUKFSs91pqD9rZuG+DQ/RkcMDTaKqA1Z8 RYmyUGI4ZKAot7ECtATnEA25UgyTqPEYcwIqOU1ytBWkZn4zDK8w6OebK1YyzZHThbUOHkPNpCQeR 1Mpeg2Cov6Z4irBISocjAWsymaOF7HrppjmwAKGkZJq/BjqF+2ekI2+XYDaqtbQGLlOqWHM6jko+O DYDDF7PsQ==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1l6f7F-00020B-Hd; Mon, 01 Feb 2021 19:44:05 +0000 Received: from mail-wm1-x34a.google.com ([2a00:1450:4864:20::34a]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1l6f6x-0001q3-FJ for linux-arm-kernel@lists.infradead.org; Mon, 01 Feb 2021 19:43:50 +0000 Received: by mail-wm1-x34a.google.com with SMTP id b201so170229wmb.9 for ; Mon, 01 Feb 2021 11:43:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=gyuOP94GIgUUwGD/R4wbCK1VboClPgYqAgz+f87pwxc=; b=p6grNAgNSSmz4spdKOzPcLMCvMxZx+ypM/Z0hojwKABm+roASresPc80+/eRGVuxel OpVYuKNtzemJ8RN9fIAsZa17CLHx4RtgQLbmecKjnjXdhypXuZLzDs9KVxxCgTAGYGLM hOCJBlwOv6oaUImzofs1uABzksjehzXQ9b67CqvHviePgGp+4zhjip3fK8cGMejierU1 bfUGAqtxMZ47Yfi+RT7MpDk1c6fejdydShZeUTjRJ/jxSOx7MUTCyhcFjJ9g3W9uZTlo kh/WrJVGsCAappLR6h6FBN9nIlo4HCgfniFUAg/ml+Ijnbqww5XHxQ4LGAhZsZie8cqI xG7A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=gyuOP94GIgUUwGD/R4wbCK1VboClPgYqAgz+f87pwxc=; b=dvYUZuGRM9//S5xmfBU7CJ8jVpGP2zjTsIT1LlTTXBR8ivwFqhJVuxbvb2DEJpRCJG CG9e5cCevKGns7/vFDWCkoRDRBguu2nOrVm5k7bjRcaKSvBSSaBkDB2Iqr/Miy1tOt+e kf9msrAiE5fkoKeKcAVld3sdS5X0Hq9NYP8EDNJB+Ria3k0xiBEEbZAw84z2ali6T32t l/ey6FtCdW6v2TE0vmQeKpcGOgG3KFRXvqXZigbIi5TGpCN+FwxeFeKdWRtMR3/S2Ar0 b02HW3HxeVVVOETyAz7sz5mMB4Nut5ICwNYVjHQo4vGFqDV//6WQ5in9y9+0DNz5dmHn MQVg== X-Gm-Message-State: AOAM530R8+pcMOonRGwsHPO9Oi+3J1y3QW7P7o2KyFs0t0GOEm7jQuBi XqrHw4vuf1vN1FEs25QdZZ9bU8oXWVAN55Px X-Google-Smtp-Source: ABdhPJzJ6kx5AQUwcSj5oiYP9SG+Bcwu0bDpq3j6i1QYTGx/ueFGRJy0q0ehWdKd19HNpyyvuA5K7r45uS6o7KUl X-Received: from andreyknvl3.muc.corp.google.com ([2a00:79e0:15:13:7220:84ff:fe09:7e9d]) (user=andreyknvl job=sendgmr) by 2002:a1c:2d8a:: with SMTP id t132mr377860wmt.119.1612208624836; Mon, 01 Feb 2021 11:43:44 -0800 (PST) Date: Mon, 1 Feb 2021 20:43:26 +0100 In-Reply-To: Message-Id: Mime-Version: 1.0 References: X-Mailer: git-send-email 2.30.0.365.g02bc693789-goog Subject: [PATCH 02/12] kasan, mm: optimize kmalloc poisoning From: Andrey Konovalov To: Catalin Marinas , Vincenzo Frascino , Dmitry Vyukov , Alexander Potapenko , Marco Elver X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210201_144347_640658_EFC40405 X-CRM114-Status: GOOD ( 26.19 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Branislav Rankov , Andrey Konovalov , Kevin Brodsky , Will Deacon , linux-kernel@vger.kernel.org, kasan-dev@googlegroups.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, Andrey Ryabinin , Andrew Morton , Peter Collingbourne , Evgenii Stepanov Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org For allocations from kmalloc caches, kasan_kmalloc() always follows kasan_slab_alloc(). Currenly, both of them unpoison the whole object, which is unnecessary. This patch provides separate implementations for both annotations: kasan_slab_alloc() unpoisons the whole object, and kasan_kmalloc() only poisons the redzone. For generic KASAN, the redzone start might not be aligned to KASAN_GRANULE_SIZE. Therefore, the poisoning is split in two parts: kasan_poison_last_granule() poisons the unaligned part, and then kasan_poison() poisons the rest. This patch also clarifies alignment guarantees of each of the poisoning functions and drops the unnecessary round_up() call for redzone_end. With this change, the early SLUB cache annotation needs to be changed to kasan_slab_alloc(), as kasan_kmalloc() doesn't unpoison objects now. The number of poisoned bytes for objects in this cache stays the same, as kmem_cache_node->object_size is equal to sizeof(struct kmem_cache_node). Signed-off-by: Andrey Konovalov Reviewed-by: Marco Elver --- mm/kasan/common.c | 93 +++++++++++++++++++++++++++++++---------------- mm/kasan/kasan.h | 43 +++++++++++++++++++++- mm/kasan/shadow.c | 28 +++++++------- mm/slub.c | 3 +- 4 files changed, 119 insertions(+), 48 deletions(-) diff --git a/mm/kasan/common.c b/mm/kasan/common.c index 374049564ea3..128cb330ca73 100644 --- a/mm/kasan/common.c +++ b/mm/kasan/common.c @@ -278,21 +278,11 @@ void __kasan_poison_object_data(struct kmem_cache *cache, void *object) * based on objects indexes, so that objects that are next to each other * get different tags. */ -static u8 assign_tag(struct kmem_cache *cache, const void *object, - bool init, bool keep_tag) +static u8 assign_tag(struct kmem_cache *cache, const void *object, bool init) { if (IS_ENABLED(CONFIG_KASAN_GENERIC)) return 0xff; - /* - * 1. When an object is kmalloc()'ed, two hooks are called: - * kasan_slab_alloc() and kasan_kmalloc(). We assign the - * tag only in the first one. - * 2. We reuse the same tag for krealloc'ed objects. - */ - if (keep_tag) - return get_tag(object); - /* * If the cache neither has a constructor nor has SLAB_TYPESAFE_BY_RCU * set, assign a tag when the object is being allocated (init == false). @@ -325,7 +315,7 @@ void * __must_check __kasan_init_slab_obj(struct kmem_cache *cache, } /* Tag is ignored in set_tag() without CONFIG_KASAN_SW/HW_TAGS */ - object = set_tag(object, assign_tag(cache, object, true, false)); + object = set_tag(object, assign_tag(cache, object, true)); return (void *)object; } @@ -413,12 +403,46 @@ static void set_alloc_info(struct kmem_cache *cache, void *object, kasan_set_track(&alloc_meta->alloc_track, flags); } +void * __must_check __kasan_slab_alloc(struct kmem_cache *cache, + void *object, gfp_t flags) +{ + u8 tag; + void *tagged_object; + + if (gfpflags_allow_blocking(flags)) + kasan_quarantine_reduce(); + + if (unlikely(object == NULL)) + return NULL; + + if (is_kfence_address(object)) + return (void *)object; + + /* + * Generate and assign random tag for tag-based modes. + * Tag is ignored in set_tag() for the generic mode. + */ + tag = assign_tag(cache, object, false); + tagged_object = set_tag(object, tag); + + /* + * Unpoison the whole object. + * For kmalloc() allocations, kasan_kmalloc() will do precise poisoning. + */ + kasan_unpoison(tagged_object, cache->object_size); + + /* Save alloc info (if possible) for non-kmalloc() allocations. */ + if (kasan_stack_collection_enabled()) + set_alloc_info(cache, (void *)object, flags, false); + + return tagged_object; +} + static void *____kasan_kmalloc(struct kmem_cache *cache, const void *object, - size_t size, gfp_t flags, bool kmalloc) + size_t size, gfp_t flags) { unsigned long redzone_start; unsigned long redzone_end; - u8 tag; if (gfpflags_allow_blocking(flags)) kasan_quarantine_reduce(); @@ -429,33 +453,41 @@ static void *____kasan_kmalloc(struct kmem_cache *cache, const void *object, if (is_kfence_address(kasan_reset_tag(object))) return (void *)object; + /* + * The object has already been unpoisoned by kasan_slab_alloc() for + * kmalloc() or by ksize() for krealloc(). + */ + + /* + * The redzone has byte-level precision for the generic mode. + * Partially poison the last object granule to cover the unaligned + * part of the redzone. + */ + if (IS_ENABLED(CONFIG_KASAN_GENERIC)) + kasan_poison_last_granule((void *)object, size); + + /* Poison the aligned part of the redzone. */ redzone_start = round_up((unsigned long)(object + size), KASAN_GRANULE_SIZE); - redzone_end = round_up((unsigned long)object + cache->object_size, - KASAN_GRANULE_SIZE); - tag = assign_tag(cache, object, false, kmalloc); - - /* Tag is ignored in set_tag without CONFIG_KASAN_SW/HW_TAGS */ - kasan_unpoison(set_tag(object, tag), size); + redzone_end = (unsigned long)object + cache->object_size; kasan_poison((void *)redzone_start, redzone_end - redzone_start, KASAN_KMALLOC_REDZONE); + /* + * Save alloc info (if possible) for kmalloc() allocations. + * This also rewrites the alloc info when called from kasan_krealloc(). + */ if (kasan_stack_collection_enabled()) - set_alloc_info(cache, (void *)object, flags, kmalloc); + set_alloc_info(cache, (void *)object, flags, true); - return set_tag(object, tag); -} - -void * __must_check __kasan_slab_alloc(struct kmem_cache *cache, - void *object, gfp_t flags) -{ - return ____kasan_kmalloc(cache, object, cache->object_size, flags, false); + /* Keep the tag that was set by kasan_slab_alloc(). */ + return (void *)object; } void * __must_check __kasan_kmalloc(struct kmem_cache *cache, const void *object, size_t size, gfp_t flags) { - return ____kasan_kmalloc(cache, object, size, flags, true); + return ____kasan_kmalloc(cache, object, size, flags); } EXPORT_SYMBOL(__kasan_kmalloc); @@ -496,8 +528,7 @@ void * __must_check __kasan_krealloc(const void *object, size_t size, gfp_t flag if (unlikely(!PageSlab(page))) return __kasan_kmalloc_large(object, size, flags); else - return ____kasan_kmalloc(page->slab_cache, object, size, - flags, true); + return ____kasan_kmalloc(page->slab_cache, object, size, flags); } void __kasan_kfree_large(void *ptr, unsigned long ip) diff --git a/mm/kasan/kasan.h b/mm/kasan/kasan.h index dd14e8870023..6a2882997f23 100644 --- a/mm/kasan/kasan.h +++ b/mm/kasan/kasan.h @@ -358,12 +358,51 @@ static inline bool kasan_byte_accessible(const void *addr) #else /* CONFIG_KASAN_HW_TAGS */ -void kasan_poison(const void *address, size_t size, u8 value); -void kasan_unpoison(const void *address, size_t size); +/** + * kasan_poison - mark the memory range as unaccessible + * @addr - range start address, must be aligned to KASAN_GRANULE_SIZE + * @size - range size + * @value - value that's written to metadata for the range + * + * The size gets aligned to KASAN_GRANULE_SIZE before marking the range. + */ +void kasan_poison(const void *addr, size_t size, u8 value); + +/** + * kasan_unpoison - mark the memory range as accessible + * @addr - range start address, must be aligned to KASAN_GRANULE_SIZE + * @size - range size + * + * For the tag-based modes, the @size gets aligned to KASAN_GRANULE_SIZE before + * marking the range. + * For the generic mode, the last granule of the memory range gets partially + * unpoisoned based on the @size. + */ +void kasan_unpoison(const void *addr, size_t size); + bool kasan_byte_accessible(const void *addr); #endif /* CONFIG_KASAN_HW_TAGS */ +#ifdef CONFIG_KASAN_GENERIC + +/** + * kasan_poison_last_granule - mark the last granule of the memory range as + * unaccessible + * @addr - range start address, must be aligned to KASAN_GRANULE_SIZE + * @size - range size + * + * This function is only available for the generic mode, as it's the only mode + * that has partially poisoned memory granules. + */ +void kasan_poison_last_granule(const void *address, size_t size); + +#else /* CONFIG_KASAN_GENERIC */ + +static inline void kasan_poison_last_granule(const void *address, size_t size) { } + +#endif /* CONFIG_KASAN_GENERIC */ + /* * Exported functions for interfaces called from assembly or from generated * code. Declarations here to avoid warning about missing declarations. diff --git a/mm/kasan/shadow.c b/mm/kasan/shadow.c index 1372a2fc0ca9..1ed7817e4ee6 100644 --- a/mm/kasan/shadow.c +++ b/mm/kasan/shadow.c @@ -69,10 +69,6 @@ void *memcpy(void *dest, const void *src, size_t len) return __memcpy(dest, src, len); } -/* - * Poisons the shadow memory for 'size' bytes starting from 'addr'. - * Memory addresses should be aligned to KASAN_GRANULE_SIZE. - */ void kasan_poison(const void *address, size_t size, u8 value) { void *shadow_start, *shadow_end; @@ -83,12 +79,12 @@ void kasan_poison(const void *address, size_t size, u8 value) * addresses to this function. */ address = kasan_reset_tag(address); - size = round_up(size, KASAN_GRANULE_SIZE); /* Skip KFENCE memory if called explicitly outside of sl*b. */ if (is_kfence_address(address)) return; + size = round_up(size, KASAN_GRANULE_SIZE); shadow_start = kasan_mem_to_shadow(address); shadow_end = kasan_mem_to_shadow(address + size); @@ -96,6 +92,16 @@ void kasan_poison(const void *address, size_t size, u8 value) } EXPORT_SYMBOL(kasan_poison); +#ifdef CONFIG_KASAN_GENERIC +void kasan_poison_last_granule(const void *address, size_t size) +{ + if (size & KASAN_GRANULE_MASK) { + u8 *shadow = (u8 *)kasan_mem_to_shadow(address + size); + *shadow = size & KASAN_GRANULE_MASK; + } +} +#endif + void kasan_unpoison(const void *address, size_t size) { u8 tag = get_tag(address); @@ -115,16 +121,12 @@ void kasan_unpoison(const void *address, size_t size) if (is_kfence_address(address)) return; + /* Unpoison round_up(size, KASAN_GRANULE_SIZE) bytes. */ kasan_poison(address, size, tag); - if (size & KASAN_GRANULE_MASK) { - u8 *shadow = (u8 *)kasan_mem_to_shadow(address + size); - - if (IS_ENABLED(CONFIG_KASAN_SW_TAGS)) - *shadow = tag; - else /* CONFIG_KASAN_GENERIC */ - *shadow = size & KASAN_GRANULE_MASK; - } + /* Partially poison the last granule for the generic mode. */ + if (IS_ENABLED(CONFIG_KASAN_GENERIC)) + kasan_poison_last_granule(address, size); } #ifdef CONFIG_MEMORY_HOTPLUG diff --git a/mm/slub.c b/mm/slub.c index 176b1cb0d006..e564008c2329 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -3565,8 +3565,7 @@ static void early_kmem_cache_node_alloc(int node) init_object(kmem_cache_node, n, SLUB_RED_ACTIVE); init_tracking(kmem_cache_node, n); #endif - n = kasan_kmalloc(kmem_cache_node, n, sizeof(struct kmem_cache_node), - GFP_KERNEL); + n = kasan_slab_alloc(kmem_cache_node, n, GFP_KERNEL); page->freelist = get_freepointer(kmem_cache_node, n); page->inuse = 1; page->frozen = 0; From patchwork Mon Feb 1 19:43:27 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrey Konovalov X-Patchwork-Id: 12059867 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.1 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 83B63C433E0 for ; Mon, 1 Feb 2021 19:45:49 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id E665664EA9 for ; Mon, 1 Feb 2021 19:45:48 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org E665664EA9 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:To:From:Subject:References:Mime-Version:Message-Id: In-Reply-To:Date:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=eZLwp5500oR/A7fEcxgZBUh6PKqo1GZEf5PrlSB3sqU=; b=z7grGXuy/IaDL2poA58yZ01zY DIbLiWXY/hEtMMWCVDcMKBXv12p59fMen1EX5CpSiyT4amHUPA9H7AMRvTBMe7g5cPMDwPaRMM3e1 yU93Db1sQ9+yKOYxFqPcsipAlTs1zPws/4OfNYutR+L0Ss9BviTgZjOQhV1KSXfzwggb0dbhluhTm 0/+wGzB9KIrpXIFwagLUfzSg5eyFI/G7D7ZXmskziZXnxYDin3bd1FhFZ2Pnj1JcQaNAv2BhgPtie DuAyC0WSAh7HR8/h06rSA53tf0WrAKZkXQkpl5PamU3BDjgfgGi731OXKP//WS0e3LDiiMR/E08VO c9zALJvpw==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1l6f7S-00026g-06; Mon, 01 Feb 2021 19:44:18 +0000 Received: from mail-qv1-xf49.google.com ([2607:f8b0:4864:20::f49]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1l6f71-0001rZ-Fs for linux-arm-kernel@lists.infradead.org; Mon, 01 Feb 2021 19:43:53 +0000 Received: by mail-qv1-xf49.google.com with SMTP id v13so8738526qvm.10 for ; Mon, 01 Feb 2021 11:43:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=5os4NdBbDONsbLVxABgdrPD9x64VnOptbOzgU3pEDKs=; b=icvI25ft44tZNJLareqZm2eV6LU3rjKFoSzXua9cf+7734+1fWcCmnYf7hEl5Qbz+D X1HGWJkAzxbfnIv7aZkYjPhjP0onDk8oLebvt5EFUyH9inK3uQ5w7A/juLiNsTrOpW6D 4QB/f/PJ605/JfJndxR6OL1twXraBxmad0by++3RFa1uvStOQIe+cLBvnXzyUaTfu+2v IDNsEAuTbXwUW2bZwOGOX6UO1kpsjI9ybzJEZjTQQpoYleLdUk5xqzFbAzvVhdKy/Qr8 WEoqEGKqEd0hnV7F9350O0q+eRFF2yKHGuEA4AGGamvMh48mEenN5FlPFbNm7at+3K3H B7wg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=5os4NdBbDONsbLVxABgdrPD9x64VnOptbOzgU3pEDKs=; b=CtQB9Zbsq0d+8DiAB1y0T/voysbylwGrhaM8vqcQz88qDomRdi+u36+gfHvK/h2Fvr 0zGHUOo+L17+z3cUaYyjc/o6shQ7EXeKqwlunwgway8WqTfyLAn4gAQINL51uvQBV/Rw wOnnoqc+EsGT9bWZmAdkQjDojtTLbTf81BYVu1hAUPWk7vtrwxQ/1Vw392dj4gRMI0GJ euKLrbmHWyPyGxKOr17VlLJ9Cfo0C/yfqzZV4iKMwNtUq20nT0ewufJbTTfhwZ8qTPKk N0Sc2dUuSBsatkJLWSBVj6YHO1uvTfvxDIIU2IkrEcACZXP/XjUcTxTNoRLxS5UYI8qn V4tw== X-Gm-Message-State: AOAM533o/hn8Q6ez+7LoahhqcwQdbXTr/DOVtdDjBj4cKtmmAp7kgcMo EHL5eY3HuHTBaSjZjn9e/svju0nsFO8tNLFI X-Google-Smtp-Source: ABdhPJzR+YEpFc2Ch11Rl6BTXMIoXfaJyZzPai737uCYCgWg2fJznC+a5+pVgD+X3paVHk7gPsjQcrXSotCeIBAj X-Received: from andreyknvl3.muc.corp.google.com ([2a00:79e0:15:13:7220:84ff:fe09:7e9d]) (user=andreyknvl job=sendgmr) by 2002:a0c:c488:: with SMTP id u8mr16899450qvi.9.1612208627196; Mon, 01 Feb 2021 11:43:47 -0800 (PST) Date: Mon, 1 Feb 2021 20:43:27 +0100 In-Reply-To: Message-Id: <8fdbf86842f4eaf2458ecd23d0844058dbc2c7a2.1612208222.git.andreyknvl@google.com> Mime-Version: 1.0 References: X-Mailer: git-send-email 2.30.0.365.g02bc693789-goog Subject: [PATCH 03/12] kasan: optimize large kmalloc poisoning From: Andrey Konovalov To: Catalin Marinas , Vincenzo Frascino , Dmitry Vyukov , Alexander Potapenko , Marco Elver X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210201_144351_589583_966E65D5 X-CRM114-Status: GOOD ( 15.11 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Branislav Rankov , Andrey Konovalov , Kevin Brodsky , Will Deacon , linux-kernel@vger.kernel.org, kasan-dev@googlegroups.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, Andrey Ryabinin , Andrew Morton , Peter Collingbourne , Evgenii Stepanov Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Similarly to kasan_kmalloc(), kasan_kmalloc_large() doesn't need to unpoison the object as it as already unpoisoned by alloc_pages() (or by ksize() for krealloc()). This patch changes kasan_kmalloc_large() to only poison the redzone. Signed-off-by: Andrey Konovalov Reviewed-by: Marco Elver --- mm/kasan/common.c | 20 +++++++++++++++----- 1 file changed, 15 insertions(+), 5 deletions(-) diff --git a/mm/kasan/common.c b/mm/kasan/common.c index 128cb330ca73..a7eb553c8e91 100644 --- a/mm/kasan/common.c +++ b/mm/kasan/common.c @@ -494,7 +494,6 @@ EXPORT_SYMBOL(__kasan_kmalloc); void * __must_check __kasan_kmalloc_large(const void *ptr, size_t size, gfp_t flags) { - struct page *page; unsigned long redzone_start; unsigned long redzone_end; @@ -504,12 +503,23 @@ void * __must_check __kasan_kmalloc_large(const void *ptr, size_t size, if (unlikely(ptr == NULL)) return NULL; - page = virt_to_page(ptr); + /* + * The object has already been unpoisoned by kasan_alloc_pages() for + * alloc_pages() or by ksize() for krealloc(). + */ + + /* + * The redzone has byte-level precision for the generic mode. + * Partially poison the last object granule to cover the unaligned + * part of the redzone. + */ + if (IS_ENABLED(CONFIG_KASAN_GENERIC)) + kasan_poison_last_granule(ptr, size); + + /* Poison the aligned part of the redzone. */ redzone_start = round_up((unsigned long)(ptr + size), KASAN_GRANULE_SIZE); - redzone_end = (unsigned long)ptr + page_size(page); - - kasan_unpoison(ptr, size); + redzone_end = (unsigned long)ptr + page_size(virt_to_page(ptr)); kasan_poison((void *)redzone_start, redzone_end - redzone_start, KASAN_PAGE_REDZONE); From patchwork Mon Feb 1 19:43:28 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrey Konovalov X-Patchwork-Id: 12059869 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.1 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id EFFB5C433DB for ; Mon, 1 Feb 2021 19:46:02 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 95CCE64DBA for ; Mon, 1 Feb 2021 19:46:02 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 95CCE64DBA Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:To:From:Subject:References:Mime-Version:Message-Id: In-Reply-To:Date:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=AQMNUjqhnv7XR7fd7Cvsftj7wAfReiAO7n5tof26pdY=; b=ADPaSPTel6ltsw5qX+ZvUJZ2d C3m8tw/palfv1FC14VZo2Tmz66er9fQdx5sUrsHBrd/7pkoMuybWrldeog5rjRV91/ty8+oqq2d2b oeDo90IjlDmQR+oszCYCWHsxmqOjmH6VB9C/8ai6ELE698FBrQtV7yNjT1NMM2TbKij+tgpwDvI4d EMZVTzSTopUOdQ6+TsZPjmMaUI0UT60Q0ybhOC8MG5ZI29llZmQv0yS6pUWf0OGglyLRY6G/D1QfY XJR3LxJnBKq09zQQ++E/0RSdqeQVUg6e6gQSzZOpouo34F0LmKGCvqDPdCpz6DAAGdMSQnJoKVj0D 9/Y6Dqe2g==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1l6f7W-00029J-PH; Mon, 01 Feb 2021 19:44:22 +0000 Received: from mail-lj1-x249.google.com ([2a00:1450:4864:20::249]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1l6f74-0001sb-F4 for linux-arm-kernel@lists.infradead.org; Mon, 01 Feb 2021 19:43:55 +0000 Received: by mail-lj1-x249.google.com with SMTP id m9so9989245ljp.16 for ; Mon, 01 Feb 2021 11:43:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=vdSQ7N299ttJfkAji9JrIXHPqKuzBqRIB40yh5wvGsM=; b=Phj+HIZS3dxn0owHV5yLJaaC0AGJkEU7TfgxvzoS86S8PoZm4Rt1g+God+bs67EJc+ oAD4CGWKbkQlIIbCUNovOV1zLvhKFi5m+ZzhOblNupaIUd5ggX6TEZOlIrDz5XWmWLta JTZs7Uw7XmdRI9FLQF1x63YHp2a9KJzeObyFVOK0Yf424coQsCFXhiGhs5WSS+Wfdr5+ DV7O2YJKktRTJmORtVpVqwjlVS56TeRw4d0zGoTpGadYHfwg5/12VFuGKUvo3egw2HjJ JyuLK5fUm/yWENE6Hh73fMc8gwZMa7zXTtIL5S8EwT3PsxdE5NtR2aqDG2mEhfgqxwDS Cr8w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=vdSQ7N299ttJfkAji9JrIXHPqKuzBqRIB40yh5wvGsM=; b=EIXMGDSRzsItTiEkYfVfj31nI1WDETKtCH2spErdkHg9wqtfdlIMTSDBh33sNgHaJQ yn6m6m//qRDk4QDWlEXHyT32K1xtR7bVUvpqCu9Si/iX5l3ueuHB5EOO9X05pM1APY22 aMez1+CWZphBv59imVPhnWhO5ZM/SSONJVevyJzhzMLLMA/xG7BvjGBabyM3uGZvkZde dg2HyoVVIaOsLnOiwPTB5BNqMrbppoBfGzIcqBg2ottU+TRmEbnlgYiXwrOXvvCB4xL7 B3FgljivWpXPLPMywUP2UQPPU+sh0w9jbCyCE5nljCO9h6Rrxl+KS7CDtql/s5zwXzx9 4CNA== X-Gm-Message-State: AOAM531HNkLKoKlF1tHGN8qGHnOeN1zNhYbqGYkR3/I8FM6pN1AgEmVV YT6b2euAoLIzf34+rmmS7WD+Lgt+uiR47Q+w X-Google-Smtp-Source: ABdhPJyTPeHvPlSg3evueEvog+d2Wbu3vFUyi7JAGczcnckPOWrvdcuYouS1E9V9e/DkjSKSFt3vDyjaDwqvcObf X-Received: from andreyknvl3.muc.corp.google.com ([2a00:79e0:15:13:7220:84ff:fe09:7e9d]) (user=andreyknvl job=sendgmr) by 2002:ac2:44b8:: with SMTP id c24mr9569637lfm.155.1612208629709; Mon, 01 Feb 2021 11:43:49 -0800 (PST) Date: Mon, 1 Feb 2021 20:43:28 +0100 In-Reply-To: Message-Id: Mime-Version: 1.0 References: X-Mailer: git-send-email 2.30.0.365.g02bc693789-goog Subject: [PATCH 04/12] kasan: clean up setting free info in kasan_slab_free From: Andrey Konovalov To: Catalin Marinas , Vincenzo Frascino , Dmitry Vyukov , Alexander Potapenko , Marco Elver X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210201_144354_556712_1F9CD2AC X-CRM114-Status: GOOD ( 11.21 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Branislav Rankov , Andrey Konovalov , Kevin Brodsky , Will Deacon , linux-kernel@vger.kernel.org, kasan-dev@googlegroups.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, Andrey Ryabinin , Andrew Morton , Peter Collingbourne , Evgenii Stepanov Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Put kasan_stack_collection_enabled() check and kasan_set_free_info() calls next to each other. The way this was previously implemented was a minor optimization that relied of the the fact that kasan_stack_collection_enabled() is always true for generic KASAN. The confusion that this brings outweights saving a few instructions. Signed-off-by: Andrey Konovalov Reviewed-by: Marco Elver --- mm/kasan/common.c | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/mm/kasan/common.c b/mm/kasan/common.c index a7eb553c8e91..086bb77292b6 100644 --- a/mm/kasan/common.c +++ b/mm/kasan/common.c @@ -350,13 +350,11 @@ static bool ____kasan_slab_free(struct kmem_cache *cache, void *object, kasan_poison(object, cache->object_size, KASAN_KMALLOC_FREE); - if (!kasan_stack_collection_enabled()) - return false; - if ((IS_ENABLED(CONFIG_KASAN_GENERIC) && !quarantine)) return false; - kasan_set_free_info(cache, object, tag); + if (kasan_stack_collection_enabled()) + kasan_set_free_info(cache, object, tag); return kasan_quarantine_put(cache, object); } From patchwork Mon Feb 1 19:43:29 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrey Konovalov X-Patchwork-Id: 12059871 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.1 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 290BEC433E0 for ; Mon, 1 Feb 2021 19:46:07 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id A6B8164EBD for ; Mon, 1 Feb 2021 19:46:06 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org A6B8164EBD Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:To:From:Subject:References:Mime-Version:Message-Id: In-Reply-To:Date:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=ER6i726DM3kwj94NnzduRy3cisMp5FVhPmJUkH7XKf0=; b=SMEb4Si1rLi6dv8bztbqKeOBq qCp0ZU9767dnxsrTmxiV8hdqcBDP+CtTKxfj+SCgcj3WRXZdX3GomItB/3rgAnDV6SMGtmHhdXNhR j7qiH+ZgCdRIISf3EKkqNSx9c+KyEZGAjE8BN6BmJTn1XASgBsy4Q6qfUBO6jv3bJRZUIBYZ3ykMf F4gkeYCUABTu8Om97XDhnv8mgSg0r9yvDBX2917HsQE9jDRia+F00ECqs8aEOV4eNsbT98M7WC6j/ VdEZeOCK8vsq3NaN+VRSXJ5bRyLVVG9QlALwJwyXzcqu+4HRUM5JT5Lpd/Gfv/38Pg8CPKhfLWwaB vcvbUF1yA==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1l6f7Z-0002Ao-8V; Mon, 01 Feb 2021 19:44:25 +0000 Received: from mail-wm1-x34a.google.com ([2a00:1450:4864:20::34a]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1l6f75-0001tj-EG for linux-arm-kernel@lists.infradead.org; Mon, 01 Feb 2021 19:43:56 +0000 Received: by mail-wm1-x34a.google.com with SMTP id y9so174803wmi.8 for ; Mon, 01 Feb 2021 11:43:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=BaU4zsBnF6d3TSTU+6bwjMLYV3de6hEhX75uuk8b+O4=; b=nJiXwOEo+jDWjutl3bPtMddLIEbYjySSqFB9nsRMPC/OyMH1cWW9MJCeCANQqWCKmB 4mwP+1VFCx3hJfom58GEOvLp6QCaI9DthbxNDJt5E19vA2IenGYT//kFZ3tt5SKMwPII JUvkgGhGqUDAruV7bZhAZMJVREM4TVtH2aoavGSYKjlNdtFGTAYff54a63RosxywMuHm 1rEjzRp24e2Mh+Qobh7XPtIFaq29nychvE8goMeWtgJmRCYM9XxEicpGU9vrO7yzeJf3 0BwkX+2NT+5+L7FlbRszkE0nJeL7H4X8e4lzext+ZMwX8Dzov8Ow4AQrlTK8RDIsbmNp XaFw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=BaU4zsBnF6d3TSTU+6bwjMLYV3de6hEhX75uuk8b+O4=; b=EoxRWB/a9NcSw3ZnQfXqcpeoUblBFZmBAQ+1i1G8185LeeG7L/zUCbonDXGAQINVxQ mU2L7HbJRJMdGSHoaoDPdo+lgxLjWDhaimafiY0JNAt7DFK81w6NHByFgr74DqdDYHdu MNvZRx+pUJyKVtijTVfOSiBhxNb6aKE6GOcALHtE4oFRymuPr7PmMOWpy96QDWETDZQk 0n1ei21EU4zTXxaqtKb4XnQcoxJL+LXQaBWq5AN6zCOr2QwRN4s00yDhkiwOUBWLdHsd MpYUJk5cDD8v570B3BB/kq4AI7d76THkVMXwF7Qv1rvLEKu/6K3FT72rbEAsoiYZfxMi 7wxw== X-Gm-Message-State: AOAM532khPUb4Nk6fisfekTzCEZ0j/SUzlf5v18VPjy2PtLVqPo8j4QQ m6UwpTBXenMjdWx/5Sb6d1mIPjQCvM6yJedA X-Google-Smtp-Source: ABdhPJw57hRKvoeOE3jkg8dsfKFGhvdZTC70UClQcRWBXmSodKiJDsoN3TWrUtEegspLXmVAevtVI9fb+W66gF3H X-Received: from andreyknvl3.muc.corp.google.com ([2a00:79e0:15:13:7220:84ff:fe09:7e9d]) (user=andreyknvl job=sendgmr) by 2002:a1c:6a02:: with SMTP id f2mr447589wmc.36.1612208631997; Mon, 01 Feb 2021 11:43:51 -0800 (PST) Date: Mon, 1 Feb 2021 20:43:29 +0100 In-Reply-To: Message-Id: Mime-Version: 1.0 References: X-Mailer: git-send-email 2.30.0.365.g02bc693789-goog Subject: [PATCH 05/12] kasan: unify large kfree checks From: Andrey Konovalov To: Catalin Marinas , Vincenzo Frascino , Dmitry Vyukov , Alexander Potapenko , Marco Elver X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210201_144355_538915_5E617885 X-CRM114-Status: GOOD ( 16.29 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Branislav Rankov , Andrey Konovalov , Kevin Brodsky , Will Deacon , linux-kernel@vger.kernel.org, kasan-dev@googlegroups.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, Andrey Ryabinin , Andrew Morton , Peter Collingbourne , Evgenii Stepanov Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Unify checks in kasan_kfree_large() and in kasan_slab_free_mempool() for large allocations as it's done for small kfree() allocations. With this change, kasan_slab_free_mempool() starts checking that the first byte of the memory that's being freed is accessible. Signed-off-by: Andrey Konovalov Reviewed-by: Marco Elver --- include/linux/kasan.h | 16 ++++++++-------- mm/kasan/common.c | 36 ++++++++++++++++++++++++++---------- 2 files changed, 34 insertions(+), 18 deletions(-) diff --git a/include/linux/kasan.h b/include/linux/kasan.h index 2d5de4092185..d53ea3c047bc 100644 --- a/include/linux/kasan.h +++ b/include/linux/kasan.h @@ -200,6 +200,13 @@ static __always_inline bool kasan_slab_free(struct kmem_cache *s, void *object) return false; } +void __kasan_kfree_large(void *ptr, unsigned long ip); +static __always_inline void kasan_kfree_large(void *ptr) +{ + if (kasan_enabled()) + __kasan_kfree_large(ptr, _RET_IP_); +} + void __kasan_slab_free_mempool(void *ptr, unsigned long ip); static __always_inline void kasan_slab_free_mempool(void *ptr) { @@ -247,13 +254,6 @@ static __always_inline void * __must_check kasan_krealloc(const void *object, return (void *)object; } -void __kasan_kfree_large(void *ptr, unsigned long ip); -static __always_inline void kasan_kfree_large(void *ptr) -{ - if (kasan_enabled()) - __kasan_kfree_large(ptr, _RET_IP_); -} - /* * Unlike kasan_check_read/write(), kasan_check_byte() is performed even for * the hardware tag-based mode that doesn't rely on compiler instrumentation. @@ -302,6 +302,7 @@ static inline bool kasan_slab_free(struct kmem_cache *s, void *object) { return false; } +static inline void kasan_kfree_large(void *ptr) {} static inline void kasan_slab_free_mempool(void *ptr) {} static inline void *kasan_slab_alloc(struct kmem_cache *s, void *object, gfp_t flags) @@ -322,7 +323,6 @@ static inline void *kasan_krealloc(const void *object, size_t new_size, { return (void *)object; } -static inline void kasan_kfree_large(void *ptr) {} static inline bool kasan_check_byte(const void *address) { return true; diff --git a/mm/kasan/common.c b/mm/kasan/common.c index 086bb77292b6..9c64a00bbf9c 100644 --- a/mm/kasan/common.c +++ b/mm/kasan/common.c @@ -364,6 +364,31 @@ bool __kasan_slab_free(struct kmem_cache *cache, void *object, unsigned long ip) return ____kasan_slab_free(cache, object, ip, true); } +static bool ____kasan_kfree_large(void *ptr, unsigned long ip) +{ + if (ptr != page_address(virt_to_head_page(ptr))) { + kasan_report_invalid_free(ptr, ip); + return true; + } + + if (!kasan_byte_accessible(ptr)) { + kasan_report_invalid_free(ptr, ip); + return true; + } + + /* + * The object will be poisoned by kasan_free_pages() or + * kasan_slab_free_mempool(). + */ + + return false; +} + +void __kasan_kfree_large(void *ptr, unsigned long ip) +{ + ____kasan_kfree_large(ptr, ip); +} + void __kasan_slab_free_mempool(void *ptr, unsigned long ip) { struct page *page; @@ -377,10 +402,8 @@ void __kasan_slab_free_mempool(void *ptr, unsigned long ip) * KMALLOC_MAX_SIZE, and kmalloc falls back onto page_alloc. */ if (unlikely(!PageSlab(page))) { - if (ptr != page_address(page)) { - kasan_report_invalid_free(ptr, ip); + if (____kasan_kfree_large(ptr, ip)) return; - } kasan_poison(ptr, page_size(page), KASAN_FREE_PAGE); } else { ____kasan_slab_free(page->slab_cache, ptr, ip, false); @@ -539,13 +562,6 @@ void * __must_check __kasan_krealloc(const void *object, size_t size, gfp_t flag return ____kasan_kmalloc(page->slab_cache, object, size, flags); } -void __kasan_kfree_large(void *ptr, unsigned long ip) -{ - if (ptr != page_address(virt_to_head_page(ptr))) - kasan_report_invalid_free(ptr, ip); - /* The object will be poisoned by kasan_free_pages(). */ -} - bool __kasan_check_byte(const void *address, unsigned long ip) { if (!kasan_byte_accessible(address)) { From patchwork Mon Feb 1 19:43:30 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrey Konovalov X-Patchwork-Id: 12059873 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.1 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B4F04C433E6 for ; Mon, 1 Feb 2021 19:46:08 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 6E90064EA9 for ; Mon, 1 Feb 2021 19:46:08 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 6E90064EA9 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:To:From:Subject:References:Mime-Version:Message-Id: In-Reply-To:Date:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=05kELMJkJ0Ehhi2Fvb8HMP6DeVD3fApS6eN7XCx+up4=; b=R9L2Pqki4PMF9UGQgKQUroPlH c+Z/GKUmDDeQc/jcI/055J+xgfS+snSNtKKZ4DWLtuHsF2s8cx4ycdSybgEKtQT+mHhexd7yopwab pAGkGJDSe4QMY1YBdddlXcgNlbfwotMLmS5BDY7xLuXKIq27u8wny1AwIGC2lKnLvrsj9+FrrcYSo My7w+oW0mep/93pIhimloshZttqxnZ571nKTZ25wwPXBaF0aLB66HzdW46WOO1nMQcKW7+OhyimLI x2eWht6LinqiJl++Sl1uAsBU4cB/ME3ZMsCkCYAalSf7GtQtsYOqW3XH7kLNdG2VaiovwxMh05a8S WrBKmTi5A==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1l6f7c-0002CZ-UZ; Mon, 01 Feb 2021 19:44:29 +0000 Received: from mail-qv1-xf49.google.com ([2607:f8b0:4864:20::f49]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1l6f77-0001v4-Qo for linux-arm-kernel@lists.infradead.org; Mon, 01 Feb 2021 19:43:59 +0000 Received: by mail-qv1-xf49.google.com with SMTP id t18so12039408qva.6 for ; Mon, 01 Feb 2021 11:43:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=XCxo0tPucACpabOxuZtkTLtcboXhH0Xr2jZowtJcY6I=; b=nnt2QvzSfV0fa4ZZ03BZbnCp26ygACCoyTH46dpZyV9pLif0zOs6SXCEH2kcnjoFoV U1X5EXE1b/4nl5HZlwRerfFqKzt7jlwQz2d6uc95LKZ+SqbWeT8/Jp5hmPe3wZ5Q5tR+ Me7mRSARwZsr3WcDQyZ/mm/huwmItQuMt6csyGU8hTuyjthwd5A/Nn8C9a4mByWlS1VZ 1MkGWO4bG9xlswrcA618UKRyKeogViFJD3prqZJKBHRo3djA6zeLo3w47rwnsunN0gz3 flvwfX50AHe7/LCWla+sbrMmy1N/cEUeB+vO+5PGoJ4Hd85oD2y2azxexCmHYTpLxzJp /Jiw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=XCxo0tPucACpabOxuZtkTLtcboXhH0Xr2jZowtJcY6I=; b=qxqUD+tdFxdkE4gXMAHwXxSsjKBQTdpdqMtnmkp/Jr9PQScyxvMzbVTvZ5LUcI483Y zUWcqzAbJNGF3s6MjnDX5UPc7tIYrOG2Q/SQ61pp394Nh30rKKygHPM979zHtDmtMzn9 2z51UyZSqdPMtmuvS0ZZqf0rWu+Lx6luKL4qgmqHHgc0bGKpDJdL9JxkZpOXB+i6PcQB 8DTf6z2hFpo3tQB6AtJnoxWlkEvhfBX85/fsioEAquadkAtxqMRthrlZ8KJCuBebC3eD Tf9Ngw9mw81kzmM2N/d1EEatLxP7gCTZIF1ckS9PvF2u9LWjwO/7NNcPi7pzJj6Fd1ly NSbA== X-Gm-Message-State: AOAM531qLiXdQQDJJ46RD5UcKcmj+Roco7ET6tE3NnMjC6/rJa44nheK wMl6Db0dm+u8XaFKItFQNF/tmJMinuAExuI9 X-Google-Smtp-Source: ABdhPJwdXbY1qM/NMUi7jaSpz4LhDzmb7rta6x+9O6xGnsVRwYvyMYrhWT+3uFTKuYlZPyvi6dklTIkdGEdweICZ X-Received: from andreyknvl3.muc.corp.google.com ([2a00:79e0:15:13:7220:84ff:fe09:7e9d]) (user=andreyknvl job=sendgmr) by 2002:ad4:40c6:: with SMTP id x6mr16803415qvp.10.1612208634261; Mon, 01 Feb 2021 11:43:54 -0800 (PST) Date: Mon, 1 Feb 2021 20:43:30 +0100 In-Reply-To: Message-Id: <995edb531f4f976277d7da9ca8a78a96a2ea356e.1612208222.git.andreyknvl@google.com> Mime-Version: 1.0 References: X-Mailer: git-send-email 2.30.0.365.g02bc693789-goog Subject: [PATCH 06/12] kasan: rework krealloc tests From: Andrey Konovalov To: Catalin Marinas , Vincenzo Frascino , Dmitry Vyukov , Alexander Potapenko , Marco Elver X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210201_144357_949442_C9BD249C X-CRM114-Status: GOOD ( 13.98 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Branislav Rankov , Andrey Konovalov , Kevin Brodsky , Will Deacon , linux-kernel@vger.kernel.org, kasan-dev@googlegroups.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, Andrey Ryabinin , Andrew Morton , Peter Collingbourne , Evgenii Stepanov Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org This patch reworks KASAN-KUnit tests for krealloc() to: 1. Check both slab and page_alloc based krealloc() implementations. 2. Allow at least one full granule to fit between old and new sizes for each KASAN mode, and check accesses to that granule accordingly. Signed-off-by: Andrey Konovalov Reviewed-by: Marco Elver --- lib/test_kasan.c | 91 ++++++++++++++++++++++++++++++++++++++++++------ 1 file changed, 81 insertions(+), 10 deletions(-) diff --git a/lib/test_kasan.c b/lib/test_kasan.c index 5699e43ca01b..2bb52853f341 100644 --- a/lib/test_kasan.c +++ b/lib/test_kasan.c @@ -258,11 +258,14 @@ static void kmalloc_large_oob_right(struct kunit *test) kfree(ptr); } -static void kmalloc_oob_krealloc_more(struct kunit *test) +static void krealloc_more_oob_helper(struct kunit *test, + size_t size1, size_t size2) { char *ptr1, *ptr2; - size_t size1 = 17; - size_t size2 = 19; + size_t middle; + + KUNIT_ASSERT_LT(test, size1, size2); + middle = size1 + (size2 - size1) / 2; ptr1 = kmalloc(size1, GFP_KERNEL); KUNIT_ASSERT_NOT_ERR_OR_NULL(test, ptr1); @@ -270,15 +273,31 @@ static void kmalloc_oob_krealloc_more(struct kunit *test) ptr2 = krealloc(ptr1, size2, GFP_KERNEL); KUNIT_ASSERT_NOT_ERR_OR_NULL(test, ptr2); - KUNIT_EXPECT_KASAN_FAIL(test, ptr2[size2 + OOB_TAG_OFF] = 'x'); + /* All offsets up to size2 must be accessible. */ + ptr2[size1 - 1] = 'x'; + ptr2[size1] = 'x'; + ptr2[middle] = 'x'; + ptr2[size2 - 1] = 'x'; + + /* Generic mode is precise, so unaligned size2 must be inaccessible. */ + if (IS_ENABLED(CONFIG_KASAN_GENERIC)) + KUNIT_EXPECT_KASAN_FAIL(test, ptr2[size2] = 'x'); + + /* For all modes first aligned offset after size2 must be inaccessible. */ + KUNIT_EXPECT_KASAN_FAIL(test, + ptr2[round_up(size2, KASAN_GRANULE_SIZE)] = 'x'); + kfree(ptr2); } -static void kmalloc_oob_krealloc_less(struct kunit *test) +static void krealloc_less_oob_helper(struct kunit *test, + size_t size1, size_t size2) { char *ptr1, *ptr2; - size_t size1 = 17; - size_t size2 = 15; + size_t middle; + + KUNIT_ASSERT_LT(test, size2, size1); + middle = size2 + (size1 - size2) / 2; ptr1 = kmalloc(size1, GFP_KERNEL); KUNIT_ASSERT_NOT_ERR_OR_NULL(test, ptr1); @@ -286,10 +305,60 @@ static void kmalloc_oob_krealloc_less(struct kunit *test) ptr2 = krealloc(ptr1, size2, GFP_KERNEL); KUNIT_ASSERT_NOT_ERR_OR_NULL(test, ptr2); - KUNIT_EXPECT_KASAN_FAIL(test, ptr2[size2 + OOB_TAG_OFF] = 'x'); + /* Must be accessible for all modes. */ + ptr2[size2 - 1] = 'x'; + + /* Generic mode is precise, so unaligned size2 must be inaccessible. */ + if (IS_ENABLED(CONFIG_KASAN_GENERIC)) + KUNIT_EXPECT_KASAN_FAIL(test, ptr2[size2] = 'x'); + + /* For all modes first aligned offset after size2 must be inaccessible. */ + KUNIT_EXPECT_KASAN_FAIL(test, + ptr2[round_up(size2, KASAN_GRANULE_SIZE)] = 'x'); + + /* + * For all modes both middle and size1 should land in separate granules + * and thus be inaccessible. + */ + KUNIT_EXPECT_LE(test, round_up(size2, KASAN_GRANULE_SIZE), + round_down(middle, KASAN_GRANULE_SIZE)); + KUNIT_EXPECT_LE(test, round_up(middle, KASAN_GRANULE_SIZE), + round_down(size1, KASAN_GRANULE_SIZE)); + KUNIT_EXPECT_KASAN_FAIL(test, ptr2[middle] = 'x'); + KUNIT_EXPECT_KASAN_FAIL(test, ptr2[size1 - 1] = 'x'); + KUNIT_EXPECT_KASAN_FAIL(test, ptr2[size1] = 'x'); + kfree(ptr2); } +static void krealloc_more_oob(struct kunit *test) +{ + krealloc_more_oob_helper(test, 201, 235); +} + +static void krealloc_less_oob(struct kunit *test) +{ + krealloc_less_oob_helper(test, 235, 201); +} + +static void krealloc_pagealloc_more_oob(struct kunit *test) +{ + /* page_alloc fallback in only implemented for SLUB. */ + KASAN_TEST_NEEDS_CONFIG_ON(test, CONFIG_SLUB); + + krealloc_more_oob_helper(test, KMALLOC_MAX_CACHE_SIZE + 201, + KMALLOC_MAX_CACHE_SIZE + 235); +} + +static void krealloc_pagealloc_less_oob(struct kunit *test) +{ + /* page_alloc fallback in only implemented for SLUB. */ + KASAN_TEST_NEEDS_CONFIG_ON(test, CONFIG_SLUB); + + krealloc_less_oob_helper(test, KMALLOC_MAX_CACHE_SIZE + 235, + KMALLOC_MAX_CACHE_SIZE + 201); +} + static void kmalloc_oob_16(struct kunit *test) { struct { @@ -983,8 +1052,10 @@ static struct kunit_case kasan_kunit_test_cases[] = { KUNIT_CASE(pagealloc_oob_right), KUNIT_CASE(pagealloc_uaf), KUNIT_CASE(kmalloc_large_oob_right), - KUNIT_CASE(kmalloc_oob_krealloc_more), - KUNIT_CASE(kmalloc_oob_krealloc_less), + KUNIT_CASE(krealloc_more_oob), + KUNIT_CASE(krealloc_less_oob), + KUNIT_CASE(krealloc_pagealloc_more_oob), + KUNIT_CASE(krealloc_pagealloc_less_oob), KUNIT_CASE(kmalloc_oob_16), KUNIT_CASE(kmalloc_uaf_16), KUNIT_CASE(kmalloc_oob_in_memset), From patchwork Mon Feb 1 19:43:31 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrey Konovalov X-Patchwork-Id: 12059875 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.1 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6BD83C43381 for ; Mon, 1 Feb 2021 19:46:34 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 9766F64EA9 for ; Mon, 1 Feb 2021 19:46:33 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 9766F64EA9 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:To:From:Subject:References:Mime-Version:Message-Id: In-Reply-To:Date:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=hbPBjYRnDd0JigRIWqVG/DKld7U5MI13NEHP1r0Mkjg=; b=wKTLzApub8/42acXP3VD5BJM5 +qpHhrihvMsClVKQlhwK52qMeXnXxSw/SVCMRPcCEJo5n81Aa/3lz/vGK7GVu1blpbknsVfJQ9P0V E/rjR9STwb7Soo934P+4JRp3nlA+0lTs1k9vbzqVfinBh6ZFzuoSnkK9rP7xkd0rn7GaOGY2fsA2U ZXGaiLx3Jq4bcVzbk6qfdNGlhop531PjGPjZshOMWuPe5wcucCaHI+bQ+tyCo1Fm0brgNvEr6osyS 1DVyrQm+l6Xrkhmr47ZHWO4fmzodsXLV60obrqs0ScQb9vM+GzEDkKMoldV7if6Z9ehv0YEAXpB9F 8ptTtuu4g==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1l6f7r-0002KJ-Oh; Mon, 01 Feb 2021 19:44:43 +0000 Received: from mail-wm1-x349.google.com ([2a00:1450:4864:20::349]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1l6f7A-0001vj-Fk for linux-arm-kernel@lists.infradead.org; Mon, 01 Feb 2021 19:44:01 +0000 Received: by mail-wm1-x349.google.com with SMTP id z188so393010wme.1 for ; Mon, 01 Feb 2021 11:43:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=yMQgrLxhHIKq4G0a2NCbXbKyDrvusdwY4yeazjCtuyk=; b=ivESTfLfXADjygdQiFNXCX+wPXeMmmyDL7MhSMwYXQFbjxX6om64VWAa9DB0UHgjky FA0IkQ9/NQQQM1j6Es5XovdS1v0MO0KZldXf+YBmCCZn47B20vO09U3YN5ZVtPh9slfU V0695NxHTqBxob7ObPmJFRiD3DbtZI77vvKRyogyBNt+B0g1mlTTZfcvclio6T9srFgt DeT+ymo2vjail6bFhgkunaozgr3fW+qjwQ7TMJkpHXXpDU0/3qtwo9c7+nWPQ+JBZxK0 m6EaK0Hm5acOgUfe9nl3oqyOCXvObbHtXRzj+nbFuJQYf48336Ff8i+FpKXIIffMt9oQ NOjQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=yMQgrLxhHIKq4G0a2NCbXbKyDrvusdwY4yeazjCtuyk=; b=JwXxUBV7HbbJZy+s2sTSzRyRMR6qfoslKM0sZ5KvJaXvrtidbCeYaOp8Iy1g978BNY 0UC3CKDLzZrN/PeumCkqmrUc8PTQxhbd95ulXYF1bKrpduQi6NoLaetHTwhwJ1nKx7Z+ EJDFw0UROWMFo/VpZpQdDjSPJpLia/KEv/GeN6cLdICRXya07NFsOqgaKDrViw1CRXmZ VWXmcIkOx5+P6CVmrXqE6odE+DwhxvgdkIeQWTHo1XWLQlqpcvnppCesa9C72LLk7ARp pPeJWHKalOMwjwCZ3PXkVV/mc0OAmekexFKkK1aurGKmm2wRqs8TUzv2Va5oi1LZ6dVU smxA== X-Gm-Message-State: AOAM532XAWnQTQYq7p4+KCPvtI8oS9mN1PYB1YAYYxCHFAQbuXyG1Utv CaZHUzpBKjh1JCaVNWpX8kBal5+mr4jWOYBo X-Google-Smtp-Source: ABdhPJzOa+LccFfH9kBnsOeOGuqeG8GyRvo7cSnKb8yeZh67FwLtjenT5WmmmRbIZ1nG1AOskfgvEC22yE1+sKw/ X-Received: from andreyknvl3.muc.corp.google.com ([2a00:79e0:15:13:7220:84ff:fe09:7e9d]) (user=andreyknvl job=sendgmr) by 2002:a1c:7711:: with SMTP id t17mr441041wmi.64.1612208636543; Mon, 01 Feb 2021 11:43:56 -0800 (PST) Date: Mon, 1 Feb 2021 20:43:31 +0100 In-Reply-To: Message-Id: <884e37ddff31b671725f4d83106111c7dcf8fb9b.1612208222.git.andreyknvl@google.com> Mime-Version: 1.0 References: X-Mailer: git-send-email 2.30.0.365.g02bc693789-goog Subject: [PATCH 07/12] kasan, mm: remove krealloc side-effect From: Andrey Konovalov To: Catalin Marinas , Vincenzo Frascino , Dmitry Vyukov , Alexander Potapenko , Marco Elver X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210201_144400_618045_AA8DE9DA X-CRM114-Status: GOOD ( 17.49 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Branislav Rankov , Andrey Konovalov , Kevin Brodsky , Will Deacon , linux-kernel@vger.kernel.org, kasan-dev@googlegroups.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, Andrey Ryabinin , Andrew Morton , Peter Collingbourne , Evgenii Stepanov Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Currently, if krealloc() is called on a freed object with KASAN enabled, it allocates and returns a new object, but doesn't copy any memory from the old one as ksize() returns 0. This makes a caller believe that krealloc() succeeded (KASAN report is printed though). This patch adds an accessibility check into __do_krealloc(). If the check fails, krealloc() returns NULL. This check duplicates the one in ksize(); this is fixed in the following patch. This patch also adds a KASAN-KUnit test to check krealloc() behaviour when it's called on a freed object. Signed-off-by: Andrey Konovalov Reviewed-by: Marco Elver --- lib/test_kasan.c | 20 ++++++++++++++++++++ mm/slab_common.c | 3 +++ 2 files changed, 23 insertions(+) diff --git a/lib/test_kasan.c b/lib/test_kasan.c index 2bb52853f341..61bc894d9f7e 100644 --- a/lib/test_kasan.c +++ b/lib/test_kasan.c @@ -359,6 +359,25 @@ static void krealloc_pagealloc_less_oob(struct kunit *test) KMALLOC_MAX_CACHE_SIZE + 201); } +/* + * Check that krealloc() detects a use-after-free, returns NULL, + * and doesn't unpoison the freed object. + */ +static void krealloc_uaf(struct kunit *test) +{ + char *ptr1, *ptr2; + int size1 = 201; + int size2 = 235; + + ptr1 = kmalloc(size1, GFP_KERNEL); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, ptr1); + kfree(ptr1); + + KUNIT_EXPECT_KASAN_FAIL(test, ptr2 = krealloc(ptr1, size2, GFP_KERNEL)); + KUNIT_ASSERT_PTR_EQ(test, (void *)ptr2, NULL); + KUNIT_EXPECT_KASAN_FAIL(test, *(volatile char *)ptr1); +} + static void kmalloc_oob_16(struct kunit *test) { struct { @@ -1056,6 +1075,7 @@ static struct kunit_case kasan_kunit_test_cases[] = { KUNIT_CASE(krealloc_less_oob), KUNIT_CASE(krealloc_pagealloc_more_oob), KUNIT_CASE(krealloc_pagealloc_less_oob), + KUNIT_CASE(krealloc_uaf), KUNIT_CASE(kmalloc_oob_16), KUNIT_CASE(kmalloc_uaf_16), KUNIT_CASE(kmalloc_oob_in_memset), diff --git a/mm/slab_common.c b/mm/slab_common.c index 39d1a8ff9bb8..dad70239b54c 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -1140,6 +1140,9 @@ static __always_inline void *__do_krealloc(const void *p, size_t new_size, void *ret; size_t ks; + if (likely(!ZERO_OR_NULL_PTR(p)) && !kasan_check_byte(p)) + return NULL; + ks = ksize(p); if (ks >= new_size) { From patchwork Mon Feb 1 19:43:32 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrey Konovalov X-Patchwork-Id: 12059877 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.1 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8CA97C433DB for ; Mon, 1 Feb 2021 19:46:40 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 036A464E3C for ; Mon, 1 Feb 2021 19:46:39 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 036A464E3C Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:To:From:Subject:References:Mime-Version:Message-Id: In-Reply-To:Date:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=7AMtUWa/BLz2OkfkN6Esy7QDmet9Z2Rje4xkPp65Xqc=; b=SV+lCXvSd4XSQHTF6FWFOeuQ+ ZGR/dP+8QhrA6V7GqpqyirUoYI/oM7LDuVMD8cwMfx6A7K/vL/VUa4Nyl+QZzxNJrQit/vtokbNd4 6tGyQV7u6vXZYgNGObyNOFRPgaw41+AflS/8ei4kxv/kZhjLvCm3PBXFVvq/X1uEDNPiFWfanUo2X 08f6CODq8vKM98qS9C9Z/pvv4lKUeuQDfAJpvQXLO60eCdzsMFeeS1tEZ85SNJnewzDuKRBn7NmNl jTIKXg6cfYl0xIxVE5VxVNHsnJQJ/Mj7N0vAtvYGsHkGd/2e+S0CzPBuN2iokXTHdG35YVPHlpWYL NPvIOTdZw==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1l6f7x-0002Nm-OV; Mon, 01 Feb 2021 19:44:49 +0000 Received: from mail-qt1-x84a.google.com ([2607:f8b0:4864:20::84a]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1l6f7C-0001xV-Ol for linux-arm-kernel@lists.infradead.org; Mon, 01 Feb 2021 19:44:04 +0000 Received: by mail-qt1-x84a.google.com with SMTP id d10so11428102qtx.8 for ; Mon, 01 Feb 2021 11:44:00 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=wTS7rGEWy/lmgEwVXVBE+5vzcvue/e9GmeYKwLI3/GI=; b=NM/EWWXfIO5X0CDVwGsXDCTqh8MrojXHR19eXM6kgDQheLb3UgxElTkZafjlwvJVMh hRX5SjP8nw46ZrfNSlgGwyvk51E5mCcJjGRICuKoVrBb2YwSa9i4QDptcJtNxcTGLJrj CeY1HpjU4xlSLRVgBGfO5+GLBbY3Lp0oNfGToUpZKWmAgO6jg6PmK+eLPiDFfbniYtjj 0oFFptb75/P6D059sJVp3xp5O4JXmkbYqtDH8y1wuU6wR8rq4N02//J4aewnw8CWdPfK zE529RUGZuEtVrAuesBdVfQ/DsQqo1DTN6iIqW3q4M7STnfb18Ldteve0sFZw56OEXzt nmdw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=wTS7rGEWy/lmgEwVXVBE+5vzcvue/e9GmeYKwLI3/GI=; b=jUjCGlnk2IvgHw1n5mP+jTbEq34vHZLjmGEvLg24CcSFKIp5dMLO5Ln8+ZjzMdvyWm o0ZHo58G0Ui05LdoOIeM+egGguGGYGzDXiFoO5t/Sy+sW42EksgKW4mH1P14Rv0pDMPj I0bf3ZMPNn9f7xAe7kS7X5KJ4SEtVuF5EBv2Q2ez0sqdTLRQaAWWPGdhcICJ9ViUXwtC MbxQfyPK/1DeJ82EA0g7JnWIiCVgq9JvpDsWxzCUSZ1wzojA6+kvN2REw1VbnHf9hIRL ZpwUDJlw3YEBYXF0GpakjF9ghBiBG4HCdj/jOOVHf41UhF6yfaFtHBqFiP5mEbryLgy9 JlbA== X-Gm-Message-State: AOAM5337pSaHsGIfqvY84hxrGBg+kA2RTlUZAMS/gOh7HBlWc+LIbUKn SSn9QEcJRcS2eQHdMGCBiFsNTTGlXOkx/yNE X-Google-Smtp-Source: ABdhPJzxW58GoVDg9DXe3VlIIu6Zz+FBlV301B0hIass2XkT/B7p+K717pXSiFgUHOf4CH1xC9QxDtyYCzLn2SR8 X-Received: from andreyknvl3.muc.corp.google.com ([2a00:79e0:15:13:7220:84ff:fe09:7e9d]) (user=andreyknvl job=sendgmr) by 2002:a05:6214:1110:: with SMTP id e16mr16696575qvs.62.1612208638752; Mon, 01 Feb 2021 11:43:58 -0800 (PST) Date: Mon, 1 Feb 2021 20:43:32 +0100 In-Reply-To: Message-Id: <431c6cfa0ac8fb2b33d7ab561a64aa84c844d1a0.1612208222.git.andreyknvl@google.com> Mime-Version: 1.0 References: X-Mailer: git-send-email 2.30.0.365.g02bc693789-goog Subject: [PATCH 08/12] kasan, mm: optimize krealloc poisoning From: Andrey Konovalov To: Catalin Marinas , Vincenzo Frascino , Dmitry Vyukov , Alexander Potapenko , Marco Elver X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210201_144402_891781_7C77D0AD X-CRM114-Status: GOOD ( 20.51 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Branislav Rankov , Andrey Konovalov , Kevin Brodsky , Will Deacon , linux-kernel@vger.kernel.org, kasan-dev@googlegroups.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, Andrey Ryabinin , Andrew Morton , Peter Collingbourne , Evgenii Stepanov Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Currently, krealloc() always calls ksize(), which unpoisons the whole object including the redzone. This is inefficient, as kasan_krealloc() repoisons the redzone for objects that fit into the same buffer. This patch changes krealloc() instrumentation to use uninstrumented __ksize() that doesn't unpoison the memory. Instead, kasan_kreallos() is changed to unpoison the memory excluding the redzone. For objects that don't fit into the old allocation, this patch disables KASAN accessibility checks when copying memory into a new object instead of unpoisoning it. Signed-off-by: Andrey Konovalov --- mm/kasan/common.c | 12 ++++++++++-- mm/slab_common.c | 20 ++++++++++++++------ 2 files changed, 24 insertions(+), 8 deletions(-) diff --git a/mm/kasan/common.c b/mm/kasan/common.c index 9c64a00bbf9c..a51d6ea580b0 100644 --- a/mm/kasan/common.c +++ b/mm/kasan/common.c @@ -476,7 +476,7 @@ static void *____kasan_kmalloc(struct kmem_cache *cache, const void *object, /* * The object has already been unpoisoned by kasan_slab_alloc() for - * kmalloc() or by ksize() for krealloc(). + * kmalloc() or by kasan_krealloc() for krealloc(). */ /* @@ -526,7 +526,7 @@ void * __must_check __kasan_kmalloc_large(const void *ptr, size_t size, /* * The object has already been unpoisoned by kasan_alloc_pages() for - * alloc_pages() or by ksize() for krealloc(). + * alloc_pages() or by kasan_krealloc() for krealloc(). */ /* @@ -554,8 +554,16 @@ void * __must_check __kasan_krealloc(const void *object, size_t size, gfp_t flag if (unlikely(object == ZERO_SIZE_PTR)) return (void *)object; + /* + * Unpoison the object's data. + * Part of it might already have been unpoisoned, but it's unknown + * how big that part is. + */ + kasan_unpoison(object, size); + page = virt_to_head_page(object); + /* Piggy-back on kmalloc() instrumentation to poison the redzone. */ if (unlikely(!PageSlab(page))) return __kasan_kmalloc_large(object, size, flags); else diff --git a/mm/slab_common.c b/mm/slab_common.c index dad70239b54c..821f657d38b5 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -1140,19 +1140,27 @@ static __always_inline void *__do_krealloc(const void *p, size_t new_size, void *ret; size_t ks; - if (likely(!ZERO_OR_NULL_PTR(p)) && !kasan_check_byte(p)) - return NULL; - - ks = ksize(p); + /* Don't use instrumented ksize to allow precise KASAN poisoning. */ + if (likely(!ZERO_OR_NULL_PTR(p))) { + if (!kasan_check_byte(p)) + return NULL; + ks = __ksize(p); + } else + ks = 0; + /* If the object still fits, repoison it precisely. */ if (ks >= new_size) { p = kasan_krealloc((void *)p, new_size, flags); return (void *)p; } ret = kmalloc_track_caller(new_size, flags); - if (ret && p) - memcpy(ret, p, ks); + if (ret && p) { + /* Disable KASAN checks as the object's redzone is accessed. */ + kasan_disable_current(); + memcpy(ret, kasan_reset_tag(p), ks); + kasan_enable_current(); + } return ret; } From patchwork Mon Feb 1 19:43:33 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrey Konovalov X-Patchwork-Id: 12059955 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.1 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id F3169C43381 for ; Mon, 1 Feb 2021 19:47:06 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 84B8B64E54 for ; Mon, 1 Feb 2021 19:47:06 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 84B8B64E54 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:To:From:Subject:References:Mime-Version:Message-Id: In-Reply-To:Date:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=K2f1fHXvYXWYfodCBPlR6DZingVTC9jSFQiVIfP6EnE=; b=doBcnEnlWF9PIXgSXWHD1CDn/ YSnupHmxsxVkJFElsyq4BWv0CERDIuWfa4DK6UxGEjn7BzlmU4R3tBkVd3csRUKCnAOgxFf2rCaFL ElK+gFXyMk8s7kmlpzPqGNwmYTROP8bKuyR6WX8pZc/gTTmg/5+vp+UAqksNXj4E9b7WEoroC3Pkx 36oYxlOC8Hrju9g2JcRfUpJC7drV7UTb+0CamaM74W1UsZ08ZT9/bTtbJDFvJ2Vz4QJgxqJh4naEJ KtfXf/WQNg3HsKe8bYx5EwUL5ZV1e1gIx/rvXsbok6e/YHTRc5748qwCiJ3A9DV6/FQqCAclm4Qi+ ztBOxQD9Q==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1l6f8D-0002Sa-MR; Mon, 01 Feb 2021 19:45:06 +0000 Received: from mail-qk1-x74a.google.com ([2607:f8b0:4864:20::74a]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1l6f7F-0001yt-Se for linux-arm-kernel@lists.infradead.org; Mon, 01 Feb 2021 19:44:09 +0000 Received: by mail-qk1-x74a.google.com with SMTP id e5so14129557qkn.2 for ; Mon, 01 Feb 2021 11:44:02 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=yIaCjepZTR5qWD3z/y90ARyLWJ/KolY2qLhEkXBJOWM=; b=LzK81n8jRDSKSrwC/tHWoCW1SthbpuTfYnfZFmsFHmvHJTYsV4KqUATa9ZGrPrvBBc oB529UpqdDeoRp4v/vVhBEGgfPBT+1NXWdYX5Xvn/kvC+ORem0DCxpCa3jI7hJSgzYEc v8XlwQYe5HeTYBz2VJ9kbwXxPh5dTuaEPFLq+BlD0tk6wRDHhS/uZ3D7uynGWY5820LH nWpqqdOVnH/n1NoSsn+bx/H1xMABqaI0urla3+6oBMVUxG+A9o43btOQxjxE8/PeEhVu mmEmr74grVtfW5johenPyPm1luTjFhX1AtLwj0Cydk2Rb6p+inUgV1CkAFZX/D9xmjfV 3vZg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=yIaCjepZTR5qWD3z/y90ARyLWJ/KolY2qLhEkXBJOWM=; b=VJT9uMiy3fKGXSL5n61mvWHjPaIKzvl5dUEKlK5b+Dlfp7iwPaLZhrM/EJFbEMsPR1 ylnSBusaw670btWAfFb3xV6NeLTc0IlUlJa5zn9Ud4gBUjsIwUgHkJy+vMo/ECZb4JXH CnAXE2+nIQ7fZQMuPaXtMvwEnvJe3zDdE3KFmeXzZMgQHmxJw1fe9efvs6sBdy7Jjxbx /thmck6Bl3L0RBRGIvXnNPPGFqtOb2wvVwOy7pvOKbOl4utbTywkzlRw16ZL51UcGjlO AjkesqkXvDjIzVxCEY0a/uEkVR9P4DG1c9bYQ8wO39+mHP0ru7sfXBe5xKn8/mcATyGW Yd+Q== X-Gm-Message-State: AOAM531MiIksAvIyfOr2Iu6RkjGG44fbGmToEjRmvKDF2SfJmi13YUmg FJuL0jonPjZI9qogd0xdKq5FyhtTtd/j6AnJ X-Google-Smtp-Source: ABdhPJw6YkwwBeDFx/5q0zc31oTDJ4pRWKYTc7OTHoCBPXE314XQpal0iqmdPiewIdDNyJd1sipvAehw/0Vpjy2b X-Received: from andreyknvl3.muc.corp.google.com ([2a00:79e0:15:13:7220:84ff:fe09:7e9d]) (user=andreyknvl job=sendgmr) by 2002:a0c:efd2:: with SMTP id a18mr16703061qvt.7.1612208641117; Mon, 01 Feb 2021 11:44:01 -0800 (PST) Date: Mon, 1 Feb 2021 20:43:33 +0100 In-Reply-To: Message-Id: Mime-Version: 1.0 References: X-Mailer: git-send-email 2.30.0.365.g02bc693789-goog Subject: [PATCH 09/12] kasan: ensure poisoning size alignment From: Andrey Konovalov To: Catalin Marinas , Vincenzo Frascino , Dmitry Vyukov , Alexander Potapenko , Marco Elver X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210201_144406_045424_D4B7C57B X-CRM114-Status: GOOD ( 18.56 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Branislav Rankov , Andrey Konovalov , Kevin Brodsky , Will Deacon , linux-kernel@vger.kernel.org, kasan-dev@googlegroups.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, Andrey Ryabinin , Andrew Morton , Peter Collingbourne , Evgenii Stepanov Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org A previous changes d99f6a10c161 ("kasan: don't round_up too much") attempted to simplify the code by adding a round_up(size) call into kasan_poison(). While this allows to have less round_up() calls around the code, this results in round_up() being called multiple times. This patch removes round_up() of size from kasan_poison() and ensures that all callers round_up() the size explicitly. This patch also adds WARN_ON() alignment checks for address and size to kasan_poison() and kasan_unpoison(). Signed-off-by: Andrey Konovalov Reviewed-by: Marco Elver --- mm/kasan/common.c | 9 ++++++--- mm/kasan/kasan.h | 33 ++++++++++++++++++++------------- mm/kasan/shadow.c | 37 ++++++++++++++++++++++--------------- 3 files changed, 48 insertions(+), 31 deletions(-) diff --git a/mm/kasan/common.c b/mm/kasan/common.c index a51d6ea580b0..5691cca69397 100644 --- a/mm/kasan/common.c +++ b/mm/kasan/common.c @@ -261,7 +261,8 @@ void __kasan_unpoison_object_data(struct kmem_cache *cache, void *object) void __kasan_poison_object_data(struct kmem_cache *cache, void *object) { - kasan_poison(object, cache->object_size, KASAN_KMALLOC_REDZONE); + kasan_poison(object, round_up(cache->object_size, KASAN_GRANULE_SIZE), + KASAN_KMALLOC_REDZONE); } /* @@ -348,7 +349,8 @@ static bool ____kasan_slab_free(struct kmem_cache *cache, void *object, return true; } - kasan_poison(object, cache->object_size, KASAN_KMALLOC_FREE); + kasan_poison(object, round_up(cache->object_size, KASAN_GRANULE_SIZE), + KASAN_KMALLOC_FREE); if ((IS_ENABLED(CONFIG_KASAN_GENERIC) && !quarantine)) return false; @@ -490,7 +492,8 @@ static void *____kasan_kmalloc(struct kmem_cache *cache, const void *object, /* Poison the aligned part of the redzone. */ redzone_start = round_up((unsigned long)(object + size), KASAN_GRANULE_SIZE); - redzone_end = (unsigned long)object + cache->object_size; + redzone_end = round_up((unsigned long)(object + cache->object_size), + KASAN_GRANULE_SIZE); kasan_poison((void *)redzone_start, redzone_end - redzone_start, KASAN_KMALLOC_REDZONE); diff --git a/mm/kasan/kasan.h b/mm/kasan/kasan.h index 6a2882997f23..2f7400a3412f 100644 --- a/mm/kasan/kasan.h +++ b/mm/kasan/kasan.h @@ -321,30 +321,37 @@ static inline u8 kasan_random_tag(void) { return 0; } #ifdef CONFIG_KASAN_HW_TAGS -static inline void kasan_poison(const void *address, size_t size, u8 value) +static inline void kasan_poison(const void *addr, size_t size, u8 value) { - address = kasan_reset_tag(address); + addr = kasan_reset_tag(addr); /* Skip KFENCE memory if called explicitly outside of sl*b. */ - if (is_kfence_address(address)) + if (is_kfence_address(addr)) return; - hw_set_mem_tag_range((void *)address, - round_up(size, KASAN_GRANULE_SIZE), value); + if (WARN_ON((u64)addr & KASAN_GRANULE_MASK)) + return; + if (WARN_ON(size & KASAN_GRANULE_MASK)) + return; + + hw_set_mem_tag_range((void *)addr, size, value); } -static inline void kasan_unpoison(const void *address, size_t size) +static inline void kasan_unpoison(const void *addr, size_t size) { - u8 tag = get_tag(address); + u8 tag = get_tag(addr); - address = kasan_reset_tag(address); + addr = kasan_reset_tag(addr); /* Skip KFENCE memory if called explicitly outside of sl*b. */ - if (is_kfence_address(address)) + if (is_kfence_address(addr)) return; - hw_set_mem_tag_range((void *)address, - round_up(size, KASAN_GRANULE_SIZE), tag); + if (WARN_ON((u64)addr & KASAN_GRANULE_MASK)) + return; + size = round_up(size, KASAN_GRANULE_SIZE); + + hw_set_mem_tag_range((void *)addr, size, tag); } static inline bool kasan_byte_accessible(const void *addr) @@ -361,7 +368,7 @@ static inline bool kasan_byte_accessible(const void *addr) /** * kasan_poison - mark the memory range as unaccessible * @addr - range start address, must be aligned to KASAN_GRANULE_SIZE - * @size - range size + * @size - range size, must be aligned to KASAN_GRANULE_SIZE * @value - value that's written to metadata for the range * * The size gets aligned to KASAN_GRANULE_SIZE before marking the range. @@ -371,7 +378,7 @@ void kasan_poison(const void *addr, size_t size, u8 value); /** * kasan_unpoison - mark the memory range as accessible * @addr - range start address, must be aligned to KASAN_GRANULE_SIZE - * @size - range size + * @size - range size, can be unaligned * * For the tag-based modes, the @size gets aligned to KASAN_GRANULE_SIZE before * marking the range. diff --git a/mm/kasan/shadow.c b/mm/kasan/shadow.c index 1ed7817e4ee6..c97f51c557ea 100644 --- a/mm/kasan/shadow.c +++ b/mm/kasan/shadow.c @@ -69,7 +69,7 @@ void *memcpy(void *dest, const void *src, size_t len) return __memcpy(dest, src, len); } -void kasan_poison(const void *address, size_t size, u8 value) +void kasan_poison(const void *addr, size_t size, u8 value) { void *shadow_start, *shadow_end; @@ -78,55 +78,62 @@ void kasan_poison(const void *address, size_t size, u8 value) * some of the callers (e.g. kasan_poison_object_data) pass tagged * addresses to this function. */ - address = kasan_reset_tag(address); + addr = kasan_reset_tag(addr); /* Skip KFENCE memory if called explicitly outside of sl*b. */ - if (is_kfence_address(address)) + if (is_kfence_address(addr)) return; - size = round_up(size, KASAN_GRANULE_SIZE); - shadow_start = kasan_mem_to_shadow(address); - shadow_end = kasan_mem_to_shadow(address + size); + if (WARN_ON((u64)addr & KASAN_GRANULE_MASK)) + return; + if (WARN_ON(size & KASAN_GRANULE_MASK)) + return; + + shadow_start = kasan_mem_to_shadow(addr); + shadow_end = kasan_mem_to_shadow(addr + size); __memset(shadow_start, value, shadow_end - shadow_start); } EXPORT_SYMBOL(kasan_poison); #ifdef CONFIG_KASAN_GENERIC -void kasan_poison_last_granule(const void *address, size_t size) +void kasan_poison_last_granule(const void *addr, size_t size) { if (size & KASAN_GRANULE_MASK) { - u8 *shadow = (u8 *)kasan_mem_to_shadow(address + size); + u8 *shadow = (u8 *)kasan_mem_to_shadow(addr + size); *shadow = size & KASAN_GRANULE_MASK; } } #endif -void kasan_unpoison(const void *address, size_t size) +void kasan_unpoison(const void *addr, size_t size) { - u8 tag = get_tag(address); + u8 tag = get_tag(addr); /* * Perform shadow offset calculation based on untagged address, as * some of the callers (e.g. kasan_unpoison_object_data) pass tagged * addresses to this function. */ - address = kasan_reset_tag(address); + addr = kasan_reset_tag(addr); /* * Skip KFENCE memory if called explicitly outside of sl*b. Also note * that calls to ksize(), where size is not a multiple of machine-word * size, would otherwise poison the invalid portion of the word. */ - if (is_kfence_address(address)) + if (is_kfence_address(addr)) + return; + + if (WARN_ON((u64)addr & KASAN_GRANULE_MASK)) return; - /* Unpoison round_up(size, KASAN_GRANULE_SIZE) bytes. */ - kasan_poison(address, size, tag); + /* Unpoison all granules that cover the object. */ + kasan_poison(addr, round_up(size, KASAN_GRANULE_SIZE), tag); /* Partially poison the last granule for the generic mode. */ if (IS_ENABLED(CONFIG_KASAN_GENERIC)) - kasan_poison_last_granule(address, size); + kasan_poison_last_granule(addr, size); } #ifdef CONFIG_MEMORY_HOTPLUG From patchwork Mon Feb 1 19:43:34 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrey Konovalov X-Patchwork-Id: 12059959 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.1 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 09771C433E0 for ; Mon, 1 Feb 2021 19:47:28 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 9E92C64E3C for ; Mon, 1 Feb 2021 19:47:27 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 9E92C64E3C Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:To:From:Subject:References:Mime-Version:Message-Id: In-Reply-To:Date:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=FfIDpabi5GDoo7PqtP1PhRUntlQG6SXd4y5w0IyP2ts=; b=z0swR780mexW59IUBCNrOm3qW ERc+pxoYhjYFDoIXDv7SVDwlRW51B17dGugAAz15dO+KOv898DlTVaoGczeQwLmF16ff/xV0qwOIK E+Nfg0FdgmwZcBGCyP+QLia9JFoUgA+NYmLFM2ZllrCJY4obw5DXjY5g4X7UJ9xCH3UfYblLDHiJs ZP15xOTIraHj9m/wTR+GCzHuL+kUkQWuuhiMmwsmNd1f6ZSAMmWtKyDFy5MD0YVHnsIlcdwUrMztB vB2JcI2qu03r3krg6QEV7pCmrNMOIpSG83zNuZ/DGQbzXI+OvvjBAh5BoQnjpobn9btTRbq9AF0/r EhL9yLb+g==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1l6f8k-0002p1-7h; Mon, 01 Feb 2021 19:45:38 +0000 Received: from mail-qk1-x74a.google.com ([2607:f8b0:4864:20::74a]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1l6f7H-00020D-51 for linux-arm-kernel@lists.infradead.org; Mon, 01 Feb 2021 19:44:13 +0000 Received: by mail-qk1-x74a.google.com with SMTP id q7so10261774qkn.7 for ; Mon, 01 Feb 2021 11:44:04 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=Odkzh1LG/DFkpV7FBjHfcEglIWvDLvMk2053ZF0TmS8=; b=lF7topMIZZyUAMPDwbNSQ4IokD79ZbQnP4p85aPkPWNYrYdUGnhVGkX1XlIlJlGN/v 60POwouAVmOoJZqP7l9CLS7BUvwybx/oFs2L/ne7iJ39ReFFa18AhXXpmHqQ/lI8hjPc iXC1G5mymCrA4y+k73BKjkX8gg0SLhGqD8Rtk/oVQk2XwtoF5FWSKcFvtQeRG5y1myfE D95+/cYum+MKroXLFIEhxiEAlmVrQDvpBa0/n6jWJNk7idpFTz+UA5n7JVGyofYwMfGI fnAv0FHmNT5TiBdf4sStkK01pblTRh0kbJmlaH6Ku+BQXmRXqzXsOfihkmt1xIJJigCi tv/A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=Odkzh1LG/DFkpV7FBjHfcEglIWvDLvMk2053ZF0TmS8=; b=tH3lM0giPRNeJ/5jB08KyH5rfIczzHCan+c0qYdlX2LYLldr8kVLksJhmn/QR4Gnx2 Wk1yOu5FJOTfg5Qi3ktq3iiaMUEajHPiGQl5zQi1jEgQ/Qkbw7hTIqwXsEO50t//bN6F aEWtCbI7JC3/kX4i95jOTu8T35YEApW+qNxiBgBpoX3lhCM8L161lqC6npDAQXKWNQNU URu4tIbfdtR9uJna+pvbEi+T/EjfccrEM3TJEP0U8V/PLLZKv4s6k7Mb8C5nixw+QkYF NjcQ6kJBZmKdKgxizowJgJsD5vY4GyZHTWXCLJZRWuiMxtPS278rlT2YPZFfO+5FIH+A oVlw== X-Gm-Message-State: AOAM530zFUJi6a8KbFWovX7lzHC++sIl8FEKdLKFwYeCrfxK7eWRp6As nf0YHGhHPQIVQxJ5Tk/meywXlDjooYNFBS1V X-Google-Smtp-Source: ABdhPJwwy203Wfi9k9SJwbsVg87njWeKNMJTs2w+lAuwP90F/9wvY20yPqJOEsyJ0pie5F8Rd3dN+mutFSzO0a2d X-Received: from andreyknvl3.muc.corp.google.com ([2a00:79e0:15:13:7220:84ff:fe09:7e9d]) (user=andreyknvl job=sendgmr) by 2002:ad4:5110:: with SMTP id g16mr16935423qvp.5.1612208643527; Mon, 01 Feb 2021 11:44:03 -0800 (PST) Date: Mon, 1 Feb 2021 20:43:34 +0100 In-Reply-To: Message-Id: <17d6bef698d193f5fe0d8baee0e232a351e23a32.1612208222.git.andreyknvl@google.com> Mime-Version: 1.0 References: X-Mailer: git-send-email 2.30.0.365.g02bc693789-goog Subject: [PATCH 10/12] arm64: kasan: simplify and inline MTE functions From: Andrey Konovalov To: Catalin Marinas , Vincenzo Frascino , Dmitry Vyukov , Alexander Potapenko , Marco Elver X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210201_144407_290894_70222E83 X-CRM114-Status: GOOD ( 19.73 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Branislav Rankov , Andrey Konovalov , Kevin Brodsky , Will Deacon , linux-kernel@vger.kernel.org, kasan-dev@googlegroups.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, Andrey Ryabinin , Andrew Morton , Peter Collingbourne , Evgenii Stepanov Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org This change provides a simpler implementation of mte_get_mem_tag(), mte_get_random_tag(), and mte_set_mem_tag_range(). Simplifications include removing system_supports_mte() checks as these functions are onlye called from KASAN runtime that had already checked system_supports_mte(). Besides that, size and address alignment checks are removed from mte_set_mem_tag_range(), as KASAN now does those. This change also moves these functions into the asm/mte-kasan.h header and implements mte_set_mem_tag_range() via inline assembly to avoid unnecessary functions calls. Co-developed-by: Vincenzo Frascino Signed-off-by: Andrey Konovalov Reviewed-by: Catalin Marinas Signed-off-by: Vincenzo Frascino --- arch/arm64/include/asm/cache.h | 1 - arch/arm64/include/asm/kasan.h | 1 + arch/arm64/include/asm/mte-def.h | 2 + arch/arm64/include/asm/mte-kasan.h | 64 ++++++++++++++++++++++++++---- arch/arm64/include/asm/mte.h | 2 - arch/arm64/kernel/mte.c | 46 --------------------- arch/arm64/lib/mte.S | 16 -------- 7 files changed, 60 insertions(+), 72 deletions(-) diff --git a/arch/arm64/include/asm/cache.h b/arch/arm64/include/asm/cache.h index 77cbbe3625f2..a074459f8f2f 100644 --- a/arch/arm64/include/asm/cache.h +++ b/arch/arm64/include/asm/cache.h @@ -6,7 +6,6 @@ #define __ASM_CACHE_H #include -#include #define CTR_L1IP_SHIFT 14 #define CTR_L1IP_MASK 3 diff --git a/arch/arm64/include/asm/kasan.h b/arch/arm64/include/asm/kasan.h index 0aaf9044cd6a..12d5f47f7dbe 100644 --- a/arch/arm64/include/asm/kasan.h +++ b/arch/arm64/include/asm/kasan.h @@ -6,6 +6,7 @@ #include #include +#include #include #define arch_kasan_set_tag(addr, tag) __tag_set(addr, tag) diff --git a/arch/arm64/include/asm/mte-def.h b/arch/arm64/include/asm/mte-def.h index 2d73a1612f09..cf241b0f0a42 100644 --- a/arch/arm64/include/asm/mte-def.h +++ b/arch/arm64/include/asm/mte-def.h @@ -11,4 +11,6 @@ #define MTE_TAG_SIZE 4 #define MTE_TAG_MASK GENMASK((MTE_TAG_SHIFT + (MTE_TAG_SIZE - 1)), MTE_TAG_SHIFT) +#define __MTE_PREAMBLE ARM64_ASM_PREAMBLE ".arch_extension memtag\n" + #endif /* __ASM_MTE_DEF_H */ diff --git a/arch/arm64/include/asm/mte-kasan.h b/arch/arm64/include/asm/mte-kasan.h index 8ad981069afb..1f090beda7e6 100644 --- a/arch/arm64/include/asm/mte-kasan.h +++ b/arch/arm64/include/asm/mte-kasan.h @@ -11,13 +11,16 @@ #include +#ifdef CONFIG_ARM64_MTE + /* - * The functions below are meant to be used only for the - * KASAN_HW_TAGS interface defined in asm/memory.h. + * These functions are meant to be only used from KASAN runtime through + * the arch_*() interface defined in asm/memory.h. + * These functions don't include system_supports_mte() checks, + * as KASAN only calls them when MTE is supported and enabled. */ -#ifdef CONFIG_ARM64_MTE -static inline u8 mte_get_ptr_tag(void *ptr) +static __always_inline u8 mte_get_ptr_tag(void *ptr) { /* Note: The format of KASAN tags is 0xF */ u8 tag = 0xF0 | (u8)(((u64)(ptr)) >> MTE_TAG_SHIFT); @@ -25,9 +28,54 @@ static inline u8 mte_get_ptr_tag(void *ptr) return tag; } -u8 mte_get_mem_tag(void *addr); -u8 mte_get_random_tag(void); -void *mte_set_mem_tag_range(void *addr, size_t size, u8 tag); +/* Get allocation tag for the address. */ +static __always_inline u8 mte_get_mem_tag(void *addr) +{ + asm(__MTE_PREAMBLE "ldg %0, [%0]" + : "+r" (addr)); + + return mte_get_ptr_tag(addr); +} + +/* Generate a random tag. */ +static __always_inline u8 mte_get_random_tag(void) +{ + void *addr; + + asm(__MTE_PREAMBLE "irg %0, %0" + : "+r" (addr)); + + return mte_get_ptr_tag(addr); +} + +/* + * Assign allocation tags for a region of memory based on the pointer tag. + * Note: The address must be non-NULL and MTE_GRANULE_SIZE aligned and + * size must be non-zero and MTE_GRANULE_SIZE aligned. + */ +static __always_inline void mte_set_mem_tag_range(void *addr, size_t size, u8 tag) +{ + u64 curr, end; + + if (!size) + return; + + curr = (u64)__tag_set(addr, tag); + end = curr + size; + + do { + /* + * 'asm volatile' is required to prevent the compiler to move + * the statement outside of the loop. + */ + asm volatile(__MTE_PREAMBLE "stg %0, [%0]" + : + : "r" (curr) + : "memory"); + + curr += MTE_GRANULE_SIZE; + } while (curr != end); +} void mte_enable_kernel_sync(void); void mte_enable_kernel_async(void); @@ -47,10 +95,12 @@ static inline u8 mte_get_mem_tag(void *addr) { return 0xFF; } + static inline u8 mte_get_random_tag(void) { return 0xFF; } + static inline void *mte_set_mem_tag_range(void *addr, size_t size, u8 tag) { return addr; diff --git a/arch/arm64/include/asm/mte.h b/arch/arm64/include/asm/mte.h index 237bb2f7309d..43169b978cd3 100644 --- a/arch/arm64/include/asm/mte.h +++ b/arch/arm64/include/asm/mte.h @@ -8,8 +8,6 @@ #include #include -#define __MTE_PREAMBLE ARM64_ASM_PREAMBLE ".arch_extension memtag\n" - #ifndef __ASSEMBLY__ #include diff --git a/arch/arm64/kernel/mte.c b/arch/arm64/kernel/mte.c index 7763ac1f2917..8b27b70e1aac 100644 --- a/arch/arm64/kernel/mte.c +++ b/arch/arm64/kernel/mte.c @@ -19,7 +19,6 @@ #include #include #include -#include #include #include @@ -88,51 +87,6 @@ int memcmp_pages(struct page *page1, struct page *page2) return ret; } -u8 mte_get_mem_tag(void *addr) -{ - if (!system_supports_mte()) - return 0xFF; - - asm(__MTE_PREAMBLE "ldg %0, [%0]" - : "+r" (addr)); - - return mte_get_ptr_tag(addr); -} - -u8 mte_get_random_tag(void) -{ - void *addr; - - if (!system_supports_mte()) - return 0xFF; - - asm(__MTE_PREAMBLE "irg %0, %0" - : "+r" (addr)); - - return mte_get_ptr_tag(addr); -} - -void *mte_set_mem_tag_range(void *addr, size_t size, u8 tag) -{ - void *ptr = addr; - - if ((!system_supports_mte()) || (size == 0)) - return addr; - - /* Make sure that size is MTE granule aligned. */ - WARN_ON(size & (MTE_GRANULE_SIZE - 1)); - - /* Make sure that the address is MTE granule aligned. */ - WARN_ON((u64)addr & (MTE_GRANULE_SIZE - 1)); - - tag = 0xF0 | tag; - ptr = (void *)__tag_set(ptr, tag); - - mte_assign_mem_tag_range(ptr, size); - - return ptr; -} - void mte_init_tags(u64 max_tag) { static bool gcr_kernel_excl_initialized; diff --git a/arch/arm64/lib/mte.S b/arch/arm64/lib/mte.S index 9e1a12e10053..351537c12f36 100644 --- a/arch/arm64/lib/mte.S +++ b/arch/arm64/lib/mte.S @@ -149,19 +149,3 @@ SYM_FUNC_START(mte_restore_page_tags) ret SYM_FUNC_END(mte_restore_page_tags) - -/* - * Assign allocation tags for a region of memory based on the pointer tag - * x0 - source pointer - * x1 - size - * - * Note: The address must be non-NULL and MTE_GRANULE_SIZE aligned and - * size must be non-zero and MTE_GRANULE_SIZE aligned. - */ -SYM_FUNC_START(mte_assign_mem_tag_range) -1: stg x0, [x0] - add x0, x0, #MTE_GRANULE_SIZE - subs x1, x1, #MTE_GRANULE_SIZE - b.gt 1b - ret -SYM_FUNC_END(mte_assign_mem_tag_range) From patchwork Mon Feb 1 19:43:35 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrey Konovalov X-Patchwork-Id: 12059957 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.1 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7FC56C433E0 for ; Mon, 1 Feb 2021 19:47:17 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 2F4A964EA8 for ; Mon, 1 Feb 2021 19:47:17 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 2F4A964EA8 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:To:From:Subject:References:Mime-Version:Message-Id: In-Reply-To:Date:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=FNChEz5M/FDkg7do34dklzcCPFjHscs+eFnJm0ipucM=; b=ADAdlAsJmb2To87s1WxZhiY0Z 4cFsVrTtLVQvWVt2VxdNA7dVHQxhoUS6DEXWrb6L1eEcTJfn1Tgm0ixSmTpFT0MmiJ3N2rRNYHfTy WU1nKbtjNB/G0TsH1QntB8SzYPDKTomKQWaPWaLJlFJfVn88+305Yc0XsANegn7o0zvPGT67oQANc x1pFFgjyIXcWH1wErlGTWmOG5X3S/ruvwtf2m8FzkacV8bjaBR41980jPza4BRU83TvhTWjKNwtMd +KSIsBZN82GdiUwbUew7sLb5ymkxQjJ7odSmsaGyYe6yQuxetgYWwPuTE+86eSDupjC2PiDJuGZNz u4njcI0Ng==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1l6f8c-0002iG-Ux; Mon, 01 Feb 2021 19:45:31 +0000 Received: from mail-qt1-x849.google.com ([2607:f8b0:4864:20::849]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1l6f7K-00021W-G8 for linux-arm-kernel@lists.infradead.org; Mon, 01 Feb 2021 19:44:13 +0000 Received: by mail-qt1-x849.google.com with SMTP id i1so3841177qtp.16 for ; Mon, 01 Feb 2021 11:44:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=NVeqfy8PnyvgWAdKHeCgVfqveCjCMDJegzzrq5UVB6U=; b=b1lXS5TDAM0t8/OObzI88t9G9V6LITLxK5/HlZBBUXX7GfRsMrQIJFL7iZyApowvVV +V+FY2IUUWYkNTWeXZxdzf34OrHTRKwVqQcNL6fICTVHh3TpEujGh1tnocawM4hwAb4b TtuC53IMGp9hqR2Xewo3b/F1Q4iYHl+9mIZ2/lzggJuzIJcI4NBXgM0WrryaLtGR0m2a sq8jarjd69t2LKd1B5QHWJUUseYqkrYYJ/nB6l/GCPk5F1j7V1sJb7Xohp/yNGB1c8xk Doov5OEX30RUucXGXr1esBwDjbnsH9P77ukc8FOoY5A7aAkf1kP4LapMQoBp2hVkGH5G FQ2g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=NVeqfy8PnyvgWAdKHeCgVfqveCjCMDJegzzrq5UVB6U=; b=NUL34LMzYzij9XtCrpLWV9obJx6izUCPq9CQTibegX1rqz4QoqHkmrzS0H0lSndKpX TjbO3wh+hsZu7h7tMRM2CDFpddgzI/JiSCVh9L6+BdwPdYttmkQFzk6gConbXKwr0/Le KuTdX13JdHx0RYLqiLbvYQ66m1kVuha2M6FF+LQXwxll+9eLo1icknBHZ7sdjYLZc64J 3B72n3PlgFYMvTLZRxT6Wq5nn4WaE2QLQdHLh0RZXAxCL828UitwHFVUqBJnpcCoHIgM WfAr3xT/snCAHKqYeeTWd3kg536ChF+XcTpKyiRi+37Vkt56mal4X3vhXQ8Jh4eyeWAJ QXdQ== X-Gm-Message-State: AOAM5333AjrbmXZK0/R5WX0pA+AOL12CU9RAvBcoXRU1V/ueJ75Du8yU I9/Rznxfkh6eIwADHFEAhDHL58360pcc8EjZ X-Google-Smtp-Source: ABdhPJy3LMcMO1RRr1wTMaZvcYIHJ2lGN1Rm3XmR+DCe/2CyvjEerR8l4ufrmsSLp/ooBcba4JiSkTlvY93N5sqV X-Received: from andreyknvl3.muc.corp.google.com ([2a00:79e0:15:13:7220:84ff:fe09:7e9d]) (user=andreyknvl job=sendgmr) by 2002:a05:6214:725:: with SMTP id c5mr16904134qvz.27.1612208645795; Mon, 01 Feb 2021 11:44:05 -0800 (PST) Date: Mon, 1 Feb 2021 20:43:35 +0100 In-Reply-To: Message-Id: <05a45017b4cb15344395650e880bbab0fe6ba3e4.1612208222.git.andreyknvl@google.com> Mime-Version: 1.0 References: X-Mailer: git-send-email 2.30.0.365.g02bc693789-goog Subject: [PATCH 11/12] kasan: always inline HW_TAGS helper functions From: Andrey Konovalov To: Catalin Marinas , Vincenzo Frascino , Dmitry Vyukov , Alexander Potapenko , Marco Elver X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210201_144410_622479_AC87F26B X-CRM114-Status: GOOD ( 12.46 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Branislav Rankov , Andrey Konovalov , Kevin Brodsky , Will Deacon , linux-kernel@vger.kernel.org, kasan-dev@googlegroups.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, Andrey Ryabinin , Andrew Morton , Peter Collingbourne , Evgenii Stepanov Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Mark all static functions in common.c and kasan.h that are used for hardware tag-based KASAN as __always_inline to avoid unnecessary function calls. Signed-off-by: Andrey Konovalov --- mm/kasan/common.c | 13 +++++++------ mm/kasan/kasan.h | 6 +++--- 2 files changed, 10 insertions(+), 9 deletions(-) diff --git a/mm/kasan/common.c b/mm/kasan/common.c index 5691cca69397..2004ecd6e43c 100644 --- a/mm/kasan/common.c +++ b/mm/kasan/common.c @@ -279,7 +279,8 @@ void __kasan_poison_object_data(struct kmem_cache *cache, void *object) * based on objects indexes, so that objects that are next to each other * get different tags. */ -static u8 assign_tag(struct kmem_cache *cache, const void *object, bool init) +static __always_inline u8 assign_tag(struct kmem_cache *cache, + const void *object, bool init) { if (IS_ENABLED(CONFIG_KASAN_GENERIC)) return 0xff; @@ -321,8 +322,8 @@ void * __must_check __kasan_init_slab_obj(struct kmem_cache *cache, return (void *)object; } -static bool ____kasan_slab_free(struct kmem_cache *cache, void *object, - unsigned long ip, bool quarantine) +static __always_inline bool ____kasan_slab_free(struct kmem_cache *cache, + void *object, unsigned long ip, bool quarantine) { u8 tag; void *tagged_object; @@ -366,7 +367,7 @@ bool __kasan_slab_free(struct kmem_cache *cache, void *object, unsigned long ip) return ____kasan_slab_free(cache, object, ip, true); } -static bool ____kasan_kfree_large(void *ptr, unsigned long ip) +static __always_inline bool ____kasan_kfree_large(void *ptr, unsigned long ip) { if (ptr != page_address(virt_to_head_page(ptr))) { kasan_report_invalid_free(ptr, ip); @@ -461,8 +462,8 @@ void * __must_check __kasan_slab_alloc(struct kmem_cache *cache, return tagged_object; } -static void *____kasan_kmalloc(struct kmem_cache *cache, const void *object, - size_t size, gfp_t flags) +static __always_inline void *____kasan_kmalloc(struct kmem_cache *cache, + const void *object, size_t size, gfp_t flags) { unsigned long redzone_start; unsigned long redzone_end; diff --git a/mm/kasan/kasan.h b/mm/kasan/kasan.h index 2f7400a3412f..d5fe72747a53 100644 --- a/mm/kasan/kasan.h +++ b/mm/kasan/kasan.h @@ -321,7 +321,7 @@ static inline u8 kasan_random_tag(void) { return 0; } #ifdef CONFIG_KASAN_HW_TAGS -static inline void kasan_poison(const void *addr, size_t size, u8 value) +static __always_inline void kasan_poison(const void *addr, size_t size, u8 value) { addr = kasan_reset_tag(addr); @@ -337,7 +337,7 @@ static inline void kasan_poison(const void *addr, size_t size, u8 value) hw_set_mem_tag_range((void *)addr, size, value); } -static inline void kasan_unpoison(const void *addr, size_t size) +static __always_inline void kasan_unpoison(const void *addr, size_t size) { u8 tag = get_tag(addr); @@ -354,7 +354,7 @@ static inline void kasan_unpoison(const void *addr, size_t size) hw_set_mem_tag_range((void *)addr, size, tag); } -static inline bool kasan_byte_accessible(const void *addr) +static __always_inline bool kasan_byte_accessible(const void *addr) { u8 ptr_tag = get_tag(addr); u8 mem_tag = hw_get_mem_tag((void *)addr); From patchwork Mon Feb 1 19:43:36 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrey Konovalov X-Patchwork-Id: 12059961 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.1 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B34CCC433E6 for ; Mon, 1 Feb 2021 19:47:47 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 685B964E54 for ; Mon, 1 Feb 2021 19:47:47 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 685B964E54 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:To:From:Subject:References:Mime-Version:Message-Id: In-Reply-To:Date:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=7GPJFjh8dNeu21f1+hyjTZYonVzonAJtN3ccr+pjKYU=; b=PQP4z+kP9wFq2aIWOQS6+7RTM G+anYWVPVDtFpETYAy1zty5R4S5tMkHAAA5DwpTwcRjnQlCN7kaHCK1yW4XR2CQJ6DEqeROuOd2Jm KBgM0H66cLC5qLXXYyFOFVSnkNL1uAYKL6pz/o7yvbzySt+Tb/iVQ2PbqAiUW1wLiRkIwsjQpDeDO xIQyWyEUHUzvhhkouxxdbVGO5OmLWgHj02Y8osKWMtOBExqjRpnR1z+gOH+9YBOKt2KR9G0gdi6KK P4wVYuADIIr8VUVVkJj5Jt+WrsBUOIV7sq6GZTlU/gAS4cCm+tJrdNJlJnnS54MCQh6lP/3c4dbW7 MFgoTeTIA==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1l6f9D-00034l-LC; Mon, 01 Feb 2021 19:46:07 +0000 Received: from mail-qt1-x849.google.com ([2607:f8b0:4864:20::849]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1l6f7M-00022Y-T4 for linux-arm-kernel@lists.infradead.org; Mon, 01 Feb 2021 19:44:16 +0000 Received: by mail-qt1-x849.google.com with SMTP id d1so11378788qtp.11 for ; Mon, 01 Feb 2021 11:44:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=9ElhXeWA9hrtQUSa92BHDv89rLolOBWVaEOaHObQHiY=; b=scCt97wjEnPKthR1OEsfg6XHzgS7EdRFiyeoCwu46hVwq5SF3m83dULHz0MWmmLGZl yY1Mcb/oHxqrO0CQyzz2RZvmAnQ0bHwXgancVwJ8Kf9cil1iktYwcALcUpt8omkBBHc3 kkd4KIyCpQULEzuHjoILesZuKjI3HCZjm6xFhHFB/41fKMgY+HzZ0fmHsvzGSo5PJQRd 1qzDE52vdQilfVBIZo6KxqHC32etk1l7O9n0C3A7cR60R+7EJMBESajOp8+nVQnDVT7t X0qHdKFBkQ62nf38KxTN6Kq6kGBIJL7HtiqGEajvgyY3xCf6RvAxjkhQOoK7XuBRQGSn QsVQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=9ElhXeWA9hrtQUSa92BHDv89rLolOBWVaEOaHObQHiY=; b=SIu3gONEa2eBuWKnKJbhub6aCidtlkEt8l6zkotgc9sp4NAPZ1Ttzo794mvm5ddO3q O20lY1jbfUELsST7mQK73UVKsE0kNyYy7M55g5fma+b14UKsSzDgbsLWIL/QKzsc5K75 t6xcBCWSl6nhb5ncC6Dgx6yVMa3f4c81L4pQNubeCMM3g8DO+jf0IqkUGpaMpnDgruPL yFBSQdxjtUVYi+ARLX0rCdbCi1R7Y2P1PQ6tkJ8/1kIChUyU+ZTSlCXel9B46PmsjZvu Q7YT/0efcQDqcyM+N22NjMfl8Cv93mAE9jNIYwpzOCu6WJOQDgh7Eu5YHgnBvwvnjB3d wPFg== X-Gm-Message-State: AOAM533/IPum3g8LTZW1jVZMqkZlLWDqItnC3c1Yk+WOn4C3k0Tr1QFu aTpvLtbhb98uFI5Pqek59azh5job0D2ulFch X-Google-Smtp-Source: ABdhPJyGsScV89G9MAWMKyK7rBzUxlUepSEM9x8u78kzqVlPK95BXbpBATuTGNJidQ0HrKqL6DgEfGQHE22rUIOh X-Received: from andreyknvl3.muc.corp.google.com ([2a00:79e0:15:13:7220:84ff:fe09:7e9d]) (user=andreyknvl job=sendgmr) by 2002:a05:6214:11ab:: with SMTP id u11mr16841497qvv.17.1612208648056; Mon, 01 Feb 2021 11:44:08 -0800 (PST) Date: Mon, 1 Feb 2021 20:43:36 +0100 In-Reply-To: Message-Id: Mime-Version: 1.0 References: X-Mailer: git-send-email 2.30.0.365.g02bc693789-goog Subject: [PATCH 12/12] arm64: kasan: export MTE symbols for KASAN tests From: Andrey Konovalov To: Catalin Marinas , Vincenzo Frascino , Dmitry Vyukov , Alexander Potapenko , Marco Elver X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210201_144413_155466_26B707C6 X-CRM114-Status: GOOD ( 11.19 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Branislav Rankov , Andrey Konovalov , Kevin Brodsky , Will Deacon , linux-kernel@vger.kernel.org, kasan-dev@googlegroups.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, Andrey Ryabinin , Andrew Morton , Peter Collingbourne , Evgenii Stepanov Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Export mte_enable_kernel() and mte_set_report_once() to fix: ERROR: modpost: "mte_enable_kernel" [lib/test_kasan.ko] undefined! ERROR: modpost: "mte_set_report_once" [lib/test_kasan.ko] undefined! Signed-off-by: Andrey Konovalov Acked-by: Catalin Marinas --- arch/arm64/kernel/mte.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/arch/arm64/kernel/mte.c b/arch/arm64/kernel/mte.c index 8b27b70e1aac..2c91bd288ea4 100644 --- a/arch/arm64/kernel/mte.c +++ b/arch/arm64/kernel/mte.c @@ -120,6 +120,7 @@ void mte_enable_kernel_sync(void) { __mte_enable_kernel("synchronous", SCTLR_ELx_TCF_SYNC); } +EXPORT_SYMBOL(mte_enable_kernel_sync); void mte_enable_kernel_async(void) { @@ -130,6 +131,7 @@ void mte_set_report_once(bool state) { WRITE_ONCE(report_fault_once, state); } +EXPORT_SYMBOL(mte_set_report_once); bool mte_report_once(void) {