From patchwork Fri Feb 5 17:34:35 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrey Konovalov X-Patchwork-Id: 12070541 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.1 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E3FC2C433E0 for ; Fri, 5 Feb 2021 17:36:24 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 57F0364DBA for ; Fri, 5 Feb 2021 17:36:24 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 57F0364DBA Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:To:From:Subject:References:Mime-Version:Message-Id: In-Reply-To:Date:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=S/gtIuhTaQEtaS7d2CvILXAVgNKkMvJ1bCJZjuq1ir8=; b=PzghTwmgGTQ5FmLd+Yk2DVR4q X0L5yfHRpV5ktPI+0CskbYOODrM6ZdScI1X5t7jDbTSZEhZLMMlueyCQFpsXcAfPdzgxc8sf6wTR8 ttjm785p0vpAvpcLUF2tHzE9VHviO1sBg6cKmiENx6/3MoF1GgM0/0Z8sb4tQu5ZRTohccod0EIo4 bAq0Emc4qZNnrXPrtXiylM6EOj1RQUPZCTCAmVYq5P8SwigwdBHj5M+zvX8Ur5ebJijWsgo/K4/D0 nIxwfvphfEMjT0/9DpWBQ2Jm5UIemvePswna0fnqPYQjhIA7JH8zS5Aapzt0DFLoz/c3gMfISjXYR PLXIUyneQ==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1l850a-0004s6-MG; Fri, 05 Feb 2021 17:35:04 +0000 Received: from mail-qt1-x84a.google.com ([2607:f8b0:4864:20::84a]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1l850S-0004n3-LX for linux-arm-kernel@lists.infradead.org; Fri, 05 Feb 2021 17:35:01 +0000 Received: by mail-qt1-x84a.google.com with SMTP id w3so5748531qti.17 for ; Fri, 05 Feb 2021 09:34:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=lqblrx40cOeXNR+PvBIbJ4346eed6vA2slz2Dc36rws=; b=g23QkriSwOFF3ULYeHRIul3KnBuvXpFVQMFgUNTP2dhadFwyqkvoS8iT1hI7oxw9f7 u1+hhH7kXrUExKdbbU67pgUS8ZTQpX4psoCpfEBqHm2T9WXmdBzKe1WOtKnmwpqGNsdH NWaYbl1NbD0oV1kF5zJQD32kZ2O8qk6Ig51aoRcBymAZjP08OevkQ0QQAgDYbZDh6noJ 8NpL3iwkxQjjiLCsAgM90Onga+590srz3TVTYwa43UOWNrA+qTqw2gHHIydZUO7ohI/s AfjVgKw8nc89oQ2NuTM42VIUoQ/XxjwVlTzAms18MaKAjcyY3UXsI+zOR/mE314FsPkv C/HQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=lqblrx40cOeXNR+PvBIbJ4346eed6vA2slz2Dc36rws=; b=tIuAmwAK0MFQM+Z9S3y4UhYuQnhFUg7spV6Gyf6gCOmOslIVINehRre1QNPSaqjgsG bVJp/r8Mj8LzkXmm9fzrUAIBj4Kb63ZIIBX5gaI8dj7h5uhBRLdY/M6/LC590uYUBSSz yY3WI7VxCbPFPtBdmfhosmAap7QWS/zTuXUE2QPpz2sXPVVfH9iORnPTg+/g1/WSOs7U OtomcIt+JkpUAjKBXYFpQMRWiJzv8grFD+FH6VmCnI+B4st0kQtcYpCydhrt/Kalc7yq JVCdumLraq40tH+VQjn9s3wvHC/pwjjYcH/c+igEb9REXwDISWJgARs9VFgmy4gVj7sx pzhg== X-Gm-Message-State: AOAM530Yb8hiG10EsTAuckG7zGCWaKgyZdQtrAPNtrGfa3EhJFJTlLyo 04+8WZi7pQlWqBw6wqg8WNuLeu52xn8F2pwC X-Google-Smtp-Source: ABdhPJyYRAxjWEwapKlGp89zabVkYAfPqiYTj/SPIKWC3AYOqdFBUlK65Ye0lDHrCSs2Bx7/IGPeRGC5Kd3DzPND X-Received: from andreyknvl3.muc.corp.google.com ([2a00:79e0:15:13:edb8:b79c:2e20:e531]) (user=andreyknvl job=sendgmr) by 2002:a0c:f143:: with SMTP id y3mr5496897qvl.62.1612546492642; Fri, 05 Feb 2021 09:34:52 -0800 (PST) Date: Fri, 5 Feb 2021 18:34:35 +0100 In-Reply-To: Message-Id: <7c673ebca8d00f40a7ad6f04ab9a2bddeeae2097.1612546384.git.andreyknvl@google.com> Mime-Version: 1.0 References: X-Mailer: git-send-email 2.30.0.365.g02bc693789-goog Subject: [PATCH v3 mm 01/13] kasan, mm: don't save alloc stacks twice From: Andrey Konovalov To: Andrew Morton , Catalin Marinas , Vincenzo Frascino , Dmitry Vyukov , Alexander Potapenko , Marco Elver X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210205_123456_763515_4BE17D59 X-CRM114-Status: GOOD ( 15.92 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Branislav Rankov , Andrey Konovalov , Kevin Brodsky , Will Deacon , linux-kernel@vger.kernel.org, kasan-dev@googlegroups.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, Andrey Ryabinin , Peter Collingbourne , Evgenii Stepanov Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Currently KASAN saves allocation stacks in both kasan_slab_alloc() and kasan_kmalloc() annotations. This patch changes KASAN to save allocation stacks for slab objects from kmalloc caches in kasan_kmalloc() only, and stacks for other slab objects in kasan_slab_alloc() only. This change requires ____kasan_kmalloc() knowing whether the object belongs to a kmalloc cache. This is implemented by adding a flag field to the kasan_info structure. That flag is only set for kmalloc caches via a new kasan_cache_create_kmalloc() annotation. Reviewed-by: Marco Elver Signed-off-by: Andrey Konovalov --- include/linux/kasan.h | 9 +++++++++ mm/kasan/common.c | 18 ++++++++++++++---- mm/slab_common.c | 1 + 3 files changed, 24 insertions(+), 4 deletions(-) diff --git a/include/linux/kasan.h b/include/linux/kasan.h index 1011e4f30284..e6ed969e74b3 100644 --- a/include/linux/kasan.h +++ b/include/linux/kasan.h @@ -83,6 +83,7 @@ static inline void kasan_disable_current(void) {} struct kasan_cache { int alloc_meta_offset; int free_meta_offset; + bool is_kmalloc; }; #ifdef CONFIG_KASAN_HW_TAGS @@ -143,6 +144,13 @@ static __always_inline void kasan_cache_create(struct kmem_cache *cache, __kasan_cache_create(cache, size, flags); } +void __kasan_cache_create_kmalloc(struct kmem_cache *cache); +static __always_inline void kasan_cache_create_kmalloc(struct kmem_cache *cache) +{ + if (kasan_enabled()) + __kasan_cache_create_kmalloc(cache); +} + size_t __kasan_metadata_size(struct kmem_cache *cache); static __always_inline size_t kasan_metadata_size(struct kmem_cache *cache) { @@ -278,6 +286,7 @@ static inline void kasan_free_pages(struct page *page, unsigned int order) {} static inline void kasan_cache_create(struct kmem_cache *cache, unsigned int *size, slab_flags_t *flags) {} +static inline void kasan_cache_create_kmalloc(struct kmem_cache *cache) {} static inline size_t kasan_metadata_size(struct kmem_cache *cache) { return 0; } static inline void kasan_poison_slab(struct page *page) {} static inline void kasan_unpoison_object_data(struct kmem_cache *cache, diff --git a/mm/kasan/common.c b/mm/kasan/common.c index fe852f3cfa42..bfdf5464f4ef 100644 --- a/mm/kasan/common.c +++ b/mm/kasan/common.c @@ -210,6 +210,11 @@ void __kasan_cache_create(struct kmem_cache *cache, unsigned int *size, *size = optimal_size; } +void __kasan_cache_create_kmalloc(struct kmem_cache *cache) +{ + cache->kasan_info.is_kmalloc = true; +} + size_t __kasan_metadata_size(struct kmem_cache *cache) { if (!kasan_stack_collection_enabled()) @@ -394,17 +399,22 @@ void __kasan_slab_free_mempool(void *ptr, unsigned long ip) } } -static void set_alloc_info(struct kmem_cache *cache, void *object, gfp_t flags) +static void set_alloc_info(struct kmem_cache *cache, void *object, + gfp_t flags, bool is_kmalloc) { struct kasan_alloc_meta *alloc_meta; + /* Don't save alloc info for kmalloc caches in kasan_slab_alloc(). */ + if (cache->kasan_info.is_kmalloc && !is_kmalloc) + return; + alloc_meta = kasan_get_alloc_meta(cache, object); if (alloc_meta) kasan_set_track(&alloc_meta->alloc_track, flags); } static void *____kasan_kmalloc(struct kmem_cache *cache, const void *object, - size_t size, gfp_t flags, bool keep_tag) + size_t size, gfp_t flags, bool is_kmalloc) { unsigned long redzone_start; unsigned long redzone_end; @@ -423,7 +433,7 @@ static void *____kasan_kmalloc(struct kmem_cache *cache, const void *object, KASAN_GRANULE_SIZE); redzone_end = round_up((unsigned long)object + cache->object_size, KASAN_GRANULE_SIZE); - tag = assign_tag(cache, object, false, keep_tag); + tag = assign_tag(cache, object, false, is_kmalloc); /* Tag is ignored in set_tag without CONFIG_KASAN_SW/HW_TAGS */ kasan_unpoison(set_tag(object, tag), size); @@ -431,7 +441,7 @@ static void *____kasan_kmalloc(struct kmem_cache *cache, const void *object, KASAN_KMALLOC_REDZONE); if (kasan_stack_collection_enabled()) - set_alloc_info(cache, (void *)object, flags); + set_alloc_info(cache, (void *)object, flags, is_kmalloc); return set_tag(object, tag); } diff --git a/mm/slab_common.c b/mm/slab_common.c index 9aa3d2fe4c55..39d1a8ff9bb8 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -647,6 +647,7 @@ struct kmem_cache *__init create_kmalloc_cache(const char *name, panic("Out of memory when creating slab %s\n", name); create_boot_cache(s, name, size, flags, useroffset, usersize); + kasan_cache_create_kmalloc(s); list_add(&s->list, &slab_caches); s->refcount = 1; return s; From patchwork Fri Feb 5 17:34:36 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrey Konovalov X-Patchwork-Id: 12070543 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.1 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0658FC433DB for ; Fri, 5 Feb 2021 17:36:41 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 9FCDF64DC4 for ; Fri, 5 Feb 2021 17:36:40 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 9FCDF64DC4 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:To:From:Subject:References:Mime-Version:Message-Id: In-Reply-To:Date:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=9r6H+0tdzu62GXfRHR6zah4T1U0GxCPVKN96VituLAU=; b=DCmrbVQuDSPzWpF3Fy1IhLgoF DkUlttg2Z8KgzDxe4CFNxeoe64i2y8bWZ6ZCWMhCk1uMr3uuBEUoKa4/D8vG+KBJMsWvtqnvHrSPm icITP4gyqjc+oA7rG75+kr74nwW83iqtx0cUXgwoiBB+LmV5qsxPwjWeDGsHNr1V1pfN4t8sks2bM T9eNZr4+MHBtFmFb/jK8uohzfWD76yZ/YVkIc2WqtGWQssJm8uDtDsqpWXisciOrv7aLBFDFEzYQl BHLfbszyN8j1PVT/dHosy8wuEfXn3f6biUloKusoYfP6mEpMxWoFw45vkh4+1caNzGm5AWCRh4mDf qLBIvMyjw==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1l850i-0004vo-RE; Fri, 05 Feb 2021 17:35:12 +0000 Received: from mail-wm1-x349.google.com ([2a00:1450:4864:20::349]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1l850U-0004o1-Fn for linux-arm-kernel@lists.infradead.org; Fri, 05 Feb 2021 17:35:07 +0000 Received: by mail-wm1-x349.google.com with SMTP id j204so4121408wmj.4 for ; Fri, 05 Feb 2021 09:34:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=UPTowWhmoRbOR7zm1ysa/6XpOoPbcb0RCUTUBjiaP4A=; b=t56fsyt49qxfrCMWGKN0VUvdOnJ1lfyApc9niZvbquzDzG4HHwUyaKJpIftA1tVtVt YydKUaNuuNmmlUbramA7U/9Sr6g5ECdl51KKHYiVdIC2vTbgJzepkWYR9N0QDg4L3m/p EBJOVwJgtGq+wQSW8VCtw7U7t5g+MZOGXWCyh1gVwuDdxRSjtVfiBaZMakwYEnOk+rZR a99zTLEkihoFmEoyNk7m65Mo1aYKg8ro4uXhBVQwdmWw3zFqnBD2+SWeYG/Ci7rsKhe9 nn+dmmc+fnatJslpS9YlapJDNq0FkiJqvT4L2bgFxT0WN7VacYU6CU4al1W6jFNyZZG1 aloA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=UPTowWhmoRbOR7zm1ysa/6XpOoPbcb0RCUTUBjiaP4A=; b=uOEBxnIVAeWbL1UFj6340Agm5O3HgEYtoQAgX5RclJlMACFsKiyw2MuXHNOd+UATwK m0ZJngAolA4FbkJ4XFWw8dceXuoTGXQSjIB6o78dEPH3MVHZPIZ7V/kIctPVAN+p6Odj D4Txt5JtfQaNUXYb1pvqkSoZXOkS/vudGXnxpcJzJuY7EWopfuvyfO6rBTLpucyfVsy+ S2Po86SXa+ucZ3LTQErIHwMegnqszv6Z4s0tWRbnHjc+NlM8mPeGfpbq3m1uvMhWu9Sf vi7nm6GB6cXlm/APTSzWB+08qle9FeRuyz7XQrxX8gHtj9z4/92CZRp91KGabD3rdEaG wx7g== X-Gm-Message-State: AOAM532VZkQfrxpp2yEpYNdMzRpbq9YeN4XfNeByG1UBVh8nNaOn2Wmc G6bVGpP4lA0Rj+OfOn4tALXvHCp+NOUSlq8x X-Google-Smtp-Source: ABdhPJxI6cEiep5fmLoRD7VwvDcrp5mbmTY5277WL0F9f+Pk7fyJDpBb3HtbO8pNwVva855QWBHiKOjfVcIkPC+J X-Received: from andreyknvl3.muc.corp.google.com ([2a00:79e0:15:13:edb8:b79c:2e20:e531]) (user=andreyknvl job=sendgmr) by 2002:a1c:a707:: with SMTP id q7mr4463252wme.15.1612546495078; Fri, 05 Feb 2021 09:34:55 -0800 (PST) Date: Fri, 5 Feb 2021 18:34:36 +0100 In-Reply-To: Message-Id: <7e3961cb52be380bc412860332063f5f7ce10d13.1612546384.git.andreyknvl@google.com> Mime-Version: 1.0 References: X-Mailer: git-send-email 2.30.0.365.g02bc693789-goog Subject: [PATCH v3 mm 02/13] kasan, mm: optimize kmalloc poisoning From: Andrey Konovalov To: Andrew Morton , Catalin Marinas , Vincenzo Frascino , Dmitry Vyukov , Alexander Potapenko , Marco Elver X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210205_123458_710048_302FF9A4 X-CRM114-Status: GOOD ( 26.11 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Branislav Rankov , Andrey Konovalov , Kevin Brodsky , Will Deacon , linux-kernel@vger.kernel.org, kasan-dev@googlegroups.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, Andrey Ryabinin , Peter Collingbourne , Evgenii Stepanov Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org For allocations from kmalloc caches, kasan_kmalloc() always follows kasan_slab_alloc(). Currenly, both of them unpoison the whole object, which is unnecessary. This patch provides separate implementations for both annotations: kasan_slab_alloc() unpoisons the whole object, and kasan_kmalloc() only poisons the redzone. For generic KASAN, the redzone start might not be aligned to KASAN_GRANULE_SIZE. Therefore, the poisoning is split in two parts: kasan_poison_last_granule() poisons the unaligned part, and then kasan_poison() poisons the rest. This patch also clarifies alignment guarantees of each of the poisoning functions and drops the unnecessary round_up() call for redzone_end. With this change, the early SLUB cache annotation needs to be changed to kasan_slab_alloc(), as kasan_kmalloc() doesn't unpoison objects now. The number of poisoned bytes for objects in this cache stays the same, as kmem_cache_node->object_size is equal to sizeof(struct kmem_cache_node). Reviewed-by: Marco Elver Signed-off-by: Andrey Konovalov --- mm/kasan/common.c | 93 +++++++++++++++++++++++++++++++---------------- mm/kasan/kasan.h | 43 +++++++++++++++++++++- mm/kasan/shadow.c | 28 +++++++------- mm/slub.c | 3 +- 4 files changed, 119 insertions(+), 48 deletions(-) diff --git a/mm/kasan/common.c b/mm/kasan/common.c index bfdf5464f4ef..00edbc3eb32e 100644 --- a/mm/kasan/common.c +++ b/mm/kasan/common.c @@ -278,21 +278,11 @@ void __kasan_poison_object_data(struct kmem_cache *cache, void *object) * based on objects indexes, so that objects that are next to each other * get different tags. */ -static u8 assign_tag(struct kmem_cache *cache, const void *object, - bool init, bool keep_tag) +static u8 assign_tag(struct kmem_cache *cache, const void *object, bool init) { if (IS_ENABLED(CONFIG_KASAN_GENERIC)) return 0xff; - /* - * 1. When an object is kmalloc()'ed, two hooks are called: - * kasan_slab_alloc() and kasan_kmalloc(). We assign the - * tag only in the first one. - * 2. We reuse the same tag for krealloc'ed objects. - */ - if (keep_tag) - return get_tag(object); - /* * If the cache neither has a constructor nor has SLAB_TYPESAFE_BY_RCU * set, assign a tag when the object is being allocated (init == false). @@ -325,7 +315,7 @@ void * __must_check __kasan_init_slab_obj(struct kmem_cache *cache, } /* Tag is ignored in set_tag() without CONFIG_KASAN_SW/HW_TAGS */ - object = set_tag(object, assign_tag(cache, object, true, false)); + object = set_tag(object, assign_tag(cache, object, true)); return (void *)object; } @@ -413,12 +403,46 @@ static void set_alloc_info(struct kmem_cache *cache, void *object, kasan_set_track(&alloc_meta->alloc_track, flags); } +void * __must_check __kasan_slab_alloc(struct kmem_cache *cache, + void *object, gfp_t flags) +{ + u8 tag; + void *tagged_object; + + if (gfpflags_allow_blocking(flags)) + kasan_quarantine_reduce(); + + if (unlikely(object == NULL)) + return NULL; + + if (is_kfence_address(object)) + return (void *)object; + + /* + * Generate and assign random tag for tag-based modes. + * Tag is ignored in set_tag() for the generic mode. + */ + tag = assign_tag(cache, object, false); + tagged_object = set_tag(object, tag); + + /* + * Unpoison the whole object. + * For kmalloc() allocations, kasan_kmalloc() will do precise poisoning. + */ + kasan_unpoison(tagged_object, cache->object_size); + + /* Save alloc info (if possible) for non-kmalloc() allocations. */ + if (kasan_stack_collection_enabled()) + set_alloc_info(cache, (void *)object, flags, false); + + return tagged_object; +} + static void *____kasan_kmalloc(struct kmem_cache *cache, const void *object, - size_t size, gfp_t flags, bool is_kmalloc) + size_t size, gfp_t flags) { unsigned long redzone_start; unsigned long redzone_end; - u8 tag; if (gfpflags_allow_blocking(flags)) kasan_quarantine_reduce(); @@ -429,33 +453,41 @@ static void *____kasan_kmalloc(struct kmem_cache *cache, const void *object, if (is_kfence_address(kasan_reset_tag(object))) return (void *)object; + /* + * The object has already been unpoisoned by kasan_slab_alloc() for + * kmalloc() or by ksize() for krealloc(). + */ + + /* + * The redzone has byte-level precision for the generic mode. + * Partially poison the last object granule to cover the unaligned + * part of the redzone. + */ + if (IS_ENABLED(CONFIG_KASAN_GENERIC)) + kasan_poison_last_granule((void *)object, size); + + /* Poison the aligned part of the redzone. */ redzone_start = round_up((unsigned long)(object + size), KASAN_GRANULE_SIZE); - redzone_end = round_up((unsigned long)object + cache->object_size, - KASAN_GRANULE_SIZE); - tag = assign_tag(cache, object, false, is_kmalloc); - - /* Tag is ignored in set_tag without CONFIG_KASAN_SW/HW_TAGS */ - kasan_unpoison(set_tag(object, tag), size); + redzone_end = (unsigned long)object + cache->object_size; kasan_poison((void *)redzone_start, redzone_end - redzone_start, KASAN_KMALLOC_REDZONE); + /* + * Save alloc info (if possible) for kmalloc() allocations. + * This also rewrites the alloc info when called from kasan_krealloc(). + */ if (kasan_stack_collection_enabled()) - set_alloc_info(cache, (void *)object, flags, is_kmalloc); + set_alloc_info(cache, (void *)object, flags, true); - return set_tag(object, tag); -} - -void * __must_check __kasan_slab_alloc(struct kmem_cache *cache, - void *object, gfp_t flags) -{ - return ____kasan_kmalloc(cache, object, cache->object_size, flags, false); + /* Keep the tag that was set by kasan_slab_alloc(). */ + return (void *)object; } void * __must_check __kasan_kmalloc(struct kmem_cache *cache, const void *object, size_t size, gfp_t flags) { - return ____kasan_kmalloc(cache, object, size, flags, true); + return ____kasan_kmalloc(cache, object, size, flags); } EXPORT_SYMBOL(__kasan_kmalloc); @@ -496,8 +528,7 @@ void * __must_check __kasan_krealloc(const void *object, size_t size, gfp_t flag if (unlikely(!PageSlab(page))) return __kasan_kmalloc_large(object, size, flags); else - return ____kasan_kmalloc(page->slab_cache, object, size, - flags, true); + return ____kasan_kmalloc(page->slab_cache, object, size, flags); } void __kasan_kfree_large(void *ptr, unsigned long ip) diff --git a/mm/kasan/kasan.h b/mm/kasan/kasan.h index 4fb8106f8e31..d0a3516e0909 100644 --- a/mm/kasan/kasan.h +++ b/mm/kasan/kasan.h @@ -355,12 +355,51 @@ static inline bool kasan_byte_accessible(const void *addr) #else /* CONFIG_KASAN_HW_TAGS */ -void kasan_poison(const void *address, size_t size, u8 value); -void kasan_unpoison(const void *address, size_t size); +/** + * kasan_poison - mark the memory range as unaccessible + * @addr - range start address, must be aligned to KASAN_GRANULE_SIZE + * @size - range size + * @value - value that's written to metadata for the range + * + * The size gets aligned to KASAN_GRANULE_SIZE before marking the range. + */ +void kasan_poison(const void *addr, size_t size, u8 value); + +/** + * kasan_unpoison - mark the memory range as accessible + * @addr - range start address, must be aligned to KASAN_GRANULE_SIZE + * @size - range size + * + * For the tag-based modes, the @size gets aligned to KASAN_GRANULE_SIZE before + * marking the range. + * For the generic mode, the last granule of the memory range gets partially + * unpoisoned based on the @size. + */ +void kasan_unpoison(const void *addr, size_t size); + bool kasan_byte_accessible(const void *addr); #endif /* CONFIG_KASAN_HW_TAGS */ +#ifdef CONFIG_KASAN_GENERIC + +/** + * kasan_poison_last_granule - mark the last granule of the memory range as + * unaccessible + * @addr - range start address, must be aligned to KASAN_GRANULE_SIZE + * @size - range size + * + * This function is only available for the generic mode, as it's the only mode + * that has partially poisoned memory granules. + */ +void kasan_poison_last_granule(const void *address, size_t size); + +#else /* CONFIG_KASAN_GENERIC */ + +static inline void kasan_poison_last_granule(const void *address, size_t size) { } + +#endif /* CONFIG_KASAN_GENERIC */ + /* * Exported functions for interfaces called from assembly or from generated * code. Declarations here to avoid warning about missing declarations. diff --git a/mm/kasan/shadow.c b/mm/kasan/shadow.c index 1372a2fc0ca9..1ed7817e4ee6 100644 --- a/mm/kasan/shadow.c +++ b/mm/kasan/shadow.c @@ -69,10 +69,6 @@ void *memcpy(void *dest, const void *src, size_t len) return __memcpy(dest, src, len); } -/* - * Poisons the shadow memory for 'size' bytes starting from 'addr'. - * Memory addresses should be aligned to KASAN_GRANULE_SIZE. - */ void kasan_poison(const void *address, size_t size, u8 value) { void *shadow_start, *shadow_end; @@ -83,12 +79,12 @@ void kasan_poison(const void *address, size_t size, u8 value) * addresses to this function. */ address = kasan_reset_tag(address); - size = round_up(size, KASAN_GRANULE_SIZE); /* Skip KFENCE memory if called explicitly outside of sl*b. */ if (is_kfence_address(address)) return; + size = round_up(size, KASAN_GRANULE_SIZE); shadow_start = kasan_mem_to_shadow(address); shadow_end = kasan_mem_to_shadow(address + size); @@ -96,6 +92,16 @@ void kasan_poison(const void *address, size_t size, u8 value) } EXPORT_SYMBOL(kasan_poison); +#ifdef CONFIG_KASAN_GENERIC +void kasan_poison_last_granule(const void *address, size_t size) +{ + if (size & KASAN_GRANULE_MASK) { + u8 *shadow = (u8 *)kasan_mem_to_shadow(address + size); + *shadow = size & KASAN_GRANULE_MASK; + } +} +#endif + void kasan_unpoison(const void *address, size_t size) { u8 tag = get_tag(address); @@ -115,16 +121,12 @@ void kasan_unpoison(const void *address, size_t size) if (is_kfence_address(address)) return; + /* Unpoison round_up(size, KASAN_GRANULE_SIZE) bytes. */ kasan_poison(address, size, tag); - if (size & KASAN_GRANULE_MASK) { - u8 *shadow = (u8 *)kasan_mem_to_shadow(address + size); - - if (IS_ENABLED(CONFIG_KASAN_SW_TAGS)) - *shadow = tag; - else /* CONFIG_KASAN_GENERIC */ - *shadow = size & KASAN_GRANULE_MASK; - } + /* Partially poison the last granule for the generic mode. */ + if (IS_ENABLED(CONFIG_KASAN_GENERIC)) + kasan_poison_last_granule(address, size); } #ifdef CONFIG_MEMORY_HOTPLUG diff --git a/mm/slub.c b/mm/slub.c index 176b1cb0d006..e564008c2329 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -3565,8 +3565,7 @@ static void early_kmem_cache_node_alloc(int node) init_object(kmem_cache_node, n, SLUB_RED_ACTIVE); init_tracking(kmem_cache_node, n); #endif - n = kasan_kmalloc(kmem_cache_node, n, sizeof(struct kmem_cache_node), - GFP_KERNEL); + n = kasan_slab_alloc(kmem_cache_node, n, GFP_KERNEL); page->freelist = get_freepointer(kmem_cache_node, n); page->inuse = 1; page->frozen = 0; From patchwork Fri Feb 5 17:34:37 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrey Konovalov X-Patchwork-Id: 12070583 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.1 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 70C01C433DB for ; Fri, 5 Feb 2021 17:37:12 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 1EBBE60C3F for ; Fri, 5 Feb 2021 17:37:12 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 1EBBE60C3F Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:To:From:Subject:References:Mime-Version:Message-Id: In-Reply-To:Date:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=Mehy53FT3eP+IUV0z8y/U7HPGF90Wf0dZrLAHwZ7Z2Y=; b=G0TxPlIQaC5GTN8P22NnYadev H2RfCRpfMcYpgnNX7vsKblLYk8cF4pYgadP4NxE4Bwt2gXr2kdLm22+Jwq+YIsNBIkInzCrs4Ru0E AedlpGubdi/WmZYufDKfMT55BiX6vCT/lv8uRo0nzrHXpmrOt0tLJIiBgq6R8wuRCbJUCL0+sTDWU SHXTlhwzR5lO19Z6YQ3qvi9bhKCBSX4r81VtQcmc+92T4C29+pzDj87PIty/zcVtybdmoVGIXRcAc op+mE/aVcVBzwXXOcy0lmWsZKJNLaH1k+EmPnp8lHMsaeewnvoCGRgKUf564cp7aDIMoS1Ik/QUHA B8oNr5Bcw==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1l850z-00053T-49; Fri, 05 Feb 2021 17:35:29 +0000 Received: from mail-wr1-x44a.google.com ([2a00:1450:4864:20::44a]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1l850X-0004p7-Ao for linux-arm-kernel@lists.infradead.org; Fri, 05 Feb 2021 17:35:09 +0000 Received: by mail-wr1-x44a.google.com with SMTP id p16so5723833wrx.10 for ; Fri, 05 Feb 2021 09:34:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=tljlSxhBzcxqidAVSdVdhM6oUJ+nJ3d1xitEmX2Ppf0=; b=EooVbzbfAmq8G1bWgvLByZcRTJzsGaZsM+gbsnttAHt59SZ03efBEYHdViNV3NOcBb AIV9WF6XYYWGw3mRwID1biNz4Xd/YnW9mVfWF3MqcCeTu8ZEhARSrRcHYLM1RcDt/U8i v3YvXKISbDMp6LV5R+hWQKWdH3XDXXiz6Y8YegizUaUeJw9yV1lF2WpCyTeZiZQMLUSY laKMOHx3s2jUXdJJtzLun9LBT3MD0Y52b8M44RMyqTJV6KG7pYS+FryEe+WFupRVaQW6 nT3pdbQDrilZswBy/TGLAEBB7NbLJO0EwX6HQFv7HTOCg1BW07zGmBMyOMgF1P3Aima1 hypg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=tljlSxhBzcxqidAVSdVdhM6oUJ+nJ3d1xitEmX2Ppf0=; b=JYfpGjWkwETY2yS24QCZIacgF6jibpHvFbEQbB0hxRsi847RLcvmYN6T4Urke5haaa xAqW9Fn2AXh9lchygecnb5ed71LK2GGLB8Qy9zwa14K0D4v09nts9iglW41BL/tzlHXC li2/yiyN5g6WEI8J960w9Jy9nqG7u2OoJTdO8tU8Wl0r6UcJp5oD8e2Ip5W9DFhCEUT/ U8tYijAKFfA5ch778UBtDJ8sHknidlFdCvVqdp6eDUDG8sOPtB/fDUQ2F9BX1yAVvm7z EIAsDOgSk5DXgOBolIiN1obtOiih43hRbGZmLjLypCJ/eqrAJET6/T+hGECZHmhgcDHi MrqA== X-Gm-Message-State: AOAM5313iILbXS3Hzh8AnUiU4Tx1YWOQC90nFnHdSoX0GQDjlnt03tuw sASKeFz3sdOytU+R1gZNNNFAXybxMapsMLyc X-Google-Smtp-Source: ABdhPJwsjA/F6RHWX96VY6maj/IIDJ50rvRshw+Hr4yR+j1mTxQLGy8N+h/cCmJm78A4s33ZYOd8+8tTyP3YonbL X-Received: from andreyknvl3.muc.corp.google.com ([2a00:79e0:15:13:edb8:b79c:2e20:e531]) (user=andreyknvl job=sendgmr) by 2002:a5d:53c3:: with SMTP id a3mr6032044wrw.43.1612546497774; Fri, 05 Feb 2021 09:34:57 -0800 (PST) Date: Fri, 5 Feb 2021 18:34:37 +0100 In-Reply-To: Message-Id: <33dee5aac0e550ad7f8e26f590c9b02c6129b4a3.1612546384.git.andreyknvl@google.com> Mime-Version: 1.0 References: X-Mailer: git-send-email 2.30.0.365.g02bc693789-goog Subject: [PATCH v3 mm 03/13] kasan: optimize large kmalloc poisoning From: Andrey Konovalov To: Andrew Morton , Catalin Marinas , Vincenzo Frascino , Dmitry Vyukov , Alexander Potapenko , Marco Elver X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210205_123501_571659_D486DA6A X-CRM114-Status: GOOD ( 15.03 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Branislav Rankov , Andrey Konovalov , Kevin Brodsky , Will Deacon , linux-kernel@vger.kernel.org, kasan-dev@googlegroups.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, Andrey Ryabinin , Peter Collingbourne , Evgenii Stepanov Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Similarly to kasan_kmalloc(), kasan_kmalloc_large() doesn't need to unpoison the object as it as already unpoisoned by alloc_pages() (or by ksize() for krealloc()). This patch changes kasan_kmalloc_large() to only poison the redzone. Reviewed-by: Marco Elver Signed-off-by: Andrey Konovalov --- mm/kasan/common.c | 20 +++++++++++++++----- 1 file changed, 15 insertions(+), 5 deletions(-) diff --git a/mm/kasan/common.c b/mm/kasan/common.c index 00edbc3eb32e..f2a6bae13053 100644 --- a/mm/kasan/common.c +++ b/mm/kasan/common.c @@ -494,7 +494,6 @@ EXPORT_SYMBOL(__kasan_kmalloc); void * __must_check __kasan_kmalloc_large(const void *ptr, size_t size, gfp_t flags) { - struct page *page; unsigned long redzone_start; unsigned long redzone_end; @@ -504,12 +503,23 @@ void * __must_check __kasan_kmalloc_large(const void *ptr, size_t size, if (unlikely(ptr == NULL)) return NULL; - page = virt_to_page(ptr); + /* + * The object has already been unpoisoned by kasan_alloc_pages() for + * alloc_pages() or by ksize() for krealloc(). + */ + + /* + * The redzone has byte-level precision for the generic mode. + * Partially poison the last object granule to cover the unaligned + * part of the redzone. + */ + if (IS_ENABLED(CONFIG_KASAN_GENERIC)) + kasan_poison_last_granule(ptr, size); + + /* Poison the aligned part of the redzone. */ redzone_start = round_up((unsigned long)(ptr + size), KASAN_GRANULE_SIZE); - redzone_end = (unsigned long)ptr + page_size(page); - - kasan_unpoison(ptr, size); + redzone_end = (unsigned long)ptr + page_size(virt_to_page(ptr)); kasan_poison((void *)redzone_start, redzone_end - redzone_start, KASAN_PAGE_REDZONE); From patchwork Fri Feb 5 17:34:38 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrey Konovalov X-Patchwork-Id: 12070581 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.1 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 55A32C433DB for ; Fri, 5 Feb 2021 17:36:59 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id EB26864DBA for ; Fri, 5 Feb 2021 17:36:58 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org EB26864DBA Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:To:From:Subject:References:Mime-Version:Message-Id: In-Reply-To:Date:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=2pTM4Abs60f8Xohq1WuYazGkHj0xpztWcE925au1Huw=; b=qw0jzMmMEopmn3iUIWrgOuf3R HM5zOSsTwRbqv2LcC1aA+RyDtn3vXv8vaYpnOOFm5GVoRglMiJ46Q86j4b3KghkUvqLxeUvwm8mvl KDc4sEXq8QnpvzW9cl8+t0v3IceXPvTixebYLAWv4j37esPredI6CnuVxO0D+d24iJuKesRLs3d57 l5LyifW5ydM9/PWgZM4mj0CW87FizG3aflZgzFG880iHvz+d4KS2237/ogPoRjBB6VVUao2VGHSXT 5MsKnLprWm2U1/Cx2Qv3+bcvbwy9IwQD6IcPRDqyIHHJea6e+pKOVzV+KL0RoOimSudLN+AUiBRvk I2hcvokRQ==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1l8519-000560-O8; Fri, 05 Feb 2021 17:35:39 +0000 Received: from mail-wr1-x44a.google.com ([2a00:1450:4864:20::44a]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1l850Z-0004qR-8a for linux-arm-kernel@lists.infradead.org; Fri, 05 Feb 2021 17:35:11 +0000 Received: by mail-wr1-x44a.google.com with SMTP id u3so5710789wri.19 for ; Fri, 05 Feb 2021 09:35:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=p2PHZd2SW9nz4o7+ZB1C3vkb9xhx8DH7M49Qc2p69KA=; b=CHtaYiFloTo1h3fJ4zIwrdm8kH9MlR88HYI0/PTh1ZWy7A7CwjuDxNrv9Gx69/d5EU /uH5wwsCw9lrcu3UgF0woJeX8LvF46mbIjl1UXDMCFstkACeUmi7LjrhuV3m6VC1O4FA kLqgo4s84+U9SrE4Ij3D8TumUHcGxjFAg+UvRisEMQ4KFqgiuA/0irNBtFKpJm7Y2yvr rv5nooR3qty/BvdTJILjp8ZF4GmyrzIoPmJkwuLqIJMGj3IC94nPtvOTvoNrBK4XD6v9 DrdflvRnrv/sqjoyw4HnrP0YYgEVd4PzvqCF+NE7zyq5KY/kgr+v0ibadQ+YFmPfzDvT 14Rg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=p2PHZd2SW9nz4o7+ZB1C3vkb9xhx8DH7M49Qc2p69KA=; b=YKFTP/HFP3DrRZhn8ijMJ0AckIC4yMdXRu727FpLnKDqLVwLYqSfHjPU1nsrB2ixcB md0FbwZuAwgis3kRUFa+JEyLMX1O1EBjAWUz1cyYw/GJO5ofJMgLVSa7WD2GSTQ9aXBW MuxAHzFqieDdHmH2iTiBxRs/T6tlv2DeuWOFtDdlf3Lbjgyln4NJ/zwGtX+Rhhh9xHu1 maPupHzY9EC5gTir9YLwqGed84rn3ALxy2aMWZy0doKSnEHHKBayn2+Y3VUUv/NPvQPM ULCEI72LOSEAQjAP4GzoIjyAWWeiYvZeYkLXZtZZEKeE9oiJeYVZMS+Lri8JAt9qEcXe H/rQ== X-Gm-Message-State: AOAM533rdwiXdePZXBpZif4mq6+q3DUz3g+s7MPYnnTsSH+N/DVCZTUf +tRGD2KuY2LGjm0I8H5SPGAiHUZ3Ii4QC723 X-Google-Smtp-Source: ABdhPJwIwsTCSScqaKOmzq93PUblNEb3imIDP3qjpFcJgdUvxRfwSWqIL28ID41mG6cwO7YBgmA2OyAuujnKhGSd X-Received: from andreyknvl3.muc.corp.google.com ([2a00:79e0:15:13:edb8:b79c:2e20:e531]) (user=andreyknvl job=sendgmr) by 2002:adf:800b:: with SMTP id 11mr4807938wrk.322.1612546500193; Fri, 05 Feb 2021 09:35:00 -0800 (PST) Date: Fri, 5 Feb 2021 18:34:38 +0100 In-Reply-To: Message-Id: Mime-Version: 1.0 References: X-Mailer: git-send-email 2.30.0.365.g02bc693789-goog Subject: [PATCH v3 mm 04/13] kasan: clean up setting free info in kasan_slab_free From: Andrey Konovalov To: Andrew Morton , Catalin Marinas , Vincenzo Frascino , Dmitry Vyukov , Alexander Potapenko , Marco Elver X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210205_123503_491234_57CD609B X-CRM114-Status: GOOD ( 11.14 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Branislav Rankov , Andrey Konovalov , Kevin Brodsky , Will Deacon , linux-kernel@vger.kernel.org, kasan-dev@googlegroups.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, Andrey Ryabinin , Peter Collingbourne , Evgenii Stepanov Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Put kasan_stack_collection_enabled() check and kasan_set_free_info() calls next to each other. The way this was previously implemented was a minor optimization that relied of the the fact that kasan_stack_collection_enabled() is always true for generic KASAN. The confusion that this brings outweights saving a few instructions. Reviewed-by: Marco Elver Signed-off-by: Andrey Konovalov --- mm/kasan/common.c | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/mm/kasan/common.c b/mm/kasan/common.c index f2a6bae13053..da24b144d46c 100644 --- a/mm/kasan/common.c +++ b/mm/kasan/common.c @@ -350,13 +350,11 @@ static bool ____kasan_slab_free(struct kmem_cache *cache, void *object, kasan_poison(object, cache->object_size, KASAN_KMALLOC_FREE); - if (!kasan_stack_collection_enabled()) - return false; - if ((IS_ENABLED(CONFIG_KASAN_GENERIC) && !quarantine)) return false; - kasan_set_free_info(cache, object, tag); + if (kasan_stack_collection_enabled()) + kasan_set_free_info(cache, object, tag); return kasan_quarantine_put(cache, object); } From patchwork Fri Feb 5 17:34:39 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrey Konovalov X-Patchwork-Id: 12070585 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.1 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 03C15C433E0 for ; Fri, 5 Feb 2021 17:37:18 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id B43AB64DBA for ; Fri, 5 Feb 2021 17:37:17 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org B43AB64DBA Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:To:From:Subject:References:Mime-Version:Message-Id: In-Reply-To:Date:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=ZIKbFlasWoqzSzBM5WuuRrQRAw1tBmF0dlli19MBzbc=; b=ECxTLX9zL78LpNgty3tNaYJnp szrCAcTse/ezcVAwG0nZuNuydgChtAIlE+LNwgE7UNapl5T0DvRr5JJLYR1UyLO/zk/gSTIvj3z4M cqxIb2D486fP3OPLl7qOcBmZAY3PXgjTNtICNKkBL83qFL+IzDvtDI1bNHErrJtVxg1nwBzcTbZk3 xorlGOMvI5HuXzbDczCEO4+LaVALGotPCXLWccYGoT3NtGNv87kp2+gancO4mHKeJzL2JsjgbSK6+ cVQ88mWLrhJRyEaLievfmd0zanRh3LBGa4201MaDVkLNwncNZqNOXpiKI6nM1WoOOuH1yKJyzuL4X Z2Lz55Vpw==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1l851J-00059P-MM; Fri, 05 Feb 2021 17:35:49 +0000 Received: from mail-qv1-xf49.google.com ([2607:f8b0:4864:20::f49]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1l850c-0004ra-96 for linux-arm-kernel@lists.infradead.org; Fri, 05 Feb 2021 17:35:17 +0000 Received: by mail-qv1-xf49.google.com with SMTP id v13so5528840qvm.10 for ; Fri, 05 Feb 2021 09:35:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=nFFQgpju2Q7ah9cWGUiX1xDaTa8Y8GPfAWjtc31c0z0=; b=r27Shr/X3udVa2hmQsmAvRhsSQcbPxddxYiMADquyC4ddoUoh3BF44I0qSECkE1QbE XyiBTM1BYB3TBJw8frdxYfnmDe87g3UcOdlqDgb20TclMhoJzVgY8OBFfc9vs9AZMjIF gDC0AV7iYQwTLplxbKhb+BBTgwxY214hlh4lwbughapRkDfgstiFZigokoykURbJ68w8 FbqJGkxclpeNP1ejzOiVkJdKV+0yAsSYMJyY8+DPhwRgzacMSLKDCP3vG3TN0W0WdRb5 tfBHk8hMane0rTGMZyTrOilNtwyf4YJ4IL2zuXBAWqi1U6SnR+UXZmCMMnVzLJEVB5eq q3Kw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=nFFQgpju2Q7ah9cWGUiX1xDaTa8Y8GPfAWjtc31c0z0=; b=i2pBmT9xHRpIDoN++HgJlRb6A6NOJUkKeoowT0M9h4vQ83ELk0Ng1kjFZZfYFfChO1 6NrTHEN80lLeu4RxXfN1rey5u93SjGuCoyMfs3/A3IIhx8AKvqU0WHwtChBBFjvpsCkM cw/f62MS1cVjkWS//2jtRN4KJ8/UvCq5pRMwYt/BwKMNJmOOqveGgS8jO13bpfwwrF5i R0UnHsI9kA+6XEJ4M6F7YtxYEZ7dLo3hm9MWAloo2N62XJdyLhyhVPpHFziuszW/6CHD 713FxYeP5cfKTuidDf7a9lfdeyylVPm3nLFpyAni+4Or8QBcrOCHlpxuesznHfaDwHKZ rd7g== X-Gm-Message-State: AOAM531wUooNTplFrOsDX7w1avHN/TDTgp5tOD6KFw1iI021RgtQPUaM iRuIoknKYJhovWY5c3RuO96e2O4ZZzCXZLYg X-Google-Smtp-Source: ABdhPJzdJEzolm2fJQSPJFKzxG4tHoyBATuOUxru6VS89YHbAkuJ/+VGGvHkSS5auuk3hRDlnvokJ5qr4Q5gNUGu X-Received: from andreyknvl3.muc.corp.google.com ([2a00:79e0:15:13:edb8:b79c:2e20:e531]) (user=andreyknvl job=sendgmr) by 2002:a05:6214:613:: with SMTP id z19mr5363210qvw.2.1612546502427; Fri, 05 Feb 2021 09:35:02 -0800 (PST) Date: Fri, 5 Feb 2021 18:34:39 +0100 In-Reply-To: Message-Id: <14ffc4cd867e0b1ed58f7527e3b748a1b4ad08aa.1612546384.git.andreyknvl@google.com> Mime-Version: 1.0 References: X-Mailer: git-send-email 2.30.0.365.g02bc693789-goog Subject: [PATCH v3 mm 05/13] kasan: unify large kfree checks From: Andrey Konovalov To: Andrew Morton , Catalin Marinas , Vincenzo Frascino , Dmitry Vyukov , Alexander Potapenko , Marco Elver X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210205_123506_419491_C519DFD4 X-CRM114-Status: GOOD ( 16.04 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Branislav Rankov , Andrey Konovalov , Kevin Brodsky , Will Deacon , linux-kernel@vger.kernel.org, kasan-dev@googlegroups.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, Andrey Ryabinin , Peter Collingbourne , Evgenii Stepanov Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Unify checks in kasan_kfree_large() and in kasan_slab_free_mempool() for large allocations as it's done for small kfree() allocations. With this change, kasan_slab_free_mempool() starts checking that the first byte of the memory that's being freed is accessible. Reviewed-by: Marco Elver Signed-off-by: Andrey Konovalov --- include/linux/kasan.h | 16 ++++++++-------- mm/kasan/common.c | 36 ++++++++++++++++++++++++++---------- 2 files changed, 34 insertions(+), 18 deletions(-) diff --git a/include/linux/kasan.h b/include/linux/kasan.h index e6ed969e74b3..14f72ec96492 100644 --- a/include/linux/kasan.h +++ b/include/linux/kasan.h @@ -200,6 +200,13 @@ static __always_inline bool kasan_slab_free(struct kmem_cache *s, void *object) return false; } +void __kasan_kfree_large(void *ptr, unsigned long ip); +static __always_inline void kasan_kfree_large(void *ptr) +{ + if (kasan_enabled()) + __kasan_kfree_large(ptr, _RET_IP_); +} + void __kasan_slab_free_mempool(void *ptr, unsigned long ip); static __always_inline void kasan_slab_free_mempool(void *ptr) { @@ -247,13 +254,6 @@ static __always_inline void * __must_check kasan_krealloc(const void *object, return (void *)object; } -void __kasan_kfree_large(void *ptr, unsigned long ip); -static __always_inline void kasan_kfree_large(void *ptr) -{ - if (kasan_enabled()) - __kasan_kfree_large(ptr, _RET_IP_); -} - /* * Unlike kasan_check_read/write(), kasan_check_byte() is performed even for * the hardware tag-based mode that doesn't rely on compiler instrumentation. @@ -302,6 +302,7 @@ static inline bool kasan_slab_free(struct kmem_cache *s, void *object) { return false; } +static inline void kasan_kfree_large(void *ptr) {} static inline void kasan_slab_free_mempool(void *ptr) {} static inline void *kasan_slab_alloc(struct kmem_cache *s, void *object, gfp_t flags) @@ -322,7 +323,6 @@ static inline void *kasan_krealloc(const void *object, size_t new_size, { return (void *)object; } -static inline void kasan_kfree_large(void *ptr) {} static inline bool kasan_check_byte(const void *address) { return true; diff --git a/mm/kasan/common.c b/mm/kasan/common.c index da24b144d46c..7ea643f7e69c 100644 --- a/mm/kasan/common.c +++ b/mm/kasan/common.c @@ -364,6 +364,31 @@ bool __kasan_slab_free(struct kmem_cache *cache, void *object, unsigned long ip) return ____kasan_slab_free(cache, object, ip, true); } +static bool ____kasan_kfree_large(void *ptr, unsigned long ip) +{ + if (ptr != page_address(virt_to_head_page(ptr))) { + kasan_report_invalid_free(ptr, ip); + return true; + } + + if (!kasan_byte_accessible(ptr)) { + kasan_report_invalid_free(ptr, ip); + return true; + } + + /* + * The object will be poisoned by kasan_free_pages() or + * kasan_slab_free_mempool(). + */ + + return false; +} + +void __kasan_kfree_large(void *ptr, unsigned long ip) +{ + ____kasan_kfree_large(ptr, ip); +} + void __kasan_slab_free_mempool(void *ptr, unsigned long ip) { struct page *page; @@ -377,10 +402,8 @@ void __kasan_slab_free_mempool(void *ptr, unsigned long ip) * KMALLOC_MAX_SIZE, and kmalloc falls back onto page_alloc. */ if (unlikely(!PageSlab(page))) { - if (ptr != page_address(page)) { - kasan_report_invalid_free(ptr, ip); + if (____kasan_kfree_large(ptr, ip)) return; - } kasan_poison(ptr, page_size(page), KASAN_FREE_PAGE); } else { ____kasan_slab_free(page->slab_cache, ptr, ip, false); @@ -539,13 +562,6 @@ void * __must_check __kasan_krealloc(const void *object, size_t size, gfp_t flag return ____kasan_kmalloc(page->slab_cache, object, size, flags); } -void __kasan_kfree_large(void *ptr, unsigned long ip) -{ - if (ptr != page_address(virt_to_head_page(ptr))) - kasan_report_invalid_free(ptr, ip); - /* The object will be poisoned by kasan_free_pages(). */ -} - bool __kasan_check_byte(const void *address, unsigned long ip) { if (!kasan_byte_accessible(address)) { From patchwork Fri Feb 5 17:34:40 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrey Konovalov X-Patchwork-Id: 12070587 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.1 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 22F9AC433DB for ; Fri, 5 Feb 2021 17:37:21 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id B8D8B64DBA for ; Fri, 5 Feb 2021 17:37:20 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org B8D8B64DBA Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:To:From:Subject:References:Mime-Version:Message-Id: In-Reply-To:Date:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=00qHrBbrlk3ng/dyiu6TUoAQgpztyayGnnKeZx20LDI=; b=GEeernNYCbznql0IsZB3exzLn Xs5giWj6ISTv/UGWBwpXh9UVS1mgONcR8yjKLZjssK9Jg/5lDLrkasBIFn2fnWCsPF3TZiJt8/MdX gHlwfqcXWwocJnN5zu8zoPjI1K6nrKdOfSNb5cwnWVFyognnorI/0otBU2pt84vDd4CeNFiWca0Uv S0nVCUq2r16QwuXtcSmKbvhXhom7UW2/8aH0wUA42CQGoh8RGnGGmMNvalC5IFA7PtiGhFZcmXcp4 ROL0lKryxIE6DE9F1+ydbwodQvwVcO2svMZ6dWvwRmW6siFWUDYnCg4dEvJ1vPfeVtyqZr6iVjPvT RduYsr/Bw==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1l851Q-0005D1-Re; Fri, 05 Feb 2021 17:35:56 +0000 Received: from mail-wr1-x44a.google.com ([2a00:1450:4864:20::44a]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1l850e-0004tT-V6 for linux-arm-kernel@lists.infradead.org; Fri, 05 Feb 2021 17:35:19 +0000 Received: by mail-wr1-x44a.google.com with SMTP id x12so5712682wrw.21 for ; Fri, 05 Feb 2021 09:35:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=WvobzSj421U2nh52f1sEoHIHe/BM9iFVUw3RqQILbkc=; b=NNedEXHnWuo6mudQu55pc85Aj8ZkARgExTe1GyiP//qvZCPgdJYoRMIytaxulPpENP 5m1UVeBdJCEJGndVwvbfCkgSYO0C9rL4pVy43HKK1B1XVeNg5b62xCGqn5v8Qh+IlOCL Z8y1xgKP51VVTVAvtqhaGHX2Gd2rlut623m9JBsMdQ0JCtXKRh0cZUakJeUCSgW01IFW 4jDbbpMJf1turd9ap5dKzKvSp1aSdk/eco4V90CWDmBtbcR0DLoHGPdb6wFqGhfy1ifP bOnqCbXQFu7nWzu5ypuM4z61ktabT+Xr2DOtVjR+7zwFld+Mi9L68l/xjVzO+c81Wm5k ynDA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=WvobzSj421U2nh52f1sEoHIHe/BM9iFVUw3RqQILbkc=; b=dXrmXg3ZTdY6YtlF/VA6Y4Z4XbHfWq/ixOLGdVGkd+c3uZMxGU2pWhY7KuF+NcoOki wpDh2+v/Ep31gqw5JscEGiriih/xTMfozBlmKYIJRV1BYLSACoVoWzsnFQUxRTLJpw/z gSXWJOGoOB2YBNjwsqiNBcAA2SFZdL5v7b7N3qvhslOFMAYjKG9sNdgITSVfOLE3QUIn 2s38uSI6cVGRDxcZhTHB8kpsk/I7lUU3cDOcB9yxl36Lp7GNTW41Cta8AqPo5/6GLIEA gUvm8/z1WhkoTMduTZ6uqP328+F+ZpyJvCgcgh7Geh9lgegPqa5iOf+IImJXSE53QVpK zuYA== X-Gm-Message-State: AOAM530oC6bhdWXqy73JS3j6mpZgxNkF/L9nkK8U6KLpmATz40Rv5K/M 3eVXOA/WdglZHaUJKj7cz59PfvYwMCirudcs X-Google-Smtp-Source: ABdhPJx4HJoO7CbKgZftXFbdt22PtMMn75VuP2ft+tzw4fvt8SPfqelmj8N1SzNcHwszBmgAjB1HSrpNRQNyRInI X-Received: from andreyknvl3.muc.corp.google.com ([2a00:79e0:15:13:edb8:b79c:2e20:e531]) (user=andreyknvl job=sendgmr) by 2002:a05:600c:154f:: with SMTP id f15mr4396184wmg.20.1612546504994; Fri, 05 Feb 2021 09:35:04 -0800 (PST) Date: Fri, 5 Feb 2021 18:34:40 +0100 In-Reply-To: Message-Id: Mime-Version: 1.0 References: X-Mailer: git-send-email 2.30.0.365.g02bc693789-goog Subject: [PATCH v3 mm 06/13] kasan: rework krealloc tests From: Andrey Konovalov To: Andrew Morton , Catalin Marinas , Vincenzo Frascino , Dmitry Vyukov , Alexander Potapenko , Marco Elver X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210205_123509_063685_CE066831 X-CRM114-Status: GOOD ( 13.83 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Branislav Rankov , Andrey Konovalov , Kevin Brodsky , Will Deacon , linux-kernel@vger.kernel.org, kasan-dev@googlegroups.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, Andrey Ryabinin , Peter Collingbourne , Evgenii Stepanov Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org This patch reworks KASAN-KUnit tests for krealloc() to: 1. Check both slab and page_alloc based krealloc() implementations. 2. Allow at least one full granule to fit between old and new sizes for each KASAN mode, and check accesses to that granule accordingly. Reviewed-by: Marco Elver Signed-off-by: Andrey Konovalov --- lib/test_kasan.c | 91 ++++++++++++++++++++++++++++++++++++++++++------ 1 file changed, 81 insertions(+), 10 deletions(-) diff --git a/lib/test_kasan.c b/lib/test_kasan.c index d16ec9e66806..ffebad2f0e6e 100644 --- a/lib/test_kasan.c +++ b/lib/test_kasan.c @@ -252,11 +252,14 @@ static void kmalloc_large_oob_right(struct kunit *test) kfree(ptr); } -static void kmalloc_oob_krealloc_more(struct kunit *test) +static void krealloc_more_oob_helper(struct kunit *test, + size_t size1, size_t size2) { char *ptr1, *ptr2; - size_t size1 = 17; - size_t size2 = 19; + size_t middle; + + KUNIT_ASSERT_LT(test, size1, size2); + middle = size1 + (size2 - size1) / 2; ptr1 = kmalloc(size1, GFP_KERNEL); KUNIT_ASSERT_NOT_ERR_OR_NULL(test, ptr1); @@ -264,15 +267,31 @@ static void kmalloc_oob_krealloc_more(struct kunit *test) ptr2 = krealloc(ptr1, size2, GFP_KERNEL); KUNIT_ASSERT_NOT_ERR_OR_NULL(test, ptr2); - KUNIT_EXPECT_KASAN_FAIL(test, ptr2[size2 + OOB_TAG_OFF] = 'x'); + /* All offsets up to size2 must be accessible. */ + ptr2[size1 - 1] = 'x'; + ptr2[size1] = 'x'; + ptr2[middle] = 'x'; + ptr2[size2 - 1] = 'x'; + + /* Generic mode is precise, so unaligned size2 must be inaccessible. */ + if (IS_ENABLED(CONFIG_KASAN_GENERIC)) + KUNIT_EXPECT_KASAN_FAIL(test, ptr2[size2] = 'x'); + + /* For all modes first aligned offset after size2 must be inaccessible. */ + KUNIT_EXPECT_KASAN_FAIL(test, + ptr2[round_up(size2, KASAN_GRANULE_SIZE)] = 'x'); + kfree(ptr2); } -static void kmalloc_oob_krealloc_less(struct kunit *test) +static void krealloc_less_oob_helper(struct kunit *test, + size_t size1, size_t size2) { char *ptr1, *ptr2; - size_t size1 = 17; - size_t size2 = 15; + size_t middle; + + KUNIT_ASSERT_LT(test, size2, size1); + middle = size2 + (size1 - size2) / 2; ptr1 = kmalloc(size1, GFP_KERNEL); KUNIT_ASSERT_NOT_ERR_OR_NULL(test, ptr1); @@ -280,10 +299,60 @@ static void kmalloc_oob_krealloc_less(struct kunit *test) ptr2 = krealloc(ptr1, size2, GFP_KERNEL); KUNIT_ASSERT_NOT_ERR_OR_NULL(test, ptr2); - KUNIT_EXPECT_KASAN_FAIL(test, ptr2[size2 + OOB_TAG_OFF] = 'x'); + /* Must be accessible for all modes. */ + ptr2[size2 - 1] = 'x'; + + /* Generic mode is precise, so unaligned size2 must be inaccessible. */ + if (IS_ENABLED(CONFIG_KASAN_GENERIC)) + KUNIT_EXPECT_KASAN_FAIL(test, ptr2[size2] = 'x'); + + /* For all modes first aligned offset after size2 must be inaccessible. */ + KUNIT_EXPECT_KASAN_FAIL(test, + ptr2[round_up(size2, KASAN_GRANULE_SIZE)] = 'x'); + + /* + * For all modes all size2, middle, and size1 should land in separate + * granules and thus the latter two offsets should be inaccessible. + */ + KUNIT_EXPECT_LE(test, round_up(size2, KASAN_GRANULE_SIZE), + round_down(middle, KASAN_GRANULE_SIZE)); + KUNIT_EXPECT_LE(test, round_up(middle, KASAN_GRANULE_SIZE), + round_down(size1, KASAN_GRANULE_SIZE)); + KUNIT_EXPECT_KASAN_FAIL(test, ptr2[middle] = 'x'); + KUNIT_EXPECT_KASAN_FAIL(test, ptr2[size1 - 1] = 'x'); + KUNIT_EXPECT_KASAN_FAIL(test, ptr2[size1] = 'x'); + kfree(ptr2); } +static void krealloc_more_oob(struct kunit *test) +{ + krealloc_more_oob_helper(test, 201, 235); +} + +static void krealloc_less_oob(struct kunit *test) +{ + krealloc_less_oob_helper(test, 235, 201); +} + +static void krealloc_pagealloc_more_oob(struct kunit *test) +{ + /* page_alloc fallback in only implemented for SLUB. */ + KASAN_TEST_NEEDS_CONFIG_ON(test, CONFIG_SLUB); + + krealloc_more_oob_helper(test, KMALLOC_MAX_CACHE_SIZE + 201, + KMALLOC_MAX_CACHE_SIZE + 235); +} + +static void krealloc_pagealloc_less_oob(struct kunit *test) +{ + /* page_alloc fallback in only implemented for SLUB. */ + KASAN_TEST_NEEDS_CONFIG_ON(test, CONFIG_SLUB); + + krealloc_less_oob_helper(test, KMALLOC_MAX_CACHE_SIZE + 235, + KMALLOC_MAX_CACHE_SIZE + 201); +} + static void kmalloc_oob_16(struct kunit *test) { struct { @@ -977,8 +1046,10 @@ static struct kunit_case kasan_kunit_test_cases[] = { KUNIT_CASE(pagealloc_oob_right), KUNIT_CASE(pagealloc_uaf), KUNIT_CASE(kmalloc_large_oob_right), - KUNIT_CASE(kmalloc_oob_krealloc_more), - KUNIT_CASE(kmalloc_oob_krealloc_less), + KUNIT_CASE(krealloc_more_oob), + KUNIT_CASE(krealloc_less_oob), + KUNIT_CASE(krealloc_pagealloc_more_oob), + KUNIT_CASE(krealloc_pagealloc_less_oob), KUNIT_CASE(kmalloc_oob_16), KUNIT_CASE(kmalloc_uaf_16), KUNIT_CASE(kmalloc_oob_in_memset), From patchwork Fri Feb 5 17:34:41 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrey Konovalov X-Patchwork-Id: 12070591 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.1 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 434DCC433E6 for ; Fri, 5 Feb 2021 17:37:25 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id D742E64DD4 for ; Fri, 5 Feb 2021 17:37:24 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org D742E64DD4 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:To:From:Subject:References:Mime-Version:Message-Id: In-Reply-To:Date:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=vc01b6T5vX6Tv93xEtlFwB+V49qmS+FbCTgXvG6ILpY=; b=SK027k++LZxfYs7zTnFcMzUyk 4c5huepSgdF8BqxuyGy67eQehM66yh063B3sxRStsCj2r/YpH+cCieedfhFGC1ealsxW09c+tajYr LUQpEHanGz7ECn6Tec2Ftfunm0L88/KG3d8dlpgDasFprc48UNTv2DQz4kVL7l41JnuR/FSOtQfwA +HNZPyaINGi6hZLvHzmDnnqgMIwYX3c7ku19/vCGxFGde8LhhxJPWpmQd3rntWV4E1Lz6orfciEMS 3K3+VPEwamoioRYzRYAMDvzrw1b7u2+OHNRpExHyb/+AhymzZOfbyZLPYRV8w7u5HYl5qNJe/mam3 4ohAGNxeg==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1l851S-0005Dt-VX; Fri, 05 Feb 2021 17:35:59 +0000 Received: from mail-wm1-x34a.google.com ([2a00:1450:4864:20::34a]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1l850h-0004uE-8I for linux-arm-kernel@lists.infradead.org; Fri, 05 Feb 2021 17:35:20 +0000 Received: by mail-wm1-x34a.google.com with SMTP id t13so4075990wmq.7 for ; Fri, 05 Feb 2021 09:35:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=6RV3PXvNWDRRLwJfhc9nA9uOtjOO4vRR2YJTlHhgy1g=; b=hr9DzLw8PMFXDmOtc8ZAyVLRhSu9H1GXAWRc7Isd9/XTI8Cgi/jOQVN1GFDvZnERyB G0HDSX6x8jQqkjNbE1UygZ3nJdUqxCSAPKSLVU27KQJjuRq2fa0ClujER0+PhSTzUcbZ nrhIBgxy8C9/vLXSIHaHysZ86Cf8ZxwaoECfAGez4vWWEVDoSH037sAnwgHeL68Yg5iV FjqojbgDKu9rTiWhHFSCGcCAnibfMdcmw/QUZF+vozj8e9iaXiBtzMtFGaPsuvLT80Kw 2VNm/yfw0oadQd2OIkqOcJCfOagcSL+70vJBOSCrH8Uczn7HUXpEpK1ijgH8yi9KwdHb prUQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=6RV3PXvNWDRRLwJfhc9nA9uOtjOO4vRR2YJTlHhgy1g=; b=IyDZ7z/9kd91twFzoijXgF+VM75bClGWExZSpBmfvYpcXP6BYK9JHcsp/vo4+3e7VL 9JYip1BuBbqluc+FY9wtjKzk3P9jYmKuJqUA9CWqBSengg/Uk+UVKEZZkL5DAZbXjnqt f+/+U8uBrWhCAR2TfrWlQjBqmkhb0S2/Gw9T6DIqj4QfhbIOilc8NexW3MnIORDoB0D4 vvE+iC48K92iY0OIe9kEfJTMBIs1Aq6NZ8cZHjyV3K49HaS/rkpAvcCQ6LdDAWUrWVmG vgCiO7BnWn+KJczH+TQ50KurDjtJT8j+UJZFmcz48Oa3uOAqo9p8YP//Eg8kDGBL/4AS DBuA== X-Gm-Message-State: AOAM533dBogtW0GrEJK+9mA9NGpPRjqt2I7Si5VTc7kW8qJL626U295N bXA+oOyc2oCrWIW1b3jAka3/l/w9TuOnjjlz X-Google-Smtp-Source: ABdhPJytRzXE5voKMAWTVE7tb2OGC2+Y1oliS+AIv9OdSPsqoc+dVJE3s1RRE+LkSF8dIUprMqGcmE4wi/aHTTwX X-Received: from andreyknvl3.muc.corp.google.com ([2a00:79e0:15:13:edb8:b79c:2e20:e531]) (user=andreyknvl job=sendgmr) by 2002:a1c:3185:: with SMTP id x127mr4375625wmx.117.1612546507585; Fri, 05 Feb 2021 09:35:07 -0800 (PST) Date: Fri, 5 Feb 2021 18:34:41 +0100 In-Reply-To: Message-Id: Mime-Version: 1.0 References: X-Mailer: git-send-email 2.30.0.365.g02bc693789-goog Subject: [PATCH v3 mm 07/13] kasan, mm: fail krealloc on freed objects From: Andrey Konovalov To: Andrew Morton , Catalin Marinas , Vincenzo Frascino , Dmitry Vyukov , Alexander Potapenko , Marco Elver X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210205_123511_342369_80A716F6 X-CRM114-Status: GOOD ( 17.62 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Branislav Rankov , Andrey Konovalov , Kevin Brodsky , Will Deacon , linux-kernel@vger.kernel.org, kasan-dev@googlegroups.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, Andrey Ryabinin , Peter Collingbourne , Evgenii Stepanov Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Currently, if krealloc() is called on a freed object with KASAN enabled, it allocates and returns a new object, but doesn't copy any memory from the old one as ksize() returns 0. This makes the caller believe that krealloc() succeeded (KASAN report is printed though). This patch adds an accessibility check into __do_krealloc(). If the check fails, krealloc() returns NULL. This check duplicates the one in ksize(); this is fixed in the following patch. This patch also adds a KASAN-KUnit test to check krealloc() behaviour when it's called on a freed object. Reviewed-by: Marco Elver Signed-off-by: Andrey Konovalov --- lib/test_kasan.c | 20 ++++++++++++++++++++ mm/slab_common.c | 3 +++ 2 files changed, 23 insertions(+) diff --git a/lib/test_kasan.c b/lib/test_kasan.c index ffebad2f0e6e..1328c468fdb5 100644 --- a/lib/test_kasan.c +++ b/lib/test_kasan.c @@ -353,6 +353,25 @@ static void krealloc_pagealloc_less_oob(struct kunit *test) KMALLOC_MAX_CACHE_SIZE + 201); } +/* + * Check that krealloc() detects a use-after-free, returns NULL, + * and doesn't unpoison the freed object. + */ +static void krealloc_uaf(struct kunit *test) +{ + char *ptr1, *ptr2; + int size1 = 201; + int size2 = 235; + + ptr1 = kmalloc(size1, GFP_KERNEL); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, ptr1); + kfree(ptr1); + + KUNIT_EXPECT_KASAN_FAIL(test, ptr2 = krealloc(ptr1, size2, GFP_KERNEL)); + KUNIT_ASSERT_PTR_EQ(test, (void *)ptr2, NULL); + KUNIT_EXPECT_KASAN_FAIL(test, *(volatile char *)ptr1); +} + static void kmalloc_oob_16(struct kunit *test) { struct { @@ -1050,6 +1069,7 @@ static struct kunit_case kasan_kunit_test_cases[] = { KUNIT_CASE(krealloc_less_oob), KUNIT_CASE(krealloc_pagealloc_more_oob), KUNIT_CASE(krealloc_pagealloc_less_oob), + KUNIT_CASE(krealloc_uaf), KUNIT_CASE(kmalloc_oob_16), KUNIT_CASE(kmalloc_uaf_16), KUNIT_CASE(kmalloc_oob_in_memset), diff --git a/mm/slab_common.c b/mm/slab_common.c index 39d1a8ff9bb8..dad70239b54c 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -1140,6 +1140,9 @@ static __always_inline void *__do_krealloc(const void *p, size_t new_size, void *ret; size_t ks; + if (likely(!ZERO_OR_NULL_PTR(p)) && !kasan_check_byte(p)) + return NULL; + ks = ksize(p); if (ks >= new_size) { From patchwork Fri Feb 5 17:34:42 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrey Konovalov X-Patchwork-Id: 12070589 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.1 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C2C1EC433DB for ; Fri, 5 Feb 2021 17:37:24 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 5695064DD4 for ; Fri, 5 Feb 2021 17:37:24 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 5695064DD4 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:To:From:Subject:References:Mime-Version:Message-Id: In-Reply-To:Date:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=KNc/Aqb0R1ljnH8mLrlAiD++LrvE1EqRwG635zi2YRA=; b=P7Yem3ZNv1UX16sHmwo2s/0rt uOisb/4QCbu2DNKANdH3VGrQ/WUhXK5rZ705o26+QvjHIIcaVpm8HhiJ6+D9o4ua2pxSg5+96wAJf zXX8oGVJ0WUZSYU8o+2Xc1nR1LJ0C961HvSV4+AXyncSkhXbDfACZ+AQqqZt6CCMUH5iwP9uKNn/3 yDFWaGCEwProD8606vDclMOd+qxOqylAdWkUvW1bQl74dTiTqweybbqPD5KroCCeK+p5DrchT+Bmz OPpdqrIVl65fq8h/P72XvSUiknCCKqByfVU6iBc+whNzytWZXzFKwWES7KWGDqMzW9bVDNMYPJU3P PW9+8mlYA==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1l851V-0005En-W5; Fri, 05 Feb 2021 17:36:02 +0000 Received: from mail-qv1-xf4a.google.com ([2607:f8b0:4864:20::f4a]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1l850j-0004vU-P4 for linux-arm-kernel@lists.infradead.org; Fri, 05 Feb 2021 17:35:23 +0000 Received: by mail-qv1-xf4a.google.com with SMTP id a12so5557320qvo.3 for ; Fri, 05 Feb 2021 09:35:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=ISHwxx5EnSKzUDUjZ1ysUZFyHBfF802klWxWhYA/428=; b=RwM46b1OK05tP9oh3ZvOyyqP5DeUCm5WEgXs9WIi2QDZ10b7a1JK5ykxxwcHrHcz9b eDMl73sT/lR0Hw6RbGi7n8FmxAtam+z6WvhhjYDN0jIge9F1av+HlKfQ0YVNHLJwTL4a wYltM3xRPcjR+cLRrDGpWcNlDeVQ0jGBPh5FGQNKahJiLulk3hX2Cp1yjBVRjR+/+5Jb tLWvCNv5ECt+IWcvAMdNWQlPComv7q0GUtli6S3pwLuKz9J3Ab96/KplUlWGVJ5DCNPP OUmhAtwjQXJraW/q+HNZTthQdlqCfh8OL9LVIJ6y66/8ZaTjbezpHKnepMfZm+KLEc5v HBiQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=ISHwxx5EnSKzUDUjZ1ysUZFyHBfF802klWxWhYA/428=; b=YRiojdFSUP0cnRqyvjzWPhEcvC9qsIOQNqnBF6uirJ/iMF9ZfUsGeoKwZJVK2zqa5H TA+xnWiFfx9eQTqO+ApWdqWAKjTqxs0J4YYbTtd0n5uDd3rVh3ydaCqGQBMrGsGyh9CZ vf9hTXmGQrURGYPa2CHMKwYlP27yu6Pswf10xRQpnsdpVQS7VYx2KNHXtN9PvAr69WvX HeFcvDrPL2trlf44bNWk7+pg5/SRKSbg0BXuc9ltzpQhYaqkRanKRalCgd60VZXPdUVr SUuRH/F6E+Yy2f11jRcQEEXrAInrNPn5hCtXC19W9FhQn6RZd6Opiqi7qp2vIPRpMbci kQHA== X-Gm-Message-State: AOAM533BHWJQRp5rI8CzQJjrWRhU6jZo4tMQerC6xa8TOaiUBQkf+xAC XxHSV6ufDhXF3QoC9Yj3sGYRCW7n5zL040j/ X-Google-Smtp-Source: ABdhPJzeuGXvKwDVMKQZvPztFjBxnrWar5viffIlpldRL8/ioMWlXHOlQwfTr/fGJyK6XQY22n8CnOgwdlEcN5sq X-Received: from andreyknvl3.muc.corp.google.com ([2a00:79e0:15:13:edb8:b79c:2e20:e531]) (user=andreyknvl job=sendgmr) by 2002:a0c:c687:: with SMTP id d7mr5237396qvj.17.1612546510164; Fri, 05 Feb 2021 09:35:10 -0800 (PST) Date: Fri, 5 Feb 2021 18:34:42 +0100 In-Reply-To: Message-Id: <9bef90327c9cb109d736c40115684fd32f49e6b0.1612546384.git.andreyknvl@google.com> Mime-Version: 1.0 References: X-Mailer: git-send-email 2.30.0.365.g02bc693789-goog Subject: [PATCH v3 mm 08/13] kasan, mm: optimize krealloc poisoning From: Andrey Konovalov To: Andrew Morton , Catalin Marinas , Vincenzo Frascino , Dmitry Vyukov , Alexander Potapenko , Marco Elver X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210205_123514_018038_094F4BE3 X-CRM114-Status: GOOD ( 20.43 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Branislav Rankov , Andrey Konovalov , Kevin Brodsky , Will Deacon , linux-kernel@vger.kernel.org, kasan-dev@googlegroups.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, Andrey Ryabinin , Peter Collingbourne , Evgenii Stepanov Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Currently, krealloc() always calls ksize(), which unpoisons the whole object including the redzone. This is inefficient, as kasan_krealloc() repoisons the redzone for objects that fit into the same buffer. This patch changes krealloc() instrumentation to use uninstrumented __ksize() that doesn't unpoison the memory. Instead, kasan_kreallos() is changed to unpoison the memory excluding the redzone. For objects that don't fit into the old allocation, this patch disables KASAN accessibility checks when copying memory into a new object instead of unpoisoning it. Signed-off-by: Andrey Konovalov Reviewed-by: Marco Elver --- mm/kasan/common.c | 12 ++++++++++-- mm/slab_common.c | 20 ++++++++++++++------ 2 files changed, 24 insertions(+), 8 deletions(-) diff --git a/mm/kasan/common.c b/mm/kasan/common.c index 7ea643f7e69c..a8a67dca5e55 100644 --- a/mm/kasan/common.c +++ b/mm/kasan/common.c @@ -476,7 +476,7 @@ static void *____kasan_kmalloc(struct kmem_cache *cache, const void *object, /* * The object has already been unpoisoned by kasan_slab_alloc() for - * kmalloc() or by ksize() for krealloc(). + * kmalloc() or by kasan_krealloc() for krealloc(). */ /* @@ -526,7 +526,7 @@ void * __must_check __kasan_kmalloc_large(const void *ptr, size_t size, /* * The object has already been unpoisoned by kasan_alloc_pages() for - * alloc_pages() or by ksize() for krealloc(). + * alloc_pages() or by kasan_krealloc() for krealloc(). */ /* @@ -554,8 +554,16 @@ void * __must_check __kasan_krealloc(const void *object, size_t size, gfp_t flag if (unlikely(object == ZERO_SIZE_PTR)) return (void *)object; + /* + * Unpoison the object's data. + * Part of it might already have been unpoisoned, but it's unknown + * how big that part is. + */ + kasan_unpoison(object, size); + page = virt_to_head_page(object); + /* Piggy-back on kmalloc() instrumentation to poison the redzone. */ if (unlikely(!PageSlab(page))) return __kasan_kmalloc_large(object, size, flags); else diff --git a/mm/slab_common.c b/mm/slab_common.c index dad70239b54c..60a2f49df6ce 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -1140,19 +1140,27 @@ static __always_inline void *__do_krealloc(const void *p, size_t new_size, void *ret; size_t ks; - if (likely(!ZERO_OR_NULL_PTR(p)) && !kasan_check_byte(p)) - return NULL; - - ks = ksize(p); + /* Don't use instrumented ksize to allow precise KASAN poisoning. */ + if (likely(!ZERO_OR_NULL_PTR(p))) { + if (!kasan_check_byte(p)) + return NULL; + ks = kfence_ksize(p) ?: __ksize(p); + } else + ks = 0; + /* If the object still fits, repoison it precisely. */ if (ks >= new_size) { p = kasan_krealloc((void *)p, new_size, flags); return (void *)p; } ret = kmalloc_track_caller(new_size, flags); - if (ret && p) - memcpy(ret, p, ks); + if (ret && p) { + /* Disable KASAN checks as the object's redzone is accessed. */ + kasan_disable_current(); + memcpy(ret, kasan_reset_tag(p), ks); + kasan_enable_current(); + } return ret; } From patchwork Fri Feb 5 17:34:43 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrey Konovalov X-Patchwork-Id: 12070595 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.1 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C0395C433E6 for ; Fri, 5 Feb 2021 17:37:34 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 3F9DD64DBA for ; Fri, 5 Feb 2021 17:37:34 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 3F9DD64DBA Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:To:From:Subject:References:Mime-Version:Message-Id: In-Reply-To:Date:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=e572fQGhNYOMeBRDgOVZsGLvg2DGNkg22FSR+KO/X1A=; b=sQaFXiq88dJ55kATuSYjnQ9wo 5gKl972KQ2hKfQQcUmEjQMcz/2mhOIfgfRMfoltzOoUt23XAXwpcu+1yGq5YNHYeb2Ca+6xxQU4+z r4JnKXn9f2csVbe2+t48MTutz0/9Pjw5tb7p+Y8u1xmouEyUiWLYV6+ra6DudayaLHq++1NwLmJa+ vdJh37wiUM0abCZjsCWmxrupZH1jZUtghVwBBjKIwfe+KD5N9tPRP/RgV6acaGeuLRCwxPrBdFbhi IR9RJMt1hBMt0Im/kz3lrZGV0I949PDxGgAB/lMiRQBxNrrVjcKh22Yj+KgZYlKbtYbZdYLOsFPVP 06GXyzIUg==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1l851b-0005I0-64; Fri, 05 Feb 2021 17:36:08 +0000 Received: from mail-ed1-x549.google.com ([2a00:1450:4864:20::549]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1l850m-0004wW-VA for linux-arm-kernel@lists.infradead.org; Fri, 05 Feb 2021 17:35:29 +0000 Received: by mail-ed1-x549.google.com with SMTP id w23so7247953edr.15 for ; Fri, 05 Feb 2021 09:35:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=qufdT2LxTRMhYRcFfEhB7241MbQS+EEn9OJq6bu53kI=; b=o3q+JKMBB2p6vxYp6UlNQonm7sinRipxEBzz0BaJM3rqo2apLOKCW0H29zn/wqpROs /iOZbrx0EpfNLyjqUm8hTQsjxWGPdm46yggYbtmuYywiKbQSRPJAFxEMFTe2i1VtBADV 3YeuKIZmcNp4jRuVBRSGExLiNPQ0/9mX7ulWbwSalSRCHJaaddgKdEGlD5UIW4uewCgH u8sbQs+sZWKs3y2rejq6DyAEJy0OdPTNc79FOttdEOg46cmZar5QKvx3Fee86H3fb02X 9JZob9YeC/kUfqm0uyfXeTmxYTa1Zi3gl3rVutvtV9CSaIRwIEessmkDKi+KSRqtI0ST 76bA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=qufdT2LxTRMhYRcFfEhB7241MbQS+EEn9OJq6bu53kI=; b=Ajg1BJcFuRW9MtLlNHrXBSX55iz19G2nnzMZQ8VLOo88zH/NZ0u1K8ECPVxy62srN2 x91LGYfM+Kxa7LUAs68g+gloHjfA4v8a7J1xKpbos0IDDmB4K32fM4i1EoC/QHzsFp9P 02L13Z5dksDhaoSVyEHuxBhlkStlYv5V9I/hXUOh13gDe0OmnABMPMOQ4lWL7B82J444 LYXFzCgZsASu5r5ku5+YF3YE6v00sQKeM0E2UBkaqD+bGNAW9YWpO4PZw/QMR0/MTQii amcEr3nt/E20/kDoEgZnOdelLFmAzz57xsmkPFcO9eZclvoTikjykUCNPB32kD6n9mcQ DELg== X-Gm-Message-State: AOAM530a+hFR24N7fr4uUiUQ04fUGhZGHKN/3T7KROdazDG0rIfklbqK lm1DsDNT7EDC0Vtktf0ndk8gVKFxGp/HB/mC X-Google-Smtp-Source: ABdhPJyWWe37oEqyJEJadVG8fp31IaMzdbGRH8rs/dpjbTP77F3EsdvHBUAogyehH3wl9neL0B5sNIh1QHWoxIWS X-Received: from andreyknvl3.muc.corp.google.com ([2a00:79e0:15:13:edb8:b79c:2e20:e531]) (user=andreyknvl job=sendgmr) by 2002:a05:6402:5193:: with SMTP id q19mr4617213edd.264.1612546512746; Fri, 05 Feb 2021 09:35:12 -0800 (PST) Date: Fri, 5 Feb 2021 18:34:43 +0100 In-Reply-To: Message-Id: <3ffe8d4a246ae67a8b5e91f65bf98cd7cba9d7b9.1612546384.git.andreyknvl@google.com> Mime-Version: 1.0 References: X-Mailer: git-send-email 2.30.0.365.g02bc693789-goog Subject: [PATCH v3 mm 09/13] kasan: ensure poisoning size alignment From: Andrey Konovalov To: Andrew Morton , Catalin Marinas , Vincenzo Frascino , Dmitry Vyukov , Alexander Potapenko , Marco Elver X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210205_123517_311055_CA5844F3 X-CRM114-Status: GOOD ( 18.48 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Branislav Rankov , Andrey Konovalov , Kevin Brodsky , Will Deacon , linux-kernel@vger.kernel.org, kasan-dev@googlegroups.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, Andrey Ryabinin , Peter Collingbourne , Evgenii Stepanov Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org A previous changes d99f6a10c161 ("kasan: don't round_up too much") attempted to simplify the code by adding a round_up(size) call into kasan_poison(). While this allows to have less round_up() calls around the code, this results in round_up() being called multiple times. This patch removes round_up() of size from kasan_poison() and ensures that all callers round_up() the size explicitly. This patch also adds WARN_ON() alignment checks for address and size to kasan_poison() and kasan_unpoison(). Reviewed-by: Marco Elver Signed-off-by: Andrey Konovalov --- mm/kasan/common.c | 9 ++++++--- mm/kasan/kasan.h | 33 ++++++++++++++++++++------------- mm/kasan/shadow.c | 37 ++++++++++++++++++++++--------------- 3 files changed, 48 insertions(+), 31 deletions(-) diff --git a/mm/kasan/common.c b/mm/kasan/common.c index a8a67dca5e55..7ffb1e6de2ef 100644 --- a/mm/kasan/common.c +++ b/mm/kasan/common.c @@ -261,7 +261,8 @@ void __kasan_unpoison_object_data(struct kmem_cache *cache, void *object) void __kasan_poison_object_data(struct kmem_cache *cache, void *object) { - kasan_poison(object, cache->object_size, KASAN_KMALLOC_REDZONE); + kasan_poison(object, round_up(cache->object_size, KASAN_GRANULE_SIZE), + KASAN_KMALLOC_REDZONE); } /* @@ -348,7 +349,8 @@ static bool ____kasan_slab_free(struct kmem_cache *cache, void *object, return true; } - kasan_poison(object, cache->object_size, KASAN_KMALLOC_FREE); + kasan_poison(object, round_up(cache->object_size, KASAN_GRANULE_SIZE), + KASAN_KMALLOC_FREE); if ((IS_ENABLED(CONFIG_KASAN_GENERIC) && !quarantine)) return false; @@ -490,7 +492,8 @@ static void *____kasan_kmalloc(struct kmem_cache *cache, const void *object, /* Poison the aligned part of the redzone. */ redzone_start = round_up((unsigned long)(object + size), KASAN_GRANULE_SIZE); - redzone_end = (unsigned long)object + cache->object_size; + redzone_end = round_up((unsigned long)(object + cache->object_size), + KASAN_GRANULE_SIZE); kasan_poison((void *)redzone_start, redzone_end - redzone_start, KASAN_KMALLOC_REDZONE); diff --git a/mm/kasan/kasan.h b/mm/kasan/kasan.h index d0a3516e0909..cc787ba47e1b 100644 --- a/mm/kasan/kasan.h +++ b/mm/kasan/kasan.h @@ -318,30 +318,37 @@ static inline u8 kasan_random_tag(void) { return 0; } #ifdef CONFIG_KASAN_HW_TAGS -static inline void kasan_poison(const void *address, size_t size, u8 value) +static inline void kasan_poison(const void *addr, size_t size, u8 value) { - address = kasan_reset_tag(address); + addr = kasan_reset_tag(addr); /* Skip KFENCE memory if called explicitly outside of sl*b. */ - if (is_kfence_address(address)) + if (is_kfence_address(addr)) return; - hw_set_mem_tag_range((void *)address, - round_up(size, KASAN_GRANULE_SIZE), value); + if (WARN_ON((unsigned long)addr & KASAN_GRANULE_MASK)) + return; + if (WARN_ON(size & KASAN_GRANULE_MASK)) + return; + + hw_set_mem_tag_range((void *)addr, size, value); } -static inline void kasan_unpoison(const void *address, size_t size) +static inline void kasan_unpoison(const void *addr, size_t size) { - u8 tag = get_tag(address); + u8 tag = get_tag(addr); - address = kasan_reset_tag(address); + addr = kasan_reset_tag(addr); /* Skip KFENCE memory if called explicitly outside of sl*b. */ - if (is_kfence_address(address)) + if (is_kfence_address(addr)) return; - hw_set_mem_tag_range((void *)address, - round_up(size, KASAN_GRANULE_SIZE), tag); + if (WARN_ON((unsigned long)addr & KASAN_GRANULE_MASK)) + return; + size = round_up(size, KASAN_GRANULE_SIZE); + + hw_set_mem_tag_range((void *)addr, size, tag); } static inline bool kasan_byte_accessible(const void *addr) @@ -358,7 +365,7 @@ static inline bool kasan_byte_accessible(const void *addr) /** * kasan_poison - mark the memory range as unaccessible * @addr - range start address, must be aligned to KASAN_GRANULE_SIZE - * @size - range size + * @size - range size, must be aligned to KASAN_GRANULE_SIZE * @value - value that's written to metadata for the range * * The size gets aligned to KASAN_GRANULE_SIZE before marking the range. @@ -368,7 +375,7 @@ void kasan_poison(const void *addr, size_t size, u8 value); /** * kasan_unpoison - mark the memory range as accessible * @addr - range start address, must be aligned to KASAN_GRANULE_SIZE - * @size - range size + * @size - range size, can be unaligned * * For the tag-based modes, the @size gets aligned to KASAN_GRANULE_SIZE before * marking the range. diff --git a/mm/kasan/shadow.c b/mm/kasan/shadow.c index 1ed7817e4ee6..63f43443f5d7 100644 --- a/mm/kasan/shadow.c +++ b/mm/kasan/shadow.c @@ -69,7 +69,7 @@ void *memcpy(void *dest, const void *src, size_t len) return __memcpy(dest, src, len); } -void kasan_poison(const void *address, size_t size, u8 value) +void kasan_poison(const void *addr, size_t size, u8 value) { void *shadow_start, *shadow_end; @@ -78,55 +78,62 @@ void kasan_poison(const void *address, size_t size, u8 value) * some of the callers (e.g. kasan_poison_object_data) pass tagged * addresses to this function. */ - address = kasan_reset_tag(address); + addr = kasan_reset_tag(addr); /* Skip KFENCE memory if called explicitly outside of sl*b. */ - if (is_kfence_address(address)) + if (is_kfence_address(addr)) return; - size = round_up(size, KASAN_GRANULE_SIZE); - shadow_start = kasan_mem_to_shadow(address); - shadow_end = kasan_mem_to_shadow(address + size); + if (WARN_ON((unsigned long)addr & KASAN_GRANULE_MASK)) + return; + if (WARN_ON(size & KASAN_GRANULE_MASK)) + return; + + shadow_start = kasan_mem_to_shadow(addr); + shadow_end = kasan_mem_to_shadow(addr + size); __memset(shadow_start, value, shadow_end - shadow_start); } EXPORT_SYMBOL(kasan_poison); #ifdef CONFIG_KASAN_GENERIC -void kasan_poison_last_granule(const void *address, size_t size) +void kasan_poison_last_granule(const void *addr, size_t size) { if (size & KASAN_GRANULE_MASK) { - u8 *shadow = (u8 *)kasan_mem_to_shadow(address + size); + u8 *shadow = (u8 *)kasan_mem_to_shadow(addr + size); *shadow = size & KASAN_GRANULE_MASK; } } #endif -void kasan_unpoison(const void *address, size_t size) +void kasan_unpoison(const void *addr, size_t size) { - u8 tag = get_tag(address); + u8 tag = get_tag(addr); /* * Perform shadow offset calculation based on untagged address, as * some of the callers (e.g. kasan_unpoison_object_data) pass tagged * addresses to this function. */ - address = kasan_reset_tag(address); + addr = kasan_reset_tag(addr); /* * Skip KFENCE memory if called explicitly outside of sl*b. Also note * that calls to ksize(), where size is not a multiple of machine-word * size, would otherwise poison the invalid portion of the word. */ - if (is_kfence_address(address)) + if (is_kfence_address(addr)) + return; + + if (WARN_ON((unsigned long)addr & KASAN_GRANULE_MASK)) return; - /* Unpoison round_up(size, KASAN_GRANULE_SIZE) bytes. */ - kasan_poison(address, size, tag); + /* Unpoison all granules that cover the object. */ + kasan_poison(addr, round_up(size, KASAN_GRANULE_SIZE), tag); /* Partially poison the last granule for the generic mode. */ if (IS_ENABLED(CONFIG_KASAN_GENERIC)) - kasan_poison_last_granule(address, size); + kasan_poison_last_granule(addr, size); } #ifdef CONFIG_MEMORY_HOTPLUG From patchwork Fri Feb 5 17:34:44 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrey Konovalov X-Patchwork-Id: 12070599 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.1 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 70DC7C433DB for ; Fri, 5 Feb 2021 17:38:02 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 0467364DBA for ; Fri, 5 Feb 2021 17:38:01 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 0467364DBA Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:To:From:Subject:References:Mime-Version:Message-Id: In-Reply-To:Date:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=kmo/cGgvw2l0JBUaLrbqYbmsSjUrkijH65DB5i90PUc=; b=J9zw6cJFmnIfvX6siLbAk5fcX iWPnZMv96YyI67/Um3r1DUSrlEwMdBCyRm9Onlp4l69Q69ufzlfKrlD6Sh1w3lQj1hCq5H+pBFfJ7 Xww756L95C8kCC1L4qAZF6X8LGh9wFp17q9DZ/YEcv32x+m08U78G5ELTpl4TXcMNrz0sEAWDzec+ OfPtdLGzkh4XhpYHRDx4ZnQ3lDhaWVe+hVng8lzdUKatPRDgbVXkNSb5EbCqEXduXsAD7WNzRVRG7 8nat5ybWUXmOdvsLDNGkbJlWu4F0TvqmuF2Hhgk1OquQyf6fab3oumD48DHUXl7NBS0Ab09s3OHPf Ywzf/b1DA==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1l8527-0005XN-6H; Fri, 05 Feb 2021 17:36:39 +0000 Received: from mail-qk1-x749.google.com ([2607:f8b0:4864:20::749]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1l850q-0004yJ-9Q for linux-arm-kernel@lists.infradead.org; Fri, 05 Feb 2021 17:35:32 +0000 Received: by mail-qk1-x749.google.com with SMTP id t186so6475211qke.5 for ; Fri, 05 Feb 2021 09:35:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=ctAIK4J2ixb9MMoTkFb1CmODQtYX8ZOk4P8i8+csvVc=; b=UtIaRW9v2FSvyHZrpUGwpYFPoy1Ze5VquXJ1iz6uGvMo/Yw3T+rNU/+4YbDkUlgcIq 274fCMAx2A4PaDOyqUrn25nxx5KgH4nJtCGILGIXDB7IGPccugIwm35XYXsTbH2RZ5qL Xh6vnLGD/n6sTDHX1FqMGfOjkDRmhl8iHZAEgI0GdyFS6YThdyLLB26YU/n80fKRBB4Y oZ+3gn7uO9Bkto02RB0j7H+iIyiqhKyLu22ZX506BzNEFtJaJBdwYBFL6Jtd0l158x7+ P0QTYpuULqdO53c7LHJQFMlCv4ixKoDfWa/gYB5U2mcB3XPiTMgIoVDw/iHUwXo+hj/o l62w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=ctAIK4J2ixb9MMoTkFb1CmODQtYX8ZOk4P8i8+csvVc=; b=oAXmLxCrvbmWZPEBoICYiS3ig5gWkc2CFmit9NvBxA4/tyi2gj1aCKUJLASduDRRJL PYpWuNYwlhYo4fc7Jp1QouPRIZNhp7TBSVVxORRwjpLFBBn5imtCJIS/I7MI0mqvGyGt xFPCrZk+hh3M+q/kDuCklPqRkgGOVUcL/9Lm7ZI/pRh9WinNjJukisyRLa/3StdMVGnw FyRC849DmkGqbdesMlkiJHxwS6bWC5Zx2BoG3Job7S7ZIAjCHG+HTzh/FtmClO4GKhaQ W8AdmfwUVIjwei9xkziChJFQtBEVYU0GtubXz1KD5+cgY4v0Ns39RCniZFAGQHP/olbf d7pw== X-Gm-Message-State: AOAM532D6Vw5XDiE4vrlukWA/JvZO9vNOiFMTGE4N3Fpp4jIF8CpegfN AOnpWqb140TAz/D2a6FMPHapAyKb+e/rgiNt X-Google-Smtp-Source: ABdhPJwuIuL8/TNu5L48K+zuPB2JexLrFAreY/Ud1v+kcNz0kgBSvMSCk88UCBtdnUXI55UOB3wRJDjTdqlNyCiC X-Received: from andreyknvl3.muc.corp.google.com ([2a00:79e0:15:13:edb8:b79c:2e20:e531]) (user=andreyknvl job=sendgmr) by 2002:a0c:a819:: with SMTP id w25mr5283889qva.6.1612546515078; Fri, 05 Feb 2021 09:35:15 -0800 (PST) Date: Fri, 5 Feb 2021 18:34:44 +0100 In-Reply-To: Message-Id: Mime-Version: 1.0 References: X-Mailer: git-send-email 2.30.0.365.g02bc693789-goog Subject: [PATCH v3 mm 10/13] arm64: kasan: simplify and inline MTE functions From: Andrey Konovalov To: Andrew Morton , Catalin Marinas , Vincenzo Frascino , Dmitry Vyukov , Alexander Potapenko , Marco Elver X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210205_123521_965550_94E7F95E X-CRM114-Status: GOOD ( 20.36 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Branislav Rankov , Andrey Konovalov , Kevin Brodsky , Will Deacon , linux-kernel@vger.kernel.org, kasan-dev@googlegroups.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, Andrey Ryabinin , Peter Collingbourne , Evgenii Stepanov Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org This change provides a simpler implementation of mte_get_mem_tag(), mte_get_random_tag(), and mte_set_mem_tag_range(). Simplifications include removing system_supports_mte() checks as these functions are onlye called from KASAN runtime that had already checked system_supports_mte(). Besides that, size and address alignment checks are removed from mte_set_mem_tag_range(), as KASAN now does those. This change also moves these functions into the asm/mte-kasan.h header and implements mte_set_mem_tag_range() via inline assembly to avoid unnecessary functions calls. Reviewed-by: Catalin Marinas Co-developed-by: Vincenzo Frascino Signed-off-by: Vincenzo Frascino Signed-off-by: Andrey Konovalov --- arch/arm64/include/asm/cache.h | 1 - arch/arm64/include/asm/kasan.h | 1 + arch/arm64/include/asm/mte-def.h | 2 + arch/arm64/include/asm/mte-kasan.h | 65 ++++++++++++++++++++++++++---- arch/arm64/include/asm/mte.h | 2 - arch/arm64/kernel/mte.c | 46 --------------------- arch/arm64/lib/mte.S | 16 -------- 7 files changed, 60 insertions(+), 73 deletions(-) diff --git a/arch/arm64/include/asm/cache.h b/arch/arm64/include/asm/cache.h index 77cbbe3625f2..a074459f8f2f 100644 --- a/arch/arm64/include/asm/cache.h +++ b/arch/arm64/include/asm/cache.h @@ -6,7 +6,6 @@ #define __ASM_CACHE_H #include -#include #define CTR_L1IP_SHIFT 14 #define CTR_L1IP_MASK 3 diff --git a/arch/arm64/include/asm/kasan.h b/arch/arm64/include/asm/kasan.h index 0aaf9044cd6a..12d5f47f7dbe 100644 --- a/arch/arm64/include/asm/kasan.h +++ b/arch/arm64/include/asm/kasan.h @@ -6,6 +6,7 @@ #include #include +#include #include #define arch_kasan_set_tag(addr, tag) __tag_set(addr, tag) diff --git a/arch/arm64/include/asm/mte-def.h b/arch/arm64/include/asm/mte-def.h index 2d73a1612f09..cf241b0f0a42 100644 --- a/arch/arm64/include/asm/mte-def.h +++ b/arch/arm64/include/asm/mte-def.h @@ -11,4 +11,6 @@ #define MTE_TAG_SIZE 4 #define MTE_TAG_MASK GENMASK((MTE_TAG_SHIFT + (MTE_TAG_SIZE - 1)), MTE_TAG_SHIFT) +#define __MTE_PREAMBLE ARM64_ASM_PREAMBLE ".arch_extension memtag\n" + #endif /* __ASM_MTE_DEF_H */ diff --git a/arch/arm64/include/asm/mte-kasan.h b/arch/arm64/include/asm/mte-kasan.h index 3748d5bb88c0..3d58489228c0 100644 --- a/arch/arm64/include/asm/mte-kasan.h +++ b/arch/arm64/include/asm/mte-kasan.h @@ -11,11 +11,14 @@ #include +#ifdef CONFIG_ARM64_MTE + /* - * The functions below are meant to be used only for the - * KASAN_HW_TAGS interface defined in asm/memory.h. + * These functions are meant to be only used from KASAN runtime through + * the arch_*() interface defined in asm/memory.h. + * These functions don't include system_supports_mte() checks, + * as KASAN only calls them when MTE is supported and enabled. */ -#ifdef CONFIG_ARM64_MTE static inline u8 mte_get_ptr_tag(void *ptr) { @@ -25,9 +28,54 @@ static inline u8 mte_get_ptr_tag(void *ptr) return tag; } -u8 mte_get_mem_tag(void *addr); -u8 mte_get_random_tag(void); -void *mte_set_mem_tag_range(void *addr, size_t size, u8 tag); +/* Get allocation tag for the address. */ +static inline u8 mte_get_mem_tag(void *addr) +{ + asm(__MTE_PREAMBLE "ldg %0, [%0]" + : "+r" (addr)); + + return mte_get_ptr_tag(addr); +} + +/* Generate a random tag. */ +static inline u8 mte_get_random_tag(void) +{ + void *addr; + + asm(__MTE_PREAMBLE "irg %0, %0" + : "+r" (addr)); + + return mte_get_ptr_tag(addr); +} + +/* + * Assign allocation tags for a region of memory based on the pointer tag. + * Note: The address must be non-NULL and MTE_GRANULE_SIZE aligned and + * size must be non-zero and MTE_GRANULE_SIZE aligned. + */ +static inline void mte_set_mem_tag_range(void *addr, size_t size, u8 tag) +{ + u64 curr, end; + + if (!size) + return; + + curr = (u64)__tag_set(addr, tag); + end = curr + size; + + do { + /* + * 'asm volatile' is required to prevent the compiler to move + * the statement outside of the loop. + */ + asm volatile(__MTE_PREAMBLE "stg %0, [%0]" + : + : "r" (curr) + : "memory"); + + curr += MTE_GRANULE_SIZE; + } while (curr != end); +} void mte_enable_kernel(void); void mte_init_tags(u64 max_tag); @@ -46,13 +94,14 @@ static inline u8 mte_get_mem_tag(void *addr) { return 0xFF; } + static inline u8 mte_get_random_tag(void) { return 0xFF; } -static inline void *mte_set_mem_tag_range(void *addr, size_t size, u8 tag) + +static inline void mte_set_mem_tag_range(void *addr, size_t size, u8 tag) { - return addr; } static inline void mte_enable_kernel(void) diff --git a/arch/arm64/include/asm/mte.h b/arch/arm64/include/asm/mte.h index d02aff9f493d..9b557a457f24 100644 --- a/arch/arm64/include/asm/mte.h +++ b/arch/arm64/include/asm/mte.h @@ -8,8 +8,6 @@ #include #include -#define __MTE_PREAMBLE ARM64_ASM_PREAMBLE ".arch_extension memtag\n" - #ifndef __ASSEMBLY__ #include diff --git a/arch/arm64/kernel/mte.c b/arch/arm64/kernel/mte.c index c63b3d7a3cd9..203108d51d40 100644 --- a/arch/arm64/kernel/mte.c +++ b/arch/arm64/kernel/mte.c @@ -19,7 +19,6 @@ #include #include #include -#include #include #include @@ -88,51 +87,6 @@ int memcmp_pages(struct page *page1, struct page *page2) return ret; } -u8 mte_get_mem_tag(void *addr) -{ - if (!system_supports_mte()) - return 0xFF; - - asm(__MTE_PREAMBLE "ldg %0, [%0]" - : "+r" (addr)); - - return mte_get_ptr_tag(addr); -} - -u8 mte_get_random_tag(void) -{ - void *addr; - - if (!system_supports_mte()) - return 0xFF; - - asm(__MTE_PREAMBLE "irg %0, %0" - : "+r" (addr)); - - return mte_get_ptr_tag(addr); -} - -void *mte_set_mem_tag_range(void *addr, size_t size, u8 tag) -{ - void *ptr = addr; - - if ((!system_supports_mte()) || (size == 0)) - return addr; - - /* Make sure that size is MTE granule aligned. */ - WARN_ON(size & (MTE_GRANULE_SIZE - 1)); - - /* Make sure that the address is MTE granule aligned. */ - WARN_ON((u64)addr & (MTE_GRANULE_SIZE - 1)); - - tag = 0xF0 | tag; - ptr = (void *)__tag_set(ptr, tag); - - mte_assign_mem_tag_range(ptr, size); - - return ptr; -} - void mte_init_tags(u64 max_tag) { static bool gcr_kernel_excl_initialized; diff --git a/arch/arm64/lib/mte.S b/arch/arm64/lib/mte.S index 9e1a12e10053..351537c12f36 100644 --- a/arch/arm64/lib/mte.S +++ b/arch/arm64/lib/mte.S @@ -149,19 +149,3 @@ SYM_FUNC_START(mte_restore_page_tags) ret SYM_FUNC_END(mte_restore_page_tags) - -/* - * Assign allocation tags for a region of memory based on the pointer tag - * x0 - source pointer - * x1 - size - * - * Note: The address must be non-NULL and MTE_GRANULE_SIZE aligned and - * size must be non-zero and MTE_GRANULE_SIZE aligned. - */ -SYM_FUNC_START(mte_assign_mem_tag_range) -1: stg x0, [x0] - add x0, x0, #MTE_GRANULE_SIZE - subs x1, x1, #MTE_GRANULE_SIZE - b.gt 1b - ret -SYM_FUNC_END(mte_assign_mem_tag_range) From patchwork Fri Feb 5 17:34:45 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrey Konovalov X-Patchwork-Id: 12070593 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.1 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B2CEEC433DB for ; Fri, 5 Feb 2021 17:37:34 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 3FA7E64DC4 for ; Fri, 5 Feb 2021 17:37:34 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 3FA7E64DC4 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:To:From:Subject:References:Mime-Version:Message-Id: In-Reply-To:Date:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=12T+l7VgFyOXn4nh8YwxsMN6nWgZ9hn1KmBNeBrNQC0=; b=xiKkvqQSaakHSkBhnlu9jrZi4 iLOuL3SlRLv7YIo+ceYOv7HviywUNFTQOiPjfqdSNhg3v/h7RZlv93gyXnRTj5QvX44USgKyGylO7 cH6prmzlLkcF5KdVoB0YEXU+lfvZ1sct4kkw8n4O5asNbfD7W4ITuJWYHO6yUYOYsNRL93ZKjxgGh 16nQ39VQtcs92giYO48KI4Fn22o3bIH5B+G+1UAVkg2Hh5jZqgSdqNTDsX46USuMGvJfa/E5amcdH WZ5hh/9vljoZmVCoVMijPhwHeFCveP596ex8qXa6DsUOtr0rr2ayw3SYQ6JaFfYwOZOi8dVH3Kq1o IFEfBjo3w==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1l851g-0005L9-Am; Fri, 05 Feb 2021 17:36:12 +0000 Received: from mail-wr1-x449.google.com ([2a00:1450:4864:20::449]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1l850r-0004z6-MX for linux-arm-kernel@lists.infradead.org; Fri, 05 Feb 2021 17:35:30 +0000 Received: by mail-wr1-x449.google.com with SMTP id o17so5757272wrv.4 for ; Fri, 05 Feb 2021 09:35:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=r9X5DpdR9qKQzOGqj5YPMHvNzUxwchyNv1XRoONrpMQ=; b=rhZcOisXxajZRSH31J6t5x8y7U35XbWt39APfdQTAyFPG0HF5mjWJSCOgw16eIMOMa FxjsONeMdcvEksZXk6bC9GsFOfyYtiJJVVxykiWoVQthkFCZ+7JIWakX1YKqLh2Hfnsw UHhGojbmi5DGaKkXKVb9OuWCEYr4RB6K9hJP+CP2j/YOxjaS7vNIFnSlhUYQR2/KU0oU ooZpFMWOGqErYIXA8pyt+sZSi7z6vjp1Cx9mo0rM+ADEMLxumJNs73rI/ZV+psV71+ij 0yCJp9ep/i82xgHJH7EhvCF5jIIaWuoWZ9xoBRvQpbg8jUeOxTCIZpjL2b4jXvijo6zj rsPQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=r9X5DpdR9qKQzOGqj5YPMHvNzUxwchyNv1XRoONrpMQ=; b=qdIlOQ+mkvIdHkDmqLF4i3EhfLej9dLGfStK9YZSKngWESQYIybVEhhhSLvuT9wVqn TQSC3d3xoMsieAZoV12Kp1wP9OlX70fa6ru3yRJuLJDV+ngeZZaQwzPwXrJGuU6OVTAY +nwVnMOr3osxdewV4RCZ4qB3ZD4J/uC29u4Qgf62wuU/4pnOUzDCmqef96xUoCK0ZgWe c7oHIYtCE2yIn1EWH2i258lIrS+pGgA976LEAsw9tQNDdz9vRpE6legjdx2hrrNh2QTP o1YdcbvtAOGSRhEDxqjcqR0wMbaebRDcAQpgxJjhzwBhI60a+LPK9HrRiCFi0AUt0Hou KQAA== X-Gm-Message-State: AOAM530PoSsXnF6XqPi0mxXAxrz+WpA3DGnFOM/GwkB1HMcjne3VIhWP Msq67QL219ADor/SvyJcVG10xeaV89XiLKPB X-Google-Smtp-Source: ABdhPJwTAziAbVGPOhm/GCfHuY7Nxua8uArDaBNKr1jq6WV9vgAgwxgQ1kegmyvTxo5ASrnPr8xSaEmusHPD5Qp2 X-Received: from andreyknvl3.muc.corp.google.com ([2a00:79e0:15:13:edb8:b79c:2e20:e531]) (user=andreyknvl job=sendgmr) by 2002:adf:f149:: with SMTP id y9mr6216013wro.144.1612546517387; Fri, 05 Feb 2021 09:35:17 -0800 (PST) Date: Fri, 5 Feb 2021 18:34:45 +0100 In-Reply-To: Message-Id: <2c94a2af0657f2b95b9337232339ff5ffa643ab5.1612546384.git.andreyknvl@google.com> Mime-Version: 1.0 References: X-Mailer: git-send-email 2.30.0.365.g02bc693789-goog Subject: [PATCH v3 mm 11/13] kasan: inline HW_TAGS helper functions From: Andrey Konovalov To: Andrew Morton , Catalin Marinas , Vincenzo Frascino , Dmitry Vyukov , Alexander Potapenko , Marco Elver X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210205_123522_050139_8C10CE5E X-CRM114-Status: GOOD ( 11.12 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Branislav Rankov , Andrey Konovalov , Kevin Brodsky , Will Deacon , linux-kernel@vger.kernel.org, kasan-dev@googlegroups.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, Andrey Ryabinin , Peter Collingbourne , Evgenii Stepanov Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Mark all static functions in common.c and kasan.h that are used for hardware tag-based KASAN as inline to avoid unnecessary function calls. Signed-off-by: Andrey Konovalov Reviewed-by: Marco Elver --- mm/kasan/common.c | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/mm/kasan/common.c b/mm/kasan/common.c index 7ffb1e6de2ef..7b53291dafa1 100644 --- a/mm/kasan/common.c +++ b/mm/kasan/common.c @@ -279,7 +279,8 @@ void __kasan_poison_object_data(struct kmem_cache *cache, void *object) * based on objects indexes, so that objects that are next to each other * get different tags. */ -static u8 assign_tag(struct kmem_cache *cache, const void *object, bool init) +static inline u8 assign_tag(struct kmem_cache *cache, + const void *object, bool init) { if (IS_ENABLED(CONFIG_KASAN_GENERIC)) return 0xff; @@ -321,8 +322,8 @@ void * __must_check __kasan_init_slab_obj(struct kmem_cache *cache, return (void *)object; } -static bool ____kasan_slab_free(struct kmem_cache *cache, void *object, - unsigned long ip, bool quarantine) +static inline bool ____kasan_slab_free(struct kmem_cache *cache, + void *object, unsigned long ip, bool quarantine) { u8 tag; void *tagged_object; @@ -366,7 +367,7 @@ bool __kasan_slab_free(struct kmem_cache *cache, void *object, unsigned long ip) return ____kasan_slab_free(cache, object, ip, true); } -static bool ____kasan_kfree_large(void *ptr, unsigned long ip) +static inline bool ____kasan_kfree_large(void *ptr, unsigned long ip) { if (ptr != page_address(virt_to_head_page(ptr))) { kasan_report_invalid_free(ptr, ip); @@ -461,8 +462,8 @@ void * __must_check __kasan_slab_alloc(struct kmem_cache *cache, return tagged_object; } -static void *____kasan_kmalloc(struct kmem_cache *cache, const void *object, - size_t size, gfp_t flags) +static inline void *____kasan_kmalloc(struct kmem_cache *cache, + const void *object, size_t size, gfp_t flags) { unsigned long redzone_start; unsigned long redzone_end; From patchwork Fri Feb 5 17:34:46 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrey Konovalov X-Patchwork-Id: 12070597 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.1 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1361AC433E6 for ; Fri, 5 Feb 2021 17:37:46 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id D013960C3F for ; Fri, 5 Feb 2021 17:37:45 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org D013960C3F Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:To:From:Subject:References:Mime-Version:Message-Id: In-Reply-To:Date:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=OJ3SuST+VEvA2KiPwIGLhRogIT5DZNw4s3JOrTw0buc=; b=pyQkwE0ttwsmUc/eT/WADx56b JqQOKbCOB+i1jlrd0lQumPxPD3NHCvWkBqyOq5Y5xOjJ4u+d+t9ZuGoKgQRhgE6hsRqHk8kLNN94g 95DAso6xTEVHkkE9bJE1Fo/35hLYqFzvwb84oSDHsLHVgox4DCH0QZkCGKh8vJ0n+LCaWpE0pqqgT Gmgfom3x/vGZRAhwkIqxbgFVXOCZqgktYEk90xuTlsBA3S/ZpfwYh7JXfeCmvgjfF1Bqt8stAC6ht ddd7++u4ZzSaZq/ihj5F6DONwQ8AZd2MKIdEAQZmWMr6Arx1sSQiDkfsWAza3q8peQAHsd5KXWaVu v2rtT6BZg==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1l851v-0005Sm-Ib; Fri, 05 Feb 2021 17:36:27 +0000 Received: from mail-qt1-x84a.google.com ([2607:f8b0:4864:20::84a]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1l850t-0004zQ-Kg for linux-arm-kernel@lists.infradead.org; Fri, 05 Feb 2021 17:35:32 +0000 Received: by mail-qt1-x84a.google.com with SMTP id o7so5775328qtw.7 for ; Fri, 05 Feb 2021 09:35:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=V3NcBz/sBuF9R5rbEs4mkivGfi7JQffIAtJVZQe6PVQ=; b=AyFbGRtZMQocl7Q5NZT214tJhO130JH2kzAWqSjD8+Setp/oN6e7y7y8sPvxLQ8TH6 UfeGhuUInm0NPCAigeTNrpGzDs0p+CeVjTEmhqouTHFWmbb5IGwrv+hy3nMqEPJ5yukw flPCVMav/pLrs8OsAbd8op8i84uJ0cBQXHIGczYU7OYXZ71NdHwo97Qy9Ohdp+Cmvc9v 5BhyH/fScylRm+CG12IxCgd5lj5u7a/9pl8SjQluL9wg2B8jUpNlP9gDHyB0ZVrHMbwx 6xr1/x0bN5rWepIoZ6NFyCEI08m5yhNOXHjLn12fmIibIaQ1QXL1I94o8s+BOLGB96yU 1XHQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=V3NcBz/sBuF9R5rbEs4mkivGfi7JQffIAtJVZQe6PVQ=; b=R2O8zwYBR4qZZZDrOtQQGTvl0ZwaYCMTj6ZD3Za2m7+ksemhkgJuxUUEtRxshnwe7/ oPMmSPr1M+m0RHXHPi7l0/Xshoo5qcS6wnIyBVChxvahzMLTwjzcObVYIaBCJmekBLNR fVNM2K1Bb4Uzhw0pi+DgQH9ZCN4XLfst00nffgaXVgmc5kK29BxN8Zj5lIWiS60uu71V NR7CyWU/7mNRMvw8F/+PJEJSiD/J5eFhwmIadQMkQLvbvdOKUfLIq+moWpWeiU4q8wwI yQnWcuv6ne+Y+rdejIQbOLO1jxpDrNxlUU15Mc5NemLLbzTyBu1deI1EUVmSfEhUoLaS wFDw== X-Gm-Message-State: AOAM533jsJnDYyQJRmaC0sy8PtoRzz2pCu1XxrFPvk8y/ZRc0HZSohJu 13uZoqjmXmtsPWT5AVxSLnUqjf/3K6CvvPnw X-Google-Smtp-Source: ABdhPJweLbaw7VFlVSS/J9c4mZHXhCjW3eIz5PFMh/DXiJO3ThcuygtX+EQLe2kTQSSSDZUfps4nI4nhEccmHL6m X-Received: from andreyknvl3.muc.corp.google.com ([2a00:79e0:15:13:edb8:b79c:2e20:e531]) (user=andreyknvl job=sendgmr) by 2002:a0c:eda6:: with SMTP id h6mr5378538qvr.19.1612546519544; Fri, 05 Feb 2021 09:35:19 -0800 (PST) Date: Fri, 5 Feb 2021 18:34:46 +0100 In-Reply-To: Message-Id: <14da91876a25a6ae5b2096f7c06ebbc6deb3cf6b.1612546384.git.andreyknvl@google.com> Mime-Version: 1.0 References: X-Mailer: git-send-email 2.30.0.365.g02bc693789-goog Subject: [PATCH v3 mm 12/13] arm64: kasan: export MTE symbols for KASAN tests From: Andrey Konovalov To: Andrew Morton , Catalin Marinas , Vincenzo Frascino , Dmitry Vyukov , Alexander Potapenko , Marco Elver X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210205_123523_787728_602D259D X-CRM114-Status: GOOD ( 10.95 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Branislav Rankov , Andrey Konovalov , Kevin Brodsky , Will Deacon , linux-kernel@vger.kernel.org, kasan-dev@googlegroups.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, Andrey Ryabinin , Peter Collingbourne , Evgenii Stepanov Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Export mte_enable_kernel() and mte_set_report_once() to fix: ERROR: modpost: "mte_enable_kernel" [lib/test_kasan.ko] undefined! ERROR: modpost: "mte_set_report_once" [lib/test_kasan.ko] undefined! Reviewed-by: Catalin Marinas Signed-off-by: Andrey Konovalov --- arch/arm64/kernel/mte.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/arch/arm64/kernel/mte.c b/arch/arm64/kernel/mte.c index 203108d51d40..a66c2806fc4d 100644 --- a/arch/arm64/kernel/mte.c +++ b/arch/arm64/kernel/mte.c @@ -113,11 +113,13 @@ void mte_enable_kernel(void) sysreg_clear_set(sctlr_el1, SCTLR_ELx_TCF_MASK, SCTLR_ELx_TCF_SYNC); isb(); } +EXPORT_SYMBOL_GPL(mte_enable_kernel); void mte_set_report_once(bool state) { WRITE_ONCE(report_fault_once, state); } +EXPORT_SYMBOL_GPL(mte_set_report_once); bool mte_report_once(void) { From patchwork Fri Feb 5 17:34:47 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrey Konovalov X-Patchwork-Id: 12070601 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.1 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2D0CBC433E6 for ; Fri, 5 Feb 2021 17:38:05 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id CE9F464DBA for ; Fri, 5 Feb 2021 17:38:04 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org CE9F464DBA Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:To:From:Subject:References:Mime-Version:Message-Id: In-Reply-To:Date:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=VLsW3cAzEnJ6T3O2IFqblufKT1/tR9JbTHlNbF+O9Mc=; b=rWE0qPiNS3Zif1LIKVmINH8Ou zY+adRGxOvnD/e5oBCun7L1pfU9hFsJNqaocEsfhk4oc4y9kSwD6eAEe/J9Ue1Deg/pqgftKeEyZT I9VIbrzr5nA/dXTBf2yEElxWMTowlV8wh919BBxYOR6pz21Msg89vYUMkC0L1ESieS8W7GlSCuz9e VDL8A3zQlQBm9KFkzFs7jYpsNDXL5aRkvFQQwPK/jPiRpk683hlqMSFHajIRyEgJ+XLSJ5iKOiStQ IGVAoHBlxiLWLLl1w2K3uHsgNMHyOF+yBx1s/bQm7dFO/La/+4KoLxnJqMy7UwVA/KQd5h5ZWPVor XJkUuMwYg==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1l852C-0005cw-PG; Fri, 05 Feb 2021 17:36:44 +0000 Received: from mail-qv1-xf49.google.com ([2607:f8b0:4864:20::f49]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1l850v-000510-IW for linux-arm-kernel@lists.infradead.org; Fri, 05 Feb 2021 17:35:36 +0000 Received: by mail-qv1-xf49.google.com with SMTP id q37so5525159qvf.14 for ; Fri, 05 Feb 2021 09:35:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=qTEJ0ZFBO6HUy56kVKTkSwNvhkuevt789FDycYnAlaw=; b=G68KUqp3HFLcXYHG+bANujiKTjP1H7aOPDJCFa1RkVjgdbj5cMzKLUYhQ88IOigD2m g5wDAziPJ9/f2dbpwetUHkPJSL+aI/MWcTpGdaiivZgXRAc1C7nVA8lhLw49A9oP0xKP YkRx4Li9ZnC0Fp2IsX/EdfVi5nLsxTLigRBe9oKmV5ex/G9k44wbWqIs5s5fsPBc6OK1 UhA4HiSIlJg20l5gKCGg1nOsKB7YDjIwiGT0etN2p6CDP2nZon8XJQMGeGLgUOz+VcNi bc8djrtRbaTCpDbTXiogPshowtlILUAlRK7BVS1XEzHo35Yv9MrTtxK4e+3RPZe4PG7Z jBtA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=qTEJ0ZFBO6HUy56kVKTkSwNvhkuevt789FDycYnAlaw=; b=meHRAhoOCRLPipGlGE5Xc0zW6A0b3tqyIriR+blNpClSozX2JxGcYOxlwabf4nWIfx B3wnIsU6Pgp2qMlaFOOOJ3hO/8TT1fgp69PMfSX0SUjhP3+/OWte3C43mqkZeGJl1kT/ +Fw3kMelHZQm9BRWvUyBYfDIQnFWqn7jTpyPfYwD/rnqWkPU9LLOt+0lHLLYAOYXIzot jZpNffOVBf2A40KFgEQ2TI4VT20Jo1VIZn0ugSew9/mrm3l/xxU3aXcpz7QTUrCyCIOY HpKdGXgHJ6GL3728xFSgKocxlcGpxG5aea1QTvcN8t9ha29sL+WrW34WTxsC/cvLjxHh 0ZZw== X-Gm-Message-State: AOAM532Y9tU88VY81HPowcznd0Q6mD30gITbGekPSm8yQIiiAXdCpWzu 7jlE8uqX/nbMZicNmmFXJDCmeC9rGgWr1Y66 X-Google-Smtp-Source: ABdhPJyrCz7kXUmcNbcxCdKw5jsfPhQUKg97pR36HY1rcIJSHaA7CzMpo0ImY3xsQ8HCnIvFo9TLiGVZ7iEPqvw/ X-Received: from andreyknvl3.muc.corp.google.com ([2a00:79e0:15:13:edb8:b79c:2e20:e531]) (user=andreyknvl job=sendgmr) by 2002:ad4:4f41:: with SMTP id eu1mr5346273qvb.34.1612546521952; Fri, 05 Feb 2021 09:35:21 -0800 (PST) Date: Fri, 5 Feb 2021 18:34:47 +0100 In-Reply-To: Message-Id: <00383ba88a47c3f8342d12263c24bdf95527b07d.1612546384.git.andreyknvl@google.com> Mime-Version: 1.0 References: X-Mailer: git-send-email 2.30.0.365.g02bc693789-goog Subject: [PATCH v3 mm 13/13] kasan: clarify that only first bug is reported in HW_TAGS From: Andrey Konovalov To: Andrew Morton , Catalin Marinas , Vincenzo Frascino , Dmitry Vyukov , Alexander Potapenko , Marco Elver X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210205_123525_730148_05ABC8CB X-CRM114-Status: GOOD ( 13.14 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Branislav Rankov , Andrey Konovalov , Kevin Brodsky , Will Deacon , linux-kernel@vger.kernel.org, kasan-dev@googlegroups.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, Andrey Ryabinin , Peter Collingbourne , Evgenii Stepanov Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Hwardware tag-based KASAN only reports the first found bug. After that MTE tag checking gets disabled. Clarify this in comments and documentation. Signed-off-by: Andrey Konovalov Reviewed-by: Marco Elver --- Documentation/dev-tools/kasan.rst | 8 ++++++-- mm/kasan/hw_tags.c | 2 +- 2 files changed, 7 insertions(+), 3 deletions(-) diff --git a/Documentation/dev-tools/kasan.rst b/Documentation/dev-tools/kasan.rst index e022b7506e37..1faabbe23e09 100644 --- a/Documentation/dev-tools/kasan.rst +++ b/Documentation/dev-tools/kasan.rst @@ -155,7 +155,7 @@ Boot parameters ~~~~~~~~~~~~~~~ Hardware tag-based KASAN mode (see the section about various modes below) is -intended for use in production as a security mitigation. Therefore it supports +intended for use in production as a security mitigation. Therefore, it supports boot parameters that allow to disable KASAN competely or otherwise control particular KASAN features. @@ -166,7 +166,8 @@ particular KASAN features. ``off``). - ``kasan.fault=report`` or ``=panic`` controls whether to only print a KASAN - report or also panic the kernel (default: ``report``). + report or also panic the kernel (default: ``report``). Note, that tag + checking gets disabled after the first reported bug. For developers ~~~~~~~~~~~~~~ @@ -296,6 +297,9 @@ Note, that enabling CONFIG_KASAN_HW_TAGS always results in in-kernel TBI being enabled. Even when kasan.mode=off is provided, or when the hardware doesn't support MTE (but supports TBI). +Hardware tag-based KASAN only reports the first found bug. After that MTE tag +checking gets disabled. + What memory accesses are sanitised by KASAN? -------------------------------------------- diff --git a/mm/kasan/hw_tags.c b/mm/kasan/hw_tags.c index e529428e7a11..6c9285c906b8 100644 --- a/mm/kasan/hw_tags.c +++ b/mm/kasan/hw_tags.c @@ -48,7 +48,7 @@ EXPORT_SYMBOL(kasan_flag_enabled); /* Whether to collect alloc/free stack traces. */ DEFINE_STATIC_KEY_FALSE(kasan_flag_stacktrace); -/* Whether panic or disable tag checking on fault. */ +/* Whether to panic or print a report and disable tag checking on fault. */ bool kasan_flag_panic __ro_after_init; /* kasan=off/on */