From patchwork Fri Feb 5 15:39:02 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrey Konovalov X-Patchwork-Id: 12070345 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.1 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 47B5AC433E0 for ; Fri, 5 Feb 2021 15:40:48 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id D9FCB650F2 for ; Fri, 5 Feb 2021 15:40:47 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org D9FCB650F2 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:To:From:Subject:References:Mime-Version:Message-Id: In-Reply-To:Date:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=cUObgtTqFnYGqUrHHlt2GL2MneOPd7kDAeIrxW9rnxI=; b=m7eCSPYcH2mCtDZSE/bRx2Dot hM+eqlxX9Ff6ZA32+xgzjPHv1Fsmc7ERmUlC/k0g4KVSgxIZaqUfayWlMyY/BrFIemC2404AesRCj vbeon5BAiyNs6jSDYT1Mp8jQfM3Y1ZQmkT03T5zE0eTZbCEnaHU53wOWrvE8FowdcPwHtaTyb4UIw XOJqh36dD7Qw0oFQGglDPEyqfJGvPMRlrjFCMjjV7iq4vqVwjl04ZiGnaTDR7lo3zcLgyJMe5bhOn xcVDJv+u6a0xe0qzWAYdyAnzPnzEzQQDEpiNIymoUxPQ8uYfRVl5SYTY11e11WY59biBfY6gjBs/S AmP2b4bcA==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1l83Ci-0007lS-Uo; Fri, 05 Feb 2021 15:39:28 +0000 Received: from mail-wr1-x44a.google.com ([2a00:1450:4864:20::44a]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1l83Cd-0007ip-Th for linux-arm-kernel@lists.infradead.org; Fri, 05 Feb 2021 15:39:25 +0000 Received: by mail-wr1-x44a.google.com with SMTP id c1so5657160wrx.2 for ; Fri, 05 Feb 2021 07:39:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=q8NBwqooOPRUby5+8HD62t04N8TexwYKtcMMLD/hsYw=; b=N8dGmHEbsbmDY3Xr1kBJ5qp2vlfezU7Wn2sOBmtqjLc7astlSDlrYrHIYgx/Hxxz+o AGiHjTmgXxeUuUr8vHVQMMFDhhu9fxy8KVJV8jT5gioitqi1phPYh2pBXdsh16SrZnE1 cZBmsz/S9moA1VbQKtAdFvKo278quUpv15w9RlEFBfL1tDDQQDKOQaKMPZe8dhtVpPmt zTe5A3rfInQaeqklm3ptrM00EOVdpUdkqBJe8YHAPLDwYJzUgKF0xceAGm9Vyd6vl0US AZalEr0wikduX6NeO5dNhgPtzDcI0LBhHy6aC93aif4F3M8ZDXlwLtQbsM7fzAsGN4WH Rakg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=q8NBwqooOPRUby5+8HD62t04N8TexwYKtcMMLD/hsYw=; b=NNiEfUjElmqxDQAAOUKtHRBOl7KDGhNWeWKLZ7ol7Jy/yYM+YBdn0LEEGDP9KNnkt0 zYF0EOul5mgaavvnsfpLna/gG73UodKcWUBc36hikBF7Sw6PLOBzGw06S4N+pebdmh2S s3ePktXux2dOv+lsj8MwH0/4U9B7ndtnH66N7OfTUkWCRqz+St5escZYgy7vPPvd0IAL m3SpUVBBkgyvic+nnRUzRxzVN9mv6v8AbLw1yz9tSYi0WXOWWKAm4+AuU7KUZEGfqW6X lJRJ+R5BCF9ayZnPZnWlauYV8sPGqblfd+/jQFqpDLMHSkVWwn3HQGMs4zpQkvLJ9GQE xh+Q== X-Gm-Message-State: AOAM532kPB3ZN4dlfQD+4CWEAIa4tzMGl+9OiQDxMHfn78TR1uFHTZR4 4mlF+7mI+3mqJWakB+km3q2As932MoAPAjDq X-Google-Smtp-Source: ABdhPJzjR0BPDS3Eaf6xpBfyulEPTn9B6ZW4BoDBH0heBGtCIrPnc5WHtwNTMu+azMOm3gvhxmLgdQ9FDPT5U+V6 X-Received: from andreyknvl3.muc.corp.google.com ([2a00:79e0:15:13:edb8:b79c:2e20:e531]) (user=andreyknvl job=sendgmr) by 2002:a05:600c:48a8:: with SMTP id j40mr143776wmp.57.1612539560487; Fri, 05 Feb 2021 07:39:20 -0800 (PST) Date: Fri, 5 Feb 2021 16:39:02 +0100 In-Reply-To: Message-Id: <4aad006aea0dcf7cd24e0cac13026dc8b93a0961.1612538932.git.andreyknvl@google.com> Mime-Version: 1.0 References: X-Mailer: git-send-email 2.30.0.365.g02bc693789-goog Subject: [PATCH v2 01/12] kasan, mm: don't save alloc stacks twice From: Andrey Konovalov To: Andrew Morton , Catalin Marinas , Vincenzo Frascino , Dmitry Vyukov , Alexander Potapenko , Marco Elver X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210205_103923_987124_C81A3B7E X-CRM114-Status: GOOD ( 16.23 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Branislav Rankov , Andrey Konovalov , Kevin Brodsky , Will Deacon , linux-kernel@vger.kernel.org, kasan-dev@googlegroups.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, Andrey Ryabinin , Peter Collingbourne , Evgenii Stepanov Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Currently KASAN saves allocation stacks in both kasan_slab_alloc() and kasan_kmalloc() annotations. This patch changes KASAN to save allocation stacks for slab objects from kmalloc caches in kasan_kmalloc() only, and stacks for other slab objects in kasan_slab_alloc() only. This change requires ____kasan_kmalloc() knowing whether the object belongs to a kmalloc cache. This is implemented by adding a flag field to the kasan_info structure. That flag is only set for kmalloc caches via a new kasan_cache_create_kmalloc() annotation. Reviewed-by: Marco Elver Signed-off-by: Andrey Konovalov --- include/linux/kasan.h | 9 +++++++++ mm/kasan/common.c | 18 ++++++++++++++---- mm/slab_common.c | 1 + 3 files changed, 24 insertions(+), 4 deletions(-) diff --git a/include/linux/kasan.h b/include/linux/kasan.h index 6d8f3227c264..2d5de4092185 100644 --- a/include/linux/kasan.h +++ b/include/linux/kasan.h @@ -83,6 +83,7 @@ static inline void kasan_disable_current(void) {} struct kasan_cache { int alloc_meta_offset; int free_meta_offset; + bool is_kmalloc; }; #ifdef CONFIG_KASAN_HW_TAGS @@ -143,6 +144,13 @@ static __always_inline void kasan_cache_create(struct kmem_cache *cache, __kasan_cache_create(cache, size, flags); } +void __kasan_cache_create_kmalloc(struct kmem_cache *cache); +static __always_inline void kasan_cache_create_kmalloc(struct kmem_cache *cache) +{ + if (kasan_enabled()) + __kasan_cache_create_kmalloc(cache); +} + size_t __kasan_metadata_size(struct kmem_cache *cache); static __always_inline size_t kasan_metadata_size(struct kmem_cache *cache) { @@ -278,6 +286,7 @@ static inline void kasan_free_pages(struct page *page, unsigned int order) {} static inline void kasan_cache_create(struct kmem_cache *cache, unsigned int *size, slab_flags_t *flags) {} +static inline void kasan_cache_create_kmalloc(struct kmem_cache *cache) {} static inline size_t kasan_metadata_size(struct kmem_cache *cache) { return 0; } static inline void kasan_poison_slab(struct page *page) {} static inline void kasan_unpoison_object_data(struct kmem_cache *cache, diff --git a/mm/kasan/common.c b/mm/kasan/common.c index fe852f3cfa42..bfdf5464f4ef 100644 --- a/mm/kasan/common.c +++ b/mm/kasan/common.c @@ -210,6 +210,11 @@ void __kasan_cache_create(struct kmem_cache *cache, unsigned int *size, *size = optimal_size; } +void __kasan_cache_create_kmalloc(struct kmem_cache *cache) +{ + cache->kasan_info.is_kmalloc = true; +} + size_t __kasan_metadata_size(struct kmem_cache *cache) { if (!kasan_stack_collection_enabled()) @@ -394,17 +399,22 @@ void __kasan_slab_free_mempool(void *ptr, unsigned long ip) } } -static void set_alloc_info(struct kmem_cache *cache, void *object, gfp_t flags) +static void set_alloc_info(struct kmem_cache *cache, void *object, + gfp_t flags, bool is_kmalloc) { struct kasan_alloc_meta *alloc_meta; + /* Don't save alloc info for kmalloc caches in kasan_slab_alloc(). */ + if (cache->kasan_info.is_kmalloc && !is_kmalloc) + return; + alloc_meta = kasan_get_alloc_meta(cache, object); if (alloc_meta) kasan_set_track(&alloc_meta->alloc_track, flags); } static void *____kasan_kmalloc(struct kmem_cache *cache, const void *object, - size_t size, gfp_t flags, bool keep_tag) + size_t size, gfp_t flags, bool is_kmalloc) { unsigned long redzone_start; unsigned long redzone_end; @@ -423,7 +433,7 @@ static void *____kasan_kmalloc(struct kmem_cache *cache, const void *object, KASAN_GRANULE_SIZE); redzone_end = round_up((unsigned long)object + cache->object_size, KASAN_GRANULE_SIZE); - tag = assign_tag(cache, object, false, keep_tag); + tag = assign_tag(cache, object, false, is_kmalloc); /* Tag is ignored in set_tag without CONFIG_KASAN_SW/HW_TAGS */ kasan_unpoison(set_tag(object, tag), size); @@ -431,7 +441,7 @@ static void *____kasan_kmalloc(struct kmem_cache *cache, const void *object, KASAN_KMALLOC_REDZONE); if (kasan_stack_collection_enabled()) - set_alloc_info(cache, (void *)object, flags); + set_alloc_info(cache, (void *)object, flags, is_kmalloc); return set_tag(object, tag); } diff --git a/mm/slab_common.c b/mm/slab_common.c index 9aa3d2fe4c55..39d1a8ff9bb8 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -647,6 +647,7 @@ struct kmem_cache *__init create_kmalloc_cache(const char *name, panic("Out of memory when creating slab %s\n", name); create_boot_cache(s, name, size, flags, useroffset, usersize); + kasan_cache_create_kmalloc(s); list_add(&s->list, &slab_caches); s->refcount = 1; return s; From patchwork Fri Feb 5 15:39:03 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrey Konovalov X-Patchwork-Id: 12070349 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.1 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 59D4DC433E0 for ; Fri, 5 Feb 2021 15:40:54 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id DC373650F2 for ; Fri, 5 Feb 2021 15:40:53 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org DC373650F2 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:To:From:Subject:References:Mime-Version:Message-Id: In-Reply-To:Date:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=E4BgpM6wZFCDcrYosEmESsLSIb/qEV0WW2fUE8yTLG8=; b=y7UV3M5G+v/aeddV7hW8TwEyG GhynVIOIQh8nChErl4JpcR2wY/1RM5oc668dVje0kaZCbUbjCYXBh9rxWqNkgb7expTY7NkYCKILF 2JC/BXw2hU33IpH83kViyDkfDZhcLZNC34BWdIdvxtoPiLRkd3fUB0HiUGOG7H0u5wHxfPCAvbsH6 LWGnq2HGyLuA0SrN+zAnSVy+kY4TwiDptgBdnAcj7qnIrEpjL4wSKkXTe5lkB6iKCesT6dE6b4kCJ qyhSIuYKxfrm02WKYTlZYJ1ZEwviy1Ms5OdyUfrzCgXtUwEIiEJsYowoYHgk00E8CsMQ1LfuAtVwJ ut0UFs79Q==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1l83Cn-0007nA-Jk; Fri, 05 Feb 2021 15:39:33 +0000 Received: from mail-wm1-x34a.google.com ([2a00:1450:4864:20::34a]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1l83Cg-0007jO-7b for linux-arm-kernel@lists.infradead.org; Fri, 05 Feb 2021 15:39:28 +0000 Received: by mail-wm1-x34a.google.com with SMTP id j204so4003683wmj.4 for ; Fri, 05 Feb 2021 07:39:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=Ba9zMe8veYMITkXBVAbBRm/Mh4dw6QKopCfTTr6LLO8=; b=eyyQwZSWrc8gIJ19+0bZ9dld0c954iMkVn3KdaCBqdcoYM/q3jyUaJHfLEVYGsOy68 H4U/Cl1PNMyp6OW5gxUGbUz9Yj+P0usA3bD/Nd3uR4tioFkhTBZZQQbmzuLiWlNjpAGo lEjlTX2Xc9XTVpzD3T2MQS2QLRGkeOI0yAkONbLD9VHHInVbAvUZuJMCvrEhmV+6yLNT XLBMpoOBn1c9uBL0hnV9Ghfg/JMi3uEalE3ALS/99ELXBxRSlLptM2aPrnKUYzQR0O+i cIWYKcSlNVhs6is6j6v78uKEa7tmEB9MezqfIAyUZcuJRBRmLYD+uqB08F9SfZeT5Jpx q0uQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=Ba9zMe8veYMITkXBVAbBRm/Mh4dw6QKopCfTTr6LLO8=; b=hZ+79UNlC6qPIGTX8pwNj9E9FfaCxZYBwBhJp7F2OzctgTQcYJz4mmvzCKwJoxfQKp pMxkxPs854c09p2kHP3j5C0rv9/PSWuQ5Gcblzmgk1Jn+zYjQtbZGWz1+5kGSSj/SP6W mZwz750p2IGVyBBpFM9uAsieXCj8Z3+mqZ/e2KxMpHZE7Vi9ksjSEFJErKBSBvhM/FMP TsVY0rcF4Nm0GLU4X5kjQESjLJz2Szk7s9u3UddTspI7I5IEu3LvmvWnqUvJu+Sv5iMC QVAnZvBt+dxZrMZxde1PheaaZnyHaukRZbgU4USvYuc/nFtU2pVhHHnhGsq3A5MH+iNW jgAw== X-Gm-Message-State: AOAM531Gys/AcOh3UM1dBpGg7pfzCR3DUfbUGLHvz/7hEBi9+PGNqbpM sZlfcy0Co0RqY2qeSOX69Br/scLzrWgEHhY0 X-Google-Smtp-Source: ABdhPJx+hLylwy+1i90XS28Cay8KBzlLsVr+OPdAChGaAjdyF9Wm6+B5mKnpB5DEbFzjGDrWLSyJ8YuiSxdBLDlt X-Received: from andreyknvl3.muc.corp.google.com ([2a00:79e0:15:13:edb8:b79c:2e20:e531]) (user=andreyknvl job=sendgmr) by 2002:a05:600c:ce:: with SMTP id u14mr4141612wmm.10.1612539562830; Fri, 05 Feb 2021 07:39:22 -0800 (PST) Date: Fri, 5 Feb 2021 16:39:03 +0100 In-Reply-To: Message-Id: <708de0add7fc6a762e66452afd3d357a9cd5420a.1612538932.git.andreyknvl@google.com> Mime-Version: 1.0 References: X-Mailer: git-send-email 2.30.0.365.g02bc693789-goog Subject: [PATCH v2 02/12] kasan, mm: optimize kmalloc poisoning From: Andrey Konovalov To: Andrew Morton , Catalin Marinas , Vincenzo Frascino , Dmitry Vyukov , Alexander Potapenko , Marco Elver X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210205_103926_376662_E56C9CBC X-CRM114-Status: GOOD ( 26.02 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Branislav Rankov , Andrey Konovalov , Kevin Brodsky , Will Deacon , linux-kernel@vger.kernel.org, kasan-dev@googlegroups.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, Andrey Ryabinin , Peter Collingbourne , Evgenii Stepanov Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org For allocations from kmalloc caches, kasan_kmalloc() always follows kasan_slab_alloc(). Currenly, both of them unpoison the whole object, which is unnecessary. This patch provides separate implementations for both annotations: kasan_slab_alloc() unpoisons the whole object, and kasan_kmalloc() only poisons the redzone. For generic KASAN, the redzone start might not be aligned to KASAN_GRANULE_SIZE. Therefore, the poisoning is split in two parts: kasan_poison_last_granule() poisons the unaligned part, and then kasan_poison() poisons the rest. This patch also clarifies alignment guarantees of each of the poisoning functions and drops the unnecessary round_up() call for redzone_end. With this change, the early SLUB cache annotation needs to be changed to kasan_slab_alloc(), as kasan_kmalloc() doesn't unpoison objects now. The number of poisoned bytes for objects in this cache stays the same, as kmem_cache_node->object_size is equal to sizeof(struct kmem_cache_node). Reviewed-by: Marco Elver Signed-off-by: Andrey Konovalov --- mm/kasan/common.c | 93 +++++++++++++++++++++++++++++++---------------- mm/kasan/kasan.h | 43 +++++++++++++++++++++- mm/kasan/shadow.c | 28 +++++++------- mm/slub.c | 3 +- 4 files changed, 119 insertions(+), 48 deletions(-) diff --git a/mm/kasan/common.c b/mm/kasan/common.c index bfdf5464f4ef..00edbc3eb32e 100644 --- a/mm/kasan/common.c +++ b/mm/kasan/common.c @@ -278,21 +278,11 @@ void __kasan_poison_object_data(struct kmem_cache *cache, void *object) * based on objects indexes, so that objects that are next to each other * get different tags. */ -static u8 assign_tag(struct kmem_cache *cache, const void *object, - bool init, bool keep_tag) +static u8 assign_tag(struct kmem_cache *cache, const void *object, bool init) { if (IS_ENABLED(CONFIG_KASAN_GENERIC)) return 0xff; - /* - * 1. When an object is kmalloc()'ed, two hooks are called: - * kasan_slab_alloc() and kasan_kmalloc(). We assign the - * tag only in the first one. - * 2. We reuse the same tag for krealloc'ed objects. - */ - if (keep_tag) - return get_tag(object); - /* * If the cache neither has a constructor nor has SLAB_TYPESAFE_BY_RCU * set, assign a tag when the object is being allocated (init == false). @@ -325,7 +315,7 @@ void * __must_check __kasan_init_slab_obj(struct kmem_cache *cache, } /* Tag is ignored in set_tag() without CONFIG_KASAN_SW/HW_TAGS */ - object = set_tag(object, assign_tag(cache, object, true, false)); + object = set_tag(object, assign_tag(cache, object, true)); return (void *)object; } @@ -413,12 +403,46 @@ static void set_alloc_info(struct kmem_cache *cache, void *object, kasan_set_track(&alloc_meta->alloc_track, flags); } +void * __must_check __kasan_slab_alloc(struct kmem_cache *cache, + void *object, gfp_t flags) +{ + u8 tag; + void *tagged_object; + + if (gfpflags_allow_blocking(flags)) + kasan_quarantine_reduce(); + + if (unlikely(object == NULL)) + return NULL; + + if (is_kfence_address(object)) + return (void *)object; + + /* + * Generate and assign random tag for tag-based modes. + * Tag is ignored in set_tag() for the generic mode. + */ + tag = assign_tag(cache, object, false); + tagged_object = set_tag(object, tag); + + /* + * Unpoison the whole object. + * For kmalloc() allocations, kasan_kmalloc() will do precise poisoning. + */ + kasan_unpoison(tagged_object, cache->object_size); + + /* Save alloc info (if possible) for non-kmalloc() allocations. */ + if (kasan_stack_collection_enabled()) + set_alloc_info(cache, (void *)object, flags, false); + + return tagged_object; +} + static void *____kasan_kmalloc(struct kmem_cache *cache, const void *object, - size_t size, gfp_t flags, bool is_kmalloc) + size_t size, gfp_t flags) { unsigned long redzone_start; unsigned long redzone_end; - u8 tag; if (gfpflags_allow_blocking(flags)) kasan_quarantine_reduce(); @@ -429,33 +453,41 @@ static void *____kasan_kmalloc(struct kmem_cache *cache, const void *object, if (is_kfence_address(kasan_reset_tag(object))) return (void *)object; + /* + * The object has already been unpoisoned by kasan_slab_alloc() for + * kmalloc() or by ksize() for krealloc(). + */ + + /* + * The redzone has byte-level precision for the generic mode. + * Partially poison the last object granule to cover the unaligned + * part of the redzone. + */ + if (IS_ENABLED(CONFIG_KASAN_GENERIC)) + kasan_poison_last_granule((void *)object, size); + + /* Poison the aligned part of the redzone. */ redzone_start = round_up((unsigned long)(object + size), KASAN_GRANULE_SIZE); - redzone_end = round_up((unsigned long)object + cache->object_size, - KASAN_GRANULE_SIZE); - tag = assign_tag(cache, object, false, is_kmalloc); - - /* Tag is ignored in set_tag without CONFIG_KASAN_SW/HW_TAGS */ - kasan_unpoison(set_tag(object, tag), size); + redzone_end = (unsigned long)object + cache->object_size; kasan_poison((void *)redzone_start, redzone_end - redzone_start, KASAN_KMALLOC_REDZONE); + /* + * Save alloc info (if possible) for kmalloc() allocations. + * This also rewrites the alloc info when called from kasan_krealloc(). + */ if (kasan_stack_collection_enabled()) - set_alloc_info(cache, (void *)object, flags, is_kmalloc); + set_alloc_info(cache, (void *)object, flags, true); - return set_tag(object, tag); -} - -void * __must_check __kasan_slab_alloc(struct kmem_cache *cache, - void *object, gfp_t flags) -{ - return ____kasan_kmalloc(cache, object, cache->object_size, flags, false); + /* Keep the tag that was set by kasan_slab_alloc(). */ + return (void *)object; } void * __must_check __kasan_kmalloc(struct kmem_cache *cache, const void *object, size_t size, gfp_t flags) { - return ____kasan_kmalloc(cache, object, size, flags, true); + return ____kasan_kmalloc(cache, object, size, flags); } EXPORT_SYMBOL(__kasan_kmalloc); @@ -496,8 +528,7 @@ void * __must_check __kasan_krealloc(const void *object, size_t size, gfp_t flag if (unlikely(!PageSlab(page))) return __kasan_kmalloc_large(object, size, flags); else - return ____kasan_kmalloc(page->slab_cache, object, size, - flags, true); + return ____kasan_kmalloc(page->slab_cache, object, size, flags); } void __kasan_kfree_large(void *ptr, unsigned long ip) diff --git a/mm/kasan/kasan.h b/mm/kasan/kasan.h index dd14e8870023..6a2882997f23 100644 --- a/mm/kasan/kasan.h +++ b/mm/kasan/kasan.h @@ -358,12 +358,51 @@ static inline bool kasan_byte_accessible(const void *addr) #else /* CONFIG_KASAN_HW_TAGS */ -void kasan_poison(const void *address, size_t size, u8 value); -void kasan_unpoison(const void *address, size_t size); +/** + * kasan_poison - mark the memory range as unaccessible + * @addr - range start address, must be aligned to KASAN_GRANULE_SIZE + * @size - range size + * @value - value that's written to metadata for the range + * + * The size gets aligned to KASAN_GRANULE_SIZE before marking the range. + */ +void kasan_poison(const void *addr, size_t size, u8 value); + +/** + * kasan_unpoison - mark the memory range as accessible + * @addr - range start address, must be aligned to KASAN_GRANULE_SIZE + * @size - range size + * + * For the tag-based modes, the @size gets aligned to KASAN_GRANULE_SIZE before + * marking the range. + * For the generic mode, the last granule of the memory range gets partially + * unpoisoned based on the @size. + */ +void kasan_unpoison(const void *addr, size_t size); + bool kasan_byte_accessible(const void *addr); #endif /* CONFIG_KASAN_HW_TAGS */ +#ifdef CONFIG_KASAN_GENERIC + +/** + * kasan_poison_last_granule - mark the last granule of the memory range as + * unaccessible + * @addr - range start address, must be aligned to KASAN_GRANULE_SIZE + * @size - range size + * + * This function is only available for the generic mode, as it's the only mode + * that has partially poisoned memory granules. + */ +void kasan_poison_last_granule(const void *address, size_t size); + +#else /* CONFIG_KASAN_GENERIC */ + +static inline void kasan_poison_last_granule(const void *address, size_t size) { } + +#endif /* CONFIG_KASAN_GENERIC */ + /* * Exported functions for interfaces called from assembly or from generated * code. Declarations here to avoid warning about missing declarations. diff --git a/mm/kasan/shadow.c b/mm/kasan/shadow.c index 1372a2fc0ca9..1ed7817e4ee6 100644 --- a/mm/kasan/shadow.c +++ b/mm/kasan/shadow.c @@ -69,10 +69,6 @@ void *memcpy(void *dest, const void *src, size_t len) return __memcpy(dest, src, len); } -/* - * Poisons the shadow memory for 'size' bytes starting from 'addr'. - * Memory addresses should be aligned to KASAN_GRANULE_SIZE. - */ void kasan_poison(const void *address, size_t size, u8 value) { void *shadow_start, *shadow_end; @@ -83,12 +79,12 @@ void kasan_poison(const void *address, size_t size, u8 value) * addresses to this function. */ address = kasan_reset_tag(address); - size = round_up(size, KASAN_GRANULE_SIZE); /* Skip KFENCE memory if called explicitly outside of sl*b. */ if (is_kfence_address(address)) return; + size = round_up(size, KASAN_GRANULE_SIZE); shadow_start = kasan_mem_to_shadow(address); shadow_end = kasan_mem_to_shadow(address + size); @@ -96,6 +92,16 @@ void kasan_poison(const void *address, size_t size, u8 value) } EXPORT_SYMBOL(kasan_poison); +#ifdef CONFIG_KASAN_GENERIC +void kasan_poison_last_granule(const void *address, size_t size) +{ + if (size & KASAN_GRANULE_MASK) { + u8 *shadow = (u8 *)kasan_mem_to_shadow(address + size); + *shadow = size & KASAN_GRANULE_MASK; + } +} +#endif + void kasan_unpoison(const void *address, size_t size) { u8 tag = get_tag(address); @@ -115,16 +121,12 @@ void kasan_unpoison(const void *address, size_t size) if (is_kfence_address(address)) return; + /* Unpoison round_up(size, KASAN_GRANULE_SIZE) bytes. */ kasan_poison(address, size, tag); - if (size & KASAN_GRANULE_MASK) { - u8 *shadow = (u8 *)kasan_mem_to_shadow(address + size); - - if (IS_ENABLED(CONFIG_KASAN_SW_TAGS)) - *shadow = tag; - else /* CONFIG_KASAN_GENERIC */ - *shadow = size & KASAN_GRANULE_MASK; - } + /* Partially poison the last granule for the generic mode. */ + if (IS_ENABLED(CONFIG_KASAN_GENERIC)) + kasan_poison_last_granule(address, size); } #ifdef CONFIG_MEMORY_HOTPLUG diff --git a/mm/slub.c b/mm/slub.c index 176b1cb0d006..e564008c2329 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -3565,8 +3565,7 @@ static void early_kmem_cache_node_alloc(int node) init_object(kmem_cache_node, n, SLUB_RED_ACTIVE); init_tracking(kmem_cache_node, n); #endif - n = kasan_kmalloc(kmem_cache_node, n, sizeof(struct kmem_cache_node), - GFP_KERNEL); + n = kasan_slab_alloc(kmem_cache_node, n, GFP_KERNEL); page->freelist = get_freepointer(kmem_cache_node, n); page->inuse = 1; page->frozen = 0; From patchwork Fri Feb 5 15:39:04 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrey Konovalov X-Patchwork-Id: 12070353 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.1 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A79A7C433E0 for ; Fri, 5 Feb 2021 15:41:24 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 39BC865093 for ; Fri, 5 Feb 2021 15:41:24 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 39BC865093 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:To:From:Subject:References:Mime-Version:Message-Id: In-Reply-To:Date:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=Mehy53FT3eP+IUV0z8y/U7HPGF90Wf0dZrLAHwZ7Z2Y=; b=a0znrPk9JEfPCeAFzzmbmmZ7X 14o1zgY+xRiexJBplvHkZj8jWETOqK797/RJqDbjfFicgLmAKXSmELTMAdRWgo17F8QilHa7UgI9J IDf2jzbuL6i9hwXrjqn10f8D8/FWAvNTy4JblrDMaocBQ1Awztyp8ZSY4J1aJ7/d+S7K56L1Vg1aq 8yohw2QpRgLHI/IWdPPqnKOTRoQ/FgiRgRWq7pIbY9+yIB+UGDFCgn1E0mgSC5WbclE4tmOCEL4xK mnJKob2Y/LIX46SDsTA7R6P0oOSfCoPMvXmQDMo5F66Pyqo0bpdqykIXLyKwEmSJ/yOaGCPMyXb/3 954aCqzWQ==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1l83DE-0007z8-MM; Fri, 05 Feb 2021 15:40:00 +0000 Received: from mail-qv1-xf4a.google.com ([2607:f8b0:4864:20::f4a]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1l83Cl-0007kF-ET for linux-arm-kernel@lists.infradead.org; Fri, 05 Feb 2021 15:39:32 +0000 Received: by mail-qv1-xf4a.google.com with SMTP id ew14so5214063qvb.21 for ; Fri, 05 Feb 2021 07:39:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=tljlSxhBzcxqidAVSdVdhM6oUJ+nJ3d1xitEmX2Ppf0=; b=syVSGwpY7OBUajXhF+JU66oy4DayPtyhrZEKrCctZ7ZSEwdKPb5Fvf+ONRWcDtxp0K l36gKS8Yr71bsitJfXedVqVkHwOpQH2t/7/hBm4ZmVz/jpErYZ6yrtAzAzFjr4bLp1mg ZNGlS6epLYCc6jCq4C1m6SSWpMXkg7iVv0VbgQuORm2mYYrrw/8PmMphfXzVC3yZ6Lv6 f992UftKMf+GCjLbYmegMakIMsmuEpSMOCwEwtz+LdHiElyCXXWkfY9EjfKHBoHqkLxc tuCvq3A1ZCy11mG8Rs7N3bouZpjWecOu83K6/n5Uzv6c/sKJo2krn++2hCMlongp3G5K c3Zg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=tljlSxhBzcxqidAVSdVdhM6oUJ+nJ3d1xitEmX2Ppf0=; b=eOXgcGvxzgAGG4x0ZHynMTaB3NsAQMadf2kj7TO8OoGO5aK8di7C8zv0GVUcZHfISj jrscmHLTK9GznNwZBjXFPy33C55R6wRJ4rDo71uK2v2tccI3PeNJmhycp4oDpU9eszfw Ig8gGSa8HTKfmoWKuBmgNqIpnGWlMBS4GrDNZ+bKHfTmR/d5ES0Fuo/IOx5+14lqd8O8 Voc8Hm7Qnn56mjwd7omxmmGEtxen44B37te7GONq+7EB7DnKlMpKEu6RpxvJgapco+y2 +afXYEwA/SBgrAhieiFznGPFHxMiA+4+xi24+00+dXrV4oCr3Qf5tIszSECEpD7LPYOk Zt2A== X-Gm-Message-State: AOAM530UFKSUFAwfJ8+EvVzSUQxvIkU8+BTlFboQ1O748F1yBUVOeyGT dlh99qWdJbK2j9759b8Ru/EDAdD0MNhjcg+d X-Google-Smtp-Source: ABdhPJyO1MocpYFwZ1RXJ8jVPh8e1tllN7Hw4na0XycrZ5x/iFUpbgSITGryg5kyrNeA7pCK+1dElwIgxYqVy2a5 X-Received: from andreyknvl3.muc.corp.google.com ([2a00:79e0:15:13:edb8:b79c:2e20:e531]) (user=andreyknvl job=sendgmr) by 2002:a0c:f94a:: with SMTP id i10mr4890478qvo.22.1612539565151; Fri, 05 Feb 2021 07:39:25 -0800 (PST) Date: Fri, 5 Feb 2021 16:39:04 +0100 In-Reply-To: Message-Id: <97581e50e594596e0bf8dd5bb3598d5e13013f18.1612538932.git.andreyknvl@google.com> Mime-Version: 1.0 References: X-Mailer: git-send-email 2.30.0.365.g02bc693789-goog Subject: [PATCH v2 03/12] kasan: optimize large kmalloc poisoning From: Andrey Konovalov To: Andrew Morton , Catalin Marinas , Vincenzo Frascino , Dmitry Vyukov , Alexander Potapenko , Marco Elver X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210205_103931_549910_2B5EB20E X-CRM114-Status: GOOD ( 15.04 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Branislav Rankov , Andrey Konovalov , Kevin Brodsky , Will Deacon , linux-kernel@vger.kernel.org, kasan-dev@googlegroups.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, Andrey Ryabinin , Peter Collingbourne , Evgenii Stepanov Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Similarly to kasan_kmalloc(), kasan_kmalloc_large() doesn't need to unpoison the object as it as already unpoisoned by alloc_pages() (or by ksize() for krealloc()). This patch changes kasan_kmalloc_large() to only poison the redzone. Reviewed-by: Marco Elver Signed-off-by: Andrey Konovalov --- mm/kasan/common.c | 20 +++++++++++++++----- 1 file changed, 15 insertions(+), 5 deletions(-) diff --git a/mm/kasan/common.c b/mm/kasan/common.c index 00edbc3eb32e..f2a6bae13053 100644 --- a/mm/kasan/common.c +++ b/mm/kasan/common.c @@ -494,7 +494,6 @@ EXPORT_SYMBOL(__kasan_kmalloc); void * __must_check __kasan_kmalloc_large(const void *ptr, size_t size, gfp_t flags) { - struct page *page; unsigned long redzone_start; unsigned long redzone_end; @@ -504,12 +503,23 @@ void * __must_check __kasan_kmalloc_large(const void *ptr, size_t size, if (unlikely(ptr == NULL)) return NULL; - page = virt_to_page(ptr); + /* + * The object has already been unpoisoned by kasan_alloc_pages() for + * alloc_pages() or by ksize() for krealloc(). + */ + + /* + * The redzone has byte-level precision for the generic mode. + * Partially poison the last object granule to cover the unaligned + * part of the redzone. + */ + if (IS_ENABLED(CONFIG_KASAN_GENERIC)) + kasan_poison_last_granule(ptr, size); + + /* Poison the aligned part of the redzone. */ redzone_start = round_up((unsigned long)(ptr + size), KASAN_GRANULE_SIZE); - redzone_end = (unsigned long)ptr + page_size(page); - - kasan_unpoison(ptr, size); + redzone_end = (unsigned long)ptr + page_size(virt_to_page(ptr)); kasan_poison((void *)redzone_start, redzone_end - redzone_start, KASAN_PAGE_REDZONE); From patchwork Fri Feb 5 15:39:05 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrey Konovalov X-Patchwork-Id: 12070351 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.1 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id EF141C433E0 for ; Fri, 5 Feb 2021 15:41:13 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 771F7650F2 for ; Fri, 5 Feb 2021 15:41:13 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 771F7650F2 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:To:From:Subject:References:Mime-Version:Message-Id: In-Reply-To:Date:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=2pTM4Abs60f8Xohq1WuYazGkHj0xpztWcE925au1Huw=; b=pHxdr0HdyLeGxMlbgLTjDg2G+ VtFL0FAls5HhYmT5vAERcScZx5k5bOMOlfPS/w8oUCGbPEx8450xPQhSqIScTTfIYQrBmRETY21f4 wYMZqscythfE+PgSp3DpV3uMYBoZ890ZcjA5narmjY5AZmj4n3dYQjZyWd2dwuA+8gN7vXC2ilbE9 v/5tPwLMpTo1MH3mMa0eawakqizLtEP5T2kFKFL2yX1f3ppehcOHswHgBZvVo8vHsJCfqYplHjojT Wbn4MmFS1AQDPIC3PQEKrBdwMPIg73QpkBICpxeZWo5sCIfcQgvXld6gl4JhdvMvF/GZu65p04cYD YzygyG3Dg==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1l83DB-0007xX-Bw; Fri, 05 Feb 2021 15:39:58 +0000 Received: from mail-qk1-x749.google.com ([2607:f8b0:4864:20::749]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1l83Cl-0007lJ-C2 for linux-arm-kernel@lists.infradead.org; Fri, 05 Feb 2021 15:39:33 +0000 Received: by mail-qk1-x749.google.com with SMTP id o16so6133690qkj.15 for ; Fri, 05 Feb 2021 07:39:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=p2PHZd2SW9nz4o7+ZB1C3vkb9xhx8DH7M49Qc2p69KA=; b=lF3JI1Wz7SI+mSyZFHoJHyreQdSUTq5QY643cGS9GBBxjPRvwFD+iqezyeavSwGDTF uMprUNPCKb6jIjNQ9H27Fur8RtWhiDepPMiibytUc1NJTu0f7IFzzowNuNjrqBgmlLiv 7Tr4SU49efZvmHAiHw87MpqB2rqt/EdI6B3iqynlt/57vW11+kQCzeiLS4vtTtF/ES7x 3KkjctyAS2VhFEOvDpZJ0NSt8AamjRfvI1fi97PPs+ONtwsp9zEj2O5BeVMpwlZmNVVm vJzDRFXwJbhi9UAyMxR5Qu1VM7ormDH+bsHuZ7T8/e7tIEZCLXPYodKzTASn4vvBqtYJ 5c/w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=p2PHZd2SW9nz4o7+ZB1C3vkb9xhx8DH7M49Qc2p69KA=; b=N6TVrlgYiTBaQ1aMbmtXbs3o3NTJRilafUcDVQlyY5LYwJy09XHndzrzg91nbuhdMC GGCjwD+y/3gqQhx0F+JA0I32NTUmTGqK85t1V/SikoetaZfmY2v6a8LOmo+wuaKAr4In IcFK/BAY/aarAdSPwTcimpx80Gj6QqpuPq1jHQq5h6vgirDTaGfSQBgYWgx4Xw7t3UkQ 9L0ij5TYcfVfnRFI1CO5eJgKdK8YE9aqnwryLWqVIKNL6vAd7lHZx0PrZoFAAgDRQS2h nWx18t8AzHmlUid4a8iDTyGhewYqH3FvdYNss5QHuvR84jKiGCKYVPD4Yp1pQUz5xaRX Kx1A== X-Gm-Message-State: AOAM5300KlSNDHH26HE+hZdtwxSO1t+fv3midTa+9ea4uHP6WDPu/Q6C 5xnk4QawM/+q46Wu0wc6r+aLBRzXekz5H5P+ X-Google-Smtp-Source: ABdhPJzwsewfz/TZInoLtcuvYaZ0I40Qn7oK2vNyTP6uAYb18yvMtG1+sQlLK+ggxuOgSY+2gruJ4PNGKknNJrtO X-Received: from andreyknvl3.muc.corp.google.com ([2a00:79e0:15:13:edb8:b79c:2e20:e531]) (user=andreyknvl job=sendgmr) by 2002:a0c:eda6:: with SMTP id h6mr4862060qvr.19.1612539567383; Fri, 05 Feb 2021 07:39:27 -0800 (PST) Date: Fri, 5 Feb 2021 16:39:05 +0100 In-Reply-To: Message-Id: Mime-Version: 1.0 References: X-Mailer: git-send-email 2.30.0.365.g02bc693789-goog Subject: [PATCH v2 04/12] kasan: clean up setting free info in kasan_slab_free From: Andrey Konovalov To: Andrew Morton , Catalin Marinas , Vincenzo Frascino , Dmitry Vyukov , Alexander Potapenko , Marco Elver X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210205_103931_544435_95CC17A3 X-CRM114-Status: GOOD ( 11.14 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Branislav Rankov , Andrey Konovalov , Kevin Brodsky , Will Deacon , linux-kernel@vger.kernel.org, kasan-dev@googlegroups.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, Andrey Ryabinin , Peter Collingbourne , Evgenii Stepanov Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Put kasan_stack_collection_enabled() check and kasan_set_free_info() calls next to each other. The way this was previously implemented was a minor optimization that relied of the the fact that kasan_stack_collection_enabled() is always true for generic KASAN. The confusion that this brings outweights saving a few instructions. Reviewed-by: Marco Elver Signed-off-by: Andrey Konovalov --- mm/kasan/common.c | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/mm/kasan/common.c b/mm/kasan/common.c index f2a6bae13053..da24b144d46c 100644 --- a/mm/kasan/common.c +++ b/mm/kasan/common.c @@ -350,13 +350,11 @@ static bool ____kasan_slab_free(struct kmem_cache *cache, void *object, kasan_poison(object, cache->object_size, KASAN_KMALLOC_FREE); - if (!kasan_stack_collection_enabled()) - return false; - if ((IS_ENABLED(CONFIG_KASAN_GENERIC) && !quarantine)) return false; - kasan_set_free_info(cache, object, tag); + if (kasan_stack_collection_enabled()) + kasan_set_free_info(cache, object, tag); return kasan_quarantine_put(cache, object); } From patchwork Fri Feb 5 15:39:06 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrey Konovalov X-Patchwork-Id: 12070355 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.1 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 60F1EC433DB for ; Fri, 5 Feb 2021 15:41:31 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id EB47865093 for ; Fri, 5 Feb 2021 15:41:30 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org EB47865093 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:To:From:Subject:References:Mime-Version:Message-Id: In-Reply-To:Date:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=xqBMiEuVcgUu/PQiGlfRYY6pvYtxBSo9cjVkUWsCzQ4=; b=fkf7Z4/IjFV1kMLIhJ09PILiq +zUP2P5eDxFyhROXp5jdIW+Fc2Flty8t+/bEpECbf1DF8KFG3ISN+2DHTyduLy9bLYtAucx2iOtlq gXPcF81FjwgIG7Iw+N6XUmzGLh3Z+y4fDTuhMmtaMIz0NEh6o8HiFImZNTbFUtXhkJzL7Qhr6drjM osPj2l4npddu8+AwzNiDhC74cmMRiSKID0JJbj/Vc4cmnqzszghGVtW+9HIM80JBxDh3IUMQFY4gX xmPI8PC4TqKTpvh3hjOCyxk0E6C2ZMfKL7uEQIIv3by9/YNRuIgeJMgMtAQL6hcUON8STRtO0f1J+ IkcGb51qQ==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1l83DP-00082w-DG; Fri, 05 Feb 2021 15:40:11 +0000 Received: from mail-wm1-x349.google.com ([2a00:1450:4864:20::349]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1l83Cn-0007lu-57 for linux-arm-kernel@lists.infradead.org; Fri, 05 Feb 2021 15:39:34 +0000 Received: by mail-wm1-x349.google.com with SMTP id y9so3108685wmj.7 for ; Fri, 05 Feb 2021 07:39:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=kg30/g/LrsxD86gPJMTES5VW5Hgvjnk4vM4kmv21+Z4=; b=LCEFnlDaJcG0Q41NRa+Ch0e8o0QaxKPH4VLE5PeiB/2TOutaBSjhOVNhtlIuqskKNL phpumV6RzdrEdBMlzF0JJ/UV8Gi5lGDAW74e0LAzZYz+t1c3Y9oGFxEzdOri6TmYTr5U na1Uut/9hwzYcfOIuhkjF9fJjKOrh+9Tqp8O7hNSKLlOFpHYyFP5w0AVgMSjecylbZlj pFazgfiM/L2igEvCu3/YSfXsjME4M0Weph+1nBqRsxnRS5ZoqKSs/FwCT7YXTazQ1F47 i3VS6u9Ncn4Er7IZkPDPnp8tQRJvJXTxGiJK3gP2p+0V3VZSUOiVr8uO6j1rSNg8VTTG ++5A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=kg30/g/LrsxD86gPJMTES5VW5Hgvjnk4vM4kmv21+Z4=; b=eWTpYQm/XFvrxtsNAVqmwsYz3pvipred5bsR5Le7f2Xc6pJoAoYKdeQ7iH0ChbOcFd e3m2LGRbMRaXIVVolFlTg3z3ZbAtQ7ClqAgFau0R5Q/1P4QcuU6TBku8bzRCuIROaNjd wGapWJCgSlfFLIUfs+VeD2BrkmYzUv7yb5QuAFfdTfR4hbrY3t0bRBpIukuQovbIWmjb h/3HKGqsvswcV1tsES8oNz4qsxPNO/F1nXvMtJ+adqlUWdr5Yr9svmZ7qA7T7adDbeXF E+RAcnZwpHVxhCwDzxw9HmwFZHEFtj2esDgg9TSoZMQ2f32t+TUDuG33c3nNHT5TPUIe MgCg== X-Gm-Message-State: AOAM5323YerTUWse3VpUEsvop2N7TPRcwHEOfF9eAXwg8EW550HN21P7 oreccQMrZrmkSMoVzsNnjResEB3gxLUDgooJ X-Google-Smtp-Source: ABdhPJwwKKgQGlKIo0fdvx1Js8AFJ3EgmJEbKeq2cWI3dxu4cKp8SA/tbBXB1FczDscVCzjNnOVVknB1qI6ECJfd X-Received: from andreyknvl3.muc.corp.google.com ([2a00:79e0:15:13:edb8:b79c:2e20:e531]) (user=andreyknvl job=sendgmr) by 2002:a7b:c2aa:: with SMTP id c10mr3901926wmk.101.1612539569588; Fri, 05 Feb 2021 07:39:29 -0800 (PST) Date: Fri, 5 Feb 2021 16:39:06 +0100 In-Reply-To: Message-Id: <492dcd5030419c5421a3762457c0ff1a7c91e628.1612538932.git.andreyknvl@google.com> Mime-Version: 1.0 References: X-Mailer: git-send-email 2.30.0.365.g02bc693789-goog Subject: [PATCH v2 05/12] kasan: unify large kfree checks From: Andrey Konovalov To: Andrew Morton , Catalin Marinas , Vincenzo Frascino , Dmitry Vyukov , Alexander Potapenko , Marco Elver X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210205_103933_262305_CDF306FA X-CRM114-Status: GOOD ( 16.21 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Branislav Rankov , Andrey Konovalov , Kevin Brodsky , Will Deacon , linux-kernel@vger.kernel.org, kasan-dev@googlegroups.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, Andrey Ryabinin , Peter Collingbourne , Evgenii Stepanov Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Unify checks in kasan_kfree_large() and in kasan_slab_free_mempool() for large allocations as it's done for small kfree() allocations. With this change, kasan_slab_free_mempool() starts checking that the first byte of the memory that's being freed is accessible. Reviewed-by: Marco Elver Signed-off-by: Andrey Konovalov --- include/linux/kasan.h | 16 ++++++++-------- mm/kasan/common.c | 36 ++++++++++++++++++++++++++---------- 2 files changed, 34 insertions(+), 18 deletions(-) diff --git a/include/linux/kasan.h b/include/linux/kasan.h index 2d5de4092185..d53ea3c047bc 100644 --- a/include/linux/kasan.h +++ b/include/linux/kasan.h @@ -200,6 +200,13 @@ static __always_inline bool kasan_slab_free(struct kmem_cache *s, void *object) return false; } +void __kasan_kfree_large(void *ptr, unsigned long ip); +static __always_inline void kasan_kfree_large(void *ptr) +{ + if (kasan_enabled()) + __kasan_kfree_large(ptr, _RET_IP_); +} + void __kasan_slab_free_mempool(void *ptr, unsigned long ip); static __always_inline void kasan_slab_free_mempool(void *ptr) { @@ -247,13 +254,6 @@ static __always_inline void * __must_check kasan_krealloc(const void *object, return (void *)object; } -void __kasan_kfree_large(void *ptr, unsigned long ip); -static __always_inline void kasan_kfree_large(void *ptr) -{ - if (kasan_enabled()) - __kasan_kfree_large(ptr, _RET_IP_); -} - /* * Unlike kasan_check_read/write(), kasan_check_byte() is performed even for * the hardware tag-based mode that doesn't rely on compiler instrumentation. @@ -302,6 +302,7 @@ static inline bool kasan_slab_free(struct kmem_cache *s, void *object) { return false; } +static inline void kasan_kfree_large(void *ptr) {} static inline void kasan_slab_free_mempool(void *ptr) {} static inline void *kasan_slab_alloc(struct kmem_cache *s, void *object, gfp_t flags) @@ -322,7 +323,6 @@ static inline void *kasan_krealloc(const void *object, size_t new_size, { return (void *)object; } -static inline void kasan_kfree_large(void *ptr) {} static inline bool kasan_check_byte(const void *address) { return true; diff --git a/mm/kasan/common.c b/mm/kasan/common.c index da24b144d46c..7ea643f7e69c 100644 --- a/mm/kasan/common.c +++ b/mm/kasan/common.c @@ -364,6 +364,31 @@ bool __kasan_slab_free(struct kmem_cache *cache, void *object, unsigned long ip) return ____kasan_slab_free(cache, object, ip, true); } +static bool ____kasan_kfree_large(void *ptr, unsigned long ip) +{ + if (ptr != page_address(virt_to_head_page(ptr))) { + kasan_report_invalid_free(ptr, ip); + return true; + } + + if (!kasan_byte_accessible(ptr)) { + kasan_report_invalid_free(ptr, ip); + return true; + } + + /* + * The object will be poisoned by kasan_free_pages() or + * kasan_slab_free_mempool(). + */ + + return false; +} + +void __kasan_kfree_large(void *ptr, unsigned long ip) +{ + ____kasan_kfree_large(ptr, ip); +} + void __kasan_slab_free_mempool(void *ptr, unsigned long ip) { struct page *page; @@ -377,10 +402,8 @@ void __kasan_slab_free_mempool(void *ptr, unsigned long ip) * KMALLOC_MAX_SIZE, and kmalloc falls back onto page_alloc. */ if (unlikely(!PageSlab(page))) { - if (ptr != page_address(page)) { - kasan_report_invalid_free(ptr, ip); + if (____kasan_kfree_large(ptr, ip)) return; - } kasan_poison(ptr, page_size(page), KASAN_FREE_PAGE); } else { ____kasan_slab_free(page->slab_cache, ptr, ip, false); @@ -539,13 +562,6 @@ void * __must_check __kasan_krealloc(const void *object, size_t size, gfp_t flag return ____kasan_kmalloc(page->slab_cache, object, size, flags); } -void __kasan_kfree_large(void *ptr, unsigned long ip) -{ - if (ptr != page_address(virt_to_head_page(ptr))) - kasan_report_invalid_free(ptr, ip); - /* The object will be poisoned by kasan_free_pages(). */ -} - bool __kasan_check_byte(const void *address, unsigned long ip) { if (!kasan_byte_accessible(address)) { From patchwork Fri Feb 5 15:39:07 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrey Konovalov X-Patchwork-Id: 12070357 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.1 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C23A3C433E0 for ; Fri, 5 Feb 2021 15:41:31 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 641FE650F2 for ; Fri, 5 Feb 2021 15:41:31 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 641FE650F2 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:To:From:Subject:References:Mime-Version:Message-Id: In-Reply-To:Date:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=sqZ8oJtNcOWKIQ0reQBO3QCgI8kTM7cWTnkTX8zRflA=; b=KVteWFAKbZAl9CXg3J3DRZii7 0lAIjYVb6A5G5xXU/0APEQDnicp5x3+pzesDoYmVDV0PoyFZo3LHS/H2bqu3lSct2fAZCrPCrgAIL kOJZR4uqOM3foFxTGgwRTgE+Gl3eBdl7mssHNqW3IXJhPDm+AYbcc9bjCt5K7OkmvNFhaRIOMff/2 aL9usSoOGkJtlPdGfsT5BRqrn3P9zPKt/CLUBKspGIsZKxpANw0WYBAslLPbyWRWQfieMWH6lM5GN bHIxpg2Rf7IO7ql+lPJk+U9Y2mDzj5IlN3Z+Z7ryeh+E55oR0aIL7HqrY2xF3Diiz+a3StdNJuJBQ wckHom5+w==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1l83DS-00084R-A4; Fri, 05 Feb 2021 15:40:14 +0000 Received: from mail-wr1-x44a.google.com ([2a00:1450:4864:20::44a]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1l83Cq-0007nR-Es for linux-arm-kernel@lists.infradead.org; Fri, 05 Feb 2021 15:39:37 +0000 Received: by mail-wr1-x44a.google.com with SMTP id u15so5575253wrn.3 for ; Fri, 05 Feb 2021 07:39:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=8mKaNk2QrOQTwEWb38aMmksU7/sGdVLcyph/DgY5wHs=; b=QfjYpmjXoRqlsRMRzIr1+KVDuXCF2mGgmlq8dFVbwnt4uNtQtYnNne3p0LEk8PZQDP LH5OZXYZHXnUePnOcLRuNPszmnd0FbfwstXEBdw/yq0puinkR3zPz/pMVKUI41FN1LRO TITdhTVPQ66BT7n+DGzJ0E4W8mHLBXe/RUdro9N/o4FevnpDSQT2EyQ2VTOhoKL0Hp51 4Yghkms3teme8JHxSmMN4lmoel1ruP8Q48RDs2QWYWOy9muAuK4Vh7IhKz+yWLzPjY7X R5eEwvcYRQPrf+CrkQtsI/MqKbLgrmOyHKWOEGwlEQK6O7TeqmH59+a4LK7xxIg906Nl 3wrw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=8mKaNk2QrOQTwEWb38aMmksU7/sGdVLcyph/DgY5wHs=; b=exk3+cbX8swqr0obYSkDFMyC1aPAzK/de7hZAx4YxvOi6xwQH/ltuaJRmPtol9LLS2 ED4J4fY8yXrxM/oy7UR71DZjfdu1nRpgglcHz2PZDNHpsKJFqNEs8VPsx8AUB/suAuQj nWAB9v9aO1yTwMDl/XAMlomvYGf9LMKbGlxsTeh1Pb6f9zYeOjyt3XhuPbr5IGPYwbWl xqGLik9LfypXf5Dr0W/9CDQYunSDqv4pxzcIvzdqgKiNQqyxpAW1XADQasz8dpv2nOle QD7jLehE4D+EvnCZJ0LILDYfFyRfepsyVos4XvKtU9bX7sTSw8HAJCmJkt1xC/7InRxV f86g== X-Gm-Message-State: AOAM532r/hYxPiTx14hl4+ju3NGZbk06L7lc2SBM/J30pg2MEOvk3AJh YQC/qaTM3/rzsKbWajkQ9A7NrO7s1o2iMmfQ X-Google-Smtp-Source: ABdhPJzYoU8WAwIEjECte+81tcJ5TXF+4Xxppz8zhonkdU8HFTaLQCKRYJRQjlnulmI0sfacZvfaDU8xnzFhQTTE X-Received: from andreyknvl3.muc.corp.google.com ([2a00:79e0:15:13:edb8:b79c:2e20:e531]) (user=andreyknvl job=sendgmr) by 2002:a05:600c:35c9:: with SMTP id r9mr499080wmq.0.1612539572043; Fri, 05 Feb 2021 07:39:32 -0800 (PST) Date: Fri, 5 Feb 2021 16:39:07 +0100 In-Reply-To: Message-Id: Mime-Version: 1.0 References: X-Mailer: git-send-email 2.30.0.365.g02bc693789-goog Subject: [PATCH v2 06/12] kasan: rework krealloc tests From: Andrey Konovalov To: Andrew Morton , Catalin Marinas , Vincenzo Frascino , Dmitry Vyukov , Alexander Potapenko , Marco Elver X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210205_103936_661650_A61705F0 X-CRM114-Status: GOOD ( 13.84 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Branislav Rankov , Andrey Konovalov , Kevin Brodsky , Will Deacon , linux-kernel@vger.kernel.org, kasan-dev@googlegroups.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, Andrey Ryabinin , Peter Collingbourne , Evgenii Stepanov Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org This patch reworks KASAN-KUnit tests for krealloc() to: 1. Check both slab and page_alloc based krealloc() implementations. 2. Allow at least one full granule to fit between old and new sizes for each KASAN mode, and check accesses to that granule accordingly. Reviewed-by: Marco Elver Signed-off-by: Andrey Konovalov --- lib/test_kasan.c | 91 ++++++++++++++++++++++++++++++++++++++++++------ 1 file changed, 81 insertions(+), 10 deletions(-) diff --git a/lib/test_kasan.c b/lib/test_kasan.c index 5699e43ca01b..6e63ba62db09 100644 --- a/lib/test_kasan.c +++ b/lib/test_kasan.c @@ -258,11 +258,14 @@ static void kmalloc_large_oob_right(struct kunit *test) kfree(ptr); } -static void kmalloc_oob_krealloc_more(struct kunit *test) +static void krealloc_more_oob_helper(struct kunit *test, + size_t size1, size_t size2) { char *ptr1, *ptr2; - size_t size1 = 17; - size_t size2 = 19; + size_t middle; + + KUNIT_ASSERT_LT(test, size1, size2); + middle = size1 + (size2 - size1) / 2; ptr1 = kmalloc(size1, GFP_KERNEL); KUNIT_ASSERT_NOT_ERR_OR_NULL(test, ptr1); @@ -270,15 +273,31 @@ static void kmalloc_oob_krealloc_more(struct kunit *test) ptr2 = krealloc(ptr1, size2, GFP_KERNEL); KUNIT_ASSERT_NOT_ERR_OR_NULL(test, ptr2); - KUNIT_EXPECT_KASAN_FAIL(test, ptr2[size2 + OOB_TAG_OFF] = 'x'); + /* All offsets up to size2 must be accessible. */ + ptr2[size1 - 1] = 'x'; + ptr2[size1] = 'x'; + ptr2[middle] = 'x'; + ptr2[size2 - 1] = 'x'; + + /* Generic mode is precise, so unaligned size2 must be inaccessible. */ + if (IS_ENABLED(CONFIG_KASAN_GENERIC)) + KUNIT_EXPECT_KASAN_FAIL(test, ptr2[size2] = 'x'); + + /* For all modes first aligned offset after size2 must be inaccessible. */ + KUNIT_EXPECT_KASAN_FAIL(test, + ptr2[round_up(size2, KASAN_GRANULE_SIZE)] = 'x'); + kfree(ptr2); } -static void kmalloc_oob_krealloc_less(struct kunit *test) +static void krealloc_less_oob_helper(struct kunit *test, + size_t size1, size_t size2) { char *ptr1, *ptr2; - size_t size1 = 17; - size_t size2 = 15; + size_t middle; + + KUNIT_ASSERT_LT(test, size2, size1); + middle = size2 + (size1 - size2) / 2; ptr1 = kmalloc(size1, GFP_KERNEL); KUNIT_ASSERT_NOT_ERR_OR_NULL(test, ptr1); @@ -286,10 +305,60 @@ static void kmalloc_oob_krealloc_less(struct kunit *test) ptr2 = krealloc(ptr1, size2, GFP_KERNEL); KUNIT_ASSERT_NOT_ERR_OR_NULL(test, ptr2); - KUNIT_EXPECT_KASAN_FAIL(test, ptr2[size2 + OOB_TAG_OFF] = 'x'); + /* Must be accessible for all modes. */ + ptr2[size2 - 1] = 'x'; + + /* Generic mode is precise, so unaligned size2 must be inaccessible. */ + if (IS_ENABLED(CONFIG_KASAN_GENERIC)) + KUNIT_EXPECT_KASAN_FAIL(test, ptr2[size2] = 'x'); + + /* For all modes first aligned offset after size2 must be inaccessible. */ + KUNIT_EXPECT_KASAN_FAIL(test, + ptr2[round_up(size2, KASAN_GRANULE_SIZE)] = 'x'); + + /* + * For all modes all size2, middle, and size1 should land in separate + * granules and thus the latter two offsets should be inaccessible. + */ + KUNIT_EXPECT_LE(test, round_up(size2, KASAN_GRANULE_SIZE), + round_down(middle, KASAN_GRANULE_SIZE)); + KUNIT_EXPECT_LE(test, round_up(middle, KASAN_GRANULE_SIZE), + round_down(size1, KASAN_GRANULE_SIZE)); + KUNIT_EXPECT_KASAN_FAIL(test, ptr2[middle] = 'x'); + KUNIT_EXPECT_KASAN_FAIL(test, ptr2[size1 - 1] = 'x'); + KUNIT_EXPECT_KASAN_FAIL(test, ptr2[size1] = 'x'); + kfree(ptr2); } +static void krealloc_more_oob(struct kunit *test) +{ + krealloc_more_oob_helper(test, 201, 235); +} + +static void krealloc_less_oob(struct kunit *test) +{ + krealloc_less_oob_helper(test, 235, 201); +} + +static void krealloc_pagealloc_more_oob(struct kunit *test) +{ + /* page_alloc fallback in only implemented for SLUB. */ + KASAN_TEST_NEEDS_CONFIG_ON(test, CONFIG_SLUB); + + krealloc_more_oob_helper(test, KMALLOC_MAX_CACHE_SIZE + 201, + KMALLOC_MAX_CACHE_SIZE + 235); +} + +static void krealloc_pagealloc_less_oob(struct kunit *test) +{ + /* page_alloc fallback in only implemented for SLUB. */ + KASAN_TEST_NEEDS_CONFIG_ON(test, CONFIG_SLUB); + + krealloc_less_oob_helper(test, KMALLOC_MAX_CACHE_SIZE + 235, + KMALLOC_MAX_CACHE_SIZE + 201); +} + static void kmalloc_oob_16(struct kunit *test) { struct { @@ -983,8 +1052,10 @@ static struct kunit_case kasan_kunit_test_cases[] = { KUNIT_CASE(pagealloc_oob_right), KUNIT_CASE(pagealloc_uaf), KUNIT_CASE(kmalloc_large_oob_right), - KUNIT_CASE(kmalloc_oob_krealloc_more), - KUNIT_CASE(kmalloc_oob_krealloc_less), + KUNIT_CASE(krealloc_more_oob), + KUNIT_CASE(krealloc_less_oob), + KUNIT_CASE(krealloc_pagealloc_more_oob), + KUNIT_CASE(krealloc_pagealloc_less_oob), KUNIT_CASE(kmalloc_oob_16), KUNIT_CASE(kmalloc_uaf_16), KUNIT_CASE(kmalloc_oob_in_memset), From patchwork Fri Feb 5 15:39:08 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrey Konovalov X-Patchwork-Id: 12070359 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.1 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id F100DC433E6 for ; Fri, 5 Feb 2021 15:41:35 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 9C061650EF for ; Fri, 5 Feb 2021 15:41:35 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 9C061650EF Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:To:From:Subject:References:Mime-Version:Message-Id: In-Reply-To:Date:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=vLEM1PmdawB7gfOxEpqPHl+nuxI8EwZc36wfe3r0//U=; b=RtqJzfO4otqO+60gjC8aScsDZ LM/vhZMZ7fMLoqhOSdmGHZVOIFV8g0pUk+XPiuOeNcah8zlC1HV6vYvUDu9vsP6UBDN0mQwPX3wlx Yc7aho0jUkgaWKdG1x+hXGMkz09/05dUObAsSywQreAmjxkLJNlfoSw20l3kIpmtsCNds/8dcm1h+ NcGwUsXwEETlTuN2daSvByYShWfS+MAPE3qRzX0XB1mn2V+IE7M048dtfD/Y+xifXQge1oZRy3I93 3dT3KIN4gE9YcFcR6RMqBZcGqe5IFbmtGsMhAvYTJE6Dwrdc0f8iW8xPFYpb9LsBD2hCHounLMBgc QpJIK6q+g==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1l83DW-000860-Ns; Fri, 05 Feb 2021 15:40:18 +0000 Received: from mail-wm1-x34a.google.com ([2a00:1450:4864:20::34a]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1l83Cs-0007oR-1w for linux-arm-kernel@lists.infradead.org; Fri, 05 Feb 2021 15:39:39 +0000 Received: by mail-wm1-x34a.google.com with SMTP id 5so3118358wmq.0 for ; Fri, 05 Feb 2021 07:39:35 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=Eem+CPI6cqDr3+sdydeGQiL8gIW8yui8+/Xm+DAKB7k=; b=SekTvXiN+P4rBMIGDiv0wwZwiSR5ud4S18Maw4kVgQUSz5FEyuQjH7o5vB8M2vmUMM Q9IwKiqWZ2Yzni/Twm1vRliwxS6Ofyr6GjmKPH0fvoYIpgsmkQdgB+otayIHcj7mH/2D jmQoetdjxOjtR114YWCAtS8Fc2juWVSQssSxdWmZlUIYr3jed51NrPHMZtJCMyGYNRLf xFoK/UyBl2mkVnEi/vaWWeRBriSKqgl/b+Wku4w85oDpVvnXWiYQLwR5eUbxAOT6jUTa Z8LefNUER57XhQirYMl1Si4r9oJMTxRRv8rujMp6fw+hnNTyANk4BtTWwAwQn6481LYu 1Bbg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=Eem+CPI6cqDr3+sdydeGQiL8gIW8yui8+/Xm+DAKB7k=; b=SFsraj+RE6wAAu1suRXOqt3RN5/UdxqDwXXlOu/pVcL/Nts8X7VE33PkdEAEkrDfEI 2ajp87nEcJRwkVUE7nYRsnD+KpeWPFrGjKQv3Pveakzzireoul/IXZ0Af8coplyGeK4a cDJrztLoCyTLEHuFeI6ywsI8OeOJeMrP5QylU7WcGVBcRmmDir+tKMjiJbw+wIFNxFWM 9QLWE/RT6Fw/bHpROiEYKCdsmH/0rhGoFQMBAu8YSugpaBfwf70xp+Ix1IDJ3jAh8Jrr B4HP4U2gANEhKYUV1Dj+gizW3RJoGzWoS4XMvGcA8gEy0NJ5LnHGFJX6DqRQdxlg2IoQ snHw== X-Gm-Message-State: AOAM533+XWOPzxQ4BCrEqIcvBhQUe2lurZl+93h8Gh2bRmG5DN13HQu6 LAlbtrMXKaBHncglEowfc2z74/zTgdnHdovH X-Google-Smtp-Source: ABdhPJyqQdKHrhOuNMT3pVa3R7FPH70A7UVF1EhvgAytPKqWBBtNMrRHXWGCczaKH2RqOThi7wWAhXKWKxUeKXJY X-Received: from andreyknvl3.muc.corp.google.com ([2a00:79e0:15:13:edb8:b79c:2e20:e531]) (user=andreyknvl job=sendgmr) by 2002:a1c:25c2:: with SMTP id l185mr4086468wml.62.1612539574962; Fri, 05 Feb 2021 07:39:34 -0800 (PST) Date: Fri, 5 Feb 2021 16:39:08 +0100 In-Reply-To: Message-Id: <419e4e7ac9ff7596a1a0956c117fcad1938e5d77.1612538932.git.andreyknvl@google.com> Mime-Version: 1.0 References: X-Mailer: git-send-email 2.30.0.365.g02bc693789-goog Subject: [PATCH v2 07/12] kasan, mm: fail krealloc on freed objects From: Andrey Konovalov To: Andrew Morton , Catalin Marinas , Vincenzo Frascino , Dmitry Vyukov , Alexander Potapenko , Marco Elver X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210205_103938_165924_69F3E038 X-CRM114-Status: GOOD ( 17.53 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Branislav Rankov , Andrey Konovalov , Kevin Brodsky , Will Deacon , linux-kernel@vger.kernel.org, kasan-dev@googlegroups.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, Andrey Ryabinin , Peter Collingbourne , Evgenii Stepanov Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Currently, if krealloc() is called on a freed object with KASAN enabled, it allocates and returns a new object, but doesn't copy any memory from the old one as ksize() returns 0. This makes the caller believe that krealloc() succeeded (KASAN report is printed though). This patch adds an accessibility check into __do_krealloc(). If the check fails, krealloc() returns NULL. This check duplicates the one in ksize(); this is fixed in the following patch. This patch also adds a KASAN-KUnit test to check krealloc() behaviour when it's called on a freed object. Reviewed-by: Marco Elver Signed-off-by: Andrey Konovalov --- lib/test_kasan.c | 20 ++++++++++++++++++++ mm/slab_common.c | 3 +++ 2 files changed, 23 insertions(+) diff --git a/lib/test_kasan.c b/lib/test_kasan.c index 6e63ba62db09..791164ef191b 100644 --- a/lib/test_kasan.c +++ b/lib/test_kasan.c @@ -359,6 +359,25 @@ static void krealloc_pagealloc_less_oob(struct kunit *test) KMALLOC_MAX_CACHE_SIZE + 201); } +/* + * Check that krealloc() detects a use-after-free, returns NULL, + * and doesn't unpoison the freed object. + */ +static void krealloc_uaf(struct kunit *test) +{ + char *ptr1, *ptr2; + int size1 = 201; + int size2 = 235; + + ptr1 = kmalloc(size1, GFP_KERNEL); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, ptr1); + kfree(ptr1); + + KUNIT_EXPECT_KASAN_FAIL(test, ptr2 = krealloc(ptr1, size2, GFP_KERNEL)); + KUNIT_ASSERT_PTR_EQ(test, (void *)ptr2, NULL); + KUNIT_EXPECT_KASAN_FAIL(test, *(volatile char *)ptr1); +} + static void kmalloc_oob_16(struct kunit *test) { struct { @@ -1056,6 +1075,7 @@ static struct kunit_case kasan_kunit_test_cases[] = { KUNIT_CASE(krealloc_less_oob), KUNIT_CASE(krealloc_pagealloc_more_oob), KUNIT_CASE(krealloc_pagealloc_less_oob), + KUNIT_CASE(krealloc_uaf), KUNIT_CASE(kmalloc_oob_16), KUNIT_CASE(kmalloc_uaf_16), KUNIT_CASE(kmalloc_oob_in_memset), diff --git a/mm/slab_common.c b/mm/slab_common.c index 39d1a8ff9bb8..dad70239b54c 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -1140,6 +1140,9 @@ static __always_inline void *__do_krealloc(const void *p, size_t new_size, void *ret; size_t ks; + if (likely(!ZERO_OR_NULL_PTR(p)) && !kasan_check_byte(p)) + return NULL; + ks = ksize(p); if (ks >= new_size) { From patchwork Fri Feb 5 15:39:09 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrey Konovalov X-Patchwork-Id: 12070361 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.1 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 234B5C433E0 for ; Fri, 5 Feb 2021 15:41:36 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id CA32365093 for ; Fri, 5 Feb 2021 15:41:35 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org CA32365093 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:To:From:Subject:References:Mime-Version:Message-Id: In-Reply-To:Date:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=KNc/Aqb0R1ljnH8mLrlAiD++LrvE1EqRwG635zi2YRA=; b=Jvg020VKez2XmzJ/lGKpXoypM l/518O90cXwvRLQFR1sLXf59Kcn1mdA+AczGImzSA9V2XFNc6uHnhLHIeFidcGaEST4ZemjNfnPEL fhNon/8bYyFJEnhNtW7vHrlBLud21c0/e0/GOJ3BrH7RYVykd0aKRlh2Zc01PZQRycEzk9PsOwaRV Vb41YXVvf8p4Up/MQmJhCSL/dcJi9r7bOvh9rGpBmrkJ7ZbCkJ46BS521ox7j9T99xMk9zG7RSOlG Tk/v8ubGOPkY11wu9lgsHjJ66Y0t1XQEURz3vscemZ536t8xHDihOUpi4IxLi2+n+1ZQqtpbCpmc8 CJQdh2JQQ==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1l83DZ-00088B-Dh; Fri, 05 Feb 2021 15:40:21 +0000 Received: from mail-wm1-x349.google.com ([2a00:1450:4864:20::349]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1l83Cu-0007pf-TN for linux-arm-kernel@lists.infradead.org; Fri, 05 Feb 2021 15:39:42 +0000 Received: by mail-wm1-x349.google.com with SMTP id q24so3955053wmc.1 for ; Fri, 05 Feb 2021 07:39:38 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=ISHwxx5EnSKzUDUjZ1ysUZFyHBfF802klWxWhYA/428=; b=TwrM5M5Ch2ZUHOoLxcPnwrE6JjOif4wmSkb2jlervFb/bxU4iYzugCMCpwAcRbHq4H N1zF1OuUuYKeX9V3GLI1wTwMILzVeF8FretvHV0csoNee1X7kwhevOuPjyG+T+OnOkVJ WVDsEz8WQh3gM3Ee9A6eYiS0TvgVp3KTN2FD9pdhx0hXv1gZr+8B5bwofshs4ZnRH/0Z eWSpUIdzy4P9bv4dq+Rq9ApiCo6ZPg5NelpiqyibiKy77/qDeLoJXoNfUEujlZouAVVQ DkWq2ovXteNYiuO0JT92Mg8SxjluiRWLibM+8zqds5UuLj66AcJhYc0GAg6plxLo1Eru l8Ig== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=ISHwxx5EnSKzUDUjZ1ysUZFyHBfF802klWxWhYA/428=; b=Sqz/IIZMqS7xNsmfx6fP4HX/y5rz+YmH/i89XPH0gEl6EUqH6V5KhUQXCxsj/MFkB7 eWJenk0dqYCCBMv9zDPoOsrLAR7le3JgEmv/Kxqkf5NDDHeLYKUui3iE+TUD92mcyj7P sDxrG5vBK0vXtlqS1vBsWoC3p/VwOBOsEsGA01AdXb6wmNSYWd0nS9U7iCt77mqf3qtg GL2RdXqrl/YYWMW03G/yTGK7hGlNvrhmaZxZqH7rEzZEJs0nYBlS+H3RYd64jd9ngbKi q0UsaTHJ+EJz8kkT637ejvS/Q2TN1d3j314kEKaDhzRO8pR5sVIt0efKlhNLh7v4fUTU CwTQ== X-Gm-Message-State: AOAM5301aKCY324XppeWLHXR+23+qXmKVRtQ1VfMg1grSkY5BzYDkjgB mBVluCJVoxh7Qa5iuZVZNbgOM87cN1bwqKYA X-Google-Smtp-Source: ABdhPJxP6vMDfthzecVC5ecdLO+5Dv7Y8pBeoXM66UJWgdoywOg+JFnFIENVMygIhQfAgTki+HWSylKOMpIDzaAZ X-Received: from andreyknvl3.muc.corp.google.com ([2a00:79e0:15:13:edb8:b79c:2e20:e531]) (user=andreyknvl job=sendgmr) by 2002:a7b:c5c1:: with SMTP id n1mr4107578wmk.163.1612539577356; Fri, 05 Feb 2021 07:39:37 -0800 (PST) Date: Fri, 5 Feb 2021 16:39:09 +0100 In-Reply-To: Message-Id: <1cf400f36ab1fd3c83e7626c3797cb11ebf9ef7f.1612538932.git.andreyknvl@google.com> Mime-Version: 1.0 References: X-Mailer: git-send-email 2.30.0.365.g02bc693789-goog Subject: [PATCH v2 08/12] kasan, mm: optimize krealloc poisoning From: Andrey Konovalov To: Andrew Morton , Catalin Marinas , Vincenzo Frascino , Dmitry Vyukov , Alexander Potapenko , Marco Elver X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210205_103941_017489_DEAFCEA2 X-CRM114-Status: GOOD ( 20.43 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Branislav Rankov , Andrey Konovalov , Kevin Brodsky , Will Deacon , linux-kernel@vger.kernel.org, kasan-dev@googlegroups.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, Andrey Ryabinin , Peter Collingbourne , Evgenii Stepanov Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Currently, krealloc() always calls ksize(), which unpoisons the whole object including the redzone. This is inefficient, as kasan_krealloc() repoisons the redzone for objects that fit into the same buffer. This patch changes krealloc() instrumentation to use uninstrumented __ksize() that doesn't unpoison the memory. Instead, kasan_kreallos() is changed to unpoison the memory excluding the redzone. For objects that don't fit into the old allocation, this patch disables KASAN accessibility checks when copying memory into a new object instead of unpoisoning it. Signed-off-by: Andrey Konovalov --- mm/kasan/common.c | 12 ++++++++++-- mm/slab_common.c | 20 ++++++++++++++------ 2 files changed, 24 insertions(+), 8 deletions(-) diff --git a/mm/kasan/common.c b/mm/kasan/common.c index 7ea643f7e69c..a8a67dca5e55 100644 --- a/mm/kasan/common.c +++ b/mm/kasan/common.c @@ -476,7 +476,7 @@ static void *____kasan_kmalloc(struct kmem_cache *cache, const void *object, /* * The object has already been unpoisoned by kasan_slab_alloc() for - * kmalloc() or by ksize() for krealloc(). + * kmalloc() or by kasan_krealloc() for krealloc(). */ /* @@ -526,7 +526,7 @@ void * __must_check __kasan_kmalloc_large(const void *ptr, size_t size, /* * The object has already been unpoisoned by kasan_alloc_pages() for - * alloc_pages() or by ksize() for krealloc(). + * alloc_pages() or by kasan_krealloc() for krealloc(). */ /* @@ -554,8 +554,16 @@ void * __must_check __kasan_krealloc(const void *object, size_t size, gfp_t flag if (unlikely(object == ZERO_SIZE_PTR)) return (void *)object; + /* + * Unpoison the object's data. + * Part of it might already have been unpoisoned, but it's unknown + * how big that part is. + */ + kasan_unpoison(object, size); + page = virt_to_head_page(object); + /* Piggy-back on kmalloc() instrumentation to poison the redzone. */ if (unlikely(!PageSlab(page))) return __kasan_kmalloc_large(object, size, flags); else diff --git a/mm/slab_common.c b/mm/slab_common.c index dad70239b54c..60a2f49df6ce 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -1140,19 +1140,27 @@ static __always_inline void *__do_krealloc(const void *p, size_t new_size, void *ret; size_t ks; - if (likely(!ZERO_OR_NULL_PTR(p)) && !kasan_check_byte(p)) - return NULL; - - ks = ksize(p); + /* Don't use instrumented ksize to allow precise KASAN poisoning. */ + if (likely(!ZERO_OR_NULL_PTR(p))) { + if (!kasan_check_byte(p)) + return NULL; + ks = kfence_ksize(p) ?: __ksize(p); + } else + ks = 0; + /* If the object still fits, repoison it precisely. */ if (ks >= new_size) { p = kasan_krealloc((void *)p, new_size, flags); return (void *)p; } ret = kmalloc_track_caller(new_size, flags); - if (ret && p) - memcpy(ret, p, ks); + if (ret && p) { + /* Disable KASAN checks as the object's redzone is accessed. */ + kasan_disable_current(); + memcpy(ret, kasan_reset_tag(p), ks); + kasan_enable_current(); + } return ret; } From patchwork Fri Feb 5 15:39:10 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrey Konovalov X-Patchwork-Id: 12070363 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.1 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9D9F1C433E0 for ; Fri, 5 Feb 2021 15:41:53 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 2FE9E650EF for ; Fri, 5 Feb 2021 15:41:53 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 2FE9E650EF Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:To:From:Subject:References:Mime-Version:Message-Id: In-Reply-To:Date:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=p/ROfo+uszhnuPaSj7unJdEesqQEtCsYFqAyMVXRPuU=; b=lbsFPvgcPov1TjZkBXwiuKaAF WjSSYKDQ7yVKWnajn+/nbsDHOziGTXy5WUbvRBZ2XyBxfwuEo3HnFPAFJMNbDyEFIJuye1Q/C6R5r XouqvZpmZ7EIcBQu0vNA8njPVC76zK2Zcu0F1m4Of8F2FKBnszpNkickGhQo99ysK9btVHJOOxhC1 X3oND07EefXzV1HoD8XNNSEuQjE/7LUeNIKvglPP7rJDhtesGvPnOK5Hid1VZDph4UiDE2n58RpAq knZQ7GhAdugJ7jEi9jnzDUsGTu7DdVDG/MZYDW5TPavJpxFmr4V8Zg+OZXACP9N7afBoKe8pJIW51 MUf9CWfFQ==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1l83Dg-0008DA-Gc; Fri, 05 Feb 2021 15:40:28 +0000 Received: from mail-wm1-x349.google.com ([2a00:1450:4864:20::349]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1l83Cw-0007qv-TI for linux-arm-kernel@lists.infradead.org; Fri, 05 Feb 2021 15:39:45 +0000 Received: by mail-wm1-x349.google.com with SMTP id z25so5732917wml.0 for ; Fri, 05 Feb 2021 07:39:40 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=sdkgj896Qo+AFN+zvttnZapcO4bCa1e5ImK6MurFn1k=; b=UwU8BZglk6GLCdc+Y+3cPUnrMSIhuxzhkzQi7L+6RBq8L/Ja+DrNoAHGZ9tg8zuien 9wQ/oFCEdEAqzF9q/ziTAcoTLx7uq4pe214ZPEfMRvBi41RZfpfTYdNJuR+aC1x1zhTG j/jMay7GnbyuMPEuOHGBxtKo4KuqGPs53RjpG84LK6ZFkEla+I+5zy8xZ15HdGMOVJzn tMaxk4y8mtz/jU4/R4uxAaF98U15EJgDBEMCthRK/vIO9tZkvEjazL4EP2GW385cy5Rd yohmn2hhYmmpppVhingV+70vLqHCr6HaWpy2qXGFbNromUr4VMO4xKXOBPfNa/3ZOwdR OO6w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=sdkgj896Qo+AFN+zvttnZapcO4bCa1e5ImK6MurFn1k=; b=DhjS0jOcZLtXTbwQONcE3OmcSzr0SixIW+Y2h/+BIAWKXh7vvNx9hTsvJqpnSiNkKp D7wgI2ce8ZPv9X7DPrCOVR+u7qYItSHCeC6+LiPhKmAvDi9GafLZeMQnSEgWWk4OWRzP Qqxrf64d/ksy70UtszIson16cKP+2oaiczpymCByu1TgzB7bT56cw/+Gq25T/0toXN1T l4usoBM2yZtwD340PSL6aqLXSIJk40EBFL6cCfz9jPjBJrRTQRqVYLO+dLum1gpDLRXr zYE12TKOf6kvkSVBYNhONSI9oXekTx3Rs6Q6b/OXhSTzzUeuJgFSi/tKAec0AqCrqVNz Ds3A== X-Gm-Message-State: AOAM531C26ztO/H89B/SlIh0s2oQU3MZvjJ6cXIwxHIPL3OimtKWRq7h qrJk2kR5cII2R1wglpGfJ6n8AS0T/NclHOgu X-Google-Smtp-Source: ABdhPJxDbLmIELc7j7WQJZu+Bpk80w6bKCbvSm27wbQqmOHDlPvAbZr77jAT0AHFGmy1CYo2Fg0HMKHW3zixhotB X-Received: from andreyknvl3.muc.corp.google.com ([2a00:79e0:15:13:edb8:b79c:2e20:e531]) (user=andreyknvl job=sendgmr) by 2002:a05:600c:21d2:: with SMTP id x18mr4105351wmj.186.1612539579825; Fri, 05 Feb 2021 07:39:39 -0800 (PST) Date: Fri, 5 Feb 2021 16:39:10 +0100 In-Reply-To: Message-Id: <3917dc59e3dbf99b2929a1e20d41df0c5930e026.1612538932.git.andreyknvl@google.com> Mime-Version: 1.0 References: X-Mailer: git-send-email 2.30.0.365.g02bc693789-goog Subject: [PATCH v2 09/12] kasan: ensure poisoning size alignment From: Andrey Konovalov To: Andrew Morton , Catalin Marinas , Vincenzo Frascino , Dmitry Vyukov , Alexander Potapenko , Marco Elver X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210205_103943_015718_F788BB91 X-CRM114-Status: GOOD ( 18.39 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Branislav Rankov , Andrey Konovalov , Kevin Brodsky , Will Deacon , linux-kernel@vger.kernel.org, kasan-dev@googlegroups.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, Andrey Ryabinin , Peter Collingbourne , Evgenii Stepanov Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org A previous changes d99f6a10c161 ("kasan: don't round_up too much") attempted to simplify the code by adding a round_up(size) call into kasan_poison(). While this allows to have less round_up() calls around the code, this results in round_up() being called multiple times. This patch removes round_up() of size from kasan_poison() and ensures that all callers round_up() the size explicitly. This patch also adds WARN_ON() alignment checks for address and size to kasan_poison() and kasan_unpoison(). Reviewed-by: Marco Elver Signed-off-by: Andrey Konovalov --- mm/kasan/common.c | 9 ++++++--- mm/kasan/kasan.h | 33 ++++++++++++++++++++------------- mm/kasan/shadow.c | 37 ++++++++++++++++++++++--------------- 3 files changed, 48 insertions(+), 31 deletions(-) diff --git a/mm/kasan/common.c b/mm/kasan/common.c index a8a67dca5e55..7ffb1e6de2ef 100644 --- a/mm/kasan/common.c +++ b/mm/kasan/common.c @@ -261,7 +261,8 @@ void __kasan_unpoison_object_data(struct kmem_cache *cache, void *object) void __kasan_poison_object_data(struct kmem_cache *cache, void *object) { - kasan_poison(object, cache->object_size, KASAN_KMALLOC_REDZONE); + kasan_poison(object, round_up(cache->object_size, KASAN_GRANULE_SIZE), + KASAN_KMALLOC_REDZONE); } /* @@ -348,7 +349,8 @@ static bool ____kasan_slab_free(struct kmem_cache *cache, void *object, return true; } - kasan_poison(object, cache->object_size, KASAN_KMALLOC_FREE); + kasan_poison(object, round_up(cache->object_size, KASAN_GRANULE_SIZE), + KASAN_KMALLOC_FREE); if ((IS_ENABLED(CONFIG_KASAN_GENERIC) && !quarantine)) return false; @@ -490,7 +492,8 @@ static void *____kasan_kmalloc(struct kmem_cache *cache, const void *object, /* Poison the aligned part of the redzone. */ redzone_start = round_up((unsigned long)(object + size), KASAN_GRANULE_SIZE); - redzone_end = (unsigned long)object + cache->object_size; + redzone_end = round_up((unsigned long)(object + cache->object_size), + KASAN_GRANULE_SIZE); kasan_poison((void *)redzone_start, redzone_end - redzone_start, KASAN_KMALLOC_REDZONE); diff --git a/mm/kasan/kasan.h b/mm/kasan/kasan.h index 6a2882997f23..98f70ffc9e1c 100644 --- a/mm/kasan/kasan.h +++ b/mm/kasan/kasan.h @@ -321,30 +321,37 @@ static inline u8 kasan_random_tag(void) { return 0; } #ifdef CONFIG_KASAN_HW_TAGS -static inline void kasan_poison(const void *address, size_t size, u8 value) +static inline void kasan_poison(const void *addr, size_t size, u8 value) { - address = kasan_reset_tag(address); + addr = kasan_reset_tag(addr); /* Skip KFENCE memory if called explicitly outside of sl*b. */ - if (is_kfence_address(address)) + if (is_kfence_address(addr)) return; - hw_set_mem_tag_range((void *)address, - round_up(size, KASAN_GRANULE_SIZE), value); + if (WARN_ON((unsigned long)addr & KASAN_GRANULE_MASK)) + return; + if (WARN_ON(size & KASAN_GRANULE_MASK)) + return; + + hw_set_mem_tag_range((void *)addr, size, value); } -static inline void kasan_unpoison(const void *address, size_t size) +static inline void kasan_unpoison(const void *addr, size_t size) { - u8 tag = get_tag(address); + u8 tag = get_tag(addr); - address = kasan_reset_tag(address); + addr = kasan_reset_tag(addr); /* Skip KFENCE memory if called explicitly outside of sl*b. */ - if (is_kfence_address(address)) + if (is_kfence_address(addr)) return; - hw_set_mem_tag_range((void *)address, - round_up(size, KASAN_GRANULE_SIZE), tag); + if (WARN_ON((unsigned long)addr & KASAN_GRANULE_MASK)) + return; + size = round_up(size, KASAN_GRANULE_SIZE); + + hw_set_mem_tag_range((void *)addr, size, tag); } static inline bool kasan_byte_accessible(const void *addr) @@ -361,7 +368,7 @@ static inline bool kasan_byte_accessible(const void *addr) /** * kasan_poison - mark the memory range as unaccessible * @addr - range start address, must be aligned to KASAN_GRANULE_SIZE - * @size - range size + * @size - range size, must be aligned to KASAN_GRANULE_SIZE * @value - value that's written to metadata for the range * * The size gets aligned to KASAN_GRANULE_SIZE before marking the range. @@ -371,7 +378,7 @@ void kasan_poison(const void *addr, size_t size, u8 value); /** * kasan_unpoison - mark the memory range as accessible * @addr - range start address, must be aligned to KASAN_GRANULE_SIZE - * @size - range size + * @size - range size, can be unaligned * * For the tag-based modes, the @size gets aligned to KASAN_GRANULE_SIZE before * marking the range. diff --git a/mm/kasan/shadow.c b/mm/kasan/shadow.c index 1ed7817e4ee6..63f43443f5d7 100644 --- a/mm/kasan/shadow.c +++ b/mm/kasan/shadow.c @@ -69,7 +69,7 @@ void *memcpy(void *dest, const void *src, size_t len) return __memcpy(dest, src, len); } -void kasan_poison(const void *address, size_t size, u8 value) +void kasan_poison(const void *addr, size_t size, u8 value) { void *shadow_start, *shadow_end; @@ -78,55 +78,62 @@ void kasan_poison(const void *address, size_t size, u8 value) * some of the callers (e.g. kasan_poison_object_data) pass tagged * addresses to this function. */ - address = kasan_reset_tag(address); + addr = kasan_reset_tag(addr); /* Skip KFENCE memory if called explicitly outside of sl*b. */ - if (is_kfence_address(address)) + if (is_kfence_address(addr)) return; - size = round_up(size, KASAN_GRANULE_SIZE); - shadow_start = kasan_mem_to_shadow(address); - shadow_end = kasan_mem_to_shadow(address + size); + if (WARN_ON((unsigned long)addr & KASAN_GRANULE_MASK)) + return; + if (WARN_ON(size & KASAN_GRANULE_MASK)) + return; + + shadow_start = kasan_mem_to_shadow(addr); + shadow_end = kasan_mem_to_shadow(addr + size); __memset(shadow_start, value, shadow_end - shadow_start); } EXPORT_SYMBOL(kasan_poison); #ifdef CONFIG_KASAN_GENERIC -void kasan_poison_last_granule(const void *address, size_t size) +void kasan_poison_last_granule(const void *addr, size_t size) { if (size & KASAN_GRANULE_MASK) { - u8 *shadow = (u8 *)kasan_mem_to_shadow(address + size); + u8 *shadow = (u8 *)kasan_mem_to_shadow(addr + size); *shadow = size & KASAN_GRANULE_MASK; } } #endif -void kasan_unpoison(const void *address, size_t size) +void kasan_unpoison(const void *addr, size_t size) { - u8 tag = get_tag(address); + u8 tag = get_tag(addr); /* * Perform shadow offset calculation based on untagged address, as * some of the callers (e.g. kasan_unpoison_object_data) pass tagged * addresses to this function. */ - address = kasan_reset_tag(address); + addr = kasan_reset_tag(addr); /* * Skip KFENCE memory if called explicitly outside of sl*b. Also note * that calls to ksize(), where size is not a multiple of machine-word * size, would otherwise poison the invalid portion of the word. */ - if (is_kfence_address(address)) + if (is_kfence_address(addr)) + return; + + if (WARN_ON((unsigned long)addr & KASAN_GRANULE_MASK)) return; - /* Unpoison round_up(size, KASAN_GRANULE_SIZE) bytes. */ - kasan_poison(address, size, tag); + /* Unpoison all granules that cover the object. */ + kasan_poison(addr, round_up(size, KASAN_GRANULE_SIZE), tag); /* Partially poison the last granule for the generic mode. */ if (IS_ENABLED(CONFIG_KASAN_GENERIC)) - kasan_poison_last_granule(address, size); + kasan_poison_last_granule(addr, size); } #ifdef CONFIG_MEMORY_HOTPLUG From patchwork Fri Feb 5 15:39:11 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrey Konovalov X-Patchwork-Id: 12070393 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.1 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A7A2AC433E0 for ; Fri, 5 Feb 2021 15:41:59 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 3566C65093 for ; Fri, 5 Feb 2021 15:41:59 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 3566C65093 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:To:From:Subject:References:Mime-Version:Message-Id: In-Reply-To:Date:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=HOn/1IQHAREzl/B2F7OfIPtMGpHtVra201lDZej1nPU=; b=AShoTQYWikRkbTGr5ei2cMOWK MVUO9KbWpBpUlW+LNcbHjbBD574fTApZZrZS/z+cf89CNtqqu9jMT9sOjbijtRiRQo8EpNkBh5Ibq ptbOFQMNC++8Aim9IVMvXt4yAkWMISKX2qOHaL/zrSoZFSL71bzaWxX4Xrtfj+ZHch+j9fbTRvwaE HvlCljNRQlhvGAYlwW12BykOokCggI/6RQXOfdnVYIVEEX9JIZOKaS5jzYS0fqhVxaJxb5QdSq8Gb MfeoePEEREYo+kHO6WKLX2xxizvWGOoAojJ/ZPTe1pdPrOmy+vpXqbvppa4ycaYhEzUWDaqyn9g1h aUvFQR5/Q==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1l83Dk-0008FI-8z; Fri, 05 Feb 2021 15:40:32 +0000 Received: from mail-ej1-x649.google.com ([2a00:1450:4864:20::649]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1l83D0-0007rv-4q for linux-arm-kernel@lists.infradead.org; Fri, 05 Feb 2021 15:39:48 +0000 Received: by mail-ej1-x649.google.com with SMTP id ar1so3582569ejc.22 for ; Fri, 05 Feb 2021 07:39:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=PREZUncIqPTZICTNMYAhqfkQODhaZIF+0hFoo18paHE=; b=konF6v6BnN243u57JzN3b95VF77A0Qj1x+GlllUtiuD8VnfG6AL7oWEx03hrBfwRj8 aYE1+nYx+DzgmpLmNYFC4bk1hHc+2drX7wYZY0+sbzyQVai8e5ZWESQ57O8F1XO3Fv6O A80n2LSxNhH5p14Cz9+YH0Sh1eL3HyDaJn+FVtAuQU8Xbu8iNidaUe7LCekS99pT+1hH Oou+pgF6FEv+zEFiwfwff+OOi0SNFMv97nl9nABkE7clP+7820Ppjz1MdS8a+Ewetltv HG/rue7AEX/E9aasLQ/JlhnWPPmj1ImsCgoBRb7fVjcCHtXw3ovNJFB74SuoOphoobHx g20Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=PREZUncIqPTZICTNMYAhqfkQODhaZIF+0hFoo18paHE=; b=gXBBGtY/Ej43dRYCbwKWOg35ea2FMMFSfEFPw1HW90REpr6CDcoZBtpfTJJr+rsb4f DA8KhLATlz5JiVyZ6epcocvgzjhaS6P5M7ZOLYuqqmQHpzCr08+1VAtckW4nUL39eLKd lEaBOkEmB8FhebgYZ17n3YCHVVNtR/5hN1w+2xpGVs5m3eqy+PXweH0vxpMGGmI8OJ8+ /2Ycc7c0hT41mrvNMakudteX1khC/joviMvUJv1bbhtFV9BJzAWi4vhUnhQ36hCZhbHZ ooRo1xHiGe4V8kV8K/VNMwg+gWcyvVmPvbs68V5xxDIXd+ZvSajY6rPlaowIlFpu3Gdp HLeQ== X-Gm-Message-State: AOAM532mg6vNBM6RARI7XbSWTsAcyef3C2WQYogf+cJcCsH6/BFC+gaj VPznYaXI1oABNBKqk7YAMZP3qMLFwlCoa297 X-Google-Smtp-Source: ABdhPJxLyOBedtBSH56Km8M+KSKGSY+TJY33YcVxBP++XHc1sb/s8ursYOVubg9JIcGS4jhcauoOBzI04W3jHs91 X-Received: from andreyknvl3.muc.corp.google.com ([2a00:79e0:15:13:edb8:b79c:2e20:e531]) (user=andreyknvl job=sendgmr) by 2002:a17:906:8611:: with SMTP id o17mr4584420ejx.145.1612539582294; Fri, 05 Feb 2021 07:39:42 -0800 (PST) Date: Fri, 5 Feb 2021 16:39:11 +0100 In-Reply-To: Message-Id: Mime-Version: 1.0 References: X-Mailer: git-send-email 2.30.0.365.g02bc693789-goog Subject: [PATCH v2 10/12] arm64: kasan: simplify and inline MTE functions From: Andrey Konovalov To: Andrew Morton , Catalin Marinas , Vincenzo Frascino , Dmitry Vyukov , Alexander Potapenko , Marco Elver X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210205_103946_255945_04BA3ED1 X-CRM114-Status: GOOD ( 20.36 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Branislav Rankov , Andrey Konovalov , Kevin Brodsky , Will Deacon , linux-kernel@vger.kernel.org, kasan-dev@googlegroups.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, Andrey Ryabinin , Peter Collingbourne , Evgenii Stepanov Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org This change provides a simpler implementation of mte_get_mem_tag(), mte_get_random_tag(), and mte_set_mem_tag_range(). Simplifications include removing system_supports_mte() checks as these functions are onlye called from KASAN runtime that had already checked system_supports_mte(). Besides that, size and address alignment checks are removed from mte_set_mem_tag_range(), as KASAN now does those. This change also moves these functions into the asm/mte-kasan.h header and implements mte_set_mem_tag_range() via inline assembly to avoid unnecessary functions calls. Reviewed-by: Catalin Marinas Co-developed-by: Vincenzo Frascino Signed-off-by: Vincenzo Frascino Signed-off-by: Andrey Konovalov --- arch/arm64/include/asm/cache.h | 1 - arch/arm64/include/asm/kasan.h | 1 + arch/arm64/include/asm/mte-def.h | 2 + arch/arm64/include/asm/mte-kasan.h | 65 ++++++++++++++++++++++++++---- arch/arm64/include/asm/mte.h | 2 - arch/arm64/kernel/mte.c | 46 --------------------- arch/arm64/lib/mte.S | 16 -------- 7 files changed, 60 insertions(+), 73 deletions(-) diff --git a/arch/arm64/include/asm/cache.h b/arch/arm64/include/asm/cache.h index 77cbbe3625f2..a074459f8f2f 100644 --- a/arch/arm64/include/asm/cache.h +++ b/arch/arm64/include/asm/cache.h @@ -6,7 +6,6 @@ #define __ASM_CACHE_H #include -#include #define CTR_L1IP_SHIFT 14 #define CTR_L1IP_MASK 3 diff --git a/arch/arm64/include/asm/kasan.h b/arch/arm64/include/asm/kasan.h index 0aaf9044cd6a..12d5f47f7dbe 100644 --- a/arch/arm64/include/asm/kasan.h +++ b/arch/arm64/include/asm/kasan.h @@ -6,6 +6,7 @@ #include #include +#include #include #define arch_kasan_set_tag(addr, tag) __tag_set(addr, tag) diff --git a/arch/arm64/include/asm/mte-def.h b/arch/arm64/include/asm/mte-def.h index 2d73a1612f09..cf241b0f0a42 100644 --- a/arch/arm64/include/asm/mte-def.h +++ b/arch/arm64/include/asm/mte-def.h @@ -11,4 +11,6 @@ #define MTE_TAG_SIZE 4 #define MTE_TAG_MASK GENMASK((MTE_TAG_SHIFT + (MTE_TAG_SIZE - 1)), MTE_TAG_SHIFT) +#define __MTE_PREAMBLE ARM64_ASM_PREAMBLE ".arch_extension memtag\n" + #endif /* __ASM_MTE_DEF_H */ diff --git a/arch/arm64/include/asm/mte-kasan.h b/arch/arm64/include/asm/mte-kasan.h index 8ad981069afb..13be3afc37ac 100644 --- a/arch/arm64/include/asm/mte-kasan.h +++ b/arch/arm64/include/asm/mte-kasan.h @@ -11,11 +11,14 @@ #include +#ifdef CONFIG_ARM64_MTE + /* - * The functions below are meant to be used only for the - * KASAN_HW_TAGS interface defined in asm/memory.h. + * These functions are meant to be only used from KASAN runtime through + * the arch_*() interface defined in asm/memory.h. + * These functions don't include system_supports_mte() checks, + * as KASAN only calls them when MTE is supported and enabled. */ -#ifdef CONFIG_ARM64_MTE static inline u8 mte_get_ptr_tag(void *ptr) { @@ -25,9 +28,54 @@ static inline u8 mte_get_ptr_tag(void *ptr) return tag; } -u8 mte_get_mem_tag(void *addr); -u8 mte_get_random_tag(void); -void *mte_set_mem_tag_range(void *addr, size_t size, u8 tag); +/* Get allocation tag for the address. */ +static inline u8 mte_get_mem_tag(void *addr) +{ + asm(__MTE_PREAMBLE "ldg %0, [%0]" + : "+r" (addr)); + + return mte_get_ptr_tag(addr); +} + +/* Generate a random tag. */ +static inline u8 mte_get_random_tag(void) +{ + void *addr; + + asm(__MTE_PREAMBLE "irg %0, %0" + : "+r" (addr)); + + return mte_get_ptr_tag(addr); +} + +/* + * Assign allocation tags for a region of memory based on the pointer tag. + * Note: The address must be non-NULL and MTE_GRANULE_SIZE aligned and + * size must be non-zero and MTE_GRANULE_SIZE aligned. + */ +static inline void mte_set_mem_tag_range(void *addr, size_t size, u8 tag) +{ + u64 curr, end; + + if (!size) + return; + + curr = (u64)__tag_set(addr, tag); + end = curr + size; + + do { + /* + * 'asm volatile' is required to prevent the compiler to move + * the statement outside of the loop. + */ + asm volatile(__MTE_PREAMBLE "stg %0, [%0]" + : + : "r" (curr) + : "memory"); + + curr += MTE_GRANULE_SIZE; + } while (curr != end); +} void mte_enable_kernel_sync(void); void mte_enable_kernel_async(void); @@ -47,13 +95,14 @@ static inline u8 mte_get_mem_tag(void *addr) { return 0xFF; } + static inline u8 mte_get_random_tag(void) { return 0xFF; } -static inline void *mte_set_mem_tag_range(void *addr, size_t size, u8 tag) + +static inline void mte_set_mem_tag_range(void *addr, size_t size, u8 tag) { - return addr; } static inline void mte_enable_kernel_sync(void) diff --git a/arch/arm64/include/asm/mte.h b/arch/arm64/include/asm/mte.h index 237bb2f7309d..43169b978cd3 100644 --- a/arch/arm64/include/asm/mte.h +++ b/arch/arm64/include/asm/mte.h @@ -8,8 +8,6 @@ #include #include -#define __MTE_PREAMBLE ARM64_ASM_PREAMBLE ".arch_extension memtag\n" - #ifndef __ASSEMBLY__ #include diff --git a/arch/arm64/kernel/mte.c b/arch/arm64/kernel/mte.c index 7763ac1f2917..8b27b70e1aac 100644 --- a/arch/arm64/kernel/mte.c +++ b/arch/arm64/kernel/mte.c @@ -19,7 +19,6 @@ #include #include #include -#include #include #include @@ -88,51 +87,6 @@ int memcmp_pages(struct page *page1, struct page *page2) return ret; } -u8 mte_get_mem_tag(void *addr) -{ - if (!system_supports_mte()) - return 0xFF; - - asm(__MTE_PREAMBLE "ldg %0, [%0]" - : "+r" (addr)); - - return mte_get_ptr_tag(addr); -} - -u8 mte_get_random_tag(void) -{ - void *addr; - - if (!system_supports_mte()) - return 0xFF; - - asm(__MTE_PREAMBLE "irg %0, %0" - : "+r" (addr)); - - return mte_get_ptr_tag(addr); -} - -void *mte_set_mem_tag_range(void *addr, size_t size, u8 tag) -{ - void *ptr = addr; - - if ((!system_supports_mte()) || (size == 0)) - return addr; - - /* Make sure that size is MTE granule aligned. */ - WARN_ON(size & (MTE_GRANULE_SIZE - 1)); - - /* Make sure that the address is MTE granule aligned. */ - WARN_ON((u64)addr & (MTE_GRANULE_SIZE - 1)); - - tag = 0xF0 | tag; - ptr = (void *)__tag_set(ptr, tag); - - mte_assign_mem_tag_range(ptr, size); - - return ptr; -} - void mte_init_tags(u64 max_tag) { static bool gcr_kernel_excl_initialized; diff --git a/arch/arm64/lib/mte.S b/arch/arm64/lib/mte.S index 9e1a12e10053..351537c12f36 100644 --- a/arch/arm64/lib/mte.S +++ b/arch/arm64/lib/mte.S @@ -149,19 +149,3 @@ SYM_FUNC_START(mte_restore_page_tags) ret SYM_FUNC_END(mte_restore_page_tags) - -/* - * Assign allocation tags for a region of memory based on the pointer tag - * x0 - source pointer - * x1 - size - * - * Note: The address must be non-NULL and MTE_GRANULE_SIZE aligned and - * size must be non-zero and MTE_GRANULE_SIZE aligned. - */ -SYM_FUNC_START(mte_assign_mem_tag_range) -1: stg x0, [x0] - add x0, x0, #MTE_GRANULE_SIZE - subs x1, x1, #MTE_GRANULE_SIZE - b.gt 1b - ret -SYM_FUNC_END(mte_assign_mem_tag_range) From patchwork Fri Feb 5 15:39:12 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrey Konovalov X-Patchwork-Id: 12070395 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.1 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9196BC433E0 for ; Fri, 5 Feb 2021 15:42:04 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 342D1650EC for ; Fri, 5 Feb 2021 15:42:04 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 342D1650EC Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:To:From:Subject:References:Mime-Version:Message-Id: In-Reply-To:Date:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=12T+l7VgFyOXn4nh8YwxsMN6nWgZ9hn1KmBNeBrNQC0=; b=PlwkrGY8L0kSxoM77QJE8ydGl evqWoKgkK80cwCvQz8/0e3BKKxWx3qx5zJWFYgRykhdPcMR0EwjYBDXmXjibBxgnla6i4i65Iak+T 56Hny54i+njRsMiS9sBnpTJcrbrbNVbYT8cu4Z9RuRhxQ7+ot/mNHfMOeTcUlzt1VHQDfk7mV9MjS M9VKP3LBUl9Mj/uoAWtQI+BQR+UYPOyC+qzbFNz1u02DdK3o/GVP1H4Hh31tyZCBHq2QaHLkwni1b 2DX7zsiaj5zelMgFPvKmuCzOohOjdnR/079knB9j5xSV6VTyloqizl7VWWcnwfziciPCx67jEpqxL fwRAnIZFg==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1l83Ds-0008Kf-Nm; Fri, 05 Feb 2021 15:40:40 +0000 Received: from mail-wr1-x44a.google.com ([2a00:1450:4864:20::44a]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1l83D2-0007t1-2t for linux-arm-kernel@lists.infradead.org; Fri, 05 Feb 2021 15:39:50 +0000 Received: by mail-wr1-x44a.google.com with SMTP id d7so5582008wri.23 for ; Fri, 05 Feb 2021 07:39:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=r9X5DpdR9qKQzOGqj5YPMHvNzUxwchyNv1XRoONrpMQ=; b=GinXCbIKDRoOgLUpKmgxzl260l9tsG2LG+b4gApdVcTyZiiVgLI843m0SQIZzi7Qae NfAp0Io+LgzQcCk69t45hFOJaqjI9KbQccxbpOgaJcJcsZgClEf6fOOjoLnqeysqDHzq FFGcssvQchbEYQAOQYIjmmCgsqDh8MRge4yFNbKPar14MoUKxkyYfLYU4IcGP5q5DCJ/ uq6GvLUb93p8KgHDIgo2/wB6GGYBaMGnvBP6ZxWTkuxb9g0ZXn0bOOLgzYqsobzl23WX ypCO6WqihQFQ1clhxfXOzKJRjdZ8iu8uQBbv8Tp6n8rPnb3bq8lPznSVsroHSUulMSsU BQGQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=r9X5DpdR9qKQzOGqj5YPMHvNzUxwchyNv1XRoONrpMQ=; b=WovxnG9Wf8dqux6kzFjbLkRhs6e7h8USXuI/XMl43mjlswC35Mpk9Axs0hh1utQ3Tk 05cPaBpFSaDMNPAn1ncTLOR9kJsdpth0pjGFfBc9X+OVmH09bLmhwFeMwQpnhWHa63O0 HDk7qcLy3On050q4Mxum5cgOi381P8adpqVLkFfXh8oLJ5kDKat5WhSS7JwP2kCoNZjp 9mFTSGpStF9iDcc5Nv+JK688p90ednNtJZwTZctfHKfx+aAnZdeAKPDzRSkssHgocI+V ZhV/V0XWHlCA2ahrdKxONhpE6vKe39NNPdtFeyiE9gDT2lXOgec/vLxsL11fCW1ftRYz O7FA== X-Gm-Message-State: AOAM5336VLJQXNn3jzM7ebyu6tv0u4gW0rFTbASAqWLAyzMXgH/JvMts AiOHiaw853ao6XGL5dfX4CMcubLKpN51KRD9 X-Google-Smtp-Source: ABdhPJwGux2zXHlarundz7K3SOkAa4ybczg07jIuC0920I9YRGfTqF2RXx4UClbc5pUiakH6JEZV6yYw43coZmqv X-Received: from andreyknvl3.muc.corp.google.com ([2a00:79e0:15:13:edb8:b79c:2e20:e531]) (user=andreyknvl job=sendgmr) by 2002:a1c:730a:: with SMTP id d10mr4025957wmb.53.1612539584906; Fri, 05 Feb 2021 07:39:44 -0800 (PST) Date: Fri, 5 Feb 2021 16:39:12 +0100 In-Reply-To: Message-Id: <182acaae3ed39231e6c3132c6bc7fc6b08ef003f.1612538932.git.andreyknvl@google.com> Mime-Version: 1.0 References: X-Mailer: git-send-email 2.30.0.365.g02bc693789-goog Subject: [PATCH v2 11/12] kasan: inline HW_TAGS helper functions From: Andrey Konovalov To: Andrew Morton , Catalin Marinas , Vincenzo Frascino , Dmitry Vyukov , Alexander Potapenko , Marco Elver X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210205_103948_167624_F005F58E X-CRM114-Status: GOOD ( 11.12 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Branislav Rankov , Andrey Konovalov , Kevin Brodsky , Will Deacon , linux-kernel@vger.kernel.org, kasan-dev@googlegroups.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, Andrey Ryabinin , Peter Collingbourne , Evgenii Stepanov Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Mark all static functions in common.c and kasan.h that are used for hardware tag-based KASAN as inline to avoid unnecessary function calls. Signed-off-by: Andrey Konovalov --- mm/kasan/common.c | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/mm/kasan/common.c b/mm/kasan/common.c index 7ffb1e6de2ef..7b53291dafa1 100644 --- a/mm/kasan/common.c +++ b/mm/kasan/common.c @@ -279,7 +279,8 @@ void __kasan_poison_object_data(struct kmem_cache *cache, void *object) * based on objects indexes, so that objects that are next to each other * get different tags. */ -static u8 assign_tag(struct kmem_cache *cache, const void *object, bool init) +static inline u8 assign_tag(struct kmem_cache *cache, + const void *object, bool init) { if (IS_ENABLED(CONFIG_KASAN_GENERIC)) return 0xff; @@ -321,8 +322,8 @@ void * __must_check __kasan_init_slab_obj(struct kmem_cache *cache, return (void *)object; } -static bool ____kasan_slab_free(struct kmem_cache *cache, void *object, - unsigned long ip, bool quarantine) +static inline bool ____kasan_slab_free(struct kmem_cache *cache, + void *object, unsigned long ip, bool quarantine) { u8 tag; void *tagged_object; @@ -366,7 +367,7 @@ bool __kasan_slab_free(struct kmem_cache *cache, void *object, unsigned long ip) return ____kasan_slab_free(cache, object, ip, true); } -static bool ____kasan_kfree_large(void *ptr, unsigned long ip) +static inline bool ____kasan_kfree_large(void *ptr, unsigned long ip) { if (ptr != page_address(virt_to_head_page(ptr))) { kasan_report_invalid_free(ptr, ip); @@ -461,8 +462,8 @@ void * __must_check __kasan_slab_alloc(struct kmem_cache *cache, return tagged_object; } -static void *____kasan_kmalloc(struct kmem_cache *cache, const void *object, - size_t size, gfp_t flags) +static inline void *____kasan_kmalloc(struct kmem_cache *cache, + const void *object, size_t size, gfp_t flags) { unsigned long redzone_start; unsigned long redzone_end; From patchwork Fri Feb 5 15:39:13 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrey Konovalov X-Patchwork-Id: 12070397 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.1 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4BD40C433DB for ; Fri, 5 Feb 2021 15:42:11 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id DCEDE650EC for ; Fri, 5 Feb 2021 15:42:10 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org DCEDE650EC Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:To:From:Subject:References:Mime-Version:Message-Id: In-Reply-To:Date:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=UdpYc+4ZJjOvyS1G/qhiEkiFjScSKb8x4fH9yfxfgKA=; b=rt5236vrG3RqGKMtX/d/c38cA +DqEsBBqNv3MoVQCgCJtw6XXxt8El1M20f8WFCpvTFUyNJWm7ph4d3XDmpw4AEuBJXMT4Sl/C+eYV IcJGrQSSaqc06D9Xck+oqGitPmi/xFq+AZ2IN6ovfC9f805y3OY/Mo+uJIrktELU5ncCuK3EZ+4O1 WCTmkYqYX1H9y4aJTbVFBbNnpkxpnkOb7EehBBnKivkqX5nYCtuMlAeB9uTbe02rBuLB4PJ8pjWaH nxKo0gvNz0tYXjXdNKkGZ2uL/zBMmzO8oiaRHzxtgvGlZg6OEy+rsJFsWOa/iLUa9NSuJljQ+84CC Cg/1U0nyA==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1l83E6-0008S4-Fo; Fri, 05 Feb 2021 15:40:54 +0000 Received: from mail-yb1-xb4a.google.com ([2607:f8b0:4864:20::b4a]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1l83D6-0007uH-1G for linux-arm-kernel@lists.infradead.org; Fri, 05 Feb 2021 15:39:53 +0000 Received: by mail-yb1-xb4a.google.com with SMTP id i2so7488716ybl.16 for ; Fri, 05 Feb 2021 07:39:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=jIaeLYAzRLuX+Bh9Y7uHrxQjgj3fqKxlpOgYc1EJ990=; b=gxafKVSjEdKfMi+epPlKRwmPrE34Nz4U8OjnRmWLqqm/8o7BQY8xx3KC3dFp6fPjPM X8HZGZ/5/LtXi1oLUzN0En3X9lftRUQVYepeq7RfrreRHKD+M+CgmYBRIJ8FpM+UXumV i6aWJhkv95qOmweLBQxRdIOcKg/8FbF/eBHX9Nor/DJ/znPBx2YZn9KDxaX2IacPF/SX x+Bb1Gmu+ncxDnZCkIpsy4Wldr5IUHQSvvG098nSFiYpcXLPRwA7OcKMnHExKNt+p0hU 6FvKas1BFmtYRhe6iDn9Q5RBGWWmZ6gZiyVg4yIVYUC1Y1IDOw9TJ0q9t8wyb9Sy536H vgGQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=jIaeLYAzRLuX+Bh9Y7uHrxQjgj3fqKxlpOgYc1EJ990=; b=ZGeA74IV5vxWJE+yjwd7MQ4cyo4WiM3mrBZ3I8GHiIZ12twFIEK9SKT9Eudch/zZ7k rVw4Ms/Sl9gdNoo43nrcJtlp4nqzZvSxmRe8TCSu+KBltjJa+Nr3IIHTgMZz4Ps6ns4K y02GMQpaae5vLumf8+uawcNLzRBjX4o8kGXj859Yh8SMZ49TCoHnDKiiXjmADnYV4s+d YZz8EBysJu+pv7F+owBw9XYuYgZU6yExVyPVlKYkelOM4x/4IKIxxxqa8lFL/mTBBMoZ 2DaFwrWRHKS+UU6pHYdCLtchwymKuTYW6hcfF1/gjflnumc4IjtBDSHJczrOsYQ555hd y7Uw== X-Gm-Message-State: AOAM531f2rcxUShSXx9s5spAUQGCTjbqOCiBoAhmw9wNd5VTEVG0BOuZ LWR4lY5GZPOMi14p7uvoD2o7e1ZngbSyPLG3 X-Google-Smtp-Source: ABdhPJywuIcjamxfCRpOr02hXFSCeN+u8fLAfBL1fEi3HK/aV7eWDTn1xkNVkVPGJ9mUhxsCfkPWbZAwBK/wAylo X-Received: from andreyknvl3.muc.corp.google.com ([2a00:79e0:15:13:edb8:b79c:2e20:e531]) (user=andreyknvl job=sendgmr) by 2002:a25:31c3:: with SMTP id x186mr6809008ybx.500.1612539587288; Fri, 05 Feb 2021 07:39:47 -0800 (PST) Date: Fri, 5 Feb 2021 16:39:13 +0100 In-Reply-To: Message-Id: <068ab897dc5e73d4a8d7c919b84339216c2f99da.1612538932.git.andreyknvl@google.com> Mime-Version: 1.0 References: X-Mailer: git-send-email 2.30.0.365.g02bc693789-goog Subject: [PATCH v2 12/12] arm64: kasan: export MTE symbols for KASAN tests From: Andrey Konovalov To: Andrew Morton , Catalin Marinas , Vincenzo Frascino , Dmitry Vyukov , Alexander Potapenko , Marco Elver X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210205_103952_160725_AF572947 X-CRM114-Status: GOOD ( 11.33 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Branislav Rankov , Andrey Konovalov , Kevin Brodsky , Will Deacon , linux-kernel@vger.kernel.org, kasan-dev@googlegroups.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, Andrey Ryabinin , Peter Collingbourne , Evgenii Stepanov Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Export mte_enable_kernel_sync() and mte_set_report_once() to fix: ERROR: modpost: "mte_enable_kernel_sync" [lib/test_kasan.ko] undefined! ERROR: modpost: "mte_set_report_once" [lib/test_kasan.ko] undefined! Reviewed-by: Catalin Marinas Signed-off-by: Andrey Konovalov --- arch/arm64/kernel/mte.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/arch/arm64/kernel/mte.c b/arch/arm64/kernel/mte.c index 8b27b70e1aac..fc753fb77940 100644 --- a/arch/arm64/kernel/mte.c +++ b/arch/arm64/kernel/mte.c @@ -120,6 +120,7 @@ void mte_enable_kernel_sync(void) { __mte_enable_kernel("synchronous", SCTLR_ELx_TCF_SYNC); } +EXPORT_SYMBOL_GPL(mte_enable_kernel_sync); void mte_enable_kernel_async(void) { @@ -130,6 +131,7 @@ void mte_set_report_once(bool state) { WRITE_ONCE(report_fault_once, state); } +EXPORT_SYMBOL_GPL(mte_set_report_once); bool mte_report_once(void) {