From patchwork Tue Sep 21 10:10:12 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marco Elver X-Patchwork-Id: 12507485 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-26.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 23B73C433FE for ; Tue, 21 Sep 2021 10:10:29 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id B556061100 for ; Tue, 21 Sep 2021 10:10:28 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org B556061100 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id 4BAEC6B0072; Tue, 21 Sep 2021 06:10:28 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 428946B0073; Tue, 21 Sep 2021 06:10:28 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 22BA1900002; Tue, 21 Sep 2021 06:10:28 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0138.hostedemail.com [216.40.44.138]) by kanga.kvack.org (Postfix) with ESMTP id 0CE246B0072 for ; Tue, 21 Sep 2021 06:10:28 -0400 (EDT) Received: from smtpin01.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id BC6C72CFE4 for ; Tue, 21 Sep 2021 10:10:27 +0000 (UTC) X-FDA: 78611160894.01.74C475D Received: from mail-wr1-f73.google.com (mail-wr1-f73.google.com [209.85.221.73]) by imf13.hostedemail.com (Postfix) with ESMTP id 817EA10B9E50 for ; Tue, 21 Sep 2021 10:10:27 +0000 (UTC) Received: by mail-wr1-f73.google.com with SMTP id r15-20020adfce8f000000b0015df1098ccbso8398256wrn.4 for ; Tue, 21 Sep 2021 03:10:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=4f6yUJ+4Yot7zlWdOonsXRBu3J0/vvau8TCpbcwqOZE=; b=FhrffzQuuvwmcqfZInJt7xW7xX09gltSG0J7M3O03dANv0GcgQW9opGQFiwzomyoHU iAWtFimWyOqrk1dBiqwcLrH5F302D9FtWfW4BT1nkjgKvovvewNaaWNayowTUh1MCjbR /kuw9LPqZWmhLsryAUecaX1Zs1ubY6C5tLNsB2rrV+8Dowzmz1sM4tvxXnUMtXTaSSRQ uzewm2xXpyaegumJhsKYghqUCiWooIJbPqrtqy5oJrE0F0nozAdbvdNdvdMPkf+ECgNI NtakRX6VkT7rP0nSVRo+cgxE4kmfvpi+I6Nepe1o7FUgVoWT1NWBQV/D6+l8b0wVuloi DwMw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=4f6yUJ+4Yot7zlWdOonsXRBu3J0/vvau8TCpbcwqOZE=; b=Y9wH+3tFDgl+wj4XuOFwr4jGlT7CxES2xQ9b/QrOwFjlG/CLgbc+Xz7L1Jc1gCkhlV g3h449fJUFsE4Vvu6cOdHjBQuVrknCodNnmePHqB4U6oHerdfW2MSGE53IERAyQa4mP8 hdSzWER8weUl/pbVN3wI8yvybPf8Pxp2OoqnJEl2Yu9AYF1XlYtajnQKlU7UutunHtcC 8jOjOSevgbfJRTMbQ9wiFur3NMnInfE2gVJ+daIUxKnCttEk64aPemqQ1PoIuGQk3aIl fOhXhxAjz9takkOfMO7th+OPMLAqU7f7eHn35M/o5NnUclL3Kg7o4R5UeN90yp8wgQpV AhpA== X-Gm-Message-State: AOAM530Cjv5nZ/u5QuQcLNNd9gBnYFGENUENuqhEM6/5ODRJdmzcF4/h SDrjgspV+n4+uu06C7wVUe6RpKSENQ== X-Google-Smtp-Source: ABdhPJwpp8di3cIrkok5g79tnAJ+pYtjLSV2AAjX0dc2ZYWN2XZib61LEjP22aKYvA6CpSXzfmZn1yvNJQ== X-Received: from elver.muc.corp.google.com ([2a00:79e0:15:13:dd03:c280:4625:60db]) (user=elver job=sendgmr) by 2002:a05:600c:4f55:: with SMTP id m21mr3560923wmq.149.1632219026385; Tue, 21 Sep 2021 03:10:26 -0700 (PDT) Date: Tue, 21 Sep 2021 12:10:12 +0200 In-Reply-To: <20210921101014.1938382-1-elver@google.com> Message-Id: <20210921101014.1938382-3-elver@google.com> Mime-Version: 1.0 References: <20210921101014.1938382-1-elver@google.com> X-Mailer: git-send-email 2.33.0.464.g1972c5931b-goog Subject: [PATCH v2 3/5] kfence: move saving stack trace of allocations into __kfence_alloc() From: Marco Elver To: elver@google.com, Andrew Morton Cc: Alexander Potapenko , Dmitry Vyukov , Jann Horn , Aleksandr Nogikh , Taras Madan , linux-kernel@vger.kernel.org, linux-mm@kvack.org, kasan-dev@googlegroups.com X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: 817EA10B9E50 Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=FhrffzQu; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf13.hostedemail.com: domain of 3kq9JYQUKCCkJQaJWLTTLQJ.HTRQNSZc-RRPaFHP.TWL@flex--elver.bounces.google.com designates 209.85.221.73 as permitted sender) smtp.mailfrom=3kq9JYQUKCCkJQaJWLTTLQJ.HTRQNSZc-RRPaFHP.TWL@flex--elver.bounces.google.com X-Stat-Signature: gzcdfypw8thjgamjuw6an5jdsb6mtmnd X-HE-Tag: 1632219027-524397 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Move the saving of the stack trace of allocations into __kfence_alloc(), so that the stack entries array can be used outside of kfence_guarded_alloc() and we avoid potentially unwinding the stack multiple times. Signed-off-by: Marco Elver Reviewed-by: Dmitry Vyukov --- v2: * New patch. --- mm/kfence/core.c | 35 ++++++++++++++++++++++++----------- 1 file changed, 24 insertions(+), 11 deletions(-) diff --git a/mm/kfence/core.c b/mm/kfence/core.c index 249d75b7e5ee..db01814f8ff0 100644 --- a/mm/kfence/core.c +++ b/mm/kfence/core.c @@ -187,19 +187,26 @@ static inline unsigned long metadata_to_pageaddr(const struct kfence_metadata *m * Update the object's metadata state, including updating the alloc/free stacks * depending on the state transition. */ -static noinline void metadata_update_state(struct kfence_metadata *meta, - enum kfence_object_state next) +static noinline void +metadata_update_state(struct kfence_metadata *meta, enum kfence_object_state next, + unsigned long *stack_entries, size_t num_stack_entries) { struct kfence_track *track = next == KFENCE_OBJECT_FREED ? &meta->free_track : &meta->alloc_track; lockdep_assert_held(&meta->lock); - /* - * Skip over 1 (this) functions; noinline ensures we do not accidentally - * skip over the caller by never inlining. - */ - track->num_stack_entries = stack_trace_save(track->stack_entries, KFENCE_STACK_DEPTH, 1); + if (stack_entries) { + memcpy(track->stack_entries, stack_entries, + num_stack_entries * sizeof(stack_entries[0])); + } else { + /* + * Skip over 1 (this) functions; noinline ensures we do not + * accidentally skip over the caller by never inlining. + */ + num_stack_entries = stack_trace_save(track->stack_entries, KFENCE_STACK_DEPTH, 1); + } + track->num_stack_entries = num_stack_entries; track->pid = task_pid_nr(current); track->cpu = raw_smp_processor_id(); track->ts_nsec = local_clock(); /* Same source as printk timestamps. */ @@ -261,7 +268,8 @@ static __always_inline void for_each_canary(const struct kfence_metadata *meta, } } -static void *kfence_guarded_alloc(struct kmem_cache *cache, size_t size, gfp_t gfp) +static void *kfence_guarded_alloc(struct kmem_cache *cache, size_t size, gfp_t gfp, + unsigned long *stack_entries, size_t num_stack_entries) { struct kfence_metadata *meta = NULL; unsigned long flags; @@ -320,7 +328,7 @@ static void *kfence_guarded_alloc(struct kmem_cache *cache, size_t size, gfp_t g addr = (void *)meta->addr; /* Update remaining metadata. */ - metadata_update_state(meta, KFENCE_OBJECT_ALLOCATED); + metadata_update_state(meta, KFENCE_OBJECT_ALLOCATED, stack_entries, num_stack_entries); /* Pairs with READ_ONCE() in kfence_shutdown_cache(). */ WRITE_ONCE(meta->cache, cache); meta->size = size; @@ -400,7 +408,7 @@ static void kfence_guarded_free(void *addr, struct kfence_metadata *meta, bool z memzero_explicit(addr, meta->size); /* Mark the object as freed. */ - metadata_update_state(meta, KFENCE_OBJECT_FREED); + metadata_update_state(meta, KFENCE_OBJECT_FREED, NULL, 0); raw_spin_unlock_irqrestore(&meta->lock, flags); @@ -742,6 +750,9 @@ void kfence_shutdown_cache(struct kmem_cache *s) void *__kfence_alloc(struct kmem_cache *s, size_t size, gfp_t flags) { + unsigned long stack_entries[KFENCE_STACK_DEPTH]; + size_t num_stack_entries; + /* * Perform size check before switching kfence_allocation_gate, so that * we don't disable KFENCE without making an allocation. @@ -786,7 +797,9 @@ void *__kfence_alloc(struct kmem_cache *s, size_t size, gfp_t flags) if (!READ_ONCE(kfence_enabled)) return NULL; - return kfence_guarded_alloc(s, size, flags); + num_stack_entries = stack_trace_save(stack_entries, KFENCE_STACK_DEPTH, 0); + + return kfence_guarded_alloc(s, size, flags, stack_entries, num_stack_entries); } size_t kfence_ksize(const void *addr)