From patchwork Mon Jan 11 09:15:43 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Marco Elver X-Patchwork-Id: 12010097 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6BEF6C433DB for ; Mon, 11 Jan 2021 09:15:56 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id D589520656 for ; Mon, 11 Jan 2021 09:15:55 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org D589520656 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id F41C88D0023; Mon, 11 Jan 2021 04:15:54 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id EEBC18D0020; Mon, 11 Jan 2021 04:15:54 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D8CF58D0023; Mon, 11 Jan 2021 04:15:54 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0085.hostedemail.com [216.40.44.85]) by kanga.kvack.org (Postfix) with ESMTP id B9EBA8D0020 for ; Mon, 11 Jan 2021 04:15:54 -0500 (EST) Received: from smtpin13.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 748252461 for ; Mon, 11 Jan 2021 09:15:54 +0000 (UTC) X-FDA: 77692937028.13.burn88_1c17ceb2750b Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin13.hostedemail.com (Postfix) with ESMTP id 569E218140B67 for ; Mon, 11 Jan 2021 09:15:54 +0000 (UTC) X-HE-Tag: burn88_1c17ceb2750b X-Filterd-Recvd-Size: 8695 Received: from mail-qv1-f74.google.com (mail-qv1-f74.google.com [209.85.219.74]) by imf27.hostedemail.com (Postfix) with ESMTP for ; Mon, 11 Jan 2021 09:15:53 +0000 (UTC) Received: by mail-qv1-f74.google.com with SMTP id i20so12767124qvk.18 for ; Mon, 11 Jan 2021 01:15:53 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:message-id:mime-version:subject:from:to:cc :content-transfer-encoding; bh=mA0JHK6xgqVKJnr+4pNXEncnoduYcHjsc3imT7Uizfs=; b=sSWE8ojTbdN1FkVPnp4Ry7+Ll9EPCJKk/KnWSf9cs9pEXp9cN5FZ190rDQZ+xM4fWH CKLujtR3RtAp0+bN5PcWfA4gEDB2KBexnB8oyvOnyHeu32hFrQ5/XHmgjWRInoq2YQG8 JefEBxyjKiLn3f/OTLefT9T4kPn3+Oq22FO5hguParALreQBrE4/HispUrLtvDWPRq8X l8Yw/aqEi+vBRg6ToTGPt0coEW9H9UTKQMLIGecqOLOuegvwPmUgYE9rry21nM+Anois gMQOmLypgMoIYL2ZvOvatClJNCLCo4T0N8dUb509awl2iL/mQ9i3GmOgGnfcViTIFdIr 0GzA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:message-id:mime-version:subject:from :to:cc:content-transfer-encoding; bh=mA0JHK6xgqVKJnr+4pNXEncnoduYcHjsc3imT7Uizfs=; b=ZMRv0zcLKtgCK6Caw+lL+pNY3K1MrglOHrUtiWdC90cim2Lvf+dYs+ZS9HRy7gJsBG A1ku7M/BeWy74mACf63f1+hBXvR4e5gGeTihsl9pr6G8Inj3PhS4Yggc85EUrPRvDnJ/ WHhSQQ/1hlWPhsTkMuzKhlcGFridabUlvzudaP7N+QbGEiTWWvvhCpCUose2vznJnzsA Xjly65Ab3+QMECquI5dRvf6vE7m6ibrSmpLhaabVDP/dQedx9eMrL/+/ox7pD+KHJUVn 0QMAKWUlPR4tc+J/xVytI7B7PoBOfNSNThdwXL/eKaHbaQx6Z/YW3jgfP2RQdQ0/DmKl +HHw== X-Gm-Message-State: AOAM531+LpT586T8+lNAkQC8sMZTK/iSFfTmtlajBw6xrQApQnf+xRMB D9Rff9jI0dxYi2Gzt4zfvqkk2I2g2A== X-Google-Smtp-Source: ABdhPJyYlHEJKPA2IGd8db8i6p9fRK2i1i2q7ieQFolg0VDYM54ZSle9WYAjTPMFcA0542Gi1yzi8aW6OQ== X-Received: from elver.muc.corp.google.com ([2a00:79e0:15:13:f693:9fff:fef4:2449]) (user=elver job=sendgmr) by 2002:ad4:438f:: with SMTP id s15mr18357736qvr.13.1610356552994; Mon, 11 Jan 2021 01:15:52 -0800 (PST) Date: Mon, 11 Jan 2021 10:15:43 +0100 Message-Id: <20210111091544.3287013-1-elver@google.com> Mime-Version: 1.0 X-Mailer: git-send-email 2.30.0.284.gd98b1dd5eaa7-goog Subject: [PATCH mm 1/2] kfence: add option to use KFENCE without static keys From: Marco Elver To: elver@google.com, akpm@linux-foundation.org Cc: glider@google.com, dvyukov@google.com, andreyknvl@google.com, jannh@google.com, mark.rutland@arm.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, kasan-dev@googlegroups.com, " =?utf-8?q?J=C3=B6rn_Eng?= =?utf-8?q?el?= " X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: For certain usecases, specifically where the sample interval is always set to a very low value such as 1ms, it can make sense to use a dynamic branch instead of static branches due to the overhead of toggling a static branch. Therefore, add a new Kconfig option to remove the static branches and instead check kfence_allocation_gate if a KFENCE allocation should be set up. Suggested-by: Jörn Engel Signed-off-by: Marco Elver Reviewed-by: Jörn Engel --- include/linux/kfence.h | 11 ++++++++++- lib/Kconfig.kfence | 12 +++++++++++- mm/kfence/core.c | 32 ++++++++++++++++++-------------- 3 files changed, 39 insertions(+), 16 deletions(-) diff --git a/include/linux/kfence.h b/include/linux/kfence.h index 76246889ecdb..dc86b69d3903 100644 --- a/include/linux/kfence.h +++ b/include/linux/kfence.h @@ -4,7 +4,6 @@ #define _LINUX_KFENCE_H #include -#include #include #ifdef CONFIG_KFENCE @@ -17,7 +16,13 @@ #define KFENCE_POOL_SIZE ((CONFIG_KFENCE_NUM_OBJECTS + 1) * 2 * PAGE_SIZE) extern char *__kfence_pool; +#ifdef CONFIG_KFENCE_STATIC_KEYS +#include DECLARE_STATIC_KEY_FALSE(kfence_allocation_key); +#else +#include +extern atomic_t kfence_allocation_gate; +#endif /** * is_kfence_address() - check if an address belongs to KFENCE pool @@ -104,7 +109,11 @@ void *__kfence_alloc(struct kmem_cache *s, size_t size, gfp_t flags); */ static __always_inline void *kfence_alloc(struct kmem_cache *s, size_t size, gfp_t flags) { +#ifdef CONFIG_KFENCE_STATIC_KEYS if (static_branch_unlikely(&kfence_allocation_key)) +#else + if (unlikely(!atomic_read(&kfence_allocation_gate))) +#endif return __kfence_alloc(s, size, flags); return NULL; } diff --git a/lib/Kconfig.kfence b/lib/Kconfig.kfence index d3ea24fa30fc..78f50ccb3b45 100644 --- a/lib/Kconfig.kfence +++ b/lib/Kconfig.kfence @@ -6,7 +6,6 @@ config HAVE_ARCH_KFENCE menuconfig KFENCE bool "KFENCE: low-overhead sampling-based memory safety error detector" depends on HAVE_ARCH_KFENCE && (SLAB || SLUB) - depends on JUMP_LABEL # To ensure performance, require jump labels select STACKTRACE help KFENCE is a low-overhead sampling-based detector of heap out-of-bounds @@ -25,6 +24,17 @@ menuconfig KFENCE if KFENCE +config KFENCE_STATIC_KEYS + bool "Use static keys to set up allocations" + default y + depends on JUMP_LABEL # To ensure performance, require jump labels + help + Use static keys (static branches) to set up KFENCE allocations. Using + static keys is normally recommended, because it avoids a dynamic + branch in the allocator's fast path. However, with very low sample + intervals, or on systems that do not support jump labels, a dynamic + branch may still be an acceptable performance trade-off. + config KFENCE_SAMPLE_INTERVAL int "Default sample interval in milliseconds" default 100 diff --git a/mm/kfence/core.c b/mm/kfence/core.c index f0816d5f5913..96a9a98e7453 100644 --- a/mm/kfence/core.c +++ b/mm/kfence/core.c @@ -88,11 +88,13 @@ struct kfence_metadata kfence_metadata[CONFIG_KFENCE_NUM_OBJECTS]; static struct list_head kfence_freelist = LIST_HEAD_INIT(kfence_freelist); static DEFINE_RAW_SPINLOCK(kfence_freelist_lock); /* Lock protecting freelist. */ +#ifdef CONFIG_KFENCE_STATIC_KEYS /* The static key to set up a KFENCE allocation. */ DEFINE_STATIC_KEY_FALSE(kfence_allocation_key); +#endif /* Gates the allocation, ensuring only one succeeds in a given period. */ -static atomic_t allocation_gate = ATOMIC_INIT(1); +atomic_t kfence_allocation_gate = ATOMIC_INIT(1); /* Statistics counters for debugfs. */ enum kfence_counter_id { @@ -583,29 +585,31 @@ late_initcall(kfence_debugfs_init); static struct delayed_work kfence_timer; static void toggle_allocation_gate(struct work_struct *work) { - unsigned long end_wait; - if (!READ_ONCE(kfence_enabled)) return; /* Enable static key, and await allocation to happen. */ - atomic_set(&allocation_gate, 0); + atomic_set(&kfence_allocation_gate, 0); +#ifdef CONFIG_KFENCE_STATIC_KEYS static_branch_enable(&kfence_allocation_key); /* * Await an allocation. Timeout after 1 second, in case the kernel stops * doing allocations, to avoid stalling this worker task for too long. */ - end_wait = jiffies + HZ; - do { - set_current_state(TASK_UNINTERRUPTIBLE); - if (atomic_read(&allocation_gate) != 0) - break; - schedule_timeout(1); - } while (time_before(jiffies, end_wait)); - __set_current_state(TASK_RUNNING); - + { + unsigned long end_wait = jiffies + HZ; + + do { + set_current_state(TASK_UNINTERRUPTIBLE); + if (atomic_read(&kfence_allocation_gate) != 0) + break; + schedule_timeout(1); + } while (time_before(jiffies, end_wait)); + __set_current_state(TASK_RUNNING); + } /* Disable static key and reset timer. */ static_branch_disable(&kfence_allocation_key); +#endif schedule_delayed_work(&kfence_timer, msecs_to_jiffies(kfence_sample_interval)); } static DECLARE_DELAYED_WORK(kfence_timer, toggle_allocation_gate); @@ -711,7 +715,7 @@ void *__kfence_alloc(struct kmem_cache *s, size_t size, gfp_t flags) * sense to continue writing to it and pay the associated contention * cost, in case we have a large number of concurrent allocations. */ - if (atomic_read(&allocation_gate) || atomic_inc_return(&allocation_gate) > 1) + if (atomic_read(&kfence_allocation_gate) || atomic_inc_return(&kfence_allocation_gate) > 1) return NULL; if (!READ_ONCE(kfence_enabled))