From patchwork Mon Apr 19 08:50:25 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marco Elver X-Patchwork-Id: 12211215 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-26.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 42C42C433B4 for ; Mon, 19 Apr 2021 08:50:59 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id C08F761107 for ; Mon, 19 Apr 2021 08:50:58 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org C08F761107 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 47CD46B006E; Mon, 19 Apr 2021 04:50:58 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 4061C6B0070; Mon, 19 Apr 2021 04:50:58 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2341B6B0071; Mon, 19 Apr 2021 04:50:58 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0199.hostedemail.com [216.40.44.199]) by kanga.kvack.org (Postfix) with ESMTP id 02B606B006E for ; Mon, 19 Apr 2021 04:50:57 -0400 (EDT) Received: from smtpin24.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id B73108249980 for ; Mon, 19 Apr 2021 08:50:57 +0000 (UTC) X-FDA: 78048496554.24.590C771 Received: from mail-wr1-f74.google.com (mail-wr1-f74.google.com [209.85.221.74]) by imf01.hostedemail.com (Postfix) with ESMTP id 5F6A9500152E for ; Mon, 19 Apr 2021 08:50:56 +0000 (UTC) Received: by mail-wr1-f74.google.com with SMTP id 93-20020adf80e60000b0290106fab45006so4162045wrl.20 for ; Mon, 19 Apr 2021 01:50:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=vYl7G57j5M+nH2KzFf9qyrbxlx0FOF9ni4b2Z/4AoS0=; b=j0gtczwPaI+cs4X0SI5bpT9FOnjc+W1uVLyPs+aAetSjO6f/HrBMPoDZh+5dwauc8V kTvpy2Rw30R4hN4dZp79bF54547BHLL5vfc8Ddi8dgyAMd32lQuK1DXC7o24cIxc4rD0 wJN7MOgPITZgUKAYi/eqGlBnLoS5ODMaseG7ew4M80eB7J0Et6VGG68cXC8dNLkyf7+1 mlHjE1wEFTENGwOCoLTjzRawVMw8mhMf7wfJPKh3oxjoQ8dsdW2DU9SC6EIeNsBcrSWE CuCA0uD5l6ccw2Iu0dGpmG88BxIntliqMSt4CRRJhmr0HEjtEl+pP4LfAJ6EpPmjps4B GiiA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=vYl7G57j5M+nH2KzFf9qyrbxlx0FOF9ni4b2Z/4AoS0=; b=E7s90/0GKFu8vQLuUdKoi+4QY6ZQe3jDjG+CBTCoxsP3Nfv6C4032g3/i9sYAW6E2+ 6zJXO/HpcVPN5t6NwFmJs8OxpMm1mZJnSVzbC6zMXDV72DDxNbJJ/Aw4/zuNfYLlEgA3 BuPsrmVrwSwV1j/yCRmSq8nB/CAxmQrZL0hcsS0FTMc6Fw0fVyv83cH+trD3L1FQnDFO 7nvr7prPdTigZyZTUFCbW7ms5yVHBrBBwdd47QRs7Kt3/mdlJ/IKqHJtD2PJ1CcCRCqb hR/i1XvM3tcmHudS+MFGRTTtf57Kq9uMHzRH9zk2QCB8uc0H2a3UgE6Oau3DazYi/UzV n0wg== X-Gm-Message-State: AOAM530Ap/4wbxz//bLRS48z9S+YmQmHFzAu2/MeUDdu6FVVNGDRZR30 Wb5e4YPbZO9oJfVXSP1RxdPf94/zTQ== X-Google-Smtp-Source: ABdhPJzkMlFib2sTj44vDpZc4hhB+ZlXfwBn2wMs7xBakSpQW9OwsdQEzis16G7Q5uQB4QEPzu3OjKS06Q== X-Received: from elver.muc.corp.google.com ([2a00:79e0:15:13:92f8:c03b:1448:ada5]) (user=elver job=sendgmr) by 2002:a05:600c:48a6:: with SMTP id j38mr20532545wmp.99.1618822256477; Mon, 19 Apr 2021 01:50:56 -0700 (PDT) Date: Mon, 19 Apr 2021 10:50:25 +0200 In-Reply-To: <20210419085027.761150-1-elver@google.com> Message-Id: <20210419085027.761150-2-elver@google.com> Mime-Version: 1.0 References: <20210419085027.761150-1-elver@google.com> X-Mailer: git-send-email 2.31.1.368.gbe11c130af-goog Subject: [PATCH 1/3] kfence: await for allocation using wait_event From: Marco Elver To: elver@google.com, akpm@linux-foundation.org Cc: glider@google.com, dvyukov@google.com, jannh@google.com, mark.rutland@arm.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, kasan-dev@googlegroups.com X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: 5F6A9500152E X-Stat-Signature: wgfa1pc9g5x7t1zarc57ras8gxqa35oq Received-SPF: none (flex--elver.bounces.google.com>: No applicable sender policy available) receiver=imf01; identity=mailfrom; envelope-from="<3cER9YAUKCPcdkudqfnnfkd.bnlkhmtw-lljuZbj.nqf@flex--elver.bounces.google.com>"; helo=mail-wr1-f74.google.com; client-ip=209.85.221.74 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1618822256-107478 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On mostly-idle systems, we have observed that toggle_allocation_gate() is a cause of frequent wake-ups, preventing an otherwise idle CPU to go into a lower power state. A late change in KFENCE's development, due to a potential deadlock [1], required changing the scheduling-friendly wait_event_timeout() and wake_up() to an open-coded wait-loop using schedule_timeout(). [1] https://lkml.kernel.org/r/000000000000c0645805b7f982e4@google.com To avoid unnecessary wake-ups, switch to using wait_event_timeout(). Unfortunately, we still cannot use a version with direct wake_up() in __kfence_alloc() due to the same potential for deadlock as in [1]. Instead, add a level of indirection via an irq_work that is scheduled if we determine that the kfence_timer requires a wake_up(). Fixes: 0ce20dd84089 ("mm: add Kernel Electric-Fence infrastructure") Signed-off-by: Marco Elver --- lib/Kconfig.kfence | 1 + mm/kfence/core.c | 58 +++++++++++++++++++++++++++++++++------------- 2 files changed, 43 insertions(+), 16 deletions(-) diff --git a/lib/Kconfig.kfence b/lib/Kconfig.kfence index 78f50ccb3b45..e641add33947 100644 --- a/lib/Kconfig.kfence +++ b/lib/Kconfig.kfence @@ -7,6 +7,7 @@ menuconfig KFENCE bool "KFENCE: low-overhead sampling-based memory safety error detector" depends on HAVE_ARCH_KFENCE && (SLAB || SLUB) select STACKTRACE + select IRQ_WORK help KFENCE is a low-overhead sampling-based detector of heap out-of-bounds access, use-after-free, and invalid-free errors. KFENCE is designed diff --git a/mm/kfence/core.c b/mm/kfence/core.c index 768dbd58170d..5f0a56041549 100644 --- a/mm/kfence/core.c +++ b/mm/kfence/core.c @@ -10,6 +10,7 @@ #include #include #include +#include #include #include #include @@ -587,6 +588,20 @@ late_initcall(kfence_debugfs_init); /* === Allocation Gate Timer ================================================ */ +#ifdef CONFIG_KFENCE_STATIC_KEYS +/* Wait queue to wake up allocation-gate timer task. */ +static DECLARE_WAIT_QUEUE_HEAD(allocation_wait); + +static void wake_up_kfence_timer(struct irq_work *work) +{ + wake_up(&allocation_wait); +} +static DEFINE_IRQ_WORK(wake_up_kfence_timer_work, wake_up_kfence_timer); + +/* Indicate if timer task is waiting, to avoid unnecessary irq_work. */ +static bool kfence_timer_waiting; +#endif + /* * Set up delayed work, which will enable and disable the static key. We need to * use a work queue (rather than a simple timer), since enabling and disabling a @@ -604,25 +619,16 @@ static void toggle_allocation_gate(struct work_struct *work) if (!READ_ONCE(kfence_enabled)) return; - /* Enable static key, and await allocation to happen. */ atomic_set(&kfence_allocation_gate, 0); #ifdef CONFIG_KFENCE_STATIC_KEYS + /* Enable static key, and await allocation to happen. */ static_branch_enable(&kfence_allocation_key); - /* - * Await an allocation. Timeout after 1 second, in case the kernel stops - * doing allocations, to avoid stalling this worker task for too long. - */ - { - unsigned long end_wait = jiffies + HZ; - - do { - set_current_state(TASK_UNINTERRUPTIBLE); - if (atomic_read(&kfence_allocation_gate) != 0) - break; - schedule_timeout(1); - } while (time_before(jiffies, end_wait)); - __set_current_state(TASK_RUNNING); - } + + WRITE_ONCE(kfence_timer_waiting, true); + smp_mb(); /* See comment in __kfence_alloc(). */ + wait_event_timeout(allocation_wait, atomic_read(&kfence_allocation_gate), HZ); + smp_store_release(&kfence_timer_waiting, false); /* Order after wait_event(). */ + /* Disable static key and reset timer. */ static_branch_disable(&kfence_allocation_key); #endif @@ -729,6 +735,26 @@ void *__kfence_alloc(struct kmem_cache *s, size_t size, gfp_t flags) */ if (atomic_read(&kfence_allocation_gate) || atomic_inc_return(&kfence_allocation_gate) > 1) return NULL; +#ifdef CONFIG_KFENCE_STATIC_KEYS + /* + * Read of kfence_timer_waiting must be ordered after write to + * kfence_allocation_gate (fully ordered per atomic_inc_return()). + * + * Conversely, the write to kfence_timer_waiting must be ordered before + * the check of kfence_allocation_gate in toggle_allocation_gate(). + * + * This ensures that toggle_allocation_gate() always sees the updated + * kfence_allocation_gate, or we see that the timer is waiting and will + * queue the work to wake it up. + */ + if (READ_ONCE(kfence_timer_waiting)) { + /* + * Calling wake_up() here may deadlock when allocations happen + * from within timer code. Use an irq_work to defer it. + */ + irq_work_queue(&wake_up_kfence_timer_work); + } +#endif if (!READ_ONCE(kfence_enabled)) return NULL;