From patchwork Wed May 10 17:02:35 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 13237092 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B7291C7EE25 for ; Wed, 10 May 2023 17:03:12 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236192AbjEJRDL (ORCPT ); Wed, 10 May 2023 13:03:11 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45488 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236142AbjEJRDJ (ORCPT ); Wed, 10 May 2023 13:03:09 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9E32340DC; Wed, 10 May 2023 10:02:44 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 328FA63F42; Wed, 10 May 2023 17:02:44 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 96B8CC433EF; Wed, 10 May 2023 17:02:43 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1683738163; bh=VXlyCdDwVi1FMedvB2vqwj5QtYQEukG0P+JwjZR1bkQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Yr24vjNQzy6y8a/ZgmUM5NN3QAqR/LXTihrlNgOixMFI6pRC8zy4Ek2uH7mOqBefT 8isdsQBXjpBZKQqsCIpG0ZTAXSLWhqJrqtQc9lZCerlA1kwklNQdqEmc7sE9AZ/ADy goPpxzLUwrVs/O/lDrKh2hb3XAgJvJGUcuHGHQoF8c+buS99nR++ztzVGEn7rDFbsa icxbWRTxlcCrgCca7jMC+zJ2c6SD7oh4t0IM+gLeASifW4MXkQomuBNN4qNGwk0cOk exusTeFR3M+Hzd627FgujhUZm+HtlE0I/bGytX+7QynTNZVakgEpGw7NK8M4vOQVkO FuzP2oPzcTV4Q== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 4C16ECE126E; Wed, 10 May 2023 10:02:43 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Uladzislau Rezki (Sony)" , "Paul E . McKenney" , Joel Fernandes Subject: [PATCH rcu 1/8] rcu/kvfree: Eliminate k[v]free_rcu() single argument macro Date: Wed, 10 May 2023 10:02:35 -0700 Message-Id: <20230510170242.2187714-1-paulmck@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <1c01c38f-3783-44d7-8c11-7416cd5b849c@paulmck-laptop> References: <1c01c38f-3783-44d7-8c11-7416cd5b849c@paulmck-laptop> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: rcu@vger.kernel.org From: "Uladzislau Rezki (Sony)" The kvfree_rcu() and kfree_rcu() APIs are hazardous in that if you forget the second argument, it works, but might sleep. This sleeping can be a correctness bug from atomic contexts, and even in non-atomic contexts it might introduce unacceptable latencies. This commit therefore removes the single-argument kvfree_rcu() and kfree_rcu() macros. Code that would have previously used these single-argument kvfree_rcu() and kfree_rcu() macros should instead use kvfree_rcu_mightsleep() or kfree_rcu_mightsleep(). [ paulmck: Apply Joel Fernandes feedback. ] Signed-off-by: Uladzislau Rezki (Sony) Signed-off-by: Paul E. McKenney Signed-off-by: Joel Fernandes (Google) --- include/linux/rcupdate.h | 29 ++++++++--------------------- 1 file changed, 8 insertions(+), 21 deletions(-) diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h index dcd2cf1e8326..744869ef930a 100644 --- a/include/linux/rcupdate.h +++ b/include/linux/rcupdate.h @@ -957,9 +957,8 @@ static inline notrace void rcu_read_unlock_sched_notrace(void) /** * kfree_rcu() - kfree an object after a grace period. - * @ptr: pointer to kfree for both single- and double-argument invocations. - * @rhf: the name of the struct rcu_head within the type of @ptr, - * but only for double-argument invocations. + * @ptr: pointer to kfree for double-argument invocations. + * @rhf: the name of the struct rcu_head within the type of @ptr. * * Many rcu callbacks functions just call kfree() on the base structure. * These functions are trivial, but their size adds up, and furthermore @@ -984,26 +983,18 @@ static inline notrace void rcu_read_unlock_sched_notrace(void) * The BUILD_BUG_ON check must not involve any function calls, hence the * checks are done in macros here. */ -#define kfree_rcu(ptr, rhf...) kvfree_rcu(ptr, ## rhf) +#define kfree_rcu(ptr, rhf) kvfree_rcu_arg_2(ptr, rhf) +#define kvfree_rcu(ptr, rhf) kvfree_rcu_arg_2(ptr, rhf) /** - * kvfree_rcu() - kvfree an object after a grace period. - * - * This macro consists of one or two arguments and it is - * based on whether an object is head-less or not. If it - * has a head then a semantic stays the same as it used - * to be before: - * - * kvfree_rcu(ptr, rhf); - * - * where @ptr is a pointer to kvfree(), @rhf is the name - * of the rcu_head structure within the type of @ptr. + * kfree_rcu_mightsleep() - kfree an object after a grace period. + * @ptr: pointer to kfree for single-argument invocations. * * When it comes to head-less variant, only one argument * is passed and that is just a pointer which has to be * freed after a grace period. Therefore the semantic is * - * kvfree_rcu(ptr); + * kfree_rcu_mightsleep(ptr); * * where @ptr is the pointer to be freed by kvfree(). * @@ -1012,13 +1003,9 @@ static inline notrace void rcu_read_unlock_sched_notrace(void) * annotation. Otherwise, please switch and embed the * rcu_head structure within the type of @ptr. */ -#define kvfree_rcu(...) KVFREE_GET_MACRO(__VA_ARGS__, \ - kvfree_rcu_arg_2, kvfree_rcu_arg_1)(__VA_ARGS__) - +#define kfree_rcu_mightsleep(ptr) kvfree_rcu_arg_1(ptr) #define kvfree_rcu_mightsleep(ptr) kvfree_rcu_arg_1(ptr) -#define kfree_rcu_mightsleep(ptr) kvfree_rcu_mightsleep(ptr) -#define KVFREE_GET_MACRO(_1, _2, NAME, ...) NAME #define kvfree_rcu_arg_2(ptr, rhf) \ do { \ typeof (ptr) ___p = (ptr); \ From patchwork Wed May 10 17:02:36 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 13237091 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 26BE6C7EE2A for ; Wed, 10 May 2023 17:03:13 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236114AbjEJRDM (ORCPT ); Wed, 10 May 2023 13:03:12 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45434 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236183AbjEJRDK (ORCPT ); Wed, 10 May 2023 13:03:10 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A8177423F; Wed, 10 May 2023 10:02:44 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 4438464A1A; Wed, 10 May 2023 17:02:44 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id A34D7C433D2; Wed, 10 May 2023 17:02:43 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1683738163; bh=Q3rFOVXyjmWxSBKHzG6f0W6zOzsmEdlFl8cDsocGYto=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Otb3WAq01eE3XPJpstZX+UNmbUyTaUU7GdBn5geALYF75KMTiGTHU3d1JqxwjATut X9975dywU8fXmcRlzrb49Zqp0pt8Oc8Vq1OnDRWgR+ZZdnqa1KT3MEzh+/zLOWuy0r wcQGV0wyeGkYCxOaOXlVlq+Odw5tHMIPpScT0sCC/jn5TadMW8q4c3E+plXg+SwhcK Mk0ts5mNWTr8/GTrKEkToIlm2a8g7ago/6jAjRSvLusFwZ4W5jRu+64KkwUTQ7/37J sjZmyw7Kck/H+dKxB9lZnbL2y6eKXUf/MaDPv8DaWFcEFxQeHXl+hggMIivcwH4bHG vTGUb0wXYwlNw== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 4EDBECE120B; Wed, 10 May 2023 10:02:43 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Paul E. McKenney" , Ziwei Dai Subject: [PATCH rcu 2/8] rcu/kvfree: Add debug to check grace periods Date: Wed, 10 May 2023 10:02:36 -0700 Message-Id: <20230510170242.2187714-2-paulmck@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <1c01c38f-3783-44d7-8c11-7416cd5b849c@paulmck-laptop> References: <1c01c38f-3783-44d7-8c11-7416cd5b849c@paulmck-laptop> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: rcu@vger.kernel.org This commit adds debugging checks to verify that the required RCU grace period has elapsed for each kvfree_rcu_bulk_data structure that arrives at the kvfree_rcu_bulk() function. These checks make use of that structure's ->gp_snap field, which has been upgraded from an unsigned long to an rcu_gp_oldstate structure. This upgrade reduces the chances of false positives to nearly zero, even on 32-bit systems, for which this structure carries 64 bits of state. Cc: Ziwei Dai Signed-off-by: Paul E. McKenney --- kernel/rcu/tree.c | 37 +++++++++++++++++++------------------ 1 file changed, 19 insertions(+), 18 deletions(-) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index f52ff7241041..91d75fd6c579 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -2756,7 +2756,7 @@ EXPORT_SYMBOL_GPL(call_rcu); */ struct kvfree_rcu_bulk_data { struct list_head list; - unsigned long gp_snap; + struct rcu_gp_oldstate gp_snap; unsigned long nr_records; void *records[]; }; @@ -2921,23 +2921,24 @@ kvfree_rcu_bulk(struct kfree_rcu_cpu *krcp, int i; debug_rcu_bhead_unqueue(bnode); - - rcu_lock_acquire(&rcu_callback_map); - if (idx == 0) { // kmalloc() / kfree(). - trace_rcu_invoke_kfree_bulk_callback( - rcu_state.name, bnode->nr_records, - bnode->records); - - kfree_bulk(bnode->nr_records, bnode->records); - } else { // vmalloc() / vfree(). - for (i = 0; i < bnode->nr_records; i++) { - trace_rcu_invoke_kvfree_callback( - rcu_state.name, bnode->records[i], 0); - - vfree(bnode->records[i]); + if (!WARN_ON_ONCE(!poll_state_synchronize_rcu_full(&bnode->gp_snap))) { + rcu_lock_acquire(&rcu_callback_map); + if (idx == 0) { // kmalloc() / kfree(). + trace_rcu_invoke_kfree_bulk_callback( + rcu_state.name, bnode->nr_records, + bnode->records); + + kfree_bulk(bnode->nr_records, bnode->records); + } else { // vmalloc() / vfree(). + for (i = 0; i < bnode->nr_records; i++) { + trace_rcu_invoke_kvfree_callback( + rcu_state.name, bnode->records[i], 0); + + vfree(bnode->records[i]); + } } + rcu_lock_release(&rcu_callback_map); } - rcu_lock_release(&rcu_callback_map); raw_spin_lock_irqsave(&krcp->lock, flags); if (put_cached_bnode(krcp, bnode)) @@ -3081,7 +3082,7 @@ kvfree_rcu_drain_ready(struct kfree_rcu_cpu *krcp) INIT_LIST_HEAD(&bulk_ready[i]); list_for_each_entry_safe_reverse(bnode, n, &krcp->bulk_head[i], list) { - if (!poll_state_synchronize_rcu(bnode->gp_snap)) + if (!poll_state_synchronize_rcu_full(&bnode->gp_snap)) break; atomic_sub(bnode->nr_records, &krcp->bulk_count[i]); @@ -3285,7 +3286,7 @@ add_ptr_to_bulk_krc_lock(struct kfree_rcu_cpu **krcp, // Finally insert and update the GP for this page. bnode->records[bnode->nr_records++] = ptr; - bnode->gp_snap = get_state_synchronize_rcu(); + get_state_synchronize_rcu_full(&bnode->gp_snap); atomic_inc(&(*krcp)->bulk_count[idx]); return true; From patchwork Wed May 10 17:02:37 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 13237093 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0CA20C7EE22 for ; Wed, 10 May 2023 17:03:15 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236142AbjEJRDO (ORCPT ); Wed, 10 May 2023 13:03:14 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45600 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236138AbjEJRDM (ORCPT ); Wed, 10 May 2023 13:03:12 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4786F1BCF; Wed, 10 May 2023 10:02:49 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 5DD6664A21; Wed, 10 May 2023 17:02:44 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id B9708C4339B; Wed, 10 May 2023 17:02:43 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1683738163; bh=GUdtbnxrMYKHSRw0zedHwvfAbvZ5RfDc9Uh+UMVsuF4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=FV7lPW9STo0XsQIh0/hNxKrBRQS4wndoPPoVFC8NGisk0B37o0XnAjykCcf/9aRT0 9svfmNbCmWkW9uDWraFp2QaZ/YyCLu3K1atSLIFI+a+w3QS9LL+ahVcDrr6RpLF6Cs lQVswOpzk9rdUesfWxWIrjPEcn2V2bwAZW9K8E1qTJ7jwxZwuaZuwhLi65aqKwPhya zGhRsxLHy0LfooHSY5LDYlJ0abX8htv3W7Y7sV6F9skS+ZZQ0HwLLat0DCLAiB2wzy cnmdpX8aDhhqhYSMV+k2WU5EHJOrJjByGNJtbtlrMCrbv6bVNqBzGV3DoVjqRzN3Yn QlE01/WKBuD6g== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 5136DCE127A; Wed, 10 May 2023 10:02:43 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Uladzislau Rezki (Sony)" , "Paul E . McKenney" Subject: [PATCH rcu 3/8] rcu/kvfree: Add debug check for GP complete for kfree_rcu_cpu list Date: Wed, 10 May 2023 10:02:37 -0700 Message-Id: <20230510170242.2187714-3-paulmck@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <1c01c38f-3783-44d7-8c11-7416cd5b849c@paulmck-laptop> References: <1c01c38f-3783-44d7-8c11-7416cd5b849c@paulmck-laptop> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: rcu@vger.kernel.org From: "Uladzislau Rezki (Sony)" Under low-memory conditions, kvfree_rcu() will use each object's rcu_head structure to queue objects in a singly linked list headed by the kfree_rcu_cpu structure's ->head field. This list is passed to call_rcu() as a unit, but there is no indication of which grace period this list needs to wait for. This in turn prevents adding debug checks in the kfree_rcu_work() as was done for the two page-of-pointers channels in the kfree_rcu_cpu structure. This commit therefore adds a ->head_free_gp_snap field to the kfree_rcu_cpu_work structure to record this grace-period number. It also adds a WARN_ON_ONCE() to kfree_rcu_monitor() that checks to make sure that the required grace period has in fact elapsed. [ paulmck: Fix kerneldoc issue raised by Stephen Rothwell. ] Signed-off-by: Uladzislau Rezki (Sony) Signed-off-by: Paul E. McKenney --- kernel/rcu/tree.c | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 91d75fd6c579..7452ba97ba34 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -2773,6 +2773,7 @@ struct kvfree_rcu_bulk_data { * struct kfree_rcu_cpu_work - single batch of kfree_rcu() requests * @rcu_work: Let queue_rcu_work() invoke workqueue handler after grace period * @head_free: List of kfree_rcu() objects waiting for a grace period + * @head_free_gp_snap: Grace-period snapshot to check for attempted premature frees. * @bulk_head_free: Bulk-List of kvfree_rcu() objects waiting for a grace period * @krcp: Pointer to @kfree_rcu_cpu structure */ @@ -2780,6 +2781,7 @@ struct kvfree_rcu_bulk_data { struct kfree_rcu_cpu_work { struct rcu_work rcu_work; struct rcu_head *head_free; + struct rcu_gp_oldstate head_free_gp_snap; struct list_head bulk_head_free[FREE_N_CHANNELS]; struct kfree_rcu_cpu *krcp; }; @@ -2985,6 +2987,7 @@ static void kfree_rcu_work(struct work_struct *work) struct rcu_head *head; struct kfree_rcu_cpu *krcp; struct kfree_rcu_cpu_work *krwp; + struct rcu_gp_oldstate head_gp_snap; int i; krwp = container_of(to_rcu_work(work), @@ -2999,6 +3002,7 @@ static void kfree_rcu_work(struct work_struct *work) // Channel 3. head = krwp->head_free; krwp->head_free = NULL; + head_gp_snap = krwp->head_free_gp_snap; raw_spin_unlock_irqrestore(&krcp->lock, flags); // Handle the first two channels. @@ -3015,7 +3019,8 @@ static void kfree_rcu_work(struct work_struct *work) * queued on a linked list through their rcu_head structures. * This list is named "Channel 3". */ - kvfree_rcu_list(head); + if (head && !WARN_ON_ONCE(!poll_state_synchronize_rcu_full(&head_gp_snap))) + kvfree_rcu_list(head); } static bool @@ -3147,6 +3152,7 @@ static void kfree_rcu_monitor(struct work_struct *work) // objects queued on the linked list. if (!krwp->head_free) { krwp->head_free = krcp->head; + get_state_synchronize_rcu_full(&krwp->head_free_gp_snap); atomic_set(&krcp->head_count, 0); WRITE_ONCE(krcp->head, NULL); } From patchwork Wed May 10 17:02:38 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 13237095 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2AF35C7EE2D for ; Wed, 10 May 2023 17:03:14 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236228AbjEJRDN (ORCPT ); Wed, 10 May 2023 13:03:13 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45486 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236215AbjEJRDL (ORCPT ); Wed, 10 May 2023 13:03:11 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DE38C4ED7; Wed, 10 May 2023 10:02:44 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 59C2564A1E; Wed, 10 May 2023 17:02:44 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id B6226C4339C; Wed, 10 May 2023 17:02:43 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1683738163; bh=877l4CUT4YO2vkuyC7nkck2lZ0w74fb2w1R/1pwbZ18=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=gEGNa6dSoOgt9+0HxMDmt57xJjCygj0zhXTETavEwjfa3Hl7hfzFrojx5XUZ56JWy BUmtu6X6B45ftCN2ByBaeiYFZixPdbX4DS2ZufelFxsC04PPONnHizxSyN6TkLmQg4 14lxT/WV7S3CxzZPcTXBz6p8mUkOxNab1JjF+0lSk4Rf6L78CxTAormgPG/TwggwV6 WMw9BmtpOklml6aKvMh3Y3UK6aSsqBVj+IJCRL6trJclf77OeTSHKyHU1VdRBbaPpP G2voZG2/d28ay4Wl+zgQT9j1dECnn9SRxQkbIrw9+3+lmhz46KIqv8o8Z1oRZkTm75 a70W2YoEFHPjw== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 53BD2CE127D; Wed, 10 May 2023 10:02:43 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, Zqiang , "Paul E . McKenney" Subject: [PATCH rcu 4/8] rcu/kvfree: Invoke debug_rcu_bhead_unqueue() after checking bnode->gp_snap Date: Wed, 10 May 2023 10:02:38 -0700 Message-Id: <20230510170242.2187714-4-paulmck@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <1c01c38f-3783-44d7-8c11-7416cd5b849c@paulmck-laptop> References: <1c01c38f-3783-44d7-8c11-7416cd5b849c@paulmck-laptop> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: rcu@vger.kernel.org From: Zqiang If kvfree_rcu_bulk() sees that the required grace period has failed to elapse, it leaks the memory because readers might still be using it. But in that case, the debug-objects subsystem still marks the relevant structures as having been freed, even though they are instead being leaked. This commit fixes this mismatch by invoking debug_rcu_bhead_unqueue() only when we are actually going to free the objects. Signed-off-by: Zqiang Signed-off-by: Paul E. McKenney --- kernel/rcu/tree.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 7452ba97ba34..426f1f3bb5f2 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -2922,8 +2922,8 @@ kvfree_rcu_bulk(struct kfree_rcu_cpu *krcp, unsigned long flags; int i; - debug_rcu_bhead_unqueue(bnode); if (!WARN_ON_ONCE(!poll_state_synchronize_rcu_full(&bnode->gp_snap))) { + debug_rcu_bhead_unqueue(bnode); rcu_lock_acquire(&rcu_callback_map); if (idx == 0) { // kmalloc() / kfree(). trace_rcu_invoke_kfree_bulk_callback( From patchwork Wed May 10 17:02:39 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 13237094 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C70B7C77B7D for ; Wed, 10 May 2023 17:03:13 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236193AbjEJRDN (ORCPT ); Wed, 10 May 2023 13:03:13 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45472 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236208AbjEJRDL (ORCPT ); Wed, 10 May 2023 13:03:11 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 489441BE1; Wed, 10 May 2023 10:02:49 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 6A89E64A23; Wed, 10 May 2023 17:02:44 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id C9363C433A0; Wed, 10 May 2023 17:02:43 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1683738163; bh=6K2M1cneLOAe74u4tZCwr+yARE8eIXwk93Jk4FtsLn4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=c4R3E6uIcqhWGD8EYNG9KFHdbkY/l9v+pVqVqsInWzWRUn36+n0VVh6N5BQxai288 qq+gR80SGvdKfiyN1tBxMPWF32Kkf+ona9PTtFoZV7UmDp8bzfcURR/QbBog3yF4tY 7YAxqFOdOUyEZWIsc2LPNg5vGkYWRSHUW1EWnvzYkZGbQmBUiu8I/BdxCDK01K2nAL hvPrrPcxbyJ8svS3HxDd9qVN3oJaU71YJvh1V7D8E4LyPhPR8mH2GWA5zKR0lwIGox EiPp0oBQb2gLUCxUS2c9DxaUzrtTvur9BuZ+wvDod4n2kvMXhMmD6rBoUL/KkIXxhz BNUiDewW6oHvQ== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 55F22CE127F; Wed, 10 May 2023 10:02:43 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, Zqiang , "Paul E . McKenney" Subject: [PATCH rcu 5/8] rcu/kvfree: Use consistent krcp when growing kfree_rcu() page cache Date: Wed, 10 May 2023 10:02:39 -0700 Message-Id: <20230510170242.2187714-5-paulmck@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <1c01c38f-3783-44d7-8c11-7416cd5b849c@paulmck-laptop> References: <1c01c38f-3783-44d7-8c11-7416cd5b849c@paulmck-laptop> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: rcu@vger.kernel.org From: Zqiang The add_ptr_to_bulk_krc_lock() function is invoked to allocate a new kfree_rcu() page, also known as a kvfree_rcu_bulk_data structure. The kfree_rcu_cpu structure's lock is used to protect this operation, except that this lock must be momentarily dropped when allocating memory. It is clearly important that the lock that is reacquired be the same lock that was acquired initially via krc_this_cpu_lock(). Unfortunately, this same krc_this_cpu_lock() function is used to re-acquire this lock, and if the task migrated to some other CPU during the memory allocation, this will result in the kvfree_rcu_bulk_data structure being added to the wrong CPU's kfree_rcu_cpu structure. This commit therefore replaces that second call to krc_this_cpu_lock() with raw_spin_lock_irqsave() in order to explicitly acquire the lock on the correct kfree_rcu_cpu structure, thus keeping things straight even when the task migrates. Signed-off-by: Zqiang Signed-off-by: Paul E. McKenney --- kernel/rcu/tree.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 426f1f3bb5f2..51d84eabf645 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -3279,7 +3279,7 @@ add_ptr_to_bulk_krc_lock(struct kfree_rcu_cpu **krcp, // scenarios. bnode = (struct kvfree_rcu_bulk_data *) __get_free_page(GFP_KERNEL | __GFP_NORETRY | __GFP_NOMEMALLOC | __GFP_NOWARN); - *krcp = krc_this_cpu_lock(flags); + raw_spin_lock_irqsave(&(*krcp)->lock, *flags); } if (!bnode) From patchwork Wed May 10 17:02:40 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 13237097 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 19535C7EE24 for ; Wed, 10 May 2023 17:03:30 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236300AbjEJRD3 (ORCPT ); Wed, 10 May 2023 13:03:29 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45644 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236215AbjEJRDZ (ORCPT ); Wed, 10 May 2023 13:03:25 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 127935592; Wed, 10 May 2023 10:03:06 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id BE92964A28; Wed, 10 May 2023 17:02:44 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 027FAC4339E; Wed, 10 May 2023 17:02:44 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1683738164; bh=lQYiBo5yWC21n7sADnCbbnl6Y7dNLkILdUz1+j0vDYU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=lvhINCbZ0cNh8wGTMn4LkrMuFyCbgVgrX+wEC6QQ5GeojTVMJES5uLsz9m83BYFPy qjdLxtc3Q03j6HB2L2V2veJcGYYsE/Ied0qS40jz+IcpgPGQol745xxul+cLranqXm Zg5cGuoyb1ljGqo0dUHlfPuG2EwPxd4Pn/CinnP9La3YMOdvU7nq8o++wYRNqp1BuB GZQYbGOGx8FWUtY+Zsh5btTKvzvNF9AES8wla0FuKeBrANCkQ3+ixm/XvMiR+Wgusd /RTQO1N9kU+aflLyTq6gJQXwjqhanJVmJPu87tvaVU+WPGSfEgg/dMUz5V9ljdI84B mXBnuew7WmoqA== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 58231CE12D5; Wed, 10 May 2023 10:02:43 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Uladzislau Rezki (Sony)" , "Paul E . McKenney" Subject: [PATCH rcu 6/8] rcu/kvfree: Do not run a page work if a cache is disabled Date: Wed, 10 May 2023 10:02:40 -0700 Message-Id: <20230510170242.2187714-6-paulmck@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <1c01c38f-3783-44d7-8c11-7416cd5b849c@paulmck-laptop> References: <1c01c38f-3783-44d7-8c11-7416cd5b849c@paulmck-laptop> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: rcu@vger.kernel.org From: "Uladzislau Rezki (Sony)" By default the cache size is 5 pages per CPU, but it can be disabled at boot time by setting the rcu_min_cached_objs to zero. When that happens, the current code will uselessly set an hrtimer to schedule refilling this cache with zero pages. This commit therefore streamlines this process by simply refusing the set the hrtimer when rcu_min_cached_objs is zero. Signed-off-by: Uladzislau Rezki (Sony) Signed-off-by: Paul E. McKenney --- kernel/rcu/tree.c | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 51d84eabf645..18f592bf6dc6 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -3225,6 +3225,10 @@ static void fill_page_cache_func(struct work_struct *work) static void run_page_cache_worker(struct kfree_rcu_cpu *krcp) { + // If cache disabled, bail out. + if (!rcu_min_cached_objs) + return; + if (rcu_scheduler_active == RCU_SCHEDULER_RUNNING && !atomic_xchg(&krcp->work_in_progress, 1)) { if (atomic_read(&krcp->backoff_page_cache_fill)) { From patchwork Wed May 10 17:02:41 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 13237096 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id BBD31C7EE22 for ; Wed, 10 May 2023 17:03:29 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236291AbjEJRD3 (ORCPT ); Wed, 10 May 2023 13:03:29 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45846 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236138AbjEJRDZ (ORCPT ); Wed, 10 May 2023 13:03:25 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 11ECF5276; Wed, 10 May 2023 10:03:06 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id C477964A2F; Wed, 10 May 2023 17:02:44 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 06901C433A1; Wed, 10 May 2023 17:02:44 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1683738164; bh=txQDJDLfcrk89uoEEXKL4LZWnIrpTUCXeBg768P6sLs=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=uIJWv3bDn2hFfC3GAznQKTfEIn92fqAADGNsdFRt24u/29/slTTyOIBocfq0g6l4N hP16khOH4RVpTapoxN0aiNt7XqwJlLFn0PWuKJ5bjUL20abVXnDMsVnqchKpVDKJ0c juQ5Z09scxOuxTXQqSA69FuE5lnAabTg8vu5bf2MKygUgoYnJtfLXjpLRWUuitfIBz hJlvVmQlWVN3HLbX90MwIFkKtEjuHeUS4hvE6Go5C/HNmOAGoRNa2GPwkxfgIfdK/1 Hw/omcb7AF7rKOB1vdAdx/rN+pn3Y/DXCXDNN1tK0UYcCCPLsvmdpKuO93+/eXZLxp Fj6l3XJrDZUnQ== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 5A4C3CE1348; Wed, 10 May 2023 10:02:43 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, Zqiang , Uladzislau Rezki , "Paul E . McKenney" Subject: [PATCH rcu 7/8] rcu/kvfree: Make fill page cache start from krcp->nr_bkv_objs Date: Wed, 10 May 2023 10:02:41 -0700 Message-Id: <20230510170242.2187714-7-paulmck@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <1c01c38f-3783-44d7-8c11-7416cd5b849c@paulmck-laptop> References: <1c01c38f-3783-44d7-8c11-7416cd5b849c@paulmck-laptop> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: rcu@vger.kernel.org From: Zqiang When the fill_page_cache_func() function is invoked, it assumes that the cache of pages is completely empty. However, there can be some time between triggering execution of this function and its actual invocation. During this time, kfree_rcu_work() might run, and might fill in part or all of this cache of pages, thus invalidating the fill_page_cache_func() function's assumption. This will not overfill the cache because put_cached_bnode() will reject the extra page. However, it will result in a needless allocation and freeing of one extra page, which might not be helpful under lowish-memory conditions. This commit therefore causes the fill_page_cache_func() to explicitly account for pages that have been placed into the cache shortly before it starts running. Signed-off-by: Zqiang Reviewed-by: Uladzislau Rezki (Sony) Signed-off-by: Paul E. McKenney --- kernel/rcu/tree.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 18f592bf6dc6..98f2e833e217 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -3201,7 +3201,7 @@ static void fill_page_cache_func(struct work_struct *work) nr_pages = atomic_read(&krcp->backoff_page_cache_fill) ? 1 : rcu_min_cached_objs; - for (i = 0; i < nr_pages; i++) { + for (i = READ_ONCE(krcp->nr_bkv_objs); i < nr_pages; i++) { bnode = (struct kvfree_rcu_bulk_data *) __get_free_page(GFP_KERNEL | __GFP_NORETRY | __GFP_NOMEMALLOC | __GFP_NOWARN); From patchwork Wed May 10 17:02:42 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 13237098 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id BC2E5C7EE22 for ; Wed, 10 May 2023 17:03:32 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236336AbjEJRDb (ORCPT ); Wed, 10 May 2023 13:03:31 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46034 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236319AbjEJRD2 (ORCPT ); Wed, 10 May 2023 13:03:28 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4C44B115; Wed, 10 May 2023 10:03:10 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id CE8FF64A2D; Wed, 10 May 2023 17:02:44 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 0F46BC433A7; Wed, 10 May 2023 17:02:44 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1683738164; bh=T2PP2RT6SZHp2JZxt61wF7BysGP/eXTSVvR5ZsRLK6E=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=LPDDps13cQGMdapFCTpx/0au0jFJD6sUSA3QEFhJN7jDo1Xz4JVaeF7oE1A1h5QLN LTsLOSn1vU6q7Huq67ko2XK6WcB8UI/vmcfPLFe1gCZMoxUOqig4zeVmC2yYkRt99L 8IEipqOgSjpRemuN1kJRFIig1Do3hp9z8+q+d8rdmBmPgOmRNx2jERhRWTAKgMPUtb 1Tme4AaYcVLjNOVLgEdbBEBfd79pPZXCNuxtBH0x5LI4yw4Qmr+R4hSh7trNKUbgmT sSh7JzxOeW+xY71EsBZIohxPFayfbJIShqENsn+sQceXT/N0WL1jgU/HlgyF6h395e wzXUhanZwEfcg== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 5C666CE1349; Wed, 10 May 2023 10:02:43 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, Zqiang , Uladzislau Rezki , "Paul E . McKenney" Subject: [PATCH rcu 8/8] rcu/kvfree: Make drain_page_cache() take early return if cache is disabled Date: Wed, 10 May 2023 10:02:42 -0700 Message-Id: <20230510170242.2187714-8-paulmck@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <1c01c38f-3783-44d7-8c11-7416cd5b849c@paulmck-laptop> References: <1c01c38f-3783-44d7-8c11-7416cd5b849c@paulmck-laptop> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: rcu@vger.kernel.org From: Zqiang If the rcutree.rcu_min_cached_objs kernel boot parameter is set to zero, then krcp->page_cache_work will never be triggered to fill page cache. In addition, the put_cached_bnode() will not fill page cache. As a result krcp->bkvcache will always be empty, so there is no need to acquire krcp->lock to get page from krcp->bkvcache. This commit therefore makes drain_page_cache() return immediately if the rcu_min_cached_objs is zero. Signed-off-by: Zqiang Reviewed-by: Uladzislau Rezki (Sony) Signed-off-by: Paul E. McKenney --- kernel/rcu/tree.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 98f2e833e217..00ed45ddc6ca 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -2902,6 +2902,9 @@ drain_page_cache(struct kfree_rcu_cpu *krcp) struct llist_node *page_list, *pos, *n; int freed = 0; + if (!rcu_min_cached_objs) + return 0; + raw_spin_lock_irqsave(&krcp->lock, flags); page_list = llist_del_all(&krcp->bkvcache); WRITE_ONCE(krcp->nr_bkv_objs, 0);