From patchwork Thu Sep 1 22:17:03 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joel Fernandes X-Patchwork-Id: 12963366 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8C2F9ECAAD3 for ; Thu, 1 Sep 2022 22:18:02 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234235AbiIAWSB (ORCPT ); Thu, 1 Sep 2022 18:18:01 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45740 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234274AbiIAWSA (ORCPT ); Thu, 1 Sep 2022 18:18:00 -0400 Received: from mail-qv1-xf32.google.com (mail-qv1-xf32.google.com [IPv6:2607:f8b0:4864:20::f32]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3DBB05809B for ; Thu, 1 Sep 2022 15:17:59 -0700 (PDT) Received: by mail-qv1-xf32.google.com with SMTP id y17so170944qvr.5 for ; Thu, 01 Sep 2022 15:17:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc; bh=+2CQoWOzORwmFm7W8hobFnBssU2v8aOfIo1J22tLn5A=; b=gYFVtvH6IT88YOW+VuemDQdOw9jN1vrqUhqD4F59DGDWIburm49TpkXGVQY87YJ84L qH6wa/UO5erX3ioCJY2/tLyTnBhtQ1+KSn2ciZl3SpPmczkva7MHX/hbiZhzntNqv0Q4 KArjDPxW26NuxuN0qO1Zj5kVlXjgrE+8nGLpo= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=+2CQoWOzORwmFm7W8hobFnBssU2v8aOfIo1J22tLn5A=; b=eNYAoMqTHCILpDM6ci+EyaOjXY1ZLLo0hcA0P7wmCb6IzgH5IQFxH7REFPDN0NyEkx zwRDFo1g4iv/sNU1FVmRQX8Fg9DFlNnWxCReh8tnoSAiV3XKc9yt5Ph9+cYtI17/H0+f /MR+lNJxuaK2sEr5G39Um3hdM8+GBPmxiiIg1tNreSd4Je1V8LicVKT/+kGdoGi5DvMa vpJz0D96Wm/OuNhCoZDBbLhcORYJIL0xcLkZFvlNbnamHZPc5+HoywV9z6TDkaHF4CCZ rh8XhQmezqMfAbCgbUtkY4fFS6q4AoDP2we4PfUxWXKRRQfJeUFzQ5wMgCTm2/InMDpe WAlQ== X-Gm-Message-State: ACgBeo2LYk+oQE5WJAj8tCTWmiTnzxz/bIgF6PeZAnnKNQ0MKJ/x9ygc 3QZnkxJGfilreolgZgccqgkishsC3te1QQ== X-Google-Smtp-Source: AA6agR4kxLtEYufGPqZXnLYzR+tmjfowzJgSuRNILICeEUEdChLKGM9hmPATH/Xdqmcnc+IW5AQBNw== X-Received: by 2002:a05:6214:2a4b:b0:499:eb9:6ed3 with SMTP id jf11-20020a0562142a4b00b004990eb96ed3mr14878236qvb.66.1662070678188; Thu, 01 Sep 2022 15:17:58 -0700 (PDT) Received: from joelboxx.c.googlers.com.com (228.221.150.34.bc.googleusercontent.com. [34.150.221.228]) by smtp.gmail.com with ESMTPSA id s16-20020ac85290000000b0034305a91aaesm11060794qtn.83.2022.09.01.15.17.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Sep 2022 15:17:57 -0700 (PDT) From: "Joel Fernandes (Google)" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, rushikesh.s.kadam@intel.com, urezki@gmail.com, neeraj.iitr10@gmail.com, frederic@kernel.org, paulmck@kernel.org, rostedt@goodmis.org, vineeth@bitbyteword.org, boqun.feng@gmail.com, Vlastimil Babka Subject: [PATCH v5 01/18] mm/slub: perform free consistency checks before call_rcu Date: Thu, 1 Sep 2022 22:17:03 +0000 Message-Id: <20220901221720.1105021-2-joel@joelfernandes.org> X-Mailer: git-send-email 2.37.2.789.g6183377224-goog In-Reply-To: <20220901221720.1105021-1-joel@joelfernandes.org> References: <20220901221720.1105021-1-joel@joelfernandes.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: rcu@vger.kernel.org From: Vlastimil Babka For SLAB_TYPESAFE_BY_RCU caches we use call_rcu to perform empty slab freeing. The rcu callback rcu_free_slab() calls __free_slab() that currently includes checking the slab consistency for caches with SLAB_CONSISTENCY_CHECKS flags. This check needs the slab->objects field to be intact. Because in the next patch we want to allow rcu_head in struct slab to become larger in debug configurations and thus potentially overwrite more fields through a union than slab_list, we want to limit the fields used in rcu_free_slab(). Thus move the consistency checks to free_slab() before call_rcu(). This can be done safely even for SLAB_TYPESAFE_BY_RCU caches where accesses to the objects can still occur after freeing them. As a result, only the slab->slab_cache field has to be physically separate from rcu_head for the freeing callback to work. We also save some cycles in the rcu callback for caches with consistency checks enabled. Signed-off-by: Vlastimil Babka --- mm/slub.c | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/mm/slub.c b/mm/slub.c index 862dbd9af4f5..d86be1b0d09f 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -2036,14 +2036,6 @@ static void __free_slab(struct kmem_cache *s, struct slab *slab) int order = folio_order(folio); int pages = 1 << order; - if (kmem_cache_debug_flags(s, SLAB_CONSISTENCY_CHECKS)) { - void *p; - - slab_pad_check(s, slab); - for_each_object(p, s, slab_address(slab), slab->objects) - check_object(s, slab, p, SLUB_RED_INACTIVE); - } - __slab_clear_pfmemalloc(slab); __folio_clear_slab(folio); folio->mapping = NULL; @@ -2062,9 +2054,17 @@ static void rcu_free_slab(struct rcu_head *h) static void free_slab(struct kmem_cache *s, struct slab *slab) { - if (unlikely(s->flags & SLAB_TYPESAFE_BY_RCU)) { + if (kmem_cache_debug_flags(s, SLAB_CONSISTENCY_CHECKS)) { + void *p; + + slab_pad_check(s, slab); + for_each_object(p, s, slab_address(slab), slab->objects) + check_object(s, slab, p, SLUB_RED_INACTIVE); + } + + if (unlikely(s->flags & SLAB_TYPESAFE_BY_RCU)) call_rcu(&slab->rcu_head, rcu_free_slab); - } else + else __free_slab(s, slab); } From patchwork Thu Sep 1 22:17:04 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joel Fernandes X-Patchwork-Id: 12963368 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E32D6C6FA87 for ; Thu, 1 Sep 2022 22:18:03 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234471AbiIAWSC (ORCPT ); Thu, 1 Sep 2022 18:18:02 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45760 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234435AbiIAWSB (ORCPT ); Thu, 1 Sep 2022 18:18:01 -0400 Received: from mail-qt1-x832.google.com (mail-qt1-x832.google.com [IPv6:2607:f8b0:4864:20::832]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E2CF7580B6 for ; Thu, 1 Sep 2022 15:17:59 -0700 (PDT) Received: by mail-qt1-x832.google.com with SMTP id a22so212348qtw.10 for ; Thu, 01 Sep 2022 15:17:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc; bh=ocxsdgSxr9WyJwVb16iinpfYqPxMSVghGuyFQwZXQfs=; b=ti/1WG1u8VpXWufM64ni12DTreKGCzniiXPz9JlY5rrgucCra9WVKDdRPJHpbwSpxj iG++ZS/nv4W4EuHP+8BlbKh7cvUFrg0n06h/UoIiKDHin6Qnra6rJCmg3+AA9tjOu7oI ZELEHVazqBBYH+5PbuvWb3Wxw/KtdCPykBIsE= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=ocxsdgSxr9WyJwVb16iinpfYqPxMSVghGuyFQwZXQfs=; b=MVvHsQSPGJb0fvgnDTe2pxECYVKnvyv1IvRb+5/L7Rk5NR+5XCbllPd9otuju9DIxh NRqnFIxcmw8iggd2PoCebCQ3hz4UI0PYP8UCuLYiGHakpBPbx2yCWxizWAqtcMhz8g2m zxUwKleeuwJHBSGgfDXFOQH+Ce6bDkUJGDowdP28bMWw9bASuKe7552L/sm5NWHWvAFM LLFw6dzAganlNzFWyu0JLtan2iE+NY991dg/tDyij0l1aRSNSTRp1ByBJKJwgJw7PwHY B62QangAj3K8JhNuhoV7+tf4NK7N9ni/GgxyJm6PPMKgo4dnWxfDjXh/OJ3SUJsOVEe+ 0Cpg== X-Gm-Message-State: ACgBeo3rz5PjZa5myJrD8XprDM857oU2+oRTwm5YRJn4oPV4O5eP+Gvj Vz1DUEGKDHC8a3zJCAQZOFtqUD/LXI8lMw== X-Google-Smtp-Source: AA6agR6eIoBqsq90vcCegOqFmOL0xyor84GJfMSmDM+50XbL31b6MpOWJKpxG/NZU2dOPdvIGdA/1Q== X-Received: by 2002:ac8:5f53:0:b0:343:a8d0:c81b with SMTP id y19-20020ac85f53000000b00343a8d0c81bmr25798028qta.489.1662070678847; Thu, 01 Sep 2022 15:17:58 -0700 (PDT) Received: from joelboxx.c.googlers.com.com (228.221.150.34.bc.googleusercontent.com. [34.150.221.228]) by smtp.gmail.com with ESMTPSA id s16-20020ac85290000000b0034305a91aaesm11060794qtn.83.2022.09.01.15.17.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Sep 2022 15:17:58 -0700 (PDT) From: "Joel Fernandes (Google)" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, rushikesh.s.kadam@intel.com, urezki@gmail.com, neeraj.iitr10@gmail.com, frederic@kernel.org, paulmck@kernel.org, rostedt@goodmis.org, vineeth@bitbyteword.org, boqun.feng@gmail.com, Vlastimil Babka , Joel Fernandes Subject: [PATCH v5 02/18] mm/sl[au]b: rearrange struct slab fields to allow larger rcu_head Date: Thu, 1 Sep 2022 22:17:04 +0000 Message-Id: <20220901221720.1105021-3-joel@joelfernandes.org> X-Mailer: git-send-email 2.37.2.789.g6183377224-goog In-Reply-To: <20220901221720.1105021-1-joel@joelfernandes.org> References: <20220901221720.1105021-1-joel@joelfernandes.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: rcu@vger.kernel.org From: Vlastimil Babka Joel reports [1] that increasing the rcu_head size for debugging purposes used to work before struct slab was split from struct page, but now runs into the various SLAB_MATCH() sanity checks of the layout. This is because the rcu_head in struct page is in union with large sub-structures and has space to grow without exceeding their size, while in struct slab (for SLAB and SLUB) it's in union only with a list_head. On closer inspection (and after the previous patch) we can put all fields except slab_cache to a union with rcu_head, as slab_cache is sufficient for the rcu freeing callbacks to work and the rest can be overwritten by rcu_head without causing issues. This is only somewhat complicated by the need to keep SLUB's freelist+counters aligned for cmpxchg_double. As a result the fields need to be reordered so that slab_cache is first (after page flags) and the union with rcu_head follows. For consistency, do that for SLAB as well, although not necessary there. As a result, the rcu_head field in struct page and struct slab is no longer at the same offset, but that doesn't matter as there is no casting that would rely on that in the slab freeing callbacks, so we can just drop the respective SLAB_MATCH() check. Also we need to update the SLAB_MATCH() for compound_head to reflect the new ordering. While at it, also add a static_assert to check the alignment needed for cmpxchg_double so mistakes are found sooner than a runtime GPF. [1] https://lore.kernel.org/all/85afd876-d8bb-0804-b2c5-48ed3055e702@joelfernandes.org/ Reported-by: Joel Fernandes Signed-off-by: Vlastimil Babka Reviewed-by: Hyeonggon Yoo <42.hyeyoo@gmail.com> Acked-by: Hyeonggon Yoo <42.hyeyoo@gmail.com> --- mm/slab.h | 54 ++++++++++++++++++++++++++++++++---------------------- 1 file changed, 32 insertions(+), 22 deletions(-) diff --git a/mm/slab.h b/mm/slab.h index 4ec82bec15ec..2c248864ea91 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -11,37 +11,43 @@ struct slab { #if defined(CONFIG_SLAB) + struct kmem_cache *slab_cache; union { - struct list_head slab_list; + struct { + struct list_head slab_list; + void *freelist; /* array of free object indexes */ + void *s_mem; /* first object */ + }; struct rcu_head rcu_head; }; - struct kmem_cache *slab_cache; - void *freelist; /* array of free object indexes */ - void *s_mem; /* first object */ unsigned int active; #elif defined(CONFIG_SLUB) - union { - struct list_head slab_list; - struct rcu_head rcu_head; -#ifdef CONFIG_SLUB_CPU_PARTIAL - struct { - struct slab *next; - int slabs; /* Nr of slabs left */ - }; -#endif - }; struct kmem_cache *slab_cache; - /* Double-word boundary */ - void *freelist; /* first free object */ union { - unsigned long counters; struct { - unsigned inuse:16; - unsigned objects:15; - unsigned frozen:1; + union { + struct list_head slab_list; +#ifdef CONFIG_SLUB_CPU_PARTIAL + struct { + struct slab *next; + int slabs; /* Nr of slabs left */ + }; +#endif + }; + /* Double-word boundary */ + void *freelist; /* first free object */ + union { + unsigned long counters; + struct { + unsigned inuse:16; + unsigned objects:15; + unsigned frozen:1; + }; + }; }; + struct rcu_head rcu_head; }; unsigned int __unused; @@ -66,9 +72,10 @@ struct slab { #define SLAB_MATCH(pg, sl) \ static_assert(offsetof(struct page, pg) == offsetof(struct slab, sl)) SLAB_MATCH(flags, __page_flags); -SLAB_MATCH(compound_head, slab_list); /* Ensure bit 0 is clear */ #ifndef CONFIG_SLOB -SLAB_MATCH(rcu_head, rcu_head); +SLAB_MATCH(compound_head, slab_cache); /* Ensure bit 0 is clear */ +#else +SLAB_MATCH(compound_head, slab_list); /* Ensure bit 0 is clear */ #endif SLAB_MATCH(_refcount, __page_refcount); #ifdef CONFIG_MEMCG @@ -76,6 +83,9 @@ SLAB_MATCH(memcg_data, memcg_data); #endif #undef SLAB_MATCH static_assert(sizeof(struct slab) <= sizeof(struct page)); +#if defined(CONFIG_HAVE_CMPXCHG_DOUBLE) && defined(CONFIG_SLUB) +static_assert(IS_ALIGNED(offsetof(struct slab, freelist), 16)); +#endif /** * folio_slab - Converts from folio to slab. From patchwork Thu Sep 1 22:17:05 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joel Fernandes X-Patchwork-Id: 12963367 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D4539C6FA82 for ; Thu, 1 Sep 2022 22:18:03 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234575AbiIAWSC (ORCPT ); Thu, 1 Sep 2022 18:18:02 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45774 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234448AbiIAWSB (ORCPT ); Thu, 1 Sep 2022 18:18:01 -0400 Received: from mail-qv1-xf31.google.com (mail-qv1-xf31.google.com [IPv6:2607:f8b0:4864:20::f31]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 75785580BE for ; Thu, 1 Sep 2022 15:18:00 -0700 (PDT) Received: by mail-qv1-xf31.google.com with SMTP id kh8so185879qvb.1 for ; Thu, 01 Sep 2022 15:18:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc; bh=wacV+wNWevSZZedckm8hHMeO8v0HhmKZhpXkr12CWiU=; b=e7rfmVttSIAE2dX9U/UrdKI2/QNY3ZP3fKeyJGley56wIpwe2qS6saz3KZesypfmqe hM1riUNN/4tHj4mrt7IXt6dpL2LLv2du/nCXvZZI8MdoBiSD0/AnUB8528he0Bh8pZQV vrMkt1S957E51Pelj+JfGSpSAlfQVDY8utxxc= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=wacV+wNWevSZZedckm8hHMeO8v0HhmKZhpXkr12CWiU=; b=6I0KdB73dVO9i7Gzz7MTCFGmqLgIs5SaeCJUT4/5OAMBwUwHJcnVmEdpgP8TgI94AN VmeFnYRw4PDnd2b0LbZ5cmpmn8BoOdHmtOrnFbktLVR6hQWKzIfNUfXuFsbaSsXhsRox XKKXqNDGYajos5TxHxw8AwWBjSRhIjhqKl346m29ET5NrPUcqOmaYCfPEDOqbdCrN6QV CPrvhoNaBMZxkE03B8bblMpmvXsDB2PrrF60AkJjz7a0d2x6wRCQ1cYKoeoVNBTCVAN2 sObDICQqJaWueNYG+fa8raK7b8Ap8IXfvbRweTdHMyIbZNQtim1drtYvXdW1fVRKQZOk nSsw== X-Gm-Message-State: ACgBeo0r1GVD8DMaJ9LtBKTW8zdAndNI83voqOtjLK37DmkNBxc9tEwJ ki7Zt4wIVgtRgqTdHH4CYVUWBrTtOKx7EQ== X-Google-Smtp-Source: AA6agR7ebVdicXTu0lYFUslN4hqrp3yrx/UXs4hw278sz5joSWluao70f6a70ft5FQ57ZlDUz+4CoA== X-Received: by 2002:a05:6214:27c1:b0:473:85c9:3eeb with SMTP id ge1-20020a05621427c100b0047385c93eebmr26877303qvb.53.1662070679524; Thu, 01 Sep 2022 15:17:59 -0700 (PDT) Received: from joelboxx.c.googlers.com.com (228.221.150.34.bc.googleusercontent.com. [34.150.221.228]) by smtp.gmail.com with ESMTPSA id s16-20020ac85290000000b0034305a91aaesm11060794qtn.83.2022.09.01.15.17.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Sep 2022 15:17:59 -0700 (PDT) From: "Joel Fernandes (Google)" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, rushikesh.s.kadam@intel.com, urezki@gmail.com, neeraj.iitr10@gmail.com, frederic@kernel.org, paulmck@kernel.org, rostedt@goodmis.org, vineeth@bitbyteword.org, boqun.feng@gmail.com, "Joel Fernandes (Google)" Subject: [PATCH v5 03/18] rcu/tree: Use READ_ONCE() for lockless read of rnp->qsmask Date: Thu, 1 Sep 2022 22:17:05 +0000 Message-Id: <20220901221720.1105021-4-joel@joelfernandes.org> X-Mailer: git-send-email 2.37.2.789.g6183377224-goog In-Reply-To: <20220901221720.1105021-1-joel@joelfernandes.org> References: <20220901221720.1105021-1-joel@joelfernandes.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: rcu@vger.kernel.org The rnp->qsmask is locklessly accessed from rcutree_dying_cpu(). This may help avoid load/store tearing due to concurrent access, KCSAN issues, and preserve sanity of people reading the mask in tracing. Signed-off-by: Joel Fernandes (Google) --- kernel/rcu/tree.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 0ca21ac0f064..5ec97e3f7468 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -2106,7 +2106,7 @@ int rcutree_dying_cpu(unsigned int cpu) if (!IS_ENABLED(CONFIG_HOTPLUG_CPU)) return 0; - blkd = !!(rnp->qsmask & rdp->grpmask); + blkd = !!(READ_ONCE(rnp->qsmask) & rdp->grpmask); trace_rcu_grace_period(rcu_state.name, READ_ONCE(rnp->gp_seq), blkd ? TPS("cpuofl-bgp") : TPS("cpuofl")); return 0; From patchwork Thu Sep 1 22:17:06 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joel Fernandes X-Patchwork-Id: 12963370 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id F02C8ECAAD3 for ; Thu, 1 Sep 2022 22:18:06 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234599AbiIAWSE (ORCPT ); Thu, 1 Sep 2022 18:18:04 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45786 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234274AbiIAWSC (ORCPT ); Thu, 1 Sep 2022 18:18:02 -0400 Received: from mail-qt1-x82d.google.com (mail-qt1-x82d.google.com [IPv6:2607:f8b0:4864:20::82d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 492B25809B for ; Thu, 1 Sep 2022 15:18:01 -0700 (PDT) Received: by mail-qt1-x82d.google.com with SMTP id h22so245555qtu.2 for ; Thu, 01 Sep 2022 15:18:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc; bh=8tBDdZaeoT3b5CBzqvW/ZB2aZTVcUlmvdmlRm1sAyM4=; b=QBkSAea3WwRWmhiRWRuvpOfEw+FTTLufqMSEdc1gAtyAq+xBunox+Nuh8C6IUqQDo1 +LxFhVcyiQIx4jvB2azfPR7AmGcN5zZvubRtp5adhPS46cT5+VACoX35Em9rNj9ukK+M L11Wz8mOl+0mNcbPFv1bMoPDD7RhIAz59Gntw= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=8tBDdZaeoT3b5CBzqvW/ZB2aZTVcUlmvdmlRm1sAyM4=; b=07CbydzWDSn8rcmKSqd3MjgfGVuyzwQBBt0IrM78cWEiFqpeeYLkyToPgFpScpuNdS kQtmgJRkaUeHINnMLZj5RMhC1u2aBHAL/Bgaw6qCtoCqlHLUjUa2aIE87l5LJhhr+Tkb OaU3x/bUfiThbthubwAkasHHFCYHgZjXnkU+D1HJUzhIfiN8rUuFq4n4QAn66r3ZTIIa R/qn0Or21bSvGQ7d+7tdEQPXNjKtKzcHCYy7qdnOENFw6IEEwy56uTcF16hKjOtgfKT7 EpkPAteffFmSsHdwpvePc8OK+NgvUEjlL0cxRabnYAwcOrSUSRaAwx6aX7ebafIIWvI6 S4+Q== X-Gm-Message-State: ACgBeo1k8TvnflZ0xf+ksLX85E/H2CdD1nv1kLPrultG69YIFQ6t4YYb cT2QTXSELxWfj19PfNVaTv8KTf8e8/2Bkw== X-Google-Smtp-Source: AA6agR4RLjnlk0HhdTMZyE6uJDeBeCmArIKBraVqa1sV/k/4xV1xpnGzfyzL2EqEG8NqbDqWrq33bg== X-Received: by 2002:a05:622a:649:b0:343:7b74:5cd1 with SMTP id a9-20020a05622a064900b003437b745cd1mr26095535qtb.407.1662070680221; Thu, 01 Sep 2022 15:18:00 -0700 (PDT) Received: from joelboxx.c.googlers.com.com (228.221.150.34.bc.googleusercontent.com. [34.150.221.228]) by smtp.gmail.com with ESMTPSA id s16-20020ac85290000000b0034305a91aaesm11060794qtn.83.2022.09.01.15.17.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Sep 2022 15:17:59 -0700 (PDT) From: "Joel Fernandes (Google)" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, rushikesh.s.kadam@intel.com, urezki@gmail.com, neeraj.iitr10@gmail.com, frederic@kernel.org, paulmck@kernel.org, rostedt@goodmis.org, vineeth@bitbyteword.org, boqun.feng@gmail.com, "Joel Fernandes (Google)" Subject: [PATCH v5 04/18] rcu: Fix late wakeup when flush of bypass cblist happens Date: Thu, 1 Sep 2022 22:17:06 +0000 Message-Id: <20220901221720.1105021-5-joel@joelfernandes.org> X-Mailer: git-send-email 2.37.2.789.g6183377224-goog In-Reply-To: <20220901221720.1105021-1-joel@joelfernandes.org> References: <20220901221720.1105021-1-joel@joelfernandes.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: rcu@vger.kernel.org When the bypass cblist gets too big or its timeout has occurred, it is flushed into the main cblist. However, the bypass timer is still running and the behavior is that it would eventually expire and wake the GP thread. Since we are going to use the bypass cblist for lazy CBs, do the wakeup soon as the flush happens. Otherwise, the lazy-timer will go off much later and the now-non-lazy cblist CBs can get stranded for the duration of the timer. This is a good thing to do anyway (regardless of this series), since it makes the behavior consistent with behavior of other code paths where queueing something into the ->cblist makes the GP kthread in a non-sleeping state quickly. Signed-off-by: Joel Fernandes (Google) Signed-off-by: Joel Fernandes (Google) --- kernel/rcu/tree_nocb.h | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/kernel/rcu/tree_nocb.h b/kernel/rcu/tree_nocb.h index 0a5f0ef41484..31068dd31315 100644 --- a/kernel/rcu/tree_nocb.h +++ b/kernel/rcu/tree_nocb.h @@ -447,7 +447,13 @@ static bool rcu_nocb_try_bypass(struct rcu_data *rdp, struct rcu_head *rhp, rcu_advance_cbs_nowake(rdp->mynode, rdp); rdp->nocb_gp_adv_time = j; } - rcu_nocb_unlock_irqrestore(rdp, flags); + + // The flush succeeded and we moved CBs into the ->cblist. + // However, the bypass timer might still be running. Wakeup the + // GP thread by calling a helper with was_all_done set so that + // wake up happens (needed if main CB list was empty before). + __call_rcu_nocb_wake(rdp, true, flags) + return true; // Callback already enqueued. } From patchwork Thu Sep 1 22:17:07 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joel Fernandes X-Patchwork-Id: 12963369 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 71ADAECAAD2 for ; Thu, 1 Sep 2022 22:18:04 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233595AbiIAWSD (ORCPT ); Thu, 1 Sep 2022 18:18:03 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45786 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234555AbiIAWSC (ORCPT ); Thu, 1 Sep 2022 18:18:02 -0400 Received: from mail-qv1-xf32.google.com (mail-qv1-xf32.google.com [IPv6:2607:f8b0:4864:20::f32]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DA07F543C2 for ; Thu, 1 Sep 2022 15:18:01 -0700 (PDT) Received: by mail-qv1-xf32.google.com with SMTP id jy14so145918qvb.12 for ; Thu, 01 Sep 2022 15:18:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc; bh=7NTlW4AVEaEifdUDNJKG5n5TSALhIocTrw3kCCpTxNA=; b=JOSQJQVjQiQE0lsAHsymw5P/1/LCdxuxYv3QrWVdzDXjflavEeAYjmd5/uZ37cCbmF zhBGTGNbXYEWseXrqhdIgH5eL3LYuesO89moXDEXUlnu/SLgEN2N3XUtV/AIkJ3Zw4BO xRsqlByOAsBzFkWDPtCz2rpRDgUAwAOEujFyU= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=7NTlW4AVEaEifdUDNJKG5n5TSALhIocTrw3kCCpTxNA=; b=jQxysGOCu3biAzsapUETF0a7g1C3dwMtC8yAtmNYXNGr+orttoo+nUn5JSwUxYMizQ thdDxL8m4Cz7h8PxSUgmrDFcNatXgcCvRlugUIxNNeB/rwrH9JeovF+s48hdBA1ATkm3 dmjZIV9qxm+30XVGxs61bqvFFkwOeil7FDIAFgGTox+vtLoU8FW5wLR4gzS+sRbS8ltS 3WGFfeTtiRFaYm+G2Wffn6KiV4ZKge7SKhEuA/Xg/81gPW8FaVgGtaW08vc+JMY8c6wC Baatp4IK2d2Bk08LRwOOXwHGaP8WEfDhqF5+xCM6SakbF0OMDgDW/BGpTD1iFXvXTwX8 RC2g== X-Gm-Message-State: ACgBeo1qWF1eI9JnLw2XK72yMMg+oIyO+wT82jEnpD09NB/sG+QBQacw g7VVBmYPBr0iBXuDS0DZR10xDcH2fSTTjg== X-Google-Smtp-Source: AA6agR5R0KAg3JQMxVIXb/rgAzL7TYmscgkTXfoxMN9Q5ya69Oe6nssTMAHS9mjy5O7k72Bf0BpSEg== X-Received: by 2002:a05:6214:29cc:b0:477:38c2:105e with SMTP id gh12-20020a05621429cc00b0047738c2105emr26673145qvb.93.1662070680808; Thu, 01 Sep 2022 15:18:00 -0700 (PDT) Received: from joelboxx.c.googlers.com.com (228.221.150.34.bc.googleusercontent.com. [34.150.221.228]) by smtp.gmail.com with ESMTPSA id s16-20020ac85290000000b0034305a91aaesm11060794qtn.83.2022.09.01.15.18.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Sep 2022 15:18:00 -0700 (PDT) From: "Joel Fernandes (Google)" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, rushikesh.s.kadam@intel.com, urezki@gmail.com, neeraj.iitr10@gmail.com, frederic@kernel.org, paulmck@kernel.org, rostedt@goodmis.org, vineeth@bitbyteword.org, boqun.feng@gmail.com, "Joel Fernandes (Google)" Subject: [PATCH v5 05/18] rcu: Move trace_rcu_callback() before bypassing Date: Thu, 1 Sep 2022 22:17:07 +0000 Message-Id: <20220901221720.1105021-6-joel@joelfernandes.org> X-Mailer: git-send-email 2.37.2.789.g6183377224-goog In-Reply-To: <20220901221720.1105021-1-joel@joelfernandes.org> References: <20220901221720.1105021-1-joel@joelfernandes.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: rcu@vger.kernel.org If any CB is queued into the bypass list, then trace_rcu_callback() does not show it. This makes it not clear when a callback was actually queued, as you only end up getting a trace_rcu_invoke_callback() trace. Fix it by moving trace_rcu_callback() before trace_rcu_nocb_try_bypass(). Signed-off-by: Joel Fernandes (Google) --- kernel/rcu/tree.c | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 5ec97e3f7468..9fe581be8696 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -2809,10 +2809,7 @@ void call_rcu(struct rcu_head *head, rcu_callback_t func) } check_cb_ovld(rdp); - if (rcu_nocb_try_bypass(rdp, head, &was_alldone, flags)) - return; // Enqueued onto ->nocb_bypass, so just leave. - // If no-CBs CPU gets here, rcu_nocb_try_bypass() acquired ->nocb_lock. - rcu_segcblist_enqueue(&rdp->cblist, head); + if (__is_kvfree_rcu_offset((unsigned long)func)) trace_rcu_kvfree_callback(rcu_state.name, head, (unsigned long)func, @@ -2821,6 +2818,11 @@ void call_rcu(struct rcu_head *head, rcu_callback_t func) trace_rcu_callback(rcu_state.name, head, rcu_segcblist_n_cbs(&rdp->cblist)); + if (rcu_nocb_try_bypass(rdp, head, &was_alldone, flags)) + return; // Enqueued onto ->nocb_bypass, so just leave. + // If no-CBs CPU gets here, rcu_nocb_try_bypass() acquired ->nocb_lock. + rcu_segcblist_enqueue(&rdp->cblist, head); + trace_rcu_segcb_stats(&rdp->cblist, TPS("SegCBQueued")); /* Go handle any RCU core processing required. */ From patchwork Thu Sep 1 22:17:08 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joel Fernandes X-Patchwork-Id: 12963373 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7D16AC54EE9 for ; Thu, 1 Sep 2022 22:18:12 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234714AbiIAWSK (ORCPT ); Thu, 1 Sep 2022 18:18:10 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45844 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234622AbiIAWSF (ORCPT ); Thu, 1 Sep 2022 18:18:05 -0400 Received: from mail-qv1-xf34.google.com (mail-qv1-xf34.google.com [IPv6:2607:f8b0:4864:20::f34]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 590E7543C2 for ; Thu, 1 Sep 2022 15:18:03 -0700 (PDT) Received: by mail-qv1-xf34.google.com with SMTP id y15so182853qvn.2 for ; Thu, 01 Sep 2022 15:18:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc; bh=PromVDxaTL5WRddWriuWKU7Rbkxnb5+n/4/JObcP9Ec=; b=q2+8jSlhJdEfVIzEptXiAisnxq1C3aFrZ6nAKdtNOLjISF2DyKf8rkqBCnHXLKiIPn ZqNhjLktTPc+ksu7IOUXfo4uKhHokkVZPpP35Bpt5qEnMGtzr/Px+VpXfgqxhV7ehd7Q a0KiUP7IlfukMeOmvA7oazBvmh8t9k2K8ZHpo= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=PromVDxaTL5WRddWriuWKU7Rbkxnb5+n/4/JObcP9Ec=; b=MuO5nr6SW53cqqyTZ5ciGkuf32H9op5/RAC/u4vMmQxhCZ+pe1vCszdOGUkKlUUgYK oZKwh6nfm0aUGX+rv5LjU8KLpkjUILdpFN3bLC+wCN0G9w7DBY3X6JHi90Qx9L2Cy3b6 GL9/yKX9KvywTqq1e+2C3iUZ/HAdgCkTxu91E1AGqRGNsuwIxHbFGehPZC++qXhWMom0 wwwTRu6PNq1tbvtI4XQc71H0uhkayh9QpealLbYxQm0mNfWHesAvJIq1Rscfo610tprL M9ZINy8o3cdXq7XkwFGyfPV5KZejZ37smEcJpTUoF8HzS6kZUbdvf6eRqXN5Qc/yZTSi KY4A== X-Gm-Message-State: ACgBeo0avkZ5iaeVaylawYeA2pf1NqNzGjALSKrM5Wp9VEcY1Gxuctna aV4fhx3RuUubeO2TrkQVo+KnOxPekKWnsw== X-Google-Smtp-Source: AA6agR7O1opNtswPRY5hmUZ4DpbRs5kCINsqiSaFc91LD+aZxx/8YeroeDAhJbG76N+5kQjZfVpHDw== X-Received: by 2002:a05:6214:2aa2:b0:477:1882:3dc with SMTP id js2-20020a0562142aa200b00477188203dcmr26038206qvb.11.1662070681727; Thu, 01 Sep 2022 15:18:01 -0700 (PDT) Received: from joelboxx.c.googlers.com.com (228.221.150.34.bc.googleusercontent.com. [34.150.221.228]) by smtp.gmail.com with ESMTPSA id s16-20020ac85290000000b0034305a91aaesm11060794qtn.83.2022.09.01.15.18.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Sep 2022 15:18:01 -0700 (PDT) From: "Joel Fernandes (Google)" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, rushikesh.s.kadam@intel.com, urezki@gmail.com, neeraj.iitr10@gmail.com, frederic@kernel.org, paulmck@kernel.org, rostedt@goodmis.org, vineeth@bitbyteword.org, boqun.feng@gmail.com, "Joel Fernandes (Google)" Subject: [PATCH v5 06/18] rcu: Introduce call_rcu_lazy() API implementation Date: Thu, 1 Sep 2022 22:17:08 +0000 Message-Id: <20220901221720.1105021-7-joel@joelfernandes.org> X-Mailer: git-send-email 2.37.2.789.g6183377224-goog In-Reply-To: <20220901221720.1105021-1-joel@joelfernandes.org> References: <20220901221720.1105021-1-joel@joelfernandes.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: rcu@vger.kernel.org Implement timer-based RCU lazy callback batching. The batch is flushed whenever a certain amount of time has passed, or the batch on a particular CPU grows too big. Also memory pressure will flush it in a future patch. To handle several corner cases automagically (such as rcu_barrier() and hotplug), we re-use bypass lists to handle lazy CBs. The bypass list length has the lazy CB length included in it. A separate lazy CB length counter is also introduced to keep track of the number of lazy CBs. Suggested-by: Paul McKenney Signed-off-by: Joel Fernandes (Google) Reported-by: Frederic Weisbecker Signed-off-by: Joel Fernandes (Google) Signed-off-by: Joel Fernandes (Google) Signed-off-by: Joel Fernandes (Google) Signed-off-by: Joel Fernandes (Google) --- include/linux/rcupdate.h | 6 ++ kernel/rcu/Kconfig | 8 ++ kernel/rcu/rcu.h | 11 +++ kernel/rcu/rcu_segcblist.c | 2 +- kernel/rcu/tree.c | 130 +++++++++++++++--------- kernel/rcu/tree.h | 13 ++- kernel/rcu/tree_nocb.h | 198 ++++++++++++++++++++++++++++++------- 7 files changed, 280 insertions(+), 88 deletions(-) diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h index 08605ce7379d..82e8a07e0856 100644 --- a/include/linux/rcupdate.h +++ b/include/linux/rcupdate.h @@ -108,6 +108,12 @@ static inline int rcu_preempt_depth(void) #endif /* #else #ifdef CONFIG_PREEMPT_RCU */ +#ifdef CONFIG_RCU_LAZY +void call_rcu_lazy(struct rcu_head *head, rcu_callback_t func); +#else +#define call_rcu_lazy(head, func) call_rcu(head, func) +#endif + /* Internal to kernel */ void rcu_init(void); extern int rcu_scheduler_active; diff --git a/kernel/rcu/Kconfig b/kernel/rcu/Kconfig index d471d22a5e21..3128d01427cb 100644 --- a/kernel/rcu/Kconfig +++ b/kernel/rcu/Kconfig @@ -311,4 +311,12 @@ config TASKS_TRACE_RCU_READ_MB Say N here if you hate read-side memory barriers. Take the default if you are unsure. +config RCU_LAZY + bool "RCU callback lazy invocation functionality" + depends on RCU_NOCB_CPU + default n + help + To save power, batch RCU callbacks and flush after delay, memory + pressure or callback list growing too big. + endmenu # "RCU Subsystem" diff --git a/kernel/rcu/rcu.h b/kernel/rcu/rcu.h index be5979da07f5..94675f14efe8 100644 --- a/kernel/rcu/rcu.h +++ b/kernel/rcu/rcu.h @@ -474,6 +474,14 @@ enum rcutorture_type { INVALID_RCU_FLAVOR }; +#if defined(CONFIG_RCU_LAZY) +unsigned long rcu_lazy_get_jiffies_till_flush(void); +void rcu_lazy_set_jiffies_till_flush(unsigned long j); +#else +static inline unsigned long rcu_lazy_get_jiffies_till_flush(void) { return 0; } +static inline void rcu_lazy_set_jiffies_till_flush(unsigned long j) { } +#endif + #if defined(CONFIG_TREE_RCU) void rcutorture_get_gp_data(enum rcutorture_type test_type, int *flags, unsigned long *gp_seq); @@ -483,6 +491,8 @@ void do_trace_rcu_torture_read(const char *rcutorturename, unsigned long c_old, unsigned long c); void rcu_gp_set_torture_wait(int duration); +void rcu_force_call_rcu_to_lazy(bool force); + #else static inline void rcutorture_get_gp_data(enum rcutorture_type test_type, int *flags, unsigned long *gp_seq) @@ -501,6 +511,7 @@ void do_trace_rcu_torture_read(const char *rcutorturename, do { } while (0) #endif static inline void rcu_gp_set_torture_wait(int duration) { } +static inline void rcu_force_call_rcu_to_lazy(bool force) { } #endif #if IS_ENABLED(CONFIG_RCU_TORTURE_TEST) || IS_MODULE(CONFIG_RCU_TORTURE_TEST) diff --git a/kernel/rcu/rcu_segcblist.c b/kernel/rcu/rcu_segcblist.c index c54ea2b6a36b..55b50e592986 100644 --- a/kernel/rcu/rcu_segcblist.c +++ b/kernel/rcu/rcu_segcblist.c @@ -38,7 +38,7 @@ void rcu_cblist_enqueue(struct rcu_cblist *rclp, struct rcu_head *rhp) * element of the second rcu_cblist structure, but ensuring that the second * rcu_cblist structure, if initially non-empty, always appears non-empty * throughout the process. If rdp is NULL, the second rcu_cblist structure - * is instead initialized to empty. + * is instead initialized to empty. Also account for lazy_len for lazy CBs. */ void rcu_cblist_flush_enqueue(struct rcu_cblist *drclp, struct rcu_cblist *srclp, diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 9fe581be8696..aaced29a0a71 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -2728,47 +2728,8 @@ static void check_cb_ovld(struct rcu_data *rdp) raw_spin_unlock_rcu_node(rnp); } -/** - * call_rcu() - Queue an RCU callback for invocation after a grace period. - * @head: structure to be used for queueing the RCU updates. - * @func: actual callback function to be invoked after the grace period - * - * The callback function will be invoked some time after a full grace - * period elapses, in other words after all pre-existing RCU read-side - * critical sections have completed. However, the callback function - * might well execute concurrently with RCU read-side critical sections - * that started after call_rcu() was invoked. - * - * RCU read-side critical sections are delimited by rcu_read_lock() - * and rcu_read_unlock(), and may be nested. In addition, but only in - * v5.0 and later, regions of code across which interrupts, preemption, - * or softirqs have been disabled also serve as RCU read-side critical - * sections. This includes hardware interrupt handlers, softirq handlers, - * and NMI handlers. - * - * Note that all CPUs must agree that the grace period extended beyond - * all pre-existing RCU read-side critical section. On systems with more - * than one CPU, this means that when "func()" is invoked, each CPU is - * guaranteed to have executed a full memory barrier since the end of its - * last RCU read-side critical section whose beginning preceded the call - * to call_rcu(). It also means that each CPU executing an RCU read-side - * critical section that continues beyond the start of "func()" must have - * executed a memory barrier after the call_rcu() but before the beginning - * of that RCU read-side critical section. Note that these guarantees - * include CPUs that are offline, idle, or executing in user mode, as - * well as CPUs that are executing in the kernel. - * - * Furthermore, if CPU A invoked call_rcu() and CPU B invoked the - * resulting RCU callback function "func()", then both CPU A and CPU B are - * guaranteed to execute a full memory barrier during the time interval - * between the call to call_rcu() and the invocation of "func()" -- even - * if CPU A and CPU B are the same CPU (but again only if the system has - * more than one CPU). - * - * Implementation of these memory-ordering guarantees is described here: - * Documentation/RCU/Design/Memory-Ordering/Tree-RCU-Memory-Ordering.rst. - */ -void call_rcu(struct rcu_head *head, rcu_callback_t func) +static void +__call_rcu_common(struct rcu_head *head, rcu_callback_t func, bool lazy) { static atomic_t doublefrees; unsigned long flags; @@ -2818,7 +2779,7 @@ void call_rcu(struct rcu_head *head, rcu_callback_t func) trace_rcu_callback(rcu_state.name, head, rcu_segcblist_n_cbs(&rdp->cblist)); - if (rcu_nocb_try_bypass(rdp, head, &was_alldone, flags)) + if (rcu_nocb_try_bypass(rdp, head, &was_alldone, flags, lazy)) return; // Enqueued onto ->nocb_bypass, so just leave. // If no-CBs CPU gets here, rcu_nocb_try_bypass() acquired ->nocb_lock. rcu_segcblist_enqueue(&rdp->cblist, head); @@ -2833,8 +2794,86 @@ void call_rcu(struct rcu_head *head, rcu_callback_t func) local_irq_restore(flags); } } -EXPORT_SYMBOL_GPL(call_rcu); +#ifdef CONFIG_RCU_LAZY +/** + * call_rcu_lazy() - Lazily queue RCU callback for invocation after grace period. + * @head: structure to be used for queueing the RCU updates. + * @func: actual callback function to be invoked after the grace period + * + * The callback function will be invoked some time after a full grace + * period elapses, in other words after all pre-existing RCU read-side + * critical sections have completed. + * + * Use this API instead of call_rcu() if you don't mind the callback being + * invoked after very long periods of time on systems without memory pressure + * and on systems which are lightly loaded or mostly idle. + * + * Other than the extra delay in callbacks being invoked, this function is + * identical to, and reuses call_rcu()'s logic. Refer to call_rcu() for more + * details about memory ordering and other functionality. + */ +void call_rcu_lazy(struct rcu_head *head, rcu_callback_t func) +{ + return __call_rcu_common(head, func, true); +} +EXPORT_SYMBOL_GPL(call_rcu_lazy); +#endif + +static bool force_call_rcu_to_lazy; + +void rcu_force_call_rcu_to_lazy(bool force) +{ + if (IS_ENABLED(CONFIG_RCU_SCALE_TEST)) + WRITE_ONCE(force_call_rcu_to_lazy, force); +} +EXPORT_SYMBOL_GPL(rcu_force_call_rcu_to_lazy); + +/** + * call_rcu() - Queue an RCU callback for invocation after a grace period. + * @head: structure to be used for queueing the RCU updates. + * @func: actual callback function to be invoked after the grace period + * + * The callback function will be invoked some time after a full grace + * period elapses, in other words after all pre-existing RCU read-side + * critical sections have completed. However, the callback function + * might well execute concurrently with RCU read-side critical sections + * that started after call_rcu() was invoked. + * + * RCU read-side critical sections are delimited by rcu_read_lock() + * and rcu_read_unlock(), and may be nested. In addition, but only in + * v5.0 and later, regions of code across which interrupts, preemption, + * or softirqs have been disabled also serve as RCU read-side critical + * sections. This includes hardware interrupt handlers, softirq handlers, + * and NMI handlers. + * + * Note that all CPUs must agree that the grace period extended beyond + * all pre-existing RCU read-side critical section. On systems with more + * than one CPU, this means that when "func()" is invoked, each CPU is + * guaranteed to have executed a full memory barrier since the end of its + * last RCU read-side critical section whose beginning preceded the call + * to call_rcu(). It also means that each CPU executing an RCU read-side + * critical section that continues beyond the start of "func()" must have + * executed a memory barrier after the call_rcu() but before the beginning + * of that RCU read-side critical section. Note that these guarantees + * include CPUs that are offline, idle, or executing in user mode, as + * well as CPUs that are executing in the kernel. + * + * Furthermore, if CPU A invoked call_rcu() and CPU B invoked the + * resulting RCU callback function "func()", then both CPU A and CPU B are + * guaranteed to execute a full memory barrier during the time interval + * between the call to call_rcu() and the invocation of "func()" -- even + * if CPU A and CPU B are the same CPU (but again only if the system has + * more than one CPU). + * + * Implementation of these memory-ordering guarantees is described here: + * Documentation/RCU/Design/Memory-Ordering/Tree-RCU-Memory-Ordering.rst. + */ +void call_rcu(struct rcu_head *head, rcu_callback_t func) +{ + return __call_rcu_common(head, func, force_call_rcu_to_lazy); +} +EXPORT_SYMBOL_GPL(call_rcu); /* Maximum number of jiffies to wait before draining a batch. */ #define KFREE_DRAIN_JIFFIES (5 * HZ) @@ -3904,7 +3943,8 @@ static void rcu_barrier_entrain(struct rcu_data *rdp) rdp->barrier_head.func = rcu_barrier_callback; debug_rcu_head_queue(&rdp->barrier_head); rcu_nocb_lock(rdp); - WARN_ON_ONCE(!rcu_nocb_flush_bypass(rdp, NULL, jiffies)); + WARN_ON_ONCE(!rcu_nocb_flush_bypass(rdp, NULL, jiffies, false, + /* wake gp thread */ true)); if (rcu_segcblist_entrain(&rdp->cblist, &rdp->barrier_head)) { atomic_inc(&rcu_state.barrier_cpu_count); } else { @@ -4325,7 +4365,7 @@ void rcutree_migrate_callbacks(int cpu) my_rdp = this_cpu_ptr(&rcu_data); my_rnp = my_rdp->mynode; rcu_nocb_lock(my_rdp); /* irqs already disabled. */ - WARN_ON_ONCE(!rcu_nocb_flush_bypass(my_rdp, NULL, jiffies)); + WARN_ON_ONCE(!rcu_nocb_flush_bypass(my_rdp, NULL, jiffies, false, false)); raw_spin_lock_rcu_node(my_rnp); /* irqs already disabled. */ /* Leverage recent GPs and set GP for new callbacks. */ needwake = rcu_advance_cbs(my_rnp, rdp) || diff --git a/kernel/rcu/tree.h b/kernel/rcu/tree.h index d4a97e40ea9c..946d819b23fc 100644 --- a/kernel/rcu/tree.h +++ b/kernel/rcu/tree.h @@ -263,14 +263,18 @@ struct rcu_data { unsigned long last_fqs_resched; /* Time of last rcu_resched(). */ unsigned long last_sched_clock; /* Jiffies of last rcu_sched_clock_irq(). */ +#ifdef CONFIG_RCU_LAZY + long lazy_len; /* Length of buffered lazy callbacks. */ +#endif int cpu; }; /* Values for nocb_defer_wakeup field in struct rcu_data. */ #define RCU_NOCB_WAKE_NOT 0 #define RCU_NOCB_WAKE_BYPASS 1 -#define RCU_NOCB_WAKE 2 -#define RCU_NOCB_WAKE_FORCE 3 +#define RCU_NOCB_WAKE_LAZY 2 +#define RCU_NOCB_WAKE 3 +#define RCU_NOCB_WAKE_FORCE 4 #define RCU_JIFFIES_TILL_FORCE_QS (1 + (HZ > 250) + (HZ > 500)) /* For jiffies_till_first_fqs and */ @@ -440,9 +444,10 @@ static struct swait_queue_head *rcu_nocb_gp_get(struct rcu_node *rnp); static void rcu_nocb_gp_cleanup(struct swait_queue_head *sq); static void rcu_init_one_nocb(struct rcu_node *rnp); static bool rcu_nocb_flush_bypass(struct rcu_data *rdp, struct rcu_head *rhp, - unsigned long j); + unsigned long j, bool lazy, bool wakegp); static bool rcu_nocb_try_bypass(struct rcu_data *rdp, struct rcu_head *rhp, - bool *was_alldone, unsigned long flags); + bool *was_alldone, unsigned long flags, + bool lazy); static void __call_rcu_nocb_wake(struct rcu_data *rdp, bool was_empty, unsigned long flags); static int rcu_nocb_need_deferred_wakeup(struct rcu_data *rdp, int level); diff --git a/kernel/rcu/tree_nocb.h b/kernel/rcu/tree_nocb.h index 31068dd31315..7e97a7b6e046 100644 --- a/kernel/rcu/tree_nocb.h +++ b/kernel/rcu/tree_nocb.h @@ -256,6 +256,31 @@ static bool wake_nocb_gp(struct rcu_data *rdp, bool force) return __wake_nocb_gp(rdp_gp, rdp, force, flags); } +/* + * LAZY_FLUSH_JIFFIES decides the maximum amount of time that + * can elapse before lazy callbacks are flushed. Lazy callbacks + * could be flushed much earlier for a number of other reasons + * however, LAZY_FLUSH_JIFFIES will ensure no lazy callbacks are + * left unsubmitted to RCU after those many jiffies. + */ +#define LAZY_FLUSH_JIFFIES (10 * HZ) +unsigned long jiffies_till_flush = LAZY_FLUSH_JIFFIES; + +#ifdef CONFIG_RCU_LAZY +// To be called only from test code. +void rcu_lazy_set_jiffies_till_flush(unsigned long jif) +{ + jiffies_till_flush = jif; +} +EXPORT_SYMBOL(rcu_lazy_set_jiffies_till_flush); + +unsigned long rcu_lazy_get_jiffies_till_flush(void) +{ + return jiffies_till_flush; +} +EXPORT_SYMBOL(rcu_lazy_get_jiffies_till_flush); +#endif + /* * Arrange to wake the GP kthread for this NOCB group at some future * time when it is safe to do so. @@ -265,23 +290,39 @@ static void wake_nocb_gp_defer(struct rcu_data *rdp, int waketype, { unsigned long flags; struct rcu_data *rdp_gp = rdp->nocb_gp_rdp; + unsigned long mod_jif = 0; raw_spin_lock_irqsave(&rdp_gp->nocb_gp_lock, flags); /* - * Bypass wakeup overrides previous deferments. In case - * of callback storm, no need to wake up too early. + * Bypass and lazy wakeup overrides previous deferments. In case of + * callback storm, no need to wake up too early. */ - if (waketype == RCU_NOCB_WAKE_BYPASS) { - mod_timer(&rdp_gp->nocb_timer, jiffies + 2); - WRITE_ONCE(rdp_gp->nocb_defer_wakeup, waketype); - } else { + switch (waketype) { + case RCU_NOCB_WAKE_LAZY: + if (rdp->nocb_defer_wakeup != RCU_NOCB_WAKE_LAZY) + mod_jif = jiffies_till_flush; + break; + + case RCU_NOCB_WAKE_BYPASS: + mod_jif = 2; + break; + + case RCU_NOCB_WAKE: + case RCU_NOCB_WAKE_FORCE: + // For these, make it wake up the soonest if we + // were in a bypass or lazy sleep before. if (rdp_gp->nocb_defer_wakeup < RCU_NOCB_WAKE) - mod_timer(&rdp_gp->nocb_timer, jiffies + 1); - if (rdp_gp->nocb_defer_wakeup < waketype) - WRITE_ONCE(rdp_gp->nocb_defer_wakeup, waketype); + mod_jif = 1; + break; } + if (mod_jif) + mod_timer(&rdp_gp->nocb_timer, jiffies + mod_jif); + + if (rdp_gp->nocb_defer_wakeup < waketype) + WRITE_ONCE(rdp_gp->nocb_defer_wakeup, waketype); + raw_spin_unlock_irqrestore(&rdp_gp->nocb_gp_lock, flags); trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, reason); @@ -293,10 +334,13 @@ static void wake_nocb_gp_defer(struct rcu_data *rdp, int waketype, * proves to be initially empty, just return false because the no-CB GP * kthread may need to be awakened in this case. * + * Return true if there was something to be flushed and it succeeded, otherwise + * false. + * * Note that this function always returns true if rhp is NULL. */ static bool rcu_nocb_do_flush_bypass(struct rcu_data *rdp, struct rcu_head *rhp, - unsigned long j) + unsigned long j, bool lazy) { struct rcu_cblist rcl; @@ -310,7 +354,18 @@ static bool rcu_nocb_do_flush_bypass(struct rcu_data *rdp, struct rcu_head *rhp, /* Note: ->cblist.len already accounts for ->nocb_bypass contents. */ if (rhp) rcu_segcblist_inc_len(&rdp->cblist); /* Must precede enqueue. */ - rcu_cblist_flush_enqueue(&rcl, &rdp->nocb_bypass, rhp); + + /* + * If the new CB requested was a lazy one, queue it onto the main + * ->cblist so we can take advantage of a sooner grade period. + */ + if (lazy && rhp) { + rcu_cblist_flush_enqueue(&rcl, &rdp->nocb_bypass, NULL); + rcu_cblist_enqueue(&rcl, rhp); + } else { + rcu_cblist_flush_enqueue(&rcl, &rdp->nocb_bypass, rhp); + } + rcu_segcblist_insert_pend_cbs(&rdp->cblist, &rcl); WRITE_ONCE(rdp->nocb_bypass_first, j); rcu_nocb_bypass_unlock(rdp); @@ -326,13 +381,20 @@ static bool rcu_nocb_do_flush_bypass(struct rcu_data *rdp, struct rcu_head *rhp, * Note that this function always returns true if rhp is NULL. */ static bool rcu_nocb_flush_bypass(struct rcu_data *rdp, struct rcu_head *rhp, - unsigned long j) + unsigned long j, bool lazy, bool wake_gp) { + bool ret; + if (!rcu_rdp_is_offloaded(rdp)) return true; rcu_lockdep_assert_cblist_protected(rdp); rcu_nocb_bypass_lock(rdp); - return rcu_nocb_do_flush_bypass(rdp, rhp, j); + ret = rcu_nocb_do_flush_bypass(rdp, rhp, j, lazy); + + if (wake_gp) + wake_nocb_gp(rdp, true); + + return ret; } /* @@ -345,7 +407,7 @@ static void rcu_nocb_try_flush_bypass(struct rcu_data *rdp, unsigned long j) if (!rcu_rdp_is_offloaded(rdp) || !rcu_nocb_bypass_trylock(rdp)) return; - WARN_ON_ONCE(!rcu_nocb_do_flush_bypass(rdp, NULL, j)); + WARN_ON_ONCE(!rcu_nocb_do_flush_bypass(rdp, NULL, j, false)); } /* @@ -367,12 +429,14 @@ static void rcu_nocb_try_flush_bypass(struct rcu_data *rdp, unsigned long j) * there is only one CPU in operation. */ static bool rcu_nocb_try_bypass(struct rcu_data *rdp, struct rcu_head *rhp, - bool *was_alldone, unsigned long flags) + bool *was_alldone, unsigned long flags, + bool lazy) { unsigned long c; unsigned long cur_gp_seq; unsigned long j = jiffies; long ncbs = rcu_cblist_n_cbs(&rdp->nocb_bypass); + bool bypass_is_lazy = (ncbs == READ_ONCE(rdp->lazy_len)); lockdep_assert_irqs_disabled(); @@ -417,23 +481,29 @@ static bool rcu_nocb_try_bypass(struct rcu_data *rdp, struct rcu_head *rhp, // If there hasn't yet been all that many ->cblist enqueues // this jiffy, tell the caller to enqueue onto ->cblist. But flush // ->nocb_bypass first. - if (rdp->nocb_nobypass_count < nocb_nobypass_lim_per_jiffy) { + // Lazy CBs throttle this back and do immediate bypass queuing. + if (rdp->nocb_nobypass_count < nocb_nobypass_lim_per_jiffy && !lazy) { rcu_nocb_lock(rdp); *was_alldone = !rcu_segcblist_pend_cbs(&rdp->cblist); if (*was_alldone) trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, TPS("FirstQ")); - WARN_ON_ONCE(!rcu_nocb_flush_bypass(rdp, NULL, j)); + + WARN_ON_ONCE(!rcu_nocb_flush_bypass(rdp, NULL, j, false, false)); + WARN_ON_ONCE(rcu_cblist_n_cbs(&rdp->nocb_bypass)); return false; // Caller must enqueue the callback. } // If ->nocb_bypass has been used too long or is too full, // flush ->nocb_bypass to ->cblist. - if ((ncbs && j != READ_ONCE(rdp->nocb_bypass_first)) || + if ((ncbs && !bypass_is_lazy && j != READ_ONCE(rdp->nocb_bypass_first)) || + (ncbs && bypass_is_lazy && + (time_after(j, READ_ONCE(rdp->nocb_bypass_first) + jiffies_till_flush))) || ncbs >= qhimark) { rcu_nocb_lock(rdp); - if (!rcu_nocb_flush_bypass(rdp, rhp, j)) { + + if (!rcu_nocb_flush_bypass(rdp, rhp, j, lazy, false)) { *was_alldone = !rcu_segcblist_pend_cbs(&rdp->cblist); if (*was_alldone) trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, @@ -460,16 +530,29 @@ static bool rcu_nocb_try_bypass(struct rcu_data *rdp, struct rcu_head *rhp, // We need to use the bypass. rcu_nocb_wait_contended(rdp); rcu_nocb_bypass_lock(rdp); + ncbs = rcu_cblist_n_cbs(&rdp->nocb_bypass); rcu_segcblist_inc_len(&rdp->cblist); /* Must precede enqueue. */ rcu_cblist_enqueue(&rdp->nocb_bypass, rhp); + + if (IS_ENABLED(CONFIG_RCU_LAZY) && lazy) + WRITE_ONCE(rdp->lazy_len, rdp->lazy_len + 1); + if (!ncbs) { WRITE_ONCE(rdp->nocb_bypass_first, j); trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, TPS("FirstBQ")); } + rcu_nocb_bypass_unlock(rdp); smp_mb(); /* Order enqueue before wake. */ - if (ncbs) { + + // We had CBs in the bypass list before. There is nothing else to do if: + // There were only non-lazy CBs before, in this case, the bypass timer + // or GP-thread will handle the CBs including any new lazy ones. + // Or, the new CB is lazy and the old bypass-CBs were also lazy. In this + // case the old lazy timer would have been setup. When that expires, + // the new lazy one will be handled. + if (ncbs && (!bypass_is_lazy || lazy)) { local_irq_restore(flags); } else { // No-CBs GP kthread might be indefinitely asleep, if so, wake. @@ -478,6 +561,10 @@ static bool rcu_nocb_try_bypass(struct rcu_data *rdp, struct rcu_head *rhp, trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, TPS("FirstBQwake")); __call_rcu_nocb_wake(rdp, true, flags); + } else if (bypass_is_lazy && !lazy) { + trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, + TPS("FirstBQwakeLazy2Non")); + __call_rcu_nocb_wake(rdp, true, flags); } else { trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, TPS("FirstBQnoWake")); @@ -499,7 +586,7 @@ static void __call_rcu_nocb_wake(struct rcu_data *rdp, bool was_alldone, { unsigned long cur_gp_seq; unsigned long j; - long len; + long len, lazy_len, bypass_len; struct task_struct *t; // If we are being polled or there is no kthread, just leave. @@ -512,9 +599,16 @@ static void __call_rcu_nocb_wake(struct rcu_data *rdp, bool was_alldone, } // Need to actually to a wakeup. len = rcu_segcblist_n_cbs(&rdp->cblist); + bypass_len = rcu_cblist_n_cbs(&rdp->nocb_bypass); + lazy_len = READ_ONCE(rdp->lazy_len); if (was_alldone) { rdp->qlen_last_fqs_check = len; - if (!irqs_disabled_flags(flags)) { + // Only lazy CBs in bypass list + if (lazy_len && bypass_len == lazy_len) { + rcu_nocb_unlock_irqrestore(rdp, flags); + wake_nocb_gp_defer(rdp, RCU_NOCB_WAKE_LAZY, + TPS("WakeLazy")); + } else if (!irqs_disabled_flags(flags)) { /* ... if queue was empty ... */ rcu_nocb_unlock_irqrestore(rdp, flags); wake_nocb_gp(rdp, false); @@ -604,8 +698,8 @@ static void nocb_gp_sleep(struct rcu_data *my_rdp, int cpu) */ static void nocb_gp_wait(struct rcu_data *my_rdp) { - bool bypass = false; - long bypass_ncbs; + bool bypass = false, lazy = false; + long bypass_ncbs, lazy_ncbs; int __maybe_unused cpu = my_rdp->cpu; unsigned long cur_gp_seq; unsigned long flags; @@ -640,24 +734,41 @@ static void nocb_gp_wait(struct rcu_data *my_rdp) * won't be ignored for long. */ list_for_each_entry(rdp, &my_rdp->nocb_head_rdp, nocb_entry_rdp) { + bool flush_bypass = false; + trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, TPS("Check")); rcu_nocb_lock_irqsave(rdp, flags); lockdep_assert_held(&rdp->nocb_lock); bypass_ncbs = rcu_cblist_n_cbs(&rdp->nocb_bypass); - if (bypass_ncbs && + lazy_ncbs = READ_ONCE(rdp->lazy_len); + + if (bypass_ncbs && (lazy_ncbs == bypass_ncbs) && + (time_after(j, READ_ONCE(rdp->nocb_bypass_first) + jiffies_till_flush) || + bypass_ncbs > 2 * qhimark)) { + flush_bypass = true; + } else if (bypass_ncbs && (lazy_ncbs != bypass_ncbs) && (time_after(j, READ_ONCE(rdp->nocb_bypass_first) + 1) || bypass_ncbs > 2 * qhimark)) { - // Bypass full or old, so flush it. - (void)rcu_nocb_try_flush_bypass(rdp, j); - bypass_ncbs = rcu_cblist_n_cbs(&rdp->nocb_bypass); + flush_bypass = true; } else if (!bypass_ncbs && rcu_segcblist_empty(&rdp->cblist)) { rcu_nocb_unlock_irqrestore(rdp, flags); continue; /* No callbacks here, try next. */ } + + if (flush_bypass) { + // Bypass full or old, so flush it. + (void)rcu_nocb_try_flush_bypass(rdp, j); + bypass_ncbs = rcu_cblist_n_cbs(&rdp->nocb_bypass); + lazy_ncbs = READ_ONCE(rdp->lazy_len); + } + if (bypass_ncbs) { trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, - TPS("Bypass")); - bypass = true; + bypass_ncbs == lazy_ncbs ? TPS("Lazy") : TPS("Bypass")); + if (bypass_ncbs == lazy_ncbs) + lazy = true; + else + bypass = true; } rnp = rdp->mynode; @@ -705,12 +816,21 @@ static void nocb_gp_wait(struct rcu_data *my_rdp) my_rdp->nocb_gp_gp = needwait_gp; my_rdp->nocb_gp_seq = needwait_gp ? wait_gp_seq : 0; - if (bypass && !rcu_nocb_poll) { - // At least one child with non-empty ->nocb_bypass, so set - // timer in order to avoid stranding its callbacks. - wake_nocb_gp_defer(my_rdp, RCU_NOCB_WAKE_BYPASS, - TPS("WakeBypassIsDeferred")); + // At least one child with non-empty ->nocb_bypass, so set + // timer in order to avoid stranding its callbacks. + if (!rcu_nocb_poll) { + // If bypass list only has lazy CBs. Add a deferred + // lazy wake up. + if (lazy && !bypass) { + wake_nocb_gp_defer(my_rdp, RCU_NOCB_WAKE_LAZY, + TPS("WakeLazyIsDeferred")); + // Otherwise add a deferred bypass wake up. + } else if (bypass) { + wake_nocb_gp_defer(my_rdp, RCU_NOCB_WAKE_BYPASS, + TPS("WakeBypassIsDeferred")); + } } + if (rcu_nocb_poll) { /* Polling, so trace if first poll in the series. */ if (gotcbs) @@ -1036,7 +1156,7 @@ static long rcu_nocb_rdp_deoffload(void *arg) * return false, which means that future calls to rcu_nocb_try_bypass() * will refuse to put anything into the bypass. */ - WARN_ON_ONCE(!rcu_nocb_flush_bypass(rdp, NULL, jiffies)); + WARN_ON_ONCE(!rcu_nocb_flush_bypass(rdp, NULL, jiffies, false, false)); /* * Start with invoking rcu_core() early. This way if the current thread * happens to preempt an ongoing call to rcu_core() in the middle, @@ -1290,6 +1410,7 @@ static void __init rcu_boot_init_nocb_percpu_data(struct rcu_data *rdp) raw_spin_lock_init(&rdp->nocb_gp_lock); timer_setup(&rdp->nocb_timer, do_nocb_deferred_wakeup_timer, 0); rcu_cblist_init(&rdp->nocb_bypass); + WRITE_ONCE(rdp->lazy_len, 0); mutex_init(&rdp->nocb_gp_kthread_mutex); } @@ -1571,13 +1692,14 @@ static void rcu_init_one_nocb(struct rcu_node *rnp) } static bool rcu_nocb_flush_bypass(struct rcu_data *rdp, struct rcu_head *rhp, - unsigned long j) + unsigned long j, bool lazy, bool wakegp) { return true; } static bool rcu_nocb_try_bypass(struct rcu_data *rdp, struct rcu_head *rhp, - bool *was_alldone, unsigned long flags) + bool *was_alldone, unsigned long flags, + bool lazy) { return false; } From patchwork Thu Sep 1 22:17:09 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joel Fernandes X-Patchwork-Id: 12963371 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 67DFFC54EE9 for ; Thu, 1 Sep 2022 22:18:07 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234274AbiIAWSF (ORCPT ); Thu, 1 Sep 2022 18:18:05 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45810 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233686AbiIAWSD (ORCPT ); Thu, 1 Sep 2022 18:18:03 -0400 Received: from mail-qv1-xf30.google.com (mail-qv1-xf30.google.com [IPv6:2607:f8b0:4864:20::f30]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C8C955809B for ; Thu, 1 Sep 2022 15:18:02 -0700 (PDT) Received: by mail-qv1-xf30.google.com with SMTP id l5so143253qvs.13 for ; Thu, 01 Sep 2022 15:18:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date; bh=5Y5kbAl2ZL0hvyMP3+b8/GM5tIuNKa2oHKBHOOJke14=; b=DLk0B/+Kjl2sEDnaJ5HNoOkF5QFsAHl3d/WO0kQdWt7MwUX6E9cr00jSheEW6r0ohh eKqnQ2mBJFjJ3kK0CphbuerOCVjht107mlZlGLAxmmWEJQT3dq2Zm0nFUcX0KPUZ68Qt UbZiWOQ9CWe4p662dEMWk9EB4AQrJh0Qr8MdY= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date; bh=5Y5kbAl2ZL0hvyMP3+b8/GM5tIuNKa2oHKBHOOJke14=; b=QdrS3njLdULSc0ep+td3HuyhS/qF20BbN6acPC4uFvaekvOZ8nlfC2d9fFHzGEuRrQ f6s6eep1iom4LgHyxx15+zcYYIiXrLKvtedMc9c6kOZo46V0okodoX3ckEtUtHD6py35 nbDDhPJyVloUI7Xdk/jhpy24Wf19fD/YTddz8hYIWWbVQHqNzXXUTALouN+PKxuZQbDv 83A+9MtGjDuSgOfK4X5yecGI/URuqiCPZ17/8OQdpETRhOT529ScdWs+afafKd1fRPlO kN5PIMDb+1bz+KtHNNKhC/vUp1SUy9BXCN5NaXeV8gVtDnOcVsAY5e7zpGECCntF6fTO 2kQw== X-Gm-Message-State: ACgBeo0khpQm64A9P8iwzLuougRb+tbiDC7/NWX0pB9mOqcLMCriRGAQ jIqM/x4KA+X276jeaYyCeQzKnxLPGJNiow== X-Google-Smtp-Source: AA6agR5SVRH/RM5xAYfxYhk8E1h0qVAjY9FUK/2Tu/dAn793HN5Tons0Nj46w1jA+Idc8q2jHDCtow== X-Received: by 2002:ad4:5bc2:0:b0:499:2c2c:1b0b with SMTP id t2-20020ad45bc2000000b004992c2c1b0bmr4677172qvt.115.1662070682335; Thu, 01 Sep 2022 15:18:02 -0700 (PDT) Received: from joelboxx.c.googlers.com.com (228.221.150.34.bc.googleusercontent.com. [34.150.221.228]) by smtp.gmail.com with ESMTPSA id s16-20020ac85290000000b0034305a91aaesm11060794qtn.83.2022.09.01.15.18.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Sep 2022 15:18:01 -0700 (PDT) From: "Joel Fernandes (Google)" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, rushikesh.s.kadam@intel.com, urezki@gmail.com, neeraj.iitr10@gmail.com, frederic@kernel.org, paulmck@kernel.org, rostedt@goodmis.org, vineeth@bitbyteword.org, boqun.feng@gmail.com, Joel Fernandes Subject: [PATCH v5 07/18] rcu: shrinker for lazy rcu Date: Thu, 1 Sep 2022 22:17:09 +0000 Message-Id: <20220901221720.1105021-8-joel@joelfernandes.org> X-Mailer: git-send-email 2.37.2.789.g6183377224-goog In-Reply-To: <20220901221720.1105021-1-joel@joelfernandes.org> References: <20220901221720.1105021-1-joel@joelfernandes.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: rcu@vger.kernel.org From: Vineeth Pillai The shrinker is used to speed up the free'ing of memory potentially held by RCU lazy callbacks. RCU kernel module test cases show this to be effective. Test is introduced in a later patch. Signed-off-by: Vineeth Pillai Signed-off-by: Joel Fernandes (Google) --- kernel/rcu/tree_nocb.h | 52 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 52 insertions(+) diff --git a/kernel/rcu/tree_nocb.h b/kernel/rcu/tree_nocb.h index 7e97a7b6e046..560ba87911c5 100644 --- a/kernel/rcu/tree_nocb.h +++ b/kernel/rcu/tree_nocb.h @@ -1333,6 +1333,55 @@ int rcu_nocb_cpu_offload(int cpu) } EXPORT_SYMBOL_GPL(rcu_nocb_cpu_offload); +static unsigned long +lazy_rcu_shrink_count(struct shrinker *shrink, struct shrink_control *sc) +{ + int cpu; + unsigned long count = 0; + + /* Snapshot count of all CPUs */ + for_each_possible_cpu(cpu) { + struct rcu_data *rdp = per_cpu_ptr(&rcu_data, cpu); + + count += READ_ONCE(rdp->lazy_len); + } + + return count ? count : SHRINK_EMPTY; +} + +static unsigned long +lazy_rcu_shrink_scan(struct shrinker *shrink, struct shrink_control *sc) +{ + int cpu; + unsigned long flags; + unsigned long count = 0; + + /* Snapshot count of all CPUs */ + for_each_possible_cpu(cpu) { + struct rcu_data *rdp = per_cpu_ptr(&rcu_data, cpu); + int _count = READ_ONCE(rdp->lazy_len); + + if (_count == 0) + continue; + rcu_nocb_lock_irqsave(rdp, flags); + WRITE_ONCE(rdp->lazy_len, 0); + rcu_nocb_unlock_irqrestore(rdp, flags); + wake_nocb_gp(rdp, false); + sc->nr_to_scan -= _count; + count += _count; + if (sc->nr_to_scan <= 0) + break; + } + return count ? count : SHRINK_STOP; +} + +static struct shrinker lazy_rcu_shrinker = { + .count_objects = lazy_rcu_shrink_count, + .scan_objects = lazy_rcu_shrink_scan, + .batch = 0, + .seeks = DEFAULT_SEEKS, +}; + void __init rcu_init_nohz(void) { int cpu; @@ -1367,6 +1416,9 @@ void __init rcu_init_nohz(void) if (!rcu_state.nocb_is_setup) return; + if (register_shrinker(&lazy_rcu_shrinker, "rcu-lazy")) + pr_err("Failed to register lazy_rcu shrinker!\n"); + #if defined(CONFIG_NO_HZ_FULL) if (tick_nohz_full_running) cpumask_or(rcu_nocb_mask, rcu_nocb_mask, tick_nohz_full_mask); From patchwork Thu Sep 1 22:17:10 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joel Fernandes X-Patchwork-Id: 12963375 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B6FA2C54EE9 for ; Thu, 1 Sep 2022 22:18:23 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233795AbiIAWSV (ORCPT ); Thu, 1 Sep 2022 18:18:21 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45908 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234299AbiIAWSJ (ORCPT ); Thu, 1 Sep 2022 18:18:09 -0400 Received: from mail-qt1-x82c.google.com (mail-qt1-x82c.google.com [IPv6:2607:f8b0:4864:20::82c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3DF54580BE for ; Thu, 1 Sep 2022 15:18:04 -0700 (PDT) Received: by mail-qt1-x82c.google.com with SMTP id c20so221148qtw.8 for ; Thu, 01 Sep 2022 15:18:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc; bh=sbbotK/X6hyBCxf4X8eHnMZLe8+ikwZ/eJHvPr5PB5Y=; b=kPA5Qz5YkoMPW7SkgX+l0Ko5jkDfQyr3bsOZBOyAN8nMIw0Iw9j1I6H4sdDZvxhdxW 7wsvjTl8ccTmCT2hpTRxUIQlxHTAEhDAge3obd9j18OBY9X0NXJOi3EOnRakNGRSgOXu 1ZyJ9+IgAB/0g4Y3K5cwMXnAb9TFR59cfhq4s= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=sbbotK/X6hyBCxf4X8eHnMZLe8+ikwZ/eJHvPr5PB5Y=; b=ajEfKI39zyxLcfHeWVcv3rTUxVk4e2uAtphv4kHdzXxxbJAhi3gIMtp02sN6HvFvxJ pbeeufQhU8TmWSnx5PMgP0EpR+GQiDhXTrHjMh49BtOS03a5pMdlcYa9xEcUdkXXt3vt UdwjFsAIPLWLcYqcJl1ZdMedf0vDoqVKoufJcXJz4LOW+KJzOQ/Bo0JT9KsStTr0QkYJ tNjmQrTzGZLMuSHenpXmuwzToiTA9+dPsXw2tKip1UympznJ9jayrshQEyscQElPAwOa cwMN1SqT5g+V+y5eLnuucCUQCFQHm7Pg0DsiKBfVnS8SNZ7ZFfr1eV8Pn3UZboCgaxW/ 1uWQ== X-Gm-Message-State: ACgBeo2K3Ed8vQcMldz34by+RsAEaUQfuFkR5m4OBuxORwALLOImZvjz fj3YLSYzR5k6eeaLKBv+SPlI7VPe09uH9A== X-Google-Smtp-Source: AA6agR6KszQ19UJa+Vd0R4VFnWWwwBVRqP48H2wjT6CRlwWeihtQuS7Q4n4Mfo+Tvf1h2BWMZDzvKA== X-Received: by 2002:a05:622a:143:b0:344:95bf:8f02 with SMTP id v3-20020a05622a014300b0034495bf8f02mr24705082qtw.202.1662070683012; Thu, 01 Sep 2022 15:18:03 -0700 (PDT) Received: from joelboxx.c.googlers.com.com (228.221.150.34.bc.googleusercontent.com. [34.150.221.228]) by smtp.gmail.com with ESMTPSA id s16-20020ac85290000000b0034305a91aaesm11060794qtn.83.2022.09.01.15.18.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Sep 2022 15:18:02 -0700 (PDT) From: "Joel Fernandes (Google)" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, rushikesh.s.kadam@intel.com, urezki@gmail.com, neeraj.iitr10@gmail.com, frederic@kernel.org, paulmck@kernel.org, rostedt@goodmis.org, vineeth@bitbyteword.org, boqun.feng@gmail.com, "Joel Fernandes (Google)" Subject: [PATCH v5 08/18] rcu: Add per-CB tracing for queuing, flush and invocation. Date: Thu, 1 Sep 2022 22:17:10 +0000 Message-Id: <20220901221720.1105021-9-joel@joelfernandes.org> X-Mailer: git-send-email 2.37.2.789.g6183377224-goog In-Reply-To: <20220901221720.1105021-1-joel@joelfernandes.org> References: <20220901221720.1105021-1-joel@joelfernandes.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: rcu@vger.kernel.org This is very useful to debug whether call_rcu_lazy() is working correctly. In the future, any BPF tools could also be modified to read the new information. The additional tracing is enabled by a new CONFIG_RCU_TRACE_CB and is kept disabled by default. Signed-off-by: Joel Fernandes (Google) Reported-by: kernel test robot Reported-by: kernel test robot --- include/linux/types.h | 44 ++++++++++++++++++++++++ include/trace/events/rcu.h | 69 +++++++++++++++++++++++++++++++++++--- kernel/rcu/Kconfig | 11 ++++++ kernel/rcu/rcu_segcblist.c | 21 ++++++++++++ kernel/rcu/rcu_segcblist.h | 8 +++++ kernel/rcu/tree.c | 46 +++++++++++++++++++++++-- kernel/rcu/tree_nocb.h | 26 +++++++++++++- 7 files changed, 217 insertions(+), 8 deletions(-) diff --git a/include/linux/types.h b/include/linux/types.h index ea8cf60a8a79..47501fdfd3c1 100644 --- a/include/linux/types.h +++ b/include/linux/types.h @@ -198,10 +198,51 @@ struct ustat { char f_fpack[6]; }; +#ifdef CONFIG_RCU_TRACE_CB +/* + * Debug information that a caller can store within a callback_head. + * Its expected to provide at least 12 bytes before BUILD_BUG starts + * complaining. + */ +enum cb_debug_flags { + CB_DEBUG_KFREE, + CB_DEBUG_LAZY, + CB_DEBUG_BYPASS, + + // A new non-lazy CB showed up and we decided to not use + // the bypass list for it. So we flushed the old ones. + CB_DEBUG_NON_LAZY_FLUSHED, + + // We decided to use the bypass list but had to flush + // the old bypass CBs because they got too old or too big. + CB_DEBUG_BYPASS_FLUSHED, + CB_DEBUG_BYPASS_LAZY_FLUSHED, + + // The GP thread flushed the bypass CBs if they got old or big. + CB_DEBUG_GPTHREAD_FLUSHED, + + // De-offload from NOCB mode. + CB_DEBUG_DEOFFLOAD_FLUSHED, + + // rcu_barrier() flushes lazy/bypass CBs for CB exec ordering. + CB_DEBUG_BARRIER_FLUSHED +}; + +struct cb_debug_info { + // 16-bit jiffie deltas can provide about 60 seconds of resolution + // at HZ=1000 before wrapping. That's enough for debug. + u16 cb_queue_jiff; + u16 first_bp_jiff; + u16 cb_flush_jiff; + enum cb_debug_flags flags:16; +}; +#endif + /** * struct callback_head - callback structure for use with RCU and task_work * @next: next update requests in a list * @func: actual update function to call after the grace period. + * @di: debug information that can be stored. * * The struct is aligned to size of pointer. On most architectures it happens * naturally due ABI requirements, but some architectures (like CRIS) have @@ -220,6 +261,9 @@ struct ustat { struct callback_head { struct callback_head *next; void (*func)(struct callback_head *head); +#ifdef CONFIG_RCU_TRACE_CB + struct cb_debug_info di; +#endif } __attribute__((aligned(sizeof(void *)))); #define rcu_head callback_head diff --git a/include/trace/events/rcu.h b/include/trace/events/rcu.h index 90b2fb0292cb..dd6baec6745c 100644 --- a/include/trace/events/rcu.h +++ b/include/trace/events/rcu.h @@ -630,24 +630,85 @@ TRACE_EVENT_RCU(rcu_batch_start, */ TRACE_EVENT_RCU(rcu_invoke_callback, - TP_PROTO(const char *rcuname, struct rcu_head *rhp), + TP_PROTO(const char *rcuname, struct rcu_head *rhp +#ifdef CONFIG_RCU_TRACE_CB + , unsigned long jiffies_first +#endif + ), - TP_ARGS(rcuname, rhp), + TP_ARGS(rcuname, rhp +#ifdef CONFIG_RCU_TRACE_CB + , jiffies_first +#endif + ), TP_STRUCT__entry( __field(const char *, rcuname) __field(void *, rhp) __field(void *, func) +#ifdef CONFIG_RCU_TRACE_CB + __field(u16, cb_debug_flags) + __field(int, cb_queue_exec_latency) + __array(char, bypass_flush_reason, 32) + __field(int, cb_bypass_slack) + __field(int, cb_queue_flush_latency) +#endif ), TP_fast_assign( __entry->rcuname = rcuname; __entry->rhp = rhp; __entry->func = rhp->func; +#ifdef CONFIG_RCU_TRACE_CB + __entry->cb_debug_flags = rhp->di.flags; + __entry->cb_queue_exec_latency = + (jiffies - jiffies_first) - rhp->di.cb_queue_jiff, + + /* The following 3 fields are valid only for bypass/lazy CBs. */ + __entry->cb_bypass_slack = + (rhp->di.flags & BIT(CB_DEBUG_BYPASS)) ? + rhp->di.cb_queue_jiff - rhp->di.first_bp_jiff : 0, + __entry->cb_queue_flush_latency = + (rhp->di.flags & BIT(CB_DEBUG_BYPASS)) ? + rhp->di.cb_flush_jiff - rhp->di.cb_queue_jiff : 0; + + + if (__entry->cb_debug_flags & BIT(CB_DEBUG_NON_LAZY_FLUSHED)) + strcpy(__entry->bypass_flush_reason, "non-lazy"); + else if (__entry->cb_debug_flags & BIT(CB_DEBUG_BYPASS_FLUSHED)) + strcpy(__entry->bypass_flush_reason, "bypass"); + else if (__entry->cb_debug_flags & BIT(CB_DEBUG_BYPASS_LAZY_FLUSHED)) + strcpy(__entry->bypass_flush_reason, "bypass-lazy"); + else if (__entry->cb_debug_flags & BIT(CB_DEBUG_GPTHREAD_FLUSHED)) + strcpy(__entry->bypass_flush_reason, "gpthread"); + else if (__entry->cb_debug_flags & BIT(CB_DEBUG_BARRIER_FLUSHED)) + strcpy(__entry->bypass_flush_reason, "rcu_barrier"); + else if (__entry->cb_debug_flags & BIT(CB_DEBUG_DEOFFLOAD_FLUSHED)) + strcpy(__entry->bypass_flush_reason, "deoffload"); +#endif ), - TP_printk("%s rhp=%p func=%ps", - __entry->rcuname, __entry->rhp, __entry->func) + TP_printk("%s rhp=%p func=%ps" +#ifdef CONFIG_RCU_TRACE_CB + " lazy=%d " + "bypass=%d " + "flush_reason=%s " + "queue_exec_latency=%d " + "bypass_slack=%d " + "queue_flush_latency=%d" +#endif + , + __entry->rcuname, __entry->rhp, __entry->func +#ifdef CONFIG_RCU_TRACE_CB + , + !!(__entry->cb_debug_flags & BIT(CB_DEBUG_LAZY)), + !!(__entry->cb_debug_flags & BIT(CB_DEBUG_BYPASS)), + __entry->bypass_flush_reason, + __entry->cb_queue_exec_latency, + __entry->cb_bypass_slack, + __entry->cb_queue_flush_latency +#endif + ) ); /* diff --git a/kernel/rcu/Kconfig b/kernel/rcu/Kconfig index 3128d01427cb..cd6ff63517f4 100644 --- a/kernel/rcu/Kconfig +++ b/kernel/rcu/Kconfig @@ -319,4 +319,15 @@ config RCU_LAZY To save power, batch RCU callbacks and flush after delay, memory pressure or callback list growing too big. +config RCU_TRACE_CB + bool "Enable additional callback tracing" + depends on RCU_LAZY + default n + help + Enable additional callback tracing for RCU. Currently only + tracing for lazy/bypass callbacks is done. In the future, it could be + extended to non-lazy callbacks as well. The trace point contains + detailed information about callback flushing reason and execution + time. This information can be retrieved by BPF tools via tracepoints. + endmenu # "RCU Subsystem" diff --git a/kernel/rcu/rcu_segcblist.c b/kernel/rcu/rcu_segcblist.c index 55b50e592986..8f0b024f7bc5 100644 --- a/kernel/rcu/rcu_segcblist.c +++ b/kernel/rcu/rcu_segcblist.c @@ -32,6 +32,27 @@ void rcu_cblist_enqueue(struct rcu_cblist *rclp, struct rcu_head *rhp) WRITE_ONCE(rclp->len, rclp->len + 1); } +/* + * Set a debug flag on all CBs in the unsegmented cblist @rclp. + * + * The callback causing the flush. Should be NULL if its a timer that does it. + */ +#ifdef CONFIG_RCU_TRACE_CB +void rcu_cblist_set_flush(struct rcu_cblist *rcl, + enum cb_debug_flags flags, + unsigned long flush_jiff) +{ + if (!rcl || !rcl->head) + return; + + for (struct rcu_head *head = rcl->head; + head && head != *(rcl->tail); head = head->next) { + head->di.flags |= flags; + head->di.cb_flush_jiff = flush_jiff; + } +} +#endif + /* * Flush the second rcu_cblist structure onto the first one, obliterating * any contents of the first. If rhp is non-NULL, enqueue it as the sole diff --git a/kernel/rcu/rcu_segcblist.h b/kernel/rcu/rcu_segcblist.h index 431cee212467..ac1f34024b84 100644 --- a/kernel/rcu/rcu_segcblist.h +++ b/kernel/rcu/rcu_segcblist.h @@ -53,6 +53,14 @@ static inline long rcu_segcblist_n_cbs(struct rcu_segcblist *rsclp) #endif } +#ifdef CONFIG_RCU_TRACE_CB +void rcu_cblist_set_flush(struct rcu_cblist *rcl, + enum cb_debug_flags flags, + unsigned long flush_jiff); +#else +#define rcu_cblist_set_flush(...) +#endif + static inline void rcu_segcblist_set_flags(struct rcu_segcblist *rsclp, int flags) { diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index aaced29a0a71..8111d9f37621 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -2179,6 +2179,12 @@ int rcutree_dead_cpu(unsigned int cpu) return 0; } +static unsigned long cb_debug_jiffies_first; + +#if defined(CONFIG_RCU_TRACE_CB) && defined(CONFIG_RCU_LAZY) +extern unsigned long jiffies_till_flush; +#endif + /* * Invoke any RCU callbacks that have made it to the end of their grace * period. Throttle as specified by rdp->blimit. @@ -2241,7 +2247,12 @@ static void rcu_do_batch(struct rcu_data *rdp) debug_rcu_head_unqueue(rhp); rcu_lock_acquire(&rcu_callback_map); - trace_rcu_invoke_callback(rcu_state.name, rhp); + + trace_rcu_invoke_callback(rcu_state.name, rhp +#ifdef CONFIG_RCU_TRACE_CB + , READ_ONCE(cb_debug_jiffies_first) +#endif + ); f = rhp->func; WRITE_ONCE(rhp->func, (rcu_callback_t)0L); @@ -2736,6 +2747,17 @@ __call_rcu_common(struct rcu_head *head, rcu_callback_t func, bool lazy) struct rcu_data *rdp; bool was_alldone; + /* + * For CB debugging, record the starting reference to jiffies while + * making sure it does not wrap around. The starting reference is + * used to calculate 16-bit jiffie deltas. To be safe, reset the + * reference once the delta exceeds 15-bits worth. + */ + if (IS_ENABLED(CONFIG_RCU_TRACE_CB) && + (!READ_ONCE(cb_debug_jiffies_first) || + (jiffies - READ_ONCE(cb_debug_jiffies_first) > (1 << 15)))) + WRITE_ONCE(cb_debug_jiffies_first, jiffies); + /* Misaligned rcu_head! */ WARN_ON_ONCE((unsigned long)head & (sizeof(void *) - 1)); @@ -2771,14 +2793,29 @@ __call_rcu_common(struct rcu_head *head, rcu_callback_t func, bool lazy) check_cb_ovld(rdp); - if (__is_kvfree_rcu_offset((unsigned long)func)) +#ifdef CONFIG_RCU_TRACE_CB + head->di.flags = 0; + WARN_ON_ONCE(jiffies - READ_ONCE(cb_debug_jiffies_first) > 65535); + head->di.cb_queue_jiff = (u16)(jiffies - READ_ONCE(cb_debug_jiffies_first)); +#endif + + if (__is_kvfree_rcu_offset((unsigned long)func)) { trace_rcu_kvfree_callback(rcu_state.name, head, (unsigned long)func, rcu_segcblist_n_cbs(&rdp->cblist)); - else +#ifdef CONFIG_RCU_TRACE_CB + head->di.flags |= BIT(CB_DEBUG_KFREE); +#endif + } else { trace_rcu_callback(rcu_state.name, head, rcu_segcblist_n_cbs(&rdp->cblist)); +#ifdef CONFIG_RCU_TRACE_CB + if (lazy) + head->di.flags |= BIT(CB_DEBUG_LAZY); +#endif + } + if (rcu_nocb_try_bypass(rdp, head, &was_alldone, flags, lazy)) return; // Enqueued onto ->nocb_bypass, so just leave. // If no-CBs CPU gets here, rcu_nocb_try_bypass() acquired ->nocb_lock. @@ -3943,6 +3980,9 @@ static void rcu_barrier_entrain(struct rcu_data *rdp) rdp->barrier_head.func = rcu_barrier_callback; debug_rcu_head_queue(&rdp->barrier_head); rcu_nocb_lock(rdp); + + rcu_cblist_set_flush(&rdp->nocb_bypass, BIT(CB_DEBUG_BARRIER_FLUSHED), + (jiffies - READ_ONCE(cb_debug_jiffies_first))); WARN_ON_ONCE(!rcu_nocb_flush_bypass(rdp, NULL, jiffies, false, /* wake gp thread */ true)); if (rcu_segcblist_entrain(&rdp->cblist, &rdp->barrier_head)) { diff --git a/kernel/rcu/tree_nocb.h b/kernel/rcu/tree_nocb.h index 560ba87911c5..ee5924ba2f3b 100644 --- a/kernel/rcu/tree_nocb.h +++ b/kernel/rcu/tree_nocb.h @@ -355,6 +355,11 @@ static bool rcu_nocb_do_flush_bypass(struct rcu_data *rdp, struct rcu_head *rhp, if (rhp) rcu_segcblist_inc_len(&rdp->cblist); /* Must precede enqueue. */ + /* The lazy CBs are being flushed, but a new one might be enqueued. */ +#ifdef CONFIG_RCU_TRACE_CB + WARN_ON_ONCE(rhp && lazy != !!(rhp->di.flags & BIT(CB_DEBUG_LAZY))); +#endif + /* * If the new CB requested was a lazy one, queue it onto the main * ->cblist so we can take advantage of a sooner grade period. @@ -407,6 +412,10 @@ static void rcu_nocb_try_flush_bypass(struct rcu_data *rdp, unsigned long j) if (!rcu_rdp_is_offloaded(rdp) || !rcu_nocb_bypass_trylock(rdp)) return; + + rcu_cblist_set_flush(&rdp->nocb_bypass, BIT(CB_DEBUG_GPTHREAD_FLUSHED), + (j - READ_ONCE(cb_debug_jiffies_first))); + WARN_ON_ONCE(!rcu_nocb_do_flush_bypass(rdp, NULL, j, false)); } @@ -489,8 +498,10 @@ static bool rcu_nocb_try_bypass(struct rcu_data *rdp, struct rcu_head *rhp, trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, TPS("FirstQ")); - WARN_ON_ONCE(!rcu_nocb_flush_bypass(rdp, NULL, j, false, false)); + rcu_cblist_set_flush(&rdp->nocb_bypass, BIT(CB_DEBUG_NON_LAZY_FLUSHED), + (j - READ_ONCE(cb_debug_jiffies_first))); + WARN_ON_ONCE(!rcu_nocb_flush_bypass(rdp, NULL, j, false, false)); WARN_ON_ONCE(rcu_cblist_n_cbs(&rdp->nocb_bypass)); return false; // Caller must enqueue the callback. } @@ -503,6 +514,10 @@ static bool rcu_nocb_try_bypass(struct rcu_data *rdp, struct rcu_head *rhp, ncbs >= qhimark) { rcu_nocb_lock(rdp); + rcu_cblist_set_flush(&rdp->nocb_bypass, + lazy ? BIT(CB_DEBUG_BYPASS_LAZY_FLUSHED) : BIT(CB_DEBUG_BYPASS_FLUSHED), + (j - READ_ONCE(cb_debug_jiffies_first))); + if (!rcu_nocb_flush_bypass(rdp, rhp, j, lazy, false)) { *was_alldone = !rcu_segcblist_pend_cbs(&rdp->cblist); if (*was_alldone) @@ -543,6 +558,11 @@ static bool rcu_nocb_try_bypass(struct rcu_data *rdp, struct rcu_head *rhp, trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, TPS("FirstBQ")); } +#ifdef CONFIG_RCU_TRACE_CB + rhp->di.flags |= BIT(CB_DEBUG_BYPASS); + rhp->di.first_bp_jiff = READ_ONCE(rdp->nocb_bypass_first) - READ_ONCE(cb_debug_jiffies_first); +#endif + rcu_nocb_bypass_unlock(rdp); smp_mb(); /* Order enqueue before wake. */ @@ -1156,6 +1176,10 @@ static long rcu_nocb_rdp_deoffload(void *arg) * return false, which means that future calls to rcu_nocb_try_bypass() * will refuse to put anything into the bypass. */ + + rcu_cblist_set_flush(&rdp->nocb_bypass, BIT(CB_DEBUG_DEOFFLOAD_FLUSHED), + (jiffies - READ_ONCE(cb_debug_jiffies_first))); + WARN_ON_ONCE(!rcu_nocb_flush_bypass(rdp, NULL, jiffies, false, false)); /* * Start with invoking rcu_core() early. This way if the current thread From patchwork Thu Sep 1 22:17:11 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joel Fernandes X-Patchwork-Id: 12963372 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id DF30AECAAD2 for ; Thu, 1 Sep 2022 22:18:10 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234456AbiIAWSJ (ORCPT ); Thu, 1 Sep 2022 18:18:09 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45882 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234641AbiIAWSG (ORCPT ); Thu, 1 Sep 2022 18:18:06 -0400 Received: from mail-qv1-xf31.google.com (mail-qv1-xf31.google.com [IPv6:2607:f8b0:4864:20::f31]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2AA4A59263 for ; Thu, 1 Sep 2022 15:18:05 -0700 (PDT) Received: by mail-qv1-xf31.google.com with SMTP id j6so152941qvu.10 for ; Thu, 01 Sep 2022 15:18:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc; bh=9dE4Rk8OIOb+ihEgM/v0EEwGQjrbZnrnEwGtQG+Tokc=; b=hduLdeo23eUYXwfIi5kOwRTpmsQWKz5WMzdJryBENxzTtKqWyj8jKS1pkZfs7K8ix/ 433izYbBsAKpcvkSRytC/Vy95tY1M4jnMjqRDxPDnm8amlDKZui5CBFRq87fL6LuqJ/2 5PT2xT3jsfon0zpijEzJBydZ5JMeqjs5NKQVw= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=9dE4Rk8OIOb+ihEgM/v0EEwGQjrbZnrnEwGtQG+Tokc=; b=5fLsjYJgP/t3kzcZjdPUZiB0seFmn7FPTcQJkKARCPagA7nHMDS5JI8YMZDpAxsLAi pap/mVpDNbephGS9jA2g4mr3aWurJZTnPmQZ2LFfU79JNSpzvvSDB5keU33rc4eaWxqI aq4dsN5rSYaAdtB7KA5PmyuetgkDQ7/gCBllqkqk19w+XV92s+brwmIIwcZLPyAtxLuL MOVzHjYSWm8LLxumVWfQuqHWm+U1XAueqOt701aXr+l6jBUp6m5zJpDU65fN7Inz4V86 gJRnpaXfMNcMeGTC+6p5FDkAGTPF+glvZ6TAA89kQQekvzDcDS6R1nDRuxsSfEJ6H315 woLQ== X-Gm-Message-State: ACgBeo0wvjqlS1VBOoDkt+g9dxvRrFZA2GzJiJ7vOSwt24Bqo3gf+F7/ sXI2o4b9i6ZlXS4wgOTd3vlwK4ScHYas6A== X-Google-Smtp-Source: AA6agR5JaoGB677mb+F1Gn0KMgIKjVq4iVwwN+1P2BSJcZKXS+rgh0glw3Qtr1+KQ96rCexOh2tcOA== X-Received: by 2002:a05:6214:b61:b0:498:ff24:9b59 with SMTP id ey1-20020a0562140b6100b00498ff249b59mr21011911qvb.98.1662070683846; Thu, 01 Sep 2022 15:18:03 -0700 (PDT) Received: from joelboxx.c.googlers.com.com (228.221.150.34.bc.googleusercontent.com. [34.150.221.228]) by smtp.gmail.com with ESMTPSA id s16-20020ac85290000000b0034305a91aaesm11060794qtn.83.2022.09.01.15.18.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Sep 2022 15:18:03 -0700 (PDT) From: "Joel Fernandes (Google)" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, rushikesh.s.kadam@intel.com, urezki@gmail.com, neeraj.iitr10@gmail.com, frederic@kernel.org, paulmck@kernel.org, rostedt@goodmis.org, vineeth@bitbyteword.org, boqun.feng@gmail.com, "Joel Fernandes (Google)" Subject: [PATCH v5 09/18] rcuscale: Add laziness and kfree tests Date: Thu, 1 Sep 2022 22:17:11 +0000 Message-Id: <20220901221720.1105021-10-joel@joelfernandes.org> X-Mailer: git-send-email 2.37.2.789.g6183377224-goog In-Reply-To: <20220901221720.1105021-1-joel@joelfernandes.org> References: <20220901221720.1105021-1-joel@joelfernandes.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: rcu@vger.kernel.org We aad 2 tests to rcuscale, first one is a startup test to check whether we are not too lazy or too hard working. Two, emulate kfree_rcu() itself to use call_rcu_lazy() and check memory pressure. In my testing, call_rcu_lazy() does well to keep memory pressure under control, similar to kfree_rcu(). Signed-off-by: Joel Fernandes (Google) --- kernel/rcu/rcuscale.c | 74 ++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 73 insertions(+), 1 deletion(-) diff --git a/kernel/rcu/rcuscale.c b/kernel/rcu/rcuscale.c index 3ef02d4a8108..6808f2e14385 100644 --- a/kernel/rcu/rcuscale.c +++ b/kernel/rcu/rcuscale.c @@ -95,6 +95,7 @@ torture_param(int, verbose, 1, "Enable verbose debugging printk()s"); torture_param(int, writer_holdoff, 0, "Holdoff (us) between GPs, zero to disable"); torture_param(int, kfree_rcu_test, 0, "Do we run a kfree_rcu() scale test?"); torture_param(int, kfree_mult, 1, "Multiple of kfree_obj size to allocate."); +torture_param(int, kfree_rcu_by_lazy, 0, "Use call_rcu_lazy() to emulate kfree_rcu()?"); static char *scale_type = "rcu"; module_param(scale_type, charp, 0444); @@ -659,6 +660,14 @@ struct kfree_obj { struct rcu_head rh; }; +/* Used if doing RCU-kfree'ing via call_rcu_lazy(). */ +static void kfree_rcu_lazy(struct rcu_head *rh) +{ + struct kfree_obj *obj = container_of(rh, struct kfree_obj, rh); + + kfree(obj); +} + static int kfree_scale_thread(void *arg) { @@ -696,6 +705,11 @@ kfree_scale_thread(void *arg) if (!alloc_ptr) return -ENOMEM; + if (kfree_rcu_by_lazy) { + call_rcu_lazy(&(alloc_ptr->rh), kfree_rcu_lazy); + continue; + } + // By default kfree_rcu_test_single and kfree_rcu_test_double are // initialized to false. If both have the same value (false or true) // both are randomly tested, otherwise only the one with value true @@ -738,6 +752,9 @@ kfree_scale_cleanup(void) { int i; + if (kfree_rcu_by_lazy) + rcu_force_call_rcu_to_lazy(false); + if (torture_cleanup_begin()) return; @@ -767,11 +784,64 @@ kfree_scale_shutdown(void *arg) return -EINVAL; } +// Used if doing RCU-kfree'ing via call_rcu_lazy(). +static unsigned long jiffies_at_lazy_cb; +static struct rcu_head lazy_test1_rh; +static int rcu_lazy_test1_cb_called; +static void call_rcu_lazy_test1(struct rcu_head *rh) +{ + jiffies_at_lazy_cb = jiffies; + WRITE_ONCE(rcu_lazy_test1_cb_called, 1); +} + static int __init kfree_scale_init(void) { long i; int firsterr = 0; + unsigned long orig_jif, jif_start; + + // If lazy-rcu based kfree'ing is requested, then for kernels that + // support it, force all call_rcu() to call_rcu_lazy() so that non-lazy + // CBs do not remove laziness of the lazy ones (since the test tries to + // stress call_rcu_lazy() for OOM). + // + // Also, do a quick self-test to ensure laziness is as much as + // expected. + if (kfree_rcu_by_lazy && !IS_ENABLED(CONFIG_RCU_LAZY)) { + pr_alert("CONFIG_RCU_LAZY is disabled, falling back to kfree_rcu() " + "for delayed RCU kfree'ing\n"); + kfree_rcu_by_lazy = 0; + } + + if (kfree_rcu_by_lazy) { + /* do a test to check the timeout. */ + orig_jif = rcu_lazy_get_jiffies_till_flush(); + + rcu_force_call_rcu_to_lazy(true); + rcu_lazy_set_jiffies_till_flush(2 * HZ); + rcu_barrier(); + + jif_start = jiffies; + jiffies_at_lazy_cb = 0; + call_rcu_lazy(&lazy_test1_rh, call_rcu_lazy_test1); + + smp_cond_load_relaxed(&rcu_lazy_test1_cb_called, VAL == 1); + + rcu_lazy_set_jiffies_till_flush(orig_jif); + + if (WARN_ON_ONCE(jiffies_at_lazy_cb - jif_start < 2 * HZ)) { + pr_alert("ERROR: Lazy CBs are not being lazy as expected!\n"); + WARN_ON_ONCE(1); + return -1; + } + + if (WARN_ON_ONCE(jiffies_at_lazy_cb - jif_start > 3 * HZ)) { + pr_alert("ERROR: Lazy CBs are being too lazy!\n"); + WARN_ON_ONCE(1); + return -1; + } + } kfree_nrealthreads = compute_real(kfree_nthreads); /* Start up the kthreads. */ @@ -784,7 +854,9 @@ kfree_scale_init(void) schedule_timeout_uninterruptible(1); } - pr_alert("kfree object size=%zu\n", kfree_mult * sizeof(struct kfree_obj)); + pr_alert("kfree object size=%zu, kfree_rcu_by_lazy=%d\n", + kfree_mult * sizeof(struct kfree_obj), + kfree_rcu_by_lazy); kfree_reader_tasks = kcalloc(kfree_nrealthreads, sizeof(kfree_reader_tasks[0]), GFP_KERNEL); From patchwork Thu Sep 1 22:17:12 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joel Fernandes X-Patchwork-Id: 12963376 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id CF7F3C54EE9 for ; Thu, 1 Sep 2022 22:18:45 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234968AbiIAWSm (ORCPT ); Thu, 1 Sep 2022 18:18:42 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45980 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234641AbiIAWSK (ORCPT ); Thu, 1 Sep 2022 18:18:10 -0400 Received: from mail-qt1-x836.google.com (mail-qt1-x836.google.com [IPv6:2607:f8b0:4864:20::836]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 845AC5A2D2 for ; Thu, 1 Sep 2022 15:18:05 -0700 (PDT) Received: by mail-qt1-x836.google.com with SMTP id cr9so202789qtb.13 for ; Thu, 01 Sep 2022 15:18:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc; bh=iF0VQ620OmNNEeBQv3gOJrSK8RXzJIsiopJiGn2gXgE=; b=SBhFYlQ3f3i6jV7HBq4Orbuwu/hinW5fDS4ifU7J74KEwQlKYPt/auQAncrY9fVdcu 5aTjF9w2HhCnuHyPQCCO0vD25oTat84cFEXTuBIKNKGkVKKSvEreggR7kMkSFMbjjldR i8wuBUjYYWqJSbR7o2Nre3jfWKBbsb2Lamrto= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=iF0VQ620OmNNEeBQv3gOJrSK8RXzJIsiopJiGn2gXgE=; b=wL168X8x6ORx0pO7WioIjapzP3rZeOe5jXICSsq7zlrD4tDb5JawCGHO/IlzhyZnUZ EVyc0Yq0jZHg4L4lIl5Ga7Vx/oDOIsWX3YfxtBCIo3uVRmeqMQoHeadn8NvRghKxzbKO 61N+JNJskroDyJ8RAjwKGvhgfZONTgBSHy8EA0DcTGiQhrFnfKLuBa2/N9ZhfN+cEBD3 /It7E92jHH4vDdL6NrOyUPKE/shwjHn/7MScNvGY1Unji9L+DZxVS8IUX4K6ykW/S2dL rwrA9FH87nUHWSzvM0pQZJfeQQjUm4Ekdr6cdIp/KRcdPstTByIVSbtxU/NckybarK7t qJ7g== X-Gm-Message-State: ACgBeo0lR/9r55ddX2IcHC5hoDA3ya0rB5QW9/UUvSGtBtLET/AKlMr2 Xzhoa64bOQVyABLZHA3DWKEt1c4IF+ZZ5g== X-Google-Smtp-Source: AA6agR7H18+EdoJRj/8Zp5aQpkjWP9gIyH82b2JWKmLuxIWI5VEu84ZwMh5yQZdQ8rViSDXh8dHaDg== X-Received: by 2002:a05:622a:614:b0:343:487:45d1 with SMTP id z20-20020a05622a061400b00343048745d1mr25242495qta.443.1662070684496; Thu, 01 Sep 2022 15:18:04 -0700 (PDT) Received: from joelboxx.c.googlers.com.com (228.221.150.34.bc.googleusercontent.com. [34.150.221.228]) by smtp.gmail.com with ESMTPSA id s16-20020ac85290000000b0034305a91aaesm11060794qtn.83.2022.09.01.15.18.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Sep 2022 15:18:04 -0700 (PDT) From: "Joel Fernandes (Google)" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, rushikesh.s.kadam@intel.com, urezki@gmail.com, neeraj.iitr10@gmail.com, frederic@kernel.org, paulmck@kernel.org, rostedt@goodmis.org, vineeth@bitbyteword.org, boqun.feng@gmail.com, "Joel Fernandes (Google)" Subject: [PATCH v5 10/18] rcutorture: Add test code for call_rcu_lazy() Date: Thu, 1 Sep 2022 22:17:12 +0000 Message-Id: <20220901221720.1105021-11-joel@joelfernandes.org> X-Mailer: git-send-email 2.37.2.789.g6183377224-goog In-Reply-To: <20220901221720.1105021-1-joel@joelfernandes.org> References: <20220901221720.1105021-1-joel@joelfernandes.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: rcu@vger.kernel.org We add a new RCU type to test call_rcu_lazy(). This allows us to just override the '.call' callback. To compensate for the laziness, we force the laziness to a small number of jiffies. The idea of this test is to stress the new code paths for stability and ensure it at least is providing behavior in parity with, or similar to, call_rcu(). The actual check for amount of laziness is in another test (rcuscale). Signed-off-by: Joel Fernandes (Google) --- kernel/rcu/rcu.h | 1 + kernel/rcu/rcutorture.c | 60 ++++++++++++++++++- kernel/rcu/tree.c | 1 + .../selftests/rcutorture/configs/rcu/CFLIST | 1 + .../selftests/rcutorture/configs/rcu/TREE11 | 18 ++++++ .../rcutorture/configs/rcu/TREE11.boot | 8 +++ 6 files changed, 88 insertions(+), 1 deletion(-) create mode 100644 tools/testing/selftests/rcutorture/configs/rcu/TREE11 create mode 100644 tools/testing/selftests/rcutorture/configs/rcu/TREE11.boot diff --git a/kernel/rcu/rcu.h b/kernel/rcu/rcu.h index 94675f14efe8..a6efb831673f 100644 --- a/kernel/rcu/rcu.h +++ b/kernel/rcu/rcu.h @@ -471,6 +471,7 @@ enum rcutorture_type { RCU_TASKS_TRACING_FLAVOR, RCU_TRIVIAL_FLAVOR, SRCU_FLAVOR, + RCU_LAZY_FLAVOR, INVALID_RCU_FLAVOR }; diff --git a/kernel/rcu/rcutorture.c b/kernel/rcu/rcutorture.c index 9ad5301385a4..b4114f29ab61 100644 --- a/kernel/rcu/rcutorture.c +++ b/kernel/rcu/rcutorture.c @@ -927,6 +927,64 @@ static struct rcu_torture_ops tasks_rude_ops = { #endif // #else #ifdef CONFIG_TASKS_RUDE_RCU +#ifdef CONFIG_RCU_LAZY + +/* + * Definitions for lazy RCU torture testing. + */ +static unsigned long orig_jiffies_till_flush; + +static void rcu_sync_torture_init_lazy(void) +{ + rcu_sync_torture_init(); + + orig_jiffies_till_flush = rcu_lazy_get_jiffies_till_flush(); + rcu_lazy_set_jiffies_till_flush(50); +} + +static void rcu_lazy_cleanup(void) +{ + rcu_lazy_set_jiffies_till_flush(orig_jiffies_till_flush); +} + +static struct rcu_torture_ops rcu_lazy_ops = { + .ttype = RCU_LAZY_FLAVOR, + .init = rcu_sync_torture_init_lazy, + .cleanup = rcu_lazy_cleanup, + .readlock = rcu_torture_read_lock, + .read_delay = rcu_read_delay, + .readunlock = rcu_torture_read_unlock, + .readlock_held = torture_readlock_not_held, + .get_gp_seq = rcu_get_gp_seq, + .gp_diff = rcu_seq_diff, + .deferred_free = rcu_torture_deferred_free, + .sync = synchronize_rcu, + .exp_sync = synchronize_rcu_expedited, + .get_gp_state = get_state_synchronize_rcu, + .start_gp_poll = start_poll_synchronize_rcu, + .poll_gp_state = poll_state_synchronize_rcu, + .cond_sync = cond_synchronize_rcu, + .call = call_rcu_lazy, + .cb_barrier = rcu_barrier, + .fqs = rcu_force_quiescent_state, + .stats = NULL, + .gp_kthread_dbg = show_rcu_gp_kthreads, + .check_boost_failed = rcu_check_boost_fail, + .stall_dur = rcu_jiffies_till_stall_check, + .irq_capable = 1, + .can_boost = IS_ENABLED(CONFIG_RCU_BOOST), + .extendables = RCUTORTURE_MAX_EXTEND, + .name = "rcu_lazy" +}; + +#define LAZY_OPS &rcu_lazy_ops, + +#else // #ifdef CONFIG_RCU_LAZY + +#define LAZY_OPS + +#endif // #else #ifdef CONFIG_RCU_LAZY + #ifdef CONFIG_TASKS_TRACE_RCU @@ -3466,7 +3524,7 @@ rcu_torture_init(void) unsigned long gp_seq = 0; static struct rcu_torture_ops *torture_ops[] = { &rcu_ops, &rcu_busted_ops, &srcu_ops, &srcud_ops, &busted_srcud_ops, - TASKS_OPS TASKS_RUDE_OPS TASKS_TRACING_OPS + TASKS_OPS TASKS_RUDE_OPS TASKS_TRACING_OPS LAZY_OPS &trivial_ops, }; diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 8111d9f37621..08bafe13c3ad 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -537,6 +537,7 @@ void rcutorture_get_gp_data(enum rcutorture_type test_type, int *flags, { switch (test_type) { case RCU_FLAVOR: + case RCU_LAZY_FLAVOR: *flags = READ_ONCE(rcu_state.gp_flags); *gp_seq = rcu_seq_current(&rcu_state.gp_seq); break; diff --git a/tools/testing/selftests/rcutorture/configs/rcu/CFLIST b/tools/testing/selftests/rcutorture/configs/rcu/CFLIST index 98b6175e5aa0..609c3370616f 100644 --- a/tools/testing/selftests/rcutorture/configs/rcu/CFLIST +++ b/tools/testing/selftests/rcutorture/configs/rcu/CFLIST @@ -5,6 +5,7 @@ TREE04 TREE05 TREE07 TREE09 +TREE11 SRCU-N SRCU-P SRCU-T diff --git a/tools/testing/selftests/rcutorture/configs/rcu/TREE11 b/tools/testing/selftests/rcutorture/configs/rcu/TREE11 new file mode 100644 index 000000000000..436013f3e015 --- /dev/null +++ b/tools/testing/selftests/rcutorture/configs/rcu/TREE11 @@ -0,0 +1,18 @@ +CONFIG_SMP=y +CONFIG_PREEMPT_NONE=n +CONFIG_PREEMPT_VOLUNTARY=n +CONFIG_PREEMPT=y +#CHECK#CONFIG_PREEMPT_RCU=y +CONFIG_HZ_PERIODIC=n +CONFIG_NO_HZ_IDLE=y +CONFIG_NO_HZ_FULL=n +CONFIG_RCU_TRACE=y +CONFIG_HOTPLUG_CPU=y +CONFIG_MAXSMP=y +CONFIG_CPUMASK_OFFSTACK=y +CONFIG_RCU_NOCB_CPU=y +CONFIG_DEBUG_LOCK_ALLOC=n +CONFIG_RCU_BOOST=n +CONFIG_DEBUG_OBJECTS_RCU_HEAD=n +CONFIG_RCU_EXPERT=y +CONFIG_RCU_LAZY=y diff --git a/tools/testing/selftests/rcutorture/configs/rcu/TREE11.boot b/tools/testing/selftests/rcutorture/configs/rcu/TREE11.boot new file mode 100644 index 000000000000..9b6f720d4ccd --- /dev/null +++ b/tools/testing/selftests/rcutorture/configs/rcu/TREE11.boot @@ -0,0 +1,8 @@ +maxcpus=8 nr_cpus=43 +rcutree.gp_preinit_delay=3 +rcutree.gp_init_delay=3 +rcutree.gp_cleanup_delay=3 +rcu_nocbs=0-7 +rcutorture.torture_type=rcu_lazy +rcutorture.nocbs_nthreads=8 +rcutorture.fwd_progress=0 From patchwork Thu Sep 1 22:17:13 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joel Fernandes X-Patchwork-Id: 12963374 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 34FB3ECAAD2 for ; Thu, 1 Sep 2022 22:18:22 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234803AbiIAWSU (ORCPT ); Thu, 1 Sep 2022 18:18:20 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45922 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234697AbiIAWSJ (ORCPT ); Thu, 1 Sep 2022 18:18:09 -0400 Received: from mail-qt1-x82e.google.com (mail-qt1-x82e.google.com [IPv6:2607:f8b0:4864:20::82e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 190375AC47 for ; Thu, 1 Sep 2022 15:18:06 -0700 (PDT) Received: by mail-qt1-x82e.google.com with SMTP id y18so234016qtv.5 for ; Thu, 01 Sep 2022 15:18:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc; bh=GmZL9QRJG3JGAUAVfGfSTA34gmVVbLMug2meh+hLHJM=; b=dWo8eLEyEw5jcTSOIhe0J42iN+xFaer3fDsss15lmAIIREY+MawFShJoU5lQXs152W 5bBsVEo8OeqyIN1jsIXSjKgdCCZW0EZC9yD75V/Uy2UCsKQA1A2x2/RnRq2779uXc7NO fuaowWwCJUf5JBRU4sL3dTWtVU2pKz7JGf3F4= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=GmZL9QRJG3JGAUAVfGfSTA34gmVVbLMug2meh+hLHJM=; b=b/UPqAJN6cviKfreBfU/J3sGPqXLGrvPPlZJ93xso6/qTJ7iC23yo/mO34qeOmwzAK gGfwteQLdHGRS9AyLgTjUrNomOEh5spoGwYaVHPSljzT1aETXWatOI9q735qTgNqZYaB vZ/Kb1FEzFTG4MlcuZfc+Ti2SKUG86k7JD/nSE9pjIGOWgNstIzwEmdapUqwOmjxRUBj dxK1++/Fox3hoTiwzuMSK5FiIJjVScNuiE8ejjQEDBQPUulBn6lFRjnon5jLjX2IhcdW v9Ofnx9RxHCeI7y2oUb7VkbCguwYlP0nN359cY5JWkfdaQvgJkHoqV4oDK/X3kGZREaU tFiw== X-Gm-Message-State: ACgBeo3K5u4+L8M0RxSwHMFjaxCmkctUVak2/xEfMey/gW+d6W5h28nL XtIGDboHh+PbB0y71AhbYROiykmAr98qxA== X-Google-Smtp-Source: AA6agR5d0+TO4TcTHxSbXssriWeXw2oFTaGpGLqzcaqhAOpXB66/FUakNqneNIoAQAkF8kDVHuirhQ== X-Received: by 2002:a05:622a:3d0:b0:343:58db:d5c7 with SMTP id k16-20020a05622a03d000b0034358dbd5c7mr25537687qtx.21.1662070685069; Thu, 01 Sep 2022 15:18:05 -0700 (PDT) Received: from joelboxx.c.googlers.com.com (228.221.150.34.bc.googleusercontent.com. [34.150.221.228]) by smtp.gmail.com with ESMTPSA id s16-20020ac85290000000b0034305a91aaesm11060794qtn.83.2022.09.01.15.18.04 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Sep 2022 15:18:04 -0700 (PDT) From: "Joel Fernandes (Google)" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, rushikesh.s.kadam@intel.com, urezki@gmail.com, neeraj.iitr10@gmail.com, frederic@kernel.org, paulmck@kernel.org, rostedt@goodmis.org, vineeth@bitbyteword.org, boqun.feng@gmail.com, "Joel Fernandes (Google)" Subject: [PATCH v5 11/18] fs: Move call_rcu() to call_rcu_lazy() in some paths Date: Thu, 1 Sep 2022 22:17:13 +0000 Message-Id: <20220901221720.1105021-12-joel@joelfernandes.org> X-Mailer: git-send-email 2.37.2.789.g6183377224-goog In-Reply-To: <20220901221720.1105021-1-joel@joelfernandes.org> References: <20220901221720.1105021-1-joel@joelfernandes.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: rcu@vger.kernel.org This is required to prevent callbacks triggering RCU machinery too quickly and too often, which adds more power to the system. When testing, we found that these paths were invoked often when the system is not doing anything (screen is ON but otherwise idle). Signed-off-by: Joel Fernandes (Google) --- fs/dcache.c | 4 ++-- fs/eventpoll.c | 2 +- fs/file_table.c | 2 +- fs/inode.c | 2 +- 4 files changed, 5 insertions(+), 5 deletions(-) diff --git a/fs/dcache.c b/fs/dcache.c index c5dc32a59c76..28a159a7e460 100644 --- a/fs/dcache.c +++ b/fs/dcache.c @@ -366,7 +366,7 @@ static void dentry_free(struct dentry *dentry) if (unlikely(dname_external(dentry))) { struct external_name *p = external_name(dentry); if (likely(atomic_dec_and_test(&p->u.count))) { - call_rcu(&dentry->d_u.d_rcu, __d_free_external); + call_rcu_lazy(&dentry->d_u.d_rcu, __d_free_external); return; } } @@ -374,7 +374,7 @@ static void dentry_free(struct dentry *dentry) if (dentry->d_flags & DCACHE_NORCU) __d_free(&dentry->d_u.d_rcu); else - call_rcu(&dentry->d_u.d_rcu, __d_free); + call_rcu_lazy(&dentry->d_u.d_rcu, __d_free); } /* diff --git a/fs/eventpoll.c b/fs/eventpoll.c index 8b56b94e2f56..530e62538d3d 100644 --- a/fs/eventpoll.c +++ b/fs/eventpoll.c @@ -728,7 +728,7 @@ static int ep_remove(struct eventpoll *ep, struct epitem *epi) * ep->mtx. The rcu read side, reverse_path_check_proc(), does not make * use of the rbn field. */ - call_rcu(&epi->rcu, epi_rcu_free); + call_rcu_lazy(&epi->rcu, epi_rcu_free); percpu_counter_dec(&ep->user->epoll_watches); diff --git a/fs/file_table.c b/fs/file_table.c index 99c6796c9f28..36b3de669771 100644 --- a/fs/file_table.c +++ b/fs/file_table.c @@ -56,7 +56,7 @@ static inline void file_free(struct file *f) security_file_free(f); if (!(f->f_mode & FMODE_NOACCOUNT)) percpu_counter_dec(&nr_files); - call_rcu(&f->f_rcuhead, file_free_rcu); + call_rcu_lazy(&f->f_rcuhead, file_free_rcu); } /* diff --git a/fs/inode.c b/fs/inode.c index 6462276dfdf0..7d1573cf3a97 100644 --- a/fs/inode.c +++ b/fs/inode.c @@ -312,7 +312,7 @@ static void destroy_inode(struct inode *inode) return; } inode->free_inode = ops->free_inode; - call_rcu(&inode->i_rcu, i_callback); + call_rcu_lazy(&inode->i_rcu, i_callback); } /** From patchwork Thu Sep 1 22:17:14 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joel Fernandes X-Patchwork-Id: 12963377 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6F800ECAAD2 for ; Thu, 1 Sep 2022 22:18:47 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235006AbiIAWSn (ORCPT ); Thu, 1 Sep 2022 18:18:43 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45984 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234653AbiIAWSK (ORCPT ); Thu, 1 Sep 2022 18:18:10 -0400 Received: from mail-qt1-x82a.google.com (mail-qt1-x82a.google.com [IPv6:2607:f8b0:4864:20::82a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id ED0DC5B061 for ; Thu, 1 Sep 2022 15:18:06 -0700 (PDT) Received: by mail-qt1-x82a.google.com with SMTP id g14so210094qto.11 for ; Thu, 01 Sep 2022 15:18:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc; bh=TrK5dOUuJ472Va/MQxlZM4Ul27mMRNSxArIaJwpPfZg=; b=TUcC0u4QM5Ny9KLvPnrxQITVTokuBxEATE/WsQfbGYFGD9PpxIg9MJCiMdUCQf7Laq X6rikyqdDpvNk6vlDW0YVq5/0S8OJYELbrNlakftwrW0WN0AsYlxa096qY1FJJBELlxI jL2uVxkUi+pIbxFoV4OaGYaWTNiex9ahiveZ0= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=TrK5dOUuJ472Va/MQxlZM4Ul27mMRNSxArIaJwpPfZg=; b=CirCsO0NJbQZZFpl+2S00ouBrLUf23c7x2xt5Y8XK42KXVXImspZqamTb8xXSZxDOr KWYmZ6o/OmjjQ38vwOaSSXfQGy+gUNJCbbY3J35YglGNHWN6URBqAmLMJlQQ87q31dge wgqEoGtgW2cVSNbD3EUewTrbD0PPkPZH4Vvawhx2hoRZNNxA6PLl/sO9CdKT6sszlqNz EXaUhLxw5L4e5dQ8e1m2ynVXBtYFpDK8vX8pdDq9Wip7/pcrOU/OOIwwJANjIRGPz2jz R05u/g7pLtjKPxNWJQyKaOcB78ugIj8OLx0Q/2DtjWrLJShFpA1Y4jGkr7XsfAbcboLV Tw3w== X-Gm-Message-State: ACgBeo0CVCv8jjEk5ghjX++CCXec5WC3/zyMzjZnVHmfNQuA2CFqtH51 tVQyLtvSRgw7hjMT7KSFK66WHUTIZanDjg== X-Google-Smtp-Source: AA6agR4uiuD53PR2JnbYvQYDsN+ofsO7SW63BGOkp2t0qORapVEs6g6ATsIcI2VvrBk+/fuLb48KeQ== X-Received: by 2002:a05:622a:53:b0:344:6f46:9b16 with SMTP id y19-20020a05622a005300b003446f469b16mr26319627qtw.664.1662070685785; Thu, 01 Sep 2022 15:18:05 -0700 (PDT) Received: from joelboxx.c.googlers.com.com (228.221.150.34.bc.googleusercontent.com. [34.150.221.228]) by smtp.gmail.com with ESMTPSA id s16-20020ac85290000000b0034305a91aaesm11060794qtn.83.2022.09.01.15.18.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Sep 2022 15:18:05 -0700 (PDT) From: "Joel Fernandes (Google)" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, rushikesh.s.kadam@intel.com, urezki@gmail.com, neeraj.iitr10@gmail.com, frederic@kernel.org, paulmck@kernel.org, rostedt@goodmis.org, vineeth@bitbyteword.org, boqun.feng@gmail.com, "Joel Fernandes (Google)" Subject: [PATCH v5 12/18] cred: Move call_rcu() to call_rcu_lazy() Date: Thu, 1 Sep 2022 22:17:14 +0000 Message-Id: <20220901221720.1105021-13-joel@joelfernandes.org> X-Mailer: git-send-email 2.37.2.789.g6183377224-goog In-Reply-To: <20220901221720.1105021-1-joel@joelfernandes.org> References: <20220901221720.1105021-1-joel@joelfernandes.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: rcu@vger.kernel.org This is required to prevent callbacks triggering RCU machinery too quickly and too often, which adds more power to the system. Signed-off-by: Joel Fernandes (Google) --- kernel/cred.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/kernel/cred.c b/kernel/cred.c index e10c15f51c1f..c7cb2e3ac73a 100644 --- a/kernel/cred.c +++ b/kernel/cred.c @@ -150,7 +150,7 @@ void __put_cred(struct cred *cred) if (cred->non_rcu) put_cred_rcu(&cred->rcu); else - call_rcu(&cred->rcu, put_cred_rcu); + call_rcu_lazy(&cred->rcu, put_cred_rcu); } EXPORT_SYMBOL(__put_cred); From patchwork Thu Sep 1 22:17:15 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joel Fernandes X-Patchwork-Id: 12963381 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A93FAC6FA82 for ; Thu, 1 Sep 2022 22:18:50 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234920AbiIAWSu (ORCPT ); Thu, 1 Sep 2022 18:18:50 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45984 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234925AbiIAWSm (ORCPT ); Thu, 1 Sep 2022 18:18:42 -0400 Received: from mail-qt1-x830.google.com (mail-qt1-x830.google.com [IPv6:2607:f8b0:4864:20::830]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8E3005E66F for ; Thu, 1 Sep 2022 15:18:07 -0700 (PDT) Received: by mail-qt1-x830.google.com with SMTP id r6so230302qtx.6 for ; Thu, 01 Sep 2022 15:18:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc; bh=5xn60rDjdB3Rf7Wir8y+6b2hKXDgCuU2pqQZqimv/ck=; b=Os4hTQX+7UasiETS+y9NSvg4ja5hDOYYPQYGOfJ5sv7/j1Yu4A2w2gXnSlW0jOFCDp bLTIIkeTRy+1cbbBwSjwl3rHEw4Ocq3dzKdcSC+TunfEtEl+yuu5I44XNzQiXUAY3Ewk QEeYCOdMfWSgWfckJw059MAJSVHel1agDUyiA= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=5xn60rDjdB3Rf7Wir8y+6b2hKXDgCuU2pqQZqimv/ck=; b=xPMYNLgANaeAELousPgs1TrX7Uw3RcOV+aHnv4Zl1pMwhHrwl+ByRXuwkFxiVGW5xz pPpvzwDQTohBwgwHsi1+KTOkyEW7md4LCHb9AO/73jLdeRXHQXF4NGYe7Oz0C7MKBbd0 i7/MFed2GcG47e48t2gvbBnQsCcftN3zPjsYt4iQTCDgEk1Ku0z2udCErUll7rJPEYFc yW2pd3bsg2bAWG6szEDfqBTrOPzKpMM3Agt+3iGvPhR7gFxGIld7hfJLmLPj0JJb2IhK wQiAQQTJpnPeLAKck3S8RNuAgRRI/qMx6E1s0FB0eLiPhWWaqaUKQwmtaZYJJVxQefD1 mbLA== X-Gm-Message-State: ACgBeo04XiyzkEUx1Fv3JUfoXwwT+rQ0Jla+huoxZg/FJJsc5NVL7ypJ qAnfU1SYCub0uZ0JK+pkrUCMkIoFwPSN9A== X-Google-Smtp-Source: AA6agR4diTGOE7swnbaEjFmZk8h1Ql35CL92QOFXkcefETWvplUeM3qM08T8V0Y69YU08rm48PY+vQ== X-Received: by 2002:ac8:4e45:0:b0:343:5faf:3af6 with SMTP id e5-20020ac84e45000000b003435faf3af6mr26987399qtw.340.1662070686395; Thu, 01 Sep 2022 15:18:06 -0700 (PDT) Received: from joelboxx.c.googlers.com.com (228.221.150.34.bc.googleusercontent.com. [34.150.221.228]) by smtp.gmail.com with ESMTPSA id s16-20020ac85290000000b0034305a91aaesm11060794qtn.83.2022.09.01.15.18.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Sep 2022 15:18:06 -0700 (PDT) From: "Joel Fernandes (Google)" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, rushikesh.s.kadam@intel.com, urezki@gmail.com, neeraj.iitr10@gmail.com, frederic@kernel.org, paulmck@kernel.org, rostedt@goodmis.org, vineeth@bitbyteword.org, boqun.feng@gmail.com, "Joel Fernandes (Google)" Subject: [PATCH v5 13/18] security: Move call_rcu() to call_rcu_lazy() Date: Thu, 1 Sep 2022 22:17:15 +0000 Message-Id: <20220901221720.1105021-14-joel@joelfernandes.org> X-Mailer: git-send-email 2.37.2.789.g6183377224-goog In-Reply-To: <20220901221720.1105021-1-joel@joelfernandes.org> References: <20220901221720.1105021-1-joel@joelfernandes.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: rcu@vger.kernel.org This is required to prevent callbacks triggering RCU machinery too quickly and too often, which adds more power to the system. Signed-off-by: Joel Fernandes (Google) --- security/security.c | 2 +- security/selinux/avc.c | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/security/security.c b/security/security.c index 14d30fec8a00..b51b4bdb567d 100644 --- a/security/security.c +++ b/security/security.c @@ -1053,7 +1053,7 @@ void security_inode_free(struct inode *inode) * The inode will be freed after the RCU grace period too. */ if (inode->i_security) - call_rcu((struct rcu_head *)inode->i_security, + call_rcu_lazy((struct rcu_head *)inode->i_security, inode_free_by_rcu); } diff --git a/security/selinux/avc.c b/security/selinux/avc.c index 9a43af0ebd7d..381f046d820f 100644 --- a/security/selinux/avc.c +++ b/security/selinux/avc.c @@ -442,7 +442,7 @@ static void avc_node_free(struct rcu_head *rhead) static void avc_node_delete(struct selinux_avc *avc, struct avc_node *node) { hlist_del_rcu(&node->list); - call_rcu(&node->rhead, avc_node_free); + call_rcu_lazy(&node->rhead, avc_node_free); atomic_dec(&avc->avc_cache.active_nodes); } @@ -458,7 +458,7 @@ static void avc_node_replace(struct selinux_avc *avc, struct avc_node *new, struct avc_node *old) { hlist_replace_rcu(&old->list, &new->list); - call_rcu(&old->rhead, avc_node_free); + call_rcu_lazy(&old->rhead, avc_node_free); atomic_dec(&avc->avc_cache.active_nodes); } From patchwork Thu Sep 1 22:17:16 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joel Fernandes X-Patchwork-Id: 12963378 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 138BEC6FA82 for ; Thu, 1 Sep 2022 22:18:48 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234653AbiIAWSp (ORCPT ); Thu, 1 Sep 2022 18:18:45 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45922 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234772AbiIAWST (ORCPT ); Thu, 1 Sep 2022 18:18:19 -0400 Received: from mail-qt1-x832.google.com (mail-qt1-x832.google.com [IPv6:2607:f8b0:4864:20::832]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DDCF958091 for ; Thu, 1 Sep 2022 15:18:07 -0700 (PDT) Received: by mail-qt1-x832.google.com with SMTP id a22so212563qtw.10 for ; Thu, 01 Sep 2022 15:18:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc; bh=Uetgjnmco7XMjJ0pPvFMMYQeK2oqyNBCbg9zI/9WK6U=; b=Qlh767M6LCeX0fcTUExKI0vDqoTRBfz2hg5EurE23Vs4l8vVL/qjU47CREwknrDU0m rLGyL/hdoXaQ3k8Vo5xqmJuRolH5BcV8urpgipRpHre9ncZoZRuwvvGiH/z84Xkf9PKx w32OhzRkeqYAp5tKbH/fi2wbMWp9nPLv/U/O4= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=Uetgjnmco7XMjJ0pPvFMMYQeK2oqyNBCbg9zI/9WK6U=; b=r36CPxqCkqkHs21H0VSXOEjeiBL70qX8lHgwo/KiUqKCfd/vM62/tt8Tr1BwHrVC97 x/jrpiX+CxJIyzvrDEvhpEGwIcgptMjQHjcoUpxcUZhpgWfiQEKlt4Q2iJ+eIFX3rwBE WQb6LRxSzV3bd64Kdvw/4Iiq6SjNOElFTkCbXvyHANBocW6SXgQmGo6cEdbwdRBxo0g/ RGacD4zVZZTeAohVqF6JE2UDepRNPjJVNbKJYM/+VchQRRix/5SSzf2Yx8jwBfncRfLM phiV7+iWNjNp4EycboTLfKyOZDydwDM3z0OGAoqM7qH2AnyE3a6S9YH2ZU3zRXdMA6GE cE1A== X-Gm-Message-State: ACgBeo3nchGnBMabLBKHqNz2X9HUsy38qiO7fVUIzp+DafLh9zIOuJKX BbUCahyeN0ig5QQD/dT1n28aO0Z6itiEXg== X-Google-Smtp-Source: AA6agR7NX0LPVmvfR8AYnY7HzxGSlvcrW6s4HEA32zP4sXGb6qTrqmmBag2GXgdyln10oQkqNMt8zw== X-Received: by 2002:ac8:5a96:0:b0:344:b3be:6931 with SMTP id c22-20020ac85a96000000b00344b3be6931mr25930091qtc.509.1662070687197; Thu, 01 Sep 2022 15:18:07 -0700 (PDT) Received: from joelboxx.c.googlers.com.com (228.221.150.34.bc.googleusercontent.com. [34.150.221.228]) by smtp.gmail.com with ESMTPSA id s16-20020ac85290000000b0034305a91aaesm11060794qtn.83.2022.09.01.15.18.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Sep 2022 15:18:06 -0700 (PDT) From: "Joel Fernandes (Google)" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, rushikesh.s.kadam@intel.com, urezki@gmail.com, neeraj.iitr10@gmail.com, frederic@kernel.org, paulmck@kernel.org, rostedt@goodmis.org, vineeth@bitbyteword.org, boqun.feng@gmail.com, "Joel Fernandes (Google)" Subject: [PATCH v5 14/18] net/core: Move call_rcu() to call_rcu_lazy() Date: Thu, 1 Sep 2022 22:17:16 +0000 Message-Id: <20220901221720.1105021-15-joel@joelfernandes.org> X-Mailer: git-send-email 2.37.2.789.g6183377224-goog In-Reply-To: <20220901221720.1105021-1-joel@joelfernandes.org> References: <20220901221720.1105021-1-joel@joelfernandes.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: rcu@vger.kernel.org This is required to prevent callbacks triggering RCU machinery too quickly and too often, which adds more power to the system. Signed-off-by: Joel Fernandes (Google) --- net/core/dst.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/net/core/dst.c b/net/core/dst.c index bc9c9be4e080..babf49e413e1 100644 --- a/net/core/dst.c +++ b/net/core/dst.c @@ -174,7 +174,7 @@ void dst_release(struct dst_entry *dst) net_warn_ratelimited("%s: dst:%p refcnt:%d\n", __func__, dst, newrefcnt); if (!newrefcnt) - call_rcu(&dst->rcu_head, dst_destroy_rcu); + call_rcu_lazy(&dst->rcu_head, dst_destroy_rcu); } } EXPORT_SYMBOL(dst_release); From patchwork Thu Sep 1 22:17:17 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joel Fernandes X-Patchwork-Id: 12963380 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 62405C54EE9 for ; Thu, 1 Sep 2022 22:18:50 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234934AbiIAWSs (ORCPT ); Thu, 1 Sep 2022 18:18:48 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46266 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234792AbiIAWST (ORCPT ); Thu, 1 Sep 2022 18:18:19 -0400 Received: from mail-qt1-x82d.google.com (mail-qt1-x82d.google.com [IPv6:2607:f8b0:4864:20::82d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9419A60504 for ; Thu, 1 Sep 2022 15:18:08 -0700 (PDT) Received: by mail-qt1-x82d.google.com with SMTP id h22so245792qtu.2 for ; Thu, 01 Sep 2022 15:18:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc; bh=3vWNR1lOZ3nlNpU4UUmxfmJYajX79MfO1ObVeBbomlg=; b=W9V/oqA+OZMtsddIG83NBCyyox5x44+ormS/AjYTPNmA6tq1LLUSu1mTv373zuQAmL stXAk/bY0nw2tv3MorrBWKQVz0OdcObvezxEjnW0fZHXmStt0csE5tLilyFNZXZtzQ7e hZ35XQl1bl0u/iHRILvxndSITlzXxbNnrQw8k= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=3vWNR1lOZ3nlNpU4UUmxfmJYajX79MfO1ObVeBbomlg=; b=V8tGseKQfxncSLJK5EIjjKZtSyKiKAGHQ27Wn/3QF71OnS7lImP+1iocKZ3p232b3C dscJzJLmQplqyO7SbsBXwB7s38MNOsUl738BSPfGhhOA6EDg3ynlGLfh314AHJnhMQN1 WVzmYpm76oCFMN+QaQx6ZZ0wzWC7v89h9XG7POwOk17mg8XUVH/hhZNPkyWUbvnpPJgi /gw9FORh4hByIBt5A0/sisNu5emndygZXTPHeFADo6pYhUh54khRy3MFP8eBngoYvmxp VxlJDh2FEpLtvYsfj/IlcylRAHhEisO/TNTkXUgSGlx0P/DYZl816f7UIvIhRejLPXDr G7fw== X-Gm-Message-State: ACgBeo3320ekoznuKJvntKf5zk4zPzmCjvfP8Cu8sHAzZ9oI/m8O90Mj KdImCxkuDqFGTqisg2EUyEnqByYdKXIUwA== X-Google-Smtp-Source: AA6agR7Md6xYqwWArMXYWwgIpGr8i6bhrZ317pWwHfXCZUjbF4/68mUbJbpxMEPOI/zS/oZi2CAc+g== X-Received: by 2002:ac8:5786:0:b0:343:3051:170d with SMTP id v6-20020ac85786000000b003433051170dmr25646460qta.429.1662070688111; Thu, 01 Sep 2022 15:18:08 -0700 (PDT) Received: from joelboxx.c.googlers.com.com (228.221.150.34.bc.googleusercontent.com. [34.150.221.228]) by smtp.gmail.com with ESMTPSA id s16-20020ac85290000000b0034305a91aaesm11060794qtn.83.2022.09.01.15.18.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Sep 2022 15:18:07 -0700 (PDT) From: "Joel Fernandes (Google)" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, rushikesh.s.kadam@intel.com, urezki@gmail.com, neeraj.iitr10@gmail.com, frederic@kernel.org, paulmck@kernel.org, rostedt@goodmis.org, vineeth@bitbyteword.org, boqun.feng@gmail.com, "Joel Fernandes (Google)" Subject: [PATCH v5 15/18] kernel: Move various core kernel usages to call_rcu_lazy() Date: Thu, 1 Sep 2022 22:17:17 +0000 Message-Id: <20220901221720.1105021-16-joel@joelfernandes.org> X-Mailer: git-send-email 2.37.2.789.g6183377224-goog In-Reply-To: <20220901221720.1105021-1-joel@joelfernandes.org> References: <20220901221720.1105021-1-joel@joelfernandes.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: rcu@vger.kernel.org Signed-off-by: Joel Fernandes (Google) --- kernel/exit.c | 2 +- kernel/pid.c | 2 +- kernel/time/posix-timers.c | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/kernel/exit.c b/kernel/exit.c index 84021b24f79e..b2a96356980a 100644 --- a/kernel/exit.c +++ b/kernel/exit.c @@ -180,7 +180,7 @@ static void delayed_put_task_struct(struct rcu_head *rhp) void put_task_struct_rcu_user(struct task_struct *task) { if (refcount_dec_and_test(&task->rcu_users)) - call_rcu(&task->rcu, delayed_put_task_struct); + call_rcu_lazy(&task->rcu, delayed_put_task_struct); } void release_task(struct task_struct *p) diff --git a/kernel/pid.c b/kernel/pid.c index 2fc0a16ec77b..5a5144519d70 100644 --- a/kernel/pid.c +++ b/kernel/pid.c @@ -153,7 +153,7 @@ void free_pid(struct pid *pid) } spin_unlock_irqrestore(&pidmap_lock, flags); - call_rcu(&pid->rcu, delayed_put_pid); + call_rcu_lazy(&pid->rcu, delayed_put_pid); } struct pid *alloc_pid(struct pid_namespace *ns, pid_t *set_tid, diff --git a/kernel/time/posix-timers.c b/kernel/time/posix-timers.c index 5dead89308b7..660daa427b41 100644 --- a/kernel/time/posix-timers.c +++ b/kernel/time/posix-timers.c @@ -485,7 +485,7 @@ static void release_posix_timer(struct k_itimer *tmr, int it_id_set) } put_pid(tmr->it_pid); sigqueue_free(tmr->sigq); - call_rcu(&tmr->rcu, k_itimer_rcu_free); + call_rcu_lazy(&tmr->rcu, k_itimer_rcu_free); } static int common_timer_create(struct k_itimer *new_timer) From patchwork Thu Sep 1 22:17:18 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joel Fernandes X-Patchwork-Id: 12963379 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4A82CECAAD3 for ; Thu, 1 Sep 2022 22:18:48 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234772AbiIAWSr (ORCPT ); Thu, 1 Sep 2022 18:18:47 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45962 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233277AbiIAWSV (ORCPT ); Thu, 1 Sep 2022 18:18:21 -0400 Received: from mail-qt1-x82a.google.com (mail-qt1-x82a.google.com [IPv6:2607:f8b0:4864:20::82a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 975B359263 for ; Thu, 1 Sep 2022 15:18:09 -0700 (PDT) Received: by mail-qt1-x82a.google.com with SMTP id l5so239084qtv.4 for ; Thu, 01 Sep 2022 15:18:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc; bh=dpVq45gOBHghNC476XxsHo12GSN2n+TlCs0w1yjI1G8=; b=LHvRA8QK5VFw/jrPAKHEmEEsl0L+oZGGdwvCDSEY1J7Zf3XW17vgFCaaiJ5p+DinkX JywVDBQDKWVZmOUD+PWJONyE7y3Pj72l2fcvyocgTGJqG/OsejHHXd4qGxJCj3m5m77v d3hMh3KrXjEdiRpUQMsxM9gI5WAwTmvMZcliE= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=dpVq45gOBHghNC476XxsHo12GSN2n+TlCs0w1yjI1G8=; b=VyvqEBEZPAgFtTr1HrycJl7f2BCp//C3qX2a1M5tNgKqXQGda+ylZGJhl+sjpKe8T+ Cy3lFn5O9XwysQEZvQh4z8Ny8QKHU6NylhkyVWwV43OMoy3w+s3qLLgttT+x6vtFJowF NzxnSElkF85tTonubR/ke9NYP8RBCwRO9euLrB/sFulJ97IYnAUF82v/M29cvbCoru2k 3eeovpe0O/ChmTQkS0yX1Lprw9RBulXtoAtyUqZCJWtkqWVnARXBCX/DQQC8eXbpjWS8 Jy3Icd8B7N+LAK+fTjVdW7nrpxK5if4i+5G41MGlIt9QLKLE2w2d848wwSiYNE4EkarY Uaig== X-Gm-Message-State: ACgBeo2Bzqk+/W4lMQnUWpB0y+LGFX3XIUdNDrcBoBGDqeVvDcjMJ/7q Q2qbcNtzfPEPOR3r+8swbiHwvwjKONeDCQ== X-Google-Smtp-Source: AA6agR5fwM2qBQofpQTaAhwXy9MXIukc637mu6NfIRAgZbUg8vF1uoX4k76JfSjiT21Y9vviNS6TMg== X-Received: by 2002:ac8:4e4a:0:b0:343:7e05:d2a8 with SMTP id e10-20020ac84e4a000000b003437e05d2a8mr26201508qtw.67.1662070689060; Thu, 01 Sep 2022 15:18:09 -0700 (PDT) Received: from joelboxx.c.googlers.com.com (228.221.150.34.bc.googleusercontent.com. [34.150.221.228]) by smtp.gmail.com with ESMTPSA id s16-20020ac85290000000b0034305a91aaesm11060794qtn.83.2022.09.01.15.18.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Sep 2022 15:18:08 -0700 (PDT) From: "Joel Fernandes (Google)" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, rushikesh.s.kadam@intel.com, urezki@gmail.com, neeraj.iitr10@gmail.com, frederic@kernel.org, paulmck@kernel.org, rostedt@goodmis.org, vineeth@bitbyteword.org, boqun.feng@gmail.com, "Joel Fernandes (Google)" Subject: [PATCH v5 16/18] lib: Move call_rcu() to call_rcu_lazy() Date: Thu, 1 Sep 2022 22:17:18 +0000 Message-Id: <20220901221720.1105021-17-joel@joelfernandes.org> X-Mailer: git-send-email 2.37.2.789.g6183377224-goog In-Reply-To: <20220901221720.1105021-1-joel@joelfernandes.org> References: <20220901221720.1105021-1-joel@joelfernandes.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: rcu@vger.kernel.org Move radix-tree and xarray to call_rcu_lazy(). This is required to prevent callbacks triggering RCU machinery too quickly and too often, which adds more power to the system. Signed-off-by: Joel Fernandes (Google) --- lib/radix-tree.c | 2 +- lib/xarray.c | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/lib/radix-tree.c b/lib/radix-tree.c index 3c78e1e8b2ad..49c821b16acc 100644 --- a/lib/radix-tree.c +++ b/lib/radix-tree.c @@ -305,7 +305,7 @@ void radix_tree_node_rcu_free(struct rcu_head *head) static inline void radix_tree_node_free(struct radix_tree_node *node) { - call_rcu(&node->rcu_head, radix_tree_node_rcu_free); + call_rcu_lazy(&node->rcu_head, radix_tree_node_rcu_free); } /* diff --git a/lib/xarray.c b/lib/xarray.c index ea9ce1f0b386..230abc8045fe 100644 --- a/lib/xarray.c +++ b/lib/xarray.c @@ -257,7 +257,7 @@ static void xa_node_free(struct xa_node *node) { XA_NODE_BUG_ON(node, !list_empty(&node->private_list)); node->array = XA_RCU_FREE; - call_rcu(&node->rcu_head, radix_tree_node_rcu_free); + call_rcu_lazy(&node->rcu_head, radix_tree_node_rcu_free); } /* From patchwork Thu Sep 1 22:17:19 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joel Fernandes X-Patchwork-Id: 12963382 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id F2A9FECAAD2 for ; Thu, 1 Sep 2022 22:18:51 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233034AbiIAWSu (ORCPT ); Thu, 1 Sep 2022 18:18:50 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46702 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234937AbiIAWSm (ORCPT ); Thu, 1 Sep 2022 18:18:42 -0400 Received: from mail-qt1-x836.google.com (mail-qt1-x836.google.com [IPv6:2607:f8b0:4864:20::836]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 088766A480 for ; Thu, 1 Sep 2022 15:18:11 -0700 (PDT) Received: by mail-qt1-x836.google.com with SMTP id x5so219571qtv.9 for ; Thu, 01 Sep 2022 15:18:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc; bh=uvMol3uyneGva4YyiwYlT7IjMY9wNfnMaY4BpajVnC0=; b=vB0TPhC+ETNKaorpvy0zXtEQMbIqrE0wkU/h5SP9iHJyWzeG/Kp/jpHxx6dt746Npa QOHvSTUY4nY6DTxMILIli4C/sMT/xiS5q23jK7GsK9NDcZWBJDyMJp2HL8fP0VFkqEZF VvTw7as9Jv9mHmLJUawLZKjNlgVaU2bZ1YoPg= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=uvMol3uyneGva4YyiwYlT7IjMY9wNfnMaY4BpajVnC0=; b=WcxjkjfmpZtim2AxM++DlupfUMcMwdXJjcZjjxgmnWLsylJT8G6S0LOdpra8Z20Xj6 l2o1lp2CaexceR/8eNoBiZ35bDBRg/FMfpQAFd76QwMxYpIrlXIsdjjYYh+tfM+vnWhI 2L1OiL8FWn9aWxyoEAUclWQYLJaog8NxE0dy5/OxfV8CHZOaYLpUdwSvEyYwfznuV8hS iNnPfc6daYM4VfzOE6bGcEo8wsuRs/fKrK4acUzq1VytCO4ccE2Dtq0y6p3L+JKA5V7g s6iFptzAmp29vGNVJmT5HvCgO0Tiqc+zCC9FQiwx5UY6AD9eU6XISNAj6BDCFW7/mo0e 5boA== X-Gm-Message-State: ACgBeo10CsSiwsun/I7EUY83unE5LcPfGPw/I4dKGiiyGqceHxn2ZjIW BCYI22HoB44iBN3qK3xMVu3FwVT3MQVuDg== X-Google-Smtp-Source: AA6agR7IUlccHMDPHEeKk4mHGnWwUpV/eTK+kjnIV8R3LGYG/QjIQq6jZqPANDerA0E/GIkXT0vkdA== X-Received: by 2002:ac8:58c8:0:b0:344:87c2:c495 with SMTP id u8-20020ac858c8000000b0034487c2c495mr25705752qta.631.1662070689867; Thu, 01 Sep 2022 15:18:09 -0700 (PDT) Received: from joelboxx.c.googlers.com.com (228.221.150.34.bc.googleusercontent.com. [34.150.221.228]) by smtp.gmail.com with ESMTPSA id s16-20020ac85290000000b0034305a91aaesm11060794qtn.83.2022.09.01.15.18.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Sep 2022 15:18:09 -0700 (PDT) From: "Joel Fernandes (Google)" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, rushikesh.s.kadam@intel.com, urezki@gmail.com, neeraj.iitr10@gmail.com, frederic@kernel.org, paulmck@kernel.org, rostedt@goodmis.org, vineeth@bitbyteword.org, boqun.feng@gmail.com, "Joel Fernandes (Google)" Subject: [PATCH v5 17/18] i915: Move call_rcu() to call_rcu_lazy() Date: Thu, 1 Sep 2022 22:17:19 +0000 Message-Id: <20220901221720.1105021-18-joel@joelfernandes.org> X-Mailer: git-send-email 2.37.2.789.g6183377224-goog In-Reply-To: <20220901221720.1105021-1-joel@joelfernandes.org> References: <20220901221720.1105021-1-joel@joelfernandes.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: rcu@vger.kernel.org This is required to prevent callbacks triggering RCU machinery too quickly and too often, which adds more power to the system. Signed-off-by: Joel Fernandes (Google) --- drivers/gpu/drm/i915/gem/i915_gem_object.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drivers/gpu/drm/i915/gem/i915_gem_object.c b/drivers/gpu/drm/i915/gem/i915_gem_object.c index ccec4055fde3..21d6f66fb394 100644 --- a/drivers/gpu/drm/i915/gem/i915_gem_object.c +++ b/drivers/gpu/drm/i915/gem/i915_gem_object.c @@ -343,7 +343,7 @@ static void __i915_gem_free_objects(struct drm_i915_private *i915, __i915_gem_free_object(obj); /* But keep the pointer alive for RCU-protected lookups */ - call_rcu(&obj->rcu, __i915_gem_free_object_rcu); + call_rcu_lazy(&obj->rcu, __i915_gem_free_object_rcu); cond_resched(); } } From patchwork Thu Sep 1 22:17:20 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joel Fernandes X-Patchwork-Id: 12963383 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D9A83C6FA82 for ; Thu, 1 Sep 2022 22:18:52 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234937AbiIAWSv (ORCPT ); Thu, 1 Sep 2022 18:18:51 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46250 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234960AbiIAWSm (ORCPT ); Thu, 1 Sep 2022 18:18:42 -0400 Received: from mail-qt1-x82f.google.com (mail-qt1-x82f.google.com [IPv6:2607:f8b0:4864:20::82f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C60EE6BD68 for ; Thu, 1 Sep 2022 15:18:11 -0700 (PDT) Received: by mail-qt1-x82f.google.com with SMTP id cb8so281104qtb.0 for ; Thu, 01 Sep 2022 15:18:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc; bh=8gmIjQiZKF8ycgdoC5ApOjU5UIrSl/T2FqJa/TgPO1I=; b=Gi604wSkIg+z78h4KmndskftB7f3p4DdvyE1ILism9RQ5yjlyt75A9qW/StOXfb+fU pDy5O0BpNJsnx5TS3sl3liH6GLjNaUxC14UjaYlRGEFOLzqWV+fbldv/OdhgigmrP6sa icbHfTEFZStTZC5sYgXRiVe8Q6pqgB7yD7VlM= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=8gmIjQiZKF8ycgdoC5ApOjU5UIrSl/T2FqJa/TgPO1I=; b=12Hhz+R3h8ob8vavJwogjKYs01Ac0JXiezGZaWBCtCcd3o56nCFd6AZqdP63/pjyAH Bf/BTBjGnYsJ3C8nYz2Ycx36wS/bnQjRC+6IrlYGppLlw2FdPxCjI4c2FUs1eMZSxw7Q 50thMXi1e3FrhYdImxE3ZDqV3NDtMOUGqU+ogX6Yvbb5HtOj5TFpmDi8utgZBAlCgnft ucFG3O3jWh2IVUoqwhZTJUzLzF2qtCJgRigf310ut1aLTJ/PBk3vTB78r6huwMIWpxQh xEOVyH1WYCaqlLjD9thFk7yZxirueez81lDvnqIW4ZmWXvN/OEXxTLmcJoPM9so41JCC JBbg== X-Gm-Message-State: ACgBeo3wiUQmjAOYDaptnU3E2tnjNNvz5F2UgGLcaAQCmiMv1B22827P CTmtNAZHnebOFAOsjP54j4788Wnbcawu5g== X-Google-Smtp-Source: AA6agR7ds55vxuronbAkhPvG+w3tnnySL6akfA2/bMY5WcFbqrnJ09MHDN2EqtY3lODJWuD4wHe3wA== X-Received: by 2002:a05:622a:198b:b0:344:7de5:c516 with SMTP id u11-20020a05622a198b00b003447de5c516mr25834335qtc.7.1662070690604; Thu, 01 Sep 2022 15:18:10 -0700 (PDT) Received: from joelboxx.c.googlers.com.com (228.221.150.34.bc.googleusercontent.com. [34.150.221.228]) by smtp.gmail.com with ESMTPSA id s16-20020ac85290000000b0034305a91aaesm11060794qtn.83.2022.09.01.15.18.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Sep 2022 15:18:10 -0700 (PDT) From: "Joel Fernandes (Google)" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, rushikesh.s.kadam@intel.com, urezki@gmail.com, neeraj.iitr10@gmail.com, frederic@kernel.org, paulmck@kernel.org, rostedt@goodmis.org, vineeth@bitbyteword.org, boqun.feng@gmail.com, "Joel Fernandes (Google)" Subject: [PATCH v5 18/18] fork: Move thread_stack_free_rcu() to call_rcu_lazy() Date: Thu, 1 Sep 2022 22:17:20 +0000 Message-Id: <20220901221720.1105021-19-joel@joelfernandes.org> X-Mailer: git-send-email 2.37.2.789.g6183377224-goog In-Reply-To: <20220901221720.1105021-1-joel@joelfernandes.org> References: <20220901221720.1105021-1-joel@joelfernandes.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: rcu@vger.kernel.org This is required to prevent callbacks triggering RCU machinery too quickly and too often, which adds more power to the system. Signed-off-by: Joel Fernandes (Google) --- kernel/fork.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/kernel/fork.c b/kernel/fork.c index 90c85b17bf69..08a2298b1f94 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -226,7 +226,7 @@ static void thread_stack_delayed_free(struct task_struct *tsk) struct vm_stack *vm_stack = tsk->stack; vm_stack->stack_vm_area = tsk->stack_vm_area; - call_rcu(&vm_stack->rcu, thread_stack_free_rcu); + call_rcu_lazy(&vm_stack->rcu, thread_stack_free_rcu); } static int free_vm_stack_cache(unsigned int cpu) @@ -353,7 +353,7 @@ static void thread_stack_delayed_free(struct task_struct *tsk) { struct rcu_head *rh = tsk->stack; - call_rcu(rh, thread_stack_free_rcu); + call_rcu_lazy(rh, thread_stack_free_rcu); } static int alloc_thread_stack_node(struct task_struct *tsk, int node) @@ -388,7 +388,7 @@ static void thread_stack_delayed_free(struct task_struct *tsk) { struct rcu_head *rh = tsk->stack; - call_rcu(rh, thread_stack_free_rcu); + call_rcu_lazy(rh, thread_stack_free_rcu); } static int alloc_thread_stack_node(struct task_struct *tsk, int node)