From patchwork Wed Jun 22 22:50:56 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joel Fernandes X-Patchwork-Id: 12891553 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 79F16CCA47F for ; Wed, 22 Jun 2022 22:51:51 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1359766AbiFVWvs (ORCPT ); Wed, 22 Jun 2022 18:51:48 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45634 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1347009AbiFVWvq (ORCPT ); Wed, 22 Jun 2022 18:51:46 -0400 Received: from mail-qv1-xf36.google.com (mail-qv1-xf36.google.com [IPv6:2607:f8b0:4864:20::f36]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0BFC540E72 for ; Wed, 22 Jun 2022 15:51:44 -0700 (PDT) Received: by mail-qv1-xf36.google.com with SMTP id 59so4714759qvb.3 for ; Wed, 22 Jun 2022 15:51:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=tZ2VmzE2lHGW4TDS5ibElbKctouxZ1GAoFxv9DWSNFQ=; b=uAvtO1wTymA1OKGzMjhdAg4Xrl4FVIKHWotoJraXcn07le/b0XJ7l01X76a8WK4Q90 kmUbwQlNMXoi/6t3IXkCKFj3BN/t49X5NpYdWKYhWTCqUvfHy+wkYe934MhNMPgDiTFa hN6xqJX72Ktoan5doV3c3TtrMl4Bo7T3X2iuo= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=tZ2VmzE2lHGW4TDS5ibElbKctouxZ1GAoFxv9DWSNFQ=; b=aNCa1weMnednvKaDxuHLapU7votxf7ESZylE0nZFoKqWfCmIV39UmY4wzNAYNHhIPx dG1A6OcgyNCKXWnyYAS6Rf3xiTQ5LN2VTvqgX3WO8es72KIXorrUefTUglixt5kbOPIb Nfv4N51z24+8ecdqRRsoW/vLSqxFDbGiNyAHO5ki/G6Dc67edJawsaa6TsEmsZeiyBoJ o9sbQFC5NjoNADJ9ygkz7UzpON6HLGnn/nQlcJgAfVFPEgSijMKurYYgL8MUz7KQtYZu DLURW17yt2bH8J0WNFZ8Vv/67t/Fuzmi6omyUKa7GXTn3tQOi2hmsM2aSaFeQWh6Vcx5 N0hw== X-Gm-Message-State: AJIora9vERdp3XcdbUzAIwYUhTfUqH31gErSC/1jqzhfrOgKQ+T3efBz 5lSsJuKF2TFvuZizKbFS7lDC/JiTOg8KJQ== X-Google-Smtp-Source: AGRyM1tPJC4HgfQNhb89caYKVnJktcVYAzMTrPx9I8PteeedjGdtVYdEKZzoNlG7gLoLRzY4NHCDLQ== X-Received: by 2002:a05:6214:f08:b0:470:519a:168a with SMTP id gw8-20020a0562140f0800b00470519a168amr9962716qvb.29.1655938303002; Wed, 22 Jun 2022 15:51:43 -0700 (PDT) Received: from joelboxx.c.googlers.com.com (228.221.150.34.bc.googleusercontent.com. [34.150.221.228]) by smtp.gmail.com with ESMTPSA id k66-20020a37a145000000b006ab91fd03fasm1794022qke.19.2022.06.22.15.51.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 22 Jun 2022 15:51:42 -0700 (PDT) From: "Joel Fernandes (Google)" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, rushikesh.s.kadam@intel.com, urezki@gmail.com, neeraj.iitr10@gmail.com, frederic@kernel.org, paulmck@kernel.org, rostedt@goodmis.org, vineeth@bitbyteword.org, Joel Fernandes Subject: [PATCH v2 2/8] rcu: shrinker for lazy rcu Date: Wed, 22 Jun 2022 22:50:56 +0000 Message-Id: <20220622225102.2112026-4-joel@joelfernandes.org> X-Mailer: git-send-email 2.37.0.rc0.104.g0611611a94-goog In-Reply-To: <20220622225102.2112026-1-joel@joelfernandes.org> References: <20220622225102.2112026-1-joel@joelfernandes.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: rcu@vger.kernel.org From: Vineeth Pillai The shrinker is used to speed up the free'ing of memory potentially held by RCU lazy callbacks. RCU kernel module test cases show this to be effective. Test is introduced in a later patch. Signed-off-by: Vineeth Pillai Signed-off-by: Joel Fernandes (Google) --- kernel/rcu/rcu_segcblist.h | 14 +++++++++-- kernel/rcu/tree_nocb.h | 48 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 60 insertions(+), 2 deletions(-) diff --git a/kernel/rcu/rcu_segcblist.h b/kernel/rcu/rcu_segcblist.h index c3d7de65b689..cf71425dbb5e 100644 --- a/kernel/rcu/rcu_segcblist.h +++ b/kernel/rcu/rcu_segcblist.h @@ -15,16 +15,26 @@ static inline long rcu_cblist_n_cbs(struct rcu_cblist *rclp) return READ_ONCE(rclp->len); } +#ifdef CONFIG_RCU_LAZY /* Return number of callbacks in the specified callback list. */ static inline long rcu_cblist_n_lazy_cbs(struct rcu_cblist *rclp) { -#ifdef CONFIG_RCU_LAZY return READ_ONCE(rclp->lazy_len); +} + +static inline void rcu_cblist_reset_lazy_len(struct rcu_cblist *rclp) +{ + WRITE_ONCE(rclp->lazy_len, 0); +} #else +static inline long rcu_cblist_n_lazy_cbs(struct rcu_cblist *rclp) +{ return 0; -#endif } +static inline void rcu_cblist_reset_lazy_len(struct rcu_cblist *rclp) {} +#endif + /* Return number of callbacks in segmented callback list by summing seglen. */ long rcu_segcblist_n_segment_cbs(struct rcu_segcblist *rsclp); diff --git a/kernel/rcu/tree_nocb.h b/kernel/rcu/tree_nocb.h index b9244f22e102..2f5da12811a5 100644 --- a/kernel/rcu/tree_nocb.h +++ b/kernel/rcu/tree_nocb.h @@ -1207,6 +1207,51 @@ int rcu_nocb_cpu_offload(int cpu) } EXPORT_SYMBOL_GPL(rcu_nocb_cpu_offload); +static unsigned long +lazy_rcu_shrink_count(struct shrinker *shrink, struct shrink_control *sc) { + int cpu; + unsigned long count = 0; + + /* Snapshot count of all CPUs */ + for_each_possible_cpu(cpu) { + struct rcu_data *rdp = per_cpu_ptr(&rcu_data, cpu); + count += rcu_cblist_n_lazy_cbs(&rdp->nocb_bypass); + } + + return count ? count : SHRINK_EMPTY; +} + +static unsigned long +lazy_rcu_shrink_scan(struct shrinker *shrink, struct shrink_control *sc) { + int cpu; + unsigned long flags; + unsigned long count = 0; + + /* Snapshot count of all CPUs */ + for_each_possible_cpu(cpu) { + struct rcu_data *rdp = per_cpu_ptr(&rcu_data, cpu); + int _count = rcu_cblist_n_lazy_cbs(&rdp->nocb_bypass); + if (_count == 0) + continue; + rcu_nocb_lock_irqsave(rdp, flags); + rcu_cblist_reset_lazy_len(&rdp->nocb_bypass); + rcu_nocb_unlock_irqrestore(rdp, flags); + wake_nocb_gp(rdp, false); + sc->nr_to_scan -= _count; + count += _count; + if (sc->nr_to_scan <= 0) + break; + } + return count ? count : SHRINK_STOP; +} + +static struct shrinker lazy_rcu_shrinker = { + .count_objects = lazy_rcu_shrink_count, + .scan_objects = lazy_rcu_shrink_scan, + .batch = 0, + .seeks = DEFAULT_SEEKS, +}; + void __init rcu_init_nohz(void) { int cpu; @@ -1244,6 +1289,9 @@ void __init rcu_init_nohz(void) if (!rcu_state.nocb_is_setup) return; + if (register_shrinker(&lazy_rcu_shrinker)) + pr_err("Failed to register lazy_rcu shrinker!\n"); + #if defined(CONFIG_NO_HZ_FULL) if (tick_nohz_full_running) cpumask_or(rcu_nocb_mask, rcu_nocb_mask, tick_nohz_full_mask);