From patchwork Tue Jul 25 23:29:06 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joel Fernandes X-Patchwork-Id: 13327316 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 85B52C001DF for ; Tue, 25 Jul 2023 23:29:31 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231714AbjGYX3a (ORCPT ); Tue, 25 Jul 2023 19:29:30 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39566 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231636AbjGYX33 (ORCPT ); Tue, 25 Jul 2023 19:29:29 -0400 Received: from mail-io1-xd31.google.com (mail-io1-xd31.google.com [IPv6:2607:f8b0:4864:20::d31]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5F963120 for ; Tue, 25 Jul 2023 16:29:28 -0700 (PDT) Received: by mail-io1-xd31.google.com with SMTP id ca18e2360f4ac-78bb3ff7cbcso113967539f.3 for ; Tue, 25 Jul 2023 16:29:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; t=1690327768; x=1690932568; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=SjqXIuVtq0u8YshojVwovW+pCVt0mgB/LilQjFvvm4k=; b=Qe9sSQNhnm2GdgDMjBEvMNzqgoUbTZUEYU+4P5xV+G0XGcheStBtG4dXcqhIqRC+Qw 6Appyh/rwaegQl6X307xDZAU2WiYBskonXGtUi++Tvwk5fjxWw6roo0BZ+1B4hrYS879 YIWAL2TlvrswSmzRpSN/9zRaKs+QVVJ2oq8tk= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1690327768; x=1690932568; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=SjqXIuVtq0u8YshojVwovW+pCVt0mgB/LilQjFvvm4k=; b=JEiwqj+FY+nLpnkTB7T5JTFlcYgoGJToz7WUOJ5NIAtm4+UDkIFmAd5k2wM25Yv5SW JwU2D0cXd/8/j+Pk0nU77TIsE/8ruEqbAWqJrkNUExLFvc/EAeUT5CeHMNN5oPZyZT9C DrwwPJX0qCG0JHu2ZLqerPkNMHEsmoPxSIU3eXEY1D6bkMYFhicjmh3Cae5KS/IBy5fr UqHvh9hoDuBzxE3TFasjYRofHPf6F7/zbtm/bZStA1BCJ1yxgrgd/HO4E7YAaBBk+6Fo zFUb+PRHy0qC4JbA4lTMOaonuaRA5hwYfxHNYKs4U+sV8G1CUW6xYhYHkoL3RbL5GEn3 SPNg== X-Gm-Message-State: ABy/qLZEQh9Tixg7yEYqmnqR7TTmHMqZnixgpA08OPZG4u7ekiar9eEK 4TVCpKJkcdCKBwJ5fSVlfhbbww== X-Google-Smtp-Source: APBJJlHSQsPaCUv83pDozY5eFiLygkk647/+RNJH82m94E6xEn7hjRbJJFpy2CXENNqGa4wtwOyDwg== X-Received: by 2002:a6b:e818:0:b0:783:6ff9:a6c7 with SMTP id f24-20020a6be818000000b007836ff9a6c7mr334374ioh.6.1690327767704; Tue, 25 Jul 2023 16:29:27 -0700 (PDT) Received: from joelboxx5.c.googlers.com.com (156.190.123.34.bc.googleusercontent.com. [34.123.190.156]) by smtp.gmail.com with ESMTPSA id m18-20020a02c892000000b0041d859c5721sm3932053jao.64.2023.07.25.16.29.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 25 Jul 2023 16:29:26 -0700 (PDT) From: "Joel Fernandes (Google)" To: linux-kernel@vger.kernel.org, Davidlohr Bueso , "Paul E. McKenney" , Josh Triplett Cc: rcu@vger.kernel.org, "Joel Fernandes (Google)" Subject: [PATCH 1/5] rcutorture: Fix stuttering races and other issues Date: Tue, 25 Jul 2023 23:29:06 +0000 Message-ID: <20230725232913.2981357-2-joel@joelfernandes.org> X-Mailer: git-send-email 2.41.0.487.g6d72f3e995-goog In-Reply-To: <20230725232913.2981357-1-joel@joelfernandes.org> References: <20230725232913.2981357-1-joel@joelfernandes.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: rcu@vger.kernel.org The stuttering code isn't functioning as expected. Ideally, it should pause the torture threads for a designated period before resuming. Yet, it fails to halt the test for the correct duration. Additionally, a race condition exists, potentially causing the stuttering code to pause for an extended period if the 'spt' variable is non-zero due to the stutter orchestration thread's inadequate CPU time. Moreover, over-stuttering can hinder RCU's progress on TREE07 kernels. This happens as the stuttering code may run within a softirq due to RCU callbacks. Consequently, ksoftirqd keeps a CPU busy for several seconds, thus obstructing RCU's progress. This situation triggers a warning message in the logs: [ 2169.481783] rcu_torture_writer: rtort_pipe_count: 9 This warning suggests that an RCU torture object, although invisible to RCU readers, couldn't make it past the pipe array and be freed -- a strong indication that there weren't enough grace periods during the stutter interval. To address these issues, this patch sets the "stutter end" time to an absolute point in the future set by the main stutter thread. This is then used for waiting in stutter_wait(). While the stutter thread still defines this absolute time, the waiters' waiting logic doesn't rely on the stutter thread receiving sufficient CPU time to halt the stuttering as the halting is now self-controlled. Signed-off-by: Joel Fernandes (Google) --- kernel/torture.c | 46 +++++++++++++--------------------------------- 1 file changed, 13 insertions(+), 33 deletions(-) diff --git a/kernel/torture.c b/kernel/torture.c index 68dba4ecab5c..63f8f2a7d960 100644 --- a/kernel/torture.c +++ b/kernel/torture.c @@ -719,7 +719,7 @@ static void torture_shutdown_cleanup(void) * suddenly applied to or removed from the system. */ static struct task_struct *stutter_task; -static int stutter_pause_test; +static ktime_t stutter_till_abs_time; static int stutter; static int stutter_gap; @@ -729,30 +729,17 @@ static int stutter_gap; */ bool stutter_wait(const char *title) { - unsigned int i = 0; bool ret = false; - int spt; + ktime_t now_ns, till_ns; cond_resched_tasks_rcu_qs(); - spt = READ_ONCE(stutter_pause_test); - for (; spt; spt = READ_ONCE(stutter_pause_test)) { - if (!ret && !rt_task(current)) { - sched_set_normal(current, MAX_NICE); - ret = true; - } - if (spt == 1) { - torture_hrtimeout_jiffies(1, NULL); - } else if (spt == 2) { - while (READ_ONCE(stutter_pause_test)) { - if (!(i++ & 0xffff)) - torture_hrtimeout_us(10, 0, NULL); - cond_resched(); - } - } else { - torture_hrtimeout_jiffies(round_jiffies_relative(HZ), NULL); - } - torture_shutdown_absorb(title); + now_ns = ktime_get(); + till_ns = READ_ONCE(stutter_till_abs_time); + if (till_ns && ktime_before(now_ns, till_ns)) { + torture_hrtimeout_ns(ktime_sub(till_ns, now_ns), 0, NULL); + ret = true; } + torture_shutdown_absorb(title); return ret; } EXPORT_SYMBOL_GPL(stutter_wait); @@ -763,23 +750,16 @@ EXPORT_SYMBOL_GPL(stutter_wait); */ static int torture_stutter(void *arg) { - DEFINE_TORTURE_RANDOM(rand); - int wtime; + ktime_t till_ns; VERBOSE_TOROUT_STRING("torture_stutter task started"); do { if (!torture_must_stop() && stutter > 1) { - wtime = stutter; - if (stutter > 2) { - WRITE_ONCE(stutter_pause_test, 1); - wtime = stutter - 3; - torture_hrtimeout_jiffies(wtime, &rand); - wtime = 2; - } - WRITE_ONCE(stutter_pause_test, 2); - torture_hrtimeout_jiffies(wtime, NULL); + till_ns = ktime_add_ns(ktime_get(), + jiffies_to_nsecs(stutter)); + WRITE_ONCE(stutter_till_abs_time, till_ns); + torture_hrtimeout_jiffies(stutter - 1, NULL); } - WRITE_ONCE(stutter_pause_test, 0); if (!torture_must_stop()) torture_hrtimeout_jiffies(stutter_gap, NULL); torture_shutdown_absorb("torture_stutter"); From patchwork Tue Jul 25 23:29:07 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joel Fernandes X-Patchwork-Id: 13327317 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 95662C001E0 for ; Tue, 25 Jul 2023 23:29:33 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231791AbjGYX3c (ORCPT ); Tue, 25 Jul 2023 19:29:32 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39590 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231636AbjGYX3b (ORCPT ); Tue, 25 Jul 2023 19:29:31 -0400 Received: from mail-io1-xd2d.google.com (mail-io1-xd2d.google.com [IPv6:2607:f8b0:4864:20::d2d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D85391BE8 for ; Tue, 25 Jul 2023 16:29:29 -0700 (PDT) Received: by mail-io1-xd2d.google.com with SMTP id ca18e2360f4ac-783549ef058so316767539f.2 for ; Tue, 25 Jul 2023 16:29:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; t=1690327769; x=1690932569; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=t1/tyaZq/EJBhi/7l+hIL3z3pGtmH2Zjtk0EiAh2ooA=; b=W4LU5bSa/QHlkTdxIb4sY4pM7LWuU9noqwcduFR+0CmKenXKzGAI5fNoSvgjHOSlUm IwiPuf4Gm81MY4HrBIdQsBc4MnQ8nMUuNbJ7L7Lgmjqm9m9qXFc6vriyFqRhMkbcPGUE QC142xSTHhGMr/9RiF3+UQBA1dMQ1hLm+EzgM= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1690327769; x=1690932569; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=t1/tyaZq/EJBhi/7l+hIL3z3pGtmH2Zjtk0EiAh2ooA=; b=dA/pH4/q1taswvfRQZ8Swg8Ira/2sHrcv/tU0zCFf1n513QO8HkoInvgRnYxPJKh4a +tq/jiRhO24JPLFfuny7TyJlCFLHRyyXml3iNltvBYAET3DgQgtLb9e/pXAEzpI2ec9J qJ41IIya5dyVF3fEeEEj+CXFxa34x0o6YO+rmY/QnE+egDJNA8GuxAMgwyLdztYdrlA2 940mXNuz+vfnGrMGTp976kujIFXH8B127d1Ayy0ghhwLOZBycFmkTeSRBs/E9dC8L59l HXnz4jfFuUEFJR8IjAhcZf5N3E6vj6wgY36JGtd7Q3v9tc4+OdbIZhyPJgsOqm6FPzDO 3ydw== X-Gm-Message-State: ABy/qLYGkDDztYPX/PF6zeizbfgSNb2bk47zmkzE+Jb+WKeVBwyHOwOj Axy9PU/Y3tTypnNpeALv182TZIGEpzZhJHfMrr0= X-Google-Smtp-Source: APBJJlFTSX89ddGUxW0LY7A4uygDts9vWsSV2ZCphFzDYxJVS7ZYCAeaT9OC8EaxmdhM7TusKTii5Q== X-Received: by 2002:a6b:5c10:0:b0:783:72d4:8c38 with SMTP id z16-20020a6b5c10000000b0078372d48c38mr341104ioh.13.1690327769166; Tue, 25 Jul 2023 16:29:29 -0700 (PDT) Received: from joelboxx5.c.googlers.com.com (156.190.123.34.bc.googleusercontent.com. [34.123.190.156]) by smtp.gmail.com with ESMTPSA id m18-20020a02c892000000b0041d859c5721sm3932053jao.64.2023.07.25.16.29.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 25 Jul 2023 16:29:28 -0700 (PDT) From: "Joel Fernandes (Google)" To: linux-kernel@vger.kernel.org, Lai Jiangshan , "Paul E. McKenney" , Josh Triplett , Steven Rostedt , Mathieu Desnoyers , rcu@vger.kernel.org (open list:SLEEPABLE READ-COPY UPDATE (SRCU)) Cc: rcu@vger.kernel.org, "Joel Fernandes (Google)" Subject: [PATCH 2/5] srcu: Fix error handling in init_srcu_struct_fields() Date: Tue, 25 Jul 2023 23:29:07 +0000 Message-ID: <20230725232913.2981357-3-joel@joelfernandes.org> X-Mailer: git-send-email 2.41.0.487.g6d72f3e995-goog In-Reply-To: <20230725232913.2981357-1-joel@joelfernandes.org> References: <20230725232913.2981357-1-joel@joelfernandes.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: rcu@vger.kernel.org The current error handling in init_srcu_struct_fields() is a bit inconsistent. If init_srcu_struct_nodes() fails, the function either returns -ENOMEM or 0 depending on whether ssp->sda_is_static is true or false. This can make init_srcu_struct_fields() return 0 even if memory allocation failed! Simplify the error handling by always returning -ENOMEM if either init_srcu_struct_nodes() or the per-CPU allocation fails. This makes the control flow easier to follow and avoids the inconsistent return values. Add goto labels to avoid duplicating the error cleanup code. Link: https://lore.kernel.org/r/20230404003508.GA254019@google.com Signed-off-by: Joel Fernandes (Google) --- kernel/rcu/srcutree.c | 31 +++++++++++++++++-------------- 1 file changed, 17 insertions(+), 14 deletions(-) diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c index 20d7a238d675..cbc37cbc1805 100644 --- a/kernel/rcu/srcutree.c +++ b/kernel/rcu/srcutree.c @@ -255,29 +255,32 @@ static int init_srcu_struct_fields(struct srcu_struct *ssp, bool is_static) ssp->srcu_sup->sda_is_static = is_static; if (!is_static) ssp->sda = alloc_percpu(struct srcu_data); - if (!ssp->sda) { - if (!is_static) - kfree(ssp->srcu_sup); - return -ENOMEM; - } + if (!ssp->sda) + goto err_free_sup; init_srcu_struct_data(ssp); ssp->srcu_sup->srcu_gp_seq_needed_exp = 0; ssp->srcu_sup->srcu_last_gp_end = ktime_get_mono_fast_ns(); if (READ_ONCE(ssp->srcu_sup->srcu_size_state) == SRCU_SIZE_SMALL && SRCU_SIZING_IS_INIT()) { - if (!init_srcu_struct_nodes(ssp, GFP_ATOMIC)) { - if (!ssp->srcu_sup->sda_is_static) { - free_percpu(ssp->sda); - ssp->sda = NULL; - kfree(ssp->srcu_sup); - return -ENOMEM; - } - } else { + if (!init_srcu_struct_nodes(ssp, GFP_ATOMIC)) + goto err_free_sda; + else WRITE_ONCE(ssp->srcu_sup->srcu_size_state, SRCU_SIZE_BIG); - } } ssp->srcu_sup->srcu_ssp = ssp; smp_store_release(&ssp->srcu_sup->srcu_gp_seq_needed, 0); /* Init done. */ return 0; + +err_free_sda: + if (!is_static) { + free_percpu(ssp->sda); + ssp->sda = NULL; + } +err_free_sup: + if (!is_static) { + kfree(ssp->srcu_sup); + ssp->srcu_sup = NULL; + } + return -ENOMEM; } #ifdef CONFIG_DEBUG_LOCK_ALLOC From patchwork Tue Jul 25 23:29:08 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joel Fernandes X-Patchwork-Id: 13327318 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5D929EB64DD for ; Tue, 25 Jul 2023 23:29:35 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231808AbjGYX3d (ORCPT ); Tue, 25 Jul 2023 19:29:33 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39600 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231790AbjGYX3c (ORCPT ); Tue, 25 Jul 2023 19:29:32 -0400 Received: from mail-io1-xd2a.google.com (mail-io1-xd2a.google.com [IPv6:2607:f8b0:4864:20::d2a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 316F419BA for ; Tue, 25 Jul 2023 16:29:31 -0700 (PDT) Received: by mail-io1-xd2a.google.com with SMTP id ca18e2360f4ac-78363cc070aso316961339f.1 for ; Tue, 25 Jul 2023 16:29:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; t=1690327770; x=1690932570; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=B6faQrb3vVDF+1ZA3HcLro9nEgsaXGn2RtvbxbNNEvc=; b=dOKqCkxitVYH7c0pdupRnZpK5/l2nGavKWDlS+ALoRAPtAH4m3BstqX7xtaGRRb1Wf d3seuANAUNcBh2wORYzQfdJRgZyYXkIXgQWow/2GUL1DlZMFSx0ijE0bHqYlnFjrE4P5 mHXFGWUEOy2hIGqzMfbeyqOFGR7zJNac5++X4= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1690327770; x=1690932570; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=B6faQrb3vVDF+1ZA3HcLro9nEgsaXGn2RtvbxbNNEvc=; b=j/Zdk31xgvonFpXmpsGKGbRskA/yw3JLcHhP2k3dZxY6wi0f1KUUtqfhloCdDwwT46 rdMu47QDugT5ovXpn/7qiuAuIKmUBdxYWITxj7v9+q0gUcNTCyVTul1xyg3HJnjAu85M BuXR3up0PvNrRpziLvdOPZLnNXNgqS29YIu+v73vPgtSqEti2blT7MXkyV3YTLrdFW65 mWVdwdHVngWeLcxDNyZcbTjM6x9xqszdo0Nh2CRqT60e3R8I5U5NXmKbbtdOK/XQHhuG 0HeleIx1tYwQ0nrLXUzpQkkSH3FG87qicydrF2NUK2JBwT2k81xNlkfUMRcpiBCMVdAv jBbQ== X-Gm-Message-State: ABy/qLb58Bat8Pmv7DGiIcqUM11C2Dn/4hb35iM/zIX7Z2mwWWrRoA2A 5kutcc2gI/GieRFHjIs8RNbS1QB8DLE69CiuSrE= X-Google-Smtp-Source: APBJJlEZFnEQQK9J1v8M71kPDaqQ8ZTeYcDEjV+OUIXcmUtYKVYoXTApZUTw4E9efh8aj3vI2iBVkA== X-Received: by 2002:a6b:e80e:0:b0:786:7100:72de with SMTP id f14-20020a6be80e000000b00786710072demr346606ioh.16.1690327770577; Tue, 25 Jul 2023 16:29:30 -0700 (PDT) Received: from joelboxx5.c.googlers.com.com (156.190.123.34.bc.googleusercontent.com. [34.123.190.156]) by smtp.gmail.com with ESMTPSA id m18-20020a02c892000000b0041d859c5721sm3932053jao.64.2023.07.25.16.29.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 25 Jul 2023 16:29:29 -0700 (PDT) From: "Joel Fernandes (Google)" To: linux-kernel@vger.kernel.org, "Paul E. McKenney" , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Josh Triplett , Boqun Feng , Steven Rostedt , Mathieu Desnoyers , Lai Jiangshan , Zqiang , rcu@vger.kernel.org (open list:READ-COPY UPDATE (RCU)) Cc: rcu@vger.kernel.org Subject: [PATCH 3/5] tree/nocb: Adjust RCU_NOCB_WAKE_* macros from weaker to stronger Date: Tue, 25 Jul 2023 23:29:08 +0000 Message-ID: <20230725232913.2981357-4-joel@joelfernandes.org> X-Mailer: git-send-email 2.41.0.487.g6d72f3e995-goog In-Reply-To: <20230725232913.2981357-1-joel@joelfernandes.org> References: <20230725232913.2981357-1-joel@joelfernandes.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: rcu@vger.kernel.org This is needed to make the next patch work correctly as we rely on the strength of the wakeup when comparing deferred-wakeup types across different CPUs. Signed-off-by: Joel Fernandes (Google) --- kernel/rcu/tree.h | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/kernel/rcu/tree.h b/kernel/rcu/tree.h index 192536916f9a..0f40a9c2b78d 100644 --- a/kernel/rcu/tree.h +++ b/kernel/rcu/tree.h @@ -288,8 +288,8 @@ struct rcu_data { /* Values for nocb_defer_wakeup field in struct rcu_data. */ #define RCU_NOCB_WAKE_NOT 0 -#define RCU_NOCB_WAKE_BYPASS 1 -#define RCU_NOCB_WAKE_LAZY 2 +#define RCU_NOCB_WAKE_LAZY 1 +#define RCU_NOCB_WAKE_BYPASS 2 #define RCU_NOCB_WAKE 3 #define RCU_NOCB_WAKE_FORCE 4 From patchwork Tue Jul 25 23:29:09 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joel Fernandes X-Patchwork-Id: 13327319 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8AEFCEB64DD for ; Tue, 25 Jul 2023 23:29:46 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231821AbjGYX3p (ORCPT ); Tue, 25 Jul 2023 19:29:45 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39770 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232010AbjGYX3k (ORCPT ); Tue, 25 Jul 2023 19:29:40 -0400 Received: from mail-io1-xd2b.google.com (mail-io1-xd2b.google.com [IPv6:2607:f8b0:4864:20::d2b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 34E311FC0 for ; Tue, 25 Jul 2023 16:29:33 -0700 (PDT) Received: by mail-io1-xd2b.google.com with SMTP id ca18e2360f4ac-78360f61ba0so307488939f.1 for ; Tue, 25 Jul 2023 16:29:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; t=1690327772; x=1690932572; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=0DIiXFGh73Irk8k8jfSTlgn3pzuU9BPg/xYZpIge3SA=; b=devazoCc1ePSllifONKGQCl8V8kDvW5uvDEsIy/u3SDJfI/vz1wlFS0SUxKZdfqrcB NEV4tcQXSLdIR+lnufehU/yii4Le4bCblWZXsvn8b0MbCrfA5CbEFQh/KvLUcVyexJgD bAOucRRZCGzD+cJEPlkY3nJk/aUwJlJqC6WxA= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1690327772; x=1690932572; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=0DIiXFGh73Irk8k8jfSTlgn3pzuU9BPg/xYZpIge3SA=; b=MFPJCHdwrLZDzcR21jyeE/Nosj1E0sFqz+wrjW0V33BMECHiQlnmGj5MiUZILiDG+3 if++O+fPaGMFa6/ibNKNmDbOPMXJNAiAyoG/0HT9xQkxEsbkKa5HXD/UJCwUrDePdEFC 8P9I+An0GcZddfP20yImwkjC80NNZh1ztRe4443ePRkBZ7BtU+Iw2QXrJXBkyRLbej29 eBtVDYeUH5EXlQtTh7CDVPKREfl/i6gLK9pJM7bPCixYeDR9+fKNHmqPeACrGz//NML0 gH89Uj/LEMmHDerAQRFAAPsMnB3cLFcjJ1lD2L1hjkIJw9wGrikg3MAlo+7edUfh1bqd jBdQ== X-Gm-Message-State: ABy/qLaSS8L8UklwhD+ShnO9vLblpDbcX3eizHVrwQ8rxck88XanqfOc YKlZGeNagrhbicYWxtdLR9Hs4Q== X-Google-Smtp-Source: APBJJlH9/vaRwB5vMSyP0PFSl3tgH3jguGqOapInYGQWm35VNrAOnNszvyRrx5BqXu6IQCKvacXg1g== X-Received: by 2002:a05:6602:257a:b0:783:5fd9:3789 with SMTP id dj26-20020a056602257a00b007835fd93789mr367880iob.1.1690327772055; Tue, 25 Jul 2023 16:29:32 -0700 (PDT) Received: from joelboxx5.c.googlers.com.com (156.190.123.34.bc.googleusercontent.com. [34.123.190.156]) by smtp.gmail.com with ESMTPSA id m18-20020a02c892000000b0041d859c5721sm3932053jao.64.2023.07.25.16.29.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 25 Jul 2023 16:29:31 -0700 (PDT) From: "Joel Fernandes (Google)" To: linux-kernel@vger.kernel.org, "Paul E. McKenney" , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Josh Triplett , Boqun Feng , Steven Rostedt , Mathieu Desnoyers , Lai Jiangshan , Zqiang , rcu@vger.kernel.org (open list:READ-COPY UPDATE (RCU)) Cc: rcu@vger.kernel.org Subject: [PATCH 4/5] tree/nocb: Improve readability of nocb_gp_wait() Date: Tue, 25 Jul 2023 23:29:09 +0000 Message-ID: <20230725232913.2981357-5-joel@joelfernandes.org> X-Mailer: git-send-email 2.41.0.487.g6d72f3e995-goog In-Reply-To: <20230725232913.2981357-1-joel@joelfernandes.org> References: <20230725232913.2981357-1-joel@joelfernandes.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: rcu@vger.kernel.org The nocb_gp_wait() function contains logic to check each rdp's bypass list, flush if needed, and decide on wakeups. This makes the function hard to follow. Split out the bypass checking and flushing into a separate helper nocb_gp_flush_wake(). The new function encapsulates the logic to: - Check if the bypass needs to be flushed - Flush if needed - Return info on wakeups (lazy, bypass or none) nocb_gp_wait() now becomes simpler by calling the helper to handle the bypass flushing and waking logic. This splitting improves the readability and maintainability of the code by encapsulating related logic into a function with a clear purpose. Signed-off-by: Joel Fernandes (Google) --- kernel/rcu/tree_nocb.h | 113 ++++++++++++++++++++++++----------------- 1 file changed, 66 insertions(+), 47 deletions(-) diff --git a/kernel/rcu/tree_nocb.h b/kernel/rcu/tree_nocb.h index 5598212d1f27..c805825c3f00 100644 --- a/kernel/rcu/tree_nocb.h +++ b/kernel/rcu/tree_nocb.h @@ -670,22 +670,68 @@ static void nocb_gp_sleep(struct rcu_data *my_rdp, int cpu) trace_rcu_nocb_wake(rcu_state.name, cpu, TPS("EndSleep")); } +/* + * Given an rdp, flush its bypass list if needed and return information about + * if a deferred-wakeup needs to be organized depending on whether things are + * still in the bypass list. Also tell caller if the list was flushed and if it + * is still empty after any flushing. + */ +static int nocb_gp_flush_wake(struct rcu_data *rdp, bool *empty, bool *flush) +{ + long bypass_ncbs; + long lazy_ncbs; + unsigned long j = jiffies; + + trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, TPS("Check")); + lockdep_assert_held(&rdp->nocb_lock); + bypass_ncbs = rcu_cblist_n_cbs(&rdp->nocb_bypass); + lazy_ncbs = READ_ONCE(rdp->lazy_len); + + *flush = false; + *empty = false; + if (bypass_ncbs && (lazy_ncbs == bypass_ncbs) && + (time_after(j, READ_ONCE(rdp->nocb_bypass_first) + jiffies_till_flush) || + bypass_ncbs > 2 * qhimark)) { + *flush = true; + } else if (bypass_ncbs && (lazy_ncbs != bypass_ncbs) && + (time_after(j, READ_ONCE(rdp->nocb_bypass_first) + 1) || + bypass_ncbs > 2 * qhimark)) { + *flush = true; + } else if (!bypass_ncbs && rcu_segcblist_empty(&rdp->cblist)) { + *empty = true; + return RCU_NOCB_WAKE_NOT; + } + + if (*flush) { + // Bypass full or old, so flush it. + (void)rcu_nocb_try_flush_bypass(rdp, j); + bypass_ncbs = rcu_cblist_n_cbs(&rdp->nocb_bypass); + lazy_ncbs = READ_ONCE(rdp->lazy_len); + } + + if (bypass_ncbs) { + trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, + bypass_ncbs == lazy_ncbs ? TPS("Lazy") : TPS("Bypass")); + return (bypass_ncbs == lazy_ncbs ? RCU_NOCB_WAKE_LAZY : + RCU_NOCB_WAKE_BYPASS); + } + return RCU_NOCB_WAKE_NOT; +} + /* * No-CBs GP kthreads come here to wait for additional callbacks to show up * or for grace periods to end. */ static void nocb_gp_wait(struct rcu_data *my_rdp) { - bool bypass = false; int __maybe_unused cpu = my_rdp->cpu; unsigned long cur_gp_seq; unsigned long flags; bool gotcbs = false; - unsigned long j = jiffies; - bool lazy = false; bool needwait_gp = false; // This prevents actual uninitialized use. bool needwake; bool needwake_gp; + int defer_wake_type = RCU_NOCB_WAKE_NOT; struct rcu_data *rdp, *rdp_toggling = NULL; struct rcu_node *rnp; unsigned long wait_gp_seq = 0; // Suppress "use uninitialized" warning. @@ -712,44 +758,24 @@ static void nocb_gp_wait(struct rcu_data *my_rdp) * won't be ignored for long. */ list_for_each_entry(rdp, &my_rdp->nocb_head_rdp, nocb_entry_rdp) { - long bypass_ncbs; - bool flush_bypass = false; - long lazy_ncbs; + int defer_wake_type_one = RCU_NOCB_WAKE_NOT; + bool flushed; + bool empty; - trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, TPS("Check")); rcu_nocb_lock_irqsave(rdp, flags); - lockdep_assert_held(&rdp->nocb_lock); - bypass_ncbs = rcu_cblist_n_cbs(&rdp->nocb_bypass); - lazy_ncbs = READ_ONCE(rdp->lazy_len); + defer_wake_type_one = nocb_gp_flush_wake(rdp, &empty, &flushed); - if (bypass_ncbs && (lazy_ncbs == bypass_ncbs) && - (time_after(j, READ_ONCE(rdp->nocb_bypass_first) + jiffies_till_flush) || - bypass_ncbs > 2 * qhimark)) { - flush_bypass = true; - } else if (bypass_ncbs && (lazy_ncbs != bypass_ncbs) && - (time_after(j, READ_ONCE(rdp->nocb_bypass_first) + 1) || - bypass_ncbs > 2 * qhimark)) { - flush_bypass = true; - } else if (!bypass_ncbs && rcu_segcblist_empty(&rdp->cblist)) { - rcu_nocb_unlock_irqrestore(rdp, flags); - continue; /* No callbacks here, try next. */ - } + // We may need to do a deferred wakeup later for bypass/lazy + // So note down what we learnt from the rdp. + defer_wake_type = max(defer_wake_type_one, defer_wake_type); - if (flush_bypass) { - // Bypass full or old, so flush it. - (void)rcu_nocb_try_flush_bypass(rdp, j); - bypass_ncbs = rcu_cblist_n_cbs(&rdp->nocb_bypass); - lazy_ncbs = READ_ONCE(rdp->lazy_len); + // Did we make any updates to main cblist? If not, no + // non-deferred wake up to do for this rdp. + if (!flushed && empty) { + rcu_nocb_unlock_irqrestore(rdp, flags); + continue; } - if (bypass_ncbs) { - trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, - bypass_ncbs == lazy_ncbs ? TPS("Lazy") : TPS("Bypass")); - if (bypass_ncbs == lazy_ncbs) - lazy = true; - else - bypass = true; - } rnp = rdp->mynode; // Advance callbacks if helpful and low contention. @@ -792,23 +818,16 @@ static void nocb_gp_wait(struct rcu_data *my_rdp) rcu_gp_kthread_wake(); } - my_rdp->nocb_gp_bypass = bypass; + my_rdp->nocb_gp_bypass = (defer_wake_type == RCU_NOCB_WAKE_BYPASS); my_rdp->nocb_gp_gp = needwait_gp; my_rdp->nocb_gp_seq = needwait_gp ? wait_gp_seq : 0; // At least one child with non-empty ->nocb_bypass, so set // timer in order to avoid stranding its callbacks. - if (!rcu_nocb_poll) { - // If bypass list only has lazy CBs. Add a deferred lazy wake up. - if (lazy && !bypass) { - wake_nocb_gp_defer(my_rdp, RCU_NOCB_WAKE_LAZY, - TPS("WakeLazyIsDeferred")); - // Otherwise add a deferred bypass wake up. - } else if (bypass) { - wake_nocb_gp_defer(my_rdp, RCU_NOCB_WAKE_BYPASS, - TPS("WakeBypassIsDeferred")); - } - } + if (!rcu_nocb_poll && defer_wake_type != RCU_NOCB_WAKE_NOT) + wake_nocb_gp_defer(my_rdp, defer_wake_type, + defer_wake_type == RCU_NOCB_WAKE_LAZY ? + TPS("WakeLazyIsDeferred") : TPS("WakeBypassIsDeferred")); if (rcu_nocb_poll) { /* Polling, so trace if first poll in the series. */ From patchwork Tue Jul 25 23:29:10 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joel Fernandes X-Patchwork-Id: 13327320 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9BEC3C001DF for ; Tue, 25 Jul 2023 23:29:51 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231923AbjGYX3t (ORCPT ); Tue, 25 Jul 2023 19:29:49 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39880 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231907AbjGYX3o (ORCPT ); Tue, 25 Jul 2023 19:29:44 -0400 Received: from mail-io1-xd35.google.com (mail-io1-xd35.google.com [IPv6:2607:f8b0:4864:20::d35]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5D9A31FCF for ; Tue, 25 Jul 2023 16:29:34 -0700 (PDT) Received: by mail-io1-xd35.google.com with SMTP id ca18e2360f4ac-78360b822abso260457239f.2 for ; Tue, 25 Jul 2023 16:29:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; t=1690327773; x=1690932573; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ynoALsv1aca8y8xpjhuiunc/CTMUKjN2qeAE0MoxzWk=; b=T1/qgW80ZW0ruvAexTl3SwPC/VD8efNMinccPq8SOmll1PeYaL7nJcC+rT2y/ICCI/ Zm7HzPisHr2CWscQ8DNq4ATfmZQ4copg8Byy3hFGTtKvq0+ing3UNCdbIzc4z9yskHeM i6XYgDFBYacc8aSIAdToK+pj+H/fS21Ias0jU= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1690327773; x=1690932573; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ynoALsv1aca8y8xpjhuiunc/CTMUKjN2qeAE0MoxzWk=; b=H880LOZW0ZAZzo+rPare5xPE0PU77ByI582ICBiQXVIIpF8dwAAKe1ab9mWBCwY96b BDDpjgcPcCHuQKyEalW4sqBmZzut4A7wl37LBTZO0wcCS6z//87VVtndgFZY7d/KMqvc VCOYXGXS5+MnP1frDWrljpRlsrRXVOgiZp91CghKMWUd722qNgRru2ClWDtDeNvSQtd/ PLOhwcmM6b5EIs3nA1RyfAio4lybWxuOTLC5varJmzJhv4UnERstvstxeqUwl2vvJUqB oUd0jzGtDZ7Ns6F+7iST4q20WTd/myntQQWzVc8Y0r7HwGmSVM/nNJ57j/jYvPiqaCW7 J2eA== X-Gm-Message-State: ABy/qLaSEpatjsIvk59alf8duks8OXtpPWb9hKxkZuefySAi7aTt6tCQ SMP8SCtXNMFWdTQdv30AbMumN6Mzl9J6OQWbHE4= X-Google-Smtp-Source: APBJJlFOYFaGXIDd2DLLQ0HZ+PPI+wBB+NlGxhegixuv5PbL5oqsOrir68iUsUdxnVKRboGwsYrdZA== X-Received: by 2002:a5e:8a4c:0:b0:787:34d:ee9e with SMTP id o12-20020a5e8a4c000000b00787034dee9emr356394iom.7.1690327773744; Tue, 25 Jul 2023 16:29:33 -0700 (PDT) Received: from joelboxx5.c.googlers.com.com (156.190.123.34.bc.googleusercontent.com. [34.123.190.156]) by smtp.gmail.com with ESMTPSA id m18-20020a02c892000000b0041d859c5721sm3932053jao.64.2023.07.25.16.29.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 25 Jul 2023 16:29:32 -0700 (PDT) From: "Joel Fernandes (Google)" To: linux-kernel@vger.kernel.org, Andy Whitcroft , Joe Perches , Dwaipayan Ray , Lukas Bulwahn Cc: rcu@vger.kernel.org, paulmck@kernel.org, "Joel Fernandes (Google)" Subject: [PATCH 5/5] Revert "checkpatch: Error out if deprecated RCU API used" Date: Tue, 25 Jul 2023 23:29:10 +0000 Message-ID: <20230725232913.2981357-6-joel@joelfernandes.org> X-Mailer: git-send-email 2.41.0.487.g6d72f3e995-goog In-Reply-To: <20230725232913.2981357-1-joel@joelfernandes.org> References: <20230725232913.2981357-1-joel@joelfernandes.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: rcu@vger.kernel.org The check for single-argument kfree_rcu() is no longer needed as all users have been converted. This reverts commit 1eacac3255495be7502d406e2ba5444fb5c3607c. Signed-off-by: Joel Fernandes (Google) --- scripts/checkpatch.pl | 9 --------- 1 file changed, 9 deletions(-) diff --git a/scripts/checkpatch.pl b/scripts/checkpatch.pl index a9841148cde2..528f619520eb 100755 --- a/scripts/checkpatch.pl +++ b/scripts/checkpatch.pl @@ -6422,15 +6422,6 @@ sub process { } } -# check for soon-to-be-deprecated single-argument k[v]free_rcu() API - if ($line =~ /\bk[v]?free_rcu\s*\([^(]+\)/) { - if ($line =~ /\bk[v]?free_rcu\s*\([^,]+\)/) { - ERROR("DEPRECATED_API", - "Single-argument k[v]free_rcu() API is deprecated, please pass rcu_head object or call k[v]free_rcu_mightsleep()." . $herecurr); - } - } - - # check for unnecessary "Out of Memory" messages if ($line =~ /^\+.*\b$logFunctions\s*\(/ && $prevline =~ /^[ \+]\s*if\s*\(\s*(\!\s*|NULL\s*==\s*)?($Lval)(\s*==\s*NULL\s*)?\s*\)/ &&