From patchwork Thu Mar 28 07:53:09 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Lai Jiangshan X-Patchwork-Id: 13608092 Received: from mail-pl1-f172.google.com (mail-pl1-f172.google.com [209.85.214.172]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id ACDE053813; Thu, 28 Mar 2024 07:50:55 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711612257; cv=none; b=cnCXNyXRY1+as3xsofdWNOCGGOITisQOui46xMY4b4MjTWrdAiZT0LAcIqNXJdMItwfcDFqIaeUOHAvpZ9iVxs6z3KfUSt78pS+ux9u3IHnfmpRIjriQG8fERDWLt/ic2XUTyK3KyH1aHw0sHWXbn5KGutSPiXEclalTeH6Fa2g= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711612257; c=relaxed/simple; bh=VlRDIqwoVM+p8KZfRcMcR2xazrEvf0Q/aFGW4bWeON8=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=Ei9wfU/LpgiI7IRoVRJOSaeUXAoKY3kBHWDtrU0V0NecBXhTvjlYyAyKSEeV4KBSLrjDIYP6PtmoTa4lZW2sqiSTSYiyP6HfYNwLZIoJy8fmLvGOPhnNztKxo/ObnkM+gZlu00ZW7kcn4j8cwSPI8T1WdlsMEuOUwoMGMAuKAhc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=f8tOIJsw; arc=none smtp.client-ip=209.85.214.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="f8tOIJsw" Received: by mail-pl1-f172.google.com with SMTP id d9443c01a7336-1e0d82d441bso6187685ad.3; Thu, 28 Mar 2024 00:50:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1711612254; x=1712217054; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=5NHnLjZYEP5UNdxD1VIIFJ0g0ZRRIIckJCqa9h6jE6A=; b=f8tOIJsw0WGxAcSb6g2IcfW0c3LjSA+nZtEBIqSMTryxVMTyWA4OQc1az6TBHHkfv2 TUkRqT8TPTS3zj0Rupzu2/qZoxP7BIxSfJI37Uj6oUIKPQNM6Ppwl90GkInusUwt2Now NH+pGMh6JWCZSRmZ7MFJm3eX3doyf6+lzK1cV4jLx9L/7ON9+O7AtQbXTY1yhPfzodmw zMwmucZGe9wLiefK53Q75ZZlDlMe9xAxrpXMeiWy6bOGfkQsKXC/6Njl3TtJmkJAK+0t P9xer2wi0baA9JAROm9PWk7+khKfYlfGUwPIZYrJ6J7/F4Gp3qBZixhJtethcokisjvG nTSA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711612254; x=1712217054; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=5NHnLjZYEP5UNdxD1VIIFJ0g0ZRRIIckJCqa9h6jE6A=; b=wGcGHyshDGykbG0K5wGWay9PBkaG8ddzBW3VJkhwpXP7vdjPgr+2+uqEVyGUa3K6jB 4nk0kWmk6ppw4Ocj81e0rbg4Ovej/lQAZGlp5N82QtKRLrWy1gHRGfLqCJOsM8499lbN 4Non1T1nSrnH3qhexjNpnBqXGlBJjSOlTcaOLPfDhD7ZZaIi1s0oVlLr2e+cGza+RcR+ /bqPWTm19pC/TR5xCHyLQwaB/iPX46CNRArl2os/9bOwJmOMZPDdXUPfqauBf9MJVcHB lrf3dHSwXdLWy6RaTln/iMY53CQH4KLH89+BigtDoXgEz6pu+8rjj/0+A3TSlq5zATrv a2Fg== X-Gm-Message-State: AOJu0Yzr4N/XDV11hqN6mM4gtnA3hNWbNVkNKo5f/LQYGJaDx2rJbKG4 yb05OHNriCDXB151rwiIw5ffUlPKov1GMf32/S/6MzymLL3fVZh9qoLYCJiY X-Google-Smtp-Source: AGHT+IG5b9hiKqIXVMmIA/V/N+96/PpxM3h9SOhxAbcOOc4hbwQ4D182LaFIDQy1+CjspQ/K/NO4Fw== X-Received: by 2002:a17:902:ef52:b0:1dd:7485:b4c9 with SMTP id e18-20020a170902ef5200b001dd7485b4c9mr2420939plx.22.1711612254606; Thu, 28 Mar 2024 00:50:54 -0700 (PDT) Received: from localhost ([198.11.178.15]) by smtp.gmail.com with ESMTPSA id k5-20020a170902694500b001e038619e34sm851133plt.221.2024.03.28.00.50.53 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 28 Mar 2024 00:50:54 -0700 (PDT) From: Lai Jiangshan To: linux-kernel@vger.kernel.org Cc: rcu@vger.kernel.org, x86@kernel.org, Lai Jiangshan , "Paul E. McKenney" , Peter Zijlstra , Frederic Weisbecker , Ingo Molnar , Will Deacon , Waiman Long , Boqun Feng , Andrew Morton Subject: [PATCH 01/10] lib: Use rcu_preempt_depth() to replace current->rcu_read_lock_nesting Date: Thu, 28 Mar 2024 15:53:09 +0800 Message-Id: <20240328075318.83039-2-jiangshanlai@gmail.com> X-Mailer: git-send-email 2.19.1.6.gb485710b In-Reply-To: <20240328075318.83039-1-jiangshanlai@gmail.com> References: <20240328075318.83039-1-jiangshanlai@gmail.com> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Lai Jiangshan Use the general wrapper rcu_preempt_depth() instead of the open code. Prepare for enabling per-cpu rcu_preempt_count, in which case current->rcu_read_lock_nesting might not be synced with the real depth. Cc: "Paul E. McKenney" Cc: Peter Zijlstra Cc: Frederic Weisbecker Signed-off-by: Lai Jiangshan --- lib/locking-selftest.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/lib/locking-selftest.c b/lib/locking-selftest.c index 6f6a5fc85b42..9bb41fb18088 100644 --- a/lib/locking-selftest.c +++ b/lib/locking-selftest.c @@ -1434,7 +1434,7 @@ static void dotest(void (*testcase_fn)(void), int expected, int lockclass_mask) #ifdef CONFIG_SMP int saved_mgd_count = current->migration_disabled; #endif - int saved_rcu_count = current->rcu_read_lock_nesting; + int saved_rcu_count = rcu_preempt_depth(); #endif WARN_ON(irqs_disabled()); @@ -1476,9 +1476,9 @@ static void dotest(void (*testcase_fn)(void), int expected, int lockclass_mask) migrate_enable(); #endif - while (current->rcu_read_lock_nesting > saved_rcu_count) + while (rcu_preempt_depth() > saved_rcu_count) rcu_read_unlock(); - WARN_ON_ONCE(current->rcu_read_lock_nesting < saved_rcu_count); + WARN_ON_ONCE(rcu_preempt_depth() < saved_rcu_count); #endif #ifdef CONFIG_TRACE_IRQFLAGS From patchwork Thu Mar 28 07:53:10 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Lai Jiangshan X-Patchwork-Id: 13608093 Received: from mail-pf1-f174.google.com (mail-pf1-f174.google.com [209.85.210.174]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 78B9B5473F; Thu, 28 Mar 2024 07:50:59 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.174 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711612260; cv=none; b=J6kDM31lRoCgXtyPTYyedi+BgwL3c8JWmqF6deFN4ycQIsa3MVxGOqRs/YeCATMeDeGZODwtKFkjgtBHqQntCurQRohqXQTttUS/5vvl6QsaLwAdmGAErbwcKrYm0+0qxW6SUOye6yiA+Sao00Zwmb+mA6NAWZ4TJ4GImvwxGTQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711612260; c=relaxed/simple; bh=IH/NVPiz0bWht5JKxdEAshiqI2BIr+nUQ1IBd+0iPuM=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=e/T6RmPBnGz7eV3yhBGG4TzqkOOstCGBYyQDFnwPc6sbnRzPy/+YThh/HZJld7ythnOPH1YXTQG4H7clbKFLTohMlj1bdukWURzbgAJb56lc4zxlaQQ7+L9Cw6lIpcM13itannyzN8BWWSjiS0hqnh9j0cOO3vFNvlnzDocAjYE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=AK+5IoRE; arc=none smtp.client-ip=209.85.210.174 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="AK+5IoRE" Received: by mail-pf1-f174.google.com with SMTP id d2e1a72fcca58-6e6b6f86975so458685b3a.1; Thu, 28 Mar 2024 00:50:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1711612258; x=1712217058; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Ulcfe4DXVNKzIDhNYM33efHgfW9bwWfLdiuO4gC4Q3M=; b=AK+5IoREw9GLCgCheA50VW4WX8B6CPX8FRgYfS/u9/eMf8Dgu+eBKm2iCXhuJBT8/T Mjrb7RhIPFiO26Wr7qDC5YUq1fPjh8HYv7BrS1m+ggxboVtux4DFHTX5kIOzHnLQdFVk xdC7dLj2xGSLtJtxog3LBxChIpKdox52S+lhzD+Rs77UJ56O0ydnz14IS4DPa85nWJtK M4SQZmqZtSY4fxUqpNaTtsbnYx10AfFF3ow0iyv7Z/ymEl4TUX6HlJMpoCd+oq7jVfX7 YPGEhGALPNs+xjF9YQaph/MeNtDumq1EVjXS+jYCTfBXlUVAwdyBK+11UZH3AFE1nfzD DyZw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711612258; x=1712217058; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Ulcfe4DXVNKzIDhNYM33efHgfW9bwWfLdiuO4gC4Q3M=; b=NNThyMhc3ZY3mU+ybsizrLGjJXKXWUO7nZZ40609Ld2KpfujGUev9ihERrpzlToCxH 6s0WI7/F72BnrVlrHX2Kfzm3xvdSsK5/qIUI83ZjHE4+0HW1poCe3yveosCA95IcMvIO 5dGH+x2O44RBQOsMe5RuUHkadeoIDPeufOooWiDmNN+cQTsueP/p+rJqQui0bz6xqUKt erZImcg5c63VngNFDzQnNDH08hQzD40CNPI9kNOc00YAPzhq5Wmc2ZhNEpXtrssnRKUt hrz7eKKdm5lYAVreNwT99jfiOzcp3k6Ai4SHhj6uWL9/yYmEEWbldp44+pJ09bO/37fC nmPA== X-Gm-Message-State: AOJu0YycMU2P43kHkr9tZQYsDON95alSjPc/tRKE4v131lH4vcsc6HaG nIOGGSM4xqmrgYqkmx+FPhFOvD/TYrqSIO5fwCdUlyJDO4kuv+1URPdQ0QI6 X-Google-Smtp-Source: AGHT+IFQkONNKHMl2jBI6NVmx5JwO2Q7pKrDHzcKuae1qfFUmz7oHXKlI4kDJJcKcEThvn3kdgW45A== X-Received: by 2002:a05:6a20:4387:b0:1a3:df2f:ab7 with SMTP id i7-20020a056a20438700b001a3df2f0ab7mr2341741pzl.24.1711612258362; Thu, 28 Mar 2024 00:50:58 -0700 (PDT) Received: from localhost ([47.254.32.37]) by smtp.gmail.com with ESMTPSA id x8-20020aa79a48000000b006e6aee6807dsm746913pfj.22.2024.03.28.00.50.57 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 28 Mar 2024 00:50:58 -0700 (PDT) From: Lai Jiangshan To: linux-kernel@vger.kernel.org Cc: rcu@vger.kernel.org, x86@kernel.org, Lai Jiangshan , "Paul E. McKenney" , Peter Zijlstra , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Josh Triplett , Boqun Feng , Steven Rostedt , Mathieu Desnoyers , Lai Jiangshan , Zqiang Subject: [PATCH 02/10] rcu: Move rcu_preempt_depth_set() to rcupdate.h Date: Thu, 28 Mar 2024 15:53:10 +0800 Message-Id: <20240328075318.83039-3-jiangshanlai@gmail.com> X-Mailer: git-send-email 2.19.1.6.gb485710b In-Reply-To: <20240328075318.83039-1-jiangshanlai@gmail.com> References: <20240328075318.83039-1-jiangshanlai@gmail.com> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Lai Jiangshan Prepare for arch-specific-defined rcu_preempt_depth_set(). No functionality change intended, but it has to be defined as a macro as rcupdate.h is a very low level header included from areas that don't even know about the task struct "current". Cc: "Paul E. McKenney" Cc: Peter Zijlstra Cc: Frederic Weisbecker Signed-off-by: Lai Jiangshan --- include/linux/rcupdate.h | 1 + kernel/rcu/tree_plugin.h | 5 ----- 2 files changed, 1 insertion(+), 5 deletions(-) diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h index 16f519914415..210f65baf47c 100644 --- a/include/linux/rcupdate.h +++ b/include/linux/rcupdate.h @@ -78,6 +78,7 @@ void __rcu_read_unlock(void); * types of kernel builds, the rcu_read_lock() nesting depth is unknowable. */ #define rcu_preempt_depth() READ_ONCE(current->rcu_read_lock_nesting) +#define rcu_preempt_depth_set(val) WRITE_ONCE(current->rcu_read_lock_nesting, (val)) #else /* #ifdef CONFIG_PREEMPT_RCU */ diff --git a/kernel/rcu/tree_plugin.h b/kernel/rcu/tree_plugin.h index 36a8b5dbf5b5..b1264096d03a 100644 --- a/kernel/rcu/tree_plugin.h +++ b/kernel/rcu/tree_plugin.h @@ -387,11 +387,6 @@ static int rcu_preempt_read_exit(void) return ret; } -static void rcu_preempt_depth_set(int val) -{ - WRITE_ONCE(current->rcu_read_lock_nesting, val); -} - /* * Preemptible RCU implementation for rcu_read_lock(). * Just increment ->rcu_read_lock_nesting, shared state will be updated From patchwork Thu Mar 28 07:53:11 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Lai Jiangshan X-Patchwork-Id: 13608094 Received: from mail-pf1-f172.google.com (mail-pf1-f172.google.com [209.85.210.172]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C5691548F6; Thu, 28 Mar 2024 07:51:02 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711612264; cv=none; b=I0MEAsjXgJkwRd83ZQPfhSg4iEq8OqS/wW9VnInIGBvMzHQPGVgvhBhR00EM0OXMffAPWLsWS8JCJ5KPg2KaMFcUcN3Pw/Zyf/LzI5uvFYCF713oY7yoEV9ad17tO+he/C8/9YnZ3hKfbfj2Bz5jZmQPM8EkfBByDmj05h+u0gA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711612264; c=relaxed/simple; bh=uCa14RngRsTw3+urcwtpjUnYiPtvWY8aUi5khb1j6Fs=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=Y99wfsqryINyEMLMHUrph5MKQlJVrwLuBPoq3LyW43IM9kDHRfkqPU+SaLOFSdJHhWJOHifNNAtohv1wU6GWYGvJBTXnW5flSkNiHdcHOoDtiKGLL7yqYS+kcqcNxLaLkdLyHXJ5PUNWoVvDODGoyOQF71F18qo7+Dl/xywM2Is= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=jD84ydRz; arc=none smtp.client-ip=209.85.210.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="jD84ydRz" Received: by mail-pf1-f172.google.com with SMTP id d2e1a72fcca58-6e6b5432439so633741b3a.1; Thu, 28 Mar 2024 00:51:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1711612261; x=1712217061; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=PDpaTJnygn2mT7ZJwVjsDsx1RmElkSFOnj98O9h9iX0=; b=jD84ydRzcSzvNlnO5JYGJqkPL1p6ZZbzei0M64btZFyg1Ze5Mee7PO8srA7Tf+UEuE /uP1Ul+BQcmTACVEkyJ6+DjRGkjUBXSLL9cZTA57Uchornrsr5Rxj3N7Mu+2YbnjHU3J +8rLua3n6uBlWijpTfpFqXtm4Kiu/FWH/u7aFLUjQb29y3VXYxSd07ovTsoKh5k8iMY/ KShKDtkpVHci/flcJEdieckqYT2e9cUD6eg1PUXpzR1mhGQIVlbRzqnU6RbGzT5qdZBh XlhOyEDeCQBuW2/3FpvsbDSPOuDBOHZw8Sehd7Ijot5iscNJPxkzsjta5vHwwMahj5KM z2Lg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711612261; x=1712217061; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=PDpaTJnygn2mT7ZJwVjsDsx1RmElkSFOnj98O9h9iX0=; b=vykGMLs9DL7/KxKrw/bdyjdnKyJVD9jWFk71BZgz+bubULfx85zweKWB7FMNNljOXl bsO7OggL70Rz+tiTCbnncPMgk+S6M216KpiODLUAqavXDZneAuy4/gSP18VbIzaLUi89 mxTqq1SZtnZgWZeJTKXU2URP4l6PXPCN5sX2Ovp1yPiuGkTvrNTB8tsz3ckiAD6IlbcJ iAfVt4Eka6LnZY7dlR7ZaFYwg+e5GoSEGSepyeEAEGSD0wEGMdIp+mdqwy23k8YZ9aon 81EUWvMpMUFj4aPGxhn5E8gH8zS7amnVb7gYIeDdk/zyQsWEKy9WzNe9jvK/FMTP+TCW hs9g== X-Gm-Message-State: AOJu0YwmSflwyVGG6zb3xrzkAEdijshfS0zyGtD7476JkOgUmo0JfTSk DHbeqaEmFZ2UiGmOXHPwbinkAEm4yONhBYCXlO1BQduSVqAlaj/g6df1admd X-Google-Smtp-Source: AGHT+IEOWHdubp7SV6rJhw98PJPgudXwwlcJgNX1Z010cKJ0fZZCbxS25WjWE/yFh6IYdrFt6kUesA== X-Received: by 2002:a05:6a21:164e:b0:19f:f059:c190 with SMTP id no14-20020a056a21164e00b0019ff059c190mr2137232pzb.24.1711612261598; Thu, 28 Mar 2024 00:51:01 -0700 (PDT) Received: from localhost ([47.254.32.37]) by smtp.gmail.com with ESMTPSA id kx13-20020a170902f94d00b001dddeb10d83sm855667plb.223.2024.03.28.00.51.00 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 28 Mar 2024 00:51:01 -0700 (PDT) From: Lai Jiangshan To: linux-kernel@vger.kernel.org Cc: rcu@vger.kernel.org, x86@kernel.org, Lai Jiangshan , "Paul E. McKenney" , Peter Zijlstra , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Josh Triplett , Boqun Feng , Steven Rostedt , Mathieu Desnoyers , Lai Jiangshan , Zqiang Subject: [PATCH 03/10] rcu: Reorder tree_exp.h after tree_plugin.h Date: Thu, 28 Mar 2024 15:53:11 +0800 Message-Id: <20240328075318.83039-4-jiangshanlai@gmail.com> X-Mailer: git-send-email 2.19.1.6.gb485710b In-Reply-To: <20240328075318.83039-1-jiangshanlai@gmail.com> References: <20240328075318.83039-1-jiangshanlai@gmail.com> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Lai Jiangshan Enable tree_exp.h using some rcu_preempt macros introduced in the next patch. The new macros touch core rcu_preempt fields and are better to be implemented in tree_plugin.h. Cc: "Paul E. McKenney" Cc: Peter Zijlstra Cc: Frederic Weisbecker Signed-off-by: Lai Jiangshan --- kernel/rcu/tree.c | 2 +- kernel/rcu/tree_plugin.h | 1 + 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index d9642dd06c25..57d1ae26861f 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -5245,6 +5245,6 @@ void __init rcu_init(void) } #include "tree_stall.h" -#include "tree_exp.h" #include "tree_nocb.h" #include "tree_plugin.h" +#include "tree_exp.h" diff --git a/kernel/rcu/tree_plugin.h b/kernel/rcu/tree_plugin.h index b1264096d03a..d899b4afc21c 100644 --- a/kernel/rcu/tree_plugin.h +++ b/kernel/rcu/tree_plugin.h @@ -103,6 +103,7 @@ static void __init rcu_bootup_announce_oddness(void) #ifdef CONFIG_PREEMPT_RCU static void rcu_report_exp_rnp(struct rcu_node *rnp, bool wake); +static bool sync_rcu_exp_done(struct rcu_node *rnp); static void rcu_read_unlock_special(struct task_struct *t); /* From patchwork Thu Mar 28 07:53:12 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Lai Jiangshan X-Patchwork-Id: 13608095 Received: from mail-pg1-f176.google.com (mail-pg1-f176.google.com [209.85.215.176]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id EF42654FA0; Thu, 28 Mar 2024 07:51:05 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.176 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711612267; cv=none; b=msizqNwohbIDXGR4HS3q0VjkcKi70BVlzs2ctyRbYA6fibD1+ilMVF3TXRjdNY2JvmSxmi909rXlCkUmUu5unPp4IWLEfIrow0ExtlHbIVTaffAejElZCNKa8dF7UnGJ5zWFq5oL6CjSHpcTBR9/9Jv2/kJtKt6QLq/CkqHJzng= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711612267; c=relaxed/simple; bh=dyDjW0emqk3ra0mcQ2J2Pyvf/leGXIhR8J75jqjVAuw=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=rQvD8dQP+xgZPIjtiVWIaiTKEsOsHXCTQld3csrc/ki9mgzRi7oAUvD1/zakZXtBYG2YzIXKU6BtGKcylEuU90Ntuk+eixU1xDxoS7vg9gqj8O1UxZbPEhOIrqAEEvEpQkprB2BBbIXeUEmEDE8qVHKwJLvPOLMlhI/4Ih9YGIw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=YLL0x/ZB; arc=none smtp.client-ip=209.85.215.176 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="YLL0x/ZB" Received: by mail-pg1-f176.google.com with SMTP id 41be03b00d2f7-5e4613f2b56so424740a12.1; Thu, 28 Mar 2024 00:51:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1711612265; x=1712217065; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ZvoBd6HF4JbE2BHoe97A/COYGhUtV93EmRmWHbLWiDc=; b=YLL0x/ZBgaAhW7Mkj57I/jN4jzQ1Sc6kOBTQg5jbfjJa5PNIfE+witr5XzMtVGFGOp RfqKhluy2qAz7ZMkgISNOG4Kif1SFVbQdcnHgW8o2RnNRrrrwa3T1GRlrzedShTNydjA VTGaiq+gKhqmugz4V1N4U3g8AJz8Yx7uuJkepzMC9pE9ckC3HX9f1167p8lsO9C3EbMx Nh91CR/bGSvB15p/4pCYRaRJ5RMwH5jIiccAwPIPOtvv2efbFNPKTVS0PfrpKwfMRLNh nsY7Vx0ezW+keupUiuwXUPGIwWh06RVb8IoEXLJjyQ8n5pc3gdClmLtGJx7T9WOUqdug TbCg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711612265; x=1712217065; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ZvoBd6HF4JbE2BHoe97A/COYGhUtV93EmRmWHbLWiDc=; b=bQQcPVFohPiY9AAQerDTPYzr/Xd7sezFIOc0Oh89XhDU52qvacXHhwQZ5IZUz3RNjC trCYUgGfrAjtEbZL5KebYXle6ZehDFhP06Ig4fNxgvVcf5+/lpJPaN7L9dnBTRanqD1b rUxVVSReYXPvkV1aoBWnVcgzQg614Vs1jCw1i7vWM1+MMoOv2VSgQYrFtE0F83J/X3gS ZGz3YUivxX9uywLLn3KvdCXRU6IxyMgOq5Dj2hrnfleZvvcOg0xFCro/7lQRSLTbWYzi 2eHpQyMEFaCEyExOuVdM2ZSRQqVt8qzQtrjtrVoEiwhsaUajVFxG5tPdewTIHL31BdO6 LFqg== X-Gm-Message-State: AOJu0YwWdI8VQm1sYrBQc0idGB9yPAfpWJ4aBSjOCHrJ+AU4quxmFo7y bNyTrz/u3sayG0CktcozZvBicH71euvjtkdNvbczMP5hiO61oE70S2XEyBCm X-Google-Smtp-Source: AGHT+IH7DhtUYNn/v811iMBSHET6XQNvp0TWBqztbY6OYCDBxmyvhgvpjskjy0i0ovFiIRQ7h/j9sA== X-Received: by 2002:a05:6a20:daa9:b0:1a3:af03:6b0 with SMTP id iy41-20020a056a20daa900b001a3af0306b0mr2913532pzb.7.1711612264778; Thu, 28 Mar 2024 00:51:04 -0700 (PDT) Received: from localhost ([47.254.32.37]) by smtp.gmail.com with ESMTPSA id l4-20020a170903244400b001e0b5d4a2a8sm857817pls.149.2024.03.28.00.51.04 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 28 Mar 2024 00:51:04 -0700 (PDT) From: Lai Jiangshan To: linux-kernel@vger.kernel.org Cc: rcu@vger.kernel.org, x86@kernel.org, Lai Jiangshan , "Paul E. McKenney" , Peter Zijlstra , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Josh Triplett , Boqun Feng , Steven Rostedt , Mathieu Desnoyers , Lai Jiangshan , Zqiang Subject: [PATCH 04/10] rcu: Add macros set_rcu_preempt_special() and clear_rcu_preempt_special() Date: Thu, 28 Mar 2024 15:53:12 +0800 Message-Id: <20240328075318.83039-5-jiangshanlai@gmail.com> X-Mailer: git-send-email 2.19.1.6.gb485710b In-Reply-To: <20240328075318.83039-1-jiangshanlai@gmail.com> References: <20240328075318.83039-1-jiangshanlai@gmail.com> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Lai Jiangshan Add wrappers accessing to t->rcu_read_unlock_special.b so that the wrappers can be adapted for synching the value with the later-introduced per-cpu rcu_preempt_count. Cc: "Paul E. McKenney" Cc: Peter Zijlstra Cc: Frederic Weisbecker Signed-off-by: Lai Jiangshan --- kernel/rcu/tree_exp.h | 2 +- kernel/rcu/tree_plugin.h | 20 +++++++++++++------- 2 files changed, 14 insertions(+), 8 deletions(-) diff --git a/kernel/rcu/tree_exp.h b/kernel/rcu/tree_exp.h index 6b83537480b1..4e91414552e6 100644 --- a/kernel/rcu/tree_exp.h +++ b/kernel/rcu/tree_exp.h @@ -725,7 +725,7 @@ static void rcu_exp_handler(void *unused) raw_spin_lock_irqsave_rcu_node(rnp, flags); if (rnp->expmask & rdp->grpmask) { WRITE_ONCE(rdp->cpu_no_qs.b.exp, true); - t->rcu_read_unlock_special.b.exp_hint = true; + set_rcu_preempt_special(exp_hint); } raw_spin_unlock_irqrestore_rcu_node(rnp, flags); return; diff --git a/kernel/rcu/tree_plugin.h b/kernel/rcu/tree_plugin.h index d899b4afc21c..3dbd5609185e 100644 --- a/kernel/rcu/tree_plugin.h +++ b/kernel/rcu/tree_plugin.h @@ -106,6 +106,14 @@ static void rcu_report_exp_rnp(struct rcu_node *rnp, bool wake); static bool sync_rcu_exp_done(struct rcu_node *rnp); static void rcu_read_unlock_special(struct task_struct *t); +#define set_rcu_preempt_special(reason) do { \ + WRITE_ONCE(current->rcu_read_unlock_special.b.reason, true); \ + } while (0) + +#define clear_rcu_preempt_special(reason) do { \ + WRITE_ONCE(current->rcu_read_unlock_special.b.reason, false); \ + } while (0) + /* * Tell them what RCU they are running. */ @@ -293,7 +301,7 @@ static void rcu_qs(void) TPS("cpuqs")); __this_cpu_write(rcu_data.cpu_no_qs.b.norm, false); barrier(); /* Coordinate with rcu_flavor_sched_clock_irq(). */ - WRITE_ONCE(current->rcu_read_unlock_special.b.need_qs, false); + clear_rcu_preempt_special(need_qs); } } @@ -325,7 +333,7 @@ void rcu_note_context_switch(bool preempt) /* Possibly blocking in an RCU read-side critical section. */ rnp = rdp->mynode; raw_spin_lock_rcu_node(rnp); - t->rcu_read_unlock_special.b.blocked = true; + set_rcu_preempt_special(blocked); t->rcu_blocked_node = rnp; /* @@ -399,7 +407,7 @@ void __rcu_read_lock(void) if (IS_ENABLED(CONFIG_PROVE_LOCKING)) WARN_ON_ONCE(rcu_preempt_depth() > RCU_NEST_PMAX); if (IS_ENABLED(CONFIG_RCU_STRICT_GRACE_PERIOD) && rcu_state.gp_kthread) - WRITE_ONCE(current->rcu_read_unlock_special.b.need_qs, true); + set_rcu_preempt_special(need_qs); barrier(); /* critical section after entry code. */ } EXPORT_SYMBOL_GPL(__rcu_read_lock); @@ -738,7 +746,7 @@ static void rcu_flavor_sched_clock_irq(int user) __this_cpu_read(rcu_data.cpu_no_qs.b.norm) && !t->rcu_read_unlock_special.b.need_qs && time_after(jiffies, rcu_state.gp_start + HZ)) - t->rcu_read_unlock_special.b.need_qs = true; + set_rcu_preempt_special(need_qs); } /* @@ -751,12 +759,10 @@ static void rcu_flavor_sched_clock_irq(int user) */ void exit_rcu(void) { - struct task_struct *t = current; - if (unlikely(!list_empty(¤t->rcu_node_entry))) { rcu_preempt_depth_set(1); barrier(); - WRITE_ONCE(t->rcu_read_unlock_special.b.blocked, true); + set_rcu_preempt_special(blocked); } else if (unlikely(rcu_preempt_depth())) { rcu_preempt_depth_set(1); } else { From patchwork Thu Mar 28 07:53:13 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Lai Jiangshan X-Patchwork-Id: 13608096 Received: from mail-oo1-f43.google.com (mail-oo1-f43.google.com [209.85.161.43]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CB4BC53E38; Thu, 28 Mar 2024 07:51:09 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.161.43 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711612271; cv=none; b=Q4O9HlggntnapK41ujF4Ry1Or6Yn30w/8oEAJjih5jlXITbv2f388qtjXEPJURaTtzCxtldYrgm2DhRArByBqzq2g/1R4I/iptoPf6nWKbR4JUeY7z4FbdkNnEeoEggepBXA2MEIul5pZLYbmvKyLPtHa5tisX+JnAr5kS5R6t4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711612271; c=relaxed/simple; bh=enZx1n4OK01lUXHrdd+7erDRpBSJsnAVbuYv7QzWELA=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=BJj6gVIIKiE/oUeiSUIJrbGMrGK22dqpplaBQF02wYRMIzMxYA4Yl1bV6DIljhH0bTH7/Fb3MBlVbzThZbK1nDqgMtFdDDDjDMpZp56NOHMPyca2N0vdSbMxrm+ESWt0WtJCsjK34eATxumwzkK2Zyz4loCzRNKixy+/hNsjUE8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=dJ6Z0Qld; arc=none smtp.client-ip=209.85.161.43 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="dJ6Z0Qld" Received: by mail-oo1-f43.google.com with SMTP id 006d021491bc7-5a4f7a648dbso347027eaf.3; Thu, 28 Mar 2024 00:51:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1711612268; x=1712217068; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=T9VU/pyKywB+unjTxJCyReyW8qdQmEQO6VzP1yTJUvM=; b=dJ6Z0QldmwkElLfKJSFgNP+9sTSj8I2hmUoOJQmArT4mZkcSr/1E8PqXVJIh2IZfpO S1Z/7FdDqzWgrVrVabIwyQuBRQhigCPyF/HsAhn8Pqvd9s63THBQlMrw61vebJN03EUM MUte3qQN+Rwn2HpIU1PuR31m75mZmpsZP9iGosxFLkX2h8dyii/PXoFZMP8pjPWW7nms zkiHfvyHC84h0XrHPtWvpe/Qbkgj/spfY95pxierhLso00CJ/wdMANPEPmB/Jsymxb5M TXxibNA84ycYeM3jDFjtNoTSG2MstASFxtUewYY4XzdgxKoa2tdVIwgjqLtgdDZ6P33Q In4g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711612268; x=1712217068; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=T9VU/pyKywB+unjTxJCyReyW8qdQmEQO6VzP1yTJUvM=; b=qFqz7X6ilcVupbilY5s70cGgs39INKIvSMnlluVGlFX4zdp7kycjMI4IfGzRIc2qWV vokr/ODZ/tdfp1GQOV9uJVgrnnTxlaffydWM4dtxHZmDzyz0LCYgTCvdoYJdnxzgrg9/ EKHhiF0iHYCaGXuuTXtNQeMNU/ot553Gczh3lyvrGwgevgQP5a+7fYU3yfcnFHCiApRL eMIgDXAQc46P49TZnyGkDHxsTKqLnxBW3tnAABrMuujZiBeEsgEig7q3gyf8rSV+4A/B vCuK52FdzwuwFij4qC8Yg7WNx8WvKsLI2kUIp2iaVKolznkIs/ExGTYxqCx8EzTcDvE+ 7LEg== X-Gm-Message-State: AOJu0YyoykFO6FItchm6Oesh9CPvtkT2Z6hRoa1YKND+XJ/R+nn44g3b OSRcu51ffCFDG5qPiFN6pB5fO9NC56vBpJMantxvbay22icxaFkngkbQtB+e X-Google-Smtp-Source: AGHT+IF4SFu3dyS1jq9c9JUIbB6hOPx5Georf4Ru081wj7PhhUuL+ItBLD0KHsC5/M3U3iMREBbs6A== X-Received: by 2002:a05:6870:c34c:b0:22a:a40a:a09f with SMTP id e12-20020a056870c34c00b0022aa40aa09fmr1115197oak.54.1711612268597; Thu, 28 Mar 2024 00:51:08 -0700 (PDT) Received: from localhost ([198.11.178.15]) by smtp.gmail.com with ESMTPSA id gx5-20020a056a001e0500b006e729dd12d5sm749531pfb.48.2024.03.28.00.51.07 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 28 Mar 2024 00:51:08 -0700 (PDT) From: Lai Jiangshan To: linux-kernel@vger.kernel.org Cc: rcu@vger.kernel.org, x86@kernel.org, Lai Jiangshan , "Paul E. McKenney" , Peter Zijlstra , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Josh Triplett , Boqun Feng , Steven Rostedt , Mathieu Desnoyers , Lai Jiangshan , Zqiang Subject: [PATCH 05/10] rcu: Make rcu_read_unlock_special() global Date: Thu, 28 Mar 2024 15:53:13 +0800 Message-Id: <20240328075318.83039-6-jiangshanlai@gmail.com> X-Mailer: git-send-email 2.19.1.6.gb485710b In-Reply-To: <20240328075318.83039-1-jiangshanlai@gmail.com> References: <20240328075318.83039-1-jiangshanlai@gmail.com> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Lai Jiangshan Make it global so that it can be used in the future inlined rcu_read_unlock(). Make it exported so that the inlined rcu_read_unlock() can be used in modules. Make it taking a void-argument so that the caller can reduce instructions obtaining the "current" task value and it can be used in THUNK later. Cc: "Paul E. McKenney" Cc: Peter Zijlstra Cc: Frederic Weisbecker Signed-off-by: Lai Jiangshan --- include/linux/rcupdate.h | 2 ++ kernel/rcu/tree_plugin.h | 7 ++++--- 2 files changed, 6 insertions(+), 3 deletions(-) diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h index 210f65baf47c..cc77d76a870b 100644 --- a/include/linux/rcupdate.h +++ b/include/linux/rcupdate.h @@ -68,6 +68,8 @@ static inline bool same_state_synchronize_rcu(unsigned long oldstate1, unsigned #ifdef CONFIG_PREEMPT_RCU +void rcu_read_unlock_special(void); + void __rcu_read_lock(void); void __rcu_read_unlock(void); diff --git a/kernel/rcu/tree_plugin.h b/kernel/rcu/tree_plugin.h index 3dbd5609185e..880b3fef1158 100644 --- a/kernel/rcu/tree_plugin.h +++ b/kernel/rcu/tree_plugin.h @@ -104,7 +104,6 @@ static void __init rcu_bootup_announce_oddness(void) static void rcu_report_exp_rnp(struct rcu_node *rnp, bool wake); static bool sync_rcu_exp_done(struct rcu_node *rnp); -static void rcu_read_unlock_special(struct task_struct *t); #define set_rcu_preempt_special(reason) do { \ WRITE_ONCE(current->rcu_read_unlock_special.b.reason, true); \ @@ -427,7 +426,7 @@ void __rcu_read_unlock(void) if (rcu_preempt_read_exit() == 0) { barrier(); // critical-section exit before .s check. if (unlikely(READ_ONCE(t->rcu_read_unlock_special.s))) - rcu_read_unlock_special(t); + rcu_read_unlock_special(); } if (IS_ENABLED(CONFIG_PROVE_LOCKING)) { int rrln = rcu_preempt_depth(); @@ -627,8 +626,9 @@ static void rcu_preempt_deferred_qs_handler(struct irq_work *iwp) * notify RCU core processing or task having blocked during the RCU * read-side critical section. */ -static void rcu_read_unlock_special(struct task_struct *t) +void rcu_read_unlock_special(void) { + struct task_struct *t = current; unsigned long flags; bool irqs_were_disabled; bool preempt_bh_were_disabled = @@ -684,6 +684,7 @@ static void rcu_read_unlock_special(struct task_struct *t) } rcu_preempt_deferred_qs_irqrestore(t, flags); } +EXPORT_SYMBOL_GPL(rcu_read_unlock_special); /* * Check that the list of blocked tasks for the newly completed grace From patchwork Thu Mar 28 07:53:14 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Lai Jiangshan X-Patchwork-Id: 13608097 Received: from mail-pf1-f178.google.com (mail-pf1-f178.google.com [209.85.210.178]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0DC7458209; Thu, 28 Mar 2024 07:51:12 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.178 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711612274; cv=none; b=s+yTDAumW22GfRXl2pSlAnM/sqBGp6tnJ1kIRBpK63pxtIoKhwX04ntCrqdTSXjLGTwv4MFvmlxYycZtlr0BQAOspWhEPVqQQj2d74iflZnAI6cuRC0oX/B6Ga8Ydtzij42ixGE75CJlZ2iqU1blr6FLt5S4U2HwuOtPIa1DVH0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711612274; c=relaxed/simple; bh=3TturtFYIwfV1u8ACm2gm3tHdeACQEBcKlzkWT/2VSg=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=DvsXjEKC5uAyp9gLZGU+IxMEN5kRoYrYG/f/MEWofKDYOFQayGyVR6ZjW5WiWokcJMkSVY8M5CntZ5CE+4hNyU7KC0M4/kMjZs2vxDk6H8CO/zvdSbdaK1PZF80U6uYHIPXUXDHd15v+3BuZe2lmLtFcxkysOhoyLaC8WADRVi0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=Qrne00b+; arc=none smtp.client-ip=209.85.210.178 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Qrne00b+" Received: by mail-pf1-f178.google.com with SMTP id d2e1a72fcca58-6e74bd85f26so607163b3a.1; Thu, 28 Mar 2024 00:51:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1711612272; x=1712217072; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=U7zjP/fD7ntbdBgOTsfd20ge3XzeOlmyb2f9DwxHaY0=; b=Qrne00b+JFmLsVv8yqzD+LXn8kxvQxd/rR2Rw8x0Qp3aga06+j9f1w6sA0K3UM2be4 B8FCz0UndJylAVTvr0QMcWx8OVvJhqQTgbFhwDV7imQdzUv6omQPx/ShdeRc9PDpfVcE Jy/rQFWS5ULSo+wV3HoFeauCVUJH03HVseDWznp1YRQ3kvh9zi75EmHZCvNcJF25+L+1 UwNwIW1KfFv4yqNO3CD9oL2YW0VaTKevyeICi/KEltNmWCe2ZKok3v25oY3JFkmp+/qG 0PbOMPlsCkT0Q6mQBrY1+rNttAqXXMnzORK5HGIWavR1qnoo33zvg3e1ShW/vkmGL1cb v5QA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711612272; x=1712217072; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=U7zjP/fD7ntbdBgOTsfd20ge3XzeOlmyb2f9DwxHaY0=; b=PWt/GIxsqAIzsmSPkQhEZej0xseVpp14I65/LcuOpSAGhu0r8AIJ0bnHqyTmWqh3XU xsQN0/R4YpV0Tpj0i4X3Sn0ypXpGbxSV4/MLxovQL3/DSEY/NWDCPquTmm1TrSIF9Pu2 8gRXfLcQSVEEv9Haq5jQzjrfijsb7YzLSkwuOD41tC5e8DbrZvZnLOHATpWxPPa016op Ch3vjlAH6TRWXPK8QYQIIJqtMGkyU26JcqOYNWeEcmuZKyvK6fRofPjZuMMq7aQw84c6 7DWoBswoRawPCvuPSr5lG28q/kzkA30pcwQ64OvTdP/gR4QNE4ciNrkmiHygzojE9jZP E9rw== X-Gm-Message-State: AOJu0YxuTiVhieumi0TpdDCXLF3WjG1VUK/7Bqz9VaZ+yqjzUrjq7iYc Hs6uQYNalXwiNI3chWglCAnjU8i7GfmZTGeU7W13zlpxuN/zZbrL0acinE5k X-Google-Smtp-Source: AGHT+IFSOSZLlO1NnrKQSPtda1+ZjQ15dhdKV5SqZhDSS9xqF0bDwi+WCn2fnHOjvcOnYPiQaKY6gg== X-Received: by 2002:a05:6a20:2583:b0:1a5:6a72:af54 with SMTP id k3-20020a056a20258300b001a56a72af54mr1867908pzd.45.1711612271919; Thu, 28 Mar 2024 00:51:11 -0700 (PDT) Received: from localhost ([47.254.32.37]) by smtp.gmail.com with ESMTPSA id h13-20020a170902680d00b001e00d9680cesm864684plk.130.2024.03.28.00.51.11 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 28 Mar 2024 00:51:11 -0700 (PDT) From: Lai Jiangshan To: linux-kernel@vger.kernel.org Cc: rcu@vger.kernel.org, x86@kernel.org, Lai Jiangshan , "Paul E. McKenney" , Peter Zijlstra , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Josh Triplett , Boqun Feng , Steven Rostedt , Mathieu Desnoyers , Lai Jiangshan , Zqiang Subject: [PATCH 06/10] rcu: Rename marco __LINUX_RCU_H to __KERNEL_RCU_H Date: Thu, 28 Mar 2024 15:53:14 +0800 Message-Id: <20240328075318.83039-7-jiangshanlai@gmail.com> X-Mailer: git-send-email 2.19.1.6.gb485710b In-Reply-To: <20240328075318.83039-1-jiangshanlai@gmail.com> References: <20240328075318.83039-1-jiangshanlai@gmail.com> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Lai Jiangshan to reflect the path of kernel/rcu/rcu.h. Cc: "Paul E. McKenney" Cc: Peter Zijlstra Cc: Frederic Weisbecker Signed-off-by: Lai Jiangshan --- kernel/rcu/rcu.h | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/kernel/rcu/rcu.h b/kernel/rcu/rcu.h index 86fce206560e..b17b2ed657fc 100644 --- a/kernel/rcu/rcu.h +++ b/kernel/rcu/rcu.h @@ -7,8 +7,8 @@ * Author: Paul E. McKenney */ -#ifndef __LINUX_RCU_H -#define __LINUX_RCU_H +#ifndef __KERNEL_RCU_H +#define __KERNEL_RCU_H #include #include @@ -668,4 +668,4 @@ int rcu_stall_notifier_call_chain(unsigned long val, void *v); static inline int rcu_stall_notifier_call_chain(unsigned long val, void *v) { return NOTIFY_DONE; } #endif // #else // #if defined(CONFIG_RCU_STALL_COMMON) && defined(CONFIG_RCU_CPU_STALL_NOTIFIER) -#endif /* __LINUX_RCU_H */ +#endif /* __KERNEL_RCU_H */ From patchwork Thu Mar 28 07:53:15 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Lai Jiangshan X-Patchwork-Id: 13608098 Received: from mail-pf1-f178.google.com (mail-pf1-f178.google.com [209.85.210.178]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C41D15B1E3; Thu, 28 Mar 2024 07:51:16 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.178 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711612278; cv=none; b=sRETaakQfKJa8DOUHEnU1gu2jed4bzVCEWhwb6VcwY0XeGCgjdcUBQ/IlR8xwBERxftYEBDUVrGsw0FDMpbee1ReXAdNhoR2lLjXW3KDoRXS6DNr7UJqN2REYwfAqbtpQY3ESdnFEmJyopd48gG6pvmL97xuRGORWQIvtPVdmdY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711612278; c=relaxed/simple; bh=N3g/1gJrg9TfDFHDhZDcoAnyjd3OaQOWBx3iO3g+BTI=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=NUBRrT/N/EVtpxVrQgWH3RR/ogUz7wY0EEng1rwHYJVoz0rll5gzjpMasMoKeN10GnqXxzxK/eKtoZHrezgn9FHb4oNC1ugkWvez6RW4RVXff5Tz8Zfn0DCwNVnw72hc8VeLtaoeF5PCQnenleM06rhmfh7hUp9q/AWHKfG25qA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=VefAOzEl; arc=none smtp.client-ip=209.85.210.178 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="VefAOzEl" Received: by mail-pf1-f178.google.com with SMTP id d2e1a72fcca58-6eac64f2205so598209b3a.2; Thu, 28 Mar 2024 00:51:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1711612276; x=1712217076; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=kZNpgupDg2bfZqoYSY4w9xqU9rN1cJ0j5BbUJd8co30=; b=VefAOzElDp++U70u98m1ge/rw+6oF+FFgwC/63cnPNFC/f3dlCHMI5IR3wVsOJNGzS j8zrfeJHWdVtLAwS9tRxsCd6Mb8CVnH2vJTNg67F6q89OatooniDd2onJJ95/Gh/OapN mgw0j1xuPqNwqnEICD/dwc/DNUoiJVVYPsAWMqJQfCd27vHxngKeMPCLtKOSB/UUAcLG 3Wn0JMnKatgLOCrz4DPzjM7rLw75lEM8Zg8rNSvpCW0EJ39UbQATNp7OFCgzvrHKcKMX xgmILeEGhcyf3bwoOfsU/zFeEQqgaynjloof6DPj5dB/JQsXwd3zX9vbYc7uEJr/tOnW px3Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711612276; x=1712217076; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=kZNpgupDg2bfZqoYSY4w9xqU9rN1cJ0j5BbUJd8co30=; b=G/ke1aEPOu/YkNZjSjTOt1jrQkhIrN5oKBT6D6XqIzfg50Y//XmBTJj2UOGh77zGlm n1WTDoJrpg7y6gzxPsNaNCTEpuq1k5q76XtMp8L8M3RFlPCJlPjANyhWQ08yft/c5rd7 5YHwW1i0jU9OkiXkimjf3lYfsr5tKAkf8yNcUD5G/zptJHAvk1txAnf+Z56ug8z7oG9a SUO5xjgWv8RefpnVcpXIXOfxcENf4TPNA2aE969xPtIplHrB+QOvGJxC/VqeaAozHDfa wCYyrLDHpC51gsRERDTdqQ/IcDsacbqUQn/Li1M0he8kCcsqa6/k/Tlesw/f24WDfh4N /hlw== X-Gm-Message-State: AOJu0YwUQY9AsFevOtdgDM2PFA+kax3p7KxqHaFHX/DB1EO4NHX2ndom HvE+mEilaivn1INzEmCwz+rKAA1yLfIVA/tQFnK0s8W30hAseS1GgVCtYXSO X-Google-Smtp-Source: AGHT+IEIaoWEFLOj5YGQX0oavQNxsIj6yrF25pRpP3Mp20xsQmW90Qs/s+goFqivAkIitJ0zpE6Mdw== X-Received: by 2002:a05:6a00:1d08:b0:6ea:df40:8a18 with SMTP id a8-20020a056a001d0800b006eadf408a18mr93965pfx.16.1711612275673; Thu, 28 Mar 2024 00:51:15 -0700 (PDT) Received: from localhost ([198.11.176.14]) by smtp.gmail.com with ESMTPSA id g5-20020aa79f05000000b006eac4d7b2e1sm746500pfr.113.2024.03.28.00.51.15 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 28 Mar 2024 00:51:15 -0700 (PDT) From: Lai Jiangshan To: linux-kernel@vger.kernel.org Cc: rcu@vger.kernel.org, x86@kernel.org, Lai Jiangshan , "Paul E. McKenney" , Peter Zijlstra , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Josh Triplett , Boqun Feng , Steven Rostedt , Mathieu Desnoyers , Lai Jiangshan , Zqiang , Ingo Molnar , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Ben Segall , Mel Gorman , Daniel Bristot de Oliveira , Valentin Schneider Subject: [PATCH 07/10] sched/core: Add rcu_preempt_switch() Date: Thu, 28 Mar 2024 15:53:15 +0800 Message-Id: <20240328075318.83039-8-jiangshanlai@gmail.com> X-Mailer: git-send-email 2.19.1.6.gb485710b In-Reply-To: <20240328075318.83039-1-jiangshanlai@gmail.com> References: <20240328075318.83039-1-jiangshanlai@gmail.com> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Lai Jiangshan When the per-cpu rcu_preempt_count is used, it has to be switched too on context-switching. And the instructions to switch the per-cpu rcu_preempt_count are meager, so it is inlined to avoid the overhead in the scheduler. Cc: "Paul E. McKenney" Cc: Peter Zijlstra Cc: Frederic Weisbecker Signed-off-by: Lai Jiangshan --- kernel/rcu/rcu.h | 5 +++++ kernel/sched/core.c | 2 ++ 2 files changed, 7 insertions(+) diff --git a/kernel/rcu/rcu.h b/kernel/rcu/rcu.h index b17b2ed657fc..ea5ae957c687 100644 --- a/kernel/rcu/rcu.h +++ b/kernel/rcu/rcu.h @@ -668,4 +668,9 @@ int rcu_stall_notifier_call_chain(unsigned long val, void *v); static inline int rcu_stall_notifier_call_chain(unsigned long val, void *v) { return NOTIFY_DONE; } #endif // #else // #if defined(CONFIG_RCU_STALL_COMMON) && defined(CONFIG_RCU_CPU_STALL_NOTIFIER) +static inline void +rcu_preempt_switch(struct task_struct *prev, struct task_struct *next) +{ +} + #endif /* __KERNEL_RCU_H */ diff --git a/kernel/sched/core.c b/kernel/sched/core.c index d44efa0d0611..7c3fedd9180c 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -95,6 +95,7 @@ #include "../workqueue_internal.h" #include "../../io_uring/io-wq.h" #include "../smpboot.h" +#include "../rcu/rcu.h" EXPORT_TRACEPOINT_SYMBOL_GPL(ipi_send_cpu); EXPORT_TRACEPOINT_SYMBOL_GPL(ipi_send_cpumask); @@ -6727,6 +6728,7 @@ static void __sched notrace __schedule(unsigned int sched_mode) */ ++*switch_count; + rcu_preempt_switch(prev, next); migrate_disable_switch(rq, prev); psi_sched_switch(prev, next, !task_on_rq_queued(prev)); From patchwork Thu Mar 28 07:53:16 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Lai Jiangshan X-Patchwork-Id: 13608099 Received: from mail-pf1-f173.google.com (mail-pf1-f173.google.com [209.85.210.173]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3F7305FB89; Thu, 28 Mar 2024 07:51:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.173 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711612281; cv=none; b=LqSxlIgXdPBVJ2hyWzTwBXHiO9hMZqOf1m0WHFfWYzx/csAPNqrajjPVMYq/MrWpPR+MByvuhcEWke/hhQw7BIOhOWhUOxF99YUh3ZtU2gGIi2VKTATlV18wfNPh+K/QtIe8mFEPY7Qr+LgfYjPKmAe3MF2Rz14vB1uKEzFepo8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711612281; c=relaxed/simple; bh=e2O4iV9sJsNP5W8dr/Q+G/8c4Lr9bkbWeoqiWsoecC4=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=PhBXAgxynvfC4YR96B4Kk4qLg5QlYmBtbEuXdAFY9X9F2j46KZo7KtYn0tepT5m3pWHqSlGMxVkDx1r5APl7533//b3t4zQ9JEQ1fbzFxz8WcMH4Sc7muVJFvphOxyklu2fXRxGKIjbkKTCOhRTIPnrvoSIVP9WQfA0+b7WcBnU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=c+vPaUDi; arc=none smtp.client-ip=209.85.210.173 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="c+vPaUDi" Received: by mail-pf1-f173.google.com with SMTP id d2e1a72fcca58-6e6b5432439so633865b3a.1; Thu, 28 Mar 2024 00:51:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1711612279; x=1712217079; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=rGirO6AjDMqe2sVAIHTJAIXsudaabgHkxaTMsIO4NV8=; b=c+vPaUDi+VTlvTWYMvruQJn+2q4IYplPBkTpJq4S+p0whW2vWO+kQ5r4lFhdfZ5Xf2 59C1iuXy5ovTJ3O/HpnSNkAnrJhaLF1vmUaui4D5GbfJncoC9TggNPz7WaekzgWEu9U5 OGnaLOLUri+/LFEzMkqNudTQ9szc53dMdwtgFsSGGP25iX+F2PBtJize5rKGw0cn0I3Z NmdmYAWyiZCVgr6/+cWAXzWAEGERgz/mREZvddfeAXb9lKDyjihgaH6SD0lFaa/XqaoO KQ9pyrj0yg+aGkwSNl1lIc1ZXFhP7Gx813RyuAp7ctw6ai95LGi47ybTN+KBBSfSSrBZ lGYA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711612279; x=1712217079; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=rGirO6AjDMqe2sVAIHTJAIXsudaabgHkxaTMsIO4NV8=; b=NDnDEwuy1has6ql6qjjmFT+uglQQWVI32oCQrZzH/2eh3xXGlEhnICtiMPZbJNUpyh 5x6TSxThEBjvwFDGUJBHd1JDvn2IMd4+o4cbqRYBc2+6a55Gwfirsj5EhZ6aQVLpR+Zw x7wLGUmAm8TUbfSadhlwURut/YW/6E6E2QYhvvZP2SWL5k2fvWgkZkUmChqFDL02PBQt wZQeUNK7XUxQOvyjWgLv+go3WYh5IqA/AY/vUunHftwOaxjr6mEb5xLtZ2fgJd3/9cWT W/K+b5QC4HLIvyEkYS16DkwA1xDkpiFzGvghotxfBuQqK7LYeX2C0SA/XXUcvCXE7mzR 95ow== X-Gm-Message-State: AOJu0YwOSuGdFFG4HyqtN/O2vlm1zt/TED8EGlDgWPs8aOZgi7t2PuD9 z4oxo6E9oL79aG95+RC4u1ITz2x1QifmA3Z5aLFi43EEkb20+LhsWtT5CqoW X-Google-Smtp-Source: AGHT+IG1e0Su/WUp6fMVEFwpsiDpgQVWG/zovyoLs1TQAfdSlcIQWafNpLlbpYUzDm1mm7e7t5ivfg== X-Received: by 2002:aa7:8889:0:b0:6e7:48ef:aeca with SMTP id z9-20020aa78889000000b006e748efaecamr2351976pfe.9.1711612279071; Thu, 28 Mar 2024 00:51:19 -0700 (PDT) Received: from localhost ([47.254.32.37]) by smtp.gmail.com with ESMTPSA id g18-20020a62e312000000b006eadf4c2b67sm71991pfh.92.2024.03.28.00.51.18 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 28 Mar 2024 00:51:18 -0700 (PDT) From: Lai Jiangshan To: linux-kernel@vger.kernel.org Cc: rcu@vger.kernel.org, x86@kernel.org, Lai Jiangshan , "Paul E. McKenney" , Peter Zijlstra , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Josh Triplett , Boqun Feng , Steven Rostedt , Mathieu Desnoyers , Lai Jiangshan , Zqiang Subject: [PATCH 08/10] rcu: Implement PCPU_RCU_PREEMPT_COUNT framework Date: Thu, 28 Mar 2024 15:53:16 +0800 Message-Id: <20240328075318.83039-9-jiangshanlai@gmail.com> X-Mailer: git-send-email 2.19.1.6.gb485710b In-Reply-To: <20240328075318.83039-1-jiangshanlai@gmail.com> References: <20240328075318.83039-1-jiangshanlai@gmail.com> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Lai Jiangshan When the arch code provides HAVE_PCPU_RCU_PREEMPT_COUNT and the corresponding functions, rcu_preempt core uses the functions to implement rcu_read_[un]lock, rcu_preempt_depth(), special bits, switching and so on. Cc: "Paul E. McKenney" Cc: Peter Zijlstra Cc: Frederic Weisbecker Signed-off-by: Lai Jiangshan --- include/linux/rcupdate.h | 33 +++++++++++++++++++++++++++++++++ kernel/rcu/Kconfig | 8 ++++++++ kernel/rcu/rcu.h | 4 ++++ kernel/rcu/tree_plugin.h | 8 ++++++++ 4 files changed, 53 insertions(+) diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h index cc77d76a870b..bf369741ef93 100644 --- a/include/linux/rcupdate.h +++ b/include/linux/rcupdate.h @@ -70,6 +70,8 @@ static inline bool same_state_synchronize_rcu(unsigned long oldstate1, unsigned void rcu_read_unlock_special(void); +#ifndef CONFIG_PCPU_RCU_PREEMPT_COUNT + void __rcu_read_lock(void); void __rcu_read_unlock(void); @@ -81,6 +83,37 @@ void __rcu_read_unlock(void); */ #define rcu_preempt_depth() READ_ONCE(current->rcu_read_lock_nesting) #define rcu_preempt_depth_set(val) WRITE_ONCE(current->rcu_read_lock_nesting, (val)) +#define pcpu_rcu_preempt_special_set() do { } while (0) +#define pcpu_rcu_preempt_special_clear() do { } while (0) + +#else /* #ifndef CONFIG_PCPU_RCU_PREEMPT_COUNT */ + +#include + +static __always_inline void __rcu_read_lock(void) +{ + pcpu_rcu_preempt_count_add(1); + barrier(); +} + +static __always_inline void __rcu_read_unlock(void) +{ + barrier(); + if (unlikely(pcpu_rcu_preempt_count_dec_and_test())) + pcpu_rcu_read_unlock_special(); +} + +static inline int rcu_preempt_depth(void) +{ + return pcpu_rcu_preempt_count(); +} + +static inline void rcu_preempt_depth_set(int val) +{ + pcpu_rcu_preempt_count_set(val); +} + +#endif /* #else #ifndef CONFIG_PCPU_RCU_PREEMPT_COUNT */ #else /* #ifdef CONFIG_PREEMPT_RCU */ diff --git a/kernel/rcu/Kconfig b/kernel/rcu/Kconfig index e7d2dd267593..5d91147bc9a3 100644 --- a/kernel/rcu/Kconfig +++ b/kernel/rcu/Kconfig @@ -345,4 +345,12 @@ config RCU_DOUBLE_CHECK_CB_TIME Say Y here if you need tighter callback-limit enforcement. Say N here if you are unsure. +config HAVE_PCPU_RCU_PREEMPT_COUNT + bool + +config PCPU_RCU_PREEMPT_COUNT + def_bool y + depends on PREEMPT_RCU && HAVE_PCPU_RCU_PREEMPT_COUNT + depends on !PROVE_LOCKING && !RCU_STRICT_GRACE_PERIOD + endmenu # "RCU Subsystem" diff --git a/kernel/rcu/rcu.h b/kernel/rcu/rcu.h index ea5ae957c687..2322b040c5cd 100644 --- a/kernel/rcu/rcu.h +++ b/kernel/rcu/rcu.h @@ -671,6 +671,10 @@ static inline int rcu_stall_notifier_call_chain(unsigned long val, void *v) { re static inline void rcu_preempt_switch(struct task_struct *prev, struct task_struct *next) { +#ifdef CONFIG_PCPU_RCU_PREEMPT_COUNT + prev->rcu_read_lock_nesting = rcu_preempt_depth(); + pcpu_rcu_preempt_switch(next->rcu_read_lock_nesting, next->rcu_read_unlock_special.s); +#endif // #ifdef CONFIG_PCPU_RCU_PREEMPT_COUNT } #endif /* __KERNEL_RCU_H */ diff --git a/kernel/rcu/tree_plugin.h b/kernel/rcu/tree_plugin.h index 880b3fef1158..db68d0c1c1f2 100644 --- a/kernel/rcu/tree_plugin.h +++ b/kernel/rcu/tree_plugin.h @@ -107,10 +107,13 @@ static bool sync_rcu_exp_done(struct rcu_node *rnp); #define set_rcu_preempt_special(reason) do { \ WRITE_ONCE(current->rcu_read_unlock_special.b.reason, true); \ + pcpu_rcu_preempt_special_set(); \ } while (0) #define clear_rcu_preempt_special(reason) do { \ WRITE_ONCE(current->rcu_read_unlock_special.b.reason, false); \ + if (!current->rcu_read_unlock_special.s) \ + pcpu_rcu_preempt_special_clear(); \ } while (0) /* @@ -379,6 +382,8 @@ static int rcu_preempt_blocked_readers_cgp(struct rcu_node *rnp) return READ_ONCE(rnp->gp_tasks) != NULL; } +#ifndef CONFIG_PCPU_RCU_PREEMPT_COUNT + /* limit value for ->rcu_read_lock_nesting. */ #define RCU_NEST_PMAX (INT_MAX / 2) @@ -436,6 +441,8 @@ void __rcu_read_unlock(void) } EXPORT_SYMBOL_GPL(__rcu_read_unlock); +#endif /* #ifndef CONFIG_PCPU_RCU_PREEMPT_COUNT */ + /* * Advance a ->blkd_tasks-list pointer to the next entry, instead * returning NULL if at the end of the list. @@ -489,6 +496,7 @@ rcu_preempt_deferred_qs_irqrestore(struct task_struct *t, unsigned long flags) return; } t->rcu_read_unlock_special.s = 0; + pcpu_rcu_preempt_special_clear(); if (special.b.need_qs) { if (IS_ENABLED(CONFIG_RCU_STRICT_GRACE_PERIOD)) { rdp->cpu_no_qs.b.norm = false; From patchwork Thu Mar 28 07:53:17 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Lai Jiangshan X-Patchwork-Id: 13608100 Received: from mail-pg1-f179.google.com (mail-pg1-f179.google.com [209.85.215.179]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 87A30548FC; Thu, 28 Mar 2024 07:51:30 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.179 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711612292; cv=none; b=EmWRRoHODtu6zgkFd8NOfQ/r0VB3nlznIycNEDo6BOCJfvgR9qwc6HFAHy8NVTMBF3KgsAy/VvyXTw7yg+uRKODE22sVb6mxYk/NjOmgHS2Yl9ewki6Nb1jLs6OWVKxTMXfsdMIf6VRh7UR/QRx1v4vNfsgTK6CcnvvDEpYYaz4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711612292; c=relaxed/simple; bh=7bkBKLnD77zEWrwezIeGk/jY1nL8DIDPDOCksC6gLzk=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=G2AdACTUprAcH+DwRFxXP/1c2npHUGqhQ51SWB74KxHj2VD6OYt7+MUDQywASfJ7UXT7TgOlkBpu/mAGV6xKfmqOvUzp8W63LyyYUqhVH2IbhQuVYiNLxot7QZIJveI572MolIE3y8aLcBS/kkbm+McpSg7+z8Bzrv/x2goiiMU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=ZmXBrK4l; arc=none smtp.client-ip=209.85.215.179 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="ZmXBrK4l" Received: by mail-pg1-f179.google.com with SMTP id 41be03b00d2f7-5e152c757a5so404467a12.2; Thu, 28 Mar 2024 00:51:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1711612289; x=1712217089; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=3Fu9XbKPcpXhQXu1sKcZUeWcfqfg6wDFC9Zk5mwxGOY=; b=ZmXBrK4lB5+xwebUa8/R5c0/kIvTKbmfc/Rpd+KPtYdD2PAqswruXa+FlnhwWFxAE3 D91qt6RwsWFasVzdjxo6Smd1kGUIAKB71cbJVc92/SHpT5iggE62BfU/NGTcLhJbm3nO CdB2W8EBW6rvnjVgHElKDhiwr9KgzUu/HTq9ecckzgCHQ5AFtYoeKzepuLLDc8cJXfC5 za66ni3WNLOyjNTWCyC45ePsHZovqqO53rugV9J9F1asb7KOEXYZC1drmZ3MmBswZRTu +sgVAlGSAp8FNjUbGeqxQvlNIEiUVyk+GhJ494QflZDweNYds2GtlUIy7Yj8rRQqd15B PJ7A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711612289; x=1712217089; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=3Fu9XbKPcpXhQXu1sKcZUeWcfqfg6wDFC9Zk5mwxGOY=; b=sfU3Q5vhWuH+rdBbW2gVCRoTtzn6soXeCfY0SND9fXq+rdIMlqrPw9geeXxIX6zlqC bw2HSuZrg9y1BfRZQrvfLs4qcC5DvHkmN4IJzmVtIEJhoIbLHcxXTCKj9DCiBfHojLNA 7eCLYXwvrW9rmNKVef+Pkd+Wx9tLhF1GK/9rkkQrqv8jpa5YokTuieTXMoCpLI5f0QRu KKl+QA/qriKlTPOHTXWdLQFpND9ta5fAsyeuhkp6sfMl15VSx3yOvu5j0WGvjrxci8gN DOhF0VNLpiypZYY44wZND2pkw8KePVSmqnI6H4qp4ZMlmW8rP7SEp9n72cIXOv5rYU+E 63uQ== X-Gm-Message-State: AOJu0Yy9yEtYrGWHJPehzhC1jweS+LMMgAvaFl3cpXIShirzN9RE1Z/0 +pEPruLilTw1QgA8ey/cHq/YDGgRakYdYjAbMkdLLBLR/lOgsyLQ22PMYcZ4 X-Google-Smtp-Source: AGHT+IFHruU5tg3ltPttVZMCiIjCktbE25klSko7+N6nSsI7NgeSJznZIu5AsRBpBKsVYWkexqhY8w== X-Received: by 2002:a05:6a20:4c1e:b0:1a3:dfd2:949c with SMTP id fm30-20020a056a204c1e00b001a3dfd2949cmr1882329pzb.54.1711612289629; Thu, 28 Mar 2024 00:51:29 -0700 (PDT) Received: from localhost ([47.88.5.130]) by smtp.gmail.com with ESMTPSA id n6-20020a170902e54600b001e0b863b815sm865604plf.96.2024.03.28.00.51.28 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 28 Mar 2024 00:51:29 -0700 (PDT) From: Lai Jiangshan To: linux-kernel@vger.kernel.org Cc: rcu@vger.kernel.org, x86@kernel.org, Lai Jiangshan , "Paul E. McKenney" , Peter Zijlstra , Frederic Weisbecker , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , Uros Bizjak , Josh Poimboeuf , Nadav Amit , Breno Leitao , Kent Overstreet , Pawan Gupta , Rick Edgecombe , Vegard Nossum , Daniel Sneddon , Nikolay Borisov Subject: [PATCH 09/10] x86/rcu: Add rcu_preempt_count Date: Thu, 28 Mar 2024 15:53:17 +0800 Message-Id: <20240328075318.83039-10-jiangshanlai@gmail.com> X-Mailer: git-send-email 2.19.1.6.gb485710b In-Reply-To: <20240328075318.83039-1-jiangshanlai@gmail.com> References: <20240328075318.83039-1-jiangshanlai@gmail.com> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Lai Jiangshan Implement PCPU_RCU_PREEMPT_COUNT for x86. Mainly copied from asm/preempt.h Make rcu_read_[un]lock() inlined for rcu-preempt. Make rcu_read_lock() only one instruction. Make rcu_read_unlock() only two instructions in the fast path. Cc: "Paul E. McKenney" Cc: Peter Zijlstra Cc: Frederic Weisbecker Signed-off-by: Lai Jiangshan --- arch/x86/Kconfig | 1 + arch/x86/include/asm/current.h | 3 + arch/x86/include/asm/rcu_preempt.h | 107 +++++++++++++++++++++++++++++ arch/x86/kernel/cpu/common.c | 7 +- 4 files changed, 115 insertions(+), 3 deletions(-) create mode 100644 arch/x86/include/asm/rcu_preempt.h diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index 78050d5d7fac..7eb17c12f7b7 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -257,6 +257,7 @@ config X86 select HAVE_OBJTOOL if X86_64 select HAVE_OPTPROBES select HAVE_PAGE_SIZE_4KB + select HAVE_PCPU_RCU_PREEMPT_COUNT select HAVE_PCSPKR_PLATFORM select HAVE_PERF_EVENTS select HAVE_PERF_EVENTS_NMI diff --git a/arch/x86/include/asm/current.h b/arch/x86/include/asm/current.h index bf5953883ec3..dcc2ef784120 100644 --- a/arch/x86/include/asm/current.h +++ b/arch/x86/include/asm/current.h @@ -24,6 +24,9 @@ struct pcpu_hot { unsigned long top_of_stack; void *hardirq_stack_ptr; u16 softirq_pending; +#ifdef CONFIG_PCPU_RCU_PREEMPT_COUNT + int rcu_preempt_count; +#endif // #ifdef CONFIG_PCPU_RCU_PREEMPT_COUNT #ifdef CONFIG_X86_64 bool hardirq_stack_inuse; #else diff --git a/arch/x86/include/asm/rcu_preempt.h b/arch/x86/include/asm/rcu_preempt.h new file mode 100644 index 000000000000..cb25ebe038a5 --- /dev/null +++ b/arch/x86/include/asm/rcu_preempt.h @@ -0,0 +1,107 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef __ASM_RCU_PREEMPT_H +#define __ASM_RCU_PREEMPT_H + +#include +#include +#include + +#ifdef CONFIG_PCPU_RCU_PREEMPT_COUNT + +/* We use the MSB mostly because its available */ +#define RCU_PREEMPT_UNLOCK_SPECIAL_INVERTED 0x80000000 + +/* + * We use the RCU_PREEMPT_UNLOCK_SPECIAL_INVERTED bit as an inverted + * current->rcu_read_unlock_special.s such that a decrement hitting 0 + * means we can and should call rcu_read_unlock_special(). + */ +#define RCU_PREEMPT_INIT (0 + RCU_PREEMPT_UNLOCK_SPECIAL_INVERTED) + +/* + * We mask the RCU_PREEMPT_UNLOCK_SPECIAL_INVERTED bit so as not to + * confuse all current users that think a non-zero value indicates we + * are in a critical section. + */ +static inline int pcpu_rcu_preempt_count(void) +{ + return raw_cpu_read_4(pcpu_hot.rcu_preempt_count) & ~RCU_PREEMPT_UNLOCK_SPECIAL_INVERTED; +} + +static inline void pcpu_rcu_preempt_count_set(int count) +{ + int old, new; + + old = raw_cpu_read_4(pcpu_hot.rcu_preempt_count); + do { + new = (old & RCU_PREEMPT_UNLOCK_SPECIAL_INVERTED) | + (count & ~RCU_PREEMPT_UNLOCK_SPECIAL_INVERTED); + } while (!raw_cpu_try_cmpxchg_4(pcpu_hot.rcu_preempt_count, &old, new)); +} + +/* + * We fold the RCU_PREEMPT_UNLOCK_SPECIAL_INVERTED bit into the RCU + * preempt count such that rcu_read_unlock() can decrement and test for + * the need of unlock-special handling with a single instruction. + * + * We invert the actual bit, so that when the decrement hits 0 we know + * we both reach a quiescent state (no rcu preempt count) and need to + * handle unlock-special (the bit is cleared), normally to report the + * quiescent state immediately. + */ + +static inline void pcpu_rcu_preempt_special_set(void) +{ + raw_cpu_and_4(pcpu_hot.rcu_preempt_count, ~RCU_PREEMPT_UNLOCK_SPECIAL_INVERTED); +} + +static inline void pcpu_rcu_preempt_special_clear(void) +{ + raw_cpu_or_4(pcpu_hot.rcu_preempt_count, RCU_PREEMPT_UNLOCK_SPECIAL_INVERTED); +} + +static inline bool pcpu_rcu_preempt_special_test(void) +{ + return !(raw_cpu_read_4(pcpu_hot.rcu_preempt_count) & RCU_PREEMPT_UNLOCK_SPECIAL_INVERTED); +} + +static inline void pcpu_rcu_preempt_switch(int count, bool special) +{ + if (likely(!special)) + raw_cpu_write(pcpu_hot.rcu_preempt_count, count | RCU_PREEMPT_UNLOCK_SPECIAL_INVERTED); + else + raw_cpu_write(pcpu_hot.rcu_preempt_count, count); +} + +/* + * The various rcu_preempt_count add/sub methods + */ + +static __always_inline void pcpu_rcu_preempt_count_add(int val) +{ + raw_cpu_add_4(pcpu_hot.rcu_preempt_count, val); +} + +static __always_inline void pcpu_rcu_preempt_count_sub(int val) +{ + raw_cpu_add_4(pcpu_hot.rcu_preempt_count, -val); +} + +/* + * Because we keep RCU_PREEMPT_UNLOCK_SPECIAL_INVERTED set when we do + * _not_ need to handle unlock-special for a fast-path decrement. + */ +static __always_inline bool pcpu_rcu_preempt_count_dec_and_test(void) +{ + return GEN_UNARY_RMWcc("decl", __my_cpu_var(pcpu_hot.rcu_preempt_count), e, + __percpu_arg([var])); +} + +#define pcpu_rcu_read_unlock_special() \ +do { \ + rcu_read_unlock_special(); \ +} while (0) + +#endif // #ifdef CONFIG_PCPU_RCU_PREEMPT_COUNT + +#endif /* __ASM_RCU_PREEMPT_H */ diff --git a/arch/x86/kernel/cpu/common.c b/arch/x86/kernel/cpu/common.c index ba8cf5e9ce56..0b204a649442 100644 --- a/arch/x86/kernel/cpu/common.c +++ b/arch/x86/kernel/cpu/common.c @@ -1992,9 +1992,10 @@ static __init int setup_clearcpuid(char *arg) __setup("clearcpuid=", setup_clearcpuid); DEFINE_PER_CPU_ALIGNED(struct pcpu_hot, pcpu_hot) = { - .current_task = &init_task, - .preempt_count = INIT_PREEMPT_COUNT, - .top_of_stack = TOP_OF_INIT_STACK, + .current_task = &init_task, + .preempt_count = INIT_PREEMPT_COUNT, + .top_of_stack = TOP_OF_INIT_STACK, + .rcu_preempt_count = RCU_PREEMPT_INIT, }; EXPORT_PER_CPU_SYMBOL(pcpu_hot); EXPORT_PER_CPU_SYMBOL(const_pcpu_hot); From patchwork Thu Mar 28 07:53:18 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Lai Jiangshan X-Patchwork-Id: 13608101 Received: from mail-oa1-f44.google.com (mail-oa1-f44.google.com [209.85.160.44]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BCA545380D; Thu, 28 Mar 2024 07:51:37 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.160.44 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711612299; cv=none; b=s+hz7s8WCUJghEhZakYWmOeAQQy2tUIsxSLYho/bTH3nvLJxNywO5SLeNzvzGY+Simqq3+PMCzJBoT+UPWS5y8QWOAANtzCQjMsWnKmW49F+WReK+mtwSGyY6ybyGveVY3ajgvqxPvd/rQ6N1HpW6Vj0UQ2AbHh5yBdRvmd9/V0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711612299; c=relaxed/simple; bh=1u4ClC0nXJYWtLmcNlGJ9Evf3oh4cLDO6utJlz4b/+Y=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=jHamKHR6XA5k8dvzRKhJDuaj/5nSsSPSE+xYIwYZOviaqkCKyaHuQ9LYJlPJGQRceYwyIZ6O+SfeEnKTbYkLJ8Ia2wM0otNAMlcF6VpADZEvWBxKY+rltR1UOwaFC/DRmxODCw1QIdrtf4IUrvYVG7A/vmUEHB0FJL36w8gS2LU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=fcVj1oRu; arc=none smtp.client-ip=209.85.160.44 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="fcVj1oRu" Received: by mail-oa1-f44.google.com with SMTP id 586e51a60fabf-2282597eff5so478301fac.3; Thu, 28 Mar 2024 00:51:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1711612296; x=1712217096; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=kb+0BwtXCOHTxNecM62LX9dxsLjU1SobPdBfLYOqsoE=; b=fcVj1oRuCRVoGBClW2Kr66cb/PPnfdvEjmoYlR+lqG1c45efv+5Bs4iQOsZyxuZfcd WWtpvEg3KsSKpBWDOw9Gr9zDsb67TfcwhorDJ3e42tVTdchSwwVhzqWiORAUjNSr7umk EpKa4KZe70WUUuFNPzzyUih5BKM4QzGEfGVOwCZOru9BoeaBxZaWlSs0tNHl0AEcNtMw 2Qxu6fgQ8Ph3WIHLY0S2Algtq+GVCye8QoKmFvY5xN/Xupu+VOfFvbEkJtJ0tW+4aA50 hnWE/dDqTg+mwgNEkKloHMGiyHTC4uxVW+JH7QnfYQ3Djv57FQlscAf67ksw1lNUwIQQ 7epQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711612296; x=1712217096; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=kb+0BwtXCOHTxNecM62LX9dxsLjU1SobPdBfLYOqsoE=; b=W5inbw0NmiJCeCMRX+XX76CLiJnTSiKvJG1jJwIADhteLIY5TnjzX9ySAE5YgkO924 v+TBDg+7eyG0hiGFWazDHdGpwh4PS06voceE7NwJ9BLjGPN5hjmSz/WWHK0QrF4j4XhJ jmc8O8ZwynKy4FESLNvBrO9RgBG9URy/b9R8xpdfwQZbfvH30PVTsZzJC5gX+Q3Yl5HX AdwonZiy8DbG5nYIgSG9B5kLYLVhF9uyTDxVyGMEcsgcZrlpjMD75CNGtRaw5ucSQJGL aYFB/f+RYuUX/nGV1UA7AQykdIuLBaiwXiSODHBp9iWHl7fJhs3WV4IsIc3k6GCbmFQi DjCA== X-Gm-Message-State: AOJu0Yyz4XEasFhK2FbBydwSHHf9fcmWahYb7Ak05H/uBHHOAqadiqTL nXLqpXNGtqLTi32fhoA/ISOQVEsyoUqzzTfA0lgjG7kxjDo+JPQcJcqpE/rg X-Google-Smtp-Source: AGHT+IEzqMx8djm+FHmNWfNhehJ+tHqeZFhrJJCl/Hxw+o3WO/76dLEGu+gqfnycvOWAL9IbEKWnGg== X-Received: by 2002:a05:6871:4406:b0:229:e636:921f with SMTP id nd6-20020a056871440600b00229e636921fmr2181447oab.49.1711612296343; Thu, 28 Mar 2024 00:51:36 -0700 (PDT) Received: from localhost ([47.88.5.130]) by smtp.gmail.com with ESMTPSA id e18-20020aa79812000000b006ea81423c65sm774667pfl.148.2024.03.28.00.51.35 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 28 Mar 2024 00:51:35 -0700 (PDT) From: Lai Jiangshan To: linux-kernel@vger.kernel.org Cc: rcu@vger.kernel.org, x86@kernel.org, Lai Jiangshan , "Paul E. McKenney" , Peter Zijlstra , Frederic Weisbecker , Andy Lutomirski , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" Subject: [PATCH 10/10] x86/rcu: Add THUNK rcu_read_unlock_special_thunk Date: Thu, 28 Mar 2024 15:53:18 +0800 Message-Id: <20240328075318.83039-11-jiangshanlai@gmail.com> X-Mailer: git-send-email 2.19.1.6.gb485710b In-Reply-To: <20240328075318.83039-1-jiangshanlai@gmail.com> References: <20240328075318.83039-1-jiangshanlai@gmail.com> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Lai Jiangshan Add rcu_read_unlock_special_thunk(), so that the inlined rcu_read_unlock() doesn't need any code to save the caller-saved registers. Make rcu_read_unlock() only two instructions in the slow path at the caller site. Cc: "Paul E. McKenney" Cc: Peter Zijlstra Cc: Frederic Weisbecker Signed-off-by: Lai Jiangshan --- arch/x86/entry/thunk_64.S | 5 +++++ arch/x86/include/asm/rcu_preempt.h | 4 +++- 2 files changed, 8 insertions(+), 1 deletion(-) diff --git a/arch/x86/entry/thunk_64.S b/arch/x86/entry/thunk_64.S index 119ebdc3d362..10c60369a67c 100644 --- a/arch/x86/entry/thunk_64.S +++ b/arch/x86/entry/thunk_64.S @@ -13,3 +13,8 @@ THUNK preempt_schedule_thunk, preempt_schedule THUNK preempt_schedule_notrace_thunk, preempt_schedule_notrace EXPORT_SYMBOL(preempt_schedule_thunk) EXPORT_SYMBOL(preempt_schedule_notrace_thunk) + +#ifdef CONFIG_PCPU_RCU_PREEMPT_COUNT +THUNK rcu_read_unlock_special_thunk, rcu_read_unlock_special +EXPORT_SYMBOL_GPL(rcu_read_unlock_special_thunk) +#endif /* #ifdef CONFIG_PCPU_RCU_PREEMPT_COUNT */ diff --git a/arch/x86/include/asm/rcu_preempt.h b/arch/x86/include/asm/rcu_preempt.h index cb25ebe038a5..acdd73b74c05 100644 --- a/arch/x86/include/asm/rcu_preempt.h +++ b/arch/x86/include/asm/rcu_preempt.h @@ -97,9 +97,11 @@ static __always_inline bool pcpu_rcu_preempt_count_dec_and_test(void) __percpu_arg([var])); } +extern asmlinkage void rcu_read_unlock_special_thunk(void); + #define pcpu_rcu_read_unlock_special() \ do { \ - rcu_read_unlock_special(); \ + asm volatile ("call rcu_read_unlock_special_thunk" : ASM_CALL_CONSTRAINT);\ } while (0) #endif // #ifdef CONFIG_PCPU_RCU_PREEMPT_COUNT