From patchwork Thu Jan 12 19:43:27 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Zijlstra X-Patchwork-Id: 13098768 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B8874C61DB3 for ; Thu, 12 Jan 2023 19:57:55 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9C66080009; Thu, 12 Jan 2023 14:57:31 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 932EE8000D; Thu, 12 Jan 2023 14:57:31 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2C41A8000F; Thu, 12 Jan 2023 14:57:31 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id F2DD880009 for ; Thu, 12 Jan 2023 14:57:30 -0500 (EST) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id CB1EE40357 for ; Thu, 12 Jan 2023 19:57:30 +0000 (UTC) X-FDA: 80347206660.07.7CF8704 Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) by imf10.hostedemail.com (Postfix) with ESMTP id 9563DC0011 for ; Thu, 12 Jan 2023 19:57:28 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=infradead.org header.s=desiato.20200630 header.b=X3sYuy6f; dmarc=none; spf=none (imf10.hostedemail.com: domain of peterz@infradead.org has no SPF policy when checking 90.155.92.199) smtp.mailfrom=peterz@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1673553449; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:references:dkim-signature; bh=U+TfVkp0QsADQiBGN/hq9wf1EsYpsa4vTkdZ0mLdl1k=; b=PF7l2aGN991fa/R3kjBqBM+84zLGbYEan10FJSjnfgYohN3k3GbAp2SPMbr0rjMI2+OIq/ djJaTsVgK81NbKhjH4ams+FilrdSLREtyf+976SQwSpEbvzaA9V+Y5hPMkEzaa/qY6tFpe x3hN1rXyy1wqQmSHbPearxNxd+FkIxk= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=infradead.org header.s=desiato.20200630 header.b=X3sYuy6f; dmarc=none; spf=none (imf10.hostedemail.com: domain of peterz@infradead.org has no SPF policy when checking 90.155.92.199) smtp.mailfrom=peterz@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1673553449; a=rsa-sha256; cv=none; b=YtSQwCMxAYtKikRWsL3meOWsISdR3tALsDTBNUtO9x0STzyKSyk2T6v67MAK9yKz/Ml1RZ SsACeBlvA6DMZ58z7dHg0oupfTl0OchfjTFLH3e30pYa8vTIa9QX24arIFXsFseR7lC7l4 Hs/LD6f5nkhqVr/ilhu2RjvLbxUoGxk= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=U+TfVkp0QsADQiBGN/hq9wf1EsYpsa4vTkdZ0mLdl1k=; b=X3sYuy6fL3RqgKJT5/kyY/T0ww A4w/aGqBXYzbjq0ovUGzSiXSnwqu0hAP6aWbAx6EM9RQMixaeSFkTeMoq5Tp8g8+2xTyyfjh6NtMy mC6f71rhHNZc8yxzfzYNgVvDLSBEmfEe+153OuiRRLXriJ46cM5MvqvIC8SrpxxjzTew2zrO+fd+H yeusXlMUu35aUpqRVpXLv7cU+tiHfK7ZE9ipHRum1o7+Yb+riAOhBcycD9sPHIAzeWqFkY5/vtieb ve85chw+D6aD1ff3Jdui4CMF6z/ux8MH+wBBaRcv1NSj17Q08cT1A6MpfchKYVWU8r5TfnmKZG09P RV5gCBxw==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.96 #2 (Red Hat Linux)) id 1pG3hD-0045oK-2O; Thu, 12 Jan 2023 19:57:10 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 2D90C30340F; Thu, 12 Jan 2023 20:57:13 +0100 (CET) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id C388F2CCF1F60; Thu, 12 Jan 2023 20:57:07 +0100 (CET) Message-ID: <20230112195540.130014793@infradead.org> User-Agent: quilt/0.66 Date: Thu, 12 Jan 2023 20:43:27 +0100 From: Peter Zijlstra To: peterz@infradead.org Cc: richard.henderson@linaro.org, ink@jurassic.park.msu.ru, mattst88@gmail.com, vgupta@kernel.org, linux@armlinux.org.uk, nsekhar@ti.com, brgl@bgdev.pl, ulli.kroll@googlemail.com, linus.walleij@linaro.org, shawnguo@kernel.org, Sascha Hauer , kernel@pengutronix.de, festevam@gmail.com, linux-imx@nxp.com, tony@atomide.com, khilman@kernel.org, krzysztof.kozlowski@linaro.org, alim.akhtar@samsung.com, catalin.marinas@arm.com, will@kernel.org, guoren@kernel.org, bcain@quicinc.com, chenhuacai@kernel.org, kernel@xen0n.name, geert@linux-m68k.org, sammy@sammy.net, monstr@monstr.eu, tsbogend@alpha.franken.de, dinguyen@kernel.org, jonas@southpole.se, stefan.kristiansson@saunalahti.fi, shorne@gmail.com, James.Bottomley@HansenPartnership.com, deller@gmx.de, mpe@ellerman.id.au, npiggin@gmail.com, christophe.leroy@csgroup.eu, paul.walmsley@sifive.com, palmer@dabbelt.com, aou@eecs.berkeley.edu, hca@linux.ibm.com, gor@linux.ibm.com, agordeev@linux.ibm.com, borntraeger@linux.ibm.com, svens@linux.ibm.com, ysato@users.sourceforge.jp, dalias@libc.org, davem@davemloft.net, richard@nod.at, anton.ivanov@cambridgegreys.com, johannes@sipsolutions.net, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, x86@kernel.org, hpa@zytor.com, acme@kernel.org, mark.rutland@arm.com, alexander.shishkin@linux.intel.com, jolsa@kernel.org, namhyung@kernel.org, jgross@suse.com, srivatsa@csail.mit.edu, amakhalov@vmware.com, pv-drivers@vmware.com, boris.ostrovsky@oracle.com, chris@zankel.net, jcmvbkbc@gmail.com, rafael@kernel.org, lenb@kernel.org, pavel@ucw.cz, gregkh@linuxfoundation.org, mturquette@baylibre.com, sboyd@kernel.org, daniel.lezcano@linaro.org, lpieralisi@kernel.org, sudeep.holla@arm.com, agross@kernel.org, andersson@kernel.org, konrad.dybcio@linaro.org, anup@brainfault.org, thierry.reding@gmail.com, jonathanh@nvidia.com, jacob.jun.pan@linux.intel.com, atishp@atishpatra.org, Arnd Bergmann , yury.norov@gmail.com, andriy.shevchenko@linux.intel.com, linux@rasmusvillemoes.dk, dennis@kernel.org, tj@kernel.org, cl@linux.com, rostedt@goodmis.org, mhiramat@kernel.org, frederic@kernel.org, paulmck@kernel.org, pmladek@suse.com, senozhatsky@chromium.org, john.ogness@linutronix.de, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, ryabinin.a.a@gmail.com, glider@google.com, andreyknvl@gmail.com, dvyukov@google.com, vincenzo.frascino@arm.com, Andrew Morton , jpoimboe@kernel.org, linux-alpha@vger.kernel.org, linux-kernel@vger.kernel.org, linux-snps-arc@lists.infradead.org, linux-omap@vger.kernel.org, linux-samsung-soc@vger.kernel.org, linux-csky@vger.kernel.org, linux-hexagon@vger.kernel.org, linux-ia64@vger.kernel.org, loongarch@lists.linux.dev, linux-m68k@lists.linux-m68k.org, linux-mips@vger.kernel.org, openrisc@lists.librecores.org, linux-parisc@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, linux-sh@vger.kernel.org, sparclinux@vger.kernel.org, linux-um@lists.infradead.org, linux-perf-users@vger.kernel.org, virtualization@lists.linux-foundation.org, linux-xtensa@linux-xtensa.org, linux-acpi@vger.kernel.org, linux-pm@vger.kernel.org, linux-clk@vger.kernel.org, linux-arm-msm@vger.kernel.org, linux-tegra@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, linux-trace-kernel@vger.kernel.org, kasan-dev@googlegroups.com, "Rafael J. Wysocki" , Ulf Hansson Subject: [PATCH v3 13/51] cpuidle: Fix ct_idle_*() usage References: <20230112194314.845371875@infradead.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 9563DC0011 X-Stat-Signature: dfe43khfso97k4ohxwq7tuhhhi5hnsih X-HE-Tag: 1673553448-959056 X-HE-Meta: U2FsdGVkX18AwAC3CnP/4Vdjik7AGAeowmUGmr459hrU68y44PunidOwemZcIb4gWbo442iDkM2LxyIVPifdDS7OpQeY8irjW2BunSNbaQzQZbzzKXURrfbjdC1Cs1MlphjjECBsUZLvlCbZJ2OzdVLDLwXw5IBeAoIF98SyR8leJwJ8cVCCtgBWag4PK7xCkceM/5Cp26y+W7q8YCGTd2R5l3xdfbouuLn6KabOulE75TpzugN1blu/yrv0TdkOk73Sid8cicfxN1WVTvDTcMZv+dw2yMIa8NTCHKyVPr1wFmgkIcRWWK1czsss1saD5QIaSbWfM9JHx7hCtlz2DgS2WRS3qO2jfWoUX0gifDeNBKBdFqC/lM/xvRnnbHzxMvLkZ8zRru3OD/sNd0XFab4T88OTjex0plvmi+O+aj+mBlQoGjbUvyQfJDUtlJp/bIaTA123d1QNmXt+/7HHiPmLsuL/aWc0jn5KuALae0jxfyIo0jhNkylp9NB17uwptpNfeROQvjp9VNGXAF069El/qKJ5K6j+cj25izt5Gnhrpy8pHuQHwXCO6/6xXJOVmoIX8h+8TyvI1xVlSzSpa8T4TjrScY5iwtsVTAj9bUsOZ5Ek7gUyWK8p6oPw6diyf2VmeW5NGIpvbcffoUEaTuu0adQW7cR/Vihe1KYqlFxHxKD9SuVR3djALA6ggIpvBHFhXyH0bYAUC9Mnp+0/AMSHPW4FA87KsZ44lb3CObzqKjMhApmcJIJ9riFveETmkaER2IdgunY+TNhFDPbA9tVamq81SqFCl5mTQ0ZnfigHwlsLNqI0lVEXcvCXAymKtVYYe98fFdXklpqZ28mxbw9YsP/jMZnaIXYLq3w0dzSOyUMz4dO92aekKSja26EUfUeN+ybepGDh5hHuDlFXnLAp2M48t1GC255tINwZ9dgBMp44KIBhp7gYtc+mZfoqoY2rBUiOr8Y9O0lzAca CLkInVnW v8V+1IGyXanlQNTXicHk767YzOzBSIEY4wLB7aoefVRGDIbLv0PqvIcI178RXiRXvUud0f9NhzE73HZKJVenUyBxLUjJBIbFdo9tOce9mikYcDElpN8LDT6aU0NF2ibqRbOkovYH8w3xIs+H00qskA9vheBFQskh3R1nIUnQ2X0QDc8N5hWkdT10xjPTxS4FemN67YSGdyOJw66IwFY6fT0pCDSLUBeq5XLdeq7vQR8tsdg9B3rjXWi/H/a0wFiL55pyzbzH7DN1xrY1LmhFzMfuUGXOuP5umUtILugasF77SeOYJza2Fi9xtudJqQ2Q3+z5DFLQsa8OM0fVuD0/YmgFz6Q6ZeNnfEGEzcu25FyGJW6XT8T5VixwCncgPYpIbP0zJZGHDllu3zp5mYJUAVHEPloFjWSB8df8uK85WGjhPsEdxxyF87EIkQVsS3hSOdljL9tX6OTO/kNS6fKSnt12dXhzxhlOYl+MSs7nUOt0wVV7mtPSJSGkaSmzhDJzZabqLF2PKTHmsFdZtJwRfhwEgpKlDtA0dUZyEMBMOzCW3VGH2JkrlMTidi6+z2K/O6CNE5CHOqEMhDjPd2DMPgv4Cv/kFcxQe1V/SB8ICUCo6iXs5UZhK5tkJgf9kpIXBXHzXA2+fPLqp8DYj6sJEFIqIydM/cEgWxcylQXtBYLoBOCJfgEJJjHe2epBpDaVY8co3l3QiOrcfdSrqxcT9QirDKZvlDBPhrVc2SVkvp+JGp8qZsQP1YpmSZ5wepsMcLx344afh+1nj5tq6Xkmnn/mcA8CKr73XjfjcqnV9JM79rzGgAdQJjr/qVD3ktXsv+BHKk6OtQOMEv3xbjRXI9EATw7Ny77QcT2sr0XY2/4SU6q/L+ohs6ct5CSwfaQNVb8PVov23wYZGjBu7g9hkswakiRfENSqiP2CLww0ysJU4PMX30WMG0hy4Gfo8nxB48rS5o08OzXozN5v8u3nA+8ikUEqy zpCnjl+C rLI6cLLir2Yk7OvtXTUMNBIoNXyy95aLkeZUdrvvrn7VyeG1LrDqvcIdes9ATLmVZQ1n+EohDumDGTXvgrcyauVVzfSN2BRIHJzDLRnfSiKWG4E84X8bA16rutXOosQwSeqz1PhLKMGfodG0k9qjeJz19GhLi+n08LcS/atXwTfSkQCxTA65jGZ62EBawbW3q0dp5YCOTS6P34voZkISNHCqZgDxc68HNRAFYds85SPVZohU7IbVTlU6SOYMFUa5zioPckb9jUe4yqb3PvnpoNZVduKvOGrWBaa/i5xwa4NEuhFDJLxhlHwxxxcOZjWROa6umW0MloNmDfhExHouIK9I8alyLv2zzjcgWRmLw36UJ/F2uOwg+DAoANeIpXeUFYAdlrC57RgwFxbVyDcKe5zJ22t3OW7DO9G/n3RmW5t7iKpehPAdxVgfXhv31agelHiTry61KCb9I1Zm5nlNlwoLLtTy5skYkn+nz74UckA9kia5XR3DpDekj1fvQMZLpdyfDa7ihQu1+MbckyGXNd2QFns0Xk26ip31tqhAI+P5MrjH0C3I0upNd+ewzm8x10jq/d16lWkKL/LL4xt5Q3RsXbwKknjUL/gulMOGmU2sJtTcE8CNhtRUZF1uV6SPqlagyfgdqSq9TaZfpD3hxw1c9e1VXitHDjDNyEhk/rcwenF5+GiLJWTFkZvE9ia85gE4h6QDzz7lhWK+3faXtxOYz+S2XW+qSGKCxnqXyVPlkZLvEs83Q7rj/NgOABO1OSR65LgBgrYgzRjv7UzRxq5tHzR9pq6Qhxzx1n1+D5yKYylnc6TKdudIRzvxAZ2V990qnGct97e7x6gJNtWmIcdV/mi5bV8hq0rSHPj6leCfrb9+ghJFxCcn7lObdWami+EkT8gIW1D5so971hMVMb0nAgyc2b+Mtx2zfXDKUjGlYDVoMFvaDlWdblNYK0+hoHrHy4VqZCeYWUlhH0tWwmiUbDzlL xjBSPTlQ mSq/obHsKh8YDv+xsekVuXbwd2pPToi4MRQ4geVd21upAnB/WsMRNC+lr6hRI+41rUbdHWDwnBQ3hTadxbXaRkDtf7FBBgFnnDrhBQKmgkB7RHUOxIqkVYOW/GCkRQHDt1FghoIeTExllMXBOSO4nXMeV8+weUKiszBL4rqXhyq8Tt3BOkCWBx1sAmmvIZeb6I4BkazylAX44ipiHPFrp+IqFPf+0S8V1fNNDBFweUTGFvUFNCGTmJ2e4TgSe89xrFQKUKymn2lsFqB5qC7SGy0eBSUJJAy/l/XdrHGu5oS4RRYkgCcReqhIN/3Zq30Vtw+14qRfl2e1PbDwALDYGTuS8a5YvW4AauKxG6tSkzcIG0tcJGxXCkdah0Md5oznOssiKq6773+rxcTix50vHYmiai9c3CPcbNZjD5RqTp8yo8iVAMSsMf2+AFZtztqsLyN0Ey+fi9iKJ953rLrh5D9DM9QmmaOPqqGlJVJ+/20rBMIusItTAIhbcIKqtHd1Gk4ayo5BUg68aspXRGyrPAdSYX7oOu2LZ31zyIX5vPO8Q2ivMNu7wyp/g6Bd1dGt363HHrYbOVrAI0xRgvctw5MO41O/ZDInd8ewBErFIG9ZvYKgKM5N5waosOSP05K4Pd1M1OrxgoX1nfHNh7bz/q5oU3XkQf1L1TDmy1lGz33mT2IqyDwZ1x0h8WLZhobI/hjQFQXvdaPgdBy2sE46Nu2pMuildCXMZGUbPRFlKZmaVHQssmuUXt1t5okQzlPuAc/dzgErr0+U6eD7FvCyc8xDpHLaP8XZvVhDH9BbPvrV8/rqxtN4IpL3YdonsvpGeeXCkmQBphtWeEbnrSLjIC8vZ9hdaDVAWbm2vBhxMQCvfez3jwo0zIIBO8VchPm5SM4PvtUxBQIvsZhSz4smOcq+axK6FhrM/onMTZ/X1PpUk3N9WjoyjWmamm3UxqU/37OhSuT0h3alKk6CULosjpxV720/J R0TxsELT KDQOxpc+P4B4eWyUVIlDdUCZxh4d/h31rrF7vzfIjvcWPNI/Fd06cE78G/CKL5oL7SQYDMbV+5Kio1vUl6U2ScrVrAq85lOVvcOLKCznQTsDsZloE5XN9IAF6GRWCvtfMgj+Y1FFoVfE8+GRqdu+w9 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The whole disable-RCU, enable-IRQS dance is very intricate since changing IRQ state is traced, which depends on RCU. Add two helpers for the cpuidle case that mirror the entry code. Signed-off-by: Peter Zijlstra (Intel) Acked-by: Rafael J. Wysocki Acked-by: Frederic Weisbecker Tested-by: Tony Lindgren Tested-by: Ulf Hansson --- arch/arm/mach-imx/cpuidle-imx6q.c | 4 +-- arch/arm/mach-imx/cpuidle-imx6sx.c | 4 +-- arch/arm/mach-omap2/cpuidle34xx.c | 4 +-- arch/arm/mach-omap2/cpuidle44xx.c | 8 +++--- drivers/acpi/processor_idle.c | 8 ++++-- drivers/cpuidle/cpuidle-big_little.c | 4 +-- drivers/cpuidle/cpuidle-mvebu-v7.c | 4 +-- drivers/cpuidle/cpuidle-psci.c | 4 +-- drivers/cpuidle/cpuidle-riscv-sbi.c | 4 +-- drivers/cpuidle/cpuidle-tegra.c | 8 +++--- drivers/cpuidle/cpuidle.c | 11 ++++---- include/linux/clockchips.h | 4 +-- include/linux/cpuidle.h | 34 ++++++++++++++++++++++++-- kernel/sched/idle.c | 45 ++++++++++------------------------- kernel/time/tick-broadcast.c | 6 +++- 15 files changed, 86 insertions(+), 66 deletions(-) --- a/arch/arm/mach-imx/cpuidle-imx6q.c +++ b/arch/arm/mach-imx/cpuidle-imx6q.c @@ -25,9 +25,9 @@ static int imx6q_enter_wait(struct cpuid imx6_set_lpm(WAIT_UNCLOCKED); raw_spin_unlock(&cpuidle_lock); - ct_idle_enter(); + ct_cpuidle_enter(); cpu_do_idle(); - ct_idle_exit(); + ct_cpuidle_exit(); raw_spin_lock(&cpuidle_lock); if (num_idle_cpus-- == num_online_cpus()) --- a/arch/arm/mach-imx/cpuidle-imx6sx.c +++ b/arch/arm/mach-imx/cpuidle-imx6sx.c @@ -47,9 +47,9 @@ static int imx6sx_enter_wait(struct cpui cpu_pm_enter(); cpu_cluster_pm_enter(); - ct_idle_enter(); + ct_cpuidle_enter(); cpu_suspend(0, imx6sx_idle_finish); - ct_idle_exit(); + ct_cpuidle_exit(); cpu_cluster_pm_exit(); cpu_pm_exit(); --- a/arch/arm/mach-omap2/cpuidle34xx.c +++ b/arch/arm/mach-omap2/cpuidle34xx.c @@ -133,9 +133,9 @@ static int omap3_enter_idle(struct cpuid } /* Execute ARM wfi */ - ct_idle_enter(); + ct_cpuidle_enter(); omap_sram_idle(); - ct_idle_exit(); + ct_cpuidle_exit(); /* * Call idle CPU PM enter notifier chain to restore --- a/arch/arm/mach-omap2/cpuidle44xx.c +++ b/arch/arm/mach-omap2/cpuidle44xx.c @@ -105,9 +105,9 @@ static int omap_enter_idle_smp(struct cp } raw_spin_unlock_irqrestore(&mpu_lock, flag); - ct_idle_enter(); + ct_cpuidle_enter(); omap4_enter_lowpower(dev->cpu, cx->cpu_state); - ct_idle_exit(); + ct_cpuidle_exit(); raw_spin_lock_irqsave(&mpu_lock, flag); if (cx->mpu_state_vote == num_online_cpus()) @@ -186,10 +186,10 @@ static int omap_enter_idle_coupled(struc } } - ct_idle_enter(); + ct_cpuidle_enter(); omap4_enter_lowpower(dev->cpu, cx->cpu_state); cpu_done[dev->cpu] = true; - ct_idle_exit(); + ct_cpuidle_exit(); /* Wakeup CPU1 only if it is not offlined */ if (dev->cpu == 0 && cpumask_test_cpu(1, cpu_online_mask)) { --- a/drivers/acpi/processor_idle.c +++ b/drivers/acpi/processor_idle.c @@ -642,6 +642,8 @@ static int __cpuidle acpi_idle_enter_bm( */ bool dis_bm = pr->flags.bm_control; + instrumentation_begin(); + /* If we can skip BM, demote to a safe state. */ if (!cx->bm_sts_skip && acpi_idle_bm_check()) { dis_bm = false; @@ -663,11 +665,11 @@ static int __cpuidle acpi_idle_enter_bm( raw_spin_unlock(&c3_lock); } - ct_idle_enter(); + ct_cpuidle_enter(); acpi_idle_do_entry(cx); - ct_idle_exit(); + ct_cpuidle_exit(); /* Re-enable bus master arbitration */ if (dis_bm) { @@ -677,6 +679,8 @@ static int __cpuidle acpi_idle_enter_bm( raw_spin_unlock(&c3_lock); } + instrumentation_end(); + return index; } --- a/drivers/cpuidle/cpuidle-big_little.c +++ b/drivers/cpuidle/cpuidle-big_little.c @@ -126,13 +126,13 @@ static int bl_enter_powerdown(struct cpu struct cpuidle_driver *drv, int idx) { cpu_pm_enter(); - ct_idle_enter(); + ct_cpuidle_enter(); cpu_suspend(0, bl_powerdown_finisher); /* signals the MCPM core that CPU is out of low power state */ mcpm_cpu_powered_up(); - ct_idle_exit(); + ct_cpuidle_exit(); cpu_pm_exit(); --- a/drivers/cpuidle/cpuidle-mvebu-v7.c +++ b/drivers/cpuidle/cpuidle-mvebu-v7.c @@ -36,9 +36,9 @@ static int mvebu_v7_enter_idle(struct cp if (drv->states[index].flags & MVEBU_V7_FLAG_DEEP_IDLE) deepidle = true; - ct_idle_enter(); + ct_cpuidle_enter(); ret = mvebu_v7_cpu_suspend(deepidle); - ct_idle_exit(); + ct_cpuidle_exit(); cpu_pm_exit(); --- a/drivers/cpuidle/cpuidle-psci.c +++ b/drivers/cpuidle/cpuidle-psci.c @@ -74,7 +74,7 @@ static int __psci_enter_domain_idle_stat else pm_runtime_put_sync_suspend(pd_dev); - ct_idle_enter(); + ct_cpuidle_enter(); state = psci_get_domain_state(); if (!state) @@ -82,7 +82,7 @@ static int __psci_enter_domain_idle_stat ret = psci_cpu_suspend_enter(state) ? -1 : idx; - ct_idle_exit(); + ct_cpuidle_exit(); if (s2idle) dev_pm_genpd_resume(pd_dev); --- a/drivers/cpuidle/cpuidle-riscv-sbi.c +++ b/drivers/cpuidle/cpuidle-riscv-sbi.c @@ -126,7 +126,7 @@ static int __sbi_enter_domain_idle_state else pm_runtime_put_sync_suspend(pd_dev); - ct_idle_enter(); + ct_cpuidle_enter(); if (sbi_is_domain_state_available()) state = sbi_get_domain_state(); @@ -135,7 +135,7 @@ static int __sbi_enter_domain_idle_state ret = sbi_suspend(state) ? -1 : idx; - ct_idle_exit(); + ct_cpuidle_exit(); if (s2idle) dev_pm_genpd_resume(pd_dev); --- a/drivers/cpuidle/cpuidle-tegra.c +++ b/drivers/cpuidle/cpuidle-tegra.c @@ -183,7 +183,7 @@ static int tegra_cpuidle_state_enter(str tegra_pm_set_cpu_in_lp2(); cpu_pm_enter(); - ct_idle_enter(); + ct_cpuidle_enter(); switch (index) { case TEGRA_C7: @@ -199,7 +199,7 @@ static int tegra_cpuidle_state_enter(str break; } - ct_idle_exit(); + ct_cpuidle_exit(); cpu_pm_exit(); tegra_pm_clear_cpu_in_lp2(); @@ -240,10 +240,10 @@ static int tegra_cpuidle_enter(struct cp if (index == TEGRA_C1) { if (do_rcu) - ct_idle_enter(); + ct_cpuidle_enter(); ret = arm_cpuidle_simple_enter(dev, drv, index); if (do_rcu) - ct_idle_exit(); + ct_cpuidle_exit(); } else ret = tegra_cpuidle_state_enter(dev, index, cpu); --- a/drivers/cpuidle/cpuidle.c +++ b/drivers/cpuidle/cpuidle.c @@ -14,6 +14,7 @@ #include #include #include +#include #include #include #include @@ -152,12 +153,12 @@ static void enter_s2idle_proper(struct c */ stop_critical_timings(); if (!(target_state->flags & CPUIDLE_FLAG_RCU_IDLE)) - ct_idle_enter(); + ct_cpuidle_enter(); target_state->enter_s2idle(dev, drv, index); if (WARN_ON_ONCE(!irqs_disabled())) - local_irq_disable(); + raw_local_irq_disable(); if (!(target_state->flags & CPUIDLE_FLAG_RCU_IDLE)) - ct_idle_exit(); + ct_cpuidle_exit(); tick_unfreeze(); start_critical_timings(); @@ -235,14 +236,14 @@ int cpuidle_enter_state(struct cpuidle_d stop_critical_timings(); if (!(target_state->flags & CPUIDLE_FLAG_RCU_IDLE)) - ct_idle_enter(); + ct_cpuidle_enter(); entered_state = target_state->enter(dev, drv, index); if (WARN_ONCE(!irqs_disabled(), "%ps leaked IRQ state", target_state->enter)) raw_local_irq_disable(); if (!(target_state->flags & CPUIDLE_FLAG_RCU_IDLE)) - ct_idle_exit(); + ct_cpuidle_exit(); start_critical_timings(); sched_clock_idle_wakeup_event(); --- a/include/linux/clockchips.h +++ b/include/linux/clockchips.h @@ -211,7 +211,7 @@ extern int tick_receive_broadcast(void); extern void tick_setup_hrtimer_broadcast(void); extern int tick_check_broadcast_expired(void); # else -static inline int tick_check_broadcast_expired(void) { return 0; } +static __always_inline int tick_check_broadcast_expired(void) { return 0; } static inline void tick_setup_hrtimer_broadcast(void) { } # endif @@ -219,7 +219,7 @@ static inline void tick_setup_hrtimer_br static inline void clockevents_suspend(void) { } static inline void clockevents_resume(void) { } -static inline int tick_check_broadcast_expired(void) { return 0; } +static __always_inline int tick_check_broadcast_expired(void) { return 0; } static inline void tick_setup_hrtimer_broadcast(void) { } #endif /* !CONFIG_GENERIC_CLOCKEVENTS */ --- a/include/linux/cpuidle.h +++ b/include/linux/cpuidle.h @@ -14,6 +14,7 @@ #include #include #include +#include #define CPUIDLE_STATE_MAX 10 #define CPUIDLE_NAME_LEN 16 @@ -115,6 +116,35 @@ struct cpuidle_device { DECLARE_PER_CPU(struct cpuidle_device *, cpuidle_devices); DECLARE_PER_CPU(struct cpuidle_device, cpuidle_dev); +static __always_inline void ct_cpuidle_enter(void) +{ + lockdep_assert_irqs_disabled(); + /* + * Idle is allowed to (temporary) enable IRQs. It + * will return with IRQs disabled. + * + * Trace IRQs enable here, then switch off RCU, and have + * arch_cpu_idle() use raw_local_irq_enable(). Note that + * ct_idle_enter() relies on lockdep IRQ state, so switch that + * last -- this is very similar to the entry code. + */ + trace_hardirqs_on_prepare(); + lockdep_hardirqs_on_prepare(); + instrumentation_end(); + ct_idle_enter(); + lockdep_hardirqs_on(_RET_IP_); +} + +static __always_inline void ct_cpuidle_exit(void) +{ + /* + * Carefully undo the above. + */ + lockdep_hardirqs_off(_RET_IP_); + ct_idle_exit(); + instrumentation_begin(); +} + /**************************** * CPUIDLE DRIVER INTERFACE * ****************************/ @@ -289,9 +319,9 @@ extern s64 cpuidle_governor_latency_req( if (!is_retention) \ __ret = cpu_pm_enter(); \ if (!__ret) { \ - ct_idle_enter(); \ + ct_cpuidle_enter(); \ __ret = low_level_idle_enter(state); \ - ct_idle_exit(); \ + ct_cpuidle_exit(); \ if (!is_retention) \ cpu_pm_exit(); \ } \ --- a/kernel/sched/idle.c +++ b/kernel/sched/idle.c @@ -51,18 +51,22 @@ __setup("hlt", cpu_idle_nopoll_setup); static noinline int __cpuidle cpu_idle_poll(void) { + instrumentation_begin(); trace_cpu_idle(0, smp_processor_id()); stop_critical_timings(); - ct_idle_enter(); - local_irq_enable(); + ct_cpuidle_enter(); + raw_local_irq_enable(); while (!tif_need_resched() && (cpu_idle_force_poll || tick_check_broadcast_expired())) cpu_relax(); + raw_local_irq_disable(); - ct_idle_exit(); + ct_cpuidle_exit(); start_critical_timings(); trace_cpu_idle(PWR_EVENT_EXIT, smp_processor_id()); + local_irq_enable(); + instrumentation_end(); return 1; } @@ -85,44 +89,21 @@ void __weak arch_cpu_idle(void) */ void __cpuidle default_idle_call(void) { - if (current_clr_polling_and_test()) { - local_irq_enable(); - } else { - + instrumentation_begin(); + if (!current_clr_polling_and_test()) { trace_cpu_idle(1, smp_processor_id()); stop_critical_timings(); - /* - * arch_cpu_idle() is supposed to enable IRQs, however - * we can't do that because of RCU and tracing. - * - * Trace IRQs enable here, then switch off RCU, and have - * arch_cpu_idle() use raw_local_irq_enable(). Note that - * ct_idle_enter() relies on lockdep IRQ state, so switch that - * last -- this is very similar to the entry code. - */ - trace_hardirqs_on_prepare(); - lockdep_hardirqs_on_prepare(); - ct_idle_enter(); - lockdep_hardirqs_on(_THIS_IP_); - + ct_cpuidle_enter(); arch_cpu_idle(); - - /* - * OK, so IRQs are enabled here, but RCU needs them disabled to - * turn itself back on.. funny thing is that disabling IRQs - * will cause tracing, which needs RCU. Jump through hoops to - * make it 'work'. - */ raw_local_irq_disable(); - lockdep_hardirqs_off(_THIS_IP_); - ct_idle_exit(); - lockdep_hardirqs_on(_THIS_IP_); - raw_local_irq_enable(); + ct_cpuidle_exit(); start_critical_timings(); trace_cpu_idle(PWR_EVENT_EXIT, smp_processor_id()); } + local_irq_enable(); + instrumentation_end(); } static int call_cpuidle_s2idle(struct cpuidle_driver *drv, --- a/kernel/time/tick-broadcast.c +++ b/kernel/time/tick-broadcast.c @@ -622,9 +622,13 @@ struct cpumask *tick_get_broadcast_onesh * to avoid a deep idle transition as we are about to get the * broadcast IPI right away. */ -int tick_check_broadcast_expired(void) +noinstr int tick_check_broadcast_expired(void) { +#ifdef _ASM_GENERIC_BITOPS_INSTRUMENTED_NON_ATOMIC_H + return arch_test_bit(smp_processor_id(), cpumask_bits(tick_broadcast_force_mask)); +#else return cpumask_test_cpu(smp_processor_id(), tick_broadcast_force_mask); +#endif } /*