From patchwork Thu Jan 12 19:44:01 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Zijlstra X-Patchwork-Id: 13098781 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A705AC61DB3 for ; Thu, 12 Jan 2023 19:58:21 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 842A48001C; Thu, 12 Jan 2023 14:57:43 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 77F6A8001B; Thu, 12 Jan 2023 14:57:43 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3ADF28001C; Thu, 12 Jan 2023 14:57:43 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 0ED208001B for ; Thu, 12 Jan 2023 14:57:43 -0500 (EST) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id E4980C0D34 for ; Thu, 12 Jan 2023 19:57:42 +0000 (UTC) X-FDA: 80347207164.08.5646AF1 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf12.hostedemail.com (Postfix) with ESMTP id 44EA54000B for ; Thu, 12 Jan 2023 19:57:40 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=sAS+cVfJ; spf=none (imf12.hostedemail.com: domain of peterz@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=peterz@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1673553461; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:references:dkim-signature; bh=grsvYP38q4MnN8oBsFSgy8h24Veck5RfPmpq2ugRYPM=; b=YikQY7GLO6fstx3Tv3hNvQXL/cAVQjFh/5j1Z8dzpxpfXf1oRXaKai0LkFG/dHOEHczoYw ItG50RaADfOrTMc63xQBas4jG25qrUyfw+68akBy3YeyqOdNpH7oozV5tp1j+b6clsTQUU U+uA2d1xjltOsUK+TtuF9e6ZK4q+rG4= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=sAS+cVfJ; spf=none (imf12.hostedemail.com: domain of peterz@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=peterz@infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1673553461; a=rsa-sha256; cv=none; b=HQYTXrxyQvVmC0TDCAF38YaG+KGaEwrDdjHwf4N1fIcKaNgexYMdsuepTR/TqHEmdNUhyh FMXrKVZIW8FYosWRUYSxI9qGQ9r0j2YazvNtp78Gkz3AHQT4ME8BS0SR5A/aF38KYO+4qS 0COtvrHEKZx42IMUwgxi4sy6JWekh9c= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=grsvYP38q4MnN8oBsFSgy8h24Veck5RfPmpq2ugRYPM=; b=sAS+cVfJZnxSjVZ+cWnQizLCZi uzGVUsOf3anafnENfDcz5DBxNwmF2u/05pvc6A0c+OdcaX2m7Qx3mQCWD2r8+RXAQmr6AABwQQUqG Xnn1+diWHWxMGA10WHWQkPrlviiLNwKR8l/igNBPuc/Urss/XPLwrgEzGEkTLbJF2i3zM2wEUdQId iXvs8/Thb5QmoGtDr5BpPsofdym80GCpPly/+P1kmFfpKKXI1eKi4k9Kt5ZS1McOXBTe8WNh0qvx1 Mira4VG3zNGY/eh9MdDKDjG1vDtIkJ7jl+e56WDxEl8scNd8fDZtJTeB77Tp7XyQzYbmD4z6YExxk sfEhC80A==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1pG3hs-005P5q-UH; Thu, 12 Jan 2023 19:57:49 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 3351230347B; Thu, 12 Jan 2023 20:57:14 +0100 (CET) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id 6E8FA2CD066EC; Thu, 12 Jan 2023 20:57:08 +0100 (CET) Message-ID: <20230112195542.212914195@infradead.org> User-Agent: quilt/0.66 Date: Thu, 12 Jan 2023 20:44:01 +0100 From: Peter Zijlstra To: peterz@infradead.org Cc: richard.henderson@linaro.org, ink@jurassic.park.msu.ru, mattst88@gmail.com, vgupta@kernel.org, linux@armlinux.org.uk, nsekhar@ti.com, brgl@bgdev.pl, ulli.kroll@googlemail.com, linus.walleij@linaro.org, shawnguo@kernel.org, Sascha Hauer , kernel@pengutronix.de, festevam@gmail.com, linux-imx@nxp.com, tony@atomide.com, khilman@kernel.org, krzysztof.kozlowski@linaro.org, alim.akhtar@samsung.com, catalin.marinas@arm.com, will@kernel.org, guoren@kernel.org, bcain@quicinc.com, chenhuacai@kernel.org, kernel@xen0n.name, geert@linux-m68k.org, sammy@sammy.net, monstr@monstr.eu, tsbogend@alpha.franken.de, dinguyen@kernel.org, jonas@southpole.se, stefan.kristiansson@saunalahti.fi, shorne@gmail.com, James.Bottomley@HansenPartnership.com, deller@gmx.de, mpe@ellerman.id.au, npiggin@gmail.com, christophe.leroy@csgroup.eu, paul.walmsley@sifive.com, palmer@dabbelt.com, aou@eecs.berkeley.edu, hca@linux.ibm.com, gor@linux.ibm.com, agordeev@linux.ibm.com, borntraeger@linux.ibm.com, svens@linux.ibm.com, ysato@users.sourceforge.jp, dalias@libc.org, davem@davemloft.net, richard@nod.at, anton.ivanov@cambridgegreys.com, johannes@sipsolutions.net, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, x86@kernel.org, hpa@zytor.com, acme@kernel.org, mark.rutland@arm.com, alexander.shishkin@linux.intel.com, jolsa@kernel.org, namhyung@kernel.org, jgross@suse.com, srivatsa@csail.mit.edu, amakhalov@vmware.com, pv-drivers@vmware.com, boris.ostrovsky@oracle.com, chris@zankel.net, jcmvbkbc@gmail.com, rafael@kernel.org, lenb@kernel.org, pavel@ucw.cz, gregkh@linuxfoundation.org, mturquette@baylibre.com, sboyd@kernel.org, daniel.lezcano@linaro.org, lpieralisi@kernel.org, sudeep.holla@arm.com, agross@kernel.org, andersson@kernel.org, konrad.dybcio@linaro.org, anup@brainfault.org, thierry.reding@gmail.com, jonathanh@nvidia.com, jacob.jun.pan@linux.intel.com, atishp@atishpatra.org, Arnd Bergmann , yury.norov@gmail.com, andriy.shevchenko@linux.intel.com, linux@rasmusvillemoes.dk, dennis@kernel.org, tj@kernel.org, cl@linux.com, rostedt@goodmis.org, mhiramat@kernel.org, frederic@kernel.org, paulmck@kernel.org, pmladek@suse.com, senozhatsky@chromium.org, john.ogness@linutronix.de, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, ryabinin.a.a@gmail.com, glider@google.com, andreyknvl@gmail.com, dvyukov@google.com, vincenzo.frascino@arm.com, Andrew Morton , jpoimboe@kernel.org, linux-alpha@vger.kernel.org, linux-kernel@vger.kernel.org, linux-snps-arc@lists.infradead.org, linux-omap@vger.kernel.org, linux-samsung-soc@vger.kernel.org, linux-csky@vger.kernel.org, linux-hexagon@vger.kernel.org, linux-ia64@vger.kernel.org, loongarch@lists.linux.dev, linux-m68k@lists.linux-m68k.org, linux-mips@vger.kernel.org, openrisc@lists.librecores.org, linux-parisc@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, linux-sh@vger.kernel.org, sparclinux@vger.kernel.org, linux-um@lists.infradead.org, linux-perf-users@vger.kernel.org, virtualization@lists.linux-foundation.org, linux-xtensa@linux-xtensa.org, linux-acpi@vger.kernel.org, linux-pm@vger.kernel.org, linux-clk@vger.kernel.org, linux-arm-msm@vger.kernel.org, linux-tegra@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, linux-trace-kernel@vger.kernel.org, kasan-dev@googlegroups.com Subject: [PATCH v3 47/51] cpuidle: Ensure ct_cpuidle_enter() is always called from noinstr/__cpuidle References: <20230112194314.845371875@infradead.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 44EA54000B X-Stat-Signature: j447zx97gexa1y37zaazw7g4rta5kfx4 X-HE-Tag: 1673553460-160284 X-HE-Meta: U2FsdGVkX18UQXb2N3bfjJ6EjJaJeK5aTqdjxcr+IayIJuh2tk+dgWReuhACV42a16kSniFOzZtzDLfFRwz+VlL6y0YVJWH+wQNbuXECit2HFV4JXV/L2inSOGTm1iYoMLyhoSsWmK1MQswebJIgEHn842l6IpJZo0bSzpZSYR0Jj/PElbVBZjG9C7LAhuAETXza5tJ3nnmI8oXpSfyXZfKSIA96Nw0ZKRvQGKnSAdygZZU62G46BBm2UwqNbDDfXJfkJVzhoAHO0QiEIlPsXopIgKK/6ej4ymCHGBk5wPin9jwPFd/vtSdWUDoZhv7yPqrYGZ1ASOdcA77wK7CygzecjAsetNl6KCJbbFpIfeFX6YyUnvmJyNXebu/0s8+NhIDgdHIuYnBTxIIeJjOqwnCM+Oru7+1Ri3qD/68++p9kROa7BR+9PVys+fGO0BD47HBvaT56+gyLCVHKCmgykaDbJy3xbm/eWldV6+IrRlG4TcCp/lerV2jwS96E8GXbgLO2o0+lgrJFSjtv9pejwkztr5AG0Bt3lGMYYzpKBkeHxQpx6W2sQsp86aLx+y6VG15uDhxUZgCBkCu7woMPPjJa2RudMJb6iWTnE9CeVqurrVY7GEWuLP6kB96tjdtZZt5HvSB1PE+NGgzDysIwc6iRm6H88I6SHJGNq5Dw47GcRDvfMMP53e1IeI3hod2oHNH4sm/G63yPa/5RBQ4TrJEVHNQHzbSdT5m5QxNLrzOVxwHG+EHHgQRrsYRrYYY3DbpbxgMq2iMQFEGb7OQHNpLEgPDSAAS9HdInPvgse9YgpVUDXks0K/iD7dqZkIHDuQFTlCUHXmbGR+XLVq/iOxoNz7yoP/8dR8TZM6GKqGULYJMsjdbgKqujKE9KYJ11b3DwZcB1i+H7assZlCuX5VbkHMqLpp6cVM+3nBHY8kHm30rLx+YdD6ZcUphtW7f8p6Ehl0hB6Oc2F0L9nce zJkKDg/5 fsFvQ3Vf/RNFnChZn5rtC7aJirDuWnw8ebDDeDunPGAz7ijifQL1BK9NiM+UCnNZ9xOJDqmLCb9IIpdskyRgdXU1qtuaEhaL/F9i+UWOh8ZP1ceO1SZLl3p8bIlQGzqNR6H7xyyCTLjtnP2SRa9cB+uoU/VQ/uy//bXJigocLt4ywuV+VAxU69iHupU3ZC2b1kPlDvkzODCdrDfAvC1MtpdMh7+F9pW9LUS6fPLXef5LCRYZiwDCzdKgmpGajVQYdRAXnouLFmBJr6CiF5BhTMGHqN79Y5Adcv79+PDYjSzMU0FCrmkaI99fjXy1N+o4jcqLBSXmqpSWAm+HOn843x4OQDFGkKO5Bk4AzuIDMgtgiuQ2cJhCVu3SJMkeMLZwkfIUapClwGfOrRqnOasGOSWOjuGBpTkzTDSxHoKkcD2/Hkpa93uWvkyWiS1Z+h5XmcEr4XguxxI1B2Rcl5DOS0/L3L5OMRp4RQgq4Xr6bYyRaBa/7aOu9rzrY4ga0hrWZF+6yuijW0bHZR/mFGP0rnfepmRGuiwH9dRcAZMZ8AosTI47B9jxDYsU01gbOVrg0o79mT4cMYSfXB/QfpoATDDSvUZL27vrkZZE+9K2I6sX4Zxf6EsdFh4pjmM8Kb+HLgxmFxiMIqHcqaAgJpc9evwlCGlCY+JdYvrteXViyldi2HlkqT9tPnT80CdnZ3DNUB6at5yxWouLolKKSKrTljDdTZezQ3AQwhGGYDKlWNbDVwn43HZOR1RIE2bP19NSQR6j2lq/ULADB6kj4+0KUvkYRLtSdWs1S9Fv5X/C7IWWrc3QyAdTO01ZicDNDxMkrUdAMh+LeO+LmohmEiG14nv9Jx/esmaasxWebJyd7SNSr90LhXj5DttLSeTsT+8ny1zmpdX6ATMCrN3lwxF8qqK5FpuV81rXSAIiG9/d5igdBhK1ss7dw1HLnCMEzEcxNLvRyl8/7JnJmzJ009zqI7PUUnopZ Moi0YPkW 89UXFurp8Q3O3XP20Mc6yXdmJG+38JAh5HIJF9jWDbsAOak66gOVY6gdN1nTe1qWzvlR+xIirkkk9aC+tMgaKg9Sgrq0hSFbGZdxpXwdFpbMs38UU5ce6j5xIcgRf6CBUzc8It/urr9+Ky6Sw8r6U8EUIvZbarWTgAuHlEWWpmW9SpoLq1LUUpGjQMJSqh+FMO1K/Jc+6U2W3/sV9BHPuwuj3RAFAUTee5DbC95yn0DYXVz7QPE17qpb+Vl/yUGKL2gEZo/HcX0I0d+5uVkI8bmPY4MtnxN7HTCgYD6JSRSp+PFCkYWyb9ML5ovb/QNP63fKvNgZ52vLlO8ROMrdxJEwsA/6+8l2UpXUyYLmRC6dwTfkQeNiwtkXPpmuM9D6gJnDySveVY4dsSSM/mAvD/hmR6cLhHXjZZc1QHg8uROvCEXp6/8Nm3t24oCVl11kC80TEodhSOIEqR/YbQmU9NPANw5B3fTaJzjXBatl5gHN1IvMF/kXQlvVmnOLPep+MpHqpS6yB9ORpIY0gKA31lHzdG8yXsnUiKPF4O+6SvB6N36ZfmJY/7zUloP9u0eouycAi7iIMSjiBs6Bz5yQsGzrB55TC62Vkj1JZ7JT0xXZHYpYLnLvrlH7RBagvAaZLovt16a2ADroCD7slOl/Or+jU7obWhpr8wKUSoxwo0xUHSDkt5j8W6nbjlDsBNf7UHD0IjYwzsv/oYbXwLCa2E7YvCNIM6ufLksK5f58L6Z75J76hdsO0nV2Fm1PGbEZAB7ahe/pABlZMq7BXKjJl+JiGl2In+fHEHPXS1psTTeKKSuy3erEMobF8MMxDM2bz+z5HBUfg/79ucoqaZQC63cVrJCinxYZG51B8v1Qfh5Wy8L9TdC18MmsLJf0NXHUgPeSgtLpsIGI7hSgLCxSNN2NOSQG6Gr1A5eI80NaaC/OqSRcLRpsCY19DllM2pO0EJAz2ZKtkMRkqOqm38I5KfclFeEK5 c0cghqCY xDHdjC1v5AO8p3+K9/XS6C2NFdHVOX9E2ZKeINqsHufbSZt+4ZpA/pw3uvrxNlYSVZnxWcqbPF9GxV8DO1pm8GIB2YVGpTWpXG19KZZyvctdHaxqdPLdVEv7wiTjtVY08QCLUqZeSIId3lfsg5KE+2Key9G4/e4e4VgvHNmsyt6E9M4MlOtM8KrW+YHx2XFXffpsKSh1neYHV/OEL/IvxpBGlo2SwmKuTgF3W+p2Or44iRqfaZaROuyGxrxbu3I81rvcOrNNAkypfL+tZf4MZ1bnN10bhtRiMH1kQU7uHbwRl3IHOKZ6J4H1/qheBGd2IFFtFvm1hmwD+yRPYjMs/c0yW2QHjOtpbBVm+cev5Cj0n/U0/QRs+3KDZIRODyp+G2XDM7qx+BwYUQyDew/4OgOFiDVe2JkE39Crz9KrST3uurJacvv8MAh5dgIneSIAlkwoc/mzaGGGKQYsH3BLTC4adHwiGcNdH6n+9Jvpty0vWP9d87aImJaDKxzuO6seutgdZt8c2NfUWUpXSxnAb2ao3WLP8q/sbj+JQYxDO8gSYyzdi0NQydZAldKpnLC3623VlR9kbnNj5GPa7ngRsFVbqUuSWqycoN/JKoTHktope8N+BLYpqkC1hx5K/r9x5ZDPNJDYAzFq9hLS1C3HmuVshMq616qjD10GASWRd40mPzV/lTXa4TgRS2TigGm5rNLhyxSlK1bNR2B1nVvxQjm/WfTZhmN9k7YktOlguJ95KWoxyWLy+bWXDbtU9WnFEkuf+U1Wjq0rJq25JuU/SVRWjl6OWZ3b5rorqGiP3jQHYYWUm3mDMziVHXZD4NhLIeogiyv6eCjobMgb9CBhpd1u4UXc5vCzr4YIB2D7wSoEQESbcTswtG3Fe6iZ04285PR+gtBbMFQ+YYIzkIJf8aWAKqm0dtxnDIzSmqlnfhDNhb6rF871tvuPoZG/Ca8JG678pmQv5eH3/bjV4rHv8Nqt9J3U5 0gmf0Gky SRr50s5DzuzBouZnMUzUNB0A+gPKOYenrQOG1uM4UORJwHUNA7ouA0JVl7ocHS7jAGpDSt/EC60Ha7IbcAfNsQHaku2s4hJZ+yz9VoabCmKBZLrUhMPI7IkD9Gf2phc8KW6iYwVrin4MbpM5jxh0hW X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Tracing (kprobes included) and other compiler instrumentation relies on a normal kernel runtime. Therefore all functions that disable RCU should be noinstr, as should all functions that are called while RCU is disabled. Signed-off-by: Peter Zijlstra (Intel) --- drivers/cpuidle/cpuidle.c | 37 ++++++++++++++++++++++++++++--------- 1 file changed, 28 insertions(+), 9 deletions(-) --- a/drivers/cpuidle/cpuidle.c +++ b/drivers/cpuidle/cpuidle.c @@ -137,11 +137,13 @@ int cpuidle_find_deepest_state(struct cp } #ifdef CONFIG_SUSPEND -static void enter_s2idle_proper(struct cpuidle_driver *drv, - struct cpuidle_device *dev, int index) +static noinstr void enter_s2idle_proper(struct cpuidle_driver *drv, + struct cpuidle_device *dev, int index) { - ktime_t time_start, time_end; struct cpuidle_state *target_state = &drv->states[index]; + ktime_t time_start, time_end; + + instrumentation_begin(); time_start = ns_to_ktime(local_clock()); @@ -152,13 +154,18 @@ static void enter_s2idle_proper(struct c * suspended is generally unsafe. */ stop_critical_timings(); - if (!(target_state->flags & CPUIDLE_FLAG_RCU_IDLE)) + if (!(target_state->flags & CPUIDLE_FLAG_RCU_IDLE)) { ct_cpuidle_enter(); + /* Annotate away the indirect call */ + instrumentation_begin(); + } target_state->enter_s2idle(dev, drv, index); if (WARN_ON_ONCE(!irqs_disabled())) raw_local_irq_disable(); - if (!(target_state->flags & CPUIDLE_FLAG_RCU_IDLE)) + if (!(target_state->flags & CPUIDLE_FLAG_RCU_IDLE)) { + instrumentation_end(); ct_cpuidle_exit(); + } tick_unfreeze(); start_critical_timings(); @@ -166,6 +173,7 @@ static void enter_s2idle_proper(struct c dev->states_usage[index].s2idle_time += ktime_us_delta(time_end, time_start); dev->states_usage[index].s2idle_usage++; + instrumentation_end(); } /** @@ -200,8 +208,9 @@ int cpuidle_enter_s2idle(struct cpuidle_ * @drv: cpuidle driver for this cpu * @index: index into the states table in @drv of the state to enter */ -int cpuidle_enter_state(struct cpuidle_device *dev, struct cpuidle_driver *drv, - int index) +noinstr int cpuidle_enter_state(struct cpuidle_device *dev, + struct cpuidle_driver *drv, + int index) { int entered_state; @@ -209,6 +218,8 @@ int cpuidle_enter_state(struct cpuidle_d bool broadcast = !!(target_state->flags & CPUIDLE_FLAG_TIMER_STOP); ktime_t time_start, time_end; + instrumentation_begin(); + /* * Tell the time framework to switch to a broadcast timer because our * local timer will be shut down. If a local timer is used from another @@ -235,15 +246,21 @@ int cpuidle_enter_state(struct cpuidle_d time_start = ns_to_ktime(local_clock()); stop_critical_timings(); - if (!(target_state->flags & CPUIDLE_FLAG_RCU_IDLE)) + if (!(target_state->flags & CPUIDLE_FLAG_RCU_IDLE)) { ct_cpuidle_enter(); + /* Annotate away the indirect call */ + instrumentation_begin(); + } entered_state = target_state->enter(dev, drv, index); + if (WARN_ONCE(!irqs_disabled(), "%ps leaked IRQ state", target_state->enter)) raw_local_irq_disable(); - if (!(target_state->flags & CPUIDLE_FLAG_RCU_IDLE)) + if (!(target_state->flags & CPUIDLE_FLAG_RCU_IDLE)) { + instrumentation_end(); ct_cpuidle_exit(); + } start_critical_timings(); sched_clock_idle_wakeup_event(); @@ -306,6 +323,8 @@ int cpuidle_enter_state(struct cpuidle_d dev->states_usage[index].rejected++; } + instrumentation_end(); + return entered_state; }