From patchwork Fri Aug 18 18:04:37 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dario Faggioli X-Patchwork-Id: 9909701 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 07D7060385 for ; Fri, 18 Aug 2017 18:06:47 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0094528CFF for ; Fri, 18 Aug 2017 18:06:47 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id E948F28D18; Fri, 18 Aug 2017 18:06:46 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.6 required=2.0 tests=BAYES_00,DKIM_SIGNED, RCVD_IN_DNSWL_MED,RCVD_IN_SORBS_SPAM,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 1CD7228D1D for ; Fri, 18 Aug 2017 18:06:46 +0000 (UTC) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dildS-0002L3-Vd; Fri, 18 Aug 2017 18:04:42 +0000 Received: from mail6.bemta6.messagelabs.com ([193.109.254.103]) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dildS-0002Kj-1L for xen-devel@lists.xenproject.org; Fri, 18 Aug 2017 18:04:42 +0000 Received: from [85.158.143.35] by server-3.bemta-6.messagelabs.com id 1D/D5-03044-93C27995; Fri, 18 Aug 2017 18:04:41 +0000 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFvrIIsWRWlGSWpSXmKPExsXiVRvkomuhMz3 SYONyZYvvWyYzOTB6HP5whSWAMYo1My8pvyKBNeP3817Wgn92FYeO/WRtYFxu3MXIySEkMINR 4s/buC5GLg4WgTWsEle6trODJCQELrFK/D6bBmHHSSxumM8MYVdKrF/4mAmiWUXi5vZVTCDNQ gI/GCU6z6wASwgL6EkcOfqDHcIOkbh/bzkbiM0mYCDxZsdeVhBbREBJ4t6qyWDNzAKTmCWm33 jKCJJgEVCVWNH8AGwbr4CPxNQj38EaOIHsFxdPsEJs9pZ493sFC4gtKiAnsfJyCytEvaDEyZl PgOIcQEM1Jdbv0gcJMwvIS2x/O4d5AqPILCRVsxCqZiGpWsDIvIpRvTi1qCy1SNdSL6koMz2j JDcxM0fX0MBMLze1uDgxPTUnMalYLzk/dxMjMPgZgGAH491NAYcYJTmYlER5f8+aEinEl5SfU pmRWJwRX1Sak1p8iFGGg0NJgjdaa3qkkGBRanpqRVpmDjAOYdISHDxKIrwrQNK8xQWJucWZ6R CpU4y6HJMObP/CJMSSl5+XKiXOWwJSJABSlFGaBzcClhIuMcpKCfMyAh0lxFOQWpSbWYIq/4p RnINRSZj3IMgUnsy8ErhNr4COYAI6wrB1GsgRJYkIKakGxhCVsyt+LN70dF9pzVbH3iOvf6+p WlH+N5+Rf/Hhqg3ha//n2kQ+P7Fxxc3q37X7t8lPNnwdGV89j5XV0T9xivHcAtFtzKyucQ8+K H9btzrRYh9D8teAV+kzJjnc3iMYkKHidLjH79/2hw+cF0apb71Zb2n4wmat0v0r8y7Gzvmaee zYp+YVs6crsRRnJBpqMRcVJwIAfZGGRgQDAAA= X-Env-Sender: raistlin.df@gmail.com X-Msg-Ref: server-2.tower-21.messagelabs.com!1503079480!64636869!1 X-Originating-IP: [74.125.82.68] X-SpamReason: No, hits=0.0 required=7.0 tests= X-StarScan-Received: X-StarScan-Version: 9.4.45; banners=-,-,- X-VirusChecked: Checked Received: (qmail 19451 invoked from network); 18 Aug 2017 18:04:40 -0000 Received: from mail-wm0-f68.google.com (HELO mail-wm0-f68.google.com) (74.125.82.68) by server-2.tower-21.messagelabs.com with AES128-GCM-SHA256 encrypted SMTP; 18 Aug 2017 18:04:40 -0000 Received: by mail-wm0-f68.google.com with SMTP id t138so4986810wmt.4 for ; Fri, 18 Aug 2017 11:04:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:subject:from:to:cc:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=vRKfM8NL3GQxW/KWCMgsR6jzXn7dIF9SaSh/ar3Cm3s=; b=fGoRXJZGQMl2tQoCEXcXskTbpe2QhSxU0SFzBE+FpXOAI271iKaPVrOysQPvbfcUtK r5U1S7lk6fQhnQKB7qSIRr/AlFoOqm6vrQ7WiiS7Y8nJO0H9AeNbRMt0GD9kpTo3wYyf X1BAWOnBB+RneMnw/u+vCkikJoaur7lbtGuJdJItr5pT7ZkWNeQNSaLJmPlQ1zzjGdKU 8qRL1+36b7ae0HIC/JsfukfsABqP6K4Zk8Bdt3KvIV2DLVSURSSAhnx0FRaiFuOJa1qD FpetL92ASvJ8VuPqRUBM5yP6mXzCwjZt/2cKWDFHOvr5WxSsnzfQIuHnPimJvBjO8XlD ZuVw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:subject:from:to:cc:date:message-id :in-reply-to:references:user-agent:mime-version :content-transfer-encoding; bh=vRKfM8NL3GQxW/KWCMgsR6jzXn7dIF9SaSh/ar3Cm3s=; b=Zlm1/RYyVRXu+A5adOT2b6CylJt/+dar6BWvJakaNB3Cm/9NSfvSR6X6IeJ0NRvpPG OZhbCOwUG8xbUPPjEInRm4eiApYuXOtazFdl3wpm5zvxSX4RvQKqkdR+IKnlzf7nEcWx Do6lMGTmLM91HUHauRh3RungkY1c4ac4nrQ9cgHLxxx4HGSNBme8voXjuUoBg/GO5EFY GEGBbVb7CiXQ2yHWeLcSVo4iTtbZCrA33SSo2Tk9VfjzJ4mYSEdzYseZ87sRDOEz5YFS 9lRe/JQZdaPmBcnCZkB0EtRI8pXQaj3BnzpoxFKgJ01BYM1QZanFRYNOTZ5bBE71Ftba a+aA== X-Gm-Message-State: AHYfb5id3Tz3XvYVcfhtCHtlYSNglq89hSRo+Dg4Qeu0uFY1aNdDu0RB g9EhjoujghDKpg== X-Received: by 10.28.189.68 with SMTP id n65mr1583727wmf.142.1503079480182; Fri, 18 Aug 2017 11:04:40 -0700 (PDT) Received: from Solace.fritz.box ([80.66.223.3]) by smtp.gmail.com with ESMTPSA id y36sm8227349wrb.53.2017.08.18.11.04.38 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 18 Aug 2017 11:04:39 -0700 (PDT) From: Dario Faggioli To: xen-devel@lists.xenproject.org Date: Fri, 18 Aug 2017 20:04:37 +0200 Message-ID: <150307947767.29525.16150424729950084786.stgit@Solace.fritz.box> In-Reply-To: <150307710991.29525.3681195976643263117.stgit@Solace.fritz.box> References: <150307710991.29525.3681195976643263117.stgit@Solace.fritz.box> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 Cc: Stefano Stabellini , Wei Liu , George Dunlap , Andrew Cooper , Ian Jackson , Tim Deegan , Julien Grall , Jan Beulich Subject: [Xen-devel] [PATCH v3 5/6] xen: RCU: avoid busy waiting until the end of grace period. X-BeenThere: xen-devel@lists.xen.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xen.org Sender: "Xen-devel" X-Virus-Scanned: ClamAV using ClamSMTP On the CPU where a callback is queued, cpu_is_haltable() returns false (due to rcu_needs_cpu() being itself false). That means the CPU would spin inside idle_loop(), continuously calling do_softirq(), and, in there, continuously checking rcu_pending(), in a tight loop. Let's instead allow the CPU to really go idle, but make sure, by arming a timer, that we periodically check whether the grace period has come to an ended. As the period of the timer, we pick a value that makes thing look like what happens in Linux, with the periodic tick (as this code comes from there). Note that the timer will *only* be armed on CPUs that are going idle while having queued RCU callbacks. On CPUs that don't, there won't be any timer, and their sleep won't be interrupted (and even for CPUs with callbacks, we only expect an handful of wakeups at most, but that depends on the system load, as much as from other things). Signed-off-by: Dario Faggioli Reviewed-by: George Dunlap --- Cc: Jan Beulich Cc: Andrew Cooper Cc: George Dunlap Cc: Ian Jackson Cc: Konrad Rzeszutek Wilk Cc: Stefano Stabellini Cc: Tim Deegan Cc: Wei Liu Cc: George Dunlap Cc: Julien Grall --- Changes from v1: * clarified changelog; * fix style/indentation issues; * deal with RCU idle timer in tick suspension logic; * as a consequence of the point above, the timer now fires, so kill the ASSERT_UNREACHABLE, and put a perfcounter there (to count the times it triggers); * add a comment about the value chosen for programming the idle timer; * avoid pointless/bogus '!!' and void* casts; * rearrange the rcu_needs_cpu() return condition; * add a comment to clarify why we don't want to check rcu_pending() in rcu_idle_timer_start(). --- xen/arch/x86/cpu/mwait-idle.c | 3 +- xen/common/rcupdate.c | 72 ++++++++++++++++++++++++++++++++++++++++- xen/common/schedule.c | 2 + xen/include/xen/perfc_defn.h | 2 + xen/include/xen/rcupdate.h | 3 ++ 5 files changed, 79 insertions(+), 3 deletions(-) diff --git a/xen/arch/x86/cpu/mwait-idle.c b/xen/arch/x86/cpu/mwait-idle.c index 762dff1..b6770ea 100644 --- a/xen/arch/x86/cpu/mwait-idle.c +++ b/xen/arch/x86/cpu/mwait-idle.c @@ -741,9 +741,8 @@ static void mwait_idle(void) } cpufreq_dbs_timer_suspend(); - sched_tick_suspend(); - /* sched_tick_suspend() can raise TIMER_SOFTIRQ. Process it now. */ + /* Timer related operations can raise TIMER_SOFTIRQ. Process it now. */ process_pending_softirqs(); /* Interrupts must be disabled for C2 and higher transitions. */ diff --git a/xen/common/rcupdate.c b/xen/common/rcupdate.c index 12ae7da..871936f 100644 --- a/xen/common/rcupdate.c +++ b/xen/common/rcupdate.c @@ -84,8 +84,37 @@ struct rcu_data { int cpu; struct rcu_head barrier; long last_rs_qlen; /* qlen during the last resched */ + + /* 3) idle CPUs handling */ + struct timer idle_timer; + bool idle_timer_active; }; +/* + * If a CPU with RCU callbacks queued goes idle, when the grace period is + * not finished yet, how can we make sure that the callbacks will eventually + * be executed? In Linux (2.6.21, the first "tickless idle" Linux kernel), + * the periodic timer tick would not be stopped for such CPU. Here in Xen, + * we (may) don't even have a periodic timer tick, so we need to use a + * special purpose timer. + * + * Such timer: + * 1) is armed only when a CPU with an RCU callback(s) queued goes idle + * before the end of the current grace period (_not_ for any CPUs that + * go idle!); + * 2) when it fires, it is only re-armed if the grace period is still + * running; + * 3) it is stopped immediately, if the CPU wakes up from idle and + * resumes 'normal' execution. + * + * About how far in the future the timer should be programmed each time, + * it's hard to tell (guess!!). Since this mimics Linux's periodic timer + * tick, take values used there as an indication. In Linux 2.6.21, tick + * period can be 10ms, 4ms, 3.33ms or 1ms. Let's use 10ms, to enable + * at least some power saving on the CPU that is going idle. + */ +#define RCU_IDLE_TIMER_PERIOD MILLISECS(10) + static DEFINE_PER_CPU(struct rcu_data, rcu_data); static int blimit = 10; @@ -404,7 +433,45 @@ int rcu_needs_cpu(int cpu) { struct rcu_data *rdp = &per_cpu(rcu_data, cpu); - return (!!rdp->curlist || rcu_pending(cpu)); + return (rdp->curlist && !rdp->idle_timer_active) || rcu_pending(cpu); +} + +/* + * Timer for making sure the CPU where a callback is queued does + * periodically poke rcu_pedning(), so that it will invoke the callback + * not too late after the end of the grace period. + */ +void rcu_idle_timer_start() +{ + struct rcu_data *rdp = &this_cpu(rcu_data); + + /* + * Note that we don't check rcu_pending() here. In fact, we don't want + * the timer armed on CPUs that are in the process of quiescing while + * going idle, unless they really are the ones with a queued callback. + */ + if (likely(!rdp->curlist)) + return; + + set_timer(&rdp->idle_timer, NOW() + RCU_IDLE_TIMER_PERIOD); + rdp->idle_timer_active = true; +} + +void rcu_idle_timer_stop() +{ + struct rcu_data *rdp = &this_cpu(rcu_data); + + if (likely(!rdp->idle_timer_active)) + return; + + rdp->idle_timer_active = false; + stop_timer(&rdp->idle_timer); +} + +static void rcu_idle_timer_handler(void* data) +{ + /* Nothing, really... Just count the number of times we fire */ + perfc_incr(rcu_idle_timer); } void rcu_check_callbacks(int cpu) @@ -425,6 +492,8 @@ static void rcu_move_batch(struct rcu_data *this_rdp, struct rcu_head *list, static void rcu_offline_cpu(struct rcu_data *this_rdp, struct rcu_ctrlblk *rcp, struct rcu_data *rdp) { + kill_timer(&rdp->idle_timer); + /* If the cpu going offline owns the grace period we can block * indefinitely waiting for it, so flush it here. */ @@ -453,6 +522,7 @@ static void rcu_init_percpu_data(int cpu, struct rcu_ctrlblk *rcp, rdp->qs_pending = 0; rdp->cpu = cpu; rdp->blimit = blimit; + init_timer(&rdp->idle_timer, rcu_idle_timer_handler, rdp, cpu); } static int cpu_callback( diff --git a/xen/common/schedule.c b/xen/common/schedule.c index c6f4817..8827921 100644 --- a/xen/common/schedule.c +++ b/xen/common/schedule.c @@ -1904,6 +1904,7 @@ void sched_tick_suspend(void) sched = per_cpu(scheduler, cpu); SCHED_OP(sched, tick_suspend, cpu); rcu_idle_enter(cpu); + rcu_idle_timer_start(); } void sched_tick_resume(void) @@ -1911,6 +1912,7 @@ void sched_tick_resume(void) struct scheduler *sched; unsigned int cpu = smp_processor_id(); + rcu_idle_timer_stop(); rcu_idle_exit(cpu); sched = per_cpu(scheduler, cpu); SCHED_OP(sched, tick_resume, cpu); diff --git a/xen/include/xen/perfc_defn.h b/xen/include/xen/perfc_defn.h index 53849af..ca446e5 100644 --- a/xen/include/xen/perfc_defn.h +++ b/xen/include/xen/perfc_defn.h @@ -12,6 +12,8 @@ PERFCOUNTER(calls_from_multicall, "calls from multicall") PERFCOUNTER(irqs, "#interrupts") PERFCOUNTER(ipis, "#IPIs") +PERFCOUNTER(rcu_idle_timer, "RCU: idle_timer") + /* Generic scheduler counters (applicable to all schedulers) */ PERFCOUNTER(sched_irq, "sched: timer") PERFCOUNTER(sched_run, "sched: runs through scheduler") diff --git a/xen/include/xen/rcupdate.h b/xen/include/xen/rcupdate.h index 561ac43..3402eb5 100644 --- a/xen/include/xen/rcupdate.h +++ b/xen/include/xen/rcupdate.h @@ -149,4 +149,7 @@ int rcu_barrier(void); void rcu_idle_enter(unsigned int cpu); void rcu_idle_exit(unsigned int cpu); +void rcu_idle_timer_start(void); +void rcu_idle_timer_stop(void); + #endif /* __XEN_RCUPDATE_H */