From patchwork Thu Jul 27 08:01:38 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dario Faggioli X-Patchwork-Id: 9866409 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id C3BCF6038C for ; Thu, 27 Jul 2017 08:03:57 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B508728782 for ; Thu, 27 Jul 2017 08:03:57 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id A9F3D2878C; Thu, 27 Jul 2017 08:03:57 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.1 required=2.0 tests=BAYES_00,DKIM_SIGNED, RCVD_IN_DNSWL_MED,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id EE0A528789 for ; Thu, 27 Jul 2017 08:03:55 +0000 (UTC) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dadjr-0000Ll-Lt; Thu, 27 Jul 2017 08:01:43 +0000 Received: from mail6.bemta5.messagelabs.com ([195.245.231.135]) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dadjq-0000LI-Sm for xen-devel@lists.xenproject.org; Thu, 27 Jul 2017 08:01:42 +0000 Received: from [85.158.139.211] by server-1.bemta-5.messagelabs.com id 83/7A-01993-6ED99795; Thu, 27 Jul 2017 08:01:42 +0000 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFvrJIsWRWlGSWpSXmKPExsVyMbThkO7TuZW RBqt2cVt83zKZyYHR4/CHKywBjFGsmXlJ+RUJrBlzD3ezFzSZVkzYMZ+1gbFBu4uRi0NIYCaj xKIL91lBHBaBNawSM7fMYQFxJAQusUrMPnedrYuRE8iJk1i9ZjMzhF0hsWzBPhYQW0hAReLm9 lVMEKN+Mkp8uPAYrEFYQE/iyNEf7BB2oMTKL98ZQWw2AQOJNzv2soLYIgJKEvdWTQZrZhZYxC hx+cQxsAYWAVWJ04+2gTXwCjhK/PpxBMzmFHAGqtkAtdlJ4vSuVWDLRAXkJFZebmGFqBeUODn zCVANB9BQTYn1u/RBwswC8hLb385hnsAoMgtJ1SyEqllIqhYwMq9iVC9OLSpLLdI100sqykzP KMlNzMzRNTQw1ctNLS5OTE/NSUwq1kvOz93ECIwABiDYwTi1wfkQoyQHk5Io7yTTikghvqT8l MqMxOKM+KLSnNTiQ4wyHBxKErzhcyojhQSLUtNTK9Iyc4CxCJOW4OBREuG1BEnzFhck5hZnpk OkTjEac1y5su4LE8eUA9u/MAmx5OXnpUqJ884DKRUAKc0ozYMbBEsRlxhlpYR5GYFOE+IpSC3 KzSxBlX/FKM7BqCTMuw9kCk9mXgncvldApzABnTKxCeyUkkSElFQDY9Xn5enBf78lTNjDsDzk /br926O3Cxs4rpOalRtWPNF9teXL7wdkcvkTRPKeWsb/2/PGf6t7rIFyh+a+yh9dtRPCGqZ9s E/LY/jUw8T0TWIiu1bIBed57z4ufiTHWHFssYSm9e1HYluV79Xu+a4lwnby0fENt/S7N906dr XGxVa3T+zKNU++6UosxRmJhlrMRcWJAMci/tAMAwAA X-Env-Sender: raistlin.df@gmail.com X-Msg-Ref: server-15.tower-206.messagelabs.com!1501142500!90713692!1 X-Originating-IP: [209.85.128.194] X-SpamReason: No, hits=0.0 required=7.0 tests= X-StarScan-Received: X-StarScan-Version: 9.4.25; banners=-,-,- X-VirusChecked: Checked Received: (qmail 17913 invoked from network); 27 Jul 2017 08:01:41 -0000 Received: from mail-wr0-f194.google.com (HELO mail-wr0-f194.google.com) (209.85.128.194) by server-15.tower-206.messagelabs.com with AES128-GCM-SHA256 encrypted SMTP; 27 Jul 2017 08:01:41 -0000 Received: by mail-wr0-f194.google.com with SMTP id o33so16264652wrb.1 for ; Thu, 27 Jul 2017 01:01:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:subject:from:to:cc:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=M8se8CpyR9hbi4DMgIguznxmKymznBMdj9rOsgqNngo=; b=aTS/Nyph4qm8Xg/GMp4sCyubhgf5YeOAjNxMpozJ554ReuK5DtaFYYk5AKixn0DNLp JpOlPCEc/5amsuxaX7OiupubRpUf2ukyXUyonS4f4VAM3kTSOEEmimDJoXvh+87Ic+k5 ngFG0t5LpCk+QBsAjg7AKf7lfmq8X8IX2C3YAxUs/M25qh+oXM9jVS805N+LyCGpqPuV sfoH5uIuP8YCCiIz9856o6QzkUTY/mai5E/5mG14qrTphU29Fn3rLfwWLRPMgqsoCO/G YCPemBBamxbC7j3IuDaXbRh0cRU/UaoaFTz8ur5irYL4AhKLGrI2tu0IWXtOFpOy9CFX KpdQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:subject:from:to:cc:date:message-id :in-reply-to:references:user-agent:mime-version :content-transfer-encoding; bh=M8se8CpyR9hbi4DMgIguznxmKymznBMdj9rOsgqNngo=; b=tKOyPwQiY9B6/iLvlDnQSi69e6PQy9Ti40IzARlqT+ct7cjXBkU3sB0lSHdZwnsdBW HNgpvobSlMWdOYdiTwKt33wk3uLOUCiRfgkniYuU+ouvGbS0J2QV0uUyzUS3cWjGEPhM aWl+4qWkPMjRcHJe+ZdxPAg5rBMoDzo4rWU5tlK9jfipGVFe28CMIMyxyOcihBynspet Aom5dXe3ARw2qBj94tt3PDjhSiFwE4KZbh4rCZ2eotAhLOIdC1+DjAsq+wEXWjgYtPOc nIKUlNu5FJbmnkPO2kJXJPFp4RVZcFxCzhJJan53kb0LVO325kzoHeG8cESEE7cxl/oW ozUw== X-Gm-Message-State: AIVw113GBku+MpNi7htoV2m59vcxxFJQTJig61Q4WyvHWWg//qDQWldM eM3ieFFqc/urPZ+O X-Received: by 10.223.160.6 with SMTP id k6mr3232183wrk.220.1501142500503; Thu, 27 Jul 2017 01:01:40 -0700 (PDT) Received: from [192.168.0.31] ([80.66.223.212]) by smtp.gmail.com with ESMTPSA id s30sm23787894wrc.13.2017.07.27.01.01.39 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 27 Jul 2017 01:01:39 -0700 (PDT) From: Dario Faggioli To: xen-devel@lists.xenproject.org Date: Thu, 27 Jul 2017 10:01:38 +0200 Message-ID: <150114249858.22910.4601418126082976816.stgit@Solace> In-Reply-To: <150114201043.22910.12807057883146318803.stgit@Solace> References: <150114201043.22910.12807057883146318803.stgit@Solace> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 Cc: Andrew Cooper , Julien Grall , Stefano Stabellini , Jan Beulich Subject: [Xen-devel] [PATCH 5/5] xen: RCU: avoid busy waiting until the end of grace period. X-BeenThere: xen-devel@lists.xen.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xen.org Sender: "Xen-devel" X-Virus-Scanned: ClamAV using ClamSMTP Instead of having the CPU where a callback is queued, busy looping on rcu_pending(), use a timer. In fact, we let the CPU go idla,e but we program a timer that will periodically wake it up, for checking whether the grace period has actually ended. It is kind of similar to introducing a periodic tick, but with a much more limited scope, and a lot less overhead. In fact, this timer is: - only active for the CPU(s) that have callbacks queued, waiting for the end of a grace period; - only active when those CPU(s) are idle (and stopped as soon as they resume execution). Signed-off-by: Dario Faggioli --- Cc: Stefano Stabellini Cc: Julien Grall Cc: Jan Beulich Cc: Andrew Cooper --- xen/arch/arm/domain.c | 4 ++- xen/arch/x86/acpi/cpu_idle.c | 6 +++-- xen/arch/x86/cpu/mwait-idle.c | 6 +++-- xen/common/rcupdate.c | 52 ++++++++++++++++++++++++++++++++++++++++- xen/include/xen/rcupdate.h | 3 ++ 5 files changed, 65 insertions(+), 6 deletions(-) diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c index 666b7ef..01da96e 100644 --- a/xen/arch/arm/domain.c +++ b/xen/arch/arm/domain.c @@ -43,8 +43,9 @@ static void do_idle(void) { unsigned int cpu = smp_processor_id(); + rcu_idle_timer_start(); sched_tick_suspend(); - /* sched_tick_suspend() can raise TIMER_SOFTIRQ. Process it now. */ + /* Timer related operations can raise TIMER_SOFTIRQ. Process it now. */ process_pending_softirqs(); local_irq_disable(); @@ -58,6 +59,7 @@ static void do_idle(void) local_irq_enable(); sched_tick_resume(); + rcu_idle_timer_stop(); } void idle_loop(void) diff --git a/xen/arch/x86/acpi/cpu_idle.c b/xen/arch/x86/acpi/cpu_idle.c index 04c52e8..b97986f 100644 --- a/xen/arch/x86/acpi/cpu_idle.c +++ b/xen/arch/x86/acpi/cpu_idle.c @@ -576,10 +576,10 @@ static void acpi_processor_idle(void) return; } + rcu_idle_timer_start(); cpufreq_dbs_timer_suspend(); - sched_tick_suspend(); - /* sched_tick_suspend() can raise TIMER_SOFTIRQ. Process it now. */ + /* Timer related operations can raise TIMER_SOFTIRQ. Process it now. */ process_pending_softirqs(); /* @@ -593,6 +593,7 @@ static void acpi_processor_idle(void) local_irq_enable(); sched_tick_resume(); cpufreq_dbs_timer_resume(); + rcu_idle_timer_stop(); return; } @@ -726,6 +727,7 @@ static void acpi_processor_idle(void) sched_tick_resume(); cpufreq_dbs_timer_resume(); + rcu_idle_timer_stop(); if ( cpuidle_current_governor->reflect ) cpuidle_current_governor->reflect(power); diff --git a/xen/arch/x86/cpu/mwait-idle.c b/xen/arch/x86/cpu/mwait-idle.c index ae9e92b..c426e41 100644 --- a/xen/arch/x86/cpu/mwait-idle.c +++ b/xen/arch/x86/cpu/mwait-idle.c @@ -743,10 +743,10 @@ static void mwait_idle(void) return; } + rcu_idle_timer_start(); cpufreq_dbs_timer_suspend(); - sched_tick_suspend(); - /* sched_tick_suspend() can raise TIMER_SOFTIRQ. Process it now. */ + /* Timer related operations can raise TIMER_SOFTIRQ. Process it now. */ process_pending_softirqs(); /* Interrupts must be disabled for C2 and higher transitions. */ @@ -756,6 +756,7 @@ static void mwait_idle(void) local_irq_enable(); sched_tick_resume(); cpufreq_dbs_timer_resume(); + rcu_idle_timer_stop(); return; } @@ -802,6 +803,7 @@ static void mwait_idle(void) sched_tick_resume(); cpufreq_dbs_timer_resume(); + rcu_idle_timer_stop(); if ( cpuidle_current_governor->reflect ) cpuidle_current_governor->reflect(power); diff --git a/xen/common/rcupdate.c b/xen/common/rcupdate.c index f0fdc87..4586f2a 100644 --- a/xen/common/rcupdate.c +++ b/xen/common/rcupdate.c @@ -84,8 +84,14 @@ struct rcu_data { int cpu; struct rcu_head barrier; long last_rs_qlen; /* qlen during the last resched */ + + /* 3) idle CPUs handling */ + struct timer idle_timer; + bool idle_timer_active; }; +#define RCU_IDLE_TIMER_PERIOD MILLISECS(10) + static DEFINE_PER_CPU(struct rcu_data, rcu_data); static int blimit = 10; @@ -402,7 +408,48 @@ int rcu_needs_cpu(int cpu) { struct rcu_data *rdp = &per_cpu(rcu_data, cpu); - return (!!rdp->curlist || rcu_pending(cpu)); + return (!!rdp->curlist || rcu_pending(cpu)) && !rdp->idle_timer_active; +} + +/* + * Timer for making sure the CPU where a callback is queued does + * periodically poke rcu_pedning(), so that it will invoke the callback + * not too late after the end of the grace period. + */ +void rcu_idle_timer_start() +{ + struct rcu_data *rdp = &this_cpu(rcu_data); + + if (likely(!rdp->curlist)) + return; + + set_timer(&rdp->idle_timer, NOW() + RCU_IDLE_TIMER_PERIOD); + rdp->idle_timer_active = true; +} + +void rcu_idle_timer_stop() +{ + struct rcu_data *rdp = &this_cpu(rcu_data); + + if (likely(!rdp->idle_timer_active)) + return; + + rdp->idle_timer_active = false; + stop_timer(&rdp->idle_timer); +} + +static void rcu_idle_timer_handler(void* data) +{ + /* + * Nothing, really... And in fact, we don't expect to ever get in here, + * as rcu_idle_timer_stop(), called while waking from idle, prevent that + * to happen by stopping the timer before the TIMER_SOFTIRQ handler has + * a chance to run. + * + * But that's fine, because all we want is the CPU that needs to execute + * the callback to be periodically woken up and check rcu_pending(). + */ + ASSERT_UNREACHABLE(); } void rcu_check_callbacks(int cpu) @@ -423,6 +470,8 @@ static void rcu_move_batch(struct rcu_data *this_rdp, struct rcu_head *list, static void rcu_offline_cpu(struct rcu_data *this_rdp, struct rcu_ctrlblk *rcp, struct rcu_data *rdp) { + kill_timer(&rdp->idle_timer); + /* If the cpu going offline owns the grace period we can block * indefinitely waiting for it, so flush it here. */ @@ -451,6 +500,7 @@ static void rcu_init_percpu_data(int cpu, struct rcu_ctrlblk *rcp, rdp->qs_pending = 0; rdp->cpu = cpu; rdp->blimit = blimit; + init_timer(&rdp->idle_timer, rcu_idle_timer_handler, (void*) rdp, cpu); } static int cpu_callback( diff --git a/xen/include/xen/rcupdate.h b/xen/include/xen/rcupdate.h index 561ac43..3402eb5 100644 --- a/xen/include/xen/rcupdate.h +++ b/xen/include/xen/rcupdate.h @@ -149,4 +149,7 @@ int rcu_barrier(void); void rcu_idle_enter(unsigned int cpu); void rcu_idle_exit(unsigned int cpu); +void rcu_idle_timer_start(void); +void rcu_idle_timer_stop(void); + #endif /* __XEN_RCUPDATE_H */