From patchwork Tue Oct 10 09:14:08 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dongli Zhang X-Patchwork-Id: 9995393 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 40062603B5 for ; Tue, 10 Oct 2017 09:17:47 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 3D68928438 for ; Tue, 10 Oct 2017 09:17:47 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 323EA28445; Tue, 10 Oct 2017 09:17:47 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=2.0 tests=BAYES_00, RCVD_IN_DNSWL_MED, UNPARSEABLE_RELAY autolearn=ham version=3.3.1 Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 9EA3628438 for ; Tue, 10 Oct 2017 09:17:46 +0000 (UTC) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1e1qd7-0002Xv-06; Tue, 10 Oct 2017 09:15:13 +0000 Received: from mail6.bemta3.messagelabs.com ([195.245.230.39]) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1e1qd5-0002Xo-VN for xen-devel@lists.xen.org; Tue, 10 Oct 2017 09:15:12 +0000 Received: from [85.158.137.68] by server-12.bemta-3.messagelabs.com id A2/4F-24316-F9F8CD95; Tue, 10 Oct 2017 09:15:11 +0000 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFmpkkeJIrShJLcpLzFFi42LpnVTnqjuv/06 kwalWTYslHxezODB6HN39mymAMYo1My8pvyKBNePA8RWMBRuFKm5cK2tg3MLfxcjFISQwkUli 5tHNTBDOb0aJ90dus0E4GxklPu9cBpXpZpTY9/g9excjJwebgI7EtAOnWEBsEQFLiReHJrGCF DEL9DJK7J47E6idg0NYIEqiv0UTpIZFQFVizt1dYGFeAXeJ/wc5QMISAnISN891MkPYhhKfNy 5lnsDIs4CRYRWjRnFqUVlqka6xgV5SUWZ6RkluYmaOrqGBsV5uanFxYnpqTmJSsV5yfu4mRqD n6xkYGHcwdp7wO8QoycGkJMpr230nUogvKT+lMiOxOCO+qDQntfgQowwHh5IE77w+oJxgUWp6 akVaZg4wBGHSEhw8SiK87CCtvMUFibnFmekQqVOM9hw/Jl35w8TRcfMukNwEJjd8f/CHSYglL z8vVUqc9wHIVAGQtozSPLihsJi5xCgrJczLyMDAIMRTkFqUm1mCKv+KUZyDUUmYNxNkCk9mXg nc7ldAZzEBnSWaBnZWSSJCSqqBke1E61O2U67T0vIvM3HxFsrJqkjPf/8h5g//jA/fLuQGvec JYrB59VFF8dHMqJXFmU1Ob+waOPd4KjVLeDDXbon9t0dE/XB4bqL/oxvu8z5b/Nu20EVo+ttg xe7D3QU7Nr4Uz+vuTVl14fTtkwf21IhbsHW0BmhMF2L9l7VP4HDQDmWlP1GvlFiKMxINtZiLi hMB2GQvaJQCAAA= X-Env-Sender: dongli.zhang@oracle.com X-Msg-Ref: server-4.tower-31.messagelabs.com!1507626908!51477906!1 X-Originating-IP: [141.146.126.69] X-SpamReason: No, hits=0.0 required=7.0 tests=sa_preprocessor: VHJ1c3RlZCBJUDogMTQxLjE0Ni4xMjYuNjkgPT4gMjc3MjE4\n X-StarScan-Received: X-StarScan-Version: 9.4.45; banners=-,-,- X-VirusChecked: Checked Received: (qmail 19122 invoked from network); 10 Oct 2017 09:15:10 -0000 Received: from aserp1040.oracle.com (HELO aserp1040.oracle.com) (141.146.126.69) by server-4.tower-31.messagelabs.com with DHE-RSA-AES256-GCM-SHA384 encrypted SMTP; 10 Oct 2017 09:15:10 -0000 Received: from userv0022.oracle.com (userv0022.oracle.com [156.151.31.74]) by aserp1040.oracle.com (Sentrion-MTA-4.3.2/Sentrion-MTA-4.3.2) with ESMTP id v9A9ELVe002504 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 10 Oct 2017 09:14:21 GMT Received: from userv0122.oracle.com (userv0122.oracle.com [156.151.31.75]) by userv0022.oracle.com (8.14.4/8.14.4) with ESMTP id v9A9ELcc025740 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 10 Oct 2017 09:14:21 GMT Received: from abhmp0006.oracle.com (abhmp0006.oracle.com [141.146.116.12]) by userv0122.oracle.com (8.14.4/8.14.4) with ESMTP id v9A9EJRq007583; Tue, 10 Oct 2017 09:14:19 GMT Received: from linux.cn.oracle.com (/10.182.70.198) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Tue, 10 Oct 2017 02:14:18 -0700 From: Dongli Zhang To: linux-kernel@vger.kernel.org, xen-devel@lists.xen.org Date: Tue, 10 Oct 2017 17:14:08 +0800 Message-Id: <1507626848-24148-1-git-send-email-dongli.zhang@oracle.com> X-Mailer: git-send-email 2.7.4 X-Source-IP: userv0022.oracle.com [156.151.31.74] Cc: xen.list@daevel.fr, peterz@infradead.org, dario.faggioli@citrix.com, bevan@bi-co.net, mingo@redhat.com, joao.m.martins@oracle.com Subject: [Xen-devel] [PATCH 1/1] sched/cputime: do not decrease steal time after live migration on xen X-BeenThere: xen-devel@lists.xen.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Errors-To: xen-devel-bounces@lists.xen.org Sender: "Xen-devel" X-Virus-Scanned: ClamAV using ClamSMTP After guest live migration on xen, steal time in /proc/stat (cpustat[CPUTIME_STEAL]) might decrease because steal returned by paravirt_steal_clock() might be less than this_rq()->prev_steal_time. For instance, steal time of each vcpu is 335 before live migration. cpu 198 0 368 200064 1962 0 0 1340 0 0 cpu0 38 0 81 50063 492 0 0 335 0 0 cpu1 65 0 97 49763 634 0 0 335 0 0 cpu2 38 0 81 50098 462 0 0 335 0 0 cpu3 56 0 107 50138 374 0 0 335 0 0 After live migration, steal time is reduced to 312. cpu 200 0 370 200330 1971 0 0 1248 0 0 cpu0 38 0 82 50123 500 0 0 312 0 0 cpu1 65 0 97 49832 634 0 0 312 0 0 cpu2 39 0 82 50167 462 0 0 312 0 0 cpu3 56 0 107 50207 374 0 0 312 0 0 The code in this patch is borrowed from do_stolen_accounting() which has already been removed from linux source code since commit ecb23dc6 ("xen: add steal_clock support on x86"). Similar and more severe issue would impact prior linux 4.8-4.10 as discussed by Michael Las at https://0xstubs.org/debugging-a-flaky-cpu-steal-time-counter-on-a-paravirtualized-xen-guest. Unlike the issue discussed by Michael Las which would overflow steal time and lead to 100% st usage in top command for linux 4.8-4.10, the issue for linux 4.11+ would only decrease but not overflow steal time after live migration. References: https://0xstubs.org/debugging-a-flaky-cpu-steal-time-counter-on-a-paravirtualized-xen-guest Signed-off-by: Dongli Zhang --- kernel/sched/cputime.c | 13 ++++++++++--- 1 file changed, 10 insertions(+), 3 deletions(-) diff --git a/kernel/sched/cputime.c b/kernel/sched/cputime.c index 14d2dbf..57d09cab 100644 --- a/kernel/sched/cputime.c +++ b/kernel/sched/cputime.c @@ -238,10 +238,17 @@ static __always_inline u64 steal_account_process_time(u64 maxtime) { #ifdef CONFIG_PARAVIRT if (static_key_false(¶virt_steal_enabled)) { - u64 steal; + u64 steal, steal_time; + s64 steal_delta; + + steal_time = paravirt_steal_clock(smp_processor_id()); + steal = steal_delta = steal_time - this_rq()->prev_steal_time; + + if (unlikely(steal_delta < 0)) { + this_rq()->prev_steal_time = steal_time; + return 0; + } - steal = paravirt_steal_clock(smp_processor_id()); - steal -= this_rq()->prev_steal_time; steal = min(steal, maxtime); account_steal_time(steal); this_rq()->prev_steal_time += steal;