From patchwork Tue Feb 6 11:46:31 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Peter Lieven X-Patchwork-Id: 10202809 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id D51E36020F for ; Tue, 6 Feb 2018 11:47:36 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D172128979 for ; Tue, 6 Feb 2018 11:47:36 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id C16CF28987; Tue, 6 Feb 2018 11:47:36 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.9 required=2.0 tests=BAYES_00,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id CDA1828978 for ; Tue, 6 Feb 2018 11:47:35 +0000 (UTC) Received: from localhost ([::1]:60599 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ej1ip-0000TD-1w for patchwork-qemu-devel@patchwork.kernel.org; Tue, 06 Feb 2018 06:47:35 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:38997) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ej1hu-0008L8-So for qemu-devel@nongnu.org; Tue, 06 Feb 2018 06:46:40 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1ej1hr-00069Y-PR for qemu-devel@nongnu.org; Tue, 06 Feb 2018 06:46:38 -0500 Received: from mx-v6.kamp.de ([2a02:248:0:51::16]:35800 helo=mx01.kamp.de) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1ej1hr-00067s-FO for qemu-devel@nongnu.org; Tue, 06 Feb 2018 06:46:35 -0500 Received: (qmail 3338 invoked by uid 89); 6 Feb 2018 11:46:32 -0000 Received: from [195.62.97.192] by client-16-kamp (envelope-from , uid 89) with qmail-scanner-2010/03/19-MF (clamdscan: 0.99.3/24290. avast: 1.2.2/17010300. spamassassin: 3.4.1. Clear:RC:1(195.62.97.192):. Processed in 0.087758 secs); 06 Feb 2018 11:46:32 -0000 Received: from kerio.kamp.de ([195.62.97.192]) by mx01.kamp.de with ESMTPS (DHE-RSA-AES256-SHA encrypted); 6 Feb 2018 11:46:31 -0000 X-GL_Whitelist: yes X-Footer: a2FtcC5kZQ== Received: from submission.kamp.de ([195.62.97.28]) by kerio.kamp.de with ESMTPS (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256 bits)) for qemu-devel@nongnu.org; Tue, 6 Feb 2018 12:46:30 +0100 Received: (qmail 1756 invoked from network); 6 Feb 2018 11:46:31 -0000 Received: from lieven-pc.kamp-intra.net (HELO ?172.21.12.60?) (pl@kamp.de@::ffff:172.21.12.60) by submission.kamp.de with ESMTPS (DHE-RSA-AES128-SHA encrypted) ESMTPA; 6 Feb 2018 11:46:31 -0000 To: "Dr. David Alan Gilbert" References: <6cc4a99c-0212-6b7b-4a12-3e898215bea9@kamp.de> <20170919143829.GH2107@work-vm> <48cdd8ff-4940-9aa1-9aba-1acf8ce74ebe@kamp.de> <20170919144130.GI2107@work-vm> <20170921123625.GE2717@work-vm> <20171212170533.GG2409@work-vm> From: Peter Lieven Message-ID: <8ed6cb1e-4770-21ec-164e-7142e649eab3@kamp.de> Date: Tue, 6 Feb 2018 12:46:31 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.5.0 MIME-Version: 1.0 In-Reply-To: <20171212170533.GG2409@work-vm> Content-Language: en-US X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 2a02:248:0:51::16 Subject: Re: [Qemu-devel] Block Migration and CPU throttling X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Fam Zheng , Stefan Hajnoczi , qemu block , Juan Quintela , "qemu-devel@nongnu.org" , jjherne@linux.vnet.ibm.com Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP Am 12.12.2017 um 18:05 schrieb Dr. David Alan Gilbert: > * Peter Lieven (pl@kamp.de) wrote: >> Am 21.09.2017 um 14:36 schrieb Dr. David Alan Gilbert: >>> * Peter Lieven (pl@kamp.de) wrote: >>>> Am 19.09.2017 um 16:41 schrieb Dr. David Alan Gilbert: >>>>> * Peter Lieven (pl@kamp.de) wrote: >>>>>> Am 19.09.2017 um 16:38 schrieb Dr. David Alan Gilbert: >>>>>>> * Peter Lieven (pl@kamp.de) wrote: >>>>>>>> Hi, >>>>>>>> >>>>>>>> I just noticed that CPU throttling and Block Migration don't work together very well. >>>>>>>> During block migration the throttling heuristic detects that we obviously make no progress >>>>>>>> in ram transfer. But the reason is the running block migration and not a too high dirty pages rate. >>>>>>>> >>>>>>>> The result is that any VM is throttled by 99% during block migration. >>>>>>> Hmm that's unfortunate; do you have a bandwidth set lower than your >>>>>>> actual network connection? I'm just wondering if it's actually going >>>>>>> between the block and RAM iterative sections or getting stuck in ne. >>>>>> It happens also if source and dest are on the same machine and speed is set to 100G. >>>>> But does it happen if they're not and the speed is set low? >>>> Yes, it does. I noticed it in our test environment between different nodes with a 10G >>>> link in between. But its totally clear why it happens. During block migration we transfer >>>> all dirty memory pages in each round (if there is moderate memory load), but all dirty >>>> pages are obviously more than 50% of the transferred ram in that round. >>>> It is more exactly 100%. But the current logic triggers on this condition. >>>> >>>> I think I will go forward and send a patch which disables auto converge during >>>> block migration bulk stage. >>> Yes, that's fair; it probably would also make sense to throttle the RAM >>> migration during the block migration bulk stage, since the chances are >>> it's not going to get far. (I think in the nbd setup, the main >>> migration process isn't started until the end of bulk). >> Catching up with the idea of delaying ram migration until block bulk has completed. >> What do you think is the easiest way to achieve this? > > > I think the answer depends whether we think this is a 'special' or we > need a new general purpose mechanism. > > If it was really general then we'd probably want to split the iterative > stage in two somehow, and only do RAM in the second half. > > But I'm not sure it's worth it; I suspect the easiest way is: > > a) Add a counter in migration/ram.c or in the RAM state somewhere > b) Make ram_save_inhibit increment the counter > c) Check the counter at the head of ram_save_iterate and just exit > if it's none 0 > d) Call ram_save_inhibit from block_save_setup > e) Then release it when you've finished the bulk stage > > Make sure you still count the RAM in the pending totals, otherwise > migration might think it's finished a bit early. Is there any culprit I don't see or is it as easy as this? Peter diff --git a/migration/ram.c b/migration/ram.c index cb1950f..c67bcf1 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -2255,6 +2255,13 @@ static int ram_save_iterate(QEMUFile *f, void *opaque)      int64_t t0;      int done = 0; +    if (blk_mig_bulk_active()) { +        /* Avoid transferring RAM during bulk phase of block migration as +         * the bulk phase will usually take a lot of time and transferring +         * RAM updates again and again is pointless. */ +        goto out; +    } +      rcu_read_lock();      if (ram_list.version != rs->last_version) {          ram_state_reset(rs); @@ -2301,6 +2308,7 @@ static int ram_save_iterate(QEMUFile *f, void *opaque)       */      ram_control_after_iterate(f, RAM_CONTROL_ROUND); +out:      qemu_put_be64(f, RAM_SAVE_FLAG_EOS);      ram_counters.transferred += 8;