From patchwork Thu May 11 12:07:28 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Zhoujian (jay)" X-Patchwork-Id: 9721553 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 4589960236 for ; Thu, 11 May 2017 13:55:03 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 38DA428650 for ; Thu, 11 May 2017 13:55:03 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 2A29D28668; Thu, 11 May 2017 13:55:03 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.9 required=2.0 tests=BAYES_00,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id A724328650 for ; Thu, 11 May 2017 13:55:02 +0000 (UTC) Received: from localhost ([::1]:48371 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1d8oYX-0005dx-N3 for patchwork-qemu-devel@patchwork.kernel.org; Thu, 11 May 2017 09:55:01 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:47495) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1d8mt1-00020t-PE for qemu-devel@nongnu.org; Thu, 11 May 2017 08:08:08 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1d8msz-0007po-3O for qemu-devel@nongnu.org; Thu, 11 May 2017 08:08:03 -0400 Received: from szxga03-in.huawei.com ([45.249.212.189]:3863) by eggs.gnu.org with esmtps (TLS1.0:RSA_ARCFOUR_SHA1:16) (Exim 4.71) (envelope-from ) id 1d8msy-0007kw-Ck for qemu-devel@nongnu.org; Thu, 11 May 2017 08:08:01 -0400 Received: from 172.30.72.56 (EHLO DGGEML401-HUB.china.huawei.com) ([172.30.72.56]) by dggrg03-dlp.huawei.com (MOS 4.4.6-GA FastPath queued) with ESMTP id ANK33375; Thu, 11 May 2017 20:07:39 +0800 (CST) Received: from DGGEML511-MBX.china.huawei.com ([169.254.1.164]) by DGGEML401-HUB.china.huawei.com ([fe80::89ed:853e:30a9:2a79%31]) with mapi id 14.03.0301.000; Thu, 11 May 2017 20:07:29 +0800 From: "Zhoujian (jay)" To: "Dr. David Alan Gilbert" , yanghongyang Thread-Topic: [Qemu-devel] About QEMU BQL and dirty log switch in Migration Thread-Index: AQHSvRnLyPsNIXfv9k+ZtGcw9lIdjKHvHZmA Date: Thu, 11 May 2017 12:07:28 +0000 Message-ID: References: <830bfc39-56c7-a901-9ebb-77d6e7a5614c@huawei.com> <874lxeovrg.fsf@secure.mitica> <7cd332ec-48d4-1feb-12e2-97b50b04e028@huawei.com> <20170424164244.GJ2362@work-vm> In-Reply-To: <20170424164244.GJ2362@work-vm> Accept-Language: zh-CN, en-US Content-Language: zh-CN X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.177.19.14] MIME-Version: 1.0 X-CFilter-Loop: Reflected X-Mirapoint-Virus-RAPID-Raw: score=unknown(0), refid=str=0001.0A020203.5914540D.004E, ss=1, re=0.000, recu=0.000, reip=0.000, cl=1, cld=1, fgs=0, ip=169.254.1.164, so=2014-11-16 11:51:01, dmn=2013-03-21 17:37:32 X-Mirapoint-Loop-Id: 86b90252629c75137f5d05c26c5336b1 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.4.x-2.6.x [generic] [fuzzy] X-Received-From: 45.249.212.189 X-Mailman-Approved-At: Thu, 11 May 2017 09:54:07 -0400 Subject: Re: [Qemu-devel] About QEMU BQL and dirty log switch in Migration X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Zhanghailiang , "quintela@redhat.com" , "wangxin \(U\)" , "qemu-devel@nongnu.org" , "Gonglei \(Arei\)" , Huangzhichao , "pbonzini@redhat.com" , "Herongguang \(Stephen\)" Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP Hi all, After applying the patch below, the time which memory_global_dirty_log_stop() function takes is down to milliseconds of a 4T memory guest, but I'm not sure whether this patch will trigger other problems. Does this patch make sense? > * Yang Hongyang (yanghongyang@huawei.com) wrote: > > > > > > On 2017/4/24 20:06, Juan Quintela wrote: > > > Yang Hongyang wrote: > > >> Hi all, > > >> > > >> We found dirty log switch costs more then 13 seconds while > > >> migrating a 4T memory guest, and dirty log switch is currently > > >> protected by QEMU BQL. This causes guest freeze for a long time > > >> when switching dirty log on, and the migration downtime is > unacceptable. > > >> Are there any chance to optimize the time cost for dirty log switch > operation? > > >> Or move the time consuming operation out of the QEMU BQL? > > > > > > Hi > > > > > > Could you specify what do you mean by dirty log switch? > > > The one inside kvm? > > > The merge between kvm one and migration bitmap? > > > > The call of the following functions: > > memory_global_dirty_log_start/stop(); > > I suppose there's a few questions; > a) Do we actually need the BQL - and if so why > b) What actually takes 13s? It's probably worth figuring out where it > goes, the whole bitmap is only 1GB isn't it even on a 4TB machine, and > even the simplest way to fill that takes way less than 13s. > > Dave > > > > > > > > > Thanks, Juan. > > > > > > > -- > > Thanks, > > Yang > -- > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK Regards, Jay Zhou diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index 464da93..fe26ee5 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -8313,6 +8313,8 @@ void kvm_arch_commit_memory_region(struct kvm *kvm, enum kvm_mr_change change) { int nr_mmu_pages = 0; + int i; + struct kvm_vcpu *vcpu; if (!kvm->arch.n_requested_mmu_pages) nr_mmu_pages = kvm_mmu_calculate_mmu_pages(kvm); @@ -8328,14 +8330,15 @@ void kvm_arch_commit_memory_region(struct kvm *kvm, * in the source machine (for example if live migration fails), small * sptes will remain around and cause bad performance. * - * Scan sptes if dirty logging has been stopped, dropping those - * which can be collapsed into a single large-page spte. Later - * page faults will create the large-page sptes. + * Reset each vcpu's mmu, then page faults will create the large-page + * sptes later. */ if ((change != KVM_MR_DELETE) && (old->flags & KVM_MEM_LOG_DIRTY_PAGES) && - !(new->flags & KVM_MEM_LOG_DIRTY_PAGES)) - kvm_mmu_zap_collapsible_sptes(kvm, new); + !(new->flags & KVM_MEM_LOG_DIRTY_PAGES)) { + kvm_for_each_vcpu(i, vcpu, kvm) + kvm_mmu_reset_context(vcpu); + } /* * Set up write protection and/or dirty logging for the new slot.