From patchwork Wed May 17 02:20:22 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Zhoujian (jay)" X-Patchwork-Id: 9729907 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id DE2A160386 for ; Wed, 17 May 2017 02:21:34 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C572328625 for ; Wed, 17 May 2017 02:21:34 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id B894C28642; Wed, 17 May 2017 02:21:34 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.9 required=2.0 tests=BAYES_00,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 010D228625 for ; Wed, 17 May 2017 02:21:33 +0000 (UTC) Received: from localhost ([::1]:45101 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dAoai-0006Hs-UY for patchwork-qemu-devel@patchwork.kernel.org; Tue, 16 May 2017 22:21:32 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:58344) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dAoa3-0006Hl-G1 for qemu-devel@nongnu.org; Tue, 16 May 2017 22:20:52 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1dAoZy-0003MY-Jp for qemu-devel@nongnu.org; Tue, 16 May 2017 22:20:51 -0400 Received: from szxga02-in.huawei.com ([45.249.212.188]:3927) by eggs.gnu.org with esmtps (TLS1.0:RSA_ARCFOUR_SHA1:16) (Exim 4.71) (envelope-from ) id 1dAoZx-0003KR-NN for qemu-devel@nongnu.org; Tue, 16 May 2017 22:20:46 -0400 Received: from 172.30.72.55 (EHLO DGGEML401-HUB.china.huawei.com) ([172.30.72.55]) by dggrg02-dlp.huawei.com (MOS 4.4.6-GA FastPath queued) with ESMTP id ANQ80825; Wed, 17 May 2017 10:20:30 +0800 (CST) Received: from DGGEML511-MBX.china.huawei.com ([169.254.1.188]) by DGGEML401-HUB.china.huawei.com ([fe80::89ed:853e:30a9:2a79%31]) with mapi id 14.03.0301.000; Wed, 17 May 2017 10:20:22 +0800 From: "Zhoujian (jay)" To: Wanpeng Li Thread-Topic: [Qemu-devel] About QEMU BQL and dirty log switch in Migration Thread-Index: AQHSvRnLyPsNIXfv9k+ZtGcw9lIdjKHvHZmA//+FMQCAABYtAIAJIEwA Date: Wed, 17 May 2017 02:20:22 +0000 Message-ID: References: <830bfc39-56c7-a901-9ebb-77d6e7a5614c@huawei.com> <874lxeovrg.fsf@secure.mitica> <7cd332ec-48d4-1feb-12e2-97b50b04e028@huawei.com> <20170424164244.GJ2362@work-vm> <85e3a0dd-20c8-8ff2-37ce-bfdf543e7787@redhat.com> In-Reply-To: Accept-Language: zh-CN, en-US Content-Language: zh-CN X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.177.19.14] MIME-Version: 1.0 X-CFilter-Loop: Reflected X-Mirapoint-Virus-RAPID-Raw: score=unknown(0), refid=str=0001.0A020203.591BB370.00A6, ss=1, re=0.000, recu=0.000, reip=0.000, cl=1, cld=1, fgs=0, ip=169.254.1.188, so=2014-11-16 11:51:01, dmn=2013-03-21 17:37:32 X-Mirapoint-Loop-Id: feb9ef4c7e5d3ec52e568c52ab096138 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.4.x-2.6.x [generic] [fuzzy] X-Received-From: 45.249.212.188 Subject: Re: [Qemu-devel] About QEMU BQL and dirty log switch in Migration X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: "Huangweidong \(C\)" , Zhanghailiang , "quintela@redhat.com" , "wangxin \(U\)" , yanghongyang , Xiao Guangrong , "qemu-devel@nongnu.org" , "Dr. David Alan Gilbert" , "Gonglei \(Arei\)" , Huangzhichao , Paolo Bonzini , "Herongguang \(Stephen\)" Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP Hi Wanpeng, > > On 11/05/2017 14:07, Zhoujian (jay) wrote: > >> - * Scan sptes if dirty logging has been stopped, dropping those > >> - * which can be collapsed into a single large-page spte. Later > >> - * page faults will create the large-page sptes. > >> + * Reset each vcpu's mmu, then page faults will create the > large-page > >> + * sptes later. > >> */ > >> if ((change != KVM_MR_DELETE) && > >> (old->flags & KVM_MEM_LOG_DIRTY_PAGES) && > >> - !(new->flags & KVM_MEM_LOG_DIRTY_PAGES)) > >> - kvm_mmu_zap_collapsible_sptes(kvm, new); > > This is an unlikely branch(unless guest live migration fails and continue > to run on the source machine) instead of hot path, do you have any > performance number for your real workloads? > Sorry to bother you again. Recently, I have tested the performance before migration and after migration failure using spec cpu2006 https://www.spec.org/cpu2006/, which is a standard performance evaluation tool. These are the results: ****** Before migration the score is 153, and the TLB miss statistics of the qemu process is: linux-sjrfac:/mnt/zhoujian # perf stat -e dTLB-load-misses,dTLB-loads,dTLB-store-misses, \ dTLB-stores,iTLB-load-misses,iTLB-loads -p 26463 sleep 10 Performance counter stats for process id '26463': 698,938 dTLB-load-misses # 0.13% of all dTLB cache hits (50.46%) 543,303,875 dTLB-loads (50.43%) 199,597 dTLB-store-misses (16.51%) 60,128,561 dTLB-stores (16.67%) 69,986 iTLB-load-misses # 6.17% of all iTLB cache hits (16.67%) 1,134,097 iTLB-loads (33.33%) 10.000684064 seconds time elapsed After migration failure the score is 149, and the TLB miss statistics of the qemu process is: linux-sjrfac:/mnt/zhoujian # perf stat -e dTLB-load-misses,dTLB-loads,dTLB-store-misses, \ dTLB-stores,iTLB-load-misses,iTLB-loads -p 26463 sleep 10 Performance counter stats for process id '26463': 765,400 dTLB-load-misses # 0.14% of all dTLB cache hits (50.50%) 540,972,144 dTLB-loads (50.47%) 207,670 dTLB-store-misses (16.50%) 58,363,787 dTLB-stores (16.67%) 109,772 iTLB-load-misses # 9.52% of all iTLB cache hits (16.67%) 1,152,784 iTLB-loads (33.32%) 10.000703078 seconds time elapsed ****** These are the steps: ====== Results of the score and TLB miss rate are almost the same, and I am confused. May I ask which tool do you use to evaluate the performance? And if my test steps are wrong, please let me know, thank you. Regards, Jay Zhou ====== (1) the version of kmod is 4.4.11(with slightly modified) and the version of qemu is 2.6.0 (with slightly modified), the kmod is applied with the following patch according to Paolo's advice: diff --git a/source/x86/x86.c b/source/x86/x86.c index 054a7d3..75a4bb3 100644 --- a/source/x86/x86.c +++ b/source/x86/x86.c @@ -8550,8 +8550,10 @@ void kvm_arch_commit_memory_region(struct kvm *kvm, */ if ((change != KVM_MR_DELETE) && (old->flags & KVM_MEM_LOG_DIRTY_PAGES) && - !(new->flags & KVM_MEM_LOG_DIRTY_PAGES)) - kvm_mmu_zap_collapsible_sptes(kvm, new); + !(new->flags & KVM_MEM_LOG_DIRTY_PAGES)) { + printk(KERN_ERR "zj make KVM_REQ_MMU_RELOAD request\n"); + kvm_make_all_cpus_request(kvm, KVM_REQ_MMU_RELOAD); + } /* * Set up write protection and/or dirty logging for the new slot. (2) I started up a memory preoccupied 10G VM(suse11sp3), which means its "RES column" in top is 10G, in order to set up the EPT table in advance. (3) And then, I run the test case 429.mcf of spec cpu2006 before migration and after migration failure. The 429.mcf is a memory intensive workload, and the migration failure is constructed deliberately with the following patch of qemu: diff --git a/migration/migration.c b/migration/migration.c index 5d725d0..88dfc59 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -625,6 +625,9 @@ static void process_incoming_migration_co(void *opaque) MIGRATION_STATUS_ACTIVE); ret = qemu_loadvm_state(f); + // deliberately construct the migration failure + exit(EXIT_FAILURE); + ps = postcopy_state_get(); trace_process_incoming_migration_co_end(ret, ps); if (ps != POSTCOPY_INCOMING_NONE) {