From patchwork Mon Sep 5 14:06:57 2011 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Philipp Hahn X-Patchwork-Id: 1124902 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by demeter2.kernel.org (8.14.4/8.14.4) with ESMTP id p85E7Qcp030650 for ; Mon, 5 Sep 2011 14:07:28 GMT Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1754122Ab1IEOHP (ORCPT ); Mon, 5 Sep 2011 10:07:15 -0400 Received: from mail.univention.de ([82.198.197.8]:2339 "EHLO mail.univention.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753494Ab1IEOHI (ORCPT ); Mon, 5 Sep 2011 10:07:08 -0400 Received: from localhost (localhost [127.0.0.1]) by slugis.knut.univention.de (Postfix) with ESMTP id CAA33673001; Mon, 5 Sep 2011 16:07:04 +0200 (CEST) Received: from localhost (localhost [127.0.0.1]) by slugis.knut.univention.de (Postfix) with ESMTP id B3465673008; Mon, 5 Sep 2011 16:07:04 +0200 (CEST) X-Virus-Scanned: by amavisd-new-2.6.1 (20080629) (Debian) at knut.univention.de Received: from mail.univention.de ([127.0.0.1]) by localhost (slugis.knut.univention.de [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id FXhjko3ejC5s; Mon, 5 Sep 2011 16:07:04 +0200 (CEST) Received: from stave.knut.univention.de (stave.knut.univention.de [192.168.0.191]) by slugis.knut.univention.de (Postfix) with ESMTPSA id 2DC41673001; Mon, 5 Sep 2011 16:07:04 +0200 (CEST) From: Philipp Hahn Organization: Univention.de To: Zachary Amsden , =?utf-8?q?=E7=BC=96=E7=A0=81=E4=BA=BA?= , Xiao Guangrong , Nikola Ciprich , Michael Tokarev Subject: Re: [BUG, PATCH-2.6.32] Fix a possible backwards warp of kvmclock Date: Mon, 5 Sep 2011 16:06:57 +0200 User-Agent: KMail/1.9.10 (enterprise35 20100903.1171286) Cc: kvm@vger.kernel.org, Avi Kivity , Marcelo Tosatti , Glauber Costa , Thomas Gleixner , John Stultz References: <1282291669-25709-1-git-send-email-zamsden@redhat.com> <1282291669-25709-17-git-send-email-zamsden@redhat.com> <201109022034.16517.hahn@univention.de> In-Reply-To: <201109022034.16517.hahn@univention.de> X-KMail-QuotePrefix: > MIME-Version: 1.0 Message-Id: <201109051607.02349.hahn@univention.de> Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org X-Greylist: IP, sender and recipient auto-whitelisted, not delayed by milter-greylist-4.2.6 (demeter2.kernel.org [140.211.167.43]); Mon, 05 Sep 2011 14:07:28 +0000 (UTC) Hello, (cc:-ing lost of people who reported similar bugs on kvm-devel) > Changing clock in KVM host may cause VM to hang > 2.6.32 guest with paravirt clock enabled hangs on 2.6.37.6 host (w qemu-kvm-0.13.0) I found a bug regarding PV-clock in the KVM-kernel module. The attached patch solves the problem of the guest being very slow after a reboot. Can you please have a look and give it a try to see if it solves your problem as well. Since the fix is only relevant for the stable 2.6.32 tree, where the code is quiet different, please have a look and forward to stable@ as appropriate. Sincerely Philipp PS: This bug is tracked in our German Bugzilla at Bug #23257: Reset tsc_timestamp on TSC writes vcpu->last_guest_tsc is updated in vcpu_enter_guest() and kvm_arch_vcpu_put() by getting the last value of the TSC from the guest. On reset, the SeaBIOS resets the TSC to 0, which triggers a bug on the next call to kvm_write_guest_time(): Since vcpu->hw_clock.tsc_timestamp still contains the old value before the reset, "max_kernel_ns = vcpu->last_guest_tsc - vcpu->hw_clock.tsc_timestamp" gets negative. Since the variable is u64, it gets translated to a large positive value. [9333.197080] vcpu->last_guest_tsc =209_328_760_015 ? vcpu->hv_clock.tsc_timestamp=209_328_708_109 vcpu->last_kernel_ns =9_333_179_830_643 kernel_ns =9_333_197_073_429 max_kernel_ns =9_333_179_847_943 ? [9336.910995] vcpu->last_guest_tsc =9_438_510_584 ? vcpu->hv_clock.tsc_timestamp=211_080_593_143 vcpu->last_kernel_ns =9_333_763_732_907 kernel_ns =9_336_910_990_771 max_kernel_ns =6_148_296_831_006_663_830 ? For completeness, here are the values for my 3 GHz CPU: vcpu->hv_clock.tsc_shift =-1 vcpu->hv_clock.tsc_to_system_mul =2_863_019_502 This makes the guest kernel crawl very slowly when clocksource=kvmclock is used: sleeps take way longer than expected and don't match wall clock any more. The times printed with printk() don't match real time and the reboot often stalls for long times. In linux-git this isn't a problem, since on every MSR_IA32_TSC write vcpu->arch.hv_clock.tsc_timestamp is reset to 0, which disables above logic. The code there is only in arch/x86/kvm/x86.c, since much of the kvm-clock related code has been refactured for 2.6.37: 99e3e30a arch/x86/kvm/x86.c (Zachary Amsden 2010-08-19 22:07:17 -1000 1084) vcpu->arch.hv_clock.tsc_timestamp = 0; Since 1d5f066e0b63271b67eac6d3752f8aa96adcbddb from 2.6.37 was back-ported to 2.6.32.40 as ad2088cabe0fd7f633f38ba106025d33ed9a2105, the following patch is needed to add the needed reset logic to 2.6.32 as well. Signed-off-by: Philipp Hahn --- a/arch/x86/kvm/vmx.c 2011-09-05 14:17:54.000000000 +0200 +++ b/arch/x86/kvm/vmx.c 2011-09-05 14:18:03.000000000 +0200 @@ -1067,6 +1067,7 @@ static int vmx_set_msr(struct kvm_vcpu * case MSR_IA32_TSC: rdtscll(host_tsc); guest_write_tsc(data, host_tsc); + vcpu->arch.hv_clock.tsc_timestamp = 0; break; case MSR_IA32_CR_PAT: if (vmcs_config.vmentry_ctrl & VM_ENTRY_LOAD_IA32_PAT) { --- a/arch/x86/kvm/svm.c 2011-09-05 14:17:57.000000000 +0200 +++ b/arch/x86/kvm/svm.c 2011-09-05 14:18:00.000000000 +0200 @@ -2256,6 +2256,7 @@ static int svm_set_msr(struct kvm_vcpu * } svm->vmcb->control.tsc_offset = tsc_offset + g_tsc_offset; + vcpu->arch.hv_clock.tsc_timestamp = 0; break; }