diff mbox

Performance issue

Message ID 201211281339.17589.vrozenfe@redhat.com (mailing list archive)
State New, archived
Headers show

Commit Message

Vadim Rozenfeld Nov. 28, 2012, 11:39 a.m. UTC
On Tuesday, November 27, 2012 11:13:12 PM George-Cristian Bîrzan wrote:
> On Tue, Nov 27, 2012 at 10:38 PM, Vadim Rozenfeld <vrozenfe@redhat.com> 
wrote:
> > I have some code which do both reference time and invariant TSC but it
> > will not work after migration. I will send it later today.
> 
> Do you mean migrating guests? This is not an issue for us.
OK, but don't say I didn't warn you :)

There are two patches, one for kvm and another one for qemu.
you will probably need to rebase them.
Add "hv_tsc" cpu parameter to activate this feature.
you will probably need to deactivate hpet by adding "-no-hpet"
parameter as well.

best regards,
Vadim.

> 
> Also, it would be much appreciated!
> 
> --
> George-Cristian Bîrzan

Comments

George-Cristian Bîrzan Nov. 28, 2012, 7:09 p.m. UTC | #1
On Wed, Nov 28, 2012 at 1:39 PM, Vadim Rozenfeld <vrozenfe@redhat.com> wrote:
> On Tuesday, November 27, 2012 11:13:12 PM George-Cristian Bîrzan wrote:
>> On Tue, Nov 27, 2012 at 10:38 PM, Vadim Rozenfeld <vrozenfe@redhat.com>
> wrote:
>> > I have some code which do both reference time and invariant TSC but it
>> > will not work after migration. I will send it later today.
>>
>> Do you mean migrating guests? This is not an issue for us.
> OK, but don't say I didn't warn you :)
>
> There are two patches, one for kvm and another one for qemu.
> you will probably need to rebase them.
> Add "hv_tsc" cpu parameter to activate this feature.
> you will probably need to deactivate hpet by adding "-no-hpet"
> parameter as well.

I've also added +hv_relaxed since then, but this is the command I'm
using now and there's no change:

/usr/bin/qemu-kvm -name b691546e-79f8-49c6-a293-81067503a6ad -S -M
pc-1.2 -enable-kvm -m 16384 -smp 9,sockets=1,cores=9,threads=1 -uuid
b691546e-79f8-49c6-a293-81067503a6ad -no-user-config -nodefaults
-chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/b691546e-79f8-49c6-a293-81067503a6ad.monitor,server,nowait
-mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc
-no-hpet -no-shutdown -device
piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive
file=/var/lib/libvirt/images/dis-magnetics-2-223101/d8b233c6-8424-4de9-ae3c-7c9a60288514,if=none,id=drive-virtio-disk0,format=qcow2,cache=writeback,aio=native
-device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1
-netdev tap,fd=35,id=hostnet0,vhost=on,vhostfd=36 -device
virtio-net-pci,netdev=hostnet0,id=net0,mac=22:2e:fb:a2:36:be,bus=pci.0,addr=0x3
-netdev tap,fd=40,id=hostnet1,vhost=on,vhostfd=41 -device
virtio-net-pci,netdev=hostnet1,id=net1,mac=22:94:44:5a:cb:24,bus=pci.0,addr=0x4
-vnc 127.0.0.1:0,password -vga cirrus -device
virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6 -cpu host,hv_tsc

I compiled qemu-1.2.0-24 after applying your patch, used the head for
KVM, and I see no difference. I've tried setting windows'
useplatformclock on and off, no change either.


Other than that, was looking into a profiling trace of the software
running and a lot of time (60%?) is spent calling two functions from
hal.dll, HalpGetPmTimerSleepModePerfCounter when I disable HPET, and
HalpHPETProgramRolloverTimer which do point at something related to
the timers.

Any other thing I can try?


--
George-Cristian Bîrzan
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
George-Cristian Bîrzan Nov. 28, 2012, 7:18 p.m. UTC | #2
On Wed, Nov 28, 2012 at 1:39 PM, Vadim Rozenfeld <vrozenfe@redhat.com> wrote:
> There are two patches, one for kvm and another one for qemu.

I just realised this. I was supposed to use qemu, or qemu-kvm? I used qemu

--
George-Cristian Bîrzan
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Gleb Natapov Nov. 28, 2012, 7:56 p.m. UTC | #3
On Wed, Nov 28, 2012 at 09:18:38PM +0200, George-Cristian Bîrzan wrote:
> On Wed, Nov 28, 2012 at 1:39 PM, Vadim Rozenfeld <vrozenfe@redhat.com> wrote:
> > There are two patches, one for kvm and another one for qemu.
> 
> I just realised this. I was supposed to use qemu, or qemu-kvm? I used qemu
> 
Does not matter, but you need to also recompile kernel with the first patch.

--
			Gleb.
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
George-Cristian Bîrzan Nov. 28, 2012, 8:01 p.m. UTC | #4
On Wed, Nov 28, 2012 at 9:56 PM, Gleb Natapov <gleb@redhat.com> wrote:
> On Wed, Nov 28, 2012 at 09:18:38PM +0200, George-Cristian Bîrzan wrote:
>> On Wed, Nov 28, 2012 at 1:39 PM, Vadim Rozenfeld <vrozenfe@redhat.com> wrote:
>> > There are two patches, one for kvm and another one for qemu.
>>
>> I just realised this. I was supposed to use qemu, or qemu-kvm? I used qemu
>>
> Does not matter, but you need to also recompile kernel with the first patch.

Do I have to recompile the kernel, or just the module? I followed the
instructions at
http://www.linux-kvm.org/page/Code#building_an_external_module_with_older_kernels
but I guess I can do the whole kernel, if it might help.

--
George-Cristian Bîrzan
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Gleb Natapov Nov. 28, 2012, 8:12 p.m. UTC | #5
On Wed, Nov 28, 2012 at 10:01:04PM +0200, George-Cristian Bîrzan wrote:
> On Wed, Nov 28, 2012 at 9:56 PM, Gleb Natapov <gleb@redhat.com> wrote:
> > On Wed, Nov 28, 2012 at 09:18:38PM +0200, George-Cristian Bîrzan wrote:
> >> On Wed, Nov 28, 2012 at 1:39 PM, Vadim Rozenfeld <vrozenfe@redhat.com> wrote:
> >> > There are two patches, one for kvm and another one for qemu.
> >>
> >> I just realised this. I was supposed to use qemu, or qemu-kvm? I used qemu
> >>
> > Does not matter, but you need to also recompile kernel with the first patch.
> 
> Do I have to recompile the kernel, or just the module? I followed the
> instructions at
> http://www.linux-kvm.org/page/Code#building_an_external_module_with_older_kernels
> but I guess I can do the whole kernel, if it might help.
> 
Module is enough, but kvm-kmod is not what you want. Just rebuild the
whole kernel if you do not know how to rebuild only the module for your
distribution's kernel.

--
			Gleb.
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Vadim Rozenfeld Nov. 29, 2012, 11:56 a.m. UTC | #6
On Wednesday, November 28, 2012 09:09:29 PM George-Cristian Bîrzan wrote:
> On Wed, Nov 28, 2012 at 1:39 PM, Vadim Rozenfeld <vrozenfe@redhat.com> 
wrote:
> > On Tuesday, November 27, 2012 11:13:12 PM George-Cristian Bîrzan wrote:
> >> On Tue, Nov 27, 2012 at 10:38 PM, Vadim Rozenfeld <vrozenfe@redhat.com>
> > 
> > wrote:
> >> > I have some code which do both reference time and invariant TSC but it
> >> > will not work after migration. I will send it later today.
> >> 
> >> Do you mean migrating guests? This is not an issue for us.
> > 
> > OK, but don't say I didn't warn you :)
> > 
> > There are two patches, one for kvm and another one for qemu.
> > you will probably need to rebase them.
> > Add "hv_tsc" cpu parameter to activate this feature.
> > you will probably need to deactivate hpet by adding "-no-hpet"
> > parameter as well.
> 
> I've also added +hv_relaxed since then, but this is the command I'm

I would suggest activating relaxed timing for all W2K8R2/Win7 guests.

> using now and there's no change:
> 
> /usr/bin/qemu-kvm -name b691546e-79f8-49c6-a293-81067503a6ad -S -M
> pc-1.2 -enable-kvm -m 16384 -smp 9,sockets=1,cores=9,threads=1 -uuid
> b691546e-79f8-49c6-a293-81067503a6ad -no-user-config -nodefaults
> -chardev
> socket,id=charmonitor,path=/var/lib/libvirt/qemu/b691546e-79f8-49c6-a293-8
> 1067503a6ad.monitor,server,nowait -mon
> chardev=charmonitor,id=monitor,mode=control -rtc base=utc
> -no-hpet -no-shutdown -device
> piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive
> file=/var/lib/libvirt/images/dis-magnetics-2-223101/d8b233c6-8424-4de9-ae3c
> -7c9a60288514,if=none,id=drive-virtio-disk0,format=qcow2,cache=writeback,ai
> o=native -device
> virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=vir
> tio-disk0,bootindex=1 -netdev tap,fd=35,id=hostnet0,vhost=on,vhostfd=36
> -device
> virtio-net-pci,netdev=hostnet0,id=net0,mac=22:2e:fb:a2:36:be,bus=pci.0,addr
> =0x3 -netdev tap,fd=40,id=hostnet1,vhost=on,vhostfd=41 -device
> virtio-net-pci,netdev=hostnet1,id=net1,mac=22:94:44:5a:cb:24,bus=pci.0,addr
> =0x4 -vnc 127.0.0.1:0,password -vga cirrus -device
> virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6 -cpu host,hv_tsc
> 
> I compiled qemu-1.2.0-24 after applying your patch, used the head for
> KVM, and I see no difference. I've tried setting windows'
> useplatformclock on and off, no change either.
> 
> 
> Other than that, was looking into a profiling trace of the software
> running and a lot of time (60%?) is spent calling two functions from
> hal.dll, HalpGetPmTimerSleepModePerfCounter when I disable HPET, and
> HalpHPETProgramRolloverTimer which do point at something related to
> the timers.
> 
It means that hyper-v time stamp source was not activated.
> Any other thing I can try?
> 
> 
> --
> George-Cristian Bîrzan
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
George-Cristian Bîrzan Nov. 29, 2012, 1:45 p.m. UTC | #7
On Thu, Nov 29, 2012 at 1:56 PM, Vadim Rozenfeld <vrozenfe@redhat.com> wrote:
>> I've also added +hv_relaxed since then, but this is the command I'm
>
> I would suggest activating relaxed timing for all W2K8R2/Win7 guests.

Is there any place I can read up on the downsides of this for Linux,
or is Just Better?

>>>> Other than that, was looking into a profiling trace of the software
>> running and a lot of time (60%?) is spent calling two functions from
>> hal.dll, HalpGetPmTimerSleepModePerfCounter when I disable HPET, and
>> HalpHPETProgramRolloverTimer which do point at something related to
>> the timers.
>>
> It means that hyper-v time stamp source was not activated.

I recompiled the whole kernel, with your patch, and while I cannot
check at 70Mbps now, a test stream of 20 seems to do better. Also, now
I don't see any of those functions, which used to account ~60% of the
time spent by the program. I'm waiting for the customer to come back
and start the 'real' stream, but from my tests, time spent in hal.dll
is now an order of magnitude smaller.

--
George-Cristian Bîrzan
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Gleb Natapov Nov. 29, 2012, 1:56 p.m. UTC | #8
On Thu, Nov 29, 2012 at 03:45:52PM +0200, George-Cristian Bîrzan wrote:
> On Thu, Nov 29, 2012 at 1:56 PM, Vadim Rozenfeld <vrozenfe@redhat.com> wrote:
> >> I've also added +hv_relaxed since then, but this is the command I'm
> >
> > I would suggest activating relaxed timing for all W2K8R2/Win7 guests.
> 
> Is there any place I can read up on the downsides of this for Linux,
> or is Just Better?
> 
You shouldn't use hyper-v flags for Linux guests. In theory Linux should
just ignore them, in practice there may be bugs that will prevent Linux
from detecting that it runs as a guest and disable optimizations.

> >>>> Other than that, was looking into a profiling trace of the software
> >> running and a lot of time (60%?) is spent calling two functions from
> >> hal.dll, HalpGetPmTimerSleepModePerfCounter when I disable HPET, and
> >> HalpHPETProgramRolloverTimer which do point at something related to
> >> the timers.
> >>
> > It means that hyper-v time stamp source was not activated.
> 
> I recompiled the whole kernel, with your patch, and while I cannot
> check at 70Mbps now, a test stream of 20 seems to do better. Also, now
> I don't see any of those functions, which used to account ~60% of the
> time spent by the program. I'm waiting for the customer to come back
> and start the 'real' stream, but from my tests, time spent in hal.dll
> is now an order of magnitude smaller.
> 
> --
> George-Cristian Bîrzan

--
			Gleb.
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Vadim Rozenfeld Nov. 29, 2012, 8:34 p.m. UTC | #9
On Thursday, November 29, 2012 03:56:10 PM Gleb Natapov wrote:
> On Thu, Nov 29, 2012 at 03:45:52PM +0200, George-Cristian Bîrzan wrote:
> > On Thu, Nov 29, 2012 at 1:56 PM, Vadim Rozenfeld <vrozenfe@redhat.com> 
wrote:
> > >> I've also added +hv_relaxed since then, but this is the command I'm
> > > 
> > > I would suggest activating relaxed timing for all W2K8R2/Win7 guests.
> > 
> > Is there any place I can read up on the downsides of this for Linux,
> > or is Just Better?
> 
> You shouldn't use hyper-v flags for Linux guests. In theory Linux should
> just ignore them, in practice there may be bugs that will prevent Linux
> from detecting that it runs as a guest and disable optimizations.
> 
As Gleb said, hyper-v flag are relevant to the Windows guests only. 
IIRC spinlocks and vapic should work for Vista and higher. Relaxed timing and
partition reference time work for Win7/W2K8R2.
> > >>>> Other than that, was looking into a profiling trace of the software
> > >> 
> > >> running and a lot of time (60%?) is spent calling two functions from
> > >> hal.dll, HalpGetPmTimerSleepModePerfCounter when I disable HPET, and
> > >> HalpHPETProgramRolloverTimer which do point at something related to
> > >> the timers.
> > > 
> > > It means that hyper-v time stamp source was not activated.
> > 
> > I recompiled the whole kernel, with your patch, and while I cannot
> > check at 70Mbps now, a test stream of 20 seems to do better. Also, now
> > I don't see any of those functions, which used to account ~60% of the
> > time spent by the program. I'm waiting for the customer to come back
> > and start the 'real' stream, but from my tests, time spent in hal.dll
> > is now an order of magnitude smaller.
> > 
> > --
> > George-Cristian Bîrzan
> 
> --
> 			Gleb.
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
diff mbox

Patch

diff --git a/target-i386/cpu.c b/target-i386/cpu.c
index f3708e6..ad77b72 100644
--- a/target-i386/cpu.c
+++ b/target-i386/cpu.c
@@ -1250,6 +1250,8 @@  static int cpu_x86_find_by_name(x86_def_t *x86_cpu_def, const char *cpu_model)
             hyperv_enable_relaxed_timing(true);
         } else if (!strcmp(featurestr, "hv_vapic")) {
             hyperv_enable_vapic_recommended(true);
+        } else if (!strcmp(featurestr, "hv_tsc")) {
+            hyperv_enable_tsc_recommended(true);
         } else {
             fprintf(stderr, "feature string `%s' not in format (+feature|-feature|feature=xyz)\n", featurestr);
             goto error;
diff --git a/target-i386/hyperv.c b/target-i386/hyperv.c
index f284e99..bd581a1 100644
--- a/target-i386/hyperv.c
+++ b/target-i386/hyperv.c
@@ -15,6 +15,12 @@ 
 static bool hyperv_vapic;
 static bool hyperv_relaxed_timing;
 static int hyperv_spinlock_attempts = HYPERV_SPINLOCK_NEVER_RETRY;
+static bool hyperv_tsc;
+
+void hyperv_enable_tsc_recommended(bool val)
+{
+    hyperv_tsc = val;
+}
 
 void hyperv_enable_vapic_recommended(bool val)
 {
@@ -42,12 +48,18 @@  bool hyperv_enabled(void)
 bool hyperv_hypercall_available(void)
 {
     if (hyperv_vapic ||
+        hyperv_tsc ||
         (hyperv_spinlock_attempts != HYPERV_SPINLOCK_NEVER_RETRY)) {
       return true;
     }
     return false;
 }
 
+bool hyperv_tsc_recommended(void)
+{
+    return hyperv_tsc;
+}
+
 bool hyperv_vapic_recommended(void)
 {
     return hyperv_vapic;
diff --git a/target-i386/hyperv.h b/target-i386/hyperv.h
index bacb1d4..94c2d6e 100644
--- a/target-i386/hyperv.h
+++ b/target-i386/hyperv.h
@@ -27,10 +27,12 @@ 
 #endif
 
 #if !defined(CONFIG_USER_ONLY) && defined(CONFIG_KVM)
+void hyperv_enable_tsc_recommended(bool val);
 void hyperv_enable_vapic_recommended(bool val);
 void hyperv_enable_relaxed_timing(bool val);
 void hyperv_set_spinlock_retries(int val);
 #else
+static inline void hyperv_enable_tsc_recommended(bool val) { }
 static inline void hyperv_enable_vapic_recommended(bool val) { }
 static inline void hyperv_enable_relaxed_timing(bool val) { }
 static inline void hyperv_set_spinlock_retries(int val) { }
@@ -38,6 +40,7 @@  static inline void hyperv_set_spinlock_retries(int val) { }
 
 bool hyperv_enabled(void);
 bool hyperv_hypercall_available(void);
+bool hyperv_tsc_recommended(void);
 bool hyperv_vapic_recommended(void);
 bool hyperv_relaxed_timing_enabled(void);
 int hyperv_get_spinlock_retries(void);
diff --git a/target-i386/kvm.c b/target-i386/kvm.c
index 5b18383..dc7f259 100644
--- a/target-i386/kvm.c
+++ b/target-i386/kvm.c
@@ -390,13 +390,17 @@  int kvm_arch_init_vcpu(CPUX86State *env)
     c = &cpuid_data.entries[cpuid_i++];
     memset(c, 0, sizeof(*c));
     c->function = KVM_CPUID_SIGNATURE;
-    if (!hyperv_enabled()) {
-        memcpy(signature, "KVMKVMKVM\0\0\0", 12);
-        c->eax = 0;
-    } else {
-        memcpy(signature, "Microsoft Hv", 12);
+    memcpy(signature, "KVMKVMKVM\0\0\0", 12);
+    if (hyperv_enabled()) {
         c->eax = HYPERV_CPUID_MIN;
     }
+//    if (!hyperv_enabled()) {
+//        memcpy(signature, "KVMKVMKVM\0\0\0", 12);
+//        c->eax = 0;
+//    } else {
+//        memcpy(signature, "Microsoft Hv", 12);
+//        c->eax = HYPERV_CPUID_MIN;
+//    }
     c->ebx = signature[0];
     c->ecx = signature[1];
     c->edx = signature[2];
@@ -427,7 +431,11 @@  int kvm_arch_init_vcpu(CPUX86State *env)
             c->eax |= HV_X64_MSR_HYPERCALL_AVAILABLE;
             c->eax |= HV_X64_MSR_APIC_ACCESS_AVAILABLE;
         }
-
+        if (hyperv_tsc_recommended()) {
+            c->eax |= HV_X64_MSR_HYPERCALL_AVAILABLE;
+            c->eax |= HV_X64_MSR_TIME_REF_COUNT_AVAILABLE;
+            c->eax |= 0x200;
+        }
         c = &cpuid_data.entries[cpuid_i++];
         memset(c, 0, sizeof(*c));
         c->function = HYPERV_CPUID_ENLIGHTMENT_INFO;
@@ -445,14 +453,14 @@  int kvm_arch_init_vcpu(CPUX86State *env)
         c->eax = 0x40;
         c->ebx = 0x40;
 
-        c = &cpuid_data.entries[cpuid_i++];
-        memset(c, 0, sizeof(*c));
-        c->function = KVM_CPUID_SIGNATURE_NEXT;
-        memcpy(signature, "KVMKVMKVM\0\0\0", 12);
-        c->eax = 0;
-        c->ebx = signature[0];
-        c->ecx = signature[1];
-        c->edx = signature[2];
+//        c = &cpuid_data.entries[cpuid_i++];
+//        memset(c, 0, sizeof(*c));
+//        c->function = KVM_CPUID_SIGNATURE_NEXT;
+//        memcpy(signature, "KVMKVMKVM\0\0\0", 12);
+//        c->eax = 0;
+//        c->ebx = signature[0];
+//        c->ecx = signature[1];
+//        c->edx = signature[2];
     }
 
     has_msr_async_pf_en = c->eax & (1 << KVM_FEATURE_ASYNC_PF);