| Message ID | 4E2D6D1B.8020903@siemens.com (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
On Mon, Jul 25, 2011 at 03:18:19PM +0200, Jan Kiszka wrote: > On 2011-07-24 18:11, Jan Kiszka wrote: > >>> I had a closer look and identified two further issues, one generic, one > >>> CPU-hotplug-specific: > >>> - (qdev) devices that are hotplugged do not receive any reset. That > >>> does not only apply to the APIC in case of CPU hotplugging, it is > >>> also broken for NICs, storage controllers, etc. when doing PCI > >>> hot-add as I just checked via gdb. > >>> - CPU hotplugging was always (or at least for a fairly long time), > >>> well, fragile as it failed to make CPU thread creation and CPU > >>> initialization atomic against APIC addition and other initialization > >>> steps. IOW, we need to create CPUs stopped, finish all init work, > >>> sync their states completely to the kernel > >>> (cpu_synchronize_post_init), and then kick them of. Actually I'm > >> Syncing the state to the kernel should be done by vcpu thread, so I it > >> cannot be stopped while the sync is done. May be I misunderstood what > >> you mean here. > > > > Stopped first of all means not entering kvm_cpu_exec before the whole > > setup and the initial sync are done. > > > > Syncing the initial state may also happen over the creating context as > > long as the vcpus are stopped (analogously to > > kvm_cpu_synchronize_post_init). > > OK, hacks below plus the following three patches make CPU hotplug work > again - with some exceptions. Here are the patches: > > 1. http://thread.gmane.org/gmane.comp.emulators.kvm.devel/76484 > 2. http://thread.gmane.org/gmane.comp.emulators.qemu/110272 > 3. http://thread.gmane.org/gmane.comp.emulators.qemu/110426 > > And here are the hacks (well, the first hunk is clearly a fix, the last > one clearly a hack, /me still undecided about the rest): > > diff --git a/hw/acpi_piix4.c b/hw/acpi_piix4.c > index c30a050..f650250 100644 > --- a/hw/acpi_piix4.c > +++ b/hw/acpi_piix4.c > @@ -92,7 +92,8 @@ static void pm_update_sci(PIIX4PMState *s) > ACPI_BITMASK_POWER_BUTTON_ENABLE | > ACPI_BITMASK_GLOBAL_LOCK_ENABLE | > ACPI_BITMASK_TIMER_ENABLE)) != 0) || > - (((s->gpe.sts[0] & s->gpe.en[0]) & PIIX4_PCI_HOTPLUG_STATUS) != 0); > + (((s->gpe.sts[0] & s->gpe.en[0]) & > + (PIIX4_PCI_HOTPLUG_STATUS | PIIX4_CPU_HOTPLUG_STATUS)) != 0); > > qemu_set_irq(s->irq, sci_level); > /* schedule a timer interruption if needed */ > diff --git a/hw/pc.c b/hw/pc.c > index c0a88e1..e5371be 100644 > --- a/hw/pc.c > +++ b/hw/pc.c > @@ -42,6 +42,7 @@ > #include "kvm.h" > #include "blockdev.h" > #include "ui/qemu-spice.h" > +#include "cpus.h" > > /* output Bochs bios info messages */ > //#define DEBUG_BIOS > @@ -936,6 +937,10 @@ CPUState *pc_new_cpu(const char *cpu_model) > #endif > } > > + if (vm_running) { > + pause_all_vcpus(); > + } > + > env = cpu_init(cpu_model); > if (!env) { > fprintf(stderr, "Unable to find x86 CPU definition\n"); > @@ -947,6 +952,11 @@ CPUState *pc_new_cpu(const char *cpu_model) > } > qemu_register_reset(pc_cpu_reset, env); > pc_cpu_reset(env); > + > + cpu_synchronize_post_init(env); > + if (vm_running) { > + resume_all_vcpus(); > + } > return env; > } > > diff --git a/hw/qdev.c b/hw/qdev.c > index 1626131..b91e2c2 100644 > --- a/hw/qdev.c > +++ b/hw/qdev.c > @@ -330,6 +330,7 @@ BusState *sysbus_get_default(void) > if (!main_system_bus) { > main_system_bus = qbus_create(&system_bus_info, NULL, > "main-system-bus"); > + main_system_bus->allow_hotplug = 1; > } > return main_system_bus; > } > > I see two remaining problems: > - kvmclock is somehow broken, either in my guest kernel (OpenSUSE HEAD > 3.0.0-2) or the host, -cpu host,-kvmclock works around sporadic > guest lockups on echo 1 > /sys... > - Seabios tends to lock up once every few system_reset after some > CPU has been hot-added - also in TCG mode. It seems to dislike any > setup of #CPUs > smp_cpus (whatever that implies in details). > Have you specified maxcpus? Something like -smp 1,maxcpus=4. -- Gleb. -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On 2011-07-25 15:21, Gleb Natapov wrote: > On Mon, Jul 25, 2011 at 03:18:19PM +0200, Jan Kiszka wrote: >> On 2011-07-24 18:11, Jan Kiszka wrote: >>>>> I had a closer look and identified two further issues, one generic, one >>>>> CPU-hotplug-specific: >>>>> - (qdev) devices that are hotplugged do not receive any reset. That >>>>> does not only apply to the APIC in case of CPU hotplugging, it is >>>>> also broken for NICs, storage controllers, etc. when doing PCI >>>>> hot-add as I just checked via gdb. >>>>> - CPU hotplugging was always (or at least for a fairly long time), >>>>> well, fragile as it failed to make CPU thread creation and CPU >>>>> initialization atomic against APIC addition and other initialization >>>>> steps. IOW, we need to create CPUs stopped, finish all init work, >>>>> sync their states completely to the kernel >>>>> (cpu_synchronize_post_init), and then kick them of. Actually I'm >>>> Syncing the state to the kernel should be done by vcpu thread, so I it >>>> cannot be stopped while the sync is done. May be I misunderstood what >>>> you mean here. >>> >>> Stopped first of all means not entering kvm_cpu_exec before the whole >>> setup and the initial sync are done. >>> >>> Syncing the initial state may also happen over the creating context as >>> long as the vcpus are stopped (analogously to >>> kvm_cpu_synchronize_post_init). >> >> OK, hacks below plus the following three patches make CPU hotplug work >> again - with some exceptions. Here are the patches: >> >> 1. http://thread.gmane.org/gmane.comp.emulators.kvm.devel/76484 >> 2. http://thread.gmane.org/gmane.comp.emulators.qemu/110272 >> 3. http://thread.gmane.org/gmane.comp.emulators.qemu/110426 >> >> And here are the hacks (well, the first hunk is clearly a fix, the last >> one clearly a hack, /me still undecided about the rest): >> >> diff --git a/hw/acpi_piix4.c b/hw/acpi_piix4.c >> index c30a050..f650250 100644 >> --- a/hw/acpi_piix4.c >> +++ b/hw/acpi_piix4.c >> @@ -92,7 +92,8 @@ static void pm_update_sci(PIIX4PMState *s) >> ACPI_BITMASK_POWER_BUTTON_ENABLE | >> ACPI_BITMASK_GLOBAL_LOCK_ENABLE | >> ACPI_BITMASK_TIMER_ENABLE)) != 0) || >> - (((s->gpe.sts[0] & s->gpe.en[0]) & PIIX4_PCI_HOTPLUG_STATUS) != 0); >> + (((s->gpe.sts[0] & s->gpe.en[0]) & >> + (PIIX4_PCI_HOTPLUG_STATUS | PIIX4_CPU_HOTPLUG_STATUS)) != 0); >> >> qemu_set_irq(s->irq, sci_level); >> /* schedule a timer interruption if needed */ >> diff --git a/hw/pc.c b/hw/pc.c >> index c0a88e1..e5371be 100644 >> --- a/hw/pc.c >> +++ b/hw/pc.c >> @@ -42,6 +42,7 @@ >> #include "kvm.h" >> #include "blockdev.h" >> #include "ui/qemu-spice.h" >> +#include "cpus.h" >> >> /* output Bochs bios info messages */ >> //#define DEBUG_BIOS >> @@ -936,6 +937,10 @@ CPUState *pc_new_cpu(const char *cpu_model) >> #endif >> } >> >> + if (vm_running) { >> + pause_all_vcpus(); >> + } >> + >> env = cpu_init(cpu_model); >> if (!env) { >> fprintf(stderr, "Unable to find x86 CPU definition\n"); >> @@ -947,6 +952,11 @@ CPUState *pc_new_cpu(const char *cpu_model) >> } >> qemu_register_reset(pc_cpu_reset, env); >> pc_cpu_reset(env); >> + >> + cpu_synchronize_post_init(env); >> + if (vm_running) { >> + resume_all_vcpus(); >> + } >> return env; >> } >> >> diff --git a/hw/qdev.c b/hw/qdev.c >> index 1626131..b91e2c2 100644 >> --- a/hw/qdev.c >> +++ b/hw/qdev.c >> @@ -330,6 +330,7 @@ BusState *sysbus_get_default(void) >> if (!main_system_bus) { >> main_system_bus = qbus_create(&system_bus_info, NULL, >> "main-system-bus"); >> + main_system_bus->allow_hotplug = 1; >> } >> return main_system_bus; >> } >> >> I see two remaining problems: >> - kvmclock is somehow broken, either in my guest kernel (OpenSUSE HEAD >> 3.0.0-2) or the host, -cpu host,-kvmclock works around sporadic >> guest lockups on echo 1 > /sys... >> - Seabios tends to lock up once every few system_reset after some >> CPU has been hot-added - also in TCG mode. It seems to dislike any >> setup of #CPUs > smp_cpus (whatever that implies in details). >> > Have you specified maxcpus? Something like -smp 1,maxcpus=4. Yes, for sure. BTW, cpu_set completely lacks any parameter sanity checks. That interface looks like "for gurus only". Hope we can do better via qdev and properties. Jan
Hi, On Mon, Jul 25, 2011 at 3:18 PM, Jan Kiszka <jan.kiszka@siemens.com> wrote: > > OK, hacks below plus the following three patches make CPU hotplug work > again - with some exceptions. Here are the patches: > > 1. http://thread.gmane.org/gmane.comp.emulators.kvm.devel/76484 > 2. http://thread.gmane.org/gmane.comp.emulators.qemu/110272 > 3. http://thread.gmane.org/gmane.comp.emulators.qemu/110426 > > And here are the hacks (well, the first hunk is clearly a fix, the last > one clearly a hack, /me still undecided about the rest): thanks for these. Testing a bit late here... Applying these 3 patches + hacks/fix on master doesn't solve the initial "CPU: not responding" dmesg hotplug issue on my setup. The ACPI event is delivered correctly, but "echo 1 > /sys/devices/system/cpu/cpuX/online" still fails. e.g. tested with "-smp 1,maxcpus=2" Could you share your qemu-kvm command line? I have been trying 2.6.39 guest kernel (as opposed to 3.0.2) but I doubt that's the issue. Can someone else confirm whether this solves the issue for their setup or not? > I see two remaining problems: > - kvmclock is somehow broken, either in my guest kernel (OpenSUSE HEAD > 3.0.0-2) or the host, -cpu host,-kvmclock works around sporadic > guest lockups on echo 1 > /sys... could you explain this a bit further? specifying "-cpu host,-kvmclock" avoids the lockups? I have tried adding this to my command line but didn't help (but then again my problem is not guest lockups: the onlining fails) > - Seabios tends to lock up once every few system_reset after some > CPU has been hot-added - also in TCG mode. It seems to dislike any > setup of #CPUs > smp_cpus (whatever that implies in details). I always get a lockedup seabios when rebooting after a hotplug attempt with these patches (or with my initial vcpu_dirty workaround FWIW) thanks, - Vasilis -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On 2011-07-27 18:35, Vasilis Liaskovitis wrote: > Hi, > > On Mon, Jul 25, 2011 at 3:18 PM, Jan Kiszka <jan.kiszka@siemens.com> wrote: >> >> OK, hacks below plus the following three patches make CPU hotplug work >> again - with some exceptions. Here are the patches: >> >> 1. http://thread.gmane.org/gmane.comp.emulators.kvm.devel/76484 >> 2. http://thread.gmane.org/gmane.comp.emulators.qemu/110272 >> 3. http://thread.gmane.org/gmane.comp.emulators.qemu/110426 >> >> And here are the hacks (well, the first hunk is clearly a fix, the last >> one clearly a hack, /me still undecided about the rest): > > thanks for these. Testing a bit late here... > Applying these 3 patches + hacks/fix on master doesn't solve the > initial "CPU: not responding" dmesg hotplug issue on my setup. > The ACPI event is delivered correctly, but "echo 1 > > /sys/devices/system/cpu/cpuX/online" still fails. > e.g. tested with "-smp 1,maxcpus=2" > > Could you share your qemu-kvm command line? Nothing special: qemu-system-x86_64 /my/image -m 1G -snapshot -smp 2,maxcpus=3 \ -cpu host,-kvmclock > I have been trying 2.6.39 > guest kernel (as opposed to 3.0.2) but > I doubt that's the issue. > Can someone else confirm whether this solves the issue for their setup or not? > >> I see two remaining problems: >> - kvmclock is somehow broken, either in my guest kernel (OpenSUSE HEAD >> 3.0.0-2) or the host, -cpu host,-kvmclock works around sporadic >> guest lockups on echo 1 > /sys... > > could you explain this a bit further? specifying "-cpu host,-kvmclock" > avoids the lockups? I have tried Yes - apparently. I haven't checked if kvmclock is actually the problem or just changing the boundary conditions so that the issue no longer shows up here. > adding this to my command line but didn't help (but then again my > problem is not guest lockups: the onlining fails) > >> - Seabios tends to lock up once every few system_reset after some >> CPU has been hot-added - also in TCG mode. It seems to dislike any >> setup of #CPUs > smp_cpus (whatever that implies in details). > > I always get a lockedup seabios when rebooting after a hotplug attempt > with these patches (or with my initial vcpu_dirty workaround FWIW) Yes, this problem is most likely independent of the above. I even got those lockups when creating smp_cpus+1 vcpus in pc_cpus_init. Jan
Hi, On Thu, Jul 28, 2011 at 6:52 PM, Jan Kiszka <jan.kiszka@siemens.com> wrote: > On 2011-07-27 18:35, Vasilis Liaskovitis wrote: >> Hi, >> >> On Mon, Jul 25, 2011 at 3:18 PM, Jan Kiszka <jan.kiszka@siemens.com> wrote: >> Applying these 3 patches + hacks/fix on master doesn't solve the >> initial "CPU: not responding" dmesg hotplug issue on my setup. >> The ACPI event is delivered correctly, but "echo 1 > >> /sys/devices/system/cpu/cpuX/online" still fails. >> e.g. tested with "-smp 1,maxcpus=2" >> >> Could you share your qemu-kvm command line? > > Nothing special: > > qemu-system-x86_64 /my/image -m 1G -snapshot -smp 2,maxcpus=3 \ > -cpu host,-kvmclock > ok, I have tested the patched qemu-kvm (guestOS ubuntu 10.10 with 2.6.35-24-generic kernel) with: qemu-system-x86_64 /my/image -m 1G -smp 1,maxcpus=2 \ -cpu host,-kvmclock on 2 different physical hosts. On the first one: vendor_id : GenuineIntel cpu family : 6 model : 37 model name : Intel(R) Core(TM) i3 CPU M 370 @ 2.40GHz cpu hotplug works ok. When onlining (echo 1 > /sys/devices/system/cpu/cpu1/online) dmesg shows: Booting Node 0 Processor 1 APIC 0x1 TSC synchronization [CPU#0 -> CPU#1]: Measured 3573498937792 cycles TSC warp between CPUs, turning off TSC clock. Marking TSC unstable due to check_tsc_sync_source failed kvm-clock: cpu 1, msr 0:1f15901, secondary cpu clock on the second system: vendor_id : AuthenticAMD cpu family : 16 model : 4 model name : AMD Phenom(tm) II X4 955 Processor onlining fails: Booting Node 0 Processor 1 APIC 0x1 CPU1: Not Responding. When I specify -kvmclock on the qemu command line for the AMD system, it seems that the clocksource on the guest is hpet. Dmesg on guest shows "Switching to clocksource hpet" and /sys/devices/system/clocksource/clocksource0/current_clocksource confirms. When I don't specify -kvmclock, then the guest clocksource is actually kvm-clock. But the onlining problem persists. I have also tried a newer guestOS (debian-testing with 2.6.39 or 3.0.0 kernel) on the second system, same problem. Also: what physical host have you used for testing? thanks, - Vasilis -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On 2011-08-02 11:46, Vasilis Liaskovitis wrote: > Hi, > > On Thu, Jul 28, 2011 at 6:52 PM, Jan Kiszka <jan.kiszka@siemens.com> wrote: >> On 2011-07-27 18:35, Vasilis Liaskovitis wrote: >>> Hi, >>> >>> On Mon, Jul 25, 2011 at 3:18 PM, Jan Kiszka <jan.kiszka@siemens.com> wrote: >>> Applying these 3 patches + hacks/fix on master doesn't solve the >>> initial "CPU: not responding" dmesg hotplug issue on my setup. >>> The ACPI event is delivered correctly, but "echo 1 > >>> /sys/devices/system/cpu/cpuX/online" still fails. >>> e.g. tested with "-smp 1,maxcpus=2" >>> >>> Could you share your qemu-kvm command line? >> >> Nothing special: >> >> qemu-system-x86_64 /my/image -m 1G -snapshot -smp 2,maxcpus=3 \ >> -cpu host,-kvmclock >> > > ok, I have tested the patched qemu-kvm (guestOS ubuntu 10.10 with > 2.6.35-24-generic kernel) with: > > qemu-system-x86_64 /my/image -m 1G -smp 1,maxcpus=2 \ > -cpu host,-kvmclock > > on 2 different physical hosts. On the first one: > > vendor_id : GenuineIntel > cpu family : 6 > model : 37 > model name : Intel(R) Core(TM) i3 CPU M 370 @ 2.40GHz > > cpu hotplug works ok. When onlining (echo 1 > > /sys/devices/system/cpu/cpu1/online) dmesg shows: > > Booting Node 0 Processor 1 APIC 0x1 > TSC synchronization [CPU#0 -> CPU#1]: > Measured 3573498937792 cycles TSC warp between CPUs, turning off TSC clock. > Marking TSC unstable due to check_tsc_sync_source failed > kvm-clock: cpu 1, msr 0:1f15901, secondary cpu clock Strange. Why does it still talk about kvm-clock? > > on the second system: > > vendor_id : AuthenticAMD > cpu family : 16 > model : 4 > model name : AMD Phenom(tm) II X4 955 Processor > > onlining fails: > > Booting Node 0 Processor 1 APIC 0x1 > CPU1: Not Responding. > > When I specify -kvmclock on the qemu command line for the AMD system, > it seems that the clocksource on the guest is hpet. > Dmesg on guest shows "Switching to clocksource hpet" and > /sys/devices/system/clocksource/clocksource0/current_clocksource > confirms. > > When I don't specify -kvmclock, then the guest clocksource is actually > kvm-clock. But the onlining problem persists. > > I have also tried a newer guestOS (debian-testing with 2.6.39 or 3.0.0 > kernel) on the second system, same problem. > > Also: what physical host have you used for testing? Intel(R) Core(TM) i7 CPU M 620 @ 2.67GHz FWIW, I've pushed my tree here: git://git.kiszka.org/qemu-kvm.git queues/cpu-hotplug Jan
On Tue, Aug 2, 2011 at 12:24 PM, Jan Kiszka <jan.kiszka@siemens.com> wrote: > FWIW, I've pushed my tree here: > > git://git.kiszka.org/qemu-kvm.git queues/cpu-hotplug > thanks. Your cpu-hotplug tree or today's git-master (commit dacdc4b10bafbb21120e1c24a9665444768ef999) works. Previously I tested the second system with master from the middle of last week (commit 497d8129ad4bca8a3b319ee84229d9a0f9cdd15c) which probably was missing something else in between merges. Apologies. > > Strange. Why does it still talk about kvm-clock? > I do get kvmclock dmesg output when onlining e.g. on the second system as well: [ 89.062098] Booting Node 0 Processor 1 APIC 0x1 [ 89.062100] smpboot cpu 1: start_ip = 98000 [ 89.074229] kvm-clock: cpu 1, msr 0:3fd12f81, secondary cpu clock [ 89.075038] NMI watchdog disabled (cpu1): hardware events not enabled [ 89.076103] Switched to NOHz mode on CPU #1 kvm_register_clock() prints the kvm-clock line, isn't this expected to be called when the new CPU is onlined? thanks, - Vasilis -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
diff --git a/hw/acpi_piix4.c b/hw/acpi_piix4.c index c30a050..f650250 100644 --- a/hw/acpi_piix4.c +++ b/hw/acpi_piix4.c @@ -92,7 +92,8 @@ static void pm_update_sci(PIIX4PMState *s) ACPI_BITMASK_POWER_BUTTON_ENABLE | ACPI_BITMASK_GLOBAL_LOCK_ENABLE | ACPI_BITMASK_TIMER_ENABLE)) != 0) || - (((s->gpe.sts[0] & s->gpe.en[0]) & PIIX4_PCI_HOTPLUG_STATUS) != 0); + (((s->gpe.sts[0] & s->gpe.en[0]) & + (PIIX4_PCI_HOTPLUG_STATUS | PIIX4_CPU_HOTPLUG_STATUS)) != 0); qemu_set_irq(s->irq, sci_level); /* schedule a timer interruption if needed */ diff --git a/hw/pc.c b/hw/pc.c index c0a88e1..e5371be 100644 --- a/hw/pc.c +++ b/hw/pc.c @@ -42,6 +42,7 @@ #include "kvm.h" #include "blockdev.h" #include "ui/qemu-spice.h" +#include "cpus.h" /* output Bochs bios info messages */ //#define DEBUG_BIOS @@ -936,6 +937,10 @@ CPUState *pc_new_cpu(const char *cpu_model) #endif } + if (vm_running) { + pause_all_vcpus(); + } + env = cpu_init(cpu_model); if (!env) { fprintf(stderr, "Unable to find x86 CPU definition\n"); @@ -947,6 +952,11 @@ CPUState *pc_new_cpu(const char *cpu_model) } qemu_register_reset(pc_cpu_reset, env); pc_cpu_reset(env); + + cpu_synchronize_post_init(env); + if (vm_running) { + resume_all_vcpus(); + } return env; } diff --git a/hw/qdev.c b/hw/qdev.c index 1626131..b91e2c2 100644 --- a/hw/qdev.c +++ b/hw/qdev.c @@ -330,6 +330,7 @@ BusState *sysbus_get_default(void) if (!main_system_bus) { main_system_bus = qbus_create(&system_bus_info, NULL, "main-system-bus"); + main_system_bus->allow_hotplug = 1; } return main_system_bus; }