From patchwork Mon Jul 25 13:18:19 2011 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jan Kiszka X-Patchwork-Id: 1004322 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by demeter2.kernel.org (8.14.4/8.14.4) with ESMTP id p6PDISb3020159 for ; Mon, 25 Jul 2011 13:18:28 GMT Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751271Ab1GYNSZ (ORCPT ); Mon, 25 Jul 2011 09:18:25 -0400 Received: from david.siemens.de ([192.35.17.14]:16782 "EHLO david.siemens.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1750827Ab1GYNSY (ORCPT ); Mon, 25 Jul 2011 09:18:24 -0400 Received: from mail1.siemens.de (localhost [127.0.0.1]) by david.siemens.de (8.13.6/8.13.6) with ESMTP id p6PDIJhm012611; Mon, 25 Jul 2011 15:18:20 +0200 Received: from mchn199C.mchp.siemens.de ([139.25.109.49]) by mail1.siemens.de (8.13.6/8.13.6) with ESMTP id p6PDIJxo029724; Mon, 25 Jul 2011 15:18:19 +0200 Message-ID: <4E2D6D1B.8020903@siemens.com> Date: Mon, 25 Jul 2011 15:18:19 +0200 From: Jan Kiszka User-Agent: Mozilla/5.0 (X11; U; Linux i686 (x86_64); de; rv:1.8.1.12) Gecko/20080226 SUSE/2.0.0.12-1.1 Thunderbird/2.0.0.12 Mnenhy/0.7.5.666 MIME-Version: 1.0 To: Gleb Natapov , Vasilis Liaskovitis CC: "kvm@vger.kernel.org" , Markus Armbruster Subject: Re: [PATCH] cpu hotplug issue References: <20110720083507.GS2400@redhat.com> <20110721113342.GB3044@redhat.com> <4E281090.9070300@siemens.com> <20110721115118.GD3044@redhat.com> <20110721124512.GI3044@redhat.com> <4E29577A.9080909@siemens.com> <20110724115647.GR3044@redhat.com> <4E2C4434.1060106@web.de> In-Reply-To: <4E2C4434.1060106@web.de> Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org X-Greylist: IP, sender and recipient auto-whitelisted, not delayed by milter-greylist-4.2.6 (demeter2.kernel.org [140.211.167.43]); Mon, 25 Jul 2011 13:18:28 +0000 (UTC) On 2011-07-24 18:11, Jan Kiszka wrote: >>> I had a closer look and identified two further issues, one generic, one >>> CPU-hotplug-specific: >>> - (qdev) devices that are hotplugged do not receive any reset. That >>> does not only apply to the APIC in case of CPU hotplugging, it is >>> also broken for NICs, storage controllers, etc. when doing PCI >>> hot-add as I just checked via gdb. >>> - CPU hotplugging was always (or at least for a fairly long time), >>> well, fragile as it failed to make CPU thread creation and CPU >>> initialization atomic against APIC addition and other initialization >>> steps. IOW, we need to create CPUs stopped, finish all init work, >>> sync their states completely to the kernel >>> (cpu_synchronize_post_init), and then kick them of. Actually I'm >> Syncing the state to the kernel should be done by vcpu thread, so I it >> cannot be stopped while the sync is done. May be I misunderstood what >> you mean here. > > Stopped first of all means not entering kvm_cpu_exec before the whole > setup and the initial sync are done. > > Syncing the initial state may also happen over the creating context as > long as the vcpus are stopped (analogously to > kvm_cpu_synchronize_post_init). OK, hacks below plus the following three patches make CPU hotplug work again - with some exceptions. Here are the patches: 1. http://thread.gmane.org/gmane.comp.emulators.kvm.devel/76484 2. http://thread.gmane.org/gmane.comp.emulators.qemu/110272 3. http://thread.gmane.org/gmane.comp.emulators.qemu/110426 And here are the hacks (well, the first hunk is clearly a fix, the last one clearly a hack, /me still undecided about the rest): I see two remaining problems: - kvmclock is somehow broken, either in my guest kernel (OpenSUSE HEAD 3.0.0-2) or the host, -cpu host,-kvmclock works around sporadic guest lockups on echo 1 > /sys... - Seabios tends to lock up once every few system_reset after some CPU has been hot-added - also in TCG mode. It seems to dislike any setup of #CPUs > smp_cpus (whatever that implies in details). Jan diff --git a/hw/acpi_piix4.c b/hw/acpi_piix4.c index c30a050..f650250 100644 --- a/hw/acpi_piix4.c +++ b/hw/acpi_piix4.c @@ -92,7 +92,8 @@ static void pm_update_sci(PIIX4PMState *s) ACPI_BITMASK_POWER_BUTTON_ENABLE | ACPI_BITMASK_GLOBAL_LOCK_ENABLE | ACPI_BITMASK_TIMER_ENABLE)) != 0) || - (((s->gpe.sts[0] & s->gpe.en[0]) & PIIX4_PCI_HOTPLUG_STATUS) != 0); + (((s->gpe.sts[0] & s->gpe.en[0]) & + (PIIX4_PCI_HOTPLUG_STATUS | PIIX4_CPU_HOTPLUG_STATUS)) != 0); qemu_set_irq(s->irq, sci_level); /* schedule a timer interruption if needed */ diff --git a/hw/pc.c b/hw/pc.c index c0a88e1..e5371be 100644 --- a/hw/pc.c +++ b/hw/pc.c @@ -42,6 +42,7 @@ #include "kvm.h" #include "blockdev.h" #include "ui/qemu-spice.h" +#include "cpus.h" /* output Bochs bios info messages */ //#define DEBUG_BIOS @@ -936,6 +937,10 @@ CPUState *pc_new_cpu(const char *cpu_model) #endif } + if (vm_running) { + pause_all_vcpus(); + } + env = cpu_init(cpu_model); if (!env) { fprintf(stderr, "Unable to find x86 CPU definition\n"); @@ -947,6 +952,11 @@ CPUState *pc_new_cpu(const char *cpu_model) } qemu_register_reset(pc_cpu_reset, env); pc_cpu_reset(env); + + cpu_synchronize_post_init(env); + if (vm_running) { + resume_all_vcpus(); + } return env; } diff --git a/hw/qdev.c b/hw/qdev.c index 1626131..b91e2c2 100644 --- a/hw/qdev.c +++ b/hw/qdev.c @@ -330,6 +330,7 @@ BusState *sysbus_get_default(void) if (!main_system_bus) { main_system_bus = qbus_create(&system_bus_info, NULL, "main-system-bus"); + main_system_bus->allow_hotplug = 1; } return main_system_bus; }