| Message ID | 20160603163519.GO14588@citrix.com (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
>>> On 03.06.16 at 18:35, <wei.liu2@citrix.com> wrote: > I got a patch ready. But QEMU upstream refuses to start on the receiving end > with following error message: > > qemu-system-i386: Unknown savevm section or instance 'cpu_common' 1 > qemu-system-i386: load of migration failed: Invalid argument > > With QEMU traditional HVM guest and PV guest, the guest works fine -- up > and running with all hot plugged cpus available. > > So I think the relevant libxl information is transmitted but we also > need to fix QEMU upstream. But that's a separate issue. Stefano, Anthony, any thoughts here? Thanks, Jan > ---8<--- > From 790ff77c6307b341dec0b4cc5e2d394e42f82e7c Mon Sep 17 00:00:00 2001 > From: Wei Liu <wei.liu2@citrix.com> > Date: Fri, 3 Jun 2016 16:38:32 +0100 > Subject: [PATCH] libxl: update vcpus bitmap in retrieved geust config > > ... because the available vcpu bitmap can change during domain life time > due to cpu hotplug and unplug. > > Reported-by: Jan Beulich <jbeulich@suse.com> > Signed-off-by: Wei Liu <wei.liu2@citrix.com> > --- > tools/libxl/libxl.c | 31 +++++++++++++++++++++++++++++++ > 1 file changed, 31 insertions(+) > > diff --git a/tools/libxl/libxl.c b/tools/libxl/libxl.c > index 006b83f..99617f3 100644 > --- a/tools/libxl/libxl.c > +++ b/tools/libxl/libxl.c > @@ -7270,6 +7270,37 @@ int libxl_retrieve_domain_configuration(libxl_ctx > *ctx, uint32_t domid, > libxl_dominfo_dispose(&info); > } > > + /* VCPUs */ > + { > + libxl_vcpuinfo *vcpus; > + libxl_bitmap *map; > + int nr_vcpus, nr_cpus; > + unsigned int i; > + > + vcpus = libxl_list_vcpu(ctx, domid, &nr_vcpus, &nr_cpus); > + if (!vcpus) { > + LOG(ERROR, "fail to get vcpu list for domain %d", domid); > + rc = ERROR_FAIL; > + goto out; > + } > + > + /* Update the avail_vcpus bitmap accordingly */ > + map = &d_config->b_info.avail_vcpus; > + > + libxl_bitmap_dispose(map); > + > + libxl_bitmap_alloc(ctx, map, nr_vcpus); > + > + libxl_bitmap_init(map); > + > + for (i = 0; i < nr_vcpus; i++) { > + if (vcpus[i].online) > + libxl_bitmap_set(map, i); > + } > + > + libxl_vcpuinfo_list_free(vcpus, nr_vcpus); > + } > + > /* Memory limits: > * > * Currently there are three memory limits: > -- > 2.1.4
On Fri, Jun 03, 2016 at 05:35:20PM +0100, Wei Liu wrote: > On Fri, Jun 03, 2016 at 08:42:11AM -0600, Jan Beulich wrote: > > >>> On 03.06.16 at 15:41, <wei.liu2@citrix.com> wrote: > > > On Fri, Jun 03, 2016 at 02:29:12AM -0600, Jan Beulich wrote: > > >> Ian, Wei, > > >> > > >> is it intentional that rebooting a (HVM) guest after having altered its > > >> vCPU count will reset it back to the vCPU count it was originally > > >> started with? That doesn't seem very natural - if one hotplugs a CPU > > >> into a physical system and then reboots, that CPU will remain there. > > >> > > > > > > This is probably an oversight. > > > > > > I've added this to my list of things to look at after the release. > > > > Thanks! > > > > I got a patch ready. But QEMU upstream refuses to start on the receiving end > with following error message: > > qemu-system-i386: Unknown savevm section or instance 'cpu_common' 1 > qemu-system-i386: load of migration failed: Invalid argument > > With QEMU traditional HVM guest and PV guest, the guest works fine -- up > and running with all hot plugged cpus available. > > So I think the relevant libxl information is transmitted but we also > need to fix QEMU upstream. But that's a separate issue. > > Wei. > > ---8<--- > From 790ff77c6307b341dec0b4cc5e2d394e42f82e7c Mon Sep 17 00:00:00 2001 > From: Wei Liu <wei.liu2@citrix.com> > Date: Fri, 3 Jun 2016 16:38:32 +0100 > Subject: [PATCH] libxl: update vcpus bitmap in retrieved geust config > > ... because the available vcpu bitmap can change during domain life time > due to cpu hotplug and unplug. > > Reported-by: Jan Beulich <jbeulich@suse.com> > Signed-off-by: Wei Liu <wei.liu2@citrix.com> This patch has two issues: 1. The code to allocate bitmap is wrong 2. The check to see if vcpu is available is wrong It happens to work on PV and qemu-trad because they don't really rely on the bitmap provided. #1 is fixed. For #2, I haven't really gotten to the point today on how to correctly get the number of online vcpus. The current issue I discover is that: xl vcpu-set jessie-hvm 4 xl list -l jessie-hvm | less # search for avail_vcpus A vcpu is not really considered online from xen's point of view, unless the guest explicitly activates it, like in guest `echo 1 > .../cpu1/online`. This is still not desirable because it would still cause qemu upstream migration to fail. I will see if there is other way to figure out how many vcpus are there. Off the top of my head I might need to interrogate QEMU for that. I will continue investigation later. Any hint on how to effectively identify online vcpus is very welcomed. Wei.
Use Stefano's new email address On Mon, Jun 06, 2016 at 06:18:06PM +0100, Wei Liu wrote: > On Fri, Jun 03, 2016 at 05:35:20PM +0100, Wei Liu wrote: > > On Fri, Jun 03, 2016 at 08:42:11AM -0600, Jan Beulich wrote: > > > >>> On 03.06.16 at 15:41, <wei.liu2@citrix.com> wrote: > > > > On Fri, Jun 03, 2016 at 02:29:12AM -0600, Jan Beulich wrote: > > > >> Ian, Wei, > > > >> > > > >> is it intentional that rebooting a (HVM) guest after having altered its > > > >> vCPU count will reset it back to the vCPU count it was originally > > > >> started with? That doesn't seem very natural - if one hotplugs a CPU > > > >> into a physical system and then reboots, that CPU will remain there. > > > >> > > > > > > > > This is probably an oversight. > > > > > > > > I've added this to my list of things to look at after the release. > > > > > > Thanks! > > > > > > > I got a patch ready. But QEMU upstream refuses to start on the receiving end > > with following error message: > > > > qemu-system-i386: Unknown savevm section or instance 'cpu_common' 1 > > qemu-system-i386: load of migration failed: Invalid argument > > > > With QEMU traditional HVM guest and PV guest, the guest works fine -- up > > and running with all hot plugged cpus available. > > > > So I think the relevant libxl information is transmitted but we also > > need to fix QEMU upstream. But that's a separate issue. > > > > Wei. > > > > ---8<--- > > From 790ff77c6307b341dec0b4cc5e2d394e42f82e7c Mon Sep 17 00:00:00 2001 > > From: Wei Liu <wei.liu2@citrix.com> > > Date: Fri, 3 Jun 2016 16:38:32 +0100 > > Subject: [PATCH] libxl: update vcpus bitmap in retrieved geust config > > > > ... because the available vcpu bitmap can change during domain life time > > due to cpu hotplug and unplug. > > > > Reported-by: Jan Beulich <jbeulich@suse.com> > > Signed-off-by: Wei Liu <wei.liu2@citrix.com> > > This patch has two issues: > 1. The code to allocate bitmap is wrong > 2. The check to see if vcpu is available is wrong > > It happens to work on PV and qemu-trad because they don't really rely > on the bitmap provided. > > #1 is fixed. > > For #2, I haven't really gotten to the point today on how to correctly > get the number of online vcpus. > > The current issue I discover is that: > > xl vcpu-set jessie-hvm 4 > xl list -l jessie-hvm | less # search for avail_vcpus > > A vcpu is not really considered online from xen's point of view, unless > the guest explicitly activates it, like in guest `echo 1 > > .../cpu1/online`. > > This is still not desirable because it would still cause qemu upstream > migration to fail. I will see if there is other way to figure out how > many vcpus are there. > > Off the top of my head I might need to interrogate QEMU for that. I will > continue investigation later. > > Any hint on how to effectively identify online vcpus is very welcomed. > > Wei.
On 06/06/16 18:20, Wei Liu wrote: > Use Stefano's new email address > > On Mon, Jun 06, 2016 at 06:18:06PM +0100, Wei Liu wrote: >> On Fri, Jun 03, 2016 at 05:35:20PM +0100, Wei Liu wrote: >>> On Fri, Jun 03, 2016 at 08:42:11AM -0600, Jan Beulich wrote: >>>>>>> On 03.06.16 at 15:41, <wei.liu2@citrix.com> wrote: >>>>> On Fri, Jun 03, 2016 at 02:29:12AM -0600, Jan Beulich wrote: >>>>>> Ian, Wei, >>>>>> >>>>>> is it intentional that rebooting a (HVM) guest after having altered its >>>>>> vCPU count will reset it back to the vCPU count it was originally >>>>>> started with? That doesn't seem very natural - if one hotplugs a CPU >>>>>> into a physical system and then reboots, that CPU will remain there. >>>>>> >>>>> This is probably an oversight. >>>>> >>>>> I've added this to my list of things to look at after the release. >>>> Thanks! >>>> >>> I got a patch ready. But QEMU upstream refuses to start on the receiving end >>> with following error message: >>> >>> qemu-system-i386: Unknown savevm section or instance 'cpu_common' 1 >>> qemu-system-i386: load of migration failed: Invalid argument >>> >>> With QEMU traditional HVM guest and PV guest, the guest works fine -- up >>> and running with all hot plugged cpus available. >>> >>> So I think the relevant libxl information is transmitted but we also >>> need to fix QEMU upstream. But that's a separate issue. >>> >>> Wei. >>> >>> ---8<--- >>> From 790ff77c6307b341dec0b4cc5e2d394e42f82e7c Mon Sep 17 00:00:00 2001 >>> From: Wei Liu <wei.liu2@citrix.com> >>> Date: Fri, 3 Jun 2016 16:38:32 +0100 >>> Subject: [PATCH] libxl: update vcpus bitmap in retrieved geust config >>> >>> ... because the available vcpu bitmap can change during domain life time >>> due to cpu hotplug and unplug. >>> >>> Reported-by: Jan Beulich <jbeulich@suse.com> >>> Signed-off-by: Wei Liu <wei.liu2@citrix.com> >> This patch has two issues: >> 1. The code to allocate bitmap is wrong >> 2. The check to see if vcpu is available is wrong >> >> It happens to work on PV and qemu-trad because they don't really rely >> on the bitmap provided. >> >> #1 is fixed. >> >> For #2, I haven't really gotten to the point today on how to correctly >> get the number of online vcpus. >> >> The current issue I discover is that: >> >> xl vcpu-set jessie-hvm 4 >> xl list -l jessie-hvm | less # search for avail_vcpus >> >> A vcpu is not really considered online from xen's point of view, unless >> the guest explicitly activates it, like in guest `echo 1 > >> .../cpu1/online`. >> >> This is still not desirable because it would still cause qemu upstream >> migration to fail. I will see if there is other way to figure out how >> many vcpus are there. >> >> Off the top of my head I might need to interrogate QEMU for that. I will >> continue investigation later. >> >> Any hint on how to effectively identify online vcpus is very welcomed. Why does qemu even care? It has nothing to do with vcpu handling. There should not be any qemu vcpu records in the first place. ~Andrew
>>> On 06.06.16 at 19:18, <wei.liu2@citrix.com> wrote: > The current issue I discover is that: > > xl vcpu-set jessie-hvm 4 > xl list -l jessie-hvm | less # search for avail_vcpus > > A vcpu is not really considered online from xen's point of view, unless > the guest explicitly activates it, like in guest `echo 1 > > .../cpu1/online`. > > This is still not desirable because it would still cause qemu upstream > migration to fail. I will see if there is other way to figure out how > many vcpus are there. > > Off the top of my head I might need to interrogate QEMU for that. I will > continue investigation later. > > Any hint on how to effectively identify online vcpus is very welcomed. How come xl itself doesn't know right from the "xl vcpu-set"? And considering the guest reboot case I originally had the issue with, the guest not activating the vCPU while still running would not mean it won't use it post reboot. E.g. if the guest OS is not CPU-hotplug capable, the reboot may be for the very reason of activating the extra vCPU(s)... Jan
On Tue, Jun 07, 2016 at 12:38:26AM -0600, Jan Beulich wrote: > >>> On 06.06.16 at 19:18, <wei.liu2@citrix.com> wrote: > > The current issue I discover is that: > > > > xl vcpu-set jessie-hvm 4 > > xl list -l jessie-hvm | less # search for avail_vcpus > > > > A vcpu is not really considered online from xen's point of view, unless > > the guest explicitly activates it, like in guest `echo 1 > > > .../cpu1/online`. > > > > This is still not desirable because it would still cause qemu upstream > > migration to fail. I will see if there is other way to figure out how > > many vcpus are there. > > > > Off the top of my head I might need to interrogate QEMU for that. I will > > continue investigation later. > > > > Any hint on how to effectively identify online vcpus is very welcomed. > > How come xl itself doesn't know right from the "xl vcpu-set"? > That is because: 1. libxl only pulls data from various sources when it needs to right at the moment you ask for domain configuration. 2. For QEMU upstream, there is no way in telling whether a hotplug is successful or not -- the return value is ignored because response from QEMU is not sensible. Not saying these things are completely unfixable though. Wei. > And considering the guest reboot case I originally had the issue > with, the guest not activating the vCPU while still running would > not mean it won't use it post reboot. E.g. if the guest OS is not > CPU-hotplug capable, the reboot may be for the very reason of > activating the extra vCPU(s)... > > Jan >
On Mon, Jun 06, 2016 at 06:34:44PM +0100, Andrew Cooper wrote: > On 06/06/16 18:20, Wei Liu wrote: > > Use Stefano's new email address > > > > On Mon, Jun 06, 2016 at 06:18:06PM +0100, Wei Liu wrote: > >> On Fri, Jun 03, 2016 at 05:35:20PM +0100, Wei Liu wrote: > >>> On Fri, Jun 03, 2016 at 08:42:11AM -0600, Jan Beulich wrote: > >>>>>>> On 03.06.16 at 15:41, <wei.liu2@citrix.com> wrote: > >>>>> On Fri, Jun 03, 2016 at 02:29:12AM -0600, Jan Beulich wrote: > >>>>>> Ian, Wei, > >>>>>> > >>>>>> is it intentional that rebooting a (HVM) guest after having altered its > >>>>>> vCPU count will reset it back to the vCPU count it was originally > >>>>>> started with? That doesn't seem very natural - if one hotplugs a CPU > >>>>>> into a physical system and then reboots, that CPU will remain there. > >>>>>> > >>>>> This is probably an oversight. > >>>>> > >>>>> I've added this to my list of things to look at after the release. > >>>> Thanks! > >>>> > >>> I got a patch ready. But QEMU upstream refuses to start on the receiving end > >>> with following error message: > >>> > >>> qemu-system-i386: Unknown savevm section or instance 'cpu_common' 1 > >>> qemu-system-i386: load of migration failed: Invalid argument > >>> > >>> With QEMU traditional HVM guest and PV guest, the guest works fine -- up > >>> and running with all hot plugged cpus available. > >>> > >>> So I think the relevant libxl information is transmitted but we also > >>> need to fix QEMU upstream. But that's a separate issue. > >>> > >>> Wei. > >>> > >>> ---8<--- > >>> From 790ff77c6307b341dec0b4cc5e2d394e42f82e7c Mon Sep 17 00:00:00 2001 > >>> From: Wei Liu <wei.liu2@citrix.com> > >>> Date: Fri, 3 Jun 2016 16:38:32 +0100 > >>> Subject: [PATCH] libxl: update vcpus bitmap in retrieved geust config > >>> > >>> ... because the available vcpu bitmap can change during domain life time > >>> due to cpu hotplug and unplug. > >>> > >>> Reported-by: Jan Beulich <jbeulich@suse.com> > >>> Signed-off-by: Wei Liu <wei.liu2@citrix.com> > >> This patch has two issues: > >> 1. The code to allocate bitmap is wrong > >> 2. The check to see if vcpu is available is wrong > >> > >> It happens to work on PV and qemu-trad because they don't really rely > >> on the bitmap provided. > >> > >> #1 is fixed. > >> > >> For #2, I haven't really gotten to the point today on how to correctly > >> get the number of online vcpus. > >> > >> The current issue I discover is that: > >> > >> xl vcpu-set jessie-hvm 4 > >> xl list -l jessie-hvm | less # search for avail_vcpus > >> > >> A vcpu is not really considered online from xen's point of view, unless > >> the guest explicitly activates it, like in guest `echo 1 > > >> .../cpu1/online`. > >> > >> This is still not desirable because it would still cause qemu upstream > >> migration to fail. I will see if there is other way to figure out how > >> many vcpus are there. > >> > >> Off the top of my head I might need to interrogate QEMU for that. I will > >> continue investigation later. > >> > >> Any hint on how to effectively identify online vcpus is very welcomed. > > Why does qemu even care? It has nothing to do with vcpu handling. > There should not be any qemu vcpu records in the first place. > IIRC upstream rejected the idea of having no cpu attached to a platform. Wei. > ~Andrew
Wei Liu writes ("Re: [Xen-devel] "xl vcpu-set" not persistent across reboot?"):
> On Mon, Jun 06, 2016 at 06:34:44PM +0100, Andrew Cooper wrote:
> > Why does qemu even care? It has nothing to do with vcpu handling.
> > There should not be any qemu vcpu records in the first place.
>
> IIRC upstream rejected the idea of having no cpu attached to a platform.
That doesn't explain why the number of vcpus that qemu believes in has
to have anything to do with the number of vcpus that the guest has.
The qemu vcpu state is a dummy, regardless of how many of it there
are, surely ?
Ian.
On Tue, Jun 14, 2016 at 05:34:22PM +0100, Ian Jackson wrote: > Wei Liu writes ("Re: [Xen-devel] "xl vcpu-set" not persistent across reboot?"): > > On Mon, Jun 06, 2016 at 06:34:44PM +0100, Andrew Cooper wrote: > > > Why does qemu even care? It has nothing to do with vcpu handling. > > > There should not be any qemu vcpu records in the first place. > > > > IIRC upstream rejected the idea of having no cpu attached to a platform. > > That doesn't explain why the number of vcpus that qemu believes in has > to have anything to do with the number of vcpus that the guest has. > > The qemu vcpu state is a dummy, regardless of how many of it there > are, surely ? > What Andrew means is that QEMU shouldn't have kept the CPU state structures in the first place. My response explains why that is not possible from a QEMU upstream point of view. Hence the unfortunate fact is that we need to live with it for now. To start QEMU we need to create a bunch of dummy CPUs to keep QEMU happy. All those dummy states need to be kept. The guest is irrelevant here. We don't care about what guest thinks, but we need to keep toolstack side view consistent so that migration and save / restore won't fail. Wei. > Ian.
Wei Liu writes ("Re: [Xen-devel] "xl vcpu-set" not persistent across reboot?"):
> What Andrew means is that QEMU shouldn't have kept the CPU state
> structures in the first place. My response explains why that is not
> possible from a QEMU upstream point of view.
I don't think it addresses my point.
> Hence the unfortunate fact is that we need to live with it for now. To
> start QEMU we need to create a bunch of dummy CPUs to keep QEMU happy.
> All those dummy states need to be kept.
Why do we need one dummy state per actual vcpu rather than just one
dummy state no matter how many vcpus ?
Or is qemu involved in hvm cpu hotplug ?
Ian.
On 14/06/16 17:57, Ian Jackson wrote: > Wei Liu writes ("Re: [Xen-devel] "xl vcpu-set" not persistent across reboot?"): >> What Andrew means is that QEMU shouldn't have kept the CPU state >> structures in the first place. My response explains why that is not >> possible from a QEMU upstream point of view. > I don't think it addresses my point. > >> Hence the unfortunate fact is that we need to live with it for now. To >> start QEMU we need to create a bunch of dummy CPUs to keep QEMU happy. >> All those dummy states need to be kept. > Why do we need one dummy state per actual vcpu rather than just one > dummy state no matter how many vcpus ? > > Or is qemu involved in hvm cpu hotplug ? Qemu has nothing to do with vcpus at all, and should not have vcpu state in its migration stream when acting as a device model. Someone needs to fix this upstream in Qemu, and that is the *only* viable option here. ~Andrew
On Tue, Jun 14, 2016 at 05:57:00PM +0100, Ian Jackson wrote: > Wei Liu writes ("Re: [Xen-devel] "xl vcpu-set" not persistent across reboot?"): > > What Andrew means is that QEMU shouldn't have kept the CPU state > > structures in the first place. My response explains why that is not > > possible from a QEMU upstream point of view. > > I don't think it addresses my point. > > > Hence the unfortunate fact is that we need to live with it for now. To > > start QEMU we need to create a bunch of dummy CPUs to keep QEMU happy. > > All those dummy states need to be kept. > > Why do we need one dummy state per actual vcpu rather than just one > dummy state no matter how many vcpus ? > We can't because ... > Or is qemu involved in hvm cpu hotplug ? > when doing hotplug, libxl uses QMP command to tell QEMU to create CPUs. Whether this can be changed I will let Anthony and Stefano to answer. Wei. > Ian.
On Tue, Jun 14, 2016 at 05:59:30PM +0100, Andrew Cooper wrote: > On 14/06/16 17:57, Ian Jackson wrote: > > Wei Liu writes ("Re: [Xen-devel] "xl vcpu-set" not persistent across reboot?"): > >> What Andrew means is that QEMU shouldn't have kept the CPU state > >> structures in the first place. My response explains why that is not > >> possible from a QEMU upstream point of view. > > I don't think it addresses my point. > > > >> Hence the unfortunate fact is that we need to live with it for now. To > >> start QEMU we need to create a bunch of dummy CPUs to keep QEMU happy. > >> All those dummy states need to be kept. > > Why do we need one dummy state per actual vcpu rather than just one > > dummy state no matter how many vcpus ? > > > > Or is qemu involved in hvm cpu hotplug ? > > Qemu has nothing to do with vcpus at all, and should not have vcpu state > in its migration stream when acting as a device model. > > Someone needs to fix this upstream in Qemu, and that is the *only* > viable option here. > Correct me if I'm wrong -- are you suggesting to add a board / platform without any CPU to QEMU? If that's the suggestion I think QEMU maintainers have made clear they won't accept such thing. Wei. > ~Andrew
On Tue, Jun 14, 2016 at 05:57:00PM +0100, Ian Jackson wrote: > Wei Liu writes ("Re: [Xen-devel] "xl vcpu-set" not persistent across reboot?"): > > What Andrew means is that QEMU shouldn't have kept the CPU state > > structures in the first place. My response explains why that is not > > possible from a QEMU upstream point of view. > > I don't think it addresses my point. > > > Hence the unfortunate fact is that we need to live with it for now. To > > start QEMU we need to create a bunch of dummy CPUs to keep QEMU happy. > > All those dummy states need to be kept. > > Why do we need one dummy state per actual vcpu rather than just one > dummy state no matter how many vcpus ? > > Or is qemu involved in hvm cpu hotplug ? It is, QEMU does the notification to the guest of a newly available CPU via an ACPI GPE or something like that. I think QEMU manage the bitfield of online CPUs then notify the guest about changes.
diff --git a/tools/libxl/libxl.c b/tools/libxl/libxl.c index 006b83f..99617f3 100644 --- a/tools/libxl/libxl.c +++ b/tools/libxl/libxl.c @@ -7270,6 +7270,37 @@ int libxl_retrieve_domain_configuration(libxl_ctx *ctx, uint32_t domid, libxl_dominfo_dispose(&info); } + /* VCPUs */ + { + libxl_vcpuinfo *vcpus; + libxl_bitmap *map; + int nr_vcpus, nr_cpus; + unsigned int i; + + vcpus = libxl_list_vcpu(ctx, domid, &nr_vcpus, &nr_cpus); + if (!vcpus) { + LOG(ERROR, "fail to get vcpu list for domain %d", domid); + rc = ERROR_FAIL; + goto out; + } + + /* Update the avail_vcpus bitmap accordingly */ + map = &d_config->b_info.avail_vcpus; + + libxl_bitmap_dispose(map); + + libxl_bitmap_alloc(ctx, map, nr_vcpus); + + libxl_bitmap_init(map); + + for (i = 0; i < nr_vcpus; i++) { + if (vcpus[i].online) + libxl_bitmap_set(map, i); + } + + libxl_vcpuinfo_list_free(vcpus, nr_vcpus); + } + /* Memory limits: * * Currently there are three memory limits: