| Message ID | 20241106130331.205020-3-salil.mehta@huawei.com (mailing list archive) |
|---|---|
| State | New |
| Headers | show |
| Series | Fixes CPUs AML & acpi-bios-tables to be x86 backward compatible | expand |
On Wed, 6 Nov 2024 13:03:30 +0000 Salil Mehta <salil.mehta@huawei.com> wrote: > Checking `is_present` first can break x86 migration from new Qemu > version to old Qemu. This is because CPRS Bit is not defined in the > older Qemu register block and will always be 0 resulting in check always > failing. Reversing the logic to first check `is_enabled` can alleviate > below problem: > > - If ((\_SB.PCI0.PRES.CPEN == One)) > - { > - Local0 = 0x0F > + If ((\_SB.PCI0.PRES.CPRS == One)) > + { > + If ((\_SB.PCI0.PRES.CPEN == One)) > + { > + Local0 = 0x0F > + } > + Else > + { > + Local0 = 0x0D > + } > } > > Suggested-by: Igor Mammedov <imammedo@redhat.com> 'Reported-by' maybe, but certainly not suggested. After more thinking and given presence is system wide that doesn't change at runtime, I don't see any reason for introducing presence bit as ABI (and undocumented on top of that). Instead changing AML code to account for it would be better, something like this: diff --git a/include/hw/acpi/cpu.h b/include/hw/acpi/cpu.h index 32654dc274..4a3e591120 100644 --- a/include/hw/acpi/cpu.h +++ b/include/hw/acpi/cpu.h @@ -55,6 +55,7 @@ void cpu_hotplug_hw_init(MemoryRegion *as, Object *owner, typedef struct CPUHotplugFeatures { bool acpi_1_compatible; bool has_legacy_cphp; + bool always_present_cpus; bool fw_unplugs_cpu; const char *smi_path; } CPUHotplugFeatures; diff --git a/hw/acpi/cpu.c b/hw/acpi/cpu.c index 5cb60ca8bc..2bcce2b31c 100644 --- a/hw/acpi/cpu.c +++ b/hw/acpi/cpu.c @@ -452,15 +452,16 @@ void build_cpus_aml(Aml *table, MachineState *machine, CPUHotplugFeatures opts, method = aml_method(CPU_STS_METHOD, 1, AML_SERIALIZED); { + uint8_t default_sta = opts.always_present_cpus?0xd:0; Aml *idx = aml_arg(0); - Aml *sta = aml_local(0); + Aml *sta = aml_local(default_sta); aml_append(method, aml_acquire(ctrl_lock, 0xFFFF)); aml_append(method, aml_store(idx, cpu_selector)); aml_append(method, aml_store(zero, sta)); ifctx = aml_if(aml_equal(is_enabled, one)); { - aml_append(ifctx, aml_store(aml_int(0xF), sta)); + aml_append(ifctx, aml_or(aml_int(0xF), sta, sta)); } aml_append(method, ifctx); aml_append(method, aml_release(ctrl_lock)) then for ARM set CPUHotplugFeatures::always_present_cpus = true to get present flag always enabled After that revert _all_ other presence bit related changes that were just merged. (I did ask to get rid of that in previous reviews but it came back again for no good reason). > Message-ID: <20241106100047.18901c9d@imammedo.users.ipa.redhat.com> > Signed-off-by: Salil Mehta <salil.mehta@huawei.com> > --- > hw/acpi/cpu.c | 22 +++++++++++----------- > 1 file changed, 11 insertions(+), 11 deletions(-) > > diff --git a/hw/acpi/cpu.c b/hw/acpi/cpu.c > index 23443f09a5..b2f7a2b27e 100644 > --- a/hw/acpi/cpu.c > +++ b/hw/acpi/cpu.c > @@ -490,22 +490,22 @@ void build_cpus_aml(Aml *table, MachineState *machine, CPUHotplugFeatures opts, > aml_append(method, aml_acquire(ctrl_lock, 0xFFFF)); > aml_append(method, aml_store(idx, cpu_selector)); > aml_append(method, aml_store(zero, sta)); > - ifctx = aml_if(aml_equal(is_present, one)); > + ifctx = aml_if(aml_equal(is_enabled, one)); > { > - ifctx2 = aml_if(aml_equal(is_enabled, one)); > - { > - /* cpu is present and enabled */ > - aml_append(ifctx2, aml_store(aml_int(0xF), sta)); > - } > - aml_append(ifctx, ifctx2); > - else_ctx = aml_else(); > + /* cpu is present and enabled */ > + aml_append(ifctx, aml_store(aml_int(0xF), sta)); > + } > + aml_append(method, ifctx); > + else_ctx = aml_else(); > + { > + ifctx2 = aml_if(aml_equal(is_present, one)); > { > /* cpu is present but disabled */ > - aml_append(else_ctx, aml_store(aml_int(0xD), sta)); > + aml_append(ifctx2, aml_store(aml_int(0xD), sta)); > } > - aml_append(ifctx, else_ctx); > + aml_append(else_ctx, ifctx2); > } > - aml_append(method, ifctx); > + aml_append(method, else_ctx); > aml_append(method, aml_release(ctrl_lock)); > aml_append(method, aml_return(sta)); > }
Hi Igor, > From: qemu-arm-bounces+salil.mehta=huawei.com@nongnu.org <qemu- > arm-bounces+salil.mehta=huawei.com@nongnu.org> On Behalf Of Igor > Mammedov > Sent: Wednesday, November 6, 2024 1:57 PM > To: Salil Mehta <salil.mehta@huawei.com> > > On Wed, 6 Nov 2024 13:03:30 +0000 > Salil Mehta <salil.mehta@huawei.com> wrote: > > > Checking `is_present` first can break x86 migration from new Qemu > > version to old Qemu. This is because CPRS Bit is not defined in the > > older Qemu register block and will always be 0 resulting in check > > always failing. Reversing the logic to first check `is_enabled` can > > alleviate below problem: > > > > - If ((\_SB.PCI0.PRES.CPEN == One)) > > - { > > - Local0 = 0x0F > > + If ((\_SB.PCI0.PRES.CPRS == One)) > > + { > > + If ((\_SB.PCI0.PRES.CPEN == One)) > > + { > > + Local0 = 0x0F > > + } > > + Else > > + { > > + Local0 = 0x0D > > + } > > } > > > > Suggested-by: Igor Mammedov <imammedo@redhat.com> > 'Reported-by' maybe, but certainly not suggested. No issues. I can change. > > After more thinking and given presence is system wide that doesn't change > at runtime, I don't see any reason for introducing presence bit as ABI (and > undocumented on top of that). This is a wrong assumption. Presence bit can change in future. We have taken into account this aspect by design in the kernel code as well. Both virtual and physical CPU hot plug can co-exists or entirely as sole features. This is a requirement. > > Instead changing AML code to account for it would be better, something like > this: > > diff --git a/include/hw/acpi/cpu.h b/include/hw/acpi/cpu.h index > 32654dc274..4a3e591120 100644 > --- a/include/hw/acpi/cpu.h > +++ b/include/hw/acpi/cpu.h > @@ -55,6 +55,7 @@ void cpu_hotplug_hw_init(MemoryRegion *as, Object > *owner, typedef struct CPUHotplugFeatures { > bool acpi_1_compatible; > bool has_legacy_cphp; > + bool always_present_cpus; This has to be fetched from architecture code. Please see other changes in the V3 patch-set. > bool fw_unplugs_cpu; > const char *smi_path; > } CPUHotplugFeatures; > diff --git a/hw/acpi/cpu.c b/hw/acpi/cpu.c index 5cb60ca8bc..2bcce2b31c > 100644 > --- a/hw/acpi/cpu.c > +++ b/hw/acpi/cpu.c > @@ -452,15 +452,16 @@ void build_cpus_aml(Aml *table, MachineState > *machine, CPUHotplugFeatures opts, > > method = aml_method(CPU_STS_METHOD, 1, AML_SERIALIZED); > { > + uint8_t default_sta = opts.always_present_cpus?0xd:0; > Aml *idx = aml_arg(0); > - Aml *sta = aml_local(0); > + Aml *sta = aml_local(default_sta); > > aml_append(method, aml_acquire(ctrl_lock, 0xFFFF)); > aml_append(method, aml_store(idx, cpu_selector)); > aml_append(method, aml_store(zero, sta)); > ifctx = aml_if(aml_equal(is_enabled, one)); > { > - aml_append(ifctx, aml_store(aml_int(0xF), sta)); > + aml_append(ifctx, aml_or(aml_int(0xF), sta, sta)); > } > aml_append(method, ifctx); > aml_append(method, aml_release(ctrl_lock)) > > then for ARM set > CPUHotplugFeatures::always_present_cpus = true to get present flag > always enabled We MUST fetch this from architecture code as this can dynamically change in future and hence, we need to keep that flexibility > > After that revert _all_ other presence bit related changes that were just > merged. > (I did ask to get rid of that in previous reviews but it came back again for no > good reason). The CPUs AML in the V2 patch-set would have broken the x86 functionality. Thanks Salil.
On Wed, 6 Nov 2024 14:45:42 +0000 Salil Mehta <salil.mehta@huawei.com> wrote: > Hi Igor, > > > From: qemu-arm-bounces+salil.mehta=huawei.com@nongnu.org <qemu- > > arm-bounces+salil.mehta=huawei.com@nongnu.org> On Behalf Of Igor > > Mammedov > > Sent: Wednesday, November 6, 2024 1:57 PM > > To: Salil Mehta <salil.mehta@huawei.com> > > > > On Wed, 6 Nov 2024 13:03:30 +0000 > > Salil Mehta <salil.mehta@huawei.com> wrote: > > > > > Checking `is_present` first can break x86 migration from new Qemu > > > version to old Qemu. This is because CPRS Bit is not defined in the > > > older Qemu register block and will always be 0 resulting in check > > > always failing. Reversing the logic to first check `is_enabled` can > > > alleviate below problem: > > > > > > - If ((\_SB.PCI0.PRES.CPEN == One)) > > > - { > > > - Local0 = 0x0F > > > + If ((\_SB.PCI0.PRES.CPRS == One)) > > > + { > > > + If ((\_SB.PCI0.PRES.CPEN == One)) > > > + { > > > + Local0 = 0x0F > > > + } > > > + Else > > > + { > > > + Local0 = 0x0D > > > + } > > > } > > > > > > Suggested-by: Igor Mammedov <imammedo@redhat.com> > > 'Reported-by' maybe, but certainly not suggested. > > > No issues. I can change. > > > > > > After more thinking and given presence is system wide that doesn't change > > at runtime, I don't see any reason for introducing presence bit as ABI (and > > undocumented on top of that). > > > This is a wrong assumption. Presence bit can change in future. We have taken > into account this aspect by design in the kernel code as well. Both virtual Above does imply that support for really hotpluggable CPUs might be implemented in the future. Can you point to relevant kernel code/patches? > and physical CPU hot plug can co-exists or entirely as sole features. This is > a requirement. I don't see any _must_ requirements here whatsoever. If/when ARM reaches point where standby and hotplug cpus are mixed within VM instance, we might have to expose presence bit to guest dynamically but it is totally not needed within observable future and premature. Your cpu class presence check also works target-wise just with more boiler code + not needed presence bit ABI for guest side, The same as my suggestion only from other side. But regardless of that as long as machine has doesn't mix always present CPUs with hotpluggable ones within the same instance, changing AML side as I've just suggested works. If ARM ever gets real cpu hotplug as your comment above hints, my suggestion also works fine. With only difference that board code would turn off always_present_cpus if hotpluggable CPUs are used instead of standby. > > > > > Instead changing AML code to account for it would be better, something like > > this: > > > > diff --git a/include/hw/acpi/cpu.h b/include/hw/acpi/cpu.h index > > 32654dc274..4a3e591120 100644 > > --- a/include/hw/acpi/cpu.h > > +++ b/include/hw/acpi/cpu.h > > @@ -55,6 +55,7 @@ void cpu_hotplug_hw_init(MemoryRegion *as, Object > > *owner, typedef struct CPUHotplugFeatures { > > bool acpi_1_compatible; > > bool has_legacy_cphp; > > + bool always_present_cpus; > > > This has to be fetched from architecture code. Please see other changes in the V3 > patch-set. I've seen, that and it doesn't have to be fetched dynamically. In my opinion the series was not ready to be merged (Michael probably picked it by mistake). We don't really need this in 9.2 as it is only ARM cpu 'hotplug' related, and the later is not ready for 9.2. I'd prefer the series being reverted and we continue improving series, instead of rushing it and fixing broken thing up. > > > > bool fw_unplugs_cpu; > > const char *smi_path; > > } CPUHotplugFeatures; > > diff --git a/hw/acpi/cpu.c b/hw/acpi/cpu.c index 5cb60ca8bc..2bcce2b31c > > 100644 > > --- a/hw/acpi/cpu.c > > +++ b/hw/acpi/cpu.c > > @@ -452,15 +452,16 @@ void build_cpus_aml(Aml *table, MachineState > > *machine, CPUHotplugFeatures opts, > > > > method = aml_method(CPU_STS_METHOD, 1, AML_SERIALIZED); > > { > > + uint8_t default_sta = opts.always_present_cpus?0xd:0; > > Aml *idx = aml_arg(0); > > - Aml *sta = aml_local(0); > > + Aml *sta = aml_local(default_sta); > > > > aml_append(method, aml_acquire(ctrl_lock, 0xFFFF)); > > aml_append(method, aml_store(idx, cpu_selector)); > > aml_append(method, aml_store(zero, sta)); > > ifctx = aml_if(aml_equal(is_enabled, one)); > > { > > - aml_append(ifctx, aml_store(aml_int(0xF), sta)); > > + aml_append(ifctx, aml_or(aml_int(0xF), sta, sta)); > > } > > aml_append(method, ifctx); > > aml_append(method, aml_release(ctrl_lock)) > > > > then for ARM set > > CPUHotplugFeatures::always_present_cpus = true to get present flag > > always enabled > > > We MUST fetch this from architecture code as this can dynamically change in > future and hence, we need to keep that flexibility CPUHotplugFeatures::always_present_cpus can be set dynamically per VM instance or per Machine type. > > > > After that revert _all_ other presence bit related changes that were just > > merged. > > (I did ask to get rid of that in previous reviews but it came back again for no > > good reason). > > > The CPUs AML in the V2 patch-set would have broken the x86 functionality. Frankly speaking, I don't see much progress. All that happens on respins is flipping between what I asked to remove before to some earlier concept. And all of that for the purpose to workaround/accommodate fake cpu hotplug hacks. > Thanks > Salil. >
Hi Igor, Thanks for replying back and the reviews. Please find my replies inline. > From: Igor Mammedov <imammedo@redhat.com> > Sent: Wednesday, November 6, 2024 4:08 PM > To: Salil Mehta <salil.mehta@huawei.com> > > On Wed, 6 Nov 2024 14:45:42 +0000 > Salil Mehta <salil.mehta@huawei.com> wrote: > > > Hi Igor, > > > > > From: qemu-arm-bounces+salil.mehta=huawei.com@nongnu.org > <qemu- > > > arm-bounces+salil.mehta=huawei.com@nongnu.org> On Behalf Of Igor > > > Mammedov > > > Sent: Wednesday, November 6, 2024 1:57 PM > > > To: Salil Mehta <salil.mehta@huawei.com> > > > > > > On Wed, 6 Nov 2024 13:03:30 +0000 > > > Salil Mehta <salil.mehta@huawei.com> wrote: > > > > > > > Checking `is_present` first can break x86 migration from new Qemu > > > > version to old Qemu. This is because CPRS Bit is not defined in > > > the > older Qemu register block and will always be 0 resulting in > > > check > always failing. Reversing the logic to first check > > > `is_enabled` can > alleviate below problem: > > > > > > > > - If ((\_SB.PCI0.PRES.CPEN == One)) > > > > - { > > > > - Local0 = 0x0F > > > > + If ((\_SB.PCI0.PRES.CPRS == One)) > > > > + { > > > > + If ((\_SB.PCI0.PRES.CPEN == One)) > > > > + { > > > > + Local0 = 0x0F > > > > + } > > > > + Else > > > > + { > > > > + Local0 = 0x0D > > > > + } > > > > } > > > > > > > > Suggested-by: Igor Mammedov <imammedo@redhat.com> > 'Reported-by' > > > maybe, but certainly not suggested. > > > > > > No issues. I can change. > > > > > > > > > > After more thinking and given presence is system wide that doesn't > > > change at runtime, I don't see any reason for introducing presence > > > bit as ABI (and undocumented on top of that). > > > > > > This is a wrong assumption. Presence bit can change in future. We have > > taken into account this aspect by design in the kernel code as well. > > Both virtual > > Above does imply that support for really hotpluggable CPUs might be > implemented in the future. > Can you point to relevant kernel code/patches? Let me make it clear so that there is no confusion, there is no support of physical "CPU" hot-plug on ARM platforms right now and nor will there be any in future as it does not makes sense to have. The point I made in the patches is about hot-plug was at different granularity which has not been denied by ARM. > > > and physical CPU hot plug can co-exists or entirely as sole features. > > This is a requirement. > > > I don't see any _must_ requirements here whatsoever. If/when ARM > reaches point where standby and hotplug cpus are mixed within VM > instance, we might have to expose presence bit to guest dynamically but it > is totally not needed within observable future and premature. > Your cpu class presence check also works target-wise just with more boiler > code > + not needed presence bit ABI for guest side, > The same as my suggestion only from other side. > > But regardless of that as long as machine has doesn't mix always present > CPUs with hotpluggable ones within the same instance, changing AML side > as I've just suggested works. Sure, I'm not disagreeing. It will work by adding the flag you've suggested but it will work even by not adding any flag and not defining a `persistent` hook for any architecture. > > If ARM ever gets real cpu hotplug as your comment above hints, my > suggestion also works fine. With only difference that board code would turn > off always_present_cpus if hotpluggable CPUs are used instead of standby. Sure, but is it necessary to define a new flag when you can do even without it? Having few lines of extra code (literally 2-3 only) should not be a major cause of worry, especially, if it makes design more clear and easy to understand. This is not a performance code either. Again, I appreciate your compact logic. I'm not disagreeing with it. > > > Instead changing AML code to account for it would be better, > > > something like > > > this: > > > > > > diff --git a/include/hw/acpi/cpu.h b/include/hw/acpi/cpu.h index > > > 32654dc274..4a3e591120 100644 > > > --- a/include/hw/acpi/cpu.h > > > +++ b/include/hw/acpi/cpu.h > > > @@ -55,6 +55,7 @@ void cpu_hotplug_hw_init(MemoryRegion *as, > Object > > > *owner, typedef struct CPUHotplugFeatures { > > > bool acpi_1_compatible; > > > bool has_legacy_cphp; > > > + bool always_present_cpus; > > > > > > This has to be fetched from architecture code. Please see other > > changes in the V3 patch-set. > > I've seen, that and it doesn't have to be fetched dynamically. Agreed, it is not necessary to be fetched. Hence, if you do not define the hook. In later case, it assumes the existing logic, which x86 has been following. It will work. > In my opinion the series was not ready to be merged (Michael probably > picked it by mistake). > > We don't really need this in 9.2 as it is only ARM cpu 'hotplug' > related, and the later is not ready for 9.2. > I'd prefer the series being reverted and we continue improving series, > instead of rushing it and fixing broken thing up. > > > > > > > > > bool fw_unplugs_cpu; > > > const char *smi_path; > > > } CPUHotplugFeatures; > > > diff --git a/hw/acpi/cpu.c b/hw/acpi/cpu.c index > > > 5cb60ca8bc..2bcce2b31c > > > 100644 > > > --- a/hw/acpi/cpu.c > > > +++ b/hw/acpi/cpu.c > > > @@ -452,15 +452,16 @@ void build_cpus_aml(Aml *table, > MachineState > > > *machine, CPUHotplugFeatures opts, > > > > > > method = aml_method(CPU_STS_METHOD, 1, AML_SERIALIZED); > > > { > > > + uint8_t default_sta = opts.always_present_cpus?0xd:0; > > > Aml *idx = aml_arg(0); > > > - Aml *sta = aml_local(0); > > > + Aml *sta = aml_local(default_sta); > > > > > > aml_append(method, aml_acquire(ctrl_lock, 0xFFFF)); > > > aml_append(method, aml_store(idx, cpu_selector)); > > > aml_append(method, aml_store(zero, sta)); > > > ifctx = aml_if(aml_equal(is_enabled, one)); > > > { > > > - aml_append(ifctx, aml_store(aml_int(0xF), sta)); > > > + aml_append(ifctx, aml_or(aml_int(0xF), sta, sta)); > > > } > > > aml_append(method, ifctx); > > > aml_append(method, aml_release(ctrl_lock)) > > > > > > then for ARM set > > > CPUHotplugFeatures::always_present_cpus = true to get present flag > > > always enabled > > > > > > We MUST fetch this from architecture code as this can dynamically > > change in future and hence, we need to keep that flexibility > > CPUHotplugFeatures::always_present_cpus can be set dynamically per VM > instance or per Machine type. Yes, but you need a code for that and I'm not sure what is being saved by introducing an extra flag then? > > > After that revert _all_ other presence bit related changes that > > > were just merged. > > > (I did ask to get rid of that in previous reviews but it came back > > > again for no good reason). > > > > > > The CPUs AML in the V2 patch-set would have broken the x86 > functionality. > > Frankly speaking, I don't see much progress. All that happens on respins is > flipping between what I asked to remove before to some earlier concept. I think then you've missed the code which addressed one of your key concerns in the V1 patch-set that would have broken x86 migration i.e. presence of the `is_enabled` state in the CPU Hot-plug VMSD. That has been removed. Maybe you would like to go through the change log of the V3 patch-set https://lore.kernel.org/qemu-devel/20241018163118.0ae01a84@imammedo.users.ipa.redhat.com/ Below is the relevant excerpt from your many comments: [...] > .fields = (const VMStateField[]) { > VMSTATE_BOOL(is_inserting, AcpiCpuStatus), > VMSTATE_BOOL(is_removing, AcpiCpuStatus), > + VMSTATE_BOOL(is_present, AcpiCpuStatus), > + VMSTATE_BOOL(is_enabled, AcpiCpuStatus), that's likely will break x86 migration, but before bothering peterx, maybe we won't need this hunk if is_enabled is migrated as part of vCPU state. [...] > And all of that for the purpose to workaround/accommodate fake cpu > hotplug hacks. I've to respectfully disagree on this. This is your assumption which is far from reality. The accompanying main series of this will never have vCPUs which are *not* present. BTW, these changes have been tested by Oracle folks with that series and are known to work. https://lore.kernel.org/qemu-devel/C4FEC9E7-69DB-428A-A85F-30170F98814B@oracle.com/ Thanks! Best regards Salil.
On Wed, 6 Nov 2024 19:05:15 +0000 Salil Mehta <salil.mehta@huawei.com> wrote: > Hi Igor, > > Thanks for replying back and the reviews. Please find my replies > inline. > > > From: Igor Mammedov <imammedo@redhat.com> > > Sent: Wednesday, November 6, 2024 4:08 PM > > To: Salil Mehta <salil.mehta@huawei.com> > > > > On Wed, 6 Nov 2024 14:45:42 +0000 > > Salil Mehta <salil.mehta@huawei.com> wrote: > > > > > Hi Igor, > > > > > > > From: qemu-arm-bounces+salil.mehta=huawei.com@nongnu.org > > <qemu- > > > > arm-bounces+salil.mehta=huawei.com@nongnu.org> On Behalf Of Igor > > > > Mammedov > > > > Sent: Wednesday, November 6, 2024 1:57 PM > > > > To: Salil Mehta <salil.mehta@huawei.com> > > > > > > > > On Wed, 6 Nov 2024 13:03:30 +0000 > > > > Salil Mehta <salil.mehta@huawei.com> wrote: > > > > > > > > > Checking `is_present` first can break x86 migration from new Qemu > > > > > version to old Qemu. This is because CPRS Bit is not defined in > > > > the > older Qemu register block and will always be 0 resulting in > > > > check > always failing. Reversing the logic to first check > > > > `is_enabled` can > alleviate below problem: > > > > > > > > > > - If ((\_SB.PCI0.PRES.CPEN == One)) > > > > > - { > > > > > - Local0 = 0x0F > > > > > + If ((\_SB.PCI0.PRES.CPRS == One)) > > > > > + { > > > > > + If ((\_SB.PCI0.PRES.CPEN == One)) > > > > > + { > > > > > + Local0 = 0x0F > > > > > + } > > > > > + Else > > > > > + { > > > > > + Local0 = 0x0D > > > > > + } > > > > > } > > > > > > > > > > Suggested-by: Igor Mammedov <imammedo@redhat.com> > > 'Reported-by' > > > > maybe, but certainly not suggested. > > > > > > > > > No issues. I can change. > > > > > > > > > > > > > > After more thinking and given presence is system wide that doesn't > > > > change at runtime, I don't see any reason for introducing presence > > > > bit as ABI (and undocumented on top of that). > > > > > > > > > This is a wrong assumption. Presence bit can change in future. We have > > > taken into account this aspect by design in the kernel code as well. > > > Both virtual > > > > Above does imply that support for really hotpluggable CPUs might be > > implemented in the future. > > Can you point to relevant kernel code/patches? > > > Let me make it clear so that there is no confusion, there is no support of > physical "CPU" hot-plug on ARM platforms right now and nor will there be > any in future as it does not makes sense to have. The point I made in the > patches is about hot-plug was at different granularity which has not been > denied by ARM. _STA and ACPI cphp registers are per logical CPU and can't be anything else per spec. how different granularity is relevant here? > > > > > > and physical CPU hot plug can co-exists or entirely as sole features. > > > This is a requirement. > > > > > > I don't see any _must_ requirements here whatsoever. If/when ARM > > reaches point where standby and hotplug cpus are mixed within VM > > instance, we might have to expose presence bit to guest dynamically but it > > is totally not needed within observable future and premature. > > > > Your cpu class presence check also works target-wise just with more boiler > > code > > + not needed presence bit ABI for guest side, > > The same as my suggestion only from other side. > > > > But regardless of that as long as machine has doesn't mix always present > > CPUs with hotpluggable ones within the same instance, changing AML side > > as I've just suggested works. > > > Sure, I'm not disagreeing. It will work by adding the flag you've suggested > but it will work even by not adding any flag and not defining a `persistent` > hook for any architecture. hook is more complicated and hidden way, than directly passing down configuration data to routine that builds AML where it is needed, but that's cosmetics in the end. However there is more to your hook approach, it exposes is_present bit in cphp flag register which is ABI to guest which we will have to maintain forever and guest will do not necessary round-trip to QEMU to query it, while alternative is much simpler AML change without any ABI changes to care about. The point is we shouldn't add new ABI unless we have to. In this case new ABI (is_presence flag) is not _must_, and much simpler static AML change is sufficient. > > > > If ARM ever gets real cpu hotplug as your comment above hints, my > > suggestion also works fine. With only difference that board code would turn > > off always_present_cpus if hotpluggable CPUs are used instead of standby. > > > Sure, but is it necessary to define a new flag when you can do even without it? > Having few lines of extra code (literally 2-3 only) should not be a major cause of > worry, especially, if it makes design more clear and easy to understand. This is > not a performance code either. > > Again, I appreciate your compact logic. I'm not disagreeing with it. > > > > > > Instead changing AML code to account for it would be better, > > > > something like > > > > this: > > > > > > > > diff --git a/include/hw/acpi/cpu.h b/include/hw/acpi/cpu.h index > > > > 32654dc274..4a3e591120 100644 > > > > --- a/include/hw/acpi/cpu.h > > > > +++ b/include/hw/acpi/cpu.h > > > > @@ -55,6 +55,7 @@ void cpu_hotplug_hw_init(MemoryRegion *as, > > Object > > > > *owner, typedef struct CPUHotplugFeatures { > > > > bool acpi_1_compatible; > > > > bool has_legacy_cphp; > > > > + bool always_present_cpus; > > > > > > > > > This has to be fetched from architecture code. Please see other > > > changes in the V3 patch-set. > > > > I've seen, that and it doesn't have to be fetched dynamically. > > > Agreed, it is not necessary to be fetched. Hence, if you do not define the > hook. In later case, it assumes the existing logic, which x86 has been following. > It will work. It is better to get rid of not necessary hook altogether, and replace it with a simple AML change. > > In my opinion the series was not ready to be merged (Michael probably > > picked it by mistake). > > > > We don't really need this in 9.2 as it is only ARM cpu 'hotplug' > > related, and the later is not ready for 9.2. > > I'd prefer the series being reverted and we continue improving series, > > instead of rushing it and fixing broken thing up. > > > > > > > > > > > > > > bool fw_unplugs_cpu; > > > > const char *smi_path; > > > > } CPUHotplugFeatures; > > > > diff --git a/hw/acpi/cpu.c b/hw/acpi/cpu.c index > > > > 5cb60ca8bc..2bcce2b31c > > > > 100644 > > > > --- a/hw/acpi/cpu.c > > > > +++ b/hw/acpi/cpu.c > > > > @@ -452,15 +452,16 @@ void build_cpus_aml(Aml *table, > > MachineState > > > > *machine, CPUHotplugFeatures opts, > > > > > > > > method = aml_method(CPU_STS_METHOD, 1, AML_SERIALIZED); > > > > { > > > > + uint8_t default_sta = opts.always_present_cpus?0xd:0; > > > > Aml *idx = aml_arg(0); > > > > - Aml *sta = aml_local(0); > > > > + Aml *sta = aml_local(default_sta); > > > > > > > > aml_append(method, aml_acquire(ctrl_lock, 0xFFFF)); > > > > aml_append(method, aml_store(idx, cpu_selector)); > > > > aml_append(method, aml_store(zero, sta)); > > > > ifctx = aml_if(aml_equal(is_enabled, one)); > > > > { > > > > - aml_append(ifctx, aml_store(aml_int(0xF), sta)); > > > > + aml_append(ifctx, aml_or(aml_int(0xF), sta, sta)); > > > > } > > > > aml_append(method, ifctx); > > > > aml_append(method, aml_release(ctrl_lock)) > > > > > > > > then for ARM set > > > > CPUHotplugFeatures::always_present_cpus = true to get present flag > > > > always enabled > > > > > > > > > We MUST fetch this from architecture code as this can dynamically > > > change in future and hence, we need to keep that flexibility > > > > CPUHotplugFeatures::always_present_cpus can be set dynamically per VM > > instance or per Machine type. > > > Yes, but you need a code for that and I'm not sure what is being saved by > introducing an extra flag then? beside snippet, I've suggested. You need to delete all is_present callback related logic, than it save quite a bit, and not only on lines of code. I guess, I need to send a patch to get point across. > > > > > > After that revert _all_ other presence bit related changes that > > > > were just merged. > > > > (I did ask to get rid of that in previous reviews but it came back > > > > again for no good reason). > > > > > > > > > The CPUs AML in the V2 patch-set would have broken the x86 > > functionality. > > > > Frankly speaking, I don't see much progress. All that happens on respins is > > flipping between what I asked to remove before to some earlier concept. > > > I think then you've missed the code which addressed one of your key concerns > in the V1 patch-set that would have broken x86 migration i.e. presence of the > `is_enabled` state in the CPU Hot-plug VMSD. That has been removed. Maybe > you would like to go through the change log of the V3 patch-set > > https://lore.kernel.org/qemu-devel/20241018163118.0ae01a84@imammedo.users.ipa.redhat.com/ > > Below is the relevant excerpt from your many comments: > > [...] > > .fields = (const VMStateField[]) { > > VMSTATE_BOOL(is_inserting, AcpiCpuStatus), > > VMSTATE_BOOL(is_removing, AcpiCpuStatus), > > + VMSTATE_BOOL(is_present, AcpiCpuStatus), > > + VMSTATE_BOOL(is_enabled, AcpiCpuStatus), > > that's likely will break x86 migration, > but before bothering peterx, maybe we won't need this hunk if > is_enabled is migrated as part of vCPU state. > what has been done in v3 is moving is_present/is_enabled, elsewhere (callbacks in this case). While argument was that both are not necessary to begin with. > [...] > > > > And all of that for the purpose to workaround/accommodate fake cpu > > hotplug hacks. > > > I've to respectfully disagree on this. This is your assumption which is far from > reality. The accompanying main series of this will never have vCPUs which are > *not* present. Reality of your last posted main series (v5), disagrees with what you are saying here [PATCH RFC V5 12/30] arm/virt: Release objects for *disabled* possible vCPUs after init https://patchew.org/QEMU/20241015100012.254223-1-salil.mehta@huawei.com/20241015100012.254223-13-salil.mehta@huawei.com/ after this patch, new is_present hook would let you bury the lie about CPU being present inside ARM specific code. > BTW, these changes have been tested by Oracle folks with that > series and are known to work. > > https://lore.kernel.org/qemu-devel/C4FEC9E7-69DB-428A-A85F-30170F98814B@oracle.com/ One can write anything and it can even work somehow, that however doesn't mean it's something merge-able, maintainable, ...whatever else come to mind... > > Thanks! > > Best regards > Salil. > >
Hi Igor, Many thanks for taking time to reply. > From: qemu-arm-bounces+salil.mehta=huawei.com@nongnu.org <qemu- > arm-bounces+salil.mehta=huawei.com@nongnu.org> On Behalf Of Igor > Mammedov > Sent: Thursday, November 7, 2024 4:57 PM > To: Salil Mehta <salil.mehta@huawei.com> > > On Wed, 6 Nov 2024 19:05:15 +0000 > Salil Mehta <salil.mehta@huawei.com> wrote: > > > Hi Igor, > > > > Thanks for replying back and the reviews. Please find my replies > > inline. > > > > > From: Igor Mammedov <imammedo@redhat.com> > > > Sent: Wednesday, November 6, 2024 4:08 PM > > > To: Salil Mehta <salil.mehta@huawei.com> > > > > > > On Wed, 6 Nov 2024 14:45:42 +0000 > > > Salil Mehta <salil.mehta@huawei.com> wrote: > > > > > > > Hi Igor, > > > > > > > > > From: qemu-arm-bounces+salil.mehta=huawei.com@nongnu.org > > > <qemu- > > > > > arm-bounces+salil.mehta=huawei.com@nongnu.org> On Behalf Of > > > Igor > > Mammedov > > Sent: Wednesday, November 6, 2024 1:57 PM > > > > > To: Salil Mehta <salil.mehta@huawei.com> > > > > On Wed, 6 > > > Nov 2024 13:03:30 +0000 > > Salil Mehta <salil.mehta@huawei.com> > > > wrote: > > > > > > > > > > > Checking `is_present` first can break x86 migration from new > > > Qemu > > > version to old Qemu. This is because CPRS Bit is not > > > defined in > > the > older Qemu register block and will always be > > > 0 resulting in > > check > always failing. Reversing the logic to > > > first check > > `is_enabled` can > alleviate below problem: > > > > > > > > > > > > - If ((\_SB.PCI0.PRES.CPEN == One)) > > > > > > - { > > > > > > - Local0 = 0x0F > > > > > > + If ((\_SB.PCI0.PRES.CPRS == One)) > > > > > > + { > > > > > > + If ((\_SB.PCI0.PRES.CPEN == One)) > > > > > > + { > > > > > > + Local0 = 0x0F > > > > > > + } > > > > > > + Else > > > > > > + { > > > > > > + Local0 = 0x0D > > > > > > + } > > > > > > } > > > > > > > > > > > > Suggested-by: Igor Mammedov <imammedo@redhat.com> > > > 'Reported-by' > > > > > maybe, but certainly not suggested. > > > > > > > > > > > > No issues. I can change. > > > > > > > > > > > > > > > > > > After more thinking and given presence is system wide that > > > doesn't > > change at runtime, I don't see any reason for > > > introducing presence > > bit as ABI (and undocumented on top of that). > > > > > > > > > > > > This is a wrong assumption. Presence bit can change in future. We > > > have > taken into account this aspect by design in the kernel code as > well. > > > > Both virtual > > > > > > Above does imply that support for really hotpluggable CPUs might be > > > implemented in the future. > > > Can you point to relevant kernel code/patches? > > > > > > Let me make it clear so that there is no confusion, there is no > > support of physical "CPU" hot-plug on ARM platforms right now and nor > > will there be any in future as it does not makes sense to have. The > > point I made in the patches is about hot-plug was at different > > granularity which has not been denied by ARM. > > _STA and ACPI cphp registers are per logical CPU and can't be anything else > per spec. > > how different granularity is relevant here? It is because hot-plug at socket level (or similar level) has not been denied. > > > > and physical CPU hot plug can co-exists or entirely as sole features. > > > > This is a requirement. > > > > > > > > > I don't see any _must_ requirements here whatsoever. If/when ARM > > > reaches point where standby and hotplug cpus are mixed within VM > > > instance, we might have to expose presence bit to guest dynamically > > > but it is totally not needed within observable future and premature. > > > > > > > Your cpu class presence check also works target-wise just with more > > > boiler code + not needed presence bit ABI for guest side, The > > > same as my suggestion only from other side. > > > > > > But regardless of that as long as machine has doesn't mix always > > > present CPUs with hotpluggable ones within the same instance, > > > changing AML side as I've just suggested works. > > > > > > Sure, I'm not disagreeing. It will work by adding the flag you've > > suggested but it will work even by not adding any flag and not > > defining a `persistent` hook for any architecture. > > hook is more complicated and hidden way, than directly passing down > configuration data to routine that builds AML where it is needed, but that's > cosmetics in the end. It is indeed an abstraction and an intentional one because decision whether some or all CPUs should remain present even after unplug action is specific to an architecture. Hence, should be left with that part. > > However there is more to your hook approach, it exposes is_present bit in > cphp flag register which is ABI to guest which we will have to maintain > forever and guest will do not necessary round-trip to QEMU to query it, > while alternative is much simpler AML change without any ABI changes to > care about. I understand your predicament about ABI but I've to respectfully disagree on the assumption that guest will never round trip and check for the presence bit. > > The point is we shouldn't add new ABI unless we have to. Ok sure. point taken. > In this case new ABI (is_presence flag) is not _must_, and much simpler > static AML change is sufficient. Agreed, there can be multiple ways to get rid of it . But I'm not sure if its simpler or easier to maintain because CPUs AML code has become even more obscure by that compact logic. It is not easy to understand at the first look. This is not a performance leg and maintainability should take over the precedence. But I understand your primary concern now (i.e. to get rid of CPU_PRESENT Bit) and allow me to propose in the lines of what you want but maybe in a slightly different way. I would still like to retain hooks though. > > > If ARM ever gets real cpu hotplug as your comment above hints, my > > > suggestion also works fine. With only difference that board code > > > would turn off always_present_cpus if hotpluggable CPUs are used > instead of standby. > > > > > > Sure, but is it necessary to define a new flag when you can do even > without it? > > Having few lines of extra code (literally 2-3 only) should not be a > > major cause of worry, especially, if it makes design more clear and > > easy to understand. This is not a performance code either. > > > > Again, I appreciate your compact logic. I'm not disagreeing with it. > > > > > > > > > Instead changing AML code to account for it would be better, > > > > > something like > > this: > > > > > > > > > > diff --git a/include/hw/acpi/cpu.h b/include/hw/acpi/cpu.h > > > index > > 32654dc274..4a3e591120 100644 > > --- > > > a/include/hw/acpi/cpu.h > > +++ b/include/hw/acpi/cpu.h > > @@ > > > -55,6 +55,7 @@ void cpu_hotplug_hw_init(MemoryRegion *as, Object > > > > > > *owner, typedef struct CPUHotplugFeatures { > > > > > bool acpi_1_compatible; > > > > > bool has_legacy_cphp; > > > > > + bool always_present_cpus; > > > > > > > > > > > > This has to be fetched from architecture code. Please see other > > > > changes in the V3 patch-set. > > > > > > I've seen, that and it doesn't have to be fetched dynamically. > > > > > > Agreed, it is not necessary to be fetched. Hence, if you do not define > > the hook. In later case, it assumes the existing logic, which x86 has been > following. > > It will work. > > It is better to get rid of not necessary hook altogether, and replace it with a > simple AML change. We still need a way to initialize flags in the architecture specific way. Hooks are clean way to fetch that architecture specific state rather fiddling with flags at multiple levels (initialization, migration etc) and you will be duplicating CPU states at ACPI level and QOM vCPU object level . To be frank that's not a very good design. That said, I believe your end goal of getting rid of the change in the ABI can still be achieved. > > > In my opinion the series was not ready to be merged (Michael > > > probably picked it by mistake). > > > > > > We don't really need this in 9.2 as it is only ARM cpu 'hotplug' > > > related, and the later is not ready for 9.2. > > > I'd prefer the series being reverted and we continue improving > > > series, instead of rushing it and fixing broken thing up. > > > > > > > > > > > > > > > > > > > bool fw_unplugs_cpu; > > > > > const char *smi_path; > > > > > } CPUHotplugFeatures; > > > > > diff --git a/hw/acpi/cpu.c b/hw/acpi/cpu.c index > > > > > 5cb60ca8bc..2bcce2b31c > > 100644 > > --- a/hw/acpi/cpu.c > > > > > +++ b/hw/acpi/cpu.c > > @@ -452,15 +452,16 @@ void > > > build_cpus_aml(Aml *table, MachineState > > *machine, > > > CPUHotplugFeatures opts, > > > > > > > method = aml_method(CPU_STS_METHOD, 1, > AML_SERIALIZED); > > > > > { > > > > > + uint8_t default_sta = opts.always_present_cpus?0xd:0; > > > > > Aml *idx = aml_arg(0); > > > > > - Aml *sta = aml_local(0); > > > > > + Aml *sta = aml_local(default_sta); > > > > > > > > > > aml_append(method, aml_acquire(ctrl_lock, 0xFFFF)); > > > > > aml_append(method, aml_store(idx, cpu_selector)); > > > > > aml_append(method, aml_store(zero, sta)); > > > > > ifctx = aml_if(aml_equal(is_enabled, one)); > > > > > { > > > > > - aml_append(ifctx, aml_store(aml_int(0xF), sta)); > > > > > + aml_append(ifctx, aml_or(aml_int(0xF), sta, sta)); > > > > > } > > > > > aml_append(method, ifctx); > > > > > aml_append(method, aml_release(ctrl_lock)) > > > > > > > > > > then for ARM set > > > > > CPUHotplugFeatures::always_present_cpus = true to get present > flag > > > > > always enabled > > > > > > > > > > > > We MUST fetch this from architecture code as this can dynamically > > > > change in future and hence, we need to keep that flexibility > > > > > > CPUHotplugFeatures::always_present_cpus can be set dynamically per > > > VM instance or per Machine type. > > > > > > Yes, but you need a code for that and I'm not sure what is being saved > > by introducing an extra flag then? > > beside snippet, I've suggested. You need to delete all is_present callback > related logic, than it save quite a bit, and not only on lines of code. > > I guess, I need to send a patch to get point across. There is some misunderstanding. We are not maintaining any `is_present` state at the ACPI level. Both flags `is_present` and `is_enabled` were removed in V3 patch-set. > > > > > After that revert _all_ other presence bit related changes > > > that > > were just merged. > > > > > (I did ask to get rid of that in previous reviews but it came > > > back > > again for no good reason). > > > > > > > > > > > > The CPUs AML in the V2 patch-set would have broken the x86 > > > functionality. > > > > > > Frankly speaking, I don't see much progress. All that happens on > > > respins is flipping between what I asked to remove before to some > earlier concept. > > > > > > I think then you've missed the code which addressed one of your key > > concerns in the V1 patch-set that would have broken x86 migration i.e. > > presence of the `is_enabled` state in the CPU Hot-plug VMSD. That has > > been removed. Maybe you would like to go through the change log of the > > V3 patch-set > > > > https://lore.kernel.org/qemu- > devel/20241018163118.0ae01a84@imammedo.us > > ers.ipa.redhat.com/ > > > > Below is the relevant excerpt from your many comments: > > > > [...] > > > .fields = (const VMStateField[]) { > > > VMSTATE_BOOL(is_inserting, AcpiCpuStatus), > > > VMSTATE_BOOL(is_removing, AcpiCpuStatus), > > > + VMSTATE_BOOL(is_present, AcpiCpuStatus), > > > + VMSTATE_BOOL(is_enabled, AcpiCpuStatus), > > > > that's likely will break x86 migration, but before bothering peterx, > > maybe we won't need this hunk if is_enabled is migrated as part of > > vCPU state. > > > > what has been done in v3 is moving is_present/is_enabled, elsewhere > (callbacks in this case). While argument was that both are not necessary to > begin with. We need to have a way to check whether CPU is _STA.Enabled or not and the meaning of the `disabled` is best decided by the architecture specific code not the ACPI code. It could mean different across architectures and the implementations. We need to keep that flexibility in design. The thin Qemu ACPI code is just an interfacing logic with the guest. It is best not to duplicate the state of the QOM objects inside the ACPI. That's a bad design practice because we will have to worry about keeping those states consistent at both the places all the time whether it is during initialization or migration. > > > [...] > > > > > > > And all of that for the purpose to workaround/accommodate fake cpu > > > hotplug hacks. > > > > > > I've to respectfully disagree on this. This is your assumption which > > is far from reality. The accompanying main series of this will never > > have vCPUs which are > > *not* present. > > Reality of your last posted main series (v5), disagrees with what you are > saying here > > [PATCH RFC V5 12/30] arm/virt: Release objects for *disabled* possible > vCPUs after init > https://patchew.org/QEMU/20241015100012.254223-1- > salil.mehta@huawei.com/20241015100012.254223-13- > salil.mehta@huawei.com/ I think you missed what I mentioned above " The accompanying main series...". To get more information about that series, please check the details mentioned in the cover-letter of the V3 patch-set (link below) which clearly says, https://lore.kernel.org/qemu-devel/20241103102419.202225-1-salil.mehta@huawei.com/ Excerpt from the cover letter: ======================== [...] This patch-set has been tested alongside ARM-specific vCPU hotplug changes (included in the upcoming RFC V6 [4]), and hot(un)plug functionalities performed as expected which concerns this patch-set. Please have a look. [...] References ========== [...] [4] Upcoming RFC V6, Support of Virtual CPU Hotplug for ARMv8 Arch Link: https://github.com/salil-mehta/qemu/commits/virt-cpuhp-armv8/rfc-v6-rc5 [...] > after this patch, new is_present hook would let you bury the lie about CPU > being present inside ARM specific code. I would request you to check the facts again! (please see previous pointers) ACPI logic should not worry about what is happening inside architecture code. It is beyond its realms and should be by design. It is an interfacing upper layer gelling the Guest with the VMM. > > BTW, these changes have been tested by Oracle folks with that series > > and are known to work. > > > > https://lore.kernel.org/qemu-devel/C4FEC9E7-69DB-428A-A85F- > 30170F98814 > > B@oracle.com/ > > One can write anything and it can even work somehow, that however > doesn't mean it's something merge-able, maintainable, ...whatever else > come to mind... It would be very helpful if you can `objectively` point the problems in the main series so that there is a context and we can have a productive discussion there. There is always a scope of improvement. Many thanks! Best regards Salil.
On Thu, 7 Nov 2024 18:59:07 +0000 Salil Mehta <salil.mehta@huawei.com> wrote: > Hi Igor, > > Many thanks for taking time to reply. > > > From: qemu-arm-bounces+salil.mehta=huawei.com@nongnu.org <qemu- > > arm-bounces+salil.mehta=huawei.com@nongnu.org> On Behalf Of Igor > > Mammedov > > Sent: Thursday, November 7, 2024 4:57 PM > > To: Salil Mehta <salil.mehta@huawei.com> > > > > On Wed, 6 Nov 2024 19:05:15 +0000 > > Salil Mehta <salil.mehta@huawei.com> wrote: > > > > > Hi Igor, > > > > > > Thanks for replying back and the reviews. Please find my replies > > > inline. > > > > > > > From: Igor Mammedov <imammedo@redhat.com> > > > > Sent: Wednesday, November 6, 2024 4:08 PM > > > > To: Salil Mehta <salil.mehta@huawei.com> > > > > > > > > On Wed, 6 Nov 2024 14:45:42 +0000 > > > > Salil Mehta <salil.mehta@huawei.com> wrote: > > > > > > > > > Hi Igor, > > > > > > > > > > > From: qemu-arm-bounces+salil.mehta=huawei.com@nongnu.org > > > > <qemu- > > > > > > arm-bounces+salil.mehta=huawei.com@nongnu.org> On Behalf Of > > > > Igor > > Mammedov > > Sent: Wednesday, November 6, 2024 1:57 PM > > > > > > To: Salil Mehta <salil.mehta@huawei.com> > > > > On Wed, 6 > > > > Nov 2024 13:03:30 +0000 > > Salil Mehta <salil.mehta@huawei.com> > > > > wrote: > > > > > > > > > > > > > Checking `is_present` first can break x86 migration from new > > > > Qemu > > > version to old Qemu. This is because CPRS Bit is not > > > > defined in > > the > older Qemu register block and will always be > > > > 0 resulting in > > check > always failing. Reversing the logic to > > > > first check > > `is_enabled` can > alleviate below problem: > > > > > > > > > > > > > > - If ((\_SB.PCI0.PRES.CPEN == One)) > > > > > > > - { > > > > > > > - Local0 = 0x0F > > > > > > > + If ((\_SB.PCI0.PRES.CPRS == One)) > > > > > > > + { > > > > > > > + If ((\_SB.PCI0.PRES.CPEN == One)) > > > > > > > + { > > > > > > > + Local0 = 0x0F > > > > > > > + } > > > > > > > + Else > > > > > > > + { > > > > > > > + Local0 = 0x0D > > > > > > > + } > > > > > > > } > > > > > > > > > > > > > > Suggested-by: Igor Mammedov <imammedo@redhat.com> > > > > 'Reported-by' > > > > > > maybe, but certainly not suggested. > > > > > > > > > > > > > > > No issues. I can change. > > > > > > > > > > > > > > > > > > > > > > After more thinking and given presence is system wide that > > > > doesn't > > change at runtime, I don't see any reason for > > > > introducing presence > > bit as ABI (and undocumented on top of that). > > > > > > > > > > > > > > > This is a wrong assumption. Presence bit can change in future. We > > > > have > taken into account this aspect by design in the kernel code as > > well. > > > > > Both virtual > > > > > > > > Above does imply that support for really hotpluggable CPUs might be > > > > implemented in the future. > > > > Can you point to relevant kernel code/patches? > > > > > > > > > Let me make it clear so that there is no confusion, there is no > > > support of physical "CPU" hot-plug on ARM platforms right now and nor > > > will there be any in future as it does not makes sense to have. The > > > point I made in the patches is about hot-plug was at different > > > granularity which has not been denied by ARM. > > > > _STA and ACPI cphp registers are per logical CPU and can't be anything else > > per spec. > > > > how different granularity is relevant here? > > > It is because hot-plug at socket level (or similar level) has not been denied. It's the same for x86. However anything above logical CPU level is not CPU devices, but container devices. Even if you plugged whole socket, in QEMU that would result in _STA evaluated on all leaf logical CPUs and guest will online those CPUs individually. Granularity here is irrelevant, it handles only vCPUs an nothing else. If there were need to do something with containers (socket/cluster/...) in hypothetical future, that should have it's own AML code instead of abusing vCPU one, and well it certainly should be added only at that time and not earlier. > > > > > and physical CPU hot plug can co-exists or entirely as sole features. > > > > > This is a requirement. > > > > > > > > > > > > I don't see any _must_ requirements here whatsoever. If/when ARM > > > > reaches point where standby and hotplug cpus are mixed within VM > > > > instance, we might have to expose presence bit to guest dynamically > > > > but it is totally not needed within observable future and premature. > > > > > > > > > > Your cpu class presence check also works target-wise just with more > > > > boiler code + not needed presence bit ABI for guest side, The > > > > same as my suggestion only from other side. > > > > > > > > But regardless of that as long as machine has doesn't mix always > > > > present CPUs with hotpluggable ones within the same instance, > > > > changing AML side as I've just suggested works. > > > > > > > > > Sure, I'm not disagreeing. It will work by adding the flag you've > > > suggested but it will work even by not adding any flag and not > > > defining a `persistent` hook for any architecture. > > > > hook is more complicated and hidden way, than directly passing down > > configuration data to routine that builds AML where it is needed, but that's > > cosmetics in the end. > > > It is indeed an abstraction and an intentional one because decision whether > some or all CPUs should remain present even after unplug action is > specific to an architecture. Hence, should be left with that part. Arch specific static AML is much simpler in this case, compared to callbacks. and this AML is still controlled by machine specific code, which opts in into always_present cpus feature (it's machine that build's AML code depending on options). We do the same in x86, see other flags in CPUHotplugFeatures. Guest opts in to change curtain parts of AML depending on machine config, same should be done for ARM. > > However there is more to your hook approach, it exposes is_present bit in > > cphp flag register which is ABI to guest which we will have to maintain > > forever and guest will do not necessary round-trip to QEMU to query it, > > while alternative is much simpler AML change without any ABI changes to > > care about. > > > I understand your predicament about ABI but I've to respectfully disagree on > the assumption that guest will never round trip and check for the presence bit. It doesn't need it now, isn't it? When guest really needs to get presence flag dynamically in the future, that's fine, but do introduce it at that time when it's actually used, i.e. not for hypothetical case. > > > > The point is we shouldn't add new ABI unless we have to. > > > Ok sure. point taken. > > > > In this case new ABI (is_presence flag) is not _must_, and much simpler > > static AML change is sufficient. > > > Agreed, there can be multiple ways to get rid of it . But I'm not sure if its > simpler or easier to maintain because CPUs AML code has become even > more obscure by that compact logic. It is not easy to understand at the > first look. This is not a performance leg and maintainability should take > over the precedence. As one who wrote and takes care ACPI and cpuhp, I'd know what code is easier to read/maintain in the arrea and to a lesser degree elsewhere in QEMU. When it comes to arch specific side, I usually leave it upto arch maintainers as they know specifics there much better. (unless I notice something that is too much to look away and asks to for bashing) > But I understand your primary concern now (i.e. to get rid of CPU_PRESENT > Bit) and allow me to propose in the lines of what you want but maybe in a > slightly different way. I would still like to retain hooks though. hooks do have it's use when logic they are executing is complex, dynamic and differs a lot between impl, but indirection they introduce makes reading code and maintaining it harder (debugging it becomes outright nightmare). Example: I did the same as you in the past, adding madt hook for some future use, in several years it began to get in the way and eventually was removed. Burden to clean up (possible in the future but not happened things) falls on new contributors. Moral of story is: not over-engineer. If hook can be replaced by config data and data driven code, I'd prefer it from ease to read/maintainability point of view. > > > > If ARM ever gets real cpu hotplug as your comment above hints, my > > > > suggestion also works fine. With only difference that board code > > > > would turn off always_present_cpus if hotpluggable CPUs are used > > instead of standby. > > > > > > > > > Sure, but is it necessary to define a new flag when you can do even > > without it? > > > Having few lines of extra code (literally 2-3 only) should not be a > > > major cause of worry, especially, if it makes design more clear and > > > easy to understand. This is not a performance code either. > > > > > > Again, I appreciate your compact logic. I'm not disagreeing with it. > > > > > > > > > > > > Instead changing AML code to account for it would be better, > > > > > > something like > > this: > > > > > > > > > > > > diff --git a/include/hw/acpi/cpu.h b/include/hw/acpi/cpu.h > > > > index > > 32654dc274..4a3e591120 100644 > > --- > > > > a/include/hw/acpi/cpu.h > > +++ b/include/hw/acpi/cpu.h > > @@ > > > > -55,6 +55,7 @@ void cpu_hotplug_hw_init(MemoryRegion *as, Object > > > > > > > > *owner, typedef struct CPUHotplugFeatures { > > > > > > bool acpi_1_compatible; > > > > > > bool has_legacy_cphp; > > > > > > + bool always_present_cpus; > > > > > > > > > > > > > > > This has to be fetched from architecture code. Please see other > > > > > changes in the V3 patch-set. > > > > > > > > I've seen, that and it doesn't have to be fetched dynamically. > > > > > > > > > Agreed, it is not necessary to be fetched. Hence, if you do not define > > > the hook. In later case, it assumes the existing logic, which x86 has been > > following. > > > It will work. > > > > It is better to get rid of not necessary hook altogether, and replace it with a > > simple AML change. > > > We still need a way to initialize flags in the architecture specific way. Hooks are > clean way to fetch that architecture specific state rather fiddling with flags > at multiple levels (initialization, migration etc) and you will be duplicating > CPU states at ACPI level and QOM vCPU object level . To be frank that's not > a very good design. cleaner way is to use properties to access object state in arch agnostic way. and cpuhp didn't access any cpu state before your bf1ecc8dad60619 commit ... - val |= cdev->cpu ? 1 : 0; + val |= check_cpu_enabled_status(DEVICE(cdev->cpu)) ? 1 : 0 ... poking into CPU internals (masked by helper function/hook) from another device that is not even parent of CPU, which is layer violation and should be avoided if possible. In case of cpuhp there is hotplug handlers infrastructure, to wire up and connect things. I particular, machine called acpi_cpu_plug_cb() which may access cpu, and in your case it can check 'realized' property and store 'enabled' state in AcpiCpuStatus state. It would be even better if it used a dedicated property "enabled", instead of abusing 'realized' one. And do opposite on unplug (can be done from unplug handler). > That said, I believe your end goal of getting rid of the change in the ABI > can still be achieved. > > > > > > In my opinion the series was not ready to be merged (Michael > > > > probably picked it by mistake). > > > > > > > > We don't really need this in 9.2 as it is only ARM cpu 'hotplug' > > > > related, and the later is not ready for 9.2. > > > > I'd prefer the series being reverted and we continue improving > > > > series, instead of rushing it and fixing broken thing up. > > > > > > > > > > > > > > > > > > > > > > > > bool fw_unplugs_cpu; > > > > > > const char *smi_path; > > > > > > } CPUHotplugFeatures; > > > > > > diff --git a/hw/acpi/cpu.c b/hw/acpi/cpu.c index > > > > > > 5cb60ca8bc..2bcce2b31c > > 100644 > > --- a/hw/acpi/cpu.c > > > > > > +++ b/hw/acpi/cpu.c > > @@ -452,15 +452,16 @@ void > > > > build_cpus_aml(Aml *table, MachineState > > *machine, > > > > CPUHotplugFeatures opts, > > > > > > > > method = aml_method(CPU_STS_METHOD, 1, > > AML_SERIALIZED); > > > > > > { > > > > > > + uint8_t default_sta = opts.always_present_cpus?0xd:0; > > > > > > Aml *idx = aml_arg(0); > > > > > > - Aml *sta = aml_local(0); > > > > > > + Aml *sta = aml_local(default_sta); > > > > > > > > > > > > aml_append(method, aml_acquire(ctrl_lock, 0xFFFF)); > > > > > > aml_append(method, aml_store(idx, cpu_selector)); > > > > > > aml_append(method, aml_store(zero, sta)); > > > > > > ifctx = aml_if(aml_equal(is_enabled, one)); > > > > > > { > > > > > > - aml_append(ifctx, aml_store(aml_int(0xF), sta)); > > > > > > + aml_append(ifctx, aml_or(aml_int(0xF), sta, sta)); > > > > > > } > > > > > > aml_append(method, ifctx); > > > > > > aml_append(method, aml_release(ctrl_lock)) > > > > > > > > > > > > then for ARM set > > > > > > CPUHotplugFeatures::always_present_cpus = true to get present > > flag > > > > > > always enabled > > > > > > > > > > > > > > > We MUST fetch this from architecture code as this can dynamically > > > > > change in future and hence, we need to keep that flexibility > > > > > > > > CPUHotplugFeatures::always_present_cpus can be set dynamically per > > > > VM instance or per Machine type. > > > > > > > > > Yes, but you need a code for that and I'm not sure what is being saved > > > by introducing an extra flag then? > > > > beside snippet, I've suggested. You need to delete all is_present callback > > related logic, than it save quite a bit, and not only on lines of code. > > > > I guess, I need to send a patch to get point across. > > > There is some misunderstanding. We are not maintaining any `is_present` > state at the ACPI level. Both flags `is_present` and `is_enabled` were > removed in V3 patch-set. That was removed from AcpiCpuStatus, however v3 does introduce present bit in register and callback to fetch it. It is the same thing, just other way around. > > > > > > After that revert _all_ other presence bit related changes > > > > that > > were just merged. > > > > > > (I did ask to get rid of that in previous reviews but it came > > > > back > > again for no good reason). > > > > > > > > > > > > > > > The CPUs AML in the V2 patch-set would have broken the x86 > > > > functionality. > > > > > > > > Frankly speaking, I don't see much progress. All that happens on > > > > respins is flipping between what I asked to remove before to some > > earlier concept. > > > > > > > > > I think then you've missed the code which addressed one of your key > > > concerns in the V1 patch-set that would have broken x86 migration i.e. > > > presence of the `is_enabled` state in the CPU Hot-plug VMSD. That has > > > been removed. Maybe you would like to go through the change log of the > > > V3 patch-set > > > > > > https://lore.kernel.org/qemu- > > devel/20241018163118.0ae01a84@imammedo.us > > > ers.ipa.redhat.com/ > > > > > > Below is the relevant excerpt from your many comments: > > > > > > [...] > > > > .fields = (const VMStateField[]) { > > > > VMSTATE_BOOL(is_inserting, AcpiCpuStatus), > > > > VMSTATE_BOOL(is_removing, AcpiCpuStatus), > > > > + VMSTATE_BOOL(is_present, AcpiCpuStatus), > > > > + VMSTATE_BOOL(is_enabled, AcpiCpuStatus), > > > > > > that's likely will break x86 migration, but before bothering peterx, > > > maybe we won't need this hunk if is_enabled is migrated as part of > > > vCPU state. > > > > > > > what has been done in v3 is moving is_present/is_enabled, elsewhere > > (callbacks in this case). While argument was that both are not necessary to > > begin with. > > > We need to have a way to check whether CPU is _STA.Enabled or not and > the meaning of the `disabled` is best decided by the architecture specific > code not the ACPI code. It could mean different across architectures and > the implementations. We need to keep that flexibility in design. I'm not arguing about 'enabled' in principle, only about the way it's implemented. And that 'present' shouldn't be part of dynamic interface. If you ever need present as dynamic in the future, introduce it at that time along with patches that would use it. > The thin Qemu ACPI code is just an interfacing logic with the guest. It is > best not to duplicate the state of the QOM objects inside the ACPI. That's > a bad design practice because we will have to worry about keeping those > states consistent at both the places all the time whether it is during > initialization or migration. if we'd follow this logic, we would have a QEMU with direct access across different objects all over the place. That might work for small project and avoid duplication, but that is very fragile and won't scale to project of QEMU size. And to configure devices we have init/property/hotplug handlers, so that machine could connect separate objects together in meaning full way and establish well known control flow. > > > [...] > > > > > > > > > > And all of that for the purpose to workaround/accommodate fake cpu > > > > hotplug hacks. > > > > > > > > > I've to respectfully disagree on this. This is your assumption which > > > is far from reality. The accompanying main series of this will never > > > have vCPUs which are > > > *not* present. > > > > Reality of your last posted main series (v5), disagrees with what you are > > saying here > > > > [PATCH RFC V5 12/30] arm/virt: Release objects for *disabled* possible > > vCPUs after init > > https://patchew.org/QEMU/20241015100012.254223-1- > > salil.mehta@huawei.com/20241015100012.254223-13- > > salil.mehta@huawei.com/ > > > I think you missed what I mentioned above " The accompanying main series...". > To get more information about that series, please check the details mentioned > in the cover-letter of the V3 patch-set (link below) which clearly says, > > https://lore.kernel.org/qemu-devel/20241103102419.202225-1-salil.mehta@huawei.com/ > > Excerpt from the cover letter: > ======================== > [...] > This patch-set has been tested alongside ARM-specific > vCPU hotplug changes (included in the upcoming RFC V6 [4]), and > hot(un)plug functionalities performed as expected which concerns this > patch-set. Please have a look. > [...] > > References > ========== > [...] > [4] Upcoming RFC V6, Support of Virtual CPU Hotplug for ARMv8 Arch > Link: https://github.com/salil-mehta/qemu/commits/virt-cpuhp-armv8/rfc-v6-rc5 > [...] that v6 series is not posted to the list, so it can disappear or change any moment. Using it as reference is pointless. There is no point to comment on something that is not on the list as later on the github branch can be totally different from what it was at the time comment was made. That is also a reason why (v3) ARM cpuhp specific ACPI patches shouldn't have been merged separately from main series. It was fine when generic acpi cpu hotplug patches where posted as separate series as those were working with several different RFCs for cpuhp for different targets that would reuse it. However posting ARM specific cpuhp acpi changes separately, without actual user along with them only lead to confusion and mess we have now. I suggest you to send revert for merged v3, and resend those (fixed up) patches as part of main series where they can be properly reviewed. (or I can sent it along with a couple of RFC patches doing the same but in acceptable way from ACPI pov. The later could be picked up into your next main series) > > after this patch, new is_present hook would let you bury the lie about CPU > > being present inside ARM specific code. > > > I would request you to check the facts again! (please see previous pointers) > > ACPI logic should not worry about what is happening inside architecture > code. It is beyond its realms and should be by design. It is an interfacing upper > layer gelling the Guest with the VMM. > > > > > BTW, these changes have been tested by Oracle folks with that series > > > and are known to work. > > > > > > https://lore.kernel.org/qemu-devel/C4FEC9E7-69DB-428A-A85F- > > 30170F98814 > > > B@oracle.com/ > > > > One can write anything and it can even work somehow, that however > > doesn't mean it's something merge-able, maintainable, ...whatever else > > come to mind... > > > It would be very helpful if you can `objectively` point the problems in the > main series so that there is a context and we can have a productive > discussion there. There is always a scope of improvement. I did point to problems in the past (the latest was v2 review), repeating it now again. Design wise issue is that ARM architecture doesn't support cpu hotplug. That's the reason why ARM kernel folks haven't implemented it there since it's not what hardware does. Quotes from [2] 'There is no hardware that does anything of this.' 'Whatever is defined for virtual machines needs to work on physical hardware too – the OS doesn’t know its in a VM' Above is also a good reason not to try implementing it on QEMU side as hotplug as well. My suggestion was to try instead standby CPUs concept that ARM does support, which looks like hotplug on guest side (kernel largely freely mixes up hotplug term with CPUs onlining). The only reply I got to this: unsure, and 'requirements' as if it set in stone and pushed on QEMU using some presentations [2] as reference, which is basically a set of issues and wishes plus some ideas how it could be achieved. Those wishes are by no means any form of consensus on how things should be. * 'must reuse device_add/del for smooth transition from x86' this one I guess comes from flowing quote: 'Should have look-and-feel of x86_64 platform VCPU hot-plug feature' and then accompanied kernel presentation phrase: 'No appetite for updating numerous specifications to describe hotplug support' On that one can easily counter: 'then make it virt hardware behave like x86 and write specs for it, instead of hacking the heck out of QEMU' But that is beside the point, management software can easily adapt to dedicated QMP command to online CPUs without abusing device_add/del. ARM's 'cpu hotplug' doesn't have to be drop in replacement for ARM, you do configure ARM VM differently, you can do 'hotplug' differently as well, nothing prevents it beside having a will to do it. * 'for large VM to start fast' that one I guess, comes from following phrase: 'allowing for faster boot time' but whole bullet point talks about scalability 'Kata Containers VM starts with a minimum amount of resources, (hot-plug everything)' that somehow turned in to requirement: 'start VM as fast as if real CPU hotplug is supported regardless of the costs' as result main series hack is to avoid creation of possible CPUs, Depending on series revision it ranged from creation of half initialized vCPU objects and then deleting them to creating half initialized vCPU and letting them dangle in memory without completing its creation in the latest series incarnation (v6). And all of that is to avoid calling expensive in current impl. cpu_realize. The rest series is some necessary boiler plate code + other hacks trying to convince the rest of QEMU that dangling half initialized CPU objects are nothing to worry about. well with standby cpu design you can also have 'allowing for faster boot time' it is not as fast as x86 with its hotplug but still faster than onlining all possible CPUs at boot time on guest side. Even 'faster startup' wish is stretching things, one just puts off delays to a later time in guest lifetime where every single vCPU hotplug incurs even bigger time penalty as guest kernel has to stop everything to hotplug a CPU. The point is: 'standby cpus' design also leads to faster boot on guest side compared normal boot with all vCPUs. As for comparing with even faster x86, fix cpu_realize to be faster (as I demonstrated in v2 review) or fix design to implement real hotplug [1], instead of another quote from [2] 'Creative Hack – Which Works! Really?' So my suggestion to try 'standby cpus' approach still stays, fully realizing CPUs. Also, you can't reuse device_add with it as that means you'd be forced to hack workflow one way or another for what it wasn't designed. Add a new QMP command instead 'device-on/device-off' that would fit standby concept better. You will very likely end up with much simpler series that it's now, probably even more simple than recent loongarch cpu hotplug [3] PS: 1) If hotplug support is really need, try to convince ARM to implement it (as minimum for virt environment, that likely would require some spec & kernel work first). Or wait until business needs wake up ARM's appetite to do it right. 2) KVMForum 2023 Presentation: Challenges Revisited in Supporting Virt CPU Hotplug on architectures that don’t Support CPU Hotplug (like ARM64) a. Kernel Link: https://kvm-forum.qemu.org/2023/KVM-forum-cpu-hotplug_7OJ1YyJ.pdf b. Qemu Link: https://kvm-forum.qemu.org/2023/Challenges_Revisited_in_Supporting_Virt_CPU_Hotplug_-__ii0iNb3.pdf 3) PS2: As a recent example of platform that does have 'appetite' to implement 'real' cpu hotplug see [PATCH v3 0/5] hw/loongarch/virt: Add cpu hotplug support https://lore.kernel.org/all/20241104063435.4130262-6-maobibo@loongson.cn/T/ and compare with ARM RFC, it looks much more cleaner and strait-forward than ARM variant when one doesn't need hack QEMU here and there to make it work. PS3: joke: typical QEMU maintainer response to large series 'rewrite half of QEMU 1st, then do you thing and we will see' ARM cpuhp gets much milder treatment 'rewrite your series in other way that matches real world' > Many thanks! > > Best regards > Salil. > >
diff --git a/hw/acpi/cpu.c b/hw/acpi/cpu.c index 23443f09a5..b2f7a2b27e 100644 --- a/hw/acpi/cpu.c +++ b/hw/acpi/cpu.c @@ -490,22 +490,22 @@ void build_cpus_aml(Aml *table, MachineState *machine, CPUHotplugFeatures opts, aml_append(method, aml_acquire(ctrl_lock, 0xFFFF)); aml_append(method, aml_store(idx, cpu_selector)); aml_append(method, aml_store(zero, sta)); - ifctx = aml_if(aml_equal(is_present, one)); + ifctx = aml_if(aml_equal(is_enabled, one)); { - ifctx2 = aml_if(aml_equal(is_enabled, one)); - { - /* cpu is present and enabled */ - aml_append(ifctx2, aml_store(aml_int(0xF), sta)); - } - aml_append(ifctx, ifctx2); - else_ctx = aml_else(); + /* cpu is present and enabled */ + aml_append(ifctx, aml_store(aml_int(0xF), sta)); + } + aml_append(method, ifctx); + else_ctx = aml_else(); + { + ifctx2 = aml_if(aml_equal(is_present, one)); { /* cpu is present but disabled */ - aml_append(else_ctx, aml_store(aml_int(0xD), sta)); + aml_append(ifctx2, aml_store(aml_int(0xD), sta)); } - aml_append(ifctx, else_ctx); + aml_append(else_ctx, ifctx2); } - aml_append(method, ifctx); + aml_append(method, else_ctx); aml_append(method, aml_release(ctrl_lock)); aml_append(method, aml_return(sta)); }
Checking `is_present` first can break x86 migration from new Qemu version to old Qemu. This is because CPRS Bit is not defined in the older Qemu register block and will always be 0 resulting in check always failing. Reversing the logic to first check `is_enabled` can alleviate below problem: - If ((\_SB.PCI0.PRES.CPEN == One)) - { - Local0 = 0x0F + If ((\_SB.PCI0.PRES.CPRS == One)) + { + If ((\_SB.PCI0.PRES.CPEN == One)) + { + Local0 = 0x0F + } + Else + { + Local0 = 0x0D + } } Suggested-by: Igor Mammedov <imammedo@redhat.com> Message-ID: <20241106100047.18901c9d@imammedo.users.ipa.redhat.com> Signed-off-by: Salil Mehta <salil.mehta@huawei.com> --- hw/acpi/cpu.c | 22 +++++++++++----------- 1 file changed, 11 insertions(+), 11 deletions(-)