| Message ID | 20241025101959.601048-22-eric.auger@redhat.com (mailing list archive) |
|---|---|
| State | New |
| Headers | show |
| Series | kvm/arm: Introduce a customizable aarch64 KVM host model | expand |
On Fri, Oct 25, 2024 at 12:17:40PM +0200, Eric Auger wrote: > From: Cornelia Huck <cohuck@redhat.com> > > Add some documentation for the custom model. > > Signed-off-by: Eric Auger <eric.auger@redhat.com> > Signed-off-by: Cornelia Huck <cohuck@redhat.com> > --- > docs/system/arm/cpu-features.rst | 55 +++++++++++++++++++++++++++----- > 1 file changed, 47 insertions(+), 8 deletions(-) > @@ -167,6 +196,16 @@ disabling many SVE vector lengths would be quite verbose, the ``sve<N>`` CPU > properties have special semantics (see "SVE CPU Property Parsing > Semantics"). > > +The ``custom`` CPU model needs to be configured via individual ID register > +field properties, for example:: > + > + $ qemu-system-aarch64 -M virt -cpu custom,SYSREG_ID_AA64ISAR0_EL1_DP=0x0 > + > +This forces ID_AA64ISAR0_EL1 DP field to 0. What is the "baseline" featureset implied by 'custom' ? On x86 we have the named CPU models each setting a baseline that matches some corresponding real world silicon. Arm has that too, with TCG at least. So that way you know what the baseline is that you're toggling features against. Experiance on x86 was that making arbitrary feature changes on top of the named models could often backfire, as there are too many scenarios where code will check for feature "Y", and assume that existance of "Y" implies existance of "A", "B" and "C" too. So if you invent custom models where Y is set, but B is missing, there's decent risk of things going wrong in horrible to debug ways. With that in mind, best practice is to try to just the vanilla named CPU models to the greatest extent possible, and keep feature toggling to an absolute minimum. This 'custom' model does not seem to give us such ability for arm. With regards, Daniel
Hi Daniel, On 10/25/24 15:13, Daniel P. Berrangé wrote: > On Fri, Oct 25, 2024 at 12:17:40PM +0200, Eric Auger wrote: >> From: Cornelia Huck <cohuck@redhat.com> >> >> Add some documentation for the custom model. >> >> Signed-off-by: Eric Auger <eric.auger@redhat.com> >> Signed-off-by: Cornelia Huck <cohuck@redhat.com> >> --- >> docs/system/arm/cpu-features.rst | 55 +++++++++++++++++++++++++++----- >> 1 file changed, 47 insertions(+), 8 deletions(-) > >> @@ -167,6 +196,16 @@ disabling many SVE vector lengths would be quite verbose, the ``sve<N>`` CPU >> properties have special semantics (see "SVE CPU Property Parsing >> Semantics"). >> >> +The ``custom`` CPU model needs to be configured via individual ID register >> +field properties, for example:: >> + >> + $ qemu-system-aarch64 -M virt -cpu custom,SYSREG_ID_AA64ISAR0_EL1_DP=0x0 >> + >> +This forces ID_AA64ISAR0_EL1 DP field to 0. > What is the "baseline" featureset implied by 'custom' ? there is no baseline at the moment. By default this is a host passthrough model. > > On x86 we have the named CPU models each setting a baseline that matches > some corresponding real world silicon. Arm has that too, with TCG at > least. So that way you know what the baseline is that you're toggling > features against. Having named models is the next thing. custom vcpu model is not a named model. But we don't want to TCG like CPU model (like A57) because we want to be able to migrate between different machines like Ampere to NVidia or different Ampere systems. So the baseline must be something usable by both hosts. > > Experiance on x86 was that making arbitrary feature changes on top of the > named models could often backfire, as there are too many scenarios where > code will check for feature "Y", and assume that existance of "Y" implies > existance of "A", "B" and "C" too. So if you invent custom models where > Y is set, but B is missing, there's decent risk of things going wrong in > horrible to debug ways. With that in mind, best practice is to try to > just the vanilla named CPU models to the greatest extent possible, and > keep feature toggling to an absolute minimum. This 'custom' model does > not seem to give us such ability for arm. The custom model is not yet a named model. This is rather something to start this kind of discussion. The code used by the custom vcpu model allows fine tuning of the ID reg fields. This code could be reused by named models. We can also imagine that libvirt does implement the named models, ie. hardcodes some IDReg fields and thus implement the named model instead. Libvirt could identify what is the baseline source and dest are the closest to, choose this baseline and tune few reg id fields if some additional tuning are needed. Thanks Eric > > With regards, > Daniel
On Fri, Oct 25, 2024 at 03:28:35PM +0200, Eric Auger wrote: > Hi Daniel, > > On 10/25/24 15:13, Daniel P. Berrangé wrote: > > On Fri, Oct 25, 2024 at 12:17:40PM +0200, Eric Auger wrote: > >> From: Cornelia Huck <cohuck@redhat.com> > >> > >> Add some documentation for the custom model. > >> > >> Signed-off-by: Eric Auger <eric.auger@redhat.com> > >> Signed-off-by: Cornelia Huck <cohuck@redhat.com> > >> --- > >> docs/system/arm/cpu-features.rst | 55 +++++++++++++++++++++++++++----- > >> 1 file changed, 47 insertions(+), 8 deletions(-) > > > >> @@ -167,6 +196,16 @@ disabling many SVE vector lengths would be quite verbose, the ``sve<N>`` CPU > >> properties have special semantics (see "SVE CPU Property Parsing > >> Semantics"). > >> > >> +The ``custom`` CPU model needs to be configured via individual ID register > >> +field properties, for example:: > >> + > >> + $ qemu-system-aarch64 -M virt -cpu custom,SYSREG_ID_AA64ISAR0_EL1_DP=0x0 > >> + > >> +This forces ID_AA64ISAR0_EL1 DP field to 0. > > What is the "baseline" featureset implied by 'custom' ? > there is no baseline at the moment. By default this is a host > passthrough model. Why do we need to create "custom" at all, as opposed to just letting users toggle features on "-cpu host" ? With regards, Daniel
On Fri, Oct 25 2024, Daniel P. Berrangé <berrange@redhat.com> wrote: > On Fri, Oct 25, 2024 at 03:28:35PM +0200, Eric Auger wrote: >> Hi Daniel, >> >> On 10/25/24 15:13, Daniel P. Berrangé wrote: >> > On Fri, Oct 25, 2024 at 12:17:40PM +0200, Eric Auger wrote: >> >> From: Cornelia Huck <cohuck@redhat.com> >> >> >> >> Add some documentation for the custom model. >> >> >> >> Signed-off-by: Eric Auger <eric.auger@redhat.com> >> >> Signed-off-by: Cornelia Huck <cohuck@redhat.com> >> >> --- >> >> docs/system/arm/cpu-features.rst | 55 +++++++++++++++++++++++++++----- >> >> 1 file changed, 47 insertions(+), 8 deletions(-) >> > >> >> @@ -167,6 +196,16 @@ disabling many SVE vector lengths would be quite verbose, the ``sve<N>`` CPU >> >> properties have special semantics (see "SVE CPU Property Parsing >> >> Semantics"). >> >> >> >> +The ``custom`` CPU model needs to be configured via individual ID register >> >> +field properties, for example:: >> >> + >> >> + $ qemu-system-aarch64 -M virt -cpu custom,SYSREG_ID_AA64ISAR0_EL1_DP=0x0 >> >> + >> >> +This forces ID_AA64ISAR0_EL1 DP field to 0. >> > What is the "baseline" featureset implied by 'custom' ? >> there is no baseline at the moment. By default this is a host >> passthrough model. > > Why do we need to create "custom" at all, as opposed to just letting > users toggle features on "-cpu host" ? We could consolidate that to the current "host" model, once we figure out how to handle the currently already existing properties. Models based on the different architecture extensions would probably be more useable in the long run; maybe "custom" has a place for testing.
On Mon, Oct 28, 2024 at 05:05:44PM +0100, Cornelia Huck wrote: > On Fri, Oct 25 2024, Daniel P. Berrangé <berrange@redhat.com> wrote: > > > On Fri, Oct 25, 2024 at 03:28:35PM +0200, Eric Auger wrote: > >> Hi Daniel, > >> > >> On 10/25/24 15:13, Daniel P. Berrangé wrote: > >> > On Fri, Oct 25, 2024 at 12:17:40PM +0200, Eric Auger wrote: > >> >> From: Cornelia Huck <cohuck@redhat.com> > >> >> > >> >> Add some documentation for the custom model. > >> >> > >> >> Signed-off-by: Eric Auger <eric.auger@redhat.com> > >> >> Signed-off-by: Cornelia Huck <cohuck@redhat.com> > >> >> --- > >> >> docs/system/arm/cpu-features.rst | 55 +++++++++++++++++++++++++++----- > >> >> 1 file changed, 47 insertions(+), 8 deletions(-) > >> > > >> >> @@ -167,6 +196,16 @@ disabling many SVE vector lengths would be quite verbose, the ``sve<N>`` CPU > >> >> properties have special semantics (see "SVE CPU Property Parsing > >> >> Semantics"). > >> >> > >> >> +The ``custom`` CPU model needs to be configured via individual ID register > >> >> +field properties, for example:: > >> >> + > >> >> + $ qemu-system-aarch64 -M virt -cpu custom,SYSREG_ID_AA64ISAR0_EL1_DP=0x0 > >> >> + > >> >> +This forces ID_AA64ISAR0_EL1 DP field to 0. > >> > What is the "baseline" featureset implied by 'custom' ? > >> there is no baseline at the moment. By default this is a host > >> passthrough model. > > > > Why do we need to create "custom" at all, as opposed to just letting > > users toggle features on "-cpu host" ? > > We could consolidate that to the current "host" model, once we figure > out how to handle the currently already existing properties. Models > based on the different architecture extensions would probably be more > useable in the long run; maybe "custom" has a place for testing. If you can set the features against "host", then any testing could be done with "host" surely, making 'custom' pointless ? With regards, Daniel
On Mon, Oct 28 2024, Daniel P. Berrangé <berrange@redhat.com> wrote: > On Mon, Oct 28, 2024 at 05:05:44PM +0100, Cornelia Huck wrote: >> On Fri, Oct 25 2024, Daniel P. Berrangé <berrange@redhat.com> wrote: >> >> > On Fri, Oct 25, 2024 at 03:28:35PM +0200, Eric Auger wrote: >> >> Hi Daniel, >> >> >> >> On 10/25/24 15:13, Daniel P. Berrangé wrote: >> >> > On Fri, Oct 25, 2024 at 12:17:40PM +0200, Eric Auger wrote: >> >> >> From: Cornelia Huck <cohuck@redhat.com> >> >> >> >> >> >> Add some documentation for the custom model. >> >> >> >> >> >> Signed-off-by: Eric Auger <eric.auger@redhat.com> >> >> >> Signed-off-by: Cornelia Huck <cohuck@redhat.com> >> >> >> --- >> >> >> docs/system/arm/cpu-features.rst | 55 +++++++++++++++++++++++++++----- >> >> >> 1 file changed, 47 insertions(+), 8 deletions(-) >> >> > >> >> >> @@ -167,6 +196,16 @@ disabling many SVE vector lengths would be quite verbose, the ``sve<N>`` CPU >> >> >> properties have special semantics (see "SVE CPU Property Parsing >> >> >> Semantics"). >> >> >> >> >> >> +The ``custom`` CPU model needs to be configured via individual ID register >> >> >> +field properties, for example:: >> >> >> + >> >> >> + $ qemu-system-aarch64 -M virt -cpu custom,SYSREG_ID_AA64ISAR0_EL1_DP=0x0 >> >> >> + >> >> >> +This forces ID_AA64ISAR0_EL1 DP field to 0. >> >> > What is the "baseline" featureset implied by 'custom' ? >> >> there is no baseline at the moment. By default this is a host >> >> passthrough model. >> > >> > Why do we need to create "custom" at all, as opposed to just letting >> > users toggle features on "-cpu host" ? >> >> We could consolidate that to the current "host" model, once we figure >> out how to handle the currently already existing properties. Models >> based on the different architecture extensions would probably be more >> useable in the long run; maybe "custom" has a place for testing. > > If you can set the features against "host", then any testing could > be done with "host" surely, making 'custom' pointless ? We might differentiate between "do some consistency checks" and "allow a completely weird wolpertinger"; if we agree that we don't need it, then we surely could drop it again.
On Fri, Oct 25, 2024 at 12:17:40PM +0200, Eric Auger wrote: > From: Cornelia Huck <cohuck@redhat.com> > > Add some documentation for the custom model. > > Signed-off-by: Eric Auger <eric.auger@redhat.com> > Signed-off-by: Cornelia Huck <cohuck@redhat.com> > --- > docs/system/arm/cpu-features.rst | 55 +++++++++++++++++++++++++++----- > 1 file changed, 47 insertions(+), 8 deletions(-) > > diff --git a/docs/system/arm/cpu-features.rst b/docs/system/arm/cpu-features.rst > index a5fb929243..962a2c6c26 100644 > --- a/docs/system/arm/cpu-features.rst > +++ b/docs/system/arm/cpu-features.rst > @@ -2,7 +2,10 @@ Arm CPU Features [...] > +Using the ``host`` type means the guest is provided all the same CPU > +features as the host CPU type has. And, for this reason, the ``host`` > +CPU type should enable all CPU features that the host has by default. > + > +In case some features need to be hidden to the guest, ``custom`` model > +shall be used instead. This is especially useful for migration purpose. > + > +The ``custom`` CPU model generally is the better choice if you want more > +flexibility or stability across different machines or with different kernel > +versions. Does "more flexibility or stability across different machines" also imply "live migration compatiblity across host CPUs"? > However, even the ``custom`` CPU model will not allow configuring > +an arbitrary set of features; the ID registers must describe a subset of the > +host's features, and all differences to the host's configuration must actually > +be supported by the kernel to be deconfigured. [...] > +The ``custom`` CPU model needs to be configured via individual ID register > +field properties, for example:: > + > + $ qemu-system-aarch64 -M virt -cpu custom,SYSREG_ID_AA64ISAR0_EL1_DP=0x0 If possible, it would be really helpful (and user-friendly) to be able to specify the CPU feature names as you see under /proc/cpuinfo, and be able to turn the flags on or off: -M virt -cpu franken,rndr=on,ts=on,fhm=off (... instead of specifying long system register IDs that groups together a bunch of CPU features. If I understand it correctly, the register "ID_AA64ISAR0_EL1" maps to a set of visible features listed here: https://docs.kernel.org/arch/arm64/cpu-feature-registers.html) Next, I prefix the below by noting that I wrote it before seeing Cornelia's reply that the name "custom" is not set in stone: https://lists.nongnu.org/archive/html/qemu-arm/2024-10/msg00987.html. I wonder if the word "custom" is starting to get overloaded; on x86: - Libvirt itself uses the term "custom" this way, to quote its documentation[1] for the 'custom' XML attribute: custom In this mode, the 'cpu' element describes the CPU that should be presented to the guest. This is the default when no 'mode' attribute is specified. This mode makes it so that a persistent guest will see the same hardware no matter what host the guest is booted on. - Some management tools also follow libvirt and use the term "custom" to refer to one of two things, (a) a specific named CPU model that libvirt and QEMU recognize, e.g. "Cascadelake-Server"; or (b) a named CPU model + extra CPU flags, e.g. this is how OpenStack uses[2] "custom" to configure CPU models, and flags that can be enabled or disabled via "+" or "-": [libvirt] cpu_mode = custom cpu_model = IvyBridge-IBRS cpu_model_extra_flags="ss,+vmx,-pcid [...]" (Note the "cpu_mode" there: it is referring to the three possible modes that libvirt and QEMU support today: 'host-passthrough', 'host-model', and named CPU models via "custom".) The above config translates to this QEMU command-line: -cpu IvyBridge-IBRS,ss=on,vmx=on,pcid=off [...] Now if QEMU introduces "custom", it is likely to create some confusion. But luckily, as referenced above, it is open to change. :) * * * FWIW, I agree with Dan here[3] that it would cause less future pain if Arm's named CPU models also decides on a "baseline that matches some corresponding real world silicon". I've experienced plenty of such debugging pain in x86-land from years of troubleshooting live migration bugs involving CPU model (in)compatibility. (Often, with help from DanPB and Jiri Denemark). [1] https://docs.openstack.org/nova/latest/admin/cpu-models.html#cpu-modes [2] https://libvirt.org/formatdomain.html#cpu-model-and-topology [3] https://lists.nongnu.org/archive/html/qemu-arm/2024-10/msg00888.html — [RFC 21/21] arm/cpu-features: Document custom vcpu model [...]
On Mon, Oct 28, 2024 at 05:29:11PM +0100, Cornelia Huck wrote: > On Mon, Oct 28 2024, Daniel P. Berrangé <berrange@redhat.com> wrote: [...] > >> We could consolidate that to the current "host" model, once we figure > >> out how to handle the currently already existing properties. Models > >> based on the different architecture extensions would probably be more > >> useable in the long run; maybe "custom" has a place for testing. > > > > If you can set the features against "host", then any testing could > > be done with "host" surely, making 'custom' pointless ? > > We might differentiate between "do some consistency checks" and "allow > a completely weird wolpertinger"; if we agree that we don't need it, > then we surely could drop it again. Yeah, FWIW, I agree that it's best to drop "custom" if all the meaningful tests can be handled by being able to add/remove CPU flags from `-cpu host`. Related: I don't see any mention of `-cpu max` here. Is it not relevant? It is currently defined as: "enables all features supported by the accelerator in the current host". Does it make sense for `max` to allow disabling features? Or is the idea that, why would you choose `-cpu max` if you want to disable features? In that case, go with either: -cpu host,feat1=off Or: -cpu some_future_named_model,$feat1=off ?
On Thu, 31 Oct 2024 at 12:24, Kashyap Chamarthy <kchamart@redhat.com> wrote: > > On Mon, Oct 28, 2024 at 05:29:11PM +0100, Cornelia Huck wrote: > > On Mon, Oct 28 2024, Daniel P. Berrangé <berrange@redhat.com> wrote: > > [...] > > > >> We could consolidate that to the current "host" model, once we figure > > >> out how to handle the currently already existing properties. Models > > >> based on the different architecture extensions would probably be more > > >> useable in the long run; maybe "custom" has a place for testing. > > > > > > If you can set the features against "host", then any testing could > > > be done with "host" surely, making 'custom' pointless ? > > > > We might differentiate between "do some consistency checks" and "allow > > a completely weird wolpertinger"; if we agree that we don't need it, > > then we surely could drop it again. > > Yeah, FWIW, I agree that it's best to drop "custom" if all the > meaningful tests can be handled by being able to add/remove CPU flags > from `-cpu host`. > > > Related: I don't see any mention of `-cpu max` here. Is it not > relevant? It is currently defined as: "enables all features supported > by the accelerator in the current host". Does it make sense for `max` > to allow disabling features? Or is the idea that, why would you choose > `-cpu max` if you want to disable features? Ideally, disabling features would work with any '-cpu' option (including our existing TCG CPUs). (The main reason for 'max' is as an option that works whether you're using TCG or KVM.) -- PMM
Hi On 10/28/24 17:09, Daniel P. Berrangé wrote: > On Mon, Oct 28, 2024 at 05:05:44PM +0100, Cornelia Huck wrote: >> On Fri, Oct 25 2024, Daniel P. Berrangé <berrange@redhat.com> wrote: >> >>> On Fri, Oct 25, 2024 at 03:28:35PM +0200, Eric Auger wrote: >>>> Hi Daniel, >>>> >>>> On 10/25/24 15:13, Daniel P. Berrangé wrote: >>>>> On Fri, Oct 25, 2024 at 12:17:40PM +0200, Eric Auger wrote: >>>>>> From: Cornelia Huck <cohuck@redhat.com> >>>>>> >>>>>> Add some documentation for the custom model. >>>>>> >>>>>> Signed-off-by: Eric Auger <eric.auger@redhat.com> >>>>>> Signed-off-by: Cornelia Huck <cohuck@redhat.com> >>>>>> --- >>>>>> docs/system/arm/cpu-features.rst | 55 +++++++++++++++++++++++++++----- >>>>>> 1 file changed, 47 insertions(+), 8 deletions(-) >>>>>> @@ -167,6 +196,16 @@ disabling many SVE vector lengths would be quite verbose, the ``sve<N>`` CPU >>>>>> properties have special semantics (see "SVE CPU Property Parsing >>>>>> Semantics"). >>>>>> >>>>>> +The ``custom`` CPU model needs to be configured via individual ID register >>>>>> +field properties, for example:: >>>>>> + >>>>>> + $ qemu-system-aarch64 -M virt -cpu custom,SYSREG_ID_AA64ISAR0_EL1_DP=0x0 >>>>>> + >>>>>> +This forces ID_AA64ISAR0_EL1 DP field to 0. >>>>> What is the "baseline" featureset implied by 'custom' ? >>>> there is no baseline at the moment. By default this is a host >>>> passthrough model. >>> Why do we need to create "custom" at all, as opposed to just letting >>> users toggle features on "-cpu host" ? >> We could consolidate that to the current "host" model, once we figure >> out how to handle the currently already existing properties. Models >> based on the different architecture extensions would probably be more >> useable in the long run; maybe "custom" has a place for testing. > If you can set the features against "host", then any testing could > be done with "host" surely, making 'custom' pointless ? Yeah I do agree that we may not need to introduce this "custom" model bus just enhance the custom host model with the capability to tweek some features. For instance we have the case where migration between 2 Ampere systems fails with host model but if you tweek 1 field in CTR_EL0 it passes. So I think in itself this modality can be useful. Same for debug/test purpose. As mentionned in the cover letter the number of writable ID regs continue to grow and this enhanced host model gives flexibility to test new support and may provide enhanced debug capabilities for migration (getting a straight understanding of which ID reg field(s) causes the migration failure could be helpful I think) Thanks Eric > > With regards, > Daniel
On Mon, Nov 04, 2024 at 03:45:13PM +0100, Eric Auger wrote: > Hi > > On 10/28/24 17:09, Daniel P. Berrangé wrote: > > On Mon, Oct 28, 2024 at 05:05:44PM +0100, Cornelia Huck wrote: > >> On Fri, Oct 25 2024, Daniel P. Berrangé <berrange@redhat.com> wrote: > >> > >>> On Fri, Oct 25, 2024 at 03:28:35PM +0200, Eric Auger wrote: > >>>> Hi Daniel, > >>>> > >>>> On 10/25/24 15:13, Daniel P. Berrangé wrote: > >>>>> On Fri, Oct 25, 2024 at 12:17:40PM +0200, Eric Auger wrote: > >>>>>> From: Cornelia Huck <cohuck@redhat.com> > >>>>>> > >>>>>> Add some documentation for the custom model. > >>>>>> > >>>>>> Signed-off-by: Eric Auger <eric.auger@redhat.com> > >>>>>> Signed-off-by: Cornelia Huck <cohuck@redhat.com> > >>>>>> --- > >>>>>> docs/system/arm/cpu-features.rst | 55 +++++++++++++++++++++++++++----- > >>>>>> 1 file changed, 47 insertions(+), 8 deletions(-) > >>>>>> @@ -167,6 +196,16 @@ disabling many SVE vector lengths would be quite verbose, the ``sve<N>`` CPU > >>>>>> properties have special semantics (see "SVE CPU Property Parsing > >>>>>> Semantics"). > >>>>>> > >>>>>> +The ``custom`` CPU model needs to be configured via individual ID register > >>>>>> +field properties, for example:: > >>>>>> + > >>>>>> + $ qemu-system-aarch64 -M virt -cpu custom,SYSREG_ID_AA64ISAR0_EL1_DP=0x0 > >>>>>> + > >>>>>> +This forces ID_AA64ISAR0_EL1 DP field to 0. > >>>>> What is the "baseline" featureset implied by 'custom' ? > >>>> there is no baseline at the moment. By default this is a host > >>>> passthrough model. > >>> Why do we need to create "custom" at all, as opposed to just letting > >>> users toggle features on "-cpu host" ? > >> We could consolidate that to the current "host" model, once we figure > >> out how to handle the currently already existing properties. Models > >> based on the different architecture extensions would probably be more > >> useable in the long run; maybe "custom" has a place for testing. > > If you can set the features against "host", then any testing could > > be done with "host" surely, making 'custom' pointless ? > Yeah I do agree that we may not need to introduce this "custom" model > bus just enhance the custom host model with the capability to tweek some > features. For instance we have the case where migration between 2 Ampere > systems fails with host model but if you tweek 1 field in CTR_EL0 it > passes. So I think in itself this modality can be useful. Same for > debug/test purpose. As mentionned in the cover letter the number of > writable ID regs continue to grow and this enhanced host model gives > flexibility to test new support and may provide enhanced debug > capabilities for migration (getting a straight understanding of which ID > reg field(s) causes the migration failure could be helpful I think) FYI, in x86 target the -cpu command has had a "migratable=bool" property for a long time , which defaults to 'true' for 'host' model. This causes QEMU to explicitly drop features which would otherwise prevent migration between two hosts with identical physical CPUs. IOW, if there are some bits present in 'host' that cause migration problems on Ampere hosts, ideally either QEMU (or KVM kmod) would detect them and turn them off automatically if migratable=true is set. See commit message in 84f1b92f & 120eee7d1fd for some background info NB "migratable" is defined in i386 target code today, but conceptually we should expand/move that to apply to all targets for consistency, even if it is effectively a no-op some targets (eg if they are guaranteed migratable out of the box already with '-cpu host'). With regards, Daniel
On Mon, Nov 04 2024, Daniel P. Berrangé <berrange@redhat.com> wrote: > On Mon, Nov 04, 2024 at 03:45:13PM +0100, Eric Auger wrote: >> Hi >> >> On 10/28/24 17:09, Daniel P. Berrangé wrote: >> > On Mon, Oct 28, 2024 at 05:05:44PM +0100, Cornelia Huck wrote: >> >> On Fri, Oct 25 2024, Daniel P. Berrangé <berrange@redhat.com> wrote: >> >> >> >>> On Fri, Oct 25, 2024 at 03:28:35PM +0200, Eric Auger wrote: >> >>>> Hi Daniel, >> >>>> >> >>>> On 10/25/24 15:13, Daniel P. Berrangé wrote: >> >>>>> On Fri, Oct 25, 2024 at 12:17:40PM +0200, Eric Auger wrote: >> >>>>>> From: Cornelia Huck <cohuck@redhat.com> >> >>>>>> >> >>>>>> Add some documentation for the custom model. >> >>>>>> >> >>>>>> Signed-off-by: Eric Auger <eric.auger@redhat.com> >> >>>>>> Signed-off-by: Cornelia Huck <cohuck@redhat.com> >> >>>>>> --- >> >>>>>> docs/system/arm/cpu-features.rst | 55 +++++++++++++++++++++++++++----- >> >>>>>> 1 file changed, 47 insertions(+), 8 deletions(-) >> >>>>>> @@ -167,6 +196,16 @@ disabling many SVE vector lengths would be quite verbose, the ``sve<N>`` CPU >> >>>>>> properties have special semantics (see "SVE CPU Property Parsing >> >>>>>> Semantics"). >> >>>>>> >> >>>>>> +The ``custom`` CPU model needs to be configured via individual ID register >> >>>>>> +field properties, for example:: >> >>>>>> + >> >>>>>> + $ qemu-system-aarch64 -M virt -cpu custom,SYSREG_ID_AA64ISAR0_EL1_DP=0x0 >> >>>>>> + >> >>>>>> +This forces ID_AA64ISAR0_EL1 DP field to 0. >> >>>>> What is the "baseline" featureset implied by 'custom' ? >> >>>> there is no baseline at the moment. By default this is a host >> >>>> passthrough model. >> >>> Why do we need to create "custom" at all, as opposed to just letting >> >>> users toggle features on "-cpu host" ? >> >> We could consolidate that to the current "host" model, once we figure >> >> out how to handle the currently already existing properties. Models >> >> based on the different architecture extensions would probably be more >> >> useable in the long run; maybe "custom" has a place for testing. >> > If you can set the features against "host", then any testing could >> > be done with "host" surely, making 'custom' pointless ? >> Yeah I do agree that we may not need to introduce this "custom" model >> bus just enhance the custom host model with the capability to tweek some >> features. For instance we have the case where migration between 2 Ampere >> systems fails with host model but if you tweek 1 field in CTR_EL0 it >> passes. So I think in itself this modality can be useful. Same for >> debug/test purpose. As mentionned in the cover letter the number of >> writable ID regs continue to grow and this enhanced host model gives >> flexibility to test new support and may provide enhanced debug >> capabilities for migration (getting a straight understanding of which ID >> reg field(s) causes the migration failure could be helpful I think) > > FYI, in x86 target the -cpu command has had a "migratable=bool" property > for a long time , which defaults to 'true' for 'host' model. This causes > QEMU to explicitly drop features which would otherwise prevent migration > between two hosts with identical physical CPUs. > > IOW, if there are some bits present in 'host' that cause migration > problems on Ampere hosts, ideally either QEMU (or KVM kmod) would > detect them and turn them off automatically if migratable=true is > set. See commit message in 84f1b92f & 120eee7d1fd for some background > info How does this work for version-sensitive features -- are they always defaulting to off? How many features are left with that in the end? > > NB "migratable" is defined in i386 target code today, but conceptually > we should expand/move that to apply to all targets for consistency, > even if it is effectively a no-op some targets (eg if they are > guaranteed migratable out of the box already with '-cpu host'). How does this compare to s390x, which defines some migration-safe cpu models, based upon the different hw generations? If I look at the QEMU code for x86 and s390x, the s390x approach seems cleaner to me (probably because it came later, and therefore could start afresh without having to care for legacy things.) Given that we'll cook up a new model for Arm migration as well, we might as well start with a clean implementation :) (Not sure what this looks like on the libvirt side.)
On Mon, Nov 04, 2024 at 04:10:12PM +0100, Cornelia Huck wrote: > On Mon, Nov 04 2024, Daniel P. Berrangé <berrange@redhat.com> wrote: > > > > > FYI, in x86 target the -cpu command has had a "migratable=bool" property > > for a long time , which defaults to 'true' for 'host' model. This causes > > QEMU to explicitly drop features which would otherwise prevent migration > > between two hosts with identical physical CPUs. > > > > IOW, if there are some bits present in 'host' that cause migration > > problems on Ampere hosts, ideally either QEMU (or KVM kmod) would > > detect them and turn them off automatically if migratable=true is > > set. See commit message in 84f1b92f & 120eee7d1fd for some background > > info > > How does this work for version-sensitive features -- are they always > defaulting to off? How many features are left with that in the end? Do you mean QEMU versions here ? The non-migratable feature list is just hardcoded in QEMU right now, and there's only 1 of them. eg grep for 'unmigratable_flags' Note, that "migratable" property is not defining a general purpose migration mask between different hw generations. It was specifically blocking just stuff that is known to make migration impossible, even if HW is identical on both sides. > > NB "migratable" is defined in i386 target code today, but conceptually > > we should expand/move that to apply to all targets for consistency, > > even if it is effectively a no-op some targets (eg if they are > > guaranteed migratable out of the box already with '-cpu host'). > > How does this compare to s390x, which defines some migration-safe cpu > models, based upon the different hw generations? If I look at the QEMU > code for x86 and s390x, the s390x approach seems cleaner to me (probably > because it came later, and therefore could start afresh without having > to care for legacy things.) Given that we'll cook up a new model for Arm > migration as well, we might as well start with a clean implementation :) The impression I get (as an distant observer) is that CPUs on s390x in general have less complexity to worry about. A combination of not having a vendor who creates loads off different SKUs for the same CPU model family with slight variations between each, and also not seeming to have a situation where CPU flags a known to disappear (or appear) arbitrarily in microcode updates. The s390x idea of a "migratable" and "non migratable" model for each HW generation is a nice simplification, but I can't see how it could be made to work for x86 when you can't predict ahead of time what features are going to be removed from existing HW definition by the next microcode update, or by the next CPU SKU that removes a feature you had assumed would always be present in a given HW generation. I don't know much about how ARM world works, but having lots of vendors competing with their own custom impls makes me worry complexity will be closer to x86 than to s390. If the ARM specifications define a minimum require featureset for each HW generation, maybe you can define a model based on that ? You might still want to have vendor specific models though, if there are compelling features they expose which are optional, or non-standardized. With regards, Daniel
Hi Kashyap, On 10/28/24 22:17, Kashyap Chamarthy wrote: > On Fri, Oct 25, 2024 at 12:17:40PM +0200, Eric Auger wrote: >> From: Cornelia Huck <cohuck@redhat.com> >> >> Add some documentation for the custom model. >> >> Signed-off-by: Eric Auger <eric.auger@redhat.com> >> Signed-off-by: Cornelia Huck <cohuck@redhat.com> >> --- >> docs/system/arm/cpu-features.rst | 55 +++++++++++++++++++++++++++----- >> 1 file changed, 47 insertions(+), 8 deletions(-) >> >> diff --git a/docs/system/arm/cpu-features.rst b/docs/system/arm/cpu-features.rst >> index a5fb929243..962a2c6c26 100644 >> --- a/docs/system/arm/cpu-features.rst >> +++ b/docs/system/arm/cpu-features.rst >> @@ -2,7 +2,10 @@ Arm CPU Features > [...] > >> +Using the ``host`` type means the guest is provided all the same CPU >> +features as the host CPU type has. And, for this reason, the ``host`` >> +CPU type should enable all CPU features that the host has by default. >> + >> +In case some features need to be hidden to the guest, ``custom`` model >> +shall be used instead. This is especially useful for migration purpose. >> + >> +The ``custom`` CPU model generally is the better choice if you want more >> +flexibility or stability across different machines or with different kernel >> +versions. > Does "more flexibility or stability across different machines" also > imply "live migration compatiblity across host CPUs"? yes that's the goal > >> However, even the ``custom`` CPU model will not allow configuring >> +an arbitrary set of features; the ID registers must describe a subset of the >> +host's features, and all differences to the host's configuration must actually >> +be supported by the kernel to be deconfigured. > [...] > >> +The ``custom`` CPU model needs to be configured via individual ID register >> +field properties, for example:: >> + >> + $ qemu-system-aarch64 -M virt -cpu custom,SYSREG_ID_AA64ISAR0_EL1_DP=0x0 > If possible, it would be really helpful (and user-friendly) to be able > to specify the CPU feature names as you see under /proc/cpuinfo, and be > able to turn the flags on or off: > > -M virt -cpu franken,rndr=on,ts=on,fhm=off > > (... instead of specifying long system register IDs that groups together > a bunch of CPU features. If I understand it correctly, the register > "ID_AA64ISAR0_EL1" maps to a set of visible features listed here: > https://docs.kernel.org/arch/arm64/cpu-feature-registers.html) Not all the writable ID regs are visible through the above technique. But indeed I think we converged on the idea to use higher level feature names than ID reg field values. However we need to study the feasibility and mappings between those high level features and ID reg field values. The cons is that we need to describe this mapping manually. Besides being cumbersome this is also error prone. > > > Next, I prefix the below by noting that I wrote it before seeing > Cornelia's reply that the name "custom" is not set in stone: > https://lists.nongnu.org/archive/html/qemu-arm/2024-10/msg00987.html. > > I wonder if the word "custom" is starting to get overloaded; on x86: > > - Libvirt itself uses the term "custom" this way, to quote its > documentation[1] for the 'custom' XML attribute: > > custom > > In this mode, the 'cpu' element describes the CPU that should be > presented to the guest. This is the default when no 'mode' > attribute is specified. This mode makes it so that a persistent > guest will see the same hardware no matter what host the guest is > booted on. > > - Some management tools also follow libvirt and use the term "custom" > to refer to one of two things, (a) a specific named CPU model that > libvirt and QEMU recognize, e.g. "Cascadelake-Server"; or (b) a > named CPU model + extra CPU flags, e.g. this is how OpenStack > uses[2] "custom" to configure CPU models, and flags that can be > enabled or disabled via "+" or "-": > > [libvirt] > cpu_mode = custom > cpu_model = IvyBridge-IBRS > cpu_model_extra_flags="ss,+vmx,-pcid [...]" > > (Note the "cpu_mode" there: it is referring to the three possible > modes that libvirt and QEMU support today: 'host-passthrough', > 'host-model', and named CPU models via "custom".) > > The above config translates to this QEMU command-line: > > -cpu IvyBridge-IBRS,ss=on,vmx=on,pcid=off [...] > > Now if QEMU introduces "custom", it is likely to create some confusion. > But luckily, as referenced above, it is open to change. :) Agreed! Thank you for the references! Eric > > * * * > > FWIW, I agree with Dan here[3] that it would cause less future pain if > Arm's named CPU models also decides on a "baseline that matches some > corresponding real world silicon". I've experienced plenty of such > debugging pain in x86-land from years of troubleshooting live migration > bugs involving CPU model (in)compatibility. (Often, with help from > DanPB and Jiri Denemark). > > [1] https://docs.openstack.org/nova/latest/admin/cpu-models.html#cpu-modes > [2] https://libvirt.org/formatdomain.html#cpu-model-and-topology > [3] https://lists.nongnu.org/archive/html/qemu-arm/2024-10/msg00888.html > — [RFC 21/21] arm/cpu-features: Document custom vcpu model > > [...] >
On Mon, Nov 04 2024, Daniel P. Berrangé <berrange@redhat.com> wrote: > On Mon, Nov 04, 2024 at 04:10:12PM +0100, Cornelia Huck wrote: >> On Mon, Nov 04 2024, Daniel P. Berrangé <berrange@redhat.com> wrote: >> >> > >> > FYI, in x86 target the -cpu command has had a "migratable=bool" property >> > for a long time , which defaults to 'true' for 'host' model. This causes >> > QEMU to explicitly drop features which would otherwise prevent migration >> > between two hosts with identical physical CPUs. >> > >> > IOW, if there are some bits present in 'host' that cause migration >> > problems on Ampere hosts, ideally either QEMU (or KVM kmod) would >> > detect them and turn them off automatically if migratable=true is >> > set. See commit message in 84f1b92f & 120eee7d1fd for some background >> > info >> >> How does this work for version-sensitive features -- are they always >> defaulting to off? How many features are left with that in the end? > > Do you mean QEMU versions here ? The non-migratable feature list is > just hardcoded in QEMU right now, and there's only 1 of them. > eg grep for 'unmigratable_flags' > > Note, that "migratable" property is not defining a general purpose > migration mask between different hw generations. It was specifically > blocking just stuff that is known to make migration impossible, even > if HW is identical on both sides. I was more thinking of dependencies on the KVM version -- QEMU versions are easier to control for, but you don't really know what kernel version you are running with. In the end, we'd probably need to mark a lot of things as unmigratable. > >> > NB "migratable" is defined in i386 target code today, but conceptually >> > we should expand/move that to apply to all targets for consistency, >> > even if it is effectively a no-op some targets (eg if they are >> > guaranteed migratable out of the box already with '-cpu host'). >> >> How does this compare to s390x, which defines some migration-safe cpu >> models, based upon the different hw generations? If I look at the QEMU >> code for x86 and s390x, the s390x approach seems cleaner to me (probably >> because it came later, and therefore could start afresh without having >> to care for legacy things.) Given that we'll cook up a new model for Arm >> migration as well, we might as well start with a clean implementation :) > > The impression I get (as an distant observer) is that CPUs on s390x in > general have less complexity to worry about. A combination of not having > a vendor who creates loads off different SKUs for the same CPU model > family with slight variations between each, and also not seeming to have > a situation where CPU flags a known to disappear (or appear) arbitrarily > in microcode updates. > > The s390x idea of a "migratable" and "non migratable" model for each > HW generation is a nice simplification, but I can't see how it could > be made to work for x86 when you can't predict ahead of time what > features are going to be removed from existing HW definition by the > next microcode update, or by the next CPU SKU that removes a feature > you had assumed would always be present in a given HW generation. > > I don't know much about how ARM world works, but having lots of vendors > competing with their own custom impls makes me worry complexity will > be closer to x86 than to s390. My concern was more about code complexity, not hw complexity. We'll probably end up with a zoo of weird creatures for Arm, but I don't see a reason why the code would need to have strange things tacked on. I.e. have a set of arch extensions that you can baseline to, and have individual cpus on top, so you can deal with both well-known cpus and more boutique ones. > > If the ARM specifications define a minimum require featureset for each > HW generation, maybe you can define a model based on that ? You might > still want to have vendor specific models though, if there are compelling > features they expose which are optional, or non-standardized. We have a list of features that are optional for a given arch extension, and a list of features that are mandatory, so I think we'd be able to generate a model with the mandatory features only. Models for individual cpus could base off these. (There are currently 13 vendors defined in MIDR, but I'm not sure how often new vendors might be added, and vendors may also be more or less active.) If you have a baseline of Arm v9.2 or so, that might already go a long way. [But I obviously have no idea how well that will work when it meats reality :)]
On Mon, 4 Nov 2024 at 15:34, Eric Auger <eric.auger@redhat.com> wrote: > > Hi Kashyap, > > On 10/28/24 22:17, Kashyap Chamarthy wrote: > > On Fri, Oct 25, 2024 at 12:17:40PM +0200, Eric Auger wrote: > >> From: Cornelia Huck <cohuck@redhat.com> > >> > >> Add some documentation for the custom model. > >> > >> Signed-off-by: Eric Auger <eric.auger@redhat.com> > >> Signed-off-by: Cornelia Huck <cohuck@redhat.com> > >> --- > >> docs/system/arm/cpu-features.rst | 55 +++++++++++++++++++++++++++----- > >> 1 file changed, 47 insertions(+), 8 deletions(-) > >> > >> diff --git a/docs/system/arm/cpu-features.rst b/docs/system/arm/cpu-features.rst > >> index a5fb929243..962a2c6c26 100644 > >> --- a/docs/system/arm/cpu-features.rst > >> +++ b/docs/system/arm/cpu-features.rst > >> @@ -2,7 +2,10 @@ Arm CPU Features > > [...] > > > >> +Using the ``host`` type means the guest is provided all the same CPU > >> +features as the host CPU type has. And, for this reason, the ``host`` > >> +CPU type should enable all CPU features that the host has by default. > >> + > >> +In case some features need to be hidden to the guest, ``custom`` model > >> +shall be used instead. This is especially useful for migration purpose. > >> + > >> +The ``custom`` CPU model generally is the better choice if you want more > >> +flexibility or stability across different machines or with different kernel > >> +versions. > > Does "more flexibility or stability across different machines" also > > imply "live migration compatiblity across host CPUs"? > yes that's the goal > > > >> However, even the ``custom`` CPU model will not allow configuring > >> +an arbitrary set of features; the ID registers must describe a subset of the > >> +host's features, and all differences to the host's configuration must actually > >> +be supported by the kernel to be deconfigured. > > [...] > > > >> +The ``custom`` CPU model needs to be configured via individual ID register > >> +field properties, for example:: > >> + > >> + $ qemu-system-aarch64 -M virt -cpu custom,SYSREG_ID_AA64ISAR0_EL1_DP=0x0 > > If possible, it would be really helpful (and user-friendly) to be able > > to specify the CPU feature names as you see under /proc/cpuinfo, and be > > able to turn the flags on or off: > > > > -M virt -cpu franken,rndr=on,ts=on,fhm=off > > > > (... instead of specifying long system register IDs that groups together > > a bunch of CPU features. If I understand it correctly, the register > > "ID_AA64ISAR0_EL1" maps to a set of visible features listed here: > > https://docs.kernel.org/arch/arm64/cpu-feature-registers.html) > Not all the writable ID regs are visible through the above technique. > But indeed I think we converged on the idea to use higher level feature > names than ID reg field values. > However we need to study the feasibility and mappings between those high > level features and ID reg field values. > The cons is that we need to describe this mapping manually. Besides > being cumbersome this is also error prone. You might be interested in "Arm Architecture Features" on https://developer.arm.com/Architectures/A-Profile%20Architecture#Downloads which includes a 1.8MB Features.json which is a machine readable version of the "what are the features and their dependencies and ID registers and so on" information. But note that (a) it is alpha quality and (b) I am not personally going to try to interpret what might be reasonable to do with it based on the legal notice attached to it: that's a matter for you and your lawyer ;-) -- PMM
Hi Peter, On 11/4/24 17:30, Peter Maydell wrote: > On Mon, 4 Nov 2024 at 15:34, Eric Auger <eric.auger@redhat.com> wrote: >> Hi Kashyap, >> >> On 10/28/24 22:17, Kashyap Chamarthy wrote: >>> On Fri, Oct 25, 2024 at 12:17:40PM +0200, Eric Auger wrote: >>>> From: Cornelia Huck <cohuck@redhat.com> >>>> >>>> Add some documentation for the custom model. >>>> >>>> Signed-off-by: Eric Auger <eric.auger@redhat.com> >>>> Signed-off-by: Cornelia Huck <cohuck@redhat.com> >>>> --- >>>> docs/system/arm/cpu-features.rst | 55 +++++++++++++++++++++++++++----- >>>> 1 file changed, 47 insertions(+), 8 deletions(-) >>>> >>>> diff --git a/docs/system/arm/cpu-features.rst b/docs/system/arm/cpu-features.rst >>>> index a5fb929243..962a2c6c26 100644 >>>> --- a/docs/system/arm/cpu-features.rst >>>> +++ b/docs/system/arm/cpu-features.rst >>>> @@ -2,7 +2,10 @@ Arm CPU Features >>> [...] >>> >>>> +Using the ``host`` type means the guest is provided all the same CPU >>>> +features as the host CPU type has. And, for this reason, the ``host`` >>>> +CPU type should enable all CPU features that the host has by default. >>>> + >>>> +In case some features need to be hidden to the guest, ``custom`` model >>>> +shall be used instead. This is especially useful for migration purpose. >>>> + >>>> +The ``custom`` CPU model generally is the better choice if you want more >>>> +flexibility or stability across different machines or with different kernel >>>> +versions. >>> Does "more flexibility or stability across different machines" also >>> imply "live migration compatiblity across host CPUs"? >> yes that's the goal >>>> However, even the ``custom`` CPU model will not allow configuring >>>> +an arbitrary set of features; the ID registers must describe a subset of the >>>> +host's features, and all differences to the host's configuration must actually >>>> +be supported by the kernel to be deconfigured. >>> [...] >>> >>>> +The ``custom`` CPU model needs to be configured via individual ID register >>>> +field properties, for example:: >>>> + >>>> + $ qemu-system-aarch64 -M virt -cpu custom,SYSREG_ID_AA64ISAR0_EL1_DP=0x0 >>> If possible, it would be really helpful (and user-friendly) to be able >>> to specify the CPU feature names as you see under /proc/cpuinfo, and be >>> able to turn the flags on or off: >>> >>> -M virt -cpu franken,rndr=on,ts=on,fhm=off >>> >>> (... instead of specifying long system register IDs that groups together >>> a bunch of CPU features. If I understand it correctly, the register >>> "ID_AA64ISAR0_EL1" maps to a set of visible features listed here: >>> https://docs.kernel.org/arch/arm64/cpu-feature-registers.html) >> Not all the writable ID regs are visible through the above technique. >> But indeed I think we converged on the idea to use higher level feature >> names than ID reg field values. >> However we need to study the feasibility and mappings between those high >> level features and ID reg field values. >> The cons is that we need to describe this mapping manually. Besides >> being cumbersome this is also error prone. > You might be interested in "Arm Architecture Features" on > https://developer.arm.com/Architectures/A-Profile%20Architecture#Downloads > which includes a 1.8MB Features.json which is a machine > readable version of the "what are the features and their > dependencies and ID registers and so on" information. thank you for the link. > > But note that (a) it is alpha quality and (b) I am not personally > going to try to interpret what might be reasonable to do with it > based on the legal notice attached to it: that's a matter for > you and your lawyer ;-) Thank you for the notice. This may be similar to the ARM xml mentioned by Marc... Eric > > -- PMM >
On Mon, Nov 04, 2024 at 04:30:17PM +0000, Peter Maydell wrote: > On Mon, 4 Nov 2024 at 15:34, Eric Auger <eric.auger@redhat.com> wrote: [...] > > > If possible, it would be really helpful (and user-friendly) to be able > > > to specify the CPU feature names as you see under /proc/cpuinfo, and be > > > able to turn the flags on or off: > > > > > > -M virt -cpu franken,rndr=on,ts=on,fhm=off > > > > > > (... instead of specifying long system register IDs that groups together > > > a bunch of CPU features. If I understand it correctly, the register > > > "ID_AA64ISAR0_EL1" maps to a set of visible features listed here: > > > https://docs.kernel.org/arch/arm64/cpu-feature-registers.html) > > Not all the writable ID regs are visible through the above technique. > > But indeed I think we converged on the idea to use higher level feature > > names than ID reg field values. > > However we need to study the feasibility and mappings between those high > > level features and ID reg field values. > > The cons is that we need to describe this mapping manually. Besides > > being cumbersome this is also error prone. > > You might be interested in "Arm Architecture Features" on > https://developer.arm.com/Architectures/A-Profile%20Architecture#Downloads > which includes a 1.8MB Features.json which is a machine > readable version of the "what are the features and their > dependencies and ID registers and so on" information. I just took a quick at the JSON file. It was a bit difficult to get a sense of the structure of the file. Maybe it makes more sense to those who are more Arm-aware than me. Or maybe that's what you meant by "alpha quality" below :) Out of curiosity, I tried to find the features for the register "ID_AA64ISAR0_EL1". I was navigating via trial-and-error with `jq`. There's a lot of "right"/"left" traversing: $> jq '.parameters[136]|.constraints[1]|.right|.left|.value' Features.json "FEAT_TME" The register name is buried under this: $> jq '.parameters[136]|.constraints[1]|.right|.right|.left|.arguments' Features.json [ { "_type": "Types.Field", "value": { "field": "TME", "instance": null, "name": "ID_AA64ISAR0_EL1", "slices": null, "state": "AArch64" } } ] I was niavely expecting a more predictable structure such as: $> jq '.register[ID_AA64ISAR0_EL1]|.fields' Features.json FEAT_RNG FEAT_TLBIOS ... > But note that (a) it is alpha quality and (b) I am not personally > going to try to interpret what might be reasonable to do with it > based on the legal notice attached to it: that's a matter for > you and your lawyer ;-) Hmm, it sounds like until point (b) is clarified, this file is out of consideration from an upstream point of view. > -- PMM >
diff --git a/docs/system/arm/cpu-features.rst b/docs/system/arm/cpu-features.rst index a5fb929243..962a2c6c26 100644 --- a/docs/system/arm/cpu-features.rst +++ b/docs/system/arm/cpu-features.rst @@ -2,7 +2,10 @@ Arm CPU Features ================ CPU features are optional features that a CPU of supporting type may -choose to implement or not. In QEMU, optional CPU features have +choose to implement or not. QEMU provides two different mechanisms +to configure those features: + +1. For most CPU models, optional CPU features may have corresponding boolean CPU proprieties that, when enabled, indicate that the feature is implemented, and, conversely, when disabled, indicate that it is not implemented. An example of an Arm CPU feature @@ -29,6 +32,16 @@ supports the feature. While ``aarch64`` currently only works with KVM, it could work with TCG. CPU features that are specific to KVM are prefixed with "kvm-" and are described in "KVM VCPU Features". +2. Alternatively, the ``custom`` CPU model allows to configure optional +CPU features via the corresponding ID registers. The host kernel allows +to write a subset of ID register fields. The custom model exposes +properties for each write ID register fields. Those options are named +SYSREG_<IDREG>_<FIELD>. IDREG and FIELD names are those used in the +ARM ARM Reference Manual. They can also be found in the linux +arch/arm64/tool/sysreg file which is used to automatically generate the +description for those registers and fields. The custom model is currently +only implemented for KVM. + CPU Feature Probing =================== @@ -106,6 +119,10 @@ As expected they are now all ``false``. Only the ``pmu`` CPU feature is available. +Probing for the ``custom`` CPU model is working differently. CPU model +expansion will return the list of available SYSREG properties (matching +writable ID register fields) + A note about CPU feature dependencies ------------------------------------- @@ -119,18 +136,30 @@ independently without error. For these reasons callers should always attempt to make their desired changes all at once in order to ensure the collection is valid. +When using the ``custom`` CPU model, the provided set of ID registers +is always evaluated as a whole. + A note about CPU models and KVM ------------------------------- Named CPU models generally do not work with KVM. There are a few cases that do work, e.g. using the named CPU model ``cortex-a57`` with KVM on a -seattle host, but mostly if KVM is enabled the ``host`` CPU type must be -used. This means the guest is provided all the same CPU features as the -host CPU type has. And, for this reason, the ``host`` CPU type should -enable all CPU features that the host has by default. Indeed it's even -a bit strange to allow disabling CPU features that the host has when using -the ``host`` CPU type, but in the absence of CPU models it's the best we can -do if we want to launch guests without all the host's CPU features enabled. +seattle host, but mostly if KVM is enabled, either the ``host`` or the +``custom`` CPU types must be used. + +Using the ``host`` type means the guest is provided all the same CPU +features as the host CPU type has. And, for this reason, the ``host`` +CPU type should enable all CPU features that the host has by default. + +In case some features need to be hidden to the guest, ``custom`` model +shall be used instead. This is especially useful for migration purpose. + +The ``custom`` CPU model generally is the better choice if you want more +flexibility or stability across different machines or with different kernel +versions. However, even the ``custom`` CPU model will not allow configuring +an arbitrary set of features; the ID registers must describe a subset of the +host's features, and all differences to the host's configuration must actually +be supported by the kernel to be deconfigured. Enabling KVM also affects the ``query-cpu-model-expansion`` QMP command. The affect is not only limited to specific features, as pointed out in example @@ -167,6 +196,16 @@ disabling many SVE vector lengths would be quite verbose, the ``sve<N>`` CPU properties have special semantics (see "SVE CPU Property Parsing Semantics"). +The ``custom`` CPU model needs to be configured via individual ID register +field properties, for example:: + + $ qemu-system-aarch64 -M virt -cpu custom,SYSREG_ID_AA64ISAR0_EL1_DP=0x0 + +This forces ID_AA64ISAR0_EL1 DP field to 0. + +Note that the other CPU feature properties are not supported when using +this model. + KVM VCPU Features =================