| Message ID | 20200817084110.2672-1-zhukeqian1@huawei.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | KVM: arm64: Add pvtime LPT support | expand |
On 2020-08-17 09:41, Keqian Zhu wrote: > Hi all, > > This patch series picks up the LPT pvtime feature originally developed > by Steven Price: https://patchwork.kernel.org/cover/10726499/ > > Backgroud: > > There is demand for cross-platform migration, which means we have to > solve different CPU features and arch counter frequency between hosts. > This patch series can solve the latter problem. > > About LPT: > > This implements support for Live Physical Time (LPT) which provides the > guest with a method to derive a stable counter of time during which the > guest is executing even when the guest is being migrated between hosts > with different physical counter frequencies. > > Changes on Steven Price's work: > 1. LPT structure: use symmatical semantics of scale multiplier, and use > fraction bits instead of "shift" to make everything clear. > 2. Structure allocation: host kernel does not allocates the LPT > structure, > instead it is allocated by userspace through VM attributes. The > save/restore > functionality can be removed. > 3. Since LPT structure just need update once for each guest run, add a > flag to > indicate the update status. This has two benifits: 1) avoid multiple > update > by each vCPUs. 2) If the update flag is not set, then return NOT > SUPPORT for > coressponding guest HVC call. > 4. Add VM device attributes interface for userspace configuration. > 5. Add a base LPT read/write layer to reduce code. > 6. Support ptimer scaling. > 7. Support timer event stream translation. > > Things need concern: > 1. https://developer.arm.com/docs/den0057/a needs update. LPT was explicitly removed from the spec because it doesn't really solve the problem, specially for the firmware: EFI knows nothing about this, for example. How is it going to work? Also, nobody was ever able to explain how this would work for nested virt. ARMv8.4 and ARMv8.6 have the feature set that is required to solve this problem without adding more PV to the kernel. M.
On 18/08/2020 15:41, Marc Zyngier wrote: > On 2020-08-17 09:41, Keqian Zhu wrote: >> Hi all, >> >> This patch series picks up the LPT pvtime feature originally developed >> by Steven Price: https://patchwork.kernel.org/cover/10726499/ >> >> Backgroud: >> >> There is demand for cross-platform migration, which means we have to >> solve different CPU features and arch counter frequency between hosts. >> This patch series can solve the latter problem. >> >> About LPT: >> >> This implements support for Live Physical Time (LPT) which provides the >> guest with a method to derive a stable counter of time during which the >> guest is executing even when the guest is being migrated between hosts >> with different physical counter frequencies. >> >> Changes on Steven Price's work: >> 1. LPT structure: use symmatical semantics of scale multiplier, and use >> fraction bits instead of "shift" to make everything clear. >> 2. Structure allocation: host kernel does not allocates the LPT >> structure, >> instead it is allocated by userspace through VM attributes. The >> save/restore >> functionality can be removed. >> 3. Since LPT structure just need update once for each guest run, add a >> flag to >> indicate the update status. This has two benifits: 1) avoid >> multiple update >> by each vCPUs. 2) If the update flag is not set, then return NOT >> SUPPORT for >> coressponding guest HVC call. >> 4. Add VM device attributes interface for userspace configuration. >> 5. Add a base LPT read/write layer to reduce code. >> 6. Support ptimer scaling. >> 7. Support timer event stream translation. >> >> Things need concern: >> 1. https://developer.arm.com/docs/den0057/a needs update. > > LPT was explicitly removed from the spec because it doesn't really > solve the problem, specially for the firmware: EFI knows > nothing about this, for example. How is it going to work? > Also, nobody was ever able to explain how this would work for > nested virt. > > ARMv8.4 and ARMv8.6 have the feature set that is required to solve > this problem without adding more PV to the kernel. Hi Marc, These are good points, however we do still have the situation that CPUs that don't have ARMv8.4/8.6 clearly cannot implement this. I presume the use-case Keqian is looking at predates the necessary support in the CPU - Keqian if you can provide more details on the architecture(s) involved that would be helpful. Nested virt is indeed more of an issue - we did have some ideas around using SDEI that never made it to the spec. However I would argue that the most pragmatic approach would be to not support the combination of nested virt and LPT. Hopefully that can wait until the counter scaling support is available and not require PV. We are discussing (re-)releasing the spec with the LPT parts added. If you have fundamental objections then please me know. Thanks, Steve
On 2020/8/19 16:54, Steven Price wrote: > On 18/08/2020 15:41, Marc Zyngier wrote: >> On 2020-08-17 09:41, Keqian Zhu wrote: >>> Hi all, >>> >>> This patch series picks up the LPT pvtime feature originally developed >>> by Steven Price: https://patchwork.kernel.org/cover/10726499/ >>> >>> Backgroud: >>> >>> There is demand for cross-platform migration, which means we have to >>> solve different CPU features and arch counter frequency between hosts. >>> This patch series can solve the latter problem. >>> >>> About LPT: >>> >>> This implements support for Live Physical Time (LPT) which provides the >>> guest with a method to derive a stable counter of time during which the >>> guest is executing even when the guest is being migrated between hosts >>> with different physical counter frequencies. >>> >>> Changes on Steven Price's work: >>> 1. LPT structure: use symmatical semantics of scale multiplier, and use >>> fraction bits instead of "shift" to make everything clear. >>> 2. Structure allocation: host kernel does not allocates the LPT structure, >>> instead it is allocated by userspace through VM attributes. The save/restore >>> functionality can be removed. >>> 3. Since LPT structure just need update once for each guest run, add a flag to >>> indicate the update status. This has two benifits: 1) avoid multiple update >>> by each vCPUs. 2) If the update flag is not set, then return NOT SUPPORT for >>> coressponding guest HVC call. >>> 4. Add VM device attributes interface for userspace configuration. >>> 5. Add a base LPT read/write layer to reduce code. >>> 6. Support ptimer scaling. >>> 7. Support timer event stream translation. >>> >>> Things need concern: >>> 1. https://developer.arm.com/docs/den0057/a needs update. >> >> LPT was explicitly removed from the spec because it doesn't really >> solve the problem, specially for the firmware: EFI knows >> nothing about this, for example. How is it going to work? >> Also, nobody was ever able to explain how this would work for >> nested virt. >> >> ARMv8.4 and ARMv8.6 have the feature set that is required to solve >> this problem without adding more PV to the kernel. > > Hi Marc, > > These are good points, however we do still have the situation that CPUs that don't have ARMv8.4/8.6 clearly cannot implement this. I presume the use-case Keqian is looking at predates the necessary support in the CPU - Keqian if you can provide more details on the architecture(s) involved that would be helpful. > > Nested virt is indeed more of an issue - we did have some ideas around using SDEI that never made it to the spec. However I would argue that the most pragmatic approach would be to not support the combination of nested virt and LPT. Hopefully that can wait until the counter scaling support is available and not require PV. > > We are discussing (re-)releasing the spec with the LPT parts added. If you have fundamental objections then please me know. > > Thanks, > > Steve > . > Hi Marc and Steven, In fact, I have realize a demo which utilize v8.6-ECV to present a constant timer freq to guest. It seems work well, but this approach has some shortcoming: 1. Guest access to cntvct cntv_ctl cntv_tval cntv_cval must trap to EL2. Every trap will take about hundreds of nano-seconds. For every timer interrupt, there is about 5~6 traps, so it will spend several us (this seems not a serious problem :-) ). But trap will cause big deviation for nano-sleep. 2. We have to make cntfrq be a context of guest. However, only the highest exception level has right to modify cntfrq. It means we have to add a new SMC call. 3. cntkctl controls event stream freq, so KVM should also translate the guest access of cntkctl. However we cannot trap guest access of that. Any solution for this problem? I think LPT as a software solution can solve these problems. However, as Marc said, UEFI knows nothing about LPT, and it will access vtimer/counter directly. The key point is how serious the impact is on UEFI. I can see that some UEFI runtime services and drivers/applications will access timer/counter. For runtime services, it is OK. Because we can translate the result which return from UEFI for Linux. For drivers/applications, they will feel time goes faster or slower after migration. This is a problem indeed :-) Thanks, Keqian
Hi Steven, On Wed, 19 Aug 2020 09:54:40 +0100, Steven Price <steven.price@arm.com> wrote: > > On 18/08/2020 15:41, Marc Zyngier wrote: > > On 2020-08-17 09:41, Keqian Zhu wrote: > >> Hi all, > >> > >> This patch series picks up the LPT pvtime feature originally developed > >> by Steven Price: https://patchwork.kernel.org/cover/10726499/ > >> > >> Backgroud: > >> > >> There is demand for cross-platform migration, which means we have to > >> solve different CPU features and arch counter frequency between hosts. > >> This patch series can solve the latter problem. > >> > >> About LPT: > >> > >> This implements support for Live Physical Time (LPT) which provides the > >> guest with a method to derive a stable counter of time during which the > >> guest is executing even when the guest is being migrated between hosts > >> with different physical counter frequencies. > >> > >> Changes on Steven Price's work: > >> 1. LPT structure: use symmatical semantics of scale multiplier, and use > >> fraction bits instead of "shift" to make everything clear. > >> 2. Structure allocation: host kernel does not allocates the LPT > >> structure, > >> instead it is allocated by userspace through VM attributes. The > >> save/restore > >> functionality can be removed. > >> 3. Since LPT structure just need update once for each guest run, > >> add a flag to > >> indicate the update status. This has two benifits: 1) avoid > >> multiple update > >> by each vCPUs. 2) If the update flag is not set, then return NOT > >> SUPPORT for > >> coressponding guest HVC call. > >> 4. Add VM device attributes interface for userspace configuration. > >> 5. Add a base LPT read/write layer to reduce code. > >> 6. Support ptimer scaling. > >> 7. Support timer event stream translation. > >> > >> Things need concern: > >> 1. https://developer.arm.com/docs/den0057/a needs update. > > > > LPT was explicitly removed from the spec because it doesn't really > > solve the problem, specially for the firmware: EFI knows > > nothing about this, for example. How is it going to work? > > Also, nobody was ever able to explain how this would work for > > nested virt. > > > > ARMv8.4 and ARMv8.6 have the feature set that is required to solve > > this problem without adding more PV to the kernel. > > Hi Marc, > > These are good points, however we do still have the situation that > CPUs that don't have ARMv8.4/8.6 clearly cannot implement this. I > presume the use-case Keqian is looking at predates the necessary > support in the CPU - Keqian if you can provide more details on the > architecture(s) involved that would be helpful. My take on this is that it is a fictional use case. In my experience, migration happens across *identical* systems, and *any* difference visible to guests will cause things to go wrong. Errata management gets in the way, as usual (name *one* integration that isn't broken one way or another!). Allowing migration across heterogeneous hosts requires a solution to the errata management problem, which everyone (including me) has decided to ignore so far (and I claim that not having a constant timer frequency exposed to guests is an architecture bug). > Nested virt is indeed more of an issue - we did have some ideas around > using SDEI that never made it to the spec. SDEI? Sigh... Why would SDEI be useful for NV and not for !NV? > However I would argue that the most pragmatic approach would be to > not support the combination of nested virt and LPT. Hopefully that > can wait until the counter scaling support is available and not > require PV. And have yet another set of band aids that paper over the fact that we can't get a consistent story on virtualization? No, thank you. NV is (IMHO) much more important than LPT as it has a chance of getting used. LPT is just another tick box, and the fact that ARM is ready to ignore sideline a decent portion of the architecture is a clear sign that it hasn't been thought out. > We are discussing (re-)releasing the spec with the LPT parts added. If > you have fundamental objections then please me know. I do, see above. I'm stating that the use case doesn't really exist given the state of the available HW and the fragmentation of the architecture, and that ignoring the most important innovation in the virtualization architecture since ARMv7 is at best short-sighted. Time scaling is just an instance of the errata management problem, and that is the issue that needs solving. Papering over part of the problem is not helping. M.
On Wed, Aug 19, 2020 at 09:54:40AM +0100, Steven Price wrote: > On 18/08/2020 15:41, Marc Zyngier wrote: > > On 2020-08-17 09:41, Keqian Zhu wrote: > We are discussing (re-)releasing the spec with the LPT parts added. If you > have fundamental objections then please me know. Like Marc, I argued strongly for the removal of the LPT bits on the premise that it didn't really work (e.g. when transistioning between SW agents) and so it was a pure maintenance burden. I don't think the technical arguments have changed, and I don't think it's a good idea to try to ressurect this. Please rope me in if this comes up in internal discussions. Mark.
Hi Marc, Sorry for the slow response, I've been on holiday. On 22/08/2020 11:31, Marc Zyngier wrote: > Hi Steven, > > On Wed, 19 Aug 2020 09:54:40 +0100, > Steven Price <steven.price@arm.com> wrote: >> >> On 18/08/2020 15:41, Marc Zyngier wrote: >>> On 2020-08-17 09:41, Keqian Zhu wrote: [...] >>>> >>>> Things need concern: >>>> 1. https://developer.arm.com/docs/den0057/a needs update. >>> >>> LPT was explicitly removed from the spec because it doesn't really >>> solve the problem, specially for the firmware: EFI knows >>> nothing about this, for example. How is it going to work? >>> Also, nobody was ever able to explain how this would work for >>> nested virt. >>> >>> ARMv8.4 and ARMv8.6 have the feature set that is required to solve >>> this problem without adding more PV to the kernel. >> >> Hi Marc, >> >> These are good points, however we do still have the situation that >> CPUs that don't have ARMv8.4/8.6 clearly cannot implement this. I >> presume the use-case Keqian is looking at predates the necessary >> support in the CPU - Keqian if you can provide more details on the >> architecture(s) involved that would be helpful. > > My take on this is that it is a fictional use case. In my experience, > migration happens across *identical* systems, and *any* difference > visible to guests will cause things to go wrong. Errata management > gets in the way, as usual (name *one* integration that isn't broken > one way or another!). Keqian appears to have a use case - but obviously I don't know the details. I guess Keqian needs to convince you of that. > Allowing migration across heterogeneous hosts requires a solution to > the errata management problem, which everyone (including me) has > decided to ignore so far (and I claim that not having a constant timer > frequency exposed to guests is an architecture bug). I agree - errata management needs to be solved before LPT. Between restricted subsets of hosts this doesn't seem impossible, but I guess we should stall LPT until a credible solution is proposed. I'm certainly not proposing one at the moment. >> Nested virt is indeed more of an issue - we did have some ideas around >> using SDEI that never made it to the spec. > > SDEI? Sigh... Why would SDEI be useful for NV and not for !NV? SDEI provides a way of injecting a synchronous exception on migration - although that certainly isn't the only possible mechanism. For NV we have the problem that a guest-guest may be running at the point of migration. However it's not practical for the host hypervisor to provide the necessary table directly to the guest-guest which means the guest-hypervisor must update the tables before the guest-guest is allowed to run on the new host. The only plausible route I could see for this is injecting a synchronous exception into the guest (per VCPU) to ensure any guest-guests running are exited at migration time. !NV is easier because we don't have to worry about multiple levels of para-virtualisation. >> However I would argue that the most pragmatic approach would be to >> not support the combination of nested virt and LPT. Hopefully that >> can wait until the counter scaling support is available and not >> require PV. > > And have yet another set of band aids that paper over the fact that we > can't get a consistent story on virtualization? No, thank you. > > NV is (IMHO) much more important than LPT as it has a chance of > getting used. LPT is just another tick box, and the fact that ARM is > ready to ignore sideline a decent portion of the architecture is a > clear sign that it hasn't been thought out. Different people have different priorities. NV is definitely important for many people. LPT may also be important if you've already got a bunch of VMs running on machines and you want to be able to (gradually) replace them with newer hosts which happen to have a different clock frequency. Those VMs running now clearly aren't using NV. However, I have to admit it's not me that has the use-case, so I'll leave it for others who might actually know the specifics to explain the details. >> We are discussing (re-)releasing the spec with the LPT parts added. If >> you have fundamental objections then please me know. > > I do, see above. I'm stating that the use case doesn't really exist > given the state of the available HW and the fragmentation of the > architecture, and that ignoring the most important innovation in the > virtualization architecture since ARMv7 is at best short-sighted. > > Time scaling is just an instance of the errata management problem, and > that is the issue that needs solving. Papering over part of the > problem is not helping. I fully agree - errata management is definitely the first step that needs solving. This is why I abandoned LPT originally because I don't have a generic solution and the testing I did involved really ugly hacks just to make the migration possible. For now I propose we (again) park LPT until some progress has been made on errata management. Thanks, Steve
Hi Marc and Steven, On 2020/9/2 18:09, Steven Price wrote: > Hi Marc, > > Sorry for the slow response, I've been on holiday. > > On 22/08/2020 11:31, Marc Zyngier wrote: >> Hi Steven, >> >> On Wed, 19 Aug 2020 09:54:40 +0100, >> Steven Price <steven.price@arm.com> wrote: >>> >>> On 18/08/2020 15:41, Marc Zyngier wrote: >>>> On 2020-08-17 09:41, Keqian Zhu wrote: > [...] >>>>> >>>>> Things need concern: >>>>> 1. https://developer.arm.com/docs/den0057/a needs update. >>>> >>>> LPT was explicitly removed from the spec because it doesn't really >>>> solve the problem, specially for the firmware: EFI knows >>>> nothing about this, for example. How is it going to work? >>>> Also, nobody was ever able to explain how this would work for >>>> nested virt. >>>> >>>> ARMv8.4 and ARMv8.6 have the feature set that is required to solve >>>> this problem without adding more PV to the kernel. >>> >>> Hi Marc, >>> >>> These are good points, however we do still have the situation that >>> CPUs that don't have ARMv8.4/8.6 clearly cannot implement this. I >>> presume the use-case Keqian is looking at predates the necessary >>> support in the CPU - Keqian if you can provide more details on the >>> architecture(s) involved that would be helpful. >> >> My take on this is that it is a fictional use case. In my experience, >> migration happens across *identical* systems, and *any* difference >> visible to guests will cause things to go wrong. Errata management >> gets in the way, as usual (name *one* integration that isn't broken >> one way or another!). > > Keqian appears to have a use case - but obviously I don't know the details. I guess Keqian needs to convince you of that. Sure, there is use case, but I'm very sorry that it's inconvenient to show the detail. Maybe cross-chip migration will be supported by arm64 eventually, so I think use case is not a key problem. > >> Allowing migration across heterogeneous hosts requires a solution to >> the errata management problem, which everyone (including me) has >> decided to ignore so far (and I claim that not having a constant timer >> frequency exposed to guests is an architecture bug). > > I agree - errata management needs to be solved before LPT. Between restricted subsets of hosts this doesn't seem impossible, but I guess we should stall LPT until a credible solution is proposed. I'm certainly not proposing one at the moment. > >>> Nested virt is indeed more of an issue - we did have some ideas around >>> using SDEI that never made it to the spec. >> >> SDEI? Sigh... Why would SDEI be useful for NV and not for !NV? > > SDEI provides a way of injecting a synchronous exception on migration - although that certainly isn't the only possible mechanism. For NV we have the problem that a guest-guest may be running at the point of migration. However it's not practical for the host hypervisor to provide the necessary table directly to the guest-guest which means the guest-hypervisor must update the tables before the guest-guest is allowed to run on the new host. The only plausible route I could see for this is injecting a synchronous exception into the guest (per VCPU) to ensure any guest-guests running are exited at migration time. > > !NV is easier because we don't have to worry about multiple levels of para-virtualisation. > >>> However I would argue that the most pragmatic approach would be to >>> not support the combination of nested virt and LPT. Hopefully that >>> can wait until the counter scaling support is available and not >>> require PV. >> >> And have yet another set of band aids that paper over the fact that we >> can't get a consistent story on virtualization? No, thank you. >> >> NV is (IMHO) much more important than LPT as it has a chance of >> getting used. LPT is just another tick box, and the fact that ARM is >> ready to ignore sideline a decent portion of the architecture is a >> clear sign that it hasn't been thought out. > > Different people have different priorities. NV is definitely important for many people. LPT may also be important if you've already got a bunch of VMs running on machines and you want to be able to (gradually) replace them with newer hosts which happen to have a different clock frequency. Those VMs running now clearly aren't using NV. > > However, I have to admit it's not me that has the use-case, so I'll leave it for others who might actually know the specifics to explain the details. > >>> We are discussing (re-)releasing the spec with the LPT parts added. If >>> you have fundamental objections then please me know. >> >> I do, see above. I'm stating that the use case doesn't really exist >> given the state of the available HW and the fragmentation of the >> architecture, and that ignoring the most important innovation in the >> virtualization architecture since ARMv7 is at best short-sighted. >> >> Time scaling is just an instance of the errata management problem, and >> that is the issue that needs solving. Papering over part of the >> problem is not helping. > > I fully agree - errata management is definitely the first step that needs solving. This is why I abandoned LPT originally because I don't have a generic solution and the testing I did involved really ugly hacks just to make the migration possible. > > For now I propose we (again) park LPT until some progress has been made on errata management. > > Thanks, > > Steve > . As we have discussed, to support the vtimer part of cross-chip migration, we still face many problems. Firstly, we have no complete solution to realize the basic functionality. For PV solution, LPT just handles Linux kernel, other SW agents are not involved. For non-PV solution, ARMv8.4 ext and ARMv8.6 ext is not enough. Besides the basic functionality, we should concern errata management and NV (I think this is not urgent). Giving above, I agree with Steven that re-park LPT. Thanks, Keqian
Hi all, This patch series picks up the LPT pvtime feature originally developed by Steven Price: https://patchwork.kernel.org/cover/10726499/ Backgroud: There is demand for cross-platform migration, which means we have to solve different CPU features and arch counter frequency between hosts. This patch series can solve the latter problem. About LPT: This implements support for Live Physical Time (LPT) which provides the guest with a method to derive a stable counter of time during which the guest is executing even when the guest is being migrated between hosts with different physical counter frequencies. Changes on Steven Price's work: 1. LPT structure: use symmatical semantics of scale multiplier, and use fraction bits instead of "shift" to make everything clear. 2. Structure allocation: host kernel does not allocates the LPT structure, instead it is allocated by userspace through VM attributes. The save/restore functionality can be removed. 3. Since LPT structure just need update once for each guest run, add a flag to indicate the update status. This has two benifits: 1) avoid multiple update by each vCPUs. 2) If the update flag is not set, then return NOT SUPPORT for coressponding guest HVC call. 4. Add VM device attributes interface for userspace configuration. 5. Add a base LPT read/write layer to reduce code. 6. Support ptimer scaling. 7. Support timer event stream translation. Things need concern: 1. https://developer.arm.com/docs/den0057/a needs update. 2. If a guest fails to initialize and use LPT, when it migrates to a host with different native frequency, it will see the virtual counter jump. However this concern is not a problem. 1) Guest does not support LPT or host does not enable LPT: we won't migrate guest to a host with different frequency. 2) Guest kernel faces problem when initilize LPT: guest can not boot up, so counter jump is not a problem at this situation. Patch 2 and 3 implement host KVM part. Patch 4 and 5 implement guest driver part. Keqian Zhu (5): KVM: arm64: Document pvtime LPT interface KVM: arm64: Support Live Physical Time reporting KVM: arm64: Provide VM device attributes for LPT time clocksource: arm_arch_timer: Add pvtime LPT initialization clocksource: arm_arch_timer: Use pvtime LPT Documentation/virt/kvm/arm/pvtime.rst | 78 +++++++++++-- Documentation/virt/kvm/devices/vm.rst | 30 +++++ arch/arm64/include/asm/arch_timer.h | 179 ++++++++++++++++++++++++++-- arch/arm64/include/asm/kvm_host.h | 14 +++ arch/arm64/include/uapi/asm/kvm.h | 5 + arch/arm64/kvm/arm.c | 71 ++++++++++++ arch/arm64/kvm/hypercalls.c | 5 + arch/arm64/kvm/pvtime.c | 212 +++++++++++++++++++++++++++++++++- drivers/clocksource/arm_arch_timer.c | 120 +++++++++++++++---- 9 files changed, 669 insertions(+), 45 deletions(-)