| Message ID | 1d1198c2-f362-840d-cb14-9a6d74da745c@web.de (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
| Series | KVM works on RPi4 | expand |
On Sat, 29 Jun 2019 19:09:37 +0200 Jan Kiszka <jan.kiszka@web.de> wrote: Hi Jan, > Hi all, > > just got KVM running on the Raspberry Pi4. Seems they now embedded all > required logic into that new SoC. Yeah, someone saw the light and decided to enter the 21st century by attaching a GICv2 to the thing. Who knows, they may plug a GICv3 and a SMMU in 2050 at that rate! ;-) > However, as the Raspberry kernel is not yet ready for 64-bit (and > upstream is not in sight), I had to use legacy 32-bit mode. And there we > stumble over the core detection. This little patch made it work, though: > > diff --git a/arch/arm/kvm/guest.c b/arch/arm/kvm/guest.c > index 2b8de885b2bf..01606aad73cc 100644 > --- a/arch/arm/kvm/guest.c > +++ b/arch/arm/kvm/guest.c > @@ -290,6 +290,7 @@ int __attribute_const__ kvm_target_cpu(void) > case ARM_CPU_PART_CORTEX_A7: > return KVM_ARM_TARGET_CORTEX_A7; > case ARM_CPU_PART_CORTEX_A15: > + case ARM_CPU_PART_CORTEX_A72: > return KVM_ARM_TARGET_CORTEX_A15; > default: > return -EINVAL; > > That raises the question if this is hack or a valid change and if there > is general interest in mapping 64-bit cores on 32-bit if they happen to > run in 32-bit mode. The real thing to do here would be to move to a generic target, much like we did on the 64bit side. Could you investigate that instead? It would also allow KVM to be used on other 32bit cores such as A12/A17/A32. Although some would argue that the *real* real thing to do would be "rm -rf arch/arm/kvm" and be done with it, but that's a discussion for next week... ;-) > Jan > > PS: The RPi device tree lacks description of the GICH maintenance > interrupts. Seems KVM is fine without that - because it has the > information hard-coded or because it can live without that interrupt? Nah, it really should have an interrupt here. You can end-up in situation where new virtual interrupts are delayed until the next natural exit if you don't get a maintenance interrupt. Feels like a bug. Anyway, if you know of any effort to get a 64bit kernel on that thing, I'm interested in helping. I bought one on Monday, but didn't get a change to do any hacking on it just yet... Thanks, M.
On 30.06.19 00:42, Marc Zyngier wrote: > On Sat, 29 Jun 2019 19:09:37 +0200 > Jan Kiszka <jan.kiszka@web.de> wrote: > > Hi Jan, > >> Hi all, >> >> just got KVM running on the Raspberry Pi4. Seems they now embedded all >> required logic into that new SoC. > > Yeah, someone saw the light and decided to enter the 21st century by > attaching a GICv2 to the thing. Who knows, they may plug a GICv3 and a > SMMU in 2050 at that rate! ;-) > Optimistic. >> However, as the Raspberry kernel is not yet ready for 64-bit (and >> upstream is not in sight), I had to use legacy 32-bit mode. And there we >> stumble over the core detection. This little patch made it work, though: >> >> diff --git a/arch/arm/kvm/guest.c b/arch/arm/kvm/guest.c >> index 2b8de885b2bf..01606aad73cc 100644 >> --- a/arch/arm/kvm/guest.c >> +++ b/arch/arm/kvm/guest.c >> @@ -290,6 +290,7 @@ int __attribute_const__ kvm_target_cpu(void) >> case ARM_CPU_PART_CORTEX_A7: >> return KVM_ARM_TARGET_CORTEX_A7; >> case ARM_CPU_PART_CORTEX_A15: >> + case ARM_CPU_PART_CORTEX_A72: >> return KVM_ARM_TARGET_CORTEX_A15; >> default: >> return -EINVAL; >> >> That raises the question if this is hack or a valid change and if there >> is general interest in mapping 64-bit cores on 32-bit if they happen to >> run in 32-bit mode. > > The real thing to do here would be to move to a generic target, much > like we did on the 64bit side. Could you investigate that instead? It > would also allow KVM to be used on other 32bit cores such as > A12/A17/A32. You mean something like KVM_ARM_TARGET_GENERIC_V8? Need to study that... > > Although some would argue that the *real* real thing to do would be "rm > -rf arch/arm/kvm" and be done with it, but that's a discussion for next > week... ;-) > >> Jan >> >> PS: The RPi device tree lacks description of the GICH maintenance >> interrupts. Seems KVM is fine without that - because it has the >> information hard-coded or because it can live without that interrupt? > > Nah, it really should have an interrupt here. You can end-up in > situation where new virtual interrupts are delayed until the next > natural exit if you don't get a maintenance interrupt. Feels like a bug. Probably just in their DT. How can I check if the maintenance IRQ is working? > > Anyway, if you know of any effort to get a 64bit kernel on that thing, > I'm interested in helping. I bought one on Monday, but didn't get a > change to do any hacking on it just yet... I played with compiling the rpi kernel for 64-bit. Lots of pieces from the graphic drivers are falling from the truck, but you can make it build at least. Not that it boots so far or gives any early messages. Probably that is the reason: https://github.com/raspberrypi/linux/issues/3032 Jan
On 30.06.19 11:34, Jan Kiszka wrote: > On 30.06.19 00:42, Marc Zyngier wrote: >> On Sat, 29 Jun 2019 19:09:37 +0200 >> Jan Kiszka <jan.kiszka@web.de> wrote: >>> However, as the Raspberry kernel is not yet ready for 64-bit (and >>> upstream is not in sight), I had to use legacy 32-bit mode. And there we >>> stumble over the core detection. This little patch made it work, though: >>> >>> diff --git a/arch/arm/kvm/guest.c b/arch/arm/kvm/guest.c >>> index 2b8de885b2bf..01606aad73cc 100644 >>> --- a/arch/arm/kvm/guest.c >>> +++ b/arch/arm/kvm/guest.c >>> @@ -290,6 +290,7 @@ int __attribute_const__ kvm_target_cpu(void) >>> case ARM_CPU_PART_CORTEX_A7: >>> return KVM_ARM_TARGET_CORTEX_A7; >>> case ARM_CPU_PART_CORTEX_A15: >>> + case ARM_CPU_PART_CORTEX_A72: >>> return KVM_ARM_TARGET_CORTEX_A15; >>> default: >>> return -EINVAL; >>> >>> That raises the question if this is hack or a valid change and if there >>> is general interest in mapping 64-bit cores on 32-bit if they happen to >>> run in 32-bit mode. >> >> The real thing to do here would be to move to a generic target, much >> like we did on the 64bit side. Could you investigate that instead? It >> would also allow KVM to be used on other 32bit cores such as >> A12/A17/A32. > > You mean something like KVM_ARM_TARGET_GENERIC_V8? Need to study that... > Hmm, looking at what KVM_ARM_TARGET_CORTEX_A7 and ..._A15 differentiates, I found nothing so far: kvm_reset_vcpu: switch (vcpu->arch.target) { case KVM_ARM_TARGET_CORTEX_A7: case KVM_ARM_TARGET_CORTEX_A15: reset_regs = &cortexa_regs_reset; vcpu->arch.midr = read_cpuid_id(); break; And arch/arm/kvm/coproc_a15.c looks like a copy of coproc_a7.c, just with some symbols renamed. What's the purpose of all that? Planned for something bigger but never implemented? From that perspective, there seems to be no need to arch.target and kvm_coproc_target_table at all. Jan
On 30.06.19 12:18, Jan Kiszka wrote: > On 30.06.19 11:34, Jan Kiszka wrote: >> On 30.06.19 00:42, Marc Zyngier wrote: >>> On Sat, 29 Jun 2019 19:09:37 +0200 >>> Jan Kiszka <jan.kiszka@web.de> wrote: >>>> However, as the Raspberry kernel is not yet ready for 64-bit (and >>>> upstream is not in sight), I had to use legacy 32-bit mode. And there we >>>> stumble over the core detection. This little patch made it work, though: >>>> >>>> diff --git a/arch/arm/kvm/guest.c b/arch/arm/kvm/guest.c >>>> index 2b8de885b2bf..01606aad73cc 100644 >>>> --- a/arch/arm/kvm/guest.c >>>> +++ b/arch/arm/kvm/guest.c >>>> @@ -290,6 +290,7 @@ int __attribute_const__ kvm_target_cpu(void) >>>> case ARM_CPU_PART_CORTEX_A7: >>>> return KVM_ARM_TARGET_CORTEX_A7; >>>> case ARM_CPU_PART_CORTEX_A15: >>>> + case ARM_CPU_PART_CORTEX_A72: >>>> return KVM_ARM_TARGET_CORTEX_A15; >>>> default: >>>> return -EINVAL; >>>> >>>> That raises the question if this is hack or a valid change and if there >>>> is general interest in mapping 64-bit cores on 32-bit if they happen to >>>> run in 32-bit mode. >>> >>> The real thing to do here would be to move to a generic target, much >>> like we did on the 64bit side. Could you investigate that instead? It >>> would also allow KVM to be used on other 32bit cores such as >>> A12/A17/A32. >> >> You mean something like KVM_ARM_TARGET_GENERIC_V8? Need to study that... >> > > Hmm, looking at what KVM_ARM_TARGET_CORTEX_A7 and ..._A15 differentiates, I > found nothing so far: > > kvm_reset_vcpu: > switch (vcpu->arch.target) { > case KVM_ARM_TARGET_CORTEX_A7: > case KVM_ARM_TARGET_CORTEX_A15: > reset_regs = &cortexa_regs_reset; > vcpu->arch.midr = read_cpuid_id(); > break; > > And arch/arm/kvm/coproc_a15.c looks like a copy of coproc_a7.c, just with some > symbols renamed. OK, found it: The reset values of SCTLR differ, in one bit. A15 starts with branch prediction (11) off, A7 with that feature enabled. Quite some boilerplate code for managing a single bit. For a generic target, can we simply assume A15 reset behaviour? Jan
On 6/30/19 11:49 AM, Jan Kiszka wrote: > On 30.06.19 12:18, Jan Kiszka wrote: >> On 30.06.19 11:34, Jan Kiszka wrote: >>> On 30.06.19 00:42, Marc Zyngier wrote: >>>> On Sat, 29 Jun 2019 19:09:37 +0200 >>>> Jan Kiszka <jan.kiszka@web.de> wrote: >>>>> However, as the Raspberry kernel is not yet ready for 64-bit (and >>>>> upstream is not in sight), I had to use legacy 32-bit mode. And there we >>>>> stumble over the core detection. This little patch made it work, though: >>>>> >>>>> diff --git a/arch/arm/kvm/guest.c b/arch/arm/kvm/guest.c >>>>> index 2b8de885b2bf..01606aad73cc 100644 >>>>> --- a/arch/arm/kvm/guest.c >>>>> +++ b/arch/arm/kvm/guest.c >>>>> @@ -290,6 +290,7 @@ int __attribute_const__ kvm_target_cpu(void) >>>>> case ARM_CPU_PART_CORTEX_A7: >>>>> return KVM_ARM_TARGET_CORTEX_A7; >>>>> case ARM_CPU_PART_CORTEX_A15: >>>>> + case ARM_CPU_PART_CORTEX_A72: >>>>> return KVM_ARM_TARGET_CORTEX_A15; >>>>> default: >>>>> return -EINVAL; >>>>> >>>>> That raises the question if this is hack or a valid change and if there >>>>> is general interest in mapping 64-bit cores on 32-bit if they happen to >>>>> run in 32-bit mode. >>>> >>>> The real thing to do here would be to move to a generic target, much >>>> like we did on the 64bit side. Could you investigate that instead? It >>>> would also allow KVM to be used on other 32bit cores such as >>>> A12/A17/A32. >>> >>> You mean something like KVM_ARM_TARGET_GENERIC_V8? Need to study that... >>> >> >> Hmm, looking at what KVM_ARM_TARGET_CORTEX_A7 and ..._A15 differentiates, I >> found nothing so far: >> >> kvm_reset_vcpu: >> switch (vcpu->arch.target) { >> case KVM_ARM_TARGET_CORTEX_A7: >> case KVM_ARM_TARGET_CORTEX_A15: >> reset_regs = &cortexa_regs_reset; >> vcpu->arch.midr = read_cpuid_id(); >> break; >> >> And arch/arm/kvm/coproc_a15.c looks like a copy of coproc_a7.c, just with some >> symbols renamed. > > OK, found it: The reset values of SCTLR differ, in one bit. A15 starts with > branch prediction (11) off, A7 with that feature enabled. Quite some boilerplate > code for managing a single bit. > > For a generic target, can we simply assume A15 reset behaviour? IIUC, it'd work only for ARCH_VIRT guest, which is known not to touch IMP_DEF registers. Unfortunately, other variants of guests (ARCH_VEXPRESS?) might touch such registers, for instance, l2ctlr is often used for querying number of populated cpus, and it might not be present at all on v8. Also, content of IMP_DEF register is not fixed and meaning of the bits may wary between different implementations, so guests may react differently. Just my 2p. Cheers Vladimir > > Jan > _______________________________________________ > kvmarm mailing list > kvmarm@lists.cs.columbia.edu > https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Hi Jan, On Sun, Jun 30, 2019 at 12:18:59PM +0200, Jan Kiszka wrote: > On 30.06.19 11:34, Jan Kiszka wrote: > >On 30.06.19 00:42, Marc Zyngier wrote: > >>On Sat, 29 Jun 2019 19:09:37 +0200 > >>Jan Kiszka <jan.kiszka@web.de> wrote: > >>>However, as the Raspberry kernel is not yet ready for 64-bit (and > >>>upstream is not in sight), I had to use legacy 32-bit mode. And there we > >>>stumble over the core detection. This little patch made it work, though: > >>> > >>>diff --git a/arch/arm/kvm/guest.c b/arch/arm/kvm/guest.c > >>>index 2b8de885b2bf..01606aad73cc 100644 > >>>--- a/arch/arm/kvm/guest.c > >>>+++ b/arch/arm/kvm/guest.c > >>>@@ -290,6 +290,7 @@ int __attribute_const__ kvm_target_cpu(void) > >>> case ARM_CPU_PART_CORTEX_A7: > >>> return KVM_ARM_TARGET_CORTEX_A7; > >>> case ARM_CPU_PART_CORTEX_A15: > >>>+ case ARM_CPU_PART_CORTEX_A72: > >>> return KVM_ARM_TARGET_CORTEX_A15; > >>> default: > >>> return -EINVAL; > >>> > >>>That raises the question if this is hack or a valid change and if there > >>>is general interest in mapping 64-bit cores on 32-bit if they happen to > >>>run in 32-bit mode. > >> > >>The real thing to do here would be to move to a generic target, much > >>like we did on the 64bit side. Could you investigate that instead? It > >>would also allow KVM to be used on other 32bit cores such as > >>A12/A17/A32. > > > >You mean something like KVM_ARM_TARGET_GENERIC_V8? Need to study that... > > > > Hmm, looking at what KVM_ARM_TARGET_CORTEX_A7 and ..._A15 differentiates, I > found nothing so far: > > kvm_reset_vcpu: > switch (vcpu->arch.target) { > case KVM_ARM_TARGET_CORTEX_A7: > case KVM_ARM_TARGET_CORTEX_A15: > reset_regs = &cortexa_regs_reset; > vcpu->arch.midr = read_cpuid_id(); > break; > > And arch/arm/kvm/coproc_a15.c looks like a copy of coproc_a7.c, just with some > symbols renamed. > > What's the purpose of all that? Planned for something bigger but never > implemented? From that perspective, there seems to be no need to arch.target and > kvm_coproc_target_table at all. > There was some speculation involved here, and we needed to figure out how we would deal with implementation defined behavior, so we built this support for each type of CPU etc. In reality, most CPUs that we support are pretty similar and that's why we did the generic CPU type instead. In practice, there might be a more light-weight appraoch to handling the minor differences between CPU implementations than what we have here. Thanks, Christoffer
On Mon, 26 Aug 2019 at 09:38, Christoffer Dall <christoffer.dall@arm.com> wrote: > On Sun, Jun 30, 2019 at 12:18:59PM +0200, Jan Kiszka wrote: > > Hmm, looking at what KVM_ARM_TARGET_CORTEX_A7 and ..._A15 differentiates, I > > found nothing so far: > > > > kvm_reset_vcpu: > > switch (vcpu->arch.target) { > > case KVM_ARM_TARGET_CORTEX_A7: > > case KVM_ARM_TARGET_CORTEX_A15: > > reset_regs = &cortexa_regs_reset; > > vcpu->arch.midr = read_cpuid_id(); > > break; > > > > And arch/arm/kvm/coproc_a15.c looks like a copy of coproc_a7.c, just with some > > symbols renamed. > > > > What's the purpose of all that? Planned for something bigger but never > > implemented? From that perspective, there seems to be no need to arch.target and > > kvm_coproc_target_table at all. > > > > There was some speculation involved here, and we needed to figure out > how we would deal with implementation defined behavior, so we built this > support for each type of CPU etc. > > In reality, most CPUs that we support are pretty similar and that's why > we did the generic CPU type instead. In practice, there might be a more > light-weight appraoch to handling the minor differences between CPU > implementations than what we have here. The other future-direction I think we were thinking about was that one day we'd want to support showing the guest a CPU other than what the host is, at which point you would want to be able to say specifically "give me a Cortex-A7" and have it work even if the host was a Cortex-A15. But there are significant unresolved design issues if we ever did want to go in that direction... thanks -- PMM
diff --git a/arch/arm/kvm/guest.c b/arch/arm/kvm/guest.c index 2b8de885b2bf..01606aad73cc 100644 --- a/arch/arm/kvm/guest.c +++ b/arch/arm/kvm/guest.c @@ -290,6 +290,7 @@ int __attribute_const__ kvm_target_cpu(void) case ARM_CPU_PART_CORTEX_A7: return KVM_ARM_TARGET_CORTEX_A7; case ARM_CPU_PART_CORTEX_A15: + case ARM_CPU_PART_CORTEX_A72: return KVM_ARM_TARGET_CORTEX_A15; default: return -EINVAL;