| Message ID | 20200210141324.21090-1-maz@kernel.org (mailing list archive) |
|---|---|
| Headers | show |
| Series | Removing support for 32bit KVM/arm host | expand |
On Mon, Feb 10, 2020 at 3:13 PM Marc Zyngier <maz@kernel.org> wrote: > > KVM/arm was merged just over 7 years ago, and has lived a very quiet > life so far. It mostly works if you're prepared to deal with its > limitations, it has been a good prototype for the arm64 version, > but it suffers a few problems: > > - It is incomplete (no debug support, no PMU) > - It hasn't followed any of the architectural evolutions > - It has zero users (I don't count myself here) > - It is more and more getting in the way of new arm64 developments > > So here it is: unless someone screams and shows that they rely on > KVM/arm to be maintained upsteam, I'll remove 32bit host support > form the tree. One of the reasons that makes me confident nobody is > using it is that I never receive *any* bug report. Yes, it is perfect. > But if you depend on KVM/arm being available in mainline, please shout. > > To reiterate: 32bit guest support for arm64 stays, of course. Only > 32bit host goes. Once this is merged, I plan to move virt/kvm/arm to > arm64, and cleanup all the now unnecessary abstractions. > > The patches have been generated with the -D option to avoid spamming > everyone with huge diffs, and there is a kvm-arm/goodbye branch in > my kernel.org repository. > > Marc Zyngier (5): > arm: Unplug KVM from the build system > arm: Remove KVM from config files > arm: Remove 32bit KVM host support > arm: Remove HYP/Stage-2 page-table support > arm: Remove GICv3 vgic compatibility macros Since I'm generally happy to drop legacy code that has no users, with the "if there are any significant users that speak up, I'll revoke my support" caveat: Acked-by: Olof Johansson <olof@lixom.net> -Olof
On Mon, Feb 10, 2020 at 02:13:19PM +0000, Marc Zyngier wrote: > KVM/arm was merged just over 7 years ago, and has lived a very quiet > life so far. It mostly works if you're prepared to deal with its > limitations, it has been a good prototype for the arm64 version, > but it suffers a few problems: > > - It is incomplete (no debug support, no PMU) > - It hasn't followed any of the architectural evolutions > - It has zero users (I don't count myself here) I tend to use it to test that it still works, but that's it. > - It is more and more getting in the way of new arm64 developments To echo this last point, we're currently looking at the possibility of using KVM to isolate VMs from the host in Android. The scope of the changes we think we'll have to make would mean effectively duplicating the existing code for 32-bit or implementing a whole load of unused and untested functionality to keep the current structure. Neither of these options are particularly satisfactory from a maintainance point of view, so removing the 32-bit code if it doesn't have any significant use would be welcomed: Acked-by: Will Deacon <will@kernel.org> Will
On Mon, Feb 10, 2020 at 4:32 PM Olof Johansson <olof@lixom.net> wrote: > On Mon, Feb 10, 2020 at 3:13 PM Marc Zyngier <maz@kernel.org> wrote: > > > > KVM/arm was merged just over 7 years ago, and has lived a very quiet > > life so far. It mostly works if you're prepared to deal with its > > limitations, it has been a good prototype for the arm64 version, > > but it suffers a few problems: > > > > - It is incomplete (no debug support, no PMU) > > - It hasn't followed any of the architectural evolutions > > - It has zero users (I don't count myself here) > > - It is more and more getting in the way of new arm64 developments > > > > So here it is: unless someone screams and shows that they rely on > > KVM/arm to be maintained upsteam, I'll remove 32bit host support > > form the tree. One of the reasons that makes me confident nobody is > > using it is that I never receive *any* bug report. Yes, it is perfect. > > But if you depend on KVM/arm being available in mainline, please shout. > > > > To reiterate: 32bit guest support for arm64 stays, of course. Only > > 32bit host goes. Once this is merged, I plan to move virt/kvm/arm to > > arm64, and cleanup all the now unnecessary abstractions. I think this makes a lot of sense: we are seeing fewer new 32-bit systems that have enough RAM to be a usable virtualization host, as most new boards with more than 1GB of RAM typically pick 64-bit SoCs, and on less than 1GB it gets awfully tight to do anything useful. > Since I'm generally happy to drop legacy code that has no users, with > the "if there are any significant users that speak up, I'll revoke my > support" caveat: > > Acked-by: Olof Johansson <olof@lixom.net> Same here Acked-by: Arnd Bergmann <arnd@arndb.de>
On Mon, Feb 10, 2020 at 02:13:19PM +0000, Marc Zyngier wrote: > KVM/arm was merged just over 7 years ago, and has lived a very quiet > life so far. It mostly works if you're prepared to deal with its > limitations, it has been a good prototype for the arm64 version, > but it suffers a few problems: > > - It is incomplete (no debug support, no PMU) > - It hasn't followed any of the architectural evolutions > - It has zero users (I don't count myself here) > - It is more and more getting in the way of new arm64 developments > > So here it is: unless someone screams and shows that they rely on > KVM/arm to be maintained upsteam, I'll remove 32bit host support > form the tree. One of the reasons that makes me confident nobody is > using it is that I never receive *any* bug report. Yes, it is perfect. > But if you depend on KVM/arm being available in mainline, please shout. It is only very recently that 64-bit ARM has really started to filter down to people like me, and people like me have setup systems which use 32-bit VMs under 64-bit hosts (about a year ago now.) In fact, everything that you presently see for the *.armlinux.org.uk domain now runs inside several 32-bit ARM VMs under a 64-bit ARM host. It isn't perfect; I've found issues with qemu and libvirt. One example is the rather sub-standard RTC implementation, which means if you use libvirt's managesave across a host reboot, the guests idea of time-of-day freezes while it's asleep, and resumes when the guest is reloaded - resulting in the guests time-of-day being rather wrong, sometimes to the point that NTP gives up. That becomes very painful if you use kerberos authentication, where time-of-day is important. So, just because you haven't received any bug reports doesn't mean there aren't any users; there certainly are, there are problems, but the problems are in places other than the kernel.
On Mon, Feb 10, 2020 at 04:25:23PM +0000, Russell King - ARM Linux admin wrote: > On Mon, Feb 10, 2020 at 02:13:19PM +0000, Marc Zyngier wrote: > > KVM/arm was merged just over 7 years ago, and has lived a very quiet > > life so far. It mostly works if you're prepared to deal with its > > limitations, it has been a good prototype for the arm64 version, > > but it suffers a few problems: > > > > - It is incomplete (no debug support, no PMU) > > - It hasn't followed any of the architectural evolutions > > - It has zero users (I don't count myself here) > > - It is more and more getting in the way of new arm64 developments > > > > So here it is: unless someone screams and shows that they rely on > > KVM/arm to be maintained upsteam, I'll remove 32bit host support > > form the tree. One of the reasons that makes me confident nobody is > > using it is that I never receive *any* bug report. Yes, it is perfect. > > But if you depend on KVM/arm being available in mainline, please shout. > > It is only very recently that 64-bit ARM has really started to filter > down to people like me, and people like me have setup systems which > use 32-bit VMs under 64-bit hosts (about a year ago now.) In fact, > everything that you presently see for the *.armlinux.org.uk domain now > runs inside several 32-bit ARM VMs under a 64-bit ARM host. > > It isn't perfect; I've found issues with qemu and libvirt. One example > is the rather sub-standard RTC implementation, which means if you > use libvirt's managesave across a host reboot, the guests idea of > time-of-day freezes while it's asleep, and resumes when the guest is > reloaded - resulting in the guests time-of-day being rather wrong, > sometimes to the point that NTP gives up. That becomes very painful > if you use kerberos authentication, where time-of-day is important. > > So, just because you haven't received any bug reports doesn't mean > there aren't any users; there certainly are, there are problems, > but the problems are in places other than the kernel. ... argh, I see, you're not removing 32-bit guest on 64-bit. Ignore the above then.
On 2/10/20 2:13 PM, Marc Zyngier wrote: > KVM/arm was merged just over 7 years ago, and has lived a very quiet > life so far. It mostly works if you're prepared to deal with its > limitations, it has been a good prototype for the arm64 version, > but it suffers a few problems: > > - It is incomplete (no debug support, no PMU) > - It hasn't followed any of the architectural evolutions > - It has zero users (I don't count myself here) > - It is more and more getting in the way of new arm64 developments > > So here it is: unless someone screams and shows that they rely on > KVM/arm to be maintained upsteam, I'll remove 32bit host support > form the tree. One of the reasons that makes me confident nobody is > using it is that I never receive *any* bug report. Yes, it is perfect. > But if you depend on KVM/arm being available in mainline, please shout. > > To reiterate: 32bit guest support for arm64 stays, of course. Only > 32bit host goes. Once this is merged, I plan to move virt/kvm/arm to > arm64, and cleanup all the now unnecessary abstractions. > > The patches have been generated with the -D option to avoid spamming > everyone with huge diffs, and there is a kvm-arm/goodbye branch in > my kernel.org repository. > > Marc Zyngier (5): > arm: Unplug KVM from the build system > arm: Remove KVM from config files > arm: Remove 32bit KVM host support > arm: Remove HYP/Stage-2 page-table support > arm: Remove GICv3 vgic compatibility macros Acked-by: Vladimir Murzin <vladimir.murzin@arm.com>
On Mon, Feb 10, 2020 at 02:13:19PM +0000, Marc Zyngier wrote: > KVM/arm was merged just over 7 years ago, and has lived a very quiet > life so far. It mostly works if you're prepared to deal with its > limitations, it has been a good prototype for the arm64 version, > but it suffers a few problems: > > - It is incomplete (no debug support, no PMU) > - It hasn't followed any of the architectural evolutions > - It has zero users (I don't count myself here) > - It is more and more getting in the way of new arm64 developments > > So here it is: unless someone screams and shows that they rely on > KVM/arm to be maintained upsteam, I'll remove 32bit host support > form the tree. One of the reasons that makes me confident nobody is > using it is that I never receive *any* bug report. Yes, it is perfect. > But if you depend on KVM/arm being available in mainline, please shout. > > To reiterate: 32bit guest support for arm64 stays, of course. Only > 32bit host goes. Once this is merged, I plan to move virt/kvm/arm to > arm64, and cleanup all the now unnecessary abstractions. > > The patches have been generated with the -D option to avoid spamming > everyone with huge diffs, and there is a kvm-arm/goodbye branch in > my kernel.org repository. > > Marc Zyngier (5): > arm: Unplug KVM from the build system > arm: Remove KVM from config files > arm: Remove 32bit KVM host support > arm: Remove HYP/Stage-2 page-table support > arm: Remove GICv3 vgic compatibility macros FWIW: Acked-by: Catalin Marinas <catalin.marinas@arm.com>
On Mon, Feb 10, 2020 at 3:13 PM Marc Zyngier <maz@kernel.org> wrote: > it has been a good prototype for the arm64 version, > but it suffers a few problems: Wittgenstein wrote that sometimes an intellectual exercise work like such that you have to throw away the ladder after you climbed it. It often happens in engineering. It also echoes Fred Brooks Mythical Man-Month: "In most projects, the first system built is barely usable....Hence plan to throw one away; you will, anyhow." > To reiterate: 32bit guest support for arm64 stays, of course. Only > 32bit host goes. That sounds more useful. I won't miss it. Acked-by: Linus Walleij <linus.walleij@linaro.org> Yours, Linus Walleij
On 2020-02-10 15:13, Marc Zyngier wrote: > KVM/arm was merged just over 7 years ago, and has lived a very quiet > life so far. It mostly works if you're prepared to deal with its > limitations, it has been a good prototype for the arm64 version, > but it suffers a few problems: > > - It is incomplete (no debug support, no PMU) > - It hasn't followed any of the architectural evolutions > - It has zero users (I don't count myself here) > - It is more and more getting in the way of new arm64 developments > > So here it is: unless someone screams and shows that they rely on > KVM/arm to be maintained upsteam, I'll remove 32bit host support > form the tree. One of the reasons that makes me confident nobody is > using it is that I never receive *any* bug report. Yes, it is perfect. Not entirely true: https://lore.kernel.org/m/e2f7196ca6c70c55463a45b490f6731a@agner.ch But, after that was fixed, it actually was perfect :-D https://blog.printk.io/2016/09/kvm-with-kvmtool-on-armv7/ That said, I never used it in a real-world application, so from my side removing it is fine. -- Stefan > But if you depend on KVM/arm being available in mainline, please shout. > > To reiterate: 32bit guest support for arm64 stays, of course. Only > 32bit host goes. Once this is merged, I plan to move virt/kvm/arm to > arm64, and cleanup all the now unnecessary abstractions. > > The patches have been generated with the -D option to avoid spamming > everyone with huge diffs, and there is a kvm-arm/goodbye branch in > my kernel.org repository. > > Marc Zyngier (5): > arm: Unplug KVM from the build system > arm: Remove KVM from config files > arm: Remove 32bit KVM host support > arm: Remove HYP/Stage-2 page-table support > arm: Remove GICv3 vgic compatibility macros > > Documentation/virt/kvm/arm/hyp-abi.txt | 5 + > arch/arm/Kconfig | 2 - > arch/arm/Makefile | 1 - > arch/arm/configs/axm55xx_defconfig | 2 - > arch/arm/include/asm/arch_gicv3.h | 114 -- > arch/arm/include/asm/kvm_arm.h | 239 ---- > arch/arm/include/asm/kvm_asm.h | 77 -- > arch/arm/include/asm/kvm_coproc.h | 36 - > arch/arm/include/asm/kvm_emulate.h | 372 ------ > arch/arm/include/asm/kvm_host.h | 459 -------- > arch/arm/include/asm/kvm_hyp.h | 127 --- > arch/arm/include/asm/kvm_mmu.h | 435 ------- > arch/arm/include/asm/kvm_ras.h | 14 - > arch/arm/include/asm/pgtable-3level.h | 20 - > arch/arm/include/asm/pgtable.h | 9 - > arch/arm/include/asm/sections.h | 6 +- > arch/arm/include/asm/stage2_pgtable.h | 75 -- > arch/arm/include/asm/virt.h | 12 - > arch/arm/include/uapi/asm/kvm.h | 314 ----- > arch/arm/kernel/asm-offsets.c | 11 - > arch/arm/kernel/vmlinux-xip.lds.S | 8 - > arch/arm/kernel/vmlinux.lds.S | 8 - > arch/arm/kernel/vmlinux.lds.h | 10 - > arch/arm/kvm/Kconfig | 59 - > arch/arm/kvm/Makefile | 43 - > arch/arm/kvm/coproc.c | 1455 ------------------------ > arch/arm/kvm/coproc.h | 130 --- > arch/arm/kvm/coproc_a15.c | 39 - > arch/arm/kvm/coproc_a7.c | 42 - > arch/arm/kvm/emulate.c | 166 --- > arch/arm/kvm/guest.c | 387 ------- > arch/arm/kvm/handle_exit.c | 175 --- > arch/arm/kvm/hyp/Makefile | 34 - > arch/arm/kvm/hyp/banked-sr.c | 70 -- > arch/arm/kvm/hyp/cp15-sr.c | 72 -- > arch/arm/kvm/hyp/entry.S | 121 -- > arch/arm/kvm/hyp/hyp-entry.S | 295 ----- > arch/arm/kvm/hyp/s2-setup.c | 22 - > arch/arm/kvm/hyp/switch.c | 242 ---- > arch/arm/kvm/hyp/tlb.c | 68 -- > arch/arm/kvm/hyp/vfp.S | 57 - > arch/arm/kvm/init.S | 157 --- > arch/arm/kvm/interrupts.S | 36 - > arch/arm/kvm/irq.h | 16 - > arch/arm/kvm/reset.c | 86 -- > arch/arm/kvm/trace.h | 86 -- > arch/arm/kvm/vgic-v3-coproc.c | 27 - > arch/arm/mach-exynos/Kconfig | 2 +- > arch/arm/mm/mmu.c | 26 - > 49 files changed, 7 insertions(+), 6262 deletions(-) > delete mode 100644 arch/arm/include/asm/kvm_arm.h > delete mode 100644 arch/arm/include/asm/kvm_asm.h > delete mode 100644 arch/arm/include/asm/kvm_coproc.h > delete mode 100644 arch/arm/include/asm/kvm_emulate.h > delete mode 100644 arch/arm/include/asm/kvm_host.h > delete mode 100644 arch/arm/include/asm/kvm_hyp.h > delete mode 100644 arch/arm/include/asm/kvm_mmu.h > delete mode 100644 arch/arm/include/asm/kvm_ras.h > delete mode 100644 arch/arm/include/asm/stage2_pgtable.h > delete mode 100644 arch/arm/include/uapi/asm/kvm.h > delete mode 100644 arch/arm/kvm/Kconfig > delete mode 100644 arch/arm/kvm/Makefile > delete mode 100644 arch/arm/kvm/coproc.c > delete mode 100644 arch/arm/kvm/coproc.h > delete mode 100644 arch/arm/kvm/coproc_a15.c > delete mode 100644 arch/arm/kvm/coproc_a7.c > delete mode 100644 arch/arm/kvm/emulate.c > delete mode 100644 arch/arm/kvm/guest.c > delete mode 100644 arch/arm/kvm/handle_exit.c > delete mode 100644 arch/arm/kvm/hyp/Makefile > delete mode 100644 arch/arm/kvm/hyp/banked-sr.c > delete mode 100644 arch/arm/kvm/hyp/cp15-sr.c > delete mode 100644 arch/arm/kvm/hyp/entry.S > delete mode 100644 arch/arm/kvm/hyp/hyp-entry.S > delete mode 100644 arch/arm/kvm/hyp/s2-setup.c > delete mode 100644 arch/arm/kvm/hyp/switch.c > delete mode 100644 arch/arm/kvm/hyp/tlb.c > delete mode 100644 arch/arm/kvm/hyp/vfp.S > delete mode 100644 arch/arm/kvm/init.S > delete mode 100644 arch/arm/kvm/interrupts.S > delete mode 100644 arch/arm/kvm/irq.h > delete mode 100644 arch/arm/kvm/reset.c > delete mode 100644 arch/arm/kvm/trace.h > delete mode 100644 arch/arm/kvm/vgic-v3-coproc.c
On Mon, Feb 10, 2020 at 02:13:19PM +0000, Marc Zyngier wrote: > KVM/arm was merged just over 7 years ago, and has lived a very quiet > life so far. It mostly works if you're prepared to deal with its > limitations, it has been a good prototype for the arm64 version, > but it suffers a few problems: > > - It is incomplete (no debug support, no PMU) > - It hasn't followed any of the architectural evolutions > - It has zero users (I don't count myself here) > - It is more and more getting in the way of new arm64 developments > > So here it is: unless someone screams and shows that they rely on > KVM/arm to be maintained upsteam, I'll remove 32bit host support > form the tree. One of the reasons that makes me confident nobody is > using it is that I never receive *any* bug report. Yes, it is perfect. > But if you depend on KVM/arm being available in mainline, please shout. > > To reiterate: 32bit guest support for arm64 stays, of course. Only > 32bit host goes. Once this is merged, I plan to move virt/kvm/arm to > arm64, and cleanup all the now unnecessary abstractions. > > The patches have been generated with the -D option to avoid spamming > everyone with huge diffs, and there is a kvm-arm/goodbye branch in > my kernel.org repository. Acked-by: Christoffer Dall <christoffer.dall@arm.com>
On Mon, Feb 10, 2020 at 3:13 PM Marc Zyngier <maz@kernel.org> wrote: > > KVM/arm was merged just over 7 years ago, and has lived a very quiet > life so far. It mostly works if you're prepared to deal with its > limitations, it has been a good prototype for the arm64 version, > but it suffers a few problems: > > - It is incomplete (no debug support, no PMU) > - It hasn't followed any of the architectural evolutions > - It has zero users (I don't count myself here) > - It is more and more getting in the way of new arm64 developments > > So here it is: unless someone screams and shows that they rely on > KVM/arm to be maintained upsteam, I'll remove 32bit host support > form the tree. One of the reasons that makes me confident nobody is > using it is that I never receive *any* bug report. Yes, it is perfect. > But if you depend on KVM/arm being available in mainline, please shout. > > To reiterate: 32bit guest support for arm64 stays, of course. Only > 32bit host goes. Once this is merged, I plan to move virt/kvm/arm to > arm64, and cleanup all the now unnecessary abstractions. > > The patches have been generated with the -D option to avoid spamming > everyone with huge diffs, and there is a kvm-arm/goodbye branch in > my kernel.org repository. Just one more thought before it's gone: is there any shared code (header files?) that is used by the jailhouse hypervisor? If there is, are there any plans to merge that into the mainline kernel for arm32 in the near future? I'm guessing the answer to at least one of those questions is 'no', so we don't need to worry about it, but it seems better to ask. Arnd
On 19.02.20 16:09, Arnd Bergmann wrote: > On Mon, Feb 10, 2020 at 3:13 PM Marc Zyngier <maz@kernel.org> wrote: >> >> KVM/arm was merged just over 7 years ago, and has lived a very quiet >> life so far. It mostly works if you're prepared to deal with its >> limitations, it has been a good prototype for the arm64 version, >> but it suffers a few problems: >> >> - It is incomplete (no debug support, no PMU) >> - It hasn't followed any of the architectural evolutions >> - It has zero users (I don't count myself here) >> - It is more and more getting in the way of new arm64 developments >> >> So here it is: unless someone screams and shows that they rely on >> KVM/arm to be maintained upsteam, I'll remove 32bit host support >> form the tree. One of the reasons that makes me confident nobody is >> using it is that I never receive *any* bug report. Yes, it is perfect. >> But if you depend on KVM/arm being available in mainline, please shout. >> >> To reiterate: 32bit guest support for arm64 stays, of course. Only >> 32bit host goes. Once this is merged, I plan to move virt/kvm/arm to >> arm64, and cleanup all the now unnecessary abstractions. >> >> The patches have been generated with the -D option to avoid spamming >> everyone with huge diffs, and there is a kvm-arm/goodbye branch in >> my kernel.org repository. > > Just one more thought before it's gone: is there any shared code > (header files?) that is used by the jailhouse hypervisor? > > If there is, are there any plans to merge that into the mainline kernel > for arm32 in the near future? > > I'm guessing the answer to at least one of those questions is 'no', so > we don't need to worry about it, but it seems better to ask. Good that you mention it: There is one thing we share on ARM (and ARM64), and that is the hypervisor enabling stub, to install our own vectors. If that was to be removed as well, we would have to patch it back downstream. So far, we only carry few EXPORT_SYMBOL patches for essential enabling. That said, I was also starting to think about how long we will continue to support Jailhouse on 32-bit ARM. We currently have no supported SoC there that comes with an SMMU, and I doubt to see one still showing up. So, Jailhouse on ARM is really just a testing/demo case, maybe useful (but I didn't get concrete feedback) for cleaner collaborative AMP for real-time purposes, without security concerns. I assume 32-bit ARM will never be part of what would be proposed of Jailhouse for upstream. Jan
On 2020-02-19 15:46, Jan Kiszka wrote: > On 19.02.20 16:09, Arnd Bergmann wrote: >> On Mon, Feb 10, 2020 at 3:13 PM Marc Zyngier <maz@kernel.org> wrote: >>> >>> KVM/arm was merged just over 7 years ago, and has lived a very quiet >>> life so far. It mostly works if you're prepared to deal with its >>> limitations, it has been a good prototype for the arm64 version, >>> but it suffers a few problems: >>> >>> - It is incomplete (no debug support, no PMU) >>> - It hasn't followed any of the architectural evolutions >>> - It has zero users (I don't count myself here) >>> - It is more and more getting in the way of new arm64 developments >>> >>> So here it is: unless someone screams and shows that they rely on >>> KVM/arm to be maintained upsteam, I'll remove 32bit host support >>> form the tree. One of the reasons that makes me confident nobody is >>> using it is that I never receive *any* bug report. Yes, it is >>> perfect. >>> But if you depend on KVM/arm being available in mainline, please >>> shout. >>> >>> To reiterate: 32bit guest support for arm64 stays, of course. Only >>> 32bit host goes. Once this is merged, I plan to move virt/kvm/arm to >>> arm64, and cleanup all the now unnecessary abstractions. >>> >>> The patches have been generated with the -D option to avoid spamming >>> everyone with huge diffs, and there is a kvm-arm/goodbye branch in >>> my kernel.org repository. >> >> Just one more thought before it's gone: is there any shared code >> (header files?) that is used by the jailhouse hypervisor? >> >> If there is, are there any plans to merge that into the mainline >> kernel >> for arm32 in the near future? >> >> I'm guessing the answer to at least one of those questions is 'no', so >> we don't need to worry about it, but it seems better to ask. > > Good that you mention it: There is one thing we share on ARM (and > ARM64), and that is the hypervisor enabling stub, to install our own > vectors. If that was to be removed as well, we would have to patch it > back downstream. So far, we only carry few EXPORT_SYMBOL patches for > essential enabling. I actually have a few extra patches on top of the series, one of them actually removing the ability to register new vectors (mostly because I don't like leaving unused stuff behind), see [1]. I'll post an update so that we can discuss whether we want this particular to stay or not. > That said, I was also starting to think about how long we will > continue to support Jailhouse on 32-bit ARM. We currently have no > supported SoC there that comes with an SMMU, and I doubt to see one > still showing up. So, Jailhouse on ARM is really just a testing/demo > case, maybe useful (but I didn't get concrete feedback) for cleaner > collaborative AMP for real-time purposes, without security concerns. I > assume 32-bit ARM will never be part of what would be proposed of > Jailhouse for upstream. I guess we all come to the same conclusion... M. [1] https://git.kernel.org/pub/scm/linux/kernel/git/maz/arm-platforms.git/commit/?h=kvm-arm/goodbye&id=0943dd119105b65197adffda52c402cce28da56d
On 2020-02-19 13:53, Stefan Agner wrote: > On 2020-02-10 15:13, Marc Zyngier wrote: >> KVM/arm was merged just over 7 years ago, and has lived a very quiet >> life so far. It mostly works if you're prepared to deal with its >> limitations, it has been a good prototype for the arm64 version, >> but it suffers a few problems: >> >> - It is incomplete (no debug support, no PMU) >> - It hasn't followed any of the architectural evolutions >> - It has zero users (I don't count myself here) >> - It is more and more getting in the way of new arm64 developments >> >> So here it is: unless someone screams and shows that they rely on >> KVM/arm to be maintained upsteam, I'll remove 32bit host support >> form the tree. One of the reasons that makes me confident nobody is >> using it is that I never receive *any* bug report. Yes, it is perfect. > > Not entirely true: > https://lore.kernel.org/m/e2f7196ca6c70c55463a45b490f6731a@agner.ch And I thank you for that. This bug was actually hitting both arm and arm64, and triggered by a bogus DT (that KVM should have handled in a nicer way). What I was trying to say is that nobody reports bugs that are specific to 32bit KVM/arm. > But, after that was fixed, it actually was perfect :-D > https://blog.printk.io/2016/09/kvm-with-kvmtool-on-armv7/ Hey, neat! not sure how useful, but neat nonetheless... ;-) > That said, I never used it in a real-world application, so from my side > removing it is fine. Thanks, M.
Hi Marc, On 10.02.2020 15:13, Marc Zyngier wrote: > KVM/arm was merged just over 7 years ago, and has lived a very quiet > life so far. It mostly works if you're prepared to deal with its > limitations, it has been a good prototype for the arm64 version, > but it suffers a few problems: > > - It is incomplete (no debug support, no PMU) > - It hasn't followed any of the architectural evolutions > - It has zero users (I don't count myself here) > - It is more and more getting in the way of new arm64 developments That is a bit sad information. Mainline Exynos finally got everything that was needed to run it on the quite popular Samsung Exynos5422-based Odroid XU4/HC1/MC1 boards. According to the Odroid related forums it is being used. We also use it internally at Samsung. > So here it is: unless someone screams and shows that they rely on > KVM/arm to be maintained upsteam, I'll remove 32bit host support > form the tree. One of the reasons that makes me confident nobody is > using it is that I never receive *any* bug report. Yes, it is perfect. > But if you depend on KVM/arm being available in mainline, please shout. > > To reiterate: 32bit guest support for arm64 stays, of course. Only > 32bit host goes. Once this is merged, I plan to move virt/kvm/arm to > arm64, and cleanup all the now unnecessary abstractions. > > The patches have been generated with the -D option to avoid spamming > everyone with huge diffs, and there is a kvm-arm/goodbye branch in > my kernel.org repository. > > Marc Zyngier (5): > arm: Unplug KVM from the build system > arm: Remove KVM from config files > arm: Remove 32bit KVM host support > arm: Remove HYP/Stage-2 page-table support > arm: Remove GICv3 vgic compatibility macros > > Documentation/virt/kvm/arm/hyp-abi.txt | 5 + > arch/arm/Kconfig | 2 - > arch/arm/Makefile | 1 - > arch/arm/configs/axm55xx_defconfig | 2 - > arch/arm/include/asm/arch_gicv3.h | 114 -- > arch/arm/include/asm/kvm_arm.h | 239 ---- > arch/arm/include/asm/kvm_asm.h | 77 -- > arch/arm/include/asm/kvm_coproc.h | 36 - > arch/arm/include/asm/kvm_emulate.h | 372 ------ > arch/arm/include/asm/kvm_host.h | 459 -------- > arch/arm/include/asm/kvm_hyp.h | 127 --- > arch/arm/include/asm/kvm_mmu.h | 435 ------- > arch/arm/include/asm/kvm_ras.h | 14 - > arch/arm/include/asm/pgtable-3level.h | 20 - > arch/arm/include/asm/pgtable.h | 9 - > arch/arm/include/asm/sections.h | 6 +- > arch/arm/include/asm/stage2_pgtable.h | 75 -- > arch/arm/include/asm/virt.h | 12 - > arch/arm/include/uapi/asm/kvm.h | 314 ----- > arch/arm/kernel/asm-offsets.c | 11 - > arch/arm/kernel/vmlinux-xip.lds.S | 8 - > arch/arm/kernel/vmlinux.lds.S | 8 - > arch/arm/kernel/vmlinux.lds.h | 10 - > arch/arm/kvm/Kconfig | 59 - > arch/arm/kvm/Makefile | 43 - > arch/arm/kvm/coproc.c | 1455 ------------------------ > arch/arm/kvm/coproc.h | 130 --- > arch/arm/kvm/coproc_a15.c | 39 - > arch/arm/kvm/coproc_a7.c | 42 - > arch/arm/kvm/emulate.c | 166 --- > arch/arm/kvm/guest.c | 387 ------- > arch/arm/kvm/handle_exit.c | 175 --- > arch/arm/kvm/hyp/Makefile | 34 - > arch/arm/kvm/hyp/banked-sr.c | 70 -- > arch/arm/kvm/hyp/cp15-sr.c | 72 -- > arch/arm/kvm/hyp/entry.S | 121 -- > arch/arm/kvm/hyp/hyp-entry.S | 295 ----- > arch/arm/kvm/hyp/s2-setup.c | 22 - > arch/arm/kvm/hyp/switch.c | 242 ---- > arch/arm/kvm/hyp/tlb.c | 68 -- > arch/arm/kvm/hyp/vfp.S | 57 - > arch/arm/kvm/init.S | 157 --- > arch/arm/kvm/interrupts.S | 36 - > arch/arm/kvm/irq.h | 16 - > arch/arm/kvm/reset.c | 86 -- > arch/arm/kvm/trace.h | 86 -- > arch/arm/kvm/vgic-v3-coproc.c | 27 - > arch/arm/mach-exynos/Kconfig | 2 +- > arch/arm/mm/mmu.c | 26 - > 49 files changed, 7 insertions(+), 6262 deletions(-) > delete mode 100644 arch/arm/include/asm/kvm_arm.h > delete mode 100644 arch/arm/include/asm/kvm_asm.h > delete mode 100644 arch/arm/include/asm/kvm_coproc.h > delete mode 100644 arch/arm/include/asm/kvm_emulate.h > delete mode 100644 arch/arm/include/asm/kvm_host.h > delete mode 100644 arch/arm/include/asm/kvm_hyp.h > delete mode 100644 arch/arm/include/asm/kvm_mmu.h > delete mode 100644 arch/arm/include/asm/kvm_ras.h > delete mode 100644 arch/arm/include/asm/stage2_pgtable.h > delete mode 100644 arch/arm/include/uapi/asm/kvm.h > delete mode 100644 arch/arm/kvm/Kconfig > delete mode 100644 arch/arm/kvm/Makefile > delete mode 100644 arch/arm/kvm/coproc.c > delete mode 100644 arch/arm/kvm/coproc.h > delete mode 100644 arch/arm/kvm/coproc_a15.c > delete mode 100644 arch/arm/kvm/coproc_a7.c > delete mode 100644 arch/arm/kvm/emulate.c > delete mode 100644 arch/arm/kvm/guest.c > delete mode 100644 arch/arm/kvm/handle_exit.c > delete mode 100644 arch/arm/kvm/hyp/Makefile > delete mode 100644 arch/arm/kvm/hyp/banked-sr.c > delete mode 100644 arch/arm/kvm/hyp/cp15-sr.c > delete mode 100644 arch/arm/kvm/hyp/entry.S > delete mode 100644 arch/arm/kvm/hyp/hyp-entry.S > delete mode 100644 arch/arm/kvm/hyp/s2-setup.c > delete mode 100644 arch/arm/kvm/hyp/switch.c > delete mode 100644 arch/arm/kvm/hyp/tlb.c > delete mode 100644 arch/arm/kvm/hyp/vfp.S > delete mode 100644 arch/arm/kvm/init.S > delete mode 100644 arch/arm/kvm/interrupts.S > delete mode 100644 arch/arm/kvm/irq.h > delete mode 100644 arch/arm/kvm/reset.c > delete mode 100644 arch/arm/kvm/trace.h > delete mode 100644 arch/arm/kvm/vgic-v3-coproc.c > Best regards
Hi Marek, On 2020-02-20 12:44, Marek Szyprowski wrote: > Hi Marc, > > On 10.02.2020 15:13, Marc Zyngier wrote: >> KVM/arm was merged just over 7 years ago, and has lived a very quiet >> life so far. It mostly works if you're prepared to deal with its >> limitations, it has been a good prototype for the arm64 version, >> but it suffers a few problems: >> >> - It is incomplete (no debug support, no PMU) >> - It hasn't followed any of the architectural evolutions >> - It has zero users (I don't count myself here) >> - It is more and more getting in the way of new arm64 developments > > That is a bit sad information. Mainline Exynos finally got everything > that was needed to run it on the quite popular Samsung Exynos5422-based > Odroid XU4/HC1/MC1 boards. According to the Odroid related forums it is > being used. We also use it internally at Samsung. Something like "too little, too late" springs to mind, but let's be constructive. Is anyone using it in a production environment, where they rely on the latest mainline kernel having KVM support? The current proposal is to still have KVM support in 5.6, as well as ongoing support for stable kernels. If that's not enough, can you please explain your precise use case? M.
On 20/02/20 14:15, Marc Zyngier wrote: >> That is a bit sad information. Mainline Exynos finally got everything >> that was needed to run it on the quite popular Samsung Exynos5422-based >> Odroid XU4/HC1/MC1 boards. According to the Odroid related forums it is >> being used. We also use it internally at Samsung. > > Something like "too little, too late" springs to mind, but let's be > constructive. Is anyone using it in a production environment, where > they rely on the latest mainline kernel having KVM support? Depends if you consider "production environment" somebody playing at home with a SBC. But it's true that, these days, most of those that support EL2 do support ARM64, even if they are used with a 32-bit userland. Paolo
On 20/02/2020 1:15 pm, Marc Zyngier wrote: > Hi Marek, > > On 2020-02-20 12:44, Marek Szyprowski wrote: >> Hi Marc, >> >> On 10.02.2020 15:13, Marc Zyngier wrote: >>> KVM/arm was merged just over 7 years ago, and has lived a very quiet >>> life so far. It mostly works if you're prepared to deal with its >>> limitations, it has been a good prototype for the arm64 version, >>> but it suffers a few problems: >>> >>> - It is incomplete (no debug support, no PMU) >>> - It hasn't followed any of the architectural evolutions >>> - It has zero users (I don't count myself here) >>> - It is more and more getting in the way of new arm64 developments >> >> That is a bit sad information. Mainline Exynos finally got everything >> that was needed to run it on the quite popular Samsung Exynos5422-based >> Odroid XU4/HC1/MC1 boards. According to the Odroid related forums it is >> being used. We also use it internally at Samsung. > > Something like "too little, too late" springs to mind, but let's be > constructive. Is anyone using it in a production environment, where > they rely on the latest mainline kernel having KVM support? > > The current proposal is to still have KVM support in 5.6, as well as > ongoing support for stable kernels. If that's not enough, can you please > explain your precise use case? Presumably there's no *technical* reason why the stable subset of v7 support couldn't be stripped down and brought back private to arch/arm if somebody really wants and is willing to step up and look after it? Robin.
On 2020-02-20 13:32, Robin Murphy wrote: > On 20/02/2020 1:15 pm, Marc Zyngier wrote: >> Hi Marek, >> >> On 2020-02-20 12:44, Marek Szyprowski wrote: >>> Hi Marc, >>> >>> On 10.02.2020 15:13, Marc Zyngier wrote: >>>> KVM/arm was merged just over 7 years ago, and has lived a very quiet >>>> life so far. It mostly works if you're prepared to deal with its >>>> limitations, it has been a good prototype for the arm64 version, >>>> but it suffers a few problems: >>>> >>>> - It is incomplete (no debug support, no PMU) >>>> - It hasn't followed any of the architectural evolutions >>>> - It has zero users (I don't count myself here) >>>> - It is more and more getting in the way of new arm64 developments >>> >>> That is a bit sad information. Mainline Exynos finally got everything >>> that was needed to run it on the quite popular Samsung >>> Exynos5422-based >>> Odroid XU4/HC1/MC1 boards. According to the Odroid related forums it >>> is >>> being used. We also use it internally at Samsung. >> >> Something like "too little, too late" springs to mind, but let's be >> constructive. Is anyone using it in a production environment, where >> they rely on the latest mainline kernel having KVM support? >> >> The current proposal is to still have KVM support in 5.6, as well as >> ongoing support for stable kernels. If that's not enough, can you >> please >> explain your precise use case? > > Presumably there's no *technical* reason why the stable subset of v7 > support couldn't be stripped down and brought back private to arch/arm > if somebody really wants and is willing to step up and look after it? There is no technical reason at all, just a maintenance effort. The main killer is the whole MMU code, which I'm butchering with NV, and that I suspect Will will also turn upside down with his stuff. Not to mention the hypercall interface that will need a complete overhaul. If we wanted to decouple the two, we'd need to make the MMU code, the hypercalls, arm.c and a number of other bits private to 32bit. M.
On 20/02/2020 2:01 pm, Marc Zyngier wrote: > On 2020-02-20 13:32, Robin Murphy wrote: >> On 20/02/2020 1:15 pm, Marc Zyngier wrote: >>> Hi Marek, >>> >>> On 2020-02-20 12:44, Marek Szyprowski wrote: >>>> Hi Marc, >>>> >>>> On 10.02.2020 15:13, Marc Zyngier wrote: >>>>> KVM/arm was merged just over 7 years ago, and has lived a very quiet >>>>> life so far. It mostly works if you're prepared to deal with its >>>>> limitations, it has been a good prototype for the arm64 version, >>>>> but it suffers a few problems: >>>>> >>>>> - It is incomplete (no debug support, no PMU) >>>>> - It hasn't followed any of the architectural evolutions >>>>> - It has zero users (I don't count myself here) >>>>> - It is more and more getting in the way of new arm64 developments >>>> >>>> That is a bit sad information. Mainline Exynos finally got everything >>>> that was needed to run it on the quite popular Samsung Exynos5422-based >>>> Odroid XU4/HC1/MC1 boards. According to the Odroid related forums it is >>>> being used. We also use it internally at Samsung. >>> >>> Something like "too little, too late" springs to mind, but let's be >>> constructive. Is anyone using it in a production environment, where >>> they rely on the latest mainline kernel having KVM support? >>> >>> The current proposal is to still have KVM support in 5.6, as well as >>> ongoing support for stable kernels. If that's not enough, can you please >>> explain your precise use case? >> >> Presumably there's no *technical* reason why the stable subset of v7 >> support couldn't be stripped down and brought back private to arch/arm >> if somebody really wants and is willing to step up and look after it? > > There is no technical reason at all, just a maintenance effort. > > The main killer is the whole MMU code, which I'm butchering with NV, > and that I suspect Will will also turn upside down with his stuff. > Not to mention the hypercall interface that will need a complete overhaul. > > If we wanted to decouple the two, we'd need to make the MMU code, the > hypercalls, arm.c and a number of other bits private to 32bit. Right, the prospective kvm-arm maintainer's gameplan would essentially be an equivalent "move virt/kvm/arm to arch/arm/kvm" patch, but then ripping out all the Armv8 and GICv3 gubbins instead. Yes, there would then be lots of *similar* code to start with, but it would only diverge further as v8 architecture development continues independently. Anyway, I just thought it seemed worth saying out loud, to reassure folks that a realistic middle-ground between "yay bye!" and "oh no the end of the world!" does exist, namely "someone else's problem" :) Robin.
Hi I found this mailing list thread and would like to express my opinion and dependence on KVM/arm32. I hope that I'm not too late already. On Monday, 10.02.2020, 14:13 +0000 Marc Zyngier wrote: > KVM/arm was merged just over 7 years ago, and has lived a very quiet > life so far. It mostly works if you're prepared to deal with its > limitations, it has been a good prototype for the arm64 version, > but it suffers a few problems: > > - It is incomplete (no debug support, no PMU) > - It hasn't followed any of the architectural evolutions > - It has zero users (I don't count myself here) I might not be an important user, but I have been for multiple years and still am a regular user of KVM/arm32 on different devices. I use KVM on my Tegra K1 Chromebook for app development and have multiple SBCs at home on which I run VMs on using KVM+libvirt. Sure, neither of these devices has many resources available, but they are working fine. I would love to keep them in service since I haven't found arm64-based replacements that don't require hours upon hours of tinkering to just get a basic OS installation running with a mainline kernel. As an example that they can still be of use in 2020 I'd like to point out that one of the SBCs is running my DNS resolver, LDAP server, RSS reader, IRC bouncer, and shared todo list just fine, each in their separate VM. > - It is more and more getting in the way of new arm64 developments > > So here it is: unless someone screams and shows that they rely on > KVM/arm to be maintained upsteam, I'll remove 32bit host support > form the tree. *scream* > One of the reasons that makes me confident nobody is > using it is that I never receive *any* bug report. Yes, it is > perfect. This assumption is deeply flawed. Most users (including me) are not subscribed to this mailing list and will never find this thread at all. I myself stumbled upon this discussion just by chance while I was browsing the web trying to find something completely unrelated. I've been using KVM on x86, ppc and arm for many years, yet I never felt the need to report a bug on the mailing list. (This is to be interpreted as a compliment to the great work the devs of KVM have done!) Just going by the number of bugs reported on a developers mailing list is not going to paint an accurate picture. I am convinced that I'm not the only one relying on KVM/arm32 in the mainline kernel and would ask you to please reconsider keeping arm32 in the mainline kernel for a few more years until adequate arm64 replacements are available on the market and have gained proper support in the mainline kernel. I myself unfortunately do neither have the knowledge nor resources to help with development in KVM or maintaining a non-mainline kernel. > But if you depend on KVM/arm being available in mainline, please > shout. > > To reiterate: 32bit guest support for arm64 stays, of course. Only > 32bit host goes. Once this is merged, I plan to move virt/kvm/arm to > arm64, and cleanup all the now unnecessary abstractions. > > The patches have been generated with the -D option to avoid spamming > everyone with huge diffs, and there is a kvm-arm/goodbye branch in > my kernel.org repository. > > [...] Thanks for your understanding. Kind regards - Yoshi
Hi I found this mailing list thread and would like to express my opinion and dependence on KVM/arm32. I hope that I'm not too late already. On Monday, 10.02.2020, 14:13 +0000 Marc Zyngier wrote: > KVM/arm was merged just over 7 years ago, and has lived a very quiet > life so far. It mostly works if you're prepared to deal with its > limitations, it has been a good prototype for the arm64 version, > but it suffers a few problems: > > - It is incomplete (no debug support, no PMU) > - It hasn't followed any of the architectural evolutions > - It has zero users (I don't count myself here) I might not be an important user, but I have been for multiple years and still am a regular user of KVM/arm32 on different devices. I use KVM on my Tegra K1 Chromebook for app development and have multiple SBCs at home on which I run VMs on using KVM+libvirt. Sure, neither of these devices has many resources available, but they are working fine. I would love to keep them in service since I haven't found arm64-based replacements that don't require hours upon hours of tinkering to just get a basic OS installation running with a mainline kernel. As an example that they can still be of use in 2020 I'd like to point out that one of the SBCs is running my DNS resolver, LDAP server, RSS reader, IRC bouncer, and shared todo list just fine, each in their separate VM. > - It is more and more getting in the way of new arm64 developments > > So here it is: unless someone screams and shows that they rely on > KVM/arm to be maintained upsteam, I'll remove 32bit host support > form the tree. *scream* > One of the reasons that makes me confident nobody is > using it is that I never receive *any* bug report. Yes, it is > perfect. This assumption is deeply flawed. Most users (including me) are not subscribed to this mailing list and will never find this thread at all. I myself stumbled upon this discussion just by chance while I was browsing the web trying to find something completely unrelated. I've been using KVM on x86, ppc and arm for many years, yet I never felt the need to report a bug on the mailing list. (This is to be interpreted as a compliment to the great work the devs of KVM have done!) Just going by the number of bugs reported on a developers mailing list is not going to paint an accurate picture. I am convinced that I'm not the only one relying on KVM/arm32 in the mainline kernel and would ask you to please reconsider keeping arm32 in the mainline kernel for a few more years until adequate arm64 replacements are available on the market and have gained proper support in the mainline kernel. I myself unfortunately do neither have the knowledge nor resources to help with development in KVM or maintaining a non-mainline kernel. > But if you depend on KVM/arm being available in mainline, please > shout. > > To reiterate: 32bit guest support for arm64 stays, of course. Only > 32bit host goes. Once this is merged, I plan to move virt/kvm/arm to > arm64, and cleanup all the now unnecessary abstractions. > > The patches have been generated with the -D option to avoid spamming > everyone with huge diffs, and there is a kvm-arm/goodbye branch in > my kernel.org repository. > > [...] Thanks for your understanding. Kind regards - Yoshi
On Sat, Feb 22, 2020 at 3:40 PM Takashi Yoshi <takashi@yoshi.email> wrote: > On Monday, 10.02.2020, 14:13 +0000 Marc Zyngier wrote: > > KVM/arm was merged just over 7 years ago, and has lived a very quiet > > life so far. It mostly works if you're prepared to deal with its > > limitations, it has been a good prototype for the arm64 version, > > but it suffers a few problems: > > > > - It is incomplete (no debug support, no PMU) > > - It hasn't followed any of the architectural evolutions > > - It has zero users (I don't count myself here) > > I might not be an important user, but I have been for multiple years > and still am a regular user of KVM/arm32 on different devices. > > I use KVM on my Tegra K1 Chromebook for app development and have > multiple SBCs at home on which I run VMs on using KVM+libvirt. > > Sure, neither of these devices has many resources available, but they > are working fine. I would love to keep them in service since I haven't > found arm64-based replacements that don't require hours upon hours of > tinkering to just get a basic OS installation running with a mainline > kernel. > > As an example that they can still be of use in 2020 I'd like to point > out that one of the SBCs is running my DNS resolver, LDAP server, > RSS reader, IRC bouncer, and shared todo list just fine, each in their > separate VM. Thank you for providing an important data point to this question. > > - It is more and more getting in the way of new arm64 developments > > > > So here it is: unless someone screams and shows that they rely on > > KVM/arm to be maintained upsteam, I'll remove 32bit host support > > form the tree. > > *scream* > > > One of the reasons that makes me confident nobody is > > using it is that I never receive *any* bug report. Yes, it is > > perfect. > > This assumption is deeply flawed. Most users (including me) are not > subscribed to this mailing list and will never find this thread at all. > I myself stumbled upon this discussion just by chance while I was > browsing the web trying to find something completely unrelated. > > I've been using KVM on x86, ppc and arm for many years, yet I never > felt the need to report a bug on the mailing list. > (This is to be interpreted as a compliment to the great work the devs > of KVM have done!) > > Just going by the number of bugs reported on a developers mailing list > is not going to paint an accurate picture. > > I am convinced that I'm not the only one relying on KVM/arm32 in the > mainline kernel and would ask you to please reconsider keeping arm32 in > the mainline kernel for a few more years until adequate arm64 > replacements are available on the market and have gained proper support > in the mainline kernel. Can you provide some more information about how you use KVM on 32-bit machines, to make it possible to better estimate how many others might do the same, and how long you will need to upgrade to newer kernels for? In particular: - What is the smallest amount of physical RAM that you have to found to make a usable ARM/KVM host? Note that the 4GB configuration of the Tegra K1 (an rk3288) Chromebooks seems to be extremely rare in other devices, while most new 32-bit SBCs come with 1GB or less these days. - How often do you update the host kernels on those 32-bit machines that you still use to newer releases? What is the oldest/newest you run at the moment? - Are you able to move the host installation to a distribution with a long-term stable release cycle such as Debian, Ubuntu that gives you a ~five year support after a kernel release? Arnd
Dear Arnd Please excuse my late response. On Sat, 22 Feb 2020 22:31:40 +0100 Arnd Bergmann <arnd@arndb.de> wrote: > On Sat, Feb 22, 2020 at 3:40 PM Takashi Yoshi <takashi@yoshi.email> > wrote: > > On Monday, 10.02.2020, 14:13 +0000 Marc Zyngier wrote: > > > KVM/arm was merged just over 7 years ago, and has lived a very > > > quiet life so far. It mostly works if you're prepared to deal > > > with its limitations, it has been a good prototype for the arm64 > > > version, but it suffers a few problems: > > > > > > - It is incomplete (no debug support, no PMU) > > > - It hasn't followed any of the architectural evolutions > > > - It has zero users (I don't count myself here) > > > > I might not be an important user, but I have been for multiple years > > and still am a regular user of KVM/arm32 on different devices. > > > > I use KVM on my Tegra K1 Chromebook for app development and have > > multiple SBCs at home on which I run VMs on using KVM+libvirt. > > > > Sure, neither of these devices has many resources available, but > > they are working fine. I would love to keep them in service since I > > haven't found arm64-based replacements that don't require hours > > upon hours of tinkering to just get a basic OS installation running > > with a mainline kernel. > > > > As an example that they can still be of use in 2020 I'd like to > > point out that one of the SBCs is running my DNS resolver, LDAP > > server, RSS reader, IRC bouncer, and shared todo list just fine, > > each in their separate VM. > > Thank you for providing an important data point to this question. > > > > - It is more and more getting in the way of new arm64 developments > > > > > > So here it is: unless someone screams and shows that they rely on > > > KVM/arm to be maintained upsteam, I'll remove 32bit host support > > > form the tree. > > > > *scream* > > > > > One of the reasons that makes me confident nobody is > > > using it is that I never receive *any* bug report. Yes, it is > > > perfect. > > > > This assumption is deeply flawed. Most users (including me) are not > > subscribed to this mailing list and will never find this thread at > > all. I myself stumbled upon this discussion just by chance while I > > was browsing the web trying to find something completely unrelated. > > > > I've been using KVM on x86, ppc and arm for many years, yet I never > > felt the need to report a bug on the mailing list. > > (This is to be interpreted as a compliment to the great work the > > devs of KVM have done!) > > > > Just going by the number of bugs reported on a developers mailing > > list is not going to paint an accurate picture. > > > > I am convinced that I'm not the only one relying on KVM/arm32 in the > > mainline kernel and would ask you to please reconsider keeping > > arm32 in the mainline kernel for a few more years until adequate > > arm64 replacements are available on the market and have gained > > proper support in the mainline kernel. > > Can you provide some more information about how you use KVM on 32-bit > machines, to make it possible to better estimate how many others might > do the same, Sure. First of all I own different ARM boards. Currently I virtualise on Banana Pi M1 (1GB), cubox-i (2GB) and my Acer Chromebook (4GB). The Chromebook is my travel laptop on which I have three VMs on (LAMP, PostgreSQL, kernel testing) which I primarily use to develop against. The others are "home servers", they run all kinds of things for my home (incl. DNS, LDAP, RSS-Reader, Wiki, Music-Database, RDBMS, Task-Management). > and how long you will need to upgrade to newer kernels for? I don't really have a strict policy regarding when to upgrade kernels. I just run whatever gets patched and works. Sometimes this is the latest stable release, most of the time this is the last longterm release. > In particular: > > - What is the smallest amount of physical RAM that you have to found > to make a usable ARM/KVM host? Not sure if I can answer this question adequately as the smallest of my ARM32 boards have at least 1GB of RAM, which works for sure. Since you're asking about the smallest amount I did some experiments. I spun up the testing VM on my Chromebook. It consists of a basic Gentoo userland currently running on a "reduced" 4.19 kernel (I'm sure it could be stripped further if one was determined enough). When I boot it up and log in the qemu-system process on the host uses 100MB. The memory usage (incl. cache) inside the VM is only 50 MB, though. Adding a few MB for the actual application one would want to run to these 100MB, I calculate with 160MB for a "lightweight" VM. This would mean that one could run 2-3 such VMs in just 512MB which I would count as "usable". If you were conservative with memory and used a lightweight distro, like Alpine, OpenWrt or built your own using Buildroot, I could imagine that you can make a nice little home router with virtualized DHCP server, DNS server (for home network) + resolver, TFTP and possibly VPN in 512MB. (Sounds like a cool experiment for the next time im bored :P) A very different use case could be to host unikernels using e.g. Solo5 (https://github.com/Solo5/solo5) whose hvt backend also uses KVM. I, unfortunately, don't have any experience with unikernels, but I assume that they consume a lot less memory than a full Linux VM. > Note that the 4GB configuration of the Tegra K1 (an rk3288) > Chromebooks seems to be extremely rare in other devices, while most > new 32-bit SBCs come with 1GB or less these days. I agree that 4GB seems to be really rare in 32-bit land outside of laptops like Chromebooks or Novena, but I don't agree that most 32-bit SBCs are so low specced. There are quite a few 2GB boards out there: ODROID U2/U3/X2/XU/XU3/XU4/HC1/HC2, Cubieboard 3/4/5, HummingBoard, CuBox i4/Pulse, ASUS TinkerBoard (S), Lenovo G66 TV-Box, Radxa ROCK PRO / Rock2, Nvidia Jetson TK1, Banana Pi M2U/M3, Firefly RK3288, BeagleBoard X15, OrangePi Plus 2E, Utilite, Wandboard, ClearFog, NetGate SC3100, and probably a lot more can be had with 2GB RAM. (If I were to buy an SBC for virtualization, I'd get one of these :-)) Also lets not forget all the powerful smartphones out there which could be used as virtualizers using postmarketos.org. > - How often do you update the host kernels on those 32-bit machines > that you still use to newer releases? I usually track the latest longterm branch (but I wait a bit before jumping to a new longterm). So at the moment the majority is running 4.19.x, but I'm in the process of upgrading to 5.4.x now that most of the annoying bugs should be fixed ;-) > What is the oldest/newest you run at the moment? Oldest: 4.15.18 (because of an annoying regression that's likely never going to get fixed) Newest: 5.4.20 > - Are you able to move the host installation to a distribution with a > long-term stable release cycle such as Debian, Ubuntu that gives you > a ~five year support after a kernel release? I could do that, but I would like to avoid being dependent on an old kernel as from personal experience even on so-called "longterm" releases regressions do slip in and tend not to get fixed if they're not too severe, especially in old distro kernels. Kind regards, - Yoshi