mbox series

[RFC,0/5] Removing support for 32bit KVM/arm host

Message ID 20200210141324.21090-1-maz@kernel.org (mailing list archive)
Headers show
Series Removing support for 32bit KVM/arm host | expand

Message

Marc Zyngier Feb. 10, 2020, 2:13 p.m. UTC
KVM/arm was merged just over 7 years ago, and has lived a very quiet
life so far. It mostly works if you're prepared to deal with its
limitations, it has been a good prototype for the arm64 version,
but it suffers a few problems:

- It is incomplete (no debug support, no PMU)
- It hasn't followed any of the architectural evolutions
- It has zero users (I don't count myself here)
- It is more and more getting in the way of new arm64 developments

So here it is: unless someone screams and shows that they rely on
KVM/arm to be maintained upsteam, I'll remove 32bit host support
form the tree. One of the reasons that makes me confident nobody is
using it is that I never receive *any* bug report. Yes, it is perfect.
But if you depend on KVM/arm being available in mainline, please shout.

To reiterate: 32bit guest support for arm64 stays, of course. Only
32bit host goes. Once this is merged, I plan to move virt/kvm/arm to
arm64, and cleanup all the now unnecessary abstractions.

The patches have been generated with the -D option to avoid spamming
everyone with huge diffs, and there is a kvm-arm/goodbye branch in
my kernel.org repository.

Marc Zyngier (5):
  arm: Unplug KVM from the build system
  arm: Remove KVM from config files
  arm: Remove 32bit KVM host support
  arm: Remove HYP/Stage-2 page-table support
  arm: Remove GICv3 vgic compatibility macros

 Documentation/virt/kvm/arm/hyp-abi.txt |    5 +
 arch/arm/Kconfig                       |    2 -
 arch/arm/Makefile                      |    1 -
 arch/arm/configs/axm55xx_defconfig     |    2 -
 arch/arm/include/asm/arch_gicv3.h      |  114 --
 arch/arm/include/asm/kvm_arm.h         |  239 ----
 arch/arm/include/asm/kvm_asm.h         |   77 --
 arch/arm/include/asm/kvm_coproc.h      |   36 -
 arch/arm/include/asm/kvm_emulate.h     |  372 ------
 arch/arm/include/asm/kvm_host.h        |  459 --------
 arch/arm/include/asm/kvm_hyp.h         |  127 ---
 arch/arm/include/asm/kvm_mmu.h         |  435 -------
 arch/arm/include/asm/kvm_ras.h         |   14 -
 arch/arm/include/asm/pgtable-3level.h  |   20 -
 arch/arm/include/asm/pgtable.h         |    9 -
 arch/arm/include/asm/sections.h        |    6 +-
 arch/arm/include/asm/stage2_pgtable.h  |   75 --
 arch/arm/include/asm/virt.h            |   12 -
 arch/arm/include/uapi/asm/kvm.h        |  314 -----
 arch/arm/kernel/asm-offsets.c          |   11 -
 arch/arm/kernel/vmlinux-xip.lds.S      |    8 -
 arch/arm/kernel/vmlinux.lds.S          |    8 -
 arch/arm/kernel/vmlinux.lds.h          |   10 -
 arch/arm/kvm/Kconfig                   |   59 -
 arch/arm/kvm/Makefile                  |   43 -
 arch/arm/kvm/coproc.c                  | 1455 ------------------------
 arch/arm/kvm/coproc.h                  |  130 ---
 arch/arm/kvm/coproc_a15.c              |   39 -
 arch/arm/kvm/coproc_a7.c               |   42 -
 arch/arm/kvm/emulate.c                 |  166 ---
 arch/arm/kvm/guest.c                   |  387 -------
 arch/arm/kvm/handle_exit.c             |  175 ---
 arch/arm/kvm/hyp/Makefile              |   34 -
 arch/arm/kvm/hyp/banked-sr.c           |   70 --
 arch/arm/kvm/hyp/cp15-sr.c             |   72 --
 arch/arm/kvm/hyp/entry.S               |  121 --
 arch/arm/kvm/hyp/hyp-entry.S           |  295 -----
 arch/arm/kvm/hyp/s2-setup.c            |   22 -
 arch/arm/kvm/hyp/switch.c              |  242 ----
 arch/arm/kvm/hyp/tlb.c                 |   68 --
 arch/arm/kvm/hyp/vfp.S                 |   57 -
 arch/arm/kvm/init.S                    |  157 ---
 arch/arm/kvm/interrupts.S              |   36 -
 arch/arm/kvm/irq.h                     |   16 -
 arch/arm/kvm/reset.c                   |   86 --
 arch/arm/kvm/trace.h                   |   86 --
 arch/arm/kvm/vgic-v3-coproc.c          |   27 -
 arch/arm/mach-exynos/Kconfig           |    2 +-
 arch/arm/mm/mmu.c                      |   26 -
 49 files changed, 7 insertions(+), 6262 deletions(-)
 delete mode 100644 arch/arm/include/asm/kvm_arm.h
 delete mode 100644 arch/arm/include/asm/kvm_asm.h
 delete mode 100644 arch/arm/include/asm/kvm_coproc.h
 delete mode 100644 arch/arm/include/asm/kvm_emulate.h
 delete mode 100644 arch/arm/include/asm/kvm_host.h
 delete mode 100644 arch/arm/include/asm/kvm_hyp.h
 delete mode 100644 arch/arm/include/asm/kvm_mmu.h
 delete mode 100644 arch/arm/include/asm/kvm_ras.h
 delete mode 100644 arch/arm/include/asm/stage2_pgtable.h
 delete mode 100644 arch/arm/include/uapi/asm/kvm.h
 delete mode 100644 arch/arm/kvm/Kconfig
 delete mode 100644 arch/arm/kvm/Makefile
 delete mode 100644 arch/arm/kvm/coproc.c
 delete mode 100644 arch/arm/kvm/coproc.h
 delete mode 100644 arch/arm/kvm/coproc_a15.c
 delete mode 100644 arch/arm/kvm/coproc_a7.c
 delete mode 100644 arch/arm/kvm/emulate.c
 delete mode 100644 arch/arm/kvm/guest.c
 delete mode 100644 arch/arm/kvm/handle_exit.c
 delete mode 100644 arch/arm/kvm/hyp/Makefile
 delete mode 100644 arch/arm/kvm/hyp/banked-sr.c
 delete mode 100644 arch/arm/kvm/hyp/cp15-sr.c
 delete mode 100644 arch/arm/kvm/hyp/entry.S
 delete mode 100644 arch/arm/kvm/hyp/hyp-entry.S
 delete mode 100644 arch/arm/kvm/hyp/s2-setup.c
 delete mode 100644 arch/arm/kvm/hyp/switch.c
 delete mode 100644 arch/arm/kvm/hyp/tlb.c
 delete mode 100644 arch/arm/kvm/hyp/vfp.S
 delete mode 100644 arch/arm/kvm/init.S
 delete mode 100644 arch/arm/kvm/interrupts.S
 delete mode 100644 arch/arm/kvm/irq.h
 delete mode 100644 arch/arm/kvm/reset.c
 delete mode 100644 arch/arm/kvm/trace.h
 delete mode 100644 arch/arm/kvm/vgic-v3-coproc.c

Comments

Olof Johansson Feb. 10, 2020, 3:21 p.m. UTC | #1
On Mon, Feb 10, 2020 at 3:13 PM Marc Zyngier <maz@kernel.org> wrote:
>
> KVM/arm was merged just over 7 years ago, and has lived a very quiet
> life so far. It mostly works if you're prepared to deal with its
> limitations, it has been a good prototype for the arm64 version,
> but it suffers a few problems:
>
> - It is incomplete (no debug support, no PMU)
> - It hasn't followed any of the architectural evolutions
> - It has zero users (I don't count myself here)
> - It is more and more getting in the way of new arm64 developments
>
> So here it is: unless someone screams and shows that they rely on
> KVM/arm to be maintained upsteam, I'll remove 32bit host support
> form the tree. One of the reasons that makes me confident nobody is
> using it is that I never receive *any* bug report. Yes, it is perfect.
> But if you depend on KVM/arm being available in mainline, please shout.
>
> To reiterate: 32bit guest support for arm64 stays, of course. Only
> 32bit host goes. Once this is merged, I plan to move virt/kvm/arm to
> arm64, and cleanup all the now unnecessary abstractions.
>
> The patches have been generated with the -D option to avoid spamming
> everyone with huge diffs, and there is a kvm-arm/goodbye branch in
> my kernel.org repository.
>
> Marc Zyngier (5):
>   arm: Unplug KVM from the build system
>   arm: Remove KVM from config files
>   arm: Remove 32bit KVM host support
>   arm: Remove HYP/Stage-2 page-table support
>   arm: Remove GICv3 vgic compatibility macros

Since I'm generally happy to drop legacy code that has no users, with
the "if there are any significant users that speak up, I'll revoke my
support" caveat:

Acked-by: Olof Johansson <olof@lixom.net>


-Olof
Will Deacon Feb. 10, 2020, 3:46 p.m. UTC | #2
On Mon, Feb 10, 2020 at 02:13:19PM +0000, Marc Zyngier wrote:
> KVM/arm was merged just over 7 years ago, and has lived a very quiet
> life so far. It mostly works if you're prepared to deal with its
> limitations, it has been a good prototype for the arm64 version,
> but it suffers a few problems:
> 
> - It is incomplete (no debug support, no PMU)
> - It hasn't followed any of the architectural evolutions
> - It has zero users (I don't count myself here)

I tend to use it to test that it still works, but that's it.

> - It is more and more getting in the way of new arm64 developments

To echo this last point, we're currently looking at the possibility of
using KVM to isolate VMs from the host in Android. The scope of the
changes we think we'll have to make would mean effectively duplicating
the existing code for 32-bit or implementing a whole load of unused and
untested functionality to keep the current structure. Neither of these
options are particularly satisfactory from a maintainance point of view,
so removing the 32-bit code if it doesn't have any significant use would
be welcomed:

Acked-by: Will Deacon <will@kernel.org>

Will
Arnd Bergmann Feb. 10, 2020, 3:54 p.m. UTC | #3
On Mon, Feb 10, 2020 at 4:32 PM Olof Johansson <olof@lixom.net> wrote:
> On Mon, Feb 10, 2020 at 3:13 PM Marc Zyngier <maz@kernel.org> wrote:
> >
> > KVM/arm was merged just over 7 years ago, and has lived a very quiet
> > life so far. It mostly works if you're prepared to deal with its
> > limitations, it has been a good prototype for the arm64 version,
> > but it suffers a few problems:
> >
> > - It is incomplete (no debug support, no PMU)
> > - It hasn't followed any of the architectural evolutions
> > - It has zero users (I don't count myself here)
> > - It is more and more getting in the way of new arm64 developments
> >
> > So here it is: unless someone screams and shows that they rely on
> > KVM/arm to be maintained upsteam, I'll remove 32bit host support
> > form the tree. One of the reasons that makes me confident nobody is
> > using it is that I never receive *any* bug report. Yes, it is perfect.
> > But if you depend on KVM/arm being available in mainline, please shout.
> >
> > To reiterate: 32bit guest support for arm64 stays, of course. Only
> > 32bit host goes. Once this is merged, I plan to move virt/kvm/arm to
> > arm64, and cleanup all the now unnecessary abstractions.

I think this makes a lot of sense: we are seeing fewer new 32-bit
systems that have enough RAM to be a usable virtualization host,
as most new boards with more than 1GB of RAM typically pick 64-bit
SoCs, and on less than 1GB it gets awfully tight to do anything useful.

> Since I'm generally happy to drop legacy code that has no users, with
> the "if there are any significant users that speak up, I'll revoke my
> support" caveat:
>
> Acked-by: Olof Johansson <olof@lixom.net>

Same here

Acked-by: Arnd Bergmann <arnd@arndb.de>
Russell King (Oracle) Feb. 10, 2020, 4:25 p.m. UTC | #4
On Mon, Feb 10, 2020 at 02:13:19PM +0000, Marc Zyngier wrote:
> KVM/arm was merged just over 7 years ago, and has lived a very quiet
> life so far. It mostly works if you're prepared to deal with its
> limitations, it has been a good prototype for the arm64 version,
> but it suffers a few problems:
> 
> - It is incomplete (no debug support, no PMU)
> - It hasn't followed any of the architectural evolutions
> - It has zero users (I don't count myself here)
> - It is more and more getting in the way of new arm64 developments
> 
> So here it is: unless someone screams and shows that they rely on
> KVM/arm to be maintained upsteam, I'll remove 32bit host support
> form the tree. One of the reasons that makes me confident nobody is
> using it is that I never receive *any* bug report. Yes, it is perfect.
> But if you depend on KVM/arm being available in mainline, please shout.

It is only very recently that 64-bit ARM has really started to filter
down to people like me, and people like me have setup systems which
use 32-bit VMs under 64-bit hosts (about a year ago now.)  In fact,
everything that you presently see for the *.armlinux.org.uk domain now
runs inside several 32-bit ARM VMs under a 64-bit ARM host.

It isn't perfect; I've found issues with qemu and libvirt.  One example
is the rather sub-standard RTC implementation, which means if you
use libvirt's managesave across a host reboot, the guests idea of
time-of-day freezes while it's asleep, and resumes when the guest is
reloaded - resulting in the guests time-of-day being rather wrong,
sometimes to the point that NTP gives up.  That becomes very painful
if you use kerberos authentication, where time-of-day is important.

So, just because you haven't received any bug reports doesn't mean
there aren't any users; there certainly are, there are problems,
but the problems are in places other than the kernel.
Russell King (Oracle) Feb. 10, 2020, 4:26 p.m. UTC | #5
On Mon, Feb 10, 2020 at 04:25:23PM +0000, Russell King - ARM Linux admin wrote:
> On Mon, Feb 10, 2020 at 02:13:19PM +0000, Marc Zyngier wrote:
> > KVM/arm was merged just over 7 years ago, and has lived a very quiet
> > life so far. It mostly works if you're prepared to deal with its
> > limitations, it has been a good prototype for the arm64 version,
> > but it suffers a few problems:
> > 
> > - It is incomplete (no debug support, no PMU)
> > - It hasn't followed any of the architectural evolutions
> > - It has zero users (I don't count myself here)
> > - It is more and more getting in the way of new arm64 developments
> > 
> > So here it is: unless someone screams and shows that they rely on
> > KVM/arm to be maintained upsteam, I'll remove 32bit host support
> > form the tree. One of the reasons that makes me confident nobody is
> > using it is that I never receive *any* bug report. Yes, it is perfect.
> > But if you depend on KVM/arm being available in mainline, please shout.
> 
> It is only very recently that 64-bit ARM has really started to filter
> down to people like me, and people like me have setup systems which
> use 32-bit VMs under 64-bit hosts (about a year ago now.)  In fact,
> everything that you presently see for the *.armlinux.org.uk domain now
> runs inside several 32-bit ARM VMs under a 64-bit ARM host.
> 
> It isn't perfect; I've found issues with qemu and libvirt.  One example
> is the rather sub-standard RTC implementation, which means if you
> use libvirt's managesave across a host reboot, the guests idea of
> time-of-day freezes while it's asleep, and resumes when the guest is
> reloaded - resulting in the guests time-of-day being rather wrong,
> sometimes to the point that NTP gives up.  That becomes very painful
> if you use kerberos authentication, where time-of-day is important.
> 
> So, just because you haven't received any bug reports doesn't mean
> there aren't any users; there certainly are, there are problems,
> but the problems are in places other than the kernel.

... argh, I see, you're not removing 32-bit guest on 64-bit.  Ignore
the above then.
Vladimir Murzin Feb. 11, 2020, 3:12 p.m. UTC | #6
On 2/10/20 2:13 PM, Marc Zyngier wrote:
> KVM/arm was merged just over 7 years ago, and has lived a very quiet
> life so far. It mostly works if you're prepared to deal with its
> limitations, it has been a good prototype for the arm64 version,
> but it suffers a few problems:
> 
> - It is incomplete (no debug support, no PMU)
> - It hasn't followed any of the architectural evolutions
> - It has zero users (I don't count myself here)
> - It is more and more getting in the way of new arm64 developments
> 
> So here it is: unless someone screams and shows that they rely on
> KVM/arm to be maintained upsteam, I'll remove 32bit host support
> form the tree. One of the reasons that makes me confident nobody is
> using it is that I never receive *any* bug report. Yes, it is perfect.
> But if you depend on KVM/arm being available in mainline, please shout.
> 
> To reiterate: 32bit guest support for arm64 stays, of course. Only
> 32bit host goes. Once this is merged, I plan to move virt/kvm/arm to
> arm64, and cleanup all the now unnecessary abstractions.
> 
> The patches have been generated with the -D option to avoid spamming
> everyone with huge diffs, and there is a kvm-arm/goodbye branch in
> my kernel.org repository.
> 
> Marc Zyngier (5):
>   arm: Unplug KVM from the build system
>   arm: Remove KVM from config files
>   arm: Remove 32bit KVM host support
>   arm: Remove HYP/Stage-2 page-table support
>   arm: Remove GICv3 vgic compatibility macros

Acked-by: Vladimir Murzin <vladimir.murzin@arm.com>
Catalin Marinas Feb. 11, 2020, 3:23 p.m. UTC | #7
On Mon, Feb 10, 2020 at 02:13:19PM +0000, Marc Zyngier wrote:
> KVM/arm was merged just over 7 years ago, and has lived a very quiet
> life so far. It mostly works if you're prepared to deal with its
> limitations, it has been a good prototype for the arm64 version,
> but it suffers a few problems:
> 
> - It is incomplete (no debug support, no PMU)
> - It hasn't followed any of the architectural evolutions
> - It has zero users (I don't count myself here)
> - It is more and more getting in the way of new arm64 developments
> 
> So here it is: unless someone screams and shows that they rely on
> KVM/arm to be maintained upsteam, I'll remove 32bit host support
> form the tree. One of the reasons that makes me confident nobody is
> using it is that I never receive *any* bug report. Yes, it is perfect.
> But if you depend on KVM/arm being available in mainline, please shout.
> 
> To reiterate: 32bit guest support for arm64 stays, of course. Only
> 32bit host goes. Once this is merged, I plan to move virt/kvm/arm to
> arm64, and cleanup all the now unnecessary abstractions.
> 
> The patches have been generated with the -D option to avoid spamming
> everyone with huge diffs, and there is a kvm-arm/goodbye branch in
> my kernel.org repository.
> 
> Marc Zyngier (5):
>   arm: Unplug KVM from the build system
>   arm: Remove KVM from config files
>   arm: Remove 32bit KVM host support
>   arm: Remove HYP/Stage-2 page-table support
>   arm: Remove GICv3 vgic compatibility macros

FWIW:

Acked-by: Catalin Marinas <catalin.marinas@arm.com>
Linus Walleij Feb. 17, 2020, 12:14 a.m. UTC | #8
On Mon, Feb 10, 2020 at 3:13 PM Marc Zyngier <maz@kernel.org> wrote:

> it has been a good prototype for the arm64 version,
> but it suffers a few problems:

Wittgenstein wrote that sometimes an intellectual exercise work like such
that you have to throw away the ladder after you climbed it. It often
happens in engineering.

It also echoes Fred Brooks Mythical Man-Month:
"In most projects, the first system built is barely usable....Hence plan to
throw one away; you will, anyhow."

> To reiterate: 32bit guest support for arm64 stays, of course. Only
> 32bit host goes.

That sounds more useful.

I won't miss it.
Acked-by: Linus Walleij <linus.walleij@linaro.org>

Yours,
Linus Walleij
Stefan Agner Feb. 19, 2020, 1:53 p.m. UTC | #9
On 2020-02-10 15:13, Marc Zyngier wrote:
> KVM/arm was merged just over 7 years ago, and has lived a very quiet
> life so far. It mostly works if you're prepared to deal with its
> limitations, it has been a good prototype for the arm64 version,
> but it suffers a few problems:
> 
> - It is incomplete (no debug support, no PMU)
> - It hasn't followed any of the architectural evolutions
> - It has zero users (I don't count myself here)
> - It is more and more getting in the way of new arm64 developments
> 
> So here it is: unless someone screams and shows that they rely on
> KVM/arm to be maintained upsteam, I'll remove 32bit host support
> form the tree. One of the reasons that makes me confident nobody is
> using it is that I never receive *any* bug report. Yes, it is perfect.

Not entirely true:
https://lore.kernel.org/m/e2f7196ca6c70c55463a45b490f6731a@agner.ch

But, after that was fixed, it actually was perfect :-D
https://blog.printk.io/2016/09/kvm-with-kvmtool-on-armv7/

That said, I never used it in a real-world application, so from my side
removing it is fine.

--
Stefan

> But if you depend on KVM/arm being available in mainline, please shout.
> 
> To reiterate: 32bit guest support for arm64 stays, of course. Only
> 32bit host goes. Once this is merged, I plan to move virt/kvm/arm to
> arm64, and cleanup all the now unnecessary abstractions.
> 
> The patches have been generated with the -D option to avoid spamming
> everyone with huge diffs, and there is a kvm-arm/goodbye branch in
> my kernel.org repository.
> 
> Marc Zyngier (5):
>   arm: Unplug KVM from the build system
>   arm: Remove KVM from config files
>   arm: Remove 32bit KVM host support
>   arm: Remove HYP/Stage-2 page-table support
>   arm: Remove GICv3 vgic compatibility macros
> 
>  Documentation/virt/kvm/arm/hyp-abi.txt |    5 +
>  arch/arm/Kconfig                       |    2 -
>  arch/arm/Makefile                      |    1 -
>  arch/arm/configs/axm55xx_defconfig     |    2 -
>  arch/arm/include/asm/arch_gicv3.h      |  114 --
>  arch/arm/include/asm/kvm_arm.h         |  239 ----
>  arch/arm/include/asm/kvm_asm.h         |   77 --
>  arch/arm/include/asm/kvm_coproc.h      |   36 -
>  arch/arm/include/asm/kvm_emulate.h     |  372 ------
>  arch/arm/include/asm/kvm_host.h        |  459 --------
>  arch/arm/include/asm/kvm_hyp.h         |  127 ---
>  arch/arm/include/asm/kvm_mmu.h         |  435 -------
>  arch/arm/include/asm/kvm_ras.h         |   14 -
>  arch/arm/include/asm/pgtable-3level.h  |   20 -
>  arch/arm/include/asm/pgtable.h         |    9 -
>  arch/arm/include/asm/sections.h        |    6 +-
>  arch/arm/include/asm/stage2_pgtable.h  |   75 --
>  arch/arm/include/asm/virt.h            |   12 -
>  arch/arm/include/uapi/asm/kvm.h        |  314 -----
>  arch/arm/kernel/asm-offsets.c          |   11 -
>  arch/arm/kernel/vmlinux-xip.lds.S      |    8 -
>  arch/arm/kernel/vmlinux.lds.S          |    8 -
>  arch/arm/kernel/vmlinux.lds.h          |   10 -
>  arch/arm/kvm/Kconfig                   |   59 -
>  arch/arm/kvm/Makefile                  |   43 -
>  arch/arm/kvm/coproc.c                  | 1455 ------------------------
>  arch/arm/kvm/coproc.h                  |  130 ---
>  arch/arm/kvm/coproc_a15.c              |   39 -
>  arch/arm/kvm/coproc_a7.c               |   42 -
>  arch/arm/kvm/emulate.c                 |  166 ---
>  arch/arm/kvm/guest.c                   |  387 -------
>  arch/arm/kvm/handle_exit.c             |  175 ---
>  arch/arm/kvm/hyp/Makefile              |   34 -
>  arch/arm/kvm/hyp/banked-sr.c           |   70 --
>  arch/arm/kvm/hyp/cp15-sr.c             |   72 --
>  arch/arm/kvm/hyp/entry.S               |  121 --
>  arch/arm/kvm/hyp/hyp-entry.S           |  295 -----
>  arch/arm/kvm/hyp/s2-setup.c            |   22 -
>  arch/arm/kvm/hyp/switch.c              |  242 ----
>  arch/arm/kvm/hyp/tlb.c                 |   68 --
>  arch/arm/kvm/hyp/vfp.S                 |   57 -
>  arch/arm/kvm/init.S                    |  157 ---
>  arch/arm/kvm/interrupts.S              |   36 -
>  arch/arm/kvm/irq.h                     |   16 -
>  arch/arm/kvm/reset.c                   |   86 --
>  arch/arm/kvm/trace.h                   |   86 --
>  arch/arm/kvm/vgic-v3-coproc.c          |   27 -
>  arch/arm/mach-exynos/Kconfig           |    2 +-
>  arch/arm/mm/mmu.c                      |   26 -
>  49 files changed, 7 insertions(+), 6262 deletions(-)
>  delete mode 100644 arch/arm/include/asm/kvm_arm.h
>  delete mode 100644 arch/arm/include/asm/kvm_asm.h
>  delete mode 100644 arch/arm/include/asm/kvm_coproc.h
>  delete mode 100644 arch/arm/include/asm/kvm_emulate.h
>  delete mode 100644 arch/arm/include/asm/kvm_host.h
>  delete mode 100644 arch/arm/include/asm/kvm_hyp.h
>  delete mode 100644 arch/arm/include/asm/kvm_mmu.h
>  delete mode 100644 arch/arm/include/asm/kvm_ras.h
>  delete mode 100644 arch/arm/include/asm/stage2_pgtable.h
>  delete mode 100644 arch/arm/include/uapi/asm/kvm.h
>  delete mode 100644 arch/arm/kvm/Kconfig
>  delete mode 100644 arch/arm/kvm/Makefile
>  delete mode 100644 arch/arm/kvm/coproc.c
>  delete mode 100644 arch/arm/kvm/coproc.h
>  delete mode 100644 arch/arm/kvm/coproc_a15.c
>  delete mode 100644 arch/arm/kvm/coproc_a7.c
>  delete mode 100644 arch/arm/kvm/emulate.c
>  delete mode 100644 arch/arm/kvm/guest.c
>  delete mode 100644 arch/arm/kvm/handle_exit.c
>  delete mode 100644 arch/arm/kvm/hyp/Makefile
>  delete mode 100644 arch/arm/kvm/hyp/banked-sr.c
>  delete mode 100644 arch/arm/kvm/hyp/cp15-sr.c
>  delete mode 100644 arch/arm/kvm/hyp/entry.S
>  delete mode 100644 arch/arm/kvm/hyp/hyp-entry.S
>  delete mode 100644 arch/arm/kvm/hyp/s2-setup.c
>  delete mode 100644 arch/arm/kvm/hyp/switch.c
>  delete mode 100644 arch/arm/kvm/hyp/tlb.c
>  delete mode 100644 arch/arm/kvm/hyp/vfp.S
>  delete mode 100644 arch/arm/kvm/init.S
>  delete mode 100644 arch/arm/kvm/interrupts.S
>  delete mode 100644 arch/arm/kvm/irq.h
>  delete mode 100644 arch/arm/kvm/reset.c
>  delete mode 100644 arch/arm/kvm/trace.h
>  delete mode 100644 arch/arm/kvm/vgic-v3-coproc.c
Christoffer Dall Feb. 19, 2020, 2:56 p.m. UTC | #10
On Mon, Feb 10, 2020 at 02:13:19PM +0000, Marc Zyngier wrote:
> KVM/arm was merged just over 7 years ago, and has lived a very quiet
> life so far. It mostly works if you're prepared to deal with its
> limitations, it has been a good prototype for the arm64 version,
> but it suffers a few problems:
> 
> - It is incomplete (no debug support, no PMU)
> - It hasn't followed any of the architectural evolutions
> - It has zero users (I don't count myself here)
> - It is more and more getting in the way of new arm64 developments
> 
> So here it is: unless someone screams and shows that they rely on
> KVM/arm to be maintained upsteam, I'll remove 32bit host support
> form the tree. One of the reasons that makes me confident nobody is
> using it is that I never receive *any* bug report. Yes, it is perfect.
> But if you depend on KVM/arm being available in mainline, please shout.
> 
> To reiterate: 32bit guest support for arm64 stays, of course. Only
> 32bit host goes. Once this is merged, I plan to move virt/kvm/arm to
> arm64, and cleanup all the now unnecessary abstractions.
> 
> The patches have been generated with the -D option to avoid spamming
> everyone with huge diffs, and there is a kvm-arm/goodbye branch in
> my kernel.org repository.

Acked-by: Christoffer Dall <christoffer.dall@arm.com>
Arnd Bergmann Feb. 19, 2020, 3:09 p.m. UTC | #11
On Mon, Feb 10, 2020 at 3:13 PM Marc Zyngier <maz@kernel.org> wrote:
>
> KVM/arm was merged just over 7 years ago, and has lived a very quiet
> life so far. It mostly works if you're prepared to deal with its
> limitations, it has been a good prototype for the arm64 version,
> but it suffers a few problems:
>
> - It is incomplete (no debug support, no PMU)
> - It hasn't followed any of the architectural evolutions
> - It has zero users (I don't count myself here)
> - It is more and more getting in the way of new arm64 developments
>
> So here it is: unless someone screams and shows that they rely on
> KVM/arm to be maintained upsteam, I'll remove 32bit host support
> form the tree. One of the reasons that makes me confident nobody is
> using it is that I never receive *any* bug report. Yes, it is perfect.
> But if you depend on KVM/arm being available in mainline, please shout.
>
> To reiterate: 32bit guest support for arm64 stays, of course. Only
> 32bit host goes. Once this is merged, I plan to move virt/kvm/arm to
> arm64, and cleanup all the now unnecessary abstractions.
>
> The patches have been generated with the -D option to avoid spamming
> everyone with huge diffs, and there is a kvm-arm/goodbye branch in
> my kernel.org repository.

Just one more thought before it's gone: is there any shared code
(header files?) that is used by the jailhouse hypervisor?

If there is, are there any plans to merge that into the mainline kernel
for arm32 in the near future?

I'm guessing the answer to at least one of those questions is 'no', so
we don't need to worry about it, but it seems better to ask.

      Arnd
Jan Kiszka Feb. 19, 2020, 3:46 p.m. UTC | #12
On 19.02.20 16:09, Arnd Bergmann wrote:
> On Mon, Feb 10, 2020 at 3:13 PM Marc Zyngier <maz@kernel.org> wrote:
>>
>> KVM/arm was merged just over 7 years ago, and has lived a very quiet
>> life so far. It mostly works if you're prepared to deal with its
>> limitations, it has been a good prototype for the arm64 version,
>> but it suffers a few problems:
>>
>> - It is incomplete (no debug support, no PMU)
>> - It hasn't followed any of the architectural evolutions
>> - It has zero users (I don't count myself here)
>> - It is more and more getting in the way of new arm64 developments
>>
>> So here it is: unless someone screams and shows that they rely on
>> KVM/arm to be maintained upsteam, I'll remove 32bit host support
>> form the tree. One of the reasons that makes me confident nobody is
>> using it is that I never receive *any* bug report. Yes, it is perfect.
>> But if you depend on KVM/arm being available in mainline, please shout.
>>
>> To reiterate: 32bit guest support for arm64 stays, of course. Only
>> 32bit host goes. Once this is merged, I plan to move virt/kvm/arm to
>> arm64, and cleanup all the now unnecessary abstractions.
>>
>> The patches have been generated with the -D option to avoid spamming
>> everyone with huge diffs, and there is a kvm-arm/goodbye branch in
>> my kernel.org repository.
> 
> Just one more thought before it's gone: is there any shared code
> (header files?) that is used by the jailhouse hypervisor?
> 
> If there is, are there any plans to merge that into the mainline kernel
> for arm32 in the near future?
> 
> I'm guessing the answer to at least one of those questions is 'no', so
> we don't need to worry about it, but it seems better to ask.

Good that you mention it: There is one thing we share on ARM (and 
ARM64), and that is the hypervisor enabling stub, to install our own 
vectors. If that was to be removed as well, we would have to patch it 
back downstream. So far, we only carry few EXPORT_SYMBOL patches for 
essential enabling.

That said, I was also starting to think about how long we will continue 
to support Jailhouse on 32-bit ARM. We currently have no supported SoC 
there that comes with an SMMU, and I doubt to see one still showing up. 
So, Jailhouse on ARM is really just a testing/demo case, maybe useful 
(but I didn't get concrete feedback) for cleaner collaborative AMP for 
real-time purposes, without security concerns. I assume 32-bit ARM will 
never be part of what would be proposed of Jailhouse for upstream.

Jan
Marc Zyngier Feb. 20, 2020, 10:29 a.m. UTC | #13
On 2020-02-19 15:46, Jan Kiszka wrote:
> On 19.02.20 16:09, Arnd Bergmann wrote:
>> On Mon, Feb 10, 2020 at 3:13 PM Marc Zyngier <maz@kernel.org> wrote:
>>> 
>>> KVM/arm was merged just over 7 years ago, and has lived a very quiet
>>> life so far. It mostly works if you're prepared to deal with its
>>> limitations, it has been a good prototype for the arm64 version,
>>> but it suffers a few problems:
>>> 
>>> - It is incomplete (no debug support, no PMU)
>>> - It hasn't followed any of the architectural evolutions
>>> - It has zero users (I don't count myself here)
>>> - It is more and more getting in the way of new arm64 developments
>>> 
>>> So here it is: unless someone screams and shows that they rely on
>>> KVM/arm to be maintained upsteam, I'll remove 32bit host support
>>> form the tree. One of the reasons that makes me confident nobody is
>>> using it is that I never receive *any* bug report. Yes, it is 
>>> perfect.
>>> But if you depend on KVM/arm being available in mainline, please 
>>> shout.
>>> 
>>> To reiterate: 32bit guest support for arm64 stays, of course. Only
>>> 32bit host goes. Once this is merged, I plan to move virt/kvm/arm to
>>> arm64, and cleanup all the now unnecessary abstractions.
>>> 
>>> The patches have been generated with the -D option to avoid spamming
>>> everyone with huge diffs, and there is a kvm-arm/goodbye branch in
>>> my kernel.org repository.
>> 
>> Just one more thought before it's gone: is there any shared code
>> (header files?) that is used by the jailhouse hypervisor?
>> 
>> If there is, are there any plans to merge that into the mainline 
>> kernel
>> for arm32 in the near future?
>> 
>> I'm guessing the answer to at least one of those questions is 'no', so
>> we don't need to worry about it, but it seems better to ask.
> 
> Good that you mention it: There is one thing we share on ARM (and
> ARM64), and that is the hypervisor enabling stub, to install our own
> vectors. If that was to be removed as well, we would have to patch it
> back downstream. So far, we only carry few EXPORT_SYMBOL patches for
> essential enabling.

I actually have a few extra patches on top of the series, one of them
actually removing the ability to register new vectors (mostly because
I don't like leaving unused stuff behind), see [1]. I'll post an update
so that we can discuss whether we want this particular to stay or not.

> That said, I was also starting to think about how long we will
> continue to support Jailhouse on 32-bit ARM. We currently have no
> supported SoC there that comes with an SMMU, and I doubt to see one
> still showing up. So, Jailhouse on ARM is really just a testing/demo
> case, maybe useful (but I didn't get concrete feedback) for cleaner
> collaborative AMP for real-time purposes, without security concerns. I
> assume 32-bit ARM will never be part of what would be proposed of
> Jailhouse for upstream.

I guess we all come to the same conclusion...

         M.

[1] 
https://git.kernel.org/pub/scm/linux/kernel/git/maz/arm-platforms.git/commit/?h=kvm-arm/goodbye&id=0943dd119105b65197adffda52c402cce28da56d
Marc Zyngier Feb. 20, 2020, 11:01 a.m. UTC | #14
On 2020-02-19 13:53, Stefan Agner wrote:
> On 2020-02-10 15:13, Marc Zyngier wrote:
>> KVM/arm was merged just over 7 years ago, and has lived a very quiet
>> life so far. It mostly works if you're prepared to deal with its
>> limitations, it has been a good prototype for the arm64 version,
>> but it suffers a few problems:
>> 
>> - It is incomplete (no debug support, no PMU)
>> - It hasn't followed any of the architectural evolutions
>> - It has zero users (I don't count myself here)
>> - It is more and more getting in the way of new arm64 developments
>> 
>> So here it is: unless someone screams and shows that they rely on
>> KVM/arm to be maintained upsteam, I'll remove 32bit host support
>> form the tree. One of the reasons that makes me confident nobody is
>> using it is that I never receive *any* bug report. Yes, it is perfect.
> 
> Not entirely true:
> https://lore.kernel.org/m/e2f7196ca6c70c55463a45b490f6731a@agner.ch

And I thank you for that. This bug was actually hitting both arm and
arm64, and triggered by a bogus DT (that KVM should have handled in a
nicer way). What I was trying to say is that nobody reports bugs that
are specific to 32bit KVM/arm.

> But, after that was fixed, it actually was perfect :-D
> https://blog.printk.io/2016/09/kvm-with-kvmtool-on-armv7/

Hey, neat! not sure how useful, but neat nonetheless... ;-)

> That said, I never used it in a real-world application, so from my side
> removing it is fine.

Thanks,

         M.
Marek Szyprowski Feb. 20, 2020, 12:44 p.m. UTC | #15
Hi Marc,

On 10.02.2020 15:13, Marc Zyngier wrote:
> KVM/arm was merged just over 7 years ago, and has lived a very quiet
> life so far. It mostly works if you're prepared to deal with its
> limitations, it has been a good prototype for the arm64 version,
> but it suffers a few problems:
>
> - It is incomplete (no debug support, no PMU)
> - It hasn't followed any of the architectural evolutions
> - It has zero users (I don't count myself here)
> - It is more and more getting in the way of new arm64 developments

That is a bit sad information. Mainline Exynos finally got everything 
that was needed to run it on the quite popular Samsung Exynos5422-based 
Odroid XU4/HC1/MC1 boards. According to the Odroid related forums it is 
being used. We also use it internally at Samsung.

> So here it is: unless someone screams and shows that they rely on
> KVM/arm to be maintained upsteam, I'll remove 32bit host support
> form the tree. One of the reasons that makes me confident nobody is
> using it is that I never receive *any* bug report. Yes, it is perfect.
> But if you depend on KVM/arm being available in mainline, please shout.
>
> To reiterate: 32bit guest support for arm64 stays, of course. Only
> 32bit host goes. Once this is merged, I plan to move virt/kvm/arm to
> arm64, and cleanup all the now unnecessary abstractions.
>
> The patches have been generated with the -D option to avoid spamming
> everyone with huge diffs, and there is a kvm-arm/goodbye branch in
> my kernel.org repository.
>
> Marc Zyngier (5):
>    arm: Unplug KVM from the build system
>    arm: Remove KVM from config files
>    arm: Remove 32bit KVM host support
>    arm: Remove HYP/Stage-2 page-table support
>    arm: Remove GICv3 vgic compatibility macros
>
>   Documentation/virt/kvm/arm/hyp-abi.txt |    5 +
>   arch/arm/Kconfig                       |    2 -
>   arch/arm/Makefile                      |    1 -
>   arch/arm/configs/axm55xx_defconfig     |    2 -
>   arch/arm/include/asm/arch_gicv3.h      |  114 --
>   arch/arm/include/asm/kvm_arm.h         |  239 ----
>   arch/arm/include/asm/kvm_asm.h         |   77 --
>   arch/arm/include/asm/kvm_coproc.h      |   36 -
>   arch/arm/include/asm/kvm_emulate.h     |  372 ------
>   arch/arm/include/asm/kvm_host.h        |  459 --------
>   arch/arm/include/asm/kvm_hyp.h         |  127 ---
>   arch/arm/include/asm/kvm_mmu.h         |  435 -------
>   arch/arm/include/asm/kvm_ras.h         |   14 -
>   arch/arm/include/asm/pgtable-3level.h  |   20 -
>   arch/arm/include/asm/pgtable.h         |    9 -
>   arch/arm/include/asm/sections.h        |    6 +-
>   arch/arm/include/asm/stage2_pgtable.h  |   75 --
>   arch/arm/include/asm/virt.h            |   12 -
>   arch/arm/include/uapi/asm/kvm.h        |  314 -----
>   arch/arm/kernel/asm-offsets.c          |   11 -
>   arch/arm/kernel/vmlinux-xip.lds.S      |    8 -
>   arch/arm/kernel/vmlinux.lds.S          |    8 -
>   arch/arm/kernel/vmlinux.lds.h          |   10 -
>   arch/arm/kvm/Kconfig                   |   59 -
>   arch/arm/kvm/Makefile                  |   43 -
>   arch/arm/kvm/coproc.c                  | 1455 ------------------------
>   arch/arm/kvm/coproc.h                  |  130 ---
>   arch/arm/kvm/coproc_a15.c              |   39 -
>   arch/arm/kvm/coproc_a7.c               |   42 -
>   arch/arm/kvm/emulate.c                 |  166 ---
>   arch/arm/kvm/guest.c                   |  387 -------
>   arch/arm/kvm/handle_exit.c             |  175 ---
>   arch/arm/kvm/hyp/Makefile              |   34 -
>   arch/arm/kvm/hyp/banked-sr.c           |   70 --
>   arch/arm/kvm/hyp/cp15-sr.c             |   72 --
>   arch/arm/kvm/hyp/entry.S               |  121 --
>   arch/arm/kvm/hyp/hyp-entry.S           |  295 -----
>   arch/arm/kvm/hyp/s2-setup.c            |   22 -
>   arch/arm/kvm/hyp/switch.c              |  242 ----
>   arch/arm/kvm/hyp/tlb.c                 |   68 --
>   arch/arm/kvm/hyp/vfp.S                 |   57 -
>   arch/arm/kvm/init.S                    |  157 ---
>   arch/arm/kvm/interrupts.S              |   36 -
>   arch/arm/kvm/irq.h                     |   16 -
>   arch/arm/kvm/reset.c                   |   86 --
>   arch/arm/kvm/trace.h                   |   86 --
>   arch/arm/kvm/vgic-v3-coproc.c          |   27 -
>   arch/arm/mach-exynos/Kconfig           |    2 +-
>   arch/arm/mm/mmu.c                      |   26 -
>   49 files changed, 7 insertions(+), 6262 deletions(-)
>   delete mode 100644 arch/arm/include/asm/kvm_arm.h
>   delete mode 100644 arch/arm/include/asm/kvm_asm.h
>   delete mode 100644 arch/arm/include/asm/kvm_coproc.h
>   delete mode 100644 arch/arm/include/asm/kvm_emulate.h
>   delete mode 100644 arch/arm/include/asm/kvm_host.h
>   delete mode 100644 arch/arm/include/asm/kvm_hyp.h
>   delete mode 100644 arch/arm/include/asm/kvm_mmu.h
>   delete mode 100644 arch/arm/include/asm/kvm_ras.h
>   delete mode 100644 arch/arm/include/asm/stage2_pgtable.h
>   delete mode 100644 arch/arm/include/uapi/asm/kvm.h
>   delete mode 100644 arch/arm/kvm/Kconfig
>   delete mode 100644 arch/arm/kvm/Makefile
>   delete mode 100644 arch/arm/kvm/coproc.c
>   delete mode 100644 arch/arm/kvm/coproc.h
>   delete mode 100644 arch/arm/kvm/coproc_a15.c
>   delete mode 100644 arch/arm/kvm/coproc_a7.c
>   delete mode 100644 arch/arm/kvm/emulate.c
>   delete mode 100644 arch/arm/kvm/guest.c
>   delete mode 100644 arch/arm/kvm/handle_exit.c
>   delete mode 100644 arch/arm/kvm/hyp/Makefile
>   delete mode 100644 arch/arm/kvm/hyp/banked-sr.c
>   delete mode 100644 arch/arm/kvm/hyp/cp15-sr.c
>   delete mode 100644 arch/arm/kvm/hyp/entry.S
>   delete mode 100644 arch/arm/kvm/hyp/hyp-entry.S
>   delete mode 100644 arch/arm/kvm/hyp/s2-setup.c
>   delete mode 100644 arch/arm/kvm/hyp/switch.c
>   delete mode 100644 arch/arm/kvm/hyp/tlb.c
>   delete mode 100644 arch/arm/kvm/hyp/vfp.S
>   delete mode 100644 arch/arm/kvm/init.S
>   delete mode 100644 arch/arm/kvm/interrupts.S
>   delete mode 100644 arch/arm/kvm/irq.h
>   delete mode 100644 arch/arm/kvm/reset.c
>   delete mode 100644 arch/arm/kvm/trace.h
>   delete mode 100644 arch/arm/kvm/vgic-v3-coproc.c
>
Best regards
Marc Zyngier Feb. 20, 2020, 1:15 p.m. UTC | #16
Hi Marek,

On 2020-02-20 12:44, Marek Szyprowski wrote:
> Hi Marc,
> 
> On 10.02.2020 15:13, Marc Zyngier wrote:
>> KVM/arm was merged just over 7 years ago, and has lived a very quiet
>> life so far. It mostly works if you're prepared to deal with its
>> limitations, it has been a good prototype for the arm64 version,
>> but it suffers a few problems:
>> 
>> - It is incomplete (no debug support, no PMU)
>> - It hasn't followed any of the architectural evolutions
>> - It has zero users (I don't count myself here)
>> - It is more and more getting in the way of new arm64 developments
> 
> That is a bit sad information. Mainline Exynos finally got everything
> that was needed to run it on the quite popular Samsung Exynos5422-based
> Odroid XU4/HC1/MC1 boards. According to the Odroid related forums it is
> being used. We also use it internally at Samsung.

Something like "too little, too late" springs to mind, but let's be
constructive. Is anyone using it in a production environment, where
they rely on the latest mainline kernel having KVM support?

The current proposal is to still have KVM support in 5.6, as well as
ongoing support for stable kernels. If that's not enough, can you please
explain your precise use case?

         M.
Paolo Bonzini Feb. 20, 2020, 1:17 p.m. UTC | #17
On 20/02/20 14:15, Marc Zyngier wrote:
>> That is a bit sad information. Mainline Exynos finally got everything
>> that was needed to run it on the quite popular Samsung Exynos5422-based
>> Odroid XU4/HC1/MC1 boards. According to the Odroid related forums it is
>> being used. We also use it internally at Samsung.
> 
> Something like "too little, too late" springs to mind, but let's be
> constructive. Is anyone using it in a production environment, where
> they rely on the latest mainline kernel having KVM support?

Depends if you consider "production environment" somebody playing at
home with a SBC.  But it's true that, these days, most of those that
support EL2 do support ARM64, even if they are used with a 32-bit userland.

Paolo
Robin Murphy Feb. 20, 2020, 1:32 p.m. UTC | #18
On 20/02/2020 1:15 pm, Marc Zyngier wrote:
> Hi Marek,
> 
> On 2020-02-20 12:44, Marek Szyprowski wrote:
>> Hi Marc,
>>
>> On 10.02.2020 15:13, Marc Zyngier wrote:
>>> KVM/arm was merged just over 7 years ago, and has lived a very quiet
>>> life so far. It mostly works if you're prepared to deal with its
>>> limitations, it has been a good prototype for the arm64 version,
>>> but it suffers a few problems:
>>>
>>> - It is incomplete (no debug support, no PMU)
>>> - It hasn't followed any of the architectural evolutions
>>> - It has zero users (I don't count myself here)
>>> - It is more and more getting in the way of new arm64 developments
>>
>> That is a bit sad information. Mainline Exynos finally got everything
>> that was needed to run it on the quite popular Samsung Exynos5422-based
>> Odroid XU4/HC1/MC1 boards. According to the Odroid related forums it is
>> being used. We also use it internally at Samsung.
> 
> Something like "too little, too late" springs to mind, but let's be
> constructive. Is anyone using it in a production environment, where
> they rely on the latest mainline kernel having KVM support?
> 
> The current proposal is to still have KVM support in 5.6, as well as
> ongoing support for stable kernels. If that's not enough, can you please
> explain your precise use case?

Presumably there's no *technical* reason why the stable subset of v7 
support couldn't be stripped down and brought back private to arch/arm 
if somebody really wants and is willing to step up and look after it?

Robin.
Marc Zyngier Feb. 20, 2020, 2:01 p.m. UTC | #19
On 2020-02-20 13:32, Robin Murphy wrote:
> On 20/02/2020 1:15 pm, Marc Zyngier wrote:
>> Hi Marek,
>> 
>> On 2020-02-20 12:44, Marek Szyprowski wrote:
>>> Hi Marc,
>>> 
>>> On 10.02.2020 15:13, Marc Zyngier wrote:
>>>> KVM/arm was merged just over 7 years ago, and has lived a very quiet
>>>> life so far. It mostly works if you're prepared to deal with its
>>>> limitations, it has been a good prototype for the arm64 version,
>>>> but it suffers a few problems:
>>>> 
>>>> - It is incomplete (no debug support, no PMU)
>>>> - It hasn't followed any of the architectural evolutions
>>>> - It has zero users (I don't count myself here)
>>>> - It is more and more getting in the way of new arm64 developments
>>> 
>>> That is a bit sad information. Mainline Exynos finally got everything
>>> that was needed to run it on the quite popular Samsung 
>>> Exynos5422-based
>>> Odroid XU4/HC1/MC1 boards. According to the Odroid related forums it 
>>> is
>>> being used. We also use it internally at Samsung.
>> 
>> Something like "too little, too late" springs to mind, but let's be
>> constructive. Is anyone using it in a production environment, where
>> they rely on the latest mainline kernel having KVM support?
>> 
>> The current proposal is to still have KVM support in 5.6, as well as
>> ongoing support for stable kernels. If that's not enough, can you 
>> please
>> explain your precise use case?
> 
> Presumably there's no *technical* reason why the stable subset of v7
> support couldn't be stripped down and brought back private to arch/arm
> if somebody really wants and is willing to step up and look after it?

There is no technical reason at all, just a maintenance effort.

The main killer is the whole MMU code, which I'm butchering with NV,
and that I suspect Will will also turn upside down with his stuff.
Not to mention the hypercall interface that will need a complete 
overhaul.

If we wanted to decouple the two, we'd need to make the MMU code, the
hypercalls, arm.c and a number of other bits private to 32bit.

         M.
Robin Murphy Feb. 20, 2020, 2:38 p.m. UTC | #20
On 20/02/2020 2:01 pm, Marc Zyngier wrote:
> On 2020-02-20 13:32, Robin Murphy wrote:
>> On 20/02/2020 1:15 pm, Marc Zyngier wrote:
>>> Hi Marek,
>>>
>>> On 2020-02-20 12:44, Marek Szyprowski wrote:
>>>> Hi Marc,
>>>>
>>>> On 10.02.2020 15:13, Marc Zyngier wrote:
>>>>> KVM/arm was merged just over 7 years ago, and has lived a very quiet
>>>>> life so far. It mostly works if you're prepared to deal with its
>>>>> limitations, it has been a good prototype for the arm64 version,
>>>>> but it suffers a few problems:
>>>>>
>>>>> - It is incomplete (no debug support, no PMU)
>>>>> - It hasn't followed any of the architectural evolutions
>>>>> - It has zero users (I don't count myself here)
>>>>> - It is more and more getting in the way of new arm64 developments
>>>>
>>>> That is a bit sad information. Mainline Exynos finally got everything
>>>> that was needed to run it on the quite popular Samsung Exynos5422-based
>>>> Odroid XU4/HC1/MC1 boards. According to the Odroid related forums it is
>>>> being used. We also use it internally at Samsung.
>>>
>>> Something like "too little, too late" springs to mind, but let's be
>>> constructive. Is anyone using it in a production environment, where
>>> they rely on the latest mainline kernel having KVM support?
>>>
>>> The current proposal is to still have KVM support in 5.6, as well as
>>> ongoing support for stable kernels. If that's not enough, can you please
>>> explain your precise use case?
>>
>> Presumably there's no *technical* reason why the stable subset of v7
>> support couldn't be stripped down and brought back private to arch/arm
>> if somebody really wants and is willing to step up and look after it?
> 
> There is no technical reason at all, just a maintenance effort.
> 
> The main killer is the whole MMU code, which I'm butchering with NV,
> and that I suspect Will will also turn upside down with his stuff.
> Not to mention the hypercall interface that will need a complete overhaul.
> 
> If we wanted to decouple the two, we'd need to make the MMU code, the
> hypercalls, arm.c and a number of other bits private to 32bit.

Right, the prospective kvm-arm maintainer's gameplan would essentially 
be an equivalent "move virt/kvm/arm to arch/arm/kvm" patch, but then 
ripping out all the Armv8 and GICv3 gubbins instead. Yes, there would 
then be lots of *similar* code to start with, but it would only diverge 
further as v8 architecture development continues independently.

Anyway, I just thought it seemed worth saying out loud, to reassure 
folks that a realistic middle-ground between "yay bye!" and "oh no the 
end of the world!" does exist, namely "someone else's problem" :)

Robin.
Takashi Yoshi Feb. 22, 2020, 2:21 p.m. UTC | #21
Hi

I found this mailing list thread and would like to express my opinion
and dependence on KVM/arm32.

I hope that I'm not too late already.


On Monday, 10.02.2020, 14:13 +0000 Marc Zyngier wrote:
> KVM/arm was merged just over 7 years ago, and has lived a very quiet
> life so far. It mostly works if you're prepared to deal with its
> limitations, it has been a good prototype for the arm64 version,
> but it suffers a few problems:
> 
> - It is incomplete (no debug support, no PMU)
> - It hasn't followed any of the architectural evolutions
> - It has zero users (I don't count myself here)

I might not be an important user, but I have been for multiple years
and still am a regular user of KVM/arm32 on different devices.

I use KVM on my Tegra K1 Chromebook for app development and have
multiple SBCs at home on which I run VMs on using KVM+libvirt.

Sure, neither of these devices has many resources available, but they
are working fine. I would love to keep them in service since I haven't
found arm64-based replacements that don't require hours upon hours of
tinkering to just get a basic OS installation running with a mainline
kernel.

As an example that they can still be of use in 2020 I'd like to point
out that one of the SBCs is running my DNS resolver, LDAP server,
RSS reader, IRC bouncer, and shared todo list just fine, each in their
separate VM.

> - It is more and more getting in the way of new arm64 developments
> 
> So here it is: unless someone screams and shows that they rely on
> KVM/arm to be maintained upsteam, I'll remove 32bit host support
> form the tree.

*scream*

> One of the reasons that makes me confident nobody is
> using it is that I never receive *any* bug report. Yes, it is
> perfect.

This assumption is deeply flawed. Most users (including me) are not
subscribed to this mailing list and will never find this thread at all.
I myself stumbled upon this discussion just by chance while I was
browsing the web trying to find something completely unrelated.

I've been using KVM on x86, ppc and arm for many years, yet I never
felt the need to report a bug on the mailing list.
(This is to be interpreted as a compliment to the great work the devs
of KVM have done!)

Just going by the number of bugs reported on a developers mailing list
is not going to paint an accurate picture.

I am convinced that I'm not the only one relying on KVM/arm32 in the
mainline kernel and would ask you to please reconsider keeping arm32 in
the mainline kernel for a few more years until adequate arm64
replacements are available on the market and have gained proper support
in the mainline kernel.

I myself unfortunately do neither have the knowledge nor resources to
help with development in KVM or maintaining a non-mainline kernel.

> But if you depend on KVM/arm being available in mainline, please
> shout.
> 
> To reiterate: 32bit guest support for arm64 stays, of course. Only
> 32bit host goes. Once this is merged, I plan to move virt/kvm/arm to
> arm64, and cleanup all the now unnecessary abstractions.
> 
> The patches have been generated with the -D option to avoid spamming
> everyone with huge diffs, and there is a kvm-arm/goodbye branch in
> my kernel.org repository.
> 
> [...]

Thanks for your understanding.

Kind regards

- Yoshi
Takashi Yoshi Feb. 22, 2020, 2:40 p.m. UTC | #22
Hi

I found this mailing list thread and would like to express my opinion
and dependence on KVM/arm32.

I hope that I'm not too late already.


On Monday, 10.02.2020, 14:13 +0000 Marc Zyngier wrote:
> KVM/arm was merged just over 7 years ago, and has lived a very quiet
> life so far. It mostly works if you're prepared to deal with its
> limitations, it has been a good prototype for the arm64 version,
> but it suffers a few problems:
> 
> - It is incomplete (no debug support, no PMU)
> - It hasn't followed any of the architectural evolutions
> - It has zero users (I don't count myself here)

I might not be an important user, but I have been for multiple years
and still am a regular user of KVM/arm32 on different devices.

I use KVM on my Tegra K1 Chromebook for app development and have
multiple SBCs at home on which I run VMs on using KVM+libvirt.

Sure, neither of these devices has many resources available, but they
are working fine. I would love to keep them in service since I haven't
found arm64-based replacements that don't require hours upon hours of
tinkering to just get a basic OS installation running with a mainline
kernel.

As an example that they can still be of use in 2020 I'd like to point
out that one of the SBCs is running my DNS resolver, LDAP server,
RSS reader, IRC bouncer, and shared todo list just fine, each in their
separate VM.

> - It is more and more getting in the way of new arm64 developments
> 
> So here it is: unless someone screams and shows that they rely on
> KVM/arm to be maintained upsteam, I'll remove 32bit host support
> form the tree.

*scream*

> One of the reasons that makes me confident nobody is
> using it is that I never receive *any* bug report. Yes, it is
> perfect.

This assumption is deeply flawed. Most users (including me) are not
subscribed to this mailing list and will never find this thread at all.
I myself stumbled upon this discussion just by chance while I was
browsing the web trying to find something completely unrelated.

I've been using KVM on x86, ppc and arm for many years, yet I never
felt the need to report a bug on the mailing list.
(This is to be interpreted as a compliment to the great work the devs
of KVM have done!)

Just going by the number of bugs reported on a developers mailing list
is not going to paint an accurate picture.

I am convinced that I'm not the only one relying on KVM/arm32 in the
mainline kernel and would ask you to please reconsider keeping arm32 in
the mainline kernel for a few more years until adequate arm64
replacements are available on the market and have gained proper support
in the mainline kernel.

I myself unfortunately do neither have the knowledge nor resources to
help with development in KVM or maintaining a non-mainline kernel.

> But if you depend on KVM/arm being available in mainline, please
> shout.
> 
> To reiterate: 32bit guest support for arm64 stays, of course. Only
> 32bit host goes. Once this is merged, I plan to move virt/kvm/arm to
> arm64, and cleanup all the now unnecessary abstractions.
> 
> The patches have been generated with the -D option to avoid spamming
> everyone with huge diffs, and there is a kvm-arm/goodbye branch in
> my kernel.org repository.
> 
> [...]

Thanks for your understanding.

Kind regards

- Yoshi
Arnd Bergmann Feb. 22, 2020, 9:31 p.m. UTC | #23
On Sat, Feb 22, 2020 at 3:40 PM Takashi Yoshi <takashi@yoshi.email> wrote:
> On Monday, 10.02.2020, 14:13 +0000 Marc Zyngier wrote:
> > KVM/arm was merged just over 7 years ago, and has lived a very quiet
> > life so far. It mostly works if you're prepared to deal with its
> > limitations, it has been a good prototype for the arm64 version,
> > but it suffers a few problems:
> >
> > - It is incomplete (no debug support, no PMU)
> > - It hasn't followed any of the architectural evolutions
> > - It has zero users (I don't count myself here)
>
> I might not be an important user, but I have been for multiple years
> and still am a regular user of KVM/arm32 on different devices.
>
> I use KVM on my Tegra K1 Chromebook for app development and have
> multiple SBCs at home on which I run VMs on using KVM+libvirt.
>
> Sure, neither of these devices has many resources available, but they
> are working fine. I would love to keep them in service since I haven't
> found arm64-based replacements that don't require hours upon hours of
> tinkering to just get a basic OS installation running with a mainline
> kernel.
>
> As an example that they can still be of use in 2020 I'd like to point
> out that one of the SBCs is running my DNS resolver, LDAP server,
> RSS reader, IRC bouncer, and shared todo list just fine, each in their
> separate VM.

Thank you for providing an important data point to this question.

> > - It is more and more getting in the way of new arm64 developments
> >
> > So here it is: unless someone screams and shows that they rely on
> > KVM/arm to be maintained upsteam, I'll remove 32bit host support
> > form the tree.
>
> *scream*
>
> > One of the reasons that makes me confident nobody is
> > using it is that I never receive *any* bug report. Yes, it is
> > perfect.
>
> This assumption is deeply flawed. Most users (including me) are not
> subscribed to this mailing list and will never find this thread at all.
> I myself stumbled upon this discussion just by chance while I was
> browsing the web trying to find something completely unrelated.
>
> I've been using KVM on x86, ppc and arm for many years, yet I never
> felt the need to report a bug on the mailing list.
> (This is to be interpreted as a compliment to the great work the devs
> of KVM have done!)
>
> Just going by the number of bugs reported on a developers mailing list
> is not going to paint an accurate picture.
>
> I am convinced that I'm not the only one relying on KVM/arm32 in the
> mainline kernel and would ask you to please reconsider keeping arm32 in
> the mainline kernel for a few more years until adequate arm64
> replacements are available on the market and have gained proper support
> in the mainline kernel.

Can you provide some more information about how you use KVM on 32-bit
machines, to make it possible to better estimate how many others might
do the same, and how long you will need to upgrade to newer kernels for?

In particular:

- What is the smallest amount of physical RAM that you have to found to
   make a usable ARM/KVM host? Note that the 4GB configuration of the
   Tegra K1 (an rk3288) Chromebooks seems to be extremely rare in other
   devices, while most new 32-bit SBCs come with 1GB or less these days.

- How often do you update the host kernels on those 32-bit machines that
  you still use to newer releases? What is the oldest/newest you run at the
  moment?

- Are you able to move the host installation to a distribution with a long-term
  stable release cycle such as Debian, Ubuntu that gives you a ~five year
  support after a kernel release?

         Arnd
Takashi Yoshi Feb. 25, 2020, 9:34 p.m. UTC | #24
Dear Arnd

Please excuse my late response.

On Sat, 22 Feb 2020 22:31:40 +0100 Arnd Bergmann <arnd@arndb.de> wrote:

> On Sat, Feb 22, 2020 at 3:40 PM Takashi Yoshi <takashi@yoshi.email>
> wrote:
> > On Monday, 10.02.2020, 14:13 +0000 Marc Zyngier wrote:
> > > KVM/arm was merged just over 7 years ago, and has lived a very
> > > quiet life so far. It mostly works if you're prepared to deal
> > > with its limitations, it has been a good prototype for the arm64
> > > version, but it suffers a few problems:
> > >
> > > - It is incomplete (no debug support, no PMU)
> > > - It hasn't followed any of the architectural evolutions
> > > - It has zero users (I don't count myself here)
> >
> > I might not be an important user, but I have been for multiple years
> > and still am a regular user of KVM/arm32 on different devices.
> >
> > I use KVM on my Tegra K1 Chromebook for app development and have
> > multiple SBCs at home on which I run VMs on using KVM+libvirt.
> >
> > Sure, neither of these devices has many resources available, but
> > they are working fine. I would love to keep them in service since I
> > haven't found arm64-based replacements that don't require hours
> > upon hours of tinkering to just get a basic OS installation running
> > with a mainline kernel.
> >
> > As an example that they can still be of use in 2020 I'd like to
> > point out that one of the SBCs is running my DNS resolver, LDAP
> > server, RSS reader, IRC bouncer, and shared todo list just fine,
> > each in their separate VM.
> 
> Thank you for providing an important data point to this question.
> 
> > > - It is more and more getting in the way of new arm64 developments
> > >
> > > So here it is: unless someone screams and shows that they rely on
> > > KVM/arm to be maintained upsteam, I'll remove 32bit host support
> > > form the tree.
> >
> > *scream*
> >
> > > One of the reasons that makes me confident nobody is
> > > using it is that I never receive *any* bug report. Yes, it is
> > > perfect.
> >
> > This assumption is deeply flawed. Most users (including me) are not
> > subscribed to this mailing list and will never find this thread at
> > all. I myself stumbled upon this discussion just by chance while I
> > was browsing the web trying to find something completely unrelated.
> >
> > I've been using KVM on x86, ppc and arm for many years, yet I never
> > felt the need to report a bug on the mailing list.
> > (This is to be interpreted as a compliment to the great work the
> > devs of KVM have done!)
> >
> > Just going by the number of bugs reported on a developers mailing
> > list is not going to paint an accurate picture.
> >
> > I am convinced that I'm not the only one relying on KVM/arm32 in the
> > mainline kernel and would ask you to please reconsider keeping
> > arm32 in the mainline kernel for a few more years until adequate
> > arm64 replacements are available on the market and have gained
> > proper support in the mainline kernel.
> 
> Can you provide some more information about how you use KVM on 32-bit
> machines, to make it possible to better estimate how many others might
> do the same,

Sure.
First of all I own different ARM boards. Currently I virtualise on
Banana Pi M1 (1GB), cubox-i (2GB) and my Acer Chromebook (4GB).

The Chromebook is my travel laptop on which I have three VMs on (LAMP,
PostgreSQL, kernel testing) which I primarily use to develop against.

The others are "home servers", they run all kinds of things for my home
(incl. DNS, LDAP, RSS-Reader, Wiki, Music-Database, RDBMS,
Task-Management).

> and how long you will need to upgrade to newer kernels for?

I don't really have a strict policy regarding when to upgrade kernels.
I just run whatever gets patched and works.

Sometimes this is the latest stable release, most of the time this is
the last longterm release.

> In particular:
> 
> - What is the smallest amount of physical RAM that you have to found
> to make a usable ARM/KVM host? 

Not sure if I can answer this question adequately as the smallest of my
ARM32 boards have at least 1GB of RAM, which works for sure.

Since you're asking about the smallest amount I did some experiments.
I spun up the testing VM on my Chromebook. It consists of a basic
Gentoo userland currently running on a "reduced" 4.19 kernel (I'm sure
it could be stripped further if one was determined enough).

When I boot it up and log in the qemu-system process on the host uses
100MB. The memory usage (incl. cache) inside the VM is only 50 MB,
though.

Adding a few MB for the actual application one would want to run to
these 100MB, I calculate with 160MB for a "lightweight" VM.

This would mean that one could run 2-3 such VMs in just 512MB which I
would count as "usable".

If you were conservative with memory and used a lightweight distro, like
Alpine, OpenWrt or built your own using Buildroot, I could imagine that
you can make a nice little home router with virtualized DHCP server,
DNS server (for home network) + resolver, TFTP and possibly VPN in
512MB.
(Sounds like a cool experiment for the next time im bored :P)


A very different use case could be to host unikernels using e.g.
Solo5 (https://github.com/Solo5/solo5) whose hvt backend also uses KVM.
I, unfortunately, don't have any experience with unikernels, but I
assume that they consume a lot less memory than a full Linux VM.


> Note that the 4GB configuration of the Tegra K1 (an rk3288)
> Chromebooks seems to be extremely rare in other devices, while most
> new 32-bit SBCs come with 1GB or less these days.

I agree that 4GB seems to be really rare in 32-bit land outside of
laptops like Chromebooks or Novena, but I don't agree that most 32-bit
SBCs are so low specced.

There are quite a few 2GB boards out there:
ODROID U2/U3/X2/XU/XU3/XU4/HC1/HC2, Cubieboard 3/4/5, HummingBoard,
CuBox i4/Pulse, ASUS TinkerBoard (S), Lenovo G66 TV-Box, Radxa ROCK PRO
/ Rock2, Nvidia Jetson TK1, Banana Pi M2U/M3, Firefly RK3288,
BeagleBoard X15, OrangePi Plus 2E, Utilite, Wandboard, ClearFog,
NetGate SC3100, and probably a lot more can be had with 2GB RAM.

(If I were to buy an SBC for virtualization, I'd get one of these :-))

Also lets not forget all the powerful smartphones out there which could
be used as virtualizers using postmarketos.org.

> - How often do you update the host kernels on those 32-bit machines
> that you still use to newer releases? 

I usually track the latest longterm branch (but I wait a bit before
jumping to a new longterm). So at the moment the majority is running
4.19.x, but I'm in the process of upgrading to 5.4.x now that most of
the annoying bugs should be fixed ;-)

> What is the oldest/newest you run at the moment?

Oldest: 4.15.18 (because of an annoying regression that's likely never
going to get fixed)
Newest: 5.4.20

> - Are you able to move the host installation to a distribution with a
> long-term stable release cycle such as Debian, Ubuntu that gives you
> a ~five year support after a kernel release?

I could do that, but I would like to avoid being dependent on an old
kernel as from personal experience even on so-called "longterm"
releases regressions do slip in and tend not to get fixed if they're
not too severe, especially in old distro kernels.

Kind regards,

- Yoshi