| Message ID | 20241008105455.2302628-1-david@redhat.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | s390x: virtio-mem support | expand |
This series has been successfully tested in s390x. Booted up a VM including:
/home/qemu/build/qemu-system-s390x \
...
-m 4G,maxmem=20G \
-object memory-backend-ram,id=mem0,size=16G,reserve=off \
-device virtio-mem-ccw,id=vmem0,memdev=mem0,dynamic-memslots=on \
...
Check the memory devices
(qemu) info memory-devices
Memory device [virtio-mem]: "vmem0"
memaddr: 0x100000000
node: 0
requested-size: 0
size: 0
max-size: 17179869184
block-size: 1048576
memdev: /objects/mem0
Hotplug memory to the maximum
(qemu) qom-set vmem0 requested-size 16G
(qemu) info memory-devices
Memory device [virtio-mem]: "vmem0"
memaddr: 0x100000000
node: 0
requested-size: 17179869184
size: 17179869184
max-size: 17179869184
block-size: 1048576
memdev: /objects/mem0
Try to unplug the device, check the error message is correct
(qemu) device_del vmem0
Error: virtio-mem device cannot get unplugged while some of its memory
is still plugged
Reduce the memory till 0 and device is successfully unplugged
(qemu) qom-set vmem0 requested-size 0
(qemu) device_del vmem0
(qemu) object_del mem0
Hotplug again the device
(qemu) object_add memory-backend-ram,id=mem0,size=8G,reserve=off
(qemu) device_add
virtio-mem-ccw,id=vmem0,memdev=mem0,dynamic-memslots=on,requested-size=1G
(qemu) info memory-devices
Memory device [virtio-mem]: "vmem0"
memaddr: 0x100000000
node: 0
requested-size: 1073741824
size: 1073741824
max-size: 8589934592
block-size: 1048576
memdev: /objects/mem0
Finally, add some extra memory and check the memslots
(qemu) qom-set vmem0 requested-size 4G
(qemu) info mtree
...
0000000100000000-00000002ffffffff (prio 0, i/o): virtio-mem
0000000100000000-000000013fffffff (prio 0, ram): alias memslot-0
@mem0 0000000000000000-000000003fffffff
0000000140000000-000000017fffffff (prio 0, ram): alias memslot-1
@mem0 0000000040000000-000000007fffffff
0000000180000000-00000001bfffffff (prio 0, ram): alias memslot-2
@mem0 0000000080000000-00000000bfffffff
00000001c0000000-00000001ffffffff (prio 0, ram): alias memslot-3
@mem0 00000000c0000000-00000000ffffffff
Tested-by: Mario Casquero <mcasquer@redhat.com>
On Tue, Oct 8, 2024 at 12:57 PM David Hildenbrand <david@redhat.com> wrote:
>
> Based on current master.
>
> There is really not much left to do on s390x, because virtio-mem already
> implements most things we need today (e.g., early-migration,
> unplugged-inaccessible). The biggest part of this series is just doing what
> we do with virtio-pci, wiring it up in the machine hotplug handler and ...
> well, messing with the physical memory layout where we can now exceed
> initial RAM size and have sparsity (memory holes).
>
> I tested a lot of things, including:
> * Memory hotplug/unplug
> * Device hotplug/unplug
> * System resets / reboots
> * Migrate to/from file (including storage attributes under KVM)
> * Basic live migration
> * Basic postcopy live migration
>
> More details on how to use it on s390x -- which is pretty much how
> we use it on other architectures, except
> s/virtio-mem-pci/virtio-mem-ccw/ --- is in the last patch.
>
> This series introduces a new diag(500) "STORAGE LIMIT" subcode that will
> be documented in the kernel and at [2] once this+kernel part go upstream.
>
> There are not many s390x-specific virtio-mem future work items, except:
> * Storage attribute migration might be improved
> * We might want to reset storage attributes of unplugged memory
> (might or might not be required for upcoming page table reclaim in
> Linux; TBD)
>
> The Linux driver is available at [3].
>
> [1] https://lkml.kernel.org/r/20240906101658.514470-1-pbonzini@redhat.com
> [2] https://gitlab.com/davidhildenbrand/s390x-os-virt-spec
> [3] https://lkml.kernel.org/r/20240910191541.2179655-6-david@redhat.com
>
> v1 -> v2:
> * "s390x/s390-virtio-hcall: prepare for more diag500 hypercalls"
> - Turn handle_diag_500() into a void function
> - Inject PGM_SPECIFICATION from handle_diag_500() directly
> * "s390x/s390-virtio-ccw: move setting the maximum guest size from sclp to
> machine code"
> - Move code to a new function to make further changes easier
> - Adjust s390_pv_vm_try_disable_async() to stay in sync with the
> maxram->ram change
> * "s390x: introduce s390_get_memory_limit()"
> - Store limit in machine
> - Move s390_set_memory_limit() from target code into machine code
> * "s390x/s390-hypercall: introduce DIAG500 STORAGE_LIMIT"
> - Move handling into a separate function now that we lookup the machine
> * "s390x/s390-stattrib-kvm: prepare for memory devices and sparse memory
> layouts"
> - Adjust to s390_get_memory_limit() changes
> * "s390x/s390-skeys: prepare for memory devices"
> - Adjust to s390_get_memory_limit() changes
> * "s390x/pv: prepare for memory devices"
> - Use s390_get_memory_limit()
> * "s390x: remember the maximum page size"
> - Store it in the machine
> - Move s390_set_max_pagesize() from target code into machine code
> - No need for s390_get_max_pagesize()
> * "s390x/virtio-ccw: add support for virtio based memory devices"
> - Move machine wire-up code from virtio-mem patch into this patch
> - Add stubs to make compilation without virtio-mem work
> * "s390x: virtio-mem support"
> - Move machine write-up code to previous patch
>
> Cc: Paolo Bonzini <pbonzini@redhat.com>
> Cc: Thomas Huth <thuth@redhat.com>
> Cc: Halil Pasic <pasic@linux.ibm.com>
> Cc: Christian Borntraeger <borntraeger@linux.ibm.com>
> Cc: Eric Farman <farman@linux.ibm.com>
> Cc: Richard Henderson <richard.henderson@linaro.org>
> Cc: Nina Schoetterl-Glausch <nsg@linux.ibm.com>
> Cc: Heiko Carstens <hca@linux.ibm.com>
> Cc: Ilya Leoshkevich <iii@linux.ibm.com>
> Cc: Janosch Frank <frankja@linux.ibm.com>
> Cc: "Michael S. Tsirkin" <mst@redhat.com>
> Cc: Cornelia Huck <cohuck@redhat.com>
>
> David Hildenbrand (14):
> s390x/s390-virtio-ccw: don't crash on weird RAM sizes
> s390x/s390-virtio-hcall: remove hypercall registration mechanism
> s390x/s390-virtio-hcall: prepare for more diag500 hypercalls
> s390x: rename s390-virtio-hcall* to s390-hypercall*
> s390x/s390-virtio-ccw: move setting the maximum guest size from sclp
> to machine code
> s390x: introduce s390_get_memory_limit()
> s390x/s390-hypercall: introduce DIAG500 STORAGE_LIMIT
> s390x/s390-stattrib-kvm: prepare for memory devices and sparse memory
> layouts
> s390x/s390-skeys: prepare for memory devices
> s390x/s390-virtio-ccw: prepare for memory devices
> s390x/pv: prepare for memory devices
> s390x: remember the maximum page size
> s390x/virtio-ccw: add support for virtio based memory devices
> s390x: virtio-mem support
>
> MAINTAINERS | 5 +
> hw/s390x/Kconfig | 1 +
> hw/s390x/meson.build | 6 +-
> hw/s390x/s390-hypercall.c | 85 +++++++++++
> hw/s390x/s390-hypercall.h | 25 ++++
> hw/s390x/s390-skeys.c | 6 +-
> hw/s390x/s390-stattrib-kvm.c | 67 ++++++---
> hw/s390x/s390-virtio-ccw.c | 165 ++++++++++++++-------
> hw/s390x/s390-virtio-hcall.c | 41 ------
> hw/s390x/s390-virtio-hcall.h | 25 ----
> hw/s390x/sclp.c | 17 +--
> hw/s390x/virtio-ccw-md-stubs.c | 24 +++
> hw/s390x/virtio-ccw-md.c | 153 +++++++++++++++++++
> hw/s390x/virtio-ccw-md.h | 44 ++++++
> hw/s390x/virtio-ccw-mem.c | 226 +++++++++++++++++++++++++++++
> hw/s390x/virtio-ccw-mem.h | 34 +++++
> hw/virtio/Kconfig | 1 +
> hw/virtio/virtio-mem.c | 4 +-
> include/hw/s390x/s390-virtio-ccw.h | 4 +
> target/s390x/cpu-sysemu.c | 15 --
> target/s390x/cpu.h | 2 -
> target/s390x/kvm/kvm.c | 18 +--
> target/s390x/kvm/pv.c | 2 +-
> target/s390x/tcg/misc_helper.c | 7 +-
> 24 files changed, 782 insertions(+), 195 deletions(-)
> create mode 100644 hw/s390x/s390-hypercall.c
> create mode 100644 hw/s390x/s390-hypercall.h
> delete mode 100644 hw/s390x/s390-virtio-hcall.c
> delete mode 100644 hw/s390x/s390-virtio-hcall.h
> create mode 100644 hw/s390x/virtio-ccw-md-stubs.c
> create mode 100644 hw/s390x/virtio-ccw-md.c
> create mode 100644 hw/s390x/virtio-ccw-md.h
> create mode 100644 hw/s390x/virtio-ccw-mem.c
> create mode 100644 hw/s390x/virtio-ccw-mem.h
>
> --
> 2.46.1
>
>
Based on current master. There is really not much left to do on s390x, because virtio-mem already implements most things we need today (e.g., early-migration, unplugged-inaccessible). The biggest part of this series is just doing what we do with virtio-pci, wiring it up in the machine hotplug handler and ... well, messing with the physical memory layout where we can now exceed initial RAM size and have sparsity (memory holes). I tested a lot of things, including: * Memory hotplug/unplug * Device hotplug/unplug * System resets / reboots * Migrate to/from file (including storage attributes under KVM) * Basic live migration * Basic postcopy live migration More details on how to use it on s390x -- which is pretty much how we use it on other architectures, except s/virtio-mem-pci/virtio-mem-ccw/ --- is in the last patch. This series introduces a new diag(500) "STORAGE LIMIT" subcode that will be documented in the kernel and at [2] once this+kernel part go upstream. There are not many s390x-specific virtio-mem future work items, except: * Storage attribute migration might be improved * We might want to reset storage attributes of unplugged memory (might or might not be required for upcoming page table reclaim in Linux; TBD) The Linux driver is available at [3]. [1] https://lkml.kernel.org/r/20240906101658.514470-1-pbonzini@redhat.com [2] https://gitlab.com/davidhildenbrand/s390x-os-virt-spec [3] https://lkml.kernel.org/r/20240910191541.2179655-6-david@redhat.com v1 -> v2: * "s390x/s390-virtio-hcall: prepare for more diag500 hypercalls" - Turn handle_diag_500() into a void function - Inject PGM_SPECIFICATION from handle_diag_500() directly * "s390x/s390-virtio-ccw: move setting the maximum guest size from sclp to machine code" - Move code to a new function to make further changes easier - Adjust s390_pv_vm_try_disable_async() to stay in sync with the maxram->ram change * "s390x: introduce s390_get_memory_limit()" - Store limit in machine - Move s390_set_memory_limit() from target code into machine code * "s390x/s390-hypercall: introduce DIAG500 STORAGE_LIMIT" - Move handling into a separate function now that we lookup the machine * "s390x/s390-stattrib-kvm: prepare for memory devices and sparse memory layouts" - Adjust to s390_get_memory_limit() changes * "s390x/s390-skeys: prepare for memory devices" - Adjust to s390_get_memory_limit() changes * "s390x/pv: prepare for memory devices" - Use s390_get_memory_limit() * "s390x: remember the maximum page size" - Store it in the machine - Move s390_set_max_pagesize() from target code into machine code - No need for s390_get_max_pagesize() * "s390x/virtio-ccw: add support for virtio based memory devices" - Move machine wire-up code from virtio-mem patch into this patch - Add stubs to make compilation without virtio-mem work * "s390x: virtio-mem support" - Move machine write-up code to previous patch Cc: Paolo Bonzini <pbonzini@redhat.com> Cc: Thomas Huth <thuth@redhat.com> Cc: Halil Pasic <pasic@linux.ibm.com> Cc: Christian Borntraeger <borntraeger@linux.ibm.com> Cc: Eric Farman <farman@linux.ibm.com> Cc: Richard Henderson <richard.henderson@linaro.org> Cc: Nina Schoetterl-Glausch <nsg@linux.ibm.com> Cc: Heiko Carstens <hca@linux.ibm.com> Cc: Ilya Leoshkevich <iii@linux.ibm.com> Cc: Janosch Frank <frankja@linux.ibm.com> Cc: "Michael S. Tsirkin" <mst@redhat.com> Cc: Cornelia Huck <cohuck@redhat.com> David Hildenbrand (14): s390x/s390-virtio-ccw: don't crash on weird RAM sizes s390x/s390-virtio-hcall: remove hypercall registration mechanism s390x/s390-virtio-hcall: prepare for more diag500 hypercalls s390x: rename s390-virtio-hcall* to s390-hypercall* s390x/s390-virtio-ccw: move setting the maximum guest size from sclp to machine code s390x: introduce s390_get_memory_limit() s390x/s390-hypercall: introduce DIAG500 STORAGE_LIMIT s390x/s390-stattrib-kvm: prepare for memory devices and sparse memory layouts s390x/s390-skeys: prepare for memory devices s390x/s390-virtio-ccw: prepare for memory devices s390x/pv: prepare for memory devices s390x: remember the maximum page size s390x/virtio-ccw: add support for virtio based memory devices s390x: virtio-mem support MAINTAINERS | 5 + hw/s390x/Kconfig | 1 + hw/s390x/meson.build | 6 +- hw/s390x/s390-hypercall.c | 85 +++++++++++ hw/s390x/s390-hypercall.h | 25 ++++ hw/s390x/s390-skeys.c | 6 +- hw/s390x/s390-stattrib-kvm.c | 67 ++++++--- hw/s390x/s390-virtio-ccw.c | 165 ++++++++++++++------- hw/s390x/s390-virtio-hcall.c | 41 ------ hw/s390x/s390-virtio-hcall.h | 25 ---- hw/s390x/sclp.c | 17 +-- hw/s390x/virtio-ccw-md-stubs.c | 24 +++ hw/s390x/virtio-ccw-md.c | 153 +++++++++++++++++++ hw/s390x/virtio-ccw-md.h | 44 ++++++ hw/s390x/virtio-ccw-mem.c | 226 +++++++++++++++++++++++++++++ hw/s390x/virtio-ccw-mem.h | 34 +++++ hw/virtio/Kconfig | 1 + hw/virtio/virtio-mem.c | 4 +- include/hw/s390x/s390-virtio-ccw.h | 4 + target/s390x/cpu-sysemu.c | 15 -- target/s390x/cpu.h | 2 - target/s390x/kvm/kvm.c | 18 +-- target/s390x/kvm/pv.c | 2 +- target/s390x/tcg/misc_helper.c | 7 +- 24 files changed, 782 insertions(+), 195 deletions(-) create mode 100644 hw/s390x/s390-hypercall.c create mode 100644 hw/s390x/s390-hypercall.h delete mode 100644 hw/s390x/s390-virtio-hcall.c delete mode 100644 hw/s390x/s390-virtio-hcall.h create mode 100644 hw/s390x/virtio-ccw-md-stubs.c create mode 100644 hw/s390x/virtio-ccw-md.c create mode 100644 hw/s390x/virtio-ccw-md.h create mode 100644 hw/s390x/virtio-ccw-mem.c create mode 100644 hw/s390x/virtio-ccw-mem.h