| Message ID | 20241008105455.2302628-1-david@redhat.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | s390x: virtio-mem support | expand |
This series has been successfully tested in s390x. Booted up a VM including:
/home/qemu/build/qemu-system-s390x \
...
-m 4G,maxmem=20G \
-object memory-backend-ram,id=mem0,size=16G,reserve=off \
-device virtio-mem-ccw,id=vmem0,memdev=mem0,dynamic-memslots=on \
...
Check the memory devices
(qemu) info memory-devices
Memory device [virtio-mem]: "vmem0"
memaddr: 0x100000000
node: 0
requested-size: 0
size: 0
max-size: 17179869184
block-size: 1048576
memdev: /objects/mem0
Hotplug memory to the maximum
(qemu) qom-set vmem0 requested-size 16G
(qemu) info memory-devices
Memory device [virtio-mem]: "vmem0"
memaddr: 0x100000000
node: 0
requested-size: 17179869184
size: 17179869184
max-size: 17179869184
block-size: 1048576
memdev: /objects/mem0
Try to unplug the device, check the error message is correct
(qemu) device_del vmem0
Error: virtio-mem device cannot get unplugged while some of its memory
is still plugged
Reduce the memory till 0 and device is successfully unplugged
(qemu) qom-set vmem0 requested-size 0
(qemu) device_del vmem0
(qemu) object_del mem0
Hotplug again the device
(qemu) object_add memory-backend-ram,id=mem0,size=8G,reserve=off
(qemu) device_add
virtio-mem-ccw,id=vmem0,memdev=mem0,dynamic-memslots=on,requested-size=1G
(qemu) info memory-devices
Memory device [virtio-mem]: "vmem0"
memaddr: 0x100000000
node: 0
requested-size: 1073741824
size: 1073741824
max-size: 8589934592
block-size: 1048576
memdev: /objects/mem0
Finally, add some extra memory and check the memslots
(qemu) qom-set vmem0 requested-size 4G
(qemu) info mtree
...
0000000100000000-00000002ffffffff (prio 0, i/o): virtio-mem
0000000100000000-000000013fffffff (prio 0, ram): alias memslot-0
@mem0 0000000000000000-000000003fffffff
0000000140000000-000000017fffffff (prio 0, ram): alias memslot-1
@mem0 0000000040000000-000000007fffffff
0000000180000000-00000001bfffffff (prio 0, ram): alias memslot-2
@mem0 0000000080000000-00000000bfffffff
00000001c0000000-00000001ffffffff (prio 0, ram): alias memslot-3
@mem0 00000000c0000000-00000000ffffffff
Tested-by: Mario Casquero <mcasquer@redhat.com>
On Tue, Oct 8, 2024 at 12:57 PM David Hildenbrand <david@redhat.com> wrote:
>
> Based on current master.
>
> There is really not much left to do on s390x, because virtio-mem already
> implements most things we need today (e.g., early-migration,
> unplugged-inaccessible). The biggest part of this series is just doing what
> we do with virtio-pci, wiring it up in the machine hotplug handler and ...
> well, messing with the physical memory layout where we can now exceed
> initial RAM size and have sparsity (memory holes).
>
> I tested a lot of things, including:
> * Memory hotplug/unplug
> * Device hotplug/unplug
> * System resets / reboots
> * Migrate to/from file (including storage attributes under KVM)
> * Basic live migration
> * Basic postcopy live migration
>
> More details on how to use it on s390x -- which is pretty much how
> we use it on other architectures, except
> s/virtio-mem-pci/virtio-mem-ccw/ --- is in the last patch.
>
> This series introduces a new diag(500) "STORAGE LIMIT" subcode that will
> be documented in the kernel and at [2] once this+kernel part go upstream.
>
> There are not many s390x-specific virtio-mem future work items, except:
> * Storage attribute migration might be improved
> * We might want to reset storage attributes of unplugged memory
> (might or might not be required for upcoming page table reclaim in
> Linux; TBD)
>
> The Linux driver is available at [3].
>
> [1] https://lkml.kernel.org/r/20240906101658.514470-1-pbonzini@redhat.com
> [2] https://gitlab.com/davidhildenbrand/s390x-os-virt-spec
> [3] https://lkml.kernel.org/r/20240910191541.2179655-6-david@redhat.com
>
> v1 -> v2:
> * "s390x/s390-virtio-hcall: prepare for more diag500 hypercalls"
> - Turn handle_diag_500() into a void function
> - Inject PGM_SPECIFICATION from handle_diag_500() directly
> * "s390x/s390-virtio-ccw: move setting the maximum guest size from sclp to
> machine code"
> - Move code to a new function to make further changes easier
> - Adjust s390_pv_vm_try_disable_async() to stay in sync with the
> maxram->ram change
> * "s390x: introduce s390_get_memory_limit()"
> - Store limit in machine
> - Move s390_set_memory_limit() from target code into machine code
> * "s390x/s390-hypercall: introduce DIAG500 STORAGE_LIMIT"
> - Move handling into a separate function now that we lookup the machine
> * "s390x/s390-stattrib-kvm: prepare for memory devices and sparse memory
> layouts"
> - Adjust to s390_get_memory_limit() changes
> * "s390x/s390-skeys: prepare for memory devices"
> - Adjust to s390_get_memory_limit() changes
> * "s390x/pv: prepare for memory devices"
> - Use s390_get_memory_limit()
> * "s390x: remember the maximum page size"
> - Store it in the machine
> - Move s390_set_max_pagesize() from target code into machine code
> - No need for s390_get_max_pagesize()
> * "s390x/virtio-ccw: add support for virtio based memory devices"
> - Move machine wire-up code from virtio-mem patch into this patch
> - Add stubs to make compilation without virtio-mem work
> * "s390x: virtio-mem support"
> - Move machine write-up code to previous patch
>
> Cc: Paolo Bonzini <pbonzini@redhat.com>
> Cc: Thomas Huth <thuth@redhat.com>
> Cc: Halil Pasic <pasic@linux.ibm.com>
> Cc: Christian Borntraeger <borntraeger@linux.ibm.com>
> Cc: Eric Farman <farman@linux.ibm.com>
> Cc: Richard Henderson <richard.henderson@linaro.org>
> Cc: Nina Schoetterl-Glausch <nsg@linux.ibm.com>
> Cc: Heiko Carstens <hca@linux.ibm.com>
> Cc: Ilya Leoshkevich <iii@linux.ibm.com>
> Cc: Janosch Frank <frankja@linux.ibm.com>
> Cc: "Michael S. Tsirkin" <mst@redhat.com>
> Cc: Cornelia Huck <cohuck@redhat.com>
>
> David Hildenbrand (14):
> s390x/s390-virtio-ccw: don't crash on weird RAM sizes
> s390x/s390-virtio-hcall: remove hypercall registration mechanism
> s390x/s390-virtio-hcall: prepare for more diag500 hypercalls
> s390x: rename s390-virtio-hcall* to s390-hypercall*
> s390x/s390-virtio-ccw: move setting the maximum guest size from sclp
> to machine code
> s390x: introduce s390_get_memory_limit()
> s390x/s390-hypercall: introduce DIAG500 STORAGE_LIMIT
> s390x/s390-stattrib-kvm: prepare for memory devices and sparse memory
> layouts
> s390x/s390-skeys: prepare for memory devices
> s390x/s390-virtio-ccw: prepare for memory devices
> s390x/pv: prepare for memory devices
> s390x: remember the maximum page size
> s390x/virtio-ccw: add support for virtio based memory devices
> s390x: virtio-mem support
>
> MAINTAINERS | 5 +
> hw/s390x/Kconfig | 1 +
> hw/s390x/meson.build | 6 +-
> hw/s390x/s390-hypercall.c | 85 +++++++++++
> hw/s390x/s390-hypercall.h | 25 ++++
> hw/s390x/s390-skeys.c | 6 +-
> hw/s390x/s390-stattrib-kvm.c | 67 ++++++---
> hw/s390x/s390-virtio-ccw.c | 165 ++++++++++++++-------
> hw/s390x/s390-virtio-hcall.c | 41 ------
> hw/s390x/s390-virtio-hcall.h | 25 ----
> hw/s390x/sclp.c | 17 +--
> hw/s390x/virtio-ccw-md-stubs.c | 24 +++
> hw/s390x/virtio-ccw-md.c | 153 +++++++++++++++++++
> hw/s390x/virtio-ccw-md.h | 44 ++++++
> hw/s390x/virtio-ccw-mem.c | 226 +++++++++++++++++++++++++++++
> hw/s390x/virtio-ccw-mem.h | 34 +++++
> hw/virtio/Kconfig | 1 +
> hw/virtio/virtio-mem.c | 4 +-
> include/hw/s390x/s390-virtio-ccw.h | 4 +
> target/s390x/cpu-sysemu.c | 15 --
> target/s390x/cpu.h | 2 -
> target/s390x/kvm/kvm.c | 18 +--
> target/s390x/kvm/pv.c | 2 +-
> target/s390x/tcg/misc_helper.c | 7 +-
> 24 files changed, 782 insertions(+), 195 deletions(-)
> create mode 100644 hw/s390x/s390-hypercall.c
> create mode 100644 hw/s390x/s390-hypercall.h
> delete mode 100644 hw/s390x/s390-virtio-hcall.c
> delete mode 100644 hw/s390x/s390-virtio-hcall.h
> create mode 100644 hw/s390x/virtio-ccw-md-stubs.c
> create mode 100644 hw/s390x/virtio-ccw-md.c
> create mode 100644 hw/s390x/virtio-ccw-md.h
> create mode 100644 hw/s390x/virtio-ccw-mem.c
> create mode 100644 hw/s390x/virtio-ccw-mem.h
>
> --
> 2.46.1
>
>
On 08.10.24 12:54, David Hildenbrand wrote: > Based on current master. > > There is really not much left to do on s390x, because virtio-mem already > implements most things we need today (e.g., early-migration, > unplugged-inaccessible). The biggest part of this series is just doing what > we do with virtio-pci, wiring it up in the machine hotplug handler and ... > well, messing with the physical memory layout where we can now exceed > initial RAM size and have sparsity (memory holes). > > I tested a lot of things, including: > * Memory hotplug/unplug > * Device hotplug/unplug > * System resets / reboots > * Migrate to/from file (including storage attributes under KVM) > * Basic live migration > * Basic postcopy live migration > > More details on how to use it on s390x -- which is pretty much how > we use it on other architectures, except > s/virtio-mem-pci/virtio-mem-ccw/ --- is in the last patch. > > This series introduces a new diag(500) "STORAGE LIMIT" subcode that will > be documented in the kernel and at [2] once this+kernel part go upstream. > > There are not many s390x-specific virtio-mem future work items, except: > * Storage attribute migration might be improved > * We might want to reset storage attributes of unplugged memory > (might or might not be required for upcoming page table reclaim in > Linux; TBD) > > The Linux driver is available at [3]. > > [1] https://lkml.kernel.org/r/20240906101658.514470-1-pbonzini@redhat.com > [2] https://gitlab.com/davidhildenbrand/s390x-os-virt-spec > [3] https://lkml.kernel.org/r/20240910191541.2179655-6-david@redhat.com Gentle ping (and thanks to Thomas for the review!). I assume the kernel portion will go upstream in the next merge window. I'd like get the QEMU parts merged soon after that. 9.2 is going to get released in roughly one month, so there is still time. @Thomas, this is mostly s390x stuff, so I guess it should go through the s390x tree? But I could also take this through my "memory devices" tree.
On 13.11.24 15:46, David Hildenbrand wrote: > On 08.10.24 12:54, David Hildenbrand wrote: >> Based on current master. >> >> There is really not much left to do on s390x, because virtio-mem already >> implements most things we need today (e.g., early-migration, >> unplugged-inaccessible). The biggest part of this series is just doing what >> we do with virtio-pci, wiring it up in the machine hotplug handler and ... >> well, messing with the physical memory layout where we can now exceed >> initial RAM size and have sparsity (memory holes). >> >> I tested a lot of things, including: >> * Memory hotplug/unplug >> * Device hotplug/unplug >> * System resets / reboots >> * Migrate to/from file (including storage attributes under KVM) >> * Basic live migration >> * Basic postcopy live migration >> >> More details on how to use it on s390x -- which is pretty much how >> we use it on other architectures, except >> s/virtio-mem-pci/virtio-mem-ccw/ --- is in the last patch. >> >> This series introduces a new diag(500) "STORAGE LIMIT" subcode that will >> be documented in the kernel and at [2] once this+kernel part go upstream. >> >> There are not many s390x-specific virtio-mem future work items, except: >> * Storage attribute migration might be improved >> * We might want to reset storage attributes of unplugged memory >> (might or might not be required for upcoming page table reclaim in >> Linux; TBD) >> >> The Linux driver is available at [3]. >> >> [1] https://lkml.kernel.org/r/20240906101658.514470-1-pbonzini@redhat.com >> [2] https://gitlab.com/davidhildenbrand/s390x-os-virt-spec >> [3] https://lkml.kernel.org/r/20240910191541.2179655-6-david@redhat.com > > Gentle ping (and thanks to Thomas for the review!). > > I assume the kernel portion will go upstream in the next merge window. > I'd like get the QEMU parts merged soon after that. > > 9.2 is going to get released in roughly one month, so there is still time. In the meantime, 9.2 was released. I don't have any changes planned. Series still applies to current master, I'll do a quick test tomorrow. > > @Thomas, this is mostly s390x stuff, so I guess it should go through the > s390x tree? But I could also take this through my "memory devices" tree. > @Thomas, any thoughts?
Based on current master. There is really not much left to do on s390x, because virtio-mem already implements most things we need today (e.g., early-migration, unplugged-inaccessible). The biggest part of this series is just doing what we do with virtio-pci, wiring it up in the machine hotplug handler and ... well, messing with the physical memory layout where we can now exceed initial RAM size and have sparsity (memory holes). I tested a lot of things, including: * Memory hotplug/unplug * Device hotplug/unplug * System resets / reboots * Migrate to/from file (including storage attributes under KVM) * Basic live migration * Basic postcopy live migration More details on how to use it on s390x -- which is pretty much how we use it on other architectures, except s/virtio-mem-pci/virtio-mem-ccw/ --- is in the last patch. This series introduces a new diag(500) "STORAGE LIMIT" subcode that will be documented in the kernel and at [2] once this+kernel part go upstream. There are not many s390x-specific virtio-mem future work items, except: * Storage attribute migration might be improved * We might want to reset storage attributes of unplugged memory (might or might not be required for upcoming page table reclaim in Linux; TBD) The Linux driver is available at [3]. [1] https://lkml.kernel.org/r/20240906101658.514470-1-pbonzini@redhat.com [2] https://gitlab.com/davidhildenbrand/s390x-os-virt-spec [3] https://lkml.kernel.org/r/20240910191541.2179655-6-david@redhat.com v1 -> v2: * "s390x/s390-virtio-hcall: prepare for more diag500 hypercalls" - Turn handle_diag_500() into a void function - Inject PGM_SPECIFICATION from handle_diag_500() directly * "s390x/s390-virtio-ccw: move setting the maximum guest size from sclp to machine code" - Move code to a new function to make further changes easier - Adjust s390_pv_vm_try_disable_async() to stay in sync with the maxram->ram change * "s390x: introduce s390_get_memory_limit()" - Store limit in machine - Move s390_set_memory_limit() from target code into machine code * "s390x/s390-hypercall: introduce DIAG500 STORAGE_LIMIT" - Move handling into a separate function now that we lookup the machine * "s390x/s390-stattrib-kvm: prepare for memory devices and sparse memory layouts" - Adjust to s390_get_memory_limit() changes * "s390x/s390-skeys: prepare for memory devices" - Adjust to s390_get_memory_limit() changes * "s390x/pv: prepare for memory devices" - Use s390_get_memory_limit() * "s390x: remember the maximum page size" - Store it in the machine - Move s390_set_max_pagesize() from target code into machine code - No need for s390_get_max_pagesize() * "s390x/virtio-ccw: add support for virtio based memory devices" - Move machine wire-up code from virtio-mem patch into this patch - Add stubs to make compilation without virtio-mem work * "s390x: virtio-mem support" - Move machine write-up code to previous patch Cc: Paolo Bonzini <pbonzini@redhat.com> Cc: Thomas Huth <thuth@redhat.com> Cc: Halil Pasic <pasic@linux.ibm.com> Cc: Christian Borntraeger <borntraeger@linux.ibm.com> Cc: Eric Farman <farman@linux.ibm.com> Cc: Richard Henderson <richard.henderson@linaro.org> Cc: Nina Schoetterl-Glausch <nsg@linux.ibm.com> Cc: Heiko Carstens <hca@linux.ibm.com> Cc: Ilya Leoshkevich <iii@linux.ibm.com> Cc: Janosch Frank <frankja@linux.ibm.com> Cc: "Michael S. Tsirkin" <mst@redhat.com> Cc: Cornelia Huck <cohuck@redhat.com> David Hildenbrand (14): s390x/s390-virtio-ccw: don't crash on weird RAM sizes s390x/s390-virtio-hcall: remove hypercall registration mechanism s390x/s390-virtio-hcall: prepare for more diag500 hypercalls s390x: rename s390-virtio-hcall* to s390-hypercall* s390x/s390-virtio-ccw: move setting the maximum guest size from sclp to machine code s390x: introduce s390_get_memory_limit() s390x/s390-hypercall: introduce DIAG500 STORAGE_LIMIT s390x/s390-stattrib-kvm: prepare for memory devices and sparse memory layouts s390x/s390-skeys: prepare for memory devices s390x/s390-virtio-ccw: prepare for memory devices s390x/pv: prepare for memory devices s390x: remember the maximum page size s390x/virtio-ccw: add support for virtio based memory devices s390x: virtio-mem support MAINTAINERS | 5 + hw/s390x/Kconfig | 1 + hw/s390x/meson.build | 6 +- hw/s390x/s390-hypercall.c | 85 +++++++++++ hw/s390x/s390-hypercall.h | 25 ++++ hw/s390x/s390-skeys.c | 6 +- hw/s390x/s390-stattrib-kvm.c | 67 ++++++--- hw/s390x/s390-virtio-ccw.c | 165 ++++++++++++++------- hw/s390x/s390-virtio-hcall.c | 41 ------ hw/s390x/s390-virtio-hcall.h | 25 ---- hw/s390x/sclp.c | 17 +-- hw/s390x/virtio-ccw-md-stubs.c | 24 +++ hw/s390x/virtio-ccw-md.c | 153 +++++++++++++++++++ hw/s390x/virtio-ccw-md.h | 44 ++++++ hw/s390x/virtio-ccw-mem.c | 226 +++++++++++++++++++++++++++++ hw/s390x/virtio-ccw-mem.h | 34 +++++ hw/virtio/Kconfig | 1 + hw/virtio/virtio-mem.c | 4 +- include/hw/s390x/s390-virtio-ccw.h | 4 + target/s390x/cpu-sysemu.c | 15 -- target/s390x/cpu.h | 2 - target/s390x/kvm/kvm.c | 18 +-- target/s390x/kvm/pv.c | 2 +- target/s390x/tcg/misc_helper.c | 7 +- 24 files changed, 782 insertions(+), 195 deletions(-) create mode 100644 hw/s390x/s390-hypercall.c create mode 100644 hw/s390x/s390-hypercall.h delete mode 100644 hw/s390x/s390-virtio-hcall.c delete mode 100644 hw/s390x/s390-virtio-hcall.h create mode 100644 hw/s390x/virtio-ccw-md-stubs.c create mode 100644 hw/s390x/virtio-ccw-md.c create mode 100644 hw/s390x/virtio-ccw-md.h create mode 100644 hw/s390x/virtio-ccw-mem.c create mode 100644 hw/s390x/virtio-ccw-mem.h