Message ID | 20240910175809.2135596-1-david@redhat.com (mailing list archive) |
---|---|
Headers | show |
Series | s390x: virtio-mem support | expand |
On Tue, Sep 10, 2024 at 07:57:55PM +0200, David Hildenbrand wrote: > This series is based on: > [PATCH v2] virtio: kconfig: memory devices are PCI only [1] > > I finally found the time (IOW forced myself) to finish virtio-mem > support on s390x. The last RFC was from 2020, so I won't talk about > what changed -- a lot changed in the meantime :) > > There is really not much left to do on s390x, because virtio-mem already > implements most things we need today (e.g., early-migration, > unplugged-inaccessible). The biggest part of this series is just doing what > we do with virtio-pci, wiring it up in the machine hotplug handler and ... > well, messing with the physical memory layout where we can now exceed > initial RAM size and have sparsity (memory holes). > > I tested a lot of things, including: > * Memory hotplug/unplug > * Device hotplug/unplug > * System resets / reboots > * Migrate to/from file (including storage attributes under KVM) > * Basic live migration > * Basic postcopy live migration > > More details on how to use it on s390x -- which is pretty much how > we use it on other architectures, except > s/virtio-mem-pci/virtio-mem-ccw/ --- is in the last patch. > > This series introduces a new diag(500) "STORAGE LIMIT" subcode that will > be documented at [2] once this+kernel part go upstream. > > There are not many s390x-specific virtio-mem future work items, except: > * Storage attribute migration might be improved > * We might want to reset storage attributes of unplugged memory > (might or might not be required for upcoming page table reclaim in > Linux; TBD) I don't see anything needing virtio specific here, let me know if I missed anything. A quick look is fine so I guess you can add Acked-by: Michael S. Tsirkin <mst@redhat.com> > I'll send out the kernel driver bits soon. > > [1] https://lkml.kernel.org/r/20240906101658.514470-1-pbonzini@redhat.com > [2] https://gitlab.com/davidhildenbrand/s390x-os-virt-spec > > Cc: Paolo Bonzini <pbonzini@redhat.com> > Cc: Thomas Huth <thuth@redhat.com> > Cc: Halil Pasic <pasic@linux.ibm.com> > Cc: Christian Borntraeger <borntraeger@linux.ibm.com> > Cc: Eric Farman <farman@linux.ibm.com> > Cc: Richard Henderson <richard.henderson@linaro.org> > Cc: David Hildenbrand <david@redhat.com> > Cc: Ilya Leoshkevich <iii@linux.ibm.com> > Cc: Janosch Frank <frankja@linux.ibm.com> > Cc: "Michael S. Tsirkin" <mst@redhat.com> > Cc: Cornelia Huck <cohuck@redhat.com> > > David Hildenbrand (14): > s390x/s390-virtio-ccw: don't crash on weird RAM sizes > s390x/s390-virtio-hcall: remove hypercall registration mechanism > s390x/s390-virtio-hcall: prepare for more diag500 hypercalls > s390x: rename s390-virtio-hcall* to s390-hypercall* > s390x/s390-virtio-ccw: move setting the maximum guest size from sclp > to machine code > s390x: introduce s390_get_memory_limit() > s390x/s390-hypercall: introduce DIAG500 STORAGE_LIMIT > s390x/s390-stattrib-kvm: prepare memory devices and sparse memory > layouts > s390x/s390-skeys: prepare for memory devices > s390x/pv: check initial, not maximum RAM size > s390x/s390-virtio-ccw: prepare for memory devices > s390x: introduce s390_get_max_pagesize() > s390x/virtio-ccw: add support for virtio based memory devices > s390x: virtio-mem support > > MAINTAINERS | 4 + > hw/s390x/Kconfig | 1 + > hw/s390x/meson.build | 4 +- > hw/s390x/s390-hypercall.c | 77 +++++++++++ > hw/s390x/s390-hypercall.h | 25 ++++ > hw/s390x/s390-skeys.c | 4 +- > hw/s390x/s390-stattrib-kvm.c | 63 +++++---- > hw/s390x/s390-virtio-ccw.c | 143 +++++++++++++-------- > hw/s390x/s390-virtio-hcall.c | 41 ------ > hw/s390x/s390-virtio-hcall.h | 25 ---- > hw/s390x/sclp.c | 17 +-- > hw/s390x/virtio-ccw-md.c | 153 ++++++++++++++++++++++ > hw/s390x/virtio-ccw-md.h | 44 +++++++ > hw/s390x/virtio-ccw-mem.c | 226 +++++++++++++++++++++++++++++++++ > hw/s390x/virtio-ccw-mem.h | 34 +++++ > hw/virtio/Kconfig | 1 + > hw/virtio/virtio-mem.c | 4 +- > target/s390x/cpu-sysemu.c | 35 ++++- > target/s390x/cpu.h | 2 + > target/s390x/kvm/kvm.c | 12 +- > target/s390x/kvm/pv.c | 2 +- > target/s390x/tcg/misc_helper.c | 6 +- > 22 files changed, 746 insertions(+), 177 deletions(-) > create mode 100644 hw/s390x/s390-hypercall.c > create mode 100644 hw/s390x/s390-hypercall.h > delete mode 100644 hw/s390x/s390-virtio-hcall.c > delete mode 100644 hw/s390x/s390-virtio-hcall.h > create mode 100644 hw/s390x/virtio-ccw-md.c > create mode 100644 hw/s390x/virtio-ccw-md.h > create mode 100644 hw/s390x/virtio-ccw-mem.c > create mode 100644 hw/s390x/virtio-ccw-mem.h > > -- > 2.46.0
On 10.09.24 20:33, Michael S. Tsirkin wrote: > On Tue, Sep 10, 2024 at 07:57:55PM +0200, David Hildenbrand wrote: >> This series is based on: >> [PATCH v2] virtio: kconfig: memory devices are PCI only [1] >> >> I finally found the time (IOW forced myself) to finish virtio-mem >> support on s390x. The last RFC was from 2020, so I won't talk about >> what changed -- a lot changed in the meantime :) >> >> There is really not much left to do on s390x, because virtio-mem already >> implements most things we need today (e.g., early-migration, >> unplugged-inaccessible). The biggest part of this series is just doing what >> we do with virtio-pci, wiring it up in the machine hotplug handler and ... >> well, messing with the physical memory layout where we can now exceed >> initial RAM size and have sparsity (memory holes). >> >> I tested a lot of things, including: >> * Memory hotplug/unplug >> * Device hotplug/unplug >> * System resets / reboots >> * Migrate to/from file (including storage attributes under KVM) >> * Basic live migration >> * Basic postcopy live migration >> >> More details on how to use it on s390x -- which is pretty much how >> we use it on other architectures, except >> s/virtio-mem-pci/virtio-mem-ccw/ --- is in the last patch. >> >> This series introduces a new diag(500) "STORAGE LIMIT" subcode that will >> be documented at [2] once this+kernel part go upstream. >> >> There are not many s390x-specific virtio-mem future work items, except: >> * Storage attribute migration might be improved >> * We might want to reset storage attributes of unplugged memory >> (might or might not be required for upcoming page table reclaim in >> Linux; TBD) > > > I don't see anything needing virtio specific here, let me know if > I missed anything. No, it's really just wiring up virtio-mem. > A quick look is fine so I guess you can add > Acked-by: Michael S. Tsirkin <mst@redhat.com> Thanks!
On 9/10/24 7:57 PM, David Hildenbrand wrote: > This series introduces a new diag(500) "STORAGE LIMIT" subcode that will > be documented at [2] once this+kernel part go upstream. Why not in Documentation/virt/kvm/s390/? s390-diag.rst is very similar already. I'd rather have it in a shared and bigger repo than in your personal gitlab one. Maybe there's a space somewhere in QEMU or the Virtio team's repos that would be a good fit if the kernel's docu isn't the right place?
On 11.09.24 13:49, Janosch Frank wrote: > On 9/10/24 7:57 PM, David Hildenbrand wrote: >> This series introduces a new diag(500) "STORAGE LIMIT" subcode that will >> be documented at [2] once this+kernel part go upstream. > > Why not in Documentation/virt/kvm/s390/? > s390-diag.rst is very similar already. I can document it anywhere people fancy, really I don't care. But this has some history. The last time we discussed it [1] there was: "Regardless what we end up with, this needs to be specified somewhere(tm)." from Conny "It must be well defined and easy to find also for kernel developers who actually have to care about memory detection code :)" from Heiko. And then "the kernel's s390-diag.rst should also point to it ... Nothing as complicated as an OASIS spec, but maybe a git??b project?" from Conny > > I'd rather have it in a shared and bigger repo than in your personal > gitlab one. Maybe there's a space somewhere in QEMU or the Virtio team's > repos that would be a good fit if the kernel's docu isn't the right place? At this point, outside of kernel/QEMU feels like the right thing to do. Conny is already a co-maintainer of my "personal" (;)) gitlab. And now I realize that I CCed Heiko on the Linux series but not the QEMU series. My bad. [1] https://lore.kernel.org/all/20200727114819.3f816010.cohuck@redhat.com/
> > > > I'd rather have it in a shared and bigger repo than in your personal > > gitlab one. Maybe there's a space somewhere in QEMU or the Virtio team's > > repos that would be a good fit if the kernel's docu isn't the right place? > > At this point, outside of kernel/QEMU feels like the right thing to do. > Conny is already a co-maintainer of my "personal" (;)) gitlab. > > > And now I realize that I CCed Heiko on the Linux series but not the QEMU > series. My bad. > > [1] https://lore.kernel.org/all/20200727114819.3f816010.cohuck@redhat.com/ No prob. Or if you want it in virtio spec, that's also fine.
On Wed, Sep 11 2024, "Michael S. Tsirkin" <mst@redhat.com> wrote: >> > >> > I'd rather have it in a shared and bigger repo than in your personal >> > gitlab one. Maybe there's a space somewhere in QEMU or the Virtio team's >> > repos that would be a good fit if the kernel's docu isn't the right place? >> >> At this point, outside of kernel/QEMU feels like the right thing to do. >> Conny is already a co-maintainer of my "personal" (;)) gitlab. >> >> >> And now I realize that I CCed Heiko on the Linux series but not the QEMU >> series. My bad. >> >> [1] https://lore.kernel.org/all/20200727114819.3f816010.cohuck@redhat.com/ > > > No prob. Or if you want it in virtio spec, that's also fine. The virtio spec makes sense for documenting things needed to implement it, but what I liked about the gitlab project is that it tries to go into more s390-specific aspects (that feel a bit out of scope for the virtio spec), and that it provides a place to document non-virtio related interfaces. Anyway, if we want to proceed with the gitlab project, would it make sense to create an org for it, so that it doesn't look like David's personal project? In any case, while I'm happy to stay on, I'm not that involved with s390 anymore, and it might make sense to add more people to it.
On 11.09.24 17:38, Cornelia Huck wrote: > On Wed, Sep 11 2024, "Michael S. Tsirkin" <mst@redhat.com> wrote: > >>>> >>>> I'd rather have it in a shared and bigger repo than in your personal >>>> gitlab one. Maybe there's a space somewhere in QEMU or the Virtio team's >>>> repos that would be a good fit if the kernel's docu isn't the right place? >>> >>> At this point, outside of kernel/QEMU feels like the right thing to do. >>> Conny is already a co-maintainer of my "personal" (;)) gitlab. >>> >>> >>> And now I realize that I CCed Heiko on the Linux series but not the QEMU >>> series. My bad. >>> >>> [1] https://lore.kernel.org/all/20200727114819.3f816010.cohuck@redhat.com/ >> >> >> No prob. Or if you want it in virtio spec, that's also fine. > > The virtio spec makes sense for documenting things needed to implement > it, but what I liked about the gitlab project is that it tries to go > into more s390-specific aspects (that feel a bit out of scope for the > virtio spec), and that it provides a place to document non-virtio > related interfaces. > > Anyway, if we want to proceed with the gitlab project, would it make > sense to create an org for it, so that it doesn't look like David's > personal project? I had that in mind as well, I can move the project. > In any case, while I'm happy to stay on, I'm not that > involved with s390 anymore, and it might make sense to add more people > to it. Indeed!
On Wed, 11 Sep 2024 21:09:27 +0200 David Hildenbrand <david@redhat.com> wrote: > > Anyway, if we want to proceed with the gitlab project, would it make > > sense to create an org for it, so that it doesn't look like David's > > personal project? Frankly, I would prefer making Documentation/virt/kvm/s390/s390-diag.rst the authoritative documentation on DIAGs. My train of thought is DIAG 500 is a KVM thing, and KVM is a linux kernel thing, so it just feels right for the documentatio to live within the linux source tree. I may have missed some of the discussion: what were the benefits of having this in its separate project/repository? Regards, Halil
On 27.09.24 20:20, Halil Pasic wrote: > On Wed, 11 Sep 2024 21:09:27 +0200 > David Hildenbrand <david@redhat.com> wrote: > >>> Anyway, if we want to proceed with the gitlab project, would it make >>> sense to create an org for it, so that it doesn't look like David's >>> personal project? > > Frankly, I would prefer making Documentation/virt/kvm/s390/s390-diag.rst > the authoritative documentation on DIAGs. > > My train of thought is DIAG 500 is a KVM thing, and KVM is a linux > kernel thing, so it just feels right for the documentatio to > live within the linux source tree. QEMU/TCG is the proof that KVM is not necessarily involved. Are you sure that no other OS out there besides Linux implements virtio on s390x, or would want to implement it? :) > > I may have missed some of the discussion: what were the benefits > of having this in its separate project/repository? Having it independent of the implementation.
On Fri, 27 Sep 2024 20:29:19 +0200 David Hildenbrand <david@redhat.com> wrote: > On 27.09.24 20:20, Halil Pasic wrote: > > On Wed, 11 Sep 2024 21:09:27 +0200 > > David Hildenbrand <david@redhat.com> wrote: > > > >>> Anyway, if we want to proceed with the gitlab project, would it make > >>> sense to create an org for it, so that it doesn't look like David's > >>> personal project? > > > > Frankly, I would prefer making Documentation/virt/kvm/s390/s390-diag.rst > > the authoritative documentation on DIAGs. > > > > My train of thought is DIAG 500 is a KVM thing, and KVM is a linux > > kernel thing, so it just feels right for the documentatio to > > live within the linux source tree. > > QEMU/TCG is the proof that KVM is not necessarily involved. > > Are you sure that no other OS out there besides Linux implements virtio > on s390x, or would want to implement it? :) > As Christian has pointed out in another thread DIAG 500 is documented as the KVM hypervisor call, and that made me argue it is a KVM thing. You are right KVM is not necessarily involved, and neither is QEMU. For me it is not about the components involved in the visualization, but about the people, projects and governance. IMHO this is basically extending the s390 architecture. We are guaranteed to not collide with the Architecture because DIAG 500 is reserved for KVM as a project I guess. > > > > I may have missed some of the discussion: what were the benefits > > of having this in its separate project/repository? > > Having it independent of the implementation. > That is a valid point. But IMHO the benefit of having this independent, does not justify the churn of having a separate project with its own governance, and communication infrastructure. And I suppose for an open(?) specification, one would need those things. No strong opinions though. If Christian, Janosch and Claudio are in favor of a separate "Specifications for open-source virtualization on s390x (IBM z Systems)" project, I'm fine with it as well. Regards, Halil
On 30.09.24 23:49, Halil Pasic wrote: > On Fri, 27 Sep 2024 20:29:19 +0200 > David Hildenbrand <david@redhat.com> wrote: > >> On 27.09.24 20:20, Halil Pasic wrote: >>> On Wed, 11 Sep 2024 21:09:27 +0200 >>> David Hildenbrand <david@redhat.com> wrote: >>> >>>>> Anyway, if we want to proceed with the gitlab project, would it make >>>>> sense to create an org for it, so that it doesn't look like David's >>>>> personal project? >>> >>> Frankly, I would prefer making Documentation/virt/kvm/s390/s390-diag.rst >>> the authoritative documentation on DIAGs. >>> >>> My train of thought is DIAG 500 is a KVM thing, and KVM is a linux >>> kernel thing, so it just feels right for the documentatio to >>> live within the linux source tree. >> >> QEMU/TCG is the proof that KVM is not necessarily involved. >> >> Are you sure that no other OS out there besides Linux implements virtio >> on s390x, or would want to implement it? :) >> > > As Christian has pointed out in another thread DIAG 500 is documented > as the KVM hypervisor call, and that made me argue it is a KVM thing. > > You are right KVM is not necessarily involved, and neither is QEMU. For > me it is not about the components involved in the visualization, but > about the people, projects and governance. > > IMHO this is basically extending the s390 architecture. We are guaranteed > to not collide with the Architecture because DIAG 500 is reserved for > KVM as a project I guess. That's my understanding. I assume because the CCW virtio machine started out as KVM-only, documenting that it is "KVM ONLY" may be because of historical reasons. > >>> >>> I may have missed some of the discussion: what were the benefits >>> of having this in its separate project/repository? >> >> Having it independent of the implementation. >> > > That is a valid point. But IMHO the benefit of having this independent, > does not justify the churn of having a separate project with its > own governance, and communication infrastructure. And I suppose for an > open(?) specification, one would need those things. I don't see the need to bring in all that bureaucracy. The original idea was simple: if QEMU/TCG or QEMU/KVM implement a hypercall (IOW: it was acked by the s390x maintainers), we document it somewhere. Implementing something in QEMU and then modifying a KVM document in the kernel tree sounded odd. It is a valid question to ask: what if any other hypervisor (cloud-hypervisor etc.) would want to implement a custom diag500 hypercall, and who would ack it. But I don't really think that we would have to sort this out this at this point in time. > > No strong opinions though. If Christian, Janosch and Claudio are in > favor of a separate "Specifications for open-source virtualization on > s390x (IBM z Systems)" project, I'm fine with it as well. I'm more than happy if we don't need that. As said, I'm happy to document wherever people tell me to document. 4 years ago we thought that having a separate repository was a good idea, maybe it is no longer. In that case, s390x mainters please let me know what to do :)
On 10/1/24 10:54 AM, David Hildenbrand wrote: > On 30.09.24 23:49, Halil Pasic wrote: >> On Fri, 27 Sep 2024 20:29:19 +0200 >> David Hildenbrand <david@redhat.com> wrote: >> >>> On 27.09.24 20:20, Halil Pasic wrote: >>>> On Wed, 11 Sep 2024 21:09:27 +0200 >>>> David Hildenbrand <david@redhat.com> wrote: [...] >> That is a valid point. But IMHO the benefit of having this independent, >> does not justify the churn of having a separate project with its >> own governance, and communication infrastructure. And I suppose for an >> open(?) specification, one would need those things. > > I don't see the need to bring in all that bureaucracy. The original idea > was simple: if QEMU/TCG or QEMU/KVM implement a hypercall (IOW: it was > acked by the s390x maintainers), we document it somewhere. > > Implementing something in QEMU and then modifying a KVM document in the > kernel tree sounded odd. > > It is a valid question to ask: what if any other hypervisor > (cloud-hypervisor etc.) would want to implement a custom diag500 > hypercall, and who would ack it. But I don't really think that we would > have to sort this out this at this point in time. > >> >> No strong opinions though. If Christian, Janosch and Claudio are in >> favor of a separate "Specifications for open-source virtualization on >> s390x (IBM z Systems)" project, I'm fine with it as well. > > I'm more than happy if we don't need that. As said, I'm happy to > document wherever people tell me to document. > > 4 years ago we thought that having a separate repository was a good > idea, maybe it is no longer. In that case, s390x mainters please let me > know what to do :) > I'd like to at least have a partial documentation in the kernel if not a full one. You can add a link to your repo to that. Even if they go out of sync I'd rather have documentation where I'd expect it (Linux) than only the repo. IMHO duplication isn't a gigantic issue here. We also have an internal space where KVM architecture is being stored (currently a handful of documents) and we'll store it there as well, including a link to the repo.
On 02.10.24 11:04, Janosch Frank wrote: > On 10/1/24 10:54 AM, David Hildenbrand wrote: >> On 30.09.24 23:49, Halil Pasic wrote: >>> On Fri, 27 Sep 2024 20:29:19 +0200 >>> David Hildenbrand <david@redhat.com> wrote: >>> >>>> On 27.09.24 20:20, Halil Pasic wrote: >>>>> On Wed, 11 Sep 2024 21:09:27 +0200 >>>>> David Hildenbrand <david@redhat.com> wrote: > > [...] > >>> That is a valid point. But IMHO the benefit of having this independent, >>> does not justify the churn of having a separate project with its >>> own governance, and communication infrastructure. And I suppose for an >>> open(?) specification, one would need those things. >> >> I don't see the need to bring in all that bureaucracy. The original idea >> was simple: if QEMU/TCG or QEMU/KVM implement a hypercall (IOW: it was >> acked by the s390x maintainers), we document it somewhere. >> >> Implementing something in QEMU and then modifying a KVM document in the >> kernel tree sounded odd. >> >> It is a valid question to ask: what if any other hypervisor >> (cloud-hypervisor etc.) would want to implement a custom diag500 >> hypercall, and who would ack it. But I don't really think that we would >> have to sort this out this at this point in time. >> >>> >>> No strong opinions though. If Christian, Janosch and Claudio are in >>> favor of a separate "Specifications for open-source virtualization on >>> s390x (IBM z Systems)" project, I'm fine with it as well. >> >> I'm more than happy if we don't need that. As said, I'm happy to >> document wherever people tell me to document. >> >> 4 years ago we thought that having a separate repository was a good >> idea, maybe it is no longer. In that case, s390x mainters please let me >> know what to do :) >> > > I'd like to at least have a partial documentation in the kernel if not a > full one. You can add a link to your repo to that. Makes sense. > > Even if they go out of sync I'd rather have documentation where I'd > expect it (Linux) than only the repo. IMHO duplication isn't a gigantic > issue here. Agreed. > > We also have an internal space where KVM architecture is being stored > (currently a handful of documents) and we'll store it there as well, > including a link to the repo. Perfect. If there is a way to enable that repository, that would be great. For the time being I'll document it there as well. Thanks!