| Message ID | 20240403140116.3002809-1-vineeth@bitbyteword.org (mailing list archive) |
|---|---|
| Headers | show |
| Series | Paravirt Scheduling (Dynamic vcpu priority management) | expand |
Sorry I missed sched_ext folks, adding them as well. Thanks, Vineeth On Wed, Apr 3, 2024 at 10:01 AM Vineeth Pillai (Google) <vineeth@bitbyteword.org> wrote: > > Double scheduling is a concern with virtualization hosts where the host > schedules vcpus without knowing whats run by the vcpu and guest schedules > tasks without knowing where the vcpu is physically running. This causes > issues related to latencies, power consumption, resource utilization > etc. An ideal solution would be to have a cooperative scheduling > framework where the guest and host shares scheduling related information > and makes an educated scheduling decision to optimally handle the > workloads. As a first step, we are taking a stab at reducing latencies > for latency sensitive workloads in the guest. > > v1 RFC[1] was posted in December 2023. The main disagreement was in the > implementation where the patch was making scheduling policy decisions > in kvm and kvm is not the right place to do it. The suggestion was to > move the polcy decisions outside of kvm and let kvm only handle the > notifications needed to make the policy decisions. This patch series is > an iterative step towards implementing the feature as a layered > design where the policy could be implemented outside of kvm as a > kernel built-in, a kernel module or a bpf program. > > This design comprises mainly of 4 components: > > - pvsched driver: Implements the scheduling policies. Register with > host with a set of callbacks that hypervisor(kvm) can use to notify > vcpu events that the driver is interested in. The callback will be > passed in the address of shared memory so that the driver can get > scheduling information shared by the guest and also update the > scheduling policies set by the driver. > - kvm component: Selects the pvsched driver for a guest and notifies > the driver via callbacks for events that the driver is interested > in. Also interface with the guest in retreiving the shared memory > region for sharing the scheduling information. > - host kernel component: Implements the APIs for: > - pvsched driver for register/unregister to the host kernel, and > - hypervisor for assingning/unassigning driver for guests. > - guest component: Implements a framework for sharing the scheduling > information with the pvsched driver through kvm. > > There is another component that we refer to as pvsched protocol. This > defines the details about shared memory layout, information sharing and > sheduling policy decisions. The protocol need not be part of the kernel > and can be defined separately based on the use case and requirements. > Both guest and the selected pvsched driver need to match the protocol > for the feature to work. Protocol shall be identified by a name and a > possible versioning scheme. Guest will advertise the protocol and then > the hypervisor can assign the driver implementing the protocol if it is > registered in the host kernel. > > This patch series only implements the first 3 components. Guest side > implementation and the protocol framework shall come as a separate > series once we finalize rest of the design. > > This series also implements a sample bpf program and a kernel-builtin > pvsched drivers. They do not do any real stuff now, but just skeletons > to demonstrate the feature. > > Rebased on 6.8.2. > > [1]: https://lwn.net/Articles/955145/ > > Vineeth Pillai (Google) (5): > pvsched: paravirt scheduling framework > kvm: Implement the paravirt sched framework for kvm > kvm: interface for managing pvsched driver for guest VMs > pvsched: bpf support for pvsched > selftests/bpf: sample implementation of a bpf pvsched driver. > > Kconfig | 2 + > arch/x86/kvm/Kconfig | 13 + > arch/x86/kvm/x86.c | 3 + > include/linux/kvm_host.h | 32 +++ > include/linux/pvsched.h | 102 +++++++ > include/uapi/linux/kvm.h | 6 + > kernel/bpf/bpf_struct_ops_types.h | 4 + > kernel/sysctl.c | 27 ++ > .../testing/selftests/bpf/progs/bpf_pvsched.c | 37 +++ > virt/Makefile | 2 +- > virt/kvm/kvm_main.c | 265 ++++++++++++++++++ > virt/pvsched/Kconfig | 12 + > virt/pvsched/Makefile | 2 + > virt/pvsched/pvsched.c | 215 ++++++++++++++ > virt/pvsched/pvsched_bpf.c | 141 ++++++++++ > 15 files changed, 862 insertions(+), 1 deletion(-) > create mode 100644 include/linux/pvsched.h > create mode 100644 tools/testing/selftests/bpf/progs/bpf_pvsched.c > create mode 100644 virt/pvsched/Kconfig > create mode 100644 virt/pvsched/Makefile > create mode 100644 virt/pvsched/pvsched.c > create mode 100644 virt/pvsched/pvsched_bpf.c > > -- > 2.40.1 >
On Wed, Apr 03, 2024, Vineeth Pillai (Google) wrote: > Double scheduling is a concern with virtualization hosts where the host > schedules vcpus without knowing whats run by the vcpu and guest schedules > tasks without knowing where the vcpu is physically running. This causes > issues related to latencies, power consumption, resource utilization > etc. An ideal solution would be to have a cooperative scheduling > framework where the guest and host shares scheduling related information > and makes an educated scheduling decision to optimally handle the > workloads. As a first step, we are taking a stab at reducing latencies > for latency sensitive workloads in the guest. > > v1 RFC[1] was posted in December 2023. The main disagreement was in the > implementation where the patch was making scheduling policy decisions > in kvm and kvm is not the right place to do it. The suggestion was to > move the polcy decisions outside of kvm and let kvm only handle the > notifications needed to make the policy decisions. This patch series is > an iterative step towards implementing the feature as a layered > design where the policy could be implemented outside of kvm as a > kernel built-in, a kernel module or a bpf program. > > This design comprises mainly of 4 components: > > - pvsched driver: Implements the scheduling policies. Register with > host with a set of callbacks that hypervisor(kvm) can use to notify > vcpu events that the driver is interested in. The callback will be > passed in the address of shared memory so that the driver can get > scheduling information shared by the guest and also update the > scheduling policies set by the driver. > - kvm component: Selects the pvsched driver for a guest and notifies > the driver via callbacks for events that the driver is interested > in. Also interface with the guest in retreiving the shared memory > region for sharing the scheduling information. > - host kernel component: Implements the APIs for: > - pvsched driver for register/unregister to the host kernel, and > - hypervisor for assingning/unassigning driver for guests. > - guest component: Implements a framework for sharing the scheduling > information with the pvsched driver through kvm. Roughly summarazing an off-list discussion. - Discovery schedulers should be handled outside of KVM and the kernel, e.g. similar to how userspace uses PCI, VMBUS, etc. to enumerate devices to the guest. - "Negotiating" features/hooks should also be handled outside of the kernel, e.g. similar to how VirtIO devices negotiate features between host and guest. - Pushing PV scheduler entities to KVM should either be done through an exported API, e.g. if the scheduler is provided by a separate kernel module, or by a KVM or VM ioctl() (especially if the desire is to have per-VM schedulers). I think those were the main takeaways? Vineeth and Joel, please chime in on anything I've missed or misremembered. The other reason I'm bringing this discussion back on-list is that I (very) briefly discussed this with Paolo, and he pointed out the proposed rseq-based mechanism that would allow userspace to request an extended time slice[*], and that if that landed it would be easy-ish to reuse the interface for KVM's steal_time PV API. I see that you're both on that thread, so presumably you're already aware of the idea, but I wanted to bring it up here to make sure that we aren't trying to design something that's more complex than is needed. Specifically, if the guest has a generic way to request an extended time slice (or boost its priority?), would that address your use cases? Or rather, how close does it get you? E.g. the guest will have no way of requesting a larger time slice or boosting priority when an event is _pending_ but not yet receiveed by the guest, but is that actually problematic in practice? [*] https://lore.kernel.org/all/20231025235413.597287e1@gandalf.local.home
> > This design comprises mainly of 4 components: > > > > - pvsched driver: Implements the scheduling policies. Register with > > host with a set of callbacks that hypervisor(kvm) can use to notify > > vcpu events that the driver is interested in. The callback will be > > passed in the address of shared memory so that the driver can get > > scheduling information shared by the guest and also update the > > scheduling policies set by the driver. > > - kvm component: Selects the pvsched driver for a guest and notifies > > the driver via callbacks for events that the driver is interested > > in. Also interface with the guest in retreiving the shared memory > > region for sharing the scheduling information. > > - host kernel component: Implements the APIs for: > > - pvsched driver for register/unregister to the host kernel, and > > - hypervisor for assingning/unassigning driver for guests. > > - guest component: Implements a framework for sharing the scheduling > > information with the pvsched driver through kvm. > > Roughly summarazing an off-list discussion. > > - Discovery schedulers should be handled outside of KVM and the kernel, e.g. > similar to how userspace uses PCI, VMBUS, etc. to enumerate devices to the guest. > > - "Negotiating" features/hooks should also be handled outside of the kernel, > e.g. similar to how VirtIO devices negotiate features between host and guest. > > - Pushing PV scheduler entities to KVM should either be done through an exported > API, e.g. if the scheduler is provided by a separate kernel module, or by a > KVM or VM ioctl() (especially if the desire is to have per-VM schedulers). > > I think those were the main takeaways? Vineeth and Joel, please chime in on > anything I've missed or misremembered. > Thanks for the brief about the offlist discussion, all the points are captured, just some minor additions. v2 implementation removed the scheduling policies outside of kvm to a separate entity called pvsched driver and could be implemented as a kernel module or bpf program. But the handshake between guest and host to decide on what pvsched driver to attach was still going through kvm. So it was suggested to move this handshake(discovery and negotiation) outside of kvm. The idea is to have a virtual device exposed by the VMM which would take care of the handshake. Guest driver for this device would talk to the device to understand the pvsched details on the host and pass the shared memory details. Once the handshake is completed, the device is responsible for loading the pvsched driver(bpf program or kernel module responsible for implementing the policies). The pvsched driver will register to the trace points exported by kvm and handle the callbacks from then on. The scheduling will be taken care of by the host scheduler, pvsched driver on host is responsible only for setting the policies(placement, priorities etc). With the above approach, the only change in kvm would be the internal tracepoints for pvsched. Host kernel will also be unchanged and all the complexities move to the VMM and the pvsched driver. Guest kernel will have a new driver to talk to the virtual pvsched device and this driver would hook into the guest kernel for passing scheduling information to the host(via tracepoints). > The other reason I'm bringing this discussion back on-list is that I (very) briefly > discussed this with Paolo, and he pointed out the proposed rseq-based mechanism > that would allow userspace to request an extended time slice[*], and that if that > landed it would be easy-ish to reuse the interface for KVM's steal_time PV API. > > I see that you're both on that thread, so presumably you're already aware of the > idea, but I wanted to bring it up here to make sure that we aren't trying to > design something that's more complex than is needed. > > Specifically, if the guest has a generic way to request an extended time slice > (or boost its priority?), would that address your use cases? Or rather, how close > does it get you? E.g. the guest will have no way of requesting a larger time > slice or boosting priority when an event is _pending_ but not yet receiveed by > the guest, but is that actually problematic in practice? > > [*] https://lore.kernel.org/all/20231025235413.597287e1@gandalf.local.home > Thanks for bringing this up. We were also very much interested in this feature and were planning to use the pvmem shared memory instead of rseq framework for guests. The motivation of paravirt scheduling framework was a bit broader than the latency issues and hence we were proposing a bit more complex design. Other than the use case for temporarily extending the time slice of vcpus, we were also looking at vcpu placements on physical cpus, educated decisions that could be made by guest scheduler if it has a picture of host cpu load etc. Having a paravirt mechanism to share scheduling information would benefit in such cases. Once we have this framework setup, the policy implementation on guest and host could be taken care of by other entities like BPF programs, modules or schedulers like sched_ext. We are working on a v3 incorporating the above ideas and would shortly be posting a design RFC soon. Thanks for all the help and inputs on this. Thanks, Vineeth
Double scheduling is a concern with virtualization hosts where the host schedules vcpus without knowing whats run by the vcpu and guest schedules tasks without knowing where the vcpu is physically running. This causes issues related to latencies, power consumption, resource utilization etc. An ideal solution would be to have a cooperative scheduling framework where the guest and host shares scheduling related information and makes an educated scheduling decision to optimally handle the workloads. As a first step, we are taking a stab at reducing latencies for latency sensitive workloads in the guest. v1 RFC[1] was posted in December 2023. The main disagreement was in the implementation where the patch was making scheduling policy decisions in kvm and kvm is not the right place to do it. The suggestion was to move the polcy decisions outside of kvm and let kvm only handle the notifications needed to make the policy decisions. This patch series is an iterative step towards implementing the feature as a layered design where the policy could be implemented outside of kvm as a kernel built-in, a kernel module or a bpf program. This design comprises mainly of 4 components: - pvsched driver: Implements the scheduling policies. Register with host with a set of callbacks that hypervisor(kvm) can use to notify vcpu events that the driver is interested in. The callback will be passed in the address of shared memory so that the driver can get scheduling information shared by the guest and also update the scheduling policies set by the driver. - kvm component: Selects the pvsched driver for a guest and notifies the driver via callbacks for events that the driver is interested in. Also interface with the guest in retreiving the shared memory region for sharing the scheduling information. - host kernel component: Implements the APIs for: - pvsched driver for register/unregister to the host kernel, and - hypervisor for assingning/unassigning driver for guests. - guest component: Implements a framework for sharing the scheduling information with the pvsched driver through kvm. There is another component that we refer to as pvsched protocol. This defines the details about shared memory layout, information sharing and sheduling policy decisions. The protocol need not be part of the kernel and can be defined separately based on the use case and requirements. Both guest and the selected pvsched driver need to match the protocol for the feature to work. Protocol shall be identified by a name and a possible versioning scheme. Guest will advertise the protocol and then the hypervisor can assign the driver implementing the protocol if it is registered in the host kernel. This patch series only implements the first 3 components. Guest side implementation and the protocol framework shall come as a separate series once we finalize rest of the design. This series also implements a sample bpf program and a kernel-builtin pvsched drivers. They do not do any real stuff now, but just skeletons to demonstrate the feature. Rebased on 6.8.2. [1]: https://lwn.net/Articles/955145/ Vineeth Pillai (Google) (5): pvsched: paravirt scheduling framework kvm: Implement the paravirt sched framework for kvm kvm: interface for managing pvsched driver for guest VMs pvsched: bpf support for pvsched selftests/bpf: sample implementation of a bpf pvsched driver. Kconfig | 2 + arch/x86/kvm/Kconfig | 13 + arch/x86/kvm/x86.c | 3 + include/linux/kvm_host.h | 32 +++ include/linux/pvsched.h | 102 +++++++ include/uapi/linux/kvm.h | 6 + kernel/bpf/bpf_struct_ops_types.h | 4 + kernel/sysctl.c | 27 ++ .../testing/selftests/bpf/progs/bpf_pvsched.c | 37 +++ virt/Makefile | 2 +- virt/kvm/kvm_main.c | 265 ++++++++++++++++++ virt/pvsched/Kconfig | 12 + virt/pvsched/Makefile | 2 + virt/pvsched/pvsched.c | 215 ++++++++++++++ virt/pvsched/pvsched_bpf.c | 141 ++++++++++ 15 files changed, 862 insertions(+), 1 deletion(-) create mode 100644 include/linux/pvsched.h create mode 100644 tools/testing/selftests/bpf/progs/bpf_pvsched.c create mode 100644 virt/pvsched/Kconfig create mode 100644 virt/pvsched/Makefile create mode 100644 virt/pvsched/pvsched.c create mode 100644 virt/pvsched/pvsched_bpf.c