| Message ID | 20240124113446.2977003-1-foxywang@tencent.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | KVM: irqchip: synchronize srcu only if needed | expand |
Gentle ping :) On Wed, Jan 24, 2024 at 7:35 PM Yi Wang <up2wing@gmail.com> wrote: > > From: Yi Wang <foxywang@tencent.com> > > We found that it may cost more than 20 milliseconds very accidentally > to enable cap of KVM_CAP_SPLIT_IRQCHIP on a host which has many vms > already. > > The reason is that when vmm(qemu/CloudHypervisor) invokes > KVM_CAP_SPLIT_IRQCHIP kvm will call synchronize_srcu_expedited() and > might_sleep and kworker of srcu may cost some delay during this period. > One way makes sence is setup empty irq routing when creating vm and > so that x86/s390 don't need to setup empty/dummy irq routing. > > Note: I have no s390 machine so the s390 patch has not been tested. > > Changelog: > ---------- > v3: > - squash setup empty routing function and use of that into one commit > - drop the comment in s390 part > > v2: > - setup empty irq routing in kvm_create_vm > - don't setup irq routing in x86 KVM_CAP_SPLIT_IRQCHIP > - don't setup irq routing in s390 KVM_CREATE_IRQCHIP > > v1: https://lore.kernel.org/kvm/20240112091128.3868059-1-foxywang@tencent.com/ > > Yi Wang (3): > KVM: setup empty irq routing when create vm > KVM: x86: don't setup empty irq routing when KVM_CAP_SPLIT_IRQCHIP > KVM: s390: don't setup dummy routing when KVM_CREATE_IRQCHIP > > arch/s390/kvm/kvm-s390.c | 9 +-------- > arch/x86/kvm/irq.h | 1 - > arch/x86/kvm/irq_comm.c | 5 ----- > arch/x86/kvm/x86.c | 3 --- > include/linux/kvm_host.h | 1 + > virt/kvm/irqchip.c | 19 +++++++++++++++++++ > virt/kvm/kvm_main.c | 4 ++++ > 7 files changed, 25 insertions(+), 17 deletions(-) > > -- > 2.39.3 >
Hi Yi, On 1/24/24 03:34, Yi Wang wrote: > From: Yi Wang <foxywang@tencent.com> > > We found that it may cost more than 20 milliseconds very accidentally > to enable cap of KVM_CAP_SPLIT_IRQCHIP on a host which has many vms > already. Would you mind explaining the reason that the *number of VMs* matters, as KVM_CAP_SPLIT_IRQCHIP is a per-VM cap? Or it meant it is more likely to have some VM workload impacted by the synchronize_srcu_expedited() as in prior discussion? https://lore.kernel.org/kvm/CAN35MuSkQf0XmBZ5ZXGhcpUCGD-kKoyTv9G7ya4QVD1xiqOxLg@mail.gmail.com/ Thank you very much! Dongli Zhang > > The reason is that when vmm(qemu/CloudHypervisor) invokes > KVM_CAP_SPLIT_IRQCHIP kvm will call synchronize_srcu_expedited() and > might_sleep and kworker of srcu may cost some delay during this period. > One way makes sence is setup empty irq routing when creating vm and > so that x86/s390 don't need to setup empty/dummy irq routing. > > Note: I have no s390 machine so the s390 patch has not been tested. > > Changelog: > ---------- > v3: > - squash setup empty routing function and use of that into one commit > - drop the comment in s390 part > > v2: > - setup empty irq routing in kvm_create_vm > - don't setup irq routing in x86 KVM_CAP_SPLIT_IRQCHIP > - don't setup irq routing in s390 KVM_CREATE_IRQCHIP > > v1: https://urldefense.com/v3/__https://lore.kernel.org/kvm/20240112091128.3868059-1-foxywang@tencent.com/__;!!ACWV5N9M2RV99hQ!LjwKfBaGVl3u1l9YQSskg_1RU6278h2-fYnYLsoihF9i43aq73eIDqolGzOmeRvO8UlPreQHLqXEL1bAuw$ > > Yi Wang (3): > KVM: setup empty irq routing when create vm > KVM: x86: don't setup empty irq routing when KVM_CAP_SPLIT_IRQCHIP > KVM: s390: don't setup dummy routing when KVM_CREATE_IRQCHIP > > arch/s390/kvm/kvm-s390.c | 9 +-------- > arch/x86/kvm/irq.h | 1 - > arch/x86/kvm/irq_comm.c | 5 ----- > arch/x86/kvm/x86.c | 3 --- > include/linux/kvm_host.h | 1 + > virt/kvm/irqchip.c | 19 +++++++++++++++++++ > virt/kvm/kvm_main.c | 4 ++++ > 7 files changed, 25 insertions(+), 17 deletions(-) >
Hi Dongli, Thanks for the reply and Happy Spring Festival to all :) On Fri, Feb 9, 2024 at 5:00 PM Dongli Zhang <dongli.zhang@oracle.com> wrote: > > Hi Yi, > > On 1/24/24 03:34, Yi Wang wrote: > > From: Yi Wang <foxywang@tencent.com> > > > > We found that it may cost more than 20 milliseconds very accidentally > > to enable cap of KVM_CAP_SPLIT_IRQCHIP on a host which has many vms > > already. > > Would you mind explaining the reason that the *number of VMs* matters, as > KVM_CAP_SPLIT_IRQCHIP is a per-VM cap? > > Or it meant it is more likely to have some VM workload impacted by the > synchronize_srcu_expedited() as in prior discussion? > > https://lore.kernel.org/kvm/CAN35MuSkQf0XmBZ5ZXGhcpUCGD-kKoyTv9G7ya4QVD1xiqOxLg@mail.gmail.com/ > The actual reason is might_sleep() and the kworker in synchronize_srcu_expedited(), which may cause some delay when there are pretty many threads in the host, so "number of VMs" is just one of the situations which can trigger the issue :) > Thank you very much! > > Dongli Zhang > > > > > The reason is that when vmm(qemu/CloudHypervisor) invokes > > KVM_CAP_SPLIT_IRQCHIP kvm will call synchronize_srcu_expedited() and > > might_sleep and kworker of srcu may cost some delay during this period. > > One way makes sence is setup empty irq routing when creating vm and > > so that x86/s390 don't need to setup empty/dummy irq routing. > > > > Note: I have no s390 machine so the s390 patch has not been tested. > > > > Changelog: > > ---------- > > v3: > > - squash setup empty routing function and use of that into one commit > > - drop the comment in s390 part > > > > v2: > > - setup empty irq routing in kvm_create_vm > > - don't setup irq routing in x86 KVM_CAP_SPLIT_IRQCHIP > > - don't setup irq routing in s390 KVM_CREATE_IRQCHIP > > > > v1: https://urldefense.com/v3/__https://lore.kernel.org/kvm/20240112091128.3868059-1-foxywang@tencent.com/__;!!ACWV5N9M2RV99hQ!LjwKfBaGVl3u1l9YQSskg_1RU6278h2-fYnYLsoihF9i43aq73eIDqolGzOmeRvO8UlPreQHLqXEL1bAuw$ > > > > Yi Wang (3): > > KVM: setup empty irq routing when create vm > > KVM: x86: don't setup empty irq routing when KVM_CAP_SPLIT_IRQCHIP > > KVM: s390: don't setup dummy routing when KVM_CREATE_IRQCHIP > > > > arch/s390/kvm/kvm-s390.c | 9 +-------- > > arch/x86/kvm/irq.h | 1 - > > arch/x86/kvm/irq_comm.c | 5 ----- > > arch/x86/kvm/x86.c | 3 --- > > include/linux/kvm_host.h | 1 + > > virt/kvm/irqchip.c | 19 +++++++++++++++++++ > > virt/kvm/kvm_main.c | 4 ++++ > > 7 files changed, 25 insertions(+), 17 deletions(-) > >
From: Yi Wang <foxywang@tencent.com> We found that it may cost more than 20 milliseconds very accidentally to enable cap of KVM_CAP_SPLIT_IRQCHIP on a host which has many vms already. The reason is that when vmm(qemu/CloudHypervisor) invokes KVM_CAP_SPLIT_IRQCHIP kvm will call synchronize_srcu_expedited() and might_sleep and kworker of srcu may cost some delay during this period. One way makes sence is setup empty irq routing when creating vm and so that x86/s390 don't need to setup empty/dummy irq routing. Note: I have no s390 machine so the s390 patch has not been tested. Changelog: ---------- v3: - squash setup empty routing function and use of that into one commit - drop the comment in s390 part v2: - setup empty irq routing in kvm_create_vm - don't setup irq routing in x86 KVM_CAP_SPLIT_IRQCHIP - don't setup irq routing in s390 KVM_CREATE_IRQCHIP v1: https://lore.kernel.org/kvm/20240112091128.3868059-1-foxywang@tencent.com/ Yi Wang (3): KVM: setup empty irq routing when create vm KVM: x86: don't setup empty irq routing when KVM_CAP_SPLIT_IRQCHIP KVM: s390: don't setup dummy routing when KVM_CREATE_IRQCHIP arch/s390/kvm/kvm-s390.c | 9 +-------- arch/x86/kvm/irq.h | 1 - arch/x86/kvm/irq_comm.c | 5 ----- arch/x86/kvm/x86.c | 3 --- include/linux/kvm_host.h | 1 + virt/kvm/irqchip.c | 19 +++++++++++++++++++ virt/kvm/kvm_main.c | 4 ++++ 7 files changed, 25 insertions(+), 17 deletions(-)