| Message ID | 1568401242-260374-1-git-send-email-suravee.suthikulpanit@amd.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | kvm: x86: Support AMD SVM AVIC w/ in-kernel irqchip mode | expand |
Ping: Hi All, Are there other concerns or suggestions for this series? Thanks, Suravee On 9/13/19 2:00 PM, Suthikulpanit, Suravee wrote: > The 'commit 67034bb9dd5e ("KVM: SVM: Add irqchip_split() checks before > enabling AVIC")' was introduced to fix miscellaneous boot-hang issues > when enable AVIC. This is mainly due to AVIC hardware doest not #vmexit > on write to LAPIC EOI register resulting in-kernel PIC and IOAPIC to > wait and do not inject new interrupts (e.g. PIT, RTC). > > This limits AVIC to only work with kernel_irqchip=split mode, which is > not currently enabled by default, and also required user-space to > support split irqchip model, which might not be the case. > > The goal of this series is to enable AVIC to work in both irqchip modes, > by allowing AVIC to be deactivated temporarily during runtime, and fallback > to legacy interrupt injection mode (w/ vINTR and interrupt windows) > when needed, and then re-enabled subsequently. > > Similar approach is also used to handle Hyper-V SynIC in the > 'commit 5c919412fe61 ("kvm/x86: Hyper-V synthetic interrupt controller")', > where APICv is permanently disabled at runtime (currently broken for > AVIC, and fixed by this series). > > This series contains three parts: > * Part 1: patch 1-2 > Introduce APICv state enum and logic for keeping track of the state > for each vm. > > * Part 2: patch 3-11 > Add support for activate/deactivate APICv at runtime > > * Part 3: patch 12-16: > Provide workaround for AVIC EOI and allow enable AVIC w/ > kernel_irqchip=on > > Pre-requisite Patch: > * commit b9c6ff94e43a ("iommu/amd: Re-factor guest virtual APIC > (de-)activation code") > (https://git.kernel.org/pub/scm/linux/kernel/git/joro/iommu.git/commit/ > ?h=next&id=b9c6ff94e43a0ee053e0c1d983fba1ac4953b762) > > This series has been tested against v5.3-rc5 as following: > * Booting Linux and Windows Server 2019 VMs upto 240 vcpus > and FreeBSD upto 128 vcpus w/ qemu option "kernel-irqchip=on" > and "-no-hpet". > * Pass-through Intel 10GbE NIC and run netperf in the VM. > > Changes from V2: (https://lkml.org/lkml/2019/8/15/672) > * Rebase to v5.3-rc5 > * Misc changes recommended by Alex and Vitaly. > * Rename APICV_DEACTIVATED to APICV_SUSPENDED > * Disable AVIC when guest booting w/ SVM support since AVIC > does not currently support guest w/ nested virt. > * Add tracepoint for APICV activate/deactivate request. (per Alex) > * Consolidate changes for handling EOI for kvm PIT emulation and > IOAPIC RTC handling in V2 into ioapic_lazy_update_eoi() in > patch 17/18 of v3 serie. > * Remove patches for providing per-vm apicv_state debug information. > > Changes from V1: (https://lkml.org/lkml/2019/3/22/1042) > * Introduce APICv state enumeration > * Introduce KVM debugfs for APICv state > * Add synchronization logic for APICv state to prevent potential > race condition (per Jan's suggestion) > * Add support for activate/deactivate posted interrupt > (per Jan's suggestion) > * Remove callback functions for handling APIC ID, DFR and LDR update > (per Paolo's suggestion) > * Add workaround for handling EOI for in-kernel PIT and IOAPIC. > > Suravee Suthikulpanit (16): > kvm: x86: Modify kvm_x86_ops.get_enable_apicv() to use struct kvm > parameter > kvm: x86: Introduce KVM APICv state > kvm: lapic: Introduce APICv update helper function > kvm: x86: Add support for activate/de-activate APICv at runtime > kvm: x86: Add APICv activate/deactivate request trace points > kvm: x86: svm: Add support to activate/deactivate posted interrupts > svm: Add support for setup/destroy virutal APIC backing page for AVIC > svm: Add support for activate/deactivate AVIC at runtime > kvm: x86: hyperv: Use APICv deactivate request interface > svm: Disable AVIC when launching guest with SVM support > svm: Temporary deactivate AVIC during ExtINT handling > kvm: x86: Introduce struct kvm_x86_ops.apicv_eoi_accelerate > kvm: lapic: Clean up APIC predefined macros > kvm: ioapic: Refactor kvm_ioapic_update_eoi() > kvm: x86: ioapic: Lazy update IOAPIC EOI > svm: Allow AVIC with in-kernel irqchip mode > > arch/x86/include/asm/kvm_host.h | 28 +++++- > arch/x86/kvm/hyperv.c | 12 ++- > arch/x86/kvm/ioapic.c | 149 +++++++++++++++++++----------- > arch/x86/kvm/lapic.c | 35 ++++--- > arch/x86/kvm/lapic.h | 2 + > arch/x86/kvm/svm.c | 198 +++++++++++++++++++++++++++++++++++++--- > arch/x86/kvm/trace.h | 30 ++++++ > arch/x86/kvm/vmx/vmx.c | 2 +- > arch/x86/kvm/x86.c | 136 ++++++++++++++++++++++++++- > 9 files changed, 506 insertions(+), 86 deletions(-) >
On 08/10/19 20:44, Suthikulpanit, Suravee wrote: > Ping: > > Hi All, > > Are there other concerns or suggestions for this series? I've now reviewed it---sorry for the delay. Paolo
The 'commit 67034bb9dd5e ("KVM: SVM: Add irqchip_split() checks before enabling AVIC")' was introduced to fix miscellaneous boot-hang issues when enable AVIC. This is mainly due to AVIC hardware doest not #vmexit on write to LAPIC EOI register resulting in-kernel PIC and IOAPIC to wait and do not inject new interrupts (e.g. PIT, RTC). This limits AVIC to only work with kernel_irqchip=split mode, which is not currently enabled by default, and also required user-space to support split irqchip model, which might not be the case. The goal of this series is to enable AVIC to work in both irqchip modes, by allowing AVIC to be deactivated temporarily during runtime, and fallback to legacy interrupt injection mode (w/ vINTR and interrupt windows) when needed, and then re-enabled subsequently. Similar approach is also used to handle Hyper-V SynIC in the 'commit 5c919412fe61 ("kvm/x86: Hyper-V synthetic interrupt controller")', where APICv is permanently disabled at runtime (currently broken for AVIC, and fixed by this series). This series contains three parts: * Part 1: patch 1-2 Introduce APICv state enum and logic for keeping track of the state for each vm. * Part 2: patch 3-11 Add support for activate/deactivate APICv at runtime * Part 3: patch 12-16: Provide workaround for AVIC EOI and allow enable AVIC w/ kernel_irqchip=on Pre-requisite Patch: * commit b9c6ff94e43a ("iommu/amd: Re-factor guest virtual APIC (de-)activation code") (https://git.kernel.org/pub/scm/linux/kernel/git/joro/iommu.git/commit/ ?h=next&id=b9c6ff94e43a0ee053e0c1d983fba1ac4953b762) This series has been tested against v5.3-rc5 as following: * Booting Linux and Windows Server 2019 VMs upto 240 vcpus and FreeBSD upto 128 vcpus w/ qemu option "kernel-irqchip=on" and "-no-hpet". * Pass-through Intel 10GbE NIC and run netperf in the VM. Changes from V2: (https://lkml.org/lkml/2019/8/15/672) * Rebase to v5.3-rc5 * Misc changes recommended by Alex and Vitaly. * Rename APICV_DEACTIVATED to APICV_SUSPENDED * Disable AVIC when guest booting w/ SVM support since AVIC does not currently support guest w/ nested virt. * Add tracepoint for APICV activate/deactivate request. (per Alex) * Consolidate changes for handling EOI for kvm PIT emulation and IOAPIC RTC handling in V2 into ioapic_lazy_update_eoi() in patch 17/18 of v3 serie. * Remove patches for providing per-vm apicv_state debug information. Changes from V1: (https://lkml.org/lkml/2019/3/22/1042) * Introduce APICv state enumeration * Introduce KVM debugfs for APICv state * Add synchronization logic for APICv state to prevent potential race condition (per Jan's suggestion) * Add support for activate/deactivate posted interrupt (per Jan's suggestion) * Remove callback functions for handling APIC ID, DFR and LDR update (per Paolo's suggestion) * Add workaround for handling EOI for in-kernel PIT and IOAPIC. Suravee Suthikulpanit (16): kvm: x86: Modify kvm_x86_ops.get_enable_apicv() to use struct kvm parameter kvm: x86: Introduce KVM APICv state kvm: lapic: Introduce APICv update helper function kvm: x86: Add support for activate/de-activate APICv at runtime kvm: x86: Add APICv activate/deactivate request trace points kvm: x86: svm: Add support to activate/deactivate posted interrupts svm: Add support for setup/destroy virutal APIC backing page for AVIC svm: Add support for activate/deactivate AVIC at runtime kvm: x86: hyperv: Use APICv deactivate request interface svm: Disable AVIC when launching guest with SVM support svm: Temporary deactivate AVIC during ExtINT handling kvm: x86: Introduce struct kvm_x86_ops.apicv_eoi_accelerate kvm: lapic: Clean up APIC predefined macros kvm: ioapic: Refactor kvm_ioapic_update_eoi() kvm: x86: ioapic: Lazy update IOAPIC EOI svm: Allow AVIC with in-kernel irqchip mode arch/x86/include/asm/kvm_host.h | 28 +++++- arch/x86/kvm/hyperv.c | 12 ++- arch/x86/kvm/ioapic.c | 149 +++++++++++++++++++----------- arch/x86/kvm/lapic.c | 35 ++++--- arch/x86/kvm/lapic.h | 2 + arch/x86/kvm/svm.c | 198 +++++++++++++++++++++++++++++++++++++--- arch/x86/kvm/trace.h | 30 ++++++ arch/x86/kvm/vmx/vmx.c | 2 +- arch/x86/kvm/x86.c | 136 ++++++++++++++++++++++++++- 9 files changed, 506 insertions(+), 86 deletions(-)