| Message ID | 20090701133007.GC27539@redhat.com (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
On Wed, 2009-07-01 at 06:30 -0700, Gleb Natapov wrote: > KVM would like to provide x2APIC interface to a guest without emulating > interrupt remapping device. The reason KVM prefers guest to use x2APIC > is that x2APIC interface is better virtualizable and provides better > performance than mmio xAPIC interface: > > - msr exits are faster than mmio (no page table walk, emulation) > - no need to read back ICR to look at the busy bit > - one 64 bit ICR write instead of two 32 bit writes > - shared code with the Hyper-V paravirt interface > > Included patch changes x2APIC enabling logic to enable it even if IR > initialization failed, but kernel runs under KVM and no apic id is > greater than 255 (if there is one spec requires BIOS to move to x2apic > mode before starting an OS). > > Signed-off-by: Gleb Natapov <gleb@redhat.com> Acked-by: Suresh Siddha <suresh.b.siddha@intel.com> -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
* Suresh Siddha <suresh.b.siddha@intel.com> wrote: > On Wed, 2009-07-01 at 06:30 -0700, Gleb Natapov wrote: > > KVM would like to provide x2APIC interface to a guest without emulating > > interrupt remapping device. The reason KVM prefers guest to use x2APIC > > is that x2APIC interface is better virtualizable and provides better > > performance than mmio xAPIC interface: > > > > - msr exits are faster than mmio (no page table walk, emulation) > > - no need to read back ICR to look at the busy bit > > - one 64 bit ICR write instead of two 32 bit writes > > - shared code with the Hyper-V paravirt interface > > > > Included patch changes x2APIC enabling logic to enable it even if IR > > initialization failed, but kernel runs under KVM and no apic id is > > greater than 255 (if there is one spec requires BIOS to move to x2apic > > mode before starting an OS). > > > > Signed-off-by: Gleb Natapov <gleb@redhat.com> > > Acked-by: Suresh Siddha <suresh.b.siddha@intel.com> Now, since this affects core x86 APIC code non-trivially so should submitted to and go via the x86 tree. (Can prepare a special branch with just this change if KVM tree wants/needs to pull it before v2.6.32.) Ingo -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Ingo Molnar <mingo@elte.hu> writes: > * Suresh Siddha <suresh.b.siddha@intel.com> wrote: > >> On Wed, 2009-07-01 at 06:30 -0700, Gleb Natapov wrote: >> > KVM would like to provide x2APIC interface to a guest without emulating >> > interrupt remapping device. The reason KVM prefers guest to use x2APIC >> > is that x2APIC interface is better virtualizable and provides better >> > performance than mmio xAPIC interface: >> > >> > - msr exits are faster than mmio (no page table walk, emulation) >> > - no need to read back ICR to look at the busy bit >> > - one 64 bit ICR write instead of two 32 bit writes >> > - shared code with the Hyper-V paravirt interface >> > >> > Included patch changes x2APIC enabling logic to enable it even if IR >> > initialization failed, but kernel runs under KVM and no apic id is >> > greater than 255 (if there is one spec requires BIOS to move to x2apic >> > mode before starting an OS). >> > >> > Signed-off-by: Gleb Natapov <gleb@redhat.com> >> >> Acked-by: Suresh Siddha <suresh.b.siddha@intel.com> > > Now, since this affects core x86 APIC code non-trivially so should > submitted to and go via the x86 tree. (Can prepare a special branch > with just this change if KVM tree wants/needs to pull it before > v2.6.32.) Please don't separate the x2apic code from the dmar code for this reason. Supporting hotplug cpus with ioapics is torture. Eric -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Sat, Jul 04, 2009 at 02:35:30AM -0700, Eric W. Biederman wrote: > Ingo Molnar <mingo@elte.hu> writes: > > > * Suresh Siddha <suresh.b.siddha@intel.com> wrote: > > > >> On Wed, 2009-07-01 at 06:30 -0700, Gleb Natapov wrote: > >> > KVM would like to provide x2APIC interface to a guest without emulating > >> > interrupt remapping device. The reason KVM prefers guest to use x2APIC > >> > is that x2APIC interface is better virtualizable and provides better > >> > performance than mmio xAPIC interface: > >> > > >> > - msr exits are faster than mmio (no page table walk, emulation) > >> > - no need to read back ICR to look at the busy bit > >> > - one 64 bit ICR write instead of two 32 bit writes > >> > - shared code with the Hyper-V paravirt interface > >> > > >> > Included patch changes x2APIC enabling logic to enable it even if IR > >> > initialization failed, but kernel runs under KVM and no apic id is > >> > greater than 255 (if there is one spec requires BIOS to move to x2apic > >> > mode before starting an OS). > >> > > >> > Signed-off-by: Gleb Natapov <gleb@redhat.com> > >> > >> Acked-by: Suresh Siddha <suresh.b.siddha@intel.com> > > > > Now, since this affects core x86 APIC code non-trivially so should > > submitted to and go via the x86 tree. (Can prepare a special branch > > with just this change if KVM tree wants/needs to pull it before > > v2.6.32.) > > Please don't separate the x2apic code from the dmar code for this > reason. > > Supporting hotplug cpus with ioapics is torture. > What is the connection between this patch and cpu hotplug? -- Gleb. -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Gleb Natapov <gleb@redhat.com> writes: > On Sat, Jul 04, 2009 at 02:35:30AM -0700, Eric W. Biederman wrote: >> Ingo Molnar <mingo@elte.hu> writes: >> >> > * Suresh Siddha <suresh.b.siddha@intel.com> wrote: >> > >> >> On Wed, 2009-07-01 at 06:30 -0700, Gleb Natapov wrote: >> >> > KVM would like to provide x2APIC interface to a guest without emulating >> >> > interrupt remapping device. The reason KVM prefers guest to use x2APIC >> >> > is that x2APIC interface is better virtualizable and provides better >> >> > performance than mmio xAPIC interface: >> >> > >> >> > - msr exits are faster than mmio (no page table walk, emulation) >> >> > - no need to read back ICR to look at the busy bit >> >> > - one 64 bit ICR write instead of two 32 bit writes >> >> > - shared code with the Hyper-V paravirt interface >> >> > >> >> > Included patch changes x2APIC enabling logic to enable it even if IR >> >> > initialization failed, but kernel runs under KVM and no apic id is >> >> > greater than 255 (if there is one spec requires BIOS to move to x2apic >> >> > mode before starting an OS). >> >> > >> >> > Signed-off-by: Gleb Natapov <gleb@redhat.com> >> >> >> >> Acked-by: Suresh Siddha <suresh.b.siddha@intel.com> >> > >> > Now, since this affects core x86 APIC code non-trivially so should >> > submitted to and go via the x86 tree. (Can prepare a special branch >> > with just this change if KVM tree wants/needs to pull it before >> > v2.6.32.) >> >> Please don't separate the x2apic code from the dmar code for this >> reason. >> >> Supporting hotplug cpus with ioapics is torture. >> > What is the connection between this patch and cpu hotplug? When I asked if cpu hotplug was a supported and more or less common feature of kvm I was told it was. Good cpu hotplug today means supporting interrupt remapping. (The code you are disabling for kvm). Therefore I don't see the point of supporting one without the other. Especially if we don't have a case where on real hardware we need to split the support. Eric -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On 07/03/2009 11:29 AM, Ingo Molnar wrote: > Now, since this affects core x86 APIC code non-trivially so should > submitted to and go via the x86 tree. (Can prepare a special branch > with just this change if KVM tree wants/needs to pull it before > v2.6.32.) > The kvm tree won't depend on this change. I'm happy with it going into 2.6.32 via tip.git.
On Sat, Jul 04, 2009 at 07:33:39AM -0700, Eric W. Biederman wrote: > Gleb Natapov <gleb@redhat.com> writes: > > > On Sat, Jul 04, 2009 at 02:35:30AM -0700, Eric W. Biederman wrote: > >> Ingo Molnar <mingo@elte.hu> writes: > >> > >> > * Suresh Siddha <suresh.b.siddha@intel.com> wrote: > >> > > >> >> On Wed, 2009-07-01 at 06:30 -0700, Gleb Natapov wrote: > >> >> > KVM would like to provide x2APIC interface to a guest without emulating > >> >> > interrupt remapping device. The reason KVM prefers guest to use x2APIC > >> >> > is that x2APIC interface is better virtualizable and provides better > >> >> > performance than mmio xAPIC interface: > >> >> > > >> >> > - msr exits are faster than mmio (no page table walk, emulation) > >> >> > - no need to read back ICR to look at the busy bit > >> >> > - one 64 bit ICR write instead of two 32 bit writes > >> >> > - shared code with the Hyper-V paravirt interface > >> >> > > >> >> > Included patch changes x2APIC enabling logic to enable it even if IR > >> >> > initialization failed, but kernel runs under KVM and no apic id is > >> >> > greater than 255 (if there is one spec requires BIOS to move to x2apic > >> >> > mode before starting an OS). > >> >> > > >> >> > Signed-off-by: Gleb Natapov <gleb@redhat.com> > >> >> > >> >> Acked-by: Suresh Siddha <suresh.b.siddha@intel.com> > >> > > >> > Now, since this affects core x86 APIC code non-trivially so should > >> > submitted to and go via the x86 tree. (Can prepare a special branch > >> > with just this change if KVM tree wants/needs to pull it before > >> > v2.6.32.) > >> > >> Please don't separate the x2apic code from the dmar code for this > >> reason. > >> > >> Supporting hotplug cpus with ioapics is torture. > >> > > What is the connection between this patch and cpu hotplug? > > When I asked if cpu hotplug was a supported and more or > less common feature of kvm I was told it was. > It is supported in a sense that cpus can be created dynamically and DSDT has objects to inform OS that a cpu state changed. I don't know how common it is. I surely don't use it daily. But 99% of the real work is done by a guest. > Good cpu hotplug today means supporting interrupt remapping. > (The code you are disabling for kvm). From you previous mails I understand that the problem is no with cpu hotplug, but with cpu offlining and this is the guest feature that cannot be controlled by KVM in any way. If you think that Linux should not support cpu offlining with ioapics fine, send a patch to disable it and cpu hot-unplug will not be supported by KVM with Linux guests, but note that KVM ioapic has no problems that you've mentioned and HW makers that provide x86 hardware may be using more sophisticated ioapics that you've tested (I don't know never saw such hardware). > > Therefore I don't see the point of supporting one without the other. x2apic provide us with other benefits as commit message explains, and doesn't add any problems that we don't have now already. > Especially if we don't have a case where on real hardware we need to > split the support. > -- Gleb. -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Gleb Natapov <gleb@redhat.com> writes: >> Therefore I don't see the point of supporting one without the other. > x2apic provide us with other benefits as commit message explains, and > doesn't add any problems that we don't have now already. If this code has a legitimate place on real hardware I am all for it. If this is just a hack to make virtualization faster I don't like the extra code paths in the middle core architecture code. That will be a support burden for the foreseeable future. More code to test etc. Quickly skimming the patch it just appears to stir a mess. Plus it adds weird paravirtualization checks, ??? If we are going to have a special code path for virtual hardware can we do it right and have something nice to use that makes life simpler? For what we want to do with ioapics they suck and are really not suitable. The only thing that recommends them is that they are standard. But you are deviating from the standard so what is the point. Eric -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On 07/05/2009 03:22 AM, Eric W. Biederman wrote: > Gleb Natapov<gleb@redhat.com> writes: > > >>> Therefore I don't see the point of supporting one without the other. >>> >> x2apic provide us with other benefits as commit message explains, and >> doesn't add any problems that we don't have now already. >> > > If this code has a legitimate place on real hardware I am all for it. > As I understood it, x2apic without interrupt remapping will work but is not a validated configuration. Interrupt remapping is only necessary if you have > 255 hardware threads + ioapics. The features are logically separate and are only tied together by the vendor's validation practices. > If this is just a hack to make virtualization faster I don't like the > extra code paths in the middle core architecture code. That will > be a support burden for the foreseeable future. More code to > test etc. > There aren't any extra code paths. The patch separates a long function into two smaller ones that each do one thing, and adds a check for kvm. Maybe it should be split into two to makes that clear. The first patch simplifies the code, the second adds a kvm check. > Quickly skimming the patch it just appears to stir a mess. > Plus it adds weird paravirtualization checks, ??? > It adds exactly one "weird paravirtualization check ???", then one described in the patch description. > If we are going to have a special code path for virtual hardware > can we do it right and have something nice to use that makes life > simpler? You mean, instead of adding one check in an initialization code path, create a new irqchip, a way of describing the topology to the guest, support code in kvm (as host)? > For what we want to do with ioapics they suck and are > really not suitable. The only thing that recommends them is that > they are standard. But you are deviating from the standard so > what is the point. > All of the code continues to work.
diff --git a/arch/x86/kernel/apic/apic.c b/arch/x86/kernel/apic/apic.c index 8c7c042..db0c07f 100644 --- a/arch/x86/kernel/apic/apic.c +++ b/arch/x86/kernel/apic/apic.c @@ -49,6 +49,7 @@ #include <asm/mtrr.h> #include <asm/smp.h> #include <asm/mce.h> +#include <asm/kvm_para.h> unsigned int num_processors; @@ -1363,52 +1364,74 @@ void enable_x2apic(void) } #endif /* CONFIG_X86_X2APIC */ -void __init enable_IR_x2apic(void) +int __init enable_IR(void) { #ifdef CONFIG_INTR_REMAP int ret; - unsigned long flags; - struct IO_APIC_route_entry **ioapic_entries = NULL; ret = dmar_table_init(); if (ret) { pr_debug("dmar_table_init() failed with %d:\n", ret); - goto ir_failed; + return 0; } if (!intr_remapping_supported()) { pr_debug("intr-remapping not supported\n"); - goto ir_failed; + return 0; } - if (!x2apic_preenabled && skip_ioapic_setup) { pr_info("Skipped enabling intr-remap because of skipping " "io-apic setup\n"); - return; + return 0; } + if (enable_intr_remapping(x2apic_supported())) + return 0; + + pr_info("Enabled Interrupt-remapping\n"); + + return 1; + +#endif + return 0; +} + +void __init enable_IR_x2apic(void) +{ + unsigned long flags; + struct IO_APIC_route_entry **ioapic_entries = NULL; + int ret, x2apic_enabled = 0; + ioapic_entries = alloc_ioapic_entries(); if (!ioapic_entries) { - pr_info("Allocate ioapic_entries failed: %d\n", ret); - goto end; + pr_info("Allocate ioapic_entries failed\n"); + goto out; } ret = save_IO_APIC_setup(ioapic_entries); if (ret) { pr_info("Saving IO-APIC state failed: %d\n", ret); - goto end; + goto out; } local_irq_save(flags); - mask_IO_APIC_setup(ioapic_entries); mask_8259A(); + mask_IO_APIC_setup(ioapic_entries); - ret = enable_intr_remapping(x2apic_supported()); - if (ret) - goto end_restore; + ret = enable_IR(); + if (!ret) { + /* IR is required if x2apic is enabled by BIOS even + * when running in kvm since this indicates present + * of APIC ID > 255 */ + if (max_physical_apicid > 255 || !kvm_para_available()) + goto nox2apic; + /* without IR all CPUs can be addressed by IOAPIC/MSI + * only in physical mode */ + x2apic_phys = 1; + } - pr_info("Enabled Interrupt-remapping\n"); + x2apic_enabled = 1; if (x2apic_supported() && !x2apic_mode) { x2apic_mode = 1; @@ -1416,41 +1439,30 @@ void __init enable_IR_x2apic(void) pr_info("Enabled x2apic\n"); } -end_restore: - if (ret) - /* - * IR enabling failed - */ +nox2apic: + if (!ret) /* IR enabling failed */ restore_IO_APIC_setup(ioapic_entries); - unmask_8259A(); local_irq_restore(flags); -end: +out: if (ioapic_entries) free_ioapic_entries(ioapic_entries); - if (!ret) + if (x2apic_enabled) return; -ir_failed: - if (x2apic_preenabled) + if (x2apic_preenabled) { +#ifdef CONFIG_INTR_REMAP panic("x2apic enabled by bios. But IR enabling failed"); - else if (cpu_has_x2apic) - pr_info("Not enabling x2apic,Intr-remapping\n"); #else - if (!cpu_has_x2apic) - return; - - if (x2apic_preenabled) panic("x2apic enabled prior OS handover," - " enable CONFIG_X86_X2APIC, CONFIG_INTR_REMAP"); + " enable CONFIG_X86_X2APIC, CONFIG_INTR_REMAP"); #endif - - return; + } else if (cpu_has_x2apic) + pr_info("Not enabling x2apic,Intr-remapping\n"); } - #ifdef CONFIG_X86_64 /* * Detect and enable local APICs on non-SMP boards. diff --git a/arch/x86/kernel/apic/probe_64.c b/arch/x86/kernel/apic/probe_64.c index bc3e880..f3b1037 100644 --- a/arch/x86/kernel/apic/probe_64.c +++ b/arch/x86/kernel/apic/probe_64.c @@ -50,11 +50,11 @@ static struct apic *apic_probe[] __initdata = { void __init default_setup_apic_routing(void) { #ifdef CONFIG_X86_X2APIC - if (x2apic_mode && (apic != &apic_x2apic_phys && + if (x2apic_mode #ifdef CONFIG_X86_UV - apic != &apic_x2apic_uv_x && + && apic != &apic_x2apic_uv_x #endif - apic != &apic_x2apic_cluster)) { + ) { if (x2apic_phys) apic = &apic_x2apic_phys; else
KVM would like to provide x2APIC interface to a guest without emulating interrupt remapping device. The reason KVM prefers guest to use x2APIC is that x2APIC interface is better virtualizable and provides better performance than mmio xAPIC interface: - msr exits are faster than mmio (no page table walk, emulation) - no need to read back ICR to look at the busy bit - one 64 bit ICR write instead of two 32 bit writes - shared code with the Hyper-V paravirt interface Included patch changes x2APIC enabling logic to enable it even if IR initialization failed, but kernel runs under KVM and no apic id is greater than 255 (if there is one spec requires BIOS to move to x2apic mode before starting an OS). Signed-off-by: Gleb Natapov <gleb@redhat.com> --- This version does not rely on x2apic_preenabled to determine max APIC id since this logic will break after kexec. Use max_physical_apicid instead. Also allow to override apic ops after call to enable_IR_x2apic() since we may decide to use physical mode instead of cluster. -- Gleb. -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html