| Message ID | 1439473570-13763-3-git-send-email-rkrcmar@redhat.com (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
On 13/08/2015 15:46, Radim Kr?má? wrote: > 1) IOAPIC inject a vector from i8254 > 2) guest reconfigures that vector's VCPU and therefore eoi_exit_bitmap > on original VCPU gets cleared > 3) guest's handler for the vector does EOI > 4) KVM's EOI handler doesn't pass that vector to IOAPIC because it is > not in that VCPU's eoi_exit_bitmap > 5) i8254 stops working > > This creates an unwanted situation if the vector is reused by a > non-IOAPIC source, but I think it is so rare that we don't want to make > the solution more sophisticated. What happens if the vector is changed in step 2? __kvm_ioapic_update_eoi won't match the redirection table entry. How do you reproduce the bug? Paolo -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
2015-08-13 16:53+0200, Paolo Bonzini: > On 13/08/2015 15:46, Radim Kr?má? wrote: >> 1) IOAPIC inject a vector from i8254 >> 2) guest reconfigures that vector's VCPU and therefore eoi_exit_bitmap >> on original VCPU gets cleared >> 3) guest's handler for the vector does EOI >> 4) KVM's EOI handler doesn't pass that vector to IOAPIC because it is >> not in that VCPU's eoi_exit_bitmap >> 5) i8254 stops working >> >> This creates an unwanted situation if the vector is reused by a >> non-IOAPIC source, but I think it is so rare that we don't want to make >> the solution more sophisticated. > > What happens if the vector is changed in step 2? > __kvm_ioapic_update_eoi won't match the redirection table entry. Yes, the EOI is going to be ignored. (With APICv, VMX won't even exit.) In the patch, I dissmissed it as "shouldn't happen in the wild" because we've always had the vector-change bug :) (Unlike the destination-change one, which was APICv-only before recent changes.) A simple solution to the vector-change would have a list of one-time fixups (vector, *ioapic) and hooks in ioapic reconfig, scan and EOI. A complex solution would replace ioapic scanning with an array of list of ioapics (it needs to be a list or small array because vectors can be shared). An ioapic would be added to list[vector] on reconfig and removed on reconfig unless an edge fixup was needed, then it would last til next EOI (I guess we won't need to consider vector in IRR and ISR). Callbacks would update the eoi_exit_bitmap on relevant changes. I considered doing the complex one, but then it occured to me that we want the destination-change fixed in stable as APICv machines are starting to get used and people might migrate old guests on them. > How do you reproduce the bug? I run rhel4 (2.6.9) kernel on 2 VCPUs and frequently alternate smp_affinity of "timer". The bug is hit within seconds. -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On 14/08/2015 10:38, Radim Kr?má? wrote: >> How do you reproduce the bug? > I run rhel4 (2.6.9) kernel on 2 VCPUs and frequently alternate > smp_affinity of "timer". The bug is hit within seconds. Nice, I'll try to make a unit test for it on the plane. :) Paolo -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On 13/08/2015 15:46, Radim Kr?má? wrote: > + if (kvm_x86_ops->sync_pir_to_irr(vcpu)) > + kvm_make_request(KVM_REQ_EVENT, vcpu); > + The call to sync_pir_to_irr belongs more in vcpu_scan_ioapic, I think. More importantly, I think that KVM_REQ_EVENT is a latent bug for kvm_vcpu_ioctl_get_lapic as well, so the call to kvm_make_request should go in vmx_sync_pir_to_irr or in a new kvm_sync_pir_to_irr wrapper. > + (e->fields.trig_mode == IOAPIC_EDGE_TRIG && > + kvm_apic_pending_eoi(vcpu, e->fields.vector))) Should we test again here that kvm_irq_has_notifier(ioapic->kvm, KVM_IRQCHIP_IOAPIC, index), to avoid unnecessarily marking other edge-triggered interrupts? Paolo -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
2015-10-07 11:29+0200, Paolo Bonzini: > On 13/08/2015 15:46, Radim Kr?má? wrote: >> + if (kvm_x86_ops->sync_pir_to_irr(vcpu)) >> + kvm_make_request(KVM_REQ_EVENT, vcpu); >> + > > The call to sync_pir_to_irr belongs more in vcpu_scan_ioapic, I think. Will do so in v2. > More importantly, I think that KVM_REQ_EVENT is a latent bug for > kvm_vcpu_ioctl_get_lapic as well, so the call to kvm_make_request should > go in vmx_sync_pir_to_irr or in a new kvm_sync_pir_to_irr wrapper. True, thanks. I'll make the request in kvm_apic_update_irr (unless you'd prefer to have it in new kvm_sync_pir_to_irr). >> + (e->fields.trig_mode == IOAPIC_EDGE_TRIG && >> + kvm_apic_pending_eoi(vcpu, e->fields.vector))) > > Should we test again here that kvm_irq_has_notifier(ioapic->kvm, > KVM_IRQCHIP_IOAPIC, index), to avoid unnecessarily marking other > edge-triggered interrupts? Other edge-triggered interrupts are skipped by a previous condition: if (e->fields.trig_mode == IOAPIC_LEVEL_TRIG || kvm_irq_has_notifier(ioapic->kvm, KVM_IRQCHIP_IOAPIC, index) || index == RTC_GSI) [we're here] I think it is ok to ignore level-triggered RTC, but we do want to include edge-triggered. -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
diff --git a/arch/x86/kvm/ioapic.c b/arch/x86/kvm/ioapic.c index 2dcda0f188ba..85d25fe25e39 100644 --- a/arch/x86/kvm/ioapic.c +++ b/arch/x86/kvm/ioapic.c @@ -239,6 +239,9 @@ void kvm_ioapic_scan_entry(struct kvm_vcpu *vcpu, u64 *eoi_exit_bitmap) union kvm_ioapic_redirect_entry *e; int index; + if (kvm_x86_ops->sync_pir_to_irr(vcpu)) + kvm_make_request(KVM_REQ_EVENT, vcpu); + spin_lock(&ioapic->lock); for (index = 0; index < IOAPIC_NUM_PINS; index++) { e = &ioapic->redirtbl[index]; @@ -246,7 +249,9 @@ void kvm_ioapic_scan_entry(struct kvm_vcpu *vcpu, u64 *eoi_exit_bitmap) kvm_irq_has_notifier(ioapic->kvm, KVM_IRQCHIP_IOAPIC, index) || index == RTC_GSI) { if (kvm_apic_match_dest(vcpu, NULL, 0, - e->fields.dest_id, e->fields.dest_mode)) + e->fields.dest_id, e->fields.dest_mode) || + (e->fields.trig_mode == IOAPIC_EDGE_TRIG && + kvm_apic_pending_eoi(vcpu, e->fields.vector))) __set_bit(e->fields.vector, (unsigned long *)eoi_exit_bitmap); }
(The main problem is that we care about EOI of edge interrupts, but our house of cards started a long time ago, so overturning that decision is not ideal for a stable fix.) KVM uses eoi_exit_bitmap to track vectors that need an action on EOI. The problem is that IOAPIC can be reconfigured while an interrupt with old configuration is pending and eoi_exit_bitmap only remembers the newest configuration so EOI from the pending interrupt is not recognized. This is not a problem for level interrupts, because IOAPIC sends interrupt with the new configuration. And then there are edge interrupts with ACK notifiers, like i8254 timer; things can happen in this order 1) IOAPIC inject a vector from i8254 2) guest reconfigures that vector's VCPU and therefore eoi_exit_bitmap on original VCPU gets cleared 3) guest's handler for the vector does EOI 4) KVM's EOI handler doesn't pass that vector to IOAPIC because it is not in that VCPU's eoi_exit_bitmap 5) i8254 stops working A simple solution is to set the IOAPIC vector in eoi_exit_bitmap if the vector is in PIR/IRR/ISR. This creates an unwanted situation if the vector is reused by a non-IOAPIC source, but I think it is so rare that we don't want to make the solution more sophisticated. The simple solution also doesn't work if we are reconfiguring the vector. (Shouldn't happen in the wild and I'd rather fix users of ACK notifiers instead of working around that.) The are no races because ioapic injection and reconfig are locked. Fixes: 638e7c03efea ("KVM: x86: Add EOI exit bitmap inference") [Before 638e7c03efea, this bug happened only with APICv.] Fixes: c7c9c56ca26f ("x86, apicv: add virtual interrupt delivery support") Cc: <stable@vger.kernel.org> Signed-off-by: Radim Kr?má? <rkrcmar@redhat.com> --- arch/x86/kvm/ioapic.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)