From patchwork Tue Jan 7 16:32:39 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: David Woodhouse X-Patchwork-Id: 13929323 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 7100EE77197 for ; Tue, 7 Jan 2025 16:32:52 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tVCVa-0003nQ-Fn; Tue, 07 Jan 2025 11:32:46 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tVCVZ-0003mz-0S for qemu-devel@nongnu.org; Tue, 07 Jan 2025 11:32:45 -0500 Received: from casper.infradead.org ([2001:8b0:10b:1236::1]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tVCVW-0003nQ-0k for qemu-devel@nongnu.org; Tue, 07 Jan 2025 11:32:44 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=MIME-Version:Content-Type:Date:Cc:To: From:Subject:Message-ID:Sender:Reply-To:Content-Transfer-Encoding:Content-ID: Content-Description:In-Reply-To:References; bh=Ny80E6ReHNK2cdCuwnO5fBIUrcRPUIxrNpVijyT9QfM=; b=D/S2IOuB8P3q5fTQvgugzys4ST emc6y0Q/FkVZ/m3uz+3ITuXdUIgzn2QsMIM12KlA0hAvMJ7EPIDsg5dIKhNIcmMwNaxohBmTKT6Ea C/0RClY94jqF8kSjZl+TVQO28/PeHmPsZtnODT6kaDbCMXyV6+LshH4wnwodgwQWiQ+M+f+tv5pkv wALGTlHAXRsFIfodAVDPagJof1jnA8A+McDHGsenfg4KXzZXj2lvKVxtslpisD5zsQOLY3JwrAGb3 pbh4hQVP9/F9Co22HlgLzfggGe8ZiIFAXhjcyQjjGgcILt9RRhy7wxZ48oqaQ01Y5MmbTnikONQzp nuNcRGDg==; Received: from 54-240-197-232.amazon.com ([54.240.197.232] helo=u09cd745991455d.ant.amazon.com) by casper.infradead.org with esmtpsa (Exim 4.98 #2 (Red Hat Linux)) id 1tVCVT-00000008FHI-3gC3; Tue, 07 Jan 2025 16:32:40 +0000 Message-ID: <975f9dab5718b441e6d4463a43845dc22b5ddf8f.camel@infradead.org> Subject: [PATCH v2] hw/i386/pc: Fix level interrupt sharing for Xen event channel GSI From: David Woodhouse To: qemu-devel@nongnu.org, Thomas Huth Cc: Paul Durrant , Paolo Bonzini , Richard Henderson , Eduardo Habkost , "Michael S. Tsirkin" , Marcel Apfelbaum Date: Tue, 07 Jan 2025 16:32:39 +0000 User-Agent: Evolution 3.52.3-0ubuntu1 MIME-Version: 1.0 X-SRS-Rewrite: SMTP reverse-path rewritten from by casper.infradead.org. See http://www.infradead.org/rpr.html Received-SPF: none client-ip=2001:8b0:10b:1236::1; envelope-from=BATV+f031519a9170f34b3d42+7807+infradead.org+dwmw2@casper.srs.infradead.org; helo=casper.infradead.org X-Spam_score_int: -43 X-Spam_score: -4.4 X-Spam_bar: ---- X-Spam_report: (-4.4 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_MED=-2.3, SPF_HELO_NONE=0.001, SPF_NONE=0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org From: David Woodhouse The system GSIs are not designed for sharing. One device might assert a shared interrupt with qemu_set_irq() and another might deassert it, and the level from the first device is lost. This could be solved by using a multiplexer which functions as an OR gate, much like the PCI code already implements for pci_set_irq() for muxing the INTx lines. Alternatively, it could be solved by having a 'resample' callback which is invoked when the interrupt is acked at the interrupt controller, and causes the devices to re-trigger the interrupt if it should still be pending. This is the model that VFIO in Linux uses, with a 'resampler' eventfd that actually unmasks the interrupt on the hardware device and thus triggers a new interrupt from it if needed. QEMU currently doesn't use that VFIO interface correctly, and just bashes on the resampler for every MMIO access to the device "just in case". This does neither of those. The Xen event channel GSI support *already* has hooks into the PC gsi_handler() code, for routing GSIs to PIRQs. So we can implement the logical OR of the external input (from PCI INTx, serial etc.) with the Xen event channel GSI by allowing that existing hook to modify the 'level' being asserted. Closes: https://gitlab.com/qemu-project/qemu/-/issues/2731 Reported-by: Thomas Huth Signed-off-by: David Woodhouse Acked-by: Michael S. Tsirkin --- v2: • Fix deassertion of PIRQ GSI. hw/i386/kvm/xen_evtchn.c | 50 +++++++++++++++++++++++++++++++--------- hw/i386/kvm/xen_evtchn.h | 2 +- hw/i386/x86-common.c | 32 ++++++++++++++++--------- 3 files changed, 61 insertions(+), 23 deletions(-) diff --git a/hw/i386/kvm/xen_evtchn.c b/hw/i386/kvm/xen_evtchn.c index bd2a3cbee0..93f68790b0 100644 --- a/hw/i386/kvm/xen_evtchn.c +++ b/hw/i386/kvm/xen_evtchn.c @@ -140,6 +140,8 @@ struct XenEvtchnState { uint64_t callback_param; bool evtchn_in_kernel; + bool setting_callback_gsi; + int extern_gsi_level; uint32_t callback_gsi; QEMUBH *gsi_bh; @@ -431,7 +433,16 @@ void xen_evtchn_set_callback_level(int level) } if (s->callback_gsi && s->callback_gsi < s->nr_callback_gsis) { - qemu_set_irq(s->callback_gsis[s->callback_gsi], level); + /* + * Ugly, but since we hold the BQL we can set this flag so that + * xen_evtchn_set_gsi() can tell the difference between this code + * setting the GSI, and an external device (PCI INTx) doing so. + */ + s->setting_callback_gsi = true; + /* Do not deassert the line if an external device is asserting it. */ + qemu_set_irq(s->callback_gsis[s->callback_gsi], + level || s->extern_gsi_level); + s->setting_callback_gsi = false; if (level) { /* Ensure the vCPU polls for deassertion */ kvm_xen_set_callback_asserted(); @@ -1596,7 +1607,7 @@ static int allocate_pirq(XenEvtchnState *s, int type, int gsi) return pirq; } -bool xen_evtchn_set_gsi(int gsi, int level) +bool xen_evtchn_set_gsi(int gsi, int *level) { XenEvtchnState *s = xen_evtchn_singleton; int pirq; @@ -1608,16 +1619,33 @@ bool xen_evtchn_set_gsi(int gsi, int level) } /* - * Check that that it *isn't* the event channel GSI, and thus - * that we are not recursing and it's safe to take s->port_lock. - * - * Locking aside, it's perfectly sane to bail out early for that - * special case, as it would make no sense for the event channel - * GSI to be routed back to event channels, when the delivery - * method is to raise the GSI... that recursion wouldn't *just* - * be a locking issue. + * For the callback_gsi we need to implement a logical OR of the event + * channel GSI and the external input (e.g. from PCI INTx), because + * QEMU itself doesn't support shared level interrupts via demux or + * resamplers. */ if (gsi && gsi == s->callback_gsi) { + /* Remember the external state of the GSI pin (e.g. from PCI INTx) */ + if (!s->setting_callback_gsi) { + s->extern_gsi_level = *level; + + /* + * Don't allow the external device to deassert the line if the + * eveht channel GSI should still be asserted. + */ + if (!s->extern_gsi_level) { + struct vcpu_info *vi = kvm_xen_get_vcpu_info_hva(0); + if (vi && vi->evtchn_upcall_pending) { + *level = 1; + } + } + } + + /* + * The event channel GSI cannot be routed to PIRQ, as that would make + * no sense. It could also deadlock on s->port_lock, if we proceed. + * So bail out now. + */ return false; } @@ -1628,7 +1656,7 @@ bool xen_evtchn_set_gsi(int gsi, int level) return false; } - if (level) { + if (*level) { int port = s->pirq[pirq].port; s->pirq_gsi_set |= (1U << gsi); diff --git a/hw/i386/kvm/xen_evtchn.h b/hw/i386/kvm/xen_evtchn.h index b740acfc0d..0521ebc092 100644 --- a/hw/i386/kvm/xen_evtchn.h +++ b/hw/i386/kvm/xen_evtchn.h @@ -23,7 +23,7 @@ void xen_evtchn_set_callback_level(int level); int xen_evtchn_set_port(uint16_t port); -bool xen_evtchn_set_gsi(int gsi, int level); +bool xen_evtchn_set_gsi(int gsi, int *level); void xen_evtchn_snoop_msi(PCIDevice *dev, bool is_msix, unsigned int vector, uint64_t addr, uint32_t data, bool is_masked); void xen_evtchn_remove_pci_device(PCIDevice *dev); diff --git a/hw/i386/x86-common.c b/hw/i386/x86-common.c index a7d46c3105..13badc26a5 100644 --- a/hw/i386/x86-common.c +++ b/hw/i386/x86-common.c @@ -450,8 +450,27 @@ static long get_file_size(FILE *f) void gsi_handler(void *opaque, int n, int level) { GSIState *s = opaque; + bool bypass_ioapic = false; trace_x86_gsi_interrupt(n, level); + +#ifdef CONFIG_XEN_EMU + /* + * Xen delivers the GSI to the Legacy PIC (not that Legacy PIC + * routing actually works properly under Xen). And then to + * *either* the PIRQ handling or the I/OAPIC depending on + * whether the former wants it. + * + * Additionally, this hook allows the Xen event channel GSI to + * work around QEMU's lack of support for shared level interrupts, + * by keeping track of the externally driven state of the pin and + * implementing a logical OR with the state of the evtchn GSI. + */ + if (xen_mode == XEN_EMULATE) { + bypass_ioapic = xen_evtchn_set_gsi(n, &level); + } +#endif + switch (n) { case 0 ... ISA_NUM_IRQS - 1: if (s->i8259_irq[n]) { @@ -460,18 +479,9 @@ void gsi_handler(void *opaque, int n, int level) } /* fall through */ case ISA_NUM_IRQS ... IOAPIC_NUM_PINS - 1: -#ifdef CONFIG_XEN_EMU - /* - * Xen delivers the GSI to the Legacy PIC (not that Legacy PIC - * routing actually works properly under Xen). And then to - * *either* the PIRQ handling or the I/OAPIC depending on - * whether the former wants it. - */ - if (xen_mode == XEN_EMULATE && xen_evtchn_set_gsi(n, level)) { - break; + if (!bypass_ioapic) { + qemu_set_irq(s->ioapic_irq[n], level); } -#endif - qemu_set_irq(s->ioapic_irq[n], level); break; case IO_APIC_SECONDARY_IRQBASE ... IO_APIC_SECONDARY_IRQBASE + IOAPIC_NUM_PINS - 1: