From patchwork Thu Nov 21 00:56:03 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Gibson X-Patchwork-Id: 11255111 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 11F8B14DB for ; Thu, 21 Nov 2019 01:03:03 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id B5AD920878 for ; Thu, 21 Nov 2019 01:03:02 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=gibson.dropbear.id.au header.i=@gibson.dropbear.id.au header.b="IcxKWzfs" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org B5AD920878 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=gibson.dropbear.id.au Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Received: from localhost ([::1]:35512 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1iXas9-0005a5-8j for patchwork-qemu-devel@patchwork.kernel.org; Wed, 20 Nov 2019 20:03:01 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:33928) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1iXao5-0001Qc-NI for qemu-devel@nongnu.org; Wed, 20 Nov 2019 19:58:51 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1iXao4-0001rb-2w for qemu-devel@nongnu.org; Wed, 20 Nov 2019 19:58:49 -0500 Received: from bilbo.ozlabs.org ([203.11.71.1]:39031 helo=ozlabs.org) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1iXao3-0001qw-CJ; Wed, 20 Nov 2019 19:58:48 -0500 Received: by ozlabs.org (Postfix, from userid 1007) id 47JLm41Hqpz9sNx; Thu, 21 Nov 2019 11:58:44 +1100 (AEDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=gibson.dropbear.id.au; s=201602; t=1574297924; bh=fREJuuqgAEuiDzU2O79CZ0GSodqvDi8dVSOI0miZGuE=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=IcxKWzfsNp+TDMm/FcmYOF/G6BK8Fwdx7sj3AbbpJBecSmOD3EvejoKZXJ9Hdo5X4 GBJcG9PY7fmZVX3w/91Q8nXcqBSmyoGG35fPguQUP2szAqbFxrua9emPjVKUaoeYJL b8W+cjT9+2k1AAe70PVOMNqijScMEkmCl0xnVIm4= From: David Gibson To: Alex Williamson , clg@kaod.org Subject: [PATCH 1/5] kvm: Introduce KVM irqchip change notifier Date: Thu, 21 Nov 2019 11:56:03 +1100 Message-Id: <20191121005607.274347-2-david@gibson.dropbear.id.au> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20191121005607.274347-1-david@gibson.dropbear.id.au> References: <20191121005607.274347-1-david@gibson.dropbear.id.au> MIME-Version: 1.0 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 203.11.71.1 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: kvm@vger.kernel.org, qemu-devel@nongnu.org, Jason Wang , Riku Voipio , Laurent Vivier , groug@kaod.org, Alexey Kardashevskiy , qemu-ppc@nongnu.org, =?utf-8?q?Marc-A?= =?utf-8?q?ndr=C3=A9_Lureau?= , Paolo Bonzini , philmd@redhat.com, David Gibson Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" Awareness of an in kernel irqchip is usually local to the machine and its top-level interrupt controller. However, in a few cases other things need to know about it. In particular vfio devices need this in order to accelerate interrupt delivery. If interrupt routing is changed, such devices may need to readjust their connection to the KVM irqchip. pci_bus_fire_intx_routing_notifier() exists to do just this. However, for the pseries machine type we have a situation where the routing remains constant but the top-level irq chip itself is changed. This occurs because of PAPR feature negotiation which allows the guest to decide between the older XICS and newer XIVE irq chip models (both of which are paravirtualized). To allow devices like vfio to adjust to this change, introduce a new notifier for the purpose kvm_irqchip_change_notify(). Cc: Alex Williamson Cc: Alexey Kardashevskiy Signed-off-by: David Gibson --- accel/kvm/kvm-all.c | 18 ++++++++++++++++++ accel/stubs/kvm-stub.c | 12 ++++++++++++ include/sysemu/kvm.h | 5 +++++ 3 files changed, 35 insertions(+) diff --git a/accel/kvm/kvm-all.c b/accel/kvm/kvm-all.c index 140b0bd8f6..ca00daa2f5 100644 --- a/accel/kvm/kvm-all.c +++ b/accel/kvm/kvm-all.c @@ -149,6 +149,9 @@ static const KVMCapabilityInfo kvm_required_capabilites[] = { KVM_CAP_LAST_INFO }; +static NotifierList kvm_irqchip_change_notifiers = + NOTIFIER_LIST_INITIALIZER(kvm_irqchip_change_notifiers); + #define kvm_slots_lock(kml) qemu_mutex_lock(&(kml)->slots_lock) #define kvm_slots_unlock(kml) qemu_mutex_unlock(&(kml)->slots_lock) @@ -1396,6 +1399,21 @@ void kvm_irqchip_release_virq(KVMState *s, int virq) trace_kvm_irqchip_release_virq(virq); } +void kvm_irqchip_add_change_notifier(Notifier *n) +{ + notifier_list_add(&kvm_irqchip_change_notifiers, n); +} + +void kvm_irqchip_remove_change_notifier(Notifier *n) +{ + notifier_remove(n); +} + +void kvm_irqchip_change_notify(void) +{ + notifier_list_notify(&kvm_irqchip_change_notifiers, NULL); +} + static unsigned int kvm_hash_msi(uint32_t data) { /* This is optimized for IA32 MSI layout. However, no other arch shall diff --git a/accel/stubs/kvm-stub.c b/accel/stubs/kvm-stub.c index 6feb66ed80..82f118d2df 100644 --- a/accel/stubs/kvm-stub.c +++ b/accel/stubs/kvm-stub.c @@ -138,6 +138,18 @@ void kvm_irqchip_commit_routes(KVMState *s) { } +void kvm_irqchip_add_change_notifier(Notifier *n) +{ +} + +void kvm_irqchip_remove_change_notifier(Notifier *n) +{ +} + +void kvm_irqchip_change_notify(void) +{ +} + int kvm_irqchip_add_adapter_route(KVMState *s, AdapterInfo *adapter) { return -ENOSYS; diff --git a/include/sysemu/kvm.h b/include/sysemu/kvm.h index 9d143282bc..9fe233b9bf 100644 --- a/include/sysemu/kvm.h +++ b/include/sysemu/kvm.h @@ -201,6 +201,7 @@ typedef struct KVMCapabilityInfo { struct KVMState; typedef struct KVMState KVMState; extern KVMState *kvm_state; +typedef struct Notifier Notifier; /* external API */ @@ -401,6 +402,10 @@ int kvm_irqchip_send_msi(KVMState *s, MSIMessage msg); void kvm_irqchip_add_irq_route(KVMState *s, int gsi, int irqchip, int pin); +void kvm_irqchip_add_change_notifier(Notifier *n); +void kvm_irqchip_remove_change_notifier(Notifier *n); +void kvm_irqchip_change_notify(void); + void kvm_get_apic_state(DeviceState *d, struct kvm_lapic_state *kapic); struct kvm_guest_debug; From patchwork Thu Nov 21 00:56:04 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Gibson X-Patchwork-Id: 11255113 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id D18FF109A for ; Thu, 21 Nov 2019 01:05:02 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id A454E20730 for ; Thu, 21 Nov 2019 01:05:02 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=gibson.dropbear.id.au header.i=@gibson.dropbear.id.au header.b="IrCBuvyd" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org A454E20730 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=gibson.dropbear.id.au Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Received: from localhost ([::1]:35526 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1iXau5-0007ME-Sz for patchwork-qemu-devel@patchwork.kernel.org; Wed, 20 Nov 2019 20:05:01 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:33932) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1iXao5-0001Qg-Js for qemu-devel@nongnu.org; Wed, 20 Nov 2019 19:58:51 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1iXao4-0001s6-HE for qemu-devel@nongnu.org; Wed, 20 Nov 2019 19:58:49 -0500 Received: from bilbo.ozlabs.org ([2401:3900:2:1::2]:35443 helo=ozlabs.org) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1iXao4-0001qz-6i; Wed, 20 Nov 2019 19:58:48 -0500 Received: by ozlabs.org (Postfix, from userid 1007) id 47JLm42L1Gz9sPW; Thu, 21 Nov 2019 11:58:44 +1100 (AEDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=gibson.dropbear.id.au; s=201602; t=1574297924; bh=M0mcg5/1en/4FPfwraZYSINmLiks1lDcs9ZPZyf4HFY=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=IrCBuvydoJarKwc/AqXBCCfhRHgtOv2SIHcrg7RMzyu44qkv46jqKR1N3sGuqc4mh I9yv01OBJiKsd+pNQA58bD4EXpoXcEVNT14kxVqV0ZDbKeXx1WZqgw8B5syMI78vvk FKKxC/uAqg99z7l1n69qh3cmU4omIQGoacX/vSQ4= From: David Gibson To: Alex Williamson , clg@kaod.org Subject: [PATCH 2/5] vfio/pci: Split vfio_intx_update() Date: Thu, 21 Nov 2019 11:56:04 +1100 Message-Id: <20191121005607.274347-3-david@gibson.dropbear.id.au> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20191121005607.274347-1-david@gibson.dropbear.id.au> References: <20191121005607.274347-1-david@gibson.dropbear.id.au> MIME-Version: 1.0 X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2401:3900:2:1::2 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: kvm@vger.kernel.org, qemu-devel@nongnu.org, Jason Wang , Riku Voipio , Laurent Vivier , groug@kaod.org, Alexey Kardashevskiy , qemu-ppc@nongnu.org, =?utf-8?q?Marc-A?= =?utf-8?q?ndr=C3=A9_Lureau?= , Paolo Bonzini , philmd@redhat.com, David Gibson Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" This splits the vfio_intx_update() function into one part doing the actual reconnection with the KVM irqchip (vfio_intx_update(), now taking an argument with the new routing) and vfio_intx_routing_notifier() which handles calls to the pci device intx routing notifier and calling vfio_intx_update() when necessary. This will make adding support for the irqchip change notifier easier. Cc: Alex Williamson Cc: Alexey Kardashevskiy Signed-off-by: David Gibson --- hw/vfio/pci.c | 39 ++++++++++++++++++++++----------------- 1 file changed, 22 insertions(+), 17 deletions(-) diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c index 0c55883bba..521289aa7d 100644 --- a/hw/vfio/pci.c +++ b/hw/vfio/pci.c @@ -216,30 +216,18 @@ static void vfio_intx_disable_kvm(VFIOPCIDevice *vdev) #endif } -static void vfio_intx_update(PCIDevice *pdev) +static void vfio_intx_update(VFIOPCIDevice *vdev, PCIINTxRoute *route) { - VFIOPCIDevice *vdev = PCI_VFIO(pdev); - PCIINTxRoute route; Error *err = NULL; - if (vdev->interrupt != VFIO_INT_INTx) { - return; - } - - route = pci_device_route_intx_to_irq(&vdev->pdev, vdev->intx.pin); - - if (!pci_intx_route_changed(&vdev->intx.route, &route)) { - return; /* Nothing changed */ - } - trace_vfio_intx_update(vdev->vbasedev.name, - vdev->intx.route.irq, route.irq); + vdev->intx.route.irq, route->irq); vfio_intx_disable_kvm(vdev); - vdev->intx.route = route; + vdev->intx.route = *route; - if (route.mode != PCI_INTX_ENABLED) { + if (route->mode != PCI_INTX_ENABLED) { return; } @@ -252,6 +240,22 @@ static void vfio_intx_update(PCIDevice *pdev) vfio_intx_eoi(&vdev->vbasedev); } +static void vfio_intx_routing_notifier(PCIDevice *pdev) +{ + VFIOPCIDevice *vdev = PCI_VFIO(pdev); + PCIINTxRoute route; + + if (vdev->interrupt != VFIO_INT_INTx) { + return; + } + + route = pci_device_route_intx_to_irq(&vdev->pdev, vdev->intx.pin); + + if (pci_intx_route_changed(&vdev->intx.route, &route)) { + vfio_intx_update(vdev, &route); + } +} + static int vfio_intx_enable(VFIOPCIDevice *vdev, Error **errp) { uint8_t pin = vfio_pci_read_config(&vdev->pdev, PCI_INTERRUPT_PIN, 1); @@ -2967,7 +2971,8 @@ static void vfio_realize(PCIDevice *pdev, Error **errp) if (vfio_pci_read_config(&vdev->pdev, PCI_INTERRUPT_PIN, 1)) { vdev->intx.mmap_timer = timer_new_ms(QEMU_CLOCK_VIRTUAL, vfio_intx_mmap_enable, vdev); - pci_device_set_intx_routing_notifier(&vdev->pdev, vfio_intx_update); + pci_device_set_intx_routing_notifier(&vdev->pdev, + vfio_intx_routing_notifier); ret = vfio_intx_enable(vdev, errp); if (ret) { goto out_teardown; From patchwork Thu Nov 21 00:56:05 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Gibson X-Patchwork-Id: 11255107 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id D3FB814C0 for ; Thu, 21 Nov 2019 00:59:52 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id AAC1C20730 for ; Thu, 21 Nov 2019 00:59:52 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=gibson.dropbear.id.au header.i=@gibson.dropbear.id.au header.b="HTZ8RZOP" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org AAC1C20730 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=gibson.dropbear.id.au Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Received: from localhost ([::1]:35484 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1iXap5-0002x6-MO for patchwork-qemu-devel@patchwork.kernel.org; Wed, 20 Nov 2019 19:59:51 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:33929) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1iXao5-0001Qd-JV for qemu-devel@nongnu.org; Wed, 20 Nov 2019 19:58:51 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1iXao4-0001rp-95 for qemu-devel@nongnu.org; Wed, 20 Nov 2019 19:58:49 -0500 Received: from bilbo.ozlabs.org ([2401:3900:2:1::2]:43401 helo=ozlabs.org) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1iXao3-0001r1-Ex; Wed, 20 Nov 2019 19:58:48 -0500 Received: by ozlabs.org (Postfix, from userid 1007) id 47JLm434w0z9sPn; Thu, 21 Nov 2019 11:58:44 +1100 (AEDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=gibson.dropbear.id.au; s=201602; t=1574297924; bh=Q2CHP0Ke6mkaX6t4hswofWQZGdJTg0Yt2bzdTOJZj8s=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=HTZ8RZOPsHWsPlug7qeJ77TO3NZIzw84Ege2mnZE9FQXPBQNOMFKManr/FpA/JEVX JpUbc+NncdPM4Lln8rWPT5XCKio3EgYMEr4M73X4n8sjX4u6pacwgYbfxPOAc+45r2 /nFqLBv+EQ3+WLKVvQgA3zAbTm5KsbKFNpAU9tfs= From: David Gibson To: Alex Williamson , clg@kaod.org Subject: [PATCH 3/5] vfio/pci: Respond to KVM irqchip change notifier Date: Thu, 21 Nov 2019 11:56:05 +1100 Message-Id: <20191121005607.274347-4-david@gibson.dropbear.id.au> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20191121005607.274347-1-david@gibson.dropbear.id.au> References: <20191121005607.274347-1-david@gibson.dropbear.id.au> MIME-Version: 1.0 X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2401:3900:2:1::2 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: kvm@vger.kernel.org, qemu-devel@nongnu.org, Jason Wang , Riku Voipio , Laurent Vivier , groug@kaod.org, Alexey Kardashevskiy , qemu-ppc@nongnu.org, =?utf-8?q?Marc-A?= =?utf-8?q?ndr=C3=A9_Lureau?= , Paolo Bonzini , philmd@redhat.com, David Gibson Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" VFIO PCI devices already respond to the pci intx routing notifier, in order to update kernel irqchip mappings when routing is updated. However this won't handle the case where the irqchip itself is replaced by a different model while retaining the same routing. This case can happen on the pseries machine type due to PAPR feature negotiation. To handle that case, add a handler for the irqchip change notifier, which does much the same thing as the routing notifier, but is unconditional, rather than being a no-op when the routing hasn't changed. Cc: Alex Williamson Cc: Alexey Kardashevskiy Signed-off-by: David Gibson --- hw/vfio/pci.c | 23 ++++++++++++++++++----- hw/vfio/pci.h | 1 + 2 files changed, 19 insertions(+), 5 deletions(-) diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c index 521289aa7d..95478c2c55 100644 --- a/hw/vfio/pci.c +++ b/hw/vfio/pci.c @@ -256,6 +256,14 @@ static void vfio_intx_routing_notifier(PCIDevice *pdev) } } +static void vfio_irqchip_change(Notifier *notify, void *data) +{ + VFIOPCIDevice *vdev = container_of(notify, VFIOPCIDevice, + irqchip_change_notifier); + + vfio_intx_update(vdev, &vdev->intx.route); +} + static int vfio_intx_enable(VFIOPCIDevice *vdev, Error **errp) { uint8_t pin = vfio_pci_read_config(&vdev->pdev, PCI_INTERRUPT_PIN, 1); @@ -2973,16 +2981,18 @@ static void vfio_realize(PCIDevice *pdev, Error **errp) vfio_intx_mmap_enable, vdev); pci_device_set_intx_routing_notifier(&vdev->pdev, vfio_intx_routing_notifier); + vdev->irqchip_change_notifier.notify = vfio_irqchip_change; + kvm_irqchip_add_change_notifier(&vdev->irqchip_change_notifier); ret = vfio_intx_enable(vdev, errp); if (ret) { - goto out_teardown; + goto out_deregister; } } if (vdev->display != ON_OFF_AUTO_OFF) { ret = vfio_display_probe(vdev, errp); if (ret) { - goto out_teardown; + goto out_deregister; } } if (vdev->enable_ramfb && vdev->dpy == NULL) { @@ -2992,11 +3002,11 @@ static void vfio_realize(PCIDevice *pdev, Error **errp) if (vdev->display_xres || vdev->display_yres) { if (vdev->dpy == NULL) { error_setg(errp, "xres and yres properties require display=on"); - goto out_teardown; + goto out_deregister; } if (vdev->dpy->edid_regs == NULL) { error_setg(errp, "xres and yres properties need edid support"); - goto out_teardown; + goto out_deregister; } } @@ -3020,8 +3030,10 @@ static void vfio_realize(PCIDevice *pdev, Error **errp) return; -out_teardown: +out_deregister: pci_device_set_intx_routing_notifier(&vdev->pdev, NULL); + kvm_irqchip_remove_change_notifier(&vdev->irqchip_change_notifier); +out_teardown: vfio_teardown_msi(vdev); vfio_bars_exit(vdev); error: @@ -3064,6 +3076,7 @@ static void vfio_exitfn(PCIDevice *pdev) vfio_unregister_req_notifier(vdev); vfio_unregister_err_notifier(vdev); pci_device_set_intx_routing_notifier(&vdev->pdev, NULL); + kvm_irqchip_remove_change_notifier(&vdev->irqchip_change_notifier); vfio_disable_interrupts(vdev); if (vdev->intx.mmap_timer) { timer_free(vdev->intx.mmap_timer); diff --git a/hw/vfio/pci.h b/hw/vfio/pci.h index b329d50338..35626cd63e 100644 --- a/hw/vfio/pci.h +++ b/hw/vfio/pci.h @@ -169,6 +169,7 @@ typedef struct VFIOPCIDevice { bool enable_ramfb; VFIODisplay *dpy; Error *migration_blocker; + Notifier irqchip_change_notifier; } VFIOPCIDevice; uint32_t vfio_pci_read_config(PCIDevice *pdev, uint32_t addr, int len); From patchwork Thu Nov 21 00:56:06 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: David Gibson X-Patchwork-Id: 11255109 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 14EC0109A for ; Thu, 21 Nov 2019 01:03:03 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id A9FC420730 for ; Thu, 21 Nov 2019 01:03:02 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=gibson.dropbear.id.au header.i=@gibson.dropbear.id.au header.b="GH/ey9I1" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org A9FC420730 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=gibson.dropbear.id.au Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Received: from localhost ([::1]:35514 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1iXas9-0005am-Aj for patchwork-qemu-devel@patchwork.kernel.org; Wed, 20 Nov 2019 20:03:01 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:33969) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1iXao7-0001R3-9h for qemu-devel@nongnu.org; Wed, 20 Nov 2019 19:58:52 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1iXao6-0001tO-AP for qemu-devel@nongnu.org; Wed, 20 Nov 2019 19:58:51 -0500 Received: from ozlabs.org ([2401:3900:2:1::2]:45307) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1iXao6-0001sA-0Q; Wed, 20 Nov 2019 19:58:50 -0500 Received: by ozlabs.org (Postfix, from userid 1007) id 47JLm44S9lz9sR2; Thu, 21 Nov 2019 11:58:44 +1100 (AEDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=gibson.dropbear.id.au; s=201602; t=1574297924; bh=uy06rYtNsupoZ3y3vcdSdik4ZpqVFHNbtPoj6Ji6YFU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=GH/ey9I17WPcqhxnfntIjzp89v+GocZFviNoVj5tc/u1wRv9nZRm70g0d6H7E4huG Ko5wf/5H1AmADYpIy9Xxx7C+drHdYhuzBZcJJdM9B/L1KHZaBNVIE79jAeczsA/ns9 js14uu6SHHwCWZDkUy0PdzSRpEXyti5fXNxUYejc= From: David Gibson To: Alex Williamson , clg@kaod.org Subject: [PATCH 4/5] spapr: Handle irq backend changes with VFIO PCI devices Date: Thu, 21 Nov 2019 11:56:06 +1100 Message-Id: <20191121005607.274347-5-david@gibson.dropbear.id.au> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20191121005607.274347-1-david@gibson.dropbear.id.au> References: <20191121005607.274347-1-david@gibson.dropbear.id.au> MIME-Version: 1.0 X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2401:3900:2:1::2 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: kvm@vger.kernel.org, qemu-devel@nongnu.org, Jason Wang , Riku Voipio , Laurent Vivier , groug@kaod.org, Alexey Kardashevskiy , qemu-ppc@nongnu.org, =?utf-8?q?Marc-A?= =?utf-8?q?ndr=C3=A9_Lureau?= , Paolo Bonzini , philmd@redhat.com, David Gibson Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" pseries machine type can have one of two different interrupt controllers in use depending on feature negotiation with the guest. Usually this is invisible to devices, because they route to a common set of qemu_irqs which in turn dispatch to the correct back end. VFIO passthrough devices, however, wire themselves up directly to the KVM irqchip for performance, which means they are affected by this change in interrupt controller. To get them to adjust correctly for the change in irqchip, we need to fire the kvm irqchip change notifier. Cc: Alex Williamson Cc: Alexey Kardashevskiy Signed-off-by: David Gibson Reviewed-by: Cédric Le Goater --- hw/ppc/spapr_irq.c | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/hw/ppc/spapr_irq.c b/hw/ppc/spapr_irq.c index 168044be85..1d27034962 100644 --- a/hw/ppc/spapr_irq.c +++ b/hw/ppc/spapr_irq.c @@ -508,6 +508,12 @@ static void set_active_intc(SpaprMachineState *spapr, } spapr->active_intc = new_intc; + + /* + * We've changed the kernel irqchip, let VFIO devices know they + * need to readjust. + */ + kvm_irqchip_change_notify(); } void spapr_irq_update_active_intc(SpaprMachineState *spapr) From patchwork Thu Nov 21 00:56:07 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: David Gibson X-Patchwork-Id: 11255105 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 8243414C0 for ; Thu, 21 Nov 2019 00:59:51 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 591F920730 for ; Thu, 21 Nov 2019 00:59:51 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=gibson.dropbear.id.au header.i=@gibson.dropbear.id.au header.b="d5Yl8X8j" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 591F920730 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=gibson.dropbear.id.au Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Received: from localhost ([::1]:35482 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1iXap4-0002ug-B1 for patchwork-qemu-devel@patchwork.kernel.org; Wed, 20 Nov 2019 19:59:50 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:33931) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1iXao5-0001Qf-Rj for qemu-devel@nongnu.org; Wed, 20 Nov 2019 19:58:50 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1iXao4-0001s0-Fe for qemu-devel@nongnu.org; Wed, 20 Nov 2019 19:58:49 -0500 Received: from bilbo.ozlabs.org ([2401:3900:2:1::2]:57659 helo=ozlabs.org) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1iXao4-0001r2-51; Wed, 20 Nov 2019 19:58:48 -0500 Received: by ozlabs.org (Postfix, from userid 1007) id 47JLm43p1Sz9sQp; Thu, 21 Nov 2019 11:58:44 +1100 (AEDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=gibson.dropbear.id.au; s=201602; t=1574297924; bh=kjLCHlKyKt7jP0NFe7tI8vqbJnysCqs9Js2O7NlIAAQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=d5Yl8X8jt5f6X3VYPrSZIeC4Sr8ahHtnfT9r4tRoBX95XiR92/bUGI6IPEAELp+p/ gj4PkS/PiYEpGg2AlbgHsnkezWv9MlYEin1RnA0QOdA8ICddY5MvLGDgMhrae1DmfQ tN1PBz0gCD0y8oOnF1ZySK7+BH5GU5e+myP++TYw= From: David Gibson To: Alex Williamson , clg@kaod.org Subject: [PATCH 5/5] spapr: Work around spurious warnings from vfio INTx initialization Date: Thu, 21 Nov 2019 11:56:07 +1100 Message-Id: <20191121005607.274347-6-david@gibson.dropbear.id.au> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20191121005607.274347-1-david@gibson.dropbear.id.au> References: <20191121005607.274347-1-david@gibson.dropbear.id.au> MIME-Version: 1.0 X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2401:3900:2:1::2 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: kvm@vger.kernel.org, qemu-devel@nongnu.org, Jason Wang , Riku Voipio , Laurent Vivier , groug@kaod.org, Alexey Kardashevskiy , qemu-ppc@nongnu.org, =?utf-8?q?Marc-A?= =?utf-8?q?ndr=C3=A9_Lureau?= , Paolo Bonzini , philmd@redhat.com, David Gibson Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" Traditional PCI INTx for vfio devices can only perform well if using an in-kernel irqchip. Therefore, vfio_intx_update() issues a warning if an in kernel irqchip is not available. We usually do have an in-kernel irqchip available for pseries machines on POWER hosts. However, because the platform allows feature negotiation of what interrupt controller model to use, we don't currently initialize it until machine reset. vfio_intx_update() is called (first) from vfio_realize() before that, so it can issue a spurious warning, even if we will have an in kernel irqchip by the time we need it. To workaround this, make a call to spapr_irq_update_active_intc() from spapr_irq_init() which is called at machine realize time, before the vfio realize. This call will be pretty much obsoleted by the later call at reset time, but it serves to suppress the spurious warning from VFIO. Cc: Alex Williamson Cc: Alexey Kardashevskiy Signed-off-by: David Gibson Reviewed-by: Cédric Le Goater --- hw/ppc/spapr_irq.c | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/hw/ppc/spapr_irq.c b/hw/ppc/spapr_irq.c index 1d27034962..d6bb7fd2d6 100644 --- a/hw/ppc/spapr_irq.c +++ b/hw/ppc/spapr_irq.c @@ -373,6 +373,14 @@ void spapr_irq_init(SpaprMachineState *spapr, Error **errp) spapr->qirqs = qemu_allocate_irqs(spapr_set_irq, spapr, smc->nr_xirqs + SPAPR_XIRQ_BASE); + + /* + * Mostly we don't actually need this until reset, except that not + * having this set up can cause VFIO devices to issue a + * false-positive warning during realize(), because they don't yet + * have an in-kernel irq chip. + */ + spapr_irq_update_active_intc(spapr); } int spapr_irq_claim(SpaprMachineState *spapr, int irq, bool lsi, Error **errp) @@ -528,7 +536,8 @@ void spapr_irq_update_active_intc(SpaprMachineState *spapr) * this. */ new_intc = SPAPR_INTC(spapr->xive); - } else if (spapr_ovec_test(spapr->ov5_cas, OV5_XIVE_EXPLOIT)) { + } else if (spapr->ov5_cas + && spapr_ovec_test(spapr->ov5_cas, OV5_XIVE_EXPLOIT)) { new_intc = SPAPR_INTC(spapr->xive); } else { new_intc = SPAPR_INTC(spapr->ics);