From patchwork Mon Oct 9 06:48:55 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?C=C3=A9dric_Le_Goater?= X-Patchwork-Id: 13412990 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 493EDE95A8E for ; Mon, 9 Oct 2023 06:49:44 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qpk4x-0006QC-PT; Mon, 09 Oct 2023 02:49:23 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qpk4w-0006Po-8P for qemu-devel@nongnu.org; Mon, 09 Oct 2023 02:49:22 -0400 Received: from gandalf.ozlabs.org ([150.107.74.76]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qpk4u-0007Gl-7q for qemu-devel@nongnu.org; Mon, 09 Oct 2023 02:49:22 -0400 Received: from gandalf.ozlabs.org (gandalf.ozlabs.org [150.107.74.76]) by gandalf.ozlabs.org (Postfix) with ESMTP id 4S3qPy6Bk9z4xWY; Mon, 9 Oct 2023 17:49:10 +1100 (AEDT) Received: from authenticated.ozlabs.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by mail.ozlabs.org (Postfix) with ESMTPSA id 4S3qPx1kYdz4xWP; Mon, 9 Oct 2023 17:49:08 +1100 (AEDT) From: =?utf-8?q?C=C3=A9dric_Le_Goater?= To: qemu-devel@nongnu.org Cc: Alex Williamson , =?utf-8?q?C=C3=A9dric_Le_G?= =?utf-8?q?oater?= Subject: [PULL 1/6] vfio/display: Fix missing update to set backing fields Date: Mon, 9 Oct 2023 08:48:55 +0200 Message-ID: <20231009064900.1465361-2-clg@redhat.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20231009064900.1465361-1-clg@redhat.com> References: <20231009064900.1465361-1-clg@redhat.com> MIME-Version: 1.0 Received-SPF: pass client-ip=150.107.74.76; envelope-from=SRS0=qD0P=FX=redhat.com=clg@ozlabs.org; helo=gandalf.ozlabs.org X-Spam_score_int: -16 X-Spam_score: -1.7 X-Spam_bar: - X-Spam_report: (-1.7 / 5.0 requ) BAYES_00=-1.9, HEADER_FROM_DIFFERENT_DOMAINS=0.249, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=no autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org From: Alex Williamson The below referenced commit renames scanout_width/height to backing_width/height, but also promotes these fields in various portions of the egl interface. Meanwhile vfio dmabuf support has never used the previous scanout fields and is therefore missed in the update. This results in a black screen when transitioning from ramfb to dmabuf display when using Intel vGPU with these features. Resolves: https://gitlab.com/qemu-project/qemu/-/issues/1891 Link: https://lists.gnu.org/archive/html/qemu-devel/2023-08/msg02726.html Fixes: 9ac06df8b684 ("virtio-gpu-udmabuf: correct naming of QemuDmaBuf size properties") Signed-off-by: Alex Williamson Tested-by: Cédric Le Goater Signed-off-by: Cédric Le Goater --- hw/vfio/display.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/hw/vfio/display.c b/hw/vfio/display.c index bec864f482f46866bf292a5a63c31753a7c84eef..837d9e6a309e834601c125e36faadf81c1c5172e 100644 --- a/hw/vfio/display.c +++ b/hw/vfio/display.c @@ -243,6 +243,8 @@ static VFIODMABuf *vfio_display_get_dmabuf(VFIOPCIDevice *vdev, dmabuf->dmabuf_id = plane.dmabuf_id; dmabuf->buf.width = plane.width; dmabuf->buf.height = plane.height; + dmabuf->buf.backing_width = plane.width; + dmabuf->buf.backing_height = plane.height; dmabuf->buf.stride = plane.stride; dmabuf->buf.fourcc = plane.drm_format; dmabuf->buf.modifier = plane.drm_format_mod; From patchwork Mon Oct 9 06:48:56 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?C=C3=A9dric_Le_Goater?= X-Patchwork-Id: 13412993 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id EFDBDE95A8D for ; Mon, 9 Oct 2023 06:50:16 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qpk4y-0006Qh-Jc; Mon, 09 Oct 2023 02:49:24 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qpk4w-0006Ps-ED for qemu-devel@nongnu.org; Mon, 09 Oct 2023 02:49:22 -0400 Received: from mail.ozlabs.org ([2404:9400:2221:ea00::3] helo=gandalf.ozlabs.org) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qpk4u-0007Gy-Rd for qemu-devel@nongnu.org; Mon, 09 Oct 2023 02:49:22 -0400 Received: from gandalf.ozlabs.org (mail.ozlabs.org [IPv6:2404:9400:2221:ea00::3]) by gandalf.ozlabs.org (Postfix) with ESMTP id 4S3qQ244hcz4xWc; Mon, 9 Oct 2023 17:49:14 +1100 (AEDT) Received: from authenticated.ozlabs.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by mail.ozlabs.org (Postfix) with ESMTPSA id 4S3qPz69DZz4xWP; Mon, 9 Oct 2023 17:49:11 +1100 (AEDT) From: =?utf-8?q?C=C3=A9dric_Le_Goater?= To: qemu-devel@nongnu.org Cc: Alex Williamson , Zhenzhong Duan , =?utf-8?q?C=C3=A9dric_Le_Goater?= Subject: [PULL 2/6] vfio/pci: rename vfio_put_device to vfio_pci_put_device Date: Mon, 9 Oct 2023 08:48:56 +0200 Message-ID: <20231009064900.1465361-3-clg@redhat.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20231009064900.1465361-1-clg@redhat.com> References: <20231009064900.1465361-1-clg@redhat.com> MIME-Version: 1.0 Received-SPF: pass client-ip=2404:9400:2221:ea00::3; envelope-from=SRS0=qD0P=FX=redhat.com=clg@ozlabs.org; helo=gandalf.ozlabs.org X-Spam_score_int: -39 X-Spam_score: -4.0 X-Spam_bar: ---- X-Spam_report: (-4.0 / 5.0 requ) BAYES_00=-1.9, HEADER_FROM_DIFFERENT_DOMAINS=0.249, RCVD_IN_DNSWL_MED=-2.3, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org From: Zhenzhong Duan vfio_put_device() is a VFIO PCI specific function, rename it with 'vfio_pci' prefix to avoid confusing. No functional change. Suggested-by: Cédric Le Goater Signed-off-by: Zhenzhong Duan Signed-off-by: Cédric Le Goater --- hw/vfio/pci.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c index 3b2ca3c24ca2ed91c03b5c6bef65bb5a0fb0298a..b2d5010b9f0effbe85a3f57b43e282e837b6383f 100644 --- a/hw/vfio/pci.c +++ b/hw/vfio/pci.c @@ -2826,7 +2826,7 @@ static void vfio_populate_device(VFIOPCIDevice *vdev, Error **errp) } } -static void vfio_put_device(VFIOPCIDevice *vdev) +static void vfio_pci_put_device(VFIOPCIDevice *vdev) { g_free(vdev->vbasedev.name); g_free(vdev->msix); @@ -3317,7 +3317,7 @@ static void vfio_instance_finalize(Object *obj) * * g_free(vdev->igd_opregion); */ - vfio_put_device(vdev); + vfio_pci_put_device(vdev); vfio_put_group(group); } From patchwork Mon Oct 9 06:48:57 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?C=C3=A9dric_Le_Goater?= X-Patchwork-Id: 13412995 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 9EF35E95A8D for ; Mon, 9 Oct 2023 06:50:37 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qpk51-0006S0-5u; Mon, 09 Oct 2023 02:49:27 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qpk4z-0006Qp-1i for qemu-devel@nongnu.org; Mon, 09 Oct 2023 02:49:25 -0400 Received: from mail.ozlabs.org ([2404:9400:2221:ea00::3] helo=gandalf.ozlabs.org) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qpk4w-0007H8-Vp for qemu-devel@nongnu.org; Mon, 09 Oct 2023 02:49:24 -0400 Received: from gandalf.ozlabs.org (gandalf.ozlabs.org [150.107.74.76]) by gandalf.ozlabs.org (Postfix) with ESMTP id 4S3qQ52RzRz4xWT; Mon, 9 Oct 2023 17:49:17 +1100 (AEDT) Received: from authenticated.ozlabs.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by mail.ozlabs.org (Postfix) with ESMTPSA id 4S3qQ30c17z4xWP; Mon, 9 Oct 2023 17:49:14 +1100 (AEDT) From: =?utf-8?q?C=C3=A9dric_Le_Goater?= To: qemu-devel@nongnu.org Cc: Alex Williamson , Jing Liu , Reinette Chatre , =?utf-8?q?C=C3=A9dric_Le_Goater?= Subject: [PULL 3/6] vfio/pci: detect the support of dynamic MSI-X allocation Date: Mon, 9 Oct 2023 08:48:57 +0200 Message-ID: <20231009064900.1465361-4-clg@redhat.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20231009064900.1465361-1-clg@redhat.com> References: <20231009064900.1465361-1-clg@redhat.com> MIME-Version: 1.0 Received-SPF: pass client-ip=2404:9400:2221:ea00::3; envelope-from=SRS0=qD0P=FX=redhat.com=clg@ozlabs.org; helo=gandalf.ozlabs.org X-Spam_score_int: -39 X-Spam_score: -4.0 X-Spam_bar: ---- X-Spam_report: (-4.0 / 5.0 requ) BAYES_00=-1.9, HEADER_FROM_DIFFERENT_DOMAINS=0.249, RCVD_IN_DNSWL_MED=-2.3, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org From: Jing Liu Kernel provides the guidance of dynamic MSI-X allocation support of passthrough device, by clearing the VFIO_IRQ_INFO_NORESIZE flag to guide user space. Fetch the flags from host to determine if dynamic MSI-X allocation is supported. Originally-by: Reinette Chatre Signed-off-by: Jing Liu Reviewed-by: Cédric Le Goater Reviewed-by: Alex Williamson Signed-off-by: Cédric Le Goater --- hw/vfio/pci.h | 1 + hw/vfio/pci.c | 16 ++++++++++++++-- hw/vfio/trace-events | 2 +- 3 files changed, 16 insertions(+), 3 deletions(-) diff --git a/hw/vfio/pci.h b/hw/vfio/pci.h index 2d836093a83d3df5f94dc8981ee6c61cfb6d3373..0d89eb761ece426ca1df40f63461b2bb1bcf1ab0 100644 --- a/hw/vfio/pci.h +++ b/hw/vfio/pci.h @@ -113,6 +113,7 @@ typedef struct VFIOMSIXInfo { uint32_t table_offset; uint32_t pba_offset; unsigned long *pending; + bool noresize; } VFIOMSIXInfo; #define TYPE_VFIO_PCI "vfio-pci" diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c index b2d5010b9f0effbe85a3f57b43e282e837b6383f..86c92c51a46e2d1e9856c3cbcd7c47548daab422 100644 --- a/hw/vfio/pci.c +++ b/hw/vfio/pci.c @@ -1493,7 +1493,9 @@ static void vfio_msix_early_setup(VFIOPCIDevice *vdev, Error **errp) uint8_t pos; uint16_t ctrl; uint32_t table, pba; - int fd = vdev->vbasedev.fd; + int ret, fd = vdev->vbasedev.fd; + struct vfio_irq_info irq_info = { .argsz = sizeof(irq_info), + .index = VFIO_PCI_MSIX_IRQ_INDEX }; VFIOMSIXInfo *msix; pos = pci_find_capability(&vdev->pdev, PCI_CAP_ID_MSIX); @@ -1530,6 +1532,15 @@ static void vfio_msix_early_setup(VFIOPCIDevice *vdev, Error **errp) msix->pba_offset = pba & ~PCI_MSIX_FLAGS_BIRMASK; msix->entries = (ctrl & PCI_MSIX_FLAGS_QSIZE) + 1; + ret = ioctl(vdev->vbasedev.fd, VFIO_DEVICE_GET_IRQ_INFO, &irq_info); + if (ret < 0) { + error_setg_errno(errp, -ret, "failed to get MSI-X irq info"); + g_free(msix); + return; + } + + msix->noresize = !!(irq_info.flags & VFIO_IRQ_INFO_NORESIZE); + /* * Test the size of the pba_offset variable and catch if it extends outside * of the specified BAR. If it is the case, we need to apply a hardware @@ -1562,7 +1573,8 @@ static void vfio_msix_early_setup(VFIOPCIDevice *vdev, Error **errp) } trace_vfio_msix_early_setup(vdev->vbasedev.name, pos, msix->table_bar, - msix->table_offset, msix->entries); + msix->table_offset, msix->entries, + msix->noresize); vdev->msix = msix; vfio_pci_fixup_msix_region(vdev); diff --git a/hw/vfio/trace-events b/hw/vfio/trace-events index e64ca4a01961b34dc4b9b7d78917f1d8847d887d..0ba3c5a0e26bc82f40a0d7737e68128d2c9334fe 100644 --- a/hw/vfio/trace-events +++ b/hw/vfio/trace-events @@ -27,7 +27,7 @@ vfio_vga_read(uint64_t addr, int size, uint64_t data) " (0x%"PRIx64", %d) = 0x%" vfio_pci_read_config(const char *name, int addr, int len, int val) " (%s, @0x%x, len=0x%x) 0x%x" vfio_pci_write_config(const char *name, int addr, int val, int len) " (%s, @0x%x, 0x%x, len=0x%x)" vfio_msi_setup(const char *name, int pos) "%s PCI MSI CAP @0x%x" -vfio_msix_early_setup(const char *name, int pos, int table_bar, int offset, int entries) "%s PCI MSI-X CAP @0x%x, BAR %d, offset 0x%x, entries %d" +vfio_msix_early_setup(const char *name, int pos, int table_bar, int offset, int entries, bool noresize) "%s PCI MSI-X CAP @0x%x, BAR %d, offset 0x%x, entries %d, noresize %d" vfio_check_pcie_flr(const char *name) "%s Supports FLR via PCIe cap" vfio_check_pm_reset(const char *name) "%s Supports PM reset" vfio_check_af_flr(const char *name) "%s Supports FLR via AF cap" From patchwork Mon Oct 9 06:48:58 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?C=C3=A9dric_Le_Goater?= X-Patchwork-Id: 13412996 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 74AA1E95A8E for ; Mon, 9 Oct 2023 06:50:41 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qpk50-0006Rb-Mh; Mon, 09 Oct 2023 02:49:26 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qpk4y-0006Qk-SH for qemu-devel@nongnu.org; Mon, 09 Oct 2023 02:49:24 -0400 Received: from gandalf.ozlabs.org ([150.107.74.76]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qpk4w-0007HN-L9 for qemu-devel@nongnu.org; Mon, 09 Oct 2023 02:49:24 -0400 Received: from gandalf.ozlabs.org (mail.ozlabs.org [IPv6:2404:9400:2221:ea00::3]) by gandalf.ozlabs.org (Postfix) with ESMTP id 4S3qQ84J32z4xWk; Mon, 9 Oct 2023 17:49:20 +1100 (AEDT) Received: from authenticated.ozlabs.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by mail.ozlabs.org (Postfix) with ESMTPSA id 4S3qQ62NsBz4xWP; Mon, 9 Oct 2023 17:49:18 +1100 (AEDT) From: =?utf-8?q?C=C3=A9dric_Le_Goater?= To: qemu-devel@nongnu.org Cc: Alex Williamson , Jing Liu , Reinette Chatre , =?utf-8?q?C=C3=A9dric_Le_Goater?= Subject: [PULL 4/6] vfio/pci: enable vector on dynamic MSI-X allocation Date: Mon, 9 Oct 2023 08:48:58 +0200 Message-ID: <20231009064900.1465361-5-clg@redhat.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20231009064900.1465361-1-clg@redhat.com> References: <20231009064900.1465361-1-clg@redhat.com> MIME-Version: 1.0 Received-SPF: pass client-ip=150.107.74.76; envelope-from=SRS0=qD0P=FX=redhat.com=clg@ozlabs.org; helo=gandalf.ozlabs.org X-Spam_score_int: -16 X-Spam_score: -1.7 X-Spam_bar: - X-Spam_report: (-1.7 / 5.0 requ) BAYES_00=-1.9, HEADER_FROM_DIFFERENT_DOMAINS=0.249, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=no autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org From: Jing Liu The vector_use callback is used to enable vector that is unmasked in guest. The kernel used to only support static MSI-X allocation. When allocating a new interrupt using "static MSI-X allocation" kernels, QEMU first disables all previously allocated vectors and then re-allocates all including the new one. The nr_vectors of VFIOPCIDevice indicates that all vectors from 0 to nr_vectors are allocated (and may be enabled), which is used to loop all the possibly used vectors when e.g., disabling MSI-X interrupts. Extend the vector_use function to support dynamic MSI-X allocation when host supports the capability. QEMU therefore can individually allocate and enable a new interrupt without affecting others or causing interrupts lost during runtime. Utilize nr_vectors to calculate the upper bound of enabled vectors in dynamic MSI-X allocation mode since looping all msix_entries_nr is not efficient and unnecessary. Signed-off-by: Jing Liu Tested-by: Reinette Chatre Reviewed-by: Alex Williamson Signed-off-by: Cédric Le Goater --- hw/vfio/pci.c | 46 ++++++++++++++++++++++++++++------------------ 1 file changed, 28 insertions(+), 18 deletions(-) diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c index 86c92c51a46e2d1e9856c3cbcd7c47548daab422..eb7627c5f9fcd67f346595ebe73011cc3205b58e 100644 --- a/hw/vfio/pci.c +++ b/hw/vfio/pci.c @@ -470,6 +470,7 @@ static int vfio_msix_vector_do_use(PCIDevice *pdev, unsigned int nr, VFIOPCIDevice *vdev = VFIO_PCI(pdev); VFIOMSIVector *vector; int ret; + bool resizing = !!(vdev->nr_vectors < nr + 1); trace_vfio_msix_vector_do_use(vdev->vbasedev.name, nr); @@ -512,33 +513,42 @@ static int vfio_msix_vector_do_use(PCIDevice *pdev, unsigned int nr, } /* - * We don't want to have the host allocate all possible MSI vectors - * for a device if they're not in use, so we shutdown and incrementally - * increase them as needed. + * When dynamic allocation is not supported, we don't want to have the + * host allocate all possible MSI vectors for a device if they're not + * in use, so we shutdown and incrementally increase them as needed. + * nr_vectors represents the total number of vectors allocated. + * + * When dynamic allocation is supported, let the host only allocate + * and enable a vector when it is in use in guest. nr_vectors represents + * the upper bound of vectors being enabled (but not all of the ranges + * is allocated or enabled). */ - if (vdev->nr_vectors < nr + 1) { + if (resizing) { vdev->nr_vectors = nr + 1; - if (!vdev->defer_kvm_irq_routing) { + } + + if (!vdev->defer_kvm_irq_routing) { + if (vdev->msix->noresize && resizing) { vfio_disable_irqindex(&vdev->vbasedev, VFIO_PCI_MSIX_IRQ_INDEX); ret = vfio_enable_vectors(vdev, true); if (ret) { error_report("vfio: failed to enable vectors, %d", ret); } - } - } else { - Error *err = NULL; - int32_t fd; - - if (vector->virq >= 0) { - fd = event_notifier_get_fd(&vector->kvm_interrupt); } else { - fd = event_notifier_get_fd(&vector->interrupt); - } + Error *err = NULL; + int32_t fd; - if (vfio_set_irq_signaling(&vdev->vbasedev, - VFIO_PCI_MSIX_IRQ_INDEX, nr, - VFIO_IRQ_SET_ACTION_TRIGGER, fd, &err)) { - error_reportf_err(err, VFIO_MSG_PREFIX, vdev->vbasedev.name); + if (vector->virq >= 0) { + fd = event_notifier_get_fd(&vector->kvm_interrupt); + } else { + fd = event_notifier_get_fd(&vector->interrupt); + } + + if (vfio_set_irq_signaling(&vdev->vbasedev, + VFIO_PCI_MSIX_IRQ_INDEX, nr, + VFIO_IRQ_SET_ACTION_TRIGGER, fd, &err)) { + error_reportf_err(err, VFIO_MSG_PREFIX, vdev->vbasedev.name); + } } } From patchwork Mon Oct 9 06:48:59 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?C=C3=A9dric_Le_Goater?= X-Patchwork-Id: 13412991 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id AAE44E95A8D for ; Mon, 9 Oct 2023 06:50:03 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qpk52-0006S3-MW; Mon, 09 Oct 2023 02:49:28 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qpk50-0006Rt-SV for qemu-devel@nongnu.org; Mon, 09 Oct 2023 02:49:26 -0400 Received: from gandalf.ozlabs.org ([150.107.74.76]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qpk4y-0007Hi-LX for qemu-devel@nongnu.org; Mon, 09 Oct 2023 02:49:26 -0400 Received: from gandalf.ozlabs.org (mail.ozlabs.org [IPv6:2404:9400:2221:ea00::3]) by gandalf.ozlabs.org (Postfix) with ESMTP id 4S3qQB4cC4z4xWl; Mon, 9 Oct 2023 17:49:22 +1100 (AEDT) Received: from authenticated.ozlabs.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by mail.ozlabs.org (Postfix) with ESMTPSA id 4S3qQ90p3Hz4xWP; Mon, 9 Oct 2023 17:49:20 +1100 (AEDT) From: =?utf-8?q?C=C3=A9dric_Le_Goater?= To: qemu-devel@nongnu.org Cc: Alex Williamson , Jing Liu , =?utf-8?q?C=C3=A9dric_Le_Goater?= Subject: [PULL 5/6] vfio/pci: use an invalid fd to enable MSI-X Date: Mon, 9 Oct 2023 08:48:59 +0200 Message-ID: <20231009064900.1465361-6-clg@redhat.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20231009064900.1465361-1-clg@redhat.com> References: <20231009064900.1465361-1-clg@redhat.com> MIME-Version: 1.0 Received-SPF: pass client-ip=150.107.74.76; envelope-from=SRS0=qD0P=FX=redhat.com=clg@ozlabs.org; helo=gandalf.ozlabs.org X-Spam_score_int: -16 X-Spam_score: -1.7 X-Spam_bar: - X-Spam_report: (-1.7 / 5.0 requ) BAYES_00=-1.9, HEADER_FROM_DIFFERENT_DOMAINS=0.249, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=no autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org From: Jing Liu Guests typically enable MSI-X with all of the vectors masked in the MSI-X vector table. To match the guest state of device, QEMU enables MSI-X by enabling vector 0 with userspace triggering and immediately release. However the release function actually does not release it due to already using userspace mode. It is no need to enable triggering on host and rely on the mask bit to avoid spurious interrupts. Use an invalid fd (i.e. fd = -1) is enough to get MSI-X enabled. After dynamic MSI-X allocation is supported, the interrupt restoring also need use such way to enable MSI-X, therefore, create a function for that. Suggested-by: Alex Williamson Signed-off-by: Jing Liu Reviewed-by: Cédric Le Goater Reviewed-by: Alex Williamson Signed-off-by: Cédric Le Goater --- hw/vfio/pci.c | 44 ++++++++++++++++++++++++++++++++++++-------- 1 file changed, 36 insertions(+), 8 deletions(-) diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c index eb7627c5f9fcd67f346595ebe73011cc3205b58e..ad508abd6f382d518871191017859c4c4c8fd4f9 100644 --- a/hw/vfio/pci.c +++ b/hw/vfio/pci.c @@ -369,6 +369,33 @@ static void vfio_msi_interrupt(void *opaque) notify(&vdev->pdev, nr); } +/* + * Get MSI-X enabled, but no vector enabled, by setting vector 0 with an invalid + * fd to kernel. + */ +static int vfio_enable_msix_no_vec(VFIOPCIDevice *vdev) +{ + g_autofree struct vfio_irq_set *irq_set = NULL; + int ret = 0, argsz; + int32_t *fd; + + argsz = sizeof(*irq_set) + sizeof(*fd); + + irq_set = g_malloc0(argsz); + irq_set->argsz = argsz; + irq_set->flags = VFIO_IRQ_SET_DATA_EVENTFD | + VFIO_IRQ_SET_ACTION_TRIGGER; + irq_set->index = VFIO_PCI_MSIX_IRQ_INDEX; + irq_set->start = 0; + irq_set->count = 1; + fd = (int32_t *)&irq_set->data; + *fd = -1; + + ret = ioctl(vdev->vbasedev.fd, VFIO_DEVICE_SET_IRQS, irq_set); + + return ret; +} + static int vfio_enable_vectors(VFIOPCIDevice *vdev, bool msix) { struct vfio_irq_set *irq_set; @@ -618,6 +645,8 @@ static void vfio_commit_kvm_msi_virq_batch(VFIOPCIDevice *vdev) static void vfio_msix_enable(VFIOPCIDevice *vdev) { + int ret; + vfio_disable_interrupts(vdev); vdev->msi_vectors = g_new0(VFIOMSIVector, vdev->msix->entries); @@ -640,8 +669,6 @@ static void vfio_msix_enable(VFIOPCIDevice *vdev) vfio_commit_kvm_msi_virq_batch(vdev); if (vdev->nr_vectors) { - int ret; - ret = vfio_enable_vectors(vdev, true); if (ret) { error_report("vfio: failed to enable vectors, %d", ret); @@ -655,13 +682,14 @@ static void vfio_msix_enable(VFIOPCIDevice *vdev) * MSI-X capability, but leaves the vector table masked. We therefore * can't rely on a vector_use callback (from request_irq() in the guest) * to switch the physical device into MSI-X mode because that may come a - * long time after pci_enable_msix(). This code enables vector 0 with - * triggering to userspace, then immediately release the vector, leaving - * the physical device with no vectors enabled, but MSI-X enabled, just - * like the guest view. + * long time after pci_enable_msix(). This code sets vector 0 with an + * invalid fd to make the physical device MSI-X enabled, but with no + * vectors enabled, just like the guest view. */ - vfio_msix_vector_do_use(&vdev->pdev, 0, NULL, NULL); - vfio_msix_vector_release(&vdev->pdev, 0); + ret = vfio_enable_msix_no_vec(vdev); + if (ret) { + error_report("vfio: failed to enable MSI-X, %d", ret); + } } trace_vfio_msix_enable(vdev->vbasedev.name); From patchwork Mon Oct 9 06:49:00 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?C=C3=A9dric_Le_Goater?= X-Patchwork-Id: 13412994 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id CE968E95A8E for ; Mon, 9 Oct 2023 06:50:19 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qpk56-0006St-73; Mon, 09 Oct 2023 02:49:32 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qpk54-0006SM-So for qemu-devel@nongnu.org; Mon, 09 Oct 2023 02:49:30 -0400 Received: from gandalf.ozlabs.org ([150.107.74.76]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qpk51-0007IM-Jp for qemu-devel@nongnu.org; Mon, 09 Oct 2023 02:49:29 -0400 Received: from gandalf.ozlabs.org (mail.ozlabs.org [IPv6:2404:9400:2221:ea00::3]) by gandalf.ozlabs.org (Postfix) with ESMTP id 4S3qQF47wKz4xWh; Mon, 9 Oct 2023 17:49:25 +1100 (AEDT) Received: from authenticated.ozlabs.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by mail.ozlabs.org (Postfix) with ESMTPSA id 4S3qQC16jjz4xWP; Mon, 9 Oct 2023 17:49:22 +1100 (AEDT) From: =?utf-8?q?C=C3=A9dric_Le_Goater?= To: qemu-devel@nongnu.org Cc: Alex Williamson , Jing Liu , =?utf-8?q?C=C3=A9dric_Le_Goater?= Subject: [PULL 6/6] vfio/pci: enable MSI-X in interrupt restoring on dynamic allocation Date: Mon, 9 Oct 2023 08:49:00 +0200 Message-ID: <20231009064900.1465361-7-clg@redhat.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20231009064900.1465361-1-clg@redhat.com> References: <20231009064900.1465361-1-clg@redhat.com> MIME-Version: 1.0 Received-SPF: pass client-ip=150.107.74.76; envelope-from=SRS0=qD0P=FX=redhat.com=clg@ozlabs.org; helo=gandalf.ozlabs.org X-Spam_score_int: -16 X-Spam_score: -1.7 X-Spam_bar: - X-Spam_report: (-1.7 / 5.0 requ) BAYES_00=-1.9, HEADER_FROM_DIFFERENT_DOMAINS=0.249, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=no autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org From: Jing Liu During migration restoring, vfio_enable_vectors() is called to restore enabling MSI-X interrupts for assigned devices. It sets the range from 0 to nr_vectors to kernel to enable MSI-X and the vectors unmasked in guest. During the MSI-X enabling, all the vectors within the range are allocated according to the VFIO_DEVICE_SET_IRQS ioctl. When dynamic MSI-X allocation is supported, we only want the guest unmasked vectors being allocated and enabled. Use vector 0 with an invalid fd to get MSI-X enabled, after that, all the vectors can be allocated in need. Signed-off-by: Jing Liu Reviewed-by: Cédric Le Goater Reviewed-by: Alex Williamson Signed-off-by: Cédric Le Goater --- hw/vfio/pci.c | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c index ad508abd6f382d518871191017859c4c4c8fd4f9..898296fd5441b07fded7e50a53b2927683dd4b1a 100644 --- a/hw/vfio/pci.c +++ b/hw/vfio/pci.c @@ -402,6 +402,23 @@ static int vfio_enable_vectors(VFIOPCIDevice *vdev, bool msix) int ret = 0, i, argsz; int32_t *fds; + /* + * If dynamic MSI-X allocation is supported, the vectors to be allocated + * and enabled can be scattered. Before kernel enabling MSI-X, setting + * nr_vectors causes all these vectors to be allocated on host. + * + * To keep allocation as needed, use vector 0 with an invalid fd to get + * MSI-X enabled first, then set vectors with a potentially sparse set of + * eventfds to enable interrupts only when enabled in guest. + */ + if (msix && !vdev->msix->noresize) { + ret = vfio_enable_msix_no_vec(vdev); + + if (ret) { + return ret; + } + } + argsz = sizeof(*irq_set) + (vdev->nr_vectors * sizeof(*fds)); irq_set = g_malloc0(argsz);