From patchwork Tue Sep 14 01:53:25 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kunkun Jiang X-Patchwork-Id: 12491235 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id EFCD2C433EF for ; Tue, 14 Sep 2021 01:55:23 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id B83AA60FDA for ; Tue, 14 Sep 2021 01:55:23 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org B83AA60FDA Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=huawei.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=nongnu.org Received: from localhost ([::1]:48172 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mPxfN-0005wP-MV for qemu-devel@archiver.kernel.org; Mon, 13 Sep 2021 21:55:21 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:40008) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mPxeF-0004VA-3x for qemu-devel@nongnu.org; Mon, 13 Sep 2021 21:54:11 -0400 Received: from szxga02-in.huawei.com ([45.249.212.188]:3135) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mPxeD-0005cK-2k for qemu-devel@nongnu.org; Mon, 13 Sep 2021 21:54:10 -0400 Received: from dggemv703-chm.china.huawei.com (unknown [172.30.72.53]) by szxga02-in.huawei.com (SkyGuard) with ESMTP id 4H7mVm3mPlz8yWt; Tue, 14 Sep 2021 09:49:36 +0800 (CST) Received: from dggema765-chm.china.huawei.com (10.1.198.207) by dggemv703-chm.china.huawei.com (10.3.19.46) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256) id 15.1.2308.8; Tue, 14 Sep 2021 09:54:00 +0800 Received: from DESKTOP-6NKE0BC.china.huawei.com (10.174.185.210) by dggema765-chm.china.huawei.com (10.1.198.207) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.2308.8; Tue, 14 Sep 2021 09:54:00 +0800 From: Kunkun Jiang To: Alex Williamson , Eric Auger , Kirti Wankhede , "open list:All patches CC here" Subject: [PATCH v2 1/2] vfio/pci: Fix vfio-pci sub-page MMIO BAR mmaping in live migration Date: Tue, 14 Sep 2021 09:53:25 +0800 Message-ID: <20210914015326.1494-2-jiangkunkun@huawei.com> X-Mailer: git-send-email 2.26.2.windows.1 In-Reply-To: <20210914015326.1494-1-jiangkunkun@huawei.com> References: <20210914015326.1494-1-jiangkunkun@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.174.185.210] X-ClientProxiedBy: dggems705-chm.china.huawei.com (10.3.19.182) To dggema765-chm.china.huawei.com (10.1.198.207) X-CFilter-Loop: Reflected Received-SPF: pass client-ip=45.249.212.188; envelope-from=jiangkunkun@huawei.com; helo=szxga02-in.huawei.com X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: wanghaibin.wang@huawei.com, Kunkun Jiang , tangnianyao@huawei.com, ganqixin@huawei.com Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" We expand MemoryRegions of vfio-pci sub-page MMIO BARs to vfio_pci_write_config to improve IO performance. The MemoryRegions of destination VM will not be expanded successful in live migration, because their addresses have been updated in vmstate_load_state (vfio_pci_load_config). So iterate BARs in vfio_pci_write_config and try to update sub-page BARs. Fixes: c5e2fb3ce4d (vfio: Add save and load functions for VFIO PCI devices) Reported-by: Nianyao Tang Reported-by: Qixin Gan Signed-off-by: Kunkun Jiang --- hw/vfio/pci.c | 15 ++++++++++++++- 1 file changed, 14 insertions(+), 1 deletion(-) diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c index e1ea1d8a23..43c7e93153 100644 --- a/hw/vfio/pci.c +++ b/hw/vfio/pci.c @@ -2453,7 +2453,12 @@ static int vfio_pci_load_config(VFIODevice *vbasedev, QEMUFile *f) { VFIOPCIDevice *vdev = container_of(vbasedev, VFIOPCIDevice, vbasedev); PCIDevice *pdev = &vdev->pdev; - int ret; + pcibus_t old_addr[PCI_NUM_REGIONS - 1]; + int bar, ret; + + for (bar = 0; bar < PCI_ROM_SLOT; bar++) { + old_addr[bar] = pdev->io_regions[bar].addr; + } ret = vmstate_load_state(f, &vmstate_vfio_pci_config, vdev, 1); if (ret) { @@ -2463,6 +2468,14 @@ static int vfio_pci_load_config(VFIODevice *vbasedev, QEMUFile *f) vfio_pci_write_config(pdev, PCI_COMMAND, pci_get_word(pdev->config + PCI_COMMAND), 2); + for (bar = 0; bar < PCI_ROM_SLOT; bar++) { + if (old_addr[bar] != pdev->io_regions[bar].addr && + vdev->bars[bar].region.size > 0 && + vdev->bars[bar].region.size < qemu_real_host_page_size) { + vfio_sub_page_bar_update_mapping(pdev, bar); + } + } + if (msi_enabled(pdev)) { vfio_msi_enable(vdev); } else if (msix_enabled(pdev)) { From patchwork Tue Sep 14 01:53:26 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kunkun Jiang X-Patchwork-Id: 12491237 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7C5D1C433EF for ; Tue, 14 Sep 2021 01:57:03 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 0E65360295 for ; Tue, 14 Sep 2021 01:57:03 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 0E65360295 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=huawei.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=nongnu.org Received: from localhost ([::1]:51424 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mPxh0-0008Et-95 for qemu-devel@archiver.kernel.org; Mon, 13 Sep 2021 21:57:02 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:39998) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mPxeD-0004TE-UJ for qemu-devel@nongnu.org; Mon, 13 Sep 2021 21:54:09 -0400 Received: from szxga01-in.huawei.com ([45.249.212.187]:2535) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mPxeB-0005cJ-RI for qemu-devel@nongnu.org; Mon, 13 Sep 2021 21:54:09 -0400 Received: from dggemv711-chm.china.huawei.com (unknown [172.30.72.57]) by szxga01-in.huawei.com (SkyGuard) with ESMTP id 4H7mZl38KVzVkJY; Tue, 14 Sep 2021 09:53:03 +0800 (CST) Received: from dggema765-chm.china.huawei.com (10.1.198.207) by dggemv711-chm.china.huawei.com (10.1.198.66) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256) id 15.1.2308.8; Tue, 14 Sep 2021 09:54:02 +0800 Received: from DESKTOP-6NKE0BC.china.huawei.com (10.174.185.210) by dggema765-chm.china.huawei.com (10.1.198.207) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.2308.8; Tue, 14 Sep 2021 09:54:02 +0800 From: Kunkun Jiang To: Alex Williamson , Eric Auger , Kirti Wankhede , "open list:All patches CC here" Subject: [PATCH v2 2/2] vfio/common: Add trace point when a MMIO RAM section less than PAGE_SIZE Date: Tue, 14 Sep 2021 09:53:26 +0800 Message-ID: <20210914015326.1494-3-jiangkunkun@huawei.com> X-Mailer: git-send-email 2.26.2.windows.1 In-Reply-To: <20210914015326.1494-1-jiangkunkun@huawei.com> References: <20210914015326.1494-1-jiangkunkun@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.174.185.210] X-ClientProxiedBy: dggems705-chm.china.huawei.com (10.3.19.182) To dggema765-chm.china.huawei.com (10.1.198.207) X-CFilter-Loop: Reflected Received-SPF: pass client-ip=45.249.212.187; envelope-from=jiangkunkun@huawei.com; helo=szxga01-in.huawei.com X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: wanghaibin.wang@huawei.com, Kunkun Jiang , tangnianyao@huawei.com, ganqixin@huawei.com Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" The MSI-X structures of some devices and other non-MSI-X structures are in the same BAR. They may share one host page, especially in the case of large page granularity, such as 64K. For example, MSIX-Table size of 82599 NIC is 0x30 and the offset in Bar 3(size 64KB) is 0x0. If host page size is 64KB. vfio_listener_region_add() will be called to map the remaining range (0x30-0xffff). And it will return early at 'int128_ge((int128_make64(iova), llend))' and hasn't any message. Let's add a trace point to informed users like commit 5c08600547c0 ("vfio: Use a trace point when a RAM section cannot be DMA mapped") did. Signed-off-by: Kunkun Jiang --- hw/vfio/common.c | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/hw/vfio/common.c b/hw/vfio/common.c index 8728d4d5c2..2fc6213c0f 100644 --- a/hw/vfio/common.c +++ b/hw/vfio/common.c @@ -892,6 +892,13 @@ static void vfio_listener_region_add(MemoryListener *listener, llend = int128_and(llend, int128_exts64(qemu_real_host_page_mask)); if (int128_ge(int128_make64(iova), llend)) { + if (memory_region_is_ram_device(section->mr)) { + trace_vfio_listener_region_add_no_dma_map( + memory_region_name(section->mr), + section->offset_within_address_space, + int128_getlo(section->size), + qemu_real_host_page_size); + } return; } end = int128_get64(int128_sub(llend, int128_one()));