From patchwork Mon Feb 12 13:56:36 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joao Martins X-Patchwork-Id: 13553296 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id CF1D3C48BC0 for ; Mon, 12 Feb 2024 13:58:04 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rZWoO-0006hq-Ju; Mon, 12 Feb 2024 08:57:33 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rZWnz-0006Uz-UP for qemu-devel@nongnu.org; Mon, 12 Feb 2024 08:57:09 -0500 Received: from mx0b-00069f02.pphosted.com ([205.220.177.32]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rZWnu-0007CE-7n for qemu-devel@nongnu.org; Mon, 12 Feb 2024 08:57:05 -0500 Received: from pps.filterd (m0246631.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 41CDJ3vJ031397; Mon, 12 Feb 2024 13:56:57 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=corp-2023-11-20; bh=Z4wJsEWEuirpA39vjBbD+wxmAqgUzY7zQonrdlIZMp0=; b=D0LZcI4X5FwSrWHrFZGCkrY1Yq4aoh4lIqCJEGnniWiLlOJV4YkCJ/u/xcDi7RPZn7r3 CEQni9re/bmsxDlqmJREYVuBcWRE+awuioFoFZiAWn6QRwBSZDB9Evnq+yXGYBSS6auk cnu9vPmcoILq5KgWaL8AX4C0C8XII38XeQqmwzKpjQ0vSc2XBvCXBkotw37PRfT4gN85 pUMq/fRbIUX5sb/C9YPfIhnS3mZ8olK0op7ozOvgeNARO/hcJ8iM+fpV2HdK2EBuu88i 7ce5udTF8HDP7m392NFqZ/fmI3Ia4c7TZb3g7lP8Wit/jH+h8nx4MUohzCMJ9TzanJ4p aQ== Received: from iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta01.appoci.oracle.com [130.35.100.223]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3w7m40g31j-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 12 Feb 2024 13:56:56 +0000 Received: from pps.filterd (iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 41CCI70P024614; Mon, 12 Feb 2024 13:56:56 GMT Received: from pps.reinject (localhost [127.0.0.1]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 3w5ykc20tq-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 12 Feb 2024 13:56:56 +0000 Received: from iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 41CDuqi5026744; Mon, 12 Feb 2024 13:56:55 GMT Received: from joaomart-mac.nl.oracle.com (dhcp-10-175-27-30.vpn.oracle.com [10.175.27.30]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTP id 3w5ykc20rs-2; Mon, 12 Feb 2024 13:56:55 +0000 From: Joao Martins To: qemu-devel@nongnu.org Cc: Yi Liu , Eric Auger , Zhenzhong Duan , Alex Williamson , Cedric Le Goater , Paolo Bonzini , "Daniel P . Berrange" , Eduardo Habkost , Eric Blake , Markus Armbruster , Jason Gunthorpe , Avihai Horon , Joao Martins Subject: [PATCH RFCv2 1/8] backends/iommufd: Introduce helper function iommufd_device_get_hw_capabilities() Date: Mon, 12 Feb 2024 13:56:36 +0000 Message-Id: <20240212135643.5858-2-joao.m.martins@oracle.com> In-Reply-To: <20240212135643.5858-1-joao.m.martins@oracle.com> References: <20240212135643.5858-1-joao.m.martins@oracle.com> MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.272,Aquarius:18.0.1011,Hydra:6.0.619,FMLib:17.11.176.26 definitions=2024-02-12_10,2024-02-12_03,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 adultscore=0 bulkscore=0 mlxlogscore=698 malwarescore=0 mlxscore=0 spamscore=0 suspectscore=0 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2311290000 definitions=main-2402120105 X-Proofpoint-GUID: fOwN1-OCnkvPzw8JFyjK-UAepyoR-rAy X-Proofpoint-ORIG-GUID: fOwN1-OCnkvPzw8JFyjK-UAepyoR-rAy Received-SPF: pass client-ip=205.220.177.32; envelope-from=joao.m.martins@oracle.com; helo=mx0b-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org The new helper will fetch vendor agnostic IOMMU capabilities supported both by hardware and software. Right now it is only iommu dirty tracking. Signed-off-by: Joao Martins --- backends/iommufd.c | 25 +++++++++++++++++++++++++ include/sysemu/iommufd.h | 2 ++ 2 files changed, 27 insertions(+) diff --git a/backends/iommufd.c b/backends/iommufd.c index d92791bba935..8486894f1b3f 100644 --- a/backends/iommufd.c +++ b/backends/iommufd.c @@ -237,3 +237,28 @@ void iommufd_device_init(IOMMUFDDevice *idev) host_iommu_base_device_init(&idev->base, HID_IOMMUFD, sizeof(IOMMUFDDevice)); } + +int iommufd_device_get_hw_capabilities(IOMMUFDDevice *idev, uint64_t *caps, + Error **errp) +{ + struct iommu_hw_info info = { + .size = sizeof(info), + .flags = 0, + .dev_id = idev->devid, + .data_len = 0, + .__reserved = 0, + .data_uptr = 0, + .out_capabilities = 0, + }; + int ret; + + ret = ioctl(idev->iommufd->fd, IOMMU_GET_HW_INFO, &info); + if (ret) { + error_setg_errno(errp, errno, + "Failed to get hardware info capabilities"); + } else { + *caps = info.out_capabilities; + } + + return ret; +} diff --git a/include/sysemu/iommufd.h b/include/sysemu/iommufd.h index c3f346976039..4afe97307dbe 100644 --- a/include/sysemu/iommufd.h +++ b/include/sysemu/iommufd.h @@ -47,4 +47,6 @@ typedef struct IOMMUFDDevice { } IOMMUFDDevice; void iommufd_device_init(IOMMUFDDevice *idev); +int iommufd_device_get_hw_capabilities(IOMMUFDDevice *idev, uint64_t *caps, + Error **errp); #endif From patchwork Mon Feb 12 13:56:37 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joao Martins X-Patchwork-Id: 13553298 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id C475AC48297 for ; Mon, 12 Feb 2024 13:58:04 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rZWoG-0006bK-BV; Mon, 12 Feb 2024 08:57:25 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rZWo3-0006WR-Ev for qemu-devel@nongnu.org; Mon, 12 Feb 2024 08:57:12 -0500 Received: from mx0b-00069f02.pphosted.com ([205.220.177.32]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rZWnx-0007Co-Me for qemu-devel@nongnu.org; Mon, 12 Feb 2024 08:57:08 -0500 Received: from pps.filterd (m0246631.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 41CDJPSn031790; Mon, 12 Feb 2024 13:57:00 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=corp-2023-11-20; bh=MYG69SyNLbthPmaYGIKZeO5BilKjofwiHEQgRnSbyC4=; b=Yi9z1KVx9spbIPH89ImK/xodo2wthWhg+4doA7FMtDFFt3MJno7N/Ou0sEDq30qcoFGu 9PwHFaIu2CRZs6ilH4yv7EB8Lrtd8QYs3kOHIPicqfexs0h2XlmjFgExebHATW7JYHfS zEH9C7LvY4gTcFoPWkxbPK9dWlNuarCA4asBKDBFQdCYI3jQdrfKTodwXafe0KgHWyoJ eeSaYXs2YuYAhbqMgzTJP88T8gdae410dW+1hQY3VrMpn6EEtiOKkLjsPX6psZPegHt5 dLFHBFs+2x9DAD+XOSjd9/PaV3c2D9zLQiIrluVCW5SFWTZk3leY31FsPiZaAJ53wRNg Mg== Received: from iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta01.appoci.oracle.com [130.35.100.223]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3w7m40g31u-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 12 Feb 2024 13:56:59 +0000 Received: from pps.filterd (iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 41CDsIXH024563; Mon, 12 Feb 2024 13:56:59 GMT Received: from pps.reinject (localhost [127.0.0.1]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 3w5ykc20vg-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 12 Feb 2024 13:56:59 +0000 Received: from iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 41CDuqi7026744; Mon, 12 Feb 2024 13:56:58 GMT Received: from joaomart-mac.nl.oracle.com (dhcp-10-175-27-30.vpn.oracle.com [10.175.27.30]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTP id 3w5ykc20rs-3; Mon, 12 Feb 2024 13:56:58 +0000 From: Joao Martins To: qemu-devel@nongnu.org Cc: Yi Liu , Eric Auger , Zhenzhong Duan , Alex Williamson , Cedric Le Goater , Paolo Bonzini , "Daniel P . Berrange" , Eduardo Habkost , Eric Blake , Markus Armbruster , Jason Gunthorpe , Avihai Horon , Joao Martins Subject: [PATCH RFCv2 2/8] vfio/iommufd: Introduce auto domain creation Date: Mon, 12 Feb 2024 13:56:37 +0000 Message-Id: <20240212135643.5858-3-joao.m.martins@oracle.com> In-Reply-To: <20240212135643.5858-1-joao.m.martins@oracle.com> References: <20240212135643.5858-1-joao.m.martins@oracle.com> MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.272,Aquarius:18.0.1011,Hydra:6.0.619,FMLib:17.11.176.26 definitions=2024-02-12_10,2024-02-12_03,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 adultscore=0 bulkscore=0 mlxlogscore=850 malwarescore=0 mlxscore=0 spamscore=0 suspectscore=0 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2311290000 definitions=main-2402120105 X-Proofpoint-GUID: VLDUzPKLAXB6tYr-LMpgLxPjtFbQffm5 X-Proofpoint-ORIG-GUID: VLDUzPKLAXB6tYr-LMpgLxPjtFbQffm5 Received-SPF: pass client-ip=205.220.177.32; envelope-from=joao.m.martins@oracle.com; helo=mx0b-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org There's generally two modes of operation for IOMMUFD: * The simple user API which intends to perform relatively simple things with IOMMUs e.g. DPDK. It generally creates an IOAS and attach to VFIO and mainly performs IOAS_MAP and UNMAP. * The native IOMMUFD API where you have fine grained control of the IOMMU domain and model it accordingly. This is where most new feature are being steered to. For dirty tracking 2) is required, as it needs to ensure that the stage-2/parent IOMMU domain will only attach devices that support dirty tracking (so far it is all homogeneous in x86, likely not the case for smmuv3). Such invariant on dirty tracking provides a useful guarantee to VMMs that will refuse incompatible device attachments for IOMMU domains. For dirty tracking such property is enabled/enforced via HWPT_ALLOC, which is responsible for creating an IOMMU domain. This is contrast to the 'simple API' where the IOMMU domain is created by IOMMUFD automatically when it attaches to VFIO (usually referred as autodomains) To support dirty tracking with the advanced IOMMUFD API, it needs similar logic, where IOMMU domains are created and devices attached to compatible domains. Essentially mimmicing kernel iommufd_device_auto_get_domain(). If this fails (i.e. mdevs) it falls back to IOAS attach. Signed-off-by: Joao Martins --- Right now the only alternative to a userspace autodomains implementation is to mimmicing all the flags being added to HWPT_ALLOC but into VFIO IOAS attach. So opted for autodomains userspace approach to avoid the duplication of hwpt-alloc flags vs attach-ioas flags. I lack mdev real drivers atm, so testing with those is still TBD. Opinions, comments, welcome! --- backends/iommufd.c | 29 +++++++++++++ backends/trace-events | 1 + hw/vfio/iommufd.c | 78 +++++++++++++++++++++++++++++++++++ include/hw/vfio/vfio-common.h | 9 ++++ include/sysemu/iommufd.h | 4 ++ 5 files changed, 121 insertions(+) diff --git a/backends/iommufd.c b/backends/iommufd.c index 8486894f1b3f..2970135af4b9 100644 --- a/backends/iommufd.c +++ b/backends/iommufd.c @@ -211,6 +211,35 @@ int iommufd_backend_unmap_dma(IOMMUFDBackend *be, uint32_t ioas_id, return ret; } +int iommufd_backend_alloc_hwpt(int iommufd, uint32_t dev_id, + uint32_t pt_id, uint32_t flags, + uint32_t data_type, uint32_t data_len, + void *data_ptr, uint32_t *out_hwpt) +{ + int ret; + struct iommu_hwpt_alloc alloc_hwpt = { + .size = sizeof(struct iommu_hwpt_alloc), + .flags = flags, + .dev_id = dev_id, + .pt_id = pt_id, + .data_type = data_type, + .data_len = data_len, + .data_uptr = (uint64_t)data_ptr, + .__reserved = 0, + }; + + ret = ioctl(iommufd, IOMMU_HWPT_ALLOC, &alloc_hwpt); + trace_iommufd_backend_alloc_hwpt(iommufd, dev_id, pt_id, flags, data_type, + data_len, (uint64_t)data_ptr, + alloc_hwpt.out_hwpt_id, ret); + if (ret) { + error_report("IOMMU_HWPT_ALLOC failed: %m"); + } else { + *out_hwpt = alloc_hwpt.out_hwpt_id; + } + return !ret ? 0 : -errno; +} + static const TypeInfo iommufd_backend_info = { .name = TYPE_IOMMUFD_BACKEND, .parent = TYPE_OBJECT, diff --git a/backends/trace-events b/backends/trace-events index d45c6e31a67e..f83a276a4253 100644 --- a/backends/trace-events +++ b/backends/trace-events @@ -13,5 +13,6 @@ iommu_backend_set_fd(int fd) "pre-opened /dev/iommu fd=%d" iommufd_backend_map_dma(int iommufd, uint32_t ioas, uint64_t iova, uint64_t size, void *vaddr, bool readonly, int ret) " iommufd=%d ioas=%d iova=0x%"PRIx64" size=0x%"PRIx64" addr=%p readonly=%d (%d)" iommufd_backend_unmap_dma_non_exist(int iommufd, uint32_t ioas, uint64_t iova, uint64_t size, int ret) " Unmap nonexistent mapping: iommufd=%d ioas=%d iova=0x%"PRIx64" size=0x%"PRIx64" (%d)" iommufd_backend_unmap_dma(int iommufd, uint32_t ioas, uint64_t iova, uint64_t size, int ret) " iommufd=%d ioas=%d iova=0x%"PRIx64" size=0x%"PRIx64" (%d)" +iommufd_backend_alloc_hwpt(int iommufd, uint32_t dev_id, uint32_t pt_id, uint32_t flags, uint32_t hwpt_type, uint32_t len, uint64_t data_ptr, uint32_t out_hwpt_id, int ret) " iommufd=%d dev_id=%u pt_id=%u flags=0x%x hwpt_type=%u len=%u data_ptr=0x%"PRIx64" out_hwpt=%u (%d)" iommufd_backend_alloc_ioas(int iommufd, uint32_t ioas, int ret) " iommufd=%d ioas=%d (%d)" iommufd_backend_free_id(int iommufd, uint32_t id, int ret) " iommufd=%d id=%d (%d)" diff --git a/hw/vfio/iommufd.c b/hw/vfio/iommufd.c index 7d39d7a5fa51..ca7ec45e725c 100644 --- a/hw/vfio/iommufd.c +++ b/hw/vfio/iommufd.c @@ -219,10 +219,82 @@ static int iommufd_cdev_detach_ioas_hwpt(VFIODevice *vbasedev, Error **errp) return ret; } +static int iommufd_cdev_autodomains_get(VFIODevice *vbasedev, + VFIOIOMMUFDContainer *container, + Error **errp) +{ + int iommufd = vbasedev->iommufd_dev.iommufd->fd; + VFIOIOASHwpt *hwpt; + Error *err = NULL; + int ret = -EINVAL; + uint32_t hwpt_id; + + /* Try to find a domain */ + QLIST_FOREACH(hwpt, &container->hwpt_list, next) { + ret = iommufd_cdev_attach_ioas_hwpt(vbasedev, hwpt->hwpt_id, &err); + if (ret) { + /* -EINVAL means the domain is incompatible with the device. */ + if (ret == -EINVAL) { + continue; + } + return ret; + } else { + vbasedev->hwpt = hwpt; + QLIST_INSERT_HEAD(&hwpt->device_list, vbasedev, hwpt_next); + return 0; + } + } + + ret = iommufd_backend_alloc_hwpt(iommufd, + vbasedev->iommufd_dev.devid, + container->ioas_id, 0, 0, 0, + NULL, &hwpt_id); + if (ret) { + error_append_hint(&err, + "Failed to allocate HWPT for device %s. Fallback to IOAS attach\n", + vbasedev->name); + warn_report_err(err); + return ret; + } + + hwpt = g_malloc0(sizeof(*hwpt)); + hwpt->hwpt_id = hwpt_id; + QLIST_INIT(&hwpt->device_list); + + ret = iommufd_cdev_attach_ioas_hwpt(vbasedev, hwpt->hwpt_id, &err); + if (ret) { + iommufd_backend_free_id(container->be, hwpt->hwpt_id); + g_free(hwpt); + return ret; + } + + vbasedev->hwpt = hwpt; + QLIST_INSERT_HEAD(&hwpt->device_list, vbasedev, hwpt_next); + QLIST_INSERT_HEAD(&container->hwpt_list, hwpt, next); + return 0; +} + +static void iommufd_cdev_autodomains_put(VFIODevice *vbasedev, + VFIOIOMMUFDContainer *container) +{ + VFIOIOASHwpt *hwpt = vbasedev->hwpt; + + QLIST_REMOVE(vbasedev, hwpt_next); + QLIST_REMOVE(hwpt, next); + if (QLIST_EMPTY(&hwpt->device_list)) { + iommufd_backend_free_id(container->be, hwpt->hwpt_id); + g_free(hwpt); + } +} + static int iommufd_cdev_attach_container(VFIODevice *vbasedev, VFIOIOMMUFDContainer *container, Error **errp) { + if (!iommufd_cdev_autodomains_get(vbasedev, container, errp)) { + return 0; + } + return iommufd_cdev_attach_ioas_hwpt(vbasedev, container->ioas_id, errp); } @@ -231,6 +303,11 @@ static void iommufd_cdev_detach_container(VFIODevice *vbasedev, { Error *err = NULL; + if (vbasedev->hwpt) { + iommufd_cdev_autodomains_put(vbasedev, container); + return; + } + if (iommufd_cdev_detach_ioas_hwpt(vbasedev, &err)) { error_report_err(err); } @@ -370,6 +447,7 @@ static int iommufd_cdev_attach(const char *name, VFIODevice *vbasedev, container = g_malloc0(sizeof(*container)); container->be = vbasedev->iommufd_dev.iommufd; container->ioas_id = ioas_id; + QLIST_INIT(&container->hwpt_list); bcontainer = &container->bcontainer; vfio_container_init(bcontainer, space, iommufd_vioc); diff --git a/include/hw/vfio/vfio-common.h b/include/hw/vfio/vfio-common.h index 9c4b60c906d9..7f7d823221e2 100644 --- a/include/hw/vfio/vfio-common.h +++ b/include/hw/vfio/vfio-common.h @@ -93,10 +93,17 @@ typedef struct VFIOHostDMAWindow { typedef struct IOMMUFDBackend IOMMUFDBackend; +typedef struct VFIOIOASHwpt { + uint32_t hwpt_id; + QLIST_HEAD(, VFIODevice) device_list; + QLIST_ENTRY(VFIOIOASHwpt) next; +} VFIOIOASHwpt; + typedef struct VFIOIOMMUFDContainer { VFIOContainerBase bcontainer; IOMMUFDBackend *be; uint32_t ioas_id; + QLIST_HEAD(, VFIOIOASHwpt) hwpt_list; } VFIOIOMMUFDContainer; /* Abstraction of host IOMMU legacy device */ @@ -136,6 +143,8 @@ typedef struct VFIODevice { IOMMULegacyDevice legacy_dev; IOMMUFDDevice iommufd_dev; }; + VFIOIOASHwpt *hwpt; + QLIST_ENTRY(VFIODevice) hwpt_next; } VFIODevice; QEMU_BUILD_BUG_ON(offsetof(VFIODevice, legacy_dev.base) != diff --git a/include/sysemu/iommufd.h b/include/sysemu/iommufd.h index 4afe97307dbe..1966b75caae2 100644 --- a/include/sysemu/iommufd.h +++ b/include/sysemu/iommufd.h @@ -49,4 +49,8 @@ typedef struct IOMMUFDDevice { void iommufd_device_init(IOMMUFDDevice *idev); int iommufd_device_get_hw_capabilities(IOMMUFDDevice *idev, uint64_t *caps, Error **errp); +int iommufd_backend_alloc_hwpt(int iommufd, uint32_t dev_id, + uint32_t pt_id, uint32_t flags, + uint32_t data_type, uint32_t data_len, + void *data_ptr, uint32_t *out_hwpt); #endif From patchwork Mon Feb 12 13:56:38 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joao Martins X-Patchwork-Id: 13553295 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id C47C5C4829D for ; Mon, 12 Feb 2024 13:58:04 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rZWoQ-0006qZ-Ie; Mon, 12 Feb 2024 08:57:34 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rZWo5-0006WX-A7 for qemu-devel@nongnu.org; Mon, 12 Feb 2024 08:57:14 -0500 Received: from mx0b-00069f02.pphosted.com ([205.220.177.32]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rZWnz-0007DX-OZ for qemu-devel@nongnu.org; Mon, 12 Feb 2024 08:57:09 -0500 Received: from pps.filterd (m0246631.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 41CDJU1X031824; Mon, 12 Feb 2024 13:57:04 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=corp-2023-11-20; bh=ZqpDDFb94Rtg9GoTde3iZeO3BEp0KehjNyFIO+jKnqw=; b=AVWLTioHL0r3fLc0qeMPNtfE/QeMZ5lkpr5gXkWmBFm3GuOLvO8aEBZTvVWUc6JdAR3n K0oOxlGqvkYJx8oGhYNB8vecEC9m8+TclvHyCoQz9yg/xj/ES/3kiM+rgPUGiO8VOTZ9 /ljkN+XaDvCu2AJatTs3GkippqWiwSjt0egQ+q/2CVo4H/SuHcThyITDfoskGmGKUx/C hPVNhgaokhbUb/JhtzeqUXUDcACg4EDH3aAVyN+9h+KypSZBY95ZW37QTJbORCbSjz9b jnyFDt0vHp6hyz2l4irM4UyhBRZ6vfp1sNqsFH6KwELxGhpOgx+qOhtZRPH9CEf5Uq6Y aQ== Received: from iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta01.appoci.oracle.com [130.35.100.223]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3w7m40g323-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 12 Feb 2024 13:57:03 +0000 Received: from pps.filterd (iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 41CDkHL3024638; Mon, 12 Feb 2024 13:57:03 GMT Received: from pps.reinject (localhost [127.0.0.1]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 3w5ykc20xh-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 12 Feb 2024 13:57:03 +0000 Received: from iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 41CDuqi9026744; Mon, 12 Feb 2024 13:57:02 GMT Received: from joaomart-mac.nl.oracle.com (dhcp-10-175-27-30.vpn.oracle.com [10.175.27.30]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTP id 3w5ykc20rs-4; Mon, 12 Feb 2024 13:57:02 +0000 From: Joao Martins To: qemu-devel@nongnu.org Cc: Yi Liu , Eric Auger , Zhenzhong Duan , Alex Williamson , Cedric Le Goater , Paolo Bonzini , "Daniel P . Berrange" , Eduardo Habkost , Eric Blake , Markus Armbruster , Jason Gunthorpe , Avihai Horon , Joao Martins Subject: [PATCH RFCv2 3/8] vfio/iommufd: Probe and request hwpt dirty tracking capability Date: Mon, 12 Feb 2024 13:56:38 +0000 Message-Id: <20240212135643.5858-4-joao.m.martins@oracle.com> In-Reply-To: <20240212135643.5858-1-joao.m.martins@oracle.com> References: <20240212135643.5858-1-joao.m.martins@oracle.com> MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.272,Aquarius:18.0.1011,Hydra:6.0.619,FMLib:17.11.176.26 definitions=2024-02-12_10,2024-02-12_03,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 adultscore=0 bulkscore=0 mlxlogscore=999 malwarescore=0 mlxscore=0 spamscore=0 suspectscore=0 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2311290000 definitions=main-2402120105 X-Proofpoint-GUID: Z3_RBTLr-c4QDTNygh7Tt0MB_KFI_V9b X-Proofpoint-ORIG-GUID: Z3_RBTLr-c4QDTNygh7Tt0MB_KFI_V9b Received-SPF: pass client-ip=205.220.177.32; envelope-from=joao.m.martins@oracle.com; helo=mx0b-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Probe hardware dirty tracking support by querying device hw capabilities via IOMMUFD_GET_HW_INFO. In preparation to using the dirty tracking UAPI, request dirty tracking in the HWPT flags when the device doesn't support dirty page tracking or has it disabled; or when support when the VF backing IOMMU supports dirty tracking. The latter is in the possibility of a device being attached that doesn't have a dirty tracker. Signed-off-by: Joao Martins --- hw/vfio/common.c | 18 ++++++++++++++++++ hw/vfio/iommufd.c | 25 ++++++++++++++++++++++++- include/hw/vfio/vfio-common.h | 2 ++ 3 files changed, 44 insertions(+), 1 deletion(-) diff --git a/hw/vfio/common.c b/hw/vfio/common.c index f7f85160be88..d8fc7077f839 100644 --- a/hw/vfio/common.c +++ b/hw/vfio/common.c @@ -216,6 +216,24 @@ bool vfio_devices_all_device_dirty_tracking(const VFIOContainerBase *bcontainer) return true; } +bool vfio_device_migration_supported(VFIODevice *vbasedev) +{ + if (!vbasedev->migration) { + return false; + } + + return vbasedev->migration->mig_flags & VFIO_MIGRATION_STOP_COPY; +} + +bool vfio_device_dirty_pages_supported(VFIODevice *vbasedev) +{ + if (vbasedev->pre_copy_dirty_page_tracking == ON_OFF_AUTO_OFF) { + return false; + } + + return !vbasedev->dirty_pages_supported; +} + /* * Check if all VFIO devices are running and migration is active, which is * essentially equivalent to the migration being in pre-copy phase. diff --git a/hw/vfio/iommufd.c b/hw/vfio/iommufd.c index ca7ec45e725c..edacb6d72748 100644 --- a/hw/vfio/iommufd.c +++ b/hw/vfio/iommufd.c @@ -219,11 +219,26 @@ static int iommufd_cdev_detach_ioas_hwpt(VFIODevice *vbasedev, Error **errp) return ret; } +static bool iommufd_dirty_pages_supported(IOMMUFDDevice *iommufd_dev, + Error **errp) +{ + uint64_t caps; + int r; + + r = iommufd_device_get_hw_capabilities(iommufd_dev, &caps, errp); + if (r) { + return false; + } + + return caps & IOMMU_HW_CAP_DIRTY_TRACKING; +} + static int iommufd_cdev_autodomains_get(VFIODevice *vbasedev, VFIOIOMMUFDContainer *container, Error **errp) { int iommufd = vbasedev->iommufd_dev.iommufd->fd; + uint32_t flags = 0; VFIOIOASHwpt *hwpt; Error *err = NULL; int ret = -EINVAL; @@ -245,9 +260,15 @@ static int iommufd_cdev_autodomains_get(VFIODevice *vbasedev, } } + if ((vfio_device_migration_supported(vbasedev) && + !vfio_device_dirty_pages_supported(vbasedev)) || + iommufd_dirty_pages_supported(&vbasedev->iommufd_dev, &err)) { + flags = IOMMU_HWPT_ALLOC_DIRTY_TRACKING; + } + ret = iommufd_backend_alloc_hwpt(iommufd, vbasedev->iommufd_dev.devid, - container->ioas_id, 0, 0, 0, + container->ioas_id, flags, 0, 0, NULL, &hwpt_id); if (ret) { error_append_hint(&err, @@ -271,6 +292,8 @@ static int iommufd_cdev_autodomains_get(VFIODevice *vbasedev, vbasedev->hwpt = hwpt; QLIST_INSERT_HEAD(&hwpt->device_list, vbasedev, hwpt_next); QLIST_INSERT_HEAD(&container->hwpt_list, hwpt, next); + container->bcontainer.dirty_pages_supported = + (flags & IOMMU_HWPT_ALLOC_DIRTY_TRACKING); return 0; } diff --git a/include/hw/vfio/vfio-common.h b/include/hw/vfio/vfio-common.h index 7f7d823221e2..a3e691c126c6 100644 --- a/include/hw/vfio/vfio-common.h +++ b/include/hw/vfio/vfio-common.h @@ -271,6 +271,8 @@ bool vfio_devices_all_running_and_mig_active(const VFIOContainerBase *bcontainer); bool vfio_devices_all_device_dirty_tracking(const VFIOContainerBase *bcontainer); +bool vfio_device_migration_supported(VFIODevice *vbasedev); +bool vfio_device_dirty_pages_supported(VFIODevice *vbasedev); int vfio_devices_query_dirty_bitmap(const VFIOContainerBase *bcontainer, VFIOBitmap *vbmap, hwaddr iova, hwaddr size); From patchwork Mon Feb 12 13:56:39 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joao Martins X-Patchwork-Id: 13553299 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 510C2C4829B for ; Mon, 12 Feb 2024 13:58:36 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rZWom-0006zF-F6; Mon, 12 Feb 2024 08:57:58 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rZWo8-0006XF-59 for qemu-devel@nongnu.org; Mon, 12 Feb 2024 08:57:16 -0500 Received: from mx0b-00069f02.pphosted.com ([205.220.177.32]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rZWo3-0007ER-6B for qemu-devel@nongnu.org; Mon, 12 Feb 2024 08:57:13 -0500 Received: from pps.filterd (m0333520.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 41CDo2xU000440; Mon, 12 Feb 2024 13:57:06 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=corp-2023-11-20; bh=NKYw2KnbG7Z3DMtUsjDy0vLuRffYtT6VEU5hwHoz/To=; b=R2a4wCixrkFccEU2MwY2CeNB6advwdYxwk9/GtBsATDZ29qsfg444qnORCFXgER3/nTo Sl1OyF7TMS6lSRUlrHyvm5TioSzrB+TBrcfcPDCa0QvlP4JJFi0QfwepZke593gsoEXp TRU5c3LUJG+SVmQ5DNSLi99bSMniGA3yAwBbbnEi+N7oyTG455nFaAbHgYATIw0INyTe lnFPkYyLZuZI+48T8fd+IrYWyP0soHubCr40x3yDOPuOKuDKfWT3vKUCC0CZ+KM82URG AqhkgyhEqRa5gnT86rP0c+7EhczYdwfbR54PxxlaX3f9ePw6kKZ2E5LIsMZ0G55MsKqU 4Q== Received: from iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta01.appoci.oracle.com [130.35.100.223]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3w7mj2g0h6-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 12 Feb 2024 13:57:06 +0000 Received: from pps.filterd (iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 41CCrK38024602; Mon, 12 Feb 2024 13:57:06 GMT Received: from pps.reinject (localhost [127.0.0.1]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 3w5ykc2103-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 12 Feb 2024 13:57:06 +0000 Received: from iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 41CDuqiB026744; Mon, 12 Feb 2024 13:57:05 GMT Received: from joaomart-mac.nl.oracle.com (dhcp-10-175-27-30.vpn.oracle.com [10.175.27.30]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTP id 3w5ykc20rs-5; Mon, 12 Feb 2024 13:57:05 +0000 From: Joao Martins To: qemu-devel@nongnu.org Cc: Yi Liu , Eric Auger , Zhenzhong Duan , Alex Williamson , Cedric Le Goater , Paolo Bonzini , "Daniel P . Berrange" , Eduardo Habkost , Eric Blake , Markus Armbruster , Jason Gunthorpe , Avihai Horon , Joao Martins Subject: [PATCH RFCv2 4/8] vfio/iommufd: Implement VFIOIOMMUClass::set_dirty_tracking support Date: Mon, 12 Feb 2024 13:56:39 +0000 Message-Id: <20240212135643.5858-5-joao.m.martins@oracle.com> In-Reply-To: <20240212135643.5858-1-joao.m.martins@oracle.com> References: <20240212135643.5858-1-joao.m.martins@oracle.com> MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.272,Aquarius:18.0.1011,Hydra:6.0.619,FMLib:17.11.176.26 definitions=2024-02-12_10,2024-02-12_03,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 adultscore=0 bulkscore=0 mlxlogscore=999 malwarescore=0 mlxscore=0 spamscore=0 suspectscore=0 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2311290000 definitions=main-2402120105 X-Proofpoint-GUID: tYafEM4yiwjKW6zarmRoeruJwAz6VQ2W X-Proofpoint-ORIG-GUID: tYafEM4yiwjKW6zarmRoeruJwAz6VQ2W Received-SPF: pass client-ip=205.220.177.32; envelope-from=joao.m.martins@oracle.com; helo=mx0b-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org ioctl(iommufd, IOMMU_HWPT_SET_DIRTY_TRACKING, arg) is the UAPI that enables or disables dirty page tracking. It is called on the whole list of iommu domains it is are tracking, and on failure it rolls it back. Signed-off-by: Joao Martins --- backends/iommufd.c | 19 +++++++++++++++++++ backends/trace-events | 1 + hw/vfio/common.c | 7 ++++++- hw/vfio/iommufd.c | 28 ++++++++++++++++++++++++++++ include/sysemu/iommufd.h | 3 +++ 5 files changed, 57 insertions(+), 1 deletion(-) diff --git a/backends/iommufd.c b/backends/iommufd.c index 2970135af4b9..954de61c2da0 100644 --- a/backends/iommufd.c +++ b/backends/iommufd.c @@ -240,6 +240,25 @@ int iommufd_backend_alloc_hwpt(int iommufd, uint32_t dev_id, return !ret ? 0 : -errno; } +int iommufd_backend_set_dirty_tracking(IOMMUFDBackend *be, uint32_t hwpt_id, + bool start) +{ + int ret; + struct iommu_hwpt_set_dirty_tracking set_dirty = { + .size = sizeof(set_dirty), + .hwpt_id = hwpt_id, + .flags = !start ? 0 : IOMMU_HWPT_DIRTY_TRACKING_ENABLE, + }; + + ret = ioctl(be->fd, IOMMU_HWPT_SET_DIRTY_TRACKING, &set_dirty); + trace_iommufd_backend_set_dirty(be->fd, hwpt_id, start, ret); + if (ret) { + error_report("IOMMU_HWPT_SET_DIRTY_TRACKING failed: %s", + strerror(errno)); + } + return !ret ? 0 : -errno; +} + static const TypeInfo iommufd_backend_info = { .name = TYPE_IOMMUFD_BACKEND, .parent = TYPE_OBJECT, diff --git a/backends/trace-events b/backends/trace-events index f83a276a4253..feba2baca5f7 100644 --- a/backends/trace-events +++ b/backends/trace-events @@ -16,3 +16,4 @@ iommufd_backend_unmap_dma(int iommufd, uint32_t ioas, uint64_t iova, uint64_t si iommufd_backend_alloc_hwpt(int iommufd, uint32_t dev_id, uint32_t pt_id, uint32_t flags, uint32_t hwpt_type, uint32_t len, uint64_t data_ptr, uint32_t out_hwpt_id, int ret) " iommufd=%d dev_id=%u pt_id=%u flags=0x%x hwpt_type=%u len=%u data_ptr=0x%"PRIx64" out_hwpt=%u (%d)" iommufd_backend_alloc_ioas(int iommufd, uint32_t ioas, int ret) " iommufd=%d ioas=%d (%d)" iommufd_backend_free_id(int iommufd, uint32_t id, int ret) " iommufd=%d id=%d (%d)" +iommufd_backend_set_dirty(int iommufd, uint32_t hwpt_id, bool start, int ret) " iommufd=%d hwpt=%d enable=%d (%d)" diff --git a/hw/vfio/common.c b/hw/vfio/common.c index d8fc7077f839..a940c0b6ede8 100644 --- a/hw/vfio/common.c +++ b/hw/vfio/common.c @@ -190,7 +190,7 @@ static bool vfio_devices_all_dirty_tracking(VFIOContainerBase *bcontainer) QLIST_FOREACH(vbasedev, &bcontainer->device_list, container_next) { VFIOMigration *migration = vbasedev->migration; - if (!migration) { + if (!migration && !vbasedev->iommufd_dev.iommufd) { return false; } @@ -199,6 +199,11 @@ static bool vfio_devices_all_dirty_tracking(VFIOContainerBase *bcontainer) vfio_device_state_is_precopy(vbasedev))) { return false; } + + if (vbasedev->iommufd_dev.iommufd && + !bcontainer->dirty_pages_supported) { + return false; + } } return true; } diff --git a/hw/vfio/iommufd.c b/hw/vfio/iommufd.c index edacb6d72748..361e659288fd 100644 --- a/hw/vfio/iommufd.c +++ b/hw/vfio/iommufd.c @@ -25,6 +25,7 @@ #include "qemu/cutils.h" #include "qemu/chardev_open.h" #include "pci.h" +#include "migration/migration.h" static int iommufd_cdev_map(const VFIOContainerBase *bcontainer, hwaddr iova, ram_addr_t size, void *vaddr, bool readonly) @@ -115,6 +116,32 @@ static void iommufd_cdev_unbind_and_disconnect(VFIODevice *vbasedev) iommufd_backend_disconnect(vbasedev->iommufd_dev.iommufd); } +static int iommufd_set_dirty_page_tracking(const VFIOContainerBase *bcontainer, + bool start) +{ + const VFIOIOMMUFDContainer *container = + container_of(bcontainer, VFIOIOMMUFDContainer, bcontainer); + int ret; + VFIOIOASHwpt *hwpt; + + QLIST_FOREACH(hwpt, &container->hwpt_list, next) { + ret = iommufd_backend_set_dirty_tracking(container->be, + hwpt->hwpt_id, start); + if (ret) { + goto err; + } + } + + return 0; + +err: + QLIST_FOREACH(hwpt, &container->hwpt_list, next) { + iommufd_backend_set_dirty_tracking(container->be, + hwpt->hwpt_id, !start); + } + return ret; +} + static int iommufd_cdev_getfd(const char *sysfs_path, Error **errp) { long int ret = -ENOTTY; @@ -737,6 +764,7 @@ static void vfio_iommu_iommufd_class_init(ObjectClass *klass, void *data) vioc->detach_device = iommufd_cdev_detach; vioc->pci_hot_reset = iommufd_cdev_pci_hot_reset; vioc->host_iommu_device_init = vfio_cdev_host_iommu_device_init; + vioc->set_dirty_page_tracking = iommufd_set_dirty_page_tracking; }; static const TypeInfo types[] = { diff --git a/include/sysemu/iommufd.h b/include/sysemu/iommufd.h index 1966b75caae2..562c189dd92c 100644 --- a/include/sysemu/iommufd.h +++ b/include/sysemu/iommufd.h @@ -53,4 +53,7 @@ int iommufd_backend_alloc_hwpt(int iommufd, uint32_t dev_id, uint32_t pt_id, uint32_t flags, uint32_t data_type, uint32_t data_len, void *data_ptr, uint32_t *out_hwpt); +int iommufd_backend_set_dirty_tracking(IOMMUFDBackend *be, uint32_t hwpt_id, + bool start); + #endif From patchwork Mon Feb 12 13:56:40 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joao Martins X-Patchwork-Id: 13553297 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id AD737C48BC1 for ; Mon, 12 Feb 2024 13:58:05 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rZWoV-0006tm-Hb; Mon, 12 Feb 2024 08:57:43 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rZWo9-0006Zc-Fe for qemu-devel@nongnu.org; Mon, 12 Feb 2024 08:57:18 -0500 Received: from mx0b-00069f02.pphosted.com ([205.220.177.32]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rZWo5-0007F3-AT for qemu-devel@nongnu.org; Mon, 12 Feb 2024 08:57:15 -0500 Received: from pps.filterd (m0246630.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 41CD4MPB019025; Mon, 12 Feb 2024 13:57:09 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=corp-2023-11-20; bh=UgpOaAovTeO1vDAdQ6kfN2u1nHhSjOolyDkJtfp+V2E=; b=G6OaRhTe+50IwwAntbWCncwoPaPcCg8ri4N/vNK2aCHAFwHxfluhIBz2xoIXrzCNEGvO 6z5fv0zIkZMq6nUKiIEufIBr/bRAtrHYdLmCURHeVJC5gztQE9RD0NcHDrgDIWHEe9Aa 1IDTugfc5QsAuDy/7oqY9Np5AlS5c9hvRXgeSBQatuSLgrRQthEM322sVW3YnLEhVQA0 tWINAnK3yVgUh5111tZ2YYSQBiP5zoeVzU8qVo+Hc2kBDv1KPkc8wYdJ6RSja/tKDOvk 44HmHNtvBym8mAzhcOoOq5jUiDDmQsXjatoP4nNrFSDHt62K+BXOTxUiZW7ECDWrK1Oj yA== Received: from iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta01.appoci.oracle.com [130.35.100.223]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3w7kw1g42p-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 12 Feb 2024 13:57:09 +0000 Received: from pps.filterd (iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 41CDS3JM024776; Mon, 12 Feb 2024 13:57:09 GMT Received: from pps.reinject (localhost [127.0.0.1]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 3w5ykc2122-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 12 Feb 2024 13:57:09 +0000 Received: from iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 41CDuqiD026744; Mon, 12 Feb 2024 13:57:08 GMT Received: from joaomart-mac.nl.oracle.com (dhcp-10-175-27-30.vpn.oracle.com [10.175.27.30]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTP id 3w5ykc20rs-6; Mon, 12 Feb 2024 13:57:08 +0000 From: Joao Martins To: qemu-devel@nongnu.org Cc: Yi Liu , Eric Auger , Zhenzhong Duan , Alex Williamson , Cedric Le Goater , Paolo Bonzini , "Daniel P . Berrange" , Eduardo Habkost , Eric Blake , Markus Armbruster , Jason Gunthorpe , Avihai Horon , Joao Martins Subject: [PATCH RFCv2 5/8] vfio/iommufd: Implement VFIOIOMMUClass::query_dirty_bitmap support Date: Mon, 12 Feb 2024 13:56:40 +0000 Message-Id: <20240212135643.5858-6-joao.m.martins@oracle.com> In-Reply-To: <20240212135643.5858-1-joao.m.martins@oracle.com> References: <20240212135643.5858-1-joao.m.martins@oracle.com> MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.272,Aquarius:18.0.1011,Hydra:6.0.619,FMLib:17.11.176.26 definitions=2024-02-12_10,2024-02-12_03,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 adultscore=0 bulkscore=0 mlxlogscore=999 malwarescore=0 mlxscore=0 spamscore=0 suspectscore=0 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2311290000 definitions=main-2402120105 X-Proofpoint-GUID: ylsCExZkejUxexDwlQQ0CDPDCC0cXTXX X-Proofpoint-ORIG-GUID: ylsCExZkejUxexDwlQQ0CDPDCC0cXTXX Received-SPF: pass client-ip=205.220.177.32; envelope-from=joao.m.martins@oracle.com; helo=mx0b-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org ioctl(iommufd, IOMMU_HWPT_GET_DIRTY_BITMAP, arg) is the UAPI that fetches the bitmap that tells what was dirty in an IOVA range. A single bitmap is allocated and used across all the hwpts sharing an IOAS which is then used in log_sync() to set Qemu global bitmaps. Signed-off-by: Joao Martins --- backends/iommufd.c | 24 ++++++++++++++++++++++++ backends/trace-events | 1 + hw/vfio/iommufd.c | 30 ++++++++++++++++++++++++++++++ include/sysemu/iommufd.h | 3 +++ 4 files changed, 58 insertions(+) diff --git a/backends/iommufd.c b/backends/iommufd.c index 954de61c2da0..dd676d493c37 100644 --- a/backends/iommufd.c +++ b/backends/iommufd.c @@ -259,6 +259,30 @@ int iommufd_backend_set_dirty_tracking(IOMMUFDBackend *be, uint32_t hwpt_id, return !ret ? 0 : -errno; } +int iommufd_backend_get_dirty_bitmap(IOMMUFDBackend *be, uint32_t hwpt_id, + uint64_t iova, ram_addr_t size, + uint64_t page_size, uint64_t *data) +{ + int ret; + struct iommu_hwpt_get_dirty_bitmap get_dirty_bitmap = { + .size = sizeof(get_dirty_bitmap), + .hwpt_id = hwpt_id, + .iova = iova, .length = size, + .page_size = page_size, .data = (uintptr_t)data, + }; + + ret = ioctl(be->fd, IOMMU_HWPT_GET_DIRTY_BITMAP, &get_dirty_bitmap); + trace_iommufd_backend_get_dirty_bitmap(be->fd, hwpt_id, iova, size, + page_size, ret); + if (ret) { + error_report("IOMMU_HWPT_GET_DIRTY_BITMAP (iova: 0x%"PRIx64 + " size: 0x%"PRIx64") failed: %s", iova, + size, strerror(errno)); + } + + return !ret ? 0 : -errno; +} + static const TypeInfo iommufd_backend_info = { .name = TYPE_IOMMUFD_BACKEND, .parent = TYPE_OBJECT, diff --git a/backends/trace-events b/backends/trace-events index feba2baca5f7..11a27cb114b6 100644 --- a/backends/trace-events +++ b/backends/trace-events @@ -17,3 +17,4 @@ iommufd_backend_alloc_hwpt(int iommufd, uint32_t dev_id, uint32_t pt_id, uint32_ iommufd_backend_alloc_ioas(int iommufd, uint32_t ioas, int ret) " iommufd=%d ioas=%d (%d)" iommufd_backend_free_id(int iommufd, uint32_t id, int ret) " iommufd=%d id=%d (%d)" iommufd_backend_set_dirty(int iommufd, uint32_t hwpt_id, bool start, int ret) " iommufd=%d hwpt=%d enable=%d (%d)" +iommufd_backend_get_dirty_bitmap(int iommufd, uint32_t hwpt_id, uint64_t iova, uint64_t size, uint64_t page_size, int ret) " iommufd=%d hwpt=%d iova=0x%"PRIx64" size=0x%"PRIx64" page_size=0x%"PRIx64" (%d)" diff --git a/hw/vfio/iommufd.c b/hw/vfio/iommufd.c index 361e659288fd..79b13bd262cc 100644 --- a/hw/vfio/iommufd.c +++ b/hw/vfio/iommufd.c @@ -25,6 +25,7 @@ #include "qemu/cutils.h" #include "qemu/chardev_open.h" #include "pci.h" +#include "exec/ram_addr.h" #include "migration/migration.h" static int iommufd_cdev_map(const VFIOContainerBase *bcontainer, hwaddr iova, @@ -142,6 +143,34 @@ err: return ret; } +static int iommufd_query_dirty_bitmap(const VFIOContainerBase *bcontainer, + VFIOBitmap *vbmap, uint64_t iova, + uint64_t size) +{ + VFIOIOMMUFDContainer *container = container_of(bcontainer, + VFIOIOMMUFDContainer, + bcontainer); + int ret; + VFIOIOASHwpt *hwpt; + unsigned long page_size; + + if (!bcontainer->dirty_pages_supported) { + return 0; + } + + page_size = qemu_real_host_page_size(); + QLIST_FOREACH(hwpt, &container->hwpt_list, next) { + ret = iommufd_backend_get_dirty_bitmap(container->be, hwpt->hwpt_id, + iova, size, page_size, + vbmap->bitmap); + if (ret) { + break; + } + } + + return ret; +} + static int iommufd_cdev_getfd(const char *sysfs_path, Error **errp) { long int ret = -ENOTTY; @@ -765,6 +794,7 @@ static void vfio_iommu_iommufd_class_init(ObjectClass *klass, void *data) vioc->pci_hot_reset = iommufd_cdev_pci_hot_reset; vioc->host_iommu_device_init = vfio_cdev_host_iommu_device_init; vioc->set_dirty_page_tracking = iommufd_set_dirty_page_tracking; + vioc->query_dirty_bitmap = iommufd_query_dirty_bitmap; }; static const TypeInfo types[] = { diff --git a/include/sysemu/iommufd.h b/include/sysemu/iommufd.h index 562c189dd92c..ba19b7ea4c19 100644 --- a/include/sysemu/iommufd.h +++ b/include/sysemu/iommufd.h @@ -55,5 +55,8 @@ int iommufd_backend_alloc_hwpt(int iommufd, uint32_t dev_id, void *data_ptr, uint32_t *out_hwpt); int iommufd_backend_set_dirty_tracking(IOMMUFDBackend *be, uint32_t hwpt_id, bool start); +int iommufd_backend_get_dirty_bitmap(IOMMUFDBackend *be, uint32_t hwpt_id, + uint64_t iova, ram_addr_t size, + uint64_t page_size, uint64_t *data); #endif From patchwork Mon Feb 12 13:56:41 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joao Martins X-Patchwork-Id: 13553302 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 7DB85C4829B for ; Mon, 12 Feb 2024 13:59:12 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rZWp8-0007ZX-Vv; Mon, 12 Feb 2024 08:58:19 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rZWoI-0006fx-5p for qemu-devel@nongnu.org; Mon, 12 Feb 2024 08:57:27 -0500 Received: from mx0b-00069f02.pphosted.com ([205.220.177.32]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rZWo7-0007FZ-Dm for qemu-devel@nongnu.org; Mon, 12 Feb 2024 08:57:18 -0500 Received: from pps.filterd (m0246630.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 41CD4PK1019057; Mon, 12 Feb 2024 13:57:12 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=corp-2023-11-20; bh=8K7Ud06VPXm69TMQsoAjMMYa+HU1h5r5mLzUbjebFGE=; b=LqFxF4YKV4JxprjYcdG8zEztCnaQwMxr7HWHouK33xdcyAro3J5zoPEGX9gyH04M6d2D WMJQ5FBSNZ82/DkMdZoBZQl8sxVB4RjFb/AroIloBnJJPvpUAOrwlslznCGJg/pOyINc HPlZ+PLHxfRVkURXQJU7ieGuFEd5IqDZeKBXwSIF9l2qi1oqSPrZ5j8KTxuPu0ucH9/M wKbLw2SLDqnabkqTx8UrTs8krk1UABtzI2ZjtNDbradn5UnP+gcPiA+f3qTM59Fn4mOG mEjkk1jXLVjcVKmidRlrXucyEVi9LGSG6I1eiJhomvV0hgASQc69wICzZuc8uBh/HpPO XA== Received: from iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta01.appoci.oracle.com [130.35.100.223]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3w7kw1g42w-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 12 Feb 2024 13:57:12 +0000 Received: from pps.filterd (iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 41CDB4xI024619; Mon, 12 Feb 2024 13:57:12 GMT Received: from pps.reinject (localhost [127.0.0.1]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 3w5ykc213q-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 12 Feb 2024 13:57:12 +0000 Received: from iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 41CDuqiF026744; Mon, 12 Feb 2024 13:57:11 GMT Received: from joaomart-mac.nl.oracle.com (dhcp-10-175-27-30.vpn.oracle.com [10.175.27.30]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTP id 3w5ykc20rs-7; Mon, 12 Feb 2024 13:57:11 +0000 From: Joao Martins To: qemu-devel@nongnu.org Cc: Yi Liu , Eric Auger , Zhenzhong Duan , Alex Williamson , Cedric Le Goater , Paolo Bonzini , "Daniel P . Berrange" , Eduardo Habkost , Eric Blake , Markus Armbruster , Jason Gunthorpe , Avihai Horon , Joao Martins Subject: [PATCH RFCv2 6/8] backends/iommufd: Add ability to disable hugepages Date: Mon, 12 Feb 2024 13:56:41 +0000 Message-Id: <20240212135643.5858-7-joao.m.martins@oracle.com> In-Reply-To: <20240212135643.5858-1-joao.m.martins@oracle.com> References: <20240212135643.5858-1-joao.m.martins@oracle.com> MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.272,Aquarius:18.0.1011,Hydra:6.0.619,FMLib:17.11.176.26 definitions=2024-02-12_10,2024-02-12_03,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 adultscore=0 bulkscore=0 mlxlogscore=999 malwarescore=0 mlxscore=0 spamscore=0 suspectscore=0 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2311290000 definitions=main-2402120105 X-Proofpoint-GUID: Kv5VI4NtVXWM-NB-7FgKHEMac4pkcU_X X-Proofpoint-ORIG-GUID: Kv5VI4NtVXWM-NB-7FgKHEMac4pkcU_X Received-SPF: pass client-ip=205.220.177.32; envelope-from=joao.m.martins@oracle.com; helo=mx0b-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Allow disabling hugepages to be dirty track at base page granularity in similar vein to vfio_type1_iommu.disable_hugepages but per IOAS. Signed-off-by: Joao Martins --- backends/iommufd.c | 36 ++++++++++++++++++++++++++++++++++++ backends/trace-events | 1 + hw/vfio/iommufd.c | 4 ++++ include/sysemu/iommufd.h | 4 ++++ qapi/qom.json | 2 +- 5 files changed, 46 insertions(+), 1 deletion(-) diff --git a/backends/iommufd.c b/backends/iommufd.c index dd676d493c37..72fd98a9a50c 100644 --- a/backends/iommufd.c +++ b/backends/iommufd.c @@ -29,6 +29,7 @@ static void iommufd_backend_init(Object *obj) be->fd = -1; be->users = 0; be->owned = true; + be->hugepages = 1; } static void iommufd_backend_finalize(Object *obj) @@ -63,6 +64,14 @@ static bool iommufd_backend_can_be_deleted(UserCreatable *uc) return !be->users; } +static void iommufd_backend_set_hugepages(Object *obj, bool enabled, + Error **errp) +{ + IOMMUFDBackend *be = IOMMUFD_BACKEND(obj); + + be->hugepages = enabled; +} + static void iommufd_backend_class_init(ObjectClass *oc, void *data) { UserCreatableClass *ucc = USER_CREATABLE_CLASS(oc); @@ -70,6 +79,11 @@ static void iommufd_backend_class_init(ObjectClass *oc, void *data) ucc->can_be_deleted = iommufd_backend_can_be_deleted; object_class_property_add_str(oc, "fd", NULL, iommufd_backend_set_fd); + + object_class_property_add_bool(oc, "hugepages", NULL, + iommufd_backend_set_hugepages); + object_class_property_set_description(oc, "hugepages", + "Set to 'off' to disable hugepages"); } int iommufd_backend_connect(IOMMUFDBackend *be, Error **errp) @@ -106,6 +120,28 @@ out: trace_iommufd_backend_disconnect(be->fd, be->users); } +int iommufd_backend_set_option(int fd, uint32_t object_id, + uint32_t option_id, uint64_t val64) +{ + int ret; + struct iommu_option option = { + .size = sizeof(option), + .option_id = option_id, + .op = IOMMU_OPTION_OP_SET, + .val64 = val64, + .object_id = object_id, + }; + + ret = ioctl(fd, IOMMU_OPTION, &option); + if (ret) { + error_report("Failed to set option %x to value %"PRIx64" %m", option_id, + val64); + } + trace_iommufd_backend_set_option(fd, object_id, option_id, val64, ret); + + return ret; +} + int iommufd_backend_alloc_ioas(IOMMUFDBackend *be, uint32_t *ioas_id, Error **errp) { diff --git a/backends/trace-events b/backends/trace-events index 11a27cb114b6..076166552881 100644 --- a/backends/trace-events +++ b/backends/trace-events @@ -15,6 +15,7 @@ iommufd_backend_unmap_dma_non_exist(int iommufd, uint32_t ioas, uint64_t iova, u iommufd_backend_unmap_dma(int iommufd, uint32_t ioas, uint64_t iova, uint64_t size, int ret) " iommufd=%d ioas=%d iova=0x%"PRIx64" size=0x%"PRIx64" (%d)" iommufd_backend_alloc_hwpt(int iommufd, uint32_t dev_id, uint32_t pt_id, uint32_t flags, uint32_t hwpt_type, uint32_t len, uint64_t data_ptr, uint32_t out_hwpt_id, int ret) " iommufd=%d dev_id=%u pt_id=%u flags=0x%x hwpt_type=%u len=%u data_ptr=0x%"PRIx64" out_hwpt=%u (%d)" iommufd_backend_alloc_ioas(int iommufd, uint32_t ioas, int ret) " iommufd=%d ioas=%d (%d)" +iommufd_backend_set_option(int iommufd, uint32_t object_id, uint32_t option_id, uint64_t val, int ret) " iommufd=%d object_id=%u option_id=%u val64=0x%"PRIx64" (%d)" iommufd_backend_free_id(int iommufd, uint32_t id, int ret) " iommufd=%d id=%d (%d)" iommufd_backend_set_dirty(int iommufd, uint32_t hwpt_id, bool start, int ret) " iommufd=%d hwpt=%d enable=%d (%d)" iommufd_backend_get_dirty_bitmap(int iommufd, uint32_t hwpt_id, uint64_t iova, uint64_t size, uint64_t page_size, int ret) " iommufd=%d hwpt=%d iova=0x%"PRIx64" size=0x%"PRIx64" page_size=0x%"PRIx64" (%d)" diff --git a/hw/vfio/iommufd.c b/hw/vfio/iommufd.c index 79b13bd262cc..697d40841d7f 100644 --- a/hw/vfio/iommufd.c +++ b/hw/vfio/iommufd.c @@ -521,6 +521,10 @@ static int iommufd_cdev_attach(const char *name, VFIODevice *vbasedev, goto err_alloc_ioas; } + if (!vbasedev->iommufd_dev.iommufd->hugepages) { + iommufd_backend_set_option(vbasedev->iommufd_dev.iommufd->fd, ioas_id, + IOMMU_OPTION_HUGE_PAGES, 0); + } trace_iommufd_cdev_alloc_ioas(vbasedev->iommufd_dev.iommufd->fd, ioas_id); container = g_malloc0(sizeof(*container)); diff --git a/include/sysemu/iommufd.h b/include/sysemu/iommufd.h index ba19b7ea4c19..bc6607e3d444 100644 --- a/include/sysemu/iommufd.h +++ b/include/sysemu/iommufd.h @@ -19,6 +19,7 @@ struct IOMMUFDBackend { /*< protected >*/ int fd; /* /dev/iommu file descriptor */ bool owned; /* is the /dev/iommu opened internally */ + bool hugepages; /* are hugepages enabled on the IOAS */ uint32_t users; /*< public >*/ @@ -30,6 +31,9 @@ void iommufd_backend_disconnect(IOMMUFDBackend *be); int iommufd_backend_alloc_ioas(IOMMUFDBackend *be, uint32_t *ioas_id, Error **errp); void iommufd_backend_free_id(IOMMUFDBackend *be, uint32_t id); +int iommufd_backend_set_option(int fd, uint32_t object_id, + uint32_t option_id, + uint64_t val64); int iommufd_backend_map_dma(IOMMUFDBackend *be, uint32_t ioas_id, hwaddr iova, ram_addr_t size, void *vaddr, bool readonly); int iommufd_backend_unmap_dma(IOMMUFDBackend *be, uint32_t ioas_id, diff --git a/qapi/qom.json b/qapi/qom.json index 84af23fe245d..9ad27e2b939b 100644 --- a/qapi/qom.json +++ b/qapi/qom.json @@ -809,7 +809,7 @@ # Since: 9.0 ## { 'struct': 'IOMMUFDProperties', - 'data': { '*fd': 'str' } } + 'data': { '*fd': 'str', '*hugepages': 'bool' } } ## # @RngProperties: From patchwork Mon Feb 12 13:56:42 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joao Martins X-Patchwork-Id: 13553301 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id C66E0C48297 for ; Mon, 12 Feb 2024 13:59:04 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rZWpb-000862-EH; Mon, 12 Feb 2024 08:58:51 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rZWoK-0006hT-2r for qemu-devel@nongnu.org; Mon, 12 Feb 2024 08:57:28 -0500 Received: from mx0b-00069f02.pphosted.com ([205.220.177.32]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rZWoH-0007G7-Aa for qemu-devel@nongnu.org; Mon, 12 Feb 2024 08:57:26 -0500 Received: from pps.filterd (m0246631.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 41CDJReS031799; Mon, 12 Feb 2024 13:57:16 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=corp-2023-11-20; bh=0NZhiOEWz7ONTB49CXHlw2TDED4An2pbk9tsG7osT8I=; b=BmwYjwT7Lnb6mS9ePN1AIDCZQTerFQoPp/Sf/9LhL2QArbqh5NvV1tvkddJ3atIqIyaS gXUQQDbECyQn3cYMWmyAB3WaxhfPRbqwFrd8pv2xbSimZoMrm1wu5gg2meBL7ZKnSNk/ uioOpryL8ITX9kDO7uWSwhilRgLxa7oePMqCXO9QqeHEyaX6xHWhmnlfH/P032XIC5Vt VGgMVBezmp2NczX946H3PLqu5G4zaf+cX1RrttNv3Zyj2G43AGmKPyshgFUbywBrlmLe sC/DSf6k2HA41gfpcHhQurj7eYvBGP6FTNd0Z1F5llBo7nTaU5emViZfj6zKCSZds8hE vg== Received: from iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta01.appoci.oracle.com [130.35.100.223]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3w7m40g32t-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 12 Feb 2024 13:57:15 +0000 Received: from pps.filterd (iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 41CDltKF024567; Mon, 12 Feb 2024 13:57:15 GMT Received: from pps.reinject (localhost [127.0.0.1]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 3w5ykc215b-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 12 Feb 2024 13:57:15 +0000 Received: from iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 41CDuqiH026744; Mon, 12 Feb 2024 13:57:14 GMT Received: from joaomart-mac.nl.oracle.com (dhcp-10-175-27-30.vpn.oracle.com [10.175.27.30]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTP id 3w5ykc20rs-8; Mon, 12 Feb 2024 13:57:14 +0000 From: Joao Martins To: qemu-devel@nongnu.org Cc: Yi Liu , Eric Auger , Zhenzhong Duan , Alex Williamson , Cedric Le Goater , Paolo Bonzini , "Daniel P . Berrange" , Eduardo Habkost , Eric Blake , Markus Armbruster , Jason Gunthorpe , Avihai Horon , Joao Martins Subject: [PATCH RFCv2 7/8] vfio/migration: Don't block migration device dirty tracking is unsupported Date: Mon, 12 Feb 2024 13:56:42 +0000 Message-Id: <20240212135643.5858-8-joao.m.martins@oracle.com> In-Reply-To: <20240212135643.5858-1-joao.m.martins@oracle.com> References: <20240212135643.5858-1-joao.m.martins@oracle.com> MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.272,Aquarius:18.0.1011,Hydra:6.0.619,FMLib:17.11.176.26 definitions=2024-02-12_10,2024-02-12_03,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 adultscore=0 bulkscore=0 mlxlogscore=999 malwarescore=0 mlxscore=0 spamscore=0 suspectscore=0 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2311290000 definitions=main-2402120105 X-Proofpoint-GUID: 1CIXOrjp6UaOMH_Z8Dg81fxCOj5oXpP1 X-Proofpoint-ORIG-GUID: 1CIXOrjp6UaOMH_Z8Dg81fxCOj5oXpP1 Received-SPF: pass client-ip=205.220.177.32; envelope-from=joao.m.martins@oracle.com; helo=mx0b-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org By default VFIO migration is set to auto, which will support live migration if the migration capability is set *and* also dirty page tracking is supported. For testing purposes one can force enable without dirty page tracking via enable-migration=on, but that option is generally left for testing purposes. So starting with IOMMU dirty tracking it can use to acomodate the lack of VF dirty page tracking allowing us to minimize the VF requirements for migration and thus enabling migration by default for those. Signed-off-by: Joao Martins --- hw/vfio/iommufd.c | 3 +-- hw/vfio/migration.c | 4 +++- include/sysemu/iommufd.h | 1 + 3 files changed, 5 insertions(+), 3 deletions(-) diff --git a/hw/vfio/iommufd.c b/hw/vfio/iommufd.c index 697d40841d7f..78d8f4391b68 100644 --- a/hw/vfio/iommufd.c +++ b/hw/vfio/iommufd.c @@ -275,8 +275,7 @@ static int iommufd_cdev_detach_ioas_hwpt(VFIODevice *vbasedev, Error **errp) return ret; } -static bool iommufd_dirty_pages_supported(IOMMUFDDevice *iommufd_dev, - Error **errp) +bool iommufd_dirty_pages_supported(IOMMUFDDevice *iommufd_dev, Error **errp) { uint64_t caps; int r; diff --git a/hw/vfio/migration.c b/hw/vfio/migration.c index 70e6b1a709f9..674e76b3f3df 100644 --- a/hw/vfio/migration.c +++ b/hw/vfio/migration.c @@ -938,7 +938,9 @@ bool vfio_migration_realize(VFIODevice *vbasedev, Error **errp) return !vfio_block_migration(vbasedev, err, errp); } - if (!vbasedev->dirty_pages_supported) { + if (!vbasedev->dirty_pages_supported && + (vbasedev->iommufd_dev.iommufd && + !iommufd_dirty_pages_supported(&vbasedev->iommufd_dev, &err))) { if (vbasedev->enable_migration == ON_OFF_AUTO_AUTO) { error_setg(&err, "%s: VFIO device doesn't support device dirty tracking", diff --git a/include/sysemu/iommufd.h b/include/sysemu/iommufd.h index bc6607e3d444..d6be49f2ac78 100644 --- a/include/sysemu/iommufd.h +++ b/include/sysemu/iommufd.h @@ -53,6 +53,7 @@ typedef struct IOMMUFDDevice { void iommufd_device_init(IOMMUFDDevice *idev); int iommufd_device_get_hw_capabilities(IOMMUFDDevice *idev, uint64_t *caps, Error **errp); +bool iommufd_dirty_pages_supported(IOMMUFDDevice *idev, Error **errp); int iommufd_backend_alloc_hwpt(int iommufd, uint32_t dev_id, uint32_t pt_id, uint32_t flags, uint32_t data_type, uint32_t data_len, From patchwork Mon Feb 12 13:56:43 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joao Martins X-Patchwork-Id: 13553300 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id D1EE9C4829B for ; Mon, 12 Feb 2024 13:58:51 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rZWor-0007BH-HV; Mon, 12 Feb 2024 08:58:01 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rZWoM-0006mC-DQ for qemu-devel@nongnu.org; Mon, 12 Feb 2024 08:57:32 -0500 Received: from mx0b-00069f02.pphosted.com ([205.220.177.32]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rZWoI-0007Gi-Qh for qemu-devel@nongnu.org; Mon, 12 Feb 2024 08:57:29 -0500 Received: from pps.filterd (m0246631.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 41CDJNIr031779; Mon, 12 Feb 2024 13:57:19 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=corp-2023-11-20; bh=bbKNI2tWJGb34lA30BIrwS3R/lj2i0DvpPnuPLxdUgY=; b=WvY2llUu2pyvC4BOrkkplAmUcXzWtnrdOx4Ys/+jEEyjRbIEZkYJujfwxHZHXREwadzm Ub8yoem+IqkWKrXeiEkI52iXiwR3FiTgall1Uqp7PRwg+jnOc37ASeGTI0ElK7+XblOt HCkwe6o1IXKmBu46MGUCE+xUqukU9NJ9MNNMMtssaPvhU3laEY/lw4sbP+zSjM54WT/I C9mnXvhBBdlWwEzOTl4bZ96t2w/8/foVJ5zgeLW6OJqjNbhlYqijU/syayho7Ni33se4 iWNfqcpUBAi1Iu67ofj6K8xT1i/SWW97qDa+/gszBSJ/Hu2Mm2UZ1Vv0BaIFKqST9L82 7w== Received: from iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta01.appoci.oracle.com [130.35.100.223]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3w7m40g331-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 12 Feb 2024 13:57:18 +0000 Received: from pps.filterd (iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 41CCGuLL024016; Mon, 12 Feb 2024 13:57:18 GMT Received: from pps.reinject (localhost [127.0.0.1]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 3w5ykc217k-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 12 Feb 2024 13:57:18 +0000 Received: from iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 41CDuqiJ026744; Mon, 12 Feb 2024 13:57:17 GMT Received: from joaomart-mac.nl.oracle.com (dhcp-10-175-27-30.vpn.oracle.com [10.175.27.30]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTP id 3w5ykc20rs-9; Mon, 12 Feb 2024 13:57:17 +0000 From: Joao Martins To: qemu-devel@nongnu.org Cc: Yi Liu , Eric Auger , Zhenzhong Duan , Alex Williamson , Cedric Le Goater , Paolo Bonzini , "Daniel P . Berrange" , Eduardo Habkost , Eric Blake , Markus Armbruster , Jason Gunthorpe , Avihai Horon , Joao Martins Subject: [PATCH RFCv2 8/8] vfio/common: Allow disabling device dirty page tracking Date: Mon, 12 Feb 2024 13:56:43 +0000 Message-Id: <20240212135643.5858-9-joao.m.martins@oracle.com> In-Reply-To: <20240212135643.5858-1-joao.m.martins@oracle.com> References: <20240212135643.5858-1-joao.m.martins@oracle.com> MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.272,Aquarius:18.0.1011,Hydra:6.0.619,FMLib:17.11.176.26 definitions=2024-02-12_10,2024-02-12_03,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 adultscore=0 bulkscore=0 mlxlogscore=999 malwarescore=0 mlxscore=0 spamscore=0 suspectscore=0 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2311290000 definitions=main-2402120105 X-Proofpoint-GUID: nihlGWdeBZ5OniBfhGB8I0HvBUYAzDGA X-Proofpoint-ORIG-GUID: nihlGWdeBZ5OniBfhGB8I0HvBUYAzDGA Received-SPF: pass client-ip=205.220.177.32; envelope-from=joao.m.martins@oracle.com; helo=mx0b-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org The property 'x-pre-copy-dirty-page-tracking' allows disabling the whole tracking of VF pre-copy phase of dirty page tracking, though it means that it will only be used at the start of the switchover phase. Add an option that disables the VF dirty page tracking, and fall back into container-based dirty page tracking. This also allows to use IOMMU dirty tracking even on VFs with their own dirty tracker scheme. Signed-off-by: Joao Martins --- hw/vfio/common.c | 7 +++++++ hw/vfio/migration.c | 3 ++- hw/vfio/pci.c | 3 +++ include/hw/vfio/vfio-common.h | 1 + 4 files changed, 13 insertions(+), 1 deletion(-) diff --git a/hw/vfio/common.c b/hw/vfio/common.c index a940c0b6ede8..9fe113ea016d 100644 --- a/hw/vfio/common.c +++ b/hw/vfio/common.c @@ -213,6 +213,9 @@ bool vfio_devices_all_device_dirty_tracking(const VFIOContainerBase *bcontainer) VFIODevice *vbasedev; QLIST_FOREACH(vbasedev, &bcontainer->device_list, container_next) { + if (vbasedev->device_dirty_page_tracking == ON_OFF_AUTO_OFF) { + return false; + } if (!vbasedev->dirty_pages_supported) { return false; } @@ -236,6 +239,10 @@ bool vfio_device_dirty_pages_supported(VFIODevice *vbasedev) return false; } + if (vbasedev->device_dirty_page_tracking == ON_OFF_AUTO_OFF) { + return false; + } + return !vbasedev->dirty_pages_supported; } diff --git a/hw/vfio/migration.c b/hw/vfio/migration.c index 674e76b3f3df..09e742fbeb9f 100644 --- a/hw/vfio/migration.c +++ b/hw/vfio/migration.c @@ -938,7 +938,8 @@ bool vfio_migration_realize(VFIODevice *vbasedev, Error **errp) return !vfio_block_migration(vbasedev, err, errp); } - if (!vbasedev->dirty_pages_supported && + if ((!vbasedev->dirty_pages_supported || + vbasedev->device_dirty_page_tracking == ON_OFF_AUTO_OFF) && (vbasedev->iommufd_dev.iommufd && !iommufd_dirty_pages_supported(&vbasedev->iommufd_dev, &err))) { if (vbasedev->enable_migration == ON_OFF_AUTO_AUTO) { diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c index dedb64fc080e..d3b516b2e4d3 100644 --- a/hw/vfio/pci.c +++ b/hw/vfio/pci.c @@ -3349,6 +3349,9 @@ static Property vfio_pci_dev_properties[] = { DEFINE_PROP_ON_OFF_AUTO("x-pre-copy-dirty-page-tracking", VFIOPCIDevice, vbasedev.pre_copy_dirty_page_tracking, ON_OFF_AUTO_ON), + DEFINE_PROP_ON_OFF_AUTO("x-device-dirty-page-tracking", VFIOPCIDevice, + vbasedev.device_dirty_page_tracking, + ON_OFF_AUTO_ON), DEFINE_PROP_ON_OFF_AUTO("display", VFIOPCIDevice, display, ON_OFF_AUTO_OFF), DEFINE_PROP_UINT32("xres", VFIOPCIDevice, display_xres, 0), diff --git a/include/hw/vfio/vfio-common.h b/include/hw/vfio/vfio-common.h index a3e691c126c6..e67f5e74cebd 100644 --- a/include/hw/vfio/vfio-common.h +++ b/include/hw/vfio/vfio-common.h @@ -136,6 +136,7 @@ typedef struct VFIODevice { VFIOMigration *migration; Error *migration_blocker; OnOffAuto pre_copy_dirty_page_tracking; + OnOffAuto device_dirty_page_tracking; bool dirty_pages_supported; bool dirty_tracking; union {