From patchwork Thu Apr 28 21:13:41 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joao Martins X-Patchwork-Id: 12831185 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E9D0DC433EF for ; Thu, 28 Apr 2022 21:14:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1352154AbiD1VRw (ORCPT ); Thu, 28 Apr 2022 17:17:52 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60702 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231442AbiD1VRu (ORCPT ); Thu, 28 Apr 2022 17:17:50 -0400 Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 087F47F227 for ; Thu, 28 Apr 2022 14:14:34 -0700 (PDT) Received: from pps.filterd (m0246632.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 23SJeCY5011361; Thu, 28 Apr 2022 21:14:09 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : content-type : mime-version; s=corp-2021-07-09; bh=JtsxKbszu8EQ5rH8Mxv9YaH+ljRMgyZYL11OCS6TaiI=; b=Ww7cfVgA8nNC39UntsnVuCdcHrzQnOF213H6r9iAmP7Df8WnmAW0uSEA21DY9T+z4QtJ 8qTw58qVWvwGvVx8u1jb3cnqZIzwWCOIMVNYItq38dMyYxnx5nQL3Iu/lgcgrNGGlE/K muGmjz7PJG9+3cywY7Zj0EmplyMkSEkS8On0qikd+pATOUUGVAaqvAt5mhWixiZbUqk/ V6qQY76fXxTdke8Odq95ghQGOJP3y0QKv8QRXKj9XTsPcFZxB3rbqju45TgVz//J6UCl ffTeFVloS5C++dx38pfp156Df1q/ocTelOYM9WxCmz2a5eRnhIok1x+Lb8TpIWlH7XXi Bw== Received: from iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta01.appoci.oracle.com [130.35.100.223]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3fmaw4nb95-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 28 Apr 2022 21:14:09 +0000 Received: from pps.filterd (iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (8.16.1.2/8.16.1.2) with SMTP id 23SL6cJ8028702; Thu, 28 Apr 2022 21:14:09 GMT Received: from nam12-mw2-obe.outbound.protection.outlook.com (mail-mw2nam12lp2044.outbound.protection.outlook.com [104.47.66.44]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com with ESMTP id 3fp5ypebh6-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 28 Apr 2022 21:14:08 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=MWQKsIZdkywgvsAcBhDXNmsPZODcyu52DQEO/bH27eiVQbjHmKRh0m+QFyTX09kLJyIBhT02RTTTUk6NKZ5JE0+R8q2K1aoiwh3IWdikmT6Uo+w3Of+6JGHuW8ycMGlPF1VLOKKUeDw7FyC+Y+RIEmtVz+nR61Jo0rgWSe4AHIdnGLpodhed76g5Ok6Kg0+dhcxgepn9ah8O5lcJCdW8rQZVOEGx88FfXPGXkOL/hR8cUvWEM+oYp44FN5qFtWVB2wej3u8Mv9RryPrIfCn5/w+JtxzX1NWkjj1RFGexjhCWmfjS2EEs4xt0HIXC4b+a9g1iFmmaqK49/BiE19FXuQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=JtsxKbszu8EQ5rH8Mxv9YaH+ljRMgyZYL11OCS6TaiI=; b=miJcY/kDNKfmDMcirfse/U6ymG6ictkqUezMrOOmNGsjLs3zGY/zupd9/QK72tV3tbhJOOp4xIa5UecfSvi4Afbhbd4mR3r4i+j3IfqYeFlp4gTh4piEsiVArQRzAI9RKNZhn3LfgLw8t69L92hXEsHmNw1l52N5bkKyTA8Qsnmu7SJpxS+DAZC8Bq4ZYBEZNOKX+y/7YRr6TeRWExL/kOWbQJXICa0JmoDZzY4BpxZDa+fB1lTJhDaGH/jgxYwr6aDXAbbI92u+JXA0z7qvVpr3hsiQ7ZWmyNaZNH8x9Eg4vuTOWvPllBsygEA51QIEswfboeFvzsesNJrP8qD5Vg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=JtsxKbszu8EQ5rH8Mxv9YaH+ljRMgyZYL11OCS6TaiI=; b=GwoyFnhLjD1CD2Twb3C8e9ylLT8PlgRKCriqWXcUWL6+O1uCD2SRYEQqdMe1LgiZGQBoNuIpzYzWR8cSvTj6FhO62Tui2igRvvxmj08LDCtiVx1Rr9cNkNXB7EkvIJ4uv+7cXhs2VYJFABoowlQR5q2Ts0nSFOQds4Qgev+oeOE= Received: from BLAPR10MB4835.namprd10.prod.outlook.com (2603:10b6:208:331::11) by CH0PR10MB5289.namprd10.prod.outlook.com (2603:10b6:610:d8::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5186.14; Thu, 28 Apr 2022 21:14:06 +0000 Received: from BLAPR10MB4835.namprd10.prod.outlook.com ([fe80::b9e5:d1b6:b4be:f9d]) by BLAPR10MB4835.namprd10.prod.outlook.com ([fe80::b9e5:d1b6:b4be:f9d%5]) with mapi id 15.20.5206.013; Thu, 28 Apr 2022 21:14:06 +0000 From: Joao Martins To: qemu-devel@nongnu.org Cc: Joao Martins , Paolo Bonzini , "Dr. David Alan Gilbert" , Richard Henderson , Eduardo Habkost , "Michael S. Tsirkin" , Marcel Apfelbaum , Peter Xu , Jason Wang , Alex Williamson , David Hildenbrand , =?utf-8?q?Philippe_Mathieu-Daud=C3=A9?= , Cornelia Huck , Juan Quintela , Eric Blake , Markus Armbruster , Jason Gunthorpe , Nicolin Chen , Yishai Hadas , Kevin Tian , Yi Liu , Eric Auger , Thanos Makatos , "John G . Johnson" , kvm@vger.kernel.org Subject: [PATCH RFC 00/10] hw/vfio, x86/iommu: IOMMUFD Dirty Tracking Date: Thu, 28 Apr 2022 22:13:41 +0100 Message-Id: <20220428211351.3897-1-joao.m.martins@oracle.com> X-Mailer: git-send-email 2.11.0 X-ClientProxiedBy: LO2P265CA0297.GBRP265.PROD.OUTLOOK.COM (2603:10a6:600:a5::21) To BLAPR10MB4835.namprd10.prod.outlook.com (2603:10b6:208:331::11) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 6668e467-e75f-40f0-0b2f-08da295c0706 X-MS-TrafficTypeDiagnostic: CH0PR10MB5289:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: LexLx5r3lnr3qTD5E0C6fdmVTYId+NhBYrs5YAf6z4n2xUUUNcxNkLO2IJDVPx3qYBITeO9R8nxFDZ0h8YvnFCo7PqcyiMuc5m6UiIHLQCsy9dfU9mlkCZnkpoQL+NK5OPAspw8/6Qa/Ll57v9cXgA7RaAEAO9JDPFSaCjLA9tryMAFalD5n8PseISfhMmvutIX+d4729VemYhi0QD8tGElyfhx96N8dfdH6jydUfsJYwYea4wJpXAtJkjx4tb9fvA6BIR5S/RzNc8KCPnK126P5yUfVX3RlOIi3dGMzHiTEQr83TcPlQytKWyReYwy8+OmTcRQ4EZSmDQQUQ2dYAfoS8+VL4nLhQIpo75I9gFCMJ0Z6TpEokQMfiSDwW+KZKaGTVbDcN8BeEt6fBb2+SgL/IdU0LUZGoyi/XjZcfLqHWiO6B336E15PBe+gpUWZ8LXcumBjseuhutLA4s0L80WrH47ZjYzDYH/B8AcIUGgeSy9ofQRwjRxcq/yOhIeIT6nn42AvzbUVam8nB4eQjy3Tn0GymGmEFzOclSDC1mAxYnRL3a2YcRUhOwbhHJ/ilgSOMSA5h4Vj+mLb3VrvrnsJbyqDwaF26eZR/9lY3YCw1QZWhRouUGC/SsY9niR4GmaS/WJY2zTzlADAgAY7KbD5A4r+orBFK5hC6Lmt5HC48Xh6nDRp9eHWIZod3Ppwr+KpwJ7w1NBfA7aNiJptXAYqT2UwI7YQkChWchjIwsY/OqZqFdJPYmw6pSxMa9uwuPO0jhrljpgeGdaApqLx4bScYZS3fkEKYYjAG/3FA/TXwDFmalGguMx2y8Ed8btC X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BLAPR10MB4835.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230001)(366004)(6666004)(54906003)(186003)(1076003)(38350700002)(38100700002)(5660300002)(2616005)(7416002)(103116003)(26005)(316002)(6916009)(36756003)(66946007)(8676002)(66476007)(4326008)(66556008)(8936002)(52116002)(86362001)(6512007)(2906002)(6486002)(508600001)(83380400001)(6506007)(966005)(14143004);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: p3kmxWeNJc9/+eizkJg2biTEO8IF7UEkMN9UlD5MvOEK7r9mxN7HyYS3OmT+be1AHTe+bF7CWSBlTBqT2y0IIxiE+Z3kDbphmy+COyImNsVA7z3XwYd+FnZKjSlzwLYtWNi6AYoRCWKaDPZ/+ilr6cEvLbUiWM3PQw8tgFsMaieXIxdRl4weF1zCB047w5stR5jf4tzHy5p1n1X0tKr9ei/fuQdlhIn7TsN+PFAROyRgIv5ndhjIY5k8xcDiWHU6xC7dlvmDkk6mBm+lGeYpzd8WJDwEwaim1DdwTzqEh97smCSxxb3UuNrQiXaHo8P6mmXEzURvFlS5WT2ONFUIyTWS6duk5MI8laP4NTb3QYLMqsAmryNXd7Lnxvief6s72qIE4HyHlKeLcxO0TywQUCCLo1NPqHzhalG0WYLpKB+Tqz9xhD75OHPz7dp9ZxDxKi/PghrZcHZ8HBvwwzClcuje4ELDOumvbYAyxNT5yWrSRt/ZNxMpqSA8gIeuXTt4G2ZVAGB1dyDuXYAbRkQd92cRXIPlT0WCT4IbiYUWFCgVVy/4woGaB2LnA0XSD6nhXHP812O1Ov2pxTGRjrd2eFu4bQClswfL2j5i2AAQzNjS35uJPTWnkRqgfCu2g26Sh3WAYLRUEPr/uqTPPRKS6wqTKKJ4uMiO9cVRI8K/aEElBTrOMEdQ2MRIwbm9l0GbDcCBCPvAQapVf2njHSJLd1X++qrpZHu5Rth0+4iNoSbhF72lDuYDNvQC3qdGYGYYaLvYhViW5+WGec6IhxbDyAxewelluoktYyh4wAuKslmqwnUI06SsSb6t5TFZIo7yIVcfT/jLZtw+B5T9qQm8NoGTyaW2t+vAfmVIM17NAQb6uUt69WnWf/9tLYPyps3IqjZTikQx3EynhTh6yKo1jsHI9hlBHZNsoVVvdieQXCOHo4wGWBq1ZGvRqxQ7HTRnF87ECMGQmZ5Ywh61HzaAv4SPHJEq0EGB8bk3sopPDodBEJ160vsXk/8d6UeZgl0qYy/0Y2kRmUjIp9Fn9N63Ul2e7gE8hFXmS2g2h3ENIUXvEtv6Akt7X9UIU8TWz8/AHz8nFSIW9Tq5LwIfZ/Coh4pcQCsf06fFKYW8dS/rjYFoXK5kuUbwgK/xBYzeCPBZc4DEs0ObH6v8+KeohLosFZZPZ5UOdHz9sY4RL057GVv1cgx5TwJt4WQ6iKS/KT5tzPbP/OcvOKb6K0ZrRU70bjhzWFrYPvV4KE55lHz/lMYTaEXLZGrajkZtyaNEFqBCYV/wWzs+MlhVkjJZPDIkZnTmQkSJ328VyAFfZg9MkYhMXr3A5dXLJgZySdvxcYDDpp7v0RYwVvndvAf64VFx0D0lMvF7vyxRX8vjxynKVaDFqwV406cni0jE9OhZs11IdQp/S22CLDEP3ZQOuHY3O+UF623iaV2Su2oJS9glWtR1H53oDDBkPDyD/XqqTV/akQH8hsun2H+Rsw/CytlSEa4sGCiA+6fVj5nKZA7J5Kh8qqIdrLW/qbavesDqdCHbuGa5Pp/s+CaAPAwQhZYuucqWyd0lOy6xME+JLrjlV3yJzf/90oDLQwFe6W3+Fp9vaPB47pMEmNi92TzpkzNh63OEEKrNvynzdpT6EENaz4WgLbV5C6T9OcMwKhM329zmW7GgK4YJLAo83pPMSTvzGDgPc9IPv3ukwqW7rvkaRy/jsrT9K4UqOyhdUIscs9o1pzm2/WXpQYqMpaQ36CkiWoDcGtZU3+km4PxCH4nIjvw= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 6668e467-e75f-40f0-0b2f-08da295c0706 X-MS-Exchange-CrossTenant-AuthSource: BLAPR10MB4835.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 28 Apr 2022 21:14:06.5099 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: Dg0b0IlAQBT6BDcR7SMxyEqXw7ndyZh70jlDHfRBwmtc5pWmUEpgwXviKTuWIfjkHQ5+XFwNR0MMTm9NuZ1m4uHTxNFCbyu2MSMuE8w9G3E= X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH0PR10MB5289 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.486,18.0.858 definitions=2022-04-28_04:2022-04-28,2022-04-28 signatures=0 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 mlxlogscore=999 mlxscore=0 suspectscore=0 malwarescore=0 spamscore=0 adultscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2202240000 definitions=main-2204280125 X-Proofpoint-GUID: SEHC40GIxpzFjRGelOc1my_p0z3Ykpav X-Proofpoint-ORIG-GUID: SEHC40GIxpzFjRGelOc1my_p0z3Ykpav Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org This series expands IOMMUFD series from Yi and Eric into supporting IOMMU Dirty Tracking. It adds both the emulated x86 IOMMUs, as well as IOMMUFD support that exercises said emulation (or H/W). It is organized into: * Patches 1 - 4: x86 IOMMU emulation that performs dirty tracking, useful for a nested guest exercising the IOMMUFD kernel counterpart for coverage and development. The caching of the PASID PTE root flags and AMD DTE is not exactly the cleanest, still wondering about a better way without having to lookup context/DTE prior to IOTLB lookup. * Patches 5 - 9: IOMMUFD backend support for dirty tracking; Sadly this wasn't exactly tested with VF Live Migration, but it adds (last 2 patches) a way to measure dirty rate via HMP and QMP. The IOMMUFD kernel patches go into extent on what each of the ioctls do, albeit the workflow is relatively simple: 1) Toggling dirty tracking via HWPT_SET_DIRTY or get -ENOTSUPP otherwise which voids any attempt of walking IO pagetables 2) Read-and-clear of Dirty IOVAs 3) Unmap vIOMMU-backed IOVAs and fetch its dirty state right-before-unmap. The series is still a WIP. The intend is to exercise the kernel side[0] aside from the selftests that provide some coverage. Comments, feedback, as always appreciated. Joao [0] https://lore.kernel.org/linux-iommu/20220428210933.3583-1-joao.m.martins@oracle.com/ Joao Martins (10): amd-iommu: Cache PTE/DTE info in IOTLB amd-iommu: Access/Dirty bit support intel-iommu: Cache PASID entry flags intel_iommu: Second Stage Access Dirty bit support linux-headers: import iommufd.h hwpt extensions vfio/iommufd: Add HWPT_SET_DIRTY support vfio/iommufd: Add HWPT_GET_DIRTY_IOVA support vfio/iommufd: Add IOAS_UNMAP_DIRTY support migration/dirtyrate: Expand dirty_bitmap to be tracked separately for devices hw/vfio: Add nr of dirty pages to tracepoints accel/kvm/kvm-all.c | 12 +++ hmp-commands.hx | 5 +- hw/i386/amd_iommu.c | 76 +++++++++++++- hw/i386/amd_iommu.h | 11 +- hw/i386/intel_iommu.c | 103 +++++++++++++++++-- hw/i386/intel_iommu_internal.h | 4 + hw/i386/trace-events | 4 + hw/iommufd/iommufd.c | 63 ++++++++++++ hw/iommufd/trace-events | 3 + hw/vfio/container.c | 20 +++- hw/vfio/iommufd.c | 179 ++++++++++++++++++++++++++++++++- hw/vfio/trace-events | 3 +- include/exec/memory.h | 10 +- include/hw/i386/intel_iommu.h | 1 + include/hw/iommufd/iommufd.h | 6 ++ linux-headers/linux/iommufd.h | 78 ++++++++++++++ migration/dirtyrate.c | 59 ++++++++--- migration/dirtyrate.h | 1 + qapi/migration.json | 15 +++ softmmu/memory.c | 5 + 20 files changed, 618 insertions(+), 40 deletions(-)