From patchwork Mon Feb 5 12:48:24 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yishai Hadas X-Patchwork-Id: 13545547 Received: from NAM10-BN7-obe.outbound.protection.outlook.com (mail-bn7nam10on2085.outbound.protection.outlook.com [40.107.92.85]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 22FA94D9E4 for ; Mon, 5 Feb 2024 12:49:05 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.92.85 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707137347; cv=fail; b=bdGASh9T6IwlhJymQ0ddcWGBvnnqGJCQlxDNKampTjLQT1uMbWPtMrof29KujAcLbcd9xkMzubZW+LL2cGg5ZQxrLVohFkMtTeddJDyIAzbzIx446+3FLQ+43fRsTtYsmgOJyZIsttMhWvT4gPwpg0OF7s353LN3uHP3zcZxSu8= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707137347; c=relaxed/simple; bh=bEStpGhp/jxgcocrPeRlB2Z2Aq4KVLJWW81SgF4OK+s=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=QDR24nzGJ5vFiVTrfjXae/68iCyc0qf03bQz5vvxxRSbDKjG6xiDY5h+qbL9P9xiNO1ZDK8Vfh6ixuYeoCd6DHCJDX9tVhabBg97317irAPhFUlRHP02lU0jFPggoDdDdKNsltIiRz9hcqLhJpl1PgoKZPvhAZgJWJYn9+niV+g= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=cRouBr1m; arc=fail smtp.client-ip=40.107.92.85 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="cRouBr1m" ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=gpXhDbAjbXD3JANTKDYvbvHdK2yxxYqtXHMNCU5W7E1jT6lGlfTnZ+eyM4u9C1BcHLD68uzkLmG0VyQqsQBt/n3KIPQ1OXLlx8+s+Yca3N+BXuMpYdLjjf+e+tO2cqATkqh4SdcAo7JvDWNxecMQRy8hk3IdHy0Etfz4ST44UxgJEeMo20kS44OFCkLnsxTMC3YhtVK83zHPu+z507Mro8mbVO0yGa+ToVdDmCnFRyHbvaf7Mcs4oDuEIKzTbltUXEwLzhsHM7TvSn4quxcn/e/lWswYDdKUlem3kwnGyAGtJcf/YiJYDvq8Wa/8BNJYYEUnPyaqoAB42K50925jWw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=9m5z4j22vPdJjJPs+MBTWAJbpLlAEXZmcstO+GhBDys=; b=VZYln1Cbs1gTSAHr1HFXmXJO5rcQVBOwRrpU59Ik3qoZN0cExjzDKdVcJJdZPkZuEeaZY3ugWrSmGu6KllaUDUaOoBulv7SBNOStNSDl8Y5zFce3Vae2p9tqEJIFwzadYka9AKeeVwKkqnSjWavFaQZEHb7le1V/8Ob6mDYBwtStbQ/Un37DBfAQQx19ERoxjuj9o120MwHKS00wefgbFcHmG/2Sp3tARI6otOOz5JX7zBHafizNE4AZOkfnG4UCqdeEo/3OpE4vvryY+TnnHbtWVey9BQPJSHWwDqfHJn/ygG4clJ4PKDapB/S5Kb5TMv9KddwPZ36+e45Ab85r8A== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.161) smtp.rcpttodomain=redhat.com smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=9m5z4j22vPdJjJPs+MBTWAJbpLlAEXZmcstO+GhBDys=; b=cRouBr1mMBWK3Qrw+fV9HSzpZbYrMKDtoHkqtmbaO0pJvgI2K930jzPWtMG2Rqt2RXTBgtRJID9DW1wMMolQZFQXb/EOpS0a2ReZc99sThFGPX9Wm0I0iHPL+xFAoInOho/W5jO/Vi+zeEaMv4o9tgjAh9mXUoG59qc52caPFkLXq67VI0KJjiTuDe1Pss6MgkdAun/eIPkpGWk5QbTADulzKNzoyI4Mr007ED3+Hk0rq4FsFLd9/kH3Vwv49yLe78xNNaxCec+DYe/T95lSeJo5MqfjnlQCjXgz2n4slxlEA1I7No8553eOzLU837tbJc15ostv2CtySyPdm8zirQ== Received: from MN2PR03CA0007.namprd03.prod.outlook.com (2603:10b6:208:23a::12) by DM4PR12MB7670.namprd12.prod.outlook.com (2603:10b6:8:105::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7270.15; Mon, 5 Feb 2024 12:49:02 +0000 Received: from MN1PEPF0000F0DF.namprd04.prod.outlook.com (2603:10b6:208:23a:cafe::bb) by MN2PR03CA0007.outlook.office365.com (2603:10b6:208:23a::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7249.36 via Frontend Transport; Mon, 5 Feb 2024 12:49:01 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.161) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.161 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.161; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.161) by MN1PEPF0000F0DF.mail.protection.outlook.com (10.167.242.37) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7249.19 via Frontend Transport; Mon, 5 Feb 2024 12:49:01 +0000 Received: from rnnvmail203.nvidia.com (10.129.68.9) by mail.nvidia.com (10.129.200.67) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Mon, 5 Feb 2024 04:48:50 -0800 Received: from rnnvmail202.nvidia.com (10.129.68.7) by rnnvmail203.nvidia.com (10.129.68.9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Mon, 5 Feb 2024 04:48:50 -0800 Received: from vdi.nvidia.com (10.127.8.10) by mail.nvidia.com (10.129.68.7) with Microsoft SMTP Server id 15.2.1258.12 via Frontend Transport; Mon, 5 Feb 2024 04:48:48 -0800 From: Yishai Hadas To: , CC: , , , , , Subject: [PATCH V1 vfio 1/5] net/mlx5: Add the IFC related bits for query tracker Date: Mon, 5 Feb 2024 14:48:24 +0200 Message-ID: <20240205124828.232701-2-yishaih@nvidia.com> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20240205124828.232701-1-yishaih@nvidia.com> References: <20240205124828.232701-1-yishaih@nvidia.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: MN1PEPF0000F0DF:EE_|DM4PR12MB7670:EE_ X-MS-Office365-Filtering-Correlation-Id: 9333d91c-27fe-40f8-85af-08dc2648d432 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: nHeJ4RqfgJ8pXOtBVZWadCmgY+4eTXy3p/D+MSKpBPqeG7+AmMt9xOqLU/UXGeXHtfWL9ijRbY5HqBWVstEbbqUX/XULnBj9x1rrB4nTvuJ2I4MN9iaktnU7PFn3w2mGr0QUZd5ebOobwRbGz/QxLottfWPdAkV5Ifp0PfNV3NnLgFY32hGufipHRZchhp9wL77AmUKb17fF/+mdEVoWtFN1lpN+8UPeM1vVU+yXPOoF62/fVbHRfi6ohVau2U31NIbHLH1qgzYmqPz8HuD2EThNFxtIG+qSu4K+zxvGQBwsdtZouMKpj4tniWIQ6sVBwgu6Yy4AcgujeftF/PZpftqKDJN6N3WoEnotZwpy+0WYXi56DZnjKuYgrQWdwZvEBM+87EvvoscEismo7gIOWoQjJsyUxGGCXCqas0wwXfCActl0oLdQ3VnuBP3oE6B8CNr6wxGebftdWCYELhDxF6++ddMcjNKITIBWnzulvoKPCHLV1U4qDxRvPyJ1Gn2CJm1FbxX2tjY+skYKhc4p56KzLBnwRehi/2m9TGIQyRzW1fuAxzMe3p3Y3fKj0lD938O3zzzxrrfPBa/zL714TfOJJ9mq/qCFY52yP9kEqWD6R3E9DyG9j80wWyd7JvtdZsC7Db+pE6ixiXKd3g3G1v4sP/B4mpUyM6h85sD6oa0UrNgZ/mMIVg49LxdgnM4+arFZ2z1UXU6Ucp7IzafaaaowSizK785V3dgtSXdwS8aca64Aul6HE3u6h/UuxG/w X-Forefront-Antispam-Report: CIP:216.228.117.161;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc6edge2.nvidia.com;CAT:NONE;SFS:(13230031)(4636009)(346002)(396003)(376002)(136003)(39860400002)(230922051799003)(1800799012)(451199024)(186009)(64100799003)(82310400011)(36840700001)(40470700004)(46966006)(36756003)(4326008)(8936002)(5660300002)(4744005)(41300700001)(2906002)(8676002)(2616005)(1076003)(107886003)(36860700001)(86362001)(6666004)(47076005)(7696005)(316002)(6636002)(40480700001)(40460700003)(478600001)(110136005)(70586007)(54906003)(70206006)(82740400003)(7636003)(356005)(26005)(426003)(336012);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 Feb 2024 12:49:01.7161 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 9333d91c-27fe-40f8-85af-08dc2648d432 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.117.161];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: MN1PEPF0000F0DF.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM4PR12MB7670 Add the IFC related bits for query tracker. Signed-off-by: Yishai Hadas --- include/linux/mlx5/mlx5_ifc.h | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/include/linux/mlx5/mlx5_ifc.h b/include/linux/mlx5/mlx5_ifc.h index 6f3631425f38..cb08b5e36c21 100644 --- a/include/linux/mlx5/mlx5_ifc.h +++ b/include/linux/mlx5/mlx5_ifc.h @@ -12619,6 +12619,11 @@ struct mlx5_ifc_modify_page_track_obj_in_bits { struct mlx5_ifc_page_track_bits obj_context; }; +struct mlx5_ifc_query_page_track_obj_out_bits { + struct mlx5_ifc_general_obj_out_cmd_hdr_bits general_obj_out_cmd_hdr; + struct mlx5_ifc_page_track_bits obj_context; +}; + struct mlx5_ifc_msecq_reg_bits { u8 reserved_at_0[0x20]; From patchwork Mon Feb 5 12:48:25 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yishai Hadas X-Patchwork-Id: 13545548 Received: from NAM11-CO1-obe.outbound.protection.outlook.com (mail-co1nam11on2078.outbound.protection.outlook.com [40.107.220.78]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BEC0539877 for ; Mon, 5 Feb 2024 12:49:11 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.220.78 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707137353; cv=fail; b=FwbPcKPIwjiwpHE16XpSGVoIo9aZ2+EcilHJPY2p30E8L9KfR70ILEBzGXaYluy+Y0JFmwRc1/XfR0pXXcVZ0PV0r8aBoB+/WX9abU3ocJZ1k8RwQMy7W/ALjXR2sgo+BZOt5oxXZ/LWEOw9baOvmyKSzA9ZiT2w0za2lz0inmg= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707137353; c=relaxed/simple; bh=Iu0Mkwkk/Zz+xDPN41cOqoKOrvRqWJ1Q7xFwp0QzU38=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=cdmNhW0JDFbWLMzrYPLE5Lp5Mw43oAciGZhP23+/jyU0PqQdp8hLA+2lU1UmvjqaSFq/bTgoL+1AEaHpdemgdeHyiSpPIfOyU/POCoTD7FsjY4dnEZCbzIuR6wYw9xddB2bW09gUco2r7BcnA9jKTYveT0df7fH2KURJ2TrVHq0= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=DkkFr+Nt; arc=fail smtp.client-ip=40.107.220.78 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="DkkFr+Nt" ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=LJiDus3BDeTEz9VdJeUQWAZEZNev6/ewraNcgnHpyNR/jMnaQLqbbj8dNuQgEgjL9pCm66DUj0mkxgF1OrkXjzvk2+J3+6Ag5bz5MkliQE6O/LoSGO9x1O6iQqrUOldvRgCQOc7So84CKJ6GmBxZMpC7RMalkqyld3N27egQvgToVlkY2RgbaLW41PnwnfDnc9reNGe/94b1yEhmrF+MUY6JAB6zILeBuVJCwysw984PVJ5EFBG2jVpM/7cRtCncuuS4r0y6Z17DpoP+4W0zqGMo+AhUFmMu/vMYOruNIkQ6AAxFKZ6bNo4vRIQIP659HKzqV87Byjtejck0G/GBGw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=bMWup6zJaSYCTJHNN7iOeX7R8nq0/cq9ivmLnVgMpg0=; b=oY0wUkWroDyXbFZg4/66tj2Kt25pi1LjFFGTaERxBOnd57wLGPHisHL/y3pR19w8Un00rhZf+wJ1UtkkOc7xGcI6MwRmOZHgbkX2nEp+SXV55GGK8JSYT/dl/a4Ulz4dxmEdPr7Nxdl9CEMUCfQZW49+SDALaYx91hRBsTBeSSb8iOypmcnW6+dtivs3WWVIufXr2p/qp4n88WlRRQZNC7F6bEIFo0s6LkOJsYa14smNNHhmWxoNLmTmK9osPfxXMerjxVpgu9oXmzS7O5LmU1o3zE67sL+0f98yI0jQIKMUYpB4TMKpdj4ZJsLdBTES3DYoUGOuaae3E3h6c5w0LA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.161) smtp.rcpttodomain=redhat.com smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=bMWup6zJaSYCTJHNN7iOeX7R8nq0/cq9ivmLnVgMpg0=; b=DkkFr+Ntsyq/IoDcx8on1V1JAujqJ/YANNO47Hr9XRIj+bSEv5sZVT7f4iTfpTcWW9PISM/R7QGoZqbSA9CFbaU1pef86rj3NghmSyQdYi2Fa0dWMRa7dkRaAVGmyHnV6hsJtNyEciKf81PB5Qo0WkBqk/fKMrZMBMf0NE08ccDFXBSsVt4MEzXUfi9B6anDJEkV+DcrWTKjtXOVpHgjHpkVtEi9ZiAO8lcnXA2mrIwyPiga3xeuP+JjgFPeCQHQ2y3auRmIEo6bIdPGScaGBXPD0KtRdOwn2oLFWvHKWuwLIvaBZuwQ+5RL3xj7rM1FoR9f82DQWk4Oz9VAoiI7Fw== Received: from MN2PR03CA0016.namprd03.prod.outlook.com (2603:10b6:208:23a::21) by SJ2PR12MB9240.namprd12.prod.outlook.com (2603:10b6:a03:563::13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7270.16; Mon, 5 Feb 2024 12:49:06 +0000 Received: from MN1PEPF0000F0DF.namprd04.prod.outlook.com (2603:10b6:208:23a:cafe::12) by MN2PR03CA0016.outlook.office365.com (2603:10b6:208:23a::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7249.34 via Frontend Transport; Mon, 5 Feb 2024 12:49:06 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.161) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.161 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.161; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.161) by MN1PEPF0000F0DF.mail.protection.outlook.com (10.167.242.37) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7249.19 via Frontend Transport; Mon, 5 Feb 2024 12:49:05 +0000 Received: from rnnvmail205.nvidia.com (10.129.68.10) by mail.nvidia.com (10.129.200.67) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Mon, 5 Feb 2024 04:48:53 -0800 Received: from rnnvmail202.nvidia.com (10.129.68.7) by rnnvmail205.nvidia.com (10.129.68.10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Mon, 5 Feb 2024 04:48:53 -0800 Received: from vdi.nvidia.com (10.127.8.10) by mail.nvidia.com (10.129.68.7) with Microsoft SMTP Server id 15.2.1258.12 via Frontend Transport; Mon, 5 Feb 2024 04:48:50 -0800 From: Yishai Hadas To: , CC: , , , , , Subject: [PATCH V1 vfio 2/5] vfio/mlx5: Add support for tracker object change event Date: Mon, 5 Feb 2024 14:48:25 +0200 Message-ID: <20240205124828.232701-3-yishaih@nvidia.com> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20240205124828.232701-1-yishaih@nvidia.com> References: <20240205124828.232701-1-yishaih@nvidia.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: MN1PEPF0000F0DF:EE_|SJ2PR12MB9240:EE_ X-MS-Office365-Filtering-Correlation-Id: a008f157-b5b4-43f2-7c61-08dc2648d6b1 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: If7dl+LSznzsRd9rAgBYs8BsH/Vx0BIunlOyy8Odo3tHxVV9bcGh8dHTUSRhVolyA8wsRAmDACh7usyBgHH2X08P5wBEtw5m4wlvWuYTi/BXoIoFdcheBlQY4dPnxg4IPR9OPuQRymSGty7olgsDSCHuklc2cwjinNqEh47Bq94ncL9eyiX2roxGWJ3Ekd1ZUH1GpQWrRjCgW/rCGQ+ojGXVT0Dd0OV+X2KWcCVdbMDQWWm/AQAK1vPrCYrsLwGp7W6A8kXcXFkhggL7TCQpSaIBuh0Qv40mLk9pkNXELEJX8AUnzYzQ9ZkmPgOnczM/lIz93oeFl0t3/Qw42u9NfP8S8Ge0IXzs6W8jVOIKtAWLc4trJjz+4uYpVGHJQouJg3+SJ2ktkTLHi4VK/TgrT4ZU31DvgrTLPe7kEmuxrgHgS55LwY4sH5mRYrUVfvYZ31fJIyJ7vOcis9Qq/0Y0hGiBjTVByOvF4bjeUEOI16xKNi5f/Z5U+75Jl+TkP+NPQlSrkmciJA45r8V0n1MChPog8+leTUiY1VwFqMkrsJEcLCGiDDJUwgnrGCJnTd/VBYBhHRkjoWJCVSQAQkBq1YF0+fYHILiz8AUB8wQUVw/Pgp7BZurU1nS7zX1/KHhO0R0xtxYo5tHdZ+Ah5A+0Ux1TaefxKzyRObCKUeCJmIDqYJEzANA5xwOUJP4AQ9pn3TP6I/UUOBbk8C89BIjhFgM5abhSaKHCHEcyIpSmm6Y= X-Forefront-Antispam-Report: CIP:216.228.117.161;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc6edge2.nvidia.com;CAT:NONE;SFS:(13230031)(4636009)(39860400002)(376002)(396003)(136003)(346002)(230922051799003)(64100799003)(1800799012)(82310400011)(186009)(451199024)(36840700001)(40470700004)(46966006)(36860700001)(336012)(47076005)(83380400001)(1076003)(2616005)(426003)(26005)(40460700003)(86362001)(7696005)(40480700001)(478600001)(107886003)(70586007)(8676002)(4326008)(6636002)(8936002)(54906003)(316002)(110136005)(70206006)(7636003)(356005)(6666004)(82740400003)(36756003)(41300700001)(2906002)(5660300002);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 Feb 2024 12:49:05.9193 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: a008f157-b5b4-43f2-7c61-08dc2648d6b1 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.117.161];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: MN1PEPF0000F0DF.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ2PR12MB9240 Add support for tracker object change event by referring to its MLX5_EVENT_TYPE_OBJECT_CHANGE event when occurs. This lets the driver recognize whether the firmware moved the tracker object to an error state. In that case, the driver will skip/block any usage of that object including an early exit in case the object was previously marked with an error. This functionality also covers the case when no CQE is delivered as of the error state. The driver was adapted to the device specification to handle the above. Signed-off-by: Yishai Hadas --- drivers/vfio/pci/mlx5/cmd.c | 48 +++++++++++++++++++++++++++++++++++++ drivers/vfio/pci/mlx5/cmd.h | 1 + 2 files changed, 49 insertions(+) diff --git a/drivers/vfio/pci/mlx5/cmd.c b/drivers/vfio/pci/mlx5/cmd.c index efd1d252cdc9..8a39ff19da28 100644 --- a/drivers/vfio/pci/mlx5/cmd.c +++ b/drivers/vfio/pci/mlx5/cmd.c @@ -149,6 +149,12 @@ int mlx5vf_cmd_query_vhca_migration_state(struct mlx5vf_pci_core_device *mvdev, return 0; } +static void set_tracker_change_event(struct mlx5vf_pci_core_device *mvdev) +{ + mvdev->tracker.object_changed = true; + complete(&mvdev->tracker_comp); +} + static void set_tracker_error(struct mlx5vf_pci_core_device *mvdev) { /* Mark the tracker under an error and wake it up if it's running */ @@ -900,6 +906,29 @@ static int mlx5vf_cmd_modify_tracker(struct mlx5_core_dev *mdev, return mlx5_cmd_exec(mdev, in, sizeof(in), out, sizeof(out)); } +static int mlx5vf_cmd_query_tracker(struct mlx5_core_dev *mdev, + struct mlx5_vhca_page_tracker *tracker) +{ + u32 out[MLX5_ST_SZ_DW(query_page_track_obj_out)] = {}; + u32 in[MLX5_ST_SZ_DW(general_obj_in_cmd_hdr)] = {}; + void *obj_context; + void *cmd_hdr; + int err; + + cmd_hdr = MLX5_ADDR_OF(modify_page_track_obj_in, in, general_obj_in_cmd_hdr); + MLX5_SET(general_obj_in_cmd_hdr, cmd_hdr, opcode, MLX5_CMD_OP_QUERY_GENERAL_OBJECT); + MLX5_SET(general_obj_in_cmd_hdr, cmd_hdr, obj_type, MLX5_OBJ_TYPE_PAGE_TRACK); + MLX5_SET(general_obj_in_cmd_hdr, cmd_hdr, obj_id, tracker->id); + + err = mlx5_cmd_exec(mdev, in, sizeof(in), out, sizeof(out)); + if (err) + return err; + + obj_context = MLX5_ADDR_OF(query_page_track_obj_out, out, obj_context); + tracker->status = MLX5_GET(page_track, obj_context, state); + return 0; +} + static int alloc_cq_frag_buf(struct mlx5_core_dev *mdev, struct mlx5_vhca_cq_buf *buf, int nent, int cqe_size) @@ -957,9 +986,11 @@ static int mlx5vf_event_notifier(struct notifier_block *nb, unsigned long type, mlx5_nb_cof(nb, struct mlx5_vhca_page_tracker, nb); struct mlx5vf_pci_core_device *mvdev = container_of( tracker, struct mlx5vf_pci_core_device, tracker); + struct mlx5_eqe_obj_change *object; struct mlx5_eqe *eqe = data; u8 event_type = (u8)type; u8 queue_type; + u32 obj_id; int qp_num; switch (event_type) { @@ -975,6 +1006,12 @@ static int mlx5vf_event_notifier(struct notifier_block *nb, unsigned long type, break; set_tracker_error(mvdev); break; + case MLX5_EVENT_TYPE_OBJECT_CHANGE: + object = &eqe->data.obj_change; + obj_id = be32_to_cpu(object->obj_id); + if (obj_id == tracker->id) + set_tracker_change_event(mvdev); + break; default: break; } @@ -1634,6 +1671,11 @@ int mlx5vf_tracker_read_and_clear(struct vfio_device *vdev, unsigned long iova, goto end; } + if (tracker->is_err) { + err = -EIO; + goto end; + } + mdev = mvdev->mdev; err = mlx5vf_cmd_modify_tracker(mdev, tracker->id, iova, length, MLX5_PAGE_TRACK_STATE_REPORTING); @@ -1652,6 +1694,12 @@ int mlx5vf_tracker_read_and_clear(struct vfio_device *vdev, unsigned long iova, dirty, &tracker->status); if (poll_err == CQ_EMPTY) { wait_for_completion(&mvdev->tracker_comp); + if (tracker->object_changed) { + tracker->object_changed = false; + err = mlx5vf_cmd_query_tracker(mdev, tracker); + if (err) + goto end; + } continue; } } diff --git a/drivers/vfio/pci/mlx5/cmd.h b/drivers/vfio/pci/mlx5/cmd.h index f2c7227fa683..0d6a2db3d801 100644 --- a/drivers/vfio/pci/mlx5/cmd.h +++ b/drivers/vfio/pci/mlx5/cmd.h @@ -162,6 +162,7 @@ struct mlx5_vhca_page_tracker { u32 id; u32 pdn; u8 is_err:1; + u8 object_changed:1; struct mlx5_uars_page *uar; struct mlx5_vhca_cq cq; struct mlx5_vhca_qp *host_qp; From patchwork Mon Feb 5 12:48:26 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yishai Hadas X-Patchwork-Id: 13545549 Received: from NAM04-MW2-obe.outbound.protection.outlook.com (mail-mw2nam04on2054.outbound.protection.outlook.com [40.107.101.54]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 557DC3A1CF for ; Mon, 5 Feb 2024 12:49:14 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.101.54 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707137356; cv=fail; b=NSCB10fjTNuRO2vhSEiSMPOcXsZkch0UjDQ15wayU66LeOPLBugw4wxdFdcVeDO02OHtNk10ovOoBWH7Ue20mPVvf+MDpmZXLUiF3Nw857xGK3BFx+PuPjArb0bTb0i3oBj+r53IsE9dUI5VYoxZXZux7LhNcgPJgOzcPqZclS0= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707137356; c=relaxed/simple; bh=uHbudj+CLGNYYVRUkL+cHrG+mz1bcEvVpMkr1Qr2vdQ=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=eguzYATOQpiaINlwqGqHeulM5H+BKXL4kC3gN/TkhyFdUCyeGQrpyL2w51uNe2SHv09aZBY8ZhjGQshPeDEXmlvCZBAKS449mxBqnjFBRncsk6kW+IJXF9Jq4hsq7gCfUV59slv/fQCY4hZbAwamee3g8bXwSBPqRKLPyMUTNhU= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=uakoUYr6; arc=fail smtp.client-ip=40.107.101.54 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="uakoUYr6" ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=PZ/73Z21k0S9+7cwgXKTQ4IyL/WTgDbXBqKck8Ez+UXWiYH30ISV1Wg+ycJFVft46KfpgK/hlFOKZsBdRGB/dD+ZzNb1oooa/gt5CvZdlB8M0W2Q5905/Ory1T0TT2xf2zpE3Vw1s2ImiwVTufNtnLSXHd0K3VYgpSxtRSu8QnP+Q6+sJUx81J5TEThpt3uRYsOoeSx0rpMJ4SnuVSjGK5TmKKsmNuaAOFctSbvFN+11h96h4039i/OYJckeMgQ4B6I5V5UlcSX1jDml+Wg2lTzNI0uMBREXIsNPjWa+QOMx4CZMfEIt8jDVvzG5rSHlELXcJDbhY8gVWZ9rhjPuQQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=Z7ddJGwq+KL68SjWmlKvzp36+D+y5O7Wo2ndtdkPdxc=; b=nXHFill+/ZxNQ3ELxkpqEP/+NTCms1yEHoquN4KGHUQ/+YVScED7CCvVN1sXsWsvZbjwYEnuFHefcXwjBqpOy3KQfzlJDPBeJBDS5ar6r/8I5qZ1vjlV10Qmpe80vtgjmX3JIW8Cw/Oonlp3Y2B3Xl05Ydhz7R+uVS2oz1oX0sD1eZMfmJCHop/XyJNkfeJUfxUAEd4xxtacr88RS2OFzQmCLSt2PwcQsTchl3My1qF9c+wlp7gq3Tb6sfuR2eVsN6jraEqGZJvJxmJBKKtvDtulbJ4Hi3+k/V+s2x04fw/VaITi1K/lWWRLTRDkjMdMo7JxNJyGsQcjaSAVB5JgqQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.161) smtp.rcpttodomain=redhat.com smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Z7ddJGwq+KL68SjWmlKvzp36+D+y5O7Wo2ndtdkPdxc=; b=uakoUYr6ltGuU6ba5UnO+5DYmjcAjiu6h4OV5V4oTD9s73rxl0VYbqR2FNX07eB8Kl80DqmQSPbW+NpXpRt/XLUn9QgFj4+Ac19wg1cS/u/edb6cpbV/UHIddk0QlWUBgV8ZffifOkxEOf6NXXvmHlNrM7aIraF7Jt9ItBA/PQWoOTHOvD1OiSTA/Er15dSdTJvRgENTyyKYp0ah2uGHxNHEi70JlT68K6tAAHOj648q+xKKl8aAv0G63lscI3apYE46MakBqJYpVbip+pmM0+ePsRmb96reQDocDWav0xfE6BO+ncaegSqnlumadxf5YMykCA+KogD5Tgo0+v+hQQ== Received: from BL1PR13CA0244.namprd13.prod.outlook.com (2603:10b6:208:2ba::9) by SJ2PR12MB8717.namprd12.prod.outlook.com (2603:10b6:a03:53d::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7270.14; Mon, 5 Feb 2024 12:49:10 +0000 Received: from MN1PEPF0000F0E0.namprd04.prod.outlook.com (2603:10b6:208:2ba:cafe::2e) by BL1PR13CA0244.outlook.office365.com (2603:10b6:208:2ba::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7270.14 via Frontend Transport; Mon, 5 Feb 2024 12:49:10 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.161) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.161 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.161; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.161) by MN1PEPF0000F0E0.mail.protection.outlook.com (10.167.242.38) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7249.19 via Frontend Transport; Mon, 5 Feb 2024 12:49:10 +0000 Received: from rnnvmail204.nvidia.com (10.129.68.6) by mail.nvidia.com (10.129.200.67) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Mon, 5 Feb 2024 04:48:56 -0800 Received: from rnnvmail202.nvidia.com (10.129.68.7) by rnnvmail204.nvidia.com (10.129.68.6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Mon, 5 Feb 2024 04:48:55 -0800 Received: from vdi.nvidia.com (10.127.8.10) by mail.nvidia.com (10.129.68.7) with Microsoft SMTP Server id 15.2.1258.12 via Frontend Transport; Mon, 5 Feb 2024 04:48:53 -0800 From: Yishai Hadas To: , CC: , , , , , Subject: [PATCH V1 vfio 3/5] vfio/mlx5: Handle the EREMOTEIO error upon the SAVE command Date: Mon, 5 Feb 2024 14:48:26 +0200 Message-ID: <20240205124828.232701-4-yishaih@nvidia.com> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20240205124828.232701-1-yishaih@nvidia.com> References: <20240205124828.232701-1-yishaih@nvidia.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: MN1PEPF0000F0E0:EE_|SJ2PR12MB8717:EE_ X-MS-Office365-Filtering-Correlation-Id: 5e4f8517-1e7e-498d-41cf-08dc2648d951 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: MvcQXCKo8NpNeJMs/O019sa2EH6nUpPL2muTFQDcKHW9PZgqArUH4n8ndtqxzbvZL8FMDzAZBlln8i2waT9GB7C8xKZP4xtf02gwkK+FkNTKZ0I38bCBVTPunFb17CoN2qigeWYhSci0EImd4DxEy6powGa2eaR8/JYBzKmKwMmBQzlkagIEsNgYUdvI0+Q8eUcA/iYuH7jdD4Vs+lwFxwiuERRohgmEwirKkcntUD0mSzagAxvy+Rb7eA4xfHRoRfMdgL5KngsUd03qfPsWigANqntXgUEeQRZZhoRiNC8laaDN9wS7NQUzLN3hOKRvsnp+sEYug3XKs1UzTZwNDp3YVL4QX5ATupHzjoqxR3Lu97+3LbEBGRohIY2pe3RjgDkqVCmJBKNimBkrzUJiSPjTNJii8te/QMiskeLCUGQ7ivCsG4mSS+fJykJ2ZODN6BsHVr4BGpGkIWIavQOAJtHy0VRgMCGXAkeziKScDA6qb+OWsHcfWhj4nUwSc6EZrhzSA/7dWthzNHWxevr44Goa8ozORRrDwK1cPcxA4f4Nl6AhjyuHpYHOIJUTusZd4CMXKfnZ33rHfuuYmNEfMon9RuUTATN7buaDDS737PVI1DlUclCJiFX8vX7oaW2kns8mGuKSIMqU3kMN254M0NoRZEP7MauqJIcMtmGpgMfPq2NJ2mqAv6IrrfiA3nocF9ATrwdZz28wndUkQtUtAHEV2qpc+2q0wKiK3xxv6Bc= X-Forefront-Antispam-Report: CIP:216.228.117.161;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc6edge2.nvidia.com;CAT:NONE;SFS:(13230031)(4636009)(346002)(136003)(39860400002)(396003)(376002)(230922051799003)(82310400011)(451199024)(64100799003)(186009)(1800799012)(46966006)(36840700001)(40470700004)(41300700001)(2906002)(8936002)(4326008)(8676002)(70586007)(5660300002)(70206006)(110136005)(6636002)(54906003)(316002)(36756003)(86362001)(36860700001)(356005)(82740400003)(7636003)(7696005)(336012)(478600001)(26005)(6666004)(83380400001)(426003)(47076005)(1076003)(107886003)(40480700001)(40460700003)(2616005);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 Feb 2024 12:49:10.3218 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 5e4f8517-1e7e-498d-41cf-08dc2648d951 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.117.161];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: MN1PEPF0000F0E0.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ2PR12MB8717 The SAVE command uses the async command interface over the PF. Upon a failure in the firmware -EREMOTEIO is returned. In that case call mlx5_cmd_out_err() to let it print the command failure details including the firmware syndrome. Note: The other commands in the driver use the sync command interface in a way that a firmware syndrome is printed upon an error inside mlx5_core. Signed-off-by: Yishai Hadas --- drivers/vfio/pci/mlx5/cmd.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/drivers/vfio/pci/mlx5/cmd.c b/drivers/vfio/pci/mlx5/cmd.c index 8a39ff19da28..6b45bd7d89ad 100644 --- a/drivers/vfio/pci/mlx5/cmd.c +++ b/drivers/vfio/pci/mlx5/cmd.c @@ -614,8 +614,13 @@ static void mlx5vf_save_callback(int status, struct mlx5_async_work *context) err: /* The error flow can't run from an interrupt context */ - if (status == -EREMOTEIO) + if (status == -EREMOTEIO) { status = MLX5_GET(save_vhca_state_out, async_data->out, status); + /* Failed in FW, print cmd out failure details */ + mlx5_cmd_out_err(migf->mvdev->mdev, MLX5_CMD_OP_SAVE_VHCA_STATE, 0, + async_data->out); + } + async_data->status = status; queue_work(migf->mvdev->cb_wq, &async_data->work); } From patchwork Mon Feb 5 12:48:27 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yishai Hadas X-Patchwork-Id: 13545550 Received: from NAM02-BN1-obe.outbound.protection.outlook.com (mail-bn1nam02on2072.outbound.protection.outlook.com [40.107.212.72]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1655F3B18D for ; Mon, 5 Feb 2024 12:49:14 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.212.72 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707137356; cv=fail; b=rtCfGKVeUYly8IFkSGxX+xVxAyfGjdcu0aN9Z+8Ywv3iKoi55qhux5rZFGCXuvwtVTFyf4KFyvdK8s3GsokY2UVb7yky5GPTcqwsFXWmDtD7Ty5/9btNB05BMIEcdAvHeDXj3vskj5IpcVRRODIfW+7vhNLU5BtzsX1Lf0D9Na4= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707137356; c=relaxed/simple; bh=kKY44mAbkzvQDJa1uCVN+G1ZxLpMZEV/V+DwtkdC8Yg=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=h8EAb5M+IYgeU5+34O6jjXH3Xya21rdzUbrcqa2YgKMvejmdlPuRaC7RaY+S9bnQ9RFQdXFJ8fu8FoImpCj1QrfkDYb7Q26aqJuyFtSr1WEBHHCMmSGjS3rdoaf5rtjBVTDTSxhZVkPmto4a5mb8zvnTtDqEb8etq/k0NyAdoTs= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=uOk8t+DV; arc=fail smtp.client-ip=40.107.212.72 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="uOk8t+DV" ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=VIBr2vIpNEyE9HFSsD//+M8VEUUY5CJ9CTyuc0gBhX2lXwf62ea9EM06OzXg9xwo0R2rGAfxMCUm2cL08yHaZg4eCllYML8fiolExm6rBtiPxt43AC085pnEh8IBvhjgQnMvYWFi0AJoFRAfDWiD0ICmKzRLt5xM/VNAmfPyCe6cwnGUtgTQTp6Q49h91yveKaeIgNA93Kedp2n5N4zRo1FGNMumm3gHaGTa+8AmqmGaSPSSi8p/AKU+tkUA0BksQ19XndkwNH1c3pTUkmslEjbxpWtOpbpTHHU34H0BR9GlozpS0c/KHkZoXrtXf/jkXhyYu0gSn1SAOBWMhLNXPA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=xqWsz5I2xppMYSxuzY+0NPD+ByLi9NzvBqvqTVebFgw=; b=UmRCvjMmlih0mRV+LzZgKm/WuAfgDhK71KfjaBAdZZ5DJKsNQZNmBzE+MT+h0PG1KX06SKNZudtb1lP5UsNK+fIxrA/yDyhZ0TmxsaM5v5YvXGTar8qycxvaq3AKGF7Hbo1JZEWNEv/3ouLjY9Wi9n/is+bYgRe2WEvNdwbDqu0s/+XkRCOl4NVxyYzwpSk0InbOFHXU5Wm02PPN/vapn8EvsGkwniiScMRiepbjdIJG+J+Dq+vub+6edi3E9gqJ6lxn2GMm6pRAoafN86BUmEVV7DPiu8X7L0+30hL5HjZnooCwbOUblzy81fYcW5npONQLrPxJDH30Cu2zFSqgsg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.160) smtp.rcpttodomain=redhat.com smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=xqWsz5I2xppMYSxuzY+0NPD+ByLi9NzvBqvqTVebFgw=; b=uOk8t+DVsbBtrKgCXwpQLGHHjP6lNuo3Tm7TOwRhAZJpUJU/qJt32cPhN2nRMZTll1vvvOe70MvJAZJ9XUTkt2hXbQIG0Z+PfJ/Q3ByRqSkBRtKgdyMNRC1ejFOB8iZK6oCmhL7BsnK2G4Vi7mnkasECLJSFbP09hlXTTXqJwvPzCBDKA0bBZ5TQ8Py5x+lGfogHyWX7h4wjls9dRm/M/3qmjWYeE+brJsl4Tow5hmhSt4QJeLmxYcVFqXo3+AeSZI0cNP5REoOHaaAAeH+MGjW6JQ0hW+iUIPdIZ8rXK6dtxndQN/EPy1obQLXv5DoCf1uHIJ+dgzupNCtd5m8iyw== Received: from BYAPR05CA0087.namprd05.prod.outlook.com (2603:10b6:a03:e0::28) by CYYPR12MB8729.namprd12.prod.outlook.com (2603:10b6:930:c2::5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7270.16; Mon, 5 Feb 2024 12:49:11 +0000 Received: from CO1PEPF000042A9.namprd03.prod.outlook.com (2603:10b6:a03:e0:cafe::1c) by BYAPR05CA0087.outlook.office365.com (2603:10b6:a03:e0::28) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7270.17 via Frontend Transport; Mon, 5 Feb 2024 12:49:11 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.160) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.160 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.160; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.160) by CO1PEPF000042A9.mail.protection.outlook.com (10.167.243.38) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7249.19 via Frontend Transport; Mon, 5 Feb 2024 12:49:11 +0000 Received: from rnnvmail204.nvidia.com (10.129.68.6) by mail.nvidia.com (10.129.200.66) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Mon, 5 Feb 2024 04:48:58 -0800 Received: from rnnvmail202.nvidia.com (10.129.68.7) by rnnvmail204.nvidia.com (10.129.68.6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Mon, 5 Feb 2024 04:48:58 -0800 Received: from vdi.nvidia.com (10.127.8.10) by mail.nvidia.com (10.129.68.7) with Microsoft SMTP Server id 15.2.1258.12 via Frontend Transport; Mon, 5 Feb 2024 04:48:56 -0800 From: Yishai Hadas To: , CC: , , , , , Subject: [PATCH V1 vfio 4/5] vfio/mlx5: Block incremental query upon migf state error Date: Mon, 5 Feb 2024 14:48:27 +0200 Message-ID: <20240205124828.232701-5-yishaih@nvidia.com> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20240205124828.232701-1-yishaih@nvidia.com> References: <20240205124828.232701-1-yishaih@nvidia.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CO1PEPF000042A9:EE_|CYYPR12MB8729:EE_ X-MS-Office365-Filtering-Correlation-Id: 32912d1d-28c4-4fdd-c4d6-08dc2648d9de X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: uxHBKHNQi4Y3DuSj0qM5wb8V4sIvt/zA3t6jXt5P1LlgTnxCnPV+7oMZ8Rl+qNuZAeKRljcNvWeTsL8+tik0J3QSii2lr8p7Am3hUOb0ObzLXC4Tl5/IgHfPQuZjnGrrgiflqLW8T0D2lJiSC9puBLUb4dvDrhng3ANDrc5UFRemftOCLuuEzBf/F75HqcxdeL6dBsyo+BMP+0UjfWfI0YlYj1U7mjDpqM0pk814cyB6mttHrIG9w04wp6T/CtY6jYhoGDGgJzqAX1F2gnA/ne1kbiayDeULeLb5Vb4rcRxZ197ABzsGV4B97nbWvQCyg1Lg9Q9g3Sbz4rQt2x5J0xFKEL2EQAy/dhKyIcUIic6phZLkqy//0+gNQe9REIXHhL5EpkvO9L5/ZAmOXvIM6YvjErcgN4qevRBGk69w1xn4gzoc1J7v31nDTLmzWuaDDj7puvVUiTUb+/Uj/LvKPuE5Oy2AS1ZAF2dH9jGnmGRga4XrJ+5HgxiVMm9SXx8PazLk+uPVe6u7r4IP54KiNTF0N87ZfTqsQot1TRx1eDJiW4kaYUCvzurKrKFQUah6iUCf8jbq9Y+MJCLo4WfhnCPCaGhygAGSg/Rh95pqVUA92c7SY1fHim930oo4r2UObP/xkmK0oyORkXcVZKl6mM8Rj16YFdb++hWherYgLzZ8FInOASSjczFwp6sMPrPQ1TXwQIJ+yqeQ4CBk9bOAlwM0LxmX5ael+VjABml5M0JxLbd1QkHkaT3z/+duAPPr X-Forefront-Antispam-Report: CIP:216.228.117.160;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc6edge1.nvidia.com;CAT:NONE;SFS:(13230031)(4636009)(396003)(136003)(39860400002)(376002)(346002)(230922051799003)(451199024)(186009)(82310400011)(64100799003)(1800799012)(40470700004)(46966006)(36840700001)(336012)(2616005)(426003)(1076003)(40460700003)(40480700001)(107886003)(26005)(41300700001)(36860700001)(70586007)(478600001)(36756003)(47076005)(54906003)(316002)(6636002)(70206006)(6666004)(7696005)(83380400001)(7636003)(82740400003)(356005)(5660300002)(4744005)(2906002)(86362001)(110136005)(8936002)(4326008)(8676002);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 Feb 2024 12:49:11.2961 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 32912d1d-28c4-4fdd-c4d6-08dc2648d9de X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.117.160];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CO1PEPF000042A9.namprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CYYPR12MB8729 Block incremental query which is state-dependent once the migration file was previously marked with state error. This may prevent redundant calls to firmware upon PRE_COPY which will end-up with a failure and a syndrome printed in dmesg. Signed-off-by: Yishai Hadas --- drivers/vfio/pci/mlx5/cmd.c | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/drivers/vfio/pci/mlx5/cmd.c b/drivers/vfio/pci/mlx5/cmd.c index 6b45bd7d89ad..6800e4ffe9ee 100644 --- a/drivers/vfio/pci/mlx5/cmd.c +++ b/drivers/vfio/pci/mlx5/cmd.c @@ -121,6 +121,11 @@ int mlx5vf_cmd_query_vhca_migration_state(struct mlx5vf_pci_core_device *mvdev, } query_flags &= ~MLX5VF_QUERY_INC; } + /* Block incremental query which is state-dependent */ + if (mvdev->saving_migf->state == MLX5_MIGF_STATE_ERROR) { + complete(&mvdev->saving_migf->save_comp); + return -ENODEV; + } } MLX5_SET(query_vhca_migration_state_in, in, opcode, From patchwork Mon Feb 5 12:48:28 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yishai Hadas X-Patchwork-Id: 13545551 Received: from NAM12-MW2-obe.outbound.protection.outlook.com (mail-mw2nam12on2043.outbound.protection.outlook.com [40.107.244.43]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B3AD62E416 for ; Mon, 5 Feb 2024 12:49:17 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.244.43 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707137359; cv=fail; b=UAE3aL0SkgxOr6Tmo2a3gsEQQFE8Rv1MXqkmPbeQ4PV6QXhhNwO6yrY9QP2laZqdAstWAOEXWJjcOAOZ6q5uXAsLnh6LGU0u2nojyEZSIXCbg3hnE/i78YU6sCkRGEnsHs5I6Sy8D1O94l3ERaA/h3Oi6id2XfNLgrRqffP4BEc= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707137359; c=relaxed/simple; bh=Ntxja4oN0lZkVDvWQPaxE+WCfSw2oLqngq5w2DwS3dI=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=C+rNuPb7W0WXwkTpjdMtwl458o8ledY3I1l+N/pDNljv6BrEOPNS4aVwm242WOyWjtzJmR2ThKSB2dh/5iVROACw0dNsqhBgPVHLWJPBjsV0LIPWYudaameDGlVg3C8VKdPtoN6pf2gvX8x2gwwVZhSlEpxd+ycSYNaqw8LbZ9c= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=lyPKDL14; arc=fail smtp.client-ip=40.107.244.43 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="lyPKDL14" ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=W8JzLpXpmAYESHlkSnf280JUiBtVJgCkGTM+GnTODomaBMjU1NEB2t3U+scx+Q9XsIBx8ExY59goSOoi6CwBbPhVGVDTun/IdZHzaAt+7YFDsEXqMOC6tEbwZCcKIXsx/Pt2rwMBw0IvhqnFxH7xUTK5Xr22P5iRUq44nEGR7Zs/KY7RdC6DO9t2VRuPbFbMlNhXUofwPeOFJlVU4Iag8pIC8Fnc8D9hWowOEL5qvN9aCU3aIaw/tP8JP7lqKED8G8XdRfiAZhowN6eNzHeU+O23uk13u4rq/aNvhGOz/Zan7FGUAiSNMih4cMrMyjQ7mH+VIGo4fwolLl9EcAl4SA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=SvK2uwQK8OjkuxkT0j16/tjZMW2dOXK80PYMkLbUKME=; b=M047fee8zFEfmIHtglfk2XZWijnJR9Kc1Vnr711LxdLEYqYxRcdkIxdT9x+Gq4G1H2/VEg61xcDokNi4R67ukhhCKocS78CShm8xFtEMqpS0T4fs6S4Moo3HHBCGc55PbW2/RiLLrvfRr6ckXFFMxYePzI52A9y1Xy/kudf10HHrtmKvbgpzo++iL4vJO+j9qG7wl3xliNkyGVj2yR1usI3DlHnUSNJJgVLupHut3lc9elucYW+pY3cGi7uE8OsGSXiYdrrrdSkkBI15FrwPfGakxGvkdfBapI9Kh8HZpgN1qMDbkLpYje3WaouXwIS336JG23ELM3oFvDhH7iUsgg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.161) smtp.rcpttodomain=redhat.com smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=SvK2uwQK8OjkuxkT0j16/tjZMW2dOXK80PYMkLbUKME=; b=lyPKDL14bUbvB9CaSV5ikMr7+rKKO9ymtQiLP3/aAxtGyfziNBRh3/cxg0tEUELu81IsDq76eNVkhCG1HUAHMDg7awCTRUrMtKCnPPdl7ayQHMRgVuTOiZ8+pxYr4FIxrJmTRW8ia8ePHKLFej0xes/8lStXIxNju/rdYxx2J5vfNlNHYc3Vp5k86M2m9gUbfsy0Q8/CdCL36bICWtCIlTR2nznZGuvTnyUGXsdiS0zPVh6pAuBkIpgk3moF9+ksyIRvmPvLHh8FgCVom+tw+8KKWT7ssErCGHIfPnbBpQExJoPjEaxkiZ33zlDknGHG2e3L6gUsyn9G8GSzePkFYg== Received: from BL1PR13CA0270.namprd13.prod.outlook.com (2603:10b6:208:2ba::35) by PH7PR12MB6563.namprd12.prod.outlook.com (2603:10b6:510:211::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7270.17; Mon, 5 Feb 2024 12:49:15 +0000 Received: from MN1PEPF0000F0E0.namprd04.prod.outlook.com (2603:10b6:208:2ba:cafe::c3) by BL1PR13CA0270.outlook.office365.com (2603:10b6:208:2ba::35) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7270.14 via Frontend Transport; Mon, 5 Feb 2024 12:49:14 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.161) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.161 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.161; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.161) by MN1PEPF0000F0E0.mail.protection.outlook.com (10.167.242.38) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7249.19 via Frontend Transport; Mon, 5 Feb 2024 12:49:14 +0000 Received: from rnnvmail204.nvidia.com (10.129.68.6) by mail.nvidia.com (10.129.200.67) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Mon, 5 Feb 2024 04:49:01 -0800 Received: from rnnvmail202.nvidia.com (10.129.68.7) by rnnvmail204.nvidia.com (10.129.68.6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Mon, 5 Feb 2024 04:49:01 -0800 Received: from vdi.nvidia.com (10.127.8.10) by mail.nvidia.com (10.129.68.7) with Microsoft SMTP Server id 15.2.1258.12 via Frontend Transport; Mon, 5 Feb 2024 04:48:59 -0800 From: Yishai Hadas To: , CC: , , , , , Subject: [PATCH V1 vfio 5/5] vfio/mlx5: Let firmware knows upon leaving PRE_COPY back to RUNNING Date: Mon, 5 Feb 2024 14:48:28 +0200 Message-ID: <20240205124828.232701-6-yishaih@nvidia.com> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20240205124828.232701-1-yishaih@nvidia.com> References: <20240205124828.232701-1-yishaih@nvidia.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: MN1PEPF0000F0E0:EE_|PH7PR12MB6563:EE_ X-MS-Office365-Filtering-Correlation-Id: fc21b9fa-e1db-41e6-248c-08dc2648dbdc X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: k6RoMm3sF8BAlVpbqmf32VHBA0np5nXVnFM8pQ0NYWpVHNBkXD9xP5GKjUFsVxMf022HdHZLOzQHHQpYJZJu5nHfIWaPLid6lDzx/HO8BTgsAwUsI8gXs9pJ9GPsk1GoG8qAJ+1s/T2NMinnaMVJd2nv8gVrFk/3mklYsdin0wrmfPIwhUAf9u4vdCAPSCe8k8f0PKIM8Aq/veqZA/6AbpvpqoGYdDr3mMznfKADPAeJmyYhinNPJxmVIuKcQbg4jIifSguCtab9CgP6zD32KfnebD2LsM2gmqVcO7M/B9rZf/rMoAnj7N9MAPFO47IKmJyWguM4oMCM3kUI4ki0x/aYHfix35thrZ2S6zFyl8nGJvAzsxEPfKx7lCLr7V0F5KJ5yn0FcJGP5mwyKrLJAVgji3yqYRYOs5WBNnMAqQRd+5ISeCaTEi+Shwc8W+EqmbrkobDKTV/9LOXDz4eiyIjlXTK5aqgc6aXLFwMwz45XItn/QqAiopS09J5YyhX0YSXsdHXdJbD4RAUZQ8u2oQuoJs9L8jJBUhmR2laM1L2JARP8cmqExUgvfTgVxrKpfTBYetiJLP6LDZQtCVNECZiy725uX0kodvgK4XmgaMODSKugy2RYF8OSzWvMrwYUrY1fBA3t8pCSxQF5nAnPjiOtzYN2LSF7hjCe/bvbPrEMaTmPYMmQWhRIm9mypSvs4yE0piHPaeH2oLC6uXEuWrGDE4zXJx3z9ilOcosRVUAa5C7mCKatMTEYwRZ/k5C/ X-Forefront-Antispam-Report: CIP:216.228.117.161;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc6edge2.nvidia.com;CAT:NONE;SFS:(13230031)(4636009)(396003)(136003)(346002)(39860400002)(376002)(230922051799003)(1800799012)(64100799003)(82310400011)(451199024)(186009)(40470700004)(36840700001)(46966006)(40460700003)(40480700001)(41300700001)(47076005)(36860700001)(36756003)(83380400001)(7696005)(6666004)(107886003)(336012)(1076003)(426003)(26005)(2906002)(356005)(82740400003)(7636003)(2616005)(70586007)(6636002)(316002)(70206006)(54906003)(478600001)(110136005)(86362001)(8676002)(4326008)(8936002)(5660300002);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 Feb 2024 12:49:14.5874 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: fc21b9fa-e1db-41e6-248c-08dc2648dbdc X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.117.161];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: MN1PEPF0000F0E0.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR12MB6563 Let firmware knows upon leaving PRE_COPY back to RUNNING as of some error in the target/migration cancellation. This will let firmware cleaning its internal resources that were turned on upon PRE_COPY. The flow is based on the device specification in this area. Signed-off-by: Yishai Hadas --- drivers/vfio/pci/mlx5/cmd.c | 14 +++++++++---- drivers/vfio/pci/mlx5/cmd.h | 4 +++- drivers/vfio/pci/mlx5/main.c | 39 +++++++++++++++++++++++++++++------- 3 files changed, 45 insertions(+), 12 deletions(-) diff --git a/drivers/vfio/pci/mlx5/cmd.c b/drivers/vfio/pci/mlx5/cmd.c index 6800e4ffe9ee..c54bcd5d0917 100644 --- a/drivers/vfio/pci/mlx5/cmd.c +++ b/drivers/vfio/pci/mlx5/cmd.c @@ -108,8 +108,9 @@ int mlx5vf_cmd_query_vhca_migration_state(struct mlx5vf_pci_core_device *mvdev, ret = wait_for_completion_interruptible(&mvdev->saving_migf->save_comp); if (ret) return ret; - if (mvdev->saving_migf->state == - MLX5_MIGF_STATE_PRE_COPY_ERROR) { + /* Upon cleanup, ignore previous pre_copy error state */ + if (mvdev->saving_migf->state == MLX5_MIGF_STATE_PRE_COPY_ERROR && + !(query_flags & MLX5VF_QUERY_CLEANUP)) { /* * In case we had a PRE_COPY error, only query full * image for final image @@ -200,7 +201,7 @@ void mlx5vf_cmd_close_migratable(struct mlx5vf_pci_core_device *mvdev) /* Must be done outside the lock to let it progress */ set_tracker_error(mvdev); mutex_lock(&mvdev->state_mutex); - mlx5vf_disable_fds(mvdev); + mlx5vf_disable_fds(mvdev, NULL); _mlx5vf_free_page_tracker_resources(mvdev); mlx5vf_state_mutex_unlock(mvdev); } @@ -639,6 +640,7 @@ int mlx5vf_cmd_save_vhca_state(struct mlx5vf_pci_core_device *mvdev, u32 in[MLX5_ST_SZ_DW(save_vhca_state_in)] = {}; struct mlx5_vhca_data_buffer *header_buf = NULL; struct mlx5vf_async_data *async_data; + bool pre_copy_cleanup = false; int err; lockdep_assert_held(&mvdev->state_mutex); @@ -649,6 +651,10 @@ int mlx5vf_cmd_save_vhca_state(struct mlx5vf_pci_core_device *mvdev, if (err) return err; + if ((migf->state == MLX5_MIGF_STATE_PRE_COPY || + migf->state == MLX5_MIGF_STATE_PRE_COPY_ERROR) && !track && !inc) + pre_copy_cleanup = true; + if (migf->state == MLX5_MIGF_STATE_PRE_COPY_ERROR) /* * In case we had a PRE_COPY error, SAVE is triggered only for @@ -667,7 +673,7 @@ int mlx5vf_cmd_save_vhca_state(struct mlx5vf_pci_core_device *mvdev, async_data = &migf->async_data; async_data->buf = buf; - async_data->stop_copy_chunk = !track; + async_data->stop_copy_chunk = (!track && !pre_copy_cleanup); async_data->out = kvzalloc(out_size, GFP_KERNEL); if (!async_data->out) { err = -ENOMEM; diff --git a/drivers/vfio/pci/mlx5/cmd.h b/drivers/vfio/pci/mlx5/cmd.h index 0d6a2db3d801..707393df36c4 100644 --- a/drivers/vfio/pci/mlx5/cmd.h +++ b/drivers/vfio/pci/mlx5/cmd.h @@ -197,6 +197,7 @@ struct mlx5vf_pci_core_device { enum { MLX5VF_QUERY_INC = (1UL << 0), MLX5VF_QUERY_FINAL = (1UL << 1), + MLX5VF_QUERY_CLEANUP = (1UL << 2), }; int mlx5vf_cmd_suspend_vhca(struct mlx5vf_pci_core_device *mvdev, u16 op_mod); @@ -232,7 +233,8 @@ int mlx5vf_add_migration_pages(struct mlx5_vhca_data_buffer *buf, struct page *mlx5vf_get_migration_page(struct mlx5_vhca_data_buffer *buf, unsigned long offset); void mlx5vf_state_mutex_unlock(struct mlx5vf_pci_core_device *mvdev); -void mlx5vf_disable_fds(struct mlx5vf_pci_core_device *mvdev); +void mlx5vf_disable_fds(struct mlx5vf_pci_core_device *mvdev, + enum mlx5_vf_migf_state *last_save_state); void mlx5vf_mig_file_cleanup_cb(struct work_struct *_work); void mlx5vf_mig_file_set_save_work(struct mlx5_vf_migration_file *migf, u8 chunk_num, size_t next_required_umem_size); diff --git a/drivers/vfio/pci/mlx5/main.c b/drivers/vfio/pci/mlx5/main.c index fe09a8c8af95..3982fcf60cf2 100644 --- a/drivers/vfio/pci/mlx5/main.c +++ b/drivers/vfio/pci/mlx5/main.c @@ -1146,7 +1146,8 @@ mlx5vf_pci_resume_device_data(struct mlx5vf_pci_core_device *mvdev) return ERR_PTR(ret); } -void mlx5vf_disable_fds(struct mlx5vf_pci_core_device *mvdev) +void mlx5vf_disable_fds(struct mlx5vf_pci_core_device *mvdev, + enum mlx5_vf_migf_state *last_save_state) { if (mvdev->resuming_migf) { mlx5vf_disable_fd(mvdev->resuming_migf); @@ -1157,6 +1158,8 @@ void mlx5vf_disable_fds(struct mlx5vf_pci_core_device *mvdev) if (mvdev->saving_migf) { mlx5_cmd_cleanup_async_ctx(&mvdev->saving_migf->async_ctx); cancel_work_sync(&mvdev->saving_migf->async_data.work); + if (last_save_state) + *last_save_state = mvdev->saving_migf->state; mlx5vf_disable_fd(mvdev->saving_migf); wake_up_interruptible(&mvdev->saving_migf->poll_wait); mlx5fv_cmd_clean_migf_resources(mvdev->saving_migf); @@ -1217,12 +1220,34 @@ mlx5vf_pci_step_device_state_locked(struct mlx5vf_pci_core_device *mvdev, return migf->filp; } - if ((cur == VFIO_DEVICE_STATE_STOP_COPY && new == VFIO_DEVICE_STATE_STOP) || - (cur == VFIO_DEVICE_STATE_PRE_COPY && new == VFIO_DEVICE_STATE_RUNNING) || + if (cur == VFIO_DEVICE_STATE_STOP_COPY && new == VFIO_DEVICE_STATE_STOP) { + mlx5vf_disable_fds(mvdev, NULL); + return NULL; + } + + if ((cur == VFIO_DEVICE_STATE_PRE_COPY && new == VFIO_DEVICE_STATE_RUNNING) || (cur == VFIO_DEVICE_STATE_PRE_COPY_P2P && new == VFIO_DEVICE_STATE_RUNNING_P2P)) { - mlx5vf_disable_fds(mvdev); - return NULL; + struct mlx5_vf_migration_file *migf = mvdev->saving_migf; + struct mlx5_vhca_data_buffer *buf; + enum mlx5_vf_migf_state state; + size_t size; + + ret = mlx5vf_cmd_query_vhca_migration_state(mvdev, &size, NULL, + MLX5VF_QUERY_INC | MLX5VF_QUERY_CLEANUP); + if (ret) + return ERR_PTR(ret); + buf = mlx5vf_get_data_buffer(migf, size, DMA_FROM_DEVICE); + if (IS_ERR(buf)) + return ERR_CAST(buf); + /* pre_copy cleanup */ + ret = mlx5vf_cmd_save_vhca_state(mvdev, migf, buf, false, false); + if (ret) { + mlx5vf_put_data_buffer(buf); + return ERR_PTR(ret); + } + mlx5vf_disable_fds(mvdev, &state); + return (state != MLX5_MIGF_STATE_ERROR) ? NULL : ERR_PTR(-EIO); } if (cur == VFIO_DEVICE_STATE_STOP && new == VFIO_DEVICE_STATE_RESUMING) { @@ -1244,7 +1269,7 @@ mlx5vf_pci_step_device_state_locked(struct mlx5vf_pci_core_device *mvdev, if (ret) return ERR_PTR(ret); } - mlx5vf_disable_fds(mvdev); + mlx5vf_disable_fds(mvdev, NULL); return NULL; } @@ -1289,7 +1314,7 @@ void mlx5vf_state_mutex_unlock(struct mlx5vf_pci_core_device *mvdev) mvdev->deferred_reset = false; spin_unlock(&mvdev->reset_lock); mvdev->mig_state = VFIO_DEVICE_STATE_RUNNING; - mlx5vf_disable_fds(mvdev); + mlx5vf_disable_fds(mvdev, NULL); goto again; } mutex_unlock(&mvdev->state_mutex);