From patchwork Mon Jun 6 08:56:18 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yishai Hadas X-Patchwork-Id: 12870171 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2C1F9C43334 for ; Mon, 6 Jun 2022 08:57:34 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232307AbiFFI5c (ORCPT ); Mon, 6 Jun 2022 04:57:32 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36832 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232266AbiFFI50 (ORCPT ); Mon, 6 Jun 2022 04:57:26 -0400 Received: from NAM12-BN8-obe.outbound.protection.outlook.com (mail-bn8nam12on2084.outbound.protection.outlook.com [40.107.237.84]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 240B91835C for ; Mon, 6 Jun 2022 01:57:21 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=TD1mk5zz4ANb6hG3yJyqJPlqTUjYQQAmTqmw/ePxTaCONacmxCXdCUZaq3J9QezVyffQKkMI0PXZ6+7GcuFeEunbZDck7J73krNCQzl5Axz10aHDY2zlFCWlgo6ioTSRTl9dzq0rSCY9/m96xKBUijirkP6gQITlQUm82j353XL5YDeV0lrW4taqPpa84RiLQHfB3mD5UvSxAjMOzIwm7qibK/ZwiM4aeyjzZAbLxkA5OlIct1+xcb/EFGM/PVOQUG3GVUevRrzlO8GzeP6W6OugAOptBGNXlW8m+GAo+38otCPNmHpacP6tUALZBN/PXB+5z14J9rO+nnJIiL48IA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=WB0vsfgtEPHIw+staE4Tb7BYwc1hahOx2vwkEWEam0A=; b=ahUzLhrt193NIxDXsDlyUNR33XrhR7dUWCnks5b3qEnfHS228uSVMz62/B+RpAFs2ThQS98J14ZgRSC2nfmjMpEp/K9BbPQSt9zxiZIxbOhtLIJiGz8MlEfMUXt3f4oY8aYJXI7PwIaEBwruv/0larQO7dVcX0r2g7uBL0AH5dckqB9GHQfr+1q2FhddoYTfZEgma8lX80secm1MHfCU9QOfVWp4/PKCzioE9NHV2rOyih/vTy6wRA0hM6JXDbclhiaGFq7YBprWO+F0GmefV73JDEjUCTdMhcQjdSWVttgtOvppUbKyQnwvrnrpIbRTGVNlhvLXM/dzRGen44bObg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 12.22.5.238) smtp.rcpttodomain=intel.com smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=WB0vsfgtEPHIw+staE4Tb7BYwc1hahOx2vwkEWEam0A=; b=sjIJchFTlv8L/kIzIINER3zLD3EqYoVOTQ3mdaMQXejFJz2Y1/KjYnHkk5VlHpzQpkLfYDKAQYGS78symcMVw2izWxwDINW9gTHiv0TtnKo1llYceRn4vQVYQZ5YZ8hJ7Vh28ynziO+A7T6iyaH41L6lQJ2RGen4EIFb+ZxCeJTDf24yunazBEWr9ZqB1/VTq+qZlQ9//omi+7/cuXw1Qjiq/a9kmLB2uq0n6P0EUQK66CHKJRuaJjBwDjKPZz9oCAeHEMSpqJxcBHkmneqwVJLF9OhxC+IsY2T29uBxNufJvQ5RwnMw675H0xFqvY4NbFpGnljZoQIAiGH8beS0+w== Received: from BN9PR03CA0035.namprd03.prod.outlook.com (2603:10b6:408:fb::10) by MWHPR12MB1838.namprd12.prod.outlook.com (2603:10b6:300:106::23) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5314.13; Mon, 6 Jun 2022 08:57:18 +0000 Received: from BN8NAM11FT051.eop-nam11.prod.protection.outlook.com (2603:10b6:408:fb:cafe::f4) by BN9PR03CA0035.outlook.office365.com (2603:10b6:408:fb::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5314.15 via Frontend Transport; Mon, 6 Jun 2022 08:57:18 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 12.22.5.238) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 12.22.5.238 as permitted sender) receiver=protection.outlook.com; client-ip=12.22.5.238; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (12.22.5.238) by BN8NAM11FT051.mail.protection.outlook.com (10.13.177.66) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.5314.12 via Frontend Transport; Mon, 6 Jun 2022 08:57:17 +0000 Received: from rnnvmail202.nvidia.com (10.129.68.7) by DRHQMAIL105.nvidia.com (10.27.9.14) with Microsoft SMTP Server (TLS) id 15.0.1497.32; Mon, 6 Jun 2022 08:57:17 +0000 Received: from rnnvmail204.nvidia.com (10.129.68.6) by rnnvmail202.nvidia.com (10.129.68.7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.22; Mon, 6 Jun 2022 01:57:16 -0700 Received: from vdi.nvidia.com (10.127.8.10) by mail.nvidia.com (10.129.68.6) with Microsoft SMTP Server id 15.2.986.22 via Frontend Transport; Mon, 6 Jun 2022 01:57:13 -0700 From: Yishai Hadas To: , , CC: , , , , , Subject: [PATCH vfio 1/2] vfio/mlx5: Protect mlx5vf_disable_fds() upon close device Date: Mon, 6 Jun 2022 11:56:18 +0300 Message-ID: <20220606085619.7757-2-yishaih@nvidia.com> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20220606085619.7757-1-yishaih@nvidia.com> References: <20220606085619.7757-1-yishaih@nvidia.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: e16caa92-c5a6-4a6b-cb32-08da479a8f45 X-MS-TrafficTypeDiagnostic: MWHPR12MB1838:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: T1cL41+ZQ7Ox2Vt3/josaUbCOqqliyb7B2v13ymb5s8NWKQaCfzz/1tQeVaYXLgV5LAL9ZVIOL1ePe54CNuoJHY/OhcU2ol9ynw3Ny/TIoOaxjYmh/G01j+yXUKWSY6R8NS2wG5lZ4hM+b1BKHxsQcENhtZhegDfJWa/EYy7jvrHQw82laqtQ3RXNF9bYSnkBOhl7TRY2o1iwmdjAKKCAfeoD9wiAFriLZxXNiCYvESynh825KgOsZrDXp6Hm0zQ/4frnKjSWoAXIBM6ijfUF80khTpw3TT46VGPYoA/GddvCTpffPQepHUkJXMz2beDLaya/VwdPuXETmM2waqugKHZl6dcy34/EFQ5zwSuf1Q6Vfqfb1Bf+Ntw9gpAhvShCIgJqpSELiMCzr5XcZ3nYfO/7y8yg+hMjWfWDEZKWpgJzHEL67tMo0uwRY6ZlOwkZCXotqfnCne8iZdE9Fto4OzS1pEVMssosFiNCECTLQFa7fRSemGf7vv102IrCemoqidwTpw9akIL9WWpOsVhEvTNn27Df5rVa7gc0YZMtX8ylVlTUt4sj3PX2nXu+NtNnLEcH913KJbSzyQvFlqHZvryf/wACYEj0RGd2HvMeUl1S2QZVqFbrfLHYGDgD50Mu42M4ZSqsRA6bvxH+AAW9L1ge7v+R4EYfY/xWdjp+Ot6dgYVm48ytimowieLMcim6jmjKzbVnDxNXUp2B5nf2w== X-Forefront-Antispam-Report: CIP:12.22.5.238;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:InfoNoRecords;CAT:NONE;SFS:(13230001)(4636009)(40470700004)(46966006)(36840700001)(5660300002)(356005)(40460700003)(8936002)(81166007)(86362001)(2906002)(508600001)(110136005)(336012)(426003)(186003)(83380400001)(36756003)(54906003)(47076005)(316002)(70206006)(36860700001)(4326008)(82310400005)(26005)(7696005)(70586007)(8676002)(2616005)(1076003)(36900700001);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Jun 2022 08:57:17.8879 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: e16caa92-c5a6-4a6b-cb32-08da479a8f45 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[12.22.5.238];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BN8NAM11FT051.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: MWHPR12MB1838 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Protect mlx5vf_disable_fds() upon close device to be called under the state mutex as done in all other places. This will prevent a race with any other flow which calls mlx5vf_disable_fds() as of health/recovery upon MLX5_PF_NOTIFY_DISABLE_VF event. Encapsulate this functionality in a separate function named mlx5vf_cmd_close_migratable() to consider migration caps and for further usage upon close device. Fixes: 6fadb021266d ("vfio/mlx5: Implement vfio_pci driver for mlx5 devices") Signed-off-by: Yishai Hadas Reviewed-by: Kevin Tian --- drivers/vfio/pci/mlx5/cmd.c | 10 ++++++++++ drivers/vfio/pci/mlx5/cmd.h | 1 + drivers/vfio/pci/mlx5/main.c | 2 +- 3 files changed, 12 insertions(+), 1 deletion(-) diff --git a/drivers/vfio/pci/mlx5/cmd.c b/drivers/vfio/pci/mlx5/cmd.c index 9b9f33ca270a..cdd0c667dc77 100644 --- a/drivers/vfio/pci/mlx5/cmd.c +++ b/drivers/vfio/pci/mlx5/cmd.c @@ -88,6 +88,16 @@ static int mlx5fv_vf_event(struct notifier_block *nb, return 0; } +void mlx5vf_cmd_close_migratable(struct mlx5vf_pci_core_device *mvdev) +{ + if (!mvdev->migrate_cap) + return; + + mutex_lock(&mvdev->state_mutex); + mlx5vf_disable_fds(mvdev); + mlx5vf_state_mutex_unlock(mvdev); +} + void mlx5vf_cmd_remove_migratable(struct mlx5vf_pci_core_device *mvdev) { if (!mvdev->migrate_cap) diff --git a/drivers/vfio/pci/mlx5/cmd.h b/drivers/vfio/pci/mlx5/cmd.h index 6c3112fdd8b1..aa692d9ce656 100644 --- a/drivers/vfio/pci/mlx5/cmd.h +++ b/drivers/vfio/pci/mlx5/cmd.h @@ -64,6 +64,7 @@ int mlx5vf_cmd_query_vhca_migration_state(struct mlx5vf_pci_core_device *mvdev, size_t *state_size); void mlx5vf_cmd_set_migratable(struct mlx5vf_pci_core_device *mvdev); void mlx5vf_cmd_remove_migratable(struct mlx5vf_pci_core_device *mvdev); +void mlx5vf_cmd_close_migratable(struct mlx5vf_pci_core_device *mvdev); int mlx5vf_cmd_save_vhca_state(struct mlx5vf_pci_core_device *mvdev, struct mlx5_vf_migration_file *migf); int mlx5vf_cmd_load_vhca_state(struct mlx5vf_pci_core_device *mvdev, diff --git a/drivers/vfio/pci/mlx5/main.c b/drivers/vfio/pci/mlx5/main.c index 0558d0649ddb..d754990f0662 100644 --- a/drivers/vfio/pci/mlx5/main.c +++ b/drivers/vfio/pci/mlx5/main.c @@ -570,7 +570,7 @@ static void mlx5vf_pci_close_device(struct vfio_device *core_vdev) struct mlx5vf_pci_core_device *mvdev = container_of( core_vdev, struct mlx5vf_pci_core_device, core_device.vdev); - mlx5vf_disable_fds(mvdev); + mlx5vf_cmd_close_migratable(mvdev); vfio_pci_core_close_device(core_vdev); } From patchwork Mon Jun 6 08:56:19 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yishai Hadas X-Patchwork-Id: 12870172 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C1B1FC433EF for ; Mon, 6 Jun 2022 08:57:35 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231836AbiFFI5e (ORCPT ); Mon, 6 Jun 2022 04:57:34 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36832 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232294AbiFFI5b (ORCPT ); Mon, 6 Jun 2022 04:57:31 -0400 Received: from NAM10-DM6-obe.outbound.protection.outlook.com (mail-dm6nam10on2063.outbound.protection.outlook.com [40.107.93.63]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8B5637CDE1 for ; Mon, 6 Jun 2022 01:57:23 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=aN7hdrIY2R2J8A97HVfll3IeYAtWvqfPmCf0GsMaMbd5uNkzMivALkC28jDcGue1nfaiaJoROpbMO69j674uDG4CvmWCv0ncuuPMff/m4DavaF5N57IuTFVtilc0g2DLpnKhHMYKj0TxBB9m0iV94R/pQ38NJLAjztrWCOZaTX+QNqGhRUMVd37d0oDtCi3WApG/GPt33Z+dk8elTXqRo2DeJS+kcYFxHRayCWw5QU+TiKbfK53X+E6vsyVUcbwGjcNwO6fq/A56wFQDnsowBDqKlaFHX5E4eiYMv2MEcdhea6PyLIv+XbVczFZVTGgtku8wWbA3Xv3nflkJWj9OrQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=IUetKQ8EMqhIOYYxY9SO9hQWp9I6WkKSeo+i7QoBZS8=; b=MBFAw6bI9zrebj4FI9k9mqXynWpd0UuzTY3Xv/ul7iQG8k9M7AOS4Fe3FBw8XFisc4M9y2mYlrOPwMPf/kk3kng4rKMRfaocB4nrpAGsyk3ZvMR4v8B5RU5v948O22mYGu0LqBLIDN+AJzltZ803TlYPPEKKbCF7mxD8Ww6bw1VrwlHN5dJd0Hga7hS2BcWyzxo5pe0siMcbg4W7EnX2e3AYJJ7gEw2sqLgp7ETYMU5PaSk+Vnmsh7evKQkjWJD77ouyGe9OZSo8pDNgrUqX4pCUf6uFBJPIyWXE9wy4ek3celGUiOHxCY9d5k4fliVGgnzGT3/MkThdPmkvqRpQhQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 12.22.5.234) smtp.rcpttodomain=intel.com smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=IUetKQ8EMqhIOYYxY9SO9hQWp9I6WkKSeo+i7QoBZS8=; b=SLbuBB++Gn4uIhna+3erkpsKLl4LtYORI2f9sQ6TI3O5lfm9+hNRKg3ECVGO9R3V1cWtIAAwyNZgNQm4MHLwY8JcuT1rxAEksVDvx70nL+nfF6ovFUwbXdyGjIApt7njVEnPVaRS8QVV8/uAvGW+hD/mebwuowIhTKoQpSja06KvKwus12bgiU2mjruKaDKSOgKPkutjYXcxjf1qxl87Mp+gm9c54rXJ8RDg9v9zk4ZeY4gYHlLxx07ReWKv17BSqYgJienFng+otNWK9O0MRfw+7PUVD2jdVPTicEPeBW2yo+OrJ0zY+qCy2h/Ytziq64p/GSXfn35lXfbE+fWDmg== Received: from BN9PR03CA0139.namprd03.prod.outlook.com (2603:10b6:408:fe::24) by DM5PR12MB1466.namprd12.prod.outlook.com (2603:10b6:4:d::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5314.15; Mon, 6 Jun 2022 08:57:21 +0000 Received: from BN8NAM11FT016.eop-nam11.prod.protection.outlook.com (2603:10b6:408:fe:cafe::a9) by BN9PR03CA0139.outlook.office365.com (2603:10b6:408:fe::24) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5314.13 via Frontend Transport; Mon, 6 Jun 2022 08:57:21 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 12.22.5.234) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 12.22.5.234 as permitted sender) receiver=protection.outlook.com; client-ip=12.22.5.234; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (12.22.5.234) by BN8NAM11FT016.mail.protection.outlook.com (10.13.176.97) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.5314.12 via Frontend Transport; Mon, 6 Jun 2022 08:57:21 +0000 Received: from rnnvmail204.nvidia.com (10.129.68.6) by DRHQMAIL101.nvidia.com (10.27.9.10) with Microsoft SMTP Server (TLS) id 15.0.1497.32; Mon, 6 Jun 2022 08:57:20 +0000 Received: from rnnvmail204.nvidia.com (10.129.68.6) by rnnvmail204.nvidia.com (10.129.68.6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.22; Mon, 6 Jun 2022 01:57:19 -0700 Received: from vdi.nvidia.com (10.127.8.10) by mail.nvidia.com (10.129.68.6) with Microsoft SMTP Server id 15.2.986.22 via Frontend Transport; Mon, 6 Jun 2022 01:57:17 -0700 From: Yishai Hadas To: , , CC: , , , , , Subject: [PATCH vfio 2/2] vfio: Split migration ops from main device ops Date: Mon, 6 Jun 2022 11:56:19 +0300 Message-ID: <20220606085619.7757-3-yishaih@nvidia.com> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20220606085619.7757-1-yishaih@nvidia.com> References: <20220606085619.7757-1-yishaih@nvidia.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 0741620e-ab54-4370-46d1-08da479a912e X-MS-TrafficTypeDiagnostic: DM5PR12MB1466:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: xS40hYuARbo2VAeS2FmFw7B9R0mItihw7EjT3kQsdMzXscad7/hzV56p/pz9Xpkap12b+08cg+NoNTcZ77Muhg2WNYKLFPzafXm2fX25wsXL34YmH2o4ujBWWUS+VNXVG9J5CXaIrZ9h746BN4BTEN+DRSIfzGSqJGWLPKshHSxkEmlGSFgNzawM1wjrqqpCjeVG8D2ts1oLRnwndRKyBbEvdvM5XFQt4OzvPyc6M0BWfebjUqHYONhwnrCitlPOQU4OVeCUYCT6tZv2B7Iz9hQn8Dxc6kMxfpafaRg/H1oynxS9cbvI4yyZ3TE/UaRlD688CtysATYHIWc4yRtzDf2OiXKdeL6+dXVyoBpKiS3gbD1FqfdZLUnPfnMfPAXMPGxtWy37LSaDHMw/7lSr7mp+qFzDUHcNW8JIVZEBWNU+bsuJSnz+IiE9x+f8so7B8uWk85Rb66SvQRw+UxNlLdDA6PBSMEsGOb8IelpabdE2e7sD5fc7NrzTxke4l+YJ5sRs77TuhYlVXOftLvq6X4FMredE3kPeA9+D4HvDqDCkZzdKWjq3fMatVajTsdlMXHpqh9Vnfmr5VdCujU+Awu5QsHDBqOT3hiXC1pVUMANaTx+fchRohjD5YbDHFR/sL8ftV0TunJQh41ix32JLZZjGY+cBY12YZQvJDObh//ZBXLHx1fOB4pnLxP/hUH/LETCn7XGOKnkgQLMWzoh20A== X-Forefront-Antispam-Report: CIP:12.22.5.234;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:InfoNoRecords;CAT:NONE;SFS:(13230001)(4636009)(36840700001)(46966006)(40470700004)(36756003)(356005)(81166007)(40460700003)(110136005)(316002)(54906003)(2616005)(70586007)(70206006)(26005)(8936002)(82310400005)(86362001)(5660300002)(8676002)(7696005)(4326008)(186003)(426003)(1076003)(36860700001)(47076005)(508600001)(83380400001)(2906002)(336012)(36900700001);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Jun 2022 08:57:21.0111 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 0741620e-ab54-4370-46d1-08da479a912e X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[12.22.5.234];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BN8NAM11FT016.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM5PR12MB1466 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org vfio core checks whether the driver sets some migration op (e.g. set_state/get_state) and accordingly calls its op. However, currently mlx5 driver sets the above ops without regards to its migration caps. This might lead to unexpected usage/Oops if user space may call to the above ops even if the driver doesn't support migration. As for example, the migration state_mutex is not initialized in that case. The cleanest way to manage that seems to split the migration ops from the main device ops, this will let the driver setting them separately from the main ops when it's applicable. As part of that, changed HISI driver to match this scheme. This scheme may enable down the road to come with some extra group of ops (e.g. DMA log) that can be set without regards to the other options based on driver caps. Fixes: 6fadb021266d ("vfio/mlx5: Implement vfio_pci driver for mlx5 devices") Signed-off-by: Yishai Hadas Reviewed-by: Kevin Tian , with one nit: --- .../vfio/pci/hisilicon/hisi_acc_vfio_pci.c | 11 +++++--- drivers/vfio/pci/mlx5/cmd.c | 4 ++- drivers/vfio/pci/mlx5/cmd.h | 3 ++- drivers/vfio/pci/mlx5/main.c | 9 ++++--- drivers/vfio/vfio.c | 13 +++++----- include/linux/vfio.h | 26 ++++++++++++------- 6 files changed, 42 insertions(+), 24 deletions(-) diff --git a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c index 4def43f5f7b6..ea762e28c1cc 100644 --- a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c +++ b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c @@ -1185,7 +1185,7 @@ static int hisi_acc_vfio_pci_open_device(struct vfio_device *core_vdev) if (ret) return ret; - if (core_vdev->ops->migration_set_state) { + if (core_vdev->mig_ops) { ret = hisi_acc_vf_qm_init(hisi_acc_vdev); if (ret) { vfio_pci_core_disable(vdev); @@ -1208,6 +1208,11 @@ static void hisi_acc_vfio_pci_close_device(struct vfio_device *core_vdev) vfio_pci_core_close_device(core_vdev); } +static const struct vfio_migration_ops hisi_acc_vfio_pci_migrn_state_ops = { + .migration_set_state = hisi_acc_vfio_pci_set_device_state, + .migration_get_state = hisi_acc_vfio_pci_get_device_state, +}; + static const struct vfio_device_ops hisi_acc_vfio_pci_migrn_ops = { .name = "hisi-acc-vfio-pci-migration", .open_device = hisi_acc_vfio_pci_open_device, @@ -1219,8 +1224,6 @@ static const struct vfio_device_ops hisi_acc_vfio_pci_migrn_ops = { .mmap = hisi_acc_vfio_pci_mmap, .request = vfio_pci_core_request, .match = vfio_pci_core_match, - .migration_set_state = hisi_acc_vfio_pci_set_device_state, - .migration_get_state = hisi_acc_vfio_pci_get_device_state, }; static const struct vfio_device_ops hisi_acc_vfio_pci_ops = { @@ -1272,6 +1275,8 @@ static int hisi_acc_vfio_pci_probe(struct pci_dev *pdev, const struct pci_device if (!ret) { vfio_pci_core_init_device(&hisi_acc_vdev->core_device, pdev, &hisi_acc_vfio_pci_migrn_ops); + hisi_acc_vdev->core_device.vdev.mig_ops = + &hisi_acc_vfio_pci_migrn_state_ops; } else { pci_warn(pdev, "migration support failed, continue with generic interface\n"); vfio_pci_core_init_device(&hisi_acc_vdev->core_device, pdev, diff --git a/drivers/vfio/pci/mlx5/cmd.c b/drivers/vfio/pci/mlx5/cmd.c index cdd0c667dc77..dd5d7bfe0a49 100644 --- a/drivers/vfio/pci/mlx5/cmd.c +++ b/drivers/vfio/pci/mlx5/cmd.c @@ -108,7 +108,8 @@ void mlx5vf_cmd_remove_migratable(struct mlx5vf_pci_core_device *mvdev) destroy_workqueue(mvdev->cb_wq); } -void mlx5vf_cmd_set_migratable(struct mlx5vf_pci_core_device *mvdev) +void mlx5vf_cmd_set_migratable(struct mlx5vf_pci_core_device *mvdev, + const struct vfio_migration_ops *mig_ops) { struct pci_dev *pdev = mvdev->core_device.pdev; int ret; @@ -149,6 +150,7 @@ void mlx5vf_cmd_set_migratable(struct mlx5vf_pci_core_device *mvdev) mvdev->core_device.vdev.migration_flags = VFIO_MIGRATION_STOP_COPY | VFIO_MIGRATION_P2P; + mvdev->core_device.vdev.mig_ops = mig_ops; end: mlx5_vf_put_core_dev(mvdev->mdev); diff --git a/drivers/vfio/pci/mlx5/cmd.h b/drivers/vfio/pci/mlx5/cmd.h index aa692d9ce656..8208f4701a90 100644 --- a/drivers/vfio/pci/mlx5/cmd.h +++ b/drivers/vfio/pci/mlx5/cmd.h @@ -62,7 +62,8 @@ int mlx5vf_cmd_suspend_vhca(struct mlx5vf_pci_core_device *mvdev, u16 op_mod); int mlx5vf_cmd_resume_vhca(struct mlx5vf_pci_core_device *mvdev, u16 op_mod); int mlx5vf_cmd_query_vhca_migration_state(struct mlx5vf_pci_core_device *mvdev, size_t *state_size); -void mlx5vf_cmd_set_migratable(struct mlx5vf_pci_core_device *mvdev); +void mlx5vf_cmd_set_migratable(struct mlx5vf_pci_core_device *mvdev, + const struct vfio_migration_ops *mig_ops); void mlx5vf_cmd_remove_migratable(struct mlx5vf_pci_core_device *mvdev); void mlx5vf_cmd_close_migratable(struct mlx5vf_pci_core_device *mvdev); int mlx5vf_cmd_save_vhca_state(struct mlx5vf_pci_core_device *mvdev, diff --git a/drivers/vfio/pci/mlx5/main.c b/drivers/vfio/pci/mlx5/main.c index d754990f0662..a9b63d15c5d3 100644 --- a/drivers/vfio/pci/mlx5/main.c +++ b/drivers/vfio/pci/mlx5/main.c @@ -574,6 +574,11 @@ static void mlx5vf_pci_close_device(struct vfio_device *core_vdev) vfio_pci_core_close_device(core_vdev); } +static const struct vfio_migration_ops mlx5vf_pci_mig_ops = { + .migration_set_state = mlx5vf_pci_set_device_state, + .migration_get_state = mlx5vf_pci_get_device_state, +}; + static const struct vfio_device_ops mlx5vf_pci_ops = { .name = "mlx5-vfio-pci", .open_device = mlx5vf_pci_open_device, @@ -585,8 +590,6 @@ static const struct vfio_device_ops mlx5vf_pci_ops = { .mmap = vfio_pci_core_mmap, .request = vfio_pci_core_request, .match = vfio_pci_core_match, - .migration_set_state = mlx5vf_pci_set_device_state, - .migration_get_state = mlx5vf_pci_get_device_state, }; static int mlx5vf_pci_probe(struct pci_dev *pdev, @@ -599,7 +602,7 @@ static int mlx5vf_pci_probe(struct pci_dev *pdev, if (!mvdev) return -ENOMEM; vfio_pci_core_init_device(&mvdev->core_device, pdev, &mlx5vf_pci_ops); - mlx5vf_cmd_set_migratable(mvdev); + mlx5vf_cmd_set_migratable(mvdev, &mlx5vf_pci_mig_ops); dev_set_drvdata(&pdev->dev, &mvdev->core_device); ret = vfio_pci_core_register_device(&mvdev->core_device); if (ret) diff --git a/drivers/vfio/vfio.c b/drivers/vfio/vfio.c index e22be13e6771..453dc2464969 100644 --- a/drivers/vfio/vfio.c +++ b/drivers/vfio/vfio.c @@ -1534,8 +1534,8 @@ vfio_ioctl_device_feature_mig_device_state(struct vfio_device *device, struct file *filp = NULL; int ret; - if (!device->ops->migration_set_state || - !device->ops->migration_get_state) + if (!device->mig_ops->migration_set_state || + !device->mig_ops->migration_get_state) return -ENOTTY; ret = vfio_check_feature(flags, argsz, @@ -1551,7 +1551,8 @@ vfio_ioctl_device_feature_mig_device_state(struct vfio_device *device, if (flags & VFIO_DEVICE_FEATURE_GET) { enum vfio_device_mig_state curr_state; - ret = device->ops->migration_get_state(device, &curr_state); + ret = device->mig_ops->migration_get_state(device, + &curr_state); if (ret) return ret; mig.device_state = curr_state; @@ -1559,7 +1560,7 @@ vfio_ioctl_device_feature_mig_device_state(struct vfio_device *device, } /* Handle the VFIO_DEVICE_FEATURE_SET */ - filp = device->ops->migration_set_state(device, mig.device_state); + filp = device->mig_ops->migration_set_state(device, mig.device_state); if (IS_ERR(filp) || !filp) goto out_copy; @@ -1582,8 +1583,8 @@ static int vfio_ioctl_device_feature_migration(struct vfio_device *device, }; int ret; - if (!device->ops->migration_set_state || - !device->ops->migration_get_state) + if (!device->mig_ops->migration_set_state || + !device->mig_ops->migration_get_state) return -ENOTTY; ret = vfio_check_feature(flags, argsz, VFIO_DEVICE_FEATURE_GET, diff --git a/include/linux/vfio.h b/include/linux/vfio.h index aa888cc51757..0453ca970221 100644 --- a/include/linux/vfio.h +++ b/include/linux/vfio.h @@ -32,6 +32,7 @@ struct vfio_device_set { struct vfio_device { struct device *dev; const struct vfio_device_ops *ops; + const struct vfio_migration_ops *mig_ops; struct vfio_group *group; struct vfio_device_set *dev_set; struct list_head dev_set_list; @@ -61,16 +62,6 @@ struct vfio_device { * match, -errno for abort (ex. match with insufficient or incorrect * additional args) * @device_feature: Optional, fill in the VFIO_DEVICE_FEATURE ioctl - * @migration_set_state: Optional callback to change the migration state for - * devices that support migration. It's mandatory for - * VFIO_DEVICE_FEATURE_MIGRATION migration support. - * The returned FD is used for data transfer according to the FSM - * definition. The driver is responsible to ensure that FD reaches end - * of stream or error whenever the migration FSM leaves a data transfer - * state or before close_device() returns. - * @migration_get_state: Optional callback to get the migration state for - * devices that support migration. It's mandatory for - * VFIO_DEVICE_FEATURE_MIGRATION migration support. */ struct vfio_device_ops { char *name; @@ -87,6 +78,21 @@ struct vfio_device_ops { int (*match)(struct vfio_device *vdev, char *buf); int (*device_feature)(struct vfio_device *device, u32 flags, void __user *arg, size_t argsz); +}; + +/** + * @migration_set_state: Optional callback to change the migration state for + * devices that support migration. It's mandatory for + * VFIO_DEVICE_FEATURE_MIGRATION migration support. + * The returned FD is used for data transfer according to the FSM + * definition. The driver is responsible to ensure that FD reaches end + * of stream or error whenever the migration FSM leaves a data transfer + * state or before close_device() returns. + * @migration_get_state: Optional callback to get the migration state for + * devices that support migration. It's mandatory for + * VFIO_DEVICE_FEATURE_MIGRATION migration support. + */ +struct vfio_migration_ops { struct file *(*migration_set_state)( struct vfio_device *device, enum vfio_device_mig_state new_state);