From patchwork Fri Aug 30 10:58:32 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dragos Tatulea X-Patchwork-Id: 13784917 Received: from NAM12-BN8-obe.outbound.protection.outlook.com (mail-bn8nam12on2051.outbound.protection.outlook.com [40.107.237.51]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 05F1018FDB3; Fri, 30 Aug 2024 10:59:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.237.51 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725015562; cv=fail; b=NBn+pIXbCpqwZaYRRrgKuyZQmyRZkvaFnvhoedGm82VFPVLoumTyRKmFx3mSTwhnB72Db8/M6kp9OhaF0LFOHb0Ymxaxw7s4aZlXDV3KhRCKkM/jep4FRHj0OkfF6AlQJgwpB1fOB/pHkdlESsR+q/F4yZH2tlnBNfzPubJnPkA= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725015562; c=relaxed/simple; bh=JhYwu9b2jQ5taeRpJIv4YfqwMBmO2TEq/EcPU1sEI8c=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=kR+zFGXDqnHYebvU7f5amZpau/Mkw4qL4dGtQywZrAA9EvhM6ycQETU7lnjGZ331UUCkIJejs3U4jKNyeZyEDijxWboWYQfYll/Bs5K+2sVTJOO088SRpfTDIcZ5b4fAE6ChmzudpN/6rSLra6iH48LtdH/D5hUm6SOCGCE7qUM= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=M75mCLYL; arc=fail smtp.client-ip=40.107.237.51 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="M75mCLYL" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=FTMXIwanUIMFHDNwUj1wpgAHeG5T0X90v+l8k5Iox9Pf6eQIlDVhRg8Ip3U06PuYw6IqIDvdy04YILZJI/4yqPkmrScB5BhjrAW/4sqAosSwG/oQB3m8FDUmpSdsLGRVKeADisE2hSJ4NS15km8LKpkJhJfHJyvlPBKd/ygAhHdsOHHbBfNM4GMEOpZ2Rm99DKFkBFJzCkcTuK9LR5KjY+9sN+8M3dN6yB/b5Md3WFRhdzJL1ImT9JC+FM3sw7bt5PokR9efe096pguW6ZbZPQarFvTD84d3OCJtLryfAKLmuDyfUncSiWnbJ8L/o2RMHpaMSEaPKkXydmP4KqAdSQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=lI4VPs6YjSGudOBwsqJySD9gzp3FRPAJISfUi3TigDs=; b=iHC/XDhtoGkttSZqoPJtmubMCy/usxwoMmx4AKDOEd1G6BR17hC4M2aema3hMoHQtdVA1ofViZa3cIL4C1GCbledTO6ttTPxZPM9yz2sqwjxwj9R8nDh7riGfj/E1fV6Dcw1kmRQi7rvCUZVplSjczhSkw1cwSJBR9w4v64yGNcYegN0USxQ1OB+ArMrkRnMlvH/43ai7QpsskIWxB7KmOGREI4XqDuB61hReEuR6zByNhyHGbuxdkTKIcW5WwdyuUn0eQFxZlTHp52em6ZHOfvj5/zVnOum8JQzhn6djIjQt99m3NsKMgdAdTR5BZVGGoN1CezDVDR/cFSH7hQEcA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.161) smtp.rcpttodomain=redhat.com smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=lI4VPs6YjSGudOBwsqJySD9gzp3FRPAJISfUi3TigDs=; b=M75mCLYL954KjPROJegPoPSYD3I34+hfbAoZvjtfro8IIaZuhUd2kzlDwA7gK98PZgAvkOw8NH8+OxveXX0wMJwy6a/d1jXNAaeLBODo/SmwbYpTgmYXBG7vU2ERjKtI1hduH2AiGzC3yFlYCCuUsGkebSZiFkm9gAu82PeY752LfmAoBua1fJNFHodDJxeJtVxkMW7J7APs4Oa6gp8IJCMKWLqPVc/GMq30JMHfweS2mJDcNCxSdgsuhjqqHxpMDYmagmxDQHeaLHY+X0H2Kyq5Vj1uKlJz2Ezq6pr4LtmPxcq7DFhVn41pNUkLnDKP8Lcep3aKec7yPBzzTWL7jA== Received: from PH1PEPF000132F8.NAMP220.PROD.OUTLOOK.COM (2603:10b6:518:1::29) by PH8PR12MB8605.namprd12.prod.outlook.com (2603:10b6:510:1cc::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7897.28; Fri, 30 Aug 2024 10:59:15 +0000 Received: from CY4PEPF0000E9D6.namprd05.prod.outlook.com (2a01:111:f403:f912::5) by PH1PEPF000132F8.outlook.office365.com (2603:1036:903:47::3) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7875.21 via Frontend Transport; Fri, 30 Aug 2024 10:59:14 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.161) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.161 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.161; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.161) by CY4PEPF0000E9D6.mail.protection.outlook.com (10.167.241.69) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7918.13 via Frontend Transport; Fri, 30 Aug 2024 10:59:14 +0000 Received: from rnnvmail204.nvidia.com (10.129.68.6) by mail.nvidia.com (10.129.200.67) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Fri, 30 Aug 2024 03:59:03 -0700 Received: from rnnvmail204.nvidia.com (10.129.68.6) by rnnvmail204.nvidia.com (10.129.68.6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Fri, 30 Aug 2024 03:59:02 -0700 Received: from c-237-113-220-225.mtl.labs.mlnx (10.127.8.12) by mail.nvidia.com (10.129.68.6) with Microsoft SMTP Server id 15.2.1544.4 via Frontend Transport; Fri, 30 Aug 2024 03:58:59 -0700 From: Dragos Tatulea To: "Michael S . Tsirkin" , Jason Wang , Eugenio Perez Martin , Si-Wei Liu , , Gal Pressman CC: Dragos Tatulea , , , Parav Pandit , Xuan Zhuo , Cosmin Ratiu Subject: [PATCH vhost v2 1/7] vdpa/mlx5: Create direct MKEYs in parallel Date: Fri, 30 Aug 2024 13:58:32 +0300 Message-ID: <20240830105838.2666587-3-dtatulea@nvidia.com> X-Mailer: git-send-email 2.45.1 In-Reply-To: <20240830105838.2666587-2-dtatulea@nvidia.com> References: <20240830105838.2666587-2-dtatulea@nvidia.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CY4PEPF0000E9D6:EE_|PH8PR12MB8605:EE_ X-MS-Office365-Filtering-Correlation-Id: 470f05b3-dbdd-46fa-7a3c-08dcc8e2c983 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|36860700013|1800799024|376014|82310400026; X-Microsoft-Antispam-Message-Info: Jgan9NRarsWUntPqiRdx40hOyp9ea7e8V62zjYCsW9wexXExm41WVUU8KPmszIcX4gaF/H3zQjklDauYtx1GLT8Jb3Ue95nK5fUoH0gG/77Woujnraz2W8iPNQSzo1aoHsiNR184obzzvQ//fFwzm1saeuVZdvTpaDI/ZySEjxYLBEwID0jgmx3HjlPbiied8PO2jTOu9PB1QFAmdZo/C/nCRBTzNLCwmqp0Jqy5lEerWorQ11cP81UfbpVZXNdfckQzBpS0YksTccpUxuq5IBp/IX/kJRuTUXdwze7P95JHOmuKLzlGGKarTBgZCa0/WuSswmLRoCoK2emREq0whb2gkBRssEe8kYtKQML6ajlkKtTUi7D/5+yCjH8vGuEQDC2aqnqwBNpBWp7Z95d98S+mPTEWnP0N3/guT1kZDAfztxH7Dc83GAuT0tiCh9eLfgHHRPiK+PLIcGu/6QX6ysCePHHykQdyQKTM/oYX2c2q7rTd/A7CqpcNwBWyYpSTP9cyS3oiau2t8KS0RMzliErhr4MgatglbDDaJ7tWUH4egX5TpCYOO7SUMS/DryZgzQQ1NdRLv1RfOncSsv1GZ2eJiQ+wz8uq3+IyR2O6ZFkK9xbIfGEQISYh8fZUjK5fIcgupIeduq1BWlLW4lwnLoQlwo2sZJLEYfm/E3HCi4qrTcYTmV11txsO9WwVBAwEZ+6tTEBEZLPWF+2pbA3SO0fPf1srMeIwkyRC3A46g9Nuj3H/eOWkt2g4uhPB9PUB3bE3SJTsfNuDeKXn05SkJmdWhvO4r16p+X6leoVAqpl3XIZUFszX4Ur3ZgpXMJ/wHXc2lSnWvc+52C/ghaIYLAx3ggGPhFdiXWvxWeJoegcatv/z/ScqOdvLv6st7T0+ECoeBGF9qEJIiIJn+Q9bKcrvpCXvAmMCWsf0V93tUTrXT1vIIyqdO0OaGGlRMImJrkffC2Oxe/XuExcQG6/avJuTThx3bbjM+Xbf3TJ7EFWeq0HtIb2/2UIHt1XkaPknuZ7I8iDHhVOjXdKp/6i5r9aKNaIXeqD1WCwWJo0ei50U8fKBnSCz5u3lN5fP3ycSiugrWHmAID+ZW2N+cDNHgeKg13jDsPMJAzpi+gGJst7FeMf/eTPu/q45Mebk5Ia5rSX9ghOZKeYEWL+T7USDZOyHPKNl5gQzguf72CDtFtQd1qTNHGz3HK6njC19yrnQqLGiDCfdO5kROo7pWe6Y6ehTSr06DMUTuRsaFavtCxIjJw02Q1q5l835vXs/V2wAzapD+qzY24ZO+FvhTg2LswveZYQR7ZmwF//S+wTUxPkOFrT+bAY63xFM70kZkI+8lXtD2+uRAhHsnTxO2GONA4bpARUnHjJG4dD0RNOC1W25xD76M67pygH8YbmTpPIZ3vtvlr7A+HjzOhot3GE1GS4B1TWMds01pm/843vlChWWLyFZsPcz8ZnZuaHB42tQ X-Forefront-Antispam-Report: CIP:216.228.117.161;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc6edge2.nvidia.com;CAT:NONE;SFS:(13230040)(36860700013)(1800799024)(376014)(82310400026);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 30 Aug 2024 10:59:14.7362 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 470f05b3-dbdd-46fa-7a3c-08dcc8e2c983 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.117.161];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CY4PEPF0000E9D6.namprd05.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH8PR12MB8605 Use the async interface to issue MTT MKEY creation. Extra care is taken at the allocation of FW input commands due to the MTT tables having variable sizes depending on MR. The indirect MKEY is still created synchronously at the end as the direct MKEYs need to be filled in. This makes create_user_mr() 3-5x faster, depending on the size of the MR. Signed-off-by: Dragos Tatulea Reviewed-by: Cosmin Ratiu --- drivers/vdpa/mlx5/core/mr.c | 120 +++++++++++++++++++++++++++++------- 1 file changed, 98 insertions(+), 22 deletions(-) diff --git a/drivers/vdpa/mlx5/core/mr.c b/drivers/vdpa/mlx5/core/mr.c index 4758914ccf86..e72fb11e353d 100644 --- a/drivers/vdpa/mlx5/core/mr.c +++ b/drivers/vdpa/mlx5/core/mr.c @@ -49,17 +49,18 @@ static void populate_mtts(struct mlx5_vdpa_direct_mr *mr, __be64 *mtt) } } -static int create_direct_mr(struct mlx5_vdpa_dev *mvdev, struct mlx5_vdpa_direct_mr *mr) +struct mlx5_create_mkey_mem { + u8 out[MLX5_ST_SZ_BYTES(create_mkey_out)]; + u8 in[MLX5_ST_SZ_BYTES(create_mkey_in)]; + __be64 mtt[]; +}; + +static void fill_create_direct_mr(struct mlx5_vdpa_dev *mvdev, + struct mlx5_vdpa_direct_mr *mr, + struct mlx5_create_mkey_mem *mem) { - int inlen; + void *in = &mem->in; void *mkc; - void *in; - int err; - - inlen = MLX5_ST_SZ_BYTES(create_mkey_in) + roundup(MLX5_ST_SZ_BYTES(mtt) * mr->nsg, 16); - in = kvzalloc(inlen, GFP_KERNEL); - if (!in) - return -ENOMEM; MLX5_SET(create_mkey_in, in, uid, mvdev->res.uid); mkc = MLX5_ADDR_OF(create_mkey_in, in, memory_key_mkey_entry); @@ -76,18 +77,25 @@ static int create_direct_mr(struct mlx5_vdpa_dev *mvdev, struct mlx5_vdpa_direct MLX5_SET(create_mkey_in, in, translations_octword_actual_size, get_octo_len(mr->end - mr->start, mr->log_size)); populate_mtts(mr, MLX5_ADDR_OF(create_mkey_in, in, klm_pas_mtt)); - err = mlx5_vdpa_create_mkey(mvdev, &mr->mr, in, inlen); - kvfree(in); - if (err) { - mlx5_vdpa_warn(mvdev, "Failed to create direct MR\n"); - return err; - } - return 0; + MLX5_SET(create_mkey_in, in, opcode, MLX5_CMD_OP_CREATE_MKEY); + MLX5_SET(create_mkey_in, in, uid, mvdev->res.uid); +} + +static void create_direct_mr_end(struct mlx5_vdpa_dev *mvdev, + struct mlx5_vdpa_direct_mr *mr, + struct mlx5_create_mkey_mem *mem) +{ + u32 mkey_index = MLX5_GET(create_mkey_out, mem->out, mkey_index); + + mr->mr = mlx5_idx_to_mkey(mkey_index); } static void destroy_direct_mr(struct mlx5_vdpa_dev *mvdev, struct mlx5_vdpa_direct_mr *mr) { + if (!mr->mr) + return; + mlx5_vdpa_destroy_mkey(mvdev, mr->mr); } @@ -179,6 +187,76 @@ static int klm_byte_size(int nklms) return 16 * ALIGN(nklms, 4); } +#define MLX5_VDPA_MTT_ALIGN 16 + +static int create_direct_keys(struct mlx5_vdpa_dev *mvdev, struct mlx5_vdpa_mr *mr) +{ + struct mlx5_vdpa_async_cmd *cmds; + struct mlx5_vdpa_direct_mr *dmr; + int err = 0; + int i = 0; + + cmds = kvcalloc(mr->num_directs, sizeof(*cmds), GFP_KERNEL); + if (!cmds) + return -ENOMEM; + + list_for_each_entry(dmr, &mr->head, list) { + struct mlx5_create_mkey_mem *cmd_mem; + int mttlen, mttcount; + + mttlen = roundup(MLX5_ST_SZ_BYTES(mtt) * dmr->nsg, MLX5_VDPA_MTT_ALIGN); + mttcount = mttlen / sizeof(cmd_mem->mtt[0]); + cmd_mem = kvcalloc(1, struct_size(cmd_mem, mtt, mttcount), GFP_KERNEL); + if (!cmd_mem) { + err = -ENOMEM; + goto done; + } + + cmds[i].out = cmd_mem->out; + cmds[i].outlen = sizeof(cmd_mem->out); + cmds[i].in = cmd_mem->in; + cmds[i].inlen = struct_size(cmd_mem, mtt, mttcount); + + fill_create_direct_mr(mvdev, dmr, cmd_mem); + + i++; + } + + err = mlx5_vdpa_exec_async_cmds(mvdev, cmds, mr->num_directs); + if (err) { + + mlx5_vdpa_err(mvdev, "error issuing MTT mkey creation for direct mrs: %d\n", err); + goto done; + } + + i = 0; + list_for_each_entry(dmr, &mr->head, list) { + struct mlx5_vdpa_async_cmd *cmd = &cmds[i++]; + struct mlx5_create_mkey_mem *cmd_mem; + + cmd_mem = container_of(cmd->out, struct mlx5_create_mkey_mem, out); + + if (!cmd->err) { + create_direct_mr_end(mvdev, dmr, cmd_mem); + } else { + err = err ? err : cmd->err; + mlx5_vdpa_err(mvdev, "error creating MTT mkey [0x%llx, 0x%llx]: %d\n", + dmr->start, dmr->end, cmd->err); + } + } + +done: + for (i = i-1; i >= 0; i--) { + struct mlx5_create_mkey_mem *cmd_mem; + + cmd_mem = container_of(cmds[i].out, struct mlx5_create_mkey_mem, out); + kvfree(cmd_mem); + } + + kvfree(cmds); + return err; +} + static int create_indirect_key(struct mlx5_vdpa_dev *mvdev, struct mlx5_vdpa_mr *mr) { int inlen; @@ -279,14 +357,8 @@ static int map_direct_mr(struct mlx5_vdpa_dev *mvdev, struct mlx5_vdpa_direct_mr goto err_map; } - err = create_direct_mr(mvdev, mr); - if (err) - goto err_direct; - return 0; -err_direct: - dma_unmap_sg_attrs(dma, mr->sg_head.sgl, mr->nsg, DMA_BIDIRECTIONAL, 0); err_map: sg_free_table(&mr->sg_head); return err; @@ -401,6 +473,10 @@ static int create_user_mr(struct mlx5_vdpa_dev *mvdev, if (err) goto err_chain; + err = create_direct_keys(mvdev, mr); + if (err) + goto err_chain; + /* Create the memory key that defines the guests's address space. This * memory key refers to the direct keys that contain the MTT * translations From patchwork Fri Aug 30 10:58:33 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Dragos Tatulea X-Patchwork-Id: 13784918 Received: from NAM12-DM6-obe.outbound.protection.outlook.com (mail-dm6nam12on2062.outbound.protection.outlook.com [40.107.243.62]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id F11DB1922F4; Fri, 30 Aug 2024 10:59:21 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.243.62 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725015563; cv=fail; b=Z5ptBufXAbCn1l8MhNLu6WcB09/sVHcvcRDEdxNaNQgIpPul+qIaklXpBXWMFmiahx9CjwdfysChz6VBfJjHZS/Ndnlya7ZNkei4oOrSWkKf4/clyKYHHdBE/2SI/mOuc+HQDz039B8sKOQgGru1cjaiSxLDIEwaVEwcYFGUtq0= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725015563; c=relaxed/simple; bh=JTYCyqAffPOq1r8bi0XVVsYkShwrQ7IXml5fZY7VTY4=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=Gz41jbYdY3pP70hT0R6AFul2IsySR2bcNnwERtH2Df1yGffvTN3u6cMx9VNzmVti47Q0e9PaTWUR3AZLZHyiwNp/Xq/9dijfRd/bwqdX+ppD65bAPcGyQWpHmeR66a2IFy8hFGdRh3SPnV9RPQGaS3yqkutcAcoJRMRZxoth87A= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=Frcrs5qE; arc=fail smtp.client-ip=40.107.243.62 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="Frcrs5qE" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=WM50P9zLx9mqkNwigFGQ9AmfQM7c9MpKCS8ztnBVhHKKLzlXcFE0Oo4VuZVpzxHOxJygNv2GMOw7PZDlacdz9omCL1SU6pl5kG1kBKPl9M4OGPX3srMa/C52+v6z7PG/4/MuQxjSwClRPk02i5HrJllDUyQXLwaRvTprK+W10jMIE0DsEQiVMQJzhUW9oXcz/M82fdkipx644IEW01MrS8H5UwKn9fA8geBiki/aQnQ+GjyXXZ1I8Ikj3kmQSyOLLtLybA0lhlYjKFLzED3PnFi7ZKH1G64IFenuyh+g97ui5i57AJV0QzWazgIx9y24TwrtMdHiJKlTdm4IEsjPkQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=uUhQYi8mfaMvGBOBT7NFXWVPvHHkvNWaiyy39e/q70c=; b=PMubnBsVlBB7t37TSCGt+E8jB+WZTzJfqgOptMz1y2WjHh9W+vFJcR7xAhipsDC+o5WKAa90kak5CVhLvjBH6QYPckx5EyleOSz8wuka6vvTm8jlbToWjyJhTe/1+e3Irmqy/cx0+sy03TkcjGvxyJfgoCCZQyUqETqKakSO+OTmAcQ8QhF4sAEkA4Jx9e1rdMPmSIiy2WUg/UQ4b5pOP19+KLagTMBffUG61A21xVb0sFV2O7Ga+rXhe3Fgbn77VPxSIQanQiNXyKWe9Cng2ZQtAqbEaaxRSq+1K8BjwGTnWcarFkRifGqsmAnWju2gBG3lULM58MebdRLTnDksxg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.160) smtp.rcpttodomain=redhat.com smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=uUhQYi8mfaMvGBOBT7NFXWVPvHHkvNWaiyy39e/q70c=; b=Frcrs5qEeJ/WStny0yD1MN4SjnCTEMI+lFM7DD7MQzaAS2SlUx+cDG5jiGdAKaBIi3cANj0HCe97XJ+DEZ6077NLRNKhuXo9b995fR7nXOY626TIj504iD2IE/Mzg0WMs8dXGXK9iKFqSMpC9vkWDy64GqjHY7vv04KxWLwy0tQVi5GpfCt6AESL7XmdSUHLSa2K5OgQrpHRDumTcmAVtSdyN4lQ414svgtyiQR9zhu9qsKkOyvyHzH6MqS1FFCF4WHHLHN+WGG4/WXeuTkzJL+yPJXsVfyHBymlZVvPl6gcKebZDstzslManFXRNpwglXcGYmMFRPhTrvSYO+SClg== Received: from CH0PR03CA0010.namprd03.prod.outlook.com (2603:10b6:610:b0::15) by SJ1PR12MB6123.namprd12.prod.outlook.com (2603:10b6:a03:45a::5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7849.22; Fri, 30 Aug 2024 10:59:17 +0000 Received: from CH2PEPF0000009A.namprd02.prod.outlook.com (2603:10b6:610:b0:cafe::e0) by CH0PR03CA0010.outlook.office365.com (2603:10b6:610:b0::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7897.28 via Frontend Transport; Fri, 30 Aug 2024 10:59:17 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.160) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.160 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.160; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.160) by CH2PEPF0000009A.mail.protection.outlook.com (10.167.244.22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7918.13 via Frontend Transport; Fri, 30 Aug 2024 10:59:17 +0000 Received: from rnnvmail203.nvidia.com (10.129.68.9) by mail.nvidia.com (10.129.200.66) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Fri, 30 Aug 2024 03:59:07 -0700 Received: from rnnvmail204.nvidia.com (10.129.68.6) by rnnvmail203.nvidia.com (10.129.68.9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Fri, 30 Aug 2024 03:59:06 -0700 Received: from c-237-113-220-225.mtl.labs.mlnx (10.127.8.12) by mail.nvidia.com (10.129.68.6) with Microsoft SMTP Server id 15.2.1544.4 via Frontend Transport; Fri, 30 Aug 2024 03:59:03 -0700 From: Dragos Tatulea To: "Michael S . Tsirkin" , Jason Wang , Eugenio Perez Martin , Si-Wei Liu , , Gal Pressman CC: Dragos Tatulea , , , Parav Pandit , Xuan Zhuo , Cosmin Ratiu Subject: [PATCH vhost v2 2/7] vdpa/mlx5: Delete direct MKEYs in parallel Date: Fri, 30 Aug 2024 13:58:33 +0300 Message-ID: <20240830105838.2666587-4-dtatulea@nvidia.com> X-Mailer: git-send-email 2.45.1 In-Reply-To: <20240830105838.2666587-2-dtatulea@nvidia.com> References: <20240830105838.2666587-2-dtatulea@nvidia.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH2PEPF0000009A:EE_|SJ1PR12MB6123:EE_ X-MS-Office365-Filtering-Correlation-Id: dc065363-0d0d-4ebd-77e2-08dcc8e2cadb X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|36860700013|1800799024|376014|82310400026; X-Microsoft-Antispam-Message-Info: =?utf-8?q?zfy1FjAskf54mafxYdQGwHzVWIYdsZs?= =?utf-8?q?aU8o+PcsRjyKrYaDXbBzJAXhXFoGMTZYNF9CBTMZurJD9+3pr3P7c54X3epursVDk?= =?utf-8?q?wrBhCDk35DuEL8N+l4PzKF/89hQKCBaAgEmAyiRUBINCHi20Kw2HeYRboa+cKTz+/?= =?utf-8?q?76M3utjYXruweHxANG1/YKINirTs4TfTSQn9mPd2Fm5v4ZideUyj9HN4rJDedw06O?= =?utf-8?q?CoTCp0tWVTW/5Vs+0wc+kk6qgE2EmnWh4Htos/gXs5J4QPj0KZtX6cvGN7dlPcSjD?= =?utf-8?q?iXDSZfBoWzytNO3xYBtX5m8wRXL5PgfsIMEEwz7SAyC2VZuxVri4H3dFHZf52dkp4?= =?utf-8?q?kN3qOoHVTlQvgiO5Lrr9zN8x6tdZTTUn/30NSA/Ub/ZdmrVdns+3JZg8SrDaCh2GI?= =?utf-8?q?/elMlKtjDqLhbJcEEJ1OvsH2rb/8FGNfw/0oWeNAmYLUG+HaoVV77vlFei+eZYDKf?= =?utf-8?q?qM794OiTw2Cbasdd+EaMQv4bSHb/lMjfXVDVHgQhCigxIKArWdjFpix2qUmpUc2wM?= =?utf-8?q?IC88d3DSu3SDiH19p648tufd9zDJOV7KaS7DSL/CJDaEdwFKrk7u5IVBbHxCnHiek?= =?utf-8?q?HmLarAr65fKZ/rC9bP4EwRcgyg0Qhz8c/VG+494diwA5FY6FT11RoBOT46UzGn2OO?= =?utf-8?q?VICUMjBVoDdM8wRzp1k+mIeJKiegZX2v3S7yUhKa/AsaaAUCLCDYWN80qLIjG05Vd?= =?utf-8?q?Oq2YZOcZk15CJwuo7A8w7OzbdxpKp6154Lqc/BaoJxcPCZkiP++m3YC0x0JsEWeq0?= =?utf-8?q?8HE2WZXRPh7jvdO8IG9LHG0cPS4WPHqliKumIUg4ArDjcFB2f4HdOee+iPhn45RfI?= =?utf-8?q?+LjqHmWZkJUpzAh2ch0tlRwGHcF5+oX1iqRI1m1+G8hxoviY4uRlb+yx1euyosIsz?= =?utf-8?q?bN+ayq8FdNCfG4Q0sEFFqVDMcnYMygVAnXxfHBPZw02BVwrnZ0lbRRRb++oBzGG0F?= =?utf-8?q?MHx+a0lHIkzxFKMfb6TSjoQXyLoJl8r9jlYhdqePC+rVZPfm5FQQJzlm/BsravFWb?= =?utf-8?q?HCPR5RdPn+IzRxnnGNbY0OeJGO+A84p42UbgtqUCu+7tsdTzztod1vG4r7LkZgqsh?= =?utf-8?q?EtwtjsORhrvAsbn3Wi1p/dpJDpYsu06SXBzWurg+i7a/XqkTAGvmjRDpUC1zdaNmQ?= =?utf-8?q?eNlnUR6o92S+BNvme9RUdVHPZSdQ+jKG4OwOBl8+I1YVUtueZuXHLAIrWh1m+xQ13?= =?utf-8?q?vL7f7Gh8yIDldHbjW3yz+3ogSCzDzQIaRLSFPe3xRIHeTy9dCkxhhx+XqVPZxzFeQ?= =?utf-8?q?ZfrQNZ/Ljg0iy6VT8ef8Gj3t3+JAc7e9cPKRRCl/3k47WRV4xioYderr2JOZ3WqOw?= =?utf-8?q?I2Kita67UI/i/eCTZnuQEMif1acbLb0jeggkn0GjpzeWhOPZIGRwE1w+9CD5zmNBz?= =?utf-8?q?4rYMDAGOgva?= X-Forefront-Antispam-Report: CIP:216.228.117.160;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc6edge1.nvidia.com;CAT:NONE;SFS:(13230040)(36860700013)(1800799024)(376014)(82310400026);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 30 Aug 2024 10:59:17.0218 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: dc065363-0d0d-4ebd-77e2-08dcc8e2cadb X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.117.160];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CH2PEPF0000009A.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ1PR12MB6123 Use the async interface to issue MTT MKEY deletion. This makes destroy_user_mr() on average 8x times faster. This number is also dependent on the size of the MR being deleted. Signed-off-by: Dragos Tatulea Reviewed-by: Cosmin Ratiu Acked-by: Eugenio Pérez --- drivers/vdpa/mlx5/core/mr.c | 64 +++++++++++++++++++++++++++++++++++++ 1 file changed, 64 insertions(+) diff --git a/drivers/vdpa/mlx5/core/mr.c b/drivers/vdpa/mlx5/core/mr.c index e72fb11e353d..64bcae2bae8a 100644 --- a/drivers/vdpa/mlx5/core/mr.c +++ b/drivers/vdpa/mlx5/core/mr.c @@ -55,6 +55,11 @@ struct mlx5_create_mkey_mem { __be64 mtt[]; }; +struct mlx5_destroy_mkey_mem { + u8 out[MLX5_ST_SZ_BYTES(destroy_mkey_out)]; + u8 in[MLX5_ST_SZ_BYTES(destroy_mkey_in)]; +}; + static void fill_create_direct_mr(struct mlx5_vdpa_dev *mvdev, struct mlx5_vdpa_direct_mr *mr, struct mlx5_create_mkey_mem *mem) @@ -91,6 +96,17 @@ static void create_direct_mr_end(struct mlx5_vdpa_dev *mvdev, mr->mr = mlx5_idx_to_mkey(mkey_index); } +static void fill_destroy_direct_mr(struct mlx5_vdpa_dev *mvdev, + struct mlx5_vdpa_direct_mr *mr, + struct mlx5_destroy_mkey_mem *mem) +{ + void *in = &mem->in; + + MLX5_SET(destroy_mkey_in, in, uid, mvdev->res.uid); + MLX5_SET(destroy_mkey_in, in, opcode, MLX5_CMD_OP_DESTROY_MKEY); + MLX5_SET(destroy_mkey_in, in, mkey_index, mlx5_mkey_to_idx(mr->mr)); +} + static void destroy_direct_mr(struct mlx5_vdpa_dev *mvdev, struct mlx5_vdpa_direct_mr *mr) { if (!mr->mr) @@ -257,6 +273,53 @@ static int create_direct_keys(struct mlx5_vdpa_dev *mvdev, struct mlx5_vdpa_mr * return err; } +DEFINE_FREE(free_cmds, struct mlx5_vdpa_async_cmd *, kvfree(_T)) +DEFINE_FREE(free_cmd_mem, struct mlx5_destroy_mkey_mem *, kvfree(_T)) + +static int destroy_direct_keys(struct mlx5_vdpa_dev *mvdev, struct mlx5_vdpa_mr *mr) +{ + struct mlx5_destroy_mkey_mem *cmd_mem __free(free_cmd_mem) = NULL; + struct mlx5_vdpa_async_cmd *cmds __free(free_cmds) = NULL; + struct mlx5_vdpa_direct_mr *dmr; + int err = 0; + int i = 0; + + cmds = kvcalloc(mr->num_directs, sizeof(*cmds), GFP_KERNEL); + cmd_mem = kvcalloc(mr->num_directs, sizeof(*cmd_mem), GFP_KERNEL); + if (!cmds || !cmd_mem) + return -ENOMEM; + + list_for_each_entry(dmr, &mr->head, list) { + cmds[i].out = cmd_mem[i].out; + cmds[i].outlen = sizeof(cmd_mem[i].out); + cmds[i].in = cmd_mem[i].in; + cmds[i].inlen = sizeof(cmd_mem[i].in); + fill_destroy_direct_mr(mvdev, dmr, &cmd_mem[i]); + i++; + } + + err = mlx5_vdpa_exec_async_cmds(mvdev, cmds, mr->num_directs); + if (err) { + + mlx5_vdpa_err(mvdev, "error issuing MTT mkey deletion for direct mrs: %d\n", err); + return err; + } + + i = 0; + list_for_each_entry(dmr, &mr->head, list) { + struct mlx5_vdpa_async_cmd *cmd = &cmds[i++]; + + dmr->mr = 0; + if (cmd->err) { + err = err ? err : cmd->err; + mlx5_vdpa_err(mvdev, "error deleting MTT mkey [0x%llx, 0x%llx]: %d\n", + dmr->start, dmr->end, cmd->err); + } + } + + return err; +} + static int create_indirect_key(struct mlx5_vdpa_dev *mvdev, struct mlx5_vdpa_mr *mr) { int inlen; @@ -565,6 +628,7 @@ static void destroy_user_mr(struct mlx5_vdpa_dev *mvdev, struct mlx5_vdpa_mr *mr struct mlx5_vdpa_direct_mr *n; destroy_indirect_key(mvdev, mr); + destroy_direct_keys(mvdev, mr); list_for_each_entry_safe_reverse(dmr, n, &mr->head, list) { list_del_init(&dmr->list); unmap_direct_mr(mvdev, dmr); From patchwork Fri Aug 30 10:58:34 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dragos Tatulea X-Patchwork-Id: 13784919 Received: from NAM12-BN8-obe.outbound.protection.outlook.com (mail-bn8nam12on2067.outbound.protection.outlook.com [40.107.237.67]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 92622192D6C; Fri, 30 Aug 2024 10:59:25 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.237.67 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725015569; cv=fail; b=VQ3mvL3mLJ6IBhPawkSea4OxYfv5hWOBeA4I6Xe4pswm4pqWwlnMoLIijRbN3CFpqsul40MYHEEleyXMIaZhcpB/zA7OAn2s5D3N97LWeVr7tHDqKwhXO5rdou84IiVORugMKkmkqTimCrLkEvtsbe9O/mxDrbyLVY7yCcqqqR4= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725015569; c=relaxed/simple; bh=BjCE2IXi+eCIR9+jAl5ssp06P2II3arGbQjk+SEgNBU=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=nvNFwjwIsNxe0TOIk33+9ANp6VoXZpJvMZC1jITKuvY3k2iXmd3goxkwIuFdJ3tsIh8ImLBOkdv2KeuhvyIxUxwuJ7x/RuZtO8pDxWsgiQTwb48MX41eyXv7nVLM8oowK3xtq6wIkHdfDA/qaaW0dzRVTERiDi3caPgPFpxIhwo= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=YuSClkX1; arc=fail smtp.client-ip=40.107.237.67 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="YuSClkX1" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=ColBTrsYsgFKkaZzeoQjbX/YwM96VAnWn58+b74+992qU/jZZ8l+zSqI8No8OCvS5aF4DdhnTSzpiVF1GSAVH+4vcYQcZhxfO7sxPSS7dqIjSeFoojf/W1gGxFlbVivpy8gNuvGPn/KFmeVUDLQ171zERQ/IPDkRva2PdMTsKnm3KScZMO3HRr5r9S/zdlbzsYbTvOll5ICPaw9osPYVKdYFesKJZQYOkizdZLP1LAsNgexiy0rA/hyK8B9QFF34CYlpVgngonnwGFAv4Fp4orM7sQQKphBdVXEitutFWGtnSdED/+9EyhxipvNUxCJXvyvIsmTPrrVl3Hcor6Vs3g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=Fp67rwtDEa6y7plVm9B7RQAtIKtBV8z4b4+Z4RpFf4c=; b=ZQ9VH9oB3HWRN1/ymNarzQgy59+rxeAmvvk2OHwgwf6BeEUirR7RfKnST7tk/ljSutUiF62LXCkyCI8jOxA6yawBR/hSXusFqgag9DOSHPxuQf7zaksIyZgg/r5LT90bKFzCc91srySF7elRdhuLvQFc+3RDFvqhllxhgbu6z1o186+g+5y7zEeriqPbxJhdk3IdlUvCDXwbZ+tFi2tejq+bglVEkAwQ+Ysl6j0Y+4wkTkxmM0BvQ3xvE5jWsQsMgsOUKXZKGZzvDn/HCEHJFrKe49Ef5Q610giYj0YKypUoLUtgsl3suGi8jtZnN5wv8vAsGOoLMcUOA98o3T0iEA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.160) smtp.rcpttodomain=redhat.com smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Fp67rwtDEa6y7plVm9B7RQAtIKtBV8z4b4+Z4RpFf4c=; b=YuSClkX17+TKLICiHJ9iWLIWeE0XIeb0p8aO4pNDrWrS8B/HScMJ8Dw1nUDhzVhq6ii0ze0yLxx3h9LxPgiNWrpaE3C7vIg8ozl9mFtfWwR+NZu2E2j4uN2FHXWPyuMzznLBEYhb3nJMtFlflLEQnqZp7asoS4WYGXjt96RjknQG26sGCljWIEOGZbvssueuwjGy4fH/g7FVx3SpaS9UAetUEie/M2CEWZRdhXHLq6W5+pv4KGuL3g1lKSPWCgdoXOjaTK+C25qMhqszyBHdcdt5eT97jUzolmClGBsiICfND0QQgwUxA+UwQi9JClaMSfvSYEnLbubdUlxXpMpCgA== Received: from MN2PR22CA0019.namprd22.prod.outlook.com (2603:10b6:208:238::24) by DS7PR12MB5888.namprd12.prod.outlook.com (2603:10b6:8:7b::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7897.27; Fri, 30 Aug 2024 10:59:21 +0000 Received: from CH2PEPF000000A0.namprd02.prod.outlook.com (2603:10b6:208:238:cafe::d7) by MN2PR22CA0019.outlook.office365.com (2603:10b6:208:238::24) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7897.29 via Frontend Transport; Fri, 30 Aug 2024 10:59:21 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.160) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.160 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.160; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.160) by CH2PEPF000000A0.mail.protection.outlook.com (10.167.244.26) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7918.13 via Frontend Transport; Fri, 30 Aug 2024 10:59:21 +0000 Received: from rnnvmail203.nvidia.com (10.129.68.9) by mail.nvidia.com (10.129.200.66) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Fri, 30 Aug 2024 03:59:10 -0700 Received: from rnnvmail204.nvidia.com (10.129.68.6) by rnnvmail203.nvidia.com (10.129.68.9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Fri, 30 Aug 2024 03:59:10 -0700 Received: from c-237-113-220-225.mtl.labs.mlnx (10.127.8.12) by mail.nvidia.com (10.129.68.6) with Microsoft SMTP Server id 15.2.1544.4 via Frontend Transport; Fri, 30 Aug 2024 03:59:07 -0700 From: Dragos Tatulea To: "Michael S . Tsirkin" , Jason Wang , Eugenio Perez Martin , Si-Wei Liu , , Gal Pressman CC: Dragos Tatulea , , , Parav Pandit , Xuan Zhuo , Cosmin Ratiu Subject: [PATCH vhost v2 3/7] vdpa/mlx5: Rename function Date: Fri, 30 Aug 2024 13:58:34 +0300 Message-ID: <20240830105838.2666587-5-dtatulea@nvidia.com> X-Mailer: git-send-email 2.45.1 In-Reply-To: <20240830105838.2666587-2-dtatulea@nvidia.com> References: <20240830105838.2666587-2-dtatulea@nvidia.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH2PEPF000000A0:EE_|DS7PR12MB5888:EE_ X-MS-Office365-Filtering-Correlation-Id: a685ae5a-e078-4f9f-834c-08dcc8e2cd4c X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|82310400026|376014|36860700013|1800799024; X-Microsoft-Antispam-Message-Info: aVrepsfhCe7Fz19scTi+nXVg5YdgjmWyXhDV3r3RSqXyQKGopJ0KR/5UsrewqMUIGABMqhNBVuGT7LvyDGxqh8Sem7Kfk1/QDCns/PDcm44VovV815soiLvMPZBiqBOpOnRpJMXm/0jzOptISW2ebnwm5QCqKS9NcYJl1MX1kqeTWSM+Kz3HS/+dN7I4lEo6ZGCYKSYviI/LJclcOFUB43/QRu2x/D+iWXgwv06gNGVZMONjPJtdff9lYOQi7UvkJ265Ih/n0hU56owqrMQ0ykrEgPPY8RD9lXGJB0ExPdndt0JLXdFB1m1PHDkwLphR6QO0hcgFKrUk4dCfKFzT7xpprgvFdPY37WbiL5mvydkiz1SDThbF47vDW40ePvZhDxD8aiaGjgHi1WStVu2q3PWxhQHgBnymwnmIRhUk/brfXu4FhfNh7CUSd1bjDW+fZWJMy9RF1m/YwMbNzY7JtU+gswquTP6VrmR9l4chyrpjYnDXBZm8ZqtIg/zbswHPK482zt9Yw65gDy66JiknNS7k55xo3cGwRjB0S277ecmolPq+iAN1yKhIDb2BAie7ssuwll1XDFaLJW1SuOi3Yl6L9bdguTiFxOzeBj4dPkwOniD6MfOYknxYT17V0KAOb5l7Jlj/SWX9h02EIh5XsHTQLGKgnqpyVhRQ0LDhtKs9EBkQ25W09GfzaQ8e5UYNBsBXLCF9JXnvXnCY4UkTqkBtNU2RGxI222lz/1sedUZRoJmuJh/eFbO+dHKNAQOt+oM4UcZ2ao0tQb5zHJC/jdnthSAO6dDHVMKI/cslcr3NGe4e6vO87JHSDtOgr1doighygGBWaSSHf9EpCZmeh29k/Pkq3mTFAbcTic180gVOUToyRsxxNQBGyRn02+k+Xg9iAfHnvfq2SbtSRrGWIbvFmZKFUcvIil6Cj63O+hmGS3R7mvAVhh0M6Ym5vapOTwjMmephXHsTo5XZCBKncsV42jgzM0poT7NVqb4F0yoM+MR7gBZiiKFlDNLH6VV/SIPeV/SAkI/Hc9B+F4uBXg6FbVt/mTtj00zf2ALLwtV7yTUGfBF1WG0aZAHt9BzfDyIb54+GHnKmMDgBNo4+NbeN5sq/upJLlKUR73dVbDEumpo4qpUDsbV7duW9Z10paRo02Pq/it1iZBWYVNOe5KBKmgMK0UM2iOKHhCuJkEHoXMxi+u8SPSwbzukTYPBok87OyRSirSIFBHhJ6AQ68quZTkzVRQn56WKqFLd/85DFnB/zZY99HLS12mZJ/eQL+ZPPnWC3kq6Ntvp22TNWDyi9BtvJpejqaBERdTCzkTXEXim6n8agihqw++rgF0K479LoxMgo/sxuM8MDIZMn/ui2e4r575jvMTAmisXhLd0lZ4IW03xPn1rkOGdQCA+s2DMkFHdxgkwmbOrhkVJL89w0wGGLvCmExkLt2iqDmIqGwDxwqfXguxYyt2uKoXp2 X-Forefront-Antispam-Report: CIP:216.228.117.160;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc6edge1.nvidia.com;CAT:NONE;SFS:(13230040)(82310400026)(376014)(36860700013)(1800799024);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 30 Aug 2024 10:59:21.1025 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: a685ae5a-e078-4f9f-834c-08dcc8e2cd4c X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.117.160];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CH2PEPF000000A0.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS7PR12MB5888 A followup patch will use this name for something else. Signed-off-by: Dragos Tatulea Reviewed-by: Cosmin Ratiu --- drivers/vdpa/mlx5/core/mlx5_vdpa.h | 2 +- drivers/vdpa/mlx5/core/mr.c | 2 +- drivers/vdpa/mlx5/net/mlx5_vnet.c | 8 ++++---- 3 files changed, 6 insertions(+), 6 deletions(-) diff --git a/drivers/vdpa/mlx5/core/mlx5_vdpa.h b/drivers/vdpa/mlx5/core/mlx5_vdpa.h index 24fa00afb24f..4d217d18239c 100644 --- a/drivers/vdpa/mlx5/core/mlx5_vdpa.h +++ b/drivers/vdpa/mlx5/core/mlx5_vdpa.h @@ -135,7 +135,7 @@ int mlx5_vdpa_create_mkey(struct mlx5_vdpa_dev *mvdev, u32 *mkey, u32 *in, int mlx5_vdpa_destroy_mkey(struct mlx5_vdpa_dev *mvdev, u32 mkey); struct mlx5_vdpa_mr *mlx5_vdpa_create_mr(struct mlx5_vdpa_dev *mvdev, struct vhost_iotlb *iotlb); -void mlx5_vdpa_destroy_mr_resources(struct mlx5_vdpa_dev *mvdev); +void mlx5_vdpa_clean_mrs(struct mlx5_vdpa_dev *mvdev); void mlx5_vdpa_get_mr(struct mlx5_vdpa_dev *mvdev, struct mlx5_vdpa_mr *mr); void mlx5_vdpa_put_mr(struct mlx5_vdpa_dev *mvdev, diff --git a/drivers/vdpa/mlx5/core/mr.c b/drivers/vdpa/mlx5/core/mr.c index 64bcae2bae8a..50bb2cc95ea2 100644 --- a/drivers/vdpa/mlx5/core/mr.c +++ b/drivers/vdpa/mlx5/core/mr.c @@ -719,7 +719,7 @@ static void mlx5_vdpa_show_mr_leaks(struct mlx5_vdpa_dev *mvdev) } -void mlx5_vdpa_destroy_mr_resources(struct mlx5_vdpa_dev *mvdev) +void mlx5_vdpa_clean_mrs(struct mlx5_vdpa_dev *mvdev) { for (int i = 0; i < MLX5_VDPA_NUM_AS; i++) mlx5_vdpa_update_mr(mvdev, NULL, i); diff --git a/drivers/vdpa/mlx5/net/mlx5_vnet.c b/drivers/vdpa/mlx5/net/mlx5_vnet.c index 822092eccb32..cf2b77ebc72b 100644 --- a/drivers/vdpa/mlx5/net/mlx5_vnet.c +++ b/drivers/vdpa/mlx5/net/mlx5_vnet.c @@ -3223,7 +3223,7 @@ static void mlx5_vdpa_set_status(struct vdpa_device *vdev, u8 status) err_driver: unregister_link_notifier(ndev); err_setup: - mlx5_vdpa_destroy_mr_resources(&ndev->mvdev); + mlx5_vdpa_clean_mrs(&ndev->mvdev); ndev->mvdev.status |= VIRTIO_CONFIG_S_FAILED; err_clear: up_write(&ndev->reslock); @@ -3275,7 +3275,7 @@ static int mlx5_vdpa_compat_reset(struct vdpa_device *vdev, u32 flags) } if (flags & VDPA_RESET_F_CLEAN_MAP) - mlx5_vdpa_destroy_mr_resources(&ndev->mvdev); + mlx5_vdpa_clean_mrs(&ndev->mvdev); ndev->mvdev.status = 0; ndev->mvdev.suspended = false; ndev->cur_num_vqs = MLX5V_DEFAULT_VQ_COUNT; @@ -3433,7 +3433,7 @@ static void mlx5_vdpa_free(struct vdpa_device *vdev) ndev = to_mlx5_vdpa_ndev(mvdev); free_fixed_resources(ndev); - mlx5_vdpa_destroy_mr_resources(mvdev); + mlx5_vdpa_clean_mrs(mvdev); if (!is_zero_ether_addr(ndev->config.mac)) { pfmdev = pci_get_drvdata(pci_physfn(mvdev->mdev->pdev)); mlx5_mpfs_del_mac(pfmdev, ndev->config.mac); @@ -4008,7 +4008,7 @@ static int mlx5_vdpa_dev_add(struct vdpa_mgmt_dev *v_mdev, const char *name, err_res2: free_fixed_resources(ndev); err_mr: - mlx5_vdpa_destroy_mr_resources(mvdev); + mlx5_vdpa_clean_mrs(mvdev); err_res: mlx5_vdpa_free_resources(&ndev->mvdev); err_mpfs: From patchwork Fri Aug 30 10:58:35 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Dragos Tatulea X-Patchwork-Id: 13784920 Received: from NAM10-DM6-obe.outbound.protection.outlook.com (mail-dm6nam10on2076.outbound.protection.outlook.com [40.107.93.76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 24E9F18FDC8; Fri, 30 Aug 2024 10:59:35 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.93.76 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725015578; cv=fail; b=J8MRK2C2rkSzA8b5XmV2gZymivTgjNsHXzm6CH7RVnCM7cm8gUtus3VSBFcK0v8N4m4esRvxeVdTZn61SyReVnWhMiz7VFpDWAEc/iPJ058gjHZ4yW55mf0r4y9nXDqzuLld1tNftGZs8EkBwqfkhLQwtnVV/31wnif82nHATIM= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725015578; c=relaxed/simple; bh=dWyAh5q+A86sqyF5vvoIzv96ciGvmPJDdWjeWGh7QkA=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=IU8Xy2A9+4TTJ0/raxA7HgP5HD+Vu+LS0U1VmoOqHgxsKAU8Rw2cJLBUqU+jjp06QW/gLdRmdH7WY0Lo7zdLbBjZQWIj/VAYGYbHhrFcXblOC4a5dSjxsg97sYBOYBzsbmwTU+TCiVWM3kpxalr29yGH5T1pRpRinn8mE4ClofQ= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=d++812bh; arc=fail smtp.client-ip=40.107.93.76 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="d++812bh" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=TnbKFDfs56W/NjZNmMBdKH+bBcshdbPD6TFJU1dL156UL3efh61Br4IUlmg9TIDq6Ft5mRHSES+w4uN+KcuTMvgeWkd90n+c3zPyBxEVs7k1QWGCYQuYbVPniyEnZhqPJJmlylqwk8HkrXmt85jNn8SeL0p2gDDiCsPQHUHpxQC37jJHS44VaFHKYPplqqPcujt4ZkRAUZ5Q9g6DOx1a05NHUfTdAyzmfY7Bvc2BpV82kqT06X0mtY59wygGk6Bljc1/qFzPYBtf3+LyKpiTfDO8Q+xIMxczs72zYL8tk2u7tZCHJsFT8Ynip3fCRYLNwZrRtuLrwx4frh2S9VoLxg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=yNNUbO7FmxX7S/l+kTnT8/Y/EDXjNREHAi7SYJ31JjI=; b=GdajGvObd7Q9lpuZMh1n7t1gA67Lb5cyIUS9NlbFmmoe57giibsF1YAuadtNceaLFKL/14S+RdsN/KNMyqAcbNnWVNokc71PonwWwKaRbT3QlExfq68bPo9lDDPW+YExjOMET46LVEz4raNIYsJuKcW248EfXrjtL1n86ztv73lyPxMA2VV9Is0hBiLzYT3wJa/oRrkaezA8YMU+89bKBijYsa/O6p6r38+CfoUDUZTFzyIP4EJrQQxOQhIBrMy8xbm33X1bPWE26n6IXI1AuS4kWL0vRDfpBG4aR1myC7dN33GNGss7o/N+f58/XoLgBJQAW1zuEG4RMsYlWQJKAQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.160) smtp.rcpttodomain=redhat.com smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=yNNUbO7FmxX7S/l+kTnT8/Y/EDXjNREHAi7SYJ31JjI=; b=d++812bhEwiKbLzWGAuB3L6q5jvigKd8e0ZHTwNt/2FytJ/NyyKixRiMR8UnzH0HIGKng9QgXX59Nl8RgSXH2la4a89Uu2pztjmNBN83RCXwnSz1zyyAXMhpncX8xAXObmZ3Ltu7bdLFwWURiadpWzr9ikR2UtgBJFyYAdR7C0D3/juP7R1nE7vqIRUnER4dwhUvLgLE8mfiIeAU1bW1SmCqJMN/GB9TpgcyYcYWNsG/JrzVFucTtyaaAEnUFENcdaEQhnN8gfHnoYaY4ctAya3Y9UTbxIrZyA4dzRIzkDp+bLWkQRc/RdbaQ+WxwdXSnCepP/vXYSPOdBwBtPSZoQ== Received: from CH0PR04CA0007.namprd04.prod.outlook.com (2603:10b6:610:76::12) by MN0PR12MB5764.namprd12.prod.outlook.com (2603:10b6:208:377::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7918.20; Fri, 30 Aug 2024 10:59:30 +0000 Received: from CH2PEPF0000009D.namprd02.prod.outlook.com (2603:10b6:610:76:cafe::20) by CH0PR04CA0007.outlook.office365.com (2603:10b6:610:76::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7897.28 via Frontend Transport; Fri, 30 Aug 2024 10:59:29 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.160) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.160 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.160; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.160) by CH2PEPF0000009D.mail.protection.outlook.com (10.167.244.25) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7918.13 via Frontend Transport; Fri, 30 Aug 2024 10:59:29 +0000 Received: from rnnvmail203.nvidia.com (10.129.68.9) by mail.nvidia.com (10.129.200.66) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Fri, 30 Aug 2024 03:59:14 -0700 Received: from rnnvmail204.nvidia.com (10.129.68.6) by rnnvmail203.nvidia.com (10.129.68.9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Fri, 30 Aug 2024 03:59:13 -0700 Received: from c-237-113-220-225.mtl.labs.mlnx (10.127.8.12) by mail.nvidia.com (10.129.68.6) with Microsoft SMTP Server id 15.2.1544.4 via Frontend Transport; Fri, 30 Aug 2024 03:59:10 -0700 From: Dragos Tatulea To: "Michael S . Tsirkin" , Jason Wang , Eugenio Perez Martin , Si-Wei Liu , , Gal Pressman CC: Dragos Tatulea , , , Parav Pandit , Xuan Zhuo , Cosmin Ratiu Subject: [PATCH vhost v2 4/7] vdpa/mlx5: Extract mr members in own resource struct Date: Fri, 30 Aug 2024 13:58:35 +0300 Message-ID: <20240830105838.2666587-6-dtatulea@nvidia.com> X-Mailer: git-send-email 2.45.1 In-Reply-To: <20240830105838.2666587-2-dtatulea@nvidia.com> References: <20240830105838.2666587-2-dtatulea@nvidia.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH2PEPF0000009D:EE_|MN0PR12MB5764:EE_ X-MS-Office365-Filtering-Correlation-Id: 9d9812b3-7e43-48a9-685c-08dcc8e2d278 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|36860700013|376014|1800799024|82310400026; X-Microsoft-Antispam-Message-Info: =?utf-8?q?6v9RaNCWTVh1DwXULBmvAJLASBN8Jx1?= =?utf-8?q?/kErVI4RQ6NI8Iam747l23o9ZqHoIyqHtymyfn+5La9wRXy4ru70R8ppcB0+Ty9lo?= =?utf-8?q?bYT6AfMVZOrlq8QVmbtyPrf46G8B/KXRgYDUBxk22wopjgkNAX8mxVyHySZZba60y?= =?utf-8?q?zRkhEOW+LGtdLEZaBavDsCo7D8JoRSSnfn8OvJYYw0A13DSruGcgS8CZ/05HuZYGj?= =?utf-8?q?LT6q704f06Yl7SWRziJkIDG5CfnEjPmhkhhkzJH15duNmzbjtjj2ZF8WbCtj/N+hj?= =?utf-8?q?OA9R4XWD4UQSerDq5OzuB7laSeQYuCOSfIa8jkYRnaKF4MJghMBnwvX/ClLEfOy9a?= =?utf-8?q?HaDVLcIfwuAukV6VskulYQO+ixJXez7LNHNpbw6WP8ouYUKNvTqvF+ApxPpnIZVul?= =?utf-8?q?ZrjQxcQkShwbAmMcGMe5Tgt7ppL8qz1tcsx6h1NPfrq0JT9MFsMpj58UFpghm2NEz?= =?utf-8?q?lWGRuj2bLi5ysI65yj7TS0qwg3pN3TE/iJYCZseU1xANml7676/ZzgfQCknoOTvwj?= =?utf-8?q?jUlO2QViqusOIFfTvcVRvxFToLzgYmTxdhdbV0hRcpdShW9OjJveih14kU15fpUnx?= =?utf-8?q?g1PnSNgpoVMJWb1tRRa8MuiSa/qj0Ba/gYGUmx/bwxp9Msp/eMB/znr/dc8FGEbH8?= =?utf-8?q?EPcCr3wa/rYxI7w9xgYdCyjcXakwyHUsodFwpsyUGTM/DZ3utrQpiER+ct21JjjQr?= =?utf-8?q?3YRXT7RK6b6TfnXIgIyylbeQOeKVwLgSZT75El6t84HZrvKpBcVO+C6Mjzcfo16ca?= =?utf-8?q?FnDauWEWQRLpqf68Qv/exK/Jm8aJk7uWBbunKnH1jR6MR1RqYqjx0+AuSPwldZiIx?= =?utf-8?q?miLYV9RW5EBYCjzAYaI2KT0K17BnIHXn79N2PDIxQRcgeuZns1x4eyZCTxu+L7lDt?= =?utf-8?q?MWpi+HwSjiQJ0k4Qw7iruzsgAMRkO/xUxqutcxJZtwXTaoPtxgZ+78T0hTOPg1JQl?= =?utf-8?q?TLwahkVW2BKf8MGjNjDDQc9QR/KuklPTGN6idqF423CpvY1IovpD66smXzNSQUSKK?= =?utf-8?q?ch24vY+DOBzV6MeRE/NT5FWW35SGLwZlFsRivJp4dN+M3WmYfOTtxVwQSx9KA0LYU?= =?utf-8?q?iAXk+OYihaU50PsRNlQ+mZWJ3wCJReNtybkUgj/Hqxj5cHv6VVk1NVqHAuyTsSf8K?= =?utf-8?q?5FfTrZQiwz5IPDU2rDKll+0Rdj6K864CHrlAFBGaIsoH3DFO3StaxoOxx3MLyBbWn?= =?utf-8?q?Kw5DQQ4qvPIuf2GgBnAF2k1sf2mZmSTFiLMiLs0D6p05pOYETr2x9grnxvkYlL1kZ?= =?utf-8?q?L99/jvIkXPRGDprVF4Yi7CU9pXiZ/WENlSWIGt3y0BtLWr3huYM6qaXDNX8BYOA2K?= =?utf-8?q?SFH5YRbXZrUmoofSzPVxHzs7IMvElreVdXCWtJ4fzEDDalYYnMI57gkUUULC9Yk1f?= =?utf-8?q?1Xmt/DfltE8?= X-Forefront-Antispam-Report: CIP:216.228.117.160;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc6edge1.nvidia.com;CAT:NONE;SFS:(13230040)(36860700013)(376014)(1800799024)(82310400026);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 30 Aug 2024 10:59:29.7475 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 9d9812b3-7e43-48a9-685c-08dcc8e2d278 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.117.160];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CH2PEPF0000009D.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN0PR12MB5764 Group all mapping related resources into their own structure. Upcoming patches will add more members in this new structure. Signed-off-by: Dragos Tatulea Reviewed-by: Cosmin Ratiu Acked-by: Eugenio Pérez --- drivers/vdpa/mlx5/core/mlx5_vdpa.h | 13 ++++++----- drivers/vdpa/mlx5/core/mr.c | 30 ++++++++++++------------- drivers/vdpa/mlx5/core/resources.c | 6 ++--- drivers/vdpa/mlx5/net/mlx5_vnet.c | 36 +++++++++++++++--------------- 4 files changed, 44 insertions(+), 41 deletions(-) diff --git a/drivers/vdpa/mlx5/core/mlx5_vdpa.h b/drivers/vdpa/mlx5/core/mlx5_vdpa.h index 4d217d18239c..5ae6deea2a8a 100644 --- a/drivers/vdpa/mlx5/core/mlx5_vdpa.h +++ b/drivers/vdpa/mlx5/core/mlx5_vdpa.h @@ -83,10 +83,18 @@ enum { MLX5_VDPA_NUM_AS = 2 }; +struct mlx5_vdpa_mr_resources { + struct mlx5_vdpa_mr *mr[MLX5_VDPA_NUM_AS]; + unsigned int group2asid[MLX5_VDPA_NUMVQ_GROUPS]; + struct list_head mr_list_head; + struct mutex mr_mtx; +}; + struct mlx5_vdpa_dev { struct vdpa_device vdev; struct mlx5_core_dev *mdev; struct mlx5_vdpa_resources res; + struct mlx5_vdpa_mr_resources mres; u64 mlx_features; u64 actual_features; @@ -95,13 +103,8 @@ struct mlx5_vdpa_dev { u16 max_idx; u32 generation; - struct mlx5_vdpa_mr *mr[MLX5_VDPA_NUM_AS]; - struct list_head mr_list_head; - /* serialize mr access */ - struct mutex mr_mtx; struct mlx5_control_vq cvq; struct workqueue_struct *wq; - unsigned int group2asid[MLX5_VDPA_NUMVQ_GROUPS]; bool suspended; struct mlx5_async_ctx async_ctx; diff --git a/drivers/vdpa/mlx5/core/mr.c b/drivers/vdpa/mlx5/core/mr.c index 50bb2cc95ea2..95087d7ae78a 100644 --- a/drivers/vdpa/mlx5/core/mr.c +++ b/drivers/vdpa/mlx5/core/mr.c @@ -666,9 +666,9 @@ static void _mlx5_vdpa_put_mr(struct mlx5_vdpa_dev *mvdev, void mlx5_vdpa_put_mr(struct mlx5_vdpa_dev *mvdev, struct mlx5_vdpa_mr *mr) { - mutex_lock(&mvdev->mr_mtx); + mutex_lock(&mvdev->mres.mr_mtx); _mlx5_vdpa_put_mr(mvdev, mr); - mutex_unlock(&mvdev->mr_mtx); + mutex_unlock(&mvdev->mres.mr_mtx); } static void _mlx5_vdpa_get_mr(struct mlx5_vdpa_dev *mvdev, @@ -683,39 +683,39 @@ static void _mlx5_vdpa_get_mr(struct mlx5_vdpa_dev *mvdev, void mlx5_vdpa_get_mr(struct mlx5_vdpa_dev *mvdev, struct mlx5_vdpa_mr *mr) { - mutex_lock(&mvdev->mr_mtx); + mutex_lock(&mvdev->mres.mr_mtx); _mlx5_vdpa_get_mr(mvdev, mr); - mutex_unlock(&mvdev->mr_mtx); + mutex_unlock(&mvdev->mres.mr_mtx); } void mlx5_vdpa_update_mr(struct mlx5_vdpa_dev *mvdev, struct mlx5_vdpa_mr *new_mr, unsigned int asid) { - struct mlx5_vdpa_mr *old_mr = mvdev->mr[asid]; + struct mlx5_vdpa_mr *old_mr = mvdev->mres.mr[asid]; - mutex_lock(&mvdev->mr_mtx); + mutex_lock(&mvdev->mres.mr_mtx); _mlx5_vdpa_put_mr(mvdev, old_mr); - mvdev->mr[asid] = new_mr; + mvdev->mres.mr[asid] = new_mr; - mutex_unlock(&mvdev->mr_mtx); + mutex_unlock(&mvdev->mres.mr_mtx); } static void mlx5_vdpa_show_mr_leaks(struct mlx5_vdpa_dev *mvdev) { struct mlx5_vdpa_mr *mr; - mutex_lock(&mvdev->mr_mtx); + mutex_lock(&mvdev->mres.mr_mtx); - list_for_each_entry(mr, &mvdev->mr_list_head, mr_list) { + list_for_each_entry(mr, &mvdev->mres.mr_list_head, mr_list) { mlx5_vdpa_warn(mvdev, "mkey still alive after resource delete: " "mr: %p, mkey: 0x%x, refcount: %u\n", mr, mr->mkey, refcount_read(&mr->refcount)); } - mutex_unlock(&mvdev->mr_mtx); + mutex_unlock(&mvdev->mres.mr_mtx); } @@ -753,7 +753,7 @@ static int _mlx5_vdpa_create_mr(struct mlx5_vdpa_dev *mvdev, if (err) goto err_iotlb; - list_add_tail(&mr->mr_list, &mvdev->mr_list_head); + list_add_tail(&mr->mr_list, &mvdev->mres.mr_list_head); return 0; @@ -779,9 +779,9 @@ struct mlx5_vdpa_mr *mlx5_vdpa_create_mr(struct mlx5_vdpa_dev *mvdev, if (!mr) return ERR_PTR(-ENOMEM); - mutex_lock(&mvdev->mr_mtx); + mutex_lock(&mvdev->mres.mr_mtx); err = _mlx5_vdpa_create_mr(mvdev, mr, iotlb); - mutex_unlock(&mvdev->mr_mtx); + mutex_unlock(&mvdev->mres.mr_mtx); if (err) goto out_err; @@ -801,7 +801,7 @@ int mlx5_vdpa_update_cvq_iotlb(struct mlx5_vdpa_dev *mvdev, { int err; - if (mvdev->group2asid[MLX5_VDPA_CVQ_GROUP] != asid) + if (mvdev->mres.group2asid[MLX5_VDPA_CVQ_GROUP] != asid) return 0; spin_lock(&mvdev->cvq.iommu_lock); diff --git a/drivers/vdpa/mlx5/core/resources.c b/drivers/vdpa/mlx5/core/resources.c index 22ea32fe007b..3e3b3049cb08 100644 --- a/drivers/vdpa/mlx5/core/resources.c +++ b/drivers/vdpa/mlx5/core/resources.c @@ -256,7 +256,7 @@ int mlx5_vdpa_alloc_resources(struct mlx5_vdpa_dev *mvdev) mlx5_vdpa_warn(mvdev, "resources already allocated\n"); return -EINVAL; } - mutex_init(&mvdev->mr_mtx); + mutex_init(&mvdev->mres.mr_mtx); res->uar = mlx5_get_uars_page(mdev); if (IS_ERR(res->uar)) { err = PTR_ERR(res->uar); @@ -301,7 +301,7 @@ int mlx5_vdpa_alloc_resources(struct mlx5_vdpa_dev *mvdev) err_uctx: mlx5_put_uars_page(mdev, res->uar); err_uars: - mutex_destroy(&mvdev->mr_mtx); + mutex_destroy(&mvdev->mres.mr_mtx); return err; } @@ -318,7 +318,7 @@ void mlx5_vdpa_free_resources(struct mlx5_vdpa_dev *mvdev) dealloc_pd(mvdev, res->pdn, res->uid); destroy_uctx(mvdev, res->uid); mlx5_put_uars_page(mvdev->mdev, res->uar); - mutex_destroy(&mvdev->mr_mtx); + mutex_destroy(&mvdev->mres.mr_mtx); res->valid = false; } diff --git a/drivers/vdpa/mlx5/net/mlx5_vnet.c b/drivers/vdpa/mlx5/net/mlx5_vnet.c index cf2b77ebc72b..3e55a7f1afcd 100644 --- a/drivers/vdpa/mlx5/net/mlx5_vnet.c +++ b/drivers/vdpa/mlx5/net/mlx5_vnet.c @@ -941,11 +941,11 @@ static int create_virtqueue(struct mlx5_vdpa_net *ndev, MLX5_SET64(virtio_q, vq_ctx, used_addr, mvq->device_addr); MLX5_SET64(virtio_q, vq_ctx, available_addr, mvq->driver_addr); - vq_mr = mvdev->mr[mvdev->group2asid[MLX5_VDPA_DATAVQ_GROUP]]; + vq_mr = mvdev->mres.mr[mvdev->mres.group2asid[MLX5_VDPA_DATAVQ_GROUP]]; if (vq_mr) MLX5_SET(virtio_q, vq_ctx, virtio_q_mkey, vq_mr->mkey); - vq_desc_mr = mvdev->mr[mvdev->group2asid[MLX5_VDPA_DATAVQ_DESC_GROUP]]; + vq_desc_mr = mvdev->mres.mr[mvdev->mres.group2asid[MLX5_VDPA_DATAVQ_DESC_GROUP]]; if (vq_desc_mr && MLX5_CAP_DEV_VDPA_EMULATION(mvdev->mdev, desc_group_mkey_supported)) MLX5_SET(virtio_q, vq_ctx, desc_group_mkey, vq_desc_mr->mkey); @@ -953,11 +953,11 @@ static int create_virtqueue(struct mlx5_vdpa_net *ndev, /* If there is no mr update, make sure that the existing ones are set * modify to ready. */ - vq_mr = mvdev->mr[mvdev->group2asid[MLX5_VDPA_DATAVQ_GROUP]]; + vq_mr = mvdev->mres.mr[mvdev->mres.group2asid[MLX5_VDPA_DATAVQ_GROUP]]; if (vq_mr) mvq->modified_fields |= MLX5_VIRTQ_MODIFY_MASK_VIRTIO_Q_MKEY; - vq_desc_mr = mvdev->mr[mvdev->group2asid[MLX5_VDPA_DATAVQ_DESC_GROUP]]; + vq_desc_mr = mvdev->mres.mr[mvdev->mres.group2asid[MLX5_VDPA_DATAVQ_DESC_GROUP]]; if (vq_desc_mr) mvq->modified_fields |= MLX5_VIRTQ_MODIFY_MASK_DESC_GROUP_MKEY; } @@ -1354,7 +1354,7 @@ static void fill_modify_virtqueue_cmd(struct mlx5_vdpa_net *ndev, } if (mvq->modified_fields & MLX5_VIRTQ_MODIFY_MASK_VIRTIO_Q_MKEY) { - vq_mr = mvdev->mr[mvdev->group2asid[MLX5_VDPA_DATAVQ_GROUP]]; + vq_mr = mvdev->mres.mr[mvdev->mres.group2asid[MLX5_VDPA_DATAVQ_GROUP]]; if (vq_mr) MLX5_SET(virtio_q, vq_ctx, virtio_q_mkey, vq_mr->mkey); @@ -1363,7 +1363,7 @@ static void fill_modify_virtqueue_cmd(struct mlx5_vdpa_net *ndev, } if (mvq->modified_fields & MLX5_VIRTQ_MODIFY_MASK_DESC_GROUP_MKEY) { - desc_mr = mvdev->mr[mvdev->group2asid[MLX5_VDPA_DATAVQ_DESC_GROUP]]; + desc_mr = mvdev->mres.mr[mvdev->mres.group2asid[MLX5_VDPA_DATAVQ_DESC_GROUP]]; if (desc_mr && MLX5_CAP_DEV_VDPA_EMULATION(mvdev->mdev, desc_group_mkey_supported)) MLX5_SET(virtio_q, vq_ctx, desc_group_mkey, desc_mr->mkey); @@ -1381,8 +1381,8 @@ static void modify_virtqueue_end(struct mlx5_vdpa_net *ndev, struct mlx5_vdpa_dev *mvdev = &ndev->mvdev; if (mvq->modified_fields & MLX5_VIRTQ_MODIFY_MASK_VIRTIO_Q_MKEY) { - unsigned int asid = mvdev->group2asid[MLX5_VDPA_DATAVQ_GROUP]; - struct mlx5_vdpa_mr *vq_mr = mvdev->mr[asid]; + unsigned int asid = mvdev->mres.group2asid[MLX5_VDPA_DATAVQ_GROUP]; + struct mlx5_vdpa_mr *vq_mr = mvdev->mres.mr[asid]; mlx5_vdpa_put_mr(mvdev, mvq->vq_mr); mlx5_vdpa_get_mr(mvdev, vq_mr); @@ -1390,8 +1390,8 @@ static void modify_virtqueue_end(struct mlx5_vdpa_net *ndev, } if (mvq->modified_fields & MLX5_VIRTQ_MODIFY_MASK_DESC_GROUP_MKEY) { - unsigned int asid = mvdev->group2asid[MLX5_VDPA_DATAVQ_DESC_GROUP]; - struct mlx5_vdpa_mr *desc_mr = mvdev->mr[asid]; + unsigned int asid = mvdev->mres.group2asid[MLX5_VDPA_DATAVQ_DESC_GROUP]; + struct mlx5_vdpa_mr *desc_mr = mvdev->mres.mr[asid]; mlx5_vdpa_put_mr(mvdev, mvq->desc_mr); mlx5_vdpa_get_mr(mvdev, desc_mr); @@ -3235,7 +3235,7 @@ static void init_group_to_asid_map(struct mlx5_vdpa_dev *mvdev) /* default mapping all groups are mapped to asid 0 */ for (i = 0; i < MLX5_VDPA_NUMVQ_GROUPS; i++) - mvdev->group2asid[i] = 0; + mvdev->mres.group2asid[i] = 0; } static bool needs_vqs_reset(const struct mlx5_vdpa_dev *mvdev) @@ -3353,7 +3353,7 @@ static int set_map_data(struct mlx5_vdpa_dev *mvdev, struct vhost_iotlb *iotlb, new_mr = NULL; } - if (!mvdev->mr[asid]) { + if (!mvdev->mres.mr[asid]) { mlx5_vdpa_update_mr(mvdev, new_mr, asid); } else { err = mlx5_vdpa_change_map(mvdev, new_mr, asid); @@ -3637,12 +3637,12 @@ static int mlx5_set_group_asid(struct vdpa_device *vdev, u32 group, if (group >= MLX5_VDPA_NUMVQ_GROUPS) return -EINVAL; - mvdev->group2asid[group] = asid; + mvdev->mres.group2asid[group] = asid; - mutex_lock(&mvdev->mr_mtx); - if (group == MLX5_VDPA_CVQ_GROUP && mvdev->mr[asid]) - err = mlx5_vdpa_update_cvq_iotlb(mvdev, mvdev->mr[asid]->iotlb, asid); - mutex_unlock(&mvdev->mr_mtx); + mutex_lock(&mvdev->mres.mr_mtx); + if (group == MLX5_VDPA_CVQ_GROUP && mvdev->mres.mr[asid]) + err = mlx5_vdpa_update_cvq_iotlb(mvdev, mvdev->mres.mr[asid]->iotlb, asid); + mutex_unlock(&mvdev->mres.mr_mtx); return err; } @@ -3962,7 +3962,7 @@ static int mlx5_vdpa_dev_add(struct vdpa_mgmt_dev *v_mdev, const char *name, if (err) goto err_mpfs; - INIT_LIST_HEAD(&mvdev->mr_list_head); + INIT_LIST_HEAD(&mvdev->mres.mr_list_head); if (MLX5_CAP_GEN(mvdev->mdev, umem_uid_0)) { err = mlx5_vdpa_create_dma_mr(mvdev); From patchwork Fri Aug 30 10:58:36 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Dragos Tatulea X-Patchwork-Id: 13784921 Received: from NAM04-BN8-obe.outbound.protection.outlook.com (mail-bn8nam04on2089.outbound.protection.outlook.com [40.107.100.89]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B26FF19007F; Fri, 30 Aug 2024 10:59:37 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.100.89 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725015579; cv=fail; b=SoZOcqZslLPZLb78c1MnAGFlzcrGZCuF57PYsPV1vDwqwDJnyOV+JSyN+Or/43/U1rsjiFef8L4jSy73u8aa2JRuDzTcbJ+QmwZ2Z/udlRyf/h6QsIF2zsd3a98cMxmwP6zZNWPIHBTq+L5WtBnyyVtK8xMmd4mFnTovCoe3ugA= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725015579; c=relaxed/simple; bh=oUd524XZ03agGn80Re3RZoCUm1QgbxORyOfezmtmWes=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=h0dY7Wkknj67MDsZjQlRaPyZjpIScKA8tP0cw6MhaWsPY/mSwP7YMlKAq/y1ZLhfX9E5bRo5k0umftwO2woM0MtoZus9YQG6voWI/wuZlGzk6hBKJN5DQ7DaVSmyacCUMx6XFnq3FYWTZAkIsbJ8HRMqcMEOHrpyQjC6YXaMsrE= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=a4gil+eT; arc=fail smtp.client-ip=40.107.100.89 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="a4gil+eT" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=GNQw7nliBHKSpxs9MQ1iowM/SuNC1iQjV74cbuXTOqjY1l7gJTpUjQMKh1EGvV8H7XB6PlOTjV2Irwfb+2t2zttqJrb+L224qeJyL0cq4LPuJs/K+RKD133cRj2SuEu3hbb60OQeJXgMV1Nn6QbBIKWLeUluqda8vNn0Rjqe9JCkpV5/j0biGXA1uey8g2PSVldJn62U/hT2p4QVfDHI4uYdOZJ05hG8FPjCYFvGkaF2+HPfKE41GewglLKDOi47QEXyk23D3L8XCUgkjG7VsEnEZHV+0KiAJs3OlRDy1DEaeot38/MupYqS0wCAVYfSs7ANvPfELtgmU3ddN12aYQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=kHe6hjjzB+AuYFTfA0Obn2d8V3Yt/lz1NalluwisKig=; b=XCoGisIyJ4uGbHt/sRd5gDJ/EKIkAmyynnxEx9OKMmaxnu3o87HFMKS/fk7OYIsGP97YhLi/hMwaao2kD3GSZbMb9grpr6/KoUPZNOCmJGvB1oLy8C3BCec/XLEts2rXqgnAWMmegbTho7Qbe8GS8anuhWfxYa/J5FR91be83lWJ+0WHkhHaL+WL+mRjgq6xGb1zWzKfoBrRV4gX1jcf1yHgZhBvUbwT7eAxcts6uCpOd1oLj1/VjmBT1eyDdLL5o2QMAHlaEMWZN5hx13HEecCqNG5LAX1iCocL4IszMldDyN/0gVEcetrW9Ri+o3ihEDS7WvgRtjAcas3TDE+DEg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.160) smtp.rcpttodomain=redhat.com smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=kHe6hjjzB+AuYFTfA0Obn2d8V3Yt/lz1NalluwisKig=; b=a4gil+eTzAbIA7KKqA88w9V49nImnyEetbzzJ+1xYk9NNO60NS3pBkBOHA69+6aY1HKVxks/rT4V64lcjatVWCqwc0P0/5gtqwPrGw0trkFUUF+oHgRtyYwnD8jOYB3uSzLIaE3kifzqicrI1GMHoLtDOHc5rnNYBcDmL/cdSRhe9yWJN4W0aXbg4qU+7Pu1Ue29FlAEcTnRbzc+aano9kJeYu/9wb8TpB46iVwS5GBdc/Hv5AUOv1cnWlPNufR5qHtN1q4/3b1wVH9PiRtqRlJ72tcDH6M8lmSgTyAcK80vFHrD2hMxWH2LyerzwY1JLXFUMQILeKvnZqmDTpjKAA== Received: from CH0PR04CA0008.namprd04.prod.outlook.com (2603:10b6:610:76::13) by DS7PR12MB6312.namprd12.prod.outlook.com (2603:10b6:8:93::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7918.20; Fri, 30 Aug 2024 10:59:33 +0000 Received: from CH2PEPF0000009D.namprd02.prod.outlook.com (2603:10b6:610:76:cafe::1f) by CH0PR04CA0008.outlook.office365.com (2603:10b6:610:76::13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7918.20 via Frontend Transport; Fri, 30 Aug 2024 10:59:33 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.160) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.160 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.160; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.160) by CH2PEPF0000009D.mail.protection.outlook.com (10.167.244.25) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7918.13 via Frontend Transport; Fri, 30 Aug 2024 10:59:33 +0000 Received: from rnnvmail203.nvidia.com (10.129.68.9) by mail.nvidia.com (10.129.200.66) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Fri, 30 Aug 2024 03:59:17 -0700 Received: from rnnvmail204.nvidia.com (10.129.68.6) by rnnvmail203.nvidia.com (10.129.68.9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Fri, 30 Aug 2024 03:59:17 -0700 Received: from c-237-113-220-225.mtl.labs.mlnx (10.127.8.12) by mail.nvidia.com (10.129.68.6) with Microsoft SMTP Server id 15.2.1544.4 via Frontend Transport; Fri, 30 Aug 2024 03:59:14 -0700 From: Dragos Tatulea To: "Michael S . Tsirkin" , Jason Wang , Eugenio Perez Martin , Si-Wei Liu , , Gal Pressman CC: Dragos Tatulea , , , Parav Pandit , Xuan Zhuo , Cosmin Ratiu Subject: [PATCH vhost v2 5/7] vdpa/mlx5: Rename mr_mtx -> lock Date: Fri, 30 Aug 2024 13:58:36 +0300 Message-ID: <20240830105838.2666587-7-dtatulea@nvidia.com> X-Mailer: git-send-email 2.45.1 In-Reply-To: <20240830105838.2666587-2-dtatulea@nvidia.com> References: <20240830105838.2666587-2-dtatulea@nvidia.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH2PEPF0000009D:EE_|DS7PR12MB6312:EE_ X-MS-Office365-Filtering-Correlation-Id: 01dc9583-d321-40d1-1580-08dcc8e2d46d X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|36860700013|82310400026|1800799024|376014; X-Microsoft-Antispam-Message-Info: =?utf-8?q?iYVbb2gABAKwqRWh/Mk8xYA5Ex5wMW3?= =?utf-8?q?tvXgM2JiXYClfkeAnpGSlowjuIC/zIUcv5bXCg4kBct0tW/wubSY3+2kBxyQZvZRK?= =?utf-8?q?ejWDd0l8LzU3Pf/SlCgirYjBx4WnHq3zh77Pt5zoL9pAbr/Co4+tcbAJsA4SHJ19L?= =?utf-8?q?BHeWClmBebYE3VuG+1BaFbyDCFI4he6JAwKLLUofA9+NOTqJE2FgPKiKPB/5fQq1m?= =?utf-8?q?HBHdsxM5VoecvmDFRJSlplXgDCbvLNzxVvrftJmKw/Cdz0pWfoip2W4MGt/bBU89P?= =?utf-8?q?DMYOSg49Y4qW2O3dHOA4wblpM/VhC3q5qjtw9DfIgVbrBd0s5o3vNBBXtc/espI8A?= =?utf-8?q?LmFVQ7shXZXF0dhuKW65t2GY1qGoujkGr5iDK+ENKxxfGcsvABkqlOov8SBiaNG0e?= =?utf-8?q?ICie/907LW+hiK8Ec9O6vYtcTTxk4Prxj5WoWQiUQeW/J8bCs4kxhbSjqCMPLdwKs?= =?utf-8?q?5PZMOT79dOgsmsDeuThj3jfoDuWS33CQqxmRt6wVLlTNHYkyHc4il5VF3DGKmaKMv?= =?utf-8?q?lj+vRPu0V/JSoWrSTJLw8GqoJCUy4rb67vUDocCacz5eYAqc3xe3UaibWLZA/VYa0?= =?utf-8?q?URmgd2jh4y2T6+TEJFZQEJxX5x6UyaagVtk4/rASbr0VD0tfTINvRMyM/qW1sme+7?= =?utf-8?q?hqRpgynzT9xxEyFJLDbqkLQ8+idTEU7CEr/80B8k7S/5HuGGnWQfsSSWVy4ipYXBq?= =?utf-8?q?qVSEtrIiJrpdNS3rYNgEQ2Me9iBgbYswIyxQcp+mpbmJGubAoM5svq5eNUTE+tGeV?= =?utf-8?q?pY9qkEXubowvu6MpPzHfwDGqJl0c1Gh9QzUYgxedaz7wf4FlrNQpsFqUCTY71G/CL?= =?utf-8?q?M/DH1eGDgOIw9w+xmtwlGK1WtRcneTRXyJuXLYpffZcFqa07QNes0uREdYEgMaG/Y?= =?utf-8?q?G/F49F033kqfCUmEuYHzwXljsFkGfUfRpvJifNUAILartilPlGle9IOovRrv9gBKa?= =?utf-8?q?4G1grdLCNLp4u95HrDQNp3oK1nd+K+eioAkIRh/XziCIxq/NOMONkO8V/WGEsfmuO?= =?utf-8?q?InAQFaPsFiYORND8uxXUj8GnnUSro4+jcpNMApRpK45Fbzv2RtCFi8oeCWWbzIk0c?= =?utf-8?q?r+QTX+/Em8gKrvh1/TFTPSkgkxXfuu7QsdUVcGLxwsZU0HsViD3M22w0FoxcTF+3C?= =?utf-8?q?zLkMxqbeyRYHbdRoHOVY4Ea1/lmlPYkR1aG3JIFwi2AWrJV8cQXS9nYDYmKGARDS4?= =?utf-8?q?ogGwoWlhoe8cNRjM2L0uBDI9ksdlfpkicefHY4Zwht9HCszzBNKzSda/fuIJftcp+?= =?utf-8?q?lIh4f5AnFg2sro8NWyRH99U2k+2doShij4xqyCs5TQaBn10dq3Xvy9X6qUlHqfklA?= =?utf-8?q?J5YV/pWrvUDb2kcbIs/0XYO9uSYYF1UxzycYJKiVaGcOjCuRAbn+iBlA2Lz8qIJVe?= =?utf-8?q?fSxGC/9lFN5?= X-Forefront-Antispam-Report: CIP:216.228.117.160;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc6edge1.nvidia.com;CAT:NONE;SFS:(13230040)(36860700013)(82310400026)(1800799024)(376014);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 30 Aug 2024 10:59:33.0288 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 01dc9583-d321-40d1-1580-08dcc8e2d46d X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.117.160];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CH2PEPF0000009D.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS7PR12MB6312 Now that the mr resources have their own namespace in the struct, give the lock a clearer name. Signed-off-by: Dragos Tatulea Reviewed-by: Cosmin Ratiu Acked-by: Eugenio Pérez --- drivers/vdpa/mlx5/core/mlx5_vdpa.h | 2 +- drivers/vdpa/mlx5/core/mr.c | 20 ++++++++++---------- drivers/vdpa/mlx5/core/resources.c | 6 +++--- drivers/vdpa/mlx5/net/mlx5_vnet.c | 4 ++-- 4 files changed, 16 insertions(+), 16 deletions(-) diff --git a/drivers/vdpa/mlx5/core/mlx5_vdpa.h b/drivers/vdpa/mlx5/core/mlx5_vdpa.h index 5ae6deea2a8a..89b564cecddf 100644 --- a/drivers/vdpa/mlx5/core/mlx5_vdpa.h +++ b/drivers/vdpa/mlx5/core/mlx5_vdpa.h @@ -87,7 +87,7 @@ struct mlx5_vdpa_mr_resources { struct mlx5_vdpa_mr *mr[MLX5_VDPA_NUM_AS]; unsigned int group2asid[MLX5_VDPA_NUMVQ_GROUPS]; struct list_head mr_list_head; - struct mutex mr_mtx; + struct mutex lock; }; struct mlx5_vdpa_dev { diff --git a/drivers/vdpa/mlx5/core/mr.c b/drivers/vdpa/mlx5/core/mr.c index 95087d7ae78a..e0412297bae5 100644 --- a/drivers/vdpa/mlx5/core/mr.c +++ b/drivers/vdpa/mlx5/core/mr.c @@ -666,9 +666,9 @@ static void _mlx5_vdpa_put_mr(struct mlx5_vdpa_dev *mvdev, void mlx5_vdpa_put_mr(struct mlx5_vdpa_dev *mvdev, struct mlx5_vdpa_mr *mr) { - mutex_lock(&mvdev->mres.mr_mtx); + mutex_lock(&mvdev->mres.lock); _mlx5_vdpa_put_mr(mvdev, mr); - mutex_unlock(&mvdev->mres.mr_mtx); + mutex_unlock(&mvdev->mres.lock); } static void _mlx5_vdpa_get_mr(struct mlx5_vdpa_dev *mvdev, @@ -683,9 +683,9 @@ static void _mlx5_vdpa_get_mr(struct mlx5_vdpa_dev *mvdev, void mlx5_vdpa_get_mr(struct mlx5_vdpa_dev *mvdev, struct mlx5_vdpa_mr *mr) { - mutex_lock(&mvdev->mres.mr_mtx); + mutex_lock(&mvdev->mres.lock); _mlx5_vdpa_get_mr(mvdev, mr); - mutex_unlock(&mvdev->mres.mr_mtx); + mutex_unlock(&mvdev->mres.lock); } void mlx5_vdpa_update_mr(struct mlx5_vdpa_dev *mvdev, @@ -694,19 +694,19 @@ void mlx5_vdpa_update_mr(struct mlx5_vdpa_dev *mvdev, { struct mlx5_vdpa_mr *old_mr = mvdev->mres.mr[asid]; - mutex_lock(&mvdev->mres.mr_mtx); + mutex_lock(&mvdev->mres.lock); _mlx5_vdpa_put_mr(mvdev, old_mr); mvdev->mres.mr[asid] = new_mr; - mutex_unlock(&mvdev->mres.mr_mtx); + mutex_unlock(&mvdev->mres.lock); } static void mlx5_vdpa_show_mr_leaks(struct mlx5_vdpa_dev *mvdev) { struct mlx5_vdpa_mr *mr; - mutex_lock(&mvdev->mres.mr_mtx); + mutex_lock(&mvdev->mres.lock); list_for_each_entry(mr, &mvdev->mres.mr_list_head, mr_list) { @@ -715,7 +715,7 @@ static void mlx5_vdpa_show_mr_leaks(struct mlx5_vdpa_dev *mvdev) mr, mr->mkey, refcount_read(&mr->refcount)); } - mutex_unlock(&mvdev->mres.mr_mtx); + mutex_unlock(&mvdev->mres.lock); } @@ -779,9 +779,9 @@ struct mlx5_vdpa_mr *mlx5_vdpa_create_mr(struct mlx5_vdpa_dev *mvdev, if (!mr) return ERR_PTR(-ENOMEM); - mutex_lock(&mvdev->mres.mr_mtx); + mutex_lock(&mvdev->mres.lock); err = _mlx5_vdpa_create_mr(mvdev, mr, iotlb); - mutex_unlock(&mvdev->mres.mr_mtx); + mutex_unlock(&mvdev->mres.lock); if (err) goto out_err; diff --git a/drivers/vdpa/mlx5/core/resources.c b/drivers/vdpa/mlx5/core/resources.c index 3e3b3049cb08..fe2ca3458f6c 100644 --- a/drivers/vdpa/mlx5/core/resources.c +++ b/drivers/vdpa/mlx5/core/resources.c @@ -256,7 +256,7 @@ int mlx5_vdpa_alloc_resources(struct mlx5_vdpa_dev *mvdev) mlx5_vdpa_warn(mvdev, "resources already allocated\n"); return -EINVAL; } - mutex_init(&mvdev->mres.mr_mtx); + mutex_init(&mvdev->mres.lock); res->uar = mlx5_get_uars_page(mdev); if (IS_ERR(res->uar)) { err = PTR_ERR(res->uar); @@ -301,7 +301,7 @@ int mlx5_vdpa_alloc_resources(struct mlx5_vdpa_dev *mvdev) err_uctx: mlx5_put_uars_page(mdev, res->uar); err_uars: - mutex_destroy(&mvdev->mres.mr_mtx); + mutex_destroy(&mvdev->mres.lock); return err; } @@ -318,7 +318,7 @@ void mlx5_vdpa_free_resources(struct mlx5_vdpa_dev *mvdev) dealloc_pd(mvdev, res->pdn, res->uid); destroy_uctx(mvdev, res->uid); mlx5_put_uars_page(mvdev->mdev, res->uar); - mutex_destroy(&mvdev->mres.mr_mtx); + mutex_destroy(&mvdev->mres.lock); res->valid = false; } diff --git a/drivers/vdpa/mlx5/net/mlx5_vnet.c b/drivers/vdpa/mlx5/net/mlx5_vnet.c index 3e55a7f1afcd..8a51c492a62a 100644 --- a/drivers/vdpa/mlx5/net/mlx5_vnet.c +++ b/drivers/vdpa/mlx5/net/mlx5_vnet.c @@ -3639,10 +3639,10 @@ static int mlx5_set_group_asid(struct vdpa_device *vdev, u32 group, mvdev->mres.group2asid[group] = asid; - mutex_lock(&mvdev->mres.mr_mtx); + mutex_lock(&mvdev->mres.lock); if (group == MLX5_VDPA_CVQ_GROUP && mvdev->mres.mr[asid]) err = mlx5_vdpa_update_cvq_iotlb(mvdev, mvdev->mres.mr[asid]->iotlb, asid); - mutex_unlock(&mvdev->mres.mr_mtx); + mutex_unlock(&mvdev->mres.lock); return err; } From patchwork Fri Aug 30 10:58:37 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dragos Tatulea X-Patchwork-Id: 13784922 Received: from NAM12-BN8-obe.outbound.protection.outlook.com (mail-bn8nam12on2086.outbound.protection.outlook.com [40.107.237.86]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 812B1199FC1; Fri, 30 Aug 2024 10:59:37 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.237.86 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725015580; cv=fail; b=Hkya80ozpD2W3vjdjfep4u1dYlM5JOu2ITYTyuObEavPNb/wkC8Vzus073ed1ehSj4x/+RJZjYPVIf9JfUCMWF/S760DmRzH4tcJahKBtuKUw84j6e+q3Z9F31DCTa4e9H7X0yx9w9w5uGDwW8djSsWAi0lsJFu+NxA3Vx8FPeQ= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725015580; c=relaxed/simple; bh=/wwMlttLHDsVbK9xHunS0oIvPddiZWpVnj+K/dDI1KA=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=PHeAJ8is4krAXeto89m+sUp5fLdSywwT6MYSK9ZTnpUb+1B1ZAgm5aYDl8en/ZnHsrV0hWuRMrzc0u59jClBBUc3EhZAqIry1E0QwSQdjalG3eug0ljSIAk+SiI0Ga6HjuOqiTFWTsb/gFPNLQYYDfI9/4EhvBBp/nbREr1L7oA= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=S9SSCK9e; arc=fail smtp.client-ip=40.107.237.86 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="S9SSCK9e" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=P6F6eGqEyil4hS50nKp6H9knWVEWSaXOg4iO6OgmQI5dqffWlbV0J4mHhneE4ZtE9zhFAdgiND6HENU/q+xxNdL4evTEF00qyLlXPJAHnZXB5RwMKx3zZddbxnxX6SgR0HbvhjIpDOu4uIM2Dv5koTAYnKhneoagmzEiC2oGuVLtZFZv0ijS4Yv/30iglmvL4LEBJcQejKd+mChVnc7ZzenyULbGmecJBUCPtggBszvFpvmu6qK5spvWj3JOgkm/FGxhgYw4CaeeKCr8jUaYjJDKWPfE1x2O1OA0ncSuy2Ttvx5YaDSr62ZLZJZHJP29vle8ZsMO2Oud/ynvN0G8vA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=iU7Aq2rGDrL2GGGmhNzGacTGEuvMlo1x3owaHepnepA=; b=x5Il4ytPXV3lnUGQgMfvvKo9BbcwZvS6DDWGvbTEs/65IBqtwiPXu16SZIpb6kyve7W3eiiFBCim2TlwKfQE1O8vw1Es85ZSO0JbXDx0cTKrutJtuse5XyWQ3MnLr/dkCmEzExuQeVmQeQ24pcs4deulxpWgWFxJcoNCr2SfY90AyQ/5Wz72Y/X7bfFWeEJ7K8pz28ZV7ICojzlmXXl/JMgBXfdAbDgBtgBGNgL1R80gKtbq0SJvdKrLbefGFpHG4HlpwcXjSFU5v6kxB0Zo8b5CiEaymSZOl4I0vJa0zJ+ky5oRaI3PuK3vTjqo8kmidvlP7IL5f2ACJCfD2GhRbw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.161) smtp.rcpttodomain=redhat.com smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=iU7Aq2rGDrL2GGGmhNzGacTGEuvMlo1x3owaHepnepA=; b=S9SSCK9eg06casC8KAimRyd+VoSK/cIAFcfR1n07UlPtPhk9S6z//6DFLyjS8IHJdoKXrmDrmg/glfclL0dF0kXd9a+r1r9ZRUQ7hXFMDvCsbfOuKnkgaaa3jcnwIOYhZO6Jbprj8h9IeriLkCCRcouY+vgNJQo3ZY+VKsSGVsGqDATWoziqDdEDKrmoQ6pUZeD6frqP/2xFr5PZUfyDwc35GUEPLLc+xYH7irT0jvsQAaL0nfM0kMHpGxWgi967863/xN3173N64b/2Hm0p86LBAPl0V5RG7Yu8MXRcFisbmsOT8dB8mCvHRIxWgAUqRHXbPeGStSbA9s9R7FpozQ== Received: from BN8PR03CA0030.namprd03.prod.outlook.com (2603:10b6:408:94::43) by IA1PR12MB7735.namprd12.prod.outlook.com (2603:10b6:208:421::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7875.20; Fri, 30 Aug 2024 10:59:34 +0000 Received: from BN2PEPF000044AC.namprd04.prod.outlook.com (2603:10b6:408:94:cafe::55) by BN8PR03CA0030.outlook.office365.com (2603:10b6:408:94::43) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7897.28 via Frontend Transport; Fri, 30 Aug 2024 10:59:34 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.161) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.161 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.161; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.161) by BN2PEPF000044AC.mail.protection.outlook.com (10.167.243.107) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7918.13 via Frontend Transport; Fri, 30 Aug 2024 10:59:33 +0000 Received: from rnnvmail203.nvidia.com (10.129.68.9) by mail.nvidia.com (10.129.200.67) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Fri, 30 Aug 2024 03:59:21 -0700 Received: from rnnvmail204.nvidia.com (10.129.68.6) by rnnvmail203.nvidia.com (10.129.68.9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Fri, 30 Aug 2024 03:59:20 -0700 Received: from c-237-113-220-225.mtl.labs.mlnx (10.127.8.12) by mail.nvidia.com (10.129.68.6) with Microsoft SMTP Server id 15.2.1544.4 via Frontend Transport; Fri, 30 Aug 2024 03:59:17 -0700 From: Dragos Tatulea To: "Michael S . Tsirkin" , Jason Wang , Eugenio Perez Martin , Si-Wei Liu , , Gal Pressman CC: Dragos Tatulea , , , Parav Pandit , Xuan Zhuo , Cosmin Ratiu Subject: [PATCH vhost v2 6/7] vdpa/mlx5: Introduce init/destroy for MR resources Date: Fri, 30 Aug 2024 13:58:37 +0300 Message-ID: <20240830105838.2666587-8-dtatulea@nvidia.com> X-Mailer: git-send-email 2.45.1 In-Reply-To: <20240830105838.2666587-2-dtatulea@nvidia.com> References: <20240830105838.2666587-2-dtatulea@nvidia.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BN2PEPF000044AC:EE_|IA1PR12MB7735:EE_ X-MS-Office365-Filtering-Correlation-Id: e7e60480-b286-4bef-7e22-08dcc8e2d500 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|1800799024|36860700013|82310400026; X-Microsoft-Antispam-Message-Info: AsJcyAW2sUnzy90Ba/QnUkav9ARmhiwLpQsd2LvVArQkeIb6/4Tkx9ncV2Y10Gu6Ymf2JixaRlaUeegLDtV1n13wrKuyztyZvxk3aJBMYKyxm/Zk3XEyO3nQlmhOeDwovBs0++oDISMOdkAetWSGzjf7LSijrQgUl3WAZOka7EkfPtuAeq5ZIPNYSG1Xy+sHSGLpNsgHoGoyUO188uMksn4Fcy5bGkbySNxo/gE48EnzM8r7aF+MFudLpBlRr5eyVwOELzp1O9OwukdEQo9ct88JYyYMMrqB8EICWgSaYAq0GSZP2rlpoEHjYxaVfWfIeVlvBeN0KeaK7UAVS1IoO98m68r22tiwkWLSN9avG0ZpVD1YeDL5BksYOzEcfOuIk7IC+V48fsmcQQ/n1Ed7DuAbl5BPOFJ8GZ7+gNztH6EryT3Kufs9l13sOEVzEiKHClTRb128ujsejFaN7VthSfgD3koNICmed8z319ixnDdmrLZDRqdmMaJ0kpFC2phLalRjCSIcz3alQ4Bu7XFDA0Tf5Br1TOz1Gp6ZZtog6Fn6ZedfjfhRkcaxfz5PZmrfVzCTOg7VLsgU4P1smP9jUZ8HC1K6vglwdi1dj4SEHfNp8M/zlXQ1DwFDwRv0i213ZYoGD5dQJvWyVaGCISDOKEhoDQfmV9bZZLEh1udx6+BSoDjxJ71WWraCC0SRhMrLDUVLy1HuOaBVL24Y+THGCGsy4ovawYDPSTGUt0WZf/iRjDlAMOVFKvBsdop8GS114eEj4zA3/Kn76892gKuHzljGFIf2R1yyfkQOJx97qcDfatmRYD8eRfU5nnlhvJltcv1AZNYLqHYkSBumrjL9dTzRZ+AHrrdH/Zt/hHdEdr9kF4OVkxh5ZkKEi8ez1dSJ7v1XAtcqdIDWE+Yg6aobTEaRq1Hwu16Q9Igjw5sv4Hd3tu9XPFkmJ162HrhnfWKad0chV6j4zQx7FYDvsjpm4qnqFWnO6n+K7NBJ05RuPTx+ZXrg4bZmwnQ7lj7LnCo39GlPlyxqT3ncIlZ27Skrlfw4qLwH5d7NYQCo0HwD1kyiWy9eHt7snIlpk7KYiIebuSCXkEI3n8kASn6TMGSY0fH8Cds2DO6On6QawIUt4LNfBnHEdceQHHBqUIIgqqdM6cbLEQ+pLT5TCfSgihfogHiuk3O6wPKflaNaYa0WpEqTUsZ03r1rh8GI9+VzCGe7Q0WP086ZJXKIjY1BMtYx+N0ea/XGGma4BgFOBzbSbfLfbiepE5c08qLDG1Hd/ROVqK+nwQoIwhQfpAblL2nf6A1AuDfrYaFM/HzBOQtaU6TLZwSgsaIaSz4EWTv7wUit43ItXyHmlouTIOYAhxLCmiqnN3HdA3+jrF/nMnXMEPrHY72k7tuPtYMevjpu0QL096/Ze5rz2hpqnsEbvMl+MM+U4MTSE3bh/8Sio3/ZmvVo/IL7+qakmbi8UeBYx4pT X-Forefront-Antispam-Report: CIP:216.228.117.161;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc6edge2.nvidia.com;CAT:NONE;SFS:(13230040)(376014)(1800799024)(36860700013)(82310400026);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 30 Aug 2024 10:59:33.9479 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: e7e60480-b286-4bef-7e22-08dcc8e2d500 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.117.161];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BN2PEPF000044AC.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA1PR12MB7735 There's currently not a lot of action happening during the init/destroy of MR resources. But more will be added in the upcoming patches. As the mr mutex lock init/destroy has been moved to these new functions, the lifetime has now shifted away from mlx5_vdpa_alloc_resources() / mlx5_vdpa_free_resources() into these new functions. However, the lifetime at the outer scope remains the same: mlx5_vdpa_dev_add() / mlx5_vdpa_dev_free() Signed-off-by: Dragos Tatulea Reviewed-by: Cosmin Ratiu --- drivers/vdpa/mlx5/core/mlx5_vdpa.h | 2 ++ drivers/vdpa/mlx5/core/mr.c | 17 +++++++++++++++++ drivers/vdpa/mlx5/core/resources.c | 3 --- drivers/vdpa/mlx5/net/mlx5_vnet.c | 9 +++++++-- 4 files changed, 26 insertions(+), 5 deletions(-) diff --git a/drivers/vdpa/mlx5/core/mlx5_vdpa.h b/drivers/vdpa/mlx5/core/mlx5_vdpa.h index 89b564cecddf..c3e17bc888e8 100644 --- a/drivers/vdpa/mlx5/core/mlx5_vdpa.h +++ b/drivers/vdpa/mlx5/core/mlx5_vdpa.h @@ -138,6 +138,8 @@ int mlx5_vdpa_create_mkey(struct mlx5_vdpa_dev *mvdev, u32 *mkey, u32 *in, int mlx5_vdpa_destroy_mkey(struct mlx5_vdpa_dev *mvdev, u32 mkey); struct mlx5_vdpa_mr *mlx5_vdpa_create_mr(struct mlx5_vdpa_dev *mvdev, struct vhost_iotlb *iotlb); +int mlx5_vdpa_init_mr_resources(struct mlx5_vdpa_dev *mvdev); +void mlx5_vdpa_destroy_mr_resources(struct mlx5_vdpa_dev *mvdev); void mlx5_vdpa_clean_mrs(struct mlx5_vdpa_dev *mvdev); void mlx5_vdpa_get_mr(struct mlx5_vdpa_dev *mvdev, struct mlx5_vdpa_mr *mr); diff --git a/drivers/vdpa/mlx5/core/mr.c b/drivers/vdpa/mlx5/core/mr.c index e0412297bae5..0bc99f159046 100644 --- a/drivers/vdpa/mlx5/core/mr.c +++ b/drivers/vdpa/mlx5/core/mr.c @@ -843,3 +843,20 @@ int mlx5_vdpa_reset_mr(struct mlx5_vdpa_dev *mvdev, unsigned int asid) return 0; } + +int mlx5_vdpa_init_mr_resources(struct mlx5_vdpa_dev *mvdev) +{ + struct mlx5_vdpa_mr_resources *mres = &mvdev->mres; + + INIT_LIST_HEAD(&mres->mr_list_head); + mutex_init(&mres->lock); + + return 0; +} + +void mlx5_vdpa_destroy_mr_resources(struct mlx5_vdpa_dev *mvdev) +{ + struct mlx5_vdpa_mr_resources *mres = &mvdev->mres; + + mutex_destroy(&mres->lock); +} diff --git a/drivers/vdpa/mlx5/core/resources.c b/drivers/vdpa/mlx5/core/resources.c index fe2ca3458f6c..aeae31d0cefa 100644 --- a/drivers/vdpa/mlx5/core/resources.c +++ b/drivers/vdpa/mlx5/core/resources.c @@ -256,7 +256,6 @@ int mlx5_vdpa_alloc_resources(struct mlx5_vdpa_dev *mvdev) mlx5_vdpa_warn(mvdev, "resources already allocated\n"); return -EINVAL; } - mutex_init(&mvdev->mres.lock); res->uar = mlx5_get_uars_page(mdev); if (IS_ERR(res->uar)) { err = PTR_ERR(res->uar); @@ -301,7 +300,6 @@ int mlx5_vdpa_alloc_resources(struct mlx5_vdpa_dev *mvdev) err_uctx: mlx5_put_uars_page(mdev, res->uar); err_uars: - mutex_destroy(&mvdev->mres.lock); return err; } @@ -318,7 +316,6 @@ void mlx5_vdpa_free_resources(struct mlx5_vdpa_dev *mvdev) dealloc_pd(mvdev, res->pdn, res->uid); destroy_uctx(mvdev, res->uid); mlx5_put_uars_page(mvdev->mdev, res->uar); - mutex_destroy(&mvdev->mres.lock); res->valid = false; } diff --git a/drivers/vdpa/mlx5/net/mlx5_vnet.c b/drivers/vdpa/mlx5/net/mlx5_vnet.c index 8a51c492a62a..fc86e33e620a 100644 --- a/drivers/vdpa/mlx5/net/mlx5_vnet.c +++ b/drivers/vdpa/mlx5/net/mlx5_vnet.c @@ -3434,6 +3434,7 @@ static void mlx5_vdpa_free(struct vdpa_device *vdev) free_fixed_resources(ndev); mlx5_vdpa_clean_mrs(mvdev); + mlx5_vdpa_destroy_mr_resources(&ndev->mvdev); if (!is_zero_ether_addr(ndev->config.mac)) { pfmdev = pci_get_drvdata(pci_physfn(mvdev->mdev->pdev)); mlx5_mpfs_del_mac(pfmdev, ndev->config.mac); @@ -3962,12 +3963,14 @@ static int mlx5_vdpa_dev_add(struct vdpa_mgmt_dev *v_mdev, const char *name, if (err) goto err_mpfs; - INIT_LIST_HEAD(&mvdev->mres.mr_list_head); + err = mlx5_vdpa_init_mr_resources(mvdev); + if (err) + goto err_res; if (MLX5_CAP_GEN(mvdev->mdev, umem_uid_0)) { err = mlx5_vdpa_create_dma_mr(mvdev); if (err) - goto err_res; + goto err_mr_res; } err = alloc_fixed_resources(ndev); @@ -4009,6 +4012,8 @@ static int mlx5_vdpa_dev_add(struct vdpa_mgmt_dev *v_mdev, const char *name, free_fixed_resources(ndev); err_mr: mlx5_vdpa_clean_mrs(mvdev); +err_mr_res: + mlx5_vdpa_destroy_mr_resources(mvdev); err_res: mlx5_vdpa_free_resources(&ndev->mvdev); err_mpfs: From patchwork Fri Aug 30 10:58:38 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dragos Tatulea X-Patchwork-Id: 13784923 Received: from NAM10-DM6-obe.outbound.protection.outlook.com (mail-dm6nam10on2045.outbound.protection.outlook.com [40.107.93.45]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 551141A287B; Fri, 30 Aug 2024 10:59:43 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.93.45 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725015585; cv=fail; b=kS9KHDc6cbqUvo8LGkl7+IuwqaAVjt6EKiDGqxTDkvVTA4dpAmLb/MvKkgfNQe0gphOrnXQ4nRU0B7zZjX608p4Rgcj6JKK/OZDi39MoC0OQHRFaOYxIqKJq0227SM40dmXZ8p8/M+mlYAfEg55pJltCPCQxZmiV+A4IEygNZUg= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725015585; c=relaxed/simple; bh=7yOK9FcltC82mZCU7xfVnl3LqAgRYMy5Wc/ISPyR8HQ=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=tm7C6OjCgiRaCyFGAZ4lEa6mieEvs3ms561LMF7BZLUR7iOXo87s40+hgthq6TE4sz53A+ifTFTkn2AJar9onlr6sErAuMCe7A0NPkIKbtdAib3i99e/nZX0ieEWrU362y5HE0+BqHCC/ytm/F3WkyBhqhB0CkoBBdS0aItoijM= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=Zy3lpdRZ; arc=fail smtp.client-ip=40.107.93.45 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="Zy3lpdRZ" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=UQtZlL4ZHpcrrw561gjANG7FkI/XKDhX0uK4QLRd3MSv6fGNDsoHyO+56USvyIIc3aUtHVkwKx2mXsjZlc1q4vxIqFmhqBo6ygEOTfacv3ViweWFQy9u0WK0v48/RvPWqe/i+1SwA4Q9vGkl0//KXk/iL1UQ9aFv10/ZgXGEk158MTwBW9i72A+f0Q8cfQfuIvb5wAqvldc6rJnydd/ZSRNrGQoDz+PvQbGnKmGC/hTzKIsSFWk9cm2FzqncBBqbvhOAb6OdbHILSa2YHQnSH94vTeYqASvGrUHyQ7IGEQj8iemHKhzuaZdwIEDUfobtXIsbO+mM77KIfsF425mIEw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=481MRpxXPZFGiUq/tv3DyRO8i2ssxDsL/9RPJ6HgHJ4=; b=LVwJOkawHiJRs2h3wSv3xIZt9bDN5V6+WLjTVOgBLSEVURGYloka+cOJZojNeRePVZNCgo/2M7snBy3tpXUr7dKdPJ4zAyqvukJaZAFMRbamArbn9SxPAt90GYhXg0drIDTzFjkZMBspsbERrurC4ntrHJ+tZBDRcJskGlLQjt+Lr1uq+S72vvNbbAq2dgFzSs4q35xDuzUi/aQ4/gDwAlPJTvtL2GQG6VKKPrlOazAoxI2phamUnL93QWDCKCdff/982UVFDVYslHeASpgrsymU0Ho/xd3PFm8VKydr7WVjOjwSj0hQ8r/HH4z4+ABypcZ/pk9S4TxzxWsd0SclnA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.161) smtp.rcpttodomain=redhat.com smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=481MRpxXPZFGiUq/tv3DyRO8i2ssxDsL/9RPJ6HgHJ4=; b=Zy3lpdRZH0ULTJhkY33lkV7+ohdeo58MdeS/b1hONGCKhLLnmQK9cEZ1CyKuJHkcks3tf6Aus6rsPizB/RD9iRKuQZXHYaB8beNW1vQFO7NdqFCEHOHGy9lRUPkmV5eVEd7FHhNe5WndV0QfaD9o7OcFyqpRe2z0aQT+2PVr+ajLGXn296/fvWzpQAx7fs3MkblPm3RXrSBqU0a7pHkS9B2IioOyWAmx85TW6yvgLhu9ev4cwGnhyQjOO8rbWAOPnb9F4cQzejCV+YaleoAbuO56g3vIaHmRm8KetFmEw1XizC/lCxFsMfrToWYxcz16+EctjjYoKAevSqypoOziFg== Received: from BN9PR03CA0459.namprd03.prod.outlook.com (2603:10b6:408:139::14) by SA1PR12MB7200.namprd12.prod.outlook.com (2603:10b6:806:2bb::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7897.25; Fri, 30 Aug 2024 10:59:39 +0000 Received: from BN2PEPF000044A5.namprd04.prod.outlook.com (2603:10b6:408:139:cafe::4) by BN9PR03CA0459.outlook.office365.com (2603:10b6:408:139::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7918.20 via Frontend Transport; Fri, 30 Aug 2024 10:59:38 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.161) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.161 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.161; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.161) by BN2PEPF000044A5.mail.protection.outlook.com (10.167.243.104) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7918.13 via Frontend Transport; Fri, 30 Aug 2024 10:59:38 +0000 Received: from rnnvmail203.nvidia.com (10.129.68.9) by mail.nvidia.com (10.129.200.67) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Fri, 30 Aug 2024 03:59:24 -0700 Received: from rnnvmail204.nvidia.com (10.129.68.6) by rnnvmail203.nvidia.com (10.129.68.9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Fri, 30 Aug 2024 03:59:24 -0700 Received: from c-237-113-220-225.mtl.labs.mlnx (10.127.8.12) by mail.nvidia.com (10.129.68.6) with Microsoft SMTP Server id 15.2.1544.4 via Frontend Transport; Fri, 30 Aug 2024 03:59:21 -0700 From: Dragos Tatulea To: "Michael S . Tsirkin" , Jason Wang , Eugenio Perez Martin , Si-Wei Liu , , Gal Pressman CC: Dragos Tatulea , , , Parav Pandit , Xuan Zhuo , Cosmin Ratiu Subject: [PATCH vhost v2 7/7] vdpa/mlx5: Postpone MR deletion Date: Fri, 30 Aug 2024 13:58:38 +0300 Message-ID: <20240830105838.2666587-9-dtatulea@nvidia.com> X-Mailer: git-send-email 2.45.1 In-Reply-To: <20240830105838.2666587-2-dtatulea@nvidia.com> References: <20240830105838.2666587-2-dtatulea@nvidia.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BN2PEPF000044A5:EE_|SA1PR12MB7200:EE_ X-MS-Office365-Filtering-Correlation-Id: af183d05-a7eb-4083-d25f-08dcc8e2d7b7 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|1800799024|36860700013|82310400026; X-Microsoft-Antispam-Message-Info: XF8k4ariKkj64iHhDO66FsSjOwiHW7dtVXnHFcmvW+oXfBSyAk3HAMPKuCzzDUizF7qcTSOt9QPRyMXeWHYapbCtOWOEabMKq0Jp6u/mZ2UQpIhu2YNlEU/MqeHmtVY1r0oxdoTqnxDPUdrdqXkyar6ljkbBK2h+V86eNeKuRDdPaHRZZteL9CjxpvIFcCwMyInmWDSij1G+Gk/N07DQwtpNATa/ZP7eml0/y4/bi4qcgpR3lzNzu3X6QvSWIHSRTeYGdpfzUtdYDsuAGgAe7BrAzlxgG7GaKFa0lazGC0OG71TBX6NyDg5H9yIraWSfu09NJOPTUA9SaG/zYDBsgU1jHduosmlW7xosFUc8t+hmGWKGhhKqovQHUDsSiZ7JLzySwQ7gdC4VK48Nv74pYmdwnwdva5I2+cHG9fV7FikRDB5p5YZR8c+MIP1TBRMhbRVuz2AeNOoJSziRhlK+AUJUl7LB8X4LR2yack85FoFNSl3+78XmjDzpdgt9gKAK/Uv8zs2eEbXQJNXlxU/+Usg6Zu6AMS3+Gp9iMFNM9MXyJAISdqwVFKJQxJbLTmEFb6zxzBwCZxdZ5wWh8iWpj1KpR1Z8+nXQYrCmKhkGRSKO++QtY0iR7YDgtZC2vR/8Z1iWTy/ebpr9YiKuRix+e/O49vZKdxxfn2JeVOcR/QoDsP5YlpTBrsF+nibTfmIAvZcRqDK9rf+iZe99dGvImvRPZVgwq4zYDv3YQG0UbYkrO5JoJdvDuXRqhp/BMTkl/nnjfKJjehZgqAzq5SELrsZ4gZ6YjjVnkDxuCLmkuC5dHgSCWepYGQGS0TpPZbSv3nV41eM2I2nV6Y5ckwp7bHt7AtDYj5JxQHyWzcPddjabxU51IjGR/AzhDxmBPSaQl5OqxX/+Otgj6DF0i7+2IfXYNwn8oF5L/h8TzGIGRUGE3dY60SMw30NwvSxu4WDDltrxJ4m+9nWXSzFsr+sMzEFFV1iK7Rsphk3pO0bYAixB1Z6uIO69741CzuIGpZE9ZUjN+pz7oPCcffLkR/bNYa9ZUo+aT6c+FKmQy8ZBh4NyGO2pnflOdzb4vYurF6WlMtxvuJ9xj/in86d6SI6NiCXNRYl3fthiuI+o6cbemyAFN0SIcMIrWTFKD2wwic0zLuG69fPM/sCWaTdn/4jQFOSOu3FuUiLvGwwvyzvfy0GdE0NTWCkAz/Nkn3L/DKlIjulRrgooXN7h50+bEq7cye7E+5JwzjFMm293D2dfhpZuihPyqueu0wfq8a4aKy94kBn+LBe1Vjvv+0kTCb+Kqj1m9cRMMmhTGfU3BdaS/5bUL30kMYdfY6hTLiJxShPWJqTifC1FpnKUkw+3X4tD/NjznO+JWJHvh6BWjE9Pw5oklYvCN4ANckew4wa+ge3M/XhcAMyYaohBqRWS/u9f0qkKNQD1NUwJTVelyHNVn8At7lBtquiyjpwDQmMOq1fg X-Forefront-Antispam-Report: CIP:216.228.117.161;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc6edge2.nvidia.com;CAT:NONE;SFS:(13230040)(376014)(1800799024)(36860700013)(82310400026);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 30 Aug 2024 10:59:38.5049 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: af183d05-a7eb-4083-d25f-08dcc8e2d7b7 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.117.161];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BN2PEPF000044A5.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA1PR12MB7200 Currently, when a new MR is set up, the old MR is deleted. MR deletion is about 30-40% the time of MR creation. As deleting the old MR is not important for the process of setting up the new MR, this operation can be postponed. This series adds a workqueue that does MR garbage collection at a later point. If the MR lock is taken, the handler will back off and reschedule. The exception during shutdown: then the handler must not postpone the work. Note that this is only a speculative optimization: if there is some mapping operation that is triggered while the garbage collector handler has the lock taken, this operation it will have to wait for the handler to finish. Signed-off-by: Dragos Tatulea Reviewed-by: Cosmin Ratiu --- drivers/vdpa/mlx5/core/mlx5_vdpa.h | 10 ++++++ drivers/vdpa/mlx5/core/mr.c | 55 ++++++++++++++++++++++++++++-- drivers/vdpa/mlx5/net/mlx5_vnet.c | 4 +-- 3 files changed, 64 insertions(+), 5 deletions(-) diff --git a/drivers/vdpa/mlx5/core/mlx5_vdpa.h b/drivers/vdpa/mlx5/core/mlx5_vdpa.h index c3e17bc888e8..2cedf7e2dbc4 100644 --- a/drivers/vdpa/mlx5/core/mlx5_vdpa.h +++ b/drivers/vdpa/mlx5/core/mlx5_vdpa.h @@ -86,8 +86,18 @@ enum { struct mlx5_vdpa_mr_resources { struct mlx5_vdpa_mr *mr[MLX5_VDPA_NUM_AS]; unsigned int group2asid[MLX5_VDPA_NUMVQ_GROUPS]; + + /* Pre-deletion mr list */ struct list_head mr_list_head; + + /* Deferred mr list */ + struct list_head mr_gc_list_head; + struct workqueue_struct *wq_gc; + struct delayed_work gc_dwork_ent; + struct mutex lock; + + atomic_t shutdown; }; struct mlx5_vdpa_dev { diff --git a/drivers/vdpa/mlx5/core/mr.c b/drivers/vdpa/mlx5/core/mr.c index 0bc99f159046..55755e97a946 100644 --- a/drivers/vdpa/mlx5/core/mr.c +++ b/drivers/vdpa/mlx5/core/mr.c @@ -653,14 +653,50 @@ static void _mlx5_vdpa_destroy_mr(struct mlx5_vdpa_dev *mvdev, struct mlx5_vdpa_ kfree(mr); } +/* There can be multiple .set_map() operations in quick succession. + * This large delay is a simple way to prevent the MR cleanup from blocking + * .set_map() MR creation in this scenario. + */ +#define MLX5_VDPA_MR_GC_TRIGGER_MS 2000 + +static void mlx5_vdpa_mr_gc_handler(struct work_struct *work) +{ + struct mlx5_vdpa_mr_resources *mres; + struct mlx5_vdpa_mr *mr, *tmp; + struct mlx5_vdpa_dev *mvdev; + + mres = container_of(work, struct mlx5_vdpa_mr_resources, gc_dwork_ent.work); + + if (atomic_read(&mres->shutdown)) { + mutex_lock(&mres->lock); + } else if (!mutex_trylock(&mres->lock)) { + queue_delayed_work(mres->wq_gc, &mres->gc_dwork_ent, + msecs_to_jiffies(MLX5_VDPA_MR_GC_TRIGGER_MS)); + return; + } + + mvdev = container_of(mres, struct mlx5_vdpa_dev, mres); + + list_for_each_entry_safe(mr, tmp, &mres->mr_gc_list_head, mr_list) { + _mlx5_vdpa_destroy_mr(mvdev, mr); + } + + mutex_unlock(&mres->lock); +} + static void _mlx5_vdpa_put_mr(struct mlx5_vdpa_dev *mvdev, struct mlx5_vdpa_mr *mr) { + struct mlx5_vdpa_mr_resources *mres = &mvdev->mres; + if (!mr) return; - if (refcount_dec_and_test(&mr->refcount)) - _mlx5_vdpa_destroy_mr(mvdev, mr); + if (refcount_dec_and_test(&mr->refcount)) { + list_move_tail(&mr->mr_list, &mres->mr_gc_list_head); + queue_delayed_work(mres->wq_gc, &mres->gc_dwork_ent, + msecs_to_jiffies(MLX5_VDPA_MR_GC_TRIGGER_MS)); + } } void mlx5_vdpa_put_mr(struct mlx5_vdpa_dev *mvdev, @@ -848,9 +884,17 @@ int mlx5_vdpa_init_mr_resources(struct mlx5_vdpa_dev *mvdev) { struct mlx5_vdpa_mr_resources *mres = &mvdev->mres; - INIT_LIST_HEAD(&mres->mr_list_head); + mres->wq_gc = create_singlethread_workqueue("mlx5_vdpa_mr_gc"); + if (!mres->wq_gc) + return -ENOMEM; + + INIT_DELAYED_WORK(&mres->gc_dwork_ent, mlx5_vdpa_mr_gc_handler); + mutex_init(&mres->lock); + INIT_LIST_HEAD(&mres->mr_list_head); + INIT_LIST_HEAD(&mres->mr_gc_list_head); + return 0; } @@ -858,5 +902,10 @@ void mlx5_vdpa_destroy_mr_resources(struct mlx5_vdpa_dev *mvdev) { struct mlx5_vdpa_mr_resources *mres = &mvdev->mres; + atomic_set(&mres->shutdown, 1); + + flush_delayed_work(&mres->gc_dwork_ent); + destroy_workqueue(mres->wq_gc); + mres->wq_gc = NULL; mutex_destroy(&mres->lock); } diff --git a/drivers/vdpa/mlx5/net/mlx5_vnet.c b/drivers/vdpa/mlx5/net/mlx5_vnet.c index fc86e33e620a..9ccbe1c1ec15 100644 --- a/drivers/vdpa/mlx5/net/mlx5_vnet.c +++ b/drivers/vdpa/mlx5/net/mlx5_vnet.c @@ -3435,6 +3435,8 @@ static void mlx5_vdpa_free(struct vdpa_device *vdev) free_fixed_resources(ndev); mlx5_vdpa_clean_mrs(mvdev); mlx5_vdpa_destroy_mr_resources(&ndev->mvdev); + mlx5_cmd_cleanup_async_ctx(&mvdev->async_ctx); + if (!is_zero_ether_addr(ndev->config.mac)) { pfmdev = pci_get_drvdata(pci_physfn(mvdev->mdev->pdev)); mlx5_mpfs_del_mac(pfmdev, ndev->config.mac); @@ -4042,8 +4044,6 @@ static void mlx5_vdpa_dev_del(struct vdpa_mgmt_dev *v_mdev, struct vdpa_device * mvdev->wq = NULL; destroy_workqueue(wq); mgtdev->ndev = NULL; - - mlx5_cmd_cleanup_async_ctx(&mvdev->async_ctx); } static const struct vdpa_mgmtdev_ops mdev_ops = {