From patchwork Sun Jan 15 12:58:14 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michael Guralnik X-Patchwork-Id: 13102233 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id F3FBAC46467 for ; Sun, 15 Jan 2023 12:58:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231175AbjAOM6g (ORCPT ); Sun, 15 Jan 2023 07:58:36 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53260 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229879AbjAOM6c (ORCPT ); Sun, 15 Jan 2023 07:58:32 -0500 Received: from NAM11-CO1-obe.outbound.protection.outlook.com (mail-co1nam11on2060.outbound.protection.outlook.com [40.107.220.60]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4E98B10A90 for ; Sun, 15 Jan 2023 04:58:31 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=QtArInQzXkQfSKI5fw3dQFaV5gx4DgY+ZkTyW0wJYOtcdtDATnGQKIgha+4jhpk5qeAX7pUmC6Liv80Gt5U+WacQKJGGkExNHCdBI6YUCiG31cWMXRbdNWCNBX2ySjoiyTkYrQJmaOICnKw4V/0OY7ZC7oagk2d7oP3hPw+q3aYfynprcCsPcvhmwmWzBBHUEUwy7jKmrDbkFFX3URl30ullqfUIYsvuS8dTSL9n1CljnVvsWQ4eJjpSyvtjf/Pr8QRRJ6uvZhsJtV94VmS7z2xFgwPlKQcDTb2zn8pacz/KoygTLpWSinm3J4PncdEYYM5ucpYEY19JRgq6wFsX7g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=TO/WvqZmz5Hi+nBLiQbD8NKYwtBwDUokf0Zm5rX+yTA=; b=m344sATKhhaIgU5aENP0QpMaNq7sbxJsrUNcsQL5cfN3BraaRXo3c0gyJFRUG0Nw/v8vAxYRRze50Sz1U6IL/q0CaQNzlSNRtpfKFTvHixBtPju/P+Xph5ZIruiNvyyZbIQcDwz2v4QVY7aUeTjkERnt5uOW3r3MRWnTlqSh4gCJuYEUFFGkT0NhO3PdmIByJqiCijg2uUmUH0ibtVpkxLLeaINHTl5TxLjf1p5RKEMt/7NUsgHi3IJa2ILrEGNe2KEbw08w2yOiMYdX8i+4kssqjYSMBQGBCh9liyhzQdiE7wHscdbKxPOj3dsCy7fg/MaUZc7gDaAQuz3mUDIFHQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.118.232) smtp.rcpttodomain=vger.kernel.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=TO/WvqZmz5Hi+nBLiQbD8NKYwtBwDUokf0Zm5rX+yTA=; b=HAWX6DuA1jFcGwscadJn6rt4LRfsU+Aybm7MSeUaUWxQKGq3oppL4fxJqwlgZYmw0t2OJThWFD+LPBWxpPRHspKY+G0Y8RBF3Ipwj+euzPXuQHDD+ByKBwslQS8rK5B9YXUndSkDGMtTjR0gSCHgay05EG5zeNXn6vdBE1BO29dGI0Z6BzM1y97dRyynHPo2srG8hfVm2YgtmUc/vd1TZhA7q4lBnAdYGJa0XN/4V9Q18iwzoaQJdYp/JVFif7e6YZzhlCKHlqEyaav4wJGpCJdcwYvZ9mt7d//pPbE56oz+FUeRX8UfvG3cdx8C+T/HBqYAdpXMnLUKC5jNFEpYsA== Received: from MW4PR03CA0287.namprd03.prod.outlook.com (2603:10b6:303:b5::22) by PH7PR12MB8122.namprd12.prod.outlook.com (2603:10b6:510:2b7::19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6002.13; Sun, 15 Jan 2023 12:58:30 +0000 Received: from CO1NAM11FT066.eop-nam11.prod.protection.outlook.com (2603:10b6:303:b5:cafe::5) by MW4PR03CA0287.outlook.office365.com (2603:10b6:303:b5::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6002.19 via Frontend Transport; Sun, 15 Jan 2023 12:58:29 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.118.232) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.118.232 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.118.232; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.118.232) by CO1NAM11FT066.mail.protection.outlook.com (10.13.175.18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6002.13 via Frontend Transport; Sun, 15 Jan 2023 12:58:29 +0000 Received: from drhqmail201.nvidia.com (10.126.190.180) by mail.nvidia.com (10.127.129.5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.36; Sun, 15 Jan 2023 04:58:27 -0800 Received: from drhqmail202.nvidia.com (10.126.190.181) by drhqmail201.nvidia.com (10.126.190.180) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.36; Sun, 15 Jan 2023 04:58:27 -0800 Received: from vdi.nvidia.com (10.127.8.13) by mail.nvidia.com (10.126.190.181) with Microsoft SMTP Server id 15.2.986.36 via Frontend Transport; Sun, 15 Jan 2023 04:58:25 -0800 From: Michael Guralnik To: , , CC: , Subject: [PATCH v3 rdma-next 1/6] RDMA/mlx5: Don't keep umrable 'page_shift' in cache entries Date: Sun, 15 Jan 2023 14:58:14 +0200 Message-ID: <20230115125819.15034-2-michaelgur@nvidia.com> X-Mailer: git-send-email 2.17.2 In-Reply-To: <20230115125819.15034-1-michaelgur@nvidia.com> References: <20230115125819.15034-1-michaelgur@nvidia.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CO1NAM11FT066:EE_|PH7PR12MB8122:EE_ X-MS-Office365-Filtering-Correlation-Id: c4d89393-d580-455d-4393-08daf6f832fc X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: urdrNQESzuw2KNDGB/hMAGITj61EjI8rHw4/ZIb/3bbXNwxEeL6K/6tkuNaGgcgU8e0LfRdicbItAjmWIgcxySAMzqjsAaNnGJ1roCd0HJNKza6p7mFyNNkA5rOngUlc+CxRA2iZpIzWemOdaAlAEdGSOPWXbmxCzKP0SDZ8tdNlagivvpMcGbZHnL3LbIlIzKIaFd93f1icBGJS4RZ1VTUncLOTS1BzkQTWeRqM0PDMY0Ci8FUyTpSBu5ART9gv/y/x/eCEuB1E8yxY6aAavjqIX6kOmu1YrvhsJteKs+iu421m2QssbzpJD9TyPQm75QoyKsWdhwHS2Zze0p6p1Bj3P+u5KxsQoIJdYo4P53rJqKe54KWXyA8ysD0aQm+d0lB6VqtyK4J0ZPJpmMl4JfxAM95U9kSXbUEBDSGN+c9SiManMn6tdU3lNQ/F/Cl46YC4q6AL8khlCiagkP3JoVHn3wSFeC00/XcURKoYYcKchbJwfBWEWHdZrSpeHiK2Arl4+jtOiUQx0oow0BzSLQTwWbIdWu1vsPxS0v/vRz3sE/jIKY6sVU03Y82Ns6aAQs7zrlyQSGRp+159t18In+bvtmwZxt2NnXe8dvL/HsmxLMq8unsQgqLLarXf32BmZQfIPcUMv9pm1hrPqBi9+YWB9gs2j3qxqUA6e0jJN0LiT0vnAanhWPgzEOGUqquX9lLJP+R7xKDdr/TEdQKCGlT7REDZpNweZgbOMoyBAic= X-Forefront-Antispam-Report: CIP:216.228.118.232;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc7edge1.nvidia.com;CAT:NONE;SFS:(13230022)(4636009)(39860400002)(376002)(346002)(136003)(396003)(451199015)(36840700001)(40470700004)(46966006)(426003)(478600001)(82740400003)(7636003)(41300700001)(356005)(47076005)(1076003)(316002)(54906003)(110136005)(40460700003)(2616005)(86362001)(26005)(40480700001)(336012)(186003)(70586007)(7696005)(70206006)(82310400005)(5660300002)(107886003)(36756003)(36860700001)(2906002)(6666004)(8676002)(83380400001)(4326008)(8936002)(2101003);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 15 Jan 2023 12:58:29.2968 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: c4d89393-d580-455d-4393-08daf6f832fc X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.118.232];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CO1NAM11FT066.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR12MB8122 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org From: Aharon Landau mkc.log_page_size can be changed using UMR. Therefore, don't treat it as a cache entry property. Removing it from struct mlx5_cache_ent. All cache mkeys will be created with default PAGE_SHIFT, and updated with the needed page_shift using UMR when passing them to a user. Issue: 2404431 Change-Id: I0e6cd6dd4eb1ef99d3373f98ef519d54a7bd568e Signed-off-by: Aharon Landau --- drivers/infiniband/hw/mlx5/mlx5_ib.h | 1 - drivers/infiniband/hw/mlx5/mr.c | 3 +-- drivers/infiniband/hw/mlx5/odp.c | 2 -- 3 files changed, 1 insertion(+), 5 deletions(-) diff --git a/drivers/infiniband/hw/mlx5/mlx5_ib.h b/drivers/infiniband/hw/mlx5/mlx5_ib.h index 8b91babdd4c0..8d985f792367 100644 --- a/drivers/infiniband/hw/mlx5/mlx5_ib.h +++ b/drivers/infiniband/hw/mlx5/mlx5_ib.h @@ -739,7 +739,6 @@ struct mlx5_cache_ent { char name[4]; u32 order; u32 access_mode; - u32 page; unsigned int ndescs; u8 disabled:1; diff --git a/drivers/infiniband/hw/mlx5/mr.c b/drivers/infiniband/hw/mlx5/mr.c index 053fe946e45a..356c99d7ec9a 100644 --- a/drivers/infiniband/hw/mlx5/mr.c +++ b/drivers/infiniband/hw/mlx5/mr.c @@ -297,7 +297,7 @@ static void set_cache_mkc(struct mlx5_cache_ent *ent, void *mkc) MLX5_SET(mkc, mkc, translations_octword_size, get_mkc_octo_size(ent->access_mode, ent->ndescs)); - MLX5_SET(mkc, mkc, log_page_size, ent->page); + MLX5_SET(mkc, mkc, log_page_size, PAGE_SHIFT); } /* Asynchronously schedule new MRs to be populated in the cache. */ @@ -765,7 +765,6 @@ int mlx5_mkey_cache_init(struct mlx5_ib_dev *dev) if (ent->order > mkey_cache_max_order(dev)) continue; - ent->page = PAGE_SHIFT; ent->ndescs = 1 << ent->order; ent->access_mode = MLX5_MKC_ACCESS_MODE_MTT; if ((dev->mdev->profile.mask & MLX5_PROF_MASK_MR_CACHE) && diff --git a/drivers/infiniband/hw/mlx5/odp.c b/drivers/infiniband/hw/mlx5/odp.c index e6e021af6aa9..8a78580a2a72 100644 --- a/drivers/infiniband/hw/mlx5/odp.c +++ b/drivers/infiniband/hw/mlx5/odp.c @@ -1594,14 +1594,12 @@ void mlx5_odp_init_mkey_cache_entry(struct mlx5_cache_ent *ent) switch (ent->order - 2) { case MLX5_IMR_MTT_CACHE_ENTRY: - ent->page = PAGE_SHIFT; ent->ndescs = MLX5_IMR_MTT_ENTRIES; ent->access_mode = MLX5_MKC_ACCESS_MODE_MTT; ent->limit = 0; break; case MLX5_IMR_KSM_CACHE_ENTRY: - ent->page = MLX5_KSM_PAGE_SHIFT; ent->ndescs = mlx5_imr_ksm_entries; ent->access_mode = MLX5_MKC_ACCESS_MODE_KSM; ent->limit = 0; From patchwork Sun Jan 15 12:58:15 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michael Guralnik X-Patchwork-Id: 13102234 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 701A6C3DA78 for ; Sun, 15 Jan 2023 12:58:40 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229907AbjAOM6i (ORCPT ); Sun, 15 Jan 2023 07:58:38 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53276 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229890AbjAOM6h (ORCPT ); Sun, 15 Jan 2023 07:58:37 -0500 Received: from NAM10-DM6-obe.outbound.protection.outlook.com (mail-dm6nam10on2073.outbound.protection.outlook.com [40.107.93.73]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1E10B10AB5 for ; Sun, 15 Jan 2023 04:58:37 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=maGCwK7wZqweYL4tvtAPpWW4e+ErAqDBpMSqLSyQHOGDYjuzNf5u84+VM93PcB/F6BgYgi9p1MMNDCm+7cCEl+hXYu5Jp1IMb2ZJ5clNyk7QZLWIOoc2DvdCESE50YeiVMgVRmhEca7OzmwaZMtUFNjkEhFHLL3z+otvdJBEvGVm9QNq1wa5n/OwoKk3ZWD8yMq5yHkqUnMbqF2KFJw4Qx5e/KyZbdKqsieTGJKaPIfUhqdWaw4lOXG8Qo2AQe0OI01ymd0l+RLUTaSX0hv6MUblmsHXvkY8gOB/VRct/0/Ofe6vqc/DCyqOyxVdvJJIyeaGvuc0tbDL5HfdJyLnbw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=PUesdKKbTncYfIQpUexXLBu26X8TYxq1AJPQI06kYlU=; b=g0ig8CSrSz+xhEfWZRisjSTyadX29OQkbIiIQ1ZMoQrHdtoPGp/sLh8C1KYM2fMG4R3mRGfUc6Z8fAcgDkAmz7/idXjlGRthI+yp8NuO94ryivth4yMqNT0JwwXpI/Bp7SeLviYE6M24W9SudckU03+aIVpiTbRTjXRfaBGjvrJuGVoAWTH4Jmov9hslGn3g7doZcHFuPY8CYjkYOUOAy6nDysq9aXNfIsQAuGIuPt+0x+XpSm9hOp/auLTMCnRSbq3VpMKmP5MIDFdeIPL9M2+UQ2Ov3nJ8OjSiOZxzGTXGoVDMRHyhhFDQ/WEmt+aNOj8CwZ+mMTXit/CrI8poRg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.118.233) smtp.rcpttodomain=vger.kernel.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=PUesdKKbTncYfIQpUexXLBu26X8TYxq1AJPQI06kYlU=; b=uIWlkVr2AYcb7duvan3FEQVWhNB+5r3Bq4TLFMMIHmCni8W5dx1FCW3MZag6TCum7nePJppbHFWEBjoCQbEmmCYm2ogs0D6BQA01fBDO+ppwwAtc3/J45ui70r/UBMOR3MoMQr58eaOyBQFwsgY/2LYYtEt/4Lj0Om4YGTbYGCpD0Oaliz5zStHhRzzripxPB1YyQbbKk/86KhED6DZEwELiWMRCqqK8gWT5wyamYXSfjUDpzA2krvif1HK86tKGShPgDx9FKTB/K9etrDjieKkG/tHaOPIlU+H4QEg7fxiiUP683gcSFFJp73q/rKZgFbTeqkh7ZQUy4pnOeB13ew== Received: from BL1P221CA0020.NAMP221.PROD.OUTLOOK.COM (2603:10b6:208:2c5::7) by DS0PR12MB7748.namprd12.prod.outlook.com (2603:10b6:8:130::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5986.19; Sun, 15 Jan 2023 12:58:35 +0000 Received: from BL02EPF000108EB.namprd05.prod.outlook.com (2603:10b6:208:2c5:cafe::bc) by BL1P221CA0020.outlook.office365.com (2603:10b6:208:2c5::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6002.16 via Frontend Transport; Sun, 15 Jan 2023 12:58:35 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.118.233) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.118.233 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.118.233; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.118.233) by BL02EPF000108EB.mail.protection.outlook.com (10.167.241.200) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6002.11 via Frontend Transport; Sun, 15 Jan 2023 12:58:35 +0000 Received: from drhqmail203.nvidia.com (10.126.190.182) by mail.nvidia.com (10.127.129.6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.36; Sun, 15 Jan 2023 04:58:29 -0800 Received: from drhqmail202.nvidia.com (10.126.190.181) by drhqmail203.nvidia.com (10.126.190.182) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.36; Sun, 15 Jan 2023 04:58:29 -0800 Received: from vdi.nvidia.com (10.127.8.13) by mail.nvidia.com (10.126.190.181) with Microsoft SMTP Server id 15.2.986.36 via Frontend Transport; Sun, 15 Jan 2023 04:58:27 -0800 From: Michael Guralnik To: , , CC: , Subject: [PATCH v3 rdma-next 2/6] RDMA/mlx5: Remove explicit ODP cache entry Date: Sun, 15 Jan 2023 14:58:15 +0200 Message-ID: <20230115125819.15034-3-michaelgur@nvidia.com> X-Mailer: git-send-email 2.17.2 In-Reply-To: <20230115125819.15034-1-michaelgur@nvidia.com> References: <20230115125819.15034-1-michaelgur@nvidia.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BL02EPF000108EB:EE_|DS0PR12MB7748:EE_ X-MS-Office365-Filtering-Correlation-Id: 2c9a0ac4-d690-4b2a-b8cf-08daf6f83688 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: uJ83bfJz58m+jPn6xfXdSvQMBoaGgu4W+lxCA3fPZRoyXKvQ04WGqh0qMDvEL9CM1QXxz1SREkpaGTKRGqyf8/KQs9ldM24H0FVI+hioMbsKUjvq631ZtNpOcq+1WhcTaQlNSq0COFmMr/TYU7HedJ5f+hInzFIommhoSQwo+Wm3CaBmNbTwwUd9sWf4r/4mkMp68xraI6QIqYEvDHJUiGYkYXOBp30CpY+0eEftB1/XgsFrIMAWyFTvZHfN/iwS54bwPQVf6xFp/U75oyrMpZQxMzflElbugbYjcoZ2PM4UxKFbOdLRCkxPd7qauxrRXUxxk+iYQvSV9B/xsZ0Kr2f76xPMXNYjyy+FIzqXjqEuq98Rc7RDFU9LfYCaplUjSI9jfQErQWa16tELQC/grJH8JRFaInqqh6oKlJHVRsbH3ofGXAMNOdjKKg9WcvxJTmrCgkCIoofpK+mFPLfb7KyNRMkRYen94kUq/aDNmOatVkcv9WKBlud8b38aON+1/1Il3Dd32HkQsIwoP73mn+qCm3KFO32UOwVHYdID+4Idtu27WDTavz53qTwh3LPfk0+HwcZZm9y8/rnKPYsR6GkJTCJRAYH2ugbnONMskP14ZrkEOmQ8vVxPd2acVk0rNKr/gGE4rdO3Ekm+d+kmfLgCWesJLGbhxBIn+gkb9U5Kw/dRrjmp8RJrbd9dkqB6J2tLVxdXXDF1vb0dRZbdWBhwmhhVc2kROPtxt1g0Sl0= X-Forefront-Antispam-Report: CIP:216.228.118.233;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc7edge2.nvidia.com;CAT:NONE;SFS:(13230022)(4636009)(376002)(136003)(396003)(39860400002)(346002)(451199015)(36840700001)(46966006)(40470700004)(36756003)(356005)(2906002)(7636003)(86362001)(5660300002)(8936002)(40460700003)(82310400005)(41300700001)(40480700001)(36860700001)(47076005)(426003)(83380400001)(82740400003)(54906003)(110136005)(478600001)(8676002)(316002)(7696005)(336012)(6666004)(2616005)(4326008)(107886003)(70586007)(1076003)(186003)(70206006)(26005)(2101003);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 15 Jan 2023 12:58:35.1723 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 2c9a0ac4-d690-4b2a-b8cf-08daf6f83688 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.118.233];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BL02EPF000108EB.namprd05.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS0PR12MB7748 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org From: Aharon Landau Explicit ODP mkey doesn't have unique properties. It shares the same properties as the order 18 cache entry. There is no need to devote a special entry for that. Issue: 2404431 Change-Id: Iedc9a56d1161b0ef24311151107c05a8c76c3518 Signed-off-by: Aharon Landau --- drivers/infiniband/hw/mlx5/odp.c | 20 +++++--------------- include/linux/mlx5/driver.h | 1 - 2 files changed, 5 insertions(+), 16 deletions(-) diff --git a/drivers/infiniband/hw/mlx5/odp.c b/drivers/infiniband/hw/mlx5/odp.c index 8a78580a2a72..72044f8ec883 100644 --- a/drivers/infiniband/hw/mlx5/odp.c +++ b/drivers/infiniband/hw/mlx5/odp.c @@ -405,6 +405,7 @@ static void mlx5_ib_page_fault_resume(struct mlx5_ib_dev *dev, static struct mlx5_ib_mr *implicit_get_child_mr(struct mlx5_ib_mr *imr, unsigned long idx) { + int order = order_base_2(MLX5_IMR_MTT_ENTRIES); struct mlx5_ib_dev *dev = mr_to_mdev(imr); struct ib_umem_odp *odp; struct mlx5_ib_mr *mr; @@ -417,7 +418,8 @@ static struct mlx5_ib_mr *implicit_get_child_mr(struct mlx5_ib_mr *imr, if (IS_ERR(odp)) return ERR_CAST(odp); - mr = mlx5_mr_cache_alloc(dev, &dev->cache.ent[MLX5_IMR_MTT_CACHE_ENTRY], + BUILD_BUG_ON(order > MKEY_CACHE_LAST_STD_ENTRY); + mr = mlx5_mr_cache_alloc(dev, &dev->cache.ent[order], imr->access_flags); if (IS_ERR(mr)) { ib_umem_odp_release(odp); @@ -1591,20 +1593,8 @@ void mlx5_odp_init_mkey_cache_entry(struct mlx5_cache_ent *ent) { if (!(ent->dev->odp_caps.general_caps & IB_ODP_SUPPORT_IMPLICIT)) return; - - switch (ent->order - 2) { - case MLX5_IMR_MTT_CACHE_ENTRY: - ent->ndescs = MLX5_IMR_MTT_ENTRIES; - ent->access_mode = MLX5_MKC_ACCESS_MODE_MTT; - ent->limit = 0; - break; - - case MLX5_IMR_KSM_CACHE_ENTRY: - ent->ndescs = mlx5_imr_ksm_entries; - ent->access_mode = MLX5_MKC_ACCESS_MODE_KSM; - ent->limit = 0; - break; - } + ent->ndescs = mlx5_imr_ksm_entries; + ent->access_mode = MLX5_MKC_ACCESS_MODE_KSM; } static const struct ib_device_ops mlx5_ib_dev_odp_ops = { diff --git a/include/linux/mlx5/driver.h b/include/linux/mlx5/driver.h index d476255c9a3f..f79c20d50eb4 100644 --- a/include/linux/mlx5/driver.h +++ b/include/linux/mlx5/driver.h @@ -734,7 +734,6 @@ enum { enum { MKEY_CACHE_LAST_STD_ENTRY = 20, - MLX5_IMR_MTT_CACHE_ENTRY, MLX5_IMR_KSM_CACHE_ENTRY, MAX_MKEY_CACHE_ENTRIES }; From patchwork Sun Jan 15 12:58:16 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michael Guralnik X-Patchwork-Id: 13102235 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 92A3AC3DA78 for ; Sun, 15 Jan 2023 12:58:44 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229879AbjAOM6n (ORCPT ); Sun, 15 Jan 2023 07:58:43 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53316 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229893AbjAOM6m (ORCPT ); Sun, 15 Jan 2023 07:58:42 -0500 Received: from NAM11-CO1-obe.outbound.protection.outlook.com (mail-co1nam11on2066.outbound.protection.outlook.com [40.107.220.66]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3490E10AB8 for ; Sun, 15 Jan 2023 04:58:41 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=mW5/enF5uxXg9/6RNrReEup3R7owg/pOTonf2xaIo7Nn94XTt2gTnDs95xrpXO1xzipwdGdTBwHXl9xuoLpRWuOlwQD84cQ58i61qcb1ar0QrRbBwr1JqYpXLO18DXCFVRLpsT/yKWioKdesAX7p4l1A01j/nnJj9XGrhokJrvd5ZCSddxsYW8B1gy6gsUimuq5wouQ51pebke93mNaf/HV2OlCHUP/RXIUVZ6UcHWGBu6WHOPNIPFfSumQw2ekIcaRxTyS5pRMM+gBT8752PTEC0zQTssOYFugBtztHD3rEgtrHZy1oIGUK8xJY/9/anfaH7DWBtatR+RrdyKK97A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=81n2WoFxnd1ZN247q4pty7rJPtCWIde6RuH22lMmMDA=; b=DDXZbsgcdxZoOVq1J/jJMjUiGCjLQ7oTwQ2Kh4NvNDX/LQxUMLZI/qVg0NUoQNOMOVAkJ5Q0NPti5Wcr6ltMMaOF7YwFhGTPPNnNKHw/heNNv5ilDqxpJElUfIvcSeAQ/sILtm2eZg3AShaxC29gFXMSaKaVRFp+DfxNX9fxLoZqdU1NozyWDleB6RuIwG/LGjrufPlwxy5NMFPtASqGTda36cotPfbvZLm7VVWvIlZV9BBZqioH7fpEAjfsHhgBMATaLd7Q2Y8WKi69l7wUPZBFLopcfUAlTX7yzOeD9kTVol8/gpTUwmSIy55LmUNM0smVaeYnOadTCbdtlZYHTA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.118.232) smtp.rcpttodomain=vger.kernel.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=81n2WoFxnd1ZN247q4pty7rJPtCWIde6RuH22lMmMDA=; b=C/DxqbKLcXNgmqC1euAazUI3f89xcg+R15qrLKVw+ufnbbrwGO945n45OuHhSxMgHf7ajx36Ktwtq0TOrANyHIWkjAUmGZ96huQ2ngNK95nPKA6Qf5rw/gguL8CAau0px8wZMqTcMpRZBrL3ywD8J7BlBY7fh55YVd0tpIzhR0nHONEmzoyOztmEvvauTbbPlrjAa+J1mXmjuqEiDGbizFb7CKhQpsPeTXPbF3qaCyW8L6rHPizqjPHwf5b+rpI3Y+YsdbLcCNMielB8dw3I4heKU613lQiU3iU6FX56qKEOvsFuUyHeFnE2ZxqGKlyIT3/qw85PDTw3JBgUBiAb1w== Received: from MW4PR04CA0331.namprd04.prod.outlook.com (2603:10b6:303:8a::6) by LV2PR12MB5895.namprd12.prod.outlook.com (2603:10b6:408:173::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5986.23; Sun, 15 Jan 2023 12:58:39 +0000 Received: from CO1NAM11FT040.eop-nam11.prod.protection.outlook.com (2603:10b6:303:8a:cafe::f5) by MW4PR04CA0331.outlook.office365.com (2603:10b6:303:8a::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6002.19 via Frontend Transport; Sun, 15 Jan 2023 12:58:39 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.118.232) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.118.232 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.118.232; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.118.232) by CO1NAM11FT040.mail.protection.outlook.com (10.13.174.140) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6002.13 via Frontend Transport; Sun, 15 Jan 2023 12:58:38 +0000 Received: from drhqmail203.nvidia.com (10.126.190.182) by mail.nvidia.com (10.127.129.5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.36; Sun, 15 Jan 2023 04:58:31 -0800 Received: from drhqmail202.nvidia.com (10.126.190.181) by drhqmail203.nvidia.com (10.126.190.182) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.36; Sun, 15 Jan 2023 04:58:31 -0800 Received: from vdi.nvidia.com (10.127.8.13) by mail.nvidia.com (10.126.190.181) with Microsoft SMTP Server id 15.2.986.36 via Frontend Transport; Sun, 15 Jan 2023 04:58:29 -0800 From: Michael Guralnik To: , , CC: , , Michael Guralnik Subject: [PATCH v3 rdma-next 3/6] RDMA/mlx5: Change the cache structure to an RB-tree Date: Sun, 15 Jan 2023 14:58:16 +0200 Message-ID: <20230115125819.15034-4-michaelgur@nvidia.com> X-Mailer: git-send-email 2.17.2 In-Reply-To: <20230115125819.15034-1-michaelgur@nvidia.com> References: <20230115125819.15034-1-michaelgur@nvidia.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CO1NAM11FT040:EE_|LV2PR12MB5895:EE_ X-MS-Office365-Filtering-Correlation-Id: c490e25c-8c44-47d6-bfa7-08daf6f838b6 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: jPQ7sUvXBgqIbbgbSQrlPKSW94jJ2Ydf0efGeS/ig2jEdnJizQUq1Cf74euEyKmwdAqIvipY/k2WvMZeVhVciOmOBt3dDoS/2PxnPSbf/dXoTOd8uv8yPZnWp9PEywD6mMsjR3K7mgc/oDRo9sG0vrj0Nb0EcrUlV887sckZC94KY0g5AxxA01PRkPZx7b8JoU6WkVXdYHpukLIxDQSgsXeBTSdCIbbvldyE6Id+z7GEKyCXkfxSZDVR4DyD2/K5AELwKKSLKcdWN3y4chKyqRex31cVZpQMqYjkfCfLL7U7aodMosZyLSr/nF5T/ti7njx1eNBCENk16eInMsHGUX/yU2nSO+tP9vEnF8heQEXDKNuUlmLLN2OLDJd9jAAPmfiB9hLhBB8J6CU1xCJbZZoOlyAtwl4NtUWw7iY3Jbw9FjcXFmKahw9SzK9rNsOyr1hGMlRJNEwgoWl2UVsYVTfvyKartILmnoyHJOIJA3BMlxQMacG23bpqJ/PqF1csDx90zBJRfL2tcmvAB2F+9vJZdSoaJOVLpQYYrPzkau37Nkn5FnXDtNhxOYNRIJrVd60pUUMtEMxnNT3HfOTqHesbggZf6r8SXe1ynb2pYK7XQJpFOQRxDIN0mt05K00joMiq0RTX+DfneK1DVQRseytW5EI4kwpxijJPBrFS12T1RzS5kMTos9ljhQ74P6aZpgem6QxBPCp01NtnyeJ8D8TZGJRX6ctypPthTtW9knM= X-Forefront-Antispam-Report: CIP:216.228.118.232;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc7edge1.nvidia.com;CAT:NONE;SFS:(13230022)(4636009)(376002)(39860400002)(346002)(136003)(396003)(451199015)(40470700004)(46966006)(36840700001)(36756003)(356005)(86362001)(4326008)(8936002)(70586007)(70206006)(8676002)(30864003)(2906002)(5660300002)(36860700001)(82740400003)(7636003)(83380400001)(40460700003)(478600001)(7696005)(110136005)(54906003)(6666004)(316002)(40480700001)(82310400005)(41300700001)(2616005)(1076003)(47076005)(107886003)(426003)(336012)(26005)(186003)(2101003);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 15 Jan 2023 12:58:38.9232 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: c490e25c-8c44-47d6-bfa7-08daf6f838b6 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.118.232];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CO1NAM11FT040.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: LV2PR12MB5895 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org Currently, the cache structure is a static linear array. Therefore, his size is limited to the number of entries in it and is not expandable. The entries are dedicated to mkeys of size 2^x and no access_flags. Mkeys with different properties are not cacheable. In this patch, we change the cache structure to an RB-tree. This will allow to extend the cache to support more entries with different mkey properties. Signed-off-by: Michael Guralnik Change-Id: Ie99a44f7151770c905a5d04b94f43b38b14cd796 --- drivers/infiniband/hw/mlx5/mlx5_ib.h | 11 +- drivers/infiniband/hw/mlx5/mr.c | 160 ++++++++++++++++++++------- drivers/infiniband/hw/mlx5/odp.c | 8 +- 3 files changed, 132 insertions(+), 47 deletions(-) diff --git a/drivers/infiniband/hw/mlx5/mlx5_ib.h b/drivers/infiniband/hw/mlx5/mlx5_ib.h index 8d985f792367..eec16db2d536 100644 --- a/drivers/infiniband/hw/mlx5/mlx5_ib.h +++ b/drivers/infiniband/hw/mlx5/mlx5_ib.h @@ -741,6 +741,8 @@ struct mlx5_cache_ent { u32 access_mode; unsigned int ndescs; + struct rb_node node; + u8 disabled:1; u8 fill_to_high_water:1; @@ -770,8 +772,9 @@ struct mlx5r_async_create_mkey { struct mlx5_mkey_cache { struct workqueue_struct *wq; - struct mlx5_cache_ent ent[MAX_MKEY_CACHE_ENTRIES]; - struct dentry *root; + struct rb_root rb_root; + struct mutex rb_lock; + struct dentry *fs_root; unsigned long last_add; }; @@ -1316,11 +1319,15 @@ void mlx5_ib_copy_pas(u64 *old, u64 *new, int step, int num); int mlx5_ib_get_cqe_size(struct ib_cq *ibcq); int mlx5_mkey_cache_init(struct mlx5_ib_dev *dev); int mlx5_mkey_cache_cleanup(struct mlx5_ib_dev *dev); +struct mlx5_cache_ent *mlx5r_cache_create_ent(struct mlx5_ib_dev *dev, + int order); struct mlx5_ib_mr *mlx5_mr_cache_alloc(struct mlx5_ib_dev *dev, struct mlx5_cache_ent *ent, int access_flags); +struct mlx5_ib_mr *mlx5_mr_cache_alloc_order(struct mlx5_ib_dev *dev, u32 order, + int access_flags); int mlx5_ib_check_mr_status(struct ib_mr *ibmr, u32 check_mask, struct ib_mr_status *mr_status); struct ib_wq *mlx5_ib_create_wq(struct ib_pd *pd, diff --git a/drivers/infiniband/hw/mlx5/mr.c b/drivers/infiniband/hw/mlx5/mr.c index 356c99d7ec9a..5cc618db277f 100644 --- a/drivers/infiniband/hw/mlx5/mr.c +++ b/drivers/infiniband/hw/mlx5/mr.c @@ -515,18 +515,22 @@ static const struct file_operations limit_fops = { static bool someone_adding(struct mlx5_mkey_cache *cache) { - unsigned int i; - - for (i = 0; i < MAX_MKEY_CACHE_ENTRIES; i++) { - struct mlx5_cache_ent *ent = &cache->ent[i]; - bool ret; + struct mlx5_cache_ent *ent; + struct rb_node *node; + bool ret; + mutex_lock(&cache->rb_lock); + for (node = rb_first(&cache->rb_root); node; node = rb_next(node)) { + ent = rb_entry(node, struct mlx5_cache_ent, node); xa_lock_irq(&ent->mkeys); ret = ent->stored < ent->limit; xa_unlock_irq(&ent->mkeys); - if (ret) + if (ret) { + mutex_unlock(&cache->rb_lock); return true; + } } + mutex_unlock(&cache->rb_lock); return false; } @@ -637,6 +641,59 @@ static void delayed_cache_work_func(struct work_struct *work) __cache_work_func(ent); } +static int mlx5_cache_ent_insert(struct mlx5_mkey_cache *cache, + struct mlx5_cache_ent *ent) +{ + struct rb_node **new = &cache->rb_root.rb_node, *parent = NULL; + struct mlx5_cache_ent *cur; + + mutex_lock(&cache->rb_lock); + /* Figure out where to put new node */ + while (*new) { + cur = rb_entry(*new, struct mlx5_cache_ent, node); + parent = *new; + if (ent->order < cur->order) + new = &((*new)->rb_left); + if (ent->order > cur->order) + new = &((*new)->rb_right); + if (ent->order == cur->order) { + mutex_unlock(&cache->rb_lock); + return -EEXIST; + } + } + + /* Add new node and rebalance tree. */ + rb_link_node(&ent->node, parent, new); + rb_insert_color(&ent->node, &cache->rb_root); + + mutex_unlock(&cache->rb_lock); + return 0; +} + +static struct mlx5_cache_ent *mkey_cache_ent_from_order(struct mlx5_ib_dev *dev, + unsigned int order) +{ + struct rb_node *node = dev->cache.rb_root.rb_node; + struct mlx5_cache_ent *cur, *smallest = NULL; + + /* + * Find the smallest ent with order >= requested_order. + */ + while (node) { + cur = rb_entry(node, struct mlx5_cache_ent, node); + if (cur->order > order) { + smallest = cur; + node = node->rb_left; + } + if (cur->order < order) + node = node->rb_right; + if (cur->order == order) + return cur; + } + + return smallest; +} + struct mlx5_ib_mr *mlx5_mr_cache_alloc(struct mlx5_ib_dev *dev, struct mlx5_cache_ent *ent, int access_flags) @@ -677,10 +734,16 @@ struct mlx5_ib_mr *mlx5_mr_cache_alloc(struct mlx5_ib_dev *dev, return mr; } -static void clean_keys(struct mlx5_ib_dev *dev, int c) +struct mlx5_ib_mr *mlx5_mr_cache_alloc_order(struct mlx5_ib_dev *dev, + u32 order, int access_flags) +{ + struct mlx5_cache_ent *ent = mkey_cache_ent_from_order(dev, order); + + return mlx5_mr_cache_alloc(dev, ent, access_flags); +} + +static void clean_keys(struct mlx5_ib_dev *dev, struct mlx5_cache_ent *ent) { - struct mlx5_mkey_cache *cache = &dev->cache; - struct mlx5_cache_ent *ent = &cache->ent[c]; u32 mkey; cancel_delayed_work(&ent->dwork); @@ -699,8 +762,8 @@ static void mlx5_mkey_cache_debugfs_cleanup(struct mlx5_ib_dev *dev) if (!mlx5_debugfs_root || dev->is_rep) return; - debugfs_remove_recursive(dev->cache.root); - dev->cache.root = NULL; + debugfs_remove_recursive(dev->cache.fs_root); + dev->cache.fs_root = NULL; } static void mlx5_mkey_cache_debugfs_init(struct mlx5_ib_dev *dev) @@ -713,12 +776,13 @@ static void mlx5_mkey_cache_debugfs_init(struct mlx5_ib_dev *dev) if (!mlx5_debugfs_root || dev->is_rep) return; - cache->root = debugfs_create_dir("mr_cache", mlx5_debugfs_get_dev_root(dev->mdev)); + dir = mlx5_debugfs_get_dev_root(dev->mdev); + cache->fs_root = debugfs_create_dir("mr_cache", dir); for (i = 0; i < MAX_MKEY_CACHE_ENTRIES; i++) { - ent = &cache->ent[i]; + ent = mkey_cache_ent_from_order(dev, i); sprintf(ent->name, "%d", ent->order); - dir = debugfs_create_dir(ent->name, cache->root); + dir = debugfs_create_dir(ent->name, cache->fs_root); debugfs_create_file("size", 0600, dir, ent, &size_fops); debugfs_create_file("limit", 0600, dir, ent, &limit_fops); debugfs_create_ulong("cur", 0400, dir, &ent->stored); @@ -733,6 +797,30 @@ static void delay_time_func(struct timer_list *t) WRITE_ONCE(dev->fill_delay, 0); } +struct mlx5_cache_ent *mlx5r_cache_create_ent(struct mlx5_ib_dev *dev, + int order) +{ + struct mlx5_cache_ent *ent; + int ret; + + ent = kzalloc(sizeof(*ent), GFP_KERNEL); + if (!ent) + return ERR_PTR(-ENOMEM); + + xa_init_flags(&ent->mkeys, XA_FLAGS_LOCK_IRQ); + ent->order = order; + ent->dev = dev; + + INIT_DELAYED_WORK(&ent->dwork, delayed_cache_work_func); + + ret = mlx5_cache_ent_insert(&dev->cache, ent); + if (ret) { + kfree(ent); + return ERR_PTR(ret); + } + return ent; +} + int mlx5_mkey_cache_init(struct mlx5_ib_dev *dev) { struct mlx5_mkey_cache *cache = &dev->cache; @@ -740,6 +828,8 @@ int mlx5_mkey_cache_init(struct mlx5_ib_dev *dev) int i; mutex_init(&dev->slow_path_mutex); + mutex_init(&dev->cache.rb_lock); + dev->cache.rb_root = RB_ROOT; cache->wq = alloc_ordered_workqueue("mkey_cache", WQ_MEM_RECLAIM); if (!cache->wq) { mlx5_ib_warn(dev, "failed to create work queue\n"); @@ -749,13 +839,7 @@ int mlx5_mkey_cache_init(struct mlx5_ib_dev *dev) mlx5_cmd_init_async_ctx(dev->mdev, &dev->async_ctx); timer_setup(&dev->delay_timer, delay_time_func, 0); for (i = 0; i < MAX_MKEY_CACHE_ENTRIES; i++) { - ent = &cache->ent[i]; - xa_init_flags(&ent->mkeys, XA_FLAGS_LOCK_IRQ); - ent->order = i + 2; - ent->dev = dev; - ent->limit = 0; - - INIT_DELAYED_WORK(&ent->dwork, delayed_cache_work_func); + ent = mlx5r_cache_create_ent(dev, i); if (i > MKEY_CACHE_LAST_STD_ENTRY) { mlx5_odp_init_mkey_cache_entry(ent); @@ -785,14 +869,16 @@ int mlx5_mkey_cache_init(struct mlx5_ib_dev *dev) int mlx5_mkey_cache_cleanup(struct mlx5_ib_dev *dev) { - unsigned int i; + struct rb_root *root = &dev->cache.rb_root; + struct mlx5_cache_ent *ent; + struct rb_node *node; if (!dev->cache.wq) return 0; - for (i = 0; i < MAX_MKEY_CACHE_ENTRIES; i++) { - struct mlx5_cache_ent *ent = &dev->cache.ent[i]; - + mutex_lock(&dev->cache.rb_lock); + for (node = rb_first(root); node; node = rb_next(node)) { + ent = rb_entry(node, struct mlx5_cache_ent, node); xa_lock_irq(&ent->mkeys); ent->disabled = true; xa_unlock_irq(&ent->mkeys); @@ -802,8 +888,15 @@ int mlx5_mkey_cache_cleanup(struct mlx5_ib_dev *dev) mlx5_mkey_cache_debugfs_cleanup(dev); mlx5_cmd_cleanup_async_ctx(&dev->async_ctx); - for (i = 0; i < MAX_MKEY_CACHE_ENTRIES; i++) - clean_keys(dev, i); + node = rb_first(root); + while (node) { + ent = rb_entry(node, struct mlx5_cache_ent, node); + node = rb_next(node); + clean_keys(dev, ent); + rb_erase(&ent->node, root); + kfree(ent); + } + mutex_unlock(&dev->cache.rb_lock); destroy_workqueue(dev->cache.wq); del_timer_sync(&dev->delay_timer); @@ -876,19 +969,6 @@ static int mkey_cache_max_order(struct mlx5_ib_dev *dev) return MLX5_MAX_UMR_SHIFT; } -static struct mlx5_cache_ent *mkey_cache_ent_from_order(struct mlx5_ib_dev *dev, - unsigned int order) -{ - struct mlx5_mkey_cache *cache = &dev->cache; - - if (order < cache->ent[0].order) - return &cache->ent[0]; - order = order - cache->ent[0].order; - if (order > MKEY_CACHE_LAST_STD_ENTRY) - return NULL; - return &cache->ent[order]; -} - static void set_mr_fields(struct mlx5_ib_dev *dev, struct mlx5_ib_mr *mr, u64 length, int access_flags, u64 iova) { diff --git a/drivers/infiniband/hw/mlx5/odp.c b/drivers/infiniband/hw/mlx5/odp.c index 72044f8ec883..71c3c611e10a 100644 --- a/drivers/infiniband/hw/mlx5/odp.c +++ b/drivers/infiniband/hw/mlx5/odp.c @@ -419,8 +419,7 @@ static struct mlx5_ib_mr *implicit_get_child_mr(struct mlx5_ib_mr *imr, return ERR_CAST(odp); BUILD_BUG_ON(order > MKEY_CACHE_LAST_STD_ENTRY); - mr = mlx5_mr_cache_alloc(dev, &dev->cache.ent[order], - imr->access_flags); + mr = mlx5_mr_cache_alloc_order(dev, order, imr->access_flags); if (IS_ERR(mr)) { ib_umem_odp_release(odp); return mr; @@ -494,9 +493,8 @@ struct mlx5_ib_mr *mlx5_ib_alloc_implicit_mr(struct mlx5_ib_pd *pd, if (IS_ERR(umem_odp)) return ERR_CAST(umem_odp); - imr = mlx5_mr_cache_alloc(dev, - &dev->cache.ent[MLX5_IMR_KSM_CACHE_ENTRY], - access_flags); + imr = mlx5_mr_cache_alloc_order(dev, MLX5_IMR_KSM_CACHE_ENTRY, + access_flags); if (IS_ERR(imr)) { ib_umem_odp_release(umem_odp); return imr; From patchwork Sun Jan 15 12:58:17 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michael Guralnik X-Patchwork-Id: 13102237 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D3028C63797 for ; Sun, 15 Jan 2023 12:58:46 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231136AbjAOM6p (ORCPT ); Sun, 15 Jan 2023 07:58:45 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53320 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230071AbjAOM6o (ORCPT ); Sun, 15 Jan 2023 07:58:44 -0500 Received: from NAM12-DM6-obe.outbound.protection.outlook.com (mail-dm6nam12on2041.outbound.protection.outlook.com [40.107.243.41]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1847A12582 for ; Sun, 15 Jan 2023 04:58:43 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=AuEUellg+cbzBCgdOuu6UviZ7CL7Pwha7yYD5CgXiy06qtLj2SM4PtC2MH/2hYNBWd0SHjojKAdg4fD++wZzN8Z5FFKhIkR87yyzshfssv+HgDULPOe3Aa0uPhs9ltAp8aPbUPVkSVj0h7Qnqzkwn2LX98lFls6oWaTqIVebmRRusBZ4Lzvo7jUy/vjxQzdLAb2LSw+BvoK7FjXNB7jkez+ScCDjAARpNMr5eGkFuIFqPSq1qhW5PDjlM7F1P5sA51jRq1ZYXCzAAy+h1GvEyCR2QnJSP9X8P/4rUNg4hvKTmRaZ7cT7rdmQ8KSqk8XcPt3ysb2/dx2jBCELADKbnw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=eBzvByxbZTTt7MajI2nwD/ZIz7StfMdlgAvPxqlp+5E=; b=KDXYNhqRs1/tNAqZPDOG2BQnhJ5zyzguQBxm+8Qk5xkFKHFMTBaejaBM5+h1AlSSxypI+RD87Gh9L98Ont5N1kEY17HUTcTatQmlyqDCkp6+A1+0+hrI3IfEWdNK3YQ3+xFZNE8IkGvpPX7wgEr5cuut7EZS2TLHnOLlijsI44pMAz5CgL29sPCycLNT7H7b+rhxf4BdlB5Myi+OsnnenmPgpxIMPYdbH+/zh6GhpkQUIgzFKLVR2hGSV9MTZw7kuiFS8NAH4gIkfTVm5hmtSV1QK2kJXrVt+nEItSdwQ799u48wtrkU4HBC2XqyJADkRtyJZV0/mtOGmkHNZwxFmQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.118.232) smtp.rcpttodomain=vger.kernel.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=eBzvByxbZTTt7MajI2nwD/ZIz7StfMdlgAvPxqlp+5E=; b=rDADmvhIvXf72Mnv03tBS8vvKMc5bC5I0XDTW+Ki8oiVrI9iND7/C4apyCTdsrd+Fr+/VZPiBMIyKXegfM/AqhY+6nqROyLhF5gSp2dE4pTt+5siQlGfVm7Ih5/pHdiBIUaPuft7fCc8OtvJnRGW3dgkqBAeSQFYe7HExQjzng69D9Us2GM3bQeLutmvbM8WbgtTgDlI12z2Dxx+Rq7q1C4WKepkPbvfZRQmv8nBODWNYvG5C/VgWAqpFpoI0kc2N1rIg/aQQGbb8/pfxO09nsZz184CoSvwzrAvFz2vnOHq4SN8OITpQ2S3LJjrcpyHfj2PIi5JrNi7ps8M6fMyeg== Received: from MW4PR04CA0355.namprd04.prod.outlook.com (2603:10b6:303:8a::30) by CY8PR12MB8411.namprd12.prod.outlook.com (2603:10b6:930:6e::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6002.13; Sun, 15 Jan 2023 12:58:41 +0000 Received: from CO1NAM11FT040.eop-nam11.prod.protection.outlook.com (2603:10b6:303:8a:cafe::fd) by MW4PR04CA0355.outlook.office365.com (2603:10b6:303:8a::30) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6002.19 via Frontend Transport; Sun, 15 Jan 2023 12:58:41 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.118.232) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.118.232 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.118.232; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.118.232) by CO1NAM11FT040.mail.protection.outlook.com (10.13.174.140) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6002.13 via Frontend Transport; Sun, 15 Jan 2023 12:58:41 +0000 Received: from drhqmail203.nvidia.com (10.126.190.182) by mail.nvidia.com (10.127.129.5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.36; Sun, 15 Jan 2023 04:58:34 -0800 Received: from drhqmail202.nvidia.com (10.126.190.181) by drhqmail203.nvidia.com (10.126.190.182) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.36; Sun, 15 Jan 2023 04:58:33 -0800 Received: from vdi.nvidia.com (10.127.8.13) by mail.nvidia.com (10.126.190.181) with Microsoft SMTP Server id 15.2.986.36 via Frontend Transport; Sun, 15 Jan 2023 04:58:31 -0800 From: Michael Guralnik To: , , CC: , , Michael Guralnik Subject: [PATCH v3 rdma-next 4/6] RDMA/mlx5: Introduce mlx5r_cache_rb_key Date: Sun, 15 Jan 2023 14:58:17 +0200 Message-ID: <20230115125819.15034-5-michaelgur@nvidia.com> X-Mailer: git-send-email 2.17.2 In-Reply-To: <20230115125819.15034-1-michaelgur@nvidia.com> References: <20230115125819.15034-1-michaelgur@nvidia.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CO1NAM11FT040:EE_|CY8PR12MB8411:EE_ X-MS-Office365-Filtering-Correlation-Id: d6b63234-2155-4ea6-aa3c-08daf6f83a0b X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: fG4u7fDAUxvUqF/758O3N+xMsCFqRSQ+w4liAEOxTuZyFP2xGw1IArDcZ2XFGisdKp6E9a1555Ogm17zCFMmGmP76ldYibKS9+XlC9OU5JdAEt31Y7jOBOEqeSYpHQWChivpV1OUIbOF62miJiz2OxGKhVYPckO4rEzDKaU60saH+cBY8ly/1Ji9VgWa/I3mjphMqn17FuyJoss8oIuUjKPC7WCQ2eJj7Qn7Fz9LIBgdvv89nMZJ1t/SwjTyohl5qZhEYteILWq5SNxWKuuV2XtyulSVDiq6BpjcJFAMaDuvByp6VBbWkhBS1NDldGUsq0RHVZuNfLseBSL3ixjweEozg+sxRXBikuWtCcQg5pptb7dZLwLOVVsWnBe/3Bo1ZQvriTI6TwbhDrndaBWEKz65rIFCA9419LccNRAXMLfNCFr82ludSPJTMsr2chBMYVsnVoSvYmbiXglwiz+iBEH1vWscGmyeS++D9GyRZvVtwodemlrVksJEMcTkKYWCBG5T4sKMoF3w44LHDe34619Z8GmgradqGvNVJUfbzgoq5/g3mVV9rQaPbX7etmQsSKKeeBHveXTnDWt4x600h1npEbOGMk5MSUKIPCctNGTN3VoRs4uCo6RWV4BfjomroNYVdNy+cmfeRtlHupGdxOkDZwdTpUZ8jdXg9JqeImi6kUZhj0KI8p3sKiyb9fHNM1MwX1Epu0Grq/DwM3+Rv4G40rXQolxBJ35RlUozTZY= X-Forefront-Antispam-Report: CIP:216.228.118.232;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc7edge1.nvidia.com;CAT:NONE;SFS:(13230022)(4636009)(346002)(136003)(376002)(39860400002)(396003)(451199015)(40470700004)(36840700001)(46966006)(82740400003)(7636003)(356005)(40460700003)(36756003)(40480700001)(82310400005)(86362001)(316002)(70206006)(70586007)(8676002)(4326008)(26005)(336012)(186003)(478600001)(54906003)(1076003)(110136005)(7696005)(2616005)(36860700001)(8936002)(426003)(47076005)(41300700001)(107886003)(83380400001)(6666004)(5660300002)(30864003)(2906002)(2101003);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 15 Jan 2023 12:58:41.1574 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: d6b63234-2155-4ea6-aa3c-08daf6f83a0b X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.118.232];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CO1NAM11FT040.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY8PR12MB8411 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org Switch from using the mkey order to using the new struct as the key to the RB tree of cache entries. The key is all the mkey properties that UMR operations can't modify. Using this key to define the cache entries and to search and create cache mkeys. Signed-off-by: Michael Guralnik Change-Id: I30d75a2404804d8c5259c27d0b256b17c1d3c24d --- drivers/infiniband/hw/mlx5/mlx5_ib.h | 27 ++-- drivers/infiniband/hw/mlx5/mr.c | 228 +++++++++++++++++++-------- drivers/infiniband/hw/mlx5/odp.c | 30 ++-- 3 files changed, 201 insertions(+), 84 deletions(-) diff --git a/drivers/infiniband/hw/mlx5/mlx5_ib.h b/drivers/infiniband/hw/mlx5/mlx5_ib.h index eec16db2d536..d560d6cbbe9b 100644 --- a/drivers/infiniband/hw/mlx5/mlx5_ib.h +++ b/drivers/infiniband/hw/mlx5/mlx5_ib.h @@ -617,6 +617,13 @@ enum mlx5_mkey_type { MLX5_MKEY_INDIRECT_DEVX, }; +struct mlx5r_cache_rb_key { + u8 ats:1; + unsigned int access_mode; + unsigned int access_flags; + unsigned int ndescs; +}; + struct mlx5_ib_mkey { u32 key; enum mlx5_mkey_type type; @@ -737,11 +744,9 @@ struct mlx5_cache_ent { unsigned long reserved; char name[4]; - u32 order; - u32 access_mode; - unsigned int ndescs; struct rb_node node; + struct mlx5r_cache_rb_key rb_key; u8 disabled:1; u8 fill_to_high_water:1; @@ -1320,14 +1325,13 @@ int mlx5_ib_get_cqe_size(struct ib_cq *ibcq); int mlx5_mkey_cache_init(struct mlx5_ib_dev *dev); int mlx5_mkey_cache_cleanup(struct mlx5_ib_dev *dev); struct mlx5_cache_ent *mlx5r_cache_create_ent(struct mlx5_ib_dev *dev, - int order); + struct mlx5r_cache_rb_key rb_key, + bool persistent_entry); struct mlx5_ib_mr *mlx5_mr_cache_alloc(struct mlx5_ib_dev *dev, - struct mlx5_cache_ent *ent, - int access_flags); + int access_flags, int access_mode, + int ndescs); -struct mlx5_ib_mr *mlx5_mr_cache_alloc_order(struct mlx5_ib_dev *dev, u32 order, - int access_flags); int mlx5_ib_check_mr_status(struct ib_mr *ibmr, u32 check_mask, struct ib_mr_status *mr_status); struct ib_wq *mlx5_ib_create_wq(struct ib_pd *pd, @@ -1350,7 +1354,7 @@ int mlx5r_odp_create_eq(struct mlx5_ib_dev *dev, struct mlx5_ib_pf_eq *eq); void mlx5_ib_odp_cleanup_one(struct mlx5_ib_dev *ibdev); int __init mlx5_ib_odp_init(void); void mlx5_ib_odp_cleanup(void); -void mlx5_odp_init_mkey_cache_entry(struct mlx5_cache_ent *ent); +int mlx5_odp_init_mkey_cache(struct mlx5_ib_dev *dev); void mlx5_odp_populate_xlt(void *xlt, size_t idx, size_t nentries, struct mlx5_ib_mr *mr, int flags); @@ -1369,7 +1373,10 @@ static inline int mlx5r_odp_create_eq(struct mlx5_ib_dev *dev, static inline void mlx5_ib_odp_cleanup_one(struct mlx5_ib_dev *ibdev) {} static inline int mlx5_ib_odp_init(void) { return 0; } static inline void mlx5_ib_odp_cleanup(void) {} -static inline void mlx5_odp_init_mkey_cache_entry(struct mlx5_cache_ent *ent) {} +static inline int mlx5_odp_init_mkey_cache(struct mlx5_ib_dev *dev) +{ + return NULL; +} static inline void mlx5_odp_populate_xlt(void *xlt, size_t idx, size_t nentries, struct mlx5_ib_mr *mr, int flags) {} diff --git a/drivers/infiniband/hw/mlx5/mr.c b/drivers/infiniband/hw/mlx5/mr.c index 5cc618db277f..7924953b9bd0 100644 --- a/drivers/infiniband/hw/mlx5/mr.c +++ b/drivers/infiniband/hw/mlx5/mr.c @@ -292,11 +292,13 @@ static void set_cache_mkc(struct mlx5_cache_ent *ent, void *mkc) set_mkc_access_pd_addr_fields(mkc, 0, 0, ent->dev->umrc.pd); MLX5_SET(mkc, mkc, free, 1); MLX5_SET(mkc, mkc, umr_en, 1); - MLX5_SET(mkc, mkc, access_mode_1_0, ent->access_mode & 0x3); - MLX5_SET(mkc, mkc, access_mode_4_2, (ent->access_mode >> 2) & 0x7); + MLX5_SET(mkc, mkc, access_mode_1_0, ent->rb_key.access_mode & 0x3); + MLX5_SET(mkc, mkc, access_mode_4_2, + (ent->rb_key.access_mode >> 2) & 0x7); MLX5_SET(mkc, mkc, translations_octword_size, - get_mkc_octo_size(ent->access_mode, ent->ndescs)); + get_mkc_octo_size(ent->rb_key.access_mode, + ent->rb_key.ndescs)); MLX5_SET(mkc, mkc, log_page_size, PAGE_SHIFT); } @@ -594,8 +596,8 @@ static void __cache_work_func(struct mlx5_cache_ent *ent) if (err != -EAGAIN) { mlx5_ib_warn( dev, - "command failed order %d, err %d\n", - ent->order, err); + "add keys command failed, err %d\n", + err); queue_delayed_work(cache->wq, &ent->dwork, msecs_to_jiffies(1000)); } @@ -641,22 +643,49 @@ static void delayed_cache_work_func(struct work_struct *work) __cache_work_func(ent); } +static int cache_ent_key_cmp(struct mlx5r_cache_rb_key key1, + struct mlx5r_cache_rb_key key2) +{ + int res; + + res = key1.ats - key2.ats; + if (res) + return res; + + res = key1.access_mode - key2.access_mode; + if (res) + return res; + + res = key1.access_flags - key2.access_flags; + if (res) + return res; + + /* + * keep ndescs the last in the compare table since the find function + * searches for an exact match on all properties and only closest + * match in size. + */ + return key1.ndescs - key2.ndescs; +} + static int mlx5_cache_ent_insert(struct mlx5_mkey_cache *cache, struct mlx5_cache_ent *ent) { struct rb_node **new = &cache->rb_root.rb_node, *parent = NULL; struct mlx5_cache_ent *cur; + int cmp; mutex_lock(&cache->rb_lock); /* Figure out where to put new node */ while (*new) { cur = rb_entry(*new, struct mlx5_cache_ent, node); parent = *new; - if (ent->order < cur->order) + cmp = cache_ent_key_cmp(cur->rb_key, ent->rb_key); + if (cmp > 0) new = &((*new)->rb_left); - if (ent->order > cur->order) + if (cmp < 0) new = &((*new)->rb_right); - if (ent->order == cur->order) { + if (cmp == 0) { mutex_unlock(&cache->rb_lock); return -EEXIST; } @@ -670,40 +699,45 @@ static int mlx5_cache_ent_insert(struct mlx5_mkey_cache *cache, return 0; } -static struct mlx5_cache_ent *mkey_cache_ent_from_order(struct mlx5_ib_dev *dev, - unsigned int order) +static struct mlx5_cache_ent * +mkey_cache_ent_from_rb_key(struct mlx5_ib_dev *dev, + struct mlx5r_cache_rb_key rb_key) { struct rb_node *node = dev->cache.rb_root.rb_node; struct mlx5_cache_ent *cur, *smallest = NULL; + int cmp; /* * Find the smallest ent with order >= requested_order. */ while (node) { cur = rb_entry(node, struct mlx5_cache_ent, node); - if (cur->order > order) { + cmp = cache_ent_key_cmp(cur->rb_key, rb_key); + if (cmp > 0) { smallest = cur; node = node->rb_left; } - if (cur->order < order) + if (cmp < 0) node = node->rb_right; - if (cur->order == order) + if (cmp == 0) return cur; } - return smallest; + return (smallest && + smallest->rb_key.access_mode == rb_key.access_mode && + smallest->rb_key.access_flags == rb_key.access_flags && + smallest->rb_key.ats == rb_key.ats) ? + smallest : + NULL; } -struct mlx5_ib_mr *mlx5_mr_cache_alloc(struct mlx5_ib_dev *dev, - struct mlx5_cache_ent *ent, - int access_flags) +static struct mlx5_ib_mr *_mlx5_mr_cache_alloc(struct mlx5_ib_dev *dev, + struct mlx5_cache_ent *ent, + int access_flags) { struct mlx5_ib_mr *mr; int err; - if (!mlx5r_umr_can_reconfig(dev, 0, access_flags)) - return ERR_PTR(-EOPNOTSUPP); - mr = kzalloc(sizeof(*mr), GFP_KERNEL); if (!mr) return ERR_PTR(-ENOMEM); @@ -734,12 +768,44 @@ struct mlx5_ib_mr *mlx5_mr_cache_alloc(struct mlx5_ib_dev *dev, return mr; } -struct mlx5_ib_mr *mlx5_mr_cache_alloc_order(struct mlx5_ib_dev *dev, - u32 order, int access_flags) +static int get_unchangeable_access_flags(struct mlx5_ib_dev *dev, + int access_flags) { - struct mlx5_cache_ent *ent = mkey_cache_ent_from_order(dev, order); + int ret = 0; - return mlx5_mr_cache_alloc(dev, ent, access_flags); + if ((access_flags & IB_ACCESS_REMOTE_ATOMIC) && + MLX5_CAP_GEN(dev->mdev, atomic) && + MLX5_CAP_GEN(dev->mdev, umr_modify_atomic_disabled)) + ret |= IB_ACCESS_REMOTE_ATOMIC; + + if ((access_flags & IB_ACCESS_RELAXED_ORDERING) && + MLX5_CAP_GEN(dev->mdev, relaxed_ordering_write) && + !MLX5_CAP_GEN(dev->mdev, relaxed_ordering_write_umr)) + ret |= IB_ACCESS_RELAXED_ORDERING; + + if ((access_flags & IB_ACCESS_RELAXED_ORDERING) && + MLX5_CAP_GEN(dev->mdev, relaxed_ordering_read) && + !MLX5_CAP_GEN(dev->mdev, relaxed_ordering_read_umr)) + ret |= IB_ACCESS_RELAXED_ORDERING; + + return ret; +} + +struct mlx5_ib_mr *mlx5_mr_cache_alloc(struct mlx5_ib_dev *dev, + int access_flags, int access_mode, + int ndescs) +{ + struct mlx5r_cache_rb_key rb_key = { + .ndescs = ndescs, + .access_mode = access_mode, + .access_flags = get_unchangeable_access_flags(dev, access_flags) + }; + struct mlx5_cache_ent *ent = mkey_cache_ent_from_rb_key(dev, rb_key); + + if (!ent) + return ERR_PTR(-EOPNOTSUPP); + + return _mlx5_mr_cache_alloc(dev, ent, access_flags); } static void clean_keys(struct mlx5_ib_dev *dev, struct mlx5_cache_ent *ent) @@ -766,28 +832,32 @@ static void mlx5_mkey_cache_debugfs_cleanup(struct mlx5_ib_dev *dev) dev->cache.fs_root = NULL; } +static void mlx5_mkey_cache_debugfs_add_ent(struct mlx5_ib_dev *dev, + struct mlx5_cache_ent *ent) +{ + int order = order_base_2(ent->rb_key.ndescs); + struct dentry *dir; + + if (ent->rb_key.access_mode == MLX5_MKC_ACCESS_MODE_KSM) + order = MLX5_IMR_KSM_CACHE_ENTRY + 2; + + sprintf(ent->name, "%d", order); + dir = debugfs_create_dir(ent->name, dev->cache.fs_root); + debugfs_create_file("size", 0600, dir, ent, &size_fops); + debugfs_create_file("limit", 0600, dir, ent, &limit_fops); + debugfs_create_ulong("cur", 0400, dir, &ent->stored); + debugfs_create_u32("miss", 0600, dir, &ent->miss); +} + static void mlx5_mkey_cache_debugfs_init(struct mlx5_ib_dev *dev) { + struct dentry *dbg_root = mlx5_debugfs_get_dev_root(dev->mdev); struct mlx5_mkey_cache *cache = &dev->cache; - struct mlx5_cache_ent *ent; - struct dentry *dir; - int i; if (!mlx5_debugfs_root || dev->is_rep) return; - dir = mlx5_debugfs_get_dev_root(dev->mdev); - cache->fs_root = debugfs_create_dir("mr_cache", dir); - - for (i = 0; i < MAX_MKEY_CACHE_ENTRIES; i++) { - ent = mkey_cache_ent_from_order(dev, i); - sprintf(ent->name, "%d", ent->order); - dir = debugfs_create_dir(ent->name, cache->fs_root); - debugfs_create_file("size", 0600, dir, ent, &size_fops); - debugfs_create_file("limit", 0600, dir, ent, &limit_fops); - debugfs_create_ulong("cur", 0400, dir, &ent->stored); - debugfs_create_u32("miss", 0600, dir, &ent->miss); - } + cache->fs_root = debugfs_create_dir("mr_cache", dbg_root); } static void delay_time_func(struct timer_list *t) @@ -798,9 +868,11 @@ static void delay_time_func(struct timer_list *t) } struct mlx5_cache_ent *mlx5r_cache_create_ent(struct mlx5_ib_dev *dev, - int order) + struct mlx5r_cache_rb_key rb_key, + bool persistent_entry) { struct mlx5_cache_ent *ent; + int order; int ret; ent = kzalloc(sizeof(*ent), GFP_KERNEL); @@ -808,7 +880,7 @@ struct mlx5_cache_ent *mlx5r_cache_create_ent(struct mlx5_ib_dev *dev, return ERR_PTR(-ENOMEM); xa_init_flags(&ent->mkeys, XA_FLAGS_LOCK_IRQ); - ent->order = order; + ent->rb_key = rb_key; ent->dev = dev; INIT_DELAYED_WORK(&ent->dwork, delayed_cache_work_func); @@ -818,13 +890,36 @@ struct mlx5_cache_ent *mlx5r_cache_create_ent(struct mlx5_ib_dev *dev, kfree(ent); return ERR_PTR(ret); } + + if (persistent_entry) { + if (rb_key.access_mode == MLX5_MKC_ACCESS_MODE_KSM) + order = MLX5_IMR_KSM_CACHE_ENTRY; + else + order = order_base_2(rb_key.ndescs) - 2; + + if ((dev->mdev->profile.mask & MLX5_PROF_MASK_MR_CACHE) && + !dev->is_rep && mlx5_core_is_pf(dev->mdev) && + mlx5r_umr_can_load_pas(dev, 0)) + ent->limit = dev->mdev->profile.mr_cache[order].limit; + else + ent->limit = 0; + + mlx5_mkey_cache_debugfs_add_ent(dev, ent); + } + return ent; } int mlx5_mkey_cache_init(struct mlx5_ib_dev *dev) { struct mlx5_mkey_cache *cache = &dev->cache; + struct rb_root *root = &dev->cache.rb_root; + struct mlx5r_cache_rb_key rb_key = { + .access_mode = MLX5_MKC_ACCESS_MODE_MTT, + }; struct mlx5_cache_ent *ent; + struct rb_node *node; + int ret; int i; mutex_init(&dev->slow_path_mutex); @@ -838,33 +933,32 @@ int mlx5_mkey_cache_init(struct mlx5_ib_dev *dev) mlx5_cmd_init_async_ctx(dev->mdev, &dev->async_ctx); timer_setup(&dev->delay_timer, delay_time_func, 0); - for (i = 0; i < MAX_MKEY_CACHE_ENTRIES; i++) { - ent = mlx5r_cache_create_ent(dev, i); - - if (i > MKEY_CACHE_LAST_STD_ENTRY) { - mlx5_odp_init_mkey_cache_entry(ent); - continue; + mlx5_mkey_cache_debugfs_init(dev); + for (i = 0; i <= mkey_cache_max_order(dev); i++) { + rb_key.ndescs = 1 << (i + 2); + ent = mlx5r_cache_create_ent(dev, rb_key, true); + if (IS_ERR(ent)) { + ret = PTR_ERR(ent); + goto err; } + } - if (ent->order > mkey_cache_max_order(dev)) - continue; + ret = mlx5_odp_init_mkey_cache(dev); + if (ret) + goto err; - ent->ndescs = 1 << ent->order; - ent->access_mode = MLX5_MKC_ACCESS_MODE_MTT; - if ((dev->mdev->profile.mask & MLX5_PROF_MASK_MR_CACHE) && - !dev->is_rep && mlx5_core_is_pf(dev->mdev) && - mlx5r_umr_can_load_pas(dev, 0)) - ent->limit = dev->mdev->profile.mr_cache[i].limit; - else - ent->limit = 0; + for (node = rb_first(root); node; node = rb_next(node)) { + ent = rb_entry(node, struct mlx5_cache_ent, node); xa_lock_irq(&ent->mkeys); queue_adjust_cache_locked(ent); xa_unlock_irq(&ent->mkeys); } - mlx5_mkey_cache_debugfs_init(dev); - return 0; + +err: + mlx5_ib_warn(dev, "failed to create mkey cache entry\n"); + return ret; } int mlx5_mkey_cache_cleanup(struct mlx5_ib_dev *dev) @@ -965,7 +1059,7 @@ static int get_octo_len(u64 addr, u64 len, int page_shift) static int mkey_cache_max_order(struct mlx5_ib_dev *dev) { if (MLX5_CAP_GEN(dev->mdev, umr_extended_translation_offset)) - return MKEY_CACHE_LAST_STD_ENTRY + 2; + return MKEY_CACHE_LAST_STD_ENTRY; return MLX5_MAX_UMR_SHIFT; } @@ -995,6 +1089,9 @@ static struct mlx5_ib_mr *alloc_cacheable_mr(struct ib_pd *pd, struct ib_umem *umem, u64 iova, int access_flags) { + struct mlx5r_cache_rb_key rb_key = { + .access_mode = MLX5_MKC_ACCESS_MODE_MTT, + }; struct mlx5_ib_dev *dev = to_mdev(pd->device); struct mlx5_cache_ent *ent; struct mlx5_ib_mr *mr; @@ -1007,8 +1104,11 @@ static struct mlx5_ib_mr *alloc_cacheable_mr(struct ib_pd *pd, 0, iova); if (WARN_ON(!page_size)) return ERR_PTR(-EINVAL); - ent = mkey_cache_ent_from_order( - dev, order_base_2(ib_umem_num_dma_blocks(umem, page_size))); + + rb_key.ndescs = ib_umem_num_dma_blocks(umem, page_size); + rb_key.ats = mlx5_umem_needs_ats(dev, umem, access_flags); + rb_key.access_flags = get_unchangeable_access_flags(dev, access_flags); + ent = mkey_cache_ent_from_rb_key(dev, rb_key); /* * Matches access in alloc_cache_mr(). If the MR can't come from the * cache then synchronously create an uncached one. @@ -1022,7 +1122,7 @@ static struct mlx5_ib_mr *alloc_cacheable_mr(struct ib_pd *pd, return mr; } - mr = mlx5_mr_cache_alloc(dev, ent, access_flags); + mr = _mlx5_mr_cache_alloc(dev, ent, access_flags); if (IS_ERR(mr)) return mr; @@ -1451,7 +1551,7 @@ static bool can_use_umr_rereg_pas(struct mlx5_ib_mr *mr, mlx5_umem_find_best_pgsz(new_umem, mkc, log_page_size, 0, iova); if (WARN_ON(!*page_size)) return false; - return (1ULL << mr->mmkey.cache_ent->order) >= + return (mr->mmkey.cache_ent->rb_key.ndescs) >= ib_umem_num_dma_blocks(new_umem, *page_size); } diff --git a/drivers/infiniband/hw/mlx5/odp.c b/drivers/infiniband/hw/mlx5/odp.c index 71c3c611e10a..c51d6c9a4c87 100644 --- a/drivers/infiniband/hw/mlx5/odp.c +++ b/drivers/infiniband/hw/mlx5/odp.c @@ -405,7 +405,6 @@ static void mlx5_ib_page_fault_resume(struct mlx5_ib_dev *dev, static struct mlx5_ib_mr *implicit_get_child_mr(struct mlx5_ib_mr *imr, unsigned long idx) { - int order = order_base_2(MLX5_IMR_MTT_ENTRIES); struct mlx5_ib_dev *dev = mr_to_mdev(imr); struct ib_umem_odp *odp; struct mlx5_ib_mr *mr; @@ -418,8 +417,9 @@ static struct mlx5_ib_mr *implicit_get_child_mr(struct mlx5_ib_mr *imr, if (IS_ERR(odp)) return ERR_CAST(odp); - BUILD_BUG_ON(order > MKEY_CACHE_LAST_STD_ENTRY); - mr = mlx5_mr_cache_alloc_order(dev, order, imr->access_flags); + mr = mlx5_mr_cache_alloc(dev, imr->access_flags, + MLX5_MKC_ACCESS_MODE_MTT, + MLX5_IMR_MTT_ENTRIES); if (IS_ERR(mr)) { ib_umem_odp_release(odp); return mr; @@ -493,8 +493,8 @@ struct mlx5_ib_mr *mlx5_ib_alloc_implicit_mr(struct mlx5_ib_pd *pd, if (IS_ERR(umem_odp)) return ERR_CAST(umem_odp); - imr = mlx5_mr_cache_alloc_order(dev, MLX5_IMR_KSM_CACHE_ENTRY, - access_flags); + imr = mlx5_mr_cache_alloc(dev, access_flags, MLX5_MKC_ACCESS_MODE_KSM, + mlx5_imr_ksm_entries); if (IS_ERR(imr)) { ib_umem_odp_release(umem_odp); return imr; @@ -1587,12 +1587,22 @@ mlx5_ib_odp_destroy_eq(struct mlx5_ib_dev *dev, struct mlx5_ib_pf_eq *eq) return err; } -void mlx5_odp_init_mkey_cache_entry(struct mlx5_cache_ent *ent) +int mlx5_odp_init_mkey_cache(struct mlx5_ib_dev *dev) { - if (!(ent->dev->odp_caps.general_caps & IB_ODP_SUPPORT_IMPLICIT)) - return; - ent->ndescs = mlx5_imr_ksm_entries; - ent->access_mode = MLX5_MKC_ACCESS_MODE_KSM; + struct mlx5r_cache_rb_key rb_key = { + .access_mode = MLX5_MKC_ACCESS_MODE_KSM, + .ndescs = mlx5_imr_ksm_entries, + }; + struct mlx5_cache_ent *ent; + + if (!(dev->odp_caps.general_caps & IB_ODP_SUPPORT_IMPLICIT)) + return 0; + + ent = mlx5r_cache_create_ent(dev, rb_key, true); + if (IS_ERR(ent)) + return PTR_ERR(ent); + + return 0; } static const struct ib_device_ops mlx5_ib_dev_odp_ops = { From patchwork Sun Jan 15 12:58:18 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michael Guralnik X-Patchwork-Id: 13102236 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 69B72C46467 for ; Sun, 15 Jan 2023 12:58:47 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230517AbjAOM6q (ORCPT ); Sun, 15 Jan 2023 07:58:46 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53314 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229890AbjAOM6o (ORCPT ); Sun, 15 Jan 2023 07:58:44 -0500 Received: from NAM11-CO1-obe.outbound.protection.outlook.com (mail-co1nam11on2058.outbound.protection.outlook.com [40.107.220.58]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DCD2212583 for ; Sun, 15 Jan 2023 04:58:43 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=FcRf3gUOnHLt7XkowvmchDBBNVXAjlEa3AeUPqjdbX9ugKCW84qLXDVBs/kTSeWRbT0Aj8Alpjpmn8okFPLWIzziA2Kp2iKxuVB6mu4TSaVFk90l2pJ6Z/qAqZCLQT0PfhfGyBDFS4kfRLEfqVxT9ixyV6X74sHg/KuW3E5FOqcfJHV8ECdX7pjqfOoflm5XVY2cOv8R1q6bzFULC00kvgwvq6rDX3pUjAn+XI+NnPAqVwDnU6uyXWq3PlojVct8CAh983bDTsCj9z6KdwBhUOj49a2E3aYQIedCwnWWF9skaTlI75B8TFzcE0ySz2ei1NtQC1DLnTYFs3Hyb/83KQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=mX5njaTIkrWE1vz37Mp8CK0TXNcqiexUnKO6n47fEUs=; b=CFhP+FtaXG/xq2KjLYl3ZsFb6VLoXtF5uwmM9uRDbX04d4fcVwi4sD/WLNisOdeDqnJIis92htdnIxlGTp8cCq8g0H2SgxaziG7fPG6CqNmyb598YIpwe42PVOnXqobbFUas8gr5oq5LP+wFlLUsbVj7hNOnf9/9uyuSQbmhirMdl3Z2Aqlqzq6LZ9nm31beO+G/QdVMSYtaj16Qa/bADIvJtNuEaaLryn3/gPbua1cfKsYKz9yLfkNGrOfTldRtC9YmVez56cQYhOGx21rZ9XgvikL4jglZZx1B5qTCf0ck/vpd2KubMvcy9XORA+1R6lBhrWZ0xKw6XKVWnx2YAA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.118.233) smtp.rcpttodomain=vger.kernel.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=mX5njaTIkrWE1vz37Mp8CK0TXNcqiexUnKO6n47fEUs=; b=Wju6b8kCRJcGEIIA/QWi6orAIUudrdpHaxVO4SWEBjNyavp0ooKlg7nBQk2w/oKJkV+rQ70rrZwiNJsFZrottXaW4aT698HDHvkCN9U4GouOh1Yjev9SmVKCZjz59CkD3yzcrM5lUyHL4BInbphQzqeZUWscSWBRwrBq/EVK9ckpBggBk6AGU7s87bP46HrxhJsJFvq5mv+asTmJ1SRkFMGjwWi4h+ZeHXUkfHHz+Lq3/dxia7VbjgQVeBJrvZIPFsMQi6f0s1QGvCqGoNaQLYxr8H26WZkt9lFj392+6cucLkqn+Q7gOMNbt2tGWHpt5OolDRZWI1AiOQqIl+2QTg== Received: from BL1P221CA0010.NAMP221.PROD.OUTLOOK.COM (2603:10b6:208:2c5::14) by DM4PR12MB5915.namprd12.prod.outlook.com (2603:10b6:8:68::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5986.19; Sun, 15 Jan 2023 12:58:42 +0000 Received: from BL02EPF000108EB.namprd05.prod.outlook.com (2603:10b6:208:2c5:cafe::de) by BL1P221CA0010.outlook.office365.com (2603:10b6:208:2c5::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6002.19 via Frontend Transport; Sun, 15 Jan 2023 12:58:42 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.118.233) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.118.233 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.118.233; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.118.233) by BL02EPF000108EB.mail.protection.outlook.com (10.167.241.200) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6002.11 via Frontend Transport; Sun, 15 Jan 2023 12:58:41 +0000 Received: from drhqmail203.nvidia.com (10.126.190.182) by mail.nvidia.com (10.127.129.6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.36; Sun, 15 Jan 2023 04:58:36 -0800 Received: from drhqmail202.nvidia.com (10.126.190.181) by drhqmail203.nvidia.com (10.126.190.182) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.36; Sun, 15 Jan 2023 04:58:35 -0800 Received: from vdi.nvidia.com (10.127.8.13) by mail.nvidia.com (10.126.190.181) with Microsoft SMTP Server id 15.2.986.36 via Frontend Transport; Sun, 15 Jan 2023 04:58:34 -0800 From: Michael Guralnik To: , , CC: , , Michael Guralnik Subject: [PATCH v3 rdma-next 5/6] RDMA/mlx5: Cache all user cacheable mkeys on dereg MR flow Date: Sun, 15 Jan 2023 14:58:18 +0200 Message-ID: <20230115125819.15034-6-michaelgur@nvidia.com> X-Mailer: git-send-email 2.17.2 In-Reply-To: <20230115125819.15034-1-michaelgur@nvidia.com> References: <20230115125819.15034-1-michaelgur@nvidia.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BL02EPF000108EB:EE_|DM4PR12MB5915:EE_ X-MS-Office365-Filtering-Correlation-Id: 0034f737-1647-4766-db19-08daf6f83a95 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: k8aO7L+bRWU9DDVPzgb6m97AJz2mzaV/Nq78ytYrLzI7S9FdlssAsCDdcK+N2cCCiQgVBIkR1PnybnLsO1762J8icsUeuW1YHKMaE4OYTBlyXJweh+FVJ+Y7t4qVpJCYwao/jF+J/UUswu9N9Ip+qFxsDEOma6a0/bIyM07qDdtrp2+6PDUNXyUTYAIyx/jdZ1zHAIIBBL3wJxxyBkhsLOYmHUKtm6N7G7+t0uzKRSH7MSHjpzSvabsbA5kOujGK+sMU2c9p2glglBs3odop2L4fLjbGkmWkpPX54Jxg0oYiZ6DUHWWwK5sYOETtiVCbahXkO4BZMFCDUtRiZkgpP1b6CQMvgl/kQPabxBpkYhKyyNOCJCcot2LQG8tpMFhBUgcQIXwatVv4kAaaiPleryEGZooDXLXQq7OMcBWz+EWkyFYEz6kFTwZwNb1n5F2er13MCbqgyqjbJbU8rzocrRwTaHW2sd6jQZ37B/Vw01RGYFypfJ7g1jGoLMmklgad16cYGxUrmxJMSNkfBeAwVHJUzVwN0+GVh4vfek/0aPGDWpEiyF1OIFatyKtHTkl5Gjw/B9QNiJepIoj9k6ibCkI4yLZ6nVC42vLqr1vWrf0Q3MdTXZv3kv3cA5YAfcTaVYCl435B+2IXk0hpMNP/vg21vTF7K7IvJTy/8+/w3r1z923t4ZFLYf5fQRibXip2YI+y5S5to2TdVknOc8F9e8IxWkKkmNeHkvhQFqK2Zaw= X-Forefront-Antispam-Report: CIP:216.228.118.233;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc7edge2.nvidia.com;CAT:NONE;SFS:(13230022)(4636009)(346002)(396003)(39860400002)(136003)(376002)(451199015)(46966006)(40470700004)(36840700001)(8936002)(5660300002)(41300700001)(7636003)(83380400001)(86362001)(426003)(47076005)(82310400005)(36756003)(40460700003)(82740400003)(356005)(2906002)(40480700001)(36860700001)(26005)(478600001)(186003)(7696005)(70206006)(4326008)(8676002)(336012)(107886003)(316002)(110136005)(54906003)(1076003)(2616005)(6666004)(70586007)(2101003);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 15 Jan 2023 12:58:41.9536 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 0034f737-1647-4766-db19-08daf6f83a95 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.118.233];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BL02EPF000108EB.namprd05.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM4PR12MB5915 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org Currently, when dereging an MR, if the mkey doesn't belong to a cache entry, it will be destroyed. As a result, the restart of applications with many non-cached mkeys is not efficient since all the mkeys are destroyed and then recreated. This process takes a long time (for 100,000 MRs, it is ~20 seconds for dereg and ~28 seconds for re-reg). To shorten the restart runtime, insert all cacheable mkeys to the cache. If there is no fitting entry to the mkey properties, create a temporary entry that fits it. After a predetermined timeout, the cache entries will shrink to the initial high limit. The mkeys will still be in the cache when consuming them again after an application restart. Therefore, the registration will be much faster (for 100,000 MRs, it is ~4 seconds for dereg and ~5 seconds for re-reg). The temporary cache entries created to store the non-cache mkeys are not exposed through sysfs like the default cache entries. Signed-off-by: Michael Guralnik Change-Id: I453176214c36af7f6cfa7491e613651bfb2100e5 --- drivers/infiniband/hw/mlx5/mlx5_ib.h | 2 + drivers/infiniband/hw/mlx5/mr.c | 55 +++++++++++++++++++++------- 2 files changed, 44 insertions(+), 13 deletions(-) diff --git a/drivers/infiniband/hw/mlx5/mlx5_ib.h b/drivers/infiniband/hw/mlx5/mlx5_ib.h index d560d6cbbe9b..6e0c0a931d78 100644 --- a/drivers/infiniband/hw/mlx5/mlx5_ib.h +++ b/drivers/infiniband/hw/mlx5/mlx5_ib.h @@ -630,6 +630,8 @@ struct mlx5_ib_mkey { unsigned int ndescs; struct wait_queue_head wait; refcount_t usecount; + /* User Mkey must hold either a rb_key or a cache_ent. */ + struct mlx5r_cache_rb_key rb_key; struct mlx5_cache_ent *cache_ent; }; diff --git a/drivers/infiniband/hw/mlx5/mr.c b/drivers/infiniband/hw/mlx5/mr.c index 7924953b9bd0..25e80529edd8 100644 --- a/drivers/infiniband/hw/mlx5/mr.c +++ b/drivers/infiniband/hw/mlx5/mr.c @@ -1110,15 +1110,14 @@ static struct mlx5_ib_mr *alloc_cacheable_mr(struct ib_pd *pd, rb_key.access_flags = get_unchangeable_access_flags(dev, access_flags); ent = mkey_cache_ent_from_rb_key(dev, rb_key); /* - * Matches access in alloc_cache_mr(). If the MR can't come from the - * cache then synchronously create an uncached one. + * If the MR can't come from the cache then synchronously create an uncached + * one. */ - if (!ent || ent->limit == 0 || - !mlx5r_umr_can_reconfig(dev, 0, access_flags) || - mlx5_umem_needs_ats(dev, umem, access_flags)) { + if (!ent) { mutex_lock(&dev->slow_path_mutex); mr = reg_create(pd, umem, iova, access_flags, page_size, false); mutex_unlock(&dev->slow_path_mutex); + mr->mmkey.rb_key = rb_key; return mr; } @@ -1209,6 +1208,7 @@ static struct mlx5_ib_mr *reg_create(struct ib_pd *pd, struct ib_umem *umem, goto err_2; } mr->mmkey.type = MLX5_MKEY_MR; + mr->mmkey.ndescs = get_octo_len(iova, umem->length, mr->page_shift); mr->umem = umem; set_mr_fields(dev, mr, umem->length, access_flags, iova); kvfree(in); @@ -1746,6 +1746,40 @@ mlx5_free_priv_descs(struct mlx5_ib_mr *mr) } } +static int cache_ent_find_and_store(struct mlx5_ib_dev *dev, + struct mlx5_ib_mr *mr) +{ + struct mlx5_mkey_cache *cache = &dev->cache; + struct mlx5_cache_ent *ent; + + if (mr->mmkey.cache_ent) { + xa_lock_irq(&mr->mmkey.cache_ent->mkeys); + mr->mmkey.cache_ent->in_use--; + xa_unlock_irq(&mr->mmkey.cache_ent->mkeys); + goto end; + } + + mutex_lock(&cache->rb_lock); + ent = mkey_cache_ent_from_rb_key(dev, mr->mmkey.rb_key); + mutex_unlock(&cache->rb_lock); + if (ent) { + if (ent->rb_key.ndescs == mr->mmkey.rb_key.ndescs) { + mr->mmkey.cache_ent = ent; + goto end; + } + } + + ent = mlx5r_cache_create_ent(dev, mr->mmkey.rb_key, false); + if (IS_ERR(ent)) + return PTR_ERR(ent); + + mr->mmkey.cache_ent = ent; + +end: + return push_mkey(mr->mmkey.cache_ent, false, + xa_mk_value(mr->mmkey.key)); +} + int mlx5_ib_dereg_mr(struct ib_mr *ibmr, struct ib_udata *udata) { struct mlx5_ib_mr *mr = to_mmr(ibmr); @@ -1791,16 +1825,11 @@ int mlx5_ib_dereg_mr(struct ib_mr *ibmr, struct ib_udata *udata) } /* Stop DMA */ - if (mr->mmkey.cache_ent) { - xa_lock_irq(&mr->mmkey.cache_ent->mkeys); - mr->mmkey.cache_ent->in_use--; - xa_unlock_irq(&mr->mmkey.cache_ent->mkeys); - + if (mr->umem && mlx5r_umr_can_load_pas(dev, mr->umem->length)) if (mlx5r_umr_revoke_mr(mr) || - push_mkey(mr->mmkey.cache_ent, false, - xa_mk_value(mr->mmkey.key))) + cache_ent_find_and_store(dev, mr)) mr->mmkey.cache_ent = NULL; - } + if (!mr->mmkey.cache_ent) { rc = destroy_mkey(to_mdev(mr->ibmr.device), mr); if (rc) From patchwork Sun Jan 15 12:58:19 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michael Guralnik X-Patchwork-Id: 13102238 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7F91FC46467 for ; Sun, 15 Jan 2023 12:58:52 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231415AbjAOM6u (ORCPT ); Sun, 15 Jan 2023 07:58:50 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53486 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231167AbjAOM6s (ORCPT ); Sun, 15 Jan 2023 07:58:48 -0500 Received: from NAM11-DM6-obe.outbound.protection.outlook.com (mail-dm6nam11on2064.outbound.protection.outlook.com [40.107.223.64]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5A4F61259D for ; Sun, 15 Jan 2023 04:58:47 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=QXfoq/VvkQN5WHYUF6lzQEJ+PPDv9GFKsp1pJW3EPTefi/MlnOFqVIObggNSb49+bitaNum/lfg6UldS/cIU6PUn+Ze9ZBIr4Uo+x9RJyKSIGXWcz2ljuZ37bJFvA8k/q7v3yND945SMQDUtCrsPwwICfoQh816Lk9eGQdRss9oJiyB4Jjf+6qollCiK8K0AjLdtXllpIJCeH0ULUjT5kxNpF8A/6rh/ZjqRhB+WAyFj3J0wdcN6y9M0iUYfnj6aYdixKMpH3OnPXuZp+UNn/28wgXZu82S/nZz//Er/KYlP8KWNLRbyJZq/aSuPohzPe7LhK9IoMzm9jRlyJemfIQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=w1lKnolIlnlkH1/XoksQrc0sffUpzSqqSyozR5Womk0=; b=nSq7suiShci0/7C/Mm+uMr9mdoiHXJ6cfFWSKclr1Y084XA0LORayBtUMp53QXQN4JnpaZ6wq6dAN1dEic9KLXQWvvDOTSp2Iqg8uuaneO3o5uzCTd8zplWfaLl5BfELSwEH0CxOQHveN7/6JVBR0lHym8gBa4Q9bnblH3BPGN81myDcPoyp7CCME9PkEyR6S+ckMGXONtjoNITbricEz8rykj67k7jwUIGj3NKlSkEs8eZZ3RhReyynLDCN1dkaCloVGkjS1ecStHwDIvSSMiHxFPc5AYNa8Sc8U7XZVa6VVwuFhOVAi1WZJGl+3H/tLSZAvJemPtUQd57S+d+mDQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.118.232) smtp.rcpttodomain=vger.kernel.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=w1lKnolIlnlkH1/XoksQrc0sffUpzSqqSyozR5Womk0=; b=H6/GV84DrV7vSiz64yAtdvFKPxwL/unigRWgcQ6liTFMVJ9WUULxwrlOYXXMCyN7Pzsaa4BqmbY0UCkVQ1On4NE9X7Il2vJBJknDjZu1/eph1utEMZJ1pG2Hce13D3E4YY11Atkhs0VcVXwtdU/qLn0a+hhh2c/UPnB6qkbvs61KlaDiPL27rxFEhFEdsmNr9BJMHXN6voxcgadhZzKjBM5W2nRM+pIRmLNRmouXY8Px9zcd08sUvvP90bGkIfZwPUwcf7FE2FDkrQGJrieZAnpV0QcH5cKCLgy5z+b1pBPLbDZHbfIYXuH/ohIb+kOyjKapDU8Wt86cuPhjvBLzvw== Received: from MW4PR04CA0331.namprd04.prod.outlook.com (2603:10b6:303:8a::6) by SA1PR12MB6701.namprd12.prod.outlook.com (2603:10b6:806:251::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5986.23; Sun, 15 Jan 2023 12:58:45 +0000 Received: from CO1NAM11FT040.eop-nam11.prod.protection.outlook.com (2603:10b6:303:8a:cafe::c) by MW4PR04CA0331.outlook.office365.com (2603:10b6:303:8a::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6002.19 via Frontend Transport; Sun, 15 Jan 2023 12:58:45 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.118.232) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.118.232 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.118.232; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.118.232) by CO1NAM11FT040.mail.protection.outlook.com (10.13.174.140) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6002.13 via Frontend Transport; Sun, 15 Jan 2023 12:58:45 +0000 Received: from drhqmail201.nvidia.com (10.126.190.180) by mail.nvidia.com (10.127.129.5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.36; Sun, 15 Jan 2023 04:58:38 -0800 Received: from drhqmail202.nvidia.com (10.126.190.181) by drhqmail201.nvidia.com (10.126.190.180) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.36; Sun, 15 Jan 2023 04:58:37 -0800 Received: from vdi.nvidia.com (10.127.8.13) by mail.nvidia.com (10.126.190.181) with Microsoft SMTP Server id 15.2.986.36 via Frontend Transport; Sun, 15 Jan 2023 04:58:36 -0800 From: Michael Guralnik To: , , CC: , , Michael Guralnik Subject: [PATCH v3 rdma-next 6/6] RDMA/mlx5: Add work to remove temporary entries from the cache Date: Sun, 15 Jan 2023 14:58:19 +0200 Message-ID: <20230115125819.15034-7-michaelgur@nvidia.com> X-Mailer: git-send-email 2.17.2 In-Reply-To: <20230115125819.15034-1-michaelgur@nvidia.com> References: <20230115125819.15034-1-michaelgur@nvidia.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CO1NAM11FT040:EE_|SA1PR12MB6701:EE_ X-MS-Office365-Filtering-Correlation-Id: 98767717-a97d-4cd5-468f-08daf6f83c69 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 7ycE/l3fLf7N0OIuMS3fhYsUkVfLYh9UpAwWcjQVuxavKkS5eeFm/BzUfBQoLI8hs7vgqA0kex6iqy/VRPQZor9WaL5PVlS+EcAwNyrAnbMQwcteBulhAc3CcCoS0Nmr6ns4uaGPggEhu5tjmULM91okye7bVvGIT+FFBzWR6AEMLmq91OWFPdvCiF/Rhvk3+VfHmqRejkDsrJIDgsmsOsIaivYumoEcplgZj7B7YVzu/aux91MgP8gtmh44e68YhtXW28rpBQCSg+qC7gHIeJE1NQZ5+w6VbLzTA5QmTXxRqGbuKhg8uI9l30SWPFWP6JIpj9pj6R0LdkrJyCN/LqmcMkXYpgA6XZjHljQxSyl68KRue+SpTWF1BmFe5/2EYBLYMElXIdvNS7TfA3/fbMIJaQ7rLKV5iuGhMd8ahMYupPL8HZJUXYZk7+bL7k0G9lExPAfY1sShkd6+ZrWqNkPi+A9GbRDoKriiS3PWDRsJ7CTLMJra6GDhI/DwCSOSdh5GELoFd7jbR8tPvf+pZKty7cMoC6fstxL1+ojVZZC4vJz+TnFBod9asx3HhvoWlrDd2FyQpZSZduG2NBI2j3rU0ldUELb/xjCfoOhenXVhmy2vNu3bKoridNT0jW6kgOmO1k3i7Grb6s98BDuPhgJNDP8ubAjJO51b8QtlnEbrN7UJy5903N3iU+CKDPgoOwQ50tOVkjLEQo+eJLjRhfne6VU8DJaDNpFqYX3Pa28= X-Forefront-Antispam-Report: CIP:216.228.118.232;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc7edge1.nvidia.com;CAT:NONE;SFS:(13230022)(4636009)(376002)(39860400002)(136003)(346002)(396003)(451199015)(46966006)(36840700001)(40470700004)(36756003)(356005)(86362001)(8936002)(8676002)(4326008)(70586007)(70206006)(2906002)(5660300002)(36860700001)(82740400003)(7636003)(83380400001)(40460700003)(478600001)(7696005)(316002)(54906003)(110136005)(6666004)(40480700001)(82310400005)(41300700001)(2616005)(1076003)(47076005)(107886003)(426003)(336012)(26005)(186003)(2101003);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 15 Jan 2023 12:58:45.1263 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 98767717-a97d-4cd5-468f-08daf6f83c69 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.118.232];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CO1NAM11FT040.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA1PR12MB6701 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org The non-cache mkeys are stored in the cache only to shorten restarting application time. Don't store them longer than needed. Configure cache entries that store non-cache MRs as temporary entries. If 30 seconds have passed and no user reclaimed the temporarily cached mkeys, an asynchronous work will destroy the mkeys entries. Signed-off-by: Michael Guralnik Change-Id: I39249384d5a1ed87cf206ca902968cdd321689d1 --- drivers/infiniband/hw/mlx5/mlx5_ib.h | 9 ++- drivers/infiniband/hw/mlx5/mr.c | 94 ++++++++++++++++++++++------ drivers/infiniband/hw/mlx5/odp.c | 2 +- 3 files changed, 82 insertions(+), 23 deletions(-) diff --git a/drivers/infiniband/hw/mlx5/mlx5_ib.h b/drivers/infiniband/hw/mlx5/mlx5_ib.h index 6e0c0a931d78..8e22bb7d4c35 100644 --- a/drivers/infiniband/hw/mlx5/mlx5_ib.h +++ b/drivers/infiniband/hw/mlx5/mlx5_ib.h @@ -750,6 +750,7 @@ struct mlx5_cache_ent { struct rb_node node; struct mlx5r_cache_rb_key rb_key; + u8 is_tmp:1; u8 disabled:1; u8 fill_to_high_water:1; @@ -783,6 +784,7 @@ struct mlx5_mkey_cache { struct mutex rb_lock; struct dentry *fs_root; unsigned long last_add; + struct delayed_work remove_ent_dwork; }; struct mlx5_ib_port_resources { @@ -1326,9 +1328,10 @@ void mlx5_ib_copy_pas(u64 *old, u64 *new, int step, int num); int mlx5_ib_get_cqe_size(struct ib_cq *ibcq); int mlx5_mkey_cache_init(struct mlx5_ib_dev *dev); int mlx5_mkey_cache_cleanup(struct mlx5_ib_dev *dev); -struct mlx5_cache_ent *mlx5r_cache_create_ent(struct mlx5_ib_dev *dev, - struct mlx5r_cache_rb_key rb_key, - bool persistent_entry); +struct mlx5_cache_ent * +mlx5r_cache_create_ent_locked(struct mlx5_ib_dev *dev, + struct mlx5r_cache_rb_key rb_key, + bool persistent_entry); struct mlx5_ib_mr *mlx5_mr_cache_alloc(struct mlx5_ib_dev *dev, int access_flags, int access_mode, diff --git a/drivers/infiniband/hw/mlx5/mr.c b/drivers/infiniband/hw/mlx5/mr.c index 25e80529edd8..37f435cdcb52 100644 --- a/drivers/infiniband/hw/mlx5/mr.c +++ b/drivers/infiniband/hw/mlx5/mr.c @@ -140,19 +140,16 @@ static void create_mkey_warn(struct mlx5_ib_dev *dev, int status, void *out) mlx5_cmd_out_err(dev->mdev, MLX5_CMD_OP_CREATE_MKEY, 0, out); } - -static int push_mkey(struct mlx5_cache_ent *ent, bool limit_pendings, - void *to_store) +static int push_mkey_locked(struct mlx5_cache_ent *ent, bool limit_pendings, + void *to_store) { XA_STATE(xas, &ent->mkeys, 0); void *curr; - xa_lock_irq(&ent->mkeys); if (limit_pendings && - (ent->reserved - ent->stored) > MAX_PENDING_REG_MR) { - xa_unlock_irq(&ent->mkeys); + (ent->reserved - ent->stored) > MAX_PENDING_REG_MR) return -EAGAIN; - } + while (1) { /* * This is cmpxchg (NULL, XA_ZERO_ENTRY) however this version @@ -191,6 +188,7 @@ static int push_mkey(struct mlx5_cache_ent *ent, bool limit_pendings, break; xa_lock_irq(&ent->mkeys); } + xa_lock_irq(&ent->mkeys); if (xas_error(&xas)) return xas_error(&xas); if (WARN_ON(curr)) @@ -198,6 +196,17 @@ static int push_mkey(struct mlx5_cache_ent *ent, bool limit_pendings, return 0; } +static int push_mkey(struct mlx5_cache_ent *ent, bool limit_pendings, + void *to_store) +{ + int ret; + + xa_lock_irq(&ent->mkeys); + ret = push_mkey_locked(ent, limit_pendings, to_store); + xa_unlock_irq(&ent->mkeys); + return ret; +} + static void undo_push_reserve_mkey(struct mlx5_cache_ent *ent) { void *old; @@ -545,7 +554,7 @@ static void queue_adjust_cache_locked(struct mlx5_cache_ent *ent) { lockdep_assert_held(&ent->mkeys.xa_lock); - if (ent->disabled || READ_ONCE(ent->dev->fill_delay)) + if (ent->disabled || READ_ONCE(ent->dev->fill_delay) || ent->is_tmp) return; if (ent->stored < ent->limit) { ent->fill_to_high_water = true; @@ -675,7 +684,6 @@ static int mlx5_cache_ent_insert(struct mlx5_mkey_cache *cache, struct mlx5_cache_ent *cur; int cmp; - mutex_lock(&cache->rb_lock); /* Figure out where to put new node */ while (*new) { cur = rb_entry(*new, struct mlx5_cache_ent, node); @@ -695,7 +703,6 @@ static int mlx5_cache_ent_insert(struct mlx5_mkey_cache *cache, rb_link_node(&ent->node, parent, new); rb_insert_color(&ent->node, &cache->rb_root); - mutex_unlock(&cache->rb_lock); return 0; } @@ -867,9 +874,10 @@ static void delay_time_func(struct timer_list *t) WRITE_ONCE(dev->fill_delay, 0); } -struct mlx5_cache_ent *mlx5r_cache_create_ent(struct mlx5_ib_dev *dev, - struct mlx5r_cache_rb_key rb_key, - bool persistent_entry) +struct mlx5_cache_ent * +mlx5r_cache_create_ent_locked(struct mlx5_ib_dev *dev, + struct mlx5r_cache_rb_key rb_key, + bool persistent_entry) { struct mlx5_cache_ent *ent; int order; @@ -882,6 +890,7 @@ struct mlx5_cache_ent *mlx5r_cache_create_ent(struct mlx5_ib_dev *dev, xa_init_flags(&ent->mkeys, XA_FLAGS_LOCK_IRQ); ent->rb_key = rb_key; ent->dev = dev; + ent->is_tmp = !persistent_entry; INIT_DELAYED_WORK(&ent->dwork, delayed_cache_work_func); @@ -906,10 +915,43 @@ struct mlx5_cache_ent *mlx5r_cache_create_ent(struct mlx5_ib_dev *dev, mlx5_mkey_cache_debugfs_add_ent(dev, ent); } + else + mod_delayed_work(ent->dev->cache.wq, + &ent->dev->cache.remove_ent_dwork, + msecs_to_jiffies(30 * 1000)); return ent; } +static void remove_ent_work_func(struct work_struct *work) +{ + struct mlx5_mkey_cache *cache; + struct mlx5_cache_ent *ent; + struct rb_node *cur; + + cache = container_of(work, struct mlx5_mkey_cache, + remove_ent_dwork.work); + mutex_lock(&cache->rb_lock); + cur = rb_last(&cache->rb_root); + while (cur) { + ent = rb_entry(cur, struct mlx5_cache_ent, node); + cur = rb_prev(cur); + mutex_unlock(&cache->rb_lock); + + xa_lock_irq(&ent->mkeys); + if (!ent->is_tmp) { + xa_unlock_irq(&ent->mkeys); + mutex_lock(&cache->rb_lock); + continue; + } + xa_unlock_irq(&ent->mkeys); + + clean_keys(ent->dev, ent); + mutex_lock(&cache->rb_lock); + } + mutex_unlock(&cache->rb_lock); +} + int mlx5_mkey_cache_init(struct mlx5_ib_dev *dev) { struct mlx5_mkey_cache *cache = &dev->cache; @@ -925,6 +967,7 @@ int mlx5_mkey_cache_init(struct mlx5_ib_dev *dev) mutex_init(&dev->slow_path_mutex); mutex_init(&dev->cache.rb_lock); dev->cache.rb_root = RB_ROOT; + INIT_DELAYED_WORK(&dev->cache.remove_ent_dwork, remove_ent_work_func); cache->wq = alloc_ordered_workqueue("mkey_cache", WQ_MEM_RECLAIM); if (!cache->wq) { mlx5_ib_warn(dev, "failed to create work queue\n"); @@ -934,9 +977,10 @@ int mlx5_mkey_cache_init(struct mlx5_ib_dev *dev) mlx5_cmd_init_async_ctx(dev->mdev, &dev->async_ctx); timer_setup(&dev->delay_timer, delay_time_func, 0); mlx5_mkey_cache_debugfs_init(dev); + mutex_lock(&cache->rb_lock); for (i = 0; i <= mkey_cache_max_order(dev); i++) { rb_key.ndescs = 1 << (i + 2); - ent = mlx5r_cache_create_ent(dev, rb_key, true); + ent = mlx5r_cache_create_ent_locked(dev, rb_key, true); if (IS_ERR(ent)) { ret = PTR_ERR(ent); goto err; @@ -947,6 +991,7 @@ int mlx5_mkey_cache_init(struct mlx5_ib_dev *dev) if (ret) goto err; + mutex_unlock(&cache->rb_lock); for (node = rb_first(root); node; node = rb_next(node)) { ent = rb_entry(node, struct mlx5_cache_ent, node); xa_lock_irq(&ent->mkeys); @@ -957,6 +1002,7 @@ int mlx5_mkey_cache_init(struct mlx5_ib_dev *dev) return 0; err: + mutex_unlock(&cache->rb_lock); mlx5_ib_warn(dev, "failed to create mkey cache entry\n"); return ret; } @@ -970,6 +1016,7 @@ int mlx5_mkey_cache_cleanup(struct mlx5_ib_dev *dev) if (!dev->cache.wq) return 0; + cancel_delayed_work_sync(&dev->cache.remove_ent_dwork); mutex_lock(&dev->cache.rb_lock); for (node = rb_first(root); node; node = rb_next(node)) { ent = rb_entry(node, struct mlx5_cache_ent, node); @@ -1751,33 +1798,42 @@ static int cache_ent_find_and_store(struct mlx5_ib_dev *dev, { struct mlx5_mkey_cache *cache = &dev->cache; struct mlx5_cache_ent *ent; + int ret; if (mr->mmkey.cache_ent) { xa_lock_irq(&mr->mmkey.cache_ent->mkeys); mr->mmkey.cache_ent->in_use--; - xa_unlock_irq(&mr->mmkey.cache_ent->mkeys); goto end; } mutex_lock(&cache->rb_lock); ent = mkey_cache_ent_from_rb_key(dev, mr->mmkey.rb_key); - mutex_unlock(&cache->rb_lock); if (ent) { if (ent->rb_key.ndescs == mr->mmkey.rb_key.ndescs) { + if (ent->disabled) { + mutex_unlock(&cache->rb_lock); + return -EOPNOTSUPP; + } mr->mmkey.cache_ent = ent; + xa_lock_irq(&mr->mmkey.cache_ent->mkeys); + mutex_unlock(&cache->rb_lock); goto end; } } - ent = mlx5r_cache_create_ent(dev, mr->mmkey.rb_key, false); + ent = mlx5r_cache_create_ent_locked(dev, mr->mmkey.rb_key, false); + mutex_unlock(&cache->rb_lock); if (IS_ERR(ent)) return PTR_ERR(ent); mr->mmkey.cache_ent = ent; + xa_lock_irq(&mr->mmkey.cache_ent->mkeys); end: - return push_mkey(mr->mmkey.cache_ent, false, - xa_mk_value(mr->mmkey.key)); + ret = push_mkey_locked(mr->mmkey.cache_ent, false, + xa_mk_value(mr->mmkey.key)); + xa_unlock_irq(&mr->mmkey.cache_ent->mkeys); + return ret; } int mlx5_ib_dereg_mr(struct ib_mr *ibmr, struct ib_udata *udata) diff --git a/drivers/infiniband/hw/mlx5/odp.c b/drivers/infiniband/hw/mlx5/odp.c index c51d6c9a4c87..6f447095218f 100644 --- a/drivers/infiniband/hw/mlx5/odp.c +++ b/drivers/infiniband/hw/mlx5/odp.c @@ -1598,7 +1598,7 @@ int mlx5_odp_init_mkey_cache(struct mlx5_ib_dev *dev) if (!(dev->odp_caps.general_caps & IB_ODP_SUPPORT_IMPLICIT)) return 0; - ent = mlx5r_cache_create_ent(dev, rb_key, true); + ent = mlx5r_cache_create_ent_locked(dev, rb_key, true); if (IS_ERR(ent)) return PTR_ERR(ent);