From patchwork Mon Nov 7 16:14:42 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michael Guralnik X-Patchwork-Id: 13034652 X-Patchwork-Delegate: jgg@ziepe.ca Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 07BDCC433FE for ; Mon, 7 Nov 2022 16:15:06 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231646AbiKGQPE (ORCPT ); Mon, 7 Nov 2022 11:15:04 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39176 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232273AbiKGQPD (ORCPT ); Mon, 7 Nov 2022 11:15:03 -0500 Received: from NAM12-BN8-obe.outbound.protection.outlook.com (mail-bn8nam12on2086.outbound.protection.outlook.com [40.107.237.86]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D9B0A65E3 for ; Mon, 7 Nov 2022 08:15:02 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=ZsTYjGiYs7MsYU9EFBD/TOGuEOD3ZiVEGvCtpnl/6vQiGoyp8c59SMMqCzYAyxFKMGdGAf2LP7wEJHzTqNmRn5q2b1hUk5Q25oVpACyIqUEg0wtRMgEAOWz70W3/HkHc6GNBG9k3g75Ge8MFYmD6eaglNz4QK3sOqTOopMT/AfiJ44Oj4y/5UYGELjpmyBtloY87H7uBEwUHv51Yi+rekcA5EBY2T4JO+z7d2Iv+xOaL/uwqMvvetyB6KJO4nlGGebaxqVx/39aPnaSnWmtArBYyxQro4py+afzsxFnh5SzCgdju511QRCOFnB8aX5g1jnGDc640a4CV/UkPLRdPmQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=TvJW13wlajFFs0c8KichCn6x5YvOhm2H97SxGYFFQIU=; b=MgKvKlSC1P5cNSyXg3doY3yzRMp49VECqEv5iJIxsQ9T+MHVSwuAZbYA5YP5V68acU5otn3TFGYttYWEirdhtVVdWfl36TdQKTovgLA89okbMb2JNB6zaZ/l3t0owU+Vs057vq6vOEYGe3/Jl/Khrvk18OMPf4tWUVwNw5qGKkSP8LLbPp8/nHqFYRL845EBSRq9shBjuoeUiiOwRrsE27apsdQJIFBp8XgiQ0lo6DQln5gtQwWL2BsaiPpQMX26ADhu9G1rGup1bv9Me71ziQmRnXUd0XC2IsxwZaXfgF1L7EMBhZj0gXjN9YoQpH+zLa6Iuys8XqLS4caU2CGYoA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.118.232) smtp.rcpttodomain=vger.kernel.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=TvJW13wlajFFs0c8KichCn6x5YvOhm2H97SxGYFFQIU=; b=XqAV9Znag325zqUZHrTD4IxQPFTcnFQFaXZAuf4/PK8Aez/DQ2DJRUsDl8CyyY0hpD48SBez4NT4GKStfn6YAsj+11Q72eWz7HdibUy+5PSbgagcnKF1Hn+OsP9/OhgdL3B86L4DflqGiVr7AVudTOYqBndGeISFL8W5k12oz1/ydRCMpcsOUy68w3IWsvPXg2tNDYqCnszFlE/QRXjX1Hiku1BGAcrbR2omc30Z9WtSRhtkVwAo7HpH6J9WxXY5ageJV7WNhGpkAsPILED2Iq6+g4ZVH4hiI2qxXBZVh5AfX9E11ajFKpiBe7FqdvTwxQYsxarp7OZ+dNRKHiyYLQ== Received: from DM6PR08CA0019.namprd08.prod.outlook.com (2603:10b6:5:80::32) by SN7PR12MB6910.namprd12.prod.outlook.com (2603:10b6:806:262::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5791.26; Mon, 7 Nov 2022 16:15:01 +0000 Received: from DM6NAM11FT051.eop-nam11.prod.protection.outlook.com (2603:10b6:5:80:cafe::32) by DM6PR08CA0019.outlook.office365.com (2603:10b6:5:80::32) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5769.15 via Frontend Transport; Mon, 7 Nov 2022 16:15:01 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.118.232) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.118.232 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.118.232; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.118.232) by DM6NAM11FT051.mail.protection.outlook.com (10.13.172.243) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5791.20 via Frontend Transport; Mon, 7 Nov 2022 16:15:01 +0000 Received: from drhqmail203.nvidia.com (10.126.190.182) by mail.nvidia.com (10.127.129.5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.26; Mon, 7 Nov 2022 08:14:55 -0800 Received: from drhqmail201.nvidia.com (10.126.190.180) by drhqmail203.nvidia.com (10.126.190.182) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.29; Mon, 7 Nov 2022 08:14:55 -0800 Received: from vdi.nvidia.com (10.127.8.13) by mail.nvidia.com (10.126.190.180) with Microsoft SMTP Server id 15.2.986.29 via Frontend Transport; Mon, 7 Nov 2022 08:14:54 -0800 From: Michael Guralnik To: , , CC: , Subject: [PATCH v1 rdma-next 1/8] RDMA/mlx5: Don't keep umrable 'page_shift' in cache entries Date: Mon, 7 Nov 2022 18:14:42 +0200 Message-ID: <20221107161449.5611-2-michaelgur@nvidia.com> X-Mailer: git-send-email 2.17.2 In-Reply-To: <20221107161449.5611-1-michaelgur@nvidia.com> References: <20221107161449.5611-1-michaelgur@nvidia.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM6NAM11FT051:EE_|SN7PR12MB6910:EE_ X-MS-Office365-Filtering-Correlation-Id: fa00a968-808b-4203-a9d3-08dac0db38ed X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: IXHkOJ5DJejoMX9vEguScHGwJ2hlk2Mb4jQzYGgoxB8P1cfpwLGmh9qxIHTiZUF1FqSsaNkusfQxGtmqRHu4HvEuUCcH+z9D80AgisekJ9vAzbZ09boDMLDZ4lirhpoC2TIRPZL0svSXa3LSG6skTWXTHTr9oFijVhrbIIjIAkW+GtISFO1Pzmzn+ImbeKp+f0BX2k+beC6HqNSGa11ZtA8B0Gcuqsx+TY/yAg2CDI9Akf4OKDQV6YiHVZMaf6OQ7+yYRw0obRbaUYVD4cwRO02oxHIPFS4UoPUO6CMsIT/VSyGdrY5RwLJ0uMbG6qRHUPPH9NHQasjnRIlGLgUp25U1G4ve6Mm4/B6P+b/ezQDfO1Zf8Lizuam8nCJNocvccMJSwF22/vkpsYf1dJlyulGARfxdiuQpb2GI3orM5QaTMDhDOBPHOA5xaBqxEdbDdk+uG6XpZ1ZEmGmutKIVLNyK1NXawOpMjLa0mNJuNFBpMQunTJ+wXPuDEV/+2cm7hnueb4xC9OqYBezdqDR8GWt5YUiOI6WhScbrI7DGqidARoSSNh86Vdzkdz7mMKd6BK64Pcg/fqC3V/jf5UC8QckrxhG2l2iY2cb9laHJQ8iuSa5vYo5SY+EKPf2phtdVp0V7RSuPQN4MJF4r9dIM+xoF9HoOLBhOGBQrDwok6v43SFA8YDGh2ZxEHb5w561ROoqv93vkEVJAq5MwJThQrkOK0oyLKsyefKpSOeCflKTFXLpt5PK0Tq5EuMK958S6JwfVgDBeeLtnuK1+6FljeEPg3y4/ivAjuvJyGr6mTZA= X-Forefront-Antispam-Report: CIP:216.228.118.232;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc7edge1.nvidia.com;CAT:NONE;SFS:(13230022)(4636009)(136003)(39860400002)(376002)(396003)(346002)(451199015)(46966006)(40470700004)(36840700001)(2616005)(47076005)(426003)(1076003)(186003)(336012)(107886003)(7696005)(6666004)(26005)(36860700001)(83380400001)(2906002)(40480700001)(40460700003)(8676002)(82310400005)(110136005)(478600001)(316002)(54906003)(41300700001)(5660300002)(8936002)(4326008)(70586007)(70206006)(36756003)(7636003)(356005)(82740400003)(86362001)(2101003);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 07 Nov 2022 16:15:01.0410 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: fa00a968-808b-4203-a9d3-08dac0db38ed X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.118.232];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: DM6NAM11FT051.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN7PR12MB6910 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org From: Aharon Landau mkc.log_page_size can be changed using UMR. Therefore, don't treat it as a cache entry property. Removing it from struct mlx5_cache_ent. All cache mkeys will be created with default PAGE_SHIFT, and updated with the needed page_shift using UMR when passing them to a user. Signed-off-by: Aharon Landau --- drivers/infiniband/hw/mlx5/mlx5_ib.h | 1 - drivers/infiniband/hw/mlx5/mr.c | 3 +-- drivers/infiniband/hw/mlx5/odp.c | 2 -- 3 files changed, 1 insertion(+), 5 deletions(-) diff --git a/drivers/infiniband/hw/mlx5/mlx5_ib.h b/drivers/infiniband/hw/mlx5/mlx5_ib.h index 4a7f7064bd0e..ac948bb3cfe1 100644 --- a/drivers/infiniband/hw/mlx5/mlx5_ib.h +++ b/drivers/infiniband/hw/mlx5/mlx5_ib.h @@ -735,7 +735,6 @@ struct mlx5_cache_ent { char name[4]; u32 order; u32 access_mode; - u32 page; unsigned int ndescs; u8 disabled:1; diff --git a/drivers/infiniband/hw/mlx5/mr.c b/drivers/infiniband/hw/mlx5/mr.c index 410cc5fd2523..29ad674a9bcd 100644 --- a/drivers/infiniband/hw/mlx5/mr.c +++ b/drivers/infiniband/hw/mlx5/mr.c @@ -297,7 +297,7 @@ static void set_cache_mkc(struct mlx5_cache_ent *ent, void *mkc) MLX5_SET(mkc, mkc, translations_octword_size, get_mkc_octo_size(ent->access_mode, ent->ndescs)); - MLX5_SET(mkc, mkc, log_page_size, ent->page); + MLX5_SET(mkc, mkc, log_page_size, PAGE_SHIFT); } /* Asynchronously schedule new MRs to be populated in the cache. */ @@ -765,7 +765,6 @@ int mlx5_mkey_cache_init(struct mlx5_ib_dev *dev) if (ent->order > mkey_cache_max_order(dev)) continue; - ent->page = PAGE_SHIFT; ent->ndescs = 1 << ent->order; ent->access_mode = MLX5_MKC_ACCESS_MODE_MTT; if ((dev->mdev->profile.mask & MLX5_PROF_MASK_MR_CACHE) && diff --git a/drivers/infiniband/hw/mlx5/odp.c b/drivers/infiniband/hw/mlx5/odp.c index bc97958818bb..e9a29adef7dc 100644 --- a/drivers/infiniband/hw/mlx5/odp.c +++ b/drivers/infiniband/hw/mlx5/odp.c @@ -1595,14 +1595,12 @@ void mlx5_odp_init_mkey_cache_entry(struct mlx5_cache_ent *ent) switch (ent->order - 2) { case MLX5_IMR_MTT_CACHE_ENTRY: - ent->page = PAGE_SHIFT; ent->ndescs = MLX5_IMR_MTT_ENTRIES; ent->access_mode = MLX5_MKC_ACCESS_MODE_MTT; ent->limit = 0; break; case MLX5_IMR_KSM_CACHE_ENTRY: - ent->page = MLX5_KSM_PAGE_SHIFT; ent->ndescs = mlx5_imr_ksm_entries; ent->access_mode = MLX5_MKC_ACCESS_MODE_KSM; ent->limit = 0; From patchwork Mon Nov 7 16:14:43 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michael Guralnik X-Patchwork-Id: 13034654 X-Patchwork-Delegate: jgg@ziepe.ca Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 63083C4332F for ; Mon, 7 Nov 2022 16:15:10 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232459AbiKGQPJ (ORCPT ); Mon, 7 Nov 2022 11:15:09 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39250 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232702AbiKGQPH (ORCPT ); Mon, 7 Nov 2022 11:15:07 -0500 Received: from NAM10-MW2-obe.outbound.protection.outlook.com (mail-mw2nam10on2076.outbound.protection.outlook.com [40.107.94.76]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 580DD6466 for ; Mon, 7 Nov 2022 08:15:05 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=SW+EwAH0IjXvlj29Us8rZYp/7qCWfojoqm0NoWYQyehgSv8PZWvqdYGlQr1MA4209Kq+LlO8Gf5La6EOzZexfvOdTre0mgxUGapwsyowtMbri8rfHNzFQI11FO3R5DGKh9o1Ip87N6E5m0W53fXAPFg0jXuNp+5dV+UrqjV7+4Dr0c4/QoelnEK6S2fAx0Waebc1c/yHiaqPH1ydM4Uo0yYeSdK0PVL096/isksTRJADLazgONh2OuLTkzniWFyHuszdEbH3fuS0+Xt0IVGb/wsw94oqzbo+Hb70rY87Cm/eSV/5mAtxeAL5EV1vh1doqce5owVbwdSxLGS5C94Yrw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=Br0kq16L5UgO2GAGNxVdLvrn0t0X4FBPdIwRgi4wteQ=; b=OMm/6N40oa/K17a7LpFr/LfcmFYzCh/TaUmumoj1BEjmhw6gTIOZ1HegQ0a7ogm7gAsE2+4jXw9UxfyqQnvF+FIb14cw1mAKjuYZ2ewyMdELR4moxEflr5XK0vgLMKBthQ1mAIrQo3ut+yyyzKJGR3SjGjZmIxfpq1TeTvUUmdM/4X0DxAf1nIMGzc7rxlFz9794Phcnezb3C+YoI37QJUeqTGUWLGIbfWTk19NUXwUUCa7QSHkdHKa1cFl0R9Vy8GEclAZ4b9SiEvbYSM2kMYEu5hPWJFz06apxw8BhOzIF56UOOLRFEC9mDB+Z/+KsJlw5YskWHRiyRW13sO+R+Q== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.118.233) smtp.rcpttodomain=vger.kernel.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Br0kq16L5UgO2GAGNxVdLvrn0t0X4FBPdIwRgi4wteQ=; b=TLqZDcbv2wFTTw4+Nqz7yXxP1dNjfqBQakkGzAM1V3UMFRjfL6oPpKapcVQGodGRoTLO1nDOgqhuRYTAz25f7xp576Uhc9yOdMalCo2g/9f0p3ZnV52QL/Ra+uCH1FjfxlYAFoeMU31V096zKiwgsiKu0Nb4w5zExq9v+Ro2+OQDGCMf3Jf3i29a5UWDkmyZNqOtoK164ukhyho2JTt61alBqS68woskmvClewpuTCkGq57ERzG99Z1X4KuvhzY6XFscRZHdUVfkWq9x534t/WsxizPA4kxM8AFX13L5RFxqtQCC9AZDLFSk/K2pFtDgc+X88rQ8P8T9ciupStcVRw== Received: from DM6PR06CA0013.namprd06.prod.outlook.com (2603:10b6:5:120::26) by PH7PR12MB5904.namprd12.prod.outlook.com (2603:10b6:510:1d8::19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5791.25; Mon, 7 Nov 2022 16:15:01 +0000 Received: from DM6NAM11FT018.eop-nam11.prod.protection.outlook.com (2603:10b6:5:120:cafe::1d) by DM6PR06CA0013.outlook.office365.com (2603:10b6:5:120::26) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5791.26 via Frontend Transport; Mon, 7 Nov 2022 16:15:00 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.118.233) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.118.233 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.118.233; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.118.233) by DM6NAM11FT018.mail.protection.outlook.com (10.13.172.110) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5791.20 via Frontend Transport; Mon, 7 Nov 2022 16:15:00 +0000 Received: from drhqmail201.nvidia.com (10.126.190.180) by mail.nvidia.com (10.127.129.6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.26; Mon, 7 Nov 2022 08:14:57 -0800 Received: from drhqmail201.nvidia.com (10.126.190.180) by drhqmail201.nvidia.com (10.126.190.180) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.29; Mon, 7 Nov 2022 08:14:57 -0800 Received: from vdi.nvidia.com (10.127.8.13) by mail.nvidia.com (10.126.190.180) with Microsoft SMTP Server id 15.2.986.29 via Frontend Transport; Mon, 7 Nov 2022 08:14:55 -0800 From: Michael Guralnik To: , , CC: , Subject: [PATCH v1 rdma-next 2/8] RDMA/mlx5: Generalize mlx5_cache_cache_mr() to fit all cacheable mkeys Date: Mon, 7 Nov 2022 18:14:43 +0200 Message-ID: <20221107161449.5611-3-michaelgur@nvidia.com> X-Mailer: git-send-email 2.17.2 In-Reply-To: <20221107161449.5611-1-michaelgur@nvidia.com> References: <20221107161449.5611-1-michaelgur@nvidia.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM6NAM11FT018:EE_|PH7PR12MB5904:EE_ X-MS-Office365-Filtering-Correlation-Id: 61f365f3-c9bb-44c2-fcaa-08dac0db38c5 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: e3KusM7mj9a+k9kWYywLvNd6qv10DHcpNgzTCq1+8q1CwWxsyVof7hF3pECAhD49m12nVcm7Svy62b8JsdH5lJUFVFaHnHh1POi4siIhT5iAlo9aSRwIB0Emi7pfhecIdh8YGxKA9ocb8Z3FUltlgKK3XSh/S/hvazomqryYAeEy6pqh1m8arY1WDYrNEcTQhhB4mBpgNAPNRJgRehWOwvoNznuSI4VfdJzXWRXGVm25JY7skrahNLrERjBxB14dPAMxhntsJ1tDbsDgy1vZOGNgsC2XO48julkO7cu0ym1mAd664RbC+Jd4oTVhf5bskG8Ax5Q/fxd4HkTvSw2at61ZbGv6H4boNiwpXGYnZfNRidFTvRkfF+I5vyXQGQ4Sk1nQnOUyT52Bl2bQe0Doz0uXQDuZhcRga98cLhTDi0s/6uL8wSnQbvL/HpVgKWLOm9Bs71kBfD3x7JTkLvevD3Ib5YJWxYFWF0SoPtcIF/Z3QrvRO6owg2fM0eTgOehMVd+ZsPYhw5HM5btzGQbnIQtnO5aoxpknMCyoDRV+d1QcBctaYqwrKMMGL8+xFMXTb+1KwCe3k7qx6p9kqpS120i6QrxGvZwBdcIkRS8ohdG39R//3W36EWMyJcudS0m72FQdBsCOS5rcitTFvptB/kn2jGh4iNyOVqYEMeD1yAo/FiZJC/lZwA4HNxUnxCQ0sjfnUj3tC1xkFnDJzX73VG8MEATeX37+ouy6Uzizai8oS1nU/CqBg/lnTclrg8Rfk7QZzIIICNsyYg31gMlilJY285NlveGAap+3ZbBwB6o= X-Forefront-Antispam-Report: CIP:216.228.118.233;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc7edge2.nvidia.com;CAT:NONE;SFS:(13230022)(4636009)(346002)(376002)(136003)(39860400002)(396003)(451199015)(36840700001)(46966006)(40470700004)(40480700001)(36756003)(7636003)(478600001)(8936002)(26005)(82740400003)(86362001)(36860700001)(82310400005)(41300700001)(107886003)(426003)(186003)(1076003)(47076005)(7696005)(356005)(2616005)(336012)(40460700003)(70206006)(70586007)(4326008)(2906002)(316002)(5660300002)(83380400001)(8676002)(54906003)(6666004)(110136005)(2101003);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 07 Nov 2022 16:15:00.7498 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 61f365f3-c9bb-44c2-fcaa-08dac0db38c5 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.118.233];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: DM6NAM11FT018.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR12MB5904 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org From: Aharon Landau All the mkeys that can use UMR, can be cached. Generalize the flow so that creation of all mkeys that can be cached will go through to the cache instead of reg_create(). This will allow us to to extend the cache in following patches to support additional non-default cache entries. Signed-off-by: Aharon Landau --- drivers/infiniband/hw/mlx5/mlx5_ib.h | 6 +- drivers/infiniband/hw/mlx5/mr.c | 147 +++++++++++++++------------ drivers/infiniband/hw/mlx5/odp.c | 9 +- 3 files changed, 87 insertions(+), 75 deletions(-) diff --git a/drivers/infiniband/hw/mlx5/mlx5_ib.h b/drivers/infiniband/hw/mlx5/mlx5_ib.h index ac948bb3cfe1..84d3b917c33e 100644 --- a/drivers/infiniband/hw/mlx5/mlx5_ib.h +++ b/drivers/infiniband/hw/mlx5/mlx5_ib.h @@ -1313,9 +1313,9 @@ int mlx5_ib_get_cqe_size(struct ib_cq *ibcq); int mlx5_mkey_cache_init(struct mlx5_ib_dev *dev); int mlx5_mkey_cache_cleanup(struct mlx5_ib_dev *dev); -struct mlx5_ib_mr *mlx5_mr_cache_alloc(struct mlx5_ib_dev *dev, - struct mlx5_cache_ent *ent, - int access_flags); +struct mlx5_ib_mr *mlx5_mr_cache_alloc(struct mlx5_ib_dev *dev, u8 access_mode, + unsigned int access_flags, + unsigned int ndescs); int mlx5_ib_check_mr_status(struct ib_mr *ibmr, u32 check_mask, struct ib_mr_status *mr_status); diff --git a/drivers/infiniband/hw/mlx5/mr.c b/drivers/infiniband/hw/mlx5/mr.c index 29ad674a9bcd..8ff10944ff16 100644 --- a/drivers/infiniband/hw/mlx5/mr.c +++ b/drivers/infiniband/hw/mlx5/mr.c @@ -287,16 +287,18 @@ static int get_mkc_octo_size(unsigned int access_mode, unsigned int ndescs) return ret; } -static void set_cache_mkc(struct mlx5_cache_ent *ent, void *mkc) +static void set_cache_mkc(struct mlx5_ib_dev *dev, u8 access_mode, + unsigned int access_flags, unsigned int ndescs, + void *mkc) { - set_mkc_access_pd_addr_fields(mkc, 0, 0, ent->dev->umrc.pd); + set_mkc_access_pd_addr_fields(mkc, access_flags, 0, dev->umrc.pd); MLX5_SET(mkc, mkc, free, 1); MLX5_SET(mkc, mkc, umr_en, 1); - MLX5_SET(mkc, mkc, access_mode_1_0, ent->access_mode & 0x3); - MLX5_SET(mkc, mkc, access_mode_4_2, (ent->access_mode >> 2) & 0x7); + MLX5_SET(mkc, mkc, access_mode_1_0, access_mode & 0x3); + MLX5_SET(mkc, mkc, access_mode_4_2, (access_mode >> 2) & 0x7); MLX5_SET(mkc, mkc, translations_octword_size, - get_mkc_octo_size(ent->access_mode, ent->ndescs)); + get_mkc_octo_size(access_mode, ndescs)); MLX5_SET(mkc, mkc, log_page_size, PAGE_SHIFT); } @@ -315,7 +317,7 @@ static int add_keys(struct mlx5_cache_ent *ent, unsigned int num) return -ENOMEM; mkc = MLX5_ADDR_OF(create_mkey_in, async_create->in, memory_key_mkey_entry); - set_cache_mkc(ent, mkc); + set_cache_mkc(ent->dev, ent->access_mode, 0, ent->ndescs, mkc); async_create->ent = ent; err = push_mkey(ent, true, NULL); @@ -340,8 +342,10 @@ static int add_keys(struct mlx5_cache_ent *ent, unsigned int num) return err; } -/* Synchronously create a MR in the cache */ -static int create_cache_mkey(struct mlx5_cache_ent *ent, u32 *mkey) +/* Synchronously create a cacheable mkey */ +static int create_cache_mkey(struct mlx5_ib_dev *dev, u8 access_mode, + unsigned int access_flags, unsigned int ndescs, + struct mlx5_ib_mkey *mkey) { size_t inlen = MLX5_ST_SZ_BYTES(create_mkey_in); void *mkc; @@ -352,14 +356,9 @@ static int create_cache_mkey(struct mlx5_cache_ent *ent, u32 *mkey) if (!in) return -ENOMEM; mkc = MLX5_ADDR_OF(create_mkey_in, in, memory_key_mkey_entry); - set_cache_mkc(ent, mkc); - - err = mlx5_core_create_mkey(ent->dev->mdev, mkey, in, inlen); - if (err) - goto free_in; + set_cache_mkc(dev, access_mode, access_flags, ndescs, mkc); - WRITE_ONCE(ent->dev->cache.last_add, jiffies); -free_in: + err = mlx5_ib_create_mkey(dev, mkey, in, inlen); kfree(in); return err; } @@ -637,41 +636,80 @@ static void delayed_cache_work_func(struct work_struct *work) __cache_work_func(ent); } -struct mlx5_ib_mr *mlx5_mr_cache_alloc(struct mlx5_ib_dev *dev, - struct mlx5_cache_ent *ent, - int access_flags) +static bool mlx5_ent_get_mkey(struct mlx5_cache_ent *ent, struct mlx5_ib_mr *mr) { - struct mlx5_ib_mr *mr; - int err; + xa_lock_irq(&ent->mkeys); + if (!ent->stored) { + queue_adjust_cache_locked(ent); + ent->miss++; + xa_unlock_irq(&ent->mkeys); + return false; + } + + mr->mmkey.key = pop_stored_mkey(ent); + mr->mmkey.ndescs = ent->ndescs; + mr->mmkey.cache_ent = ent; + queue_adjust_cache_locked(ent); + ent->in_use++; + xa_unlock_irq(&ent->mkeys); + return true; +} + +static struct mlx5_cache_ent *mkey_cache_ent_from_order(struct mlx5_ib_dev *dev, + unsigned int order) +{ + struct mlx5_mkey_cache *cache = &dev->cache; + + if (order < cache->ent[0].order) + return &cache->ent[0]; + order = order - cache->ent[0].order; + if (order > MKEY_CACHE_LAST_STD_ENTRY) + return NULL; + return &cache->ent[order]; +} + +static bool mlx5_cache_get_mkey(struct mlx5_ib_dev *dev, u8 access_mode, + u8 access_flags, unsigned int ndescs, + struct mlx5_ib_mr *mr) +{ + struct mlx5_cache_ent *ent; if (!mlx5r_umr_can_reconfig(dev, 0, access_flags)) - return ERR_PTR(-EOPNOTSUPP); + return false; + + if (access_mode == MLX5_MKC_ACCESS_MODE_KSM) + ent = &dev->cache.ent[MLX5_IMR_KSM_CACHE_ENTRY]; + + ent = mkey_cache_ent_from_order(dev, order_base_2(ndescs)); + if (!ent) + return false; + + return mlx5_ent_get_mkey(ent, mr); +} + +struct mlx5_ib_mr *mlx5_mr_cache_alloc(struct mlx5_ib_dev *dev, u8 access_mode, + unsigned int access_flags, + unsigned int ndescs) +{ + struct mlx5_ib_mr *mr; + int err; mr = kzalloc(sizeof(*mr), GFP_KERNEL); if (!mr) return ERR_PTR(-ENOMEM); - xa_lock_irq(&ent->mkeys); - ent->in_use++; - - if (!ent->stored) { - queue_adjust_cache_locked(ent); - ent->miss++; - xa_unlock_irq(&ent->mkeys); - err = create_cache_mkey(ent, &mr->mmkey.key); + if (!mlx5_cache_get_mkey(dev, access_mode, access_flags, ndescs, mr)) { + /* + * Didn't find an mkey in cache. + * Create an mkey with the exact needed size. + */ + err = create_cache_mkey(dev, access_mode, access_flags, ndescs, + &mr->mmkey); if (err) { - xa_lock_irq(&ent->mkeys); - ent->in_use--; - xa_unlock_irq(&ent->mkeys); kfree(mr); return ERR_PTR(err); } - } else { - mr->mmkey.key = pop_stored_mkey(ent); - queue_adjust_cache_locked(ent); - xa_unlock_irq(&ent->mkeys); } - mr->mmkey.cache_ent = ent; mr->mmkey.type = MLX5_MKEY_MR; init_waitqueue_head(&mr->mmkey.wait); return mr; @@ -876,19 +914,6 @@ static int mkey_cache_max_order(struct mlx5_ib_dev *dev) return MLX5_MAX_UMR_SHIFT; } -static struct mlx5_cache_ent *mkey_cache_ent_from_order(struct mlx5_ib_dev *dev, - unsigned int order) -{ - struct mlx5_mkey_cache *cache = &dev->cache; - - if (order < cache->ent[0].order) - return &cache->ent[0]; - order = order - cache->ent[0].order; - if (order > MKEY_CACHE_LAST_STD_ENTRY) - return NULL; - return &cache->ent[order]; -} - static void set_mr_fields(struct mlx5_ib_dev *dev, struct mlx5_ib_mr *mr, u64 length, int access_flags, u64 iova) { @@ -916,9 +941,8 @@ static struct mlx5_ib_mr *alloc_cacheable_mr(struct ib_pd *pd, int access_flags) { struct mlx5_ib_dev *dev = to_mdev(pd->device); - struct mlx5_cache_ent *ent; + unsigned int page_size, ndescs; struct mlx5_ib_mr *mr; - unsigned int page_size; if (umem->is_dmabuf) page_size = mlx5_umem_dmabuf_default_pgsz(umem, iova); @@ -927,22 +951,11 @@ static struct mlx5_ib_mr *alloc_cacheable_mr(struct ib_pd *pd, 0, iova); if (WARN_ON(!page_size)) return ERR_PTR(-EINVAL); - ent = mkey_cache_ent_from_order( - dev, order_base_2(ib_umem_num_dma_blocks(umem, page_size))); - /* - * Matches access in alloc_cache_mr(). If the MR can't come from the - * cache then synchronously create an uncached one. - */ - if (!ent || ent->limit == 0 || - !mlx5r_umr_can_reconfig(dev, 0, access_flags) || - mlx5_umem_needs_ats(dev, umem, access_flags)) { - mutex_lock(&dev->slow_path_mutex); - mr = reg_create(pd, umem, iova, access_flags, page_size, false); - mutex_unlock(&dev->slow_path_mutex); - return mr; - } - mr = mlx5_mr_cache_alloc(dev, ent, access_flags); + ndescs = ib_umem_num_dma_blocks(umem, page_size); + + mr = mlx5_mr_cache_alloc(dev, MLX5_MKC_ACCESS_MODE_MTT, + access_flags, ndescs); if (IS_ERR(mr)) return mr; diff --git a/drivers/infiniband/hw/mlx5/odp.c b/drivers/infiniband/hw/mlx5/odp.c index e9a29adef7dc..f7c9eeaa8e79 100644 --- a/drivers/infiniband/hw/mlx5/odp.c +++ b/drivers/infiniband/hw/mlx5/odp.c @@ -418,8 +418,8 @@ static struct mlx5_ib_mr *implicit_get_child_mr(struct mlx5_ib_mr *imr, if (IS_ERR(odp)) return ERR_CAST(odp); - mr = mlx5_mr_cache_alloc(dev, &dev->cache.ent[MLX5_IMR_MTT_CACHE_ENTRY], - imr->access_flags); + mr = mlx5_mr_cache_alloc(dev, MLX5_MKC_ACCESS_MODE_MTT, + imr->access_flags, MLX5_IMR_MTT_ENTRIES); if (IS_ERR(mr)) { ib_umem_odp_release(odp); return mr; @@ -493,9 +493,8 @@ struct mlx5_ib_mr *mlx5_ib_alloc_implicit_mr(struct mlx5_ib_pd *pd, if (IS_ERR(umem_odp)) return ERR_CAST(umem_odp); - imr = mlx5_mr_cache_alloc(dev, - &dev->cache.ent[MLX5_IMR_KSM_CACHE_ENTRY], - access_flags); + imr = mlx5_mr_cache_alloc(dev, MLX5_MKC_ACCESS_MODE_KSM, access_flags, + mlx5_imr_ksm_entries); if (IS_ERR(imr)) { ib_umem_odp_release(umem_odp); return imr; From patchwork Mon Nov 7 16:14:44 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michael Guralnik X-Patchwork-Id: 13034653 X-Patchwork-Delegate: jgg@ziepe.ca Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C9852C433FE for ; Mon, 7 Nov 2022 16:15:09 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232258AbiKGQPI (ORCPT ); Mon, 7 Nov 2022 11:15:08 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39252 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232725AbiKGQPH (ORCPT ); Mon, 7 Nov 2022 11:15:07 -0500 Received: from NAM04-MW2-obe.outbound.protection.outlook.com (mail-mw2nam04on2085.outbound.protection.outlook.com [40.107.101.85]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A579D2737 for ; Mon, 7 Nov 2022 08:15:06 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=j6dy78Zh5BE0z/hmTXt/96dJ9aE2yLIVdN2zpm4z6U6/g+6uSH0YrDcZ9xjjuDUJnUe7uquoyLFgr0lboS0kUVn04Y/XNhmh4Z9zfJdyA5tSn6vWvhx4JguuQg3wsy1hQ0fsqYN0wffCDqioqjw0clcv8y0fdJyeBGrQSes1Jyt0A9wrg35W1u2XEWtEOwGulWdvGGevAveWw7AqUxKFGT4UzedoYBnxbUDM+ZmBXJDe0Hv7A9PW4iGBwLUGdBp86Ht54Y4gHqGPMy4dIAPR20zDYsMpj4UNE7MMQAuMfmnWZAkxjmE2szYP3N7isbXOz7ZtLwSiqlsY39U06h5L0w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=7ZwWeRDh+jRV/9MGXPBY0mEL+YbAyqUIYblFR78ssNk=; b=QbDBJpQZJny8edtjiSmrSC5Sn7R9rILU50pHLGOYpjfkNNUp+gZXUNPwJKchG17ReF8Z30ZMhKc4zzXwf4UmwAFFxh/h7eJcgR51ONFC1qn1OO7TdGrFcnGQB9E9+s0I65czJWSKdDDpohdr184Mk4PBYCnybOwwnSSvZp5J1LhGzYmfmTzS+sRxraxqTRsfc4mw0KWe1nffRvMpbybiIVO5osuB2RiHxiOMDIln5F7OArjL7S8wwjUdRk31nJOHJDd2lzMmRXQTpE87lQDemnyu4QZ8le9IeWWwAr78e0pkt2kO+wR5rQisJ1CCn1gREvHU6xTNu2lEOvp0wI6PkA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.118.233) smtp.rcpttodomain=vger.kernel.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=7ZwWeRDh+jRV/9MGXPBY0mEL+YbAyqUIYblFR78ssNk=; b=N4aGqKGcbrXyr1u87FmiMKV/Pl888vLEiikTM/1kr2DGJ90H77MxP4eLBPsPU6Eau93xk19bc619fPT7EXIqhMHLWwOhasj7C71baQFiBbwue2doX2BV/onD3hwWUTRaguPtLBCKh8C4U5d9B5siR9y7V4OMwvOUI8JDZbPGIW2VQDbO2MTyA82So0hOLlW71maCWoZXgJmAVk5NEfnDb8m/lyS6oYlgkINva1W9Wva9PvB1OOQn5asE2jboDK5HsOuIA72BPhS8nWrSPP0XOMQFh4v2ZNV52qCEILAuWnDX8b1//snzbf5B6+0OmDjUoF6H0SmqG/Fd+YE0hYdB4g== Received: from DM6PR06CA0015.namprd06.prod.outlook.com (2603:10b6:5:120::28) by SJ0PR12MB6710.namprd12.prod.outlook.com (2603:10b6:a03:44c::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5791.25; Mon, 7 Nov 2022 16:15:04 +0000 Received: from DM6NAM11FT018.eop-nam11.prod.protection.outlook.com (2603:10b6:5:120:cafe::a0) by DM6PR06CA0015.outlook.office365.com (2603:10b6:5:120::28) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5791.26 via Frontend Transport; Mon, 7 Nov 2022 16:15:03 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.118.233) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.118.233 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.118.233; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.118.233) by DM6NAM11FT018.mail.protection.outlook.com (10.13.172.110) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5791.20 via Frontend Transport; Mon, 7 Nov 2022 16:15:03 +0000 Received: from drhqmail203.nvidia.com (10.126.190.182) by mail.nvidia.com (10.127.129.6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.26; Mon, 7 Nov 2022 08:14:59 -0800 Received: from drhqmail201.nvidia.com (10.126.190.180) by drhqmail203.nvidia.com (10.126.190.182) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.29; Mon, 7 Nov 2022 08:14:59 -0800 Received: from vdi.nvidia.com (10.127.8.13) by mail.nvidia.com (10.126.190.180) with Microsoft SMTP Server id 15.2.986.29 via Frontend Transport; Mon, 7 Nov 2022 08:14:57 -0800 From: Michael Guralnik To: , , CC: , Subject: [PATCH v1 rdma-next 3/8] RDMA/mlx5: Remove explicit ODP cache entry Date: Mon, 7 Nov 2022 18:14:44 +0200 Message-ID: <20221107161449.5611-4-michaelgur@nvidia.com> X-Mailer: git-send-email 2.17.2 In-Reply-To: <20221107161449.5611-1-michaelgur@nvidia.com> References: <20221107161449.5611-1-michaelgur@nvidia.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM6NAM11FT018:EE_|SJ0PR12MB6710:EE_ X-MS-Office365-Filtering-Correlation-Id: f74bd1a0-a7ac-4e03-575f-08dac0db3a6e X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 5iPFdWaQBrzWbEYH/pG5vA3bSl+YJAshP7L4Fd5COO70otrhEIljD329BxbvvtM+EpBdEHEPjqxw+VPdiiTEJn9dSrWbVjUeKYc7R8DXfGcPUiHWVtLjsl1WQF87OZYNbxBZzxOER5/U8tGAp9+7X5/rIDoikmBEKNFDhmncFpm1fnwhRUupALIF1i7kFFRZjW1p1azAWtSk/vmHWT6sgj/ZFmhQt+LJ8gvaMmv+ertOV3wc7qSuM4HPzLyn7T06UGNfvGbXs+0829BL0sHIxp2ogPcTuleW0pR8WgMx0HA1b9DtwbYDFdUhqLTOh+3yHDSTvXMrfdaoNP+SxK0PMEMOZxxDmJ8090r9uFE5DSwlT2vOlKzXpBGs1iHIie/w7SkGh1lzAsOVjzuLgu/0Kw0x17iapo1X8OaToTHDQO2Pd7uc/k01sGqfZJvbPCCLkC6G3xwjDDP4XNt9cNRTlqA/IQ+eRuaDiwQ727Ww97wA397cy2mww4hRkOTorqABoLpiLc8zytXKeMNkIsKtzR/jkhH1BxhL6cQ7apP7rG++3hEl113PKRAQtmBUWTIl/YA8VUpFVNYldo4FD5dBCAcOPLjJmUbuUeMaNRpdMsaU25kinNfr1tQp4NZSF0XUaxNy5n4FREMOXJn4gaUmnuGuWoNv10R4PbtP3oHYj5IuCqQGLEJ5sNiRQJswsHQBfp8VP3DKimviwCksqJZBZ++/WBcZ9VB0OK0otDZ/CZXVk5arL2hhlYbG68QpCppIHqafwmeITU1jG4OvaUUuTeV0Bsdh7MbTawGglpQzVP8= X-Forefront-Antispam-Report: CIP:216.228.118.233;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc7edge2.nvidia.com;CAT:NONE;SFS:(13230022)(4636009)(396003)(39860400002)(346002)(136003)(376002)(451199015)(46966006)(36840700001)(40470700004)(82310400005)(5660300002)(2906002)(36756003)(6666004)(7696005)(107886003)(82740400003)(70206006)(70586007)(8676002)(4326008)(36860700001)(316002)(54906003)(110136005)(7636003)(86362001)(356005)(83380400001)(41300700001)(40480700001)(336012)(26005)(40460700003)(8936002)(426003)(47076005)(1076003)(186003)(2616005)(478600001)(2101003);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 07 Nov 2022 16:15:03.5621 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: f74bd1a0-a7ac-4e03-575f-08dac0db3a6e X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.118.233];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: DM6NAM11FT018.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR12MB6710 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org From: Aharon Landau Explicit ODP mkey doesn't have unique properties. There is no need to devote to it a special entry. Removing it. Signed-off-by: Aharon Landau --- drivers/infiniband/hw/mlx5/odp.c | 16 ++-------------- include/linux/mlx5/driver.h | 1 - 2 files changed, 2 insertions(+), 15 deletions(-) diff --git a/drivers/infiniband/hw/mlx5/odp.c b/drivers/infiniband/hw/mlx5/odp.c index f7c9eeaa8e79..137143da5959 100644 --- a/drivers/infiniband/hw/mlx5/odp.c +++ b/drivers/infiniband/hw/mlx5/odp.c @@ -1591,20 +1591,8 @@ void mlx5_odp_init_mkey_cache_entry(struct mlx5_cache_ent *ent) { if (!(ent->dev->odp_caps.general_caps & IB_ODP_SUPPORT_IMPLICIT)) return; - - switch (ent->order - 2) { - case MLX5_IMR_MTT_CACHE_ENTRY: - ent->ndescs = MLX5_IMR_MTT_ENTRIES; - ent->access_mode = MLX5_MKC_ACCESS_MODE_MTT; - ent->limit = 0; - break; - - case MLX5_IMR_KSM_CACHE_ENTRY: - ent->ndescs = mlx5_imr_ksm_entries; - ent->access_mode = MLX5_MKC_ACCESS_MODE_KSM; - ent->limit = 0; - break; - } + ent->ndescs = mlx5_imr_ksm_entries; + ent->access_mode = MLX5_MKC_ACCESS_MODE_KSM; } static const struct ib_device_ops mlx5_ib_dev_odp_ops = { diff --git a/include/linux/mlx5/driver.h b/include/linux/mlx5/driver.h index a12929bc31b2..bc13143f1a87 100644 --- a/include/linux/mlx5/driver.h +++ b/include/linux/mlx5/driver.h @@ -736,7 +736,6 @@ enum { enum { MKEY_CACHE_LAST_STD_ENTRY = 20, - MLX5_IMR_MTT_CACHE_ENTRY, MLX5_IMR_KSM_CACHE_ENTRY, MAX_MKEY_CACHE_ENTRIES }; From patchwork Mon Nov 7 16:14:45 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michael Guralnik X-Patchwork-Id: 13034656 X-Patchwork-Delegate: jgg@ziepe.ca Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id BF136C4332F for ; Mon, 7 Nov 2022 16:15:19 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232776AbiKGQPS (ORCPT ); Mon, 7 Nov 2022 11:15:18 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39412 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232725AbiKGQPQ (ORCPT ); Mon, 7 Nov 2022 11:15:16 -0500 Received: from NAM02-SN1-obe.outbound.protection.outlook.com (mail-sn1anam02on2050.outbound.protection.outlook.com [40.107.96.50]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 707016466 for ; Mon, 7 Nov 2022 08:15:11 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=UITEUmA4h/riFfSAmumbAZOKLnd/sV5EXCyBb5ljRDJdyda2pTVsAodpu/rJG7ZE+a9qaTt+IjG9GhlXMQrVP6n2FH2MwKWDIvw/WMVhmdLodD+hRR83teu1THcdWuGX8D661GUT2SI4CcZiYE9gawBzoBHEEBTYaqvoAA5r3ZvchG+lWbNn11GseqgGuXYh5SNUqCi/iAy2uQxAkJeh4sykX/cYb5m8rT0DWH/o7TcxE8hsd9kWO79ERRodwCnPOSNAstAL1yavBZ23jgG/kLSZ0kp+/V6deGkSAdyIMFNiu4ATqEXuX+17nVwmAx3iScnMzNf+cnbiBCJUc/oUig== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=N2VkzcA0wbArXp1xXbA4tyOBtgr6XmD0B1qi4RvkWhQ=; b=JNprs5fqaPypfwI2yVbqSPjP++UU6y2Qr1VFyc7ke+lRb9laKhoIz6UCOgpTGE+Rr0yR8WH1Fsyv5V4LXiOIA9Z+8Z3wYoe2igAgn1CJMusB5Tz1LS7XNHhMB4OiEqq6GWUz5uVPQjN7IgJqGyvBhXmvpKf4q+3dBJvaIzaXMLfXFPmqbctuYwfiZrj3t2EgzQ1v4YpaCrmHCPwusWE13QcgWRWcq1lrYD2XovVxrmoaca3iYE70CinZR+NJmwO1umM22NvP0tMHBNfQ9KTb/X/+dmifpycKK79eFENxHXf+bvGXgvumdcpVF7fwFTXjlIUNOppKjIFN3AqMdAwgsw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.118.233) smtp.rcpttodomain=vger.kernel.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=N2VkzcA0wbArXp1xXbA4tyOBtgr6XmD0B1qi4RvkWhQ=; b=R345SjWOWtdHN+tf3tNfKUz3EdO7FtJf93iefNYU5nljGhhEVlFMBVRj5jTDfb/OXYxpZVHXuMp1qM2rNQK/h6ZMlvL64ga4FThG/cIKVDC8p9/9KnDRFJd8WKz+zizJIxE0Bx5OEjg1i9XUkX5Zy94C2KuHe6yoQD8bLZSwqCzMNJHYZ63ZAAH/4ZW22vNDW2+zzD4hPfuc/LUtgf79wieo1EriaR2D3oJVEmTZ46O4kSVW5umA/QUp/WhUh06htNHvAsrZdRU4Gp6Fu9cCvvbxuDGTBuLqJ2qyx0Ryvcing0zm8u62TMdLeLgOBAmtOBNx6b/IKscEVzXKRnEzhg== Received: from DM6PR07CA0125.namprd07.prod.outlook.com (2603:10b6:5:330::7) by CY5PR12MB6406.namprd12.prod.outlook.com (2603:10b6:930:3d::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5791.22; Mon, 7 Nov 2022 16:15:09 +0000 Received: from DM6NAM11FT094.eop-nam11.prod.protection.outlook.com (2603:10b6:5:330:cafe::72) by DM6PR07CA0125.outlook.office365.com (2603:10b6:5:330::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5791.26 via Frontend Transport; Mon, 7 Nov 2022 16:15:09 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.118.233) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.118.233 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.118.233; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.118.233) by DM6NAM11FT094.mail.protection.outlook.com (10.13.172.195) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5791.20 via Frontend Transport; Mon, 7 Nov 2022 16:15:08 +0000 Received: from drhqmail201.nvidia.com (10.126.190.180) by mail.nvidia.com (10.127.129.6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.26; Mon, 7 Nov 2022 08:15:01 -0800 Received: from drhqmail201.nvidia.com (10.126.190.180) by drhqmail201.nvidia.com (10.126.190.180) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.29; Mon, 7 Nov 2022 08:15:01 -0800 Received: from vdi.nvidia.com (10.127.8.13) by mail.nvidia.com (10.126.190.180) with Microsoft SMTP Server id 15.2.986.29 via Frontend Transport; Mon, 7 Nov 2022 08:14:59 -0800 From: Michael Guralnik To: , , CC: , Subject: [PATCH v1 rdma-next 4/8] RDMA/mlx5: Allow rereg all the mkeys that can load pas with UMR Date: Mon, 7 Nov 2022 18:14:45 +0200 Message-ID: <20221107161449.5611-5-michaelgur@nvidia.com> X-Mailer: git-send-email 2.17.2 In-Reply-To: <20221107161449.5611-1-michaelgur@nvidia.com> References: <20221107161449.5611-1-michaelgur@nvidia.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM6NAM11FT094:EE_|CY5PR12MB6406:EE_ X-MS-Office365-Filtering-Correlation-Id: c9a6595d-8aed-4318-6d7c-08dac0db3d8e X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: mmR6obKeGWSGfMbtPugXnT/20IUp12juxXzx3yPVNcpqQFNQf/qi7NIgdIWL4Frucie5XWvftb/tiC23NCakg22cr5/ESc0xJR/MHhqIPjLAkaXoVrLrp0aPQnMMgJxGL5k7luT3foGZqgwZc6gV6LwE/zD+AlqgJmYfzsXVdGW0DyVQZTKaygABKozuy5JgNtVTjNcaV4rioKZIJyY0AaMJi0uaz3BNygZDQq1D4DZsZXJKS/A0PJI6f+dvV//RH0hwpjg0hgTZJeqe3PU4FnTYZiRV7FiH/HRS3IMRFn+ZEU+lgwbxCoKXlfLa+bksX3kc6pepl68VZioVjV0B3ohS6fv8HjQKNpJEGZS5ojkMusvfjizcbRRB9VDYegODyE0Q2XlzuqxRxJveYRnSel4i8+7t+tiYt6st+1/SI5mB7M6TD1bYy6H19BuZySpQwYC0vm+LG8yWVAT+3poNWzjshE6J8nq9KAwVgOMX/8BYSd3XA6vhLZW2Bszwn4mubptGAmp6xYfjut0/FbL9yUwezY3qYSbwIr03D//Cxl0F6ka7bLzX1Jmtuai5fcDlFKVbFM5vpD0go7XQeR9ObCFaSYl8JxtL3w9ZRP7sFpv7zYIfrbo4zB8nDMudwgKVSG4klLnoPpO25UOI8DC/Cx/95zIZknS4aYF1XXCRzzqDu2/5+2RH4+VMs5Sz+wDxgi4s45NantvLg98jRiEf2r1TN/zuqLJyGhHrBByJTGXHZ+B/Hgp+O12h7BNXt/P3G4oFH1DwOwzjMvct8xLnM/dv0girFzMtxHk6jvt6DA4= X-Forefront-Antispam-Report: CIP:216.228.118.233;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc7edge2.nvidia.com;CAT:NONE;SFS:(13230022)(4636009)(376002)(396003)(136003)(39860400002)(346002)(451199015)(40470700004)(46966006)(36840700001)(82310400005)(36756003)(70586007)(2906002)(7696005)(107886003)(36860700001)(82740400003)(8936002)(6666004)(316002)(40480700001)(5660300002)(54906003)(4326008)(41300700001)(86362001)(7636003)(8676002)(2616005)(70206006)(356005)(478600001)(40460700003)(186003)(47076005)(336012)(26005)(110136005)(83380400001)(1076003)(426003)(2101003);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 07 Nov 2022 16:15:08.7274 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: c9a6595d-8aed-4318-6d7c-08dac0db3d8e X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.118.233];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: DM6NAM11FT094.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY5PR12MB6406 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org From: Aharon Landau Keep track of the mkey size of all cacheable mkeys, and by this allow to rereg them. Signed-off-by: Aharon Landau --- drivers/infiniband/hw/mlx5/mr.c | 7 ++----- 1 file changed, 2 insertions(+), 5 deletions(-) diff --git a/drivers/infiniband/hw/mlx5/mr.c b/drivers/infiniband/hw/mlx5/mr.c index 8ff10944ff16..fe5567c57897 100644 --- a/drivers/infiniband/hw/mlx5/mr.c +++ b/drivers/infiniband/hw/mlx5/mr.c @@ -709,6 +709,7 @@ struct mlx5_ib_mr *mlx5_mr_cache_alloc(struct mlx5_ib_dev *dev, u8 access_mode, kfree(mr); return ERR_PTR(err); } + mr->mmkey.ndescs = ndescs; } mr->mmkey.type = MLX5_MKEY_MR; init_waitqueue_head(&mr->mmkey.wait); @@ -1374,9 +1375,6 @@ static bool can_use_umr_rereg_pas(struct mlx5_ib_mr *mr, { struct mlx5_ib_dev *dev = to_mdev(mr->ibmr.device); - /* We only track the allocated sizes of MRs from the cache */ - if (!mr->mmkey.cache_ent) - return false; if (!mlx5r_umr_can_load_pas(dev, new_umem->length)) return false; @@ -1384,8 +1382,7 @@ static bool can_use_umr_rereg_pas(struct mlx5_ib_mr *mr, mlx5_umem_find_best_pgsz(new_umem, mkc, log_page_size, 0, iova); if (WARN_ON(!*page_size)) return false; - return (1ULL << mr->mmkey.cache_ent->order) >= - ib_umem_num_dma_blocks(new_umem, *page_size); + return mr->mmkey.ndescs >= ib_umem_num_dma_blocks(new_umem, *page_size); } static int umr_rereg_pas(struct mlx5_ib_mr *mr, struct ib_pd *pd, From patchwork Mon Nov 7 16:14:46 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michael Guralnik X-Patchwork-Id: 13034655 X-Patchwork-Delegate: jgg@ziepe.ca Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B7DFEC433FE for ; Mon, 7 Nov 2022 16:15:18 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232511AbiKGQPR (ORCPT ); Mon, 7 Nov 2022 11:15:17 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39402 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232464AbiKGQPQ (ORCPT ); Mon, 7 Nov 2022 11:15:16 -0500 Received: from NAM11-CO1-obe.outbound.protection.outlook.com (mail-co1nam11on2053.outbound.protection.outlook.com [40.107.220.53]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3519FB48B for ; Mon, 7 Nov 2022 08:15:14 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=nr7TXi2dBvCVEv42IU2zOzj/3V1YtZf/vt+tnolfLIh3nwO7NClnZJWDc79Q7wg3gGX5/r85N+a0v2EzMmiGr53JS5Vdfn/1sMEI6e/fQIFLgWkrJtqngV+h26r+ttTZxGsi+vqNHaorLWnPBTE53QAfrFU6PYQ7qzvQkrOJgYX4jsj6/G5SKTB9r4j8uhRvEgot7FwoHfXv+9T9LGLEq9fp9rsBlaV2hvsBbNJH08E/P5sFuGmgJlTvFKs2UgzQ/PDyx3EUUDf6zp+wVypQ7hQl2bp3eh8tKcA3nLPG98/t/s9abIWmlUYQ69Uzqz1sVnK+ICyAATZuCwj/VpT1Aw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=04Nm7OrzAcL/CD/gEDxOkjTgj43WyMSTk4jirbzT46c=; b=DQSRUWgkdPG5wNPqf5k3dCW1YTh7NkDJKIrPtvyViIuq9MUb8BcrbUM5eh2FelkqbFI3L7L6TepZGEcG6L/9S4xBVXwwpsADyqNg5sJXrbgiTWLBOl7CsmtgQCV4ZmjVf2MlGSZXextBoTBFMcFFrvZ1dT/gmMIiqgfz/v9iM3pN7gq2IMYXCL/6LImw5sEDe/1qMELJnY/e868dSPBBdafJEE7pWdwPeaoYwFCIvbNK9HZ1/3L9tL6GWEdaw/OouoobJWTtknWsjR+YAUyGBx5U+eiJawjmtm71Nb+dChCY7sb3PdqusRJ7bSP0MLvynCL/GxiiR/bNnxLoVeJMDw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.118.232) smtp.rcpttodomain=vger.kernel.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=04Nm7OrzAcL/CD/gEDxOkjTgj43WyMSTk4jirbzT46c=; b=kpbP1L/eCqO29KAVu9f8McsJYOx4Azuh4o9GH0slnN6pKZQsmQ4M7ONayNIiOk93UvgFr76q43iuLH3Jx7hNkKCE2avkIexYxu5RMzUCYeKDkbRQzFtH8oOYB9kw3/22L+wOX5Gy4nR2J43pqQLtBpkY9R1r8T3TZeZKVXqcBOALMl5UgbVpxjPo6pHKy/voSBY3OkNgXIUtr4AC+w/2OA5cSky7h2f+2j1x3HXiurhQTeq60MIr8ubBKTqg1fvsSgNssAZp3uXbJ7Qix0pZ48C2ybmjWE6cogUpCQmJ+2bVIlgtzZah5WD8bMtgNvKOgskaDLqKcVv0EmBqynS9yw== Received: from DM6PR10CA0002.namprd10.prod.outlook.com (2603:10b6:5:60::15) by PH7PR12MB5784.namprd12.prod.outlook.com (2603:10b6:510:1d3::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5791.26; Mon, 7 Nov 2022 16:15:12 +0000 Received: from DM6NAM11FT084.eop-nam11.prod.protection.outlook.com (2603:10b6:5:60:cafe::af) by DM6PR10CA0002.outlook.office365.com (2603:10b6:5:60::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5791.26 via Frontend Transport; Mon, 7 Nov 2022 16:15:12 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.118.232) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.118.232 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.118.232; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.118.232) by DM6NAM11FT084.mail.protection.outlook.com (10.13.172.132) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5791.20 via Frontend Transport; Mon, 7 Nov 2022 16:15:12 +0000 Received: from drhqmail201.nvidia.com (10.126.190.180) by mail.nvidia.com (10.127.129.5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.26; Mon, 7 Nov 2022 08:15:03 -0800 Received: from drhqmail201.nvidia.com (10.126.190.180) by drhqmail201.nvidia.com (10.126.190.180) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.29; Mon, 7 Nov 2022 08:15:03 -0800 Received: from vdi.nvidia.com (10.127.8.13) by mail.nvidia.com (10.126.190.180) with Microsoft SMTP Server id 15.2.986.29 via Frontend Transport; Mon, 7 Nov 2022 08:15:01 -0800 From: Michael Guralnik To: , , CC: , Subject: [PATCH v1 rdma-next 5/8] RDMA/mlx5: Introduce mlx5r_cache_rb_key Date: Mon, 7 Nov 2022 18:14:46 +0200 Message-ID: <20221107161449.5611-6-michaelgur@nvidia.com> X-Mailer: git-send-email 2.17.2 In-Reply-To: <20221107161449.5611-1-michaelgur@nvidia.com> References: <20221107161449.5611-1-michaelgur@nvidia.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM6NAM11FT084:EE_|PH7PR12MB5784:EE_ X-MS-Office365-Filtering-Correlation-Id: d209da53-27e4-4290-b067-08dac0db3f9e X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 6FmRbbhyPq08JzidSzf6kgMqUslQMMmOJsK7MexYn1qucwPgrwdpPT2GwEm8h7dZDApcWohRJXNSfQZHpJldwiicIomIGndMAPCVklL2eaF9iFMOHXf3S5STEDZhOlny9x4GSe217oi2+Z0gDD9G9POg/TQhXf/z6usbT80PZ4LtBrJ7LOOrVDxGLQj/82ink6xTl3TFNLpepEODCyCjdFpKq8IHS2/s/XQhmqjPD3W+iumrdZQk2I5ZSaYE8Bo0Y5iU5Uh4kxaVFl2IXhnqcwhxLdigtjoR6D/aEtwSppY306nKUgYElg7WUFLUryziIPzWedL4A13P0wq+ce/rppq2OvEA+dLfbtQWszX1UTzybODmsvTek8UUYV9j6ZQ3tEOt0kINlR+KEiV546sDnQnLQYkJ6oAjz4jL5+BKzVJyFXvDSL5nQSR4/obNwDdyYruVmGVe/o4MR3J/fcnjTpiCO0J9OtsO9mBc/eGv00FDgM31dPqbn29eFkXncvutxjura9I4cEsrU9RzXanefTqMNvtWySQjdXBMnSqz0nV4SCS1TH0WSoy6Zge1hT7DRhFEf1Oi/wHalECmoGMeiYY84zx9A5APoiwbvIXjCcEuuzaVPviJMiGsnw2HLQImwtpw1fNNnQ4eU9FORu7nbuW9R62+li1a/iFeR9EPv1LltR7KsHraPRXrI21EU6J4YkEje/MLOTWVUiTQ4JST38bF3jopdkuyAqsomc9TgxIwOC9h3vs1ijmWJsZDkaESq+AqM3U+wRIBaCSrbZxDlzrc7Yjk1DyUfLH1lTOvvTc= X-Forefront-Antispam-Report: CIP:216.228.118.232;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc7edge1.nvidia.com;CAT:NONE;SFS:(13230022)(4636009)(136003)(346002)(39860400002)(376002)(396003)(451199015)(46966006)(36840700001)(40470700004)(336012)(107886003)(426003)(2616005)(26005)(6666004)(186003)(1076003)(47076005)(7696005)(36860700001)(83380400001)(2906002)(40460700003)(40480700001)(82310400005)(110136005)(54906003)(478600001)(41300700001)(5660300002)(8936002)(4326008)(70586007)(316002)(70206006)(8676002)(36756003)(82740400003)(356005)(7636003)(86362001)(2101003);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 07 Nov 2022 16:15:12.2363 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: d209da53-27e4-4290-b067-08dac0db3f9e X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.118.232];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: DM6NAM11FT084.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR12MB5784 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org From: Aharon Landau In the next patch, I will change the cache structure to an RB tree. Introducing the key of the tree. The key is all the mkey properties that UMR operations can't modify. Using this key to define the cache entries and to search and create cache mkeys. Signed-off-by: Aharon Landau --- drivers/infiniband/hw/mlx5/mlx5_ib.h | 14 +++++- drivers/infiniband/hw/mlx5/mr.c | 73 +++++++++++++++++++--------- drivers/infiniband/hw/mlx5/odp.c | 8 ++- 3 files changed, 69 insertions(+), 26 deletions(-) diff --git a/drivers/infiniband/hw/mlx5/mlx5_ib.h b/drivers/infiniband/hw/mlx5/mlx5_ib.h index 84d3b917c33e..939ec3759eba 100644 --- a/drivers/infiniband/hw/mlx5/mlx5_ib.h +++ b/drivers/infiniband/hw/mlx5/mlx5_ib.h @@ -613,6 +613,16 @@ enum mlx5_mkey_type { MLX5_MKEY_INDIRECT_DEVX, }; +struct mlx5r_cache_rb_key { + unsigned int access_mode; + unsigned int access_flags; + /* + * keep ndescs as the last member so entries with about the same ndescs + * will be close in the tree + */ + unsigned int ndescs; +}; + struct mlx5_ib_mkey { u32 key; enum mlx5_mkey_type type; @@ -732,10 +742,10 @@ struct mlx5_cache_ent { unsigned long stored; unsigned long reserved; + struct mlx5r_cache_rb_key rb_key; + char name[4]; u32 order; - u32 access_mode; - unsigned int ndescs; u8 disabled:1; u8 fill_to_high_water:1; diff --git a/drivers/infiniband/hw/mlx5/mr.c b/drivers/infiniband/hw/mlx5/mr.c index fe5567c57897..e7a3d4fa52d0 100644 --- a/drivers/infiniband/hw/mlx5/mr.c +++ b/drivers/infiniband/hw/mlx5/mr.c @@ -287,18 +287,18 @@ static int get_mkc_octo_size(unsigned int access_mode, unsigned int ndescs) return ret; } -static void set_cache_mkc(struct mlx5_ib_dev *dev, u8 access_mode, - unsigned int access_flags, unsigned int ndescs, - void *mkc) +static void set_cache_mkc(struct mlx5_ib_dev *dev, + struct mlx5r_cache_rb_key rb_key, void *mkc) { - set_mkc_access_pd_addr_fields(mkc, access_flags, 0, dev->umrc.pd); + set_mkc_access_pd_addr_fields(mkc, rb_key.access_flags, 0, + dev->umrc.pd); MLX5_SET(mkc, mkc, free, 1); MLX5_SET(mkc, mkc, umr_en, 1); - MLX5_SET(mkc, mkc, access_mode_1_0, access_mode & 0x3); - MLX5_SET(mkc, mkc, access_mode_4_2, (access_mode >> 2) & 0x7); + MLX5_SET(mkc, mkc, access_mode_1_0, rb_key.access_mode & 0x3); + MLX5_SET(mkc, mkc, access_mode_4_2, (rb_key.access_mode >> 2) & 0x7); MLX5_SET(mkc, mkc, translations_octword_size, - get_mkc_octo_size(access_mode, ndescs)); + get_mkc_octo_size(rb_key.access_mode, rb_key.ndescs)); MLX5_SET(mkc, mkc, log_page_size, PAGE_SHIFT); } @@ -317,7 +317,7 @@ static int add_keys(struct mlx5_cache_ent *ent, unsigned int num) return -ENOMEM; mkc = MLX5_ADDR_OF(create_mkey_in, async_create->in, memory_key_mkey_entry); - set_cache_mkc(ent->dev, ent->access_mode, 0, ent->ndescs, mkc); + set_cache_mkc(ent->dev, ent->rb_key, mkc); async_create->ent = ent; err = push_mkey(ent, true, NULL); @@ -343,8 +343,8 @@ static int add_keys(struct mlx5_cache_ent *ent, unsigned int num) } /* Synchronously create a cacheable mkey */ -static int create_cache_mkey(struct mlx5_ib_dev *dev, u8 access_mode, - unsigned int access_flags, unsigned int ndescs, +static int create_cache_mkey(struct mlx5_ib_dev *dev, + struct mlx5r_cache_rb_key rb_key, struct mlx5_ib_mkey *mkey) { size_t inlen = MLX5_ST_SZ_BYTES(create_mkey_in); @@ -356,7 +356,7 @@ static int create_cache_mkey(struct mlx5_ib_dev *dev, u8 access_mode, if (!in) return -ENOMEM; mkc = MLX5_ADDR_OF(create_mkey_in, in, memory_key_mkey_entry); - set_cache_mkc(dev, access_mode, access_flags, ndescs, mkc); + set_cache_mkc(dev, rb_key, mkc); err = mlx5_ib_create_mkey(dev, mkey, in, inlen); kfree(in); @@ -647,7 +647,7 @@ static bool mlx5_ent_get_mkey(struct mlx5_cache_ent *ent, struct mlx5_ib_mr *mr) } mr->mmkey.key = pop_stored_mkey(ent); - mr->mmkey.ndescs = ent->ndescs; + mr->mmkey.ndescs = ent->rb_key.ndescs; mr->mmkey.cache_ent = ent; queue_adjust_cache_locked(ent); ent->in_use++; @@ -668,29 +668,57 @@ static struct mlx5_cache_ent *mkey_cache_ent_from_order(struct mlx5_ib_dev *dev, return &cache->ent[order]; } -static bool mlx5_cache_get_mkey(struct mlx5_ib_dev *dev, u8 access_mode, - u8 access_flags, unsigned int ndescs, +static bool mlx5_cache_get_mkey(struct mlx5_ib_dev *dev, + struct mlx5r_cache_rb_key rb_key, struct mlx5_ib_mr *mr) { struct mlx5_cache_ent *ent; - if (!mlx5r_umr_can_reconfig(dev, 0, access_flags)) + if (!mlx5r_umr_can_reconfig(dev, 0, rb_key.access_flags)) return false; - if (access_mode == MLX5_MKC_ACCESS_MODE_KSM) + if (rb_key.access_mode == MLX5_MKC_ACCESS_MODE_KSM) ent = &dev->cache.ent[MLX5_IMR_KSM_CACHE_ENTRY]; - ent = mkey_cache_ent_from_order(dev, order_base_2(ndescs)); + ent = mkey_cache_ent_from_order(dev, order_base_2(rb_key.ndescs)); if (!ent) return false; return mlx5_ent_get_mkey(ent, mr); } +static int get_uchangeable_access_flags(struct mlx5_ib_dev *dev, + int access_flags) +{ + int ret = 0; + + if ((access_flags & IB_ACCESS_REMOTE_ATOMIC) && + MLX5_CAP_GEN(dev->mdev, atomic) && + MLX5_CAP_GEN(dev->mdev, umr_modify_atomic_disabled)) + ret |= IB_ACCESS_REMOTE_ATOMIC; + + if ((access_flags & IB_ACCESS_RELAXED_ORDERING) && + MLX5_CAP_GEN(dev->mdev, relaxed_ordering_write) && + !MLX5_CAP_GEN(dev->mdev, relaxed_ordering_write_umr)) + ret |= IB_ACCESS_RELAXED_ORDERING; + + if ((access_flags & IB_ACCESS_RELAXED_ORDERING) && + MLX5_CAP_GEN(dev->mdev, relaxed_ordering_read) && + !MLX5_CAP_GEN(dev->mdev, relaxed_ordering_read_umr)) + ret |= IB_ACCESS_RELAXED_ORDERING; + + return ret; +} + struct mlx5_ib_mr *mlx5_mr_cache_alloc(struct mlx5_ib_dev *dev, u8 access_mode, unsigned int access_flags, unsigned int ndescs) { + struct mlx5r_cache_rb_key rb_key = { + .access_mode = access_mode, + .access_flags = get_uchangeable_access_flags(dev, access_flags), + .ndescs = ndescs + }; struct mlx5_ib_mr *mr; int err; @@ -698,13 +726,12 @@ struct mlx5_ib_mr *mlx5_mr_cache_alloc(struct mlx5_ib_dev *dev, u8 access_mode, if (!mr) return ERR_PTR(-ENOMEM); - if (!mlx5_cache_get_mkey(dev, access_mode, access_flags, ndescs, mr)) { + if (!mlx5_cache_get_mkey(dev, rb_key, mr)) { /* * Didn't find an mkey in cache. * Create an mkey with the exact needed size. */ - err = create_cache_mkey(dev, access_mode, access_flags, ndescs, - &mr->mmkey); + err = create_cache_mkey(dev, rb_key, &mr->mmkey); if (err) { kfree(mr); return ERR_PTR(err); @@ -774,6 +801,8 @@ static void delay_time_func(struct timer_list *t) int mlx5_mkey_cache_init(struct mlx5_ib_dev *dev) { + struct mlx5r_cache_rb_key rb_key = { .access_mode = + MLX5_MKC_ACCESS_MODE_MTT }; struct mlx5_mkey_cache *cache = &dev->cache; struct mlx5_cache_ent *ent; int i; @@ -804,8 +833,8 @@ int mlx5_mkey_cache_init(struct mlx5_ib_dev *dev) if (ent->order > mkey_cache_max_order(dev)) continue; - ent->ndescs = 1 << ent->order; - ent->access_mode = MLX5_MKC_ACCESS_MODE_MTT; + rb_key.ndescs = 1 << ent->order; + ent->rb_key = rb_key; if ((dev->mdev->profile.mask & MLX5_PROF_MASK_MR_CACHE) && !dev->is_rep && mlx5_core_is_pf(dev->mdev) && mlx5r_umr_can_load_pas(dev, 0)) diff --git a/drivers/infiniband/hw/mlx5/odp.c b/drivers/infiniband/hw/mlx5/odp.c index 137143da5959..90339edddfed 100644 --- a/drivers/infiniband/hw/mlx5/odp.c +++ b/drivers/infiniband/hw/mlx5/odp.c @@ -1589,10 +1589,14 @@ mlx5_ib_odp_destroy_eq(struct mlx5_ib_dev *dev, struct mlx5_ib_pf_eq *eq) void mlx5_odp_init_mkey_cache_entry(struct mlx5_cache_ent *ent) { + struct mlx5r_cache_rb_key rb_key = { + .access_mode = MLX5_MKC_ACCESS_MODE_KSM, + .ndescs = mlx5_imr_ksm_entries + }; + if (!(ent->dev->odp_caps.general_caps & IB_ODP_SUPPORT_IMPLICIT)) return; - ent->ndescs = mlx5_imr_ksm_entries; - ent->access_mode = MLX5_MKC_ACCESS_MODE_KSM; + ent->rb_key = rb_key; } static const struct ib_device_ops mlx5_ib_dev_odp_ops = { From patchwork Mon Nov 7 16:14:47 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michael Guralnik X-Patchwork-Id: 13034657 X-Patchwork-Delegate: jgg@ziepe.ca Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 97B72C433FE for ; Mon, 7 Nov 2022 16:15:21 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232273AbiKGQPU (ORCPT ); Mon, 7 Nov 2022 11:15:20 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39410 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232770AbiKGQPS (ORCPT ); Mon, 7 Nov 2022 11:15:18 -0500 Received: from NAM12-MW2-obe.outbound.protection.outlook.com (mail-mw2nam12on2057.outbound.protection.outlook.com [40.107.244.57]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5BBD61572D for ; Mon, 7 Nov 2022 08:15:16 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=oAKmrflfUHbVqZ1rIVTuc7i+Jgf0VXLARMJQHW7p0+rq0e1QXgJsFaeFR5bBsRTABbeRzahqhMNLsDvjfylrKUB4bt9gru6l3znEJP15XBjQ/JK+vkXAMKOCi+zZq1Ir3fr9xs5Zj4PLMnbyqXAU/4IeJo6V83yCo6WcPd+2NRNnlGTteNHOify7mbO4lG7lvcYfTnO5s8+8o+AtNAz8fAB9Hsuy0XDyr6jOB3Q3B5VYYniEpaJ9MeJFYA6sNQ50nM804yOr6BLrWMILWB68/V2pTDktAEmybQNvVJP3mL0ypkFJmGEkfVKGixvNMQmefVuBQ6XkxDsGDTYmRmvuWQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=wm4TiMf/sAYn0CFpzz9S3Fd13nxZc1cz+81Z2ZPJ7aE=; b=lEU2B3mJFAG3bCh+16BYk8VY+yd038+q3h3FuclmbMWSFTBctTWW6JTKYRBKAQxOUNJEGQP4rWLkMDdGfc+Tr4ZA5fQ/4Qqmw1pkaDShBdVYcsiBAGUwAVosajsPuHZa9P19vj5aHE0rxK0DxTUGIARIIpSrk+6E2tj5kxYaZtKbB7PboegZ7Zd7ig+wvzX4Guk0lz04P4BUQzYU19qcKD7sjP7ZnKIIyu9T7TpZB6VlYED/5RjiWMG+98590JTo7sICKuOKrABZl9jK8hG0/n5xvByu6hoj+Ftw3l1sCr77BCdxMkaBlwDR6+gwipamVPJtcqSS4jPQPyB7SfgEqQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.118.233) smtp.rcpttodomain=vger.kernel.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=wm4TiMf/sAYn0CFpzz9S3Fd13nxZc1cz+81Z2ZPJ7aE=; b=SfT9voRgUVcxz/P1kaaRWPBmPk3R/9paYIpkTvKPmdXD7S6rfF3gjcpodBT61Icp3zIDI/fqWBTIEx0gsnK+wvYe0x9vgggsAzwwU7MmGms85GIhvwBk8PAa2mPpJdt5bS0YL6nrpU/zNc1EMk4b2zseMuGAHNGASAlVXbDYmEo0vSlAfFsHFVy/IuEQlrpIH/MkG9IR5OuaQ+8AG3+wdsZ8oQTEPcCV8MM9IluR20xV0vILu4tw48HrcljJPy85lfKSF6RRhhZLE+wGaoXQ81V5MECc1uMJoBpECDKDJgB4baG1McGadFpjWo3YSJ7mcfE1nnhz00Ytku8g6Ah0MQ== Received: from DS7PR06CA0053.namprd06.prod.outlook.com (2603:10b6:8:54::8) by CH3PR12MB7642.namprd12.prod.outlook.com (2603:10b6:610:14a::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5791.22; Mon, 7 Nov 2022 16:15:14 +0000 Received: from DM6NAM11FT037.eop-nam11.prod.protection.outlook.com (2603:10b6:8:54:cafe::9d) by DS7PR06CA0053.outlook.office365.com (2603:10b6:8:54::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5791.25 via Frontend Transport; Mon, 7 Nov 2022 16:15:14 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.118.233) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.118.233 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.118.233; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.118.233) by DM6NAM11FT037.mail.protection.outlook.com (10.13.172.122) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5791.20 via Frontend Transport; Mon, 7 Nov 2022 16:15:14 +0000 Received: from drhqmail202.nvidia.com (10.126.190.181) by mail.nvidia.com (10.127.129.6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.26; Mon, 7 Nov 2022 08:15:05 -0800 Received: from drhqmail201.nvidia.com (10.126.190.180) by drhqmail202.nvidia.com (10.126.190.181) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.29; Mon, 7 Nov 2022 08:15:05 -0800 Received: from vdi.nvidia.com (10.127.8.13) by mail.nvidia.com (10.126.190.180) with Microsoft SMTP Server id 15.2.986.29 via Frontend Transport; Mon, 7 Nov 2022 08:15:03 -0800 From: Michael Guralnik To: , , CC: , Subject: [PATCH v1 rdma-next 6/8] RDMA/mlx5: Change the cache structure to an RB-tree Date: Mon, 7 Nov 2022 18:14:47 +0200 Message-ID: <20221107161449.5611-7-michaelgur@nvidia.com> X-Mailer: git-send-email 2.17.2 In-Reply-To: <20221107161449.5611-1-michaelgur@nvidia.com> References: <20221107161449.5611-1-michaelgur@nvidia.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM6NAM11FT037:EE_|CH3PR12MB7642:EE_ X-MS-Office365-Filtering-Correlation-Id: ba4d1ff9-f8b7-4856-40a1-08dac0db40c0 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: hgMCOZCaxqmAIJReJ8Hk83pAscyriVvjNp2JMV+4a8Y6CgmYfDc9cPz07R+mekqqwkv1eb9mcmqokBnw+8lJlZCtwEzcbteSh6YgtWrQj3tZDg0PDYJhFSVwbX7PwLpLBhVIJM5k5H1fJFqhsutabw2d/qnrM+y88ea4XTVm8MXh3VoHwYgrI+b0AWtSv7A3uHxouhj/cfFDU6P0jPvDZyOVOcxuKQpzXQIYtMtugMXwF/fwYXXKfEqzBFlLgWnSNuNUp+Y4VGWbU4pU+wVkViqEC6tiIVHwb5sJBbv6l0CS5txRt8USdu4R2yS2p2M7K4XCjGZY5McDz3uA6J48yBobD5UZz/f2pJN1+76tWK3mqg4IiWLUI25KKE9x6hCqQaLcldYDU/oenEcFTMgHot9/V4It+arsFP8nUo1hZ2sPeP5EHxflf+0KBrERcLHxDjoED17aObFxGUSnr85fezTc/DxCip01gQifMQR9AqlOlGo2qr5Iu3d25iuenn++DVIH4VnB0LLtPO2nlRHuLy4JRmxwRXRXabtwR5JgHz9YaSvG3PTH1WbgXstFEFm87EvLpAV2+3Vrmf7JO67Sd9db/xM8BVbdeTFk4OmziMr5g3+V01HZcu2rE9GIQ0f1JoWErgDx2Gp+k8ZFjKmmq5AwMB+5Mr5nmUL0e3TqxvjJFwdULLSy+CzhqV8Ay5sX9zuqZfAn5pD3ugA2G5LccSbKz3W3ORmDYWxNM0xmEeXnTP0YcwhhAfokHYEyfySAhyA16t0ifdP5HJftMM9wzyfGYvGkhMukhLqijYSOOZE= X-Forefront-Antispam-Report: CIP:216.228.118.233;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc7edge2.nvidia.com;CAT:NONE;SFS:(13230022)(4636009)(136003)(39860400002)(346002)(376002)(396003)(451199015)(36840700001)(40470700004)(46966006)(54906003)(8676002)(41300700001)(110136005)(478600001)(36860700001)(4326008)(5660300002)(8936002)(70586007)(70206006)(83380400001)(7636003)(356005)(40480700001)(86362001)(7696005)(107886003)(82740400003)(6666004)(2616005)(186003)(1076003)(40460700003)(426003)(30864003)(26005)(47076005)(316002)(336012)(82310400005)(36756003)(2906002)(2101003);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 07 Nov 2022 16:15:14.1652 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: ba4d1ff9-f8b7-4856-40a1-08dac0db40c0 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.118.233];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: DM6NAM11FT037.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH3PR12MB7642 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org From: Aharon Landau Currently, the cache structure is a static linear array. Therefore, his size is limited to the number of entries in it and is not expandable. The entries are dedicated to mkeys of size 2^x and no access_flags. Mkeys with different properties are not cacheable. In this patch, we change the cache structure to an RB-tree. By this, we enable caching all user mkeys that can use umr. Signed-off-by: Aharon Landau --- drivers/infiniband/hw/mlx5/mlx5_ib.h | 17 +- drivers/infiniband/hw/mlx5/mr.c | 307 +++++++++++++++++++-------- drivers/infiniband/hw/mlx5/odp.c | 9 +- 3 files changed, 237 insertions(+), 96 deletions(-) diff --git a/drivers/infiniband/hw/mlx5/mlx5_ib.h b/drivers/infiniband/hw/mlx5/mlx5_ib.h index 939ec3759eba..0bf2511fdfe4 100644 --- a/drivers/infiniband/hw/mlx5/mlx5_ib.h +++ b/drivers/infiniband/hw/mlx5/mlx5_ib.h @@ -742,10 +742,10 @@ struct mlx5_cache_ent { unsigned long stored; unsigned long reserved; + struct rb_node node; struct mlx5r_cache_rb_key rb_key; char name[4]; - u32 order; u8 disabled:1; u8 fill_to_high_water:1; @@ -776,8 +776,9 @@ struct mlx5r_async_create_mkey { struct mlx5_mkey_cache { struct workqueue_struct *wq; - struct mlx5_cache_ent ent[MAX_MKEY_CACHE_ENTRIES]; - struct dentry *root; + struct rb_root rb_root; + struct mutex rb_lock; + struct dentry *fs_root; unsigned long last_add; }; @@ -1322,6 +1323,8 @@ void mlx5_ib_copy_pas(u64 *old, u64 *new, int step, int num); int mlx5_ib_get_cqe_size(struct ib_cq *ibcq); int mlx5_mkey_cache_init(struct mlx5_ib_dev *dev); int mlx5_mkey_cache_cleanup(struct mlx5_ib_dev *dev); +struct mlx5_cache_ent *mlx5r_cache_create_ent(struct mlx5_ib_dev *dev, + struct mlx5r_cache_rb_key rb_key); struct mlx5_ib_mr *mlx5_mr_cache_alloc(struct mlx5_ib_dev *dev, u8 access_mode, unsigned int access_flags, @@ -1349,7 +1352,7 @@ int mlx5r_odp_create_eq(struct mlx5_ib_dev *dev, struct mlx5_ib_pf_eq *eq); void mlx5_ib_odp_cleanup_one(struct mlx5_ib_dev *ibdev); int __init mlx5_ib_odp_init(void); void mlx5_ib_odp_cleanup(void); -void mlx5_odp_init_mkey_cache_entry(struct mlx5_cache_ent *ent); +struct mlx5_cache_ent *mlx5_odp_init_mkey_cache_entry(struct mlx5_ib_dev *dev); void mlx5_odp_populate_xlt(void *xlt, size_t idx, size_t nentries, struct mlx5_ib_mr *mr, int flags); @@ -1368,7 +1371,11 @@ static inline int mlx5r_odp_create_eq(struct mlx5_ib_dev *dev, static inline void mlx5_ib_odp_cleanup_one(struct mlx5_ib_dev *ibdev) {} static inline int mlx5_ib_odp_init(void) { return 0; } static inline void mlx5_ib_odp_cleanup(void) {} -static inline void mlx5_odp_init_mkey_cache_entry(struct mlx5_cache_ent *ent) {} +static inline struct mlx5_cache_ent * +mlx5_odp_init_mkey_cache_entry(struct mlx5_ib_dev *dev) +{ + return NULL; +} static inline void mlx5_odp_populate_xlt(void *xlt, size_t idx, size_t nentries, struct mlx5_ib_mr *mr, int flags) {} diff --git a/drivers/infiniband/hw/mlx5/mr.c b/drivers/infiniband/hw/mlx5/mr.c index e7a3d4fa52d0..df3e551ad7cd 100644 --- a/drivers/infiniband/hw/mlx5/mr.c +++ b/drivers/infiniband/hw/mlx5/mr.c @@ -514,18 +514,22 @@ static const struct file_operations limit_fops = { static bool someone_adding(struct mlx5_mkey_cache *cache) { - unsigned int i; - - for (i = 0; i < MAX_MKEY_CACHE_ENTRIES; i++) { - struct mlx5_cache_ent *ent = &cache->ent[i]; - bool ret; + struct mlx5_cache_ent *ent; + struct rb_node *node; + bool ret; + mutex_lock(&cache->rb_lock); + for (node = rb_first(&cache->rb_root); node; node = rb_next(node)) { + ent = rb_entry(node, struct mlx5_cache_ent, node); xa_lock_irq(&ent->mkeys); ret = ent->stored < ent->limit; xa_unlock_irq(&ent->mkeys); - if (ret) + if (ret) { + mutex_unlock(&cache->rb_lock); return true; + } } + mutex_unlock(&cache->rb_lock); return false; } @@ -589,8 +593,8 @@ static void __cache_work_func(struct mlx5_cache_ent *ent) if (err != -EAGAIN) { mlx5_ib_warn( dev, - "command failed order %d, err %d\n", - ent->order, err); + "command failed order %s, err %d\n", + ent->name, err); queue_delayed_work(cache->wq, &ent->dwork, msecs_to_jiffies(1000)); } @@ -636,6 +640,86 @@ static void delayed_cache_work_func(struct work_struct *work) __cache_work_func(ent); } +static int cache_rb_key_cmp(struct mlx5r_cache_rb_key cur, + struct mlx5r_cache_rb_key lookup) +{ + int ret; + + ret = cur.access_mode - lookup.access_mode; + if (ret) + return ret; + + ret = cur.access_flags - lookup.access_flags; + if (ret) + return ret; + + return cur.ndescs - lookup.ndescs; +} + +static int mlx5_cache_ent_insert(struct mlx5_mkey_cache *cache, + struct mlx5_cache_ent *ent) +{ + struct rb_node **new = &cache->rb_root.rb_node, *parent = NULL; + struct mlx5_cache_ent *cur; + int cmp; + + mutex_lock(&cache->rb_lock); + /* Figure out where to put new node */ + while (*new) { + cur = rb_entry(*new, struct mlx5_cache_ent, node); + parent = *new; + cmp = cache_rb_key_cmp(ent->rb_key, cur->rb_key); + if (cmp < 0) + new = &((*new)->rb_left); + if (cmp > 0) + new = &((*new)->rb_right); + if (cmp == 0) { + mutex_unlock(&cache->rb_lock); + return -EEXIST; + } + } + + /* Add new node and rebalance tree. */ + rb_link_node(&ent->node, parent, new); + rb_insert_color(&ent->node, &cache->rb_root); + + mutex_unlock(&cache->rb_lock); + return 0; +} + +static struct rb_node * +mlx5_cache_find_smallest_ent(struct mlx5_mkey_cache *cache, + struct mlx5r_cache_rb_key rb_key) +{ + struct rb_node *node = cache->rb_root.rb_node; + struct mlx5_cache_ent *cur, *smallest = NULL; + int cmp; + + /* + * Find the smallest ent with ent.rb_key >= rb_key. + */ + while (node) { + cur = rb_entry(node, struct mlx5_cache_ent, node); + + cmp = cache_rb_key_cmp(rb_key, cur->rb_key); + if (cmp < 0) { + /* cur.rb_key > rb_key */ + smallest = cur; + node = node->rb_left; + } + if (cmp > 0) + node = node->rb_right; + if (cmp == 0) + return &cur->node; + } + + return (smallest && + smallest->rb_key.access_mode == rb_key.access_mode && + smallest->rb_key.access_flags == rb_key.access_flags) ? + &smallest->node : + NULL; +} + static bool mlx5_ent_get_mkey(struct mlx5_cache_ent *ent, struct mlx5_ib_mr *mr) { xa_lock_irq(&ent->mkeys); @@ -655,36 +739,41 @@ static bool mlx5_ent_get_mkey(struct mlx5_cache_ent *ent, struct mlx5_ib_mr *mr) return true; } -static struct mlx5_cache_ent *mkey_cache_ent_from_order(struct mlx5_ib_dev *dev, - unsigned int order) -{ - struct mlx5_mkey_cache *cache = &dev->cache; - - if (order < cache->ent[0].order) - return &cache->ent[0]; - order = order - cache->ent[0].order; - if (order > MKEY_CACHE_LAST_STD_ENTRY) - return NULL; - return &cache->ent[order]; -} - static bool mlx5_cache_get_mkey(struct mlx5_ib_dev *dev, struct mlx5r_cache_rb_key rb_key, struct mlx5_ib_mr *mr) { + struct mlx5_mkey_cache *cache = &dev->cache; + unsigned int order, upper_bound; struct mlx5_cache_ent *ent; + struct rb_node *node; - if (!mlx5r_umr_can_reconfig(dev, 0, rb_key.access_flags)) - return false; + order = order_base_2(rb_key.ndescs) > 2 ? + order_base_2(rb_key.ndescs) : 2; + upper_bound = 1 << order; - if (rb_key.access_mode == MLX5_MKC_ACCESS_MODE_KSM) - ent = &dev->cache.ent[MLX5_IMR_KSM_CACHE_ENTRY]; + /* + * Find the smallest node within the range with available mkeys. + */ + mutex_lock(&cache->rb_lock); + node = mlx5_cache_find_smallest_ent(cache, rb_key); + while (node) { + ent = rb_entry(node, struct mlx5_cache_ent, node); + if (ent->rb_key.access_mode != rb_key.access_mode || + ent->rb_key.access_flags != rb_key.access_flags || + ent->rb_key.ndescs > upper_bound) + break; - ent = mkey_cache_ent_from_order(dev, order_base_2(rb_key.ndescs)); - if (!ent) - return false; + if (mlx5_ent_get_mkey(ent, mr)) { + mutex_unlock(&cache->rb_lock); + return true; + } - return mlx5_ent_get_mkey(ent, mr); + node = rb_next(node); + } + mutex_unlock(&cache->rb_lock); + + return false; } static int get_uchangeable_access_flags(struct mlx5_ib_dev *dev, @@ -743,10 +832,8 @@ struct mlx5_ib_mr *mlx5_mr_cache_alloc(struct mlx5_ib_dev *dev, u8 access_mode, return mr; } -static void clean_keys(struct mlx5_ib_dev *dev, int c) +static void clean_keys(struct mlx5_ib_dev *dev, struct mlx5_cache_ent *ent) { - struct mlx5_mkey_cache *cache = &dev->cache; - struct mlx5_cache_ent *ent = &cache->ent[c]; u32 mkey; cancel_delayed_work(&ent->dwork); @@ -765,26 +852,19 @@ static void mlx5_mkey_cache_debugfs_cleanup(struct mlx5_ib_dev *dev) if (!mlx5_debugfs_root || dev->is_rep) return; - debugfs_remove_recursive(dev->cache.root); - dev->cache.root = NULL; + debugfs_remove_recursive(dev->cache.fs_root); + dev->cache.fs_root = NULL; } -static void mlx5_mkey_cache_debugfs_init(struct mlx5_ib_dev *dev) +static void mlx5_cache_ent_debugfs_init(struct mlx5_ib_dev *dev, + struct mlx5_cache_ent *ent, int order) { struct mlx5_mkey_cache *cache = &dev->cache; - struct mlx5_cache_ent *ent; struct dentry *dir; - int i; - if (!mlx5_debugfs_root || dev->is_rep) - return; - - cache->root = debugfs_create_dir("mr_cache", mlx5_debugfs_get_dev_root(dev->mdev)); - - for (i = 0; i < MAX_MKEY_CACHE_ENTRIES; i++) { - ent = &cache->ent[i]; - sprintf(ent->name, "%d", ent->order); - dir = debugfs_create_dir(ent->name, cache->root); + if (cache->fs_root) { + sprintf(ent->name, "%d", order); + dir = debugfs_create_dir(ent->name, cache->fs_root); debugfs_create_file("size", 0600, dir, ent, &size_fops); debugfs_create_file("limit", 0600, dir, ent, &limit_fops); debugfs_create_ulong("cur", 0400, dir, &ent->stored); @@ -799,68 +879,114 @@ static void delay_time_func(struct timer_list *t) WRITE_ONCE(dev->fill_delay, 0); } -int mlx5_mkey_cache_init(struct mlx5_ib_dev *dev) +struct mlx5_cache_ent *mlx5r_cache_create_ent(struct mlx5_ib_dev *dev, + struct mlx5r_cache_rb_key rb_key) +{ + struct mlx5_cache_ent *ent; + int ret; + + ent = kzalloc(sizeof(*ent), GFP_KERNEL); + if (!ent) + return ERR_PTR(-ENOMEM); + + xa_init_flags(&ent->mkeys, XA_FLAGS_LOCK_IRQ); + ent->rb_key = rb_key; + ent->dev = dev; + + INIT_DELAYED_WORK(&ent->dwork, delayed_cache_work_func); + + ret = mlx5_cache_ent_insert(&dev->cache, ent); + if (ret) { + kfree(ent); + return ERR_PTR(ret); + } + return ent; +} + +static int mlx5_cache_init_default_entries(struct mlx5_ib_dev *dev) { struct mlx5r_cache_rb_key rb_key = { .access_mode = MLX5_MKC_ACCESS_MODE_MTT }; struct mlx5_mkey_cache *cache = &dev->cache; + bool can_use_cache, need_cache; struct mlx5_cache_ent *ent; - int i; + int order; + + if (mlx5_debugfs_root && !dev->is_rep) + cache->fs_root = debugfs_create_dir( + "mr_cache", mlx5_debugfs_get_dev_root(dev->mdev)); + + can_use_cache = !dev->is_rep && mlx5r_umr_can_load_pas(dev, 0); + need_cache = (dev->mdev->profile.mask & MLX5_PROF_MASK_MR_CACHE) && + mlx5_core_is_pf(dev->mdev); + + for (order = 2; order <= MKEY_CACHE_LAST_STD_ENTRY + 2; order++) { + rb_key.ndescs = 1 << order; + ent = mlx5r_cache_create_ent(dev, rb_key); + if (IS_ERR(ent)) + return PTR_ERR(ent); + + mlx5_cache_ent_debugfs_init(dev, ent, order); + + if (can_use_cache && need_cache && + order <= mkey_cache_max_order(dev)) { + ent->limit = + dev->mdev->profile.mr_cache[order - 2].limit; + xa_lock_irq(&ent->mkeys); + queue_adjust_cache_locked(ent); + xa_unlock_irq(&ent->mkeys); + } + } + + ent = mlx5_odp_init_mkey_cache_entry(dev); + if (ent) { + if (IS_ERR(ent)) + return PTR_ERR(ent); + + mlx5_cache_ent_debugfs_init(dev, ent, + MLX5_IMR_KSM_CACHE_ENTRY + 2); + } + + return 0; +} + +int mlx5_mkey_cache_init(struct mlx5_ib_dev *dev) +{ + int err; mutex_init(&dev->slow_path_mutex); - cache->wq = alloc_ordered_workqueue("mkey_cache", WQ_MEM_RECLAIM); - if (!cache->wq) { + mutex_init(&dev->cache.rb_lock); + dev->cache.rb_root = RB_ROOT; + dev->cache.wq = alloc_ordered_workqueue("mkey_cache", WQ_MEM_RECLAIM); + if (!dev->cache.wq) { mlx5_ib_warn(dev, "failed to create work queue\n"); return -ENOMEM; } mlx5_cmd_init_async_ctx(dev->mdev, &dev->async_ctx); timer_setup(&dev->delay_timer, delay_time_func, 0); - for (i = 0; i < MAX_MKEY_CACHE_ENTRIES; i++) { - ent = &cache->ent[i]; - xa_init_flags(&ent->mkeys, XA_FLAGS_LOCK_IRQ); - ent->order = i + 2; - ent->dev = dev; - ent->limit = 0; - - INIT_DELAYED_WORK(&ent->dwork, delayed_cache_work_func); - - if (i > MKEY_CACHE_LAST_STD_ENTRY) { - mlx5_odp_init_mkey_cache_entry(ent); - continue; - } - - if (ent->order > mkey_cache_max_order(dev)) - continue; - - rb_key.ndescs = 1 << ent->order; - ent->rb_key = rb_key; - if ((dev->mdev->profile.mask & MLX5_PROF_MASK_MR_CACHE) && - !dev->is_rep && mlx5_core_is_pf(dev->mdev) && - mlx5r_umr_can_load_pas(dev, 0)) - ent->limit = dev->mdev->profile.mr_cache[i].limit; - else - ent->limit = 0; - xa_lock_irq(&ent->mkeys); - queue_adjust_cache_locked(ent); - xa_unlock_irq(&ent->mkeys); - } - - mlx5_mkey_cache_debugfs_init(dev); + err = mlx5_cache_init_default_entries(dev); + if (err) + goto err; return 0; +err: + mlx5_mkey_cache_cleanup(dev); + return err; } int mlx5_mkey_cache_cleanup(struct mlx5_ib_dev *dev) { - unsigned int i; + struct rb_root *root = &dev->cache.rb_root; + struct mlx5_cache_ent *ent; + struct rb_node *node; if (!dev->cache.wq) return 0; - for (i = 0; i < MAX_MKEY_CACHE_ENTRIES; i++) { - struct mlx5_cache_ent *ent = &dev->cache.ent[i]; - + mutex_lock(&dev->cache.rb_lock); + for (node = rb_first(root); node; node = rb_next(node)) { + ent = rb_entry(node, struct mlx5_cache_ent, node); xa_lock_irq(&ent->mkeys); ent->disabled = true; xa_unlock_irq(&ent->mkeys); @@ -870,8 +996,15 @@ int mlx5_mkey_cache_cleanup(struct mlx5_ib_dev *dev) mlx5_mkey_cache_debugfs_cleanup(dev); mlx5_cmd_cleanup_async_ctx(&dev->async_ctx); - for (i = 0; i < MAX_MKEY_CACHE_ENTRIES; i++) - clean_keys(dev, i); + node = rb_first(root); + while (node) { + ent = rb_entry(node, struct mlx5_cache_ent, node); + node = rb_next(node); + clean_keys(dev, ent); + rb_erase(&ent->node, root); + kfree(ent); + } + mutex_unlock(&dev->cache.rb_lock); destroy_workqueue(dev->cache.wq); del_timer_sync(&dev->delay_timer); diff --git a/drivers/infiniband/hw/mlx5/odp.c b/drivers/infiniband/hw/mlx5/odp.c index 90339edddfed..b29ec4e0d8ff 100644 --- a/drivers/infiniband/hw/mlx5/odp.c +++ b/drivers/infiniband/hw/mlx5/odp.c @@ -1587,16 +1587,17 @@ mlx5_ib_odp_destroy_eq(struct mlx5_ib_dev *dev, struct mlx5_ib_pf_eq *eq) return err; } -void mlx5_odp_init_mkey_cache_entry(struct mlx5_cache_ent *ent) +struct mlx5_cache_ent *mlx5_odp_init_mkey_cache_entry(struct mlx5_ib_dev *dev) { struct mlx5r_cache_rb_key rb_key = { .access_mode = MLX5_MKC_ACCESS_MODE_KSM, .ndescs = mlx5_imr_ksm_entries }; - if (!(ent->dev->odp_caps.general_caps & IB_ODP_SUPPORT_IMPLICIT)) - return; - ent->rb_key = rb_key; + if (!(dev->odp_caps.general_caps & IB_ODP_SUPPORT_IMPLICIT)) + return NULL; + + return mlx5r_cache_create_ent(dev, rb_key); } static const struct ib_device_ops mlx5_ib_dev_odp_ops = { From patchwork Mon Nov 7 16:14:48 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michael Guralnik X-Patchwork-Id: 13034658 X-Patchwork-Delegate: jgg@ziepe.ca Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2A420C433FE for ; Mon, 7 Nov 2022 16:15:35 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232782AbiKGQPd (ORCPT ); Mon, 7 Nov 2022 11:15:33 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39542 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232642AbiKGQPY (ORCPT ); Mon, 7 Nov 2022 11:15:24 -0500 Received: from NAM02-SN1-obe.outbound.protection.outlook.com (mail-sn1anam02on2060.outbound.protection.outlook.com [40.107.96.60]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 005B2271B for ; Mon, 7 Nov 2022 08:15:22 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=DmtrrzgMkMs2FVfBXqHpneBeG//VwsveOm+iuWTcB3CSL1L9yEXvNUKCKNfxvkcg5IMTK2AcAOAbgxX7yEJ+/sBPZLBw8bat3bdERxSsZeH6l0j+rjc4lecXTzqrpwYRUV3hq9UTanyVKC7hQAdQwN1am5BxsZ7TykUWMYmVu//vqBtwHEVIx2VV4k4KeSmxe+DqWx28SmVsXg/eoFh3TLUmpHHwOEtcOqK9t6kkk7WR2Fc8iFz8HSkwCnQx5xv/r+10SbBbj8PMq4S2YSqGWydipfEVIR9kKRp106tXCmGwE+n3zhEN7H34O28SSCP9SLxiJocdxJl6tyjTauvr6Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=TwveOi1M2EVEa1DDa8BHDjWgyVGcCfx1MrrP+v+Od/4=; b=n9EPsxRmKqjwg2ef0fxTHuVQ+ZTxQpfUgz8fHcL45YAamVvu5wM7WQgVB2XAlFfro7pDZSvgQ3wFnjvzsap6uqENbgjJ5GhnBB5RGYR/AeLfImKHjj9ii+ufxsOap1F/qE8sdS+7wC7KROydaQMD3xq53zkH8L7+nLvwVsdmTmGKYuCqU2pu0I1lXvNZ7C4crfoeNvRsxofFgWPmxpk16b+FrOr4TEkKQsr0+18rqJEEyiYD5Nx2eP/WmhSbzn1lJq1v+s9uJPalhpsqxNjb4vOTJczrJ9cWbTA4xMTGxPM94Jp5bGXIgTJLHwIn2U+8LZkNxJVAoUA4yZtTAIq4NA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.118.232) smtp.rcpttodomain=vger.kernel.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=TwveOi1M2EVEa1DDa8BHDjWgyVGcCfx1MrrP+v+Od/4=; b=RSVz7sP0C7siX1qOfULdrbdNADSt8SfoIsSqOWief8IrOdBeCxJjWruIM+J4chwoKsP8GBR2sEFhh2lPtaOjvh9wDRlNVvYQY3X6/461EIx23yb9r8EsSnrtUU45dVJjlqR2CLIgYolujUzT4bF4PgvOa8fi+kKqXvPt3/GJq1AJYSEk1JuNW5zFB7F2a2xx0QvOpITTHMkEaJxHyHOsYyiXI5LPY8AE+brJgxgOUcXVX/tvGIEeo5J9pnDw2GUi7Hds/4pQDjZ8/CFKyBkhAgVjatJSaoV4Kw0aWQV/pXilVbWdo6XiIyM/w3D2e7IrTa+lrkSj3gjn49vKyRtJSA== Received: from DM6PR08CA0059.namprd08.prod.outlook.com (2603:10b6:5:1e0::33) by SJ1PR12MB6122.namprd12.prod.outlook.com (2603:10b6:a03:45b::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5791.26; Mon, 7 Nov 2022 16:15:19 +0000 Received: from DM6NAM11FT034.eop-nam11.prod.protection.outlook.com (2603:10b6:5:1e0:cafe::cc) by DM6PR08CA0059.outlook.office365.com (2603:10b6:5:1e0::33) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5791.26 via Frontend Transport; Mon, 7 Nov 2022 16:15:19 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.118.232) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.118.232 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.118.232; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.118.232) by DM6NAM11FT034.mail.protection.outlook.com (10.13.173.47) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5791.20 via Frontend Transport; Mon, 7 Nov 2022 16:15:19 +0000 Received: from drhqmail202.nvidia.com (10.126.190.181) by mail.nvidia.com (10.127.129.5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.26; Mon, 7 Nov 2022 08:15:07 -0800 Received: from drhqmail201.nvidia.com (10.126.190.180) by drhqmail202.nvidia.com (10.126.190.181) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.29; Mon, 7 Nov 2022 08:15:06 -0800 Received: from vdi.nvidia.com (10.127.8.13) by mail.nvidia.com (10.126.190.180) with Microsoft SMTP Server id 15.2.986.29 via Frontend Transport; Mon, 7 Nov 2022 08:15:05 -0800 From: Michael Guralnik To: , , CC: , Subject: [PATCH v1 rdma-next 7/8] RDMA/mlx5: Cache all user cacheable mkeys on dereg MR flow Date: Mon, 7 Nov 2022 18:14:48 +0200 Message-ID: <20221107161449.5611-8-michaelgur@nvidia.com> X-Mailer: git-send-email 2.17.2 In-Reply-To: <20221107161449.5611-1-michaelgur@nvidia.com> References: <20221107161449.5611-1-michaelgur@nvidia.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM6NAM11FT034:EE_|SJ1PR12MB6122:EE_ X-MS-Office365-Filtering-Correlation-Id: c772dde4-00b6-4c42-34ee-08dac0db43d3 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: f97bRAnHvg8ot4oA5Oo27lZ6sMFDaihnBUZhIB0a7+TZyyEqWy/IZBTN6wUeLzvcYjCkje6XQhz8/38IqzmyR/gxE8oyIe88HuE7rqkfxNAoxaKAk/qHpj3mWMXbU7TlWPW4SIkOlgMan85ieQerK5SdhW/rj/CY0i5xGd5Je2fcWgRkUun6tp/1MJ7T6o+W/jhX4WWuknr+Gm3bcGez0oCTu3PgHhLqrUDHdfVCDPZ68ejKwu80lLxU3nsnUDzLcTD9dMamiagcwaAqtDYRTSKBzycfRkUyUv5ri+mJYcPia4Z6IwFwSokuNjzD6Cbg7+EsVG4jgFL9BB2jRpnCdaQPYsiysfFa9yjuFnof+0OCyDt15jG+d0W/AlxUhrVQu3wJ43UIjrl6suAeTXug1LmpG/54qNUyIJasFiWpbQfhLP3Ssbjnj4ade9IPvYg8D+fKmllbRD7YXN6ZBDOxn2GwrUwDw/jrKK97B+VqXVeHJclSyr8s8XFtMumNGMLF0wzf44z2CAVzhAUMr4jqyqFQuZlR5BioDhE3kbGH/8BGgTs21IS+CZFB4zpFYKHp1Dig2rbY/NmNCbSvAX3x4MOZV8i6ghBxI+cwM5+jMfLUryavi9DtnjXFhiiJipJkkKAKZl/3uvsvLN/vtaYCxNrolkdh8BAhbfVjUx+KAY5z8gEx6J683ot3sVsubLsv4CAgPMkejAmUO5UHKPEZYsV8dOdCYubza84eonBkSJKhzPI+fAaJ5gRmvjGEtV3NXrg4TYRzKip3qF8c3MtgNUyiSxoydMRgvMle/iV/+ek= X-Forefront-Antispam-Report: CIP:216.228.118.232;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc7edge1.nvidia.com;CAT:NONE;SFS:(13230022)(4636009)(39860400002)(396003)(376002)(136003)(346002)(451199015)(36840700001)(46966006)(40470700004)(86362001)(36756003)(7636003)(82740400003)(40460700003)(40480700001)(8936002)(2906002)(107886003)(26005)(2616005)(1076003)(186003)(426003)(336012)(7696005)(47076005)(6666004)(83380400001)(36860700001)(110136005)(54906003)(356005)(82310400005)(5660300002)(70206006)(70586007)(316002)(478600001)(4326008)(41300700001)(8676002)(2101003);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 07 Nov 2022 16:15:19.3382 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: c772dde4-00b6-4c42-34ee-08dac0db43d3 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.118.232];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: DM6NAM11FT034.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ1PR12MB6122 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org From: Aharon Landau Currently, when dereging an MR, if the mkey doesn't belong to a cache entry, it will be destroyed. As a result, the restart of applications with many non-cached mkeys is not efficient since all the mkeys are destroyed and then recreated. This process takes a long time (for 100,000 MRs, it is ~20 seconds for dereg and ~28 seconds for re-reg). To shorten the restart runtime, insert all cacheable mkeys to the cache. If there is no fitting entry to the mkey properties, create a temporary entry that fits it. After a predetermined timeout, the cache entries will shrink to the initial high limit. The mkeys will still be in the cache when consuming them again after an application restart. Therefore, the registration will be much faster (for 100,000 MRs, it is ~4 seconds for dereg and ~5 seconds for re-reg). The temporary cache entries created to store the non-cache mkeys are not exposed through sysfs like the default cache entries. Signed-off-by: Aharon Landau --- drivers/infiniband/hw/mlx5/mlx5_ib.h | 2 ++ drivers/infiniband/hw/mlx5/mr.c | 48 +++++++++++++++++++++++----- 2 files changed, 42 insertions(+), 8 deletions(-) diff --git a/drivers/infiniband/hw/mlx5/mlx5_ib.h b/drivers/infiniband/hw/mlx5/mlx5_ib.h index 0bf2511fdfe4..dafd4c34a968 100644 --- a/drivers/infiniband/hw/mlx5/mlx5_ib.h +++ b/drivers/infiniband/hw/mlx5/mlx5_ib.h @@ -629,6 +629,8 @@ struct mlx5_ib_mkey { unsigned int ndescs; struct wait_queue_head wait; refcount_t usecount; + /* User Mkey must hold either a cache_key or a cache_ent. */ + struct mlx5r_cache_rb_key rb_key; struct mlx5_cache_ent *cache_ent; }; diff --git a/drivers/infiniband/hw/mlx5/mr.c b/drivers/infiniband/hw/mlx5/mr.c index df3e551ad7cd..78d0bc10b821 100644 --- a/drivers/infiniband/hw/mlx5/mr.c +++ b/drivers/infiniband/hw/mlx5/mr.c @@ -826,6 +826,7 @@ struct mlx5_ib_mr *mlx5_mr_cache_alloc(struct mlx5_ib_dev *dev, u8 access_mode, return ERR_PTR(err); } mr->mmkey.ndescs = ndescs; + mr->mmkey.rb_key = rb_key; } mr->mmkey.type = MLX5_MKEY_MR; init_waitqueue_head(&mr->mmkey.wait); @@ -1738,6 +1739,42 @@ mlx5_free_priv_descs(struct mlx5_ib_mr *mr) } } +static int cache_ent_find_and_store(struct mlx5_ib_dev *dev, + struct mlx5_ib_mr *mr) +{ + struct mlx5_mkey_cache *cache = &dev->cache; + struct mlx5_cache_ent *ent; + struct rb_node *node; + + if (mr->mmkey.cache_ent) { + xa_lock_irq(&mr->mmkey.cache_ent->mkeys); + mr->mmkey.cache_ent->in_use--; + xa_unlock_irq(&mr->mmkey.cache_ent->mkeys); + goto end; + } + + mutex_lock(&cache->rb_lock); + node = mlx5_cache_find_smallest_ent(&dev->cache, mr->mmkey.rb_key); + mutex_unlock(&cache->rb_lock); + if (node) { + ent = rb_entry(node, struct mlx5_cache_ent, node); + if (ent->rb_key.ndescs == mr->mmkey.rb_key.ndescs) { + mr->mmkey.cache_ent = ent; + goto end; + } + } + + ent = mlx5r_cache_create_ent(dev, mr->mmkey.rb_key); + if (IS_ERR(ent)) + return PTR_ERR(ent); + + mr->mmkey.cache_ent = ent; + +end: + return push_mkey(mr->mmkey.cache_ent, false, + xa_mk_value(mr->mmkey.key)); +} + int mlx5_ib_dereg_mr(struct ib_mr *ibmr, struct ib_udata *udata) { struct mlx5_ib_mr *mr = to_mmr(ibmr); @@ -1783,16 +1820,11 @@ int mlx5_ib_dereg_mr(struct ib_mr *ibmr, struct ib_udata *udata) } /* Stop DMA */ - if (mr->mmkey.cache_ent) { - xa_lock_irq(&mr->mmkey.cache_ent->mkeys); - mr->mmkey.cache_ent->in_use--; - xa_unlock_irq(&mr->mmkey.cache_ent->mkeys); - + if (mr->umem && mlx5r_umr_can_load_pas(dev, mr->umem->length)) if (mlx5r_umr_revoke_mr(mr) || - push_mkey(mr->mmkey.cache_ent, false, - xa_mk_value(mr->mmkey.key))) + cache_ent_find_and_store(dev, mr)) mr->mmkey.cache_ent = NULL; - } + if (!mr->mmkey.cache_ent) { rc = destroy_mkey(to_mdev(mr->ibmr.device), mr); if (rc) From patchwork Mon Nov 7 16:14:49 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michael Guralnik X-Patchwork-Id: 13034659 X-Patchwork-Delegate: jgg@ziepe.ca Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 10A7FC43219 for ; Mon, 7 Nov 2022 16:15:36 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232434AbiKGQPe (ORCPT ); Mon, 7 Nov 2022 11:15:34 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39628 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232770AbiKGQPY (ORCPT ); Mon, 7 Nov 2022 11:15:24 -0500 Received: from NAM12-MW2-obe.outbound.protection.outlook.com (mail-mw2nam12on2064.outbound.protection.outlook.com [40.107.244.64]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D09401FCFB for ; Mon, 7 Nov 2022 08:15:23 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=aTODt3Eh1qL1QtpGK3YMWTZgLHzuqrzAOu7E4zp8l3tBVYubyMqXk4fcaoCOIOfp+t3ErfZM89yTnjxncXi9bW0O7PfOSYprZ+9g7ZKT1ByVmLAs3NWmjUFTBc2iQMCZRsN/9Wfsuriz3UpncFPrrMO3l4wK9y3UOC2ZNtQcppECau6lFdRM1LxccT7ameXbHjzCbMxrwCoRmEB+0d32hYDNH5/evJzO64DISlGPcUKVwjU2mFrbD7JUt/3tZ2ofKy4B8bwrprGPLrewh1r1JyIGevnniIeO9Xx4lu7Al0cEfL3CreJQK0LArjqVxfuyLauyo/k/5x9abPEbyqMlOw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=L8PAvcIWAMprzh2/fEatQN7wj2WBggGKUgU/XHFyECE=; b=TgnGKyV9CKpyJL8Q+Ik78yDGLf8lCmimVwLCZETsYrsmsRizjJz0VKpGfrJgKjHY9zhln7sWCl286OEa5rrSdE813Cxj3Z1ZAkVRIwzmMLeqMYTGSnWmO/V3O/Qk16/sFy5Xi4dBeFIzxyEzY8En4ohtGe20B4YSbIJQ27QwRWxgvoISxJZuomqgmxF9llD4xWhSPco0KbkoMUs1jTwHLFOxpddZQL36JfZOthcbcaOjTm9jJryxyvi0GF+UpikBDYsDaSCiUdB762G2l2V5seHrhmWuG20iUbgw0CWQVe9nICkYEmWjsEnF00VJps4xauiChsujnA1h5nHV0SDe8g== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.118.232) smtp.rcpttodomain=vger.kernel.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=L8PAvcIWAMprzh2/fEatQN7wj2WBggGKUgU/XHFyECE=; b=boBOYjYjswsTBgdieX3rEuQTaqMbKtpiaS7WyThYw3WHkJ9lrxJoHpmBSs5YXQ36cgUu88DOkgiD5uh/IFHkaGenwMaljd5P4s/9h8K9+j/OiaetmRZgCvZs2h5HGrN0Wb1LRVQCVFHUazQoDr8kofsIQnDdzFS62Tnm8Dqjynzc19fGtpd2rWN8Hw6EkrHFoQ6/1zC61yh6mmwxlkp1cG0ZHjn0YYlQ503dKnMrMhrsdH4k78LulBuauS99/QmKlIBpPEYsVnU5P7HJjlhX6nk2Ogif86Ijejf4qsvk89vMg8f9bvWt31ijALZangCVaRyZXO6E6/Dt3F08ZWWvlg== Received: from DM6PR04CA0010.namprd04.prod.outlook.com (2603:10b6:5:334::15) by CH0PR12MB5074.namprd12.prod.outlook.com (2603:10b6:610:e1::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5791.26; Mon, 7 Nov 2022 16:15:22 +0000 Received: from DM6NAM11FT049.eop-nam11.prod.protection.outlook.com (2603:10b6:5:334:cafe::9b) by DM6PR04CA0010.outlook.office365.com (2603:10b6:5:334::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5791.27 via Frontend Transport; Mon, 7 Nov 2022 16:15:22 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.118.232) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.118.232 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.118.232; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.118.232) by DM6NAM11FT049.mail.protection.outlook.com (10.13.172.188) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5791.20 via Frontend Transport; Mon, 7 Nov 2022 16:15:22 +0000 Received: from drhqmail202.nvidia.com (10.126.190.181) by mail.nvidia.com (10.127.129.5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.26; Mon, 7 Nov 2022 08:15:09 -0800 Received: from drhqmail201.nvidia.com (10.126.190.180) by drhqmail202.nvidia.com (10.126.190.181) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.29; Mon, 7 Nov 2022 08:15:08 -0800 Received: from vdi.nvidia.com (10.127.8.13) by mail.nvidia.com (10.126.190.180) with Microsoft SMTP Server id 15.2.986.29 via Frontend Transport; Mon, 7 Nov 2022 08:15:07 -0800 From: Michael Guralnik To: , , CC: , Subject: [PATCH v1 rdma-next 8/8] RDMA/mlx5: Add work to remove temporary entries from the cache Date: Mon, 7 Nov 2022 18:14:49 +0200 Message-ID: <20221107161449.5611-9-michaelgur@nvidia.com> X-Mailer: git-send-email 2.17.2 In-Reply-To: <20221107161449.5611-1-michaelgur@nvidia.com> References: <20221107161449.5611-1-michaelgur@nvidia.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM6NAM11FT049:EE_|CH0PR12MB5074:EE_ X-MS-Office365-Filtering-Correlation-Id: 97d448d6-2f27-46fe-6595-08dac0db456e X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: vLrGDSFK2xTdjf3kutNdq2NFsDyfwihIDtDw1A1OViias5jHoed0s9y4oSSSbqa6ttmbq5++DLmi8ZOJ4j1Tk6QS29CN9pPH/HgMCWpQOrEMe4RY80fT9Zdaao9IXhBzIQR8KJv6+imc486WeY/LoryA2Lmu/SW731xDGKLsPnn5TEn+WRFAcZVpgcJ4775z5kG8aeQy8ahsxdNyAw6UvFva1+W/pAPAcLt/I+ubNbuTVs7mG6FghRYwhPb0r1ARkJYFgkL5+9EUx/mXKwMex98rM5/7+DlTyBRmAgT8D9HMQAq4P1yHrMBIoyCQbZuNhIOcQRwZqrgcshVU+n5uj2cOLui31AzIxHiv23Z7UTor60P4xNng6a0LGbn236GO70q309J6yI3M+uSazK51byO5hugxfelevuiNGJeya8uGHWgwt7k0eAkSjnQf6KRfmuA+XxTKkVxKzlzBaR6h8wXa1u2cq/qwT5FkXNJMiDT9TVNpMva8ACV+ygJxntoiREOcmIab2M7GGTkspM6yzMRSO/shXaJj7fnSc4cvOt/upa0sLu1pKWKMsn7Dv8XIzURY6pYercLOIDpTJOR3KtCuKLC35kmiHh45Vsu4rAGZjXTjTKjCFeJEcZiNM16XJuz0/udtt9F0yJgAECrr7JxwJmQgCqDkIAZTxqAzCBL4zjOZ1U2ad+rCjU01ciFytiH0L3I+cUmYF0cSBjiWfEPSOOiUGCoXK6bHbxdlF4qi01i+Q1pJCjrEG3wJmEl+MoHrfFsZCaXpHhLyqlTW2Mdt/qD/Evt01R4+GEUfiBc= X-Forefront-Antispam-Report: CIP:216.228.118.232;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc7edge1.nvidia.com;CAT:NONE;SFS:(13230022)(4636009)(396003)(346002)(376002)(136003)(39860400002)(451199015)(40470700004)(36840700001)(46966006)(36756003)(86362001)(82740400003)(7636003)(356005)(40460700003)(40480700001)(5660300002)(110136005)(7696005)(107886003)(6666004)(1076003)(2616005)(83380400001)(186003)(26005)(336012)(426003)(47076005)(36860700001)(2906002)(70586007)(70206006)(54906003)(478600001)(82310400005)(41300700001)(8936002)(8676002)(4326008)(316002)(2101003);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 07 Nov 2022 16:15:22.0174 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 97d448d6-2f27-46fe-6595-08dac0db456e X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.118.232];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: DM6NAM11FT049.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH0PR12MB5074 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org From: Aharon Landau The non-cache mkeys are stored in the cache only to shorten restarting application time. Don't store them longer than needed. Configure cache entries that store non-cache MR as temporary entries. If 30 seconds have passed and no user reclaimed the temporarily cached mkeys, an asynchronous work will destroy the mkeys and the temporary entries. When allocating an mkey from a temporary entry, don't keep a pointer to the entry, as it might be destroyed. Signed-off-by: Aharon Landau --- drivers/infiniband/hw/mlx5/mlx5_ib.h | 2 + drivers/infiniband/hw/mlx5/mr.c | 95 ++++++++++++++++++++++++---- 2 files changed, 83 insertions(+), 14 deletions(-) diff --git a/drivers/infiniband/hw/mlx5/mlx5_ib.h b/drivers/infiniband/hw/mlx5/mlx5_ib.h index dafd4c34a968..0a72d8e766b8 100644 --- a/drivers/infiniband/hw/mlx5/mlx5_ib.h +++ b/drivers/infiniband/hw/mlx5/mlx5_ib.h @@ -749,6 +749,7 @@ struct mlx5_cache_ent { char name[4]; + u8 is_tmp:1; u8 disabled:1; u8 fill_to_high_water:1; @@ -782,6 +783,7 @@ struct mlx5_mkey_cache { struct mutex rb_lock; struct dentry *fs_root; unsigned long last_add; + struct delayed_work remove_ent_dwork; }; struct mlx5_ib_port_resources { diff --git a/drivers/infiniband/hw/mlx5/mr.c b/drivers/infiniband/hw/mlx5/mr.c index 78d0bc10b821..d920a3f7803c 100644 --- a/drivers/infiniband/hw/mlx5/mr.c +++ b/drivers/infiniband/hw/mlx5/mr.c @@ -141,18 +141,16 @@ static void create_mkey_warn(struct mlx5_ib_dev *dev, int status, void *out) } -static int push_mkey(struct mlx5_cache_ent *ent, bool limit_pendings, - void *to_store) +static int push_mkey_locked(struct mlx5_cache_ent *ent, bool limit_pendings, + void *to_store) { XA_STATE(xas, &ent->mkeys, 0); void *curr; - xa_lock_irq(&ent->mkeys); if (limit_pendings && - (ent->reserved - ent->stored) > MAX_PENDING_REG_MR) { - xa_unlock_irq(&ent->mkeys); + (ent->reserved - ent->stored) > MAX_PENDING_REG_MR) return -EAGAIN; - } + while (1) { /* * This is cmpxchg (NULL, XA_ZERO_ENTRY) however this version @@ -191,6 +189,7 @@ static int push_mkey(struct mlx5_cache_ent *ent, bool limit_pendings, break; xa_lock_irq(&ent->mkeys); } + xa_lock_irq(&ent->mkeys); if (xas_error(&xas)) return xas_error(&xas); if (WARN_ON(curr)) @@ -198,6 +197,17 @@ static int push_mkey(struct mlx5_cache_ent *ent, bool limit_pendings, return 0; } +static int push_mkey(struct mlx5_cache_ent *ent, bool limit_pendings, + void *to_store) +{ + int ret; + + xa_lock_irq(&ent->mkeys); + ret = push_mkey_locked(ent, limit_pendings, to_store); + xa_unlock_irq(&ent->mkeys); + return ret; +} + static void undo_push_reserve_mkey(struct mlx5_cache_ent *ent) { void *old; @@ -542,7 +552,7 @@ static void queue_adjust_cache_locked(struct mlx5_cache_ent *ent) { lockdep_assert_held(&ent->mkeys.xa_lock); - if (ent->disabled || READ_ONCE(ent->dev->fill_delay)) + if (ent->disabled || READ_ONCE(ent->dev->fill_delay) || ent->is_tmp) return; if (ent->stored < ent->limit) { ent->fill_to_high_water = true; @@ -732,9 +742,18 @@ static bool mlx5_ent_get_mkey(struct mlx5_cache_ent *ent, struct mlx5_ib_mr *mr) mr->mmkey.key = pop_stored_mkey(ent); mr->mmkey.ndescs = ent->rb_key.ndescs; - mr->mmkey.cache_ent = ent; - queue_adjust_cache_locked(ent); - ent->in_use++; + if (!ent->is_tmp) { + mr->mmkey.cache_ent = ent; + queue_adjust_cache_locked(ent); + ent->in_use++; + + } else { + mr->mmkey.rb_key = ent->rb_key; + mod_delayed_work(ent->dev->cache.wq, + &ent->dev->cache.remove_ent_dwork, + msecs_to_jiffies(30 * 1000)); + } + xa_unlock_irq(&ent->mkeys); return true; } @@ -904,6 +923,38 @@ struct mlx5_cache_ent *mlx5r_cache_create_ent(struct mlx5_ib_dev *dev, return ent; } +static void remove_ent_work_func(struct work_struct *work) +{ + struct mlx5_mkey_cache *cache; + struct mlx5_cache_ent *ent; + struct rb_node *cur; + + cache = container_of(work, struct mlx5_mkey_cache, + remove_ent_dwork.work); + mutex_lock(&cache->rb_lock); + cur = rb_last(&cache->rb_root); + while (cur) { + ent = rb_entry(cur, struct mlx5_cache_ent, node); + cur = rb_prev(cur); + mutex_unlock(&cache->rb_lock); + + xa_lock_irq(&ent->mkeys); + if (!ent->is_tmp) { + xa_unlock_irq(&ent->mkeys); + mutex_lock(&cache->rb_lock); + continue; + } + ent->disabled = true; + xa_unlock_irq(&ent->mkeys); + + clean_keys(ent->dev, ent); + mutex_lock(&cache->rb_lock); + rb_erase(&ent->node, &cache->rb_root); + kfree(ent); + } + mutex_unlock(&cache->rb_lock); +} + static int mlx5_cache_init_default_entries(struct mlx5_ib_dev *dev) { struct mlx5r_cache_rb_key rb_key = { .access_mode = @@ -958,6 +1009,7 @@ int mlx5_mkey_cache_init(struct mlx5_ib_dev *dev) mutex_init(&dev->slow_path_mutex); mutex_init(&dev->cache.rb_lock); dev->cache.rb_root = RB_ROOT; + INIT_DELAYED_WORK(&dev->cache.remove_ent_dwork, remove_ent_work_func); dev->cache.wq = alloc_ordered_workqueue("mkey_cache", WQ_MEM_RECLAIM); if (!dev->cache.wq) { mlx5_ib_warn(dev, "failed to create work queue\n"); @@ -985,6 +1037,7 @@ int mlx5_mkey_cache_cleanup(struct mlx5_ib_dev *dev) if (!dev->cache.wq) return 0; + cancel_delayed_work_sync(&dev->cache.remove_ent_dwork); mutex_lock(&dev->cache.rb_lock); for (node = rb_first(root); node; node = rb_next(node)) { ent = rb_entry(node, struct mlx5_cache_ent, node); @@ -1745,34 +1798,48 @@ static int cache_ent_find_and_store(struct mlx5_ib_dev *dev, struct mlx5_mkey_cache *cache = &dev->cache; struct mlx5_cache_ent *ent; struct rb_node *node; + int ret; if (mr->mmkey.cache_ent) { xa_lock_irq(&mr->mmkey.cache_ent->mkeys); mr->mmkey.cache_ent->in_use--; - xa_unlock_irq(&mr->mmkey.cache_ent->mkeys); goto end; } mutex_lock(&cache->rb_lock); + mod_delayed_work(cache->wq, &cache->remove_ent_dwork, + msecs_to_jiffies(30 * 1000)); node = mlx5_cache_find_smallest_ent(&dev->cache, mr->mmkey.rb_key); - mutex_unlock(&cache->rb_lock); if (node) { ent = rb_entry(node, struct mlx5_cache_ent, node); if (ent->rb_key.ndescs == mr->mmkey.rb_key.ndescs) { + if (ent->disabled) { + mutex_unlock(&cache->rb_lock); + return -EOPNOTSUPP; + } + mr->mmkey.cache_ent = ent; + xa_lock_irq(&mr->mmkey.cache_ent->mkeys); + mutex_unlock(&cache->rb_lock); goto end; } } + mutex_unlock(&cache->rb_lock); ent = mlx5r_cache_create_ent(dev, mr->mmkey.rb_key); if (IS_ERR(ent)) return PTR_ERR(ent); mr->mmkey.cache_ent = ent; + xa_lock_irq(&mr->mmkey.cache_ent->mkeys); + ent->is_tmp = true; end: - return push_mkey(mr->mmkey.cache_ent, false, - xa_mk_value(mr->mmkey.key)); + ret = push_mkey_locked(mr->mmkey.cache_ent, false, + xa_mk_value(mr->mmkey.key)); + xa_unlock_irq(&mr->mmkey.cache_ent->mkeys); + return ret; + } int mlx5_ib_dereg_mr(struct ib_mr *ibmr, struct ib_udata *udata)