From patchwork Thu Sep 1 14:20:53 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 12962761 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 794C1ECAAD1 for ; Thu, 1 Sep 2022 14:21:26 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 46BC810E72E; Thu, 1 Sep 2022 14:21:12 +0000 (UTC) Received: from NAM10-BN7-obe.outbound.protection.outlook.com (mail-bn7nam10on2058.outbound.protection.outlook.com [40.107.92.58]) by gabe.freedesktop.org (Postfix) with ESMTPS id BCC8E10E726 for ; Thu, 1 Sep 2022 14:21:05 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=ZF4l2qBbA/GTr4diSaTEIU72aHyYUmfEBowy0+4ybWBGQSGqO6staDmNC1ihpQtczf8HcorT1SPC7MCcPguK/gMF1eKmzUcqQ4AvcFhJo1RGJ6FC0fLvPc4Z7s6X9DrEpnGgRZNwhwhvpejT/nMoLuGNdXo+25y3S65we/2i3Xv08NTDjeZC8YO7yREzRa14QzKWj62GOuH+HFPihraF3aio0IBW13boXVDXV4uwRBWdrGyB3afC7Bkj3tNBZDZLJlFOsiNBssP31mwzoC3Yc4JDjFaLIeY7YAtbYOEnSX0yFOwXo392P0sXYuV0ql0h4VwEG78crA4XuOB3ShMA8A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=TetcQo6Slx3iQaRByv9RxzXzo0lNntzkAteTIzw8ubk=; b=krWZzWNHdvCiIYkHBp6yLYc8uXx5IkHGsEs5t4fyXkBBlln+ij7HEN3UGvk30BbLiEqBM2oOh/JTg4OHlad6MQpVTr1SZQKV6m3+eC3bstToG3E2Kkn5kQbskcKrQXNBay4sDm79u5MIl/JbMmklBO57Rj/yCINQ4+yJrlP4k34DTsg9yiP6wE/GycQSjQVa1/iNpeAEwHuErRabotthxHrTDk7Z7YCSd+GhtXC9ThdJwSKWnEYuMas3W9627VXMj273/dgX2Ae5YZBk/oeRkXBROMqIqv2wT8UWtlde98DpS/qEa9cr/xlzA7q3tmz3Z9cCfYhXnBWgrQFs9BEWIQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=TetcQo6Slx3iQaRByv9RxzXzo0lNntzkAteTIzw8ubk=; b=j15rZclJEMFlMQtDBxZuMSHDN6PxTAZQ1R4XivSsI+35iPIyAXdVPpRGxBO/UjHyJaWt5AOv7a3BgHA61KoS1Olo5pQ9T2RFuSG8MgcuqLxZkDI0aqCPqkcrQUyGUq4bDWAgNNVICUInubquBvxLtSUqwMAzYOVNHsCm13+9CXaqrzq9nMwxrBgU5Xi2+R5l7zy+fLx7QjNSoidwM/ZkNeenhQ0s2Ix6uYHlUXc+VL7j4HIvbWR2bvLTHWRpv4E172o8JSonnCUuiZVUFjWCmBIe0bH69MFXnVSqd8bgf8C/I2EPw0EnDwlsgYUUGMpmbIn6yNDTw5PzyYd6xMbpgQ== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from MN2PR12MB4192.namprd12.prod.outlook.com (2603:10b6:208:1d5::15) by BY5PR12MB5000.namprd12.prod.outlook.com (2603:10b6:a03:1d7::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5566.15; Thu, 1 Sep 2022 14:21:03 +0000 Received: from MN2PR12MB4192.namprd12.prod.outlook.com ([fe80::462:7fe:f04f:d0d5]) by MN2PR12MB4192.namprd12.prod.outlook.com ([fe80::462:7fe:f04f:d0d5%7]) with mapi id 15.20.5588.012; Thu, 1 Sep 2022 14:21:03 +0000 From: Jason Gunthorpe To: =?utf-8?q?Christian_K=C3=B6nig?= , dri-devel@lists.freedesktop.org, Leon Romanovsky , linaro-mm-sig@lists.linaro.org, linux-media@vger.kernel.org, linux-rdma@vger.kernel.org, netdev@vger.kernel.org, Saeed Mahameed , Sumit Semwal Subject: [PATCH 1/4] net/mlx5: Add IFC bits for mkey ATS Date: Thu, 1 Sep 2022 11:20:53 -0300 Message-Id: <1-v1-bd147097458e+ede-umem_dmabuf_jgg@nvidia.com> In-Reply-To: <0-v1-bd147097458e+ede-umem_dmabuf_jgg@nvidia.com> References: X-ClientProxiedBy: MN2PR01CA0020.prod.exchangelabs.com (2603:10b6:208:10c::33) To MN2PR12MB4192.namprd12.prod.outlook.com (2603:10b6:208:1d5::15) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: c7b45e0d-59cc-4244-844f-08da8c253389 X-MS-TrafficTypeDiagnostic: BY5PR12MB5000:EE_ X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: YJMx1GK/5X6bGaw4rKDULHfRdcmw2BJODwd+WaOWJgjte4q26HoeueMLOWPA6Z1Th19A92dhKrXprykBfH+p0xWOmpVUoIT6SUIfVNa2eSFZDm0YQxJ/ma8PtvgPwn12ekK78lS7E+bLPya5OFol5s2MweTnKLua3zR59D+aQXLcWGNnXbBvxBjfyUU95xeEgd9eCMFBv+yecVg5iEZSj3jcItT5VpkwPaecDEBOzP8LmVYMtVSsUYJI/mhtDzjbtU42pKCODoHQk27N7gl5PkaEQDCiXySUCTTafZr2oHqFsHTK2LK3oK0X6sbwGkPX/7oyjArxpvbxw2NWYVQEtmARbd9lf5sgLbt3PoFxt8yXclfEQX1VacaKOl5BO9qifIMQgjrn+fziP7JAUF4Dl6qjmfvnHWOatJGo5CiiqncKg45gHa9KfjI0666GqGVstIpQuuyQX1SX187J6pcAsO+pyZVWM+DK1JTsR/7mE255KHBaG5uf4B+lEp7ya8e35akDh8EN19275toluaRkoGkVWqHJEjeSEKcNGMOFbf1nA0eu7T4/L5bkvAarGk1sa0nxJty8jfllIuV1lunJX3ojND+u49DJhFeMZaxhMYf9GJAZ6EAbEUP0kX8j408Z74Lm43agd7dczNr44RFNvIXkmSbC1qOVD1b8KqO5POfDPbcLrHDRyQKWzUlMdeRwsPHHY8OZFb3w35Ct7oL8xi0YuBB57qwjl/h9tH5936HEeAMC4pYGXlbhXi7U+Kii X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:MN2PR12MB4192.namprd12.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230016)(4636009)(39860400002)(376002)(346002)(136003)(366004)(396003)(6512007)(66946007)(4326008)(5660300002)(66476007)(6666004)(6506007)(478600001)(107886003)(8676002)(66556008)(6486002)(8936002)(86362001)(41300700001)(83380400001)(2906002)(2616005)(26005)(38100700002)(186003)(316002)(36756003)(110136005)(54906003)(4216001); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: a1NZsy/0nFprincwE7vaAbYPL+ebQyNOGV738qboH6ovKQqBG4CK4aJV3WsKmlwTkUaDfizKVn1i07fQT6uIJzW4y11DNupgRmmekOWh5RS62Ia+tBpqV5riFzQvPJ1XGodQWBjJKiVA0MpzXPltkP8dFSBBTP9I9A9Ig0RL9yZ3k9hXbWJV7UuwGKVhaBzNR/Yo60ouei2mh51tkb5NPDHTjgVCCVLKdTS53r02yE9y3gJjg4zeVgtWM4zR/rgpdEx557z64iB5VL2Dt4Tobs1J8G1806GE0nSXPJ0ZXhs98o48az6DuDskYHpT4sH+1+zaA7iiywmmj9korNFeLA1m6Gfa/UBhdZog23RyuYp6rAfQykLYnl/6XL+ozNPlnWMt2zSp39/Ze82ahkvbeprhltSzY6/m0sTK+pZWZztHWg4wIIrLa/5UM6hxVTr0gXbVeVSELjb4QgP/gMX7kbcfUYsMWd3at3apaNef6NiCRA0daW5CO8k+Kx5ud3q/L6WGTblyR5v9XJ6T4wubb4ZhrZkpjiG2Y3OcPGeLjtTMuzebD7+2XJOgFf/fGw9IlmDAWYPDZ1624PaxMcJsYSvRLCeHnbsKGcEIsJwwY5MDjk+oDx9RsFo/Oy4dhNamtA6z6yF8FFU7Q9y9UWaVKL6m9ekfrIS+ElvkAhA7Cy04JlOzJ+MRR9S3oGufkKauWREpgqy0GYGVv051PhtfXJXFKQJhUcVwvHZzuc8vFsuUwywjdVdPLdYTfznZvIDFAnZvYr4JV7yJlO+vlYmKxYwOfVDYKlOgNmUkz/0blRQA0p32VSaUM3nvrk1vj1r7hDrSDHm0wgrYOzPpYGP17ZPpFUvuVlTdz+IEzyWcHACBijXufhHs1Bv7RRmJUjzjsvwkkUVdpuBJDrwU6styQ/nk6V1i5c0l02KXp6ANvjoS+n0WUZVYzLnNfYk6/TbYEon0mzEaJwGV7DxvI5qUOAkklxMNBsyUW0qjGMnNBrC4ZGE3ExzfxfSgULSm6WfNPTjnpaoG1otOIAbqrTuYoFe9E+mgTV3CoVw2bMkqRm6GhuPuhBmrPlpt0FvTAVRyrTBUKSLOi5f7jZDnSuwtv9lxu9gUL+nEaDfczSPDyuFmUTEmqYS6WHtkNh6Oe+yZI7faUTfjCi46Yhwly/onD7r8HbQFXESLR1c89NACi9dZVLumBnYoafvAAry/ITLXkXnXQIcT/SvMcLrrNRzmRUrR2OR2QHK+2jh8cFU+gfI3fABWUVS/o786wEzoy2CP2P/U08gFQngRigUANnC4TAv8mbe8yVACWp6pcBFzquYXSYNFx9fhxQxcYu0dNg6c5vc7CGRdP4aNNW4ZnrzB+rzHjtsIuS5Nka7CEArxa5XagLr1XDyPiBE2DBiXK4hrO7rQ4iCItsaaER6mE9YD9Q5YmnYplvBGbSJC2MfKHq0REnHLoaHiC80aHR3hFQH3afpPKgwWiMC83gBhbMW2NicT8YBWapkWXB9hT1w6Q+gwgJh0dn5oMyRhw3TZmxHWIdnNyuAi2hb7dujUOc2rVMOP66ym82orULuT2jATON/yizKCwEXVZsCpAGzHnp1I X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: c7b45e0d-59cc-4244-844f-08da8c253389 X-MS-Exchange-CrossTenant-AuthSource: MN2PR12MB4192.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 01 Sep 2022 14:21:03.4237 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: qaMiBYTjHTCtmcbCfweY+g8VQd3jdygj9NjSwhlvgSMEEbEoEJOzfDSmhhuUmZQn X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY5PR12MB5000 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Mohammad Kabat , Kamal Heib Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" Allows telling a mkey to use PCI ATS for DMA that flows through it. Signed-off-by: Jason Gunthorpe --- include/linux/mlx5/mlx5_ifc.h | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/include/linux/mlx5/mlx5_ifc.h b/include/linux/mlx5/mlx5_ifc.h index 4acd5610e96bc0..92602e33a82c42 100644 --- a/include/linux/mlx5/mlx5_ifc.h +++ b/include/linux/mlx5/mlx5_ifc.h @@ -1707,7 +1707,9 @@ struct mlx5_ifc_cmd_hca_cap_bits { u8 steering_format_version[0x4]; u8 create_qp_start_hint[0x18]; - u8 reserved_at_460[0x3]; + u8 reserved_at_460[0x1]; + u8 ats[0x1]; + u8 reserved_at_462[0x1]; u8 log_max_uctx[0x5]; u8 reserved_at_468[0x2]; u8 ipsec_offload[0x1]; @@ -3873,7 +3875,9 @@ struct mlx5_ifc_mkc_bits { u8 lw[0x1]; u8 lr[0x1]; u8 access_mode_1_0[0x2]; - u8 reserved_at_18[0x8]; + u8 reserved_at_18[0x2]; + u8 ma_translation_mode[0x2]; + u8 reserved_at_1c[0x4]; u8 qpn[0x18]; u8 mkey_7_0[0x8]; @@ -11134,7 +11138,8 @@ struct mlx5_ifc_dealloc_memic_out_bits { struct mlx5_ifc_umem_bits { u8 reserved_at_0[0x80]; - u8 reserved_at_80[0x1b]; + u8 ats[0x1]; + u8 reserved_at_81[0x1a]; u8 log_page_size[0x5]; u8 page_offset[0x20]; From patchwork Thu Sep 1 14:20:54 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 12962757 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 6D9BBECAAD3 for ; Thu, 1 Sep 2022 14:21:04 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 2454010E725; Thu, 1 Sep 2022 14:21:03 +0000 (UTC) Received: from NAM12-BN8-obe.outbound.protection.outlook.com (mail-bn8nam12on2074.outbound.protection.outlook.com [40.107.237.74]) by gabe.freedesktop.org (Postfix) with ESMTPS id 8209310E726 for ; Thu, 1 Sep 2022 14:21:00 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=mQDuxmPMKgq+5eat+j7TfqmgVBa9QJwwxIWNNAdarunVYn1CdbZSx0bCz9W3WPZuBzedFgg/fWp2Ep97Dg4OCi+2AgPpeYCPXRnoO4Ua8WfXa2DikxXdTz6zcFec8mxA1KD/kEoY7e1fA1yDi8OG1y6RXP25hGKA0rsMBvoRkxpOb+rh8cePmJoOhFlBmAGbUDZzLJ+9GCNvIkRpgNuMLHa0l+rwwX8DXtdeyo+eWkJljV567N9ePeDvelpCS9f2AhjsB3CqGKP7zSpb6OwbBMBT6A2NAQ5PJesxsY+spYj1QkBBN84sGQUaEcJqPOL90stDTVKKI9YoYmfijtIfxQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=/aB2fdyup4zXlO7Fs2az03ubJD/zt2/t60pbinRpFNQ=; b=aBt/156TwEDGCmZgi6MygQeDDQjTKmCV4Lpa575aKRhkvFmmM2GlvYsi/jLGILTvIzy7945rIBP9W8WLj8uPraxebGaHYO7AoTH7paTEA7AOUfin54hzwvMVTD30ZWhjsLDjfzIgywRV+NYT5nyRxaoyAKlmwZy6StQRFQO4Y5cJy08wf5iOGJBf0X4jOyJUIZCbGtWftS1C4USbdPdxO9NQU5IOZklRdeQTEQESIohmZ1L5/4CKvWDIbO4abqUu6Hk2ANGh2MLHcMpQkPGvv8fbBs6fuh5hLUedC7l591RbIADJZwZ8ojzAfIgn9Bhw/rIXQkokmtOSSjR1z97BlQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=/aB2fdyup4zXlO7Fs2az03ubJD/zt2/t60pbinRpFNQ=; b=fhZmiN7U8rZs0slqgczpOzdTFY4A/es294tSxlbS6qTlnkg2JPTIWmw60A2jFK4Mbc7bbnpjIINDfTqK+L725dh1267yyQfGzpk3TGTEJmaLTEmzSGri+y+6zknHB0Achf6pIbb9D7XVkfet5IZbV3YOdEvLZJ2hh6oqtTTLlpMQdz6SushKoQ4cHG4z9I8wWGKFEahlCsB99nVPcsHYi3TZC9kUiTBpX9+EiqAN3eRn5wIIGN3627I0KFlPfY6WfKEP+9eQ3fRlzRP6Kcn4V9pjAslq1Ae8hU5pXcwsKcGZ2DmRSIUWgIv8ZlvyRNp7vUbVIi0GIGXUMlrThJlHCQ== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from MN2PR12MB4192.namprd12.prod.outlook.com (2603:10b6:208:1d5::15) by BY5PR12MB5000.namprd12.prod.outlook.com (2603:10b6:a03:1d7::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5566.15; Thu, 1 Sep 2022 14:20:57 +0000 Received: from MN2PR12MB4192.namprd12.prod.outlook.com ([fe80::462:7fe:f04f:d0d5]) by MN2PR12MB4192.namprd12.prod.outlook.com ([fe80::462:7fe:f04f:d0d5%7]) with mapi id 15.20.5588.012; Thu, 1 Sep 2022 14:20:57 +0000 From: Jason Gunthorpe To: =?utf-8?q?Christian_K=C3=B6nig?= , dri-devel@lists.freedesktop.org, Leon Romanovsky , linaro-mm-sig@lists.linaro.org, linux-media@vger.kernel.org, linux-rdma@vger.kernel.org, netdev@vger.kernel.org, Saeed Mahameed , Sumit Semwal Subject: [PATCH 2/4] RDMA/core: Add UVERBS_ATTR_RAW_FD Date: Thu, 1 Sep 2022 11:20:54 -0300 Message-Id: <2-v1-bd147097458e+ede-umem_dmabuf_jgg@nvidia.com> In-Reply-To: <0-v1-bd147097458e+ede-umem_dmabuf_jgg@nvidia.com> References: X-ClientProxiedBy: MN2PR01CA0007.prod.exchangelabs.com (2603:10b6:208:10c::20) To MN2PR12MB4192.namprd12.prod.outlook.com (2603:10b6:208:1d5::15) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 9e4e12a6-d60d-496c-015d-08da8c253009 X-MS-TrafficTypeDiagnostic: BY5PR12MB5000:EE_ X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: pDIhVsbNPCH+LhanUheLaUnlGaYL1t0kAYkMcEFk7HH6dbs2Lero7nX8e0hrGuY7rZARQW4B6415QmY63s5m2X+TSJ/7b+gP4hfid6K4MnfwamloC5R2XOBxe+SyUGKYG+2p5PrGBXJwBFhhgnXtYfwZbHf2dtYRzGl0NfXnRk5s72fsd4d720RKSDelHTZTZQwj3pOf/BKHYLrOeMSxO2KOOQ5iorgJ9ydF6BLnqB57wiDOYfKvs0JLCC0KVQX39qRwJMNOajOKLaQ7YTJZWXsnQEI//H0ACkW21qxu2hAEDGQx3r3lk4MjxKSOBaga2c2SQpUL1RjlTOLIL1DbAmR1/Oy2uDFFahiZDZ486bI4UrCQ8O+O8/tBPcDxmnIbaGtaKHBBJlmpcg0IsD5XzwSPU0p3gOkjMeLhgbMGu7K05kKntJni2xThuq9oyXtVdTHMHxTPs73rze7jlsc/L8O9iToIKoTUIIzAc/Jgh/4uizYPQ1MirPXt4jRn2401OtNxH7qKlVYRlW/ShZxVXGWRn1uq/Hk5Nq0QYXEwARdh65d22iFhBJMeAH5ac8wsujBh3CeLzfdZ+r/N6KeC2dj2t09HKm5FvoyzzmkkKWdCXzTh93QSZ0CC+FJcUhFz7NSbTylkE1uBo3CxQzykzVOG2f+ejXXS76902UqgQaYrTD+dVI6chZiincaTmkAXEt8kRXdgd6UBzmpAG0tDZgukU48HQxXKJ+wCYwFdRHwhU2Au2iKPmb67+NRmBnck X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:MN2PR12MB4192.namprd12.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230016)(4636009)(39860400002)(376002)(346002)(136003)(366004)(396003)(6512007)(66946007)(4326008)(5660300002)(66476007)(6666004)(6506007)(478600001)(107886003)(8676002)(66556008)(6486002)(8936002)(86362001)(41300700001)(83380400001)(2906002)(2616005)(26005)(38100700002)(186003)(316002)(36756003)(110136005)(54906003)(4216001); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: UKAP33sN7HmkgVXsDBVvWQEhWbPRMDmw/iLMiUsV8ThAzAdGlYubAp8NbzN7eYeXjcAjlxp3rGVGt2m2BSn6WJtuR91xB1OPI+BamoAcbuHe3f8TSLy9c3mmsd2EVniUrece6LnuGT75txH7gKjsDK+LQ95e7R9LxE5HhANp+hH60zPOt+Zl2kd4tPCxm9XqGiqUj77prExKaI8txIviBk4+QBPyIZx53SDsOImHAtvDlGVHGHwn36lewtfhXDu9lgemDYzOpATLJKFhB77AW4Kveh19XgFUv6pzGGb0C1NXCm0pxb8P7GuOQwKGXh+41Eiy64qJBuLZrguJWJwgiFmjaxd+HxmZJZ1TcVdM/heR/jtGHunaR24dp95KV1sx6TWGFdy20peNHVc1pMvvu/fZmsDuqi4ke7ZiqACzz9SaXmmYK8iNqILxuCGYRp0yPTQz1NC+F1L+PRULtxWnqy3LoO4e3F8IezK14ABC7TqAMXQ0gRU7GSi99TOGYOySXim9CRKWa+vzQiRpeGv3RLA1iv7+MLsux9v/7UcvX5vE4nbwVuASTjxdEnsapifeJ79ByhkffsEVMoQK6z26NOX9jWMe31qEXLag0Zwvtlo+XOWgtDDpfNIZw42CxMuCEcgnLf+2QUeGDuEYvoQ3Cigf/0TZbT8jyUjCrVVKtQDsFskelfB7MrDH1yMHUmR/T7ZvoQsRzff9Gog4PgBwuDtToI89wJ+zXUcaPxcI8po0F8DdYHpK0wVeq+o74CT4nuOqugpTUaDHTs/YEEzcriQ3b/ZbkbTmcAPyvczMFFhxq98k03up9nPESlaGjzpnd/Xc6UVc3YljU0AQNM+oJ+tteye8vccTPkMNT/BVSevKlHK8O0JcwC29TTPR5sF2IBf7CKrl3cDLTjJduUPomZfX1x4KPY3oNHUqKXxomHhiprsxMSpl1Wa0REL17cMrGmMyCrgla486trzuOejP9CS/RL3E5jYpu5qpkEMbgJCPeDtKjYfjqHVNYfcK9vTWU4wKVDdsqcZbc3/fdTnMFKcEZWvIlo5jyZeRwYKNF4mnnU8aaDHQv9tLM+QdVTaOx3rML99BAqnx5QW1QXHZa/okrLP5nErLILJYrKRrpNUa/vwT+oKw/Jq2y8TIR3LH3BRI46s4YiSVotDeKPYcObYD7QesOajWpSnzrcuw+seSAIOjEW/f2Z+LqqteX82zexU2UYGXriRFkZLa0Sxqv13mXm19ZoMB5GrUb6KW1Ab58AD+JbjQJNfzitK4o6co6HZBrtJPQRGl445yA0sDoF70yiVimFPqTTw+mflnpSIbiRMAP7SqT07hAJyJOKvHGIyR/KRRc/dCiqgqh/BeTgIi/EWdBhp9PeoRTzmcvqW2wP1ceR/q/YaVfrihBKURFMFLVescSqODH8oGF7giiP2FwMfaLvPYdTxcUhG8/erM6pqIx20idLDVAZ2Y0sDlJ6uJV3LQEviab4+DGLGGnW0kS95iitfWtoHh5PcS/2+JyZs+VXj0Wt6TcDleOPzxgXgFWEuYkQLjz5fcvHRP914O3X8Br9AEXwdn1TKRkIe26/2FoQXGVXaAmj2uRdag X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 9e4e12a6-d60d-496c-015d-08da8c253009 X-MS-Exchange-CrossTenant-AuthSource: MN2PR12MB4192.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 01 Sep 2022 14:20:57.5171 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 3MayONxtN/zL4PW7xCehQvlc2ay4mi/jVluEOkJMaoVCyd0vXa+ydBenYoQ9bb9D X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY5PR12MB5000 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Mohammad Kabat , Kamal Heib Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" This uses the same passing protocol as UVERBS_ATTR_FD (eg len = 0 data_s64 = fd), except that the FD is not required to be a uverbs object and the core code does not covert the FD to an object handle automatically. Access to the int fd is provided by uverbs_get_raw_fd(). Signed-off-by: Jason Gunthorpe --- drivers/infiniband/core/uverbs_ioctl.c | 8 ++++++++ include/rdma/uverbs_ioctl.h | 13 +++++++++++++ 2 files changed, 21 insertions(+) diff --git a/drivers/infiniband/core/uverbs_ioctl.c b/drivers/infiniband/core/uverbs_ioctl.c index 990f0724acc6b6..d9799706c58e99 100644 --- a/drivers/infiniband/core/uverbs_ioctl.c +++ b/drivers/infiniband/core/uverbs_ioctl.c @@ -337,6 +337,14 @@ static int uverbs_process_attr(struct bundle_priv *pbundle, break; + case UVERBS_ATTR_TYPE_RAW_FD: + if (uattr->attr_data.reserved || uattr->len != 0 || + uattr->data_s64 < INT_MIN || uattr->data_s64 > INT_MAX) + return -EINVAL; + /* _uverbs_get_const_signed() is the accessor */ + e->ptr_attr.data = uattr->data_s64; + break; + case UVERBS_ATTR_TYPE_IDRS_ARRAY: return uverbs_process_idrs_array(pbundle, attr_uapi, &e->objs_arr_attr, uattr, diff --git a/include/rdma/uverbs_ioctl.h b/include/rdma/uverbs_ioctl.h index 23bb404aba12c0..9d45a5b203169e 100644 --- a/include/rdma/uverbs_ioctl.h +++ b/include/rdma/uverbs_ioctl.h @@ -24,6 +24,7 @@ enum uverbs_attr_type { UVERBS_ATTR_TYPE_PTR_OUT, UVERBS_ATTR_TYPE_IDR, UVERBS_ATTR_TYPE_FD, + UVERBS_ATTR_TYPE_RAW_FD, UVERBS_ATTR_TYPE_ENUM_IN, UVERBS_ATTR_TYPE_IDRS_ARRAY, }; @@ -521,6 +522,11 @@ struct uapi_definition { .u.obj.access = _access, \ __VA_ARGS__ } }) +#define UVERBS_ATTR_RAW_FD(_attr_id, ...) \ + (&(const struct uverbs_attr_def){ \ + .id = (_attr_id), \ + .attr = { .type = UVERBS_ATTR_TYPE_RAW_FD, __VA_ARGS__ } }) + #define UVERBS_ATTR_PTR_IN(_attr_id, _type, ...) \ (&(const struct uverbs_attr_def){ \ .id = _attr_id, \ @@ -999,4 +1005,11 @@ _uverbs_get_const_unsigned(u64 *to, uverbs_get_const_default_unsigned(_to, _attrs_bundle, _idx, \ _default)) +static inline int +uverbs_get_raw_fd(int *to, const struct uverbs_attr_bundle *attrs_bundle, + size_t idx) +{ + return uverbs_get_const_signed(to, attrs_bundle, idx); +} + #endif From patchwork Thu Sep 1 14:20:55 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 12962759 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 06284C0502C for ; Thu, 1 Sep 2022 14:21:18 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 9874A10E726; Thu, 1 Sep 2022 14:21:06 +0000 (UTC) Received: from NAM12-BN8-obe.outbound.protection.outlook.com (mail-bn8nam12on2074.outbound.protection.outlook.com [40.107.237.74]) by gabe.freedesktop.org (Postfix) with ESMTPS id CAF0C10E727 for ; Thu, 1 Sep 2022 14:21:00 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=K5tbApyaECetOsLuBxZYmayjz0xSYDjBbWNtw016PGnobXegOkdpO0ngdN5pf9RqoTviepjQLoyTcF93fHCKMtaamz4plDE7UmaQ19OEOsw7TU+bxP7s/Px/KT/RHJD79L0NEGsbFLb9V1RamEdjj8DgOPYKYpNn8uBsOYiR/MH2TWJMaCHEMTk3n2YWzlvixlQDoBAhi1g4pMcVpc8Kiv/PH2Qaka22y+aXu44vgwt8NskfdT/vWhM7AVRT0SrhF/IMPHry4D89lYT5nTx1sZehybtdACH7PgQLsrsNXbGnzuYcWLLc6T1g83OnVUxjyUzEn0LTTDqj3qw9P/sAag== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=LWT5g/0ubjMdjjud63JCmVADsQoTILLj1AtF/TxuYy0=; b=Pal8oJwtncEhzWW6LqV8XomMRjLQVXAYyB502KymR4O4meUFq3TX22kaN0IA4IP+NvidCI72g3O73a0PQqrN/+TV4bg9JexB1aY1j2NLBNQ8JS4eeRQhDD6Ajb+xpzq6TeQ4cVoyFeqrb+4LEe5CDCeDvOhuyRo9FxXx+G6klUT4ETnZ+GckE3yPYCZwytd6i5XyNR8HvF7LRjuKhWXDBmug2Mr4jqOhNEuoYx1m7w4SUPkTFWxrueXNaatsZji2TNKCRF1qccBJ3Q9xwFRP1mhR3ZeJjs1XC+MAAaxcrmsbJ2215qdv09deQlBOJi1GNMxWisyOCza+7Pb4M206Yw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=LWT5g/0ubjMdjjud63JCmVADsQoTILLj1AtF/TxuYy0=; b=rJRKUx/oVnsFHmhgjWQI2DhI41O1+ogltyDsCpwce0xfPwrW+0UDjNCMgm0Njt3Lrzbl7cF2qVZEU3jCkD2GERt8992Pm1zI56z9JG3IGH+hoZwgS/hyKb0/C1MNtyFX3J5g8xHEaf2C3iJNcqcebBaAMCcmAuRK35Y0GvqqFvYSmQb2N0pzNzcxhCNzocyn7vFIQwNkeqU759bHDOKKKidl/OKlVkX0ykwBnHpqc7Lg/2G15lOLN8TO5JH7m13QxYZL0naa1ktmXYO3vwmgZz8GiCcqtTJooJ1xKaKeViUIoxWwDbRLvQIBtg0UKHe9khGVUkDlmYNWD9xLu1QGxA== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from MN2PR12MB4192.namprd12.prod.outlook.com (2603:10b6:208:1d5::15) by BY5PR12MB5000.namprd12.prod.outlook.com (2603:10b6:a03:1d7::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5566.15; Thu, 1 Sep 2022 14:20:58 +0000 Received: from MN2PR12MB4192.namprd12.prod.outlook.com ([fe80::462:7fe:f04f:d0d5]) by MN2PR12MB4192.namprd12.prod.outlook.com ([fe80::462:7fe:f04f:d0d5%7]) with mapi id 15.20.5588.012; Thu, 1 Sep 2022 14:20:58 +0000 From: Jason Gunthorpe To: =?utf-8?q?Christian_K=C3=B6nig?= , dri-devel@lists.freedesktop.org, Leon Romanovsky , linaro-mm-sig@lists.linaro.org, linux-media@vger.kernel.org, linux-rdma@vger.kernel.org, netdev@vger.kernel.org, Saeed Mahameed , Sumit Semwal Subject: [PATCH 3/4] RDMA/mlx5: Add support for dmabuf to devx umem Date: Thu, 1 Sep 2022 11:20:55 -0300 Message-Id: <3-v1-bd147097458e+ede-umem_dmabuf_jgg@nvidia.com> In-Reply-To: <0-v1-bd147097458e+ede-umem_dmabuf_jgg@nvidia.com> References: X-ClientProxiedBy: BLAPR05CA0025.namprd05.prod.outlook.com (2603:10b6:208:335::6) To MN2PR12MB4192.namprd12.prod.outlook.com (2603:10b6:208:1d5::15) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 147bd910-42ed-4492-60c5-08da8c25302a X-MS-TrafficTypeDiagnostic: BY5PR12MB5000:EE_ X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: Z+09MsS3nqOVvrJ8ALEfz/eCUWrTis5gYMj7e5GPU1wFD2qqqnJwFrN5Pwh7pSPTCT4YZoE2+Fo8k+QhPhOAw3bBImpek63/aFc09DlRxW6r0H5CJimzhh7bzfaMIqjgshGPh/bzgu9KWXkmhktsKqbk7YqeCjJJsnP0s7cKFXCkKOSoWBIGsHUkKx9YP64eNwBWo9w8iIhcbBoh0LwBnNHMwGYvIo4cdmC45Qmb7RWN+3Bhthr3mGVZ9x5OuJiNXvUfykehfqmKU3p0foRBLbg8tWuB40ssEEwFs9jOyzij9Y3gr4d0o6MCLRxI8xiH7BfuXe/A7W/XW0JfJaeQE6w2sgHu0ex6uc9xs94I1Y2qpTos0IgYJNYd7ovHXRz8plpG7Ru91kkeYLhAbIu/ArPHadHU27mUxm66+7+EkOiwMICS1sXkwOZcR0Hx7dgc8MuqE7XgaQKlh8oIDeDw5AAfWjG3j0aJtIXcm8C9BGbiAkL0puTYRbHCYkgdkjOoqGBboF8aPQrDbFhQguBkqHEWXzZH5VWZ1z1GPbUS1uvMTQxtN7ukS9lHaRHSE0VWK4xbvt36HtE9gZT2oUFkBhntkgcvlTGt5K2vAFXaHTbdVeVaprkzILSlalUSc8OI9BSTrRzHj5Vc6yvNf3VpU2RSN8TnI+vV/O85NGS+ezPWog0iYfMvCOpN4V4tm9fs0OWFp/C+VtW1N6d6lxl3osXejhHsk5UWn/ZCe5ZQ0DsjhTJMX4kMtBMtYtct3vCu X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:MN2PR12MB4192.namprd12.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230016)(4636009)(39860400002)(376002)(346002)(136003)(366004)(396003)(6512007)(66946007)(4326008)(5660300002)(66476007)(6666004)(6506007)(478600001)(107886003)(8676002)(66556008)(6486002)(8936002)(86362001)(41300700001)(83380400001)(2906002)(2616005)(26005)(38100700002)(186003)(316002)(36756003)(110136005)(54906003)(4216001); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: x6dXc0wVw7ppooyxGAXnR90j/mLxcjgTVjGXGbcdGGtFC+IO+z2gucv3/kwP4WBt3RQZXOzeNBpdIxg6wY0KWQHt9gVkDVhB1EQMGnFuZbVXNdB1lWEmEv74v/pN8qSPn/Cd4q2Ko60gTeaSTVgLVTx/Kabx/t3Xv7GgpRGhy1mG2rkUQLkN1RpHTHQNeiPREKsbeUGDkDMkLwR1nUWx+wsCvm8r35yFldNeQD8FOEm5Lmi6D2jwUCrcHiITwKcnqyEeIE5MgmpcO79ZSJDpIsPCzWleNdMF8fLOot4+xzxmK2fEZb4/f/98hLA+v/gQgHJd49ATIrVkch3tq05BnFVi3JoKAWG1a0Hak60J8ZewKgXPvGutRn0EFV7PVxCAeZWxHKCMhg7bIJpElHtdF5krPsB9G1XSq/grnOXCzM2xtr019N1HkGfmP0Cc1pkLka0duF9PzyFX6XxO0umiG4Lw9/708JrDN2OoQcpDgZOege+CyYTnWsPEnANsgQZRuTNuItcAkMjJdFEi+aO/BY0SBY0JA+mHdmPHnJCUkBf0dG5BUGCshCrsQUtl5XPH5u7PQuAuGT9T3Hbsw+DJBpxFi97iwynXkdrvA/Ao7HpMy3NXxDi0z3ocY9PqMTNjRr0g8f1SpuUwaZgPh8wLCtDqkXYRdwPqvBaNm5hq9/1B6EciJj91yTUENGIyLIl1gVOlMEpgAlaCvHvc2vu0Deg/qopjsMpRsmdqbqnDGC2cI2M8FT2WHwaPq895SdkhyBtvDn7WgpmcBuTAbX4TYsulUf6dXxzv5TWBxpYO4EWdRFWGglbuntxjFKx5CU+sU+ABmgv+I0Al8/7uogw9LeqFxEZ4qZC0TX2kA8KhcEKw9knrjxpVUxAo9m3pJtXcybHERaAWmLTPS82W5Y91lZyhc8UPuElV/Y4RxhB3V4dcmBVQMeCdti6lwcIRL0NhlbIlYu6yEevQYlqy9OLu91j55MpDJIEadc/zLT0rv6KnMo5yHgh3mt55rc/SikOF1puF8h8W6G3w32rBqdzQ5UUiokndEqFL/IrT0QRlCBkleEi+7DxMFmvQ3l6z4g/DWTPop29Cw6nGdQnIpTqhtEQ8MApCqqlls+fJ6YbFpkNamAu1j3tzeoIh9Jou6fXIEorR4L80jdWlPCZ2WcG9hd8/Co5IEuUchWDRTkRYbAMgto1VSPlvk+fD4Ms2mpB3KBQ3FNX1O6x8v5xBGaMNOYbInoZpcW6yzmsDOTxVBL5PVLNDa91Z8RNSAcquEsGvPIvejsMIJSTlGFqhddrRAcNqhYZCYLVGjs5BrRKcy8BmbbR545J4aMuYQIiB5RrhVQK/1ioyOeLLm8QwLeuasLZE46wrB2VuQSw3t+9pvkkKfgJ5sBsxiBvmjMsirGqCtyLoSucXuA6MSqtI3c/mxt2s1TDNHt4QfvGB+a+Ag7g6apsothD7PZGkctKYNV6nnq5XQBlLo+iMHUakASomSbQB257S0QBQkUsAiJApsZkxdY/PMCIwGL8KIJ0vGsSa8mVE7H2QhVa5bfwhRDhJSUja/yPF/tSUh+5ciYaeB0Ri1fxhhp/z9PJOLM8Vmf7/ X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 147bd910-42ed-4492-60c5-08da8c25302a X-MS-Exchange-CrossTenant-AuthSource: MN2PR12MB4192.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 01 Sep 2022 14:20:57.7046 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: MIZlG+iQqfBzkvtWFOLpWw6rQThS4EMk4hdwKJg7EO6Rp0EuZQbZjsyMaj6RmdLR X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY5PR12MB5000 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Mohammad Kabat , Kamal Heib Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" This is modeled after the similar EFA enablement in commit 66f4817b5712 ("RDMA/efa: Add support for dmabuf memory regions"). Like EFA there is no support for revocation so we simply call the ib_umem_dmabuf_get_pinned() to obtain a umem instead of the normal ib_umem_get(). Everything else stays the same. Signed-off-by: Jason Gunthorpe --- drivers/infiniband/hw/mlx5/devx.c | 24 +++++++++++++++++++++--- include/uapi/rdma/mlx5_user_ioctl_cmds.h | 1 + 2 files changed, 22 insertions(+), 3 deletions(-) diff --git a/drivers/infiniband/hw/mlx5/devx.c b/drivers/infiniband/hw/mlx5/devx.c index 2a2a9e9afc9dad..291e73d7928276 100644 --- a/drivers/infiniband/hw/mlx5/devx.c +++ b/drivers/infiniband/hw/mlx5/devx.c @@ -2181,9 +2181,25 @@ static int devx_umem_get(struct mlx5_ib_dev *dev, struct ib_ucontext *ucontext, if (err) return err; - obj->umem = ib_umem_get(&dev->ib_dev, addr, size, access); - if (IS_ERR(obj->umem)) - return PTR_ERR(obj->umem); + if (uverbs_attr_is_valid(attrs, MLX5_IB_ATTR_DEVX_UMEM_REG_DMABUF_FD)) { + struct ib_umem_dmabuf *umem_dmabuf; + int dmabuf_fd; + + err = uverbs_get_raw_fd(&dmabuf_fd, attrs, + MLX5_IB_ATTR_DEVX_UMEM_REG_DMABUF_FD); + if (err) + return -EFAULT; + + umem_dmabuf = ib_umem_dmabuf_get_pinned( + &dev->ib_dev, addr, size, dmabuf_fd, access); + if (IS_ERR(umem_dmabuf)) + return PTR_ERR(umem_dmabuf); + obj->umem = &umem_dmabuf->umem; + } else { + obj->umem = ib_umem_get(&dev->ib_dev, addr, size, access); + if (IS_ERR(obj->umem)) + return PTR_ERR(obj->umem); + } return 0; } @@ -2833,6 +2849,8 @@ DECLARE_UVERBS_NAMED_METHOD( UVERBS_ATTR_PTR_IN(MLX5_IB_ATTR_DEVX_UMEM_REG_LEN, UVERBS_ATTR_TYPE(u64), UA_MANDATORY), + UVERBS_ATTR_RAW_FD(MLX5_IB_ATTR_DEVX_UMEM_REG_DMABUF_FD, + UA_OPTIONAL), UVERBS_ATTR_FLAGS_IN(MLX5_IB_ATTR_DEVX_UMEM_REG_ACCESS, enum ib_access_flags), UVERBS_ATTR_CONST_IN(MLX5_IB_ATTR_DEVX_UMEM_REG_PGSZ_BITMAP, diff --git a/include/uapi/rdma/mlx5_user_ioctl_cmds.h b/include/uapi/rdma/mlx5_user_ioctl_cmds.h index 3bee490eb5857f..595edad03dfe54 100644 --- a/include/uapi/rdma/mlx5_user_ioctl_cmds.h +++ b/include/uapi/rdma/mlx5_user_ioctl_cmds.h @@ -174,6 +174,7 @@ enum mlx5_ib_devx_umem_reg_attrs { MLX5_IB_ATTR_DEVX_UMEM_REG_ACCESS, MLX5_IB_ATTR_DEVX_UMEM_REG_OUT_ID, MLX5_IB_ATTR_DEVX_UMEM_REG_PGSZ_BITMAP, + MLX5_IB_ATTR_DEVX_UMEM_REG_DMABUF_FD, }; enum mlx5_ib_devx_umem_dereg_attrs { From patchwork Thu Sep 1 14:20:56 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 12962760 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 91852ECAAD3 for ; Thu, 1 Sep 2022 14:21:22 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 07F8C10E72A; Thu, 1 Sep 2022 14:21:07 +0000 (UTC) Received: from NAM11-BN8-obe.outbound.protection.outlook.com (mail-bn8nam11on2040.outbound.protection.outlook.com [40.107.236.40]) by gabe.freedesktop.org (Postfix) with ESMTPS id 7EE9810E725 for ; Thu, 1 Sep 2022 14:21:00 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=gzzzBDcPNMfi5VGvJeusidb7bepykYArYAXIPI233EaL4RbFTFHgOYLyV3y4x0ILY8fzB7HnzlTWvc1LQhQGzBfzesQEEhiFUz1re97lLW1yA1mQRtkWAhi5qJGi7dxREUMD/3rBEbkcdCzr9v7TQNtQoFRM2NRkJVJczYm8E2aafxmOpBdJnDjK9Qi8UqLsQR2Kt7q2c1UjzB1Msfo0N+Q8HkrcCHKCLE3RH9Uhp3o1GULsDjoo17nrHmWQeO75cjYIElv2+Mi5tAn4rsHgXG5c0aqUDbGuV/AWUf2vN0X3zNiq+2F4PPjRIndc75AZUbTePnDhPosZMm8QwZSh2A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=6Ldn7BcH4Fc4o6NLHshrDaZYqiE1GryuyqNfaP2v+tQ=; b=Fb61O1+MeLrBamNwC7tK4trcrNtAIFfSM5FUWjOcq1j9MmBPJ7XDlzrvZRqyV++IMndod0cnvWpJOHaOSJCm9JPtZw1ZNPw2Mn1N9X15Tj4PV2aKOopG9cxOtR7uescXpb5vC5FhFaCqHFrGgQ061qSTbLm2qTF/0Pb1xs+J4LJoinDlmiAd8UsYHscHxgXZ1n1tpZJhFZD5tmaCfU4qYfkToTsJjGqR5o30WgO9vWV0QxESG+ZaOm44yUo1ueytJVxg5HirscsWF/7w3LsZh0/XQ2DSxMerecU10qQ8genLaKlDAlIPrA103WOhQJLwqeEJ1TJDcxxmwR3u8Q3hfg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=6Ldn7BcH4Fc4o6NLHshrDaZYqiE1GryuyqNfaP2v+tQ=; b=pxLcvQFJlhu0V+OBYgLCE7vja8H2kmMRdVoeupKpK9l/jbYZ8Nz9phvw5nwmZE6vW4ZVW0T61OaJagGO7HPQ/Z0ngCz/7MTOep7Jl/reomsr2vrkWkUyZco32zn9fhXU8w8T+pE0y6YCTqEOkboq8z+0gp8ykBBTeerk+KyWYcLNLgP+R7P+lyGBLXbC1hCJ5cApVsFmjqbKbnx2gqIIafDowx5iMsE0l2crlpchqVY3hSDfpHlaaSdbbFUne5L1rxPc0qAfHBXAAa2NuK63WSfaukmJ9KcW8EXM3U5DpesJ9HvvmLffaW0U79PgvpaFJVyDvoDRAPqWFRdA1QrQuA== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from MN2PR12MB4192.namprd12.prod.outlook.com (2603:10b6:208:1d5::15) by IA1PR12MB6041.namprd12.prod.outlook.com (2603:10b6:208:3d7::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5566.21; Thu, 1 Sep 2022 14:20:58 +0000 Received: from MN2PR12MB4192.namprd12.prod.outlook.com ([fe80::462:7fe:f04f:d0d5]) by MN2PR12MB4192.namprd12.prod.outlook.com ([fe80::462:7fe:f04f:d0d5%7]) with mapi id 15.20.5588.012; Thu, 1 Sep 2022 14:20:58 +0000 From: Jason Gunthorpe To: =?utf-8?q?Christian_K=C3=B6nig?= , dri-devel@lists.freedesktop.org, Leon Romanovsky , linaro-mm-sig@lists.linaro.org, linux-media@vger.kernel.org, linux-rdma@vger.kernel.org, netdev@vger.kernel.org, Saeed Mahameed , Sumit Semwal Subject: [PATCH 4/4] RDMA/mlx5: Enable ATS support for MRs and umems Date: Thu, 1 Sep 2022 11:20:56 -0300 Message-Id: <4-v1-bd147097458e+ede-umem_dmabuf_jgg@nvidia.com> In-Reply-To: <0-v1-bd147097458e+ede-umem_dmabuf_jgg@nvidia.com> References: X-ClientProxiedBy: MN2PR15CA0002.namprd15.prod.outlook.com (2603:10b6:208:1b4::15) To MN2PR12MB4192.namprd12.prod.outlook.com (2603:10b6:208:1d5::15) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 98e81480-a7f3-4b36-c5d3-08da8c25300d X-MS-TrafficTypeDiagnostic: IA1PR12MB6041:EE_ X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 2XjB1Vb6mgSsjXuPyIquXbqc2YtrnGtu1V/byME1PLJM599L1edw8l0T6TLuzlIPTvpuQ5+BVz8lq5GdqkF3dT0rZMhQicgg3cNcqez3SvfRF9v5KoG+dW7BVwIBog+PYMo1+o83Fhbr7TMEQfscniFurhEOGgpfEwCHTxTERGAiNOEv8uTSZo5zIB5o3ozQTSnC9uBx4xIEqsn7l/3xGrUBwxcdzmPkE0WnkoD1xgKV5UVGzf9Q6t4B7qCDlldp7dlsKVKkSDRghuXFsUzOMNl0Pqz7TnMv+VFU7SlwMoLBprdVMLGHTz7p141exvJtYeo39Z6gMKBDHvAYqqop/pEmz+92mIXVL7+qiTkWOnZLD4GKT8lpeFEg6JCRjW3dW7yyM9YRG2Nv8CEkQ8i3faEUGRjlpTWvSZ9PQVhIAwYsBzGEodc8Q43miHbpX6wK3H0RPrXzSWLYDf3p3XLjTSUoOFKO0RfgVd8/6dhnYp46iK8cvdtK7eET7kP+gFlh8asRutew+edCSkLm+1wtQrIZgFl3aaZWrQJTYSfx501Pd+XQO6nFlpRVSOdp3sMLzcjp26f8HT+cQ+dzx45OdsmmseXt8WLTmj/17fte80mg3oTgTS5cK4fLZtDEAnWxqOhsAiiZZqtEIEdZYF8vUYYvLYJ7HV5HoHASHzmdvBW3Z4YMoE36Y1AsieVr2UaYg8EIgVZG5kwjyZWt+RSPky727HBqC3gNuxqfaCfD+IcWWY8LlxXp2ThxA6Z0vVDG X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:MN2PR12MB4192.namprd12.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230016)(4636009)(346002)(136003)(396003)(39860400002)(366004)(376002)(6512007)(2616005)(107886003)(26005)(2906002)(38100700002)(6506007)(83380400001)(186003)(4326008)(66556008)(110136005)(8676002)(6486002)(66476007)(54906003)(66946007)(8936002)(316002)(41300700001)(5660300002)(36756003)(86362001)(478600001)(4216001); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: X72JPHBzAwSalejS6P0B+j4HymKeRtfD+mfUxzwebzUGQ+MpxS3eh4Lt5ZkNDilqxdyY2erBGt7wD0rZ/KzFvjezT72ClF5fRftyJopgRP9InoNCWnieIRJjy9gFKeDldsmYoqyotpUYgLgMcy3G3iiPToYkObiWbwa2SVYNg94PO9hmuQMYruLMc2R9aQuVrkkiTgkssBQJ7ehRQVRJks09f/EZ8JCtGl1TPM2Vb6VuUl02xYFwKngJfPn6sNV5BLtiMMXvzQB6eTdKZAC9I6fp4rhpOyRv29QNFYWsKKDZrUiiC+b0v4MGN3dXkx7SZxjmjfCKN37eLdNpQXvbGrhfTP5V0UVhwr+TBz/j0oXaoTujCvf9eCSdt+EHbt1dlKEYy7UgwovcEZgJZ1g/YZUKzA26Ho3kvdh/FKpnI6JixJpQuL5RqtSL00/6DjJgrm52Rm5xqR/t98OVaOo5VbHphmoqKtMfGsG7swpQ8h/FZCHs2xd835/hfDMDkXGraQNDT1Rl65PZobc7F588IEgrN/A17/xWfzJNKyDzGcIYE6AgjKBuvhv4QvH6lGiBXQcSybFK3kW+77FylnGjgn4MKm3AV206A4/AHBgvX/mqaCTW147eHNVYfs3J64gEKMEnPWxB2ijss4Jlhy6WN7DorshdTrzQ/BvkB3TtlbhRX3qjPDL5qPcgBqSTIWAQpA07K3TBaNWczXf4K+Xkt0D6hm5csbeY2Id1eUEwzGowes+8+nLTlKsPOrVK0BkSXbeGFskaDzXRJyiXzZjrkn0TfV3/X489ELdSx+8REj0VE1DHr7f0cKVfzwQRon8RUurPKwaMhA9i6b1uhzVTQTGKwGjRm0bQz1pwQMrFJvEzWJ5vyw3Y7HBGkSQtYGp2crL8yEU2R5VejcEOX/gHY/Y51PHhPWU3u2OMhClWFa3oPg7qzhqrAY6N6LmpciCqd9w594wIq2R5aebolEuaOldsT9AmC9QOMEOaoxCZBsj3B9sBUrIBKJICJWropJncjDY4Y0jIcSy5gVgoPkR8EU3eHW3LtJ/kZojNGn1u78Gq7OENtsFJawztoj6XVNI49jmjiMlZsS6Lmv0quemirZ++mY6k5CPquEYqFievv4gfaLAgUyY6TXj2kDmZs/HNAgkOBnHIdnujyhaan3UFDcJ6/2iq5/IAQTkyXBCHXFG1gKroB+I96vjVYg+2IkGxllpHIQ+3gc7s+/qkmwVIbFoKisE4291cHBqRALI4PBfXvl7+3xvYoRHIOLR4Pzx7ntiEJJikPT0IBEC92/jIbLglIrAxcqB87AZdgATBbPq+ncFLDDJjc2rn5hylcP3HdTY4Q0VXoJd1R8DrzwcWxoxypKJbzgiJrsVsfpV8RjbtlVF1jqMy/BnRl5g9NzfwFsC45sYZMwRPa9eOZ20JHimQi+lfU80cWe9ROhYXzLhAftZd2SO+drRiC8iwyTjCyfVZEGNhfViouxk3tlmRGEW3lTzoa+JDdm76hLRbijA8xxkyUv0NUzaRiLbwD4/aDmDteJc5gE7ueINhkw+FJAe3yPoz6fjepfpi67UyZned51mGqvpgnwyJ7u+Khd5m X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 98e81480-a7f3-4b36-c5d3-08da8c25300d X-MS-Exchange-CrossTenant-AuthSource: MN2PR12MB4192.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 01 Sep 2022 14:20:57.5796 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 6H7cld3vBDck1/Y4kymq3qaWLmIZ/6tmtnJC5D74eRUZ9WrbE/gjG08szHhsdzke X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA1PR12MB6041 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Mohammad Kabat , Kamal Heib Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" For mlx5 if ATS is enabled in the PCI config then the device will use ATS requests for only certain DMA operations. This has to be opted in by the SW side based on the mkey or umem settings. ATS slows down the PCI performance, so it should only be set in cases when it is needed. All of these cases revolve around optimizing PCI P2P transfers and avoiding bad cases where the bus just doesn't work. Signed-off-by: Jason Gunthorpe --- drivers/infiniband/hw/mlx5/devx.c | 37 ++++++++++++++++------------ drivers/infiniband/hw/mlx5/mlx5_ib.h | 36 +++++++++++++++++++++++++++ drivers/infiniband/hw/mlx5/mr.c | 5 +++- 3 files changed, 61 insertions(+), 17 deletions(-) diff --git a/drivers/infiniband/hw/mlx5/devx.c b/drivers/infiniband/hw/mlx5/devx.c index 291e73d7928276..c900977e6ccdb7 100644 --- a/drivers/infiniband/hw/mlx5/devx.c +++ b/drivers/infiniband/hw/mlx5/devx.c @@ -2158,26 +2158,17 @@ static int UVERBS_HANDLER(MLX5_IB_METHOD_DEVX_SUBSCRIBE_EVENT)( static int devx_umem_get(struct mlx5_ib_dev *dev, struct ib_ucontext *ucontext, struct uverbs_attr_bundle *attrs, - struct devx_umem *obj) + struct devx_umem *obj, u32 access_flags) { u64 addr; size_t size; - u32 access; int err; if (uverbs_copy_from(&addr, attrs, MLX5_IB_ATTR_DEVX_UMEM_REG_ADDR) || uverbs_copy_from(&size, attrs, MLX5_IB_ATTR_DEVX_UMEM_REG_LEN)) return -EFAULT; - err = uverbs_get_flags32(&access, attrs, - MLX5_IB_ATTR_DEVX_UMEM_REG_ACCESS, - IB_ACCESS_LOCAL_WRITE | - IB_ACCESS_REMOTE_WRITE | - IB_ACCESS_REMOTE_READ); - if (err) - return err; - - err = ib_check_mr_access(&dev->ib_dev, access); + err = ib_check_mr_access(&dev->ib_dev, access_flags); if (err) return err; @@ -2191,12 +2182,12 @@ static int devx_umem_get(struct mlx5_ib_dev *dev, struct ib_ucontext *ucontext, return -EFAULT; umem_dmabuf = ib_umem_dmabuf_get_pinned( - &dev->ib_dev, addr, size, dmabuf_fd, access); + &dev->ib_dev, addr, size, dmabuf_fd, access_flags); if (IS_ERR(umem_dmabuf)) return PTR_ERR(umem_dmabuf); obj->umem = &umem_dmabuf->umem; } else { - obj->umem = ib_umem_get(&dev->ib_dev, addr, size, access); + obj->umem = ib_umem_get(&dev->ib_dev, addr, size, access_flags); if (IS_ERR(obj->umem)) return PTR_ERR(obj->umem); } @@ -2238,7 +2229,8 @@ static unsigned int devx_umem_find_best_pgsize(struct ib_umem *umem, static int devx_umem_reg_cmd_alloc(struct mlx5_ib_dev *dev, struct uverbs_attr_bundle *attrs, struct devx_umem *obj, - struct devx_umem_reg_cmd *cmd) + struct devx_umem_reg_cmd *cmd, + int access) { unsigned long pgsz_bitmap; unsigned int page_size; @@ -2287,6 +2279,9 @@ static int devx_umem_reg_cmd_alloc(struct mlx5_ib_dev *dev, MLX5_SET(umem, umem, page_offset, ib_umem_dma_offset(obj->umem, page_size)); + if (mlx5_umem_needs_ats(dev, obj->umem, access)) + MLX5_SET(umem, umem, ats, 1); + mlx5_ib_populate_pas(obj->umem, page_size, mtt, (obj->umem->writable ? MLX5_IB_MTT_WRITE : 0) | MLX5_IB_MTT_READ); @@ -2304,20 +2299,30 @@ static int UVERBS_HANDLER(MLX5_IB_METHOD_DEVX_UMEM_REG)( struct mlx5_ib_ucontext *c = rdma_udata_to_drv_context( &attrs->driver_udata, struct mlx5_ib_ucontext, ibucontext); struct mlx5_ib_dev *dev = to_mdev(c->ibucontext.device); + int access_flags; int err; if (!c->devx_uid) return -EINVAL; + err = uverbs_get_flags32(&access_flags, attrs, + MLX5_IB_ATTR_DEVX_UMEM_REG_ACCESS, + IB_ACCESS_LOCAL_WRITE | + IB_ACCESS_REMOTE_WRITE | + IB_ACCESS_REMOTE_READ | + IB_ACCESS_RELAXED_ORDERING); + if (err) + return err; + obj = kzalloc(sizeof(struct devx_umem), GFP_KERNEL); if (!obj) return -ENOMEM; - err = devx_umem_get(dev, &c->ibucontext, attrs, obj); + err = devx_umem_get(dev, &c->ibucontext, attrs, obj, access_flags); if (err) goto err_obj_free; - err = devx_umem_reg_cmd_alloc(dev, attrs, obj, &cmd); + err = devx_umem_reg_cmd_alloc(dev, attrs, obj, &cmd, access_flags); if (err) goto err_umem_release; diff --git a/drivers/infiniband/hw/mlx5/mlx5_ib.h b/drivers/infiniband/hw/mlx5/mlx5_ib.h index 2e2ad391838583..7e2c4a3782209d 100644 --- a/drivers/infiniband/hw/mlx5/mlx5_ib.h +++ b/drivers/infiniband/hw/mlx5/mlx5_ib.h @@ -1550,4 +1550,40 @@ static inline bool rt_supported(int ts_cap) return ts_cap == MLX5_TIMESTAMP_FORMAT_CAP_REAL_TIME || ts_cap == MLX5_TIMESTAMP_FORMAT_CAP_FREE_RUNNING_AND_REAL_TIME; } + +/* + * PCI Peer to Peer is a trainwreck. If no switch is present then things + * sometimes work, depending on the pci_distance_p2p logic for excluding broken + * root complexes. However if a switch is present in the path, then things get + * really ugly depending on how the switch is setup. This table assumes that the + * root complex is strict and is validating that all req/reps are matches + * perfectly - so any scenario where it sees only half the transaction is a + * failure. + * + * CR/RR/DT ATS RO P2P + * 00X X X OK + * 010 X X fails (request is routed to root but root never sees comp) + * 011 0 X fails (request is routed to root but root never sees comp) + * 011 1 X OK + * 10X X 1 OK + * 101 X 0 fails (completion is routed to root but root didn't see req) + * 110 X 0 SLOW + * 111 0 0 SLOW + * 111 1 0 fails (completion is routed to root but root didn't see req) + * 111 1 1 OK + * + * Unfortunately we cannot reliably know if a switch is present or what the + * CR/RR/DT ACS settings are, as in a VM that is all hidden. Assume that + * CR/RR/DT is 111 if the ATS cap is enabled and follow the last three rows. + * + * For now assume if the umem is a dma_buf then it is P2P. + */ +static inline bool mlx5_umem_needs_ats(struct mlx5_ib_dev *dev, + struct ib_umem *umem, int access_flags) +{ + if (!MLX5_CAP_GEN(dev->mdev, ats) || !umem->is_dmabuf) + return false; + return access_flags & IB_ACCESS_RELAXED_ORDERING; +} + #endif /* MLX5_IB_H */ diff --git a/drivers/infiniband/hw/mlx5/mr.c b/drivers/infiniband/hw/mlx5/mr.c index 129d531bd01bc8..7fd3adea370290 100644 --- a/drivers/infiniband/hw/mlx5/mr.c +++ b/drivers/infiniband/hw/mlx5/mr.c @@ -937,7 +937,8 @@ static struct mlx5_ib_mr *alloc_cacheable_mr(struct ib_pd *pd, * cache then synchronously create an uncached one. */ if (!ent || ent->limit == 0 || - !mlx5r_umr_can_reconfig(dev, 0, access_flags)) { + !mlx5r_umr_can_reconfig(dev, 0, access_flags) || + mlx5_umem_needs_ats(dev, umem, access_flags)) { mutex_lock(&dev->slow_path_mutex); mr = reg_create(pd, umem, iova, access_flags, page_size, false); mutex_unlock(&dev->slow_path_mutex); @@ -1018,6 +1019,8 @@ static struct mlx5_ib_mr *reg_create(struct ib_pd *pd, struct ib_umem *umem, MLX5_SET(mkc, mkc, translations_octword_size, get_octo_len(iova, umem->length, mr->page_shift)); MLX5_SET(mkc, mkc, log_page_size, mr->page_shift); + if (mlx5_umem_needs_ats(dev, umem, access_flags)) + MLX5_SET(mkc, mkc, ma_translation_mode, 1); if (populate) { MLX5_SET(create_mkey_in, in, translations_octword_actual_size, get_octo_len(iova, umem->length, mr->page_shift));