From patchwork Mon Nov 23 17:53:00 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Xiong, Jianxin" X-Patchwork-Id: 11925793 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A1223C63777 for ; Mon, 23 Nov 2020 17:39:13 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 56C7720756 for ; Mon, 23 Nov 2020 17:39:13 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 56C7720756 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=intel.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=dri-devel-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 1EC8B6E071; Mon, 23 Nov 2020 17:39:12 +0000 (UTC) Received: from mga06.intel.com (mga06.intel.com [134.134.136.31]) by gabe.freedesktop.org (Postfix) with ESMTPS id 8B2B86E081 for ; Mon, 23 Nov 2020 17:39:10 +0000 (UTC) IronPort-SDR: WK2yyhMiz0R8r+uhb2pVbhrNJtZp7Vlb0eOYGyv6PlQD4+lKyAFel68y8Aus6vM68mpSZ/4NtC 8bvXuIb+NPjQ== X-IronPort-AV: E=McAfee;i="6000,8403,9814"; a="233421234" X-IronPort-AV: E=Sophos;i="5.78,364,1599548400"; d="scan'208";a="233421234" X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga004.fm.intel.com ([10.253.24.48]) by orsmga104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 23 Nov 2020 09:39:09 -0800 IronPort-SDR: 6S5X1On1jW+bAw0bHRCSyY61wr5vW9+lWQX/wPDHYNLyX7DWdT/yLVetqFZPVSLIJArPJY1vw/ 7p9uPa58HCuw== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.78,364,1599548400"; d="scan'208";a="361525789" Received: from cst-dev.jf.intel.com ([10.23.221.69]) by fmsmga004.fm.intel.com with ESMTP; 23 Nov 2020 09:39:09 -0800 From: Jianxin Xiong To: linux-rdma@vger.kernel.org, dri-devel@lists.freedesktop.org Subject: [PATCH rdma-core 1/5] verbs: Support dma-buf based memory region Date: Mon, 23 Nov 2020 09:53:00 -0800 Message-Id: <1606153984-104583-2-git-send-email-jianxin.xiong@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1606153984-104583-1-git-send-email-jianxin.xiong@intel.com> References: <1606153984-104583-1-git-send-email-jianxin.xiong@intel.com> X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Leon Romanovsky , Jason Gunthorpe , Doug Ledford , Daniel Vetter , Christian Koenig , Jianxin Xiong MIME-Version: 1.0 Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" Add new API function and new provider method for registering dma-buf based memory region. Update the man page and bump the API version. Signed-off-by: Jianxin Xiong --- kernel-headers/rdma/ib_user_ioctl_cmds.h | 14 ++++++++++++ libibverbs/cmd_mr.c | 38 ++++++++++++++++++++++++++++++++ libibverbs/driver.h | 7 ++++++ libibverbs/dummy_ops.c | 11 +++++++++ libibverbs/libibverbs.map.in | 6 +++++ libibverbs/man/ibv_reg_mr.3 | 21 ++++++++++++++++-- libibverbs/verbs.c | 19 ++++++++++++++++ libibverbs/verbs.h | 10 +++++++++ 8 files changed, 124 insertions(+), 2 deletions(-) diff --git a/kernel-headers/rdma/ib_user_ioctl_cmds.h b/kernel-headers/rdma/ib_user_ioctl_cmds.h index 7968a18..dafc7eb 100644 --- a/kernel-headers/rdma/ib_user_ioctl_cmds.h +++ b/kernel-headers/rdma/ib_user_ioctl_cmds.h @@ -1,5 +1,6 @@ /* * Copyright (c) 2018, Mellanox Technologies inc. All rights reserved. + * Copyright (c) 2020, Intel Corporation. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU @@ -251,6 +252,7 @@ enum uverbs_methods_mr { UVERBS_METHOD_MR_DESTROY, UVERBS_METHOD_ADVISE_MR, UVERBS_METHOD_QUERY_MR, + UVERBS_METHOD_REG_DMABUF_MR, }; enum uverbs_attrs_mr_destroy_ids { @@ -272,6 +274,18 @@ enum uverbs_attrs_query_mr_cmd_attr_ids { UVERBS_ATTR_QUERY_MR_RESP_IOVA, }; +enum uverbs_attrs_reg_dmabuf_mr_cmd_attr_ids { + UVERBS_ATTR_REG_DMABUF_MR_HANDLE, + UVERBS_ATTR_REG_DMABUF_MR_PD_HANDLE, + UVERBS_ATTR_REG_DMABUF_MR_OFFSET, + UVERBS_ATTR_REG_DMABUF_MR_LENGTH, + UVERBS_ATTR_REG_DMABUF_MR_IOVA, + UVERBS_ATTR_REG_DMABUF_MR_FD, + UVERBS_ATTR_REG_DMABUF_MR_ACCESS_FLAGS, + UVERBS_ATTR_REG_DMABUF_MR_RESP_LKEY, + UVERBS_ATTR_REG_DMABUF_MR_RESP_RKEY, +}; + enum uverbs_attrs_create_counters_cmd_attr_ids { UVERBS_ATTR_CREATE_COUNTERS_HANDLE, }; diff --git a/libibverbs/cmd_mr.c b/libibverbs/cmd_mr.c index 42dbe42..91ce2ef 100644 --- a/libibverbs/cmd_mr.c +++ b/libibverbs/cmd_mr.c @@ -1,5 +1,6 @@ /* * Copyright (c) 2018 Mellanox Technologies, Ltd. All rights reserved. + * Copyright (c) 2020 Intel Corporation. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU @@ -116,3 +117,40 @@ int ibv_cmd_query_mr(struct ibv_pd *pd, struct verbs_mr *vmr, return 0; } +int ibv_cmd_reg_dmabuf_mr(struct ibv_pd *pd, uint64_t offset, size_t length, + uint64_t iova, int fd, int access, + struct verbs_mr *vmr) +{ + DECLARE_COMMAND_BUFFER(cmdb, UVERBS_OBJECT_MR, + UVERBS_METHOD_REG_DMABUF_MR, + 9); + struct ib_uverbs_attr *handle; + uint32_t lkey, rkey; + int ret; + + handle = fill_attr_out_obj(cmdb, UVERBS_ATTR_REG_DMABUF_MR_HANDLE); + fill_attr_out_ptr(cmdb, UVERBS_ATTR_REG_DMABUF_MR_RESP_LKEY, &lkey); + fill_attr_out_ptr(cmdb, UVERBS_ATTR_REG_DMABUF_MR_RESP_RKEY, &rkey); + + fill_attr_in_obj(cmdb, UVERBS_ATTR_REG_DMABUF_MR_PD_HANDLE, pd->handle); + fill_attr_in_uint64(cmdb, UVERBS_ATTR_REG_DMABUF_MR_OFFSET, offset); + fill_attr_in_uint64(cmdb, UVERBS_ATTR_REG_DMABUF_MR_LENGTH, length); + fill_attr_in_uint64(cmdb, UVERBS_ATTR_REG_DMABUF_MR_IOVA, iova); + fill_attr_in_uint32(cmdb, UVERBS_ATTR_REG_DMABUF_MR_FD, fd); + fill_attr_in_uint32(cmdb, UVERBS_ATTR_REG_DMABUF_MR_ACCESS_FLAGS, access); + + ret = execute_ioctl(pd->context, cmdb); + if (ret) + return errno; + + vmr->ibv_mr.handle = + read_attr_obj(UVERBS_ATTR_REG_DMABUF_MR_HANDLE, handle); + vmr->ibv_mr.context = pd->context; + vmr->ibv_mr.lkey = lkey; + vmr->ibv_mr.rkey = rkey; + vmr->ibv_mr.pd = pd; + vmr->ibv_mr.addr = (void *)offset; + vmr->ibv_mr.length = length; + vmr->mr_type = IBV_MR_TYPE_MR; + return 0; +} diff --git a/libibverbs/driver.h b/libibverbs/driver.h index ab80f4b..d6a9d0a 100644 --- a/libibverbs/driver.h +++ b/libibverbs/driver.h @@ -2,6 +2,7 @@ * Copyright (c) 2004, 2005 Topspin Communications. All rights reserved. * Copyright (c) 2005, 2006 Cisco Systems, Inc. All rights reserved. * Copyright (c) 2005 PathScale, Inc. All rights reserved. + * Copyright (c) 2020 Intel Corporation. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU @@ -373,6 +374,9 @@ struct verbs_context_ops { struct ibv_mr *(*reg_dm_mr)(struct ibv_pd *pd, struct ibv_dm *dm, uint64_t dm_offset, size_t length, unsigned int access); + struct ibv_mr *(*reg_dmabuf_mr)(struct ibv_pd *pd, uint64_t offset, + size_t length, uint64_t iova, + int fd, int access); struct ibv_mr *(*reg_mr)(struct ibv_pd *pd, void *addr, size_t length, uint64_t hca_va, int access); int (*req_notify_cq)(struct ibv_cq *cq, int solicited_only); @@ -498,6 +502,9 @@ int ibv_cmd_advise_mr(struct ibv_pd *pd, uint32_t flags, struct ibv_sge *sg_list, uint32_t num_sge); +int ibv_cmd_reg_dmabuf_mr(struct ibv_pd *pd, uint64_t offset, size_t length, + uint64_t iova, int fd, int access, + struct verbs_mr *vmr); int ibv_cmd_alloc_mw(struct ibv_pd *pd, enum ibv_mw_type type, struct ibv_mw *mw, struct ibv_alloc_mw *cmd, size_t cmd_size, diff --git a/libibverbs/dummy_ops.c b/libibverbs/dummy_ops.c index e5af9e4..dca96d2 100644 --- a/libibverbs/dummy_ops.c +++ b/libibverbs/dummy_ops.c @@ -1,5 +1,6 @@ /* * Copyright (c) 2017 Mellanox Technologies, Inc. All rights reserved. + * Copyright (c) 2020 Intel Corporation. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU @@ -452,6 +453,14 @@ static struct ibv_mr *reg_mr(struct ibv_pd *pd, void *addr, size_t length, return NULL; } +static struct ibv_mr *reg_dmabuf_mr(struct ibv_pd *pd, uint64_t offset, + size_t length, uint64_t iova, + int fd, int access) +{ + errno = ENOSYS; + return NULL; +} + static int req_notify_cq(struct ibv_cq *cq, int solicited_only) { return EOPNOTSUPP; @@ -560,6 +569,7 @@ const struct verbs_context_ops verbs_dummy_ops = { query_srq, read_counters, reg_dm_mr, + reg_dmabuf_mr, reg_mr, req_notify_cq, rereg_mr, @@ -689,6 +699,7 @@ void verbs_set_ops(struct verbs_context *vctx, SET_PRIV_OP_IC(vctx, set_ece); SET_PRIV_OP_IC(vctx, unimport_mr); SET_PRIV_OP_IC(vctx, unimport_pd); + SET_OP(vctx, reg_dmabuf_mr); #undef SET_OP #undef SET_OP2 diff --git a/libibverbs/libibverbs.map.in b/libibverbs/libibverbs.map.in index b5ccaca..f67e1ef 100644 --- a/libibverbs/libibverbs.map.in +++ b/libibverbs/libibverbs.map.in @@ -148,6 +148,11 @@ IBVERBS_1.11 { _ibv_query_gid_table; } IBVERBS_1.10; +IBVERBS_1.12 { + global: + ibv_reg_dmabuf_mr; +} IBVERBS_1.11; + /* If any symbols in this stanza change ABI then the entire staza gets a new symbol version. See the top level CMakeLists.txt for this setting. */ @@ -211,6 +216,7 @@ IBVERBS_PRIVATE_@IBVERBS_PABI_VERSION@ { ibv_cmd_query_srq; ibv_cmd_read_counters; ibv_cmd_reg_dm_mr; + ibv_cmd_reg_dmabuf_mr; ibv_cmd_reg_mr; ibv_cmd_req_notify_cq; ibv_cmd_rereg_mr; diff --git a/libibverbs/man/ibv_reg_mr.3 b/libibverbs/man/ibv_reg_mr.3 index 2bfc955..4975c79 100644 --- a/libibverbs/man/ibv_reg_mr.3 +++ b/libibverbs/man/ibv_reg_mr.3 @@ -3,7 +3,7 @@ .\" .TH IBV_REG_MR 3 2006-10-31 libibverbs "Libibverbs Programmer's Manual" .SH "NAME" -ibv_reg_mr, ibv_reg_mr_iova, ibv_dereg_mr \- register or deregister a memory region (MR) +ibv_reg_mr, ibv_reg_mr_iova, ibv_reg_dmabuf_mr, ibv_dereg_mr \- register or deregister a memory region (MR) .SH "SYNOPSIS" .nf .B #include @@ -15,6 +15,9 @@ ibv_reg_mr, ibv_reg_mr_iova, ibv_dereg_mr \- register or deregister a memory reg .BI " size_t " "length" ", uint64_t " "hca_va" , .BI " int " "access" ); .sp +.BI "struct ibv_mr *ibv_reg_dmabuf_mr(struct ibv_pd " "*pd" ", uint64_t " "offset" , +.BI " size_t " "length" ", int " "access" ); +.sp .BI "int ibv_dereg_mr(struct ibv_mr " "*mr" ); .fi .SH "DESCRIPTION" @@ -71,11 +74,25 @@ a lkey or rkey. The offset in the memory region is computed as 'addr + (iova - hca_va)'. Specifying 0 for hca_va has the same effect as IBV_ACCESS_ZERO_BASED. .PP +.B ibv_reg_dmabuf_mr() +registers a dma-buf based memory region (MR) associated with the protection domain +.I pd\fR. +The MR starts at +.I offset +of the dma-buf and its size is +.I length\fR. +The dma-buf is identified by the file descriptor +.I fd\fR. +The argument +.I access +describes the desired memory protection attributes; it is similar to the ibv_reg_mr case except that only the following flags are supported: +.B IBV_ACCESS_LOCAL_WRITE, IBV_ACCESS_REMOTE_WRITE, IBV_ACCESS_REMOTE_READ, IBV_ACCESS_REMOTE_ATOMIC, IBV_ACCESS_RELAXED_ORDERING. +.PP .B ibv_dereg_mr() deregisters the MR .I mr\fR. .SH "RETURN VALUE" -.B ibv_reg_mr() / ibv_reg_mr_iova() +.B ibv_reg_mr() / ibv_reg_mr_iova() / ibv_reg_dmabuf_mr() returns a pointer to the registered MR, or NULL if the request fails. The local key (\fBL_Key\fR) field .B lkey diff --git a/libibverbs/verbs.c b/libibverbs/verbs.c index 2b0ede8..84ddac7 100644 --- a/libibverbs/verbs.c +++ b/libibverbs/verbs.c @@ -1,6 +1,7 @@ /* * Copyright (c) 2005 Topspin Communications. All rights reserved. * Copyright (c) 2006, 2007 Cisco Systems, Inc. All rights reserved. + * Copyright (c) 2020 Intel Corperation. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU @@ -367,6 +368,24 @@ void ibv_unimport_mr(struct ibv_mr *mr) get_ops(mr->context)->unimport_mr(mr); } +LATEST_SYMVER_FUNC(ibv_reg_dmabuf_mr, 1_12, "IBVERBS_1.12", + struct ibv_mr *, + struct ibv_pd *pd, uint64_t offset, + size_t length, int fd, int access) +{ + struct ibv_mr *mr; + + mr = get_ops(pd->context)->reg_dmabuf_mr(pd, offset, length, offset, + fd, access); + if (mr) { + mr->context = pd->context; + mr->pd = pd; + mr->addr = (void *)offset; + mr->length = length; + } + return mr; +} + LATEST_SYMVER_FUNC(ibv_rereg_mr, 1_1, "IBVERBS_1.1", int, struct ibv_mr *mr, int flags, diff --git a/libibverbs/verbs.h b/libibverbs/verbs.h index ee57e05..3961b1e 100644 --- a/libibverbs/verbs.h +++ b/libibverbs/verbs.h @@ -3,6 +3,7 @@ * Copyright (c) 2004, 2011-2012 Intel Corporation. All rights reserved. * Copyright (c) 2005, 2006, 2007 Cisco Systems, Inc. All rights reserved. * Copyright (c) 2005 PathScale, Inc. All rights reserved. + * Copyright (c) 2020 Intel Corporation. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU @@ -2133,6 +2134,9 @@ struct verbs_context { struct ibv_xrcd * (*open_xrcd)(struct ibv_context *context, struct ibv_xrcd_init_attr *xrcd_init_attr); int (*close_xrcd)(struct ibv_xrcd *xrcd); + struct ibv_mr * (*reg_dmabuf_mr)(struct ibv_pd *pd, uint64_t offset, + size_t length, uint64_t iova, + int fd, int access); uint64_t _ABI_placeholder3; size_t sz; /* Must be immediately before struct ibv_context */ struct ibv_context context; /* Must be last field in the struct */ @@ -2535,6 +2539,12 @@ __ibv_reg_mr_iova(struct ibv_pd *pd, void *addr, size_t length, uint64_t iova, __builtin_constant_p( \ ((access) & IBV_ACCESS_OPTIONAL_RANGE) == 0)) +/** + * ibv_reg_dmabuf_mr - Register a dambuf-based memory region + */ +struct ibv_mr *ibv_reg_dmabuf_mr(struct ibv_pd *pd, uint64_t offset, size_t length, + int fd, int access); + enum ibv_rereg_mr_err_code { /* Old MR is valid, invalid input */ IBV_REREG_MR_ERR_INPUT = -1, From patchwork Mon Nov 23 17:53:01 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Xiong, Jianxin" X-Patchwork-Id: 11925795 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5C8C4C63697 for ; Mon, 23 Nov 2020 17:39:14 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 0D71620756 for ; Mon, 23 Nov 2020 17:39:13 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 0D71620756 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=intel.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=dri-devel-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 795D36E07D; Mon, 23 Nov 2020 17:39:12 +0000 (UTC) Received: from mga06.intel.com (mga06.intel.com [134.134.136.31]) by gabe.freedesktop.org (Postfix) with ESMTPS id BEFF86E07D for ; Mon, 23 Nov 2020 17:39:10 +0000 (UTC) IronPort-SDR: o2dJKoxFB4o8PMoIYlFOQWexJJocKgvs1NjsysiKLBT4e2lZanHdJ4FVDcGf0RepTl9TzsMgBx ICiN2ni5mzaA== X-IronPort-AV: E=McAfee;i="6000,8403,9814"; a="233421239" X-IronPort-AV: E=Sophos;i="5.78,364,1599548400"; d="scan'208";a="233421239" X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga004.fm.intel.com ([10.253.24.48]) by orsmga104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 23 Nov 2020 09:39:09 -0800 IronPort-SDR: TTxBlRjTclW3F7N+bORXXUlHYLyZ9c+Ws7dmJKMST0PUulg+mH2kzcnYh4OgBHys6XoYYZrNng NUMETnpfioCA== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.78,364,1599548400"; d="scan'208";a="361525793" Received: from cst-dev.jf.intel.com ([10.23.221.69]) by fmsmga004.fm.intel.com with ESMTP; 23 Nov 2020 09:39:09 -0800 From: Jianxin Xiong To: linux-rdma@vger.kernel.org, dri-devel@lists.freedesktop.org Subject: [PATCH rdma-core 2/5] mlx5: Support dma-buf based memory region Date: Mon, 23 Nov 2020 09:53:01 -0800 Message-Id: <1606153984-104583-3-git-send-email-jianxin.xiong@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1606153984-104583-1-git-send-email-jianxin.xiong@intel.com> References: <1606153984-104583-1-git-send-email-jianxin.xiong@intel.com> X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Leon Romanovsky , Jason Gunthorpe , Doug Ledford , Daniel Vetter , Christian Koenig , Jianxin Xiong MIME-Version: 1.0 Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" Implement the new provider method for registering dma-buf based memory regions. Signed-off-by: Jianxin Xiong --- providers/mlx5/mlx5.c | 2 ++ providers/mlx5/mlx5.h | 3 +++ providers/mlx5/verbs.c | 23 +++++++++++++++++++++++ 3 files changed, 28 insertions(+) diff --git a/providers/mlx5/mlx5.c b/providers/mlx5/mlx5.c index 1378acf..b3e2d57 100644 --- a/providers/mlx5/mlx5.c +++ b/providers/mlx5/mlx5.c @@ -1,5 +1,6 @@ /* * Copyright (c) 2012 Mellanox Technologies, Inc. All rights reserved. + * Copyright (c) 2020 Intel Corporation. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU @@ -96,6 +97,7 @@ static const struct verbs_context_ops mlx5_ctx_common_ops = { .async_event = mlx5_async_event, .dealloc_pd = mlx5_free_pd, .reg_mr = mlx5_reg_mr, + .reg_dmabuf_mr = mlx5_reg_dmabuf_mr, .rereg_mr = mlx5_rereg_mr, .dereg_mr = mlx5_dereg_mr, .alloc_mw = mlx5_alloc_mw, diff --git a/providers/mlx5/mlx5.h b/providers/mlx5/mlx5.h index 8c94f72..17a2470 100644 --- a/providers/mlx5/mlx5.h +++ b/providers/mlx5/mlx5.h @@ -1,5 +1,6 @@ /* * Copyright (c) 2012 Mellanox Technologies, Inc. All rights reserved. + * Copyright (c) 2020 Intel Corporation. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU @@ -903,6 +904,8 @@ void mlx5_async_event(struct ibv_context *context, struct ibv_mr *mlx5_alloc_null_mr(struct ibv_pd *pd); struct ibv_mr *mlx5_reg_mr(struct ibv_pd *pd, void *addr, size_t length, uint64_t hca_va, int access); +struct ibv_mr *mlx5_reg_dmabuf_mr(struct ibv_pd *pd, uint64_t offset, size_t length, + uint64_t iova, int fd, int access); int mlx5_rereg_mr(struct verbs_mr *mr, int flags, struct ibv_pd *pd, void *addr, size_t length, int access); int mlx5_dereg_mr(struct verbs_mr *mr); diff --git a/providers/mlx5/verbs.c b/providers/mlx5/verbs.c index b956156..6279993 100644 --- a/providers/mlx5/verbs.c +++ b/providers/mlx5/verbs.c @@ -1,5 +1,6 @@ /* * Copyright (c) 2012 Mellanox Technologies, Inc. All rights reserved. + * Copyright (c) 2020 Intel Corporation. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU @@ -647,6 +648,28 @@ struct ibv_mr *mlx5_reg_mr(struct ibv_pd *pd, void *addr, size_t length, return &mr->vmr.ibv_mr; } +struct ibv_mr *mlx5_reg_dmabuf_mr(struct ibv_pd *pd, uint64_t offset, size_t length, + uint64_t iova, int fd, int acc) +{ + struct mlx5_mr *mr; + int ret; + enum ibv_access_flags access = (enum ibv_access_flags)acc; + + mr = calloc(1, sizeof(*mr)); + if (!mr) + return NULL; + + ret = ibv_cmd_reg_dmabuf_mr(pd, offset, length, iova, fd, access, + &mr->vmr); + if (ret) { + free(mr); + return NULL; + } + mr->alloc_flags = acc; + + return &mr->vmr.ibv_mr; +} + struct ibv_mr *mlx5_alloc_null_mr(struct ibv_pd *pd) { struct mlx5_mr *mr; From patchwork Mon Nov 23 17:53:02 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Xiong, Jianxin" X-Patchwork-Id: 11925799 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2E6BCC63697 for ; Mon, 23 Nov 2020 17:39:26 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id CFD522072C for ; Mon, 23 Nov 2020 17:39:25 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org CFD522072C Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=intel.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=dri-devel-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 483AF6E072; Mon, 23 Nov 2020 17:39:25 +0000 (UTC) Received: from mga06.intel.com (mga06.intel.com [134.134.136.31]) by gabe.freedesktop.org (Postfix) with ESMTPS id 1A26F6E07D for ; Mon, 23 Nov 2020 17:39:11 +0000 (UTC) IronPort-SDR: GLH9NOHQuoymTt1SmnK67GOSSUUykkjwKVlhxMGUZpfYUczwaRh/i+TuheijJj5S2lsyWw+Cql DWxfhN7OXccA== X-IronPort-AV: E=McAfee;i="6000,8403,9814"; a="233421242" X-IronPort-AV: E=Sophos;i="5.78,364,1599548400"; d="scan'208";a="233421242" X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga004.fm.intel.com ([10.253.24.48]) by orsmga104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 23 Nov 2020 09:39:10 -0800 IronPort-SDR: 9vils/WBwvnYkMVY6uFUzvlVAqGTQOlBYNR3BgVlnfcFzsZOOHUL5Bx62n9fAzH74NBJKIp/AU kdDImGrG5y9w== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.78,364,1599548400"; d="scan'208";a="361525796" Received: from cst-dev.jf.intel.com ([10.23.221.69]) by fmsmga004.fm.intel.com with ESMTP; 23 Nov 2020 09:39:09 -0800 From: Jianxin Xiong To: linux-rdma@vger.kernel.org, dri-devel@lists.freedesktop.org Subject: [PATCH rdma-core 3/5] pyverbs: Add dma-buf based MR support Date: Mon, 23 Nov 2020 09:53:02 -0800 Message-Id: <1606153984-104583-4-git-send-email-jianxin.xiong@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1606153984-104583-1-git-send-email-jianxin.xiong@intel.com> References: <1606153984-104583-1-git-send-email-jianxin.xiong@intel.com> X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Leon Romanovsky , Jason Gunthorpe , Doug Ledford , Daniel Vetter , Christian Koenig , Jianxin Xiong MIME-Version: 1.0 Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" Define a new sub-class of 'MR' that uses dma-buf object for the memory region. Define a new class 'DmaBuf' for dma-buf object allocation. Signed-off-by: Jianxin Xiong --- pyverbs/CMakeLists.txt | 2 ++ pyverbs/dmabuf.pxd | 13 +++++++++ pyverbs/dmabuf.pyx | 58 +++++++++++++++++++++++++++++++++++++ pyverbs/libibverbs.pxd | 2 ++ pyverbs/mr.pxd | 5 ++++ pyverbs/mr.pyx | 77 ++++++++++++++++++++++++++++++++++++++++++++++++-- 6 files changed, 155 insertions(+), 2 deletions(-) create mode 100644 pyverbs/dmabuf.pxd create mode 100644 pyverbs/dmabuf.pyx diff --git a/pyverbs/CMakeLists.txt b/pyverbs/CMakeLists.txt index 9542c4b..5aee02b 100644 --- a/pyverbs/CMakeLists.txt +++ b/pyverbs/CMakeLists.txt @@ -1,5 +1,6 @@ # SPDX-License-Identifier: (GPL-2.0 OR Linux-OpenIB) # Copyright (c) 2019, Mellanox Technologies. All rights reserved. See COPYING file +# Copyright (c) 2020, Intel Corporation. All rights reserved. rdma_cython_module(pyverbs "" addr.pyx @@ -16,6 +17,7 @@ rdma_cython_module(pyverbs "" wr.pyx xrcd.pyx srq.pyx + dmabuf.pyx ) rdma_python_module(pyverbs diff --git a/pyverbs/dmabuf.pxd b/pyverbs/dmabuf.pxd new file mode 100644 index 0000000..040db4b --- /dev/null +++ b/pyverbs/dmabuf.pxd @@ -0,0 +1,13 @@ +# SPDX-License-Identifier: (GPL-2.0 OR Linux-OpenIB) +# Copyright (c) 2020, Intel Corporation. All rights reserved. + +#cython: language_level=3 + +cdef class DmaBuf: + cdef int dri_fd + cdef int handle + cdef int fd + cdef unsigned long size + cdef unsigned long map_offset + cdef object dmabuf_mrs + cdef add_ref(self, obj) diff --git a/pyverbs/dmabuf.pyx b/pyverbs/dmabuf.pyx new file mode 100644 index 0000000..6c7622d --- /dev/null +++ b/pyverbs/dmabuf.pyx @@ -0,0 +1,58 @@ +# SPDX-License-Identifier: (GPL-2.0 OR Linux-OpenIB) +# Copyright (c) 2020, Intel Corporation. All rights reserved. + +#cython: language_level=3 + +import weakref + +from os import open, close, O_RDWR +from fcntl import ioctl +from struct import pack_into, unpack +from pyverbs.base cimport close_weakrefs +from pyverbs.mr cimport DmaBufMR + +cdef extern from "drm/drm.h": + cdef int DRM_IOCTL_MODE_CREATE_DUMB + cdef int DRM_IOCTL_MODE_MAP_DUMB + cdef int DRM_IOCTL_MODE_DESTROY_DUMB + cdef int DRM_IOCTL_PRIME_HANDLE_TO_FD + +cdef class DmaBuf: + def __init__(self, size, unit=0): + """ + Allocate DmaBuf object from a GPU device. This is done through the + DRI device interface (/dev/dri/card*). Usually this requires the + effective user id being root or being a member of the 'video' group. + :param size: The size (in number of bytes) of the buffer. + :param unit: The unit number of the GPU to allocate the buffer from. + :return: The newly created DmaBuf object on success. + """ + self.dmabuf_mrs = weakref.WeakSet() + self.dri_fd = open('/dev/dri/card'+str(unit), O_RDWR) + + args = bytearray(32) + pack_into('=iiiiiiq', args, 0, 1, size, 8, 0, 0, 0, 0) + ioctl(self.dri_fd, DRM_IOCTL_MODE_CREATE_DUMB, args) + a, b, c, d, self.handle, e, self.size = unpack('=iiiiiiq', args) + + args = bytearray(12) + pack_into('=iii', args, 0, self.handle, O_RDWR, 0) + ioctl(self.dri_fd, DRM_IOCTL_PRIME_HANDLE_TO_FD, args) + a, b, self.fd = unpack('=iii', args) + + args = bytearray(16) + pack_into('=iiq', args, 0, self.handle, 0, 0) + ioctl(self.dri_fd, DRM_IOCTL_MODE_MAP_DUMB, args); + a, b, self.map_offset = unpack('=iiq', args); + + def __dealloc__(self): + close_weakrefs([self.dmabuf_mrs]) + args = bytearray(4) + pack_into('=i', args, 0, self.handle) + ioctl(self.dri_fd, DRM_IOCTL_MODE_DESTROY_DUMB, args) + close(self.dri_fd) + + cdef add_ref(self, obj): + if isinstance(obj, DmaBufMR): + self.dmabuf_mrs.add(obj) + diff --git a/pyverbs/libibverbs.pxd b/pyverbs/libibverbs.pxd index 6fbba54..95e51e1 100644 --- a/pyverbs/libibverbs.pxd +++ b/pyverbs/libibverbs.pxd @@ -507,6 +507,8 @@ cdef extern from 'infiniband/verbs.h': ibv_pd *ibv_alloc_pd(ibv_context *context) int ibv_dealloc_pd(ibv_pd *pd) ibv_mr *ibv_reg_mr(ibv_pd *pd, void *addr, size_t length, int access) + ibv_mr *ibv_reg_dmabuf_mr(ibv_pd *pd, uint64_t offset, size_t length, + int fd, int access) int ibv_dereg_mr(ibv_mr *mr) int ibv_advise_mr(ibv_pd *pd, uint32_t advice, uint32_t flags, ibv_sge *sg_list, uint32_t num_sge) diff --git a/pyverbs/mr.pxd b/pyverbs/mr.pxd index ebe8ada..b89cf02 100644 --- a/pyverbs/mr.pxd +++ b/pyverbs/mr.pxd @@ -1,5 +1,6 @@ # SPDX-License-Identifier: (GPL-2.0 OR Linux-OpenIB) # Copyright (c) 2019, Mellanox Technologies. All rights reserved. See COPYING file +# Copyright (c) 2020, Intel Corporation. All rights reserved. #cython: language_level=3 @@ -33,3 +34,7 @@ cdef class MW(PyverbsCM): cdef class DMMR(MR): cdef object dm + +cdef class DmaBufMR(MR): + cdef object dmabuf + cdef unsigned long offset diff --git a/pyverbs/mr.pyx b/pyverbs/mr.pyx index 7011da1..4102d3c 100644 --- a/pyverbs/mr.pyx +++ b/pyverbs/mr.pyx @@ -1,11 +1,12 @@ # SPDX-License-Identifier: (GPL-2.0 OR Linux-OpenIB) # Copyright (c) 2019, Mellanox Technologies. All rights reserved. See COPYING file +# Copyright (c) 2020, Intel Corporation. All rights reserved. import resource import logging from posix.mman cimport mmap, munmap, MAP_PRIVATE, PROT_READ, PROT_WRITE, \ - MAP_ANONYMOUS, MAP_HUGETLB + MAP_ANONYMOUS, MAP_HUGETLB, MAP_SHARED from pyverbs.pyverbs_error import PyverbsError, PyverbsRDMAError, \ PyverbsUserError from libc.stdint cimport uintptr_t, SIZE_MAX @@ -14,9 +15,10 @@ from posix.stdlib cimport posix_memalign from libc.string cimport memcpy, memset cimport pyverbs.libibverbs_enums as e from pyverbs.device cimport DM -from libc.stdlib cimport free +from libc.stdlib cimport free, malloc from .cmid cimport CMID from .pd cimport PD +from .dmabuf cimport DmaBuf cdef extern from 'sys/mman.h': cdef void* MAP_FAILED @@ -348,6 +350,77 @@ cdef class DMMR(MR): cpdef read(self, length, offset): return self.dm.copy_from_dm(offset, length) +cdef class DmaBufMR(MR): + def __init__(self, PD pd not None, length=0, access=0, DmaBuf dmabuf=None, offset=0): + """ + Initializes a DmaBufMR (DMA-BUF Memory Region) of the given length + and access flags using the given PD and DmaBuf objects. + :param pd: A PD object + :param length: Length in bytes + :param access: Access flags, see ibv_access_flags enum + :param dmabuf: A DmaBuf object + :param offset: Byte offset from the beginning of the dma-buf + :return: The newly create DMABUFMR + """ + self.logger = logging.getLogger(self.__class__.__name__) + if dmabuf is None: + dmabuf = DmaBuf(length + offset) + self.mr = v.ibv_reg_dmabuf_mr(pd.pd, offset, length, dmabuf.fd, access) + if self.mr == NULL: + raise PyverbsRDMAErrno('Failed to register a dma-buf MR. length: {len}, access flags: {flags}'. + format(len=length, flags=access,)) + super().__init__(pd, length, access) + self.pd = pd + self.dmabuf = dmabuf + self.offset = offset + pd.add_ref(self) + dmabuf.add_ref(self) + self.logger.debug('Registered dma-buf ibv_mr. Length: {len}, access flags {flags}'. + format(len=length, flags=access)) + + def write(self, data, length, offset=0): + """ + Write user data to the dma-buf backing the MR + :param data: User data to write + :param length: Length of the data to write + :param offset: Writing offset + :return: None + """ + if isinstance(data, str): + data = data.encode() + # can't access the attributes if self.dmabuf is used w/o cast + dmabuf = self.dmabuf + cdef int off = offset + self.offset + cdef void *buf = mmap(NULL, length + off, PROT_READ | PROT_WRITE, + MAP_SHARED, dmabuf.dri_fd, dmabuf.map_offset) + if buf == MAP_FAILED: + raise PyverbsError('Failed to map dma-buf of size {l}'. + format(l=length)) + memcpy((buf + off), data, length) + munmap(buf, length + off) + + cpdef read(self, length, offset): + """ + Reads data from the dma-buf backing the MR + :param length: Length of data to read + :param offset: Reading offset + :return: The data on the buffer in the requested offset + """ + # can't access the attributes if self.dmabuf is used w/o cast + dmabuf = self.dmabuf + cdef int off = offset + self.offset + cdef void *buf = mmap(NULL, length + off, PROT_READ | PROT_WRITE, + MAP_SHARED, dmabuf.dri_fd, dmabuf.map_offset) + if buf == MAP_FAILED: + raise PyverbsError('Failed to map dma-buf of size {l}'. + format(l=length)) + cdef char *data =malloc(length) + memset(data, 0, length) + memcpy(data, (buf + off), length) + munmap(buf, length + off) + res = data[:length] + free(data) + return res def mwtype2str(mw_type): mw_types = {1:'IBV_MW_TYPE_1', 2:'IBV_MW_TYPE_2'} From patchwork Mon Nov 23 17:53:03 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Xiong, Jianxin" X-Patchwork-Id: 11925797 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 25D59C63697 for ; Mon, 23 Nov 2020 17:39:22 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id C12DE2072C for ; Mon, 23 Nov 2020 17:39:21 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org C12DE2072C Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=intel.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=dri-devel-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 26FA96E06B; Mon, 23 Nov 2020 17:39:21 +0000 (UTC) Received: from mga06.intel.com (mga06.intel.com [134.134.136.31]) by gabe.freedesktop.org (Postfix) with ESMTPS id 60ACF6E083 for ; Mon, 23 Nov 2020 17:39:11 +0000 (UTC) IronPort-SDR: DkGR7k22GBso8xI/Hj7DpCc/rDZyW3BxsojnIPTlIM0oDMuyhE4XEgT5+dloB3SOpsyRujW7/6 sPVxOgfLQxQg== X-IronPort-AV: E=McAfee;i="6000,8403,9814"; a="233421244" X-IronPort-AV: E=Sophos;i="5.78,364,1599548400"; d="scan'208";a="233421244" X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga004.fm.intel.com ([10.253.24.48]) by orsmga104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 23 Nov 2020 09:39:10 -0800 IronPort-SDR: glbKYsYF/s4/8XJkW7gu3ZfUKhZDWV0rCHbLSxwq8/+giGmEeseqmjIcnENUdheNVW6vyYmGmv 6+9Imk/FTg6A== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.78,364,1599548400"; d="scan'208";a="361525799" Received: from cst-dev.jf.intel.com ([10.23.221.69]) by fmsmga004.fm.intel.com with ESMTP; 23 Nov 2020 09:39:09 -0800 From: Jianxin Xiong To: linux-rdma@vger.kernel.org, dri-devel@lists.freedesktop.org Subject: [PATCH rdma-core 4/5] tests: Add tests for dma-buf based memory regions Date: Mon, 23 Nov 2020 09:53:03 -0800 Message-Id: <1606153984-104583-5-git-send-email-jianxin.xiong@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1606153984-104583-1-git-send-email-jianxin.xiong@intel.com> References: <1606153984-104583-1-git-send-email-jianxin.xiong@intel.com> X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Leon Romanovsky , Jason Gunthorpe , Doug Ledford , Daniel Vetter , Christian Koenig , Jianxin Xiong MIME-Version: 1.0 Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" Define a full set of tests similar to regular MR tests. Add a utility function to generate access flags for dma-buf based MRs because the set of supported flags is smaller. Signed-off-by: Jianxin Xiong --- tests/test_mr.py | 130 ++++++++++++++++++++++++++++++++++++++++++++++++++++++- tests/utils.py | 25 +++++++++++ 2 files changed, 154 insertions(+), 1 deletion(-) diff --git a/tests/test_mr.py b/tests/test_mr.py index adc649c..8d7f002 100644 --- a/tests/test_mr.py +++ b/tests/test_mr.py @@ -9,9 +9,10 @@ import errno from tests.base import PyverbsAPITestCase, RCResources, RDMATestCase from pyverbs.pyverbs_error import PyverbsRDMAError, PyverbsError -from pyverbs.mr import MR, MW, DMMR, MWBindInfo, MWBind +from pyverbs.mr import MR, MW, DMMR, DmaBufMR, MWBindInfo, MWBind from pyverbs.qp import QPCap, QPInitAttr, QPAttr, QP from pyverbs.wr import SendWR +from pyverbs.dmabuf import DmaBuf import pyverbs.device as d from pyverbs.pd import PD import pyverbs.enums as e @@ -366,3 +367,130 @@ class DMMRTest(PyverbsAPITestCase): dm_mr = DMMR(pd, dm_mr_len, e.IBV_ACCESS_ZERO_BASED, dm=dm, offset=dm_mr_offset) dm_mr.close() + +class DmaBufMRTest(PyverbsAPITestCase): + """ + Test various functionalities of the DmaBufMR class. + """ + def test_dmabuf_reg_mr(self): + """ + Test ibv_reg_dmabuf_mr() + """ + for ctx, attr, attr_ex in self.devices: + with PD(ctx) as pd: + flags = u.get_dmabuf_access_flags(ctx) + for f in flags: + len = u.get_mr_length() + off = random.randint(0, len//2) + with DmaBufMR(pd, len, f, offset=off) as mr: + pass + + def test_dmabuf_dereg_mr(self): + """ + Test ibv_dereg_mr() with DmaBufMR + """ + for ctx, attr, attr_ex in self.devices: + with PD(ctx) as pd: + flags = u.get_dmabuf_access_flags(ctx) + for f in flags: + len = u.get_mr_length() + off = random.randint(0, len//2) + with DmaBufMR(pd, len, f, offset=off) as mr: + mr.close() + + def test_dmabuf_dereg_mr_twice(self): + """ + Verify that explicit call to DmaBufMR's close() doesn't fail + """ + for ctx, attr, attr_ex in self.devices: + with PD(ctx) as pd: + flags = u.get_dmabuf_access_flags(ctx) + for f in flags: + len = u.get_mr_length() + off = random.randint(0, len//2) + with DmaBufMR(pd, len, f, offset=off) as mr: + # Pyverbs supports multiple destruction of objects, + # we are not expecting an exception here. + mr.close() + mr.close() + + def test_dmabuf_reg_mr_bad_flags(self): + """ + Verify that illegal flags combination fails as expected + """ + for ctx, attr, attr_ex in self.devices: + with PD(ctx) as pd: + for i in range(5): + flags = random.sample([e.IBV_ACCESS_REMOTE_WRITE, + e.IBV_ACCESS_REMOTE_ATOMIC], + random.randint(1, 2)) + mr_flags = 0 + for i in flags: + mr_flags += i.value + try: + DmaBufMR(pd, u.get_mr_length(), mr_flags) + except PyverbsRDMAError as err: + assert 'Failed to register a dma-buf MR' in err.args[0] + else: + raise PyverbsRDMAError('Registered a dma-buf MR with illegal falgs') + + def test_dmabuf_write(self): + """ + Test writing to DmaBufMR's buffer + """ + for ctx, attr, attr_ex in self.devices: + with PD(ctx) as pd: + for i in range(10): + mr_len = u.get_mr_length() + mr_off = random.randint(0, mr_len//2) + flags = u.get_dmabuf_access_flags(ctx) + for f in flags: + with DmaBufMR(pd, mr_len, f, offset=mr_off) as mr: + write_len = min(random.randint(1, MAX_IO_LEN), + mr_len) + mr.write('a' * write_len, write_len) + + def test_dmabuf_read(self): + """ + Test reading from DmaBufMR's buffer + """ + for ctx, attr, attr_ex in self.devices: + with PD(ctx) as pd: + for i in range(10): + mr_len = u.get_mr_length() + mr_off = random.randint(0, mr_len//2) + flags = u.get_dmabuf_access_flags(ctx) + for f in flags: + with DmaBufMR(pd, mr_len, f, offset=mr_off) as mr: + write_len = min(random.randint(1, MAX_IO_LEN), + mr_len) + write_str = 'a' * write_len + mr.write(write_str, write_len) + read_len = random.randint(1, write_len) + offset = random.randint(0, write_len-read_len) + read_str = mr.read(read_len, offset).decode() + assert read_str in write_str + + def test_dmabuf_lkey(self): + """ + Test reading lkey property + """ + for ctx, attr, attr_ex in self.devices: + with PD(ctx) as pd: + length = u.get_mr_length() + flags = u.get_dmabuf_access_flags(ctx) + for f in flags: + with DmaBufMR(pd, length, f) as mr: + mr.lkey + + def test_dmabuf_rkey(self): + """ + Test reading rkey property + """ + for ctx, attr, attr_ex in self.devices: + with PD(ctx) as pd: + length = u.get_mr_length() + flags = u.get_dmabuf_access_flags(ctx) + for f in flags: + with DmaBufMR(pd, length, f) as mr: + mr.rkey diff --git a/tests/utils.py b/tests/utils.py index 7039f41..0ad7110 100644 --- a/tests/utils.py +++ b/tests/utils.py @@ -94,6 +94,31 @@ def get_access_flags(ctx): return arr +def get_dmabuf_access_flags(ctx): + """ + Similar to get_access_flags, except that dma-buf MR only support + a subset of the flags. + :param ctx: Device Context to check capabilities + :return: A random legal value for MR flags + """ + attr = ctx.query_device() + vals = [e.IBV_ACCESS_LOCAL_WRITE, e.IBV_ACCESS_REMOTE_WRITE, + e.IBV_ACCESS_REMOTE_READ, e.IBV_ACCESS_REMOTE_ATOMIC, + e.IBV_ACCESS_RELAXED_ORDERING] + if not attr.atomic_caps & e.IBV_ATOMIC_HCA: + vals.remove(e.IBV_ACCESS_REMOTE_ATOMIC) + arr = [] + for i in range(1, len(vals)): + tmp = list(com(vals, i)) + tmp = filter(filter_illegal_access_flags, tmp) + for t in tmp: # Iterate legal combinations and bitwise OR them + val = 0 + for flag in t: + val += flag.value + arr.append(val) + return arr + + def get_dm_attrs(dm_len): """ Initializes an AllocDmAttr member with the given length and random From patchwork Mon Nov 23 17:53:04 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Xiong, Jianxin" X-Patchwork-Id: 11925801 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 81BD7C63697 for ; Mon, 23 Nov 2020 17:39:29 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 2CCC72072C for ; Mon, 23 Nov 2020 17:39:29 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 2CCC72072C Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=intel.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=dri-devel-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 70EA989F97; Mon, 23 Nov 2020 17:39:28 +0000 (UTC) Received: from mga06.intel.com (mga06.intel.com [134.134.136.31]) by gabe.freedesktop.org (Postfix) with ESMTPS id 7C0736E084 for ; Mon, 23 Nov 2020 17:39:11 +0000 (UTC) IronPort-SDR: reaTaVTx99uAiC/euAA0JvojGMlI8ZLnOxP0yEWynyXKRsE3DJoiApe8i0+/efO+CwM/d768nc Ayz456MsUA1A== X-IronPort-AV: E=McAfee;i="6000,8403,9814"; a="233421246" X-IronPort-AV: E=Sophos;i="5.78,364,1599548400"; d="scan'208";a="233421246" X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga004.fm.intel.com ([10.253.24.48]) by orsmga104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 23 Nov 2020 09:39:10 -0800 IronPort-SDR: oH3phGhklkDmg9YwGLBlv5uAttjT2t2C5x7KxrhvQ1P+urIfs9DpBb3MrkWE1ax8HbJp7CRmMG DEfvrRpKzFrg== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.78,364,1599548400"; d="scan'208";a="361525804" Received: from cst-dev.jf.intel.com ([10.23.221.69]) by fmsmga004.fm.intel.com with ESMTP; 23 Nov 2020 09:39:10 -0800 From: Jianxin Xiong To: linux-rdma@vger.kernel.org, dri-devel@lists.freedesktop.org Subject: [PATCH rdma-core 5/5] tests: Bug fix for get_access_flags() Date: Mon, 23 Nov 2020 09:53:04 -0800 Message-Id: <1606153984-104583-6-git-send-email-jianxin.xiong@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1606153984-104583-1-git-send-email-jianxin.xiong@intel.com> References: <1606153984-104583-1-git-send-email-jianxin.xiong@intel.com> X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Leon Romanovsky , Jason Gunthorpe , Doug Ledford , Daniel Vetter , Christian Koenig , Jianxin Xiong MIME-Version: 1.0 Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" The filter defintion is wrong and causes get_access_flags() always returning empty list. As the result the MR tests using this function are effectively skipped (but report success). Also fix a typo in the comments. Signed-off-by: Jianxin Xiong --- tests/utils.py | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/tests/utils.py b/tests/utils.py index 0ad7110..eee44b4 100644 --- a/tests/utils.py +++ b/tests/utils.py @@ -55,8 +55,8 @@ def filter_illegal_access_flags(element): :param element: A list of access flags to check :return: True if this list is legal, else False """ - if e.IBV_ACCESS_REMOTE_ATOMIC in element or e.IBV_ACCESS_REMOTE_WRITE: - if e.IBV_ACCESS_LOCAL_WRITE: + if e.IBV_ACCESS_REMOTE_ATOMIC in element or e.IBV_ACCESS_REMOTE_WRITE in element: + if not e.IBV_ACCESS_LOCAL_WRITE in element: return False return True @@ -69,7 +69,7 @@ def get_access_flags(ctx): added as well. After verifying that the flags selection is legal, it is appended to an array, assuming it wasn't previously appended. - :param ctx: Device Context to check capabilities + :param ctx: Device Coyyntext to check capabilities :param num: Size of initial collection :return: A random legal value for MR flags """