From patchwork Mon Dec 25 13:56:56 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yishai Hadas X-Patchwork-Id: 10132563 X-Patchwork-Delegate: leon@leon.nu Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 26549605B4 for ; Mon, 25 Dec 2017 13:57:35 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 173BF2F135 for ; Mon, 25 Dec 2017 13:57:35 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 0BF8F2F13C; Mon, 25 Dec 2017 13:57:35 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.9 required=2.0 tests=BAYES_00, RCVD_IN_DNSWL_HI, UNPARSEABLE_RELAY autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id BAB1A2F140 for ; Mon, 25 Dec 2017 13:57:32 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752562AbdLYN53 (ORCPT ); Mon, 25 Dec 2017 08:57:29 -0500 Received: from mail-il-dmz.mellanox.com ([193.47.165.129]:37302 "EHLO mellanox.co.il" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1752403AbdLYN51 (ORCPT ); Mon, 25 Dec 2017 08:57:27 -0500 Received: from Internal Mail-Server by MTLPINE1 (envelope-from yishaih@mellanox.com) with ESMTPS (AES256-SHA encrypted); 25 Dec 2017 15:57:18 +0200 Received: from vnc17.mtl.labs.mlnx (vnc17.mtl.labs.mlnx [10.7.2.17]) by labmailer.mlnx (8.13.8/8.13.8) with ESMTP id vBPDvIJC016289; Mon, 25 Dec 2017 15:57:18 +0200 Received: from vnc17.mtl.labs.mlnx (vnc17.mtl.labs.mlnx [127.0.0.1]) by vnc17.mtl.labs.mlnx (8.13.8/8.13.8) with ESMTP id vBPDvIuq010657; Mon, 25 Dec 2017 15:57:18 +0200 Received: (from yishaih@localhost) by vnc17.mtl.labs.mlnx (8.13.8/8.13.8/Submit) id vBPDvI16010656; Mon, 25 Dec 2017 15:57:18 +0200 From: Yishai Hadas To: linux-rdma@vger.kernel.org Cc: yishaih@mellanox.com, Alexr@mellanox.com, jgg@mellanox.com, majd@mellanox.com Subject: [PATCH rdma-core 4/9] mlx5: Add support for ibv_parent_domain Date: Mon, 25 Dec 2017 15:56:56 +0200 Message-Id: <1514210221-10466-5-git-send-email-yishaih@mellanox.com> X-Mailer: git-send-email 1.8.2.3 In-Reply-To: <1514210221-10466-1-git-send-email-yishaih@mellanox.com> References: <1514210221-10466-1-git-send-email-yishaih@mellanox.com> Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP This patch adds support for ibv_alloc_parent domain verb in the mlx5 driver. The driver uses internally the mlx5_pd structure which as its prefix holds the core ibv_pd so that the return ibv_pd can be used with any verb that gets a protection domain. A parent domain must include a valid protection domain pointer and optionally can hold a thread domain. When used as a parent domain the core ibv_pd completely duplicates the protection domain core fields, so the two pointers are totally interchangeable in all core APIs, minimizing driver changes. Signed-off-by: Yishai Hadas --- providers/mlx5/mlx5.c | 1 + providers/mlx5/mlx5.h | 31 ++++++++++++++++++++++++- providers/mlx5/verbs.c | 61 +++++++++++++++++++++++++++++++++++++++++++++++++- 3 files changed, 91 insertions(+), 2 deletions(-) diff --git a/providers/mlx5/mlx5.c b/providers/mlx5/mlx5.c index d078548..d76964f 100644 --- a/providers/mlx5/mlx5.c +++ b/providers/mlx5/mlx5.c @@ -1028,6 +1028,7 @@ static int mlx5_init_context(struct verbs_device *vdev, v_ctx->modify_cq = mlx5_modify_cq; v_ctx->alloc_td = mlx5_alloc_td; v_ctx->dealloc_td = mlx5_dealloc_td; + v_ctx->alloc_parent_domain = mlx5_alloc_parent_domain; memset(&device_attr, 0, sizeof(device_attr)); if (!mlx5_query_device_ex(ctx, NULL, &device_attr, diff --git a/providers/mlx5/mlx5.h b/providers/mlx5/mlx5.h index 4d4c583..ed75409 100644 --- a/providers/mlx5/mlx5.h +++ b/providers/mlx5/mlx5.h @@ -329,6 +329,13 @@ struct mlx5_td { struct mlx5_pd { struct ibv_pd ibv_pd; uint32_t pdn; + atomic_int refcount; + struct mlx5_pd *mprotection_domain; +}; + +struct mlx5_parent_domain { + struct mlx5_pd mpd; + struct mlx5_td *mtd; }; enum { @@ -557,9 +564,27 @@ static inline struct mlx5_context *to_mctx(struct ibv_context *ibctx) return to_mxxx(ctx, context); } +/* to_mpd always returns the real mlx5_pd object ie the protection domain. */ static inline struct mlx5_pd *to_mpd(struct ibv_pd *ibpd) { - return to_mxxx(pd, pd); + struct mlx5_pd *mpd = to_mxxx(pd, pd); + + if (mpd->mprotection_domain) + return mpd->mprotection_domain; + + return mpd; +} + +static inline struct mlx5_parent_domain *to_mparent_domain(struct ibv_pd *ibpd) +{ + struct mlx5_parent_domain *mparent_domain = + ibpd ? container_of(ibpd, struct mlx5_parent_domain, mpd.ibv_pd) : NULL; + + if (mparent_domain && mparent_domain->mpd.mprotection_domain) + return mparent_domain; + + /* Otherwise ibpd isn't a parent_domain */ + return NULL; } static inline struct mlx5_cq *to_mcq(struct ibv_cq *ibcq) @@ -767,6 +792,10 @@ int mlx5_post_srq_ops(struct ibv_srq *srq, struct ibv_td *mlx5_alloc_td(struct ibv_context *context, struct ibv_td_init_attr *init_attr); int mlx5_dealloc_td(struct ibv_td *td); +struct ibv_pd *mlx5_alloc_parent_domain(struct ibv_context *context, + struct ibv_parent_domain_init_attr *attr); + + static inline void *mlx5_find_uidx(struct mlx5_context *ctx, uint32_t uidx) { int tind = uidx >> MLX5_UIDX_TABLE_SHIFT; diff --git a/providers/mlx5/verbs.c b/providers/mlx5/verbs.c index 43db501..13af0c2 100644 --- a/providers/mlx5/verbs.c +++ b/providers/mlx5/verbs.c @@ -148,6 +148,7 @@ struct ibv_pd *mlx5_alloc_pd(struct ibv_context *context) return NULL; } + atomic_init(&pd->refcount, 1); pd->pdn = resp.pdn; return &pd->ibv_pd; @@ -203,15 +204,73 @@ int mlx5_dealloc_td(struct ibv_td *ib_td) return 0; } +struct ibv_pd * +mlx5_alloc_parent_domain(struct ibv_context *context, + struct ibv_parent_domain_init_attr *attr) +{ + struct mlx5_parent_domain *mparent_domain; + + if (ibv_check_alloc_parent_domain(attr)) + return NULL; + + if (attr->comp_mask) { + errno = EINVAL; + return NULL; + } + + mparent_domain = calloc(1, sizeof(*mparent_domain)); + if (!mparent_domain) { + errno = ENOMEM; + return NULL; + } + + if (attr->td) { + mparent_domain->mtd = to_mtd(attr->td); + atomic_fetch_add(&mparent_domain->mtd->refcount, 1); + } + + mparent_domain->mpd.mprotection_domain = to_mpd(attr->pd); + atomic_fetch_add(&mparent_domain->mpd.mprotection_domain->refcount, 1); + atomic_init(&mparent_domain->mpd.refcount, 1); + + ibv_initialize_parent_domain( + &mparent_domain->mpd.ibv_pd, + &mparent_domain->mpd.mprotection_domain->ibv_pd); + + return &mparent_domain->mpd.ibv_pd; +} + +static int mlx5_dealloc_parent_domain(struct mlx5_parent_domain *mparent_domain) +{ + if (atomic_load(&mparent_domain->mpd.refcount) > 1) + return EBUSY; + + atomic_fetch_sub(&mparent_domain->mpd.mprotection_domain->refcount, 1); + + if (mparent_domain->mtd) + atomic_fetch_sub(&mparent_domain->mtd->refcount, 1); + + free(mparent_domain); + return 0; +} + int mlx5_free_pd(struct ibv_pd *pd) { int ret; + struct mlx5_parent_domain *mparent_domain = to_mparent_domain(pd); + struct mlx5_pd *mpd = to_mpd(pd); + + if (mparent_domain) + return mlx5_dealloc_parent_domain(mparent_domain); + + if (atomic_load(&mpd->refcount) > 1) + return EBUSY; ret = ibv_cmd_dealloc_pd(pd); if (ret) return ret; - free(to_mpd(pd)); + free(mpd); return 0; }