From patchwork Sun Nov 12 21:41:39 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yishai Hadas X-Patchwork-Id: 10055003 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id D9C6360586 for ; Sun, 12 Nov 2017 23:40:53 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id CAF5E292C7 for ; Sun, 12 Nov 2017 23:40:53 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id BE703292CC; Sun, 12 Nov 2017 23:40:53 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.9 required=2.0 tests=BAYES_00, RCVD_IN_DNSWL_HI, UNPARSEABLE_RELAY autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 539DA292C7 for ; Sun, 12 Nov 2017 23:40:53 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751417AbdKLXku (ORCPT ); Sun, 12 Nov 2017 18:40:50 -0500 Received: from mail-il-dmz.mellanox.com ([193.47.165.129]:58488 "EHLO mellanox.co.il" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1751085AbdKLXkr (ORCPT ); Sun, 12 Nov 2017 18:40:47 -0500 Received: from Internal Mail-Server by MTLPINE1 (envelope-from yishaih@mellanox.com) with ESMTPS (AES256-SHA encrypted); 12 Nov 2017 23:42:05 +0200 Received: from vnc17.mtl.labs.mlnx (vnc17.mtl.labs.mlnx [10.7.2.17]) by labmailer.mlnx (8.13.8/8.13.8) with ESMTP id vACLg5vj010438; Sun, 12 Nov 2017 23:42:05 +0200 Received: from vnc17.mtl.labs.mlnx (vnc17.mtl.labs.mlnx [127.0.0.1]) by vnc17.mtl.labs.mlnx (8.13.8/8.13.8) with ESMTP id vACLg559006990; Sun, 12 Nov 2017 23:42:05 +0200 Received: (from yishaih@localhost) by vnc17.mtl.labs.mlnx (8.13.8/8.13.8/Submit) id vACLg5RN006989; Sun, 12 Nov 2017 23:42:05 +0200 From: Yishai Hadas To: linux-rdma@vger.kernel.org Cc: yishaih@mellanox.com, majd@mellanox.com, Alexr@mellanox.com, dledford@redhat.com Subject: [PATCH RFC rdma-core 1/5] verbs: Introduce thread domain and its related verbs Date: Sun, 12 Nov 2017 23:41:39 +0200 Message-Id: <1510522903-6838-2-git-send-email-yishaih@mellanox.com> X-Mailer: git-send-email 1.8.2.3 In-Reply-To: <1510522903-6838-1-git-send-email-yishaih@mellanox.com> References: <1510522903-6838-1-git-send-email-yishaih@mellanox.com> Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP The ibv_thread_domain is a user space object that can be used by an application to mark that few verbs resources will be used from single thread. As a result the driver can share some hardware resources for those resources and can drop few locks when it accesses them. This patch introduces this object and its related verbs, more details about the expected usage are described as part of the man page of this commit. Signed-off-by: Yishai Hadas --- libibverbs/man/ibv_alloc_td.3 | 52 +++++++++++++++++++++++++++++++++++++++++++ libibverbs/verbs.h | 35 +++++++++++++++++++++++++++++ 2 files changed, 87 insertions(+) create mode 100755 libibverbs/man/ibv_alloc_td.3 diff --git a/libibverbs/man/ibv_alloc_td.3 b/libibverbs/man/ibv_alloc_td.3 new file mode 100755 index 0000000..8fc2eef --- /dev/null +++ b/libibverbs/man/ibv_alloc_td.3 @@ -0,0 +1,52 @@ +.\" -*- nroff -*- +.\" Licensed under the OpenIB.org BSD license (FreeBSD Variant) - See COPYING.md +.\" +.TH IBV_ALLOC_TD 3 2017-11-06 libibverbs "Libibverbs Programmer's Manual" +.SH "NAME" +ibv_alloc_td(), ibv_dealloc_td() \- allocate and deallocate thread domain object +.SH "SYNOPSIS" +.nf +.B #include +.sp +.BI "struct ibv_td *ibv_alloc_td(struct ibv_context " "*context"); +.sp +.BI "int ibv_dealloc_td(struct ibv_td " "*td"); +.fi +.SH "DESCRIPTION" +.B ibv_alloc_td() +allocates a thread domain object for the RDMA device context +.I context\fR. +.sp +The thread domain object defines how the verbs libraries will use locks and additional HW mapping to achieve best performance for handling multi-thread or single-thread protection. +An application defines the serialization thread accesses scope by choosing which thread domain object will be used when creating verbs resources. +.sp +All verbs resources created with a unique thread domain object should be accessed in a serialized sequence by no more than single thread at a time. +.sp +Using verbs resources without a thread domain object means application can't commit to any threading access serialization model and the verbs library has to provide full thread safe access. +.sp +A +.I struct ibv_td +can be added to a parent domain via +.B ibv_alloc_parent_domain()\fR. +.sp +.B ibv_dealloc_td() +will deallocate the thread domain +.I td\fR. +All resources created with the +.I td +should be destroyed prior to deallocating the +.I td\fR. +.SH "RETURN VALUE" +.B ibv_alloc_td() +returns a pointer to the allocated struct +.I ibv_td +object, or NULL if the request fails (and sets errno to indicates the failure reason). +.sp +.B ibv_dealloc_td() +returns 0 on success, or the value of errno on failure (which indicates the failure reason). +.SH "SEE ALSO" +.BR ibv_alloc_parent_domain (3), +.SH "AUTHORS" +.TP +Alex Rosenbaum + diff --git a/libibverbs/verbs.h b/libibverbs/verbs.h index 6465aa9..c6c536f 100644 --- a/libibverbs/verbs.h +++ b/libibverbs/verbs.h @@ -549,6 +549,10 @@ struct ibv_pd { uint32_t handle; }; +struct ibv_td { + struct ibv_context *context; +}; + enum ibv_xrcd_init_attr_mask { IBV_XRCD_INIT_ATTR_FD = 1 << 0, IBV_XRCD_INIT_ATTR_OFLAGS = 1 << 1, @@ -1637,6 +1641,8 @@ enum verbs_context_mask { struct verbs_context { /* "grows up" - new fields go here */ + int (*dealloc_td)(struct ibv_td *td); + struct ibv_td *(*alloc_td)(struct ibv_context *context); int (*post_srq_ops)(struct ibv_srq *srq, struct ibv_ops_wr *op, struct ibv_ops_wr **bad_op); @@ -2177,6 +2183,35 @@ ibv_create_qp_ex(struct ibv_context *context, struct ibv_qp_init_attr_ex *qp_ini } /** + * ibv_alloc_td - Allocate a thread domain + */ +static inline struct ibv_td *ibv_alloc_td(struct ibv_context *context) +{ + struct verbs_context *vctx; + + vctx = verbs_get_ctx_op(context, alloc_td); + if (!vctx) { + errno = ENOSYS; + return NULL; + } + return vctx->alloc_td(context); +} + +/** + * ibv_dealloc_td - Free a thread domain + */ +static inline int ibv_dealloc_td(struct ibv_td *td) +{ + struct verbs_context *vctx; + + vctx = verbs_get_ctx_op(td->context, dealloc_td); + if (!vctx) + return ENOSYS; + + return vctx->dealloc_td(td); +} + +/** * ibv_query_rt_values_ex - Get current real time @values of a device. * @values - in/out - defines the attributes we need to query/queried. * (Or's bits of enum ibv_values_mask on values->comp_mask field)