From patchwork Tue Dec 8 15:17:04 2015 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yishai Hadas X-Patchwork-Id: 7798631 Return-Path: X-Original-To: patchwork-linux-rdma@patchwork.kernel.org Delivered-To: patchwork-parsemail@patchwork1.web.kernel.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.136]) by patchwork1.web.kernel.org (Postfix) with ESMTP id 973F99F387 for ; Tue, 8 Dec 2015 15:17:57 +0000 (UTC) Received: from mail.kernel.org (localhost [127.0.0.1]) by mail.kernel.org (Postfix) with ESMTP id B437320396 for ; Tue, 8 Dec 2015 15:17:56 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id CE9AA20376 for ; Tue, 8 Dec 2015 15:17:55 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S932922AbbLHPRy (ORCPT ); Tue, 8 Dec 2015 10:17:54 -0500 Received: from [193.47.165.129] ([193.47.165.129]:39386 "EHLO mellanox.co.il" rhost-flags-FAIL-FAIL-OK-FAIL) by vger.kernel.org with ESMTP id S932946AbbLHPRy (ORCPT ); Tue, 8 Dec 2015 10:17:54 -0500 Received: from Internal Mail-Server by MTLPINE1 (envelope-from yishaih@mellanox.com) with ESMTPS (AES256-SHA encrypted); 8 Dec 2015 17:17:19 +0200 Received: from vnc17.mtl.labs.mlnx (vnc17.mtl.labs.mlnx [10.7.2.17]) by labmailer.mlnx (8.13.8/8.13.8) with ESMTP id tB8FHJq4032032; Tue, 8 Dec 2015 17:17:19 +0200 Received: from vnc17.mtl.labs.mlnx (localhost.localdomain [127.0.0.1]) by vnc17.mtl.labs.mlnx (8.13.8/8.13.8) with ESMTP id tB8FHGGv026491; Tue, 8 Dec 2015 17:17:16 +0200 Received: (from yishaih@localhost) by vnc17.mtl.labs.mlnx (8.13.8/8.13.8/Submit) id tB8FHGTu026490; Tue, 8 Dec 2015 17:17:16 +0200 From: Yishai Hadas To: dledford@redhat.com Cc: linux-rdma@vger.kernel.org, yishaih@mellanox.com, ogerlitz@mellanox.com, talal@mellanox.com Subject: [RFC libibverbs 1/2] Add contiguous pages support Date: Tue, 8 Dec 2015 17:17:04 +0200 Message-Id: <1449587825-26444-2-git-send-email-yishaih@mellanox.com> X-Mailer: git-send-email 1.7.11.3 In-Reply-To: <1449587825-26444-1-git-send-email-yishaih@mellanox.com> References: <1449587825-26444-1-git-send-email-yishaih@mellanox.com> Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Spam-Status: No, score=-6.9 required=5.0 tests=BAYES_00, RCVD_IN_DNSWL_HI, T_RP_MATCHES_RCVD, UNPARSEABLE_RELAY autolearn=ham version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on mail.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP To enable contiguous the application should supply NULL address as part of ibv_reg_mr and turn on a bit in the access flags asking for. In that mode the provider allocates contiguous pages and it also manages the un/do_fork_range calls. An extra extended verb between the library and the provider asking for the ibv_mr access flags was added so that the library can know whether to call ibv_dofork_range in its code upon ibv_dereg_mr. Signed-off-by: Yishai Hadas --- include/infiniband/verbs.h | 2 ++ src/verbs.c | 29 +++++++++++++++++++++++------ 2 files changed, 25 insertions(+), 6 deletions(-) diff --git a/include/infiniband/verbs.h b/include/infiniband/verbs.h index ae22768..b80eb4f 100644 --- a/include/infiniband/verbs.h +++ b/include/infiniband/verbs.h @@ -384,6 +384,7 @@ enum ibv_access_flags { IBV_ACCESS_REMOTE_ATOMIC = (1<<3), IBV_ACCESS_MW_BIND = (1<<4), IBV_ACCESS_ON_DEMAND = (1<<6), + IBV_ACCESS_ALLOC_MR = (1<<7), }; struct ibv_pd { @@ -1021,6 +1022,7 @@ enum verbs_context_mask { struct verbs_context { /* "grows up" - new fields go here */ + int (*get_mr_access_flags)(struct ibv_mr *ibv_mr); int (*query_device_ex)(struct ibv_context *context, const struct ibv_query_device_ex_input *input, struct ibv_device_attr_ex *attr, diff --git a/src/verbs.c b/src/verbs.c index ada3515..32d36d6 100644 --- a/src/verbs.c +++ b/src/verbs.c @@ -211,18 +211,27 @@ struct ibv_mr *__ibv_reg_mr(struct ibv_pd *pd, void *addr, size_t length, int access) { struct ibv_mr *mr; + int is_contig; - if (ibv_dontfork_range(addr, length)) - return NULL; + is_contig = !addr && !!((access & IBV_ACCESS_ALLOC_MR)); + + /* fork support for contig is handled by the provider */ + if (!is_contig) { + if (ibv_dontfork_range(addr, length)) + return NULL; + } mr = pd->context->ops.reg_mr(pd, addr, length, access); if (mr) { mr->context = pd->context; mr->pd = pd; - mr->addr = addr; + if (!is_contig) + /* for contig addr is set internally */ + mr->addr = addr; mr->length = length; - } else + } else if (!is_contig) { ibv_dofork_range(addr, length); + } return mr; } @@ -233,10 +242,18 @@ int __ibv_dereg_mr(struct ibv_mr *mr) int ret; void *addr = mr->addr; size_t length = mr->length; + int access_flags = 0; + struct verbs_context *vctx; + + vctx = verbs_get_ctx_op(mr->context, get_mr_access_flags); + if (vctx) + access_flags = vctx->get_mr_access_flags(mr); ret = mr->context->ops.dereg_mr(mr); - if (!ret) - ibv_dofork_range(addr, length); + if (!ret) { + if (!(access_flags & IBV_ACCESS_ALLOC_MR)) + ibv_dofork_range(addr, length); + } return ret; }