From patchwork Tue May 1 23:17:36 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Rohit Zambre X-Patchwork-Id: 10374625 X-Patchwork-Delegate: leon@leon.nu Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 8060860234 for ; Tue, 1 May 2018 23:17:54 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6EB7422A68 for ; Tue, 1 May 2018 23:17:54 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 62CDF24151; Tue, 1 May 2018 23:17:54 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.8 required=2.0 tests=BAYES_00,DKIM_SIGNED, MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI, T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id F2EEC22A68 for ; Tue, 1 May 2018 23:17:53 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751848AbeEAXRx (ORCPT ); Tue, 1 May 2018 19:17:53 -0400 Received: from mail-io0-f195.google.com ([209.85.223.195]:36635 "EHLO mail-io0-f195.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751747AbeEAXRw (ORCPT ); Tue, 1 May 2018 19:17:52 -0400 Received: by mail-io0-f195.google.com with SMTP id d73-v6so15382653iog.3 for ; Tue, 01 May 2018 16:17:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=uci-edu.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=PZQzBBKW+3qXh2KsgqW9PBLYOhzn5+bBPeqMKrcN6Gw=; b=c/Jh5k6ABmyjsCM7LQ37N8SniemhnmWQ/dGIqjMajppUJVfAMNQPrIQyHkIXbBf4bz UUSZYSRLcZOcT4C0MGf2ml8rLXlj644gXNP3nhS2O84JlYo0YPwb8/Rg/CXjTyQzltdC kOD6LDvifHa4q68Owj6gy9fydSZaqXfhRLRPKD9xRSoTi0egM6rmBdgkdLj7dcUgpkOQ VtKKnWXyp+x5rNCQaaAyL3HEqWRt5cd4u9u388dhfgZgOPVbrhb1O23DhKdAYzTPwYTn lQcNafZIDoGIoNvOyqA+eUdT1uWt16WGJDvnJNXNw0o5RJ9TSd1DKtgkj9GdHV7TCrWQ jdUg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=PZQzBBKW+3qXh2KsgqW9PBLYOhzn5+bBPeqMKrcN6Gw=; b=gIj4PEMzStcRY2Q//OuHRrTteHs9WjmaNpMO/NtTkRI+i4KEEDoPmR7wE263buFuQU ksfP5sJFLTC7sQD1+B3R/XyhjxI0UU/ztUAhXMb+FvdMcBrefh1a8V1ENbAVQxjEtk3+ KHlsXZjpBV8eB1Oo5ty20XtvEJsZAOzl5YJA85yX+VqWgN7xDCEHq1Os4iRFQ/txxcws 9fJpbGDZiCA0x5f2uHHmSJSrVddjziYWtideYSqZtvDATfFy1dDSNTtjMMxm50oL7f/e dsAVR9gbfMUJisKiSj5+XcukFPvHcmT8u3Supazj8Ap+yGYtBGrRcKe6uELH1ZQHaHh2 74fA== X-Gm-Message-State: ALQs6tDjnwN0WpIFZroLQM4j3tFa0TYdXjy9P4G8PCQ9qJkQkeGLWZcp bZKbn9v8OLT+xbTugHFmcw1UjmwHcSg= X-Google-Smtp-Source: AB8JxZpruojq7t5wGyPhWU3Y27NuKSjJAQXdDYloQBtiEJCtuUPt5lSESF6jl+6z1+Jf74MuJ2AgWw== X-Received: by 2002:a6b:3306:: with SMTP id z6-v6mr18201372ioz.88.1525216672009; Tue, 01 May 2018 16:17:52 -0700 (PDT) Received: from jlselogin2.ftm.alcf.anl.gov (jlselogin2.pub.jlse.anl.gov. [140.221.96.131]) by smtp.gmail.com with ESMTPSA id f63-v6sm5154938itb.2.2018.05.01.16.17.51 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 01 May 2018 16:17:51 -0700 (PDT) From: Rohit Zambre To: dledford@redhat.com, jgg@mellanox.com Cc: linux-rdma@vger.kernel.org, balaji@anl.gov, amowli@uci.edu, Rohit Zambre Subject: [PATCH rdma-core v2 2/2] mlx5: Disable locking on a QP if it is assigned to a thread domain Date: Tue, 1 May 2018 23:17:36 +0000 Message-Id: <1525216656-11549-3-git-send-email-rzambre@uci.edu> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1525216656-11549-1-git-send-email-rzambre@uci.edu> References: <1525216656-11549-1-git-send-email-rzambre@uci.edu> Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP By creating a QP within a thread domain the application is guaranteeing that the QP will not be accessed concurrently from multiple user threads. Hence, a lock is not needed for QP that is assigned to a thread domain. This patch disables locking on the QP if a thread domain is passed during QP-creation. To do so, it uses a new mlx5_spinlock_init_pd function which will check to see if the ibv_pd is a Parent Domain. If the Parent Domain contains a Thread Domain, it will eliminate the need of a lock on that object. All objects associated with a Protection Domain will use this function to initialize their locks. Signed-off-by: Rohit Zambre --- providers/mlx5/mlx5.h | 14 ++++++++++++++ providers/mlx5/verbs.c | 10 +++++----- 2 files changed, 19 insertions(+), 5 deletions(-) diff --git a/providers/mlx5/mlx5.h b/providers/mlx5/mlx5.h index 08ab610..f0f376c 100644 --- a/providers/mlx5/mlx5.h +++ b/providers/mlx5/mlx5.h @@ -882,6 +882,20 @@ static inline int mlx5_spinlock_init(struct mlx5_spinlock *lock, int need_lock) return pthread_spin_init(&lock->lock, PTHREAD_PROCESS_PRIVATE); } +static inline int mlx5_spinlock_init_pd(struct mlx5_spinlock *lock, struct ibv_pd *pd) +{ + struct mlx5_parent_domain *mparent_domain; + int thread_safe; + + mparent_domain = to_mparent_domain(pd); + if (mparent_domain && mparent_domain->mtd) + thread_safe = 1; + else + thread_safe = mlx5_single_threaded; + + return mlx5_spinlock_init(lock, !thread_safe); +} + static inline int mlx5_spinlock_destroy(struct mlx5_spinlock *lock) { return pthread_spin_destroy(&lock->lock); diff --git a/providers/mlx5/verbs.c b/providers/mlx5/verbs.c index f05b2f3..71728c8 100644 --- a/providers/mlx5/verbs.c +++ b/providers/mlx5/verbs.c @@ -907,7 +907,7 @@ struct ibv_srq *mlx5_create_srq(struct ibv_pd *pd, ibsrq = &srq->vsrq.srq; memset(&cmd, 0, sizeof cmd); - if (mlx5_spinlock_init(&srq->lock, !mlx5_single_threaded)) { + if (mlx5_spinlock_init_pd(&srq->lock, pd)) { fprintf(stderr, "%s-%d:\n", __func__, __LINE__); goto err; } @@ -1751,8 +1751,8 @@ static struct ibv_qp *create_qp(struct ibv_context *context, mlx5_init_qp_indices(qp); - if (mlx5_spinlock_init(&qp->sq.lock, !mlx5_single_threaded) || - mlx5_spinlock_init(&qp->rq.lock, !mlx5_single_threaded)) + if (mlx5_spinlock_init_pd(&qp->sq.lock, attr->pd) || + mlx5_spinlock_init_pd(&qp->rq.lock, attr->pd)) goto err_free_qp_buf; qp->db = mlx5_alloc_dbrec(ctx); @@ -2495,7 +2495,7 @@ struct ibv_srq *mlx5_create_srq_ex(struct ibv_context *context, memset(&cmd, 0, sizeof(cmd)); memset(&resp, 0, sizeof(resp)); - if (mlx5_spinlock_init(&msrq->lock, !mlx5_single_threaded)) { + if (mlx5_spinlock_init_pd(&msrq->lock, attr->pd)) { fprintf(stderr, "%s-%d:\n", __func__, __LINE__); goto err; } @@ -2799,7 +2799,7 @@ static struct ibv_wq *create_wq(struct ibv_context *context, mlx5_init_rwq_indices(rwq); - if (mlx5_spinlock_init(&rwq->rq.lock, !mlx5_single_threaded)) + if (mlx5_spinlock_init_pd(&rwq->rq.lock, attr->pd)) goto err_free_rwq_buf; rwq->db = mlx5_alloc_dbrec(ctx);