From patchwork Wed Oct 10 06:27:27 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Devesh Sharma X-Patchwork-Id: 10633995 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 36A3D112B for ; Wed, 10 Oct 2018 06:27:47 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 3C1B129622 for ; Wed, 10 Oct 2018 06:27:47 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 2FE6B29633; Wed, 10 Oct 2018 06:27:47 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id CD22629622 for ; Wed, 10 Oct 2018 06:27:46 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1725864AbeJJNsX (ORCPT ); Wed, 10 Oct 2018 09:48:23 -0400 Received: from mail-pf1-f196.google.com ([209.85.210.196]:41186 "EHLO mail-pf1-f196.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725757AbeJJNsX (ORCPT ); Wed, 10 Oct 2018 09:48:23 -0400 Received: by mail-pf1-f196.google.com with SMTP id m77-v6so2092861pfi.8 for ; Tue, 09 Oct 2018 23:27:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=jb2Mg9ICqHU1CnDdG/kdd+nnL8XMKDUCZ5oGfN6QpsA=; b=X3JTrH+PnmlB5quesyo20amBrIQm4bjWPPn6OfQZAFMtPB8K5fW9NJ1q2UYkt5C2NX lNiYFygKZegh/5pqi0dkJC8LWlLNnPLLNzBhYj4No8SJMSMXSYHHDf6924RXn0DVCHhm 0k6jeEI2wkOCpYGxSvL1aLMT9srl4VEAQCYJE= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=jb2Mg9ICqHU1CnDdG/kdd+nnL8XMKDUCZ5oGfN6QpsA=; b=nKduA14bPJwesYeFcbGEKX+9nU+goI/sxYMKJ794RaLs11Rh8KPEJHND/Ulp4WMldx ZAKVHnw8Y2GBOD7dTNbqy/DVoeFI+V3hGakdEsSRqRp+ne76gj9TVM04m4PM7Qq8Nii5 7e32tVYU88hp9GyWlaipBtJPBNf0nuvHs2IgCv9r+KUnRD1zoR/Gt/DJt6GcD8EqSmQP tQ0Sc6VXnc6Csx24k2mMMWbeEuE+LSPfhiNE6yyg9moKRKGpNdpfjOrCszf5hSVeQY7l Inxvb5/JkrN2ucOk+qESdG3bJ5RGpcRl4vF20O/9tuVYOE+4mG+VQeFc7+cy++zacAgb L24A== X-Gm-Message-State: ABuFfoj2nu+Qklg4IPfUjMnxEj3EIbsxGE+xObeGzMn0pcozMtYvLtb4 z9f8p5KhrtsE0khCOGSnnqLTIw== X-Google-Smtp-Source: ACcGV603dbMSqwKB5L/o/fIYydcCstBPZB9FXjRPaMqZ8CPzzHQ7duWsWIqxJYNWMqGfig08netmtA== X-Received: by 2002:a63:7419:: with SMTP id p25-v6mr28624596pgc.395.1539152865304; Tue, 09 Oct 2018 23:27:45 -0700 (PDT) Received: from neo00-el73.dhcp.broadcom.net ([192.19.234.250]) by smtp.gmail.com with ESMTPSA id r65-v6sm31330790pfj.5.2018.10.09.23.27.42 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 09 Oct 2018 23:27:44 -0700 (PDT) From: Devesh Sharma To: jgg@mellanox.com, dledford@redhat.com Cc: linux-rdma@vger.kernel.org, NMoreyChaisemartin@suse.de, Devesh Sharma , Jonathan Richardson , JD Zheng Subject: [PATCH rdma-core V2 1/4] bnxt_re/lib: Reduce memory barrier calls Date: Wed, 10 Oct 2018 02:27:27 -0400 Message-Id: <1539152850-24357-2-git-send-email-devesh.sharma@broadcom.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1539152850-24357-1-git-send-email-devesh.sharma@broadcom.com> References: <1539152850-24357-1-git-send-email-devesh.sharma@broadcom.com> Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Move wmb calls (ring doorbell) out of the loop when processing work requests in post send. This reduces the number of calls and increases performance. in some cases it improves the performance by 35%. Signed-off-by: Devesh Sharma Signed-off-by: Jonathan Richardson Signed-off-by: JD Zheng --- providers/bnxt_re/verbs.c | 28 +++++++++++++++++----------- 1 file changed, 17 insertions(+), 11 deletions(-) diff --git a/providers/bnxt_re/verbs.c b/providers/bnxt_re/verbs.c index 0036cc5..9ce1454 100644 --- a/providers/bnxt_re/verbs.c +++ b/providers/bnxt_re/verbs.c @@ -1222,31 +1222,32 @@ int bnxt_re_post_send(struct ibv_qp *ibvqp, struct ibv_send_wr *wr, struct bnxt_re_bsqe *hdr; struct bnxt_re_wrid *wrid; struct bnxt_re_psns *psns; - void *sqe; - int ret = 0, bytes = 0; uint8_t is_inline = false; + int ret = 0, bytes = 0; + bool ring_db = false; + void *sqe; pthread_spin_lock(&sq->qlock); while (wr) { if ((qp->qpst != IBV_QPS_RTS) && (qp->qpst != IBV_QPS_SQD)) { *bad = wr; - pthread_spin_unlock(&sq->qlock); - return EINVAL; + ret = EINVAL; + goto bad_wr; } if ((qp->qptyp == IBV_QPT_UD) && (wr->opcode != IBV_WR_SEND && wr->opcode != IBV_WR_SEND_WITH_IMM)) { *bad = wr; - pthread_spin_unlock(&sq->qlock); - return EINVAL; + ret = EINVAL; + goto bad_wr; } if (bnxt_re_is_que_full(sq) || wr->num_sge > qp->cap.max_ssge) { *bad = wr; - pthread_spin_unlock(&sq->qlock); - return ENOMEM; + ret = ENOMEM; + goto bad_wr; } sqe = (void *)(sq->va + (sq->tail * sq->stride)); @@ -1305,9 +1306,10 @@ int bnxt_re_post_send(struct ibv_qp *ibvqp, struct ibv_send_wr *wr, bnxt_re_incr_tail(sq); qp->wqe_cnt++; wr = wr->next; - bnxt_re_ring_sq_db(qp); - if (qp->wqe_cnt == BNXT_RE_UD_QP_HW_STALL && qp->qptyp == - IBV_QPT_UD) { + ring_db = true; + + if (qp->wqe_cnt == BNXT_RE_UD_QP_HW_STALL && + qp->qptyp == IBV_QPT_UD) { /* Move RTS to RTS since it is time. */ struct ibv_qp_attr attr; int attr_mask; @@ -1319,6 +1321,10 @@ int bnxt_re_post_send(struct ibv_qp *ibvqp, struct ibv_send_wr *wr, } } +bad_wr: + if (ring_db) + bnxt_re_ring_sq_db(qp); + pthread_spin_unlock(&sq->qlock); return ret; }