From patchwork Wed Jan 31 15:28:49 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Devesh Sharma X-Patchwork-Id: 10194135 X-Patchwork-Delegate: leon@leon.nu Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 8F1FF601A0 for ; Wed, 31 Jan 2018 15:29:08 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7B59628710 for ; Wed, 31 Jan 2018 15:29:08 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 6FFD328718; Wed, 31 Jan 2018 15:29:08 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.8 required=2.0 tests=BAYES_00,DKIM_SIGNED, RCVD_IN_DNSWL_HI,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0EA3728710 for ; Wed, 31 Jan 2018 15:29:08 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753347AbeAaP3H (ORCPT ); Wed, 31 Jan 2018 10:29:07 -0500 Received: from mail-qk0-f196.google.com ([209.85.220.196]:46122 "EHLO mail-qk0-f196.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752497AbeAaP3G (ORCPT ); Wed, 31 Jan 2018 10:29:06 -0500 Received: by mail-qk0-f196.google.com with SMTP id l64so15206908qke.13 for ; Wed, 31 Jan 2018 07:29:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=Sy0L2FWo/a+QJiujj6BqWQnCxXeBydEHZUa4i4Ttj38=; b=HYbm94D+IHr2D5lvpzMTd7ncV939OzRCOfumcImOB4tN0vgjOS6O/+E7sVoyOjwrIm EyiBVZv4INJKdPZqspaFNc4ctlewt5236zNay2RBJp1H1q0u29Vgzh8XamhPxZkkWU5G RjTc0WzdJCUIowSHuQCLP/chrhLWsgdkrkVoY= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=Sy0L2FWo/a+QJiujj6BqWQnCxXeBydEHZUa4i4Ttj38=; b=tKtlC0Y2eRFk61C4qQ4+zXUPc9ImR2HTMMOYPY+6nk+wbf8WlOUP5pvQdSOdQCRQQY SCcgK6rQxObtTetsl5ZtU9zqC8P64Ew4JA7Zs+5+xNz3lsX+RBpSNVqIsy5Het+Ow0vl ltCC5JWsqs1Q/T7ZgC5h6uYg3wHyO2emWeBKe+OpnLu5lJ2WCpGstVqvD+F54mUY1/Cx C/gwazxPt4DPRyGzncQGuBlNijiYrpWrkjLTrWb+1JHSfRLdeuoKbqmY13qeVPThxVY/ CPZJs99rZRjhD2pFaa44/UNWu0ylxGaOBm4UlAxi7KaFdwgrqeyx3XkdmnFco7eE87uk 9xXg== X-Gm-Message-State: AKwxytefmHX8yl/mk2bTDMEjBVuIunWjbkslGhJ9EefBsKgdew6CCn4Z 0kG8YDk2AW6L/8WHZr03+mxAhQ== X-Google-Smtp-Source: AH8x226ORMlhjQb97+VVUSHS4lyo/oD4tzfFSkCjnEvZJ5JXLA+hUUFSFduOVGW7W0+Uqe0J9baQJA== X-Received: by 10.55.175.65 with SMTP id y62mr9404050qke.20.1517412546005; Wed, 31 Jan 2018 07:29:06 -0800 (PST) Received: from neo00-el73.dhcp.broadcom.net ([192.19.239.250]) by smtp.gmail.com with ESMTPSA id a27sm11324645qkj.94.2018.01.31.07.29.02 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 31 Jan 2018 07:29:05 -0800 (PST) From: Devesh Sharma To: dledford@redhat.com, jgg@mellanox.com, leon@kernel.org Cc: linux-rdma@vger.kernel.org, Devesh Sharma , Somnath Kotur , Selvin Xavier Subject: [rdma-core v2 1/3] libbnxt_re: Fix lat test failure in event mode Date: Wed, 31 Jan 2018 10:28:49 -0500 Message-Id: <1517412531-25195-2-git-send-email-devesh.sharma@broadcom.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1517412531-25195-1-git-send-email-devesh.sharma@broadcom.com> References: <1517412531-25195-1-git-send-email-devesh.sharma@broadcom.com> Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP The application assumes that, when CQ is armed, it gives interrupt for the new CQEs generated and not for the existing CQEs. This is in-line with the IB-Spec. However, Broadcom HW generates an interrupt for any unread CQEs not just new ones. This results in a scenario where the application is expecting a completion for a SEND operation but it receives a completion for a prior incoming-send/RQE that was not yet consumed as per the HW thereby leading to failure. Workaround this by deferring the ARM-ing of the CQ when invoked in the notify_cq hook to 'poll_cq' so that the CQ is armed after all completions are consumed. Signed-off-by: Devesh Sharma Signed-off-by: Somnath Kotur Signed-off-by: Selvin Xavier --- providers/bnxt_re/main.h | 3 +++ providers/bnxt_re/verbs.c | 13 ++++++++++++- 2 files changed, 15 insertions(+), 1 deletion(-) diff --git a/providers/bnxt_re/main.h b/providers/bnxt_re/main.h index affe24f..08aa277 100644 --- a/providers/bnxt_re/main.h +++ b/providers/bnxt_re/main.h @@ -76,6 +76,9 @@ struct bnxt_re_cq { struct list_head rfhead; uint32_t cqe_size; uint8_t phase; + int deferred_arm_flags; + bool first_arm; + bool deferred_arm; }; struct bnxt_re_srq { diff --git a/providers/bnxt_re/verbs.c b/providers/bnxt_re/verbs.c index 09ac333..2e88304 100644 --- a/providers/bnxt_re/verbs.c +++ b/providers/bnxt_re/verbs.c @@ -202,6 +202,7 @@ struct ibv_cq *bnxt_re_create_cq(struct ibv_context *ibvctx, int ncqe, cq->phase = resp.phase; cq->cqq.tail = resp.tail; cq->udpi = &cntx->udpi; + cq->first_arm = true; list_head_init(&cq->sfhead); list_head_init(&cq->rfhead); @@ -654,6 +655,11 @@ int bnxt_re_poll_cq(struct ibv_cq *ibvcq, int nwc, struct ibv_wc *wc) pthread_spin_lock(&cq->cqq.qlock); dqed = bnxt_re_poll_one(cq, nwc, wc); + if (cq->deferred_arm) { + bnxt_re_ring_cq_arm_db(cq, cq->deferred_arm_flags); + cq->deferred_arm = false; + cq->deferred_arm_flags = 0; + } pthread_spin_unlock(&cq->cqq.qlock); /* Check if anything is there to flush. */ pthread_spin_lock(&cntx->fqlock); @@ -718,7 +724,12 @@ int bnxt_re_arm_cq(struct ibv_cq *ibvcq, int flags) pthread_spin_lock(&cq->cqq.qlock); flags = !flags ? BNXT_RE_QUE_TYPE_CQ_ARMALL : BNXT_RE_QUE_TYPE_CQ_ARMSE; - bnxt_re_ring_cq_arm_db(cq, flags); + if (cq->first_arm) { + bnxt_re_ring_cq_arm_db(cq, flags); + cq->first_arm = false; + } + cq->deferred_arm = true; + cq->deferred_arm_flags = flags; pthread_spin_unlock(&cq->cqq.qlock); return 0;