From patchwork Wed Jan 31 14:51:49 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Devesh Sharma X-Patchwork-Id: 10194085 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 096B9603EE for ; Wed, 31 Jan 2018 14:52:11 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id EEC642872A for ; Wed, 31 Jan 2018 14:52:10 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id E23F72872D; Wed, 31 Jan 2018 14:52:10 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.8 required=2.0 tests=BAYES_00,DKIM_SIGNED, RCVD_IN_DNSWL_HI,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 880F22872A for ; Wed, 31 Jan 2018 14:52:10 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752826AbeAaOwJ (ORCPT ); Wed, 31 Jan 2018 09:52:09 -0500 Received: from mail-qt0-f196.google.com ([209.85.216.196]:45733 "EHLO mail-qt0-f196.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752337AbeAaOwI (ORCPT ); Wed, 31 Jan 2018 09:52:08 -0500 Received: by mail-qt0-f196.google.com with SMTP id x27so22094075qtm.12 for ; Wed, 31 Jan 2018 06:52:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=Sy0L2FWo/a+QJiujj6BqWQnCxXeBydEHZUa4i4Ttj38=; b=Ko3pO2DdVFQbKYJ+5mZz+eTISOqbAQKklSWF1NEfK6uUbqC0eJxQTISPc8rrZn/usJ EQxKeaLRz09joMVFADzbrL4d8zVvxeROWBkaHGujTnSUSXO4n6tDj7bpWdUrmI3ynNVp HQoKoNXzy3CqKu4eC03FZ/Hq6A6HS9NrRIr1M= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=Sy0L2FWo/a+QJiujj6BqWQnCxXeBydEHZUa4i4Ttj38=; b=lvJnTtiu8+djL9jK9me84/JUE50CAgzVBP5bhF2C17cHB8xK2731ujMsBhLUq+GWfI cin61bCOaQkX0Q7KzL2/LcpK/6TM5lWvf8yHqFkCY6uMW8ZGxGLjnGA3vWgy4HqhM1k1 qFeyDc7okkRWvDS29E0CyaqROiS24SyILCmQ3a0+5apCWv5aIBBzI2hy2VIQwDtaknZr wC5vduK2m/h9/5/gXAbK9dI+UAh764zD8gZh14IekQM6E9OZk1cY2+qtA1MdusFgYF5h v6oe3qZ/moT7xrEOLtcBc8bjTl6s7L2YDQbZoaqzeZhTauFtjbPAmeTWWQtZYxBN0KRl ZzFg== X-Gm-Message-State: AKwxytfME85h/KNo6rvPOSaTjnDC1KxIeYR7TG856d5FY3xa4EqzDXSu b7L7v43GzmZKZoIm+59PiBCY5g== X-Google-Smtp-Source: AH8x225djHFoQmcjOD5c8SNl3oOQZ6gOjiCfiDwOw6nyHDnqePFCsg2r7DWMl2MgIXn+aopBLD4cPQ== X-Received: by 10.237.41.163 with SMTP id o32mr53417736qtd.80.1517410328157; Wed, 31 Jan 2018 06:52:08 -0800 (PST) Received: from neo00-el73.dhcp.broadcom.net ([192.19.239.250]) by smtp.gmail.com with ESMTPSA id s204sm7926932qke.4.2018.01.31.06.52.04 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 31 Jan 2018 06:52:07 -0800 (PST) From: Devesh Sharma To: dledford@redhat.com, jgg@mellanox.com, leon@kernel.org Cc: linux-rdma@vger.kernel.org, Devesh Sharma , Somnath Kotur , Selvin Xavier Subject: [rdma-core 1/3] libbnxt_re: Fix lat test failure in event mode Date: Wed, 31 Jan 2018 09:51:49 -0500 Message-Id: <1517410311-23623-2-git-send-email-devesh.sharma@broadcom.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1517410311-23623-1-git-send-email-devesh.sharma@broadcom.com> References: <1517410311-23623-1-git-send-email-devesh.sharma@broadcom.com> Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP The application assumes that, when CQ is armed, it gives interrupt for the new CQEs generated and not for the existing CQEs. This is in-line with the IB-Spec. However, Broadcom HW generates an interrupt for any unread CQEs not just new ones. This results in a scenario where the application is expecting a completion for a SEND operation but it receives a completion for a prior incoming-send/RQE that was not yet consumed as per the HW thereby leading to failure. Workaround this by deferring the ARM-ing of the CQ when invoked in the notify_cq hook to 'poll_cq' so that the CQ is armed after all completions are consumed. Signed-off-by: Devesh Sharma Signed-off-by: Somnath Kotur Signed-off-by: Selvin Xavier --- providers/bnxt_re/main.h | 3 +++ providers/bnxt_re/verbs.c | 13 ++++++++++++- 2 files changed, 15 insertions(+), 1 deletion(-) diff --git a/providers/bnxt_re/main.h b/providers/bnxt_re/main.h index affe24f..08aa277 100644 --- a/providers/bnxt_re/main.h +++ b/providers/bnxt_re/main.h @@ -76,6 +76,9 @@ struct bnxt_re_cq { struct list_head rfhead; uint32_t cqe_size; uint8_t phase; + int deferred_arm_flags; + bool first_arm; + bool deferred_arm; }; struct bnxt_re_srq { diff --git a/providers/bnxt_re/verbs.c b/providers/bnxt_re/verbs.c index 09ac333..2e88304 100644 --- a/providers/bnxt_re/verbs.c +++ b/providers/bnxt_re/verbs.c @@ -202,6 +202,7 @@ struct ibv_cq *bnxt_re_create_cq(struct ibv_context *ibvctx, int ncqe, cq->phase = resp.phase; cq->cqq.tail = resp.tail; cq->udpi = &cntx->udpi; + cq->first_arm = true; list_head_init(&cq->sfhead); list_head_init(&cq->rfhead); @@ -654,6 +655,11 @@ int bnxt_re_poll_cq(struct ibv_cq *ibvcq, int nwc, struct ibv_wc *wc) pthread_spin_lock(&cq->cqq.qlock); dqed = bnxt_re_poll_one(cq, nwc, wc); + if (cq->deferred_arm) { + bnxt_re_ring_cq_arm_db(cq, cq->deferred_arm_flags); + cq->deferred_arm = false; + cq->deferred_arm_flags = 0; + } pthread_spin_unlock(&cq->cqq.qlock); /* Check if anything is there to flush. */ pthread_spin_lock(&cntx->fqlock); @@ -718,7 +724,12 @@ int bnxt_re_arm_cq(struct ibv_cq *ibvcq, int flags) pthread_spin_lock(&cq->cqq.qlock); flags = !flags ? BNXT_RE_QUE_TYPE_CQ_ARMALL : BNXT_RE_QUE_TYPE_CQ_ARMSE; - bnxt_re_ring_cq_arm_db(cq, flags); + if (cq->first_arm) { + bnxt_re_ring_cq_arm_db(cq, flags); + cq->first_arm = false; + } + cq->deferred_arm = true; + cq->deferred_arm_flags = flags; pthread_spin_unlock(&cq->cqq.qlock); return 0;