From patchwork Thu Jun 8 10:24:52 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Selvin Xavier X-Patchwork-Id: 13271919 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id ECFA7C7EE45 for ; Thu, 8 Jun 2023 10:37:03 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232139AbjFHKhC (ORCPT ); Thu, 8 Jun 2023 06:37:02 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49338 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235957AbjFHKhA (ORCPT ); Thu, 8 Jun 2023 06:37:00 -0400 Received: from mail-pl1-x633.google.com (mail-pl1-x633.google.com [IPv6:2607:f8b0:4864:20::633]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D363AE6C for ; Thu, 8 Jun 2023 03:36:59 -0700 (PDT) Received: by mail-pl1-x633.google.com with SMTP id d9443c01a7336-1b01d3bb571so1885945ad.2 for ; Thu, 08 Jun 2023 03:36:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; t=1686220619; x=1688812619; h=references:in-reply-to:message-id:date:subject:cc:to:from:from:to :cc:subject:date:message-id:reply-to; bh=5uWpYILjByNrdhhrXHCKg6f9gV/+49ynHktODwaTeZk=; b=eznfhcFWjR5BL94/wx2N5GEZSziFwVUxb0QTLAkP7txA6Iffi3mPEH8f+KyXgES1Gz jAYgkzFmVGajYdKgb6MrEbvVn3G0eBvKyY3GDwp1Q/wY2Y2aGpXTGxBzPJw6DASY/DCr yOiw/xGPfERz/lealAxdb40wn6toKy6kwLQ1I= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1686220619; x=1688812619; h=references:in-reply-to:message-id:date:subject:cc:to:from :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=5uWpYILjByNrdhhrXHCKg6f9gV/+49ynHktODwaTeZk=; b=aWy0wkh6lp0OXbyiodC1Vugylnj8dzruHMMAq7oMIJqogPXqKc/F2oEnIfhSb65bqE hYplKslg17RugD4KpKq5XX/o04+TNhSPljNhq1TM5tFPscLKzaFBDfS9+iBk+3PeQt+6 Yx+Fl+EJT9JqM/1/CIVX/HFpFfQteKGKP1AAm+zTjQjpb9ZI4GuOHBz4+n4SXkeftQCD jE6MF+NbYjBgDG18UROW2yIe+gUxYPU62RErVHZpw/vtOM5gHSEDNporE1H7tQrqopWv MewATJuBBKU76eAYkQMUJVviCOC8xHHweH7jqgKg2GKhyDUsBXzExy2zlRRU/Ffa51fu b8hg== X-Gm-Message-State: AC+VfDw5bNysRdzi7nJiV6YIwOhP7Qe++uun4HKqBIm0rPwc890Qow0S Uy4WQCDM2A6My45ymiaNUA7FoA== X-Google-Smtp-Source: ACHHUZ7V8aRPmF1TX3mUAyBePHaYOD0rg5p0tt/gugpjg5yAw4yV0Phnlcase3n6Y1LZzDr2oV1RBw== X-Received: by 2002:a17:903:2450:b0:1ad:bc86:851 with SMTP id l16-20020a170903245000b001adbc860851mr4600826pls.45.1686220619251; Thu, 08 Jun 2023 03:36:59 -0700 (PDT) Received: from dhcp-10-192-206-197.iig.avagotech.net.net ([192.19.234.250]) by smtp.gmail.com with ESMTPSA id jj11-20020a170903048b00b001a980a23802sm1128510plb.111.2023.06.08.03.36.56 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 08 Jun 2023 03:36:58 -0700 (PDT) From: Selvin Xavier To: jgg@ziepe.ca, leon@kernel.org Cc: linux-rdma@vger.kernel.org, andrew.gospodarek@broadcom.com, Kashyap Desai , Selvin Xavier Subject: [PATCH for-next 01/17] RDMA/bnxt_re: wraparound mbox producer index Date: Thu, 8 Jun 2023 03:24:52 -0700 Message-Id: <1686219908-11181-2-git-send-email-selvin.xavier@broadcom.com> X-Mailer: git-send-email 2.5.5 In-Reply-To: <1686219908-11181-1-git-send-email-selvin.xavier@broadcom.com> References: <1686219908-11181-1-git-send-email-selvin.xavier@broadcom.com> Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org From: Kashyap Desai Driver is not handling the wraparound of the mbox producer index correctly. Currently the wraparound happens once u32 max is reached. Bit 31 of the producer index register is special and should be set only once for the first command. Because the producer index overflow setting bit31 after a long time, FW goes to initialization sequence and this causes FW hang. Fix is to wraparound the mbox producer index once it reaches u16 max. Fixes: cee0c7bba486 ("RDMA/bnxt_re: Refactor command queue management code") Fixes: 1ac5a4047975 ("RDMA/bnxt_re: Add bnxt_re RoCE driver") Signed-off-by: Kashyap Desai Signed-off-by: Selvin Xavier --- drivers/infiniband/hw/bnxt_re/qplib_rcfw.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/drivers/infiniband/hw/bnxt_re/qplib_rcfw.c b/drivers/infiniband/hw/bnxt_re/qplib_rcfw.c index 688eaa0..d4ce82b 100644 --- a/drivers/infiniband/hw/bnxt_re/qplib_rcfw.c +++ b/drivers/infiniband/hw/bnxt_re/qplib_rcfw.c @@ -180,7 +180,7 @@ static int __send_message(struct bnxt_qplib_rcfw *rcfw, } while (bsize > 0); cmdq->seq_num++; - cmdq_prod = hwq->prod; + cmdq_prod = hwq->prod & 0xFFFF; if (test_bit(FIRMWARE_FIRST_FLAG, &cmdq->flags)) { /* The very first doorbell write * is required to set this flag @@ -599,7 +599,7 @@ int bnxt_qplib_alloc_rcfw_channel(struct bnxt_qplib_res *res, rcfw->cmdq_depth = BNXT_QPLIB_CMDQE_MAX_CNT_8192; sginfo.pgsize = bnxt_qplib_cmdqe_page_size(rcfw->cmdq_depth); - hwq_attr.depth = rcfw->cmdq_depth; + hwq_attr.depth = rcfw->cmdq_depth & 0x7FFFFFFF; hwq_attr.stride = BNXT_QPLIB_CMDQE_UNITS; hwq_attr.type = HWQ_TYPE_CTX; if (bnxt_qplib_alloc_init_hwq(&cmdq->hwq, &hwq_attr)) { From patchwork Thu Jun 8 10:24:53 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Selvin Xavier X-Patchwork-Id: 13271920 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 89F61C7EE23 for ; Thu, 8 Jun 2023 10:37:12 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236062AbjFHKhK (ORCPT ); Thu, 8 Jun 2023 06:37:10 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49398 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235994AbjFHKhI (ORCPT ); Thu, 8 Jun 2023 06:37:08 -0400 Received: from mail-pl1-x631.google.com (mail-pl1-x631.google.com [IPv6:2607:f8b0:4864:20::631]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B1AC3E4D for ; Thu, 8 Jun 2023 03:37:02 -0700 (PDT) Received: by mail-pl1-x631.google.com with SMTP id d9443c01a7336-1b01bd7093aso1883995ad.1 for ; Thu, 08 Jun 2023 03:37:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; t=1686220622; x=1688812622; h=references:in-reply-to:message-id:date:subject:cc:to:from:from:to :cc:subject:date:message-id:reply-to; bh=gLutcpfZkuIDhI7geDLcW1Kg2azRF+u1Y0FLWhLWpEk=; b=TfQm9WTlj0sec30yeVfRQvfC0v0PGUJKyDoyQQ1rIUfFKWdWBN0ydzULrrxJLlVKkz jrL3vSyZrKDeNrgLmdBl8xnr/Mn9OrXuQuSlbsMUvir6ldkBmg6i64bGtA83HTKdpq/D 7l3p1CqR6zwaBJj0LSlLP5lYsE7ESlkYZ0Lc0= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1686220622; x=1688812622; h=references:in-reply-to:message-id:date:subject:cc:to:from :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=gLutcpfZkuIDhI7geDLcW1Kg2azRF+u1Y0FLWhLWpEk=; b=AEGKz9VBZ7Lo5lXTTwFqW4x+a261+Kx/qW1FIsMIloOh3kh2ikOxxnAdIcBwi6o4Hp MVwX/LcI6knsyoHyocW8eSJG2oGTfmmxGRrPzpFCjB82gtkl3AYeY6KXcLn52b0hB0zU OFCwEneC7znaNDKpx17h+Q6slLobNhkvrQvDMx2UCkFikZCOdYzESvxP/OVU2FiJPUfI gfkaQ+6ezuWSp9EsZ1aCYGdeTIvYED5b/GibeGELSYuhoj3ItMT3fEzW/Q2aN9XIFe1w FJsLyxa/+4CD3PJLEUfbTJWzntnJJzKKvXp3k1aBLpztojeHjDQZK0n2ihYp9OEqW8ok soEA== X-Gm-Message-State: AC+VfDyhES4ELaYDaqovQDFGqHmj9/nTBVBjEwj6G4fMv3OO/Qe1FniH vFYDLNP+WSPkT+rX3BgOkHGgBA== X-Google-Smtp-Source: ACHHUZ6AjQjteJ795O1PfMU5v0UvQSt4kzOicM+8lfDyIBwFHjVAamvvceGk9NYClWP67doLN976oA== X-Received: by 2002:a17:902:7b96:b0:1ac:451d:34b with SMTP id w22-20020a1709027b9600b001ac451d034bmr4876321pll.9.1686220622048; Thu, 08 Jun 2023 03:37:02 -0700 (PDT) Received: from dhcp-10-192-206-197.iig.avagotech.net.net ([192.19.234.250]) by smtp.gmail.com with ESMTPSA id jj11-20020a170903048b00b001a980a23802sm1128510plb.111.2023.06.08.03.36.59 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 08 Jun 2023 03:37:01 -0700 (PDT) From: Selvin Xavier To: jgg@ziepe.ca, leon@kernel.org Cc: linux-rdma@vger.kernel.org, andrew.gospodarek@broadcom.com, Kashyap Desai , Selvin Xavier Subject: [PATCH for-next 02/17] RDMA/bnxt_re: Avoid calling wake_up threads from spin_lock context Date: Thu, 8 Jun 2023 03:24:53 -0700 Message-Id: <1686219908-11181-3-git-send-email-selvin.xavier@broadcom.com> X-Mailer: git-send-email 2.5.5 In-Reply-To: <1686219908-11181-1-git-send-email-selvin.xavier@broadcom.com> References: <1686219908-11181-1-git-send-email-selvin.xavier@broadcom.com> Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org From: Kashyap Desai bnxt_qplib_service_creq can be called from interrupt or tasklet or process context. So the function take irq variant of spin_lock. But when wake_up is invoked with the lock held, it is putting the calling context to sleep. [exception RIP: __wake_up_common+190] RIP: ffffffffb7539d7e RSP: ffffa73300207ad8 RFLAGS: 00000083 RAX: 0000000000000001 RBX: ffff91fa295f69b8 RCX: dead000000000200 RDX: ffffa733344af940 RSI: ffffa73336527940 RDI: ffffa73336527940 RBP: 000000000000001c R8: 0000000000000002 R9: 00000000000299c0 R10: 0000017230de82c5 R11: 0000000000000002 R12: ffffa73300207b28 R13: 0000000000000000 R14: ffffa733341bf928 R15: 0000000000000000 ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018 Call the wakeup after releasing the lock. Fixes: 1ac5a4047975 ("RDMA/bnxt_re: Add bnxt_re RoCE driver") Signed-off-by: Kashyap Desai Signed-off-by: Selvin Xavier --- drivers/infiniband/hw/bnxt_re/qplib_rcfw.c | 13 ++++++++++--- 1 file changed, 10 insertions(+), 3 deletions(-) diff --git a/drivers/infiniband/hw/bnxt_re/qplib_rcfw.c b/drivers/infiniband/hw/bnxt_re/qplib_rcfw.c index d4ce82b..c11b8e7 100644 --- a/drivers/infiniband/hw/bnxt_re/qplib_rcfw.c +++ b/drivers/infiniband/hw/bnxt_re/qplib_rcfw.c @@ -295,7 +295,8 @@ static int bnxt_qplib_process_func_event(struct bnxt_qplib_rcfw *rcfw, } static int bnxt_qplib_process_qp_event(struct bnxt_qplib_rcfw *rcfw, - struct creq_qp_event *qp_event) + struct creq_qp_event *qp_event, + u32 *num_wait) { struct creq_qp_error_notification *err_event; struct bnxt_qplib_hwq *hwq = &rcfw->cmdq.hwq; @@ -304,6 +305,7 @@ static int bnxt_qplib_process_qp_event(struct bnxt_qplib_rcfw *rcfw, u16 cbit, blocked = 0; struct pci_dev *pdev; unsigned long flags; + u32 wait_cmds = 0; __le16 mcookie; u16 cookie; int rc = 0; @@ -363,9 +365,10 @@ static int bnxt_qplib_process_qp_event(struct bnxt_qplib_rcfw *rcfw, crsqe->req_size = 0; if (!blocked) - wake_up(&rcfw->cmdq.waitq); + wait_cmds++; spin_unlock_irqrestore(&hwq->lock, flags); } + *num_wait += wait_cmds; return rc; } @@ -379,6 +382,7 @@ static void bnxt_qplib_service_creq(struct tasklet_struct *t) struct creq_base *creqe; u32 sw_cons, raw_cons; unsigned long flags; + u32 num_wakeup = 0; /* Service the CREQ until budget is over */ spin_lock_irqsave(&hwq->lock, flags); @@ -397,7 +401,8 @@ static void bnxt_qplib_service_creq(struct tasklet_struct *t) switch (type) { case CREQ_BASE_TYPE_QP_EVENT: bnxt_qplib_process_qp_event - (rcfw, (struct creq_qp_event *)creqe); + (rcfw, (struct creq_qp_event *)creqe, + &num_wakeup); creq->stats.creq_qp_event_processed++; break; case CREQ_BASE_TYPE_FUNC_EVENT: @@ -425,6 +430,8 @@ static void bnxt_qplib_service_creq(struct tasklet_struct *t) rcfw->res->cctx, true); } spin_unlock_irqrestore(&hwq->lock, flags); + if (num_wakeup) + wake_up_nr(&rcfw->cmdq.waitq, num_wakeup); } static irqreturn_t bnxt_qplib_creq_irq(int irq, void *dev_instance) From patchwork Thu Jun 8 10:24:54 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Selvin Xavier X-Patchwork-Id: 13271921 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id EBF27C7EE25 for ; Thu, 8 Jun 2023 10:37:13 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235957AbjFHKhM (ORCPT ); Thu, 8 Jun 2023 06:37:12 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49496 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236135AbjFHKhI (ORCPT ); Thu, 8 Jun 2023 06:37:08 -0400 Received: from mail-pf1-x433.google.com (mail-pf1-x433.google.com [IPv6:2607:f8b0:4864:20::433]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 81B8B272D for ; Thu, 8 Jun 2023 03:37:05 -0700 (PDT) Received: by mail-pf1-x433.google.com with SMTP id d2e1a72fcca58-6537d2a8c20so221758b3a.2 for ; Thu, 08 Jun 2023 03:37:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; t=1686220625; x=1688812625; h=references:in-reply-to:message-id:date:subject:cc:to:from:from:to :cc:subject:date:message-id:reply-to; bh=4mkihjp5Nk82g01aGKW/hZHh1APp1U/2DYux+leDQsg=; b=VEaB67RwIWy/8QqPauhetvCj027toZuBe4T0mazj1b2Z/z42a1krD6EB5zBF2PzNuK TISzB2fh6n2jW5468AjblE86n2p8cXncgmBaZ3aKwHNzNXzOYpZiX0HL9aRHsUn4TzPE eUsdt3lVmL7jpk0TGFpymM/x7O9zt5qoDedeA= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1686220625; x=1688812625; h=references:in-reply-to:message-id:date:subject:cc:to:from :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=4mkihjp5Nk82g01aGKW/hZHh1APp1U/2DYux+leDQsg=; b=jfxFucpY3hfDa/tMVc3Yweqax+zPJ5B1xDsbD9+NdjKGWkANlzQVlJFVO2BcjXv0KB iq2NboZ21f3+hVvIIMom3xhQttP5rsjcIlrF1FyGFPb0Dv84vmu7W6hicPgltRe/RD2c aIW09JSp+bstZZc7ql+jf6Rv4C3zbXlA1CxlJjxCI9NOyXIsT54Zyu7c2FRzM7talXYP tvYp7TD8KzPb4ol2pRVYo4xtYXcZqXjCVnNncJkHb3Xt6lcc9oCn+RdTB66j8v0ZfLmg 2yu+Qqr6k9tJ185RTCNwjJjeBmJnt4iVfvGq80yUJvTB1Juhr2ZV5GWdHvf3f5erSkoH 2g8g== X-Gm-Message-State: AC+VfDzZUc1e9V5i4AJFUFeImABSrsfRKTTa4KWp83hlWLchU6Dk6SCE ZW+4+iFmVNY49fYhM6BfuSnFWA== X-Google-Smtp-Source: ACHHUZ7zL8BO+9iSyqyEoqf0mNs73fR1mvbR8IeDrwr51CnpoinJs+G195IOZD7hR9McO9V8o+i7wg== X-Received: by 2002:a05:6a21:6815:b0:114:f112:d8f0 with SMTP id wr21-20020a056a21681500b00114f112d8f0mr3368138pzb.35.1686220624941; Thu, 08 Jun 2023 03:37:04 -0700 (PDT) Received: from dhcp-10-192-206-197.iig.avagotech.net.net ([192.19.234.250]) by smtp.gmail.com with ESMTPSA id jj11-20020a170903048b00b001a980a23802sm1128510plb.111.2023.06.08.03.37.02 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 08 Jun 2023 03:37:04 -0700 (PDT) From: Selvin Xavier To: jgg@ziepe.ca, leon@kernel.org Cc: linux-rdma@vger.kernel.org, andrew.gospodarek@broadcom.com, Kashyap Desai , Selvin Xavier Subject: [PATCH for-next 03/17] RDMA/bnxt_re: remove virt_func check while creating RoCE FW channel Date: Thu, 8 Jun 2023 03:24:54 -0700 Message-Id: <1686219908-11181-4-git-send-email-selvin.xavier@broadcom.com> X-Mailer: git-send-email 2.5.5 In-Reply-To: <1686219908-11181-1-git-send-email-selvin.xavier@broadcom.com> References: <1686219908-11181-1-git-send-email-selvin.xavier@broadcom.com> Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org From: Kashyap Desai There is a common FW communication offset for both PF and VF. Removed code around virt_fn check while creating FW channel. Signed-off-by: Kashyap Desai Signed-off-by: Selvin Xavier --- drivers/infiniband/hw/bnxt_re/main.c | 2 +- drivers/infiniband/hw/bnxt_re/qplib_rcfw.c | 15 ++++++--------- drivers/infiniband/hw/bnxt_re/qplib_rcfw.h | 5 ++--- 3 files changed, 9 insertions(+), 13 deletions(-) diff --git a/drivers/infiniband/hw/bnxt_re/main.c b/drivers/infiniband/hw/bnxt_re/main.c index 4718af6..51372de 100644 --- a/drivers/infiniband/hw/bnxt_re/main.c +++ b/drivers/infiniband/hw/bnxt_re/main.c @@ -1202,7 +1202,7 @@ static int bnxt_re_dev_init(struct bnxt_re_dev *rdev, u8 wqe_mode) db_offt = bnxt_re_get_nqdb_offset(rdev, BNXT_RE_AEQ_IDX); vid = rdev->en_dev->msix_entries[BNXT_RE_AEQ_IDX].vector; rc = bnxt_qplib_enable_rcfw_channel(&rdev->rcfw, - vid, db_offt, rdev->is_virtfn, + vid, db_offt, &bnxt_re_aeq_handler); if (rc) { ibdev_err(&rdev->ibdev, "Failed to enable RCFW channel: %#x\n", diff --git a/drivers/infiniband/hw/bnxt_re/qplib_rcfw.c b/drivers/infiniband/hw/bnxt_re/qplib_rcfw.c index c11b8e7..67140eb 100644 --- a/drivers/infiniband/hw/bnxt_re/qplib_rcfw.c +++ b/drivers/infiniband/hw/bnxt_re/qplib_rcfw.c @@ -724,13 +724,11 @@ int bnxt_qplib_rcfw_start_irq(struct bnxt_qplib_rcfw *rcfw, int msix_vector, return 0; } -static int bnxt_qplib_map_cmdq_mbox(struct bnxt_qplib_rcfw *rcfw, bool is_vf) +static int bnxt_qplib_map_cmdq_mbox(struct bnxt_qplib_rcfw *rcfw) { struct bnxt_qplib_cmdq_mbox *mbox; resource_size_t bar_reg; struct pci_dev *pdev; - u16 prod_offt; - int rc = 0; pdev = rcfw->pdev; mbox = &rcfw->cmdq.cmdq_mbox; @@ -755,11 +753,10 @@ static int bnxt_qplib_map_cmdq_mbox(struct bnxt_qplib_rcfw *rcfw, bool is_vf) return -ENOMEM; } - prod_offt = is_vf ? RCFW_VF_COMM_PROD_OFFSET : - RCFW_PF_COMM_PROD_OFFSET; - mbox->prod = (void __iomem *)(mbox->reg.bar_reg + prod_offt); + mbox->prod = (void __iomem *)(mbox->reg.bar_reg + + RCFW_PF_VF_COMM_PROD_OFFSET); mbox->db = (void __iomem *)(mbox->reg.bar_reg + RCFW_COMM_TRIG_OFFSET); - return rc; + return 0; } static int bnxt_qplib_map_creq_db(struct bnxt_qplib_rcfw *rcfw, u32 reg_offt) @@ -820,7 +817,7 @@ static void bnxt_qplib_start_rcfw(struct bnxt_qplib_rcfw *rcfw) int bnxt_qplib_enable_rcfw_channel(struct bnxt_qplib_rcfw *rcfw, int msix_vector, - int cp_bar_reg_off, int virt_fn, + int cp_bar_reg_off, aeq_handler_t aeq_handler) { struct bnxt_qplib_cmdq_ctx *cmdq; @@ -840,7 +837,7 @@ int bnxt_qplib_enable_rcfw_channel(struct bnxt_qplib_rcfw *rcfw, creq->stats.creq_func_event_processed = 0; creq->aeq_handler = aeq_handler; - rc = bnxt_qplib_map_cmdq_mbox(rcfw, virt_fn); + rc = bnxt_qplib_map_cmdq_mbox(rcfw); if (rc) return rc; diff --git a/drivers/infiniband/hw/bnxt_re/qplib_rcfw.h b/drivers/infiniband/hw/bnxt_re/qplib_rcfw.h index 92f7a25..0dff56e 100644 --- a/drivers/infiniband/hw/bnxt_re/qplib_rcfw.h +++ b/drivers/infiniband/hw/bnxt_re/qplib_rcfw.h @@ -45,8 +45,7 @@ #define RCFW_COMM_PCI_BAR_REGION 0 #define RCFW_COMM_CONS_PCI_BAR_REGION 2 #define RCFW_COMM_BASE_OFFSET 0x600 -#define RCFW_PF_COMM_PROD_OFFSET 0xc -#define RCFW_VF_COMM_PROD_OFFSET 0xc +#define RCFW_PF_VF_COMM_PROD_OFFSET 0xc #define RCFW_COMM_TRIG_OFFSET 0x100 #define RCFW_COMM_SIZE 0x104 @@ -235,7 +234,7 @@ int bnxt_qplib_rcfw_start_irq(struct bnxt_qplib_rcfw *rcfw, int msix_vector, bool need_init); int bnxt_qplib_enable_rcfw_channel(struct bnxt_qplib_rcfw *rcfw, int msix_vector, - int cp_bar_reg_off, int virt_fn, + int cp_bar_reg_off, aeq_handler_t aeq_handler); struct bnxt_qplib_rcfw_sbuf *bnxt_qplib_rcfw_alloc_sbuf( From patchwork Thu Jun 8 10:24:55 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Selvin Xavier X-Patchwork-Id: 13271922 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 52046C7EE23 for ; Thu, 8 Jun 2023 10:37:15 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235981AbjFHKhO (ORCPT ); Thu, 8 Jun 2023 06:37:14 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49468 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236159AbjFHKhJ (ORCPT ); Thu, 8 Jun 2023 06:37:09 -0400 Received: from mail-pj1-x1029.google.com (mail-pj1-x1029.google.com [IPv6:2607:f8b0:4864:20::1029]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4262E2D47 for ; Thu, 8 Jun 2023 03:37:08 -0700 (PDT) Received: by mail-pj1-x1029.google.com with SMTP id 98e67ed59e1d1-256932cea7aso210398a91.3 for ; Thu, 08 Jun 2023 03:37:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; t=1686220627; x=1688812627; h=references:in-reply-to:message-id:date:subject:cc:to:from:from:to :cc:subject:date:message-id:reply-to; bh=SGLwNFaZlCJOOUxjkjGnaGg4Tc47jJ1sRXF5hytEiww=; b=JwZLadgNc02/43dKG90WHvLau1JBX0p4BoKzyOsAF+X+sDvlHj/2eb1U2Hne0qSuzc w9ri3LpMhhgrTpd9Eq59tqkFN221vACKSIIuUbQxrQbzG6wbuaUmKcl1y8nooeouvYkH JzDNgYCJGl3fuByVws3ozlvzsidDWrjG+9nLY= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1686220627; x=1688812627; h=references:in-reply-to:message-id:date:subject:cc:to:from :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=SGLwNFaZlCJOOUxjkjGnaGg4Tc47jJ1sRXF5hytEiww=; b=CWjJom8g21zFaVWqPQ1uWy4SDtYOTgsRwqS4QNnO9Ql8wS9pkOFasb+6ivB2B/C/7g 39RgafYYFyZF9loJ4aItde3iByauwJz716AT5bXtkJC//XvBCU1h1zumuZFGQbGMI8A2 2XLkWeuNhOO2iXINSpIi1S5OEAUGAV1oc5PSvq4fJUObKsuwhhI2tde2Hy1XnL/trwHS kVGmU0np0s9fTu9JMih8+ZXcr/BtRvNBhVsij11uO+B3BQXMK9h45k29XWsAF2FhfuGd 4MHIbZUvdcDs0TfgZpJvdeqLu9t9ez4UDs0STgO1YKLrM7usQvRkWvPBvanmWkbeO3e6 7lIQ== X-Gm-Message-State: AC+VfDw6hzGvZPdT8bm5CS36XtaJqwrBponsUXFdp1i2gvNvHBi72bpW wFlKjkCGflyUx5vIt/x1O4kJhg== X-Google-Smtp-Source: ACHHUZ5OEVXrzvnhqgwQeOaBBgucpjvO1N2MnaaCbsvhuh6OiNuOCE32UoXG+1656aNkBp5KuG0Scw== X-Received: by 2002:a17:90a:498a:b0:255:b1d3:5220 with SMTP id d10-20020a17090a498a00b00255b1d35220mr4781314pjh.20.1686220627589; Thu, 08 Jun 2023 03:37:07 -0700 (PDT) Received: from dhcp-10-192-206-197.iig.avagotech.net.net ([192.19.234.250]) by smtp.gmail.com with ESMTPSA id jj11-20020a170903048b00b001a980a23802sm1128510plb.111.2023.06.08.03.37.05 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 08 Jun 2023 03:37:06 -0700 (PDT) From: Selvin Xavier To: jgg@ziepe.ca, leon@kernel.org Cc: linux-rdma@vger.kernel.org, andrew.gospodarek@broadcom.com, Kashyap Desai , Selvin Xavier Subject: [PATCH for-next 04/17] RDMA/bnxt_re: set fixed command queue depth Date: Thu, 8 Jun 2023 03:24:55 -0700 Message-Id: <1686219908-11181-5-git-send-email-selvin.xavier@broadcom.com> X-Mailer: git-send-email 2.5.5 In-Reply-To: <1686219908-11181-1-git-send-email-selvin.xavier@broadcom.com> References: <1686219908-11181-1-git-send-email-selvin.xavier@broadcom.com> Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org From: Kashyap Desai There is no need of setting max command queue entries based on firmware version check. Removing deperecated code. Signed-off-by: Kashyap Desai Signed-off-by: Selvin Xavier --- drivers/infiniband/hw/bnxt_re/qplib_rcfw.c | 6 ++---- drivers/infiniband/hw/bnxt_re/qplib_rcfw.h | 7 ++----- 2 files changed, 4 insertions(+), 9 deletions(-) diff --git a/drivers/infiniband/hw/bnxt_re/qplib_rcfw.c b/drivers/infiniband/hw/bnxt_re/qplib_rcfw.c index 67140eb..66121fb 100644 --- a/drivers/infiniband/hw/bnxt_re/qplib_rcfw.c +++ b/drivers/infiniband/hw/bnxt_re/qplib_rcfw.c @@ -600,10 +600,8 @@ int bnxt_qplib_alloc_rcfw_channel(struct bnxt_qplib_res *res, "HW channel CREQ allocation failed\n"); goto fail; } - if (ctx->hwrm_intf_ver < HWRM_VERSION_RCFW_CMDQ_DEPTH_CHECK) - rcfw->cmdq_depth = BNXT_QPLIB_CMDQE_MAX_CNT_256; - else - rcfw->cmdq_depth = BNXT_QPLIB_CMDQE_MAX_CNT_8192; + + rcfw->cmdq_depth = BNXT_QPLIB_CMDQE_MAX_CNT; sginfo.pgsize = bnxt_qplib_cmdqe_page_size(rcfw->cmdq_depth); hwq_attr.depth = rcfw->cmdq_depth & 0x7FFFFFFF; diff --git a/drivers/infiniband/hw/bnxt_re/qplib_rcfw.h b/drivers/infiniband/hw/bnxt_re/qplib_rcfw.h index 0dff56e..32e5b67 100644 --- a/drivers/infiniband/hw/bnxt_re/qplib_rcfw.h +++ b/drivers/infiniband/hw/bnxt_re/qplib_rcfw.h @@ -69,8 +69,7 @@ static inline void bnxt_qplib_rcfw_cmd_prep(struct cmdq_base *req, #define RCFW_CMD_WAIT_TIME_MS 20000 /* 20 Seconds timeout */ /* CMDQ elements */ -#define BNXT_QPLIB_CMDQE_MAX_CNT_256 256 -#define BNXT_QPLIB_CMDQE_MAX_CNT_8192 8192 +#define BNXT_QPLIB_CMDQE_MAX_CNT 8192 #define BNXT_QPLIB_CMDQE_BYTES(depth) ((depth) * BNXT_QPLIB_CMDQE_UNITS) static inline u32 bnxt_qplib_cmdqe_npages(u32 depth) @@ -105,12 +104,10 @@ static inline u32 bnxt_qplib_set_cmd_slots(struct cmdq_base *req) return cmd_byte; } -#define RCFW_MAX_COOKIE_VALUE 0x7FFF +#define RCFW_MAX_COOKIE_VALUE (BNXT_QPLIB_CMDQE_MAX_CNT - 1) #define RCFW_CMD_IS_BLOCKING 0x8000 #define RCFW_BLOCKED_CMD_WAIT_COUNT 20000000UL /* 20 sec */ -#define HWRM_VERSION_RCFW_CMDQ_DEPTH_CHECK 0x1000900020011ULL - /* Crsq buf is 1024-Byte */ struct bnxt_qplib_crsbe { u8 data[1024]; From patchwork Thu Jun 8 10:24:56 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Selvin Xavier X-Patchwork-Id: 13271923 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 649A2C7EE23 for ; Thu, 8 Jun 2023 10:37:18 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230290AbjFHKhR (ORCPT ); Thu, 8 Jun 2023 06:37:17 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49532 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236031AbjFHKhO (ORCPT ); Thu, 8 Jun 2023 06:37:14 -0400 Received: from mail-pj1-x1030.google.com (mail-pj1-x1030.google.com [IPv6:2607:f8b0:4864:20::1030]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E46ABE6C for ; Thu, 8 Jun 2023 03:37:10 -0700 (PDT) Received: by mail-pj1-x1030.google.com with SMTP id 98e67ed59e1d1-25690e009c8so213257a91.0 for ; Thu, 08 Jun 2023 03:37:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; t=1686220630; x=1688812630; h=references:in-reply-to:message-id:date:subject:cc:to:from:from:to :cc:subject:date:message-id:reply-to; bh=5SOsUE+OI2hVxdrf4xvI3ntlBJ4TpipHVqKAc0N8Hok=; b=H/tPlTYt1dU8jd2n1kM+NgxP8Fr02ywN3f0EWvTIrm9ZBpCJky+GpJT0w2ViRfZ4o4 bW0MJOEYwiQntTv884XNLt0UUASZ/GMBsDnyuJa1rmlt1cukDKf+1Nlc+5+lCs0Bz99S 4H8HGe8Lul1Y0xK65B4J6wO8JuokgX7ko/Bpo= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1686220630; x=1688812630; h=references:in-reply-to:message-id:date:subject:cc:to:from :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=5SOsUE+OI2hVxdrf4xvI3ntlBJ4TpipHVqKAc0N8Hok=; b=LCX1wrIkNZpNky7wEafos28zc880ioGY4r5S5e5RumGqm/kMFPxllymUoXVYADJmoz TTqJeW/t+iXVM3gJkwMeZrYmB/PC9XvhYUjZUTGhosUZI+pGYChhDCD0NiYO46zLcW7Q RwvH3IUG+H5qEjKquMW97slxdv0p9CezkXjYN5bOYrSNW0Xd20GT4KFzGhRqBGFG2iST UaP5vd7rdw0xHECxX0jhS4D5Mukg3piRYdn7TqP0R5bO7+zTBcrvkKJHMLE6k5euHOdv +eevy1EBrWVX8iDOhjnqtvqyEKOT9UqEGErlEOnj4HwVKFXpcm9mOYvuKN45R7d0DtVS At9g== X-Gm-Message-State: AC+VfDxRrBRI45RhCytkjNKrJWFqlU2tzGNnLrhxp8ihcBcb0NoXv8LG ANrk7mBBwOlfUdKdfCnfaDX3MQ== X-Google-Smtp-Source: ACHHUZ5o76ww0DXdwubhJk7D3r0iVV2V1LMOPcB/LkXJmfn4Z+ikMxsPMgVAFuTQydMhDXU5ML/aoA== X-Received: by 2002:a17:90b:38c5:b0:253:828a:28f8 with SMTP id nn5-20020a17090b38c500b00253828a28f8mr3759740pjb.25.1686220630243; Thu, 08 Jun 2023 03:37:10 -0700 (PDT) Received: from dhcp-10-192-206-197.iig.avagotech.net.net ([192.19.234.250]) by smtp.gmail.com with ESMTPSA id jj11-20020a170903048b00b001a980a23802sm1128510plb.111.2023.06.08.03.37.07 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 08 Jun 2023 03:37:09 -0700 (PDT) From: Selvin Xavier To: jgg@ziepe.ca, leon@kernel.org Cc: linux-rdma@vger.kernel.org, andrew.gospodarek@broadcom.com, Kashyap Desai , Selvin Xavier Subject: [PATCH for-next 05/17] RDMA/bnxt_re: Enhance the existing functions that wait for FW responses Date: Thu, 8 Jun 2023 03:24:56 -0700 Message-Id: <1686219908-11181-6-git-send-email-selvin.xavier@broadcom.com> X-Mailer: git-send-email 2.5.5 In-Reply-To: <1686219908-11181-1-git-send-email-selvin.xavier@broadcom.com> References: <1686219908-11181-1-git-send-email-selvin.xavier@broadcom.com> Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org From: Kashyap Desai Use jiffies based timewait instead of counting iteration for commands that block for FW response. Also add a poll routine for control path commands. This is for polling completion if the waiting commands timeout. This avoids cases where the driver misses completion interrupts. Signed-off-by: Kashyap Desai Signed-off-by: Selvin Xavier --- drivers/infiniband/hw/bnxt_re/qplib_rcfw.c | 65 +++++++++++++++++++++++------- 1 file changed, 51 insertions(+), 14 deletions(-) diff --git a/drivers/infiniband/hw/bnxt_re/qplib_rcfw.c b/drivers/infiniband/hw/bnxt_re/qplib_rcfw.c index 66121fb..3b242cc 100644 --- a/drivers/infiniband/hw/bnxt_re/qplib_rcfw.c +++ b/drivers/infiniband/hw/bnxt_re/qplib_rcfw.c @@ -53,37 +53,74 @@ static void bnxt_qplib_service_creq(struct tasklet_struct *t); -/* Hardware communication channel */ +/** + * __wait_for_resp - Don't hold the cpu context and wait for response + * @rcfw - rcfw channel instance of rdev + * @cookie - cookie to track the command + * + * Wait for command completion in sleepable context. + * + * Returns: + * 0 if command is completed by firmware. + * Non zero error code for rest of the case. + */ static int __wait_for_resp(struct bnxt_qplib_rcfw *rcfw, u16 cookie) { struct bnxt_qplib_cmdq_ctx *cmdq; u16 cbit; - int rc; + int ret; cmdq = &rcfw->cmdq; cbit = cookie % rcfw->cmdq_depth; - rc = wait_event_timeout(cmdq->waitq, - !test_bit(cbit, cmdq->cmdq_bitmap), - msecs_to_jiffies(RCFW_CMD_WAIT_TIME_MS)); - return rc ? 0 : -ETIMEDOUT; + + do { + /* Non zero means command completed */ + ret = wait_event_timeout(cmdq->waitq, + !test_bit(cbit, cmdq->cmdq_bitmap), + msecs_to_jiffies(10000)); + + if (!test_bit(cbit, cmdq->cmdq_bitmap)) + return 0; + + bnxt_qplib_service_creq(&rcfw->creq.creq_tasklet); + + if (!test_bit(cbit, cmdq->cmdq_bitmap)) + return 0; + + } while (true); }; +/** + * __block_for_resp - hold the cpu context and wait for response + * @rcfw - rcfw channel instance of rdev + * @cookie - cookie to track the command + * + * This function will hold the cpu (non-sleepable context) and + * wait for command completion. Maximum holding interval is 8 second. + * + * Returns: + * -ETIMEOUT if command is not completed in specific time interval. + * 0 if command is completed by firmware. + */ static int __block_for_resp(struct bnxt_qplib_rcfw *rcfw, u16 cookie) { - u32 count = RCFW_BLOCKED_CMD_WAIT_COUNT; - struct bnxt_qplib_cmdq_ctx *cmdq; + struct bnxt_qplib_cmdq_ctx *cmdq = &rcfw->cmdq; + unsigned long issue_time = 0; u16 cbit; - cmdq = &rcfw->cmdq; cbit = cookie % rcfw->cmdq_depth; - if (!test_bit(cbit, cmdq->cmdq_bitmap)) - goto done; + issue_time = jiffies; + do { udelay(1); + bnxt_qplib_service_creq(&rcfw->creq.creq_tasklet); - } while (test_bit(cbit, cmdq->cmdq_bitmap) && --count); -done: - return count ? 0 : -ETIMEDOUT; + if (!test_bit(cbit, cmdq->cmdq_bitmap)) + return 0; + + } while (time_before(jiffies, issue_time + (8 * HZ))); + + return -ETIMEDOUT; }; static int __send_message(struct bnxt_qplib_rcfw *rcfw, From patchwork Thu Jun 8 10:24:57 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Selvin Xavier X-Patchwork-Id: 13271924 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D3C63C7EE23 for ; Thu, 8 Jun 2023 10:37:20 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236037AbjFHKhT (ORCPT ); Thu, 8 Jun 2023 06:37:19 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49570 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235963AbjFHKhS (ORCPT ); Thu, 8 Jun 2023 06:37:18 -0400 Received: from mail-pl1-x62f.google.com (mail-pl1-x62f.google.com [IPv6:2607:f8b0:4864:20::62f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B45582D53 for ; Thu, 8 Jun 2023 03:37:13 -0700 (PDT) Received: by mail-pl1-x62f.google.com with SMTP id d9443c01a7336-1b025d26f4fso2752935ad.1 for ; Thu, 08 Jun 2023 03:37:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; t=1686220633; x=1688812633; h=references:in-reply-to:message-id:date:subject:cc:to:from:from:to :cc:subject:date:message-id:reply-to; bh=2VsZ55l/p5N1O/trRd6wo49opUNJKzs0Yf4+iMapfe4=; b=U7WB141eME8/iGe7ltZKwuEpp4ibQQStXosGEfw1PP1zCDSIbCICMHZI7+9bYmNaau Hbkprj0DzBHCy9vmvgECAYoucj47Za831osv++ZFBiqOQy1vpe+k6Lyn0pC4t2/ikCqS 4eOy9EHY4fNS9goM+0kPr6W5t9cwuoBVtwPAM= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1686220633; x=1688812633; h=references:in-reply-to:message-id:date:subject:cc:to:from :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=2VsZ55l/p5N1O/trRd6wo49opUNJKzs0Yf4+iMapfe4=; b=dSmxKyqnGPRjR/O3j9MAsjbtphkuPFI0NBWk5MpsscEARzZLKGPxDUThL5c+lkmQcO 2d0jJWJQYiZHkn2szpgMW9mh7jolb9yg+NSfaZYnMM1lQUZ7dKo1QCDMXGHMo3n3mOcV HWgb0JLOp6KkirxE7MljP0xEGwoAwGD+gmYHnGIiCmqU7ph2wqKmnY/npJSrIefW9BM2 EjcVdPYNWrJYoi5rc7gHRhdFSBtwnMPabS9+iNeXlkeieC2d6TJkNxHk2ZBgOvv26X3Q FhlaxNZp232eSg9p4Vs+nbGnkLexM0av3L4iRoZ4irlupljShf72mrApLcv95/CupwEo yZFA== X-Gm-Message-State: AC+VfDwaiiZrweWxTy/eZqA0hvq4GP9DeuhgH3egRtRc+94FfMyAfc1G Sl33ExOVnX3imt+oBqkqlQawFQ== X-Google-Smtp-Source: ACHHUZ50r/7QJvLvCBCRbpU5EzcNIWy8CFG7QlycDdmhAz/yn+zjQSwVqwJwN21FkuZVEUoxTBMSlA== X-Received: by 2002:a17:903:248:b0:1af:a349:3f0d with SMTP id j8-20020a170903024800b001afa3493f0dmr9554696plh.23.1686220632934; Thu, 08 Jun 2023 03:37:12 -0700 (PDT) Received: from dhcp-10-192-206-197.iig.avagotech.net.net ([192.19.234.250]) by smtp.gmail.com with ESMTPSA id jj11-20020a170903048b00b001a980a23802sm1128510plb.111.2023.06.08.03.37.10 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 08 Jun 2023 03:37:12 -0700 (PDT) From: Selvin Xavier To: jgg@ziepe.ca, leon@kernel.org Cc: linux-rdma@vger.kernel.org, andrew.gospodarek@broadcom.com, Kashyap Desai , Selvin Xavier Subject: [PATCH for-next 06/17] RDMA/bnxt_re: Avoid the command wait if firmware is inactive Date: Thu, 8 Jun 2023 03:24:57 -0700 Message-Id: <1686219908-11181-7-git-send-email-selvin.xavier@broadcom.com> X-Mailer: git-send-email 2.5.5 In-Reply-To: <1686219908-11181-1-git-send-email-selvin.xavier@broadcom.com> References: <1686219908-11181-1-git-send-email-selvin.xavier@broadcom.com> Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org From: Kashyap Desai Add a check to avoid waiting if driver already detects a FW timeout. Return success for resource destroy in case the device is detached. Add helper function to map timeout error code to success. Signed-off-by: Kashyap Desai Signed-off-by: Selvin Xavier --- drivers/infiniband/hw/bnxt_re/qplib_rcfw.c | 52 +++++++++++++++++++++++++++--- 1 file changed, 48 insertions(+), 4 deletions(-) diff --git a/drivers/infiniband/hw/bnxt_re/qplib_rcfw.c b/drivers/infiniband/hw/bnxt_re/qplib_rcfw.c index 3b242cc..3c4f72a 100644 --- a/drivers/infiniband/hw/bnxt_re/qplib_rcfw.c +++ b/drivers/infiniband/hw/bnxt_re/qplib_rcfw.c @@ -54,9 +54,46 @@ static void bnxt_qplib_service_creq(struct tasklet_struct *t); /** + * bnxt_qplib_map_rc - map return type based on opcode + * @opcode - roce slow path opcode + * + * In some cases like firmware halt is detected, the driver is supposed to + * remap the error code of the timed out command. + * + * It is not safe to assume hardware is really inactive so certain opcodes + * like destroy qp etc are not safe to be returned success, but this function + * will be called when FW already reports a timeout. This would be possible + * only when FW crashes and resets. This will clear all the HW resources. + * + * Returns: + * 0 to communicate success to caller. + * Non zero error code to communicate failure to caller. + */ +static int bnxt_qplib_map_rc(u8 opcode) +{ + switch (opcode) { + case CMDQ_BASE_OPCODE_DESTROY_QP: + case CMDQ_BASE_OPCODE_DESTROY_SRQ: + case CMDQ_BASE_OPCODE_DESTROY_CQ: + case CMDQ_BASE_OPCODE_DEALLOCATE_KEY: + case CMDQ_BASE_OPCODE_DEREGISTER_MR: + case CMDQ_BASE_OPCODE_DELETE_GID: + case CMDQ_BASE_OPCODE_DESTROY_QP1: + case CMDQ_BASE_OPCODE_DESTROY_AH: + case CMDQ_BASE_OPCODE_DEINITIALIZE_FW: + case CMDQ_BASE_OPCODE_MODIFY_ROCE_CC: + case CMDQ_BASE_OPCODE_SET_LINK_AGGR_MODE: + return 0; + default: + return -ETIMEDOUT; + } +} + +/** * __wait_for_resp - Don't hold the cpu context and wait for response * @rcfw - rcfw channel instance of rdev * @cookie - cookie to track the command + * @opcode - rcfw submitted for given opcode * * Wait for command completion in sleepable context. * @@ -64,7 +101,7 @@ static void bnxt_qplib_service_creq(struct tasklet_struct *t); * 0 if command is completed by firmware. * Non zero error code for rest of the case. */ -static int __wait_for_resp(struct bnxt_qplib_rcfw *rcfw, u16 cookie) +static int __wait_for_resp(struct bnxt_qplib_rcfw *rcfw, u16 cookie, u8 opcode) { struct bnxt_qplib_cmdq_ctx *cmdq; u16 cbit; @@ -74,6 +111,9 @@ static int __wait_for_resp(struct bnxt_qplib_rcfw *rcfw, u16 cookie) cbit = cookie % rcfw->cmdq_depth; do { + if (test_bit(ERR_DEVICE_DETACHED, &cmdq->flags)) + return bnxt_qplib_map_rc(opcode); + /* Non zero means command completed */ ret = wait_event_timeout(cmdq->waitq, !test_bit(cbit, cmdq->cmdq_bitmap), @@ -94,6 +134,7 @@ static int __wait_for_resp(struct bnxt_qplib_rcfw *rcfw, u16 cookie) * __block_for_resp - hold the cpu context and wait for response * @rcfw - rcfw channel instance of rdev * @cookie - cookie to track the command + * @opcode - rcfw submitted for given opcode * * This function will hold the cpu (non-sleepable context) and * wait for command completion. Maximum holding interval is 8 second. @@ -102,7 +143,7 @@ static int __wait_for_resp(struct bnxt_qplib_rcfw *rcfw, u16 cookie) * -ETIMEOUT if command is not completed in specific time interval. * 0 if command is completed by firmware. */ -static int __block_for_resp(struct bnxt_qplib_rcfw *rcfw, u16 cookie) +static int __block_for_resp(struct bnxt_qplib_rcfw *rcfw, u16 cookie, u8 opcode) { struct bnxt_qplib_cmdq_ctx *cmdq = &rcfw->cmdq; unsigned long issue_time = 0; @@ -112,6 +153,9 @@ static int __block_for_resp(struct bnxt_qplib_rcfw *rcfw, u16 cookie) issue_time = jiffies; do { + if (test_bit(ERR_DEVICE_DETACHED, &cmdq->flags)) + return bnxt_qplib_map_rc(opcode); + udelay(1); bnxt_qplib_service_creq(&rcfw->creq.creq_tasklet); @@ -267,9 +311,9 @@ int bnxt_qplib_rcfw_send_message(struct bnxt_qplib_rcfw *rcfw, } while (retry_cnt--); if (msg->block) - rc = __block_for_resp(rcfw, cookie); + rc = __block_for_resp(rcfw, cookie, opcode); else - rc = __wait_for_resp(rcfw, cookie); + rc = __wait_for_resp(rcfw, cookie, opcode); if (rc) { /* timed out */ dev_err(&rcfw->pdev->dev, "cmdq[%#x]=%#x timedout (%d)msec\n", From patchwork Thu Jun 8 10:24:58 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Selvin Xavier X-Patchwork-Id: 13271925 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 39279C7EE25 for ; Thu, 8 Jun 2023 10:37:27 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234306AbjFHKh0 (ORCPT ); Thu, 8 Jun 2023 06:37:26 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49862 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235695AbjFHKhY (ORCPT ); Thu, 8 Jun 2023 06:37:24 -0400 Received: from mail-pl1-x635.google.com (mail-pl1-x635.google.com [IPv6:2607:f8b0:4864:20::635]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 76F4F1712 for ; Thu, 8 Jun 2023 03:37:16 -0700 (PDT) Received: by mail-pl1-x635.google.com with SMTP id d9443c01a7336-1b04706c974so2889195ad.2 for ; Thu, 08 Jun 2023 03:37:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; t=1686220636; x=1688812636; h=references:in-reply-to:message-id:date:subject:cc:to:from:from:to :cc:subject:date:message-id:reply-to; bh=AChhikG6yh1WerfUWTyfgKQ7r/qWyJ9gkI+8JTMPYYc=; b=hgGti8x8Y6NUHfcYlaTatuKB/2Nc0+FpDSTQuftCm0nW1X+TCfl+gF03lCWtt0csQN vzgeoafEk+C5rIMHmmknf3bohg2HJ9iwlx0Z5w3++l9GA+BHTmVpHkvADCAuj0F0PFIR l8X1NaZToqdFYHPBCmo4PXtJ7OKLIY20qcuIA= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1686220636; x=1688812636; h=references:in-reply-to:message-id:date:subject:cc:to:from :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=AChhikG6yh1WerfUWTyfgKQ7r/qWyJ9gkI+8JTMPYYc=; b=ee4nRHKEZyKOQ6ZoSM+NSX4R0jl7efKE7bmWtDWLVDArcDR4Y0m+fLC99OJIkRIOaG w6OZslzFa1yvRr8ig9JaBS448NgHv9to7Kl68jckntgXlWAEsmsfa7A2BP8eG6t9GxmV YiMwGcd2gvnbKlhnN7OOE6Yij+W9qgHSuUBuLGQsWkuNUiwG43mXP355Vv9Ug7FmhEhF oAFGjLPF0Hf+Fnhdcx3MD4uDeDv0zKald9my273vPcQceTMUpAo+YboESAJ33Z3ElCA2 8IjdJwArO9usmHNb+jMGQlvNr7foMJVSlQeeiTG50JEnY5GLuSvlQLFrtkJJMgd0LoXU ABSQ== X-Gm-Message-State: AC+VfDw6QjbrPYprvo6uWYVJjIu1R+0kJIFhnbwYS/yh0gcdd3RYgMNU JN5dTNeKyQNvOJnu3B8vpmBL8Q== X-Google-Smtp-Source: ACHHUZ4AFGp55LZkUoDJMk8ZN9yy9ko1y8LhDt+2fvhETTV6YUBsARRRlb/B+rmBIKkMIddTQcVJDg== X-Received: by 2002:a17:903:32cb:b0:1b0:3742:e732 with SMTP id i11-20020a17090332cb00b001b03742e732mr9453957plr.23.1686220635771; Thu, 08 Jun 2023 03:37:15 -0700 (PDT) Received: from dhcp-10-192-206-197.iig.avagotech.net.net ([192.19.234.250]) by smtp.gmail.com with ESMTPSA id jj11-20020a170903048b00b001a980a23802sm1128510plb.111.2023.06.08.03.37.13 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 08 Jun 2023 03:37:15 -0700 (PDT) From: Selvin Xavier To: jgg@ziepe.ca, leon@kernel.org Cc: linux-rdma@vger.kernel.org, andrew.gospodarek@broadcom.com, Kashyap Desai , Selvin Xavier Subject: [PATCH for-next 07/17] RDMA/bnxt_re: use shadow qd while posting non blocking rcfw command Date: Thu, 8 Jun 2023 03:24:58 -0700 Message-Id: <1686219908-11181-8-git-send-email-selvin.xavier@broadcom.com> X-Mailer: git-send-email 2.5.5 In-Reply-To: <1686219908-11181-1-git-send-email-selvin.xavier@broadcom.com> References: <1686219908-11181-1-git-send-email-selvin.xavier@broadcom.com> Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org From: Kashyap Desai Whenever there is a fast path IO and create/destroy resources from the slow path is happening in parallel, we may notice high latency of slow path command completion. Introduces a shadow queue depth to prevent the outstanding requests to the FW. Driver will not allow more than #RCFW_CMD_NON_BLOCKING_SHADOW_QD non-blocking commands to the Firmware. Shadow queue depth is a soft limit only for non-blocking commands. Blocking commands will be posted to the firmware as long as there is a free slot. Signed-off-by: Kashyap Desai Signed-off-by: Selvin Xavier --- drivers/infiniband/hw/bnxt_re/qplib_rcfw.c | 60 +++++++++++++++++++++++++++++- drivers/infiniband/hw/bnxt_re/qplib_rcfw.h | 3 ++ 2 files changed, 61 insertions(+), 2 deletions(-) diff --git a/drivers/infiniband/hw/bnxt_re/qplib_rcfw.c b/drivers/infiniband/hw/bnxt_re/qplib_rcfw.c index 3c4f72a..f7d1238 100644 --- a/drivers/infiniband/hw/bnxt_re/qplib_rcfw.c +++ b/drivers/infiniband/hw/bnxt_re/qplib_rcfw.c @@ -281,8 +281,21 @@ static int __send_message(struct bnxt_qplib_rcfw *rcfw, return 0; } -int bnxt_qplib_rcfw_send_message(struct bnxt_qplib_rcfw *rcfw, - struct bnxt_qplib_cmdqmsg *msg) +/** + * __bnxt_qplib_rcfw_send_message - qplib interface to send + * and complete rcfw command. + * @rcfw - rcfw channel instance of rdev + * @msg - qplib message internal + * + * This function does not account shadow queue depth. It will send + * all the command unconditionally as long as send queue is not full. + * + * Returns: + * 0 if command completed by firmware. + * Non zero if the command is not completed by firmware. + */ +static int __bnxt_qplib_rcfw_send_message(struct bnxt_qplib_rcfw *rcfw, + struct bnxt_qplib_cmdqmsg *msg) { struct creq_qp_event *evnt = (struct creq_qp_event *)msg->resp; u16 cookie; @@ -331,6 +344,48 @@ int bnxt_qplib_rcfw_send_message(struct bnxt_qplib_rcfw *rcfw, return rc; } + +/** + * bnxt_qplib_rcfw_send_message - qplib interface to send + * and complete rcfw command. + * @rcfw - rcfw channel instance of rdev + * @msg - qplib message internal + * + * Driver interact with Firmware through rcfw channel/slow path in two ways. + * a. Blocking rcfw command send. In this path, driver cannot hold + * the context for longer period since it is holding cpu until + * command is not completed. + * b. Non-blocking rcfw command send. In this path, driver can hold the + * context for longer period. There may be many pending command waiting + * for completion because of non-blocking nature. + * + * Driver will use shadow queue depth. Current queue depth of 8K + * (due to size of rcfw message there can be actual ~4K rcfw outstanding) + * is not optimal for rcfw command processing in firmware. + * + * Restrict at max #RCFW_CMD_NON_BLOCKING_SHADOW_QD Non-Blocking rcfw commands. + * Allow all blocking commands until there is no queue full. + * + * Returns: + * 0 if command completed by firmware. + * Non zero if the command is not completed by firmware. + */ +int bnxt_qplib_rcfw_send_message(struct bnxt_qplib_rcfw *rcfw, + struct bnxt_qplib_cmdqmsg *msg) +{ + int ret; + + if (!msg->block) { + down(&rcfw->rcfw_inflight); + ret = __bnxt_qplib_rcfw_send_message(rcfw, msg); + up(&rcfw->rcfw_inflight); + } else { + ret = __bnxt_qplib_rcfw_send_message(rcfw, msg); + } + + return ret; +} + /* Completions */ static int bnxt_qplib_process_func_event(struct bnxt_qplib_rcfw *rcfw, struct creq_func_event *func_event) @@ -932,6 +987,7 @@ int bnxt_qplib_enable_rcfw_channel(struct bnxt_qplib_rcfw *rcfw, return rc; } + sema_init(&rcfw->rcfw_inflight, RCFW_CMD_NON_BLOCKING_SHADOW_QD); bnxt_qplib_start_rcfw(rcfw); return 0; diff --git a/drivers/infiniband/hw/bnxt_re/qplib_rcfw.h b/drivers/infiniband/hw/bnxt_re/qplib_rcfw.h index 32e5b67..862bfbf 100644 --- a/drivers/infiniband/hw/bnxt_re/qplib_rcfw.h +++ b/drivers/infiniband/hw/bnxt_re/qplib_rcfw.h @@ -66,6 +66,8 @@ static inline void bnxt_qplib_rcfw_cmd_prep(struct cmdq_base *req, req->cmd_size = cmd_size; } +/* Shadow queue depth for non blocking command */ +#define RCFW_CMD_NON_BLOCKING_SHADOW_QD 64 #define RCFW_CMD_WAIT_TIME_MS 20000 /* 20 Seconds timeout */ /* CMDQ elements */ @@ -197,6 +199,7 @@ struct bnxt_qplib_rcfw { u64 oos_prev; u32 init_oos_stats; u32 cmdq_depth; + struct semaphore rcfw_inflight; }; struct bnxt_qplib_cmdqmsg { From patchwork Thu Jun 8 10:24:59 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Selvin Xavier X-Patchwork-Id: 13271926 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E03BDC7EE25 for ; Thu, 8 Jun 2023 10:37:31 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235994AbjFHKha (ORCPT ); Thu, 8 Jun 2023 06:37:30 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49844 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236025AbjFHKh3 (ORCPT ); Thu, 8 Jun 2023 06:37:29 -0400 Received: from mail-pl1-x62f.google.com (mail-pl1-x62f.google.com [IPv6:2607:f8b0:4864:20::62f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1CA4C2D47 for ; Thu, 8 Jun 2023 03:37:19 -0700 (PDT) Received: by mail-pl1-x62f.google.com with SMTP id d9443c01a7336-1b02fcde49aso1913685ad.0 for ; Thu, 08 Jun 2023 03:37:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; t=1686220638; x=1688812638; h=references:in-reply-to:message-id:date:subject:cc:to:from:from:to :cc:subject:date:message-id:reply-to; bh=0WVYXaNV4s7CiTig4zoZw9ITPO8rLwz66S+Xf+1lSY0=; b=cel1kwWTIjy2wyXoQBweWhzjxYm7QvyuaSUthIkpWDsPeb+VarEbBJIn5GPdAvhwkd M8ZA2A4v+XNUeA3QGrxOPS9Hytrii4afxHX6sfAocgAHa8+YnxpFeb/XPXZ5gANGKGP8 71DORX26IIlpD+Mm75ItyVQOt8WV86n1N7FIw= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1686220638; x=1688812638; h=references:in-reply-to:message-id:date:subject:cc:to:from :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=0WVYXaNV4s7CiTig4zoZw9ITPO8rLwz66S+Xf+1lSY0=; b=f4v2AuCfeNVdqIvd0jENokoiAteFT9jv6/auDe+pFBawiiUIIh1CBnuZ/nWqaq+iYx 36nNOsRK5l8jDRN/UjbbibNg9n8+Q2Gb1I32n+LRKyZKU7ed7+hmtImvwLubjT14xs32 AGKrrmAVEaFpFrjdQSD2VQK/eEeKdLB9EeQDPbQcYn/xfJlgJYnX8apVsttSdlBGRmxb CFEUwWqMtg1m1zpQ97NJa5FUa/SJWPuC9mIwEWBogRhs8h8qjKu8eCamN7YHhFbVeBtw ilLuXwq9Kd4JolQQ6cQ3Ht5czkx2sqmeFS57tqEa8h7075omMPGvT4mMUrqviKLlIW39 DmNg== X-Gm-Message-State: AC+VfDzSmULSvcnFGP4APUmjWiEa4QRnx2JdGidBPOmmPiOgH3lrBS+s vgNejqQvNaENp1UW/CTr0VLKKQ== X-Google-Smtp-Source: ACHHUZ7ZCPguSOk0+bJKsAoHKvhya7Hwt83VzOrW0xVHmiwLVxpwrsu2AREC7PfgIdDTzf9QO74KAw== X-Received: by 2002:a17:902:ef91:b0:1aa:ff24:f8f0 with SMTP id iz17-20020a170902ef9100b001aaff24f8f0mr3928755plb.4.1686220638411; Thu, 08 Jun 2023 03:37:18 -0700 (PDT) Received: from dhcp-10-192-206-197.iig.avagotech.net.net ([192.19.234.250]) by smtp.gmail.com with ESMTPSA id jj11-20020a170903048b00b001a980a23802sm1128510plb.111.2023.06.08.03.37.16 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 08 Jun 2023 03:37:17 -0700 (PDT) From: Selvin Xavier To: jgg@ziepe.ca, leon@kernel.org Cc: linux-rdma@vger.kernel.org, andrew.gospodarek@broadcom.com, Kashyap Desai , Selvin Xavier Subject: [PATCH for-next 08/17] RDMA/bnxt_re: Simplify the function that sends the FW commands Date: Thu, 8 Jun 2023 03:24:59 -0700 Message-Id: <1686219908-11181-9-git-send-email-selvin.xavier@broadcom.com> X-Mailer: git-send-email 2.5.5 In-Reply-To: <1686219908-11181-1-git-send-email-selvin.xavier@broadcom.com> References: <1686219908-11181-1-git-send-email-selvin.xavier@broadcom.com> Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org From: Kashyap Desai - Use __send_message_basic_sanity helper function. - Do not retry posting same command if there is a queue full detection. - ENXIO is used to indicate controller recovery. - In the case of ERR_DEVICE_DETACHED state, the driver should not post commands to the firmware, but also return fabricated written code. Signed-off-by: Kashyap Desai Signed-off-by: Selvin Xavier --- drivers/infiniband/hw/bnxt_re/qplib_rcfw.c | 125 +++++++++++++++-------------- drivers/infiniband/hw/bnxt_re/qplib_rcfw.h | 22 +++++ 2 files changed, 86 insertions(+), 61 deletions(-) diff --git a/drivers/infiniband/hw/bnxt_re/qplib_rcfw.c b/drivers/infiniband/hw/bnxt_re/qplib_rcfw.c index f7d1238..15f6793 100644 --- a/drivers/infiniband/hw/bnxt_re/qplib_rcfw.c +++ b/drivers/infiniband/hw/bnxt_re/qplib_rcfw.c @@ -170,34 +170,22 @@ static int __block_for_resp(struct bnxt_qplib_rcfw *rcfw, u16 cookie, u8 opcode) static int __send_message(struct bnxt_qplib_rcfw *rcfw, struct bnxt_qplib_cmdqmsg *msg) { - struct bnxt_qplib_cmdq_ctx *cmdq = &rcfw->cmdq; - struct bnxt_qplib_hwq *hwq = &cmdq->hwq; + u32 bsize, opcode, free_slots, required_slots; + struct bnxt_qplib_cmdq_ctx *cmdq; struct bnxt_qplib_crsqe *crsqe; struct bnxt_qplib_cmdqe *cmdqe; + struct bnxt_qplib_hwq *hwq; u32 sw_prod, cmdq_prod; struct pci_dev *pdev; unsigned long flags; - u32 bsize, opcode; u16 cookie, cbit; u8 *preq; + cmdq = &rcfw->cmdq; + hwq = &cmdq->hwq; pdev = rcfw->pdev; opcode = __get_cmdq_base_opcode(msg->req, msg->req_sz); - if (!test_bit(FIRMWARE_INITIALIZED_FLAG, &cmdq->flags) && - (opcode != CMDQ_BASE_OPCODE_QUERY_FUNC && - opcode != CMDQ_BASE_OPCODE_INITIALIZE_FW && - opcode != CMDQ_BASE_OPCODE_QUERY_VERSION)) { - dev_err(&pdev->dev, - "RCFW not initialized, reject opcode 0x%x\n", opcode); - return -EINVAL; - } - - if (test_bit(FIRMWARE_INITIALIZED_FLAG, &cmdq->flags) && - opcode == CMDQ_BASE_OPCODE_INITIALIZE_FW) { - dev_err(&pdev->dev, "RCFW already initialized!\n"); - return -EINVAL; - } if (test_bit(FIRMWARE_TIMED_OUT, &cmdq->flags)) return -ETIMEDOUT; @@ -206,40 +194,37 @@ static int __send_message(struct bnxt_qplib_rcfw *rcfw, * cmdqe */ spin_lock_irqsave(&hwq->lock, flags); - if (msg->req->cmd_size >= HWQ_FREE_SLOTS(hwq)) { - dev_err(&pdev->dev, "RCFW: CMDQ is full!\n"); + required_slots = bnxt_qplib_get_cmd_slots(msg->req); + free_slots = HWQ_FREE_SLOTS(hwq); + cookie = cmdq->seq_num & RCFW_MAX_COOKIE_VALUE; + cbit = cookie % rcfw->cmdq_depth; + + if (required_slots >= free_slots || + test_bit(cbit, cmdq->cmdq_bitmap)) { + dev_info_ratelimited(&pdev->dev, + "CMDQ is full req/free %d/%d!", + required_slots, free_slots); spin_unlock_irqrestore(&hwq->lock, flags); return -EAGAIN; } - - - cookie = cmdq->seq_num & RCFW_MAX_COOKIE_VALUE; - cbit = cookie % rcfw->cmdq_depth; if (msg->block) cookie |= RCFW_CMD_IS_BLOCKING; - set_bit(cbit, cmdq->cmdq_bitmap); __set_cmdq_base_cookie(msg->req, msg->req_sz, cpu_to_le16(cookie)); crsqe = &rcfw->crsqe_tbl[cbit]; - if (crsqe->resp) { - spin_unlock_irqrestore(&hwq->lock, flags); - return -EBUSY; - } - - /* change the cmd_size to the number of 16byte cmdq unit. - * req->cmd_size is modified here - */ bsize = bnxt_qplib_set_cmd_slots(msg->req); - - memset(msg->resp, 0, sizeof(*msg->resp)); + crsqe->free_slots = free_slots; crsqe->resp = (struct creq_qp_event *)msg->resp; crsqe->resp->cookie = cpu_to_le16(cookie); crsqe->req_size = __get_cmdq_base_cmd_size(msg->req, msg->req_sz); if (__get_cmdq_base_resp_size(msg->req, msg->req_sz) && msg->sb) { struct bnxt_qplib_rcfw_sbuf *sbuf = msg->sb; - __set_cmdq_base_resp_addr(msg->req, msg->req_sz, cpu_to_le64(sbuf->dma_addr)); + + __set_cmdq_base_resp_addr(msg->req, msg->req_sz, + cpu_to_le64(sbuf->dma_addr)); __set_cmdq_base_resp_size(msg->req, msg->req_sz, - ALIGN(sbuf->size, BNXT_QPLIB_CMDQE_UNITS)); + ALIGN(sbuf->size, + BNXT_QPLIB_CMDQE_UNITS)); } preq = (u8 *)msg->req; @@ -247,11 +232,6 @@ static int __send_message(struct bnxt_qplib_rcfw *rcfw, /* Locate the next cmdq slot */ sw_prod = HWQ_CMP(hwq->prod, hwq); cmdqe = bnxt_qplib_get_qe(hwq, sw_prod, NULL); - if (!cmdqe) { - dev_err(&pdev->dev, - "RCFW request failed with no cmdqe!\n"); - goto done; - } /* Copy a segment of the req cmd to the cmdq */ memset(cmdqe, 0, sizeof(*cmdqe)); memcpy(cmdqe, preq, min_t(u32, bsize, sizeof(*cmdqe))); @@ -275,12 +255,43 @@ static int __send_message(struct bnxt_qplib_rcfw *rcfw, wmb(); writel(cmdq_prod, cmdq->cmdq_mbox.prod); writel(RCFW_CMDQ_TRIG_VAL, cmdq->cmdq_mbox.db); -done: spin_unlock_irqrestore(&hwq->lock, flags); /* Return the CREQ response pointer */ return 0; } +static int __send_message_basic_sanity(struct bnxt_qplib_rcfw *rcfw, + struct bnxt_qplib_cmdqmsg *msg) +{ + struct bnxt_qplib_cmdq_ctx *cmdq; + u32 opcode; + + cmdq = &rcfw->cmdq; + opcode = __get_cmdq_base_opcode(msg->req, msg->req_sz); + + /* Prevent posting if f/w is not in a state to process */ + if (test_bit(ERR_DEVICE_DETACHED, &rcfw->cmdq.flags)) + return -ENXIO; + + if (test_bit(FIRMWARE_INITIALIZED_FLAG, &cmdq->flags) && + opcode == CMDQ_BASE_OPCODE_INITIALIZE_FW) { + dev_err(&rcfw->pdev->dev, "QPLIB: RCFW already initialized!"); + return -EINVAL; + } + + if (!test_bit(FIRMWARE_INITIALIZED_FLAG, &cmdq->flags) && + (opcode != CMDQ_BASE_OPCODE_QUERY_FUNC && + opcode != CMDQ_BASE_OPCODE_INITIALIZE_FW && + opcode != CMDQ_BASE_OPCODE_QUERY_VERSION)) { + dev_err(&rcfw->pdev->dev, + "QPLIB: RCFW not initialized, reject opcode 0x%x", + opcode); + return -EOPNOTSUPP; + } + + return 0; +} + /** * __bnxt_qplib_rcfw_send_message - qplib interface to send * and complete rcfw command. @@ -299,29 +310,21 @@ static int __bnxt_qplib_rcfw_send_message(struct bnxt_qplib_rcfw *rcfw, { struct creq_qp_event *evnt = (struct creq_qp_event *)msg->resp; u16 cookie; - u8 opcode, retry_cnt = 0xFF; int rc = 0; + u8 opcode; - /* Prevent posting if f/w is not in a state to process */ - if (test_bit(ERR_DEVICE_DETACHED, &rcfw->cmdq.flags)) - return 0; + opcode = __get_cmdq_base_opcode(msg->req, msg->req_sz); - do { - opcode = __get_cmdq_base_opcode(msg->req, msg->req_sz); - rc = __send_message(rcfw, msg); - cookie = le16_to_cpu(__get_cmdq_base_cookie(msg->req, msg->req_sz)) & - RCFW_MAX_COOKIE_VALUE; - if (!rc) - break; - if (!retry_cnt || (rc != -EAGAIN && rc != -EBUSY)) { - /* send failed */ - dev_err(&rcfw->pdev->dev, "cmdq[%#x]=%#x send failed\n", - cookie, opcode); - return rc; - } - msg->block ? mdelay(1) : usleep_range(500, 1000); + rc = __send_message_basic_sanity(rcfw, msg); + if (rc) + return rc == -ENXIO ? bnxt_qplib_map_rc(opcode) : rc; + + rc = __send_message(rcfw, msg); + if (rc) + return rc; - } while (retry_cnt--); + cookie = le16_to_cpu(__get_cmdq_base_cookie(msg->req, msg->req_sz)) + & RCFW_MAX_COOKIE_VALUE; if (msg->block) rc = __block_for_resp(rcfw, cookie, opcode); diff --git a/drivers/infiniband/hw/bnxt_re/qplib_rcfw.h b/drivers/infiniband/hw/bnxt_re/qplib_rcfw.h index 862bfbf..b7bbbae 100644 --- a/drivers/infiniband/hw/bnxt_re/qplib_rcfw.h +++ b/drivers/infiniband/hw/bnxt_re/qplib_rcfw.h @@ -89,6 +89,26 @@ static inline u32 bnxt_qplib_cmdqe_page_size(u32 depth) return (bnxt_qplib_cmdqe_npages(depth) * PAGE_SIZE); } +/* Get the number of command units required for the req. The + * function returns correct value only if called before + * setting using bnxt_qplib_set_cmd_slots + */ +static inline u32 bnxt_qplib_get_cmd_slots(struct cmdq_base *req) +{ + u32 cmd_units = 0; + + if (HAS_TLV_HEADER(req)) { + struct roce_tlv *tlv_req = (struct roce_tlv *)req; + + cmd_units = tlv_req->total_size; + } else { + cmd_units = (req->cmd_size + BNXT_QPLIB_CMDQE_UNITS - 1) / + BNXT_QPLIB_CMDQE_UNITS; + } + + return cmd_units; +} + static inline u32 bnxt_qplib_set_cmd_slots(struct cmdq_base *req) { u32 cmd_byte = 0; @@ -130,6 +150,8 @@ typedef int (*aeq_handler_t)(struct bnxt_qplib_rcfw *, void *, void *); struct bnxt_qplib_crsqe { struct creq_qp_event *resp; u32 req_size; + /* Free slots at the time of submission */ + u32 free_slots; }; struct bnxt_qplib_rcfw_sbuf { From patchwork Thu Jun 8 10:25:00 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Selvin Xavier X-Patchwork-Id: 13271927 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B029BC7EE23 for ; Thu, 8 Jun 2023 10:37:38 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236156AbjFHKhh (ORCPT ); Thu, 8 Jun 2023 06:37:37 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50028 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235695AbjFHKhd (ORCPT ); Thu, 8 Jun 2023 06:37:33 -0400 Received: from mail-pg1-x52c.google.com (mail-pg1-x52c.google.com [IPv6:2607:f8b0:4864:20::52c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1BBF319AC for ; Thu, 8 Jun 2023 03:37:22 -0700 (PDT) Received: by mail-pg1-x52c.google.com with SMTP id 41be03b00d2f7-51452556acdso171131a12.2 for ; Thu, 08 Jun 2023 03:37:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; t=1686220641; x=1688812641; h=references:in-reply-to:message-id:date:subject:cc:to:from:from:to :cc:subject:date:message-id:reply-to; bh=jOXsEoYtLnSSwCR/KPNwP+poRh7SwQECvWJsNVX177k=; b=M407A7X70+RGpnxRj2z6KGxgI+obbDVX7v7Bs/3j0dVg9MZVgkB7dd5SCh/l6fZyOR W5iCeIABn+Bjqf27B02LNBYmY3GPsE1yguTOnhKUoX1r3PN+s231ClY3MkmieeZKN+W8 fH8lKdM98z4/Q6Jfk0Wa2vcSH5fnPQWRMoDLE= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1686220641; x=1688812641; h=references:in-reply-to:message-id:date:subject:cc:to:from :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=jOXsEoYtLnSSwCR/KPNwP+poRh7SwQECvWJsNVX177k=; b=ATUl8skgdGri1sguE8SQogeAiIXdvddAZ7vg2fxC+/LL0x1C/5zI63DFPzPC1oNBsx 8xuLzvu1SXToTHAUMkawwLdPtuwy12rKaAYInY7/5x2QGk/wIOyXne9Wnna0bzi3HG4F I45U7Cl0iZsioQOnlH2HgYCEyHfZX/J9mZork6x41TpP3yYkmsN4Dw+jEaLeFLymTP8q 1otvN4kFj3mopHnnSc5kkJRPw6igyLFIxgHwmI+e8JcAI9NKma6hjy8U+o/0jlcHbvs5 uiEWx67bNZ0uh7UOwwnWDveZD0An4rBY5GD311ikf1S/d5A8s6C59p4dFKqXAFL4KrqF uhag== X-Gm-Message-State: AC+VfDz9iqlqdTDVcCgK8CxlAPHBTDjNff/B4Tz6CObJeYqdq9OvRtd6 v173ozQ4veKbnC2P/6zaahYE3A== X-Google-Smtp-Source: ACHHUZ4FypsE7tBOlHWG79KpcNtmJxq1dpazBl2nxN5POaDKffTp9zsR/tzBAYncZR6kmf1uWbmuDg== X-Received: by 2002:a17:902:ee82:b0:1ae:2b97:f387 with SMTP id a2-20020a170902ee8200b001ae2b97f387mr4808268pld.21.1686220641146; Thu, 08 Jun 2023 03:37:21 -0700 (PDT) Received: from dhcp-10-192-206-197.iig.avagotech.net.net ([192.19.234.250]) by smtp.gmail.com with ESMTPSA id jj11-20020a170903048b00b001a980a23802sm1128510plb.111.2023.06.08.03.37.18 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 08 Jun 2023 03:37:20 -0700 (PDT) From: Selvin Xavier To: jgg@ziepe.ca, leon@kernel.org Cc: linux-rdma@vger.kernel.org, andrew.gospodarek@broadcom.com, Kashyap Desai , Selvin Xavier Subject: [PATCH for-next 09/17] RDMA/bnxt_re: add helper function __poll_for_resp Date: Thu, 8 Jun 2023 03:25:00 -0700 Message-Id: <1686219908-11181-10-git-send-email-selvin.xavier@broadcom.com> X-Mailer: git-send-email 2.5.5 In-Reply-To: <1686219908-11181-1-git-send-email-selvin.xavier@broadcom.com> References: <1686219908-11181-1-git-send-email-selvin.xavier@broadcom.com> Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org From: Kashyap Desai This interface will be used if the driver has not enabled interrupt and/or interrupt is disabled for a short period of time. Completion is not possible from interrupt so this interface does self-polling. Signed-off-by: Kashyap Desai Signed-off-by: Selvin Xavier --- drivers/infiniband/hw/bnxt_re/qplib_rcfw.c | 44 +++++++++++++++++++++++++++++- drivers/infiniband/hw/bnxt_re/qplib_rcfw.h | 1 + 2 files changed, 44 insertions(+), 1 deletion(-) diff --git a/drivers/infiniband/hw/bnxt_re/qplib_rcfw.c b/drivers/infiniband/hw/bnxt_re/qplib_rcfw.c index 15f6793..3215f8a 100644 --- a/drivers/infiniband/hw/bnxt_re/qplib_rcfw.c +++ b/drivers/infiniband/hw/bnxt_re/qplib_rcfw.c @@ -260,6 +260,44 @@ static int __send_message(struct bnxt_qplib_rcfw *rcfw, return 0; } +/** + * __poll_for_resp - self poll completion for rcfw command + * @rcfw - rcfw channel instance of rdev + * @cookie - cookie to track the command + * @opcode - rcfw submitted for given opcode + * + * It works same as __wait_for_resp except this function will + * do self polling in sort interval since interrupt is disabled. + * This function can not be called from non-sleepable context. + * + * Returns: + * -ETIMEOUT if command is not completed in specific time interval. + * 0 if command is completed by firmware. + */ +static int __poll_for_resp(struct bnxt_qplib_rcfw *rcfw, u16 cookie, + u8 opcode) +{ + struct bnxt_qplib_cmdq_ctx *cmdq = &rcfw->cmdq; + unsigned long issue_time; + u16 cbit; + + cbit = cookie % rcfw->cmdq_depth; + issue_time = jiffies; + + do { + if (test_bit(ERR_DEVICE_DETACHED, &cmdq->flags)) + return bnxt_qplib_map_rc(opcode); + + usleep_range(1000, 1001); + + bnxt_qplib_service_creq(&rcfw->creq.creq_tasklet); + if (!test_bit(cbit, cmdq->cmdq_bitmap)) + return 0; + if (jiffies_to_msecs(jiffies - issue_time) > 10000) + return -ETIMEDOUT; + } while (true); +}; + static int __send_message_basic_sanity(struct bnxt_qplib_rcfw *rcfw, struct bnxt_qplib_cmdqmsg *msg) { @@ -328,8 +366,10 @@ static int __bnxt_qplib_rcfw_send_message(struct bnxt_qplib_rcfw *rcfw, if (msg->block) rc = __block_for_resp(rcfw, cookie, opcode); - else + else if (atomic_read(&rcfw->rcfw_intr_enabled)) rc = __wait_for_resp(rcfw, cookie, opcode); + else + rc = __poll_for_resp(rcfw, cookie, opcode); if (rc) { /* timed out */ dev_err(&rcfw->pdev->dev, "cmdq[%#x]=%#x timedout (%d)msec\n", @@ -796,6 +836,7 @@ void bnxt_qplib_rcfw_stop_irq(struct bnxt_qplib_rcfw *rcfw, bool kill) kfree(creq->irq_name); creq->irq_name = NULL; creq->requested = false; + atomic_set(&rcfw->rcfw_intr_enabled, 0); } void bnxt_qplib_disable_rcfw_channel(struct bnxt_qplib_rcfw *rcfw) @@ -857,6 +898,7 @@ int bnxt_qplib_rcfw_start_irq(struct bnxt_qplib_rcfw *rcfw, int msix_vector, creq->requested = true; bnxt_qplib_ring_nq_db(&creq->creq_db.dbinfo, res->cctx, true); + atomic_inc(&rcfw->rcfw_intr_enabled); return 0; } diff --git a/drivers/infiniband/hw/bnxt_re/qplib_rcfw.h b/drivers/infiniband/hw/bnxt_re/qplib_rcfw.h index b7bbbae..089e616 100644 --- a/drivers/infiniband/hw/bnxt_re/qplib_rcfw.h +++ b/drivers/infiniband/hw/bnxt_re/qplib_rcfw.h @@ -221,6 +221,7 @@ struct bnxt_qplib_rcfw { u64 oos_prev; u32 init_oos_stats; u32 cmdq_depth; + atomic_t rcfw_intr_enabled; struct semaphore rcfw_inflight; }; From patchwork Thu Jun 8 10:25:01 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Selvin Xavier X-Patchwork-Id: 13271928 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id DC8D2C7EE23 for ; Thu, 8 Jun 2023 10:37:41 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235695AbjFHKhk (ORCPT ); Thu, 8 Jun 2023 06:37:40 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50120 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236155AbjFHKhh (ORCPT ); Thu, 8 Jun 2023 06:37:37 -0400 Received: from mail-pf1-x42d.google.com (mail-pf1-x42d.google.com [IPv6:2607:f8b0:4864:20::42d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9F79E2D4A for ; Thu, 8 Jun 2023 03:37:24 -0700 (PDT) Received: by mail-pf1-x42d.google.com with SMTP id d2e1a72fcca58-65055aa4ed7so218175b3a.3 for ; Thu, 08 Jun 2023 03:37:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; t=1686220644; x=1688812644; h=references:in-reply-to:message-id:date:subject:cc:to:from:from:to :cc:subject:date:message-id:reply-to; bh=tq7eOSDAeLLk5LknjFHp5iJKCx5zSSFDlnOSqSoGxxQ=; b=D+BIkPGN0v1la3Oje7+Ym9PodL8nr/mqUyx2K2thkJOGQbKburYkMKEhnZrye6HXBF 35nAIj1dUnJrFQwhuZYydJk92FvWx20X2v2Qp3/N2S/nyk24tPfVzdfpzyrmko70q5Xx HdYv7ZZxXD0yVGQSZvZJpHLQ9LkyXuiI9Hjl0= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1686220644; x=1688812644; h=references:in-reply-to:message-id:date:subject:cc:to:from :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=tq7eOSDAeLLk5LknjFHp5iJKCx5zSSFDlnOSqSoGxxQ=; b=Po+S7KTIjAZVt5HsKwJJPhReIMzambu1v6m2VrfXsHeBxu3ru76m5t/vF3G8ynqp5T NvrywDoKqVDcjWhTNdAPF5baq/jKDezbuJxNScctuPXJ9VZimPNyvg9PXl/uEMIERkH7 62co3FwYXTuV5dkAQsb3LBJ8OKyW/9Chtj1bC45Z8RgRfoMvirjsW2Zfc5A0IZ30NHxb K0GPRO0K/B+JqBj7Wm7Nm2Gunsvz6lP0XtvkOvox9QfyOvIuK1CkuPTK9ijKegTDOJN0 dZ1UbRRDs2+bVACWQ+uOWLpr4GJyIe50LqGrsyx31xV1s0fUjWCUJ3CMYHeBsMul7uDN 26BQ== X-Gm-Message-State: AC+VfDyBqMgC7xqtaBLylP8uguy/msEnByYGAXeD18pp7q7UiOzEojmR SRa5e8Hbxxhr4fC2gSJ7H/FAsw== X-Google-Smtp-Source: ACHHUZ6nHO7EUYVXNabAN880qe47lfqLfX/AOkbefHePfqb8Jt5zQBVyUitE3qGtnl3yyYWCZo8JRw== X-Received: by 2002:a05:6a20:42a3:b0:112:be13:5a54 with SMTP id o35-20020a056a2042a300b00112be135a54mr3850053pzj.50.1686220643983; Thu, 08 Jun 2023 03:37:23 -0700 (PDT) Received: from dhcp-10-192-206-197.iig.avagotech.net.net ([192.19.234.250]) by smtp.gmail.com with ESMTPSA id jj11-20020a170903048b00b001a980a23802sm1128510plb.111.2023.06.08.03.37.21 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 08 Jun 2023 03:37:23 -0700 (PDT) From: Selvin Xavier To: jgg@ziepe.ca, leon@kernel.org Cc: linux-rdma@vger.kernel.org, andrew.gospodarek@broadcom.com, Kashyap Desai , Selvin Xavier Subject: [PATCH for-next 10/17] RDMA/bnxt_re: handle command completions after driver detect a timedout Date: Thu, 8 Jun 2023 03:25:01 -0700 Message-Id: <1686219908-11181-11-git-send-email-selvin.xavier@broadcom.com> X-Mailer: git-send-email 2.5.5 In-Reply-To: <1686219908-11181-1-git-send-email-selvin.xavier@broadcom.com> References: <1686219908-11181-1-git-send-email-selvin.xavier@broadcom.com> Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org From: Kashyap Desai If calling context detect command timeout, associated memory stored on stack will not be valid. If firmware complete the same command later, this causes incorrect memory access by driver. Added is_waiter_alive to handle delayed completion by firmware. is_waiter_alive is set and reset under command queue lock. Signed-off-by: Kashyap Desai Signed-off-by: Selvin Xavier --- drivers/infiniband/hw/bnxt_re/qplib_rcfw.c | 46 +++++++++++++++++++----------- drivers/infiniband/hw/bnxt_re/qplib_rcfw.h | 1 + 2 files changed, 30 insertions(+), 17 deletions(-) diff --git a/drivers/infiniband/hw/bnxt_re/qplib_rcfw.c b/drivers/infiniband/hw/bnxt_re/qplib_rcfw.c index 3215f8a..e105961 100644 --- a/drivers/infiniband/hw/bnxt_re/qplib_rcfw.c +++ b/drivers/infiniband/hw/bnxt_re/qplib_rcfw.c @@ -216,6 +216,7 @@ static int __send_message(struct bnxt_qplib_rcfw *rcfw, crsqe->free_slots = free_slots; crsqe->resp = (struct creq_qp_event *)msg->resp; crsqe->resp->cookie = cpu_to_le16(cookie); + crsqe->is_waiter_alive = true; crsqe->req_size = __get_cmdq_base_cmd_size(msg->req, msg->req_sz); if (__get_cmdq_base_resp_size(msg->req, msg->req_sz) && msg->sb) { struct bnxt_qplib_rcfw_sbuf *sbuf = msg->sb; @@ -347,7 +348,9 @@ static int __bnxt_qplib_rcfw_send_message(struct bnxt_qplib_rcfw *rcfw, struct bnxt_qplib_cmdqmsg *msg) { struct creq_qp_event *evnt = (struct creq_qp_event *)msg->resp; - u16 cookie; + struct bnxt_qplib_crsqe *crsqe; + unsigned long flags; + u16 cookie, cbit; int rc = 0; u8 opcode; @@ -363,6 +366,7 @@ static int __bnxt_qplib_rcfw_send_message(struct bnxt_qplib_rcfw *rcfw, cookie = le16_to_cpu(__get_cmdq_base_cookie(msg->req, msg->req_sz)) & RCFW_MAX_COOKIE_VALUE; + cbit = cookie % rcfw->cmdq_depth; if (msg->block) rc = __block_for_resp(rcfw, cookie, opcode); @@ -378,6 +382,14 @@ static int __bnxt_qplib_rcfw_send_message(struct bnxt_qplib_rcfw *rcfw, return rc; } + if (rc) { + spin_lock_irqsave(&rcfw->cmdq.hwq.lock, flags); + crsqe = &rcfw->crsqe_tbl[cbit]; + crsqe->is_waiter_alive = false; + spin_unlock_irqrestore(&rcfw->cmdq.hwq.lock, flags); + return -ETIMEDOUT; + } + if (evnt->status) { /* failed with status */ dev_err(&rcfw->pdev->dev, "cmdq[%#x]=%#x status %#x\n", @@ -480,15 +492,16 @@ static int bnxt_qplib_process_qp_event(struct bnxt_qplib_rcfw *rcfw, struct creq_qp_error_notification *err_event; struct bnxt_qplib_hwq *hwq = &rcfw->cmdq.hwq; struct bnxt_qplib_crsqe *crsqe; + u32 qp_id, tbl_indx, req_size; struct bnxt_qplib_qp *qp; u16 cbit, blocked = 0; + bool is_waiter_alive; struct pci_dev *pdev; unsigned long flags; u32 wait_cmds = 0; __le16 mcookie; u16 cookie; int rc = 0; - u32 qp_id, tbl_indx; pdev = rcfw->pdev; switch (qp_event->event) { @@ -525,26 +538,25 @@ static int bnxt_qplib_process_qp_event(struct bnxt_qplib_rcfw *rcfw, cookie &= RCFW_MAX_COOKIE_VALUE; cbit = cookie % rcfw->cmdq_depth; crsqe = &rcfw->crsqe_tbl[cbit]; - if (crsqe->resp && - crsqe->resp->cookie == mcookie) { - memcpy(crsqe->resp, qp_event, sizeof(*qp_event)); - crsqe->resp = NULL; - } else { - if (crsqe->resp && crsqe->resp->cookie) - dev_err(&pdev->dev, - "CMD %s cookie sent=%#x, recd=%#x\n", - crsqe->resp ? "mismatch" : "collision", - crsqe->resp ? crsqe->resp->cookie : 0, - mcookie); - } if (!test_and_clear_bit(cbit, rcfw->cmdq.cmdq_bitmap)) dev_warn(&pdev->dev, "CMD bit %d was not requested\n", cbit); - hwq->cons += crsqe->req_size; + + if (crsqe->is_waiter_alive) { + if (crsqe->resp) + memcpy(crsqe->resp, qp_event, sizeof(*qp_event)); + if (!blocked) + wait_cmds++; + } + + req_size = crsqe->req_size; + is_waiter_alive = crsqe->is_waiter_alive; + crsqe->req_size = 0; + if (!is_waiter_alive) + crsqe->resp = NULL; - if (!blocked) - wait_cmds++; + hwq->cons += req_size; spin_unlock_irqrestore(&hwq->lock, flags); } *num_wait += wait_cmds; diff --git a/drivers/infiniband/hw/bnxt_re/qplib_rcfw.h b/drivers/infiniband/hw/bnxt_re/qplib_rcfw.h index 089e616..6ed81c1 100644 --- a/drivers/infiniband/hw/bnxt_re/qplib_rcfw.h +++ b/drivers/infiniband/hw/bnxt_re/qplib_rcfw.h @@ -152,6 +152,7 @@ struct bnxt_qplib_crsqe { u32 req_size; /* Free slots at the time of submission */ u32 free_slots; + bool is_waiter_alive; }; struct bnxt_qplib_rcfw_sbuf { From patchwork Thu Jun 8 10:25:02 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Selvin Xavier X-Patchwork-Id: 13271929 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id F401BC7EE25 for ; Thu, 8 Jun 2023 10:37:43 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236066AbjFHKhm (ORCPT ); Thu, 8 Jun 2023 06:37:42 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50078 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235171AbjFHKhj (ORCPT ); Thu, 8 Jun 2023 06:37:39 -0400 Received: from mail-pl1-x62f.google.com (mail-pl1-x62f.google.com [IPv6:2607:f8b0:4864:20::62f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4FAE92D79 for ; Thu, 8 Jun 2023 03:37:27 -0700 (PDT) Received: by mail-pl1-x62f.google.com with SMTP id d9443c01a7336-1b01d912924so2778685ad.1 for ; Thu, 08 Jun 2023 03:37:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; t=1686220647; x=1688812647; h=references:in-reply-to:message-id:date:subject:cc:to:from:from:to :cc:subject:date:message-id:reply-to; bh=Rp5SH4mx4LQQb0t4McS9lJO7HHlITxCh8MgqQP63uUw=; b=c1wc4yxcyPCSAFM7PjJ4gICg8NGdmaWnh+3hvr8zFTtJ2GyrzDWrZFew4u846X61xb ezJZvfvB9uiauuiCXK8C4/AHdxv//ro+Kg67u/5Jcrd6diS4SkcadVVW3aX1yAafRGVg ujvkffoGv3x2fGFDeOTFaxVaUyrPBPzIgNvjw= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1686220647; x=1688812647; h=references:in-reply-to:message-id:date:subject:cc:to:from :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Rp5SH4mx4LQQb0t4McS9lJO7HHlITxCh8MgqQP63uUw=; b=NuwY/d6k6iY7kGl3RoJXj5Q2Q5+3XiAic0soNKcfMSdrEfmQHfp+GaMZ6UVgstqqVF xALvyrRwjfUKw979p0l1K13j/tOSccEAB90/+TKkOhytmH7GrPlCxHOp9jUZwkYfpb1y z10gCeHcQ12DCYCPVRVy1lngJVEycnvVD4oGvl5mOgPl907HVlI6uIkP1zXVWXpgziY2 rGvOpqQx/LmfX4ruKhn7OMnjq7j93v35kESXZpty0qR7FcJakxIyLYrSt6MG1c7yDZ06 6iKZGLAhEtkh3RBFVZtqHZ10zzlV7CDNqSXtDc+X0n8G//dXppJcmprRoFhC/dgy9vUL bQ0Q== X-Gm-Message-State: AC+VfDwTCOGuyVaQr4mnGjqTamq9QHbq9T+lKYTKlpACRD1HQupxTrJ/ E0dPDCk7MQm/H0LFRDESHhdcNN+sWcuwDGm4bYk= X-Google-Smtp-Source: ACHHUZ7QcB7jwue8DBKbmygnqfFCcHDlcbI28cNZkpZM1VsWizQVl20eLDV5Unk91YwPW/x35IjRnw== X-Received: by 2002:a17:903:22c9:b0:1ac:4a41:d38d with SMTP id y9-20020a17090322c900b001ac4a41d38dmr9033716plg.51.1686220646660; Thu, 08 Jun 2023 03:37:26 -0700 (PDT) Received: from dhcp-10-192-206-197.iig.avagotech.net.net ([192.19.234.250]) by smtp.gmail.com with ESMTPSA id jj11-20020a170903048b00b001a980a23802sm1128510plb.111.2023.06.08.03.37.24 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 08 Jun 2023 03:37:25 -0700 (PDT) From: Selvin Xavier To: jgg@ziepe.ca, leon@kernel.org Cc: linux-rdma@vger.kernel.org, andrew.gospodarek@broadcom.com, Kashyap Desai , Selvin Xavier Subject: [PATCH for-next 11/17] RDMA/bnxt_re: Add firmware stall check detection Date: Thu, 8 Jun 2023 03:25:02 -0700 Message-Id: <1686219908-11181-12-git-send-email-selvin.xavier@broadcom.com> X-Mailer: git-send-email 2.5.5 In-Reply-To: <1686219908-11181-1-git-send-email-selvin.xavier@broadcom.com> References: <1686219908-11181-1-git-send-email-selvin.xavier@broadcom.com> Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org From: Kashyap Desai Every completion will update last_seen value in the unit of jiffies. last_seen field will be used to know if firmware is alive and is useful to detect firmware stall. Non blocking interface __wait_for_resp will have logic to detect firmware stall. After every 10 second interval if __wait_for_resp has not received completion for a given command it will check for firmware stall condition. If current jiffies is greater than last_seen jiffies + RCFW_FW_STALL_TIMEOUT_SEC * HZ, it is a firmware stall. Signed-off-by: Kashyap Desai Signed-off-by: Selvin Xavier --- drivers/infiniband/hw/bnxt_re/qplib_rcfw.c | 42 +++++++++++++++++++++++++----- drivers/infiniband/hw/bnxt_re/qplib_rcfw.h | 5 ++-- 2 files changed, 38 insertions(+), 9 deletions(-) diff --git a/drivers/infiniband/hw/bnxt_re/qplib_rcfw.c b/drivers/infiniband/hw/bnxt_re/qplib_rcfw.c index e105961..4d208ac 100644 --- a/drivers/infiniband/hw/bnxt_re/qplib_rcfw.c +++ b/drivers/infiniband/hw/bnxt_re/qplib_rcfw.c @@ -113,11 +113,14 @@ static int __wait_for_resp(struct bnxt_qplib_rcfw *rcfw, u16 cookie, u8 opcode) do { if (test_bit(ERR_DEVICE_DETACHED, &cmdq->flags)) return bnxt_qplib_map_rc(opcode); + if (test_bit(FIRMWARE_STALL_DETECTED, &cmdq->flags)) + return -ETIMEDOUT; /* Non zero means command completed */ ret = wait_event_timeout(cmdq->waitq, !test_bit(cbit, cmdq->cmdq_bitmap), - msecs_to_jiffies(10000)); + msecs_to_jiffies(RCFW_FW_STALL_TIMEOUT_SEC + * 1000)); if (!test_bit(cbit, cmdq->cmdq_bitmap)) return 0; @@ -127,6 +130,11 @@ static int __wait_for_resp(struct bnxt_qplib_rcfw *rcfw, u16 cookie, u8 opcode) if (!test_bit(cbit, cmdq->cmdq_bitmap)) return 0; + /* Firmware stall is detected */ + if (time_after(jiffies, cmdq->last_seen + + (RCFW_FW_STALL_TIMEOUT_SEC * HZ))) + return -ENODEV; + } while (true); }; @@ -155,6 +163,8 @@ static int __block_for_resp(struct bnxt_qplib_rcfw *rcfw, u16 cookie, u8 opcode) do { if (test_bit(ERR_DEVICE_DETACHED, &cmdq->flags)) return bnxt_qplib_map_rc(opcode); + if (test_bit(FIRMWARE_STALL_DETECTED, &cmdq->flags)) + return -ETIMEDOUT; udelay(1); @@ -187,9 +197,6 @@ static int __send_message(struct bnxt_qplib_rcfw *rcfw, opcode = __get_cmdq_base_opcode(msg->req, msg->req_sz); - if (test_bit(FIRMWARE_TIMED_OUT, &cmdq->flags)) - return -ETIMEDOUT; - /* Cmdq are in 16-byte units, each request can consume 1 or more * cmdqe */ @@ -288,14 +295,21 @@ static int __poll_for_resp(struct bnxt_qplib_rcfw *rcfw, u16 cookie, do { if (test_bit(ERR_DEVICE_DETACHED, &cmdq->flags)) return bnxt_qplib_map_rc(opcode); + if (test_bit(FIRMWARE_STALL_DETECTED, &cmdq->flags)) + return -ETIMEDOUT; usleep_range(1000, 1001); bnxt_qplib_service_creq(&rcfw->creq.creq_tasklet); if (!test_bit(cbit, cmdq->cmdq_bitmap)) return 0; - if (jiffies_to_msecs(jiffies - issue_time) > 10000) - return -ETIMEDOUT; + if (jiffies_to_msecs(jiffies - issue_time) > + (RCFW_FW_STALL_TIMEOUT_SEC * 1000)) { + /* Firmware stall is detected */ + if (time_after(jiffies, cmdq->last_seen + + (RCFW_FW_STALL_TIMEOUT_SEC * HZ))) + return -ENODEV; + } } while (true); }; @@ -311,6 +325,8 @@ static int __send_message_basic_sanity(struct bnxt_qplib_rcfw *rcfw, /* Prevent posting if f/w is not in a state to process */ if (test_bit(ERR_DEVICE_DETACHED, &rcfw->cmdq.flags)) return -ENXIO; + if (test_bit(FIRMWARE_STALL_DETECTED, &cmdq->flags)) + return -ETIMEDOUT; if (test_bit(FIRMWARE_INITIALIZED_FLAG, &cmdq->flags) && opcode == CMDQ_BASE_OPCODE_INITIALIZE_FW) { @@ -378,7 +394,6 @@ static int __bnxt_qplib_rcfw_send_message(struct bnxt_qplib_rcfw *rcfw, /* timed out */ dev_err(&rcfw->pdev->dev, "cmdq[%#x]=%#x timedout (%d)msec\n", cookie, opcode, RCFW_CMD_WAIT_TIME_MS); - set_bit(FIRMWARE_TIMED_OUT, &rcfw->cmdq.flags); return rc; } @@ -386,6 +401,8 @@ static int __bnxt_qplib_rcfw_send_message(struct bnxt_qplib_rcfw *rcfw, spin_lock_irqsave(&rcfw->cmdq.hwq.lock, flags); crsqe = &rcfw->crsqe_tbl[cbit]; crsqe->is_waiter_alive = false; + if (rc == -ENODEV) + set_bit(FIRMWARE_STALL_DETECTED, &rcfw->cmdq.flags); spin_unlock_irqrestore(&rcfw->cmdq.hwq.lock, flags); return -ETIMEDOUT; } @@ -538,6 +555,16 @@ static int bnxt_qplib_process_qp_event(struct bnxt_qplib_rcfw *rcfw, cookie &= RCFW_MAX_COOKIE_VALUE; cbit = cookie % rcfw->cmdq_depth; crsqe = &rcfw->crsqe_tbl[cbit]; + + if (WARN_ONCE(test_bit(FIRMWARE_STALL_DETECTED, + &rcfw->cmdq.flags), + "QPLIB: Unreponsive rcfw channel detected.!!")) { + dev_info(&pdev->dev, "rcfw timedout: cookie = %#x," + " free_slots = %d", cookie, crsqe->free_slots); + spin_unlock_irqrestore(&hwq->lock, flags); + return rc; + } + if (!test_and_clear_bit(cbit, rcfw->cmdq.cmdq_bitmap)) dev_warn(&pdev->dev, "CMD bit %d was not requested\n", cbit); @@ -587,6 +614,7 @@ static void bnxt_qplib_service_creq(struct tasklet_struct *t) * reading any further. */ dma_rmb(); + rcfw->cmdq.last_seen = jiffies; type = creqe->type & CREQ_BASE_TYPE_MASK; switch (type) { diff --git a/drivers/infiniband/hw/bnxt_re/qplib_rcfw.h b/drivers/infiniband/hw/bnxt_re/qplib_rcfw.h index 6ed81c1..54576f1 100644 --- a/drivers/infiniband/hw/bnxt_re/qplib_rcfw.h +++ b/drivers/infiniband/hw/bnxt_re/qplib_rcfw.h @@ -51,6 +51,7 @@ #define RCFW_DBR_PCI_BAR_REGION 2 #define RCFW_DBR_BASE_PAGE_SHIFT 12 +#define RCFW_FW_STALL_TIMEOUT_SEC 40 /* Cmdq contains a fix number of a 16-Byte slots */ struct bnxt_qplib_cmdqe { @@ -128,7 +129,6 @@ static inline u32 bnxt_qplib_set_cmd_slots(struct cmdq_base *req) #define RCFW_MAX_COOKIE_VALUE (BNXT_QPLIB_CMDQE_MAX_CNT - 1) #define RCFW_CMD_IS_BLOCKING 0x8000 -#define RCFW_BLOCKED_CMD_WAIT_COUNT 20000000UL /* 20 sec */ /* Crsq buf is 1024-Byte */ struct bnxt_qplib_crsbe { @@ -170,7 +170,7 @@ struct bnxt_qplib_qp_node { #define FIRMWARE_INITIALIZED_FLAG (0) #define FIRMWARE_FIRST_FLAG (31) -#define FIRMWARE_TIMED_OUT (3) +#define FIRMWARE_STALL_DETECTED (3) #define ERR_DEVICE_DETACHED (4) struct bnxt_qplib_cmdq_mbox { @@ -185,6 +185,7 @@ struct bnxt_qplib_cmdq_ctx { wait_queue_head_t waitq; unsigned long flags; unsigned long *cmdq_bitmap; + unsigned long last_seen; u32 seq_num; }; From patchwork Thu Jun 8 10:25:03 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Selvin Xavier X-Patchwork-Id: 13271930 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E7F84C7EE23 for ; Thu, 8 Jun 2023 10:37:47 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234380AbjFHKhq (ORCPT ); Thu, 8 Jun 2023 06:37:46 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50022 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234388AbjFHKhp (ORCPT ); Thu, 8 Jun 2023 06:37:45 -0400 Received: from mail-pl1-x633.google.com (mail-pl1-x633.google.com [IPv6:2607:f8b0:4864:20::633]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1ED152D42 for ; Thu, 8 Jun 2023 03:37:30 -0700 (PDT) Received: by mail-pl1-x633.google.com with SMTP id d9443c01a7336-1b24eba184bso1792525ad.0 for ; Thu, 08 Jun 2023 03:37:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; t=1686220649; x=1688812649; h=references:in-reply-to:message-id:date:subject:cc:to:from:from:to :cc:subject:date:message-id:reply-to; bh=bkr4h3jmbZEKUu1ORuVEewoKSGIQpNT88aUTHIfD7Gc=; b=XT3tC3DG9BVQffRrsRCkV/ouWFlZi+zxF+i8uplUPYDwKPexi2G6wroWohfCi/7ntK qMFrQ02lpY13Pucf2BpwkXbpQz7getC+hq9s/dvXL2pXzmi1DggvOrElgtY8urUmCoX/ wT6dFW/8MpWa8IIpyO/uucssRg+xbhxuGtvXs= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1686220649; x=1688812649; h=references:in-reply-to:message-id:date:subject:cc:to:from :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=bkr4h3jmbZEKUu1ORuVEewoKSGIQpNT88aUTHIfD7Gc=; b=cgmmLtqE3ujPzZStAZdIjlzuD3BRHVXZZerNeYyMvBwKrUFAI3l+7y89vJqo0NGxra 95ncGXUGzYKbCcy0vSy7G/XXQgxEPiPE2zOQPRa+tq9RbvbZFC4LjLPkSAikxKCqYeB7 fCBf1EEk+zOGcOnuwf5Pv4jDegzJ/fcUqLPu0InUGWCrV15S1Z6sJQxiqkXwbD+l5l8G XxWS5nfcfEoqXgcrPGzxT5kkIvx4SEuy15XjAOvRiYXO8ZnW8ojS9dYUfHedWQQ8JR7b IcpiIzIVJ0DlxPqgTPC1RjB5J0bhTx3ThNltz669KdBtxrP6WXU6yxDt1SwKMDA/oeU3 os7Q== X-Gm-Message-State: AC+VfDx2BBKUip4XtrQuHRa+0kD6G85tC3jG1o9U0iDPpqF43GaUYlzJ RSRDuuwo3QpyBZVW6CGrS9dv3w== X-Google-Smtp-Source: ACHHUZ5If6NwLXh43coNDS0nKDP6OWr0/1Zve52rjTIl5KIUaCiTgIl25Pqb668hS+863SMGPwAvLg== X-Received: by 2002:a17:902:cecb:b0:1b0:3b07:705b with SMTP id d11-20020a170902cecb00b001b03b07705bmr3860600plg.7.1686220649428; Thu, 08 Jun 2023 03:37:29 -0700 (PDT) Received: from dhcp-10-192-206-197.iig.avagotech.net.net ([192.19.234.250]) by smtp.gmail.com with ESMTPSA id jj11-20020a170903048b00b001a980a23802sm1128510plb.111.2023.06.08.03.37.27 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 08 Jun 2023 03:37:28 -0700 (PDT) From: Selvin Xavier To: jgg@ziepe.ca, leon@kernel.org Cc: linux-rdma@vger.kernel.org, andrew.gospodarek@broadcom.com, Kashyap Desai , Selvin Xavier Subject: [PATCH for-next 12/17] RDMA/bnxt_re: post destroy_ah for delayed completion of AH creation Date: Thu, 8 Jun 2023 03:25:03 -0700 Message-Id: <1686219908-11181-13-git-send-email-selvin.xavier@broadcom.com> X-Mailer: git-send-email 2.5.5 In-Reply-To: <1686219908-11181-1-git-send-email-selvin.xavier@broadcom.com> References: <1686219908-11181-1-git-send-email-selvin.xavier@broadcom.com> Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org From: Kashyap Desai AH create may be called from interrpt context and driver has a special timeout (8 sec) for this command. This is to avoid soft lockups when the FW command takes more time. Driver returns -ETIMEOUT and fail create AH, without waiting for actual completion from firmware. When FW completion is received, use is_waiter_alive flag to avoid a regular completion path. If create_ah opcode is detected in completion path which does not have waiter alive, driver will fetch ah_id from successful firmware completion in the interrupt context and sends destroy_ah command for same ah_id. This special post is done in quick manner using helper function __send_message_no_waiter. timeout_send is only used for debugging purposes. If timeout_send value keeps incrementing, it indicates out of sync active ah counter between driver and firmware. This is a limitation but graceful handling is possible in future. Signed-off-by: Kashyap Desai Signed-off-by: Selvin Xavier --- drivers/infiniband/hw/bnxt_re/qplib_rcfw.c | 108 +++++++++++++++++++++++++++++ drivers/infiniband/hw/bnxt_re/qplib_rcfw.h | 2 + 2 files changed, 110 insertions(+) diff --git a/drivers/infiniband/hw/bnxt_re/qplib_rcfw.c b/drivers/infiniband/hw/bnxt_re/qplib_rcfw.c index 4d208ac..3461e3b 100644 --- a/drivers/infiniband/hw/bnxt_re/qplib_rcfw.c +++ b/drivers/infiniband/hw/bnxt_re/qplib_rcfw.c @@ -177,6 +177,73 @@ static int __block_for_resp(struct bnxt_qplib_rcfw *rcfw, u16 cookie, u8 opcode) return -ETIMEDOUT; }; +/* __send_message_no_waiter - get cookie and post the message. + * @rcfw - rcfw channel instance of rdev + * @msg - qplib message internal + * + * This function will just post and don't bother about completion. + * Current design of this function is - + * user must hold the completion queue hwq->lock. + * user must have used existing completion and free the resources. + * this function will not check queue full condition. + * this function will explicitly set is_waiter_alive=false. + * current use case is - send destroy_ah if create_ah is return + * after waiter of create_ah is lost. It can be extended for other + * use case as well. + * + * Returns: Nothing + * + */ +static void __send_message_no_waiter(struct bnxt_qplib_rcfw *rcfw, + struct bnxt_qplib_cmdqmsg *msg) +{ + struct bnxt_qplib_cmdq_ctx *cmdq = &rcfw->cmdq; + struct bnxt_qplib_hwq *hwq = &cmdq->hwq; + struct bnxt_qplib_crsqe *crsqe; + struct bnxt_qplib_cmdqe *cmdqe; + u32 sw_prod, cmdq_prod; + u16 cookie, cbit; + u32 bsize; + u8 *preq; + + cookie = cmdq->seq_num & RCFW_MAX_COOKIE_VALUE; + cbit = cookie % rcfw->cmdq_depth; + + set_bit(cbit, cmdq->cmdq_bitmap); + __set_cmdq_base_cookie(msg->req, msg->req_sz, cpu_to_le16(cookie)); + crsqe = &rcfw->crsqe_tbl[cbit]; + + /* Set cmd_size in terms of 16B slots in req. */ + bsize = bnxt_qplib_set_cmd_slots(msg->req); + /* GET_CMD_SIZE would return number of slots in either case of tlv + * and non-tlv commands after call to bnxt_qplib_set_cmd_slots() + */ + crsqe->is_internal_cmd = true; + crsqe->is_waiter_alive = false; + crsqe->req_size = __get_cmdq_base_cmd_size(msg->req, msg->req_sz); + + preq = (u8 *)msg->req; + do { + /* Locate the next cmdq slot */ + sw_prod = HWQ_CMP(hwq->prod, hwq); + cmdqe = bnxt_qplib_get_qe(hwq, sw_prod, NULL); + /* Copy a segment of the req cmd to the cmdq */ + memset(cmdqe, 0, sizeof(*cmdqe)); + memcpy(cmdqe, preq, min_t(u32, bsize, sizeof(*cmdqe))); + preq += min_t(u32, bsize, sizeof(*cmdqe)); + bsize -= min_t(u32, bsize, sizeof(*cmdqe)); + hwq->prod++; + } while (bsize > 0); + cmdq->seq_num++; + + cmdq_prod = hwq->prod; + atomic_inc(&rcfw->timeout_send); + /* ring CMDQ DB */ + wmb(); + writel(cmdq_prod, cmdq->cmdq_mbox.prod); + writel(RCFW_CMDQ_TRIG_VAL, cmdq->cmdq_mbox.db); +} + static int __send_message(struct bnxt_qplib_rcfw *rcfw, struct bnxt_qplib_cmdqmsg *msg) { @@ -223,6 +290,7 @@ static int __send_message(struct bnxt_qplib_rcfw *rcfw, crsqe->free_slots = free_slots; crsqe->resp = (struct creq_qp_event *)msg->resp; crsqe->resp->cookie = cpu_to_le16(cookie); + crsqe->is_internal_cmd = false; crsqe->is_waiter_alive = true; crsqe->req_size = __get_cmdq_base_cmd_size(msg->req, msg->req_sz); if (__get_cmdq_base_resp_size(msg->req, msg->req_sz) && msg->sb) { @@ -347,6 +415,26 @@ static int __send_message_basic_sanity(struct bnxt_qplib_rcfw *rcfw, return 0; } +/* This function will just post and do not bother about completion */ +static void __destroy_timedout_ah(struct bnxt_qplib_rcfw *rcfw, + struct creq_create_ah_resp *create_ah_resp) +{ + struct bnxt_qplib_cmdqmsg msg = {}; + struct cmdq_destroy_ah req = {}; + + bnxt_qplib_rcfw_cmd_prep((struct cmdq_base *)&req, + CMDQ_BASE_OPCODE_DESTROY_AH, + sizeof(req)); + req.ah_cid = create_ah_resp->xid; + msg.req = (struct cmdq_base *)&req; + msg.req_sz = sizeof(req); + __send_message_no_waiter(rcfw, &msg); + dev_info_ratelimited(&rcfw->pdev->dev, + "From %s: ah_cid = %d timeout_send %d\n", + __func__, req.ah_cid, + atomic_read(&rcfw->timeout_send)); +} + /** * __bnxt_qplib_rcfw_send_message - qplib interface to send * and complete rcfw command. @@ -568,6 +656,8 @@ static int bnxt_qplib_process_qp_event(struct bnxt_qplib_rcfw *rcfw, if (!test_and_clear_bit(cbit, rcfw->cmdq.cmdq_bitmap)) dev_warn(&pdev->dev, "CMD bit %d was not requested\n", cbit); + if (crsqe->is_internal_cmd && !qp_event->status) + atomic_dec(&rcfw->timeout_send); if (crsqe->is_waiter_alive) { if (crsqe->resp) @@ -584,6 +674,24 @@ static int bnxt_qplib_process_qp_event(struct bnxt_qplib_rcfw *rcfw, crsqe->resp = NULL; hwq->cons += req_size; + + /* This is a case to handle below scenario - + * Create AH is completed successfully by firmware, + * but completion took more time and driver already lost + * the context of create_ah from caller. + * We have already return failure for create_ah verbs, + * so let's destroy the same address vector since it is + * no more used in stack. We don't care about completion + * in __send_message_no_waiter. + * If destroy_ah is failued by firmware, there will be AH + * resource leak and relatively not critical + unlikely + * scenario. Current design is not to handle such case. + */ + if (!is_waiter_alive && !qp_event->status && + qp_event->event == CREQ_QP_EVENT_EVENT_CREATE_AH) + __destroy_timedout_ah(rcfw, + (struct creq_create_ah_resp *) + qp_event); spin_unlock_irqrestore(&hwq->lock, flags); } *num_wait += wait_cmds; diff --git a/drivers/infiniband/hw/bnxt_re/qplib_rcfw.h b/drivers/infiniband/hw/bnxt_re/qplib_rcfw.h index 54576f1..338bf6a 100644 --- a/drivers/infiniband/hw/bnxt_re/qplib_rcfw.h +++ b/drivers/infiniband/hw/bnxt_re/qplib_rcfw.h @@ -153,6 +153,7 @@ struct bnxt_qplib_crsqe { /* Free slots at the time of submission */ u32 free_slots; bool is_waiter_alive; + bool is_internal_cmd; }; struct bnxt_qplib_rcfw_sbuf { @@ -225,6 +226,7 @@ struct bnxt_qplib_rcfw { u32 cmdq_depth; atomic_t rcfw_intr_enabled; struct semaphore rcfw_inflight; + atomic_t timeout_send; }; struct bnxt_qplib_cmdqmsg { From patchwork Thu Jun 8 10:25:04 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Selvin Xavier X-Patchwork-Id: 13271931 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E1A3DC7EE23 for ; Thu, 8 Jun 2023 10:37:50 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233806AbjFHKht (ORCPT ); Thu, 8 Jun 2023 06:37:49 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50290 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234388AbjFHKhs (ORCPT ); Thu, 8 Jun 2023 06:37:48 -0400 Received: from mail-pg1-x534.google.com (mail-pg1-x534.google.com [IPv6:2607:f8b0:4864:20::534]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B487F2D43 for ; Thu, 8 Jun 2023 03:37:32 -0700 (PDT) Received: by mail-pg1-x534.google.com with SMTP id 41be03b00d2f7-53fbb3a013dso234818a12.1 for ; Thu, 08 Jun 2023 03:37:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; t=1686220652; x=1688812652; h=references:in-reply-to:message-id:date:subject:cc:to:from:from:to :cc:subject:date:message-id:reply-to; bh=w6HW1Tve1A5DQfVVPzLnGqYQquumLqFDrnWccriu8+g=; b=ak/rdo7UJjUTKHZ1M9pnF+zxicw4sLU9N0pKV4GItpY6/xIpwH4Bba/s3wphvgfA5/ Sq1s+b1RuhMArsg1l8vjjEekTU0jOJBEXj9ZI9RXWCUhbV2jMmnWut4yklOwQOrNPOQm 6aAVSnTXKcnw/lYs1Yg+0sah/0ULg+rgO2ukM= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1686220652; x=1688812652; h=references:in-reply-to:message-id:date:subject:cc:to:from :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=w6HW1Tve1A5DQfVVPzLnGqYQquumLqFDrnWccriu8+g=; b=OIfEOfPMinUhe/RplRabHhUVMp1FMf6OrAWgUYYrnF9BKyP4WgQfd5mFmpx7GhwGvQ 5m0N68jcTpzoKt87AuihM3MoH5RDz3pgImFBl+2VQ41c99tKziFyo+5qkY68qhvrv6wZ TnLQWNKcFIa33l+Pzz7zIPUYBC+nqkK/BW12Axt1BKmUoUaqxwuxPzMrJoP+nF77f4qv YCG5Elx+m3JZ8lEO4C12wzATgvcLkrEZPYax1Vwu3i81IJqmCbPx9MdBQ56cxsEgIGnO IuTa9nXF5iFFHE6lesEU5za65PtlBb1OJTmd24RCjT0YJsR5SI5ZIQ3IaZOdcAgz6dBW zjaA== X-Gm-Message-State: AC+VfDzIHKUimwYkxuueHS0UyxjlNRLV+G23R7fPGeQhC1Mv0IDBHM6G 4pdnaYWps1ZUDEKof45otlljSQ== X-Google-Smtp-Source: ACHHUZ4PUp3sS3+oaZYQpnMn1LqPGGQI4W/TnHS1rRsfq+S0c8Xktx0rXVkb9WH/eRK48bhvsupQaQ== X-Received: by 2002:a05:6a20:1611:b0:10b:cb77:5403 with SMTP id l17-20020a056a20161100b0010bcb775403mr6992375pzj.52.1686220652130; Thu, 08 Jun 2023 03:37:32 -0700 (PDT) Received: from dhcp-10-192-206-197.iig.avagotech.net.net ([192.19.234.250]) by smtp.gmail.com with ESMTPSA id jj11-20020a170903048b00b001a980a23802sm1128510plb.111.2023.06.08.03.37.29 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 08 Jun 2023 03:37:31 -0700 (PDT) From: Selvin Xavier To: jgg@ziepe.ca, leon@kernel.org Cc: linux-rdma@vger.kernel.org, andrew.gospodarek@broadcom.com, Kashyap Desai , Selvin Xavier Subject: [PATCH for-next 13/17] RDMA/bnxt_re: consider timeout of destroy ah as success. Date: Thu, 8 Jun 2023 03:25:04 -0700 Message-Id: <1686219908-11181-14-git-send-email-selvin.xavier@broadcom.com> X-Mailer: git-send-email 2.5.5 In-Reply-To: <1686219908-11181-1-git-send-email-selvin.xavier@broadcom.com> References: <1686219908-11181-1-git-send-email-selvin.xavier@broadcom.com> Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org From: Kashyap Desai If destroy_ah is timed out, it is likely to be destroyed by firmware but it is taking longer time due to temporary slowness in processing the rcfw command. In worst case, there might be AH resource leak in firmware. Sending timeout return value can dump warning message from ib_core which can be avoided if we map timeout of destroy_ah as success. Signed-off-by: Kashyap Desai Signed-off-by: Selvin Xavier --- drivers/infiniband/hw/bnxt_re/bnxt_re.h | 2 ++ drivers/infiniband/hw/bnxt_re/ib_verbs.c | 16 ++++++++++++---- drivers/infiniband/hw/bnxt_re/qplib_sp.c | 8 +++++--- drivers/infiniband/hw/bnxt_re/qplib_sp.h | 4 ++-- 4 files changed, 21 insertions(+), 9 deletions(-) diff --git a/drivers/infiniband/hw/bnxt_re/bnxt_re.h b/drivers/infiniband/hw/bnxt_re/bnxt_re.h index 5a2baf4..80f198e 100644 --- a/drivers/infiniband/hw/bnxt_re/bnxt_re.h +++ b/drivers/infiniband/hw/bnxt_re/bnxt_re.h @@ -181,6 +181,8 @@ struct bnxt_re_dev { #define BNXT_RE_ROCEV2_IPV4_PACKET 2 #define BNXT_RE_ROCEV2_IPV6_PACKET 3 +#define BNXT_RE_CHECK_RC(x) ((x) && (x != -ETIMEDOUT)) + static inline struct device *rdev_to_dev(struct bnxt_re_dev *rdev) { if (rdev) diff --git a/drivers/infiniband/hw/bnxt_re/ib_verbs.c b/drivers/infiniband/hw/bnxt_re/ib_verbs.c index e86afec..053afc9 100644 --- a/drivers/infiniband/hw/bnxt_re/ib_verbs.c +++ b/drivers/infiniband/hw/bnxt_re/ib_verbs.c @@ -613,12 +613,20 @@ int bnxt_re_destroy_ah(struct ib_ah *ib_ah, u32 flags) { struct bnxt_re_ah *ah = container_of(ib_ah, struct bnxt_re_ah, ib_ah); struct bnxt_re_dev *rdev = ah->rdev; + bool block = true; + int rc = 0; - bnxt_qplib_destroy_ah(&rdev->qplib_res, &ah->qplib_ah, - !(flags & RDMA_DESTROY_AH_SLEEPABLE)); + block = !(flags & RDMA_DESTROY_AH_SLEEPABLE); + rc = bnxt_qplib_destroy_ah(&rdev->qplib_res, &ah->qplib_ah, block); + if (BNXT_RE_CHECK_RC(rc)) { + if (rc == -ETIMEDOUT) + rc = 0; + else + goto fail; + } atomic_dec(&rdev->ah_count); - - return 0; +fail: + return rc; } static u8 bnxt_re_stack_to_dev_nw_type(enum rdma_network_type ntype) diff --git a/drivers/infiniband/hw/bnxt_re/qplib_sp.c b/drivers/infiniband/hw/bnxt_re/qplib_sp.c index dbb0e4e..910d17d 100644 --- a/drivers/infiniband/hw/bnxt_re/qplib_sp.c +++ b/drivers/infiniband/hw/bnxt_re/qplib_sp.c @@ -460,13 +460,14 @@ int bnxt_qplib_create_ah(struct bnxt_qplib_res *res, struct bnxt_qplib_ah *ah, return 0; } -void bnxt_qplib_destroy_ah(struct bnxt_qplib_res *res, struct bnxt_qplib_ah *ah, - bool block) +int bnxt_qplib_destroy_ah(struct bnxt_qplib_res *res, struct bnxt_qplib_ah *ah, + bool block) { struct bnxt_qplib_rcfw *rcfw = res->rcfw; struct creq_destroy_ah_resp resp = {}; struct bnxt_qplib_cmdqmsg msg = {}; struct cmdq_destroy_ah req = {}; + int rc; /* Clean up the AH table in the device */ bnxt_qplib_rcfw_cmd_prep((struct cmdq_base *)&req, @@ -477,7 +478,8 @@ void bnxt_qplib_destroy_ah(struct bnxt_qplib_res *res, struct bnxt_qplib_ah *ah, bnxt_qplib_fill_cmdqmsg(&msg, &req, &resp, NULL, sizeof(req), sizeof(resp), block); - bnxt_qplib_rcfw_send_message(rcfw, &msg); + rc = bnxt_qplib_rcfw_send_message(rcfw, &msg); + return rc; } /* MRW */ diff --git a/drivers/infiniband/hw/bnxt_re/qplib_sp.h b/drivers/infiniband/hw/bnxt_re/qplib_sp.h index 5de87465..4061616 100644 --- a/drivers/infiniband/hw/bnxt_re/qplib_sp.h +++ b/drivers/infiniband/hw/bnxt_re/qplib_sp.h @@ -327,8 +327,8 @@ int bnxt_qplib_set_func_resources(struct bnxt_qplib_res *res, struct bnxt_qplib_ctx *ctx); int bnxt_qplib_create_ah(struct bnxt_qplib_res *res, struct bnxt_qplib_ah *ah, bool block); -void bnxt_qplib_destroy_ah(struct bnxt_qplib_res *res, struct bnxt_qplib_ah *ah, - bool block); +int bnxt_qplib_destroy_ah(struct bnxt_qplib_res *res, struct bnxt_qplib_ah *ah, + bool block); int bnxt_qplib_alloc_mrw(struct bnxt_qplib_res *res, struct bnxt_qplib_mrw *mrw); int bnxt_qplib_dereg_mrw(struct bnxt_qplib_res *res, struct bnxt_qplib_mrw *mrw, From patchwork Thu Jun 8 10:25:05 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Selvin Xavier X-Patchwork-Id: 13271932 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 285CDC7EE43 for ; Thu, 8 Jun 2023 10:37:53 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234388AbjFHKhv (ORCPT ); Thu, 8 Jun 2023 06:37:51 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50348 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235253AbjFHKhu (ORCPT ); Thu, 8 Jun 2023 06:37:50 -0400 Received: from mail-pl1-x62e.google.com (mail-pl1-x62e.google.com [IPv6:2607:f8b0:4864:20::62e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6897230C7 for ; Thu, 8 Jun 2023 03:37:35 -0700 (PDT) Received: by mail-pl1-x62e.google.com with SMTP id d9443c01a7336-1b02d0942caso1640995ad.1 for ; Thu, 08 Jun 2023 03:37:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; t=1686220655; x=1688812655; h=references:in-reply-to:message-id:date:subject:cc:to:from:from:to :cc:subject:date:message-id:reply-to; bh=xXNBUhvWKZ2gdj45Hr9XWCQfaApid+t7wJQiKNdg72k=; b=DT7H4ZQUQ91L8b+kxRceq/1nDNJSpo7Xz/fx6Jxef6XuS0Fm3VUYG3xQ7HyNou51HZ z9dk99Z8/tgsKxivNvw/oU07paz+hJDrp77RYBXLCeEiR1UvOowlA3IaWUdUdmWsHGhZ nkOv5sdjkHNyuCCefIqgZQSc1GRA62zz/JBSc= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1686220655; x=1688812655; h=references:in-reply-to:message-id:date:subject:cc:to:from :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=xXNBUhvWKZ2gdj45Hr9XWCQfaApid+t7wJQiKNdg72k=; b=LdGkBBcrKk43n74H70g9O1BCVBkM3zUv9U21Motsm1pPxKg3F6+F8BsXi8tgGp8jkw c8a6eEhFPFzIlqny6zqFzxU0fQEq1zmj+AJEX9gUHQv64iDiauSzsAa0/jwzh+2W8Hk7 Mm9K2KsziHYJ9qnX174b68tWdtrLp6XDkmnWBQ2zkz+7yC0OjNs05lyjVTIKjs7K6aFx 3mNpXLg289QJBG79gWPlB91QbOky9DM8P9uUqLJaU9mWs7gPziQEehGjHbwllR5E+8hX 6Zg5CCqIk1Ed4oxs3s1+6RZaQFscRROU4gdzOUsdEu3T1tz6QsbnGd0gp2NFPVt5JJug Hy+A== X-Gm-Message-State: AC+VfDzQQ8cNImpyU1cJK1NOYTw0r/xKMy2b8VU18dY6Yxfj75I68Kyq GBxOoeTUIb9H93gYc32yKLxp6A== X-Google-Smtp-Source: ACHHUZ7/MYkTRGCtv6fSgl+ZnFotHIIHiQ1f59049iM/1h58l9sxSgbJ522KwIty36YC8DJQENhuyw== X-Received: by 2002:a17:903:1251:b0:1b0:4527:3716 with SMTP id u17-20020a170903125100b001b045273716mr4905441plh.15.1686220654813; Thu, 08 Jun 2023 03:37:34 -0700 (PDT) Received: from dhcp-10-192-206-197.iig.avagotech.net.net ([192.19.234.250]) by smtp.gmail.com with ESMTPSA id jj11-20020a170903048b00b001a980a23802sm1128510plb.111.2023.06.08.03.37.32 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 08 Jun 2023 03:37:34 -0700 (PDT) From: Selvin Xavier To: jgg@ziepe.ca, leon@kernel.org Cc: linux-rdma@vger.kernel.org, andrew.gospodarek@broadcom.com, Kashyap Desai , Selvin Xavier Subject: [PATCH for-next 14/17] RDMA/bnxt_re: cancel all control path command waiters upon error. Date: Thu, 8 Jun 2023 03:25:05 -0700 Message-Id: <1686219908-11181-15-git-send-email-selvin.xavier@broadcom.com> X-Mailer: git-send-email 2.5.5 In-Reply-To: <1686219908-11181-1-git-send-email-selvin.xavier@broadcom.com> References: <1686219908-11181-1-git-send-email-selvin.xavier@broadcom.com> Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org From: Kashyap Desai When an error is detected in FW, wake up all the waiters as the all of them need to be completed with timeout. Add the device error state also as a wait condition. Signed-off-by: Kashyap Desai Signed-off-by: Selvin Xavier --- drivers/infiniband/hw/bnxt_re/main.c | 1 + drivers/infiniband/hw/bnxt_re/qplib_rcfw.c | 3 ++- 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/drivers/infiniband/hw/bnxt_re/main.c b/drivers/infiniband/hw/bnxt_re/main.c index 51372de..8241154 100644 --- a/drivers/infiniband/hw/bnxt_re/main.c +++ b/drivers/infiniband/hw/bnxt_re/main.c @@ -1495,6 +1495,7 @@ static int bnxt_re_suspend(struct auxiliary_device *adev, pm_message_t state) */ set_bit(BNXT_RE_FLAG_ERR_DEVICE_DETACHED, &rdev->flags); set_bit(ERR_DEVICE_DETACHED, &rdev->rcfw.cmdq.flags); + wake_up_all(&rdev->rcfw.cmdq.waitq); mutex_unlock(&bnxt_re_mutex); return 0; diff --git a/drivers/infiniband/hw/bnxt_re/qplib_rcfw.c b/drivers/infiniband/hw/bnxt_re/qplib_rcfw.c index 3461e3b..4f36644 100644 --- a/drivers/infiniband/hw/bnxt_re/qplib_rcfw.c +++ b/drivers/infiniband/hw/bnxt_re/qplib_rcfw.c @@ -118,7 +118,8 @@ static int __wait_for_resp(struct bnxt_qplib_rcfw *rcfw, u16 cookie, u8 opcode) /* Non zero means command completed */ ret = wait_event_timeout(cmdq->waitq, - !test_bit(cbit, cmdq->cmdq_bitmap), + !test_bit(cbit, cmdq->cmdq_bitmap) || + test_bit(ERR_DEVICE_DETACHED, &cmdq->flags), msecs_to_jiffies(RCFW_FW_STALL_TIMEOUT_SEC * 1000)); From patchwork Thu Jun 8 10:25:06 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Selvin Xavier X-Patchwork-Id: 13271933 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C30C7C7EE25 for ; Thu, 8 Jun 2023 10:37:55 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235171AbjFHKhy (ORCPT ); Thu, 8 Jun 2023 06:37:54 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50408 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235059AbjFHKhx (ORCPT ); Thu, 8 Jun 2023 06:37:53 -0400 Received: from mail-pl1-x62d.google.com (mail-pl1-x62d.google.com [IPv6:2607:f8b0:4864:20::62d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4AE272D6D for ; Thu, 8 Jun 2023 03:37:38 -0700 (PDT) Received: by mail-pl1-x62d.google.com with SMTP id d9443c01a7336-1b0314f057cso2010775ad.1 for ; Thu, 08 Jun 2023 03:37:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; t=1686220658; x=1688812658; h=references:in-reply-to:message-id:date:subject:cc:to:from:from:to :cc:subject:date:message-id:reply-to; bh=Ww0q9fpRyaBvzYRhuPEEV+DRtPaxNTZDMxjiruNuri4=; b=aPcrki9O2+q1VyNQzTBpo53aRvHmhHBFgp/Nqd1Pdl4nIeLv/k4HFqa+KdldHUKiPA u9pfX2pd6/iPEtV08pcz+4qcYg3Me3ho7tR5ZGnSPxAV8nNj8nHd9OHeO34ksP6i91AB aHyjkU3QGNWsh4d2kuuS7qxXrJUJDsKwwbQRc= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1686220658; x=1688812658; h=references:in-reply-to:message-id:date:subject:cc:to:from :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Ww0q9fpRyaBvzYRhuPEEV+DRtPaxNTZDMxjiruNuri4=; b=EQXZz4qUDksk8LAUeQQDz+ssw78Xt3WtqYgGsY0Ai6EIISzTbbzHCx1I0Wsm82431i 6yWZDmJb8r7sB7nLqEo6voTJGpI/TAqdgyxiGa38Wx/kM8dr5YU0O85KOvOTdgwCNJfx DLV4zW2JcNCLTPsxBygxoPErDB7SfvVMg49TUNYjaa3YmeTf3gCXYjB+LmVPa5y7nV2q f3ECq7HXCdn6KLTRWlopBGZOQVMEgZmki7ESgw2WkVc1pWDxAJJdtpDmrfQ4lE9uY+YA pkID0eWE7Md6PJwjiDQJR0EHtGDSZVLaxXRYOUmmOlgiqeOUMPpvYeZ90czC92Xn58dr nmUw== X-Gm-Message-State: AC+VfDwDXBR3AqwaZ5rBaWGyAqgMHJog09Nq/chnaTjbgoyial2UufRI u13kjrNslEMYYoC2L+X9EwljiA== X-Google-Smtp-Source: ACHHUZ4zd/zs97m1vWfYe/KhBoTIeS1N2DZC9FAAm/8aEKtpuW+fCXZG6QeX+rZ45b4BGI2DqTp/0Q== X-Received: by 2002:a17:903:32cf:b0:1b2:1ee9:682 with SMTP id i15-20020a17090332cf00b001b21ee90682mr6370016plr.23.1686220657687; Thu, 08 Jun 2023 03:37:37 -0700 (PDT) Received: from dhcp-10-192-206-197.iig.avagotech.net.net ([192.19.234.250]) by smtp.gmail.com with ESMTPSA id jj11-20020a170903048b00b001a980a23802sm1128510plb.111.2023.06.08.03.37.35 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 08 Jun 2023 03:37:36 -0700 (PDT) From: Selvin Xavier To: jgg@ziepe.ca, leon@kernel.org Cc: linux-rdma@vger.kernel.org, andrew.gospodarek@broadcom.com, Kashyap Desai , Selvin Xavier Subject: [PATCH for-next 15/17] RDMA/bnxt_re: use firmware provided max request timeout Date: Thu, 8 Jun 2023 03:25:06 -0700 Message-Id: <1686219908-11181-16-git-send-email-selvin.xavier@broadcom.com> X-Mailer: git-send-email 2.5.5 In-Reply-To: <1686219908-11181-1-git-send-email-selvin.xavier@broadcom.com> References: <1686219908-11181-1-git-send-email-selvin.xavier@broadcom.com> Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org From: Kashyap Desai Firmware provides max request timeout value as part of hwrm_ver_get API. Driver gets the timeout from firmware and if that interface is not available then fall back to hardcoded timeout value. Also, Add a helper function to check the FW status. Signed-off-by: Kashyap Desai Signed-off-by: Selvin Xavier --- drivers/infiniband/hw/bnxt_re/main.c | 8 +++++ drivers/infiniband/hw/bnxt_re/qplib_rcfw.c | 57 ++++++++++++++++++++++++------ drivers/infiniband/hw/bnxt_re/qplib_rcfw.h | 4 ++- drivers/infiniband/hw/bnxt_re/qplib_res.h | 1 + 4 files changed, 58 insertions(+), 12 deletions(-) diff --git a/drivers/infiniband/hw/bnxt_re/main.c b/drivers/infiniband/hw/bnxt_re/main.c index 8241154..a2c7d3f 100644 --- a/drivers/infiniband/hw/bnxt_re/main.c +++ b/drivers/infiniband/hw/bnxt_re/main.c @@ -1041,6 +1041,7 @@ static void bnxt_re_query_hwrm_intf_version(struct bnxt_re_dev *rdev) struct bnxt_en_dev *en_dev = rdev->en_dev; struct hwrm_ver_get_output resp = {0}; struct hwrm_ver_get_input req = {0}; + struct bnxt_qplib_chip_ctx *cctx; struct bnxt_fw_msg fw_msg; int rc = 0; @@ -1058,11 +1059,18 @@ static void bnxt_re_query_hwrm_intf_version(struct bnxt_re_dev *rdev) rc); return; } + + cctx = rdev->chip_ctx; rdev->qplib_ctx.hwrm_intf_ver = (u64)le16_to_cpu(resp.hwrm_intf_major) << 48 | (u64)le16_to_cpu(resp.hwrm_intf_minor) << 32 | (u64)le16_to_cpu(resp.hwrm_intf_build) << 16 | le16_to_cpu(resp.hwrm_intf_patch); + + cctx->hwrm_cmd_max_timeout = le16_to_cpu(resp.max_req_timeout); + + if (!cctx->hwrm_cmd_max_timeout) + cctx->hwrm_cmd_max_timeout = RCFW_FW_STALL_MAX_TIMEOUT; } static int bnxt_re_ib_init(struct bnxt_re_dev *rdev) diff --git a/drivers/infiniband/hw/bnxt_re/qplib_rcfw.c b/drivers/infiniband/hw/bnxt_re/qplib_rcfw.c index 4f36644..8d08715 100644 --- a/drivers/infiniband/hw/bnxt_re/qplib_rcfw.c +++ b/drivers/infiniband/hw/bnxt_re/qplib_rcfw.c @@ -90,6 +90,41 @@ static int bnxt_qplib_map_rc(u8 opcode) } /** + * bnxt_re_is_fw_stalled - Check firmware health + * @rcfw - rcfw channel instance of rdev + * @cookie - cookie to track the command + * @opcode - rcfw submitted for given opcode + * @cbit - bitmap entry of cookie + * + * If firmware has not responded any rcfw command within + * rcfw->max_timeout, consider firmware as stalled. + * + * Returns: + * 0 if firmware is responding + * -ENODEV if firmware is not responding + */ +static int bnxt_re_is_fw_stalled(struct bnxt_qplib_rcfw *rcfw, + u16 cookie, u8 opcode, u16 cbit) +{ + struct bnxt_qplib_cmdq_ctx *cmdq; + + cmdq = &rcfw->cmdq; + + if (time_after(jiffies, cmdq->last_seen + + (rcfw->max_timeout * HZ))) { + dev_warn_ratelimited(&rcfw->pdev->dev, + "%s: FW STALL Detected. cmdq[%#x]=%#x waited (%d > %d) msec active %d ", + __func__, cookie, opcode, + jiffies_to_msecs(jiffies - cmdq->last_seen), + rcfw->max_timeout * 1000, + test_bit(cbit, cmdq->cmdq_bitmap)); + return -ENODEV; + } + + return 0; +} + +/** * __wait_for_resp - Don't hold the cpu context and wait for response * @rcfw - rcfw channel instance of rdev * @cookie - cookie to track the command @@ -120,8 +155,7 @@ static int __wait_for_resp(struct bnxt_qplib_rcfw *rcfw, u16 cookie, u8 opcode) ret = wait_event_timeout(cmdq->waitq, !test_bit(cbit, cmdq->cmdq_bitmap) || test_bit(ERR_DEVICE_DETACHED, &cmdq->flags), - msecs_to_jiffies(RCFW_FW_STALL_TIMEOUT_SEC - * 1000)); + msecs_to_jiffies(rcfw->max_timeout * 1000)); if (!test_bit(cbit, cmdq->cmdq_bitmap)) return 0; @@ -131,10 +165,9 @@ static int __wait_for_resp(struct bnxt_qplib_rcfw *rcfw, u16 cookie, u8 opcode) if (!test_bit(cbit, cmdq->cmdq_bitmap)) return 0; - /* Firmware stall is detected */ - if (time_after(jiffies, cmdq->last_seen + - (RCFW_FW_STALL_TIMEOUT_SEC * HZ))) - return -ENODEV; + ret = bnxt_re_is_fw_stalled(rcfw, cookie, opcode, cbit); + if (ret) + return ret; } while (true); }; @@ -357,6 +390,7 @@ static int __poll_for_resp(struct bnxt_qplib_rcfw *rcfw, u16 cookie, struct bnxt_qplib_cmdq_ctx *cmdq = &rcfw->cmdq; unsigned long issue_time; u16 cbit; + int ret; cbit = cookie % rcfw->cmdq_depth; issue_time = jiffies; @@ -373,11 +407,10 @@ static int __poll_for_resp(struct bnxt_qplib_rcfw *rcfw, u16 cookie, if (!test_bit(cbit, cmdq->cmdq_bitmap)) return 0; if (jiffies_to_msecs(jiffies - issue_time) > - (RCFW_FW_STALL_TIMEOUT_SEC * 1000)) { - /* Firmware stall is detected */ - if (time_after(jiffies, cmdq->last_seen + - (RCFW_FW_STALL_TIMEOUT_SEC * HZ))) - return -ENODEV; + (rcfw->max_timeout * 1000)) { + ret = bnxt_re_is_fw_stalled(rcfw, cookie, opcode, cbit); + if (ret) + return ret; } } while (true); }; @@ -957,6 +990,8 @@ int bnxt_qplib_alloc_rcfw_channel(struct bnxt_qplib_res *res, if (!rcfw->qp_tbl) goto fail; + rcfw->max_timeout = res->cctx->hwrm_cmd_max_timeout; + return 0; fail: diff --git a/drivers/infiniband/hw/bnxt_re/qplib_rcfw.h b/drivers/infiniband/hw/bnxt_re/qplib_rcfw.h index 338bf6a..b644dcc 100644 --- a/drivers/infiniband/hw/bnxt_re/qplib_rcfw.h +++ b/drivers/infiniband/hw/bnxt_re/qplib_rcfw.h @@ -51,7 +51,7 @@ #define RCFW_DBR_PCI_BAR_REGION 2 #define RCFW_DBR_BASE_PAGE_SHIFT 12 -#define RCFW_FW_STALL_TIMEOUT_SEC 40 +#define RCFW_FW_STALL_MAX_TIMEOUT 40 /* Cmdq contains a fix number of a 16-Byte slots */ struct bnxt_qplib_cmdqe { @@ -227,6 +227,8 @@ struct bnxt_qplib_rcfw { atomic_t rcfw_intr_enabled; struct semaphore rcfw_inflight; atomic_t timeout_send; + /* cached from chip cctx for quick reference in slow path */ + u16 max_timeout; }; struct bnxt_qplib_cmdqmsg { diff --git a/drivers/infiniband/hw/bnxt_re/qplib_res.h b/drivers/infiniband/hw/bnxt_re/qplib_res.h index 982e2c9..77f0b84 100644 --- a/drivers/infiniband/hw/bnxt_re/qplib_res.h +++ b/drivers/infiniband/hw/bnxt_re/qplib_res.h @@ -55,6 +55,7 @@ struct bnxt_qplib_chip_ctx { u8 chip_rev; u8 chip_metal; u16 hw_stats_size; + u16 hwrm_cmd_max_timeout; struct bnxt_qplib_drv_modes modes; }; From patchwork Thu Jun 8 10:25:07 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Selvin Xavier X-Patchwork-Id: 13271934 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E3FF5C7EE25 for ; Thu, 8 Jun 2023 10:38:02 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235059AbjFHKiB (ORCPT ); Thu, 8 Jun 2023 06:38:01 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50492 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233965AbjFHKiA (ORCPT ); Thu, 8 Jun 2023 06:38:00 -0400 Received: from mail-pl1-x62e.google.com (mail-pl1-x62e.google.com [IPv6:2607:f8b0:4864:20::62e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 13E0A1712 for ; Thu, 8 Jun 2023 03:37:41 -0700 (PDT) Received: by mail-pl1-x62e.google.com with SMTP id d9443c01a7336-1b065154b79so12087835ad.1 for ; Thu, 08 Jun 2023 03:37:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; t=1686220660; x=1688812660; h=references:in-reply-to:message-id:date:subject:cc:to:from:from:to :cc:subject:date:message-id:reply-to; bh=nZORGEz5YcIDqb9yaf9IaIdONJL6308EeG2LuLD1oaY=; b=M3+kGAvfjmjNTCVBJ8Qjpa+8wlYdYIBuFRY4l2IbyvxZ62nDD0E/Nfbw5SFmzezk9H xT2ZnFlkcf7aqV2XZWmNhXbjZ9eA+AvzG3VMSnYWdOfuYcexHR4OabGqjG1pm+eoyf5S hUXSIgPdGNGNl4+afos/2L4E2360Zbw7AquOY= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1686220660; x=1688812660; h=references:in-reply-to:message-id:date:subject:cc:to:from :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=nZORGEz5YcIDqb9yaf9IaIdONJL6308EeG2LuLD1oaY=; b=PlIA60uk2bR+d8V7G2Rc2Rmhz+bkBcSeR0bVHk05EnblIGbjBMRQTsuDCb06wKPUVg v0T8Bh5RXxfEEJbKSWcB3EzGHZTpC2+UKwXMSzFCr8GhoBBSCOao3wVHKKNpmkz1z7vu evao0fC0DKFCsb54j6mHvBmjni4k3ioBejAhFcuM98QgAgCMIMmfMoIqcjQV11bpWAzF IlI61vcGJCXHkgXpyFC4iqiqFgbXJiRYqqv89rRAUel1yPKpnApOQZBuIF0vtTdFI+GH CxxAB2jOjbt6S+iHHRLV+0lxbP5/QSKcI/8WDNlhAYaVg0L/Sv1gqOPyzf38KSQV6TK6 grxw== X-Gm-Message-State: AC+VfDwZsOdIRCt9iV45qo6ac9YiiaUTQZLL12Eu5PAyEfKrQmTTjIlS CxjLHNJFhRD4ouL1PelJFobv8Q== X-Google-Smtp-Source: ACHHUZ43RpzWnbeOjiCxvlVikNLkumsRrpuBrUsEQvDkqjkRTERutQv6FJEhd3uKWFh2VUtt7oKBRQ== X-Received: by 2002:a17:902:ab59:b0:1aa:d235:6dd4 with SMTP id ij25-20020a170902ab5900b001aad2356dd4mr1680386plb.19.1686220660372; Thu, 08 Jun 2023 03:37:40 -0700 (PDT) Received: from dhcp-10-192-206-197.iig.avagotech.net.net ([192.19.234.250]) by smtp.gmail.com with ESMTPSA id jj11-20020a170903048b00b001a980a23802sm1128510plb.111.2023.06.08.03.37.38 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 08 Jun 2023 03:37:39 -0700 (PDT) From: Selvin Xavier To: jgg@ziepe.ca, leon@kernel.org Cc: linux-rdma@vger.kernel.org, andrew.gospodarek@broadcom.com, Kashyap Desai , Selvin Xavier Subject: [PATCH for-next 16/17] RDMA/bnxt_re: remove redundant cmdq_bitmap Date: Thu, 8 Jun 2023 03:25:07 -0700 Message-Id: <1686219908-11181-17-git-send-email-selvin.xavier@broadcom.com> X-Mailer: git-send-email 2.5.5 In-Reply-To: <1686219908-11181-1-git-send-email-selvin.xavier@broadcom.com> References: <1686219908-11181-1-git-send-email-selvin.xavier@broadcom.com> Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org From: Kashyap Desai cmdq_bitmap is used to derive the next available index in the CMDQ. This is not required as the we can get the next index using the existing bnxt_qplib_crsqe array. Driver will use bnxt_qplib_crsqe array and flag is_in_used to derive valid entries. is_in_used is replacement of cmdq_bitmap. There is no change in the existing mechanism of the circular buffer used to get index. Added opcode field in bnxt_qplib_crsqe array so that it is easy to map opcode associated with pending rcfw command. Signed-off-by: Kashyap Desai Signed-off-by: Selvin Xavier --- drivers/infiniband/hw/bnxt_re/qplib_rcfw.c | 81 ++++++++++++------------------ drivers/infiniband/hw/bnxt_re/qplib_rcfw.h | 3 +- 2 files changed, 35 insertions(+), 49 deletions(-) diff --git a/drivers/infiniband/hw/bnxt_re/qplib_rcfw.c b/drivers/infiniband/hw/bnxt_re/qplib_rcfw.c index 8d08715..45bbf5f 100644 --- a/drivers/infiniband/hw/bnxt_re/qplib_rcfw.c +++ b/drivers/infiniband/hw/bnxt_re/qplib_rcfw.c @@ -95,6 +95,7 @@ static int bnxt_qplib_map_rc(u8 opcode) * @cookie - cookie to track the command * @opcode - rcfw submitted for given opcode * @cbit - bitmap entry of cookie + * @in_used - command is in used or freed * * If firmware has not responded any rcfw command within * rcfw->max_timeout, consider firmware as stalled. @@ -104,7 +105,7 @@ static int bnxt_qplib_map_rc(u8 opcode) * -ENODEV if firmware is not responding */ static int bnxt_re_is_fw_stalled(struct bnxt_qplib_rcfw *rcfw, - u16 cookie, u8 opcode, u16 cbit) + u16 cookie, u8 opcode, bool in_used) { struct bnxt_qplib_cmdq_ctx *cmdq; @@ -117,7 +118,7 @@ static int bnxt_re_is_fw_stalled(struct bnxt_qplib_rcfw *rcfw, __func__, cookie, opcode, jiffies_to_msecs(jiffies - cmdq->last_seen), rcfw->max_timeout * 1000, - test_bit(cbit, cmdq->cmdq_bitmap)); + in_used); return -ENODEV; } @@ -139,11 +140,11 @@ static int bnxt_re_is_fw_stalled(struct bnxt_qplib_rcfw *rcfw, static int __wait_for_resp(struct bnxt_qplib_rcfw *rcfw, u16 cookie, u8 opcode) { struct bnxt_qplib_cmdq_ctx *cmdq; - u16 cbit; + struct bnxt_qplib_crsqe *crsqe; int ret; cmdq = &rcfw->cmdq; - cbit = cookie % rcfw->cmdq_depth; + crsqe = &rcfw->crsqe_tbl[cookie]; do { if (test_bit(ERR_DEVICE_DETACHED, &cmdq->flags)) @@ -153,19 +154,19 @@ static int __wait_for_resp(struct bnxt_qplib_rcfw *rcfw, u16 cookie, u8 opcode) /* Non zero means command completed */ ret = wait_event_timeout(cmdq->waitq, - !test_bit(cbit, cmdq->cmdq_bitmap) || + !crsqe->is_in_used || test_bit(ERR_DEVICE_DETACHED, &cmdq->flags), msecs_to_jiffies(rcfw->max_timeout * 1000)); - if (!test_bit(cbit, cmdq->cmdq_bitmap)) + if (!crsqe->is_in_used) return 0; bnxt_qplib_service_creq(&rcfw->creq.creq_tasklet); - if (!test_bit(cbit, cmdq->cmdq_bitmap)) + if (!crsqe->is_in_used) return 0; - ret = bnxt_re_is_fw_stalled(rcfw, cookie, opcode, cbit); + ret = bnxt_re_is_fw_stalled(rcfw, cookie, opcode, crsqe->is_in_used); if (ret) return ret; @@ -188,11 +189,11 @@ static int __wait_for_resp(struct bnxt_qplib_rcfw *rcfw, u16 cookie, u8 opcode) static int __block_for_resp(struct bnxt_qplib_rcfw *rcfw, u16 cookie, u8 opcode) { struct bnxt_qplib_cmdq_ctx *cmdq = &rcfw->cmdq; + struct bnxt_qplib_crsqe *crsqe; unsigned long issue_time = 0; - u16 cbit; - cbit = cookie % rcfw->cmdq_depth; issue_time = jiffies; + crsqe = &rcfw->crsqe_tbl[cookie]; do { if (test_bit(ERR_DEVICE_DETACHED, &cmdq->flags)) @@ -203,7 +204,7 @@ static int __block_for_resp(struct bnxt_qplib_rcfw *rcfw, u16 cookie, u8 opcode) udelay(1); bnxt_qplib_service_creq(&rcfw->creq.creq_tasklet); - if (!test_bit(cbit, cmdq->cmdq_bitmap)) + if (!crsqe->is_in_used) return 0; } while (time_before(jiffies, issue_time + (8 * HZ))); @@ -236,16 +237,13 @@ static void __send_message_no_waiter(struct bnxt_qplib_rcfw *rcfw, struct bnxt_qplib_crsqe *crsqe; struct bnxt_qplib_cmdqe *cmdqe; u32 sw_prod, cmdq_prod; - u16 cookie, cbit; + u16 cookie; u32 bsize; u8 *preq; cookie = cmdq->seq_num & RCFW_MAX_COOKIE_VALUE; - cbit = cookie % rcfw->cmdq_depth; - - set_bit(cbit, cmdq->cmdq_bitmap); __set_cmdq_base_cookie(msg->req, msg->req_sz, cpu_to_le16(cookie)); - crsqe = &rcfw->crsqe_tbl[cbit]; + crsqe = &rcfw->crsqe_tbl[cookie]; /* Set cmd_size in terms of 16B slots in req. */ bsize = bnxt_qplib_set_cmd_slots(msg->req); @@ -254,6 +252,7 @@ static void __send_message_no_waiter(struct bnxt_qplib_rcfw *rcfw, */ crsqe->is_internal_cmd = true; crsqe->is_waiter_alive = false; + crsqe->is_in_used = true; crsqe->req_size = __get_cmdq_base_cmd_size(msg->req, msg->req_sz); preq = (u8 *)msg->req; @@ -281,7 +280,7 @@ static void __send_message_no_waiter(struct bnxt_qplib_rcfw *rcfw, static int __send_message(struct bnxt_qplib_rcfw *rcfw, struct bnxt_qplib_cmdqmsg *msg) { - u32 bsize, opcode, free_slots, required_slots; + u32 bsize, free_slots, required_slots; struct bnxt_qplib_cmdq_ctx *cmdq; struct bnxt_qplib_crsqe *crsqe; struct bnxt_qplib_cmdqe *cmdqe; @@ -289,7 +288,8 @@ static int __send_message(struct bnxt_qplib_rcfw *rcfw, u32 sw_prod, cmdq_prod; struct pci_dev *pdev; unsigned long flags; - u16 cookie, cbit; + u16 cookie; + u8 opcode; u8 *preq; cmdq = &rcfw->cmdq; @@ -305,10 +305,9 @@ static int __send_message(struct bnxt_qplib_rcfw *rcfw, required_slots = bnxt_qplib_get_cmd_slots(msg->req); free_slots = HWQ_FREE_SLOTS(hwq); cookie = cmdq->seq_num & RCFW_MAX_COOKIE_VALUE; - cbit = cookie % rcfw->cmdq_depth; + crsqe = &rcfw->crsqe_tbl[cookie]; - if (required_slots >= free_slots || - test_bit(cbit, cmdq->cmdq_bitmap)) { + if (required_slots >= free_slots) { dev_info_ratelimited(&pdev->dev, "CMDQ is full req/free %d/%d!", required_slots, free_slots); @@ -317,15 +316,17 @@ static int __send_message(struct bnxt_qplib_rcfw *rcfw, } if (msg->block) cookie |= RCFW_CMD_IS_BLOCKING; - set_bit(cbit, cmdq->cmdq_bitmap); __set_cmdq_base_cookie(msg->req, msg->req_sz, cpu_to_le16(cookie)); - crsqe = &rcfw->crsqe_tbl[cbit]; + bsize = bnxt_qplib_set_cmd_slots(msg->req); crsqe->free_slots = free_slots; crsqe->resp = (struct creq_qp_event *)msg->resp; crsqe->resp->cookie = cpu_to_le16(cookie); crsqe->is_internal_cmd = false; crsqe->is_waiter_alive = true; + crsqe->is_in_used = true; + crsqe->opcode = opcode; + crsqe->req_size = __get_cmdq_base_cmd_size(msg->req, msg->req_sz); if (__get_cmdq_base_resp_size(msg->req, msg->req_sz) && msg->sb) { struct bnxt_qplib_rcfw_sbuf *sbuf = msg->sb; @@ -388,12 +389,12 @@ static int __poll_for_resp(struct bnxt_qplib_rcfw *rcfw, u16 cookie, u8 opcode) { struct bnxt_qplib_cmdq_ctx *cmdq = &rcfw->cmdq; + struct bnxt_qplib_crsqe *crsqe; unsigned long issue_time; - u16 cbit; int ret; - cbit = cookie % rcfw->cmdq_depth; issue_time = jiffies; + crsqe = &rcfw->crsqe_tbl[cookie]; do { if (test_bit(ERR_DEVICE_DETACHED, &cmdq->flags)) @@ -404,11 +405,11 @@ static int __poll_for_resp(struct bnxt_qplib_rcfw *rcfw, u16 cookie, usleep_range(1000, 1001); bnxt_qplib_service_creq(&rcfw->creq.creq_tasklet); - if (!test_bit(cbit, cmdq->cmdq_bitmap)) + if (!crsqe->is_in_used) return 0; if (jiffies_to_msecs(jiffies - issue_time) > (rcfw->max_timeout * 1000)) { - ret = bnxt_re_is_fw_stalled(rcfw, cookie, opcode, cbit); + ret = bnxt_re_is_fw_stalled(rcfw, cookie, opcode, crsqe->is_in_used); if (ret) return ret; } @@ -488,7 +489,7 @@ static int __bnxt_qplib_rcfw_send_message(struct bnxt_qplib_rcfw *rcfw, struct creq_qp_event *evnt = (struct creq_qp_event *)msg->resp; struct bnxt_qplib_crsqe *crsqe; unsigned long flags; - u16 cookie, cbit; + u16 cookie; int rc = 0; u8 opcode; @@ -504,7 +505,6 @@ static int __bnxt_qplib_rcfw_send_message(struct bnxt_qplib_rcfw *rcfw, cookie = le16_to_cpu(__get_cmdq_base_cookie(msg->req, msg->req_sz)) & RCFW_MAX_COOKIE_VALUE; - cbit = cookie % rcfw->cmdq_depth; if (msg->block) rc = __block_for_resp(rcfw, cookie, opcode); @@ -521,7 +521,7 @@ static int __bnxt_qplib_rcfw_send_message(struct bnxt_qplib_rcfw *rcfw, if (rc) { spin_lock_irqsave(&rcfw->cmdq.hwq.lock, flags); - crsqe = &rcfw->crsqe_tbl[cbit]; + crsqe = &rcfw->crsqe_tbl[cookie]; crsqe->is_waiter_alive = false; if (rc == -ENODEV) set_bit(FIRMWARE_STALL_DETECTED, &rcfw->cmdq.flags); @@ -633,13 +633,12 @@ static int bnxt_qplib_process_qp_event(struct bnxt_qplib_rcfw *rcfw, struct bnxt_qplib_crsqe *crsqe; u32 qp_id, tbl_indx, req_size; struct bnxt_qplib_qp *qp; - u16 cbit, blocked = 0; + u16 cookie, blocked = 0; bool is_waiter_alive; struct pci_dev *pdev; unsigned long flags; u32 wait_cmds = 0; __le16 mcookie; - u16 cookie; int rc = 0; pdev = rcfw->pdev; @@ -675,8 +674,8 @@ static int bnxt_qplib_process_qp_event(struct bnxt_qplib_rcfw *rcfw, mcookie = qp_event->cookie; blocked = cookie & RCFW_CMD_IS_BLOCKING; cookie &= RCFW_MAX_COOKIE_VALUE; - cbit = cookie % rcfw->cmdq_depth; - crsqe = &rcfw->crsqe_tbl[cbit]; + crsqe = &rcfw->crsqe_tbl[cookie]; + crsqe->is_in_used = false; if (WARN_ONCE(test_bit(FIRMWARE_STALL_DETECTED, &rcfw->cmdq.flags), @@ -687,9 +686,6 @@ static int bnxt_qplib_process_qp_event(struct bnxt_qplib_rcfw *rcfw, return rc; } - if (!test_and_clear_bit(cbit, rcfw->cmdq.cmdq_bitmap)) - dev_warn(&pdev->dev, - "CMD bit %d was not requested\n", cbit); if (crsqe->is_internal_cmd && !qp_event->status) atomic_dec(&rcfw->timeout_send); @@ -924,7 +920,6 @@ int bnxt_qplib_init_rcfw(struct bnxt_qplib_rcfw *rcfw, void bnxt_qplib_free_rcfw_channel(struct bnxt_qplib_rcfw *rcfw) { - bitmap_free(rcfw->cmdq.cmdq_bitmap); kfree(rcfw->qp_tbl); kfree(rcfw->crsqe_tbl); bnxt_qplib_free_hwq(rcfw->res, &rcfw->cmdq.hwq); @@ -979,10 +974,6 @@ int bnxt_qplib_alloc_rcfw_channel(struct bnxt_qplib_res *res, if (!rcfw->crsqe_tbl) goto fail; - cmdq->cmdq_bitmap = bitmap_zalloc(rcfw->cmdq_depth, GFP_KERNEL); - if (!cmdq->cmdq_bitmap) - goto fail; - /* Allocate one extra to hold the QP1 entries */ rcfw->qp_tbl_size = qp_tbl_sz + 1; rcfw->qp_tbl = kcalloc(rcfw->qp_tbl_size, sizeof(struct bnxt_qplib_qp_node), @@ -1027,7 +1018,6 @@ void bnxt_qplib_disable_rcfw_channel(struct bnxt_qplib_rcfw *rcfw) { struct bnxt_qplib_creq_ctx *creq; struct bnxt_qplib_cmdq_ctx *cmdq; - unsigned long indx; creq = &rcfw->creq; cmdq = &rcfw->cmdq; @@ -1037,11 +1027,6 @@ void bnxt_qplib_disable_rcfw_channel(struct bnxt_qplib_rcfw *rcfw) iounmap(cmdq->cmdq_mbox.reg.bar_reg); iounmap(creq->creq_db.reg.bar_reg); - indx = find_first_bit(cmdq->cmdq_bitmap, rcfw->cmdq_depth); - if (indx != rcfw->cmdq_depth) - dev_err(&rcfw->pdev->dev, - "disabling RCFW with pending cmd-bit %lx\n", indx); - cmdq->cmdq_mbox.reg.bar_reg = NULL; creq->creq_db.reg.bar_reg = NULL; creq->aeq_handler = NULL; diff --git a/drivers/infiniband/hw/bnxt_re/qplib_rcfw.h b/drivers/infiniband/hw/bnxt_re/qplib_rcfw.h index b644dcc..f46de07 100644 --- a/drivers/infiniband/hw/bnxt_re/qplib_rcfw.h +++ b/drivers/infiniband/hw/bnxt_re/qplib_rcfw.h @@ -152,8 +152,10 @@ struct bnxt_qplib_crsqe { u32 req_size; /* Free slots at the time of submission */ u32 free_slots; + u8 opcode; bool is_waiter_alive; bool is_internal_cmd; + bool is_in_used; }; struct bnxt_qplib_rcfw_sbuf { @@ -185,7 +187,6 @@ struct bnxt_qplib_cmdq_ctx { struct bnxt_qplib_cmdq_mbox cmdq_mbox; wait_queue_head_t waitq; unsigned long flags; - unsigned long *cmdq_bitmap; unsigned long last_seen; u32 seq_num; }; From patchwork Thu Jun 8 10:25:08 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Selvin Xavier X-Patchwork-Id: 13271935 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 99D8EC7EE23 for ; Thu, 8 Jun 2023 10:38:07 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233965AbjFHKiG (ORCPT ); Thu, 8 Jun 2023 06:38:06 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50576 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235462AbjFHKiE (ORCPT ); Thu, 8 Jun 2023 06:38:04 -0400 Received: from mail-pg1-x533.google.com (mail-pg1-x533.google.com [IPv6:2607:f8b0:4864:20::533]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E4D3D30D0 for ; Thu, 8 Jun 2023 03:37:43 -0700 (PDT) Received: by mail-pg1-x533.google.com with SMTP id 41be03b00d2f7-543c692db30so232852a12.3 for ; Thu, 08 Jun 2023 03:37:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; t=1686220663; x=1688812663; h=references:in-reply-to:message-id:date:subject:cc:to:from:from:to :cc:subject:date:message-id:reply-to; bh=H7X2h68Oxs4Gb9veWaY48teMprU/YVNoF3JPTJbDMQo=; b=eorCv2W31kAhrQXiWBe5SSZpfed/ikPrYsgVtmYNJ51Z8qhXNNgsBH04cT7H+pgYDN 6MyYgkMU5amLbICX1KvaatfRHSxkcilb/4eGD7MfiSdKbChjlOi1pKyG8mfwqIB2GCcZ nQmLDhuOZHrwI2zMWbnBC5X5iezZPD5j4op7g= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1686220663; x=1688812663; h=references:in-reply-to:message-id:date:subject:cc:to:from :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=H7X2h68Oxs4Gb9veWaY48teMprU/YVNoF3JPTJbDMQo=; b=ElPZw/Gpe/TFbr9DkQlr75X4/iOmBN+j5Ht6gDJ6SBWlfneljzaRvqzCILZGhUGjem Y6ybFYkisxZh5PuOKiSDVA/qtpNZJEeVfsRWZ5YRaeijSySwJcQZf8T0ZaO61WrbHQEC aTxtlpzHoPQJoNOtjdEQMrckHBA050TygPh3wpy39T+rOyZvLlyI9EsgxaxQO9sA3W2h cKWokMVjXKSs0UGXqsjexjlE5R0ypmGURBVo/ZY6RWmzTLTW3o3gce3KRTp2efZOO+uA USXhrVINe3ufC8imp/SWiZJeQzD1VPLSYx+tvuIwKoaCfnVAyZoPvF0AYj4LFI2SHz9m VboA== X-Gm-Message-State: AC+VfDx1lQK8ngW8FvBS9pLwnDp4KIQ9FWtSzFwxmPSW7uZ9RLIMC0Od 7RTCuY0aHUoKqYd7aGhmze9hPg== X-Google-Smtp-Source: ACHHUZ7jgF7fRe9SpxSqVcv82A4I7wJFrVIbMdCJRVQj/nZhYFWAB02R69oPWlXdvuNdThc6KAGC2Q== X-Received: by 2002:a17:90a:4d:b0:255:b07a:3d55 with SMTP id 13-20020a17090a004d00b00255b07a3d55mr7830376pjb.28.1686220663098; Thu, 08 Jun 2023 03:37:43 -0700 (PDT) Received: from dhcp-10-192-206-197.iig.avagotech.net.net ([192.19.234.250]) by smtp.gmail.com with ESMTPSA id jj11-20020a170903048b00b001a980a23802sm1128510plb.111.2023.06.08.03.37.40 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 08 Jun 2023 03:37:42 -0700 (PDT) From: Selvin Xavier To: jgg@ziepe.ca, leon@kernel.org Cc: linux-rdma@vger.kernel.org, andrew.gospodarek@broadcom.com, Kashyap Desai , Selvin Xavier Subject: [PATCH for-next 17/17] RDMA/bnxt_re: optimize the parameters passed to helper functions Date: Thu, 8 Jun 2023 03:25:08 -0700 Message-Id: <1686219908-11181-18-git-send-email-selvin.xavier@broadcom.com> X-Mailer: git-send-email 2.5.5 In-Reply-To: <1686219908-11181-1-git-send-email-selvin.xavier@broadcom.com> References: <1686219908-11181-1-git-send-email-selvin.xavier@broadcom.com> Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org From: Kashyap Desai Avoid passing arguments like Opcode which can be retrieved from bnxt_qplib_crsqe structure. Signed-off-by: Kashyap Desai Signed-off-by: Selvin Xavier --- drivers/infiniband/hw/bnxt_re/qplib_rcfw.c | 44 +++++++++++++----------------- 1 file changed, 19 insertions(+), 25 deletions(-) diff --git a/drivers/infiniband/hw/bnxt_re/qplib_rcfw.c b/drivers/infiniband/hw/bnxt_re/qplib_rcfw.c index 45bbf5f..3526518 100644 --- a/drivers/infiniband/hw/bnxt_re/qplib_rcfw.c +++ b/drivers/infiniband/hw/bnxt_re/qplib_rcfw.c @@ -93,9 +93,6 @@ static int bnxt_qplib_map_rc(u8 opcode) * bnxt_re_is_fw_stalled - Check firmware health * @rcfw - rcfw channel instance of rdev * @cookie - cookie to track the command - * @opcode - rcfw submitted for given opcode - * @cbit - bitmap entry of cookie - * @in_used - command is in used or freed * * If firmware has not responded any rcfw command within * rcfw->max_timeout, consider firmware as stalled. @@ -105,20 +102,22 @@ static int bnxt_qplib_map_rc(u8 opcode) * -ENODEV if firmware is not responding */ static int bnxt_re_is_fw_stalled(struct bnxt_qplib_rcfw *rcfw, - u16 cookie, u8 opcode, bool in_used) + u16 cookie) { struct bnxt_qplib_cmdq_ctx *cmdq; + struct bnxt_qplib_crsqe *crsqe; + crsqe = &rcfw->crsqe_tbl[cookie]; cmdq = &rcfw->cmdq; if (time_after(jiffies, cmdq->last_seen + (rcfw->max_timeout * HZ))) { dev_warn_ratelimited(&rcfw->pdev->dev, "%s: FW STALL Detected. cmdq[%#x]=%#x waited (%d > %d) msec active %d ", - __func__, cookie, opcode, + __func__, cookie, crsqe->opcode, jiffies_to_msecs(jiffies - cmdq->last_seen), rcfw->max_timeout * 1000, - in_used); + crsqe->is_in_used); return -ENODEV; } @@ -129,7 +128,6 @@ static int bnxt_re_is_fw_stalled(struct bnxt_qplib_rcfw *rcfw, * __wait_for_resp - Don't hold the cpu context and wait for response * @rcfw - rcfw channel instance of rdev * @cookie - cookie to track the command - * @opcode - rcfw submitted for given opcode * * Wait for command completion in sleepable context. * @@ -137,7 +135,7 @@ static int bnxt_re_is_fw_stalled(struct bnxt_qplib_rcfw *rcfw, * 0 if command is completed by firmware. * Non zero error code for rest of the case. */ -static int __wait_for_resp(struct bnxt_qplib_rcfw *rcfw, u16 cookie, u8 opcode) +static int __wait_for_resp(struct bnxt_qplib_rcfw *rcfw, u16 cookie) { struct bnxt_qplib_cmdq_ctx *cmdq; struct bnxt_qplib_crsqe *crsqe; @@ -148,7 +146,7 @@ static int __wait_for_resp(struct bnxt_qplib_rcfw *rcfw, u16 cookie, u8 opcode) do { if (test_bit(ERR_DEVICE_DETACHED, &cmdq->flags)) - return bnxt_qplib_map_rc(opcode); + return bnxt_qplib_map_rc(crsqe->opcode); if (test_bit(FIRMWARE_STALL_DETECTED, &cmdq->flags)) return -ETIMEDOUT; @@ -166,7 +164,7 @@ static int __wait_for_resp(struct bnxt_qplib_rcfw *rcfw, u16 cookie, u8 opcode) if (!crsqe->is_in_used) return 0; - ret = bnxt_re_is_fw_stalled(rcfw, cookie, opcode, crsqe->is_in_used); + ret = bnxt_re_is_fw_stalled(rcfw, cookie); if (ret) return ret; @@ -177,7 +175,6 @@ static int __wait_for_resp(struct bnxt_qplib_rcfw *rcfw, u16 cookie, u8 opcode) * __block_for_resp - hold the cpu context and wait for response * @rcfw - rcfw channel instance of rdev * @cookie - cookie to track the command - * @opcode - rcfw submitted for given opcode * * This function will hold the cpu (non-sleepable context) and * wait for command completion. Maximum holding interval is 8 second. @@ -186,7 +183,7 @@ static int __wait_for_resp(struct bnxt_qplib_rcfw *rcfw, u16 cookie, u8 opcode) * -ETIMEOUT if command is not completed in specific time interval. * 0 if command is completed by firmware. */ -static int __block_for_resp(struct bnxt_qplib_rcfw *rcfw, u16 cookie, u8 opcode) +static int __block_for_resp(struct bnxt_qplib_rcfw *rcfw, u16 cookie) { struct bnxt_qplib_cmdq_ctx *cmdq = &rcfw->cmdq; struct bnxt_qplib_crsqe *crsqe; @@ -197,7 +194,7 @@ static int __block_for_resp(struct bnxt_qplib_rcfw *rcfw, u16 cookie, u8 opcode) do { if (test_bit(ERR_DEVICE_DETACHED, &cmdq->flags)) - return bnxt_qplib_map_rc(opcode); + return bnxt_qplib_map_rc(crsqe->opcode); if (test_bit(FIRMWARE_STALL_DETECTED, &cmdq->flags)) return -ETIMEDOUT; @@ -375,7 +372,6 @@ static int __send_message(struct bnxt_qplib_rcfw *rcfw, * __poll_for_resp - self poll completion for rcfw command * @rcfw - rcfw channel instance of rdev * @cookie - cookie to track the command - * @opcode - rcfw submitted for given opcode * * It works same as __wait_for_resp except this function will * do self polling in sort interval since interrupt is disabled. @@ -385,8 +381,7 @@ static int __send_message(struct bnxt_qplib_rcfw *rcfw, * -ETIMEOUT if command is not completed in specific time interval. * 0 if command is completed by firmware. */ -static int __poll_for_resp(struct bnxt_qplib_rcfw *rcfw, u16 cookie, - u8 opcode) +static int __poll_for_resp(struct bnxt_qplib_rcfw *rcfw, u16 cookie) { struct bnxt_qplib_cmdq_ctx *cmdq = &rcfw->cmdq; struct bnxt_qplib_crsqe *crsqe; @@ -398,7 +393,7 @@ static int __poll_for_resp(struct bnxt_qplib_rcfw *rcfw, u16 cookie, do { if (test_bit(ERR_DEVICE_DETACHED, &cmdq->flags)) - return bnxt_qplib_map_rc(opcode); + return bnxt_qplib_map_rc(crsqe->opcode); if (test_bit(FIRMWARE_STALL_DETECTED, &cmdq->flags)) return -ETIMEDOUT; @@ -409,7 +404,7 @@ static int __poll_for_resp(struct bnxt_qplib_rcfw *rcfw, u16 cookie, return 0; if (jiffies_to_msecs(jiffies - issue_time) > (rcfw->max_timeout * 1000)) { - ret = bnxt_re_is_fw_stalled(rcfw, cookie, opcode, crsqe->is_in_used); + ret = bnxt_re_is_fw_stalled(rcfw, cookie); if (ret) return ret; } @@ -417,13 +412,12 @@ static int __poll_for_resp(struct bnxt_qplib_rcfw *rcfw, u16 cookie, }; static int __send_message_basic_sanity(struct bnxt_qplib_rcfw *rcfw, - struct bnxt_qplib_cmdqmsg *msg) + struct bnxt_qplib_cmdqmsg *msg, + u8 opcode) { struct bnxt_qplib_cmdq_ctx *cmdq; - u32 opcode; cmdq = &rcfw->cmdq; - opcode = __get_cmdq_base_opcode(msg->req, msg->req_sz); /* Prevent posting if f/w is not in a state to process */ if (test_bit(ERR_DEVICE_DETACHED, &rcfw->cmdq.flags)) @@ -495,7 +489,7 @@ static int __bnxt_qplib_rcfw_send_message(struct bnxt_qplib_rcfw *rcfw, opcode = __get_cmdq_base_opcode(msg->req, msg->req_sz); - rc = __send_message_basic_sanity(rcfw, msg); + rc = __send_message_basic_sanity(rcfw, msg, opcode); if (rc) return rc == -ENXIO ? bnxt_qplib_map_rc(opcode) : rc; @@ -507,11 +501,11 @@ static int __bnxt_qplib_rcfw_send_message(struct bnxt_qplib_rcfw *rcfw, & RCFW_MAX_COOKIE_VALUE; if (msg->block) - rc = __block_for_resp(rcfw, cookie, opcode); + rc = __block_for_resp(rcfw, cookie); else if (atomic_read(&rcfw->rcfw_intr_enabled)) - rc = __wait_for_resp(rcfw, cookie, opcode); + rc = __wait_for_resp(rcfw, cookie); else - rc = __poll_for_resp(rcfw, cookie, opcode); + rc = __poll_for_resp(rcfw, cookie); if (rc) { /* timed out */ dev_err(&rcfw->pdev->dev, "cmdq[%#x]=%#x timedout (%d)msec\n",