From patchwork Mon May 14 17:47:39 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Devesh Sharma X-Patchwork-Id: 10399057 X-Patchwork-Delegate: dledford@redhat.com Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 480D1601E7 for ; Mon, 14 May 2018 17:48:17 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 3802128382 for ; Mon, 14 May 2018 17:48:17 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 2C4FA28396; Mon, 14 May 2018 17:48:17 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.8 required=2.0 tests=BAYES_00,DKIM_SIGNED, MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI, T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6ABBE28382 for ; Mon, 14 May 2018 17:48:16 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752031AbeENRsP (ORCPT ); Mon, 14 May 2018 13:48:15 -0400 Received: from mail-qk0-f196.google.com ([209.85.220.196]:38282 "EHLO mail-qk0-f196.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751992AbeENRsO (ORCPT ); Mon, 14 May 2018 13:48:14 -0400 Received: by mail-qk0-f196.google.com with SMTP id b39-v6so10707039qkb.5 for ; Mon, 14 May 2018 10:48:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=O1OBDj/7EQWs1/486aYlIxcpN7fo3J/T2rksdIMWgAY=; b=d6ABYO/EE6ww5ka9TXUcZJxg1K6wPRrTwW1gXI4LbMClEeoOI+/g0xtWbJB0ifHHnw 5HrUCupZvBEA5CjbbUUfDpsdVPL8tW+B4Yxan1XfAjPl2Wv/K3DGSf7WAOv8O0X4+icl 5wA0VWFcQPKI64TAdefSlf29U7th1Z/y56gVY= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=O1OBDj/7EQWs1/486aYlIxcpN7fo3J/T2rksdIMWgAY=; b=naWtz7NSau+ekeRMYOAlTifh5auGBVMG1+i4nwoKnNkixP33tPXsyxVUYWOCzZUr4q ATUSRIyBS+9dN09xCydpVpI/4qLno5w2QNGl2L4sLPfry8U/5XX9x8D72ShdKaQO55Ki SjdoiYNUtX6oJWrLg9qF5hNcj3LX7IAvdMAS2h/gQsi+SRdQOnc7XmU4zfappV8L7sYe N9OIIcdCrPEA98dXcbVlzoKIP1Jcuuy0/p09/KRqmVHpWZyhPqdiRDZY1KT292+ykXZn 1S7RX50HdMeCUmjAJqCpqP4TjHjGeB7vf2oC47skEEkacZRhGx02M+CDHHlKuvbHAu1l rqAQ== X-Gm-Message-State: ALKqPwcVzYlzrELbnW1D2JejKC6GYK5Pqu0IuGpIJF3Cx8/efHKuqbOV UOwQ5hDilR1mKSU68vI5wS26Lw== X-Google-Smtp-Source: AB8JxZpHflMBrysLK/xyanM5/jMX+szAdh7jFRNNycXM1tlln10M/2KwoHWijbx1erxQpU38WU8d3w== X-Received: by 2002:a37:aa4b:: with SMTP id t72-v6mr9068569qke.215.1526320093566; Mon, 14 May 2018 10:48:13 -0700 (PDT) Received: from neo00-el73.dhcp.broadcom.net ([192.19.234.250]) by smtp.gmail.com with ESMTPSA id l1-v6sm7875419qki.32.2018.05.14.10.48.10 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 14 May 2018 10:48:13 -0700 (PDT) From: Devesh Sharma To: dledford@redhat.com, jgg@mellanox.com Cc: linux-rdma@vger.kernel.org, jtoppins@redhat.com, ddutile@redhat.com, Devesh Sharma Subject: [rdma-next 1/2] bnxt_re: refactor creq irq init and fp irq init Date: Mon, 14 May 2018 13:47:39 -0400 Message-Id: <1526320060-9517-2-git-send-email-devesh.sharma@broadcom.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1526320060-9517-1-git-send-email-devesh.sharma@broadcom.com> References: <1526320060-9517-1-git-send-email-devesh.sharma@broadcom.com> Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Refactoring the code to accommodate changes required to adjust msix vectors using the new scheme of vector allocation/start/stop. Signed-off-by: Devesh Sharma --- drivers/infiniband/hw/bnxt_re/qplib_fp.c | 91 +++++++++++++++++++----------- drivers/infiniband/hw/bnxt_re/qplib_rcfw.c | 53 +++++++++++------ 2 files changed, 93 insertions(+), 51 deletions(-) diff --git a/drivers/infiniband/hw/bnxt_re/qplib_fp.c b/drivers/infiniband/hw/bnxt_re/qplib_fp.c index 3a78fab..e56f063 100644 --- a/drivers/infiniband/hw/bnxt_re/qplib_fp.c +++ b/drivers/infiniband/hw/bnxt_re/qplib_fp.c @@ -336,22 +336,31 @@ static irqreturn_t bnxt_qplib_nq_irq(int irq, void *dev_instance) return IRQ_HANDLED; } -void bnxt_qplib_disable_nq(struct bnxt_qplib_nq *nq) +static void bnxt_qplib_nq_stop_irq(struct bnxt_qplib_nq *nq) { - if (nq->cqn_wq) { - destroy_workqueue(nq->cqn_wq); - nq->cqn_wq = NULL; - } - /* Make sure the HW is stopped! */ - synchronize_irq(nq->vector); tasklet_disable(&nq->worker); + /* Mask h/w interrupt */ + NQ_DB(nq->bar_reg_iomem, nq->hwq.cons, nq->hwq.max_elements); + /* Sync with last running IRQ handler */ + synchronize_irq(nq->vector); tasklet_kill(&nq->worker); - if (nq->requested) { irq_set_affinity_hint(nq->vector, NULL); free_irq(nq->vector, nq); nq->requested = false; } +} + +void bnxt_qplib_disable_nq(struct bnxt_qplib_nq *nq) +{ + if (nq->cqn_wq) { + destroy_workqueue(nq->cqn_wq); + nq->cqn_wq = NULL; + } + + /* Make sure the HW is stopped! */ + bnxt_qplib_nq_stop_irq(nq); + if (nq->bar_reg_iomem) iounmap(nq->bar_reg_iomem); nq->bar_reg_iomem = NULL; @@ -361,6 +370,38 @@ void bnxt_qplib_disable_nq(struct bnxt_qplib_nq *nq) nq->vector = 0; } +static int bnxt_qplib_nq_start_irq(struct bnxt_qplib_nq *nq, + int nq_indx, int msix_vector) +{ + int rc; + + if (nq->requested) + return -EFAULT; + + nq->vector = msix_vector; + tasklet_init(&nq->worker, bnxt_qplib_service_nq, + (unsigned long)nq); + + memset(nq->name, 0, 32); + sprintf(nq->name, "bnxt_qplib_nq-%d", nq_indx); + rc = request_irq(nq->vector, bnxt_qplib_nq_irq, 0, nq->name, nq); + if (rc) + return rc; + + cpumask_clear(&nq->mask); + cpumask_set_cpu(nq_indx, &nq->mask); + rc = irq_set_affinity_hint(nq->vector, &nq->mask); + if (rc) { + dev_warn(&nq->pdev->dev, + "QPLIB: set affinity failed; vector: %d nq_idx: %d\n", + nq->vector, nq_indx); + } + nq->requested = true; + NQ_DB_REARM(nq->bar_reg_iomem, nq->hwq.cons, nq->hwq.max_elements); + + return rc; +} + int bnxt_qplib_enable_nq(struct pci_dev *pdev, struct bnxt_qplib_nq *nq, int nq_idx, int msix_vector, int bar_reg_offset, int (*cqn_handler)(struct bnxt_qplib_nq *nq, @@ -372,41 +413,17 @@ int bnxt_qplib_enable_nq(struct pci_dev *pdev, struct bnxt_qplib_nq *nq, resource_size_t nq_base; int rc = -1; - nq->pdev = pdev; - nq->vector = msix_vector; if (cqn_handler) nq->cqn_handler = cqn_handler; if (srqn_handler) nq->srqn_handler = srqn_handler; - tasklet_init(&nq->worker, bnxt_qplib_service_nq, (unsigned long)nq); - /* Have a task to schedule CQ notifiers in post send case */ nq->cqn_wq = create_singlethread_workqueue("bnxt_qplib_nq"); if (!nq->cqn_wq) - goto fail; - - nq->requested = false; - memset(nq->name, 0, 32); - sprintf(nq->name, "bnxt_qplib_nq-%d", nq_idx); - rc = request_irq(nq->vector, bnxt_qplib_nq_irq, 0, nq->name, nq); - if (rc) { - dev_err(&nq->pdev->dev, - "Failed to request IRQ for NQ: %#x", rc); - goto fail; - } - - cpumask_clear(&nq->mask); - cpumask_set_cpu(nq_idx, &nq->mask); - rc = irq_set_affinity_hint(nq->vector, &nq->mask); - if (rc) { - dev_warn(&nq->pdev->dev, - "QPLIB: set affinity failed; vector: %d nq_idx: %d\n", - nq->vector, nq_idx); - } + return -ENOMEM; - nq->requested = true; nq->bar_reg = NQ_CONS_PCI_BAR_REGION; nq->bar_reg_off = bar_reg_offset; nq_base = pci_resource_start(pdev, nq->bar_reg); @@ -419,7 +436,13 @@ int bnxt_qplib_enable_nq(struct pci_dev *pdev, struct bnxt_qplib_nq *nq, rc = -ENOMEM; goto fail; } - NQ_DB_REARM(nq->bar_reg_iomem, nq->hwq.cons, nq->hwq.max_elements); + + rc = bnxt_qplib_nq_start_irq(nq, nq_idx, msix_vector); + if (rc) { + dev_err(&nq->pdev->dev, + "QPLIB: Failed to request irq for nq-idx %d", nq_idx); + goto fail; + } return 0; fail: diff --git a/drivers/infiniband/hw/bnxt_re/qplib_rcfw.c b/drivers/infiniband/hw/bnxt_re/qplib_rcfw.c index 80027a4..fe9e2aa 100644 --- a/drivers/infiniband/hw/bnxt_re/qplib_rcfw.c +++ b/drivers/infiniband/hw/bnxt_re/qplib_rcfw.c @@ -582,19 +582,28 @@ int bnxt_qplib_alloc_rcfw_channel(struct pci_dev *pdev, return -ENOMEM; } -void bnxt_qplib_disable_rcfw_channel(struct bnxt_qplib_rcfw *rcfw) +static void bnxt_qplib_rcfw_stop_irq(struct bnxt_qplib_rcfw *rcfw) { - unsigned long indx; - - /* Make sure the HW channel is stopped! */ - synchronize_irq(rcfw->vector); tasklet_disable(&rcfw->worker); + /* Mask h/w interrupts */ + CREQ_DB(rcfw->creq_bar_reg_iomem, rcfw->creq.cons, + rcfw->creq.max_elements); + /* Sync with last running IRQ-handler */ + synchronize_irq(rcfw->vector); tasklet_kill(&rcfw->worker); if (rcfw->requested) { free_irq(rcfw->vector, rcfw); rcfw->requested = false; } +} + +void bnxt_qplib_disable_rcfw_channel(struct bnxt_qplib_rcfw *rcfw) +{ + unsigned long indx; + + bnxt_qplib_rcfw_stop_irq(rcfw); + if (rcfw->cmdq_bar_reg_iomem) iounmap(rcfw->cmdq_bar_reg_iomem); rcfw->cmdq_bar_reg_iomem = NULL; @@ -614,6 +623,26 @@ void bnxt_qplib_disable_rcfw_channel(struct bnxt_qplib_rcfw *rcfw) rcfw->vector = 0; } +static int bnxt_qplib_rcfw_start_irq(struct bnxt_qplib_rcfw *rcfw, + int msix_vector) +{ + int rc; + + if (rcfw->requested) + return -EFAULT; + rcfw->vector = msix_vector; + tasklet_init(&rcfw->worker, + bnxt_qplib_service_creq, (unsigned long)rcfw); + rc = request_irq(rcfw->vector, bnxt_qplib_creq_irq, 0, + "bnxt_qplib_creq", rcfw); + if (rc) + return rc; + rcfw->requested = true; + CREQ_DB_REARM(rcfw->creq_bar_reg_iomem, rcfw->creq.cons, + rcfw->creq.max_elements); + return 0; +} + int bnxt_qplib_enable_rcfw_channel(struct pci_dev *pdev, struct bnxt_qplib_rcfw *rcfw, int msix_vector, @@ -675,27 +704,17 @@ int bnxt_qplib_enable_rcfw_channel(struct pci_dev *pdev, rcfw->creq_qp_event_processed = 0; rcfw->creq_func_event_processed = 0; - rcfw->vector = msix_vector; if (aeq_handler) rcfw->aeq_handler = aeq_handler; + init_waitqueue_head(&rcfw->waitq); - tasklet_init(&rcfw->worker, bnxt_qplib_service_creq, - (unsigned long)rcfw); - - rcfw->requested = false; - rc = request_irq(rcfw->vector, bnxt_qplib_creq_irq, 0, - "bnxt_qplib_creq", rcfw); + rc = bnxt_qplib_rcfw_start_irq(rcfw, msix_vector); if (rc) { dev_err(&rcfw->pdev->dev, "QPLIB: Failed to request IRQ for CREQ rc = 0x%x", rc); bnxt_qplib_disable_rcfw_channel(rcfw); return rc; } - rcfw->requested = true; - - init_waitqueue_head(&rcfw->waitq); - - CREQ_DB_REARM(rcfw->creq_bar_reg_iomem, 0, rcfw->creq.max_elements); init.cmdq_pbl = cpu_to_le64(rcfw->cmdq.pbl[PBL_LVL_0].pg_map_arr[0]); init.cmdq_size_cmdq_lvl = cpu_to_le16(