From patchwork Thu Nov 14 09:49:05 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Selvin Xavier X-Patchwork-Id: 13874873 Received: from mail-pl1-f176.google.com (mail-pl1-f176.google.com [209.85.214.176]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 17C841F9402 for ; Thu, 14 Nov 2024 10:09:57 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.176 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731578999; cv=none; b=i8Od21q68RbZ/bWceDVYiP0WqZ6oKuDP5EDQ9KqRw1qvA5z9jK+Vn99R/ebmGg0KdMnrN5LlyaySsiJ06F9HAryvClSQQwbYOPbLDbxN2vLYxTumPxXPMSobL+x6OBasRMJetfyWJrFZMJUsVf1rvLqvTeG9CV3nuY48F3YUBqw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731578999; c=relaxed/simple; bh=O71aN2JbigWuk34uSQkNwJuIEcj/OKyfbDig1P0aKk4=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References; b=iefyMV2KINBMk9etONkrohBowo5pAHNXVu+f3YffwYrhy/v6xVsKVb6uu4xQ2QnNrPHYs4uNkLTVQIdsCaHzrZo/l62Q009mM4UwfUkc6Lb6GpUv2rF3yG+quJ3etlxRmrzSNkoLX6E8500vZGuK/Xker8mkQEREqbQWV4pCtJA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=broadcom.com; spf=fail smtp.mailfrom=broadcom.com; dkim=pass (1024-bit key) header.d=broadcom.com header.i=@broadcom.com header.b=OUA9POLm; arc=none smtp.client-ip=209.85.214.176 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=broadcom.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=broadcom.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=broadcom.com header.i=@broadcom.com header.b="OUA9POLm" Received: by mail-pl1-f176.google.com with SMTP id d9443c01a7336-20c803787abso3229205ad.0 for ; Thu, 14 Nov 2024 02:09:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; t=1731578997; x=1732183797; darn=vger.kernel.org; h=references:in-reply-to:message-id:date:subject:cc:to:from:from:to :cc:subject:date:message-id:reply-to; bh=JFkF6hN7CIrpVleCjythFCzqJs/7/QyjmUB4pSrKtVk=; b=OUA9POLmNFqz7IOhkCQhL7rahfbViv+2d2cNAjSuKZQCWvObc9MHBwHjQqePS///rU xTx70OeRAB4V0sDS9XrF8xxRBFNMRj1UVp9KI6ax2v9VwathTBkzHD/E8nprnQQYFeaP GUZEH7aBt40CEATq8M8Y1DBbMTSfar4LdDHbM= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1731578997; x=1732183797; h=references:in-reply-to:message-id:date:subject:cc:to:from :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=JFkF6hN7CIrpVleCjythFCzqJs/7/QyjmUB4pSrKtVk=; b=hgII68UrCsdVOqO53uLJa8eZVPd8EPCuHlKRwWh1dl3cPUsHCmP4wu4iYltijZtflw KSpmhFZ2zglCzSi/dO6MLQo+xwX8V3CO57cysCrLs7LnS49S5uk3KGXtmkVXALl5MsVQ ZTXzTh097SvICnP5KT0BxFwneWRE84XFCME2wEVezqRYeeDI4A4lt+goGd9JlhzQ5aDV /GXYsyxRB4r4a5VcDfGEL2mU+tr6l7+VhjrDn+hOMn7gJndZ5e0szXh6M7IOyn7Q3tG0 rh2Ly4y109Vw+EzloMfnb43t1hHdKxdaXUsW//BzYZF023+cOrUyYPUkw6HgbvBiCJru t/9Q== X-Gm-Message-State: AOJu0YyCxv8LeT5ogFXZc42ZFmDfv3M7K8vKYI83dcLoH1QTG9lt8SRT N5RZaZAEBLua+VyhXuPJ0bnBJpJyWYfmrkz6BHuQISr8KpnGEfjJmR/CbaxwT4oU6iVWPSJ6a3y xNQ== X-Google-Smtp-Source: AGHT+IFmG+p0wfeaPELqYwmEMPdSFi/xoF4jNe4BgG6nNP91y5zUW5SGXD+ETkCiL/VUzo5Vv7dyGg== X-Received: by 2002:a17:903:244b:b0:20c:c18f:c39e with SMTP id d9443c01a7336-211c0fab017mr40978525ad.21.1731578997294; Thu, 14 Nov 2024 02:09:57 -0800 (PST) Received: from sxavier-dev.dhcp.broadcom.net ([192.19.234.250]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-211c7d062absm7251135ad.206.2024.11.14.02.09.55 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 14 Nov 2024 02:09:56 -0800 (PST) From: Selvin Xavier To: leon@kernel.org, jgg@ziepe.ca Cc: linux-rdma@vger.kernel.org, andrew.gospodarek@broadcom.com, kalesh-anakkur.purayil@broadcom.com, Selvin Xavier Subject: [PATCH rdma-next v2 1/4] RDMA/bnxt_re: Fail probe early when not enough MSI-x vectors are reserved Date: Thu, 14 Nov 2024 01:49:05 -0800 Message-Id: <1731577748-1804-2-git-send-email-selvin.xavier@broadcom.com> X-Mailer: git-send-email 2.5.5 In-Reply-To: <1731577748-1804-1-git-send-email-selvin.xavier@broadcom.com> References: <1731577748-1804-1-git-send-email-selvin.xavier@broadcom.com> Precedence: bulk X-Mailing-List: linux-rdma@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: From: Kalesh AP L2 driver allocates and populates the MSI-x vector details for RoCE in the en_dev structure. RoCE driver requires minimum 2 MSIx vectors. Hence during probe, driver has to check and bail out if there are not enough MSI-x vectors reserved for it before proceeding further initialization. Reviewed-by: Andy Gospodarek Reviewed-by: Ajit Khaparde Reviewed-by: Hongguang Gao Reviewed-by: Bhargava Chenna Marreddy Reviewed-by: Kashyap Desai Reviewed-by: Chandramohan Akula Signed-off-by: Kalesh AP Signed-off-by: Selvin Xavier --- drivers/infiniband/hw/bnxt_re/bnxt_re.h | 2 ++ drivers/infiniband/hw/bnxt_re/main.c | 22 ++++++++++++---------- 2 files changed, 14 insertions(+), 10 deletions(-) diff --git a/drivers/infiniband/hw/bnxt_re/bnxt_re.h b/drivers/infiniband/hw/bnxt_re/bnxt_re.h index d1b7c20..7abc37b 100644 --- a/drivers/infiniband/hw/bnxt_re/bnxt_re.h +++ b/drivers/infiniband/hw/bnxt_re/bnxt_re.h @@ -154,6 +154,8 @@ struct bnxt_re_pacing { #define BNXT_RE_GRC_FIFO_REG_BASE 0x2000 +#define BNXT_RE_MIN_MSIX 2 + #define MAX_CQ_HASH_BITS (16) #define MAX_SRQ_HASH_BITS (16) diff --git a/drivers/infiniband/hw/bnxt_re/main.c b/drivers/infiniband/hw/bnxt_re/main.c index cb61941..c262a16 100644 --- a/drivers/infiniband/hw/bnxt_re/main.c +++ b/drivers/infiniband/hw/bnxt_re/main.c @@ -1936,6 +1936,18 @@ static int bnxt_re_dev_init(struct bnxt_re_dev *rdev, u8 op_type) } set_bit(BNXT_RE_FLAG_NETDEV_REGISTERED, &rdev->flags); + if (rdev->en_dev->ulp_tbl->msix_requested < BNXT_RE_MIN_MSIX) { + ibdev_err(&rdev->ibdev, + "RoCE requires minimum 2 MSI-X vectors, but only %d reserved\n", + rdev->en_dev->ulp_tbl->msix_requested); + bnxt_unregister_dev(rdev->en_dev); + clear_bit(BNXT_RE_FLAG_NETDEV_REGISTERED, &rdev->flags); + return -EINVAL; + } + ibdev_dbg(&rdev->ibdev, "Got %d MSI-X vectors\n", + rdev->en_dev->ulp_tbl->msix_requested); + rdev->num_msix = rdev->en_dev->ulp_tbl->msix_requested; + rc = bnxt_re_setup_chip_ctx(rdev); if (rc) { bnxt_unregister_dev(rdev->en_dev); @@ -1947,16 +1959,6 @@ static int bnxt_re_dev_init(struct bnxt_re_dev *rdev, u8 op_type) /* Check whether VF or PF */ bnxt_re_get_sriov_func_type(rdev); - if (!rdev->en_dev->ulp_tbl->msix_requested) { - ibdev_err(&rdev->ibdev, - "Failed to get MSI-X vectors: %#x\n", rc); - rc = -EINVAL; - goto fail; - } - ibdev_dbg(&rdev->ibdev, "Got %d MSI-X vectors\n", - rdev->en_dev->ulp_tbl->msix_requested); - rdev->num_msix = rdev->en_dev->ulp_tbl->msix_requested; - bnxt_re_query_hwrm_intf_version(rdev); /* Establish RCFW Communication Channel to initialize the context From patchwork Thu Nov 14 09:49:06 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Selvin Xavier X-Patchwork-Id: 13874874 Received: from mail-pl1-f173.google.com (mail-pl1-f173.google.com [209.85.214.173]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CBE821F9402 for ; Thu, 14 Nov 2024 10:10:00 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.173 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731579002; cv=none; b=LYyD22mDEWfzGU4Ppx8kVBFCaC7zi1PKNIe7KmVrMgiw+Fj7ds8aQOz4dxSKSfuwLiwG5C6zJW6dlh5SQDP9aGbHCsSX3v+GnK+ulbr3dZ/d9o7Y5nUh6YCNe/H0xhQR8GvP2/pZUF4kR3d1a/euLU2RTScQ6j6kpbs2zgKvDI4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731579002; c=relaxed/simple; bh=EeUiJCqmf6isxVGhFGdJE4ORofhDL1kCqg7tx6zu0bw=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References; b=k5dxsvQ1J1eOI2PhZpfsVT4yiiKMH7HeaHhicXnFOWXW3aSSP2JX65h886/UYtxBaqlqlabIGBDvjBJTIpqwo8f3iM6d4wuhGuqs8/eOD7FAzDgiM1lKo0H/8MGm+9JLMiE+ulwvcaGwBZJgHMYhz4mcCbzh7qsrM0c/MAb2xTc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=broadcom.com; spf=fail smtp.mailfrom=broadcom.com; dkim=pass (1024-bit key) header.d=broadcom.com header.i=@broadcom.com header.b=EpJ0JlVu; arc=none smtp.client-ip=209.85.214.173 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=broadcom.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=broadcom.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=broadcom.com header.i=@broadcom.com header.b="EpJ0JlVu" Received: by mail-pl1-f173.google.com with SMTP id d9443c01a7336-20c6f492d2dso4241575ad.0 for ; Thu, 14 Nov 2024 02:10:00 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; t=1731579000; x=1732183800; darn=vger.kernel.org; h=references:in-reply-to:message-id:date:subject:cc:to:from:from:to :cc:subject:date:message-id:reply-to; bh=xNrcfbkd0EcY0VZf4z3OydrR5ENv5ZcLyUEyn0TusQ8=; b=EpJ0JlVuptMB47FvxLvVSDa8RG89powbnrQKrZ7x/I1iAA+/j5HWk/LmZyZf38Hoo1 o+BM7UMltiJzSNotp1rWiu45rd+EMYvVKp/18BVyj7M9UYFSYm79WBRDpqWGVSlCjElv +oa3IrXKxjw2JqfxX33+CjdXYXZ5Cfr6PBIQo= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1731579000; x=1732183800; h=references:in-reply-to:message-id:date:subject:cc:to:from :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=xNrcfbkd0EcY0VZf4z3OydrR5ENv5ZcLyUEyn0TusQ8=; b=qQRlRLI3M6KoyjH6moXq8mKcgYTwkxUBqMHTwtqe83xyHyrj3TwY2iAg7koMBHcFeP JZbVvEwTvjs/0vKEcDpBx4DzSt85e25lFqXzoxu46tp8APP3/wAEBLq2J/LazlAlxTOC Eh5Vhm0O8gdiozvX67J8mnIic4VBkR20ufEmmIGA4Y+A7UFnZoGcNvjd+ditFe75AB2D I+XdWZs/4hwMpIJ/ByrtuAQzqbLsEhx1h02c38HOI3y0A6zoyQXpOG3D+6aqcYUgcg/9 sn+koUYmCFRvZT+dCDmIZ+NhjulfNsvof9vUiaoKy3BUif/xDERMp5th7CsaXGeFJ7b1 c7AQ== X-Gm-Message-State: AOJu0Yy/bVylTGQ1Fw3eW5X1SYq9u0yoL3KhHo/1lrdEolh9KzxSMxWy dGSZ2HnNVdSsxtmbQd9s4w0bhFcMGpZFnE2IFBQQyPilsZpWdL6QUOAzqcMIMw== X-Google-Smtp-Source: AGHT+IGEgqH0r8gVqaU48foJkuMb6hv75e5WjGAFUSQtohczZB0f77icTblOGXwWBkKeMaUuOUf04A== X-Received: by 2002:a17:902:ef50:b0:1fb:57e7:5bb4 with SMTP id d9443c01a7336-211c5060604mr17004635ad.37.1731579000040; Thu, 14 Nov 2024 02:10:00 -0800 (PST) Received: from sxavier-dev.dhcp.broadcom.net ([192.19.234.250]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-211c7d062absm7251135ad.206.2024.11.14.02.09.57 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 14 Nov 2024 02:09:59 -0800 (PST) From: Selvin Xavier To: leon@kernel.org, jgg@ziepe.ca Cc: linux-rdma@vger.kernel.org, andrew.gospodarek@broadcom.com, kalesh-anakkur.purayil@broadcom.com, Selvin Xavier Subject: [PATCH rdma-next v2 2/4] RDMA/bnxt_re: Refactor NQ allocation Date: Thu, 14 Nov 2024 01:49:06 -0800 Message-Id: <1731577748-1804-3-git-send-email-selvin.xavier@broadcom.com> X-Mailer: git-send-email 2.5.5 In-Reply-To: <1731577748-1804-1-git-send-email-selvin.xavier@broadcom.com> References: <1731577748-1804-1-git-send-email-selvin.xavier@broadcom.com> Precedence: bulk X-Mailing-List: linux-rdma@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: From: Kalesh AP Move NQ related data structures from rdev to a new structure named "struct bnxt_re_nq_record" by keeping a pointer to in the rdev structure. Allocate the memory for it dynamically. This change is needed for subsequent patches in the series. Also, removed the nq_task variable from rdev structure as it is redundant and no longer used. This change would help to reduce the size of the driver private structure as well. Reviewed-by: Chandramohan Akula Signed-off-by: Kalesh AP Signed-off-by: Selvin Xavier --- drivers/infiniband/hw/bnxt_re/bnxt_re.h | 13 +++--- drivers/infiniband/hw/bnxt_re/ib_verbs.c | 6 +-- drivers/infiniband/hw/bnxt_re/main.c | 74 +++++++++++++++++++++----------- 3 files changed, 60 insertions(+), 33 deletions(-) diff --git a/drivers/infiniband/hw/bnxt_re/bnxt_re.h b/drivers/infiniband/hw/bnxt_re/bnxt_re.h index 7abc37b..d1c839e 100644 --- a/drivers/infiniband/hw/bnxt_re/bnxt_re.h +++ b/drivers/infiniband/hw/bnxt_re/bnxt_re.h @@ -155,6 +155,11 @@ struct bnxt_re_pacing { #define BNXT_RE_GRC_FIFO_REG_BASE 0x2000 #define BNXT_RE_MIN_MSIX 2 +#define BNXT_RE_MAX_MSIX BNXT_MAX_ROCE_MSIX +struct bnxt_re_nq_record { + struct bnxt_qplib_nq nq[BNXT_RE_MAX_MSIX]; + int num_msix; +}; #define MAX_CQ_HASH_BITS (16) #define MAX_SRQ_HASH_BITS (16) @@ -183,21 +188,17 @@ struct bnxt_re_dev { unsigned int version, major, minor; struct bnxt_qplib_chip_ctx *chip_ctx; struct bnxt_en_dev *en_dev; - int num_msix; int id; struct delayed_work worker; u8 cur_prio_map; - /* FP Notification Queue (CQ & SRQ) */ - struct tasklet_struct nq_task; - /* RCFW Channel */ struct bnxt_qplib_rcfw rcfw; - /* NQ */ - struct bnxt_qplib_nq nq[BNXT_MAX_ROCE_MSIX]; + /* NQ record */ + struct bnxt_re_nq_record *nqr; /* Device Resources */ struct bnxt_qplib_dev_attr dev_attr; diff --git a/drivers/infiniband/hw/bnxt_re/ib_verbs.c b/drivers/infiniband/hw/bnxt_re/ib_verbs.c index 9a188cc..a9c32c0 100644 --- a/drivers/infiniband/hw/bnxt_re/ib_verbs.c +++ b/drivers/infiniband/hw/bnxt_re/ib_verbs.c @@ -1872,8 +1872,8 @@ int bnxt_re_create_srq(struct ib_srq *ib_srq, srq->qplib_srq.wqe_size = bnxt_re_get_rwqe_size(dev_attr->max_srq_sges); srq->qplib_srq.threshold = srq_init_attr->attr.srq_limit; srq->srq_limit = srq_init_attr->attr.srq_limit; - srq->qplib_srq.eventq_hw_ring_id = rdev->nq[0].ring_id; - nq = &rdev->nq[0]; + srq->qplib_srq.eventq_hw_ring_id = rdev->nqr->nq[0].ring_id; + nq = &rdev->nqr->nq[0]; if (udata) { rc = bnxt_re_init_user_srq(rdev, pd, srq, udata); @@ -3122,7 +3122,7 @@ int bnxt_re_create_cq(struct ib_cq *ibcq, const struct ib_cq_init_attr *attr, * used for getting the NQ index. */ nq_alloc_cnt = atomic_inc_return(&rdev->nq_alloc_cnt); - nq = &rdev->nq[nq_alloc_cnt % (rdev->num_msix - 1)]; + nq = &rdev->nqr->nq[nq_alloc_cnt % (rdev->nqr->num_msix - 1)]; cq->qplib_cq.max_wqe = entries; cq->qplib_cq.cnq_hw_ring_id = nq->ring_id; cq->qplib_cq.nq = nq; diff --git a/drivers/infiniband/hw/bnxt_re/main.c b/drivers/infiniband/hw/bnxt_re/main.c index c262a16..9669def 100644 --- a/drivers/infiniband/hw/bnxt_re/main.c +++ b/drivers/infiniband/hw/bnxt_re/main.c @@ -326,8 +326,8 @@ static void bnxt_re_stop_irq(void *handle) rdev = en_info->rdev; rcfw = &rdev->rcfw; - for (indx = BNXT_RE_NQ_IDX; indx < rdev->num_msix; indx++) { - nq = &rdev->nq[indx - 1]; + for (indx = BNXT_RE_NQ_IDX; indx < rdev->nqr->num_msix; indx++) { + nq = &rdev->nqr->nq[indx - 1]; bnxt_qplib_nq_stop_irq(nq, false); } @@ -362,7 +362,7 @@ static void bnxt_re_start_irq(void *handle, struct bnxt_msix_entry *ent) /* Vectors may change after restart, so update with new vectors * in device sctructure. */ - for (indx = 0; indx < rdev->num_msix; indx++) + for (indx = 0; indx < rdev->nqr->num_msix; indx++) rdev->en_dev->msix_entries[indx].vector = ent[indx].vector; rc = bnxt_qplib_rcfw_start_irq(rcfw, msix_ent[BNXT_RE_AEQ_IDX].vector, @@ -371,8 +371,8 @@ static void bnxt_re_start_irq(void *handle, struct bnxt_msix_entry *ent) ibdev_warn(&rdev->ibdev, "Failed to reinit CREQ\n"); return; } - for (indx = BNXT_RE_NQ_IDX ; indx < rdev->num_msix; indx++) { - nq = &rdev->nq[indx - 1]; + for (indx = BNXT_RE_NQ_IDX ; indx < rdev->nqr->num_msix; indx++) { + nq = &rdev->nqr->nq[indx - 1]; rc = bnxt_qplib_nq_start_irq(nq, indx - 1, msix_ent[indx].vector, false); if (rc) { @@ -1206,7 +1206,7 @@ static int bnxt_re_register_ib(struct bnxt_re_dev *rdev) addrconf_addr_eui48((u8 *)&ibdev->node_guid, rdev->netdev->dev_addr); - ibdev->num_comp_vectors = rdev->num_msix - 1; + ibdev->num_comp_vectors = rdev->nqr->num_msix - 1; ibdev->dev.parent = &rdev->en_dev->pdev->dev; ibdev->local_dma_lkey = BNXT_QPLIB_RSVD_LKEY; @@ -1551,8 +1551,8 @@ static void bnxt_re_cleanup_res(struct bnxt_re_dev *rdev) { int i; - for (i = 1; i < rdev->num_msix; i++) - bnxt_qplib_disable_nq(&rdev->nq[i - 1]); + for (i = 1; i < rdev->nqr->num_msix; i++) + bnxt_qplib_disable_nq(&rdev->nqr->nq[i - 1]); if (rdev->qplib_res.rcfw) bnxt_qplib_cleanup_res(&rdev->qplib_res); @@ -1566,9 +1566,9 @@ static int bnxt_re_init_res(struct bnxt_re_dev *rdev) bnxt_qplib_init_res(&rdev->qplib_res); - for (i = 1; i < rdev->num_msix ; i++) { + for (i = 1; i < rdev->nqr->num_msix ; i++) { db_offt = rdev->en_dev->msix_entries[i].db_offset; - rc = bnxt_qplib_enable_nq(rdev->en_dev->pdev, &rdev->nq[i - 1], + rc = bnxt_qplib_enable_nq(rdev->en_dev->pdev, &rdev->nqr->nq[i - 1], i - 1, rdev->en_dev->msix_entries[i].vector, db_offt, &bnxt_re_cqn_handler, &bnxt_re_srqn_handler); @@ -1582,20 +1582,22 @@ static int bnxt_re_init_res(struct bnxt_re_dev *rdev) return 0; fail: for (i = num_vec_enabled; i >= 0; i--) - bnxt_qplib_disable_nq(&rdev->nq[i]); + bnxt_qplib_disable_nq(&rdev->nqr->nq[i]); return rc; } static void bnxt_re_free_nq_res(struct bnxt_re_dev *rdev) { + struct bnxt_qplib_nq *nq; u8 type; int i; - for (i = 0; i < rdev->num_msix - 1; i++) { + for (i = 0; i < rdev->nqr->num_msix - 1; i++) { type = bnxt_qplib_get_ring_type(rdev->chip_ctx); - bnxt_re_net_ring_free(rdev, rdev->nq[i].ring_id, type); - bnxt_qplib_free_nq(&rdev->nq[i]); - rdev->nq[i].res = NULL; + nq = &rdev->nqr->nq[i]; + bnxt_re_net_ring_free(rdev, nq->ring_id, type); + bnxt_qplib_free_nq(nq); + nq->res = NULL; } } @@ -1637,12 +1639,12 @@ static int bnxt_re_alloc_res(struct bnxt_re_dev *rdev) if (rc) goto dealloc_res; - for (i = 0; i < rdev->num_msix - 1; i++) { + for (i = 0; i < rdev->nqr->num_msix - 1; i++) { struct bnxt_qplib_nq *nq; - nq = &rdev->nq[i]; + nq = &rdev->nqr->nq[i]; nq->hwq.max_elements = BNXT_QPLIB_NQE_MAX_CNT; - rc = bnxt_qplib_alloc_nq(&rdev->qplib_res, &rdev->nq[i]); + rc = bnxt_qplib_alloc_nq(&rdev->qplib_res, nq); if (rc) { ibdev_err(&rdev->ibdev, "Alloc Failed NQ%d rc:%#x", i, rc); @@ -1650,7 +1652,7 @@ static int bnxt_re_alloc_res(struct bnxt_re_dev *rdev) } type = bnxt_qplib_get_ring_type(rdev->chip_ctx); rattr.dma_arr = nq->hwq.pbl[PBL_LVL_0].pg_map_arr; - rattr.pages = nq->hwq.pbl[rdev->nq[i].hwq.level].pg_count; + rattr.pages = nq->hwq.pbl[rdev->nqr->nq[i].hwq.level].pg_count; rattr.type = type; rattr.mode = RING_ALLOC_REQ_INT_MODE_MSIX; rattr.depth = BNXT_QPLIB_NQE_MAX_CNT - 1; @@ -1660,7 +1662,7 @@ static int bnxt_re_alloc_res(struct bnxt_re_dev *rdev) ibdev_err(&rdev->ibdev, "Failed to allocate NQ fw id with rc = 0x%x", rc); - bnxt_qplib_free_nq(&rdev->nq[i]); + bnxt_qplib_free_nq(nq); goto free_nq; } num_vec_created++; @@ -1669,8 +1671,8 @@ static int bnxt_re_alloc_res(struct bnxt_re_dev *rdev) free_nq: for (i = num_vec_created - 1; i >= 0; i--) { type = bnxt_qplib_get_ring_type(rdev->chip_ctx); - bnxt_re_net_ring_free(rdev, rdev->nq[i].ring_id, type); - bnxt_qplib_free_nq(&rdev->nq[i]); + bnxt_re_net_ring_free(rdev, rdev->nqr->nq[i].ring_id, type); + bnxt_qplib_free_nq(&rdev->nqr->nq[i]); } bnxt_qplib_dealloc_dpi(&rdev->qplib_res, &rdev->dpi_privileged); @@ -1865,6 +1867,21 @@ static int bnxt_re_ib_init(struct bnxt_re_dev *rdev) return rc; } +static int bnxt_re_alloc_nqr_mem(struct bnxt_re_dev *rdev) +{ + rdev->nqr = kzalloc(sizeof(*rdev->nqr), GFP_KERNEL); + if (!rdev->nqr) + return -ENOMEM; + + return 0; +} + +static void bnxt_re_free_nqr_mem(struct bnxt_re_dev *rdev) +{ + kfree(rdev->nqr); + rdev->nqr = NULL; +} + static void bnxt_re_dev_uninit(struct bnxt_re_dev *rdev, u8 op_type) { u8 type; @@ -1894,11 +1911,12 @@ static void bnxt_re_dev_uninit(struct bnxt_re_dev *rdev, u8 op_type) bnxt_qplib_free_rcfw_channel(&rdev->rcfw); } - rdev->num_msix = 0; + rdev->nqr->num_msix = 0; if (rdev->pacing.dbr_pacing) bnxt_re_deinitialize_dbr_pacing(rdev); + bnxt_re_free_nqr_mem(rdev); bnxt_re_destroy_chip_ctx(rdev); if (op_type == BNXT_RE_COMPLETE_REMOVE) { if (test_and_clear_bit(BNXT_RE_FLAG_NETDEV_REGISTERED, &rdev->flags)) @@ -1946,7 +1964,6 @@ static int bnxt_re_dev_init(struct bnxt_re_dev *rdev, u8 op_type) } ibdev_dbg(&rdev->ibdev, "Got %d MSI-X vectors\n", rdev->en_dev->ulp_tbl->msix_requested); - rdev->num_msix = rdev->en_dev->ulp_tbl->msix_requested; rc = bnxt_re_setup_chip_ctx(rdev); if (rc) { @@ -1956,6 +1973,15 @@ static int bnxt_re_dev_init(struct bnxt_re_dev *rdev, u8 op_type) return -EINVAL; } + rc = bnxt_re_alloc_nqr_mem(rdev); + if (rc) { + bnxt_re_destroy_chip_ctx(rdev); + bnxt_unregister_dev(rdev->en_dev); + clear_bit(BNXT_RE_FLAG_NETDEV_REGISTERED, &rdev->flags); + return rc; + } + rdev->nqr->num_msix = rdev->en_dev->ulp_tbl->msix_requested; + /* Check whether VF or PF */ bnxt_re_get_sriov_func_type(rdev); From patchwork Thu Nov 14 09:49:07 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Selvin Xavier X-Patchwork-Id: 13874875 Received: from mail-pl1-f170.google.com (mail-pl1-f170.google.com [209.85.214.170]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6E2DA1F942D for ; Thu, 14 Nov 2024 10:10:03 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.170 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731579005; cv=none; b=pVc81N/IqMKCm1QStOQ3hsHV3OOAJK+44bv0pvt9GvgrgSB5kg6FGgkwA1uwY3t07cpXKtlxkxyQNUupUZAIh5FSj45cs5fUEDUYiN163mUEgC2u/NgAYx5YWBz/Xs9IsxVRt45X+wsxu4ppWUMGMk6zTNb0B+wqT6r4kdVxmfE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731579005; c=relaxed/simple; bh=baxiIC1nPKsiXM67V+rYXNFN5IskF/tU2Kspg9dAL4Q=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References; b=JgW63Sjig3TF0fcsbOcUyG2WgpzqffbnyCNqmX8S2RGDOJn/fKHlC7pm312hw6mWf2kz0ZGT3+Y86ArmjISgjfdC+GFwE8eP+3g99X0WnTEXp/3DEnTA/MbUePFZefxIwmbbtKzPfujZrXoGqc72hWTksyGr0mCRUK2FZq9h/Vo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=broadcom.com; spf=fail smtp.mailfrom=broadcom.com; dkim=pass (1024-bit key) header.d=broadcom.com header.i=@broadcom.com header.b=Pu+JYaJS; arc=none smtp.client-ip=209.85.214.170 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=broadcom.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=broadcom.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=broadcom.com header.i=@broadcom.com header.b="Pu+JYaJS" Received: by mail-pl1-f170.google.com with SMTP id d9443c01a7336-20cb47387ceso4523865ad.1 for ; Thu, 14 Nov 2024 02:10:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; t=1731579003; x=1732183803; darn=vger.kernel.org; h=references:in-reply-to:message-id:date:subject:cc:to:from:from:to :cc:subject:date:message-id:reply-to; bh=Gk841z4BLx62XFc2h9J4aU0K9bzkZNC3kcYlHZpWf9g=; b=Pu+JYaJSMdGW666Wse0deNwQ+5oU/qM7C1gCA/5JpHoLgrudT5CCp//wyTAODeVx+0 9UbO+ATys24fMTylmvP886PxjYosCUptD17MDZJBMZYxHgnimryv6SoP7W6IQLNBgVvu UIGGAokUpkwEIl8geVG1NOl3AbS9A9cQd2Mto= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1731579003; x=1732183803; h=references:in-reply-to:message-id:date:subject:cc:to:from :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Gk841z4BLx62XFc2h9J4aU0K9bzkZNC3kcYlHZpWf9g=; b=K+pqfdeAPXY+Ov0cnl1jek7LraetzSLcDEeCH83jyhGKKLzq3S2u+lXI5FF8ozzx6U B6gXuNcEuIhqFkD33tkJm2hYIB8C3fNBF3fYmV1vdYErYAPkBUVzxkKFGoSpCDQ1pP81 1aXuhO/Q/TQBdijNn6VaE84DPO+gEuRYDyYfO69rSSF52Kk2WbLq7NS8lbyTViwmPD3s Wt8a8ZLSzAGJHT1+NjtfyZq3zfNEJNPiiv4We+1REC5SmYp851jqsJyb5EV4B2H9dbbM 2p4OBoP7wsFB3hf73/bJY7BaH/tsIIKFqwG9Sd291lBq4pSkXcqV4WvCyLu1tG32NR7s lXIQ== X-Gm-Message-State: AOJu0YxIxFh0qAjr2Ai1JhIgRj/Qdo4iz5YRM/Mpq+SN4VIWNzoYdhgc K2hsbkcRM+LUfmsCyqlD2O41l6feli9kiaTB37R8oPKVUfD6+o2Mt2e5ai0LJg== X-Google-Smtp-Source: AGHT+IFvFl0NnfUqxRhoxrkc28locKjNC1fqUoEDd2eLqFdjd6teAY0mfAEcJiwB2qY16kKhAttoBw== X-Received: by 2002:a17:903:1cf:b0:20c:f39e:4c15 with SMTP id d9443c01a7336-211835268a9mr346791105ad.22.1731579002683; Thu, 14 Nov 2024 02:10:02 -0800 (PST) Received: from sxavier-dev.dhcp.broadcom.net ([192.19.234.250]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-211c7d062absm7251135ad.206.2024.11.14.02.10.00 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 14 Nov 2024 02:10:02 -0800 (PST) From: Selvin Xavier To: leon@kernel.org, jgg@ziepe.ca Cc: linux-rdma@vger.kernel.org, andrew.gospodarek@broadcom.com, kalesh-anakkur.purayil@broadcom.com, Selvin Xavier Subject: [PATCH rdma-next v2 3/4] RDMA/bnxt_re: Refurbish CQ to NQ hash calculation Date: Thu, 14 Nov 2024 01:49:07 -0800 Message-Id: <1731577748-1804-4-git-send-email-selvin.xavier@broadcom.com> X-Mailer: git-send-email 2.5.5 In-Reply-To: <1731577748-1804-1-git-send-email-selvin.xavier@broadcom.com> References: <1731577748-1804-1-git-send-email-selvin.xavier@broadcom.com> Precedence: bulk X-Mailing-List: linux-rdma@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: From: Kalesh AP There are few use cases where CQ create and destroy is seen before re-creating the CQ, this kind of use case is disturbing the RR distribution and all the active CQ getting mapped to only 2 NQ alternatively. Fixing the CQ to NQ hash calculation by implementing a quick load sorting mechanism under a mutex. Using this, if the CQ was allocated and destroyed before using it, the nq selecting algorithm still obtains the least loaded CQ. Thus balancing the load on NQs. Signed-off-by: Selvin Xavier Signed-off-by: Kalesh AP --- drivers/infiniband/hw/bnxt_re/bnxt_re.h | 2 ++ drivers/infiniband/hw/bnxt_re/ib_verbs.c | 37 ++++++++++++++++++++++---------- drivers/infiniband/hw/bnxt_re/main.c | 2 ++ drivers/infiniband/hw/bnxt_re/qplib_fp.c | 1 + drivers/infiniband/hw/bnxt_re/qplib_fp.h | 1 + 5 files changed, 32 insertions(+), 11 deletions(-) diff --git a/drivers/infiniband/hw/bnxt_re/bnxt_re.h b/drivers/infiniband/hw/bnxt_re/bnxt_re.h index d1c839e..5f64fc4 100644 --- a/drivers/infiniband/hw/bnxt_re/bnxt_re.h +++ b/drivers/infiniband/hw/bnxt_re/bnxt_re.h @@ -159,6 +159,8 @@ struct bnxt_re_pacing { struct bnxt_re_nq_record { struct bnxt_qplib_nq nq[BNXT_RE_MAX_MSIX]; int num_msix; + /* serialize NQ access */ + struct mutex load_lock; }; #define MAX_CQ_HASH_BITS (16) diff --git a/drivers/infiniband/hw/bnxt_re/ib_verbs.c b/drivers/infiniband/hw/bnxt_re/ib_verbs.c index a9c32c0..f6e9eef 100644 --- a/drivers/infiniband/hw/bnxt_re/ib_verbs.c +++ b/drivers/infiniband/hw/bnxt_re/ib_verbs.c @@ -3029,6 +3029,28 @@ int bnxt_re_post_recv(struct ib_qp *ib_qp, const struct ib_recv_wr *wr, return rc; } +static struct bnxt_qplib_nq *bnxt_re_get_nq(struct bnxt_re_dev *rdev) +{ + int min, indx; + + mutex_lock(&rdev->nqr->load_lock); + for (indx = 0, min = 0; indx < (rdev->nqr->num_msix - 1); indx++) { + if (rdev->nqr->nq[min].load > rdev->nqr->nq[indx].load) + min = indx; + } + rdev->nqr->nq[min].load++; + mutex_unlock(&rdev->nqr->load_lock); + + return &rdev->nqr->nq[min]; +} + +static void bnxt_re_put_nq(struct bnxt_re_dev *rdev, struct bnxt_qplib_nq *nq) +{ + mutex_lock(&rdev->nqr->load_lock); + nq->load--; + mutex_unlock(&rdev->nqr->load_lock); +} + /* Completion Queues */ int bnxt_re_destroy_cq(struct ib_cq *ib_cq, struct ib_udata *udata) { @@ -3047,6 +3069,8 @@ int bnxt_re_destroy_cq(struct ib_cq *ib_cq, struct ib_udata *udata) hash_del(&cq->hash_entry); } bnxt_qplib_destroy_cq(&rdev->qplib_res, &cq->qplib_cq); + + bnxt_re_put_nq(rdev, nq); ib_umem_release(cq->umem); atomic_dec(&rdev->stats.res.cq_count); @@ -3065,8 +3089,6 @@ int bnxt_re_create_cq(struct ib_cq *ibcq, const struct ib_cq_init_attr *attr, rdma_udata_to_drv_context(udata, struct bnxt_re_ucontext, ib_uctx); struct bnxt_qplib_dev_attr *dev_attr = &rdev->dev_attr; struct bnxt_qplib_chip_ctx *cctx; - struct bnxt_qplib_nq *nq = NULL; - unsigned int nq_alloc_cnt; int cqe = attr->cqe; int rc, entries; u32 active_cqs; @@ -3117,16 +3139,10 @@ int bnxt_re_create_cq(struct ib_cq *ibcq, const struct ib_cq_init_attr *attr, cq->qplib_cq.dpi = &rdev->dpi_privileged; } - /* - * Allocating the NQ in a round robin fashion. nq_alloc_cnt is a - * used for getting the NQ index. - */ - nq_alloc_cnt = atomic_inc_return(&rdev->nq_alloc_cnt); - nq = &rdev->nqr->nq[nq_alloc_cnt % (rdev->nqr->num_msix - 1)]; cq->qplib_cq.max_wqe = entries; - cq->qplib_cq.cnq_hw_ring_id = nq->ring_id; - cq->qplib_cq.nq = nq; cq->qplib_cq.coalescing = &rdev->cq_coalescing; + cq->qplib_cq.nq = bnxt_re_get_nq(rdev); + cq->qplib_cq.cnq_hw_ring_id = cq->qplib_cq.nq->ring_id; rc = bnxt_qplib_create_cq(&rdev->qplib_res, &cq->qplib_cq); if (rc) { @@ -3136,7 +3152,6 @@ int bnxt_re_create_cq(struct ib_cq *ibcq, const struct ib_cq_init_attr *attr, cq->ib_cq.cqe = entries; cq->cq_period = cq->qplib_cq.period; - nq->budget++; active_cqs = atomic_inc_return(&rdev->stats.res.cq_count); if (active_cqs > rdev->stats.res.cq_watermark) diff --git a/drivers/infiniband/hw/bnxt_re/main.c b/drivers/infiniband/hw/bnxt_re/main.c index 9669def..fcaf2b3 100644 --- a/drivers/infiniband/hw/bnxt_re/main.c +++ b/drivers/infiniband/hw/bnxt_re/main.c @@ -1566,6 +1566,8 @@ static int bnxt_re_init_res(struct bnxt_re_dev *rdev) bnxt_qplib_init_res(&rdev->qplib_res); + mutex_init(&rdev->nqr->load_lock); + for (i = 1; i < rdev->nqr->num_msix ; i++) { db_offt = rdev->en_dev->msix_entries[i].db_offset; rc = bnxt_qplib_enable_nq(rdev->en_dev->pdev, &rdev->nqr->nq[i - 1], diff --git a/drivers/infiniband/hw/bnxt_re/qplib_fp.c b/drivers/infiniband/hw/bnxt_re/qplib_fp.c index e2eea71..e56f42f 100644 --- a/drivers/infiniband/hw/bnxt_re/qplib_fp.c +++ b/drivers/infiniband/hw/bnxt_re/qplib_fp.c @@ -551,6 +551,7 @@ int bnxt_qplib_enable_nq(struct pci_dev *pdev, struct bnxt_qplib_nq *nq, nq->pdev = pdev; nq->cqn_handler = cqn_handler; nq->srqn_handler = srqn_handler; + nq->load = 0; /* Have a task to schedule CQ notifiers in post send case */ nq->cqn_wq = create_singlethread_workqueue("bnxt_qplib_nq"); diff --git a/drivers/infiniband/hw/bnxt_re/qplib_fp.h b/drivers/infiniband/hw/bnxt_re/qplib_fp.h index b5a90581..8ff56d7 100644 --- a/drivers/infiniband/hw/bnxt_re/qplib_fp.h +++ b/drivers/infiniband/hw/bnxt_re/qplib_fp.h @@ -519,6 +519,7 @@ struct bnxt_qplib_nq { struct tasklet_struct nq_tasklet; bool requested; int budget; + u32 load; cqn_handler_t cqn_handler; srqn_handler_t srqn_handler; From patchwork Thu Nov 14 09:49:08 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Selvin Xavier X-Patchwork-Id: 13874876 Received: from mail-pl1-f182.google.com (mail-pl1-f182.google.com [209.85.214.182]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id EB1CF1F9402 for ; Thu, 14 Nov 2024 10:10:05 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.182 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731579007; cv=none; b=jNbwtxi+38UZulDJbSAKsutCMdZqFoB/DtyKXBLSYx4cJBiJbc1B3I7VDadKfiLxLsqqXIdWRFQywMRy4qoc3MoCzz8diYGLPVZSc7NE+YgW9zmed/oz8mkqo8YNWr6r7o0n6erQsVmwLI4YPhCMDgncJCNf9djDvHeEt/fn9cA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731579007; c=relaxed/simple; bh=5lOkGFyJrdaN2V/I0r/cxjezp4xMm0O7wmVD3juhGlI=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References; b=olrWg8UOlJDUibpW+2rIzh4X/a33SC8DBgSv9aLi5Y7GuYnb1NxgN6PhJcfWFr3ZkliYGRyRSNH7K7wu3eaAUh3S3jny03bYIWQGuP4i7k/5xtc1HFgHupcER36441ktl1HOKoa5hipy6uDeaAx+h6uhzCVZxSxflq7lLqeLaRc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=broadcom.com; spf=fail smtp.mailfrom=broadcom.com; dkim=pass (1024-bit key) header.d=broadcom.com header.i=@broadcom.com header.b=RfcAopzE; arc=none smtp.client-ip=209.85.214.182 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=broadcom.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=broadcom.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=broadcom.com header.i=@broadcom.com header.b="RfcAopzE" Received: by mail-pl1-f182.google.com with SMTP id d9443c01a7336-20c693b68f5so4210465ad.1 for ; Thu, 14 Nov 2024 02:10:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; t=1731579005; x=1732183805; darn=vger.kernel.org; h=references:in-reply-to:message-id:date:subject:cc:to:from:from:to :cc:subject:date:message-id:reply-to; bh=DbScZsKBGM/Zlei9B1dlGPa7BBIqOFtWKOcxzU+BIkI=; b=RfcAopzE8O8zH6O88MegEkNhqNNod5n3Hutefp0zR/OpWC2/0JupUIreMZgIKBUlFb pgoDWl8lCQ77s2MTiO3DxpM58X3523I81P0xcnjrjH8kwqFE21xeM6JF6pAH6Z3saj13 cursQQ7gPfGAC7H8ROjRGTu8JwCaF7s613TUg= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1731579005; x=1732183805; h=references:in-reply-to:message-id:date:subject:cc:to:from :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=DbScZsKBGM/Zlei9B1dlGPa7BBIqOFtWKOcxzU+BIkI=; b=PdwnBO9CSwd3QT52fCe63fo/RvsgVERmm5DQUzQ2QeoLEkAC70YjZk+kc8poSuF6pQ Jl9+pT/peKQBMnPkc9EqxMMXeGYxIWyQWgSDCshZ18k4o6eSy2x8UnIFxQbz7CRwYj1w 65jQmXE3F+G61ArgqodvFXNCPgC2r3n3EKtXmykzxAuKVRj+5wYMNHP84DQ7I89dZ2ZF orSCyshMRGcOmzYVLaV6bwgKeYJdV8Q5Y1xW1H0Rh+S9UrIfUt/DVv62RnBLxCI0dDYy xAOzV4whaERIM+umpyhGhxw7z7T2YwWTAGoBdZh49l4mVAJ0Vwvp1Fjv0OVd1qRkj93n YoPg== X-Gm-Message-State: AOJu0YyeDRc3Qfvt5kPj3H0EGOY/AzdVYi7leCTGIH1ehjORpXwXHtXM RoU2l9xVp1AkuWIjXxi85nP9VZ3dhJxq4mZTk3PPn2JX2yLnn7NvalVF8muUGA== X-Google-Smtp-Source: AGHT+IGB0DgHcfEdKjHXOnZYsRrLWagXYAu4Hp97gHlAF3+U3IhkbI8u2jakdKYdNQ4KLqzrUpReAw== X-Received: by 2002:a17:902:ec8f:b0:211:31ac:89f9 with SMTP id d9443c01a7336-211b5d6b0c7mr64716195ad.55.1731579005343; Thu, 14 Nov 2024 02:10:05 -0800 (PST) Received: from sxavier-dev.dhcp.broadcom.net ([192.19.234.250]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-211c7d062absm7251135ad.206.2024.11.14.02.10.03 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 14 Nov 2024 02:10:04 -0800 (PST) From: Selvin Xavier To: leon@kernel.org, jgg@ziepe.ca Cc: linux-rdma@vger.kernel.org, andrew.gospodarek@broadcom.com, kalesh-anakkur.purayil@broadcom.com, Selvin Xavier Subject: [PATCH rdma-next v2 4/4] RDMA/bnxt_re: Cache MSIx info to a local structure Date: Thu, 14 Nov 2024 01:49:08 -0800 Message-Id: <1731577748-1804-5-git-send-email-selvin.xavier@broadcom.com> X-Mailer: git-send-email 2.5.5 In-Reply-To: <1731577748-1804-1-git-send-email-selvin.xavier@broadcom.com> References: <1731577748-1804-1-git-send-email-selvin.xavier@broadcom.com> Precedence: bulk X-Mailing-List: linux-rdma@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: From: Kalesh AP L2 driver allocates the vectors for RoCE and pass it through the en_dev structure to RoCE. During probe, cache the MSIx related info to a local structure. Signed-off-by: Selvin Xavier Signed-off-by: Kalesh AP --- drivers/infiniband/hw/bnxt_re/bnxt_re.h | 1 + drivers/infiniband/hw/bnxt_re/main.c | 18 ++++++++++-------- 2 files changed, 11 insertions(+), 8 deletions(-) diff --git a/drivers/infiniband/hw/bnxt_re/bnxt_re.h b/drivers/infiniband/hw/bnxt_re/bnxt_re.h index 5f64fc4..2975b11 100644 --- a/drivers/infiniband/hw/bnxt_re/bnxt_re.h +++ b/drivers/infiniband/hw/bnxt_re/bnxt_re.h @@ -157,6 +157,7 @@ struct bnxt_re_pacing { #define BNXT_RE_MIN_MSIX 2 #define BNXT_RE_MAX_MSIX BNXT_MAX_ROCE_MSIX struct bnxt_re_nq_record { + struct bnxt_msix_entry msix_entries[BNXT_RE_MAX_MSIX]; struct bnxt_qplib_nq nq[BNXT_RE_MAX_MSIX]; int num_msix; /* serialize NQ access */ diff --git a/drivers/infiniband/hw/bnxt_re/main.c b/drivers/infiniband/hw/bnxt_re/main.c index fcaf2b3..533b9f1 100644 --- a/drivers/infiniband/hw/bnxt_re/main.c +++ b/drivers/infiniband/hw/bnxt_re/main.c @@ -347,7 +347,7 @@ static void bnxt_re_start_irq(void *handle, struct bnxt_msix_entry *ent) return; rdev = en_info->rdev; - msix_ent = rdev->en_dev->msix_entries; + msix_ent = rdev->nqr->msix_entries; rcfw = &rdev->rcfw; if (!ent) { /* Not setting the f/w timeout bit in rcfw. @@ -363,7 +363,7 @@ static void bnxt_re_start_irq(void *handle, struct bnxt_msix_entry *ent) * in device sctructure. */ for (indx = 0; indx < rdev->nqr->num_msix; indx++) - rdev->en_dev->msix_entries[indx].vector = ent[indx].vector; + rdev->nqr->msix_entries[indx].vector = ent[indx].vector; rc = bnxt_qplib_rcfw_start_irq(rcfw, msix_ent[BNXT_RE_AEQ_IDX].vector, false); @@ -1569,9 +1569,9 @@ static int bnxt_re_init_res(struct bnxt_re_dev *rdev) mutex_init(&rdev->nqr->load_lock); for (i = 1; i < rdev->nqr->num_msix ; i++) { - db_offt = rdev->en_dev->msix_entries[i].db_offset; + db_offt = rdev->nqr->msix_entries[i].db_offset; rc = bnxt_qplib_enable_nq(rdev->en_dev->pdev, &rdev->nqr->nq[i - 1], - i - 1, rdev->en_dev->msix_entries[i].vector, + i - 1, rdev->nqr->msix_entries[i].vector, db_offt, &bnxt_re_cqn_handler, &bnxt_re_srqn_handler); if (rc) { @@ -1658,7 +1658,7 @@ static int bnxt_re_alloc_res(struct bnxt_re_dev *rdev) rattr.type = type; rattr.mode = RING_ALLOC_REQ_INT_MODE_MSIX; rattr.depth = BNXT_QPLIB_NQE_MAX_CNT - 1; - rattr.lrid = rdev->en_dev->msix_entries[i + 1].ring_idx; + rattr.lrid = rdev->nqr->msix_entries[i + 1].ring_idx; rc = bnxt_re_net_ring_alloc(rdev, &rattr, &nq->ring_id); if (rc) { ibdev_err(&rdev->ibdev, @@ -1983,6 +1983,8 @@ static int bnxt_re_dev_init(struct bnxt_re_dev *rdev, u8 op_type) return rc; } rdev->nqr->num_msix = rdev->en_dev->ulp_tbl->msix_requested; + memcpy(rdev->nqr->msix_entries, rdev->en_dev->msix_entries, + sizeof(struct bnxt_msix_entry) * rdev->nqr->num_msix); /* Check whether VF or PF */ bnxt_re_get_sriov_func_type(rdev); @@ -2008,14 +2010,14 @@ static int bnxt_re_dev_init(struct bnxt_re_dev *rdev, u8 op_type) rattr.type = type; rattr.mode = RING_ALLOC_REQ_INT_MODE_MSIX; rattr.depth = BNXT_QPLIB_CREQE_MAX_CNT - 1; - rattr.lrid = rdev->en_dev->msix_entries[BNXT_RE_AEQ_IDX].ring_idx; + rattr.lrid = rdev->nqr->msix_entries[BNXT_RE_AEQ_IDX].ring_idx; rc = bnxt_re_net_ring_alloc(rdev, &rattr, &creq->ring_id); if (rc) { ibdev_err(&rdev->ibdev, "Failed to allocate CREQ: %#x\n", rc); goto free_rcfw; } - db_offt = rdev->en_dev->msix_entries[BNXT_RE_AEQ_IDX].db_offset; - vid = rdev->en_dev->msix_entries[BNXT_RE_AEQ_IDX].vector; + db_offt = rdev->nqr->msix_entries[BNXT_RE_AEQ_IDX].db_offset; + vid = rdev->nqr->msix_entries[BNXT_RE_AEQ_IDX].vector; rc = bnxt_qplib_enable_rcfw_channel(&rdev->rcfw, vid, db_offt, &bnxt_re_aeq_handler);