From patchwork Wed Nov 6 08:44:34 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Selvin Xavier X-Patchwork-Id: 13864141 X-Patchwork-Delegate: kuba@kernel.org Received: from mail-pl1-f176.google.com (mail-pl1-f176.google.com [209.85.214.176]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4F9231DD0DF for ; Wed, 6 Nov 2024 09:05:33 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.176 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1730883934; cv=none; b=PjJjmdkSk2y2mrf1iQo7X5p+DKYteFqa69YtQYdu29wswFqz8theIoMpocQE5nrqHOauv7pyqbtzTdVZ+18XP1+idgg8qxPovvqidJUUSF0Itz/DL8o8zhQYpnVT1JRUnq81Z2OQaEMOMjsSrtPjSnuD7OrzD73AjxNy4H+O9DM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1730883934; c=relaxed/simple; bh=MxDFVykjeASFttygGlVFETzBcH7gqvDqyuqwVQvN+wE=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References; b=X8+VQfYtAL3ss65y7XNMIFAvhUFcFKJUCmLs9xSHLZBuM9umq4cV/P8cSjoXCWUTgQIV36LNjOWXwHgzet0LxHq8FOXGYHSpf9Yvde+EmEXehK3nWy8HWlkarcjQ3xHMzBMgMnsEGRSWUVlD2pNVeukfNqGGG7wK1QQS+k3xGy4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=broadcom.com; spf=fail smtp.mailfrom=broadcom.com; dkim=pass (1024-bit key) header.d=broadcom.com header.i=@broadcom.com header.b=dnqMeUsE; arc=none smtp.client-ip=209.85.214.176 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=broadcom.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=broadcom.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=broadcom.com header.i=@broadcom.com header.b="dnqMeUsE" Received: by mail-pl1-f176.google.com with SMTP id d9443c01a7336-20e6981ca77so66899175ad.2 for ; Wed, 06 Nov 2024 01:05:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; t=1730883932; x=1731488732; darn=vger.kernel.org; h=references:in-reply-to:message-id:date:subject:cc:to:from:from:to :cc:subject:date:message-id:reply-to; bh=ojUfmXZSzyIQau3R+jJmvfUsS9SSCZ+gkgKPe64SPcE=; b=dnqMeUsEmi/fesTOpciCKdgiO+I0c4reLAnFWIpss6wZV6KjaAO3OOfHJ08uMtxECQ TSkDi+f6nQva/eL98l6lkfHMyTjnH7IFRIhaw4LTAb98JtwIGsdTPTBBRLI/JibfO0Nk OU0TOJsQNHaPgBCFrDwJd4/XsXrEfSkrqsX0I= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1730883932; x=1731488732; h=references:in-reply-to:message-id:date:subject:cc:to:from :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=ojUfmXZSzyIQau3R+jJmvfUsS9SSCZ+gkgKPe64SPcE=; b=c5pZsqciO86VZDdbGaRbwNPIfSJEgdj07WiYrcUFqKa6wyBa+i0/dEYu5HfUHHVKZh Frt+gUOjtsl+B2jP1BYWRqmzGsTKjJ3bKcbmCQ8dLKyH29Nt/Ft0Jecmo5QEYOaX5U6T 633EQHSOTyx+fOIKe5n5jmz7RQhdd7tG2jFc1N48zIBlVe/tQgMIj5C9QkmqICU6dwws BSs0MNzn2FNNEQI+NGIZQUoqfjEuTAWWa+jj7P43vYKoc+Dmm1meH6JXaZDKcMhxfiYB +RDMbC/VVmvVBRvGPJSI+Y/ELnCQMBtc6XBJcOqnG8W03eOkASeLn4ZBVkD+WMikiB0U EXfA== X-Forwarded-Encrypted: i=1; AJvYcCV76xw2Q7O3PKiDfMNleQ86ocOfDk+ZdhgQRb6zzLw27xPtwqt8/xi6SAoPvc+YyRlHRnc94xM=@vger.kernel.org X-Gm-Message-State: AOJu0Yxesl5qv7i79lB1d/tc5BWLo2fUfFodqWFneXpDaylntdAYAMfB EYqe2mWFZq+7GngVoLHdruq0FqGBxoyQHl1X3TmTEvYscmOkM2hAHKhVGtzqSQ== X-Google-Smtp-Source: AGHT+IE4ZaLYh3wQeBnRAPKoJPKr/cbn1fwHu8TzegfnNFk6B+kXJBzu0nwlPpEFAXFbVj9t+b2MpQ== X-Received: by 2002:a17:902:cf4c:b0:20b:7be8:8eb9 with SMTP id d9443c01a7336-21103c971c0mr271733425ad.54.1730883932544; Wed, 06 Nov 2024 01:05:32 -0800 (PST) Received: from sxavier-dev.dhcp.broadcom.net ([192.19.234.250]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-211057084casm91715395ad.92.2024.11.06.01.05.28 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Wed, 06 Nov 2024 01:05:31 -0800 (PST) From: Selvin Xavier To: leon@kernel.org, jgg@ziepe.ca Cc: linux-rdma@vger.kernel.org, netdev@vger.kernel.org, davem@davemloft.net, edumazet@google.com, kuba@kernel.org, pabeni@redhat.com, michael.chan@broadcom.com, andrew.gospodarek@broadcom.com, kalesh-anakkur.purayil@broadcom.com, Vikas Gupta , Selvin Xavier Subject: [PATCH rdma-next v2 1/3] bnxt_en: Add support for RoCE sriov configuration Date: Wed, 6 Nov 2024 00:44:34 -0800 Message-Id: <1730882676-24434-2-git-send-email-selvin.xavier@broadcom.com> X-Mailer: git-send-email 2.5.5 In-Reply-To: <1730882676-24434-1-git-send-email-selvin.xavier@broadcom.com> References: <1730882676-24434-1-git-send-email-selvin.xavier@broadcom.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: X-Patchwork-Delegate: kuba@kernel.org From: Vikas Gupta During driver load, PF RDMA driver provisions resources to the RDMA VFs. This logic takes into consideration of the total number of VFs supported on the PF while allocating resources. Firmware now advertises a capability where NIC driver can allocate resources for RDMA VFs when the user actually creates a VF. So this resource distribution can be based on the number of active VFs. This patch adds the support to check for the firmware capability and follow the new RDMA VF resource allocation strategy. The current logic in the RDMA driver will be removed for the newer Firmware versions in a subsequent patch in this series. Signed-off-by: Vikas Gupta Reviewed-by: Selvin Xavier Reviewed-by: Pavan Chebbi Reviewed-by: Michael Chan Signed-off-by: Kalesh AP Signed-off-by: Selvin Xavier --- drivers/net/ethernet/broadcom/bnxt/bnxt.c | 6 +++ drivers/net/ethernet/broadcom/bnxt/bnxt.h | 6 +++ drivers/net/ethernet/broadcom/bnxt/bnxt_sriov.c | 53 +++++++++++++++++++++++++ 3 files changed, 65 insertions(+) diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt.c b/drivers/net/ethernet/broadcom/bnxt/bnxt.c index 6e422e2..70230c5 100644 --- a/drivers/net/ethernet/broadcom/bnxt/bnxt.c +++ b/drivers/net/ethernet/broadcom/bnxt/bnxt.c @@ -8151,6 +8151,9 @@ static int bnxt_hwrm_func_qcfg(struct bnxt *bp) if (flags & FUNC_QCFG_RESP_FLAGS_RING_MONITOR_ENABLED) bp->fw_cap |= BNXT_FW_CAP_RING_MONITOR; + if (flags & FUNC_QCFG_RESP_FLAGS_ENABLE_RDMA_SRIOV) + bp->fw_cap |= BNXT_FW_CAP_ENABLE_RDMA_SRIOV; + switch (resp->port_partition_type) { case FUNC_QCFG_RESP_PORT_PARTITION_TYPE_NPAR1_0: case FUNC_QCFG_RESP_PORT_PARTITION_TYPE_NPAR1_5: @@ -9177,6 +9180,9 @@ static int __bnxt_hwrm_func_qcaps(struct bnxt *bp) bp->flags |= BNXT_FLAG_UDP_GSO_CAP; if (flags_ext2 & FUNC_QCAPS_RESP_FLAGS_EXT2_TX_PKT_TS_CMPL_SUPPORTED) bp->fw_cap |= BNXT_FW_CAP_TX_TS_CMP; + if (BNXT_PF(bp) && + (flags_ext2 & FUNC_QCAPS_RESP_FLAGS_EXT2_ROCE_VF_RESOURCE_MGMT_SUPPORTED)) + bp->fw_cap |= BNXT_FW_CAP_ROCE_VF_RESC_MGMT_SUPPORTED; bp->tx_push_thresh = 0; if ((flags & FUNC_QCAPS_RESP_FLAGS_PUSH_MODE_SUPPORTED) && diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt.h b/drivers/net/ethernet/broadcom/bnxt/bnxt.h index 69231e8..2da6c7b 100644 --- a/drivers/net/ethernet/broadcom/bnxt/bnxt.h +++ b/drivers/net/ethernet/broadcom/bnxt/bnxt.h @@ -2406,6 +2406,8 @@ struct bnxt { #define BNXT_FW_CAP_DCBX_AGENT BIT_ULL(2) #define BNXT_FW_CAP_NEW_RM BIT_ULL(3) #define BNXT_FW_CAP_IF_CHANGE BIT_ULL(4) + #define BNXT_FW_CAP_ENABLE_RDMA_SRIOV BIT_ULL(5) + #define BNXT_FW_CAP_ROCE_VF_RESC_MGMT_SUPPORTED BIT_ULL(6) #define BNXT_FW_CAP_KONG_MB_CHNL BIT_ULL(7) #define BNXT_FW_CAP_OVS_64BIT_HANDLE BIT_ULL(10) #define BNXT_FW_CAP_TRUSTED_VF BIT_ULL(11) @@ -2452,6 +2454,10 @@ struct bnxt { #define BNXT_SUPPORTS_QUEUE_API(bp) \ (BNXT_PF(bp) && BNXT_SUPPORTS_NTUPLE_VNIC(bp) && \ ((bp)->fw_cap & BNXT_FW_CAP_VNIC_RE_FLUSH)) +#define BNXT_RDMA_SRIOV_EN(bp) \ + ((bp)->fw_cap & BNXT_FW_CAP_ENABLE_RDMA_SRIOV) +#define BNXT_ROCE_VF_RESC_CAP(bp) \ + ((bp)->fw_cap & BNXT_FW_CAP_ROCE_VF_RESC_MGMT_SUPPORTED) u32 hwrm_spec_code; u16 hwrm_cmd_seq; diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt_sriov.c b/drivers/net/ethernet/broadcom/bnxt/bnxt_sriov.c index 7bb8a5d..12b6ed5 100644 --- a/drivers/net/ethernet/broadcom/bnxt/bnxt_sriov.c +++ b/drivers/net/ethernet/broadcom/bnxt/bnxt_sriov.c @@ -520,6 +520,56 @@ static int __bnxt_set_vf_params(struct bnxt *bp, int vf_id) return hwrm_req_send(bp, req); } +static void bnxt_hwrm_roce_sriov_cfg(struct bnxt *bp, int num_vfs) +{ + struct hwrm_func_qcaps_output *resp; + struct hwrm_func_cfg_input *cfg_req; + struct hwrm_func_qcaps_input *req; + int rc; + + rc = hwrm_req_init(bp, req, HWRM_FUNC_QCAPS); + if (rc) + return; + + req->fid = cpu_to_le16(0xffff); + resp = hwrm_req_hold(bp, req); + rc = hwrm_req_send(bp, req); + if (rc) + goto err; + + rc = hwrm_req_init(bp, cfg_req, HWRM_FUNC_CFG); + if (rc) + goto err; + + cfg_req->fid = cpu_to_le16(0xffff); + cfg_req->enables2 = + cpu_to_le32(FUNC_CFG_REQ_ENABLES2_ROCE_MAX_AV_PER_VF | + FUNC_CFG_REQ_ENABLES2_ROCE_MAX_CQ_PER_VF | + FUNC_CFG_REQ_ENABLES2_ROCE_MAX_MRW_PER_VF | + FUNC_CFG_REQ_ENABLES2_ROCE_MAX_QP_PER_VF | + FUNC_CFG_REQ_ENABLES2_ROCE_MAX_SRQ_PER_VF | + FUNC_CFG_REQ_ENABLES2_ROCE_MAX_GID_PER_VF); + cfg_req->roce_max_av_per_vf = + cpu_to_le32(le32_to_cpu(resp->roce_vf_max_av) / num_vfs); + cfg_req->roce_max_cq_per_vf = + cpu_to_le32(le32_to_cpu(resp->roce_vf_max_cq) / num_vfs); + cfg_req->roce_max_mrw_per_vf = + cpu_to_le32(le32_to_cpu(resp->roce_vf_max_mrw) / num_vfs); + cfg_req->roce_max_qp_per_vf = + cpu_to_le32(le32_to_cpu(resp->roce_vf_max_qp) / num_vfs); + cfg_req->roce_max_srq_per_vf = + cpu_to_le32(le32_to_cpu(resp->roce_vf_max_srq) / num_vfs); + cfg_req->roce_max_gid_per_vf = + cpu_to_le32(le32_to_cpu(resp->roce_vf_max_gid) / num_vfs); + + rc = hwrm_req_send(bp, cfg_req); + +err: + hwrm_req_drop(bp, req); + if (rc) + netdev_err(bp->dev, "RoCE sriov configuration failed\n"); +} + /* Only called by PF to reserve resources for VFs, returns actual number of * VFs configured, or < 0 on error. */ @@ -759,6 +809,9 @@ int bnxt_cfg_hw_sriov(struct bnxt *bp, int *num_vfs, bool reset) *num_vfs = rc; } + if (BNXT_RDMA_SRIOV_EN(bp) && BNXT_ROCE_VF_RESC_CAP(bp)) + bnxt_hwrm_roce_sriov_cfg(bp, *num_vfs); + return 0; } From patchwork Wed Nov 6 08:44:35 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Selvin Xavier X-Patchwork-Id: 13864142 X-Patchwork-Delegate: kuba@kernel.org Received: from mail-pl1-f180.google.com (mail-pl1-f180.google.com [209.85.214.180]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 482751DB34C for ; Wed, 6 Nov 2024 09:05:38 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.180 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1730883939; cv=none; b=tB7yqBPSl3cKoWk21biXfht0xwZ+hKGLXIYz6PMqZp94R0YalkRks+ew5PPrVwW/+9+6LRK9ftrHn7SJoZ6fAZ5Oia4V07jl1oIURN4HGR0N1jEeiqLxNFedAoSatMoVsOSSCgEpPqcTTgMCGZepXyNxdOq3eU3rlZiNneuoyLs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1730883939; c=relaxed/simple; bh=jD+cShbR5j46hyMs5F6R7f7NqxdyrvSwGdYQL8CPPGs=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References; b=MucLyV68hOG3VFCaBEdX82lVzNCZ0IkZW3tjkfbG+VAU3jWxhZDv/tc2X8RTJWu5G7EqyCoP1PydEYIzEbS39HCwRgkLYSIrhLLrkIUO5mPIeSXiEr/C15HBONWmvh/T4y8MFEzzIiWSCQ1baSH8s3kRmlsDSNoxHCsY9mjyGjg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=broadcom.com; spf=fail smtp.mailfrom=broadcom.com; dkim=pass (1024-bit key) header.d=broadcom.com header.i=@broadcom.com header.b=SeJjxQ+P; arc=none smtp.client-ip=209.85.214.180 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=broadcom.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=broadcom.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=broadcom.com header.i=@broadcom.com header.b="SeJjxQ+P" Received: by mail-pl1-f180.google.com with SMTP id d9443c01a7336-20cb7139d9dso60007795ad.1 for ; Wed, 06 Nov 2024 01:05:38 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; t=1730883938; x=1731488738; darn=vger.kernel.org; h=references:in-reply-to:message-id:date:subject:cc:to:from:from:to :cc:subject:date:message-id:reply-to; bh=eA1PTP98RNcNMUiOZVXtR51oHg/PYJgYgWSQcEEreX4=; b=SeJjxQ+PS1L2XtmN5yISaAtjGPRF4I3sA5wF8BWZbEH58BQKyrfwa69u9TEBDbfvHy jQfiHDcfn9jRTzPsbXIzxSsR343WboVdQc3mVeD6LQiCm8IwvU4hYjJJxXG9vuuBnKoQ kQ9D1mfzAoMcQXWIuKo7k2c38+eHzBmmNxXmk= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1730883938; x=1731488738; h=references:in-reply-to:message-id:date:subject:cc:to:from :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=eA1PTP98RNcNMUiOZVXtR51oHg/PYJgYgWSQcEEreX4=; b=hkf3XGl6UtoFcy4Xr8TiZ/Q1bg3QnBBG2Zre1ZzgK2ga8gxeJb+ehgtDWpUWqdGmSA S+RiJFQILhVNOCUkxUzK9oZmD1SFsxkrDoVgOBLa7asWVepxo2eiZZB+JBC7ryESsI+w vIKEPw1KxmZLKC/+HJQW4/wk7asRa0iMW56gcQPXujZIbJL/gUP2JsnKnpHXYGD9E7/0 R+oYW6DiH38BdoGdDSTFIVhMFBlSfzOC8U7hcwG6DYxGakm56keBivSZLBh0mmgtDkbY NiuSgT2kTJSIt1A8m0gBOCDWdmn3GmvNOkekRbycMBZXZ97q6si+sQRwZpCPDs8f9nmR r3ow== X-Forwarded-Encrypted: i=1; AJvYcCUCIcaSw+KY49yZYGK0iei7eE/Xi0lxL+LeGWKI+Tokzl1CR4oonJVoPInqAbx43jRHNsmhvXQ=@vger.kernel.org X-Gm-Message-State: AOJu0Yyb3eb6MhSs26XgKhAx35x+g1nb8y4trbrgOTWeJFrnXBQicjI5 Rnujqqzkgk20KwcE5E5adMMjt+FecYvtYde0AgaAZ4Pp8yyTC3JyeMF2GWzkIg== X-Google-Smtp-Source: AGHT+IFhSjGTXVWMwpfzA852xVj2q0jvhY8dnkm99UfTnrlbksQ23xkyb29wBxVqbOhNNEjlPIE0oQ== X-Received: by 2002:a17:902:f542:b0:20b:8bf5:cd72 with SMTP id d9443c01a7336-210c6c6f727mr517113365ad.49.1730883937535; Wed, 06 Nov 2024 01:05:37 -0800 (PST) Received: from sxavier-dev.dhcp.broadcom.net ([192.19.234.250]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-211057084casm91715395ad.92.2024.11.06.01.05.32 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Wed, 06 Nov 2024 01:05:36 -0800 (PST) From: Selvin Xavier To: leon@kernel.org, jgg@ziepe.ca Cc: linux-rdma@vger.kernel.org, netdev@vger.kernel.org, davem@davemloft.net, edumazet@google.com, kuba@kernel.org, pabeni@redhat.com, michael.chan@broadcom.com, andrew.gospodarek@broadcom.com, kalesh-anakkur.purayil@broadcom.com, Bhargava Chenna Marreddy , Selvin Xavier Subject: [PATCH rdma-next v2 2/3] RDMA/bnxt_re: Enhance RoCE SRIOV resource configuration design Date: Wed, 6 Nov 2024 00:44:35 -0800 Message-Id: <1730882676-24434-3-git-send-email-selvin.xavier@broadcom.com> X-Mailer: git-send-email 2.5.5 In-Reply-To: <1730882676-24434-1-git-send-email-selvin.xavier@broadcom.com> References: <1730882676-24434-1-git-send-email-selvin.xavier@broadcom.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: X-Patchwork-Delegate: kuba@kernel.org From: Bhargava Chenna Marreddy Refine RoCE SRIOV resource configuration design, using the INITIALIZE_FW's flag as an indication for the new design to the firmware. RoCE driver does not have to provision resources to VF when firmware advertises support for RoCE resource management by NIC driver. Signed-off-by: Bhargava Chenna Marreddy Signed-off-by: Kalesh AP Reviewed-by: Vikas Gupta Reviewed-by: Selvin Xavier CC: Michael Chan Signed-off-by: Selvin Xavier --- drivers/infiniband/hw/bnxt_re/main.c | 13 ++++++++----- drivers/infiniband/hw/bnxt_re/qplib_rcfw.c | 2 ++ drivers/infiniband/hw/bnxt_re/qplib_res.h | 3 +++ drivers/infiniband/hw/bnxt_re/roce_hsi.h | 1 + drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c | 2 ++ drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h | 1 + 6 files changed, 17 insertions(+), 5 deletions(-) diff --git a/drivers/infiniband/hw/bnxt_re/main.c b/drivers/infiniband/hw/bnxt_re/main.c index 4127227..dd528dd 100644 --- a/drivers/infiniband/hw/bnxt_re/main.c +++ b/drivers/infiniband/hw/bnxt_re/main.c @@ -184,6 +184,7 @@ static int bnxt_re_setup_chip_ctx(struct bnxt_re_dev *rdev) rdev->rcfw.res = &rdev->qplib_res; rdev->qplib_res.dattr = &rdev->dev_attr; rdev->qplib_res.is_vf = BNXT_EN_VF(en_dev); + rdev->qplib_res.en_dev = en_dev; bnxt_re_set_drv_mode(rdev); @@ -285,6 +286,10 @@ static void bnxt_re_set_resource_limits(struct bnxt_re_dev *rdev) static void bnxt_re_vf_res_config(struct bnxt_re_dev *rdev) { + /* + * Use the total VF count since the actual VF count may not be + * available at this point. + */ rdev->num_vfs = pci_sriov_get_totalvfs(rdev->en_dev->pdev); if (!bnxt_qplib_is_chip_gen_p5_p7(rdev->chip_ctx)) { bnxt_re_set_resource_limits(rdev); @@ -2056,11 +2061,9 @@ static int bnxt_re_dev_init(struct bnxt_re_dev *rdev, u8 op_type) INIT_DELAYED_WORK(&rdev->worker, bnxt_re_worker); set_bit(BNXT_RE_FLAG_QOS_WORK_REG, &rdev->flags); schedule_delayed_work(&rdev->worker, msecs_to_jiffies(30000)); - /* - * Use the total VF count since the actual VF count may not be - * available at this point. - */ - bnxt_re_vf_res_config(rdev); + + if (!(rdev->qplib_res.en_dev->flags & BNXT_EN_FLAG_ROCE_VF_RES_MGMT)) + bnxt_re_vf_res_config(rdev); } hash_init(rdev->cq_hash); if (rdev->chip_ctx->modes.toggle_bits & BNXT_QPLIB_SRQ_TOGGLE_BIT) diff --git a/drivers/infiniband/hw/bnxt_re/qplib_rcfw.c b/drivers/infiniband/hw/bnxt_re/qplib_rcfw.c index f5713e3..005079b 100644 --- a/drivers/infiniband/hw/bnxt_re/qplib_rcfw.c +++ b/drivers/infiniband/hw/bnxt_re/qplib_rcfw.c @@ -910,6 +910,8 @@ int bnxt_qplib_init_rcfw(struct bnxt_qplib_rcfw *rcfw, flags |= CMDQ_INITIALIZE_FW_FLAGS_HW_REQUESTER_RETX_SUPPORTED; if (_is_optimize_modify_qp_supported(rcfw->res->dattr->dev_cap_flags2)) flags |= CMDQ_INITIALIZE_FW_FLAGS_OPTIMIZE_MODIFY_QP_SUPPORTED; + if (rcfw->res->en_dev->flags & BNXT_EN_FLAG_ROCE_VF_RES_MGMT) + flags |= CMDQ_INITIALIZE_FW_FLAGS_L2_VF_RESOURCE_MGMT; req.flags |= cpu_to_le16(flags); req.stat_ctx_id = cpu_to_le32(ctx->stats.fw_id); bnxt_qplib_fill_cmdqmsg(&msg, &req, &resp, NULL, sizeof(req), sizeof(resp), 0); diff --git a/drivers/infiniband/hw/bnxt_re/qplib_res.h b/drivers/infiniband/hw/bnxt_re/qplib_res.h index 115910c..21fb148 100644 --- a/drivers/infiniband/hw/bnxt_re/qplib_res.h +++ b/drivers/infiniband/hw/bnxt_re/qplib_res.h @@ -39,6 +39,8 @@ #ifndef __BNXT_QPLIB_RES_H__ #define __BNXT_QPLIB_RES_H__ +#include "bnxt_ulp.h" + extern const struct bnxt_qplib_gid bnxt_qplib_gid_zero; #define CHIP_NUM_57508 0x1750 @@ -302,6 +304,7 @@ struct bnxt_qplib_res { struct bnxt_qplib_chip_ctx *cctx; struct bnxt_qplib_dev_attr *dattr; struct net_device *netdev; + struct bnxt_en_dev *en_dev; struct bnxt_qplib_rcfw *rcfw; struct bnxt_qplib_pd_tbl pd_tbl; /* To protect the pd table bit map */ diff --git a/drivers/infiniband/hw/bnxt_re/roce_hsi.h b/drivers/infiniband/hw/bnxt_re/roce_hsi.h index d9c5373..a98fc9c 100644 --- a/drivers/infiniband/hw/bnxt_re/roce_hsi.h +++ b/drivers/infiniband/hw/bnxt_re/roce_hsi.h @@ -217,6 +217,7 @@ struct cmdq_initialize_fw { #define CMDQ_INITIALIZE_FW_FLAGS_MRAV_RESERVATION_SPLIT 0x1UL #define CMDQ_INITIALIZE_FW_FLAGS_HW_REQUESTER_RETX_SUPPORTED 0x2UL #define CMDQ_INITIALIZE_FW_FLAGS_OPTIMIZE_MODIFY_QP_SUPPORTED 0x8UL + #define CMDQ_INITIALIZE_FW_FLAGS_L2_VF_RESOURCE_MGMT 0x10UL __le16 cookie; u8 resp_size; u8 reserved8; diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c b/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c index fdd6356..b771c84 100644 --- a/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c +++ b/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c @@ -414,6 +414,8 @@ static void bnxt_set_edev_info(struct bnxt_en_dev *edev, struct bnxt *bp) edev->flags |= BNXT_EN_FLAG_ROCEV2_CAP; if (bp->flags & BNXT_FLAG_VF) edev->flags |= BNXT_EN_FLAG_VF; + if (BNXT_ROCE_VF_RESC_CAP(bp)) + edev->flags |= BNXT_EN_FLAG_ROCE_VF_RES_MGMT; edev->chip_num = bp->chip_num; edev->hw_ring_stats_size = bp->hw_ring_stats_size; diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h b/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h index 4f4914f5..5d6aac6 100644 --- a/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h +++ b/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h @@ -64,6 +64,7 @@ struct bnxt_en_dev { #define BNXT_EN_FLAG_ULP_STOPPED 0x8 #define BNXT_EN_FLAG_VF 0x10 #define BNXT_EN_VF(edev) ((edev)->flags & BNXT_EN_FLAG_VF) + #define BNXT_EN_FLAG_ROCE_VF_RES_MGMT 0x20 struct bnxt_ulp *ulp_tbl; int l2_db_size; /* Doorbell BAR size in From patchwork Wed Nov 6 08:44:36 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Selvin Xavier X-Patchwork-Id: 13864143 Received: from mail-pg1-f176.google.com (mail-pg1-f176.google.com [209.85.215.176]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B1F7E1DD880 for ; Wed, 6 Nov 2024 09:05:42 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.176 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1730883944; cv=none; b=MZaoBqHJQU2iku6icci63MK0XAVQlOeR71hjFyiwOKXswzOiBCyl9GqbHGilxYcyNHWPj5BzR6QBnaSI58fEQHQ4DebfNwT5U/m+s1vs+nOjM+ggvSU6y39umtc1W9NJeSfhpr0sn2PQMvc8RT7iI/AMl+rY70zt/XsBexTqlCM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1730883944; c=relaxed/simple; bh=Paw3NHGBKcpYWKx6/8kixwuXZd4axLGz+zE8p3Aik4E=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References; b=JEeEP9+Ea5htuZTNpQTj1wPXkUJUvqU52c2jqEmw3hwjHA6BK6BKFO4hM/YsaP2B7cz4qKfLK1lYyRvk/GRekYnnYGu2Aw9DDdXl/KrNCDjFdHW94V1FOszWC4k9LyzId1WRl98HdOizQCeXQi3EBOgI2OxFa1mdXIEwmhpr8Jg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=broadcom.com; spf=fail smtp.mailfrom=broadcom.com; dkim=pass (1024-bit key) header.d=broadcom.com header.i=@broadcom.com header.b=gpP+p9cm; arc=none smtp.client-ip=209.85.215.176 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=broadcom.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=broadcom.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=broadcom.com header.i=@broadcom.com header.b="gpP+p9cm" Received: by mail-pg1-f176.google.com with SMTP id 41be03b00d2f7-7ea9739647bso4626409a12.0 for ; Wed, 06 Nov 2024 01:05:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; t=1730883942; x=1731488742; darn=vger.kernel.org; h=references:in-reply-to:message-id:date:subject:cc:to:from:from:to :cc:subject:date:message-id:reply-to; bh=KpyF7OfeG3D1SjYyfRqhBhCivmBUEISeT2uU0LeHwvU=; b=gpP+p9cmKNiWbEScfck4H+8B0FdENOMlK0Gdc0zbeWB+0QIzFoh5iNnqzoZJd2DleJ Hx7gKsgOdk7WrajzW4xDqu1QA3qJr0xdX07H9UAwFPcFI0QwhqdXpfLRAoUYefmnhb81 illP8wwORUJ9dRvBSJT3Aaud8LTE4zjYHA8VI= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1730883942; x=1731488742; h=references:in-reply-to:message-id:date:subject:cc:to:from :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=KpyF7OfeG3D1SjYyfRqhBhCivmBUEISeT2uU0LeHwvU=; b=Ikkc9jtgf3FegE6epBvEBy2dgyl+6awI2aVI6I3im65PseDfN2rzeCKPIkR7jEDIN/ WwypoRgK15889L8Y40i8CC5IpVD0m/AxbInVgqHEiqOH/c9rgs9OiFfPo/HsL+Gp7rRW /jjTB7+vchz0mCT4Xr0uEjh1dxPAshA/RKbojvdnK7eu9okf53i2a/HvkfSPvmLgbJUi j3xvSyV1K6OBHkcKI8M1J8Zwtq4KTTqaEMLfaRw88YeKGLPzNjxCREPFPXhB/UoYnpGa aiCfGMcldKKXvnSzzJ379oCxLGRqLJgHO/oeiEkkoFbRpXxeM7e1DBo/gl8/TdT95ygT Iu0w== X-Forwarded-Encrypted: i=1; AJvYcCUjBHxHTAHErPp7OlFKV2QEg1wVjkHDo55fcoCdDr2gGJXC8q9pTNIr5xzCNRUBidNgiuqInY4=@vger.kernel.org X-Gm-Message-State: AOJu0Ywmel8/oh7qJNLOs8SbvKYvciQ2nbCUINakcg4Lm6W4OgBk0RiG aY8INXCQeh5USCt7Sss3s/g9sLZ09oDEL2fIYo7tmtYPudRLko0IV7yIv6oKmQ== X-Google-Smtp-Source: AGHT+IE2DaJ2SRKntdWmojupvyRQ48n1fwCI4ZCg/P3nRIBrDbwGV5WYxIPHn57KcSlLclx78itmsA== X-Received: by 2002:a05:6a20:c6c1:b0:1cf:9a86:6cb7 with SMTP id adf61e73a8af0-1d9a83d5671mr52453452637.20.1730883941925; Wed, 06 Nov 2024 01:05:41 -0800 (PST) Received: from sxavier-dev.dhcp.broadcom.net ([192.19.234.250]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-211057084casm91715395ad.92.2024.11.06.01.05.37 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Wed, 06 Nov 2024 01:05:40 -0800 (PST) From: Selvin Xavier To: leon@kernel.org, jgg@ziepe.ca Cc: linux-rdma@vger.kernel.org, netdev@vger.kernel.org, davem@davemloft.net, edumazet@google.com, kuba@kernel.org, pabeni@redhat.com, michael.chan@broadcom.com, andrew.gospodarek@broadcom.com, kalesh-anakkur.purayil@broadcom.com, Selvin Xavier Subject: [PATCH rdma-next v2 3/3] RDMA/bnxt_re: Add set_func_resources support for P5/P7 adapters Date: Wed, 6 Nov 2024 00:44:36 -0800 Message-Id: <1730882676-24434-4-git-send-email-selvin.xavier@broadcom.com> X-Mailer: git-send-email 2.5.5 In-Reply-To: <1730882676-24434-1-git-send-email-selvin.xavier@broadcom.com> References: <1730882676-24434-1-git-send-email-selvin.xavier@broadcom.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: From: Kalesh AP Enable set_func_resources for P5 and P7 adapters to handle VF resource distribution. Remove setting max resources per VF during PF initialization. This change is required for firmwares which does not support RoCE VF resource management by NIC driver. The code is same for all adapters now. Reviewed-by: Stephen Shi Reviewed-by: Rukhsana Ansari Signed-off-by: Kalesh AP Signed-off-by: Selvin Xavier --- drivers/infiniband/hw/bnxt_re/main.c | 11 ++++++----- drivers/infiniband/hw/bnxt_re/qplib_rcfw.c | 11 +---------- 2 files changed, 7 insertions(+), 15 deletions(-) diff --git a/drivers/infiniband/hw/bnxt_re/main.c b/drivers/infiniband/hw/bnxt_re/main.c index dd528dd..cb61941 100644 --- a/drivers/infiniband/hw/bnxt_re/main.c +++ b/drivers/infiniband/hw/bnxt_re/main.c @@ -291,11 +291,12 @@ static void bnxt_re_vf_res_config(struct bnxt_re_dev *rdev) * available at this point. */ rdev->num_vfs = pci_sriov_get_totalvfs(rdev->en_dev->pdev); - if (!bnxt_qplib_is_chip_gen_p5_p7(rdev->chip_ctx)) { - bnxt_re_set_resource_limits(rdev); - bnxt_qplib_set_func_resources(&rdev->qplib_res, &rdev->rcfw, - &rdev->qplib_ctx); - } + if (!rdev->num_vfs) + return; + + bnxt_re_set_resource_limits(rdev); + bnxt_qplib_set_func_resources(&rdev->qplib_res, &rdev->rcfw, + &rdev->qplib_ctx); } static void bnxt_re_shutdown(struct auxiliary_device *adev) diff --git a/drivers/infiniband/hw/bnxt_re/qplib_rcfw.c b/drivers/infiniband/hw/bnxt_re/qplib_rcfw.c index 005079b..7072991 100644 --- a/drivers/infiniband/hw/bnxt_re/qplib_rcfw.c +++ b/drivers/infiniband/hw/bnxt_re/qplib_rcfw.c @@ -851,10 +851,8 @@ int bnxt_qplib_init_rcfw(struct bnxt_qplib_rcfw *rcfw, * shall setup this area for VF. Skipping the * HW programming */ - if (is_virtfn) + if (is_virtfn || bnxt_qplib_is_chip_gen_p5_p7(rcfw->res->cctx)) goto skip_ctx_setup; - if (bnxt_qplib_is_chip_gen_p5_p7(rcfw->res->cctx)) - goto config_vf_res; lvl = ctx->qpc_tbl.level; pgsz = bnxt_qplib_base_pg_size(&ctx->qpc_tbl); @@ -898,13 +896,6 @@ int bnxt_qplib_init_rcfw(struct bnxt_qplib_rcfw *rcfw, req.number_of_srq = cpu_to_le32(ctx->srqc_tbl.max_elements); req.number_of_cq = cpu_to_le32(ctx->cq_tbl.max_elements); -config_vf_res: - req.max_qp_per_vf = cpu_to_le32(ctx->vf_res.max_qp_per_vf); - req.max_mrw_per_vf = cpu_to_le32(ctx->vf_res.max_mrw_per_vf); - req.max_srq_per_vf = cpu_to_le32(ctx->vf_res.max_srq_per_vf); - req.max_cq_per_vf = cpu_to_le32(ctx->vf_res.max_cq_per_vf); - req.max_gid_per_vf = cpu_to_le32(ctx->vf_res.max_gid_per_vf); - skip_ctx_setup: if (BNXT_RE_HW_RETX(rcfw->res->dattr->dev_cap_flags)) flags |= CMDQ_INITIALIZE_FW_FLAGS_HW_REQUESTER_RETX_SUPPORTED;