From patchwork Thu Jun 27 02:41:03 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Selvin Xavier X-Patchwork-Id: 13713766 Received: from mail-pg1-f181.google.com (mail-pg1-f181.google.com [209.85.215.181]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id EA778F9E4 for ; Thu, 27 Jun 2024 03:01:59 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.181 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719457321; cv=none; b=WNPEVlJDfE9IPxW3OkcfETJm/X2EveByrntL31r30cguusrGKowKkxCj0xOBODHFbpGIkhc2Mx6eHrDIGwU3hRZgiMvvdb82jXLiVonexcJXnKwMML7QIenBmA13wv996c+e4G3B96JRo0Bb5xsKgt17XL/0LDVuxmFcQiLEUDs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719457321; c=relaxed/simple; bh=1yjIGELiQA3lMiP3LpOYsS4EzyN6dHVTg3ptYNYhJec=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: Content-Type; b=LItFUX1/YbWzVouF90tw2IXd0E1BBPuWKUXyLcura6AACsyDS1zMaL1liuy79uZLS69Ky4nOcIgo7qiedcESn5DFXBZEGFvPNYmt98cwDQwrZAD6mkhxGrdx2YBSA7dLNIHU/Ac4v2OTlDNLMlWMmKd5sul7eUKkYc57bGStDmM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=broadcom.com; spf=fail smtp.mailfrom=broadcom.com; dkim=pass (1024-bit key) header.d=broadcom.com header.i=@broadcom.com header.b=C+BTrpwg; arc=none smtp.client-ip=209.85.215.181 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=broadcom.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=broadcom.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=broadcom.com header.i=@broadcom.com header.b="C+BTrpwg" Received: by mail-pg1-f181.google.com with SMTP id 41be03b00d2f7-725a7b0fc55so795843a12.3 for ; Wed, 26 Jun 2024 20:01:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; t=1719457319; x=1720062119; darn=vger.kernel.org; h=references:in-reply-to:message-id:date:subject:cc:to:from:from:to :cc:subject:date:message-id:reply-to; bh=iHAC9FGVqvLCNFsGqlyNt0g7E3PC19iLYRwVC4l6Ww8=; b=C+BTrpwg/WWOV/xZg7ty0GkIFP1yu8CCCDaaOR1GdzLRv7qLRhNPrB1NtsEuisoUK6 wNiUXlYpgJ12q+NYAlDRXyy92M5DQxaCb46/EdXF7P+vLf5IRAxjyU5BTCdvtibwiDHd iJ18iLseNSPOzs6+NTfC2K0yKw86TLgK5e4k8= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1719457319; x=1720062119; h=references:in-reply-to:message-id:date:subject:cc:to:from :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=iHAC9FGVqvLCNFsGqlyNt0g7E3PC19iLYRwVC4l6Ww8=; b=lcQh0w5vHlSmEM+DNqz8EgmD4BORAJodoQ3D5292jeFetc/lah6+Vw5EHTi/ssQ6a5 zY0Vn/MI9WsKD/ZhqWSUHKpxxq+G7cO7fSqIbkKh/9v9SUD17+9Ahl67af6Fxp/HVeeD a+U6wcoh4j7y6/9Ld/7eVkRdfRgOi1pZsGuYu6H8Fw6eLpwpJJ4LJ8z9XYHbzJT4m1Ks PVlSKJ/ofzlqF1Pp3XwlfWXhxLFR8gNvChbqrmRajcCp2aGzPI1zWLle+YJUZWvHQz1W NaDtSV2fKqMj6jG3QqITz2Lzzzw5IZqFQRIbrzE7XurOfxJ5j5b7sOBDXKvwBADwxIsr +puw== X-Gm-Message-State: AOJu0YwQxc3nvBO+6eoQUsSOcD2aBgKfKUOebO1eFM8tXH9CIpWrlcON xfpCqvKvYKQoUZi55Mk10loB/lWQltelmLVfU7dARpjR9SjPPWxEsBKXUSo6vQ== X-Google-Smtp-Source: AGHT+IFQ43BEMg3xIe3HUm55aK0AUj67/DJr3+qon30VQTUsbdUYmoTT7ccfRDwYWoduEqn9kokiaw== X-Received: by 2002:a05:6a20:354b:b0:1bd:2d0b:e143 with SMTP id adf61e73a8af0-1bd2d0be318mr3628783637.52.1719457319122; Wed, 26 Jun 2024 20:01:59 -0700 (PDT) Received: from sxavier-dev.dhcp.broadcom.net ([192.19.234.250]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-1faac997da7sm2103285ad.216.2024.06.26.20.01.56 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Wed, 26 Jun 2024 20:01:58 -0700 (PDT) From: Selvin Xavier To: leon@kernel.org, jgg@ziepe.ca Cc: linux-rdma@vger.kernel.org, andrew.gospodarek@broadcom.com, Selvin Xavier , Chandramohan Akula , Ajit Khaparde Subject: [PATCH for-next 1/3] RDMA/bnxt_re: Update the correct DB FIFO depth and mask for GenP7 Date: Wed, 26 Jun 2024 19:41:03 -0700 Message-Id: <1719456065-27394-2-git-send-email-selvin.xavier@broadcom.com> X-Mailer: git-send-email 2.5.5 In-Reply-To: <1719456065-27394-1-git-send-email-selvin.xavier@broadcom.com> References: <1719456065-27394-1-git-send-email-selvin.xavier@broadcom.com> Precedence: bulk X-Mailing-List: linux-rdma@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: GenP5 and P7 devices have different DB FIFO depth. Use different values based on the chip context. Instead of hardcoding doorbell FIFO related values, get it from the HWRM interface. Maintain backward compatibility by having default values when FW is not providing the doorbell FIFO related values. Signed-off-by: Chandramohan Akula Signed-off-by: Ajit Khaparde Signed-off-by: Selvin Xavier --- drivers/infiniband/hw/bnxt_re/bnxt_re.h | 12 ++++++++--- drivers/infiniband/hw/bnxt_re/main.c | 38 +++++++++++++++++++++------------ 2 files changed, 33 insertions(+), 17 deletions(-) diff --git a/drivers/infiniband/hw/bnxt_re/bnxt_re.h b/drivers/infiniband/hw/bnxt_re/bnxt_re.h index 9dca451..0880b4b 100644 --- a/drivers/infiniband/hw/bnxt_re/bnxt_re.h +++ b/drivers/infiniband/hw/bnxt_re/bnxt_re.h @@ -131,9 +131,15 @@ struct bnxt_re_pacing { #define BNXT_RE_PACING_ALARM_TH_MULTIPLE 2 /* Multiple of pacing algo threshold */ /* Default do_pacing value when there is no congestion */ #define BNXT_RE_DBR_DO_PACING_NO_CONGESTION 0x7F /* 1 in 512 probability */ -#define BNXT_RE_DB_FIFO_ROOM_MASK 0x1FFF8000 -#define BNXT_RE_MAX_FIFO_DEPTH 0x2c00 -#define BNXT_RE_DB_FIFO_ROOM_SHIFT 15 + +#define BNXT_RE_MAX_FIFO_DEPTH_P5 0x2c00 +#define BNXT_RE_MAX_FIFO_DEPTH_P7 0x8000 + +#define BNXT_RE_MAX_FIFO_DEPTH(ctx) \ + (bnxt_qplib_is_chip_gen_p7((ctx)) ? \ + BNXT_RE_MAX_FIFO_DEPTH_P7 :\ + BNXT_RE_MAX_FIFO_DEPTH_P5) + #define BNXT_RE_GRC_FIFO_REG_BASE 0x2000 #define MAX_CQ_HASH_BITS (16) diff --git a/drivers/infiniband/hw/bnxt_re/main.c b/drivers/infiniband/hw/bnxt_re/main.c index 54b4d2f..2a727f4 100644 --- a/drivers/infiniband/hw/bnxt_re/main.c +++ b/drivers/infiniband/hw/bnxt_re/main.c @@ -444,6 +444,7 @@ int bnxt_re_hwrm_qcaps(struct bnxt_re_dev *rdev) static int bnxt_re_hwrm_dbr_pacing_qcfg(struct bnxt_re_dev *rdev) { + struct bnxt_qplib_db_pacing_data *pacing_data = rdev->qplib_res.pacing_data; struct hwrm_func_dbr_pacing_qcfg_output resp = {}; struct hwrm_func_dbr_pacing_qcfg_input req = {}; struct bnxt_en_dev *en_dev = rdev->en_dev; @@ -465,6 +466,13 @@ static int bnxt_re_hwrm_dbr_pacing_qcfg(struct bnxt_re_dev *rdev) cctx->dbr_stat_db_fifo = le32_to_cpu(resp.dbr_stat_db_fifo_reg) & ~FUNC_DBR_PACING_QCFG_RESP_DBR_STAT_DB_FIFO_REG_ADDR_SPACE_MASK; + + pacing_data->fifo_max_depth = le32_to_cpu(resp.dbr_stat_db_max_fifo_depth); + if (!pacing_data->fifo_max_depth) + pacing_data->fifo_max_depth = BNXT_RE_MAX_FIFO_DEPTH(cctx); + pacing_data->fifo_room_mask = le32_to_cpu(resp.dbr_stat_db_fifo_reg_fifo_room_mask); + pacing_data->fifo_room_shift = resp.dbr_stat_db_fifo_reg_fifo_room_shift; + return 0; } @@ -481,6 +489,7 @@ static void bnxt_re_set_default_pacing_data(struct bnxt_re_dev *rdev) static void __wait_for_fifo_occupancy_below_th(struct bnxt_re_dev *rdev) { + struct bnxt_qplib_db_pacing_data *pacing_data = rdev->qplib_res.pacing_data; u32 read_val, fifo_occup; /* loop shouldn't run infintely as the occupancy usually goes @@ -488,14 +497,14 @@ static void __wait_for_fifo_occupancy_below_th(struct bnxt_re_dev *rdev) */ while (1) { read_val = readl(rdev->en_dev->bar0 + rdev->pacing.dbr_db_fifo_reg_off); - fifo_occup = BNXT_RE_MAX_FIFO_DEPTH - - ((read_val & BNXT_RE_DB_FIFO_ROOM_MASK) >> - BNXT_RE_DB_FIFO_ROOM_SHIFT); + fifo_occup = pacing_data->fifo_max_depth - + ((read_val & pacing_data->fifo_room_mask) >> + pacing_data->fifo_room_shift); /* Fifo occupancy cannot be greater the MAX FIFO depth */ - if (fifo_occup > BNXT_RE_MAX_FIFO_DEPTH) + if (fifo_occup > pacing_data->fifo_max_depth) break; - if (fifo_occup < rdev->qplib_res.pacing_data->pacing_th) + if (fifo_occup < pacing_data->pacing_th) break; } } @@ -553,9 +562,9 @@ static void bnxt_re_pacing_timer_exp(struct work_struct *work) pacing_data = rdev->qplib_res.pacing_data; read_val = readl(rdev->en_dev->bar0 + rdev->pacing.dbr_db_fifo_reg_off); - fifo_occup = BNXT_RE_MAX_FIFO_DEPTH - - ((read_val & BNXT_RE_DB_FIFO_ROOM_MASK) >> - BNXT_RE_DB_FIFO_ROOM_SHIFT); + fifo_occup = pacing_data->fifo_max_depth - + ((read_val & pacing_data->fifo_room_mask) >> + pacing_data->fifo_room_shift); if (fifo_occup > pacing_data->pacing_th) goto restart_timer; @@ -594,7 +603,7 @@ void bnxt_re_pacing_alert(struct bnxt_re_dev *rdev) * Increase the alarm_th to max so that other user lib instances do not * keep alerting the driver. */ - pacing_data->alarm_th = BNXT_RE_MAX_FIFO_DEPTH; + pacing_data->alarm_th = pacing_data->fifo_max_depth; pacing_data->do_pacing = BNXT_RE_MAX_DBR_DO_PACING; cancel_work_sync(&rdev->dbq_fifo_check_work); schedule_work(&rdev->dbq_fifo_check_work); @@ -603,8 +612,6 @@ void bnxt_re_pacing_alert(struct bnxt_re_dev *rdev) static int bnxt_re_initialize_dbr_pacing(struct bnxt_re_dev *rdev) { - if (bnxt_re_hwrm_dbr_pacing_qcfg(rdev)) - return -EIO; /* Allocate a page for app use */ rdev->pacing.dbr_page = (void *)__get_free_page(GFP_KERNEL); @@ -614,6 +621,12 @@ static int bnxt_re_initialize_dbr_pacing(struct bnxt_re_dev *rdev) memset((u8 *)rdev->pacing.dbr_page, 0, PAGE_SIZE); rdev->qplib_res.pacing_data = (struct bnxt_qplib_db_pacing_data *)rdev->pacing.dbr_page; + if (bnxt_re_hwrm_dbr_pacing_qcfg(rdev)) { + free_page((u64)rdev->pacing.dbr_page); + rdev->pacing.dbr_page = NULL; + return -EIO; + } + /* MAP HW window 2 for reading db fifo depth */ writel(rdev->chip_ctx->dbr_stat_db_fifo & BNXT_GRC_BASE_MASK, rdev->en_dev->bar0 + BNXT_GRCPF_REG_WINDOW_BASE_OUT + 4); @@ -627,9 +640,6 @@ static int bnxt_re_initialize_dbr_pacing(struct bnxt_re_dev *rdev) rdev->pacing.dbq_pacing_time = BNXT_RE_DBR_PACING_TIME; rdev->pacing.dbr_def_do_pacing = BNXT_RE_DBR_DO_PACING_NO_CONGESTION; rdev->pacing.do_pacing_save = rdev->pacing.dbr_def_do_pacing; - rdev->qplib_res.pacing_data->fifo_max_depth = BNXT_RE_MAX_FIFO_DEPTH; - rdev->qplib_res.pacing_data->fifo_room_mask = BNXT_RE_DB_FIFO_ROOM_MASK; - rdev->qplib_res.pacing_data->fifo_room_shift = BNXT_RE_DB_FIFO_ROOM_SHIFT; rdev->qplib_res.pacing_data->grc_reg_offset = rdev->pacing.dbr_db_fifo_reg_off; bnxt_re_set_default_pacing_data(rdev); /* Initialize worker for DBR Pacing */ From patchwork Thu Jun 27 02:41:04 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Selvin Xavier X-Patchwork-Id: 13713767 Received: from mail-pl1-f170.google.com (mail-pl1-f170.google.com [209.85.214.170]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 482FEF9E4 for ; Thu, 27 Jun 2024 03:02:03 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.170 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719457324; cv=none; b=A5MbLpZFLD7CxenDTCa5SSLUELBe9VgJOi5Ios+mrlb0LAjTqxdyz6UQ1Ofz0K/yeFqATfl8CKYBo1NPmZeMTZ9yYJmqazdZncH86TZiA8SPdRqJBmNTUVkG0unP1vP1K5nHkSd9Y/toodyL4R08LDMPKm+hx3R01wH9GVwzgAA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719457324; c=relaxed/simple; bh=cQ2CM5xAZJunftQNliRO/vPRPTFuCLA7gZLdBGlMgOw=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: Content-Type; b=pVUppYgBa3zoMtJgtBCqgaML8ZHzDiNPjZX8q34hI8YaqdX9Hiz9KocoUDvRF1yo6KDoJmqUBcsajjtnwojIwR+89Lex3cqHpoVDSN4c8KeGkjGKlmS7ALVocf0eSVrc7VlNXu9GMhJNVkQsneDy0pvwwdrALLqcBO95vOLVoTI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=broadcom.com; spf=fail smtp.mailfrom=broadcom.com; dkim=pass (1024-bit key) header.d=broadcom.com header.i=@broadcom.com header.b=HoW9m7FO; arc=none smtp.client-ip=209.85.214.170 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=broadcom.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=broadcom.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=broadcom.com header.i=@broadcom.com header.b="HoW9m7FO" Received: by mail-pl1-f170.google.com with SMTP id d9443c01a7336-1f480624d10so61501865ad.1 for ; Wed, 26 Jun 2024 20:02:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; t=1719457322; x=1720062122; darn=vger.kernel.org; h=references:in-reply-to:message-id:date:subject:cc:to:from:from:to :cc:subject:date:message-id:reply-to; bh=bSsf9znd4oVsM5HZXF24KiNx2k95El3gJPGyz2GdLX0=; b=HoW9m7FO49r0/MvuoBAVSKI6YcXAW4SpEbGOeiIizuWc+7g+7BSUxarFhoHvZIa0KU FtWciuy09RtxjoiLYNjrw+NvQm/07TJZT1XJb1b38TGL+igIGYoEnnsm0kO31RH1zJ+Z SGTJLe/jDeKJB0vES2RUCESXjM7WS/am9Ju70= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1719457322; x=1720062122; h=references:in-reply-to:message-id:date:subject:cc:to:from :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=bSsf9znd4oVsM5HZXF24KiNx2k95El3gJPGyz2GdLX0=; b=cmR6QL0eFr5Bk33ulRYRbyCD5Htec+TxfApBeEh8/l/bjyOeMCqUeYUzPbVvbHz/St AHPAG7avBIJSIJABdJNPMU60tfJWAt5oQrV9t1TBq4d9mb/5bd8KCb1/MRYr/JdjPE4o SPqdFtfkG3XUkJF1AdZAiLN2XoMU0485ACH2jlDC8XqmjfmXMtlU682d8TpWqKfYw+Rx SUsifNfnOrW6Q68M3XFgMDEZpU086F/jlZ9AzVgZNklYtmTKcz+Ga0apOt0w9X83nfA3 NsV5PQqOlIUlmyJS8sCfqfYyRlGcmhTYSYjj8C4WlGyPZw5tvn0ZxxKzok92ikYOiYJk BuJg== X-Gm-Message-State: AOJu0YzlYCNFtZ5rAi1RZMrKPYfoEElVzYgzCwF3pEnohB73jhmuYRAl fbyIX8k2a31dLo8uY6AYmSTH1EqfsYemGreXMp73UToce5ncUAxj26xKABVlDw== X-Google-Smtp-Source: AGHT+IGh7dRDZkvmYPBMX0ii2ilWlS3gAeq/irzL856vTJdk40hg/UO4oi15URKsu5Ffn1VOeHRA8Q== X-Received: by 2002:a17:902:e995:b0:1f9:8f78:5553 with SMTP id d9443c01a7336-1fa1d4fdc46mr121817505ad.18.1719457322417; Wed, 26 Jun 2024 20:02:02 -0700 (PDT) Received: from sxavier-dev.dhcp.broadcom.net ([192.19.234.250]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-1faac997da7sm2103285ad.216.2024.06.26.20.01.59 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Wed, 26 Jun 2024 20:02:01 -0700 (PDT) From: Selvin Xavier To: leon@kernel.org, jgg@ziepe.ca Cc: linux-rdma@vger.kernel.org, andrew.gospodarek@broadcom.com, Selvin Xavier , Chandramohan Akula , Ajit Khaparde Subject: [PATCH for-next 2/3] RDMA/bnxt_re: Enable DB moderation for genP7 adapters Date: Wed, 26 Jun 2024 19:41:04 -0700 Message-Id: <1719456065-27394-3-git-send-email-selvin.xavier@broadcom.com> X-Mailer: git-send-email 2.5.5 In-Reply-To: <1719456065-27394-1-git-send-email-selvin.xavier@broadcom.com> References: <1719456065-27394-1-git-send-email-selvin.xavier@broadcom.com> Precedence: bulk X-Mailing-List: linux-rdma@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Enable DB moderation support for GenP7 adapters also. Query from FW and update the status. Signed-off-by: Chandramohan Akula Signed-off-by: Ajit Khaparde Signed-off-by: Selvin Xavier --- drivers/infiniband/hw/bnxt_re/main.c | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/drivers/infiniband/hw/bnxt_re/main.c b/drivers/infiniband/hw/bnxt_re/main.c index 2a727f4..2c5282f 100644 --- a/drivers/infiniband/hw/bnxt_re/main.c +++ b/drivers/infiniband/hw/bnxt_re/main.c @@ -423,6 +423,7 @@ int bnxt_re_hwrm_qcaps(struct bnxt_re_dev *rdev) struct hwrm_func_qcaps_input req = {}; struct bnxt_qplib_chip_ctx *cctx; struct bnxt_fw_msg fw_msg = {}; + u32 flags_ext2; int rc; cctx = rdev->chip_ctx; @@ -436,9 +437,9 @@ int bnxt_re_hwrm_qcaps(struct bnxt_re_dev *rdev) return rc; cctx->modes.db_push = le32_to_cpu(resp.flags) & FUNC_QCAPS_RESP_FLAGS_WCB_PUSH_MODE; - cctx->modes.dbr_pacing = - le32_to_cpu(resp.flags_ext2) & - FUNC_QCAPS_RESP_FLAGS_EXT2_DBR_PACING_EXT_SUPPORTED; + flags_ext2 = le32_to_cpu(resp.flags_ext2); + cctx->modes.dbr_pacing = flags_ext2 & FUNC_QCAPS_RESP_FLAGS_EXT2_DBR_PACING_EXT_SUPPORTED || + flags_ext2 & FUNC_QCAPS_RESP_FLAGS_EXT2_DBR_PACING_V0_SUPPORTED; return 0; } From patchwork Thu Jun 27 02:41:05 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Selvin Xavier X-Patchwork-Id: 13713768 Received: from mail-pf1-f171.google.com (mail-pf1-f171.google.com [209.85.210.171]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 91D35F9E4 for ; Thu, 27 Jun 2024 03:02:05 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.171 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719457327; cv=none; b=fau+EVQaUyqKXKsC9sq+meQe9ohpQO0vbasNnIbR9HJgYl1BDGYTppgtX6YmQbLOKyPOS8VsHny92O2ztUvQs4HjtwumKLJpGqMt8jkBH5JajQnY/zeEh87/ZbfmTHMuI5j1qYbiaBWwzDPZR0XnLlbwVowfM2tWwfIll+t7Hdg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719457327; c=relaxed/simple; bh=HKjXnfLm2m4bFdvLBpF3QCDc1g3UNbWlBNNtpbsQFSM=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: Content-Type; b=cn2BGFfuKIvQMd/J71wkoAGzEqhE0sLUxP55WVMsq/x9GrXFtZqII4gzLTWanKt0nlJQm51Ire+0IIXm3iQN5ULrRO2TzlZEYxjbZpCfU1H+RwHx9gYOL2s9GFzpLr215Ah0Rs1kF0+uyVdGSKGBzwW8oz95Bt2nSxfnLSpZWVw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=broadcom.com; spf=fail smtp.mailfrom=broadcom.com; dkim=pass (1024-bit key) header.d=broadcom.com header.i=@broadcom.com header.b=Pf6z3DZb; arc=none smtp.client-ip=209.85.210.171 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=broadcom.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=broadcom.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=broadcom.com header.i=@broadcom.com header.b="Pf6z3DZb" Received: by mail-pf1-f171.google.com with SMTP id d2e1a72fcca58-7065e2fe7d9so4200149b3a.3 for ; Wed, 26 Jun 2024 20:02:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; t=1719457325; x=1720062125; darn=vger.kernel.org; h=references:in-reply-to:message-id:date:subject:cc:to:from:from:to :cc:subject:date:message-id:reply-to; bh=BxR07j9lWmVKzHRqDbkJYOyqaI1iQZ6WqhMvpuLEi98=; b=Pf6z3DZbot/q3xNLy7CiwywxsDZptHq2r+I9AgOtAtx5jP9LFW80h/Fw4iTyHfP+qL FhvKSk7ER6Jg6XJfHbKRNuIKv06dKLxehYs6+HZUCBpXvbdkdw05Tl8WIFYa7ay+vttL DQueFNxsU3WAehYDmnJuRVfA/NxfyVYwmphYk= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1719457325; x=1720062125; h=references:in-reply-to:message-id:date:subject:cc:to:from :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=BxR07j9lWmVKzHRqDbkJYOyqaI1iQZ6WqhMvpuLEi98=; b=Nhb+4kDjeseB3kCEr91r5kz/tNU/rgR3jWl/FuQ+hR15pbikP77Unzx6Nt9hKR9OON LEu83/+sUcLbJYx+5pwvcUVLK24vt6qsZjV/2ENWSnKlPF9Gyx/H1Ub0w22GF5kMDfuh EqG6M3MHQ11OdIiDU5UPOgdNsAxqXsqyUOAXlv+VhweHf1DkdASF9XiCNeWWtYGfb0g+ yg/Tg4l6fjHep6KaJ9o7vNP/LhCZJ13KOim5N33QCnImbQNQDIIVl39J3xI8YSTK1daW vwmEhMCON4hyqfyhfhFdO0C3y4AKDuyAg2lqe6CVhD/eNT5KTQMZoh5GTyAS/Fla/kHH 2MKA== X-Gm-Message-State: AOJu0Yxdmsylvw6GMvAPE4MBLvJQ19nOyTTILL7N7RSgy3oWKDeaw9ml tTLUImJun6TbkF9LMOogpxpqZyZ8U/YXOmNN7KykFr11kIYHHjoZXplKYXoTOA== X-Google-Smtp-Source: AGHT+IGlDtCJ+D1onvo8x8HSvajywOz1qx3QqKg4H1CNQ/xF+fd79si4IBDrjJA/aGg7IQMO5Kw6+A== X-Received: by 2002:a05:6a20:3b05:b0:1bd:27fd:ff56 with SMTP id adf61e73a8af0-1bd27fe00a6mr5601925637.58.1719457324887; Wed, 26 Jun 2024 20:02:04 -0700 (PDT) Received: from sxavier-dev.dhcp.broadcom.net ([192.19.234.250]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-1faac997da7sm2103285ad.216.2024.06.26.20.02.02 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Wed, 26 Jun 2024 20:02:04 -0700 (PDT) From: Selvin Xavier To: leon@kernel.org, jgg@ziepe.ca Cc: linux-rdma@vger.kernel.org, andrew.gospodarek@broadcom.com, Selvin Xavier Subject: [PATCH for-next 3/3] RDMA/bnxt_re: Disable doorbell moderation if hardware register read fails Date: Wed, 26 Jun 2024 19:41:05 -0700 Message-Id: <1719456065-27394-4-git-send-email-selvin.xavier@broadcom.com> X-Mailer: git-send-email 2.5.5 In-Reply-To: <1719456065-27394-1-git-send-email-selvin.xavier@broadcom.com> References: <1719456065-27394-1-git-send-email-selvin.xavier@broadcom.com> Precedence: bulk X-Mailing-List: linux-rdma@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: If the HW register read fails, the FIFO will be always shown as full. DB moderation doesn't work in that case and the traffic fails. So disable this feature and log a message. Signed-off-by: Selvin Xavier --- drivers/infiniband/hw/bnxt_re/main.c | 45 +++++++++++++++++++++++++++--------- 1 file changed, 34 insertions(+), 11 deletions(-) diff --git a/drivers/infiniband/hw/bnxt_re/main.c b/drivers/infiniband/hw/bnxt_re/main.c index 2c5282f..9714b9a 100644 --- a/drivers/infiniband/hw/bnxt_re/main.c +++ b/drivers/infiniband/hw/bnxt_re/main.c @@ -488,19 +488,40 @@ static void bnxt_re_set_default_pacing_data(struct bnxt_re_dev *rdev) pacing_data->pacing_th * BNXT_RE_PACING_ALARM_TH_MULTIPLE; } -static void __wait_for_fifo_occupancy_below_th(struct bnxt_re_dev *rdev) +static u32 __get_fifo_occupancy(struct bnxt_re_dev *rdev) { struct bnxt_qplib_db_pacing_data *pacing_data = rdev->qplib_res.pacing_data; u32 read_val, fifo_occup; + read_val = readl(rdev->en_dev->bar0 + rdev->pacing.dbr_db_fifo_reg_off); + fifo_occup = pacing_data->fifo_max_depth - + ((read_val & pacing_data->fifo_room_mask) >> + pacing_data->fifo_room_shift); + return fifo_occup; +} + +static bool is_dbr_fifo_full(struct bnxt_re_dev *rdev) +{ + u32 max_occup, fifo_occup; + + fifo_occup = __get_fifo_occupancy(rdev); + max_occup = BNXT_RE_MAX_FIFO_DEPTH(rdev->chip_ctx) - 1; + if (fifo_occup == max_occup) + return true; + + return false; +} + +static void __wait_for_fifo_occupancy_below_th(struct bnxt_re_dev *rdev) +{ + struct bnxt_qplib_db_pacing_data *pacing_data = rdev->qplib_res.pacing_data; + u32 fifo_occup; + /* loop shouldn't run infintely as the occupancy usually goes * below pacing algo threshold as soon as pacing kicks in. */ while (1) { - read_val = readl(rdev->en_dev->bar0 + rdev->pacing.dbr_db_fifo_reg_off); - fifo_occup = pacing_data->fifo_max_depth - - ((read_val & pacing_data->fifo_room_mask) >> - pacing_data->fifo_room_shift); + fifo_occup = __get_fifo_occupancy(rdev); /* Fifo occupancy cannot be greater the MAX FIFO depth */ if (fifo_occup > pacing_data->fifo_max_depth) break; @@ -556,16 +577,13 @@ static void bnxt_re_pacing_timer_exp(struct work_struct *work) struct bnxt_re_dev *rdev = container_of(work, struct bnxt_re_dev, dbq_pacing_work.work); struct bnxt_qplib_db_pacing_data *pacing_data; - u32 read_val, fifo_occup; + u32 fifo_occup; if (!mutex_trylock(&rdev->pacing.dbq_lock)) return; pacing_data = rdev->qplib_res.pacing_data; - read_val = readl(rdev->en_dev->bar0 + rdev->pacing.dbr_db_fifo_reg_off); - fifo_occup = pacing_data->fifo_max_depth - - ((read_val & pacing_data->fifo_room_mask) >> - pacing_data->fifo_room_shift); + fifo_occup = __get_fifo_occupancy(rdev); if (fifo_occup > pacing_data->pacing_th) goto restart_timer; @@ -613,7 +631,6 @@ void bnxt_re_pacing_alert(struct bnxt_re_dev *rdev) static int bnxt_re_initialize_dbr_pacing(struct bnxt_re_dev *rdev) { - /* Allocate a page for app use */ rdev->pacing.dbr_page = (void *)__get_free_page(GFP_KERNEL); if (!rdev->pacing.dbr_page) @@ -637,6 +654,12 @@ static int bnxt_re_initialize_dbr_pacing(struct bnxt_re_dev *rdev) rdev->pacing.dbr_bar_addr = pci_resource_start(rdev->qplib_res.pdev, 0) + rdev->pacing.dbr_db_fifo_reg_off; + if (is_dbr_fifo_full(rdev)) { + free_page((u64)rdev->pacing.dbr_page); + rdev->pacing.dbr_page = NULL; + return -EIO; + } + rdev->pacing.pacing_algo_th = BNXT_RE_PACING_ALGO_THRESHOLD; rdev->pacing.dbq_pacing_time = BNXT_RE_DBR_PACING_TIME; rdev->pacing.dbr_def_do_pacing = BNXT_RE_DBR_DO_PACING_NO_CONGESTION;