From patchwork Wed Sep 11 04:08:27 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Selvin Xavier X-Patchwork-Id: 13799668 Received: from mail-pf1-f172.google.com (mail-pf1-f172.google.com [209.85.210.172]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0E3DBC144 for ; Wed, 11 Sep 2024 04:29:26 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1726028968; cv=none; b=EdNpPWTY7ohRaq5aTpLbWTR6iCEH6odGSu+eFdSgJj1W2Lqfxu1WUQK9Yb1/8f0C55tYwjel6mPUBzEN5gzq5YyHOsXJ97cylrNgQcaRjIXlvstGQ1tdOvqXg35FWCnTWVad7qgjfIO2w9od6mYxiPLBxHkFpxYxwBdu8sivXfM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1726028968; c=relaxed/simple; bh=TVu4ranskAbNFxfmLNliVBfZtsJxLRVxQXpuObN/QSk=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References; b=ifAxaPfQfbPCm7WOWv3+sIZ1NPdv9Bbp+8W+jzxSVpeDNifjAViTljfkw1dwfkx9S6rXIUN/tjKwUyVLuLUoXuhgaMfkfOIxhjR0U0DHo6He7a/s3/WGx9IrOtECahsACkHbjU9GHOFg9PkRrIBtQib6yXWSx7NIsc8W1fUomj4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=broadcom.com; spf=fail smtp.mailfrom=broadcom.com; dkim=pass (1024-bit key) header.d=broadcom.com header.i=@broadcom.com header.b=L6AHa1Eg; arc=none smtp.client-ip=209.85.210.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=broadcom.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=broadcom.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=broadcom.com header.i=@broadcom.com header.b="L6AHa1Eg" Received: by mail-pf1-f172.google.com with SMTP id d2e1a72fcca58-7179802b91fso3993284b3a.3 for ; Tue, 10 Sep 2024 21:29:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; t=1726028966; x=1726633766; darn=vger.kernel.org; h=references:in-reply-to:message-id:date:subject:cc:to:from:from:to :cc:subject:date:message-id:reply-to; bh=TW3Ul13S7ZGDx8wPnj3gwFimrABuOrRGEyLFve7q/VA=; b=L6AHa1EgYH8/VzSgjcFA4AiHa8FtrPLPp4Av8Z0ZP4inOmdl8ZVYvGvgQf1mLUCevm uHn0j1vts5s26lbziIyKOe+CMyyQRKF/GfhZUusw5ZdgzeMSTw//LyxWbsFpjhvViM7b NTw8n6IvoGIun1o7Efy22TEpSVdT3ZSCvYCa8= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1726028966; x=1726633766; h=references:in-reply-to:message-id:date:subject:cc:to:from :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=TW3Ul13S7ZGDx8wPnj3gwFimrABuOrRGEyLFve7q/VA=; b=cb0Y3LPmD7NcePSJbbvEKPSRTTSlNhdKc9tkJBvcg3ZteAXbSnQh82T5CrGtUJV3w5 9+M1Itmj7mxm/f/5fDqv1/rI10RgCP0MuEhEu8B3fl4O1FhoBI2yesiyyvjRcWms3dgh HwInBN/MCPYBDzBgeFqOi/EH3jYIIPp/g0Vz4qd3wPkyHj+NVmeGNPbN7ntmNsP1vqEe 4RqwtlOInoRQ7a3yAIz0cQWv+9zIDF+5s/Prn9j4qKoy/SzG0OT9IRHXBgq3OahxyJig hKJ9h2zfaV/SdcdCk3UTDbV/nYp/5YbYztGBVjA7kTtUff9Z9DEUAhLdhCMWzvkZb4vd +KCA== X-Gm-Message-State: AOJu0YxwTVdDnBo1RXyugz7SZTWcDgovLOSkKQYSXEVQzoMSp8Bv2xAE nZGZa1ViB0wxyOXqcs2WEjQ8mfaiTZuReVNWIQqC7kCbGZiLB6QmuF7nOa+cIQ== X-Google-Smtp-Source: AGHT+IH4/hLo8mytflkglXMCXnV05a9fe+TwqcLA4oAYef9iaUI+NhJjDYg3riuP/l+TMM/YUZC9Yg== X-Received: by 2002:a05:6a20:cd0e:b0:1cf:50e6:fee4 with SMTP id adf61e73a8af0-1cf5e19f82fmr4486124637.49.1726028966196; Tue, 10 Sep 2024 21:29:26 -0700 (PDT) Received: from sxavier-dev.dhcp.broadcom.net ([192.19.234.250]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-71909095184sm2123562b3a.106.2024.09.10.21.29.21 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 10 Sep 2024 21:29:23 -0700 (PDT) From: Selvin Xavier To: leon@kernel.org, jgg@ziepe.ca Cc: linux-rdma@vger.kernel.org, andrew.gospodarek@broadcom.com, kalesh-anakkur.purayil@broadcom.com, Chandramohan Akula , Selvin Xavier Subject: [PATCH for-next v2 1/4] RDMA/bnxt_re: Change aux driver data to en_info to hold more information Date: Tue, 10 Sep 2024 21:08:27 -0700 Message-Id: <1726027710-2292-2-git-send-email-selvin.xavier@broadcom.com> X-Mailer: git-send-email 2.5.5 In-Reply-To: <1726027710-2292-1-git-send-email-selvin.xavier@broadcom.com> References: <1726027710-2292-1-git-send-email-selvin.xavier@broadcom.com> Precedence: bulk X-Mailing-List: linux-rdma@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: From: Chandramohan Akula rdev will be destroyed and recreated during the FW error recovery scenarios. So to keep the state, if any, use an en_info structure which gets created/freed based on auxiliary device initialization/de-initialization. Signed-off-by: Chandramohan Akula Reviewed-by: Kashyap Desai Reviewed-by: Kalesh AP Signed-off-by: Selvin Xavier --- drivers/infiniband/hw/bnxt_re/bnxt_re.h | 6 +++ drivers/infiniband/hw/bnxt_re/main.c | 71 ++++++++++++++++++++++++++++----- 2 files changed, 66 insertions(+), 11 deletions(-) diff --git a/drivers/infiniband/hw/bnxt_re/bnxt_re.h b/drivers/infiniband/hw/bnxt_re/bnxt_re.h index 2be9a62..5df3ce1 100644 --- a/drivers/infiniband/hw/bnxt_re/bnxt_re.h +++ b/drivers/infiniband/hw/bnxt_re/bnxt_re.h @@ -107,6 +107,11 @@ struct bnxt_re_gsi_context { struct bnxt_re_sqp_entries *sqp_tbl; }; +struct bnxt_re_en_dev_info { + struct bnxt_en_dev *en_dev; + struct bnxt_re_dev *rdev; +}; + #define BNXT_RE_AEQ_IDX 0 #define BNXT_RE_NQ_IDX 1 #define BNXT_RE_GEN_P5_MAX_VF 64 @@ -155,6 +160,7 @@ struct bnxt_re_dev { #define BNXT_RE_FLAG_ERR_DEVICE_DETACHED 17 #define BNXT_RE_FLAG_ISSUE_ROCE_STATS 29 struct net_device *netdev; + struct auxiliary_device *adev; struct notifier_block nb; unsigned int version, major, minor; struct bnxt_qplib_chip_ctx *chip_ctx; diff --git a/drivers/infiniband/hw/bnxt_re/main.c b/drivers/infiniband/hw/bnxt_re/main.c index 16a84ca..085a03c 100644 --- a/drivers/infiniband/hw/bnxt_re/main.c +++ b/drivers/infiniband/hw/bnxt_re/main.c @@ -292,10 +292,13 @@ static void bnxt_re_vf_res_config(struct bnxt_re_dev *rdev) static void bnxt_re_shutdown(struct auxiliary_device *adev) { - struct bnxt_re_dev *rdev = auxiliary_get_drvdata(adev); + struct bnxt_re_en_dev_info *en_info = auxiliary_get_drvdata(adev); + struct bnxt_re_dev *rdev; - if (!rdev) + if (!en_info) return; + + rdev = en_info->rdev; ib_unregister_device(&rdev->ibdev); bnxt_re_dev_uninit(rdev); } @@ -1794,14 +1797,33 @@ static int bnxt_re_dev_init(struct bnxt_re_dev *rdev) return rc; } +static void bnxt_re_update_en_info_rdev(struct bnxt_re_dev *rdev, + struct bnxt_re_en_dev_info *en_info, + struct auxiliary_device *adev) +{ + /* Before updating the rdev pointer in bnxt_re_en_dev_info structure, + * take the rtnl lock to avoid accessing invalid rdev pointer from + * L2 ULP callbacks. This is applicable in all the places where rdev + * pointer is updated in bnxt_re_en_dev_info. + */ + rtnl_lock(); + en_info->rdev = rdev; + rdev->adev = adev; + rtnl_unlock(); +} + static int bnxt_re_add_device(struct auxiliary_device *adev) { struct bnxt_aux_priv *aux_priv = container_of(adev, struct bnxt_aux_priv, aux_dev); + struct bnxt_re_en_dev_info *en_info; struct bnxt_en_dev *en_dev; struct bnxt_re_dev *rdev; int rc; + en_info = auxiliary_get_drvdata(adev); + en_dev = en_info->en_dev; + /* en_dev should never be NULL as long as adev and aux_dev are valid. */ en_dev = aux_priv->edev; @@ -1811,6 +1833,8 @@ static int bnxt_re_add_device(struct auxiliary_device *adev) goto exit; } + bnxt_re_update_en_info_rdev(rdev, en_info, adev); + rc = bnxt_re_dev_init(rdev); if (rc) goto re_dev_dealloc; @@ -1821,11 +1845,11 @@ static int bnxt_re_add_device(struct auxiliary_device *adev) aux_priv->aux_dev.name); goto re_dev_uninit; } - auxiliary_set_drvdata(adev, rdev); return 0; re_dev_uninit: + bnxt_re_update_en_info_rdev(NULL, en_info, adev); bnxt_re_dev_uninit(rdev); re_dev_dealloc: ib_dealloc_device(&rdev->ibdev); @@ -1911,12 +1935,18 @@ static int bnxt_re_netdev_event(struct notifier_block *notifier, static void bnxt_re_remove(struct auxiliary_device *adev) { - struct bnxt_re_dev *rdev = auxiliary_get_drvdata(adev); + struct bnxt_re_en_dev_info *en_info = auxiliary_get_drvdata(adev); + struct bnxt_re_dev *rdev; - if (!rdev) + mutex_lock(&bnxt_re_mutex); + if (!en_info) { + mutex_unlock(&bnxt_re_mutex); return; + } + rdev = en_info->rdev; + if (!rdev) + goto skip_remove; - mutex_lock(&bnxt_re_mutex); if (rdev->nb.notifier_call) { unregister_netdevice_notifier(&rdev->nb); rdev->nb.notifier_call = NULL; @@ -1931,16 +1961,31 @@ static void bnxt_re_remove(struct auxiliary_device *adev) bnxt_re_dev_uninit(rdev); ib_dealloc_device(&rdev->ibdev); skip_remove: + kfree(en_info); mutex_unlock(&bnxt_re_mutex); } static int bnxt_re_probe(struct auxiliary_device *adev, const struct auxiliary_device_id *id) { + struct bnxt_aux_priv *aux_priv = + container_of(adev, struct bnxt_aux_priv, aux_dev); + struct bnxt_re_en_dev_info *en_info; + struct bnxt_en_dev *en_dev; struct bnxt_re_dev *rdev; int rc; + en_dev = aux_priv->edev; + mutex_lock(&bnxt_re_mutex); + en_info = kzalloc(sizeof(*en_info), GFP_KERNEL); + if (!en_info) { + mutex_unlock(&bnxt_re_mutex); + return -ENOMEM; + } + en_info->en_dev = en_dev; + + auxiliary_set_drvdata(adev, en_info); rc = bnxt_re_add_device(adev); if (rc) { @@ -1948,7 +1993,7 @@ static int bnxt_re_probe(struct auxiliary_device *adev, return rc; } - rdev = auxiliary_get_drvdata(adev); + rdev = en_info->rdev; rdev->nb.notifier_call = bnxt_re_netdev_event; rc = register_netdevice_notifier(&rdev->nb); @@ -1972,11 +2017,13 @@ static int bnxt_re_probe(struct auxiliary_device *adev, static int bnxt_re_suspend(struct auxiliary_device *adev, pm_message_t state) { - struct bnxt_re_dev *rdev = auxiliary_get_drvdata(adev); + struct bnxt_re_en_dev_info *en_info = auxiliary_get_drvdata(adev); + struct bnxt_re_dev *rdev; - if (!rdev) + if (!en_info) return 0; + rdev = en_info->rdev; mutex_lock(&bnxt_re_mutex); /* L2 driver may invoke this callback during device error/crash or device * reset. Current RoCE driver doesn't recover the device in case of @@ -2009,11 +2056,13 @@ static int bnxt_re_suspend(struct auxiliary_device *adev, pm_message_t state) static int bnxt_re_resume(struct auxiliary_device *adev) { - struct bnxt_re_dev *rdev = auxiliary_get_drvdata(adev); + struct bnxt_re_en_dev_info *en_info = auxiliary_get_drvdata(adev); + struct bnxt_re_dev *rdev; - if (!rdev) + if (!en_info) return 0; + rdev = en_info->rdev; mutex_lock(&bnxt_re_mutex); /* L2 driver may invoke this callback during device recovery, resume. * reset. Current RoCE driver doesn't recover the device in case of From patchwork Wed Sep 11 04:08:28 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Selvin Xavier X-Patchwork-Id: 13799669 Received: from mail-pg1-f172.google.com (mail-pg1-f172.google.com [209.85.215.172]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 28B25C144 for ; Wed, 11 Sep 2024 04:29:31 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1726028973; cv=none; b=VXjYi3R2dMcGKoaDT2RlIvIdxyhO9gDM35k8tWVBO5u8hWWVpz0qCqqKem1xGn9m5sjpJ9ym4J6oXrtYYeY4ws/yN1j8Vr/CZhR0vnl9MbokO9VwkXkbriI1Mn6YkKl0l4u/SMbvgHtsomsCgoC4Bml7rAmcb2VKrC8OTF6h2WM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1726028973; c=relaxed/simple; bh=GbRpg1h0UWw1sY8RVEVPG8k18EfZQJuKsBwJRdJBQT4=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References; b=X09CpyeA5bYqQ+zJMbqavgCawKEU6jkK4PBslct5B6Te6pZLOJQdUydaUknzQmJO4AiopHM9u8DSGLSgwnGRxRyrjVZEGRXJiiGmK6jYHtLeqZcgM8l+PwTlavam7WYIEF5qvB8fKMWr2tdn36s0MJKwd8s3KajgxhEdB5zObQs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=broadcom.com; spf=fail smtp.mailfrom=broadcom.com; dkim=pass (1024-bit key) header.d=broadcom.com header.i=@broadcom.com header.b=QQEr0APA; arc=none smtp.client-ip=209.85.215.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=broadcom.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=broadcom.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=broadcom.com header.i=@broadcom.com header.b="QQEr0APA" Received: by mail-pg1-f172.google.com with SMTP id 41be03b00d2f7-7db0fb03df5so192385a12.3 for ; Tue, 10 Sep 2024 21:29:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; t=1726028971; x=1726633771; darn=vger.kernel.org; h=references:in-reply-to:message-id:date:subject:cc:to:from:from:to :cc:subject:date:message-id:reply-to; bh=r2eul6sBj4rHlzswzbS767ryuCxywOlCbgVkH5S8zQs=; b=QQEr0APAWOk6FDCWe+oCU3/vumpc4g9B9fpzYezHdY9dRdkKFdq3aqJi/SUDNfiROW hqYg9ttP0QHKyUNxSDNwoyEMMjYSPkwKfyjtngQOToCBnohe5G229u238kM4NEH1jDxl +LFbYgn3kDGWgtsSCyfxRwsl+vMltxXcR5wzA= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1726028971; x=1726633771; h=references:in-reply-to:message-id:date:subject:cc:to:from :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=r2eul6sBj4rHlzswzbS767ryuCxywOlCbgVkH5S8zQs=; b=dxf/HmN3hWZ4/7dr1nQxGam5hHT+Tsal6+r4phxCPjM3GxkVktNV1C+efBsBtGr4a3 gF9lC3nQEnyOfUOKSRrddH9jH08hBbP5MF+BrlZpg6tUWZLkV1eZP9kBbPqrECNEn3ik vORsi8SZGs/A+6BDy6NfCP0y8piiWrWbpybdSPR+D8hclt5mK/DfH+Bo06x20wNeITff 6z5EBmjZM+AFEH00L3Eud1/4YxzYwpuQ7pPLLxz7vP5bWQm8J8mGTz9WMMdNKzrBiT+s b44fnsHb09ZumgHhx3MK4sPN6VTu4EffnT22MiOC/kqOSgQagG+aLvR+RfrNcBhLfXI6 nMBA== X-Gm-Message-State: AOJu0YyNCAotEFua2X61wbPOjdeUvtZyFN0wxNXqyQSF/tcYJIwD2EYF PlVnvMULR9QB49WbOI1VZ/1q/zLfsOxYZx4n8xjUNllA4h5htpc9Lnva3RLB1Q== X-Google-Smtp-Source: AGHT+IFhy8Iq3WejVOjuLSiqYzK1v6Js4HGnjClVX+dioJV8MMw/4Bb4KuepzRicngA+pUWS6SejRg== X-Received: by 2002:a05:6a20:d528:b0:1cf:32d1:489 with SMTP id adf61e73a8af0-1cf5e03f737mr4890729637.1.1726028971209; Tue, 10 Sep 2024 21:29:31 -0700 (PDT) Received: from sxavier-dev.dhcp.broadcom.net ([192.19.234.250]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-71909095184sm2123562b3a.106.2024.09.10.21.29.26 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 10 Sep 2024 21:29:28 -0700 (PDT) From: Selvin Xavier To: leon@kernel.org, jgg@ziepe.ca Cc: linux-rdma@vger.kernel.org, andrew.gospodarek@broadcom.com, kalesh-anakkur.purayil@broadcom.com, Chandramohan Akula , Selvin Xavier Subject: [PATCH for-next v2 2/4] RDMA/bnxt_re: Use the aux device for L2 ULP callbacks Date: Tue, 10 Sep 2024 21:08:28 -0700 Message-Id: <1726027710-2292-3-git-send-email-selvin.xavier@broadcom.com> X-Mailer: git-send-email 2.5.5 In-Reply-To: <1726027710-2292-1-git-send-email-selvin.xavier@broadcom.com> References: <1726027710-2292-1-git-send-email-selvin.xavier@broadcom.com> Precedence: bulk X-Mailing-List: linux-rdma@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: From: Chandramohan Akula While registering with the L2 for ULP operations, use the aux device pointer as the handle. Aux device has the data bnxt_re_en_dev_info, which is used to store required information for the bnxt_re_suspend and bnxt_re_resume functions. Signed-off-by: Chandramohan Akula Reviewed-by: Kalesh AP Reviewed-by: Kashyap Desai Signed-off-by: Selvin Xavier --- drivers/infiniband/hw/bnxt_re/main.c | 26 ++++++++++++++++++++------ 1 file changed, 20 insertions(+), 6 deletions(-) diff --git a/drivers/infiniband/hw/bnxt_re/main.c b/drivers/infiniband/hw/bnxt_re/main.c index 085a03c..2a91699 100644 --- a/drivers/infiniband/hw/bnxt_re/main.c +++ b/drivers/infiniband/hw/bnxt_re/main.c @@ -305,11 +305,18 @@ static void bnxt_re_shutdown(struct auxiliary_device *adev) static void bnxt_re_stop_irq(void *handle) { - struct bnxt_re_dev *rdev = (struct bnxt_re_dev *)handle; - struct bnxt_qplib_rcfw *rcfw = &rdev->rcfw; + struct bnxt_re_en_dev_info *en_info = auxiliary_get_drvdata(handle); + struct bnxt_qplib_rcfw *rcfw; + struct bnxt_re_dev *rdev; struct bnxt_qplib_nq *nq; int indx; + if (!en_info) + return; + + rdev = en_info->rdev; + rcfw = &rdev->rcfw; + for (indx = BNXT_RE_NQ_IDX; indx < rdev->num_msix; indx++) { nq = &rdev->nq[indx - 1]; bnxt_qplib_nq_stop_irq(nq, false); @@ -320,12 +327,19 @@ static void bnxt_re_stop_irq(void *handle) static void bnxt_re_start_irq(void *handle, struct bnxt_msix_entry *ent) { - struct bnxt_re_dev *rdev = (struct bnxt_re_dev *)handle; - struct bnxt_msix_entry *msix_ent = rdev->en_dev->msix_entries; - struct bnxt_qplib_rcfw *rcfw = &rdev->rcfw; + struct bnxt_re_en_dev_info *en_info = auxiliary_get_drvdata(handle); + struct bnxt_msix_entry *msix_ent; + struct bnxt_qplib_rcfw *rcfw; + struct bnxt_re_dev *rdev; struct bnxt_qplib_nq *nq; int indx, rc; + if (!en_info) + return; + + rdev = en_info->rdev; + msix_ent = rdev->en_dev->msix_entries; + rcfw = &rdev->rcfw; if (!ent) { /* Not setting the f/w timeout bit in rcfw. * During the driver unload the first command @@ -374,7 +388,7 @@ static int bnxt_re_register_netdev(struct bnxt_re_dev *rdev) en_dev = rdev->en_dev; - rc = bnxt_register_dev(en_dev, &bnxt_re_ulp_ops, rdev); + rc = bnxt_register_dev(en_dev, &bnxt_re_ulp_ops, rdev->adev); if (!rc) rdev->qplib_res.pdev = rdev->en_dev->pdev; return rc; From patchwork Wed Sep 11 04:08:29 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Selvin Xavier X-Patchwork-Id: 13799670 Received: from mail-pf1-f177.google.com (mail-pf1-f177.google.com [209.85.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D9C33C144 for ; Wed, 11 Sep 2024 04:29:36 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.177 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1726028978; cv=none; b=M+oRHkQ6r2Dh6wNMinyMIORrrpIYfMlgyO5TMec4aOpZEaYWDng8igV/qgC6Jhhdqu5P9AwMhjRh+gYxr3mL6XO6//hTKIeVxVUob2DC/5Hr++t86BxNioo/7DX9rj1N1B+RKjewpmeOq4BV6hT6PLC0UtF281nsvKF6bzjhYL4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1726028978; c=relaxed/simple; bh=5LxvtAbuEc9ZYXKjugmj/JehWdjeJaSPW5xgXl5XuS8=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References; b=su0o6fjmLVS9JT81X3d/iLM4WO0HkIvFpR+SW1ELx3v8Yohol5xR5mpp2c4wZncPS5vBDzBsglY8fiALdtyTyDi1zkCCjA6pqoKi98NQuKiIYY9V/OI1jl4LsUhHvI/cxA6M9lqw8RScViOHp8ouXtazUQIOg2xKuWxeUxPE9y8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=broadcom.com; spf=fail smtp.mailfrom=broadcom.com; dkim=pass (1024-bit key) header.d=broadcom.com header.i=@broadcom.com header.b=W1vF6+hI; arc=none smtp.client-ip=209.85.210.177 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=broadcom.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=broadcom.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=broadcom.com header.i=@broadcom.com header.b="W1vF6+hI" Received: by mail-pf1-f177.google.com with SMTP id d2e1a72fcca58-71911585ac4so1092704b3a.1 for ; Tue, 10 Sep 2024 21:29:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; t=1726028976; x=1726633776; darn=vger.kernel.org; h=references:in-reply-to:message-id:date:subject:cc:to:from:from:to :cc:subject:date:message-id:reply-to; bh=sRe8rJNyl95P8tZdEPQ0yoe4cOHwkuQlLU1cuTL5PJM=; b=W1vF6+hIl8uzM42WhPfVvrSfGAHOR1HUo8NueNm+82tIV8Om6NMabsJUJa2Cwfm0YI 6pmbFyv1KUTZ40ZtZHpB5XIk9JCzHTHmHPIsvA3j53TqjfmT6Kx6ouNBpb48mDaIrjLM LlrwXvSQJYBePCr6mvUdD/cfdxst6yNbgM2zM= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1726028976; x=1726633776; h=references:in-reply-to:message-id:date:subject:cc:to:from :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=sRe8rJNyl95P8tZdEPQ0yoe4cOHwkuQlLU1cuTL5PJM=; b=cQ0TivlBoPQfhksHw6NlvwY8aGibvp4gDDlFS2i8LMKZvRsrGTlKODLHsu2U6V3DhI XjjI9vDZ/OexI+cuT9ii8q1JOawQ6jiVbOU3cga77l40fVmaRbxSHF+IjGIyo605qbD7 lO2kqkdEG/0p6nOJzNomiWK3iSyXf2U8PNKQw0pDPseA8OcfAcoef1rhul5Crfa9TYmz grtv7wkTVdO8cQqsxoRbY0GpwCZVsPFnpwblPB27GXPkQqNiC0nvhPJxqn9FzFTq28R5 iKMx1oz2G8ZiKxc0YCrks7//UbOwu6CtLmDRZza6ptNfOr51qPKRKAEgazh2OvmyMKIs 3nKQ== X-Gm-Message-State: AOJu0YxKgaAKPd6GFklOu3bTWbHZNuSKqvVNLzOUDGzgD4DK3+UgRKPq dVbku4kQ3ajy82aKaUsebp6JuwxdG0Wcm0A4t+REaT7v6UNWRJ0gBO4LF4hqng== X-Google-Smtp-Source: AGHT+IGzBtUt5U1trOUOCpQquHw1RkMmg/ktmOklb1SDWWQykH00Edpg1EYFd0OthQIpXGlrLRjIxw== X-Received: by 2002:a05:6a00:987:b0:717:8e49:37ce with SMTP id d2e1a72fcca58-718d5f3293cmr23400113b3a.21.1726028975830; Tue, 10 Sep 2024 21:29:35 -0700 (PDT) Received: from sxavier-dev.dhcp.broadcom.net ([192.19.234.250]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-71909095184sm2123562b3a.106.2024.09.10.21.29.31 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 10 Sep 2024 21:29:33 -0700 (PDT) From: Selvin Xavier To: leon@kernel.org, jgg@ziepe.ca Cc: linux-rdma@vger.kernel.org, andrew.gospodarek@broadcom.com, kalesh-anakkur.purayil@broadcom.com, Selvin Xavier , Chandramohan Akula Subject: [PATCH for-next v2 3/4] RDMA/bnxt_re: Group all operations under add_device and remove_device Date: Tue, 10 Sep 2024 21:08:29 -0700 Message-Id: <1726027710-2292-4-git-send-email-selvin.xavier@broadcom.com> X-Mailer: git-send-email 2.5.5 In-Reply-To: <1726027710-2292-1-git-send-email-selvin.xavier@broadcom.com> References: <1726027710-2292-1-git-send-email-selvin.xavier@broadcom.com> Precedence: bulk X-Mailing-List: linux-rdma@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Adding and removing device need to be handled from multiple contexts when Firmware error recovery is supported. So group all the add and remove operations to add_device and remove_device function. Signed-off-by: Chandramohan Akula Reviewed-by: Kalesh AP Signed-off-by: Selvin Xavier --- drivers/infiniband/hw/bnxt_re/main.c | 65 ++++++++++++++++++------------------ 1 file changed, 33 insertions(+), 32 deletions(-) diff --git a/drivers/infiniband/hw/bnxt_re/main.c b/drivers/infiniband/hw/bnxt_re/main.c index 2a91699..dc63ad0 100644 --- a/drivers/infiniband/hw/bnxt_re/main.c +++ b/drivers/infiniband/hw/bnxt_re/main.c @@ -88,6 +88,7 @@ static int bnxt_re_hwrm_qcaps(struct bnxt_re_dev *rdev); static int bnxt_re_hwrm_qcfg(struct bnxt_re_dev *rdev, u32 *db_len, u32 *offset); +static void bnxt_re_setup_cc(struct bnxt_re_dev *rdev, bool enable); static void bnxt_re_set_db_offset(struct bnxt_re_dev *rdev) { struct bnxt_qplib_chip_ctx *cctx; @@ -1860,6 +1861,16 @@ static int bnxt_re_add_device(struct auxiliary_device *adev) goto re_dev_uninit; } + rdev->nb.notifier_call = bnxt_re_netdev_event; + rc = register_netdevice_notifier(&rdev->nb); + if (rc) { + rdev->nb.notifier_call = NULL; + pr_err("%s: Cannot register to netdevice_notifier", + ROCE_DRV_MODULE_NAME); + return rc; + } + bnxt_re_setup_cc(rdev, true); + return 0; re_dev_uninit: @@ -1947,20 +1958,9 @@ static int bnxt_re_netdev_event(struct notifier_block *notifier, #define BNXT_ADEV_NAME "bnxt_en" -static void bnxt_re_remove(struct auxiliary_device *adev) +static void bnxt_re_remove_device(struct bnxt_re_dev *rdev, + struct auxiliary_device *aux_dev) { - struct bnxt_re_en_dev_info *en_info = auxiliary_get_drvdata(adev); - struct bnxt_re_dev *rdev; - - mutex_lock(&bnxt_re_mutex); - if (!en_info) { - mutex_unlock(&bnxt_re_mutex); - return; - } - rdev = en_info->rdev; - if (!rdev) - goto skip_remove; - if (rdev->nb.notifier_call) { unregister_netdevice_notifier(&rdev->nb); rdev->nb.notifier_call = NULL; @@ -1968,13 +1968,30 @@ static void bnxt_re_remove(struct auxiliary_device *adev) /* If notifier is null, we should have already done a * clean up before coming here. */ - goto skip_remove; + return; } bnxt_re_setup_cc(rdev, false); ib_unregister_device(&rdev->ibdev); bnxt_re_dev_uninit(rdev); ib_dealloc_device(&rdev->ibdev); -skip_remove: +} + +static void bnxt_re_remove(struct auxiliary_device *adev) +{ + struct bnxt_re_en_dev_info *en_info = auxiliary_get_drvdata(adev); + struct bnxt_en_dev *en_dev; + struct bnxt_re_dev *rdev; + + mutex_lock(&bnxt_re_mutex); + if (!en_info) { + mutex_unlock(&bnxt_re_mutex); + return; + } + en_dev = en_info->en_dev; + rdev = en_info->rdev; + + if (rdev) + bnxt_re_remove_device(rdev, adev); kfree(en_info); mutex_unlock(&bnxt_re_mutex); } @@ -1986,7 +2003,6 @@ static int bnxt_re_probe(struct auxiliary_device *adev, container_of(adev, struct bnxt_aux_priv, aux_dev); struct bnxt_re_en_dev_info *en_info; struct bnxt_en_dev *en_dev; - struct bnxt_re_dev *rdev; int rc; en_dev = aux_priv->edev; @@ -2002,23 +2018,8 @@ static int bnxt_re_probe(struct auxiliary_device *adev, auxiliary_set_drvdata(adev, en_info); rc = bnxt_re_add_device(adev); - if (rc) { - mutex_unlock(&bnxt_re_mutex); - return rc; - } - - rdev = en_info->rdev; - - rdev->nb.notifier_call = bnxt_re_netdev_event; - rc = register_netdevice_notifier(&rdev->nb); - if (rc) { - rdev->nb.notifier_call = NULL; - pr_err("%s: Cannot register to netdevice_notifier", - ROCE_DRV_MODULE_NAME); + if (rc) goto err; - } - - bnxt_re_setup_cc(rdev, true); mutex_unlock(&bnxt_re_mutex); return 0; From patchwork Wed Sep 11 04:08:30 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Selvin Xavier X-Patchwork-Id: 13799671 Received: from mail-pf1-f182.google.com (mail-pf1-f182.google.com [209.85.210.182]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5B1FEC144 for ; Wed, 11 Sep 2024 04:29:42 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.182 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1726028984; cv=none; b=CFHLr+bjLNNhPr1hRigxDwXIW0QKD7rE2Ep/wsCJeCGyYHz79m3vC9SfyXEHFkC2vdc46XGD5BLqyKzBQhkSyFN9ClWdP/s47bB3b/HGPJX6Bsxyv7FNZtIowxI+OBf4wczDkC6Zus4ukKg2xRG9W/PYb/9hHmLCvINjO6z5PuA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1726028984; c=relaxed/simple; bh=YPvIcX2EZsZ9Jx2QA9o4RSACQ+xRLTs6IWnFEX1aX2w=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References; b=RmAF9PYD5LQJwmicsQpUxMD5dbPMZXzWK6sceHGonUUrSFuKCsYJ29YRSHBJ8ia5r4QwhPNfLB8lyr23S7NDCLsU7xNZYLb5u21Ktp4o9bX0oPbc8FqU4M/cvEclHMZZjeTX6qO6/UYTshgnubpwSQcPInPRfR5940feVeESt10= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=broadcom.com; spf=fail smtp.mailfrom=broadcom.com; dkim=pass (1024-bit key) header.d=broadcom.com header.i=@broadcom.com header.b=FexQO3C9; arc=none smtp.client-ip=209.85.210.182 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=broadcom.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=broadcom.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=broadcom.com header.i=@broadcom.com header.b="FexQO3C9" Received: by mail-pf1-f182.google.com with SMTP id d2e1a72fcca58-718f4fd89e5so1468886b3a.0 for ; Tue, 10 Sep 2024 21:29:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; t=1726028982; x=1726633782; darn=vger.kernel.org; h=references:in-reply-to:message-id:date:subject:cc:to:from:from:to :cc:subject:date:message-id:reply-to; bh=JQ2nrpdpUtq/LFuizqLWUBVnskmrs44so0Ymt7LDKME=; b=FexQO3C9CvXpqZ32+7eNV1uVjHm86uKZ+XhxpUjEkIiP3iV5nxImeFzdWWLKpA3VUq GJU1gFPK67BJoEdzVtjtgy6422ApqsVqp26rpPF82XNOtHTOx4k0JUs83EWfpP6cUJuM elALmEMyyNbDyJQC+3Ftn9paIy8YqIgcEwQqo= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1726028982; x=1726633782; h=references:in-reply-to:message-id:date:subject:cc:to:from :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=JQ2nrpdpUtq/LFuizqLWUBVnskmrs44so0Ymt7LDKME=; b=gof9yXURrLzeINNVr4knSwMy6hfUCUOPOvscl5Vde9R4LMuT0owrCW+hNEhQM0nkBU 0dCz7P80ATCDKmALNZu0ySe+8HcZ8evu2eALr1tUaZoUPwwrjKmovYnD6Bg4rL2wg2ZH 06pFyHaT0mSDqfzS1FWZdmqvVCsvU8QCD2UJoZng4pJG7GJgmOoDApOMt2IlQdgPjmns hsXhvJ4gNIgOZITNHv8UX19Ki+G1K9RUpswW19x7Iii0dLOqlLDNw2hXtlcsM75iEo3o BlIS01NM8wBLN3bAlzLmNWcFDuLb5DKQjGZ8v5lQFE9WSS/vE1bbq+qejGywKq47vwV+ FzOQ== X-Gm-Message-State: AOJu0YwXPWBOnpgJYMgRZykDT5SXkThdBQGpv2FXhK5vBRzd93/WZFfB VsBaQBTlvL8RHOTtR4XyJDdQqt1+iVtJOYEqHlkXV4JHHBucel6/KWAWxG5IeQ== X-Google-Smtp-Source: AGHT+IH4ZoUStFjdOF3Hwm4RqsQWKIFUvrHaP80wCTPGO0ufKKyZqxc7eN+6cPhzLpuupDJxY9L7IA== X-Received: by 2002:a05:6a00:9a1:b0:713:f127:ad5f with SMTP id d2e1a72fcca58-71916e98be4mr2993771b3a.22.1726028981321; Tue, 10 Sep 2024 21:29:41 -0700 (PDT) Received: from sxavier-dev.dhcp.broadcom.net ([192.19.234.250]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-71909095184sm2123562b3a.106.2024.09.10.21.29.36 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 10 Sep 2024 21:29:38 -0700 (PDT) From: Selvin Xavier To: leon@kernel.org, jgg@ziepe.ca Cc: linux-rdma@vger.kernel.org, andrew.gospodarek@broadcom.com, kalesh-anakkur.purayil@broadcom.com, Selvin Xavier , Chandramohan Akula Subject: [PATCH for-next v2 4/4] RDMA/bnxt_re: Recover the device when FW error is detected Date: Tue, 10 Sep 2024 21:08:30 -0700 Message-Id: <1726027710-2292-5-git-send-email-selvin.xavier@broadcom.com> X-Mailer: git-send-email 2.5.5 In-Reply-To: <1726027710-2292-1-git-send-email-selvin.xavier@broadcom.com> References: <1726027710-2292-1-git-send-email-selvin.xavier@broadcom.com> Precedence: bulk X-Mailing-List: linux-rdma@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: If the FW crashes, L2 driver gets notified and it notifies the RoCE driver. Currently driver doesn't re-initialize the device. Add support for re-initialize the RoCE device. RoCE device is removed and re-attached in the ulp_stop and ulp_start respectively. The recovery logic expects the RoCE driver to be registered with L2 driver while its being removed. So the driver avoids unregistering with L2 driver in the recovery path. Signed-off-by: Chandramohan Akula Signed-off-by: Kalesh AP Signed-off-by: Selvin Xavier --- drivers/infiniband/hw/bnxt_re/bnxt_re.h | 15 +++++++ drivers/infiniband/hw/bnxt_re/main.c | 70 +++++++++++++++++-------------- drivers/infiniband/hw/bnxt_re/qplib_res.h | 1 + 3 files changed, 55 insertions(+), 31 deletions(-) diff --git a/drivers/infiniband/hw/bnxt_re/bnxt_re.h b/drivers/infiniband/hw/bnxt_re/bnxt_re.h index 5df3ce1..e94518b 100644 --- a/drivers/infiniband/hw/bnxt_re/bnxt_re.h +++ b/drivers/infiniband/hw/bnxt_re/bnxt_re.h @@ -91,6 +91,15 @@ struct bnxt_re_ring_attr { u8 mode; }; +/* + * Data structure and defines to handle + * recovery + */ +#define BNXT_RE_PRE_RECOVERY_REMOVE 0x1 +#define BNXT_RE_COMPLETE_REMOVE 0x2 +#define BNXT_RE_POST_RECOVERY_INIT 0x4 +#define BNXT_RE_COMPLETE_INIT 0x8 + struct bnxt_re_sqp_entries { struct bnxt_qplib_sge sge; u64 wrid; @@ -224,4 +233,10 @@ static inline struct device *rdev_to_dev(struct bnxt_re_dev *rdev) } extern const struct uapi_definition bnxt_re_uapi_defs[]; + +static inline void bnxt_re_set_pacing_dev_state(struct bnxt_re_dev *rdev) +{ + rdev->qplib_res.pacing_data->dev_err_state = + test_bit(BNXT_RE_FLAG_ERR_DEVICE_DETACHED, &rdev->flags); +} #endif diff --git a/drivers/infiniband/hw/bnxt_re/main.c b/drivers/infiniband/hw/bnxt_re/main.c index dc63ad0..adff9e4 100644 --- a/drivers/infiniband/hw/bnxt_re/main.c +++ b/drivers/infiniband/hw/bnxt_re/main.c @@ -83,7 +83,7 @@ static void bnxt_re_dev_stop(struct bnxt_re_dev *rdev); static int bnxt_re_netdev_event(struct notifier_block *notifier, unsigned long event, void *ptr); static struct bnxt_re_dev *bnxt_re_from_netdev(struct net_device *netdev); -static void bnxt_re_dev_uninit(struct bnxt_re_dev *rdev); +static void bnxt_re_dev_uninit(struct bnxt_re_dev *rdev, u8 op_type); static int bnxt_re_hwrm_qcaps(struct bnxt_re_dev *rdev); static int bnxt_re_hwrm_qcfg(struct bnxt_re_dev *rdev, u32 *db_len, @@ -169,6 +169,7 @@ static int bnxt_re_setup_chip_ctx(struct bnxt_re_dev *rdev) en_dev = rdev->en_dev; + rdev->qplib_res.pdev = en_dev->pdev; chip_ctx = kzalloc(sizeof(*chip_ctx), GFP_KERNEL); if (!chip_ctx) return -ENOMEM; @@ -301,7 +302,7 @@ static void bnxt_re_shutdown(struct auxiliary_device *adev) rdev = en_info->rdev; ib_unregister_device(&rdev->ibdev); - bnxt_re_dev_uninit(rdev); + bnxt_re_dev_uninit(rdev, BNXT_RE_COMPLETE_REMOVE); } static void bnxt_re_stop_irq(void *handle) @@ -385,14 +386,9 @@ static struct bnxt_ulp_ops bnxt_re_ulp_ops = { static int bnxt_re_register_netdev(struct bnxt_re_dev *rdev) { struct bnxt_en_dev *en_dev; - int rc; en_dev = rdev->en_dev; - - rc = bnxt_register_dev(en_dev, &bnxt_re_ulp_ops, rdev->adev); - if (!rc) - rdev->qplib_res.pdev = rdev->en_dev->pdev; - return rc; + return bnxt_register_dev(en_dev, &bnxt_re_ulp_ops, rdev->adev); } static void bnxt_re_init_hwrm_hdr(struct input *hdr, u16 opcd) @@ -1593,7 +1589,7 @@ static int bnxt_re_ib_init(struct bnxt_re_dev *rdev) return rc; } -static void bnxt_re_dev_uninit(struct bnxt_re_dev *rdev) +static void bnxt_re_dev_uninit(struct bnxt_re_dev *rdev, u8 op_type) { u8 type; int rc; @@ -1626,8 +1622,10 @@ static void bnxt_re_dev_uninit(struct bnxt_re_dev *rdev) bnxt_re_deinitialize_dbr_pacing(rdev); bnxt_re_destroy_chip_ctx(rdev); - if (test_and_clear_bit(BNXT_RE_FLAG_NETDEV_REGISTERED, &rdev->flags)) - bnxt_unregister_dev(rdev->en_dev); + if (op_type == BNXT_RE_COMPLETE_REMOVE) { + if (test_and_clear_bit(BNXT_RE_FLAG_NETDEV_REGISTERED, &rdev->flags)) + bnxt_unregister_dev(rdev->en_dev); + } } /* worker thread for polling periodic events. Now used for QoS programming*/ @@ -1640,7 +1638,7 @@ static void bnxt_re_worker(struct work_struct *work) schedule_delayed_work(&rdev->worker, msecs_to_jiffies(30000)); } -static int bnxt_re_dev_init(struct bnxt_re_dev *rdev) +static int bnxt_re_dev_init(struct bnxt_re_dev *rdev, u8 op_type) { struct bnxt_re_ring_attr rattr = {}; struct bnxt_qplib_creq_ctx *creq; @@ -1649,12 +1647,14 @@ static int bnxt_re_dev_init(struct bnxt_re_dev *rdev) u8 type; int rc; - /* Registered a new RoCE device instance to netdev */ - rc = bnxt_re_register_netdev(rdev); - if (rc) { - ibdev_err(&rdev->ibdev, - "Failed to register with netedev: %#x\n", rc); - return -EINVAL; + if (op_type == BNXT_RE_COMPLETE_INIT) { + /* Registered a new RoCE device instance to netdev */ + rc = bnxt_re_register_netdev(rdev); + if (rc) { + ibdev_err(&rdev->ibdev, + "Failed to register with netedev: %#x\n", rc); + return -EINVAL; + } } set_bit(BNXT_RE_FLAG_NETDEV_REGISTERED, &rdev->flags); @@ -1807,7 +1807,7 @@ static int bnxt_re_dev_init(struct bnxt_re_dev *rdev) free_rcfw: bnxt_qplib_free_rcfw_channel(&rdev->rcfw); fail: - bnxt_re_dev_uninit(rdev); + bnxt_re_dev_uninit(rdev, BNXT_RE_COMPLETE_REMOVE); return rc; } @@ -1827,7 +1827,7 @@ static void bnxt_re_update_en_info_rdev(struct bnxt_re_dev *rdev, rtnl_unlock(); } -static int bnxt_re_add_device(struct auxiliary_device *adev) +static int bnxt_re_add_device(struct auxiliary_device *adev, u8 op_type) { struct bnxt_aux_priv *aux_priv = container_of(adev, struct bnxt_aux_priv, aux_dev); @@ -1839,8 +1839,6 @@ static int bnxt_re_add_device(struct auxiliary_device *adev) en_info = auxiliary_get_drvdata(adev); en_dev = en_info->en_dev; - /* en_dev should never be NULL as long as adev and aux_dev are valid. */ - en_dev = aux_priv->edev; rdev = bnxt_re_dev_add(aux_priv, en_dev); if (!rdev || !rdev_to_dev(rdev)) { @@ -1850,7 +1848,7 @@ static int bnxt_re_add_device(struct auxiliary_device *adev) bnxt_re_update_en_info_rdev(rdev, en_info, adev); - rc = bnxt_re_dev_init(rdev); + rc = bnxt_re_dev_init(rdev, op_type); if (rc) goto re_dev_dealloc; @@ -1875,7 +1873,7 @@ static int bnxt_re_add_device(struct auxiliary_device *adev) re_dev_uninit: bnxt_re_update_en_info_rdev(NULL, en_info, adev); - bnxt_re_dev_uninit(rdev); + bnxt_re_dev_uninit(rdev, BNXT_RE_COMPLETE_REMOVE); re_dev_dealloc: ib_dealloc_device(&rdev->ibdev); exit: @@ -1958,7 +1956,7 @@ static int bnxt_re_netdev_event(struct notifier_block *notifier, #define BNXT_ADEV_NAME "bnxt_en" -static void bnxt_re_remove_device(struct bnxt_re_dev *rdev, +static void bnxt_re_remove_device(struct bnxt_re_dev *rdev, u8 op_type, struct auxiliary_device *aux_dev) { if (rdev->nb.notifier_call) { @@ -1972,7 +1970,7 @@ static void bnxt_re_remove_device(struct bnxt_re_dev *rdev, } bnxt_re_setup_cc(rdev, false); ib_unregister_device(&rdev->ibdev); - bnxt_re_dev_uninit(rdev); + bnxt_re_dev_uninit(rdev, op_type); ib_dealloc_device(&rdev->ibdev); } @@ -1991,7 +1989,7 @@ static void bnxt_re_remove(struct auxiliary_device *adev) rdev = en_info->rdev; if (rdev) - bnxt_re_remove_device(rdev, adev); + bnxt_re_remove_device(rdev, BNXT_RE_COMPLETE_REMOVE, adev); kfree(en_info); mutex_unlock(&bnxt_re_mutex); } @@ -2017,7 +2015,7 @@ static int bnxt_re_probe(struct auxiliary_device *adev, auxiliary_set_drvdata(adev, en_info); - rc = bnxt_re_add_device(adev); + rc = bnxt_re_add_device(adev, BNXT_RE_COMPLETE_INIT); if (rc) goto err; mutex_unlock(&bnxt_re_mutex); @@ -2033,12 +2031,14 @@ static int bnxt_re_probe(struct auxiliary_device *adev, static int bnxt_re_suspend(struct auxiliary_device *adev, pm_message_t state) { struct bnxt_re_en_dev_info *en_info = auxiliary_get_drvdata(adev); + struct bnxt_en_dev *en_dev; struct bnxt_re_dev *rdev; if (!en_info) return 0; rdev = en_info->rdev; + en_dev = en_info->en_dev; mutex_lock(&bnxt_re_mutex); /* L2 driver may invoke this callback during device error/crash or device * reset. Current RoCE driver doesn't recover the device in case of @@ -2057,13 +2057,20 @@ static int bnxt_re_suspend(struct auxiliary_device *adev, pm_message_t state) set_bit(ERR_DEVICE_DETACHED, &rdev->rcfw.cmdq.flags); bnxt_re_dev_stop(rdev); - bnxt_re_stop_irq(rdev); + bnxt_re_stop_irq(adev); /* Move the device states to detached and avoid sending any more * commands to HW */ set_bit(BNXT_RE_FLAG_ERR_DEVICE_DETACHED, &rdev->flags); set_bit(ERR_DEVICE_DETACHED, &rdev->rcfw.cmdq.flags); wake_up_all(&rdev->rcfw.cmdq.waitq); + + if (rdev->pacing.dbr_pacing) + bnxt_re_set_pacing_dev_state(rdev); + + ibdev_info(&rdev->ibdev, "%s: L2 driver notified to stop en_state 0x%lx", + __func__, en_dev->en_state); + bnxt_re_remove_device(rdev, BNXT_RE_PRE_RECOVERY_REMOVE, adev); mutex_unlock(&bnxt_re_mutex); return 0; @@ -2077,7 +2084,6 @@ static int bnxt_re_resume(struct auxiliary_device *adev) if (!en_info) return 0; - rdev = en_info->rdev; mutex_lock(&bnxt_re_mutex); /* L2 driver may invoke this callback during device recovery, resume. * reset. Current RoCE driver doesn't recover the device in case of @@ -2086,7 +2092,9 @@ static int bnxt_re_resume(struct auxiliary_device *adev) * L2 driver want to modify the MSIx table. */ - ibdev_info(&rdev->ibdev, "Handle device resume call"); + bnxt_re_add_device(adev, BNXT_RE_POST_RECOVERY_INIT); + rdev = en_info->rdev; + ibdev_info(&rdev->ibdev, "Device resume completed"); mutex_unlock(&bnxt_re_mutex); return 0; diff --git a/drivers/infiniband/hw/bnxt_re/qplib_res.h b/drivers/infiniband/hw/bnxt_re/qplib_res.h index 049805a..c2f7103 100644 --- a/drivers/infiniband/hw/bnxt_re/qplib_res.h +++ b/drivers/infiniband/hw/bnxt_re/qplib_res.h @@ -82,6 +82,7 @@ struct bnxt_qplib_db_pacing_data { u32 fifo_room_mask; u32 fifo_room_shift; u32 grc_reg_offset; + u32 dev_err_state; }; #define BNXT_QPLIB_DBR_PF_DB_OFFSET 0x10000