From patchwork Mon Nov 25 08:39:29 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Selvin Xavier X-Patchwork-Id: 11260005 X-Patchwork-Delegate: jgg@ziepe.ca Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id D51E515AC for ; Mon, 25 Nov 2019 08:40:06 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id B6CD12071E for ; Mon, 25 Nov 2019 08:40:06 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=broadcom.com header.i=@broadcom.com header.b="c5CoBM3w" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726694AbfKYIj7 (ORCPT ); Mon, 25 Nov 2019 03:39:59 -0500 Received: from mail-wm1-f68.google.com ([209.85.128.68]:51733 "EHLO mail-wm1-f68.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725793AbfKYIj7 (ORCPT ); Mon, 25 Nov 2019 03:39:59 -0500 Received: by mail-wm1-f68.google.com with SMTP id g206so14158674wme.1 for ; Mon, 25 Nov 2019 00:39:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=nWm85geAm8QkOdDtHQwPIjdz6MlCOkq1CZjRdUm73Ho=; b=c5CoBM3w/yU4083JbhmuE6QMWQ5+kzQ5659gXLpkZ9YtXmIl+uQ9522+YH45E4T8S9 t55BBjIAoIeqAjfETBYkH46zKB8Lktlt5q4QLHO9HSs8cJfk5fIHqWLUthfkbprBYVKN pq5eDWkrXe9MdMGREd2faUbdE9A0xMsVA4dp4= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=nWm85geAm8QkOdDtHQwPIjdz6MlCOkq1CZjRdUm73Ho=; b=c3ycRCBfdb74UrPHmZTeBg7J8qGgn6cemXluc2/GB+RhGHEChKufBxrwbHUnNjTBYS 0EjQ+yNejN0pcXNadWCRFB36BFZogDb/JA64adEgN7smjj6tePLZZoGkhOK3jiD1IQae Lw70nT/L7VRHtPPVhQXR2BQ1veOQHJ8N5GNj0rWPKKozHTZXXveNx9ZDOmJ2+2tTL8sC ipVTxzZdvy8KKlVpWcDoGKWIUBlmpPv0HFbdNFTAlmw7Uk9wxqxIF87WlEyk8TYXeW3J qisH6eqgsBBMTLWSw9HUbMaMMg13rmPPNfeDby3Nj1hudOGu9u7mS5IFbyat7BzgtYoX uO7A== X-Gm-Message-State: APjAAAX+T6GlFDYEiXoCv3oRlADHQ8u8gP+43ysN8vekj4Z6E1qQCZ6x G5Wg34prxLxBFVZ1vamGa1kX9w== X-Google-Smtp-Source: APXvYqz14yhuycBvqquKzYz9x7Ax5iaGoIeazz0dGNRZ9/LZ0eKhZieDnftLbuPa58ALi0/eSMa8Wg== X-Received: by 2002:a1c:f407:: with SMTP id z7mr22812281wma.148.1574671196554; Mon, 25 Nov 2019 00:39:56 -0800 (PST) Received: from dhcp-10-192-206-197.iig.avagotech.net.net ([192.19.234.250]) by smtp.gmail.com with ESMTPSA id k4sm7996995wmk.26.2019.11.25.00.39.53 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 25 Nov 2019 00:39:55 -0800 (PST) From: Selvin Xavier To: dledford@redhat.com, jgg@ziepe.ca Cc: linux-rdma@vger.kernel.org, Selvin Xavier Subject: [PATCH for-next 1/6] RDMA/bnxt_re: Avoid freeing MR resources if dereg fails Date: Mon, 25 Nov 2019 00:39:29 -0800 Message-Id: <1574671174-5064-2-git-send-email-selvin.xavier@broadcom.com> X-Mailer: git-send-email 2.5.5 In-Reply-To: <1574671174-5064-1-git-send-email-selvin.xavier@broadcom.com> References: <1574671174-5064-1-git-send-email-selvin.xavier@broadcom.com> Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org Driver returns error code for MR dereg, but frees the MR structure. When the MR dereg is retried due to previous error, the system crashes as the structure is already freed. [45545.547748] BUG: unable to handle kernel NULL pointer dereference at 00000000000001b8 [45545.557020] PGD 0 P4D 0 [45545.560370] Oops: 0000 [#1] SMP PTI [45545.564778] CPU: 7 PID: 12178 Comm: ib_send_bw Kdump: loaded Not tainted 4.18.0-124.el8.x86_64 #1 [45545.575211] Hardware name: Dell Inc. PowerEdge R430/03XKDV, BIOS 1.1.10 03/10/2015 [45545.584202] RIP: 0010:__dev_printk+0x2a/0x70 [45545.589495] Code: 0f 1f 44 00 00 49 89 d1 48 85 f6 0f 84 f6 2b 00 00 4c 8b 46 70 4d 85 c0 75 04 4c 8b 46 10 48 8b 86 a8 00 00 00 48 85 c0 74 16 <48> 8b 08 0f be 7f 01 48 c7 c2 13 ac ac 83 83 ef 30 e9 10 fe ff ff [45545.611538] RSP: 0018:ffffaf7c04607a60 EFLAGS: 00010006 [45545.617903] RAX: 00000000000001b8 RBX: ffffa0010c91c488 RCX: 0000000000000246 [45545.626416] RDX: ffffaf7c04607a68 RSI: ffffa0010c91caa8 RDI: ffffffff83a788eb [45545.634929] RBP: ffffaf7c04607ac8 R08: 0000000000000000 R09: ffffaf7c04607a68 [45545.643433] R10: 0000000000000000 R11: 0000000000000001 R12: ffffaf7c04607b90 [45545.651924] R13: 000000000000000e R14: 0000000000000000 R15: 00000000ffffa001 [45545.660411] FS: 0000146fa1f1cdc0(0000) GS:ffffa0012fac0000(0000) knlGS:0000000000000000 [45545.669969] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [45545.676910] CR2: 00000000000001b8 CR3: 000000007680a003 CR4: 00000000001606e0 [45545.685405] Call Trace: [45545.688661] dev_err+0x6c/0x90 [45545.692592] ? dev_printk_emit+0x4e/0x70 [45545.697490] bnxt_qplib_rcfw_send_message+0x594/0x660 [bnxt_re] [45545.704619] ? dev_err+0x6c/0x90 [45545.708727] bnxt_qplib_free_mrw+0x80/0xe0 [bnxt_re] [45545.714782] bnxt_re_dereg_mr+0x2e/0xd0 [bnxt_re] [45545.720552] ib_dereg_mr+0x2f/0x50 [ib_core] [45545.725835] destroy_hw_idr_uobject+0x20/0x70 [ib_uverbs] [45545.732375] uverbs_destroy_uobject+0x2e/0x170 [ib_uverbs] [45545.739010] __uverbs_cleanup_ufile+0x6e/0x90 [ib_uverbs] [45545.745544] uverbs_destroy_ufile_hw+0x61/0x130 [ib_uverbs] [45545.752272] ib_uverbs_close+0x1f/0x80 [ib_uverbs] [45545.758126] __fput+0xb7/0x230 [45545.762033] task_work_run+0x8a/0xb0 [45545.766518] do_exit+0x2da/0xb40 ... [45545.841546] RIP: 0033:0x146fa113a387 [45545.845934] Code: Bad RIP value. [45545.849931] RSP: 002b:00007fff945d1478 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff02 [45545.858783] RAX: 0000000000000000 RBX: 000055a248908d70 RCX: 0000000000000000 [45545.867145] RDX: 0000146fa1f2b000 RSI: 0000000000000001 RDI: 000055a248906488 [45545.875503] RBP: 000055a248909630 R08: 0000000000010000 R09: 0000000000000000 [45545.883849] R10: 0000000000000000 R11: 0000000000000000 R12: 000055a248906488 [45545.892180] R13: 0000000000000001 R14: 0000000000000000 R15: 000055a2489095f0 Do not free the MR structures, when driver returns error to the stack. Fixes: 872f3578241d ("RDMA/bnxt_re: Add support for MRs with Huge pages") Signed-off-by: Selvin Xavier --- drivers/infiniband/hw/bnxt_re/ib_verbs.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/drivers/infiniband/hw/bnxt_re/ib_verbs.c b/drivers/infiniband/hw/bnxt_re/ib_verbs.c index 9b6ca15..ad5112a 100644 --- a/drivers/infiniband/hw/bnxt_re/ib_verbs.c +++ b/drivers/infiniband/hw/bnxt_re/ib_verbs.c @@ -3305,8 +3305,10 @@ int bnxt_re_dereg_mr(struct ib_mr *ib_mr, struct ib_udata *udata) int rc; rc = bnxt_qplib_free_mrw(&rdev->qplib_res, &mr->qplib_mr); - if (rc) + if (rc) { dev_err(rdev_to_dev(rdev), "Dereg MR failed: %#x\n", rc); + return rc; + } if (mr->pages) { rc = bnxt_qplib_free_fast_reg_page_list(&rdev->qplib_res, From patchwork Mon Nov 25 08:39:30 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Selvin Xavier X-Patchwork-Id: 11260007 X-Patchwork-Delegate: jgg@ziepe.ca Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 085B71390 for ; Mon, 25 Nov 2019 08:40:07 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id DE5472071E for ; Mon, 25 Nov 2019 08:40:06 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=broadcom.com header.i=@broadcom.com header.b="XGjAcvfc" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726696AbfKYIkC (ORCPT ); Mon, 25 Nov 2019 03:40:02 -0500 Received: from mail-wr1-f68.google.com ([209.85.221.68]:43266 "EHLO mail-wr1-f68.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725793AbfKYIkB (ORCPT ); Mon, 25 Nov 2019 03:40:01 -0500 Received: by mail-wr1-f68.google.com with SMTP id n1so16819877wra.10 for ; Mon, 25 Nov 2019 00:40:00 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=z47Ojrrp4Sr0aVG121k4/W8LgZr4mIcsGLIopDAFISU=; b=XGjAcvfcummEjUTitFMwW42z+zT+5KrFfIulzMFwDFWv6G9EX0pNTb2DBEgzYmM1ek ZhWAsfbOn7p4kOCU2GEJCULXNK9xg3oG1HpPKHg/YQnnPTbygdZ4NffjVd0IVxN0Z0SH YGD7Rd4YGgvYFI2HM/NCEDKoja7OycU19l/Vg= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=z47Ojrrp4Sr0aVG121k4/W8LgZr4mIcsGLIopDAFISU=; b=BJYeT9JaJM6psp+ULsvR59hVNpdaFDyFDxq67aAS6oPxLRHE2vDK2+SvWHzqAWYAkW mKPpz/lVjwT3IwBJRJIUkMVkyUL/l5ZjXgX/FZkXUC9UIfs1ZM5UMrgS7Va475dVtTY7 zVHIRrZSlJVcbus1AtSq+RCu9iQ5Dy7uKfuV/gOonFZi0x7gyf6kZUS0rnaEATELUmB6 zwDUQpqJZ9FniDOwBUK4MugUKGNW8BNGXOq/wWz+STh8K9JlLxtEhHzFv/c063TNHlGH 0ssMHQmTqBQhs6U7SzXFUqNMIc0kxfl8FoougsGWeLzhJxcnv8a5MFY46KUw2jjbf4Uo FtQA== X-Gm-Message-State: APjAAAVB1a2adlDWyk+ZbuCT+6R9ge5VTbcLocLl1dROQ33n1MWbK9hK YHI54/qL1iNl00xf7WzjF/CJxA== X-Google-Smtp-Source: APXvYqx+vmTJgYgR/0ofEZH+gYv00GghqILCtp7Y9tXo5JRamYzXrJK9mYN9rXHn8lN6AGhxZZ3YFQ== X-Received: by 2002:adf:f489:: with SMTP id l9mr29004000wro.337.1574671199526; Mon, 25 Nov 2019 00:39:59 -0800 (PST) Received: from dhcp-10-192-206-197.iig.avagotech.net.net ([192.19.234.250]) by smtp.gmail.com with ESMTPSA id k4sm7996995wmk.26.2019.11.25.00.39.56 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 25 Nov 2019 00:39:58 -0800 (PST) From: Selvin Xavier To: dledford@redhat.com, jgg@ziepe.ca Cc: linux-rdma@vger.kernel.org, Selvin Xavier Subject: [PATCH for-next 2/6] RDMA/bnxt_re: Fix Send Work Entry state check while polling completions Date: Mon, 25 Nov 2019 00:39:30 -0800 Message-Id: <1574671174-5064-3-git-send-email-selvin.xavier@broadcom.com> X-Mailer: git-send-email 2.5.5 In-Reply-To: <1574671174-5064-1-git-send-email-selvin.xavier@broadcom.com> References: <1574671174-5064-1-git-send-email-selvin.xavier@broadcom.com> Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org Some adapters need fence Work Entry to handle retransmission. Currently the driver checks for this condition, only if the Send queue entry is signalled. Implement the condition check, irrespective of the signalled state of the Work queue entries Fixes: 9152e0b722b2 ("RDMA/bnxt_re: HW workarounds for handling specific conditions") Signed-off-by: Selvin Xavier --- drivers/infiniband/hw/bnxt_re/qplib_fp.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/drivers/infiniband/hw/bnxt_re/qplib_fp.c b/drivers/infiniband/hw/bnxt_re/qplib_fp.c index 958c1ff..4d07d22 100644 --- a/drivers/infiniband/hw/bnxt_re/qplib_fp.c +++ b/drivers/infiniband/hw/bnxt_re/qplib_fp.c @@ -2283,13 +2283,13 @@ static int bnxt_qplib_cq_process_req(struct bnxt_qplib_cq *cq, /* Add qp to flush list of the CQ */ bnxt_qplib_add_flush_qp(qp); } else { + /* Before we complete, do WA 9060 */ + if (do_wa9060(qp, cq, cq_cons, sw_sq_cons, + cqe_sq_cons)) { + *lib_qp = qp; + goto out; + } if (swq->flags & SQ_SEND_FLAGS_SIGNAL_COMP) { - /* Before we complete, do WA 9060 */ - if (do_wa9060(qp, cq, cq_cons, sw_sq_cons, - cqe_sq_cons)) { - *lib_qp = qp; - goto out; - } cqe->status = CQ_REQ_STATUS_OK; cqe++; (*budget)--; From patchwork Mon Nov 25 08:39:31 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Selvin Xavier X-Patchwork-Id: 11260009 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 2FC8117F0 for ; Mon, 25 Nov 2019 08:40:07 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 1161F2071E for ; Mon, 25 Nov 2019 08:40:07 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=broadcom.com header.i=@broadcom.com header.b="Q7D2ZAlR" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726873AbfKYIkF (ORCPT ); Mon, 25 Nov 2019 03:40:05 -0500 Received: from mail-wr1-f68.google.com ([209.85.221.68]:37767 "EHLO mail-wr1-f68.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725793AbfKYIkF (ORCPT ); Mon, 25 Nov 2019 03:40:05 -0500 Received: by mail-wr1-f68.google.com with SMTP id t1so16793047wrv.4 for ; Mon, 25 Nov 2019 00:40:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=3kSw11JvHCo4FkPc2oNgF2bt6NULAqHJCzqBOnn9g+c=; b=Q7D2ZAlR7VrNTsEqXfnTRnZhIQ/gKFCTP+QwgrzfW9J0dYnOoeUb5XP6KubcfHMP7/ T6YKzPGvX9mMybkg99UFcz04DrEgV0DjI3Gf5X9PwWMnuy8gOJUnsXq3PFCw2cH6ax5F ec+cqWYsUo/zc3TzKL1jn2teHshTJBXHvhjsE= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=3kSw11JvHCo4FkPc2oNgF2bt6NULAqHJCzqBOnn9g+c=; b=AcHoZzkVGXMhALO0jbONH5sckKrQ6kh0y9c7Ho2OpId0oBD2jNaMZ8HIMG3NWnCVsJ jWjdkr/G011yXlA29rPwlCKOvYSolJd938qXR0KaziW77qrMnQbQEh3seAal9pEsSHmy B/T+X5+MSyA5piBhr9DFZC1bImnzfCpres5qyFHD8pdIcM6jLqJAFK8YcxUcb70lkbrX X2ENJNTVg9dC3SAjVZVXIIQDOjqvxbwN0gBXr4e5mjo4FzFGVg6C3igQ5ke+qG37d7pd siqLuP6IQVcI1vO20J97q8e/x3wzU3IarZwMheX+OqdzjiAnB+2bNICqk4dhMxZDWAXo 1+Dw== X-Gm-Message-State: APjAAAX/FhT8p2RrK+rpclJT9OO702R1gnMXz2SHad087wrt8J8Zqxxs kJ7BoqgAcriQNHqVZ2Hz7SJlS7YmtAs= X-Google-Smtp-Source: APXvYqxkXknOQd4wDVuaJFxAegnS7jWzCaiiWIQhdHdq2bCNnAmcrdVIeN25/SrbdHSMsve8mhbeDw== X-Received: by 2002:adf:f108:: with SMTP id r8mr18511696wro.390.1574671202471; Mon, 25 Nov 2019 00:40:02 -0800 (PST) Received: from dhcp-10-192-206-197.iig.avagotech.net.net ([192.19.234.250]) by smtp.gmail.com with ESMTPSA id k4sm7996995wmk.26.2019.11.25.00.39.59 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 25 Nov 2019 00:40:01 -0800 (PST) From: Selvin Xavier To: dledford@redhat.com, jgg@ziepe.ca Cc: linux-rdma@vger.kernel.org, Selvin Xavier Subject: [PATCH for-next 3/6] RDMA/bnxt_re: Add more flags in device init and uninit path Date: Mon, 25 Nov 2019 00:39:31 -0800 Message-Id: <1574671174-5064-4-git-send-email-selvin.xavier@broadcom.com> X-Mailer: git-send-email 2.5.5 In-Reply-To: <1574671174-5064-1-git-send-email-selvin.xavier@broadcom.com> References: <1574671174-5064-1-git-send-email-selvin.xavier@broadcom.com> Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org Add more flags for better granularity in in device init/uninit path Signed-off-by: Selvin Xavier --- drivers/infiniband/hw/bnxt_re/bnxt_re.h | 16 ++++++++---- drivers/infiniband/hw/bnxt_re/main.c | 45 ++++++++++++++++++--------------- 2 files changed, 35 insertions(+), 26 deletions(-) diff --git a/drivers/infiniband/hw/bnxt_re/bnxt_re.h b/drivers/infiniband/hw/bnxt_re/bnxt_re.h index 725b235..3be495d 100644 --- a/drivers/infiniband/hw/bnxt_re/bnxt_re.h +++ b/drivers/infiniband/hw/bnxt_re/bnxt_re.h @@ -118,11 +118,17 @@ struct bnxt_re_dev { #define BNXT_RE_FLAG_IBDEV_REGISTERED 1 #define BNXT_RE_FLAG_GOT_MSIX 2 #define BNXT_RE_FLAG_HAVE_L2_REF 3 -#define BNXT_RE_FLAG_RCFW_CHANNEL_EN 4 -#define BNXT_RE_FLAG_QOS_WORK_REG 5 -#define BNXT_RE_FLAG_RESOURCES_ALLOCATED 7 -#define BNXT_RE_FLAG_RESOURCES_INITIALIZED 8 -#define BNXT_RE_FLAG_ISSUE_ROCE_STATS 29 +#define BNXT_RE_FLAG_ALLOC_RCFW 4 +#define BNXT_RE_FLAG_NET_RING_ALLOC 5 +#define BNXT_RE_FLAG_RCFW_CHANNEL_EN 6 +#define BNXT_RE_FLAG_ALLOC_CTX 7 +#define BNXT_RE_FLAG_STATS_CTX_ALLOC 8 +#define BNXT_RE_FLAG_STATS_CTX2_ALLOC 9 +#define BNXT_RE_FLAG_RCFW_CHANNEL_INIT 10 +#define BNXT_RE_FLAG_QOS_WORK_REG 11 +#define BNXT_RE_FLAG_RESOURCES_ALLOCATED 12 +#define BNXT_RE_FLAG_RESOURCES_INITIALIZED 13 +#define BNXT_RE_FLAG_ISSUE_ROCE_STATS 29 struct net_device *netdev; unsigned int version, major, minor; struct bnxt_qplib_chip_ctx chip_ctx; diff --git a/drivers/infiniband/hw/bnxt_re/main.c b/drivers/infiniband/hw/bnxt_re/main.c index e7e8a0f..fbe3192 100644 --- a/drivers/infiniband/hw/bnxt_re/main.c +++ b/drivers/infiniband/hw/bnxt_re/main.c @@ -1315,18 +1315,23 @@ static void bnxt_re_ib_unreg(struct bnxt_re_dev *rdev) if (test_and_clear_bit(BNXT_RE_FLAG_RESOURCES_ALLOCATED, &rdev->flags)) bnxt_re_free_res(rdev); - if (test_and_clear_bit(BNXT_RE_FLAG_RCFW_CHANNEL_EN, &rdev->flags)) { + if (test_and_clear_bit(BNXT_RE_FLAG_RCFW_CHANNEL_INIT, &rdev->flags)) { rc = bnxt_qplib_deinit_rcfw(&rdev->rcfw); if (rc) dev_warn(rdev_to_dev(rdev), "Failed to deinitialize RCFW: %#x", rc); + } + if (test_and_clear_bit(BNXT_RE_FLAG_STATS_CTX_ALLOC, &rdev->flags)) bnxt_re_net_stats_ctx_free(rdev, rdev->qplib_ctx.stats.fw_id); + if (test_and_clear_bit(BNXT_RE_FLAG_ALLOC_CTX, &rdev->flags)) bnxt_qplib_free_ctx(rdev->en_dev->pdev, &rdev->qplib_ctx); + if (test_and_clear_bit(BNXT_RE_FLAG_RCFW_CHANNEL_EN, &rdev->flags)) bnxt_qplib_disable_rcfw_channel(&rdev->rcfw); - type = bnxt_qplib_get_ring_type(&rdev->chip_ctx); + type = bnxt_qplib_get_ring_type(&rdev->chip_ctx); + if (test_and_clear_bit(BNXT_RE_FLAG_NET_RING_ALLOC, &rdev->flags)) bnxt_re_net_ring_free(rdev, rdev->rcfw.creq_ring_id, type); + if (test_and_clear_bit(BNXT_RE_FLAG_ALLOC_RCFW, &rdev->flags)) bnxt_qplib_free_rcfw_channel(&rdev->rcfw); - } if (test_and_clear_bit(BNXT_RE_FLAG_GOT_MSIX, &rdev->flags)) { rc = bnxt_re_free_msix(rdev); if (rc) @@ -1404,6 +1409,9 @@ static int bnxt_re_ib_reg(struct bnxt_re_dev *rdev) pr_err("Failed to allocate RCFW Channel: %#x\n", rc); goto fail; } + + set_bit(BNXT_RE_FLAG_ALLOC_RCFW, &rdev->flags); + type = bnxt_qplib_get_ring_type(&rdev->chip_ctx); pg_map = rdev->rcfw.creq.pbl[PBL_LVL_0].pg_map_arr; pages = rdev->rcfw.creq.pbl[rdev->rcfw.creq.level].pg_count; @@ -1413,8 +1421,10 @@ static int bnxt_re_ib_reg(struct bnxt_re_dev *rdev) ridx, &rdev->rcfw.creq_ring_id); if (rc) { pr_err("Failed to allocate CREQ: %#x\n", rc); - goto free_rcfw; + goto fail; } + set_bit(BNXT_RE_FLAG_NET_RING_ALLOC, &rdev->flags); + db_offt = bnxt_re_get_nqdb_offset(rdev, BNXT_RE_AEQ_IDX); vid = rdev->msix_entries[BNXT_RE_AEQ_IDX].vector; rc = bnxt_qplib_enable_rcfw_channel(rdev->en_dev->pdev, &rdev->rcfw, @@ -1422,13 +1432,14 @@ static int bnxt_re_ib_reg(struct bnxt_re_dev *rdev) &bnxt_re_aeq_handler); if (rc) { pr_err("Failed to enable RCFW channel: %#x\n", rc); - goto free_ring; + goto fail; } + set_bit(BNXT_RE_FLAG_RCFW_CHANNEL_EN, &rdev->flags); rc = bnxt_qplib_get_dev_attr(&rdev->rcfw, &rdev->dev_attr, rdev->is_virtfn); if (rc) - goto disable_rcfw; + goto fail; bnxt_re_set_resource_limits(rdev); @@ -1436,23 +1447,26 @@ static int bnxt_re_ib_reg(struct bnxt_re_dev *rdev) bnxt_qplib_is_chip_gen_p5(&rdev->chip_ctx)); if (rc) { pr_err("Failed to allocate QPLIB context: %#x\n", rc); - goto disable_rcfw; + goto fail; } + set_bit(BNXT_RE_FLAG_ALLOC_CTX, &rdev->flags); + rc = bnxt_re_net_stats_ctx_alloc(rdev, rdev->qplib_ctx.stats.dma_map, &rdev->qplib_ctx.stats.fw_id); if (rc) { pr_err("Failed to allocate stats context: %#x\n", rc); - goto free_ctx; + goto fail; } + set_bit(BNXT_RE_FLAG_STATS_CTX_ALLOC, &rdev->flags); rc = bnxt_qplib_init_rcfw(&rdev->rcfw, &rdev->qplib_ctx, rdev->is_virtfn); if (rc) { pr_err("Failed to initialize RCFW: %#x\n", rc); - goto free_sctx; + goto fail; } - set_bit(BNXT_RE_FLAG_RCFW_CHANNEL_EN, &rdev->flags); + set_bit(BNXT_RE_FLAG_RCFW_CHANNEL_INIT, &rdev->flags); /* Resources based on the 'new' device caps */ rc = bnxt_re_alloc_res(rdev); @@ -1496,17 +1510,6 @@ static int bnxt_re_ib_reg(struct bnxt_re_dev *rdev) bnxt_re_dispatch_event(&rdev->ibdev, NULL, 1, IB_EVENT_PORT_ACTIVE); return 0; -free_sctx: - bnxt_re_net_stats_ctx_free(rdev, rdev->qplib_ctx.stats.fw_id); -free_ctx: - bnxt_qplib_free_ctx(rdev->en_dev->pdev, &rdev->qplib_ctx); -disable_rcfw: - bnxt_qplib_disable_rcfw_channel(&rdev->rcfw); -free_ring: - type = bnxt_qplib_get_ring_type(&rdev->chip_ctx); - bnxt_re_net_ring_free(rdev, rdev->rcfw.creq_ring_id, type); -free_rcfw: - bnxt_qplib_free_rcfw_channel(&rdev->rcfw); fail: if (!locked) rtnl_lock(); From patchwork Mon Nov 25 08:39:32 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Selvin Xavier X-Patchwork-Id: 11260011 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B36A71390 for ; Mon, 25 Nov 2019 08:40:09 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 8BFDC2071E for ; Mon, 25 Nov 2019 08:40:09 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=broadcom.com header.i=@broadcom.com header.b="O1a37BgZ" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726571AbfKYIkI (ORCPT ); Mon, 25 Nov 2019 03:40:08 -0500 Received: from mail-wm1-f65.google.com ([209.85.128.65]:51749 "EHLO mail-wm1-f65.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725793AbfKYIkI (ORCPT ); Mon, 25 Nov 2019 03:40:08 -0500 Received: by mail-wm1-f65.google.com with SMTP id g206so14159107wme.1 for ; Mon, 25 Nov 2019 00:40:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=dnCWu2YByTvrq6H4KNxnq0uugqxVE2UA7neUdMCLUOA=; b=O1a37BgZAOYBbIifxdbamQKfzZlPhlBYqtstU4FfGTY4c9vIXG3wTqjBS0UnIhblGL PZEw1WD6LkIOtAXdTnML7iF6tpJivq4i4FT4WHcnyvryvwzusbjrdXfOVFIX9QuqZ/VE TUjB5OOPkC99Ek3gUjf9T+E9/AgKVHgCqRiAI= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=dnCWu2YByTvrq6H4KNxnq0uugqxVE2UA7neUdMCLUOA=; b=VM4NbSlPQHx8+R2mo2kRhFHswNs9P1jh4kZDqPfIvkPWMbdhSfJb65RKq9ZwocyQvQ qMwYDgadyZC33Zbm9Tw4HmX9DvGZxzekhXf2JQ0dfkJfQ+swZ98D6K6/y5KJIcR2fqlC moZufdUL8sHR5s+JFKo66d0bgpZKomDurxzH1uWXpEOfBpNbvzCd1cu5b7ZbLpxarp7N h+1nBT0XM4WHoOb33yYJ+pVQP7Ue2F89eC4ZdQk7xf4rv1482A0u4Kuz406TNtOsVC5J ts79V5YfWT/RgVXoC35NFiPe9Y60COa3duQX3Uy3fdLaS066C4v/cHtc59sKGzcQLsm+ xLqA== X-Gm-Message-State: APjAAAWX/VFfnCpiGhkOsp9a7BWaegOsGV8qrbj1IdVjaKFcaFOfn8Jv ZejVhZwvoM2VCyK/rNNM2IfLHg== X-Google-Smtp-Source: APXvYqyIfQPdDelULJk/jLvbW8FvDgjoa71Yz38tqmeSMKqbn3ldET0YrxP8xZ9VXxkgpfGYm89n0g== X-Received: by 2002:a05:600c:2945:: with SMTP id n5mr29259513wmd.80.1574671205375; Mon, 25 Nov 2019 00:40:05 -0800 (PST) Received: from dhcp-10-192-206-197.iig.avagotech.net.net ([192.19.234.250]) by smtp.gmail.com with ESMTPSA id k4sm7996995wmk.26.2019.11.25.00.40.02 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 25 Nov 2019 00:40:04 -0800 (PST) From: Selvin Xavier To: dledford@redhat.com, jgg@ziepe.ca Cc: linux-rdma@vger.kernel.org, Selvin Xavier Subject: [PATCH for-next 4/6] RDMA/bnxt_re: Refactor device add/remove functionalities Date: Mon, 25 Nov 2019 00:39:32 -0800 Message-Id: <1574671174-5064-5-git-send-email-selvin.xavier@broadcom.com> X-Mailer: git-send-email 2.5.5 In-Reply-To: <1574671174-5064-1-git-send-email-selvin.xavier@broadcom.com> References: <1574671174-5064-1-git-send-email-selvin.xavier@broadcom.com> Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org - bnxt_re_ib_reg() handles two main functionalities - initializing the device and registering with the IB stack. Split it into 2 functions i.e. bnxt_re_dev_init() and bnxt_re_ib_init() to account for the same thereby improve modularity. Do the same for bnxt_re_ib_unreg()i.e. split into two functions - bnxt_re_dev_uninit() and bnxt_re_ib_uninit(). - Simplify the code by combining the different steps to add and remove the device into two functions. Signed-off-by: Selvin Xavier --- drivers/infiniband/hw/bnxt_re/main.c | 133 ++++++++++++++++++++--------------- 1 file changed, 78 insertions(+), 55 deletions(-) diff --git a/drivers/infiniband/hw/bnxt_re/main.c b/drivers/infiniband/hw/bnxt_re/main.c index fbe3192..0cf38a4 100644 --- a/drivers/infiniband/hw/bnxt_re/main.c +++ b/drivers/infiniband/hw/bnxt_re/main.c @@ -78,7 +78,8 @@ static struct list_head bnxt_re_dev_list = LIST_HEAD_INIT(bnxt_re_dev_list); /* Mutex to protect the list of bnxt_re devices added */ static DEFINE_MUTEX(bnxt_re_dev_lock); static struct workqueue_struct *bnxt_re_wq; -static void bnxt_re_ib_unreg(struct bnxt_re_dev *rdev); +static void bnxt_re_remove_device(struct bnxt_re_dev *rdev); +static void bnxt_re_ib_uninit(struct bnxt_re_dev *rdev); static void bnxt_re_destroy_chip_ctx(struct bnxt_re_dev *rdev) { @@ -222,7 +223,9 @@ static void bnxt_re_shutdown(void *p) if (!rdev) return; - bnxt_re_ib_unreg(rdev); + bnxt_re_ib_uninit(rdev); + /* rtnl_lock held by L2 before coming here */ + bnxt_re_remove_device(rdev); } static void bnxt_re_stop_irq(void *handle) @@ -1297,7 +1300,37 @@ static void bnxt_re_query_hwrm_intf_version(struct bnxt_re_dev *rdev) le16_to_cpu(resp.hwrm_intf_patch); } -static void bnxt_re_ib_unreg(struct bnxt_re_dev *rdev) +static void bnxt_re_ib_uninit(struct bnxt_re_dev *rdev) +{ + /* Cleanup ib dev */ + if (test_bit(BNXT_RE_FLAG_IBDEV_REGISTERED, &rdev->flags)) { + ib_unregister_device(&rdev->ibdev); + clear_bit(BNXT_RE_FLAG_IBDEV_REGISTERED, &rdev->flags); + } +} + +int bnxt_re_ib_init(struct bnxt_re_dev *rdev) +{ + int rc = 0; + + /* Register ib dev */ + rc = bnxt_re_register_ib(rdev); + if (rc) { + pr_err("Failed to register with IB: %#x\n", rc); + return rc; + } + set_bit(BNXT_RE_FLAG_IBDEV_REGISTERED, &rdev->flags); + dev_info(rdev_to_dev(rdev), "Device registered successfully"); + ib_get_eth_speed(&rdev->ibdev, 1, &rdev->active_speed, + &rdev->active_width); + set_bit(BNXT_RE_FLAG_ISSUE_ROCE_STATS, &rdev->flags); + bnxt_re_dispatch_event(&rdev->ibdev, NULL, 1, IB_EVENT_PORT_ACTIVE); + bnxt_re_dispatch_event(&rdev->ibdev, NULL, 1, IB_EVENT_GID_CHANGE); + + return rc; +} + +static void bnxt_re_dev_uninit(struct bnxt_re_dev *rdev) { u8 type; int rc; @@ -1358,19 +1391,14 @@ static void bnxt_re_worker(struct work_struct *work) schedule_delayed_work(&rdev->worker, msecs_to_jiffies(30000)); } -static int bnxt_re_ib_reg(struct bnxt_re_dev *rdev) +static int bnxt_re_dev_init(struct bnxt_re_dev *rdev) { dma_addr_t *pg_map; u32 db_offt, ridx; int pages, vid; - bool locked; u8 type; int rc; - /* Acquire rtnl lock through out this function */ - rtnl_lock(); - locked = true; - /* Registered a new RoCE device instance to netdev */ rc = bnxt_re_register_netdev(rdev); if (rc) { @@ -1493,29 +1521,10 @@ static int bnxt_re_ib_reg(struct bnxt_re_dev *rdev) schedule_delayed_work(&rdev->worker, msecs_to_jiffies(30000)); } - rtnl_unlock(); - locked = false; - - /* Register ib dev */ - rc = bnxt_re_register_ib(rdev); - if (rc) { - pr_err("Failed to register with IB: %#x\n", rc); - goto fail; - } - set_bit(BNXT_RE_FLAG_IBDEV_REGISTERED, &rdev->flags); - dev_info(rdev_to_dev(rdev), "Device registered successfully"); - ib_get_eth_speed(&rdev->ibdev, 1, &rdev->active_speed, - &rdev->active_width); - set_bit(BNXT_RE_FLAG_ISSUE_ROCE_STATS, &rdev->flags); - bnxt_re_dispatch_event(&rdev->ibdev, NULL, 1, IB_EVENT_PORT_ACTIVE); - return 0; fail: - if (!locked) - rtnl_lock(); - bnxt_re_ib_unreg(rdev); - rtnl_unlock(); + bnxt_re_dev_uninit(rdev); return rc; } @@ -1555,9 +1564,35 @@ static int bnxt_re_dev_reg(struct bnxt_re_dev **rdev, struct net_device *netdev) return rc; } -static void bnxt_re_remove_one(struct bnxt_re_dev *rdev) +static void bnxt_re_remove_device(struct bnxt_re_dev *rdev) { + bnxt_re_dev_uninit(rdev); pci_dev_put(rdev->en_dev->pdev); + bnxt_re_dev_unreg(rdev); +} + +static int bnxt_re_add_device(struct bnxt_re_dev **rdev, + struct net_device *netdev) +{ + int rc; + + rc = bnxt_re_dev_reg(rdev, netdev); + if (rc == -ENODEV) + return rc; + if (rc) { + pr_err("Failed to register with the device %s: %#x\n", + netdev->name, rc); + return rc; + } + + pci_dev_get((*rdev)->en_dev->pdev); + rc = bnxt_re_dev_init(*rdev); + if (rc) { + pci_dev_put((*rdev)->en_dev->pdev); + bnxt_re_dev_unreg(*rdev); + } + + return rc; } /* Handle all deferred netevents tasks */ @@ -1572,16 +1607,17 @@ static void bnxt_re_task(struct work_struct *work) if (re_work->event != NETDEV_REGISTER && !test_bit(BNXT_RE_FLAG_IBDEV_REGISTERED, &rdev->flags)) - return; + goto done; switch (re_work->event) { case NETDEV_REGISTER: - rc = bnxt_re_ib_reg(rdev); + rc = bnxt_re_ib_init(rdev); if (rc) { dev_err(rdev_to_dev(rdev), "Failed to register with IB: %#x", rc); - bnxt_re_remove_one(rdev); - bnxt_re_dev_unreg(rdev); + rtnl_lock(); + bnxt_re_remove_device(rdev); + rtnl_unlock(); goto exit; } break; @@ -1604,17 +1640,13 @@ static void bnxt_re_task(struct work_struct *work) default: break; } +done: smp_mb__before_atomic(); atomic_dec(&rdev->sched_count); exit: kfree(re_work); } -static void bnxt_re_init_one(struct bnxt_re_dev *rdev) -{ - pci_dev_get(rdev->en_dev->pdev); -} - /* * "Notifier chain callback can be invoked for the same chain from * different CPUs at the same time". @@ -1652,16 +1684,9 @@ static int bnxt_re_netdev_event(struct notifier_block *notifier, case NETDEV_REGISTER: if (rdev) break; - rc = bnxt_re_dev_reg(&rdev, real_dev); - if (rc == -ENODEV) - break; - if (rc) { - pr_err("Failed to register with the device %s: %#x\n", - real_dev->name, rc); - break; - } - bnxt_re_init_one(rdev); - sch_work = true; + rc = bnxt_re_add_device(&rdev, real_dev); + if (!rc) + sch_work = true; break; case NETDEV_UNREGISTER: @@ -1670,9 +1695,8 @@ static int bnxt_re_netdev_event(struct notifier_block *notifier, */ if (atomic_read(&rdev->sched_count) > 0) goto exit; - bnxt_re_ib_unreg(rdev); - bnxt_re_remove_one(rdev); - bnxt_re_dev_unreg(rdev); + bnxt_re_ib_uninit(rdev); + bnxt_re_remove_device(rdev); break; default: @@ -1749,12 +1773,11 @@ static void __exit bnxt_re_mod_exit(void) */ flush_workqueue(bnxt_re_wq); bnxt_re_dev_stop(rdev); + bnxt_re_ib_uninit(rdev); /* Acquire the rtnl_lock as the L2 resources are freed here */ rtnl_lock(); - bnxt_re_ib_unreg(rdev); + bnxt_re_remove_device(rdev); rtnl_unlock(); - bnxt_re_remove_one(rdev); - bnxt_re_dev_unreg(rdev); } unregister_netdevice_notifier(&bnxt_re_netdev_notifier); if (bnxt_re_wq) From patchwork Mon Nov 25 08:39:33 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Selvin Xavier X-Patchwork-Id: 11260013 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 5CAA9138C for ; Mon, 25 Nov 2019 08:40:12 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 357FB20823 for ; Mon, 25 Nov 2019 08:40:12 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=broadcom.com header.i=@broadcom.com header.b="EnHGZVc+" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726947AbfKYIkL (ORCPT ); Mon, 25 Nov 2019 03:40:11 -0500 Received: from mail-wm1-f66.google.com ([209.85.128.66]:38571 "EHLO mail-wm1-f66.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725793AbfKYIkL (ORCPT ); Mon, 25 Nov 2019 03:40:11 -0500 Received: by mail-wm1-f66.google.com with SMTP id z19so14861482wmk.3 for ; Mon, 25 Nov 2019 00:40:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=rKMFStMwnjz7q1hlJIu8NYh75xwdbICmdZIplHFUWU8=; b=EnHGZVc+kkOtpdWVA+tMBYe/kJBTqzdJ2iaFX6iiPSLLlHyWruNDOZV6D/VD6Wte1S Ias8bwBWv4zVbaCuxnkQ5I5IopALRb+xf89YhRUay4Bh+06T7tevDUBXEQWu9azEPbZY GBNPXo9lOX0hxqSOJd24ZRl0YEoSDIK/BE4Lk= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=rKMFStMwnjz7q1hlJIu8NYh75xwdbICmdZIplHFUWU8=; b=SOzOkuR6FFRwGcYjJXaO0iyQet8Wy5XQe1IuTJhZuB3Y5OltmMSHhcETSAiNjdYo8I 1pKR/6lMmeBMqCk9RzUWTUsq1ogKADiGuLoAePZTUBwz6cZNUzy5Y0+aPdY5WAQCeeju BVuuDuNvajCItOJ7pichkn/8mEgkatnTlxXoAUGozETy2KukOfgi+h50ykXrZCTCVUti acxi4qv6QDqxiDUsAzgQz9GcrkVRUHz12sN/Zk6W655WeK36TWMRJYz8VXi1x5knf76P 6VIbQPQGHnfFFF8rppPm2Qrf8VLsy/+6ICI0bAShjuzuACslILoRuGhzIgRkBNhWbYjL jU+A== X-Gm-Message-State: APjAAAVdQ9Fkv2H9YdTtB3FSHJ4iDBEnn7IeaMRjaQg8pAcVUYYtB0De WuLTCvkojD3arX70aH9Mfl2eFwZKndM= X-Google-Smtp-Source: APXvYqxZAHWtp09gsjg/sJ0JCeEffg7fD3Y1a1ivQgvV8IeX/qooGEGyA72Pdmabz29gn6o42donTA== X-Received: by 2002:a05:600c:2301:: with SMTP id 1mr17435576wmo.143.1574671208307; Mon, 25 Nov 2019 00:40:08 -0800 (PST) Received: from dhcp-10-192-206-197.iig.avagotech.net.net ([192.19.234.250]) by smtp.gmail.com with ESMTPSA id k4sm7996995wmk.26.2019.11.25.00.40.05 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 25 Nov 2019 00:40:07 -0800 (PST) From: Selvin Xavier To: dledford@redhat.com, jgg@ziepe.ca Cc: linux-rdma@vger.kernel.org, Selvin Xavier Subject: [PATCH for-next 5/6] RDMA/bnxt_re: Use driver_unregister and unregistration API Date: Mon, 25 Nov 2019 00:39:33 -0800 Message-Id: <1574671174-5064-6-git-send-email-selvin.xavier@broadcom.com> X-Mailer: git-send-email 2.5.5 In-Reply-To: <1574671174-5064-1-git-send-email-selvin.xavier@broadcom.com> References: <1574671174-5064-1-git-send-email-selvin.xavier@broadcom.com> Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org Using the new unregister APIs provided by the core. Provide the dealloc_driver hook for the core to callback at the time of device un-registration. Also, simplify bnxt_re_from_netdev by retrieving the netdev information from the core API- ib_device_get_by_netdev. bnxt_re VF resources are created by the corresponding PF driver. During driver unload, if PFs gets destroyed first, driver need not send down any command to FW. So adding a state (res_deinit) in the per device structures to detect driver removal and avoid failures VF destroy due to driver removal. Signed-off-by: Selvin Xavier --- drivers/infiniband/hw/bnxt_re/main.c | 100 ++++++++++++----------------- drivers/infiniband/hw/bnxt_re/qplib_rcfw.c | 7 ++ drivers/infiniband/hw/bnxt_re/qplib_rcfw.h | 1 + 3 files changed, 48 insertions(+), 60 deletions(-) diff --git a/drivers/infiniband/hw/bnxt_re/main.c b/drivers/infiniband/hw/bnxt_re/main.c index 0cf38a4..9ea4ce7 100644 --- a/drivers/infiniband/hw/bnxt_re/main.c +++ b/drivers/infiniband/hw/bnxt_re/main.c @@ -79,7 +79,7 @@ static struct list_head bnxt_re_dev_list = LIST_HEAD_INIT(bnxt_re_dev_list); static DEFINE_MUTEX(bnxt_re_dev_lock); static struct workqueue_struct *bnxt_re_wq; static void bnxt_re_remove_device(struct bnxt_re_dev *rdev); -static void bnxt_re_ib_uninit(struct bnxt_re_dev *rdev); +static void bnxt_re_dealloc_driver(struct ib_device *ib_dev); static void bnxt_re_destroy_chip_ctx(struct bnxt_re_dev *rdev) { @@ -223,9 +223,7 @@ static void bnxt_re_shutdown(void *p) if (!rdev) return; - bnxt_re_ib_uninit(rdev); - /* rtnl_lock held by L2 before coming here */ - bnxt_re_remove_device(rdev); + ib_unregister_device_queued(&rdev->ibdev); } static void bnxt_re_stop_irq(void *handle) @@ -527,17 +525,12 @@ static bool is_bnxt_re_dev(struct net_device *netdev) static struct bnxt_re_dev *bnxt_re_from_netdev(struct net_device *netdev) { - struct bnxt_re_dev *rdev; + struct ib_device *ibdev = + ib_device_get_by_netdev(netdev, RDMA_DRIVER_BNXT_RE); + if (!ibdev) + return NULL; - rcu_read_lock(); - list_for_each_entry_rcu(rdev, &bnxt_re_dev_list, list) { - if (rdev->netdev == netdev) { - rcu_read_unlock(); - return rdev; - } - } - rcu_read_unlock(); - return NULL; + return container_of(ibdev, struct bnxt_re_dev, ibdev); } static void bnxt_re_dev_unprobe(struct net_device *netdev, @@ -611,11 +604,6 @@ static const struct attribute_group bnxt_re_dev_attr_group = { .attrs = bnxt_re_attributes, }; -static void bnxt_re_unregister_ib(struct bnxt_re_dev *rdev) -{ - ib_unregister_device(&rdev->ibdev); -} - static const struct ib_device_ops bnxt_re_dev_ops = { .owner = THIS_MODULE, .driver_id = RDMA_DRIVER_BNXT_RE, @@ -630,6 +618,7 @@ static const struct ib_device_ops bnxt_re_dev_ops = { .create_cq = bnxt_re_create_cq, .create_qp = bnxt_re_create_qp, .create_srq = bnxt_re_create_srq, + .dealloc_driver = bnxt_re_dealloc_driver, .dealloc_pd = bnxt_re_dealloc_pd, .dealloc_ucontext = bnxt_re_dealloc_ucontext, .del_gid = bnxt_re_del_gid, @@ -726,15 +715,11 @@ static void bnxt_re_dev_remove(struct bnxt_re_dev *rdev) { dev_put(rdev->netdev); rdev->netdev = NULL; - mutex_lock(&bnxt_re_dev_lock); list_del_rcu(&rdev->list); mutex_unlock(&bnxt_re_dev_lock); synchronize_rcu(); - - ib_dealloc_device(&rdev->ibdev); - /* rdev is gone */ } static struct bnxt_re_dev *bnxt_re_dev_add(struct net_device *netdev, @@ -767,6 +752,7 @@ static struct bnxt_re_dev *bnxt_re_dev_add(struct net_device *netdev, mutex_lock(&bnxt_re_dev_lock); list_add_tail_rcu(&rdev->list, &bnxt_re_dev_list); mutex_unlock(&bnxt_re_dev_lock); + return rdev; } @@ -1300,15 +1286,6 @@ static void bnxt_re_query_hwrm_intf_version(struct bnxt_re_dev *rdev) le16_to_cpu(resp.hwrm_intf_patch); } -static void bnxt_re_ib_uninit(struct bnxt_re_dev *rdev) -{ - /* Cleanup ib dev */ - if (test_bit(BNXT_RE_FLAG_IBDEV_REGISTERED, &rdev->flags)) { - ib_unregister_device(&rdev->ibdev); - clear_bit(BNXT_RE_FLAG_IBDEV_REGISTERED, &rdev->flags); - } -} - int bnxt_re_ib_init(struct bnxt_re_dev *rdev) { int rc = 0; @@ -1335,10 +1312,6 @@ static void bnxt_re_dev_uninit(struct bnxt_re_dev *rdev) u8 type; int rc; - if (test_and_clear_bit(BNXT_RE_FLAG_IBDEV_REGISTERED, &rdev->flags)) { - /* Cleanup ib dev */ - bnxt_re_unregister_ib(rdev); - } if (test_and_clear_bit(BNXT_RE_FLAG_QOS_WORK_REG, &rdev->flags)) cancel_delayed_work_sync(&rdev->worker); @@ -1595,6 +1568,19 @@ static int bnxt_re_add_device(struct bnxt_re_dev **rdev, return rc; } +static void bnxt_re_dealloc_driver(struct ib_device *ib_dev) +{ + struct bnxt_re_dev *rdev = + container_of(ib_dev, struct bnxt_re_dev, ibdev); + + clear_bit(BNXT_RE_FLAG_IBDEV_REGISTERED, &rdev->flags); + dev_info(rdev_to_dev(rdev), "Unregistering Device"); + + rtnl_lock(); + bnxt_re_remove_device(rdev); + rtnl_unlock(); +} + /* Handle all deferred netevents tasks */ static void bnxt_re_task(struct work_struct *work) { @@ -1669,6 +1655,7 @@ static int bnxt_re_netdev_event(struct notifier_block *notifier, struct bnxt_re_dev *rdev; int rc = 0; bool sch_work = false; + bool release = true; real_dev = rdma_vlan_dev_real_dev(netdev); if (!real_dev) @@ -1676,7 +1663,8 @@ static int bnxt_re_netdev_event(struct notifier_block *notifier, rdev = bnxt_re_from_netdev(real_dev); if (!rdev && event != NETDEV_REGISTER) - goto exit; + return NOTIFY_OK; + if (real_dev != netdev) goto exit; @@ -1687,6 +1675,7 @@ static int bnxt_re_netdev_event(struct notifier_block *notifier, rc = bnxt_re_add_device(&rdev, real_dev); if (!rc) sch_work = true; + release = false; break; case NETDEV_UNREGISTER: @@ -1695,8 +1684,7 @@ static int bnxt_re_netdev_event(struct notifier_block *notifier, */ if (atomic_read(&rdev->sched_count) > 0) goto exit; - bnxt_re_ib_uninit(rdev); - bnxt_re_remove_device(rdev); + ib_unregister_device_queued(&rdev->ibdev); break; default: @@ -1718,6 +1706,8 @@ static int bnxt_re_netdev_event(struct notifier_block *notifier, } exit: + if (release) + ib_device_put(&rdev->ibdev); return NOTIFY_DONE; } @@ -1753,32 +1743,22 @@ static int __init bnxt_re_mod_init(void) static void __exit bnxt_re_mod_exit(void) { - struct bnxt_re_dev *rdev, *next; - LIST_HEAD(to_be_deleted); + struct bnxt_re_dev *rdev; + flush_workqueue(bnxt_re_wq); mutex_lock(&bnxt_re_dev_lock); - /* Free all adapter allocated resources */ - if (!list_empty(&bnxt_re_dev_list)) - list_splice_init(&bnxt_re_dev_list, &to_be_deleted); - mutex_unlock(&bnxt_re_dev_lock); - /* - * Cleanup the devices in reverse order so that the VF device - * cleanup is done before PF cleanup - */ - list_for_each_entry_safe_reverse(rdev, next, &to_be_deleted, list) { - dev_info(rdev_to_dev(rdev), "Unregistering Device"); + list_for_each_entry(rdev, &bnxt_re_dev_list, list) { /* - * Flush out any scheduled tasks before destroying the - * resources + * Set unreg flag to avoid VF resource cleanup + * in module unload path. This is required because + * dealloc_driver for VF can come after PF cleaning + * the VF resources. */ - flush_workqueue(bnxt_re_wq); - bnxt_re_dev_stop(rdev); - bnxt_re_ib_uninit(rdev); - /* Acquire the rtnl_lock as the L2 resources are freed here */ - rtnl_lock(); - bnxt_re_remove_device(rdev); - rtnl_unlock(); + if (rdev->is_virtfn) + rdev->rcfw.res_deinit = true; } + mutex_unlock(&bnxt_re_dev_lock); + ib_unregister_driver(RDMA_DRIVER_BNXT_RE); unregister_netdevice_notifier(&bnxt_re_netdev_notifier); if (bnxt_re_wq) destroy_workqueue(bnxt_re_wq); diff --git a/drivers/infiniband/hw/bnxt_re/qplib_rcfw.c b/drivers/infiniband/hw/bnxt_re/qplib_rcfw.c index 5cdfa84..2304f97 100644 --- a/drivers/infiniband/hw/bnxt_re/qplib_rcfw.c +++ b/drivers/infiniband/hw/bnxt_re/qplib_rcfw.c @@ -211,6 +211,13 @@ int bnxt_qplib_rcfw_send_message(struct bnxt_qplib_rcfw *rcfw, u8 opcode, retry_cnt = 0xFF; int rc = 0; + /* Check if the command needs to be send down to HW. + * Return success if resources are de-initialized + */ + + if (rcfw->res_deinit) + return 0; + do { opcode = req->opcode; rc = __send_message(rcfw, req, resp, sb, is_block); diff --git a/drivers/infiniband/hw/bnxt_re/qplib_rcfw.h b/drivers/infiniband/hw/bnxt_re/qplib_rcfw.h index dfeadc1..c2b930e 100644 --- a/drivers/infiniband/hw/bnxt_re/qplib_rcfw.h +++ b/drivers/infiniband/hw/bnxt_re/qplib_rcfw.h @@ -265,6 +265,7 @@ struct bnxt_qplib_rcfw { u64 oos_prev; u32 init_oos_stats; u32 cmdq_depth; + bool res_deinit; }; void bnxt_qplib_free_rcfw_channel(struct bnxt_qplib_rcfw *rcfw); From patchwork Mon Nov 25 08:39:34 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Selvin Xavier X-Patchwork-Id: 11260015 X-Patchwork-Delegate: jgg@ziepe.ca Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id AFBCD138C for ; Mon, 25 Nov 2019 08:40:15 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 90B9B20815 for ; Mon, 25 Nov 2019 08:40:15 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=broadcom.com header.i=@broadcom.com header.b="GlLvqa86" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726957AbfKYIkO (ORCPT ); Mon, 25 Nov 2019 03:40:14 -0500 Received: from mail-wm1-f66.google.com ([209.85.128.66]:52683 "EHLO mail-wm1-f66.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725793AbfKYIkO (ORCPT ); Mon, 25 Nov 2019 03:40:14 -0500 Received: by mail-wm1-f66.google.com with SMTP id l1so14435469wme.2 for ; Mon, 25 Nov 2019 00:40:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=KFy9Pc54+og40BCKJ07VzGkp54TwLhEqGBhlNfNaDpk=; b=GlLvqa86v3Tif4eb4BdAk9KpHVtmoIipyta/FOSAwR3Cq8N36SaeRDczf2e2z9hR26 d+mXcUJtd+uV3hxUKsF+LTRd2T+n+W0dMVldqhjR6e8MHi5/yozYWEVyA4XeDGcmmwEq oYqk74p8+yNiDwMopMrzy68y/Q1yCnVjD3HxI= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=KFy9Pc54+og40BCKJ07VzGkp54TwLhEqGBhlNfNaDpk=; b=AlYVogw/uZfT9um2Fy6r3DnXL9W12okXonBG+KcfCDMtVvdB9B9SsmV6wAQ0ywVXxd A20vBsPaCGTj1qWuqvavUFSyi2pT9a7Nv2p/r1crk1bchI0UWq8VOxDVp2se1ud9HB2v WHI1ktWcNY796H+UDzHEQCyzMW3uHWay4a8zCJYK0Nr1Vqwo0gZa4+/MSC2lVbRqv6ju w0WL7VBhUQOjDJrTLW2uvUbNa267dEIBFjOWuqqxTbIvnlQs6ErZ7HZLksvR4g6gAQm1 gya7KKXeeDp2KsfMV6uxknEgJkBlRGPOQDjF4yxOIBUtpq3YJ8aM91trB09hDoMfyNV+ YjWQ== X-Gm-Message-State: APjAAAUpF13zHCaGIvXyhoOtu1Tk3TN3u0bv9MAholr1cpKg89t7xwY1 MNjCbN7wWpNYwks30kw73kG1+A== X-Google-Smtp-Source: APXvYqyCmNZEolpUM/O1DnlVPr6M+KVRBszl4lcaLnkumg4dhGPD177y6Jm8xOu4YjggKnwheMkDiA== X-Received: by 2002:a1c:8153:: with SMTP id c80mr9641209wmd.58.1574671211238; Mon, 25 Nov 2019 00:40:11 -0800 (PST) Received: from dhcp-10-192-206-197.iig.avagotech.net.net ([192.19.234.250]) by smtp.gmail.com with ESMTPSA id k4sm7996995wmk.26.2019.11.25.00.40.08 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 25 Nov 2019 00:40:10 -0800 (PST) From: Selvin Xavier To: dledford@redhat.com, jgg@ziepe.ca Cc: linux-rdma@vger.kernel.org, Selvin Xavier Subject: [PATCH for-next 6/6] RDMA/bnxt_re: Report more number of completion vectors Date: Mon, 25 Nov 2019 00:39:34 -0800 Message-Id: <1574671174-5064-7-git-send-email-selvin.xavier@broadcom.com> X-Mailer: git-send-email 2.5.5 In-Reply-To: <1574671174-5064-1-git-send-email-selvin.xavier@broadcom.com> References: <1574671174-5064-1-git-send-email-selvin.xavier@broadcom.com> Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org Report the the data path MSIx vectors allocated by driver as number of completion vectors. One interrupt vector is used for Control path. So reporting one less than the total number of MSIx vectors allocated by the driver. Signed-off-by: Selvin Xavier --- drivers/infiniband/hw/bnxt_re/main.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drivers/infiniband/hw/bnxt_re/main.c b/drivers/infiniband/hw/bnxt_re/main.c index 9ea4ce7..2e400da 100644 --- a/drivers/infiniband/hw/bnxt_re/main.c +++ b/drivers/infiniband/hw/bnxt_re/main.c @@ -669,7 +669,7 @@ static int bnxt_re_register_ib(struct bnxt_re_dev *rdev) bnxt_qplib_get_guid(rdev->netdev->dev_addr, (u8 *)&ibdev->node_guid); - ibdev->num_comp_vectors = 1; + ibdev->num_comp_vectors = rdev->num_msix - 1; ibdev->dev.parent = &rdev->en_dev->pdev->dev; ibdev->local_dma_lkey = BNXT_QPLIB_RSVD_LKEY;