From patchwork Mon Feb 24 10:49:53 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Selvin Xavier X-Patchwork-Id: 11400199 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 8B89E14D5 for ; Mon, 24 Feb 2020 10:50:11 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 6080B20732 for ; Mon, 24 Feb 2020 10:50:11 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=broadcom.com header.i=@broadcom.com header.b="LQAQxvuw" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727202AbgBXKuK (ORCPT ); Mon, 24 Feb 2020 05:50:10 -0500 Received: from mail-wr1-f65.google.com ([209.85.221.65]:45662 "EHLO mail-wr1-f65.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726778AbgBXKuK (ORCPT ); Mon, 24 Feb 2020 05:50:10 -0500 Received: by mail-wr1-f65.google.com with SMTP id g3so9770962wrs.12 for ; Mon, 24 Feb 2020 02:50:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=id9545e4k3uqojt7fpw66/OXpIo0MOpueljiI8GftZ0=; b=LQAQxvuwN/RTXpR1GcbVx5aegGLcwHkNFr2eGCyzLABxU3Y/gr022LtewgchLwp+S4 cItZ3CJyjpxJHRL/dMhaobmfysR1IRkPqLOuFbcc8dI80yYt7e5mxq0zcvXeBVZe7t1m OmCyEeLfCUpu5IMnx2STodi5opSijwCjr6QvU= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=id9545e4k3uqojt7fpw66/OXpIo0MOpueljiI8GftZ0=; b=JkWZR9urvj8xh/6zm8ApsZlyMnz/c9E2Y/JmpxbPxx2zPmadi7CCWhwUGpnompiH06 DYZXvsp9JtUYapMMesEw2D+c1iGWjqDLzjOmVhfzcYE/DHADn2TgNSGSIKPL7BfPPj5b spl9HXIZQzC4mucRaifVt5UskAbiAaKvoAwhltznVZH8mfVFiJgc3sYRdI513GTtBgRB txt9vpG5vptVE+kEoXjs6GhLV6UxLjamQ2G92tyJjYzbrwy+raTs194at6LBfdHtfP1G XCN76lU2RWPidm0YdFvHTsdnkb+tdgfyx3dChukzx904We+pQb5HNp1CgHbsPA9aiRME 4N3Q== X-Gm-Message-State: APjAAAWJer5dzTsY2ddFAV0c2iwoTZsUaZJxoDCIM/lw7J8ibim5fUOt fdK+4AcpEHmeEgS1013A8dB+yQ== X-Google-Smtp-Source: APXvYqxY4Bfz8fjpLFd60hWboK4Yobnqr255WZuCm634VEBG3amhRMKtduN4X1Kcb8N4Cd5U9btaQw== X-Received: by 2002:adf:e401:: with SMTP id g1mr10494254wrm.165.1582541407653; Mon, 24 Feb 2020 02:50:07 -0800 (PST) Received: from dhcp-10-192-206-197.iig.avagotech.net.net ([192.19.234.250]) by smtp.gmail.com with ESMTPSA id c9sm18096426wrq.44.2020.02.24.02.50.05 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 24 Feb 2020 02:50:06 -0800 (PST) From: Selvin Xavier To: dledford@redhat.com, jgg@mellanox.com Cc: linux-rdma@vger.kernel.org, Selvin Xavier Subject: [PATCH for-next v2 1/3] RDMA/bnxt_re: Add more flags in device init and uninit path Date: Mon, 24 Feb 2020 02:49:53 -0800 Message-Id: <1582541395-19409-2-git-send-email-selvin.xavier@broadcom.com> X-Mailer: git-send-email 2.5.5 In-Reply-To: <1582541395-19409-1-git-send-email-selvin.xavier@broadcom.com> References: <1582541395-19409-1-git-send-email-selvin.xavier@broadcom.com> Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org Add more flags for better granularity in in device init/uninit path Signed-off-by: Selvin Xavier --- drivers/infiniband/hw/bnxt_re/bnxt_re.h | 16 +++++++++---- drivers/infiniband/hw/bnxt_re/main.c | 41 ++++++++++++++++----------------- 2 files changed, 31 insertions(+), 26 deletions(-) diff --git a/drivers/infiniband/hw/bnxt_re/bnxt_re.h b/drivers/infiniband/hw/bnxt_re/bnxt_re.h index c736e82..0babc66 100644 --- a/drivers/infiniband/hw/bnxt_re/bnxt_re.h +++ b/drivers/infiniband/hw/bnxt_re/bnxt_re.h @@ -135,11 +135,17 @@ struct bnxt_re_dev { #define BNXT_RE_FLAG_IBDEV_REGISTERED 1 #define BNXT_RE_FLAG_GOT_MSIX 2 #define BNXT_RE_FLAG_HAVE_L2_REF 3 -#define BNXT_RE_FLAG_RCFW_CHANNEL_EN 4 -#define BNXT_RE_FLAG_QOS_WORK_REG 5 -#define BNXT_RE_FLAG_RESOURCES_ALLOCATED 7 -#define BNXT_RE_FLAG_RESOURCES_INITIALIZED 8 -#define BNXT_RE_FLAG_ISSUE_ROCE_STATS 29 +#define BNXT_RE_FLAG_ALLOC_RCFW 4 +#define BNXT_RE_FLAG_NET_RING_ALLOC 5 +#define BNXT_RE_FLAG_RCFW_CHANNEL_EN 6 +#define BNXT_RE_FLAG_ALLOC_CTX 7 +#define BNXT_RE_FLAG_STATS_CTX_ALLOC 8 +#define BNXT_RE_FLAG_STATS_CTX2_ALLOC 9 +#define BNXT_RE_FLAG_RCFW_CHANNEL_INIT 10 +#define BNXT_RE_FLAG_QOS_WORK_REG 11 +#define BNXT_RE_FLAG_RESOURCES_ALLOCATED 12 +#define BNXT_RE_FLAG_RESOURCES_INITIALIZED 13 +#define BNXT_RE_FLAG_ISSUE_ROCE_STATS 29 struct net_device *netdev; unsigned int version, major, minor; struct bnxt_qplib_chip_ctx *chip_ctx; diff --git a/drivers/infiniband/hw/bnxt_re/main.c b/drivers/infiniband/hw/bnxt_re/main.c index b5128cc..1402130 100644 --- a/drivers/infiniband/hw/bnxt_re/main.c +++ b/drivers/infiniband/hw/bnxt_re/main.c @@ -1335,18 +1335,23 @@ static void bnxt_re_ib_unreg(struct bnxt_re_dev *rdev) if (test_and_clear_bit(BNXT_RE_FLAG_RESOURCES_ALLOCATED, &rdev->flags)) bnxt_re_free_res(rdev); - if (test_and_clear_bit(BNXT_RE_FLAG_RCFW_CHANNEL_EN, &rdev->flags)) { + if (test_and_clear_bit(BNXT_RE_FLAG_RCFW_CHANNEL_INIT, &rdev->flags)) { rc = bnxt_qplib_deinit_rcfw(&rdev->rcfw); if (rc) ibdev_warn(&rdev->ibdev, "Failed to deinitialize RCFW: %#x", rc); + } + if (test_and_clear_bit(BNXT_RE_FLAG_STATS_CTX_ALLOC, &rdev->flags)) bnxt_re_net_stats_ctx_free(rdev, rdev->qplib_ctx.stats.fw_id); + if (test_and_clear_bit(BNXT_RE_FLAG_ALLOC_CTX, &rdev->flags)) bnxt_qplib_free_ctx(&rdev->qplib_res, &rdev->qplib_ctx); + if (test_and_clear_bit(BNXT_RE_FLAG_RCFW_CHANNEL_EN, &rdev->flags)) bnxt_qplib_disable_rcfw_channel(&rdev->rcfw); - type = bnxt_qplib_get_ring_type(rdev->chip_ctx); + type = bnxt_qplib_get_ring_type(rdev->chip_ctx); + if (test_and_clear_bit(BNXT_RE_FLAG_NET_RING_ALLOC, &rdev->flags)) bnxt_re_net_ring_free(rdev, rdev->rcfw.creq.ring_id, type); + if (test_and_clear_bit(BNXT_RE_FLAG_ALLOC_RCFW, &rdev->flags)) bnxt_qplib_free_rcfw_channel(&rdev->rcfw); - } if (test_and_clear_bit(BNXT_RE_FLAG_GOT_MSIX, &rdev->flags)) { rc = bnxt_re_free_msix(rdev); if (rc) @@ -1430,6 +1435,7 @@ static int bnxt_re_ib_reg(struct bnxt_re_dev *rdev) goto fail; } + set_bit(BNXT_RE_FLAG_ALLOC_RCFW, &rdev->flags); type = bnxt_qplib_get_ring_type(rdev->chip_ctx); creq = &rdev->rcfw.creq; rattr.dma_arr = creq->hwq.pbl[PBL_LVL_0].pg_map_arr; @@ -1441,8 +1447,9 @@ static int bnxt_re_ib_reg(struct bnxt_re_dev *rdev) rc = bnxt_re_net_ring_alloc(rdev, &rattr, &creq->ring_id); if (rc) { ibdev_err(&rdev->ibdev, "Failed to allocate CREQ: %#x\n", rc); - goto free_rcfw; + goto fail; } + set_bit(BNXT_RE_FLAG_NET_RING_ALLOC, &rdev->flags); db_offt = bnxt_re_get_nqdb_offset(rdev, BNXT_RE_AEQ_IDX); vid = rdev->msix_entries[BNXT_RE_AEQ_IDX].vector; rc = bnxt_qplib_enable_rcfw_channel(&rdev->rcfw, @@ -1451,13 +1458,14 @@ static int bnxt_re_ib_reg(struct bnxt_re_dev *rdev) if (rc) { ibdev_err(&rdev->ibdev, "Failed to enable RCFW channel: %#x\n", rc); - goto free_ring; + goto fail; } + set_bit(BNXT_RE_FLAG_RCFW_CHANNEL_EN, &rdev->flags); rc = bnxt_qplib_get_dev_attr(&rdev->rcfw, &rdev->dev_attr, rdev->is_virtfn); if (rc) - goto disable_rcfw; + goto fail; bnxt_re_set_resource_limits(rdev); @@ -1466,25 +1474,27 @@ static int bnxt_re_ib_reg(struct bnxt_re_dev *rdev) if (rc) { ibdev_err(&rdev->ibdev, "Failed to allocate QPLIB context: %#x\n", rc); - goto disable_rcfw; + goto fail; } + set_bit(BNXT_RE_FLAG_ALLOC_CTX, &rdev->flags); rc = bnxt_re_net_stats_ctx_alloc(rdev, rdev->qplib_ctx.stats.dma_map, &rdev->qplib_ctx.stats.fw_id); if (rc) { ibdev_err(&rdev->ibdev, "Failed to allocate stats context: %#x\n", rc); - goto free_ctx; + goto fail; } + set_bit(BNXT_RE_FLAG_STATS_CTX_ALLOC, &rdev->flags); rc = bnxt_qplib_init_rcfw(&rdev->rcfw, &rdev->qplib_ctx, rdev->is_virtfn); if (rc) { ibdev_err(&rdev->ibdev, "Failed to initialize RCFW: %#x\n", rc); - goto free_sctx; + goto fail; } - set_bit(BNXT_RE_FLAG_RCFW_CHANNEL_EN, &rdev->flags); + set_bit(BNXT_RE_FLAG_RCFW_CHANNEL_INIT, &rdev->flags); /* Resources based on the 'new' device caps */ rc = bnxt_re_alloc_res(rdev); @@ -1532,17 +1542,6 @@ static int bnxt_re_ib_reg(struct bnxt_re_dev *rdev) bnxt_re_dispatch_event(&rdev->ibdev, NULL, 1, IB_EVENT_PORT_ACTIVE); return 0; -free_sctx: - bnxt_re_net_stats_ctx_free(rdev, rdev->qplib_ctx.stats.fw_id); -free_ctx: - bnxt_qplib_free_ctx(&rdev->qplib_res, &rdev->qplib_ctx); -disable_rcfw: - bnxt_qplib_disable_rcfw_channel(&rdev->rcfw); -free_ring: - type = bnxt_qplib_get_ring_type(rdev->chip_ctx); - bnxt_re_net_ring_free(rdev, rdev->rcfw.creq.ring_id, type); -free_rcfw: - bnxt_qplib_free_rcfw_channel(&rdev->rcfw); fail: if (!locked) rtnl_lock(); From patchwork Mon Feb 24 10:49:54 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Selvin Xavier X-Patchwork-Id: 11400201 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 601711395 for ; Mon, 24 Feb 2020 10:50:14 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 34EFE20732 for ; Mon, 24 Feb 2020 10:50:14 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=broadcom.com header.i=@broadcom.com header.b="RRxAjh0h" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726838AbgBXKuN (ORCPT ); Mon, 24 Feb 2020 05:50:13 -0500 Received: from mail-wr1-f68.google.com ([209.85.221.68]:41536 "EHLO mail-wr1-f68.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726778AbgBXKuN (ORCPT ); Mon, 24 Feb 2020 05:50:13 -0500 Received: by mail-wr1-f68.google.com with SMTP id c9so9810500wrw.8 for ; Mon, 24 Feb 2020 02:50:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=UYc4YubWH5Kmwr3B3MeH+4mb+bQlFGCCqkM+VNqFF0Q=; b=RRxAjh0hQNnCT15I1I1a8cF7yr4ZZdl/Dsa5sf2R8M6xNCWRgSONp6iCO3VKA1NGAm Gs52z3hVZWbbEpfuizmNdq4EunwqeVoT6brJTyuADubJNhc4qNWAGrDzpAFmSnDCAsdc fu6Y2VSC/u0QqoZUjbiUe+5dkZhxQKH178e48= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=UYc4YubWH5Kmwr3B3MeH+4mb+bQlFGCCqkM+VNqFF0Q=; b=qD2H2d9jMNG93LJHDbchDbErexmF+TUqXyLV02FMthKGZZPYFqHJLoTKm2y5UeV5uE X9fI9oBEphsI+g4cSjVajrStJJ2RUlD3qaSgFRjR8u/4Q9hOA6MiTUyViD6m/Z+IzBi7 CdzUPxfNlNfPYPFRb+Tb1GjMYHx+c/I96S9M+LW3hrEYw+fmcDld5x54Prc9Acugtwyo W68HqQYwR/ge3K06z/CJMK48sOeRV6IrV+N7LGYwntjZ7bfdYBSyzItG0yq2p+gX591L c3vFrsT5T7x6F37rgIRSonRv7ww8P8GVOcXjeWArWoGCJhHbY/ib74FkQ05MqKeRrrjO fFdA== X-Gm-Message-State: APjAAAUXj+AQA5JdYLh/CjLrElwUjVs5RDWVNOPitwfl23+wG6jVorjf B980tn1ZnuSwNmG5gTcFSVg1gg== X-Google-Smtp-Source: APXvYqxOcmMc9H+FhT6OgDTsomuP8ZNdI9JAiPLybYIVeF+X9YIcPzH4pv1I4HFqfNjBP/lyMnhheA== X-Received: by 2002:a5d:4a06:: with SMTP id m6mr66497694wrq.155.1582541410874; Mon, 24 Feb 2020 02:50:10 -0800 (PST) Received: from dhcp-10-192-206-197.iig.avagotech.net.net ([192.19.234.250]) by smtp.gmail.com with ESMTPSA id c9sm18096426wrq.44.2020.02.24.02.50.07 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 24 Feb 2020 02:50:09 -0800 (PST) From: Selvin Xavier To: dledford@redhat.com, jgg@mellanox.com Cc: linux-rdma@vger.kernel.org, Selvin Xavier Subject: [PATCH for-next v2 2/3] RDMA/bnxt_re: Refactor device add/remove functionalities Date: Mon, 24 Feb 2020 02:49:54 -0800 Message-Id: <1582541395-19409-3-git-send-email-selvin.xavier@broadcom.com> X-Mailer: git-send-email 2.5.5 In-Reply-To: <1582541395-19409-1-git-send-email-selvin.xavier@broadcom.com> References: <1582541395-19409-1-git-send-email-selvin.xavier@broadcom.com> Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org - bnxt_re_ib_reg() handles two main functionalities - initializing the device and registering with the IB stack. Split it into 2 functions i.e. bnxt_re_dev_init() and bnxt_re_ib_init() to account for the same thereby improve modularity. Do the same for bnxt_re_ib_unreg()i.e. split into two functions - bnxt_re_dev_uninit() and bnxt_re_ib_uninit(). - Simplify the code by combining the different steps to add and remove the device into two functions. Signed-off-by: Selvin Xavier --- drivers/infiniband/hw/bnxt_re/main.c | 135 ++++++++++++++++++++--------------- 1 file changed, 78 insertions(+), 57 deletions(-) diff --git a/drivers/infiniband/hw/bnxt_re/main.c b/drivers/infiniband/hw/bnxt_re/main.c index 1402130..3a548b1 100644 --- a/drivers/infiniband/hw/bnxt_re/main.c +++ b/drivers/infiniband/hw/bnxt_re/main.c @@ -78,7 +78,8 @@ static struct list_head bnxt_re_dev_list = LIST_HEAD_INIT(bnxt_re_dev_list); /* Mutex to protect the list of bnxt_re devices added */ static DEFINE_MUTEX(bnxt_re_dev_lock); static struct workqueue_struct *bnxt_re_wq; -static void bnxt_re_ib_unreg(struct bnxt_re_dev *rdev); +static void bnxt_re_remove_device(struct bnxt_re_dev *rdev); +static void bnxt_re_ib_uninit(struct bnxt_re_dev *rdev); static void bnxt_re_destroy_chip_ctx(struct bnxt_re_dev *rdev) { @@ -237,7 +238,9 @@ static void bnxt_re_shutdown(void *p) if (!rdev) return; - bnxt_re_ib_unreg(rdev); + bnxt_re_ib_uninit(rdev); + ASSERT_RTNL(); + bnxt_re_remove_device(rdev); } static void bnxt_re_stop_irq(void *handle) @@ -1317,7 +1320,37 @@ static void bnxt_re_query_hwrm_intf_version(struct bnxt_re_dev *rdev) le16_to_cpu(resp.hwrm_intf_patch); } -static void bnxt_re_ib_unreg(struct bnxt_re_dev *rdev) +static void bnxt_re_ib_uninit(struct bnxt_re_dev *rdev) +{ + /* Cleanup ib dev */ + if (test_bit(BNXT_RE_FLAG_IBDEV_REGISTERED, &rdev->flags)) { + ib_unregister_device(&rdev->ibdev); + clear_bit(BNXT_RE_FLAG_IBDEV_REGISTERED, &rdev->flags); + } +} + +int bnxt_re_ib_init(struct bnxt_re_dev *rdev) +{ + int rc = 0; + + /* Register ib dev */ + rc = bnxt_re_register_ib(rdev); + if (rc) { + pr_err("Failed to register with IB: %#x\n", rc); + return rc; + } + set_bit(BNXT_RE_FLAG_IBDEV_REGISTERED, &rdev->flags); + dev_info(rdev_to_dev(rdev), "Device registered successfully"); + ib_get_eth_speed(&rdev->ibdev, 1, &rdev->active_speed, + &rdev->active_width); + set_bit(BNXT_RE_FLAG_ISSUE_ROCE_STATS, &rdev->flags); + bnxt_re_dispatch_event(&rdev->ibdev, NULL, 1, IB_EVENT_PORT_ACTIVE); + bnxt_re_dispatch_event(&rdev->ibdev, NULL, 1, IB_EVENT_GID_CHANGE); + + return rc; +} + +static void bnxt_re_dev_uninit(struct bnxt_re_dev *rdev) { u8 type; int rc; @@ -1378,20 +1411,15 @@ static void bnxt_re_worker(struct work_struct *work) schedule_delayed_work(&rdev->worker, msecs_to_jiffies(30000)); } -static int bnxt_re_ib_reg(struct bnxt_re_dev *rdev) +static int bnxt_re_dev_init(struct bnxt_re_dev *rdev) { struct bnxt_qplib_creq_ctx *creq; struct bnxt_re_ring_attr rattr; u32 db_offt; - bool locked; int vid; u8 type; int rc; - /* Acquire rtnl lock through out this function */ - rtnl_lock(); - locked = true; - /* Registered a new RoCE device instance to netdev */ memset(&rattr, 0, sizeof(rattr)); rc = bnxt_re_register_netdev(rdev); @@ -1524,30 +1552,10 @@ static int bnxt_re_ib_reg(struct bnxt_re_dev *rdev) schedule_delayed_work(&rdev->worker, msecs_to_jiffies(30000)); } - rtnl_unlock(); - locked = false; - - /* Register ib dev */ - rc = bnxt_re_register_ib(rdev); - if (rc) { - ibdev_err(&rdev->ibdev, - "Failed to register with IB: %#x\n", rc); - goto fail; - } - set_bit(BNXT_RE_FLAG_IBDEV_REGISTERED, &rdev->flags); - ibdev_info(&rdev->ibdev, "Device registered successfully"); - ib_get_eth_speed(&rdev->ibdev, 1, &rdev->active_speed, - &rdev->active_width); - set_bit(BNXT_RE_FLAG_ISSUE_ROCE_STATS, &rdev->flags); - bnxt_re_dispatch_event(&rdev->ibdev, NULL, 1, IB_EVENT_PORT_ACTIVE); - return 0; fail: - if (!locked) - rtnl_lock(); - bnxt_re_ib_unreg(rdev); - rtnl_unlock(); + bnxt_re_dev_uninit(rdev); return rc; } @@ -1588,9 +1596,35 @@ static int bnxt_re_dev_reg(struct bnxt_re_dev **rdev, struct net_device *netdev) return rc; } -static void bnxt_re_remove_one(struct bnxt_re_dev *rdev) +static void bnxt_re_remove_device(struct bnxt_re_dev *rdev) { + bnxt_re_dev_uninit(rdev); pci_dev_put(rdev->en_dev->pdev); + bnxt_re_dev_unreg(rdev); +} + +static int bnxt_re_add_device(struct bnxt_re_dev **rdev, + struct net_device *netdev) +{ + int rc; + + rc = bnxt_re_dev_reg(rdev, netdev); + if (rc == -ENODEV) + return rc; + if (rc) { + pr_err("Failed to register with the device %s: %#x\n", + netdev->name, rc); + return rc; + } + + pci_dev_get((*rdev)->en_dev->pdev); + rc = bnxt_re_dev_init(*rdev); + if (rc) { + pci_dev_put((*rdev)->en_dev->pdev); + bnxt_re_dev_unreg(*rdev); + } + + return rc; } /* Handle all deferred netevents tasks */ @@ -1605,16 +1639,17 @@ static void bnxt_re_task(struct work_struct *work) if (re_work->event != NETDEV_REGISTER && !test_bit(BNXT_RE_FLAG_IBDEV_REGISTERED, &rdev->flags)) - return; + goto done; switch (re_work->event) { case NETDEV_REGISTER: - rc = bnxt_re_ib_reg(rdev); + rc = bnxt_re_ib_init(rdev); if (rc) { ibdev_err(&rdev->ibdev, "Failed to register with IB: %#x", rc); - bnxt_re_remove_one(rdev); - bnxt_re_dev_unreg(rdev); + rtnl_lock(); + bnxt_re_remove_device(rdev); + rtnl_unlock(); goto exit; } break; @@ -1637,17 +1672,13 @@ static void bnxt_re_task(struct work_struct *work) default: break; } +done: smp_mb__before_atomic(); atomic_dec(&rdev->sched_count); exit: kfree(re_work); } -static void bnxt_re_init_one(struct bnxt_re_dev *rdev) -{ - pci_dev_get(rdev->en_dev->pdev); -} - /* * "Notifier chain callback can be invoked for the same chain from * different CPUs at the same time". @@ -1685,17 +1716,9 @@ static int bnxt_re_netdev_event(struct notifier_block *notifier, case NETDEV_REGISTER: if (rdev) break; - rc = bnxt_re_dev_reg(&rdev, real_dev); - if (rc == -ENODEV) - break; - if (rc) { - ibdev_err(&rdev->ibdev, - "Failed to register with the device %s: %#x\n", - real_dev->name, rc); - break; - } - bnxt_re_init_one(rdev); - sch_work = true; + rc = bnxt_re_add_device(&rdev, real_dev); + if (!rc) + sch_work = true; break; case NETDEV_UNREGISTER: @@ -1704,9 +1727,8 @@ static int bnxt_re_netdev_event(struct notifier_block *notifier, */ if (atomic_read(&rdev->sched_count) > 0) goto exit; - bnxt_re_ib_unreg(rdev); - bnxt_re_remove_one(rdev); - bnxt_re_dev_unreg(rdev); + bnxt_re_ib_uninit(rdev); + bnxt_re_remove_device(rdev); break; default: @@ -1783,12 +1805,11 @@ static void __exit bnxt_re_mod_exit(void) */ flush_workqueue(bnxt_re_wq); bnxt_re_dev_stop(rdev); + bnxt_re_ib_uninit(rdev); /* Acquire the rtnl_lock as the L2 resources are freed here */ rtnl_lock(); - bnxt_re_ib_unreg(rdev); + bnxt_re_remove_device(rdev); rtnl_unlock(); - bnxt_re_remove_one(rdev); - bnxt_re_dev_unreg(rdev); } unregister_netdevice_notifier(&bnxt_re_netdev_notifier); if (bnxt_re_wq) From patchwork Mon Feb 24 10:49:55 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Selvin Xavier X-Patchwork-Id: 11400203 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 714B11395 for ; Mon, 24 Feb 2020 10:50:17 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 479CB20828 for ; Mon, 24 Feb 2020 10:50:17 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=broadcom.com header.i=@broadcom.com header.b="LVJiV9zp" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726765AbgBXKuQ (ORCPT ); Mon, 24 Feb 2020 05:50:16 -0500 Received: from mail-wr1-f68.google.com ([209.85.221.68]:38113 "EHLO mail-wr1-f68.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726778AbgBXKuQ (ORCPT ); Mon, 24 Feb 2020 05:50:16 -0500 Received: by mail-wr1-f68.google.com with SMTP id e8so9807452wrm.5 for ; Mon, 24 Feb 2020 02:50:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=BmppQlbJRz0w8KNJ0nbzuNuPVPH7cIIQhQrpCUHRnOY=; b=LVJiV9zp34c+6J6pdTB1DKDBB0ncHQ0DQWS9JVdJPnnraQBAuBGAOYpepMMapf5XgH tdV66jlyuywedywY52gAaCaZ2VpppJeIqvUxg7rPX1fGp0J3GZq6pVqw510tKgUomfrK aqyaaRT6hAVvX+t+gv87ps5oFPZC1ncS1R8/A= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=BmppQlbJRz0w8KNJ0nbzuNuPVPH7cIIQhQrpCUHRnOY=; b=tqPpgSnET+iOTJDEKGNl8bJcd68VL8sFZSAlIqqMCMHJ0iPznxuGSu0wrJFpdxcJKT rB7rqS+6xIeklfF4WGAYQFvvpLvQYWxB+c5tGvzQHRGrFPJdO2p7+Ys+KfQ4fyIf0FGY YZMGt2OpKYPUKvXuLHtGuNy+Cp5W54VeWRinSHfulUX8mn08ZaosuRWK3Gbk/HWR+HZg 0mUb6u2mBiwZ4vMAfAVbb/0Ow2lfQ453vTQrSst118hc0DeZIrwzdyt6LmhLjOK+eiVu nkzO/Ls/P/T6rJaTFO3JfVsfGurM8eJW6i5SjEhtgwxR4wB9SV3Am4wzCuEDiJ0WRtoP +m/g== X-Gm-Message-State: APjAAAU9suPZb85Rd4mF4ricuNIyTiySEefC9QI0ublQdYdGsDZO2jHM 4g1/8YVFqllrYbWmOEVs1Pl6Tg== X-Google-Smtp-Source: APXvYqz/g3v0Xw5V1SCjcI5bkUHB24x5mvxYdmGWWeJeSJlaWJYFzUHrgg66iWsfzQFT9PM7z6N41A== X-Received: by 2002:adf:db48:: with SMTP id f8mr62540198wrj.146.1582541413697; Mon, 24 Feb 2020 02:50:13 -0800 (PST) Received: from dhcp-10-192-206-197.iig.avagotech.net.net ([192.19.234.250]) by smtp.gmail.com with ESMTPSA id c9sm18096426wrq.44.2020.02.24.02.50.11 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 24 Feb 2020 02:50:13 -0800 (PST) From: Selvin Xavier To: dledford@redhat.com, jgg@mellanox.com Cc: linux-rdma@vger.kernel.org, Selvin Xavier Subject: [PATCH for-next v2 3/3] RDMA/bnxt_re: Use driver_unregister and unregistration API Date: Mon, 24 Feb 2020 02:49:55 -0800 Message-Id: <1582541395-19409-4-git-send-email-selvin.xavier@broadcom.com> X-Mailer: git-send-email 2.5.5 In-Reply-To: <1582541395-19409-1-git-send-email-selvin.xavier@broadcom.com> References: <1582541395-19409-1-git-send-email-selvin.xavier@broadcom.com> Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org Using the new unregister APIs provided by the core. Provide the dealloc_driver hook for the core to callback at the time of device un-registration. bnxt_re VF resources are created by the corresponding PF driver. During ib_unregister_driver, PF might get removed before VF and this could cause failure when VFs are removed. Driver is explicitly queuing the removal of VF devices before calling ib_unregister_driver. Commands to any VFs after the PF removal will timeout and driver will proceed with unregisteration and free up the host resources. Signed-off-by: Selvin Xavier --- drivers/infiniband/hw/bnxt_re/main.c | 105 +++++++++++++++-------------------- 1 file changed, 44 insertions(+), 61 deletions(-) diff --git a/drivers/infiniband/hw/bnxt_re/main.c b/drivers/infiniband/hw/bnxt_re/main.c index 3a548b1..6dec09d 100644 --- a/drivers/infiniband/hw/bnxt_re/main.c +++ b/drivers/infiniband/hw/bnxt_re/main.c @@ -79,7 +79,8 @@ static struct list_head bnxt_re_dev_list = LIST_HEAD_INIT(bnxt_re_dev_list); static DEFINE_MUTEX(bnxt_re_dev_lock); static struct workqueue_struct *bnxt_re_wq; static void bnxt_re_remove_device(struct bnxt_re_dev *rdev); -static void bnxt_re_ib_uninit(struct bnxt_re_dev *rdev); +static void bnxt_re_dealloc_driver(struct ib_device *ib_dev); +static void bnxt_re_stop_irq(void *handle); static void bnxt_re_destroy_chip_ctx(struct bnxt_re_dev *rdev) { @@ -237,10 +238,10 @@ static void bnxt_re_shutdown(void *p) if (!rdev) return; - - bnxt_re_ib_uninit(rdev); ASSERT_RTNL(); - bnxt_re_remove_device(rdev); + /* Release the MSIx vectors before queuing unregister */ + bnxt_re_stop_irq(rdev); + ib_unregister_device_queued(&rdev->ibdev); } static void bnxt_re_stop_irq(void *handle) @@ -542,17 +543,12 @@ static bool is_bnxt_re_dev(struct net_device *netdev) static struct bnxt_re_dev *bnxt_re_from_netdev(struct net_device *netdev) { - struct bnxt_re_dev *rdev; + struct ib_device *ibdev = + ib_device_get_by_netdev(netdev, RDMA_DRIVER_BNXT_RE); + if (!ibdev) + return NULL; - rcu_read_lock(); - list_for_each_entry_rcu(rdev, &bnxt_re_dev_list, list) { - if (rdev->netdev == netdev) { - rcu_read_unlock(); - return rdev; - } - } - rcu_read_unlock(); - return NULL; + return container_of(ibdev, struct bnxt_re_dev, ibdev); } static void bnxt_re_dev_unprobe(struct net_device *netdev, @@ -626,11 +622,6 @@ static const struct attribute_group bnxt_re_dev_attr_group = { .attrs = bnxt_re_attributes, }; -static void bnxt_re_unregister_ib(struct bnxt_re_dev *rdev) -{ - ib_unregister_device(&rdev->ibdev); -} - static const struct ib_device_ops bnxt_re_dev_ops = { .owner = THIS_MODULE, .driver_id = RDMA_DRIVER_BNXT_RE, @@ -645,6 +636,7 @@ static const struct ib_device_ops bnxt_re_dev_ops = { .create_cq = bnxt_re_create_cq, .create_qp = bnxt_re_create_qp, .create_srq = bnxt_re_create_srq, + .dealloc_driver = bnxt_re_dealloc_driver, .dealloc_pd = bnxt_re_dealloc_pd, .dealloc_ucontext = bnxt_re_dealloc_ucontext, .del_gid = bnxt_re_del_gid, @@ -741,15 +733,11 @@ static void bnxt_re_dev_remove(struct bnxt_re_dev *rdev) { dev_put(rdev->netdev); rdev->netdev = NULL; - mutex_lock(&bnxt_re_dev_lock); list_del_rcu(&rdev->list); mutex_unlock(&bnxt_re_dev_lock); synchronize_rcu(); - - ib_dealloc_device(&rdev->ibdev); - /* rdev is gone */ } static struct bnxt_re_dev *bnxt_re_dev_add(struct net_device *netdev, @@ -1320,15 +1308,6 @@ static void bnxt_re_query_hwrm_intf_version(struct bnxt_re_dev *rdev) le16_to_cpu(resp.hwrm_intf_patch); } -static void bnxt_re_ib_uninit(struct bnxt_re_dev *rdev) -{ - /* Cleanup ib dev */ - if (test_bit(BNXT_RE_FLAG_IBDEV_REGISTERED, &rdev->flags)) { - ib_unregister_device(&rdev->ibdev); - clear_bit(BNXT_RE_FLAG_IBDEV_REGISTERED, &rdev->flags); - } -} - int bnxt_re_ib_init(struct bnxt_re_dev *rdev) { int rc = 0; @@ -1355,10 +1334,6 @@ static void bnxt_re_dev_uninit(struct bnxt_re_dev *rdev) u8 type; int rc; - if (test_and_clear_bit(BNXT_RE_FLAG_IBDEV_REGISTERED, &rdev->flags)) { - /* Cleanup ib dev */ - bnxt_re_unregister_ib(rdev); - } if (test_and_clear_bit(BNXT_RE_FLAG_QOS_WORK_REG, &rdev->flags)) cancel_delayed_work_sync(&rdev->worker); @@ -1627,6 +1602,19 @@ static int bnxt_re_add_device(struct bnxt_re_dev **rdev, return rc; } +static void bnxt_re_dealloc_driver(struct ib_device *ib_dev) +{ + struct bnxt_re_dev *rdev = + container_of(ib_dev, struct bnxt_re_dev, ibdev); + + clear_bit(BNXT_RE_FLAG_IBDEV_REGISTERED, &rdev->flags); + dev_info(rdev_to_dev(rdev), "Unregistering Device"); + + rtnl_lock(); + bnxt_re_remove_device(rdev); + rtnl_unlock(); +} + /* Handle all deferred netevents tasks */ static void bnxt_re_task(struct work_struct *work) { @@ -1701,6 +1689,7 @@ static int bnxt_re_netdev_event(struct notifier_block *notifier, struct bnxt_re_dev *rdev; int rc = 0; bool sch_work = false; + bool release = true; real_dev = rdma_vlan_dev_real_dev(netdev); if (!real_dev) @@ -1708,7 +1697,8 @@ static int bnxt_re_netdev_event(struct notifier_block *notifier, rdev = bnxt_re_from_netdev(real_dev); if (!rdev && event != NETDEV_REGISTER) - goto exit; + return NOTIFY_OK; + if (real_dev != netdev) goto exit; @@ -1719,6 +1709,7 @@ static int bnxt_re_netdev_event(struct notifier_block *notifier, rc = bnxt_re_add_device(&rdev, real_dev); if (!rc) sch_work = true; + release = false; break; case NETDEV_UNREGISTER: @@ -1727,8 +1718,7 @@ static int bnxt_re_netdev_event(struct notifier_block *notifier, */ if (atomic_read(&rdev->sched_count) > 0) goto exit; - bnxt_re_ib_uninit(rdev); - bnxt_re_remove_device(rdev); + ib_unregister_device_queued(&rdev->ibdev); break; default: @@ -1750,6 +1740,8 @@ static int bnxt_re_netdev_event(struct notifier_block *notifier, } exit: + if (rdev && release) + ib_device_put(&rdev->ibdev); return NOTIFY_DONE; } @@ -1785,32 +1777,23 @@ static int __init bnxt_re_mod_init(void) static void __exit bnxt_re_mod_exit(void) { - struct bnxt_re_dev *rdev, *next; - LIST_HEAD(to_be_deleted); + struct bnxt_re_dev *rdev; + flush_workqueue(bnxt_re_wq); mutex_lock(&bnxt_re_dev_lock); - /* Free all adapter allocated resources */ - if (!list_empty(&bnxt_re_dev_list)) - list_splice_init(&bnxt_re_dev_list, &to_be_deleted); - mutex_unlock(&bnxt_re_dev_lock); - /* - * Cleanup the devices in reverse order so that the VF device - * cleanup is done before PF cleanup - */ - list_for_each_entry_safe_reverse(rdev, next, &to_be_deleted, list) { - ibdev_info(&rdev->ibdev, "Unregistering Device"); - /* - * Flush out any scheduled tasks before destroying the - * resources + list_for_each_entry(rdev, &bnxt_re_dev_list, list) { + /* VF device removal should be called before the removal + * of PF device. Queue VFs unregister first, so that VFs + * shall be removed before the PF during the call of + * ib_unregister_driver. Commands to any VFs after the + * PF removal will timeout and driver will proceed with + * unregisteration and free up the host resources. */ - flush_workqueue(bnxt_re_wq); - bnxt_re_dev_stop(rdev); - bnxt_re_ib_uninit(rdev); - /* Acquire the rtnl_lock as the L2 resources are freed here */ - rtnl_lock(); - bnxt_re_remove_device(rdev); - rtnl_unlock(); + if (rdev->is_virtfn) + ib_unregister_device_queued(&rdev->ibdev); } + mutex_unlock(&bnxt_re_dev_lock); + ib_unregister_driver(RDMA_DRIVER_BNXT_RE); unregister_netdevice_notifier(&bnxt_re_netdev_notifier); if (bnxt_re_wq) destroy_workqueue(bnxt_re_wq);