From patchwork Sun Jan 8 10:35:29 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Arinzon, David" X-Patchwork-Id: 13092458 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 002FEC53210 for ; Sun, 8 Jan 2023 10:35:52 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232110AbjAHKfu (ORCPT ); Sun, 8 Jan 2023 05:35:50 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35530 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229520AbjAHKfs (ORCPT ); Sun, 8 Jan 2023 05:35:48 -0500 Received: from smtp-fw-80007.amazon.com (smtp-fw-80007.amazon.com [99.78.197.218]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4AD95E099 for ; Sun, 8 Jan 2023 02:35:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1673174148; x=1704710148; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=jmMFhHcQzWvtylhhO2ZZKHohq5cIvThpGTZA9jpMnx4=; b=Fek9kL15xUziBR5IXcNjhkZM/jcdATpDOtHQK4oyDyg1ZufseRh3bN7b EfJkUvxSHFShifjEprYKSx98CxGGJjrgTFoJt1/qaMGITbFs/8Uc5o8L0 pRNveIZR9lnMr5jXtWDNJYSDgVaXKXjnRHrCIGU9qcDXtX2bhazX32MGn I=; X-IronPort-AV: E=Sophos;i="5.96,310,1665446400"; d="scan'208";a="168928150" Received: from pdx4-co-svc-p1-lb2-vlan3.amazon.com (HELO email-inbound-relay-iad-1e-m6i4x-b538c141.us-east-1.amazon.com) ([10.25.36.214]) by smtp-border-fw-80007.pdx80.corp.amazon.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 Jan 2023 10:35:45 +0000 Received: from EX13MTAUWB002.ant.amazon.com (iad12-ws-svc-p26-lb9-vlan2.iad.amazon.com [10.40.163.34]) by email-inbound-relay-iad-1e-m6i4x-b538c141.us-east-1.amazon.com (Postfix) with ESMTPS id 871BABF7F2; Sun, 8 Jan 2023 10:35:43 +0000 (UTC) Received: from EX19D002UWA002.ant.amazon.com (10.13.138.246) by EX13MTAUWB002.ant.amazon.com (10.43.161.202) with Microsoft SMTP Server (TLS) id 15.0.1497.42; Sun, 8 Jan 2023 10:35:42 +0000 Received: from EX13MTAUWA001.ant.amazon.com (10.43.160.58) by EX19D002UWA002.ant.amazon.com (10.13.138.246) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA) id 15.2.1118.7; Sun, 8 Jan 2023 10:35:42 +0000 Received: from dev-dsk-darinzon-1c-05962a8d.eu-west-1.amazon.com (172.19.80.187) by mail-relay.amazon.com (10.43.160.118) with Microsoft SMTP Server id 15.0.1497.42 via Frontend Transport; Sun, 8 Jan 2023 10:35:40 +0000 From: David Arinzon To: David Miller , Jakub Kicinski , CC: David Arinzon , "Machulsky, Zorik" , "Matushevsky, Alexander" , Saeed Bshara , "Bshara, Nafea" , "Saidi, Ali" , "Kiyanovski, Arthur" , "Dagan, Noam" , "Agroskin, Shay" , "Itzko, Shahar" , "Abboud, Osama" Subject: [PATCH V1 net-next 1/5] net: ena: Register ena device to devlink Date: Sun, 8 Jan 2023 10:35:29 +0000 Message-ID: <20230108103533.10104-2-darinzon@amazon.com> X-Mailer: git-send-email 2.38.1 In-Reply-To: <20230108103533.10104-1-darinzon@amazon.com> References: <20230108103533.10104-1-darinzon@amazon.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: kuba@kernel.org This patch registers ena as a device that supports devlink. This makes it listed when running $ devlink dev show The patch lands the base upon which the driver's devlink callbacks will be added. Signed-off-by: Shay Agroskin Signed-off-by: David Arinzon --- drivers/net/ethernet/amazon/Kconfig | 1 + drivers/net/ethernet/amazon/ena/Makefile | 2 +- drivers/net/ethernet/amazon/ena/ena_devlink.c | 42 +++++++++++++++++++ drivers/net/ethernet/amazon/ena/ena_devlink.h | 20 +++++++++ drivers/net/ethernet/amazon/ena/ena_netdev.c | 24 ++++++++++- drivers/net/ethernet/amazon/ena/ena_netdev.h | 2 + 6 files changed, 88 insertions(+), 3 deletions(-) create mode 100644 drivers/net/ethernet/amazon/ena/ena_devlink.c create mode 100644 drivers/net/ethernet/amazon/ena/ena_devlink.h diff --git a/drivers/net/ethernet/amazon/Kconfig b/drivers/net/ethernet/amazon/Kconfig index c37fa393b99e..8f1b3302a89e 100644 --- a/drivers/net/ethernet/amazon/Kconfig +++ b/drivers/net/ethernet/amazon/Kconfig @@ -18,6 +18,7 @@ if NET_VENDOR_AMAZON config ENA_ETHERNET tristate "Elastic Network Adapter (ENA) support" + select NET_DEVLINK depends on PCI_MSI && !CPU_BIG_ENDIAN select DIMLIB help diff --git a/drivers/net/ethernet/amazon/ena/Makefile b/drivers/net/ethernet/amazon/ena/Makefile index f1f752a8f7bb..34abcedd9132 100644 --- a/drivers/net/ethernet/amazon/ena/Makefile +++ b/drivers/net/ethernet/amazon/ena/Makefile @@ -5,4 +5,4 @@ obj-$(CONFIG_ENA_ETHERNET) += ena.o -ena-y := ena_netdev.o ena_com.o ena_eth_com.o ena_ethtool.o +ena-y := ena_netdev.o ena_com.o ena_eth_com.o ena_ethtool.o ena_devlink.o diff --git a/drivers/net/ethernet/amazon/ena/ena_devlink.c b/drivers/net/ethernet/amazon/ena/ena_devlink.c new file mode 100644 index 000000000000..6897d60d8376 --- /dev/null +++ b/drivers/net/ethernet/amazon/ena/ena_devlink.c @@ -0,0 +1,42 @@ +// SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB +/* + * Copyright 2015-2021 Amazon.com, Inc. or its affiliates. All rights reserved. + */ + +#include "linux/pci.h" + +#include "ena_devlink.h" + +static const struct devlink_ops ena_devlink_ops = {}; + +struct devlink *ena_devlink_alloc(struct ena_adapter *adapter) +{ + struct device *dev = &adapter->pdev->dev; + struct devlink *devlink; + + devlink = devlink_alloc(&ena_devlink_ops, sizeof(struct ena_adapter *), dev); + if (!devlink) { + netdev_err(adapter->netdev, "Failed to allocate devlink struct\n"); + return NULL; + } + + ENA_DEVLINK_PRIV(devlink) = adapter; + adapter->devlink = devlink; + + return devlink; +} + +void ena_devlink_free(struct devlink *devlink) +{ + devlink_free(devlink); +} + +void ena_devlink_register(struct devlink *devlink, struct device *dev) +{ + devlink_register(devlink); +} + +void ena_devlink_unregister(struct devlink *devlink) +{ + devlink_unregister(devlink); +} diff --git a/drivers/net/ethernet/amazon/ena/ena_devlink.h b/drivers/net/ethernet/amazon/ena/ena_devlink.h new file mode 100644 index 000000000000..6f737884b850 --- /dev/null +++ b/drivers/net/ethernet/amazon/ena/ena_devlink.h @@ -0,0 +1,20 @@ +/* SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB */ +/* + * Copyright 2015-2021 Amazon.com, Inc. or its affiliates. All rights reserved. + */ + +#ifndef DEVLINK_H +#define DEVLINK_H + +#include "ena_netdev.h" +#include + +#define ENA_DEVLINK_PRIV(devlink) \ + (*(struct ena_adapter **)devlink_priv(devlink)) + +struct devlink *ena_devlink_alloc(struct ena_adapter *adapter); +void ena_devlink_free(struct devlink *devlink); +void ena_devlink_register(struct devlink *devlink, struct device *dev); +void ena_devlink_unregister(struct devlink *devlink); + +#endif /* DEVLINK_H */ diff --git a/drivers/net/ethernet/amazon/ena/ena_netdev.c b/drivers/net/ethernet/amazon/ena/ena_netdev.c index e8ad5ea31aff..ce79a0c42e6a 100644 --- a/drivers/net/ethernet/amazon/ena/ena_netdev.c +++ b/drivers/net/ethernet/amazon/ena/ena_netdev.c @@ -22,6 +22,8 @@ #include #include "ena_pci_id_tbl.h" +#include "ena_devlink.h" + MODULE_AUTHOR("Amazon.com, Inc. or its affiliates"); MODULE_DESCRIPTION(DEVICE_NAME); MODULE_LICENSE("GPL"); @@ -4243,6 +4245,7 @@ static int ena_probe(struct pci_dev *pdev, const struct pci_device_id *ent) struct ena_adapter *adapter; struct net_device *netdev; static int adapters_found; + struct devlink *devlink; u32 max_num_io_queues; bool wd_state; int bars, rc; @@ -4308,12 +4311,18 @@ static int ena_probe(struct pci_dev *pdev, const struct pci_device_id *ent) pci_set_drvdata(pdev, adapter); - rc = ena_device_init(ena_dev, pdev, &get_feat_ctx, &wd_state); + devlink = ena_devlink_alloc(adapter); + if (!devlink) { + netdev_err(netdev, "ena_devlink_alloc failed\n"); + goto err_netdev_destroy; + } + + rc = ena_device_init(adapter, pdev, &get_feat_ctx, &wd_state); if (rc) { dev_err(&pdev->dev, "ENA device init failed\n"); if (rc == -ETIME) rc = -EPROBE_DEFER; - goto err_netdev_destroy; + goto err_devlink_destroy; } rc = ena_map_llq_mem_bar(pdev, ena_dev, bars); @@ -4362,6 +4371,7 @@ static int ena_probe(struct pci_dev *pdev, const struct pci_device_id *ent) "Failed to query interrupt moderation feature\n"); goto err_device_destroy; } + ena_init_io_rings(adapter, 0, adapter->xdp_num_queues + @@ -4420,6 +4430,8 @@ static int ena_probe(struct pci_dev *pdev, const struct pci_device_id *ent) adapters_found++; + ena_devlink_register(devlink, &pdev->dev); + return 0; err_rss: @@ -4436,6 +4448,8 @@ static int ena_probe(struct pci_dev *pdev, const struct pci_device_id *ent) err_device_destroy: ena_com_delete_host_info(ena_dev); ena_com_admin_destroy(ena_dev); +err_devlink_destroy: + ena_devlink_free(devlink); err_netdev_destroy: free_netdev(netdev); err_free_region: @@ -4462,10 +4476,15 @@ static void __ena_shutoff(struct pci_dev *pdev, bool shutdown) struct ena_adapter *adapter = pci_get_drvdata(pdev); struct ena_com_dev *ena_dev; struct net_device *netdev; + struct devlink *devlink; ena_dev = adapter->ena_dev; netdev = adapter->netdev; + devlink = adapter->devlink; + ena_devlink_unregister(devlink); + ena_devlink_free(devlink); + #ifdef CONFIG_RFS_ACCEL if ((adapter->msix_vecs >= 1) && (netdev->rx_cpu_rmap)) { free_irq_cpu_rmap(netdev->rx_cpu_rmap); @@ -4482,6 +4501,7 @@ static void __ena_shutoff(struct pci_dev *pdev, bool shutdown) rtnl_lock(); /* lock released inside the below if-else block */ adapter->reset_reason = ENA_REGS_RESET_SHUTDOWN; ena_destroy_device(adapter, true); + if (shutdown) { netif_device_detach(netdev); dev_close(netdev); diff --git a/drivers/net/ethernet/amazon/ena/ena_netdev.h b/drivers/net/ethernet/amazon/ena/ena_netdev.h index 2cb141079474..c6132aa229df 100644 --- a/drivers/net/ethernet/amazon/ena/ena_netdev.h +++ b/drivers/net/ethernet/amazon/ena/ena_netdev.h @@ -313,6 +313,8 @@ struct ena_adapter { struct net_device *netdev; struct pci_dev *pdev; + struct devlink *devlink; + /* rx packets that shorter that this len will be copied to the skb * header */ From patchwork Sun Jan 8 10:35:30 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Arinzon, David" X-Patchwork-Id: 13092460 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A19FBC53210 for ; Sun, 8 Jan 2023 10:36:09 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232711AbjAHKgI (ORCPT ); Sun, 8 Jan 2023 05:36:08 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35852 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233000AbjAHKgF (ORCPT ); Sun, 8 Jan 2023 05:36:05 -0500 Received: from smtp-fw-9103.amazon.com (smtp-fw-9103.amazon.com [207.171.188.200]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7177EE0C2 for ; Sun, 8 Jan 2023 02:36:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1673174163; x=1704710163; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=4z/K6HIGmMlECX3NMbZs5X48Um0ZTGBuE68PJlxE9/k=; b=lAajQOipOrCDBSfenfquRMXQpA0PAQsDmeqV/tYf6Id4vt4Be3bv4xOF fA/acsc4KGwyGF1OhyuQ7yVkQcRnXo6A6TzEqoxhi6mcQ2OUkBMEgp720 xXHrvoXdo+VUez77q+My1awQD1fr2oNoXMicc+6AXnFnNdhTetm3e3V+P o=; X-IronPort-AV: E=Sophos;i="5.96,310,1665446400"; d="scan'208";a="1090394641" Received: from pdx4-co-svc-p1-lb2-vlan3.amazon.com (HELO email-inbound-relay-iad-1d-m6i4x-25ac6bd5.us-east-1.amazon.com) ([10.25.36.214]) by smtp-border-fw-9103.sea19.amazon.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 Jan 2023 10:35:58 +0000 Received: from EX13MTAUWC001.ant.amazon.com (iad12-ws-svc-p26-lb9-vlan2.iad.amazon.com [10.40.163.34]) by email-inbound-relay-iad-1d-m6i4x-25ac6bd5.us-east-1.amazon.com (Postfix) with ESMTPS id AFDEF42BFC; Sun, 8 Jan 2023 10:35:56 +0000 (UTC) Received: from EX19D002UWC003.ant.amazon.com (10.13.138.183) by EX13MTAUWC001.ant.amazon.com (10.43.162.135) with Microsoft SMTP Server (TLS) id 15.0.1497.42; Sun, 8 Jan 2023 10:35:46 +0000 Received: from EX13MTAUWA001.ant.amazon.com (10.43.160.58) by EX19D002UWC003.ant.amazon.com (10.13.138.183) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA) id 15.2.1118.7; Sun, 8 Jan 2023 10:35:46 +0000 Received: from dev-dsk-darinzon-1c-05962a8d.eu-west-1.amazon.com (172.19.80.187) by mail-relay.amazon.com (10.43.160.118) with Microsoft SMTP Server id 15.0.1497.42 via Frontend Transport; Sun, 8 Jan 2023 10:35:44 +0000 From: David Arinzon To: David Miller , Jakub Kicinski , CC: David Arinzon , "Machulsky, Zorik" , "Matushevsky, Alexander" , Saeed Bshara , "Bshara, Nafea" , "Saidi, Ali" , "Kiyanovski, Arthur" , "Dagan, Noam" , "Agroskin, Shay" , "Itzko, Shahar" , "Abboud, Osama" Subject: [PATCH V1 net-next 2/5] net: ena: Add devlink reload functionality Date: Sun, 8 Jan 2023 10:35:30 +0000 Message-ID: <20230108103533.10104-3-darinzon@amazon.com> X-Mailer: git-send-email 2.38.1 In-Reply-To: <20230108103533.10104-1-darinzon@amazon.com> References: <20230108103533.10104-1-darinzon@amazon.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: kuba@kernel.org This system allows to reload most of the driver's resources. This functionality is added to our driver to apply configurations which require a reset (such as changing LLQ entry size). For the implementation of reload functionality, the driver performs the same sequence that device reset performs with a few exceptions: - The reset occurs immediately rather than setting a reset flag which would cause the timer routine to trigger the reset. This is done to provide a smoother user experience, which makes sure the reset operation is done by the time the 'devlink reload' command returns. - Destruction of driver resources (using ena_destroy_device()) and their re-initialization (using ena_restore_device()) is done without holding the rtnl_lock() throughout the 'devlink reload' execution, but rather with holding it for *each* of the operations separately. Signed-off-by: Shay Agroskin Signed-off-by: David Arinzon --- drivers/net/ethernet/amazon/ena/ena_devlink.c | 73 ++++++++++++++++++- drivers/net/ethernet/amazon/ena/ena_netdev.c | 6 +- drivers/net/ethernet/amazon/ena/ena_netdev.h | 3 + 3 files changed, 77 insertions(+), 5 deletions(-) diff --git a/drivers/net/ethernet/amazon/ena/ena_devlink.c b/drivers/net/ethernet/amazon/ena/ena_devlink.c index 6897d60d8376..2568ade34c2a 100644 --- a/drivers/net/ethernet/amazon/ena/ena_devlink.c +++ b/drivers/net/ethernet/amazon/ena/ena_devlink.c @@ -7,7 +7,76 @@ #include "ena_devlink.h" -static const struct devlink_ops ena_devlink_ops = {}; +static int ena_devlink_reload_down(struct devlink *devlink, + bool netns_change, + enum devlink_reload_action action, + enum devlink_reload_limit limit, + struct netlink_ext_ack *extack) +{ + struct ena_adapter *adapter = ENA_DEVLINK_PRIV(devlink); + + if (netns_change) { + NL_SET_ERR_MSG_MOD(extack, "Namespace change is not supported"); + return -EOPNOTSUPP; + } + + if (action != DEVLINK_RELOAD_ACTION_DRIVER_REINIT) { + NL_SET_ERR_MSG_MOD(extack, "Action is not supported"); + return -EOPNOTSUPP; + } + + if (limit != DEVLINK_RELOAD_LIMIT_UNSPEC) { + NL_SET_ERR_MSG_MOD(extack, "Driver reload doesn't support limitations"); + return -EOPNOTSUPP; + } + + rtnl_lock(); + ena_destroy_device(adapter, false); + rtnl_unlock(); + + return 0; +} + +static int ena_devlink_reload_up(struct devlink *devlink, + enum devlink_reload_action action, + enum devlink_reload_limit limit, + u32 *actions_performed, + struct netlink_ext_ack *extack) +{ + struct ena_adapter *adapter = ENA_DEVLINK_PRIV(devlink); + int err = 0; + + if (action != DEVLINK_RELOAD_ACTION_DRIVER_REINIT) { + NL_SET_ERR_MSG_MOD(extack, "Action is not supported"); + return -EOPNOTSUPP; + } + + if (limit != DEVLINK_RELOAD_LIMIT_UNSPEC) { + NL_SET_ERR_MSG_MOD(extack, "Driver reload doesn't support limitations"); + return -EOPNOTSUPP; + } + + rtnl_lock(); + /* Check that no other routine initialized the device (e.g. + * ena_fw_reset_device()). Also we're under devlink_mutex here, + * so devlink isn't freed under our feet. + */ + if (!test_bit(ENA_FLAG_DEVICE_RUNNING, &adapter->flags)) + err = ena_restore_device(adapter); + + rtnl_unlock(); + + if (!err) + *actions_performed = BIT(DEVLINK_RELOAD_ACTION_DRIVER_REINIT); + + return err; +} + +static const struct devlink_ops ena_devlink_ops = { + .reload_actions = BIT(DEVLINK_RELOAD_ACTION_DRIVER_REINIT), + .reload_down = ena_devlink_reload_down, + .reload_up = ena_devlink_reload_up, +}; struct devlink *ena_devlink_alloc(struct ena_adapter *adapter) { @@ -20,6 +89,8 @@ struct devlink *ena_devlink_alloc(struct ena_adapter *adapter) return NULL; } + devlink_set_features(devlink, DEVLINK_F_RELOAD); + ENA_DEVLINK_PRIV(devlink) = adapter; adapter->devlink = devlink; diff --git a/drivers/net/ethernet/amazon/ena/ena_netdev.c b/drivers/net/ethernet/amazon/ena/ena_netdev.c index ce79a0c42e6a..a42db781472c 100644 --- a/drivers/net/ethernet/amazon/ena/ena_netdev.c +++ b/drivers/net/ethernet/amazon/ena/ena_netdev.c @@ -44,8 +44,6 @@ MODULE_DEVICE_TABLE(pci, ena_pci_tbl); static int ena_rss_init_default(struct ena_adapter *adapter); static void check_for_admin_com_state(struct ena_adapter *adapter); -static void ena_destroy_device(struct ena_adapter *adapter, bool graceful); -static int ena_restore_device(struct ena_adapter *adapter); static void ena_init_io_rings(struct ena_adapter *adapter, int first_index, int count); @@ -3587,7 +3585,7 @@ static int ena_enable_msix_and_set_admin_interrupts(struct ena_adapter *adapter) return rc; } -static void ena_destroy_device(struct ena_adapter *adapter, bool graceful) +void ena_destroy_device(struct ena_adapter *adapter, bool graceful) { struct net_device *netdev = adapter->netdev; struct ena_com_dev *ena_dev = adapter->ena_dev; @@ -3633,7 +3631,7 @@ static void ena_destroy_device(struct ena_adapter *adapter, bool graceful) clear_bit(ENA_FLAG_DEVICE_RUNNING, &adapter->flags); } -static int ena_restore_device(struct ena_adapter *adapter) +int ena_restore_device(struct ena_adapter *adapter) { struct ena_com_dev_get_features_ctx get_feat_ctx; struct ena_com_dev *ena_dev = adapter->ena_dev; diff --git a/drivers/net/ethernet/amazon/ena/ena_netdev.h b/drivers/net/ethernet/amazon/ena/ena_netdev.h index c6132aa229df..244c80af6974 100644 --- a/drivers/net/ethernet/amazon/ena/ena_netdev.h +++ b/drivers/net/ethernet/amazon/ena/ena_netdev.h @@ -452,4 +452,7 @@ static inline enum ena_xdp_errors_t ena_xdp_allowed(struct ena_adapter *adapter) return rc; } +void ena_destroy_device(struct ena_adapter *adapter, bool graceful); +int ena_restore_device(struct ena_adapter *adapter); + #endif /* !(ENA_H) */ From patchwork Sun Jan 8 10:35:31 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Arinzon, David" X-Patchwork-Id: 13092463 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 37CFEC54EBC for ; Sun, 8 Jan 2023 10:36:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233000AbjAHKgf (ORCPT ); Sun, 8 Jan 2023 05:36:35 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36082 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232985AbjAHKgU (ORCPT ); Sun, 8 Jan 2023 05:36:20 -0500 Received: from smtp-fw-6002.amazon.com (smtp-fw-6002.amazon.com [52.95.49.90]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3C1C662D2 for ; Sun, 8 Jan 2023 02:36:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1673174178; x=1704710178; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=dQcm+cOOxJBC7N26Urrc2pvccfWbsgrlFnnvrKfUAbA=; b=m1oVB1O6nooPCfSB1jRgYMW9h1kMmx6NRMdvrAQaIXw24z4lctCAoRx8 JYRhBm+KAueq1VRT5kNOc0GdQlR24ldBeGNnUjgV0i4/aKjAdHW7cAZFX ATNViAgD5wjLpkhSk5Ubw7BSiF1PmpAqU6yS0f5sTlpFU3AqqhGzIoHUu I=; X-IronPort-AV: E=Sophos;i="5.96,310,1665446400"; d="scan'208";a="284419000" Received: from iad12-co-svc-p1-lb1-vlan3.amazon.com (HELO email-inbound-relay-iad-1a-m6i4x-9fe6ad2f.us-east-1.amazon.com) ([10.43.8.6]) by smtp-border-fw-6002.iad6.amazon.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 Jan 2023 10:36:17 +0000 Received: from EX13MTAUWB001.ant.amazon.com (iad12-ws-svc-p26-lb9-vlan3.iad.amazon.com [10.40.163.38]) by email-inbound-relay-iad-1a-m6i4x-9fe6ad2f.us-east-1.amazon.com (Postfix) with ESMTPS id 70D5882285; Sun, 8 Jan 2023 10:36:14 +0000 (UTC) Received: from EX19D002UWA002.ant.amazon.com (10.13.138.246) by EX13MTAUWB001.ant.amazon.com (10.43.161.207) with Microsoft SMTP Server (TLS) id 15.0.1497.42; Sun, 8 Jan 2023 10:35:50 +0000 Received: from EX13MTAUWA001.ant.amazon.com (10.43.160.58) by EX19D002UWA002.ant.amazon.com (10.13.138.246) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA) id 15.2.1118.7; Sun, 8 Jan 2023 10:35:50 +0000 Received: from dev-dsk-darinzon-1c-05962a8d.eu-west-1.amazon.com (172.19.80.187) by mail-relay.amazon.com (10.43.160.118) with Microsoft SMTP Server id 15.0.1497.42 via Frontend Transport; Sun, 8 Jan 2023 10:35:48 +0000 From: David Arinzon To: David Miller , Jakub Kicinski , CC: David Arinzon , "Machulsky, Zorik" , "Matushevsky, Alexander" , Saeed Bshara , "Bshara, Nafea" , "Saidi, Ali" , "Kiyanovski, Arthur" , "Dagan, Noam" , "Agroskin, Shay" , "Itzko, Shahar" , "Abboud, Osama" Subject: [PATCH V1 net-next 3/5] net: ena: Configure large LLQ using devlink params Date: Sun, 8 Jan 2023 10:35:31 +0000 Message-ID: <20230108103533.10104-4-darinzon@amazon.com> X-Mailer: git-send-email 2.38.1 In-Reply-To: <20230108103533.10104-1-darinzon@amazon.com> References: <20230108103533.10104-1-darinzon@amazon.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: kuba@kernel.org This change introduces devlink params infrastructure to the ena driver as well as the ability to enable large LLQ configuration through the infrastructure. Default LLQ entry size is 128 bytes. 128 bytes entry size allows for a maximum of 96 bytes of packet header size which sometimes is not enough (e.g. when using tunneling). Increasing LLQ entry size to 256 bytes, by enabling large LLQ through devlink, allows a maximum header size of 224 bytes. This comes with the penalty of reducing the number of LLQ entries in the TX queue by 2 (i.e. from 1024 to 512). Signed-off-by: Shay Agroskin Signed-off-by: David Arinzon --- drivers/net/ethernet/amazon/ena/ena_devlink.c | 106 +++++++++++++++++- drivers/net/ethernet/amazon/ena/ena_devlink.h | 2 + drivers/net/ethernet/amazon/ena/ena_netdev.c | 47 +++++++- drivers/net/ethernet/amazon/ena/ena_netdev.h | 8 ++ 4 files changed, 157 insertions(+), 6 deletions(-) diff --git a/drivers/net/ethernet/amazon/ena/ena_devlink.c b/drivers/net/ethernet/amazon/ena/ena_devlink.c index 2568ade34c2a..25194c365299 100644 --- a/drivers/net/ethernet/amazon/ena/ena_devlink.c +++ b/drivers/net/ethernet/amazon/ena/ena_devlink.c @@ -7,6 +7,72 @@ #include "ena_devlink.h" +static int ena_devlink_llq_header_validate(struct devlink *devlink, u32 id, + union devlink_param_value val, + struct netlink_ext_ack *extack); + +enum ena_devlink_param_id { + ENA_DEVLINK_PARAM_ID_BASE = DEVLINK_PARAM_GENERIC_ID_MAX, + ENA_DEVLINK_PARAM_ID_LLQ_HEADER_SIZE, +}; + +static const struct devlink_param ena_devlink_params[] = { + DEVLINK_PARAM_DRIVER(ENA_DEVLINK_PARAM_ID_LLQ_HEADER_SIZE, + "large_llq_header", DEVLINK_PARAM_TYPE_BOOL, + BIT(DEVLINK_PARAM_CMODE_DRIVERINIT), + NULL, NULL, ena_devlink_llq_header_validate), +}; + +static int ena_devlink_llq_header_validate(struct devlink *devlink, u32 id, + union devlink_param_value val, + struct netlink_ext_ack *extack) +{ + struct ena_adapter *adapter = ENA_DEVLINK_PRIV(devlink); + bool value = val.vbool; + + if (!value) + return 0; + + if (adapter->ena_dev->tx_mem_queue_type == ENA_ADMIN_PLACEMENT_POLICY_HOST) { + NL_SET_ERR_MSG_MOD(extack, "Instance doesn't support LLQ"); + return -EOPNOTSUPP; + } + + if (!adapter->large_llq_header_supported) { + NL_SET_ERR_MSG_MOD(extack, "Instance doesn't support large LLQ"); + return -EOPNOTSUPP; + } + + return 0; +} + +void ena_devlink_params_get(struct devlink *devlink) +{ + struct ena_adapter *adapter = ENA_DEVLINK_PRIV(devlink); + union devlink_param_value val; + int err; + + err = devlink_param_driverinit_value_get(devlink, + ENA_DEVLINK_PARAM_ID_LLQ_HEADER_SIZE, + &val); + if (err) { + netdev_err(adapter->netdev, "Failed to query LLQ header size param\n"); + return; + } + + adapter->large_llq_header_enabled = val.vbool; +} + +void ena_devlink_disable_large_llq_header_param(struct devlink *devlink) +{ + union devlink_param_value value; + + value.vbool = false; + devlink_param_driverinit_value_set(devlink, + ENA_DEVLINK_PARAM_ID_LLQ_HEADER_SIZE, + value); +} + static int ena_devlink_reload_down(struct devlink *devlink, bool netns_change, enum devlink_reload_action action, @@ -78,6 +144,29 @@ static const struct devlink_ops ena_devlink_ops = { .reload_up = ena_devlink_reload_up, }; +static int ena_devlink_configure_params(struct devlink *devlink) +{ + struct ena_adapter *adapter = ENA_DEVLINK_PRIV(devlink); + union devlink_param_value value; + int rc; + + rc = devlink_params_register(devlink, ena_devlink_params, + ARRAY_SIZE(ena_devlink_params)); + if (rc) { + netdev_err(adapter->netdev, "Failed to register devlink params\n"); + return rc; + } + + value.vbool = adapter->large_llq_header_enabled; + devlink_param_driverinit_value_set(devlink, + ENA_DEVLINK_PARAM_ID_LLQ_HEADER_SIZE, + value); + + devlink_set_features(devlink, DEVLINK_F_RELOAD); + + return 0; +} + struct devlink *ena_devlink_alloc(struct ena_adapter *adapter) { struct device *dev = &adapter->pdev->dev; @@ -89,16 +178,29 @@ struct devlink *ena_devlink_alloc(struct ena_adapter *adapter) return NULL; } - devlink_set_features(devlink, DEVLINK_F_RELOAD); - ENA_DEVLINK_PRIV(devlink) = adapter; adapter->devlink = devlink; + if (ena_devlink_configure_params(devlink)) + goto free_devlink; + return devlink; +free_devlink: + devlink_free(devlink); + + return NULL; +} + +static void ena_devlink_configure_params_clean(struct devlink *devlink) +{ + devlink_params_unregister(devlink, ena_devlink_params, + ARRAY_SIZE(ena_devlink_params)); } void ena_devlink_free(struct devlink *devlink) { + ena_devlink_configure_params_clean(devlink); + devlink_free(devlink); } diff --git a/drivers/net/ethernet/amazon/ena/ena_devlink.h b/drivers/net/ethernet/amazon/ena/ena_devlink.h index 6f737884b850..9db6038ecd62 100644 --- a/drivers/net/ethernet/amazon/ena/ena_devlink.h +++ b/drivers/net/ethernet/amazon/ena/ena_devlink.h @@ -16,5 +16,7 @@ struct devlink *ena_devlink_alloc(struct ena_adapter *adapter); void ena_devlink_free(struct devlink *devlink); void ena_devlink_register(struct devlink *devlink, struct device *dev); void ena_devlink_unregister(struct devlink *devlink); +void ena_devlink_params_get(struct devlink *devlink); +void ena_devlink_disable_large_llq_header_param(struct devlink *devlink); #endif /* DEVLINK_H */ diff --git a/drivers/net/ethernet/amazon/ena/ena_netdev.c b/drivers/net/ethernet/amazon/ena/ena_netdev.c index a42db781472c..24b95765bb04 100644 --- a/drivers/net/ethernet/amazon/ena/ena_netdev.c +++ b/drivers/net/ethernet/amazon/ena/ena_netdev.c @@ -3385,13 +3385,30 @@ static int ena_device_validate_params(struct ena_adapter *adapter, return 0; } -static void set_default_llq_configurations(struct ena_llq_configurations *llq_config) +static void set_default_llq_configurations(struct ena_adapter *adapter, + struct ena_llq_configurations *llq_config, + struct ena_admin_feature_llq_desc *llq) { + struct ena_com_dev *ena_dev = adapter->ena_dev; + llq_config->llq_header_location = ENA_ADMIN_INLINE_HEADER; llq_config->llq_stride_ctrl = ENA_ADMIN_MULTIPLE_DESCS_PER_ENTRY; llq_config->llq_num_decs_before_header = ENA_ADMIN_LLQ_NUM_DESCS_BEFORE_HEADER_2; - llq_config->llq_ring_entry_size = ENA_ADMIN_LIST_ENTRY_SIZE_128B; - llq_config->llq_ring_entry_size_value = 128; + + adapter->large_llq_header_supported = + !!(ena_dev->supported_features & (1 << ENA_ADMIN_LLQ)); + adapter->large_llq_header_supported &= + !!(llq->entry_size_ctrl_supported & + ENA_ADMIN_LIST_ENTRY_SIZE_256B); + + if ((llq->entry_size_ctrl_supported & ENA_ADMIN_LIST_ENTRY_SIZE_256B) && + adapter->large_llq_header_enabled) { + llq_config->llq_ring_entry_size = ENA_ADMIN_LIST_ENTRY_SIZE_256B; + llq_config->llq_ring_entry_size_value = 256; + } else { + llq_config->llq_ring_entry_size = ENA_ADMIN_LIST_ENTRY_SIZE_128B; + llq_config->llq_ring_entry_size_value = 128; + } } static int ena_set_queues_placement_policy(struct pci_dev *pdev, @@ -3493,6 +3510,8 @@ static int ena_device_init(struct ena_com_dev *ena_dev, struct pci_dev *pdev, goto err_mmio_read_less; } + ena_devlink_params_get(adapter->devlink); + /* ENA admin level init */ rc = ena_com_admin_init(ena_dev, &aenq_handlers); if (rc) { @@ -3533,7 +3552,7 @@ static int ena_device_init(struct ena_com_dev *ena_dev, struct pci_dev *pdev, *wd_state = !!(aenq_groups & BIT(ENA_ADMIN_KEEP_ALIVE)); - set_default_llq_configurations(&llq_config); + set_default_llq_configurations(adapter, &llq_config, &get_feat_ctx->llq); rc = ena_set_queues_placement_policy(pdev, ena_dev, &get_feat_ctx->llq, &llq_config); @@ -4212,6 +4231,26 @@ static void ena_calc_io_queue_size(struct ena_adapter *adapter, max_tx_queue_size = rounddown_pow_of_two(max_tx_queue_size); max_rx_queue_size = rounddown_pow_of_two(max_rx_queue_size); + /* When forcing large headers, we multiply the entry size by 2, + * and therefore divide the queue size by 2, leaving the amount + * of memory used by the queues unchanged. + */ + if (adapter->large_llq_header_enabled) { + if ((llq->entry_size_ctrl_supported & ENA_ADMIN_LIST_ENTRY_SIZE_256B) && + (ena_dev->tx_mem_queue_type == ENA_ADMIN_PLACEMENT_POLICY_DEV)) { + max_tx_queue_size /= 2; + dev_info(&adapter->pdev->dev, + "Forcing large headers and decreasing maximum TX queue size to %d\n", + max_tx_queue_size); + } else { + dev_err(&adapter->pdev->dev, + "Forcing large headers failed: LLQ is disabled or device does not support large headers\n"); + + adapter->large_llq_header_enabled = false; + ena_devlink_disable_large_llq_header_param(adapter->devlink); + } + } + tx_queue_size = clamp_val(tx_queue_size, ENA_MIN_RING_SIZE, max_tx_queue_size); rx_queue_size = clamp_val(rx_queue_size, ENA_MIN_RING_SIZE, diff --git a/drivers/net/ethernet/amazon/ena/ena_netdev.h b/drivers/net/ethernet/amazon/ena/ena_netdev.h index 244c80af6974..b6faf48373d2 100644 --- a/drivers/net/ethernet/amazon/ena/ena_netdev.h +++ b/drivers/net/ethernet/amazon/ena/ena_netdev.h @@ -336,6 +336,14 @@ struct ena_adapter { u32 msg_enable; + /* The flag is used for two purposes: + * 1. Indicates that large LLQ has been requested. + * 2. Indicates whether large LLQ is set or not after device + * initialization / configuration. + */ + bool large_llq_header_enabled; + bool large_llq_header_supported; + u16 max_tx_sgl_size; u16 max_rx_sgl_size; From patchwork Sun Jan 8 10:35:32 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Arinzon, David" X-Patchwork-Id: 13092461 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7D443C54EBC for ; Sun, 8 Jan 2023 10:36:11 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232838AbjAHKgJ (ORCPT ); Sun, 8 Jan 2023 05:36:09 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35854 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233005AbjAHKgF (ORCPT ); Sun, 8 Jan 2023 05:36:05 -0500 Received: from smtp-fw-9102.amazon.com (smtp-fw-9102.amazon.com [207.171.184.29]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 83AFCE0C3 for ; Sun, 8 Jan 2023 02:36:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1673174164; x=1704710164; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=hPgABhr7ie7jATqagQ7xcHqNsDrm8OL8O6BdbfdvoFo=; b=CtZFlr17qzX0YiX4zPBPCfu8GCwag3P6qkaTAqChRHJwyxAdgW+VYfVt e8HtTzML2ihnpgVMlfWOQjIoTdoMl/3nnh0yMUbt9Vw1QgKybPPvMgpUI UCMjSCTCWv1kzeGezX0diKTmN9p6j4hZPCHOI7/VLibC++zICXzKu5wWx 8=; X-IronPort-AV: E=Sophos;i="5.96,310,1665446400"; d="scan'208";a="297950532" Received: from pdx4-co-svc-p1-lb2-vlan2.amazon.com (HELO email-inbound-relay-iad-1a-m6i4x-47cc8a4c.us-east-1.amazon.com) ([10.25.36.210]) by smtp-border-fw-9102.sea19.amazon.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 Jan 2023 10:36:03 +0000 Received: from EX13MTAUWC001.ant.amazon.com (iad12-ws-svc-p26-lb9-vlan2.iad.amazon.com [10.40.163.34]) by email-inbound-relay-iad-1a-m6i4x-47cc8a4c.us-east-1.amazon.com (Postfix) with ESMTPS id DF8B3160C2D; Sun, 8 Jan 2023 10:36:01 +0000 (UTC) Received: from EX19D002UWC004.ant.amazon.com (10.13.138.186) by EX13MTAUWC001.ant.amazon.com (10.43.162.135) with Microsoft SMTP Server (TLS) id 15.0.1497.42; Sun, 8 Jan 2023 10:35:54 +0000 Received: from EX13MTAUWA001.ant.amazon.com (10.43.160.58) by EX19D002UWC004.ant.amazon.com (10.13.138.186) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA) id 15.2.1118.7; Sun, 8 Jan 2023 10:35:53 +0000 Received: from dev-dsk-darinzon-1c-05962a8d.eu-west-1.amazon.com (172.19.80.187) by mail-relay.amazon.com (10.43.160.118) with Microsoft SMTP Server id 15.0.1497.42 via Frontend Transport; Sun, 8 Jan 2023 10:35:51 +0000 From: David Arinzon To: David Miller , Jakub Kicinski , CC: David Arinzon , "Machulsky, Zorik" , "Matushevsky, Alexander" , Saeed Bshara , "Bshara, Nafea" , "Saidi, Ali" , "Kiyanovski, Arthur" , "Dagan, Noam" , "Agroskin, Shay" , "Itzko, Shahar" , "Abboud, Osama" Subject: [PATCH V1 net-next 4/5] net: ena: Several changes to support large LLQ configuration Date: Sun, 8 Jan 2023 10:35:32 +0000 Message-ID: <20230108103533.10104-5-darinzon@amazon.com> X-Mailer: git-send-email 2.38.1 In-Reply-To: <20230108103533.10104-1-darinzon@amazon.com> References: <20230108103533.10104-1-darinzon@amazon.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: kuba@kernel.org 1. Allow queue related parameters to be recalculated upon every VF reset. 2. Handling of queues configured before ena_device_init() function is called. 3. Prevention of large LLQ configuration if LLQ is not supported. Signed-off-by: Shay Agroskin Signed-off-by: David Arinzon --- drivers/net/ethernet/amazon/ena/ena_netdev.c | 48 +++++++++++++------- 1 file changed, 31 insertions(+), 17 deletions(-) diff --git a/drivers/net/ethernet/amazon/ena/ena_netdev.c b/drivers/net/ethernet/amazon/ena/ena_netdev.c index 24b95765bb04..fac8c913de6b 100644 --- a/drivers/net/ethernet/amazon/ena/ena_netdev.c +++ b/drivers/net/ethernet/amazon/ena/ena_netdev.c @@ -44,6 +44,8 @@ MODULE_DEVICE_TABLE(pci, ena_pci_tbl); static int ena_rss_init_default(struct ena_adapter *adapter); static void check_for_admin_com_state(struct ena_adapter *adapter); +static void ena_calc_io_queue_size(struct ena_adapter *adapter, + struct ena_com_dev_get_features_ctx *get_feat_ctx); static void ena_init_io_rings(struct ena_adapter *adapter, int first_index, int count); @@ -3427,6 +3429,13 @@ static int ena_set_queues_placement_policy(struct pci_dev *pdev, return 0; } + if (!ena_dev->mem_bar) { + netdev_err(ena_dev->net_device, + "LLQ is advertised as supported but device doesn't expose mem bar\n"); + ena_dev->tx_mem_queue_type = ENA_ADMIN_PLACEMENT_POLICY_HOST; + return 0; + } + rc = ena_com_config_dev_mode(ena_dev, llq, llq_default_configurations); if (unlikely(rc)) { dev_err(&pdev->dev, @@ -3442,15 +3451,8 @@ static int ena_map_llq_mem_bar(struct pci_dev *pdev, struct ena_com_dev *ena_dev { bool has_mem_bar = !!(bars & BIT(ENA_MEM_BAR)); - if (!has_mem_bar) { - if (ena_dev->tx_mem_queue_type == ENA_ADMIN_PLACEMENT_POLICY_DEV) { - dev_err(&pdev->dev, - "ENA device does not expose LLQ bar. Fallback to host mode policy.\n"); - ena_dev->tx_mem_queue_type = ENA_ADMIN_PLACEMENT_POLICY_HOST; - } - + if (!has_mem_bar) return 0; - } ena_dev->mem_bar = devm_ioremap_wc(&pdev->dev, pci_resource_start(pdev, ENA_MEM_BAR), @@ -3462,10 +3464,11 @@ static int ena_map_llq_mem_bar(struct pci_dev *pdev, struct ena_com_dev *ena_dev return 0; } -static int ena_device_init(struct ena_com_dev *ena_dev, struct pci_dev *pdev, +static int ena_device_init(struct ena_adapter *adapter, struct pci_dev *pdev, struct ena_com_dev_get_features_ctx *get_feat_ctx, bool *wd_state) { + struct ena_com_dev *ena_dev = adapter->ena_dev; struct ena_llq_configurations llq_config; struct device *dev = &pdev->dev; bool readless_supported; @@ -3561,6 +3564,8 @@ static int ena_device_init(struct ena_com_dev *ena_dev, struct pci_dev *pdev, goto err_admin_init; } + ena_calc_io_queue_size(adapter, get_feat_ctx); + return 0; err_admin_init: @@ -3659,7 +3664,7 @@ int ena_restore_device(struct ena_adapter *adapter) int rc; set_bit(ENA_FLAG_ONGOING_RESET, &adapter->flags); - rc = ena_device_init(ena_dev, adapter->pdev, &get_feat_ctx, &wd_state); + rc = ena_device_init(adapter, adapter->pdev, &get_feat_ctx, &wd_state); if (rc) { dev_err(&pdev->dev, "Can not initialize device\n"); goto err; @@ -4187,9 +4192,19 @@ static void ena_calc_io_queue_size(struct ena_adapter *adapter, struct ena_com_dev *ena_dev = adapter->ena_dev; u32 tx_queue_size = ENA_DEFAULT_RING_SIZE; u32 rx_queue_size = ENA_DEFAULT_RING_SIZE; + bool tx_configured, rx_configured; u32 max_tx_queue_size; u32 max_rx_queue_size; + /* If this function is called after driver load, the ring sizes have + * already been configured. Take it into account when recalculating ring + * size. + */ + tx_configured = !!adapter->tx_ring[0].ring_size; + rx_configured = !!adapter->rx_ring[0].ring_size; + tx_queue_size = tx_configured ? adapter->tx_ring[0].ring_size : tx_queue_size; + rx_queue_size = rx_configured ? adapter->rx_ring[0].ring_size : rx_queue_size; + if (ena_dev->supported_features & BIT(ENA_ADMIN_MAX_QUEUES_EXT)) { struct ena_admin_queue_ext_feature_fields *max_queue_ext = &get_feat_ctx->max_queue_ext.max_queue_ext; @@ -4354,6 +4369,12 @@ static int ena_probe(struct pci_dev *pdev, const struct pci_device_id *ent) goto err_netdev_destroy; } + rc = ena_map_llq_mem_bar(pdev, ena_dev, bars); + if (rc) { + dev_err(&pdev->dev, "ENA LLQ bar mapping failed\n"); + goto err_devlink_destroy; + } + rc = ena_device_init(adapter, pdev, &get_feat_ctx, &wd_state); if (rc) { dev_err(&pdev->dev, "ENA device init failed\n"); @@ -4362,12 +4383,6 @@ static int ena_probe(struct pci_dev *pdev, const struct pci_device_id *ent) goto err_devlink_destroy; } - rc = ena_map_llq_mem_bar(pdev, ena_dev, bars); - if (rc) { - dev_err(&pdev->dev, "ENA llq bar mapping failed\n"); - goto err_device_destroy; - } - /* Initial TX and RX interrupt delay. Assumes 1 usec granularity. * Updated during device initialization with the real granularity */ @@ -4375,7 +4390,6 @@ static int ena_probe(struct pci_dev *pdev, const struct pci_device_id *ent) ena_dev->intr_moder_rx_interval = ENA_INTR_INITIAL_RX_INTERVAL_USECS; ena_dev->intr_delay_resolution = ENA_DEFAULT_INTR_DELAY_RESOLUTION; max_num_io_queues = ena_calc_max_io_queue_num(pdev, ena_dev, &get_feat_ctx); - ena_calc_io_queue_size(adapter, &get_feat_ctx); if (unlikely(!max_num_io_queues)) { rc = -EFAULT; goto err_device_destroy; From patchwork Sun Jan 8 10:35:33 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Arinzon, David" X-Patchwork-Id: 13092462 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C3BB1C54EBE for ; Sun, 8 Jan 2023 10:36:13 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232895AbjAHKgK (ORCPT ); Sun, 8 Jan 2023 05:36:10 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35836 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231184AbjAHKgH (ORCPT ); Sun, 8 Jan 2023 05:36:07 -0500 Received: from smtp-fw-80006.amazon.com (smtp-fw-80006.amazon.com [99.78.197.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id EC25CE0C6 for ; Sun, 8 Jan 2023 02:36:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1673174166; x=1704710166; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=Tp8YrA/QmF9BGSvmkOyBb6k+v1Soq/Sj2/zR8dpdDGM=; b=KP2YBycWQJ8Ur1kPK0LrEDEvmLMNkcXISNj6pLPz5MEOwJbm1vf+SFan T4xVA0qmgJqkz+Q4dCqrURq+2qVI6jipcSTf+wTmMvFAySIik9/w1yDkC lXp+iQnmrmbzLHs/s0WhSDruW+ldOevMymFYIQxXouXJDb2b2LcD1ohOB g=; X-IronPort-AV: E=Sophos;i="5.96,310,1665446400"; d="scan'208";a="168893040" Received: from pdx4-co-svc-p1-lb2-vlan2.amazon.com (HELO email-inbound-relay-pdx-2b-m6i4x-ed19f671.us-west-2.amazon.com) ([10.25.36.210]) by smtp-border-fw-80006.pdx80.corp.amazon.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 Jan 2023 10:36:04 +0000 Received: from EX13MTAUWC002.ant.amazon.com (pdx1-ws-svc-p6-lb9-vlan3.pdx.amazon.com [10.236.137.198]) by email-inbound-relay-pdx-2b-m6i4x-ed19f671.us-west-2.amazon.com (Postfix) with ESMTPS id F312881155; Sun, 8 Jan 2023 10:36:02 +0000 (UTC) Received: from EX19D002UWC002.ant.amazon.com (10.13.138.166) by EX13MTAUWC002.ant.amazon.com (10.43.162.240) with Microsoft SMTP Server (TLS) id 15.0.1497.42; Sun, 8 Jan 2023 10:35:58 +0000 Received: from EX13MTAUWA001.ant.amazon.com (10.43.160.58) by EX19D002UWC002.ant.amazon.com (10.13.138.166) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA) id 15.2.1118.7; Sun, 8 Jan 2023 10:35:57 +0000 Received: from dev-dsk-darinzon-1c-05962a8d.eu-west-1.amazon.com (172.19.80.187) by mail-relay.amazon.com (10.43.160.118) with Microsoft SMTP Server id 15.0.1497.42 via Frontend Transport; Sun, 8 Jan 2023 10:35:55 +0000 From: David Arinzon To: David Miller , Jakub Kicinski , CC: David Arinzon , "Machulsky, Zorik" , "Matushevsky, Alexander" , Saeed Bshara , "Bshara, Nafea" , "Saidi, Ali" , "Kiyanovski, Arthur" , "Dagan, Noam" , "Agroskin, Shay" , "Itzko, Shahar" , "Abboud, Osama" Subject: [PATCH V1 net-next 5/5] net: ena: Add devlink documentation Date: Sun, 8 Jan 2023 10:35:33 +0000 Message-ID: <20230108103533.10104-6-darinzon@amazon.com> X-Mailer: git-send-email 2.38.1 In-Reply-To: <20230108103533.10104-1-darinzon@amazon.com> References: <20230108103533.10104-1-darinzon@amazon.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: kuba@kernel.org Update the documentation with a devlink section, the added files, as well as large LLQ enablement. Signed-off-by: Shay Agroskin Signed-off-by: David Arinzon --- .../device_drivers/ethernet/amazon/ena.rst | 30 +++++++++++++++++++ 1 file changed, 30 insertions(+) diff --git a/Documentation/networking/device_drivers/ethernet/amazon/ena.rst b/Documentation/networking/device_drivers/ethernet/amazon/ena.rst index 8bcb173e0353..1229732a8c91 100644 --- a/Documentation/networking/device_drivers/ethernet/amazon/ena.rst +++ b/Documentation/networking/device_drivers/ethernet/amazon/ena.rst @@ -53,6 +53,7 @@ ena_common_defs.h Common definitions for ena_com layer. ena_regs_defs.h Definition of ENA PCI memory-mapped (MMIO) registers. ena_netdev.[ch] Main Linux kernel driver. ena_ethtool.c ethtool callbacks. +ena_devlink.[ch] devlink files (see `devlink support`_ for more info) ena_pci_id_tbl.h Supported device IDs. ================= ====================================================== @@ -253,6 +254,35 @@ RSS - The user can provide a hash key, hash function, and configure the indirection table through `ethtool(8)`. +.. _`devlink support`: +DEVLINK SUPPORT +=============== +.. _`devlink`: https://www.kernel.org/doc/html/latest/networking/devlink/index.html + +`devlink`_ supports toggling LLQ entry size between the default 128 bytes and 256 +bytes. +A 128 bytes entry size allows for a maximum of 96 bytes of packet header size +which sometimes is not enough (e.g. when using tunneling). +Increasing LLQ entry size to 256 bytes, allows a maximum header size of 224 +bytes. This comes with the penalty of reducing the number of LLQ entries in the +TX queue by 2 (i.e. from 1024 to 512). + +The entry size can be toggled by enabling/disabling the large_llq_header devlink +param and reloading the driver to make it take effect, e.g. + +.. code-block:: shell + + sudo devlink dev param set pci/0000:00:06.0 name large_llq_header value true cmode driverinit + sudo devlink dev reload pci/0000:00:06.0 + +One way to verify that the TX queue entry size has indeed increased is to check +that the maximum TX queue depth is 512. This can be checked, for example, by +using: + +.. code-block:: shell + + ethtool -g [interface] + DATA PATH =========