From patchwork Tue Oct 17 07:42:46 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ido Schimmel X-Patchwork-Id: 13424696 X-Patchwork-Delegate: bhelgaas@google.com Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3E3DECDB474 for ; Tue, 17 Oct 2023 07:44:01 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234613AbjJQHoA (ORCPT ); Tue, 17 Oct 2023 03:44:00 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43202 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232134AbjJQHn7 (ORCPT ); Tue, 17 Oct 2023 03:43:59 -0400 Received: from NAM10-MW2-obe.outbound.protection.outlook.com (mail-mw2nam10on2059.outbound.protection.outlook.com [40.107.94.59]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0959B8F; Tue, 17 Oct 2023 00:43:58 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=ODYxErI3Cb/ZADnq9QJjXWtU0pTgJq5MVOPRKGAO9cwlgWkIQrN6gaBhG6I7z8GhcGCjNLevB/oC8ktVIfszPM7aKvvxOUJ1IE81pwZpatNthQYgVzBtTukkSiIDaBSazYlgIJwtY5/4uRT9AA7w7DagZ6ypblzCp6FfloJDJ+RzIZOQY+16g2dEQ6krTrIuNUtXlSzAaGV65lTvI0R2tbgn3QCyOj/IF4CWHeQQbXdYSj2fm2CI6Q2aDwtBnVBqBNqJiHsOk+XqdtAgRH1lsceCT821TNTCGk8ZCfuQuqt7hJdSWCXJh/uCkICBl9TwDj74KJ2rqVzCZvKmk7XksQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=6+UdztoCxdDVBnRnQBFgbG5vg+ieKEhrNnRjwC41O3Y=; b=fOV4skR6e1iDOKfndWd5VoUdstJ0zVeARaJbO9GKjTBeGUP0QLARwOaDqYKYxRRB/DYPXmnSs5473ZtWpvb4k2z+MfQS3c7glQsXkZtlHtrZPk+MbPXuiGUGeumcQlDJMJj+RkTwF2C0DuLKGTGPW3m0tBOXxg/H/M83A7J4L49NiUmrwWMHIScOMdCLIpHR/h8J9FX9PLlvT6CxtzwoV+BxBi9o9H+vFc39pnhvmrBvm6Po2GseCTSn1wOXWAJKq1RVtLPccFkwoU0lIlVLzNySLqzGGJtkhC7JSkB8ObswLEECNjr2rdTwX51X2PwxnBb4HNAniLZWAhGx0mpvyg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.160) smtp.rcpttodomain=vger.kernel.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=6+UdztoCxdDVBnRnQBFgbG5vg+ieKEhrNnRjwC41O3Y=; b=SBpiMCfOMQgjXKedeeJA0O/hGiIwYfu9dQfl436LB99aqpm4dJYkpIUAq02TxcmI80TinEvj2wBBcQ2AnA93vnz/x+jdaDiXsdht0wFNq7VYVMaH0EP8KdGCcSIKFkgVOVO0ztm5Q8AN/tmZgaqBNKtRnVrNiZb1S3v/Kwj+O7S7Oph00BKScV2WYCFlQWCAAsFOECWN7DaWE6XbADoiKOvpbKi/cfMGgNrNQQUo3p46su+QMWRM3jglehzzZuP1dhsLPUZfkvl8fUsOTBIwfFcT6ww0UKT7eKGwimDane5hY9WDp6qvB6QS3XvA+5ele2Wu41XZv0U6JOI6bQjkQg== Received: from BL1PR13CA0142.namprd13.prod.outlook.com (2603:10b6:208:2bb::27) by DM3PR12MB9288.namprd12.prod.outlook.com (2603:10b6:0:4a::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6886.36; Tue, 17 Oct 2023 07:43:56 +0000 Received: from BL6PEPF0001AB78.namprd02.prod.outlook.com (2603:10b6:208:2bb:cafe::60) by BL1PR13CA0142.outlook.office365.com (2603:10b6:208:2bb::27) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6907.18 via Frontend Transport; Tue, 17 Oct 2023 07:43:55 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.160) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.160 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.160; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.160) by BL6PEPF0001AB78.mail.protection.outlook.com (10.167.242.171) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6907.21 via Frontend Transport; Tue, 17 Oct 2023 07:43:55 +0000 Received: from rnnvmail201.nvidia.com (10.129.68.8) by mail.nvidia.com (10.129.200.66) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Tue, 17 Oct 2023 00:43:40 -0700 Received: from dev-r-vrt-155.mtr.labs.mlnx (10.126.230.35) by rnnvmail201.nvidia.com (10.129.68.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Tue, 17 Oct 2023 00:43:37 -0700 From: Ido Schimmel To: , CC: , , , , , , , , , , Ido Schimmel Subject: [RFC PATCH net-next 01/12] netdevsim: Block until all devices are released Date: Tue, 17 Oct 2023 10:42:46 +0300 Message-ID: <20231017074257.3389177-2-idosch@nvidia.com> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20231017074257.3389177-1-idosch@nvidia.com> References: <20231017074257.3389177-1-idosch@nvidia.com> MIME-Version: 1.0 X-Originating-IP: [10.126.230.35] X-ClientProxiedBy: rnnvmail201.nvidia.com (10.129.68.8) To rnnvmail201.nvidia.com (10.129.68.8) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BL6PEPF0001AB78:EE_|DM3PR12MB9288:EE_ X-MS-Office365-Filtering-Correlation-Id: 1ef68985-ec7f-41a9-bea1-08dbcee4d0fd X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: zsGUp/medVjjw7SRL3fLbHlx2zms0NNX0vQU2RzOaCVt6hmel0ki/u2QrCW8HPnGjWEV0cEqCR93Nw0ZjCkHjRzrLMuxXDUEJShK5VJYthiKlyLy/UtDpcmQDrzqH13rk6fuEbbACy8JY7SRC1w4tarwBtzH9uO6nk+onESRJZxt8PVfKd9SL+h+fxyB9DLy8194V7gxEn085Y96YGXMjyjj/Up1Y6cAq37hCxmetJM/mOYJ+ARUSW7mJiuWvAE86UKIsTFSa5Xp4+qsdydaKekZSCmOBvfHNRHnvlpLk0yjBjZc2ThMpTQseuFc++u+s67d5B0E3PvDZoLAEZl1WTfuqrwmiacnUQJOvSTl7AkfxDYMFgK5BoTtHZO24t09cTSXYPYjQOuv8ixa3pwR9Sr1F7KXyP/cJo1yE7cY0WsK68bwkz4u951bAnUhs5xVOjjewej52RNJxg/kFN3WkMuFdGuRegPRrlPiX94G4f7jzrdx0mVyf2jKhV82ZJQjwSYILuo4AgUKeeL8pHyMpyvHayXvQyKzfxD3fAI5r9e02B7EZ9VbD+slRdCcZABcuI5ZGbq/gVEW1r9Nz7fezzeutRIbPtBu7Ov5FgHENXx03305KHB74t3WoSAQDS4gBBIYtL8hU+iCOkeLcXIlUS0OqdSBViehI+SzNIUCYyosTiJwbdUH6Ona7NKoEIwd3RuFwP7n2jqUoEorES6SUbH5Cw3kiDxrKbrNRMUltQq0dHkRIARoN1chVJ87PPpKcpScQUHf5efv68lCXQekVwyTtklbpQSRR+6H8Mzc8oY= X-Forefront-Antispam-Report: CIP:216.228.117.160;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc6edge1.nvidia.com;CAT:NONE;SFS:(13230031)(4636009)(346002)(39860400002)(376002)(136003)(396003)(230922051799003)(451199024)(64100799003)(186009)(1800799009)(82310400011)(36840700001)(40470700004)(46966006)(2616005)(40460700003)(40480700001)(966005)(36860700001)(82740400003)(7636003)(356005)(47076005)(83380400001)(6666004)(16526019)(26005)(36756003)(70586007)(70206006)(54906003)(478600001)(1076003)(107886003)(426003)(110136005)(336012)(316002)(86362001)(4326008)(41300700001)(2906002)(8936002)(8676002)(5660300002);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 17 Oct 2023 07:43:55.5181 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 1ef68985-ec7f-41a9-bea1-08dbcee4d0fd X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.117.160];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BL6PEPF0001AB78.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM3PR12MB9288 Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org Like other buses, devices on the netdevsim bus have a release callback that is invoked when the reference count of the device drops to zero. However, unlike other buses such as PCI, the release callback is not necessarily built into the kernel, as netdevsim can be built as a module. This above is problematic as nothing prevents the module from being unloaded before the release callback has been invoked, which can happen asynchronously. One such example is going to be added in subsequent patches where devlink will call put_device() from an RCU callback. The issue is not theoretical and the reproducer in [1] can reliably crash the kernel. The conclusion of this discussion was that the issue should be solved in netdevsim, which is what this patch is trying to do. Add a reference count that is increased when a device is added to the bus and decreased when a device is released. Signal a completion when the reference count drops to zero and wait for the completion when unloading the module so that the module will not be unloaded before all the devices were released. The reference count is initialized to one so that completion is only signaled when unloading the module. With this patch, the reproducer in [1] no longer crashes the kernel. [1] https://lore.kernel.org/netdev/20230619125015.1541143-2-idosch@nvidia.com/ Signed-off-by: Ido Schimmel Reviewed-by: Jakub Kicinski --- drivers/net/netdevsim/bus.c | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/drivers/net/netdevsim/bus.c b/drivers/net/netdevsim/bus.c index 0787ad252dd9..bcbc1e19edde 100644 --- a/drivers/net/netdevsim/bus.c +++ b/drivers/net/netdevsim/bus.c @@ -3,11 +3,13 @@ * Copyright (C) 2019 Mellanox Technologies. All rights reserved */ +#include #include #include #include #include #include +#include #include #include @@ -17,6 +19,8 @@ static DEFINE_IDA(nsim_bus_dev_ids); static LIST_HEAD(nsim_bus_dev_list); static DEFINE_MUTEX(nsim_bus_dev_list_lock); static bool nsim_bus_enable; +static refcount_t nsim_bus_devs; /* Including the bus itself. */ +static DECLARE_COMPLETION(nsim_bus_devs_released); static struct nsim_bus_dev *to_nsim_bus_dev(struct device *dev) { @@ -121,6 +125,8 @@ static void nsim_bus_dev_release(struct device *dev) nsim_bus_dev = container_of(dev, struct nsim_bus_dev, dev); kfree(nsim_bus_dev); + if (refcount_dec_and_test(&nsim_bus_devs)) + complete(&nsim_bus_devs_released); } static struct device_type nsim_bus_dev_type = { @@ -170,6 +176,7 @@ new_device_store(const struct bus_type *bus, const char *buf, size_t count) goto err; } + refcount_inc(&nsim_bus_devs); /* Allow using nsim_bus_dev */ smp_store_release(&nsim_bus_dev->init, true); @@ -326,6 +333,7 @@ int nsim_bus_init(void) err = driver_register(&nsim_driver); if (err) goto err_bus_unregister; + refcount_set(&nsim_bus_devs, 1); /* Allow using resources */ smp_store_release(&nsim_bus_enable, true); return 0; @@ -341,6 +349,8 @@ void nsim_bus_exit(void) /* Disallow using resources */ smp_store_release(&nsim_bus_enable, false); + if (refcount_dec_and_test(&nsim_bus_devs)) + complete(&nsim_bus_devs_released); mutex_lock(&nsim_bus_dev_list_lock); list_for_each_entry_safe(nsim_bus_dev, tmp, &nsim_bus_dev_list, list) { @@ -349,6 +359,8 @@ void nsim_bus_exit(void) } mutex_unlock(&nsim_bus_dev_list_lock); + wait_for_completion(&nsim_bus_devs_released); + driver_unregister(&nsim_driver); bus_unregister(&nsim_bus); } From patchwork Tue Oct 17 07:42:47 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ido Schimmel X-Patchwork-Id: 13424697 X-Patchwork-Delegate: bhelgaas@google.com Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E8A2FCDB474 for ; Tue, 17 Oct 2023 07:44:03 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232134AbjJQHoD (ORCPT ); Tue, 17 Oct 2023 03:44:03 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43244 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234647AbjJQHoD (ORCPT ); Tue, 17 Oct 2023 03:44:03 -0400 Received: from NAM11-CO1-obe.outbound.protection.outlook.com (mail-co1nam11on2089.outbound.protection.outlook.com [40.107.220.89]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4225395; Tue, 17 Oct 2023 00:44:01 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=WFKjmwrewh9jL9rTJpnFEPJV9qCuhV3XOarTRWEBjX6DRUqwVIPUlsuQTzv1aQEvrSRwUBE+LqMhaNCT+cFlQCuN1rW0WMgR1AEGOaCf7p3hpaH5TZ81evp8NIXs7Ipyh7c2hF1kOPtHJp+83MIDvWAhIfq8j42h6jvrPDnqA35TzzNrrrwRbNkNjnT2eoPsBpEruu/6pq8UiVc0skCrVSmjPXd0x4lcDQI0g+iAX3PO7n+J09cNTafMxn7Tjq2QEvpGhheY3twF7Ia8LWkbc4fClYtxgyxI/1lVBydFj5bYIiBLUW6O5+YDuPOrOxkqTl+Jhk3/YXFOI4Hw2BEDXQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=mm2TIS8vlCdsD1nFe6vuUEyYctb55iQFP4PEjqu58pU=; b=UAnjU9sQIHpD+8FImNHEz4JVr47tSiWnEN5lbuEHK3NkilYg90MxPKfXDOiQFYQeEdnroYmqUG74tViZT854c50RZN8jIR/6cJjqwo1C8maO6W6mVAg6WZdUQ1FEjRs2YOEPemm58Jn8sfpuio3QqLp2DrltAoBf4EO0DHoQWWbfPnU/5K3u3Tx9U/ojFbD1IjGl7dKjD7axTRkux6AvQ3hoeZMvE6cQNn6DrVEdGx1CMffu0MCfQkIjB0oaskxocfW6bC/ZqalUO95HpX8v8xm27Zdy4VvhwG3kIdE6b1xf4t68Tk7V2Umh/fxFZ0APQljAqSP2zCCqfFeLzi8dLw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.160) smtp.rcpttodomain=vger.kernel.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=mm2TIS8vlCdsD1nFe6vuUEyYctb55iQFP4PEjqu58pU=; b=hfG7grrKmKUYrKHfNYn7vdob08mI8ixo2SyoOLi05UfpjgE2D+AedJNDcXqiZqR3a4/8DPwx6MT31o6NBSrEi9cpWRtWufc2+ohFjiaMwrHi8/NZq9GCVRmPmCDmjtxwX7VUg4sPpBdlarW7URFoIShqNBG/W0BIpORCrIrbPdaQCyv0JYOMxm9L+LXbz+G5blHAbopMMVmrnbfzRgvxUYgWU/svJ3E21vRuUme6pHnwtb5g7z7tRPeHGXHPYpVyBGCblEIIl/mD1Gl4/AIYUSU6DCeDxg3czodkrIYB0gtrPP1kJIFoe9BKsMJjGz+EhzYL9t0dbeoWZKbQAPP0cQ== Received: from BLAPR05CA0007.namprd05.prod.outlook.com (2603:10b6:208:36e::7) by CH0PR12MB8508.namprd12.prod.outlook.com (2603:10b6:610:18c::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6886.35; Tue, 17 Oct 2023 07:43:59 +0000 Received: from BL6PEPF0001AB71.namprd02.prod.outlook.com (2603:10b6:208:36e:cafe::db) by BLAPR05CA0007.outlook.office365.com (2603:10b6:208:36e::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6907.17 via Frontend Transport; Tue, 17 Oct 2023 07:43:59 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.160) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.160 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.160; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.160) by BL6PEPF0001AB71.mail.protection.outlook.com (10.167.242.164) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6907.20 via Frontend Transport; Tue, 17 Oct 2023 07:43:59 +0000 Received: from rnnvmail201.nvidia.com (10.129.68.8) by mail.nvidia.com (10.129.200.66) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Tue, 17 Oct 2023 00:43:44 -0700 Received: from dev-r-vrt-155.mtr.labs.mlnx (10.126.230.35) by rnnvmail201.nvidia.com (10.129.68.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Tue, 17 Oct 2023 00:43:41 -0700 From: Ido Schimmel To: , CC: , , , , , , , , , , Ido Schimmel Subject: [RFC PATCH net-next 02/12] devlink: Hold a reference on parent device Date: Tue, 17 Oct 2023 10:42:47 +0300 Message-ID: <20231017074257.3389177-3-idosch@nvidia.com> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20231017074257.3389177-1-idosch@nvidia.com> References: <20231017074257.3389177-1-idosch@nvidia.com> MIME-Version: 1.0 X-Originating-IP: [10.126.230.35] X-ClientProxiedBy: rnnvmail201.nvidia.com (10.129.68.8) To rnnvmail201.nvidia.com (10.129.68.8) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BL6PEPF0001AB71:EE_|CH0PR12MB8508:EE_ X-MS-Office365-Filtering-Correlation-Id: 563afb95-2527-43e3-6ba8-08dbcee4d315 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: d+7nLyCX2Vcb3mjI1rafa0CqXIV8aFllzmpoq2PMoay+0O+GqtcGtN/DzRTMDgPMVSKHf8iDsklVvVaLy7Z/xleehSi1WKUws7ghAm/gGhu5pbNwNdj5tmSL/Lq8+fqEeHy312WX7z96mTZHFz4CXDk0Nt/t/Jz8B/Wd1C76lVV5l0FquArXU05aAt96reKFwlInyGn98Ng0wLxqOEhfYuY4LQU7Y/cboWcJporHMbuiNP3uvlOp0kTpxYq+l09jx2/gC4tPo3jAU689ENg9l+khLrpUCbMe2QBoOEd4pVJbZ+UrpZOmFCI7PFMYivWhSr1UMrW1wB80MoZulj0F+cBIXbXyoTzbgB4mgamTIWG8kj5k4rps5coUUQIRb7fBZy6ZDdaAeUSWWX891GBVBeGoHuh99TVJUmueNH7rJqG82r3KmN/wh61AeBXpER29Y2jcpiiasIQMd1PB72kcoi1F4S7VfEBVNO6ZZDEMfR4o0p/z+c4NJpmjsSZmvvh1JQpuH7kyyGIzBLuJu6PoICOTxcaooWbCJKzNcvZL4dXVoa6z54yZxLJ1r/Ud9kXfDINa3iY4oMX9wEKeMQ2yujsoCzqa1/PIuPwAMOKMLFbM1bzE8nPADUkft63TKBA+DF7dqd2jzyzE4Jr5PmDi+1qL+lqNBGXY6wcwkATipx5d3AQMRLct5IUC8g5lgYP2p3shc8N5MHCs9QyEKXxElV9N8DANfjyZASb2pK0jxHPbiMosaOdEslHP+NLZoyy9 X-Forefront-Antispam-Report: CIP:216.228.117.160;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc6edge1.nvidia.com;CAT:NONE;SFS:(13230031)(4636009)(39860400002)(396003)(376002)(136003)(346002)(230922051799003)(186009)(1800799009)(82310400011)(451199024)(64100799003)(36840700001)(46966006)(40470700004)(41300700001)(70206006)(110136005)(478600001)(54906003)(70586007)(16526019)(426003)(1076003)(26005)(336012)(107886003)(316002)(2616005)(4326008)(8676002)(2906002)(8936002)(5660300002)(36756003)(47076005)(7636003)(36860700001)(83380400001)(82740400003)(356005)(86362001)(40460700003)(40480700001);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 17 Oct 2023 07:43:59.0131 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 563afb95-2527-43e3-6ba8-08dbcee4d315 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.117.160];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BL6PEPF0001AB71.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH0PR12MB8508 Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org Each devlink instance is associated with a parent device and a pointer to this device is stored in the devlink structure, but devlink does not hold a reference on this device. This is going to be a problem in the next patch where - among other things - devlink will acquire the device lock during netns dismantle, before the reload operation. Since netns dismantle is performed asynchronously and since a reference is not held on the parent device, it will be possible to hit a use-after-free. Prepare for the upcoming change by holding a reference on the parent device. Signed-off-by: Ido Schimmel Reviewed-by: Jiri Pirko --- net/devlink/core.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/net/devlink/core.c b/net/devlink/core.c index bcbbb952569f..5b8b692b8c76 100644 --- a/net/devlink/core.c +++ b/net/devlink/core.c @@ -4,6 +4,7 @@ * Copyright (c) 2016 Jiri Pirko */ +#include #include #define CREATE_TRACE_POINTS #include @@ -310,6 +311,7 @@ static void devlink_release(struct work_struct *work) mutex_destroy(&devlink->lock); lockdep_unregister_key(&devlink->lock_key); + put_device(devlink->dev); kfree(devlink); } @@ -425,6 +427,7 @@ struct devlink *devlink_alloc_ns(const struct devlink_ops *ops, if (ret < 0) goto err_xa_alloc; + get_device(dev); devlink->dev = dev; devlink->ops = ops; xa_init_flags(&devlink->ports, XA_FLAGS_ALLOC); From patchwork Tue Oct 17 07:42:48 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ido Schimmel X-Patchwork-Id: 13424699 X-Patchwork-Delegate: bhelgaas@google.com Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 61508CDB482 for ; Tue, 17 Oct 2023 07:44:10 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234672AbjJQHoJ (ORCPT ); Tue, 17 Oct 2023 03:44:09 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55632 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234647AbjJQHoI (ORCPT ); Tue, 17 Oct 2023 03:44:08 -0400 Received: from NAM04-BN8-obe.outbound.protection.outlook.com (mail-bn8nam04on2056.outbound.protection.outlook.com [40.107.100.56]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 87B83F1; Tue, 17 Oct 2023 00:44:06 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=JoN/ZnJfE2CGa5S9CvA47+rIo7ZPk/pYmz0wDZzvsgfAaQOOwaGW8rweLY30bDa+DZaNzABSm0HyrSPwBHNqUw5JBNGWqtigeQ+qWJa2j2ozuY5Fj5QLXlvk3uisldb2nsV6o6rSkIhAPr+nt2HQ0CtRUdROXQ97bgP14UG8WCgOogq5sqFuBJCHTEWe3oCiAUKtNYMDoQJSAWbUjvL/LfN/jwlYohAANfMpuPJxB6ncQ/xlziOuQpGpR20OftiS2Nag2wXnlGH2cKl+ZiFyMExUJg+we0nwmp2Y7TgGXDVVDD2aPry+udV8wIJVk2uqjq1r6Hi5VCtrIMQGXyKhkw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=NHg6h6blho5eQEpN8tp3swpuvzBzcqSeBV65e/vt1rc=; b=lPbOZFFh2SqfFGUMjyzwnaeKpaGLqvN9dZZs/ur36u5a7zRXn53AslKAfalh1SaGqLrfTG5iaEJPb8IQ/33UG9jsUAmrdTcUTUFlWM1kkBMcljh8Bd9duTDj2HjFOmV1oW/DcSU6PsgFL2xO+XeyfpWlaZ5al0H8a1wCq7KOltm+Xsjq01VrVyE9YqAaEZb3WbQdPgybtJGDEgcqr3/QYqYJCWEesyglr/KYGae3deXq3qgcE+tq29tYolI94NFE3dELMKIMI1s5Jr5XK/cqp/5CcN7HhKa8DUn59Uqdg09NcOf2zyML9do7E9MWrb48tAOZPKzrT3i2jZHNfVh06w== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.160) smtp.rcpttodomain=vger.kernel.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=NHg6h6blho5eQEpN8tp3swpuvzBzcqSeBV65e/vt1rc=; b=tfcGyWF5xUT77Ovf7AyqcDdTLYIirBCSFyiujekUA6MZdPzcwYwnKSRy+/G2j8QlgmfQjaFTNwZ4KCZLkS/NTB4H5RJxf54Rjx+wIgWuB8VS5YdGeuDh/RRTl73KUpK+QwaFmztEEiwmHtoMWt/IWmmK0xHlBFlLLFS3HvVkCrwdVOyIJq7iWKVd3czm6srzvd3cNmCecnXE1xCAp0qUouB2pfpyOsnObI+vSvAXc8/dLnYwqTPGhlKLZsD0TgogMGVmN/QCBBiqenKUq3lMlKvb73I1okf0d3WmOOJhqUSoPEm9NlkKpZ/1kpK+pY/QUCk/+6hgjlVnbw3PMdXtAw== Received: from BL1PR13CA0152.namprd13.prod.outlook.com (2603:10b6:208:2bd::7) by BL1PR12MB5899.namprd12.prod.outlook.com (2603:10b6:208:397::19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6886.35; Tue, 17 Oct 2023 07:44:04 +0000 Received: from BL6PEPF0001AB72.namprd02.prod.outlook.com (2603:10b6:208:2bd:cafe::6a) by BL1PR13CA0152.outlook.office365.com (2603:10b6:208:2bd::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6907.18 via Frontend Transport; Tue, 17 Oct 2023 07:44:04 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.160) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.160 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.160; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.160) by BL6PEPF0001AB72.mail.protection.outlook.com (10.167.242.165) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6907.20 via Frontend Transport; Tue, 17 Oct 2023 07:44:04 +0000 Received: from rnnvmail201.nvidia.com (10.129.68.8) by mail.nvidia.com (10.129.200.66) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Tue, 17 Oct 2023 00:43:48 -0700 Received: from dev-r-vrt-155.mtr.labs.mlnx (10.126.230.35) by rnnvmail201.nvidia.com (10.129.68.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Tue, 17 Oct 2023 00:43:44 -0700 From: Ido Schimmel To: , CC: , , , , , , , , , , Ido Schimmel Subject: [RFC PATCH net-next 03/12] devlink: Acquire device lock during reload Date: Tue, 17 Oct 2023 10:42:48 +0300 Message-ID: <20231017074257.3389177-4-idosch@nvidia.com> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20231017074257.3389177-1-idosch@nvidia.com> References: <20231017074257.3389177-1-idosch@nvidia.com> MIME-Version: 1.0 X-Originating-IP: [10.126.230.35] X-ClientProxiedBy: rnnvmail201.nvidia.com (10.129.68.8) To rnnvmail201.nvidia.com (10.129.68.8) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BL6PEPF0001AB72:EE_|BL1PR12MB5899:EE_ X-MS-Office365-Filtering-Correlation-Id: aadf613b-702d-4966-0979-08dbcee4d620 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 5LKLCp3+TqqXMpTpI0w/V9eqcYbGZGrdgdAMoHzyhsaF2z/m+ek3LEf2YHoUgn+LMJP4GDdNmMQdXyP3kdFdR0rJHG3d1zHKmImM5jn4aWKWH1Cxro5vEmO19IjRG3dVpA97XEeYp3KWDDBA6kWb2VQw+MpPW/YJbI5CGq/jmRv85EqfvW91ArvFVUxQlUuri6ho/kRFh4X4yN6rL7HCXN6EbmpbXqQEeOijGuWKlCETlcaCUJkYycarkoNZvU22dmL8EhFVKnExn9OhXCFEFlwTW91snOYuvK6oLpru/rcCQCPbuxxcKBoDtreUA/dB75dGsOY6nLNtMllYIskgd5gxhM7aejDzFGr4msKzsSD7a78Pyn7CrkXaJeDL0oKO6HfswrsONR0uggrpNz9wUHhApPKPCLVJongal4VU2r3LRsBi9ZCaUytJECaH2yp+B4X9q9QOovJw/Fob2zD6TB8nr853g9bXIqo/JFJvT8G3QkwVgmrkEa49mDAE8dUjdLwKx+sqorcYn5OhXpY8yD/+KsDYQ8YlKdD3MkjPZ2os0ESINjuJnxTDE6j0UE9QZp08I1rInVHEFU3itFNf5FbWfuizOJSj5D1CFVZoLndPaLbbuiO/56xoGVmU14dMwek1zn0aaiLdO+r4YexJ7K6JhbZsWbzVMmG1E1CFEHEJCB8oomo0O96S1OsQh0eBCDVLVrG+3GofDz0BJJZypK64R/OEfRxZdVsH7h9md1OYvEaGpRB2kA1pKuON3xPx X-Forefront-Antispam-Report: CIP:216.228.117.160;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc6edge1.nvidia.com;CAT:NONE;SFS:(13230031)(4636009)(39860400002)(396003)(376002)(136003)(346002)(230922051799003)(186009)(82310400011)(64100799003)(451199024)(1800799009)(40470700004)(46966006)(36840700001)(478600001)(70206006)(110136005)(70586007)(54906003)(107886003)(1076003)(16526019)(426003)(41300700001)(26005)(336012)(316002)(2616005)(8676002)(8936002)(2906002)(5660300002)(4326008)(36756003)(86362001)(47076005)(7636003)(36860700001)(83380400001)(82740400003)(356005)(40460700003)(40480700001);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 17 Oct 2023 07:44:04.1315 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: aadf613b-702d-4966-0979-08dbcee4d620 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.117.160];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BL6PEPF0001AB72.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL1PR12MB5899 Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org Device drivers register with devlink from their probe routines (under the device lock) by acquiring the devlink instance lock and calling devl_register(). Drivers that support a devlink reload usually implement the reload_{down, up}() operations in a similar fashion to their remove and probe routines, respectively. However, while the remove and probe routines are invoked with the device lock held, the reload operations are only invoked with the devlink instance lock held. It is therefore impossible for drivers to acquire the device lock from their reload operations, as this would result in lock inversion. The motivating use case for invoking the reload operations with the device lock held is in mlxsw which needs to trigger a PCI reset as part of the reload. The driver cannot call pci_reset_function() as this function acquires the device lock. Instead, it needs to call __pci_reset_function_locked which expects the device lock to be held. To that end, adjust devlink to always acquire the device lock before the devlink instance lock when performing a reload. Do that both when reload is triggered explicitly by user space and when it is triggered as part of netns dismantle. Tested the following flows with netdevsim and mlxsw while lockdep is enabled: netdevsim: # echo "10 1" > /sys/bus/netdevsim/new_device # devlink dev reload netdevsim/netdevsim10 # ip netns add bla # devlink dev reload netdevsim/netdevsim10 netns bla # ip netns del bla # echo 10 > /sys/bus/netdevsim/del_device mlxsw: # devlink dev reload pci/0000:01:00.0 # ip netns add bla # devlink dev reload pci/0000:01:00.0 netns bla # ip netns del bla # echo 1 > /sys/bus/pci/devices/0000\:01\:00.0/remove # echo 1 > /sys/bus/pci/rescan Signed-off-by: Ido Schimmel --- net/devlink/core.c | 4 ++-- net/devlink/dev.c | 8 ++++++++ net/devlink/devl_internal.h | 19 ++++++++++++++++++- net/devlink/health.c | 3 ++- net/devlink/netlink.c | 21 ++++++++++++++------- net/devlink/region.c | 3 ++- 6 files changed, 46 insertions(+), 12 deletions(-) diff --git a/net/devlink/core.c b/net/devlink/core.c index 5b8b692b8c76..0f866f2cbaf6 100644 --- a/net/devlink/core.c +++ b/net/devlink/core.c @@ -502,14 +502,14 @@ static void __net_exit devlink_pernet_pre_exit(struct net *net) * all devlink instances from this namespace into init_net. */ devlinks_xa_for_each_registered_get(net, index, devlink) { - devl_lock(devlink); + devl_dev_lock(devlink, true); err = 0; if (devl_is_registered(devlink)) err = devlink_reload(devlink, &init_net, DEVLINK_RELOAD_ACTION_DRIVER_REINIT, DEVLINK_RELOAD_LIMIT_UNSPEC, &actions_performed, NULL); - devl_unlock(devlink); + devl_dev_unlock(devlink, true); devlink_put(devlink); if (err && err != -EOPNOTSUPP) pr_warn("Failed to reload devlink instance into init_net\n"); diff --git a/net/devlink/dev.c b/net/devlink/dev.c index dc8039ca2b38..70cebe716187 100644 --- a/net/devlink/dev.c +++ b/net/devlink/dev.c @@ -4,6 +4,7 @@ * Copyright (c) 2016 Jiri Pirko */ +#include #include #include #include "devl_internal.h" @@ -433,6 +434,13 @@ int devlink_reload(struct devlink *devlink, struct net *dest_net, struct net *curr_net; int err; + /* Make sure the reload operations are invoked with the device lock + * held to allow drivers to trigger functionality that expects it + * (e.g., PCI reset) and to close possible races between these + * operations and probe/remove. + */ + device_lock_assert(devlink->dev); + memcpy(remote_reload_stats, devlink->stats.remote_reload_stats, sizeof(remote_reload_stats)); diff --git a/net/devlink/devl_internal.h b/net/devlink/devl_internal.h index 741d1bf1bec8..a9c5e52c40a7 100644 --- a/net/devlink/devl_internal.h +++ b/net/devlink/devl_internal.h @@ -3,6 +3,7 @@ * Copyright (c) 2016 Jiri Pirko */ +#include #include #include #include @@ -96,6 +97,20 @@ static inline bool devl_is_registered(struct devlink *devlink) return xa_get_mark(&devlinks, devlink->index, DEVLINK_REGISTERED); } +static inline void devl_dev_lock(struct devlink *devlink, bool dev_lock) +{ + if (dev_lock) + device_lock(devlink->dev); + devl_lock(devlink); +} + +static inline void devl_dev_unlock(struct devlink *devlink, bool dev_lock) +{ + devl_unlock(devlink); + if (dev_lock) + device_unlock(devlink->dev); +} + typedef void devlink_rel_notify_cb_t(struct devlink *devlink, u32 obj_index); typedef void devlink_rel_cleanup_cb_t(struct devlink *devlink, u32 obj_index, u32 rel_index); @@ -113,6 +128,7 @@ int devlink_rel_devlink_handle_put(struct sk_buff *msg, struct devlink *devlink, /* Netlink */ #define DEVLINK_NL_FLAG_NEED_PORT BIT(0) #define DEVLINK_NL_FLAG_NEED_DEVLINK_OR_PORT BIT(1) +#define DEVLINK_NL_FLAG_NEED_DEV_LOCK BIT(2) enum devlink_multicast_groups { DEVLINK_MCGRP_CONFIG, @@ -140,7 +156,8 @@ typedef int devlink_nl_dump_one_func_t(struct sk_buff *msg, int flags); struct devlink * -devlink_get_from_attrs_lock(struct net *net, struct nlattr **attrs); +devlink_get_from_attrs_lock(struct net *net, struct nlattr **attrs, + bool dev_lock); int devlink_nl_dumpit(struct sk_buff *msg, struct netlink_callback *cb, devlink_nl_dump_one_func_t *dump_one); diff --git a/net/devlink/health.c b/net/devlink/health.c index 51e6e81e31bb..3c4c049c3636 100644 --- a/net/devlink/health.c +++ b/net/devlink/health.c @@ -1266,7 +1266,8 @@ devlink_health_reporter_get_from_cb_lock(struct netlink_callback *cb) struct nlattr **attrs = info->attrs; struct devlink *devlink; - devlink = devlink_get_from_attrs_lock(sock_net(cb->skb->sk), attrs); + devlink = devlink_get_from_attrs_lock(sock_net(cb->skb->sk), attrs, + false); if (IS_ERR(devlink)) return NULL; diff --git a/net/devlink/netlink.c b/net/devlink/netlink.c index 499304d9de49..14d598000d72 100644 --- a/net/devlink/netlink.c +++ b/net/devlink/netlink.c @@ -124,7 +124,8 @@ int devlink_nl_msg_reply_and_new(struct sk_buff **msg, struct genl_info *info) } struct devlink * -devlink_get_from_attrs_lock(struct net *net, struct nlattr **attrs) +devlink_get_from_attrs_lock(struct net *net, struct nlattr **attrs, + bool dev_lock) { struct devlink *devlink; unsigned long index; @@ -138,12 +139,12 @@ devlink_get_from_attrs_lock(struct net *net, struct nlattr **attrs) devname = nla_data(attrs[DEVLINK_ATTR_DEV_NAME]); devlinks_xa_for_each_registered_get(net, index, devlink) { - devl_lock(devlink); + devl_dev_lock(devlink, dev_lock); if (devl_is_registered(devlink) && strcmp(devlink->dev->bus->name, busname) == 0 && strcmp(dev_name(devlink->dev), devname) == 0) return devlink; - devl_unlock(devlink); + devl_dev_unlock(devlink, dev_lock); devlink_put(devlink); } @@ -155,9 +156,12 @@ static int __devlink_nl_pre_doit(struct sk_buff *skb, struct genl_info *info, { struct devlink_port *devlink_port; struct devlink *devlink; + bool dev_lock; int err; - devlink = devlink_get_from_attrs_lock(genl_info_net(info), info->attrs); + dev_lock = !!(flags & DEVLINK_NL_FLAG_NEED_DEV_LOCK); + devlink = devlink_get_from_attrs_lock(genl_info_net(info), info->attrs, + dev_lock); if (IS_ERR(devlink)) return PTR_ERR(devlink); @@ -177,7 +181,7 @@ static int __devlink_nl_pre_doit(struct sk_buff *skb, struct genl_info *info, return 0; unlock: - devl_unlock(devlink); + devl_dev_unlock(devlink, dev_lock); devlink_put(devlink); return err; } @@ -205,9 +209,11 @@ void devlink_nl_post_doit(const struct genl_split_ops *ops, struct sk_buff *skb, struct genl_info *info) { struct devlink *devlink; + bool dev_lock; + dev_lock = !!(ops->internal_flags & DEVLINK_NL_FLAG_NEED_DEV_LOCK); devlink = info->user_ptr[0]; - devl_unlock(devlink); + devl_dev_unlock(devlink, dev_lock); devlink_put(devlink); } @@ -219,7 +225,7 @@ static int devlink_nl_inst_single_dumpit(struct sk_buff *msg, struct devlink *devlink; int err; - devlink = devlink_get_from_attrs_lock(sock_net(msg->sk), attrs); + devlink = devlink_get_from_attrs_lock(sock_net(msg->sk), attrs, false); if (IS_ERR(devlink)) return PTR_ERR(devlink); err = dump_one(msg, devlink, cb, flags | NLM_F_DUMP_FILTERED); @@ -420,6 +426,7 @@ static const struct genl_small_ops devlink_nl_small_ops[40] = { .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = devlink_nl_cmd_reload, .flags = GENL_ADMIN_PERM, + .internal_flags = DEVLINK_NL_FLAG_NEED_DEV_LOCK, }, { .cmd = DEVLINK_CMD_PARAM_SET, diff --git a/net/devlink/region.c b/net/devlink/region.c index d197cdb662db..30c6c49ec10b 100644 --- a/net/devlink/region.c +++ b/net/devlink/region.c @@ -883,7 +883,8 @@ int devlink_nl_cmd_region_read_dumpit(struct sk_buff *skb, start_offset = state->start_offset; - devlink = devlink_get_from_attrs_lock(sock_net(cb->skb->sk), attrs); + devlink = devlink_get_from_attrs_lock(sock_net(cb->skb->sk), attrs, + false); if (IS_ERR(devlink)) return PTR_ERR(devlink); From patchwork Tue Oct 17 07:42:49 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ido Schimmel X-Patchwork-Id: 13424698 X-Patchwork-Delegate: bhelgaas@google.com Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3D8C4CDB474 for ; Tue, 17 Oct 2023 07:44:09 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234658AbjJQHoI (ORCPT ); Tue, 17 Oct 2023 03:44:08 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55616 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234672AbjJQHoH (ORCPT ); Tue, 17 Oct 2023 03:44:07 -0400 Received: from NAM02-BN1-obe.outbound.protection.outlook.com (mail-bn1nam02on2047.outbound.protection.outlook.com [40.107.212.47]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6F0FD8F; Tue, 17 Oct 2023 00:44:05 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=ikEsNTIMI1rtDZE/aDLfIBhbUvjVYJL00vA3UCv/xndAU3uVDqcOplK33YUjT2jwjMuRZTOtnjHAB7RMCVHFb+xp4J54cjf0Vu+YtwniJFZfFJOX9cdoU9H5JQFdOWllk2LdTcUgEPIp+It+t0CXOUZLBiRd63qjfHIat2Ob4UM1tYUKq+Vo6lHPb4cXKquMnfW1mUsJxePZIudzz2bxypFPhZDU9VoCbnKPo2jrKAxEmKisV40EmsLJ8eJUwgNpGEZ1H/vsBKT5WEnH2BljJMUts3uHYuHUUKD94NEwhhWo2x7BJkjwrjz03o9gfWJasjHgxICPUXqv4wBy5zRl9A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=n91lZ8OzIl4a+vNqew2S3qyrxSEe3wp3gOPs4Xh0hIg=; b=iMrhLbQWHca/jgmuSdWxyVfkH7kHIsY7wxdR0UyC9KddU/9ydQai+ihvEchFtEdjuoCwK30P/gdHRYmZHwsrcayIR981l+VcGCDXgEWBrWjmEB3MCWO8rD1GC7ES1tpPSc2caPOJ0HPJKfahkA3WDBPG73TbfY0mgEY4Au3/atVko6UTBGg8mCLOp086MEvUHoUcDq4/DT8O5jCIv9v9c856ouiKVGuN38dvI0af9dbp/kBy7j1CkuftLdLiX75I4mCUjiNhW+Mg4GCdOMphjjnR2sBzUrG3VfF6tC3Tthm0F0u9ZZwF1mBdxiwVMzp2R+StIpmh6cd1tsVWq2XiKw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.161) smtp.rcpttodomain=vger.kernel.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=n91lZ8OzIl4a+vNqew2S3qyrxSEe3wp3gOPs4Xh0hIg=; b=CNg9qzYLS1tUuMGwC+hf+EXovKp9D9c3XWq/hknNVGFfblM0XPSUyrDI7gjH7/YWLpYKVm4Fq4DrzrUlHxZ9vMfacUQ3PIqN9UcVc7XmchZ9cJf3MGzeo2qMQ2JrFAFSWM1FJJUT0V6ehidR/Ii8tCxIRAG79TbkVg/xN5dnrRMcdfnHpSgI1OsmguRFxqY97i1MwJ/xygmVLQaC08nULJT98bS8jbpAcBUPmC/tOpYoGZjJiKyWdSyvTkO/7uR0F6HZyvsz8LF/ksGJolw9XvIM1kvSoMvXZ/HtLvAWQYw1EAU/yZ6mFcnDr5W42tCCDXwfFGHstKh8kBNSOz1njQ== Received: from CY8P220CA0036.NAMP220.PROD.OUTLOOK.COM (2603:10b6:930:47::8) by DM6PR12MB4329.namprd12.prod.outlook.com (2603:10b6:5:211::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6863.37; Tue, 17 Oct 2023 07:44:03 +0000 Received: from CY4PEPF0000EE32.namprd05.prod.outlook.com (2603:10b6:930:47:cafe::9a) by CY8P220CA0036.outlook.office365.com (2603:10b6:930:47::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6886.34 via Frontend Transport; Tue, 17 Oct 2023 07:44:03 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.161) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.161 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.161; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.161) by CY4PEPF0000EE32.mail.protection.outlook.com (10.167.242.38) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6838.22 via Frontend Transport; Tue, 17 Oct 2023 07:44:03 +0000 Received: from rnnvmail201.nvidia.com (10.129.68.8) by mail.nvidia.com (10.129.200.67) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Tue, 17 Oct 2023 00:43:52 -0700 Received: from dev-r-vrt-155.mtr.labs.mlnx (10.126.230.35) by rnnvmail201.nvidia.com (10.129.68.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Tue, 17 Oct 2023 00:43:48 -0700 From: Ido Schimmel To: , CC: , , , , , , , , , , Ido Schimmel Subject: [RFC PATCH net-next 04/12] PCI: Add no PM reset quirk for NVIDIA Spectrum devices Date: Tue, 17 Oct 2023 10:42:49 +0300 Message-ID: <20231017074257.3389177-5-idosch@nvidia.com> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20231017074257.3389177-1-idosch@nvidia.com> References: <20231017074257.3389177-1-idosch@nvidia.com> MIME-Version: 1.0 X-Originating-IP: [10.126.230.35] X-ClientProxiedBy: rnnvmail201.nvidia.com (10.129.68.8) To rnnvmail201.nvidia.com (10.129.68.8) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CY4PEPF0000EE32:EE_|DM6PR12MB4329:EE_ X-MS-Office365-Filtering-Correlation-Id: 76bdbd3b-06d8-4ade-eec9-08dbcee4d5aa X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: BNbSAhjIxV04I+eE3Rr8XODNpCk+JThkymm0RP0oKmKuP8ByeFgntITLZzb+JQQmoMC7w1CGqB5jX/bCN17s4/3GFP5xMj1P+EwqJ8Ze0aSsxM0hlsG9Lh9u9lmzMHRgqp35OIavPCBMqy9+ZOwNHrx6N9+xgIJtglvqjA/ylHLJaUDdnAuq5XXRp4CsDbpCglFpllge2dyWbr+zVHtj8VBcBjKgFqAuSJAyxRSdrmhb3qyGZ6oI+r+GsKS/fdqot53gGHn82orbZSpQXF3S6BJNWrEKrEchgIAjSaGYReUHDpHK9eMWu0QH+7/OGGIp9+kFkoRGIprusWAhgK75miEvOs+MeOFJSxbdXhq/SDbQZnJRw3H+GQgQA5bFV3gPh6g/dBHkUIU8bMIAHuaoRk7HyTs3GF6jxOrMciB+eFMD74HA2Us9g23wyXM12924CC8FAb6Edu4BAK3gfzKy7sXRfrOvy2snH0f/f03Om2mR6pedJHsSmUcCiIW7ECHzNuQJkh7wz4oUq8otQ40sFsbKbAE1G1X3+GpaHX6UalzNkSq6SAblGkUFtRjU3no0FOp2LkPyVK5iZaWGb5NkaqWF9ykqxd0pkLTdKVs6pz75kl0C3AV6noJGLVzHJjot+767BpsSTnJj8KzryvdVL8m5EQxTZJYV7v8lBEVTps+vAC4HUvwX2JnGbeHlX1NgYYRAdFlZk+Y6EKoZ5BBZX1nIrSHT8SqC5vjNTzxxAjZWPMd2288H+woIG2EGP7vQ X-Forefront-Antispam-Report: CIP:216.228.117.161;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc6edge2.nvidia.com;CAT:NONE;SFS:(13230031)(4636009)(39860400002)(376002)(346002)(396003)(136003)(230922051799003)(451199024)(1800799009)(186009)(82310400011)(64100799003)(36840700001)(46966006)(40470700004)(426003)(86362001)(356005)(7636003)(36756003)(82740400003)(40480700001)(2906002)(47076005)(8676002)(8936002)(41300700001)(5660300002)(4326008)(478600001)(107886003)(336012)(40460700003)(70206006)(110136005)(316002)(26005)(70586007)(16526019)(54906003)(2616005)(1076003)(36860700001);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 17 Oct 2023 07:44:03.4265 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 76bdbd3b-06d8-4ade-eec9-08dbcee4d5aa X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.117.161];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CY4PEPF0000EE32.namprd05.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR12MB4329 Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org Spectrum-{1,2,3,4} devices report that a D3hot->D0 transition causes a reset (i.e., they advertise NoSoftRst-). However, this transition seems to have no effect on the device: It continues to be operational and network ports remain up. Advertising this support makes it seem as if a PM reset is viable for these devices. Mark it as unavailable to skip it when testing reset methods. Before: # cat /sys/bus/pci/devices/0000\:03\:00.0/reset_method pm bus After: # cat /sys/bus/pci/devices/0000\:03\:00.0/reset_method bus Signed-off-by: Ido Schimmel Acked-by: Bjorn Helgaas --- drivers/pci/quirks.c | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c index eeec1d6f9023..23f6bd2184e2 100644 --- a/drivers/pci/quirks.c +++ b/drivers/pci/quirks.c @@ -3784,6 +3784,19 @@ static void quirk_no_pm_reset(struct pci_dev *dev) DECLARE_PCI_FIXUP_CLASS_HEADER(PCI_VENDOR_ID_ATI, PCI_ANY_ID, PCI_CLASS_DISPLAY_VGA, 8, quirk_no_pm_reset); +/* + * Spectrum-{1,2,3,4} devices report that a D3hot->D0 transition causes a reset + * (i.e., they advertise NoSoftRst-). However, this transition seems to have no + * effect on the device: It continues to be operational and network ports + * remain up. Advertising this support makes it seem as if a PM reset is viable + * for these devices. Mark it as unavailable to skip it when testing reset + * methods. + */ +DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_MELLANOX, 0xcb84, quirk_no_pm_reset); +DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_MELLANOX, 0xcf6c, quirk_no_pm_reset); +DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_MELLANOX, 0xcf70, quirk_no_pm_reset); +DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_MELLANOX, 0xcf80, quirk_no_pm_reset); + /* * Thunderbolt controllers with broken MSI hotplug signaling: * Entire 1st generation (Light Ridge, Eagle Ridge, Light Peak) and part From patchwork Tue Oct 17 07:42:50 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ido Schimmel X-Patchwork-Id: 13424700 X-Patchwork-Delegate: bhelgaas@google.com Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 88D2ECDB483 for ; Tue, 17 Oct 2023 07:44:16 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234661AbjJQHoP (ORCPT ); Tue, 17 Oct 2023 03:44:15 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55682 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234698AbjJQHoO (ORCPT ); Tue, 17 Oct 2023 03:44:14 -0400 Received: from NAM02-BN1-obe.outbound.protection.outlook.com (mail-bn1nam02on2042.outbound.protection.outlook.com [40.107.212.42]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5F11EF2; Tue, 17 Oct 2023 00:44:11 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=QbB6iCwkk9CQrzOLbE5Vw+jTUTfLjM9ZgBgGEMMT/CbTi/4uWPQHAajwmqXwXtphLIFC4T54HiDZfhTIUStoMO4gHQavkAYcRDdPHB55xFhJDewSNsgv++/hnrXLzg1heDs4CiZGHjKorFOm59hg7aWZ02fWtUT2MC2mgo3QMYTYZ9q4LKOz5i3HxFtyV3RspDGOjrDFmy6THSDx5giOfwCDvqE5yevhQmDYewft4tP9Q4gmhWvCeAmrwoBBYhUxMLWHC/lTXsL848i8l4NUl2A2mBs6yf0aw4abCTpZ4UBbhDz9dxr6mjPL3yc95slTkgUiEIn2jW+aXQ10QF964A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=JcAaM35r7yZL7svTVidSM0MqJyjI++7E0U7a77YAHlk=; b=UoBmvpHxJ992/NtW6wV0/tQpCrmwvythrWlil4tkWx9GLOMMB1bZFCQ0VDL852laehgyqhTkRWFl0t1VoyO/bXDtwYiQHHRf+KsnVpVVVwGHy10Rb8bxz2b0EULrVKTqvEvbR2eEH0vTZzU4lRx01vCAEazmtmCCUA1QKU2SqVzCnSFtFfqY1Te2tEijTD841SuIvbo/vKXa5oJYYd84DLneAVAdYXyHlYguqxnakNcJnVDz6DE4PJwGozvzGXL/PLLCuCjvowLrp0rjhwTlrVP1A2OYiZL4NQOa3uCiwy8tKZw4u47CrINTYVSqZ0PSuj0SapMz1UZMiaudexcUEQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.160) smtp.rcpttodomain=vger.kernel.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=JcAaM35r7yZL7svTVidSM0MqJyjI++7E0U7a77YAHlk=; b=tOLrr8XtLm3WTg2AEx+WBqZRx89xoCouTICmUUyRPDavXPeNZd1/TplAXvzNApDpk2eKDQ7a1XSiFuhy3mYLsAXRISgATWpK0Nux+eN2URddjLKLyJj+D/u7H/bStjx6kIQ/T3MgbteNczKS0EdAQg3Csogk4W6DJsOiZqWA+2lfxUKhyUx9FEUJXWV5VL0Szi2QEyandVdAlIC4Fmd4fzWzapk3qV3uuPQtSIJsd1zujLcGwISomPwiVYgPcLRRuSKOv/img2C7IMQ1o7fusV+jeR1wiAB8sbfscAm3V8P4BLPq0ens1fN4BtOlWFRWHU9Omh+s2dgIgxz73X8ZBg== Received: from BL0PR01CA0036.prod.exchangelabs.com (2603:10b6:208:71::49) by BL1PR12MB5255.namprd12.prod.outlook.com (2603:10b6:208:315::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6886.35; Tue, 17 Oct 2023 07:44:09 +0000 Received: from BL6PEPF0001AB77.namprd02.prod.outlook.com (2603:10b6:208:71:cafe::38) by BL0PR01CA0036.outlook.office365.com (2603:10b6:208:71::49) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6886.35 via Frontend Transport; Tue, 17 Oct 2023 07:44:09 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.160) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.160 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.160; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.160) by BL6PEPF0001AB77.mail.protection.outlook.com (10.167.242.170) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6907.20 via Frontend Transport; Tue, 17 Oct 2023 07:44:09 +0000 Received: from rnnvmail201.nvidia.com (10.129.68.8) by mail.nvidia.com (10.129.200.66) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Tue, 17 Oct 2023 00:43:56 -0700 Received: from dev-r-vrt-155.mtr.labs.mlnx (10.126.230.35) by rnnvmail201.nvidia.com (10.129.68.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Tue, 17 Oct 2023 00:43:52 -0700 From: Ido Schimmel To: , CC: , , , , , , , , , , Ido Schimmel Subject: [RFC PATCH net-next 05/12] PCI: Add device-specific reset for NVIDIA Spectrum devices Date: Tue, 17 Oct 2023 10:42:50 +0300 Message-ID: <20231017074257.3389177-6-idosch@nvidia.com> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20231017074257.3389177-1-idosch@nvidia.com> References: <20231017074257.3389177-1-idosch@nvidia.com> MIME-Version: 1.0 X-Originating-IP: [10.126.230.35] X-ClientProxiedBy: rnnvmail201.nvidia.com (10.129.68.8) To rnnvmail201.nvidia.com (10.129.68.8) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BL6PEPF0001AB77:EE_|BL1PR12MB5255:EE_ X-MS-Office365-Filtering-Correlation-Id: 4391690c-3c89-4abb-eb86-08dbcee4d910 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: fIhu5V9NaeTbtulm1Y8jdthAZ9FDnRyzNTA8rdvLDBOGGhzZDU0LUmWyYV9P/yE6qif88Qr2V5kPozPzZFKIx+kE9NciNL51xGVUgFzbr01DeezqkzjfMJZYYSSbeuEcs6F4+FIXG8ZLV/XLEDHg/TqIqv6k30lWLvjs/hVC+NQzsIp7lxsQ2Sx83jTlToEI9a2lGwoMRxe7f984SwgRbxUoz+dfP32+mAy/PJcPrNeXuCNDFgHrAcW9+cWyoQBm3aqUI6Say+GVhR4NoCEM5yUGjDcvDUKQ4rggoSdKM8/eECdXTJlWXAfy7MP2ubfQ3drkndfj8jBfs85u+YuQ0VeLKphb91Ft3KHY4IuoryF30CZOha1o1SraQxilHaD+aA7l7VpOWcWc+L2MtIyOKyrx29E/EsjqUoCOaD2uZx68dI0o9WiBuv/B+0cVQp+wy55GaFPBDMJCVOs3rkysazmu1lqTwy1KZZ26Mt26D8UzWBBeP0ZmWPHu+PxoBZ/1JMyA0k10c0M8bictCPDvHxOJmcJ825gQIC/b6kU+P1YaUtPhjSNbJfrZZHCMF77McIp1+RpSFja48mXp1Eyuhz94fKgFcyHRGBJpNEev0/AgpQeNozC/eMXV2h/Lj+fyDhSNQKjZu4A0Z4n0IvJqINeYJcFRHazZxcWP+SRCVQWMuvI0Z9EhRPJ/YLDl36NgK9Q+DLnUffKSmO0W208uZ5W6Y6ms0N6r+ZXD9xl/7cqmkOG9J4ny/indFeTe4nYr X-Forefront-Antispam-Report: CIP:216.228.117.160;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc6edge1.nvidia.com;CAT:NONE;SFS:(13230031)(4636009)(39860400002)(396003)(376002)(136003)(346002)(230922051799003)(64100799003)(186009)(1800799009)(82310400011)(451199024)(36840700001)(40470700004)(46966006)(478600001)(47076005)(70586007)(70206006)(110136005)(54906003)(26005)(16526019)(107886003)(41300700001)(336012)(1076003)(316002)(426003)(2616005)(4326008)(2906002)(8936002)(8676002)(5660300002)(36756003)(86362001)(7636003)(36860700001)(83380400001)(356005)(82740400003)(40460700003)(40480700001);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 17 Oct 2023 07:44:09.0655 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 4391690c-3c89-4abb-eb86-08dbcee4d910 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.117.160];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BL6PEPF0001AB77.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL1PR12MB5255 Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org The PCIe specification defines two methods to trigger a hot reset across a link: Bus reset and link disablement (r6.0.1, sec 7.1, sec 6.6.1). In the first method, the Secondary Bus Reset (SBR) bit in the Bridge Control Register of the Downstream Port is asserted for at least 1ms (r6.0.1, sec 7.5.1.3.13). In the second method, the Link Disable bit in the Link Control Register of the Downstream Port is asserted and then cleared to disable and enable the link (r6.0.1, sec 7.5.3.7). While the two methods are identical from the perspective of the Downstream device, they are different as far as the host is concerned. In the first method, the Link Training and Status State Machine (LTSSM) of the Downstream Port is expected to be in the Hot Reset state as long as the SBR bit is asserted. In the second method, the LTSSM of the Downstream Port is expected to be in the Disabled state as long as the Link Disable bit is asserted. This above difference is of importance because the specification requires the LTTSM to exit from the Hot Reset state to the Detect state within a 2ms timeout (r6.0.1, sec 4.2.7.11). NVIDIA Spectrum devices cannot guarantee it and a host enforcing such a behavior might fail to communicate with the device after issuing a Secondary Bus Reset. With the link disablement method, the host can leave the link disabled for enough time to allow the device to undergo a hot reset and reach the Detect state. After enabling the link, the host will exit from the Disabled state to Detect state (r6.0.1, sec 4.2.7.9) and observe that the device is already in the Detect state. The PCI core only implements the first method, which might not work with NVIDIA Spectrum devices on certain hosts, as explained above. Therefore, implement the link disablement method as a device-specific method for NVIDIA Spectrum devices. Specifically, disable the link, wait for 500ms, enable the link and then wait for the device to become accessible. Signed-off-by: Ido Schimmel --- drivers/pci/quirks.c | 29 +++++++++++++++++++++++++++++ 1 file changed, 29 insertions(+) diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c index 23f6bd2184e2..a6e308bb934c 100644 --- a/drivers/pci/quirks.c +++ b/drivers/pci/quirks.c @@ -4182,6 +4182,31 @@ static int reset_hinic_vf_dev(struct pci_dev *pdev, bool probe) return 0; } +#define PCI_DEVICE_ID_MELLANOX_SPECTRUM 0xcb84 +#define PCI_DEVICE_ID_MELLANOX_SPECTRUM2 0xcf6c +#define PCI_DEVICE_ID_MELLANOX_SPECTRUM3 0xcf70 +#define PCI_DEVICE_ID_MELLANOX_SPECTRUM4 0xcf80 + +static int reset_mlx(struct pci_dev *pdev, bool probe) +{ + struct pci_dev *bridge = pdev->bus->self; + + if (probe) + return 0; + + /* + * Disable the link on the Downstream port in order to trigger a hot + * reset in the Downstream device. Wait for 500ms before enabling the + * link so that the firmware on the device will have enough time to + * transition the Upstream port to the Detect state. + */ + pcie_capability_set_word(bridge, PCI_EXP_LNKCTL, PCI_EXP_LNKCTL_LD); + msleep(500); + pcie_capability_clear_word(bridge, PCI_EXP_LNKCTL, PCI_EXP_LNKCTL_LD); + + return pci_bridge_wait_for_secondary_bus(bridge, "link toggle"); +} + static const struct pci_dev_reset_methods pci_dev_reset_methods[] = { { PCI_VENDOR_ID_INTEL, PCI_DEVICE_ID_INTEL_82599_SFP_VF, reset_intel_82599_sfp_virtfn }, @@ -4197,6 +4222,10 @@ static const struct pci_dev_reset_methods pci_dev_reset_methods[] = { reset_chelsio_generic_dev }, { PCI_VENDOR_ID_HUAWEI, PCI_DEVICE_ID_HINIC_VF, reset_hinic_vf_dev }, + { PCI_VENDOR_ID_MELLANOX, PCI_DEVICE_ID_MELLANOX_SPECTRUM, reset_mlx }, + { PCI_VENDOR_ID_MELLANOX, PCI_DEVICE_ID_MELLANOX_SPECTRUM2, reset_mlx }, + { PCI_VENDOR_ID_MELLANOX, PCI_DEVICE_ID_MELLANOX_SPECTRUM3, reset_mlx }, + { PCI_VENDOR_ID_MELLANOX, PCI_DEVICE_ID_MELLANOX_SPECTRUM4, reset_mlx }, { 0 } }; From patchwork Tue Oct 17 07:42:51 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ido Schimmel X-Patchwork-Id: 13424701 X-Patchwork-Delegate: bhelgaas@google.com Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id BC8C6CDB474 for ; Tue, 17 Oct 2023 07:44:21 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234701AbjJQHoU (ORCPT ); Tue, 17 Oct 2023 03:44:20 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43712 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234722AbjJQHoS (ORCPT ); Tue, 17 Oct 2023 03:44:18 -0400 Received: from NAM10-MW2-obe.outbound.protection.outlook.com (mail-mw2nam10on2081.outbound.protection.outlook.com [40.107.94.81]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3B3B3106; Tue, 17 Oct 2023 00:44:15 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=oa8eJmDGTGlsyR4ZIHtEi04CNBmulXk1HgHTXRzLtjasUD2F9p+aKXWLNVUxSv6PSR9bq+WmGqTA3Cd9dQKgWPEoY6DXion+vEGMWqx6rFtqZQJ2q/FJX/c/QqLMhs+6Rx3GQYzVWlip3m9ftGeG39VBsJ5rXJJ78up9mkcCWHkKt1nTwtjlnH+YrgtfmYPIbbGQj5bvYaq434GS8gwpOh5yD5pqb89HHZI27OZJzJoQ4FoF+mWDcLbWYNXeuTTN2BIulw9fQkAwJhzFlHhYhyPedOFViFJTuJl+g85lG5EUCOw7tuYLO0RBRDsxXLqe/p/B4qjrt9m6XKxP/4px8Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=uGnYnIIBghGO5sxadBSTNx7BruaTJrIx1/d1ZKzo8HQ=; b=S52IDoE+APRkokwIJAAjuG6l+JwktanbATARbDhgjxzFqBZaJqOapNy9mC0CvRJqlQMtb+mA+dvIVEWNnGzR7uzkP/kv0kP0sbdm9j/hnaD4WAshtF/zVqSNwh2+ZdjtBYiyXMwQIq+qwZPfNqdLYcAoMHxKcKb2V++FmI3Y4oliZ2nr7VeFTXQDFIrCrsncjPPSTvPSbWxbOf0Hw+yxClBfhmFpJPWbS6YySp10iwnvPSnWnuUU2Zz2BI1xWxu1BAAiiZ+glDc7+EhdyqaHshXyK8KrnNhZWKs+7/BuuXCbDo1sZ+E2tstpMmxWT26UbGRbiR42UA1cud96Ff7Miw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.160) smtp.rcpttodomain=vger.kernel.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=uGnYnIIBghGO5sxadBSTNx7BruaTJrIx1/d1ZKzo8HQ=; b=KtqHFKu5Ytd2Oy7x7S6cH7CII0H985ypzau1X+SUBNTYbAAlJp55zsBm0zD9SeHU75Ecq51yeVXYpN7w1WaWBw3FnR6dULzEJqbp/Q+2ozNCNrLRSTdIjSqgtBfR7HdCv6ZcCiq4THnzY+x43BtuL3lv8kDIIbS/ZhfJ2vKWkluDunlT/yblA9ALh+pdOd1wAN4HyiO6rVljGf+fLVTIDcEheJSMlml9Bi4wo+lsS36IqMxFQyRvbnXRpuhau7C9d5TJ3ACo4sT6QyIUKucXRuwF/3TUxeyMB33OD5FXE7kvHxVDdKM59o0l2Lr8+WldtNA8Mvftmluv+1bvuoMfxQ== Received: from BL1PR13CA0153.namprd13.prod.outlook.com (2603:10b6:208:2bd::8) by MW4PR12MB7000.namprd12.prod.outlook.com (2603:10b6:303:20a::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6886.36; Tue, 17 Oct 2023 07:44:13 +0000 Received: from BL6PEPF0001AB76.namprd02.prod.outlook.com (2603:10b6:208:2bd:cafe::b3) by BL1PR13CA0153.outlook.office365.com (2603:10b6:208:2bd::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6907.18 via Frontend Transport; Tue, 17 Oct 2023 07:44:13 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.160) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.160 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.160; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.160) by BL6PEPF0001AB76.mail.protection.outlook.com (10.167.242.169) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6907.21 via Frontend Transport; Tue, 17 Oct 2023 07:44:12 +0000 Received: from rnnvmail201.nvidia.com (10.129.68.8) by mail.nvidia.com (10.129.200.66) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Tue, 17 Oct 2023 00:43:59 -0700 Received: from dev-r-vrt-155.mtr.labs.mlnx (10.126.230.35) by rnnvmail201.nvidia.com (10.129.68.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Tue, 17 Oct 2023 00:43:56 -0700 From: Ido Schimmel To: , CC: , , , , , , , , , , Ido Schimmel Subject: [RFC PATCH net-next 06/12] PCI: Add debug print for device ready delay Date: Tue, 17 Oct 2023 10:42:51 +0300 Message-ID: <20231017074257.3389177-7-idosch@nvidia.com> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20231017074257.3389177-1-idosch@nvidia.com> References: <20231017074257.3389177-1-idosch@nvidia.com> MIME-Version: 1.0 X-Originating-IP: [10.126.230.35] X-ClientProxiedBy: rnnvmail201.nvidia.com (10.129.68.8) To rnnvmail201.nvidia.com (10.129.68.8) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BL6PEPF0001AB76:EE_|MW4PR12MB7000:EE_ X-MS-Office365-Filtering-Correlation-Id: ca9585d7-aeec-4b8e-ea70-08dbcee4db53 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: MzoXim9qrHSwM5Vsbs18jaNlgq2n66lw9TvVQBmkbbXA4a/7X/DkyDvvE1v3TRVIBjGS0oQv2v7iodwkwMEFfZ1bVpcE6ars6QYid5lpBFl10VBLpGh8fvwSLQaJRZD8jHsU4Oq2XXJO5isFksowcjACwCX0VD3hDlJbXcQ0veqNI6DVBNXbBLIbLfmThy3ActAP5CpT4cK2x9dsy8LjGONi6pj4rqMlKMs2Ety27iLFGdbpfVI2VwVz65HhAqcdthB2l/2NZSEPC1FFFOsgl/vn2bjb+J0+zrGGo1bqKzlrPPjj1qBzf9WqgZlfTpTqPVVzlBZGLsQJpNfgAdKwIw3MzBmX4SBE3rTQ2UdCoXoywptNechFHmeFQ+zJm31PMu9Znl1z8BJQ4JWaqRKn40MJYTUGLTg2TprZENPURcqG2MvquWpefz//oOTnYXn7kZjlX96yExbrvRWYr6aXqdaMOs3NEZ+qpCb6RG35FWEdMUeM2oDK8O4JwMQzrwfILNGuHlvgWmbDGgtapjq5xtDwFalyp4NZ/Xxk06mlCzDGW/Uq4w87N7jC/+Y3G9ZK0iuwwEJX74fdKvkP00zJd4wLNOdxr4bvnfr0PqV2IFk7jItDD48nwHSIr375Re++KCCHyfpPsE8VJA4qmFmoo1n3VG/Hff24WEbyQxl78A9qYCjpxNe1JcuwNnWqiRJux7pXjKQkBqhox3aUBbM+L5SV5y83jR9lzWnWXy8xckQY/UF3eZhAJibLh0dF2p+T X-Forefront-Antispam-Report: CIP:216.228.117.160;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc6edge1.nvidia.com;CAT:NONE;SFS:(13230031)(4636009)(346002)(136003)(376002)(396003)(39860400002)(230922051799003)(64100799003)(186009)(1800799009)(451199024)(82310400011)(36840700001)(46966006)(40470700004)(316002)(40460700003)(70586007)(70206006)(54906003)(110136005)(16526019)(2616005)(1076003)(426003)(336012)(26005)(47076005)(107886003)(83380400001)(40480700001)(7636003)(356005)(86362001)(82740400003)(36860700001)(36756003)(478600001)(6666004)(2906002)(41300700001)(5660300002)(4326008)(8676002)(8936002);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 17 Oct 2023 07:44:12.8601 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: ca9585d7-aeec-4b8e-ea70-08dbcee4db53 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.117.160];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BL6PEPF0001AB76.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW4PR12MB7000 Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org Currently, the time it took a PCI device to become ready after reset is only printed if it was longer than 1000ms ('PCI_RESET_WAIT'). However, for debugging purposes it is useful to know this time even if it was shorter. For example, with the device I am working on, hardware engineers asked to verify that it becomes ready on the first try (no delay). To that end, add a debug level print that can be enabled using dynamic debug. Example: # echo 1 > /sys/bus/pci/devices/0000\:01\:00.0/reset # dmesg -c | grep ready # echo "file drivers/pci/pci.c +p" > /sys/kernel/debug/dynamic_debug/control # echo 1 > /sys/bus/pci/devices/0000\:01\:00.0/reset # dmesg -c | grep ready [ 396.060335] mlxsw_spectrum4 0000:01:00.0: ready 0ms after link toggle # echo "file drivers/pci/pci.c -p" > /sys/kernel/debug/dynamic_debug/control # echo 1 > /sys/bus/pci/devices/0000\:01\:00.0/reset # dmesg -c | grep ready Signed-off-by: Ido Schimmel Acked-by: Bjorn Helgaas --- drivers/pci/pci.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c index 59c01d68c6d5..0a708e65c5c4 100644 --- a/drivers/pci/pci.c +++ b/drivers/pci/pci.c @@ -1216,6 +1216,9 @@ static int pci_dev_wait(struct pci_dev *dev, char *reset_type, int timeout) if (delay > PCI_RESET_WAIT) pci_info(dev, "ready %dms after %s\n", delay - 1, reset_type); + else + pci_dbg(dev, "ready %dms after %s\n", delay - 1, + reset_type); return 0; } From patchwork Tue Oct 17 07:42:52 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ido Schimmel X-Patchwork-Id: 13424702 X-Patchwork-Delegate: bhelgaas@google.com Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0CCD4C46CA1 for ; Tue, 17 Oct 2023 07:44:26 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1343493AbjJQHoZ (ORCPT ); Tue, 17 Oct 2023 03:44:25 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43768 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234698AbjJQHoW (ORCPT ); Tue, 17 Oct 2023 03:44:22 -0400 Received: from NAM11-DM6-obe.outbound.protection.outlook.com (mail-dm6nam11on2082.outbound.protection.outlook.com [40.107.223.82]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1876E114; Tue, 17 Oct 2023 00:44:20 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=l7ugib52Pd4qBbqAmBcIlAZ2jhSEc6OEa4ulq7fRzVUY7A8NUykzFNIG8lL31ikJgSHL+xb4coJsvkypUOulH618N3LeCfl29Ce162yNpXWuREpqkMDJzNJZiAtA4xJ6uIdO5FEXKsSzQvA/CgZuUdQIG/6DwOAx3y4cc8f1UBoid3TeReCtMLRMoeGf32wzDRHA89os37RPywfy4XhfMUt8nnd/6Iaav+Db2C4Ao4OgKmW+1PlNUyU8kaGirgJMZvmc9xY83Nfu0WFHh76v6RPDnMNs8G/RyYxa6hYIoNVGMj/qPoPwcXYK6K1f6AZY4Y2s1NwYGEkOnKOjyWFXmw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=U+Wxkya7jtrraxFiIxfX4SLA1OlxnpDaImArPrytwp4=; b=ZcCbb36a8H9LuEvYKewO6lT8EUrM8yFIttmKKfAUTu6tGEZefrAC2iuvwr53R6BGSlmqUQkBRnInLF126A4w/EiRKPzUjvEtjggXzZQs8aAibhd0WBZ0VM2J7tgknB5gUBz3Xk3DKamHbnbKzTpZBjAprUuG/gRe8dM87ai2tM9F3UMG+pRlIXfxwfY/pvKqRhUBD58jyzbpo7HsxuqBA2pkxfmN3FEpEwpNVv4yFrEa1jd5meWu9W3hgVHOP/mnID+Jf/o56JErD+ESh90VcZ5Wrv9m4WcyOZliUyX/LyondoECIuXQajJbWhNmaIKL9K/52lJTiS8Px+tLpRPGcw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.160) smtp.rcpttodomain=vger.kernel.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=U+Wxkya7jtrraxFiIxfX4SLA1OlxnpDaImArPrytwp4=; b=tRR3C4vTrt6+Rq6OXwEveZsM6lGYvHg1s156Q/tAK+8QQ2BabqRwEZcPMGfP/1HlLQhw00AleJTF93JAz7D+DVY5aSe8E7M+mtuabtQKCX0v2cHbobZQ751M3QeCFbkFnL0ygEGtkfU+d9oNunt5Ya7b0c4Foxd98qkfo6kz7Ih3vk4ZfOeeXwQK+IKj4NZP/B8whqmXldgUx9hjRNgX3GpvfUj3MLf25ch1nPF2HUw7W7yNAtQeIB3k6DmlGEt7OR3YYiNRtGESs9k7Bl0ojm+RX63J34ZK2cd6We/jrykzgrfCl9jXhXFJdyrZsiA0xrx/NF2hDEbKjzZxo3xRfQ== Received: from BL1PR13CA0163.namprd13.prod.outlook.com (2603:10b6:208:2bd::18) by PH8PR12MB7350.namprd12.prod.outlook.com (2603:10b6:510:216::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6886.35; Tue, 17 Oct 2023 07:44:18 +0000 Received: from BL6PEPF0001AB76.namprd02.prod.outlook.com (2603:10b6:208:2bd:cafe::7c) by BL1PR13CA0163.outlook.office365.com (2603:10b6:208:2bd::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6907.18 via Frontend Transport; Tue, 17 Oct 2023 07:44:17 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.160) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.160 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.160; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.160) by BL6PEPF0001AB76.mail.protection.outlook.com (10.167.242.169) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6907.21 via Frontend Transport; Tue, 17 Oct 2023 07:44:17 +0000 Received: from rnnvmail201.nvidia.com (10.129.68.8) by mail.nvidia.com (10.129.200.66) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Tue, 17 Oct 2023 00:44:03 -0700 Received: from dev-r-vrt-155.mtr.labs.mlnx (10.126.230.35) by rnnvmail201.nvidia.com (10.129.68.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Tue, 17 Oct 2023 00:44:00 -0700 From: Ido Schimmel To: , CC: , , , , , , , , , , Amit Cohen , Ido Schimmel Subject: [RFC PATCH net-next 07/12] mlxsw: Extend MRSR pack() function to support new commands Date: Tue, 17 Oct 2023 10:42:52 +0300 Message-ID: <20231017074257.3389177-8-idosch@nvidia.com> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20231017074257.3389177-1-idosch@nvidia.com> References: <20231017074257.3389177-1-idosch@nvidia.com> MIME-Version: 1.0 X-Originating-IP: [10.126.230.35] X-ClientProxiedBy: rnnvmail201.nvidia.com (10.129.68.8) To rnnvmail201.nvidia.com (10.129.68.8) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BL6PEPF0001AB76:EE_|PH8PR12MB7350:EE_ X-MS-Office365-Filtering-Correlation-Id: 5de35ad0-aa49-4c7c-89c7-08dbcee4de40 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: OIN44ee5Nugsnm52TsPawf0b4D+XynEaPnWV4sFUxogBR9v11AEkKcLhdODsTFXFaqgqdboswioJLVbNx6akSHIiIofRNDGvusbqTunuW953usMsLqnIWaCTsRKd772axuU+mzbNnauLXO6RK1apRe4u2pI4CpSCIvum/sKIYGKkQSEUGOqQhFyBsF+zZvjovzZW1BrJRooQDXvHzFYH7DTYbLPhHZdsUcI8gHq+9JjS8jg3Cc7uZ9YSyF9nzjt7CD8iEHrSkhMxA0n/OZyoliSf0w1E2BvpsUU4/w8cw1oL4vzP9G70nMNajqWQA6DAfVHgp/QoXc/bXSY/K7wB9cWpgpaerYVrDLc+1cGmOx8/aI5DUiafLhUz+dTa08is7xLiu1A0Ze6ipHUB0twM+CcXzOL66xfp6Q4CTyLL551BIT6kghhVK+FmIpZmKjqdcmi261cWD5w0MxZjSNLt3GJho/jvt6Sz5xgnDuyYR0CGUc9TDDia54wWKmDzais6zZBGr47A8VD44l6NqDjFEuFvwADUhbFDGufwyWWiWkxUtQOxEknezAb/OTcw+S2WOwTNjITn2P1JPQWsqXKTr3Y90jJuw62O9cqyjgjjDkB6C+HnT+aa5nyvq7UfIeQLmbfUJPA4JVs1i17U5a+t6xmBZkApTcfPIRc9NgMmYuKKkpBam9ZTiLML6x5IG2Bp0ZEx+b6TxanMhPfEA70m1ekMqYTQWCbFDTr3l0kNcoo= X-Forefront-Antispam-Report: CIP:216.228.117.160;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc6edge1.nvidia.com;CAT:NONE;SFS:(13230031)(4636009)(136003)(396003)(39860400002)(346002)(376002)(230922051799003)(82310400011)(451199024)(1800799009)(186009)(64100799003)(36840700001)(46966006)(40470700004)(40480700001)(5660300002)(40460700003)(2906002)(26005)(36756003)(36860700001)(2616005)(426003)(336012)(107886003)(16526019)(6666004)(83380400001)(7636003)(86362001)(47076005)(82740400003)(356005)(1076003)(41300700001)(316002)(54906003)(110136005)(70586007)(70206006)(8676002)(4326008)(8936002)(478600001);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 17 Oct 2023 07:44:17.6881 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 5de35ad0-aa49-4c7c-89c7-08dbcee4de40 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.117.160];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BL6PEPF0001AB76.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH8PR12MB7350 Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org From: Amit Cohen Currently mlxsw_reg_mrsr_pack() always sets 'command=1'. As preparation for support of new reset flow, pass the command as an argument to the function and add an enum for this field. For now, always pass 'command=1' to the pack() function. Signed-off-by: Amit Cohen Reviewed-by: Petr Machata Signed-off-by: Ido Schimmel --- drivers/net/ethernet/mellanox/mlxsw/pci.c | 2 +- drivers/net/ethernet/mellanox/mlxsw/reg.h | 14 ++++++++++++-- 2 files changed, 13 insertions(+), 3 deletions(-) diff --git a/drivers/net/ethernet/mellanox/mlxsw/pci.c b/drivers/net/ethernet/mellanox/mlxsw/pci.c index 7fae963b2608..afa7df273202 100644 --- a/drivers/net/ethernet/mellanox/mlxsw/pci.c +++ b/drivers/net/ethernet/mellanox/mlxsw/pci.c @@ -1479,7 +1479,7 @@ static int mlxsw_pci_sw_reset(struct mlxsw_pci *mlxsw_pci, return err; } - mlxsw_reg_mrsr_pack(mrsr_pl); + mlxsw_reg_mrsr_pack(mrsr_pl, MLXSW_REG_MRSR_COMMAND_SOFTWARE_RESET); err = mlxsw_reg_write(mlxsw_pci->core, MLXSW_REG(mrsr), mrsr_pl); if (err) return err; diff --git a/drivers/net/ethernet/mellanox/mlxsw/reg.h b/drivers/net/ethernet/mellanox/mlxsw/reg.h index 9970921ceef3..44f528326394 100644 --- a/drivers/net/ethernet/mellanox/mlxsw/reg.h +++ b/drivers/net/ethernet/mellanox/mlxsw/reg.h @@ -10122,6 +10122,15 @@ mlxsw_reg_mgir_unpack(char *payload, u32 *hw_rev, char *fw_info_psid, MLXSW_REG_DEFINE(mrsr, MLXSW_REG_MRSR_ID, MLXSW_REG_MRSR_LEN); +enum mlxsw_reg_mrsr_command { + /* Switch soft reset, does not reset PCI firmware. */ + MLXSW_REG_MRSR_COMMAND_SOFTWARE_RESET = 1, + /* Reset will be done when PCI link will be disabled. + * This command will reset PCI firmware also. + */ + MLXSW_REG_MRSR_COMMAND_RESET_AT_PCI_DISABLE = 6, +}; + /* reg_mrsr_command * Reset/shutdown command * 0 - do nothing @@ -10130,10 +10139,11 @@ MLXSW_REG_DEFINE(mrsr, MLXSW_REG_MRSR_ID, MLXSW_REG_MRSR_LEN); */ MLXSW_ITEM32(reg, mrsr, command, 0x00, 0, 4); -static inline void mlxsw_reg_mrsr_pack(char *payload) +static inline void mlxsw_reg_mrsr_pack(char *payload, + enum mlxsw_reg_mrsr_command command) { MLXSW_REG_ZERO(mrsr, payload); - mlxsw_reg_mrsr_command_set(payload, 1); + mlxsw_reg_mrsr_command_set(payload, command); } /* MLCR - Management LED Control Register From patchwork Tue Oct 17 07:42:53 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ido Schimmel X-Patchwork-Id: 13424703 X-Patchwork-Delegate: bhelgaas@google.com Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6E91DCDB482 for ; Tue, 17 Oct 2023 07:44:27 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1343505AbjJQHo0 (ORCPT ); Tue, 17 Oct 2023 03:44:26 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43748 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1343491AbjJQHoX (ORCPT ); Tue, 17 Oct 2023 03:44:23 -0400 Received: from NAM10-MW2-obe.outbound.protection.outlook.com (mail-mw2nam10on2077.outbound.protection.outlook.com [40.107.94.77]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 33BD4116; Tue, 17 Oct 2023 00:44:20 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=FgHrtpF+gZRbIB5nPyRvUrn+8hPfBgmJRdoCYHwBxzYP6Zh4pYalFe7zsiAGoFydSXwN3Lzc821/D4AT7s1z/kWjhHbIaExyaH53/mVgRCARFaPCNyFG8a3vEQP+NlBaGqVljTEZCSUBmLEJ24/II2CwS6jIhThj61U1FZp949LsQfGYsBgWiPFcl2xpjlOAlYTnu1WzM3OCWJ/DOs0cU+37NZf2MVciRl9AIQEEciICjgXpqi/HKk7CCBiEIs1R2WFnRouqFA02/U+ayaqvgubOtNypxzv6ixZXVcPd2g1k/GJn0rpwQvxL6mkP3C2YEm+lO0HpD8xMOICs+weT6g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=AvrjWKnInD3FEV7atI/fMRbwGrB4JpQa0u66Dj1Pow4=; b=CjfuwBT56S5n4OE98WavkLLzXE5xo2UMAOQTz78M0mGSoRkmG+ZWgnX3IJGzPtnM0PEBliWV05d95AYZI53eu9LJGcRn1jULOPmK4TT9PpheZVlwm9L6YIJmFkdkDCqn2sy0SsQGv5zkyXYUS6N3F04HoB7Djs7sq+UcPjxHwg4bd9hkxFqidk6zb+krFid4z6WNckvHbDfrON8U+8yAGzZjwedsqNpvkBIXPHnzvxMCeeTV7dskaE1XMo7CnqijvqhOZHJ5CSgLY8c/xLCMx0Zth7lSza+ntFpLY/Pvu4vhKRwKhlEAYqQwsWy9T6iJZThJM656Ejc6AGl2ayt8Aw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.161) smtp.rcpttodomain=vger.kernel.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=AvrjWKnInD3FEV7atI/fMRbwGrB4JpQa0u66Dj1Pow4=; b=aZeIgtktILD1X0ZfeS8COThKNgHrhMforLZEYhqjUzGidCTLvVrpUA8x3UZwHLWN4J2ZW/FGa4VYXC6QPORoC8bu6Xn+3n2xaPP0BHKXB9Vw7iHNG0jkV1pOk5LyCqRz1JsExvmac5bUAXAwwFnjtWCf9qBnCnK4OUk1JIgShdNtajfM7zbS60HTKW7zemWZ8N6StgdUEicQ4TrZZiJdFcHjEua9Q2wRxpXk8/ii03kLBten1ZjyYa+YAgK/51nNqhjiAIDp6KnLtAWS+LbJaXfPoKe1VHpPyF76dcIA1A7aLxcOJ/aTJXDzBEOreqERzaTu6bmpiIaoC/Wr1xFq/g== Received: from CY5PR22CA0089.namprd22.prod.outlook.com (2603:10b6:930:65::20) by SA3PR12MB7923.namprd12.prod.outlook.com (2603:10b6:806:317::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6863.45; Tue, 17 Oct 2023 07:44:17 +0000 Received: from CY4PEPF0000EE35.namprd05.prod.outlook.com (2603:10b6:930:65:cafe::eb) by CY5PR22CA0089.outlook.office365.com (2603:10b6:930:65::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6886.36 via Frontend Transport; Tue, 17 Oct 2023 07:44:17 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.161) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.161 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.161; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.161) by CY4PEPF0000EE35.mail.protection.outlook.com (10.167.242.41) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6838.22 via Frontend Transport; Tue, 17 Oct 2023 07:44:17 +0000 Received: from rnnvmail201.nvidia.com (10.129.68.8) by mail.nvidia.com (10.129.200.67) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Tue, 17 Oct 2023 00:44:07 -0700 Received: from dev-r-vrt-155.mtr.labs.mlnx (10.126.230.35) by rnnvmail201.nvidia.com (10.129.68.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Tue, 17 Oct 2023 00:44:04 -0700 From: Ido Schimmel To: , CC: , , , , , , , , , , Amit Cohen , Ido Schimmel Subject: [RFC PATCH net-next 08/12] mlxsw: pci: Rename mlxsw_pci_sw_reset() Date: Tue, 17 Oct 2023 10:42:53 +0300 Message-ID: <20231017074257.3389177-9-idosch@nvidia.com> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20231017074257.3389177-1-idosch@nvidia.com> References: <20231017074257.3389177-1-idosch@nvidia.com> MIME-Version: 1.0 X-Originating-IP: [10.126.230.35] X-ClientProxiedBy: rnnvmail201.nvidia.com (10.129.68.8) To rnnvmail201.nvidia.com (10.129.68.8) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CY4PEPF0000EE35:EE_|SA3PR12MB7923:EE_ X-MS-Office365-Filtering-Correlation-Id: 84173d44-40ad-4c22-97be-08dbcee4de1a X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: QzS/yOHGxCIUfpXTuxbJHNGSHoBn6X+et2xpwH+kM97KhaBiEijG2lVYppT9ylazpCyJqdIkF0ehwcxIeJBryZu2QU6lI4I+aE5T5W1MU+BpTaLBCSMvjXsa8GT0wuX8xbbdhCLH+m0mWWO3FA48ygqy1IRSLEVcSoC46t0ZGwNUsWfqfb82dguc5Lrq3ugRrGQgFpkmQfg7LjMYlR316amF+IUJpsSt5uKSZ4aR7daB0KzRfNDF+CyYQVCXM6ShGavm5WQLAln4Rraukb4yh8vgLw8ry+QNSSUcKNHwDNcLc/0SNJ1hOFk+edijZ2PGwQ6cLaIAS5VFtabW4sSezj9hZitdc4tV5aaxjCwgQyx0XXHPCzW/toq1ZvjI3cPIvvA4Nz3t+LbwpEZKXNZ3Ahgd8PEaI0e6+uCozMIQFgqU5Fkq+/A3Dduy5BrxmV10zTkb8f36FCSdxeEviT5cocxS0yLzM+AEQQHa4QRoWH6h8IVFkpALyCZ7lAt1eE94wghGsd+KoBdPP0AwxNtuyNf/eq4LG4ra41p5m4aIuvew5WIEdAck7TzGuaMO5/sHJtkf88zuSJ/EPvmXPQVPL8PROJfobYG6g2VGcjYpzzyOVIitK2y/rUGS59bFnbgFocbAFgWnPJafJr/zrMGIMGDdPjYajsg/XpddmE1cpEU5dr+nphv3I29V1izQT2mW+o4KPTWwacfD6FfXusnuS9qs/MQwjkJc0AWL+b0/fj0Acmq1VhNfYkHL/178ItKJ X-Forefront-Antispam-Report: CIP:216.228.117.161;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc6edge2.nvidia.com;CAT:NONE;SFS:(13230031)(4636009)(376002)(136003)(39860400002)(396003)(346002)(230922051799003)(1800799009)(186009)(64100799003)(451199024)(82310400011)(36840700001)(46966006)(40470700004)(40460700003)(316002)(70206006)(70586007)(110136005)(54906003)(2616005)(6666004)(478600001)(107886003)(1076003)(8676002)(8936002)(4326008)(336012)(426003)(36756003)(5660300002)(86362001)(7636003)(36860700001)(26005)(16526019)(41300700001)(83380400001)(356005)(40480700001)(47076005)(2906002)(82740400003);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 17 Oct 2023 07:44:17.5801 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 84173d44-40ad-4c22-97be-08dbcee4de1a X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.117.161];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CY4PEPF0000EE35.namprd05.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA3PR12MB7923 Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org From: Amit Cohen In the next patches, mlxsw_pci_sw_reset() will be extended to support more reset types and will not necessarily issue a software reset. Rename the function to reflect that. Signed-off-by: Amit Cohen Reviewed-by: Petr Machata Signed-off-by: Ido Schimmel --- drivers/net/ethernet/mellanox/mlxsw/pci.c | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/drivers/net/ethernet/mellanox/mlxsw/pci.c b/drivers/net/ethernet/mellanox/mlxsw/pci.c index afa7df273202..af47d450332f 100644 --- a/drivers/net/ethernet/mellanox/mlxsw/pci.c +++ b/drivers/net/ethernet/mellanox/mlxsw/pci.c @@ -1464,8 +1464,8 @@ static int mlxsw_pci_sys_ready_wait(struct mlxsw_pci *mlxsw_pci, return -EBUSY; } -static int mlxsw_pci_sw_reset(struct mlxsw_pci *mlxsw_pci, - const struct pci_device_id *id) +static int +mlxsw_pci_reset(struct mlxsw_pci *mlxsw_pci, const struct pci_device_id *id) { struct pci_dev *pdev = mlxsw_pci->pdev; char mrsr_pl[MLXSW_REG_MRSR_LEN]; @@ -1525,9 +1525,9 @@ static int mlxsw_pci_init(void *bus_priv, struct mlxsw_core *mlxsw_core, if (!mbox) return -ENOMEM; - err = mlxsw_pci_sw_reset(mlxsw_pci, mlxsw_pci->id); + err = mlxsw_pci_reset(mlxsw_pci, mlxsw_pci->id); if (err) - goto err_sw_reset; + goto err_reset; err = mlxsw_pci_alloc_irq_vectors(mlxsw_pci); if (err < 0) { @@ -1659,7 +1659,7 @@ static int mlxsw_pci_init(void *bus_priv, struct mlxsw_core *mlxsw_core, err_query_fw: mlxsw_pci_free_irq_vectors(mlxsw_pci); err_alloc_irq: -err_sw_reset: +err_reset: mbox_put: mlxsw_cmd_mbox_free(mbox); return err; From patchwork Tue Oct 17 07:42:54 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ido Schimmel X-Patchwork-Id: 13424709 X-Patchwork-Delegate: bhelgaas@google.com Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 74BECCDB474 for ; Tue, 17 Oct 2023 07:44:34 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234724AbjJQHod (ORCPT ); Tue, 17 Oct 2023 03:44:33 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42572 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234708AbjJQHob (ORCPT ); Tue, 17 Oct 2023 03:44:31 -0400 Received: from NAM11-DM6-obe.outbound.protection.outlook.com (mail-dm6nam11on2085.outbound.protection.outlook.com [40.107.223.85]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9B30D10C; Tue, 17 Oct 2023 00:44:28 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=cRPTomM0b//O0jAo5TvPPu8mg98dQQYFBH/wML6pbxa7Yx3AEIoUzj7B5tMfmZeHihrGCaOLFjdBe6CVROBVmNDxXA5whUmGqJNeRYfh8Y6LjgAdFlsVSL2ulYhxDb9xZr4pHpfBvDedqd9cWxmDQji4ozn6LP66LXl0QhAhGgEEzp//0VakdJhqwQLequRf6YfcykMp96nATql0lVd8PmNl3ENE7roaWy4hY+fCKECtEk8BUlXWQPrjjNMvP203ZZTGCjW0B6oFJgPYFydpNJThXjjlXCSWe5THndOLv2rPzsgx8FvNW6O76Q8QfD/M5WrhxoQRmKvUERKN4S8Pmg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=0chJtnb3ns48sVZviSEcdyNMJg/VRQQAECZhiVjdrAI=; b=Qb7xh9nO0UISYpUZoii3LMVgIBUrZI8xnVAkjmwWyCMCIO9tGZ9rxiZpB80mHE8Zjnbf03p3qhxbdv2R7PcXeo8Sb0q3YxmVb8SfIs6XgpYyhHQi7QX6ZsKFRks047zx2zkG65Yt73elSnAD4rZkAPFJPi0JDQhfsc6a23Vw+RikDXtf9/IC9kVc8wR+m1EeSA1ts6w70cQ/5maXsvE5sybEajfFAIqamVyEEly6Y+5mANl3zbRAfQUUYvhRgaspT1Kl/aTaTpy8At3+ya+0Lk4EYZpUeVHCGWmW2bhaZ2vh6xZ+dR+azMt4iIGSYlJ0G1pmkMS55X7LMZREdTi3Jw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.160) smtp.rcpttodomain=vger.kernel.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=0chJtnb3ns48sVZviSEcdyNMJg/VRQQAECZhiVjdrAI=; b=HMxRj4CvK6kHmJBAlmNyPYjB155up+GsjeRp6rj6WhEe08g6oRrcN6BpbjE2xg83J//T6s6HG3+S/1PGATibJ2/CBk9B4yM3WVqqAOyvQhgnHUhC8RB0n5AVkpHK0NxwVUtavw1rrj2OgukkXpVad9quPn5y1SZfazA6QkYqcbKkFkD68GWgrBqdAbGlo5YYr8VviXBFYXtNPkMwAgesvxRJpENR9DI3HS4c1IXPBYEsVihYJq+um1ZKL+BlCz3vpUS5OR3bZ5YhVrIs3IKMJPJSbKXrlqeY7Zm+I7VuTcldQcmWiXKvfYPn7iPYx8HpBuOGaiSbj+UBxi5Bvcq61A== Received: from BL1PR13CA0173.namprd13.prod.outlook.com (2603:10b6:208:2bd::28) by CO6PR12MB5475.namprd12.prod.outlook.com (2603:10b6:5:354::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6907.21; Tue, 17 Oct 2023 07:44:26 +0000 Received: from BL6PEPF0001AB76.namprd02.prod.outlook.com (2603:10b6:208:2bd:cafe::3b) by BL1PR13CA0173.outlook.office365.com (2603:10b6:208:2bd::28) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6907.18 via Frontend Transport; Tue, 17 Oct 2023 07:44:25 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.160) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.160 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.160; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.160) by BL6PEPF0001AB76.mail.protection.outlook.com (10.167.242.169) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6907.21 via Frontend Transport; Tue, 17 Oct 2023 07:44:25 +0000 Received: from rnnvmail201.nvidia.com (10.129.68.8) by mail.nvidia.com (10.129.200.66) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Tue, 17 Oct 2023 00:44:11 -0700 Received: from dev-r-vrt-155.mtr.labs.mlnx (10.126.230.35) by rnnvmail201.nvidia.com (10.129.68.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Tue, 17 Oct 2023 00:44:07 -0700 From: Ido Schimmel To: , CC: , , , , , , , , , , Amit Cohen , Ido Schimmel Subject: [RFC PATCH net-next 09/12] mlxsw: pci: Move software reset code to a separate function Date: Tue, 17 Oct 2023 10:42:54 +0300 Message-ID: <20231017074257.3389177-10-idosch@nvidia.com> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20231017074257.3389177-1-idosch@nvidia.com> References: <20231017074257.3389177-1-idosch@nvidia.com> MIME-Version: 1.0 X-Originating-IP: [10.126.230.35] X-ClientProxiedBy: rnnvmail201.nvidia.com (10.129.68.8) To rnnvmail201.nvidia.com (10.129.68.8) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BL6PEPF0001AB76:EE_|CO6PR12MB5475:EE_ X-MS-Office365-Filtering-Correlation-Id: d69bb817-e09f-404f-120d-08dbcee4e2fe X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 6wTh/GRZjORzrulnHjP3iB7v7wPoYAUBVmuqI+Z0NI2wYjbo3FSqsGQxhH7eg5VucY7/iPhEv9ewAcdqLJ6rV5PjXlC06OUQGh/bEqC2V9tVAE2rCe/SdwdplJtKsUuLvAC/5NgTP0cZMg7eqGn6LzjtrxzP+ZfFpsKieopq8LwG5ouWzuaN+zTZKEBcV/xaw3IImxrD7drW52S7zzBBnNpR3YNKbWHKnek7agGPtpagB8HwqnZBiocYCbbsbFPFGn+stUD46MkCTI+62lo3wc91i/9npzBKR/gNXKtE7879yQBF5+AZPId4VdNXsp4jEuyzqIHXrf9Hen2y7xQshm/QbjOx0xvoJ/R7VDZIPEWfhVcTWgo/TVxoizZK3m/CwxWhsah/Xo/aVh5kG/ZcNQpsnkXh97zwFQH2DkrBXDSI6axr04Tn6OmNGB14Ij/lZnCVGhN646OAMgwLZ4YREVkVmXetlVP6U2VrFI0S+8rD1PN+7yrJ0VsgCydeI4kIuHc6Ob9v8mXoSnjy+M9HOmCuwdIfWX5/ZhFFAE501ibQLJyvVMV8EBabpzAOREbj6PE4dAhjgHElvzs8j4P9gx3XCoEBU8sPJ5rQ8DslIzH/7FjDHIPWiscyc0AOpcK9tpHvWwUX6Kon/7HXwYNnxf+drdNICFYjjm3dblDsCYc5MYP0U7NNqbTuFWyehj96Lmbobzy4v7wdGfabEoQyOoBQJEuHVfVejbxAOrl977Y= X-Forefront-Antispam-Report: CIP:216.228.117.160;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc6edge1.nvidia.com;CAT:NONE;SFS:(13230031)(4636009)(346002)(376002)(396003)(39860400002)(136003)(230922051799003)(1800799009)(451199024)(82310400011)(64100799003)(186009)(40470700004)(36840700001)(46966006)(82740400003)(356005)(7636003)(40460700003)(8936002)(8676002)(4326008)(41300700001)(5660300002)(86362001)(2906002)(40480700001)(478600001)(47076005)(6666004)(16526019)(426003)(336012)(107886003)(2616005)(36756003)(26005)(83380400001)(1076003)(54906003)(316002)(110136005)(70586007)(70206006)(36860700001);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 17 Oct 2023 07:44:25.7192 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: d69bb817-e09f-404f-120d-08dbcee4e2fe X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.117.160];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BL6PEPF0001AB76.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CO6PR12MB5475 Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org From: Amit Cohen In general, the existing flow of software reset in the driver is: 1. Wait for system ready status. 2. Send MRSR command, to start the reset. 3. Wait for system ready status. This flow will be extended once a new reset command is supported. As a preparation, move step #2 to a separate function. Signed-off-by: Amit Cohen Reviewed-by: Petr Machata Signed-off-by: Ido Schimmel --- drivers/net/ethernet/mellanox/mlxsw/pci.c | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/drivers/net/ethernet/mellanox/mlxsw/pci.c b/drivers/net/ethernet/mellanox/mlxsw/pci.c index af47d450332f..1980343ff873 100644 --- a/drivers/net/ethernet/mellanox/mlxsw/pci.c +++ b/drivers/net/ethernet/mellanox/mlxsw/pci.c @@ -1464,11 +1464,18 @@ static int mlxsw_pci_sys_ready_wait(struct mlxsw_pci *mlxsw_pci, return -EBUSY; } +static int mlxsw_pci_reset_sw(struct mlxsw_pci *mlxsw_pci) +{ + char mrsr_pl[MLXSW_REG_MRSR_LEN]; + + mlxsw_reg_mrsr_pack(mrsr_pl, MLXSW_REG_MRSR_COMMAND_SOFTWARE_RESET); + return mlxsw_reg_write(mlxsw_pci->core, MLXSW_REG(mrsr), mrsr_pl); +} + static int mlxsw_pci_reset(struct mlxsw_pci *mlxsw_pci, const struct pci_device_id *id) { struct pci_dev *pdev = mlxsw_pci->pdev; - char mrsr_pl[MLXSW_REG_MRSR_LEN]; u32 sys_status; int err; @@ -1479,8 +1486,7 @@ mlxsw_pci_reset(struct mlxsw_pci *mlxsw_pci, const struct pci_device_id *id) return err; } - mlxsw_reg_mrsr_pack(mrsr_pl, MLXSW_REG_MRSR_COMMAND_SOFTWARE_RESET); - err = mlxsw_reg_write(mlxsw_pci->core, MLXSW_REG(mrsr), mrsr_pl); + err = mlxsw_pci_reset_sw(mlxsw_pci); if (err) return err; From patchwork Tue Oct 17 07:42:55 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ido Schimmel X-Patchwork-Id: 13424710 X-Patchwork-Delegate: bhelgaas@google.com Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D2B98CDB483 for ; Tue, 17 Oct 2023 07:44:35 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234708AbjJQHoe (ORCPT ); Tue, 17 Oct 2023 03:44:34 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42556 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234718AbjJQHoc (ORCPT ); Tue, 17 Oct 2023 03:44:32 -0400 Received: from NAM12-DM6-obe.outbound.protection.outlook.com (mail-dm6nam12on2078.outbound.protection.outlook.com [40.107.243.78]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 45B9D124; Tue, 17 Oct 2023 00:44:29 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=kCq8abt49Atv70bttgoHO+B06kvVngCYty2Qk62FfdCmvae6f4xkdWNT6a3QYJ3nKwX4kyNiv1vjSyeIWAw43DtHW99+IyXdFwIvVyDSlhqV8CDR3j7D+3eWQ2vv/1JuNPqawvPtFEy3pIeo9ESFiVQLT1OLvTv2duG6XMhP7PxRWlMr3DY1k+4eyOPiEfErcYPiiZpzCqPAOxZts1wpmLWeXY/z1TG+wcSeu5Ydkx9b2arTrEE4DCX6/xTsNYPUXb6bqGuejBVbKxdOu7ejYOFP6lNbga2qTvzN4CJAMpswMw8x5ikdlkI85IoqnJ658u3fa0vr3CABiSiSCOy9Pw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=7vqs+C06eO7czzQVki9lE5wuURywSDuMWoKlaORdiDo=; b=Uex6y1v9hJJ44Qbmf1vOjyUW5LlimXi//jVrSlIEQPUmSqypwZUFhwn9fWt9sVHXnQHZGfIGLSABaqW9cCiK6KQVW6HDe2aYNHNCPd4hpMtOIM3F+rQJ9N7L8hzE96GFsZVrvKjM0bOyJCZ7NECtaMBMdEtina6Q6ls/JT9eKI8czaZz61nvCq0CNyUE7IK+1UU3NqsZMv/wlBKEv9BsFtMr7hvVfGW6MoYD1wKJ3GDFgZ7EU7qpbWphqcbFjbP8fhhSxvTb8JCPJGc5SUySYKv9uu0L3futHcjGgOCSMpijimU2vBYWApoDu4O3FFopqp0W4gyOdeoCbjy7wA9D+w== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.161) smtp.rcpttodomain=vger.kernel.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=7vqs+C06eO7czzQVki9lE5wuURywSDuMWoKlaORdiDo=; b=dSUmTm9pOgi6iN4D5cdiW2BFFjnQVbRYZzoLDgXv2DiMQqSjD2e+GQq4Vqt20Oiin41PknmYe0OZMI2MeiplCNU9F6T7qYC7Z9pnkqwVyC6UGMgSQrCHxy4YQEP+bwXWQoMwGJWPCGy1C0ehcBGE//muijgHV07h+SmiFe/TQ201nIv1rca66oUsB2rlWnwRy+/hp0faK4ZO0Rr6szuu0t4+sO3jLTRVDU6/WJYre1/fviBBT9e3USM8pjppXJJjD+Ax/TyHc6vpp3V4XkYS0vTyE2qlIpqMomRGJtezeDMFxXbCfkjkcACrKNlbyBRAt6q5l0EFTS7vBo6Na3/Rcg== Received: from CY5PR22CA0107.namprd22.prod.outlook.com (2603:10b6:930:65::17) by LV8PR12MB9182.namprd12.prod.outlook.com (2603:10b6:408:192::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6863.38; Tue, 17 Oct 2023 07:44:27 +0000 Received: from CY4PEPF0000EE35.namprd05.prod.outlook.com (2603:10b6:930:65:cafe::59) by CY5PR22CA0107.outlook.office365.com (2603:10b6:930:65::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6886.36 via Frontend Transport; Tue, 17 Oct 2023 07:44:27 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.161) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.161 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.161; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.161) by CY4PEPF0000EE35.mail.protection.outlook.com (10.167.242.41) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6838.22 via Frontend Transport; Tue, 17 Oct 2023 07:44:26 +0000 Received: from rnnvmail201.nvidia.com (10.129.68.8) by mail.nvidia.com (10.129.200.67) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Tue, 17 Oct 2023 00:44:15 -0700 Received: from dev-r-vrt-155.mtr.labs.mlnx (10.126.230.35) by rnnvmail201.nvidia.com (10.129.68.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Tue, 17 Oct 2023 00:44:11 -0700 From: Ido Schimmel To: , CC: , , , , , , , , , , Ido Schimmel Subject: [RFC PATCH net-next 10/12] mlxsw: pci: Add support for new reset flow Date: Tue, 17 Oct 2023 10:42:55 +0300 Message-ID: <20231017074257.3389177-11-idosch@nvidia.com> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20231017074257.3389177-1-idosch@nvidia.com> References: <20231017074257.3389177-1-idosch@nvidia.com> MIME-Version: 1.0 X-Originating-IP: [10.126.230.35] X-ClientProxiedBy: rnnvmail201.nvidia.com (10.129.68.8) To rnnvmail201.nvidia.com (10.129.68.8) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CY4PEPF0000EE35:EE_|LV8PR12MB9182:EE_ X-MS-Office365-Filtering-Correlation-Id: dd83813e-52aa-46ad-5589-08dbcee4e3ac X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: wgeSfGR0RS3z8+4W7X/GG3zzgYxIUdRZrxABG1u+1OswTTUV7rlYpsNp1CLH7i/ROSEeKQfVPBXU63LT0sI35wAruQ4lOtqMH+DL3Nx5DW4CSGAc7FDw62Xkk7Iorz7weoCVDnQJ4E3TsH+o6ormW3M6JhxnNrP2crEkHI8oCfwutKKJ3ME4fCFPX5qEfLGyAN9ibHv0pDfUC1R9QN0v24HyvSAz1WyoOd1MoNY3pQMX37pWwxShj7Qgh3Rh2Z/9y1DqdJyvmzpfTaSMzWd+OgET6YB1fdVyb1jm3ebZqKRwVChrd7a0QwHGc3c2YXy/0BOuGiPWihcxGrCTmxT+UQtS3CgKReKoaq6ATKyUaHdMoxBcbZh1eKrEfDr1dTOJm1pIwFt1sP3NWG3GGdtKwx4+y5Tfc13OM0Qtf8szAsro2tjY5iSulWLpZygd28YwvW3oAfY3Pglq9kcKoK4bXAScRv9/3x+++yvMZ6dMsGVcdQvjEJ4ssmckno26X+ovXChZLIgpXvp7/BF997wtwioumFJyfPvAcRSQU3+16uMDJqVJgPkG2A0b4+VXtGuKZuLy201B0q/miSKw8jP5qJ3Y8jfvDiSh1jzzfAm1JMjcm8Z7iUpdCJpDSOdudziHt4hkiFAv0SZDd9/FIpnmwHGAGwAP/iEUZXv1PQ6GJqu7nN2WIqBNXiMNqsy2qiK959To8nkUV+ZDL8vr5Sd6YfziRwQRMNG3oE6lQUGjUWEOgs4/7KIDm7LacH8kUXj1 X-Forefront-Antispam-Report: CIP:216.228.117.161;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc6edge2.nvidia.com;CAT:NONE;SFS:(13230031)(4636009)(136003)(39860400002)(346002)(396003)(376002)(230922051799003)(451199024)(186009)(1800799009)(64100799003)(82310400011)(36840700001)(40470700004)(46966006)(8936002)(40480700001)(478600001)(316002)(47076005)(54906003)(40460700003)(110136005)(26005)(107886003)(2616005)(336012)(70586007)(16526019)(83380400001)(70206006)(426003)(36860700001)(82740400003)(8676002)(2906002)(6666004)(1076003)(5660300002)(4326008)(356005)(36756003)(7636003)(41300700001)(86362001);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 17 Oct 2023 07:44:26.9082 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: dd83813e-52aa-46ad-5589-08dbcee4e3ac X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.117.161];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CY4PEPF0000EE35.namprd05.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: LV8PR12MB9182 Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org The driver resets the device during probe and during a devlink reload. The current reset method reloads the current firmware version or a pending one, if one was previously flashed using devlink. However, the current reset method does not result in a PCI hot reset, preventing the PCI firmware from being upgraded, unless the system is rebooted. To solve this problem, a new reset command (6) was implemented in the firmware. Unlike the current command (1), after issuing the new command the device will not start the reset immediately, but only after a PCI hot reset. Implement the new reset method by first verifying that it is supported by the current firmware version by querying the Management Capabilities Mask (MCAM) register. If supported, issue the new reset command (6) via MRSR register followed by a PCI reset by calling __pci_reset_function_locked(). Once the PCI firmware is operational, go back to the regular reset flow and wait for the entire device to become ready. That is, repeatedly read the "system_status" register from the BAR until a value of "FW_READY" (0x5E) appears. Tested: # for i in $(seq 1 10); do devlink dev reload pci/0000:01:00.0; done Signed-off-by: Ido Schimmel --- drivers/net/ethernet/mellanox/mlxsw/pci.c | 44 ++++++++++++++++++++++- drivers/net/ethernet/mellanox/mlxsw/reg.h | 2 ++ 2 files changed, 45 insertions(+), 1 deletion(-) diff --git a/drivers/net/ethernet/mellanox/mlxsw/pci.c b/drivers/net/ethernet/mellanox/mlxsw/pci.c index 1980343ff873..b5bb47b0215f 100644 --- a/drivers/net/ethernet/mellanox/mlxsw/pci.c +++ b/drivers/net/ethernet/mellanox/mlxsw/pci.c @@ -1464,6 +1464,33 @@ static int mlxsw_pci_sys_ready_wait(struct mlxsw_pci *mlxsw_pci, return -EBUSY; } +static int mlxsw_pci_reset_at_pci_disable(struct mlxsw_pci *mlxsw_pci) +{ + struct pci_dev *pdev = mlxsw_pci->pdev; + char mrsr_pl[MLXSW_REG_MRSR_LEN]; + int err; + + mlxsw_reg_mrsr_pack(mrsr_pl, + MLXSW_REG_MRSR_COMMAND_RESET_AT_PCI_DISABLE); + err = mlxsw_reg_write(mlxsw_pci->core, MLXSW_REG(mrsr), mrsr_pl); + if (err) + return err; + + device_lock_assert(&pdev->dev); + + pci_cfg_access_lock(pdev); + pci_save_state(pdev); + + err = __pci_reset_function_locked(pdev); + if (err) + pci_err(pdev, "PCI function reset failed with %d\n", err); + + pci_restore_state(pdev); + pci_cfg_access_unlock(pdev); + + return err; +} + static int mlxsw_pci_reset_sw(struct mlxsw_pci *mlxsw_pci) { char mrsr_pl[MLXSW_REG_MRSR_LEN]; @@ -1476,6 +1503,8 @@ static int mlxsw_pci_reset(struct mlxsw_pci *mlxsw_pci, const struct pci_device_id *id) { struct pci_dev *pdev = mlxsw_pci->pdev; + char mcam_pl[MLXSW_REG_MCAM_LEN]; + bool pci_reset_supported; u32 sys_status; int err; @@ -1486,10 +1515,23 @@ mlxsw_pci_reset(struct mlxsw_pci *mlxsw_pci, const struct pci_device_id *id) return err; } - err = mlxsw_pci_reset_sw(mlxsw_pci); + mlxsw_reg_mcam_pack(mcam_pl, + MLXSW_REG_MCAM_FEATURE_GROUP_ENHANCED_FEATURES); + err = mlxsw_reg_query(mlxsw_pci->core, MLXSW_REG(mcam), mcam_pl); if (err) return err; + mlxsw_reg_mcam_unpack(mcam_pl, MLXSW_REG_MCAM_PCI_RESET, + &pci_reset_supported); + + if (pci_reset_supported) { + pci_dbg(pdev, "Starting PCI reset flow\n"); + err = mlxsw_pci_reset_at_pci_disable(mlxsw_pci); + } else { + pci_dbg(pdev, "Starting software reset flow\n"); + err = mlxsw_pci_reset_sw(mlxsw_pci); + } + err = mlxsw_pci_sys_ready_wait(mlxsw_pci, id, &sys_status); if (err) { dev_err(&pdev->dev, "Failed to reach system ready status after reset. Status is 0x%x\n", diff --git a/drivers/net/ethernet/mellanox/mlxsw/reg.h b/drivers/net/ethernet/mellanox/mlxsw/reg.h index 44f528326394..c314afd4a8ff 100644 --- a/drivers/net/ethernet/mellanox/mlxsw/reg.h +++ b/drivers/net/ethernet/mellanox/mlxsw/reg.h @@ -10594,6 +10594,8 @@ MLXSW_ITEM32(reg, mcam, feature_group, 0x00, 16, 8); enum mlxsw_reg_mcam_mng_feature_cap_mask_bits { /* If set, MCIA supports 128 bytes payloads. Otherwise, 48 bytes. */ MLXSW_REG_MCAM_MCIA_128B = 34, + /* If set, MRSR.command=6 is supported. */ + MLXSW_REG_MCAM_PCI_RESET = 48, }; #define MLXSW_REG_BYTES_PER_DWORD 0x4 From patchwork Tue Oct 17 07:42:56 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ido Schimmel X-Patchwork-Id: 13424711 X-Patchwork-Delegate: bhelgaas@google.com Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 21127CDB474 for ; Tue, 17 Oct 2023 07:44:42 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234735AbjJQHok (ORCPT ); Tue, 17 Oct 2023 03:44:40 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42722 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234689AbjJQHoh (ORCPT ); Tue, 17 Oct 2023 03:44:37 -0400 Received: from NAM12-MW2-obe.outbound.protection.outlook.com (mail-mw2nam12on2064.outbound.protection.outlook.com [40.107.244.64]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 522B5124; Tue, 17 Oct 2023 00:44:34 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=ndIqh2proNqp7wGzOyezUsPm0FAXnjtmQbUQjLSGdBmqIIe1LOTVfWPWHDYKhUjIS22c0+Z2v6S8tPynYFMu7Jyvmc6//Bm+N0zZbDPCs8VxNzwhn9DNHdyt+T6T1fmNfuknoy+r4gvJjHA5sVo0ufYkPiHtiKbLRF0aIlgFDpw3kdpBA69rdtqaU2q1gbcGvA4MH6T7Lag42VluT6q/C+ScDCv/vVJcgYRurppr3+jQwOaHGF5VQy4O9J5UCOozwrehgo0/f8C/MzWwTvA/LsIpNfaFgSiyGB+yTn1GGMVyd7UUL3Tpp3muEXDoQQvxuXrWDymHgoILXiL8J2K5CA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=tRWUnk6t5TArjd2YyK/IisbcCMSDer+3jvzA+PByoB0=; b=baKm/zHlGitLPysIjossdn6yQSUc6UdjGSqS2F73ji3n70ZxBKJBuXHocfFDKb0OIPRBsj5KEY+hwtKTZgxwCkQ7WuWr0QJ5W/x/MluiVkObErRGRo0sjt5VrIst2lUhg6YJLxNuT34QgtHuI5KTp1DafPmUUgeFqUSSceTZP9to/BaxsdtCxi1XRkfp2VMd/K9uLzWMrZ0+/9xFAO+2T29XKSXUovQu7TcgBAW1SUx+FzlSqgQ5V1Fgr4UzYAHVF4dMHwiq/vdIgZ+0mXsDNJlUvqVFsjMykjhOfWgz8JAwZ3CicgJMyzfClwpyezu18ld+iShgoPMPj3FV3JL0zQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.161) smtp.rcpttodomain=vger.kernel.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=tRWUnk6t5TArjd2YyK/IisbcCMSDer+3jvzA+PByoB0=; b=rcrS/2uHx2of//PW2CznQRCq1BWDvlYbYg9Dkix8d4jaiqCXMxJQ6V5Gaf+AZym+ArKPXLnbaCh1m1/pDAamBVhP1fomFoq2J6lHPV7Ip1VMEHpVOjmVMT/onF0mF2DSUtD0sAL7oGama9b7DEj7MA71BU+FGfZURfreGvFs4qDw6aS82vXhbgwJjwVMx1UMQ4QLb0w6HWcG7iLccAkJZE70dsLne7J4rFAz+ejPWQhmUro4qwKuMsHhDZoCMbCXgCCxOtPULVb7zgzxAjvSHvKpsRHAkGmuj5HoL4SqLTCXqzRnZi+1KtuYZe6cdcPtgxsWTeq560fRdNE4EHmo0w== Received: from CY5PR22CA0087.namprd22.prod.outlook.com (2603:10b6:930:65::6) by BY5PR12MB5511.namprd12.prod.outlook.com (2603:10b6:a03:1d8::13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6907.21; Tue, 17 Oct 2023 07:44:31 +0000 Received: from CY4PEPF0000EE35.namprd05.prod.outlook.com (2603:10b6:930:65:cafe::ea) by CY5PR22CA0087.outlook.office365.com (2603:10b6:930:65::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6886.36 via Frontend Transport; Tue, 17 Oct 2023 07:44:31 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.161) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.161 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.161; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.161) by CY4PEPF0000EE35.mail.protection.outlook.com (10.167.242.41) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6838.22 via Frontend Transport; Tue, 17 Oct 2023 07:44:31 +0000 Received: from rnnvmail201.nvidia.com (10.129.68.8) by mail.nvidia.com (10.129.200.67) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Tue, 17 Oct 2023 00:44:19 -0700 Received: from dev-r-vrt-155.mtr.labs.mlnx (10.126.230.35) by rnnvmail201.nvidia.com (10.129.68.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Tue, 17 Oct 2023 00:44:15 -0700 From: Ido Schimmel To: , CC: , , , , , , , , , , Ido Schimmel Subject: [RFC PATCH net-next 11/12] mlxsw: pci: Implement PCI reset handlers Date: Tue, 17 Oct 2023 10:42:56 +0300 Message-ID: <20231017074257.3389177-12-idosch@nvidia.com> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20231017074257.3389177-1-idosch@nvidia.com> References: <20231017074257.3389177-1-idosch@nvidia.com> MIME-Version: 1.0 X-Originating-IP: [10.126.230.35] X-ClientProxiedBy: rnnvmail201.nvidia.com (10.129.68.8) To rnnvmail201.nvidia.com (10.129.68.8) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CY4PEPF0000EE35:EE_|BY5PR12MB5511:EE_ X-MS-Office365-Filtering-Correlation-Id: 12f1d7b0-e023-46cb-83de-08dbcee4e664 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: g+LbF0wRSmfxtvtPbaxcRme+VBk2a93suCi1XErMN8b96ByABXogutb2y7Al6ACsnLmhK5LOF2nSKNQ6L/ARJgkN+q/+33tY00G+Ut5rMjpgacYN5j4EAfcOnA0SPsEQFOgnKNlQITku04mG9sy5QfN6ni7KECp1p/r1I7cL6QI2s+W+BOlrP++JfzERcKyB3eG8vIsm8vejcHNuKJd++rWFbP6TPn0mUDf5CAn0J+NoXkFrT5KIyG8ZHLw+R4dLLyTSyAfGDX8zmcKDba10U39ix1gUF+YIxwQtkMVydn1JzFmOPQ4caiUt42GErfwwNVIx8x4VWStJ6MHkWUfwBCvw9dsJoGwjDAgFMBXkY5/pFNlyZep/GsZLno5NtV/3WGK3IpcYFvaZeLy6tAAEb9X9Q8EyBrm4QsKq2wyN0pdeRDZtgse23Vnd8aCImGS1jN0iGe/qp/EBVUZbtqqtdmdiCn+CWa3YxbJJ8lOVBGVfERIoYaR7DSpBQD4VtpJy5ePeO8DSHuT/qYz/UZ4rgsxR84wneyJJu6qzaZDjHn48HTTuGfcWms+b5SDHfGfqDNYE5F6jnvLspOLA+CXh1XdpzzZ6O24d4aAcxC5SiXlZCdURfCqYUSnHjTZrFK3w3daQnU8fohkE++jjLLzeuWbsjrcmbtN15sb0e27d6J+vi5FyT6gGbQ2fCGeAEcZFtXttDD72pb8x0KQ0ORB7M+izXOm+r+idYIleSMWvtQatIFGGPI4mbz4+TrWPaet6 X-Forefront-Antispam-Report: CIP:216.228.117.161;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc6edge2.nvidia.com;CAT:NONE;SFS:(13230031)(4636009)(396003)(136003)(39860400002)(346002)(376002)(230922051799003)(82310400011)(186009)(1800799009)(64100799003)(451199024)(40470700004)(46966006)(36840700001)(40480700001)(36756003)(40460700003)(110136005)(316002)(54906003)(70586007)(70206006)(86362001)(82740400003)(7636003)(356005)(8936002)(36860700001)(426003)(336012)(107886003)(26005)(1076003)(2616005)(16526019)(6666004)(2906002)(478600001)(41300700001)(47076005)(5660300002)(8676002)(4326008);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 17 Oct 2023 07:44:31.4863 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 12f1d7b0-e023-46cb-83de-08dbcee4e664 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.117.161];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CY4PEPF0000EE35.namprd05.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY5PR12MB5511 Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org Implement reset_prepare() and reset_done() handlers that are invoked by the PCI core before and after issuing a PCI reset, respectively. Specifically, implement reset_prepare() by calling mlxsw_core_bus_device_unregister() and reset_done() by calling mlxsw_core_bus_device_register(). This is the same implementation as the reload_{down,up}() devlink operations with the following differences: 1. The devlink instance is unregistered and then registered again after the reset. 2. A reset via the device's command interface (using MRSR register) is not issued during reset_done() as PCI core already issued a PCI reset. Tested: # for i in $(seq 1 10); do echo 1 > /sys/bus/pci/devices/0000\:01\:00.0/reset; done Reviewed-by: Petr Machata Signed-off-by: Ido Schimmel --- drivers/net/ethernet/mellanox/mlxsw/pci.c | 28 +++++++++++++++++++++++ 1 file changed, 28 insertions(+) diff --git a/drivers/net/ethernet/mellanox/mlxsw/pci.c b/drivers/net/ethernet/mellanox/mlxsw/pci.c index b5bb47b0215f..8de953902918 100644 --- a/drivers/net/ethernet/mellanox/mlxsw/pci.c +++ b/drivers/net/ethernet/mellanox/mlxsw/pci.c @@ -128,6 +128,7 @@ struct mlxsw_pci { const struct pci_device_id *id; enum mlxsw_pci_cqe_v max_cqe_ver; /* Maximal supported CQE version */ u8 num_sdq_cqs; /* Number of CQs used for SDQs */ + bool skip_reset; }; static void mlxsw_pci_queue_tasklet_schedule(struct mlxsw_pci_queue *q) @@ -1515,6 +1516,10 @@ mlxsw_pci_reset(struct mlxsw_pci *mlxsw_pci, const struct pci_device_id *id) return err; } + /* PCI core already issued a PCI reset, do not issue another reset. */ + if (mlxsw_pci->skip_reset) + return 0; + mlxsw_reg_mcam_pack(mcam_pl, MLXSW_REG_MCAM_FEATURE_GROUP_ENHANCED_FEATURES); err = mlxsw_reg_query(mlxsw_pci->core, MLXSW_REG(mcam), mcam_pl); @@ -2085,11 +2090,34 @@ static void mlxsw_pci_remove(struct pci_dev *pdev) kfree(mlxsw_pci); } +static void mlxsw_pci_reset_prepare(struct pci_dev *pdev) +{ + struct mlxsw_pci *mlxsw_pci = pci_get_drvdata(pdev); + + mlxsw_core_bus_device_unregister(mlxsw_pci->core, false); +} + +static void mlxsw_pci_reset_done(struct pci_dev *pdev) +{ + struct mlxsw_pci *mlxsw_pci = pci_get_drvdata(pdev); + + mlxsw_pci->skip_reset = true; + mlxsw_core_bus_device_register(&mlxsw_pci->bus_info, &mlxsw_pci_bus, + mlxsw_pci, false, NULL, NULL); + mlxsw_pci->skip_reset = false; +} + +static const struct pci_error_handlers mlxsw_pci_err_handler = { + .reset_prepare = mlxsw_pci_reset_prepare, + .reset_done = mlxsw_pci_reset_done, +}; + int mlxsw_pci_driver_register(struct pci_driver *pci_driver) { pci_driver->probe = mlxsw_pci_probe; pci_driver->remove = mlxsw_pci_remove; pci_driver->shutdown = mlxsw_pci_remove; + pci_driver->err_handler = &mlxsw_pci_err_handler; return pci_register_driver(pci_driver); } EXPORT_SYMBOL(mlxsw_pci_driver_register); From patchwork Tue Oct 17 07:42:57 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ido Schimmel X-Patchwork-Id: 13424712 X-Patchwork-Delegate: bhelgaas@google.com Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 510CECDB474 for ; Tue, 17 Oct 2023 07:44:46 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234689AbjJQHop (ORCPT ); Tue, 17 Oct 2023 03:44:45 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38716 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234726AbjJQHon (ORCPT ); Tue, 17 Oct 2023 03:44:43 -0400 Received: from NAM12-DM6-obe.outbound.protection.outlook.com (mail-dm6nam12on2042.outbound.protection.outlook.com [40.107.243.42]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 80FBA95; Tue, 17 Oct 2023 00:44:40 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=XUqakudbFl2Qb+DCT4GweEEX2wEofOPze8PW423VWl6kaPH9hD84dVXC0+0BC/mrVhYWvtzfTiAnABvLvruKNTnPDiUEX1Zcq2F8chS+ATCINtimG1mnUYXj5VG3Ub2v2XrtirwxrOfnpER+eeGAHp3DX6lQViL4bGxyriGE+j26WJq2cf62NQENmtV1evAie2H59IE8N0+wCq0H6ejNz4sDaTBVEiTies3q3RV0Ht5u4n/5Opw1DA+m4FA8kf16V3NB3vcFGHnFPuiCRXxaVmzRG6w/IuVZcyXMyT+0kVtgeu84PqEXA8033XiwPRzBvFaSoaMnpfvfmam4709PSQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=0ySBn8rgWCnhaBR3SuwY+BznTeAwRhbcFy4gg/uvGpk=; b=ipdNCWReu1kxM4kfO/I/EaRINwOSa8EH2C44qMGsLjBHHMkbs3x835YlsrLFieKMTkoUva23YFjLouaabdQaPsYcw78wTJKqtfIhRhicRa5D9dHlZbYrYdpcVSOyyLTfxR6NY/s40N8QANAksMc1XgbLFMXo7wX38QGXGzU32Vtj6LFAXDa0ogQ55My91Fz+2tbhZJXjdm/9+GSb6NqGsVOylGjpO41wgNzvYfffxT3DGCEAfzXgdyW4Wq6KiHCAi4MvtofglveoCzodF6RvdnVhkkvWZ1lXpeKxVSnCFwX/fQR40SNZdsC8hTh5vrNhxWi5Ief/KdW/1Ns9ShacyQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.160) smtp.rcpttodomain=vger.kernel.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=0ySBn8rgWCnhaBR3SuwY+BznTeAwRhbcFy4gg/uvGpk=; b=W3oE10QDk1D+IImh8he1PxBtBk+B6ae9YYh2t67quPmwb2yaa1bkbVGjbUTD/L9xjM4ItEGYVoLPIemz35muSK7YIqacOADTsWncmB3VRX0+9dVOJ7/PkHMbN3NKy+Aw9tV/GCXM33opS9bpjzO16seVmvFZVSiuLTkc/XM+Vwwhtwtl3/2Qv0JoSG3+o1ibGDIi7q0wrJoj0HLpBmNi/4KbSoodOieSJ6bwehCdXWMl1MVgVNSmUEvuBpOFRFJm5dcbD1qn9MsaKq+Up2/FkBxW/UT9NeEa6KHSZn6SjbkvDzasw9x06OUQ+ivEKxHaJLIzA2aJ+ZHBv4njAcHj1w== Received: from BL1PR13CA0144.namprd13.prod.outlook.com (2603:10b6:208:2bb::29) by SJ2PR12MB7822.namprd12.prod.outlook.com (2603:10b6:a03:4ca::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6886.35; Tue, 17 Oct 2023 07:44:38 +0000 Received: from BL6PEPF0001AB78.namprd02.prod.outlook.com (2603:10b6:208:2bb:cafe::e3) by BL1PR13CA0144.outlook.office365.com (2603:10b6:208:2bb::29) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6907.17 via Frontend Transport; Tue, 17 Oct 2023 07:44:38 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.160) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.160 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.160; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.160) by BL6PEPF0001AB78.mail.protection.outlook.com (10.167.242.171) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6907.21 via Frontend Transport; Tue, 17 Oct 2023 07:44:37 +0000 Received: from rnnvmail201.nvidia.com (10.129.68.8) by mail.nvidia.com (10.129.200.66) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Tue, 17 Oct 2023 00:44:23 -0700 Received: from dev-r-vrt-155.mtr.labs.mlnx (10.126.230.35) by rnnvmail201.nvidia.com (10.129.68.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Tue, 17 Oct 2023 00:44:19 -0700 From: Ido Schimmel To: , CC: , , , , , , , , , , Ido Schimmel Subject: [RFC PATCH net-next 12/12] selftests: mlxsw: Add PCI reset test Date: Tue, 17 Oct 2023 10:42:57 +0300 Message-ID: <20231017074257.3389177-13-idosch@nvidia.com> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20231017074257.3389177-1-idosch@nvidia.com> References: <20231017074257.3389177-1-idosch@nvidia.com> MIME-Version: 1.0 X-Originating-IP: [10.126.230.35] X-ClientProxiedBy: rnnvmail201.nvidia.com (10.129.68.8) To rnnvmail201.nvidia.com (10.129.68.8) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BL6PEPF0001AB78:EE_|SJ2PR12MB7822:EE_ X-MS-Office365-Filtering-Correlation-Id: 3d58036d-dafa-4f8f-2019-08dbcee4ea4e X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 2UXhb02n1HGrsDI8PGxz8mZdX7G1+W140PvAfQCZRazoEDKjg+6wTt1W7RE/Ht3e7ZiZ2SB6Ultic2eXFUPwJISwSFQyoQ7v5iD9GJ3eb94QGf94n7PdznGZrRjk8ukISGMOcqDfxRdqmTmXHFlVkIzwQCtkiUiDXhoMsZgWz7Q6DSyuzfur5OyQooOitL5t6/5t3YJSPB9zQVFqbU0MWrVV0AmoXFCYKWAl0vW8QY/tR+SQhUutad2eTsGR9IAVJB5AHLUc4mnLL+JftGOox4GXfEdGJqgakzTJvug++z2RkphZI6thQvr7NKlQkD/Ji1rooZWQI6VsHixQ57PeF9cgqey7TUB1OqAcoVwLa2yx9rdqJBwGABiQK/tKDzWVr8A1p066tZ2GXMpHRhkvEQjMTELZ/x6O8QXdCnDis+3YZVOtYrsUPjcAXnDwv+A8m/rwCZL4WCb6chUu1XRUOncyJHFowEKh1d+BaI/0409EkFP26hoCzl7j2jCFvlStUkzDd7GZU5gB0CRB9WZ6/wGh+/CyfoV1GTP127/9MEf7Lcwv9nkWCIuyfYDcA6P1L7+itGgRov4bvehLcGhcISjIXOMpZKy9hIH39D65UOveeZG/05bV7ebBKazQLQMlWJQvZY45BZO78Krmt2Ti3eMmYZVdLGv7jRgTQk6iyNklGrRWX0rLw9NzT+W0sG83FhX0ThOB/fVeGI5rdqaAbe9N+kqgmhnwGr70i/ZZF7ZkrrUiMIVyuq58jRv7pGBU X-Forefront-Antispam-Report: CIP:216.228.117.160;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc6edge1.nvidia.com;CAT:NONE;SFS:(13230031)(4636009)(39860400002)(396003)(346002)(136003)(376002)(230922051799003)(64100799003)(1800799009)(82310400011)(186009)(451199024)(46966006)(40470700004)(36840700001)(478600001)(70586007)(70206006)(110136005)(54906003)(6666004)(26005)(16526019)(107886003)(1076003)(336012)(316002)(426003)(2616005)(8936002)(2906002)(4326008)(8676002)(5660300002)(41300700001)(36756003)(86362001)(7636003)(47076005)(36860700001)(83380400001)(356005)(82740400003)(40480700001)(40460700003);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 17 Oct 2023 07:44:37.9879 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 3d58036d-dafa-4f8f-2019-08dbcee4ea4e X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.117.160];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BL6PEPF0001AB78.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ2PR12MB7822 Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org Test that PCI reset works correctly by verifying that only the expected reset methods are supported and that after issuing the reset the ifindex of the port changes. Signed-off-by: Ido Schimmel --- .../selftests/drivers/net/mlxsw/pci_reset.sh | 58 +++++++++++++++++++ 1 file changed, 58 insertions(+) create mode 100755 tools/testing/selftests/drivers/net/mlxsw/pci_reset.sh diff --git a/tools/testing/selftests/drivers/net/mlxsw/pci_reset.sh b/tools/testing/selftests/drivers/net/mlxsw/pci_reset.sh new file mode 100755 index 000000000000..2ea22806d530 --- /dev/null +++ b/tools/testing/selftests/drivers/net/mlxsw/pci_reset.sh @@ -0,0 +1,58 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 +# +# Test that PCI reset works correctly by verifying that only the expected reset +# methods are supported and that after issuing the reset the ifindex of the +# port changes. + +lib_dir=$(dirname $0)/../../../net/forwarding + +ALL_TESTS=" + pci_reset_test +" +NUM_NETIFS=1 +source $lib_dir/lib.sh +source $lib_dir/devlink_lib.sh + +pci_reset_test() +{ + RET=0 + + local bus=$(echo $DEVLINK_DEV | cut -d '/' -f 1) + local bdf=$(echo $DEVLINK_DEV | cut -d '/' -f 2) + + if [ $bus != "pci" ]; then + check_err 1 "devlink device is not a PCI device" + log_test "pci reset" + return + fi + + if [ ! -f /sys/bus/pci/devices/$bdf/reset_method ]; then + check_err 1 "reset is not supported" + log_test "pci reset" + return + fi + + [[ $(cat /sys/bus/pci/devices/$bdf/reset_method) == "device_specific bus" ]] + check_err $? "only \"device_specific\" and \"bus\" reset methods should be supported" + + local ifindex_pre=$(ip -j link show dev $swp1 | jq '.[]["ifindex"]') + + echo 1 > /sys/bus/pci/devices/$bdf/reset + check_err $? "reset failed" + + # Wait for udev to rename newly created netdev. + udevadm settle + + local ifindex_post=$(ip -j link show dev $swp1 | jq '.[]["ifindex"]') + + [[ $ifindex_pre != $ifindex_post ]] + check_err $? "reset not performed" + + log_test "pci reset" +} + +swp1=${NETIFS[p1]} +tests_run + +exit $EXIT_STATUS