From patchwork Mon Mar 28 23:00:06 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tanjore Suresh X-Patchwork-Id: 12794310 X-Patchwork-Delegate: bhelgaas@google.com Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 05DFDC433EF for ; Mon, 28 Mar 2022 23:00:30 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230235AbiC1XCJ (ORCPT ); Mon, 28 Mar 2022 19:02:09 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42410 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230218AbiC1XCB (ORCPT ); Mon, 28 Mar 2022 19:02:01 -0400 Received: from mail-yb1-xb49.google.com (mail-yb1-xb49.google.com [IPv6:2607:f8b0:4864:20::b49]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id F0E0F24F09 for ; Mon, 28 Mar 2022 16:00:18 -0700 (PDT) Received: by mail-yb1-xb49.google.com with SMTP id j6-20020a25ec06000000b00633c6f3e072so11963333ybh.12 for ; Mon, 28 Mar 2022 16:00:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=HE+KJmElkBiRdLqH+GjHP+Rl3V6ftUnhFGl2WadF25Q=; b=MWRcatLqV8jhdG9ohWm7mM4E3R/YbC1/FIFzwNDD6TlY6EYMe/xuHPFJHuooAj3aXf 4fJnfu9sk+tQmJn/OsEvJ644ufB1KJUyOS57p1Kr1FoZ2LkMV+NcO2Tn61yRVx0vSZ02 As2kMTmHM3+2vcZPWir+uiPFUfpWiidfUOVa+5s0fN+X+T9z4gpHxuLcE4GCGTO2yUBG E/YhxqmsZ3PfScXtfw7d4xHeWcdVdSlZuDFK0jHSsZq/Rn8vo+L4KVzXyuxl9W3E33RA Tfd62IaObJUTQy+NaMZBs9PavgPC+FdcHfKDuvN8hcAamVIJRXhLmFfvIuDTIza+ZbO7 XpSQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=HE+KJmElkBiRdLqH+GjHP+Rl3V6ftUnhFGl2WadF25Q=; b=jYO67H1LOQYmAbhViBzMd9JfXmZ3q8kiV/0kjOaO0Qj3ySniAbMK1izeZ+AJVYLoei fINNnshMTpPyA3c3G6pmEIxMPQGDv7LPQZJOiqMwQsKkznAONRXKwWlFsvS5Is6hZS5N A2XRqE2M3cRcLN+QIqbgsMCwJzpXXIwLk8Ll4kSiXmTuc0n5E5XQOYXxPib0xVUQkNFb sa6T+fXAvDDAjeVHq9nkhgGS9nLspQfADJrS0AcJrwYobNJ0iosnEnlckJA20efu6GG4 lV8DslHbgubgO6KN/GBlFImAsQrXTu1vwiMUaTgKIyihEd43+gSjnmc0mM7Y4Rducsny NZtQ== X-Gm-Message-State: AOAM532BSCrEJ42A+MYlfpj2z3pSQp/m+1Rh4CmZZ0GONPJqoeBW/tcO KLAZkFy4Pe89JIaIkbJPSqq6EPQjadlpNIo= X-Google-Smtp-Source: ABdhPJyJLGwD6p/MqUDfw8nEh0DJ9s5L0oPyAGDvmxNwqd6Cp0ER8bUrxaaN8M+8r2L4xk3m5KFU7Ql3rq3c7tw= X-Received: from tansuresh.svl.corp.google.com ([2620:15c:2c5:13:3dbb:6bbc:98be:a31e]) (user=tansuresh job=sendgmr) by 2002:a81:4ed6:0:b0:2e6:aee7:6b7a with SMTP id c205-20020a814ed6000000b002e6aee76b7amr28282562ywb.399.1648508418228; Mon, 28 Mar 2022 16:00:18 -0700 (PDT) Date: Mon, 28 Mar 2022 16:00:06 -0700 In-Reply-To: <20220328230008.3587975-1-tansuresh@google.com> Message-Id: <20220328230008.3587975-2-tansuresh@google.com> Mime-Version: 1.0 References: <20220328230008.3587975-1-tansuresh@google.com> X-Mailer: git-send-email 2.35.1.1021.g381101b075-goog Subject: [PATCH v1 1/3] driver core: Support asynchronous driver shutdown From: Tanjore Suresh To: Greg Kroah-Hartman , "Rafael J . Wysocki" , Christoph Hellwig , Sagi Grimberg , Bjorn Helgaas Cc: linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org, linux-pci@vger.kernel.org, Tanjore Suresh Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org This changes the bus driver interface with additional entry points to enable devices to implement asynchronous shutdown. The existing synchronous interface to shutdown is unmodified and retained for backward compatibility. This changes the common device shutdown code to enable devices to participate in asynchronous shutdown implementation. Signed-off-by: Tanjore Suresh --- drivers/base/core.c | 39 +++++++++++++++++++++++++++++++++++++- include/linux/device/bus.h | 10 ++++++++++ 2 files changed, 48 insertions(+), 1 deletion(-) diff --git a/drivers/base/core.c b/drivers/base/core.c index 3d6430eb0c6a..359e7067e8b8 100644 --- a/drivers/base/core.c +++ b/drivers/base/core.c @@ -4479,6 +4479,7 @@ EXPORT_SYMBOL_GPL(device_change_owner); void device_shutdown(void) { struct device *dev, *parent; + LIST_HEAD(async_shutdown_list); wait_for_device_probe(); device_block_probing(); @@ -4523,7 +4524,14 @@ void device_shutdown(void) dev_info(dev, "shutdown_pre\n"); dev->class->shutdown_pre(dev); } - if (dev->bus && dev->bus->shutdown) { + + if (dev->bus && dev->bus->shutdown_pre) { + if (initcall_debug) + dev_info(dev, "shutdown_pre\n"); + dev->bus->shutdown_pre(dev); + list_add(&dev->kobj.entry, + &async_shutdown_list); + } else if (dev->bus && dev->bus->shutdown) { if (initcall_debug) dev_info(dev, "shutdown\n"); dev->bus->shutdown(dev); @@ -4543,6 +4551,35 @@ void device_shutdown(void) spin_lock(&devices_kset->list_lock); } spin_unlock(&devices_kset->list_lock); + + /* + * Second pass spin for only devices, that have configured + * Asynchronous shutdown. + */ + while (!list_empty(&async_shutdown_list)) { + dev = list_entry(async_shutdown_list.next, struct device, + kobj.entry); + parent = get_device(dev->parent); + get_device(dev); + /* + * Make sure the device is off the list + */ + list_del_init(&dev->kobj.entry); + if (parent) + device_lock(parent); + device_lock(dev); + if (dev->bus && dev->bus->shutdown_post) { + if (initcall_debug) + dev_info(dev, + "shutdown_post called\n"); + dev->bus->shutdown_post(dev); + } + device_unlock(dev); + if (parent) + device_unlock(parent); + put_device(dev); + put_device(parent); + } } /* diff --git a/include/linux/device/bus.h b/include/linux/device/bus.h index a039ab809753..e261819601e9 100644 --- a/include/linux/device/bus.h +++ b/include/linux/device/bus.h @@ -49,6 +49,14 @@ struct fwnode_handle; * will never get called until they do. * @remove: Called when a device removed from this bus. * @shutdown: Called at shut-down time to quiesce the device. + * @shutdown_pre: Called at the shutdown-time to start the shutdown + * process on the device. This entry point will be called + * only when the bus driver has indicated it would like + * to participate in asynchronous shutdown completion. + * @shutdown_post: Called at shutdown-time to complete the shutdown + * process of the device. This entry point will be called + * only when the bus drive has indicated it would like to + * participate in the asynchronous shutdown completion. * * @online: Called to put the device back online (after offlining it). * @offline: Called to put the device offline for hot-removal. May fail. @@ -93,6 +101,8 @@ struct bus_type { void (*sync_state)(struct device *dev); void (*remove)(struct device *dev); void (*shutdown)(struct device *dev); + void (*shutdown_pre)(struct device *dev); + void (*shutdown_post)(struct device *dev); int (*online)(struct device *dev); int (*offline)(struct device *dev); From patchwork Mon Mar 28 23:00:07 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tanjore Suresh X-Patchwork-Id: 12794312 X-Patchwork-Delegate: bhelgaas@google.com Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D728AC4332F for ; Mon, 28 Mar 2022 23:00:33 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230203AbiC1XCM (ORCPT ); Mon, 28 Mar 2022 19:02:12 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42788 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230242AbiC1XCJ (ORCPT ); Mon, 28 Mar 2022 19:02:09 -0400 Received: from mail-yw1-x114a.google.com (mail-yw1-x114a.google.com [IPv6:2607:f8b0:4864:20::114a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1FD8126131 for ; Mon, 28 Mar 2022 16:00:23 -0700 (PDT) Received: by mail-yw1-x114a.google.com with SMTP id 00721157ae682-2e68c93bb30so130428717b3.18 for ; Mon, 28 Mar 2022 16:00:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=PjCJgoS657I/tk8eQdlHI01Sy2nbJ1vs0eOca3BDeMA=; b=YrW1oHI6OeNS97qb0yJLiWsmS/O0yisj3J+4OLHulXFrVW++E6+RYQOIE9kRM8gvPl nYVOVSMnQ38vUWHmvPhbD7Di+LW8cxAlhFJk6rylh8ho4c7W6RbVgiqFR3kqbUGal79N AnGraxxjbZ/qEIuvv/3hskPbqVzITszKNTauorm3U5A10LAz33yyLW+qRHr39PSjgZEc gFxP0vIK4JjSTg72HAxnLs1TerTNewB0WjKzK2tZmdEGrsphjP+GpJ/8nRfVGtQ6/6oh 8DnJjPZ35lUPzzKkyIyhHXn9zk+j+m6UjTXpgJYAHJ8LEDsxYZSRsjbEB7hd4MEVv7fE U7mw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=PjCJgoS657I/tk8eQdlHI01Sy2nbJ1vs0eOca3BDeMA=; b=IvdtviJqqhw41R0j9ynipyw7TNgZGZ4MQsALjws2Cc7eATLszFbYT4ebzspHnQjZcL DGf+nFOORpCOGzMyaYoZgH0nlXleyFnBCTDP9KZKACM7y3YVsQ/84NEogR+aVgmRaxGe oOHVGZZV2ah+SVAVCT4aPSx5ZpuwZkE4yoA8BduSUAOZBy526gNJ1y8ygkL5DdWKfBBp S1pugVY3ackIfd//oP1uJpFruX3DV3lylvwWaBGJRazSDbcY2sCI9PvHQkr0cMSLGCOk AA9Dq7ZAcy6sGULcZWEjaWEumwolwcS7gfF4Te/Qrpwj8UyJPMq8uD0ZzcZ88fq40jS9 v/DA== X-Gm-Message-State: AOAM532gYIe4kRTeD6Ats036WJAbtT3T6TG4kTZzMWq3erMpBO3oJ27q NuIgPNU6RVTxio8DfZlwaHgIsrh3vVXpFjg= X-Google-Smtp-Source: ABdhPJzCP0I8+FlderVfuOTtetUSRQ7p5Ej5WF/C0Zw2u46sRNYBzp8kbYF//zgxfxG0NFdnd84urn2FfzqAcr0= X-Received: from tansuresh.svl.corp.google.com ([2620:15c:2c5:13:3dbb:6bbc:98be:a31e]) (user=tansuresh job=sendgmr) by 2002:a81:c02:0:b0:2e5:b6af:2c8 with SMTP id 2-20020a810c02000000b002e5b6af02c8mr29031195ywm.190.1648508422372; Mon, 28 Mar 2022 16:00:22 -0700 (PDT) Date: Mon, 28 Mar 2022 16:00:07 -0700 In-Reply-To: <20220328230008.3587975-2-tansuresh@google.com> Message-Id: <20220328230008.3587975-3-tansuresh@google.com> Mime-Version: 1.0 References: <20220328230008.3587975-1-tansuresh@google.com> <20220328230008.3587975-2-tansuresh@google.com> X-Mailer: git-send-email 2.35.1.1021.g381101b075-goog Subject: [PATCH v1 2/3] PCI: Support asynchronous shutdown From: Tanjore Suresh To: Greg Kroah-Hartman , "Rafael J . Wysocki" , Christoph Hellwig , Sagi Grimberg , Bjorn Helgaas Cc: linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org, linux-pci@vger.kernel.org, Tanjore Suresh Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org Enhances the base PCI driver to add support for asynchronous shutdown. Assume a device takes n secs to shutdown. If a machine has been populated with M such devices, the total time spent in shutting down all the devices will be M * n secs, if the shutdown is done synchronously. For example, if NVMe PCI Controllers take 5 secs to shutdown and if there are 16 such NVMe controllers in a system, system will spend a total of 80 secs to shutdown all NVMe devices in that system. In order to speed up the shutdown time, asynchronous interface to shutdown has been implemented. This will significantly reduce the machine reboot time. Signed-off-by: Tanjore Suresh --- drivers/pci/pci-driver.c | 17 ++++++++++++++--- include/linux/pci.h | 2 ++ 2 files changed, 16 insertions(+), 3 deletions(-) diff --git a/drivers/pci/pci-driver.c b/drivers/pci/pci-driver.c index 4ceeb75fc899..0d0b46d71e88 100644 --- a/drivers/pci/pci-driver.c +++ b/drivers/pci/pci-driver.c @@ -501,14 +501,16 @@ static void pci_device_remove(struct device *dev) pci_dev_put(pci_dev); } -static void pci_device_shutdown(struct device *dev) +static void pci_device_shutdown_pre(struct device *dev) { struct pci_dev *pci_dev = to_pci_dev(dev); struct pci_driver *drv = pci_dev->driver; pm_runtime_resume(dev); - if (drv && drv->shutdown) + if (drv && drv->shutdown_pre) + drv->shutdown_pre(pci_dev); + else if (drv && drv->shutdown) drv->shutdown(pci_dev); /* @@ -522,6 +524,14 @@ static void pci_device_shutdown(struct device *dev) pci_clear_master(pci_dev); } +static void pci_device_shutdown_post(struct device *dev) +{ + struct pci_dev *pci_dev = to_pci_dev(dev); + struct pci_driver *drv = pci_dev->driver; + + if (drv && drv->shutdown_post) + drv->shutdown_post(pci_dev); +} #ifdef CONFIG_PM /* Auxiliary functions used for system resume and run-time resume. */ @@ -1625,7 +1635,8 @@ struct bus_type pci_bus_type = { .uevent = pci_uevent, .probe = pci_device_probe, .remove = pci_device_remove, - .shutdown = pci_device_shutdown, + .shutdown_pre = pci_device_shutdown_pre, + .shutdown_post = pci_device_shutdown_post, .dev_groups = pci_dev_groups, .bus_groups = pci_bus_groups, .drv_groups = pci_drv_groups, diff --git a/include/linux/pci.h b/include/linux/pci.h index b957eeb89c7a..19047fcb3c8a 100644 --- a/include/linux/pci.h +++ b/include/linux/pci.h @@ -905,6 +905,8 @@ struct pci_driver { int (*suspend)(struct pci_dev *dev, pm_message_t state); /* Device suspended */ int (*resume)(struct pci_dev *dev); /* Device woken up */ void (*shutdown)(struct pci_dev *dev); + void (*shutdown_pre)(struct pci_dev *dev); + void (*shutdown_post)(struct pci_dev *dev); int (*sriov_configure)(struct pci_dev *dev, int num_vfs); /* On PF */ int (*sriov_set_msix_vec_count)(struct pci_dev *vf, int msix_vec_count); /* On PF */ u32 (*sriov_get_vf_total_msix)(struct pci_dev *pf); From patchwork Mon Mar 28 23:00:08 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tanjore Suresh X-Patchwork-Id: 12794311 X-Patchwork-Delegate: bhelgaas@google.com Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B511CC433FE for ; Mon, 28 Mar 2022 23:00:33 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230163AbiC1XCM (ORCPT ); Mon, 28 Mar 2022 19:02:12 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42830 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230203AbiC1XCK (ORCPT ); Mon, 28 Mar 2022 19:02:10 -0400 Received: from mail-yb1-xb49.google.com (mail-yb1-xb49.google.com [IPv6:2607:f8b0:4864:20::b49]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4966324BD1 for ; Mon, 28 Mar 2022 16:00:27 -0700 (PDT) Received: by mail-yb1-xb49.google.com with SMTP id j6-20020a25ec06000000b00633c6f3e072so11963813ybh.12 for ; Mon, 28 Mar 2022 16:00:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=955rcqvOq7CsT/HV+ZQ8Dkn9KmNXcvBcomGd/b5MLgQ=; b=gO5CRr/TXyxTRWCtZsVymoT2ODodCfiLk/7e0a5/s+rpuLrMsYBy8zIGuFvyjEaNBM e7i0sO2DzP+rPlctXexI4PbVn2xSZzTaFh2YlquL0G3NXG0pzE3Rh88gD2kqC5WqQrDS Fk1NgdRk7pF6ltSkbJV0WYBw3r4t5pA6ylE59XrQbX4v82R/rtNXYwWozyv37a5mbCTh P1XedmNmKXoarMABv2QNCIYKORnhsHfndbxbJIxoVwBsk7qyFFJzOcL1Eiz0WKV+QFIT LXZUCNMKTwfW78lEnowDdoZ/KZgLS97oti6pOL/UajtAGLYg3p6sH9hvXsvT1kdU8uND QvkA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=955rcqvOq7CsT/HV+ZQ8Dkn9KmNXcvBcomGd/b5MLgQ=; b=x8A41b1DIFoExhQbYXGlWcW3+qz9iieUM2u1aAQMjKh7V+z8GGKLPR/BL00EAy5Tmq HNtNsVylxjOGqAAGmv/tLwsjKgnHdwGeq4/AXX9SqEmvVyg4eCrAneBcHHJlNc9Ab+nS AjPcS58rwEkE7rew1prhWwp4FjYTP6Ys4Kgi3VvQS5byPRZQSE6o+kxL6Pnes5Sy5tCY xx2Rb3oulLkkTHR6QL7ThSh9lf1TcgRfi8DAgGjr9U76ZWlSwUKaBM+nfOuuyriQG/nI LBQfJLhLI0LvGT6Qr0v194msnbRPziDvgffeXXx3jjDppj3VtTdwr7wQnIQV5UEcuEMq JPHQ== X-Gm-Message-State: AOAM53368dbbaXkMNLrAmI8YQYfl6AiyQSUuCIrRMyl7j7YDj9S3LLxB 8HsyKHOyBlCX954zxkOaud/kUJcxM2BCIrY= X-Google-Smtp-Source: ABdhPJxE0/bKX2+jSyl68rZKYUHtgtBXK2/N9OPvaCvXK1Cu6aGcC3/ZnsNS8hKxWB3hJuPR8XHfBwkuCvtOxqE= X-Received: from tansuresh.svl.corp.google.com ([2620:15c:2c5:13:3dbb:6bbc:98be:a31e]) (user=tansuresh job=sendgmr) by 2002:a25:3cc7:0:b0:634:568:a950 with SMTP id j190-20020a253cc7000000b006340568a950mr24418522yba.138.1648508427222; Mon, 28 Mar 2022 16:00:27 -0700 (PDT) Date: Mon, 28 Mar 2022 16:00:08 -0700 In-Reply-To: <20220328230008.3587975-3-tansuresh@google.com> Message-Id: <20220328230008.3587975-4-tansuresh@google.com> Mime-Version: 1.0 References: <20220328230008.3587975-1-tansuresh@google.com> <20220328230008.3587975-2-tansuresh@google.com> <20220328230008.3587975-3-tansuresh@google.com> X-Mailer: git-send-email 2.35.1.1021.g381101b075-goog Subject: [PATCH v1 3/3] nvme: Add async shutdown support From: Tanjore Suresh To: Greg Kroah-Hartman , "Rafael J . Wysocki" , Christoph Hellwig , Sagi Grimberg , Bjorn Helgaas Cc: linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org, linux-pci@vger.kernel.org, Tanjore Suresh Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org This works with the asynchronous shutdown mechanism setup for the PCI drivers and participates to provide both pre and post shutdown routines at pci_driver structure level. The shutdown_pre routine starts the shutdown and does not wait for the shutdown to complete. The shutdown_post routine waits for the shutdown to complete on individual controllers that this driver instance controls. This mechanism optimizes to speed up the shutdown in a system which host many controllers. Signed-off-by: Tanjore Suresh --- drivers/nvme/host/core.c | 28 ++++++++++---- drivers/nvme/host/nvme.h | 8 ++++ drivers/nvme/host/pci.c | 80 +++++++++++++++++++++++++--------------- 3 files changed, 80 insertions(+), 36 deletions(-) diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c index 677fa4bf76d3..24b08789fd34 100644 --- a/drivers/nvme/host/core.c +++ b/drivers/nvme/host/core.c @@ -2173,16 +2173,30 @@ EXPORT_SYMBOL_GPL(nvme_enable_ctrl); int nvme_shutdown_ctrl(struct nvme_ctrl *ctrl) { - unsigned long timeout = jiffies + (ctrl->shutdown_timeout * HZ); - u32 csts; int ret; + ret = nvme_shutdown_ctrl_start(ctrl); + if (ret) + return ret; + return nvme_wait_for_shutdown_cmpl(ctrl); +} +EXPORT_SYMBOL_GPL(nvme_shutdown_ctrl); + +int nvme_shutdown_ctrl_start(struct nvme_ctrl *ctrl) +{ + ctrl->ctrl_config &= ~NVME_CC_SHN_MASK; ctrl->ctrl_config |= NVME_CC_SHN_NORMAL; - ret = ctrl->ops->reg_write32(ctrl, NVME_REG_CC, ctrl->ctrl_config); - if (ret) - return ret; + return ctrl->ops->reg_write32(ctrl, NVME_REG_CC, ctrl->ctrl_config); +} +EXPORT_SYMBOL_GPL(nvme_shutdown_ctrl_start); + +int nvme_wait_for_shutdown_cmpl(struct nvme_ctrl *ctrl) +{ + unsigned long deadline = jiffies + (ctrl->shutdown_timeout * HZ); + u32 csts; + int ret; while ((ret = ctrl->ops->reg_read32(ctrl, NVME_REG_CSTS, &csts)) == 0) { if ((csts & NVME_CSTS_SHST_MASK) == NVME_CSTS_SHST_CMPLT) @@ -2191,7 +2205,7 @@ int nvme_shutdown_ctrl(struct nvme_ctrl *ctrl) msleep(100); if (fatal_signal_pending(current)) return -EINTR; - if (time_after(jiffies, timeout)) { + if (time_after(jiffies, deadline)) { dev_err(ctrl->device, "Device shutdown incomplete; abort shutdown\n"); return -ENODEV; @@ -2200,7 +2214,7 @@ int nvme_shutdown_ctrl(struct nvme_ctrl *ctrl) return ret; } -EXPORT_SYMBOL_GPL(nvme_shutdown_ctrl); +EXPORT_SYMBOL_GPL(nvme_wait_for_shutdown_cmpl); static int nvme_configure_timestamp(struct nvme_ctrl *ctrl) { diff --git a/drivers/nvme/host/nvme.h b/drivers/nvme/host/nvme.h index f4b674a8ce20..87f5803ef577 100644 --- a/drivers/nvme/host/nvme.h +++ b/drivers/nvme/host/nvme.h @@ -170,6 +170,12 @@ enum { NVME_REQ_USERCMD = (1 << 1), }; +enum shutdown_type { + DO_NOT_SHUTDOWN = 0, + SHUTDOWN_TYPE_SYNC = 1, + SHUTDOWN_TYPE_ASYNC = 2, +}; + static inline struct nvme_request *nvme_req(struct request *req) { return blk_mq_rq_to_pdu(req); @@ -672,6 +678,8 @@ bool nvme_wait_reset(struct nvme_ctrl *ctrl); int nvme_disable_ctrl(struct nvme_ctrl *ctrl); int nvme_enable_ctrl(struct nvme_ctrl *ctrl); int nvme_shutdown_ctrl(struct nvme_ctrl *ctrl); +int nvme_shutdown_ctrl_start(struct nvme_ctrl *ctrl); +int nvme_wait_for_shutdown_cmpl(struct nvme_ctrl *ctrl); int nvme_init_ctrl(struct nvme_ctrl *ctrl, struct device *dev, const struct nvme_ctrl_ops *ops, unsigned long quirks); void nvme_uninit_ctrl(struct nvme_ctrl *ctrl); diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c index 2e98ac3f3ad6..dc72fe7d8994 100644 --- a/drivers/nvme/host/pci.c +++ b/drivers/nvme/host/pci.c @@ -107,7 +107,7 @@ MODULE_PARM_DESC(noacpi, "disable acpi bios quirks"); struct nvme_dev; struct nvme_queue; -static void nvme_dev_disable(struct nvme_dev *dev, bool shutdown); +static void nvme_dev_disable(struct nvme_dev *dev, int shutdown_type); static bool __nvme_disable_io_queues(struct nvme_dev *dev, u8 opcode); /* @@ -1357,7 +1357,7 @@ static enum blk_eh_timer_return nvme_timeout(struct request *req, bool reserved) */ if (nvme_should_reset(dev, csts)) { nvme_warn_reset(dev, csts); - nvme_dev_disable(dev, false); + nvme_dev_disable(dev, DO_NOT_SHUTDOWN); nvme_reset_ctrl(&dev->ctrl); return BLK_EH_DONE; } @@ -1392,7 +1392,7 @@ static enum blk_eh_timer_return nvme_timeout(struct request *req, bool reserved) "I/O %d QID %d timeout, disable controller\n", req->tag, nvmeq->qid); nvme_req(req)->flags |= NVME_REQ_CANCELLED; - nvme_dev_disable(dev, true); + nvme_dev_disable(dev, SHUTDOWN_TYPE_SYNC); return BLK_EH_DONE; case NVME_CTRL_RESETTING: return BLK_EH_RESET_TIMER; @@ -1410,7 +1410,7 @@ static enum blk_eh_timer_return nvme_timeout(struct request *req, bool reserved) "I/O %d QID %d timeout, reset controller\n", req->tag, nvmeq->qid); nvme_req(req)->flags |= NVME_REQ_CANCELLED; - nvme_dev_disable(dev, false); + nvme_dev_disable(dev, DO_NOT_SHUTDOWN); nvme_reset_ctrl(&dev->ctrl); return BLK_EH_DONE; @@ -1503,11 +1503,13 @@ static void nvme_suspend_io_queues(struct nvme_dev *dev) nvme_suspend_queue(&dev->queues[i]); } -static void nvme_disable_admin_queue(struct nvme_dev *dev, bool shutdown) +static void nvme_disable_admin_queue(struct nvme_dev *dev, int shutdown_type) { struct nvme_queue *nvmeq = &dev->queues[0]; - if (shutdown) + if (shutdown_type == SHUTDOWN_TYPE_ASYNC) + nvme_shutdown_ctrl_start(&dev->ctrl); + else if (shutdown_type == SHUTDOWN_TYPE_SYNC) nvme_shutdown_ctrl(&dev->ctrl); else nvme_disable_ctrl(&dev->ctrl); @@ -2669,7 +2671,7 @@ static void nvme_pci_disable(struct nvme_dev *dev) } } -static void nvme_dev_disable(struct nvme_dev *dev, bool shutdown) +static void nvme_dev_disable(struct nvme_dev *dev, int shutdown_type) { bool dead = true, freeze = false; struct pci_dev *pdev = to_pci_dev(dev->dev); @@ -2691,14 +2693,14 @@ static void nvme_dev_disable(struct nvme_dev *dev, bool shutdown) * Give the controller a chance to complete all entered requests if * doing a safe shutdown. */ - if (!dead && shutdown && freeze) + if (!dead && (shutdown_type != DO_NOT_SHUTDOWN) && freeze) nvme_wait_freeze_timeout(&dev->ctrl, NVME_IO_TIMEOUT); nvme_stop_queues(&dev->ctrl); if (!dead && dev->ctrl.queue_count > 0) { nvme_disable_io_queues(dev); - nvme_disable_admin_queue(dev, shutdown); + nvme_disable_admin_queue(dev, shutdown_type); } nvme_suspend_io_queues(dev); nvme_suspend_queue(&dev->queues[0]); @@ -2710,12 +2712,12 @@ static void nvme_dev_disable(struct nvme_dev *dev, bool shutdown) blk_mq_tagset_wait_completed_request(&dev->tagset); blk_mq_tagset_wait_completed_request(&dev->admin_tagset); - /* - * The driver will not be starting up queues again if shutting down so - * must flush all entered requests to their failed completion to avoid - * deadlocking blk-mq hot-cpu notifier. - */ - if (shutdown) { + if (shutdown_type == SHUTDOWN_TYPE_SYNC) { + /* + * The driver will not be starting up queues again if shutting down so + * must flush all entered requests to their failed completion to avoid + * deadlocking blk-mq hot-cpu notifier. + */ nvme_start_queues(&dev->ctrl); if (dev->ctrl.admin_q && !blk_queue_dying(dev->ctrl.admin_q)) nvme_start_admin_queue(&dev->ctrl); @@ -2723,11 +2725,11 @@ static void nvme_dev_disable(struct nvme_dev *dev, bool shutdown) mutex_unlock(&dev->shutdown_lock); } -static int nvme_disable_prepare_reset(struct nvme_dev *dev, bool shutdown) +static int nvme_disable_prepare_reset(struct nvme_dev *dev, int type) { if (!nvme_wait_reset(&dev->ctrl)) return -EBUSY; - nvme_dev_disable(dev, shutdown); + nvme_dev_disable(dev, type); return 0; } @@ -2785,7 +2787,7 @@ static void nvme_remove_dead_ctrl(struct nvme_dev *dev) */ nvme_change_ctrl_state(&dev->ctrl, NVME_CTRL_DELETING); nvme_get_ctrl(&dev->ctrl); - nvme_dev_disable(dev, false); + nvme_dev_disable(dev, DO_NOT_SHUTDOWN); nvme_kill_queues(&dev->ctrl); if (!queue_work(nvme_wq, &dev->remove_work)) nvme_put_ctrl(&dev->ctrl); @@ -2810,7 +2812,7 @@ static void nvme_reset_work(struct work_struct *work) * moving on. */ if (dev->ctrl.ctrl_config & NVME_CC_ENABLE) - nvme_dev_disable(dev, false); + nvme_dev_disable(dev, DO_NOT_SHUTDOWN); nvme_sync_queues(&dev->ctrl); mutex_lock(&dev->shutdown_lock); @@ -3151,7 +3153,7 @@ static void nvme_reset_prepare(struct pci_dev *pdev) * state as pci_dev device lock is held, making it impossible to race * with ->remove(). */ - nvme_disable_prepare_reset(dev, false); + nvme_disable_prepare_reset(dev, DO_NOT_SHUTDOWN); nvme_sync_queues(&dev->ctrl); } @@ -3163,13 +3165,32 @@ static void nvme_reset_done(struct pci_dev *pdev) flush_work(&dev->ctrl.reset_work); } -static void nvme_shutdown(struct pci_dev *pdev) +static void nvme_shutdown_pre(struct pci_dev *pdev) { struct nvme_dev *dev = pci_get_drvdata(pdev); - nvme_disable_prepare_reset(dev, true); + nvme_disable_prepare_reset(dev, SHUTDOWN_TYPE_ASYNC); } +static void nvme_shutdown_post(struct pci_dev *pdev) +{ + struct nvme_dev *dev = pci_get_drvdata(pdev); + + mutex_lock(&dev->shutdown_lock); + nvme_wait_for_shutdown_cmpl(&dev->ctrl); + + /* + * The driver will not be starting up queues again if shutting down so + * must flush all entered requests to their failed completion to avoid + * deadlocking blk-mq hot-cpu notifier. + */ + nvme_start_queues(&dev->ctrl); + if (dev->ctrl.admin_q && !blk_queue_dying(dev->ctrl.admin_q)) + nvme_start_admin_queue(&dev->ctrl); + + mutex_unlock(&dev->shutdown_lock); + +} static void nvme_remove_attrs(struct nvme_dev *dev) { if (dev->attrs_added) @@ -3191,13 +3212,13 @@ static void nvme_remove(struct pci_dev *pdev) if (!pci_device_is_present(pdev)) { nvme_change_ctrl_state(&dev->ctrl, NVME_CTRL_DEAD); - nvme_dev_disable(dev, true); + nvme_dev_disable(dev, SHUTDOWN_TYPE_SYNC); } flush_work(&dev->ctrl.reset_work); nvme_stop_ctrl(&dev->ctrl); nvme_remove_namespaces(&dev->ctrl); - nvme_dev_disable(dev, true); + nvme_dev_disable(dev, SHUTDOWN_TYPE_SYNC); nvme_remove_attrs(dev); nvme_free_host_mem(dev); nvme_dev_remove_admin(dev); @@ -3259,7 +3280,7 @@ static int nvme_suspend(struct device *dev) if (pm_suspend_via_firmware() || !ctrl->npss || !pcie_aspm_enabled(pdev) || (ndev->ctrl.quirks & NVME_QUIRK_SIMPLE_SUSPEND)) - return nvme_disable_prepare_reset(ndev, true); + return nvme_disable_prepare_reset(ndev, SHUTDOWN_TYPE_SYNC); nvme_start_freeze(ctrl); nvme_wait_freeze(ctrl); @@ -3302,7 +3323,7 @@ static int nvme_suspend(struct device *dev) * Clearing npss forces a controller reset on resume. The * correct value will be rediscovered then. */ - ret = nvme_disable_prepare_reset(ndev, true); + ret = nvme_disable_prepare_reset(ndev, SHUTDOWN_TYPE_SYNC); ctrl->npss = 0; } unfreeze: @@ -3314,7 +3335,7 @@ static int nvme_simple_suspend(struct device *dev) { struct nvme_dev *ndev = pci_get_drvdata(to_pci_dev(dev)); - return nvme_disable_prepare_reset(ndev, true); + return nvme_disable_prepare_reset(ndev, SHUTDOWN_TYPE_SYNC); } static int nvme_simple_resume(struct device *dev) @@ -3351,7 +3372,7 @@ static pci_ers_result_t nvme_error_detected(struct pci_dev *pdev, case pci_channel_io_frozen: dev_warn(dev->ctrl.device, "frozen state error detected, reset controller\n"); - nvme_dev_disable(dev, false); + nvme_dev_disable(dev, DO_NOT_SHUTDOWN); return PCI_ERS_RESULT_NEED_RESET; case pci_channel_io_perm_failure: dev_warn(dev->ctrl.device, @@ -3478,7 +3499,8 @@ static struct pci_driver nvme_driver = { .id_table = nvme_id_table, .probe = nvme_probe, .remove = nvme_remove, - .shutdown = nvme_shutdown, + .shutdown_pre = nvme_shutdown_pre, + .shutdown_post = nvme_shutdown_post, #ifdef CONFIG_PM_SLEEP .driver = { .pm = &nvme_dev_pm_ops,