From patchwork Tue Nov 1 15:00:37 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christoph Hellwig X-Patchwork-Id: 13027078 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B9379C433FE for ; Tue, 1 Nov 2022 15:08:04 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231153AbiKAPID (ORCPT ); Tue, 1 Nov 2022 11:08:03 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60668 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229814AbiKAPHo (ORCPT ); Tue, 1 Nov 2022 11:07:44 -0400 Received: from bombadil.infradead.org (bombadil.infradead.org [IPv6:2607:7c80:54:3::133]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3741621E35 for ; Tue, 1 Nov 2022 08:01:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20210309; h=Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender :Reply-To:Content-Type:Content-ID:Content-Description; bh=hbXzmOarrYYIi/4tfheWgFaaR6daBypbKfv0/Gh5BWg=; b=uHGuiw72lRt/acc1FD6KrBz0qO VsQLQqd0kRliJeb8X3lo1JF7J9yWuFQGsGTPPDGfboAPJr+f3hA4W7sNt4hQNJ2KTcRmsofz+AEmE /EQ3fhYIEzfQlVkFDxz+X86rNC8MKvp79FI6CBEWY7+m9TlLsp7elmnvWw8mWmyFwv3oEKJqsw9AX HovgZR0a9Fe2adX9hTf+YRER5ux+ouUc4kKmH2nWEYProwkfFhqydFlmfzdTIQrY9sAH/EiDW5Xx4 5DarENfuDB5O9Wl1ddKb1nBzHLjWbMncYo9litkLYv0venbfMkhB0zzi7J+DaO43N8TqhgCtkgipm HZF3KkVg==; Received: from [2001:4bb8:180:e42a:50da:325f:4a06:8830] (helo=localhost) by bombadil.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1opsl6-005gdH-Ik; Tue, 01 Nov 2022 15:00:57 +0000 From: Christoph Hellwig To: Jens Axboe , Keith Busch , Sagi Grimberg , Chao Leng Cc: Ming Lei , linux-nvme@lists.infradead.org, linux-block@vger.kernel.org Subject: [PATCH 01/14] block: set the disk capacity to 0 in blk_mark_disk_dead Date: Tue, 1 Nov 2022 16:00:37 +0100 Message-Id: <20221101150050.3510-2-hch@lst.de> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20221101150050.3510-1-hch@lst.de> References: <20221101150050.3510-1-hch@lst.de> MIME-Version: 1.0 X-SRS-Rewrite: SMTP reverse-path rewritten from by bombadil.infradead.org. See http://www.infradead.org/rpr.html Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org nvme and xen-blkfront are already doing this to stop buffered writes from creating dirty pages that can't be written out later. Move it to the common code. This also removes the comment about the ordering from nvme, as bd_mutex not only is gone entirely, but also hasn't been used for locking updates to the disk size long before that, and thus the ordering requirement documented there doesn't apply any more. Signed-off-by: Christoph Hellwig Reviewed-by: Keith Busch Reviewed-by: Sagi Grimberg Reviewed-by: Ming Lei Reviewed-by: Chao Leng Reviewed-by: Chaitanya Kulkarni --- block/genhd.c | 5 +++++ drivers/block/xen-blkfront.c | 1 - drivers/nvme/host/core.c | 7 +------ 3 files changed, 6 insertions(+), 7 deletions(-) diff --git a/block/genhd.c b/block/genhd.c index 493b93faee9c8..e7bd036024fab 100644 --- a/block/genhd.c +++ b/block/genhd.c @@ -555,6 +555,11 @@ void blk_mark_disk_dead(struct gendisk *disk) { set_bit(GD_DEAD, &disk->state); blk_queue_start_drain(disk->queue); + + /* + * Stop buffered writers from dirtying pages that can't be written out. + */ + set_capacity_and_notify(disk, 0); } EXPORT_SYMBOL_GPL(blk_mark_disk_dead); diff --git a/drivers/block/xen-blkfront.c b/drivers/block/xen-blkfront.c index 35b9bcad9db90..b28489290323f 100644 --- a/drivers/block/xen-blkfront.c +++ b/drivers/block/xen-blkfront.c @@ -2129,7 +2129,6 @@ static void blkfront_closing(struct blkfront_info *info) if (info->rq && info->gd) { blk_mq_stop_hw_queues(info->rq); blk_mark_disk_dead(info->gd); - set_capacity(info->gd, 0); } for_each_rinfo(info, rinfo, i) { diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c index 0090dc0b3ae6f..aea0f89acf409 100644 --- a/drivers/nvme/host/core.c +++ b/drivers/nvme/host/core.c @@ -5116,10 +5116,7 @@ static void nvme_stop_ns_queue(struct nvme_ns *ns) /* * Prepare a queue for teardown. * - * This must forcibly unquiesce queues to avoid blocking dispatch, and only set - * the capacity to 0 after that to avoid blocking dispatchers that may be - * holding bd_butex. This will end buffered writers dirtying pages that can't - * be synced. + * This must forcibly unquiesce queues to avoid blocking dispatch. */ static void nvme_set_queue_dying(struct nvme_ns *ns) { @@ -5128,8 +5125,6 @@ static void nvme_set_queue_dying(struct nvme_ns *ns) blk_mark_disk_dead(ns->disk); nvme_start_ns_queue(ns); - - set_capacity_and_notify(ns->disk, 0); } /** From patchwork Tue Nov 1 15:00:38 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christoph Hellwig X-Patchwork-Id: 13027079 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id BFF81C43219 for ; Tue, 1 Nov 2022 15:08:05 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229814AbiKAPIE (ORCPT ); Tue, 1 Nov 2022 11:08:04 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59982 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231381AbiKAPHo (ORCPT ); Tue, 1 Nov 2022 11:07:44 -0400 Received: from bombadil.infradead.org (bombadil.infradead.org [IPv6:2607:7c80:54:3::133]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3739C1CFEC for ; Tue, 1 Nov 2022 08:01:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20210309; h=Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender :Reply-To:Content-Type:Content-ID:Content-Description; bh=RMeBBtuZ6jN1c2xr45VBpasHrL1bqnzj0i7LLV0DoSY=; b=gVn4z3tLk2y8kIsfYRVJuFq3Cp NhHck4o9kuBisUJYWpfDuTm9cOiWA5CrKci7CCkljhWbXhwHxfgs/F/SzXAJ6V7QXcVkrhp6mUNiI d+ytgNVdoQT6fafmdt9BLdGHDbf3kwQM/E4fUkBSCbF1Q/d33Ae8NG2gCQGOFeIP6Pv+/7N6IXM7/ osELVWXILNTGsVxWNb/gSx4Y5jeLQwI3rYs7vGERtW5xLNF2SQE0t5WRsOTEClzTr2X2vNrc/vV4Y T29DfXN3V/j5bu/cX5Usl/GzlrqztMIVcDjvQ3NHPsKnnwczaEwICjdK42fR093Q9dq2LYwQZsh43 9YL2K9+g==; Received: from [2001:4bb8:180:e42a:50da:325f:4a06:8830] (helo=localhost) by bombadil.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1opsl9-005ge8-7t; Tue, 01 Nov 2022 15:00:59 +0000 From: Christoph Hellwig To: Jens Axboe , Keith Busch , Sagi Grimberg , Chao Leng Cc: Ming Lei , linux-nvme@lists.infradead.org, linux-block@vger.kernel.org Subject: [PATCH 02/14] nvme-pci: refactor the tagset handling in nvme_reset_work Date: Tue, 1 Nov 2022 16:00:38 +0100 Message-Id: <20221101150050.3510-3-hch@lst.de> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20221101150050.3510-1-hch@lst.de> References: <20221101150050.3510-1-hch@lst.de> MIME-Version: 1.0 X-SRS-Rewrite: SMTP reverse-path rewritten from by bombadil.infradead.org. See http://www.infradead.org/rpr.html Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org The code to create, update or delete a tagset and namespaces in nvme_reset_work is a bit convoluted. Refactor it with a two high-level conditionals for first probe vs reset and I/O queues vs no I/O queues to make the code flow more clear. Signed-off-by: Christoph Hellwig Reviewed-by: Chaitanya Kulkarni --- drivers/nvme/host/pci.c | 46 ++++++++++++++++++++++++++--------------- 1 file changed, 29 insertions(+), 17 deletions(-) diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c index 5f1d71ac00865..0e8410edb41bf 100644 --- a/drivers/nvme/host/pci.c +++ b/drivers/nvme/host/pci.c @@ -2897,25 +2897,37 @@ static void nvme_reset_work(struct work_struct *work) result = nvme_setup_io_queues(dev); if (result) goto out; - - /* - * Keep the controller around but remove all namespaces if we don't have - * any working I/O queue. - */ - if (dev->online_queues < 2) { - dev_warn(dev->ctrl.device, "IO queues not created\n"); - nvme_kill_queues(&dev->ctrl); - nvme_remove_namespaces(&dev->ctrl); - nvme_free_tagset(dev); + + if (dev->ctrl.tagset) { + /* + * This is a controller reset and we already have a tagset. + * Freeze and update the number of I/O queues as thos might have + * changed. If there are no I/O queues left after this reset, + * keep the controller around but remove all namespaces. + */ + if (dev->online_queues > 1) { + nvme_start_queues(&dev->ctrl); + nvme_wait_freeze(&dev->ctrl); + nvme_pci_update_nr_queues(dev); + nvme_dbbuf_set(dev); + nvme_unfreeze(&dev->ctrl); + } else { + dev_warn(dev->ctrl.device, "IO queues lost\n"); + nvme_kill_queues(&dev->ctrl); + nvme_remove_namespaces(&dev->ctrl); + nvme_free_tagset(dev); + } } else { - nvme_start_queues(&dev->ctrl); - nvme_wait_freeze(&dev->ctrl); - if (!dev->ctrl.tagset) + /* + * First probe. Still allow the controller to show up even if + * there are no namespaces. + */ + if (dev->online_queues > 1) { nvme_pci_alloc_tag_set(dev); - else - nvme_pci_update_nr_queues(dev); - nvme_dbbuf_set(dev); - nvme_unfreeze(&dev->ctrl); + nvme_dbbuf_set(dev); + } else { + dev_warn(dev->ctrl.device, "IO queues not created\n"); + } } /* From patchwork Tue Nov 1 15:00:39 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christoph Hellwig X-Patchwork-Id: 13027080 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5E16EC4332F for ; Tue, 1 Nov 2022 15:08:10 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231425AbiKAPIH (ORCPT ); Tue, 1 Nov 2022 11:08:07 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59218 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231258AbiKAPHs (ORCPT ); Tue, 1 Nov 2022 11:07:48 -0400 Received: from bombadil.infradead.org (bombadil.infradead.org [IPv6:2607:7c80:54:3::133]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2177622501 for ; Tue, 1 Nov 2022 08:01:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20210309; h=Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender :Reply-To:Content-Type:Content-ID:Content-Description; bh=UWJJITfRoSb//57QGaQg0nyw7JGk+8FSnAd0y+59s9Q=; b=h6jEHoDFnd0GlzMsGzargZA2YN lJxkEkNTpbbDr3bb0dFR00femFT60FCbN1l6wiGG3XiiycpLhXktZI5jfRTEXg6GSva5AMBOy8gxs qI8f7up2Qmr/SiPCqEmnuBB4TbnRwsS5MQiu/G6TBYZU/PwRa683onOj+RIbUQH9DBwwqjhEZ4zw2 zEV8uovyboyL9+gcq7hLQZPJqprahSJTZyE4TgEF19nN8t6d/dR0sthTR3qSOU7ypWVeXdZRdRZx8 ni+pydZiBK+rioV9vJNeUXvlLabTTd2YYt7UozfUZc+x+JKKUZCwhJUJJSOvSoy/28jhh5oHu9HSs ZZRKCB+g==; Received: from [2001:4bb8:180:e42a:50da:325f:4a06:8830] (helo=localhost) by bombadil.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1opslC-005gex-65; Tue, 01 Nov 2022 15:01:05 +0000 From: Christoph Hellwig To: Jens Axboe , Keith Busch , Sagi Grimberg , Chao Leng Cc: Ming Lei , linux-nvme@lists.infradead.org, linux-block@vger.kernel.org Subject: [PATCH 03/14] nvme: don't remove namespaces in nvme_passthru_end Date: Tue, 1 Nov 2022 16:00:39 +0100 Message-Id: <20221101150050.3510-4-hch@lst.de> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20221101150050.3510-1-hch@lst.de> References: <20221101150050.3510-1-hch@lst.de> MIME-Version: 1.0 X-SRS-Rewrite: SMTP reverse-path rewritten from by bombadil.infradead.org. See http://www.infradead.org/rpr.html Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org The call to nvme_remove_invalid_namespaces made sense when nvme_passthru_end revalidated all namespaces and had to remove those that didn't exist any more. Since we don't revalidate from nvme_passthru_end now, this call is entirely spurious. Signed-off-by: Christoph Hellwig Reviewed-by: Sagi Grimberg Reviewed-by: Chaitanya Kulkarni --- drivers/nvme/host/core.c | 1 - 1 file changed, 1 deletion(-) diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c index aea0f89acf409..35386aa24f589 100644 --- a/drivers/nvme/host/core.c +++ b/drivers/nvme/host/core.c @@ -1118,7 +1118,6 @@ void nvme_passthru_end(struct nvme_ctrl *ctrl, u32 effects, nvme_unfreeze(ctrl); nvme_mpath_unfreeze(ctrl->subsys); mutex_unlock(&ctrl->subsys->lock); - nvme_remove_invalid_namespaces(ctrl, NVME_NSID_ALL); mutex_unlock(&ctrl->scan_lock); } if (effects & NVME_CMD_EFFECTS_CCC) From patchwork Tue Nov 1 15:00:40 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christoph Hellwig X-Patchwork-Id: 13027082 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4A041C433FE for ; Tue, 1 Nov 2022 15:08:19 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231496AbiKAPIR (ORCPT ); Tue, 1 Nov 2022 11:08:17 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60522 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231356AbiKAPHz (ORCPT ); Tue, 1 Nov 2022 11:07:55 -0400 Received: from bombadil.infradead.org (bombadil.infradead.org [IPv6:2607:7c80:54:3::133]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 541F61E709 for ; Tue, 1 Nov 2022 08:01:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20210309; h=Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender :Reply-To:Content-Type:Content-ID:Content-Description; bh=VftSGhpevZu6/5y82Vykr3F9H9L8vKbEyML7+10xdA4=; b=sO/Ocqo4arwe7z6saGd3iA1SxP wJhVK1QAXSrMda9/aBLdRa7oCX8GB90Wtx9w0SvtXVPwzym/FhLbp5JYjER3ls3tv5JvnS+z7nbT5 S2TYlIqe8tMDJWivUpqGVvPJa8fTOv/xGia6chhB34Tl1Vg5O9eSS2ofGeTThYJ3VhZKVzkl3TlR9 17lfU6U+ncDcKrm+6AKKsv2LQGE9NwmAc6SxE8riTPTt1+CVOq+C4fyXZ/SKnwduaV2zjn4x4XDxd pt8LpfyWXpVeT3Skn5sktbU9xcw2AnsBM+QInAtWVmHBjfbhnP7dSNELroe2VXk9DekMQOsxt1bdC p5urW9xg==; Received: from [2001:4bb8:180:e42a:50da:325f:4a06:8830] (helo=localhost) by bombadil.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1opslI-005giM-8a; Tue, 01 Nov 2022 15:01:08 +0000 From: Christoph Hellwig To: Jens Axboe , Keith Busch , Sagi Grimberg , Chao Leng Cc: Ming Lei , linux-nvme@lists.infradead.org, linux-block@vger.kernel.org Subject: [PATCH 04/14] nvme: remove the NVME_NS_DEAD check in nvme_remove_invalid_namespaces Date: Tue, 1 Nov 2022 16:00:40 +0100 Message-Id: <20221101150050.3510-5-hch@lst.de> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20221101150050.3510-1-hch@lst.de> References: <20221101150050.3510-1-hch@lst.de> MIME-Version: 1.0 X-SRS-Rewrite: SMTP reverse-path rewritten from by bombadil.infradead.org. See http://www.infradead.org/rpr.html Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org The NVME_NS_DEAD check only made sense when we revalidated namespaces in nvme_passthrough_end for commands that affected the namespace inventory. These days NVME_NS_DEAD is only set during reset or when tearing down namespaces, and we always remove all namespaces right after that. Signed-off-by: Christoph Hellwig Reviewed-by: Sagi Grimberg Reviewed-by: Chaitanya Kulkarni --- drivers/nvme/host/core.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c index 35386aa24f589..390f2a2db3238 100644 --- a/drivers/nvme/host/core.c +++ b/drivers/nvme/host/core.c @@ -4407,7 +4407,7 @@ static void nvme_remove_invalid_namespaces(struct nvme_ctrl *ctrl, down_write(&ctrl->namespaces_rwsem); list_for_each_entry_safe(ns, next, &ctrl->namespaces, list) { - if (ns->head->ns_id > nsid || test_bit(NVME_NS_DEAD, &ns->flags)) + if (ns->head->ns_id > nsid) list_move_tail(&ns->list, &rm_list); } up_write(&ctrl->namespaces_rwsem); From patchwork Tue Nov 1 15:00:41 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christoph Hellwig X-Patchwork-Id: 13027081 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B11F9C4332F for ; Tue, 1 Nov 2022 15:08:17 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231312AbiKAPIP (ORCPT ); Tue, 1 Nov 2022 11:08:15 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60498 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231298AbiKAPHx (ORCPT ); Tue, 1 Nov 2022 11:07:53 -0400 Received: from bombadil.infradead.org (bombadil.infradead.org [IPv6:2607:7c80:54:3::133]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 60CFD1E728 for ; Tue, 1 Nov 2022 08:01:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20210309; h=Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender :Reply-To:Content-Type:Content-ID:Content-Description; bh=+0iu+A0Po72NG6PsBiW6InjA/PL4RlYWhhZ8s6hwjKQ=; b=Y4v5OIc0tTDNx87TVR4V5J16uP m7Hys8t+VdOYhuqVUnB14wX/4vdYhUhsu1ixbsyb9/MNa13cB4Dh1+gO3pTXyVXoH3EZOksHo8O2F 0n0kVkoyPxmAfiY9DLqbC5EsN+wJFbSncObpCP1nwdyHndGxoZ8mFF+/xIYMJIm98eJkEU6DNZhlc KrxtxD/oztyme6FhCI37+eOmHy9RwnVAVrcpW+IBWsIJZdJiGV9k1dXbx9eKme98OAq5KjKq4wv40 Z3b1VPErwJH+lRrl9ybeVji6vegOaVIW3IeiPRgadC30GPE0chKzt6j1ZtcifuZRUrZl7rprBFw1N +pTKdpPQ==; Received: from [2001:4bb8:180:e42a:50da:325f:4a06:8830] (helo=localhost) by bombadil.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1opslL-005gj1-0g; Tue, 01 Nov 2022 15:01:11 +0000 From: Christoph Hellwig To: Jens Axboe , Keith Busch , Sagi Grimberg , Chao Leng Cc: Ming Lei , linux-nvme@lists.infradead.org, linux-block@vger.kernel.org Subject: [PATCH 05/14] nvme: remove the NVME_NS_DEAD check in nvme_validate_ns Date: Tue, 1 Nov 2022 16:00:41 +0100 Message-Id: <20221101150050.3510-6-hch@lst.de> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20221101150050.3510-1-hch@lst.de> References: <20221101150050.3510-1-hch@lst.de> MIME-Version: 1.0 X-SRS-Rewrite: SMTP reverse-path rewritten from by bombadil.infradead.org. See http://www.infradead.org/rpr.html Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org At the point where namespaces are marked dead, the controller is in a non-live state and we won't get pass the identify commands. Signed-off-by: Christoph Hellwig Reviewed-by: Chaitanya Kulkarni --- drivers/nvme/host/core.c | 4 ---- 1 file changed, 4 deletions(-) diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c index 390f2a2db3238..871a8ab7ec199 100644 --- a/drivers/nvme/host/core.c +++ b/drivers/nvme/host/core.c @@ -4333,10 +4333,6 @@ static void nvme_validate_ns(struct nvme_ns *ns, struct nvme_ns_info *info) { int ret = NVME_SC_INVALID_NS | NVME_SC_DNR; - if (test_bit(NVME_NS_DEAD, &ns->flags)) - goto out; - - ret = NVME_SC_INVALID_NS | NVME_SC_DNR; if (!nvme_ns_ids_equal(&ns->head->ids, &info->ids)) { dev_err(ns->ctrl->device, "identifiers changed for nsid %d\n", ns->head->ns_id); From patchwork Tue Nov 1 15:00:42 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christoph Hellwig X-Patchwork-Id: 13027083 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id DFF39C433FE for ; Tue, 1 Nov 2022 15:08:31 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230376AbiKAPI3 (ORCPT ); Tue, 1 Nov 2022 11:08:29 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60540 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231132AbiKAPH6 (ORCPT ); Tue, 1 Nov 2022 11:07:58 -0400 Received: from bombadil.infradead.org (bombadil.infradead.org [IPv6:2607:7c80:54:3::133]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 60C1B1E725 for ; Tue, 1 Nov 2022 08:01:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20210309; h=Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender :Reply-To:Content-Type:Content-ID:Content-Description; bh=PLmgUevMzzrLRuQx97jRlnGVQcnqlnKGWkKm3cCI3M0=; b=CwyOGoskjE+aWQ8ZiBlLYAz1Td zRUxGkWNGJ3w1UADQkUdCAZpEztBPuwRCITBqTfqgBjBU8bzHJzbnArlQlAfi9YIWrAGkgUwopNYe DCEWumWOgVd9TBf6lcRSnfkf36Whd0ZgZtliLJIQBtC49xWdeRlHqw0ys+YLhQJEZm7jeOJ9k08rz w5wyK82f2kpNWXG1mlyQD1An1uNezPCEH22s7/pFJZhruyaWV+bUBX79NJFOMLzH8TRPEEKvDePW/ VmvpqZVCRvY9aiIrHBM+Ph3B7FvdTbc5sjHx8v1x4TlA7D3sxY2/LxBACsJqy/wjkd/78imn2DVgS AIn5OYBQ==; Received: from [2001:4bb8:180:e42a:50da:325f:4a06:8830] (helo=localhost) by bombadil.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1opslN-005gkj-Vh; Tue, 01 Nov 2022 15:01:14 +0000 From: Christoph Hellwig To: Jens Axboe , Keith Busch , Sagi Grimberg , Chao Leng Cc: Ming Lei , linux-nvme@lists.infradead.org, linux-block@vger.kernel.org Subject: [PATCH 06/14] nvme: don't unquiesce the admin queue in nvme_kill_queues Date: Tue, 1 Nov 2022 16:00:42 +0100 Message-Id: <20221101150050.3510-7-hch@lst.de> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20221101150050.3510-1-hch@lst.de> References: <20221101150050.3510-1-hch@lst.de> MIME-Version: 1.0 X-SRS-Rewrite: SMTP reverse-path rewritten from by bombadil.infradead.org. See http://www.infradead.org/rpr.html Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org None of the callers of nvme_kill_queues needs it to unquiesce the admin queues, as all of them already do it themselves: 1) nvme_reset_work explicit call nvme_start_admin_queue toward the beginning of the function. The extra call to nvme_start_admin_queue in nvme_reset_work this won't do anything as NVME_CTRL_ADMIN_Q_STOPPED will already be cleared. 2) nvme_remove calls nvme_dev_disable with shutdown flag set to true at the very beginning of the function if the PCIe device was not present, which is the precondition for the call to nvme_kill_queues. nvme_dev_disable already calls nvme_start_admin_queue toward the end of the function when the shutdown flag is set to true, so the admin queue is already enabled at this point. 3) nvme_remove_dead_ctrl schedules a workqueue to unbind the driver, which will end up in nvme_remove, which calls nvme_dev_disable with the shutdown flag. This case will call nvme_start_admin_queue a bit later than before. 4) apple_nvme_remove uses the same sequence as nvme_remove_dead_ctrl above. 5) nvme_remove_namespaces only calls nvme_kill_queues when the controller is in the DEAD state. That can only happen in the PCIe driver, and only from nvme_remove. See item 2) above for the conditions there. So it is safe to just remove the call to nvme_start_admin_queue in nvme_kill_queues without replacement. Signed-off-by: Christoph Hellwig Reviewed-by: Sagi Grimberg Reviewed-by: Chaitanya Kulkarni --- drivers/nvme/host/core.c | 4 ---- 1 file changed, 4 deletions(-) diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c index 871a8ab7ec199..bb62803de5b21 100644 --- a/drivers/nvme/host/core.c +++ b/drivers/nvme/host/core.c @@ -5135,10 +5135,6 @@ void nvme_kill_queues(struct nvme_ctrl *ctrl) down_read(&ctrl->namespaces_rwsem); - /* Forcibly unquiesce queues to avoid blocking dispatch */ - if (ctrl->admin_q && !blk_queue_dying(ctrl->admin_q)) - nvme_start_admin_queue(ctrl); - list_for_each_entry(ns, &ctrl->namespaces, list) nvme_set_queue_dying(ns); From patchwork Tue Nov 1 15:00:43 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christoph Hellwig X-Patchwork-Id: 13027085 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 403F1C433FE for ; Tue, 1 Nov 2022 15:08:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231381AbiKAPIf (ORCPT ); Tue, 1 Nov 2022 11:08:35 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59706 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230421AbiKAPIF (ORCPT ); Tue, 1 Nov 2022 11:08:05 -0400 Received: from bombadil.infradead.org (bombadil.infradead.org [IPv6:2607:7c80:54:3::133]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 551141E73D for ; Tue, 1 Nov 2022 08:01:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20210309; h=Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender :Reply-To:Content-Type:Content-ID:Content-Description; bh=Be+zd3xtXP6zSt/C9rrJRBJ64ghK1c1HAkxsmDcJqRk=; b=CaqmACPD9AoE3KLkg1+BmxZwUR DCinCE7alPLSP08Vu0IeX2mjqJb+PTaDYaUTA5dwFHb1VCO/Y/jrjSZT27lyQCv85pdKSTObgnvQE 491/P5uisUxbNFaM/xAxIp84tj8uwG1Ce025e/mjD6CtbY5E655itC349XqslUvgyJqzHg7abEWPY /pkFUDrtZlAQ/A1T/JxO9vRFnz+tlX/kHkhv5v5EXj5JZrTgX5PAhfHatkCBjgXe7sNb0MzgcYuuL 1iLO6M78GHV73v9X4SVKkYBRgM75cUs5DeMzHvVfg2/tJW6yu/I8DBMMrough45qvq/uAw0NUQHHp AeZXp0vQ==; Received: from [2001:4bb8:180:e42a:50da:325f:4a06:8830] (helo=localhost) by bombadil.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1opslQ-005glu-Rj; Tue, 01 Nov 2022 15:01:17 +0000 From: Christoph Hellwig To: Jens Axboe , Keith Busch , Sagi Grimberg , Chao Leng Cc: Ming Lei , linux-nvme@lists.infradead.org, linux-block@vger.kernel.org Subject: [PATCH 07/14] nvme: split nvme_kill_queues Date: Tue, 1 Nov 2022 16:00:43 +0100 Message-Id: <20221101150050.3510-8-hch@lst.de> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20221101150050.3510-1-hch@lst.de> References: <20221101150050.3510-1-hch@lst.de> MIME-Version: 1.0 X-SRS-Rewrite: SMTP reverse-path rewritten from by bombadil.infradead.org. See http://www.infradead.org/rpr.html Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org nvme_kill_queues does two things: 1) mark the gendisk of all namespaces dead 2) unquiesce all I/O queues These used to be be intertwined due to block layer issues, but aren't any more. So move the unquiscing of the I/O queues into the callers, and rename the rest of the function to the now more descriptive nvme_mark_namespaces_dead. Signed-off-by: Christoph Hellwig Reviewed-by: Sagi Grimberg Reviewed-by: Chaitanya Kulkarni --- drivers/nvme/host/apple.c | 3 ++- drivers/nvme/host/core.c | 36 ++++++++---------------------------- drivers/nvme/host/nvme.h | 3 +-- drivers/nvme/host/pci.c | 6 ++++-- 4 files changed, 15 insertions(+), 33 deletions(-) diff --git a/drivers/nvme/host/apple.c b/drivers/nvme/host/apple.c index a89b06a7099f1..6c09703ffe922 100644 --- a/drivers/nvme/host/apple.c +++ b/drivers/nvme/host/apple.c @@ -1153,7 +1153,8 @@ static void apple_nvme_reset_work(struct work_struct *work) nvme_change_ctrl_state(&anv->ctrl, NVME_CTRL_DELETING); nvme_get_ctrl(&anv->ctrl); apple_nvme_disable(anv, false); - nvme_kill_queues(&anv->ctrl); + nvme_mark_namespaces_dead(&anv->ctrl); + nvme_start_queues(&anv->ctrl); if (!queue_work(nvme_wq, &anv->remove_work)) nvme_put_ctrl(&anv->ctrl); } diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c index bb62803de5b21..ed06fcb87f93a 100644 --- a/drivers/nvme/host/core.c +++ b/drivers/nvme/host/core.c @@ -4561,8 +4561,10 @@ void nvme_remove_namespaces(struct nvme_ctrl *ctrl) * removing the namespaces' disks; fail all the queues now to avoid * potentially having to clean up the failed sync later. */ - if (ctrl->state == NVME_CTRL_DEAD) - nvme_kill_queues(ctrl); + if (ctrl->state == NVME_CTRL_DEAD) { + nvme_mark_namespaces_dead(ctrl); + nvme_start_queues(ctrl); + } /* this is a no-op when called from the controller reset handler */ nvme_change_ctrl_state(ctrl, NVME_CTRL_DELETING_NOIO); @@ -5108,39 +5110,17 @@ static void nvme_stop_ns_queue(struct nvme_ns *ns) blk_mq_wait_quiesce_done(ns->queue); } -/* - * Prepare a queue for teardown. - * - * This must forcibly unquiesce queues to avoid blocking dispatch. - */ -static void nvme_set_queue_dying(struct nvme_ns *ns) -{ - if (test_and_set_bit(NVME_NS_DEAD, &ns->flags)) - return; - - blk_mark_disk_dead(ns->disk); - nvme_start_ns_queue(ns); -} - -/** - * nvme_kill_queues(): Ends all namespace queues - * @ctrl: the dead controller that needs to end - * - * Call this function when the driver determines it is unable to get the - * controller in a state capable of servicing IO. - */ -void nvme_kill_queues(struct nvme_ctrl *ctrl) +/* let I/O to all namespaces fail in preparation for surprise removal */ +void nvme_mark_namespaces_dead(struct nvme_ctrl *ctrl) { struct nvme_ns *ns; down_read(&ctrl->namespaces_rwsem); - list_for_each_entry(ns, &ctrl->namespaces, list) - nvme_set_queue_dying(ns); - + blk_mark_disk_dead(ns->disk); up_read(&ctrl->namespaces_rwsem); } -EXPORT_SYMBOL_GPL(nvme_kill_queues); +EXPORT_SYMBOL_GPL(nvme_mark_namespaces_dead); void nvme_unfreeze(struct nvme_ctrl *ctrl) { diff --git a/drivers/nvme/host/nvme.h b/drivers/nvme/host/nvme.h index a29877217ee65..1cd7168002438 100644 --- a/drivers/nvme/host/nvme.h +++ b/drivers/nvme/host/nvme.h @@ -483,7 +483,6 @@ struct nvme_ns { unsigned long features; unsigned long flags; #define NVME_NS_REMOVING 0 -#define NVME_NS_DEAD 1 #define NVME_NS_ANA_PENDING 2 #define NVME_NS_FORCE_RO 3 #define NVME_NS_READY 4 @@ -758,7 +757,7 @@ void nvme_stop_queues(struct nvme_ctrl *ctrl); void nvme_start_queues(struct nvme_ctrl *ctrl); void nvme_stop_admin_queue(struct nvme_ctrl *ctrl); void nvme_start_admin_queue(struct nvme_ctrl *ctrl); -void nvme_kill_queues(struct nvme_ctrl *ctrl); +void nvme_mark_namespaces_dead(struct nvme_ctrl *ctrl); void nvme_sync_queues(struct nvme_ctrl *ctrl); void nvme_sync_io_queues(struct nvme_ctrl *ctrl); void nvme_unfreeze(struct nvme_ctrl *ctrl); diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c index 0e8410edb41bf..046fc02914a5a 100644 --- a/drivers/nvme/host/pci.c +++ b/drivers/nvme/host/pci.c @@ -2788,7 +2788,8 @@ static void nvme_remove_dead_ctrl(struct nvme_dev *dev) nvme_change_ctrl_state(&dev->ctrl, NVME_CTRL_DELETING); nvme_get_ctrl(&dev->ctrl); nvme_dev_disable(dev, false); - nvme_kill_queues(&dev->ctrl); + nvme_mark_namespaces_dead(&dev->ctrl); + nvme_start_queues(&dev->ctrl); if (!queue_work(nvme_wq, &dev->remove_work)) nvme_put_ctrl(&dev->ctrl); } @@ -2913,7 +2914,8 @@ static void nvme_reset_work(struct work_struct *work) nvme_unfreeze(&dev->ctrl); } else { dev_warn(dev->ctrl.device, "IO queues lost\n"); - nvme_kill_queues(&dev->ctrl); + nvme_mark_namespaces_dead(&dev->ctrl); + nvme_start_queues(&dev->ctrl); nvme_remove_namespaces(&dev->ctrl); nvme_free_tagset(dev); } From patchwork Tue Nov 1 15:00:44 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christoph Hellwig X-Patchwork-Id: 13027084 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6FC93C4332F for ; Tue, 1 Nov 2022 15:08:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231382AbiKAPIg (ORCPT ); Tue, 1 Nov 2022 11:08:36 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60476 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230291AbiKAPIH (ORCPT ); Tue, 1 Nov 2022 11:08:07 -0400 Received: from bombadil.infradead.org (bombadil.infradead.org [IPv6:2607:7c80:54:3::133]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AD22D2251A for ; Tue, 1 Nov 2022 08:01:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20210309; h=Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender :Reply-To:Content-Type:Content-ID:Content-Description; bh=+Ufifm1k1HpMzIHQoATSkCrRQA+hWQleQGdeWF3/Tzw=; b=rIzIPqzAjcUfTQi1SOO0l0QHVO rHgq0tBoUTLhRCR7w4WbVRbu39grwCG+ziZG7XHV/pd9wu7T+/aqJf0uKaqAbHsZxCHRDhBXgjeJd ts1u93rFmCWIbZ5DvBczEKIbD3vi1Ct7b3yMTzPfIXPLBbg7gennLOMpwBHE2ygxvavlH4lhhbZIC +WbxxNl0SIpVPUbwOGFeH1/e4TYlJ9G+O6+M41e4+Op3oGKzVE4O6Q4VfjVsB/UGYAoI0HbFfiDC8 sqICAvK/SsowLLxqqGtdP1zuf8hB/xUIOnyY/Op7043VbPOAYZ4WQ1cXFL2mickpSSoc4jIFzS/Cb uaM5zqKA==; Received: from [2001:4bb8:180:e42a:50da:325f:4a06:8830] (helo=localhost) by bombadil.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1opslT-005gn8-Tq; Tue, 01 Nov 2022 15:01:20 +0000 From: Christoph Hellwig To: Jens Axboe , Keith Busch , Sagi Grimberg , Chao Leng Cc: Ming Lei , linux-nvme@lists.infradead.org, linux-block@vger.kernel.org Subject: [PATCH 08/14] nvme-pci: don't unquiesce the I/O queues in nvme_remove_dead_ctrl Date: Tue, 1 Nov 2022 16:00:44 +0100 Message-Id: <20221101150050.3510-9-hch@lst.de> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20221101150050.3510-1-hch@lst.de> References: <20221101150050.3510-1-hch@lst.de> MIME-Version: 1.0 X-SRS-Rewrite: SMTP reverse-path rewritten from by bombadil.infradead.org. See http://www.infradead.org/rpr.html Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org nvme_remove_dead_ctrl schedules nvme_remove to be called, which will call nvme_dev_disable and unquiesce the I/O queues. Signed-off-by: Christoph Hellwig Reviewed-by: Chaitanya Kulkarni --- drivers/nvme/host/pci.c | 1 - 1 file changed, 1 deletion(-) diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c index 046fc02914a5a..159bc8918bbf0 100644 --- a/drivers/nvme/host/pci.c +++ b/drivers/nvme/host/pci.c @@ -2789,7 +2789,6 @@ static void nvme_remove_dead_ctrl(struct nvme_dev *dev) nvme_get_ctrl(&dev->ctrl); nvme_dev_disable(dev, false); nvme_mark_namespaces_dead(&dev->ctrl); - nvme_start_queues(&dev->ctrl); if (!queue_work(nvme_wq, &dev->remove_work)) nvme_put_ctrl(&dev->ctrl); } From patchwork Tue Nov 1 15:00:45 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christoph Hellwig X-Patchwork-Id: 13027086 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D89AFC433FE for ; Tue, 1 Nov 2022 15:08:43 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231265AbiKAPIm (ORCPT ); Tue, 1 Nov 2022 11:08:42 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59688 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230442AbiKAPIO (ORCPT ); Tue, 1 Nov 2022 11:08:14 -0400 Received: from bombadil.infradead.org (bombadil.infradead.org [IPv6:2607:7c80:54:3::133]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4918722B12 for ; Tue, 1 Nov 2022 08:01:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20210309; h=Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender :Reply-To:Content-Type:Content-ID:Content-Description; bh=rm/IRbGeMyvZjy9/i6jToMGsSHuJMehfKODHL4Mhra0=; b=P6YQMMwxgbRqoCVh28US5cnadX oHHufJqdT7y3Al9m/vYuJUdawzkTwPvAn/gG8WHHLniixMVt1Z4h7kWssAM6k+zB36hwT7AuhNUwT zwm36LHv44MywKhCFUb+AFtu7ZKvx8VqEZAY6kLLdfjTyw78sU5NiyOqekKiiGu5+NE347cOZggcK N/2h9nKb+1P1qoQ/pco19u2vJqjVHMz2mYo/+W2Qnx8Zc668GfWg+E9x1KVTx7I2vmqcirVaEBVWF jd8iD2zA8lbkHC2c8UsEAv+4SHBNiBevggM99q1dSkvccaoabjixDeJt7SrFZryhac+rFv3Dr0dA2 Z43R9jSg==; Received: from [2001:4bb8:180:e42a:50da:325f:4a06:8830] (helo=localhost) by bombadil.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1opslW-005gov-Tt; Tue, 01 Nov 2022 15:01:23 +0000 From: Christoph Hellwig To: Jens Axboe , Keith Busch , Sagi Grimberg , Chao Leng Cc: Ming Lei , linux-nvme@lists.infradead.org, linux-block@vger.kernel.org Subject: [PATCH 09/14] nvme-apple: don't unquiesce the I/O queues in apple_nvme_reset_work Date: Tue, 1 Nov 2022 16:00:45 +0100 Message-Id: <20221101150050.3510-10-hch@lst.de> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20221101150050.3510-1-hch@lst.de> References: <20221101150050.3510-1-hch@lst.de> MIME-Version: 1.0 X-SRS-Rewrite: SMTP reverse-path rewritten from by bombadil.infradead.org. See http://www.infradead.org/rpr.html Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org apple_nvme_reset_work schedules apple_nvme_remove, to be called, which will call apple_nvme_disable and unquiesce the I/O queues. Signed-off-by: Christoph Hellwig Reviewed-by: Sagi Grimberg Reviewed-by: Chaitanya Kulkarni --- drivers/nvme/host/apple.c | 1 - 1 file changed, 1 deletion(-) diff --git a/drivers/nvme/host/apple.c b/drivers/nvme/host/apple.c index 6c09703ffe922..24e224c279a41 100644 --- a/drivers/nvme/host/apple.c +++ b/drivers/nvme/host/apple.c @@ -1154,7 +1154,6 @@ static void apple_nvme_reset_work(struct work_struct *work) nvme_get_ctrl(&anv->ctrl); apple_nvme_disable(anv, false); nvme_mark_namespaces_dead(&anv->ctrl); - nvme_start_queues(&anv->ctrl); if (!queue_work(nvme_wq, &anv->remove_work)) nvme_put_ctrl(&anv->ctrl); } From patchwork Tue Nov 1 15:00:46 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christoph Hellwig X-Patchwork-Id: 13027087 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 557DEC433FE for ; Tue, 1 Nov 2022 15:08:48 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230503AbiKAPIq (ORCPT ); Tue, 1 Nov 2022 11:08:46 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60520 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231298AbiKAPIP (ORCPT ); Tue, 1 Nov 2022 11:08:15 -0400 Received: from bombadil.infradead.org (bombadil.infradead.org [IPv6:2607:7c80:54:3::133]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 39F6922B2A for ; Tue, 1 Nov 2022 08:01:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20210309; h=Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender :Reply-To:Content-Type:Content-ID:Content-Description; bh=vFIffKMuruxaQXIDQ4KlSxJoVoIR2c5sW2+Wi5AWYDI=; b=DtetXMpKTvWn3HezyW+8KtmYTv RvZOTXigRlmLrfHy9FPjMf9y1in4QugdE5e+SDkMJ9WzVBxjnjGUFw9aEhX0LKqfBkWOCdIgTbol4 GTfAML6bK6CBT1AqH/3UJGIzqtmBUEJAWxHryXu3XrDvdXame5pBrlutovfxwwEd8pw84pBgJYx1M YSUQ4P6GBFDdkayI47yScvsOGq2bdCNK4Fid0o1oLl6crNrtknhLk3z399K/0wYplFynNTgtZ0UYg HZ6s9N1wwC7u76MPM3OLOe9IkEAzm4O5O86l4+aClhsc5RV8UmoscVzNRpaqnMG17G6C+wpjsKBmZ pkZ/c/QQ==; Received: from [2001:4bb8:180:e42a:50da:325f:4a06:8830] (helo=localhost) by bombadil.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1opsla-005gqg-2H; Tue, 01 Nov 2022 15:01:26 +0000 From: Christoph Hellwig To: Jens Axboe , Keith Busch , Sagi Grimberg , Chao Leng Cc: Ming Lei , linux-nvme@lists.infradead.org, linux-block@vger.kernel.org, Hannes Reinecke Subject: [PATCH 10/14] blk-mq: skip non-mq queues in blk_mq_quiesce_queue Date: Tue, 1 Nov 2022 16:00:46 +0100 Message-Id: <20221101150050.3510-11-hch@lst.de> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20221101150050.3510-1-hch@lst.de> References: <20221101150050.3510-1-hch@lst.de> MIME-Version: 1.0 X-SRS-Rewrite: SMTP reverse-path rewritten from by bombadil.infradead.org. See http://www.infradead.org/rpr.html Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org For submit_bio based queues there is no (S)RCU critical section during I/O submission and thus nothing to wait for in blk_mq_wait_quiesce_done, so skip doing any synchronization. No non-mq driver should be calling this, but for now we have core callers that unconditionally call into it. Signed-off-by: Christoph Hellwig Reviewed-by: Keith Busch Reviewed-by: Sagi Grimberg Reviewed-by: Ming Lei Reviewed-by: Hannes Reinecke Reviewed-by: Chaitanya Kulkarni --- block/blk-mq.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/block/blk-mq.c b/block/blk-mq.c index 623e8a506539c..b8f37cfd3ae22 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -280,7 +280,9 @@ EXPORT_SYMBOL_GPL(blk_mq_wait_quiesce_done); void blk_mq_quiesce_queue(struct request_queue *q) { blk_mq_quiesce_queue_nowait(q); - blk_mq_wait_quiesce_done(q); + /* nothing to wait for non-mq queues */ + if (queue_is_mq(q)) + blk_mq_wait_quiesce_done(q); } EXPORT_SYMBOL_GPL(blk_mq_quiesce_queue); From patchwork Tue Nov 1 15:00:47 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christoph Hellwig X-Patchwork-Id: 13027088 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 02079C4332F for ; Tue, 1 Nov 2022 15:08:51 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230515AbiKAPIu (ORCPT ); Tue, 1 Nov 2022 11:08:50 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60668 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231166AbiKAPIb (ORCPT ); Tue, 1 Nov 2022 11:08:31 -0400 Received: from bombadil.infradead.org (bombadil.infradead.org [IPv6:2607:7c80:54:3::133]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 99AB91DDF4 for ; Tue, 1 Nov 2022 08:01:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20210309; h=Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender :Reply-To:Content-Type:Content-ID:Content-Description; bh=zmbgApDCz9PmuMoSQG07GoSCQLaKF32hxFbS25zR6Y4=; b=od7BN4mKQSQXsOzRnL+OKZsePA tswAmW0SQjt31Q4lylB5Jl+fgwb/NxusP8YGQYdBoCMHSKjqNTXyTLsFPoTEtDBPYxxo7KAxaR9ak 2GhMaAxPU9HZu2lKRtZx/vGKhDlYHNwzS9KyBhSwxEgGWtO+2cHGRb6BKah6hA9SZ11XCIADvgVdZ NarJH1+e9YCxoQjP2fnACABE7Cjof3xwyqmJLT6ZmQAIp0KOHmEueglCPFw7f2iW3AEJTURn5+fd/ 2f+fvIdaMObbhCC8iD+dxBycFsFuIw5omfZlpxopcA3Uea/4xHrMoUuXKeZn5s2rabItyjbLo5cEI znBJi/sw==; Received: from [2001:4bb8:180:e42a:50da:325f:4a06:8830] (helo=localhost) by bombadil.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1opslc-005gsj-Ti; Tue, 01 Nov 2022 15:01:29 +0000 From: Christoph Hellwig To: Jens Axboe , Keith Busch , Sagi Grimberg , Chao Leng Cc: Ming Lei , linux-nvme@lists.infradead.org, linux-block@vger.kernel.org, Hannes Reinecke Subject: [PATCH 11/14] blk-mq: move the srcu_struct used for quiescing to the tagset Date: Tue, 1 Nov 2022 16:00:47 +0100 Message-Id: <20221101150050.3510-12-hch@lst.de> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20221101150050.3510-1-hch@lst.de> References: <20221101150050.3510-1-hch@lst.de> MIME-Version: 1.0 X-SRS-Rewrite: SMTP reverse-path rewritten from by bombadil.infradead.org. See http://www.infradead.org/rpr.html Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org All I/O submissions have fairly similar latencies, and a tagset-wide quiesce is a fairly common operation. Signed-off-by: Christoph Hellwig Reviewed-by: Keith Busch Reviewed-by: Ming Lei Reviewed-by: Chao Leng Reviewed-by: Sagi Grimberg Reviewed-by: Hannes Reinecke Reviewed-by: Chaitanya Kulkarni --- block/blk-core.c | 27 +++++---------------------- block/blk-mq.c | 33 +++++++++++++++++++++++++-------- block/blk-mq.h | 14 +++++++------- block/blk-sysfs.c | 9 ++------- block/blk.h | 9 +-------- block/genhd.c | 2 +- include/linux/blk-mq.h | 4 ++++ include/linux/blkdev.h | 9 --------- 8 files changed, 45 insertions(+), 62 deletions(-) diff --git a/block/blk-core.c b/block/blk-core.c index 5d50dd16e2a59..e9e2bf15cd909 100644 --- a/block/blk-core.c +++ b/block/blk-core.c @@ -65,7 +65,6 @@ DEFINE_IDA(blk_queue_ida); * For queue allocation */ struct kmem_cache *blk_requestq_cachep; -struct kmem_cache *blk_requestq_srcu_cachep; /* * Controlling structure to kblockd @@ -373,26 +372,20 @@ static void blk_timeout_work(struct work_struct *work) { } -struct request_queue *blk_alloc_queue(int node_id, bool alloc_srcu) +struct request_queue *blk_alloc_queue(int node_id) { struct request_queue *q; - q = kmem_cache_alloc_node(blk_get_queue_kmem_cache(alloc_srcu), - GFP_KERNEL | __GFP_ZERO, node_id); + q = kmem_cache_alloc_node(blk_requestq_cachep, GFP_KERNEL | __GFP_ZERO, + node_id); if (!q) return NULL; - if (alloc_srcu) { - blk_queue_flag_set(QUEUE_FLAG_HAS_SRCU, q); - if (init_srcu_struct(q->srcu) != 0) - goto fail_q; - } - q->last_merge = NULL; q->id = ida_alloc(&blk_queue_ida, GFP_KERNEL); if (q->id < 0) - goto fail_srcu; + goto fail_q; q->stats = blk_alloc_queue_stats(); if (!q->stats) @@ -435,11 +428,8 @@ struct request_queue *blk_alloc_queue(int node_id, bool alloc_srcu) blk_free_queue_stats(q->stats); fail_id: ida_free(&blk_queue_ida, q->id); -fail_srcu: - if (alloc_srcu) - cleanup_srcu_struct(q->srcu); fail_q: - kmem_cache_free(blk_get_queue_kmem_cache(alloc_srcu), q); + kmem_cache_free(blk_requestq_cachep, q); return NULL; } @@ -1172,9 +1162,6 @@ int __init blk_dev_init(void) sizeof_field(struct request, cmd_flags)); BUILD_BUG_ON(REQ_OP_BITS + REQ_FLAG_BITS > 8 * sizeof_field(struct bio, bi_opf)); - BUILD_BUG_ON(ALIGN(offsetof(struct request_queue, srcu), - __alignof__(struct request_queue)) != - sizeof(struct request_queue)); /* used for unplugging and affects IO latency/throughput - HIGHPRI */ kblockd_workqueue = alloc_workqueue("kblockd", @@ -1185,10 +1172,6 @@ int __init blk_dev_init(void) blk_requestq_cachep = kmem_cache_create("request_queue", sizeof(struct request_queue), 0, SLAB_PANIC, NULL); - blk_requestq_srcu_cachep = kmem_cache_create("request_queue_srcu", - sizeof(struct request_queue) + - sizeof(struct srcu_struct), 0, SLAB_PANIC, NULL); - blk_debugfs_root = debugfs_create_dir("block", NULL); return 0; diff --git a/block/blk-mq.c b/block/blk-mq.c index b8f37cfd3ae22..3479325fc86c3 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -261,8 +261,8 @@ EXPORT_SYMBOL_GPL(blk_mq_quiesce_queue_nowait); */ void blk_mq_wait_quiesce_done(struct request_queue *q) { - if (blk_queue_has_srcu(q)) - synchronize_srcu(q->srcu); + if (q->tag_set->flags & BLK_MQ_F_BLOCKING) + synchronize_srcu(q->tag_set->srcu); else synchronize_rcu(); } @@ -4003,7 +4003,7 @@ static struct request_queue *blk_mq_init_queue_data(struct blk_mq_tag_set *set, struct request_queue *q; int ret; - q = blk_alloc_queue(set->numa_node, set->flags & BLK_MQ_F_BLOCKING); + q = blk_alloc_queue(set->numa_node); if (!q) return ERR_PTR(-ENOMEM); q->queuedata = queuedata; @@ -4168,9 +4168,6 @@ static void blk_mq_update_poll_flag(struct request_queue *q) int blk_mq_init_allocated_queue(struct blk_mq_tag_set *set, struct request_queue *q) { - WARN_ON_ONCE(blk_queue_has_srcu(q) != - !!(set->flags & BLK_MQ_F_BLOCKING)); - /* mark the queue as mq asap */ q->mq_ops = set->ops; @@ -4428,9 +4425,19 @@ int blk_mq_alloc_tag_set(struct blk_mq_tag_set *set) */ if (set->nr_maps == 1 && set->nr_hw_queues > nr_cpu_ids) set->nr_hw_queues = nr_cpu_ids; + + if (set->flags & BLK_MQ_F_BLOCKING) { + set->srcu = kmalloc(sizeof(*set->srcu), GFP_KERNEL); + if (!set->srcu) + return -ENOMEM; + ret = init_srcu_struct(set->srcu); + if (ret) + goto out_free_srcu; + } - if (blk_mq_alloc_tag_set_tags(set, set->nr_hw_queues) < 0) - return -ENOMEM; + ret = blk_mq_alloc_tag_set_tags(set, set->nr_hw_queues); + if (ret) + goto out_cleanup_srcu; ret = -ENOMEM; for (i = 0; i < set->nr_maps; i++) { @@ -4460,6 +4467,12 @@ int blk_mq_alloc_tag_set(struct blk_mq_tag_set *set) } kfree(set->tags); set->tags = NULL; +out_cleanup_srcu: + if (set->flags & BLK_MQ_F_BLOCKING) + cleanup_srcu_struct(set->srcu); +out_free_srcu: + if (set->flags & BLK_MQ_F_BLOCKING) + kfree(set->srcu); return ret; } EXPORT_SYMBOL(blk_mq_alloc_tag_set); @@ -4499,6 +4512,10 @@ void blk_mq_free_tag_set(struct blk_mq_tag_set *set) kfree(set->tags); set->tags = NULL; + if (set->flags & BLK_MQ_F_BLOCKING) { + cleanup_srcu_struct(set->srcu); + kfree(set->srcu); + } } EXPORT_SYMBOL(blk_mq_free_tag_set); diff --git a/block/blk-mq.h b/block/blk-mq.h index 0b2870839cdd6..ef59fee62780d 100644 --- a/block/blk-mq.h +++ b/block/blk-mq.h @@ -377,17 +377,17 @@ static inline bool hctx_may_queue(struct blk_mq_hw_ctx *hctx, /* run the code block in @dispatch_ops with rcu/srcu read lock held */ #define __blk_mq_run_dispatch_ops(q, check_sleep, dispatch_ops) \ do { \ - if (!blk_queue_has_srcu(q)) { \ - rcu_read_lock(); \ - (dispatch_ops); \ - rcu_read_unlock(); \ - } else { \ + if ((q)->tag_set->flags & BLK_MQ_F_BLOCKING) { \ int srcu_idx; \ \ might_sleep_if(check_sleep); \ - srcu_idx = srcu_read_lock((q)->srcu); \ + srcu_idx = srcu_read_lock((q)->tag_set->srcu); \ (dispatch_ops); \ - srcu_read_unlock((q)->srcu, srcu_idx); \ + srcu_read_unlock((q)->tag_set->srcu, srcu_idx); \ + } else { \ + rcu_read_lock(); \ + (dispatch_ops); \ + rcu_read_unlock(); \ } \ } while (0) diff --git a/block/blk-sysfs.c b/block/blk-sysfs.c index 7b98c7074771d..02e94c4beff17 100644 --- a/block/blk-sysfs.c +++ b/block/blk-sysfs.c @@ -742,10 +742,8 @@ queue_attr_store(struct kobject *kobj, struct attribute *attr, static void blk_free_queue_rcu(struct rcu_head *rcu_head) { - struct request_queue *q = container_of(rcu_head, struct request_queue, - rcu_head); - - kmem_cache_free(blk_get_queue_kmem_cache(blk_queue_has_srcu(q)), q); + kmem_cache_free(blk_requestq_cachep, + container_of(rcu_head, struct request_queue, rcu_head)); } /** @@ -782,9 +780,6 @@ static void blk_release_queue(struct kobject *kobj) if (queue_is_mq(q)) blk_mq_release(q); - if (blk_queue_has_srcu(q)) - cleanup_srcu_struct(q->srcu); - ida_free(&blk_queue_ida, q->id); call_rcu(&q->rcu_head, blk_free_queue_rcu); } diff --git a/block/blk.h b/block/blk.h index 7f9e089ab1f75..37faec6e558b0 100644 --- a/block/blk.h +++ b/block/blk.h @@ -27,7 +27,6 @@ struct blk_flush_queue { }; extern struct kmem_cache *blk_requestq_cachep; -extern struct kmem_cache *blk_requestq_srcu_cachep; extern struct kobj_type blk_queue_ktype; extern struct ida blk_queue_ida; @@ -428,13 +427,7 @@ int bio_add_hw_page(struct request_queue *q, struct bio *bio, struct page *page, unsigned int len, unsigned int offset, unsigned int max_sectors, bool *same_page); -static inline struct kmem_cache *blk_get_queue_kmem_cache(bool srcu) -{ - if (srcu) - return blk_requestq_srcu_cachep; - return blk_requestq_cachep; -} -struct request_queue *blk_alloc_queue(int node_id, bool alloc_srcu); +struct request_queue *blk_alloc_queue(int node_id); int disk_scan_partitions(struct gendisk *disk, fmode_t mode); diff --git a/block/genhd.c b/block/genhd.c index e7bd036024fab..09cde914e0548 100644 --- a/block/genhd.c +++ b/block/genhd.c @@ -1414,7 +1414,7 @@ struct gendisk *__blk_alloc_disk(int node, struct lock_class_key *lkclass) struct request_queue *q; struct gendisk *disk; - q = blk_alloc_queue(node, false); + q = blk_alloc_queue(node); if (!q) return NULL; diff --git a/include/linux/blk-mq.h b/include/linux/blk-mq.h index 569053ed959d5..f059edebb11d8 100644 --- a/include/linux/blk-mq.h +++ b/include/linux/blk-mq.h @@ -7,6 +7,7 @@ #include #include #include +#include struct blk_mq_tags; struct blk_flush_queue; @@ -500,6 +501,8 @@ enum hctx_type { * @tag_list_lock: Serializes tag_list accesses. * @tag_list: List of the request queues that use this tag set. See also * request_queue.tag_set_list. + * @srcu: Use as lock when type of the request queue is blocking + * (BLK_MQ_F_BLOCKING). */ struct blk_mq_tag_set { struct blk_mq_queue_map map[HCTX_MAX_TYPES]; @@ -520,6 +523,7 @@ struct blk_mq_tag_set { struct mutex tag_list_lock; struct list_head tag_list; + struct srcu_struct *srcu; }; /** diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h index 32137d85c9ad5..6a6fa167fc828 100644 --- a/include/linux/blkdev.h +++ b/include/linux/blkdev.h @@ -22,7 +22,6 @@ #include #include #include -#include #include #include @@ -543,18 +542,11 @@ struct request_queue { struct mutex debugfs_mutex; bool mq_sysfs_init_done; - - /** - * @srcu: Sleepable RCU. Use as lock when type of the request queue - * is blocking (BLK_MQ_F_BLOCKING). Must be the last member - */ - struct srcu_struct srcu[]; }; /* Keep blk_queue_flag_name[] in sync with the definitions below */ #define QUEUE_FLAG_STOPPED 0 /* queue is stopped */ #define QUEUE_FLAG_DYING 1 /* queue being torn down */ -#define QUEUE_FLAG_HAS_SRCU 2 /* SRCU is allocated */ #define QUEUE_FLAG_NOMERGES 3 /* disable merge attempts */ #define QUEUE_FLAG_SAME_COMP 4 /* complete on same CPU-group */ #define QUEUE_FLAG_FAIL_IO 5 /* fake timeout */ @@ -590,7 +582,6 @@ bool blk_queue_flag_test_and_set(unsigned int flag, struct request_queue *q); #define blk_queue_stopped(q) test_bit(QUEUE_FLAG_STOPPED, &(q)->queue_flags) #define blk_queue_dying(q) test_bit(QUEUE_FLAG_DYING, &(q)->queue_flags) -#define blk_queue_has_srcu(q) test_bit(QUEUE_FLAG_HAS_SRCU, &(q)->queue_flags) #define blk_queue_init_done(q) test_bit(QUEUE_FLAG_INIT_DONE, &(q)->queue_flags) #define blk_queue_nomerges(q) test_bit(QUEUE_FLAG_NOMERGES, &(q)->queue_flags) #define blk_queue_noxmerges(q) \ From patchwork Tue Nov 1 15:00:48 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christoph Hellwig X-Patchwork-Id: 13027089 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0FB7AC433FE for ; Tue, 1 Nov 2022 15:08:53 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231350AbiKAPIv (ORCPT ); Tue, 1 Nov 2022 11:08:51 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60462 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231367AbiKAPIe (ORCPT ); Tue, 1 Nov 2022 11:08:34 -0400 Received: from bombadil.infradead.org (bombadil.infradead.org [IPv6:2607:7c80:54:3::133]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 68CE11EACA for ; Tue, 1 Nov 2022 08:01:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20210309; h=Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender :Reply-To:Content-Type:Content-ID:Content-Description; bh=LKSp7sTVrN/SlopF/xOv1f/B3XQWGZ/lAzcwBtezQBE=; b=M1GNlE5waHTMcKk0HQFjnrFDh8 d7q3mFNaq+QTAeqAbv/CKX4vnlmQqgEGKy6T2X+2T5nevUSB7UL7+YK4sPE6PETIx+XodPh1RhsR/ STjcSTjfSWYY+fUe9IMlCvEj1XuEcYxq1fPe199FGHt0W+qgIgvbnR0UJ27IXmrguwJuJHYtJLFEe M9QlzQrbdfAQC98dYObqMOigHQfcscaksRai8o4j6a4E/yj8PaH5AvQ7F6U0UDZGIjRKDRiQ387a3 dTnQPXWSytYfpetKxjsfiYUgh1KD1q5zKrTPeS82dJ+JRrDrO76XNONBloKIBIhLAxoRoywggRZXE V/+2+R+g==; Received: from [2001:4bb8:180:e42a:50da:325f:4a06:8830] (helo=localhost) by bombadil.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1opslf-005gvW-Q4; Tue, 01 Nov 2022 15:01:32 +0000 From: Christoph Hellwig To: Jens Axboe , Keith Busch , Sagi Grimberg , Chao Leng Cc: Ming Lei , linux-nvme@lists.infradead.org, linux-block@vger.kernel.org, Hannes Reinecke Subject: [PATCH 12/14] blk-mq: pass a tagset to blk_mq_wait_quiesce_done Date: Tue, 1 Nov 2022 16:00:48 +0100 Message-Id: <20221101150050.3510-13-hch@lst.de> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20221101150050.3510-1-hch@lst.de> References: <20221101150050.3510-1-hch@lst.de> MIME-Version: 1.0 X-SRS-Rewrite: SMTP reverse-path rewritten from by bombadil.infradead.org. See http://www.infradead.org/rpr.html Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org Nothing in blk_mq_wait_quiesce_done needs the request_queue now, so just pass the tagset, and move the non-mq check into the only caller that needs it. Signed-off-by: Christoph Hellwig Reviewed-by: Keith Busch Reviewed-by: Sagi Grimberg Reviewed-by: Chao Leng Reviewed-by: Hannes Reinecke Reviewed-by: Chaitanya Kulkarni --- block/blk-mq.c | 16 +++++++++------- drivers/nvme/host/core.c | 4 ++-- drivers/scsi/scsi_lib.c | 2 +- include/linux/blk-mq.h | 2 +- 4 files changed, 13 insertions(+), 11 deletions(-) diff --git a/block/blk-mq.c b/block/blk-mq.c index 3479325fc86c3..b358328a1dce3 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -254,15 +254,17 @@ EXPORT_SYMBOL_GPL(blk_mq_quiesce_queue_nowait); /** * blk_mq_wait_quiesce_done() - wait until in-progress quiesce is done - * @q: request queue. + * @set: tag_set to wait on * * Note: it is driver's responsibility for making sure that quiesce has - * been started. + * been started on or more of the request_queues of the tag_set. This + * function only waits for the quiesce on those request_queues that had + * the quiesce flag set using blk_mq_quiesce_queue_nowait. */ -void blk_mq_wait_quiesce_done(struct request_queue *q) +void blk_mq_wait_quiesce_done(struct blk_mq_tag_set *set) { - if (q->tag_set->flags & BLK_MQ_F_BLOCKING) - synchronize_srcu(q->tag_set->srcu); + if (set->flags & BLK_MQ_F_BLOCKING) + synchronize_srcu(set->srcu); else synchronize_rcu(); } @@ -282,7 +284,7 @@ void blk_mq_quiesce_queue(struct request_queue *q) blk_mq_quiesce_queue_nowait(q); /* nothing to wait for non-mq queues */ if (queue_is_mq(q)) - blk_mq_wait_quiesce_done(q); + blk_mq_wait_quiesce_done(q->tag_set); } EXPORT_SYMBOL_GPL(blk_mq_quiesce_queue); @@ -1623,7 +1625,7 @@ static void blk_mq_timeout_work(struct work_struct *work) * uses srcu or rcu, wait for a synchronization point to * ensure all running submits have finished */ - blk_mq_wait_quiesce_done(q); + blk_mq_wait_quiesce_done(q->tag_set); expired.next = 0; blk_mq_queue_tag_busy_iter(q, blk_mq_handle_expired, &expired); diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c index ed06fcb87f93a..66b0b6e110024 100644 --- a/drivers/nvme/host/core.c +++ b/drivers/nvme/host/core.c @@ -5107,7 +5107,7 @@ static void nvme_stop_ns_queue(struct nvme_ns *ns) if (!test_and_set_bit(NVME_NS_STOPPED, &ns->flags)) blk_mq_quiesce_queue(ns->queue); else - blk_mq_wait_quiesce_done(ns->queue); + blk_mq_wait_quiesce_done(ns->queue->tag_set); } /* let I/O to all namespaces fail in preparation for surprise removal */ @@ -5197,7 +5197,7 @@ void nvme_stop_admin_queue(struct nvme_ctrl *ctrl) if (!test_and_set_bit(NVME_CTRL_ADMIN_Q_STOPPED, &ctrl->flags)) blk_mq_quiesce_queue(ctrl->admin_q); else - blk_mq_wait_quiesce_done(ctrl->admin_q); + blk_mq_wait_quiesce_done(ctrl->admin_q->tag_set); } EXPORT_SYMBOL_GPL(nvme_stop_admin_queue); diff --git a/drivers/scsi/scsi_lib.c b/drivers/scsi/scsi_lib.c index 8b89fab7c4206..249757ddd8fea 100644 --- a/drivers/scsi/scsi_lib.c +++ b/drivers/scsi/scsi_lib.c @@ -2735,7 +2735,7 @@ static void scsi_stop_queue(struct scsi_device *sdev, bool nowait) blk_mq_quiesce_queue(sdev->request_queue); } else { if (!nowait) - blk_mq_wait_quiesce_done(sdev->request_queue); + blk_mq_wait_quiesce_done(sdev->request_queue->tag_set); } } diff --git a/include/linux/blk-mq.h b/include/linux/blk-mq.h index f059edebb11d8..061ea6e7af017 100644 --- a/include/linux/blk-mq.h +++ b/include/linux/blk-mq.h @@ -880,7 +880,7 @@ void blk_mq_start_hw_queues(struct request_queue *q); void blk_mq_start_stopped_hw_queue(struct blk_mq_hw_ctx *hctx, bool async); void blk_mq_start_stopped_hw_queues(struct request_queue *q, bool async); void blk_mq_quiesce_queue(struct request_queue *q); -void blk_mq_wait_quiesce_done(struct request_queue *q); +void blk_mq_wait_quiesce_done(struct blk_mq_tag_set *set); void blk_mq_unquiesce_queue(struct request_queue *q); void blk_mq_delay_run_hw_queue(struct blk_mq_hw_ctx *hctx, unsigned long msecs); void blk_mq_run_hw_queue(struct blk_mq_hw_ctx *hctx, bool async); From patchwork Tue Nov 1 15:00:49 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christoph Hellwig X-Patchwork-Id: 13027091 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id AE590C4332F for ; Tue, 1 Nov 2022 15:09:10 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231184AbiKAPIx (ORCPT ); Tue, 1 Nov 2022 11:08:53 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60106 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230499AbiKAPIe (ORCPT ); Tue, 1 Nov 2022 11:08:34 -0400 Received: from bombadil.infradead.org (bombadil.infradead.org [IPv6:2607:7c80:54:3::133]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B97ED1EADA for ; Tue, 1 Nov 2022 08:01:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20210309; h=Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender :Reply-To:Content-Type:Content-ID:Content-Description; bh=VbGBWZhiSpQF0UZ5IwyygVVuVA9A0T3gNpvRoR7aO0A=; b=Jq4Wuo20lSTbsmzyFLTWGOKwGQ fRYR4N+VfGeWAv44ZgtoqqFGW9Mre4LUOixpr2l0ui5jKSLGQdtuF0JZ78YUsdcQ+OQDXYLpVtZDA wM72mQtqNIzK4dpT5Pb20ayrfiWQq52jxkqeWabjO8NqrvO33ft8O5jsPIeXp+SYF2D/Za/ycki5e jVAJiD7UlQ/GiPv5p/GShAg2ze3fTLn9bgo5QpoxZzzto2cEE7aRjlMpXj01dCqM37hmvBBbdz67x ebv7oR0/5OedkOdFcXFKsXfUuGhCSYLV/7JgQLg88WMIqSsBDP+EunsV0A04ROO9AOmEQSkXjO5Go 4uE6taWA==; Received: from [2001:4bb8:180:e42a:50da:325f:4a06:8830] (helo=localhost) by bombadil.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1opsli-005gy9-Ov; Tue, 01 Nov 2022 15:01:35 +0000 From: Christoph Hellwig To: Jens Axboe , Keith Busch , Sagi Grimberg , Chao Leng Cc: Ming Lei , linux-nvme@lists.infradead.org, linux-block@vger.kernel.org, Hannes Reinecke Subject: [PATCH 13/14] blk-mq: add tagset quiesce interface Date: Tue, 1 Nov 2022 16:00:49 +0100 Message-Id: <20221101150050.3510-14-hch@lst.de> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20221101150050.3510-1-hch@lst.de> References: <20221101150050.3510-1-hch@lst.de> MIME-Version: 1.0 X-SRS-Rewrite: SMTP reverse-path rewritten from by bombadil.infradead.org. See http://www.infradead.org/rpr.html Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org From: Chao Leng Drivers that have shared tagsets may need to quiesce potentially a lot of request queues that all share a single tagset (e.g. nvme). Add an interface to quiesce all the queues on a given tagset. This interface is useful because it can speedup the quiesce by doing it in parallel. Because some queues should not need to be quiesced (e.g. the nvme connect_q) when quiescing the tagset, introduce a QUEUE_FLAG_SKIP_TAGSET_QUIESCE flag to allow this new interface to ski quiescing a particular queue. Signed-off-by: Chao Leng [hch: simplify for the per-tag_set srcu_struct] Signed-off-by: Christoph Hellwig Reviewed-by: Keith Busch Reviewed-by: Sagi Grimberg Reviewed-by: Ming Lei Reviewed-by: Chao Leng Reviewed-by: Hannes Reinecke Reviewed-by: Chaitanya Kulkarni --- block/blk-mq.c | 27 +++++++++++++++++++++++++++ include/linux/blk-mq.h | 2 ++ include/linux/blkdev.h | 3 +++ 3 files changed, 32 insertions(+) diff --git a/block/blk-mq.c b/block/blk-mq.c index b358328a1dce3..f0dd07bbf4c16 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -315,6 +315,33 @@ void blk_mq_unquiesce_queue(struct request_queue *q) } EXPORT_SYMBOL_GPL(blk_mq_unquiesce_queue); +void blk_mq_quiesce_tagset(struct blk_mq_tag_set *set) +{ + struct request_queue *q; + + mutex_lock(&set->tag_list_lock); + list_for_each_entry(q, &set->tag_list, tag_set_list) { + if (!blk_queue_skip_tagset_quiesce(q)) + blk_mq_quiesce_queue_nowait(q); + } + blk_mq_wait_quiesce_done(set); + mutex_unlock(&set->tag_list_lock); +} +EXPORT_SYMBOL_GPL(blk_mq_quiesce_tagset); + +void blk_mq_unquiesce_tagset(struct blk_mq_tag_set *set) +{ + struct request_queue *q; + + mutex_lock(&set->tag_list_lock); + list_for_each_entry(q, &set->tag_list, tag_set_list) { + if (!blk_queue_skip_tagset_quiesce(q)) + blk_mq_unquiesce_queue(q); + } + mutex_unlock(&set->tag_list_lock); +} +EXPORT_SYMBOL_GPL(blk_mq_unquiesce_tagset); + void blk_mq_wake_waiters(struct request_queue *q) { struct blk_mq_hw_ctx *hctx; diff --git a/include/linux/blk-mq.h b/include/linux/blk-mq.h index 061ea6e7af017..109a0e30c4704 100644 --- a/include/linux/blk-mq.h +++ b/include/linux/blk-mq.h @@ -881,6 +881,8 @@ void blk_mq_start_stopped_hw_queue(struct blk_mq_hw_ctx *hctx, bool async); void blk_mq_start_stopped_hw_queues(struct request_queue *q, bool async); void blk_mq_quiesce_queue(struct request_queue *q); void blk_mq_wait_quiesce_done(struct blk_mq_tag_set *set); +void blk_mq_quiesce_tagset(struct blk_mq_tag_set *set); +void blk_mq_unquiesce_tagset(struct blk_mq_tag_set *set); void blk_mq_unquiesce_queue(struct request_queue *q); void blk_mq_delay_run_hw_queue(struct blk_mq_hw_ctx *hctx, unsigned long msecs); void blk_mq_run_hw_queue(struct blk_mq_hw_ctx *hctx, bool async); diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h index 6a6fa167fc828..9188aa3f62595 100644 --- a/include/linux/blkdev.h +++ b/include/linux/blkdev.h @@ -571,6 +571,7 @@ struct request_queue { #define QUEUE_FLAG_HCTX_ACTIVE 28 /* at least one blk-mq hctx is active */ #define QUEUE_FLAG_NOWAIT 29 /* device supports NOWAIT */ #define QUEUE_FLAG_SQ_SCHED 30 /* single queue style io dispatch */ +#define QUEUE_FLAG_SKIP_TAGSET_QUIESCE 31 /* quiesce_tagset skip the queue*/ #define QUEUE_FLAG_MQ_DEFAULT ((1UL << QUEUE_FLAG_IO_STAT) | \ (1UL << QUEUE_FLAG_SAME_COMP) | \ @@ -610,6 +611,8 @@ bool blk_queue_flag_test_and_set(unsigned int flag, struct request_queue *q); #define blk_queue_pm_only(q) atomic_read(&(q)->pm_only) #define blk_queue_registered(q) test_bit(QUEUE_FLAG_REGISTERED, &(q)->queue_flags) #define blk_queue_sq_sched(q) test_bit(QUEUE_FLAG_SQ_SCHED, &(q)->queue_flags) +#define blk_queue_skip_tagset_quiesce(q) \ + test_bit(QUEUE_FLAG_SKIP_TAGSET_QUIESCE, &(q)->queue_flags) extern void blk_set_pm_only(struct request_queue *q); extern void blk_clear_pm_only(struct request_queue *q); From patchwork Tue Nov 1 15:00:50 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christoph Hellwig X-Patchwork-Id: 13027090 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E500EC433FE for ; Tue, 1 Nov 2022 15:09:09 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231189AbiKAPIx (ORCPT ); Tue, 1 Nov 2022 11:08:53 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60688 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231184AbiKAPIf (ORCPT ); Tue, 1 Nov 2022 11:08:35 -0400 Received: from bombadil.infradead.org (bombadil.infradead.org [IPv6:2607:7c80:54:3::133]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2F2DC1C905 for ; Tue, 1 Nov 2022 08:01:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20210309; h=Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender :Reply-To:Content-Type:Content-ID:Content-Description; bh=lGwLgdy//29RWSKEL4CO4hAmaZjxKGL0EJmVog3/l4I=; b=mNPdlA9S28uo0AvK3dq4YoTaJo erblOFZlqvwlcWksjJ6oZBEW88fuvkEH1elu4l2510a6BamRsMmbTLqnqTzOe2NpLuv3ycJwmOfLz H2vbWwQmBbbnBUMbw1xlrJfsvxXJGVpJTkgjNIiBdHwdzx/3B6IVfWIuLKrwUGbLRGLRoUnS1Hjjm pJ/BAKKL5H6ywHIx7KjgUA4bfwuXw/nirj+nchwwxHccxUvWJfxSelY90OM2DbFkxpjVOHWDs4u3A 4bO2CnxIiwYTVlPu69AplCwNAbVcA7ia29Y4krh/W0gFrQSIIEc40dPbch+o8IRGJtf/7fHgxJWFT f6SFabHw==; Received: from [2001:4bb8:180:e42a:50da:325f:4a06:8830] (helo=localhost) by bombadil.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1opsll-005h10-RS; Tue, 01 Nov 2022 15:01:38 +0000 From: Christoph Hellwig To: Jens Axboe , Keith Busch , Sagi Grimberg , Chao Leng Cc: Ming Lei , linux-nvme@lists.infradead.org, linux-block@vger.kernel.org, Hannes Reinecke Subject: [PATCH 14/14] nvme: use blk_mq_[un]quiesce_tagset Date: Tue, 1 Nov 2022 16:00:50 +0100 Message-Id: <20221101150050.3510-15-hch@lst.de> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20221101150050.3510-1-hch@lst.de> References: <20221101150050.3510-1-hch@lst.de> MIME-Version: 1.0 X-SRS-Rewrite: SMTP reverse-path rewritten from by bombadil.infradead.org. See http://www.infradead.org/rpr.html Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org From: Chao Leng All controller namespaces share the same tagset, so we can use this interface which does the optimal operation for parallel quiesce based on the tagset type(e.g. blocking tagsets and non-blocking tagsets). nvme connect_q should not be quiesced when quiesce tagset, so set the QUEUE_FLAG_SKIP_TAGSET_QUIESCE to skip it when init connect_q. Currently we use NVME_NS_STOPPED to ensure pairing quiescing and unquiescing. If use blk_mq_[un]quiesce_tagset, NVME_NS_STOPPED will be invalided, so introduce NVME_CTRL_STOPPED to replace NVME_NS_STOPPED. In addition, we never really quiesce a single namespace. It is a better choice to move the flag from ns to ctrl. Signed-off-by: Chao Leng [hch: rebased on top of prep patches] Signed-off-by: Christoph Hellwig Reviewed-by: Keith Busch Reviewed-by: Chao Leng Reviewed-by: Hannes Reinecke Reviewed-by: Sagi Grimberg Reviewed-by: Chaitanya Kulkarni --- drivers/nvme/host/core.c | 34 ++++++++-------------------------- drivers/nvme/host/nvme.h | 2 +- 2 files changed, 9 insertions(+), 27 deletions(-) diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c index 66b0b6e110024..f94b05c585cbc 100644 --- a/drivers/nvme/host/core.c +++ b/drivers/nvme/host/core.c @@ -4895,6 +4895,8 @@ int nvme_alloc_io_tag_set(struct nvme_ctrl *ctrl, struct blk_mq_tag_set *set, ret = PTR_ERR(ctrl->connect_q); goto out_free_tag_set; } + blk_queue_flag_set(QUEUE_FLAG_SKIP_TAGSET_QUIESCE, + ctrl->connect_q); } ctrl->tagset = set; @@ -5096,20 +5098,6 @@ int nvme_init_ctrl(struct nvme_ctrl *ctrl, struct device *dev, } EXPORT_SYMBOL_GPL(nvme_init_ctrl); -static void nvme_start_ns_queue(struct nvme_ns *ns) -{ - if (test_and_clear_bit(NVME_NS_STOPPED, &ns->flags)) - blk_mq_unquiesce_queue(ns->queue); -} - -static void nvme_stop_ns_queue(struct nvme_ns *ns) -{ - if (!test_and_set_bit(NVME_NS_STOPPED, &ns->flags)) - blk_mq_quiesce_queue(ns->queue); - else - blk_mq_wait_quiesce_done(ns->queue->tag_set); -} - /* let I/O to all namespaces fail in preparation for surprise removal */ void nvme_mark_namespaces_dead(struct nvme_ctrl *ctrl) { @@ -5172,23 +5160,17 @@ EXPORT_SYMBOL_GPL(nvme_start_freeze); void nvme_stop_queues(struct nvme_ctrl *ctrl) { - struct nvme_ns *ns; - - down_read(&ctrl->namespaces_rwsem); - list_for_each_entry(ns, &ctrl->namespaces, list) - nvme_stop_ns_queue(ns); - up_read(&ctrl->namespaces_rwsem); + if (!test_and_set_bit(NVME_CTRL_STOPPED, &ctrl->flags)) + blk_mq_quiesce_tagset(ctrl->tagset); + else + blk_mq_wait_quiesce_done(ctrl->tagset); } EXPORT_SYMBOL_GPL(nvme_stop_queues); void nvme_start_queues(struct nvme_ctrl *ctrl) { - struct nvme_ns *ns; - - down_read(&ctrl->namespaces_rwsem); - list_for_each_entry(ns, &ctrl->namespaces, list) - nvme_start_ns_queue(ns); - up_read(&ctrl->namespaces_rwsem); + if (test_and_clear_bit(NVME_CTRL_STOPPED, &ctrl->flags)) + blk_mq_unquiesce_tagset(ctrl->tagset); } EXPORT_SYMBOL_GPL(nvme_start_queues); diff --git a/drivers/nvme/host/nvme.h b/drivers/nvme/host/nvme.h index 1cd7168002438..f9df10653f3c5 100644 --- a/drivers/nvme/host/nvme.h +++ b/drivers/nvme/host/nvme.h @@ -237,6 +237,7 @@ enum nvme_ctrl_flags { NVME_CTRL_FAILFAST_EXPIRED = 0, NVME_CTRL_ADMIN_Q_STOPPED = 1, NVME_CTRL_STARTED_ONCE = 2, + NVME_CTRL_STOPPED = 3, }; struct nvme_ctrl { @@ -486,7 +487,6 @@ struct nvme_ns { #define NVME_NS_ANA_PENDING 2 #define NVME_NS_FORCE_RO 3 #define NVME_NS_READY 4 -#define NVME_NS_STOPPED 5 struct cdev cdev; struct device cdev_device;