From patchwork Mon Oct 31 13:53:18 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Laurence Oberman X-Patchwork-Id: 9405491 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 551EC60588 for ; Mon, 31 Oct 2016 13:54:13 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6082E2920B for ; Mon, 31 Oct 2016 13:54:13 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 557DA29246; Mon, 31 Oct 2016 13:54:13 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.2 required=2.0 tests=BAYES_00, RCVD_IN_DNSWL_HI, URIBL_BLACK autolearn=unavailable version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id CD630292A2 for ; Mon, 31 Oct 2016 13:54:12 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S943231AbcJaNyK convert rfc822-to-8bit (ORCPT ); Mon, 31 Oct 2016 09:54:10 -0400 Received: from mx3-phx2.redhat.com ([209.132.183.24]:52539 "EHLO mx3-phx2.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S941468AbcJaNyJ (ORCPT ); Mon, 31 Oct 2016 09:54:09 -0400 Received: from zmail22.collab.prod.int.phx2.redhat.com (zmail22.collab.prod.int.phx2.redhat.com [10.5.83.26]) by mx3-phx2.redhat.com (8.13.8/8.13.8) with ESMTP id u9VDrIxa020597; Mon, 31 Oct 2016 09:53:18 -0400 Date: Mon, 31 Oct 2016 09:53:18 -0400 (EDT) From: Laurence Oberman To: Bart Van Assche Cc: Jens Axboe , Christoph Hellwig , James Bottomley , "Martin K. Petersen" , Mike Snitzer , Doug Ledford , Keith Busch , Ming Lei , Konrad Rzeszutek Wilk , Roger Pau =?utf-8?Q?Monn=C3=A9?= , linux-block@vger.kernel.org, linux-scsi@vger.kernel.org, linux-rdma@vger.kernel.org, linux-nvme@lists.infradead.org Message-ID: <540193784.5466628.1477921998345.JavaMail.zimbra@redhat.com> In-Reply-To: References: <7460e8b2-2cfd-c0d5-7ae7-7f662d89dad3@sandisk.com> Subject: Re: [PATCH v5 14/14] nvme: Use BLK_MQ_S_STOPPED instead of QUEUE_FLAG_STOPPED in blk-mq code MIME-Version: 1.0 X-Originating-IP: [10.3.239.35] X-Mailer: Zimbra 8.0.6_GA_5922 (ZimbraWebClient - FF45 (Linux)/8.0.6_GA_5922) Thread-Topic: nvme: Use BLK_MQ_S_STOPPED instead of QUEUE_FLAG_STOPPED in blk-mq code Thread-Index: I4NeSjDoGA+05c4LhblUp2fHnBhTww== Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP ----- Original Message ----- > From: "Bart Van Assche" > To: "Jens Axboe" > Cc: "Christoph Hellwig" , "James Bottomley" , "Martin K. Petersen" > , "Mike Snitzer" , "Doug Ledford" , "Keith > Busch" , "Ming Lei" , "Konrad Rzeszutek Wilk" > , "Roger Pau Monné" , "Laurence Oberman" , > linux-block@vger.kernel.org, linux-scsi@vger.kernel.org, linux-rdma@vger.kernel.org, linux-nvme@lists.infradead.org > Sent: Friday, October 28, 2016 8:23:40 PM > Subject: [PATCH v5 14/14] nvme: Use BLK_MQ_S_STOPPED instead of QUEUE_FLAG_STOPPED in blk-mq code > > Make nvme_requeue_req() check BLK_MQ_S_STOPPED instead of > QUEUE_FLAG_STOPPED. Remove the QUEUE_FLAG_STOPPED manipulations > that became superfluous because of this change. Change > blk_queue_stopped() tests into blk_mq_queue_stopped(). > > This patch fixes a race condition: using queue_flag_clear_unlocked() > is not safe if any other function that manipulates the queue flags > can be called concurrently, e.g. blk_cleanup_queue(). > > Signed-off-by: Bart Van Assche > Cc: Keith Busch > Cc: Christoph Hellwig > Cc: Sagi Grimberg > --- > drivers/nvme/host/core.c | 16 ++-------------- > 1 file changed, 2 insertions(+), 14 deletions(-) > > diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c > index fe15d94..45dd237 100644 > --- a/drivers/nvme/host/core.c > +++ b/drivers/nvme/host/core.c > @@ -201,13 +201,7 @@ static struct nvme_ns *nvme_get_ns_from_disk(struct > gendisk *disk) > > void nvme_requeue_req(struct request *req) > { > - unsigned long flags; > - > - blk_mq_requeue_request(req, false); > - spin_lock_irqsave(req->q->queue_lock, flags); > - if (!blk_queue_stopped(req->q)) > - blk_mq_kick_requeue_list(req->q); > - spin_unlock_irqrestore(req->q->queue_lock, flags); > + blk_mq_requeue_request(req, !blk_mq_queue_stopped(req->q)); > } > EXPORT_SYMBOL_GPL(nvme_requeue_req); > > @@ -2078,13 +2072,8 @@ void nvme_stop_queues(struct nvme_ctrl *ctrl) > struct nvme_ns *ns; > > mutex_lock(&ctrl->namespaces_mutex); > - list_for_each_entry(ns, &ctrl->namespaces, list) { > - spin_lock_irq(ns->queue->queue_lock); > - queue_flag_set(QUEUE_FLAG_STOPPED, ns->queue); > - spin_unlock_irq(ns->queue->queue_lock); > - > + list_for_each_entry(ns, &ctrl->namespaces, list) > blk_mq_quiesce_queue(ns->queue); > - } > mutex_unlock(&ctrl->namespaces_mutex); > } > EXPORT_SYMBOL_GPL(nvme_stop_queues); > @@ -2095,7 +2084,6 @@ void nvme_start_queues(struct nvme_ctrl *ctrl) > > mutex_lock(&ctrl->namespaces_mutex); > list_for_each_entry(ns, &ctrl->namespaces, list) { > - queue_flag_clear_unlocked(QUEUE_FLAG_STOPPED, ns->queue); > blk_mq_start_stopped_hw_queues(ns->queue, true); > blk_mq_kick_requeue_list(ns->queue); > } > -- > 2.10.1 > > -- > To unsubscribe from this list: send the line "unsubscribe linux-rdma" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > Hello Bart Thanks for all this work. Applied all 14 patches, also corrected the part of the xen-blkfront.c blkif_recover patch in patchv5-5/14. Ran multiple read/write buffered and directio tests via RDMA/SRP and mlx5 (100Gbit) with max_sectors_kb set to 1024, 2048, 4096 and 8196 Ran multiple read/write buffered and directio tests via RDMA/SRP and mlx4 (56Gbit) with max_sectors_kb set to 1024, 2048, 4096 and 8196 Reset the SRP hosts multiple times with multipath set to no_path_retry queue Ran basic NVME read/write testing with no hot plug disconnects on multiple block sizes All tests passed. For the series: Tested-by: Laurence Oberman --- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html diff --git a/drivers/block/xen-blkfront.c b/drivers/block/xen-blkfront.c index 9908597..60fff99 100644 --- a/drivers/block/xen-blkfront.c +++ b/drivers/block/xen-blkfront.c @@ -2045,6 +2045,7 @@ static int blkif_recover(struct blkfront_info *info) BUG_ON(req->nr_phys_segments > segs); blk_mq_requeue_request(req); } + blk_mq_start_stopped_hw_queues(infrq, true); *** Corrected blk_mq_kick_requeue_list(infrq); while ((bio = bio_list_pop(&infbio_list)) != NULL) {