From patchwork Mon Nov 26 16:35:49 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 10698617 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 52BB713BB for ; Mon, 26 Nov 2018 16:36:06 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 4273E29735 for ; Mon, 26 Nov 2018 16:36:06 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 3650129B2B; Mon, 26 Nov 2018 16:36:06 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7EFF029735 for ; Mon, 26 Nov 2018 16:36:05 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726328AbeK0Dal (ORCPT ); Mon, 26 Nov 2018 22:30:41 -0500 Received: from mail-it1-f196.google.com ([209.85.166.196]:35056 "EHLO mail-it1-f196.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726253AbeK0Dal (ORCPT ); Mon, 26 Nov 2018 22:30:41 -0500 Received: by mail-it1-f196.google.com with SMTP id p197so3164058itp.0 for ; Mon, 26 Nov 2018 08:36:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=wY8wSFZwGvj9JmcMtDWhg1FyLukBykQhyIYxMSU20DI=; b=lZrf3BOaNHJV7dFivtB2fbcZnUugPOpuIyPxGXAskJSl2WC3mXExzl2+g5bui7k4q5 tDZ1xjy3RsZdZkv0oi3Y1LHGGWQA3CZoyBtx9A+gYfjjrVQ/eNGsr9qPwyp6jIyKQsXT PO+T9l6CIBJCHXt9CSIWhsPQCWTDZCRPSkeN5kNqxikLvB/bS9P5Bb5bDZTTozR0YBaD ubNvMY4RysUEJM0cfE3O+PJpUkpn88NtdVrZuzT76LCY1I8Y/avUGmFEdSehguGVscTc f7xLRjDpZ9Sb0p9mTmmaaS9w6EYJVFPOoeYQq74kge2WjKH3R9LiCOECqsdcX1xv0lHf 7n6A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=wY8wSFZwGvj9JmcMtDWhg1FyLukBykQhyIYxMSU20DI=; b=RASLHOZD8Tywt3ESIdnQrJLvQi5S8xozpeDNMJMz47iIptQXulTCnXeCQRu8z+j3lG 1aOYyH9SF8BHNXRrdsRk/a0VXt2u3uXcUVoNYZ+gJp0GMj9EUqX1zBlNdYAcP93hvfqw RKHLsUCZi/2jwHyNSMZjtb9fIQUaL3IFjWuyDT0QC6lTlqkVKgceSUhdQ84fDAB0CZHy uWx1Mr/yULwGO0EM1M7REbx1jT44ZDYHRtnn2wC+ghvyE8v9AjW56ttDp7amxveb9c01 S++r6DRdGQN3Ggz0fTp5VWw7x2CTS22bX7ATytIAKEvVMkEWlVR02LhacngQN/xSRsd7 a+4g== X-Gm-Message-State: AGRZ1gKxgkkxS5sfdaMtULBLj5kS+tdNsxEjA8fzNVyZ0epzB7Jb2BKb vhboF4B/umB3v/ZjYfYyYwgB+aZTaJE= X-Google-Smtp-Source: AFSGD/XkjBYC70cM+UHR1//zgjPb2WTo+2N9Nzntcow/ICB12kXO6f/VUugJuXIbBuY36xgwUxRbng== X-Received: by 2002:a24:fdc4:: with SMTP id m187mr22908628ith.75.1543250162894; Mon, 26 Nov 2018 08:36:02 -0800 (PST) Received: from localhost.localdomain ([216.160.245.98]) by smtp.gmail.com with ESMTPSA id r11sm263717iog.46.2018.11.26.08.36.00 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 26 Nov 2018 08:36:01 -0800 (PST) From: Jens Axboe To: linux-block@vger.kernel.org, linux-nvme@lists.infradead.org Cc: Jens Axboe Subject: [PATCH 1/8] block: sum requests in the plug structure Date: Mon, 26 Nov 2018 09:35:49 -0700 Message-Id: <20181126163556.5181-2-axboe@kernel.dk> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181126163556.5181-1-axboe@kernel.dk> References: <20181126163556.5181-1-axboe@kernel.dk> Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP This isn't exactly the same as the previous count, as it includes requests for all devices. But that really doesn't matter, if we have more than the threshold (16) queued up, flush it. It's not worth it to have an expensive list loop for this. Signed-off-by: Jens Axboe Reviewed-by: Christoph Hellwig --- block/blk-core.c | 30 ++++-------------------------- block/blk-mq.c | 16 +++++----------- block/blk.h | 2 -- include/linux/blkdev.h | 1 + 4 files changed, 10 insertions(+), 39 deletions(-) diff --git a/block/blk-core.c b/block/blk-core.c index 9af56dbb84f1..be9233400314 100644 --- a/block/blk-core.c +++ b/block/blk-core.c @@ -736,7 +736,6 @@ bool bio_attempt_discard_merge(struct request_queue *q, struct request *req, * Caller must ensure !blk_queue_nomerges(q) beforehand. */ bool blk_attempt_plug_merge(struct request_queue *q, struct bio *bio, - unsigned int *request_count, struct request **same_queue_rq) { struct blk_plug *plug; @@ -746,22 +745,19 @@ bool blk_attempt_plug_merge(struct request_queue *q, struct bio *bio, plug = current->plug; if (!plug) return false; - *request_count = 0; plug_list = &plug->mq_list; list_for_each_entry_reverse(rq, plug_list, queuelist) { bool merged = false; - if (rq->q == q) { - (*request_count)++; + if (rq->q == q && same_queue_rq) { /* * Only blk-mq multiple hardware queues case checks the * rq in the same queue, there should be only one such * rq in a queue **/ - if (same_queue_rq) - *same_queue_rq = rq; + *same_queue_rq = rq; } if (rq->q != q || !blk_rq_merge_ok(rq, bio)) @@ -788,26 +784,6 @@ bool blk_attempt_plug_merge(struct request_queue *q, struct bio *bio, return false; } -unsigned int blk_plug_queued_count(struct request_queue *q) -{ - struct blk_plug *plug; - struct request *rq; - struct list_head *plug_list; - unsigned int ret = 0; - - plug = current->plug; - if (!plug) - goto out; - - plug_list = &plug->mq_list; - list_for_each_entry(rq, plug_list, queuelist) { - if (rq->q == q) - ret++; - } -out: - return ret; -} - void blk_init_request_from_bio(struct request *req, struct bio *bio) { if (bio->bi_opf & REQ_RAHEAD) @@ -1803,6 +1779,8 @@ void blk_start_plug(struct blk_plug *plug) INIT_LIST_HEAD(&plug->mq_list); INIT_LIST_HEAD(&plug->cb_list); + plug->rq_count = 0; + /* * Store ordering should not be needed here, since a potential * preempt will imply a full memory barrier diff --git a/block/blk-mq.c b/block/blk-mq.c index 37674c1766a7..99c66823d52f 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -1676,6 +1676,7 @@ void blk_mq_flush_plug_list(struct blk_plug *plug, bool from_schedule) unsigned int depth; list_splice_init(&plug->mq_list, &list); + plug->rq_count = 0; list_sort(NULL, &list, plug_rq_cmp); @@ -1872,7 +1873,6 @@ static blk_qc_t blk_mq_make_request(struct request_queue *q, struct bio *bio) const int is_flush_fua = op_is_flush(bio->bi_opf); struct blk_mq_alloc_data data = { .flags = 0, .cmd_flags = bio->bi_opf }; struct request *rq; - unsigned int request_count = 0; struct blk_plug *plug; struct request *same_queue_rq = NULL; blk_qc_t cookie; @@ -1885,7 +1885,7 @@ static blk_qc_t blk_mq_make_request(struct request_queue *q, struct bio *bio) return BLK_QC_T_NONE; if (!is_flush_fua && !blk_queue_nomerges(q) && - blk_attempt_plug_merge(q, bio, &request_count, &same_queue_rq)) + blk_attempt_plug_merge(q, bio, &same_queue_rq)) return BLK_QC_T_NONE; if (blk_mq_sched_bio_merge(q, bio)) @@ -1916,20 +1916,12 @@ static blk_qc_t blk_mq_make_request(struct request_queue *q, struct bio *bio) blk_insert_flush(rq); blk_mq_run_hw_queue(data.hctx, true); } else if (plug && q->nr_hw_queues == 1) { + unsigned int request_count = plug->rq_count; struct request *last = NULL; blk_mq_put_ctx(data.ctx); blk_mq_bio_to_request(rq, bio); - /* - * @request_count may become stale because of schedule - * out, so check the list again. - */ - if (list_empty(&plug->mq_list)) - request_count = 0; - else if (blk_queue_nomerges(q)) - request_count = blk_plug_queued_count(q); - if (!request_count) trace_block_plug(q); else @@ -1942,6 +1934,7 @@ static blk_qc_t blk_mq_make_request(struct request_queue *q, struct bio *bio) } list_add_tail(&rq->queuelist, &plug->mq_list); + plug->rq_count++; } else if (plug && !blk_queue_nomerges(q)) { blk_mq_bio_to_request(rq, bio); @@ -1957,6 +1950,7 @@ static blk_qc_t blk_mq_make_request(struct request_queue *q, struct bio *bio) if (same_queue_rq) list_del_init(&same_queue_rq->queuelist); list_add_tail(&rq->queuelist, &plug->mq_list); + plug->rq_count++; blk_mq_put_ctx(data.ctx); diff --git a/block/blk.h b/block/blk.h index 610948157a5b..848278c52030 100644 --- a/block/blk.h +++ b/block/blk.h @@ -161,9 +161,7 @@ bool bio_attempt_back_merge(struct request_queue *q, struct request *req, bool bio_attempt_discard_merge(struct request_queue *q, struct request *req, struct bio *bio); bool blk_attempt_plug_merge(struct request_queue *q, struct bio *bio, - unsigned int *request_count, struct request **same_queue_rq); -unsigned int blk_plug_queued_count(struct request_queue *q); void blk_account_io_start(struct request *req, bool new_io); void blk_account_io_completion(struct request *req, unsigned int bytes); diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h index e3c0a8ec16a7..02732cae6080 100644 --- a/include/linux/blkdev.h +++ b/include/linux/blkdev.h @@ -1130,6 +1130,7 @@ extern void blk_set_queue_dying(struct request_queue *); struct blk_plug { struct list_head mq_list; /* blk-mq requests */ struct list_head cb_list; /* md requires an unplug callback */ + unsigned short rq_count; }; #define BLK_MAX_REQUEST_COUNT 16 #define BLK_PLUG_FLUSH_SIZE (128 * 1024) From patchwork Mon Nov 26 16:35:50 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 10698619 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id AA2F413BB for ; Mon, 26 Nov 2018 16:36:07 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9A9D129735 for ; Mon, 26 Nov 2018 16:36:07 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 8EA8829B2B; Mon, 26 Nov 2018 16:36:07 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 14B7229735 for ; Mon, 26 Nov 2018 16:36:07 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726412AbeK0Dan (ORCPT ); Mon, 26 Nov 2018 22:30:43 -0500 Received: from mail-io1-f68.google.com ([209.85.166.68]:34219 "EHLO mail-io1-f68.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726253AbeK0Dam (ORCPT ); Mon, 26 Nov 2018 22:30:42 -0500 Received: by mail-io1-f68.google.com with SMTP id f6so14453176iob.1 for ; Mon, 26 Nov 2018 08:36:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=XXvNiHPYXxX5XgSQnfFMNCnnoUcJikRAPxMVajab6f8=; b=T8xXpaYGpsx+hN5FjhoUjPJR5bpLgtCjq5JfGSWiWbEvxlvfUx3jCeCTqPSyxdmpmE L8ciU7325zjScr2g12Ek5ckh18vqnIFM/on6GtbmzScvMU4rUqHpg1CnioxJyPvancwT tz+MkqSUPo97xk7hDMyaiUYRnokXug2xgzyVabcatwplYWW1B4/vs9rukAs9c+RxEjlO cfRU7lPOdHc6KIw73WLEov4MOA5XfvUYkXY+rMw9tQdRXGy5i7u+HBMSPH6FFoMF+nEp DCPfio62/kpjHDX2d4bzWZyz7MbxBAgOeAmevytHd0yUxXrY3fDjyykZqMai7CKa+m4x m7rg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=XXvNiHPYXxX5XgSQnfFMNCnnoUcJikRAPxMVajab6f8=; b=bSo77lx7ZVa7F2blwaC1dkz/GEnlZzlk2RmG9tf7Zi+VbBk0Md3f40LmEBmVr4yQgB mDK7AUCNhDu69n2oWcApxRC65wja0JoU3y8JDNk/0qwYbcSkjw8VlwleK8y62x8VtbII jfyItpXLKAzYLZzCgElHva16Pz5SRc6WwiFOTPsGPqJbm9G6IO9x1wWE0iQ/QERZWBkj L/fATj33yRT8fA1n1+CKtkxLFd0CJkm5sfUpkjh0mgcI8NXu5/IreQtKEkM48Sbw7sny rYoo7HP2UAgga1HUitTkJb0kDN4xoxSY+UW9kpYgZWMXjE+0Dn5+UayQxuxot7N33j4W CiPQ== X-Gm-Message-State: AA+aEWY+FfmrYe0aTeMFs3RDYe7a6ob7IPyDTLpvdy3IbC1SMfDxWCfv A30sTg6xYutbq/qIgMGB6XUTlCaMw9s= X-Google-Smtp-Source: AFSGD/XbUkapnAgNL5HAZiSvhTJ8m/NT2GBdkNlSJvNQhKjHIQSrp9lgXIlVY62Esopjm5fvPe+idA== X-Received: by 2002:a6b:fe13:: with SMTP id x19mr20651748ioh.294.1543250164682; Mon, 26 Nov 2018 08:36:04 -0800 (PST) Received: from localhost.localdomain ([216.160.245.98]) by smtp.gmail.com with ESMTPSA id r11sm263717iog.46.2018.11.26.08.36.02 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 26 Nov 2018 08:36:03 -0800 (PST) From: Jens Axboe To: linux-block@vger.kernel.org, linux-nvme@lists.infradead.org Cc: Jens Axboe Subject: [PATCH 2/8] block: improve logic around when to sort a plug list Date: Mon, 26 Nov 2018 09:35:50 -0700 Message-Id: <20181126163556.5181-3-axboe@kernel.dk> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181126163556.5181-1-axboe@kernel.dk> References: <20181126163556.5181-1-axboe@kernel.dk> Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Do it for the nr_hw_queues == 1 case, but only do it for the multi queue case if we have requests for multiple devices in the plug. Signed-off-by: Jens Axboe --- block/blk-core.c | 1 + block/blk-mq.c | 7 +++++-- include/linux/blkdev.h | 1 + 3 files changed, 7 insertions(+), 2 deletions(-) diff --git a/block/blk-core.c b/block/blk-core.c index be9233400314..c9758d185357 100644 --- a/block/blk-core.c +++ b/block/blk-core.c @@ -1780,6 +1780,7 @@ void blk_start_plug(struct blk_plug *plug) INIT_LIST_HEAD(&plug->mq_list); INIT_LIST_HEAD(&plug->cb_list); plug->rq_count = 0; + plug->do_sort = false; /* * Store ordering should not be needed here, since a potential diff --git a/block/blk-mq.c b/block/blk-mq.c index 99c66823d52f..6a249bf6ed00 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -1678,7 +1678,8 @@ void blk_mq_flush_plug_list(struct blk_plug *plug, bool from_schedule) list_splice_init(&plug->mq_list, &list); plug->rq_count = 0; - list_sort(NULL, &list, plug_rq_cmp); + if (plug->do_sort) + list_sort(NULL, &list, plug_rq_cmp); this_q = NULL; this_hctx = NULL; @@ -1935,6 +1936,7 @@ static blk_qc_t blk_mq_make_request(struct request_queue *q, struct bio *bio) list_add_tail(&rq->queuelist, &plug->mq_list); plug->rq_count++; + plug->do_sort = true; } else if (plug && !blk_queue_nomerges(q)) { blk_mq_bio_to_request(rq, bio); @@ -1958,7 +1960,8 @@ static blk_qc_t blk_mq_make_request(struct request_queue *q, struct bio *bio) data.hctx = same_queue_rq->mq_hctx; blk_mq_try_issue_directly(data.hctx, same_queue_rq, &cookie); - } + } else if (plug->rq_count > 1) + plug->do_sort = true; } else if ((q->nr_hw_queues > 1 && is_sync) || (!q->elevator && !data.hctx->dispatch_busy)) { blk_mq_put_ctx(data.ctx); diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h index 02732cae6080..b6d0b33feee7 100644 --- a/include/linux/blkdev.h +++ b/include/linux/blkdev.h @@ -1131,6 +1131,7 @@ struct blk_plug { struct list_head mq_list; /* blk-mq requests */ struct list_head cb_list; /* md requires an unplug callback */ unsigned short rq_count; + bool do_sort; }; #define BLK_MAX_REQUEST_COUNT 16 #define BLK_PLUG_FLUSH_SIZE (128 * 1024) From patchwork Mon Nov 26 16:35:51 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 10698621 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 154DC13BF for ; Mon, 26 Nov 2018 16:36:10 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 04E6E29735 for ; Mon, 26 Nov 2018 16:36:10 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id ED6AD29B2B; Mon, 26 Nov 2018 16:36:09 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 72FD229735 for ; Mon, 26 Nov 2018 16:36:09 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726340AbeK0Dap (ORCPT ); Mon, 26 Nov 2018 22:30:45 -0500 Received: from mail-it1-f194.google.com ([209.85.166.194]:35891 "EHLO mail-it1-f194.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726253AbeK0Dap (ORCPT ); Mon, 26 Nov 2018 22:30:45 -0500 Received: by mail-it1-f194.google.com with SMTP id c9so28699308itj.1 for ; Mon, 26 Nov 2018 08:36:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=NLWOXWk+blJHoTtHNBJpfBgQguikfitnIRG4gEI4XWw=; b=c5Si/kEKF/yRyk2CG3Eu0e+qLMV/MgLni0l5CEIqIiQnu/vB90IVjpDBBw7rwviK4c EnET2LUkl+/cmF2O4NmnjjmN1+MXNr8mnFXdgMlCwDREb0Db2DyJ7Jlc3TTp6xtDEurt hdRMcSs+QkAd9t83CWP5cAwtfY6kBFE+NOBFyWyjzkoaEJLDkb+GQsE1F9ZpywftwPDo VlsT+TNKGUF7u5PcvSyytlVjw+Cr4s/wL59cxCKwjzJCi0x0oDn6oBkRAdx0r+1JySji V9y1KMmTtToqxM0EMurUflMLyU66iitwZx2bB6AfRNcvR1FKzGknMQgyOkqiDPpo6dNX Z4Bg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=NLWOXWk+blJHoTtHNBJpfBgQguikfitnIRG4gEI4XWw=; b=NmyZ/zSRzeISdAgoMvzaBWh40XuntDGY1iP/bKHJ4KwpEeo1V0xhQDPfz1y2o3MBlw cAoM4l4z1hcVQ7W3XbygL2vp+8MuIQQccLWecf0oW4+wzvaLpEmVt6zKsQ/2TlDrv4qI DHkoBEPQo0Dm7SQsqitWptG8InxeFo9aRm/1t8EkIZTFTJJWEe7n3NLb3ocyG1bnF32n ewE+psA5rDYJSYLftvt9+fhI2wVjSLB2pnQrMLgqbQBI+E4V7tYjFzasYknNdY5ctFc9 ks7y47jtgSwXc2ELtpMllx+v34c6qXd7B5HBLRa5S5L5ztkz4+t+am+W8EGY5K6sFNZq yvoA== X-Gm-Message-State: AA+aEWYEtURTY5Ft9/IDN1oIc3moKQH9H/inlZA7rn7a7RrDrQSvgd4y YJfsShdnINX+zwHuH87sBVLo/Mk+cMg= X-Google-Smtp-Source: AJdET5fF+XzT2t210RKA1bZC/Q71WxmtBlJ2JIQ/Euf3r2IyP0NFnH801RCZqsYcEjBGSpLZLNACRA== X-Received: by 2002:a02:93c2:: with SMTP id z60mr23380443jah.51.1543250166950; Mon, 26 Nov 2018 08:36:06 -0800 (PST) Received: from localhost.localdomain ([216.160.245.98]) by smtp.gmail.com with ESMTPSA id r11sm263717iog.46.2018.11.26.08.36.04 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 26 Nov 2018 08:36:05 -0800 (PST) From: Jens Axboe To: linux-block@vger.kernel.org, linux-nvme@lists.infradead.org Cc: Jens Axboe Subject: [PATCH 3/8] blk-mq: add mq_ops->commit_rqs() Date: Mon, 26 Nov 2018 09:35:51 -0700 Message-Id: <20181126163556.5181-4-axboe@kernel.dk> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181126163556.5181-1-axboe@kernel.dk> References: <20181126163556.5181-1-axboe@kernel.dk> Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP blk-mq passes information to the hardware about any given request being the last that we will issue in this sequence. The point is that hardware can defer costly doorbell type writes to the last request. But if we run into errors issuing a sequence of requests, we may never send the request with bd->last == true set. For that case, we need a hook that tells the hardware that nothing else is coming right now. For failures returned by the drivers ->queue_rq() hook, the driver is responsible for flushing pending requests, if it uses bd->last to optimize that part. This works like before, no changes there. Signed-off-by: Jens Axboe Reviewed-by: Omar Sandoval Reviewed-by: Ming Lei Reviewed-by: Christoph Hellwig --- include/linux/blk-mq.h | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/include/linux/blk-mq.h b/include/linux/blk-mq.h index ca0520ca6437..1fd139b65a6e 100644 --- a/include/linux/blk-mq.h +++ b/include/linux/blk-mq.h @@ -117,6 +117,7 @@ struct blk_mq_queue_data { typedef blk_status_t (queue_rq_fn)(struct blk_mq_hw_ctx *, const struct blk_mq_queue_data *); +typedef void (commit_rqs_fn)(struct blk_mq_hw_ctx *); /* takes rq->cmd_flags as input, returns a hardware type index */ typedef int (rq_flags_to_type_fn)(struct request_queue *, unsigned int); typedef bool (get_budget_fn)(struct blk_mq_hw_ctx *); @@ -144,6 +145,15 @@ struct blk_mq_ops { */ queue_rq_fn *queue_rq; + /* + * If a driver uses bd->last to judge when to submit requests to + * hardware, it must define this function. In case of errors that + * make us stop issuing further requests, this hook serves the + * purpose of kicking the hardware (which the last request otherwise + * would have done). + */ + commit_rqs_fn *commit_rqs; + /* * Return a queue map type for the given request/bio flags */ From patchwork Mon Nov 26 16:35:52 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 10698623 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 81B301869 for ; Mon, 26 Nov 2018 16:36:12 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 72C3629735 for ; Mon, 26 Nov 2018 16:36:12 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 670E52992C; Mon, 26 Nov 2018 16:36:12 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id CAE7629B2B for ; Mon, 26 Nov 2018 16:36:11 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726167AbeK0Dar (ORCPT ); Mon, 26 Nov 2018 22:30:47 -0500 Received: from mail-io1-f67.google.com ([209.85.166.67]:39034 "EHLO mail-io1-f67.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726253AbeK0Dar (ORCPT ); Mon, 26 Nov 2018 22:30:47 -0500 Received: by mail-io1-f67.google.com with SMTP id k7so4574637iob.6 for ; Mon, 26 Nov 2018 08:36:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=dQNpMrnHiA8Sq6l5G1VGOnfunRBbHhhFUOFsxMZRY6o=; b=DwKeFjZAG8fw1993XjAg0QLcyl+9+CVEyCM0k5OSMU9z4luAQ175CJFmH4+3Kd/wE/ LgDQQMfHZQdT2dzue0Zzi0Itxp9cAXi8Wnc09x/vo+9sNEDmJVCm2wdkiuk9R8xy6Zzd uCBK7lW2oZ37knZc3P/3ouzTMalIa8f2hcuPGQXsAxPjdb5Me6DK3DapTUNFBLeUxQp4 fOprsVcmWP0moBpHGcHd/sMMem/8/epTFK0rkWnhbnJFvL2q/93qPbU9SXMkuE1Ka55w lBmQyTvSjEXy7ZZ4h12kwUr2ziwVOCY+R91Vl244auZrTH1gwnLrfDu/ONC2FBUj+6zP /GdA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=dQNpMrnHiA8Sq6l5G1VGOnfunRBbHhhFUOFsxMZRY6o=; b=nfhpj5N2OUCLpQ/K7BzbRetq7y+LNPkD9ZKudbsm9Rbi2bCoJNia+Xo9ZM4s2zG886 J2U8+6X+4rcrBxDdj+pEn0jKRr2i+HSdh26GjZ5FGD+hSe9j3Hu9qtngo2Ig7ImPdcIK h/jUu54KTK4vL2uGDKUgRokKcmFPWbyK3Z6NC5gE7O8QRm7HHLZW1khKUL1ofFGRm0EV ceInDTKPDTx4fjyqwMpk/MjJTpvnCL1qR2YV1dAoXBc1Wb09KMs5SXJ6enoT7M9qqrXI yZd7s6b0RSs1N/jpZkyHvyryKFqqbTKfjLSC5zJh72v3Huzqd4Hq+48YJPZS5Eq8cBRY eW9Q== X-Gm-Message-State: AA+aEWamGT5K3o7AJilN2sAkEF858mFQnbaVq8fXcCjxLQnzhT8yH2HA 2yuu/1a4kyjYnap1elmrr6bOaTtoHug= X-Google-Smtp-Source: AFSGD/WyesAa5saWos3qvLjDmOhwwkDw32e2g6Szojfa1CsYC9l0r6J34H9EP73aA0obu7dRYIBk3w== X-Received: by 2002:a6b:14c6:: with SMTP id 189mr21191603iou.179.1543250168835; Mon, 26 Nov 2018 08:36:08 -0800 (PST) Received: from localhost.localdomain ([216.160.245.98]) by smtp.gmail.com with ESMTPSA id r11sm263717iog.46.2018.11.26.08.36.07 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 26 Nov 2018 08:36:07 -0800 (PST) From: Jens Axboe To: linux-block@vger.kernel.org, linux-nvme@lists.infradead.org Cc: Jens Axboe Subject: [PATCH 4/8] nvme: implement mq_ops->commit_rqs() hook Date: Mon, 26 Nov 2018 09:35:52 -0700 Message-Id: <20181126163556.5181-5-axboe@kernel.dk> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181126163556.5181-1-axboe@kernel.dk> References: <20181126163556.5181-1-axboe@kernel.dk> Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Split the command submission and the SQ doorbell ring, and add the doorbell ring as our ->commit_rqs() hook. This allows a list of requests to be issued, with nvme only writing the SQ update when it's necessary. This is more efficient if we have lists of requests to issue, particularly on virtualized hardware, where writing the SQ doorbell is more expensive than on real hardware. For those cases, performance increases of 2-3x have been observed. The use case for this is plugged IO, where blk-mq flushes a batch of requests at the time. Signed-off-by: Jens Axboe --- drivers/nvme/host/pci.c | 52 +++++++++++++++++++++++++++++++++-------- 1 file changed, 42 insertions(+), 10 deletions(-) diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c index 73effe586e5f..d503bf6cd8ba 100644 --- a/drivers/nvme/host/pci.c +++ b/drivers/nvme/host/pci.c @@ -203,6 +203,7 @@ struct nvme_queue { u16 q_depth; s16 cq_vector; u16 sq_tail; + u16 last_sq_tail; u16 cq_head; u16 last_cq_head; u16 qid; @@ -522,22 +523,52 @@ static int nvme_pci_map_queues(struct blk_mq_tag_set *set) return 0; } +static inline void nvme_write_sq_db(struct nvme_queue *nvmeq) +{ + if (nvme_dbbuf_update_and_check_event(nvmeq->sq_tail, + nvmeq->dbbuf_sq_db, nvmeq->dbbuf_sq_ei)) + writel(nvmeq->sq_tail, nvmeq->q_db); + nvmeq->last_sq_tail = nvmeq->sq_tail; +} + +static inline int nvme_next_ring_index(struct nvme_queue *nvmeq, u16 index) +{ + if (++index == nvmeq->q_depth) + return 0; + + return index; +} + /** * nvme_submit_cmd() - Copy a command into a queue and ring the doorbell * @nvmeq: The queue to use * @cmd: The command to send + * @write_sq: whether to write to the SQ doorbell */ -static void nvme_submit_cmd(struct nvme_queue *nvmeq, struct nvme_command *cmd) +static void nvme_submit_cmd(struct nvme_queue *nvmeq, struct nvme_command *cmd, + bool write_sq) { spin_lock(&nvmeq->sq_lock); memcpy(&nvmeq->sq_cmds[nvmeq->sq_tail], cmd, sizeof(*cmd)); - if (++nvmeq->sq_tail == nvmeq->q_depth) - nvmeq->sq_tail = 0; - if (nvme_dbbuf_update_and_check_event(nvmeq->sq_tail, - nvmeq->dbbuf_sq_db, nvmeq->dbbuf_sq_ei)) - writel(nvmeq->sq_tail, nvmeq->q_db); + /* + * Write sq tail if we have to, OR if the next command would wrap + */ + nvmeq->sq_tail = nvme_next_ring_index(nvmeq, nvmeq->sq_tail); + if (write_sq || + nvme_next_ring_index(nvmeq, nvmeq->sq_tail) == nvmeq->last_sq_tail) + nvme_write_sq_db(nvmeq); + spin_unlock(&nvmeq->sq_lock); +} + +static void nvme_commit_rqs(struct blk_mq_hw_ctx *hctx) +{ + struct nvme_queue *nvmeq = hctx->driver_data; + + spin_lock(&nvmeq->sq_lock); + if (nvmeq->sq_tail != nvmeq->last_sq_tail) + nvme_write_sq_db(nvmeq); spin_unlock(&nvmeq->sq_lock); } @@ -923,7 +954,7 @@ static blk_status_t nvme_queue_rq(struct blk_mq_hw_ctx *hctx, } blk_mq_start_request(req); - nvme_submit_cmd(nvmeq, &cmnd); + nvme_submit_cmd(nvmeq, &cmnd, bd->last); return BLK_STS_OK; out_cleanup_iod: nvme_free_iod(dev, req); @@ -999,8 +1030,7 @@ static void nvme_complete_cqes(struct nvme_queue *nvmeq, u16 start, u16 end) { while (start != end) { nvme_handle_cqe(nvmeq, start); - if (++start == nvmeq->q_depth) - start = 0; + start = nvme_next_ring_index(nvmeq, start); } } @@ -1108,7 +1138,7 @@ static void nvme_pci_submit_async_event(struct nvme_ctrl *ctrl) memset(&c, 0, sizeof(c)); c.common.opcode = nvme_admin_async_event; c.common.command_id = NVME_AQ_BLK_MQ_DEPTH; - nvme_submit_cmd(nvmeq, &c); + nvme_submit_cmd(nvmeq, &c, true); } static int adapter_delete_queue(struct nvme_dev *dev, u8 opcode, u16 id) @@ -1531,6 +1561,7 @@ static void nvme_init_queue(struct nvme_queue *nvmeq, u16 qid) spin_lock_irq(&nvmeq->cq_lock); nvmeq->sq_tail = 0; + nvmeq->last_sq_tail = 0; nvmeq->cq_head = 0; nvmeq->cq_phase = 1; nvmeq->q_db = &dev->dbs[qid * 2 * dev->db_stride]; @@ -1603,6 +1634,7 @@ static const struct blk_mq_ops nvme_mq_admin_ops = { #define NVME_SHARED_MQ_OPS \ .queue_rq = nvme_queue_rq, \ + .commit_rqs = nvme_commit_rqs, \ .rq_flags_to_type = nvme_rq_flags_to_type, \ .complete = nvme_pci_complete_rq, \ .init_hctx = nvme_init_hctx, \ From patchwork Mon Nov 26 16:35:53 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 10698625 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 38CDF13BF for ; Mon, 26 Nov 2018 16:36:14 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 2861529735 for ; Mon, 26 Nov 2018 16:36:14 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 1C5BD29B2B; Mon, 26 Nov 2018 16:36:14 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9F87B29735 for ; Mon, 26 Nov 2018 16:36:13 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726634AbeK0Dat (ORCPT ); Mon, 26 Nov 2018 22:30:49 -0500 Received: from mail-it1-f196.google.com ([209.85.166.196]:51809 "EHLO mail-it1-f196.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726253AbeK0Dat (ORCPT ); Mon, 26 Nov 2018 22:30:49 -0500 Received: by mail-it1-f196.google.com with SMTP id x19so29251760itl.1 for ; Mon, 26 Nov 2018 08:36:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=Dg+etYC2drYnfrLWClPG4P6hpiYhQ/56nszkzahxJB8=; b=EhJkWjQI2JP92wWMckYenWDZben7TEVnaTCmcoMvxAnGgskAEZFkeP84PdAuilkSce oIofQpUBmVmpJV5Y1Y3/CwCpK++S5GiRTy/UbHSbZsDAYI48Om1nxk+hnRlTwyldAl7l nJQZ3pbUYGPLR6clzQ2FrTndxIPXKzuNNNlkoZqDAaZ8WmB3TIx2AdDqUGYfo+4qGwch aCIWP/QoNQHMnnnhnpr5SPN86y+XdF6fJ3r0T807+le+lUywKIX84JpZouBXRjd1xunw tlkWcIyboX1sM1eZICFLv3Ds3fhaAYuBmpBfO6my+nuFFzizsL25uKSf6tBXYR4NSKlp 90JQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=Dg+etYC2drYnfrLWClPG4P6hpiYhQ/56nszkzahxJB8=; b=h67J7u9l5IFa8QB203diAcII8XRq4Wv7m0tqjczTjD0j5r7MFTSUquZ5RX/gUUE58o vN6H5iEIN6mu6h3qTymyEcGdMB/wUtDI/xPBAzI4XWBwPyUpu/ywhgelmjsm4LYSQRtj bkk+bS29quGdfwUekRh+dcR4k+2qZ38hpjBuF1zsQrO1qcOHP94oV1avYwwOKSuw8aoO RhiTiVROnTSV3ZSaJyZxDw1xDmokL9fN+HyFrk3bKGAIyJJcGB68VJzhhTVVnFqF+Q5w x0CtWrtbp4PdBeye6ij8TR0FfPrixeWHVdDpLqIcsO+pxJPYH4mE7h1ItdVqKz6TXdP9 6F0Q== X-Gm-Message-State: AA+aEWZ3qcHTZn8mQpU0S4qVn5/a5rQ5Yo4m/rE8jHBj1uNvCcAafqUU aAKhA+CWPIZ4VaK6jagq3PMhtSdxBmc= X-Google-Smtp-Source: AJdET5fKZfS1T0DGCM+D8/+UpX+tglfd9Li+rspSnSa1OaYX1Kk9ez99fq8dXt9fh5oxLOD7pFRgxA== X-Received: by 2002:a24:5989:: with SMTP id p131-v6mr22798269itb.0.1543250171582; Mon, 26 Nov 2018 08:36:11 -0800 (PST) Received: from localhost.localdomain ([216.160.245.98]) by smtp.gmail.com with ESMTPSA id r11sm263717iog.46.2018.11.26.08.36.08 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 26 Nov 2018 08:36:10 -0800 (PST) From: Jens Axboe To: linux-block@vger.kernel.org, linux-nvme@lists.infradead.org Cc: Jens Axboe Subject: [PATCH 5/8] virtio_blk: implement mq_ops->commit_rqs() hook Date: Mon, 26 Nov 2018 09:35:53 -0700 Message-Id: <20181126163556.5181-6-axboe@kernel.dk> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181126163556.5181-1-axboe@kernel.dk> References: <20181126163556.5181-1-axboe@kernel.dk> Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP We need this for blk-mq to kick things into gear, if we told it that we had more IO coming, but then failed to deliver on that promise. Signed-off-by: Jens Axboe Reviewed-by: Omar Sandoval Acked-by: Michael S. Tsirkin Reviewed-by: Christoph Hellwig Reviewed-by: Ming Lei --- drivers/block/virtio_blk.c | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/drivers/block/virtio_blk.c b/drivers/block/virtio_blk.c index 6e869d05f91e..b49c57e77780 100644 --- a/drivers/block/virtio_blk.c +++ b/drivers/block/virtio_blk.c @@ -214,6 +214,20 @@ static void virtblk_done(struct virtqueue *vq) spin_unlock_irqrestore(&vblk->vqs[qid].lock, flags); } +static void virtio_commit_rqs(struct blk_mq_hw_ctx *hctx) +{ + struct virtio_blk *vblk = hctx->queue->queuedata; + int qid = hctx->queue_num; + bool kick; + + spin_lock_irq(&vblk->vqs[qid].lock); + kick = virtqueue_kick_prepare(vblk->vqs[qid].vq); + spin_unlock_irq(&vblk->vqs[qid].lock); + + if (kick) + virtqueue_notify(vblk->vqs[qid].vq); +} + static blk_status_t virtio_queue_rq(struct blk_mq_hw_ctx *hctx, const struct blk_mq_queue_data *bd) { @@ -638,6 +652,7 @@ static void virtblk_initialize_rq(struct request *req) static const struct blk_mq_ops virtio_mq_ops = { .queue_rq = virtio_queue_rq, + .commit_rqs = virtio_commit_rqs, .complete = virtblk_request_done, .init_request = virtblk_init_request, #ifdef CONFIG_VIRTIO_BLK_SCSI From patchwork Mon Nov 26 16:35:54 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 10698627 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 287E913BB for ; Mon, 26 Nov 2018 16:36:16 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 1939D29735 for ; Mon, 26 Nov 2018 16:36:16 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 0D52F29B2B; Mon, 26 Nov 2018 16:36:16 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 94F2629735 for ; Mon, 26 Nov 2018 16:36:15 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726852AbeK0Dav (ORCPT ); Mon, 26 Nov 2018 22:30:51 -0500 Received: from mail-it1-f196.google.com ([209.85.166.196]:54599 "EHLO mail-it1-f196.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726588AbeK0Dav (ORCPT ); Mon, 26 Nov 2018 22:30:51 -0500 Received: by mail-it1-f196.google.com with SMTP id m123-v6so5009405ita.4 for ; Mon, 26 Nov 2018 08:36:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=/IBFVWV8/9cHpHivEgC4qsXTRnszXOXgHHSPu+O1gfg=; b=BstM0UcleU6Oj9/CJ5rEqrksTn04L/iRF/r+wlVu0zZH6nQR8LMmxudeNKDPEMaGiZ XLnGj5syo8dBYK2VlhUmDrg7knsCbbxaLrTgE0Jv8FI6Qym1rKRr/qcKBQISQdRDTx8q fKJZ6JP4f3d28RT6l0RXaom8QfvFKbVCN+EIxpYB2RtjMHlhM5vdqGIJq+wdRROd7k4h 9rAfRntJ5KLpqVxZGZeYiNAThAKlcvQ66gKYe2ZArjFfUC1M59clSIVxXxrK0SpR0+4a XEg/iPYXFk0PmcJL57zTrAv8CzzbFDh4kon7Ttx3qKPxnkUdoxDqCwgrCs0Vvc9hO5w+ uBqQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=/IBFVWV8/9cHpHivEgC4qsXTRnszXOXgHHSPu+O1gfg=; b=mvcJByZLFZaocavy7X2k3uHWcdctGlm2pvMHu5Cm8Fsx+Rt9X/wvxwk8ZEg+U51yVK UWpWGhMJsIESvb2P39T+zsE1FEAwvQkwMT68kmt7DjIjLby9D0OxAOOQGh9HBWgHBvB6 BIHM0l90RZ43j5eCG5lb0vKs4sVZPJkEufrYlXdONW9xzSVWorrWG8pNuDGDzjHViBb6 D5mueXiddsJNNFli9Mz+JD15yQNUWgXs0wgLWUcmY0h+0jfrOJRCCTOnCOO8TnjXEMPF sg2W8RZdiKpheMwmAGLhDmc/WV9WpofFmxpqJf4v7lAmE/YWg7QwZQwqiZTX3WrYZt3R hebQ== X-Gm-Message-State: AA+aEWbHzSx9x+yuZSpH0ZBgLlsFBRdhxzD358nAGgIHOUua0U/ZRzgO yIt0fUlIDqsDk/nAnaJkRs8XomdU3Yk= X-Google-Smtp-Source: AJdET5cLtxZdynO8Oh5MGhpDWy98RkZzCjGVC1E99MT+CinWhh7yWEI8HjLRfRQ9wuMUhgUU57INpw== X-Received: by 2002:a24:3390:: with SMTP id k138mr22782463itk.132.1543250173429; Mon, 26 Nov 2018 08:36:13 -0800 (PST) Received: from localhost.localdomain ([216.160.245.98]) by smtp.gmail.com with ESMTPSA id r11sm263717iog.46.2018.11.26.08.36.11 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 26 Nov 2018 08:36:12 -0800 (PST) From: Jens Axboe To: linux-block@vger.kernel.org, linux-nvme@lists.infradead.org Cc: Jens Axboe Subject: [PATCH 6/8] ataflop: implement mq_ops->commit_rqs() hook Date: Mon, 26 Nov 2018 09:35:54 -0700 Message-Id: <20181126163556.5181-7-axboe@kernel.dk> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181126163556.5181-1-axboe@kernel.dk> References: <20181126163556.5181-1-axboe@kernel.dk> Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP We need this for blk-mq to kick things into gear, if we told it that we had more IO coming, but then failed to deliver on that promise. Signed-off-by: Jens Axboe Reviewed-by: Omar Sandoval --- drivers/block/ataflop.c | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/drivers/block/ataflop.c b/drivers/block/ataflop.c index f88b4c26d422..475cb972f324 100644 --- a/drivers/block/ataflop.c +++ b/drivers/block/ataflop.c @@ -1471,6 +1471,13 @@ static void setup_req_params( int drive ) ReqTrack, ReqSector, (unsigned long)ReqData )); } +static void ataflop_commit_rqs(struct blk_mq_hw_ctx *hctx) +{ + spin_lock_irq(&ataflop_lock); + finish_fdc(); + spin_unlock_irq(&ataflop_lock); +} + static blk_status_t ataflop_queue_rq(struct blk_mq_hw_ctx *hctx, const struct blk_mq_queue_data *bd) { @@ -1947,6 +1954,7 @@ static const struct block_device_operations floppy_fops = { static const struct blk_mq_ops ataflop_mq_ops = { .queue_rq = ataflop_queue_rq, + .commit_rqs = ataflop_commit_rqs, }; static struct kobject *floppy_find(dev_t dev, int *part, void *data) From patchwork Mon Nov 26 16:35:55 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 10698631 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 8F99013BB for ; Mon, 26 Nov 2018 16:36:19 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7FA4729735 for ; Mon, 26 Nov 2018 16:36:19 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 73CBC29B2B; Mon, 26 Nov 2018 16:36:19 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 427DE29735 for ; Mon, 26 Nov 2018 16:36:18 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726872AbeK0Day (ORCPT ); Mon, 26 Nov 2018 22:30:54 -0500 Received: from mail-io1-f67.google.com ([209.85.166.67]:45362 "EHLO mail-io1-f67.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726588AbeK0Day (ORCPT ); Mon, 26 Nov 2018 22:30:54 -0500 Received: by mail-io1-f67.google.com with SMTP id w7so14430703iom.12 for ; Mon, 26 Nov 2018 08:36:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=Rqs+kcz4289Ghf5rqO8+g1izjb6etgUeESUQoGcKqu8=; b=u92aavNBLoOy1UkOEPDZw/jyK2q+4Gc4EQPqfb7sg9dQPJSeBeG9K9hQe2bNRlxC1s huykcWFTsWkXAgnU0moFJVbIalMKX3seyjy0LFUgXJfpULtLzytt43YniRYzXUuo5CSK 2zEDfF7OCm7cJaA8d8R/1huLpZRZgEkO3vdPlnxq1rmueo2Rec6Wp6FcEfWDsU+N6Ee6 jFhug5WgRPVnEs5gA8fYjcM8HVTrKjLoWa2AIRjfPK0cOvxYO0rPPwysmEWWC2E2hPLm BPJOiq9OnOjWltLwlC4YDeHScHbCU6RHdrirBppKImIzM0pNAS3S0Z0WuocSo1XEJLmn y6JA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=Rqs+kcz4289Ghf5rqO8+g1izjb6etgUeESUQoGcKqu8=; b=dRpaC0aSGj9n9ieuEz74D2FhtHe3sXquZAP/i5OcY5hRTXbUg+Gkgund9dtumLfKm9 sM9RzM2brpLwxHdH8oBwu2M6iVCECtLOGns8vUbcaPoS3G9NAICeedMouuWKkun9wkhN tgUjg9FtHQAucVD7iCyEHa6coHB53P0pu7RMLD/aN/P2atmSACz9pJiMf5VYAT5Ku+Gb YXia59DHHMo62zn59dC7LQfwQaYgfN35VipzmyBFsO2nxI8NUkxPEscAzOTHGy4Y1M3y VMbQznoehycXkgU77i6zWPgD1wWBpPkChn9YeLTM0dKRoryzkfOAZPGZNa9Ox+trSHb3 qAyQ== X-Gm-Message-State: AA+aEWZHjJOKGbvByZ4AM0onjP9PIxUAABjFu343g4WI5cT4rmWPWWhc NguJrDLRhsmaCAwnwlfycxcnhiLoL4I= X-Google-Smtp-Source: AFSGD/VcAq91nLKc3FPyvp+zfjJXLTEABcFOQKtReaeZQTcN3xKO/6q0znoccfU04cyf1jD6Vvc7gg== X-Received: by 2002:a5e:9804:: with SMTP id s4mr18902345ioj.4.1543250175739; Mon, 26 Nov 2018 08:36:15 -0800 (PST) Received: from localhost.localdomain ([216.160.245.98]) by smtp.gmail.com with ESMTPSA id r11sm263717iog.46.2018.11.26.08.36.13 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 26 Nov 2018 08:36:13 -0800 (PST) From: Jens Axboe To: linux-block@vger.kernel.org, linux-nvme@lists.infradead.org Cc: Jens Axboe Subject: [PATCH 7/8] blk-mq: use bd->last == true for list inserts Date: Mon, 26 Nov 2018 09:35:55 -0700 Message-Id: <20181126163556.5181-8-axboe@kernel.dk> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181126163556.5181-1-axboe@kernel.dk> References: <20181126163556.5181-1-axboe@kernel.dk> Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP If we are issuing a list of requests, we know if we're at the last one. If we fail issuing, ensure that we call ->commits_rqs() to flush any potential previous requests. Signed-off-by: Jens Axboe Reviewed-by: Omar Sandoval --- block/blk-core.c | 2 +- block/blk-mq.c | 32 ++++++++++++++++++++++++-------- block/blk-mq.h | 2 +- 3 files changed, 26 insertions(+), 10 deletions(-) diff --git a/block/blk-core.c b/block/blk-core.c index c9758d185357..808a65d23f1a 100644 --- a/block/blk-core.c +++ b/block/blk-core.c @@ -1334,7 +1334,7 @@ blk_status_t blk_insert_cloned_request(struct request_queue *q, struct request * * bypass a potential scheduler on the bottom device for * insert. */ - return blk_mq_request_issue_directly(rq); + return blk_mq_request_issue_directly(rq, true); } EXPORT_SYMBOL_GPL(blk_insert_cloned_request); diff --git a/block/blk-mq.c b/block/blk-mq.c index 6a249bf6ed00..0a12cec0b426 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -1260,6 +1260,14 @@ bool blk_mq_dispatch_rq_list(struct request_queue *q, struct list_head *list, if (!list_empty(list)) { bool needs_restart; + /* + * If we didn't flush the entire list, we could have told + * the driver there was more coming, but that turned out to + * be a lie. + */ + if (q->mq_ops->commit_rqs) + q->mq_ops->commit_rqs(hctx); + spin_lock(&hctx->lock); list_splice_init(list, &hctx->dispatch); spin_unlock(&hctx->lock); @@ -1736,12 +1744,12 @@ static blk_qc_t request_to_qc_t(struct blk_mq_hw_ctx *hctx, struct request *rq) static blk_status_t __blk_mq_issue_directly(struct blk_mq_hw_ctx *hctx, struct request *rq, - blk_qc_t *cookie) + blk_qc_t *cookie, bool last) { struct request_queue *q = rq->q; struct blk_mq_queue_data bd = { .rq = rq, - .last = true, + .last = last, }; blk_qc_t new_cookie; blk_status_t ret; @@ -1776,7 +1784,7 @@ static blk_status_t __blk_mq_issue_directly(struct blk_mq_hw_ctx *hctx, static blk_status_t __blk_mq_try_issue_directly(struct blk_mq_hw_ctx *hctx, struct request *rq, blk_qc_t *cookie, - bool bypass_insert) + bool bypass_insert, bool last) { struct request_queue *q = rq->q; bool run_queue = true; @@ -1805,7 +1813,7 @@ static blk_status_t __blk_mq_try_issue_directly(struct blk_mq_hw_ctx *hctx, goto insert; } - return __blk_mq_issue_directly(hctx, rq, cookie); + return __blk_mq_issue_directly(hctx, rq, cookie, last); insert: if (bypass_insert) return BLK_STS_RESOURCE; @@ -1824,7 +1832,7 @@ static void blk_mq_try_issue_directly(struct blk_mq_hw_ctx *hctx, hctx_lock(hctx, &srcu_idx); - ret = __blk_mq_try_issue_directly(hctx, rq, cookie, false); + ret = __blk_mq_try_issue_directly(hctx, rq, cookie, false, true); if (ret == BLK_STS_RESOURCE || ret == BLK_STS_DEV_RESOURCE) blk_mq_sched_insert_request(rq, false, true, false); else if (ret != BLK_STS_OK) @@ -1833,7 +1841,7 @@ static void blk_mq_try_issue_directly(struct blk_mq_hw_ctx *hctx, hctx_unlock(hctx, srcu_idx); } -blk_status_t blk_mq_request_issue_directly(struct request *rq) +blk_status_t blk_mq_request_issue_directly(struct request *rq, bool last) { blk_status_t ret; int srcu_idx; @@ -1841,7 +1849,7 @@ blk_status_t blk_mq_request_issue_directly(struct request *rq) struct blk_mq_hw_ctx *hctx = rq->mq_hctx; hctx_lock(hctx, &srcu_idx); - ret = __blk_mq_try_issue_directly(hctx, rq, &unused_cookie, true); + ret = __blk_mq_try_issue_directly(hctx, rq, &unused_cookie, true, last); hctx_unlock(hctx, srcu_idx); return ret; @@ -1856,7 +1864,7 @@ void blk_mq_try_issue_list_directly(struct blk_mq_hw_ctx *hctx, queuelist); list_del_init(&rq->queuelist); - ret = blk_mq_request_issue_directly(rq); + ret = blk_mq_request_issue_directly(rq, list_empty(list)); if (ret != BLK_STS_OK) { if (ret == BLK_STS_RESOURCE || ret == BLK_STS_DEV_RESOURCE) { @@ -1866,6 +1874,14 @@ void blk_mq_try_issue_list_directly(struct blk_mq_hw_ctx *hctx, blk_mq_end_request(rq, ret); } } + + /* + * If we didn't flush the entire list, we could have told + * the driver there was more coming, but that turned out to + * be a lie. + */ + if (!list_empty(list) && hctx->queue->mq_ops->commit_rqs) + hctx->queue->mq_ops->commit_rqs(hctx); } static blk_qc_t blk_mq_make_request(struct request_queue *q, struct bio *bio) diff --git a/block/blk-mq.h b/block/blk-mq.h index 9ae8e9f8f8b1..7291e5379358 100644 --- a/block/blk-mq.h +++ b/block/blk-mq.h @@ -69,7 +69,7 @@ void blk_mq_insert_requests(struct blk_mq_hw_ctx *hctx, struct blk_mq_ctx *ctx, struct list_head *list); /* Used by blk_insert_cloned_request() to issue request directly */ -blk_status_t blk_mq_request_issue_directly(struct request *rq); +blk_status_t blk_mq_request_issue_directly(struct request *rq, bool last); void blk_mq_try_issue_list_directly(struct blk_mq_hw_ctx *hctx, struct list_head *list); From patchwork Mon Nov 26 16:35:56 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 10698633 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 241B513BF for ; Mon, 26 Nov 2018 16:36:21 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 14C9029735 for ; Mon, 26 Nov 2018 16:36:21 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 0923629B2B; Mon, 26 Nov 2018 16:36:21 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 8A82729735 for ; Mon, 26 Nov 2018 16:36:20 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727003AbeK0Da4 (ORCPT ); Mon, 26 Nov 2018 22:30:56 -0500 Received: from mail-io1-f67.google.com ([209.85.166.67]:38873 "EHLO mail-io1-f67.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726588AbeK0Da4 (ORCPT ); Mon, 26 Nov 2018 22:30:56 -0500 Received: by mail-io1-f67.google.com with SMTP id l14so14301067ioj.5 for ; Mon, 26 Nov 2018 08:36:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=9+3qzXhfJkyKm0ac9rnSYO7z3XNAZpFNPEOllN9qF64=; b=HsHW5XSiXqzM6T9q8jzNDAiPzXj7Brt8W5gyNtxhHxxE1Lv+UE37gAKG55IsGnMMyK ft4msdtjG4jEwo3FzVk4pXZV5J3B+LqIo119NPTFZ/uAzI9ZyaBEPWcbtj7n6SyOPQoz 4sgL3UDjdvwLaLzygitl2HRd0dVv61+z0DPbofpVBTnmW2nNLJn0uldSseQBLXxzXpAf 6wesSKt+5snbjfPsNdEaql7NiZDjvro8oD+igRW6bYGIJk2dQ6+xQsL+DeMmUjo1o1xx kZTF49vqBtuDb0nqHkVleEa/vPngTH8v7M3ZlTEcihr2cogTOLONyOa0Z+9KUMNhjSqS MYuQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=9+3qzXhfJkyKm0ac9rnSYO7z3XNAZpFNPEOllN9qF64=; b=iHlDHO0o36rMWr2BUlF4NeK2L9sxBeIwYWredZO5Q+oVuesEHmf1kFS/DJqy2eeAyM E1/fIyEAO+m5GxRBu22VgBATFawz3YAkjzyeqzFVsGHrVIrguvlhaKiKeEcblOt5I0BL URMgaQo17H+KsjX20fnLwLj9OzWUT2sy0jnlyZv6n2aC9WsME7PQ58ku3CbUXen1nDWa +OudhM+16NYbGQX2YBp3EtbTxwUMENVpFymMDSUBmalDuF6aOF1si/+iWae1DFskDhuk E+nLBMiJY7YOG2sopoPZNclDqB3iD6VwsSUVOH7K4THCTvq5VV8VZTyAbbuwwpcrLLv5 /m5A== X-Gm-Message-State: AA+aEWZwU1whu7b+m/fbFHng4+sZKIveCe1hr5JPHnHTaLScHcqM99k4 vXIegBVs9rhkFmwUdt40amARYvbvkws= X-Google-Smtp-Source: AFSGD/Wxm5jgDXfCZr2swfnwRChizBSWgWReuHnFfm5/Xmiq2ifhp8G2eOcUYyZnX63D/4vav+X4MA== X-Received: by 2002:a6b:4e14:: with SMTP id c20mr20270853iob.65.1543250178384; Mon, 26 Nov 2018 08:36:18 -0800 (PST) Received: from localhost.localdomain ([216.160.245.98]) by smtp.gmail.com with ESMTPSA id r11sm263717iog.46.2018.11.26.08.36.15 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 26 Nov 2018 08:36:16 -0800 (PST) From: Jens Axboe To: linux-block@vger.kernel.org, linux-nvme@lists.infradead.org Cc: Jens Axboe Subject: [PATCH 8/8] blk-mq: add plug case for devices that implement ->commits_rqs() Date: Mon, 26 Nov 2018 09:35:56 -0700 Message-Id: <20181126163556.5181-9-axboe@kernel.dk> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181126163556.5181-1-axboe@kernel.dk> References: <20181126163556.5181-1-axboe@kernel.dk> Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP If we have that hook, we know the driver handles bd->last == true in a smart fashion. If it does, even for multiple hardware queues, it's a good idea to flush batches of requests to the device, if we have batches of requests from the submitter. Signed-off-by: Jens Axboe --- block/blk-mq.c | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+) diff --git a/block/blk-mq.c b/block/blk-mq.c index 0a12cec0b426..232f4914c8db 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -1953,6 +1953,32 @@ static blk_qc_t blk_mq_make_request(struct request_queue *q, struct bio *bio) list_add_tail(&rq->queuelist, &plug->mq_list); plug->rq_count++; plug->do_sort = true; + } else if (plug && q->mq_ops->commit_rqs) { + /* + * If we have a ->commit_rqs(), then we know the driver can + * batch submission doorbell updates. Add rq to plug list, + * and flush if we exceed the plug count only. + */ + blk_mq_bio_to_request(rq, bio); + blk_mq_put_ctx(data.ctx); + + /* + * If we have requests for more than one queue here, we + * should sort the list when it's flushed. + */ + if (!plug->do_sort && !list_empty(&plug->mq_list)) { + same_queue_rq = list_first_entry(&plug->mq_list, + struct request, queuelist); + if (same_queue_rq->q != q) + plug->do_sort = true; + } + + list_add_tail(&rq->queuelist, &plug->mq_list); + plug->rq_count++; + if (plug->rq_count >= BLK_MAX_REQUEST_COUNT) { + blk_flush_plug_list(plug, false); + trace_block_plug(q); + } } else if (plug && !blk_queue_nomerges(q)) { blk_mq_bio_to_request(rq, bio);