From patchwork Thu Nov 15 19:51:25 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 10684963 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id CC1BC17EE for ; Thu, 15 Nov 2018 19:51:44 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id BE8DE2D168 for ; Thu, 15 Nov 2018 19:51:44 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id BCE482D0D0; Thu, 15 Nov 2018 19:51:44 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 514072D132 for ; Thu, 15 Nov 2018 19:51:44 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726007AbeKPGAz (ORCPT ); Fri, 16 Nov 2018 01:00:55 -0500 Received: from mail-pl1-f194.google.com ([209.85.214.194]:44896 "EHLO mail-pl1-f194.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725869AbeKPGAz (ORCPT ); Fri, 16 Nov 2018 01:00:55 -0500 Received: by mail-pl1-f194.google.com with SMTP id s5-v6so9953079plq.11 for ; Thu, 15 Nov 2018 11:51:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=0hpfobUzk41/y0GfZii9AW4JlRiFqpFtiDvsBHwkxOc=; b=fPbbJdb5dG26CtmSNiv9jIF/ybkn5kvrIbkQ7Hb3ywOwbBDLBW1w9ME7KkA066yZU8 73cQUG/NaRLNBaF0gAvwE9+AOZtccInaU3yHFuWpmv3U7dL5G/zXg40KXOug1+NVmI4x vZipoW4XoypuWBAVDSUlVL49gXUnsifEYSvNKZDAnb2EVy8gEN3eoHSNdNrF93va81FU J4waZl9FjH+uh8rF4hUSyyXDsvGtzZfEMjhylmwSz4gSUx46RyFXF14zCUeoO2F7ItZb nJUYl0APvaLiX4zHPIyJziegBCt863QBb5NAg/SqxXLCzBb5rpJjALWqNh4r2kgmQLNe fccA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=0hpfobUzk41/y0GfZii9AW4JlRiFqpFtiDvsBHwkxOc=; b=JcWkUUpTy+AH/Oh0Fi4ciXNJqf/BcEFB1tg7XTnt3o30BqEyCd7it3jOOfscUu2vbL Sk6mRIKd55ScBMb9sJdksjTnjFP9KAe8FuW8Vg35BfDe9a1wkNH5CO7av3ozLUy5tBRA MLA0sdXKxaUPCb5tujKas38NoibX9f9IUX5XbVezEDqMKV2um2BilbcLvjXjEmqLOufN SM20uwYV2JRynlnMUM3XsctLf7fbP68910pWIXMAF8dWyHPX6oQIarqJ9y1EJI0kkflj qTdDza1qmHsd6gxGoDPnHcVMBYduopG+Bu7vuzD2AAq7Pfq/8u7jEAxnApaE6kRYO3W9 7Xxg== X-Gm-Message-State: AGRZ1gJYq8//OK2moKhoES7GPZhJ5YqgeGBQ3U1kcjMQf9t4y8zmmgAp gmatA1CKa+zM7/VcnrikyuCxOTQmwzo= X-Google-Smtp-Source: AJdET5c+6kGIFk5Ui7lrL4XTmHob0BxE7zGNpFulsQkrBsnN6O0S3rjVdpaHUXtM+fgAwKLoRcEUZw== X-Received: by 2002:a17:902:8210:: with SMTP id x16-v6mr7703705pln.129.1542311502562; Thu, 15 Nov 2018 11:51:42 -0800 (PST) Received: from x1.localdomain ([2620:10d:c090:180::1:7813]) by smtp.gmail.com with ESMTPSA id d68-v6sm30670766pfa.80.2018.11.15.11.51.41 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 15 Nov 2018 11:51:41 -0800 (PST) From: Jens Axboe To: linux-block@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 01/11] nvme: provide optimized poll function for separate poll queues Date: Thu, 15 Nov 2018 12:51:25 -0700 Message-Id: <20181115195135.22812-2-axboe@kernel.dk> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181115195135.22812-1-axboe@kernel.dk> References: <20181115195135.22812-1-axboe@kernel.dk> Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP If we have separate poll queues, we know that they aren't using interrupts. Hence we don't need to disable interrupts around finding completions. Provide a separate set of blk_mq_ops for such devices. Signed-off-by: Jens Axboe --- drivers/nvme/host/pci.c | 45 +++++++++++++++++++++++++++++++++-------- 1 file changed, 37 insertions(+), 8 deletions(-) diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c index ffbab5b01df4..fc7dd49f22fc 100644 --- a/drivers/nvme/host/pci.c +++ b/drivers/nvme/host/pci.c @@ -1082,6 +1082,23 @@ static int nvme_poll(struct blk_mq_hw_ctx *hctx, unsigned int tag) return __nvme_poll(nvmeq, tag); } +static int nvme_poll_noirq(struct blk_mq_hw_ctx *hctx, unsigned int tag) +{ + struct nvme_queue *nvmeq = hctx->driver_data; + u16 start, end; + bool found; + + if (!nvme_cqe_pending(nvmeq)) + return 0; + + spin_lock(&nvmeq->cq_lock); + found = nvme_process_cq(nvmeq, &start, &end, tag); + spin_unlock(&nvmeq->cq_lock); + + nvme_complete_cqes(nvmeq, start, end); + return found; +} + static void nvme_pci_submit_async_event(struct nvme_ctrl *ctrl) { struct nvme_dev *dev = to_nvme_dev(ctrl); @@ -1584,17 +1601,25 @@ static const struct blk_mq_ops nvme_mq_admin_ops = { .timeout = nvme_timeout, }; +#define NVME_SHARED_MQ_OPS \ + .queue_rq = nvme_queue_rq, \ + .rq_flags_to_type = nvme_rq_flags_to_type, \ + .complete = nvme_pci_complete_rq, \ + .init_hctx = nvme_init_hctx, \ + .init_request = nvme_init_request, \ + .map_queues = nvme_pci_map_queues, \ + .timeout = nvme_timeout \ + static const struct blk_mq_ops nvme_mq_ops = { - .queue_rq = nvme_queue_rq, - .rq_flags_to_type = nvme_rq_flags_to_type, - .complete = nvme_pci_complete_rq, - .init_hctx = nvme_init_hctx, - .init_request = nvme_init_request, - .map_queues = nvme_pci_map_queues, - .timeout = nvme_timeout, + NVME_SHARED_MQ_OPS, .poll = nvme_poll, }; +static const struct blk_mq_ops nvme_mq_poll_noirq_ops = { + NVME_SHARED_MQ_OPS, + .poll = nvme_poll_noirq, +}; + static void nvme_dev_remove_admin(struct nvme_dev *dev) { if (dev->ctrl.admin_q && !blk_queue_dying(dev->ctrl.admin_q)) { @@ -2274,7 +2299,11 @@ static int nvme_dev_add(struct nvme_dev *dev) int ret; if (!dev->ctrl.tagset) { - dev->tagset.ops = &nvme_mq_ops; + if (!dev->io_queues[NVMEQ_TYPE_POLL]) + dev->tagset.ops = &nvme_mq_ops; + else + dev->tagset.ops = &nvme_mq_poll_noirq_ops; + dev->tagset.nr_hw_queues = dev->online_queues - 1; dev->tagset.nr_maps = NVMEQ_TYPE_NR; dev->tagset.timeout = NVME_IO_TIMEOUT; From patchwork Thu Nov 15 19:51:26 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 10684965 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id F30A813BB for ; Thu, 15 Nov 2018 19:51:48 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E31672D133 for ; Thu, 15 Nov 2018 19:51:48 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id E105B2D11C; Thu, 15 Nov 2018 19:51:48 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E6FED2D133 for ; Thu, 15 Nov 2018 19:51:47 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1725809AbeKPGA7 (ORCPT ); Fri, 16 Nov 2018 01:00:59 -0500 Received: from mail-pg1-f196.google.com ([209.85.215.196]:35645 "EHLO mail-pg1-f196.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725869AbeKPGA6 (ORCPT ); Fri, 16 Nov 2018 01:00:58 -0500 Received: by mail-pg1-f196.google.com with SMTP id 32-v6so9479894pgu.2 for ; Thu, 15 Nov 2018 11:51:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=zuE5HiN6neQwo6VH85lAFmz18mDze9gV2PKq91gHoV8=; b=0x/C4GSEu5uWMnS9zXJLj2Vu0t48P29eeuRKNyER7XKQN0LML2PtJn/wtOcw37ODq5 hGsWs2403kqk+g8mNtWWsfArE+SsSfNVF8+c8OoIk9B3SHIL0SzkoYKE3TurAHvZfZF/ P+PMxO1WDxo3N9uV94++NFS40Xx/DmFpVY2cXqwfIrE2O1mFSYfLYwxCCUkxIROG2kPs TzD+dj4eGiVaWPTpBARucDQ6LYLYBITp8JvgyWiE1xXXRUksD3dCFeVbfR7ITEuoc1Tl Lt8CmL80Bo9V64a65bP9LMddB86TPaaRH1A8xmrM4pzVL7Y6PWFcEnik4FPPJjydoVYK Vv1Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=zuE5HiN6neQwo6VH85lAFmz18mDze9gV2PKq91gHoV8=; b=hOsjmuj2+S2N7U3pBbyzO49eCcP/uwQGfYhNwAzGAcNdbV2fcG57w/kDg697iDTf41 XOdRGNePp99i9BrY0lBNDuv6xACCDlmC6hUVYmzmudzJzxNNStJ9Xe21HQyOi90TOgqV rrcePUP9BMYyWn0OlsKvjpJLYaG45MURCVDTBG54KC+ZL0A1k6pAF7XzTLKvvNBqoWRB 3HvMKZvZ0OH+K/BBo9GxaAxmKIJAqYpO6NgKSxe+1tv2SVrTmrF1r3jLxn9bKQe+lSbv s1bbJvUm8qGja0ueQWyGPsPBU4n3RYmxXdAWwSDj+uRd3wxZrh69eYEV7PWbxrFOcH1X xg+w== X-Gm-Message-State: AGRZ1gIN8oyjcq2h6lqxKQ7mTyifXF8eY4114brpxjbVea5fWiv31aMu s5GqsQqIBbNWvNmjWxb1qlB6gaF6AtA= X-Google-Smtp-Source: AJdET5drrjnTtzAvgAN40qUt4zjgMLaOLqQ+y8m+GpF8m8VLy+EnqpVobXqMoX+Rk9xzIFhj5LmVJA== X-Received: by 2002:a63:a41:: with SMTP id z1mr7037228pgk.117.1542311504663; Thu, 15 Nov 2018 11:51:44 -0800 (PST) Received: from x1.localdomain ([2620:10d:c090:180::1:7813]) by smtp.gmail.com with ESMTPSA id d68-v6sm30670766pfa.80.2018.11.15.11.51.42 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 15 Nov 2018 11:51:43 -0800 (PST) From: Jens Axboe To: linux-block@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 02/11] block: add queue_is_mq() helper Date: Thu, 15 Nov 2018 12:51:26 -0700 Message-Id: <20181115195135.22812-3-axboe@kernel.dk> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181115195135.22812-1-axboe@kernel.dk> References: <20181115195135.22812-1-axboe@kernel.dk> Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Various spots check for q->mq_ops being non-NULL, but provide a helper to do this instead. Where the ->mq_ops != NULL check is redundant, remove it. Since mq == rq-based now that legacy is gone, get rid of the queue_is_rq_based() and just use queue_is_mq() everywhere. Signed-off-by: Jens Axboe Reviewed-by: Christoph Hellwig --- block/blk-cgroup.c | 8 ++++---- block/blk-core.c | 12 ++++++------ block/blk-flush.c | 3 +-- block/blk-mq.c | 2 +- block/blk-sysfs.c | 14 +++++++------- block/blk-throttle.c | 2 +- block/blk-wbt.c | 2 +- block/blk-zoned.c | 2 +- block/bsg.c | 2 +- block/elevator.c | 11 +++++------ block/genhd.c | 8 ++++---- drivers/md/dm-rq.c | 2 +- drivers/md/dm-table.c | 4 ++-- include/linux/blkdev.h | 6 +----- 14 files changed, 36 insertions(+), 42 deletions(-) diff --git a/block/blk-cgroup.c b/block/blk-cgroup.c index 0f6b44614165..63d226a084cd 100644 --- a/block/blk-cgroup.c +++ b/block/blk-cgroup.c @@ -1324,7 +1324,7 @@ int blkcg_activate_policy(struct request_queue *q, if (blkcg_policy_enabled(q, pol)) return 0; - if (q->mq_ops) + if (queue_is_mq(q)) blk_mq_freeze_queue(q); pd_prealloc: if (!pd_prealloc) { @@ -1363,7 +1363,7 @@ int blkcg_activate_policy(struct request_queue *q, spin_unlock_irq(&q->queue_lock); out_bypass_end: - if (q->mq_ops) + if (queue_is_mq(q)) blk_mq_unfreeze_queue(q); if (pd_prealloc) pol->pd_free_fn(pd_prealloc); @@ -1387,7 +1387,7 @@ void blkcg_deactivate_policy(struct request_queue *q, if (!blkcg_policy_enabled(q, pol)) return; - if (q->mq_ops) + if (queue_is_mq(q)) blk_mq_freeze_queue(q); spin_lock_irq(&q->queue_lock); @@ -1405,7 +1405,7 @@ void blkcg_deactivate_policy(struct request_queue *q, spin_unlock_irq(&q->queue_lock); - if (q->mq_ops) + if (queue_is_mq(q)) blk_mq_unfreeze_queue(q); } EXPORT_SYMBOL_GPL(blkcg_deactivate_policy); diff --git a/block/blk-core.c b/block/blk-core.c index 92b6b200e9fb..0b684a520a11 100644 --- a/block/blk-core.c +++ b/block/blk-core.c @@ -232,7 +232,7 @@ void blk_sync_queue(struct request_queue *q) del_timer_sync(&q->timeout); cancel_work_sync(&q->timeout_work); - if (q->mq_ops) { + if (queue_is_mq(q)) { struct blk_mq_hw_ctx *hctx; int i; @@ -281,7 +281,7 @@ void blk_set_queue_dying(struct request_queue *q) */ blk_freeze_queue_start(q); - if (q->mq_ops) + if (queue_is_mq(q)) blk_mq_wake_waiters(q); /* Make blk_queue_enter() reexamine the DYING flag. */ @@ -356,7 +356,7 @@ void blk_cleanup_queue(struct request_queue *q) * blk_freeze_queue() should be enough for cases of passthrough * request. */ - if (q->mq_ops && blk_queue_init_done(q)) + if (queue_is_mq(q) && blk_queue_init_done(q)) blk_mq_quiesce_queue(q); /* for synchronous bio-based driver finish in-flight integrity i/o */ @@ -374,7 +374,7 @@ void blk_cleanup_queue(struct request_queue *q) blk_exit_queue(q); - if (q->mq_ops) + if (queue_is_mq(q)) blk_mq_free_queue(q); percpu_ref_exit(&q->q_usage_counter); @@ -982,7 +982,7 @@ generic_make_request_checks(struct bio *bio) * For a REQ_NOWAIT based request, return -EOPNOTSUPP * if queue is not a request based queue. */ - if ((bio->bi_opf & REQ_NOWAIT) && !queue_is_rq_based(q)) + if ((bio->bi_opf & REQ_NOWAIT) && !queue_is_mq(q)) goto not_supported; if (should_fail_bio(bio)) @@ -1657,7 +1657,7 @@ EXPORT_SYMBOL_GPL(rq_flush_dcache_pages); */ int blk_lld_busy(struct request_queue *q) { - if (q->mq_ops && q->mq_ops->busy) + if (queue_is_mq(q) && q->mq_ops->busy) return q->mq_ops->busy(q); return 0; diff --git a/block/blk-flush.c b/block/blk-flush.c index fcd18b158fd6..a3fc7191c694 100644 --- a/block/blk-flush.c +++ b/block/blk-flush.c @@ -273,8 +273,7 @@ static void blk_kick_flush(struct request_queue *q, struct blk_flush_queue *fq, * assigned to empty flushes, and we deadlock if we are expecting * other requests to make progress. Don't defer for that case. */ - if (!list_empty(&fq->flush_data_in_flight) && - !(q->mq_ops && q->elevator) && + if (!list_empty(&fq->flush_data_in_flight) && q->elevator && time_before(jiffies, fq->flush_pending_since + FLUSH_PENDING_TIMEOUT)) return; diff --git a/block/blk-mq.c b/block/blk-mq.c index 3b823891b3ef..32b246ed44c0 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -150,7 +150,7 @@ void blk_freeze_queue_start(struct request_queue *q) freeze_depth = atomic_inc_return(&q->mq_freeze_depth); if (freeze_depth == 1) { percpu_ref_kill(&q->q_usage_counter); - if (q->mq_ops) + if (queue_is_mq(q)) blk_mq_run_hw_queues(q, false); } } diff --git a/block/blk-sysfs.c b/block/blk-sysfs.c index 1e370207a20e..80eef48fddc8 100644 --- a/block/blk-sysfs.c +++ b/block/blk-sysfs.c @@ -68,7 +68,7 @@ queue_requests_store(struct request_queue *q, const char *page, size_t count) unsigned long nr; int ret, err; - if (!q->mq_ops) + if (!queue_is_mq(q)) return -EINVAL; ret = queue_var_store(&nr, page, count); @@ -835,12 +835,12 @@ static void __blk_release_queue(struct work_struct *work) blk_queue_free_zone_bitmaps(q); - if (q->mq_ops) + if (queue_is_mq(q)) blk_mq_release(q); blk_trace_shutdown(q); - if (q->mq_ops) + if (queue_is_mq(q)) blk_mq_debugfs_unregister(q); bioset_exit(&q->bio_split); @@ -914,7 +914,7 @@ int blk_register_queue(struct gendisk *disk) goto unlock; } - if (q->mq_ops) { + if (queue_is_mq(q)) { __blk_mq_register_dev(dev, q); blk_mq_debugfs_register(q); } @@ -925,7 +925,7 @@ int blk_register_queue(struct gendisk *disk) blk_throtl_register_queue(q); - if ((q->mq_ops && q->elevator)) { + if (q->elevator) { ret = elv_register_queue(q); if (ret) { mutex_unlock(&q->sysfs_lock); @@ -974,7 +974,7 @@ void blk_unregister_queue(struct gendisk *disk) * Remove the sysfs attributes before unregistering the queue data * structures that can be modified through sysfs. */ - if (q->mq_ops) + if (queue_is_mq(q)) blk_mq_unregister_dev(disk_to_dev(disk), q); mutex_unlock(&q->sysfs_lock); @@ -983,7 +983,7 @@ void blk_unregister_queue(struct gendisk *disk) blk_trace_remove_sysfs(disk_to_dev(disk)); mutex_lock(&q->sysfs_lock); - if (q->mq_ops && q->elevator) + if (q->elevator) elv_unregister_queue(q); mutex_unlock(&q->sysfs_lock); diff --git a/block/blk-throttle.c b/block/blk-throttle.c index d0a23f0bb3ed..8f0a104770ee 100644 --- a/block/blk-throttle.c +++ b/block/blk-throttle.c @@ -2456,7 +2456,7 @@ void blk_throtl_register_queue(struct request_queue *q) td->throtl_slice = DFL_THROTL_SLICE_HD; #endif - td->track_bio_latency = !queue_is_rq_based(q); + td->track_bio_latency = !queue_is_mq(q); if (!td->track_bio_latency) blk_stat_enable_accounting(q); } diff --git a/block/blk-wbt.c b/block/blk-wbt.c index 9f142b84dc85..d051ebfb4852 100644 --- a/block/blk-wbt.c +++ b/block/blk-wbt.c @@ -701,7 +701,7 @@ void wbt_enable_default(struct request_queue *q) if (!test_bit(QUEUE_FLAG_REGISTERED, &q->queue_flags)) return; - if (q->mq_ops && IS_ENABLED(CONFIG_BLK_WBT_MQ)) + if (queue_is_mq(q) && IS_ENABLED(CONFIG_BLK_WBT_MQ)) wbt_init(q); } EXPORT_SYMBOL_GPL(wbt_enable_default); diff --git a/block/blk-zoned.c b/block/blk-zoned.c index 13ba2011a306..e9c332b1d9da 100644 --- a/block/blk-zoned.c +++ b/block/blk-zoned.c @@ -421,7 +421,7 @@ int blk_revalidate_disk_zones(struct gendisk *disk) * BIO based queues do not use a scheduler so only q->nr_zones * needs to be updated so that the sysfs exposed value is correct. */ - if (!queue_is_rq_based(q)) { + if (!queue_is_mq(q)) { q->nr_zones = nr_zones; return 0; } diff --git a/block/bsg.c b/block/bsg.c index 9a442c23a715..44f6028b9567 100644 --- a/block/bsg.c +++ b/block/bsg.c @@ -471,7 +471,7 @@ int bsg_register_queue(struct request_queue *q, struct device *parent, /* * we need a proper transport to send commands, not a stacked device */ - if (!queue_is_rq_based(q)) + if (!queue_is_mq(q)) return 0; bcd = &q->bsg_dev; diff --git a/block/elevator.c b/block/elevator.c index 796436270682..f05e90d4e695 100644 --- a/block/elevator.c +++ b/block/elevator.c @@ -667,7 +667,7 @@ static int __elevator_change(struct request_queue *q, const char *name) /* * Special case for mq, turn off scheduling */ - if (q->mq_ops && !strncmp(name, "none", 4)) + if (!strncmp(name, "none", 4)) return elevator_switch(q, NULL); strlcpy(elevator_name, name, sizeof(elevator_name)); @@ -685,8 +685,7 @@ static int __elevator_change(struct request_queue *q, const char *name) static inline bool elv_support_iosched(struct request_queue *q) { - if (q->mq_ops && q->tag_set && (q->tag_set->flags & - BLK_MQ_F_NO_SCHED)) + if (q->tag_set && (q->tag_set->flags & BLK_MQ_F_NO_SCHED)) return false; return true; } @@ -696,7 +695,7 @@ ssize_t elv_iosched_store(struct request_queue *q, const char *name, { int ret; - if (!q->mq_ops || !elv_support_iosched(q)) + if (!queue_is_mq(q) || !elv_support_iosched(q)) return count; ret = __elevator_change(q, name); @@ -713,7 +712,7 @@ ssize_t elv_iosched_show(struct request_queue *q, char *name) struct elevator_type *__e; int len = 0; - if (!queue_is_rq_based(q)) + if (!queue_is_mq(q)) return sprintf(name, "none\n"); if (!q->elevator) @@ -732,7 +731,7 @@ ssize_t elv_iosched_show(struct request_queue *q, char *name) } spin_unlock(&elv_list_lock); - if (q->mq_ops && q->elevator) + if (q->elevator) len += sprintf(name+len, "none"); len += sprintf(len+name, "\n"); diff --git a/block/genhd.c b/block/genhd.c index cff6bdf27226..0145bcb0cc76 100644 --- a/block/genhd.c +++ b/block/genhd.c @@ -47,7 +47,7 @@ static void disk_release_events(struct gendisk *disk); void part_inc_in_flight(struct request_queue *q, struct hd_struct *part, int rw) { - if (q->mq_ops) + if (queue_is_mq(q)) return; atomic_inc(&part->in_flight[rw]); @@ -57,7 +57,7 @@ void part_inc_in_flight(struct request_queue *q, struct hd_struct *part, int rw) void part_dec_in_flight(struct request_queue *q, struct hd_struct *part, int rw) { - if (q->mq_ops) + if (queue_is_mq(q)) return; atomic_dec(&part->in_flight[rw]); @@ -68,7 +68,7 @@ void part_dec_in_flight(struct request_queue *q, struct hd_struct *part, int rw) void part_in_flight(struct request_queue *q, struct hd_struct *part, unsigned int inflight[2]) { - if (q->mq_ops) { + if (queue_is_mq(q)) { blk_mq_in_flight(q, part, inflight); return; } @@ -85,7 +85,7 @@ void part_in_flight(struct request_queue *q, struct hd_struct *part, void part_in_flight_rw(struct request_queue *q, struct hd_struct *part, unsigned int inflight[2]) { - if (q->mq_ops) { + if (queue_is_mq(q)) { blk_mq_in_flight_rw(q, part, inflight); return; } diff --git a/drivers/md/dm-rq.c b/drivers/md/dm-rq.c index 7cd36e4d1310..1f1fe9a618ea 100644 --- a/drivers/md/dm-rq.c +++ b/drivers/md/dm-rq.c @@ -43,7 +43,7 @@ static unsigned dm_get_blk_mq_queue_depth(void) int dm_request_based(struct mapped_device *md) { - return queue_is_rq_based(md->queue); + return queue_is_mq(md->queue); } void dm_start_queue(struct request_queue *q) diff --git a/drivers/md/dm-table.c b/drivers/md/dm-table.c index 9038c302d5c2..844f7d0f2ef8 100644 --- a/drivers/md/dm-table.c +++ b/drivers/md/dm-table.c @@ -919,12 +919,12 @@ static int device_is_rq_based(struct dm_target *ti, struct dm_dev *dev, struct request_queue *q = bdev_get_queue(dev->bdev); struct verify_rq_based_data *v = data; - if (q->mq_ops) + if (queue_is_mq(q)) v->mq_count++; else v->sq_count++; - return queue_is_rq_based(q); + return queue_is_mq(q); } static int dm_table_determine_type(struct dm_table *t) diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h index 1d185f1fc333..41aaa05e42c1 100644 --- a/include/linux/blkdev.h +++ b/include/linux/blkdev.h @@ -656,11 +656,7 @@ static inline bool blk_account_rq(struct request *rq) #define rq_data_dir(rq) (op_is_write(req_op(rq)) ? WRITE : READ) -/* - * Driver can handle struct request, if it either has an old style - * request_fn defined, or is blk-mq based. - */ -static inline bool queue_is_rq_based(struct request_queue *q) +static inline bool queue_is_mq(struct request_queue *q) { return q->mq_ops; } From patchwork Thu Nov 15 19:51:27 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 10684967 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id CFAEB13BB for ; Thu, 15 Nov 2018 19:51:49 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C2A6D2D14F for ; Thu, 15 Nov 2018 19:51:49 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id B65072D16E; Thu, 15 Nov 2018 19:51:49 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 25ED52D14F for ; Thu, 15 Nov 2018 19:51:49 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726941AbeKPGBA (ORCPT ); Fri, 16 Nov 2018 01:01:00 -0500 Received: from mail-pf1-f195.google.com ([209.85.210.195]:32972 "EHLO mail-pf1-f195.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725869AbeKPGA7 (ORCPT ); Fri, 16 Nov 2018 01:00:59 -0500 Received: by mail-pf1-f195.google.com with SMTP id v68-v6so10203136pfk.0 for ; Thu, 15 Nov 2018 11:51:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=0CsOxzWNiXiFSdVT8Ap2AwFn+e3sW43Qpf/ppSkfHZc=; b=CIEAgpUwfncK5yL71e0vdNcIrM3V78MAvcqc5PPBYeEwrA1wNemtYJtn46/sviuHB9 ji0od33E7vwTJ4s/t8BK+dWWh0pOtMvQ1jJGlMCKuXrFQ8h9PladOZq9cfaU9kS5X4Sg XzvncLlMuCDcwpCzE2rDU4Vj2kzZZU2aRWjDhFmv92IpCgSA46RaojFxm9p+lsNK/OZY GZiDW0IuLurz/MOuO8AXinFJGa+OU5AnYR0W8pfN/dLyIXZQ1WLwzpNK0ooEIMhD6zwO 1aQyF1oL1sbRHHUwonUzp37xl+52rO4qY34OWK9SOWZ/dk+C8yRqIpZCQcCH8HSWWIQx ND0Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=0CsOxzWNiXiFSdVT8Ap2AwFn+e3sW43Qpf/ppSkfHZc=; b=q/mg8YpLI67f9pRYJkYn1kYudPHpo7szxgJKSvj0J78zJCZg8bJv/CijgTKcq/+rXc 5NmYnrUvOQ/MMbOq42JM8e0JmRmaVuuvkf1hU2g5FYC6HH3omeT/D0+p0G/VAXbuvrpU 0+m7Wt/JskEPuVM9VgsVhYj1awfVhbVpjG+jviwugdDF2vuAS1+WWzZuDeNRVjmqbfuY C6CahmV368D86/r061hb9WYSqLhkF4L8BcMfyrQXDCAbFnoUT9GzOuzSj9XUHOQ76z4g y9Uy4baD18DljaKqjtb0fkj5VNDxx6UN/UHk3oYAWiPa7Qgd5LgxAr77rpMkmLYLceyq mZRg== X-Gm-Message-State: AGRZ1gI+JPF4j/tvexAx/rhptaTmUh0qTtjYXnF58OOt5876arN9veR3 gMFJTkqUPqW6q/guBzjqLxeLcDbE8S8= X-Google-Smtp-Source: AJdET5ew1XUfMbFTJpKUgMF13MdNwtlrs+RU3k+uiIDYq2/ETXQVW9SCesGN/4Gl/5oIHvT5uIrsCQ== X-Received: by 2002:a63:db02:: with SMTP id e2mr7112727pgg.419.1542311506944; Thu, 15 Nov 2018 11:51:46 -0800 (PST) Received: from x1.localdomain ([2620:10d:c090:180::1:7813]) by smtp.gmail.com with ESMTPSA id d68-v6sm30670766pfa.80.2018.11.15.11.51.44 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 15 Nov 2018 11:51:45 -0800 (PST) From: Jens Axboe To: linux-block@vger.kernel.org Cc: Jens Axboe , Josef Bacik Subject: [PATCH 03/11] blk-rq-qos: inline check for q->rq_qos functions Date: Thu, 15 Nov 2018 12:51:27 -0700 Message-Id: <20181115195135.22812-4-axboe@kernel.dk> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181115195135.22812-1-axboe@kernel.dk> References: <20181115195135.22812-1-axboe@kernel.dk> Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Put the short code in the fast path, where we don't have any functions attached to the queue. This minimizes the impact on the hot path in the core code. Cc: Josef Bacik Signed-off-by: Jens Axboe Reviewed-by: Christoph Hellwig --- block/blk-rq-qos.c | 63 +++++++++++++++++++++------------------------- block/blk-rq-qos.h | 59 +++++++++++++++++++++++++++++++++++++------ 2 files changed, 80 insertions(+), 42 deletions(-) diff --git a/block/blk-rq-qos.c b/block/blk-rq-qos.c index f8a4d3fbb98c..80f603b76f61 100644 --- a/block/blk-rq-qos.c +++ b/block/blk-rq-qos.c @@ -27,74 +27,67 @@ bool rq_wait_inc_below(struct rq_wait *rq_wait, unsigned int limit) return atomic_inc_below(&rq_wait->inflight, limit); } -void rq_qos_cleanup(struct request_queue *q, struct bio *bio) +void __rq_qos_cleanup(struct rq_qos *rqos, struct bio *bio) { - struct rq_qos *rqos; - - for (rqos = q->rq_qos; rqos; rqos = rqos->next) { + do { if (rqos->ops->cleanup) rqos->ops->cleanup(rqos, bio); - } + rqos = rqos->next; + } while (rqos); } -void rq_qos_done(struct request_queue *q, struct request *rq) +void __rq_qos_done(struct rq_qos *rqos, struct request *rq) { - struct rq_qos *rqos; - - for (rqos = q->rq_qos; rqos; rqos = rqos->next) { + do { if (rqos->ops->done) rqos->ops->done(rqos, rq); - } + rqos = rqos->next; + } while (rqos); } -void rq_qos_issue(struct request_queue *q, struct request *rq) +void __rq_qos_issue(struct rq_qos *rqos, struct request *rq) { - struct rq_qos *rqos; - - for(rqos = q->rq_qos; rqos; rqos = rqos->next) { + do { if (rqos->ops->issue) rqos->ops->issue(rqos, rq); - } + rqos = rqos->next; + } while (rqos); } -void rq_qos_requeue(struct request_queue *q, struct request *rq) +void __rq_qos_requeue(struct rq_qos *rqos, struct request *rq) { - struct rq_qos *rqos; - - for(rqos = q->rq_qos; rqos; rqos = rqos->next) { + do { if (rqos->ops->requeue) rqos->ops->requeue(rqos, rq); - } + rqos = rqos->next; + } while (rqos); } -void rq_qos_throttle(struct request_queue *q, struct bio *bio) +void __rq_qos_throttle(struct rq_qos *rqos, struct bio *bio) { - struct rq_qos *rqos; - - for(rqos = q->rq_qos; rqos; rqos = rqos->next) { + do { if (rqos->ops->throttle) rqos->ops->throttle(rqos, bio); - } + rqos = rqos->next; + } while (rqos); } -void rq_qos_track(struct request_queue *q, struct request *rq, struct bio *bio) +void __rq_qos_track(struct rq_qos *rqos, struct request *rq, struct bio *bio) { - struct rq_qos *rqos; - - for(rqos = q->rq_qos; rqos; rqos = rqos->next) { + do { if (rqos->ops->track) rqos->ops->track(rqos, rq, bio); - } + rqos = rqos->next; + } while (rqos); } -void rq_qos_done_bio(struct request_queue *q, struct bio *bio) +void __rq_qos_done_bio(struct rq_qos *rqos, struct bio *bio) { - struct rq_qos *rqos; - - for(rqos = q->rq_qos; rqos; rqos = rqos->next) { + do { if (rqos->ops->done_bio) rqos->ops->done_bio(rqos, bio); - } + rqos = rqos->next; + } while (rqos); } /* diff --git a/block/blk-rq-qos.h b/block/blk-rq-qos.h index b6b11d496007..6e09e98b93ea 100644 --- a/block/blk-rq-qos.h +++ b/block/blk-rq-qos.h @@ -98,12 +98,57 @@ void rq_depth_scale_up(struct rq_depth *rqd); void rq_depth_scale_down(struct rq_depth *rqd, bool hard_throttle); bool rq_depth_calc_max_depth(struct rq_depth *rqd); -void rq_qos_cleanup(struct request_queue *, struct bio *); -void rq_qos_done(struct request_queue *, struct request *); -void rq_qos_issue(struct request_queue *, struct request *); -void rq_qos_requeue(struct request_queue *, struct request *); -void rq_qos_done_bio(struct request_queue *q, struct bio *bio); -void rq_qos_throttle(struct request_queue *, struct bio *); -void rq_qos_track(struct request_queue *q, struct request *, struct bio *); +void __rq_qos_cleanup(struct rq_qos *rqos, struct bio *bio); +void __rq_qos_done(struct rq_qos *rqos, struct request *rq); +void __rq_qos_issue(struct rq_qos *rqos, struct request *rq); +void __rq_qos_requeue(struct rq_qos *rqos, struct request *rq); +void __rq_qos_throttle(struct rq_qos *rqos, struct bio *bio); +void __rq_qos_track(struct rq_qos *rqos, struct request *rq, struct bio *bio); +void __rq_qos_done_bio(struct rq_qos *rqos, struct bio *bio); + +static inline void rq_qos_cleanup(struct request_queue *q, struct bio *bio) +{ + if (q->rq_qos) + __rq_qos_cleanup(q->rq_qos, bio); +} + +static inline void rq_qos_done(struct request_queue *q, struct request *rq) +{ + if (q->rq_qos) + __rq_qos_done(q->rq_qos, rq); +} + +static inline void rq_qos_issue(struct request_queue *q, struct request *rq) +{ + if (q->rq_qos) + __rq_qos_issue(q->rq_qos, rq); +} + +static inline void rq_qos_requeue(struct request_queue *q, struct request *rq) +{ + if (q->rq_qos) + __rq_qos_requeue(q->rq_qos, rq); +} + +static inline void rq_qos_done_bio(struct request_queue *q, struct bio *bio) +{ + if (q->rq_qos) + __rq_qos_done_bio(q->rq_qos, bio); +} + +static inline void rq_qos_throttle(struct request_queue *q, struct bio *bio) +{ + if (q->rq_qos) + __rq_qos_throttle(q->rq_qos, bio); +} + +static inline void rq_qos_track(struct request_queue *q, struct request *rq, + struct bio *bio) +{ + if (q->rq_qos) + __rq_qos_track(q->rq_qos, rq, bio); +} + void rq_qos_exit(struct request_queue *); + #endif From patchwork Thu Nov 15 19:51:28 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 10684969 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id C097F14D6 for ; Thu, 15 Nov 2018 19:51:51 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id AE0152D0FE for ; Thu, 15 Nov 2018 19:51:51 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 9EF582CC9B; Thu, 15 Nov 2018 19:51:51 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0798E2D14E for ; Thu, 15 Nov 2018 19:51:51 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727651AbeKPGBC (ORCPT ); Fri, 16 Nov 2018 01:01:02 -0500 Received: from mail-pl1-f196.google.com ([209.85.214.196]:44904 "EHLO mail-pl1-f196.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725869AbeKPGBB (ORCPT ); Fri, 16 Nov 2018 01:01:01 -0500 Received: by mail-pl1-f196.google.com with SMTP id s5-v6so9953185plq.11 for ; Thu, 15 Nov 2018 11:51:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=MP5P1AyoI9WZaMp7Yr6iSxvqTi6dTbNRiicBr3AUvQI=; b=zqcKF6d/G2RY0eSaetQhZ4NiwwGA+kOFBmETwTDrKAGeM9bRGMj6b8ZLV5vtE5yyuI NxqUpXWPKB5PQZDvBUZxqWEQAi9Hwwtzl9HfcKTelEk/WrJ+qayfSLIvSW/L+FoIizID nMc+tyC1bFsEviQ2BmMPTgBTuj9BsLacYezNYzpuGnX3Hw2rjvCeOaY8TMnyhqMxP5X5 uxEc2HISp6pEMAemRqFiiDrAwwXPKCZZ7xFWH4+lEYKJguWehoHm2BeToprEvHEt7PVO M4WqkvNAon0vYZBUCm2pf3OuAExQ746SjBp9Zb2flv18w28SX+B3HR44z1404LJl92bB WfRg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=MP5P1AyoI9WZaMp7Yr6iSxvqTi6dTbNRiicBr3AUvQI=; b=OM3Qfl5SZ33HJn6CTcSaUQQ0D4grrjdODfusq5HI4xsUxOmIi06pbL0/xrj7zPPH4k GToZFBWj8u4YUdGVvPGj18ZjmK1CmSRXuXJR/ejMCGVb/H+zg3cTaprGvGy8fGDg8maM yWhSH4jkjeTPI2p3CMOSQ8oJZV8YEhmSbrxhq6f6beJwRpbheYy7TsYA5bc/voLjF576 VeSXppVundgxTbWsWvBPhEoxqI/svL31r4TAF3D6GSdwIYDwcGph7WAoF8z/jn8cwDHz NzeeWCBML+iYpuSQX/Z5PlHeoYo3MjKDBUHz1wyLBoiJNi0OQduo86Phvxc/iLF8NMOW JCqQ== X-Gm-Message-State: AGRZ1gLDzsWrDNV6NLYxPDM6aXAF7+XzxCRIFPgD2y7gNnf8uMnZji85 8oJX5Bz+uLxJPzr13hpDpZ7b1rV8jv8= X-Google-Smtp-Source: AJdET5dme0s+6Zn5RRzTfdeWkoif/Qq0pM+yH+vSIIuv7YOBG6HBxjpDxVUBb/XkV3FpN69e7AemVw== X-Received: by 2002:a17:902:2ec5:: with SMTP id r63-v6mr7839372plb.330.1542311508808; Thu, 15 Nov 2018 11:51:48 -0800 (PST) Received: from x1.localdomain ([2620:10d:c090:180::1:7813]) by smtp.gmail.com with ESMTPSA id d68-v6sm30670766pfa.80.2018.11.15.11.51.47 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 15 Nov 2018 11:51:47 -0800 (PST) From: Jens Axboe To: linux-block@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 04/11] block: avoid ordered task state change for polled IO Date: Thu, 15 Nov 2018 12:51:28 -0700 Message-Id: <20181115195135.22812-5-axboe@kernel.dk> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181115195135.22812-1-axboe@kernel.dk> References: <20181115195135.22812-1-axboe@kernel.dk> Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Ensure that writes to the dio/bio waiter field are ordered correctly. With the smp_rmb() before the READ_ONCE() check, we should be able to use a more relaxed ordering for the task state setting. We don't need a heavier barrier on the wakeup side after writing the waiter field, since we either going to be in the task we care about, or go through wake_up_process() which implies a strong enough barrier. For the core poll helper, the task state setting don't need to imply any atomics, as it's the current task itself that is being modified and we're not going to sleep. Signed-off-by: Jens Axboe --- block/blk-mq.c | 4 ++-- fs/block_dev.c | 9 +++++++-- fs/iomap.c | 4 +++- mm/page_io.c | 4 +++- 4 files changed, 15 insertions(+), 6 deletions(-) diff --git a/block/blk-mq.c b/block/blk-mq.c index 32b246ed44c0..7fc4abb4cc36 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -3331,12 +3331,12 @@ static bool __blk_mq_poll(struct blk_mq_hw_ctx *hctx, struct request *rq) ret = q->mq_ops->poll(hctx, rq->tag); if (ret > 0) { hctx->poll_success++; - set_current_state(TASK_RUNNING); + __set_current_state(TASK_RUNNING); return true; } if (signal_pending_state(state, current)) - set_current_state(TASK_RUNNING); + __set_current_state(TASK_RUNNING); if (current->state == TASK_RUNNING) return true; diff --git a/fs/block_dev.c b/fs/block_dev.c index c039abfb2052..5b754f84c814 100644 --- a/fs/block_dev.c +++ b/fs/block_dev.c @@ -237,9 +237,12 @@ __blkdev_direct_IO_simple(struct kiocb *iocb, struct iov_iter *iter, qc = submit_bio(&bio); for (;;) { - set_current_state(TASK_UNINTERRUPTIBLE); + __set_current_state(TASK_UNINTERRUPTIBLE); + + smp_rmb(); if (!READ_ONCE(bio.bi_private)) break; + if (!(iocb->ki_flags & IOCB_HIPRI) || !blk_poll(bdev_get_queue(bdev), qc)) io_schedule(); @@ -403,7 +406,9 @@ __blkdev_direct_IO(struct kiocb *iocb, struct iov_iter *iter, int nr_pages) return -EIOCBQUEUED; for (;;) { - set_current_state(TASK_UNINTERRUPTIBLE); + __set_current_state(TASK_UNINTERRUPTIBLE); + + smp_rmb(); if (!READ_ONCE(dio->waiter)) break; diff --git a/fs/iomap.c b/fs/iomap.c index f61d13dfdf09..3373ea4984d9 100644 --- a/fs/iomap.c +++ b/fs/iomap.c @@ -1888,7 +1888,9 @@ iomap_dio_rw(struct kiocb *iocb, struct iov_iter *iter, return -EIOCBQUEUED; for (;;) { - set_current_state(TASK_UNINTERRUPTIBLE); + __set_current_state(TASK_UNINTERRUPTIBLE); + + smp_rmb(); if (!READ_ONCE(dio->submit.waiter)) break; diff --git a/mm/page_io.c b/mm/page_io.c index d4d1c89bcddd..008f6d00c47c 100644 --- a/mm/page_io.c +++ b/mm/page_io.c @@ -405,7 +405,9 @@ int swap_readpage(struct page *page, bool synchronous) bio_get(bio); qc = submit_bio(bio); while (synchronous) { - set_current_state(TASK_UNINTERRUPTIBLE); + __set_current_state(TASK_UNINTERRUPTIBLE); + + smp_rmb(); if (!READ_ONCE(bio->bi_private)) break; From patchwork Thu Nov 15 19:51:29 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 10684971 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 64E3414D6 for ; Thu, 15 Nov 2018 19:51:53 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 57A362D0DB for ; Thu, 15 Nov 2018 19:51:53 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 55D1A2D169; Thu, 15 Nov 2018 19:51:53 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B256D2D0DB for ; Thu, 15 Nov 2018 19:51:52 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1725869AbeKPGBD (ORCPT ); Fri, 16 Nov 2018 01:01:03 -0500 Received: from mail-pg1-f194.google.com ([209.85.215.194]:44107 "EHLO mail-pg1-f194.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725781AbeKPGBD (ORCPT ); Fri, 16 Nov 2018 01:01:03 -0500 Received: by mail-pg1-f194.google.com with SMTP id t13so816187pgr.11 for ; Thu, 15 Nov 2018 11:51:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=FMTs3C4r9EHC31s/sU7DDsf/1xn5v8SxFDlkXY61qcI=; b=Juge2bx+u+W9xtKjSrjv7BrE6t9ix65K3/FhVQABXyl6ukFxi4L3ILMwwEJ1SrOoyL Kk2vQmG4OpWCquoHikM2NoZl3NO26t03aXawqH2EI0/pGYacpZcONoQ9uEez+7oXBgh1 a9pBXgUXoBJfqAs/JxBGaJQmeQ+CrWev29wAT8KSXvaCyB8D071PrU7bmrYyiEz3Xfp8 2lcbJ90EQeyXczXzFaDlC87qVbF7H2ZwJl5YYZBs3XxT/irK+zwjTd1B+4KEhexR+gfR McW3QtUqmsTSKXXScwCM/W9srJFUM+eiYQGDmAfZM2C6ohaXzROdEwWMvFSyVCeHhe84 /Edw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=FMTs3C4r9EHC31s/sU7DDsf/1xn5v8SxFDlkXY61qcI=; b=Q9k4gQtyzFeLjIVnabLJDJIyw5n+cAN4Bq9W6EpCqeDxeNvPtQAMvjWZMwNV/O7qmD /zqAwfT4fT48PMbL20O2wenhSluNhOWG8c7SEbli5xSQL4FPKjLD+IRhS5J7VO2DlqWf hBiMWcvjhT8txp1u+IXrh6km7DS1KmWfuR6qA421+FtEhnY8Cd4XnS3gjP5SZ1TByM8x g8UL2ITxJTGW1i432G7/p4XGyXBksYSCk3x1srtXNBODjyfnqImclP8WlmLztnBJ6fik XJ2SZXbPApiaLUcFa06bO1We9/rSjwueMjwPKgVJqSC0d/4ibv2JXo5fHI4WiaH2Bj2d 7GMQ== X-Gm-Message-State: AGRZ1gIf6DHCdqLGLpAL5A7ErC6SUotpB+xOCC5wICygsd0a9C9wmSjx 1qa1/cCC8llg4G+deYd8FGsxvCJNcyY= X-Google-Smtp-Source: AJdET5cEKlTcH7fQETz1Ob4oMCJEpgdf7hlZqOt45OCd2/SraTspObtPXltJbROABrBAK0eJfOLFfg== X-Received: by 2002:a62:704a:: with SMTP id l71-v6mr7832104pfc.68.1542311510912; Thu, 15 Nov 2018 11:51:50 -0800 (PST) Received: from x1.localdomain ([2620:10d:c090:180::1:7813]) by smtp.gmail.com with ESMTPSA id d68-v6sm30670766pfa.80.2018.11.15.11.51.48 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 15 Nov 2018 11:51:49 -0800 (PST) From: Jens Axboe To: linux-block@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 05/11] block: add polled wakeup task helper Date: Thu, 15 Nov 2018 12:51:29 -0700 Message-Id: <20181115195135.22812-6-axboe@kernel.dk> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181115195135.22812-1-axboe@kernel.dk> References: <20181115195135.22812-1-axboe@kernel.dk> Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP If we're polling for IO on a device that doesn't use interrupts, then IO completion loop (and wake of task) is done by submitting task itself. If that is the case, then we don't need to enter the wake_up_process() function, we can simply mark ourselves as TASK_RUNNING. Signed-off-by: Jens Axboe Reviewed-by: Christoph Hellwig --- fs/block_dev.c | 4 ++-- fs/iomap.c | 2 +- include/linux/blkdev.h | 13 +++++++++++++ mm/page_io.c | 2 +- 4 files changed, 17 insertions(+), 4 deletions(-) diff --git a/fs/block_dev.c b/fs/block_dev.c index 5b754f84c814..0ed9be8906a8 100644 --- a/fs/block_dev.c +++ b/fs/block_dev.c @@ -181,7 +181,7 @@ static void blkdev_bio_end_io_simple(struct bio *bio) struct task_struct *waiter = bio->bi_private; WRITE_ONCE(bio->bi_private, NULL); - wake_up_process(waiter); + blk_wake_io_task(waiter); } static ssize_t @@ -308,7 +308,7 @@ static void blkdev_bio_end_io(struct bio *bio) struct task_struct *waiter = dio->waiter; WRITE_ONCE(dio->waiter, NULL); - wake_up_process(waiter); + blk_wake_io_task(waiter); } } diff --git a/fs/iomap.c b/fs/iomap.c index 3373ea4984d9..38c9bc63296a 100644 --- a/fs/iomap.c +++ b/fs/iomap.c @@ -1525,7 +1525,7 @@ static void iomap_dio_bio_end_io(struct bio *bio) if (dio->wait_for_completion) { struct task_struct *waiter = dio->submit.waiter; WRITE_ONCE(dio->submit.waiter, NULL); - wake_up_process(waiter); + blk_wake_io_task(waiter); } else if (dio->flags & IOMAP_DIO_WRITE) { struct inode *inode = file_inode(dio->iocb->ki_filp); diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h index 41aaa05e42c1..91c44f7a7f62 100644 --- a/include/linux/blkdev.h +++ b/include/linux/blkdev.h @@ -1772,4 +1772,17 @@ static inline int blkdev_issue_flush(struct block_device *bdev, gfp_t gfp_mask, #endif /* CONFIG_BLOCK */ +static inline void blk_wake_io_task(struct task_struct *waiter) +{ + /* + * If we're polling, the task itself is doing the completions. For + * that case, we don't need to signal a wakeup, it's enough to just + * mark us as RUNNING. + */ + if (waiter == current) + __set_current_state(TASK_RUNNING); + else + wake_up_process(waiter); +} + #endif diff --git a/mm/page_io.c b/mm/page_io.c index 008f6d00c47c..f277459db805 100644 --- a/mm/page_io.c +++ b/mm/page_io.c @@ -140,7 +140,7 @@ static void end_swap_bio_read(struct bio *bio) unlock_page(page); WRITE_ONCE(bio->bi_private, NULL); bio_put(bio); - wake_up_process(waiter); + blk_wake_io_task(waiter); put_task_struct(waiter); } From patchwork Thu Nov 15 19:51:30 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 10684973 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id D103913BB for ; Thu, 15 Nov 2018 19:51:54 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C3D372D158 for ; Thu, 15 Nov 2018 19:51:54 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id C1DB32D132; Thu, 15 Nov 2018 19:51:54 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 5056C2D14F for ; Thu, 15 Nov 2018 19:51:54 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729077AbeKPGBF (ORCPT ); Fri, 16 Nov 2018 01:01:05 -0500 Received: from mail-pl1-f193.google.com ([209.85.214.193]:34254 "EHLO mail-pl1-f193.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725781AbeKPGBF (ORCPT ); Fri, 16 Nov 2018 01:01:05 -0500 Received: by mail-pl1-f193.google.com with SMTP id f12-v6so9984247plo.1 for ; Thu, 15 Nov 2018 11:51:53 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=DEomzWGyNQSltl7n4uFzYYpxk2U93JIuvGhRDl2wTS4=; b=WAHT/CdtxhpfrRLOwqLki37Sd1i9P8q4k9QyLqjTCGobXKSJLt7o7Ivh3eozwL/u5O KmCX3gsMLO4Ut8ThlYX6Q4bx1oEeiZkR/DjJoeMetfL8aMnIZZ01jdMZu+2pD2byj8E9 +MxHbIRF8qCKSq0ecWbJEeAIRlj2VdFnwCOV81BsL7jZg5bNvq+/QY19Y+vyBEoMnyNL TWtBq6uJBBaehw+EMiH8Rcq2foMpob+eKZTkMdLDrQsZBHYsmtT5zVQch2vJRVeFZjyR DnciBAVOJ6Chw0oRvWOLBD3JE8YSFn8WRUVRoEcpkvFl2K/ZX/wv4su98siwgZaaFZFd CQZQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=DEomzWGyNQSltl7n4uFzYYpxk2U93JIuvGhRDl2wTS4=; b=uOPWt7eL3fm8tWEEDkO6T4JJG1WbRPiKHXvwAydnhCmHId2XGHYHHOBJZ/fUqM9vOA HzTJRaxpFNRTqmUoJyr7fdHH4uik3dk5qopqRl8MIB5E9Z0INYXWtAdATaWbJ/nO0oD0 PPKtQp5THIMfXOvJLmimBuYy9u+TpAQMnpFI5Mzw6xdeXVngjU1n0i3FUbmpD9Zym7lU lgkv1ce9D0FbGFtssJU2f3aILh9GvqLa9CBSoEmZU67yPTSw5EQi+PXFjTwJAX+fYiIU n/keaykxdu0PJigxo+Dama3mPbzG8TKbuAk1yfT8mzt6aylFDX0A/NHH2BSoCrxAVTgE yT2w== X-Gm-Message-State: AGRZ1gJkhd92GXLxSm3Ga41EgEzsCZAq8kY/frwsDRndli4AVjwL5Pti sm2iAwDoaTZ+3qFMLpat+B73Vr7oDN4= X-Google-Smtp-Source: AJdET5c76YU2UKow5wsiosXlK1NI9obmubFXNRBl49ArSazLKvc0T4YcVhIsC29pjlpRcxak46orvA== X-Received: by 2002:a17:902:70cb:: with SMTP id l11-v6mr7884739plt.30.1542311512716; Thu, 15 Nov 2018 11:51:52 -0800 (PST) Received: from x1.localdomain ([2620:10d:c090:180::1:7813]) by smtp.gmail.com with ESMTPSA id d68-v6sm30670766pfa.80.2018.11.15.11.51.50 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 15 Nov 2018 11:51:51 -0800 (PST) From: Jens Axboe To: linux-block@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 06/11] block: have ->poll_fn() return number of entries polled Date: Thu, 15 Nov 2018 12:51:30 -0700 Message-Id: <20181115195135.22812-7-axboe@kernel.dk> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181115195135.22812-1-axboe@kernel.dk> References: <20181115195135.22812-1-axboe@kernel.dk> Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP We currently only really support sync poll, ie poll with 1 IO in flight. This prepares us for supporting async poll. Note that the returned value isn't necessarily 100% accurate. If poll races with IRQ completion, we assume that the fact that the task is now runnable means we found at least one entry. In reality it could be more than 1, or not even 1. This is fine, the caller will just need to take this into account. Signed-off-by: Jens Axboe --- block/blk-mq.c | 18 +++++++++--------- drivers/nvme/host/multipath.c | 4 ++-- include/linux/blkdev.h | 2 +- 3 files changed, 12 insertions(+), 12 deletions(-) diff --git a/block/blk-mq.c b/block/blk-mq.c index 7fc4abb4cc36..52b1c97cd7c6 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -38,7 +38,7 @@ #include "blk-mq-sched.h" #include "blk-rq-qos.h" -static bool blk_mq_poll(struct request_queue *q, blk_qc_t cookie); +static int blk_mq_poll(struct request_queue *q, blk_qc_t cookie); static void blk_mq_poll_stats_start(struct request_queue *q); static void blk_mq_poll_stats_fn(struct blk_stat_callback *cb); @@ -3305,7 +3305,7 @@ static bool blk_mq_poll_hybrid_sleep(struct request_queue *q, return true; } -static bool __blk_mq_poll(struct blk_mq_hw_ctx *hctx, struct request *rq) +static int __blk_mq_poll(struct blk_mq_hw_ctx *hctx, struct request *rq) { struct request_queue *q = hctx->queue; long state; @@ -3318,7 +3318,7 @@ static bool __blk_mq_poll(struct blk_mq_hw_ctx *hctx, struct request *rq) * straight to the busy poll loop. */ if (blk_mq_poll_hybrid_sleep(q, hctx, rq)) - return true; + return 1; hctx->poll_considered++; @@ -3332,30 +3332,30 @@ static bool __blk_mq_poll(struct blk_mq_hw_ctx *hctx, struct request *rq) if (ret > 0) { hctx->poll_success++; __set_current_state(TASK_RUNNING); - return true; + return ret; } if (signal_pending_state(state, current)) __set_current_state(TASK_RUNNING); if (current->state == TASK_RUNNING) - return true; + return 1; if (ret < 0) break; cpu_relax(); } __set_current_state(TASK_RUNNING); - return false; + return 0; } -static bool blk_mq_poll(struct request_queue *q, blk_qc_t cookie) +static int blk_mq_poll(struct request_queue *q, blk_qc_t cookie) { struct blk_mq_hw_ctx *hctx; struct request *rq; if (!test_bit(QUEUE_FLAG_POLL, &q->queue_flags)) - return false; + return 0; hctx = q->queue_hw_ctx[blk_qc_t_to_queue_num(cookie)]; if (!blk_qc_t_is_internal(cookie)) @@ -3369,7 +3369,7 @@ static bool blk_mq_poll(struct request_queue *q, blk_qc_t cookie) * so we should be safe with just the NULL check. */ if (!rq) - return false; + return 0; } return __blk_mq_poll(hctx, rq); diff --git a/drivers/nvme/host/multipath.c b/drivers/nvme/host/multipath.c index b82b0d3ca39a..65539c8df11d 100644 --- a/drivers/nvme/host/multipath.c +++ b/drivers/nvme/host/multipath.c @@ -220,11 +220,11 @@ static blk_qc_t nvme_ns_head_make_request(struct request_queue *q, return ret; } -static bool nvme_ns_head_poll(struct request_queue *q, blk_qc_t qc) +static int nvme_ns_head_poll(struct request_queue *q, blk_qc_t qc) { struct nvme_ns_head *head = q->queuedata; struct nvme_ns *ns; - bool found = false; + int found = 0; int srcu_idx; srcu_idx = srcu_read_lock(&head->srcu); diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h index 91c44f7a7f62..e96dc16ef8aa 100644 --- a/include/linux/blkdev.h +++ b/include/linux/blkdev.h @@ -283,7 +283,7 @@ static inline unsigned short req_get_ioprio(struct request *req) struct blk_queue_ctx; typedef blk_qc_t (make_request_fn) (struct request_queue *q, struct bio *bio); -typedef bool (poll_q_fn) (struct request_queue *q, blk_qc_t); +typedef int (poll_q_fn) (struct request_queue *q, blk_qc_t); struct bio_vec; typedef int (dma_drain_needed_fn)(struct request *); From patchwork Thu Nov 15 19:51:31 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 10684975 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 2E85513BB for ; Thu, 15 Nov 2018 19:51:57 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 1DD462D0D0 for ; Thu, 15 Nov 2018 19:51:57 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 11FF82D08F; Thu, 15 Nov 2018 19:51:57 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 8D3CE2D14C for ; Thu, 15 Nov 2018 19:51:56 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729190AbeKPGBH (ORCPT ); Fri, 16 Nov 2018 01:01:07 -0500 Received: from mail-pl1-f194.google.com ([209.85.214.194]:39914 "EHLO mail-pl1-f194.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725781AbeKPGBH (ORCPT ); Fri, 16 Nov 2018 01:01:07 -0500 Received: by mail-pl1-f194.google.com with SMTP id b5-v6so9974084pla.6 for ; Thu, 15 Nov 2018 11:51:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=tiaZa8WSa4N6JxUKx5nvzdX+JfaYEZWZ99+zxTGrJas=; b=wlO24RLQtEyMJExogq+QI6U0bCrb7WEWIynPYO8hC2LM7GuU8xDogNOBd3zukv+ybx mCpQud80cb8Ig6T3yvIJGY+kYZ2gJx4wV7jaQ1vY/BT4Gu46UTL9pe+yZCduoFRDx58a duhJpOFctRKDipC2fj1jT/+B/7IQS27Tl1oJNXX1wGdfhaWBU/xVRpI67/xnL2O/7P8/ cVprf35LqVscNhL2nIkjzV0BVhWxhQ/dyrQrX97qEAM7A19uzQodOsqDKI9c2dNqWKyR zyTwIzprGzjq2IZXh1sQ7i6MdXGPDRz4mqD42dukkx6SQIq+lyKYhZCs1qrOJo7bvMCo BAdQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=tiaZa8WSa4N6JxUKx5nvzdX+JfaYEZWZ99+zxTGrJas=; b=XRo+o0eGPLM5rKxKtnASPrOYC8eoyE4iK0rVWDe9ajEDuaQ/1UbWhluWOWRfMEbxuv dEIe4bzwv4Z4+y7vxO3LrW+axQXiiK1SBStLAYtStsYTBDvpZiMi1rcN5KSQkhjzo/4i 0RCNVG6aKdyaT7nwRe3HD9e3cGv7rECvNuZxnzp+rAvzAOZAqsK04SDMY4ZsEVao7WNa b870rMhJkipBVR7c+YxgDQwUoVnXRGpGGD9arH7sqiSdTStVzh8Ntnn/eU2DXMCbSNCn egp6gxRBcCQBlnNDmcxgvAiTRvUSYhJjjmPLbJSA+r/IKxHF/IWMBN0vfKGAf3OCIdzY OsDw== X-Gm-Message-State: AGRZ1gIeMXfEAVlwrPSx78S5PyowfjtgB0OeoPj3K78hR9FNGjIVUbUm Nyd6Z1FIwk5D0QXNuslkTk8OJP7oV5Y= X-Google-Smtp-Source: AJdET5fO4UWyjNUaWm3RGR1J9ioLZtr1kpayeBIOmBpdi04AyJsQyI3NcjWeRoVHy88hyYBuLiV7Rw== X-Received: by 2002:a17:902:33c1:: with SMTP id b59-v6mr7480679plc.71.1542311514481; Thu, 15 Nov 2018 11:51:54 -0800 (PST) Received: from x1.localdomain ([2620:10d:c090:180::1:7813]) by smtp.gmail.com with ESMTPSA id d68-v6sm30670766pfa.80.2018.11.15.11.51.52 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 15 Nov 2018 11:51:53 -0800 (PST) From: Jens Axboe To: linux-block@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 07/11] blk-mq: when polling for IO, look for any completion Date: Thu, 15 Nov 2018 12:51:31 -0700 Message-Id: <20181115195135.22812-8-axboe@kernel.dk> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181115195135.22812-1-axboe@kernel.dk> References: <20181115195135.22812-1-axboe@kernel.dk> Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP If we want to support async IO polling, then we have to allow finding completions that aren't just for the one we are looking for. Always pass in -1 to the mq_ops->poll() helper, and have that return how many events were found in this poll loop. Signed-off-by: Jens Axboe --- block/blk-mq.c | 69 +++++++++++++++++++++++------------------ drivers/nvme/host/pci.c | 14 ++++----- 2 files changed, 46 insertions(+), 37 deletions(-) diff --git a/block/blk-mq.c b/block/blk-mq.c index 52b1c97cd7c6..3ca00d712158 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -3266,9 +3266,7 @@ static bool blk_mq_poll_hybrid_sleep(struct request_queue *q, * 0: use half of prev avg * >0: use this specific value */ - if (q->poll_nsec == -1) - return false; - else if (q->poll_nsec > 0) + if (q->poll_nsec > 0) nsecs = q->poll_nsec; else nsecs = blk_mq_poll_nsecs(q, hctx, rq); @@ -3305,21 +3303,36 @@ static bool blk_mq_poll_hybrid_sleep(struct request_queue *q, return true; } -static int __blk_mq_poll(struct blk_mq_hw_ctx *hctx, struct request *rq) +static bool blk_mq_poll_hybrid(struct request_queue *q, + struct blk_mq_hw_ctx *hctx, blk_qc_t cookie) +{ + struct request *rq; + + if (q->poll_nsec == -1) + return false; + + if (!blk_qc_t_is_internal(cookie)) + rq = blk_mq_tag_to_rq(hctx->tags, blk_qc_t_to_tag(cookie)); + else { + rq = blk_mq_tag_to_rq(hctx->sched_tags, blk_qc_t_to_tag(cookie)); + /* + * With scheduling, if the request has completed, we'll + * get a NULL return here, as we clear the sched tag when + * that happens. The request still remains valid, like always, + * so we should be safe with just the NULL check. + */ + if (!rq) + return false; + } + + return blk_mq_poll_hybrid_sleep(q, hctx, rq); +} + +static int __blk_mq_poll(struct blk_mq_hw_ctx *hctx) { struct request_queue *q = hctx->queue; long state; - /* - * If we sleep, have the caller restart the poll loop to reset - * the state. Like for the other success return cases, the - * caller is responsible for checking if the IO completed. If - * the IO isn't complete, we'll get called again and will go - * straight to the busy poll loop. - */ - if (blk_mq_poll_hybrid_sleep(q, hctx, rq)) - return 1; - hctx->poll_considered++; state = current->state; @@ -3328,7 +3341,7 @@ static int __blk_mq_poll(struct blk_mq_hw_ctx *hctx, struct request *rq) hctx->poll_invoked++; - ret = q->mq_ops->poll(hctx, rq->tag); + ret = q->mq_ops->poll(hctx, -1U); if (ret > 0) { hctx->poll_success++; __set_current_state(TASK_RUNNING); @@ -3352,27 +3365,23 @@ static int __blk_mq_poll(struct blk_mq_hw_ctx *hctx, struct request *rq) static int blk_mq_poll(struct request_queue *q, blk_qc_t cookie) { struct blk_mq_hw_ctx *hctx; - struct request *rq; if (!test_bit(QUEUE_FLAG_POLL, &q->queue_flags)) return 0; hctx = q->queue_hw_ctx[blk_qc_t_to_queue_num(cookie)]; - if (!blk_qc_t_is_internal(cookie)) - rq = blk_mq_tag_to_rq(hctx->tags, blk_qc_t_to_tag(cookie)); - else { - rq = blk_mq_tag_to_rq(hctx->sched_tags, blk_qc_t_to_tag(cookie)); - /* - * With scheduling, if the request has completed, we'll - * get a NULL return here, as we clear the sched tag when - * that happens. The request still remains valid, like always, - * so we should be safe with just the NULL check. - */ - if (!rq) - return 0; - } - return __blk_mq_poll(hctx, rq); + /* + * If we sleep, have the caller restart the poll loop to reset + * the state. Like for the other success return cases, the + * caller is responsible for checking if the IO completed. If + * the IO isn't complete, we'll get called again and will go + * straight to the busy poll loop. + */ + if (blk_mq_poll_hybrid(q, hctx, cookie)) + return 1; + + return __blk_mq_poll(hctx); } unsigned int blk_mq_rq_cpu(struct request *rq) diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c index fc7dd49f22fc..6c03461ad988 100644 --- a/drivers/nvme/host/pci.c +++ b/drivers/nvme/host/pci.c @@ -1012,15 +1012,15 @@ static inline void nvme_update_cq_head(struct nvme_queue *nvmeq) } } -static inline bool nvme_process_cq(struct nvme_queue *nvmeq, u16 *start, - u16 *end, int tag) +static inline int nvme_process_cq(struct nvme_queue *nvmeq, u16 *start, + u16 *end, unsigned int tag) { - bool found = false; + int found = 0; *start = nvmeq->cq_head; - while (!found && nvme_cqe_pending(nvmeq)) { - if (nvmeq->cqes[nvmeq->cq_head].command_id == tag) - found = true; + while (nvme_cqe_pending(nvmeq)) { + if (tag == -1U || nvmeq->cqes[nvmeq->cq_head].command_id == tag) + found++; nvme_update_cq_head(nvmeq); } *end = nvmeq->cq_head; @@ -1062,7 +1062,7 @@ static irqreturn_t nvme_irq_check(int irq, void *data) static int __nvme_poll(struct nvme_queue *nvmeq, unsigned int tag) { u16 start, end; - bool found; + int found; if (!nvme_cqe_pending(nvmeq)) return 0; From patchwork Thu Nov 15 19:51:32 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 10684977 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 8238814D6 for ; Thu, 15 Nov 2018 19:51:59 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 73CAF2CE3F for ; Thu, 15 Nov 2018 19:51:59 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 723A92D16A; Thu, 15 Nov 2018 19:51:59 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 544592D153 for ; Thu, 15 Nov 2018 19:51:58 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729225AbeKPGBJ (ORCPT ); Fri, 16 Nov 2018 01:01:09 -0500 Received: from mail-pl1-f194.google.com ([209.85.214.194]:38577 "EHLO mail-pl1-f194.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725781AbeKPGBJ (ORCPT ); Fri, 16 Nov 2018 01:01:09 -0500 Received: by mail-pl1-f194.google.com with SMTP id p4-v6so9965340plo.5 for ; Thu, 15 Nov 2018 11:51:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=PnDYIJyXFWSMKRTntl3dYnBofM0cebTckZzxLQWXx1A=; b=kjxLnI88E6RRZuGO0gtn/Kw9s0cYHUjvlrETW9odLhtLzwrYpAEbY+4LKaHVWoipAK aCRopC1XfyScmM/IoMBa8lHq0sxIOnJ+tXlA3S/vYD+5lsYlNsoazCJBKYWeZs/TMAIe cNwzG/2jdOZ2PZNOKDNcmDA6Zi/jTLFH7gk4P/UfYrM1BHVqhxW7aT/pe1p0fuZ+AqC3 bP2OJ55rYrLnEJvHSrOu1knzBVeKGpBSMYFQgRm0n8IkzRClp2qJNNSOCdzIkfd3ZxJw zd8BycLK2Ov1ZlRtMovIH9T0cTXSX4QoENBf8Hg9etxSD9VCscQ3rFH1K4hoEEHaQ6ux ZXfA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=PnDYIJyXFWSMKRTntl3dYnBofM0cebTckZzxLQWXx1A=; b=TX9CoHl/I8IEZWc9Xei/8/gy2LuX7NTmWUv0lPi9UI84QkKAR/O3Aoutzi6WT30FLH CdAeII34zimXV5XbYb8KtfYdsHOlCDnn1bGFhtFj0RPaSM4Ty7eefR0r/pfoQ3I+/+eO szBJ7ICixwILsROMkZsPXwz6NUTsruU+6eRb6GpDH9f2SgftFbh93wYYkahgquIdGG20 tq0FehkAAFL0NHYZ9o0XXiDbW3fvT/ZO9RQ2ILScA1klF3UEu3fsgCuR1GBrd+qjuUD4 6sZW0Zv3g33wTp+Z7hCu/O7gWKflbtJ4hgU9tkXCi7mdeMuGowbC5DfevAhd7VpkUMYu /LvQ== X-Gm-Message-State: AGRZ1gImXY+yVHJDewmvPzma5qdUTQbZe5qhOrppNqQxOCCrAWhDe7Go OBLRt5iKlWHYUEAvjh5BAsjk2RGmZSU= X-Google-Smtp-Source: AJdET5c2qd3RByXjV/UCkgVwmVkEwOdp9f16L2zcpdxzFrl2YMafrGaztMtcZ/ZzwUWCd7dqvTXgtQ== X-Received: by 2002:a17:902:9047:: with SMTP id w7mr4647393plz.270.1542311516327; Thu, 15 Nov 2018 11:51:56 -0800 (PST) Received: from x1.localdomain ([2620:10d:c090:180::1:7813]) by smtp.gmail.com with ESMTPSA id d68-v6sm30670766pfa.80.2018.11.15.11.51.54 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 15 Nov 2018 11:51:55 -0800 (PST) From: Jens Axboe To: linux-block@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 08/11] block: make blk_poll() take a parameter on whether to spin or not Date: Thu, 15 Nov 2018 12:51:32 -0700 Message-Id: <20181115195135.22812-9-axboe@kernel.dk> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181115195135.22812-1-axboe@kernel.dk> References: <20181115195135.22812-1-axboe@kernel.dk> Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP blk_poll() has always kept spinning until it found an IO. This is fine for SYNC polling, since we need to find one request we have pending, but in preparation for ASYNC polling it can be beneficial to just check if we have any entries available or not. Existing callers are converted to pass in 'spin == true', to retain the old behavior. Signed-off-by: Jens Axboe --- block/blk-core.c | 4 ++-- block/blk-mq.c | 10 +++++----- drivers/nvme/host/multipath.c | 4 ++-- drivers/nvme/target/io-cmd-bdev.c | 2 +- fs/block_dev.c | 4 ++-- fs/direct-io.c | 2 +- fs/iomap.c | 2 +- include/linux/blkdev.h | 4 ++-- mm/page_io.c | 2 +- 9 files changed, 17 insertions(+), 17 deletions(-) diff --git a/block/blk-core.c b/block/blk-core.c index 0b684a520a11..ccf40f853afd 100644 --- a/block/blk-core.c +++ b/block/blk-core.c @@ -1284,14 +1284,14 @@ blk_qc_t submit_bio(struct bio *bio) } EXPORT_SYMBOL(submit_bio); -bool blk_poll(struct request_queue *q, blk_qc_t cookie) +bool blk_poll(struct request_queue *q, blk_qc_t cookie, bool spin) { if (!q->poll_fn || !blk_qc_t_valid(cookie)) return false; if (current->plug) blk_flush_plug_list(current->plug, false); - return q->poll_fn(q, cookie); + return q->poll_fn(q, cookie, spin); } EXPORT_SYMBOL_GPL(blk_poll); diff --git a/block/blk-mq.c b/block/blk-mq.c index 3ca00d712158..695aa9363a6e 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -38,7 +38,7 @@ #include "blk-mq-sched.h" #include "blk-rq-qos.h" -static int blk_mq_poll(struct request_queue *q, blk_qc_t cookie); +static int blk_mq_poll(struct request_queue *q, blk_qc_t cookie, bool spin); static void blk_mq_poll_stats_start(struct request_queue *q); static void blk_mq_poll_stats_fn(struct blk_stat_callback *cb); @@ -3328,7 +3328,7 @@ static bool blk_mq_poll_hybrid(struct request_queue *q, return blk_mq_poll_hybrid_sleep(q, hctx, rq); } -static int __blk_mq_poll(struct blk_mq_hw_ctx *hctx) +static int __blk_mq_poll(struct blk_mq_hw_ctx *hctx, bool spin) { struct request_queue *q = hctx->queue; long state; @@ -3353,7 +3353,7 @@ static int __blk_mq_poll(struct blk_mq_hw_ctx *hctx) if (current->state == TASK_RUNNING) return 1; - if (ret < 0) + if (ret < 0 || !spin) break; cpu_relax(); } @@ -3362,7 +3362,7 @@ static int __blk_mq_poll(struct blk_mq_hw_ctx *hctx) return 0; } -static int blk_mq_poll(struct request_queue *q, blk_qc_t cookie) +static int blk_mq_poll(struct request_queue *q, blk_qc_t cookie, bool spin) { struct blk_mq_hw_ctx *hctx; @@ -3381,7 +3381,7 @@ static int blk_mq_poll(struct request_queue *q, blk_qc_t cookie) if (blk_mq_poll_hybrid(q, hctx, cookie)) return 1; - return __blk_mq_poll(hctx); + return __blk_mq_poll(hctx, spin); } unsigned int blk_mq_rq_cpu(struct request *rq) diff --git a/drivers/nvme/host/multipath.c b/drivers/nvme/host/multipath.c index 65539c8df11d..c83bb3302684 100644 --- a/drivers/nvme/host/multipath.c +++ b/drivers/nvme/host/multipath.c @@ -220,7 +220,7 @@ static blk_qc_t nvme_ns_head_make_request(struct request_queue *q, return ret; } -static int nvme_ns_head_poll(struct request_queue *q, blk_qc_t qc) +static int nvme_ns_head_poll(struct request_queue *q, blk_qc_t qc, bool spin) { struct nvme_ns_head *head = q->queuedata; struct nvme_ns *ns; @@ -230,7 +230,7 @@ static int nvme_ns_head_poll(struct request_queue *q, blk_qc_t qc) srcu_idx = srcu_read_lock(&head->srcu); ns = srcu_dereference(head->current_path[numa_node_id()], &head->srcu); if (likely(ns && nvme_path_is_optimized(ns))) - found = ns->queue->poll_fn(q, qc); + found = ns->queue->poll_fn(q, qc, spin); srcu_read_unlock(&head->srcu, srcu_idx); return found; } diff --git a/drivers/nvme/target/io-cmd-bdev.c b/drivers/nvme/target/io-cmd-bdev.c index c1ec3475a140..f6971b45bc54 100644 --- a/drivers/nvme/target/io-cmd-bdev.c +++ b/drivers/nvme/target/io-cmd-bdev.c @@ -116,7 +116,7 @@ static void nvmet_bdev_execute_rw(struct nvmet_req *req) cookie = submit_bio(bio); - blk_poll(bdev_get_queue(req->ns->bdev), cookie); + blk_poll(bdev_get_queue(req->ns->bdev), cookie, true); } static void nvmet_bdev_execute_flush(struct nvmet_req *req) diff --git a/fs/block_dev.c b/fs/block_dev.c index 0ed9be8906a8..7810f5b588ea 100644 --- a/fs/block_dev.c +++ b/fs/block_dev.c @@ -244,7 +244,7 @@ __blkdev_direct_IO_simple(struct kiocb *iocb, struct iov_iter *iter, break; if (!(iocb->ki_flags & IOCB_HIPRI) || - !blk_poll(bdev_get_queue(bdev), qc)) + !blk_poll(bdev_get_queue(bdev), qc, true)) io_schedule(); } __set_current_state(TASK_RUNNING); @@ -413,7 +413,7 @@ __blkdev_direct_IO(struct kiocb *iocb, struct iov_iter *iter, int nr_pages) break; if (!(iocb->ki_flags & IOCB_HIPRI) || - !blk_poll(bdev_get_queue(bdev), qc)) + !blk_poll(bdev_get_queue(bdev), qc, true)) io_schedule(); } __set_current_state(TASK_RUNNING); diff --git a/fs/direct-io.c b/fs/direct-io.c index ea07d5a34317..a5a4e5a1423e 100644 --- a/fs/direct-io.c +++ b/fs/direct-io.c @@ -518,7 +518,7 @@ static struct bio *dio_await_one(struct dio *dio) dio->waiter = current; spin_unlock_irqrestore(&dio->bio_lock, flags); if (!(dio->iocb->ki_flags & IOCB_HIPRI) || - !blk_poll(dio->bio_disk->queue, dio->bio_cookie)) + !blk_poll(dio->bio_disk->queue, dio->bio_cookie, true)) io_schedule(); /* wake up sets us TASK_RUNNING */ spin_lock_irqsave(&dio->bio_lock, flags); diff --git a/fs/iomap.c b/fs/iomap.c index 38c9bc63296a..1ef4e063f068 100644 --- a/fs/iomap.c +++ b/fs/iomap.c @@ -1897,7 +1897,7 @@ iomap_dio_rw(struct kiocb *iocb, struct iov_iter *iter, if (!(iocb->ki_flags & IOCB_HIPRI) || !dio->submit.last_queue || !blk_poll(dio->submit.last_queue, - dio->submit.cookie)) + dio->submit.cookie, true)) io_schedule(); } __set_current_state(TASK_RUNNING); diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h index e96dc16ef8aa..e83ad6f15281 100644 --- a/include/linux/blkdev.h +++ b/include/linux/blkdev.h @@ -283,7 +283,7 @@ static inline unsigned short req_get_ioprio(struct request *req) struct blk_queue_ctx; typedef blk_qc_t (make_request_fn) (struct request_queue *q, struct bio *bio); -typedef int (poll_q_fn) (struct request_queue *q, blk_qc_t); +typedef int (poll_q_fn) (struct request_queue *q, blk_qc_t, bool spin); struct bio_vec; typedef int (dma_drain_needed_fn)(struct request *); @@ -868,7 +868,7 @@ extern void blk_execute_rq_nowait(struct request_queue *, struct gendisk *, int blk_status_to_errno(blk_status_t status); blk_status_t errno_to_blk_status(int errno); -bool blk_poll(struct request_queue *q, blk_qc_t cookie); +bool blk_poll(struct request_queue *q, blk_qc_t cookie, bool spin); static inline struct request_queue *bdev_get_queue(struct block_device *bdev) { diff --git a/mm/page_io.c b/mm/page_io.c index f277459db805..1518f459866d 100644 --- a/mm/page_io.c +++ b/mm/page_io.c @@ -411,7 +411,7 @@ int swap_readpage(struct page *page, bool synchronous) if (!READ_ONCE(bio->bi_private)) break; - if (!blk_poll(disk->queue, qc)) + if (!blk_poll(disk->queue, qc, true)) break; } __set_current_state(TASK_RUNNING); From patchwork Thu Nov 15 19:51:33 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 10684979 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id E1FAA17EE for ; Thu, 15 Nov 2018 19:51:59 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D174D2D0EC for ; Thu, 15 Nov 2018 19:51:59 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id CFC112D16A; Thu, 15 Nov 2018 19:51:59 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 71F812D14E for ; Thu, 15 Nov 2018 19:51:59 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729263AbeKPGBK (ORCPT ); Fri, 16 Nov 2018 01:01:10 -0500 Received: from mail-pl1-f193.google.com ([209.85.214.193]:42847 "EHLO mail-pl1-f193.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725781AbeKPGBK (ORCPT ); Fri, 16 Nov 2018 01:01:10 -0500 Received: by mail-pl1-f193.google.com with SMTP id x21-v6so7241864pln.9 for ; Thu, 15 Nov 2018 11:51:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=UjKFPPY66VB4W6EBmXfarxZuGQQ5lRuC6yiN12FUde8=; b=Z0vv1NFk8YfS0Xj6QE28UswnkjV973Br90CAQ8O6HCiICvXWAb2OMXpKOdgRDrBShq fgMc2QH/gu3UkcxH+WvZrgDYGOLdtNmC5l0EAQ5fXWhZuWomVrx7CsErgwbt9qAmV5C4 M/6nu6f3UUyH2l7xf7V379Fi7Xqdhb8AKlj9AvgG4+Znb3jgydg6C4aygysDhK2bfrPt u5PZSHe3R/v2lC7i4Qsv2SnHMnvG8JD+EM8DQ1epe6p3e0BZIRkPQ8OhNy/7XjPOPZZn wHT/zU7XLGd29fw0nT6BwF8k4DyOpw16oY0qXaIcY84JgOouOVXlHK+phoGypW3eOBKA Z5iQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=UjKFPPY66VB4W6EBmXfarxZuGQQ5lRuC6yiN12FUde8=; b=Irj5Rmnrp19Uh1XjlD5BexmZRWsIyuEymoCH3KeyeqdARjc+8vF+AzpzCZY1UU283E /U7KNgZzf6QTAZVMHGYe2xQlfDn3lQoOcdpBCk6805NPr1sBWdpaWuQkSgpeoE12RWKB 7PB9oMH0clt7Sr3YotfGKQOea7SwjgZKrOak+dpwIgWQB+N0qGVPWLyGtBWSEFpI6tCa Fab8n7N2E2u01ljI0SRdMs1pANerkXYgNpopQGRM+lnC8haAiyCSopjgJIhPkMxd+j4g j5D1lkI4/w3aertiV+4YMrCL6AqBGxLXU8FHvJAoiRtBWhzV9sXZTxATSezyjGk5lhF1 OqgQ== X-Gm-Message-State: AGRZ1gKdwbhYm61roqqMw3a0hboFjMMevWSGNEAIQqoAFWABH0OJebLv CPOF8ia1JfdZoYSs7/uq4JYprSvdUf4= X-Google-Smtp-Source: AJdET5dKp4SIOsgjibHOjeqebQ2SYxZIXGybbY6STVljeN5N95MW/QlKZJ4X5Fo3mV6Q2iCkh+gJwQ== X-Received: by 2002:a17:902:2bc5:: with SMTP id l63-v6mr7681411plb.241.1542311517808; Thu, 15 Nov 2018 11:51:57 -0800 (PST) Received: from x1.localdomain ([2620:10d:c090:180::1:7813]) by smtp.gmail.com with ESMTPSA id d68-v6sm30670766pfa.80.2018.11.15.11.51.56 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 15 Nov 2018 11:51:56 -0800 (PST) From: Jens Axboe To: linux-block@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 09/11] blk-mq: ensure mq_ops ->poll() is entered at least once Date: Thu, 15 Nov 2018 12:51:33 -0700 Message-Id: <20181115195135.22812-10-axboe@kernel.dk> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181115195135.22812-1-axboe@kernel.dk> References: <20181115195135.22812-1-axboe@kernel.dk> Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Right now we immediately bail if need_resched() is true, but we need to do at least one loop in case we have entries waiting. So just invert the need_resched() check, putting it at the bottom of the loop. Signed-off-by: Jens Axboe --- block/blk-mq.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/block/blk-mq.c b/block/blk-mq.c index 695aa9363a6e..0ff70dfc8c0e 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -3336,7 +3336,7 @@ static int __blk_mq_poll(struct blk_mq_hw_ctx *hctx, bool spin) hctx->poll_considered++; state = current->state; - while (!need_resched()) { + do { int ret; hctx->poll_invoked++; @@ -3356,7 +3356,7 @@ static int __blk_mq_poll(struct blk_mq_hw_ctx *hctx, bool spin) if (ret < 0 || !spin) break; cpu_relax(); - } + } while (!need_resched()); __set_current_state(TASK_RUNNING); return 0; From patchwork Thu Nov 15 19:51:34 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 10684981 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id DD08613BB for ; Thu, 15 Nov 2018 19:52:01 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id CB76C2D14E for ; Thu, 15 Nov 2018 19:52:01 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id C9B412D15E; Thu, 15 Nov 2018 19:52:01 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 695E92D14E for ; Thu, 15 Nov 2018 19:52:01 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728772AbeKPGBM (ORCPT ); Fri, 16 Nov 2018 01:01:12 -0500 Received: from mail-pf1-f196.google.com ([209.85.210.196]:44728 "EHLO mail-pf1-f196.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725781AbeKPGBM (ORCPT ); Fri, 16 Nov 2018 01:01:12 -0500 Received: by mail-pf1-f196.google.com with SMTP id b81-v6so9694680pfe.11 for ; Thu, 15 Nov 2018 11:52:00 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=WYF2DQMSuLBvvC6oyWCzDINndqOwZo6FPKocRmVKp6k=; b=T5sxAa0ZLNhXZpxss4WoFTAqUFa3kyESjrlyZu9kD3Y8OU5u3OKuAjcIw6l0YNXVkk 5P2pUgzHOp0w1oNae4/v9GQtSyo1vfxi9k6LZdfS34r6u3kG94lWUT3cBw1hPH14TPJw VgppQZJFdHpMOXTA9Lu3rQwicPcQu7gidv4M8TPUVZr/CNLjQYvx989aKqgXqu4EwkW/ lv1jgByNxezdU1sH87DqduSW2/WdkEPVA/8pYm+lU1WQQUhc7tHTwTB6ONJUAeYodwxx r2qNwBsipxL25Df4bU9KZYm8dAqfHt3YFp37HZVQXPC3H6KtPasmle0t1DqKp8AK+IpQ PH/A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=WYF2DQMSuLBvvC6oyWCzDINndqOwZo6FPKocRmVKp6k=; b=YZxtPqq4eiZrr49L3gLo6vYTp/jO4m478j5q8/8PVQ2kqj90JLDeI/hcX/f6QnZ6IP 1EgvtCep+oy7hTlHIjqwWEjgfqVJIHblE7FxvwVb+htqw997V1H5aUZhYXGleYPR8PGm Uwmfcnt5ogR80ciXV7/FEuXUywgvb8ZXeArEHO8Pl9hX5cN2ORAUQ4XGi1FAOrrY1nwA sSLqI/B6MRRiDr0aDqXNQv07esPLU1VzUELVloDsu5I75YRIkVgf4NBmOA58uXjSGOl9 8/ott7Z1EHgErG+W3SwKR5GO9duqavuv4YaEvUogulJBoOMazKT8CC56kJwC9Z9NQfVg NJSg== X-Gm-Message-State: AGRZ1gI+PHQX5BxZo3cChnWSqYFK6kRmtI6bg7EpYlU1ynZWg+VdSU0L 9Hn8lEaxEX31LREOcwbvWdpzqdMHNDo= X-Google-Smtp-Source: AJdET5ed0D/bh3cdJA0Xz8YbicN07WRf9ub7ivRjcjJjGR41EvQaranHOJU8Hi37wz1PICenjsaqKQ== X-Received: by 2002:a63:170c:: with SMTP id x12mr6869920pgl.364.1542311519713; Thu, 15 Nov 2018 11:51:59 -0800 (PST) Received: from x1.localdomain ([2620:10d:c090:180::1:7813]) by smtp.gmail.com with ESMTPSA id d68-v6sm30670766pfa.80.2018.11.15.11.51.57 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 15 Nov 2018 11:51:58 -0800 (PST) From: Jens Axboe To: linux-block@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 10/11] block: for async O_DIRECT, mark us as polling if asked to Date: Thu, 15 Nov 2018 12:51:34 -0700 Message-Id: <20181115195135.22812-11-axboe@kernel.dk> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181115195135.22812-1-axboe@kernel.dk> References: <20181115195135.22812-1-axboe@kernel.dk> Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Inherit the iocb IOCB_HIPRI flag, and pass on REQ_HIPRI for those kinds of requests. Signed-off-by: Jens Axboe Reviewed-by: Christoph Hellwig --- fs/block_dev.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/fs/block_dev.c b/fs/block_dev.c index 7810f5b588ea..c124982b810d 100644 --- a/fs/block_dev.c +++ b/fs/block_dev.c @@ -386,6 +386,9 @@ __blkdev_direct_IO(struct kiocb *iocb, struct iov_iter *iter, int nr_pages) nr_pages = iov_iter_npages(iter, BIO_MAX_PAGES); if (!nr_pages) { + if (iocb->ki_flags & IOCB_HIPRI) + bio->bi_opf |= REQ_HIPRI; + qc = submit_bio(bio); break; } From patchwork Thu Nov 15 19:51:35 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 10684983 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B5D6314D6 for ; Thu, 15 Nov 2018 19:52:03 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A725E2D0D3 for ; Thu, 15 Nov 2018 19:52:03 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 95B3E2D14E; Thu, 15 Nov 2018 19:52:03 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 3154A2D08F for ; Thu, 15 Nov 2018 19:52:03 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1725868AbeKPGBO (ORCPT ); Fri, 16 Nov 2018 01:01:14 -0500 Received: from mail-pg1-f196.google.com ([209.85.215.196]:36192 "EHLO mail-pg1-f196.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725781AbeKPGBO (ORCPT ); Fri, 16 Nov 2018 01:01:14 -0500 Received: by mail-pg1-f196.google.com with SMTP id n2so2298137pgm.3 for ; Thu, 15 Nov 2018 11:52:02 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=Fz9g1ov0KEBk0jca5hC7QbGAUKLsaBIiDlim4iRDh3o=; b=2RATox6BMIW4EUoQcZ4YDFq8hclWiTuXeB0mphIjgCKp5th1/sVoIJ8ylG8PQoF2DZ NreEVIM6RSDoPPZBYglwNOHYxM6eH2ECaACQ+vstVVz64rVelgNDJbGTfTkX+90SuEdg Z0l50QcL7XI6neZX1tPhLZhdhaC4NcB/kMWonvxGLi4foG9GmsQyjSF68Hdal2lKxFZo VJozM1hrlqQL8XZVvCWGOcZLBY6h48g5dqXW1TwxQKmZoT87gFr24KncOm4Vg1F/0y+E BxYqKHt+kzOp/rjD0zkGtTUDs1+Jq4Uv4T+Uf0vr3ovf/z2p8un//jf5frh8aOju+3xr zd9w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=Fz9g1ov0KEBk0jca5hC7QbGAUKLsaBIiDlim4iRDh3o=; b=q9UB3fo9n83DqakGbuz26In2EL1VdbV+j6zN3inPODbuVgRdseSuNchk7BUgZz4+EF jjav8Ap5AuMadnJb4WtKagYaYD7ejnH7wPMH2VQ3/c7VRo4cCawu3Rwa3ltsyDzBRA+O n6lavGGe7KsbkwXRoeo7foWsjWetQ08KKntIpK34D/QRAMdVss5xN1xS03EAiYB12d+a cItHI59HsqgGT+AHA3uEtMvXj6IzzeFJ2wa5K7k+eZuVZvR0le6xaXmhtT8RX526d/M9 gj0w+gug1wc82p8ord+BrBgHRMA6WR7y4razYq5DYp0e4s+mrfjKutVcuX/djIFkUDLv 56Vg== X-Gm-Message-State: AGRZ1gIpTVE8gWBgMdu7Ed5V3w6IawJnNleDSh95mxlP6TYNgMYFjpN7 N7iz2CLSu9aNXVHtzaDZq5GErpU6Reo= X-Google-Smtp-Source: AJdET5c7FMGsxqg4HwcXgEXNRIEpm3boNBIOUyyK+dfkELu4a+gcwhVkpHgjOTnVApGReG9hSNw6Ow== X-Received: by 2002:a63:d747:: with SMTP id w7mr6918420pgi.360.1542311521459; Thu, 15 Nov 2018 11:52:01 -0800 (PST) Received: from x1.localdomain ([2620:10d:c090:180::1:7813]) by smtp.gmail.com with ESMTPSA id d68-v6sm30670766pfa.80.2018.11.15.11.51.59 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 15 Nov 2018 11:52:00 -0800 (PST) From: Jens Axboe To: linux-block@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 11/11] block: don't plug for aio/O_DIRECT HIPRI IO Date: Thu, 15 Nov 2018 12:51:35 -0700 Message-Id: <20181115195135.22812-12-axboe@kernel.dk> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181115195135.22812-1-axboe@kernel.dk> References: <20181115195135.22812-1-axboe@kernel.dk> Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Those will go straight to issue inside blk-mq, so don't bother setting up a block plug for them. Signed-off-by: Jens Axboe Reviewed-by: Christoph Hellwig --- fs/block_dev.c | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/fs/block_dev.c b/fs/block_dev.c index c124982b810d..9dc695a3af4e 100644 --- a/fs/block_dev.c +++ b/fs/block_dev.c @@ -356,7 +356,13 @@ __blkdev_direct_IO(struct kiocb *iocb, struct iov_iter *iter, int nr_pages) dio->multi_bio = false; dio->should_dirty = is_read && iter_is_iovec(iter); - blk_start_plug(&plug); + /* + * Don't plug for HIPRI/polled IO, as those should go straight + * to issue + */ + if (!(iocb->ki_flags & IOCB_HIPRI)) + blk_start_plug(&plug); + for (;;) { bio_set_dev(bio, bdev); bio->bi_iter.bi_sector = pos >> 9; @@ -403,7 +409,9 @@ __blkdev_direct_IO(struct kiocb *iocb, struct iov_iter *iter, int nr_pages) submit_bio(bio); bio = bio_alloc(GFP_KERNEL, nr_pages); } - blk_finish_plug(&plug); + + if (!(iocb->ki_flags & IOCB_HIPRI)) + blk_finish_plug(&plug); if (!is_sync) return -EIOCBQUEUED;