From patchwork Tue Nov 13 15:42:30 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 10681041 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 6047C1709 for ; Tue, 13 Nov 2018 15:42:56 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 4E9832AE1D for ; Tue, 13 Nov 2018 15:42:56 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 4D00E2AE46; Tue, 13 Nov 2018 15:42:56 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9ACCC2AE1D for ; Tue, 13 Nov 2018 15:42:55 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2387779AbeKNBld (ORCPT ); Tue, 13 Nov 2018 20:41:33 -0500 Received: from mail-it1-f193.google.com ([209.85.166.193]:39486 "EHLO mail-it1-f193.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1733221AbeKNBlc (ORCPT ); Tue, 13 Nov 2018 20:41:32 -0500 Received: by mail-it1-f193.google.com with SMTP id m15so19048419itl.4 for ; Tue, 13 Nov 2018 07:42:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=EHx8l7IMdoDaN4dvTj41uPWQUtQjoAD8b6vGeSugKx4=; b=PiY2ZWaYMJtI/vujNb9OCNrjIbFvyHgJCYKoEUf31QZaag+IFwV+wF8yiJOZmYrE0h KaMcFCkCm2XosZw0/cK3DOG1BrLnefqi3ttlsY9lTjkMk3T8YpPMrDY40AER53kH4bsm R0CxKf0pj6Hu8aq4vSdDTar8ZqcvRqjJ4cpQn2m4yW3fsscLDaZpWWt27EXc20fMoqoZ wnNPBSeGbsOR32CYljAIoBnbRAFvtSzCDhpVhbEk1E292CnAuMCzGM2Zh6RGCVG8NFMa SDzAuth02ti+qqWPOpYO0DzdmYMGVbETMHjgea8WU5ITQEGbp8Yl4/Ftm4Xtwm2qsDVJ 8q6Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=EHx8l7IMdoDaN4dvTj41uPWQUtQjoAD8b6vGeSugKx4=; b=mNlQZVbxf2/iGAkYg3MChXtJfui71E1fTAluf8xYs3myL3NvnNSxYy906GAnRujF6q bjLzK6yHCsRqfblFMFH4qpUZikD+MUnoks/9YpFTC4WGJJsaPpbpSZ06Of+YJds9fg55 xatWCHfzQ9FzcpSMabYNhFtMT24WIy2i1j0mgM1s74myV7+jI1nOO4LE2NBd+XI2XnVV et/cqfcaQ9DiBw/rhUR1pWwdAUGiKBmCjG9pPK9guZWC9M4y28dElXagEkk4CIWC5/ac WChIZhj5vc8QzciAnm0zAnTTLL4RL1TkDf+r+QG/DNPEzFdkxisSfsKhEbM3v31TCDJJ 1jTA== X-Gm-Message-State: AGRZ1gKur6CnCUCl020w51ODjdoInwl/J7lb0HNwmRmzoqhQf2jbAGlU TxxSsbScFPHZbE5kpEpuAYlL0I4rbS8= X-Google-Smtp-Source: AJdET5e3SQmsEohHfHqIdQ4p/ynnyMnaUDcWtTKys55FqNXTZrfhRlGUbG04ff4Fmtslk/JkfIh/1g== X-Received: by 2002:a24:fd09:: with SMTP id m9mr4036186ith.81.1542123772883; Tue, 13 Nov 2018 07:42:52 -0800 (PST) Received: from x1.localdomain ([216.160.245.98]) by smtp.gmail.com with ESMTPSA id o14-v6sm6721987ito.3.2018.11.13.07.42.51 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 13 Nov 2018 07:42:51 -0800 (PST) From: Jens Axboe To: linux-block@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 08/11] blk-mq: when polling for IO, look for any completion Date: Tue, 13 Nov 2018 08:42:30 -0700 Message-Id: <20181113154233.15256-9-axboe@kernel.dk> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181113154233.15256-1-axboe@kernel.dk> References: <20181113154233.15256-1-axboe@kernel.dk> Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP If we want to support async IO polling, then we have to allow finding completions that aren't just for the one we are looking for. Always pass in -1 to the mq_ops->poll() helper, and have that return how many events were found in this poll loop. Signed-off-by: Jens Axboe --- block/blk-mq.c | 69 +++++++++++++++++++++++------------------ drivers/nvme/host/pci.c | 32 +++++++++---------- 2 files changed, 54 insertions(+), 47 deletions(-) diff --git a/block/blk-mq.c b/block/blk-mq.c index f8c2e6544903..03b1af0151ca 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -3266,9 +3266,7 @@ static bool blk_mq_poll_hybrid_sleep(struct request_queue *q, * 0: use half of prev avg * >0: use this specific value */ - if (q->poll_nsec == -1) - return false; - else if (q->poll_nsec > 0) + if (q->poll_nsec > 0) nsecs = q->poll_nsec; else nsecs = blk_mq_poll_nsecs(q, hctx, rq); @@ -3305,21 +3303,36 @@ static bool blk_mq_poll_hybrid_sleep(struct request_queue *q, return true; } -static int __blk_mq_poll(struct blk_mq_hw_ctx *hctx, struct request *rq) +static bool blk_mq_poll_hybrid(struct request_queue *q, + struct blk_mq_hw_ctx *hctx, blk_qc_t cookie) +{ + struct request *rq; + + if (q->poll_nsec == -1) + return false; + + if (!blk_qc_t_is_internal(cookie)) + rq = blk_mq_tag_to_rq(hctx->tags, blk_qc_t_to_tag(cookie)); + else { + rq = blk_mq_tag_to_rq(hctx->sched_tags, blk_qc_t_to_tag(cookie)); + /* + * With scheduling, if the request has completed, we'll + * get a NULL return here, as we clear the sched tag when + * that happens. The request still remains valid, like always, + * so we should be safe with just the NULL check. + */ + if (!rq) + return false; + } + + return blk_mq_poll_hybrid_sleep(q, hctx, rq); +} + +static int __blk_mq_poll(struct blk_mq_hw_ctx *hctx) { struct request_queue *q = hctx->queue; long state; - /* - * If we sleep, have the caller restart the poll loop to reset - * the state. Like for the other success return cases, the - * caller is responsible for checking if the IO completed. If - * the IO isn't complete, we'll get called again and will go - * straight to the busy poll loop. - */ - if (blk_mq_poll_hybrid_sleep(q, hctx, rq)) - return 1; - hctx->poll_considered++; state = current->state; @@ -3328,7 +3341,7 @@ static int __blk_mq_poll(struct blk_mq_hw_ctx *hctx, struct request *rq) hctx->poll_invoked++; - ret = q->mq_ops.poll(hctx, rq->tag); + ret = q->mq_ops.poll(hctx, -1U); if (ret > 0) { hctx->poll_success++; set_current_state(TASK_RUNNING); @@ -3352,27 +3365,23 @@ static int __blk_mq_poll(struct blk_mq_hw_ctx *hctx, struct request *rq) static int blk_mq_poll(struct request_queue *q, blk_qc_t cookie) { struct blk_mq_hw_ctx *hctx; - struct request *rq; if (!test_bit(QUEUE_FLAG_POLL, &q->queue_flags)) return 0; hctx = q->queue_hw_ctx[blk_qc_t_to_queue_num(cookie)]; - if (!blk_qc_t_is_internal(cookie)) - rq = blk_mq_tag_to_rq(hctx->tags, blk_qc_t_to_tag(cookie)); - else { - rq = blk_mq_tag_to_rq(hctx->sched_tags, blk_qc_t_to_tag(cookie)); - /* - * With scheduling, if the request has completed, we'll - * get a NULL return here, as we clear the sched tag when - * that happens. The request still remains valid, like always, - * so we should be safe with just the NULL check. - */ - if (!rq) - return 0; - } - return __blk_mq_poll(hctx, rq); + /* + * If we sleep, have the caller restart the poll loop to reset + * the state. Like for the other success return cases, the + * caller is responsible for checking if the IO completed. If + * the IO isn't complete, we'll get called again and will go + * straight to the busy poll loop. + */ + if (blk_mq_poll_hybrid(q, hctx, cookie)) + return 1; + + return __blk_mq_poll(hctx); } unsigned int blk_mq_rq_cpu(struct request *rq) diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c index bb22ae567208..adeb8f516bf9 100644 --- a/drivers/nvme/host/pci.c +++ b/drivers/nvme/host/pci.c @@ -995,13 +995,18 @@ static inline void nvme_handle_cqe(struct nvme_queue *nvmeq, u16 idx) nvme_end_request(req, cqe->status, cqe->result); } -static void nvme_complete_cqes(struct nvme_queue *nvmeq, u16 start, u16 end) +static int nvme_complete_cqes(struct nvme_queue *nvmeq, u16 start, u16 end) { + int nr = 0; + while (start != end) { + nr++; nvme_handle_cqe(nvmeq, start); if (++start == nvmeq->q_depth) start = 0; } + + return nr; } static inline void nvme_update_cq_head(struct nvme_queue *nvmeq) @@ -1012,22 +1017,17 @@ static inline void nvme_update_cq_head(struct nvme_queue *nvmeq) } } -static inline bool nvme_process_cq(struct nvme_queue *nvmeq, u16 *start, - u16 *end, int tag) +static inline void nvme_process_cq(struct nvme_queue *nvmeq, u16 *start, + u16 *end) { - bool found = false; - *start = nvmeq->cq_head; - while (!found && nvme_cqe_pending(nvmeq)) { - if (nvmeq->cqes[nvmeq->cq_head].command_id == tag) - found = true; + while (nvme_cqe_pending(nvmeq)) nvme_update_cq_head(nvmeq); - } + *end = nvmeq->cq_head; if (*start != *end) nvme_ring_cq_doorbell(nvmeq); - return found; } static irqreturn_t nvme_irq(int irq, void *data) @@ -1039,7 +1039,7 @@ static irqreturn_t nvme_irq(int irq, void *data) spin_lock(&nvmeq->cq_lock); if (nvmeq->cq_head != nvmeq->last_cq_head) ret = IRQ_HANDLED; - nvme_process_cq(nvmeq, &start, &end, -1); + nvme_process_cq(nvmeq, &start, &end); nvmeq->last_cq_head = nvmeq->cq_head; spin_unlock(&nvmeq->cq_lock); @@ -1062,7 +1062,6 @@ static irqreturn_t nvme_irq_check(int irq, void *data) static int __nvme_poll(struct nvme_queue *nvmeq, unsigned int tag) { u16 start, end; - bool found; if (!nvme_cqe_pending(nvmeq)) return 0; @@ -1074,14 +1073,13 @@ static int __nvme_poll(struct nvme_queue *nvmeq, unsigned int tag) local_irq_disable(); spin_lock(&nvmeq->cq_lock); - found = nvme_process_cq(nvmeq, &start, &end, tag); + nvme_process_cq(nvmeq, &start, &end); spin_unlock(&nvmeq->cq_lock); if (!nvmeq->polled) local_irq_enable(); - nvme_complete_cqes(nvmeq, start, end); - return found; + return nvme_complete_cqes(nvmeq, start, end); } static int nvme_poll(struct blk_mq_hw_ctx *hctx, unsigned int tag) @@ -1414,7 +1412,7 @@ static void nvme_disable_admin_queue(struct nvme_dev *dev, bool shutdown) nvme_disable_ctrl(&dev->ctrl, dev->ctrl.cap); spin_lock_irq(&nvmeq->cq_lock); - nvme_process_cq(nvmeq, &start, &end, -1); + nvme_process_cq(nvmeq, &start, &end); spin_unlock_irq(&nvmeq->cq_lock); nvme_complete_cqes(nvmeq, start, end); @@ -2209,7 +2207,7 @@ static void nvme_del_cq_end(struct request *req, blk_status_t error) unsigned long flags; spin_lock_irqsave(&nvmeq->cq_lock, flags); - nvme_process_cq(nvmeq, &start, &end, -1); + nvme_process_cq(nvmeq, &start, &end); spin_unlock_irqrestore(&nvmeq->cq_lock, flags); nvme_complete_cqes(nvmeq, start, end);