From patchwork Fri Nov 23 18:34:06 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 10696261 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 72FCA1750 for ; Fri, 23 Nov 2018 18:34:22 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 62E122B533 for ; Fri, 23 Nov 2018 18:34:22 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 56FF52BB6C; Fri, 23 Nov 2018 18:34:22 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 99EAD2BA7B for ; Fri, 23 Nov 2018 18:34:21 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2389278AbeKXFTo (ORCPT ); Sat, 24 Nov 2018 00:19:44 -0500 Received: from mail-pg1-f195.google.com ([209.85.215.195]:35226 "EHLO mail-pg1-f195.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726368AbeKXFTn (ORCPT ); Sat, 24 Nov 2018 00:19:43 -0500 Received: by mail-pg1-f195.google.com with SMTP id s198so3380048pgs.2 for ; Fri, 23 Nov 2018 10:34:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=CnlK+mr+M53qC/l7xd/anJL6Z+EIH23uc9OwRaNRk8U=; b=cm/bE8T3Kta2xHCzhGZUR0YiZpOTT9eA4UFaAuwuDrrbIgz6U1hfQ+BUQ+EPKo2ZsY mVYYnuWMAuUU+x1Tw1i4RjDa+gPWVBeSOdTK7b9q3vJa+yR3MtKp1XpUNcAKPh3Ftw9G KZcipwf8t/QJ8cE82fRikn+i8bYK6mE/AXhBIds/K5HqSHv1VUL1ISQVnSvk2FWd/HK2 AUtEn3BrbOITycXI6IodiFm/Lej7vJEHh5pC9cSnyguFgnuKCS2DIQ1T6aNyIFidpoRa 3AJPImRkL2430sS+LvXSgNWOV+idADORnTiavhoEWl1oW+71wgST75aBJoZ2t5n3yCt6 Mbzw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=CnlK+mr+M53qC/l7xd/anJL6Z+EIH23uc9OwRaNRk8U=; b=YghPcGs32Pi+vjhRwGrq+mDSs5Ci5iObLDQgU5fn7uGc4O0/zfF64s2udjm0d+tQvQ CrgYcl6K2t0mQCuney6kozQZ76UTrA1IfODfGwrsttEgyukigNpqdULw2KABqL0jA2bh ZSpulCqyI8B3LYZc+qyAd0Ri7RWdGbX38noQnpFWnlwJr9JzMHC2z1+JXYBFdJ0NxKeW 5F6ugXiAIIg2rVLdHx+PB0zu6DuZLZyStoLzVIAi2LYJ5Dvl03ZEWpqcHYQ7EhkxZUdY xKCi9XWeHIJ7uZ+D47MqHQGI+CZjvEMTrGsIKpPePbwbEzXuiW0dRcUDhtcNAj3eyNd3 xHlA== X-Gm-Message-State: AA+aEWYbyRuHi116v79ixWN4TDLDJPJyh5/GJ66GQSboDh31j/MwsOVj yTomwJ5o09dXC5kJ8MAOwCpSFci1OxQ= X-Google-Smtp-Source: AFSGD/WiVRAY+8yD38P7lLwJM2Y723RYUBarHR9DOQA824plr0pZG9ZK0Jbp6XyRFyzvN8V3YTnVCw== X-Received: by 2002:a63:8c0d:: with SMTP id m13mr15154819pgd.422.1542998056989; Fri, 23 Nov 2018 10:34:16 -0800 (PST) Received: from x1.thefacebook.com (66.29.188.166.static.utbb.net. [66.29.188.166]) by smtp.gmail.com with ESMTPSA id s37sm52294260pgm.19.2018.11.23.10.34.15 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 23 Nov 2018 10:34:16 -0800 (PST) From: Jens Axboe To: linux-block@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 1/6] blk-mq: when polling for IO, look for any completion Date: Fri, 23 Nov 2018 11:34:06 -0700 Message-Id: <20181123183411.8708-2-axboe@kernel.dk> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181123183411.8708-1-axboe@kernel.dk> References: <20181123183411.8708-1-axboe@kernel.dk> Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP If we want to support async IO polling, then we have to allow finding completions that aren't just for the one we are looking for. Always pass in -1 to the mq_ops->poll() helper, and have that return how many events were found in this poll loop. Signed-off-by: Jens Axboe --- block/blk-core.c | 2 +- block/blk-mq.c | 71 ++++++++++++++++++++-------------------- drivers/nvme/host/pci.c | 14 ++++---- drivers/nvme/host/rdma.c | 39 +++++++++------------- include/linux/blkdev.h | 2 +- 5 files changed, 60 insertions(+), 68 deletions(-) diff --git a/block/blk-core.c b/block/blk-core.c index 04f5be473638..9e99aa852d6e 100644 --- a/block/blk-core.c +++ b/block/blk-core.c @@ -1273,7 +1273,7 @@ blk_qc_t submit_bio(struct bio *bio) } EXPORT_SYMBOL(submit_bio); -bool blk_poll(struct request_queue *q, blk_qc_t cookie) +int blk_poll(struct request_queue *q, blk_qc_t cookie) { if (!q->poll_fn || !blk_qc_t_valid(cookie)) return false; diff --git a/block/blk-mq.c b/block/blk-mq.c index b16204df65d1..ec6c79578332 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -3285,15 +3285,12 @@ static bool blk_mq_poll_hybrid_sleep(struct request_queue *q, return false; /* - * poll_nsec can be: + * If we get here, hybrid polling is enabled. Hence poll_nsec can be: * - * -1: don't ever hybrid sleep * 0: use half of prev avg * >0: use this specific value */ - if (q->poll_nsec == -1) - return false; - else if (q->poll_nsec > 0) + if (q->poll_nsec > 0) nsecs = q->poll_nsec; else nsecs = blk_mq_poll_nsecs(q, hctx, rq); @@ -3330,11 +3327,41 @@ static bool blk_mq_poll_hybrid_sleep(struct request_queue *q, return true; } -static int __blk_mq_poll(struct blk_mq_hw_ctx *hctx, struct request *rq) +static bool blk_mq_poll_hybrid(struct request_queue *q, + struct blk_mq_hw_ctx *hctx, blk_qc_t cookie) { - struct request_queue *q = hctx->queue; + struct request *rq; + + if (q->poll_nsec == -1) + return false; + + if (!blk_qc_t_is_internal(cookie)) + rq = blk_mq_tag_to_rq(hctx->tags, blk_qc_t_to_tag(cookie)); + else { + rq = blk_mq_tag_to_rq(hctx->sched_tags, blk_qc_t_to_tag(cookie)); + /* + * With scheduling, if the request has completed, we'll + * get a NULL return here, as we clear the sched tag when + * that happens. The request still remains valid, like always, + * so we should be safe with just the NULL check. + */ + if (!rq) + return false; + } + + return blk_mq_poll_hybrid_sleep(q, hctx, rq); +} + +static int blk_mq_poll(struct request_queue *q, blk_qc_t cookie) +{ + struct blk_mq_hw_ctx *hctx; long state; + if (!test_bit(QUEUE_FLAG_POLL, &q->queue_flags)) + return 0; + + hctx = q->queue_hw_ctx[blk_qc_t_to_queue_num(cookie)]; + /* * If we sleep, have the caller restart the poll loop to reset * the state. Like for the other success return cases, the @@ -3342,7 +3369,7 @@ static int __blk_mq_poll(struct blk_mq_hw_ctx *hctx, struct request *rq) * the IO isn't complete, we'll get called again and will go * straight to the busy poll loop. */ - if (blk_mq_poll_hybrid_sleep(q, hctx, rq)) + if (blk_mq_poll_hybrid(q, hctx, cookie)) return 1; hctx->poll_considered++; @@ -3353,7 +3380,7 @@ static int __blk_mq_poll(struct blk_mq_hw_ctx *hctx, struct request *rq) hctx->poll_invoked++; - ret = q->mq_ops->poll(hctx, rq->tag); + ret = q->mq_ops->poll(hctx, -1U); if (ret > 0) { hctx->poll_success++; __set_current_state(TASK_RUNNING); @@ -3374,32 +3401,6 @@ static int __blk_mq_poll(struct blk_mq_hw_ctx *hctx, struct request *rq) return 0; } -static int blk_mq_poll(struct request_queue *q, blk_qc_t cookie) -{ - struct blk_mq_hw_ctx *hctx; - struct request *rq; - - if (!test_bit(QUEUE_FLAG_POLL, &q->queue_flags)) - return 0; - - hctx = q->queue_hw_ctx[blk_qc_t_to_queue_num(cookie)]; - if (!blk_qc_t_is_internal(cookie)) - rq = blk_mq_tag_to_rq(hctx->tags, blk_qc_t_to_tag(cookie)); - else { - rq = blk_mq_tag_to_rq(hctx->sched_tags, blk_qc_t_to_tag(cookie)); - /* - * With scheduling, if the request has completed, we'll - * get a NULL return here, as we clear the sched tag when - * that happens. The request still remains valid, like always, - * so we should be safe with just the NULL check. - */ - if (!rq) - return 0; - } - - return __blk_mq_poll(hctx, rq); -} - unsigned int blk_mq_rq_cpu(struct request *rq) { return rq->mq_ctx->cpu; diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c index 57e790391b82..de50d80ecc84 100644 --- a/drivers/nvme/host/pci.c +++ b/drivers/nvme/host/pci.c @@ -1012,15 +1012,15 @@ static inline void nvme_update_cq_head(struct nvme_queue *nvmeq) } } -static inline bool nvme_process_cq(struct nvme_queue *nvmeq, u16 *start, - u16 *end, int tag) +static inline int nvme_process_cq(struct nvme_queue *nvmeq, u16 *start, + u16 *end, unsigned int tag) { - bool found = false; + int found = 0; *start = nvmeq->cq_head; - while (!found && nvme_cqe_pending(nvmeq)) { - if (nvmeq->cqes[nvmeq->cq_head].command_id == tag) - found = true; + while (nvme_cqe_pending(nvmeq)) { + if (tag == -1U || nvmeq->cqes[nvmeq->cq_head].command_id == tag) + found++; nvme_update_cq_head(nvmeq); } *end = nvmeq->cq_head; @@ -1062,7 +1062,7 @@ static irqreturn_t nvme_irq_check(int irq, void *data) static int __nvme_poll(struct nvme_queue *nvmeq, unsigned int tag) { u16 start, end; - bool found; + int found; if (!nvme_cqe_pending(nvmeq)) return 0; diff --git a/drivers/nvme/host/rdma.c b/drivers/nvme/host/rdma.c index d181cafedc58..c2c3e1a5b7af 100644 --- a/drivers/nvme/host/rdma.c +++ b/drivers/nvme/host/rdma.c @@ -1409,12 +1409,11 @@ static void nvme_rdma_submit_async_event(struct nvme_ctrl *arg) WARN_ON_ONCE(ret); } -static int nvme_rdma_process_nvme_rsp(struct nvme_rdma_queue *queue, - struct nvme_completion *cqe, struct ib_wc *wc, int tag) +static void nvme_rdma_process_nvme_rsp(struct nvme_rdma_queue *queue, + struct nvme_completion *cqe, struct ib_wc *wc) { struct request *rq; struct nvme_rdma_request *req; - int ret = 0; rq = blk_mq_tag_to_rq(nvme_rdma_tagset(queue), cqe->command_id); if (!rq) { @@ -1422,7 +1421,7 @@ static int nvme_rdma_process_nvme_rsp(struct nvme_rdma_queue *queue, "tag 0x%x on QP %#x not found\n", cqe->command_id, queue->qp->qp_num); nvme_rdma_error_recovery(queue->ctrl); - return ret; + return; } req = blk_mq_rq_to_pdu(rq); @@ -1437,6 +1436,8 @@ static int nvme_rdma_process_nvme_rsp(struct nvme_rdma_queue *queue, nvme_rdma_error_recovery(queue->ctrl); } } else if (req->mr) { + int ret; + ret = nvme_rdma_inv_rkey(queue, req); if (unlikely(ret < 0)) { dev_err(queue->ctrl->ctrl.device, @@ -1445,19 +1446,14 @@ static int nvme_rdma_process_nvme_rsp(struct nvme_rdma_queue *queue, nvme_rdma_error_recovery(queue->ctrl); } /* the local invalidation completion will end the request */ - return 0; + return; } - if (refcount_dec_and_test(&req->ref)) { - if (rq->tag == tag) - ret = 1; + if (refcount_dec_and_test(&req->ref)) nvme_end_request(rq, req->status, req->result); - } - - return ret; } -static int __nvme_rdma_recv_done(struct ib_cq *cq, struct ib_wc *wc, int tag) +static void nvme_rdma_recv_done(struct ib_cq *cq, struct ib_wc *wc) { struct nvme_rdma_qe *qe = container_of(wc->wr_cqe, struct nvme_rdma_qe, cqe); @@ -1465,11 +1461,10 @@ static int __nvme_rdma_recv_done(struct ib_cq *cq, struct ib_wc *wc, int tag) struct ib_device *ibdev = queue->device->dev; struct nvme_completion *cqe = qe->data; const size_t len = sizeof(struct nvme_completion); - int ret = 0; if (unlikely(wc->status != IB_WC_SUCCESS)) { nvme_rdma_wr_error(cq, wc, "RECV"); - return 0; + return; } ib_dma_sync_single_for_cpu(ibdev, qe->dma, len, DMA_FROM_DEVICE); @@ -1484,16 +1479,10 @@ static int __nvme_rdma_recv_done(struct ib_cq *cq, struct ib_wc *wc, int tag) nvme_complete_async_event(&queue->ctrl->ctrl, cqe->status, &cqe->result); else - ret = nvme_rdma_process_nvme_rsp(queue, cqe, wc, tag); + nvme_rdma_process_nvme_rsp(queue, cqe, wc); ib_dma_sync_single_for_device(ibdev, qe->dma, len, DMA_FROM_DEVICE); nvme_rdma_post_recv(queue, qe); - return ret; -} - -static void nvme_rdma_recv_done(struct ib_cq *cq, struct ib_wc *wc) -{ - __nvme_rdma_recv_done(cq, wc, -1); } static int nvme_rdma_conn_established(struct nvme_rdma_queue *queue) @@ -1758,10 +1747,12 @@ static int nvme_rdma_poll(struct blk_mq_hw_ctx *hctx, unsigned int tag) struct ib_cqe *cqe = wc.wr_cqe; if (cqe) { - if (cqe->done == nvme_rdma_recv_done) - found |= __nvme_rdma_recv_done(cq, &wc, tag); - else + if (cqe->done == nvme_rdma_recv_done) { + nvme_rdma_recv_done(cq, &wc); + found++; + } else { cqe->done(cq, &wc); + } } } diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h index 9b53db06ad08..f3015e9b5ae3 100644 --- a/include/linux/blkdev.h +++ b/include/linux/blkdev.h @@ -867,7 +867,7 @@ extern void blk_execute_rq_nowait(struct request_queue *, struct gendisk *, int blk_status_to_errno(blk_status_t status); blk_status_t errno_to_blk_status(int errno); -bool blk_poll(struct request_queue *q, blk_qc_t cookie); +int blk_poll(struct request_queue *q, blk_qc_t cookie); static inline struct request_queue *bdev_get_queue(struct block_device *bdev) { From patchwork Fri Nov 23 18:34:07 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 10696263 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id F326E175A for ; Fri, 23 Nov 2018 18:34:22 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E40EF2968A for ; Fri, 23 Nov 2018 18:34:22 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id D80282BA7B; Fri, 23 Nov 2018 18:34:22 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 78D2A2968A for ; Fri, 23 Nov 2018 18:34:21 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726368AbeKXFTo (ORCPT ); Sat, 24 Nov 2018 00:19:44 -0500 Received: from mail-pg1-f194.google.com ([209.85.215.194]:42650 "EHLO mail-pg1-f194.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2389262AbeKXFTn (ORCPT ); Sat, 24 Nov 2018 00:19:43 -0500 Received: by mail-pg1-f194.google.com with SMTP id d72so3365946pga.9 for ; Fri, 23 Nov 2018 10:34:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=qbf7THVfKfKXHautth08FWEWMofN2ha8Jnuk3gPYbfc=; b=qSXeMJPFL9gsW4gIHe46rwtOsvgaVomkTl1t3ylO5XG2fgYqWsgBZ1839dQBlyJFNK 1GrXwQ75FEPaO8STano21k2yPxjvr1Ksl/CRRtPPD/soDRwvlrJx4cx7ijZaDDaqFUaf E9Qh3YMAwTPqe8hHpzqB8bw8FMiMdgTWXUeSAccUX+jN/29J/75gCe8gM1Knt7/LHFt0 0KOEon/6/KiNjN2pGTY+eetoSQ0T2rdgdLgN9iB0v9XcMGSLjCmVzA0ZRm701Y1jK1Y/ TMyEwgW7DWj8mLX5NJMklsU0i84YPa1XIJzGzpbySZ7icMmY1hSjCuomF4CNohScQ3ND UNmQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=qbf7THVfKfKXHautth08FWEWMofN2ha8Jnuk3gPYbfc=; b=lyS5w/RKq6hIUNOgd2qGfRSbkzK7HUZmHAnpvgxBX/7bAxFKKxhKHVnSHiC1F4x6Sz skKrdlfv0K0qmprLOOI9a+rLMtKRbnVPeSxbipnCzuJZSF94uwrjpnQP2I024JAqECU/ xQV0Jc98cVwh/umLw4aE+aeNl/U2dRRYAQGZcUJW5XFW4JKLM1oCGgrpRMVEfeHL7G0k Qx2Et7vUC2WhLL8VbzgWsjSHIo7ocJbSCttX+zs3iHbrpUTh7cGhWH2b8bPQTnr/6oV2 NBzwWiSUcWb6EXcNCJHNz1kK+AxZIqEWTMgWJ3uunSxOg3yt4FwjNVP89Sy1nbP+lnc0 fdPw== X-Gm-Message-State: AGRZ1gKO+oAqsJayAExPRirPuYrvzv6YMUFERDbJY5eiz2VVEpDLbPPg ZYdCTDPGbjdAE+DB3gZ5UY5aTsC4FFE= X-Google-Smtp-Source: AJdET5egHqJvGclTe+5PKMxVSrabW/6ejRfiBWLPsL59bifod8/6U9ln4tpw3zD/fUw1JMH/HOq7+g== X-Received: by 2002:a62:5dd1:: with SMTP id n78mr16805683pfj.58.1542998058730; Fri, 23 Nov 2018 10:34:18 -0800 (PST) Received: from x1.thefacebook.com (66.29.188.166.static.utbb.net. [66.29.188.166]) by smtp.gmail.com with ESMTPSA id s37sm52294260pgm.19.2018.11.23.10.34.17 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 23 Nov 2018 10:34:17 -0800 (PST) From: Jens Axboe To: linux-block@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 2/6] blk-mq: remove 'tag' parameter from mq_ops->poll() Date: Fri, 23 Nov 2018 11:34:07 -0700 Message-Id: <20181123183411.8708-3-axboe@kernel.dk> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181123183411.8708-1-axboe@kernel.dk> References: <20181123183411.8708-1-axboe@kernel.dk> Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP We always pass in -1 now and none of the callers use the tag value, remove the parameter. Reviewed-by: Christoph Hellwig Signed-off-by: Jens Axboe --- block/blk-mq.c | 2 +- drivers/nvme/host/pci.c | 8 ++++---- drivers/nvme/host/rdma.c | 2 +- include/linux/blk-mq.h | 2 +- 4 files changed, 7 insertions(+), 7 deletions(-) diff --git a/block/blk-mq.c b/block/blk-mq.c index ec6c79578332..b66cca3ce1e5 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -3380,7 +3380,7 @@ static int blk_mq_poll(struct request_queue *q, blk_qc_t cookie) hctx->poll_invoked++; - ret = q->mq_ops->poll(hctx, -1U); + ret = q->mq_ops->poll(hctx); if (ret > 0) { hctx->poll_success++; __set_current_state(TASK_RUNNING); diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c index de50d80ecc84..73effe586e5f 100644 --- a/drivers/nvme/host/pci.c +++ b/drivers/nvme/host/pci.c @@ -1075,14 +1075,14 @@ static int __nvme_poll(struct nvme_queue *nvmeq, unsigned int tag) return found; } -static int nvme_poll(struct blk_mq_hw_ctx *hctx, unsigned int tag) +static int nvme_poll(struct blk_mq_hw_ctx *hctx) { struct nvme_queue *nvmeq = hctx->driver_data; - return __nvme_poll(nvmeq, tag); + return __nvme_poll(nvmeq, -1); } -static int nvme_poll_noirq(struct blk_mq_hw_ctx *hctx, unsigned int tag) +static int nvme_poll_noirq(struct blk_mq_hw_ctx *hctx) { struct nvme_queue *nvmeq = hctx->driver_data; u16 start, end; @@ -1092,7 +1092,7 @@ static int nvme_poll_noirq(struct blk_mq_hw_ctx *hctx, unsigned int tag) return 0; spin_lock(&nvmeq->cq_lock); - found = nvme_process_cq(nvmeq, &start, &end, tag); + found = nvme_process_cq(nvmeq, &start, &end, -1); spin_unlock(&nvmeq->cq_lock); nvme_complete_cqes(nvmeq, start, end); diff --git a/drivers/nvme/host/rdma.c b/drivers/nvme/host/rdma.c index c2c3e1a5b7af..ccfde6c7c0a5 100644 --- a/drivers/nvme/host/rdma.c +++ b/drivers/nvme/host/rdma.c @@ -1736,7 +1736,7 @@ static blk_status_t nvme_rdma_queue_rq(struct blk_mq_hw_ctx *hctx, return BLK_STS_IOERR; } -static int nvme_rdma_poll(struct blk_mq_hw_ctx *hctx, unsigned int tag) +static int nvme_rdma_poll(struct blk_mq_hw_ctx *hctx) { struct nvme_rdma_queue *queue = hctx->driver_data; struct ib_cq *cq = queue->ib_cq; diff --git a/include/linux/blk-mq.h b/include/linux/blk-mq.h index 929e8abc5535..ca0520ca6437 100644 --- a/include/linux/blk-mq.h +++ b/include/linux/blk-mq.h @@ -132,7 +132,7 @@ typedef void (exit_request_fn)(struct blk_mq_tag_set *set, struct request *, typedef bool (busy_iter_fn)(struct blk_mq_hw_ctx *, struct request *, void *, bool); typedef bool (busy_tag_iter_fn)(struct request *, void *, bool); -typedef int (poll_fn)(struct blk_mq_hw_ctx *, unsigned int); +typedef int (poll_fn)(struct blk_mq_hw_ctx *); typedef int (map_queues_fn)(struct blk_mq_tag_set *set); typedef bool (busy_fn)(struct request_queue *); typedef void (complete_fn)(struct request *); From patchwork Fri Nov 23 18:34:08 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 10696265 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id C235713AD for ; Fri, 23 Nov 2018 18:34:24 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B2B9C2968A for ; Fri, 23 Nov 2018 18:34:24 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id A72122BA7B; Fri, 23 Nov 2018 18:34:24 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 5FFED2968A for ; Fri, 23 Nov 2018 18:34:24 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2389262AbeKXFTq (ORCPT ); Sat, 24 Nov 2018 00:19:46 -0500 Received: from mail-pf1-f195.google.com ([209.85.210.195]:33213 "EHLO mail-pf1-f195.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2391577AbeKXFTp (ORCPT ); Sat, 24 Nov 2018 00:19:45 -0500 Received: by mail-pf1-f195.google.com with SMTP id c123so2984983pfb.0 for ; Fri, 23 Nov 2018 10:34:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=Dwmb4KEzJ3RrNx1bsdmJxba6f/9aWI5P0VkiLMifhJ8=; b=khsHIbZFKhVLzIw9QQo4SZny7rDIWQJxNCghMeH0tU1zvIJMtAKp2Sk7FqFAjX4vHM LyMTlm2maUpmJq11fterAu5ctCvYIomhlmcAv5eRcSucsw4My+k05B0A041MrNLht2MX w0qTxRou50exed+gbDnFvzpxmWEIER3/bJyZpnX6W9YRc8fFkMcRGK8EOlcMQ4N1n5lT 4lmxH9KydSf7+m2hmYqG0voKmuc8iR9/GU3WQRM28ieuKj5i241Ai6SiItigFuux5OIW LCtP7aSQTx6qiEqLEZdNUPgR/QXNX3Q8F8UIhLVjWBbZcmMVS9/C002MW+58okpy9wo4 xNUA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=Dwmb4KEzJ3RrNx1bsdmJxba6f/9aWI5P0VkiLMifhJ8=; b=LkAfaXHsDz7uZnOf9HjDlAFv4x42ckMd9sJbf5t4OP+zLgKRroluCyENSn3YFeea6F CUxX6FY6Tt+OquT4KSqfAXBqoF9fwbqF+xaWLZ3BgPTxTC9vqpEn5gFKFNutYlp9WduJ DTULV/Nfv8nJFv1cyh8/wj0o/qonFN/9yaCaYxzdu/xd3ZWORgZuxyt+3VAxEaYqihE1 R/p2NIyT0k4uIVhtY7FX3LKxcg5NNLVlE3sWNsQBjiEZ/+hAu9G6aN18dZvO3tTYNnI8 buRumR4fR1zL+r1z+YWEfEXZwPax9lOfgH0uf+g7Kyh18Itdn6NeC1Cn6hD6rdWNvCEt XmSg== X-Gm-Message-State: AA+aEWZZeME6JedvBtWsSrOD7BsRsM+JzAHurAh9nLsWQFWDFTomM3uH OiiTE2dYAzrV0SCaR17rZk+7pq5E8SQ= X-Google-Smtp-Source: AFSGD/UAj8XypGdSTB7q0oXikBFFJT6yTzGHKrS0y7RAmaerVLe0i9WTTVTbAAR6SlpnUaG6zxnCZQ== X-Received: by 2002:a62:8f8c:: with SMTP id n134mr3220980pfd.137.1542998060438; Fri, 23 Nov 2018 10:34:20 -0800 (PST) Received: from x1.thefacebook.com (66.29.188.166.static.utbb.net. [66.29.188.166]) by smtp.gmail.com with ESMTPSA id s37sm52294260pgm.19.2018.11.23.10.34.18 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 23 Nov 2018 10:34:19 -0800 (PST) From: Jens Axboe To: linux-block@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 3/6] nvme: remove opportunistic polling from bdev target Date: Fri, 23 Nov 2018 11:34:08 -0700 Message-Id: <20181123183411.8708-4-axboe@kernel.dk> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181123183411.8708-1-axboe@kernel.dk> References: <20181123183411.8708-1-axboe@kernel.dk> Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP It doesn't set HIPRI on the bio, so polling for it is pretty silly. Signed-off-by: Jens Axboe Reviewed-by: Christoph Hellwig --- drivers/nvme/target/io-cmd-bdev.c | 2 -- 1 file changed, 2 deletions(-) diff --git a/drivers/nvme/target/io-cmd-bdev.c b/drivers/nvme/target/io-cmd-bdev.c index c1ec3475a140..c1cb2ed5531c 100644 --- a/drivers/nvme/target/io-cmd-bdev.c +++ b/drivers/nvme/target/io-cmd-bdev.c @@ -115,8 +115,6 @@ static void nvmet_bdev_execute_rw(struct nvmet_req *req) } cookie = submit_bio(bio); - - blk_poll(bdev_get_queue(req->ns->bdev), cookie); } static void nvmet_bdev_execute_flush(struct nvmet_req *req) From patchwork Fri Nov 23 18:34:09 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 10696267 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 8123C13AD for ; Fri, 23 Nov 2018 18:34:27 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 709762BA7B for ; Fri, 23 Nov 2018 18:34:27 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 6325C2BB6C; Fri, 23 Nov 2018 18:34:27 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C04F12968A for ; Fri, 23 Nov 2018 18:34:26 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2395280AbeKXFTt (ORCPT ); Sat, 24 Nov 2018 00:19:49 -0500 Received: from mail-pg1-f194.google.com ([209.85.215.194]:35871 "EHLO mail-pg1-f194.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2391577AbeKXFTr (ORCPT ); Sat, 24 Nov 2018 00:19:47 -0500 Received: by mail-pg1-f194.google.com with SMTP id n2so3379879pgm.3 for ; Fri, 23 Nov 2018 10:34:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=B1nsWSJKOOcjQaULw0z7/oVTYVRfudblENSpnK7eGZ8=; b=ni/CP0J18Ys2Y8Be+uFtJHaiJHvMlJV8XtELpONKtAwgqtlClEr774LIpwrAVp2NJH gCWRzROAcHGXH0slnUJNFtVPbA858SMnoK7itKTPFYYnZhkxqFy0flSqAJjAVHMXRJWc 2nuoqBjPNnmW4C5wlyhnhURMJttDrcUjzD9JBf6Aa+eMASqwrRh5JB/lmfE+SGjtFd6h LGLbOZUytVkcZfgv3MFCChGwT2TpOSCa+4L5wzKGdxQbzRzOnW3awicc0Hg5sQN/djqj zyK1Ck4vnpy+nMICyTQbYzOFOEyHHdDI81C815GmLH/EzVbwHCausK4W7APl5OovqyL2 4AuQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=B1nsWSJKOOcjQaULw0z7/oVTYVRfudblENSpnK7eGZ8=; b=gT9KimL3IhKrdhtUZ6MKVoWbwOUwtXi+axW/qbqH6l0lkVuvX6K+C8rJaxh8GSxNFD Nha2V7hNBWwiCBJv16VN0fGql8HUA8fNT0SyOGehCnooQ3iAEoo86CZZ7UGlzA9neBFp 7FEP+7FrprJWVcwyp/I1vVUhb6L/h8jdnv6JZRsKnlh0w1XOOR7fHyLgMzv3INTSfx/l RSedck2Cnb1VaZENAjZCeOeHDCDdljDK5FnSvqYw9HNVxH69K/aN9GdwzNIqyEFVmeGT klliXhmKo+irkr+I7tNzDWXR4fnqEqO6rdrzc15VeiXw5XAGX3fsFs+j253fDEOp5K5n U2mA== X-Gm-Message-State: AGRZ1gLn4R7LmrkTTXxYulYiqk3Ff9wQeq3grMt9aisl5mZpk8VhCLOk 6df9iBUzLhZCYq/8065Klrb4R3WeGD4= X-Google-Smtp-Source: AJdET5fUOAwSrW4W2e9e/RDDwvMn2DWaEOz38DUDgkT/XMJdVRSdh1tTTglFjd1GmX/jSQcj4Nnk0w== X-Received: by 2002:a62:528e:: with SMTP id g136mr17814574pfb.111.1542998062297; Fri, 23 Nov 2018 10:34:22 -0800 (PST) Received: from x1.thefacebook.com (66.29.188.166.static.utbb.net. [66.29.188.166]) by smtp.gmail.com with ESMTPSA id s37sm52294260pgm.19.2018.11.23.10.34.20 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 23 Nov 2018 10:34:21 -0800 (PST) From: Jens Axboe To: linux-block@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 4/6] block: make blk_poll() take a parameter on whether to spin or not Date: Fri, 23 Nov 2018 11:34:09 -0700 Message-Id: <20181123183411.8708-5-axboe@kernel.dk> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181123183411.8708-1-axboe@kernel.dk> References: <20181123183411.8708-1-axboe@kernel.dk> Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP blk_poll() has always kept spinning until it found an IO. This is fine for SYNC polling, since we need to find one request we have pending, but in preparation for ASYNC polling it can be beneficial to just check if we have any entries available or not. Existing callers are converted to pass in 'spin == true', to retain the old behavior. Signed-off-by: Jens Axboe --- block/blk-core.c | 4 ++-- block/blk-mq.c | 6 +++--- drivers/nvme/host/multipath.c | 4 ++-- fs/block_dev.c | 4 ++-- fs/direct-io.c | 2 +- fs/iomap.c | 2 +- include/linux/blkdev.h | 4 ++-- mm/page_io.c | 2 +- 8 files changed, 14 insertions(+), 14 deletions(-) diff --git a/block/blk-core.c b/block/blk-core.c index 9e99aa852d6e..f7ffc43ada14 100644 --- a/block/blk-core.c +++ b/block/blk-core.c @@ -1273,14 +1273,14 @@ blk_qc_t submit_bio(struct bio *bio) } EXPORT_SYMBOL(submit_bio); -int blk_poll(struct request_queue *q, blk_qc_t cookie) +int blk_poll(struct request_queue *q, blk_qc_t cookie, bool spin) { if (!q->poll_fn || !blk_qc_t_valid(cookie)) return false; if (current->plug) blk_flush_plug_list(current->plug, false); - return q->poll_fn(q, cookie); + return q->poll_fn(q, cookie, spin); } EXPORT_SYMBOL_GPL(blk_poll); diff --git a/block/blk-mq.c b/block/blk-mq.c index b66cca3ce1e5..c2751f0a3ccc 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -38,7 +38,7 @@ #include "blk-mq-sched.h" #include "blk-rq-qos.h" -static int blk_mq_poll(struct request_queue *q, blk_qc_t cookie); +static int blk_mq_poll(struct request_queue *q, blk_qc_t cookie, bool spin); static void blk_mq_poll_stats_start(struct request_queue *q); static void blk_mq_poll_stats_fn(struct blk_stat_callback *cb); @@ -3352,7 +3352,7 @@ static bool blk_mq_poll_hybrid(struct request_queue *q, return blk_mq_poll_hybrid_sleep(q, hctx, rq); } -static int blk_mq_poll(struct request_queue *q, blk_qc_t cookie) +static int blk_mq_poll(struct request_queue *q, blk_qc_t cookie, bool spin) { struct blk_mq_hw_ctx *hctx; long state; @@ -3392,7 +3392,7 @@ static int blk_mq_poll(struct request_queue *q, blk_qc_t cookie) if (current->state == TASK_RUNNING) return 1; - if (ret < 0) + if (ret < 0 || !spin) break; cpu_relax(); } diff --git a/drivers/nvme/host/multipath.c b/drivers/nvme/host/multipath.c index f9eeb3b58632..ffebdd0ae34b 100644 --- a/drivers/nvme/host/multipath.c +++ b/drivers/nvme/host/multipath.c @@ -220,7 +220,7 @@ static blk_qc_t nvme_ns_head_make_request(struct request_queue *q, return ret; } -static int nvme_ns_head_poll(struct request_queue *q, blk_qc_t qc) +static int nvme_ns_head_poll(struct request_queue *q, blk_qc_t qc, bool spin) { struct nvme_ns_head *head = q->queuedata; struct nvme_ns *ns; @@ -230,7 +230,7 @@ static int nvme_ns_head_poll(struct request_queue *q, blk_qc_t qc) srcu_idx = srcu_read_lock(&head->srcu); ns = srcu_dereference(head->current_path[numa_node_id()], &head->srcu); if (likely(ns && nvme_path_is_optimized(ns))) - found = ns->queue->poll_fn(q, qc); + found = ns->queue->poll_fn(q, qc, spin); srcu_read_unlock(&head->srcu, srcu_idx); return found; } diff --git a/fs/block_dev.c b/fs/block_dev.c index 64ba27b8b754..d233a59ea364 100644 --- a/fs/block_dev.c +++ b/fs/block_dev.c @@ -243,7 +243,7 @@ __blkdev_direct_IO_simple(struct kiocb *iocb, struct iov_iter *iter, break; if (!(iocb->ki_flags & IOCB_HIPRI) || - !blk_poll(bdev_get_queue(bdev), qc)) + !blk_poll(bdev_get_queue(bdev), qc, true)) io_schedule(); } __set_current_state(TASK_RUNNING); @@ -423,7 +423,7 @@ __blkdev_direct_IO(struct kiocb *iocb, struct iov_iter *iter, int nr_pages) break; if (!(iocb->ki_flags & IOCB_HIPRI) || - !blk_poll(bdev_get_queue(bdev), qc)) + !blk_poll(bdev_get_queue(bdev), qc, true)) io_schedule(); } __set_current_state(TASK_RUNNING); diff --git a/fs/direct-io.c b/fs/direct-io.c index ea07d5a34317..a5a4e5a1423e 100644 --- a/fs/direct-io.c +++ b/fs/direct-io.c @@ -518,7 +518,7 @@ static struct bio *dio_await_one(struct dio *dio) dio->waiter = current; spin_unlock_irqrestore(&dio->bio_lock, flags); if (!(dio->iocb->ki_flags & IOCB_HIPRI) || - !blk_poll(dio->bio_disk->queue, dio->bio_cookie)) + !blk_poll(dio->bio_disk->queue, dio->bio_cookie, true)) io_schedule(); /* wake up sets us TASK_RUNNING */ spin_lock_irqsave(&dio->bio_lock, flags); diff --git a/fs/iomap.c b/fs/iomap.c index c5df035ace6f..74c1f37f0fd6 100644 --- a/fs/iomap.c +++ b/fs/iomap.c @@ -1896,7 +1896,7 @@ iomap_dio_rw(struct kiocb *iocb, struct iov_iter *iter, if (!(iocb->ki_flags & IOCB_HIPRI) || !dio->submit.last_queue || !blk_poll(dio->submit.last_queue, - dio->submit.cookie)) + dio->submit.cookie, true)) io_schedule(); } __set_current_state(TASK_RUNNING); diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h index f3015e9b5ae3..e3c0a8ec16a7 100644 --- a/include/linux/blkdev.h +++ b/include/linux/blkdev.h @@ -283,7 +283,7 @@ static inline unsigned short req_get_ioprio(struct request *req) struct blk_queue_ctx; typedef blk_qc_t (make_request_fn) (struct request_queue *q, struct bio *bio); -typedef int (poll_q_fn) (struct request_queue *q, blk_qc_t); +typedef int (poll_q_fn) (struct request_queue *q, blk_qc_t, bool spin); struct bio_vec; typedef int (dma_drain_needed_fn)(struct request *); @@ -867,7 +867,7 @@ extern void blk_execute_rq_nowait(struct request_queue *, struct gendisk *, int blk_status_to_errno(blk_status_t status); blk_status_t errno_to_blk_status(int errno); -int blk_poll(struct request_queue *q, blk_qc_t cookie); +int blk_poll(struct request_queue *q, blk_qc_t cookie, bool spin); static inline struct request_queue *bdev_get_queue(struct block_device *bdev) { diff --git a/mm/page_io.c b/mm/page_io.c index a7271fa481f6..5bdfd21c1bd9 100644 --- a/mm/page_io.c +++ b/mm/page_io.c @@ -410,7 +410,7 @@ int swap_readpage(struct page *page, bool synchronous) if (!READ_ONCE(bio->bi_private)) break; - if (!blk_poll(disk->queue, qc)) + if (!blk_poll(disk->queue, qc, true)) break; } __set_current_state(TASK_RUNNING); From patchwork Fri Nov 23 18:34:10 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 10696269 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 8D203175A for ; Fri, 23 Nov 2018 18:34:27 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7C8962968A for ; Fri, 23 Nov 2018 18:34:27 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 7111C2BA93; Fri, 23 Nov 2018 18:34:27 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 234A02B533 for ; Fri, 23 Nov 2018 18:34:27 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2395347AbeKXFTu (ORCPT ); Sat, 24 Nov 2018 00:19:50 -0500 Received: from mail-pg1-f194.google.com ([209.85.215.194]:43226 "EHLO mail-pg1-f194.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2395303AbeKXFTt (ORCPT ); Sat, 24 Nov 2018 00:19:49 -0500 Received: by mail-pg1-f194.google.com with SMTP id v28so3364286pgk.10 for ; Fri, 23 Nov 2018 10:34:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=MzbWbo9n2Y/u3PwfH0Qv5yW2T3HdVyOokQf5GbAztyc=; b=k01RBKiyGd5Uc6Lx4XrGzKH448UFAMA0Rxwb7vM8CWgBhCqFVhBInOP9Gm8z4ySj0r tQ8QhZzsSZRm777rpJN9OGcq23gpg6SlMtIS18LlH5uISLaQoJWC6uQNDODUo50iKyLz kiWjnr5H3MyGRnwGDCj+Tvb7nplcJk1C122oCxivaIDf46NCWRcDjbFciPXSIfpEVgMf 8bZ1l65Ieu5BST7FWbZ27Cb8f+qV0hq3vDfyEAEXfrgHB9EXN8viviX/2z+yDNUcixjt A2OPGg+B1lXg31u8Dmp8RNUR02y8AkpAvdUby18etzza1IKOZ1D1G3J3cWlW6+5nNr6D d+pQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=MzbWbo9n2Y/u3PwfH0Qv5yW2T3HdVyOokQf5GbAztyc=; b=Ka/GU4Xp8uT0uIu9WOiKnTktHX58nrW+8HJgmdCxGz6HW0vpCNBokzTAW2WBsM1gTm fColf2qvMq1Mhggr6QEfeql1FIiZSPuEmxFVuro5/8XztGCI4400mK5qBZOnZ4POvyuf Ia6PC3p9I8PpgDFSZ0TojVQVRV1h1oJDkOmv+73ynS1FK8/LXrK+GJOc+CWJygIrsW5V eWZ5AF4ZdZVAtY//fwgii+mqUWd5G1QE6/jP0/WLmcYvJjfKOn/+NGqQR/mKikV7lv67 uUCaVsKqG836NrnDZFavPr3f9S/r05BbBXQawumYBEfHoNL+EjqJUG5Wox70i9iXXdsC dedA== X-Gm-Message-State: AA+aEWZ9I3kEFk2xdQWTcuPCudEAKZvRecEpVH66GNItqjzydYhqeTrt fT3teSKqhFw2MvgU0ApGdtJCDCScp3s= X-Google-Smtp-Source: AFSGD/Vaf0Iq7EynLoS+OLdv+SeJGy+7MwcUJV2WiVsVquFe6eJibD6oml5hSNocpKGAyWXd2L6Spg== X-Received: by 2002:a63:5b48:: with SMTP id l8mr15193444pgm.80.1542998063787; Fri, 23 Nov 2018 10:34:23 -0800 (PST) Received: from x1.thefacebook.com (66.29.188.166.static.utbb.net. [66.29.188.166]) by smtp.gmail.com with ESMTPSA id s37sm52294260pgm.19.2018.11.23.10.34.22 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 23 Nov 2018 10:34:22 -0800 (PST) From: Jens Axboe To: linux-block@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 5/6] blk-mq: ensure mq_ops ->poll() is entered at least once Date: Fri, 23 Nov 2018 11:34:10 -0700 Message-Id: <20181123183411.8708-6-axboe@kernel.dk> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181123183411.8708-1-axboe@kernel.dk> References: <20181123183411.8708-1-axboe@kernel.dk> Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Right now we immediately bail if need_resched() is true, but we need to do at least one loop in case we have entries waiting. So just invert the need_resched() check, putting it at the bottom of the loop. Reviewed-by: Christoph Hellwig Signed-off-by: Jens Axboe --- block/blk-mq.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/block/blk-mq.c b/block/blk-mq.c index c2751f0a3ccc..ba3c7b6476b7 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -3375,7 +3375,7 @@ static int blk_mq_poll(struct request_queue *q, blk_qc_t cookie, bool spin) hctx->poll_considered++; state = current->state; - while (!need_resched()) { + do { int ret; hctx->poll_invoked++; @@ -3395,7 +3395,7 @@ static int blk_mq_poll(struct request_queue *q, blk_qc_t cookie, bool spin) if (ret < 0 || !spin) break; cpu_relax(); - } + } while (!need_resched()); __set_current_state(TASK_RUNNING); return 0; From patchwork Fri Nov 23 18:34:11 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 10696271 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 19B4513AD for ; Fri, 23 Nov 2018 18:34:30 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0A7D72B533 for ; Fri, 23 Nov 2018 18:34:30 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id F349B2BB6C; Fri, 23 Nov 2018 18:34:29 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A87DA2B533 for ; Fri, 23 Nov 2018 18:34:29 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2395303AbeKXFTw (ORCPT ); Sat, 24 Nov 2018 00:19:52 -0500 Received: from mail-pg1-f195.google.com ([209.85.215.195]:36971 "EHLO mail-pg1-f195.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2391577AbeKXFTv (ORCPT ); Sat, 24 Nov 2018 00:19:51 -0500 Received: by mail-pg1-f195.google.com with SMTP id 80so3379012pge.4 for ; Fri, 23 Nov 2018 10:34:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=xoHXsz5i1U24MlbYVPJenFoEGlB/YU7byvbkdX2oJsY=; b=PNrCgJqwBBKX+K+gMDO2afHWKPeSztXiTdqTpQLl2ZDQAHxXgClD12avT+AU8t9FIt uu9Zpth+pqBk9nZkRs/kxGXAD6sOJgKTfrkl1d6rX8Wu34J5BpB52OQMvU/MHt0IRnFv KqfjGeUqeIgdruNms2/K8vFcF5+kFp+E3VlYkfoOsg/yUDufhp6dnPxpHXRMuCqmLyKQ mA/kzsWE/4SOxqhJEbndikGYrmr42WvD1+h4JosJDl3jNcOqUbEF9Q6fDE5DwWeXVNSB WySjIMAOz2x9bayJ6lC825P80xGOTng6uKVYdWZXyUrpOEp9ySNJs8Ylh1ZOrIOL7t1L i3zA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=xoHXsz5i1U24MlbYVPJenFoEGlB/YU7byvbkdX2oJsY=; b=KH1UwW08BVvq7RT2i0P93KbwBwI+PvWAOJ3hkJttmrVCGb9BdfOR84iqOgHV/xlbqf qTvqKi/8Xb8HRXeUcipEWy7DsjirLqnLOV2f/DpaWinthVUZxvpMxQLs6BJuqHpDFWmZ 9SB8fgM8Y71N0Z8sq5d3I0mqjVteCnTQP+Z/uWT5v3sc56roPBoOouInF8aUgRswpxQ0 yKVPIpdD9XOz+Wt4mGOavEEz4CpE6z862j3ckVVxkxSQdb/kDKA5FzYbGhUa5TB6C1j7 8sDlEgonZn5zo+/ixbJrdk9H523gKBbMvB958sruPjHxyeY5DgPVil6+MN4HdBpl18aH rpWQ== X-Gm-Message-State: AGRZ1gKa5UhHixe2m1dGvQkOJol/VShf3H35D99FE4f9w1f/MexUvi2V ND9aKJ7j1mvlN+EVEB9/qlO+WDc2UcE= X-Google-Smtp-Source: AJdET5ejkCdg6NJcUI8WeDPYfubq7oMGfmO0yFl7Bi+nrpST3Do2reFtHrgICGDAbOnSCX0MexAEdQ== X-Received: by 2002:a62:5d0c:: with SMTP id r12mr17615236pfb.0.1542998065654; Fri, 23 Nov 2018 10:34:25 -0800 (PST) Received: from x1.thefacebook.com (66.29.188.166.static.utbb.net. [66.29.188.166]) by smtp.gmail.com with ESMTPSA id s37sm52294260pgm.19.2018.11.23.10.34.23 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 23 Nov 2018 10:34:24 -0800 (PST) From: Jens Axboe To: linux-block@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 6/6] blk-mq: never redirect polled IO completions Date: Fri, 23 Nov 2018 11:34:11 -0700 Message-Id: <20181123183411.8708-7-axboe@kernel.dk> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181123183411.8708-1-axboe@kernel.dk> References: <20181123183411.8708-1-axboe@kernel.dk> Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP It's pointless to do so, we are by definition on the CPU we want/need to be, as that's the one waiting for a completion event. Signed-off-by: Jens Axboe Reviewed-by: Christoph Hellwig --- block/blk-mq.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/block/blk-mq.c b/block/blk-mq.c index ba3c7b6476b7..37674c1766a7 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -585,7 +585,12 @@ static void __blk_mq_complete_request(struct request *rq) return; } - if (!test_bit(QUEUE_FLAG_SAME_COMP, &q->queue_flags)) { + /* + * For a polled request, always complete locallly, it's pointless + * to redirect the completion. + */ + if ((rq->cmd_flags & REQ_HIPRI) || + !test_bit(QUEUE_FLAG_SAME_COMP, &q->queue_flags)) { q->mq_ops->complete(rq); return; }