From patchwork Tue Jan 9 16:29:51 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tejun Heo X-Patchwork-Id: 10152835 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id B91CC60223 for ; Tue, 9 Jan 2018 16:30:27 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C95D0285B9 for ; Tue, 9 Jan 2018 16:30:27 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id C81D92871D; Tue, 9 Jan 2018 16:30:27 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.8 required=2.0 tests=BAYES_00,DKIM_SIGNED, RCVD_IN_DNSWL_HI,T_DKIM_INVALID autolearn=unavailable version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 596E92873C for ; Tue, 9 Jan 2018 16:30:27 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1759475AbeAIQaZ (ORCPT ); Tue, 9 Jan 2018 11:30:25 -0500 Received: from mail-qt0-f194.google.com ([209.85.216.194]:35290 "EHLO mail-qt0-f194.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1759557AbeAIQaU (ORCPT ); Tue, 9 Jan 2018 11:30:20 -0500 Received: by mail-qt0-f194.google.com with SMTP id u10so18485754qtg.2; Tue, 09 Jan 2018 08:30:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:from:to:cc:subject:date:message-id:in-reply-to:references; bh=Me4JwB5kXIGTjrIGEjpuQf5UKr1woRsuqoc3e51vjr4=; b=R1SpowzFc8UGfGJcw2Oys1GXOSkpDkupAkDcB5KO96YOh0fDUc9lHYKw06Ax/PpByT 0VujkmWhsKdbtW+pXyMcuxjobJoYDXX6AwmDIKs6YRieqOdDgr0+sTqDp6QGip5VLS+t I1E2flPDZwEYRSgHVeAeWc0BNGGU4a9IJMT/fHjYg95CvEZh6vzaiqtG8yyefOSRdP5Y v9CUouE62Lzn93JUl0lPvs1xsPsdcfIguh5PvD3gLPB7LrX+FiMuRzk0E7m+7MD8Tgb6 rYm7BBoNLa7UHrGezWVS63PKelrOWYIyGIHMgWhiySEpUraa6/Jbma3Czfs1vEppxS/P XhQQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:from:to:cc:subject:date:message-id :in-reply-to:references; bh=Me4JwB5kXIGTjrIGEjpuQf5UKr1woRsuqoc3e51vjr4=; b=BWKPdbvYyQb0U/FzDe9oQQlMJOEuo9nyU9S6p94sAChQPe0JFjTCLQLx+Q+cc5Uogn sPD+IfS1eVoDkKLUeGKuk2MJTh/zybZg2W03RPaHOdHDEMoXkbVtrTPBZCaDvmjD+pRX QRPZ5uQJQQUyE0qOyThirnr7o/tlqUJseC2+8m/C/B18DptmJqvZEJ/cPRQFHI0sDMXh 6Zfg2iggX+fqkXLhIVVsIOmI7GFU+TxJUf26/JprtuJElsZCvPA2nLObMPROEb+lxhvQ o5/to+zGq8c9keZCH0p5ZdC6MqyTzfbtL7dCSpd6AVqY6M1OCrSbrMHN/5SD//16I6Z6 czzQ== X-Gm-Message-State: AKwxytcC/AOI2a8XnJ6cYzr13Egny7Bt9DuHjmwECb1Gd1XkfntzDZqd 7GJBLygzU/E5+nsgMBpzbF0= X-Google-Smtp-Source: ACJfBouwsSJ6F3XSx659RPfEgddpO+otS17wi5eUU0f56z+2O7Q4i9fgKgTGRPboIHMCs3w+iRh7xA== X-Received: by 10.200.50.253 with SMTP id a58mr21935945qtb.13.1515515419607; Tue, 09 Jan 2018 08:30:19 -0800 (PST) Received: from localhost (dhcp-ec-8-6b-ed-7a-cf.cpe.echoes.net. [72.28.5.223]) by smtp.gmail.com with ESMTPSA id i37sm1376635qtc.80.2018.01.09.08.30.18 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 09 Jan 2018 08:30:18 -0800 (PST) From: Tejun Heo To: jack@suse.cz, axboe@kernel.dk, clm@fb.com, jbacik@fb.com Cc: kernel-team@fb.com, linux-kernel@vger.kernel.org, linux-btrfs@vger.kernel.org, peterz@infradead.org, jianchao.w.wang@oracle.com, Bart.VanAssche@wdc.com, linux-block@vger.kernel.org, Tejun Heo Subject: [PATCH 6/8] blk-mq: remove REQ_ATOM_COMPLETE usages from blk-mq Date: Tue, 9 Jan 2018 08:29:51 -0800 Message-Id: <20180109162953.1211451-7-tj@kernel.org> X-Mailer: git-send-email 2.9.5 In-Reply-To: <20180109162953.1211451-1-tj@kernel.org> References: <20180109162953.1211451-1-tj@kernel.org> Sender: linux-btrfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP After the recent updates to use generation number and state based synchronization, blk-mq no longer depends on REQ_ATOM_COMPLETE except to avoid firing the same timeout multiple times. Remove all REQ_ATOM_COMPLETE usages and use a new rq_flags flag RQF_MQ_TIMEOUT_EXPIRED to avoid firing the same timeout multiple times. This removes atomic bitops from hot paths too. v2: Removed blk_clear_rq_complete() from blk_mq_rq_timed_out(). v3: Added RQF_MQ_TIMEOUT_EXPIRED flag. Signed-off-by: Tejun Heo Cc: "jianchao.wang" --- block/blk-mq.c | 15 +++++++-------- block/blk-timeout.c | 1 + include/linux/blkdev.h | 2 ++ 3 files changed, 10 insertions(+), 8 deletions(-) diff --git a/block/blk-mq.c b/block/blk-mq.c index b419746..3f297f0 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -634,8 +634,7 @@ void blk_mq_complete_request(struct request *rq) * hctx_lock() covers both issue and completion paths. */ hctx_lock(hctx, &srcu_idx); - if (blk_mq_rq_aborted_gstate(rq) != rq->gstate && - !blk_mark_rq_complete(rq)) + if (blk_mq_rq_aborted_gstate(rq) != rq->gstate) __blk_mq_complete_request(rq); hctx_unlock(hctx, srcu_idx); } @@ -685,8 +684,6 @@ void blk_mq_start_request(struct request *rq) preempt_enable(); set_bit(REQ_ATOM_STARTED, &rq->atomic_flags); - if (test_bit(REQ_ATOM_COMPLETE, &rq->atomic_flags)) - clear_bit(REQ_ATOM_COMPLETE, &rq->atomic_flags); if (q->dma_drain_size && blk_rq_bytes(rq)) { /* @@ -837,6 +834,8 @@ static void blk_mq_rq_timed_out(struct request *req, bool reserved) if (!test_bit(REQ_ATOM_STARTED, &req->atomic_flags)) return; + req->rq_flags |= RQF_MQ_TIMEOUT_EXPIRED; + if (ops->timeout) ret = ops->timeout(req, reserved); @@ -852,7 +851,6 @@ static void blk_mq_rq_timed_out(struct request *req, bool reserved) */ blk_mq_rq_update_aborted_gstate(req, 0); blk_add_timer(req); - blk_clear_rq_complete(req); break; case BLK_EH_NOT_HANDLED: break; @@ -871,7 +869,8 @@ static void blk_mq_check_expired(struct blk_mq_hw_ctx *hctx, might_sleep(); - if (!test_bit(REQ_ATOM_STARTED, &rq->atomic_flags)) + if ((rq->rq_flags & RQF_MQ_TIMEOUT_EXPIRED) || + !test_bit(REQ_ATOM_STARTED, &rq->atomic_flags)) return; /* read coherent snapshots of @rq->state_gen and @rq->deadline */ @@ -906,8 +905,8 @@ static void blk_mq_terminate_expired(struct blk_mq_hw_ctx *hctx, * now guaranteed to see @rq->aborted_gstate and yield. If * @rq->aborted_gstate still matches @rq->gstate, @rq is ours. */ - if (READ_ONCE(rq->gstate) == rq->aborted_gstate && - !blk_mark_rq_complete(rq)) + if (!(rq->rq_flags & RQF_MQ_TIMEOUT_EXPIRED) && + READ_ONCE(rq->gstate) == rq->aborted_gstate) blk_mq_rq_timed_out(rq, reserved); } diff --git a/block/blk-timeout.c b/block/blk-timeout.c index 4f04cd1..ebe9996 100644 --- a/block/blk-timeout.c +++ b/block/blk-timeout.c @@ -214,6 +214,7 @@ void blk_add_timer(struct request *req) req->timeout = q->rq_timeout; req->deadline = jiffies + req->timeout; + req->rq_flags &= ~RQF_MQ_TIMEOUT_EXPIRED; /* * Only the non-mq case needs to add the request to a protected list. diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h index ae563d0..007a7cf 100644 --- a/include/linux/blkdev.h +++ b/include/linux/blkdev.h @@ -125,6 +125,8 @@ typedef __u32 __bitwise req_flags_t; #define RQF_SPECIAL_PAYLOAD ((__force req_flags_t)(1 << 18)) /* The per-zone write lock is held for this request */ #define RQF_ZONE_WRITE_LOCKED ((__force req_flags_t)(1 << 19)) +/* timeout is expired */ +#define RQF_MQ_TIMEOUT_EXPIRED ((__force req_flags_t)(1 << 20)) /* flags that prevent us from merging requests: */ #define RQF_NOMERGE_FLAGS \