From patchwork Tue Nov 23 17:10:56 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 12634827 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7E490C433F5 for ; Tue, 23 Nov 2021 17:11:03 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239012AbhKWROK (ORCPT ); Tue, 23 Nov 2021 12:14:10 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45000 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239020AbhKWROJ (ORCPT ); Tue, 23 Nov 2021 12:14:09 -0500 Received: from mail-io1-xd30.google.com (mail-io1-xd30.google.com [IPv6:2607:f8b0:4864:20::d30]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B658CC061574 for ; Tue, 23 Nov 2021 09:11:01 -0800 (PST) Received: by mail-io1-xd30.google.com with SMTP id 14so28709371ioe.2 for ; Tue, 23 Nov 2021 09:11:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20210112.gappssmtp.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=FlIXO1+Df28dRCWMrnsPEvj1eS6L9MP2lhaGQ2y3v+g=; b=5BcNaLUG5id7+Cklik7xvrZYfFbEDQ1zS34ZdTtnD7EUeoh9U3Ths5/tQjK0vQglAe zZRWwVqtNajw4+jS3yxGiieK45IeKtF6jbsbNw6SYGNm3y1tTegyWmeytXGW292TnNeI SKqSytb9wyVcqsWDEH4NXUSa1oyuMcUU53ZwI4pwwPiDsdAc7J9nf1m2nGXZCo0F/3He YBY/9PwW21rJJkSaVPqkbIH2I9jvMkQ7YhghHAGkzhGomSR0sXqCfYncha5epQYTTsNn X26pp3WdZkpszfAhPPL2DJpsFRvxQJlaaFVx3e2n/PYZqILhjz2vIKGXwVKvKkrrcgrr RA7w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=FlIXO1+Df28dRCWMrnsPEvj1eS6L9MP2lhaGQ2y3v+g=; b=8Ikkb0GWqRYXE5vISSBArUbsZGeRl42ObLmFLbgUJgl4d8SGQB4t23lVmcCBJ+BE90 4oNhQ712ohBsBZ2lA6Ai67hIANvUD3jJbxMSFilCtK8nW2tJznutwaHWV7PYAVohJWiG jdlew03LHGKGJ0xBqelpGNUiGWJmW8u8xyBhhSi8/5EWGDof+klNvg+XYY0x2TDwpUhd XqcJtHMUHzhw03UhxWVx4G4KacojPE5R8RSf923hBgnHicpIhff/2bapqwsUmYoQuqo7 g8HWn/o6RCraG/J2dGtOaSLUv5BPpeQbh6PjgwW2O88SDugN/ohURjXhXkO+0Qe3VcaJ bw2w== X-Gm-Message-State: AOAM531efCbckN6UrRoHkUEmsU0tGD5+xqmV3aWXozSgIyNrkgW0JJJI OoCn9tD9cGhvE6n8C+f4jUtknEeTbDxbDnma X-Google-Smtp-Source: ABdhPJyqN0pyUqocEWYrJK+CohDl3f6E0vV+7B5+MaDcON1+VKmEcBVFJXNVMr7bBqdzSBYXr1BxYQ== X-Received: by 2002:a05:6638:1395:: with SMTP id w21mr8086533jad.125.1637687460892; Tue, 23 Nov 2021 09:11:00 -0800 (PST) Received: from p1.localdomain ([207.135.234.126]) by smtp.gmail.com with ESMTPSA id i20sm8251283iow.9.2021.11.23.09.11.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 23 Nov 2021 09:11:00 -0800 (PST) From: Jens Axboe To: linux-block@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 1/3] block: move io_context creation into where it's needed Date: Tue, 23 Nov 2021 10:10:56 -0700 Message-Id: <20211123171058.346084-2-axboe@kernel.dk> X-Mailer: git-send-email 2.34.0 In-Reply-To: <20211123171058.346084-1-axboe@kernel.dk> References: <20211123171058.346084-1-axboe@kernel.dk> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org The only user of the io_context for IO is BFQ, yet we put the checking and logic of it into the normal IO path. Put the creation into blk_mq_sched_assign_ioc(), and have BFQ use that helper. Signed-off-by: Jens Axboe --- block/bfq-iosched.c | 2 ++ block/blk-core.c | 9 --------- block/blk-ioc.c | 1 + block/blk-mq-sched.c | 5 +++++ block/blk-mq.c | 3 --- 5 files changed, 8 insertions(+), 12 deletions(-) diff --git a/block/bfq-iosched.c b/block/bfq-iosched.c index fec18118dc30..1ce1a99a7160 100644 --- a/block/bfq-iosched.c +++ b/block/bfq-iosched.c @@ -6573,6 +6573,8 @@ static struct bfq_queue *bfq_get_bfqq_handle_split(struct bfq_data *bfqd, */ static void bfq_prepare_request(struct request *rq) { + blk_mq_sched_assign_ioc(rq); + /* * Regardless of whether we have an icq attached, we have to * clear the scheduler pointers, as they might point to diff --git a/block/blk-core.c b/block/blk-core.c index 6443f2dfe43e..6ae8297b033f 100644 --- a/block/blk-core.c +++ b/block/blk-core.c @@ -750,15 +750,6 @@ noinline_for_stack bool submit_bio_checks(struct bio *bio) break; } - /* - * Various block parts want %current->io_context, so allocate it up - * front rather than dealing with lots of pain to allocate it only - * where needed. This may fail and the block layer knows how to live - * with it. - */ - if (unlikely(!current->io_context)) - create_task_io_context(current, GFP_ATOMIC, q->node); - if (blk_throtl_bio(bio)) return false; diff --git a/block/blk-ioc.c b/block/blk-ioc.c index 57299f860d41..736e0280d76f 100644 --- a/block/blk-ioc.c +++ b/block/blk-ioc.c @@ -286,6 +286,7 @@ int create_task_io_context(struct task_struct *task, gfp_t gfp_flags, int node) return ret; } +EXPORT_SYMBOL_GPL(create_task_io_context); /** * get_task_io_context - get io_context of a task diff --git a/block/blk-mq-sched.c b/block/blk-mq-sched.c index ba21449439cc..b942b38000e5 100644 --- a/block/blk-mq-sched.c +++ b/block/blk-mq-sched.c @@ -24,6 +24,10 @@ void blk_mq_sched_assign_ioc(struct request *rq) struct io_context *ioc; struct io_cq *icq; + /* create task io_context, if we don't have one already */ + if (unlikely(!current->io_context)) + create_task_io_context(current, GFP_ATOMIC, q->node); + /* * May not have an IO context if it's a passthrough request */ @@ -43,6 +47,7 @@ void blk_mq_sched_assign_ioc(struct request *rq) get_io_context(icq->ioc); rq->elv.icq = icq; } +EXPORT_SYMBOL_GPL(blk_mq_sched_assign_ioc); /* * Mark a hardware queue as needing a restart. For shared queues, maintain diff --git a/block/blk-mq.c b/block/blk-mq.c index 4c00b22590cc..20a6445f6a01 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -406,9 +406,6 @@ static struct request *blk_mq_rq_ctx_init(struct blk_mq_alloc_data *data, if (!op_is_flush(data->cmd_flags) && e->type->ops.prepare_request) { - if (e->type->icq_cache) - blk_mq_sched_assign_ioc(rq); - e->type->ops.prepare_request(rq); rq->rq_flags |= RQF_ELVPRIV; } From patchwork Tue Nov 23 17:10:57 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 12634831 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 25451C4332F for ; Tue, 23 Nov 2021 17:11:05 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239020AbhKWROM (ORCPT ); Tue, 23 Nov 2021 12:14:12 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45012 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239039AbhKWROM (ORCPT ); Tue, 23 Nov 2021 12:14:12 -0500 Received: from mail-io1-xd2a.google.com (mail-io1-xd2a.google.com [IPv6:2607:f8b0:4864:20::d2a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DB115C061714 for ; Tue, 23 Nov 2021 09:11:03 -0800 (PST) Received: by mail-io1-xd2a.google.com with SMTP id 14so28709523ioe.2 for ; Tue, 23 Nov 2021 09:11:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20210112.gappssmtp.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=obLep7hKn/j/EoRL8hQqMnp3Wbyin21r4clAytoEnPo=; b=ZMlpb2GAFTKUPTURx50fJst+8Gol+ffBBhtExcfky83LdAqP7SMElpuz/tejcHe7z8 3r8pJfvbcwrT3YBvC/pDrrYwcIg69pB+EIZulWNNAyym6wg2+zLhIfoglwjy/mXhfgfT TJvMQmVVG9J/I9kx4m78Xfv5yjfAg+NwhB/R8ckHwIb4G27fPSIjuyfOR3AsexOOnWDK A0Lkx59fN3pcdxJ+BGnJcrsYilCaH/ML5h588lonQiIBS+nFwy2wAX6e/nc3oJYH8pF7 Ja2F+Q8IkJcUuVcZtm0tf4LHfgqhVDlhpQHw3o69zyQ7ISL2XwLewudEnn3h8F3Il8ew B14g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=obLep7hKn/j/EoRL8hQqMnp3Wbyin21r4clAytoEnPo=; b=CNaXJX1XtXxAdj/HUplJrJ578+y33g7IubIxCzg0aHg8bPoktft77l6H3MMTSEyJPk 5DCDkXRnPxbG9LlMdU9aq1vRVLbaLJH5tFjxI7pJfxPcTDCWRx01pPlSw4hit/y2+dr8 qsqT+eAkjFRWZqA0lR1azFLkbYquYQQvqbpZgTzDnHPvTCO8lQRZ6jQECWAHTBXjdmdu OaZyUCo9h62b8RiioX7a1s0yIbw0hK8zYhgxnog+bMDYCJPJK/Nnd6Yid0GR7bgsJVcW RSeLkjQ9gYKduOcFp1GfGNGBIHZqW9CX8+4PZnLxZG2CN30yUp5lywm80UUbF8tIR6bF XuoQ== X-Gm-Message-State: AOAM5338nxfMPBWhgUqEiYNiGwFXpLAasJ9aBeCyUdEeNgtymzRQTi9D YJh1B/3+c5JE0hkvxPD07Tn2TXaH6wxBSR41 X-Google-Smtp-Source: ABdhPJyp1CXd3JMpFO6fBE5/sIVLd5PeMerSD45dlq/EzsQCS4De68fNl18HKBdWikviqMI4mK7kRw== X-Received: by 2002:a02:6666:: with SMTP id l38mr7815860jaf.146.1637687461732; Tue, 23 Nov 2021 09:11:01 -0800 (PST) Received: from p1.localdomain ([207.135.234.126]) by smtp.gmail.com with ESMTPSA id i20sm8251283iow.9.2021.11.23.09.11.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 23 Nov 2021 09:11:01 -0800 (PST) From: Jens Axboe To: linux-block@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 2/3] blk-ioprio: don't set bio priority if not needed Date: Tue, 23 Nov 2021 10:10:57 -0700 Message-Id: <20211123171058.346084-3-axboe@kernel.dk> X-Mailer: git-send-email 2.34.0 In-Reply-To: <20211123171058.346084-1-axboe@kernel.dk> References: <20211123171058.346084-1-axboe@kernel.dk> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org We don't need to write to the bio if: 1) No ioprio value has ever been assigned to the blkcg 2) We wouldn't anyway, depending on bio and blkcg IO priority Signed-off-by: Jens Axboe Reviewed-by: Christoph Hellwig --- block/blk-ioprio.c | 13 ++++++++++--- 1 file changed, 10 insertions(+), 3 deletions(-) diff --git a/block/blk-ioprio.c b/block/blk-ioprio.c index 332a07761bf8..2e7f10e1c03f 100644 --- a/block/blk-ioprio.c +++ b/block/blk-ioprio.c @@ -62,6 +62,7 @@ struct ioprio_blkg { struct ioprio_blkcg { struct blkcg_policy_data cpd; enum prio_policy prio_policy; + bool prio_set; }; static inline struct ioprio_blkg *pd_to_ioprio(struct blkg_policy_data *pd) @@ -112,7 +113,7 @@ static ssize_t ioprio_set_prio_policy(struct kernfs_open_file *of, char *buf, if (ret < 0) return ret; blkcg->prio_policy = ret; - + blkcg->prio_set = true; return nbytes; } @@ -190,6 +191,10 @@ static void blkcg_ioprio_track(struct rq_qos *rqos, struct request *rq, struct bio *bio) { struct ioprio_blkcg *blkcg = ioprio_blkcg_from_bio(bio); + u16 prio; + + if (!blkcg->prio_set) + return; /* * Except for IOPRIO_CLASS_NONE, higher I/O priority numbers @@ -199,8 +204,10 @@ static void blkcg_ioprio_track(struct rq_qos *rqos, struct request *rq, * bio I/O priority is not modified. If the bio I/O priority equals * IOPRIO_CLASS_NONE, the cgroup I/O priority is assigned to the bio. */ - bio->bi_ioprio = max_t(u16, bio->bi_ioprio, - IOPRIO_PRIO_VALUE(blkcg->prio_policy, 0)); + prio = max_t(u16, bio->bi_ioprio, + IOPRIO_PRIO_VALUE(blkcg->prio_policy, 0)); + if (prio > bio->bi_ioprio) + bio->bi_ioprio = prio; } static void blkcg_ioprio_exit(struct rq_qos *rqos) From patchwork Tue Nov 23 17:10:58 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 12634829 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id CBD3AC433FE for ; Tue, 23 Nov 2021 17:11:04 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239048AbhKWROM (ORCPT ); Tue, 23 Nov 2021 12:14:12 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45006 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239020AbhKWROL (ORCPT ); Tue, 23 Nov 2021 12:14:11 -0500 Received: from mail-io1-xd2d.google.com (mail-io1-xd2d.google.com [IPv6:2607:f8b0:4864:20::d2d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 08D2AC061574 for ; Tue, 23 Nov 2021 09:11:03 -0800 (PST) Received: by mail-io1-xd2d.google.com with SMTP id z18so28857313iof.5 for ; Tue, 23 Nov 2021 09:11:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20210112.gappssmtp.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=064ELPy97acrp52hALNERxgS/c5GK/HHcBvjHkrSoso=; b=OJkQkESVqccMeCZqVkmuvzLsQoArI0v8dMIhc5Rut7ULt7krDY+c9ZwfMKMHfbdS6E 9I4/Ek78n+epvP3lmv8BZ03xebDeDzqp+gR168zDoKBc7EAjEMFX5rH+cQMERPmkr10r s08EJtAkbmrwAlDipJEbafCVDTZ10n4Jh2d0k1yaX3K2YRIVeNfY+8VbIgpKP68XKajA eL49ZKXmRsv+wQxShhm9zHK1oCQNmglNxfOJnsjl0bwbVvYGDfNeqgtVgB4SJTFnoXMm UTAv+9MiEqIu0qJsN2FpsMwYj8+ZtqTXgI6YkCQX85B3BK8G7ukTWZ47oLSnkd9wY3Tt RBvg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=064ELPy97acrp52hALNERxgS/c5GK/HHcBvjHkrSoso=; b=dARpLfmICCOpiM0A3h4h4ak5usb6+iYATMEQnTSpSqIDd1LtkjXaUoTKEyBhLFOH6P zU/rpFygwwlpmVzEkcc734bYdXSu773oCLxFCV2swRoM/z/bkiyLyFqbfXEJ2G5Q4sSJ qPZTP8o98kauSd+myGFbk1/7IyJtrOsqrc+sAZ/jJ6PMTRS95APi0AyDU6EFzjSzksr8 6saMR66+up6DNuvDycOFBYfGf1KzuPFdb2xuceHAfzYMPEWzk37q2s4FGDsisr9MZ0Ap E20uHAr+ijfduCzXMWG6ytTYJgOw+MqfovinBsZewfb0r3VlG8sBMhUHGORli8g7w1bd xQXQ== X-Gm-Message-State: AOAM533cm0zcz+xIjNdC4bkgP3v0BCFF8A+JYMC/+UMC5NdvD1sGWy48 gmf9XmMkkw/WnKLQ9NiSg3+2hPnZ4PmpN0oS X-Google-Smtp-Source: ABdhPJwOtdzHSFuEf+gUqwQCH9UCBi5QkGZUsBjv1K9DdRfyTUwd2tvGekVOagVI2qQIzUrrDjaosA== X-Received: by 2002:a02:cc91:: with SMTP id s17mr8346541jap.3.1637687462276; Tue, 23 Nov 2021 09:11:02 -0800 (PST) Received: from p1.localdomain ([207.135.234.126]) by smtp.gmail.com with ESMTPSA id i20sm8251283iow.9.2021.11.23.09.11.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 23 Nov 2021 09:11:01 -0800 (PST) From: Jens Axboe To: linux-block@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 3/3] block: only allocate poll_stats if there's a user of them Date: Tue, 23 Nov 2021 10:10:58 -0700 Message-Id: <20211123171058.346084-4-axboe@kernel.dk> X-Mailer: git-send-email 2.34.0 In-Reply-To: <20211123171058.346084-1-axboe@kernel.dk> References: <20211123171058.346084-1-axboe@kernel.dk> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org This is essentially never used, yet it's about 1/3rd of the total queue size. Allocate it when needed, and don't embed it in the queue. Kill the queue flag for this while at it, since we can just check the assigned pointer now. Signed-off-by: Jens Axboe --- block/blk-mq-debugfs.c | 1 - block/blk-mq.c | 23 +++++++++++++++++++---- block/blk-stat.c | 6 ------ block/blk-sysfs.c | 3 ++- include/linux/blkdev.h | 10 +++++++--- 5 files changed, 28 insertions(+), 15 deletions(-) diff --git a/block/blk-mq-debugfs.c b/block/blk-mq-debugfs.c index 4f2cf8399f3d..f4022b198580 100644 --- a/block/blk-mq-debugfs.c +++ b/block/blk-mq-debugfs.c @@ -122,7 +122,6 @@ static const char *const blk_queue_flag_name[] = { QUEUE_FLAG_NAME(FUA), QUEUE_FLAG_NAME(DAX), QUEUE_FLAG_NAME(STATS), - QUEUE_FLAG_NAME(POLL_STATS), QUEUE_FLAG_NAME(REGISTERED), QUEUE_FLAG_NAME(QUIESCED), QUEUE_FLAG_NAME(PCI_P2PDMA), diff --git a/block/blk-mq.c b/block/blk-mq.c index 20a6445f6a01..4c59b24690d7 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -4577,9 +4577,25 @@ EXPORT_SYMBOL_GPL(blk_mq_update_nr_hw_queues); /* Enable polling stats and return whether they were already enabled. */ static bool blk_poll_stats_enable(struct request_queue *q) { - if (test_bit(QUEUE_FLAG_POLL_STATS, &q->queue_flags) || - blk_queue_flag_test_and_set(QUEUE_FLAG_POLL_STATS, q)) + struct blk_rq_stat *poll_stat; + + if (q->poll_stat) return true; + + poll_stat = kcalloc(BLK_MQ_POLL_STATS_BKTS, sizeof(*poll_stat), + GFP_ATOMIC); + if (!poll_stat) + return false; + + spin_lock_irq(&q->stats->lock); + if (q->poll_stat) { + spin_unlock_irq(&q->stats->lock); + kfree(poll_stat); + return true; + } + q->poll_stat = poll_stat; + spin_unlock_irq(&q->stats->lock); + blk_stat_add_callback(q, q->poll_cb); return false; } @@ -4590,8 +4606,7 @@ static void blk_mq_poll_stats_start(struct request_queue *q) * We don't arm the callback if polling stats are not enabled or the * callback is already active. */ - if (!test_bit(QUEUE_FLAG_POLL_STATS, &q->queue_flags) || - blk_stat_is_active(q->poll_cb)) + if (!q->poll_stat || blk_stat_is_active(q->poll_cb)) return; blk_stat_activate_msecs(q->poll_cb, 100); diff --git a/block/blk-stat.c b/block/blk-stat.c index ae3dd1fb8e61..7ba504166d1b 100644 --- a/block/blk-stat.c +++ b/block/blk-stat.c @@ -12,12 +12,6 @@ #include "blk-mq.h" #include "blk.h" -struct blk_queue_stats { - struct list_head callbacks; - spinlock_t lock; - bool enable_accounting; -}; - void blk_rq_stat_init(struct blk_rq_stat *stat) { stat->min = -1ULL; diff --git a/block/blk-sysfs.c b/block/blk-sysfs.c index cd75b0f73dc6..c079be1c58a3 100644 --- a/block/blk-sysfs.c +++ b/block/blk-sysfs.c @@ -785,11 +785,12 @@ static void blk_release_queue(struct kobject *kobj) might_sleep(); - if (test_bit(QUEUE_FLAG_POLL_STATS, &q->queue_flags)) + if (q->poll_stat) blk_stat_remove_callback(q, q->poll_cb); blk_stat_free_callback(q->poll_cb); blk_free_queue_stats(q->stats); + kfree(q->poll_stat); blk_exit_queue(q); diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h index bd4370baccca..90dac4a67cfc 100644 --- a/include/linux/blkdev.h +++ b/include/linux/blkdev.h @@ -28,10 +28,15 @@ struct blk_flush_queue; struct kiocb; struct pr_ops; struct rq_qos; -struct blk_queue_stats; struct blk_stat_callback; struct blk_crypto_profile; +struct blk_queue_stats { + struct list_head callbacks; + spinlock_t lock; + bool enable_accounting; +}; + /* Must be consistent with blk_mq_poll_stats_bkt() */ #define BLK_MQ_POLL_STATS_BKTS 16 @@ -267,7 +272,7 @@ struct request_queue { int poll_nsec; struct blk_stat_callback *poll_cb; - struct blk_rq_stat poll_stat[BLK_MQ_POLL_STATS_BKTS]; + struct blk_rq_stat *poll_stat; struct timer_list timeout; struct work_struct timeout_work; @@ -397,7 +402,6 @@ struct request_queue { #define QUEUE_FLAG_FUA 18 /* device supports FUA writes */ #define QUEUE_FLAG_DAX 19 /* device supports DAX */ #define QUEUE_FLAG_STATS 20 /* track IO start and completion times */ -#define QUEUE_FLAG_POLL_STATS 21 /* collecting stats for hybrid polling */ #define QUEUE_FLAG_REGISTERED 22 /* queue has been registered to a disk */ #define QUEUE_FLAG_QUIESCED 24 /* queue has been quiesced */ #define QUEUE_FLAG_PCI_P2PDMA 25 /* device supports PCI p2p requests */