From patchwork Tue Jun 8 23:06:50 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bart Van Assche X-Patchwork-Id: 12308361 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 11027C47082 for ; Tue, 8 Jun 2021 23:07:28 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id E728461361 for ; Tue, 8 Jun 2021 23:07:27 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232208AbhFHXJU (ORCPT ); Tue, 8 Jun 2021 19:09:20 -0400 Received: from mail-pj1-f51.google.com ([209.85.216.51]:34329 "EHLO mail-pj1-f51.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234074AbhFHXJU (ORCPT ); Tue, 8 Jun 2021 19:09:20 -0400 Received: by mail-pj1-f51.google.com with SMTP id g6-20020a17090adac6b029015d1a9a6f1aso2567232pjx.1 for ; Tue, 08 Jun 2021 16:07:12 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=1i84vOGYAYDqgpeOnmCS0aBbdkIOmIlw00w56H4OlYg=; b=Gsjn7qwYzhUjRowI7JSE4tTvdPvVw4je8gB2WKuvnKur8ARU53io+TR4gxmE1d5val VxHewR6YZKI1/e0IDTqstZO2kYNNa4JA2kRLhdrxIDBgFJuAlWMks6TBjZogzEemQVT8 jnBWQwlWvytxuJyayTTMNj3RXbdzqpx8sJ2f8k6zNJdmgzm/ZJVw74cwC5SAfLUWVZFJ TwBEYUhjEuLgH8k50GO/uI+lJzEqb1SPB8FdHWpxXNdC4wZtmQ+mo1+Rr6N2ekXZ/1pF zKTEmdX7OO0g7yp/aTeIo1z9hJz9A7oFbcQTS8RbS64ffd/j45BWHns9lG5L2Qd9ou13 z/Ng== X-Gm-Message-State: AOAM530B2/Wf18aZArj5TjOGzcJZ/sIkA7s+78ubbllzTfgVQNyobZoJ IRrIo9gsHrXMMD8KsNMD5Cg= X-Google-Smtp-Source: ABdhPJwIX/b1mR9cDzzb8ZAXoVrj/trqqlLItxMoBYdnudTjJjqCAqJm/aB25SNw7sLosQRp22IPKg== X-Received: by 2002:a17:903:1c6:b029:113:2b74:56ad with SMTP id e6-20020a17090301c6b02901132b7456admr2426532plh.55.1623193632572; Tue, 08 Jun 2021 16:07:12 -0700 (PDT) Received: from asus.hsd1.ca.comcast.net (c-73-241-217-19.hsd1.ca.comcast.net. [73.241.217.19]) by smtp.gmail.com with ESMTPSA id s21sm11395838pfw.57.2021.06.08.16.07.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 08 Jun 2021 16:07:12 -0700 (PDT) From: Bart Van Assche To: Jens Axboe Cc: linux-block@vger.kernel.org, Christoph Hellwig , Jaegeuk Kim , Bart Van Assche , Damien Le Moal , Hannes Reinecke , Ming Lei , Johannes Thumshirn , Himanshu Madhani Subject: [PATCH 01/14] block/Kconfig: Make the BLK_WBT and BLK_WBT_MQ entries consecutive Date: Tue, 8 Jun 2021 16:06:50 -0700 Message-Id: <20210608230703.19510-2-bvanassche@acm.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210608230703.19510-1-bvanassche@acm.org> References: <20210608230703.19510-1-bvanassche@acm.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org These entries were consecutive at the time of their introduction but are no longer consecutive. Make these again consecutive. Additionally, modify the help text since it refers to blk-mq and since the legacy block layer has been removed. Cc: Damien Le Moal Cc: Hannes Reinecke Cc: Christoph Hellwig Cc: Ming Lei Cc: Johannes Thumshirn Cc: Himanshu Madhani Signed-off-by: Bart Van Assche Reviewed-by: Damien Le Moal Reviewed-by: Johannes Thumshirn Reviewed-by: Hannes Reinecke --- block/Kconfig | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/block/Kconfig b/block/Kconfig index a2297edfdde8..6685578b2a20 100644 --- a/block/Kconfig +++ b/block/Kconfig @@ -133,6 +133,13 @@ config BLK_WBT dynamically on an algorithm loosely based on CoDel, factoring in the realtime performance of the disk. +config BLK_WBT_MQ + bool "Enable writeback throttling by default" + default y + depends on BLK_WBT + help + Enable writeback throttling by default for request-based block devices. + config BLK_CGROUP_IOLATENCY bool "Enable support for latency based cgroup IO protection" depends on BLK_CGROUP=y @@ -155,13 +162,6 @@ config BLK_CGROUP_IOCOST distributes IO capacity between different groups based on their share of the overall weight distribution. -config BLK_WBT_MQ - bool "Multiqueue writeback throttling" - default y - depends on BLK_WBT - help - Enable writeback throttling by default on multiqueue devices. - config BLK_DEBUG_FS bool "Block layer debugging information in debugfs" default y From patchwork Tue Jun 8 23:06:51 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bart Van Assche X-Patchwork-Id: 12308359 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 501B7C47082 for ; Tue, 8 Jun 2021 23:07:25 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 30B9A61361 for ; Tue, 8 Jun 2021 23:07:25 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234060AbhFHXJR (ORCPT ); Tue, 8 Jun 2021 19:09:17 -0400 Received: from mail-pg1-f169.google.com ([209.85.215.169]:34575 "EHLO mail-pg1-f169.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233927AbhFHXJQ (ORCPT ); Tue, 8 Jun 2021 19:09:16 -0400 Received: by mail-pg1-f169.google.com with SMTP id l1so17826199pgm.1 for ; Tue, 08 Jun 2021 16:07:14 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=4qLgR9vvJmrAZEerRG4VFRoyx7aAeQCali5c03o+eYw=; b=KS0BcRHNUEpduW00BC/Fs1LJxtKiW4i5xDoyoWWSIr5SPSReunxqWKdOcnX7lY+IU2 i8JAgUxea5+/h9KGPycEy6Mzn+aurOR7sKSAch4DEc0TAg2SW5L6sv4Z+mrTqhu9/XBt yzyIFMEdR/Se7/34Ptlk3p62cNUFLkIxswm43rjRASTqTvhq69HvXHEJ8yDm4ETG2efz nSiSgDQJutACS4Y04oeQheUod/pFspc3yX6rKRz52wUNHmJyXjy+ybFSUGZZ8xEiIHgs HW8SrBULdz2gtjCf1SXEgf/rqHh2CYZKLJuxVZfBvH+l6Wjr/INPQVEAh3hbGc5hTXqg 4JdQ== X-Gm-Message-State: AOAM531at2pXOmsumpc/+ssX2X7V5MLX5TGUeV0UBA2gFmCeHX9X/nuV V4a9k4jHV85AkHC12JSQ/k0= X-Google-Smtp-Source: ABdhPJz//lQjT8Ia79YujdTPyd1bi87zHCr/FOF0pYKNEcOGAhO1WDkNqB+SUkGG7uFEtdqeMCSXTQ== X-Received: by 2002:aa7:8c02:0:b029:2e9:c513:1e10 with SMTP id c2-20020aa78c020000b02902e9c5131e10mr2133250pfd.2.1623193633977; Tue, 08 Jun 2021 16:07:13 -0700 (PDT) Received: from asus.hsd1.ca.comcast.net (c-73-241-217-19.hsd1.ca.comcast.net. [73.241.217.19]) by smtp.gmail.com with ESMTPSA id s21sm11395838pfw.57.2021.06.08.16.07.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 08 Jun 2021 16:07:13 -0700 (PDT) From: Bart Van Assche To: Jens Axboe Cc: linux-block@vger.kernel.org, Christoph Hellwig , Jaegeuk Kim , Bart Van Assche , Tejun Heo , Damien Le Moal , Hannes Reinecke , Ming Lei , Johannes Thumshirn , Himanshu Madhani Subject: [PATCH 02/14] block/blk-cgroup: Swap the blk_throtl_init() and blk_iolatency_init() calls Date: Tue, 8 Jun 2021 16:06:51 -0700 Message-Id: <20210608230703.19510-3-bvanassche@acm.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210608230703.19510-1-bvanassche@acm.org> References: <20210608230703.19510-1-bvanassche@acm.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org Before adding more calls in this function, simplify the error path. Cc: Tejun Heo Cc: Damien Le Moal Cc: Hannes Reinecke Cc: Christoph Hellwig Cc: Ming Lei Cc: Johannes Thumshirn Cc: Himanshu Madhani Signed-off-by: Bart Van Assche Reviewed-by: Damien Le Moal Reviewed-by: Johannes Thumshirn Reviewed-by: Hannes Reinecke --- block/blk-cgroup.c | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/block/blk-cgroup.c b/block/blk-cgroup.c index d169e2055158..3b0f6efaa2b6 100644 --- a/block/blk-cgroup.c +++ b/block/blk-cgroup.c @@ -1183,15 +1183,14 @@ int blkcg_init_queue(struct request_queue *q) if (preloaded) radix_tree_preload_end(); - ret = blk_throtl_init(q); + ret = blk_iolatency_init(q); if (ret) goto err_destroy_all; - ret = blk_iolatency_init(q); - if (ret) { - blk_throtl_exit(q); + ret = blk_throtl_init(q); + if (ret) goto err_destroy_all; - } + return 0; err_destroy_all: From patchwork Tue Jun 8 23:06:52 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bart Van Assche X-Patchwork-Id: 12308367 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3A889C4743F for ; Tue, 8 Jun 2021 23:07:33 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 219BE61278 for ; Tue, 8 Jun 2021 23:07:33 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234074AbhFHXJZ (ORCPT ); Tue, 8 Jun 2021 19:09:25 -0400 Received: from mail-pg1-f181.google.com ([209.85.215.181]:41576 "EHLO mail-pg1-f181.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234779AbhFHXJY (ORCPT ); Tue, 8 Jun 2021 19:09:24 -0400 Received: by mail-pg1-f181.google.com with SMTP id l184so2274444pgd.8 for ; Tue, 08 Jun 2021 16:07:15 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=n9mlbGrne/i9hM5ojviqHTqgYxuBYYuowh8cFarteBM=; b=GAa0SsTRpUsO5ClecgGJwUjcwEvk323kNqjekRUB15Bd0lNcdr1bpamYmqL86aZloq 773u8vI6CJztKeXx5KTOrAW6sM0purk8QWl5n4WoVYtDKnfGoG3Hm6LtSeyHrqVaUwVN lQcH4EGtspTGtIZO8yXPjhmbvv7yLF+MxpxTT059/tUvoPN2Npbtx9K0tO5/gjIj0ovj NdxHvtqnAT5SGy4CFQ1ODTOkPfGacGqaOk0zq/Ae3UFimQftVdmnrrzNea3pL56eZEmF vj2LLTDX/eMFTM8qglOXemOfS+y27FZBLtWomP/UY9FrFnPMH9MnEmQZ2VWhqLAyOe5U DCPg== X-Gm-Message-State: AOAM532yu0haaXE0UdIRfiWa9TKDES5/C0V1zr76ckbTUgT46iKfB/sa 4hQwvYziUAvoxgz3E2db16I= X-Google-Smtp-Source: ABdhPJxP/M7v3Ck/aUbO/4UGNYO99vEw8JRe36cgyKEe17Kb9t1VlneEWEPABZUUThVVjo6mMsUZPw== X-Received: by 2002:a63:e709:: with SMTP id b9mr614796pgi.153.1623193635335; Tue, 08 Jun 2021 16:07:15 -0700 (PDT) Received: from asus.hsd1.ca.comcast.net (c-73-241-217-19.hsd1.ca.comcast.net. [73.241.217.19]) by smtp.gmail.com with ESMTPSA id s21sm11395838pfw.57.2021.06.08.16.07.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 08 Jun 2021 16:07:14 -0700 (PDT) From: Bart Van Assche To: Jens Axboe Cc: linux-block@vger.kernel.org, Christoph Hellwig , Jaegeuk Kim , Bart Van Assche , Damien Le Moal , Hannes Reinecke , Ming Lei , Johannes Thumshirn , Himanshu Madhani Subject: [PATCH 03/14] block/blk-rq-qos: Move a function from a header file into a C file Date: Tue, 8 Jun 2021 16:06:52 -0700 Message-Id: <20210608230703.19510-4-bvanassche@acm.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210608230703.19510-1-bvanassche@acm.org> References: <20210608230703.19510-1-bvanassche@acm.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org rq_qos_id_to_name() is not in the hot path so move it from a .h into a .c file. Cc: Damien Le Moal Cc: Hannes Reinecke Cc: Christoph Hellwig Cc: Ming Lei Cc: Johannes Thumshirn Cc: Himanshu Madhani Signed-off-by: Bart Van Assche Reviewed-by: Damien Le Moal Reviewed-by: Hannes Reinecke --- block/blk-rq-qos.c | 13 +++++++++++++ block/blk-rq-qos.h | 13 +------------ 2 files changed, 14 insertions(+), 12 deletions(-) diff --git a/block/blk-rq-qos.c b/block/blk-rq-qos.c index 656460636ad3..61dccf584c68 100644 --- a/block/blk-rq-qos.c +++ b/block/blk-rq-qos.c @@ -2,6 +2,19 @@ #include "blk-rq-qos.h" +const char *rq_qos_id_to_name(enum rq_qos_id id) +{ + switch (id) { + case RQ_QOS_WBT: + return "wbt"; + case RQ_QOS_LATENCY: + return "latency"; + case RQ_QOS_COST: + return "cost"; + } + return "unknown"; +} + /* * Increment 'v', if 'v' is below 'below'. Returns true if we succeeded, * false if 'v' + 1 would be bigger than 'below'. diff --git a/block/blk-rq-qos.h b/block/blk-rq-qos.h index 2bc43e94f4c4..6b5f9ae69883 100644 --- a/block/blk-rq-qos.h +++ b/block/blk-rq-qos.h @@ -78,18 +78,7 @@ static inline struct rq_qos *blkcg_rq_qos(struct request_queue *q) return rq_qos_id(q, RQ_QOS_LATENCY); } -static inline const char *rq_qos_id_to_name(enum rq_qos_id id) -{ - switch (id) { - case RQ_QOS_WBT: - return "wbt"; - case RQ_QOS_LATENCY: - return "latency"; - case RQ_QOS_COST: - return "cost"; - } - return "unknown"; -} +const char *rq_qos_id_to_name(enum rq_qos_id id); static inline void rq_wait_init(struct rq_wait *rq_wait) { From patchwork Tue Jun 8 23:06:53 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bart Van Assche X-Patchwork-Id: 12308355 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id BA234C47082 for ; Tue, 8 Jun 2021 23:07:18 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 9277761364 for ; Tue, 8 Jun 2021 23:07:18 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232310AbhFHXJK (ORCPT ); Tue, 8 Jun 2021 19:09:10 -0400 Received: from mail-pg1-f180.google.com ([209.85.215.180]:38429 "EHLO mail-pg1-f180.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232208AbhFHXJK (ORCPT ); Tue, 8 Jun 2021 19:09:10 -0400 Received: by mail-pg1-f180.google.com with SMTP id t17so1252798pga.5 for ; Tue, 08 Jun 2021 16:07:17 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=K5VjFrX5EZz8KxWa/6vPDxuEdxlNFseJ5efdQkCIwWQ=; b=Ehv9pQnYNBDIE9nQOCw9k4IrVgCAx7YF8ujdr6NWAbTrH7DNVZDqScfN9ylicjGRoS J0xn85jxEaIIHCMHRi3jz4y0cdPIWlmY8SBkD7CuMprqsg0ytI3MrFmReoPfbcrL1eSm Bip7XqQS/jwEaFhcso/l4TtE3HGWbHxNTbw4/YPiPurvyLqijYjT49zWwvMnDzIZ9ESJ Ya/P7tKKgtN9evQmZxNnUR7y4lJs7vefUEc5uBHL481J1A9TEAepnIA+XlN/qlmRvLVn PDNNZ4goP+66tgxNQIC5K5ejQpZxd90ql9B4at7uBo2n8iuKyfQCTcu2d6rVLYGe5q7c H9pw== X-Gm-Message-State: AOAM533tHoyZIyfW0MR8MRFjIZkddeHNK0Cz79lrAjAtyz4aZ5EZuskx HUYv1q90rtSBS9YUaaQzw/P2ziPMROI= X-Google-Smtp-Source: ABdhPJzadO1cZWiWxXFxpMGtYpgDDkUJseRXthffOwrzPQCKGKt1tTio174ARlg5T06KHF7+A5XnpQ== X-Received: by 2002:a63:a805:: with SMTP id o5mr638979pgf.328.1623193636720; Tue, 08 Jun 2021 16:07:16 -0700 (PDT) Received: from asus.hsd1.ca.comcast.net (c-73-241-217-19.hsd1.ca.comcast.net. [73.241.217.19]) by smtp.gmail.com with ESMTPSA id s21sm11395838pfw.57.2021.06.08.16.07.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 08 Jun 2021 16:07:16 -0700 (PDT) From: Bart Van Assche To: Jens Axboe Cc: linux-block@vger.kernel.org, Christoph Hellwig , Jaegeuk Kim , Bart Van Assche , Damien Le Moal , Hannes Reinecke , Ming Lei , Johannes Thumshirn , Himanshu Madhani Subject: [PATCH 04/14] block: Introduce the ioprio rq-qos policy Date: Tue, 8 Jun 2021 16:06:53 -0700 Message-Id: <20210608230703.19510-5-bvanassche@acm.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210608230703.19510-1-bvanassche@acm.org> References: <20210608230703.19510-1-bvanassche@acm.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org Introduce an rq-qos policy that assigns an I/O priority to requests based on blk-cgroup configuration settings. This policy has the following advantages over the ioprio_set() system call: - This policy is cgroup based so it has all the advantages of cgroups. - While ioprio_set() does not affect page cache writeback I/O, this rq-qos controller affects page cache writeback I/O for filesystems that support assiociating a cgroup with writeback I/O. See also Documentation/admin-guide/cgroup-v2.rst. Cc: Damien Le Moal Cc: Hannes Reinecke Cc: Christoph Hellwig Cc: Ming Lei Cc: Johannes Thumshirn Cc: Himanshu Madhani Signed-off-by: Bart Van Assche --- Documentation/admin-guide/cgroup-v2.rst | 26 +++ block/Kconfig | 9 + block/Makefile | 1 + block/blk-cgroup.c | 5 + block/blk-ioprio.c | 236 ++++++++++++++++++++++++ block/blk-ioprio.h | 19 ++ block/blk-rq-qos.c | 2 + block/blk-rq-qos.h | 1 + 8 files changed, 299 insertions(+) create mode 100644 block/blk-ioprio.c create mode 100644 block/blk-ioprio.h diff --git a/Documentation/admin-guide/cgroup-v2.rst b/Documentation/admin-guide/cgroup-v2.rst index b1e81aa8598a..cf8c8073b474 100644 --- a/Documentation/admin-guide/cgroup-v2.rst +++ b/Documentation/admin-guide/cgroup-v2.rst @@ -56,6 +56,7 @@ v1 is available under :ref:`Documentation/admin-guide/cgroup-v1/index.rst #include #include "blk.h" +#include "blk-ioprio.h" /* * blkcg_pol_mutex protects blkcg_policy[] and policy [de]activation. @@ -1187,6 +1188,10 @@ int blkcg_init_queue(struct request_queue *q) if (ret) goto err_destroy_all; + ret = blk_ioprio_init(q); + if (ret) + goto err_destroy_all; + ret = blk_throtl_init(q); if (ret) goto err_destroy_all; diff --git a/block/blk-ioprio.c b/block/blk-ioprio.c new file mode 100644 index 000000000000..d7292cdef67d --- /dev/null +++ b/block/blk-ioprio.c @@ -0,0 +1,236 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Block rq-qos policy for assigning an I/O priority to requests. + * + * Using an rq-qos policy for assigning I/O priority has two advantages over + * using the ioprio_set() system call: + * + * - This policy is cgroup based so it has all the advantages of cgroups. + * - While ioprio_set() does not affect page cache writeback I/O, this rq-qos + * controller affects page cache writeback I/O for filesystems that support + * assiociating a cgroup with writeback I/O. See also + * Documentation/admin-guide/cgroup-v2.rst. + */ + +#include +#include +#include +#include +#include +#include "blk-ioprio.h" +#include "blk-rq-qos.h" + +/* + * The accepted I/O priority values are 0..IOPRIO_CLASS_MAX(3). 0 (default) + * means do not modify the I/O priority. Values 1..3 are used to restrict the + * I/O priority for a cgroup to the specified priority. See also + * . + */ +#define IOPRIO_CLASS_MAX IOPRIO_CLASS_IDLE + +static struct blkcg_policy ioprio_policy; + +/** + * struct ioprio_blkg - Per (cgroup, request queue) data. + * @pd: blkg_policy_data structure. + */ +struct ioprio_blkg { + struct blkg_policy_data pd; +}; + +/** + * struct ioprio_blkcg - Per cgroup data. + * @cpd: blkcg_policy_data structure. + * @prio_class: One of the IOPRIO_CLASS_* values. See also . + */ +struct ioprio_blkcg { + struct blkcg_policy_data cpd; + u8 prio_class; +}; + +static inline struct ioprio_blkg *pd_to_ioprio(struct blkg_policy_data *pd) +{ + return pd ? container_of(pd, struct ioprio_blkg, pd) : NULL; +} + +static struct ioprio_blkcg * +ioprio_blkcg_from_css(struct cgroup_subsys_state *css) +{ + struct blkcg *blkcg = css_to_blkcg(css); + struct blkcg_policy_data *cpd = blkcg_to_cpd(blkcg, &ioprio_policy); + + return container_of(cpd, struct ioprio_blkcg, cpd); +} + +static struct ioprio_blkcg *ioprio_blkcg_from_bio(struct bio *bio) +{ + struct blkg_policy_data *pd; + + pd = blkg_to_pd(bio->bi_blkg, &ioprio_policy); + if (!pd) + return NULL; + + return container_of(blkcg_to_cpd(pd->blkg->blkcg, &ioprio_policy), + struct ioprio_blkcg, cpd); +} + +static u64 ioprio_show_prio_class(struct cgroup_subsys_state *css, + struct cftype *cft) +{ + struct ioprio_blkcg *blkcg = ioprio_blkcg_from_css(css); + + return blkcg->prio_class; +} + +static int ioprio_set_prio_class(struct cgroup_subsys_state *css, + struct cftype *cft, u64 val) +{ + struct ioprio_blkcg *blkcg = ioprio_blkcg_from_css(css); + + if (val > IOPRIO_CLASS_MAX) + return -EINVAL; + + blkcg->prio_class = val; + return 0; +} + +static struct blkg_policy_data *ioprio_alloc_pd(gfp_t gfp, + struct request_queue *q, + struct blkcg *blkcg) +{ + struct ioprio_blkg *ioprio_blkg; + + ioprio_blkg = kzalloc(sizeof(*ioprio_blkg), gfp); + if (!ioprio_blkg) + return NULL; + + return &ioprio_blkg->pd; +} + +static void ioprio_free_pd(struct blkg_policy_data *pd) +{ + struct ioprio_blkg *ioprio_blkg = pd_to_ioprio(pd); + + kfree(ioprio_blkg); +} + +static struct blkcg_policy_data *ioprio_alloc_cpd(gfp_t gfp) +{ + struct ioprio_blkcg *blkcg; + + blkcg = kzalloc(sizeof(*blkcg), gfp); + if (!blkcg) + return NULL; + blkcg->prio_class = IOPRIO_CLASS_NONE; + return &blkcg->cpd; +} + +static void ioprio_free_cpd(struct blkcg_policy_data *cpd) +{ + struct ioprio_blkcg *blkcg = container_of(cpd, typeof(*blkcg), cpd); + + kfree(blkcg); +} + +#define IOPRIO_ATTRS \ + { \ + .name = "prio.class", \ + .read_u64 = ioprio_show_prio_class, \ + .write_u64 = ioprio_set_prio_class, \ + }, \ + { } /* sentinel */ + +/* cgroup v2 attributes */ +static struct cftype ioprio_files[] = { + IOPRIO_ATTRS +}; + +/* cgroup v1 attributes */ +static struct cftype ioprio_legacy_files[] = { + IOPRIO_ATTRS +}; + +static struct blkcg_policy ioprio_policy = { + .dfl_cftypes = ioprio_files, + .legacy_cftypes = ioprio_legacy_files, + + .cpd_alloc_fn = ioprio_alloc_cpd, + .cpd_free_fn = ioprio_free_cpd, + + .pd_alloc_fn = ioprio_alloc_pd, + .pd_free_fn = ioprio_free_pd, +}; + +struct blk_ioprio { + struct rq_qos rqos; +}; + +static void blkcg_ioprio_track(struct rq_qos *rqos, struct request *rq, + struct bio *bio) +{ + struct ioprio_blkcg *blkcg = ioprio_blkcg_from_bio(bio); + + /* + * Except for IOPRIO_CLASS_NONE, higher I/O priority numbers + * correspond to a lower priority. Hence, the max_t() below selects + * the lower priority of bi_ioprio and the cgroup I/O priority class. + * If the cgroup priority has been set to IOPRIO_CLASS_NONE == 0, the + * bio I/O priority is not modified. If the bio I/O priority equals + * IOPRIO_CLASS_NONE, the cgroup I/O priority is assigned to the bio. + */ + bio->bi_ioprio = max_t(u16, bio->bi_ioprio, + IOPRIO_PRIO_VALUE(blkcg->prio_class, 0)); +} + +static void blkcg_ioprio_exit(struct rq_qos *rqos) +{ + struct blk_ioprio *blkioprio_blkg = + container_of(rqos, typeof(*blkioprio_blkg), rqos); + + blkcg_deactivate_policy(rqos->q, &ioprio_policy); + kfree(blkioprio_blkg); +} + +static struct rq_qos_ops blkcg_ioprio_ops = { + .track = blkcg_ioprio_track, + .exit = blkcg_ioprio_exit, +}; + +int blk_ioprio_init(struct request_queue *q) +{ + struct blk_ioprio *blkioprio_blkg; + struct rq_qos *rqos; + int ret; + + blkioprio_blkg = kzalloc(sizeof(*blkioprio_blkg), GFP_KERNEL); + if (!blkioprio_blkg) + return -ENOMEM; + + ret = blkcg_activate_policy(q, &ioprio_policy); + if (ret) { + kfree(blkioprio_blkg); + return ret; + } + + rqos = &blkioprio_blkg->rqos; + rqos->id = RQ_QOS_IOPRIO; + rqos->ops = &blkcg_ioprio_ops; + rqos->q = q; + + rq_qos_add(q, rqos); + + return 0; +} + +static int __init ioprio_init(void) +{ + return blkcg_policy_register(&ioprio_policy); +} + +static void __exit ioprio_exit(void) +{ + blkcg_policy_unregister(&ioprio_policy); +} + +module_init(ioprio_init); +module_exit(ioprio_exit); diff --git a/block/blk-ioprio.h b/block/blk-ioprio.h new file mode 100644 index 000000000000..a7785c2f1aea --- /dev/null +++ b/block/blk-ioprio.h @@ -0,0 +1,19 @@ +/* SPDX-License-Identifier: GPL-2.0 */ + +#ifndef _BLK_IOPRIO_H_ +#define _BLK_IOPRIO_H_ + +#include + +struct request_queue; + +#ifdef CONFIG_BLK_CGROUP_IOPRIO +int blk_ioprio_init(struct request_queue *q); +#else +static inline int blk_ioprio_init(struct request_queue *q) +{ + return 0; +} +#endif + +#endif /* _BLK_IOPRIO_H_ */ diff --git a/block/blk-rq-qos.c b/block/blk-rq-qos.c index 61dccf584c68..3a5400b11af7 100644 --- a/block/blk-rq-qos.c +++ b/block/blk-rq-qos.c @@ -11,6 +11,8 @@ const char *rq_qos_id_to_name(enum rq_qos_id id) return "latency"; case RQ_QOS_COST: return "cost"; + case RQ_QOS_IOPRIO: + return "ioprio"; } return "unknown"; } diff --git a/block/blk-rq-qos.h b/block/blk-rq-qos.h index 6b5f9ae69883..1d4c6e951aa6 100644 --- a/block/blk-rq-qos.h +++ b/block/blk-rq-qos.h @@ -16,6 +16,7 @@ enum rq_qos_id { RQ_QOS_WBT, RQ_QOS_LATENCY, RQ_QOS_COST, + RQ_QOS_IOPRIO, }; struct rq_wait { From patchwork Tue Jun 8 23:06:54 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bart Van Assche X-Patchwork-Id: 12308365 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4A363C47082 for ; Tue, 8 Jun 2021 23:07:32 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 31C5961278 for ; Tue, 8 Jun 2021 23:07:32 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234331AbhFHXJY (ORCPT ); Tue, 8 Jun 2021 19:09:24 -0400 Received: from mail-pg1-f169.google.com ([209.85.215.169]:45941 "EHLO mail-pg1-f169.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234256AbhFHXJY (ORCPT ); Tue, 8 Jun 2021 19:09:24 -0400 Received: by mail-pg1-f169.google.com with SMTP id q15so17793156pgg.12 for ; Tue, 08 Jun 2021 16:07:18 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=D0iD14Ru51kND5VtafJkBd0EUPHs/gOKbFP4JvK7IqU=; b=V5NAP/8cI85V4xQb2x504x14vCsDG7H3a+6h1+jhgi0WYVnW2PQlVo173EncMrea2I QP0NC0pWhAhnrpHexCSknAx7uzIq/lhjFhSBhWiHZNkvW183UeIrTS4lMC5JuEvZa6yF rt5lPlDtV5cHjTheJzpfJgT/zmN8fLkktGgoZ1OWAcvHe2EOnZtQIlmDEo/wEp1GvHCv RLcDp9ysV+OnUPZKvkffdszU1NoRyJOHca5NTNMKlqW+RkTUeiLNEY3eyNRoDgbbTHKu DpIngF1/tlvZ9pI7XiA98lxxb5ZiSrjbc1D24RTGTI0/IaKs+RXnlt3mmJhYWxMOHEKT JBog== X-Gm-Message-State: AOAM530s+dDTmMRq/UeI/1gsYXHAPiXhNhwhqz4CtBmoOx5am9dU7xX+ NZ9IBzB7D815bugUq51QceU= X-Google-Smtp-Source: ABdhPJyLnkBrnqhMw+F2CPAhs+fgkLaAXMS6mUMY9nu+Reqf+9ZhtXnd+VEGW2SxHHzyGaR/Oxm2GQ== X-Received: by 2002:a63:5c49:: with SMTP id n9mr605097pgm.223.1623193638099; Tue, 08 Jun 2021 16:07:18 -0700 (PDT) Received: from asus.hsd1.ca.comcast.net (c-73-241-217-19.hsd1.ca.comcast.net. [73.241.217.19]) by smtp.gmail.com with ESMTPSA id s21sm11395838pfw.57.2021.06.08.16.07.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 08 Jun 2021 16:07:17 -0700 (PDT) From: Bart Van Assche To: Jens Axboe Cc: linux-block@vger.kernel.org, Christoph Hellwig , Jaegeuk Kim , Bart Van Assche , Damien Le Moal , Johannes Thumshirn , Himanshu Madhani , Hannes Reinecke , Ming Lei Subject: [PATCH 05/14] block/mq-deadline: Add several comments Date: Tue, 8 Jun 2021 16:06:54 -0700 Message-Id: <20210608230703.19510-6-bvanassche@acm.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210608230703.19510-1-bvanassche@acm.org> References: <20210608230703.19510-1-bvanassche@acm.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org Make the code easier to read by adding more comments. Reviewed-by: Damien Le Moal Reviewed-by: Johannes Thumshirn Reviewed-by: Himanshu Madhani Cc: Christoph Hellwig Cc: Hannes Reinecke Cc: Ming Lei Signed-off-by: Bart Van Assche Reviewed-by: Hannes Reinecke --- block/mq-deadline.c | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/block/mq-deadline.c b/block/mq-deadline.c index 8eea2cbf2bf4..31418e9ce9e2 100644 --- a/block/mq-deadline.c +++ b/block/mq-deadline.c @@ -139,6 +139,9 @@ static void dd_request_merged(struct request_queue *q, struct request *req, } } +/* + * Callback function that is invoked after @next has been merged into @req. + */ static void dd_merged_requests(struct request_queue *q, struct request *req, struct request *next) { @@ -375,6 +378,8 @@ static struct request *__dd_dispatch_request(struct deadline_data *dd) } /* + * Called from blk_mq_run_hw_queue() -> __blk_mq_sched_dispatch_requests(). + * * One confusing aspect here is that we get called for a specific * hardware queue, but we may return a request that is for a * different hardware queue. This is because mq-deadline has shared @@ -438,6 +443,10 @@ static int dd_init_queue(struct request_queue *q, struct elevator_type *e) return 0; } +/* + * Try to merge @bio into an existing request. If @bio has been merged into + * an existing request, store the pointer to that request into *@rq. + */ static int dd_request_merge(struct request_queue *q, struct request **rq, struct bio *bio) { @@ -461,6 +470,10 @@ static int dd_request_merge(struct request_queue *q, struct request **rq, return ELEVATOR_NO_MERGE; } +/* + * Attempt to merge a bio into an existing request. This function is called + * before @bio is associated with a request. + */ static bool dd_bio_merge(struct request_queue *q, struct bio *bio, unsigned int nr_segs) { @@ -518,6 +531,9 @@ static void dd_insert_request(struct blk_mq_hw_ctx *hctx, struct request *rq, } } +/* + * Called from blk_mq_sched_insert_request() or blk_mq_sched_insert_requests(). + */ static void dd_insert_requests(struct blk_mq_hw_ctx *hctx, struct list_head *list, bool at_head) { @@ -544,6 +560,8 @@ static void dd_prepare_request(struct request *rq) } /* + * Callback from inside blk_mq_free_request(). + * * For zoned block devices, write unlock the target zone of * completed write requests. Do this while holding the zone lock * spinlock so that the zone is never unlocked while deadline_fifo_request() From patchwork Tue Jun 8 23:06:55 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bart Van Assche X-Patchwork-Id: 12308357 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2DD86C4743F for ; Tue, 8 Jun 2021 23:07:21 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 127EB61361 for ; Tue, 8 Jun 2021 23:07:21 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233574AbhFHXJN (ORCPT ); Tue, 8 Jun 2021 19:09:13 -0400 Received: from mail-pg1-f181.google.com ([209.85.215.181]:44849 "EHLO mail-pg1-f181.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232208AbhFHXJN (ORCPT ); Tue, 8 Jun 2021 19:09:13 -0400 Received: by mail-pg1-f181.google.com with SMTP id y11so9667950pgp.11 for ; Tue, 08 Jun 2021 16:07:20 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=sC//DSflrYYUUqwO5cZ8eH2Jsx8jSOvCgTNVQdA4IeM=; b=SWpeOU6bhRzoXrM3xM5h9m0wlWXHcWSvZVGDQ61m2cYl4roDgRvZmi7VKeZyWAfVwX Vt3QP5Q06PPquwnWc0FtbpDJlkizLlT3mUZPob5EDWJPouwvLl3hyxGJI9SWuxHfJs1V gZPHzUQJFGsmHqo7AKSm6ZxdCSGzpL7qCuWoAMXADnurs8qhe1NAZq6P5jcOA16WX6U4 ZOeBkqj+LJ1SohpNdGWCMEZFjwYtd0+YytDSFLvseN2aTQCpiysMaUKqTKh1ficrakJx QGBUzFdNaHrhle0Ob/GJoAtojhIgBy60zqVgzFz4QEojX0IOhfemyA+gEtlY7RG5ouy/ GjHQ== X-Gm-Message-State: AOAM533FOGXh5k2e2CyBjslQ13E3ScHJ7WiSx3k37despiyF/Zhm5uui dFqxnvUzFdNd19YrCnWYkw8= X-Google-Smtp-Source: ABdhPJx/VSoaOvVnVaGwWYqpAfDQeMZUdXzjsd/WR7q5fpvGfArhtD0/WYwzgXcrT+OhB+5nP7ODpg== X-Received: by 2002:a62:5285:0:b029:2e9:e0d5:67dc with SMTP id g127-20020a6252850000b02902e9e0d567dcmr2302182pfb.79.1623193639782; Tue, 08 Jun 2021 16:07:19 -0700 (PDT) Received: from asus.hsd1.ca.comcast.net (c-73-241-217-19.hsd1.ca.comcast.net. [73.241.217.19]) by smtp.gmail.com with ESMTPSA id s21sm11395838pfw.57.2021.06.08.16.07.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 08 Jun 2021 16:07:18 -0700 (PDT) From: Bart Van Assche To: Jens Axboe Cc: linux-block@vger.kernel.org, Christoph Hellwig , Jaegeuk Kim , Bart Van Assche , Chaitanya Kulkarni , Damien Le Moal , Hannes Reinecke , Johannes Thumshirn , Himanshu Madhani , Ming Lei Subject: [PATCH 06/14] block/mq-deadline: Add two lockdep_assert_held() statements Date: Tue, 8 Jun 2021 16:06:55 -0700 Message-Id: <20210608230703.19510-7-bvanassche@acm.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210608230703.19510-1-bvanassche@acm.org> References: <20210608230703.19510-1-bvanassche@acm.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org Document the locking strategy by adding two lockdep_assert_held() statements. Reviewed-by: Chaitanya Kulkarni Reviewed-by: Damien Le Moal Reviewed-by: Hannes Reinecke Reviewed-by: Johannes Thumshirn Reviewed-by: Himanshu Madhani Cc: Damien Le Moal Cc: Christoph Hellwig Cc: Ming Lei Signed-off-by: Bart Van Assche Reviewed-by: Hannes Reinecke --- block/mq-deadline.c | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/block/mq-deadline.c b/block/mq-deadline.c index 31418e9ce9e2..191ff5ce629c 100644 --- a/block/mq-deadline.c +++ b/block/mq-deadline.c @@ -279,6 +279,8 @@ static struct request *__dd_dispatch_request(struct deadline_data *dd) bool reads, writes; int data_dir; + lockdep_assert_held(&dd->lock); + if (!list_empty(&dd->dispatch)) { rq = list_first_entry(&dd->dispatch, struct request, queuelist); list_del_init(&rq->queuelist); @@ -501,6 +503,8 @@ static void dd_insert_request(struct blk_mq_hw_ctx *hctx, struct request *rq, struct deadline_data *dd = q->elevator->elevator_data; const int data_dir = rq_data_dir(rq); + lockdep_assert_held(&dd->lock); + /* * This may be a requeue of a write request that has locked its * target zone. If it is the case, this releases the zone lock. From patchwork Tue Jun 8 23:06:56 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bart Van Assche X-Patchwork-Id: 12308373 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 53FADC4743D for ; Tue, 8 Jun 2021 23:07:34 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 3D4926135D for ; Tue, 8 Jun 2021 23:07:34 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234256AbhFHXJ0 (ORCPT ); Tue, 8 Jun 2021 19:09:26 -0400 Received: from mail-pf1-f182.google.com ([209.85.210.182]:38658 "EHLO mail-pf1-f182.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235066AbhFHXJ0 (ORCPT ); Tue, 8 Jun 2021 19:09:26 -0400 Received: by mail-pf1-f182.google.com with SMTP id z26so16893755pfj.5 for ; Tue, 08 Jun 2021 16:07:21 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=8rkBqJ3UYA2WzpE+Shn97+2YWtxkFcR7b/FvEGWfprc=; b=bYPpzuyvALCXIN6XUvCs36Rr1oVZESlwyEBVk308JHIARsgATD05yLO6RGhz6P6iIs +nMwQt3oLO4hfQ/PKxL0qEOXmWy6f3WiwiJxihWhF53lsLKeQXpvZDCi7DGM0YSUL8q5 3ksFpuIQ0lZOFWjWnX9cLFa94l8K2cqXpy7EbJjR8gns3/RVE0/gtk4s2AtYa7nha7NN NmGpSUpipngSmJa6UU2jKaKtO6kLGxMsX5nyspgd4WLgQKyf9kAxIpVtnwmcefcl/bA5 Ax+3+2xiWzj7jCf2cDSu6aJ9G+nEGNak5RzsGS4UynGJSpvVGQxG8le4asYQPL7VYFgg ATOQ== X-Gm-Message-State: AOAM531BigXFQIzmoJjcwwG0InmGBeKU9wswAIYNPGjtisLzt6yAPhgW boS30GU+s4gG2/jzH7FQprM= X-Google-Smtp-Source: ABdhPJwVoFogZdrxecKZalsNHDC45x0mGOYuZwEGjyZLuFqj16SzqpSiqsKzfW3JeLTLX6mWaYob2g== X-Received: by 2002:a62:760a:0:b029:2e9:e039:20bc with SMTP id r10-20020a62760a0000b02902e9e03920bcmr2322869pfc.7.1623193641224; Tue, 08 Jun 2021 16:07:21 -0700 (PDT) Received: from asus.hsd1.ca.comcast.net (c-73-241-217-19.hsd1.ca.comcast.net. [73.241.217.19]) by smtp.gmail.com with ESMTPSA id s21sm11395838pfw.57.2021.06.08.16.07.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 08 Jun 2021 16:07:20 -0700 (PDT) From: Bart Van Assche To: Jens Axboe Cc: linux-block@vger.kernel.org, Christoph Hellwig , Jaegeuk Kim , Bart Van Assche , Chaitanya Kulkarni , Damien Le Moal , Hannes Reinecke , Johannes Thumshirn , Himanshu Madhani , Ming Lei Subject: [PATCH 07/14] block/mq-deadline: Remove two local variables Date: Tue, 8 Jun 2021 16:06:56 -0700 Message-Id: <20210608230703.19510-8-bvanassche@acm.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210608230703.19510-1-bvanassche@acm.org> References: <20210608230703.19510-1-bvanassche@acm.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org Make __dd_dispatch_request() easier to read by removing two local variables. Reviewed-by: Chaitanya Kulkarni Reviewed-by: Damien Le Moal Reviewed-by: Hannes Reinecke Reviewed-by: Johannes Thumshirn Reviewed-by: Himanshu Madhani Cc: Damien Le Moal Cc: Christoph Hellwig Cc: Ming Lei Signed-off-by: Bart Van Assche Reviewed-by: Hannes Reinecke --- block/mq-deadline.c | 8 ++------ 1 file changed, 2 insertions(+), 6 deletions(-) diff --git a/block/mq-deadline.c b/block/mq-deadline.c index 191ff5ce629c..caa438f62a4d 100644 --- a/block/mq-deadline.c +++ b/block/mq-deadline.c @@ -276,7 +276,6 @@ deadline_next_request(struct deadline_data *dd, int data_dir) static struct request *__dd_dispatch_request(struct deadline_data *dd) { struct request *rq, *next_rq; - bool reads, writes; int data_dir; lockdep_assert_held(&dd->lock); @@ -287,9 +286,6 @@ static struct request *__dd_dispatch_request(struct deadline_data *dd) goto done; } - reads = !list_empty(&dd->fifo_list[READ]); - writes = !list_empty(&dd->fifo_list[WRITE]); - /* * batches are currently reads XOR writes */ @@ -306,7 +302,7 @@ static struct request *__dd_dispatch_request(struct deadline_data *dd) * data direction (read / write) */ - if (reads) { + if (!list_empty(&dd->fifo_list[READ])) { BUG_ON(RB_EMPTY_ROOT(&dd->sort_list[READ])); if (deadline_fifo_request(dd, WRITE) && @@ -322,7 +318,7 @@ static struct request *__dd_dispatch_request(struct deadline_data *dd) * there are either no reads or writes have been starved */ - if (writes) { + if (!list_empty(&dd->fifo_list[WRITE])) { dispatch_writes: BUG_ON(RB_EMPTY_ROOT(&dd->sort_list[WRITE])); From patchwork Tue Jun 8 23:06:57 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bart Van Assche X-Patchwork-Id: 12308369 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E48B5C48BCD for ; Tue, 8 Jun 2021 23:07:37 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id C75C561361 for ; Tue, 8 Jun 2021 23:07:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235084AbhFHXJa (ORCPT ); Tue, 8 Jun 2021 19:09:30 -0400 Received: from mail-pf1-f169.google.com ([209.85.210.169]:35836 "EHLO mail-pf1-f169.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235116AbhFHXJ3 (ORCPT ); Tue, 8 Jun 2021 19:09:29 -0400 Received: by mail-pf1-f169.google.com with SMTP id h12so14066195pfe.2 for ; Tue, 08 Jun 2021 16:07:22 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=KvzcpYyxUXsxaJFPAswnHIuD6teUMMAJH/ZJ8ZfJmVQ=; b=Qcre7h2a4M9bRAle/elrg1uo5Hm3bmJPe4cPrjMnwfNubd+9suVDN+vSdZzSigPoiW 6QxjlQUQxUBicr0qkAxoaUcKI8p5s8FGZbzNBuldDYvRg3HWcPNqe4eMWugbUP5jw5/E 3L/2bnm1uSjEfCJDfP6cZhqrrd+eZNLyML1pOZgawysWjWfuDi5zF0506OUPsqGYsI1M iLn2qydKBCC/4JNsncLTmZS9950JxBmz/Cnbrs4PLcRZ7mEP3YFCRhOhRzGGLBqAq4qD z9/IqJRa/aJ2uWkX9lAOIS+2O71He59o9dJccDGNfn9Ni/X04G1/tPwjPTFMVS/ZNpNH XVlA== X-Gm-Message-State: AOAM530bNFKVCaM8E/cBRCZoqT1TDm94DAYtR0Cg2zpqeWL13yVUd8Ye e+95cLO2OQyLTZGmj8Rm3go= X-Google-Smtp-Source: ABdhPJzDWFCjKgw1M+OrP/pUzJPEGZHgCbB/GXnv4i4bC2swZS9skM0IVjT4E8ZAGK0fsMfcC+dZUQ== X-Received: by 2002:a63:451f:: with SMTP id s31mr596413pga.209.1623193642588; Tue, 08 Jun 2021 16:07:22 -0700 (PDT) Received: from asus.hsd1.ca.comcast.net (c-73-241-217-19.hsd1.ca.comcast.net. [73.241.217.19]) by smtp.gmail.com with ESMTPSA id s21sm11395838pfw.57.2021.06.08.16.07.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 08 Jun 2021 16:07:22 -0700 (PDT) From: Bart Van Assche To: Jens Axboe Cc: linux-block@vger.kernel.org, Christoph Hellwig , Jaegeuk Kim , Bart Van Assche , Chaitanya Kulkarni , Damien Le Moal , Hannes Reinecke , Johannes Thumshirn , Himanshu Madhani , Ming Lei Subject: [PATCH 08/14] block/mq-deadline: Rename dd_init_queue() and dd_exit_queue() Date: Tue, 8 Jun 2021 16:06:57 -0700 Message-Id: <20210608230703.19510-9-bvanassche@acm.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210608230703.19510-1-bvanassche@acm.org> References: <20210608230703.19510-1-bvanassche@acm.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org Change "queue" into "sched" to make the function names reflect better the purpose of these functions. Reviewed-by: Chaitanya Kulkarni Reviewed-by: Damien Le Moal Reviewed-by: Hannes Reinecke Reviewed-by: Johannes Thumshirn Reviewed-by: Himanshu Madhani Cc: Damien Le Moal Cc: Christoph Hellwig Cc: Ming Lei Signed-off-by: Bart Van Assche --- block/mq-deadline.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/block/mq-deadline.c b/block/mq-deadline.c index caa438f62a4d..d823ba7cb084 100644 --- a/block/mq-deadline.c +++ b/block/mq-deadline.c @@ -395,7 +395,7 @@ static struct request *dd_dispatch_request(struct blk_mq_hw_ctx *hctx) return rq; } -static void dd_exit_queue(struct elevator_queue *e) +static void dd_exit_sched(struct elevator_queue *e) { struct deadline_data *dd = e->elevator_data; @@ -408,7 +408,7 @@ static void dd_exit_queue(struct elevator_queue *e) /* * initialize elevator private data (deadline_data). */ -static int dd_init_queue(struct request_queue *q, struct elevator_type *e) +static int dd_init_sched(struct request_queue *q, struct elevator_type *e) { struct deadline_data *dd; struct elevator_queue *eq; @@ -800,8 +800,8 @@ static struct elevator_type mq_deadline = { .requests_merged = dd_merged_requests, .request_merged = dd_request_merged, .has_work = dd_has_work, - .init_sched = dd_init_queue, - .exit_sched = dd_exit_queue, + .init_sched = dd_init_sched, + .exit_sched = dd_exit_sched, }, #ifdef CONFIG_BLK_DEBUG_FS From patchwork Tue Jun 8 23:06:58 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bart Van Assche X-Patchwork-Id: 12308371 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-14.0 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,UNWANTED_LANGUAGE_BODY, URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 150A3C48BCF for ; Tue, 8 Jun 2021 23:07:38 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id EE1516135D for ; Tue, 8 Jun 2021 23:07:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234779AbhFHXJa (ORCPT ); Tue, 8 Jun 2021 19:09:30 -0400 Received: from mail-pf1-f179.google.com ([209.85.210.179]:41856 "EHLO mail-pf1-f179.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235198AbhFHXJa (ORCPT ); Tue, 8 Jun 2021 19:09:30 -0400 Received: by mail-pf1-f179.google.com with SMTP id x73so16912757pfc.8 for ; Tue, 08 Jun 2021 16:07:24 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=V9uU9/GDXAReAxWAZlxO0lics4XD9Pdny3ARKtCA2Ls=; b=cIrkpzhRNVavy/IgbJV7ax+ebJijLAmtS6vlxWXaGlOi7/KbtkGcZrhMzXnhZtjf6K AxC3/LP9Tq4DoOjr39w74S9VWN1YQjATO/isvrEX/Vxx+WGZ2f1hOlBbs5OF3m3prsNe iDBdnFBm5fv3tGCUChneSaljMxfC2dlkvdjAnzXyEkbQiKRnXbcKOzZiFFBd+S7GpT6f FxXN7TA2jj+KRyoISvXJQUD0EQ4jqi18pACmeT8mWjfb3HFeTJxI5Os0X90f7fJXtLyW bPy/WJGiCFDmteOJBq4zwM5HADUu8Eiy9Nq+iH5yyLb+Dd6hJ7nbfkfCktF7Cs8a4+3M 1pTw== X-Gm-Message-State: AOAM533a6uxhmX06HUCQWqzl4wGDZExRNs7e+8Rq/HKGKzG0SbYLsLtn dgi693alwd9woR8qxYOLEqA= X-Google-Smtp-Source: ABdhPJwsC1Xf/dm3hUlQ1P5Ji9ZZ8Fto5ZNUXL7p1/ZXPcbTSZJsm17AcnwV+1+9xzlHhk4bKZPrAg== X-Received: by 2002:a05:6a00:ac9:b029:2de:a06d:a52f with SMTP id c9-20020a056a000ac9b02902dea06da52fmr2411078pfl.4.1623193644046; Tue, 08 Jun 2021 16:07:24 -0700 (PDT) Received: from asus.hsd1.ca.comcast.net (c-73-241-217-19.hsd1.ca.comcast.net. [73.241.217.19]) by smtp.gmail.com with ESMTPSA id s21sm11395838pfw.57.2021.06.08.16.07.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 08 Jun 2021 16:07:23 -0700 (PDT) From: Bart Van Assche To: Jens Axboe Cc: linux-block@vger.kernel.org, Christoph Hellwig , Jaegeuk Kim , Bart Van Assche , Chaitanya Kulkarni , Hannes Reinecke , Johannes Thumshirn , Himanshu Madhani , Damien Le Moal , Ming Lei Subject: [PATCH 09/14] block/mq-deadline: Improve compile-time argument checking Date: Tue, 8 Jun 2021 16:06:58 -0700 Message-Id: <20210608230703.19510-10-bvanassche@acm.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210608230703.19510-1-bvanassche@acm.org> References: <20210608230703.19510-1-bvanassche@acm.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org Modern compilers complain if an out-of-range value is passed to a function argument that has an enumeration type. Let the compiler detect out-of-range data direction arguments instead of verifying the data_dir argument at runtime. Reviewed-by: Chaitanya Kulkarni Reviewed-by: Hannes Reinecke Reviewed-by: Johannes Thumshirn Reviewed-by: Himanshu Madhani Cc: Damien Le Moal Cc: Christoph Hellwig Cc: Ming Lei Signed-off-by: Bart Van Assche --- block/mq-deadline.c | 96 +++++++++++++++++++++++---------------------- 1 file changed, 49 insertions(+), 47 deletions(-) diff --git a/block/mq-deadline.c b/block/mq-deadline.c index d823ba7cb084..69126beff77d 100644 --- a/block/mq-deadline.c +++ b/block/mq-deadline.c @@ -35,6 +35,13 @@ static const int writes_starved = 2; /* max times reads can starve a write */ static const int fifo_batch = 16; /* # of sequential requests treated as one by the above parameters. For throughput. */ +enum dd_data_dir { + DD_READ = READ, + DD_WRITE = WRITE, +}; + +enum { DD_DIR_COUNT = 2 }; + struct deadline_data { /* * run time data @@ -43,20 +50,20 @@ struct deadline_data { /* * requests (deadline_rq s) are present on both sort_list and fifo_list */ - struct rb_root sort_list[2]; - struct list_head fifo_list[2]; + struct rb_root sort_list[DD_DIR_COUNT]; + struct list_head fifo_list[DD_DIR_COUNT]; /* * next in sort order. read, write or both are NULL */ - struct request *next_rq[2]; + struct request *next_rq[DD_DIR_COUNT]; unsigned int batching; /* number of sequential requests made */ unsigned int starved; /* times reads have starved writes */ /* * settings that change how the i/o scheduler behaves */ - int fifo_expire[2]; + int fifo_expire[DD_DIR_COUNT]; int fifo_batch; int writes_starved; int front_merges; @@ -97,7 +104,7 @@ deadline_add_rq_rb(struct deadline_data *dd, struct request *rq) static inline void deadline_del_rq_rb(struct deadline_data *dd, struct request *rq) { - const int data_dir = rq_data_dir(rq); + const enum dd_data_dir data_dir = rq_data_dir(rq); if (dd->next_rq[data_dir] == rq) dd->next_rq[data_dir] = deadline_latter_request(rq); @@ -169,10 +176,10 @@ static void dd_merged_requests(struct request_queue *q, struct request *req, static void deadline_move_request(struct deadline_data *dd, struct request *rq) { - const int data_dir = rq_data_dir(rq); + const enum dd_data_dir data_dir = rq_data_dir(rq); - dd->next_rq[READ] = NULL; - dd->next_rq[WRITE] = NULL; + dd->next_rq[DD_READ] = NULL; + dd->next_rq[DD_WRITE] = NULL; dd->next_rq[data_dir] = deadline_latter_request(rq); /* @@ -185,9 +192,10 @@ deadline_move_request(struct deadline_data *dd, struct request *rq) * deadline_check_fifo returns 0 if there are no expired requests on the fifo, * 1 otherwise. Requires !list_empty(&dd->fifo_list[data_dir]) */ -static inline int deadline_check_fifo(struct deadline_data *dd, int ddir) +static inline int deadline_check_fifo(struct deadline_data *dd, + enum dd_data_dir data_dir) { - struct request *rq = rq_entry_fifo(dd->fifo_list[ddir].next); + struct request *rq = rq_entry_fifo(dd->fifo_list[data_dir].next); /* * rq is expired! @@ -203,19 +211,16 @@ static inline int deadline_check_fifo(struct deadline_data *dd, int ddir) * dispatch using arrival ordered lists. */ static struct request * -deadline_fifo_request(struct deadline_data *dd, int data_dir) +deadline_fifo_request(struct deadline_data *dd, enum dd_data_dir data_dir) { struct request *rq; unsigned long flags; - if (WARN_ON_ONCE(data_dir != READ && data_dir != WRITE)) - return NULL; - if (list_empty(&dd->fifo_list[data_dir])) return NULL; rq = rq_entry_fifo(dd->fifo_list[data_dir].next); - if (data_dir == READ || !blk_queue_is_zoned(rq->q)) + if (data_dir == DD_READ || !blk_queue_is_zoned(rq->q)) return rq; /* @@ -223,7 +228,7 @@ deadline_fifo_request(struct deadline_data *dd, int data_dir) * an unlocked target zone. */ spin_lock_irqsave(&dd->zone_lock, flags); - list_for_each_entry(rq, &dd->fifo_list[WRITE], queuelist) { + list_for_each_entry(rq, &dd->fifo_list[DD_WRITE], queuelist) { if (blk_req_can_dispatch_to_zone(rq)) goto out; } @@ -239,19 +244,16 @@ deadline_fifo_request(struct deadline_data *dd, int data_dir) * dispatch using sector position sorted lists. */ static struct request * -deadline_next_request(struct deadline_data *dd, int data_dir) +deadline_next_request(struct deadline_data *dd, enum dd_data_dir data_dir) { struct request *rq; unsigned long flags; - if (WARN_ON_ONCE(data_dir != READ && data_dir != WRITE)) - return NULL; - rq = dd->next_rq[data_dir]; if (!rq) return NULL; - if (data_dir == READ || !blk_queue_is_zoned(rq->q)) + if (data_dir == DD_READ || !blk_queue_is_zoned(rq->q)) return rq; /* @@ -276,7 +278,7 @@ deadline_next_request(struct deadline_data *dd, int data_dir) static struct request *__dd_dispatch_request(struct deadline_data *dd) { struct request *rq, *next_rq; - int data_dir; + enum dd_data_dir data_dir; lockdep_assert_held(&dd->lock); @@ -289,9 +291,9 @@ static struct request *__dd_dispatch_request(struct deadline_data *dd) /* * batches are currently reads XOR writes */ - rq = deadline_next_request(dd, WRITE); + rq = deadline_next_request(dd, DD_WRITE); if (!rq) - rq = deadline_next_request(dd, READ); + rq = deadline_next_request(dd, DD_READ); if (rq && dd->batching < dd->fifo_batch) /* we have a next request are still entitled to batch */ @@ -302,14 +304,14 @@ static struct request *__dd_dispatch_request(struct deadline_data *dd) * data direction (read / write) */ - if (!list_empty(&dd->fifo_list[READ])) { - BUG_ON(RB_EMPTY_ROOT(&dd->sort_list[READ])); + if (!list_empty(&dd->fifo_list[DD_READ])) { + BUG_ON(RB_EMPTY_ROOT(&dd->sort_list[DD_READ])); - if (deadline_fifo_request(dd, WRITE) && + if (deadline_fifo_request(dd, DD_WRITE) && (dd->starved++ >= dd->writes_starved)) goto dispatch_writes; - data_dir = READ; + data_dir = DD_READ; goto dispatch_find_request; } @@ -318,13 +320,13 @@ static struct request *__dd_dispatch_request(struct deadline_data *dd) * there are either no reads or writes have been starved */ - if (!list_empty(&dd->fifo_list[WRITE])) { + if (!list_empty(&dd->fifo_list[DD_WRITE])) { dispatch_writes: - BUG_ON(RB_EMPTY_ROOT(&dd->sort_list[WRITE])); + BUG_ON(RB_EMPTY_ROOT(&dd->sort_list[DD_WRITE])); dd->starved = 0; - data_dir = WRITE; + data_dir = DD_WRITE; goto dispatch_find_request; } @@ -399,8 +401,8 @@ static void dd_exit_sched(struct elevator_queue *e) { struct deadline_data *dd = e->elevator_data; - BUG_ON(!list_empty(&dd->fifo_list[READ])); - BUG_ON(!list_empty(&dd->fifo_list[WRITE])); + BUG_ON(!list_empty(&dd->fifo_list[DD_READ])); + BUG_ON(!list_empty(&dd->fifo_list[DD_WRITE])); kfree(dd); } @@ -424,12 +426,12 @@ static int dd_init_sched(struct request_queue *q, struct elevator_type *e) } eq->elevator_data = dd; - INIT_LIST_HEAD(&dd->fifo_list[READ]); - INIT_LIST_HEAD(&dd->fifo_list[WRITE]); - dd->sort_list[READ] = RB_ROOT; - dd->sort_list[WRITE] = RB_ROOT; - dd->fifo_expire[READ] = read_expire; - dd->fifo_expire[WRITE] = write_expire; + INIT_LIST_HEAD(&dd->fifo_list[DD_READ]); + INIT_LIST_HEAD(&dd->fifo_list[DD_WRITE]); + dd->sort_list[DD_READ] = RB_ROOT; + dd->sort_list[DD_WRITE] = RB_ROOT; + dd->fifo_expire[DD_READ] = read_expire; + dd->fifo_expire[DD_WRITE] = write_expire; dd->writes_starved = writes_starved; dd->front_merges = 1; dd->fifo_batch = fifo_batch; @@ -497,7 +499,7 @@ static void dd_insert_request(struct blk_mq_hw_ctx *hctx, struct request *rq, { struct request_queue *q = hctx->queue; struct deadline_data *dd = q->elevator->elevator_data; - const int data_dir = rq_data_dir(rq); + const enum dd_data_dir data_dir = rq_data_dir(rq); lockdep_assert_held(&dd->lock); @@ -585,7 +587,7 @@ static void dd_finish_request(struct request *rq) spin_lock_irqsave(&dd->zone_lock, flags); blk_req_zone_write_unlock(rq); - if (!list_empty(&dd->fifo_list[WRITE])) + if (!list_empty(&dd->fifo_list[DD_WRITE])) blk_mq_sched_mark_restart_hctx(rq->mq_hctx); spin_unlock_irqrestore(&dd->zone_lock, flags); } @@ -626,8 +628,8 @@ static ssize_t __FUNC(struct elevator_queue *e, char *page) \ __data = jiffies_to_msecs(__data); \ return deadline_var_show(__data, (page)); \ } -SHOW_FUNCTION(deadline_read_expire_show, dd->fifo_expire[READ], 1); -SHOW_FUNCTION(deadline_write_expire_show, dd->fifo_expire[WRITE], 1); +SHOW_FUNCTION(deadline_read_expire_show, dd->fifo_expire[DD_READ], 1); +SHOW_FUNCTION(deadline_write_expire_show, dd->fifo_expire[DD_WRITE], 1); SHOW_FUNCTION(deadline_writes_starved_show, dd->writes_starved, 0); SHOW_FUNCTION(deadline_front_merges_show, dd->front_merges, 0); SHOW_FUNCTION(deadline_fifo_batch_show, dd->fifo_batch, 0); @@ -649,8 +651,8 @@ static ssize_t __FUNC(struct elevator_queue *e, const char *page, size_t count) *(__PTR) = __data; \ return count; \ } -STORE_FUNCTION(deadline_read_expire_store, &dd->fifo_expire[READ], 0, INT_MAX, 1); -STORE_FUNCTION(deadline_write_expire_store, &dd->fifo_expire[WRITE], 0, INT_MAX, 1); +STORE_FUNCTION(deadline_read_expire_store, &dd->fifo_expire[DD_READ], 0, INT_MAX, 1); +STORE_FUNCTION(deadline_write_expire_store, &dd->fifo_expire[DD_WRITE], 0, INT_MAX, 1); STORE_FUNCTION(deadline_writes_starved_store, &dd->writes_starved, INT_MIN, INT_MAX, 0); STORE_FUNCTION(deadline_front_merges_store, &dd->front_merges, 0, 1, 0); STORE_FUNCTION(deadline_fifo_batch_store, &dd->fifo_batch, 0, INT_MAX, 0); @@ -717,8 +719,8 @@ static int deadline_##name##_next_rq_show(void *data, \ __blk_mq_debugfs_rq_show(m, rq); \ return 0; \ } -DEADLINE_DEBUGFS_DDIR_ATTRS(READ, read) -DEADLINE_DEBUGFS_DDIR_ATTRS(WRITE, write) +DEADLINE_DEBUGFS_DDIR_ATTRS(DD_READ, read) +DEADLINE_DEBUGFS_DDIR_ATTRS(DD_WRITE, write) #undef DEADLINE_DEBUGFS_DDIR_ATTRS static int deadline_batching_show(void *data, struct seq_file *m) From patchwork Tue Jun 8 23:06:59 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bart Van Assche X-Patchwork-Id: 12308375 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8D191C47082 for ; Tue, 8 Jun 2021 23:07:40 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 78E866135D for ; Tue, 8 Jun 2021 23:07:40 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235198AbhFHXJd (ORCPT ); Tue, 8 Jun 2021 19:09:33 -0400 Received: from mail-pl1-f178.google.com ([209.85.214.178]:43867 "EHLO mail-pl1-f178.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235116AbhFHXJc (ORCPT ); Tue, 8 Jun 2021 19:09:32 -0400 Received: by mail-pl1-f178.google.com with SMTP id v12so11509672plo.10 for ; Tue, 08 Jun 2021 16:07:25 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=hFU+v6PrYbG3GwJsak4IZ52nG2edvhdZ46b6T0mqhvA=; b=W7/wUjnU1uV4PZWHDRQE5Fx2ASrYwA7DN9lf4KxM5lYHs9O32NJc040ZUHNaLN6DNA YUu9NZm5i6zwOoexNY4nb9BNBHJ0CF1Ix3CcpG7eMi9MWiTeXRReJVTTEaU7jtj1efzn SXJJUt9XuRdKFRa4YC2ixmSvxIaNo+Y2RxWEzXoHmdUWembJmnmdq2y53xZogNT9+27C u9ptBpyJDbpbn2fOGJJY9fTDxX0N+gKcCRLLN7sRvQJSlUD3tJ8TxBwJXf9eR+cULi49 DrVDUuf75nbrT22HKJov5yzi7+MELs71myrA8B4Y2d/I8dw1Y0SFqIhLSXGGNladCn4O MvEA== X-Gm-Message-State: AOAM533j73i13lJ+6U1x1RwmfuCfHar9B8msSEmm5YrvW+bX1lvM8VKi 06WPnlQIxIk0wni4z3+ngYA= X-Google-Smtp-Source: ABdhPJwwauikGuzYrfCr4eZYGDSBY9DrDyFof9rAYmvnPaR4U89gMseMNZ/WTlwPYhX+igP9108mUw== X-Received: by 2002:a17:902:b58d:b029:114:7f9a:efd9 with SMTP id a13-20020a170902b58db02901147f9aefd9mr2024090pls.63.1623193645461; Tue, 08 Jun 2021 16:07:25 -0700 (PDT) Received: from asus.hsd1.ca.comcast.net (c-73-241-217-19.hsd1.ca.comcast.net. [73.241.217.19]) by smtp.gmail.com with ESMTPSA id s21sm11395838pfw.57.2021.06.08.16.07.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 08 Jun 2021 16:07:24 -0700 (PDT) From: Bart Van Assche To: Jens Axboe Cc: linux-block@vger.kernel.org, Christoph Hellwig , Jaegeuk Kim , Bart Van Assche , Damien Le Moal , Hannes Reinecke , Ming Lei , Johannes Thumshirn , Himanshu Madhani Subject: [PATCH 10/14] block/mq-deadline: Improve the sysfs show and store macros Date: Tue, 8 Jun 2021 16:06:59 -0700 Message-Id: <20210608230703.19510-11-bvanassche@acm.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210608230703.19510-1-bvanassche@acm.org> References: <20210608230703.19510-1-bvanassche@acm.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org Define separate macros for integers and jiffies to improve readability. Use sysfs_emit() and kstrtoint() instead of sprintf() and simple_strtol() since the former functions are the recommended functions. Cc: Damien Le Moal Cc: Hannes Reinecke Cc: Christoph Hellwig Cc: Ming Lei Cc: Johannes Thumshirn Cc: Himanshu Madhani Signed-off-by: Bart Van Assche Reviewed-by: Hannes Reinecke --- block/mq-deadline.c | 66 ++++++++++++++++++++------------------------- 1 file changed, 29 insertions(+), 37 deletions(-) diff --git a/block/mq-deadline.c b/block/mq-deadline.c index 69126beff77d..1d1bb7a41d2a 100644 --- a/block/mq-deadline.c +++ b/block/mq-deadline.c @@ -605,58 +605,50 @@ static bool dd_has_work(struct blk_mq_hw_ctx *hctx) /* * sysfs parts below */ -static ssize_t -deadline_var_show(int var, char *page) -{ - return sprintf(page, "%d\n", var); -} - -static void -deadline_var_store(int *var, const char *page) -{ - char *p = (char *) page; - - *var = simple_strtol(p, &p, 10); -} - -#define SHOW_FUNCTION(__FUNC, __VAR, __CONV) \ +#define SHOW_INT(__FUNC, __VAR) \ static ssize_t __FUNC(struct elevator_queue *e, char *page) \ { \ struct deadline_data *dd = e->elevator_data; \ - int __data = __VAR; \ - if (__CONV) \ - __data = jiffies_to_msecs(__data); \ - return deadline_var_show(__data, (page)); \ -} -SHOW_FUNCTION(deadline_read_expire_show, dd->fifo_expire[DD_READ], 1); -SHOW_FUNCTION(deadline_write_expire_show, dd->fifo_expire[DD_WRITE], 1); -SHOW_FUNCTION(deadline_writes_starved_show, dd->writes_starved, 0); -SHOW_FUNCTION(deadline_front_merges_show, dd->front_merges, 0); -SHOW_FUNCTION(deadline_fifo_batch_show, dd->fifo_batch, 0); -#undef SHOW_FUNCTION + \ + return sysfs_emit((page), "%d\n", __VAR); \ +} +#define SHOW_JIFFIES(__FUNC, __VAR) SHOW_INT(__FUNC, jiffies_to_msecs(__VAR)) +SHOW_JIFFIES(deadline_read_expire_show, dd->fifo_expire[DD_READ]); +SHOW_JIFFIES(deadline_write_expire_show, dd->fifo_expire[DD_WRITE]); +SHOW_INT(deadline_writes_starved_show, dd->writes_starved); +SHOW_INT(deadline_front_merges_show, dd->front_merges); +SHOW_INT(deadline_fifo_batch_show, dd->fifo_batch); +#undef SHOW_INT +#undef SHOW_JIFFIES #define STORE_FUNCTION(__FUNC, __PTR, MIN, MAX, __CONV) \ static ssize_t __FUNC(struct elevator_queue *e, const char *page, size_t count) \ { \ struct deadline_data *dd = e->elevator_data; \ - int __data; \ - deadline_var_store(&__data, (page)); \ + int __data, __ret; \ + \ + __ret = kstrtoint((page), 0, &__data); \ + if (__ret < 0) \ + return __ret; \ if (__data < (MIN)) \ __data = (MIN); \ else if (__data > (MAX)) \ __data = (MAX); \ - if (__CONV) \ - *(__PTR) = msecs_to_jiffies(__data); \ - else \ - *(__PTR) = __data; \ + *(__PTR) = __CONV(__data); \ return count; \ } -STORE_FUNCTION(deadline_read_expire_store, &dd->fifo_expire[DD_READ], 0, INT_MAX, 1); -STORE_FUNCTION(deadline_write_expire_store, &dd->fifo_expire[DD_WRITE], 0, INT_MAX, 1); -STORE_FUNCTION(deadline_writes_starved_store, &dd->writes_starved, INT_MIN, INT_MAX, 0); -STORE_FUNCTION(deadline_front_merges_store, &dd->front_merges, 0, 1, 0); -STORE_FUNCTION(deadline_fifo_batch_store, &dd->fifo_batch, 0, INT_MAX, 0); +#define STORE_INT(__FUNC, __PTR, MIN, MAX) \ + STORE_FUNCTION(__FUNC, __PTR, MIN, MAX, ) +#define STORE_JIFFIES(__FUNC, __PTR, MIN, MAX) \ + STORE_FUNCTION(__FUNC, __PTR, MIN, MAX, msecs_to_jiffies) +STORE_JIFFIES(deadline_read_expire_store, &dd->fifo_expire[DD_READ], 0, INT_MAX); +STORE_JIFFIES(deadline_write_expire_store, &dd->fifo_expire[DD_WRITE], 0, INT_MAX); +STORE_INT(deadline_writes_starved_store, &dd->writes_starved, INT_MIN, INT_MAX); +STORE_INT(deadline_front_merges_store, &dd->front_merges, 0, 1); +STORE_INT(deadline_fifo_batch_store, &dd->fifo_batch, 0, INT_MAX); #undef STORE_FUNCTION +#undef STORE_INT +#undef STORE_JIFFIES #define DD_ATTR(name) \ __ATTR(name, 0644, deadline_##name##_show, deadline_##name##_store) From patchwork Tue Jun 8 23:07:00 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bart Van Assche X-Patchwork-Id: 12308383 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0E88DC48BDF for ; Tue, 8 Jun 2021 23:07:44 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id E7C8961278 for ; Tue, 8 Jun 2021 23:07:43 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235308AbhFHXJg (ORCPT ); Tue, 8 Jun 2021 19:09:36 -0400 Received: from mail-pl1-f172.google.com ([209.85.214.172]:40827 "EHLO mail-pl1-f172.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235225AbhFHXJg (ORCPT ); Tue, 8 Jun 2021 19:09:36 -0400 Received: by mail-pl1-f172.google.com with SMTP id e7so11519805plj.7 for ; Tue, 08 Jun 2021 16:07:28 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=DKj20C0bcnYEdi2no4p//QdL1zitAi+dZIPfHAogfi4=; b=HtXbnJEGzK+uG2rr4QWv7ZeNZ3/IXU2keJuEgT2A9gQdDZ2du9ztjhvA0XVgD9VUHL /RZnMuISqcNblVR3dvLhdYelNyMrVfAWXG7yw06nMwyVeADTywSm6lHrVWvFNqcRrEbN Y/CJjiiF8G+nRqaEPmHE3uiY2cFqkMgqMpF/yPefrX7dUA0SCqtv6YGzzmHkc0kHkKrD heT6j0yWHuPpjCZNBGynqj4I/Y+JJDbOwWfizeAoNKDtcu25/sMyTKKAXZ71jd1s3l54 3MKuXuFLzGvyEB/h+X2uUqKEUeMYOikexo6ow3INHPV1kUVEQM0ebz8GXXhE2DZxbg5x /+/A== X-Gm-Message-State: AOAM530OUzJYt4+mlZgtxA+wMIZHP6qf0phArVUFtEPR5r1t+C/kT24R imoZN/buktp1Fsgpv5NFdYU= X-Google-Smtp-Source: ABdhPJzAXkQGT2eBDgLCNWGRzUpLfOZhR0WuDVmyj3TAQEa4od/srKyR6+ufSc//U+yN+QEu4D/HXg== X-Received: by 2002:a17:90a:5b17:: with SMTP id o23mr28175525pji.14.1623193647698; Tue, 08 Jun 2021 16:07:27 -0700 (PDT) Received: from asus.hsd1.ca.comcast.net (c-73-241-217-19.hsd1.ca.comcast.net. [73.241.217.19]) by smtp.gmail.com with ESMTPSA id s21sm11395838pfw.57.2021.06.08.16.07.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 08 Jun 2021 16:07:27 -0700 (PDT) From: Bart Van Assche To: Jens Axboe Cc: linux-block@vger.kernel.org, Christoph Hellwig , Jaegeuk Kim , Bart Van Assche , Damien Le Moal , Hannes Reinecke , Ming Lei , Johannes Thumshirn , Himanshu Madhani Subject: [PATCH 11/14] block/mq-deadline: Reserve 25% of scheduler tags for synchronous requests Date: Tue, 8 Jun 2021 16:07:00 -0700 Message-Id: <20210608230703.19510-12-bvanassche@acm.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210608230703.19510-1-bvanassche@acm.org> References: <20210608230703.19510-1-bvanassche@acm.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org For interactive workloads it is important that synchronous requests are not delayed. Hence reserve 25% of scheduler tags for synchronous requests. This patch still allows asynchronous requests to fill the hardware queues since blk_mq_init_sched() makes sure that the number of scheduler requests is the double of the hardware queue depth. From blk_mq_init_sched(): q->nr_requests = 2 * min_t(unsigned int, q->tag_set->queue_depth, BLKDEV_MAX_RQ); Cc: Damien Le Moal Cc: Hannes Reinecke Cc: Christoph Hellwig Cc: Ming Lei Cc: Johannes Thumshirn Cc: Himanshu Madhani Signed-off-by: Bart Van Assche --- block/mq-deadline.c | 52 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 52 insertions(+) diff --git a/block/mq-deadline.c b/block/mq-deadline.c index 1d1bb7a41d2a..a7d0584437d1 100644 --- a/block/mq-deadline.c +++ b/block/mq-deadline.c @@ -67,6 +67,7 @@ struct deadline_data { int fifo_batch; int writes_starved; int front_merges; + u32 async_depth; spinlock_t lock; spinlock_t zone_lock; @@ -397,6 +398,44 @@ static struct request *dd_dispatch_request(struct blk_mq_hw_ctx *hctx) return rq; } +/* + * Called by __blk_mq_alloc_request(). The shallow_depth value set by this + * function is used by __blk_mq_get_tag(). + */ +static void dd_limit_depth(unsigned int op, struct blk_mq_alloc_data *data) +{ + struct deadline_data *dd = data->q->elevator->elevator_data; + + /* Do not throttle synchronous reads. */ + if (op_is_sync(op) && !op_is_write(op)) + return; + + /* + * Throttle asynchronous requests and writes such that these requests + * do not block the allocation of synchronous requests. + */ + data->shallow_depth = dd->async_depth; +} + +/* Called by blk_mq_update_nr_requests(). */ +static void dd_depth_updated(struct blk_mq_hw_ctx *hctx) +{ + struct request_queue *q = hctx->queue; + struct deadline_data *dd = q->elevator->elevator_data; + struct blk_mq_tags *tags = hctx->sched_tags; + + dd->async_depth = max(1UL, 3 * q->nr_requests / 4); + + sbitmap_queue_min_shallow_depth(tags->bitmap_tags, dd->async_depth); +} + +/* Called by blk_mq_init_hctx() and blk_mq_init_sched(). */ +static int dd_init_hctx(struct blk_mq_hw_ctx *hctx, unsigned int hctx_idx) +{ + dd_depth_updated(hctx); + return 0; +} + static void dd_exit_sched(struct elevator_queue *e) { struct deadline_data *dd = e->elevator_data; @@ -733,6 +772,15 @@ static int deadline_starved_show(void *data, struct seq_file *m) return 0; } +static int dd_async_depth_show(void *data, struct seq_file *m) +{ + struct request_queue *q = data; + struct deadline_data *dd = q->elevator->elevator_data; + + seq_printf(m, "%u\n", dd->async_depth); + return 0; +} + static void *deadline_dispatch_start(struct seq_file *m, loff_t *pos) __acquires(&dd->lock) { @@ -775,6 +823,7 @@ static const struct blk_mq_debugfs_attr deadline_queue_debugfs_attrs[] = { DEADLINE_QUEUE_DDIR_ATTRS(write), {"batching", 0400, deadline_batching_show}, {"starved", 0400, deadline_starved_show}, + {"async_depth", 0400, dd_async_depth_show}, {"dispatch", 0400, .seq_ops = &deadline_dispatch_seq_ops}, {}, }; @@ -783,6 +832,8 @@ static const struct blk_mq_debugfs_attr deadline_queue_debugfs_attrs[] = { static struct elevator_type mq_deadline = { .ops = { + .depth_updated = dd_depth_updated, + .limit_depth = dd_limit_depth, .insert_requests = dd_insert_requests, .dispatch_request = dd_dispatch_request, .prepare_request = dd_prepare_request, @@ -796,6 +847,7 @@ static struct elevator_type mq_deadline = { .has_work = dd_has_work, .init_sched = dd_init_sched, .exit_sched = dd_exit_sched, + .init_hctx = dd_init_hctx, }, #ifdef CONFIG_BLK_DEBUG_FS From patchwork Tue Jun 8 23:07:01 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bart Van Assche X-Patchwork-Id: 12308381 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8D35DC48BE0 for ; Tue, 8 Jun 2021 23:07:43 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 6E00F6135D for ; Tue, 8 Jun 2021 23:07:43 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235066AbhFHXJg (ORCPT ); Tue, 8 Jun 2021 19:09:36 -0400 Received: from mail-pj1-f42.google.com ([209.85.216.42]:54000 "EHLO mail-pj1-f42.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235225AbhFHXJf (ORCPT ); Tue, 8 Jun 2021 19:09:35 -0400 Received: by mail-pj1-f42.google.com with SMTP id ei4so244479pjb.3 for ; Tue, 08 Jun 2021 16:07:29 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=h4UhUwQsGVbqI68Pq882qG8sntQOLhzXkVT2RaGYaMU=; b=WlTLqkpD/Zlne0ahM5uhsOdMgN7IYmTrf1R9gGX49SldTRlnptF0BpV7VuDHDkEWEQ UXQmoCQDRJLNo73ToHxxJ92Q4RW0yTn2CGA/V1fRAJz8Dk55NvMgy5GMn4IqRzyrm9zq nVDh7y8IngtchZh+BHbYbRg4dqmtBSLjRYUwzjdFMEmd0OK7zCq6c4dj57WqyYTYsK1+ hyVcYzlxYj1jEGRj3daFevcGfHk0T7GlYD628dSaX8iqabFUR1CV4D0AT154K8W6PcUD olru0Gy6NLehHGJCTpL1W8K/sHzeSM1TyhXpQxsMRfM41lTqOCRwjW7livo8YEWDJ2fL bsXQ== X-Gm-Message-State: AOAM532rVlN8nYNZv6MkrarqufiVJAGMGVsX8QHXzcXrgFxm32eyfIz2 fBNKlX941d5eYjk9YCDmmV4= X-Google-Smtp-Source: ABdhPJwEDI5s6SUYR0kpXSYe9pMnsX7yUbhXhCBr0Ff/DV6r1im6q1m+wpgfg2g8QvAYqK0iGoYhSw== X-Received: by 2002:a17:902:7087:b029:104:6133:6d2d with SMTP id z7-20020a1709027087b029010461336d2dmr2432353plk.39.1623193649117; Tue, 08 Jun 2021 16:07:29 -0700 (PDT) Received: from asus.hsd1.ca.comcast.net (c-73-241-217-19.hsd1.ca.comcast.net. [73.241.217.19]) by smtp.gmail.com with ESMTPSA id s21sm11395838pfw.57.2021.06.08.16.07.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 08 Jun 2021 16:07:28 -0700 (PDT) From: Bart Van Assche To: Jens Axboe Cc: linux-block@vger.kernel.org, Christoph Hellwig , Jaegeuk Kim , Bart Van Assche , Damien Le Moal , Hannes Reinecke , Ming Lei , Johannes Thumshirn , Himanshu Madhani Subject: [PATCH 12/14] block/mq-deadline: Add I/O priority support Date: Tue, 8 Jun 2021 16:07:01 -0700 Message-Id: <20210608230703.19510-13-bvanassche@acm.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210608230703.19510-1-bvanassche@acm.org> References: <20210608230703.19510-1-bvanassche@acm.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org Maintain one dispatch list and FIFO one list per I/O priority class: RT, BE and IDLE. Maintain statistics for each priority level. Split the debugfs attributes per priority level as follows: $ ls /sys/kernel/debug/block/.../sched/ async_depth dispatch2 read_next_rq write2_fifo_list batching read0_fifo_list starved write_next_rq dispatch0 read1_fifo_list write0_fifo_list dispatch1 read2_fifo_list write1_fifo_list Cc: Damien Le Moal Cc: Hannes Reinecke Cc: Christoph Hellwig Cc: Ming Lei Cc: Johannes Thumshirn Cc: Himanshu Madhani Signed-off-by: Bart Van Assche --- block/mq-deadline.c | 355 ++++++++++++++++++++++++++++++++++---------- 1 file changed, 279 insertions(+), 76 deletions(-) diff --git a/block/mq-deadline.c b/block/mq-deadline.c index a7d0584437d1..776ff49713c3 100644 --- a/block/mq-deadline.c +++ b/block/mq-deadline.c @@ -42,16 +42,40 @@ enum dd_data_dir { enum { DD_DIR_COUNT = 2 }; +enum dd_prio { + DD_RT_PRIO = 0, + DD_BE_PRIO = 1, + DD_IDLE_PRIO = 2, + DD_PRIO_MAX = 2, +}; + +enum { DD_PRIO_COUNT = 3 }; + +/* I/O statistics per I/O priority. */ +struct io_stats_per_prio { + local_t inserted; + local_t merged; + local_t dispatched; + local_t completed; +}; + +/* I/O statistics for all I/O priorities (enum dd_prio). */ +struct io_stats { + struct io_stats_per_prio stats[DD_PRIO_COUNT]; +}; + struct deadline_data { /* * run time data */ /* - * requests (deadline_rq s) are present on both sort_list and fifo_list + * Requests are present on both sort_list[] and fifo_list[][]. The + * first index of fifo_list[][] is the I/O priority class (DD_*_PRIO). + * The second index is the data direction (rq_data_dir(rq)). */ struct rb_root sort_list[DD_DIR_COUNT]; - struct list_head fifo_list[DD_DIR_COUNT]; + struct list_head fifo_list[DD_PRIO_COUNT][DD_DIR_COUNT]; /* * next in sort order. read, write or both are NULL @@ -60,6 +84,8 @@ struct deadline_data { unsigned int batching; /* number of sequential requests made */ unsigned int starved; /* times reads have starved writes */ + struct io_stats __percpu *stats; + /* * settings that change how the i/o scheduler behaves */ @@ -71,7 +97,42 @@ struct deadline_data { spinlock_t lock; spinlock_t zone_lock; - struct list_head dispatch; + struct list_head dispatch[DD_PRIO_COUNT]; +}; + +/* Count one event of type 'event_type' and with I/O priority 'prio' */ +#define dd_count(dd, event_type, prio) do { \ + struct io_stats *io_stats = get_cpu_ptr((dd)->stats); \ + \ + BUILD_BUG_ON(!__same_type((dd), struct deadline_data *)); \ + BUILD_BUG_ON(!__same_type((prio), enum dd_prio)); \ + local_inc(&io_stats->stats[(prio)].event_type); \ + put_cpu_ptr(io_stats); \ +} while (0) + +/* + * Returns the total number of dd_count(dd, event_type, prio) calls across all + * CPUs. No locking or barriers since it is fine if the returned sum is slightly + * outdated. + */ +#define dd_sum(dd, event_type, prio) ({ \ + unsigned int cpu; \ + u32 sum = 0; \ + \ + BUILD_BUG_ON(!__same_type((dd), struct deadline_data *)); \ + BUILD_BUG_ON(!__same_type((prio), enum dd_prio)); \ + for_each_present_cpu(cpu) \ + sum += local_read(&per_cpu_ptr((dd)->stats, cpu)-> \ + stats[(prio)].event_type); \ + sum; \ +}) + +/* Maps an I/O priority class to a deadline scheduler priority. */ +static const enum dd_prio ioprio_class_to_prio[] = { + [IOPRIO_CLASS_NONE] = DD_BE_PRIO, + [IOPRIO_CLASS_RT] = DD_RT_PRIO, + [IOPRIO_CLASS_BE] = DD_BE_PRIO, + [IOPRIO_CLASS_IDLE] = DD_IDLE_PRIO, }; static inline struct rb_root * @@ -147,12 +208,31 @@ static void dd_request_merged(struct request_queue *q, struct request *req, } } +/* + * Returns the I/O priority class (IOPRIO_CLASS_*) that has been assigned to a + * request. + */ +static u8 dd_rq_ioclass(struct request *rq) +{ + return IOPRIO_PRIO_CLASS(req_get_ioprio(rq)); +} + /* * Callback function that is invoked after @next has been merged into @req. */ static void dd_merged_requests(struct request_queue *q, struct request *req, struct request *next) { + struct deadline_data *dd = q->elevator->elevator_data; + const u8 ioprio_class = dd_rq_ioclass(next); + const enum dd_prio prio = ioprio_class_to_prio[ioprio_class]; + + if (next->elv.priv[0]) { + dd_count(dd, merged, prio); + } else { + WARN_ON_ONCE(true); + } + /* * if next expires before rq, assign its expire time to rq * and move into next position (next will be deleted) in fifo @@ -189,14 +269,21 @@ deadline_move_request(struct deadline_data *dd, struct request *rq) deadline_remove_request(rq->q, rq); } +/* Number of requests queued for a given priority level. */ +static u32 dd_queued(struct deadline_data *dd, enum dd_prio prio) +{ + return dd_sum(dd, inserted, prio) - dd_sum(dd, completed, prio); +} + /* * deadline_check_fifo returns 0 if there are no expired requests on the fifo, * 1 otherwise. Requires !list_empty(&dd->fifo_list[data_dir]) */ static inline int deadline_check_fifo(struct deadline_data *dd, + enum dd_prio prio, enum dd_data_dir data_dir) { - struct request *rq = rq_entry_fifo(dd->fifo_list[data_dir].next); + struct request *rq = rq_entry_fifo(dd->fifo_list[prio][data_dir].next); /* * rq is expired! @@ -212,15 +299,16 @@ static inline int deadline_check_fifo(struct deadline_data *dd, * dispatch using arrival ordered lists. */ static struct request * -deadline_fifo_request(struct deadline_data *dd, enum dd_data_dir data_dir) +deadline_fifo_request(struct deadline_data *dd, enum dd_prio prio, + enum dd_data_dir data_dir) { struct request *rq; unsigned long flags; - if (list_empty(&dd->fifo_list[data_dir])) + if (list_empty(&dd->fifo_list[prio][data_dir])) return NULL; - rq = rq_entry_fifo(dd->fifo_list[data_dir].next); + rq = rq_entry_fifo(dd->fifo_list[prio][data_dir].next); if (data_dir == DD_READ || !blk_queue_is_zoned(rq->q)) return rq; @@ -229,7 +317,7 @@ deadline_fifo_request(struct deadline_data *dd, enum dd_data_dir data_dir) * an unlocked target zone. */ spin_lock_irqsave(&dd->zone_lock, flags); - list_for_each_entry(rq, &dd->fifo_list[DD_WRITE], queuelist) { + list_for_each_entry(rq, &dd->fifo_list[prio][DD_WRITE], queuelist) { if (blk_req_can_dispatch_to_zone(rq)) goto out; } @@ -245,7 +333,8 @@ deadline_fifo_request(struct deadline_data *dd, enum dd_data_dir data_dir) * dispatch using sector position sorted lists. */ static struct request * -deadline_next_request(struct deadline_data *dd, enum dd_data_dir data_dir) +deadline_next_request(struct deadline_data *dd, enum dd_prio prio, + enum dd_data_dir data_dir) { struct request *rq; unsigned long flags; @@ -276,15 +365,18 @@ deadline_next_request(struct deadline_data *dd, enum dd_data_dir data_dir) * deadline_dispatch_requests selects the best request according to * read/write expire, fifo_batch, etc */ -static struct request *__dd_dispatch_request(struct deadline_data *dd) +static struct request *__dd_dispatch_request(struct deadline_data *dd, + enum dd_prio prio) { struct request *rq, *next_rq; enum dd_data_dir data_dir; + u8 ioprio_class; lockdep_assert_held(&dd->lock); - if (!list_empty(&dd->dispatch)) { - rq = list_first_entry(&dd->dispatch, struct request, queuelist); + if (!list_empty(&dd->dispatch[prio])) { + rq = list_first_entry(&dd->dispatch[prio], struct request, + queuelist); list_del_init(&rq->queuelist); goto done; } @@ -292,9 +384,9 @@ static struct request *__dd_dispatch_request(struct deadline_data *dd) /* * batches are currently reads XOR writes */ - rq = deadline_next_request(dd, DD_WRITE); + rq = deadline_next_request(dd, prio, DD_WRITE); if (!rq) - rq = deadline_next_request(dd, DD_READ); + rq = deadline_next_request(dd, prio, DD_READ); if (rq && dd->batching < dd->fifo_batch) /* we have a next request are still entitled to batch */ @@ -305,10 +397,10 @@ static struct request *__dd_dispatch_request(struct deadline_data *dd) * data direction (read / write) */ - if (!list_empty(&dd->fifo_list[DD_READ])) { + if (!list_empty(&dd->fifo_list[prio][DD_READ])) { BUG_ON(RB_EMPTY_ROOT(&dd->sort_list[DD_READ])); - if (deadline_fifo_request(dd, DD_WRITE) && + if (deadline_fifo_request(dd, prio, DD_WRITE) && (dd->starved++ >= dd->writes_starved)) goto dispatch_writes; @@ -321,7 +413,7 @@ static struct request *__dd_dispatch_request(struct deadline_data *dd) * there are either no reads or writes have been starved */ - if (!list_empty(&dd->fifo_list[DD_WRITE])) { + if (!list_empty(&dd->fifo_list[prio][DD_WRITE])) { dispatch_writes: BUG_ON(RB_EMPTY_ROOT(&dd->sort_list[DD_WRITE])); @@ -338,14 +430,14 @@ static struct request *__dd_dispatch_request(struct deadline_data *dd) /* * we are not running a batch, find best request for selected data_dir */ - next_rq = deadline_next_request(dd, data_dir); - if (deadline_check_fifo(dd, data_dir) || !next_rq) { + next_rq = deadline_next_request(dd, prio, data_dir); + if (deadline_check_fifo(dd, prio, data_dir) || !next_rq) { /* * A deadline has expired, the last request was in the other * direction, or we have run out of higher-sectored requests. * Start again from the request with the earliest expiry time. */ - rq = deadline_fifo_request(dd, data_dir); + rq = deadline_fifo_request(dd, prio, data_dir); } else { /* * The last req was the same dir and we have a next request in @@ -370,6 +462,13 @@ static struct request *__dd_dispatch_request(struct deadline_data *dd) dd->batching++; deadline_move_request(dd, rq); done: + ioprio_class = dd_rq_ioclass(rq); + prio = ioprio_class_to_prio[ioprio_class]; + if (rq->elv.priv[0]) { + dd_count(dd, dispatched, prio); + } else { + WARN_ON_ONCE(true); + } /* * If the request needs its target zone locked, do it. */ @@ -390,9 +489,14 @@ static struct request *dd_dispatch_request(struct blk_mq_hw_ctx *hctx) { struct deadline_data *dd = hctx->queue->elevator->elevator_data; struct request *rq; + enum dd_prio prio; spin_lock(&dd->lock); - rq = __dd_dispatch_request(dd); + for (prio = 0; prio <= DD_PRIO_MAX; prio++) { + rq = __dd_dispatch_request(dd, prio); + if (rq) + break; + } spin_unlock(&dd->lock); return rq; @@ -439,9 +543,14 @@ static int dd_init_hctx(struct blk_mq_hw_ctx *hctx, unsigned int hctx_idx) static void dd_exit_sched(struct elevator_queue *e) { struct deadline_data *dd = e->elevator_data; + enum dd_prio prio; + + for (prio = 0; prio <= DD_PRIO_MAX; prio++) { + WARN_ON_ONCE(!list_empty(&dd->fifo_list[prio][DD_READ])); + WARN_ON_ONCE(!list_empty(&dd->fifo_list[prio][DD_WRITE])); + } - BUG_ON(!list_empty(&dd->fifo_list[DD_READ])); - BUG_ON(!list_empty(&dd->fifo_list[DD_WRITE])); + free_percpu(dd->stats); kfree(dd); } @@ -453,20 +562,29 @@ static int dd_init_sched(struct request_queue *q, struct elevator_type *e) { struct deadline_data *dd; struct elevator_queue *eq; + enum dd_prio prio; + int ret = -ENOMEM; eq = elevator_alloc(q, e); if (!eq) - return -ENOMEM; + return ret; dd = kzalloc_node(sizeof(*dd), GFP_KERNEL, q->node); - if (!dd) { - kobject_put(&eq->kobj); - return -ENOMEM; - } + if (!dd) + goto put_eq; + eq->elevator_data = dd; - INIT_LIST_HEAD(&dd->fifo_list[DD_READ]); - INIT_LIST_HEAD(&dd->fifo_list[DD_WRITE]); + dd->stats = alloc_percpu_gfp(typeof(*dd->stats), + GFP_KERNEL | __GFP_ZERO); + if (!dd->stats) + goto free_dd; + + for (prio = 0; prio <= DD_PRIO_MAX; prio++) { + INIT_LIST_HEAD(&dd->fifo_list[prio][DD_READ]); + INIT_LIST_HEAD(&dd->fifo_list[prio][DD_WRITE]); + INIT_LIST_HEAD(&dd->dispatch[prio]); + } dd->sort_list[DD_READ] = RB_ROOT; dd->sort_list[DD_WRITE] = RB_ROOT; dd->fifo_expire[DD_READ] = read_expire; @@ -476,10 +594,16 @@ static int dd_init_sched(struct request_queue *q, struct elevator_type *e) dd->fifo_batch = fifo_batch; spin_lock_init(&dd->lock); spin_lock_init(&dd->zone_lock); - INIT_LIST_HEAD(&dd->dispatch); q->elevator = eq; return 0; + +free_dd: + kfree(dd); + +put_eq: + kobject_put(&eq->kobj); + return ret; } /* @@ -539,6 +663,9 @@ static void dd_insert_request(struct blk_mq_hw_ctx *hctx, struct request *rq, struct request_queue *q = hctx->queue; struct deadline_data *dd = q->elevator->elevator_data; const enum dd_data_dir data_dir = rq_data_dir(rq); + u16 ioprio = req_get_ioprio(rq); + u8 ioprio_class = IOPRIO_PRIO_CLASS(ioprio); + enum dd_prio prio; lockdep_assert_held(&dd->lock); @@ -548,13 +675,18 @@ static void dd_insert_request(struct blk_mq_hw_ctx *hctx, struct request *rq, */ blk_req_zone_write_unlock(rq); + prio = ioprio_class_to_prio[ioprio_class]; + dd_count(dd, inserted, prio); + WARN_ON_ONCE(rq->elv.priv[0]); + rq->elv.priv[0] = (void *)1ULL; + if (blk_mq_sched_try_insert_merge(q, rq)) return; trace_block_rq_insert(rq); if (at_head) { - list_add(&rq->queuelist, &dd->dispatch); + list_add(&rq->queuelist, &dd->dispatch[prio]); } else { deadline_add_rq_rb(dd, rq); @@ -568,7 +700,7 @@ static void dd_insert_request(struct blk_mq_hw_ctx *hctx, struct request *rq, * set expire time and add to fifo list */ rq->fifo_time = jiffies + dd->fifo_expire[data_dir]; - list_add_tail(&rq->queuelist, &dd->fifo_list[data_dir]); + list_add_tail(&rq->queuelist, &dd->fifo_list[prio][data_dir]); } } @@ -592,12 +724,10 @@ static void dd_insert_requests(struct blk_mq_hw_ctx *hctx, spin_unlock(&dd->lock); } -/* - * Nothing to do here. This is defined only to ensure that .finish_request - * method is called upon request completion. - */ +/* Callback from inside blk_mq_rq_ctx_init(). */ static void dd_prepare_request(struct request *rq) { + rq->elv.priv[0] = NULL; } /* @@ -619,26 +749,41 @@ static void dd_prepare_request(struct request *rq) static void dd_finish_request(struct request *rq) { struct request_queue *q = rq->q; + struct deadline_data *dd = q->elevator->elevator_data; + const u8 ioprio_class = dd_rq_ioclass(rq); + const enum dd_prio prio = ioprio_class_to_prio[ioprio_class]; + + if (rq->elv.priv[0]) + dd_count(dd, completed, prio); if (blk_queue_is_zoned(q)) { - struct deadline_data *dd = q->elevator->elevator_data; unsigned long flags; spin_lock_irqsave(&dd->zone_lock, flags); blk_req_zone_write_unlock(rq); - if (!list_empty(&dd->fifo_list[DD_WRITE])) + if (!list_empty(&dd->fifo_list[prio][DD_WRITE])) blk_mq_sched_mark_restart_hctx(rq->mq_hctx); spin_unlock_irqrestore(&dd->zone_lock, flags); } } +static bool dd_has_work_for_prio(struct deadline_data *dd, enum dd_prio prio) +{ + return !list_empty_careful(&dd->dispatch[prio]) || + !list_empty_careful(&dd->fifo_list[prio][DD_READ]) || + !list_empty_careful(&dd->fifo_list[prio][DD_WRITE]); +} + static bool dd_has_work(struct blk_mq_hw_ctx *hctx) { struct deadline_data *dd = hctx->queue->elevator->elevator_data; + enum dd_prio prio; + + for (prio = 0; prio <= DD_PRIO_MAX; prio++) + if (dd_has_work_for_prio(dd, prio)) + return true; - return !list_empty_careful(&dd->dispatch) || - !list_empty_careful(&dd->fifo_list[0]) || - !list_empty_careful(&dd->fifo_list[1]); + return false; } /* @@ -702,7 +847,7 @@ static struct elv_fs_entry deadline_attrs[] = { }; #ifdef CONFIG_BLK_DEBUG_FS -#define DEADLINE_DEBUGFS_DDIR_ATTRS(ddir, name) \ +#define DEADLINE_DEBUGFS_DDIR_ATTRS(prio, data_dir, name) \ static void *deadline_##name##_fifo_start(struct seq_file *m, \ loff_t *pos) \ __acquires(&dd->lock) \ @@ -711,7 +856,7 @@ static void *deadline_##name##_fifo_start(struct seq_file *m, \ struct deadline_data *dd = q->elevator->elevator_data; \ \ spin_lock(&dd->lock); \ - return seq_list_start(&dd->fifo_list[ddir], *pos); \ + return seq_list_start(&dd->fifo_list[prio][data_dir], *pos); \ } \ \ static void *deadline_##name##_fifo_next(struct seq_file *m, void *v, \ @@ -720,7 +865,7 @@ static void *deadline_##name##_fifo_next(struct seq_file *m, void *v, \ struct request_queue *q = m->private; \ struct deadline_data *dd = q->elevator->elevator_data; \ \ - return seq_list_next(v, &dd->fifo_list[ddir], pos); \ + return seq_list_next(v, &dd->fifo_list[prio][data_dir], pos); \ } \ \ static void deadline_##name##_fifo_stop(struct seq_file *m, void *v) \ @@ -737,22 +882,31 @@ static const struct seq_operations deadline_##name##_fifo_seq_ops = { \ .next = deadline_##name##_fifo_next, \ .stop = deadline_##name##_fifo_stop, \ .show = blk_mq_debugfs_rq_show, \ -}; \ - \ +}; + +#define DEADLINE_DEBUGFS_NEXT_RQ(data_dir, name) \ static int deadline_##name##_next_rq_show(void *data, \ struct seq_file *m) \ { \ struct request_queue *q = data; \ struct deadline_data *dd = q->elevator->elevator_data; \ - struct request *rq = dd->next_rq[ddir]; \ + struct request *rq = dd->next_rq[data_dir]; \ \ if (rq) \ __blk_mq_debugfs_rq_show(m, rq); \ return 0; \ } -DEADLINE_DEBUGFS_DDIR_ATTRS(DD_READ, read) -DEADLINE_DEBUGFS_DDIR_ATTRS(DD_WRITE, write) + +DEADLINE_DEBUGFS_DDIR_ATTRS(DD_RT_PRIO, DD_READ, read0) +DEADLINE_DEBUGFS_DDIR_ATTRS(DD_RT_PRIO, DD_WRITE, write0) +DEADLINE_DEBUGFS_DDIR_ATTRS(DD_BE_PRIO, DD_READ, read1) +DEADLINE_DEBUGFS_DDIR_ATTRS(DD_BE_PRIO, DD_WRITE, write1) +DEADLINE_DEBUGFS_DDIR_ATTRS(DD_IDLE_PRIO, DD_READ, read2) +DEADLINE_DEBUGFS_DDIR_ATTRS(DD_IDLE_PRIO, DD_WRITE, write2) +DEADLINE_DEBUGFS_NEXT_RQ(DD_READ, read) +DEADLINE_DEBUGFS_NEXT_RQ(DD_WRITE, write) #undef DEADLINE_DEBUGFS_DDIR_ATTRS +#undef DEADLINE_DEBUGFS_NEXT_RQ static int deadline_batching_show(void *data, struct seq_file *m) { @@ -781,50 +935,99 @@ static int dd_async_depth_show(void *data, struct seq_file *m) return 0; } -static void *deadline_dispatch_start(struct seq_file *m, loff_t *pos) - __acquires(&dd->lock) +static int dd_queued_show(void *data, struct seq_file *m) { - struct request_queue *q = m->private; + struct request_queue *q = data; struct deadline_data *dd = q->elevator->elevator_data; - spin_lock(&dd->lock); - return seq_list_start(&dd->dispatch, *pos); + seq_printf(m, "%u %u %u\n", dd_queued(dd, DD_RT_PRIO), + dd_queued(dd, DD_BE_PRIO), + dd_queued(dd, DD_IDLE_PRIO)); + return 0; } -static void *deadline_dispatch_next(struct seq_file *m, void *v, loff_t *pos) +/* Number of requests owned by the block driver for a given priority. */ +static u32 dd_owned_by_driver(struct deadline_data *dd, enum dd_prio prio) { - struct request_queue *q = m->private; - struct deadline_data *dd = q->elevator->elevator_data; - - return seq_list_next(v, &dd->dispatch, pos); + return dd_sum(dd, dispatched, prio) + dd_sum(dd, merged, prio) + - dd_sum(dd, completed, prio); } -static void deadline_dispatch_stop(struct seq_file *m, void *v) - __releases(&dd->lock) +static int dd_owned_by_driver_show(void *data, struct seq_file *m) { - struct request_queue *q = m->private; + struct request_queue *q = data; struct deadline_data *dd = q->elevator->elevator_data; - spin_unlock(&dd->lock); + seq_printf(m, "%u %u %u\n", dd_owned_by_driver(dd, DD_RT_PRIO), + dd_owned_by_driver(dd, DD_BE_PRIO), + dd_owned_by_driver(dd, DD_IDLE_PRIO)); + return 0; } -static const struct seq_operations deadline_dispatch_seq_ops = { - .start = deadline_dispatch_start, - .next = deadline_dispatch_next, - .stop = deadline_dispatch_stop, - .show = blk_mq_debugfs_rq_show, -}; +#define DEADLINE_DISPATCH_ATTR(prio) \ +static void *deadline_dispatch##prio##_start(struct seq_file *m, \ + loff_t *pos) \ + __acquires(&dd->lock) \ +{ \ + struct request_queue *q = m->private; \ + struct deadline_data *dd = q->elevator->elevator_data; \ + \ + spin_lock(&dd->lock); \ + return seq_list_start(&dd->dispatch[prio], *pos); \ +} \ + \ +static void *deadline_dispatch##prio##_next(struct seq_file *m, \ + void *v, loff_t *pos) \ +{ \ + struct request_queue *q = m->private; \ + struct deadline_data *dd = q->elevator->elevator_data; \ + \ + return seq_list_next(v, &dd->dispatch[prio], pos); \ +} \ + \ +static void deadline_dispatch##prio##_stop(struct seq_file *m, void *v) \ + __releases(&dd->lock) \ +{ \ + struct request_queue *q = m->private; \ + struct deadline_data *dd = q->elevator->elevator_data; \ + \ + spin_unlock(&dd->lock); \ +} \ + \ +static const struct seq_operations deadline_dispatch##prio##_seq_ops = { \ + .start = deadline_dispatch##prio##_start, \ + .next = deadline_dispatch##prio##_next, \ + .stop = deadline_dispatch##prio##_stop, \ + .show = blk_mq_debugfs_rq_show, \ +} + +DEADLINE_DISPATCH_ATTR(0); +DEADLINE_DISPATCH_ATTR(1); +DEADLINE_DISPATCH_ATTR(2); +#undef DEADLINE_DISPATCH_ATTR -#define DEADLINE_QUEUE_DDIR_ATTRS(name) \ - {#name "_fifo_list", 0400, .seq_ops = &deadline_##name##_fifo_seq_ops}, \ +#define DEADLINE_QUEUE_DDIR_ATTRS(name) \ + {#name "_fifo_list", 0400, \ + .seq_ops = &deadline_##name##_fifo_seq_ops} +#define DEADLINE_NEXT_RQ_ATTR(name) \ {#name "_next_rq", 0400, deadline_##name##_next_rq_show} static const struct blk_mq_debugfs_attr deadline_queue_debugfs_attrs[] = { - DEADLINE_QUEUE_DDIR_ATTRS(read), - DEADLINE_QUEUE_DDIR_ATTRS(write), + DEADLINE_QUEUE_DDIR_ATTRS(read0), + DEADLINE_QUEUE_DDIR_ATTRS(write0), + DEADLINE_QUEUE_DDIR_ATTRS(read1), + DEADLINE_QUEUE_DDIR_ATTRS(write1), + DEADLINE_QUEUE_DDIR_ATTRS(read2), + DEADLINE_QUEUE_DDIR_ATTRS(write2), + DEADLINE_NEXT_RQ_ATTR(read), + DEADLINE_NEXT_RQ_ATTR(write), {"batching", 0400, deadline_batching_show}, {"starved", 0400, deadline_starved_show}, {"async_depth", 0400, dd_async_depth_show}, - {"dispatch", 0400, .seq_ops = &deadline_dispatch_seq_ops}, + {"dispatch0", 0400, .seq_ops = &deadline_dispatch0_seq_ops}, + {"dispatch1", 0400, .seq_ops = &deadline_dispatch1_seq_ops}, + {"dispatch2", 0400, .seq_ops = &deadline_dispatch2_seq_ops}, + {"owned_by_driver", 0400, dd_owned_by_driver_show}, + {"queued", 0400, dd_queued_show}, {}, }; #undef DEADLINE_QUEUE_DDIR_ATTRS @@ -874,6 +1077,6 @@ static void __exit deadline_exit(void) module_init(deadline_init); module_exit(deadline_exit); -MODULE_AUTHOR("Jens Axboe"); +MODULE_AUTHOR("Jens Axboe, Damien Le Moal and Bart Van Assche"); MODULE_LICENSE("GPL"); MODULE_DESCRIPTION("MQ deadline IO scheduler"); From patchwork Tue Jun 8 23:07:02 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bart Van Assche X-Patchwork-Id: 12308379 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 59AC1C48BD1 for ; Tue, 8 Jun 2021 23:07:42 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 4483261361 for ; Tue, 8 Jun 2021 23:07:42 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235255AbhFHXJe (ORCPT ); Tue, 8 Jun 2021 19:09:34 -0400 Received: from mail-pj1-f43.google.com ([209.85.216.43]:42773 "EHLO mail-pj1-f43.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235066AbhFHXJe (ORCPT ); Tue, 8 Jun 2021 19:09:34 -0400 Received: by mail-pj1-f43.google.com with SMTP id md2-20020a17090b23c2b029016de4440381so247031pjb.1 for ; Tue, 08 Jun 2021 16:07:31 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=gBFsEB9EnxjTh68lBTEfYlcHmXH6z+4Ffq6g5I1YRPI=; b=lTGmBqnrY44cXeInD96POwCHLT4C6+tROGzunrXx3BE0PmM6t5M9/gJYG9EBow6ypC UZs+VwZ4L9E5/cfpeF9aS3tAKJrWnC9BFfWmajjNqXhKqrRleR+ke3wxN9cyDNJkgEvl 3wVQdht+2YeG67EkYfVCgryHM9yvRz6DBVli2Tm2jJxiQUaynMie0yY6dmOMe7irvcqw bTc4LOTlzTXIEwYyhHz+v1nfnusFAQN4SWo3mNSNrhYSyaxqqJAiNxlYlXbQnnCju+HT hric5h1GqZolz1aDDhhnybtZKqFXqfdyN78/FoYxcpd2NV1lRb1kDE7j2B53oh9mcorn pwyg== X-Gm-Message-State: AOAM530UKWuDmy+WTziuGFR5znE2mJ8WNfr59wNycOfO6FsreYKnTpAC KiapAp6ZYrJL0sQmFFb5lZ4= X-Google-Smtp-Source: ABdhPJyz8E/p4ps6cDLIHqcTHDZsg2N+zJ6MDv4zgTQTB7GAEM5EeBoQX2UAET2IShT+T+ymYC1QLQ== X-Received: by 2002:a17:903:2281:b029:113:1edb:97d0 with SMTP id b1-20020a1709032281b02901131edb97d0mr2109313plh.64.1623193650532; Tue, 08 Jun 2021 16:07:30 -0700 (PDT) Received: from asus.hsd1.ca.comcast.net (c-73-241-217-19.hsd1.ca.comcast.net. [73.241.217.19]) by smtp.gmail.com with ESMTPSA id s21sm11395838pfw.57.2021.06.08.16.07.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 08 Jun 2021 16:07:29 -0700 (PDT) From: Bart Van Assche To: Jens Axboe Cc: linux-block@vger.kernel.org, Christoph Hellwig , Jaegeuk Kim , Bart Van Assche , Damien Le Moal , Hannes Reinecke , Ming Lei , Johannes Thumshirn , Himanshu Madhani Subject: [PATCH 13/14] block/mq-deadline: Add cgroup support Date: Tue, 8 Jun 2021 16:07:02 -0700 Message-Id: <20210608230703.19510-14-bvanassche@acm.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210608230703.19510-1-bvanassche@acm.org> References: <20210608230703.19510-1-bvanassche@acm.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org Maintain statistics per cgroup and export these to user space. These statistics are essential for verifying whether the proper I/O priorities have been assigned to requests. With this patch applied the output of (cd /sys/fs/cgroup/blkio/ && ls) is as follows: blkio.dd.dispatched_pri0 blkio.dd.inserted_pri3 cgroup.clone_children blkio.dd.dispatched_pri1 blkio.dd.merged_pri0 cgroup.procs blkio.dd.dispatched_pri2 blkio.dd.merged_pri1 cgroup.sane_behavior blkio.dd.dispatched_pri3 blkio.dd.merged_pri2 notify_on_release blkio.dd.inserted_pri0 blkio.dd.merged_pri3 release_agent blkio.dd.inserted_pri1 blkio.prio.class tasks blkio.dd.inserted_pri2 blkio.reset_stats Cc: Damien Le Moal Cc: Hannes Reinecke Cc: Christoph Hellwig Cc: Ming Lei Cc: Johannes Thumshirn Cc: Himanshu Madhani Signed-off-by: Bart Van Assche --- block/Kconfig.iosched | 6 + block/Makefile | 2 + block/mq-deadline-cgroup.c | 185 ++++++++++++++++++++ block/mq-deadline-cgroup.h | 112 ++++++++++++ block/{mq-deadline.c => mq-deadline-main.c} | 85 +++++++-- 5 files changed, 376 insertions(+), 14 deletions(-) create mode 100644 block/mq-deadline-cgroup.c create mode 100644 block/mq-deadline-cgroup.h rename block/{mq-deadline.c => mq-deadline-main.c} (94%) diff --git a/block/Kconfig.iosched b/block/Kconfig.iosched index 2f2158e05a91..64053d67a97b 100644 --- a/block/Kconfig.iosched +++ b/block/Kconfig.iosched @@ -9,6 +9,12 @@ config MQ_IOSCHED_DEADLINE help MQ version of the deadline IO scheduler. +config MQ_IOSCHED_DEADLINE_CGROUP + tristate + default y + depends on MQ_IOSCHED_DEADLINE + depends on BLK_CGROUP + config MQ_IOSCHED_KYBER tristate "Kyber I/O scheduler" default y diff --git a/block/Makefile b/block/Makefile index af3d044abaf1..b9db5d4edfc8 100644 --- a/block/Makefile +++ b/block/Makefile @@ -21,6 +21,8 @@ obj-$(CONFIG_BLK_CGROUP_IOPRIO) += blk-ioprio.o obj-$(CONFIG_BLK_CGROUP_IOLATENCY) += blk-iolatency.o obj-$(CONFIG_BLK_CGROUP_IOCOST) += blk-iocost.o obj-$(CONFIG_MQ_IOSCHED_DEADLINE) += mq-deadline.o +mq-deadline-y += mq-deadline-main.o +mq-deadline-$(CONFIG_MQ_IOSCHED_DEADLINE_CGROUP)+= mq-deadline-cgroup.o obj-$(CONFIG_MQ_IOSCHED_KYBER) += kyber-iosched.o bfq-y := bfq-iosched.o bfq-wf2q.o bfq-cgroup.o obj-$(CONFIG_IOSCHED_BFQ) += bfq.o diff --git a/block/mq-deadline-cgroup.c b/block/mq-deadline-cgroup.c new file mode 100644 index 000000000000..73b04e5a577f --- /dev/null +++ b/block/mq-deadline-cgroup.c @@ -0,0 +1,185 @@ +// SPDX-License-Identifier: GPL-2.0 + +#include +#include + +#include "mq-deadline-cgroup.h" + +static struct blkcg_policy dd_blkcg_policy; + +static struct dd_blkcg *dd_blkcg_from_css(struct cgroup_subsys_state *css) +{ + struct blkcg *blkcg = css_to_blkcg(css); + struct blkcg_policy_data *cpd = blkcg_to_cpd(blkcg, &dd_blkcg_policy); + + return container_of(cpd, struct dd_blkcg, cpd); +} + +/* + * Number of inserted but not completed requests for priority class + * cft->private. + */ +static u64 dd_show_inserted(struct cgroup_subsys_state *css, + struct cftype *cft) +{ + struct dd_blkcg *blkcg = dd_blkcg_from_css(css); + const u8 prio = cft->private; + + return ddcg_sum(blkcg, inserted, prio) - + ddcg_sum(blkcg, completed, prio); +} + +/* Number of merged requests for priority class cft->private. */ +static u64 dd_show_merged(struct cgroup_subsys_state *css, struct cftype *cft) +{ + struct dd_blkcg *blkcg = dd_blkcg_from_css(css); + const u8 prio = cft->private; + + return ddcg_sum(blkcg, merged, prio); +} + +/* + * Number of dispatched but not completed requests for priority class + * cft->private. + */ +static u64 dd_show_dispatched(struct cgroup_subsys_state *css, + struct cftype *cft) +{ + struct dd_blkcg *blkcg = dd_blkcg_from_css(css); + const u8 prio = cft->private; + + return ddcg_sum(blkcg, dispatched, prio) + + ddcg_sum(blkcg, merged, prio) - + ddcg_sum(blkcg, completed, prio); +} + +#define IOPRI_ATTRS(pfx, cft_flags, callback) \ + { \ + .name = pfx "0", \ + .private = IOPRIO_CLASS_NONE, \ + .flags = cft_flags, \ + .read_u64 = callback, \ + }, \ + { \ + .name = pfx "1", \ + .private = IOPRIO_CLASS_RT, \ + .flags = cft_flags, \ + .read_u64 = callback, \ + }, \ + { \ + .name = pfx "2", \ + .private = IOPRIO_CLASS_BE, \ + .flags = cft_flags, \ + .read_u64 = callback, \ + }, \ + { \ + .name = pfx "3", \ + .private = IOPRIO_CLASS_IDLE, \ + .flags = cft_flags, \ + .read_u64 = callback, \ + } + +#define DD_ATTRS \ + IOPRI_ATTRS("dd.inserted_pri", 0, dd_show_inserted), \ + IOPRI_ATTRS("dd.merged_pri", 0, dd_show_merged), \ + IOPRI_ATTRS("dd.dispatched_pri", 0, dd_show_dispatched), \ + { } /* sentinel */ + +/* cgroup v2 attributes */ +static struct cftype dd_blkcg_files[] = { + DD_ATTRS +}; + +/* cgroup v1 attributes */ +static struct cftype dd_blkcg_legacy_files[] = { + DD_ATTRS +}; + +static struct blkcg_policy_data *dd_cpd_alloc(gfp_t gfp) +{ + struct dd_blkcg *pd; + + pd = kzalloc(sizeof(*pd), gfp); + if (!pd) + return NULL; + pd->stats = alloc_percpu_gfp(typeof(*pd->stats), + GFP_KERNEL | __GFP_ZERO); + if (!pd->stats) { + kfree(pd); + return NULL; + } + return &pd->cpd; +} + +static void dd_cpd_free(struct blkcg_policy_data *cpd) +{ + struct dd_blkcg *dd_blkcg = container_of(cpd, typeof(*dd_blkcg), cpd); + + free_percpu(dd_blkcg->stats); + kfree(dd_blkcg); +} + +/* + * Convert an association between a block cgroup and a request queue into a + * pointer to the mq-deadline information associated with a (blkcg, queue) pair. + */ +struct dd_blkcg *dd_blkcg_from_bio(struct bio *bio) +{ + struct blkg_policy_data *pd; + + pd = blkg_to_pd(bio->bi_blkg, &dd_blkcg_policy); + if (!pd) + return NULL; + + return container_of(blkcg_to_cpd(pd->blkg->blkcg, &dd_blkcg_policy), + struct dd_blkcg, cpd); +} + +static struct blkg_policy_data *dd_pd_alloc(gfp_t gfp, struct request_queue *q, + struct blkcg *blkcg) +{ + struct dd_blkg *pd; + + pd = kzalloc(sizeof(*pd), gfp); + if (!pd) + return NULL; + return &pd->pd; +} + +static void dd_pd_free(struct blkg_policy_data *pd) +{ + struct dd_blkg *dd_blkg = container_of(pd, typeof(*dd_blkg), pd); + + kfree(dd_blkg); +} + +static struct blkcg_policy dd_blkcg_policy = { + .dfl_cftypes = dd_blkcg_files, + .legacy_cftypes = dd_blkcg_legacy_files, + + .cpd_alloc_fn = dd_cpd_alloc, + .cpd_free_fn = dd_cpd_free, + + .pd_alloc_fn = dd_pd_alloc, + .pd_free_fn = dd_pd_free, +}; + +int dd_activate_policy(struct request_queue *q) +{ + return blkcg_activate_policy(q, &dd_blkcg_policy); +} + +void dd_deactivate_policy(struct request_queue *q) +{ + blkcg_deactivate_policy(q, &dd_blkcg_policy); +} + +int __init dd_blkcg_init(void) +{ + return blkcg_policy_register(&dd_blkcg_policy); +} + +void __exit dd_blkcg_exit(void) +{ + blkcg_policy_unregister(&dd_blkcg_policy); +} diff --git a/block/mq-deadline-cgroup.h b/block/mq-deadline-cgroup.h new file mode 100644 index 000000000000..1e263557c881 --- /dev/null +++ b/block/mq-deadline-cgroup.h @@ -0,0 +1,112 @@ +/* SPDX-License-Identifier: GPL-2.0 */ + +#if !defined(_MQ_DEADLINE_CGROUP_H_) +#define _MQ_DEADLINE_CGROUP_H_ + +#include + +struct request_queue; + +/** + * struct io_stats_per_prio - I/O statistics per I/O priority class. + * @inserted: Number of inserted requests. + * @merged: Number of merged requests. + * @dispatched: Number of dispatched requests. + * @completed: Number of I/O completions. + */ +struct io_stats_per_prio { + local_t inserted; + local_t merged; + local_t dispatched; + local_t completed; +}; + +/* I/O statistics per I/O cgroup per I/O priority class (IOPRIO_CLASS_*). */ +struct blkcg_io_stats { + struct io_stats_per_prio stats[4]; +}; + +/** + * struct dd_blkcg - Per cgroup data. + * @cpd: blkcg_policy_data structure. + * @stats: I/O statistics. + */ +struct dd_blkcg { + struct blkcg_policy_data cpd; /* must be the first member */ + struct blkcg_io_stats __percpu *stats; +}; + +/* + * Count one event of type 'event_type' and with I/O priority class + * 'prio_class'. + */ +#define ddcg_count(ddcg, event_type, prio_class) do { \ + struct blkcg_io_stats *io_stats = get_cpu_ptr((ddcg)->stats); \ + \ + BUILD_BUG_ON(!__same_type((ddcg), struct dd_blkcg *)); \ + BUILD_BUG_ON(!__same_type((prio_class), u8)); \ + local_inc(&io_stats->stats[(prio_class)].event_type); \ + put_cpu_ptr(io_stats); \ +} while (0) + +/* + * Returns the total number of ddcg_count(ddcg, event_type, prio_class) calls + * across all CPUs. No locking or barriers since it is fine if the returned + * sum is slightly outdated. + */ +#define ddcg_sum(ddcg, event_type, prio) ({ \ + unsigned int cpu; \ + u32 sum = 0; \ + \ + BUILD_BUG_ON(!__same_type((ddcg), struct dd_blkcg *)); \ + BUILD_BUG_ON(!__same_type((prio), u8)); \ + for_each_present_cpu(cpu) \ + sum += local_read(&per_cpu_ptr((ddcg)->stats, cpu)-> \ + stats[(prio)].event_type); \ + sum; \ +}) + +#ifdef CONFIG_BLK_CGROUP + +/** + * struct dd_blkg - Per (cgroup, request queue) data. + * @pd: blkg_policy_data structure. + */ +struct dd_blkg { + struct blkg_policy_data pd; /* must be the first member */ +}; + +struct dd_blkcg *dd_blkcg_from_bio(struct bio *bio); +int dd_activate_policy(struct request_queue *q); +void dd_deactivate_policy(struct request_queue *q); +int __init dd_blkcg_init(void); +void __exit dd_blkcg_exit(void); + +#else /* CONFIG_BLK_CGROUP */ + +static inline struct dd_blkcg *dd_blkcg_from_bio(struct bio *bio) +{ + return NULL; +} + +static inline int dd_activate_policy(struct request_queue *q) +{ + return 0; +} + +static inline void dd_deactivate_policy(struct request_queue *q) +{ +} + +static inline int dd_blkcg_init(void) +{ + return 0; +} + +static inline void dd_blkcg_exit(void) +{ +} + +#endif /* CONFIG_BLK_CGROUP */ + +#endif /* _MQ_DEADLINE_CGROUP_H_ */ diff --git a/block/mq-deadline.c b/block/mq-deadline-main.c similarity index 94% rename from block/mq-deadline.c rename to block/mq-deadline-main.c index 776ff49713c3..1b2b6c1de5b8 100644 --- a/block/mq-deadline.c +++ b/block/mq-deadline-main.c @@ -25,6 +25,7 @@ #include "blk-mq-debugfs.h" #include "blk-mq-tag.h" #include "blk-mq-sched.h" +#include "mq-deadline-cgroup.h" /* * See Documentation/block/deadline-iosched.rst @@ -51,14 +52,6 @@ enum dd_prio { enum { DD_PRIO_COUNT = 3 }; -/* I/O statistics per I/O priority. */ -struct io_stats_per_prio { - local_t inserted; - local_t merged; - local_t dispatched; - local_t completed; -}; - /* I/O statistics for all I/O priorities (enum dd_prio). */ struct io_stats { struct io_stats_per_prio stats[DD_PRIO_COUNT]; @@ -69,6 +62,9 @@ struct deadline_data { * run time data */ + /* Request queue that owns this data structure. */ + struct request_queue *queue; + /* * Requests are present on both sort_list[] and fifo_list[][]. The * first index of fifo_list[][] is the I/O priority class (DD_*_PRIO). @@ -226,11 +222,15 @@ static void dd_merged_requests(struct request_queue *q, struct request *req, struct deadline_data *dd = q->elevator->elevator_data; const u8 ioprio_class = dd_rq_ioclass(next); const enum dd_prio prio = ioprio_class_to_prio[ioprio_class]; + struct dd_blkcg *blkcg = next->elv.priv[0]; - if (next->elv.priv[0]) { + if (blkcg) { dd_count(dd, merged, prio); + ddcg_count(blkcg, merged, ioprio_class); } else { +#if defined(CONFIG_BLK_CGROUP) WARN_ON_ONCE(true); +#endif } /* @@ -370,6 +370,7 @@ static struct request *__dd_dispatch_request(struct deadline_data *dd, { struct request *rq, *next_rq; enum dd_data_dir data_dir; + struct dd_blkcg *blkcg; u8 ioprio_class; lockdep_assert_held(&dd->lock); @@ -462,12 +463,16 @@ static struct request *__dd_dispatch_request(struct deadline_data *dd, dd->batching++; deadline_move_request(dd, rq); done: + blkcg = rq->elv.priv[0]; ioprio_class = dd_rq_ioclass(rq); prio = ioprio_class_to_prio[ioprio_class]; - if (rq->elv.priv[0]) { + if (blkcg) { dd_count(dd, dispatched, prio); + ddcg_count(blkcg, dispatched, ioprio_class); } else { +#if defined(CONFIG_BLK_CGROUP) WARN_ON_ONCE(true); +#endif } /* * If the request needs its target zone locked, do it. @@ -545,6 +550,8 @@ static void dd_exit_sched(struct elevator_queue *e) struct deadline_data *dd = e->elevator_data; enum dd_prio prio; + dd_deactivate_policy(dd->queue); + for (prio = 0; prio <= DD_PRIO_MAX; prio++) { WARN_ON_ONCE(!list_empty(&dd->fifo_list[prio][DD_READ])); WARN_ON_ONCE(!list_empty(&dd->fifo_list[prio][DD_WRITE])); @@ -556,7 +563,7 @@ static void dd_exit_sched(struct elevator_queue *e) } /* - * initialize elevator private data (deadline_data). + * Initialize elevator private data (deadline_data) and associate with blkcg. */ static int dd_init_sched(struct request_queue *q, struct elevator_type *e) { @@ -565,6 +572,12 @@ static int dd_init_sched(struct request_queue *q, struct elevator_type *e) enum dd_prio prio; int ret = -ENOMEM; + /* + * Initialization would be very tricky if the queue is not frozen, + * hence the warning statement below. + */ + WARN_ON_ONCE(!percpu_ref_is_zero(&q->q_usage_counter)); + eq = elevator_alloc(q, e); if (!eq) return ret; @@ -580,6 +593,8 @@ static int dd_init_sched(struct request_queue *q, struct elevator_type *e) if (!dd->stats) goto free_dd; + dd->queue = q; + for (prio = 0; prio <= DD_PRIO_MAX; prio++) { INIT_LIST_HEAD(&dd->fifo_list[prio][DD_READ]); INIT_LIST_HEAD(&dd->fifo_list[prio][DD_WRITE]); @@ -595,9 +610,17 @@ static int dd_init_sched(struct request_queue *q, struct elevator_type *e) spin_lock_init(&dd->lock); spin_lock_init(&dd->zone_lock); + ret = dd_activate_policy(q); + if (ret) + goto free_stats; + + ret = 0; q->elevator = eq; return 0; +free_stats: + free_percpu(dd->stats); + free_dd: kfree(dd); @@ -666,6 +689,7 @@ static void dd_insert_request(struct blk_mq_hw_ctx *hctx, struct request *rq, u16 ioprio = req_get_ioprio(rq); u8 ioprio_class = IOPRIO_PRIO_CLASS(ioprio); enum dd_prio prio; + struct dd_blkcg *blkcg; lockdep_assert_held(&dd->lock); @@ -675,10 +699,25 @@ static void dd_insert_request(struct blk_mq_hw_ctx *hctx, struct request *rq, */ blk_req_zone_write_unlock(rq); + /* + * If a block cgroup has been associated with the submitter and if an + * I/O priority has been set in the associated block cgroup, use the + * lowest of the cgroup priority and the request priority for the + * request. If no priority has been set in the request, use the cgroup + * priority. + */ + blkcg = dd_blkcg_from_bio(rq->bio); + if (blkcg) { + ddcg_count(blkcg, inserted, ioprio_class); + } else { +#if defined(CONFIG_BLK_CGROUP) + WARN_ON_ONCE(true); +#endif + } prio = ioprio_class_to_prio[ioprio_class]; dd_count(dd, inserted, prio); WARN_ON_ONCE(rq->elv.priv[0]); - rq->elv.priv[0] = (void *)1ULL; + rq->elv.priv[0] = blkcg; if (blk_mq_sched_try_insert_merge(q, rq)) return; @@ -750,11 +789,14 @@ static void dd_finish_request(struct request *rq) { struct request_queue *q = rq->q; struct deadline_data *dd = q->elevator->elevator_data; + struct dd_blkcg *blkcg = rq->elv.priv[0]; const u8 ioprio_class = dd_rq_ioclass(rq); const enum dd_prio prio = ioprio_class_to_prio[ioprio_class]; - if (rq->elv.priv[0]) + if (blkcg) { dd_count(dd, completed, prio); + ddcg_count(blkcg, completed, ioprio_class); + } if (blk_queue_is_zoned(q)) { unsigned long flags; @@ -1066,11 +1108,26 @@ MODULE_ALIAS("mq-deadline-iosched"); static int __init deadline_init(void) { - return elv_register(&mq_deadline); + int ret; + + ret = elv_register(&mq_deadline); + if (ret) + goto out; + ret = dd_blkcg_init(); + if (ret) + goto unreg; + +out: + return ret; + +unreg: + elv_unregister(&mq_deadline); + goto out; } static void __exit deadline_exit(void) { + dd_blkcg_exit(); elv_unregister(&mq_deadline); } From patchwork Tue Jun 8 23:07:03 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bart Van Assche X-Patchwork-Id: 12308377 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2AAF3C4743F for ; Tue, 8 Jun 2021 23:07:41 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 1767261361 for ; Tue, 8 Jun 2021 23:07:41 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235116AbhFHXJd (ORCPT ); Tue, 8 Jun 2021 19:09:33 -0400 Received: from mail-pj1-f42.google.com ([209.85.216.42]:43964 "EHLO mail-pj1-f42.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235066AbhFHXJc (ORCPT ); Tue, 8 Jun 2021 19:09:32 -0400 Received: by mail-pj1-f42.google.com with SMTP id x21-20020a17090aa395b029016e25313bfcso244454pjp.2 for ; Tue, 08 Jun 2021 16:07:32 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=U+8O8o+tE8LH0yHV6I53G6dFgrWnWaOANTEONxgpi80=; b=fdGCtm+6Eel0k9q+J1LZtCFKzQyyeX25uWD/5sxfiJLpTzm61UgpH1q7FQlVPDW+5R 8oFYbhBiuq6YgsGgLl4WbFg8RgyoCQ85en7OTma/zU67SHcfFv/RqWu1+kR2bOSpJTNy Y7xzzeR4W0Qx5IOi1CQ6drq62oh/UwaSNe4ML4RgcqdqgzBF8OZw50oaThCPRMOV3X6I hzKOYbLw8T63St/csB/OtDYTo8vqs0elTUJEHC0suziH3WoM8NFftVL/bErAcaCzN7A6 HGPPgTYnSo5jYbXhe+eYjEkN1g6Flg1+h/OX5ilCAg7m0aZaSUQ71Eb6uuMg+YduS+o2 hzyQ== X-Gm-Message-State: AOAM530pRuLb52VlLy6P24DnRDiErKMseQwgsmhl9x4kWt0R+DG0mar5 KF0lKuUC5IUwbnNuinouUno= X-Google-Smtp-Source: ABdhPJx6HDS2zzoW+VbZsj3aaZSYdlZHwpuEUAAneKvpqxb06vGS5Jdv1vrfvDt5Mrl5atv6uLvcRw== X-Received: by 2002:a17:90a:1141:: with SMTP id d1mr28874970pje.56.1623193652341; Tue, 08 Jun 2021 16:07:32 -0700 (PDT) Received: from asus.hsd1.ca.comcast.net (c-73-241-217-19.hsd1.ca.comcast.net. [73.241.217.19]) by smtp.gmail.com with ESMTPSA id s21sm11395838pfw.57.2021.06.08.16.07.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 08 Jun 2021 16:07:31 -0700 (PDT) From: Bart Van Assche To: Jens Axboe Cc: linux-block@vger.kernel.org, Christoph Hellwig , Jaegeuk Kim , Bart Van Assche , Damien Le Moal , Hannes Reinecke , Ming Lei , Johannes Thumshirn , Himanshu Madhani Subject: [PATCH 14/14] block/mq-deadline: Prioritize high-priority requests Date: Tue, 8 Jun 2021 16:07:03 -0700 Message-Id: <20210608230703.19510-15-bvanassche@acm.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210608230703.19510-1-bvanassche@acm.org> References: <20210608230703.19510-1-bvanassche@acm.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org While one or more requests with a certain I/O priority are pending, do not dispatch lower priority requests. Dispatch lower priority requests anyway after the "aging" time has expired. This patch has been tested as follows: modprobe scsi_debug ndelay=1000000 max_queue=16 && sd='' && while [ -z "$sd" ]; do sd=/dev/$(basename /sys/bus/pseudo/drivers/scsi_debug/adapter*/host*/target*/*/block/*) done && echo $((100*1000)) > /sys/block/$sd/queue/iosched/aging_expire && cd /sys/fs/cgroup/blkio/ && echo $$ >cgroup.procs && echo 2 >blkio.prio.class && mkdir -p hipri && cd hipri && echo 1 >blkio.prio.class && { max-iops -a1 -d32 -j1 -e mq-deadline $sd >& ~/low-pri.txt & } && echo $$ >cgroup.procs && max-iops -a1 -d32 -j1 -e mq-deadline $sd >& ~/hi-pri.txt Result: * 11000 IOPS for the high-priority job * 400 IOPS for the low-priority job If the aging expiry time is changed from 100s into 0, the IOPS results change into 6712 and 6796 IOPS. The max-iops script is a script that runs fio with the following arguments: --bs=4K --gtod_reduce=1 --ioengine=libaio --ioscheduler=${arg_e} --runtime=60 --norandommap --rw=read --thread --buffered=0 --numjobs=${arg_j} --iodepth=${arg_d} --iodepth_batch_submit=${arg_a} --iodepth_batch_complete=$((arg_d / 2)) --name=${positional_argument_1} --filename=${positional_argument_1} Cc: Damien Le Moal Cc: Hannes Reinecke Cc: Christoph Hellwig Cc: Ming Lei Cc: Johannes Thumshirn Cc: Himanshu Madhani Signed-off-by: Bart Van Assche Reviewed-by: Damien Le Moal Reviewed-by: Hannes Reinecke --- block/mq-deadline-main.c | 42 +++++++++++++++++++++++++++++++++++----- 1 file changed, 37 insertions(+), 5 deletions(-) diff --git a/block/mq-deadline-main.c b/block/mq-deadline-main.c index 1b2b6c1de5b8..cecdb475a610 100644 --- a/block/mq-deadline-main.c +++ b/block/mq-deadline-main.c @@ -32,6 +32,11 @@ */ static const int read_expire = HZ / 2; /* max time before a read is submitted. */ static const int write_expire = 5 * HZ; /* ditto for writes, these limits are SOFT! */ +/* + * Time after which to dispatch lower priority requests even if higher + * priority requests are pending. + */ +static const int aging_expire = 10 * HZ; static const int writes_starved = 2; /* max times reads can starve a write */ static const int fifo_batch = 16; /* # of sequential requests treated as one by the above parameters. For throughput. */ @@ -90,6 +95,7 @@ struct deadline_data { int writes_starved; int front_merges; u32 async_depth; + int aging_expire; spinlock_t lock; spinlock_t zone_lock; @@ -363,10 +369,10 @@ deadline_next_request(struct deadline_data *dd, enum dd_prio prio, /* * deadline_dispatch_requests selects the best request according to - * read/write expire, fifo_batch, etc + * read/write expire, fifo_batch, etc and with a start time <= @latest. */ static struct request *__dd_dispatch_request(struct deadline_data *dd, - enum dd_prio prio) + enum dd_prio prio, u64 latest_start_ns) { struct request *rq, *next_rq; enum dd_data_dir data_dir; @@ -378,6 +384,8 @@ static struct request *__dd_dispatch_request(struct deadline_data *dd, if (!list_empty(&dd->dispatch[prio])) { rq = list_first_entry(&dd->dispatch[prio], struct request, queuelist); + if (rq->start_time_ns > latest_start_ns) + return NULL; list_del_init(&rq->queuelist); goto done; } @@ -457,6 +465,8 @@ static struct request *__dd_dispatch_request(struct deadline_data *dd, dd->batching = 0; dispatch_request: + if (rq->start_time_ns > latest_start_ns) + return NULL; /* * rq is the selected appropriate request. */ @@ -493,15 +503,32 @@ static struct request *__dd_dispatch_request(struct deadline_data *dd, static struct request *dd_dispatch_request(struct blk_mq_hw_ctx *hctx) { struct deadline_data *dd = hctx->queue->elevator->elevator_data; - struct request *rq; + const u64 now_ns = ktime_get_ns(); + struct request *rq = NULL; enum dd_prio prio; spin_lock(&dd->lock); - for (prio = 0; prio <= DD_PRIO_MAX; prio++) { - rq = __dd_dispatch_request(dd, prio); + /* + * Start with dispatching requests whose deadline expired more than + * aging_expire jiffies ago. + */ + for (prio = DD_BE_PRIO; prio <= DD_PRIO_MAX; prio++) { + rq = __dd_dispatch_request(dd, prio, now_ns - + jiffies_to_nsecs(dd->aging_expire)); if (rq) + goto unlock; + } + /* + * Next, dispatch requests in priority order. Ignore lower priority + * requests if any higher priority requests are pending. + */ + for (prio = 0; prio <= DD_PRIO_MAX; prio++) { + rq = __dd_dispatch_request(dd, prio, now_ns); + if (rq || dd_queued(dd, prio)) break; } + +unlock: spin_unlock(&dd->lock); return rq; @@ -607,6 +634,7 @@ static int dd_init_sched(struct request_queue *q, struct elevator_type *e) dd->writes_starved = writes_starved; dd->front_merges = 1; dd->fifo_batch = fifo_batch; + dd->aging_expire = aging_expire; spin_lock_init(&dd->lock); spin_lock_init(&dd->zone_lock); @@ -725,6 +753,7 @@ static void dd_insert_request(struct blk_mq_hw_ctx *hctx, struct request *rq, trace_block_rq_insert(rq); if (at_head) { + rq->fifo_time = jiffies; list_add(&rq->queuelist, &dd->dispatch[prio]); } else { deadline_add_rq_rb(dd, rq); @@ -841,6 +870,7 @@ static ssize_t __FUNC(struct elevator_queue *e, char *page) \ #define SHOW_JIFFIES(__FUNC, __VAR) SHOW_INT(__FUNC, jiffies_to_msecs(__VAR)) SHOW_JIFFIES(deadline_read_expire_show, dd->fifo_expire[DD_READ]); SHOW_JIFFIES(deadline_write_expire_show, dd->fifo_expire[DD_WRITE]); +SHOW_JIFFIES(deadline_aging_expire_show, dd->aging_expire); SHOW_INT(deadline_writes_starved_show, dd->writes_starved); SHOW_INT(deadline_front_merges_show, dd->front_merges); SHOW_INT(deadline_fifo_batch_show, dd->fifo_batch); @@ -869,6 +899,7 @@ static ssize_t __FUNC(struct elevator_queue *e, const char *page, size_t count) STORE_FUNCTION(__FUNC, __PTR, MIN, MAX, msecs_to_jiffies) STORE_JIFFIES(deadline_read_expire_store, &dd->fifo_expire[DD_READ], 0, INT_MAX); STORE_JIFFIES(deadline_write_expire_store, &dd->fifo_expire[DD_WRITE], 0, INT_MAX); +STORE_JIFFIES(deadline_aging_expire_store, &dd->aging_expire, 0, INT_MAX); STORE_INT(deadline_writes_starved_store, &dd->writes_starved, INT_MIN, INT_MAX); STORE_INT(deadline_front_merges_store, &dd->front_merges, 0, 1); STORE_INT(deadline_fifo_batch_store, &dd->fifo_batch, 0, INT_MAX); @@ -885,6 +916,7 @@ static struct elv_fs_entry deadline_attrs[] = { DD_ATTR(writes_starved), DD_ATTR(front_merges), DD_ATTR(fifo_batch), + DD_ATTR(aging_expire), __ATTR_NULL };