From patchwork Thu May 27 01:01:26 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bart Van Assche X-Patchwork-Id: 12283105 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id EBD8BC47082 for ; Thu, 27 May 2021 01:01:49 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id CC040613C5 for ; Thu, 27 May 2021 01:01:49 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233490AbhE0BDU (ORCPT ); Wed, 26 May 2021 21:03:20 -0400 Received: from mail-pf1-f169.google.com ([209.85.210.169]:33596 "EHLO mail-pf1-f169.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231364AbhE0BDU (ORCPT ); Wed, 26 May 2021 21:03:20 -0400 Received: by mail-pf1-f169.google.com with SMTP id f22so2366134pfn.0 for ; Wed, 26 May 2021 18:01:47 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Yeomt37aZvE3pGOlGPZ2DrhRIPFlungSBjtQq0xgL9A=; b=bNWBybfQdTChF6Z4rAMGmohEgqvChdsh09wZKaYv0d65ZCGJO9HMBWyMWMwNdDtImj wKd78TAZWwZ9wXoPsvNCDgWUNYHrWASBhA0JJ6CuAKs5ysiH1AB6ahrcIJ3jbfetZY9N 04o5QVH50ZWHnjY3927UhWJ/3PgN/R3KPWfaq20r/yrSIfEC0qpjXw3i0hXIx8PbgLra GL3H+rC4hKCDFEVxPKC9IqFbMNi9w9DBWu2J7gFoPshbjvS9jEzvqrWj4ljytKTkSMCo kUZJQbOR/UdT1XEReqCupnKksv6cf89kJBYJzLm9fv3hXDTBXWN+UJIuSdPuIUPCTdjx 6uOA== X-Gm-Message-State: AOAM530xVwY8Yf2hHw4GQSWhsXngOUVPg5JybrHzwr+jM4xT3z8UePmQ zgvHadyGb+Txfaj6BcObzCY= X-Google-Smtp-Source: ABdhPJyaz4mdTGJLNsp0MuFHgKv9qD1MTY8emL8kG8QlNdJxpMwHZgs3hGiB5eQyLXgdalSkQx1TQQ== X-Received: by 2002:a63:64c6:: with SMTP id y189mr1215345pgb.333.1622077307049; Wed, 26 May 2021 18:01:47 -0700 (PDT) Received: from asus.hsd1.ca.comcast.net (c-73-241-217-19.hsd1.ca.comcast.net. [73.241.217.19]) by smtp.gmail.com with ESMTPSA id j22sm310707pfd.215.2021.05.26.18.01.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 26 May 2021 18:01:46 -0700 (PDT) From: Bart Van Assche To: Jens Axboe Cc: linux-block@vger.kernel.org, Christoph Hellwig , Jaegeuk Kim , Adam Manzanares , Bart Van Assche , Damien Le Moal , Ming Lei Subject: [PATCH 1/9] block/mq-deadline: Add several comments Date: Wed, 26 May 2021 18:01:26 -0700 Message-Id: <20210527010134.32448-2-bvanassche@acm.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210527010134.32448-1-bvanassche@acm.org> References: <20210527010134.32448-1-bvanassche@acm.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org Make the code easier to read by adding more comments. Cc: Damien Le Moal Cc: Christoph Hellwig Cc: Ming Lei Signed-off-by: Bart Van Assche Reviewed-by: Damien Le Moal Reviewed-by: Johannes Thumshirn --- block/mq-deadline.c | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/block/mq-deadline.c b/block/mq-deadline.c index 8eea2cbf2bf4..64cabbc157ea 100644 --- a/block/mq-deadline.c +++ b/block/mq-deadline.c @@ -139,6 +139,9 @@ static void dd_request_merged(struct request_queue *q, struct request *req, } } +/* + * Callback function that is invoked after @next has been merged into @req. + */ static void dd_merged_requests(struct request_queue *q, struct request *req, struct request *next) { @@ -375,6 +378,8 @@ static struct request *__dd_dispatch_request(struct deadline_data *dd) } /* + * Called from blk_mq_run_hw_queue() -> __blk_mq_sched_dispatch_requests(). + * * One confusing aspect here is that we get called for a specific * hardware queue, but we may return a request that is for a * different hardware queue. This is because mq-deadline has shared @@ -438,6 +443,10 @@ static int dd_init_queue(struct request_queue *q, struct elevator_type *e) return 0; } +/* + * Try to merge @bio into an existing request. If @bio has been merged into + * an existing request, store the pointer to that request into *@rq. + */ static int dd_request_merge(struct request_queue *q, struct request **rq, struct bio *bio) { @@ -461,6 +470,10 @@ static int dd_request_merge(struct request_queue *q, struct request **rq, return ELEVATOR_NO_MERGE; } +/* + * Attempt to merge a bio into an existing request. This function is called + * before @bio is associated with a request. + */ static bool dd_bio_merge(struct request_queue *q, struct bio *bio, unsigned int nr_segs) { @@ -518,6 +531,9 @@ static void dd_insert_request(struct blk_mq_hw_ctx *hctx, struct request *rq, } } +/* + * Called from blk_mq_sched_insert_request() or blk_mq_sched_insert_requests(). + */ static void dd_insert_requests(struct blk_mq_hw_ctx *hctx, struct list_head *list, bool at_head) { @@ -544,6 +560,8 @@ static void dd_prepare_request(struct request *rq) } /* + * Callback function called from inside blk_mq_free_request(). + * * For zoned block devices, write unlock the target zone of * completed write requests. Do this while holding the zone lock * spinlock so that the zone is never unlocked while deadline_fifo_request() From patchwork Thu May 27 01:01:27 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bart Van Assche X-Patchwork-Id: 12283107 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 59DBCC4708A for ; Thu, 27 May 2021 01:01:51 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 3711A6128B for ; Thu, 27 May 2021 01:01:51 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233491AbhE0BDW (ORCPT ); Wed, 26 May 2021 21:03:22 -0400 Received: from mail-pf1-f174.google.com ([209.85.210.174]:35824 "EHLO mail-pf1-f174.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231364AbhE0BDW (ORCPT ); Wed, 26 May 2021 21:03:22 -0400 Received: by mail-pf1-f174.google.com with SMTP id g18so2348297pfr.2 for ; Wed, 26 May 2021 18:01:49 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=CBVcGmIsQ4t8SQd6Cf5AhXexoCdVAbFCg7YYTBO9lk0=; b=DevwN+9CRswWsVUoiBQa9hKcFSgfLyCHRkM7u7QQGTVnr6Nup4iOt5F/gZ7HvGBl8p EjHH5XPQ9e2vopXt1sWr7k+f4E704q/+c9AKSb+6p6f9MM/QhXAa9KM91x7yQmSRvz2b 7wJczVfs6N2PPf/7G7a1lqlE7YFhMyFnU9EA41PvwXG8f27NGPGvr6mlqgflttqPbmah i6sZLcaFjAZ+zuUqKChb8CvGoXPffzUWdlunTMzR+odFW1pnbgnK8p8yez+L+ZIkm8OG 31MQ5gODYISbjFzP4or3qItrohw3Rne0xhZGJ23eK65VK5bbaicV7jCcEe2MukZq+4TZ j3cw== X-Gm-Message-State: AOAM530mB3bZcpAw98QYJRC29qcHyx3gHIcud11e0oYRAhXxHoWLL5dK yUIldgl273ZuuuXhv572urnzZ6xeHE8= X-Google-Smtp-Source: ABdhPJwSSR4A1A7bHNEM05bJapun4betSESNjjqj4m3hymTBGBLBEtQdOXX+nZxGcgv8HZaSckuvFg== X-Received: by 2002:a05:6a00:1a0a:b029:2e0:11d1:6d73 with SMTP id g10-20020a056a001a0ab02902e011d16d73mr1209737pfv.34.1622077308618; Wed, 26 May 2021 18:01:48 -0700 (PDT) Received: from asus.hsd1.ca.comcast.net (c-73-241-217-19.hsd1.ca.comcast.net. [73.241.217.19]) by smtp.gmail.com with ESMTPSA id j22sm310707pfd.215.2021.05.26.18.01.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 26 May 2021 18:01:47 -0700 (PDT) From: Bart Van Assche To: Jens Axboe Cc: linux-block@vger.kernel.org, Christoph Hellwig , Jaegeuk Kim , Adam Manzanares , Bart Van Assche , Damien Le Moal , Ming Lei Subject: [PATCH 2/9] block/mq-deadline: Add two lockdep_assert_held() statements Date: Wed, 26 May 2021 18:01:27 -0700 Message-Id: <20210527010134.32448-3-bvanassche@acm.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210527010134.32448-1-bvanassche@acm.org> References: <20210527010134.32448-1-bvanassche@acm.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org Document the locking strategy by adding two lockdep_assert_held() statements. Cc: Damien Le Moal Cc: Christoph Hellwig Cc: Ming Lei Signed-off-by: Bart Van Assche Reviewed-by: Chaitanya Kulkarni Reviewed-by: Damien Le Moal Reviewed-by: Hannes Reinecke Reviewed-by: Johannes Thumshirn --- block/mq-deadline.c | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/block/mq-deadline.c b/block/mq-deadline.c index 64cabbc157ea..4da0bd3b9827 100644 --- a/block/mq-deadline.c +++ b/block/mq-deadline.c @@ -279,6 +279,8 @@ static struct request *__dd_dispatch_request(struct deadline_data *dd) bool reads, writes; int data_dir; + lockdep_assert_held(&dd->lock); + if (!list_empty(&dd->dispatch)) { rq = list_first_entry(&dd->dispatch, struct request, queuelist); list_del_init(&rq->queuelist); @@ -501,6 +503,8 @@ static void dd_insert_request(struct blk_mq_hw_ctx *hctx, struct request *rq, struct deadline_data *dd = q->elevator->elevator_data; const int data_dir = rq_data_dir(rq); + lockdep_assert_held(&dd->lock); + /* * This may be a requeue of a write request that has locked its * target zone. If it is the case, this releases the zone lock. From patchwork Thu May 27 01:01:28 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bart Van Assche X-Patchwork-Id: 12283109 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C43DBC47082 for ; Thu, 27 May 2021 01:01:53 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id A5FBD613C5 for ; Thu, 27 May 2021 01:01:53 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231364AbhE0BDY (ORCPT ); Wed, 26 May 2021 21:03:24 -0400 Received: from mail-pg1-f178.google.com ([209.85.215.178]:44870 "EHLO mail-pg1-f178.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233528AbhE0BDY (ORCPT ); Wed, 26 May 2021 21:03:24 -0400 Received: by mail-pg1-f178.google.com with SMTP id 29so2403326pgu.11 for ; Wed, 26 May 2021 18:01:51 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=4edXMHzGiEojjOWUWm/R1SRrs1nterPwfP9eCCDFqEE=; b=joYYRpojv1xNoHiKDq3IzYkdrU2V68T71Ou9YUYp5c1qFDAVVekyxjQ0Mh1RJJWr1C oXy5B/wEsXa6C2M+ql4BSy7MNvjZtSfTl60wiZHiICBytEhcu9AuR1MCZILnDEPbrZFj 7GRm3QNWbZDXzEbuWsmfYADiPJsXhPawt/oXhIoZfrIrtJfRkNFyaKEdgzuaBTBiS5xl hHQuL1QPtpJDJheUAG3aQ27GrB4PamBV6W+xgvbp7/Xgq9/iIc5tKgMdr2v9UEZrRj5h e9k/HP/7B64/UnI73tTQLT59+gn5JOGAkvzdfcvJeh3Sn9EHj7XWZbLH/e8O5T1Ayhdq sa9w== X-Gm-Message-State: AOAM531727FP07Mh98o0u8Tjtpk5+y/NEBfIF4g9S2X6VHEM2aPiMrzj jKDTPsytuhpZv6gj7TUBI0g= X-Google-Smtp-Source: ABdhPJzA/RN/PgULyICnmurS/sqpc5Au6b65XYMwOpUgI2H6BzePXQZfeg1v8mYAwLnkuM1tqV1iiA== X-Received: by 2002:a63:1d09:: with SMTP id d9mr1243805pgd.302.1622077310905; Wed, 26 May 2021 18:01:50 -0700 (PDT) Received: from asus.hsd1.ca.comcast.net (c-73-241-217-19.hsd1.ca.comcast.net. [73.241.217.19]) by smtp.gmail.com with ESMTPSA id j22sm310707pfd.215.2021.05.26.18.01.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 26 May 2021 18:01:49 -0700 (PDT) From: Bart Van Assche To: Jens Axboe Cc: linux-block@vger.kernel.org, Christoph Hellwig , Jaegeuk Kim , Adam Manzanares , Bart Van Assche , Damien Le Moal , Ming Lei Subject: [PATCH 3/9] block/mq-deadline: Remove two local variables Date: Wed, 26 May 2021 18:01:28 -0700 Message-Id: <20210527010134.32448-4-bvanassche@acm.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210527010134.32448-1-bvanassche@acm.org> References: <20210527010134.32448-1-bvanassche@acm.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org Make __dd_dispatch_request() easier to read by removing two local variables. Cc: Damien Le Moal Cc: Christoph Hellwig Cc: Ming Lei Signed-off-by: Bart Van Assche Reviewed-by: Chaitanya Kulkarni Reviewed-by: Damien Le Moal Reviewed-by: Hannes Reinecke Reviewed-by: Johannes Thumshirn Reviewed-by: Himanshu Madhani --- block/mq-deadline.c | 8 ++------ 1 file changed, 2 insertions(+), 6 deletions(-) diff --git a/block/mq-deadline.c b/block/mq-deadline.c index 4da0bd3b9827..cc9d0fc72db2 100644 --- a/block/mq-deadline.c +++ b/block/mq-deadline.c @@ -276,7 +276,6 @@ deadline_next_request(struct deadline_data *dd, int data_dir) static struct request *__dd_dispatch_request(struct deadline_data *dd) { struct request *rq, *next_rq; - bool reads, writes; int data_dir; lockdep_assert_held(&dd->lock); @@ -287,9 +286,6 @@ static struct request *__dd_dispatch_request(struct deadline_data *dd) goto done; } - reads = !list_empty(&dd->fifo_list[READ]); - writes = !list_empty(&dd->fifo_list[WRITE]); - /* * batches are currently reads XOR writes */ @@ -306,7 +302,7 @@ static struct request *__dd_dispatch_request(struct deadline_data *dd) * data direction (read / write) */ - if (reads) { + if (!list_empty(&dd->fifo_list[READ])) { BUG_ON(RB_EMPTY_ROOT(&dd->sort_list[READ])); if (deadline_fifo_request(dd, WRITE) && @@ -322,7 +318,7 @@ static struct request *__dd_dispatch_request(struct deadline_data *dd) * there are either no reads or writes have been starved */ - if (writes) { + if (!list_empty(&dd->fifo_list[WRITE])) { dispatch_writes: BUG_ON(RB_EMPTY_ROOT(&dd->sort_list[WRITE])); From patchwork Thu May 27 01:01:29 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bart Van Assche X-Patchwork-Id: 12283111 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E3969C47089 for ; Thu, 27 May 2021 01:01:54 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id BD73A6128B for ; Thu, 27 May 2021 01:01:54 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233528AbhE0BD0 (ORCPT ); Wed, 26 May 2021 21:03:26 -0400 Received: from mail-pj1-f48.google.com ([209.85.216.48]:53101 "EHLO mail-pj1-f48.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233592AbhE0BDZ (ORCPT ); Wed, 26 May 2021 21:03:25 -0400 Received: by mail-pj1-f48.google.com with SMTP id q6so1781905pjj.2 for ; Wed, 26 May 2021 18:01:52 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Q8hyuqg/tAJ17moIrqTXZyuv3MUbSAHq4QLZ5IGOWpY=; b=Xam8NIY8u7uBLCa6mJK4Y2x4mpEv9RrYIcwVs1hiU3pE6BKM4zhI5IzwA2OjLY8EvH F9SXYUVqhgL5aJsZomxIOhVJy2HLJ8aKX8CEzxn6GUEnLhFnO7qL+hVZECW9iEXeCZ7v tZZvMFIyc6+ADp1Bx2EuqrLP7mv2WR8zxdBTYHJ86azSHEl5uO86B9laJACAgbsrXBbE T2FgkU+6/O8CnU2byc4q4jpcnJmFfTtuyqW8lU2kQvVjCSXa+55kp3OvHRndmePLcnbX CXcI81r5cIYcZ7m/WEl2B3EhCYUUrjXBJJ6uFUv9KwECfPhDPB5CbvuF5YR0SjsKAxCI fGfA== X-Gm-Message-State: AOAM531RejY8Ty9Xp3y7MDNFkqIlNPRobN7Sh+o2b/adFOWXcvrzgo3v 8aWz7WK2T7TnZfly5/u//3E= X-Google-Smtp-Source: ABdhPJzmifAG0Rp1KT4nHuCId/lkxiaDeZHJqE1uy3WLklKVSiSy0kS2RFysk7ehin+7tC4HTi1T/g== X-Received: by 2002:a17:902:d4cd:b029:f5:4ec0:d593 with SMTP id o13-20020a170902d4cdb02900f54ec0d593mr841845plg.19.1622077312342; Wed, 26 May 2021 18:01:52 -0700 (PDT) Received: from asus.hsd1.ca.comcast.net (c-73-241-217-19.hsd1.ca.comcast.net. [73.241.217.19]) by smtp.gmail.com with ESMTPSA id j22sm310707pfd.215.2021.05.26.18.01.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 26 May 2021 18:01:51 -0700 (PDT) From: Bart Van Assche To: Jens Axboe Cc: linux-block@vger.kernel.org, Christoph Hellwig , Jaegeuk Kim , Adam Manzanares , Bart Van Assche , Damien Le Moal , Ming Lei Subject: [PATCH 4/9] block/mq-deadline: Rename dd_init_queue() and dd_exit_queue() Date: Wed, 26 May 2021 18:01:29 -0700 Message-Id: <20210527010134.32448-5-bvanassche@acm.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210527010134.32448-1-bvanassche@acm.org> References: <20210527010134.32448-1-bvanassche@acm.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org Change "queue" into "sched" to make the function names reflect better the purpose of these functions. Cc: Damien Le Moal Cc: Christoph Hellwig Cc: Ming Lei Signed-off-by: Bart Van Assche Reviewed-by: Chaitanya Kulkarni Reviewed-by: Damien Le Moal Reviewed-by: Hannes Reinecke Reviewed-by: Johannes Thumshirn Reviewed-by: Himanshu Madhani --- block/mq-deadline.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/block/mq-deadline.c b/block/mq-deadline.c index cc9d0fc72db2..c4f51e7884fb 100644 --- a/block/mq-deadline.c +++ b/block/mq-deadline.c @@ -395,7 +395,7 @@ static struct request *dd_dispatch_request(struct blk_mq_hw_ctx *hctx) return rq; } -static void dd_exit_queue(struct elevator_queue *e) +static void dd_exit_sched(struct elevator_queue *e) { struct deadline_data *dd = e->elevator_data; @@ -408,7 +408,7 @@ static void dd_exit_queue(struct elevator_queue *e) /* * initialize elevator private data (deadline_data). */ -static int dd_init_queue(struct request_queue *q, struct elevator_type *e) +static int dd_init_sched(struct request_queue *q, struct elevator_type *e) { struct deadline_data *dd; struct elevator_queue *eq; @@ -800,8 +800,8 @@ static struct elevator_type mq_deadline = { .requests_merged = dd_merged_requests, .request_merged = dd_request_merged, .has_work = dd_has_work, - .init_sched = dd_init_queue, - .exit_sched = dd_exit_queue, + .init_sched = dd_init_sched, + .exit_sched = dd_exit_sched, }, #ifdef CONFIG_BLK_DEBUG_FS From patchwork Thu May 27 01:01:30 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bart Van Assche X-Patchwork-Id: 12283113 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E0135C47082 for ; Thu, 27 May 2021 01:01:56 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id BD18C613CC for ; Thu, 27 May 2021 01:01:56 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233788AbhE0BD1 (ORCPT ); Wed, 26 May 2021 21:03:27 -0400 Received: from mail-pl1-f172.google.com ([209.85.214.172]:43594 "EHLO mail-pl1-f172.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233801AbhE0BD1 (ORCPT ); Wed, 26 May 2021 21:03:27 -0400 Received: by mail-pl1-f172.google.com with SMTP id v12so1489699plo.10 for ; Wed, 26 May 2021 18:01:54 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=/9lr2FlM8aPVNr8W5stKehP5sCAwrhh9DM0XcB/Dpj4=; b=atiasqZzlSmlHBkTyjSeLxo6f0d3JtZS5r9BZWQQ6h1gmt76MaUqdnzmMHJd7+HHyU 3mIoubKpbVo0eKrTzP+3U2w/YLZaBCvwxEbbEiwBCl3P8Hc1ODBKTiuBfJ3LQHZNWq37 NthZh6at5Saoa59fW06i454XvDHtrNXNtdoVQ/h/q7uclIDxwY4VqZBwpzqovxTqa512 lNIQV2U/j4EpzSjXEmDxO4OiFL+TU1MetB/HRE202KcBIAvXasBt/8NfTkXwWDXSqwUn iCZdqpAoUp8CFM9LMJbjJG9yUdpBajf3smdWkCkU+9dVY4WigpyXt6+ySpOkQaGF1nMO RtSA== X-Gm-Message-State: AOAM5331VS5zUHHbhUN7visyuntDTvKP90ACjin4fSAdFjaqLG8HCQz+ Bp/4iD9ozjpsqqSJQPn9r9Y= X-Google-Smtp-Source: ABdhPJy1dXqu6+8aniIUpGofx4tJknaZ/a5UT7JWbpeKof3OAafdfLV/u1MzJWAYi4bj7oKA2OcbXw== X-Received: by 2002:a17:90a:3041:: with SMTP id q1mr6693684pjl.191.1622077313874; Wed, 26 May 2021 18:01:53 -0700 (PDT) Received: from asus.hsd1.ca.comcast.net (c-73-241-217-19.hsd1.ca.comcast.net. [73.241.217.19]) by smtp.gmail.com with ESMTPSA id j22sm310707pfd.215.2021.05.26.18.01.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 26 May 2021 18:01:53 -0700 (PDT) From: Bart Van Assche To: Jens Axboe Cc: linux-block@vger.kernel.org, Christoph Hellwig , Jaegeuk Kim , Adam Manzanares , Bart Van Assche , Damien Le Moal , Ming Lei Subject: [PATCH 5/9] block/mq-deadline: Improve compile-time argument checking Date: Wed, 26 May 2021 18:01:30 -0700 Message-Id: <20210527010134.32448-6-bvanassche@acm.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210527010134.32448-1-bvanassche@acm.org> References: <20210527010134.32448-1-bvanassche@acm.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org Modern compilers complain if an out-of-range value is passed to a function argument that has an enumeration type. Let the compiler detect out-of-range data direction arguments instead of verifying the data_dir argument at runtime. Cc: Damien Le Moal Cc: Christoph Hellwig Cc: Ming Lei Signed-off-by: Bart Van Assche Reviewed-by: Chaitanya Kulkarni Reviewed-by: Hannes Reinecke Reviewed-by: Johannes Thumshirn Reviewed-by: Himanshu Madhani --- block/mq-deadline.c | 96 +++++++++++++++++++++++---------------------- 1 file changed, 49 insertions(+), 47 deletions(-) diff --git a/block/mq-deadline.c b/block/mq-deadline.c index c4f51e7884fb..dfbc6b77fa71 100644 --- a/block/mq-deadline.c +++ b/block/mq-deadline.c @@ -35,6 +35,13 @@ static const int writes_starved = 2; /* max times reads can starve a write */ static const int fifo_batch = 16; /* # of sequential requests treated as one by the above parameters. For throughput. */ +enum dd_data_dir { + DD_READ = READ, + DD_WRITE = WRITE, +} __packed; + +enum { DD_DIR_COUNT = 2 }; + struct deadline_data { /* * run time data @@ -43,20 +50,20 @@ struct deadline_data { /* * requests (deadline_rq s) are present on both sort_list and fifo_list */ - struct rb_root sort_list[2]; - struct list_head fifo_list[2]; + struct rb_root sort_list[DD_DIR_COUNT]; + struct list_head fifo_list[DD_DIR_COUNT]; /* * next in sort order. read, write or both are NULL */ - struct request *next_rq[2]; + struct request *next_rq[DD_DIR_COUNT]; unsigned int batching; /* number of sequential requests made */ unsigned int starved; /* times reads have starved writes */ /* * settings that change how the i/o scheduler behaves */ - int fifo_expire[2]; + int fifo_expire[DD_DIR_COUNT]; int fifo_batch; int writes_starved; int front_merges; @@ -97,7 +104,7 @@ deadline_add_rq_rb(struct deadline_data *dd, struct request *rq) static inline void deadline_del_rq_rb(struct deadline_data *dd, struct request *rq) { - const int data_dir = rq_data_dir(rq); + const enum dd_data_dir data_dir = rq_data_dir(rq); if (dd->next_rq[data_dir] == rq) dd->next_rq[data_dir] = deadline_latter_request(rq); @@ -169,10 +176,10 @@ static void dd_merged_requests(struct request_queue *q, struct request *req, static void deadline_move_request(struct deadline_data *dd, struct request *rq) { - const int data_dir = rq_data_dir(rq); + const enum dd_data_dir data_dir = rq_data_dir(rq); - dd->next_rq[READ] = NULL; - dd->next_rq[WRITE] = NULL; + dd->next_rq[DD_READ] = NULL; + dd->next_rq[DD_WRITE] = NULL; dd->next_rq[data_dir] = deadline_latter_request(rq); /* @@ -185,9 +192,10 @@ deadline_move_request(struct deadline_data *dd, struct request *rq) * deadline_check_fifo returns 0 if there are no expired requests on the fifo, * 1 otherwise. Requires !list_empty(&dd->fifo_list[data_dir]) */ -static inline int deadline_check_fifo(struct deadline_data *dd, int ddir) +static inline int deadline_check_fifo(struct deadline_data *dd, + enum dd_data_dir data_dir) { - struct request *rq = rq_entry_fifo(dd->fifo_list[ddir].next); + struct request *rq = rq_entry_fifo(dd->fifo_list[data_dir].next); /* * rq is expired! @@ -203,19 +211,16 @@ static inline int deadline_check_fifo(struct deadline_data *dd, int ddir) * dispatch using arrival ordered lists. */ static struct request * -deadline_fifo_request(struct deadline_data *dd, int data_dir) +deadline_fifo_request(struct deadline_data *dd, enum dd_data_dir data_dir) { struct request *rq; unsigned long flags; - if (WARN_ON_ONCE(data_dir != READ && data_dir != WRITE)) - return NULL; - if (list_empty(&dd->fifo_list[data_dir])) return NULL; rq = rq_entry_fifo(dd->fifo_list[data_dir].next); - if (data_dir == READ || !blk_queue_is_zoned(rq->q)) + if (data_dir == DD_READ || !blk_queue_is_zoned(rq->q)) return rq; /* @@ -223,7 +228,7 @@ deadline_fifo_request(struct deadline_data *dd, int data_dir) * an unlocked target zone. */ spin_lock_irqsave(&dd->zone_lock, flags); - list_for_each_entry(rq, &dd->fifo_list[WRITE], queuelist) { + list_for_each_entry(rq, &dd->fifo_list[DD_WRITE], queuelist) { if (blk_req_can_dispatch_to_zone(rq)) goto out; } @@ -239,19 +244,16 @@ deadline_fifo_request(struct deadline_data *dd, int data_dir) * dispatch using sector position sorted lists. */ static struct request * -deadline_next_request(struct deadline_data *dd, int data_dir) +deadline_next_request(struct deadline_data *dd, enum dd_data_dir data_dir) { struct request *rq; unsigned long flags; - if (WARN_ON_ONCE(data_dir != READ && data_dir != WRITE)) - return NULL; - rq = dd->next_rq[data_dir]; if (!rq) return NULL; - if (data_dir == READ || !blk_queue_is_zoned(rq->q)) + if (data_dir == DD_READ || !blk_queue_is_zoned(rq->q)) return rq; /* @@ -276,7 +278,7 @@ deadline_next_request(struct deadline_data *dd, int data_dir) static struct request *__dd_dispatch_request(struct deadline_data *dd) { struct request *rq, *next_rq; - int data_dir; + enum dd_data_dir data_dir; lockdep_assert_held(&dd->lock); @@ -289,9 +291,9 @@ static struct request *__dd_dispatch_request(struct deadline_data *dd) /* * batches are currently reads XOR writes */ - rq = deadline_next_request(dd, WRITE); + rq = deadline_next_request(dd, DD_WRITE); if (!rq) - rq = deadline_next_request(dd, READ); + rq = deadline_next_request(dd, DD_READ); if (rq && dd->batching < dd->fifo_batch) /* we have a next request are still entitled to batch */ @@ -302,14 +304,14 @@ static struct request *__dd_dispatch_request(struct deadline_data *dd) * data direction (read / write) */ - if (!list_empty(&dd->fifo_list[READ])) { - BUG_ON(RB_EMPTY_ROOT(&dd->sort_list[READ])); + if (!list_empty(&dd->fifo_list[DD_READ])) { + BUG_ON(RB_EMPTY_ROOT(&dd->sort_list[DD_READ])); - if (deadline_fifo_request(dd, WRITE) && + if (deadline_fifo_request(dd, DD_WRITE) && (dd->starved++ >= dd->writes_starved)) goto dispatch_writes; - data_dir = READ; + data_dir = DD_READ; goto dispatch_find_request; } @@ -318,13 +320,13 @@ static struct request *__dd_dispatch_request(struct deadline_data *dd) * there are either no reads or writes have been starved */ - if (!list_empty(&dd->fifo_list[WRITE])) { + if (!list_empty(&dd->fifo_list[DD_WRITE])) { dispatch_writes: - BUG_ON(RB_EMPTY_ROOT(&dd->sort_list[WRITE])); + BUG_ON(RB_EMPTY_ROOT(&dd->sort_list[DD_WRITE])); dd->starved = 0; - data_dir = WRITE; + data_dir = DD_WRITE; goto dispatch_find_request; } @@ -399,8 +401,8 @@ static void dd_exit_sched(struct elevator_queue *e) { struct deadline_data *dd = e->elevator_data; - BUG_ON(!list_empty(&dd->fifo_list[READ])); - BUG_ON(!list_empty(&dd->fifo_list[WRITE])); + BUG_ON(!list_empty(&dd->fifo_list[DD_READ])); + BUG_ON(!list_empty(&dd->fifo_list[DD_WRITE])); kfree(dd); } @@ -424,12 +426,12 @@ static int dd_init_sched(struct request_queue *q, struct elevator_type *e) } eq->elevator_data = dd; - INIT_LIST_HEAD(&dd->fifo_list[READ]); - INIT_LIST_HEAD(&dd->fifo_list[WRITE]); - dd->sort_list[READ] = RB_ROOT; - dd->sort_list[WRITE] = RB_ROOT; - dd->fifo_expire[READ] = read_expire; - dd->fifo_expire[WRITE] = write_expire; + INIT_LIST_HEAD(&dd->fifo_list[DD_READ]); + INIT_LIST_HEAD(&dd->fifo_list[DD_WRITE]); + dd->sort_list[DD_READ] = RB_ROOT; + dd->sort_list[DD_WRITE] = RB_ROOT; + dd->fifo_expire[DD_READ] = read_expire; + dd->fifo_expire[DD_WRITE] = write_expire; dd->writes_starved = writes_starved; dd->front_merges = 1; dd->fifo_batch = fifo_batch; @@ -497,7 +499,7 @@ static void dd_insert_request(struct blk_mq_hw_ctx *hctx, struct request *rq, { struct request_queue *q = hctx->queue; struct deadline_data *dd = q->elevator->elevator_data; - const int data_dir = rq_data_dir(rq); + const enum dd_data_dir data_dir = rq_data_dir(rq); lockdep_assert_held(&dd->lock); @@ -585,7 +587,7 @@ static void dd_finish_request(struct request *rq) spin_lock_irqsave(&dd->zone_lock, flags); blk_req_zone_write_unlock(rq); - if (!list_empty(&dd->fifo_list[WRITE])) + if (!list_empty(&dd->fifo_list[DD_WRITE])) blk_mq_sched_mark_restart_hctx(rq->mq_hctx); spin_unlock_irqrestore(&dd->zone_lock, flags); } @@ -626,8 +628,8 @@ static ssize_t __FUNC(struct elevator_queue *e, char *page) \ __data = jiffies_to_msecs(__data); \ return deadline_var_show(__data, (page)); \ } -SHOW_FUNCTION(deadline_read_expire_show, dd->fifo_expire[READ], 1); -SHOW_FUNCTION(deadline_write_expire_show, dd->fifo_expire[WRITE], 1); +SHOW_FUNCTION(deadline_read_expire_show, dd->fifo_expire[DD_READ], 1); +SHOW_FUNCTION(deadline_write_expire_show, dd->fifo_expire[DD_WRITE], 1); SHOW_FUNCTION(deadline_writes_starved_show, dd->writes_starved, 0); SHOW_FUNCTION(deadline_front_merges_show, dd->front_merges, 0); SHOW_FUNCTION(deadline_fifo_batch_show, dd->fifo_batch, 0); @@ -649,8 +651,8 @@ static ssize_t __FUNC(struct elevator_queue *e, const char *page, size_t count) *(__PTR) = __data; \ return count; \ } -STORE_FUNCTION(deadline_read_expire_store, &dd->fifo_expire[READ], 0, INT_MAX, 1); -STORE_FUNCTION(deadline_write_expire_store, &dd->fifo_expire[WRITE], 0, INT_MAX, 1); +STORE_FUNCTION(deadline_read_expire_store, &dd->fifo_expire[DD_READ], 0, INT_MAX, 1); +STORE_FUNCTION(deadline_write_expire_store, &dd->fifo_expire[DD_WRITE], 0, INT_MAX, 1); STORE_FUNCTION(deadline_writes_starved_store, &dd->writes_starved, INT_MIN, INT_MAX, 0); STORE_FUNCTION(deadline_front_merges_store, &dd->front_merges, 0, 1, 0); STORE_FUNCTION(deadline_fifo_batch_store, &dd->fifo_batch, 0, INT_MAX, 0); @@ -717,8 +719,8 @@ static int deadline_##name##_next_rq_show(void *data, \ __blk_mq_debugfs_rq_show(m, rq); \ return 0; \ } -DEADLINE_DEBUGFS_DDIR_ATTRS(READ, read) -DEADLINE_DEBUGFS_DDIR_ATTRS(WRITE, write) +DEADLINE_DEBUGFS_DDIR_ATTRS(DD_READ, read) +DEADLINE_DEBUGFS_DDIR_ATTRS(DD_WRITE, write) #undef DEADLINE_DEBUGFS_DDIR_ATTRS static int deadline_batching_show(void *data, struct seq_file *m) From patchwork Thu May 27 01:01:31 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bart Van Assche X-Patchwork-Id: 12283115 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 87FAFC47089 for ; Thu, 27 May 2021 01:01:59 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 6379B613C5 for ; Thu, 27 May 2021 01:01:59 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233801AbhE0BDa (ORCPT ); Wed, 26 May 2021 21:03:30 -0400 Received: from mail-pl1-f170.google.com ([209.85.214.170]:35789 "EHLO mail-pl1-f170.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233592AbhE0BD3 (ORCPT ); Wed, 26 May 2021 21:03:29 -0400 Received: by mail-pl1-f170.google.com with SMTP id t21so1503497plo.2 for ; Wed, 26 May 2021 18:01:56 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=YPnT1JBCPsUBvPWopNHHnlw5sGKoI9+ydtrjdwNEk/U=; b=setXY8E1uVAtF/sLjiTvgkEpAxxnUb7Y2ibiUnnzGq+EITnow0waLVkfN83rP2MHJ8 m3I5o+K4Jvecj8yRBwHxaSVae7PP2qKu/AI9h7qZvXo1lTWCWJPlITW7EXVXmbBHPBRF y5Q6i6f/7b4Lt7ZDPIykyWdeMSXUY/HSl7/Y9Vp3qEey+YW18PuYF07Dik7BQFwH3uGb E0hs8ScKuCY+f6Xn4/ZMbYkThj8V8VziJrkwl0wJRtA541Uwie6xVdUPrD5cqdBwtAxE v/gxyVwOiPH3KVzpdwwBHWHz5l3RokSOMIUibYz4LUwOcXlyQiPFHyuZqDrt9mJZqtyn 05FQ== X-Gm-Message-State: AOAM530Pz069wixk6zS1C1GbFUDHRsSDn0UX3WIyT8ShFXQvPufrz8Fc GoXEGQAug4lvedjlKh2l0xY= X-Google-Smtp-Source: ABdhPJwCFSxLBLkZNgPOqiBb7AfRp2zslkhXyIJOEltSaUS5RePAIRb3EBBY4cjipb49t44ZH4yfSg== X-Received: by 2002:a17:90a:bf91:: with SMTP id d17mr5897599pjs.17.1622077315356; Wed, 26 May 2021 18:01:55 -0700 (PDT) Received: from asus.hsd1.ca.comcast.net (c-73-241-217-19.hsd1.ca.comcast.net. [73.241.217.19]) by smtp.gmail.com with ESMTPSA id j22sm310707pfd.215.2021.05.26.18.01.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 26 May 2021 18:01:54 -0700 (PDT) From: Bart Van Assche To: Jens Axboe Cc: linux-block@vger.kernel.org, Christoph Hellwig , Jaegeuk Kim , Adam Manzanares , Bart Van Assche , Damien Le Moal , Ming Lei Subject: [PATCH 6/9] block/mq-deadline: Reduce the read expiry time for non-rotational media Date: Wed, 26 May 2021 18:01:31 -0700 Message-Id: <20210527010134.32448-7-bvanassche@acm.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210527010134.32448-1-bvanassche@acm.org> References: <20210527010134.32448-1-bvanassche@acm.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org Rotational media (hard disks) benefit more from merging requests than non-rotational media. Reduce the read expire time for non-rotational media to reduce read latency. Cc: Damien Le Moal Cc: Christoph Hellwig Cc: Ming Lei Signed-off-by: Bart Van Assche Reviewed-by: Chaitanya Kulkarni Reviewed-by: Himanshu Madhani --- block/mq-deadline.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/block/mq-deadline.c b/block/mq-deadline.c index dfbc6b77fa71..2ab844a4b6b5 100644 --- a/block/mq-deadline.c +++ b/block/mq-deadline.c @@ -29,7 +29,9 @@ /* * See Documentation/block/deadline-iosched.rst */ -static const int read_expire = HZ / 2; /* max time before a read is submitted. */ +/* max time before a read is submitted. */ +static const int read_expire_rot = HZ / 2; +static const int read_expire_nonrot = 1; static const int write_expire = 5 * HZ; /* ditto for writes, these limits are SOFT! */ static const int writes_starved = 2; /* max times reads can starve a write */ static const int fifo_batch = 16; /* # of sequential requests treated as one @@ -430,7 +432,8 @@ static int dd_init_sched(struct request_queue *q, struct elevator_type *e) INIT_LIST_HEAD(&dd->fifo_list[DD_WRITE]); dd->sort_list[DD_READ] = RB_ROOT; dd->sort_list[DD_WRITE] = RB_ROOT; - dd->fifo_expire[DD_READ] = read_expire; + dd->fifo_expire[DD_READ] = blk_queue_nonrot(q) ? read_expire_nonrot : + read_expire_rot; dd->fifo_expire[DD_WRITE] = write_expire; dd->writes_starved = writes_starved; dd->front_merges = 1; From patchwork Thu May 27 01:01:32 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bart Van Assche X-Patchwork-Id: 12283117 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id EB883C4708A for ; Thu, 27 May 2021 01:02:00 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id D08F66128B for ; Thu, 27 May 2021 01:02:00 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233800AbhE0BDb (ORCPT ); Wed, 26 May 2021 21:03:31 -0400 Received: from mail-pj1-f42.google.com ([209.85.216.42]:38657 "EHLO mail-pj1-f42.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234083AbhE0BDa (ORCPT ); Wed, 26 May 2021 21:03:30 -0400 Received: by mail-pj1-f42.google.com with SMTP id mp9-20020a17090b1909b029015fd1e3ad5aso1329558pjb.3 for ; Wed, 26 May 2021 18:01:57 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Nq+cjmm3jqmpucjC3WcUjC8nein4jmI17n/Vl9OOIsM=; b=gX4Rhs91Q0m3HB/GPLNLy1zHqBQEJ66KDhgKwgML0SugcgOdOhc4+3KaqzWiuZLh+l 9WL1YASemYW3DAd2Gp8gdhy4ixt0Y267sHFp0OJ9yqg0D9rrgSz0+/V6zhD+zuKoSB1N uxjdZx5HV3aGDpCSaq6LfTcMjuoGyDTsJ3/f9jAJhK7d5l8oDHWrpBHPWTp6bKOC4Mia WBNSou8SZ4OBfEWO78v16Ts5a9/rwQeQjR6DuOCmdoAU+whUzG301DyOSUINq+qLSB35 rerUmzUH7gY7LLL0uMwDy8ZJ+q0wad4cqm98RqCjJr8VRXK9rgvljM3ZR3uH/pYKoe0c AsFg== X-Gm-Message-State: AOAM531alngrm/6/3U2qhCf8C7jsNaQrjN7sRvU6kvIOI1c3PW+MCGqI X8TnXNLtPRDti4dJQG6Rw3A= X-Google-Smtp-Source: ABdhPJzg7D3lkcY6c29dPHD6x8y+GJ1a4Epgvc0hxrd+D0EDYTV4SKJ+3rHpEeTKLzLGejVn56QyjA== X-Received: by 2002:a17:90a:e508:: with SMTP id t8mr886334pjy.221.1622077317475; Wed, 26 May 2021 18:01:57 -0700 (PDT) Received: from asus.hsd1.ca.comcast.net (c-73-241-217-19.hsd1.ca.comcast.net. [73.241.217.19]) by smtp.gmail.com with ESMTPSA id j22sm310707pfd.215.2021.05.26.18.01.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 26 May 2021 18:01:56 -0700 (PDT) From: Bart Van Assche To: Jens Axboe Cc: linux-block@vger.kernel.org, Christoph Hellwig , Jaegeuk Kim , Adam Manzanares , Bart Van Assche , Damien Le Moal , Ming Lei Subject: [PATCH 7/9] block/mq-deadline: Reserve 25% of tags for synchronous requests Date: Wed, 26 May 2021 18:01:32 -0700 Message-Id: <20210527010134.32448-8-bvanassche@acm.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210527010134.32448-1-bvanassche@acm.org> References: <20210527010134.32448-1-bvanassche@acm.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org For interactive workloads it is important that synchronous requests are not delayed. Hence reserve 25% of tags for synchronous requests. This patch still allows asynchronous requests to fill the hardware queues since blk_mq_init_sched() makes sure that the number of scheduler requests is the double of the hardware queue depth. From blk_mq_init_sched(): q->nr_requests = 2 * min_t(unsigned int, q->tag_set->queue_depth, BLKDEV_MAX_RQ); Cc: Damien Le Moal Cc: Christoph Hellwig Cc: Ming Lei Signed-off-by: Bart Van Assche --- block/mq-deadline.c | 46 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 46 insertions(+) diff --git a/block/mq-deadline.c b/block/mq-deadline.c index 2ab844a4b6b5..81f487d77e09 100644 --- a/block/mq-deadline.c +++ b/block/mq-deadline.c @@ -69,6 +69,7 @@ struct deadline_data { int fifo_batch; int writes_starved; int front_merges; + u32 async_depth; spinlock_t lock; spinlock_t zone_lock; @@ -399,6 +400,38 @@ static struct request *dd_dispatch_request(struct blk_mq_hw_ctx *hctx) return rq; } +static void dd_limit_depth(unsigned int op, struct blk_mq_alloc_data *data) +{ + struct deadline_data *dd = data->q->elevator->elevator_data; + + /* Do not throttle synchronous reads. */ + if (op_is_sync(op) && !op_is_write(op)) + return; + + /* + * Throttle asynchronous requests and writes such that these requests + * do not block the allocation of synchronous requests. + */ + data->shallow_depth = dd->async_depth; +} + +static void dd_depth_updated(struct blk_mq_hw_ctx *hctx) +{ + struct request_queue *q = hctx->queue; + struct deadline_data *dd = q->elevator->elevator_data; + struct blk_mq_tags *tags = hctx->sched_tags; + + dd->async_depth = 3 * q->nr_requests / 4; + + sbitmap_queue_min_shallow_depth(tags->bitmap_tags, dd->async_depth); +} + +static int dd_init_hctx(struct blk_mq_hw_ctx *hctx, unsigned int hctx_idx) +{ + dd_depth_updated(hctx); + return 0; +} + static void dd_exit_sched(struct elevator_queue *e) { struct deadline_data *dd = e->elevator_data; @@ -744,6 +777,15 @@ static int deadline_starved_show(void *data, struct seq_file *m) return 0; } +static int dd_async_depth_show(void *data, struct seq_file *m) +{ + struct request_queue *q = data; + struct deadline_data *dd = q->elevator->elevator_data; + + seq_printf(m, "%u\n", dd->async_depth); + return 0; +} + static void *deadline_dispatch_start(struct seq_file *m, loff_t *pos) __acquires(&dd->lock) { @@ -786,6 +828,7 @@ static const struct blk_mq_debugfs_attr deadline_queue_debugfs_attrs[] = { DEADLINE_QUEUE_DDIR_ATTRS(write), {"batching", 0400, deadline_batching_show}, {"starved", 0400, deadline_starved_show}, + {"async_depth", 0400, dd_async_depth_show}, {"dispatch", 0400, .seq_ops = &deadline_dispatch_seq_ops}, {}, }; @@ -794,6 +837,8 @@ static const struct blk_mq_debugfs_attr deadline_queue_debugfs_attrs[] = { static struct elevator_type mq_deadline = { .ops = { + .depth_updated = dd_depth_updated, + .limit_depth = dd_limit_depth, .insert_requests = dd_insert_requests, .dispatch_request = dd_dispatch_request, .prepare_request = dd_prepare_request, @@ -807,6 +852,7 @@ static struct elevator_type mq_deadline = { .has_work = dd_has_work, .init_sched = dd_init_sched, .exit_sched = dd_exit_sched, + .init_hctx = dd_init_hctx, }, #ifdef CONFIG_BLK_DEBUG_FS From patchwork Thu May 27 01:01:33 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bart Van Assche X-Patchwork-Id: 12283119 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id AA21FC4708B for ; Thu, 27 May 2021 01:02:02 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 8C4CB613CC for ; Thu, 27 May 2021 01:02:02 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234142AbhE0BDd (ORCPT ); Wed, 26 May 2021 21:03:33 -0400 Received: from mail-pj1-f43.google.com ([209.85.216.43]:50931 "EHLO mail-pj1-f43.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233906AbhE0BDc (ORCPT ); Wed, 26 May 2021 21:03:32 -0400 Received: by mail-pj1-f43.google.com with SMTP id t11so1797724pjm.0 for ; Wed, 26 May 2021 18:01:59 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=XnxiebgqyAoC7h8USczT5VXqx4IzpNayB9gs9NtLVgA=; b=pUthFMOyufepHcV1cGUkllEvCjjcYJU774afPxebVGm6Oq1VR+1KN27C34uJknCprB LnAgE7kdLtd/A9NaEdH9ETDPr+5jDhlOL1VnMmcANxsB7+SF+hM3veEw3JiL4EFYgkRT KyXCxgQ7tMq5Al5YfOhRiQpBpHXGmKEktKiBWoOURucOJp4gLIWHcRhXjK/VmsezXTKq 2X9o6qfDCDWGNTpRlEhrl5c5W8Vtg8x16hF1ByNgfFqXywnGR52Sw8uFWY5Q7d05LGJG EHjAeZfoCGMWPNUDYnW8DdueQksofTqFgrRxYyWjzyv6TSBY/M/OlscG7PQpiEF4iFP4 PD/g== X-Gm-Message-State: AOAM533fUTnqH7JsxO/upwygD2TOQnp0fPsLnh+b71imSJBKyF8FST6M uKyITM+zmzoSw8baChveZoQ= X-Google-Smtp-Source: ABdhPJxqa5MD/ZAZzUWczTewm1KMAAui6FgTyIwMCE0BRhZ5Kz/2dn/BybGh6wn2lntKb3YAO7qkaA== X-Received: by 2002:a17:902:8ec4:b029:fb:33ad:e86f with SMTP id x4-20020a1709028ec4b02900fb33ade86fmr969761plo.4.1622077319009; Wed, 26 May 2021 18:01:59 -0700 (PDT) Received: from asus.hsd1.ca.comcast.net (c-73-241-217-19.hsd1.ca.comcast.net. [73.241.217.19]) by smtp.gmail.com with ESMTPSA id j22sm310707pfd.215.2021.05.26.18.01.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 26 May 2021 18:01:58 -0700 (PDT) From: Bart Van Assche To: Jens Axboe Cc: linux-block@vger.kernel.org, Christoph Hellwig , Jaegeuk Kim , Adam Manzanares , Bart Van Assche , Damien Le Moal , Ming Lei Subject: [PATCH 8/9] block/mq-deadline: Add I/O priority support Date: Wed, 26 May 2021 18:01:33 -0700 Message-Id: <20210527010134.32448-9-bvanassche@acm.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210527010134.32448-1-bvanassche@acm.org> References: <20210527010134.32448-1-bvanassche@acm.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org Maintain one FIFO list per I/O priority: RT, BE and IDLE. Only dispatch requests for a lower priority after all higher priority requests have finished. Maintain statistics for each priority level. Split the debugfs attributes per priority level as follows: $ ls /sys/kernel/debug/block/.../sched/ async_depth dispatch2 read_next_rq write2_fifo_list batching read0_fifo_list starved write_next_rq dispatch0 read1_fifo_list write0_fifo_list dispatch1 read2_fifo_list write1_fifo_list Cc: Damien Le Moal Cc: Christoph Hellwig Cc: Ming Lei Signed-off-by: Bart Van Assche --- block/mq-deadline.c | 293 +++++++++++++++++++++++++++++++++----------- 1 file changed, 223 insertions(+), 70 deletions(-) diff --git a/block/mq-deadline.c b/block/mq-deadline.c index 81f487d77e09..5a703e1228fa 100644 --- a/block/mq-deadline.c +++ b/block/mq-deadline.c @@ -44,16 +44,27 @@ enum dd_data_dir { enum { DD_DIR_COUNT = 2 }; +enum dd_prio { + DD_RT_PRIO = 0, + DD_BE_PRIO = 1, + DD_IDLE_PRIO = 2, + DD_PRIO_MAX = 2, +} __packed; + +enum { DD_PRIO_COUNT = 3 }; + struct deadline_data { /* * run time data */ /* - * requests (deadline_rq s) are present on both sort_list and fifo_list + * Requests are present on both sort_list[] and fifo_list[][]. The + * first index of fifo_list[][] is the I/O priority class (DD_*_PRIO). + * The second index is the data direction (rq_data_dir(rq)). */ struct rb_root sort_list[DD_DIR_COUNT]; - struct list_head fifo_list[DD_DIR_COUNT]; + struct list_head fifo_list[DD_PRIO_COUNT][DD_DIR_COUNT]; /* * next in sort order. read, write or both are NULL @@ -62,6 +73,12 @@ struct deadline_data { unsigned int batching; /* number of sequential requests made */ unsigned int starved; /* times reads have starved writes */ + /* statistics */ + atomic_t inserted[DD_PRIO_COUNT]; + atomic_t dispatched[DD_PRIO_COUNT]; + atomic_t merged[DD_PRIO_COUNT]; + atomic_t completed[DD_PRIO_COUNT]; + /* * settings that change how the i/o scheduler behaves */ @@ -73,7 +90,15 @@ struct deadline_data { spinlock_t lock; spinlock_t zone_lock; - struct list_head dispatch; + struct list_head dispatch[DD_PRIO_COUNT]; +}; + +/* Maps an I/O priority class to a deadline scheduler priority. */ +static const enum dd_prio ioprio_class_to_prio[] = { + [IOPRIO_CLASS_NONE] = DD_BE_PRIO, + [IOPRIO_CLASS_RT] = DD_RT_PRIO, + [IOPRIO_CLASS_BE] = DD_BE_PRIO, + [IOPRIO_CLASS_IDLE] = DD_IDLE_PRIO, }; static inline struct rb_root * @@ -149,12 +174,28 @@ static void dd_request_merged(struct request_queue *q, struct request *req, } } +/* Returns the I/O class that has been assigned by dd_insert_request(). */ +static u8 dd_rq_ioclass(struct request *rq) +{ + return (uintptr_t)rq->elv.priv[1]; +} + /* * Callback function that is invoked after @next has been merged into @req. */ static void dd_merged_requests(struct request_queue *q, struct request *req, struct request *next) { + struct deadline_data *dd = q->elevator->elevator_data; + const u8 ioprio_class = dd_rq_ioclass(next); + const enum dd_prio prio = ioprio_class_to_prio[ioprio_class]; + + if (next->elv.priv[0]) { + atomic_inc(&dd->merged[prio]); + } else { + WARN_ON_ONCE(true); + } + /* * if next expires before rq, assign its expire time to rq * and move into next position (next will be deleted) in fifo @@ -191,14 +232,22 @@ deadline_move_request(struct deadline_data *dd, struct request *rq) deadline_remove_request(rq->q, rq); } +/* Number of requests queued for a given priority level. */ +static u64 dd_queued(struct deadline_data *dd, enum dd_prio prio) +{ + return atomic_read(&dd->inserted[prio]) - + atomic_read(&dd->completed[prio]); +} + /* * deadline_check_fifo returns 0 if there are no expired requests on the fifo, * 1 otherwise. Requires !list_empty(&dd->fifo_list[data_dir]) */ static inline int deadline_check_fifo(struct deadline_data *dd, + enum dd_prio prio, enum dd_data_dir data_dir) { - struct request *rq = rq_entry_fifo(dd->fifo_list[data_dir].next); + struct request *rq = rq_entry_fifo(dd->fifo_list[prio][data_dir].next); /* * rq is expired! @@ -214,15 +263,16 @@ static inline int deadline_check_fifo(struct deadline_data *dd, * dispatch using arrival ordered lists. */ static struct request * -deadline_fifo_request(struct deadline_data *dd, enum dd_data_dir data_dir) +deadline_fifo_request(struct deadline_data *dd, enum dd_prio prio, + enum dd_data_dir data_dir) { struct request *rq; unsigned long flags; - if (list_empty(&dd->fifo_list[data_dir])) + if (list_empty(&dd->fifo_list[prio][data_dir])) return NULL; - rq = rq_entry_fifo(dd->fifo_list[data_dir].next); + rq = rq_entry_fifo(dd->fifo_list[prio][data_dir].next); if (data_dir == DD_READ || !blk_queue_is_zoned(rq->q)) return rq; @@ -231,7 +281,7 @@ deadline_fifo_request(struct deadline_data *dd, enum dd_data_dir data_dir) * an unlocked target zone. */ spin_lock_irqsave(&dd->zone_lock, flags); - list_for_each_entry(rq, &dd->fifo_list[DD_WRITE], queuelist) { + list_for_each_entry(rq, &dd->fifo_list[prio][DD_WRITE], queuelist) { if (blk_req_can_dispatch_to_zone(rq)) goto out; } @@ -247,7 +297,8 @@ deadline_fifo_request(struct deadline_data *dd, enum dd_data_dir data_dir) * dispatch using sector position sorted lists. */ static struct request * -deadline_next_request(struct deadline_data *dd, enum dd_data_dir data_dir) +deadline_next_request(struct deadline_data *dd, enum dd_prio prio, + enum dd_data_dir data_dir) { struct request *rq; unsigned long flags; @@ -278,15 +329,18 @@ deadline_next_request(struct deadline_data *dd, enum dd_data_dir data_dir) * deadline_dispatch_requests selects the best request according to * read/write expire, fifo_batch, etc */ -static struct request *__dd_dispatch_request(struct deadline_data *dd) +static struct request *__dd_dispatch_request(struct deadline_data *dd, + enum dd_prio prio) { struct request *rq, *next_rq; enum dd_data_dir data_dir; + u8 ioprio_class; lockdep_assert_held(&dd->lock); - if (!list_empty(&dd->dispatch)) { - rq = list_first_entry(&dd->dispatch, struct request, queuelist); + if (!list_empty(&dd->dispatch[prio])) { + rq = list_first_entry(&dd->dispatch[prio], struct request, + queuelist); list_del_init(&rq->queuelist); goto done; } @@ -294,9 +348,9 @@ static struct request *__dd_dispatch_request(struct deadline_data *dd) /* * batches are currently reads XOR writes */ - rq = deadline_next_request(dd, DD_WRITE); + rq = deadline_next_request(dd, prio, DD_WRITE); if (!rq) - rq = deadline_next_request(dd, DD_READ); + rq = deadline_next_request(dd, prio, DD_READ); if (rq && dd->batching < dd->fifo_batch) /* we have a next request are still entitled to batch */ @@ -307,10 +361,10 @@ static struct request *__dd_dispatch_request(struct deadline_data *dd) * data direction (read / write) */ - if (!list_empty(&dd->fifo_list[DD_READ])) { + if (!list_empty(&dd->fifo_list[prio][DD_READ])) { BUG_ON(RB_EMPTY_ROOT(&dd->sort_list[DD_READ])); - if (deadline_fifo_request(dd, DD_WRITE) && + if (deadline_fifo_request(dd, prio, DD_WRITE) && (dd->starved++ >= dd->writes_starved)) goto dispatch_writes; @@ -323,7 +377,7 @@ static struct request *__dd_dispatch_request(struct deadline_data *dd) * there are either no reads or writes have been starved */ - if (!list_empty(&dd->fifo_list[DD_WRITE])) { + if (!list_empty(&dd->fifo_list[prio][DD_WRITE])) { dispatch_writes: BUG_ON(RB_EMPTY_ROOT(&dd->sort_list[DD_WRITE])); @@ -340,14 +394,14 @@ static struct request *__dd_dispatch_request(struct deadline_data *dd) /* * we are not running a batch, find best request for selected data_dir */ - next_rq = deadline_next_request(dd, data_dir); - if (deadline_check_fifo(dd, data_dir) || !next_rq) { + next_rq = deadline_next_request(dd, prio, data_dir); + if (deadline_check_fifo(dd, prio, data_dir) || !next_rq) { /* * A deadline has expired, the last request was in the other * direction, or we have run out of higher-sectored requests. * Start again from the request with the earliest expiry time. */ - rq = deadline_fifo_request(dd, data_dir); + rq = deadline_fifo_request(dd, prio, data_dir); } else { /* * The last req was the same dir and we have a next request in @@ -372,6 +426,13 @@ static struct request *__dd_dispatch_request(struct deadline_data *dd) dd->batching++; deadline_move_request(dd, rq); done: + ioprio_class = dd_rq_ioclass(rq); + prio = ioprio_class_to_prio[ioprio_class]; + if (rq->elv.priv[0]) { + atomic_inc(&dd->dispatched[prio]); + } else { + WARN_ON_ONCE(true); + } /* * If the request needs its target zone locked, do it. */ @@ -392,9 +453,14 @@ static struct request *dd_dispatch_request(struct blk_mq_hw_ctx *hctx) { struct deadline_data *dd = hctx->queue->elevator->elevator_data; struct request *rq; + enum dd_prio prio; spin_lock(&dd->lock); - rq = __dd_dispatch_request(dd); + for (prio = 0; prio <= DD_PRIO_MAX; prio++) { + rq = __dd_dispatch_request(dd, prio); + if (rq || dd_queued(dd, prio)) + break; + } spin_unlock(&dd->lock); return rq; @@ -435,9 +501,12 @@ static int dd_init_hctx(struct blk_mq_hw_ctx *hctx, unsigned int hctx_idx) static void dd_exit_sched(struct elevator_queue *e) { struct deadline_data *dd = e->elevator_data; + enum dd_prio prio; - BUG_ON(!list_empty(&dd->fifo_list[DD_READ])); - BUG_ON(!list_empty(&dd->fifo_list[DD_WRITE])); + for (prio = 0; prio <= DD_PRIO_MAX; prio++) { + WARN_ON_ONCE(!list_empty(&dd->fifo_list[prio][DD_READ])); + WARN_ON_ONCE(!list_empty(&dd->fifo_list[prio][DD_WRITE])); + } kfree(dd); } @@ -449,6 +518,7 @@ static int dd_init_sched(struct request_queue *q, struct elevator_type *e) { struct deadline_data *dd; struct elevator_queue *eq; + enum dd_prio prio; eq = elevator_alloc(q, e); if (!eq) @@ -461,8 +531,11 @@ static int dd_init_sched(struct request_queue *q, struct elevator_type *e) } eq->elevator_data = dd; - INIT_LIST_HEAD(&dd->fifo_list[DD_READ]); - INIT_LIST_HEAD(&dd->fifo_list[DD_WRITE]); + for (prio = 0; prio <= DD_PRIO_MAX; prio++) { + INIT_LIST_HEAD(&dd->fifo_list[prio][DD_READ]); + INIT_LIST_HEAD(&dd->fifo_list[prio][DD_WRITE]); + INIT_LIST_HEAD(&dd->dispatch[prio]); + } dd->sort_list[DD_READ] = RB_ROOT; dd->sort_list[DD_WRITE] = RB_ROOT; dd->fifo_expire[DD_READ] = blk_queue_nonrot(q) ? read_expire_nonrot : @@ -473,7 +546,6 @@ static int dd_init_sched(struct request_queue *q, struct elevator_type *e) dd->fifo_batch = fifo_batch; spin_lock_init(&dd->lock); spin_lock_init(&dd->zone_lock); - INIT_LIST_HEAD(&dd->dispatch); q->elevator = eq; return 0; @@ -536,6 +608,9 @@ static void dd_insert_request(struct blk_mq_hw_ctx *hctx, struct request *rq, struct request_queue *q = hctx->queue; struct deadline_data *dd = q->elevator->elevator_data; const enum dd_data_dir data_dir = rq_data_dir(rq); + u16 ioprio = req_get_ioprio(rq); + u8 ioprio_class = IOPRIO_PRIO_CLASS(ioprio); + enum dd_prio prio; lockdep_assert_held(&dd->lock); @@ -545,13 +620,19 @@ static void dd_insert_request(struct blk_mq_hw_ctx *hctx, struct request *rq, */ blk_req_zone_write_unlock(rq); + prio = ioprio_class_to_prio[ioprio_class]; + atomic_inc(&dd->inserted[prio]); + WARN_ON_ONCE(rq->elv.priv[0]); + rq->elv.priv[0] = (void *)1ULL; + rq->elv.priv[1] = (void *)(uintptr_t)ioprio_class; + if (blk_mq_sched_try_insert_merge(q, rq)) return; trace_block_rq_insert(rq); if (at_head) { - list_add(&rq->queuelist, &dd->dispatch); + list_add(&rq->queuelist, &dd->dispatch[prio]); } else { deadline_add_rq_rb(dd, rq); @@ -565,7 +646,7 @@ static void dd_insert_request(struct blk_mq_hw_ctx *hctx, struct request *rq, * set expire time and add to fifo list */ rq->fifo_time = jiffies + dd->fifo_expire[data_dir]; - list_add_tail(&rq->queuelist, &dd->fifo_list[data_dir]); + list_add_tail(&rq->queuelist, &dd->fifo_list[prio][data_dir]); } } @@ -589,12 +670,10 @@ static void dd_insert_requests(struct blk_mq_hw_ctx *hctx, spin_unlock(&dd->lock); } -/* - * Nothing to do here. This is defined only to ensure that .finish_request - * method is called upon request completion. - */ +/* Callback function called from inside blk_mq_rq_ctx_init(). */ static void dd_prepare_request(struct request *rq) { + rq->elv.priv[0] = NULL; } /* @@ -616,26 +695,41 @@ static void dd_prepare_request(struct request *rq) static void dd_finish_request(struct request *rq) { struct request_queue *q = rq->q; + struct deadline_data *dd = q->elevator->elevator_data; + const u8 ioprio_class = dd_rq_ioclass(rq); + const enum dd_prio prio = ioprio_class_to_prio[ioprio_class]; + + if (rq->elv.priv[0]) + atomic_inc(&dd->completed[prio]); if (blk_queue_is_zoned(q)) { - struct deadline_data *dd = q->elevator->elevator_data; unsigned long flags; spin_lock_irqsave(&dd->zone_lock, flags); blk_req_zone_write_unlock(rq); - if (!list_empty(&dd->fifo_list[DD_WRITE])) + if (!list_empty(&dd->fifo_list[prio][DD_WRITE])) blk_mq_sched_mark_restart_hctx(rq->mq_hctx); spin_unlock_irqrestore(&dd->zone_lock, flags); } } +static bool dd_has_work_for_prio(struct deadline_data *dd, enum dd_prio prio) +{ + return !list_empty_careful(&dd->dispatch[prio]) || + !list_empty_careful(&dd->fifo_list[prio][DD_READ]) || + !list_empty_careful(&dd->fifo_list[prio][DD_WRITE]); +} + static bool dd_has_work(struct blk_mq_hw_ctx *hctx) { struct deadline_data *dd = hctx->queue->elevator->elevator_data; + enum dd_prio prio; + + for (prio = 0; prio <= DD_PRIO_MAX; prio++) + if (dd_has_work_for_prio(dd, prio)) + return true; - return !list_empty_careful(&dd->dispatch) || - !list_empty_careful(&dd->fifo_list[0]) || - !list_empty_careful(&dd->fifo_list[1]); + return false; } /* @@ -707,7 +801,7 @@ static struct elv_fs_entry deadline_attrs[] = { }; #ifdef CONFIG_BLK_DEBUG_FS -#define DEADLINE_DEBUGFS_DDIR_ATTRS(ddir, name) \ +#define DEADLINE_DEBUGFS_DDIR_ATTRS(prio, data_dir, name) \ static void *deadline_##name##_fifo_start(struct seq_file *m, \ loff_t *pos) \ __acquires(&dd->lock) \ @@ -716,7 +810,7 @@ static void *deadline_##name##_fifo_start(struct seq_file *m, \ struct deadline_data *dd = q->elevator->elevator_data; \ \ spin_lock(&dd->lock); \ - return seq_list_start(&dd->fifo_list[ddir], *pos); \ + return seq_list_start(&dd->fifo_list[prio][data_dir], *pos); \ } \ \ static void *deadline_##name##_fifo_next(struct seq_file *m, void *v, \ @@ -725,7 +819,7 @@ static void *deadline_##name##_fifo_next(struct seq_file *m, void *v, \ struct request_queue *q = m->private; \ struct deadline_data *dd = q->elevator->elevator_data; \ \ - return seq_list_next(v, &dd->fifo_list[ddir], pos); \ + return seq_list_next(v, &dd->fifo_list[prio][data_dir], pos); \ } \ \ static void deadline_##name##_fifo_stop(struct seq_file *m, void *v) \ @@ -742,22 +836,31 @@ static const struct seq_operations deadline_##name##_fifo_seq_ops = { \ .next = deadline_##name##_fifo_next, \ .stop = deadline_##name##_fifo_stop, \ .show = blk_mq_debugfs_rq_show, \ -}; \ - \ +}; + +#define DEADLINE_DEBUGFS_NEXT_RQ(data_dir, name) \ static int deadline_##name##_next_rq_show(void *data, \ struct seq_file *m) \ { \ struct request_queue *q = data; \ struct deadline_data *dd = q->elevator->elevator_data; \ - struct request *rq = dd->next_rq[ddir]; \ + struct request *rq = dd->next_rq[data_dir]; \ \ if (rq) \ __blk_mq_debugfs_rq_show(m, rq); \ return 0; \ } -DEADLINE_DEBUGFS_DDIR_ATTRS(DD_READ, read) -DEADLINE_DEBUGFS_DDIR_ATTRS(DD_WRITE, write) + +DEADLINE_DEBUGFS_DDIR_ATTRS(DD_RT_PRIO, DD_READ, read0) +DEADLINE_DEBUGFS_DDIR_ATTRS(DD_RT_PRIO, DD_WRITE, write0) +DEADLINE_DEBUGFS_DDIR_ATTRS(DD_BE_PRIO, DD_READ, read1) +DEADLINE_DEBUGFS_DDIR_ATTRS(DD_BE_PRIO, DD_WRITE, write1) +DEADLINE_DEBUGFS_DDIR_ATTRS(DD_IDLE_PRIO, DD_READ, read2) +DEADLINE_DEBUGFS_DDIR_ATTRS(DD_IDLE_PRIO, DD_WRITE, write2) +DEADLINE_DEBUGFS_NEXT_RQ(DD_READ, read) +DEADLINE_DEBUGFS_NEXT_RQ(DD_WRITE, write) #undef DEADLINE_DEBUGFS_DDIR_ATTRS +#undef DEADLINE_DEBUGFS_NEXT_RQ static int deadline_batching_show(void *data, struct seq_file *m) { @@ -786,50 +889,100 @@ static int dd_async_depth_show(void *data, struct seq_file *m) return 0; } -static void *deadline_dispatch_start(struct seq_file *m, loff_t *pos) - __acquires(&dd->lock) +static int dd_queued_show(void *data, struct seq_file *m) { - struct request_queue *q = m->private; + struct request_queue *q = data; struct deadline_data *dd = q->elevator->elevator_data; - spin_lock(&dd->lock); - return seq_list_start(&dd->dispatch, *pos); + seq_printf(m, "%llu %llu %llu\n", dd_queued(dd, DD_RT_PRIO), + dd_queued(dd, DD_BE_PRIO), + dd_queued(dd, DD_IDLE_PRIO)); + return 0; } -static void *deadline_dispatch_next(struct seq_file *m, void *v, loff_t *pos) +/* Number of requests owned by the block driver for a given priority. */ +static u64 dd_owned_by_driver(struct deadline_data *dd, enum dd_prio prio) { - struct request_queue *q = m->private; - struct deadline_data *dd = q->elevator->elevator_data; - - return seq_list_next(v, &dd->dispatch, pos); + return atomic_read(&dd->dispatched[prio]) + + atomic_read(&dd->merged[prio]) - + atomic_read(&dd->completed[prio]); } -static void deadline_dispatch_stop(struct seq_file *m, void *v) - __releases(&dd->lock) +static int dd_owned_by_driver_show(void *data, struct seq_file *m) { - struct request_queue *q = m->private; + struct request_queue *q = data; struct deadline_data *dd = q->elevator->elevator_data; - spin_unlock(&dd->lock); + seq_printf(m, "%llu %llu %llu\n", dd_owned_by_driver(dd, DD_RT_PRIO), + dd_owned_by_driver(dd, DD_BE_PRIO), + dd_owned_by_driver(dd, DD_IDLE_PRIO)); + return 0; } -static const struct seq_operations deadline_dispatch_seq_ops = { - .start = deadline_dispatch_start, - .next = deadline_dispatch_next, - .stop = deadline_dispatch_stop, - .show = blk_mq_debugfs_rq_show, +#define DEADLINE_DISPATCH_ATTR(prio) \ +static void *deadline_dispatch##prio##_start(struct seq_file *m, \ + loff_t *pos) \ + __acquires(&dd->lock) \ +{ \ + struct request_queue *q = m->private; \ + struct deadline_data *dd = q->elevator->elevator_data; \ + \ + spin_lock(&dd->lock); \ + return seq_list_start(&dd->dispatch[prio], *pos); \ +} \ + \ +static void *deadline_dispatch##prio##_next(struct seq_file *m, \ + void *v, loff_t *pos) \ +{ \ + struct request_queue *q = m->private; \ + struct deadline_data *dd = q->elevator->elevator_data; \ + \ + return seq_list_next(v, &dd->dispatch[prio], pos); \ +} \ + \ +static void deadline_dispatch##prio##_stop(struct seq_file *m, void *v) \ + __releases(&dd->lock) \ +{ \ + struct request_queue *q = m->private; \ + struct deadline_data *dd = q->elevator->elevator_data; \ + \ + spin_unlock(&dd->lock); \ +} \ + \ +static const struct seq_operations deadline_dispatch##prio##_seq_ops = { \ + .start = deadline_dispatch##prio##_start, \ + .next = deadline_dispatch##prio##_next, \ + .stop = deadline_dispatch##prio##_stop, \ + .show = blk_mq_debugfs_rq_show, \ }; -#define DEADLINE_QUEUE_DDIR_ATTRS(name) \ - {#name "_fifo_list", 0400, .seq_ops = &deadline_##name##_fifo_seq_ops}, \ +DEADLINE_DISPATCH_ATTR(0); +DEADLINE_DISPATCH_ATTR(1); +DEADLINE_DISPATCH_ATTR(2); +#undef DEADLINE_DISPATCH_ATTR + +#define DEADLINE_QUEUE_DDIR_ATTRS(name) \ + {#name "_fifo_list", 0400, \ + .seq_ops = &deadline_##name##_fifo_seq_ops} +#define DEADLINE_NEXT_RQ_ATTR(name) \ {#name "_next_rq", 0400, deadline_##name##_next_rq_show} static const struct blk_mq_debugfs_attr deadline_queue_debugfs_attrs[] = { - DEADLINE_QUEUE_DDIR_ATTRS(read), - DEADLINE_QUEUE_DDIR_ATTRS(write), + DEADLINE_QUEUE_DDIR_ATTRS(read0), + DEADLINE_QUEUE_DDIR_ATTRS(write0), + DEADLINE_QUEUE_DDIR_ATTRS(read1), + DEADLINE_QUEUE_DDIR_ATTRS(write1), + DEADLINE_QUEUE_DDIR_ATTRS(read2), + DEADLINE_QUEUE_DDIR_ATTRS(write2), + DEADLINE_NEXT_RQ_ATTR(read), + DEADLINE_NEXT_RQ_ATTR(write), {"batching", 0400, deadline_batching_show}, {"starved", 0400, deadline_starved_show}, {"async_depth", 0400, dd_async_depth_show}, - {"dispatch", 0400, .seq_ops = &deadline_dispatch_seq_ops}, + {"dispatch0", 0400, .seq_ops = &deadline_dispatch0_seq_ops}, + {"dispatch1", 0400, .seq_ops = &deadline_dispatch1_seq_ops}, + {"dispatch2", 0400, .seq_ops = &deadline_dispatch2_seq_ops}, + {"owned_by_driver", 0400, dd_owned_by_driver_show}, + {"queued", 0400, dd_queued_show}, {}, }; #undef DEADLINE_QUEUE_DDIR_ATTRS @@ -879,6 +1032,6 @@ static void __exit deadline_exit(void) module_init(deadline_init); module_exit(deadline_exit); -MODULE_AUTHOR("Jens Axboe"); +MODULE_AUTHOR("Jens Axboe, Damien Le Moal and Bart Van Assche"); MODULE_LICENSE("GPL"); MODULE_DESCRIPTION("MQ deadline IO scheduler"); From patchwork Thu May 27 01:01:34 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bart Van Assche X-Patchwork-Id: 12283121 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B98C6C47082 for ; Thu, 27 May 2021 01:02:03 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 99FFF6128B for ; Thu, 27 May 2021 01:02:03 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233817AbhE0BDe (ORCPT ); Wed, 26 May 2021 21:03:34 -0400 Received: from mail-pg1-f169.google.com ([209.85.215.169]:35362 "EHLO mail-pg1-f169.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233592AbhE0BDe (ORCPT ); Wed, 26 May 2021 21:03:34 -0400 Received: by mail-pg1-f169.google.com with SMTP id m190so2425620pga.2 for ; Wed, 26 May 2021 18:02:01 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=RoFE4Kf4cG+tMimiDO0vvQxxNfLoJUV32uixhmxPv1M=; b=HKmnfXrXpDCdxaHZLs16uIF5hP4dzAYI7P7K6Jqy/jnivkoSyWPOV6aaw7oaWRO88T nvaDDiUo690IszuwtyUymcWf7JzbdBab4vz6rFwj7STbMxkgRr5ixClcVKMBpoTG6op7 AlzYwVZ2w7oJNngOvvT1l7ES0BOlAOi6hAxvK5FKjDiQmMmziHB4PBgAnTyeTIjYZFf2 kTLe0T60PYB6Syr8GUDTy8DBADSMhcd5E8Ykq+5ggE4vvgETuFfhEC3PWFS+tGI6/+sZ wqWNKm1p/7GfIul+EVvoK6F60kJRM/UiJqtaFKb7TsjZ4K4gnbwT1uox2WLjeI3+dZ7M hSCA== X-Gm-Message-State: AOAM530oHsAragkBNv3EeghIrlqwmPHqKmjv2YI7+F0Q40Xhw8JRzQcp 4IALHAK1REd9MqLMJRbW+vE= X-Google-Smtp-Source: ABdhPJzthpcWHxisIl+f/FIO0hpmZOPLBCIbhG0BrIAIUDGOfjzyrBj4DNaKd2qR9DTrOKXSs3faRg== X-Received: by 2002:a65:4887:: with SMTP id n7mr1267012pgs.284.1622077320897; Wed, 26 May 2021 18:02:00 -0700 (PDT) Received: from asus.hsd1.ca.comcast.net (c-73-241-217-19.hsd1.ca.comcast.net. [73.241.217.19]) by smtp.gmail.com with ESMTPSA id j22sm310707pfd.215.2021.05.26.18.01.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 26 May 2021 18:02:00 -0700 (PDT) From: Bart Van Assche To: Jens Axboe Cc: linux-block@vger.kernel.org, Christoph Hellwig , Jaegeuk Kim , Adam Manzanares , Bart Van Assche , Damien Le Moal , Ming Lei Subject: [PATCH 9/9] block/mq-deadline: Add cgroup support Date: Wed, 26 May 2021 18:01:34 -0700 Message-Id: <20210527010134.32448-10-bvanassche@acm.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210527010134.32448-1-bvanassche@acm.org> References: <20210527010134.32448-1-bvanassche@acm.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org Add support for configuring I/O priority per block cgroup. Assign the lowest of the following two priorities to a request: rq->ioprio and blkcg->ioprio. Maintain statistics per cgroup and make these available in sysfs. This patch has been tested as follows: SHELL 1 modprobe scsi_debug ndelay=1000000 max_queue=16 && while [ -z "$sd" ]; do sd=/dev/$(basename /sys/bus/pseudo/drivers/scsi_debug/adapter*/host*/target*/*/block/*); done && cd /sys/fs/cgroup/blkio/ && echo 2 >blkio.dd.prio && mkdir -p hipri && cd hipri && echo 1 >blkio.dd.prio && echo $$ >cgroup.procs && max-iops -a1 -d32 -j1 -e mq-deadline $sd SHELL 2 sd=/dev/$(basename /sys/bus/pseudo/drivers/scsi_debug/adapter*/host*/target*/*/block/*) && max-iops -a1 -d32 -j1 -e mq-deadline $sd Result: * 12000 IOPS in shell 1 * 2000 IOPS in shell 2 The max-iops script is a script that runs fio with the following arguments: --bs=4K --gtod_reduce=1 --ioengine=libaio --ioscheduler=${arg_e} --runtime=60 --norandommap --rw=read --thread --buffered=0 --numjobs=${arg_j} --iodepth=${arg_d} --iodepth_batch_submit=${arg_a} --iodepth_batch_complete=$((arg_d / 2)) --name=${positional_argument_1} --filename=${positional_argument_1} Cc: Damien Le Moal Cc: Christoph Hellwig Cc: Ming Lei Signed-off-by: Bart Van Assche --- block/Kconfig.iosched | 6 ++ block/Makefile | 1 + block/mq-deadline-cgroup.c | 206 +++++++++++++++++++++++++++++++++++++ block/mq-deadline-cgroup.h | 73 +++++++++++++ block/mq-deadline.c | 96 ++++++++++++++--- 5 files changed, 370 insertions(+), 12 deletions(-) create mode 100644 block/mq-deadline-cgroup.c create mode 100644 block/mq-deadline-cgroup.h diff --git a/block/Kconfig.iosched b/block/Kconfig.iosched index 2f2158e05a91..64053d67a97b 100644 --- a/block/Kconfig.iosched +++ b/block/Kconfig.iosched @@ -9,6 +9,12 @@ config MQ_IOSCHED_DEADLINE help MQ version of the deadline IO scheduler. +config MQ_IOSCHED_DEADLINE_CGROUP + tristate + default y + depends on MQ_IOSCHED_DEADLINE + depends on BLK_CGROUP + config MQ_IOSCHED_KYBER tristate "Kyber I/O scheduler" default y diff --git a/block/Makefile b/block/Makefile index 8d841f5f986f..7d5763d5f988 100644 --- a/block/Makefile +++ b/block/Makefile @@ -20,6 +20,7 @@ obj-$(CONFIG_BLK_DEV_THROTTLING) += blk-throttle.o obj-$(CONFIG_BLK_CGROUP_IOLATENCY) += blk-iolatency.o obj-$(CONFIG_BLK_CGROUP_IOCOST) += blk-iocost.o obj-$(CONFIG_MQ_IOSCHED_DEADLINE) += mq-deadline.o +obj-$(CONFIG_MQ_IOSCHED_DEADLINE_CGROUP)+= mq-deadline-cgroup.o obj-$(CONFIG_MQ_IOSCHED_KYBER) += kyber-iosched.o bfq-y := bfq-iosched.o bfq-wf2q.o bfq-cgroup.o obj-$(CONFIG_IOSCHED_BFQ) += bfq.o diff --git a/block/mq-deadline-cgroup.c b/block/mq-deadline-cgroup.c new file mode 100644 index 000000000000..3983795dda60 --- /dev/null +++ b/block/mq-deadline-cgroup.c @@ -0,0 +1,206 @@ +// SPDX-License-Identifier: GPL-2.0 + +#include +#include + +#include "mq-deadline-cgroup.h" + +static struct blkcg_policy dd_blkcg_policy; + +static struct dd_blkcg *dd_blkcg_from_css(struct cgroup_subsys_state *css) +{ + struct blkcg *blkcg = css_to_blkcg(css); + struct blkcg_policy_data *cpd = blkcg_to_cpd(blkcg, &dd_blkcg_policy); + + return container_of(cpd, struct dd_blkcg, cpd); +} + +static u64 dd_show_prio(struct cgroup_subsys_state *css, struct cftype *cft) +{ + return dd_blkcg_from_css(css)->prio; +} + +static int dd_set_prio(struct cgroup_subsys_state *css, struct cftype *cft, + u64 val) +{ + struct dd_blkcg *dd_blkcg = dd_blkcg_from_css(css); + + if (val < 0 || val > 3) + return -EINVAL; + dd_blkcg->prio = val; + return 0; +} + +/* + * Total number of inserted requests not associated with any cgroup and + * with I/O priority cft->private. + */ +static u64 dd_show_nocg(struct cgroup_subsys_state *css, + struct cftype *cft) +{ + const u8 prio = cft->private; + + return atomic_read(&nocg_inserted[prio]); +} + +/* Number of inserted but not completed requests for priority cft->private. */ +static u64 dd_show_inserted(struct cgroup_subsys_state *css, + struct cftype *cft) +{ + struct dd_blkcg *blkcg = dd_blkcg_from_css(css); + const u8 prio = cft->private; + + return atomic_read(&blkcg->inserted[prio]) - + atomic_read(&blkcg->completed[prio]); +} + +static u64 dd_show_merged(struct cgroup_subsys_state *css, struct cftype *cft) +{ + struct dd_blkcg *blkcg = dd_blkcg_from_css(css); + const u8 prio = cft->private; + + return atomic_read(&blkcg->merged[prio]); +} + +/* Number of dispatched but not completed requests for priority cft->private. */ +static u64 dd_show_dispatched(struct cgroup_subsys_state *css, + struct cftype *cft) +{ + struct dd_blkcg *blkcg = dd_blkcg_from_css(css); + const u8 prio = cft->private; + + return atomic_read(&blkcg->dispatched[prio]) + + atomic_read(&blkcg->merged[prio]) - + atomic_read(&blkcg->completed[prio]); +} + +#define IOPRI_ATTRS(pfx, cft_flags, callback) \ + { \ + .name = pfx "0", \ + .private = IOPRIO_CLASS_NONE, \ + .flags = cft_flags, \ + .read_u64 = callback, \ + }, \ + { \ + .name = pfx "1", \ + .private = IOPRIO_CLASS_RT, \ + .flags = cft_flags, \ + .read_u64 = callback, \ + }, \ + { \ + .name = pfx "2", \ + .private = IOPRIO_CLASS_BE, \ + .flags = cft_flags, \ + .read_u64 = callback, \ + }, \ + { \ + .name = pfx "3", \ + .private = IOPRIO_CLASS_IDLE, \ + .flags = cft_flags, \ + .read_u64 = callback, \ + } + +#define DD_ATTRS \ + { \ + .name = "dd.prio", \ + .read_u64 = dd_show_prio, \ + .write_u64 = dd_set_prio, \ + }, \ + IOPRI_ATTRS("dd.nocg_pri", CFTYPE_ONLY_ON_ROOT, dd_show_nocg), \ + IOPRI_ATTRS("dd.inserted_pri", 0, dd_show_inserted), \ + IOPRI_ATTRS("dd.merged_pri", 0, dd_show_merged), \ + IOPRI_ATTRS("dd.dispatched_pri", 0, dd_show_dispatched), \ + { } /* sentinel */ + +/* cgroup v2 attributes */ +static struct cftype dd_blkcg_files[] = { + DD_ATTRS +}; + +/* cgroup v1 attributes */ +static struct cftype dd_blkcg_legacy_files[] = { + DD_ATTRS +}; + +static struct blkcg_policy_data *dd_cpd_alloc(gfp_t gfp) +{ + struct dd_blkcg *pd; + + pd = kzalloc(sizeof(*pd), gfp); + if (!pd) + return NULL; + pd->prio = IOPRIO_CLASS_NONE; + return &pd->cpd; +} + +static void dd_cpd_free(struct blkcg_policy_data *cpd) +{ + struct dd_blkcg *dd_blkcg = container_of(cpd, typeof(*dd_blkcg), cpd); + + kfree(dd_blkcg); +} + +/* + * Convert an association between a block cgroup and a request queue into a + * pointer to the mq-deadline information associated with a (blkcg, queue) pair. + */ +struct dd_blkcg *dd_blkcg_from_bio(struct bio *bio) +{ + struct blkg_policy_data *pd; + + pd = blkg_to_pd(bio->bi_blkg, &dd_blkcg_policy); + if (!pd) + return NULL; + + return container_of(blkcg_to_cpd(pd->blkg->blkcg, &dd_blkcg_policy), + struct dd_blkcg, cpd); +} + +static struct blkg_policy_data *dd_pd_alloc(gfp_t gfp, struct request_queue *q, + struct blkcg *blkcg) +{ + struct dd_blkg *pd; + + pd = kzalloc(sizeof(*pd), gfp); + if (!pd) + return NULL; + return &pd->pd; +} + +static void dd_pd_free(struct blkg_policy_data *pd) +{ + struct dd_blkg *dd_blkg = container_of(pd, typeof(*dd_blkg), pd); + + kfree(dd_blkg); +} + +static struct blkcg_policy dd_blkcg_policy = { + .dfl_cftypes = dd_blkcg_files, + .legacy_cftypes = dd_blkcg_legacy_files, + + .cpd_alloc_fn = dd_cpd_alloc, + .cpd_free_fn = dd_cpd_free, + + .pd_alloc_fn = dd_pd_alloc, + .pd_free_fn = dd_pd_free, +}; + +int dd_activate_policy(struct request_queue *q) +{ + return blkcg_activate_policy(q, &dd_blkcg_policy); +} + +void dd_deactivate_policy(struct request_queue *q) +{ + blkcg_deactivate_policy(q, &dd_blkcg_policy); +} + +int __init dd_blkcg_init(void) +{ + return blkcg_policy_register(&dd_blkcg_policy); +} + +void __exit dd_blkcg_exit(void) +{ + blkcg_policy_unregister(&dd_blkcg_policy); +} diff --git a/block/mq-deadline-cgroup.h b/block/mq-deadline-cgroup.h new file mode 100644 index 000000000000..0996a1a28412 --- /dev/null +++ b/block/mq-deadline-cgroup.h @@ -0,0 +1,73 @@ +/* SPDX-License-Identifier: GPL-2.0 */ + +#if !defined(_MQ_DEADLINE_CGROUP_H_) +#define _MQ_DEADLINE_CGROUP_H_ + +#include + +struct request_queue; + +extern atomic_t nocg_inserted[4]; + +/** + * struct dd_blkcg - Per cgroup data. + * @cpd: blkcg_policy_data structure. + * @prio: One of the IOPRIO_CLASS_* values. See also . + * @inserted: Number of inserted requests for the given IOPRIO_CLASS_*. + * @merged: Number of merged requests for the given I/O priority level. + * @dispatched: Number of dispatched requests for the given IOPRIO_CLASS_*. + * @completed: Number of completions per I/O priority level. + */ +struct dd_blkcg { + struct blkcg_policy_data cpd; /* must be the first member */ + u8 prio; + atomic_t inserted[4]; + atomic_t merged[4]; + atomic_t dispatched[4]; + atomic_t completed[4]; +}; + +#ifdef CONFIG_BLK_CGROUP + +/** + * struct dd_blkg - Per (cgroup, request queue) data. + * @pd: blkg_policy_data structure. + */ +struct dd_blkg { + struct blkg_policy_data pd; /* must be the first member */ +}; + +struct dd_blkcg *dd_blkcg_from_bio(struct bio *bio); +int dd_activate_policy(struct request_queue *q); +void dd_deactivate_policy(struct request_queue *q); +int __init dd_blkcg_init(void); +void __exit dd_blkcg_exit(void); + +#else /* CONFIG_BLK_CGROUP */ + +static inline struct dd_blkcg *dd_blkcg_from_bio(struct bio *bio) +{ + return NULL; +} + +static inline int dd_activate_policy(struct request_queue *q) +{ + return 0; +} + +static inline void dd_deactivate_policy(struct request_queue *q) +{ +} + +static inline int dd_blkcg_init(void) +{ + return 0; +} + +static inline void dd_blkcg_exit(void) +{ +} + +#endif /* CONFIG_BLK_CGROUP */ + +#endif /* _MQ_DEADLINE_CGROUP_H_ */ diff --git a/block/mq-deadline.c b/block/mq-deadline.c index 5a703e1228fa..0aa14cc302fd 100644 --- a/block/mq-deadline.c +++ b/block/mq-deadline.c @@ -25,6 +25,7 @@ #include "blk-mq-debugfs.h" #include "blk-mq-tag.h" #include "blk-mq-sched.h" +#include "mq-deadline-cgroup.h" /* * See Documentation/block/deadline-iosched.rst @@ -58,6 +59,9 @@ struct deadline_data { * run time data */ + /* Request queue that owns this data structure. */ + struct request_queue *queue; + /* * Requests are present on both sort_list[] and fifo_list[][]. The * first index of fifo_list[][] is the I/O priority class (DD_*_PRIO). @@ -101,6 +105,9 @@ static const enum dd_prio ioprio_class_to_prio[] = { [IOPRIO_CLASS_IDLE] = DD_IDLE_PRIO, }; +/* To track requests not associated with any cgroup. */ +atomic_t nocg_inserted[4]; + static inline struct rb_root * deadline_rb_root(struct deadline_data *dd, struct request *rq) { @@ -189,9 +196,11 @@ static void dd_merged_requests(struct request_queue *q, struct request *req, struct deadline_data *dd = q->elevator->elevator_data; const u8 ioprio_class = dd_rq_ioclass(next); const enum dd_prio prio = ioprio_class_to_prio[ioprio_class]; + struct dd_blkcg *blkcg = next->elv.priv[0]; - if (next->elv.priv[0]) { + if (blkcg) { atomic_inc(&dd->merged[prio]); + atomic_inc(&blkcg->merged[ioprio_class]); } else { WARN_ON_ONCE(true); } @@ -334,6 +343,7 @@ static struct request *__dd_dispatch_request(struct deadline_data *dd, { struct request *rq, *next_rq; enum dd_data_dir data_dir; + struct dd_blkcg *blkcg; u8 ioprio_class; lockdep_assert_held(&dd->lock); @@ -426,10 +436,12 @@ static struct request *__dd_dispatch_request(struct deadline_data *dd, dd->batching++; deadline_move_request(dd, rq); done: + blkcg = rq->elv.priv[0]; ioprio_class = dd_rq_ioclass(rq); prio = ioprio_class_to_prio[ioprio_class]; - if (rq->elv.priv[0]) { + if (blkcg) { atomic_inc(&dd->dispatched[prio]); + atomic_inc(&blkcg->dispatched[ioprio_class]); } else { WARN_ON_ONCE(true); } @@ -503,6 +515,8 @@ static void dd_exit_sched(struct elevator_queue *e) struct deadline_data *dd = e->elevator_data; enum dd_prio prio; + dd_deactivate_policy(dd->queue); + for (prio = 0; prio <= DD_PRIO_MAX; prio++) { WARN_ON_ONCE(!list_empty(&dd->fifo_list[prio][DD_READ])); WARN_ON_ONCE(!list_empty(&dd->fifo_list[prio][DD_WRITE])); @@ -512,25 +526,33 @@ static void dd_exit_sched(struct elevator_queue *e) } /* - * initialize elevator private data (deadline_data). + * Initialize elevator private data (deadline_data) and associate with blkcg. */ static int dd_init_sched(struct request_queue *q, struct elevator_type *e) { struct deadline_data *dd; struct elevator_queue *eq; enum dd_prio prio; + int ret = -ENOMEM; + + /* + * Initialization would be very tricky if the queue is not frozen, + * hence the warning statement below. + */ + WARN_ON_ONCE(!percpu_ref_is_zero(&q->q_usage_counter)); eq = elevator_alloc(q, e); if (!eq) - return -ENOMEM; + goto out; dd = kzalloc_node(sizeof(*dd), GFP_KERNEL, q->node); - if (!dd) { - kobject_put(&eq->kobj); - return -ENOMEM; - } + if (!dd) + goto put_eq; + eq->elevator_data = dd; + dd->queue = q; + for (prio = 0; prio <= DD_PRIO_MAX; prio++) { INIT_LIST_HEAD(&dd->fifo_list[prio][DD_READ]); INIT_LIST_HEAD(&dd->fifo_list[prio][DD_WRITE]); @@ -547,8 +569,23 @@ static int dd_init_sched(struct request_queue *q, struct elevator_type *e) spin_lock_init(&dd->lock); spin_lock_init(&dd->zone_lock); + ret = dd_activate_policy(q); + if (ret) + goto free_dd; + + ret = 0; q->elevator = eq; - return 0; + +out: + return ret; + +free_dd: + eq->elevator_data = NULL; + kfree(dd); + +put_eq: + kobject_put(&eq->kobj); + goto out; } /* @@ -611,6 +648,7 @@ static void dd_insert_request(struct blk_mq_hw_ctx *hctx, struct request *rq, u16 ioprio = req_get_ioprio(rq); u8 ioprio_class = IOPRIO_PRIO_CLASS(ioprio); enum dd_prio prio; + struct dd_blkcg *blkcg; lockdep_assert_held(&dd->lock); @@ -620,10 +658,27 @@ static void dd_insert_request(struct blk_mq_hw_ctx *hctx, struct request *rq, */ blk_req_zone_write_unlock(rq); + /* + * If a block cgroup has been associated with the submitter and if an + * I/O priority has been set in the associated block cgroup, use the + * lowest of the cgroup priority and the request priority for the + * request. If no priority has been set in the request, use the cgroup + * priority. + */ + blkcg = dd_blkcg_from_bio(rq->bio); + if (blkcg) { + ioprio_class = max(ioprio_class, blkcg->prio); + rq->ioprio = IOPRIO_PRIO_VALUE(ioprio_class, + IOPRIO_PRIO_DATA(ioprio)); + atomic_inc(&blkcg->inserted[ioprio_class]); + } else { + WARN_ON_ONCE(true); + atomic_inc(&nocg_inserted[ioprio_class]); + } prio = ioprio_class_to_prio[ioprio_class]; atomic_inc(&dd->inserted[prio]); WARN_ON_ONCE(rq->elv.priv[0]); - rq->elv.priv[0] = (void *)1ULL; + rq->elv.priv[0] = blkcg; rq->elv.priv[1] = (void *)(uintptr_t)ioprio_class; if (blk_mq_sched_try_insert_merge(q, rq)) @@ -696,11 +751,14 @@ static void dd_finish_request(struct request *rq) { struct request_queue *q = rq->q; struct deadline_data *dd = q->elevator->elevator_data; + struct dd_blkcg *blkcg = rq->elv.priv[0]; const u8 ioprio_class = dd_rq_ioclass(rq); const enum dd_prio prio = ioprio_class_to_prio[ioprio_class]; - if (rq->elv.priv[0]) + if (blkcg) { atomic_inc(&dd->completed[prio]); + atomic_inc(&blkcg->completed[ioprio_class]); + } if (blk_queue_is_zoned(q)) { unsigned long flags; @@ -1021,11 +1079,25 @@ MODULE_ALIAS("mq-deadline-iosched"); static int __init deadline_init(void) { - return elv_register(&mq_deadline); + int ret; + ret = elv_register(&mq_deadline); + if (ret) + goto out; + ret = dd_blkcg_init(); + if (ret) + goto unreg; + +out: + return ret; + +unreg: + elv_unregister(&mq_deadline); + goto out; } static void __exit deadline_exit(void) { + dd_blkcg_exit(); elv_unregister(&mq_deadline); }