From patchwork Mon Jan 25 19:02:43 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Paolo Valente X-Patchwork-Id: 12045467 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3F934C433DB for ; Tue, 26 Jan 2021 05:33:31 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 0EE4F229C4 for ; Tue, 26 Jan 2021 05:33:31 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2387767AbhAZFdZ (ORCPT ); Tue, 26 Jan 2021 00:33:25 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48220 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1731285AbhAYTD7 (ORCPT ); Mon, 25 Jan 2021 14:03:59 -0500 Received: from mail-wr1-x42c.google.com (mail-wr1-x42c.google.com [IPv6:2a00:1450:4864:20::42c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7A593C061756 for ; Mon, 25 Jan 2021 11:03:16 -0800 (PST) Received: by mail-wr1-x42c.google.com with SMTP id d16so13419304wro.11 for ; Mon, 25 Jan 2021 11:03:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=ZgGFbvXxHvS5eUAN/hln5OKJlAJbciVhefRf0mMTswU=; b=KjoB8VpoBeqeCFf+1js5OWLn3XeZLjEhCQcs7QPGEfZdwqHQ0iMln4FfmlXmM/dYdw Od+jRiFZo7ddkhBE8ByTSWYCVO8Fo84QHRIuAkBIp9XJpb4dqrxVNOgMECme+VNZ8s7P p4aVOdjjCtBcMXs/Zy4+fhKkBJgEpyu7bM8cSAupcricts2FrbLFeCT16jTjUeAutXcq NYcUx+UDSAqXkI/itiY5J7zhUUPB8IvJFqN73NkEjkQr3/W9CjDjSq7yrNn0qqwgyzey SypkN40Dq8GyymVh7g12ZbEEUTV7fYeK8BZJ/PBWHtqCKIueciFmbkiQCBu/noHV96aO 7PaA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=ZgGFbvXxHvS5eUAN/hln5OKJlAJbciVhefRf0mMTswU=; b=U3zigaSYQ063/LysqziWHdbxthLOP3256xT3ewKRVameVfOS7ffOoQuOrOI5QK84B6 AtFSnsEPTBf8jwkBe5/PR53QRX2g2+0Ct3tmzpC248XEc21sEa1BpG1lGyjcwDJ9WkPF w5VaDIqqJaDFsRg/c5vh5tdHAF9VndOJXDPfKzU96tlfpFmnJ49QrvpIMK0pgAi9ygib DxR3Uz2ZHxy4oZM+tF8Ks1Gib15bITd32hfIpXvCMyz8Xn6BfR+DTkTEwWUab5xOU7bu ndyjM6i1mS6DpxvYqi791f84aJZTVRMkBmb5J0kCTChIFih2AncF9ftgewKoeM9qxSJE pUEQ== X-Gm-Message-State: AOAM531sNov2lwZfYpNzOIk9rr5JEUASWVM8uwMGLMQXfHQXUFyeXrDJ hXkNJSh6oxUiyJAdV8u4SWpjiw== X-Google-Smtp-Source: ABdhPJwUWAsFf6dK2EcfP+GGoeBvK78AUFHELoUYX26fRo4bwu35ni+MdVc7neA/Qw7dzWF+RzE3UA== X-Received: by 2002:adf:decb:: with SMTP id i11mr1213090wrn.78.1611601395112; Mon, 25 Jan 2021 11:03:15 -0800 (PST) Received: from localhost.localdomain ([37.160.159.175]) by smtp.gmail.com with ESMTPSA id g194sm222534wme.39.2021.01.25.11.03.13 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Mon, 25 Jan 2021 11:03:14 -0800 (PST) From: Paolo Valente To: Jens Axboe Cc: linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, Paolo Valente , Jan Kara Subject: [PATCH BUGFIX/IMPROVEMENT 1/6] block, bfq: replace mechanism for evaluating I/O intensity Date: Mon, 25 Jan 2021 20:02:43 +0100 Message-Id: <20210125190248.49338-2-paolo.valente@linaro.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210125190248.49338-1-paolo.valente@linaro.org> References: <20210125190248.49338-1-paolo.valente@linaro.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org Some BFQ mechanisms make their decisions on a bfq_queue basing also on whether the bfq_queue is I/O bound. In this respect, the current logic for evaluating whether a bfq_queue is I/O bound is rather rough. This commits replaces this logic with a more effective one. The new logic measures the percentage of time during which a bfq_queue is active, and marks the bfq_queue as I/O bound if the latter if this percentage is above a fixed threshold. Tested-by: Jan Kara Signed-off-by: Paolo Valente --- block/bfq-iosched.c | 63 +++++++++++++++++++++++++++++++-------------- block/bfq-iosched.h | 16 ++++++------ 2 files changed, 52 insertions(+), 27 deletions(-) diff --git a/block/bfq-iosched.c b/block/bfq-iosched.c index c045613ce927..db393f5d70ba 100644 --- a/block/bfq-iosched.c +++ b/block/bfq-iosched.c @@ -1026,6 +1026,8 @@ bfq_bfqq_resume_state(struct bfq_queue *bfqq, struct bfq_data *bfqd, bfqq->entity.new_weight = bic->saved_weight; bfqq->ttime = bic->saved_ttime; + bfqq->io_start_time = bic->saved_io_start_time; + bfqq->tot_idle_time = bic->saved_tot_idle_time; bfqq->wr_coeff = bic->saved_wr_coeff; bfqq->wr_start_at_switch_to_srt = bic->saved_wr_start_at_switch_to_srt; bfqq->last_wr_start_finish = bic->saved_last_wr_start_finish; @@ -1721,17 +1723,6 @@ static void bfq_bfqq_handle_idle_busy_switch(struct bfq_data *bfqd, bfq_clear_bfqq_just_created(bfqq); - - if (!bfq_bfqq_IO_bound(bfqq)) { - if (arrived_in_time) { - bfqq->requests_within_timer++; - if (bfqq->requests_within_timer >= - bfqd->bfq_requests_within_timer) - bfq_mark_bfqq_IO_bound(bfqq); - } else - bfqq->requests_within_timer = 0; - } - if (bfqd->low_latency) { if (unlikely(time_is_after_jiffies(bfqq->split_time))) /* wraparound */ @@ -1865,6 +1856,36 @@ static void bfq_reset_inject_limit(struct bfq_data *bfqd, bfqq->decrease_time_jif = jiffies; } +static void bfq_update_io_intensity(struct bfq_queue *bfqq, u64 now_ns) +{ + u64 tot_io_time = now_ns - bfqq->io_start_time; + + if (RB_EMPTY_ROOT(&bfqq->sort_list) && bfqq->dispatched == 0) + bfqq->tot_idle_time += + now_ns - bfqq->ttime.last_end_request; + + if (unlikely(bfq_bfqq_just_created(bfqq))) + return; + + /* + * Must be busy for at least about 80% of the time to be + * considered I/O bound. + */ + if (bfqq->tot_idle_time * 5 > tot_io_time) + bfq_clear_bfqq_IO_bound(bfqq); + else + bfq_mark_bfqq_IO_bound(bfqq); + + /* + * Keep an observation window of at most 200 ms in the past + * from now. + */ + if (tot_io_time > 200 * NSEC_PER_MSEC) { + bfqq->io_start_time = now_ns - (tot_io_time>>1); + bfqq->tot_idle_time >>= 1; + } +} + static void bfq_add_request(struct request *rq) { struct bfq_queue *bfqq = RQ_BFQQ(rq); @@ -1872,6 +1893,7 @@ static void bfq_add_request(struct request *rq) struct request *next_rq, *prev; unsigned int old_wr_coeff = bfqq->wr_coeff; bool interactive = false; + u64 now_ns = ktime_get_ns(); bfq_log_bfqq(bfqd, bfqq, "add_request %d", rq_is_sync(rq)); bfqq->queued[rq_is_sync(rq)]++; @@ -1934,7 +1956,7 @@ static void bfq_add_request(struct request *rq) */ if (bfqd->last_completed_rq_bfqq && !bfq_bfqq_has_short_ttime(bfqq) && - ktime_get_ns() - bfqd->last_completion < + now_ns - bfqd->last_completion < 4 * NSEC_PER_MSEC) { if (bfqd->last_completed_rq_bfqq != bfqq && bfqd->last_completed_rq_bfqq != @@ -2051,6 +2073,9 @@ static void bfq_add_request(struct request *rq) } } + if (bfq_bfqq_sync(bfqq)) + bfq_update_io_intensity(bfqq, now_ns); + elv_rb_add(&bfqq->sort_list, rq); /* @@ -2712,6 +2737,8 @@ static void bfq_bfqq_save_state(struct bfq_queue *bfqq) bic->saved_ttime = bfqq->ttime; bic->saved_has_short_ttime = bfq_bfqq_has_short_ttime(bfqq); bic->saved_IO_bound = bfq_bfqq_IO_bound(bfqq); + bic->saved_io_start_time = bfqq->io_start_time; + bic->saved_tot_idle_time = bfqq->tot_idle_time; bic->saved_in_large_burst = bfq_bfqq_in_large_burst(bfqq); bic->was_in_burst_list = !hlist_unhashed(&bfqq->burst_list_node); if (unlikely(bfq_bfqq_just_created(bfqq) && @@ -3979,10 +4006,6 @@ void bfq_bfqq_expire(struct bfq_data *bfqd, bfq_bfqq_budget_left(bfqq) >= entity->budget / 3))) bfq_bfqq_charge_time(bfqd, bfqq, delta); - if (reason == BFQQE_TOO_IDLE && - entity->service <= 2 * entity->budget / 10) - bfq_clear_bfqq_IO_bound(bfqq); - if (bfqd->low_latency && bfqq->wr_coeff == 1) bfqq->last_wr_start_finish = jiffies; @@ -5088,6 +5111,8 @@ static void bfq_check_ioprio_change(struct bfq_io_cq *bic, struct bio *bio) static void bfq_init_bfqq(struct bfq_data *bfqd, struct bfq_queue *bfqq, struct bfq_io_cq *bic, pid_t pid, int is_sync) { + u64 now_ns = ktime_get_ns(); + RB_CLEAR_NODE(&bfqq->entity.rb_node); INIT_LIST_HEAD(&bfqq->fifo); INIT_HLIST_NODE(&bfqq->burst_list_node); @@ -5115,7 +5140,9 @@ static void bfq_init_bfqq(struct bfq_data *bfqd, struct bfq_queue *bfqq, bfq_clear_bfqq_sync(bfqq); /* set end request to minus infinity from now */ - bfqq->ttime.last_end_request = ktime_get_ns() + 1; + bfqq->ttime.last_end_request = now_ns + 1; + + bfqq->io_start_time = now_ns; bfq_mark_bfqq_IO_bound(bfqq); @@ -6529,8 +6556,6 @@ static int bfq_init_queue(struct request_queue *q, struct elevator_type *e) bfqd->bfq_slice_idle = bfq_slice_idle; bfqd->bfq_timeout = bfq_timeout; - bfqd->bfq_requests_within_timer = 120; - bfqd->bfq_large_burst_thresh = 8; bfqd->bfq_burst_interval = msecs_to_jiffies(180); diff --git a/block/bfq-iosched.h b/block/bfq-iosched.h index 703895224562..c913b06016b3 100644 --- a/block/bfq-iosched.h +++ b/block/bfq-iosched.h @@ -291,6 +291,11 @@ struct bfq_queue { /* associated @bfq_ttime struct */ struct bfq_ttime ttime; + /* when bfqq started to do I/O within the last observation window */ + u64 io_start_time; + /* how long bfqq has remained empty during the last observ. window */ + u64 tot_idle_time; + /* bit vector: a 1 for each seeky requests in history */ u32 seek_history; @@ -407,6 +412,9 @@ struct bfq_io_cq { */ bool saved_IO_bound; + u64 saved_io_start_time; + u64 saved_tot_idle_time; + /* * Same purpose as the previous fields for the value of the * field keeping the queue's belonging to a large burst @@ -641,14 +649,6 @@ struct bfq_data { */ unsigned int bfq_timeout; - /* - * Number of consecutive requests that must be issued within - * the idle time slice to set again idling to a queue which - * was marked as non-I/O-bound (see the definition of the - * IO_bound flag for further details). - */ - unsigned int bfq_requests_within_timer; - /* * Force device idling whenever needed to provide accurate * service guarantees, without caring about throughput From patchwork Mon Jan 25 19:02:44 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Paolo Valente X-Patchwork-Id: 12045469 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id EE320C433E0 for ; Tue, 26 Jan 2021 05:33:44 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id B7C1F229C4 for ; Tue, 26 Jan 2021 05:33:44 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2387784AbhAZFdj (ORCPT ); Tue, 26 Jan 2021 00:33:39 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48450 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1731533AbhAYTFF (ORCPT ); Mon, 25 Jan 2021 14:05:05 -0500 Received: from mail-wr1-x42a.google.com (mail-wr1-x42a.google.com [IPv6:2a00:1450:4864:20::42a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B8422C0613ED for ; Mon, 25 Jan 2021 11:03:17 -0800 (PST) Received: by mail-wr1-x42a.google.com with SMTP id g10so14111059wrx.1 for ; Mon, 25 Jan 2021 11:03:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=9YzGYM8jUXZx1tyLF6Ot8at1yHr+fDjnAOr+cHSlKDo=; b=eKB/v1ZKYxKzCyfUQ3RlHEbASNKmHj/1ABogRXpO8XGKL7h6fVjTkJLMCrk+8Zv/1h 6eu1Vr68gXKGnjoaQfE2wGqpS2x6oGkoEBLpsvv8u65yqJ0vMIBk/b/saUKPxIeHdlXo kJs/jbX5n88ivf/e2JmAJB0GsuOlM18Htul9WHQDSPLTzNz6aJr9dvDlQ3ZVDBq8FOnX czXD/VVqqXQs5QiOXYKCwyMCOiwgYay2ahkX7aUhlZpbs2+vpBA4b4zKyrqs9t3Xx301 JsL0MHYmbNzLDGaTpHNXSHi8p9tjOEFxsKpvqthRk6kaFT3Ko2E2itMlpTQ6BR93q9Tm HR5A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=9YzGYM8jUXZx1tyLF6Ot8at1yHr+fDjnAOr+cHSlKDo=; b=SkgSexjm4bLGJhgfmwvmfYEX/9NJryGFmNfdHeeKMM2SVuXJePSu7apAUqfL9qQMxN n0UonjtdkcHEjO4+eohe9mtFSW/kuaIu83K2i9+9aUKY1HwpRQ1L8k/X4g0J+GZZMQce RmKgPjtrALbOWC2nMYUrCFcWgL24lCluwh5FQMDDx+WGIQu5jIf6evFbWfx/6cvN4fUk 7ysCjm5UFAVAhjL3Z/C8EsrOkMW8fVq9cIcuTLbqTmhgPPYN1jxyHIDV0BKEqjjH3kBP J9WJ/aMR1+WnRliC0qgomdpDk5gz3yYgddQEI19NZBzqQoDV2xC1kpVY5agUv7eVAqkX TbOw== X-Gm-Message-State: AOAM530PKrlMsmjo3nl31jh6FoOBRZYx0E4Y2FhZCJku+mn7ZErCHriK vJnTnwpE1KvHysnma1N2TeRubg== X-Google-Smtp-Source: ABdhPJzrKET5xztkr2KE3VrQH01OZZtc811nnF8Yy2xPDMMGnJnt/o4zYl+o47XMQccmdAoHDzHLdg== X-Received: by 2002:adf:f743:: with SMTP id z3mr2543929wrp.165.1611601396486; Mon, 25 Jan 2021 11:03:16 -0800 (PST) Received: from localhost.localdomain ([37.160.159.175]) by smtp.gmail.com with ESMTPSA id g194sm222534wme.39.2021.01.25.11.03.15 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Mon, 25 Jan 2021 11:03:15 -0800 (PST) From: Paolo Valente To: Jens Axboe Cc: linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, Paolo Valente , Jan Kara Subject: [PATCH BUGFIX/IMPROVEMENT 2/6] block, bfq: re-evaluate convenience of I/O plugging on rq arrivals Date: Mon, 25 Jan 2021 20:02:44 +0100 Message-Id: <20210125190248.49338-3-paolo.valente@linaro.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210125190248.49338-1-paolo.valente@linaro.org> References: <20210125190248.49338-1-paolo.valente@linaro.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org Upon an I/O-dispatch attempt, BFQ may detect that it was better to plug I/O dispatch, and to wait for a new request to arrive for the currently in-service queue. But the arrival of a new request for an empty bfq_queue, and thus the switch from idle to busy of the bfq_queue, may cause the scenario to change, and make plugging no longer needed for service guarantees, or more convenient for throughput. In this case, keeping I/O-dispatch plugged would certainly lower throughput. To address this issue, this commit makes such a check, and stops plugging I/O if it is better to stop plugging I/O. Tested-by: Jan Kara Signed-off-by: Paolo Valente --- block/bfq-iosched.c | 24 +++++++++++++++++++----- 1 file changed, 19 insertions(+), 5 deletions(-) diff --git a/block/bfq-iosched.c b/block/bfq-iosched.c index db393f5d70ba..6a02a12ff553 100644 --- a/block/bfq-iosched.c +++ b/block/bfq-iosched.c @@ -1649,6 +1649,8 @@ static bool bfq_bfqq_higher_class_or_weight(struct bfq_queue *bfqq, return bfqq_weight > in_serv_weight; } +static bool bfq_better_to_idle(struct bfq_queue *bfqq); + static void bfq_bfqq_handle_idle_busy_switch(struct bfq_data *bfqd, struct bfq_queue *bfqq, int old_wr_coeff, @@ -1750,10 +1752,10 @@ static void bfq_bfqq_handle_idle_busy_switch(struct bfq_data *bfqd, bfq_add_bfqq_busy(bfqd, bfqq); /* - * Expire in-service queue only if preemption may be needed - * for guarantees. In particular, we care only about two - * cases. The first is that bfqq has to recover a service - * hole, as explained in the comments on + * Expire in-service queue if preemption may be needed for + * guarantees or throughput. As for guarantees, we care + * explicitly about two cases. The first is that bfqq has to + * recover a service hole, as explained in the comments on * bfq_bfqq_update_budg_for_activation(), i.e., that * bfqq_wants_to_preempt is true. However, if bfqq does not * carry time-critical I/O, then bfqq's bandwidth is less @@ -1780,11 +1782,23 @@ static void bfq_bfqq_handle_idle_busy_switch(struct bfq_data *bfqd, * timestamps of the in-service queue would need to be * updated, and this operation is quite costly (see the * comments on bfq_bfqq_update_budg_for_activation()). + * + * As for throughput, we ask bfq_better_to_idle() whether we + * still need to plug I/O dispatching. If bfq_better_to_idle() + * says no, then plugging is not needed any longer, either to + * boost throughput or to perserve service guarantees. Then + * the best option is to stop plugging I/O, as not doing so + * would certainly lower throughput. We may end up in this + * case if: (1) upon a dispatch attempt, we detected that it + * was better to plug I/O dispatch, and to wait for a new + * request to arrive for the currently in-service queue, but + * (2) this switch of bfqq to busy changes the scenario. */ if (bfqd->in_service_queue && ((bfqq_wants_to_preempt && bfqq->wr_coeff >= bfqd->in_service_queue->wr_coeff) || - bfq_bfqq_higher_class_or_weight(bfqq, bfqd->in_service_queue)) && + bfq_bfqq_higher_class_or_weight(bfqq, bfqd->in_service_queue) || + !bfq_better_to_idle(bfqd->in_service_queue)) && next_queue_may_preempt(bfqd)) bfq_bfqq_expire(bfqd, bfqd->in_service_queue, false, BFQQE_PREEMPTED); From patchwork Mon Jan 25 19:02:45 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Paolo Valente X-Patchwork-Id: 12048109 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id CD610C433E0 for ; Tue, 26 Jan 2021 19:49:58 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id A7FDA2220B for ; Tue, 26 Jan 2021 19:49:58 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2387777AbhAZFdg (ORCPT ); Tue, 26 Jan 2021 00:33:36 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48452 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1731525AbhAYTFB (ORCPT ); Mon, 25 Jan 2021 14:05:01 -0500 Received: from mail-wm1-x32e.google.com (mail-wm1-x32e.google.com [IPv6:2a00:1450:4864:20::32e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 57816C061788 for ; Mon, 25 Jan 2021 11:03:19 -0800 (PST) Received: by mail-wm1-x32e.google.com with SMTP id 190so379219wmz.0 for ; Mon, 25 Jan 2021 11:03:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=VBkn8blA/6+QqkMVIGbS0xGWL5eTh//t50DpyLtTo/8=; b=vqsaLV7pZBnPICnzIgmqorwk2fyo45c5UKZBvj6tVNO/j3qga7xpo2Ha8grkSszN3A mZnbVNrlC62jx5xVWKCIKvxE1MyVwNu0h1gUVW17Pyzi3OK+uM3x4+iOKYYu7vtS+PQD +cRVLfrDNzVVtZ2Jz7EhZHV6ZUXp3KsVeVYbmz4HubIkDNrwgRWcPPMalW2cM1gCrd9Q iBevB2gs3Q4wo3DmleUZTog8Kx/uhjAesYQrrzCDs6XYqmRcOBKAXc97pRy4HQJN/tqo XSwGu72eqRjpk3R+dw96+AA4bt3dL0egLqL4WpqT1cWD7bPimewOBIRMd101afM7/XkM xUHw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=VBkn8blA/6+QqkMVIGbS0xGWL5eTh//t50DpyLtTo/8=; b=ufYeC07BR9x+mJOrB6of6Bo3iGA/9PS8h+jduF0DKgHSv2o56VAoazcJZ4kzH0/GSi RZugWW8srCspDjfxAb2I2uFzjwGJHs9ad35DnGS2I0cNHNwjejMeNRaLRlsL0WmUWglM SGdbw/dQ2UdZRU59sb4UvU7+XX0pIZ4CGJO16USK9dZ+V+B4px0KwyuBr2nzWy5zlc5m vyC4hagzz268Q301nUn7CQHU2BRjp/1aSLkuhlXnq0Fpm5POhC1bFclgOMyI8pyambJR j0l1o8LNMJTOuLVEfNQ4vChLuzPwLJUmlso08+zYpiYBUXncUnsGZUsJVq8cUSYwAfaW Oazg== X-Gm-Message-State: AOAM532LGfUSD5BI5XeS0BLi+oAtoErnyJtKPFQzWVDSb0wflohBrzTZ CBW7ozYXGD9blfShI4QCXu+ZDA== X-Google-Smtp-Source: ABdhPJzGmeJTvaTg1ITDmlviE9TGzuSOK9EMZHfgNtiPJsBjq/PBozhjmh5XEBzl7dO7ymaRlYvfGg== X-Received: by 2002:a05:600c:230e:: with SMTP id 14mr1413130wmo.161.1611601398093; Mon, 25 Jan 2021 11:03:18 -0800 (PST) Received: from localhost.localdomain ([37.160.159.175]) by smtp.gmail.com with ESMTPSA id g194sm222534wme.39.2021.01.25.11.03.16 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Mon, 25 Jan 2021 11:03:17 -0800 (PST) From: Paolo Valente To: Jens Axboe Cc: linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, Paolo Valente , Jan Kara Subject: [PATCH BUGFIX/IMPROVEMENT 3/6] block, bfq: fix switch back from soft-rt weitgh-raising Date: Mon, 25 Jan 2021 20:02:45 +0100 Message-Id: <20210125190248.49338-4-paolo.valente@linaro.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210125190248.49338-1-paolo.valente@linaro.org> References: <20210125190248.49338-1-paolo.valente@linaro.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org A bfq_queue may happen to be deemed as soft real-time while it is still enjoying interactive weight-raising. If this happens because of a false positive, then the bfq_queue is likely to loose its soft real-time status soon. Upon losing such a status, the bfq_queue must get back its interactive weight-raising, if its interactive period is not over yet. But this case is not handled. This commit corrects this error. Tested-by: Jan Kara Signed-off-by: Paolo Valente --- block/bfq-iosched.c | 22 ++++++++++++++++++++-- 1 file changed, 20 insertions(+), 2 deletions(-) diff --git a/block/bfq-iosched.c b/block/bfq-iosched.c index 6a02a12ff553..9e5242b2788a 100644 --- a/block/bfq-iosched.c +++ b/block/bfq-iosched.c @@ -5293,8 +5293,26 @@ bfq_update_io_seektime(struct bfq_data *bfqd, struct bfq_queue *bfqq, if (bfqq->wr_coeff > 1 && bfqq->wr_cur_max_time == bfqd->bfq_wr_rt_max_time && - BFQQ_TOTALLY_SEEKY(bfqq)) - bfq_bfqq_end_wr(bfqq); + BFQQ_TOTALLY_SEEKY(bfqq)) { + if (time_is_before_jiffies(bfqq->wr_start_at_switch_to_srt + + bfq_wr_duration(bfqd))) { + /* + * In soft_rt weight raising with the + * interactive-weight-raising period + * elapsed (so no switch back to + * interactive weight raising). + */ + bfq_bfqq_end_wr(bfqq); + } else { /* + * stopping soft_rt weight raising + * while still in interactive period, + * switch back to interactive weight + * raising + */ + switch_back_to_interactive_wr(bfqq, bfqd); + bfqq->entity.prio_changed = 1; + } + } } static void bfq_update_has_short_ttime(struct bfq_data *bfqd, From patchwork Mon Jan 25 19:02:46 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Paolo Valente X-Patchwork-Id: 12044021 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6B94EC433DB for ; Mon, 25 Jan 2021 19:05:58 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 3B61D2067B for ; Mon, 25 Jan 2021 19:05:58 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1731542AbhAYTFd (ORCPT ); Mon, 25 Jan 2021 14:05:33 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48456 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1731534AbhAYTFF (ORCPT ); Mon, 25 Jan 2021 14:05:05 -0500 Received: from mail-wr1-x431.google.com (mail-wr1-x431.google.com [IPv6:2a00:1450:4864:20::431]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B06ACC06178B for ; Mon, 25 Jan 2021 11:03:20 -0800 (PST) Received: by mail-wr1-x431.google.com with SMTP id p15so7436347wrq.8 for ; Mon, 25 Jan 2021 11:03:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=z3N9Fc9Oym1a/uSXoPjk2H2RgyHj8WV7XIwzdfEee54=; b=aSRERs4AkNneRj0RmPWLV03SycGYdHxwG1cq6GOgodsaPX4VY1olaYz37kZGHA8Vdh ctrahRmB7MbwOQao585A4TjlLBdoLU3C/l67U4bz/qwmkaR6cHFxIYMSfIeCdxT4K1a3 VRUwrlgnZdKVIyhB8meFItX0n6xS5rjc5A5A81xQ9z7d+ErHgdl4Mhl29Is2hTNXG5AD OiYo8DudLASRR4Fgoqcu+y38TQmttdkiqJYh2q+giBtn4Rq0+CpjPpG6vF86PcZt5EVm jHfkFcfAfc/Paxz0OfHshMxWfZJt9LQYq9D6hgMjSHtU8hWfmvl1ewO1JLtdEE63knJ4 20nw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=z3N9Fc9Oym1a/uSXoPjk2H2RgyHj8WV7XIwzdfEee54=; b=oAYSNyJ73PUdxw/9iyzlDtNSRtv/DDbWz3WXb46VqejtwLx+iaBq0Z8o6QHoDl1Lio D6VTINrSEf5D3L37dA6Lpy8G+QAaf8mi/L8/OGfoaMztOYSNhWPVHAr44E1BZkHUPljh tywT6v9md6MlGACtnF50/qEnOjqtjHMrb2RnUhmmJirU+kNy9DHGjZo3mArI8FNnPRqz wOdf1p52xg1HSn2myGXMaKIf5j+rQBJk3+nLUCGUSPfPEe9GrhYEWHyLFaadIqMhhc+H iZzScoXfyuSP/Jp4CDP8WwutDhl+H7gXFvQ7GFRZNZ7C/6wZi/gPsGrLzaz1Qvc7CwEy htnw== X-Gm-Message-State: AOAM533yc8w0XTOWiIimNCQsmnbCvEaSJqPONYIxhKYma4qzEQ9qfLe4 ZEizgpE0BzsPSy0q1btbXq+aJg== X-Google-Smtp-Source: ABdhPJxwcZ2dcN1I/y1hi9Lq/ZxxKwclSEVGgd921GuPKSsEsHU54wwyDcs3IEFHCU6qxvfnrGnyXw== X-Received: by 2002:a5d:5051:: with SMTP id h17mr2632482wrt.164.1611601399469; Mon, 25 Jan 2021 11:03:19 -0800 (PST) Received: from localhost.localdomain ([37.160.159.175]) by smtp.gmail.com with ESMTPSA id g194sm222534wme.39.2021.01.25.11.03.18 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Mon, 25 Jan 2021 11:03:18 -0800 (PST) From: Paolo Valente To: Jens Axboe Cc: linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, Paolo Valente , Jan Kara Subject: [PATCH BUGFIX/IMPROVEMENT 4/6] block, bfq: save also weight-raised service on queue merging Date: Mon, 25 Jan 2021 20:02:46 +0100 Message-Id: <20210125190248.49338-5-paolo.valente@linaro.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210125190248.49338-1-paolo.valente@linaro.org> References: <20210125190248.49338-1-paolo.valente@linaro.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org To prevent weight-raising information from being lost on bfq_queue merging, also the amount of service that a bfq_queue receives must be saved and restored when the bfq_queue is merged and split, respectively. Tested-by: Jan Kara Signed-off-by: Paolo Valente --- block/bfq-iosched.c | 2 ++ block/bfq-iosched.h | 1 + 2 files changed, 3 insertions(+) diff --git a/block/bfq-iosched.c b/block/bfq-iosched.c index 9e5242b2788a..56ad6067d41d 100644 --- a/block/bfq-iosched.c +++ b/block/bfq-iosched.c @@ -1029,6 +1029,7 @@ bfq_bfqq_resume_state(struct bfq_queue *bfqq, struct bfq_data *bfqd, bfqq->io_start_time = bic->saved_io_start_time; bfqq->tot_idle_time = bic->saved_tot_idle_time; bfqq->wr_coeff = bic->saved_wr_coeff; + bfqq->service_from_wr = bic->saved_service_from_wr; bfqq->wr_start_at_switch_to_srt = bic->saved_wr_start_at_switch_to_srt; bfqq->last_wr_start_finish = bic->saved_last_wr_start_finish; bfqq->wr_cur_max_time = bic->saved_wr_cur_max_time; @@ -2775,6 +2776,7 @@ static void bfq_bfqq_save_state(struct bfq_queue *bfqq) bic->saved_wr_coeff = bfqq->wr_coeff; bic->saved_wr_start_at_switch_to_srt = bfqq->wr_start_at_switch_to_srt; + bic->saved_service_from_wr = bfqq->service_from_wr; bic->saved_last_wr_start_finish = bfqq->last_wr_start_finish; bic->saved_wr_cur_max_time = bfqq->wr_cur_max_time; } diff --git a/block/bfq-iosched.h b/block/bfq-iosched.h index c913b06016b3..d15299d59f89 100644 --- a/block/bfq-iosched.h +++ b/block/bfq-iosched.h @@ -440,6 +440,7 @@ struct bfq_io_cq { */ unsigned long saved_wr_coeff; unsigned long saved_last_wr_start_finish; + unsigned long saved_service_from_wr; unsigned long saved_wr_start_at_switch_to_srt; unsigned int saved_wr_cur_max_time; struct bfq_ttime saved_ttime; From patchwork Mon Jan 25 19:02:47 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Paolo Valente X-Patchwork-Id: 12048145 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B56D3C433E0 for ; Tue, 26 Jan 2021 19:53:53 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 80C0322A83 for ; Tue, 26 Jan 2021 19:53:53 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2387770AbhAZFd3 (ORCPT ); Tue, 26 Jan 2021 00:33:29 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48460 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1731527AbhAYTFA (ORCPT ); Mon, 25 Jan 2021 14:05:00 -0500 Received: from mail-wr1-x433.google.com (mail-wr1-x433.google.com [IPv6:2a00:1450:4864:20::433]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 472FAC061793 for ; Mon, 25 Jan 2021 11:03:22 -0800 (PST) Received: by mail-wr1-x433.google.com with SMTP id p15so7436417wrq.8 for ; Mon, 25 Jan 2021 11:03:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=uemGCY4/316d8GBRWMu2x2DHphVSGyL2sreD9v73IaY=; b=tWxtMW8Cz4ayJUVqWs2aqwqtCLJwusxbOqpBiLFCZDJ9nxUC8sXT3a8RsrdC0RGD4A LhpOgHOSEIYWqHbFqTlrpT/GgvPAi99nMaTuWSC1+ObZbWlxukmgQPkMnRBHF4RMk7V+ BJUVjJ51c/n73FnFmuRyOLLA/JIg1PAGjJtNWK1yvMfl5nm1myE4dzk3i2DtujZdq5O3 nSN5d4T/OQUOZx3TH0yxML+f/SwouhbnsRc8ER3RrK3pI+YrDiOYSPHDF6g8Dedrfy13 Vo8OK99KQqS27gvQ24Z75kSv3+VBvuGPlQ+fly+FYo1vxqqHnAlc+sk+syUJ0Nlwnga7 DRfA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=uemGCY4/316d8GBRWMu2x2DHphVSGyL2sreD9v73IaY=; b=Y4JpCS6S8qtqN9EwZYc35gaTKFvtAFP5lyQMYMp00mv8lFzI/W3NBf41r9M3O37K1/ +x+KqlYm2imVDJS8/g9F0TiP7vi+UPFWbFFVwG8Gpb67Yh7c4zaSl8T15EPOXWwkkwK4 10HpiDJ28zca2n7dH/4bCESpullqK3ZdKaQ7LMcIKrtSCX3Ernmk/W5vkM23siFA/Bm8 E//l/TbR1gnvbqYP/i4zzKPiUt3iDrdyN+hnJ6x9QfZ9QA5V5PQ3lrdAp1fVajBv99Os VHYHbBvHkJFjKExN0vPHUNo0Bw8Ca8WyrzMTrEJcy209NYzjXMeNUrgQhiuULW/zouo8 SeQw== X-Gm-Message-State: AOAM530UJPPeSEy/yKTKb5AAPy/zIMCHJX+mET0PlYgFZB5W5HXSFlUF JcLqMhui4Zb98xD+coNU0jTGeA== X-Google-Smtp-Source: ABdhPJxrzx3VM5nd+6B4NAg/5ad3yjMTjSwlywag1cH6B40RsYY5iCAXFuH2UwP6jr5IzPSgFpIdcg== X-Received: by 2002:a05:6000:1543:: with SMTP id 3mr2593302wry.254.1611601401012; Mon, 25 Jan 2021 11:03:21 -0800 (PST) Received: from localhost.localdomain ([37.160.159.175]) by smtp.gmail.com with ESMTPSA id g194sm222534wme.39.2021.01.25.11.03.19 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Mon, 25 Jan 2021 11:03:20 -0800 (PST) From: Paolo Valente To: Jens Axboe Cc: linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, Paolo Valente , Jan Kara Subject: [PATCH BUGFIX/IMPROVEMENT 5/6] block, bfq: save also injection state on queue merging Date: Mon, 25 Jan 2021 20:02:47 +0100 Message-Id: <20210125190248.49338-6-paolo.valente@linaro.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210125190248.49338-1-paolo.valente@linaro.org> References: <20210125190248.49338-1-paolo.valente@linaro.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org To prevent injection information from being lost on bfq_queue merging, also the amount of service that a bfq_queue receives must be saved and restored when the bfq_queue is merged and split, respectively. Tested-by: Jan Kara Signed-off-by: Paolo Valente --- block/bfq-iosched.c | 8 ++++++++ block/bfq-iosched.h | 5 +++++ 2 files changed, 13 insertions(+) diff --git a/block/bfq-iosched.c b/block/bfq-iosched.c index 56ad6067d41d..e56ee60df014 100644 --- a/block/bfq-iosched.c +++ b/block/bfq-iosched.c @@ -1024,6 +1024,10 @@ bfq_bfqq_resume_state(struct bfq_queue *bfqq, struct bfq_data *bfqd, else bfq_clear_bfqq_IO_bound(bfqq); + bfqq->last_serv_time_ns = bic->saved_last_serv_time_ns; + bfqq->inject_limit = bic->saved_inject_limit; + bfqq->decrease_time_jif = bic->saved_decrease_time_jif; + bfqq->entity.new_weight = bic->saved_weight; bfqq->ttime = bic->saved_ttime; bfqq->io_start_time = bic->saved_io_start_time; @@ -2748,6 +2752,10 @@ static void bfq_bfqq_save_state(struct bfq_queue *bfqq) if (!bic) return; + bic->saved_last_serv_time_ns = bfqq->last_serv_time_ns; + bic->saved_inject_limit = bfqq->inject_limit; + bic->saved_decrease_time_jif = bfqq->decrease_time_jif; + bic->saved_weight = bfqq->entity.orig_weight; bic->saved_ttime = bfqq->ttime; bic->saved_has_short_ttime = bfq_bfqq_has_short_ttime(bfqq); diff --git a/block/bfq-iosched.h b/block/bfq-iosched.h index d15299d59f89..3f350fa3c5fd 100644 --- a/block/bfq-iosched.h +++ b/block/bfq-iosched.h @@ -444,6 +444,11 @@ struct bfq_io_cq { unsigned long saved_wr_start_at_switch_to_srt; unsigned int saved_wr_cur_max_time; struct bfq_ttime saved_ttime; + + /* Save also injection state */ + u64 saved_last_serv_time_ns; + unsigned int saved_inject_limit; + unsigned long saved_decrease_time_jif; }; /** From patchwork Mon Jan 25 19:02:48 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Paolo Valente X-Patchwork-Id: 12048143 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E2C39C433DB for ; Tue, 26 Jan 2021 19:53:52 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id A1FCD22A85 for ; Tue, 26 Jan 2021 19:53:52 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1731527AbhAZFdb (ORCPT ); Tue, 26 Jan 2021 00:33:31 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48464 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1731529AbhAYTFA (ORCPT ); Mon, 25 Jan 2021 14:05:00 -0500 Received: from mail-wr1-x433.google.com (mail-wr1-x433.google.com [IPv6:2a00:1450:4864:20::433]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 896CCC061797 for ; Mon, 25 Jan 2021 11:03:24 -0800 (PST) Received: by mail-wr1-x433.google.com with SMTP id h9so3845513wrr.9 for ; Mon, 25 Jan 2021 11:03:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=mBtrbFwWUbHdCDpM5SiVyGA3LyuDrRqzb7zQyqRYGq0=; b=p9GSSB5J4VKyYCQ6cOc3y0WntPOPd4tW17j6qu5wDnKIFPCSLUrJuJGQ83oIdnif2L KGVpmhZQp1RhdRyGcFzLuNvDEAT7THPotg9hEDq5RVzeNj6f0l9F7e4Iz0ovIEVxIHs6 5aSILS3/1hiDs5UAN1ja+Ue22g9cKXixRLA4XW24KBQ5KAQ/ww2xrAbZOgIJlmsNBccr z/1ezJuIidYtBcmY0CJ3NKG+E5AKFSlxSbu5TnvCzKysyzxAp4UVeeLrezhplBXSX11V 0PeZhtMowMddbCGw3/LswgE5vLpuDBeh8zRiDeCB+miE6NYSftb7SRs0wtc3WEJVg5VE Iglw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=mBtrbFwWUbHdCDpM5SiVyGA3LyuDrRqzb7zQyqRYGq0=; b=pNVe6NhhTDmyD/nc00vPu74Xbu+1SkWvX9s6Y8nvh1ehsme5rjWylf2EIKyP7lkspS WHthzgrWXjULmtbQE5t2bWNJhYOIG+Mp8jXvK3FF6Qy3gxDYBvgGJqR2NqT7FfJ8uIOU /oiJ1E+7SW8xAcT8vtckp7i/BpNseBOjRsbPxJweOW5vRAOlv67WvN5IKSAjvwVDWRWE Je333QAXtjC7uw1SYK0eRy20hOVCIaDjBaMZuKDDWrUXr1pps8JwBszbfndOCAVixEu9 fErvE2gbhlslczaBktbIpWroiaonqpSzkC24V5Ng2ABuTkalPEj+w9LZt/RMu0SYnBbF wxDg== X-Gm-Message-State: AOAM532VmvMXQy5F0zOpvClqQ9nL/uxuTpxT8Fr19C37K/zX5S+Bw0S3 CZCqz37jmFSBdxgrG/rgCLAO2kpFY9ZDmg== X-Google-Smtp-Source: ABdhPJyrLVkgmvRn21KH1LTSFELBuNlDN1V1usXfDnA1I+XJ/hPJjhKl9HZ8B0pHQkTJHJbOZMjD9w== X-Received: by 2002:adf:ba0c:: with SMTP id o12mr2507895wrg.322.1611601403210; Mon, 25 Jan 2021 11:03:23 -0800 (PST) Received: from localhost.localdomain ([37.160.159.175]) by smtp.gmail.com with ESMTPSA id g194sm222534wme.39.2021.01.25.11.03.21 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Mon, 25 Jan 2021 11:03:22 -0800 (PST) From: Paolo Valente To: Jens Axboe Cc: linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, Paolo Valente , Jan Kara Subject: [PATCH BUGFIX/IMPROVEMENT 6/6] block, bfq: make waker-queue detection more robust Date: Mon, 25 Jan 2021 20:02:48 +0100 Message-Id: <20210125190248.49338-7-paolo.valente@linaro.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210125190248.49338-1-paolo.valente@linaro.org> References: <20210125190248.49338-1-paolo.valente@linaro.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org In the presence of many parallel I/O flows, the detection of waker bfq_queues suffers from false positives. This commits addresses this issue by making the filtering of actual wakers more selective. In more detail, a candidate waker must be found to meet waker requirements three times before being promoted to actual waker. Tested-by: Jan Kara Signed-off-by: Paolo Valente --- block/bfq-iosched.c | 211 +++++++++++++++++++++----------------------- block/bfq-iosched.h | 7 +- 2 files changed, 108 insertions(+), 110 deletions(-) diff --git a/block/bfq-iosched.c b/block/bfq-iosched.c index e56ee60df014..445cef9c0bb9 100644 --- a/block/bfq-iosched.c +++ b/block/bfq-iosched.c @@ -158,7 +158,6 @@ BFQ_BFQQ_FNS(in_large_burst); BFQ_BFQQ_FNS(coop); BFQ_BFQQ_FNS(split_coop); BFQ_BFQQ_FNS(softrt_update); -BFQ_BFQQ_FNS(has_waker); #undef BFQ_BFQQ_FNS \ /* Expiration time of sync (0) and async (1) requests, in ns. */ @@ -1905,6 +1904,107 @@ static void bfq_update_io_intensity(struct bfq_queue *bfqq, u64 now_ns) } } +/* + * Detect whether bfqq's I/O seems synchronized with that of some + * other queue, i.e., whether bfqq, after remaining empty, happens to + * receive new I/O only right after some I/O request of the other + * queue has been completed. We call waker queue the other queue, and + * we assume, for simplicity, that bfqq may have at most one waker + * queue. + * + * A remarkable throughput boost can be reached by unconditionally + * injecting the I/O of the waker queue, every time a new + * bfq_dispatch_request happens to be invoked while I/O is being + * plugged for bfqq. In addition to boosting throughput, this + * unblocks bfqq's I/O, thereby improving bandwidth and latency for + * bfqq. Note that these same results may be achieved with the general + * injection mechanism, but less effectively. For details on this + * aspect, see the comments on the choice of the queue for injection + * in bfq_select_queue(). + * + * Turning back to the detection of a waker queue, a queue Q is deemed + * as a waker queue for bfqq if, for three consecutive times, bfqq + * happens to become non empty right after a request of Q has been + * completed. In particular, on the first time, Q is tentatively set + * as a candidate waker queue, while on the third consecutive time + * that Q is detected, the field waker_bfqq is set to Q, to confirm + * that Q is a waker queue for bfqq. These detection steps are + * performed only if bfqq has a long think time, so as to make it more + * likely that bfqq's I/O is actually being blocked by a + * synchronization. This last filter, plus the above three-times + * requirement, make false positives less likely. + * + * NOTE + * + * The sooner a waker queue is detected, the sooner throughput can be + * boosted by injecting I/O from the waker queue. Fortunately, + * detection is likely to be actually fast, for the following + * reasons. While blocked by synchronization, bfqq has a long think + * time. This implies that bfqq's inject limit is at least equal to 1 + * (see the comments in bfq_update_inject_limit()). So, thanks to + * injection, the waker queue is likely to be served during the very + * first I/O-plugging time interval for bfqq. This triggers the first + * step of the detection mechanism. Thanks again to injection, the + * candidate waker queue is then likely to be confirmed no later than + * during the next I/O-plugging interval for bfqq. + * + * ISSUE + * + * On queue merging all waker information is lost. + */ +void bfq_check_waker(struct bfq_data *bfqd, struct bfq_queue *bfqq, u64 now_ns) +{ + if (!bfqd->last_completed_rq_bfqq || + bfqd->last_completed_rq_bfqq == bfqq || + bfq_bfqq_has_short_ttime(bfqq) || + now_ns - bfqd->last_completion >= 4 * NSEC_PER_MSEC || + bfqd->last_completed_rq_bfqq == bfqq->waker_bfqq) + return; + + if (bfqd->last_completed_rq_bfqq != + bfqq->tentative_waker_bfqq) { + /* + * First synchronization detected with a + * candidate waker queue, or with a different + * candidate waker queue from the current one. + */ + bfqq->tentative_waker_bfqq = + bfqd->last_completed_rq_bfqq; + bfqq->num_waker_detections = 1; + } else /* Same tentative waker queue detected again */ + bfqq->num_waker_detections++; + + if (bfqq->num_waker_detections == 3) { + bfqq->waker_bfqq = bfqd->last_completed_rq_bfqq; + bfqq->tentative_waker_bfqq = NULL; + + /* + * If the waker queue disappears, then + * bfqq->waker_bfqq must be reset. To + * this goal, we maintain in each + * waker queue a list, woken_list, of + * all the queues that reference the + * waker queue through their + * waker_bfqq pointer. When the waker + * queue exits, the waker_bfqq pointer + * of all the queues in the woken_list + * is reset. + * + * In addition, if bfqq is already in + * the woken_list of a waker queue, + * then, before being inserted into + * the woken_list of a new waker + * queue, bfqq must be removed from + * the woken_list of the old waker + * queue. + */ + if (!hlist_unhashed(&bfqq->woken_list_node)) + hlist_del_init(&bfqq->woken_list_node); + hlist_add_head(&bfqq->woken_list_node, + &bfqd->last_completed_rq_bfqq->woken_list); + } +} + static void bfq_add_request(struct request *rq) { struct bfq_queue *bfqq = RQ_BFQQ(rq); @@ -1919,111 +2019,7 @@ static void bfq_add_request(struct request *rq) bfqd->queued++; if (RB_EMPTY_ROOT(&bfqq->sort_list) && bfq_bfqq_sync(bfqq)) { - /* - * Detect whether bfqq's I/O seems synchronized with - * that of some other queue, i.e., whether bfqq, after - * remaining empty, happens to receive new I/O only - * right after some I/O request of the other queue has - * been completed. We call waker queue the other - * queue, and we assume, for simplicity, that bfqq may - * have at most one waker queue. - * - * A remarkable throughput boost can be reached by - * unconditionally injecting the I/O of the waker - * queue, every time a new bfq_dispatch_request - * happens to be invoked while I/O is being plugged - * for bfqq. In addition to boosting throughput, this - * unblocks bfqq's I/O, thereby improving bandwidth - * and latency for bfqq. Note that these same results - * may be achieved with the general injection - * mechanism, but less effectively. For details on - * this aspect, see the comments on the choice of the - * queue for injection in bfq_select_queue(). - * - * Turning back to the detection of a waker queue, a - * queue Q is deemed as a waker queue for bfqq if, for - * two consecutive times, bfqq happens to become non - * empty right after a request of Q has been - * completed. In particular, on the first time, Q is - * tentatively set as a candidate waker queue, while - * on the second time, the flag - * bfq_bfqq_has_waker(bfqq) is set to confirm that Q - * is a waker queue for bfqq. These detection steps - * are performed only if bfqq has a long think time, - * so as to make it more likely that bfqq's I/O is - * actually being blocked by a synchronization. This - * last filter, plus the above two-times requirement, - * make false positives less likely. - * - * NOTE - * - * The sooner a waker queue is detected, the sooner - * throughput can be boosted by injecting I/O from the - * waker queue. Fortunately, detection is likely to be - * actually fast, for the following reasons. While - * blocked by synchronization, bfqq has a long think - * time. This implies that bfqq's inject limit is at - * least equal to 1 (see the comments in - * bfq_update_inject_limit()). So, thanks to - * injection, the waker queue is likely to be served - * during the very first I/O-plugging time interval - * for bfqq. This triggers the first step of the - * detection mechanism. Thanks again to injection, the - * candidate waker queue is then likely to be - * confirmed no later than during the next - * I/O-plugging interval for bfqq. - */ - if (bfqd->last_completed_rq_bfqq && - !bfq_bfqq_has_short_ttime(bfqq) && - now_ns - bfqd->last_completion < - 4 * NSEC_PER_MSEC) { - if (bfqd->last_completed_rq_bfqq != bfqq && - bfqd->last_completed_rq_bfqq != - bfqq->waker_bfqq) { - /* - * First synchronization detected with - * a candidate waker queue, or with a - * different candidate waker queue - * from the current one. - */ - bfqq->waker_bfqq = bfqd->last_completed_rq_bfqq; - - /* - * If the waker queue disappears, then - * bfqq->waker_bfqq must be reset. To - * this goal, we maintain in each - * waker queue a list, woken_list, of - * all the queues that reference the - * waker queue through their - * waker_bfqq pointer. When the waker - * queue exits, the waker_bfqq pointer - * of all the queues in the woken_list - * is reset. - * - * In addition, if bfqq is already in - * the woken_list of a waker queue, - * then, before being inserted into - * the woken_list of a new waker - * queue, bfqq must be removed from - * the woken_list of the old waker - * queue. - */ - if (!hlist_unhashed(&bfqq->woken_list_node)) - hlist_del_init(&bfqq->woken_list_node); - hlist_add_head(&bfqq->woken_list_node, - &bfqd->last_completed_rq_bfqq->woken_list); - - bfq_clear_bfqq_has_waker(bfqq); - } else if (bfqd->last_completed_rq_bfqq == - bfqq->waker_bfqq && - !bfq_bfqq_has_waker(bfqq)) { - /* - * synchronization with waker_bfqq - * seen for the second time - */ - bfq_mark_bfqq_has_waker(bfqq); - } - } + bfq_check_waker(bfqd, bfqq, now_ns); /* * Periodically reset inject limit, to make sure that @@ -4569,7 +4565,7 @@ static struct bfq_queue *bfq_select_queue(struct bfq_data *bfqd) bfq_serv_to_charge(async_bfqq->next_rq, async_bfqq) <= bfq_bfqq_budget_left(async_bfqq)) bfqq = bfqq->bic->bfqq[0]; - else if (bfq_bfqq_has_waker(bfqq) && + else if (bfqq->waker_bfqq && bfq_bfqq_busy(bfqq->waker_bfqq) && bfqq->waker_bfqq->next_rq && bfq_serv_to_charge(bfqq->waker_bfqq->next_rq, @@ -4976,7 +4972,6 @@ void bfq_put_queue(struct bfq_queue *bfqq) hlist_for_each_entry_safe(item, n, &bfqq->woken_list, woken_list_node) { item->waker_bfqq = NULL; - bfq_clear_bfqq_has_waker(item); hlist_del_init(&item->woken_list_node); } diff --git a/block/bfq-iosched.h b/block/bfq-iosched.h index 3f350fa3c5fd..b8e793c34ff1 100644 --- a/block/bfq-iosched.h +++ b/block/bfq-iosched.h @@ -376,6 +376,11 @@ struct bfq_queue { * bfq_select_queue(). */ struct bfq_queue *waker_bfqq; + /* pointer to the curr. tentative waker queue, see bfq_check_waker() */ + struct bfq_queue *tentative_waker_bfqq; + /* number of times the same tentative waker has been detected */ + unsigned int num_waker_detections; + /* node for woken_list, see below */ struct hlist_node woken_list_node; /* @@ -776,7 +781,6 @@ enum bfqq_state_flags { */ BFQQF_coop, /* bfqq is shared */ BFQQF_split_coop, /* shared bfqq will be split */ - BFQQF_has_waker /* bfqq has a waker queue */ }; #define BFQ_BFQQ_FNS(name) \ @@ -796,7 +800,6 @@ BFQ_BFQQ_FNS(in_large_burst); BFQ_BFQQ_FNS(coop); BFQ_BFQQ_FNS(split_coop); BFQ_BFQQ_FNS(softrt_update); -BFQ_BFQQ_FNS(has_waker); #undef BFQ_BFQQ_FNS /* Expiration reasons. */