From patchwork Wed Sep 4 19:45:52 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tejun Heo X-Patchwork-Id: 11131303 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 8EC231805 for ; Wed, 4 Sep 2019 19:46:05 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 6718222CF5 for ; Wed, 4 Sep 2019 19:46:05 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1567626365; bh=G5YRcN1eewC/hQJEJyW7hKch5p3TJ9B+7YuXF/YOcds=; h=From:To:Cc:Subject:Date:In-Reply-To:References:List-ID:From; b=IOke8N2YLPjHMLdWQqmjN5u9tqcmQ//ja+K3sVcPLNHvmysWR/uvxSaQ2nSMHqcU2 Rg1wagforUD2WQTOOb20VFoQFE+G2+gSaQifsJPgxGL1Y07M90qoshVcTgpaOebV00 K1GxbGelathK0m47VMHKYNZ//SJo7vp14iE7KQR4= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727387AbfIDTqE (ORCPT ); Wed, 4 Sep 2019 15:46:04 -0400 Received: from mail-qk1-f194.google.com ([209.85.222.194]:43568 "EHLO mail-qk1-f194.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1729677AbfIDTqD (ORCPT ); Wed, 4 Sep 2019 15:46:03 -0400 Received: by mail-qk1-f194.google.com with SMTP id m2so20785341qkd.10; Wed, 04 Sep 2019 12:46:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:from:to:cc:subject:date:message-id:in-reply-to:references; bh=ZJvcHGT5+ZK1nLs2NfUHYcJOsCB431R3Hn7hw0Oz7B8=; b=E0uqMjQRayYNy8veSJkjeJkP2qgDyGUTV4k/f6AyhPt+/1jGiXulUJyYomLTjgBaVU MIak9tyLL5QLx7o6wQ699v8gdPm2jLzcMrC8/1PqxPQEYyQZZyRvUr9KbUpkPx2e+fXn cUCCrv/pYQYuDRjzlLnWXlQv7zY0BBWoD1yCeSFwdRuyM+ERVdE0W2XOxr+BBKFLiyua FHO4Pyu3boOi5WI6b0WXNtrI/2xFvUoAof9YIe+TocItH4ZBEWUHwLy8KNgupBNwCaKA 3jsIYT+WL7LKdzby6xjH8RvzFmY5pFbW5b3rDQZu7YPjvNCkGAGLBUi7RhkTAHhH7rqT Kb9A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:from:to:cc:subject:date:message-id :in-reply-to:references; bh=ZJvcHGT5+ZK1nLs2NfUHYcJOsCB431R3Hn7hw0Oz7B8=; b=Vaqp890WFDG1fN+ii+/4Fgyv/Zi+C6WWSr6rArX/Bwo9nM5VFuGMsg1sJuGSbVVnV9 GL9UOuVfAtnGe5u2DQheUKZNczGd6O0CZKB6nSob7tuLukjA3QSYcxCdtSybaTDGuwXE lty28k7Yt6MPV++2ENlPJ8Uuu0aga+t0m8JNxjJbD9JN7tXmCh408I0UPqbAnV4jP4cs FZWNN6+cY7foMhDkTv0AIUxUBi3yh6T7qVwv6BI+yiydtIRyXNX/kzuMgElsYu7px1UU f4hszbEExCW91Ms779rBsFkriH2Jft4hw0z2S0306NI4shn0+N/t4GNMlz1E2KwMC01t vA4A== X-Gm-Message-State: APjAAAUT4gVk+w3SYKtQzgIHiYpnOkdJjQS51qx3NqdQlzphWftty3gV Qyz60XaK31QMt4WnTDla+00= X-Google-Smtp-Source: APXvYqwAccyHdhgbgFCLP2mOEbgNl6xGrrLLy5l4LEf/X3ptmN7K1gvWtWPlEy0nuDujNY5UcNvhlA== X-Received: by 2002:a37:5187:: with SMTP id f129mr13390651qkb.382.1567626362436; Wed, 04 Sep 2019 12:46:02 -0700 (PDT) Received: from localhost ([2620:10d:c091:500::3:33b1]) by smtp.gmail.com with ESMTPSA id 64sm4710038qkk.63.2019.09.04.12.46.01 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 04 Sep 2019 12:46:01 -0700 (PDT) From: Tejun Heo To: axboe@kernel.dk Cc: linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, kernel-team@vger.kernel.org, cgroups@vger.kernel.org, newella@fb.com, Tejun Heo Subject: [PATCH 1/5] blk-iocost: Account force-charged overage in absolute vtime Date: Wed, 4 Sep 2019 12:45:52 -0700 Message-Id: <20190904194556.2984857-2-tj@kernel.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190904194556.2984857-1-tj@kernel.org> References: <20190904194556.2984857-1-tj@kernel.org> Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org Currently, when a bio needs to be force-charged and there isn't enough budget, vtime is simply pushed into the future. This means that the cost of the whole bio is scaled using the current hweight and then charged immediately. Until the global vtime advances beyond this future vtime, the cgroup won't be allowed to issue normal IOs. This is incorrect and can lead to, for example, exploding vrate or extended stalls if vrate range is constrained. Consider the following scenario. 1. A cgroup with a very low hweight runs out of budget. 2. A storm of swap-out happens on it. All of them are scaled according to the current low hweight and charged to vtime pushing it to a far future. 3. All other cgroups go idle and now the above cgroup has access to the whole device. However, because vtime is already wound using the past low hweight, what its current hweight is doesn't matter until global vtime catches up to the local vtime. 4. As a result, either vrate gets ramped up extremely or the IOs stall while the underlying device is idle. This is because the hweight the overage is calculated at is different from the hweight that it's being paid at. Fix it by remembering the overage in absoulte vtime and continuously paying with the actual budget according to the current hweight at each period. Note that non-forced bios which wait already remembers the cost in absolute vtime. This brings forced-bio accounting in line. Signed-off-by: Tejun Heo --- block/blk-iocost.c | 62 ++++++++++++++++++++++++++++++++++++++++------ 1 file changed, 55 insertions(+), 7 deletions(-) diff --git a/block/blk-iocost.c b/block/blk-iocost.c index 2aae8ec391ef..5b0c024ee5dd 100644 --- a/block/blk-iocost.c +++ b/block/blk-iocost.c @@ -469,6 +469,7 @@ struct ioc_gq { */ atomic64_t vtime; atomic64_t done_vtime; + atomic64_t abs_vdebt; u64 last_vtime; /* @@ -653,13 +654,21 @@ static struct ioc_cgrp *blkcg_to_iocc(struct blkcg *blkcg) /* * Scale @abs_cost to the inverse of @hw_inuse. The lower the hierarchical - * weight, the more expensive each IO. + * weight, the more expensive each IO. Must round up. */ static u64 abs_cost_to_cost(u64 abs_cost, u32 hw_inuse) { return DIV64_U64_ROUND_UP(abs_cost * HWEIGHT_WHOLE, hw_inuse); } +/* + * The inverse of abs_cost_to_cost(). Must round up. + */ +static u64 cost_to_abs_cost(u64 cost, u32 hw_inuse) +{ + return DIV64_U64_ROUND_UP(cost * hw_inuse, HWEIGHT_WHOLE); +} + static void iocg_commit_bio(struct ioc_gq *iocg, struct bio *bio, u64 cost) { bio->bi_iocost_cost = cost; @@ -1132,16 +1141,36 @@ static void iocg_kick_waitq(struct ioc_gq *iocg, struct ioc_now *now) struct iocg_wake_ctx ctx = { .iocg = iocg }; u64 margin_ns = (u64)(ioc->period_us * WAITQ_TIMER_MARGIN_PCT / 100) * NSEC_PER_USEC; - u64 vshortage, expires, oexpires; + u64 abs_vdebt, vdebt, vshortage, expires, oexpires; + s64 vbudget; + u32 hw_inuse; lockdep_assert_held(&iocg->waitq.lock); + current_hweight(iocg, NULL, &hw_inuse); + vbudget = now->vnow - atomic64_read(&iocg->vtime); + + /* pay off debt */ + abs_vdebt = atomic64_read(&iocg->abs_vdebt); + vdebt = abs_cost_to_cost(abs_vdebt, hw_inuse); + if (vdebt && vbudget > 0) { + u64 delta = min_t(u64, vbudget, vdebt); + u64 abs_delta = min(cost_to_abs_cost(delta, hw_inuse), + abs_vdebt); + + atomic64_add(delta, &iocg->vtime); + atomic64_add(delta, &iocg->done_vtime); + atomic64_sub(abs_delta, &iocg->abs_vdebt); + if (WARN_ON_ONCE(atomic64_read(&iocg->abs_vdebt) < 0)) + atomic64_set(&iocg->abs_vdebt, 0); + } + /* * Wake up the ones which are due and see how much vtime we'll need * for the next one. */ - current_hweight(iocg, NULL, &ctx.hw_inuse); - ctx.vbudget = now->vnow - atomic64_read(&iocg->vtime); + ctx.hw_inuse = hw_inuse; + ctx.vbudget = vbudget - vdebt; __wake_up_locked_key(&iocg->waitq, TASK_NORMAL, &ctx); if (!waitqueue_active(&iocg->waitq)) return; @@ -1187,6 +1216,11 @@ static void iocg_kick_delay(struct ioc_gq *iocg, struct ioc_now *now, u64 cost) u64 vmargin = ioc->margin_us * now->vrate; u64 margin_ns = ioc->margin_us * NSEC_PER_USEC; u64 expires, oexpires; + u32 hw_inuse; + + /* debt-adjust vtime */ + current_hweight(iocg, NULL, &hw_inuse); + vtime += abs_cost_to_cost(atomic64_read(&iocg->abs_vdebt), hw_inuse); /* clear or maintain depending on the overage */ if (time_before_eq64(vtime, now->vnow)) { @@ -1332,12 +1366,14 @@ static void ioc_timer_fn(struct timer_list *timer) * should have woken up in the last period and expire idle iocgs. */ list_for_each_entry_safe(iocg, tiocg, &ioc->active_iocgs, active_list) { - if (!waitqueue_active(&iocg->waitq) && !iocg_is_idle(iocg)) + if (!waitqueue_active(&iocg->waitq) && + !atomic64_read(&iocg->abs_vdebt) && !iocg_is_idle(iocg)) continue; spin_lock(&iocg->waitq.lock); - if (waitqueue_active(&iocg->waitq)) { + if (waitqueue_active(&iocg->waitq) || + atomic64_read(&iocg->abs_vdebt)) { /* might be oversleeping vtime / hweight changes, kick */ iocg_kick_waitq(iocg, &now); iocg_kick_delay(iocg, &now, 0); @@ -1673,13 +1709,24 @@ static void ioc_rqos_throttle(struct rq_qos *rqos, struct bio *bio) * in a while which is fine. */ if (!waitqueue_active(&iocg->waitq) && + !atomic64_read(&iocg->abs_vdebt) && time_before_eq64(vtime + cost, now.vnow)) { iocg_commit_bio(iocg, bio, cost); return; } + /* + * We're over budget. If @bio has to be issued regardless, + * remember the abs_cost instead of advancing vtime. + * iocg_kick_waitq() will pay off the debt before waking more IOs. + * This way, the debt is continuously paid off each period with the + * actual budget available to the cgroup. If we just wound vtime, + * we would incorrectly use the current hw_inuse for the entire + * amount which, for example, can lead to the cgroup staying + * blocked for a long time even with substantially raised hw_inuse. + */ if (bio_issue_as_root_blkg(bio) || fatal_signal_pending(current)) { - iocg_commit_bio(iocg, bio, cost); + atomic64_add(abs_cost, &iocg->abs_vdebt); iocg_kick_delay(iocg, &now, cost); return; } @@ -1928,6 +1975,7 @@ static void ioc_pd_init(struct blkg_policy_data *pd) iocg->ioc = ioc; atomic64_set(&iocg->vtime, now.vnow); atomic64_set(&iocg->done_vtime, now.vnow); + atomic64_set(&iocg->abs_vdebt, 0); atomic64_set(&iocg->active_period, atomic64_read(&ioc->cur_period)); INIT_LIST_HEAD(&iocg->active_list); iocg->hweight_active = HWEIGHT_WHOLE; From patchwork Wed Sep 4 19:45:53 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tejun Heo X-Patchwork-Id: 11131313 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 16E1B13B1 for ; Wed, 4 Sep 2019 19:46:30 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id EC8B1233A1 for ; Wed, 4 Sep 2019 19:46:29 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1567626390; bh=Gh1HhtE9kcYKMxIBtZJWgIkRvIBaBU+cPpa/NjXVivQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:List-ID:From; b=ma81+nCFr8Fi43d1o+7jMAfNrC8hKbLo001HKKQVjTQrVfm2MIlke8npmJMwxckM1 /+0xLNW0H/dGgOqkwcpzDQi56rbUcAKzJLS2t1NSw0ZsBDtwd0ReX1NVey63pvG9Fc 5AchtlOcjV4U1SgwOES1aaj+XSUIwCijyK2IV2k0= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730131AbfIDTqG (ORCPT ); Wed, 4 Sep 2019 15:46:06 -0400 Received: from mail-qt1-f196.google.com ([209.85.160.196]:32886 "EHLO mail-qt1-f196.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730059AbfIDTqF (ORCPT ); Wed, 4 Sep 2019 15:46:05 -0400 Received: by mail-qt1-f196.google.com with SMTP id r5so20592187qtd.0; Wed, 04 Sep 2019 12:46:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:from:to:cc:subject:date:message-id:in-reply-to:references; bh=20hd4msrraab6jp0F0s3lMKgksvtNidxbTFc6K2dRls=; b=CupX71bIVnfpPA8VGUrRsLsNQaXHSnSrVgTiMAhqvTUTIhXuj/bvxSRlaVRiANA7Z7 W6lK0f2NMmzXwkjXayxc91oQmH4j/o4uQkRffIRsRwb8+njuyWCFetp90/r9jD94bYdk cFSShDfZhragdYKCx7McGmf7vS/mAWvMuLK/QaWOu7qHR/VE/tB5DPouZagXNU8mbefP 10UO9oCiSORQNsx88g6WFqmOTixTwCTygs1Ii3g5KOPq/fc26lgbTzZc06xATfiZxmbl BZhbP3ZOsXL4Lj8XDj7iq38DoHYra/0lw/G1bbIYVud1VBcWhHDbE4lA5tCkd3Q/N6gE JBjQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:from:to:cc:subject:date:message-id :in-reply-to:references; bh=20hd4msrraab6jp0F0s3lMKgksvtNidxbTFc6K2dRls=; b=l1iIFMJO5na2lrDBypFIAhxO5pNgcTLyu8Jh11MOIJTb8EtZu1yCHZSwexQyHJkUck yAjfYmdr48DxAKtjuvfJ/h4khqMmjsFUqgfecezYPZ8kBdvF2ouL3TR7SADk2pY4TR16 opRLaJJImWJSvEVldH2Ao1Rrl/MPSebpZcDGfbzxYsJR7InG2mY+rYYDGXYOVrK3fnEy QuBofX8blgJeE1JSNLGgW+dtQaYpQI5gp7GtzMV1oPsWE0pHJUrc+Q+bbYAv1VQROAfp fuFn4Vj/N8z/P+eu/4fVR+n/TMm/Ziju94G2B7vEX1lutMwK2zKXDIa6u1jN9dicG45Q zAdw== X-Gm-Message-State: APjAAAWmG8kXfLu7rt3qPk+myw3iGL8BTQlmMUWcfYak3X1TqQJmeW7b nVQjw8wJNszF6Z4xZeFr6Y9ajy5t X-Google-Smtp-Source: APXvYqwVauzkyPShVeuhPhrQW+nWyc/CGdd+OPcgHHe6zgC0wvjj3tjQnlwmh5wMCT7WIRFbd9qP/A== X-Received: by 2002:ac8:6919:: with SMTP id e25mr39885610qtr.317.1567626364386; Wed, 04 Sep 2019 12:46:04 -0700 (PDT) Received: from localhost ([2620:10d:c091:500::3:33b1]) by smtp.gmail.com with ESMTPSA id e17sm11343561qkn.61.2019.09.04.12.46.03 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 04 Sep 2019 12:46:03 -0700 (PDT) From: Tejun Heo To: axboe@kernel.dk Cc: linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, kernel-team@vger.kernel.org, cgroups@vger.kernel.org, newella@fb.com, Tejun Heo Subject: [PATCH 2/5] blk-iocost: Don't let merges push vtime into the future Date: Wed, 4 Sep 2019 12:45:53 -0700 Message-Id: <20190904194556.2984857-3-tj@kernel.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190904194556.2984857-1-tj@kernel.org> References: <20190904194556.2984857-1-tj@kernel.org> Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org Merges have the same problem that forced-bios had which is fixed by the previous patch. The cost of a merge is calculated at the time of issue and force-advances vtime into the future. Until global vtime catches up, how the cgroup's hweight changes in the meantime doesn't matter and it often leads to situations where the cost is calculated at one hweight and paid at a very different one. See the previous patch for more details. Fix it by never advancing vtime into the future for merges. If budget is available, vtime is advanced. Otherwise, the cost is charged as debt. This brings merge cost handling in line with issue cost handling in ioc_rqos_throttle(). Signed-off-by: Tejun Heo --- block/blk-iocost.c | 25 ++++++++++++++++++------- 1 file changed, 18 insertions(+), 7 deletions(-) diff --git a/block/blk-iocost.c b/block/blk-iocost.c index 5b0c024ee5dd..6000ce9b10bb 100644 --- a/block/blk-iocost.c +++ b/block/blk-iocost.c @@ -1784,28 +1784,39 @@ static void ioc_rqos_merge(struct rq_qos *rqos, struct request *rq, struct bio *bio) { struct ioc_gq *iocg = blkg_to_iocg(bio->bi_blkg); + struct ioc *ioc = iocg->ioc; sector_t bio_end = bio_end_sector(bio); + struct ioc_now now; u32 hw_inuse; u64 abs_cost, cost; - /* add iff the existing request has cost assigned */ - if (!rq->bio || !rq->bio->bi_iocost_cost) + /* bypass if disabled or for root cgroup */ + if (!ioc->enabled || !iocg->level) return; abs_cost = calc_vtime_cost(bio, iocg, true); if (!abs_cost) return; + ioc_now(ioc, &now); + current_hweight(iocg, NULL, &hw_inuse); + cost = abs_cost_to_cost(abs_cost, hw_inuse); + /* update cursor if backmerging into the request at the cursor */ if (blk_rq_pos(rq) < bio_end && blk_rq_pos(rq) + blk_rq_sectors(rq) == iocg->cursor) iocg->cursor = bio_end; - current_hweight(iocg, NULL, &hw_inuse); - cost = div64_u64(abs_cost * HWEIGHT_WHOLE, hw_inuse); - bio->bi_iocost_cost = cost; - - atomic64_add(cost, &iocg->vtime); + /* + * Charge if there's enough vtime budget and the existing request + * has cost assigned. Otherwise, account it as debt. See debt + * handling in ioc_rqos_throttle() for details. + */ + if (rq->bio && rq->bio->bi_iocost_cost && + time_before_eq64(atomic64_read(&iocg->vtime) + cost, now.vnow)) + iocg_commit_bio(iocg, bio, cost); + else + atomic64_add(abs_cost, &iocg->abs_vdebt); } static void ioc_rqos_done_bio(struct rq_qos *rqos, struct bio *bio) From patchwork Wed Sep 4 19:45:54 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tejun Heo X-Patchwork-Id: 11131307 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 770D313B1 for ; Wed, 4 Sep 2019 19:46:10 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 5364F2339E for ; Wed, 4 Sep 2019 19:46:10 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1567626370; bh=rEUT35aSw14M/bJYzLui66b7CBo3bPD5cPsfeZvG0hQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:List-ID:From; b=JKxmZ7edZeLXVTzA8pZB05o2kjV6pdcI1EN5pgMNgxzmqnsg8mynN8y4rddqAMDAR +IM4ivakcox5W+fowjziORHuU34Eq8dS3wgAsRYNMipI4jQRkjBzO/oFuBdsw6NHtt BAIUP2X/IPSWYBN2PSRUkDxibqN9ZrnvVxaqhLfs= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730214AbfIDTqI (ORCPT ); Wed, 4 Sep 2019 15:46:08 -0400 Received: from mail-qt1-f193.google.com ([209.85.160.193]:42388 "EHLO mail-qt1-f193.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730185AbfIDTqI (ORCPT ); Wed, 4 Sep 2019 15:46:08 -0400 Received: by mail-qt1-f193.google.com with SMTP id c9so492911qth.9; Wed, 04 Sep 2019 12:46:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:from:to:cc:subject:date:message-id:in-reply-to:references; bh=TRsIsTqTTaD4KvJ5qqrHTq3GaA09a3g8vTgRyaRa+1g=; b=Cy/rW5q/7uBhRpjIRkte/wVsq531S30qrwpCgl8Fl8aWv8B9RYhNXyyRJUdYJFVqZ1 hQZurp7uO2Ep5E4LHaqNjkNXCVTnByOGz+o/2DwqUkKkljYGXsxPdQu5Upwnr+mCe/Vr Dgpu0m62CwYkw9XgXFso9Q5d1PsdjZmqklzoyxyaSzR3Guh1bx2usmf0HSL0D/ZAaHZL jMxtBEzmlq/mwV41VVXfRowouGAC/lMkxV+Jq/Ra1kqJbmcyVpk8oFHX0miW/qm2Q1Do bqrEjPahv/I/K/sCSnaYdVBmWT4K4rug7ZkIbuuOzvCOF0Xy0I2Mh/YomkF5TTHovzXk rFng== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:from:to:cc:subject:date:message-id :in-reply-to:references; bh=TRsIsTqTTaD4KvJ5qqrHTq3GaA09a3g8vTgRyaRa+1g=; b=ULOrx5RkCZ1hEInKFbhbtPzXnkmaHIY94SZLueuzb8JSulXmW/9ubCeU2VD3w/cEGe st55rxcHMTd3sQb8Zr9pvksqno0K5QbXZqhSyHax2WdQa9sY2fgUkHpIZC8pxFj+X+Th WKz3RXJcK1H32TSzRHxxgbQsqh9QHUV2viZIBs1V5YnKzLfVUY3rF6Qb3AwW6I4MH/et kA/V7nr46wh/z9eUM8IRil4l10iQyteMuE5phlc8tYyUsC14hvQT/NUIHX8wCqh8ll0W 1BWTsVMamKiUKrzRJ51z+1DAxczxwdETmtIx4CFH5ZZlh4MArtA9w3wqEf1V7zt3M6+c g6BA== X-Gm-Message-State: APjAAAUlKqkkWUwDCkla2SZJcuH2cFbSpLidD5Bd12ql6zebQaJlCjpA ji7avsX0X7BR6Xawt1P22t0= X-Google-Smtp-Source: APXvYqxpAqeBUOEQ8CYHvJNOvr4kq5J0rhwF1VSB2xCy2s8AAwMhbesfB2T/fBUPNkojbGTUKjEFTQ== X-Received: by 2002:ac8:414b:: with SMTP id e11mr41201878qtm.174.1567626366859; Wed, 04 Sep 2019 12:46:06 -0700 (PDT) Received: from localhost ([2620:10d:c091:500::3:33b1]) by smtp.gmail.com with ESMTPSA id z7sm9114582qka.72.2019.09.04.12.46.05 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 04 Sep 2019 12:46:06 -0700 (PDT) From: Tejun Heo To: axboe@kernel.dk Cc: linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, kernel-team@vger.kernel.org, cgroups@vger.kernel.org, newella@fb.com, Tejun Heo Subject: [PATCH 3/5] iocost_monitor: Always use strings for json values Date: Wed, 4 Sep 2019 12:45:54 -0700 Message-Id: <20190904194556.2984857-4-tj@kernel.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190904194556.2984857-1-tj@kernel.org> References: <20190904194556.2984857-1-tj@kernel.org> Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org Json has limited accuracy for numbers and can silently truncate 64bit values, which can be extremely confusing. Let's consistently use string encapsulated values for json output. While at it, convert an unnecesary f-string to str(). Signed-off-by: Tejun Heo --- tools/cgroup/iocost_monitor.py | 40 +++++++++++++++++----------------- 1 file changed, 20 insertions(+), 20 deletions(-) diff --git a/tools/cgroup/iocost_monitor.py b/tools/cgroup/iocost_monitor.py index 2c9445e966d8..8f6b4ac377bd 100644 --- a/tools/cgroup/iocost_monitor.py +++ b/tools/cgroup/iocost_monitor.py @@ -111,14 +111,14 @@ autop_names = { def dict(self, now): return { 'device' : devname, - 'timestamp' : now, - 'enabled' : self.enabled, - 'running' : self.running, - 'period_ms' : self.period_ms, - 'period_at' : self.period_at, - 'period_vtime_at' : self.vperiod_at, - 'busy_level' : self.busy_level, - 'vrate_pct' : self.vrate_pct, } + 'timestamp' : str(now), + 'enabled' : str(int(self.enabled)), + 'running' : str(int(self.running)), + 'period_ms' : str(self.period_ms), + 'period_at' : str(self.period_at), + 'period_vtime_at' : str(self.vperiod_at), + 'busy_level' : str(self.busy_level), + 'vrate_pct' : str(self.vrate_pct), } def table_preamble_str(self): state = ('RUN' if self.running else 'IDLE') if self.enabled else 'OFF' @@ -171,19 +171,19 @@ autop_names = { def dict(self, now, path): out = { 'cgroup' : path, - 'timestamp' : now, - 'is_active' : self.is_active, - 'weight' : self.weight, - 'weight_active' : self.active, - 'weight_inuse' : self.inuse, - 'hweight_active_pct' : self.hwa_pct, - 'hweight_inuse_pct' : self.hwi_pct, - 'inflight_pct' : self.inflight_pct, - 'use_delay' : self.use_delay, - 'delay_ms' : self.delay_ms, - 'usage_pct' : self.usage } + 'timestamp' : str(now), + 'is_active' : str(int(self.is_active)), + 'weight' : str(self.weight), + 'weight_active' : str(self.active), + 'weight_inuse' : str(self.inuse), + 'hweight_active_pct' : str(self.hwa_pct), + 'hweight_inuse_pct' : str(self.hwi_pct), + 'inflight_pct' : str(self.inflight_pct), + 'use_delay' : str(self.use_delay), + 'delay_ms' : str(self.delay_ms), + 'usage_pct' : str(self.usage) } for i in range(len(self.usages)): - out[f'usage_pct_{i}'] = f'{self.usages[i]}' + out[f'usage_pct_{i}'] = str(self.usages[i]) return out def table_row_str(self, path): From patchwork Wed Sep 4 19:45:55 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tejun Heo X-Patchwork-Id: 11131309 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id C3CEF1395 for ; Wed, 4 Sep 2019 19:46:14 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id A2CAC22CED for ; Wed, 4 Sep 2019 19:46:14 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1567626374; bh=tH3kXeIKpW2pNLjHyed4Whx2+7/Kx6O3RjakgddoqyE=; h=From:To:Cc:Subject:Date:In-Reply-To:References:List-ID:From; b=Q61YTJ3K65qP1DOooGCd2oBuLqdMVRPrzaSvtAbtW14xoby84gcPxHh2C0z+z3Awu tEm5YOZbHhgGM8LpRlFoaGsZJNZDt4rfvPehd2ITFmmRoo5xE5MUhZ5NFDUSYqE4lo dm2XD1qD3iyKG6azoXU3muBz0eFBJgMCmt1cUOSU= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730290AbfIDTqM (ORCPT ); Wed, 4 Sep 2019 15:46:12 -0400 Received: from mail-qk1-f196.google.com ([209.85.222.196]:37153 "EHLO mail-qk1-f196.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730232AbfIDTqK (ORCPT ); Wed, 4 Sep 2019 15:46:10 -0400 Received: by mail-qk1-f196.google.com with SMTP id s14so20854706qkm.4; Wed, 04 Sep 2019 12:46:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:from:to:cc:subject:date:message-id:in-reply-to:references; bh=iI9MwVhDc7RAqkPvomtO9iKQGF6mvrirQ+L7ZaA22N4=; b=vIyNi+SSpBFV02XHRcFiU9QGtz6rnHbfRIsKoGrthXgk0sdQVAXBI+WeCYlcYrFY9B eB9CTcGogSly1Aa92JWc/hMa04vsKdDocBNkcfP0pLj2rSC8LLgDtQUbDRGEwqQhTug2 s7kNEpY6sjbPd42tv/iLO2VKb9Ks5j3Ly2zqsmuMsCwGPQsLyfBeTS7tHJAtlJZmdiAn yhEHs5pEysE9NGXmG2R+48Rfw/qZrPT2MPxCKQ25clVabwpMfcxR6YguiXsPC4UdSapl ZR2cwgxvYBXFRQvjqYtfi7msm+LLM5x5SaD/dKpAi4YgZS4uQu6Y2qF5oC5vRTOQwZU6 wE0A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:from:to:cc:subject:date:message-id :in-reply-to:references; bh=iI9MwVhDc7RAqkPvomtO9iKQGF6mvrirQ+L7ZaA22N4=; b=YrANn/Lou+wYLhzcbrWkEGeA1n3kmQ7i8rEwWcRF+fdWX2mamz0dnM06UbjaNmUe1b 81L7sGD7njiS0uF/lu7e8/SrUZU7m0iR4YWDD6XXh54WodjC+Xm9vpt3Q6Ny6qwt8Unb JCAIKg8HBBSKslUvy4ymknRcatNQArT+h/vQwf8X00NsKRYsVDspzkpwDMFq7//j5JW2 PimuxqWkifVhSAMzGWukUNuN4Vcrm9tiJebYxqKBrvhCOEQsjdQeBtXemezDSAYa8yqt klAOEOhipIXLm/fUqJE74HJJjeVvaCDftyKuslFCgdBIzLVrVscXCSztMtmlqvalPlCZ BEKA== X-Gm-Message-State: APjAAAU1pgYg5Dpw4adQrZr0gCpOZXqaHvgmyqE1qgxLNsP3TM5Lf5C8 i0AkayCAmCmQp1biHltTruE= X-Google-Smtp-Source: APXvYqxmmaj1DbMhYCY30sd/yTDtXcWZVCbrlR37ssZllxSIxdbtRX+bARbUlfZQSD6KbztFYRVASg== X-Received: by 2002:a37:afc6:: with SMTP id y189mr41691870qke.7.1567626369152; Wed, 04 Sep 2019 12:46:09 -0700 (PDT) Received: from localhost ([2620:10d:c091:500::3:33b1]) by smtp.gmail.com with ESMTPSA id b4sm7798573qkd.121.2019.09.04.12.46.08 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 04 Sep 2019 12:46:08 -0700 (PDT) From: Tejun Heo To: axboe@kernel.dk Cc: linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, kernel-team@vger.kernel.org, cgroups@vger.kernel.org, newella@fb.com, Tejun Heo Subject: [PATCH 4/5] iocost_monitor: Report more info with higher accuracy Date: Wed, 4 Sep 2019 12:45:55 -0700 Message-Id: <20190904194556.2984857-5-tj@kernel.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190904194556.2984857-1-tj@kernel.org> References: <20190904194556.2984857-1-tj@kernel.org> Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org When outputting json: * Don't truncate numbers. * Report address of iocg to ease drilling down further. When outputting table: * Use math.ceil() for delay_ms so that small delays don't read as 0. Signed-off-by: Tejun Heo --- tools/cgroup/iocost_monitor.py | 18 +++++++++++------- 1 file changed, 11 insertions(+), 7 deletions(-) diff --git a/tools/cgroup/iocost_monitor.py b/tools/cgroup/iocost_monitor.py index 8f6b4ac377bd..5d8bac603ffa 100644 --- a/tools/cgroup/iocost_monitor.py +++ b/tools/cgroup/iocost_monitor.py @@ -13,6 +13,7 @@ import sys import re import time import json +import math import drgn from drgn import container_of @@ -95,7 +96,7 @@ autop_names = { self.enabled = ioc.enabled.value_() self.running = ioc.running.value_() == IOC_RUNNING - self.period_ms = round(ioc.period_us.value_() / 1_000) + self.period_ms = ioc.period_us.value_() / 1_000 self.period_at = ioc.period_at.value_() / 1_000_000 self.vperiod_at = ioc.period_at_vtime.value_() / VTIME_PER_SEC self.vrate_pct = ioc.vtime_rate.counter.value_() * 100 / VTIME_PER_USEC @@ -147,6 +148,7 @@ autop_names = { self.inuse = iocg.inuse.value_() self.hwa_pct = iocg.hweight_active.value_() * 100 / HWEIGHT_WHOLE self.hwi_pct = iocg.hweight_inuse.value_() * 100 / HWEIGHT_WHOLE + self.address = iocg.value_() vdone = iocg.done_vtime.counter.value_() vtime = iocg.vtime.counter.value_() @@ -157,15 +159,15 @@ autop_names = { else: self.inflight_pct = 0 - self.use_delay = min(blkg.use_delay.counter.value_(), 99) - self.delay_ms = min(round(blkg.delay_nsec.counter.value_() / 1_000_000), 999) + self.use_delay = blkg.use_delay.counter.value_() + self.delay_ms = blkg.delay_nsec.counter.value_() / 1_000_000 usage_idx = iocg.usage_idx.value_() self.usages = [] self.usage = 0 for i in range(NR_USAGE_SLOTS): usage = iocg.usages[(usage_idx + i) % NR_USAGE_SLOTS].value_() - upct = min(usage * 100 / HWEIGHT_WHOLE, 999) + upct = usage * 100 / HWEIGHT_WHOLE self.usages.append(upct) self.usage = max(self.usage, upct) @@ -181,7 +183,8 @@ autop_names = { 'inflight_pct' : str(self.inflight_pct), 'use_delay' : str(self.use_delay), 'delay_ms' : str(self.delay_ms), - 'usage_pct' : str(self.usage) } + 'usage_pct' : str(self.usage), + 'address' : str(hex(self.address)) } for i in range(len(self.usages)): out[f'usage_pct_{i}'] = str(self.usages[i]) return out @@ -192,9 +195,10 @@ autop_names = { f'{self.inuse:5}/{self.active:5} ' \ f'{self.hwi_pct:6.2f}/{self.hwa_pct:6.2f} ' \ f'{self.inflight_pct:6.2f} ' \ - f'{self.use_delay:2}*{self.delay_ms:03} ' + f'{min(self.use_delay, 99):2}*'\ + f'{min(math.ceil(self.delay_ms), 999):03} ' for u in self.usages: - out += f'{round(u):03d}:' + out += f'{min(round(u), 999):03d}:' out = out.rstrip(':') return out From patchwork Wed Sep 4 19:45:56 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tejun Heo X-Patchwork-Id: 11131311 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 80CC713B1 for ; Wed, 4 Sep 2019 19:46:24 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 6200822DBF for ; Wed, 4 Sep 2019 19:46:24 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1567626384; bh=C45AUngWHtNaVWGA8Rh6aD9tJ05Bcdujt4ucft/pUoY=; h=From:To:Cc:Subject:Date:In-Reply-To:References:List-ID:From; b=dIeeJP2uQ0XMMzPyBXTGnefsJIgkjFXQb59yNSS/rQm4H6ugGRl6JLwuHb72GxiPU 5auKX++Pn10SoaDzpDSenwAMxveJ4wHFnxByqVnaRLVVG18+mKPyttVU9ft/JyqX86 KR5yhy7nSfGViCTK1DxGF7WxmqFLG/doEoG8Kn60= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729398AbfIDTqX (ORCPT ); Wed, 4 Sep 2019 15:46:23 -0400 Received: from mail-qt1-f171.google.com ([209.85.160.171]:38247 "EHLO mail-qt1-f171.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730310AbfIDTqN (ORCPT ); Wed, 4 Sep 2019 15:46:13 -0400 Received: by mail-qt1-f171.google.com with SMTP id b2so22337557qtq.5; Wed, 04 Sep 2019 12:46:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:from:to:cc:subject:date:message-id:in-reply-to:references; bh=jL4d81pypGq7ODywNqxdQtdPeeIUOLBTqveh64eDASc=; b=Z0oBR0eUEqPyun6hEblYCICjdKNvWRr+b29/35hl2k9IPCpnLWYk0YRY/8L5vtThT9 rqb7sJxaWR3X8K4fC+O5IkrzkOK1cnraVOQlHpbAUryTpGEd8x1V2MF9f4AKzgabbPN4 RURLEtu/yEDEf+b64CY/JEYDCcpg0DIRHptVcP+sl4fVElmdNupPsKEc1YTszbljredC 6QJXBlhEqOK2ua2rUmzwgc0UOAFR/+1fjYgDmuwXttlJgNlzx8QdHsY1mQ6Iq345qlJM /UXcMmB/pCl4//OeGXfgwloUyMdpYFvINjlHOxBwvbqzG6c++Nio0VNFmvhBGqoSgXN2 FXyQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:from:to:cc:subject:date:message-id :in-reply-to:references; bh=jL4d81pypGq7ODywNqxdQtdPeeIUOLBTqveh64eDASc=; b=KGVq1t6PfGSA3cCiii1+KgvFKr18TEva/tAjDnPwEtwqeJkmuSUSJaU0v/kfTikWT4 9foY5q7JVd6itxIbOfFxhu+p0fXpicOHD6/dzuJ/hTi+fdSpJoxRJwOW9oaCA5wWG4Kw 3arNcom5km4tuaKTnt2sR3UuCbYmHRGOkUcXMfrlkHScaUp0sAx3jL4XbqJONZmpXKXU wBytDpeMesyBjAO0+0L0QWPmnVEA3iGeABWVxjt5vmX+m1mGZFzfcPuGjIlZh97Zf258 WWddOhD/UosPLoja+Qllqd06q+O5qTuMnVVlzCgtVEQb74n4NWa8T8GZGc5T4LASKupy wqLg== X-Gm-Message-State: APjAAAWWJgPxudWG6NSeTWJh3IpMB6QqTORGzHJGGU7uIlzy1x8sxdCx slAJOOjcZre9wU+3Ye/zfwM= X-Google-Smtp-Source: APXvYqy3B8uCoovpU+dggZaKK6MmRowSu3/rxq1PqkkYanMHQXH3mw68IiIC36wBDFPsVTXt4ivzVA== X-Received: by 2002:a0c:d084:: with SMTP id z4mr17865362qvg.63.1567626371193; Wed, 04 Sep 2019 12:46:11 -0700 (PDT) Received: from localhost ([2620:10d:c091:500::3:33b1]) by smtp.gmail.com with ESMTPSA id w15sm107958qtt.71.2019.09.04.12.46.10 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 04 Sep 2019 12:46:10 -0700 (PDT) From: Tejun Heo To: axboe@kernel.dk Cc: linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, kernel-team@vger.kernel.org, cgroups@vger.kernel.org, newella@fb.com, Tejun Heo Subject: [PATCH 5/5] iocost_monitor: Report debt Date: Wed, 4 Sep 2019 12:45:56 -0700 Message-Id: <20190904194556.2984857-6-tj@kernel.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190904194556.2984857-1-tj@kernel.org> References: <20190904194556.2984857-1-tj@kernel.org> Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org Report debt and rename del_ms row to delay for consistency. Signed-off-by: Tejun Heo --- block/blk-iocost.c | 6 +++--- tools/cgroup/iocost_monitor.py | 5 ++++- 2 files changed, 7 insertions(+), 4 deletions(-) diff --git a/block/blk-iocost.c b/block/blk-iocost.c index 6000ce9b10bb..f0c5bfd4b4a8 100644 --- a/block/blk-iocost.c +++ b/block/blk-iocost.c @@ -161,9 +161,9 @@ * https://github.com/osandov/drgn. The ouput looks like the following. * * sdb RUN per=300ms cur_per=234.218:v203.695 busy= +1 vrate= 62.12% - * active weight hweight% inflt% del_ms usages% - * test/a * 50/ 50 33.33/ 33.33 27.65 0*041 033:033:033 - * test/b * 100/ 100 66.67/ 66.67 17.56 0*000 066:079:077 + * active weight hweight% inflt% dbt delay usages% + * test/a * 50/ 50 33.33/ 33.33 27.65 2 0*041 033:033:033 + * test/b * 100/ 100 66.67/ 66.67 17.56 0 0*000 066:079:077 * * - per : Timer period * - cur_per : Internal wall and device vtime clock diff --git a/tools/cgroup/iocost_monitor.py b/tools/cgroup/iocost_monitor.py index 5d8bac603ffa..f79b23582a1d 100644 --- a/tools/cgroup/iocost_monitor.py +++ b/tools/cgroup/iocost_monitor.py @@ -135,7 +135,7 @@ autop_names = { def table_header_str(self): return f'{"":25} active {"weight":>9} {"hweight%":>13} {"inflt%":>6} ' \ - f'{"del_ms":>6} {"usages%"}' + f'{"dbt":>3} {"delay":>6} {"usages%"}' class IocgStat: def __init__(self, iocg): @@ -159,6 +159,7 @@ autop_names = { else: self.inflight_pct = 0 + self.debt_ms = iocg.abs_vdebt.counter.value_() / VTIME_PER_USEC / 1000 self.use_delay = blkg.use_delay.counter.value_() self.delay_ms = blkg.delay_nsec.counter.value_() / 1_000_000 @@ -181,6 +182,7 @@ autop_names = { 'hweight_active_pct' : str(self.hwa_pct), 'hweight_inuse_pct' : str(self.hwi_pct), 'inflight_pct' : str(self.inflight_pct), + 'debt_ms' : str(self.debt_ms), 'use_delay' : str(self.use_delay), 'delay_ms' : str(self.delay_ms), 'usage_pct' : str(self.usage), @@ -195,6 +197,7 @@ autop_names = { f'{self.inuse:5}/{self.active:5} ' \ f'{self.hwi_pct:6.2f}/{self.hwa_pct:6.2f} ' \ f'{self.inflight_pct:6.2f} ' \ + f'{min(math.ceil(self.debt_ms), 999):3} ' \ f'{min(self.use_delay, 99):2}*'\ f'{min(math.ceil(self.delay_ms), 999):03} ' for u in self.usages: