From patchwork Mon Mar 27 17:51:39 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Shaohua Li X-Patchwork-Id: 9647289 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 4154D60349 for ; Mon, 27 Mar 2017 17:54:06 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 33F43283E1 for ; Mon, 27 Mar 2017 17:54:06 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 32EA12840B; Mon, 27 Mar 2017 17:54:06 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, RCVD_IN_DNSWL_HI autolearn=unavailable version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id BAD60283FB for ; Mon, 27 Mar 2017 17:54:05 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751924AbdC0Rxz (ORCPT ); Mon, 27 Mar 2017 13:53:55 -0400 Received: from mx0b-00082601.pphosted.com ([67.231.153.30]:53653 "EHLO mx0a-00082601.pphosted.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1751564AbdC0RxV (ORCPT ); Mon, 27 Mar 2017 13:53:21 -0400 Received: from pps.filterd (m0089730.ppops.net [127.0.0.1]) by m0089730.ppops.net (8.16.0.20/8.16.0.20) with SMTP id v2RHhRae023001 for ; Mon, 27 Mar 2017 10:51:49 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-type; s=facebook; bh=2FnlrDnVXCFBLzDe4gdKOCu/1y0dRe//3V8qgkeGTNw=; b=pyZvd6NsHF1ajHz55Bu6HUqyNWgKQgBxmJ+soGwicskKigRLE16E3gm23kC0tsb3NFNm Ovzgg0k5AJpiIrma/GEviN34TJMy8iEHcmFra+3tsit/tz3WPwkJI82T154O/T4qfAyt SJiZ/UnVPD+KU9DQdDRvq9HD0GRF5+AiJ7Q= Received: from mail.thefacebook.com ([199.201.64.23]) by m0089730.ppops.net with ESMTP id 29f66w0ar5-5 (version=TLSv1 cipher=ECDHE-RSA-AES256-SHA bits=256 verify=NOT) for ; Mon, 27 Mar 2017 10:51:49 -0700 Received: from mx-out.facebook.com (192.168.52.123) by PRN-CHUB01.TheFacebook.com (192.168.16.11) with Microsoft SMTP Server (TLS) id 14.3.319.2; Mon, 27 Mar 2017 10:51:47 -0700 Received: from facebook.com (2401:db00:21:603d:face:0:19:0) by mx-out.facebook.com (10.102.107.99) with ESMTP id 0c0b004a131611e7a29c0002c99293a0-bc9fd9a0 for ; Mon, 27 Mar 2017 10:51:47 -0700 Received: by devbig638.prn2.facebook.com (Postfix, from userid 11222) id EA8BB43A3BAE; Mon, 27 Mar 2017 10:51:46 -0700 (PDT) Smtp-Origin-Hostprefix: devbig From: Shaohua Li Smtp-Origin-Hostname: devbig638.prn2.facebook.com To: , CC: , , Vivek Goyal , , Smtp-Origin-Cluster: prn2c22 Subject: [PATCH V7 11/18] blk-throttle: detect completed idle cgroup Date: Mon, 27 Mar 2017 10:51:39 -0700 Message-ID: X-Mailer: git-send-email 2.9.3 In-Reply-To: References: X-FB-Internal: Safe MIME-Version: 1.0 X-Proofpoint-Spam-Reason: safe X-FB-Internal: Safe X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:, , definitions=2017-03-27_16:, , signatures=0 Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP cgroup could be assigned a limit, but doesn't dispatch enough IO, eg the cgroup is idle. When this happens, the cgroup doesn't hit its limit, so we can't move the state machine to higher level and all cgroups will be throttled to their lower limit, so we waste bandwidth. Detecting idle cgroup is hard. This patch handles a simple case, a cgroup doesn't dispatch any IO. We ignore such cgroup's limit, so other cgroups can use the bandwidth. Please note this will be replaced with a more sophisticated algorithm later, but this demonstrates the idea how we handle idle cgroups, so I leave it here. Signed-off-by: Shaohua Li --- block/blk-throttle.c | 19 ++++++++++++++++++- 1 file changed, 18 insertions(+), 1 deletion(-) diff --git a/block/blk-throttle.c b/block/blk-throttle.c index d00c1c1..014b2e9 100644 --- a/block/blk-throttle.c +++ b/block/blk-throttle.c @@ -149,6 +149,8 @@ struct throtl_grp { unsigned long last_check_time; + unsigned long last_dispatch_time[2]; + /* When did we start a new slice */ unsigned long slice_start[2]; unsigned long slice_end[2]; @@ -445,11 +447,14 @@ static void tg_update_has_rules(struct throtl_grp *tg) static void throtl_pd_online(struct blkg_policy_data *pd) { + struct throtl_grp *tg = pd_to_tg(pd); /* * We don't want new groups to escape the limits of its ancestors. * Update has_rules[] after a new group is brought online. */ - tg_update_has_rules(pd_to_tg(pd)); + tg_update_has_rules(tg); + tg->last_dispatch_time[READ] = jiffies; + tg->last_dispatch_time[WRITE] = jiffies; } static void blk_throtl_update_limit_valid(struct throtl_data *td) @@ -1615,6 +1620,12 @@ static bool throtl_tg_can_upgrade(struct throtl_grp *tg) if (write_limit && sq->nr_queued[WRITE] && (!read_limit || sq->nr_queued[READ])) return true; + + if (time_after_eq(jiffies, + tg->last_dispatch_time[READ] + tg->td->throtl_slice) && + time_after_eq(jiffies, + tg->last_dispatch_time[WRITE] + tg->td->throtl_slice)) + return true; return false; } @@ -1692,6 +1703,11 @@ static bool throtl_tg_can_downgrade(struct throtl_grp *tg) struct throtl_data *td = tg->td; unsigned long now = jiffies; + if (time_after_eq(now, tg->last_dispatch_time[READ] + + td->throtl_slice) && + time_after_eq(now, tg->last_dispatch_time[WRITE] + + td->throtl_slice)) + return false; /* * If cgroup is below low limit, consider downgrade and throttle other * cgroups @@ -1800,6 +1816,7 @@ bool blk_throtl_bio(struct request_queue *q, struct blkcg_gq *blkg, again: while (true) { + tg->last_dispatch_time[rw] = jiffies; if (tg->last_low_overflow_time[rw] == 0) tg->last_low_overflow_time[rw] = jiffies; throtl_downgrade_check(tg);