From patchwork Wed Sep 25 23:02:07 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tejun Heo X-Patchwork-Id: 11161691 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 58113912 for ; Wed, 25 Sep 2019 23:02:28 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 2DA1D21D7B for ; Wed, 25 Sep 2019 23:02:28 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1569452548; bh=aF5eWj/Bp4MAgMXMsU/tILGukQe7wEsm+krU50KJ4aM=; h=Date:From:To:Cc:Subject:List-ID:From; b=2IxXubNxVheoFzAJ1PMEGbp+Gg7ZZq/B4U4j1765TK18lgvwgHxxMCoYIaLULDiAZ 6AFNGva8nXoFWrKkAgwtiPNH0YLxyubiDewlcFDc6+TXVZ27JGwD86rzH2YZlvtUHw 5Net5Jffu+PRYl5f+a9OD58slxP5UUPQvqun8ndc= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727368AbfIYXCY (ORCPT ); Wed, 25 Sep 2019 19:02:24 -0400 Received: from mail-qk1-f194.google.com ([209.85.222.194]:43125 "EHLO mail-qk1-f194.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726921AbfIYXCY (ORCPT ); Wed, 25 Sep 2019 19:02:24 -0400 Received: by mail-qk1-f194.google.com with SMTP id h126so195627qke.10; Wed, 25 Sep 2019 16:02:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:date:from:to:cc:subject:message-id:mime-version :content-disposition:user-agent; bh=J6xx84NYBZMzyAcWsbF/M7RN9YSA/RHntuLBn1Nqazg=; b=SIfEzvpvUIu7/aCIY6SH2SkyCjOd9o0i5PqDvqyQHvE2InvTaTZQsVQCaxK5q1n/Cj yumJMyFWRX5aLUZd6WU9ijwVcG7VOjbiBe83nr6b6ydDBgYuXARTJf9DFAPC08+spzNo lUUQILezVaRYjlSnhG9eGY0RP1Hd2B2Vt9F8TaDReIT63toDYPKKzmTKEbTR868/tRhG AlAKT1UT4L82ewB5aZ1Sz017fFwxNLGqgxsyPmzjD7ZzoHFV7NWhiDpx9tVnBa86/yP+ YgSGjq91jgvX1ZBsL/sJT2p9Dmg68iJXvjHCqi5Yt4gQ8UoZIivfDeKn7jA2r66/x0R5 ou0g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:from:to:cc:subject:message-id :mime-version:content-disposition:user-agent; bh=J6xx84NYBZMzyAcWsbF/M7RN9YSA/RHntuLBn1Nqazg=; b=hPAp7SNGvOncwU8XdsEPoAA35NumTULvoxTqdRAyIx4/m2FizTJelAWE7Y6L0jSO0q EWa5CFV3blOH3eBtHh1H6T9jC+da9qJ+XKoSq5aYsFQC7wfU5RPL/Mto58BBJ4e0U4c8 xPCJec/3mdfax7D/sLsRhl0mTvdqDVX0lAhdxyxF+1n86UEwen29KXcxl8RhQzbQUNFN 5+KN9rJRCPxNf3DgH2niORDrY4oc5uDHLKYKtaY2qHuyMXkAPw96NyXiNKkWGTvY4IPN xXaAuFN0rDPHtlDStppFZ1DFAxJGTtwODzPh0tzyvsJ6ntl6l6ds5B7bzmyATTNET8ES hMeg== X-Gm-Message-State: APjAAAWfKNdytNvl1XH6ZcOqZcupAD/LUMtK+Hl9sD2R2gZRYsxjkmYn fZLKYkgHRgEyTe8iTzf92Yg= X-Google-Smtp-Source: APXvYqzlN0SByh/FHkgjqHVCuxYeLgWE2FPBsrHJAD9J0BHXXCNAE+h7pH/AOECmnivaxcM7ltQ9Ow== X-Received: by 2002:a37:ef11:: with SMTP id j17mr391810qkk.1.1569452542573; Wed, 25 Sep 2019 16:02:22 -0700 (PDT) Received: from localhost ([2620:10d:c091:500::3:a001]) by smtp.gmail.com with ESMTPSA id c41sm301087qte.8.2019.09.25.16.02.20 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 25 Sep 2019 16:02:20 -0700 (PDT) Date: Wed, 25 Sep 2019 16:02:07 -0700 From: Tejun Heo To: Jens Axboe Cc: linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, kernel-team@fb.com, cgroups@vger.kernel.org Subject: [PATCH 1/3 for-5.4/block] iocost: better trace vrate changes Message-ID: <20190925230207.GI2233839@devbig004.ftw2.facebook.com> MIME-Version: 1.0 Content-Disposition: inline User-Agent: Mutt/1.5.21 (2010-09-15) Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org vrate_adj tracepoint traces vrate changes; however, it does so only when busy_level is non-zero. busy_level turning to zero can sometimes be as interesting an event. This patch also enables vrate_adj tracepoint on other vrate related events - busy_level changes and non-zero nr_lagging. Signed-off-by: Tejun Heo --- Hello, Jens. I've encountered vrate regulation issues while testing on a hard disk machine. These are three patches to improve vrate adj visibility and fix the issue. Thanks. block/blk-iocost.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) --- a/block/blk-iocost.c +++ b/block/blk-iocost.c @@ -1343,7 +1343,7 @@ static void ioc_timer_fn(struct timer_li u32 ppm_wthr = MILLION - ioc->params.qos[QOS_WPPM]; u32 missed_ppm[2], rq_wait_pct; u64 period_vtime; - int i; + int prev_busy_level, i; /* how were the latencies during the period? */ ioc_lat_stat(ioc, missed_ppm, &rq_wait_pct); @@ -1531,6 +1531,7 @@ skip_surplus_transfers: * and experiencing shortages but not surpluses, we're too stingy * and should increase vtime rate. */ + prev_busy_level = ioc->busy_level; if (rq_wait_pct > RQ_WAIT_BUSY_PCT || missed_ppm[READ] > ppm_rthr || missed_ppm[WRITE] > ppm_wthr) { @@ -1592,6 +1593,10 @@ skip_surplus_transfers: atomic64_set(&ioc->vtime_rate, vrate); ioc->inuse_margin_vtime = DIV64_U64_ROUND_UP( ioc->period_us * vrate * INUSE_MARGIN_PCT, 100); + } else if (ioc->busy_level != prev_busy_level || nr_lagging) { + trace_iocost_ioc_vrate_adj(ioc, atomic64_read(&ioc->vtime_rate), + &missed_ppm, rq_wait_pct, nr_lagging, + nr_shortages, nr_surpluses); } ioc_refresh_params(ioc, false); From patchwork Wed Sep 25 23:03:09 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tejun Heo X-Patchwork-Id: 11161693 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id D06EE14DB for ; Wed, 25 Sep 2019 23:03:13 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id AF36421D7E for ; Wed, 25 Sep 2019 23:03:13 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1569452593; bh=TwQJTP4AyYkE4IHVrTInCcWmRbo/3iQm27uvwuSr2Lc=; h=Date:From:To:Cc:Subject:References:In-Reply-To:List-ID:From; b=cqo9R3mEgrX1/Wfl2paSUlCDyUcQnVzMalnIkBspCiuKuknOuLAxaZR6kZh/WjQzD X3Ik3uKKBZwL/j7+SBH8XK4a2EDubLXt6+Aj0XWCxeOgpHFVjbVW0sY1eQWcR+YiRX /3860PUbekEiPxGH8YaB6yqUaQmLEpTxJMMbBE6Q= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727202AbfIYXDN (ORCPT ); Wed, 25 Sep 2019 19:03:13 -0400 Received: from mail-qk1-f196.google.com ([209.85.222.196]:44062 "EHLO mail-qk1-f196.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726206AbfIYXDN (ORCPT ); Wed, 25 Sep 2019 19:03:13 -0400 Received: by mail-qk1-f196.google.com with SMTP id u22so191460qkk.11; Wed, 25 Sep 2019 16:03:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to:user-agent; bh=+2s0oXR538dsUd5mHpXcur3xcbIHPtS2CMh60hp/fzM=; b=lirB5BC8U7A7hYKAMl7kBbxc6YJxcklp0YnCGMGTuWHyBImdAtixJC+AFZylYcBLG2 g8hlcI40h40Ks1Y9AkE+Frij4whfpsgJzSBjJkQDI2LVIGiu+xxdPep+rQK7G2czxNPp bBCx1Vv+ddbACfvLlGig3qz6WPm5mJWE/8nm+uNcagUIluMY3PRYOPIkRzC5R8Pnmgz4 tExhI0IiMCyAblozYrZ0X2Zdk32mmIjfckrx20gSLbRHYbFs+93Ga1G1+UFr5deF89Ta FBcaReTOPRiYyxAoVgH/pMHTmOp9w5zLvQ0Y8lnVKaNuj5J8HkvGZFa5Gd7SMQLVB5X6 lUWQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:from:to:cc:subject:message-id :references:mime-version:content-disposition:in-reply-to:user-agent; bh=+2s0oXR538dsUd5mHpXcur3xcbIHPtS2CMh60hp/fzM=; b=DolbGWu+V6AaKG7c6Uych8cHDN7PSXleBMHjSAWVJr7ROvFk2aWFwpNV+5KWp5B4bW PWyGc8lp0dZ5Q6hM075I5oHrASKHglVjRTD/vfWLatHxDJIOCohN86A3szGe0jYVGt7w B5SVdu9WtB33YGK9hc7LIW48cjCMrxqzR29uQJ7tGxopjSzcp6qAeMiLfQKb+yWI9hGI I2j2s4kI2NW2PJgB0j1Kxrm9Cc9S/2z1GJa/4ToxpPWuPy/b5FilaMPNYnCJXuW/RY8D kpiBzp1SmajmHI5dh1uU7NV02dTMYYtYz7/Up7A+lGBHHIWMtIlNzp06l008p8j4JGXZ tKiA== X-Gm-Message-State: APjAAAVVzKSm9SpCIZOR9wJiK4SNyRQAwQF4uMxPT5ZCGRti64W/sFGr 3bsmVqcWkt+TBbxbuXHK5uY= X-Google-Smtp-Source: APXvYqz6X5CLZiRgdLEkLrgwcksoK5XgwtkoG+UNUHxhzTE3TndXVJwyw7Xkm6jDRldgFVN+VHl8fA== X-Received: by 2002:a05:620a:15d2:: with SMTP id o18mr375515qkm.341.1569452591643; Wed, 25 Sep 2019 16:03:11 -0700 (PDT) Received: from localhost ([2620:10d:c091:500::3:a001]) by smtp.gmail.com with ESMTPSA id b26sm119056qkl.43.2019.09.25.16.03.10 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 25 Sep 2019 16:03:11 -0700 (PDT) Date: Wed, 25 Sep 2019 16:03:09 -0700 From: Tejun Heo To: Jens Axboe Cc: linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, kernel-team@fb.com, cgroups@vger.kernel.org Subject: [PATCH 2/3 for-5.4/block] iocost: improve nr_lagging handling Message-ID: <20190925230309.GJ2233839@devbig004.ftw2.facebook.com> References: <20190925230207.GI2233839@devbig004.ftw2.facebook.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20190925230207.GI2233839@devbig004.ftw2.facebook.com> User-Agent: Mutt/1.5.21 (2010-09-15) Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org Some IOs may span multiple periods. As latencies are collected on completion, the inbetween periods won't register them and may incorrectly decide to increase vrate. nr_lagging tracks these IOs to avoid those situations. Currently, whenever there are IOs which are spanning from the previous period, busy_level is reset to 0 if negative thus suppressing vrate increase. This has the following two problems. * When latency target percentiles aren't set, vrate adjustment should only be governed by queue depth depletion; however, the current code keeps nr_lagging active which pulls in latency results and can keep down vrate unexpectedly. * When lagging condition is detected, it resets the entire negative busy_level. This turned out to be way too aggressive on some devices which sometimes experience extended latencies on a small subset of commands. In addition, a lagging IO will be accounted as latency target miss on completion anyway and resetting busy_level amplifies its impact unnecessarily. This patch fixes the above two problems by disabling nr_lagging counting when latency target percentiles aren't set and blocking vrate increases when there are lagging IOs while leaving busy_level as-is. Signed-off-by: Tejun Heo --- block/blk-iocost.c | 19 +++++++++++-------- 1 file changed, 11 insertions(+), 8 deletions(-) --- a/block/blk-iocost.c +++ b/block/blk-iocost.c @@ -1407,7 +1407,8 @@ static void ioc_timer_fn(struct timer_li * comparing vdone against period start. If lagging behind * IOs from past periods, don't increase vrate. */ - if (!atomic_read(&iocg_to_blkg(iocg)->use_delay) && + if ((ppm_rthr != MILLION || ppm_wthr != MILLION) && + !atomic_read(&iocg_to_blkg(iocg)->use_delay) && time_after64(vtime, vdone) && time_after64(vtime, now.vnow - MAX_LAGGING_PERIODS * period_vtime) && @@ -1537,21 +1538,23 @@ skip_surplus_transfers: missed_ppm[WRITE] > ppm_wthr) { ioc->busy_level = max(ioc->busy_level, 0); ioc->busy_level++; - } else if (nr_lagging) { - ioc->busy_level = max(ioc->busy_level, 0); - } else if (nr_shortages && !nr_surpluses && - rq_wait_pct <= RQ_WAIT_BUSY_PCT * UNBUSY_THR_PCT / 100 && + } else if (rq_wait_pct <= RQ_WAIT_BUSY_PCT * UNBUSY_THR_PCT / 100 && missed_ppm[READ] <= ppm_rthr * UNBUSY_THR_PCT / 100 && missed_ppm[WRITE] <= ppm_wthr * UNBUSY_THR_PCT / 100) { - ioc->busy_level = min(ioc->busy_level, 0); - ioc->busy_level--; + /* take action iff there is contention */ + if (nr_shortages && !nr_lagging) { + ioc->busy_level = min(ioc->busy_level, 0); + /* redistribute surpluses first */ + if (!nr_surpluses) + ioc->busy_level--; + } } else { ioc->busy_level = 0; } ioc->busy_level = clamp(ioc->busy_level, -1000, 1000); - if (ioc->busy_level) { + if (ioc->busy_level > 0 || (ioc->busy_level < 0 && !nr_lagging)) { u64 vrate = atomic64_read(&ioc->vtime_rate); u64 vrate_min = ioc->vrate_min, vrate_max = ioc->vrate_max; From patchwork Wed Sep 25 23:03:35 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tejun Heo X-Patchwork-Id: 11161695 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id BF32314DB for ; Wed, 25 Sep 2019 23:03:42 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 9C8F720640 for ; Wed, 25 Sep 2019 23:03:42 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1569452622; bh=30m8Bj/9Yrhu6rdft1hnYF9b79GkkQHGW8a4CX47k+Q=; h=Date:From:To:Cc:Subject:References:In-Reply-To:List-ID:From; b=zDoluOGVfGT35JyItCloQfbmgSeHkLZf1UK4K+Za3Q17DNIy8/956FPPInnfqGO8i 0/i7r+OTAgxOTHuBxzuq96Wma9uCcnVm0fNEy+XJLedmnqIMLnd8IX8Kn9Idv8Ovuj +kXJCG4r97SBH5+ceCv+ZwcKAmc62jKzVFzl7JLA= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730614AbfIYXDi (ORCPT ); Wed, 25 Sep 2019 19:03:38 -0400 Received: from mail-qt1-f195.google.com ([209.85.160.195]:39666 "EHLO mail-qt1-f195.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726206AbfIYXDi (ORCPT ); Wed, 25 Sep 2019 19:03:38 -0400 Received: by mail-qt1-f195.google.com with SMTP id n7so535466qtb.6; Wed, 25 Sep 2019 16:03:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to:user-agent; bh=WfbXh+sodiUMPQSO5yCeqiJJDmHL2yMhTTI21O9c//4=; b=UrO+9FhI4EIs4jI4yXfKf1L+DbuNMlCxcVZ3jN5cwJ77HfdAjqNgQ6FQPxLj6MbdI5 EQ/qM8/S1CQq7aJbPc4BEHeXaKa0t8kWWT5CI/Ss8KJpfdfe8ErjJVNEUNWUDkWlTdu7 CHchGLYEiaLnnldmMtFmLQKgaZqfp6lWYUGq4/CRCyN96qPXjBLljijuJ4/6rOpNNE/B P/9roPgUV+Yc+bU/97dcTbYlgJnckw5Lls8VQSeJnLllwEXdKFmD+ONt+yzarNqHB0kZ WhlCKuVX9JQyMt2nUCrCRDD5K/0JcLluu+G+Yq3HIEN3rNKrvMOGWYzCiyxaKhpZuEsy Of9w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:from:to:cc:subject:message-id :references:mime-version:content-disposition:in-reply-to:user-agent; bh=WfbXh+sodiUMPQSO5yCeqiJJDmHL2yMhTTI21O9c//4=; b=ekAeYulb6w4JboYVzWkX/GBojQ73c5G1/mDiGRmwFxIdmx4zZ+s9vktB59bfSQPSfK ue/0atMEHabcS8kUt0dlRJBB6V2ZMjNMiln0vxCqv8wxE2foJmMhFjTsHojV6EhKwLeU IFvzLS0BX9S1HvKxSjy29yKODkZhuaN6lUx7Gv8riU8lbEeQCn8RjPXAlxm7MhCTth0K CiUZRY8iEsmi8rZmWgJ/1KjX93uhW+k8uj0EWgLDuAuTtBa/mQOHtCZoVQcXyXOhI7rW QVitvmqzbdIoPnraESfM/H7v5uq3M7tRVrKrGAvsnDtKb0KrU+/WrSNRyzCiPCrenMnt nZqw== X-Gm-Message-State: APjAAAWg4QhdiWFSaEkCwx5xefZdpcDDdEBaEaSg7+BfTjmoO94Ndi6C 0HT8K9LqdV+AAJWmCMh2vnw= X-Google-Smtp-Source: APXvYqyC1FyjGChJ2Qd1VW5+w/5vvUgYmAgGVElejt3WCWCezoA05O48V25SM3ipL5VdElk9+zyedw== X-Received: by 2002:ac8:100b:: with SMTP id z11mr953244qti.377.1569452617529; Wed, 25 Sep 2019 16:03:37 -0700 (PDT) Received: from localhost ([2620:10d:c091:500::3:a001]) by smtp.gmail.com with ESMTPSA id u43sm265769qte.19.2019.09.25.16.03.36 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 25 Sep 2019 16:03:37 -0700 (PDT) Date: Wed, 25 Sep 2019 16:03:35 -0700 From: Tejun Heo To: Jens Axboe Cc: linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, kernel-team@fb.com, cgroups@vger.kernel.org Subject: [PATCH 3/3 for-5.4/block] iocost: bump up default latency targets for hard disks Message-ID: <20190925230335.GK2233839@devbig004.ftw2.facebook.com> References: <20190925230207.GI2233839@devbig004.ftw2.facebook.com> <20190925230309.GJ2233839@devbig004.ftw2.facebook.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20190925230309.GJ2233839@devbig004.ftw2.facebook.com> User-Agent: Mutt/1.5.21 (2010-09-15) Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org The default hard disk param sets latency targets at 50ms. As the default target percentiles are zero, these don't directly regulate vrate; however, they're still used to calculate the period length - 100ms in this case. This is excessively low. A SATA drive with QD32 saturated with random IOs can easily reach avg completion latency of several hundred msecs. A period duration which is substantially lower than avg completion latency can lead to wildly fluctuating vrate. Let's bump up the default latency targets to 250ms so that the period duration is sufficiently long. Signed-off-by: Tejun Heo --- block/blk-iocost.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) --- a/block/blk-iocost.c +++ b/block/blk-iocost.c @@ -529,8 +529,8 @@ struct iocg_wake_ctx { static const struct ioc_params autop[] = { [AUTOP_HDD] = { .qos = { - [QOS_RLAT] = 50000, /* 50ms */ - [QOS_WLAT] = 50000, + [QOS_RLAT] = 250000, /* 250ms */ + [QOS_WLAT] = 250000, [QOS_MIN] = VRATE_MIN_PPM, [QOS_MAX] = VRATE_MAX_PPM, },