From patchwork Thu Jan 26 00:30:28 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dario Faggioli X-Patchwork-Id: 9538203 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id A7FCE604A7 for ; Thu, 26 Jan 2017 00:32:40 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9AF7726B39 for ; Thu, 26 Jan 2017 00:32:40 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 8F65E27F85; Thu, 26 Jan 2017 00:32:40 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.6 required=2.0 tests=BAYES_00,DKIM_SIGNED, RCVD_IN_DNSWL_MED,RCVD_IN_SORBS_SPAM,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 34D7826B39 for ; Thu, 26 Jan 2017 00:32:40 +0000 (UTC) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1cWXxR-0005uN-UU; Thu, 26 Jan 2017 00:30:33 +0000 Received: from mail6.bemta6.messagelabs.com ([193.109.254.103]) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1cWXxQ-0005tM-Au for xen-devel@lists.xenproject.org; Thu, 26 Jan 2017 00:30:32 +0000 Received: from [193.109.254.147] by server-10.bemta-6.messagelabs.com id 91/F3-13192-72349885; Thu, 26 Jan 2017 00:30:31 +0000 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFvrDIsWRWlGSWpSXmKPExsXiVRvkqKvm3Bl h8O6DmcX3LZOZHBg9Dn+4whLAGMWamZeUX5HAmjGr4zhbwWnpij+HpzA3ML4T7WLk4hASmMko 8fbRcdYuRk4OFoE1rBI3J/mDJCQELrFK3J68kgkkISEQI7FuzTZ2CLtCYvmNlWANQgIqEje3r 2KCmDSDSeLVpkWMIAlhAT2JI0d/sEPYoRJzzv9mA7HZBAwk3uzYC9YsIqAkcW/VZLAFzAJREm eWNzNDXKEq0bp2LtAcDg5eAW+JjweEQMKcQOaMyR+YQMJCAl4S0w7kgIRFBeQkVl5uAZvIKyA ocXLmExaQEmYBTYn1u/QhhstLbH87h3kCo8gsJFWzEKpmIalawMi8ilGjOLWoLLVI18hAL6ko Mz2jJDcxM0fX0MBMLze1uDgxPTUnMalYLzk/dxMjMPQZgGAH469lAYcYJTmYlER5b2p3RgjxJ eWnVGYkFmfEF5XmpBYfYtTg4BCYcHbudCYplrz8vFQlCV4bJ6A6waLU9NSKtMwcYHTClEpw8C iJ8HqCpHmLCxJzizPTIVKnGHU5du26/JJJCGyGlDivHUiRAEhRRmke3AhYorjEKCslzMsIdKA QT0FqUW5mCar8K0ZxDkYlYV4PkCk8mXklcJteAR3BBHTEBeZ2kCNKEhFSUg2MAm8CdyrWrNZT k8hfduOzwiL+JTa50g913kva+9pM8S7QzpQtmzWfUfhsnMm/8MyP0cHP/is5VBts0T08p5k/6 fOMyyGLd/5tNeSxdGQX9Fi/u/AuW/C+Rb+Y3k1/sOejAU+zY9Y5BQ/DGHaj5UYBQkv1HR52/3 m0Zebff/dCijnzdh55zDpHiaU4I9FQi7moOBEAYCOVrA8DAAA= X-Env-Sender: raistlin.df@gmail.com X-Msg-Ref: server-8.tower-27.messagelabs.com!1485390630!73485999!1 X-Originating-IP: [74.125.82.65] X-SpamReason: No, hits=0.5 required=7.0 tests=BODY_RANDOM_LONG X-StarScan-Received: X-StarScan-Version: 9.1.1; banners=-,-,- X-VirusChecked: Checked Received: (qmail 14460 invoked from network); 26 Jan 2017 00:30:30 -0000 Received: from mail-wm0-f65.google.com (HELO mail-wm0-f65.google.com) (74.125.82.65) by server-8.tower-27.messagelabs.com with AES128-GCM-SHA256 encrypted SMTP; 26 Jan 2017 00:30:30 -0000 Received: by mail-wm0-f65.google.com with SMTP id r144so46497436wme.0 for ; Wed, 25 Jan 2017 16:30:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:subject:from:to:cc:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=4NkvDWzLhiyIYW2XKb4IoPFvJCRRmJOfK9OuCPdT/4U=; b=TVfu/VhQjQlbA2YlP/4jpYC3JeDJFecsT3ObPobLPTl4pYbnXHEfZjz1WZZdY5xPsU 0hmzc8E7OVaEPluBzXkXbHlO41w61T3cCJ9LLRL/atHHyBp+ZV3ZdupAI3dz08sGdbC1 0aK9SnGqU5/OVDuxSrrLLALNhkHW/eKuuVYlz1opyJnhkwD5MkBSHWXfF5cZB67kTvl4 crL9Qcec632zxxVV5F6btONUOT4rq6VpmfQ+g7MWpgdHrpJ5nRP3viz2otkhhYMEuBGv 73qnwg5lG6X/8P97dZduzXctad4LkrCBIjRrXQNxVWLQBZLQHJB4rLxcdgNo/17pxKyv xLCA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:subject:from:to:cc:date:message-id :in-reply-to:references:user-agent:mime-version :content-transfer-encoding; bh=4NkvDWzLhiyIYW2XKb4IoPFvJCRRmJOfK9OuCPdT/4U=; b=cJ47qYcBTGElyHJ2LQp1mMw9sppmZEOQhUCHUIIIQU5sfvPiI3Qdqmoka2ITFs87fk P+0gz8GUM1K3FRhJgLc3+lnkUDba6QoP45v+K+Me3DxEZiBBHbfz/L4qpUy8XOI6f53H larlS/0gPn2u+rLYsO4bBIRgBwTgvx6qjIxOjQEl9B4b2h49AhqaOa8+ugxxxdWfHMAI eE+wjSkJUC2NWM9YjVtuYMg56cdK+1lFN6qvE6ZXTSdXUgsTnqPwj/KICIkdwxcly4OC 5MgVrPbS0EwrOjXniym+4QLeKLPOhAXqMndZTwWJXCl/DUe8B02V4/SwW3yXcbJ38mYL mHjA== X-Gm-Message-State: AIkVDXLZU7u+eNON4iC3ty/FsEZrbzUE4ToxemAPvSbSpLGuoc79lemcEeTDaheWN3LFkQ== X-Received: by 10.223.145.227 with SMTP id 90mr85510wri.156.1485390630419; Wed, 25 Jan 2017 16:30:30 -0800 (PST) Received: from Solace.fritz.box (58-209-66-80.hosts.abilene.it. [80.66.209.58]) by smtp.gmail.com with ESMTPSA id x135sm918141wme.23.2017.01.25.16.30.29 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 25 Jan 2017 16:30:29 -0800 (PST) From: Dario Faggioli To: xen-devel@lists.xenproject.org Date: Thu, 26 Jan 2017 01:30:28 +0100 Message-ID: <148539062805.5464.13572346399281383738.stgit@Solace.fritz.box> In-Reply-To: <148539008889.5464.5896389113741708672.stgit@Solace.fritz.box> References: <148539008889.5464.5896389113741708672.stgit@Solace.fritz.box> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 Cc: George Dunlap , Anshul Makkar Subject: [Xen-devel] [PATCH 6/9] xen: credit2: don't miss accounting while doing a credit reset. X-BeenThere: xen-devel@lists.xen.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xen.org Sender: "Xen-devel" X-Virus-Scanned: ClamAV using ClamSMTP A credit reset basically means going through all the vCPUs of a runqueue and altering their credits, as a consequence of a 'scheduling epoch' having come to an end. Blocked or runnable vCPUs are fine, all the credits they've spent running so far have been accounted to them when they were scheduled out. But if a vCPU is running on a pCPU, when a reset event occurs (on another pCPU), that does not get properly accounted. Let's therefore begin to do so, for better accuracy and fairness. In fact, after this patch, we see this in a trace: csched2:schedule cpu 10, rq# 1, busy, not tickled csched2:burn_credits d1v5, credit = 9998353, delta = 202996 runstate_continue d1v5 running->running ... csched2:schedule cpu 12, rq# 1, busy, not tickled csched2:burn_credits d1v6, credit = -1327, delta = 9999544 csched2:reset_credits d0v13, credit_start = 10500000, credit_end = 10500000, mult = 1 csched2:reset_credits d0v14, credit_start = 10500000, credit_end = 10500000, mult = 1 csched2:reset_credits d0v7, credit_start = 10500000, credit_end = 10500000, mult = 1 csched2:burn_credits d1v5, credit = 201805, delta = 9796548 csched2:reset_credits d1v5, credit_start = 201805, credit_end = 10201805, mult = 1 csched2:burn_credits d1v6, credit = -1327, delta = 0 csched2:reset_credits d1v6, credit_start = -1327, credit_end = 9998673, mult = 1 Which shows how d1v5 actually executed for ~9.796 ms, on pCPU 10, when reset_credit() is executed, on pCPU 12, because of d1v6's credits going below 0. Without this patch, this 9.796ms are not accounted to anyone. With this patch, d1v5 is charged for that, and its credits drop down from 9796548 to 201805. And this is important, as it means that it will begin the new epoch with 10201805 credits, instead of 10500000 (which he would have, before this patch). Basically, we were forgetting one round of accounting in epoch x, for the vCPUs that are running at the time the epoch ends. And this meant favouring a little bit these same vCPUs, in epoch x+1, providing them with the chance of execute longer than their fair share. Signed-off-by: Dario Faggioli --- Cc: George Dunlap Cc: Anshul Makkar --- xen/common/sched_credit2.c | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-) diff --git a/xen/common/sched_credit2.c b/xen/common/sched_credit2.c index bab9667..07e0a6d 100644 --- a/xen/common/sched_credit2.c +++ b/xen/common/sched_credit2.c @@ -1341,18 +1341,28 @@ static void reset_credit(const struct scheduler *ops, int cpu, s_time_t now, list_for_each( iter, &rqd->svc ) { + unsigned int svc_cpu; struct csched2_vcpu * svc; int start_credit; svc = list_entry(iter, struct csched2_vcpu, rqd_elem); + svc_cpu = svc->vcpu->processor; ASSERT(!is_idle_vcpu(svc->vcpu)); ASSERT(svc->rqd == rqd); + /* + * If svc is running, it is our responsibility to make sure, here, + * that the credit it has spent so far get accounted. + */ + if ( svc->vcpu == curr_on_cpu(svc_cpu) ) + burn_credits(rqd, svc, now); + start_credit = svc->credit; - /* And add INIT * m, avoiding integer multiplication in the - * common case. */ + /* + * Add INIT * m, avoiding integer multiplication in the common case. + */ if ( likely(m==1) ) svc->credit += CSCHED2_CREDIT_INIT; else