From patchwork Fri Sep 30 02:53:46 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dario Faggioli X-Patchwork-Id: 9357457 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id BEEFE6077A for ; Fri, 30 Sep 2016 02:56:08 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B3A2729D40 for ; Fri, 30 Sep 2016 02:56:08 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id A7F6829D44; Fri, 30 Sep 2016 02:56:08 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.6 required=2.0 tests=BAYES_00,DKIM_SIGNED, RCVD_IN_DNSWL_MED,RCVD_IN_SORBS_SPAM,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 2C58329D40 for ; Fri, 30 Sep 2016 02:56:08 +0000 (UTC) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1bpnxO-0001sd-Jt; Fri, 30 Sep 2016 02:53:50 +0000 Received: from mail6.bemta6.messagelabs.com ([193.109.254.103]) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1bpnxO-0001s6-3X for xen-devel@lists.xenproject.org; Fri, 30 Sep 2016 02:53:50 +0000 Received: from [193.109.254.147] by server-3.bemta-6.messagelabs.com id 64/B1-04595-DB3DDE75; Fri, 30 Sep 2016 02:53:49 +0000 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFvrPIsWRWlGSWpSXmKPExsXiVRvkrLvn8tt wg4u7lCy+b5nM5MDocfjDFZYAxijWzLyk/IoE1oyVi76xFzxXr1i8rYm5gfG/TBcjF4eQwHRG iUuvD7KDOCwCa1glXqxbzATiSAhcYpX4NO01axcjJ5ATI/H50Hl2CLtS4vGSM4wgtpCAisTN7 auYIEb9YpR4032eBSQhLKAnceToD3YI20di1fEbYA1sAgYSb3bsBRsqIqAkcW/VZCYQm1kgSu LM8mZmEJtFQFXi/+nbYPW8At4SG9c/BavnFPCV+LXtBQvEYh+J38uXs4HYogJyEisvt7BC1At KnJz5BKiGA2impsT6XfoQ4+Ultr+dwzyBUWQWkqpZCFWzkFQtYGRexahRnFpUllqka2igl1SU mZ5RkpuYmQPkmenlphYXJ6an5iQmFesl5+duYgRGAAMQ7GC8tyzgEKMkB5OSKK/s0bfhQnxJ+ SmVGYnFGfFFpTmpxYcYNTg4BCacnTudSYolLz8vVUmCd+0loDrBotT01Iq0zBxgjMKUSnDwKI nwrgRJ8xYXJOYWZ6ZDpE4x6nJsmXpvLZMQ2Awpcd4mkCIBkKKM0jy4EbB0cYlRVkqYlxHoQCG egtSi3MwSVPlXjOIcjErCvNtApvBk5pXAbXoFdAQT0BH5R9+AHFGSiJCSamBsnrwr2/FWavWs C6cWeSa0NSv+26JpvF9wV3jiHe5zfn9fld74smFJi/CbUjbnXUKTm61q5Va5F735KakQMv01n yJXqPefCQnJ79ZZ/LNSPTj3yhnW8wecjt2criXOuadw7q8LstZSf6bmt311nzPJ72XGn4IfGZ J96v3FMv7/zi/KFHif/tFMiaU4I9FQi7moOBEABqAfeBIDAAA= X-Env-Sender: raistlin.df@gmail.com X-Msg-Ref: server-15.tower-27.messagelabs.com!1475204028!9516894!1 X-Originating-IP: [74.125.82.67] X-SpamReason: No, hits=0.0 required=7.0 tests= X-StarScan-Received: X-StarScan-Version: 8.84; banners=-,-,- X-VirusChecked: Checked Received: (qmail 13170 invoked from network); 30 Sep 2016 02:53:48 -0000 Received: from mail-wm0-f67.google.com (HELO mail-wm0-f67.google.com) (74.125.82.67) by server-15.tower-27.messagelabs.com with AES128-GCM-SHA256 encrypted SMTP; 30 Sep 2016 02:53:48 -0000 Received: by mail-wm0-f67.google.com with SMTP id w72so1397066wmf.1 for ; Thu, 29 Sep 2016 19:53:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:subject:from:to:cc:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=m/1CiGEjpheLoTL1L015a30zFjaqAUw82oaFRfswKy8=; b=GX2KkTaeTWG+AhLokcw9Co24FV5vug/NJq0vqCMg2TnyMJd6D5/ILlpx6hOzzMW+a9 4XQt/4CV/XJopjgaNoxT8A5FN0q3DSt/uOskeDDXbmaMMy6zUjYpjdEPAmYqvLphmGG9 fRhQk2f530bMxBAb17vGZCKoPiWf3vUSUMSvCtArCQ0D/hiq0F4PTDdRh9aZlCSP3zxQ q4+1TKddxeBir2bg8DW5kSutVlGmNYPr5v3k+aIrjPtU4g3FfQMwJEVsnZ/x0oS0p2LU rhxs4Fd0TlAYHT6cMORft79pwDZfm3fQG1QYb8ftQ+DOb9z6AWbJGq2l/SiXBQPpjCpG Ctug== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:sender:subject:from:to:cc:date:message-id :in-reply-to:references:user-agent:mime-version :content-transfer-encoding; bh=m/1CiGEjpheLoTL1L015a30zFjaqAUw82oaFRfswKy8=; b=YrMlTczj++MoSO3kHgZw4nKbjqc3d2RWChiwJIdyYzSd85Mu+aikQG4BqeW1YeE/gm q/BKe4owlgVgMIpIRNbKKcJCjmlc+8Q1u1WGnRDBLehifiiYq1HRXk0EhTSqeIHyK2wd iKtS4jOo1tvy2Ydrkl/tBC4lEjKBYuvm8tI/SEpAlo+leCo1bfyVwk6S8yvwAJghnjv8 k/SjK0gPFGwjfS6ukqHNrC7RnrAUjoZU7miaDrt2gy9vc9azyozou2LM52Vq1aX+4vks /1rEDqpwVZ/Uj+XJ1XFeXMQCbUY6iwBsIGHODbJctOuFAmBbRFIicDPpSnrf8nS3CmgC cLDw== X-Gm-Message-State: AA6/9RkAkmqGgSA1+0v0M1PxHZl4uuSFZmtqzG5CxR1q4b7CQZzym7lhRuhEK3/hg5YALA== X-Received: by 10.194.87.197 with SMTP id ba5mr4394434wjb.69.1475204028172; Thu, 29 Sep 2016 19:53:48 -0700 (PDT) Received: from Solace.fritz.box ([80.66.223.126]) by smtp.gmail.com with ESMTPSA id ab9sm17194147wjc.7.2016.09.29.19.53.47 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 29 Sep 2016 19:53:47 -0700 (PDT) From: Dario Faggioli To: xen-devel@lists.xenproject.org Date: Fri, 30 Sep 2016 04:53:46 +0200 Message-ID: <147520402650.22544.2188671255927985759.stgit@Solace.fritz.box> In-Reply-To: <147520253247.22544.10673844222866363947.stgit@Solace.fritz.box> References: <147520253247.22544.10673844222866363947.stgit@Solace.fritz.box> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 Cc: George Dunlap , Anshul Makkar Subject: [Xen-devel] [PATCH v2 04/10] xen: credit2: only reset credit on reset condition X-BeenThere: xen-devel@lists.xen.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xen.org Sender: "Xen-devel" X-Virus-Scanned: ClamAV using ClamSMTP The condition for a Credit2 scheduling epoch coming to an end is that the vcpu at the front of the runqueue has negative credits. However, it is possible, that runq_candidate() does not actually return to the scheduler the first vcpu in the runqueue (e.g., because such vcpu can't run on the cpu that is going through the scheduler, because of hard-affinity). If that happens, we should not trigger a credit reset, or we risk altering the lenght of a scheduler epoch, wrt what the original idea of the algorithm was. Signed-off-by: Dario Faggioli Reviewed-by: George Dunlap --- Cc: George Dunlap Cc: Anshul Makkar --- Changes from v1: * new patch, containing part of what was in patch 5; * (wrt v1 patch 5) 'pos' parameter to runq_candidate renamed 'skipped', as requested during review. --- xen/common/sched_credit2.c | 33 ++++++++++++++++++++++++++++----- 1 file changed, 28 insertions(+), 5 deletions(-) diff --git a/xen/common/sched_credit2.c b/xen/common/sched_credit2.c index 3986441..72e31b5 100644 --- a/xen/common/sched_credit2.c +++ b/xen/common/sched_credit2.c @@ -2244,12 +2244,15 @@ void __dump_execstate(void *unused); static struct csched2_vcpu * runq_candidate(struct csched2_runqueue_data *rqd, struct csched2_vcpu *scurr, - int cpu, s_time_t now) + int cpu, s_time_t now, + unsigned int *skipped) { struct list_head *iter; struct csched2_vcpu *snext = NULL; struct csched2_private *prv = CSCHED2_PRIV(per_cpu(scheduler, cpu)); + *skipped = 0; + /* Default to current if runnable, idle otherwise */ if ( vcpu_runnable(scurr->vcpu) ) snext = scurr; @@ -2273,7 +2276,10 @@ runq_candidate(struct csched2_runqueue_data *rqd, /* Only consider vcpus that are allowed to run on this processor. */ if ( !cpumask_test_cpu(cpu, svc->vcpu->cpu_hard_affinity) ) + { + (*skipped)++; continue; + } /* * If a vcpu is meant to be picked up by another processor, and such @@ -2282,6 +2288,7 @@ runq_candidate(struct csched2_runqueue_data *rqd, if ( svc->tickled_cpu != -1 && svc->tickled_cpu != cpu && cpumask_test_cpu(svc->tickled_cpu, &rqd->tickled) ) { + (*skipped)++; SCHED_STAT_CRANK(deferred_to_tickled_cpu); continue; } @@ -2291,6 +2298,7 @@ runq_candidate(struct csched2_runqueue_data *rqd, if ( svc->vcpu->processor != cpu && snext->credit + CSCHED2_MIGRATE_RESIST > svc->credit ) { + (*skipped)++; SCHED_STAT_CRANK(migrate_resisted); continue; } @@ -2308,11 +2316,12 @@ runq_candidate(struct csched2_runqueue_data *rqd, { struct { unsigned vcpu:16, dom:16; - unsigned tickled_cpu; + unsigned tickled_cpu, skipped; } d; d.dom = snext->vcpu->domain->domain_id; d.vcpu = snext->vcpu->vcpu_id; d.tickled_cpu = snext->tickled_cpu; + d.skipped = *skipped; __trace_var(TRC_CSCHED2_RUNQ_CANDIDATE, 1, sizeof(d), (unsigned char *)&d); @@ -2336,6 +2345,7 @@ csched2_schedule( struct csched2_runqueue_data *rqd; struct csched2_vcpu * const scurr = CSCHED2_VCPU(current); struct csched2_vcpu *snext = NULL; + unsigned int skipped_vcpus = 0; struct task_slice ret; SCHED_STAT_CRANK(schedule); @@ -2385,7 +2395,7 @@ csched2_schedule( snext = CSCHED2_VCPU(idle_vcpu[cpu]); } else - snext = runq_candidate(rqd, scurr, cpu, now); + snext = runq_candidate(rqd, scurr, cpu, now, &skipped_vcpus); /* If switching from a non-idle runnable vcpu, put it * back on the runqueue. */ @@ -2409,8 +2419,21 @@ csched2_schedule( __set_bit(__CSFLAG_scheduled, &snext->flags); } - /* Check for the reset condition */ - if ( snext->credit <= CSCHED2_CREDIT_RESET ) + /* + * The reset condition is "has a scheduler epoch come to an end?". + * The way this is enforced is checking whether the vcpu at the top + * of the runqueue has negative credits. This means the epochs have + * variable lenght, as in one epoch expores when: + * 1) the vcpu at the top of the runqueue has executed for + * around 10 ms (with default parameters); + * 2) no other vcpu with higher credits wants to run. + * + * Here, where we want to check for reset, we need to make sure the + * proper vcpu is being used. In fact, runqueue_candidate() may have + * not returned the first vcpu in the runqueue, for various reasons + * (e.g., affinity). Only trigger a reset when it does. + */ + if ( skipped_vcpus == 0 && snext->credit <= CSCHED2_CREDIT_RESET ) { reset_credit(ops, cpu, now, snext); balance_load(ops, cpu, now);