From patchwork Thu Jan 26 00:30:14 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dario Faggioli X-Patchwork-Id: 9538201 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id F00BA604A7 for ; Thu, 26 Jan 2017 00:32:39 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E13D426B39 for ; Thu, 26 Jan 2017 00:32:39 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id D576327F85; Thu, 26 Jan 2017 00:32:39 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.6 required=2.0 tests=BAYES_00,DKIM_SIGNED, RCVD_IN_DNSWL_MED,RCVD_IN_SORBS_SPAM,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id C6C4526B39 for ; Thu, 26 Jan 2017 00:32:38 +0000 (UTC) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1cWXxE-0005nk-Ab; Thu, 26 Jan 2017 00:30:20 +0000 Received: from mail6.bemta3.messagelabs.com ([195.245.230.39]) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1cWXxC-0005nG-Mc for xen-devel@lists.xenproject.org; Thu, 26 Jan 2017 00:30:18 +0000 Received: from [85.158.137.68] by server-16.bemta-3.messagelabs.com id 73/75-03637-91349885; Thu, 26 Jan 2017 00:30:17 +0000 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFvrLIsWRWlGSWpSXmKPExsXiVRvkoivh3Bl h8PqsucX3LZOZHBg9Dn+4whLAGMWamZeUX5HAmrHmSVTBjJmMFUebqhsY+0q6GLk4hARmMko0 HXjCDuKwCKxhldiwZC8biCMhcIlV4sOpNqAMJ5ATIzFz9wMou1Li/JtNTCC2kICKxM3tq5ggR k1jktjwdRcLSEJYQE/iyNEf7BC2p8SW9WfYQGw2AQOJNzv2soLYIgJKEvdWTQYbxCwQJXFmeT MziM0ioCoxo30VWJxXwEviStc2sHpOAW+JGZM/AMU5gJZ5SUw7kAMSFhWQk1h5uYUVolxQ4uT MJywgJcwCmhLrd+lDTJeX2P52DvMERpFZSKpmIVTNQlK1gJF5FaNGcWpRWWqRrqGRXlJRZnpG SW5iZo6uoYGxXm5qcXFiempOYlKxXnJ+7iZGYPAzAMEOxlfdzocYJTmYlER5b2p3RgjxJeWnV GYkFmfEF5XmpBYfYtTg4BCYcHbudCYplrz8vFQlCd4zjkB1gkWp6akVaZk5wPiEKZXg4FES4Z 0AkuYtLkjMLc5Mh0idYtTl2LXr8ksmIbAZUuK8m0CKBECKMkrz4EbAUsUlRlkpYV5GoAOFeAp Si3IzS1DlXzGKczAqCfOeB5nCk5lXArfpFdARTEBHXGBuBzmiJBEhJdXAaK9fmuLRIfpqGWP2 z/dOmW8m72lY8e3OIsacrbzCMqXJsZesY8xfyD+/97uvW/+jpE9I4/7DnRpZz/5+0lvBdf0lm 1b6kb+vj1wuN8yXbHT+MC3+2aul6ieKRdtXOn6pmxIVltERtTt8Qv7V+lrPW8/szuWw5XFVpb j3lp+9+bPUj+d1a4iVEktxRqKhFnNRcSIAsItkDBADAAA= X-Env-Sender: raistlin.df@gmail.com X-Msg-Ref: server-5.tower-31.messagelabs.com!1485390616!79182895!1 X-Originating-IP: [74.125.82.68] X-SpamReason: No, hits=0.0 required=7.0 tests= X-StarScan-Received: X-StarScan-Version: 9.1.1; banners=-,-,- X-VirusChecked: Checked Received: (qmail 3503 invoked from network); 26 Jan 2017 00:30:16 -0000 Received: from mail-wm0-f68.google.com (HELO mail-wm0-f68.google.com) (74.125.82.68) by server-5.tower-31.messagelabs.com with AES128-GCM-SHA256 encrypted SMTP; 26 Jan 2017 00:30:16 -0000 Received: by mail-wm0-f68.google.com with SMTP id r144so46496211wme.0 for ; Wed, 25 Jan 2017 16:30:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:subject:from:to:cc:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=bysDTeO6HQIWnJ3JaegnMw0ZWZ5y10bniwPqTid5s4U=; b=Aea69dreXIqLXBz9xLMxjLc6kYTPTYvMm+BsXJR6NyBwvwQh43GYBMAkLXXgw0q452 33hyPtTvo9+mUEAciqAaamGqtdZwRGXWaixrkuN42bpmowYXWBMjd9o7bkU+xuxJ9jKW oAi+YqDRsseMtceUa7pD2f4jAIdfvB2tGormyLmPn37mOVgbjke1QiLoQ/onwAWVLyaD 77z+QRiG+ANoIX95P+Kms7IcQ+iiuItB517NmS4YoIMSmcHCaeG9S/NnWlvP59RCCtpb fnY65iShZCDxBD9kw3EGRf3Rromor+YXJWCgyITr8CKeHmOZyEmpRfY1jaFS7kWcZ0EF 1yaQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:subject:from:to:cc:date:message-id :in-reply-to:references:user-agent:mime-version :content-transfer-encoding; bh=bysDTeO6HQIWnJ3JaegnMw0ZWZ5y10bniwPqTid5s4U=; b=pmLstvJn4yjKdPqvAAwQ89S1X77xX6mpVqDvX1Q9q1RbYbQLMfVObFl0Oaiig3n+Xd jxbssB9maKB49CbKVz1V2OEep32Otvs3rBL/FRJDcPGiTYLIeg8CCzobNtaJR7uFWfat U+i/WF6dwMCq3Tu0fDkB+9wc2fPfNaCCoPfafFR4B4ldG4VDPs3EyLE820h9zRxqMH55 NjCwyB6d8yA5ikezzbKLBNtdSL10+xEIiQt3uxMfm1WQF89u09WW4NBwnSbzs5TBKDSq rjyWHgQNj1qw6qffMBsjQfbzUL7TsApgQK+d39nTZddMKqr8ItKcSQPsn9zkPqZ+zhcR v2Zw== X-Gm-Message-State: AIkVDXKcuC+e2CkM4dqQm07TRDASIKY1i8D83qPndkG+djiVusTft5Mu0sOdDoXTWmkTVw== X-Received: by 10.28.141.199 with SMTP id p190mr368427wmd.89.1485390616171; Wed, 25 Jan 2017 16:30:16 -0800 (PST) Received: from Solace.fritz.box (58-209-66-80.hosts.abilene.it. [80.66.209.58]) by smtp.gmail.com with ESMTPSA id y1sm949720wme.15.2017.01.25.16.30.15 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 25 Jan 2017 16:30:15 -0800 (PST) From: Dario Faggioli To: xen-devel@lists.xenproject.org Date: Thu, 26 Jan 2017 01:30:14 +0100 Message-ID: <148539061397.5464.1056583005783258880.stgit@Solace.fritz.box> In-Reply-To: <148539008889.5464.5896389113741708672.stgit@Solace.fritz.box> References: <148539008889.5464.5896389113741708672.stgit@Solace.fritz.box> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 Cc: George Dunlap , Anshul Makkar Subject: [Xen-devel] [PATCH 4/9] xen: credit2: group the runq manipulating functions. X-BeenThere: xen-devel@lists.xen.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xen.org Sender: "Xen-devel" X-Virus-Scanned: ClamAV using ClamSMTP So that they're all close among each other, and also near to the comment describind the runqueue organization (which is also moved). No functional change intended. Signed-off-by: Dario Faggioli --- Cc: George Dunlap Cc: Anshul Makkar --- xen/common/sched_credit2.c | 572 ++++++++++++++++++++++---------------------- 1 file changed, 286 insertions(+), 286 deletions(-) diff --git a/xen/common/sched_credit2.c b/xen/common/sched_credit2.c index b764cf9..f4c0ae4 100644 --- a/xen/common/sched_credit2.c +++ b/xen/common/sched_credit2.c @@ -295,63 +295,6 @@ static int __read_mostly opt_overload_balance_tolerance = -3; integer_param("credit2_balance_over", opt_overload_balance_tolerance); /* - * Runqueue organization. - * - * The various cpus are to be assigned each one to a runqueue, and we - * want that to happen basing on topology. At the moment, it is possible - * to choose to arrange runqueues to be: - * - * - per-core: meaning that there will be one runqueue per each physical - * core of the host. This will happen if the opt_runqueue - * parameter is set to 'core'; - * - * - per-socket: meaning that there will be one runqueue per each physical - * socket (AKA package, which often, but not always, also - * matches a NUMA node) of the host; This will happen if - * the opt_runqueue parameter is set to 'socket'; - * - * - per-node: meaning that there will be one runqueue per each physical - * NUMA node of the host. This will happen if the opt_runqueue - * parameter is set to 'node'; - * - * - global: meaning that there will be only one runqueue to which all the - * (logical) processors of the host belong. This will happen if - * the opt_runqueue parameter is set to 'all'. - * - * Depending on the value of opt_runqueue, therefore, cpus that are part of - * either the same physical core, the same physical socket, the same NUMA - * node, or just all of them, will be put together to form runqueues. - */ -#define OPT_RUNQUEUE_CORE 0 -#define OPT_RUNQUEUE_SOCKET 1 -#define OPT_RUNQUEUE_NODE 2 -#define OPT_RUNQUEUE_ALL 3 -static const char *const opt_runqueue_str[] = { - [OPT_RUNQUEUE_CORE] = "core", - [OPT_RUNQUEUE_SOCKET] = "socket", - [OPT_RUNQUEUE_NODE] = "node", - [OPT_RUNQUEUE_ALL] = "all" -}; -static int __read_mostly opt_runqueue = OPT_RUNQUEUE_SOCKET; - -static void parse_credit2_runqueue(const char *s) -{ - unsigned int i; - - for ( i = 0; i < ARRAY_SIZE(opt_runqueue_str); i++ ) - { - if ( !strcmp(s, opt_runqueue_str[i]) ) - { - opt_runqueue = i; - return; - } - } - - printk("WARNING, unrecognized value of credit2_runqueue option!\n"); -} -custom_param("credit2_runqueue", parse_credit2_runqueue); - -/* * Per-runqueue data */ struct csched2_runqueue_data { @@ -563,45 +506,304 @@ static int get_fallback_cpu(struct csched2_vcpu *svc) return cpumask_first(cpumask_scratch_cpu(cpu)); } - ASSERT(!cpumask_empty(cpumask_scratch_cpu(cpu))); - - return cpumask_first(cpumask_scratch_cpu(cpu)); + ASSERT(!cpumask_empty(cpumask_scratch_cpu(cpu))); + + return cpumask_first(cpumask_scratch_cpu(cpu)); +} + +/* + * Time-to-credit, credit-to-time. + * + * We keep track of the "residual" time to make sure that frequent short + * schedules still get accounted for in the end. + * + * FIXME: Do pre-calculated division? + */ +static void t2c_update(struct csched2_runqueue_data *rqd, s_time_t time, + struct csched2_vcpu *svc) +{ + uint64_t val = time * rqd->max_weight + svc->residual; + + svc->residual = do_div(val, svc->weight); + svc->credit -= val; +} + +static s_time_t c2t(struct csched2_runqueue_data *rqd, s_time_t credit, struct csched2_vcpu *svc) +{ + return credit * svc->weight / rqd->max_weight; +} + +/* + * Runqueue related code. + */ + +/* + * Runqueue organization. + * + * The various cpus are to be assigned each one to a runqueue, and we + * want that to happen basing on topology. At the moment, it is possible + * to choose to arrange runqueues to be: + * + * - per-core: meaning that there will be one runqueue per each physical + * core of the host. This will happen if the opt_runqueue + * parameter is set to 'core'; + * + * - per-socket: meaning that there will be one runqueue per each physical + * socket (AKA package, which often, but not always, also + * matches a NUMA node) of the host; This will happen if + * the opt_runqueue parameter is set to 'socket'; + * + * - per-node: meaning that there will be one runqueue per each physical + * NUMA node of the host. This will happen if the opt_runqueue + * parameter is set to 'node'; + * + * - global: meaning that there will be only one runqueue to which all the + * (logical) processors of the host belong. This will happen if + * the opt_runqueue parameter is set to 'all'. + * + * Depending on the value of opt_runqueue, therefore, cpus that are part of + * either the same physical core, the same physical socket, the same NUMA + * node, or just all of them, will be put together to form runqueues. + */ +#define OPT_RUNQUEUE_CORE 0 +#define OPT_RUNQUEUE_SOCKET 1 +#define OPT_RUNQUEUE_NODE 2 +#define OPT_RUNQUEUE_ALL 3 +static const char *const opt_runqueue_str[] = { + [OPT_RUNQUEUE_CORE] = "core", + [OPT_RUNQUEUE_SOCKET] = "socket", + [OPT_RUNQUEUE_NODE] = "node", + [OPT_RUNQUEUE_ALL] = "all" +}; +static int __read_mostly opt_runqueue = OPT_RUNQUEUE_SOCKET; + +static void parse_credit2_runqueue(const char *s) +{ + unsigned int i; + + for ( i = 0; i < ARRAY_SIZE(opt_runqueue_str); i++ ) + { + if ( !strcmp(s, opt_runqueue_str[i]) ) + { + opt_runqueue = i; + return; + } + } + + printk("WARNING, unrecognized value of credit2_runqueue option!\n"); +} +custom_param("credit2_runqueue", parse_credit2_runqueue); + +static inline int vcpu_on_runq(struct csched2_vcpu *svc) +{ + return !list_empty(&svc->runq_elem); +} + +static struct csched2_vcpu * runq_elem(struct list_head *elem) +{ + return list_entry(elem, struct csched2_vcpu, runq_elem); +} + +static void activate_runqueue(struct csched2_private *prv, int rqi) +{ + struct csched2_runqueue_data *rqd; + + rqd = prv->rqd + rqi; + + BUG_ON(!cpumask_empty(&rqd->active)); + + rqd->max_weight = 1; + rqd->id = rqi; + INIT_LIST_HEAD(&rqd->svc); + INIT_LIST_HEAD(&rqd->runq); + spin_lock_init(&rqd->lock); + + __cpumask_set_cpu(rqi, &prv->active_queues); +} + +static void deactivate_runqueue(struct csched2_private *prv, int rqi) +{ + struct csched2_runqueue_data *rqd; + + rqd = prv->rqd + rqi; + + BUG_ON(!cpumask_empty(&rqd->active)); + + rqd->id = -1; + + __cpumask_clear_cpu(rqi, &prv->active_queues); +} + +static inline bool_t same_node(unsigned int cpua, unsigned int cpub) +{ + return cpu_to_node(cpua) == cpu_to_node(cpub); +} + +static inline bool_t same_socket(unsigned int cpua, unsigned int cpub) +{ + return cpu_to_socket(cpua) == cpu_to_socket(cpub); +} + +static inline bool_t same_core(unsigned int cpua, unsigned int cpub) +{ + return same_socket(cpua, cpub) && + cpu_to_core(cpua) == cpu_to_core(cpub); +} + +static unsigned int +cpu_to_runqueue(struct csched2_private *prv, unsigned int cpu) +{ + struct csched2_runqueue_data *rqd; + unsigned int rqi; + + for ( rqi = 0; rqi < nr_cpu_ids; rqi++ ) + { + unsigned int peer_cpu; + + /* + * As soon as we come across an uninitialized runqueue, use it. + * In fact, either: + * - we are initializing the first cpu, and we assign it to + * runqueue 0. This is handy, especially if we are dealing + * with the boot cpu (if credit2 is the default scheduler), + * as we would not be able to use cpu_to_socket() and similar + * helpers anyway (they're result of which is not reliable yet); + * - we have gone through all the active runqueues, and have not + * found anyone whose cpus' topology matches the one we are + * dealing with, so activating a new runqueue is what we want. + */ + if ( prv->rqd[rqi].id == -1 ) + break; + + rqd = prv->rqd + rqi; + BUG_ON(cpumask_empty(&rqd->active)); + + peer_cpu = cpumask_first(&rqd->active); + BUG_ON(cpu_to_socket(cpu) == XEN_INVALID_SOCKET_ID || + cpu_to_socket(peer_cpu) == XEN_INVALID_SOCKET_ID); + + if ( opt_runqueue == OPT_RUNQUEUE_ALL || + (opt_runqueue == OPT_RUNQUEUE_CORE && same_core(peer_cpu, cpu)) || + (opt_runqueue == OPT_RUNQUEUE_SOCKET && same_socket(peer_cpu, cpu)) || + (opt_runqueue == OPT_RUNQUEUE_NODE && same_node(peer_cpu, cpu)) ) + break; + } + + /* We really expect to be able to assign each cpu to a runqueue. */ + BUG_ON(rqi >= nr_cpu_ids); + + return rqi; +} + +/* Find the domain with the highest weight. */ +static void update_max_weight(struct csched2_runqueue_data *rqd, int new_weight, + int old_weight) +{ + /* Try to avoid brute-force search: + * - If new_weight is larger, max_weigth <- new_weight + * - If old_weight != max_weight, someone else is still max_weight + * (No action required) + * - If old_weight == max_weight, brute-force search for max weight + */ + if ( new_weight > rqd->max_weight ) + { + rqd->max_weight = new_weight; + SCHED_STAT_CRANK(upd_max_weight_quick); + } + else if ( old_weight == rqd->max_weight ) + { + struct list_head *iter; + int max_weight = 1; + + list_for_each( iter, &rqd->svc ) + { + struct csched2_vcpu * svc = list_entry(iter, struct csched2_vcpu, rqd_elem); + + if ( svc->weight > max_weight ) + max_weight = svc->weight; + } + + rqd->max_weight = max_weight; + SCHED_STAT_CRANK(upd_max_weight_full); + } + + if ( unlikely(tb_init_done) ) + { + struct { + unsigned rqi:16, max_weight:16; + } d; + d.rqi = rqd->id; + d.max_weight = rqd->max_weight; + __trace_var(TRC_CSCHED2_RUNQ_MAX_WEIGHT, 1, + sizeof(d), + (unsigned char *)&d); + } +} + +/* Add and remove from runqueue assignment (not active run queue) */ +static void +_runq_assign(struct csched2_vcpu *svc, struct csched2_runqueue_data *rqd) +{ + + svc->rqd = rqd; + list_add_tail(&svc->rqd_elem, &svc->rqd->svc); + + update_max_weight(svc->rqd, svc->weight, 0); + + /* Expected new load based on adding this vcpu */ + rqd->b_avgload += svc->avgload; + + if ( unlikely(tb_init_done) ) + { + struct { + unsigned vcpu:16, dom:16; + unsigned rqi:16; + } d; + d.dom = svc->vcpu->domain->domain_id; + d.vcpu = svc->vcpu->vcpu_id; + d.rqi=rqd->id; + __trace_var(TRC_CSCHED2_RUNQ_ASSIGN, 1, + sizeof(d), + (unsigned char *)&d); + } + } -/* - * Time-to-credit, credit-to-time. - * - * We keep track of the "residual" time to make sure that frequent short - * schedules still get accounted for in the end. - * - * FIXME: Do pre-calculated division? - */ -static void t2c_update(struct csched2_runqueue_data *rqd, s_time_t time, - struct csched2_vcpu *svc) +static void +runq_assign(const struct scheduler *ops, struct vcpu *vc) { - uint64_t val = time * rqd->max_weight + svc->residual; + struct csched2_vcpu *svc = vc->sched_priv; - svc->residual = do_div(val, svc->weight); - svc->credit -= val; + ASSERT(svc->rqd == NULL); + + _runq_assign(svc, c2rqd(ops, vc->processor)); } -static s_time_t c2t(struct csched2_runqueue_data *rqd, s_time_t credit, struct csched2_vcpu *svc) +static void +_runq_deassign(struct csched2_vcpu *svc) { - return credit * svc->weight / rqd->max_weight; -} + struct csched2_runqueue_data *rqd = svc->rqd; -/* - * Runqueue related code - */ + ASSERT(!vcpu_on_runq(svc)); + ASSERT(!(svc->flags & CSFLAG_scheduled)); -static inline int vcpu_on_runq(struct csched2_vcpu *svc) -{ - return !list_empty(&svc->runq_elem); + list_del_init(&svc->rqd_elem); + update_max_weight(rqd, 0, svc->weight); + + /* Expected new load based on removing this vcpu */ + rqd->b_avgload = max_t(s_time_t, rqd->b_avgload - svc->avgload, 0); + + svc->rqd = NULL; } -static struct csched2_vcpu * runq_elem(struct list_head *elem) +static void +runq_deassign(const struct scheduler *ops, struct vcpu *vc) { - return list_entry(elem, struct csched2_vcpu, runq_elem); + struct csched2_vcpu *svc = vc->sched_priv; + + ASSERT(svc->rqd == c2rqd(ops, vc->processor)); + + _runq_deassign(svc); } /* @@ -1219,51 +1421,6 @@ void burn_credits(struct csched2_runqueue_data *rqd, } } -/* Find the domain with the highest weight. */ -static void update_max_weight(struct csched2_runqueue_data *rqd, int new_weight, - int old_weight) -{ - /* Try to avoid brute-force search: - * - If new_weight is larger, max_weigth <- new_weight - * - If old_weight != max_weight, someone else is still max_weight - * (No action required) - * - If old_weight == max_weight, brute-force search for max weight - */ - if ( new_weight > rqd->max_weight ) - { - rqd->max_weight = new_weight; - SCHED_STAT_CRANK(upd_max_weight_quick); - } - else if ( old_weight == rqd->max_weight ) - { - struct list_head *iter; - int max_weight = 1; - - list_for_each( iter, &rqd->svc ) - { - struct csched2_vcpu * svc = list_entry(iter, struct csched2_vcpu, rqd_elem); - - if ( svc->weight > max_weight ) - max_weight = svc->weight; - } - - rqd->max_weight = max_weight; - SCHED_STAT_CRANK(upd_max_weight_full); - } - - if ( unlikely(tb_init_done) ) - { - struct { - unsigned rqi:16, max_weight:16; - } d; - d.rqi = rqd->id; - d.max_weight = rqd->max_weight; - __trace_var(TRC_CSCHED2_RUNQ_MAX_WEIGHT, 1, - sizeof(d), - (unsigned char *)&d); - } -} - #ifndef NDEBUG static inline void csched2_vcpu_check(struct vcpu *vc) @@ -1328,72 +1485,6 @@ csched2_alloc_vdata(const struct scheduler *ops, struct vcpu *vc, void *dd) return svc; } -/* Add and remove from runqueue assignment (not active run queue) */ -static void -_runq_assign(struct csched2_vcpu *svc, struct csched2_runqueue_data *rqd) -{ - - svc->rqd = rqd; - list_add_tail(&svc->rqd_elem, &svc->rqd->svc); - - update_max_weight(svc->rqd, svc->weight, 0); - - /* Expected new load based on adding this vcpu */ - rqd->b_avgload += svc->avgload; - - if ( unlikely(tb_init_done) ) - { - struct { - unsigned vcpu:16, dom:16; - unsigned rqi:16; - } d; - d.dom = svc->vcpu->domain->domain_id; - d.vcpu = svc->vcpu->vcpu_id; - d.rqi=rqd->id; - __trace_var(TRC_CSCHED2_RUNQ_ASSIGN, 1, - sizeof(d), - (unsigned char *)&d); - } - -} - -static void -runq_assign(const struct scheduler *ops, struct vcpu *vc) -{ - struct csched2_vcpu *svc = vc->sched_priv; - - ASSERT(svc->rqd == NULL); - - _runq_assign(svc, c2rqd(ops, vc->processor)); -} - -static void -_runq_deassign(struct csched2_vcpu *svc) -{ - struct csched2_runqueue_data *rqd = svc->rqd; - - ASSERT(!vcpu_on_runq(svc)); - ASSERT(!(svc->flags & CSFLAG_scheduled)); - - list_del_init(&svc->rqd_elem); - update_max_weight(rqd, 0, svc->weight); - - /* Expected new load based on removing this vcpu */ - rqd->b_avgload = max_t(s_time_t, rqd->b_avgload - svc->avgload, 0); - - svc->rqd = NULL; -} - -static void -runq_deassign(const struct scheduler *ops, struct vcpu *vc) -{ - struct csched2_vcpu *svc = vc->sched_priv; - - ASSERT(svc->rqd == c2rqd(ops, vc->processor)); - - _runq_deassign(svc); -} - static void csched2_vcpu_sleep(const struct scheduler *ops, struct vcpu *vc) { @@ -2778,97 +2869,6 @@ csched2_dump(const struct scheduler *ops) #undef cpustr } -static void activate_runqueue(struct csched2_private *prv, int rqi) -{ - struct csched2_runqueue_data *rqd; - - rqd = prv->rqd + rqi; - - BUG_ON(!cpumask_empty(&rqd->active)); - - rqd->max_weight = 1; - rqd->id = rqi; - INIT_LIST_HEAD(&rqd->svc); - INIT_LIST_HEAD(&rqd->runq); - spin_lock_init(&rqd->lock); - - __cpumask_set_cpu(rqi, &prv->active_queues); -} - -static void deactivate_runqueue(struct csched2_private *prv, int rqi) -{ - struct csched2_runqueue_data *rqd; - - rqd = prv->rqd + rqi; - - BUG_ON(!cpumask_empty(&rqd->active)); - - rqd->id = -1; - - __cpumask_clear_cpu(rqi, &prv->active_queues); -} - -static inline bool_t same_node(unsigned int cpua, unsigned int cpub) -{ - return cpu_to_node(cpua) == cpu_to_node(cpub); -} - -static inline bool_t same_socket(unsigned int cpua, unsigned int cpub) -{ - return cpu_to_socket(cpua) == cpu_to_socket(cpub); -} - -static inline bool_t same_core(unsigned int cpua, unsigned int cpub) -{ - return same_socket(cpua, cpub) && - cpu_to_core(cpua) == cpu_to_core(cpub); -} - -static unsigned int -cpu_to_runqueue(struct csched2_private *prv, unsigned int cpu) -{ - struct csched2_runqueue_data *rqd; - unsigned int rqi; - - for ( rqi = 0; rqi < nr_cpu_ids; rqi++ ) - { - unsigned int peer_cpu; - - /* - * As soon as we come across an uninitialized runqueue, use it. - * In fact, either: - * - we are initializing the first cpu, and we assign it to - * runqueue 0. This is handy, especially if we are dealing - * with the boot cpu (if credit2 is the default scheduler), - * as we would not be able to use cpu_to_socket() and similar - * helpers anyway (they're result of which is not reliable yet); - * - we have gone through all the active runqueues, and have not - * found anyone whose cpus' topology matches the one we are - * dealing with, so activating a new runqueue is what we want. - */ - if ( prv->rqd[rqi].id == -1 ) - break; - - rqd = prv->rqd + rqi; - BUG_ON(cpumask_empty(&rqd->active)); - - peer_cpu = cpumask_first(&rqd->active); - BUG_ON(cpu_to_socket(cpu) == XEN_INVALID_SOCKET_ID || - cpu_to_socket(peer_cpu) == XEN_INVALID_SOCKET_ID); - - if ( opt_runqueue == OPT_RUNQUEUE_ALL || - (opt_runqueue == OPT_RUNQUEUE_CORE && same_core(peer_cpu, cpu)) || - (opt_runqueue == OPT_RUNQUEUE_SOCKET && same_socket(peer_cpu, cpu)) || - (opt_runqueue == OPT_RUNQUEUE_NODE && same_node(peer_cpu, cpu)) ) - break; - } - - /* We really expect to be able to assign each cpu to a runqueue. */ - BUG_ON(rqi >= nr_cpu_ids); - - return rqi; -} - /* Returns the ID of the runqueue the cpu is assigned to. */ static unsigned init_pdata(struct csched2_private *prv, unsigned int cpu)