From patchwork Tue May 28 10:32:24 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: =?utf-8?b?SsO8cmdlbiBHcm/Dnw==?= X-Patchwork-Id: 10964651 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B15AE92A for ; Tue, 28 May 2019 10:35:30 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9DA4D1FF29 for ; Tue, 28 May 2019 10:35:30 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 8EC272839C; Tue, 28 May 2019 10:35:30 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.2 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 19B542857D for ; Tue, 28 May 2019 10:35:29 +0000 (UTC) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1hVZQ7-0004g9-Pm; Tue, 28 May 2019 10:33:27 +0000 Received: from us1-rack-dfw2.inumbo.com ([104.130.134.6]) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1hVZQ5-0004at-49 for xen-devel@lists.xenproject.org; Tue, 28 May 2019 10:33:25 +0000 X-Inumbo-ID: 03cb4571-8134-11e9-8980-bc764e045a96 Received: from mx1.suse.de (unknown [195.135.220.15]) by us1-rack-dfw2.inumbo.com (Halon) with ESMTPS id 03cb4571-8134-11e9-8980-bc764e045a96; Tue, 28 May 2019 10:33:22 +0000 (UTC) X-Virus-Scanned: by amavisd-new at test-mx.suse.de Received: from relay2.suse.de (unknown [195.135.220.254]) by mx1.suse.de (Postfix) with ESMTP id 54952B031; Tue, 28 May 2019 10:33:19 +0000 (UTC) From: Juergen Gross To: xen-devel@lists.xenproject.org Date: Tue, 28 May 2019 12:32:24 +0200 Message-Id: <20190528103313.1343-12-jgross@suse.com> X-Mailer: git-send-email 2.16.4 In-Reply-To: <20190528103313.1343-1-jgross@suse.com> References: <20190528103313.1343-1-jgross@suse.com> Subject: [Xen-devel] [PATCH 11/60] xen/sched: move per cpu scheduler private data into struct sched_resource X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: Juergen Gross , Stefano Stabellini , Wei Liu , Konrad Rzeszutek Wilk , George Dunlap , Andrew Cooper , Ian Jackson , Robert VanVossen , Tim Deegan , Julien Grall , Josh Whitehead , Meng Xu , Jan Beulich , Dario Faggioli , =?utf-8?q?Roger_Pau_Monn=C3=A9?= MIME-Version: 1.0 Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" X-Virus-Scanned: ClamAV using ClamSMTP This prepares support of larger scheduling granularities, e.g. core scheduling. While at it move sched_has_urgent_vcpu() from include/asm-x86/cpuidle.h into schedule.c removing the need for including sched-if.h in cpuidle.h and multiple other C sources. Signed-off-by: Juergen Gross Acked-by: Jan Beulich Reviewed-by: Dario Faggioli --- V1: move sched_has_urgent_vcpu() --- xen/arch/x86/acpi/cpu_idle.c | 1 - xen/arch/x86/cpu/mcheck/mce.c | 1 - xen/arch/x86/cpu/mcheck/mctelem.c | 1 - xen/arch/x86/setup.c | 1 - xen/arch/x86/smpboot.c | 1 - xen/common/sched_arinc653.c | 4 +-- xen/common/sched_credit.c | 12 +++---- xen/common/sched_credit2.c | 21 +++++++------ xen/common/sched_null.c | 6 ++-- xen/common/sched_rt.c | 9 +++--- xen/common/schedule.c | 66 ++++++++++++++++++++++----------------- xen/include/asm-x86/cpuidle.h | 11 ------- xen/include/xen/sched-if.h | 20 +++++------- xen/include/xen/sched.h | 1 + 14 files changed, 73 insertions(+), 82 deletions(-) diff --git a/xen/arch/x86/acpi/cpu_idle.c b/xen/arch/x86/acpi/cpu_idle.c index 8846722bca..9f66f70986 100644 --- a/xen/arch/x86/acpi/cpu_idle.c +++ b/xen/arch/x86/acpi/cpu_idle.c @@ -38,7 +38,6 @@ #include #include #include -#include #include #include #include diff --git a/xen/arch/x86/cpu/mcheck/mce.c b/xen/arch/x86/cpu/mcheck/mce.c index 30cdb06401..726db75180 100644 --- a/xen/arch/x86/cpu/mcheck/mce.c +++ b/xen/arch/x86/cpu/mcheck/mce.c @@ -10,7 +10,6 @@ #include #include #include -#include #include #include #include diff --git a/xen/arch/x86/cpu/mcheck/mctelem.c b/xen/arch/x86/cpu/mcheck/mctelem.c index 3bb13e5265..012a9b95e5 100644 --- a/xen/arch/x86/cpu/mcheck/mctelem.c +++ b/xen/arch/x86/cpu/mcheck/mctelem.c @@ -18,7 +18,6 @@ #include #include #include -#include #include #include diff --git a/xen/arch/x86/setup.c b/xen/arch/x86/setup.c index 0ed94a613a..5ad66667ef 100644 --- a/xen/arch/x86/setup.c +++ b/xen/arch/x86/setup.c @@ -3,7 +3,6 @@ #include #include #include -#include #include #include #include diff --git a/xen/arch/x86/smpboot.c b/xen/arch/x86/smpboot.c index 274865a705..153bfbb4b7 100644 --- a/xen/arch/x86/smpboot.c +++ b/xen/arch/x86/smpboot.c @@ -25,7 +25,6 @@ #include #include #include -#include #include #include #include diff --git a/xen/common/sched_arinc653.c b/xen/common/sched_arinc653.c index 7f26e6a0b0..6e7b2c9968 100644 --- a/xen/common/sched_arinc653.c +++ b/xen/common/sched_arinc653.c @@ -475,7 +475,7 @@ a653sched_unit_sleep(const struct scheduler *ops, struct sched_unit *unit) * If the VCPU being put to sleep is the same one that is currently * running, raise a softirq to invoke the scheduler to switch domains. */ - if ( per_cpu(schedule_data, vc->processor).curr == unit ) + if ( get_sched_res(vc->processor)->curr == unit ) cpu_raise_softirq(vc->processor, SCHEDULE_SOFTIRQ); } @@ -642,7 +642,7 @@ static spinlock_t * a653_switch_sched(struct scheduler *new_ops, unsigned int cpu, void *pdata, void *vdata) { - struct schedule_data *sd = &per_cpu(schedule_data, cpu); + struct sched_resource *sd = get_sched_res(cpu); arinc653_vcpu_t *svc = vdata; ASSERT(!pdata && svc && is_idle_vcpu(svc->vc)); diff --git a/xen/common/sched_credit.c b/xen/common/sched_credit.c index bc87b813dc..fe4fc5abb1 100644 --- a/xen/common/sched_credit.c +++ b/xen/common/sched_credit.c @@ -82,7 +82,7 @@ #define CSCHED_PRIV(_ops) \ ((struct csched_private *)((_ops)->sched_data)) #define CSCHED_PCPU(_c) \ - ((struct csched_pcpu *)per_cpu(schedule_data, _c).sched_priv) + ((struct csched_pcpu *)get_sched_res(_c)->sched_priv) #define CSCHED_UNIT(unit) ((struct csched_unit *) (unit)->priv) #define CSCHED_DOM(_dom) ((struct csched_dom *) (_dom)->sched_priv) #define RUNQ(_cpu) (&(CSCHED_PCPU(_cpu)->runq)) @@ -248,7 +248,7 @@ static inline bool_t is_runq_idle(unsigned int cpu) /* * We're peeking at cpu's runq, we must hold the proper lock. */ - ASSERT(spin_is_locked(per_cpu(schedule_data, cpu).schedule_lock)); + ASSERT(spin_is_locked(get_sched_res(cpu)->schedule_lock)); return list_empty(RUNQ(cpu)) || is_idle_vcpu(__runq_elem(RUNQ(cpu)->next)->vcpu); @@ -257,7 +257,7 @@ static inline bool_t is_runq_idle(unsigned int cpu) static inline void inc_nr_runnable(unsigned int cpu) { - ASSERT(spin_is_locked(per_cpu(schedule_data, cpu).schedule_lock)); + ASSERT(spin_is_locked(get_sched_res(cpu)->schedule_lock)); CSCHED_PCPU(cpu)->nr_runnable++; } @@ -265,7 +265,7 @@ inc_nr_runnable(unsigned int cpu) static inline void dec_nr_runnable(unsigned int cpu) { - ASSERT(spin_is_locked(per_cpu(schedule_data, cpu).schedule_lock)); + ASSERT(spin_is_locked(get_sched_res(cpu)->schedule_lock)); ASSERT(CSCHED_PCPU(cpu)->nr_runnable >= 1); CSCHED_PCPU(cpu)->nr_runnable--; } @@ -615,7 +615,7 @@ csched_init_pdata(const struct scheduler *ops, void *pdata, int cpu) { unsigned long flags; struct csched_private *prv = CSCHED_PRIV(ops); - struct schedule_data *sd = &per_cpu(schedule_data, cpu); + struct sched_resource *sd = get_sched_res(cpu); /* * This is called either during during boot, resume or hotplug, in @@ -635,7 +635,7 @@ static spinlock_t * csched_switch_sched(struct scheduler *new_ops, unsigned int cpu, void *pdata, void *vdata) { - struct schedule_data *sd = &per_cpu(schedule_data, cpu); + struct sched_resource *sd = get_sched_res(cpu); struct csched_private *prv = CSCHED_PRIV(new_ops); struct csched_unit *svc = vdata; diff --git a/xen/common/sched_credit2.c b/xen/common/sched_credit2.c index 36201815e3..99e993b32f 100644 --- a/xen/common/sched_credit2.c +++ b/xen/common/sched_credit2.c @@ -567,7 +567,7 @@ static inline struct csched2_private *csched2_priv(const struct scheduler *ops) static inline struct csched2_pcpu *csched2_pcpu(unsigned int cpu) { - return per_cpu(schedule_data, cpu).sched_priv; + return get_sched_res(cpu)->sched_priv; } static inline struct csched2_unit *csched2_unit(const struct sched_unit *unit) @@ -1276,7 +1276,7 @@ runq_insert(const struct scheduler *ops, struct csched2_unit *svc) struct list_head * runq = &c2rqd(ops, cpu)->runq; int pos = 0; - ASSERT(spin_is_locked(per_cpu(schedule_data, cpu).schedule_lock)); + ASSERT(spin_is_locked(get_sched_res(cpu)->schedule_lock)); ASSERT(!vcpu_on_runq(svc)); ASSERT(c2r(cpu) == c2r(svc->vcpu->processor)); @@ -1797,7 +1797,7 @@ static bool vcpu_grab_budget(struct csched2_unit *svc) struct csched2_dom *sdom = svc->sdom; unsigned int cpu = svc->vcpu->processor; - ASSERT(spin_is_locked(per_cpu(schedule_data, cpu).schedule_lock)); + ASSERT(spin_is_locked(get_sched_res(cpu)->schedule_lock)); if ( svc->budget > 0 ) return true; @@ -1844,7 +1844,7 @@ vcpu_return_budget(struct csched2_unit *svc, struct list_head *parked) struct csched2_dom *sdom = svc->sdom; unsigned int cpu = svc->vcpu->processor; - ASSERT(spin_is_locked(per_cpu(schedule_data, cpu).schedule_lock)); + ASSERT(spin_is_locked(get_sched_res(cpu)->schedule_lock)); ASSERT(list_empty(parked)); /* budget_lock nests inside runqueue lock. */ @@ -2101,7 +2101,7 @@ csched2_unit_wake(const struct scheduler *ops, struct sched_unit *unit) unsigned int cpu = vc->processor; s_time_t now; - ASSERT(spin_is_locked(per_cpu(schedule_data, cpu).schedule_lock)); + ASSERT(spin_is_locked(get_sched_res(cpu)->schedule_lock)); ASSERT(!is_idle_vcpu(vc)); @@ -2229,7 +2229,7 @@ csched2_res_pick(const struct scheduler *ops, struct sched_unit *unit) * just grab the prv lock. Instead, we'll have to trylock, and * do something else reasonable if we fail. */ - ASSERT(spin_is_locked(per_cpu(schedule_data, cpu).schedule_lock)); + ASSERT(spin_is_locked(get_sched_res(cpu)->schedule_lock)); if ( !read_trylock(&prv->lock) ) { @@ -2569,7 +2569,7 @@ static void balance_load(const struct scheduler *ops, int cpu, s_time_t now) * on either side may be empty). */ - ASSERT(spin_is_locked(per_cpu(schedule_data, cpu).schedule_lock)); + ASSERT(spin_is_locked(get_sched_res(cpu)->schedule_lock)); st.lrqd = c2rqd(ops, cpu); update_runq_load(ops, st.lrqd, 0, now); @@ -3475,7 +3475,7 @@ csched2_schedule( rqd = c2rqd(ops, cpu); BUG_ON(!cpumask_test_cpu(cpu, &rqd->active)); - ASSERT(spin_is_locked(per_cpu(schedule_data, cpu).schedule_lock)); + ASSERT(spin_is_locked(get_sched_res(cpu)->schedule_lock)); BUG_ON(!is_idle_vcpu(scurr->vcpu) && scurr->rqd != rqd); @@ -3863,7 +3863,7 @@ csched2_init_pdata(const struct scheduler *ops, void *pdata, int cpu) rqi = init_pdata(prv, pdata, cpu); /* Move the scheduler lock to the new runq lock. */ - per_cpu(schedule_data, cpu).schedule_lock = &prv->rqd[rqi].lock; + get_sched_res(cpu)->schedule_lock = &prv->rqd[rqi].lock; /* _Not_ pcpu_schedule_unlock(): schedule_lock may have changed! */ spin_unlock(old_lock); @@ -3877,6 +3877,7 @@ csched2_switch_sched(struct scheduler *new_ops, unsigned int cpu, { struct csched2_private *prv = csched2_priv(new_ops); struct csched2_unit *svc = vdata; + struct sched_resource *sd = get_sched_res(cpu); unsigned rqi; ASSERT(pdata && svc && is_idle_vcpu(svc->vcpu)); @@ -3902,7 +3903,7 @@ csched2_switch_sched(struct scheduler *new_ops, unsigned int cpu, * this scheduler, and so it's safe to have taken it /before/ our * private global lock. */ - ASSERT(per_cpu(schedule_data, cpu).schedule_lock != &prv->rqd[rqi].lock); + ASSERT(sd->schedule_lock != &prv->rqd[rqi].lock); write_unlock(&prv->lock); diff --git a/xen/common/sched_null.c b/xen/common/sched_null.c index defcdfcf1e..9df6f867aa 100644 --- a/xen/common/sched_null.c +++ b/xen/common/sched_null.c @@ -168,7 +168,7 @@ static void init_pdata(struct null_private *prv, unsigned int cpu) static void null_init_pdata(const struct scheduler *ops, void *pdata, int cpu) { struct null_private *prv = null_priv(ops); - struct schedule_data *sd = &per_cpu(schedule_data, cpu); + struct sched_resource *sd = get_sched_res(cpu); /* alloc_pdata is not implemented, so we want this to be NULL. */ ASSERT(!pdata); @@ -277,7 +277,7 @@ pick_res(struct null_private *prv, struct sched_unit *unit) unsigned int cpu = v->processor, new_cpu; cpumask_t *cpus = cpupool_domain_cpumask(v->domain); - ASSERT(spin_is_locked(per_cpu(schedule_data, cpu).schedule_lock)); + ASSERT(spin_is_locked(get_sched_res(cpu)->schedule_lock)); for_each_affinity_balance_step( bs ) { @@ -389,7 +389,7 @@ static spinlock_t *null_switch_sched(struct scheduler *new_ops, unsigned int cpu, void *pdata, void *vdata) { - struct schedule_data *sd = &per_cpu(schedule_data, cpu); + struct sched_resource *sd = get_sched_res(cpu); struct null_private *prv = null_priv(new_ops); struct null_unit *nvc = vdata; diff --git a/xen/common/sched_rt.c b/xen/common/sched_rt.c index 8caec5c5dc..cee0d69d54 100644 --- a/xen/common/sched_rt.c +++ b/xen/common/sched_rt.c @@ -75,7 +75,7 @@ /* * Locking: * A global system lock is used to protect the RunQ and DepletedQ. - * The global lock is referenced by schedule_data.schedule_lock + * The global lock is referenced by sched_res->schedule_lock * from all physical cpus. * * The lock is already grabbed when calling wake/sleep/schedule/ functions @@ -176,7 +176,7 @@ static void repl_timer_handler(void *data); /* * System-wide private data, include global RunQueue/DepletedQ - * Global lock is referenced by schedule_data.schedule_lock from all + * Global lock is referenced by sched_res->schedule_lock from all * physical cpus. It can be grabbed via vcpu_schedule_lock_irq() */ struct rt_private { @@ -723,7 +723,7 @@ rt_init_pdata(const struct scheduler *ops, void *pdata, int cpu) } /* Move the scheduler lock to our global runqueue lock. */ - per_cpu(schedule_data, cpu).schedule_lock = &prv->lock; + get_sched_res(cpu)->schedule_lock = &prv->lock; /* _Not_ pcpu_schedule_unlock(): per_cpu().schedule_lock changed! */ spin_unlock_irqrestore(old_lock, flags); @@ -736,6 +736,7 @@ rt_switch_sched(struct scheduler *new_ops, unsigned int cpu, { struct rt_private *prv = rt_priv(new_ops); struct rt_unit *svc = vdata; + struct sched_resource *sd = get_sched_res(cpu); ASSERT(!pdata && svc && is_idle_vcpu(svc->vcpu)); @@ -745,7 +746,7 @@ rt_switch_sched(struct scheduler *new_ops, unsigned int cpu, * another scheduler, but that is how things need to be, for * preventing races. */ - ASSERT(per_cpu(schedule_data, cpu).schedule_lock != &prv->lock); + ASSERT(sd->schedule_lock != &prv->lock); /* * If we are the absolute first cpu being switched toward this diff --git a/xen/common/schedule.c b/xen/common/schedule.c index c0cec0b6ca..ea53bd7183 100644 --- a/xen/common/schedule.c +++ b/xen/common/schedule.c @@ -61,7 +61,6 @@ static void vcpu_singleshot_timer_fn(void *data); static void poll_timer_fn(void *data); /* This is global for now so that private implementations can reach it */ -DEFINE_PER_CPU(struct schedule_data, schedule_data); DEFINE_PER_CPU(struct scheduler *, scheduler); DEFINE_PER_CPU(struct sched_resource *, sched_res); @@ -157,7 +156,7 @@ static inline void vcpu_urgent_count_update(struct vcpu *v) !test_bit(v->vcpu_id, v->domain->poll_mask) ) { v->is_urgent = 0; - atomic_dec(&per_cpu(schedule_data,v->processor).urgent_count); + atomic_dec(&get_sched_res(v->processor)->urgent_count); } } else @@ -166,7 +165,7 @@ static inline void vcpu_urgent_count_update(struct vcpu *v) unlikely(test_bit(v->vcpu_id, v->domain->poll_mask)) ) { v->is_urgent = 1; - atomic_inc(&per_cpu(schedule_data,v->processor).urgent_count); + atomic_inc(&get_sched_res(v->processor)->urgent_count); } } } @@ -177,7 +176,7 @@ static inline void vcpu_runstate_change( s_time_t delta; ASSERT(v->runstate.state != new_state); - ASSERT(spin_is_locked(per_cpu(schedule_data,v->processor).schedule_lock)); + ASSERT(spin_is_locked(get_sched_res(v->processor)->schedule_lock)); vcpu_urgent_count_update(v); @@ -334,7 +333,7 @@ int sched_init_vcpu(struct vcpu *v, unsigned int processor) /* Idle VCPUs are scheduled immediately, so don't put them in runqueue. */ if ( is_idle_domain(d) ) { - per_cpu(schedule_data, v->processor).curr = unit; + get_sched_res(v->processor)->curr = unit; v->is_running = 1; } else @@ -459,7 +458,7 @@ void sched_destroy_vcpu(struct vcpu *v) kill_timer(&v->singleshot_timer); kill_timer(&v->poll_timer); if ( test_and_clear_bool(v->is_urgent) ) - atomic_dec(&per_cpu(schedule_data, v->processor).urgent_count); + atomic_dec(&get_sched_res(v->processor)->urgent_count); sched_remove_unit(vcpu_scheduler(v), unit); sched_free_vdata(vcpu_scheduler(v), unit->priv); sched_free_unit(unit); @@ -506,7 +505,7 @@ void sched_destroy_domain(struct domain *d) void vcpu_sleep_nosync_locked(struct vcpu *v) { - ASSERT(spin_is_locked(per_cpu(schedule_data,v->processor).schedule_lock)); + ASSERT(spin_is_locked(get_sched_res(v->processor)->schedule_lock)); if ( likely(!vcpu_runnable(v)) ) { @@ -601,8 +600,8 @@ static void vcpu_move_locked(struct vcpu *v, unsigned int new_cpu) */ if ( unlikely(v->is_urgent) && (old_cpu != new_cpu) ) { - atomic_inc(&per_cpu(schedule_data, new_cpu).urgent_count); - atomic_dec(&per_cpu(schedule_data, old_cpu).urgent_count); + atomic_inc(&get_sched_res(new_cpu)->urgent_count); + atomic_dec(&get_sched_res(old_cpu)->urgent_count); } /* @@ -668,20 +667,20 @@ static void vcpu_migrate_finish(struct vcpu *v) * are not correct any longer after evaluating old and new cpu holding * the locks. */ - old_lock = per_cpu(schedule_data, old_cpu).schedule_lock; - new_lock = per_cpu(schedule_data, new_cpu).schedule_lock; + old_lock = get_sched_res(old_cpu)->schedule_lock; + new_lock = get_sched_res(new_cpu)->schedule_lock; sched_spin_lock_double(old_lock, new_lock, &flags); old_cpu = v->processor; - if ( old_lock == per_cpu(schedule_data, old_cpu).schedule_lock ) + if ( old_lock == get_sched_res(old_cpu)->schedule_lock ) { /* * If we selected a CPU on the previosu iteration, check if it * remains suitable for running this vCPU. */ if ( pick_called && - (new_lock == per_cpu(schedule_data, new_cpu).schedule_lock) && + (new_lock == get_sched_res(new_cpu)->schedule_lock) && cpumask_test_cpu(new_cpu, v->cpu_hard_affinity) && cpumask_test_cpu(new_cpu, v->domain->cpupool->cpu_valid) ) break; @@ -689,7 +688,7 @@ static void vcpu_migrate_finish(struct vcpu *v) /* Select a new CPU. */ new_cpu = sched_pick_resource(vcpu_scheduler(v), v->sched_unit)->processor; - if ( (new_lock == per_cpu(schedule_data, new_cpu).schedule_lock) && + if ( (new_lock == get_sched_res(new_cpu)->schedule_lock) && cpumask_test_cpu(new_cpu, v->domain->cpupool->cpu_valid) ) break; pick_called = 1; @@ -1472,7 +1471,7 @@ static void schedule(void) struct scheduler *sched; unsigned long *tasklet_work = &this_cpu(tasklet_work_to_do); bool_t tasklet_work_scheduled = 0; - struct schedule_data *sd; + struct sched_resource *sd; spinlock_t *lock; struct task_slice next_slice; int cpu = smp_processor_id(); @@ -1481,7 +1480,7 @@ static void schedule(void) SCHED_STAT_CRANK(sched_run); - sd = &this_cpu(schedule_data); + sd = get_sched_res(cpu); /* Update tasklet scheduling status. */ switch ( *tasklet_work ) @@ -1623,15 +1622,14 @@ static void poll_timer_fn(void *data) static int cpu_schedule_up(unsigned int cpu) { - struct schedule_data *sd = &per_cpu(schedule_data, cpu); + struct sched_resource *sd; void *sched_priv; - struct sched_resource *res; - res = xzalloc(struct sched_resource); - if ( res == NULL ) + sd = xzalloc(struct sched_resource); + if ( sd == NULL ) return -ENOMEM; - res->processor = cpu; - set_sched_res(cpu, res); + sd->processor = cpu; + set_sched_res(cpu, sd); per_cpu(scheduler, cpu) = &ops; spin_lock_init(&sd->_lock); @@ -1687,7 +1685,7 @@ static int cpu_schedule_up(unsigned int cpu) static void cpu_schedule_down(unsigned int cpu) { - struct schedule_data *sd = &per_cpu(schedule_data, cpu); + struct sched_resource *sd = get_sched_res(cpu); struct scheduler *sched = per_cpu(scheduler, cpu); sched_free_pdata(sched, sd->sched_priv, cpu); @@ -1707,7 +1705,7 @@ static int cpu_schedule_callback( { unsigned int cpu = (unsigned long)hcpu; struct scheduler *sched = per_cpu(scheduler, cpu); - struct schedule_data *sd = &per_cpu(schedule_data, cpu); + struct sched_resource *sd = get_sched_res(cpu); int rc = 0; /* @@ -1864,10 +1862,10 @@ void __init scheduler_init(void) idle_domain->max_vcpus = nr_cpu_ids; if ( vcpu_create(idle_domain, 0, 0) == NULL ) BUG(); - this_cpu(schedule_data).curr = idle_vcpu[0]->sched_unit; - this_cpu(schedule_data).sched_priv = sched_alloc_pdata(&ops, 0); - BUG_ON(IS_ERR(this_cpu(schedule_data).sched_priv)); - sched_init_pdata(&ops, this_cpu(schedule_data).sched_priv, 0); + get_sched_res(0)->curr = idle_vcpu[0]->sched_unit; + get_sched_res(0)->sched_priv = sched_alloc_pdata(&ops, 0); + BUG_ON(IS_ERR(get_sched_res(0)->sched_priv)); + sched_init_pdata(&ops, get_sched_res(0)->sched_priv, 0); } /* @@ -1888,7 +1886,7 @@ int schedule_cpu_switch(unsigned int cpu, struct cpupool *c) struct scheduler *old_ops = per_cpu(scheduler, cpu); struct scheduler *new_ops = (c == NULL) ? &ops : c->sched; struct cpupool *old_pool = per_cpu(cpupool, cpu); - struct schedule_data *sd = &per_cpu(schedule_data, cpu); + struct sched_resource *sd = get_sched_res(cpu); spinlock_t *old_lock, *new_lock; /* @@ -2074,6 +2072,16 @@ void wait(void) schedule(); } +/* + * vcpu is urgent if vcpu is polling event channel + * + * if urgent vcpu exists, CPU should not enter deep C state + */ +int sched_has_urgent_vcpu(void) +{ + return atomic_read(&get_sched_res(smp_processor_id())->urgent_count); +} + #ifdef CONFIG_COMPAT #include "compat/schedule.c" #endif diff --git a/xen/include/asm-x86/cpuidle.h b/xen/include/asm-x86/cpuidle.h index 488f708305..5d7dffd228 100644 --- a/xen/include/asm-x86/cpuidle.h +++ b/xen/include/asm-x86/cpuidle.h @@ -4,7 +4,6 @@ #include #include #include -#include extern struct acpi_processor_power *processor_powers[]; @@ -27,14 +26,4 @@ void update_idle_stats(struct acpi_processor_power *, void update_last_cx_stat(struct acpi_processor_power *, struct acpi_processor_cx *, uint64_t); -/* - * vcpu is urgent if vcpu is polling event channel - * - * if urgent vcpu exists, CPU should not enter deep C state - */ -static inline int sched_has_urgent_vcpu(void) -{ - return atomic_read(&this_cpu(schedule_data).urgent_count); -} - #endif /* __X86_ASM_CPUIDLE_H__ */ diff --git a/xen/include/xen/sched-if.h b/xen/include/xen/sched-if.h index 9b7855e775..0443fe1d7e 100644 --- a/xen/include/xen/sched-if.h +++ b/xen/include/xen/sched-if.h @@ -33,22 +33,18 @@ extern int sched_ratelimit_us; * For cache betterness, keep the actual lock in the same cache area * as the rest of the struct. Just have the scheduler point to the * one it wants (This may be the one right in front of it).*/ -struct schedule_data { +struct sched_resource { spinlock_t *schedule_lock, _lock; struct sched_unit *curr; /* current task */ void *sched_priv; struct timer s_timer; /* scheduling timer */ atomic_t urgent_count; /* how many urgent vcpus */ + unsigned int processor; }; -#define curr_on_cpu(c) (per_cpu(schedule_data, c).curr) - -struct sched_resource { - unsigned int processor; -}; +#define curr_on_cpu(c) (get_sched_res(c)->curr) -DECLARE_PER_CPU(struct schedule_data, schedule_data); DECLARE_PER_CPU(struct scheduler *, scheduler); DECLARE_PER_CPU(struct cpupool *, cpupool); DECLARE_PER_CPU(struct sched_resource *, sched_res); @@ -79,7 +75,7 @@ static inline spinlock_t *kind##_schedule_lock##irq(param EXTRA_TYPE(arg)) \ { \ for ( ; ; ) \ { \ - spinlock_t *lock = per_cpu(schedule_data, cpu).schedule_lock; \ + spinlock_t *lock = get_sched_res(cpu)->schedule_lock; \ /* \ * v->processor may change when grabbing the lock; but \ * per_cpu(v->processor) may also change, if changing cpu pool \ @@ -89,7 +85,7 @@ static inline spinlock_t *kind##_schedule_lock##irq(param EXTRA_TYPE(arg)) \ * lock may be the same; this will succeed in that case. \ */ \ spin_lock##irq(lock, ## arg); \ - if ( likely(lock == per_cpu(schedule_data, cpu).schedule_lock) ) \ + if ( likely(lock == get_sched_res(cpu)->schedule_lock) ) \ return lock; \ spin_unlock##irq(lock, ## arg); \ } \ @@ -99,7 +95,7 @@ static inline spinlock_t *kind##_schedule_lock##irq(param EXTRA_TYPE(arg)) \ static inline void kind##_schedule_unlock##irq(spinlock_t *lock \ EXTRA_TYPE(arg), param) \ { \ - ASSERT(lock == per_cpu(schedule_data, cpu).schedule_lock); \ + ASSERT(lock == get_sched_res(cpu)->schedule_lock); \ spin_unlock##irq(lock, ## arg); \ } @@ -128,11 +124,11 @@ sched_unlock(vcpu, const struct vcpu *v, v->processor, _irqrestore, flags) static inline spinlock_t *pcpu_schedule_trylock(unsigned int cpu) { - spinlock_t *lock = per_cpu(schedule_data, cpu).schedule_lock; + spinlock_t *lock = get_sched_res(cpu)->schedule_lock; if ( !spin_trylock(lock) ) return NULL; - if ( lock == per_cpu(schedule_data, cpu).schedule_lock ) + if ( lock == get_sched_res(cpu)->schedule_lock ) return lock; spin_unlock(lock); return NULL; diff --git a/xen/include/xen/sched.h b/xen/include/xen/sched.h index d3a1a31c86..ee316cddd7 100644 --- a/xen/include/xen/sched.h +++ b/xen/include/xen/sched.h @@ -900,6 +900,7 @@ int vcpu_pin_override(struct vcpu *v, int cpu); void vcpu_runstate_get(struct vcpu *v, struct vcpu_runstate_info *runstate); uint64_t get_cpu_idle_time(unsigned int cpu); +int sched_has_urgent_vcpu(void); /* * Used by idle loop to decide whether there is work to do: