From patchwork Fri Sep 27 07:00:25 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Juergen Gross X-Patchwork-Id: 11163937 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 6052776 for ; Fri, 27 Sep 2019 07:03:03 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 3AE5A207FF for ; Fri, 27 Sep 2019 07:03:03 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 3AE5A207FF Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=suse.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1iDkGI-0003yT-OL; Fri, 27 Sep 2019 07:01:54 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1iDkGG-0003vK-Pv for xen-devel@lists.xenproject.org; Fri, 27 Sep 2019 07:01:52 +0000 X-Inumbo-ID: 91f3ca12-e0f4-11e9-bf31-bc764e2007e4 Received: from mx1.suse.de (unknown [195.135.220.15]) by localhost (Halon) with ESMTPS id 91f3ca12-e0f4-11e9-bf31-bc764e2007e4; Fri, 27 Sep 2019 07:01:04 +0000 (UTC) X-Virus-Scanned: by amavisd-new at test-mx.suse.de Received: from relay2.suse.de (unknown [195.135.220.254]) by mx1.suse.de (Postfix) with ESMTP id 075EBACEF; Fri, 27 Sep 2019 07:01:01 +0000 (UTC) From: Juergen Gross To: xen-devel@lists.xenproject.org Date: Fri, 27 Sep 2019 09:00:25 +0200 Message-Id: <20190927070050.12405-22-jgross@suse.com> X-Mailer: git-send-email 2.16.4 In-Reply-To: <20190927070050.12405-1-jgross@suse.com> References: <20190927070050.12405-1-jgross@suse.com> Subject: [Xen-devel] [PATCH v4 21/46] xen/sched: use sched_resource cpu instead smp_processor_id in schedulers X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: Juergen Gross , Tim Deegan , Stefano Stabellini , Wei Liu , Konrad Rzeszutek Wilk , George Dunlap , Andrew Cooper , Ian Jackson , Robert VanVossen , Dario Faggioli , Julien Grall , Josh Whitehead , Meng Xu , Jan Beulich MIME-Version: 1.0 Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" Especially in the do_schedule() functions of the different schedulers using smp_processor_id() for the local cpu number is correct only if the sched_unit is a single vcpu. As soon as larger sched_units are used most uses should be replaced by the master_cpu number of the local sched_resource instead. Add a helper to get that sched_resource master_cpu and modify the schedulers to use it in a correct way. Signed-off-by: Juergen Gross Reviewed-by: Dario Faggioli --- V3: - style correction (Jan Beulich) - rename cpu to cur_cpu when introducing sched_cpu in same function (Jan Beulich) - undo change in sched_idle_schedule() (Jan Beulich) V4: - simplify test for correct cpu (Jan Beulich) --- xen/common/sched_arinc653.c | 2 +- xen/common/sched_credit.c | 23 ++++++++++--------- xen/common/sched_credit2.c | 55 +++++++++++++++++++++++---------------------- xen/common/sched_null.c | 25 +++++++++++---------- xen/common/sched_rt.c | 19 ++++++++-------- xen/include/xen/sched-if.h | 5 +++++ 6 files changed, 69 insertions(+), 60 deletions(-) diff --git a/xen/common/sched_arinc653.c b/xen/common/sched_arinc653.c index f04d9c9cb1..2bc187c92b 100644 --- a/xen/common/sched_arinc653.c +++ b/xen/common/sched_arinc653.c @@ -519,7 +519,7 @@ a653sched_do_schedule( static unsigned int sched_index = 0; static s_time_t next_switch_time; a653sched_priv_t *sched_priv = SCHED_PRIV(ops); - const unsigned int cpu = smp_processor_id(); + const unsigned int cpu = sched_get_resource_cpu(smp_processor_id()); unsigned long flags; spin_lock_irqsave(&sched_priv->lock, flags); diff --git a/xen/common/sched_credit.c b/xen/common/sched_credit.c index 21c3a033b9..7f6ba35766 100644 --- a/xen/common/sched_credit.c +++ b/xen/common/sched_credit.c @@ -1684,7 +1684,7 @@ csched_load_balance(struct csched_private *prv, int cpu, int peer_cpu, first_cpu, peer_node, bstep; int node = cpu_to_node(cpu); - BUG_ON( cpu != sched_unit_master(snext->unit) ); + BUG_ON(get_sched_res(cpu) != snext->unit->res); online = cpupool_online_cpumask(c); /* @@ -1825,8 +1825,9 @@ static struct task_slice csched_schedule( const struct scheduler *ops, s_time_t now, bool_t tasklet_work_scheduled) { - const int cpu = smp_processor_id(); - struct list_head * const runq = RUNQ(cpu); + const unsigned int cur_cpu = smp_processor_id(); + const unsigned int sched_cpu = sched_get_resource_cpu(cur_cpu); + struct list_head * const runq = RUNQ(sched_cpu); struct sched_unit *unit = current->sched_unit; struct csched_unit * const scurr = CSCHED_UNIT(unit); struct csched_private *prv = CSCHED_PRIV(ops); @@ -1847,7 +1848,7 @@ csched_schedule( struct { unsigned cpu:16, tasklet:8, idle:8; } d; - d.cpu = cpu; + d.cpu = cur_cpu; d.tasklet = tasklet_work_scheduled; d.idle = is_idle_unit(unit); __trace_var(TRC_CSCHED_SCHEDULE, 1, sizeof(d), @@ -1937,7 +1938,7 @@ csched_schedule( { BUG_ON( is_idle_unit(unit) || list_empty(runq) ); /* Current has blocked. Update the runnable counter for this cpu. */ - dec_nr_runnable(cpu); + dec_nr_runnable(sched_cpu); } snext = __runq_elem(runq->next); @@ -1947,7 +1948,7 @@ csched_schedule( if ( tasklet_work_scheduled ) { TRACE_0D(TRC_CSCHED_SCHED_TASKLET); - snext = CSCHED_UNIT(sched_idle_unit(cpu)); + snext = CSCHED_UNIT(sched_idle_unit(sched_cpu)); snext->pri = CSCHED_PRI_TS_BOOST; } @@ -1967,7 +1968,7 @@ csched_schedule( if ( snext->pri > CSCHED_PRI_TS_OVER ) __runq_remove(snext); else - snext = csched_load_balance(prv, cpu, snext, &ret.migrated); + snext = csched_load_balance(prv, sched_cpu, snext, &ret.migrated); /* * Update idlers mask if necessary. When we're idling, other CPUs @@ -1975,12 +1976,12 @@ csched_schedule( */ if ( !tasklet_work_scheduled && snext->pri == CSCHED_PRI_IDLE ) { - if ( !cpumask_test_cpu(cpu, prv->idlers) ) - cpumask_set_cpu(cpu, prv->idlers); + if ( !cpumask_test_cpu(sched_cpu, prv->idlers) ) + cpumask_set_cpu(sched_cpu, prv->idlers); } - else if ( cpumask_test_cpu(cpu, prv->idlers) ) + else if ( cpumask_test_cpu(sched_cpu, prv->idlers) ) { - cpumask_clear_cpu(cpu, prv->idlers); + cpumask_clear_cpu(sched_cpu, prv->idlers); } if ( !is_idle_unit(snext->unit) ) diff --git a/xen/common/sched_credit2.c b/xen/common/sched_credit2.c index 3ee76323df..c4c6c69a0e 100644 --- a/xen/common/sched_credit2.c +++ b/xen/common/sched_credit2.c @@ -3450,7 +3450,8 @@ static struct task_slice csched2_schedule( const struct scheduler *ops, s_time_t now, bool tasklet_work_scheduled) { - const int cpu = smp_processor_id(); + const unsigned int cur_cpu = smp_processor_id(); + const unsigned int sched_cpu = sched_get_resource_cpu(cur_cpu); struct csched2_runqueue_data *rqd; struct sched_unit *currunit = current->sched_unit; struct csched2_unit * const scurr = csched2_unit(currunit); @@ -3462,22 +3463,22 @@ csched2_schedule( SCHED_STAT_CRANK(schedule); CSCHED2_UNIT_CHECK(currunit); - BUG_ON(!cpumask_test_cpu(cpu, &csched2_priv(ops)->initialized)); + BUG_ON(!cpumask_test_cpu(sched_cpu, &csched2_priv(ops)->initialized)); - rqd = c2rqd(ops, cpu); - BUG_ON(!cpumask_test_cpu(cpu, &rqd->active)); + rqd = c2rqd(ops, sched_cpu); + BUG_ON(!cpumask_test_cpu(sched_cpu, &rqd->active)); - ASSERT(spin_is_locked(get_sched_res(cpu)->schedule_lock)); + ASSERT(spin_is_locked(get_sched_res(sched_cpu)->schedule_lock)); BUG_ON(!is_idle_unit(currunit) && scurr->rqd != rqd); /* Clear "tickled" bit now that we've been scheduled */ - tickled = cpumask_test_cpu(cpu, &rqd->tickled); + tickled = cpumask_test_cpu(sched_cpu, &rqd->tickled); if ( tickled ) { - __cpumask_clear_cpu(cpu, &rqd->tickled); + __cpumask_clear_cpu(sched_cpu, &rqd->tickled); cpumask_andnot(cpumask_scratch, &rqd->idle, &rqd->tickled); - smt_idle_mask_set(cpu, cpumask_scratch, &rqd->smt_idle); + smt_idle_mask_set(sched_cpu, cpumask_scratch, &rqd->smt_idle); } if ( unlikely(tb_init_done) ) @@ -3486,11 +3487,11 @@ csched2_schedule( unsigned cpu:16, rq_id:16; unsigned tasklet:8, idle:8, smt_idle:8, tickled:8; } d; - d.cpu = cpu; - d.rq_id = c2r(cpu); + d.cpu = cur_cpu; + d.rq_id = c2r(sched_cpu); d.tasklet = tasklet_work_scheduled; d.idle = is_idle_unit(currunit); - d.smt_idle = cpumask_test_cpu(cpu, &rqd->smt_idle); + d.smt_idle = cpumask_test_cpu(sched_cpu, &rqd->smt_idle); d.tickled = tickled; __trace_var(TRC_CSCHED2_SCHEDULE, 1, sizeof(d), @@ -3530,10 +3531,10 @@ csched2_schedule( { __clear_bit(__CSFLAG_unit_yield, &scurr->flags); trace_var(TRC_CSCHED2_SCHED_TASKLET, 1, 0, NULL); - snext = csched2_unit(sched_idle_unit(cpu)); + snext = csched2_unit(sched_idle_unit(sched_cpu)); } else - snext = runq_candidate(rqd, scurr, cpu, now, &skipped_units); + snext = runq_candidate(rqd, scurr, sched_cpu, now, &skipped_units); /* If switching from a non-idle runnable unit, put it * back on the runqueue. */ @@ -3558,10 +3559,10 @@ csched2_schedule( } /* Clear the idle mask if necessary */ - if ( cpumask_test_cpu(cpu, &rqd->idle) ) + if ( cpumask_test_cpu(sched_cpu, &rqd->idle) ) { - __cpumask_clear_cpu(cpu, &rqd->idle); - smt_idle_mask_clear(cpu, &rqd->smt_idle); + __cpumask_clear_cpu(sched_cpu, &rqd->idle); + smt_idle_mask_clear(sched_cpu, &rqd->smt_idle); } /* @@ -3580,18 +3581,18 @@ csched2_schedule( */ if ( skipped_units == 0 && snext->credit <= CSCHED2_CREDIT_RESET ) { - reset_credit(ops, cpu, now, snext); - balance_load(ops, cpu, now); + reset_credit(ops, sched_cpu, now, snext); + balance_load(ops, sched_cpu, now); } snext->start_time = now; snext->tickled_cpu = -1; /* Safe because lock for old processor is held */ - if ( sched_unit_master(snext->unit) != cpu ) + if ( sched_unit_master(snext->unit) != sched_cpu ) { snext->credit += CSCHED2_MIGRATE_COMPENSATION; - sched_set_res(snext->unit, get_sched_res(cpu)); + sched_set_res(snext->unit, get_sched_res(sched_cpu)); SCHED_STAT_CRANK(migrated); ret.migrated = 1; } @@ -3604,17 +3605,17 @@ csched2_schedule( */ if ( tasklet_work_scheduled ) { - if ( cpumask_test_cpu(cpu, &rqd->idle) ) + if ( cpumask_test_cpu(sched_cpu, &rqd->idle) ) { - __cpumask_clear_cpu(cpu, &rqd->idle); - smt_idle_mask_clear(cpu, &rqd->smt_idle); + __cpumask_clear_cpu(sched_cpu, &rqd->idle); + smt_idle_mask_clear(sched_cpu, &rqd->smt_idle); } } - else if ( !cpumask_test_cpu(cpu, &rqd->idle) ) + else if ( !cpumask_test_cpu(sched_cpu, &rqd->idle) ) { - __cpumask_set_cpu(cpu, &rqd->idle); + __cpumask_set_cpu(sched_cpu, &rqd->idle); cpumask_andnot(cpumask_scratch, &rqd->idle, &rqd->tickled); - smt_idle_mask_set(cpu, cpumask_scratch, &rqd->smt_idle); + smt_idle_mask_set(sched_cpu, cpumask_scratch, &rqd->smt_idle); } /* Make sure avgload gets updated periodically even * if there's no activity */ @@ -3624,7 +3625,7 @@ csched2_schedule( /* * Return task to run next... */ - ret.time = csched2_runtime(ops, cpu, snext, now); + ret.time = csched2_runtime(ops, sched_cpu, snext, now); ret.task = snext->unit; CSCHED2_UNIT_CHECK(ret.task); diff --git a/xen/common/sched_null.c b/xen/common/sched_null.c index 4c9eed81ae..51edc3dbb9 100644 --- a/xen/common/sched_null.c +++ b/xen/common/sched_null.c @@ -784,7 +784,8 @@ static struct task_slice null_schedule(const struct scheduler *ops, bool_t tasklet_work_scheduled) { unsigned int bs; - const unsigned int cpu = smp_processor_id(); + const unsigned int cur_cpu = smp_processor_id(); + const unsigned int sched_cpu = sched_get_resource_cpu(cur_cpu); struct null_private *prv = null_priv(ops); struct null_unit *wvc; struct task_slice ret; @@ -798,16 +799,16 @@ static struct task_slice null_schedule(const struct scheduler *ops, uint16_t tasklet, cpu; int16_t unit, dom; } d; - d.cpu = cpu; + d.cpu = cur_cpu; d.tasklet = tasklet_work_scheduled; - if ( per_cpu(npc, cpu).unit == NULL ) + if ( per_cpu(npc, sched_cpu).unit == NULL ) { d.unit = d.dom = -1; } else { - d.unit = per_cpu(npc, cpu).unit->unit_id; - d.dom = per_cpu(npc, cpu).unit->domain->domain_id; + d.unit = per_cpu(npc, sched_cpu).unit->unit_id; + d.dom = per_cpu(npc, sched_cpu).unit->domain->domain_id; } __trace_var(TRC_SNULL_SCHEDULE, 1, sizeof(d), &d); } @@ -815,10 +816,10 @@ static struct task_slice null_schedule(const struct scheduler *ops, if ( tasklet_work_scheduled ) { trace_var(TRC_SNULL_TASKLET, 1, 0, NULL); - ret.task = sched_idle_unit(cpu); + ret.task = sched_idle_unit(sched_cpu); } else - ret.task = per_cpu(npc, cpu).unit; + ret.task = per_cpu(npc, sched_cpu).unit; ret.migrated = 0; ret.time = -1; @@ -849,9 +850,9 @@ static struct task_slice null_schedule(const struct scheduler *ops, !has_soft_affinity(wvc->unit) ) continue; - if ( unit_check_affinity(wvc->unit, cpu, bs) ) + if ( unit_check_affinity(wvc->unit, sched_cpu, bs) ) { - unit_assign(prv, wvc->unit, cpu); + unit_assign(prv, wvc->unit, sched_cpu); list_del_init(&wvc->waitq_elem); ret.task = wvc->unit; goto unlock; @@ -861,12 +862,12 @@ static struct task_slice null_schedule(const struct scheduler *ops, unlock: spin_unlock(&prv->waitq_lock); - if ( ret.task == NULL && !cpumask_test_cpu(cpu, &prv->cpus_free) ) - cpumask_set_cpu(cpu, &prv->cpus_free); + if ( ret.task == NULL && !cpumask_test_cpu(sched_cpu, &prv->cpus_free) ) + cpumask_set_cpu(sched_cpu, &prv->cpus_free); } if ( unlikely(ret.task == NULL || !unit_runnable(ret.task)) ) - ret.task = sched_idle_unit(cpu); + ret.task = sched_idle_unit(sched_cpu); NULL_UNIT_CHECK(ret.task); return ret; diff --git a/xen/common/sched_rt.c b/xen/common/sched_rt.c index 3e8852c348..151353b9a0 100644 --- a/xen/common/sched_rt.c +++ b/xen/common/sched_rt.c @@ -1056,7 +1056,8 @@ runq_pick(const struct scheduler *ops, const cpumask_t *mask) static struct task_slice rt_schedule(const struct scheduler *ops, s_time_t now, bool_t tasklet_work_scheduled) { - const int cpu = smp_processor_id(); + const unsigned int cur_cpu = smp_processor_id(); + const unsigned int sched_cpu = sched_get_resource_cpu(cur_cpu); struct rt_private *prv = rt_priv(ops); struct rt_unit *const scurr = rt_unit(current->sched_unit); struct rt_unit *snext = NULL; @@ -1068,9 +1069,9 @@ rt_schedule(const struct scheduler *ops, s_time_t now, bool_t tasklet_work_sched struct __packed { unsigned cpu:16, tasklet:8, tickled:4, idle:4; } d; - d.cpu = cpu; + d.cpu = cur_cpu; d.tasklet = tasklet_work_scheduled; - d.tickled = cpumask_test_cpu(cpu, &prv->tickled); + d.tickled = cpumask_test_cpu(sched_cpu, &prv->tickled); d.idle = is_idle_unit(currunit); trace_var(TRC_RTDS_SCHEDULE, 1, sizeof(d), @@ -1078,7 +1079,7 @@ rt_schedule(const struct scheduler *ops, s_time_t now, bool_t tasklet_work_sched } /* clear ticked bit now that we've been scheduled */ - cpumask_clear_cpu(cpu, &prv->tickled); + cpumask_clear_cpu(sched_cpu, &prv->tickled); /* burn_budget would return for IDLE UNIT */ burn_budget(ops, scurr, now); @@ -1086,13 +1087,13 @@ rt_schedule(const struct scheduler *ops, s_time_t now, bool_t tasklet_work_sched if ( tasklet_work_scheduled ) { trace_var(TRC_RTDS_SCHED_TASKLET, 1, 0, NULL); - snext = rt_unit(sched_idle_unit(cpu)); + snext = rt_unit(sched_idle_unit(sched_cpu)); } else { - snext = runq_pick(ops, cpumask_of(cpu)); + snext = runq_pick(ops, cpumask_of(sched_cpu)); if ( snext == NULL ) - snext = rt_unit(sched_idle_unit(cpu)); + snext = rt_unit(sched_idle_unit(sched_cpu)); /* if scurr has higher priority and budget, still pick scurr */ if ( !is_idle_unit(currunit) && @@ -1117,9 +1118,9 @@ rt_schedule(const struct scheduler *ops, s_time_t now, bool_t tasklet_work_sched q_remove(snext); __set_bit(__RTDS_scheduled, &snext->flags); } - if ( sched_unit_master(snext->unit) != cpu ) + if ( sched_unit_master(snext->unit) != sched_cpu ) { - sched_set_res(snext->unit, get_sched_res(cpu)); + sched_set_res(snext->unit, get_sched_res(sched_cpu)); ret.migrated = 1; } ret.time = snext->cur_budget; /* invoke the scheduler next time */ diff --git a/xen/include/xen/sched-if.h b/xen/include/xen/sched-if.h index 4797ba3f2b..d7fad0cbcc 100644 --- a/xen/include/xen/sched-if.h +++ b/xen/include/xen/sched-if.h @@ -150,6 +150,11 @@ static inline struct sched_unit *sched_idle_unit(unsigned int cpu) return idle_vcpu[cpu]->sched_unit; } +static inline unsigned int sched_get_resource_cpu(unsigned int cpu) +{ + return get_sched_res(cpu)->master_cpu; +} + /* * Scratch space, for avoiding having too many cpumask_t on the stack. * Within each scheduler, when using the scratch mask of one pCPU: