| Message ID | 148977477739.22479.7043154591081622447.stgit@Palanthas.fritz.box (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
On 17/03/17 19:19, Dario Faggioli wrote: > Within context_saved(), we call the context_saved hook, > and we use VCPU2OP() to determine from what scheduler. > VCPU2OP uses DOM2OP, which uses d->cpupool, which is > NULL when d is the idle domain. And in that case, > DOM2OP just returns ops, the scheduler of cpupool0. > > Therefore, if: > - cpupool0's scheduler defines context_saved (like > Credit2 and RTDS do), > - we are not in cpupool0 (i.e., our scheduler is > not ops), > - we are context switching from idle, > > we call VCPU2OP(idle_vcpu), which means > DOM2OP(idle->cpupool), which is ops. > > Therefore, we both: > - check if context_saved is defined in the wrong > scheduler; > - if yes, call the wrong one. > > When using Credit2 at boot, and also Credit2 in > the other cpupool, this is wrong but innocuous, > because it only involves the idle vcpus. > > When using Credit2 at boot, and Credit1 in the > other cpupool, this is *totally* wrong, and > it's by chance it does not explode! > > When using Credit2 and other schedulers I'm > developping, I hit the following assert (in > sched_credit2.c, on a CPU inside a cpupool that > does not use Credit2): > > csched2_context_saved() > { > ... > ASSERT(!vcpu_on_runq(svc)); > ... > } > > Fix this by dealing explicitly, in VCPU2OP, with > idle vcpus, returning the scheduler of the pCPU > they (always) run on. > > While there, rename VCPU2OP itself to something > that makes it easier to understand what it does. > > Signed-off-by: Dario Faggioli <dario.faggioli@citrix.com> Reviewed-by: Juergen Gross <jgross@suse.com> Juergen
On 17/03/17 19:19, Dario Faggioli wrote: > Within context_saved(), we call the context_saved hook, > and we use VCPU2OP() to determine from what scheduler. > VCPU2OP uses DOM2OP, which uses d->cpupool, which is > NULL when d is the idle domain. And in that case, > DOM2OP just returns ops, the scheduler of cpupool0. > > Therefore, if: > - cpupool0's scheduler defines context_saved (like > Credit2 and RTDS do), > - we are not in cpupool0 (i.e., our scheduler is > not ops), > - we are context switching from idle, > > we call VCPU2OP(idle_vcpu), which means > DOM2OP(idle->cpupool), which is ops. > > Therefore, we both: > - check if context_saved is defined in the wrong > scheduler; > - if yes, call the wrong one. > > When using Credit2 at boot, and also Credit2 in > the other cpupool, this is wrong but innocuous, > because it only involves the idle vcpus. > > When using Credit2 at boot, and Credit1 in the > other cpupool, this is *totally* wrong, and > it's by chance it does not explode! > > When using Credit2 and other schedulers I'm > developping, I hit the following assert (in > sched_credit2.c, on a CPU inside a cpupool that > does not use Credit2): > > csched2_context_saved() > { > ... > ASSERT(!vcpu_on_runq(svc)); > ... > } > > Fix this by dealing explicitly, in VCPU2OP, with > idle vcpus, returning the scheduler of the pCPU > they (always) run on. > > While there, rename VCPU2OP itself to something > that makes it easier to understand what it does. > > Signed-off-by: Dario Faggioli <dario.faggioli@citrix.com> Reviewed-by: Juergen Gross <jgross@suse.com> Juergen
>>> On 17.03.17 at 19:29, <jgross@suse.com> wrote: > On 17/03/17 19:19, Dario Faggioli wrote: >> Within context_saved(), we call the context_saved hook, >> and we use VCPU2OP() to determine from what scheduler. >> VCPU2OP uses DOM2OP, which uses d->cpupool, which is >> NULL when d is the idle domain. And in that case, >> DOM2OP just returns ops, the scheduler of cpupool0. >> >> Therefore, if: >> - cpupool0's scheduler defines context_saved (like >> Credit2 and RTDS do), >> - we are not in cpupool0 (i.e., our scheduler is >> not ops), >> - we are context switching from idle, >> >> we call VCPU2OP(idle_vcpu), which means >> DOM2OP(idle->cpupool), which is ops. >> >> Therefore, we both: >> - check if context_saved is defined in the wrong >> scheduler; >> - if yes, call the wrong one. >> >> When using Credit2 at boot, and also Credit2 in >> the other cpupool, this is wrong but innocuous, >> because it only involves the idle vcpus. >> >> When using Credit2 at boot, and Credit1 in the >> other cpupool, this is *totally* wrong, and >> it's by chance it does not explode! >> >> When using Credit2 and other schedulers I'm >> developping, I hit the following assert (in >> sched_credit2.c, on a CPU inside a cpupool that >> does not use Credit2): >> >> csched2_context_saved() >> { >> ... >> ASSERT(!vcpu_on_runq(svc)); >> ... >> } >> >> Fix this by dealing explicitly, in VCPU2OP, with >> idle vcpus, returning the scheduler of the pCPU >> they (always) run on. >> >> While there, rename VCPU2OP itself to something >> that makes it easier to understand what it does. >> >> Signed-off-by: Dario Faggioli <dario.faggioli@citrix.com> > > Reviewed-by: Juergen Gross <jgross@suse.com> This came through twice, so I'm now wondering whether one of them was meant for 2/2, or whether this was just some infrastructure glitch. Jan
On 17/03/17 18:19, Dario Faggioli wrote: > Within context_saved(), we call the context_saved hook, > and we use VCPU2OP() to determine from what scheduler. > VCPU2OP uses DOM2OP, which uses d->cpupool, which is > NULL when d is the idle domain. And in that case, > DOM2OP just returns ops, the scheduler of cpupool0. > > Therefore, if: > - cpupool0's scheduler defines context_saved (like > Credit2 and RTDS do), > - we are not in cpupool0 (i.e., our scheduler is > not ops), > - we are context switching from idle, > > we call VCPU2OP(idle_vcpu), which means > DOM2OP(idle->cpupool), which is ops. > > Therefore, we both: > - check if context_saved is defined in the wrong > scheduler; > - if yes, call the wrong one. > > When using Credit2 at boot, and also Credit2 in > the other cpupool, this is wrong but innocuous, > because it only involves the idle vcpus. > > When using Credit2 at boot, and Credit1 in the > other cpupool, this is *totally* wrong, and > it's by chance it does not explode! > > When using Credit2 and other schedulers I'm > developping, I hit the following assert (in > sched_credit2.c, on a CPU inside a cpupool that > does not use Credit2): > > csched2_context_saved() > { > ... > ASSERT(!vcpu_on_runq(svc)); > ... > } > > Fix this by dealing explicitly, in VCPU2OP, with > idle vcpus, returning the scheduler of the pCPU > they (always) run on. > > While there, rename VCPU2OP itself to something > that makes it easier to understand what it does. > > Signed-off-by: Dario Faggioli <dario.faggioli@citrix.com> Reviewed-by: George Dunlap <george.dunlap@citrix.com>
On 17/03/17 18:19, Dario Faggioli wrote: > Within context_saved(), we call the context_saved hook, > and we use VCPU2OP() to determine from what scheduler. > VCPU2OP uses DOM2OP, which uses d->cpupool, which is > NULL when d is the idle domain. And in that case, > DOM2OP just returns ops, the scheduler of cpupool0. > > Therefore, if: > - cpupool0's scheduler defines context_saved (like > Credit2 and RTDS do), > - we are not in cpupool0 (i.e., our scheduler is > not ops), > - we are context switching from idle, > > we call VCPU2OP(idle_vcpu), which means > DOM2OP(idle->cpupool), which is ops. > > Therefore, we both: > - check if context_saved is defined in the wrong > scheduler; > - if yes, call the wrong one. > > When using Credit2 at boot, and also Credit2 in > the other cpupool, this is wrong but innocuous, > because it only involves the idle vcpus. > > When using Credit2 at boot, and Credit1 in the > other cpupool, this is *totally* wrong, and > it's by chance it does not explode! > > When using Credit2 and other schedulers I'm > developping, I hit the following assert (in > sched_credit2.c, on a CPU inside a cpupool that > does not use Credit2): > > csched2_context_saved() > { > ... > ASSERT(!vcpu_on_runq(svc)); > ... > } > > Fix this by dealing explicitly, in VCPU2OP, with > idle vcpus, returning the scheduler of the pCPU > they (always) run on. > > While there, rename VCPU2OP itself to something > that makes it easier to understand what it does. This seems to have been moved into patch 2/2 -- I'll remove this paragraph on commit if that's OK with you. -George
On Mon, 2017-03-27 at 12:49 +0100, George Dunlap wrote: > On 17/03/17 18:19, Dario Faggioli wrote: > > Fix this by dealing explicitly, in VCPU2OP, with > > idle vcpus, returning the scheduler of the pCPU > > they (always) run on. > > > > While there, rename VCPU2OP itself to something > > that makes it easier to understand what it does. > > This seems to have been moved into patch 2/2 -- I'll remove this > paragraph on commit if that's OK with you. > Mmm... Yes, sorry for this. I'm ok with you dropping the paragraph. Thanks. Dario
diff --git a/xen/common/schedule.c b/xen/common/schedule.c index 223a120..d344b7c 100644 --- a/xen/common/schedule.c +++ b/xen/common/schedule.c @@ -78,7 +78,27 @@ static struct scheduler __read_mostly ops; : (typeof((opsptr)->fn(opsptr, ##__VA_ARGS__)))0 ) #define DOM2OP(_d) (((_d)->cpupool == NULL) ? &ops : ((_d)->cpupool->sched)) -#define VCPU2OP(_v) (DOM2OP((_v)->domain)) +static inline struct scheduler *VCPU2OP(const struct vcpu *v) +{ + struct domain *d = v->domain; + + if ( likely(d->cpupool != NULL) ) + return d->cpupool->sched; + + /* + * If d->cpupool is NULL, this is a vCPU of the idle domain. And this + * case is special because the idle domain does not really belong to + * a cpupool and, hence, doesn't really have a scheduler). In fact, its + * vCPUs (may) run on pCPUs which are in different pools, with different + * schedulers. + * + * What we want, in this case, is the scheduler of the pCPU where this + * particular idle vCPU is running. And, since v->processor never changes + * for idle vCPUs, it is safe to use it, with no locks, to figure that out. + */ + ASSERT(is_idle_domain(d)); + return per_cpu(scheduler, v->processor); +} #define VCPU2ONLINE(_v) cpupool_domain_cpumask((_v)->domain) static inline void trace_runstate_change(struct vcpu *v, int new_state)
Within context_saved(), we call the context_saved hook, and we use VCPU2OP() to determine from what scheduler. VCPU2OP uses DOM2OP, which uses d->cpupool, which is NULL when d is the idle domain. And in that case, DOM2OP just returns ops, the scheduler of cpupool0. Therefore, if: - cpupool0's scheduler defines context_saved (like Credit2 and RTDS do), - we are not in cpupool0 (i.e., our scheduler is not ops), - we are context switching from idle, we call VCPU2OP(idle_vcpu), which means DOM2OP(idle->cpupool), which is ops. Therefore, we both: - check if context_saved is defined in the wrong scheduler; - if yes, call the wrong one. When using Credit2 at boot, and also Credit2 in the other cpupool, this is wrong but innocuous, because it only involves the idle vcpus. When using Credit2 at boot, and Credit1 in the other cpupool, this is *totally* wrong, and it's by chance it does not explode! When using Credit2 and other schedulers I'm developping, I hit the following assert (in sched_credit2.c, on a CPU inside a cpupool that does not use Credit2): csched2_context_saved() { ... ASSERT(!vcpu_on_runq(svc)); ... } Fix this by dealing explicitly, in VCPU2OP, with idle vcpus, returning the scheduler of the pCPU they (always) run on. While there, rename VCPU2OP itself to something that makes it easier to understand what it does. Signed-off-by: Dario Faggioli <dario.faggioli@citrix.com> --- Cc: George Dunlap <george.dunlap@citrix.com> Cc: Juergen Gross <jgross@suse.com> Cc: Jan Beulich <jbeulich@suse.com> --- Changes from v1: - refactored according to review comments. Added code comments and an ASSERT(). --- Cc-ing Jan, as this should be backported at least to 4.8, but, IMO, as back as possible. --- xen/common/schedule.c | 22 +++++++++++++++++++++- 1 file changed, 21 insertions(+), 1 deletion(-)