| Message ID | 20160212093705.16002.46867.stgit@Solace.station (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
>>> On 12.02.16 at 10:37, <dario.faggioli@citrix.com> wrote: > @@ -787,6 +788,16 @@ _csched_cpu_pick(const struct scheduler *ops, struct vcpu *vc, bool_t commit) > static int > csched_cpu_pick(const struct scheduler *ops, struct vcpu *vc) > { > + struct csched_vcpu *svc = CSCHED_VCPU(vc); > + > + /* > + * We have been called by vcpu_migrate() (in schedule.c), as part > + * of the process of seeing if vc can be migrated to another pcpu. > + * We make a note about this in svc->flags so that later, in > + * csched_vcpu_wake() (still called from vcpu_migrate()) we won't > + * get boosted, which we don't deserve as we are "only" migrating. > + */ > + set_bit(CSCHED_FLAG_VCPU_MIGRATING, &svc->flags); > return _csched_cpu_pick(ops, vc, 1); > } I think you either want __set_bit() here or ... > @@ -1022,11 +1033,18 @@ csched_vcpu_wake(const struct scheduler *ops, struct vcpu *vc) > * more CPU resource intensive VCPUs without impacting overall > * system fairness. > * > - * The one exception is for VCPUs of capped domains unpausing > - * after earning credits they had overspent. We don't boost > - * those. > + * There are two cases, when we don't want to boost: > + * - VCPUs that are waking up after a migration, rather than > + * after having block; > + * - VCPUs of capped domains unpausing after earning credits > + * they had overspent. > + * > + * Note that checking whether we are "only" migrating must be > + * done up front, as we do not want the clearing of the bit we > + * set in csched_cpu_pick() to be short-circuited away. > */ > - if ( svc->pri == CSCHED_PRI_TS_UNDER && > + if ( !__test_and_clear_bit(CSCHED_FLAG_VCPU_MIGRATING, &svc->flags) && > + svc->pri == CSCHED_PRI_TS_UNDER && > !test_bit(CSCHED_FLAG_VCPU_PARKED, &svc->flags) ) > { ... you ought to use test_and_clear_bit() here. Jan
On Fri, 2016-02-12 at 02:50 -0700, Jan Beulich wrote: > > > > On 12.02.16 at 10:37, <dario.faggioli@citrix.com> wrote: > > @@ -787,6 +788,16 @@ _csched_cpu_pick(const struct scheduler *ops, > > struct vcpu *vc, bool_t commit) > > static int > > csched_cpu_pick(const struct scheduler *ops, struct vcpu *vc) > > { > > + struct csched_vcpu *svc = CSCHED_VCPU(vc); > > + > > + /* > > + * We have been called by vcpu_migrate() (in schedule.c), as > > part > > + * of the process of seeing if vc can be migrated to another > > pcpu. > > + * We make a note about this in svc->flags so that later, in > > + * csched_vcpu_wake() (still called from vcpu_migrate()) we > > won't > > + * get boosted, which we don't deserve as we are "only" > > migrating. > > + */ > > + set_bit(CSCHED_FLAG_VCPU_MIGRATING, &svc->flags); > > return _csched_cpu_pick(ops, vc, 1); > > } > > I think you either want __set_bit() here or ... > Yes, this is completely serialized by the vcpu's scheduler lock, so I indeed want __set_bit(), sorry for the overlook. Thanks and Regards, Dario
[Yes, replying to myself] On Fri, 2016-02-12 at 11:50 +0100, Dario Faggioli wrote: > On Fri, 2016-02-12 at 02:50 -0700, Jan Beulich wrote: > > > > > On 12.02.16 at 10:37, <dario.faggioli@citrix.com> wrote: > > > @@ -787,6 +788,16 @@ _csched_cpu_pick(const struct scheduler > > > *ops, > > > struct vcpu *vc, bool_t commit) > > > static int > > > csched_cpu_pick(const struct scheduler *ops, struct vcpu *vc) > > > { > > > + struct csched_vcpu *svc = CSCHED_VCPU(vc); > > > + > > > + /* > > > + * We have been called by vcpu_migrate() (in schedule.c), as > > > part > > > + * of the process of seeing if vc can be migrated to another > > > pcpu. > > > + * We make a note about this in svc->flags so that later, in > > > + * csched_vcpu_wake() (still called from vcpu_migrate()) we > > > won't > > > + * get boosted, which we don't deserve as we are "only" > > > migrating. > > > + */ > > > + set_bit(CSCHED_FLAG_VCPU_MIGRATING, &svc->flags); > > > return _csched_cpu_pick(ops, vc, 1); > > > } > > > > I think you either want __set_bit() here or ... > > > Yes, this is completely serialized by the vcpu's scheduler lock, so I > indeed want __set_bit(), sorry for the overlook. > Which is indeed the case, in the case of this svc->flags, but not for other cases when svc->flags is used, for manipulating the other two existing flags (see, for instance be6507509454adf3bb5a50b9406c88504e996d5a "credit1: Use atomic bit operations for the flags structure"). So what I want is really the opposite of what I said above: set_bit() is ok, and I need the atomic test_and_clear(). -ENEEDMORECOFFEEATMORNING :-/ Thanks again and Regards, Dario
diff --git a/xen/common/sched_credit.c b/xen/common/sched_credit.c index 5708701..756e884 100644 --- a/xen/common/sched_credit.c +++ b/xen/common/sched_credit.c @@ -66,6 +66,7 @@ */ #define CSCHED_FLAG_VCPU_PARKED 0x0 /* VCPU over capped credits */ #define CSCHED_FLAG_VCPU_YIELD 0x1 /* VCPU yielding */ +#define CSCHED_FLAG_VCPU_MIGRATING 0x2 /* VCPU may have moved to a new pcpu */ /* @@ -787,6 +788,16 @@ _csched_cpu_pick(const struct scheduler *ops, struct vcpu *vc, bool_t commit) static int csched_cpu_pick(const struct scheduler *ops, struct vcpu *vc) { + struct csched_vcpu *svc = CSCHED_VCPU(vc); + + /* + * We have been called by vcpu_migrate() (in schedule.c), as part + * of the process of seeing if vc can be migrated to another pcpu. + * We make a note about this in svc->flags so that later, in + * csched_vcpu_wake() (still called from vcpu_migrate()) we won't + * get boosted, which we don't deserve as we are "only" migrating. + */ + set_bit(CSCHED_FLAG_VCPU_MIGRATING, &svc->flags); return _csched_cpu_pick(ops, vc, 1); } @@ -1022,11 +1033,18 @@ csched_vcpu_wake(const struct scheduler *ops, struct vcpu *vc) * more CPU resource intensive VCPUs without impacting overall * system fairness. * - * The one exception is for VCPUs of capped domains unpausing - * after earning credits they had overspent. We don't boost - * those. + * There are two cases, when we don't want to boost: + * - VCPUs that are waking up after a migration, rather than + * after having block; + * - VCPUs of capped domains unpausing after earning credits + * they had overspent. + * + * Note that checking whether we are "only" migrating must be + * done up front, as we do not want the clearing of the bit we + * set in csched_cpu_pick() to be short-circuited away. */ - if ( svc->pri == CSCHED_PRI_TS_UNDER && + if ( !__test_and_clear_bit(CSCHED_FLAG_VCPU_MIGRATING, &svc->flags) && + svc->pri == CSCHED_PRI_TS_UNDER && !test_bit(CSCHED_FLAG_VCPU_PARKED, &svc->flags) ) { TRACE_2D(TRC_CSCHED_BOOST_START, vc->domain->domain_id, vc->vcpu_id);
Moving a vCPU to a different pCPU means offlining it and then waking it up, on the new pCPU. Credit1 grants BOOST priority to vCPUs that wakes up, with the aim of improving I/O latency. The net effect of this all is that vCPUs get boosted when migrating, which shouldn't happen. For instance, this causes scheduling anomalies and, potentially, performance problems, as reported here: http://lists.xen.org/archives/html/xen-devel/2015-10/msg02851.html This patch fixes this by noting down (by means of a flag) the fact that the vCPU is about to undergo a migration. This way we can tell, later, during a wakeup, whether the vCPU is migrating or unblocking, and decide whether or not to apply the boosting. Signed-off-by: Dario Faggioli <dario.faggioli@citrix.com> --- Cc: George Dunlap <george.dunlap@eu.citrix.com> Cc: Jan Beulich <JBeulich@suse.com> --- Changes from v1: * rewritten, following suggestion got during review: there are no wakeup flags any longer, and all is done in sched_credit.c by setting a flag in csched_cpu_pick() and testing (and cleating) it in csched_vcpu_wake(). --- xen/common/sched_credit.c | 26 ++++++++++++++++++++++---- 1 file changed, 22 insertions(+), 4 deletions(-)