| Message ID | 5CD2CFA1020000780022CCA6@prv1-mh.provo.novell.com (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
| Series | x86: EOI timer corrections / improvements | expand |
On Wed, May 08, 2019 at 06:46:25AM -0600, Jan Beulich wrote: > The timer needs to remain active only until all pending IRQ instances > have seen EOIs from their respective domains. Stop it when the in-flight > count has reached zero in desc_guest_eoi(). Note that this is race free > (with __do_IRQ_guest()), as the IRQ descriptor lock is being held at > that point. > > Also pull up stopping of the timer in __do_IRQ_guest() itself: Instead > of stopping it immediately before re-setting, stop it as soon as we've > made it past any early returns from the function (and hence we're sure > it'll get set again). > > Finally bail from the actual timer handler in case we find the timer > already active again by the time we've managed to acquire the IRQ > descriptor lock. Without this we may forcibly EOI an IRQ immediately > after it got sent to a guest. For this, timer_is_active() gets split out > of active_timer(), deliberately moving just one of the two ASSERT()s (to > allow the function to be used also on a never initialized timer). AFAICT timer_is_active is exclusively used in irq_guest_eoi_timer_fn, which must have initialized the timer in order for irq_guest_eoi_timer_fn to be called, and hence I'm not sure why you need to be able to call timer_is_active with an uninitialized timer. Is this maybe used by other patches? Thanks, Roger.
>>> On 16.05.19 at 12:32, <roger.pau@citrix.com> wrote: > On Wed, May 08, 2019 at 06:46:25AM -0600, Jan Beulich wrote: >> The timer needs to remain active only until all pending IRQ instances >> have seen EOIs from their respective domains. Stop it when the in-flight >> count has reached zero in desc_guest_eoi(). Note that this is race free >> (with __do_IRQ_guest()), as the IRQ descriptor lock is being held at >> that point. >> >> Also pull up stopping of the timer in __do_IRQ_guest() itself: Instead >> of stopping it immediately before re-setting, stop it as soon as we've >> made it past any early returns from the function (and hence we're sure >> it'll get set again). >> >> Finally bail from the actual timer handler in case we find the timer >> already active again by the time we've managed to acquire the IRQ >> descriptor lock. Without this we may forcibly EOI an IRQ immediately >> after it got sent to a guest. For this, timer_is_active() gets split out >> of active_timer(), deliberately moving just one of the two ASSERT()s (to >> allow the function to be used also on a never initialized timer). > > AFAICT timer_is_active is exclusively used in irq_guest_eoi_timer_fn, > which must have initialized the timer in order for > irq_guest_eoi_timer_fn to be called, and hence I'm not sure why you > need to be able to call timer_is_active with an uninitialized timer. It's not needed here, but I consider this useful behavior when used outside of the specific timer's handler. > Is this maybe used by other patches? None that I would have in the works. Jan
On Thu, May 16, 2019 at 04:50:22AM -0600, Jan Beulich wrote: > >>> On 16.05.19 at 12:32, <roger.pau@citrix.com> wrote: > > On Wed, May 08, 2019 at 06:46:25AM -0600, Jan Beulich wrote: > >> The timer needs to remain active only until all pending IRQ instances > >> have seen EOIs from their respective domains. Stop it when the in-flight > >> count has reached zero in desc_guest_eoi(). Note that this is race free > >> (with __do_IRQ_guest()), as the IRQ descriptor lock is being held at > >> that point. > >> > >> Also pull up stopping of the timer in __do_IRQ_guest() itself: Instead > >> of stopping it immediately before re-setting, stop it as soon as we've > >> made it past any early returns from the function (and hence we're sure > >> it'll get set again). > >> > >> Finally bail from the actual timer handler in case we find the timer > >> already active again by the time we've managed to acquire the IRQ > >> descriptor lock. Without this we may forcibly EOI an IRQ immediately > >> after it got sent to a guest. For this, timer_is_active() gets split out > >> of active_timer(), deliberately moving just one of the two ASSERT()s (to > >> allow the function to be used also on a never initialized timer). > > > > AFAICT timer_is_active is exclusively used in irq_guest_eoi_timer_fn, > > which must have initialized the timer in order for > > irq_guest_eoi_timer_fn to be called, and hence I'm not sure why you > > need to be able to call timer_is_active with an uninitialized timer. > > It's not needed here, but I consider this useful behavior when used > outside of the specific timer's handler. > > > Is this maybe used by other patches? > > None that I would have in the works. Then IMO I would rather make timer_is_active a replacement for active_timer (or just move active_timer to the header) if there's no user that can call timer_is_active with an uninitialized timer. Ie: I would keep the asserts as restrictive as possible unless there's a user that requires less restrictive assertions. Anyway, the change is an improvement, so with or without that changed: Reviewed-by: Roger Pau Monné <roger.pau@citrix.com> Thanks.
On 08/05/2019 13:46, Jan Beulich wrote: > The timer needs to remain active only until all pending IRQ instances > have seen EOIs from their respective domains. Stop it when the in-flight > count has reached zero in desc_guest_eoi(). Note that this is race free > (with __do_IRQ_guest()), as the IRQ descriptor lock is being held at > that point. > > Also pull up stopping of the timer in __do_IRQ_guest() itself: Instead > of stopping it immediately before re-setting, stop it as soon as we've > made it past any early returns from the function (and hence we're sure > it'll get set again). Why this this a good thing? > > Finally bail from the actual timer handler in case we find the timer > already active again by the time we've managed to acquire the IRQ > descriptor lock. Without this we may forcibly EOI an IRQ immediately > after it got sent to a guest. For this, timer_is_active() gets split out > of active_timer(), deliberately moving just one of the two ASSERT()s (to > allow the function to be used also on a never initialized timer). > > Signed-off-by: Jan Beulich <jbeulich@suse.com> > > --- a/xen/arch/x86/irq.c > +++ b/xen/arch/x86/irq.c > @@ -1115,6 +1115,9 @@ static void irq_guest_eoi_timer_fn(void > > action = (irq_guest_action_t *)desc->action; > /* Another instance of this timer already running? Skip everything to avoid forcing an EOI early. */ ~Andrew > + if ( timer_is_active(&action->eoi_timer) ) > + goto out; > + > if ( action->ack_type != ACKTYPE_NONE ) > { > unsigned int i; >
>>> On 05.06.19 at 19:04, <andrew.cooper3@citrix.com> wrote: > On 08/05/2019 13:46, Jan Beulich wrote: >> The timer needs to remain active only until all pending IRQ instances >> have seen EOIs from their respective domains. Stop it when the in-flight >> count has reached zero in desc_guest_eoi(). Note that this is race free >> (with __do_IRQ_guest()), as the IRQ descriptor lock is being held at >> that point. >> >> Also pull up stopping of the timer in __do_IRQ_guest() itself: Instead >> of stopping it immediately before re-setting, stop it as soon as we've >> made it past any early returns from the function (and hence we're sure >> it'll get set again). > > Why this this a good thing? For it to not fire when it doesn't need to. If we're about to set a new timeout, we clearly don't want the previous one to have any effect anymore. >> --- a/xen/arch/x86/irq.c >> +++ b/xen/arch/x86/irq.c >> @@ -1115,6 +1115,9 @@ static void irq_guest_eoi_timer_fn(void >> >> action = (irq_guest_action_t *)desc->action; >> > > /* Another instance of this timer already running? Skip everything to > avoid forcing an EOI early. */ Fine with me, added. Jan
On 06/06/2019 09:08, Jan Beulich wrote: >>>> On 05.06.19 at 19:04, <andrew.cooper3@citrix.com> wrote: >> On 08/05/2019 13:46, Jan Beulich wrote: >>> The timer needs to remain active only until all pending IRQ instances >>> have seen EOIs from their respective domains. Stop it when the in-flight >>> count has reached zero in desc_guest_eoi(). Note that this is race free >>> (with __do_IRQ_guest()), as the IRQ descriptor lock is being held at >>> that point. >>> >>> Also pull up stopping of the timer in __do_IRQ_guest() itself: Instead >>> of stopping it immediately before re-setting, stop it as soon as we've >>> made it past any early returns from the function (and hence we're sure >>> it'll get set again). >> Why this this a good thing? > For it to not fire when it doesn't need to. If we're about to set > a new timeout, we clearly don't want the previous one to have > any effect anymore. Sounds like an excellent addition to the code, now that there is a order-of-returns dependency. With a suitable comment, Reviewed-by: Andrew Cooper <andrew.cooper3@citrix.com>
--- a/xen/arch/x86/irq.c +++ b/xen/arch/x86/irq.c @@ -1115,6 +1115,9 @@ static void irq_guest_eoi_timer_fn(void action = (irq_guest_action_t *)desc->action; + if ( timer_is_active(&action->eoi_timer) ) + goto out; + if ( action->ack_type != ACKTYPE_NONE ) { unsigned int i; @@ -1167,6 +1170,9 @@ static void __do_IRQ_guest(int irq) return; } + if ( action->ack_type != ACKTYPE_NONE ) + stop_timer(&action->eoi_timer); + if ( action->ack_type == ACKTYPE_EOI ) { sp = pending_eoi_sp(peoi); @@ -1194,7 +1200,6 @@ static void __do_IRQ_guest(int irq) if ( action->ack_type != ACKTYPE_NONE ) { - stop_timer(&action->eoi_timer); migrate_timer(&action->eoi_timer, smp_processor_id()); set_timer(&action->eoi_timer, NOW() + MILLISECS(1)); } @@ -1457,6 +1462,8 @@ void desc_guest_eoi(struct irq_desc *des return; } + stop_timer(&action->eoi_timer); + if ( action->ack_type == ACKTYPE_UNMASK ) { ASSERT(cpumask_empty(action->cpu_eoi_map)); --- a/xen/common/timer.c +++ b/xen/common/timer.c @@ -282,11 +282,10 @@ static inline void timer_unlock(struct t }) -static bool_t active_timer(struct timer *timer) +static bool active_timer(const struct timer *timer) { ASSERT(timer->status >= TIMER_STATUS_inactive); - ASSERT(timer->status <= TIMER_STATUS_in_list); - return (timer->status >= TIMER_STATUS_in_heap); + return timer_is_active(timer); } --- a/xen/include/xen/timer.h +++ b/xen/include/xen/timer.h @@ -75,6 +75,19 @@ bool timer_expires_before(struct timer * #define timer_is_expired(t) timer_expires_before(t, NOW()) +/* + * True if a timer is active. + * + * Unlike for timer_expires_before(), it is the caller's responsibility to + * use suitable locking such that the returned value isn't stale by the time + * it gets acted upon. + */ +static inline bool timer_is_active(const struct timer *timer) +{ + ASSERT(timer->status <= TIMER_STATUS_in_list); + return timer->status >= TIMER_STATUS_in_heap; +} + /* Migrate a timer to a different CPU. The timer may be currently active. */ void migrate_timer(struct timer *timer, unsigned int new_cpu);
The timer needs to remain active only until all pending IRQ instances have seen EOIs from their respective domains. Stop it when the in-flight count has reached zero in desc_guest_eoi(). Note that this is race free (with __do_IRQ_guest()), as the IRQ descriptor lock is being held at that point. Also pull up stopping of the timer in __do_IRQ_guest() itself: Instead of stopping it immediately before re-setting, stop it as soon as we've made it past any early returns from the function (and hence we're sure it'll get set again). Finally bail from the actual timer handler in case we find the timer already active again by the time we've managed to acquire the IRQ descriptor lock. Without this we may forcibly EOI an IRQ immediately after it got sent to a guest. For this, timer_is_active() gets split out of active_timer(), deliberately moving just one of the two ASSERT()s (to allow the function to be used also on a never initialized timer). Signed-off-by: Jan Beulich <jbeulich@suse.com>