| Message ID | 5CD2CFBB020000780022CCC6@prv1-mh.provo.novell.com (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
| Series | x86: EOI timer corrections / improvements | expand |
On Wed, May 08, 2019 at 06:46:51AM -0600, Jan Beulich wrote: > There's no point entering the loop in the function in this case. Instead > there still being something in flight _after_ the loop would be an > actual problem: No timer would be running anymore for issuing the EOI > eventually, and hence this IRQ (and possibly lower priority ones) would > be blocked, perhaps indefinitely. > > Issue a warning instead and prefer breaking some (presumably > misbehaving) guest over stalling perhaps the entire system. > > Signed-off-by: Jan Beulich <jbeulich@suse.com> > > --- a/xen/arch/x86/irq.c > +++ b/xen/arch/x86/irq.c > @@ -1115,7 +1115,7 @@ static void irq_guest_eoi_timer_fn(void > > action = (irq_guest_action_t *)desc->action; > > - if ( timer_is_active(&action->eoi_timer) ) > + if ( !action->in_flight || timer_is_active(&action->eoi_timer) ) > goto out; > > if ( action->ack_type != ACKTYPE_NONE ) > @@ -1130,8 +1130,10 @@ static void irq_guest_eoi_timer_fn(void > } > } > > - if ( action->in_flight != 0 ) > - goto out; > + if ( action->in_flight ) > + printk(XENLOG_G_WARNING > + "IRQ%d: %d handlers still in flight at forced EOI\n", > + desc->irq, action->in_flight); AFAICT action->in_flight should contain the number of guests pirqs that have the pirq masked (pirq->masked == true), because in_flight is only increased by __do_IRQ_guest when the pirq is not already masked. At guest EOI (desc_guest_eoi) the in_flight count is also only decreased if the pirq is unmasked. Hence I think this condition could be turned into an ASSERT, but I'm likely missing something. Thanks, Roger.
>>> On 16.05.19 at 13:37, <roger.pau@citrix.com> wrote: > On Wed, May 08, 2019 at 06:46:51AM -0600, Jan Beulich wrote: >> There's no point entering the loop in the function in this case. Instead >> there still being something in flight _after_ the loop would be an >> actual problem: No timer would be running anymore for issuing the EOI >> eventually, and hence this IRQ (and possibly lower priority ones) would >> be blocked, perhaps indefinitely. >> >> Issue a warning instead and prefer breaking some (presumably >> misbehaving) guest over stalling perhaps the entire system. >> >> Signed-off-by: Jan Beulich <jbeulich@suse.com> >> >> --- a/xen/arch/x86/irq.c >> +++ b/xen/arch/x86/irq.c >> @@ -1115,7 +1115,7 @@ static void irq_guest_eoi_timer_fn(void >> >> action = (irq_guest_action_t *)desc->action; >> >> - if ( timer_is_active(&action->eoi_timer) ) >> + if ( !action->in_flight || timer_is_active(&action->eoi_timer) ) >> goto out; >> >> if ( action->ack_type != ACKTYPE_NONE ) >> @@ -1130,8 +1130,10 @@ static void irq_guest_eoi_timer_fn(void >> } >> } >> >> - if ( action->in_flight != 0 ) >> - goto out; >> + if ( action->in_flight ) >> + printk(XENLOG_G_WARNING >> + "IRQ%d: %d handlers still in flight at forced EOI\n", >> + desc->irq, action->in_flight); > > AFAICT action->in_flight should contain the number of guests pirqs > that have the pirq masked (pirq->masked == true), because in_flight is > only increased by __do_IRQ_guest when the pirq is not already masked. > At guest EOI (desc_guest_eoi) the in_flight count is also only > decreased if the pirq is unmasked. > > Hence I think this condition could be turned into an ASSERT, but I'm > likely missing something. I don't think you are. Going from if() straight to ASSERT() simply seemed too harsh to me, the more in a subsystem where I could easily have overlooked some corner case, due to how convoluted some of the implementation is. Jan
On Thu, May 16, 2019 at 06:02:15AM -0600, Jan Beulich wrote: > >>> On 16.05.19 at 13:37, <roger.pau@citrix.com> wrote: > > On Wed, May 08, 2019 at 06:46:51AM -0600, Jan Beulich wrote: > >> There's no point entering the loop in the function in this case. Instead > >> there still being something in flight _after_ the loop would be an > >> actual problem: No timer would be running anymore for issuing the EOI > >> eventually, and hence this IRQ (and possibly lower priority ones) would > >> be blocked, perhaps indefinitely. > >> > >> Issue a warning instead and prefer breaking some (presumably > >> misbehaving) guest over stalling perhaps the entire system. > >> > >> Signed-off-by: Jan Beulich <jbeulich@suse.com> > >> > >> --- a/xen/arch/x86/irq.c > >> +++ b/xen/arch/x86/irq.c > >> @@ -1115,7 +1115,7 @@ static void irq_guest_eoi_timer_fn(void > >> > >> action = (irq_guest_action_t *)desc->action; > >> > >> - if ( timer_is_active(&action->eoi_timer) ) > >> + if ( !action->in_flight || timer_is_active(&action->eoi_timer) ) > >> goto out; > >> > >> if ( action->ack_type != ACKTYPE_NONE ) > >> @@ -1130,8 +1130,10 @@ static void irq_guest_eoi_timer_fn(void > >> } > >> } > >> > >> - if ( action->in_flight != 0 ) > >> - goto out; > >> + if ( action->in_flight ) > >> + printk(XENLOG_G_WARNING > >> + "IRQ%d: %d handlers still in flight at forced EOI\n", > >> + desc->irq, action->in_flight); > > > > AFAICT action->in_flight should contain the number of guests pirqs > > that have the pirq masked (pirq->masked == true), because in_flight is > > only increased by __do_IRQ_guest when the pirq is not already masked. > > At guest EOI (desc_guest_eoi) the in_flight count is also only > > decreased if the pirq is unmasked. > > > > Hence I think this condition could be turned into an ASSERT, but I'm > > likely missing something. > > I don't think you are. Going from if() straight to ASSERT() simply > seemed too harsh to me, the more in a subsystem where I could > easily have overlooked some corner case, due to how convoluted > some of the implementation is. I agree it's quite convoluted. I think it would be helpful to add an ASSERT_UNREACHABLE together with the warning message. With that: Reviewed-by: Roger Pau Monné <roger.pau@citrix.com> Thanks, Roger.
On 08/05/2019 13:46, Jan Beulich wrote: > There's no point entering the loop in the function in this case. Instead > there still being something in flight _after_ the loop would be an > actual problem: No timer would be running anymore for issuing the EOI > eventually, and hence this IRQ (and possibly lower priority ones) would > be blocked, perhaps indefinitely. > > Issue a warning instead and prefer breaking some (presumably > misbehaving) guest over stalling perhaps the entire system. > > Signed-off-by: Jan Beulich <jbeulich@suse.com> > > --- a/xen/arch/x86/irq.c > +++ b/xen/arch/x86/irq.c > @@ -1115,7 +1115,7 @@ static void irq_guest_eoi_timer_fn(void > > action = (irq_guest_action_t *)desc->action; > > - if ( timer_is_active(&action->eoi_timer) ) > + if ( !action->in_flight || timer_is_active(&action->eoi_timer) ) > goto out; > > if ( action->ack_type != ACKTYPE_NONE ) > @@ -1130,8 +1130,10 @@ static void irq_guest_eoi_timer_fn(void > } > } > > - if ( action->in_flight != 0 ) > - goto out; > + if ( action->in_flight ) > + printk(XENLOG_G_WARNING > + "IRQ%d: %d handlers still in flight at forced EOI\n", > + desc->irq, action->in_flight); AFACIT, this condition can be triggered by a buggy/malicious guest, by it simply ignoring or masking the line interrupt at the vIO-APIC. The message would be far more useful if it identified the domain in question, which looks like it can be obtained from the middle of the loop. ~Andrew
>>> On 05.06.19 at 19:15, <andrew.cooper3@citrix.com> wrote: > On 08/05/2019 13:46, Jan Beulich wrote: >> @@ -1130,8 +1130,10 @@ static void irq_guest_eoi_timer_fn(void >> } >> } >> >> - if ( action->in_flight != 0 ) >> - goto out; >> + if ( action->in_flight ) >> + printk(XENLOG_G_WARNING >> + "IRQ%d: %d handlers still in flight at forced EOI\n", >> + desc->irq, action->in_flight); > > AFACIT, this condition can be triggered by a buggy/malicious guest, by > it simply ignoring or masking the line interrupt at the vIO-APIC. I don't think it can, no. Or else the ASSERT_UNREACHABLE() below here would be invalid to add. > The message would be far more useful if it identified the domain in > question, which looks like it can be obtained from the middle of the loop. That very loop has just taken care of decrementing ->in_flight for all such guests. Also note that there could be more than one offending domain, for shared IRQs. Plus the loop you're referring to can specifically _not_ be used for identifying the domain(s), because for the ones processed there we _did_ decrement ->in_flight. If this message gets logged, we simply have no idea why ->in_flight is _still_ non- zero. This could be a BUG_ON(), but it seems more in line with our general idea of how we would like to deal with such cases to try and keep the system running here in release builds. Jan
On 06/06/2019 09:17, Jan Beulich wrote: >>>> On 05.06.19 at 19:15, <andrew.cooper3@citrix.com> wrote: >> On 08/05/2019 13:46, Jan Beulich wrote: >>> @@ -1130,8 +1130,10 @@ static void irq_guest_eoi_timer_fn(void >>> } >>> } >>> >>> - if ( action->in_flight != 0 ) >>> - goto out; >>> + if ( action->in_flight ) >>> + printk(XENLOG_G_WARNING >>> + "IRQ%d: %d handlers still in flight at forced EOI\n", >>> + desc->irq, action->in_flight); >> AFACIT, this condition can be triggered by a buggy/malicious guest, by >> it simply ignoring or masking the line interrupt at the vIO-APIC. > I don't think it can, no. Or else the ASSERT_UNREACHABLE() below > here would be invalid to add. Which ASSERT_UNREACHABLE() ? I know Roger asked for one, but I don't see it anywhere in the code. > >> The message would be far more useful if it identified the domain in >> question, which looks like it can be obtained from the middle of the loop. > That very loop has just taken care of decrementing ->in_flight for > all such guests. > > Also note that there could be more than one offending domain, for > shared IRQs. Plus the loop you're referring to can specifically _not_ > be used for identifying the domain(s), because for the ones > processed there we _did_ decrement ->in_flight. If this message > gets logged, we simply have no idea why ->in_flight is _still_ non- > zero. This could be a BUG_ON(), but it seems more in line with our > general idea of how we would like to deal with such cases to try > and keep the system running here in release builds. Ok - lets go with this for now. It is a net improvement, and we can evaluate the guest-triggerability at a later point. Reviewed-by: Andrew Cooper <andrew.cooper3@citrix.com>
>>> On 06.06.19 at 13:34, <andrew.cooper3@citrix.com> wrote: > On 06/06/2019 09:17, Jan Beulich wrote: >>>>> On 05.06.19 at 19:15, <andrew.cooper3@citrix.com> wrote: >>> On 08/05/2019 13:46, Jan Beulich wrote: >>>> @@ -1130,8 +1130,10 @@ static void irq_guest_eoi_timer_fn(void >>>> } >>>> } >>>> >>>> - if ( action->in_flight != 0 ) >>>> - goto out; >>>> + if ( action->in_flight ) >>>> + printk(XENLOG_G_WARNING >>>> + "IRQ%d: %d handlers still in flight at forced EOI\n", >>>> + desc->irq, action->in_flight); >>> AFACIT, this condition can be triggered by a buggy/malicious guest, by >>> it simply ignoring or masking the line interrupt at the vIO-APIC. >> I don't think it can, no. Or else the ASSERT_UNREACHABLE() below >> here would be invalid to add. > > Which ASSERT_UNREACHABLE() ? I know Roger asked for one, but I don't > see it anywhere in the code. Because so far there was no real reason to re-post. It's right here, as Roger did ask for, and as I did (hesitantly) agree: if ( action->in_flight ) { printk(XENLOG_G_WARNING "IRQ%u: %d/%d handler(s) still in flight at forced EOI\n", irq, action->in_flight, action->nr_guests); ASSERT_UNREACHABLE(); } >>> The message would be far more useful if it identified the domain in >>> question, which looks like it can be obtained from the middle of the loop. >> That very loop has just taken care of decrementing ->in_flight for >> all such guests. >> >> Also note that there could be more than one offending domain, for >> shared IRQs. Plus the loop you're referring to can specifically _not_ >> be used for identifying the domain(s), because for the ones >> processed there we _did_ decrement ->in_flight. If this message >> gets logged, we simply have no idea why ->in_flight is _still_ non- >> zero. This could be a BUG_ON(), but it seems more in line with our >> general idea of how we would like to deal with such cases to try >> and keep the system running here in release builds. > > Ok - lets go with this for now. It is a net improvement, and we can > evaluate the guest-triggerability at a later point. > > Reviewed-by: Andrew Cooper <andrew.cooper3@citrix.com> Thanks much. I'll assume this holds also for the adjustments requested by Roger. Jan
On 06/06/2019 12:43, Jan Beulich wrote: >>>> On 06.06.19 at 13:34, <andrew.cooper3@citrix.com> wrote: >> On 06/06/2019 09:17, Jan Beulich wrote: >>>>>> On 05.06.19 at 19:15, <andrew.cooper3@citrix.com> wrote: >>>> On 08/05/2019 13:46, Jan Beulich wrote: >>>>> @@ -1130,8 +1130,10 @@ static void irq_guest_eoi_timer_fn(void >>>>> } >>>>> } >>>>> >>>>> - if ( action->in_flight != 0 ) >>>>> - goto out; >>>>> + if ( action->in_flight ) >>>>> + printk(XENLOG_G_WARNING >>>>> + "IRQ%d: %d handlers still in flight at forced EOI\n", >>>>> + desc->irq, action->in_flight); >>>> AFACIT, this condition can be triggered by a buggy/malicious guest, by >>>> it simply ignoring or masking the line interrupt at the vIO-APIC. >>> I don't think it can, no. Or else the ASSERT_UNREACHABLE() below >>> here would be invalid to add. >> Which ASSERT_UNREACHABLE() ? I know Roger asked for one, but I don't >> see it anywhere in the code. > Because so far there was no real reason to re-post. It's right here, > as Roger did ask for, and as I did (hesitantly) agree: > > if ( action->in_flight ) > { > printk(XENLOG_G_WARNING > "IRQ%u: %d/%d handler(s) still in flight at forced EOI\n", > irq, action->in_flight, action->nr_guests); > ASSERT_UNREACHABLE(); > } > >>>> The message would be far more useful if it identified the domain in >>>> question, which looks like it can be obtained from the middle of the loop. >>> That very loop has just taken care of decrementing ->in_flight for >>> all such guests. >>> >>> Also note that there could be more than one offending domain, for >>> shared IRQs. Plus the loop you're referring to can specifically _not_ >>> be used for identifying the domain(s), because for the ones >>> processed there we _did_ decrement ->in_flight. If this message >>> gets logged, we simply have no idea why ->in_flight is _still_ non- >>> zero. This could be a BUG_ON(), but it seems more in line with our >>> general idea of how we would like to deal with such cases to try >>> and keep the system running here in release builds. >> Ok - lets go with this for now. It is a net improvement, and we can >> evaluate the guest-triggerability at a later point. >> >> Reviewed-by: Andrew Cooper <andrew.cooper3@citrix.com> > Thanks much. I'll assume this holds also for the adjustments > requested by Roger. Fine. At least that should make things obvious in a debug build. ~Andrew
--- a/xen/arch/x86/irq.c +++ b/xen/arch/x86/irq.c @@ -1115,7 +1115,7 @@ static void irq_guest_eoi_timer_fn(void action = (irq_guest_action_t *)desc->action; - if ( timer_is_active(&action->eoi_timer) ) + if ( !action->in_flight || timer_is_active(&action->eoi_timer) ) goto out; if ( action->ack_type != ACKTYPE_NONE ) @@ -1130,8 +1130,10 @@ static void irq_guest_eoi_timer_fn(void } } - if ( action->in_flight != 0 ) - goto out; + if ( action->in_flight ) + printk(XENLOG_G_WARNING + "IRQ%d: %d handlers still in flight at forced EOI\n", + desc->irq, action->in_flight); switch ( action->ack_type ) {
There's no point entering the loop in the function in this case. Instead there still being something in flight _after_ the loop would be an actual problem: No timer would be running anymore for issuing the EOI eventually, and hence this IRQ (and possibly lower priority ones) would be blocked, perhaps indefinitely. Issue a warning instead and prefer breaking some (presumably misbehaving) guest over stalling perhaps the entire system. Signed-off-by: Jan Beulich <jbeulich@suse.com>