| Message ID | 1513148407-2611-1-git-send-email-hejianet@gmail.com (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
Hi Jia, On 13/12/17 07:00, Jia He wrote: > In our Armv8a server (qualcomm Amberwing, non VHE), after applying > Christoffer's timer optimizing patchset(Optimize arch timer register > handling), the guest is hang during kernel booting. > > The error root cause might be as follows: > 1. in kvm_arch_timer_handler, it reset vtimer->cnt_ctl with current > cntv_ctl register value. And then it missed some cases to update timer's > irq (irq.level) when kvm_timer_irq_can_fire() is false > 2. It causes kvm_vcpu_check_block return 0 instead of -EINTR > kvm_vcpu_check_block > kvm_cpu_has_pending_timer > kvm_timer_is_pending > kvm_timer_should_fire > 3. Thus, the kvm hyp code can not break the loop in kvm_vcpu_block (halt > poll process) and the guest is hang forever > > Fixes: b103cc3f10c0 ("KVM: arm/arm64: Avoid timer save/restore in vcpu entry/exit") > Signed-off-by: Jia He <jia.he@hxt-semitech.com> > --- > virt/kvm/arm/arch_timer.c | 1 - > 1 file changed, 1 deletion(-) > > diff --git a/virt/kvm/arm/arch_timer.c b/virt/kvm/arm/arch_timer.c > index f9555b1..bb86433 100644 > --- a/virt/kvm/arm/arch_timer.c > +++ b/virt/kvm/arm/arch_timer.c > @@ -100,7 +100,6 @@ static irqreturn_t kvm_arch_timer_handler(int irq, void *dev_id) > vtimer = vcpu_vtimer(vcpu); > > if (!vtimer->irq.level) { > - vtimer->cnt_ctl = read_sysreg_el0(cntv_ctl); > if (kvm_timer_irq_can_fire(vtimer)) > kvm_timer_update_irq(vcpu, true, vtimer); > } > Which patches are you looking at? The current code in mainline looks like this: vtimer = vcpu_vtimer(vcpu); vtimer->cnt_ctl = read_sysreg_el0(cntv_ctl); if (kvm_timer_irq_can_fire(vtimer)) kvm_timer_update_irq(vcpu, true, vtimer); I'd suggest you use mainline and report if this doesn't work. Thanks, M.
Marc, On 13/12/17 09:56, Marc Zyngier wrote: > Hi Jia, > > On 13/12/17 07:00, Jia He wrote: >> In our Armv8a server (qualcomm Amberwing, non VHE), after applying >> Christoffer's timer optimizing patchset(Optimize arch timer register >> handling), the guest is hang during kernel booting. >> >> The error root cause might be as follows: >> 1. in kvm_arch_timer_handler, it reset vtimer->cnt_ctl with current >> cntv_ctl register value. And then it missed some cases to update timer's >> irq (irq.level) when kvm_timer_irq_can_fire() is false >> 2. It causes kvm_vcpu_check_block return 0 instead of -EINTR >> kvm_vcpu_check_block >> kvm_cpu_has_pending_timer >> kvm_timer_is_pending >> kvm_timer_should_fire >> 3. Thus, the kvm hyp code can not break the loop in kvm_vcpu_block (halt >> poll process) and the guest is hang forever >> >> Fixes: b103cc3f10c0 ("KVM: arm/arm64: Avoid timer save/restore in vcpu entry/exit") >> Signed-off-by: Jia He <jia.he@hxt-semitech.com> >> --- >> virt/kvm/arm/arch_timer.c | 1 - >> 1 file changed, 1 deletion(-) >> >> diff --git a/virt/kvm/arm/arch_timer.c b/virt/kvm/arm/arch_timer.c >> index f9555b1..bb86433 100644 >> --- a/virt/kvm/arm/arch_timer.c >> +++ b/virt/kvm/arm/arch_timer.c >> @@ -100,7 +100,6 @@ static irqreturn_t kvm_arch_timer_handler(int irq, void *dev_id) >> vtimer = vcpu_vtimer(vcpu); >> >> if (!vtimer->irq.level) { >> - vtimer->cnt_ctl = read_sysreg_el0(cntv_ctl); >> if (kvm_timer_irq_can_fire(vtimer)) >> kvm_timer_update_irq(vcpu, true, vtimer); >> } >> > > Which patches are you looking at? The current code in mainline looks > like this: > > vtimer = vcpu_vtimer(vcpu); > > vtimer->cnt_ctl = read_sysreg_el0(cntv_ctl); > if (kvm_timer_irq_can_fire(vtimer)) > kvm_timer_update_irq(vcpu, true, vtimer); > > I'd suggest you use mainline and report if this doesn't work the removal of if (!vtimer->irq.level) test happened in: [PATCH v7 3/8] KVM: arm/arm64: Don't cache the timer IRQ level which is not upstream. Thanks Eric > > Thanks, > > M. >
On Tue, Dec 12, 2017 at 11:00:07PM -0800, Jia He wrote: > In our Armv8a server (qualcomm Amberwing, non VHE), after applying > Christoffer's timer optimizing patchset(Optimize arch timer register > handling), the guest is hang during kernel booting. > > The error root cause might be as follows: > 1. in kvm_arch_timer_handler, it reset vtimer->cnt_ctl with current > cntv_ctl register value. And then it missed some cases to update timer's > irq (irq.level) when kvm_timer_irq_can_fire() is false Why should it set the irq level to true when the timer cannot fire? > 2. It causes kvm_vcpu_check_block return 0 instead of -EINTR > kvm_vcpu_check_block > kvm_cpu_has_pending_timer > kvm_timer_is_pending > kvm_timer_should_fire > 3. Thus, the kvm hyp code can not break the loop in kvm_vcpu_block (halt > poll process) and the guest is hang forever This is just a polling loop which will expire after some time, so it shouldn't halt the guest indefinitely, but merely slow it down for some while, if we have a bug. Is that the behavior you're seeing or are you seeing the guest coming to a complete halt? > > Fixes: b103cc3f10c0 ("KVM: arm/arm64: Avoid timer save/restore in vcpu entry/exit") > Signed-off-by: Jia He <jia.he@hxt-semitech.com> > --- > virt/kvm/arm/arch_timer.c | 1 - > 1 file changed, 1 deletion(-) > > diff --git a/virt/kvm/arm/arch_timer.c b/virt/kvm/arm/arch_timer.c > index f9555b1..bb86433 100644 > --- a/virt/kvm/arm/arch_timer.c > +++ b/virt/kvm/arm/arch_timer.c > @@ -100,7 +100,6 @@ static irqreturn_t kvm_arch_timer_handler(int irq, void *dev_id) > vtimer = vcpu_vtimer(vcpu); > > if (!vtimer->irq.level) { > - vtimer->cnt_ctl = read_sysreg_el0(cntv_ctl); This fix is clearly not correct, as it would prevent forwarding timer interrupts in some cases. > if (kvm_timer_irq_can_fire(vtimer)) > kvm_timer_update_irq(vcpu, true, vtimer); > } > -- > 2.7.4 > I actually don't see how the above scenario you painted can happen. If you're in the polling loop, that means that the timer state is loaded on the vcpu, and that means you can take interrupts from the timer, and when you take interrupts, you will set the irq.level. And here's the first bit of logic in kvm_timer_is_pending(): if (vtimer->irq.level || ptimer->irq.level) return true; So that would break the loop. I'm not able to reproduce on my side with a non-VHE platform. What is the workload you're running to reproduce this, and what is the exact kernel tree and kernel configuration you're using? Thanks, -Christoffer
On Wed, Dec 13, 2017 at 08:56:12AM +0000, Marc Zyngier wrote: > Hi Jia, > > On 13/12/17 07:00, Jia He wrote: > > In our Armv8a server (qualcomm Amberwing, non VHE), after applying > > Christoffer's timer optimizing patchset(Optimize arch timer register > > handling), the guest is hang during kernel booting. > > > > The error root cause might be as follows: > > 1. in kvm_arch_timer_handler, it reset vtimer->cnt_ctl with current > > cntv_ctl register value. And then it missed some cases to update timer's > > irq (irq.level) when kvm_timer_irq_can_fire() is false > > 2. It causes kvm_vcpu_check_block return 0 instead of -EINTR > > kvm_vcpu_check_block > > kvm_cpu_has_pending_timer > > kvm_timer_is_pending > > kvm_timer_should_fire > > 3. Thus, the kvm hyp code can not break the loop in kvm_vcpu_block (halt > > poll process) and the guest is hang forever > > > > Fixes: b103cc3f10c0 ("KVM: arm/arm64: Avoid timer save/restore in vcpu entry/exit") > > Signed-off-by: Jia He <jia.he@hxt-semitech.com> > > --- > > virt/kvm/arm/arch_timer.c | 1 - > > 1 file changed, 1 deletion(-) > > > > diff --git a/virt/kvm/arm/arch_timer.c b/virt/kvm/arm/arch_timer.c > > index f9555b1..bb86433 100644 > > --- a/virt/kvm/arm/arch_timer.c > > +++ b/virt/kvm/arm/arch_timer.c > > @@ -100,7 +100,6 @@ static irqreturn_t kvm_arch_timer_handler(int irq, void *dev_id) > > vtimer = vcpu_vtimer(vcpu); > > > > if (!vtimer->irq.level) { > > - vtimer->cnt_ctl = read_sysreg_el0(cntv_ctl); > > if (kvm_timer_irq_can_fire(vtimer)) > > kvm_timer_update_irq(vcpu, true, vtimer); > > } > > > > Which patches are you looking at? The current code in mainline looks > like this: > > vtimer = vcpu_vtimer(vcpu); > > vtimer->cnt_ctl = read_sysreg_el0(cntv_ctl); > if (kvm_timer_irq_can_fire(vtimer)) > kvm_timer_update_irq(vcpu, true, vtimer); > > I'd suggest you use mainline and report if this doesn't work. > That looks like you have the level-triggered mapped series applied? That would be an interesting data point to get from Jia as well though. Jia, can you try applying this series and see if it helps? git://git.kernel.org/pub/scm/linux/kernel/git/cdall/linux.git level-mapped-v7 Thanks, -Christoffer > Thanks, > > M. > -- > Jazz is not dead. It just smells funny...
On 13/12/17 09:08, Auger Eric wrote: > Marc, > On 13/12/17 09:56, Marc Zyngier wrote: >> Hi Jia, >> >> On 13/12/17 07:00, Jia He wrote: >>> In our Armv8a server (qualcomm Amberwing, non VHE), after applying >>> Christoffer's timer optimizing patchset(Optimize arch timer register >>> handling), the guest is hang during kernel booting. >>> >>> The error root cause might be as follows: >>> 1. in kvm_arch_timer_handler, it reset vtimer->cnt_ctl with current >>> cntv_ctl register value. And then it missed some cases to update timer's >>> irq (irq.level) when kvm_timer_irq_can_fire() is false >>> 2. It causes kvm_vcpu_check_block return 0 instead of -EINTR >>> kvm_vcpu_check_block >>> kvm_cpu_has_pending_timer >>> kvm_timer_is_pending >>> kvm_timer_should_fire >>> 3. Thus, the kvm hyp code can not break the loop in kvm_vcpu_block (halt >>> poll process) and the guest is hang forever >>> >>> Fixes: b103cc3f10c0 ("KVM: arm/arm64: Avoid timer save/restore in vcpu entry/exit") >>> Signed-off-by: Jia He <jia.he@hxt-semitech.com> >>> --- >>> virt/kvm/arm/arch_timer.c | 1 - >>> 1 file changed, 1 deletion(-) >>> >>> diff --git a/virt/kvm/arm/arch_timer.c b/virt/kvm/arm/arch_timer.c >>> index f9555b1..bb86433 100644 >>> --- a/virt/kvm/arm/arch_timer.c >>> +++ b/virt/kvm/arm/arch_timer.c >>> @@ -100,7 +100,6 @@ static irqreturn_t kvm_arch_timer_handler(int irq, void *dev_id) >>> vtimer = vcpu_vtimer(vcpu); >>> >>> if (!vtimer->irq.level) { >>> - vtimer->cnt_ctl = read_sysreg_el0(cntv_ctl); >>> if (kvm_timer_irq_can_fire(vtimer)) >>> kvm_timer_update_irq(vcpu, true, vtimer); >>> } >>> >> >> Which patches are you looking at? The current code in mainline looks >> like this: >> >> vtimer = vcpu_vtimer(vcpu); >> >> vtimer->cnt_ctl = read_sysreg_el0(cntv_ctl); >> if (kvm_timer_irq_can_fire(vtimer)) >> kvm_timer_update_irq(vcpu, true, vtimer); >> >> I'd suggest you use mainline and report if this doesn't work > the removal of if (!vtimer->irq.level) test happened in: > [PATCH v7 3/8] KVM: arm/arm64: Don't cache the timer IRQ level > > which is not upstream. Ah, my bad (I have that series in my working tree already...). I still think Jia's approach to this is not quite right. If you don't update the status of the timer by reading the HW value, how can you decide whether the timer can fire or not? Thanks, M.
On Wed, Dec 13, 2017 at 10:27 AM, Marc Zyngier <marc.zyngier@arm.com> wrote: > On 13/12/17 09:08, Auger Eric wrote: >> Marc, >> On 13/12/17 09:56, Marc Zyngier wrote: >>> Hi Jia, >>> >>> On 13/12/17 07:00, Jia He wrote: >>>> In our Armv8a server (qualcomm Amberwing, non VHE), after applying >>>> Christoffer's timer optimizing patchset(Optimize arch timer register >>>> handling), the guest is hang during kernel booting. >>>> >>>> The error root cause might be as follows: >>>> 1. in kvm_arch_timer_handler, it reset vtimer->cnt_ctl with current >>>> cntv_ctl register value. And then it missed some cases to update timer's >>>> irq (irq.level) when kvm_timer_irq_can_fire() is false >>>> 2. It causes kvm_vcpu_check_block return 0 instead of -EINTR >>>> kvm_vcpu_check_block >>>> kvm_cpu_has_pending_timer >>>> kvm_timer_is_pending >>>> kvm_timer_should_fire >>>> 3. Thus, the kvm hyp code can not break the loop in kvm_vcpu_block (halt >>>> poll process) and the guest is hang forever >>>> >>>> Fixes: b103cc3f10c0 ("KVM: arm/arm64: Avoid timer save/restore in vcpu entry/exit") >>>> Signed-off-by: Jia He <jia.he@hxt-semitech.com> >>>> --- >>>> virt/kvm/arm/arch_timer.c | 1 - >>>> 1 file changed, 1 deletion(-) >>>> >>>> diff --git a/virt/kvm/arm/arch_timer.c b/virt/kvm/arm/arch_timer.c >>>> index f9555b1..bb86433 100644 >>>> --- a/virt/kvm/arm/arch_timer.c >>>> +++ b/virt/kvm/arm/arch_timer.c >>>> @@ -100,7 +100,6 @@ static irqreturn_t kvm_arch_timer_handler(int irq, void *dev_id) >>>> vtimer = vcpu_vtimer(vcpu); >>>> >>>> if (!vtimer->irq.level) { >>>> - vtimer->cnt_ctl = read_sysreg_el0(cntv_ctl); >>>> if (kvm_timer_irq_can_fire(vtimer)) >>>> kvm_timer_update_irq(vcpu, true, vtimer); >>>> } >>>> >>> >>> Which patches are you looking at? The current code in mainline looks >>> like this: >>> >>> vtimer = vcpu_vtimer(vcpu); >>> >>> vtimer->cnt_ctl = read_sysreg_el0(cntv_ctl); >>> if (kvm_timer_irq_can_fire(vtimer)) >>> kvm_timer_update_irq(vcpu, true, vtimer); >>> >>> I'd suggest you use mainline and report if this doesn't work >> the removal of if (!vtimer->irq.level) test happened in: >> [PATCH v7 3/8] KVM: arm/arm64: Don't cache the timer IRQ level >> >> which is not upstream. > Ah, my bad (I have that series in my working tree already...). > > I still think Jia's approach to this is not quite right. If you don't > update the status of the timer by reading the HW value, how can you > decide whether the timer can fire or not? > Exactly. We need to know the exact kernel source, symptoms, how to reproduce, and then trace what's going on. It may be needed to tweak kvm_timer_is_pending(), but I don't yet see a case where it breaks. Thanks, -Christoffer
diff --git a/virt/kvm/arm/arch_timer.c b/virt/kvm/arm/arch_timer.c index f9555b1..bb86433 100644 --- a/virt/kvm/arm/arch_timer.c +++ b/virt/kvm/arm/arch_timer.c @@ -100,7 +100,6 @@ static irqreturn_t kvm_arch_timer_handler(int irq, void *dev_id) vtimer = vcpu_vtimer(vcpu); if (!vtimer->irq.level) { - vtimer->cnt_ctl = read_sysreg_el0(cntv_ctl); if (kvm_timer_irq_can_fire(vtimer)) kvm_timer_update_irq(vcpu, true, vtimer); }
In our Armv8a server (qualcomm Amberwing, non VHE), after applying Christoffer's timer optimizing patchset(Optimize arch timer register handling), the guest is hang during kernel booting. The error root cause might be as follows: 1. in kvm_arch_timer_handler, it reset vtimer->cnt_ctl with current cntv_ctl register value. And then it missed some cases to update timer's irq (irq.level) when kvm_timer_irq_can_fire() is false 2. It causes kvm_vcpu_check_block return 0 instead of -EINTR kvm_vcpu_check_block kvm_cpu_has_pending_timer kvm_timer_is_pending kvm_timer_should_fire 3. Thus, the kvm hyp code can not break the loop in kvm_vcpu_block (halt poll process) and the guest is hang forever Fixes: b103cc3f10c0 ("KVM: arm/arm64: Avoid timer save/restore in vcpu entry/exit") Signed-off-by: Jia He <jia.he@hxt-semitech.com> --- virt/kvm/arm/arch_timer.c | 1 - 1 file changed, 1 deletion(-)