| Message ID | 1472615791-8664-7-git-send-email-feng.wu@intel.com (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
>>> On 31.08.16 at 05:56, <feng.wu@intel.com> wrote: > +void vmx_pi_desc_fixup(int cpu) unsigned int > +{ > + unsigned int new_cpu, dest; > + unsigned long flags; > + struct arch_vmx_struct *vmx, *tmp; > + spinlock_t *new_lock, *old_lock = &per_cpu(vmx_pi_blocking, cpu).lock; > + struct list_head *blocked_vcpus = &per_cpu(vmx_pi_blocking, cpu).list; > + > + if ( !iommu_intpost ) > + return; > + > + spin_lock_irqsave(old_lock, flags); > + > + list_for_each_entry_safe(vmx, tmp, blocked_vcpus, pi_blocking.list) > + { > + /* > + * We need to find an online cpu as the NDST of the PI descriptor, it > + * doesn't matter whether it is within the cpupool of the domain or > + * not. As long as it is online, the vCPU will be woken up once the > + * notification event arrives. > + */ > +restart: I'd prefer if you did this without label and goto, but in any case labels should be indented by at least one space. Yet ... > + new_cpu = cpumask_any(&cpu_online_map); > + new_lock = &per_cpu(vmx_pi_blocking, new_cpu).lock; > + > + spin_lock(new_lock); > + > + /* > + * If the new_cpu is not online, that means it became offline between > + * we got 'new_cpu' and acquiring its lock above, we need to find > + * another online cpu instead. Such as, this fucntion is being called > + * on 'new_cpu' at the same time. Can this happen?? > + */ > + if ( !cpu_online(new_cpu) ) > + { > + spin_unlock(new_lock); > + goto restart; > + } ... I think this too has been discussed before: Is this case really possible? You're in the context of a CPU_DEAD or CPU_UP_CANCELED notification, which both get issued with cpu_add_remove_lock held. How can a second CPU go down in parallel? Jan
> -----Original Message----- > From: Jan Beulich [mailto:JBeulich@suse.com] > Sent: Thursday, September 1, 2016 4:49 PM > To: Wu, Feng <feng.wu@intel.com> > Cc: andrew.cooper3@citrix.com; dario.faggioli@citrix.com; > george.dunlap@eu.citrix.com; Tian, Kevin <kevin.tian@intel.com>; xen- > devel@lists.xen.org > Subject: Re: [PATCH v3 6/6] VMX: Fixup PI descritpor when cpu is offline > > >>> On 31.08.16 at 05:56, <feng.wu@intel.com> wrote: > > +void vmx_pi_desc_fixup(int cpu) > > unsigned int > > > +{ > > + unsigned int new_cpu, dest; > > + unsigned long flags; > > + struct arch_vmx_struct *vmx, *tmp; > > + spinlock_t *new_lock, *old_lock = &per_cpu(vmx_pi_blocking, cpu).lock; > > + struct list_head *blocked_vcpus = &per_cpu(vmx_pi_blocking, cpu).list; > > + > > + if ( !iommu_intpost ) > > + return; > > + > > + spin_lock_irqsave(old_lock, flags); > > + > > + list_for_each_entry_safe(vmx, tmp, blocked_vcpus, pi_blocking.list) > > + { > > + /* > > + * We need to find an online cpu as the NDST of the PI descriptor, it > > + * doesn't matter whether it is within the cpupool of the domain or > > + * not. As long as it is online, the vCPU will be woken up once the > > + * notification event arrives. > > + */ > > +restart: > > I'd prefer if you did this without label and goto, but in any case > labels should be indented by at least one space. Yet ... > > > + new_cpu = cpumask_any(&cpu_online_map); > > + new_lock = &per_cpu(vmx_pi_blocking, new_cpu).lock; > > + > > + spin_lock(new_lock); > > + > > + /* > > + * If the new_cpu is not online, that means it became offline between > > + * we got 'new_cpu' and acquiring its lock above, we need to find > > + * another online cpu instead. Such as, this fucntion is being called > > + * on 'new_cpu' at the same time. Can this happen?? > > + */ > > + if ( !cpu_online(new_cpu) ) > > + { > > + spin_unlock(new_lock); > > + goto restart; > > + } > > ... I think this too has been discussed before: Is this case really > possible? You're in the context of a CPU_DEAD or CPU_UP_CANCELED > notification, which both get issued with cpu_add_remove_lock held. > How can a second CPU go down in parallel? Here is the call chain: cpu_down() -> stop_machine_run() -> get_cpu_maps() /* Try to hold the cpu_add_remove_lock */ ...... put_cpu_maps() /* Release the lock */ notifier_call_chain(..., CPU_DEAD, ...) -> vmx_vcpu_dead() -> vmx_pi_desc_fixup() Seems vmx_pi_desc_fixup() is not calling with holding cpu_add_remove_lock? Or do I miss something? Thanks for further comments in advance! Thanks, Feng > > Jan
>>> On 02.09.16 at 05:25, <feng.wu@intel.com> wrote: >> From: Jan Beulich [mailto:JBeulich@suse.com] >> Sent: Thursday, September 1, 2016 4:49 PM >> >>> On 31.08.16 at 05:56, <feng.wu@intel.com> wrote: >> > + new_cpu = cpumask_any(&cpu_online_map); >> > + new_lock = &per_cpu(vmx_pi_blocking, new_cpu).lock; >> > + >> > + spin_lock(new_lock); >> > + >> > + /* >> > + * If the new_cpu is not online, that means it became offline between >> > + * we got 'new_cpu' and acquiring its lock above, we need to find >> > + * another online cpu instead. Such as, this fucntion is being called >> > + * on 'new_cpu' at the same time. Can this happen?? >> > + */ >> > + if ( !cpu_online(new_cpu) ) >> > + { >> > + spin_unlock(new_lock); >> > + goto restart; >> > + } >> >> ... I think this too has been discussed before: Is this case really >> possible? You're in the context of a CPU_DEAD or CPU_UP_CANCELED >> notification, which both get issued with cpu_add_remove_lock held. >> How can a second CPU go down in parallel? > > Here is the call chain: > > cpu_down() -> > stop_machine_run() -> > get_cpu_maps() /* Try to hold the cpu_add_remove_lock */ > ...... > put_cpu_maps() /* Release the lock */ > notifier_call_chain(..., CPU_DEAD, ...) -> > vmx_vcpu_dead() -> > vmx_pi_desc_fixup() > > Seems vmx_pi_desc_fixup() is not calling with holding cpu_add_remove_lock? > Or do I miss something? Thanks for further comments in advance! The only place I see CPU_DEAD being passed to the notifier is right in cpu_down(), with cpu_hotplug_done() (which is basically an alias of put_cpu_maps()) clearly called a few lines later. Jan
diff --git a/xen/arch/x86/hvm/vmx/vmcs.c b/xen/arch/x86/hvm/vmx/vmcs.c index 1bd875a..f554d4c 100644 --- a/xen/arch/x86/hvm/vmx/vmcs.c +++ b/xen/arch/x86/hvm/vmx/vmcs.c @@ -578,6 +578,7 @@ void vmx_cpu_dead(unsigned int cpu) vmx_free_vmcs(per_cpu(vmxon_region, cpu)); per_cpu(vmxon_region, cpu) = 0; nvmx_cpu_dead(cpu); + vmx_pi_desc_fixup(cpu); } int vmx_cpu_up(void) diff --git a/xen/arch/x86/hvm/vmx/vmx.c b/xen/arch/x86/hvm/vmx/vmx.c index 071c063..5f428b7 100644 --- a/xen/arch/x86/hvm/vmx/vmx.c +++ b/xen/arch/x86/hvm/vmx/vmx.c @@ -211,6 +211,60 @@ static void vmx_pi_blocking_cleanup(struct vcpu *v) vmx_pi_remove_vcpu_from_blocking_list(v); } +void vmx_pi_desc_fixup(int cpu) +{ + unsigned int new_cpu, dest; + unsigned long flags; + struct arch_vmx_struct *vmx, *tmp; + spinlock_t *new_lock, *old_lock = &per_cpu(vmx_pi_blocking, cpu).lock; + struct list_head *blocked_vcpus = &per_cpu(vmx_pi_blocking, cpu).list; + + if ( !iommu_intpost ) + return; + + spin_lock_irqsave(old_lock, flags); + + list_for_each_entry_safe(vmx, tmp, blocked_vcpus, pi_blocking.list) + { + /* + * We need to find an online cpu as the NDST of the PI descriptor, it + * doesn't matter whether it is within the cpupool of the domain or + * not. As long as it is online, the vCPU will be woken up once the + * notification event arrives. + */ +restart: + new_cpu = cpumask_any(&cpu_online_map); + new_lock = &per_cpu(vmx_pi_blocking, new_cpu).lock; + + spin_lock(new_lock); + + /* + * If the new_cpu is not online, that means it became offline between + * we got 'new_cpu' and acquiring its lock above, we need to find + * another online cpu instead. Such as, this fucntion is being called + * on 'new_cpu' at the same time. Can this happen?? + */ + if ( !cpu_online(new_cpu) ) + { + spin_unlock(new_lock); + goto restart; + } + + ASSERT(vmx->pi_blocking.lock == old_lock); + + dest = cpu_physical_id(new_cpu); + write_atomic(&vmx->pi_desc.ndst, + x2apic_enabled ? dest : MASK_INSR(dest, PI_xAPIC_NDST_MASK)); + + list_move(&vmx->pi_blocking.list, + &per_cpu(vmx_pi_blocking, new_cpu).list); + vmx->pi_blocking.lock = new_lock; + spin_unlock(new_lock); + } + + spin_unlock_irqrestore(old_lock, flags); +} + /* This function is called when pcidevs_lock is held */ void vmx_pi_hooks_assign(struct domain *d) { diff --git a/xen/include/asm-x86/hvm/vmx/vmx.h b/xen/include/asm-x86/hvm/vmx/vmx.h index 4cdd9b1..9783c70 100644 --- a/xen/include/asm-x86/hvm/vmx/vmx.h +++ b/xen/include/asm-x86/hvm/vmx/vmx.h @@ -569,6 +569,7 @@ void free_p2m_hap_data(struct p2m_domain *p2m); void p2m_init_hap_data(struct p2m_domain *p2m); void vmx_pi_per_cpu_init(unsigned int cpu); +void vmx_pi_desc_fixup(int cpu); void vmx_pi_hooks_assign(struct domain *d); void vmx_pi_hooks_deassign(struct domain *d);
When cpu is offline, we need to move all the vcpus in its blocking list to another online cpu, this patch handles it. Signed-off-by: Feng Wu <feng.wu@intel.com> --- xen/arch/x86/hvm/vmx/vmcs.c | 1 + xen/arch/x86/hvm/vmx/vmx.c | 54 +++++++++++++++++++++++++++++++++++++++ xen/include/asm-x86/hvm/vmx/vmx.h | 1 + 3 files changed, 56 insertions(+)