From patchwork Fri Jun 3 19:31:57 2011 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 848152 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by demeter2.kernel.org (8.14.4/8.14.3) with ESMTP id p53JWKSI023334 for ; Fri, 3 Jun 2011 19:32:20 GMT Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1754723Ab1FCTcO (ORCPT ); Fri, 3 Jun 2011 15:32:14 -0400 Received: from e8.ny.us.ibm.com ([32.97.182.138]:45439 "EHLO e8.ny.us.ibm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1754811Ab1FCTcM (ORCPT ); Fri, 3 Jun 2011 15:32:12 -0400 Received: from d01relay03.pok.ibm.com (d01relay03.pok.ibm.com [9.56.227.235]) by e8.ny.us.ibm.com (8.14.4/8.13.1) with ESMTP id p53JL2Ol032550 for ; Fri, 3 Jun 2011 15:21:02 -0400 Received: from d01av01.pok.ibm.com (d01av01.pok.ibm.com [9.56.224.215]) by d01relay03.pok.ibm.com (8.13.8/8.13.8/NCO v10.0) with ESMTP id p53JVxdX123194 for ; Fri, 3 Jun 2011 15:31:59 -0400 Received: from d01av01.pok.ibm.com (loopback [127.0.0.1]) by d01av01.pok.ibm.com (8.14.4/8.13.1/NCO v10.0 AVout) with ESMTP id p53JVvxw021435 for ; Fri, 3 Jun 2011 15:31:59 -0400 Received: from paulmck-ThinkPad-W500 (paulmck-ThinkPad-W500.beaverton.ibm.com [9.47.24.65]) by d01av01.pok.ibm.com (8.14.4/8.13.1/NCO v10.0 AVin) with ESMTP id p53JVvnY021403; Fri, 3 Jun 2011 15:31:57 -0400 Received: by paulmck-ThinkPad-W500 (Postfix, from userid 1000) id 2DCF213F7C6; Fri, 3 Jun 2011 12:31:57 -0700 (PDT) Date: Fri, 3 Jun 2011 12:31:57 -0700 From: "Paul E. McKenney" To: Sasha Levin Cc: Ingo Molnar , Yinghai Lu , Avi Kivity , Takuya Yoshikawa , Pekka Enberg , kvm@vger.kernel.org, asias.hejun@gmail.com, gorcunov@gmail.com, prasadjoshi124@gmail.com, takuya.yoshikawa@gmail.com Subject: Re: [PATCH v2 6/8] kvm tools: Add rwlock wrapper Message-ID: <20110603193157.GO2333@linux.vnet.ibm.com> Reply-To: paulmck@linux.vnet.ibm.com References: <20110530201110.f3bf20b5.yoshikawa.takuya@oss.ntt.co.jp> <1306753954.14564.92.camel@lappy> <20110530202646.eff0ea28.yoshikawa.takuya@oss.ntt.co.jp> <4DE381DB.8040804@redhat.com> <20110530114949.GD22324@elte.hu> <4DE387DA.3020307@redhat.com> <20110530123602.GK22324@elte.hu> <1307086079.13088.8.camel@lappy> <20110603073427.GF15375@elte.hu> <1307087659.13088.10.camel@lappy> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <1307087659.13088.10.camel@lappy> User-Agent: Mutt/1.5.20 (2009-06-14) Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org X-Greylist: IP, sender and recipient auto-whitelisted, not delayed by milter-greylist-4.2.6 (demeter2.kernel.org [140.211.167.43]); Fri, 03 Jun 2011 19:32:20 +0000 (UTC) On Fri, Jun 03, 2011 at 10:54:19AM +0300, Sasha Levin wrote: > On Fri, 2011-06-03 at 09:34 +0200, Ingo Molnar wrote: > > * Sasha Levin wrote: > > > > > > with no apparent progress being made. > > > > > > Since it's something that worked in 2.6.37, I've looked into it to > > > find what might have caused this issue. > > > > > > I've bisected guest kernels and found that the problem starts with: > > > > > > a26ac2455ffcf3be5c6ef92bc6df7182700f2114 is the first bad commit > > > commit a26ac2455ffcf3be5c6ef92bc6df7182700f2114 > > > Author: Paul E. McKenney > > > Date: Wed Jan 12 14:10:23 2011 -0800 > > > > > > rcu: move TREE_RCU from softirq to kthread > > > > > > Ingo, could you confirm that the problem goes away for you when you > > > use an earlier commit? > > > > testing will have to wait, but there's a recent upstream fix: > > > > d72bce0e67e8: rcu: Cure load woes > > > > That *might* perhaps address this problem too. > > > I've re-tested with Linus's current git, the problem is still there. > > > If not then this appears to be some sort of RCU related livelock with > > brutally overcommitted vcpus. On native this would show up too, in a > > less drastic form, as a spurious bootup delay. > > I don't think it was overcommited by *that* much. With that commit it > usually hangs at 20-40 vcpus, while without it I can go up to 255. Here is a diagnostic patch, untested. It assumes that your system has only a few CPUs (maybe 8-16) and that timers are still running. It dumps out some RCU state if grace periods extend for more than a few seconds. To activate it, call rcu_diag_timer_start() from process context. To stop it, call rcu_diag_timer_stop(), also from process context. Thoughts? Thanx, Paul ------------------------------------------------------------------------ rcu: diagnostic check of kthread state Not-signed-off-by: Paul E. McKenney --- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h index 99f9aa7..489ea1b 100644 --- a/include/linux/rcupdate.h +++ b/include/linux/rcupdate.h @@ -80,6 +80,8 @@ extern void call_rcu_sched(struct rcu_head *head, extern void synchronize_sched(void); extern void rcu_barrier_bh(void); extern void rcu_barrier_sched(void); +extern void rcu_diag_timer_start(void); +extern void rcu_diag_timer_stop(void); static inline void __rcu_read_lock_bh(void) { diff --git a/kernel/rcutree.c b/kernel/rcutree.c index 89419ff..bb61574 100644 --- a/kernel/rcutree.c +++ b/kernel/rcutree.c @@ -2423,3 +2423,48 @@ void __init rcu_init(void) } #include "rcutree_plugin.h" + +/* Diagnostic code for boot-time hangs observed in early 3.0 days. */ + +static int rcu_diag_timer_must_stop; +struct timer_list rcu_diag_timer; +#define RCU_DIAG_TIMER_PERIOD (10 * HZ) + +static void rcu_diag_timer_handler(unsigned long unused) +{ + int cpu; + + if (rcu_diag_timer_must_stop) + return; + + if(ULONG_CMP_GE(jiffies, + rcu_sched_state.gp_start + RCU_DIAG_TIMER_PERIOD)) + for_each_online_cpu(cpu) { + printk(KERN_ALERT "rcu_diag: rcuc%d %u/%u/%d ", + cpu, + per_cpu(rcu_cpu_kthread_status, cpu), + per_cpu(rcu_cpu_kthread_loops, cpu), + per_cpu(rcu_cpu_has_work, cpu)); + sched_show_task(current); + } + + if (rcu_diag_timer_must_stop) + return; + mod_timer(&rcu_diag_timer, RCU_DIAG_TIMER_PERIOD + jiffies); +} + +void rcu_diag_timer_start(void) +{ + rcu_diag_timer_must_stop = 0; + setup_timer(&rcu_diag_timer, + rcu_diag_timer_handler, (unsigned long) NULL); + mod_timer(&rcu_diag_timer, RCU_DIAG_TIMER_PERIOD + jiffies); +} +EXPORT_SYMBOL_GPL(rcu_diag_timer_start); + +void rcu_diag_timer_stop(void) +{ + rcu_diag_timer_must_stop = 1; + del_timer(&rcu_diag_timer); +} +EXPORT_SYMBOL_GPL(rcu_diag_timer_stop);