From patchwork Wed Jun 22 00:52:08 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joonsoo Kim X-Patchwork-Id: 9191627 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 2C7496075A for ; Wed, 22 Jun 2016 00:52:00 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 1AA3128319 for ; Wed, 22 Jun 2016 00:52:00 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 0F3B82838C; Wed, 22 Jun 2016 00:52:00 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=2.0 tests=BAYES_00, RCVD_IN_DNSWL_MED autolearn=unavailable version=3.3.1 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.9]) (using TLSv1.2 with cipher AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 7D2CD28319 for ; Wed, 22 Jun 2016 00:51:59 +0000 (UTC) Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.85_2 #1 (Red Hat Linux)) id 1bFWMo-0002Vw-6b; Wed, 22 Jun 2016 00:50:06 +0000 Received: from lgeamrelo13.lge.com ([156.147.23.53]) by bombadil.infradead.org with esmtp (Exim 4.85_2 #1 (Red Hat Linux)) id 1bFWMk-0001kg-BR for linux-arm-kernel@lists.infradead.org; Wed, 22 Jun 2016 00:50:04 +0000 Received: from unknown (HELO lgemrelse7q.lge.com) (156.147.1.151) by 156.147.23.53 with ESMTP; 22 Jun 2016 09:49:38 +0900 X-Original-SENDERIP: 156.147.1.151 X-Original-MAILFROM: iamjoonsoo.kim@lge.com Received: from unknown (HELO localhost) (10.177.222.138) by 156.147.1.151 with ESMTP; 22 Jun 2016 09:49:35 +0900 X-Original-SENDERIP: 10.177.222.138 X-Original-MAILFROM: iamjoonsoo.kim@lge.com Date: Wed, 22 Jun 2016 09:52:08 +0900 From: Joonsoo Kim To: "Paul E. McKenney" Subject: Re: Boot failure on emev2/kzm9d (was: Re: [PATCH v2 11/11] mm/slab: lockless decision to grow cache) Message-ID: <20160622005208.GB25106@js1304-P5Q-DELUXE> References: <20160614062456.GB13753@js1304-P5Q-DELUXE> <20160614081125.GA17700@js1304-P5Q-DELUXE> <20160615022325.GA19863@js1304-P5Q-DELUXE> <20160620063942.GA13747@js1304-P5Q-DELUXE> <20160620131254.GO3923@linux.vnet.ibm.com> <20160621064302.GA20635@js1304-P5Q-DELUXE> <20160621125406.GF3923@linux.vnet.ibm.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20160621125406.GF3923@linux.vnet.ibm.com> User-Agent: Mutt/1.5.21 (2010-09-15) X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20160621_175002_761658_935ED157 X-CRM114-Status: GOOD ( 38.63 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: linux-renesas-soc@vger.kernel.org, David Rientjes , "linux-kernel@vger.kernel.org" , Pekka Enberg , Linux MM , Geert Uytterhoeven , Jesper Dangaard Brouer , Andrew Morton , Christoph Lameter , "linux-arm-kernel@lists.infradead.org" Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Virus-Scanned: ClamAV using ClamSMTP On Tue, Jun 21, 2016 at 05:54:06AM -0700, Paul E. McKenney wrote: > On Tue, Jun 21, 2016 at 03:43:02PM +0900, Joonsoo Kim wrote: > > On Mon, Jun 20, 2016 at 06:12:54AM -0700, Paul E. McKenney wrote: > > > On Mon, Jun 20, 2016 at 03:39:43PM +0900, Joonsoo Kim wrote: > > > > CCing Paul to ask some question. > > > > > > > > On Wed, Jun 15, 2016 at 10:39:47AM +0200, Geert Uytterhoeven wrote: > > > > > Hi Joonsoo, > > > > > > > > > > On Wed, Jun 15, 2016 at 4:23 AM, Joonsoo Kim wrote: > > > > > > On Tue, Jun 14, 2016 at 12:45:14PM +0200, Geert Uytterhoeven wrote: > > > > > >> On Tue, Jun 14, 2016 at 10:11 AM, Joonsoo Kim wrote: > > > > > >> > On Tue, Jun 14, 2016 at 09:31:23AM +0200, Geert Uytterhoeven wrote: > > > > > >> >> On Tue, Jun 14, 2016 at 8:24 AM, Joonsoo Kim wrote: > > > > > >> >> > On Mon, Jun 13, 2016 at 09:43:13PM +0200, Geert Uytterhoeven wrote: > > > > > >> >> >> On Tue, Apr 12, 2016 at 6:51 AM, wrote: > > > > > >> >> >> > From: Joonsoo Kim > > > > > >> >> >> > To check whther free objects exist or not precisely, we need to grab a > > > > > >> >> >> > lock. But, accuracy isn't that important because race window would be > > > > > >> >> >> > even small and if there is too much free object, cache reaper would reap > > > > > >> >> >> > it. So, this patch makes the check for free object exisistence not to > > > > > >> >> >> > hold a lock. This will reduce lock contention in heavily allocation case. > > > > > > > > > > >> >> >> I've bisected a boot failure (no output at all) in v4.7-rc2 on emev2/kzm9d > > > > > >> >> >> (Renesas dual Cortex A9) to this patch, which is upstream commit > > > > > >> >> >> 801faf0db8947e01877920e848a4d338dd7a99e7. > > > > > > > > > > > It's curious that synchronize_sched() has some effect in this early > > > > > > phase. In synchronize_sched(), rcu_blocking_is_gp() is called and > > > > > > it checks num_online_cpus <= 1. If so, synchronize_sched() does nothing. > > > > > > > > > > > > It would be related to might_sleep() in rcu_blocking_is_gp() but I'm not sure now. > > > > > > > > > > > > First, I'd like to confirm that num_online_cpus() is correct. > > > > > > Could you try following patch and give me a dmesg? > > > > > > > > > > > > Thanks. > > > > > > > > > > > > ------->8---------- > > > > > > diff --git a/mm/slab.c b/mm/slab.c > > > > > > index 763096a..5b7300a 100644 > > > > > > --- a/mm/slab.c > > > > > > +++ b/mm/slab.c > > > > > > @@ -964,8 +964,10 @@ static int setup_kmem_cache_node(struct kmem_cache *cachep, > > > > > > * guaranteed to be valid until irq is re-enabled, because it will be > > > > > > * freed after synchronize_sched(). > > > > > > */ > > > > > > - if (force_change) > > > > > > - synchronize_sched(); > > > > > > + if (force_change) { > > > > > > + WARN_ON_ONCE(num_online_cpus() <= 1); > > > > > > + WARN_ON_ONCE(num_online_cpus() > 1); > > > > > > + } > > > > > > > > > > Full dmesg output below. > > > > > > > > > > I also tested whether it's the call to synchronize_sched() before or after > > > > > secondary CPU bringup that hangs. > > > > > > > > > > if (force_change && num_online_cpus() <= 1) > > > > > synchronize_sched(); > > > > > > > > > > boots. > > > > > > > > > > if (force_change && num_online_cpus() > 1) > > > > > synchronize_sched(); > > > > > > > > > > hangs. > > > > > > > > Hello, Paul. > > > > > > > > I changed slab.c to use synchronize_sched() for full memory barrier. First > > > > call happens on kmem_cache_init_late() and it would not be a problem > > > > because, at this time, num_online_cpus() <= 1 and synchronize_sched() > > > > would return immediately. Second call site would be shmem_init() > > > > and it seems that system hangs on it. Since smp is already initialized > > > > at that time, there would be some effect of synchronize_sched() but I > > > > can't imagine what's wrong here. Is it invalid moment to call > > > > synchronize_sched()? > > > > > > > > Note that my x86 virtual machine works fine even if > > > > synchronize_sched() is called in shmem_init() but Geert's some ARM > > > > machines (not all ARM machine) don't work well with it. > > > > > > Color me confused. > > > > > > Is Geert's ARM system somehow adding the second CPU before > > > rcu_spawn_gp_kthread() is called, that is, before or during > > > early_initcall() time? > > > > Hang would happen on shmem_init() which is called in do_basic_setup(). > > do_basic_setup() is called after early_initcall(). > > Thank you for the info! > > That should be lat enough that the RCU kthreads are alive and well. > > Can you get sysalt-t output? > > > Hmm... Is it okay to call synchronize_sched() by kernel thread? > > Yes, it can, in fact, rcutorture does this all the time. As do any > number of other kthreads. Paul, thanks for confirmation. Geert, we need to try more debugging. Could you try below patch to check who causes the hang? And, if sysalt-t works when hang, could you get sysalt-t output? I haven't used it before but Paul could find some culprit on it. :) Thanks. ----->8----- diff --git a/mm/slab.c b/mm/slab.c index 763096a..9652d38 100644 --- a/mm/slab.c +++ b/mm/slab.c @@ -964,8 +964,13 @@ static int setup_kmem_cache_node(struct kmem_cache *cachep, * guaranteed to be valid until irq is re-enabled, because it will be * freed after synchronize_sched(). */ - if (force_change) + if (force_change) { + if (num_online_cpus() > 1) + dump_stack(); synchronize_sched(); + if (num_online_cpus() > 1) + dump_stack(); + } fail: kfree(old_shared);