From patchwork Thu Jun 23 02:53:29 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 9194411 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 10A596075C for ; Thu, 23 Jun 2016 02:55:26 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 03AFF2841D for ; Thu, 23 Jun 2016 02:55:26 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id EBDA828420; Thu, 23 Jun 2016 02:55:25 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=2.0 tests=BAYES_00, RCVD_IN_DNSWL_MED autolearn=unavailable version=3.3.1 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.9]) (using TLSv1.2 with cipher AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 59EA42841D for ; Thu, 23 Jun 2016 02:55:25 +0000 (UTC) Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.85_2 #1 (Red Hat Linux)) id 1bFumG-0000wO-Mf; Thu, 23 Jun 2016 02:54:00 +0000 Received: from mx0b-001b2d01.pphosted.com ([148.163.158.5] helo=mx0a-001b2d01.pphosted.com) by bombadil.infradead.org with esmtps (Exim 4.85_2 #1 (Red Hat Linux)) id 1bFumD-0000uk-LL for linux-arm-kernel@lists.infradead.org; Thu, 23 Jun 2016 02:53:58 +0000 Received: from pps.filterd (m0098419.ppops.net [127.0.0.1]) by mx0b-001b2d01.pphosted.com (8.16.0.11/8.16.0.11) with SMTP id u5N2mQin022355 for ; Wed, 22 Jun 2016 22:53:35 -0400 Received: from e33.co.us.ibm.com (e33.co.us.ibm.com [32.97.110.151]) by mx0b-001b2d01.pphosted.com with ESMTP id 23q6wdxw5w-1 (version=TLSv1.2 cipher=AES256-SHA bits=256 verify=NOT) for ; Wed, 22 Jun 2016 22:53:34 -0400 Received: from localhost by e33.co.us.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Wed, 22 Jun 2016 20:53:34 -0600 Received: from d03dlp02.boulder.ibm.com (9.17.202.178) by e33.co.us.ibm.com (192.168.1.133) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; Wed, 22 Jun 2016 20:53:33 -0600 X-IBM-Helo: d03dlp02.boulder.ibm.com X-IBM-MailFrom: paulmck@linux.vnet.ibm.com Received: from b01cxnp23034.gho.pok.ibm.com (b01cxnp23034.gho.pok.ibm.com [9.57.198.29]) by d03dlp02.boulder.ibm.com (Postfix) with ESMTP id 695ED3E4004C for ; Wed, 22 Jun 2016 20:53:32 -0600 (MDT) Received: from d01av01.pok.ibm.com (d01av01.pok.ibm.com [9.56.224.215]) by b01cxnp23034.gho.pok.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id u5N2rXG310158402 for ; Thu, 23 Jun 2016 02:53:33 GMT Received: from d01av01.pok.ibm.com (localhost [127.0.0.1]) by d01av01.pok.ibm.com (8.14.4/8.14.4/NCO v10.0 AVout) with ESMTP id u5N2rUS8018051 for ; Wed, 22 Jun 2016 22:53:31 -0400 Received: from paulmck-ThinkPad-W541 (sig-9-65-69-8.ibm.com [9.65.69.8]) by d01av01.pok.ibm.com (8.14.4/8.14.4/NCO v10.0 AVin) with ESMTP id u5N2rTgJ018027; Wed, 22 Jun 2016 22:53:30 -0400 Received: by paulmck-ThinkPad-W541 (Postfix, from userid 1000) id 7E15E16C15A6; Wed, 22 Jun 2016 19:53:29 -0700 (PDT) Date: Wed, 22 Jun 2016 19:53:29 -0700 From: "Paul E. McKenney" To: Joonsoo Kim Subject: Re: Boot failure on emev2/kzm9d (was: Re: [PATCH v2 11/11] mm/slab: lockless decision to grow cache) References: <20160620063942.GA13747@js1304-P5Q-DELUXE> <20160620131254.GO3923@linux.vnet.ibm.com> <20160621064302.GA20635@js1304-P5Q-DELUXE> <20160621125406.GF3923@linux.vnet.ibm.com> <20160622005208.GB25106@js1304-P5Q-DELUXE> <20160622190859.GA1473@linux.vnet.ibm.com> <20160623004935.GA20752@linux.vnet.ibm.com> <20160623023756.GA30438@js1304-P5Q-DELUXE> <20160623024742.GD1473@linux.vnet.ibm.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20160623024742.GD1473@linux.vnet.ibm.com> User-Agent: Mutt/1.5.21 (2010-09-15) X-TM-AS-MML: disable X-Content-Scanned: Fidelis XPS MAILER x-cbid: 16062302-0008-0000-0000-000004E2AE60 X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 16062302-0009-0000-0000-000038A73D65 Message-Id: <20160623025329.GA13095@linux.vnet.ibm.com> X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:, , definitions=2016-06-23_01:, , signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 spamscore=0 suspectscore=0 malwarescore=0 phishscore=0 adultscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1604210000 definitions=main-1606230028 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20160622_195357_885310_22C43CE7 X-CRM114-Status: GOOD ( 37.30 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: paulmck@linux.vnet.ibm.com Cc: linux-renesas-soc@vger.kernel.org, David Rientjes , "linux-kernel@vger.kernel.org" , Pekka Enberg , Linux MM , Geert Uytterhoeven , Jesper Dangaard Brouer , Andrew Morton , Christoph Lameter , "linux-arm-kernel@lists.infradead.org" Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Virus-Scanned: ClamAV using ClamSMTP On Wed, Jun 22, 2016 at 07:47:42PM -0700, Paul E. McKenney wrote: > On Thu, Jun 23, 2016 at 11:37:56AM +0900, Joonsoo Kim wrote: > > On Wed, Jun 22, 2016 at 05:49:35PM -0700, Paul E. McKenney wrote: > > > On Wed, Jun 22, 2016 at 12:08:59PM -0700, Paul E. McKenney wrote: > > > > On Wed, Jun 22, 2016 at 05:01:35PM +0200, Geert Uytterhoeven wrote: > > > > > On Wed, Jun 22, 2016 at 2:52 AM, Joonsoo Kim wrote: > > > > > > Could you try below patch to check who causes the hang? > > > > > > > > > > > > And, if sysalt-t works when hang, could you get sysalt-t output? I haven't > > > > > > used it before but Paul could find some culprit on it. :) > > > > > > > > > > > > Thanks. > > > > > > > > > > > > > > > > > > ----->8----- > > > > > > diff --git a/mm/slab.c b/mm/slab.c > > > > > > index 763096a..9652d38 100644 > > > > > > --- a/mm/slab.c > > > > > > +++ b/mm/slab.c > > > > > > @@ -964,8 +964,13 @@ static int setup_kmem_cache_node(struct kmem_cache *cachep, > > > > > > * guaranteed to be valid until irq is re-enabled, because it will be > > > > > > * freed after synchronize_sched(). > > > > > > */ > > > > > > - if (force_change) > > > > > > + if (force_change) { > > > > > > + if (num_online_cpus() > 1) > > > > > > + dump_stack(); > > > > > > synchronize_sched(); > > > > > > + if (num_online_cpus() > 1) > > > > > > + dump_stack(); > > > > > > + } > > > > > > > > > > I've only added the first one, as I would never see the second one. All of > > > > > this happens before the serial console is activated, earlycon is not supported, > > > > > and I only have remote access. > > > > > > > > > > Brought up 2 CPUs > > > > > SMP: Total of 2 processors activated (2132.00 BogoMIPS). > > > > > CPU: All CPU(s) started in SVC mode. > > > > > CPU: 0 PID: 1 Comm: swapper/0 Not tainted > > > > > 4.7.0-rc4-kzm9d-00404-g4a235e6dde4404dd-dirty #89 > > > > > Hardware name: Generic Emma Mobile EV2 (Flattened Device Tree) > > > > > [] (unwind_backtrace) from [] (show_stack+0x10/0x14) > > > > > [] (show_stack) from [] (dump_stack+0x7c/0x9c) > > > > > [] (dump_stack) from [] (setup_kmem_cache_node+0x140/0x170) > > > > > [] (setup_kmem_cache_node) from [] > > > > > (__do_tune_cpucache+0xf4/0x114) > > > > > [] (__do_tune_cpucache) from [] (enable_cpucache+0xf8/0x148) > > > > > [] (enable_cpucache) from [] > > > > > (__kmem_cache_create+0x1a8/0x1d0) > > > > > [] (__kmem_cache_create) from [] > > > > > (kmem_cache_create+0xbc/0x190) > > > > > [] (kmem_cache_create) from [] (shmem_init+0x34/0xb0) > > > > > [] (shmem_init) from [] (kernel_init_freeable+0x98/0x1ec) > > > > > [] (kernel_init_freeable) from [] (kernel_init+0x8/0x110) > > > > > [] (kernel_init) from [] (ret_from_fork+0x14/0x3c) > > > > > devtmpfs: initialized > > > > > > > > I don't see anything here that would prevent grace periods from completing. > > > > > > > > The CPUs are using the normal hotplug sequence to come online, correct? > > > > > > And either way, could you please apply the patch below and then > > > invoke rcu_dump_rcu_sched_tree() just before the offending call to > > > synchronize_sched()? That will tell me what CPUs RCU believes exist, > > > and perhaps also which CPU is holding it up. > > > > I can't find rcu_dump_rcu_sched_tree(). Do you mean > > rcu_dump_rcu_node_tree()? Anyway, there is no patch below so I attach > > one which does what Paul want, maybe. > > One of those days, I guess! :-/ > > Your patch is exactly what I intended to send, thank you! Ah, but your telepathy was not sufficient to intuit the additional information I need. Please see the patch at the end. Your hunk in mm/slab.c is needed on top of my patch. So I am clearly having difficulties reading as well as including patches today... Thanx, Paul > > Thanks. > > > > ------->8--------- > > diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c > > index 88d3f95..6b650f0 100644 > > --- a/kernel/rcu/tree.c > > +++ b/kernel/rcu/tree.c > > @@ -4171,7 +4171,7 @@ static void __init rcu_init_geometry(void) > > * Dump out the structure of the rcu_node combining tree associated > > * with the rcu_state structure referenced by rsp. > > */ > > -static void __init rcu_dump_rcu_node_tree(struct rcu_state *rsp) > > +static void rcu_dump_rcu_node_tree(struct rcu_state *rsp) > > { > > int level = 0; > > struct rcu_node *rnp; > > @@ -4189,6 +4189,11 @@ static void __init rcu_dump_rcu_node_tree(struct rcu_state *rsp) > > pr_cont("\n"); > > } > > > > +void rcu_dump_rcu_sched_tree(void) > > +{ > > + rcu_dump_rcu_node_tree(&rcu_sched_state); > > +} > > + > > void __init rcu_init(void) > > { > > int cpu; > > diff --git a/mm/slab.c b/mm/slab.c > > index 763096a..d88976c 100644 > > --- a/mm/slab.c > > +++ b/mm/slab.c > > @@ -909,6 +909,8 @@ static int init_cache_node_node(int node) > > return 0; > > } > > > > +extern void rcu_dump_rcu_sched_tree(void); > > + > > static int setup_kmem_cache_node(struct kmem_cache *cachep, > > int node, gfp_t gfp, bool force_change) > > { > > @@ -964,8 +966,10 @@ static int setup_kmem_cache_node(struct kmem_cache *cachep, > > * guaranteed to be valid until irq is re-enabled, because it will be > > * freed after synchronize_sched(). > > */ > > - if (force_change) > > + if (force_change) { > > + rcu_dump_rcu_sched_tree(); > > synchronize_sched(); > > + } > > > > fail: > > kfree(old_shared); > > ------------------------------------------------------------------------ diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index c7f1bc4f817c..2eda7bece401 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -4707,7 +4707,7 @@ static void __init rcu_init_geometry(void) * Dump out the structure of the rcu_node combining tree associated * with the rcu_state structure referenced by rsp. */ -static void __init rcu_dump_rcu_node_tree(struct rcu_state *rsp) +static void rcu_dump_rcu_node_tree(struct rcu_state *rsp) { int level = 0; struct rcu_node *rnp; @@ -4720,11 +4720,18 @@ static void __init rcu_dump_rcu_node_tree(struct rcu_state *rsp) pr_info(" "); level = rnp->level; } - pr_cont("%d:%d ^%d ", rnp->grplo, rnp->grphi, rnp->grpnum); + pr_cont("%d:%d/%#lx/%#lx ^%d ", rnp->grplo, rnp->grphi, + rnp->qsmask, + rnp->qsmaskinit | rnp->qsmaskinitnext, rnp->grpnum); } pr_cont("\n"); } +void rcu_dump_rcu_sched_tree(void) +{ + rcu_dump_rcu_node_tree(&rcu_sched_state); +} + void __init rcu_init(void) { int cpu;