From patchwork Tue Dec 12 16:49:00 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 10107879 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 17884602B3 for ; Tue, 12 Dec 2017 16:49:32 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 071F7285B5 for ; Tue, 12 Dec 2017 16:49:32 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id F002D29932; Tue, 12 Dec 2017 16:49:31 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from bombadil.infradead.org (bombadil.infradead.org [65.50.211.133]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 7C085285B5 for ; Tue, 12 Dec 2017 16:49:31 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:Reply-To:List-Subscribe:List-Help: List-Post:List-Archive:List-Unsubscribe:List-Id:Message-Id:In-Reply-To: MIME-Version:References:Subject:To:From:Date:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=0uPGwWiU+oD6TTOlZWhGwSTDNLNXWYVdaVYaQmqILl8=; b=A8MN+HrRbo8dJ4 aQDlf6l6u8mrHEN7xdvegCOVXpo5riVx8cGEoz6RSNqJJo7VcTwbYkmOB7fQI1lwfqAshvZBJnY2k q9vNCck2u2ZIx/YQ2JanoQEy3whfQBkLbbXC+mNfc/d0Yu4B7QakHxNwxzAmt7uSHHHy8Nnjjer8u FNk04d8IkPOeOsKweY49YDxnEMuvb4E614f4XNKPqu4p1EDgRlLb6eD3fRxXFVEDWj9a30WSy2sq8 VWaiSdrwDJ3R7b+plImy68EX4Pc8Or/DvIHKt68UiWX4OYYUpPzZUwV95po8HKo6Q5A0ICF7vh5B0 AJ/9GV0NNyCyf8q9j9oQ==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.87 #1 (Red Hat Linux)) id 1eOnkI-0003tt-Uc; Tue, 12 Dec 2017 16:49:30 +0000 Received: from mx0b-001b2d01.pphosted.com ([148.163.158.5] helo=mx0a-001b2d01.pphosted.com) by bombadil.infradead.org with esmtps (Exim 4.87 #1 (Red Hat Linux)) id 1eOnkG-0003s9-2i for linux-arm-kernel@lists.infradead.org; Tue, 12 Dec 2017 16:49:30 +0000 Received: from pps.filterd (m0098416.ppops.net [127.0.0.1]) by mx0b-001b2d01.pphosted.com (8.16.0.21/8.16.0.21) with SMTP id vBCGlAVT025874 for ; Tue, 12 Dec 2017 11:49:06 -0500 Received: from e17.ny.us.ibm.com (e17.ny.us.ibm.com [129.33.205.207]) by mx0b-001b2d01.pphosted.com with ESMTP id 2etjs68kr3-1 (version=TLSv1.2 cipher=AES256-SHA bits=256 verify=NOT) for ; Tue, 12 Dec 2017 11:49:05 -0500 Received: from localhost by e17.ny.us.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Tue, 12 Dec 2017 11:49:04 -0500 Received: from b01cxnp22036.gho.pok.ibm.com (9.57.198.26) by e17.ny.us.ibm.com (146.89.104.204) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; Tue, 12 Dec 2017 11:49:01 -0500 Received: from b01ledav003.gho.pok.ibm.com (b01ledav003.gho.pok.ibm.com [9.57.199.108]) by b01cxnp22036.gho.pok.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id vBCGn03W46792718; Tue, 12 Dec 2017 16:49:00 GMT Received: from b01ledav003.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id C8DF3B204D; Tue, 12 Dec 2017 11:46:06 -0500 (EST) Received: from paulmck-ThinkPad-W541 (unknown [9.85.141.99]) by b01ledav003.gho.pok.ibm.com (Postfix) with ESMTP id 8094CB2046; Tue, 12 Dec 2017 11:46:06 -0500 (EST) Received: by paulmck-ThinkPad-W541 (Postfix, from userid 1000) id 7AB3416C0B86; Tue, 12 Dec 2017 08:49:00 -0800 (PST) Date: Tue, 12 Dec 2017 08:49:00 -0800 From: "Paul E. McKenney" To: Russell King - ARM Linux Subject: Re: WARNING: suspicious RCU usage References: <20171210113930.zimywmxzsutwzzmz@linux-u7w5.ap.freescale.net> <20171210120012.GM10595@n2100.armlinux.org.uk> <20171210190727.GJ7829@linux.vnet.ibm.com> <20171210193438.GP10595@n2100.armlinux.org.uk> <20171210213930.GL7829@linux.vnet.ibm.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20171210213930.GL7829@linux.vnet.ibm.com> User-Agent: Mutt/1.5.21 (2010-09-15) X-TM-AS-GCONF: 00 x-cbid: 17121216-0040-0000-0000-000003D04782 X-IBM-SpamModules-Scores: X-IBM-SpamModules-Versions: BY=3.00008194; HX=3.00000241; KW=3.00000007; PH=3.00000004; SC=3.00000244; SDB=6.00959208; UDB=6.00485086; IPR=6.00739231; BA=6.00005738; NDR=6.00000001; ZLA=6.00000005; ZF=6.00000009; ZB=6.00000000; ZP=6.00000000; ZH=6.00000000; ZU=6.00000002; MB=3.00018504; XFM=3.00000015; UTC=2017-12-12 16:49:03 X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 17121216-0041-0000-0000-000007C588AB Message-Id: <20171212164900.GA6673@linux.vnet.ibm.com> X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:, , definitions=2017-12-12_10:, , signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 impostorscore=0 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1709140000 definitions=main-1712120240 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20171212_084928_254425_709E5F2A X-CRM114-Status: GOOD ( 27.21 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: paulmck@linux.vnet.ibm.com Cc: Peng Fan , fabio.estevam@nxp.com, shawnguo@kernel.org, aisheng.dong@nxp.com, linux-arm-kernel@lists.infradead.org Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Virus-Scanned: ClamAV using ClamSMTP On Sun, Dec 10, 2017 at 01:39:30PM -0800, Paul E. McKenney wrote: > On Sun, Dec 10, 2017 at 07:34:39PM +0000, Russell King - ARM Linux wrote: > > On Sun, Dec 10, 2017 at 11:07:27AM -0800, Paul E. McKenney wrote: > > > On Sun, Dec 10, 2017 at 12:00:12PM +0000, Russell King - ARM Linux wrote: > > > > +Paul > > > > > > > > Annoyingly, it looks like calling "complete()" from a dying CPU is > > > > triggering the RCU usage warning. From what I remember, this is an > > > > old problem, and we still have no better solution for this other than > > > > to persist with the warning. > > > > > > I thought that this issue was resolved with tglx's use of IPIs from > > > the outgoing CPU. Or is this due to an additional complete() from the > > > ARM code? If so, could it also use tglx's IPI trick? > > > > I don't think it was tglx's IPI trick, I've had code sitting in my tree > > for a while for it, but it has its own set of problems which are not > > resolvable: > > > > 1. it needs more IPIs than we have available on all platforms > > OK, I will ask the stupid question... Is it possible to multiplex > the IPIs, for example, by using smp_call_function_single()? On the perhaps unlikely off-chance that it is both useful and welcome, the (untested, probably does not even build) patch below illustrates the use of smp_call_function_single(). This is based on the patch Russell sent -- for all I know, it might well be that there are other places needing similar changes. But something to try out for anyone wishing to do so. Thanx, Paul ------------------------------------------------------------------------ commit c579a1494ccbc7ebf5548115571a2988ea1a1fe5 Author: Paul E. McKenney Date: Mon Dec 11 09:40:58 2017 -0800 ARM: CPU hotplug: Delegate complete() to surviving CPU The ARM implementation of arch_cpu_idle_dead() invokes complete(), but does so after RCU has stopped watching the outgoing CPU, which results in lockdep complaints because complete() invokes functions containing RCU readers. This patch therefore uses Thomas Gleixner's trick of delegating the complete() call to a surviving CPU via smp_call_function_single(). This patch is untested, and probably does not even build. Reported-by: Peng Fan Reported-by: Russell King - ARM Linux Signed-off-by: Paul E. McKenney Tested-by: Fabio Estevam diff --git a/arch/arm/kernel/smp.c b/arch/arm/kernel/smp.c index b4fbf00ee4ad..75f85e20aafa 100644 --- a/arch/arm/kernel/smp.c +++ b/arch/arm/kernel/smp.c @@ -267,6 +267,14 @@ void __cpu_die(unsigned int cpu) } /* + * Invoke complete() on behalf of the outgoing CPU. + */ +static void arch_cpu_idle_dead_complete(void *arg) +{ + complete(&cpu_died); +} + +/* * Called from the idle thread for the CPU which has been shutdown. * * Note that we disable IRQs here, but do not re-enable them @@ -293,9 +301,11 @@ void arch_cpu_idle_dead(void) /* * Tell __cpu_die() that this CPU is now safe to dispose of. Once * this returns, power and/or clocks can be removed at any point - * from this CPU and its cache by platform_cpu_kill(). + * from this CPU and its cache by platform_cpu_kill(). We cannot + * call complete() this late, so we delegate it to an online CPU. */ - complete(&cpu_died); + smp_call_function_single(cpumask_first(cpu_online_mask), + arch_cpu_idle_dead_complete, NULL, 0); /* * Ensure that the cache lines associated with that completion are