From patchwork Fri Dec 15 18:23:40 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 10115827 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 7F5CA6019C for ; Fri, 15 Dec 2017 18:24:40 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 75FA22A080 for ; Fri, 15 Dec 2017 18:24:40 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 68CDA2A0BB; Fri, 15 Dec 2017 18:24:40 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from bombadil.infradead.org (bombadil.infradead.org [65.50.211.133]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 554112A080 for ; Fri, 15 Dec 2017 18:24:39 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:Reply-To:List-Subscribe:List-Help: List-Post:List-Archive:List-Unsubscribe:List-Id:Message-Id:In-Reply-To: MIME-Version:References:Subject:To:From:Date:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=a0erP0MxBNFbZZ4LAMIFbpvNrO0AISdYQCdcQ9i22vQ=; b=XPrguYXyZfL+Mo T0yu8Z/fIDjQ1yM5b8CQ+gBVxmo3adcmHLiCeldKLibYAmtI2pmzVtNfcAYUGPgoZoJfYyx09XDSA Ae3p4MnjT6n0XumleoUJmg2VwBDuDdo8DWnnbYevg60mdO/uIBLQ1fCml7bSafdJ5Y15WE6gVQYD0 1XPy7cAcIy73jFEzb/c1OKq/ZkZ1D51aQ8h8x3wBjdkOjC4bB69XMFmY25M1hmG3Qu2R5I/ce81Fv 3Ci59Hhqd1CJVyEZ7A4spi8b30bXxyT25q+7wUTievxiLYtmcbKyiDHlo8bL8a4iNchQqoBYzC4uV 861xggSUTUlCCUeDO9Kw==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.87 #1 (Red Hat Linux)) id 1ePuej-00007I-9b; Fri, 15 Dec 2017 18:24:21 +0000 Received: from mx0b-001b2d01.pphosted.com ([148.163.158.5] helo=mx0a-001b2d01.pphosted.com) by bombadil.infradead.org with esmtps (Exim 4.87 #1 (Red Hat Linux)) id 1ePueZ-00005h-69 for linux-arm-kernel@lists.infradead.org; Fri, 15 Dec 2017 18:24:19 +0000 Received: from pps.filterd (m0098420.ppops.net [127.0.0.1]) by mx0b-001b2d01.pphosted.com (8.16.0.21/8.16.0.21) with SMTP id vBFIIpM3070700 for ; Fri, 15 Dec 2017 13:23:40 -0500 Received: from e11.ny.us.ibm.com (e11.ny.us.ibm.com [129.33.205.201]) by mx0b-001b2d01.pphosted.com with ESMTP id 2evhk4wfrd-1 (version=TLSv1.2 cipher=AES256-SHA bits=256 verify=NOT) for ; Fri, 15 Dec 2017 13:23:39 -0500 Received: from localhost by e11.ny.us.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Fri, 15 Dec 2017 13:23:39 -0500 Received: from b01cxnp22034.gho.pok.ibm.com (9.57.198.24) by e11.ny.us.ibm.com (146.89.104.198) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; Fri, 15 Dec 2017 13:23:35 -0500 Received: from b01ledav003.gho.pok.ibm.com (b01ledav003.gho.pok.ibm.com [9.57.199.108]) by b01cxnp22034.gho.pok.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id vBFINZL653411918; Fri, 15 Dec 2017 18:23:35 GMT Received: from b01ledav003.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id DC7E0B2054; Fri, 15 Dec 2017 13:20:40 -0500 (EST) Received: from paulmck-ThinkPad-W541 (unknown [9.85.141.99]) by b01ledav003.gho.pok.ibm.com (Postfix) with ESMTP id 8C05FB2052; Fri, 15 Dec 2017 13:20:40 -0500 (EST) Received: by paulmck-ThinkPad-W541 (Postfix, from userid 1000) id 5406716C1773; Fri, 15 Dec 2017 10:23:40 -0800 (PST) Date: Fri, 15 Dec 2017 10:23:40 -0800 From: "Paul E. McKenney" To: Fabio Estevam Subject: Re: WARNING: suspicious RCU usage References: <20171210193438.GP10595@n2100.armlinux.org.uk> <20171210213930.GL7829@linux.vnet.ibm.com> <20171212164900.GA6673@linux.vnet.ibm.com> <20171212173450.GD10595@n2100.armlinux.org.uk> <20171213091245.GH10595@n2100.armlinux.org.uk> <20171215063817.GX7829@linux.vnet.ibm.com> <20171215155218.GB7829@linux.vnet.ibm.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20171215155218.GB7829@linux.vnet.ibm.com> User-Agent: Mutt/1.5.21 (2010-09-15) X-TM-AS-GCONF: 00 x-cbid: 17121518-2213-0000-0000-0000024B139E X-IBM-SpamModules-Scores: X-IBM-SpamModules-Versions: BY=3.00008207; HX=3.00000241; KW=3.00000007; PH=3.00000004; SC=3.00000244; SDB=6.00960675; UDB=6.00485935; IPR=6.00740694; BA=6.00005743; NDR=6.00000001; ZLA=6.00000005; ZF=6.00000009; ZB=6.00000000; ZP=6.00000000; ZH=6.00000000; ZU=6.00000002; MB=3.00018562; XFM=3.00000015; UTC=2017-12-15 18:23:38 X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 17121518-2214-0000-0000-0000587402BF Message-Id: <20171215182340.GA30443@linux.vnet.ibm.com> X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:, , definitions=2017-12-15_10:, , signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 impostorscore=0 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1709140000 definitions=main-1712150255 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20171215_102411_841638_D35BE2D8 X-CRM114-Status: GOOD ( 23.78 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: paulmck@linux.vnet.ibm.com Cc: Dong Aisheng , Will Deacon , Russell King - ARM Linux , Fabio Estevam , Peng Fan , Shawn Guo , "linux-arm-kernel@lists.infradead.org" Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Virus-Scanned: ClamAV using ClamSMTP On Fri, Dec 15, 2017 at 07:52:18AM -0800, Paul E. McKenney wrote: > On Fri, Dec 15, 2017 at 11:16:43AM -0200, Fabio Estevam wrote: > > Hi Paul, > > > > On Fri, Dec 15, 2017 at 4:38 AM, Paul E. McKenney > > wrote: > > > > > For your amusement, I have a patch below that takes a paranoid view of > > > the possible answers to these questions. This patch is untested and > > > probably does not even build. Plus its polling loop is quite naive. > > > > I tried to build it, but if fails to link: > > > > LD vmlinux.o > > MODPOST vmlinux.o > > arch/arm/kernel/smp.o: In function `__cpu_die': > > smp.c:(.text+0x44c): undefined reference to `__bad_xchg' > > Makefile:1024: recipe for target 'vmlinux' failed > > make: *** [vmlinux] Error 1 > > OK, I will need to make a better choice of atomic operation. > > Thank you for testing this! How about this one, also untested etc.? Thanx, Paul ------------------------------------------------------------------------ commit 66c038bfa64ff75e0fcdf6756f6225d4253f5a81 Author: Paul E. McKenney Date: Mon Dec 11 09:40:58 2017 -0800 ARM: CPU hotplug: Delegate complete() to surviving CPU The ARM implementation of arch_cpu_idle_dead() invokes complete(), but does so after RCU has stopped watching the outgoing CPU, which results in lockdep complaints because complete() invokes functions containing RCU readers. In addition, if the outgoing CPU really were to consume several seconds of its five-second allotted time, multiple RCU updates could complete, possibly giving the outgoing CPU an inconsistent view of the scheduler data structures on which complete() relies. This (untested, probably does not build) commit avoids this problem by polling the outgoing CPU. The polling strategy in this prototype patch is quite naive, with one jiffy between each poll and without any sort of adaptive spin phase. The key point is that the polling CPU uses atomic_dec_and_test(), which evicts the flag from the outgoing CPU's cache. The outgoing CPU simply does an atomic_set() of the value 1 which causes the next atomic_dec_and_test() to return true, and which also minimizes opportunities for other data to get pulled into the outgoing CPU's cache. This pulling of values from the outgoing CPU's cache is important because the outgoing CPU might be unceremoniously powered off before it has time to execute any code after the atomic_set(). Underflow is avoided because there can be at most 5,000 invocations of atomic_dec_and_test() for a given offline operation, and the counter is set back to zero each time. Reported-by: Peng Fan Reported-by: Russell King - ARM Linux Signed-off-by: Paul E. McKenney Cc: Russell King Cc: Ingo Molnar Cc: "Peter Zijlstra (Intel)" Cc: Michal Hocko Cc: Thomas Gleixner Cc: Fabio Estevam Cc: Signed-off-by: Paul E. McKenney diff --git a/arch/arm/kernel/smp.c b/arch/arm/kernel/smp.c index b4fbf00ee4ad..2fcffccf26ab 100644 --- a/arch/arm/kernel/smp.c +++ b/arch/arm/kernel/smp.c @@ -241,7 +241,7 @@ int __cpu_disable(void) return 0; } -static DECLARE_COMPLETION(cpu_died); +static atomic_t cpu_died; /* * called on the thread which is asking for a CPU to be shutdown - @@ -249,7 +249,17 @@ static DECLARE_COMPLETION(cpu_died); */ void __cpu_die(unsigned int cpu) { - if (!wait_for_completion_timeout(&cpu_died, msecs_to_jiffies(5000))) { + unsigned long deadline = jiffies + msecs_to_jiffies(5000); + char ret; + + while (time_before(jiffies, deadline)) { + ret = atomic_dec_and_test(&cpu_died); + if (ret) + break; + schedule_timeout_interruptible(1); + } + atomic_set(&cpu_died, 0); + if (!ret) { pr_err("CPU%u: cpu didn't die\n", cpu); return; } @@ -295,7 +305,7 @@ void arch_cpu_idle_dead(void) * this returns, power and/or clocks can be removed at any point * from this CPU and its cache by platform_cpu_kill(). */ - complete(&cpu_died); + atomic_set(&cpu_died, 1); /* * Ensure that the cache lines associated with that completion are