From patchwork Thu May 15 04:11:30 2014 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ashwin Chaugule X-Patchwork-Id: 4179121 Return-Path: X-Original-To: patchwork-linux-arm@patchwork.kernel.org Delivered-To: patchwork-parsemail@patchwork1.web.kernel.org Received: from mail.kernel.org (mail.kernel.org [198.145.19.201]) by patchwork1.web.kernel.org (Postfix) with ESMTP id E73AC9F271 for ; Thu, 15 May 2014 04:15:38 +0000 (UTC) Received: from mail.kernel.org (localhost [127.0.0.1]) by mail.kernel.org (Postfix) with ESMTP id C73DE20377 for ; Thu, 15 May 2014 04:15:37 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.9]) (using TLSv1.2 with cipher DHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id C5F212035D for ; Thu, 15 May 2014 04:15:36 +0000 (UTC) Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.80.1 #2 (Red Hat Linux)) id 1Wkn1t-00066C-Jl; Thu, 15 May 2014 04:12:25 +0000 Received: from mail-qc0-f173.google.com ([209.85.216.173]) by bombadil.infradead.org with esmtps (Exim 4.80.1 #2 (Red Hat Linux)) id 1Wkn1d-00062m-Rh for linux-arm-kernel@lists.infradead.org; Thu, 15 May 2014 04:12:11 +0000 Received: by mail-qc0-f173.google.com with SMTP id i8so873258qcq.18 for ; Wed, 14 May 2014 21:11:48 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=IU/Ya91BvJ3TMKxEDpowFD4YqrL0+2Q+MK8jVPm+TrE=; b=AEHWfZ0d+hhUdNanOreWKLNKzaCkpZNZgMx7BpNLXRUQIMzqmpIAxV2nfZBLxKio2C qLHMpcOsKqA/b2jRzlBRLxYh3NAK/fX3CyCgSa0DZ/UyhVEMKuyPGZ8ET2s5Tyf9LCcj Oav9ix0tZvvHWxehnrHNcBD+tNnexqCJIOg60KHJdjcFH7mTRE3exKv0vDmfJaIiQN8A euM1AcQfy7EHv/PgaWGXgKUdtu8N+x1lRDWFbanu3Gif0/EHOzgp8NkWcvxyzOxKPi5B LSJlP4ZkY2Di/UN9hyGnBjV13Sm+wbjrCnt431EmIUUNcIOrGTZIO8nSo0gaj/BS8yQL 2vTA== X-Gm-Message-State: ALoCoQluLBXZYkG7AUpTJk6ibYGqDoZTWYu6zafjNVLZEVAzF5IkJjGbKymhjjLbytItjZyGWLHt X-Received: by 10.140.44.72 with SMTP id f66mr11164202qga.78.1400127108003; Wed, 14 May 2014 21:11:48 -0700 (PDT) Received: from localhost.localdomain (cpe-098-027-049-158.nc.res.rr.com. [98.27.49.158]) by mx.google.com with ESMTPSA id k9sm6107147qaj.25.2014.05.14.21.11.46 for (version=TLSv1.1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Wed, 14 May 2014 21:11:47 -0700 (PDT) From: Ashwin Chaugule To: Catalin.Marinas@arm.com Subject: [PATCH v10 3/3] ARM: Check if a CPU has gone offline Date: Thu, 15 May 2014 00:11:30 -0400 Message-Id: <1400127090-1320-4-git-send-email-ashwin.chaugule@linaro.org> X-Mailer: git-send-email 1.8.3.2 In-Reply-To: <1400127090-1320-1-git-send-email-ashwin.chaugule@linaro.org> References: <1400127090-1320-1-git-send-email-ashwin.chaugule@linaro.org> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20140514_211210_072918_291F8D65 X-CRM114-Status: GOOD ( 21.26 ) X-Spam-Score: -0.7 (/) Cc: mark.rutland@arm.com, rob.herring@linaro.org, anup.patel@linaro.org, linaro-acpi@lists.linaro.org, Marc.Zyngier@arm.com, patches@linaro.org, Ashwin Chaugule , linux-arm-kernel@lists.infradead.org, christoffer.dall@linaro.org X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Spam-Status: No, score=-2.5 required=5.0 tests=BAYES_00,RP_MATCHES_RCVD, UNPARSEABLE_RELAY autolearn=unavailable version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on mail.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP PSCIv0.2 adds a new function called AFFINITY_INFO, which can be used to query if a specified CPU has actually gone offline. Calling this function via cpu_kill ensures that a CPU has quiesced after a call to cpu_die. This helps prevent the CPU from doing arbitrary bad things when data or instructions are clobbered (as happens with kexec) in the window between a CPU announcing that it is dead and said CPU leaving the kernel. Signed-off-by: Ashwin Chaugule Signed-off-by: Mark Rutland Reviewed-by: Rob Herring Acked-by: Catalin Marinas --- arch/arm/kernel/psci_smp.c | 33 +++++++++++++++++++++++++++++++++ arch/arm64/include/asm/cpu_ops.h | 2 ++ arch/arm64/kernel/psci.c | 31 +++++++++++++++++++++++++++++++ arch/arm64/kernel/smp.c | 22 ++++++++++++++++++++++ 4 files changed, 88 insertions(+) diff --git a/arch/arm/kernel/psci_smp.c b/arch/arm/kernel/psci_smp.c index 570a48c..28a1db4 100644 --- a/arch/arm/kernel/psci_smp.c +++ b/arch/arm/kernel/psci_smp.c @@ -16,6 +16,8 @@ #include #include #include +#include +#include #include #include @@ -66,6 +68,36 @@ void __ref psci_cpu_die(unsigned int cpu) /* We should never return */ panic("psci: cpu %d failed to shutdown\n", cpu); } + +int __ref psci_cpu_kill(unsigned int cpu) +{ + int err, i; + + if (!psci_ops.affinity_info) + return 1; + /* + * cpu_kill could race with cpu_die and we can + * potentially end up declaring this cpu undead + * while it is dying. So, try again a few times. + */ + + for (i = 0; i < 10; i++) { + err = psci_ops.affinity_info(cpu_logical_map(cpu), 0); + if (err == PSCI_0_2_AFFINITY_LEVEL_OFF) { + pr_info("CPU%d killed.\n", cpu); + return 1; + } + + msleep(10); + pr_info("Retrying again to check for CPU kill\n"); + } + + pr_warn("CPU%d may not have shut down cleanly (AFFINITY_INFO reports %d)\n", + cpu, err); + /* Make platform_cpu_kill() fail. */ + return 0; +} + #endif bool __init psci_smp_available(void) @@ -78,5 +110,6 @@ struct smp_operations __initdata psci_smp_ops = { .smp_boot_secondary = psci_boot_secondary, #ifdef CONFIG_HOTPLUG_CPU .cpu_die = psci_cpu_die, + .cpu_kill = psci_cpu_kill, #endif }; diff --git a/arch/arm64/include/asm/cpu_ops.h b/arch/arm64/include/asm/cpu_ops.h index 1524130..d7b4b38 100644 --- a/arch/arm64/include/asm/cpu_ops.h +++ b/arch/arm64/include/asm/cpu_ops.h @@ -39,6 +39,7 @@ struct device_node; * from the cpu to be killed. * @cpu_die: Makes a cpu leave the kernel. Must not fail. Called from the * cpu being killed. + * @cpu_kill: Ensures a cpu has left the kernel. Called from another cpu. * @cpu_suspend: Suspends a cpu and saves the required context. May fail owing * to wrong parameters or error conditions. Called from the * CPU being suspended. Must be called with IRQs disabled. @@ -52,6 +53,7 @@ struct cpu_operations { #ifdef CONFIG_HOTPLUG_CPU int (*cpu_disable)(unsigned int cpu); void (*cpu_die)(unsigned int cpu); + int (*cpu_kill)(unsigned int cpu); #endif #ifdef CONFIG_ARM64_CPU_SUSPEND int (*cpu_suspend)(unsigned long); diff --git a/arch/arm64/kernel/psci.c b/arch/arm64/kernel/psci.c index 90df6e6..9e9798f 100644 --- a/arch/arm64/kernel/psci.c +++ b/arch/arm64/kernel/psci.c @@ -20,6 +20,7 @@ #include #include #include +#include #include #include @@ -403,6 +404,35 @@ static void cpu_psci_cpu_die(unsigned int cpu) pr_crit("unable to power off CPU%u (%d)\n", cpu, ret); } + +static int cpu_psci_cpu_kill(unsigned int cpu) +{ + int err, i; + + if (!psci_ops.affinity_info) + return 1; + /* + * cpu_kill could race with cpu_die and we can + * potentially end up declaring this cpu undead + * while it is dying. So, try again a few times. + */ + + for (i = 0; i < 10; i++) { + err = psci_ops.affinity_info(cpu_logical_map(cpu), 0); + if (err == PSCI_0_2_AFFINITY_LEVEL_OFF) { + pr_info("CPU%d killed.\n", cpu); + return 1; + } + + msleep(10); + pr_info("Retrying again to check for CPU kill\n"); + } + + pr_warn("CPU%d may not have shut down cleanly (AFFINITY_INFO reports %d)\n", + cpu, err); + /* Make op_cpu_kill() fail. */ + return 0; +} #endif const struct cpu_operations cpu_psci_ops = { @@ -413,6 +443,7 @@ const struct cpu_operations cpu_psci_ops = { #ifdef CONFIG_HOTPLUG_CPU .cpu_disable = cpu_psci_cpu_disable, .cpu_die = cpu_psci_cpu_die, + .cpu_kill = cpu_psci_cpu_kill, #endif }; diff --git a/arch/arm64/kernel/smp.c b/arch/arm64/kernel/smp.c index f0a141d..c3cb160 100644 --- a/arch/arm64/kernel/smp.c +++ b/arch/arm64/kernel/smp.c @@ -228,6 +228,19 @@ int __cpu_disable(void) return 0; } +static int op_cpu_kill(unsigned int cpu) +{ + /* + * If we have no means of synchronising with the dying CPU, then assume + * that it is really dead. We can only wait for an arbitrary length of + * time and hope that it's dead, so let's skip the wait and just hope. + */ + if (!cpu_ops[cpu]->cpu_kill) + return 1; + + return cpu_ops[cpu]->cpu_kill(cpu); +} + static DECLARE_COMPLETION(cpu_died); /* @@ -241,6 +254,15 @@ void __cpu_die(unsigned int cpu) return; } pr_notice("CPU%u: shutdown\n", cpu); + + /* + * Now that the dying CPU is beyond the point of no return w.r.t. + * in-kernel synchronisation, try to get the firwmare to help us to + * verify that it has really left the kernel before we consider + * clobbering anything it might still be using. + */ + if (!op_cpu_kill(cpu)) + pr_warn("CPU%d may not have shut down cleanly\n", cpu); } /*