From patchwork Thu Aug 11 07:41:47 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xianting Tian X-Patchwork-Id: 12941217 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id A0F44C19F2D for ; Thu, 11 Aug 2022 07:42:51 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:List-Subscribe:List-Help: List-Post:List-Archive:List-Unsubscribe:List-Id:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=CtQ3hc3Ecc7fAwGSHET2Gx1JaLUUCbCYkGYn9W0wtQA=; b=esXkDY2kX3wNB7 MGqXARMxZaAcoOPPVLrEFgnRgupTvIsAUZnd5BQlvd0VbCEf0PNDN1q+KbnWgAzhiLwPX/5+Uwz01 pdEu+tQB1EXqk0++8FIKhpYf4rDO2OLYnr84/2G/PXAdk0+KTi9TCG6vPJ8W6+zIAQVHg0vwyp2cY Dm5Y/9QGk5q5rcJA51AxPjmCCHk4XMkLtOxbmHBjEt5fZNCGjXPg5LYTde9C8ixC0nHIExDjqeXL2 DmQzWLMz1EKLLKb6+16U5vXUaV3p/dPVl9uwejb60KvqDf7GNVz2pJ9JaAYIVJrN/Jzk5poGpQR6y LfJn1DwfGYfl4DzhL7IA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1oM2py-009mdT-Er; Thu, 11 Aug 2022 07:42:38 +0000 Received: from out199-11.us.a.mail.aliyun.com ([47.90.199.11]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1oM2pQ-009lhB-FZ; Thu, 11 Aug 2022 07:42:16 +0000 X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R801e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=ay29a033018045192;MF=xianting.tian@linux.alibaba.com;NM=1;PH=DS;RN=23;SR=0;TI=SMTPD_---0VLyDALq_1660203717; Received: from localhost(mailfrom:xianting.tian@linux.alibaba.com fp:SMTPD_---0VLyDALq_1660203717) by smtp.aliyun-inc.com; Thu, 11 Aug 2022 15:41:58 +0800 From: Xianting Tian To: paul.walmsley@sifive.com, palmer@dabbelt.com, aou@eecs.berkeley.edu, anup@brainfault.org, heiko@sntech.de, guoren@kernel.org, mick@ics.forth.gr, alexandre.ghiti@canonical.com, bhe@redhat.com, vgoyal@redhat.com, dyoung@redhat.com, corbet@lwn.net, Conor.Dooley@microchip.com Cc: kexec@lists.infradead.org, linux-doc@vger.kernel.org, linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, crash-utility@redhat.com, heinrich.schuchardt@canonical.com, k-hagio-ab@nec.com, hschauhan@nulltrace.org, yixun.lan@gmail.com, Xianting Tian Subject: [PATCH V6 3/6] RISC-V: Fixup schedule out issue in machine_crash_shutdown() Date: Thu, 11 Aug 2022 15:41:47 +0800 Message-Id: <20220811074150.3020189-4-xianting.tian@linux.alibaba.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20220811074150.3020189-1-xianting.tian@linux.alibaba.com> References: <20220811074150.3020189-1-xianting.tian@linux.alibaba.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220811_004213_973128_5437DFAC X-CRM114-Status: GOOD ( 11.90 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org Current task of executing crash kexec will be schedule out when panic is triggered by RCU Stall, as it needs to wait rcu completion. It lead to inability to enter the crash system. The implementation of machine_crash_shutdown() is non-standard for RISC-V according to other Arch's implementation(eg, x86, arm64), we need to send IPI to stop secondary harts. [224521.877268] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: [224521.883471] rcu: 0-...0: (3 GPs behind) idle=cfa/0/0x1 softirq=3968793/3968793 fqs=2495 [224521.891742] (detected by 2, t=5255 jiffies, g=60855593, q=328) [224521.897754] Task dump for CPU 0: [224521.901074] task:swapper/0 state:R running task stack: 0 pid: 0 ppid: 0 flags:0x00000008 [224521.911090] Call Trace: [224521.913638] [] __schedule+0x208/0x5ea [224521.918957] Kernel panic - not syncing: RCU Stall [224521.923773] bad: scheduling from the idle thread! [224521.928571] CPU: 2 PID: 0 Comm: swapper/2 Kdump: loaded Tainted: G O 5.10.113-yocto-standard #1 [224521.938658] Call Trace: [224521.941200] [] walk_stackframe+0x0/0xaa [224521.946689] [] show_stack+0x32/0x3e [224521.951830] [] dump_stack_lvl+0x7e/0xa2 [224521.957317] [] dump_stack+0x14/0x1c [224521.962459] [] dequeue_task_idle+0x2c/0x40 [224521.968207] [] __schedule+0x41e/0x5ea [224521.973520] [] schedule+0x34/0xe4 [224521.978487] [] schedule_timeout+0xc6/0x170 [224521.984234] [] wait_for_completion+0x98/0xf2 [224521.990157] [] __wait_rcu_gp+0x148/0x14a [224521.995733] [] synchronize_rcu+0x5c/0x66 [224522.001307] [] rcu_sync_enter+0x54/0xe6 [224522.006795] [] percpu_down_write+0x32/0x11c [224522.012629] [] _cpu_down+0x92/0x21a [224522.017771] [] smp_shutdown_nonboot_cpus+0x90/0x118 [224522.024299] [] machine_crash_shutdown+0x30/0x4a [224522.030483] [] __crash_kexec+0x62/0xa6 [224522.035884] [] panic+0xfa/0x2b6 [224522.040678] [] rcu_sched_clock_irq+0xc26/0xcb8 [224522.046774] [] update_process_times+0x62/0x8a [224522.052785] [] tick_sched_timer+0x9e/0x102 [224522.058533] [] __hrtimer_run_queues+0x16a/0x318 [224522.064716] [] hrtimer_interrupt+0xd4/0x228 [224522.070551] [] riscv_timer_interrupt+0x3c/0x48 [224522.076646] [] handle_percpu_devid_irq+0xb0/0x24c [224522.083004] [] __handle_domain_irq+0xa8/0x122 [224522.089014] [] riscv_intc_irq+0x38/0x60 [224522.094501] [] ret_from_exception+0x0/0xc [224522.100161] [] rcu_eqs_enter.constprop.0+0x8c/0xb8 With the patch, it can enter crash system when RCU Stall occur. Fixes: e53d28180d4d ("RISC-V: Add kdump support") Signed-off-by: Xianting Tian --- arch/riscv/kernel/machine_kexec.c | 26 ++++++++++++++++++++++---- 1 file changed, 22 insertions(+), 4 deletions(-) diff --git a/arch/riscv/kernel/machine_kexec.c b/arch/riscv/kernel/machine_kexec.c index 86d1b5f9dfb5..ee79e6839b86 100644 --- a/arch/riscv/kernel/machine_kexec.c +++ b/arch/riscv/kernel/machine_kexec.c @@ -138,19 +138,37 @@ void machine_shutdown(void) #endif } +/* Override the weak function in kernel/panic.c */ +void crash_smp_send_stop(void) +{ + static int cpus_stopped; + + /* + * This function can be called twice in panic path, but obviously + * we execute this only once. + */ + if (cpus_stopped) + return; + + smp_send_stop(); + cpus_stopped = 1; +} + /* * machine_crash_shutdown - Prepare to kexec after a kernel crash * * This function is called by crash_kexec just before machine_kexec - * below and its goal is similar to machine_shutdown, but in case of - * a kernel crash. Since we don't handle such cases yet, this function - * is empty. + * and its goal is to shutdown non-crashing cpus and save registers. */ void machine_crash_shutdown(struct pt_regs *regs) { + local_irq_disable(); + + /* shutdown non-crashing cpus */ + crash_smp_send_stop(); + crash_save_cpu(regs, smp_processor_id()); - machine_shutdown(); pr_info("Starting crashdump kernel...\n"); }