From patchwork Fri Nov 11 19:49:56 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joel Fernandes X-Patchwork-Id: 13040789 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id DFE26C4332F for ; Fri, 11 Nov 2022 20:58:33 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 6313910E8CC; Fri, 11 Nov 2022 20:58:25 +0000 (UTC) Received: from mail-qt1-x833.google.com (mail-qt1-x833.google.com [IPv6:2607:f8b0:4864:20::833]) by gabe.freedesktop.org (Postfix) with ESMTPS id BC21510E0CE for ; Fri, 11 Nov 2022 19:50:15 +0000 (UTC) Received: by mail-qt1-x833.google.com with SMTP id h24so3246435qta.9 for ; Fri, 11 Nov 2022 11:50:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=YU/dVE0ERRRiwLc0HKRhu2fVlyQiYjgDvyxIAHOZXz8=; b=UTwWvPMR4j2f2p7up8Le/KvMZlwgGMWDnXQ8RcwwGzzkoHuyjeJwmie5C9GVXHQOeM 4nOE7CQbS2Rp5oNTgauj01OBG+ZZjUdaZ0z0IfnaKX4HNjv6IZ2fHaOWlPCl9B5mridB 2dt9pYevIUdALZ+hmKfj0ybst/f34okv6UyGw= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=YU/dVE0ERRRiwLc0HKRhu2fVlyQiYjgDvyxIAHOZXz8=; b=eu0hfSscfIaPGi1eW6PcJS6A8XSe43FCIRhYMrGrWLq6SIuTp5jKBjOKkulbpkC4Eq olG/G2J3RDIxmi4pPqxcDWxEJdzoQhdNkZxU0aP7oh0zjIHldz/1a/xebZiRKLeLxCdx bs6QFOsWdRbpvb8A+F3C5JJ3NdZV4EkP4kvQmTdsmEeI2XMus3Cy6ndphRaL8qy8HJYu yL+5ePPg0M+OxeeZIxMgMn6uvLlemmM9GMOhijv1AkuRQsi6gFIiz2fa6Vlkjf2dLk+/ 97GiQicT7BDF7kcZXskfmtFH3mbMIhYG+w0vts8nPE9StGMi/X/WutNBIHHc+C0RheMG BXzg== X-Gm-Message-State: ANoB5pk6WYaREfOyp2apV8FB9Lgs3BAbbpPYqH4mbw6YhSHeqcKnecFk u/nsS7yGvLVsVqUdfD/zVjCwiA== X-Google-Smtp-Source: AA0mqf4f379m8qQ4NDG8WIoePs17+iR7zz+GPC8xaMp8RokfgQW72ZP6ejmf+n05qXiKrn00RS9fgw== X-Received: by 2002:a05:622a:5083:b0:3a5:404:4c36 with SMTP id fp3-20020a05622a508300b003a504044c36mr2767678qtb.653.1668196214671; Fri, 11 Nov 2022 11:50:14 -0800 (PST) Received: from joelboxx.c.googlers.com.com (228.221.150.34.bc.googleusercontent.com. [34.150.221.228]) by smtp.gmail.com with ESMTPSA id y11-20020a05620a25cb00b006ef1a8f1b81sm1985277qko.5.2022.11.11.11.50.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 11 Nov 2022 11:50:13 -0800 (PST) From: "Joel Fernandes (Google)" To: linux-kernel@vger.kernel.org Subject: [PATCH 1/2] adreno: Shutdown the GPU properly Date: Fri, 11 Nov 2022 19:49:56 +0000 Message-Id: <20221111194957.4046771-1-joel@joelfernandes.org> X-Mailer: git-send-email 2.38.1.493.g58b659f92b-goog MIME-Version: 1.0 X-Mailman-Approved-At: Fri, 11 Nov 2022 20:58:22 +0000 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Rob Clark , Emma Anholt , Akhil P Oommen , freedreno@lists.freedesktop.org, linux-arm-msm@vger.kernel.org, Ricardo Ribalda , Vladimir Lypak , Abhinav Kumar , Steven Rostedt , Dmitry Baryshkov , dri-devel@lists.freedesktop.org, Ross Zwisler , "Joel Fernandes \(Google\)" , Sean Paul Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" During kexec on ARM device, we notice that device_shutdown() only calls pm_runtime_force_suspend() while shutting down the GPU. This means the GPU kthread is still running and further, there maybe active submits. This causes all kinds of issues during a kexec reboot: Warning from shutdown path: [ 292.509662] WARNING: CPU: 0 PID: 6304 at [...] adreno_runtime_suspend+0x3c/0x44 [ 292.509863] Hardware name: Google Lazor (rev3 - 8) with LTE (DT) [ 292.509872] pstate: 80400009 (Nzcv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 292.509881] pc : adreno_runtime_suspend+0x3c/0x44 [ 292.509891] lr : pm_generic_runtime_suspend+0x30/0x44 [ 292.509905] sp : ffffffc014473bf0 [...] [ 292.510043] Call trace: [ 292.510051] adreno_runtime_suspend+0x3c/0x44 [ 292.510061] pm_generic_runtime_suspend+0x30/0x44 [ 292.510071] pm_runtime_force_suspend+0x54/0xc8 [ 292.510081] adreno_shutdown+0x1c/0x28 [ 292.510090] platform_shutdown+0x2c/0x38 [ 292.510104] device_shutdown+0x158/0x210 [ 292.510119] kernel_restart_prepare+0x40/0x4c And here from GPU kthread, an SError OOPs: [ 192.648789] el1h_64_error+0x7c/0x80 [ 192.648812] el1_interrupt+0x20/0x58 [ 192.648833] el1h_64_irq_handler+0x18/0x24 [ 192.648854] el1h_64_irq+0x7c/0x80 [ 192.648873] local_daif_inherit+0x10/0x18 [ 192.648900] el1h_64_sync_handler+0x48/0xb4 [ 192.648921] el1h_64_sync+0x7c/0x80 [ 192.648941] a6xx_gmu_set_oob+0xbc/0x1fc [ 192.648968] a6xx_hw_init+0x44/0xe38 [ 192.648991] msm_gpu_hw_init+0x48/0x80 [ 192.649013] msm_gpu_submit+0x5c/0x1a8 [ 192.649034] msm_job_run+0xb0/0x11c [ 192.649058] drm_sched_main+0x170/0x434 [ 192.649086] kthread+0x134/0x300 [ 192.649114] ret_from_fork+0x10/0x20 Fix by calling adreno_system_suspend() in the device_shutdown() path. Cc: Rob Clark Cc: Steven Rostedt Cc: Ricardo Ribalda Cc: Ross Zwisler Signed-off-by: Joel Fernandes (Google) --- drivers/gpu/drm/msm/adreno/adreno_device.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/drivers/gpu/drm/msm/adreno/adreno_device.c b/drivers/gpu/drm/msm/adreno/adreno_device.c index 24b489b6129a..f0cff62812c3 100644 --- a/drivers/gpu/drm/msm/adreno/adreno_device.c +++ b/drivers/gpu/drm/msm/adreno/adreno_device.c @@ -607,9 +607,12 @@ static int adreno_remove(struct platform_device *pdev) return 0; } +static int adreno_system_suspend(struct device *dev); static void adreno_shutdown(struct platform_device *pdev) { - pm_runtime_force_suspend(&pdev->dev); + struct msm_gpu *gpu = dev_to_gpu(&pdev->dev); + + WARN_ON_ONCE(adreno_system_suspend(&pdev->dev)); } static const struct of_device_id dt_match[] = { From patchwork Fri Nov 11 19:49:57 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joel Fernandes X-Patchwork-Id: 13040788 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 2DE8FC433FE for ; Fri, 11 Nov 2022 20:58:25 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 0366810E8C6; Fri, 11 Nov 2022 20:58:24 +0000 (UTC) Received: from mail-qt1-x82d.google.com (mail-qt1-x82d.google.com [IPv6:2607:f8b0:4864:20::82d]) by gabe.freedesktop.org (Postfix) with ESMTPS id D50AF10E8AD for ; Fri, 11 Nov 2022 19:50:16 +0000 (UTC) Received: by mail-qt1-x82d.google.com with SMTP id l2so3242618qtq.11 for ; Fri, 11 Nov 2022 11:50:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=1KH+9DnybePERv04muJMWfidDb3fPimLOhb4bQHi9hY=; b=RHjf12jHGHZlV6LxazAvl/5fZwStrXPkGDsdKj20bcVMeSXNquWMtMx/Bsdq/o5JYm eAfmViLAt2OLTkbQbsClDh4FCyBu1w5nu3AzI7kME+tNrhg4OayMskXk5hBLP5WSLQYN vTRUqiZVfCQevVB5j1H2sbOljVncceXkIvv5Y= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=1KH+9DnybePERv04muJMWfidDb3fPimLOhb4bQHi9hY=; b=o6cqA1ICD81wV0aTSJrc1BetkJCwuQ0YfyW/S6ALKcrQkaHanMcEw1SypzPZJglAeS 7li6gm12jvIF528Ukx1uRhvAgFsREPG89bNwis+cSEmfyfnpr9+PDfu8mxl/NthFq5Jo aSFagP0cfdEJNmiSybdt7vD/5dbSkNryNZYj9SUeZS6z3I2ovj6kt4mXVAStt+6C6jWr niekYmvFnAD5zMFq4dXmSzjHRfgf5K6owfzCbR6D13DQh20kcBh18lAzJFEmgw35MTZ4 8dM6FgmEE/Y9wdkTgm/rgg1O1JQbQwjWajCbtnPwBQ5wlyLJcbSOxEI3kVczsWVEKevZ jvFg== X-Gm-Message-State: ANoB5pnbNYDWS0oXuPee3tXtIjipHARBuQWJC84gbaseGClG3G/zrqzg om0+Z65Kl7/H+2ICopUxY/K55Q== X-Google-Smtp-Source: AA0mqf5c6zKhevDvTLNf9Ctrgvya3+388MKGvFGBoi33TAcyrdTqZHtjc+V5gDh69jQ6VTNdJcrmDQ== X-Received: by 2002:ac8:44d7:0:b0:39c:c7ba:4af4 with SMTP id b23-20020ac844d7000000b0039cc7ba4af4mr2862729qto.99.1668196215864; Fri, 11 Nov 2022 11:50:15 -0800 (PST) Received: from joelboxx.c.googlers.com.com (228.221.150.34.bc.googleusercontent.com. [34.150.221.228]) by smtp.gmail.com with ESMTPSA id y11-20020a05620a25cb00b006ef1a8f1b81sm1985277qko.5.2022.11.11.11.50.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 11 Nov 2022 11:50:15 -0800 (PST) From: "Joel Fernandes (Google)" To: linux-kernel@vger.kernel.org Subject: [PATCH 2/2] adreno: Detect shutdown during get_param() Date: Fri, 11 Nov 2022 19:49:57 +0000 Message-Id: <20221111194957.4046771-2-joel@joelfernandes.org> X-Mailer: git-send-email 2.38.1.493.g58b659f92b-goog In-Reply-To: <20221111194957.4046771-1-joel@joelfernandes.org> References: <20221111194957.4046771-1-joel@joelfernandes.org> MIME-Version: 1.0 X-Mailman-Approved-At: Fri, 11 Nov 2022 20:58:22 +0000 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Rob Clark , Emma Anholt , Akhil P Oommen , freedreno@lists.freedesktop.org, linux-arm-msm@vger.kernel.org, Ricardo Ribalda , Vladimir Lypak , Abhinav Kumar , Steven Rostedt , Dmitry Baryshkov , dri-devel@lists.freedesktop.org, Ross Zwisler , "Joel Fernandes \(Google\)" , Sean Paul Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" Even though the GPU is shut down, during kexec reboot we can have userspace still running. This is especially true if KEXEC_JUMP is not enabled, because we do not freeze userspace in this case. To prevent crashes, track that the GPU is shutdown and prevent get_param() from accessing GPU resources if we find it shutdown. This fixes the following crash during kexec reboot on an ARM64 device with adreno GPU: [ 292.534314] Kernel panic - not syncing: Asynchronous SError Interrupt [ 292.534323] Hardware name: Google Lazor (rev3 - 8) with LTE (DT) [ 292.534326] Call trace: [ 292.534328] dump_backtrace+0x0/0x1d4 [ 292.534337] show_stack+0x20/0x2c [ 292.534342] dump_stack_lvl+0x60/0x78 [ 292.534347] dump_stack+0x18/0x38 [ 292.534352] panic+0x148/0x3b0 [ 292.534357] nmi_panic+0x80/0x94 [ 292.534364] arm64_serror_panic+0x70/0x7c [ 292.534369] do_serror+0x0/0x7c [ 292.534372] do_serror+0x54/0x7c [ 292.534377] el1h_64_error_handler+0x34/0x4c [ 292.534381] el1h_64_error+0x7c/0x80 [ 292.534386] el1_interrupt+0x20/0x58 [ 292.534389] el1h_64_irq_handler+0x18/0x24 [ 292.534395] el1h_64_irq+0x7c/0x80 [ 292.534399] local_daif_inherit+0x10/0x18 [ 292.534405] el1h_64_sync_handler+0x48/0xb4 [ 292.534410] el1h_64_sync+0x7c/0x80 [ 292.534414] a6xx_gmu_set_oob+0xbc/0x1fc [ 292.534422] a6xx_get_timestamp+0x40/0xb4 [ 292.534426] adreno_get_param+0x12c/0x1e0 [ 292.534433] msm_ioctl_get_param+0x64/0x70 [ 292.534440] drm_ioctl_kernel+0xe8/0x158 [ 292.534448] drm_ioctl+0x208/0x320 [ 292.534453] __arm64_sys_ioctl+0x98/0xd0 [ 292.534461] invoke_syscall+0x4c/0x118 [ 292.534467] el0_svc_common+0x98/0x104 [ 292.534473] do_el0_svc+0x30/0x80 [ 292.534478] el0_svc+0x20/0x50 [ 292.534481] el0t_64_sync_handler+0x78/0x108 [ 292.534485] el0t_64_sync+0x1a4/0x1a8 [ 292.534632] Kernel Offset: 0x1a5f800000 from 0xffffffc008000000 [ 292.534635] PHYS_OFFSET: 0x80000000 [ 292.534638] CPU features: 0x40018541,a3300e42 [ 292.534644] Memory Limit: none Cc: Rob Clark Cc: Steven Rostedt Cc: Ricardo Ribalda Cc: Ross Zwisler Signed-off-by: Joel Fernandes (Google) --- drivers/gpu/drm/msm/adreno/adreno_device.c | 1 + drivers/gpu/drm/msm/adreno/adreno_gpu.c | 2 +- drivers/gpu/drm/msm/msm_gpu.h | 3 +++ 3 files changed, 5 insertions(+), 1 deletion(-) diff --git a/drivers/gpu/drm/msm/adreno/adreno_device.c b/drivers/gpu/drm/msm/adreno/adreno_device.c index f0cff62812c3..03d912dc0130 100644 --- a/drivers/gpu/drm/msm/adreno/adreno_device.c +++ b/drivers/gpu/drm/msm/adreno/adreno_device.c @@ -612,6 +612,7 @@ static void adreno_shutdown(struct platform_device *pdev) { struct msm_gpu *gpu = dev_to_gpu(&pdev->dev); + gpu->is_shutdown = true; WARN_ON_ONCE(adreno_system_suspend(&pdev->dev)); } diff --git a/drivers/gpu/drm/msm/adreno/adreno_gpu.c b/drivers/gpu/drm/msm/adreno/adreno_gpu.c index 382fb7f9e497..6903c6892469 100644 --- a/drivers/gpu/drm/msm/adreno/adreno_gpu.c +++ b/drivers/gpu/drm/msm/adreno/adreno_gpu.c @@ -251,7 +251,7 @@ int adreno_get_param(struct msm_gpu *gpu, struct msm_file_private *ctx, struct adreno_gpu *adreno_gpu = to_adreno_gpu(gpu); /* No pointer params yet */ - if (*len != 0) + if (*len != 0 || gpu->is_shutdown) return -EINVAL; switch (param) { diff --git a/drivers/gpu/drm/msm/msm_gpu.h b/drivers/gpu/drm/msm/msm_gpu.h index ff911e7305ce..f18b0a91442b 100644 --- a/drivers/gpu/drm/msm/msm_gpu.h +++ b/drivers/gpu/drm/msm/msm_gpu.h @@ -214,6 +214,9 @@ struct msm_gpu { /* does gpu need hw_init? */ bool needs_hw_init; + /* is the GPU shutdown? */ + bool is_shutdown; + /** * global_faults: number of GPU hangs not attributed to a particular * address space