From patchwork Mon Jan 9 22:25:47 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joel Fernandes X-Patchwork-Id: 13094432 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id D41A8C54EBD for ; Mon, 9 Jan 2023 22:25:58 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 15FFF10E0AE; Mon, 9 Jan 2023 22:25:58 +0000 (UTC) Received: from mail-qv1-xf30.google.com (mail-qv1-xf30.google.com [IPv6:2607:f8b0:4864:20::f30]) by gabe.freedesktop.org (Postfix) with ESMTPS id B3ED710E0AE for ; Mon, 9 Jan 2023 22:25:55 +0000 (UTC) Received: by mail-qv1-xf30.google.com with SMTP id m12so5263197qvt.9 for ; Mon, 09 Jan 2023 14:25:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=YkeaBylOW4jhgYHb/D3fun9Z5f+LSYx+wsvJq/ZnfRM=; b=tsF++OQqIotu0iXwaD5qqq9cRqzAPro8aeajLf0eNZVSMxhvR5A0/fLalEPA9WRwSA gYg1NfZBtQX8Bt9uQt0xzQzzANMMGxkFphnfIdvzuMoWxrvH92BvKVUiRXJgoz6bx6Os OIjE72HQgYaANJPM5Fm4H/rrAah3StrbdyFes= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=YkeaBylOW4jhgYHb/D3fun9Z5f+LSYx+wsvJq/ZnfRM=; b=1SDt96IzVff6P4bQc/s++i3IPevk0LXc38wKEnbqP3nrmBV5z1gW+Umfrtu3CdK6LV kCFzkzYr2Liey/SYfo4KY2MxlptVc6rkczGwr4tNUdxhHD29JOYOv3Yb5ti2RufYGywZ YRtRW0COPcRjv1PIr3gnSn8BJ4zXdG0+ZvHJNHV4sI9bGxKfUAooP0XVnbuvwPAdD3rk ju7L8onuOWOH9IBdaDPyBJs+ssIJDpIVjRF3zS0cLm/oB3/1NoDF9869je7uxrwyhGUv 5tGIpLE8+hBKVjMaeht8Lw9zYAw+tfOwM8dLvi6A7qCikOiSgzTQvtTA/2bzbR6JRGUU J91A== X-Gm-Message-State: AFqh2kpmHBxoiWn+a9YbNiEgkDQa7qa9voPvwj8kO63wrHcIVQJ96gc8 prZzp4PBKwVyOoR9E9gpJl/DmQ== X-Google-Smtp-Source: AMrXdXt5WcdGViw3hX3Ad5LFg/Chk9zTGtRgfE3ElNQE7MvJ1vaJebzVyAG3hxHZ6x4rwSWjeVvPUQ== X-Received: by 2002:a05:6214:322:b0:532:2409:2eec with SMTP id j2-20020a056214032200b0053224092eecmr15693944qvu.18.1673303154591; Mon, 09 Jan 2023 14:25:54 -0800 (PST) Received: from joelboxx.c.googlers.com.com (48.230.85.34.bc.googleusercontent.com. [34.85.230.48]) by smtp.gmail.com with ESMTPSA id b5-20020a05620a04e500b006ee8874f5fasm6000671qkh.53.2023.01.09.14.25.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 09 Jan 2023 14:25:53 -0800 (PST) From: "Joel Fernandes (Google)" To: linux-kernel@vger.kernel.org Subject: [PATCH v2 RESEND] adreno: Shutdown the GPU properly Date: Mon, 9 Jan 2023 22:25:47 +0000 Message-Id: <20230109222547.1368644-1-joel@joelfernandes.org> X-Mailer: git-send-email 2.39.0.314.g84b9a713c41-goog MIME-Version: 1.0 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Rob Clark , Akhil P Oommen , freedreno@lists.freedesktop.org, linux-arm-msm@vger.kernel.org, Ricardo Ribalda , Konrad Dybcio , Abhinav Kumar , Steven Rostedt , Dmitry Baryshkov , dri-devel@lists.freedesktop.org, Ross Zwisler , "Joel Fernandes \(Google\)" , Sean Paul , AngeloGioacchino Del Regno Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" During kexec on ARM device, we notice that device_shutdown() only calls pm_runtime_force_suspend() while shutting down the GPU. This means the GPU kthread is still running and further, there maybe active submits. This causes all kinds of issues during a kexec reboot: Warning from shutdown path: [ 292.509662] WARNING: CPU: 0 PID: 6304 at [...] adreno_runtime_suspend+0x3c/0x44 [ 292.509863] Hardware name: Google Lazor (rev3 - 8) with LTE (DT) [ 292.509872] pstate: 80400009 (Nzcv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 292.509881] pc : adreno_runtime_suspend+0x3c/0x44 [ 292.509891] lr : pm_generic_runtime_suspend+0x30/0x44 [ 292.509905] sp : ffffffc014473bf0 [...] [ 292.510043] Call trace: [ 292.510051] adreno_runtime_suspend+0x3c/0x44 [ 292.510061] pm_generic_runtime_suspend+0x30/0x44 [ 292.510071] pm_runtime_force_suspend+0x54/0xc8 [ 292.510081] adreno_shutdown+0x1c/0x28 [ 292.510090] platform_shutdown+0x2c/0x38 [ 292.510104] device_shutdown+0x158/0x210 [ 292.510119] kernel_restart_prepare+0x40/0x4c And here from GPU kthread, an SError OOPs: [ 192.648789] el1h_64_error+0x7c/0x80 [ 192.648812] el1_interrupt+0x20/0x58 [ 192.648833] el1h_64_irq_handler+0x18/0x24 [ 192.648854] el1h_64_irq+0x7c/0x80 [ 192.648873] local_daif_inherit+0x10/0x18 [ 192.648900] el1h_64_sync_handler+0x48/0xb4 [ 192.648921] el1h_64_sync+0x7c/0x80 [ 192.648941] a6xx_gmu_set_oob+0xbc/0x1fc [ 192.648968] a6xx_hw_init+0x44/0xe38 [ 192.648991] msm_gpu_hw_init+0x48/0x80 [ 192.649013] msm_gpu_submit+0x5c/0x1a8 [ 192.649034] msm_job_run+0xb0/0x11c [ 192.649058] drm_sched_main+0x170/0x434 [ 192.649086] kthread+0x134/0x300 [ 192.649114] ret_from_fork+0x10/0x20 Fix by calling adreno_system_suspend() in the device_shutdown() path. [ Applied Rob Clark feedback on fixing adreno_unbind() similarly, also tested as above. ] Cc: Rob Clark Cc: Steven Rostedt Cc: Ricardo Ribalda Cc: Ross Zwisler Signed-off-by: Joel Fernandes (Google) Reviewed-by: Ricardo Ribalda Reviewed-by: Rob Clark --- drivers/gpu/drm/msm/adreno/adreno_device.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/drivers/gpu/drm/msm/adreno/adreno_device.c b/drivers/gpu/drm/msm/adreno/adreno_device.c index 628806423f7d..36f062c7582f 100644 --- a/drivers/gpu/drm/msm/adreno/adreno_device.c +++ b/drivers/gpu/drm/msm/adreno/adreno_device.c @@ -551,13 +551,14 @@ static int adreno_bind(struct device *dev, struct device *master, void *data) return 0; } +static int adreno_system_suspend(struct device *dev); static void adreno_unbind(struct device *dev, struct device *master, void *data) { struct msm_drm_private *priv = dev_get_drvdata(master); struct msm_gpu *gpu = dev_to_gpu(dev); - pm_runtime_force_suspend(dev); + WARN_ON_ONCE(adreno_system_suspend(dev)); gpu->funcs->destroy(gpu); priv->gpu_pdev = NULL; @@ -609,7 +610,7 @@ static int adreno_remove(struct platform_device *pdev) static void adreno_shutdown(struct platform_device *pdev) { - pm_runtime_force_suspend(&pdev->dev); + WARN_ON_ONCE(adreno_system_suspend(&pdev->dev)); } static const struct of_device_id dt_match[] = {