From patchwork Thu Nov 28 15:37:04 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Raag Jadav X-Patchwork-Id: 13888183 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 06F45D69114 for ; Thu, 28 Nov 2024 15:38:33 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id A8AA110ECE8; Thu, 28 Nov 2024 15:38:32 +0000 (UTC) Authentication-Results: gabe.freedesktop.org; dkim=pass (2048-bit key; unprotected) header.d=intel.com header.i=@intel.com header.b="ckDNGcLi"; dkim-atps=neutral Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.20]) by gabe.freedesktop.org (Postfix) with ESMTPS id 3C39910ECE7; Thu, 28 Nov 2024 15:38:31 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1732808311; x=1764344311; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=RaqAoTENE1o2dueh5FZ4gpUZZcXnB+IeysPpYMs4yeM=; b=ckDNGcLiSCcPChvRKJY/HO7IvkA/S+w7M1myNffwOdSa7iv0uIVhbSJO NZhLl2Yqa+RkmpC8Y0xhIAuySBpEO+493iFs8QJna0J2RMWz2HTl9ggni 0JSUBJg+uFbLhcNMsDarwIwW5oKYycRZpi1QFioq6PDf2A3UOruMzgRyL aLyGmyRxNQp2yf6tzbJlTlNmZ9v9zP4pKYrPF4zeY+/+DN42nqs5Ra9PA Mm0k11s4warBcibvOQ7GcPVSW4sHiWFJVx2+4NJlZzl4bH+JEUkHzjwcA 2PApUk0TIsoEBOXnQN36psC6WA+KxdBMX5l9BSU7Sf61ypbs9dsly8tj9 g==; X-CSE-ConnectionGUID: ZiPoBnNcTBC12gCt7wSCTw== X-CSE-MsgGUID: UclX8ZhaR7Ws7Hk/r/89Hw== X-IronPort-AV: E=McAfee;i="6700,10204,11270"; a="32792632" X-IronPort-AV: E=Sophos;i="6.12,192,1728975600"; d="scan'208";a="32792632" Received: from orviesa009.jf.intel.com ([10.64.159.149]) by orvoesa112.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 28 Nov 2024 07:38:31 -0800 X-CSE-ConnectionGUID: zEOBGx8iQAC60c2hyjttAg== X-CSE-MsgGUID: 1NLlvij6Tey0GrJ/huh4Zw== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.12,192,1728975600"; d="scan'208";a="92193083" Received: from jraag-nuc8i7beh.iind.intel.com ([10.145.169.79]) by orviesa009.jf.intel.com with ESMTP; 28 Nov 2024 07:38:25 -0800 From: Raag Jadav To: airlied@gmail.com, simona@ffwll.ch, lucas.demarchi@intel.com, rodrigo.vivi@intel.com, jani.nikula@linux.intel.com, andriy.shevchenko@linux.intel.com, lina@asahilina.net, michal.wajdeczko@intel.com, christian.koenig@amd.com Cc: intel-xe@lists.freedesktop.org, intel-gfx@lists.freedesktop.org, dri-devel@lists.freedesktop.org, himal.prasad.ghimiray@intel.com, aravind.iddamsetty@linux.intel.com, anshuman.gupta@intel.com, alexander.deucher@amd.com, andrealmeid@igalia.com, amd-gfx@lists.freedesktop.org, kernel-dev@igalia.com, Raag Jadav Subject: [PATCH v10 1/4] drm: Introduce device wedged event Date: Thu, 28 Nov 2024 21:07:04 +0530 Message-Id: <20241128153707.1294347-2-raag.jadav@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20241128153707.1294347-1-raag.jadav@intel.com> References: <20241128153707.1294347-1-raag.jadav@intel.com> MIME-Version: 1.0 X-BeenThere: intel-gfx@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Intel graphics driver community testing & development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: intel-gfx-bounces@lists.freedesktop.org Sender: "Intel-gfx" Introduce device wedged event, which notifies userspace of 'wedged' (hanged/unusable) state of the DRM device through a uevent. This is useful especially in cases where the device is no longer operating as expected and has become unrecoverable from driver context. Purpose of this implementation is to provide drivers a generic way to recover with the help of userspace intervention without taking any drastic measures in the driver. A 'wedged' device is basically a dead device that needs attention. The uevent is the notification that is sent to userspace along with a hint about what could possibly be attempted to recover the device and bring it back to usable state. Different drivers may have different ideas of a 'wedged' device depending on their hardware implementation, and hence the vendor agnostic nature of the event. It is up to the drivers to decide when they see the need for device recovery and how they want to recover from the available methods. Driver prerequisites -------------------- The driver, before opting for recovery, needs to make sure that the 'wedged' device doesn't harm the system as a whole by taking care of the prerequisites. Necessary actions must include disabling DMA to system memory as well as any communication channels with other devices. Further, the driver must ensure that all dma_fences are signalled and any device state that the core kernel might depend on is cleaned up. Any existing mmap should be invalidated and page faults should be redirected to a dummy page. Once the event is sent, the device must be kept in 'wedged' state until the recovery is performed. New accesses to the device (IOCTLs) should be blocked, preferably with an error code that resembles the type of failure the device has encountered. This will signify the reason for wedging, which can be reported to the application if needed. Recovery -------- Current implementation defines three recovery methods, out of which, drivers can use any one, multiple or none. Method(s) of choice will be sent in the uevent environment as ``WEDGED=[,]`` in order of less to more side-effects. If driver is unsure about recovery or method is unknown (like soft/hard reboot, firmware flashing, hardware replacement or any other procedure which can't be attempted on the fly), ``WEDGED=unknown`` will be sent instead. Userspace consumers can parse this event and attempt recovery as per the following expectations. =============== ================================ Recovery method Consumer expectations =============== ================================ none optional telemetry collection rebind unbind + bind driver bus-reset unbind + reset bus device + bind unknown admin/user policy =============== ================================ The only exception to this is ``WEDGED=none``, which signifies that the device was temporarily 'wedged' at some point but was able to recover using device specific methods like reset. No explicit device recovery is expected from the consumer in this case, but it can still take additional steps like gathering telemetry information (devcoredump, syslog). This is useful because the first hang is usually the most critical one which can result in consequential hangs or complete wedging. Consumer prerequisites ---------------------- It is the responsibility of the consumer to make sure that the device or its resources are not in use by any process before attempting recovery. With IOCTLs blocked and device already 'wedged', all device memory should be unmapped and file descriptors should be closed to prevent leaks. Example ------- Udev rule:: SUBSYSTEM=="drm", ENV{WEDGED}=="rebind", DEVPATH=="*/drm/card[0-9]", RUN+="/path/to/rebind.sh $env{DEVPATH}" Recovery script:: #!/bin/sh DEVPATH=$(readlink -f /sys/$1/device) DEVICE=$(basename $DEVPATH) DRIVER=$(readlink -f $DEVPATH/driver) echo -n $DEVICE > $DRIVER/unbind sleep 1 echo -n $DEVICE > $DRIVER/bind Customization ------------- Although basic recovery is possible with a simple script, admin/users can define custom policies around recovery action. For example, if the driver supports multiple recovery methods, consumers can opt for the suitable one based on policy definition. Consumers can also choose to have the device available for debugging or additional data collection before performing the recovery. This is useful especially when the driver is unsure about recovery or method is unknown. v4: s/drm_dev_wedged/drm_dev_wedged_event Use drm_info() (Jani) Kernel doc adjustment (Aravind) v5: Send recovery method with uevent (Lina) v6: Access wedge_recovery_opts[] using helper function (Jani) Use snprintf() (Jani) v7: Convert recovery helpers into regular functions (Andy, Jani) Aesthetic adjustments (Andy) Handle invalid method cases v8: Allow sending multiple methods with uevent (Lucas, Michal) static_assert() globally (Andy) v9: Provide 'none' method for reset cases (Christian) Provide recovery opts using switch cases Signed-off-by: Raag Jadav --- drivers/gpu/drm/drm_drv.c | 66 +++++++++++++++++++++++++++++++++++++++ include/drm/drm_device.h | 8 +++++ include/drm/drm_drv.h | 1 + 3 files changed, 75 insertions(+) diff --git a/drivers/gpu/drm/drm_drv.c b/drivers/gpu/drm/drm_drv.c index c2c172eb25df..e6583ebc6b05 100644 --- a/drivers/gpu/drm/drm_drv.c +++ b/drivers/gpu/drm/drm_drv.c @@ -26,6 +26,7 @@ * DEALINGS IN THE SOFTWARE. */ +#include #include #include #include @@ -33,6 +34,7 @@ #include #include #include +#include #include #include @@ -497,6 +499,70 @@ void drm_dev_unplug(struct drm_device *dev) } EXPORT_SYMBOL(drm_dev_unplug); +/* + * Available recovery methods for wedged device. To be sent along with device + * wedged uevent. + */ +static const char *drm_get_wedge_recovery(unsigned int opt) +{ + switch (BIT(opt)) { + case DRM_WEDGE_RECOVERY_NONE: + return "none"; + case DRM_WEDGE_RECOVERY_REBIND: + return "rebind"; + case DRM_WEDGE_RECOVERY_BUS_RESET: + return "bus-reset"; + default: + return NULL; + } +} + +/** + * drm_dev_wedged_event - generate a device wedged uevent + * @dev: DRM device + * @method: method(s) to be used for recovery + * + * This generates a device wedged uevent for the DRM device specified by @dev. + * Recovery @method\(s) of choice will be sent in the uevent environment as + * ``WEDGED=[,]`` in order of less to more side-effects. If + * caller is unsure about recovery or @method is unknown (0), ``WEDGED=unknown`` + * will be sent instead. + * + * Refer to 'Device Wedging' chapter in Documentation/gpu/drm-uapi.rst for more + * details. + * + * Returns: 0 on success, negative error code otherwise. + */ +int drm_dev_wedged_event(struct drm_device *dev, unsigned long method) +{ + const char *recovery = NULL; + unsigned int len, opt; + /* Event string length up to 28+ characters with available methods */ + char event_string[32]; + char *envp[] = { event_string, NULL }; + + len = scnprintf(event_string, sizeof(event_string), "%s", "WEDGED="); + + for_each_set_bit(opt, &method, BITS_PER_TYPE(method)) { + recovery = drm_get_wedge_recovery(opt); + if (drm_WARN(dev, !recovery, "device wedged, invalid recovery method %u\n", opt)) + break; + + len += scnprintf(event_string + len, sizeof(event_string), "%s,", recovery); + } + + if (recovery) + /* Get rid of trailing comma */ + event_string[len - 1] = '\0'; + else + /* Caller is unsure about recovery, do the best we can at this point. */ + snprintf(event_string, sizeof(event_string), "%s", "WEDGED=unknown"); + + drm_info(dev, "device wedged, needs recovery\n"); + return kobject_uevent_env(&dev->primary->kdev->kobj, KOBJ_CHANGE, envp); +} +EXPORT_SYMBOL(drm_dev_wedged_event); + /* * DRM internal mount * We want to be able to allocate our own "struct address_space" to control diff --git a/include/drm/drm_device.h b/include/drm/drm_device.h index c91f87b5242d..6ea54a578cda 100644 --- a/include/drm/drm_device.h +++ b/include/drm/drm_device.h @@ -21,6 +21,14 @@ struct inode; struct pci_dev; struct pci_controller; +/* + * Recovery methods for wedged device in order of less to more side-effects. + * To be used with drm_dev_wedged_event() as recovery @method. Callers can + * use any one, multiple (or'd) or none depending on their needs. + */ +#define DRM_WEDGE_RECOVERY_NONE BIT(0) /* optional telemetry collection */ +#define DRM_WEDGE_RECOVERY_REBIND BIT(1) /* unbind + bind driver */ +#define DRM_WEDGE_RECOVERY_BUS_RESET BIT(2) /* unbind + reset bus device + bind */ /** * enum switch_power_state - power state of drm device diff --git a/include/drm/drm_drv.h b/include/drm/drm_drv.h index 1bbbcb8e2d23..f41a82839e28 100644 --- a/include/drm/drm_drv.h +++ b/include/drm/drm_drv.h @@ -479,6 +479,7 @@ void drm_put_dev(struct drm_device *dev); bool drm_dev_enter(struct drm_device *dev, int *idx); void drm_dev_exit(int idx); void drm_dev_unplug(struct drm_device *dev); +int drm_dev_wedged_event(struct drm_device *dev, unsigned long method); /** * drm_dev_is_unplugged - is a DRM device unplugged From patchwork Thu Nov 28 15:37:05 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Raag Jadav X-Patchwork-Id: 13888184 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id C86F9D6911A for ; Thu, 28 Nov 2024 15:38:39 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 4B6C110ECEC; Thu, 28 Nov 2024 15:38:39 +0000 (UTC) Authentication-Results: gabe.freedesktop.org; dkim=pass (2048-bit key; unprotected) header.d=intel.com header.i=@intel.com header.b="hzX0XID3"; dkim-atps=neutral Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.20]) by gabe.freedesktop.org (Postfix) with ESMTPS id 6FF3110ECEB; Thu, 28 Nov 2024 15:38:37 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1732808317; x=1764344317; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=evxs1HgW7KzrjOU/lEVYgqDdo+ldzoYSVqHeHY9nrls=; b=hzX0XID3ysTLnDbRAfG8FgKNsPHfwy2JbD25o2BE2eeEvCcWEO7tLZ5R A9gIuiC7G+X//630SCZZw6ddQ+yjmBRpXNNN1skc42byZkN7CXMFNFkK6 efDbudNIgM4sF49Tw74eiRC04rb5nhUR8Q/V8946MFnth0s1TcnGL9wSm 5JmDMdexS7RxseWKg9gI5xAuy9BxEROlhkbjcMXNmPe5FNLV72aY1As2P bRQtVd5hqOvgHDKcQtz+oKatH+/4etFdwDJpmXYc91ERc7MUZvRdNzRK0 EkoBlZYQLPUjNyiPc2Zy9xjGGugsxjjARdG2L/SEiMAUDVHSTxHZCoZcq Q==; X-CSE-ConnectionGUID: 9TYtjgbdSvOEfUviFH2e1w== X-CSE-MsgGUID: l9F35e9eQ8aoWBkFAlBDaA== X-IronPort-AV: E=McAfee;i="6700,10204,11270"; a="32792645" X-IronPort-AV: E=Sophos;i="6.12,192,1728975600"; d="scan'208";a="32792645" Received: from orviesa009.jf.intel.com ([10.64.159.149]) by orvoesa112.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 28 Nov 2024 07:38:37 -0800 X-CSE-ConnectionGUID: 5Lp+ZF+NQ5mVsAsk+PRlxg== X-CSE-MsgGUID: +nb2B1ijR7Wnl95HZ/EK5A== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.12,192,1728975600"; d="scan'208";a="92193089" Received: from jraag-nuc8i7beh.iind.intel.com ([10.145.169.79]) by orviesa009.jf.intel.com with ESMTP; 28 Nov 2024 07:38:32 -0800 From: Raag Jadav To: airlied@gmail.com, simona@ffwll.ch, lucas.demarchi@intel.com, rodrigo.vivi@intel.com, jani.nikula@linux.intel.com, andriy.shevchenko@linux.intel.com, lina@asahilina.net, michal.wajdeczko@intel.com, christian.koenig@amd.com Cc: intel-xe@lists.freedesktop.org, intel-gfx@lists.freedesktop.org, dri-devel@lists.freedesktop.org, himal.prasad.ghimiray@intel.com, aravind.iddamsetty@linux.intel.com, anshuman.gupta@intel.com, alexander.deucher@amd.com, andrealmeid@igalia.com, amd-gfx@lists.freedesktop.org, kernel-dev@igalia.com, Raag Jadav Subject: [PATCH v10 2/4] drm/doc: Document device wedged event Date: Thu, 28 Nov 2024 21:07:05 +0530 Message-Id: <20241128153707.1294347-3-raag.jadav@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20241128153707.1294347-1-raag.jadav@intel.com> References: <20241128153707.1294347-1-raag.jadav@intel.com> MIME-Version: 1.0 X-BeenThere: intel-gfx@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Intel graphics driver community testing & development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: intel-gfx-bounces@lists.freedesktop.org Sender: "Intel-gfx" Add documentation for device wedged event in a new 'Device wedging' chapter. The describes basic definitions, prerequisites and consumer expectations along with an example. v8: Improve documentation (Christian, Rodrigo) v9: Add prerequisites section (Christian) v10: Clarify mmap cleanup and consumer prerequisites (Christian, Aravind) Signed-off-by: Raag Jadav Reviewed-by: Christian König --- Documentation/gpu/drm-uapi.rst | 112 ++++++++++++++++++++++++++++++++- 1 file changed, 109 insertions(+), 3 deletions(-) diff --git a/Documentation/gpu/drm-uapi.rst b/Documentation/gpu/drm-uapi.rst index b75cc9a70d1f..da2927cde53d 100644 --- a/Documentation/gpu/drm-uapi.rst +++ b/Documentation/gpu/drm-uapi.rst @@ -371,9 +371,115 @@ Reporting causes of resets Apart from propagating the reset through the stack so apps can recover, it's really useful for driver developers to learn more about what caused the reset in -the first place. DRM devices should make use of devcoredump to store relevant -information about the reset, so this information can be added to user bug -reports. +the first place. For this, drivers can make use of devcoredump to store relevant +information about the reset and send device wedged event without recovery method +(as explained in next chapter) to notify userspace, so this information can be +collected and added to user bug reports. + +Device wedging +============== + +Drivers can optionally make use of device wedged event (implemented as +drm_dev_wedged_event() in DRM subsystem), which notifies userspace of 'wedged' +(hanged/unusable) state of the DRM device through a uevent. This is useful +especially in cases where the device is no longer operating as expected and +has become unrecoverable from driver context. Purpose of this implementation +is to provide drivers a generic way to recover with the help of userspace +intervention without taking any drastic measures in the driver. + +A 'wedged' device is basically a dead device that needs attention. The +uevent is the notification that is sent to userspace along with a hint about +what could possibly be attempted to recover the device and bring it back to +usable state. Different drivers may have different ideas of a 'wedged' device +depending on their hardware implementation, and hence the vendor agnostic +nature of the event. It is up to the drivers to decide when they see the need +for device recovery and how they want to recover from the available methods. + +Driver prerequisites +-------------------- + +The driver, before opting for recovery, needs to make sure that the 'wedged' +device doesn't harm the system as a whole by taking care of the prerequisites. +Necessary actions must include disabling DMA to system memory as well as any +communication channels with other devices. Further, the driver must ensure +that all dma_fences are signalled and any device state that the core kernel +might depend on is cleaned up. Any existing mmap should be invalidated and +page faults should be redirected to a dummy page. Once the event is sent, the +device must be kept in 'wedged' state until the recovery is performed. New +accesses to the device (IOCTLs) should be blocked, preferably with an error +code that resembles the type of failure the device has encountered. This will +signify the reason for wedging, which can be reported to the application if +needed. + +Recovery +-------- + +Current implementation defines three recovery methods, out of which, drivers +can use any one, multiple or none. Method(s) of choice will be sent in the +uevent environment as ``WEDGED=[,]`` in order of less to +more side-effects. If driver is unsure about recovery or method is unknown +(like soft/hard reboot, firmware flashing, hardware replacement or any other +procedure which can't be attempted on the fly), ``WEDGED=unknown`` will be +sent instead. + +Userspace consumers can parse this event and attempt recovery as per the +following expectations. + + =============== ================================ + Recovery method Consumer expectations + =============== ================================ + none optional telemetry collection + rebind unbind + bind driver + bus-reset unbind + reset bus device + bind + unknown admin/user policy + =============== ================================ + +The only exception to this is ``WEDGED=none``, which signifies that the +device was temporarily 'wedged' at some point but was able to recover using +device specific methods like reset. No explicit device recovery is expected +from the consumer in this case, but it can still take additional steps like +gathering telemetry information (devcoredump, syslog). This is useful +because the first hang is usually the most critical one which can result in +consequential hangs or complete wedging. + +Consumer prerequisites +---------------------- + +It is the responsibility of the consumer to make sure that the device or +its resources are not in use by any process before attempting recovery. +With IOCTLs blocked and device already 'wedged', all device memory should +be unmapped and file descriptors should be closed to prevent leaks. + +Example +------- + +Udev rule:: + + SUBSYSTEM=="drm", ENV{WEDGED}=="rebind", DEVPATH=="*/drm/card[0-9]", + RUN+="/path/to/rebind.sh $env{DEVPATH}" + +Recovery script:: + + #!/bin/sh + + DEVPATH=$(readlink -f /sys/$1/device) + DEVICE=$(basename $DEVPATH) + DRIVER=$(readlink -f $DEVPATH/driver) + + echo -n $DEVICE > $DRIVER/unbind + sleep 1 + echo -n $DEVICE > $DRIVER/bind + +Customization +------------- + +Although basic recovery is possible with a simple script, admin/users can +define custom policies around recovery action. For example, if the driver +supports multiple recovery methods, consumers can opt for the suitable one +based on policy definition. Consumers can also choose to have the device +available for debugging or additional data collection before performing the +recovery. This is useful especially when the driver is unsure about recovery +or method is unknown. .. _drm_driver_ioctl: From patchwork Thu Nov 28 15:37:06 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Raag Jadav X-Patchwork-Id: 13888185 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id A825BD69119 for ; Thu, 28 Nov 2024 15:38:44 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 5574110ECE7; Thu, 28 Nov 2024 15:38:44 +0000 (UTC) Authentication-Results: gabe.freedesktop.org; dkim=pass (2048-bit key; unprotected) header.d=intel.com header.i=@intel.com header.b="eY/CytXu"; dkim-atps=neutral Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.20]) by gabe.freedesktop.org (Postfix) with ESMTPS id 7627010ECED; Thu, 28 Nov 2024 15:38:43 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1732808323; x=1764344323; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=bp14nKhtV1N2OY82d4MjLXSDZ5hPp86QDbW+7UNJ1A4=; b=eY/CytXuR9M5H7xKh/SiyYHm5lFRAnfJzf8x1ls6lxiQlelCPFCdgwe3 CevEskxdqMr2AfvdQVl1NVDdRRrC6S9+5aEq8lA9hff5M2kDzittqDy3H 5cmocZesXT5RW6LiJjhZBPN8GL6RkBZ00bga+dR4xdOEaSXSqs3542N/0 fTm0WekfK4ZE2nGOhrNeOph77kTb/5hkd0ZgNoWSJu5y1V7HEXi4/xKRF +qM1pfY9XlAuCM7pgFbdJMpijcAVWw1Z7YN1QM3Z1Zr6YDoe5dT8c6ZBk fYbMg7EK1IzavyVJn1ld1eDMLTxuWQ0vPP9GybD21wLKggkUITB58W8Ho w==; X-CSE-ConnectionGUID: 8jraKiyBSai9PgD2fzfecw== X-CSE-MsgGUID: PsKLlbnQSRuyEJxBWh5ipg== X-IronPort-AV: E=McAfee;i="6700,10204,11270"; a="32792660" X-IronPort-AV: E=Sophos;i="6.12,192,1728975600"; d="scan'208";a="32792660" Received: from orviesa009.jf.intel.com ([10.64.159.149]) by orvoesa112.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 28 Nov 2024 07:38:43 -0800 X-CSE-ConnectionGUID: H0vY0WCOQKyq0QMsigDBgA== X-CSE-MsgGUID: M70I2a6iT6K9u/k3EowGXw== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.12,192,1728975600"; d="scan'208";a="92193097" Received: from jraag-nuc8i7beh.iind.intel.com ([10.145.169.79]) by orviesa009.jf.intel.com with ESMTP; 28 Nov 2024 07:38:38 -0800 From: Raag Jadav To: airlied@gmail.com, simona@ffwll.ch, lucas.demarchi@intel.com, rodrigo.vivi@intel.com, jani.nikula@linux.intel.com, andriy.shevchenko@linux.intel.com, lina@asahilina.net, michal.wajdeczko@intel.com, christian.koenig@amd.com Cc: intel-xe@lists.freedesktop.org, intel-gfx@lists.freedesktop.org, dri-devel@lists.freedesktop.org, himal.prasad.ghimiray@intel.com, aravind.iddamsetty@linux.intel.com, anshuman.gupta@intel.com, alexander.deucher@amd.com, andrealmeid@igalia.com, amd-gfx@lists.freedesktop.org, kernel-dev@igalia.com, Raag Jadav Subject: [PATCH v10 3/4] drm/xe: Use device wedged event Date: Thu, 28 Nov 2024 21:07:06 +0530 Message-Id: <20241128153707.1294347-4-raag.jadav@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20241128153707.1294347-1-raag.jadav@intel.com> References: <20241128153707.1294347-1-raag.jadav@intel.com> MIME-Version: 1.0 X-BeenThere: intel-gfx@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Intel graphics driver community testing & development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: intel-gfx-bounces@lists.freedesktop.org Sender: "Intel-gfx" This was previously attempted as xe specific reset uevent but dropped in commit 77a0d4d1cea2 ("drm/xe/uapi: Remove reset uevent for now") as part of refactoring. Now that we have device wedged event provided by DRM core, make use of it and support both driver rebind and bus-reset based recovery. With this in place userspace will be notified of wedged device, on the basis of which, userspace may take respective action to recover the device. $ udevadm monitor --property --kernel monitor will print the received events for: KERNEL - the kernel uevent KERNEL[265.802982] change /devices/pci0000:00/0000:00:01.0/0000:01:00.0/0000:02:01.0/0000:03:00.0/drm/card0 (drm) ACTION=change DEVPATH=/devices/pci0000:00/0000:00:01.0/0000:01:00.0/0000:02:01.0/0000:03:00.0/drm/card0 SUBSYSTEM=drm WEDGED=rebind,bus-reset DEVNAME=/dev/dri/card0 DEVTYPE=drm_minor SEQNUM=5208 MAJOR=226 MINOR=0 v2: Change authorship to Himal (Aravind) Add uevent for all device wedged cases (Aravind) v3: Generic re-implementation in DRM subsystem (Lucas) v4: Change authorship to Raag (Aravind) Signed-off-by: Raag Jadav Reviewed-by: Aravind Iddamsetty --- drivers/gpu/drm/xe/xe_device.c | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/drivers/gpu/drm/xe/xe_device.c b/drivers/gpu/drm/xe/xe_device.c index 930bb2750e2e..f04737854887 100644 --- a/drivers/gpu/drm/xe/xe_device.c +++ b/drivers/gpu/drm/xe/xe_device.c @@ -991,11 +991,12 @@ static void xe_device_wedged_fini(struct drm_device *drm, void *arg) * xe_device_declare_wedged - Declare device wedged * @xe: xe device instance * - * This is a final state that can only be cleared with a mudule + * This is a final state that can only be cleared with a module * re-probe (unbind + bind). * In this state every IOCTL will be blocked so the GT cannot be used. * In general it will be called upon any critical error such as gt reset - * failure or guc loading failure. + * failure or guc loading failure. Userspace will be notified of this state + * by a DRM uevent. * If xe.wedged module parameter is set to 2, this function will be called * on every single execution timeout (a.k.a. GPU hang) right after devcoredump * snapshot capture. In this mode, GT reset won't be attempted so the state of @@ -1025,6 +1026,10 @@ void xe_device_declare_wedged(struct xe_device *xe) "IOCTLs and executions are blocked. Only a rebind may clear the failure\n" "Please file a _new_ bug report at https://gitlab.freedesktop.org/drm/xe/kernel/issues/new\n", dev_name(xe->drm.dev)); + + /* Notify userspace of wedged device */ + drm_dev_wedged_event(&xe->drm, + DRM_WEDGE_RECOVERY_REBIND | DRM_WEDGE_RECOVERY_BUS_RESET); } for_each_gt(gt, xe, id) From patchwork Thu Nov 28 15:37:07 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Raag Jadav X-Patchwork-Id: 13888186 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id CBA5CD69119 for ; Thu, 28 Nov 2024 15:38:51 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 719AF10ECF3; Thu, 28 Nov 2024 15:38:51 +0000 (UTC) Authentication-Results: gabe.freedesktop.org; dkim=pass (2048-bit key; unprotected) header.d=intel.com header.i=@intel.com header.b="WmXwsviS"; dkim-atps=neutral Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.20]) by gabe.freedesktop.org (Postfix) with ESMTPS id 5B07710ECF4; Thu, 28 Nov 2024 15:38:49 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1732808329; x=1764344329; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=+HmVSqga3n7mRAEm5//05RCGz/gUnv1oBQxeHSTnpPg=; b=WmXwsviSgD0bZuj8cNYUqyqoyhFOiootOwXwckjqcsUx+cvb6WkaFek/ GhRcafnheF+t9xPt05luNLMoqaagJSgT23czPH9XFxFxNlk1Ewe4Cc1E9 jkWM4SuILnoAfPLlWqQwA6LRFO2627cGJT5cV209mGFelbCiCcbsZeL9B iTaOv3uiiK9Cyls6vt3SYxWUyoUwxWhzhwLzMABUKlbocosVs/5Z+KNb8 epuGfsBvyIA/gUgVxvc6iMpMgZSViyaiAEShj1LVyF7bQnimjXWK3awsC dN1/8+a7idTPtVH8qodnYdIK4lWo24D5NAlobhgBVyX9BZUEinA5W8zp+ g==; X-CSE-ConnectionGUID: /YWxIreCR+CuUrxr6cN3gA== X-CSE-MsgGUID: IFk68TvvQ3aedVymuuHVFQ== X-IronPort-AV: E=McAfee;i="6700,10204,11270"; a="32792670" X-IronPort-AV: E=Sophos;i="6.12,192,1728975600"; d="scan'208";a="32792670" Received: from orviesa009.jf.intel.com ([10.64.159.149]) by orvoesa112.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 28 Nov 2024 07:38:49 -0800 X-CSE-ConnectionGUID: mnoeAkoNQDWISrnO+Gimcg== X-CSE-MsgGUID: 9/b8I9QJSTSTono2eFBw/Q== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.12,192,1728975600"; d="scan'208";a="92193101" Received: from jraag-nuc8i7beh.iind.intel.com ([10.145.169.79]) by orviesa009.jf.intel.com with ESMTP; 28 Nov 2024 07:38:44 -0800 From: Raag Jadav To: airlied@gmail.com, simona@ffwll.ch, lucas.demarchi@intel.com, rodrigo.vivi@intel.com, jani.nikula@linux.intel.com, andriy.shevchenko@linux.intel.com, lina@asahilina.net, michal.wajdeczko@intel.com, christian.koenig@amd.com Cc: intel-xe@lists.freedesktop.org, intel-gfx@lists.freedesktop.org, dri-devel@lists.freedesktop.org, himal.prasad.ghimiray@intel.com, aravind.iddamsetty@linux.intel.com, anshuman.gupta@intel.com, alexander.deucher@amd.com, andrealmeid@igalia.com, amd-gfx@lists.freedesktop.org, kernel-dev@igalia.com, Raag Jadav Subject: [PATCH v10 4/4] drm/i915: Use device wedged event Date: Thu, 28 Nov 2024 21:07:07 +0530 Message-Id: <20241128153707.1294347-5-raag.jadav@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20241128153707.1294347-1-raag.jadav@intel.com> References: <20241128153707.1294347-1-raag.jadav@intel.com> MIME-Version: 1.0 X-BeenThere: intel-gfx@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Intel graphics driver community testing & development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: intel-gfx-bounces@lists.freedesktop.org Sender: "Intel-gfx" Now that we have device wedged event provided by DRM core, make use of it and support both driver rebind and bus-reset based recovery. With this in place, userspace will be notified of wedged device on gt reset failure. Signed-off-by: Raag Jadav Reviewed-by: Aravind Iddamsetty --- drivers/gpu/drm/i915/gt/intel_reset.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/drivers/gpu/drm/i915/gt/intel_reset.c b/drivers/gpu/drm/i915/gt/intel_reset.c index f42f21632306..18cf50a1e84d 100644 --- a/drivers/gpu/drm/i915/gt/intel_reset.c +++ b/drivers/gpu/drm/i915/gt/intel_reset.c @@ -1418,6 +1418,9 @@ static void intel_gt_reset_global(struct intel_gt *gt, if (!test_bit(I915_WEDGED, >->reset.flags)) kobject_uevent_env(kobj, KOBJ_CHANGE, reset_done_event); + else + drm_dev_wedged_event(>->i915->drm, + DRM_WEDGE_RECOVERY_REBIND | DRM_WEDGE_RECOVERY_BUS_RESET); } /**