[1/5] auto-t: end process_io on HUP signal, detect process crash

Commit Message

James Prestwood Feb. 29, 2024, 5:07 p.m. UTC

When HUP is received the IO read callback was never completing which
caused it to block indefinitely until waited for. This didn't matter
for most transient processes but for IWD, hostapd, wpa_supplicant
it would cause test-runner to hang if the process crashed.

Detecting a crash is somewhat hacky because we have no process
management like systemd and the return code isn't reliable as some
processes return non-zero under normal circumstances. So to detect
a crash the process output is being checked for the string:
"++++++++ backtrace ++++++++". This isn't 100% reliable obviously
since its dependent on how the binary is compiled, but even if the
crash itself isn't detected any test should still fail if written
correctly.

Doing this allows auto-tests to handle IWD crashes gracefully by
failing the test, printing the exception (event without debugging)
and continue with other tests.
---
 tools/utils.py | 10 +++++++---
 1 file changed, 7 insertions(+), 3 deletions(-)

Comments

Denis Kenzior Feb. 29, 2024, 8:36 p.m. UTC | #1

Hi James,

On 2/29/24 11:07, James Prestwood wrote:
> When HUP is received the IO read callback was never completing which
> caused it to block indefinitely until waited for. This didn't matter
> for most transient processes but for IWD, hostapd, wpa_supplicant
> it would cause test-runner to hang if the process crashed.
> 
> Detecting a crash is somewhat hacky because we have no process
> management like systemd and the return code isn't reliable as some
> processes return non-zero under normal circumstances. So to detect
> a crash the process output is being checked for the string:
> "++++++++ backtrace ++++++++". This isn't 100% reliable obviously
> since its dependent on how the binary is compiled, but even if the
> crash itself isn't detected any test should still fail if written
> correctly.
> 
> Doing this allows auto-tests to handle IWD crashes gracefully by
> failing the test, printing the exception (event without debugging)
> and continue with other tests.
> ---
>   tools/utils.py | 10 +++++++---
>   1 file changed, 7 insertions(+), 3 deletions(-)
> 

All applied, thanks.

Regards,
-Denis

Patch

diff --git a/tools/utils.py b/tools/utils.py
index 5984fc69..d5445ea7 100644
--- a/tools/utils.py
+++ b/tools/utils.py
@@ -175,23 +175,27 @@  class Process(subprocess.Popen):
 	def process_io(self, source, condition):
 		if condition & GLib.IO_HUP:
 			self.hup = True
+			self.wait()
+			bt = self.out.partition("++++++++ backtrace ++++++++")
+			if bt[1]:
+				raise Exception(f"Process {self.args[0]} crashed!\n{bt[1] + bt[2]}")
 
 		data = source.read()
 
 		if not data:
-			return True
+			return not self.hup
 
 		try:
 			data = data.decode('utf-8')
 		except:
-			return True
+			return not self.hup
 
 		# Save data away in case the caller needs it (e.g. list_sta)
 		self.out += data
 
 		self._write_io(self, data)
 
-		return True
+		return not self.hup
 
 	def _append_outfile(self, file, append=True):
 		gid = int(os.environ.get('SUDO_GID', os.getgid()))