| Message ID | 20220329214248.1330868-1-dlatypov@google.com (mailing list archive) |
|---|---|
| State | Accepted |
| Commit | 9660209d9418f2295d31fea0d32e313e9b2c1200 |
| Delegated to: | Brendan Higgins |
| Headers | show |
| Series | kunit: tool: print clearer error message when there's no TAP output | expand |
On Wed, Mar 30, 2022 at 5:43 AM Daniel Latypov <dlatypov@google.com> wrote: > > Before: > $ ./tools/testing/kunit/kunit.py parse /dev/null > ... > [ERROR] Test : invalid KTAP input! > > After: > $ ./tools/testing/kunit/kunit.py parse /dev/null > ... > [ERROR] Test <missing>: could not find any KTAP output! > > This error message gets printed out when extract_tap_output() yielded no > lines. So while it could be because of malformed KTAP output from KUnit, > it could also be due to to not having any KTAP output at all. > > Try and make the error message here more clear. > > Signed-off-by: Daniel Latypov <dlatypov@google.com> > --- At first I thought that this was "working as intended", but I agree that it's a bit confusing, so changing it is for the best. (And there's no sense getting too bogged down in the philosophical difference between "invalid TAP" and "not valid TAP" :-)) This works fine here, and the code looks sensible. I tested it with the --json option as well, and the result ("test_cases" being empty) makes sense to me: --- { "name": "KUnit Test Group", "arch": "UM", "defconfig": "kunit_defconfig", "build_environment": "", "sub_groups": [], "test_cases": [], "lab_name": null, "kernel": null, "job": null, "git_branch": "kselftest" } --- So, this is: Reviewed-by: David Gow <davidgow@google.com> Cheers, -- David
On Tue, Mar 29, 2022 at 10:17 PM David Gow <davidgow@google.com> wrote: > > On Wed, Mar 30, 2022 at 5:43 AM Daniel Latypov <dlatypov@google.com> wrote: > > > > Before: > > $ ./tools/testing/kunit/kunit.py parse /dev/null > > ... > > [ERROR] Test : invalid KTAP input! > > > > After: > > $ ./tools/testing/kunit/kunit.py parse /dev/null > > ... > > [ERROR] Test <missing>: could not find any KTAP output! > > > > This error message gets printed out when extract_tap_output() yielded no > > lines. So while it could be because of malformed KTAP output from KUnit, > > it could also be due to to not having any KTAP output at all. > > > > Try and make the error message here more clear. > > > > Signed-off-by: Daniel Latypov <dlatypov@google.com> > > --- > > At first I thought that this was "working as intended", but I agree > that it's a bit confusing, so changing it is for the best. > (And there's no sense getting too bogged down in the philosophical > difference between "invalid TAP" and "not valid TAP" :-)) extract_tap_output() returning something means it just saw a TAP header. So this error realistically only happens when there's no TAP at all :P My allusion to malformed KTAP is in the sense that KTA<random stuff>P ver<more stuff>sion 1 is malformed. We could potentially be more completely precise w/ our error message and say "no K?TAP header found", but I initially thought this might be a bit more user-friendly.
On Tue, Mar 29, 2022 at 5:43 PM Daniel Latypov <dlatypov@google.com> wrote: > > Before: > $ ./tools/testing/kunit/kunit.py parse /dev/null > ... > [ERROR] Test : invalid KTAP input! > > After: > $ ./tools/testing/kunit/kunit.py parse /dev/null > ... > [ERROR] Test <missing>: could not find any KTAP output! > > This error message gets printed out when extract_tap_output() yielded no > lines. So while it could be because of malformed KTAP output from KUnit, > it could also be due to to not having any KTAP output at all. > > Try and make the error message here more clear. > > Signed-off-by: Daniel Latypov <dlatypov@google.com> Reviewed-by: Brendan Higgins <brendanhiggins@google.com>
diff --git a/tools/testing/kunit/kunit_parser.py b/tools/testing/kunit/kunit_parser.py index 05ff334761dd..103d95a66a7e 100644 --- a/tools/testing/kunit/kunit_parser.py +++ b/tools/testing/kunit/kunit_parser.py @@ -817,7 +817,8 @@ def parse_run_tests(kernel_output: Iterable[str]) -> Test: lines = extract_tap_lines(kernel_output) test = Test() if not lines: - test.add_error('invalid KTAP input!') + test.name = '<missing>' + test.add_error('could not find any KTAP output!') test.status = TestStatus.FAILURE_TO_PARSE_TESTS else: test = parse_test(lines, 0, []) diff --git a/tools/testing/kunit/kunit_tool_test.py b/tools/testing/kunit/kunit_tool_test.py index 352369dffbd9..f14934853ea1 100755 --- a/tools/testing/kunit/kunit_tool_test.py +++ b/tools/testing/kunit/kunit_tool_test.py @@ -226,7 +226,7 @@ class KUnitParserTest(unittest.TestCase): with open(crash_log) as file: result = kunit_parser.parse_run_tests( kunit_parser.extract_tap_lines(file.readlines())) - print_mock.assert_any_call(StrContains('invalid KTAP input!')) + print_mock.assert_any_call(StrContains('could not find any KTAP output!')) print_mock.stop() self.assertEqual(0, len(result.subtests)) @@ -559,7 +559,7 @@ class KUnitMainTest(unittest.TestCase): self.assertEqual(e.exception.code, 1) self.assertEqual(self.linux_source_mock.build_reconfig.call_count, 1) self.assertEqual(self.linux_source_mock.run_kernel.call_count, 1) - self.print_mock.assert_any_call(StrContains('invalid KTAP input!')) + self.print_mock.assert_any_call(StrContains('could not find any KTAP output!')) def test_exec_no_tests(self): self.linux_source_mock.run_kernel = mock.Mock(return_value=['TAP version 14', '1..0'])
Before: $ ./tools/testing/kunit/kunit.py parse /dev/null ... [ERROR] Test : invalid KTAP input! After: $ ./tools/testing/kunit/kunit.py parse /dev/null ... [ERROR] Test <missing>: could not find any KTAP output! This error message gets printed out when extract_tap_output() yielded no lines. So while it could be because of malformed KTAP output from KUnit, it could also be due to to not having any KTAP output at all. Try and make the error message here more clear. Signed-off-by: Daniel Latypov <dlatypov@google.com> --- tools/testing/kunit/kunit_parser.py | 3 ++- tools/testing/kunit/kunit_tool_test.py | 4 ++-- 2 files changed, 4 insertions(+), 3 deletions(-) base-commit: 13776ebb9964b2ea66ffb8c824c0762eed6da784