| Message ID | 20250228180007.83325-1-kuba@kernel.org (mailing list archive) |
|---|---|
| State | Accepted |
| Commit | d110dbf1490b130f70fd0cd73a658ff495bc75e0 |
| Delegated to: | Netdev Maintainers |
| Headers | show |
| Series | [net-next] selftests: net: report output format as TAP 13 in Python tests | expand |
Hi Jakub, On 28/02/2025 19:00, Jakub Kicinski wrote: > The Python lib based tests report that they are producing > "KTAP version 1", but really we aren't making use of any > KTAP features, like subtests. Our output is plain TAP. > > Report TAP 13 instead of KTAP 1, this is what mptcp tests do, Indeed, and also TC tests, and all the ones using kselftest_harness.h I think. > and what NIPA knows how to parse best. For HW testing we need > precise subtest result tracking. I guess the best would be to have the kselftests infrastructure fully supporting KTAP, and then have a way for each test to print subtests correctly, not nested in the comments like it is done for the moment. But that looks harder to put in place, and the current solution is used in a few areas. Good idea to switch to 'TAP version 13' for the moment, and re-use the existing parsing in NIPA! Acked-by: Matthieu Baerts (NGI0) <matttbe@kernel.org> Cheers, Matt
On Sat, 1 Mar 2025 11:52:06 +0100 Matthieu Baerts wrote: > On 28/02/2025 19:00, Jakub Kicinski wrote: > > The Python lib based tests report that they are producing > > "KTAP version 1", but really we aren't making use of any > > KTAP features, like subtests. Our output is plain TAP. > > > > Report TAP 13 instead of KTAP 1, this is what mptcp tests do, > > Indeed, and also TC tests, and all the ones using kselftest_harness.h I > think. > > > and what NIPA knows how to parse best. For HW testing we need > > precise subtest result tracking. > > I guess the best would be to have the kselftests infrastructure fully > supporting KTAP, and then have a way for each test to print subtests > correctly, not nested in the comments like it is done for the moment. > But that looks harder to put in place, and the current solution is used > in a few areas. Right. So for additional context for folks less familiar with selftests the two minor inconveniences which made "supporting KTAP" a less exciting option are: (a) we run make run_tests which wraps things in an outer TAP, so we'd end up with a hybrid, anyway; (b) NIPA only supports one level of subtests. Thanks for the review!
Hello: This patch was applied to netdev/net-next.git (main) by Jakub Kicinski <kuba@kernel.org>: On Fri, 28 Feb 2025 10:00:07 -0800 you wrote: > The Python lib based tests report that they are producing > "KTAP version 1", but really we aren't making use of any > KTAP features, like subtests. Our output is plain TAP. > > Report TAP 13 instead of KTAP 1, this is what mptcp tests do, > and what NIPA knows how to parse best. For HW testing we need > precise subtest result tracking. > > [...] Here is the summary with links: - [net-next] selftests: net: report output format as TAP 13 in Python tests https://git.kernel.org/netdev/net-next/c/d110dbf1490b You are awesome, thank you!
diff --git a/tools/testing/selftests/drivers/net/README.rst b/tools/testing/selftests/drivers/net/README.rst index 3b6a29e6564b..eb838ae94844 100644 --- a/tools/testing/selftests/drivers/net/README.rst +++ b/tools/testing/selftests/drivers/net/README.rst @@ -107,7 +107,7 @@ Example 1..1 # timeout set to 45 # selftests: drivers/net: ping.py - # KTAP version 1 + # TAP version 13 # 1..3 # ok 1 ping.test_v4 # ok 2 ping.test_v6 @@ -128,7 +128,7 @@ Example Run the test:: [/root] # ./ksft-net-drv/drivers/net/ping.py - KTAP version 1 + TAP version 13 1..3 ok 1 ping.test_v4 ok 2 ping.test_v6 # SKIP Test requires IPv6 connectivity diff --git a/tools/testing/selftests/net/lib/py/ksft.py b/tools/testing/selftests/net/lib/py/ksft.py index fd23349fa8ca..3cfad0fd4570 100644 --- a/tools/testing/selftests/net/lib/py/ksft.py +++ b/tools/testing/selftests/net/lib/py/ksft.py @@ -207,7 +207,7 @@ KSFT_DISRUPTIVE = True totals = {"pass": 0, "fail": 0, "skip": 0, "xfail": 0} - print("KTAP version 1") + print("TAP version 13") print("1.." + str(len(cases))) global KSFT_RESULT
The Python lib based tests report that they are producing "KTAP version 1", but really we aren't making use of any KTAP features, like subtests. Our output is plain TAP. Report TAP 13 instead of KTAP 1, this is what mptcp tests do, and what NIPA knows how to parse best. For HW testing we need precise subtest result tracking. Signed-off-by: Jakub Kicinski <kuba@kernel.org> --- CC: shuah@kernel.org CC: petrm@nvidia.com CC: matttbe@kernel.org CC: willemb@google.com CC: linux-kselftest@vger.kernel.org --- tools/testing/selftests/drivers/net/README.rst | 4 ++-- tools/testing/selftests/net/lib/py/ksft.py | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-)