| Message ID | 20230717-upstream-net-next-20230712-selftests-mptcp-subtests-v1-0-695127e0ad83@tessares.net (mailing list archive) |
|---|---|
| Headers | show |
| Series | selftests: mptcp: format subtests results in TAP | expand |
Hello: This series was applied to netdev/net-next.git (main) by David S. Miller <davem@davemloft.net>: On Mon, 17 Jul 2023 15:21:20 +0200 you wrote: > The current selftests infrastructure formats the results in TAP 13. This > version doesn't support subtests and only the end result of each > selftest is taken into account. It means that a single issue in a > subtest of a selftest containing multiple subtests forces the whole > selftest to be marked as failed. It also means that subtests results are > not tracked by CI executing selftests. > > [...] Here is the summary with links: - [net-next,01/13] selftests: mptcp: connect: don't stop if error https://git.kernel.org/netdev/net-next/c/edbc16c43b27 - [net-next,02/13] selftests: mptcp: userspace pm: don't stop if error https://git.kernel.org/netdev/net-next/c/e141c1e8e4c1 - [net-next,03/13] selftests: mptcp: userspace_pm: fix shellcheck warnings https://git.kernel.org/netdev/net-next/c/8320b1387a15 - [net-next,04/13] selftests: mptcp: userspace_pm: uniform results printing https://git.kernel.org/netdev/net-next/c/e198ad759273 - [net-next,05/13] selftests: mptcp: userspace_pm: reduce dup code around printf https://git.kernel.org/netdev/net-next/c/d8463d81652d - [net-next,06/13] selftests: mptcp: lib: format subtests results in TAP https://git.kernel.org/netdev/net-next/c/c4192967e62f - [net-next,07/13] selftests: mptcp: connect: format subtests results in TAP https://git.kernel.org/netdev/net-next/c/dd350f46e35e - [net-next,08/13] selftests: mptcp: pm_netlink: format subtests results in TAP https://git.kernel.org/netdev/net-next/c/d85555ac11f9 - [net-next,09/13] selftests: mptcp: join: format subtests results in TAP https://git.kernel.org/netdev/net-next/c/7f117cd37c61 - [net-next,10/13] selftests: mptcp: diag: format subtests results in TAP https://git.kernel.org/netdev/net-next/c/ce9902573652 - [net-next,11/13] selftests: mptcp: simult flows: format subtests results in TAP https://git.kernel.org/netdev/net-next/c/675d99338e7a - [net-next,12/13] selftests: mptcp: sockopt: format subtests results in TAP https://git.kernel.org/netdev/net-next/c/9e86a297796b - [net-next,13/13] selftests: mptcp: userspace_pm: format subtests results in TAP https://git.kernel.org/netdev/net-next/c/f589234e1af0 You are awesome, thank you!
The current selftests infrastructure formats the results in TAP 13. This version doesn't support subtests and only the end result of each selftest is taken into account. It means that a single issue in a subtest of a selftest containing multiple subtests forces the whole selftest to be marked as failed. It also means that subtests results are not tracked by CI executing selftests. MPTCP selftests run hundreds of various subtests. It is then important to track each of them and not one result per selftest. It is particularly interesting to do that when validating stable kernels with the last version of the test suite: tests might fail because a feature is not supported but the test didn't skip that part. In this case, if subtests are not tracked, the whole selftest will be marked as failed making the other subtests useless because their results are ignored. Regarding this patch set: - The two first patches modify connect and userspace_pm selftests to continue executing other tests if there is an error before the end. This is what is done in the other MPTCP selftests. - Patches 3-5 are refactoring the code in userspace_pm selftest to reduce duplicated code, suppress some shellcheck warnings and prepare subtests' support by using new helpers. - Patch 6 adds new helpers in mptcp_lib.sh to easily support printing the subtests results in the different MPTCP selftests. - Patch 7-13 format subtests results in TAP 13 in the different MPTCP selftests. Signed-off-by: Matthieu Baerts <matthieu.baerts@tessares.net> --- Matthieu Baerts (13): selftests: mptcp: connect: don't stop if error selftests: mptcp: userspace pm: don't stop if error selftests: mptcp: userspace_pm: fix shellcheck warnings selftests: mptcp: userspace_pm: uniform results printing selftests: mptcp: userspace_pm: reduce dup code around printf selftests: mptcp: lib: format subtests results in TAP selftests: mptcp: connect: format subtests results in TAP selftests: mptcp: pm_netlink: format subtests results in TAP selftests: mptcp: join: format subtests results in TAP selftests: mptcp: diag: format subtests results in TAP selftests: mptcp: simult flows: format subtests results in TAP selftests: mptcp: sockopt: format subtests results in TAP selftests: mptcp: userspace_pm: format subtests results in TAP tools/testing/selftests/net/mptcp/diag.sh | 7 + tools/testing/selftests/net/mptcp/mptcp_connect.sh | 66 ++++++-- tools/testing/selftests/net/mptcp/mptcp_join.sh | 37 ++++- tools/testing/selftests/net/mptcp/mptcp_lib.sh | 66 ++++++++ tools/testing/selftests/net/mptcp/mptcp_sockopt.sh | 20 ++- tools/testing/selftests/net/mptcp/pm_netlink.sh | 6 +- tools/testing/selftests/net/mptcp/simult_flows.sh | 4 + tools/testing/selftests/net/mptcp/userspace_pm.sh | 181 +++++++++++++-------- 8 files changed, 298 insertions(+), 89 deletions(-) --- base-commit: 60cc1f7d0605598b47ee3c0c2b4b6fbd4da50a06 change-id: 20230712-upstream-net-next-20230712-selftests-mptcp-subtests-25d250d77886 Best regards,