[bpf-next] selftests/bpf: Fix flaky selftest lwt_redirect/lwt_reroute

Commit Message

Yonghong Song Feb. 5, 2024, 5:29 a.m. UTC

Recently, when running './test_progs -j', I occasionally hit the
following errors:

  test_lwt_redirect:PASS:pthread_create 0 nsec
  test_lwt_redirect_run:FAIL:netns_create unexpected error: 256 (errno 0)
  #142/2   lwt_redirect/lwt_redirect_normal_nomac:FAIL
  #142     lwt_redirect:FAIL
  test_lwt_reroute:PASS:pthread_create 0 nsec
  test_lwt_reroute_run:FAIL:netns_create unexpected error: 256 (errno 0)
  test_lwt_reroute:PASS:pthread_join 0 nsec
  #143/2   lwt_reroute/lwt_reroute_qdisc_dropped:FAIL
  #143     lwt_reroute:FAIL

The netns_create() definition looks like below:

  #define NETNS "ns_lwt"
  static inline int netns_create(void)
  {
        return system("ip netns add " NETNS);
  }

One possibility is that both lwt_redirect and lwt_reroute create
netns with the same name "ns_lwt" which may cause conflict. I tried
the following example:
  $ sudo ip netns add abc
  $ echo $?
  0
  $ sudo ip netns add abc
  Cannot create namespace file "/var/run/netns/abc": File exists
  $ echo $?
  1
  $

The return code for above netns_create() is 256. The internet search
suggests that the return value for 'ip netns add ns_lwt' is 1, which
matches the above 'sudo ip netns add abc' example.

This patch tried to use different netns names for two tests to avoid
'ip netns add <name>' failure.

I ran './test_progs -j' 10 times and all succeeded with
lwt_redirect/lwt_reroute tests.

Signed-off-by: Yonghong Song <[e-mail]>
---
 tools/testing/selftests/bpf/prog_tests/lwt_helpers.h  | 2 --
 tools/testing/selftests/bpf/prog_tests/lwt_redirect.c | 1 +
 tools/testing/selftests/bpf/prog_tests/lwt_reroute.c  | 1 +
 3 files changed, 2 insertions(+), 2 deletions(-)

Comments

Eduard Zingerman Feb. 5, 2024, 4:02 p.m. UTC | #1

On Sun, 2024-02-04 at 21:29 -0800, Yonghong Song wrote:
> Recently, when running './test_progs -j', I occasionally hit the
> following errors:
> 
>   test_lwt_redirect:PASS:pthread_create 0 nsec
>   test_lwt_redirect_run:FAIL:netns_create unexpected error: 256 (errno 0)
>   #142/2   lwt_redirect/lwt_redirect_normal_nomac:FAIL
>   #142     lwt_redirect:FAIL
>   test_lwt_reroute:PASS:pthread_create 0 nsec
>   test_lwt_reroute_run:FAIL:netns_create unexpected error: 256 (errno 0)
>   test_lwt_reroute:PASS:pthread_join 0 nsec
>   #143/2   lwt_reroute/lwt_reroute_qdisc_dropped:FAIL
>   #143     lwt_reroute:FAIL
> 
> The netns_create() definition looks like below:
> 
>   #define NETNS "ns_lwt"
>   static inline int netns_create(void)
>   {
>         return system("ip netns add " NETNS);
>   }
> 
> One possibility is that both lwt_redirect and lwt_reroute create
> netns with the same name "ns_lwt" which may cause conflict. I tried
> the following example:
>   $ sudo ip netns add abc
>   $ echo $?
>   0
>   $ sudo ip netns add abc
>   Cannot create namespace file "/var/run/netns/abc": File exists
>   $ echo $?
>   1
>   $
> 
> The return code for above netns_create() is 256. The internet search
> suggests that the return value for 'ip netns add ns_lwt' is 1, which
> matches the above 'sudo ip netns add abc' example.
> 
> This patch tried to use different netns names for two tests to avoid
> 'ip netns add <name>' failure.
> 
> I ran './test_progs -j' 10 times and all succeeded with
> lwt_redirect/lwt_reroute tests.
> 
> Signed-off-by: Yonghong Song <[e-mail]>
> ---

Tested-by: Eduard Zingerman <[e-mail]>

Can confirm, w/o this patch tests reliably step on each others toes
when using command:

  ./test_progs -j  -a "lwt_*"
  
This patch resolves the issue.

[e-mail] Feb. 5, 2024, 7:20 p.m. UTC | #2

Hello:

This patch was applied to bpf/bpf-next.git (master)
by Andrii Nakryiko <[e-mail]>:

On Sun,  4 Feb 2024 21:29:14 -0800 you wrote:
> Recently, when running './test_progs -j', I occasionally hit the
> following errors:
> 
>   test_lwt_redirect:PASS:pthread_create 0 nsec
>   test_lwt_redirect_run:FAIL:netns_create unexpected error: 256 (errno 0)
>   #142/2   lwt_redirect/lwt_redirect_normal_nomac:FAIL
>   #142     lwt_redirect:FAIL
>   test_lwt_reroute:PASS:pthread_create 0 nsec
>   test_lwt_reroute_run:FAIL:netns_create unexpected error: 256 (errno 0)
>   test_lwt_reroute:PASS:pthread_join 0 nsec
>   #143/2   lwt_reroute/lwt_reroute_qdisc_dropped:FAIL
>   #143     lwt_reroute:FAIL
> 
> [...]

Here is the summary with links:
  - [bpf-next] selftests/bpf: Fix flaky selftest lwt_redirect/lwt_reroute
    https://git.kernel.org/bpf/bpf-next/c/e7f31873176a

You are awesome, thank you!

Patch

diff --git a/tools/testing/selftests/bpf/prog_tests/lwt_helpers.h b/tools/testing/selftests/bpf/prog_tests/lwt_helpers.h
index e9190574e79f..fb1eb8c67361 100644
--- a/tools/testing/selftests/bpf/prog_tests/lwt_helpers.h
+++ b/tools/testing/selftests/bpf/prog_tests/lwt_helpers.h
@@ -27,8 +27,6 @@ 
 			}                                                     \
 	})
 
-#define NETNS "ns_lwt"
-
 static inline int netns_create(void)
 {
 	return system("ip netns add " NETNS);
diff --git a/tools/testing/selftests/bpf/prog_tests/lwt_redirect.c b/tools/testing/selftests/bpf/prog_tests/lwt_redirect.c
index b5b9e74b1044..835a1d756c16 100644
--- a/tools/testing/selftests/bpf/prog_tests/lwt_redirect.c
+++ b/tools/testing/selftests/bpf/prog_tests/lwt_redirect.c
@@ -54,6 +54,7 @@ 
 #include <stdbool.h>
 #include <stdlib.h>
 
+#define NETNS "ns_lwt_redirect"
 #include "lwt_helpers.h"
 #include "test_progs.h"
 #include "network_helpers.h"
diff --git a/tools/testing/selftests/bpf/prog_tests/lwt_reroute.c b/tools/testing/selftests/bpf/prog_tests/lwt_reroute.c
index 5610bc76928d..03825d2b45a8 100644
--- a/tools/testing/selftests/bpf/prog_tests/lwt_reroute.c
+++ b/tools/testing/selftests/bpf/prog_tests/lwt_reroute.c
@@ -48,6 +48,7 @@ 
  *  For case 2, force UDP packets to overflow fq limit. As long as kernel
  *  is not crashed, it is considered successful.
  */
+#define NETNS "ns_lwt_reroute"
 #include "lwt_helpers.h"
 #include "network_helpers.h"
 #include <linux/net_tstamp.h>