diff mbox series

[kselftest] fix single bpf test

Message ID 20241118140608.53524-1-mrpre@163.com (mailing list archive)
State Handled Elsewhere
Delegated to: BPF
Headers show
Series [kselftest] fix single bpf test | expand

Checks

Context Check Description
netdev/tree_selection success Not a local patch
bpf/vmtest-bpf-next-VM_Test-1 success Logs for ShellCheck
bpf/vmtest-bpf-next-VM_Test-0 success Logs for Lint
bpf/vmtest-bpf-next-VM_Test-2 success Logs for Unittests
bpf/vmtest-bpf-next-VM_Test-3 success Logs for Validate matrix.py
bpf/vmtest-bpf-next-VM_Test-5 success Logs for aarch64-gcc / build-release
bpf/vmtest-bpf-next-VM_Test-6 success Logs for aarch64-gcc / test (test_maps, false, 360) / test_maps on aarch64 with gcc
bpf/vmtest-bpf-next-VM_Test-4 success Logs for aarch64-gcc / build / build for aarch64 with gcc
bpf/vmtest-bpf-next-VM_Test-12 success Logs for s390x-gcc / build-release
bpf/vmtest-bpf-next-VM_Test-9 success Logs for aarch64-gcc / test (test_verifier, false, 360) / test_verifier on aarch64 with gcc
bpf/vmtest-bpf-next-VM_Test-10 success Logs for aarch64-gcc / veristat
bpf/vmtest-bpf-next-VM_Test-7 success Logs for aarch64-gcc / test (test_progs, false, 360) / test_progs on aarch64 with gcc
bpf/vmtest-bpf-next-VM_Test-8 success Logs for aarch64-gcc / test (test_progs_no_alu32, false, 360) / test_progs_no_alu32 on aarch64 with gcc
bpf/vmtest-bpf-next-VM_Test-11 success Logs for s390x-gcc / build / build for s390x with gcc
bpf/vmtest-bpf-next-VM_Test-17 success Logs for set-matrix
bpf/vmtest-bpf-next-VM_Test-16 success Logs for s390x-gcc / veristat
bpf/vmtest-bpf-next-VM_Test-18 success Logs for x86_64-gcc / build / build for x86_64 with gcc
bpf/vmtest-bpf-next-VM_Test-27 success Logs for x86_64-llvm-17 / build / build for x86_64 with llvm-17
bpf/vmtest-bpf-next-VM_Test-19 success Logs for x86_64-gcc / build-release
bpf/vmtest-bpf-next-VM_Test-28 success Logs for x86_64-llvm-17 / build-release / build for x86_64 with llvm-17-O2
bpf/vmtest-bpf-next-VM_Test-33 success Logs for x86_64-llvm-17 / veristat
bpf/vmtest-bpf-next-VM_Test-34 success Logs for x86_64-llvm-18 / build / build for x86_64 with llvm-18
bpf/vmtest-bpf-next-VM_Test-35 success Logs for x86_64-llvm-18 / build-release / build for x86_64 with llvm-18-O2
bpf/vmtest-bpf-next-VM_Test-41 success Logs for x86_64-llvm-18 / veristat
bpf/vmtest-bpf-next-VM_Test-15 success Logs for s390x-gcc / test (test_verifier, false, 360) / test_verifier on s390x with gcc
bpf/vmtest-bpf-next-VM_Test-14 fail Logs for s390x-gcc / test (test_progs_no_alu32, false, 360) / test_progs_no_alu32 on s390x with gcc
bpf/vmtest-bpf-next-VM_Test-25 success Logs for x86_64-gcc / test (test_verifier, false, 360) / test_verifier on x86_64 with gcc
bpf/vmtest-bpf-next-VM_Test-32 success Logs for x86_64-llvm-17 / test (test_verifier, false, 360) / test_verifier on x86_64 with llvm-17
bpf/vmtest-bpf-next-VM_Test-40 success Logs for x86_64-llvm-18 / test (test_verifier, false, 360) / test_verifier on x86_64 with llvm-18
bpf/vmtest-bpf-next-VM_Test-20 success Logs for x86_64-gcc / test (test_maps, false, 360) / test_maps on x86_64 with gcc
bpf/vmtest-bpf-next-VM_Test-22 success Logs for x86_64-gcc / test (test_progs_no_alu32, false, 360) / test_progs_no_alu32 on x86_64 with gcc
bpf/vmtest-bpf-next-VM_Test-21 success Logs for x86_64-gcc / test (test_progs, false, 360) / test_progs on x86_64 with gcc
bpf/vmtest-bpf-next-VM_Test-23 success Logs for x86_64-gcc / test (test_progs_no_alu32_parallel, true, 30) / test_progs_no_alu32_parallel on x86_64 with gcc
bpf/vmtest-bpf-next-VM_Test-24 success Logs for x86_64-gcc / test (test_progs_parallel, true, 30) / test_progs_parallel on x86_64 with gcc
bpf/vmtest-bpf-next-VM_Test-26 success Logs for x86_64-gcc / veristat / veristat on x86_64 with gcc
bpf/vmtest-bpf-next-VM_Test-29 success Logs for x86_64-llvm-17 / test (test_maps, false, 360) / test_maps on x86_64 with llvm-17
bpf/vmtest-bpf-next-VM_Test-30 success Logs for x86_64-llvm-17 / test (test_progs, false, 360) / test_progs on x86_64 with llvm-17
bpf/vmtest-bpf-next-VM_Test-31 success Logs for x86_64-llvm-17 / test (test_progs_no_alu32, false, 360) / test_progs_no_alu32 on x86_64 with llvm-17
bpf/vmtest-bpf-next-VM_Test-38 success Logs for x86_64-llvm-18 / test (test_progs_cpuv4, false, 360) / test_progs_cpuv4 on x86_64 with llvm-18
bpf/vmtest-bpf-next-VM_Test-37 success Logs for x86_64-llvm-18 / test (test_progs, false, 360) / test_progs on x86_64 with llvm-18
bpf/vmtest-bpf-next-VM_Test-36 success Logs for x86_64-llvm-18 / test (test_maps, false, 360) / test_maps on x86_64 with llvm-18
bpf/vmtest-bpf-next-VM_Test-39 success Logs for x86_64-llvm-18 / test (test_progs_no_alu32, false, 360) / test_progs_no_alu32 on x86_64 with llvm-18
bpf/vmtest-bpf-next-PR fail PR summary
bpf/vmtest-bpf-next-VM_Test-13 success Logs for s390x-gcc / test (test_progs, false, 360) / test_progs on s390x with gcc

Commit Message

Jiayuan Chen Nov. 18, 2024, 2:06 p.m. UTC 
Currently, when testing a certain target in selftests, executing the
command 'make TARGETS=XX -C tools/testing/selftests' succeeds for non-BPF,
but a similar command fails for BPF:
'''
make TARGETS=bpf -C tools/testing/selftests

make: Entering directory '/linux-kselftest/tools/testing/selftests'
make: *** [Makefile:197: all] Error 1
make: Leaving directory '/linux-kselftest/tools/testing/selftests'
'''

The reason is that the previous commit:
commit 7a6eb7c34a78 ("selftests: Skip BPF seftests by default")
led to the default filtering of bpf in TARGETS which make TARGETS empty.
That commit also mentioned that building BPF tests requires external
commands to run. This caused target like 'bpf' or 'sched_ext' defined
in SKIP_TARGETS to need an additional specification of SKIP_TARGETS as
empty to avoid skipping it, for example:
'''
make TARGETS=bpf SKIP_TARGETS="" -C tools/testing/selftests
'''

If special steps are required to execute certain test, it is extremely
unfair. We need a fairer way to treat different test targets.

This commit provider a way: If a user has specified a single TARGETS,
it indicates an expectation to run the specified target, and thus the
object should not be skipped.

Another way is to change TARGETS to DEFAULT_TARGETS in the Makefile and
then check if the user specified TARGETS and decide whether filter or not,
though this approach requires too many modifications.
Signed-off-by: Jiayuan Chen <mrpre@163.com>
---
 tools/testing/selftests/Makefile | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)


base-commit: 67b6d342fb6d5abfbeb71e0f23141b9b96cf7bb1

Comments

Shuah Khan Nov. 22, 2024, 3:14 p.m. UTC | #1 
On 11/18/24 07:06, Jiayuan Chen wrote:
> Currently, when testing a certain target in selftests, executing the
> command 'make TARGETS=XX -C tools/testing/selftests' succeeds for non-BPF,
> but a similar command fails for BPF:
> '''
> make TARGETS=bpf -C tools/testing/selftests
> 
> make: Entering directory '/linux-kselftest/tools/testing/selftests'
> make: *** [Makefile:197: all] Error 1
> make: Leaving directory '/linux-kselftest/tools/testing/selftests'
> '''
> 
> The reason is that the previous commit:
> commit 7a6eb7c34a78 ("selftests: Skip BPF seftests by default")
> led to the default filtering of bpf in TARGETS which make TARGETS empty.
> That commit also mentioned that building BPF tests requires external
> commands to run. This caused target like 'bpf' or 'sched_ext' defined
> in SKIP_TARGETS to need an additional specification of SKIP_TARGETS as
> empty to avoid skipping it, for example:
> '''
> make TARGETS=bpf SKIP_TARGETS="" -C tools/testing/selftests
> '''
> 
> If special steps are required to execute certain test, it is extremely
> unfair. We need a fairer way to treat different test targets.
> 

Note: Adding Mark, author for commit 7a6eb7c34a78 to the thread

The reason we did this was bpf test depends on newer versions
of LLVM tool chain.

A better solution would be to check for compile time dependencies in
bpf Makefile and check run-time dependencies from bpf test or a wrapper
script invoked from run_tests to the skip the test if test can't run.

I would like to see us go that route over addressing this problem
with SKIP_TARGETS solution.

The commit 7a6eb7c34a78 went in 4 years ago? DO we have a better
story for the LLVM tool chain to get rid of skipping bpf and sched_ext?

Running make -C tools/testing/selftests/bpf/ gave me the following error.
Does this mean we still can't include bpf in default run?

make -C tools/testing/selftests/bpf/
make: Entering directory '/linux/linux_6.12/tools/testing/selftests/bpf'

Auto-detecting system features:
...                                    llvm: [ OFF ]


   GEN     /linux/linux_6.12/tools/testing/selftests/bpf/tools/build/bpftool/vmlinux.h
libbpf: failed to find '.BTF' ELF section in /linux/linux_6.12/vmlinux
Error: failed to load BTF from /linux/linux_6.12/vmlinux: No data available
make[1]: *** [Makefile:209: /linux/linux_6.12/tools/testing/selftests/bpf/tools/build/bpftool/vmlinux.h] Error 195
make[1]: *** Deleting file '/linux/linux_6.12/tools/testing/selftests/bpf/tools/build/bpftool/vmlinux.h'
make: *** [Makefile:369: /linux/linux_6.12/tools/testing/selftests/bpf/tools/sbin/bpftool] Error 2
make: Leaving directory '/linux/linux_6.12/tools/testing/selftests/bpf'

> This commit provider a way: If a user has specified a single TARGETS,
> it indicates an expectation to run the specified target, and thus the
> object should not be skipped.
> 
> Another way is to change TARGETS to DEFAULT_TARGETS in the Makefile and
> then check if the user specified TARGETS and decide whether filter or not,
> though this approach requires too many modifications.
> Signed-off-by: Jiayuan Chen <mrpre@163.com>
> ---
>   tools/testing/selftests/Makefile | 7 +++++--
>   1 file changed, 5 insertions(+), 2 deletions(-)
> 
> diff --git a/tools/testing/selftests/Makefile b/tools/testing/selftests/Makefile
> index 363d031a16f7..d76c1781ec09 100644
> --- a/tools/testing/selftests/Makefile
> +++ b/tools/testing/selftests/Makefile
> @@ -116,7 +116,7 @@ TARGETS += vDSO
>   TARGETS += mm
>   TARGETS += x86
>   TARGETS += zram
> -#Please keep the TARGETS list alphabetically sorted
> +# Please keep the TARGETS list alphabetically sorted
>   # Run "make quicktest=1 run_tests" or
>   # "make quicktest=1 kselftest" from top level Makefile
>   
> @@ -132,12 +132,15 @@ endif
>   
>   # User can optionally provide a TARGETS skiplist. By default we skip
>   # targets using BPF since it has cutting edge build time dependencies
> -# which require more effort to install.
> +# If user provide custom TARGETS, we just ignore SKIP_TARGETS so that
> +# user can easy to test single target which defined in SKIP_TARGETS
>   SKIP_TARGETS ?= bpf sched_ext
>   ifneq ($(SKIP_TARGETS),)
> +ifneq ($(words $(TARGETS)), 1)
>   	TMP := $(filter-out $(SKIP_TARGETS), $(TARGETS))
>   	override TARGETS := $(TMP)
>   endif
> +endif
>   
>   # User can set FORCE_TARGETS to 1 to require all targets to be successfully
>   # built; make will fail if any of the targets cannot be built. If
> 
> base-commit: 67b6d342fb6d5abfbeb71e0f23141b9b96cf7bb1

thanks,
-- Shuah
Mark Brown Nov. 25, 2024, 1:11 p.m. UTC | #2 
On Fri, Nov 22, 2024 at 08:14:58AM -0700, Shuah Khan wrote:

> The commit 7a6eb7c34a78 went in 4 years ago? DO we have a better
> story for the LLVM tool chain to get rid of skipping bpf and sched_ext?

> Running make -C tools/testing/selftests/bpf/ gave me the following error.
> Does this mean we still can't include bpf in default run?

> make -C tools/testing/selftests/bpf/
> make: Entering directory '/linux/linux_6.12/tools/testing/selftests/bpf'
> 
> Auto-detecting system features:
> ...                                    llvm: [ OFF ]

The toolchain is in a better place now and we can run with released LLVM
versions.  The detection above is still needed since the LLVM version
needed is a bit newer than the version needed for the kernel itself,
it's LLVM 18 for BPF, and not everyone has LLVM.

>   GEN     /linux/linux_6.12/tools/testing/selftests/bpf/tools/build/bpftool/vmlinux.h
> libbpf: failed to find '.BTF' ELF section in /linux/linux_6.12/vmlinux
> Error: failed to load BTF from /linux/linux_6.12/vmlinux: No data available
> make[1]: *** [Makefile:209: /linux/linux_6.12/tools/testing/selftests/bpf/tools/build/bpftool/vmlinux.h] Error 195
> make[1]: *** Deleting file '/linux/linux_6.12/tools/testing/selftests/bpf/tools/build/bpftool/vmlinux.h'
> make: *** [Makefile:369: /linux/linux_6.12/tools/testing/selftests/bpf/tools/sbin/bpftool] Error 2
> make: Leaving directory '/linux/linux_6.12/tools/testing/selftests/bpf'

This bit still needs some attention - the build needs a kernel binary
with BTF information built in via CONFIG_DEBUG_INFO_BTF.  That is
enabled by the config fragment for BTF tests but it's not compatible
with the arm64 defconfig since that sets CONFIG_DEBUG_INFO_REDUCED which
isn't compatible with _BTF, and in general having it missing should be
handled a bit more gracefully.  I believe some of the tests would run
happily without the BTF information.  

TBH I'm a bit surprised we even tried to do this bit with LLVM not
available...
diff mbox series

Patch

diff --git a/tools/testing/selftests/Makefile b/tools/testing/selftests/Makefile
index 363d031a16f7..d76c1781ec09 100644
--- a/tools/testing/selftests/Makefile
+++ b/tools/testing/selftests/Makefile
@@ -116,7 +116,7 @@  TARGETS += vDSO
 TARGETS += mm
 TARGETS += x86
 TARGETS += zram
-#Please keep the TARGETS list alphabetically sorted
+# Please keep the TARGETS list alphabetically sorted
 # Run "make quicktest=1 run_tests" or
 # "make quicktest=1 kselftest" from top level Makefile
 
@@ -132,12 +132,15 @@  endif
 
 # User can optionally provide a TARGETS skiplist. By default we skip
 # targets using BPF since it has cutting edge build time dependencies
-# which require more effort to install.
+# If user provide custom TARGETS, we just ignore SKIP_TARGETS so that
+# user can easy to test single target which defined in SKIP_TARGETS
 SKIP_TARGETS ?= bpf sched_ext
 ifneq ($(SKIP_TARGETS),)
+ifneq ($(words $(TARGETS)), 1)
 	TMP := $(filter-out $(SKIP_TARGETS), $(TARGETS))
 	override TARGETS := $(TMP)
 endif
+endif
 
 # User can set FORCE_TARGETS to 1 to require all targets to be successfully
 # built; make will fail if any of the targets cannot be built. If