| Message ID | cover.1646957399.git.delyank@fb.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | Subskeleton support for BPF libraries | expand |
On 3/10/22 4:11 PM, Delyan Kratunov wrote: > In the quest for ever more modularity, a new need has arisen - the ability to > access data associated with a BPF library from a corresponding userspace library. > The catch is that we don't want the userspace library to know about the structure of the > final BPF object that the BPF library is linked into. > > In pursuit of this modularity, this patch series introduces *subskeletons.* > Subskeletons are similar in use and design to skeletons with a couple of differences: > > 1. The generated storage types do not rely on contiguous storage for the library's > variables because they may be interspersed randomly throughout the final BPF object's sections. > > 2. Subskeletons do not own objects and instead require a loaded bpf_object* to > be passed at runtime in order to be initialized. By extension, symbols are resolved at > runtime by parsing the final object's BTF. This has the interesting effect that the same > userspace code can interoperate with the library BPF code *linked into different final objects.* > > 3. Subskeletons allow access to all global variables, programs, and custom maps. They also expose > the internal maps *of the final object*. This allows bpf_var_skeleton objects to contain a bpf_map** > instead of a section name. > > Changes since v1: > - Introduced new strict mode knob for single-routine-in-.text compatibility behavior, which > disproportionately affects library objects. bpftool works in 1.0 mode so subskeleton generation > doesn't have to worry about this now. > - Made bpf_map_btf_value_type_id available earlier and used it wherever applicable. > - Refactoring in bpftool gen.c per review comments. > - Subskels now use typeof() for array and func proto globals to avoid the need for runtime split btf. > - Expanded the subskeleton test to include arrays, custom maps, extern maps, weak symbols, and kconfigs. > - selftests/bpf/Makefile now generates a subskel.h for every skel.h it would make. > > For reference, here is a shortened subskeleton header: > > #ifndef __TEST_SUBSKELETON_LIB_SUBSKEL_H__ > #define __TEST_SUBSKELETON_LIB_SUBSKEL_H__ > > struct test_subskeleton_lib { > struct bpf_object *obj; > struct bpf_object_subskeleton *subskel; > struct { > struct bpf_map *map2; > struct bpf_map *map1; > struct bpf_map *data; > struct bpf_map *rodata; > struct bpf_map *bss; > struct bpf_map *kconfig; > } maps; > struct { > struct bpf_program *lib_perf_handler; > } progs; > struct test_subskeleton_lib__data { > int *var6; > int *var2; > int *var5; > } data; > struct test_subskeleton_lib__rodata { > int *var1; > } rodata; > struct test_subskeleton_lib__bss { > struct { > int var3_1; > __s64 var3_2; > } *var3; > int *libout1; > typeof(int[4]) *var4; > typeof(int (*)()) *fn_ptr; > } bss; > struct test_subskeleton_lib__kconfig { > _Bool *CONFIG_BPF_SYSCALL; > } kconfig; > > static inline struct test_subskeleton_lib * > test_subskeleton_lib__open(const struct bpf_object *src) > { > struct test_subskeleton_lib *obj; > struct bpf_object_subskeleton *s; > int err; > > ... > s = (struct bpf_object_subskeleton *)calloc(1, sizeof(*s)); > ... > > s->var_cnt = 9; > ... > > s->vars[0].name = "var6"; > s->vars[0].map = &obj->maps.data; > s->vars[0].addr = (void**) &obj->data.var6; > ... > > /* maps */ > ... > > /* programs */ > s->prog_cnt = 1; > ... > > err = bpf_object__open_subskeleton(s); > ... > return obj; > } > #endif /* __TEST_SUBSKELETON_LIB_SUBSKEL_H__ */ When I tried to build the patch set with parallel mode (-j), make -C tools/testing/selftests/bpf -j I hit the following errors: /bin/sh: line 1: 3484984 Bus error (core dumped) /home/yhs/work/bpf-next/tools/testing/selftests/bpf/tools/sbin/bpftool gen skeleton /home/yhs/work/bpf-next/tools/testing/selftests/bpf/test_ksyms_weak.linked3.o name test_ksyms_weak > /home/yhs/work/bpf-next/tools/testing/selftests/bpf/test_ksyms_weak.skel.h make: *** [Makefile:496: /home/yhs/work/bpf-next/tools/testing/selftests/bpf/test_ksyms_weak.skel.h] Error 135 make: *** Deleting file '/home/yhs/work/bpf-next/tools/testing/selftests/bpf/test_ksyms_weak.skel.h' make: *** Waiting for unfinished jobs.... make: Leaving directory '/home/yhs/work/bpf-next/tools/testing/selftests/bpf' Probably some make file related issues. I didn't hit this issue before without this patch set. > > Delyan Kratunov (5): > libbpf: add new strict flag for .text subprograms > libbpf: init btf_{key,value}_type_id on internal map open > libbpf: add subskeleton scaffolding > bpftool: add support for subskeletons > selftests/bpf: test subskeleton functionality > > .../bpf/bpftool/Documentation/bpftool-gen.rst | 25 + > tools/bpf/bpftool/bash-completion/bpftool | 14 +- > tools/bpf/bpftool/gen.c | 589 ++++++++++++++++-- > tools/lib/bpf/libbpf.c | 151 ++++- > tools/lib/bpf/libbpf.h | 29 + > tools/lib/bpf/libbpf.map | 2 + > tools/lib/bpf/libbpf_legacy.h | 6 + > tools/testing/selftests/bpf/.gitignore | 1 + > tools/testing/selftests/bpf/Makefile | 10 +- > .../selftests/bpf/prog_tests/subskeleton.c | 83 +++ > .../selftests/bpf/progs/test_subskeleton.c | 23 + > .../bpf/progs/test_subskeleton_lib.c | 56 ++ > .../bpf/progs/test_subskeleton_lib2.c | 16 + > 13 files changed, 919 insertions(+), 86 deletions(-) > create mode 100644 tools/testing/selftests/bpf/prog_tests/subskeleton.c > create mode 100644 tools/testing/selftests/bpf/progs/test_subskeleton.c > create mode 100644 tools/testing/selftests/bpf/progs/test_subskeleton_lib.c > create mode 100644 tools/testing/selftests/bpf/progs/test_subskeleton_lib2.c > > -- > 2.34.1
On Thu, 2022-03-10 at 21:10 -0800, Yonghong Song wrote: > > On 3/10/22 4:11 PM, Delyan Kratunov wrote: [..] > > When I tried to build the patch set with parallel mode (-j), > make -C tools/testing/selftests/bpf -j > I hit the following errors: > > /bin/sh: line 1: 3484984 Bus error (core dumped) > /home/yhs/work/bpf-next/tools/testing/selftests/bpf/tools/sbin/bpftool > gen skeleton > /home/yhs/work/bpf-next/tools/testing/selftests/bpf/test_ksyms_weak.linked3.o > name test_ksyms_weak > > /home/yhs/work/bpf-next/tools/testing/selftests/bpf/test_ksyms_weak.skel.h > make: *** [Makefile:496: > /home/yhs/work/bpf-next/tools/testing/selftests/bpf/test_ksyms_weak.skel.h] > Error 135 > make: *** Deleting file > '/home/yhs/work/bpf-next/tools/testing/selftests/bpf/test_ksyms_weak.skel.h' > make: *** Waiting for unfinished jobs.... > make: Leaving directory > '/home/yhs/work/bpf-next/tools/testing/selftests/bpf' > > Probably some make file related issues. > I didn't hit this issue before without this patch set. Hm, that's interesting, can you reproduce it? I build everything with -j and have not seen any bpftool issues. I also use ASAN for bpftool and that's not complaining about anything either. SIGBUS suggests a memory mapped file was not there. I'll try and come up with ways that can happen, especially given that it's a `gen skeleton` invocation, which I haven't changed at all. --Delyan
On 3/11/22 10:18 AM, Delyan Kratunov wrote: > On Thu, 2022-03-10 at 21:10 -0800, Yonghong Song wrote: >> >> On 3/10/22 4:11 PM, Delyan Kratunov wrote: > [..] >> >> When I tried to build the patch set with parallel mode (-j), >> make -C tools/testing/selftests/bpf -j >> I hit the following errors: >> >> /bin/sh: line 1: 3484984 Bus error (core dumped) >> /home/yhs/work/bpf-next/tools/testing/selftests/bpf/tools/sbin/bpftool >> gen skeleton >> /home/yhs/work/bpf-next/tools/testing/selftests/bpf/test_ksyms_weak.linked3.o >> name test_ksyms_weak > >> /home/yhs/work/bpf-next/tools/testing/selftests/bpf/test_ksyms_weak.skel.h >> make: *** [Makefile:496: >> /home/yhs/work/bpf-next/tools/testing/selftests/bpf/test_ksyms_weak.skel.h] >> Error 135 >> make: *** Deleting file >> '/home/yhs/work/bpf-next/tools/testing/selftests/bpf/test_ksyms_weak.skel.h' >> make: *** Waiting for unfinished jobs.... >> make: Leaving directory >> '/home/yhs/work/bpf-next/tools/testing/selftests/bpf' >> >> Probably some make file related issues. >> I didn't hit this issue before without this patch set. > > Hm, that's interesting, can you reproduce it? I build everything with -j and > have not seen any bpftool issues. I also use ASAN for bpftool and that's not > complaining about anything either. I can reproduce it in 50% of tries with command line: make -C tools/testing/selftests/bpf -j > > SIGBUS suggests a memory mapped file was not there. I'll try and come up with > ways that can happen, especially given that it's a `gen skeleton` invocation, > which I haven't changed at all. > > --Delyan
In the quest for ever more modularity, a new need has arisen - the ability to access data associated with a BPF library from a corresponding userspace library. The catch is that we don't want the userspace library to know about the structure of the final BPF object that the BPF library is linked into. In pursuit of this modularity, this patch series introduces *subskeletons.* Subskeletons are similar in use and design to skeletons with a couple of differences: 1. The generated storage types do not rely on contiguous storage for the library's variables because they may be interspersed randomly throughout the final BPF object's sections. 2. Subskeletons do not own objects and instead require a loaded bpf_object* to be passed at runtime in order to be initialized. By extension, symbols are resolved at runtime by parsing the final object's BTF. This has the interesting effect that the same userspace code can interoperate with the library BPF code *linked into different final objects.* 3. Subskeletons allow access to all global variables, programs, and custom maps. They also expose the internal maps *of the final object*. This allows bpf_var_skeleton objects to contain a bpf_map** instead of a section name. Changes since v1: - Introduced new strict mode knob for single-routine-in-.text compatibility behavior, which disproportionately affects library objects. bpftool works in 1.0 mode so subskeleton generation doesn't have to worry about this now. - Made bpf_map_btf_value_type_id available earlier and used it wherever applicable. - Refactoring in bpftool gen.c per review comments. - Subskels now use typeof() for array and func proto globals to avoid the need for runtime split btf. - Expanded the subskeleton test to include arrays, custom maps, extern maps, weak symbols, and kconfigs. - selftests/bpf/Makefile now generates a subskel.h for every skel.h it would make. For reference, here is a shortened subskeleton header: #ifndef __TEST_SUBSKELETON_LIB_SUBSKEL_H__ #define __TEST_SUBSKELETON_LIB_SUBSKEL_H__ struct test_subskeleton_lib { struct bpf_object *obj; struct bpf_object_subskeleton *subskel; struct { struct bpf_map *map2; struct bpf_map *map1; struct bpf_map *data; struct bpf_map *rodata; struct bpf_map *bss; struct bpf_map *kconfig; } maps; struct { struct bpf_program *lib_perf_handler; } progs; struct test_subskeleton_lib__data { int *var6; int *var2; int *var5; } data; struct test_subskeleton_lib__rodata { int *var1; } rodata; struct test_subskeleton_lib__bss { struct { int var3_1; __s64 var3_2; } *var3; int *libout1; typeof(int[4]) *var4; typeof(int (*)()) *fn_ptr; } bss; struct test_subskeleton_lib__kconfig { _Bool *CONFIG_BPF_SYSCALL; } kconfig; static inline struct test_subskeleton_lib * test_subskeleton_lib__open(const struct bpf_object *src) { struct test_subskeleton_lib *obj; struct bpf_object_subskeleton *s; int err; ... s = (struct bpf_object_subskeleton *)calloc(1, sizeof(*s)); ... s->var_cnt = 9; ... s->vars[0].name = "var6"; s->vars[0].map = &obj->maps.data; s->vars[0].addr = (void**) &obj->data.var6; ... /* maps */ ... /* programs */ s->prog_cnt = 1; ... err = bpf_object__open_subskeleton(s); ... return obj; } #endif /* __TEST_SUBSKELETON_LIB_SUBSKEL_H__ */ Delyan Kratunov (5): libbpf: add new strict flag for .text subprograms libbpf: init btf_{key,value}_type_id on internal map open libbpf: add subskeleton scaffolding bpftool: add support for subskeletons selftests/bpf: test subskeleton functionality .../bpf/bpftool/Documentation/bpftool-gen.rst | 25 + tools/bpf/bpftool/bash-completion/bpftool | 14 +- tools/bpf/bpftool/gen.c | 589 ++++++++++++++++-- tools/lib/bpf/libbpf.c | 151 ++++- tools/lib/bpf/libbpf.h | 29 + tools/lib/bpf/libbpf.map | 2 + tools/lib/bpf/libbpf_legacy.h | 6 + tools/testing/selftests/bpf/.gitignore | 1 + tools/testing/selftests/bpf/Makefile | 10 +- .../selftests/bpf/prog_tests/subskeleton.c | 83 +++ .../selftests/bpf/progs/test_subskeleton.c | 23 + .../bpf/progs/test_subskeleton_lib.c | 56 ++ .../bpf/progs/test_subskeleton_lib2.c | 16 + 13 files changed, 919 insertions(+), 86 deletions(-) create mode 100644 tools/testing/selftests/bpf/prog_tests/subskeleton.c create mode 100644 tools/testing/selftests/bpf/progs/test_subskeleton.c create mode 100644 tools/testing/selftests/bpf/progs/test_subskeleton_lib.c create mode 100644 tools/testing/selftests/bpf/progs/test_subskeleton_lib2.c -- 2.34.1