| Message ID | 20250109185950.653110-1-ihor.solodrai@pm.me (mailing list archive) |
|---|---|
| Headers | show |
| Series | pahole: faster reproducible BTF encoding | expand |
On Thu, Jan 09, 2025 at 06:59:54PM +0000, Ihor Solodrai wrote: > Note: a resend due to https://lore.kernel.org/dwarves/Z4AWJBNsGJvBU7ZY@x1/ > > This is v4 of the series aiming to speed up parallel reproducible BTF > encoding. This version mostly addresses feedback from Jiri Olsa on v3. > > A notable adition is a patch 10/10, which changes func_states in > btf_encoder from a list to an array. > > Testing: > > vmlinux=/home/theihor/kernels/bpf-next/kbuild-output/.tmp_vmlinux1 PATH=$(realpath build):$PATH ./tests/tests > 1: Validation of BTF encoding of functions; this may take some time: Ok > 2: Default BTF on a system without BTF: Ok > 3: Flexible arrays accounting: pahole: type 'nft_pipapo_elem' not found > pahole: type 'tls_rec' not found > pahole: type 'fuse_direntplus' not found > pahole: type 'nft_rhash_elem' not found > pahole: type 'nft_hash_elem' not found > pahole: type 'nft_bitmap_elem' not found > pahole: type 'ipt_standard' not found > pahole: type 'nft_rule_dp_last' not found > pahole: type 'ip6t_standard' not found > pahole: type 'ipt_error' not found > pahole: type 'ip6t_error' not found > pahole: type 'nft_rbtree_elem' not found > Ok > 4: Check that pfunct can print btf_decl_tags read from BTF: Ok > 5: Pretty printing of files using DWARF type information: Ok > 6: Parallel reproducible DWARF Loading/Serial BTF encoding: Ok > > The warnings about not found types are also present at pahole/next, so > not related to this patchset. > > > Performance check. This patchset (always reproducible): > > Performance counter stats for '/home/theihor/dev/dwarves/build/pahole -J -j24 --btf_features=encode_force,var,float,enum64,decl_tag,type_tag,optimized_func,consistent_func,decl_tag_kfuncs,reproducible_build --btf_encode_detached=/dev/null --lang_exclude=rust /home/theihor/kernels/bpf-next/kbuild-output/.tmp_vmlinux1' (13 runs): > > 5,788.22 msec cpu-clock:u # 3.776 CPUs utilized ( +- 0.17% ) > > 1.53288 +- 0.00334 seconds time elapsed ( +- 0.22% ) > > > pahole/next (d444eb6), parallel non-reproducible: > > Performance counter stats for '/home/theihor/dev/dwarves/build/pahole -J -j24 --btf_features=encode_force,var,float,enum64,decl_tag,type_tag,optimized_func,consistent_func,decl_tag_kfuncs --btf_encode_detached=/dev/null --lang_exclude=rust /home/theihor/kernels/bpf-next/kbuild-output/.tmp_vmlinux1' (13 runs): > > 10,462.38 msec cpu-clock:u # 6.678 CPUs utilized ( +- 0.15% ) > > 1.56670 +- 0.00548 seconds time elapsed ( +- 0.35% ) > > > pahole/next (d444eb6), parallel reproducible: > > Performance counter stats for '/home/theihor/dev/dwarves/build/pahole -J -j24 --btf_features=encode_force,var,float,enum64,decl_tag,type_tag,optimized_func,consistent_func,decl_tag_kfuncs,reproducible_build --btf_encode_detached=/dev/null --lang_exclude=rust /home/theihor/kernels/bpf-next/kbuild-output/.tmp_vmlinux1' (13 runs): > > 6,399.88 msec cpu-clock:u # 3.164 CPUs utilized ( +- 0.22% ) > > 2.02269 +- 0.00359 seconds time elapsed ( +- 0.18% ) > > > v3: https://lore.kernel.org/dwarves/20241221012245.243845-1-ihor.solodrai@pm.me/ > v2: https://lore.kernel.org/dwarves/20241213223641.564002-1-ihor.solodrai@pm.me/ > v1: https://lore.kernel.org/dwarves/20241128012341.4081072-1-ihor.solodrai@pm.me/ > > Alan Maguire (2): > btf_encoder: simplify function encoding > btf_encoder: separate elf function, saved function representations > > Ihor Solodrai (8): > btf_encoder: free encoder->secinfo in btf_encoder__delete > btf_encoder: introduce elf_functions struct type > btf_encoder: introduce elf_functions_list > btf_encoder: remove skip_encoding_inconsistent_proto > dwarf_loader: introduce cu->id > dwarf_loader: multithreading with a job/worker model > btf_encoder: clean up global encoders list > btf_encoder: switch func_states from a list to an array Acked-by: Jiri Olsa <jolsa@kernel.org> thanks, jirka > > btf_encoder.c | 662 +++++++++++++++++++----------------- > btf_encoder.h | 7 +- > btf_loader.c | 2 +- > ctf_loader.c | 2 +- > dwarf_loader.c | 335 ++++++++++++------ > dwarves.c | 44 --- > dwarves.h | 20 +- > pahole.c | 230 ++----------- > pdwtags.c | 3 +- > pfunct.c | 3 +- > tests/reproducible_build.sh | 5 +- > 11 files changed, 623 insertions(+), 690 deletions(-) > > -- > 2.47.1 > >
Note: a resend due to https://lore.kernel.org/dwarves/Z4AWJBNsGJvBU7ZY@x1/ This is v4 of the series aiming to speed up parallel reproducible BTF encoding. This version mostly addresses feedback from Jiri Olsa on v3. A notable adition is a patch 10/10, which changes func_states in btf_encoder from a list to an array. Testing: vmlinux=/home/theihor/kernels/bpf-next/kbuild-output/.tmp_vmlinux1 PATH=$(realpath build):$PATH ./tests/tests 1: Validation of BTF encoding of functions; this may take some time: Ok 2: Default BTF on a system without BTF: Ok 3: Flexible arrays accounting: pahole: type 'nft_pipapo_elem' not found pahole: type 'tls_rec' not found pahole: type 'fuse_direntplus' not found pahole: type 'nft_rhash_elem' not found pahole: type 'nft_hash_elem' not found pahole: type 'nft_bitmap_elem' not found pahole: type 'ipt_standard' not found pahole: type 'nft_rule_dp_last' not found pahole: type 'ip6t_standard' not found pahole: type 'ipt_error' not found pahole: type 'ip6t_error' not found pahole: type 'nft_rbtree_elem' not found Ok 4: Check that pfunct can print btf_decl_tags read from BTF: Ok 5: Pretty printing of files using DWARF type information: Ok 6: Parallel reproducible DWARF Loading/Serial BTF encoding: Ok The warnings about not found types are also present at pahole/next, so not related to this patchset. Performance check. This patchset (always reproducible): Performance counter stats for '/home/theihor/dev/dwarves/build/pahole -J -j24 --btf_features=encode_force,var,float,enum64,decl_tag,type_tag,optimized_func,consistent_func,decl_tag_kfuncs,reproducible_build --btf_encode_detached=/dev/null --lang_exclude=rust /home/theihor/kernels/bpf-next/kbuild-output/.tmp_vmlinux1' (13 runs): 5,788.22 msec cpu-clock:u # 3.776 CPUs utilized ( +- 0.17% ) 1.53288 +- 0.00334 seconds time elapsed ( +- 0.22% ) pahole/next (d444eb6), parallel non-reproducible: Performance counter stats for '/home/theihor/dev/dwarves/build/pahole -J -j24 --btf_features=encode_force,var,float,enum64,decl_tag,type_tag,optimized_func,consistent_func,decl_tag_kfuncs --btf_encode_detached=/dev/null --lang_exclude=rust /home/theihor/kernels/bpf-next/kbuild-output/.tmp_vmlinux1' (13 runs): 10,462.38 msec cpu-clock:u # 6.678 CPUs utilized ( +- 0.15% ) 1.56670 +- 0.00548 seconds time elapsed ( +- 0.35% ) pahole/next (d444eb6), parallel reproducible: Performance counter stats for '/home/theihor/dev/dwarves/build/pahole -J -j24 --btf_features=encode_force,var,float,enum64,decl_tag,type_tag,optimized_func,consistent_func,decl_tag_kfuncs,reproducible_build --btf_encode_detached=/dev/null --lang_exclude=rust /home/theihor/kernels/bpf-next/kbuild-output/.tmp_vmlinux1' (13 runs): 6,399.88 msec cpu-clock:u # 3.164 CPUs utilized ( +- 0.22% ) 2.02269 +- 0.00359 seconds time elapsed ( +- 0.18% ) v3: https://lore.kernel.org/dwarves/20241221012245.243845-1-ihor.solodrai@pm.me/ v2: https://lore.kernel.org/dwarves/20241213223641.564002-1-ihor.solodrai@pm.me/ v1: https://lore.kernel.org/dwarves/20241128012341.4081072-1-ihor.solodrai@pm.me/ Alan Maguire (2): btf_encoder: simplify function encoding btf_encoder: separate elf function, saved function representations Ihor Solodrai (8): btf_encoder: free encoder->secinfo in btf_encoder__delete btf_encoder: introduce elf_functions struct type btf_encoder: introduce elf_functions_list btf_encoder: remove skip_encoding_inconsistent_proto dwarf_loader: introduce cu->id dwarf_loader: multithreading with a job/worker model btf_encoder: clean up global encoders list btf_encoder: switch func_states from a list to an array btf_encoder.c | 662 +++++++++++++++++++----------------- btf_encoder.h | 7 +- btf_loader.c | 2 +- ctf_loader.c | 2 +- dwarf_loader.c | 335 ++++++++++++------ dwarves.c | 44 --- dwarves.h | 20 +- pahole.c | 230 ++----------- pdwtags.c | 3 +- pfunct.c | 3 +- tests/reproducible_build.sh | 5 +- 11 files changed, 623 insertions(+), 690 deletions(-)