| Message ID | 20250122062332.577009-1-irogers@google.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | Support dynamic opening of capstone/llvm remove BUILD_NONDISTRO | expand |
On Tue, Jan 21, 2025 at 10:23:15PM -0800, Ian Rogers wrote: > Linking against libcapstone and libLLVM can be a significant increase > in dependencies and size of memory footprint. For something like `perf > record` the disassembler and addr2line functionality won't be > used. Support dynamically loading these libraries using dlopen and > then calling the appropriate functions found using dlsym. It's unclear to me what this actually fixes. If the code is not used it should not be faulted in and the dynamic linker is lazy too, so if it's not used, it won't even be linked. I don't see any numbers, but it won't surprise me if it improved actual run time or memory usage significantly. -Andi
On Wed, Jan 22, 2025 at 7:21 AM Andi Kleen <ak@linux.intel.com> wrote: > > On Tue, Jan 21, 2025 at 10:23:15PM -0800, Ian Rogers wrote: > > Linking against libcapstone and libLLVM can be a significant increase > > in dependencies and size of memory footprint. For something like `perf > > record` the disassembler and addr2line functionality won't be > > used. Support dynamically loading these libraries using dlopen and > > then calling the appropriate functions found using dlsym. > > It's unclear to me what this actually fixes. If the code is not used > it should not be faulted in and the dynamic linker is lazy too, so > if it's not used, it won't even be linked. > > I don't see any numbers, but it won't surprise me if it improved > actual run time or memory usage significantly. In certain scenarios, like data centers, it can be useful to statically link all your dependencies to avoid dll hell. The X86 disassembler alone in libllvm is of a size comparable to the perf tool - I think this speaks to us doing a reasonably good job of size optimization of the events/metrics in the perf tool. We want these dependencies for the performance over forking objdump and addr2line, but we don't want it baked in - unless the person doing the build wants this and this is still the default if the libraries are detected by Makefile.config. Using dlopen also means distributions can have a perf tool that doesn't drag in libLLVM.so and a universe of dependencies, but when it is installed get the performance advantages. In data centres having fast disassembly/addr2line is less of a priority over the binary size cost replicated over 10,000s of machines because those machines don't tend to be running the annotate/report commands. Fwiw, Namhyung's uftrace is doing something similar for python: https://github.com/namhyung/uftrace/blob/master/utils/script-python.c#L139 and I wish the perf tool were also doing this. I think it is much nicer to have the tool fail at runtime because of a missing dependency which you can then install should you want it, rather than doing an equivalent within the code base with #ifdefs and needing users to recompile. This patch series significantly reduces the #ifdefs in places like the core disasm code. Thanks, Ian
> In certain scenarios, like data centers, it can be useful to > statically link all your dependencies to avoid dll hell. Yes but it won't be loaded into memory if not used. Executable loading is all lazy. Maybe look a page fault trace for loading perf if you don't believe me. So you're trying to optimize disk space here? I didn't see that in the cover letter. It doesn't seem like a very good reason for such an intrusive patch kit. If it's a serious concern maybe investigate an executable compressor? > The X86 > disassembler alone in libllvm is of a size comparable to the perf tool I agree that LLVM is a serious bloat and DLL hell concern, but I don't think dlopen is the answer here. -Andi
On Thu, Jan 23, 2025 at 10:19 AM Andi Kleen <ak@linux.intel.com> wrote: > > > In certain scenarios, like data centers, it can be useful to > > statically link all your dependencies to avoid dll hell. > > Yes but it won't be loaded into memory if not used. Executable > loading is all lazy. Maybe look a page fault trace for loading > perf if you don't believe me. > > So you're trying to optimize disk space here? > > I didn't see that in the cover letter. For me yes, for distributions it is dependencies. This is already in the v3 message: https://lore.kernel.org/lkml/20250122174308.350350-1-irogers@google.com/ > It doesn't seem like a very good reason for such an intrusive patch kit. The capstone and LLVM code is preexisting. Moving the capstone/llvm code to their own files isn't dependent on dlopen, it does make it nicer to have a single place we're doing dlopen. The change to shim the capstone/LLVM calls looks like this: https://github.com/googleprodkernel/linux-perf/blob/google_tools_master/tools/perf/util/llvm.c#L160-L182 That is a shim is introduced that either calls through to the function if we're linking against libcapstone/llvm or does the dlsym. There are 7 such functions in the LLVM code. I don't think shimming 7 functions is at the scale of hugely intrusive. > If it's a serious concern maybe investigate an executable compressor? Perhaps just have a squashfs partition. Fwiw, excluding dependencies I think compression on the events is a good solution. Convert json events/metrics to a sysfs file with the cpuid in the path, add the compressed file to the binary as data, find "json" events by iterating the directories in the compressed file, etc. A single filesystem approach to event lookup can mean we do some kind of unionfs style lookup of events, which could support users adding their own events/metrics in a directory. Zip doesn't support compressing across files, which is something of a requirement here, other formats do but it's a case of optimizing for some kind of libarchive sweet spot. The opportunity here is that about 70% of the binary is event encodings, a compressed file is about 30% of the current binary size, so we could reduce the binary size by about 40%. > > The X86 > > disassembler alone in libllvm is of a size comparable to the perf tool > > I agree that LLVM is a serious bloat and DLL hell concern, but I don't think > dlopen is the answer here. Agreed, but it's where the code is at. addr2line command or use LLVM for some performance. I think having an inbuilt solution would be best longer term - we spend energy trying to parse and understand text output from tools/libraries when the information is just sitting there in the instruction encoding. Such a solution would be brittle for things like new dwarf information, so we may want to have fallbacks like LLVM but having a loosely coupled dependency using dlopen feels preferable there, to aid package maintainers. Thanks, Ian
On Tue, Jan 21, 2025 at 10:23:15PM -0800, Ian Rogers wrote: > Linking against libcapstone and libLLVM can be a significant increase > in dependencies and size of memory footprint. For something like `perf > record` the disassembler and addr2line functionality won't be > used. Support dynamically loading these libraries using dlopen and > then calling the appropriate functions found using dlsym. It's not clear from the description how you would use dlopen/dlsym. Based on an offline discussion, you want to leave the current linking model as is, and to support dlopen/dlsym when it's NOT detected at build-time, right? For that, you need to carry some definitions of the functions and types for the used APIs. But I'm not sure if it's right to carry them in the perf code base. > > BUILD_NONDISTRO is used to build perf against the license incompatible > libbfd and libiberty libraries. As this has been opt-in for nearly 2 > years, commit dd317df07207 ("perf build: Make binutil libraries opt > in"), remove the code to simplify the code base. This part can be a separate series. > > The patch series: > 1) does some initial clean up; > 2) moves the capstone and LLVM code to their own C files, > 3) simplifies a little the capstone code; I like changes up to this in general. Let me take a look at the patches. Thanks, Namhyung > 4) adds perf_ variants of the functions that will either directly call > the function or use dlsym to discover it; > 5) adds BPF JIT disassembly support to LLVM and capstone disassembly; > 6) removes the BUILD_NONDISTRO code, reduces scope and removes what's possible. > > The addr2line LLVM functionality is written in C++. To avoid linking > against libLLVM for this, a new LIBLLVM_DYNAMIC option is added where > the C++ code with the libLLVM dependency will be built into a > libperf-llvm.so and that dlsym-ed and called against. Ideally LLVM > would extend their C API to avoid this. > > The libbfd BPF disassembly supported source lines, this wasn't ported > to the capstone and LLVM disassembly. > > v2: Add mangling of the function names in libperf-llvm.so to avoid > potential infinite recursion. Add BPF JIT disassembly support to > LLVM and capstone. Add/rebase the BUILD_NONDISTRO cleanup onto the > series from: > https://lore.kernel.org/lkml/20250111202851.1075338-1-irogers@google.com/ > Some other minor additional clean up. > > Ian Rogers (17): > perf build: Remove libtracefs configuration > perf map: Constify objdump offset/address conversion APIs > perf capstone: Move capstone functionality into its own file > perf llvm: Move llvm functionality into its own file > perf capstone: Remove open_capstone_handle > perf capstone: Support for dlopen-ing libcapstone.so > perf llvm: Support for dlopen-ing libLLVM.so > perf llvm: Mangle libperf-llvm.so function names > perf dso: Move read_symbol from llvm/capstone to dso > perf dso: Support BPF programs in dso__read_symbol > perf llvm: Disassemble cleanup > perf dso: Clean up read_symbol error handling > perf build: Remove libbfd support > perf build: Remove libiberty support > perf build: Remove unused defines > perf disasm: Remove disasm_bpf > perf disasm: Make ins__scnprintf and ins__is_nop static > > tools/perf/Documentation/perf-check.txt | 1 - > tools/perf/Makefile.config | 90 +--- > tools/perf/Makefile.perf | 35 +- > tools/perf/builtin-check.c | 1 - > tools/perf/builtin-script.c | 2 - > tools/perf/tests/Build | 1 - > tools/perf/tests/builtin-test.c | 1 - > tools/perf/tests/make | 4 +- > tools/perf/tests/pe-file-parsing.c | 101 ---- > tools/perf/tests/tests.h | 1 - > tools/perf/util/Build | 5 +- > tools/perf/util/annotate.h | 1 - > tools/perf/util/capstone.c | 682 ++++++++++++++++++++++++ > tools/perf/util/capstone.h | 24 + > tools/perf/util/demangle-cxx.cpp | 22 +- > tools/perf/util/disasm.c | 632 +--------------------- > tools/perf/util/disasm.h | 5 +- > tools/perf/util/disasm_bpf.c | 195 ------- > tools/perf/util/disasm_bpf.h | 12 - > tools/perf/util/dso.c | 98 ++++ > tools/perf/util/dso.h | 4 + > tools/perf/util/llvm-c-helpers.cpp | 120 ++++- > tools/perf/util/llvm-c-helpers.h | 24 +- > tools/perf/util/llvm.c | 489 +++++++++++++++++ > tools/perf/util/llvm.h | 24 + > tools/perf/util/map.c | 19 +- > tools/perf/util/map.h | 6 +- > tools/perf/util/print_insn.c | 117 +--- > tools/perf/util/srcline.c | 306 +---------- > tools/perf/util/srcline.h | 6 + > tools/perf/util/symbol-elf.c | 95 ---- > tools/perf/util/symbol.c | 135 ----- > tools/perf/util/symbol.h | 4 - > 33 files changed, 1552 insertions(+), 1710 deletions(-) > delete mode 100644 tools/perf/tests/pe-file-parsing.c > create mode 100644 tools/perf/util/capstone.c > create mode 100644 tools/perf/util/capstone.h > delete mode 100644 tools/perf/util/disasm_bpf.c > delete mode 100644 tools/perf/util/disasm_bpf.h > create mode 100644 tools/perf/util/llvm.c > create mode 100644 tools/perf/util/llvm.h > > -- > 2.48.0.rc2.279.g1de40edade-goog >
On Thu, Jan 23, 2025 at 1:59 PM Namhyung Kim <namhyung@kernel.org> wrote: > > On Tue, Jan 21, 2025 at 10:23:15PM -0800, Ian Rogers wrote: > > Linking against libcapstone and libLLVM can be a significant increase > > in dependencies and size of memory footprint. For something like `perf > > record` the disassembler and addr2line functionality won't be > > used. Support dynamically loading these libraries using dlopen and > > then calling the appropriate functions found using dlsym. > > It's not clear from the description how you would use dlopen/dlsym. > Based on an offline discussion, you want to leave the current linking > model as is, and to support dlopen/dlsym when it's NOT detected at > build-time, right? Yep. Current behavior is no header file than these options fail, new behavior is that we try to use dlopen/dlsym and fail if the dlopen fails. > For that, you need to carry some definitions of the functions and types > for the used APIs. But I'm not sure if it's right to carry them in the > perf code base. Right. I mention that here: https://lore.kernel.org/lkml/CAP-5=fUhNuybCU-2_5EgcCwgwXnxvyFMvyhzKe=ZP1bssQwXHw@mail.gmail.com/ For LLVM we need 3 typedefs and 5 #defines, for capstone we need 2 structs and 5 enums (if we #ifdef some x86 only formatting code). The problem in not carrying those definitions is: 1) if the header file isn't present a build won't support LLVM/capstone even by dlopen - everything falls through to objdump/addr2line that have known performance issues; 2) package maintainers either need to spot a warning message to realize they've done this by having a missing header file (hard to spot and brittle) or we require the build to fail and people without capstone.h opt out of the build error with NO_CAPSTONE=1 - something perf developers will probably not like; 3) the LLVM/capstone code needs #ifdefs and __maybe_unused to suppress compiler warnings, or perhaps we have a minimal version of those files, leading to extra code complexity. I believe the approach here is no worse than what we do with vmlinux.h for BPF code and is robust as depending on dlsym being able to look up the function names. It is not perfect but I think it is more perfect and less complex than the alternative. > > > > BUILD_NONDISTRO is used to build perf against the license incompatible > > libbfd and libiberty libraries. As this has been opt-in for nearly 2 > > years, commit dd317df07207 ("perf build: Make binutil libraries opt > > in"), remove the code to simplify the code base. > > This part can be a separate series. Right, I posted it as a series here: https://lore.kernel.org/lkml/20250111202851.1075338-1-irogers@google.com/ as mentioned in the v2 notes below. The issue was that Arnaldo pointed out removing BUILD_NONDISTRO removed disassemble_bpf that had only been implemented for libbfd. This series adds the LLVM/capstone definitions built into the symbol__disassemble refactor. Merging that series would conflict with this series, so I posted everything together to avoid having series of patches depending upon one another. I also wanted to check that what is in disasm.c in the end is reasonable, which I believe it is with significantly reduced ifdef-ery. > > > > The patch series: > > 1) does some initial clean up; > > 2) moves the capstone and LLVM code to their own C files, > > 3) simplifies a little the capstone code; > > I like changes up to this in general. Let me take a look at the > patches. Thanks, Ian Ian > Thanks, > Namhyung
On Thu, Jan 23, 2025 at 3:36 PM Ian Rogers <irogers@google.com> wrote: > On Thu, Jan 23, 2025 at 1:59 PM Namhyung Kim <namhyung@kernel.org> wrote: > > I like changes up to this in general. Let me take a look at the > > patches. So it would be nice to make progress with this series given some level of happiness, I don't see any actions currently on the patch series as is. If I may be so bold as to recap the issues that have come up: 1) Andi Kleen mentions that dlopen is inferior to linking against libraries and those libraries aren't a memory overhead if unused. I agree but pointed-out the data center use case means that saving size on binaries can be important to some (me). We've also been trying to reduce perf's dependencies for distributions as perf dragging in say the whole of libLLVM can be annoying for making minimal distributions that contain perf. Perhaps somebody (Arnaldo?) more involved with distributions can confirm or deny the distribution problem, I'm hoping it is self-evident. 2) Namhyung Kim was uncomfortable with the code defining types/constants that were in header files as the two may drift over time I agree but in the same way as a function name is an ABI for dlysym, the types/constants are too. Yes a header file may change, but in doing so the ABI has changed and so it would be an incompatible change and everything would be broken. We'd need to fix the code for this, say as we did when libbpf moved to version 1.0, but using a header file would only weakly guard against this problem. The problem with including the header files is that then the build either breaks without the header or we need to support a no linking against a library and not using dlopen case. I suspect a lot of distributions wouldn't understand the build subtlety in this, the necessary build options and things installed, and we'd end up not using things like libLLVM even when it is known to be a large performance win. I also hope one day we can move from parsing text out of forked commands, as it is slower and more brittle, to just directly using libraries. Making dlopen the fallback (probably with a warning on failure) seems like the right direction for this except we won't get it if we need to drag in extra dependency header files for the build to succeed (well we could have a no library or dlopen option, but then we'd probably find distributions packaging this and things like perf annotate getting broken as they don't even know how to dlopen a library). 3) Namhyung Kim (and I) also raises that the libcapstone patch can be smaller by dropping the print_capstone_detail support on x86 Note, given the similarity between capstone and libLLVM for disassembly, it is curious that only capstone gives the extra detail. I agree. Given the capstone disassembly output will be compromised we should warn for this, probably in Makefile.config to avoid running afoul of -Werror. It isn't clear that having a warning is a good move given the handful of structs needed to support print_capstone_detail. I'd prefer to keep the structs so that we haven't got a warning that looks like it needs cleaning up. 4) Namhyung Kim raised concerns over #if placement Namhyung raised that he'd prefer: ``` #if HAVE_LIBCAPSTONE_SUPPORT // lots of code #else // lots of code #endif ``` rather than the #ifs being inside or around individual functions. I raised that the large #ifs is a problem in the current code as you lose context when trying to understand a function. You may look at a function but not realize it isn't being used because of a #if 10s or 100s of lines above. Namhyung raised that the large #ifs is closer to kernel style, I disagreed as I think kernel style is only doing this when it stubs out a bunch of API functions, not when more context would be useful. Hopefully as the person writing the patches the style choice I've made can be respected. 5) Daniel Xu raised issues with the removal of libbfd for Rust support, as the code implies libbfd C++ demangling is a pre-requisite of legacy rust symbol demangling A separate patch was posted adding Rust v0 symbol demangling with no libbfd dependency: https://lore.kernel.org/lkml/20250129193037.573431-1-irogers@google.com/ The legacy support should work with the non-libbfd demanglers as that's what we have today. We should really clean up Rust demangling and have tests. This is blocked on the Rust community responding to: https://github.com/rust-lang/rust/issues/60705 Thanks, Ian
Linking against libcapstone and libLLVM can be a significant increase in dependencies and size of memory footprint. For something like `perf record` the disassembler and addr2line functionality won't be used. Support dynamically loading these libraries using dlopen and then calling the appropriate functions found using dlsym. BUILD_NONDISTRO is used to build perf against the license incompatible libbfd and libiberty libraries. As this has been opt-in for nearly 2 years, commit dd317df07207 ("perf build: Make binutil libraries opt in"), remove the code to simplify the code base. The patch series: 1) does some initial clean up; 2) moves the capstone and LLVM code to their own C files, 3) simplifies a little the capstone code; 4) adds perf_ variants of the functions that will either directly call the function or use dlsym to discover it; 5) adds BPF JIT disassembly support to LLVM and capstone disassembly; 6) removes the BUILD_NONDISTRO code, reduces scope and removes what's possible. The addr2line LLVM functionality is written in C++. To avoid linking against libLLVM for this, a new LIBLLVM_DYNAMIC option is added where the C++ code with the libLLVM dependency will be built into a libperf-llvm.so and that dlsym-ed and called against. Ideally LLVM would extend their C API to avoid this. The libbfd BPF disassembly supported source lines, this wasn't ported to the capstone and LLVM disassembly. v2: Add mangling of the function names in libperf-llvm.so to avoid potential infinite recursion. Add BPF JIT disassembly support to LLVM and capstone. Add/rebase the BUILD_NONDISTRO cleanup onto the series from: https://lore.kernel.org/lkml/20250111202851.1075338-1-irogers@google.com/ Some other minor additional clean up. Ian Rogers (17): perf build: Remove libtracefs configuration perf map: Constify objdump offset/address conversion APIs perf capstone: Move capstone functionality into its own file perf llvm: Move llvm functionality into its own file perf capstone: Remove open_capstone_handle perf capstone: Support for dlopen-ing libcapstone.so perf llvm: Support for dlopen-ing libLLVM.so perf llvm: Mangle libperf-llvm.so function names perf dso: Move read_symbol from llvm/capstone to dso perf dso: Support BPF programs in dso__read_symbol perf llvm: Disassemble cleanup perf dso: Clean up read_symbol error handling perf build: Remove libbfd support perf build: Remove libiberty support perf build: Remove unused defines perf disasm: Remove disasm_bpf perf disasm: Make ins__scnprintf and ins__is_nop static tools/perf/Documentation/perf-check.txt | 1 - tools/perf/Makefile.config | 90 +--- tools/perf/Makefile.perf | 35 +- tools/perf/builtin-check.c | 1 - tools/perf/builtin-script.c | 2 - tools/perf/tests/Build | 1 - tools/perf/tests/builtin-test.c | 1 - tools/perf/tests/make | 4 +- tools/perf/tests/pe-file-parsing.c | 101 ---- tools/perf/tests/tests.h | 1 - tools/perf/util/Build | 5 +- tools/perf/util/annotate.h | 1 - tools/perf/util/capstone.c | 682 ++++++++++++++++++++++++ tools/perf/util/capstone.h | 24 + tools/perf/util/demangle-cxx.cpp | 22 +- tools/perf/util/disasm.c | 632 +--------------------- tools/perf/util/disasm.h | 5 +- tools/perf/util/disasm_bpf.c | 195 ------- tools/perf/util/disasm_bpf.h | 12 - tools/perf/util/dso.c | 98 ++++ tools/perf/util/dso.h | 4 + tools/perf/util/llvm-c-helpers.cpp | 120 ++++- tools/perf/util/llvm-c-helpers.h | 24 +- tools/perf/util/llvm.c | 489 +++++++++++++++++ tools/perf/util/llvm.h | 24 + tools/perf/util/map.c | 19 +- tools/perf/util/map.h | 6 +- tools/perf/util/print_insn.c | 117 +--- tools/perf/util/srcline.c | 306 +---------- tools/perf/util/srcline.h | 6 + tools/perf/util/symbol-elf.c | 95 ---- tools/perf/util/symbol.c | 135 ----- tools/perf/util/symbol.h | 4 - 33 files changed, 1552 insertions(+), 1710 deletions(-) delete mode 100644 tools/perf/tests/pe-file-parsing.c create mode 100644 tools/perf/util/capstone.c create mode 100644 tools/perf/util/capstone.h delete mode 100644 tools/perf/util/disasm_bpf.c delete mode 100644 tools/perf/util/disasm_bpf.h create mode 100644 tools/perf/util/llvm.c create mode 100644 tools/perf/util/llvm.h