| Message ID | 20240419205747.1102933-3-acme@kernel.org (mailing list archive) |
|---|---|
| State | Not Applicable |
| Headers | show |
| Series | Introduce --btf_features=+extra_features syntax | expand |
| Context | Check | Description |
|---|---|---|
| netdev/tree_selection | success | Not a local patch |
On Fri, Apr 19, 2024 at 1:58 PM Arnaldo Carvalho de Melo <acme@kernel.org> wrote: > > From: Arnaldo Carvalho de Melo <acme@redhat.com> > > Instead of the somewhat confusing: > > --btf_features=all,reproducible_build > > That means "'all' the standard BTF features plus the 'reproducible_build' > extra BTF feature", use + directly, making it more compact: > > --btf_features=+reproducible_build > for older paholes that don't yet know about + syntax, but support --btf_features, will this effectively disable all features or how will it work? I'm thinking from the perspective of using +reproducible_build unconditionally in kernel's build scripts, will it regress something for current pahole versions? > In the future we may want the '-' counterpart as a way to _remove_ some > of the standard set of BTF features. > > Cc: Alan Maguire <alan.maguire@oracle.com> > Cc: Andrii Nakryiko <andrii.nakryiko@gmail.com> > Cc: Daniel Xu <dxu@dxuuu.xyz> > Cc: Eduard Zingerman <eddyz87@gmail.com> > Cc: Jiri Olsa <jolsa@kernel.org> > Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com> > --- > man-pages/pahole.1 | 6 ++++++ > pahole.c | 6 ++++++ > tests/reproducible_build.sh | 2 +- > 3 files changed, 13 insertions(+), 1 deletion(-) > [...]
On Fri, Apr 26, 2024 at 01:26:40PM -0700, Andrii Nakryiko wrote: > On Fri, Apr 19, 2024 at 1:58 PM Arnaldo Carvalho de Melo > <acme@kernel.org> wrote: > > > > From: Arnaldo Carvalho de Melo <acme@redhat.com> > > > > Instead of the somewhat confusing: > > > > --btf_features=all,reproducible_build > > > > That means "'all' the standard BTF features plus the 'reproducible_build' > > extra BTF feature", use + directly, making it more compact: > > > > --btf_features=+reproducible_build > > > > for older paholes that don't yet know about + syntax, but support > --btf_features, will this effectively disable all features or how will > it work? > > I'm thinking from the perspective of using +reproducible_build > unconditionally in kernel's build scripts, will it regress something > for current pahole versions? Well, I think it will end up being discarded just like "all" or "default", no? I.e. those were keywords not grokked by older pahole versions, so ignored as we're not using --btf_features_strict, right? Alan? But then we're not yet using --btf_features in scripts/Makefile.btf, right? But as Daniel pointed out and Alan (I think) agreed, for things like scripts we probably end up using the most verbose thing as: --btf_features=default,reproducible_build to mean a set (the default set of BTF options) + an optional/extra feature (reproducibe_build), that for people not used to the + syntax may be more descriptive (I really think that both are confusing for beginners knowing nothing about BTF and its evolution, etc). Alan, also we released 1.26 with "all" meaning what we now call "default", so we need to keep both meaning the same thing, right? - Arnaldo > > In the future we may want the '-' counterpart as a way to _remove_ some > > of the standard set of BTF features. > > > > Cc: Alan Maguire <alan.maguire@oracle.com> > > Cc: Andrii Nakryiko <andrii.nakryiko@gmail.com> > > Cc: Daniel Xu <dxu@dxuuu.xyz> > > Cc: Eduard Zingerman <eddyz87@gmail.com> > > Cc: Jiri Olsa <jolsa@kernel.org> > > Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com> > > --- > > man-pages/pahole.1 | 6 ++++++ > > pahole.c | 6 ++++++ > > tests/reproducible_build.sh | 2 +- > > 3 files changed, 13 insertions(+), 1 deletion(-) > > > > [...]
On 26/04/2024 21:47, Arnaldo Carvalho de Melo wrote: > On Fri, Apr 26, 2024 at 01:26:40PM -0700, Andrii Nakryiko wrote: >> On Fri, Apr 19, 2024 at 1:58 PM Arnaldo Carvalho de Melo >> <acme@kernel.org> wrote: >>> >>> From: Arnaldo Carvalho de Melo <acme@redhat.com> >>> >>> Instead of the somewhat confusing: >>> >>> --btf_features=all,reproducible_build >>> >>> That means "'all' the standard BTF features plus the 'reproducible_build' >>> extra BTF feature", use + directly, making it more compact: >>> >>> --btf_features=+reproducible_build >>> >> >> for older paholes that don't yet know about + syntax, but support >> --btf_features, will this effectively disable all features or how will >> it work? >> >> I'm thinking from the perspective of using +reproducible_build >> unconditionally in kernel's build scripts, will it regress something >> for current pahole versions? > > Well, I think it will end up being discarded just like "all" or > "default", no? I.e. those were keywords not grokked by older pahole > versions, so ignored as we're not using --btf_features_strict, right? > > Alan? > Yep, it would just be ignored, so wouldn't have the desired behaviour of enabling defaults + reproducible build option. > But then we're not yet using --btf_features in scripts/Makefile.btf, > right? > > But as Daniel pointed out and Alan (I think) agreed, for things like > scripts we probably end up using the most verbose thing as: > > --btf_features=default,reproducible_build > > to mean a set (the default set of BTF options) + an optional/extra > feature (reproducibe_build), that for people not used to the + syntax > may be more descriptive (I really think that both are confusing for > beginners knowing nothing about BTF and its evolution, etc). > > Alan, also we released 1.26 with "all" meaning what we now call > "default", so we need to keep both meaning the same thing, right? > I might be missing something here, but I think we should always call out explicitly the set of features we want in the kernel Makefile.btf (something like [1]). The reason for this is that the concept of what is "default" may evolve over time; for example it's going to include Daniel's kfunc definitions for soon. That's a good thing, but it could conceivably cause problems down the line. Consider a newer pahole - with a newer set of defaults - running on an older kernel. In that case, we could end up encoding BTF features we don't want. By contrast, if we always call out the full set of BTF features we want via --btf_features=feature1,feature2 etc we'll always get the expected set. Plus for folks consulting the code, it's much clearer which BTF features are in use when they look at the Makefiles for a particular kernel. So my sense of the value of "default" is as a shortcut for testing the latest and greatest set of BTF feature encoding, but not for use in the kernel tree Makefiles. Thanks! Alan [1] https://lore.kernel.org/bpf/20240424154806.3417662-7-alan.maguire@oracle.com/ > - Arnaldo > >>> In the future we may want the '-' counterpart as a way to _remove_ some >>> of the standard set of BTF features. >>> >>> Cc: Alan Maguire <alan.maguire@oracle.com> >>> Cc: Andrii Nakryiko <andrii.nakryiko@gmail.com> >>> Cc: Daniel Xu <dxu@dxuuu.xyz> >>> Cc: Eduard Zingerman <eddyz87@gmail.com> >>> Cc: Jiri Olsa <jolsa@kernel.org> >>> Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com> >>> --- >>> man-pages/pahole.1 | 6 ++++++ >>> pahole.c | 6 ++++++ >>> tests/reproducible_build.sh | 2 +- >>> 3 files changed, 13 insertions(+), 1 deletion(-) >>> >> >> [...]
On Mon, Apr 29, 2024 at 4:16 AM Alan Maguire <alan.maguire@oracle.com> wrote: > > On 26/04/2024 21:47, Arnaldo Carvalho de Melo wrote: > > On Fri, Apr 26, 2024 at 01:26:40PM -0700, Andrii Nakryiko wrote: > >> On Fri, Apr 19, 2024 at 1:58 PM Arnaldo Carvalho de Melo > >> <acme@kernel.org> wrote: > >>> > >>> From: Arnaldo Carvalho de Melo <acme@redhat.com> > >>> > >>> Instead of the somewhat confusing: > >>> > >>> --btf_features=all,reproducible_build > >>> > >>> That means "'all' the standard BTF features plus the 'reproducible_build' > >>> extra BTF feature", use + directly, making it more compact: > >>> > >>> --btf_features=+reproducible_build > >>> > >> > >> for older paholes that don't yet know about + syntax, but support > >> --btf_features, will this effectively disable all features or how will > >> it work? > >> > >> I'm thinking from the perspective of using +reproducible_build > >> unconditionally in kernel's build scripts, will it regress something > >> for current pahole versions? > > > > Well, I think it will end up being discarded just like "all" or > > "default", no? I.e. those were keywords not grokked by older pahole > > versions, so ignored as we're not using --btf_features_strict, right? > > > > Alan? > > > > Yep, it would just be ignored, so wouldn't have the desired behaviour > of enabling defaults + reproducible build option. > > > But then we're not yet using --btf_features in scripts/Makefile.btf, > > right? > > > > But as Daniel pointed out and Alan (I think) agreed, for things like > > scripts we probably end up using the most verbose thing as: > > > > --btf_features=default,reproducible_build > > > > to mean a set (the default set of BTF options) + an optional/extra > > feature (reproducibe_build), that for people not used to the + syntax > > may be more descriptive (I really think that both are confusing for > > beginners knowing nothing about BTF and its evolution, etc). > > > > Alan, also we released 1.26 with "all" meaning what we now call > > "default", so we need to keep both meaning the same thing, right? > > > > I might be missing something here, but I think we should always call out > explicitly the set of features we want in the kernel Makefile.btf > (something like [1]). The reason for this is that the concept of what is > "default" may evolve over time; for example it's going to include > Daniel's kfunc definitions for soon. That's a good thing, but it could > conceivably cause problems down the line. Consider a newer pahole - with > a newer set of defaults - running on an older kernel. In that case, we > could end up encoding BTF features we don't want. By contrast, if we > always call out the full set of BTF features we want via > --btf_features=feature1,feature2 etc we'll always get the expected set. > Plus for folks consulting the code, it's much clearer which BTF features > are in use when they look at the Makefiles for a particular kernel. > So my sense of the value of "default" is as a shortcut for testing the > latest and greatest set of BTF feature encoding, but not for use in the > kernel tree Makefiles. Thanks! Yep, I agree, the whole point was to not regress older kernel builds with newer pahole, so we need to explicitly list used features. > > Alan > > [1] > https://lore.kernel.org/bpf/20240424154806.3417662-7-alan.maguire@oracle.com/ > > > - Arnaldo > > > >>> In the future we may want the '-' counterpart as a way to _remove_ some > >>> of the standard set of BTF features. > >>> > >>> Cc: Alan Maguire <alan.maguire@oracle.com> > >>> Cc: Andrii Nakryiko <andrii.nakryiko@gmail.com> > >>> Cc: Daniel Xu <dxu@dxuuu.xyz> > >>> Cc: Eduard Zingerman <eddyz87@gmail.com> > >>> Cc: Jiri Olsa <jolsa@kernel.org> > >>> Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com> > >>> --- > >>> man-pages/pahole.1 | 6 ++++++ > >>> pahole.c | 6 ++++++ > >>> tests/reproducible_build.sh | 2 +- > >>> 3 files changed, 13 insertions(+), 1 deletion(-) > >>> > >> > >> [...]
On Mon, Apr 29, 2024 at 12:16:18PM +0100, Alan Maguire wrote: > On 26/04/2024 21:47, Arnaldo Carvalho de Melo wrote: > > On Fri, Apr 26, 2024 at 01:26:40PM -0700, Andrii Nakryiko wrote: > >> On Fri, Apr 19, 2024 at 1:58 PM Arnaldo Carvalho de Melo <acme@kernel.org> wrote: > >> for older paholes that don't yet know about + syntax, but support > >> --btf_features, will this effectively disable all features or how will > >> it work? > >> > >> I'm thinking from the perspective of using +reproducible_build > >> unconditionally in kernel's build scripts, will it regress something > >> for current pahole versions? > > > > Well, I think it will end up being discarded just like "all" or > > "default", no? I.e. those were keywords not grokked by older pahole > > versions, so ignored as we're not using --btf_features_strict, right? > > Alan? > Yep, it would just be ignored, so wouldn't have the desired behaviour > of enabling defaults + reproducible build option. > > But then we're not yet using --btf_features in scripts/Makefile.btf, > > right? > > But as Daniel pointed out and Alan (I think) agreed, for things like > > scripts we probably end up using the most verbose thing as: > > --btf_features=default,reproducible_build > > to mean a set (the default set of BTF options) + an optional/extra > > feature (reproducibe_build), that for people not used to the + syntax > > may be more descriptive (I really think that both are confusing for > > beginners knowing nothing about BTF and its evolution, etc). > > Alan, also we released 1.26 with "all" meaning what we now call > > "default", so we need to keep both meaning the same thing, right? > I might be missing something here, but I think we should always call out No you're not, I was just talking about what Daniel pointed out, that when using a script using 'default,extra_feature' is more descriptive than '+extra_feature', but you're right, for the kernel we go on adding the features we need and older pahole versions will just ignore those. The explanation you gave on this message is interesting to clarify when 'default' should be used and perhaps also use the kernel build process needs/use of a list of features, etc. It would be good to have it in the man page, can you provide a formal patch doing that? Thanks, - Arnaldo > explicitly the set of features we want in the kernel Makefile.btf > (something like [1]). The reason for this is that the concept of what is > "default" may evolve over time; for example it's going to include > Daniel's kfunc definitions for soon. That's a good thing, but it could > conceivably cause problems down the line. Consider a newer pahole - with > a newer set of defaults - running on an older kernel. In that case, we > could end up encoding BTF features we don't want. By contrast, if we > always call out the full set of BTF features we want via > --btf_features=feature1,feature2 etc we'll always get the expected set. > Plus for folks consulting the code, it's much clearer which BTF features > are in use when they look at the Makefiles for a particular kernel. > So my sense of the value of "default" is as a shortcut for testing the > latest and greatest set of BTF feature encoding, but not for use in the > kernel tree Makefiles. Thanks! > > Alan > > [1] > https://lore.kernel.org/bpf/20240424154806.3417662-7-alan.maguire@oracle.com/ > > > - Arnaldo > > > >>> In the future we may want the '-' counterpart as a way to _remove_ some > >>> of the standard set of BTF features. > >>> > >>> Cc: Alan Maguire <alan.maguire@oracle.com> > >>> Cc: Andrii Nakryiko <andrii.nakryiko@gmail.com> > >>> Cc: Daniel Xu <dxu@dxuuu.xyz> > >>> Cc: Eduard Zingerman <eddyz87@gmail.com> > >>> Cc: Jiri Olsa <jolsa@kernel.org> > >>> Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com> > >>> --- > >>> man-pages/pahole.1 | 6 ++++++ > >>> pahole.c | 6 ++++++ > >>> tests/reproducible_build.sh | 2 +- > >>> 3 files changed, 13 insertions(+), 1 deletion(-) > >>> > >> > >> [...]
diff --git a/man-pages/pahole.1 b/man-pages/pahole.1 index 64de3438b5f9a77a..2f4f42f8323efd6e 100644 --- a/man-pages/pahole.1 +++ b/man-pages/pahole.1 @@ -320,6 +320,12 @@ Supported non-standard features (not enabled for 'all') So for example, specifying \-\-btf_encode=var,enum64 will result in a BTF encoding that (as well as encoding basic BTF information) will contain variables and enum64 values. +.fi + +If one wants to add an extra feature to the set of standard ones, the '+' prefix can be used, i.e.: +\-\-btf_features=+reproducible_build will add all standard features plus the 'reproducible_build' extra +feature. + .TP .B \-\-btf_features_strict Identical to \-\-btf_features above, but pahole will exit if it encounters an unrecognized feature. diff --git a/pahole.c b/pahole.c index af94d2a45ee96cbe..42c5b03ee1d1a8f8 100644 --- a/pahole.c +++ b/pahole.c @@ -1364,6 +1364,12 @@ static void parse_btf_features(const char *features, bool strict) return; } + // Adding extra features to the set of standard features. + if (strstarts(features, "+")) { + btf_features__enable_for_all(); + ++features; + } + strncpy(f, features, BTF_MAX_FEATURE_STR)[BTF_MAX_FEATURE_STR] = '\0'; s = f; while ((feature_name = strtok_r(s, ",", &saveptr)) != NULL) { diff --git a/tests/reproducible_build.sh b/tests/reproducible_build.sh index e2f836081b125119..1222cb42c6639235 100755 --- a/tests/reproducible_build.sh +++ b/tests/reproducible_build.sh @@ -29,7 +29,7 @@ nr_proc=$(getconf _NPROCESSORS_ONLN) for threads in $(seq $nr_proc) ; do test -n "$VERBOSE" && echo $threads threads encoding - pahole -j$threads --btf_features=all,reproducible_build --btf_encode_detached=$outdir/vmlinux.btf.parallel.reproducible $vmlinux & + pahole -j$threads --btf_features=+reproducible_build --btf_encode_detached=$outdir/vmlinux.btf.parallel.reproducible $vmlinux & pahole=$! # HACK: Wait a bit for pahole to start its threads sleep 0.3s