diff mbox series

[2/2] pahole: Allow asking for extra features using the '+' prefix in --btf_features

Message ID 20240419205747.1102933-3-acme@kernel.org (mailing list archive)
State Not Applicable
Headers show
Series Introduce --btf_features=+extra_features syntax | expand

Checks

Context Check Description
netdev/tree_selection success Not a local patch

Commit Message

Arnaldo Carvalho de Melo April 19, 2024, 8:57 p.m. UTC
From: Arnaldo Carvalho de Melo <acme@redhat.com>

Instead of the somewhat confusing:

  --btf_features=all,reproducible_build

That means "'all' the standard BTF features plus the 'reproducible_build'
extra BTF feature", use + directly, making it more compact:

  --btf_features=+reproducible_build

In the future we may want the '-' counterpart as a way to _remove_ some
of the standard set of BTF features.

Cc: Alan Maguire <alan.maguire@oracle.com>
Cc: Andrii Nakryiko <andrii.nakryiko@gmail.com>
Cc: Daniel Xu <dxu@dxuuu.xyz>
Cc: Eduard Zingerman <eddyz87@gmail.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
---
 man-pages/pahole.1          | 6 ++++++
 pahole.c                    | 6 ++++++
 tests/reproducible_build.sh | 2 +-
 3 files changed, 13 insertions(+), 1 deletion(-)

Comments

Andrii Nakryiko April 26, 2024, 8:26 p.m. UTC | #1
On Fri, Apr 19, 2024 at 1:58 PM Arnaldo Carvalho de Melo
<acme@kernel.org> wrote:
>
> From: Arnaldo Carvalho de Melo <acme@redhat.com>
>
> Instead of the somewhat confusing:
>
>   --btf_features=all,reproducible_build
>
> That means "'all' the standard BTF features plus the 'reproducible_build'
> extra BTF feature", use + directly, making it more compact:
>
>   --btf_features=+reproducible_build
>

for older paholes that don't yet know about + syntax, but support
--btf_features, will this effectively disable all features or how will
it work?

I'm thinking from the perspective of using +reproducible_build
unconditionally in kernel's build scripts, will it regress something
for current pahole versions?

> In the future we may want the '-' counterpart as a way to _remove_ some
> of the standard set of BTF features.
>
> Cc: Alan Maguire <alan.maguire@oracle.com>
> Cc: Andrii Nakryiko <andrii.nakryiko@gmail.com>
> Cc: Daniel Xu <dxu@dxuuu.xyz>
> Cc: Eduard Zingerman <eddyz87@gmail.com>
> Cc: Jiri Olsa <jolsa@kernel.org>
> Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
> ---
>  man-pages/pahole.1          | 6 ++++++
>  pahole.c                    | 6 ++++++
>  tests/reproducible_build.sh | 2 +-
>  3 files changed, 13 insertions(+), 1 deletion(-)
>

[...]
Arnaldo Carvalho de Melo April 26, 2024, 8:47 p.m. UTC | #2
On Fri, Apr 26, 2024 at 01:26:40PM -0700, Andrii Nakryiko wrote:
> On Fri, Apr 19, 2024 at 1:58 PM Arnaldo Carvalho de Melo
> <acme@kernel.org> wrote:
> >
> > From: Arnaldo Carvalho de Melo <acme@redhat.com>
> >
> > Instead of the somewhat confusing:
> >
> >   --btf_features=all,reproducible_build
> >
> > That means "'all' the standard BTF features plus the 'reproducible_build'
> > extra BTF feature", use + directly, making it more compact:
> >
> >   --btf_features=+reproducible_build
> >
> 
> for older paholes that don't yet know about + syntax, but support
> --btf_features, will this effectively disable all features or how will
> it work?
> 
> I'm thinking from the perspective of using +reproducible_build
> unconditionally in kernel's build scripts, will it regress something
> for current pahole versions?

Well, I think it will end up being discarded just like "all" or
"default", no? I.e. those were keywords not grokked by older pahole
versions, so ignored as we're not using --btf_features_strict, right?

Alan?

But then we're not yet using --btf_features in scripts/Makefile.btf,
right?

But as Daniel pointed out and Alan (I think) agreed, for things like
scripts we probably end up using the most verbose thing as:

	--btf_features=default,reproducible_build

to mean a set (the default set of BTF options) + an optional/extra
feature (reproducibe_build), that for people not used to the + syntax
may be more descriptive (I really think that both are confusing for
beginners knowing nothing about BTF and its evolution, etc).

Alan, also we released 1.26 with "all" meaning what we now call
"default", so we need to keep both meaning the same thing, right?

- Arnaldo
 
> > In the future we may want the '-' counterpart as a way to _remove_ some
> > of the standard set of BTF features.
> >
> > Cc: Alan Maguire <alan.maguire@oracle.com>
> > Cc: Andrii Nakryiko <andrii.nakryiko@gmail.com>
> > Cc: Daniel Xu <dxu@dxuuu.xyz>
> > Cc: Eduard Zingerman <eddyz87@gmail.com>
> > Cc: Jiri Olsa <jolsa@kernel.org>
> > Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
> > ---
> >  man-pages/pahole.1          | 6 ++++++
> >  pahole.c                    | 6 ++++++
> >  tests/reproducible_build.sh | 2 +-
> >  3 files changed, 13 insertions(+), 1 deletion(-)
> >
> 
> [...]
Alan Maguire April 29, 2024, 11:16 a.m. UTC | #3
On 26/04/2024 21:47, Arnaldo Carvalho de Melo wrote:
> On Fri, Apr 26, 2024 at 01:26:40PM -0700, Andrii Nakryiko wrote:
>> On Fri, Apr 19, 2024 at 1:58 PM Arnaldo Carvalho de Melo
>> <acme@kernel.org> wrote:
>>>
>>> From: Arnaldo Carvalho de Melo <acme@redhat.com>
>>>
>>> Instead of the somewhat confusing:
>>>
>>>   --btf_features=all,reproducible_build
>>>
>>> That means "'all' the standard BTF features plus the 'reproducible_build'
>>> extra BTF feature", use + directly, making it more compact:
>>>
>>>   --btf_features=+reproducible_build
>>>
>>
>> for older paholes that don't yet know about + syntax, but support
>> --btf_features, will this effectively disable all features or how will
>> it work?
>>
>> I'm thinking from the perspective of using +reproducible_build
>> unconditionally in kernel's build scripts, will it regress something
>> for current pahole versions?
> 
> Well, I think it will end up being discarded just like "all" or
> "default", no? I.e. those were keywords not grokked by older pahole
> versions, so ignored as we're not using --btf_features_strict, right?
> 
> Alan?
> 

Yep, it would just be ignored, so wouldn't have the desired behaviour
of enabling defaults + reproducible build option.

> But then we're not yet using --btf_features in scripts/Makefile.btf,
> right?
> 
> But as Daniel pointed out and Alan (I think) agreed, for things like
> scripts we probably end up using the most verbose thing as:
> 
> 	--btf_features=default,reproducible_build
> 
> to mean a set (the default set of BTF options) + an optional/extra
> feature (reproducibe_build), that for people not used to the + syntax
> may be more descriptive (I really think that both are confusing for
> beginners knowing nothing about BTF and its evolution, etc).
> 
> Alan, also we released 1.26 with "all" meaning what we now call
> "default", so we need to keep both meaning the same thing, right?
>

I might be missing something here, but I think we should always call out
explicitly the set of features we want in the kernel Makefile.btf
(something like [1]). The reason for this is that the concept of what is
"default" may evolve over time; for example it's going to include
Daniel's kfunc definitions for soon. That's a good thing, but it could
conceivably cause problems down the line. Consider a newer pahole - with
a newer set of defaults - running on an older kernel. In that case, we
could end up encoding BTF features we don't want.  By contrast, if we
always call out the full set of BTF features we want via
--btf_features=feature1,feature2 etc we'll always get the expected set.
Plus for folks consulting the code, it's much clearer which BTF features
are in use when they look at the Makefiles for a particular kernel.
So my sense of the value of "default" is as a shortcut for testing the
latest and greatest set of BTF feature encoding, but not for use in the
kernel tree Makefiles. Thanks!

Alan

[1]
https://lore.kernel.org/bpf/20240424154806.3417662-7-alan.maguire@oracle.com/

> - Arnaldo
>  
>>> In the future we may want the '-' counterpart as a way to _remove_ some
>>> of the standard set of BTF features.
>>>
>>> Cc: Alan Maguire <alan.maguire@oracle.com>
>>> Cc: Andrii Nakryiko <andrii.nakryiko@gmail.com>
>>> Cc: Daniel Xu <dxu@dxuuu.xyz>
>>> Cc: Eduard Zingerman <eddyz87@gmail.com>
>>> Cc: Jiri Olsa <jolsa@kernel.org>
>>> Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
>>> ---
>>>  man-pages/pahole.1          | 6 ++++++
>>>  pahole.c                    | 6 ++++++
>>>  tests/reproducible_build.sh | 2 +-
>>>  3 files changed, 13 insertions(+), 1 deletion(-)
>>>
>>
>> [...]
Andrii Nakryiko April 29, 2024, 4:46 p.m. UTC | #4
On Mon, Apr 29, 2024 at 4:16 AM Alan Maguire <alan.maguire@oracle.com> wrote:
>
> On 26/04/2024 21:47, Arnaldo Carvalho de Melo wrote:
> > On Fri, Apr 26, 2024 at 01:26:40PM -0700, Andrii Nakryiko wrote:
> >> On Fri, Apr 19, 2024 at 1:58 PM Arnaldo Carvalho de Melo
> >> <acme@kernel.org> wrote:
> >>>
> >>> From: Arnaldo Carvalho de Melo <acme@redhat.com>
> >>>
> >>> Instead of the somewhat confusing:
> >>>
> >>>   --btf_features=all,reproducible_build
> >>>
> >>> That means "'all' the standard BTF features plus the 'reproducible_build'
> >>> extra BTF feature", use + directly, making it more compact:
> >>>
> >>>   --btf_features=+reproducible_build
> >>>
> >>
> >> for older paholes that don't yet know about + syntax, but support
> >> --btf_features, will this effectively disable all features or how will
> >> it work?
> >>
> >> I'm thinking from the perspective of using +reproducible_build
> >> unconditionally in kernel's build scripts, will it regress something
> >> for current pahole versions?
> >
> > Well, I think it will end up being discarded just like "all" or
> > "default", no? I.e. those were keywords not grokked by older pahole
> > versions, so ignored as we're not using --btf_features_strict, right?
> >
> > Alan?
> >
>
> Yep, it would just be ignored, so wouldn't have the desired behaviour
> of enabling defaults + reproducible build option.
>
> > But then we're not yet using --btf_features in scripts/Makefile.btf,
> > right?
> >
> > But as Daniel pointed out and Alan (I think) agreed, for things like
> > scripts we probably end up using the most verbose thing as:
> >
> >       --btf_features=default,reproducible_build
> >
> > to mean a set (the default set of BTF options) + an optional/extra
> > feature (reproducibe_build), that for people not used to the + syntax
> > may be more descriptive (I really think that both are confusing for
> > beginners knowing nothing about BTF and its evolution, etc).
> >
> > Alan, also we released 1.26 with "all" meaning what we now call
> > "default", so we need to keep both meaning the same thing, right?
> >
>
> I might be missing something here, but I think we should always call out
> explicitly the set of features we want in the kernel Makefile.btf
> (something like [1]). The reason for this is that the concept of what is
> "default" may evolve over time; for example it's going to include
> Daniel's kfunc definitions for soon. That's a good thing, but it could
> conceivably cause problems down the line. Consider a newer pahole - with
> a newer set of defaults - running on an older kernel. In that case, we
> could end up encoding BTF features we don't want.  By contrast, if we
> always call out the full set of BTF features we want via
> --btf_features=feature1,feature2 etc we'll always get the expected set.
> Plus for folks consulting the code, it's much clearer which BTF features
> are in use when they look at the Makefiles for a particular kernel.
> So my sense of the value of "default" is as a shortcut for testing the
> latest and greatest set of BTF feature encoding, but not for use in the
> kernel tree Makefiles. Thanks!

Yep, I agree, the whole point was to not regress older kernel builds
with newer pahole, so we need to explicitly list used features.

>
> Alan
>
> [1]
> https://lore.kernel.org/bpf/20240424154806.3417662-7-alan.maguire@oracle.com/
>
> > - Arnaldo
> >
> >>> In the future we may want the '-' counterpart as a way to _remove_ some
> >>> of the standard set of BTF features.
> >>>
> >>> Cc: Alan Maguire <alan.maguire@oracle.com>
> >>> Cc: Andrii Nakryiko <andrii.nakryiko@gmail.com>
> >>> Cc: Daniel Xu <dxu@dxuuu.xyz>
> >>> Cc: Eduard Zingerman <eddyz87@gmail.com>
> >>> Cc: Jiri Olsa <jolsa@kernel.org>
> >>> Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
> >>> ---
> >>>  man-pages/pahole.1          | 6 ++++++
> >>>  pahole.c                    | 6 ++++++
> >>>  tests/reproducible_build.sh | 2 +-
> >>>  3 files changed, 13 insertions(+), 1 deletion(-)
> >>>
> >>
> >> [...]
Arnaldo Carvalho de Melo April 29, 2024, 6:48 p.m. UTC | #5
On Mon, Apr 29, 2024 at 12:16:18PM +0100, Alan Maguire wrote:
> On 26/04/2024 21:47, Arnaldo Carvalho de Melo wrote:
> > On Fri, Apr 26, 2024 at 01:26:40PM -0700, Andrii Nakryiko wrote:
> >> On Fri, Apr 19, 2024 at 1:58 PM Arnaldo Carvalho de Melo <acme@kernel.org> wrote:
> >> for older paholes that don't yet know about + syntax, but support
> >> --btf_features, will this effectively disable all features or how will
> >> it work?
> >>
> >> I'm thinking from the perspective of using +reproducible_build
> >> unconditionally in kernel's build scripts, will it regress something
> >> for current pahole versions?
> > 
> > Well, I think it will end up being discarded just like "all" or
> > "default", no? I.e. those were keywords not grokked by older pahole
> > versions, so ignored as we're not using --btf_features_strict, right?

> > Alan?
 
> Yep, it would just be ignored, so wouldn't have the desired behaviour
> of enabling defaults + reproducible build option.
 
> > But then we're not yet using --btf_features in scripts/Makefile.btf,
> > right?

> > But as Daniel pointed out and Alan (I think) agreed, for things like
> > scripts we probably end up using the most verbose thing as:

> > 	--btf_features=default,reproducible_build

> > to mean a set (the default set of BTF options) + an optional/extra
> > feature (reproducibe_build), that for people not used to the + syntax
> > may be more descriptive (I really think that both are confusing for
> > beginners knowing nothing about BTF and its evolution, etc).

> > Alan, also we released 1.26 with "all" meaning what we now call
> > "default", so we need to keep both meaning the same thing, right?

> I might be missing something here, but I think we should always call out

No you're not, I was just talking about what Daniel pointed out, that
when using a script using 'default,extra_feature' is more descriptive
than '+extra_feature', but you're right, for the kernel we go on adding
the features we need and older pahole versions will just ignore those.

The explanation you gave on this message is interesting to clarify when
'default' should be used and perhaps also use the kernel build process
needs/use of a list of features, etc. It would be good to have it in the
man page, can you provide a formal patch doing that?

Thanks,

- Arnaldo

> explicitly the set of features we want in the kernel Makefile.btf
> (something like [1]). The reason for this is that the concept of what is
> "default" may evolve over time; for example it's going to include
> Daniel's kfunc definitions for soon. That's a good thing, but it could
> conceivably cause problems down the line. Consider a newer pahole - with
> a newer set of defaults - running on an older kernel. In that case, we
> could end up encoding BTF features we don't want.  By contrast, if we
> always call out the full set of BTF features we want via
> --btf_features=feature1,feature2 etc we'll always get the expected set.
> Plus for folks consulting the code, it's much clearer which BTF features
> are in use when they look at the Makefiles for a particular kernel.
> So my sense of the value of "default" is as a shortcut for testing the
> latest and greatest set of BTF feature encoding, but not for use in the
> kernel tree Makefiles. Thanks!
> 
> Alan
> 
> [1]
> https://lore.kernel.org/bpf/20240424154806.3417662-7-alan.maguire@oracle.com/
> 
> > - Arnaldo
> >  
> >>> In the future we may want the '-' counterpart as a way to _remove_ some
> >>> of the standard set of BTF features.
> >>>
> >>> Cc: Alan Maguire <alan.maguire@oracle.com>
> >>> Cc: Andrii Nakryiko <andrii.nakryiko@gmail.com>
> >>> Cc: Daniel Xu <dxu@dxuuu.xyz>
> >>> Cc: Eduard Zingerman <eddyz87@gmail.com>
> >>> Cc: Jiri Olsa <jolsa@kernel.org>
> >>> Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
> >>> ---
> >>>  man-pages/pahole.1          | 6 ++++++
> >>>  pahole.c                    | 6 ++++++
> >>>  tests/reproducible_build.sh | 2 +-
> >>>  3 files changed, 13 insertions(+), 1 deletion(-)
> >>>
> >>
> >> [...]
diff mbox series

Patch

diff --git a/man-pages/pahole.1 b/man-pages/pahole.1
index 64de3438b5f9a77a..2f4f42f8323efd6e 100644
--- a/man-pages/pahole.1
+++ b/man-pages/pahole.1
@@ -320,6 +320,12 @@  Supported non-standard features (not enabled for 'all')
 
 So for example, specifying \-\-btf_encode=var,enum64 will result in a BTF encoding that (as well as encoding basic BTF information) will contain variables and enum64 values.
 
+.fi
+
+If one wants to add an extra feature to the set of standard ones, the '+' prefix can be used, i.e.:
+\-\-btf_features=+reproducible_build will add all standard features plus the 'reproducible_build' extra
+feature.
+
 .TP
 .B \-\-btf_features_strict
 Identical to \-\-btf_features above, but pahole will exit if it encounters an unrecognized feature.
diff --git a/pahole.c b/pahole.c
index af94d2a45ee96cbe..42c5b03ee1d1a8f8 100644
--- a/pahole.c
+++ b/pahole.c
@@ -1364,6 +1364,12 @@  static void parse_btf_features(const char *features, bool strict)
 		return;
 	}
 
+	// Adding extra features to the set of standard features.
+	if (strstarts(features, "+")) {
+		btf_features__enable_for_all();
+		++features;
+	}
+
 	strncpy(f, features, BTF_MAX_FEATURE_STR)[BTF_MAX_FEATURE_STR] = '\0';
 	s = f;
 	while ((feature_name = strtok_r(s, ",", &saveptr)) != NULL) {
diff --git a/tests/reproducible_build.sh b/tests/reproducible_build.sh
index e2f836081b125119..1222cb42c6639235 100755
--- a/tests/reproducible_build.sh
+++ b/tests/reproducible_build.sh
@@ -29,7 +29,7 @@  nr_proc=$(getconf _NPROCESSORS_ONLN)
 
 for threads in $(seq $nr_proc) ; do
 	test -n "$VERBOSE" && echo $threads threads encoding
-	pahole -j$threads --btf_features=all,reproducible_build --btf_encode_detached=$outdir/vmlinux.btf.parallel.reproducible $vmlinux &
+	pahole -j$threads --btf_features=+reproducible_build --btf_encode_detached=$outdir/vmlinux.btf.parallel.reproducible $vmlinux &
 	pahole=$!
 	# HACK: Wait a bit for pahole to start its threads
 	sleep 0.3s