| Message ID | 20240701173133.3283312-1-alan.maguire@oracle.com (mailing list archive) |
|---|---|
| State | New |
| Headers | show |
| Series | [bpf-next] kbuild, bpf: reproducible BTF from pahole when KBUILD_BUILD_TIMESTAMP set | expand |
On Mon, Jul 01, 2024 at 06:31:33PM +0100, Alan Maguire wrote: > Reproducible builds [1] require that the same source code with > the same set of tools can build identical objects each time, > but pahole in parallel mode was non-deterministic in > BTF generation prior to > > dba7b5e ("pahole: Encode BTF serially in a reproducible build") > > This was a problem since said BTF is baked into kernels and modules in > .BTF sections, so parallel pahole was causing non-reproducible binary > generation. Now with the above commit we have support for parallel > reproducible BTF generation in pahole. > > KBUILD_BUILD_TIMESTAMP is set for reproducible builds, so if it > is set, add reproducible_build to --btf_features. > > [1] Documentation/kbuild/reproducible-builds.rst > > Signed-off-by: Alan Maguire <alan.maguire@oracle.com> makes sense Acked-by: Jiri Olsa <jolsa@kernel.org> jirka > --- > scripts/Makefile.btf | 4 ++++ > 1 file changed, 4 insertions(+) > > diff --git a/scripts/Makefile.btf b/scripts/Makefile.btf > index b75f09f3f424..40bb72662967 100644 > --- a/scripts/Makefile.btf > +++ b/scripts/Makefile.btf > @@ -21,6 +21,10 @@ else > # Switch to using --btf_features for v1.26 and later. > pahole-flags-$(call test-ge, $(pahole-ver), 126) = -j --btf_features=encode_force,var,float,enum64,decl_tag,type_tag,optimized_func,consistent_func,decl_tag_kfuncs > > +ifneq ($(KBUILD_BUILD_TIMESTAMP),) > +pahole-flags-$(call test-ge, $(pahole-ver), 126) += --btf_features=reproducible_build > +endif > + > ifneq ($(KBUILD_EXTMOD),) > module-pahole-flags-$(call test-ge, $(pahole-ver), 126) += --btf_features=distilled_base > endif > -- > 2.31.1 >
On Tue, Jul 2, 2024 at 2:32 AM Alan Maguire <alan.maguire@oracle.com> wrote: > > Reproducible builds [1] require that the same source code with > the same set of tools can build identical objects each time, > but pahole in parallel mode was non-deterministic in > BTF generation prior to > > dba7b5e ("pahole: Encode BTF serially in a reproducible build") > > This was a problem since said BTF is baked into kernels and modules in > .BTF sections, so parallel pahole was causing non-reproducible binary > generation. Now with the above commit we have support for parallel > reproducible BTF generation in pahole. > > KBUILD_BUILD_TIMESTAMP is set for reproducible builds, so if it > is set, add reproducible_build to --btf_features. > > [1] Documentation/kbuild/reproducible-builds.rst > > Signed-off-by: Alan Maguire <alan.maguire@oracle.com> Does not make sense. KBUILD_BUILD_TIMESTAMP is not a switch for "please enable the reproducible build". KBUILD_BUILD_TIMESTAMP requires the build code to use the given time in the output where timestamps are used. Your patch does not use the timestamp at all. If --btf_features=reproducible_build has no downside, please add it whenever supported. > --- > scripts/Makefile.btf | 4 ++++ > 1 file changed, 4 insertions(+) > > diff --git a/scripts/Makefile.btf b/scripts/Makefile.btf > index b75f09f3f424..40bb72662967 100644 > --- a/scripts/Makefile.btf > +++ b/scripts/Makefile.btf > @@ -21,6 +21,10 @@ else > # Switch to using --btf_features for v1.26 and later. > pahole-flags-$(call test-ge, $(pahole-ver), 126) = -j --btf_features=encode_force,var,float,enum64,decl_tag,type_tag,optimized_func,consistent_func,decl_tag_kfuncs > > +ifneq ($(KBUILD_BUILD_TIMESTAMP),) > +pahole-flags-$(call test-ge, $(pahole-ver), 126) += --btf_features=reproducible_build > +endif > + > ifneq ($(KBUILD_EXTMOD),) > module-pahole-flags-$(call test-ge, $(pahole-ver), 126) += --btf_features=distilled_base > endif > -- > 2.31.1 >
Masahiro Yamada wrote on Tue, Jul 02, 2024 at 04:58:50PM +0900: > If --btf_features=reproducible_build has no downside, > please add it whenever supported. It makes the build slightly slower (from [1], 3.858 -> 3.991 (+3%) on my crippled machine for the vmlinux BTF phase -- the modules also get similar treatment, I'm not sure how big the total time exactly is -- for large kernels with tons of modules it definitely adds up but for small kernels it's probably "short enough") [1] https://lkml.kernel.org/r/20240626032253.3406460-1-asmadeus@codewreck.org I don't particularly mind either way, so this is mostly out of curiosity: do we have any other setting that would be closer better than this KBUILD_BUILD_TIMESTAMP to say "make this reproducible", or is the kernel build supposed to be reproducible except for the date by default? Thank you,
On 02/07/2024 08:58, Masahiro Yamada wrote: > On Tue, Jul 2, 2024 at 2:32 AM Alan Maguire <alan.maguire@oracle.com> wrote: >> >> Reproducible builds [1] require that the same source code with >> the same set of tools can build identical objects each time, >> but pahole in parallel mode was non-deterministic in >> BTF generation prior to >> >> dba7b5e ("pahole: Encode BTF serially in a reproducible build") >> >> This was a problem since said BTF is baked into kernels and modules in >> .BTF sections, so parallel pahole was causing non-reproducible binary >> generation. Now with the above commit we have support for parallel >> reproducible BTF generation in pahole. >> >> KBUILD_BUILD_TIMESTAMP is set for reproducible builds, so if it >> is set, add reproducible_build to --btf_features. >> >> [1] Documentation/kbuild/reproducible-builds.rst >> >> Signed-off-by: Alan Maguire <alan.maguire@oracle.com> > > > > > Does not make sense. > > > > KBUILD_BUILD_TIMESTAMP is not a switch for > "please enable the reproducible build". > > > KBUILD_BUILD_TIMESTAMP requires the build code > to use the given time in the output where timestamps are used. > > Your patch does not use the timestamp at all. > No, and that's not the intention. It is used as a signal to pahole to enable reproducibility in parallel build. There is a cost to this so it's not advisable in all scenarios [1]. Is there a kbuild-approved way to determine if reproducible builds are in operation? Alan [1] https://lore.kernel.org/dwarves/20240412211604.789632-12-acme@kernel.org/ > > If --btf_features=reproducible_build has no downside, > please add it whenever supported. > > > > > > > >> --- >> scripts/Makefile.btf | 4 ++++ >> 1 file changed, 4 insertions(+) >> >> diff --git a/scripts/Makefile.btf b/scripts/Makefile.btf >> index b75f09f3f424..40bb72662967 100644 >> --- a/scripts/Makefile.btf >> +++ b/scripts/Makefile.btf >> @@ -21,6 +21,10 @@ else >> # Switch to using --btf_features for v1.26 and later. >> pahole-flags-$(call test-ge, $(pahole-ver), 126) = -j --btf_features=encode_force,var,float,enum64,decl_tag,type_tag,optimized_func,consistent_func,decl_tag_kfuncs >> >> +ifneq ($(KBUILD_BUILD_TIMESTAMP),) >> +pahole-flags-$(call test-ge, $(pahole-ver), 126) += --btf_features=reproducible_build >> +endif >> + >> ifneq ($(KBUILD_EXTMOD),) >> module-pahole-flags-$(call test-ge, $(pahole-ver), 126) += --btf_features=distilled_base >> endif >> -- >> 2.31.1 >> > >
On Tue, Jul 2, 2024 at 6:49 PM Alan Maguire <alan.maguire@oracle.com> wrote: > > On 02/07/2024 08:58, Masahiro Yamada wrote: > > On Tue, Jul 2, 2024 at 2:32 AM Alan Maguire <alan.maguire@oracle.com> wrote: > >> > >> Reproducible builds [1] require that the same source code with > >> the same set of tools can build identical objects each time, > >> but pahole in parallel mode was non-deterministic in > >> BTF generation prior to > >> > >> dba7b5e ("pahole: Encode BTF serially in a reproducible build") > >> > >> This was a problem since said BTF is baked into kernels and modules in > >> .BTF sections, so parallel pahole was causing non-reproducible binary > >> generation. Now with the above commit we have support for parallel > >> reproducible BTF generation in pahole. > >> > >> KBUILD_BUILD_TIMESTAMP is set for reproducible builds, so if it > >> is set, add reproducible_build to --btf_features. > >> > >> [1] Documentation/kbuild/reproducible-builds.rst > >> > >> Signed-off-by: Alan Maguire <alan.maguire@oracle.com> > > > > > > > > > > Does not make sense. > > > > > > > > KBUILD_BUILD_TIMESTAMP is not a switch for > > "please enable the reproducible build". > > > > > > KBUILD_BUILD_TIMESTAMP requires the build code > > to use the given time in the output where timestamps are used. > > > > Your patch does not use the timestamp at all. > > > > No, and that's not the intention. It is used as a signal to pahole to > enable reproducibility in parallel build. There is a cost to this so > it's not advisable in all scenarios [1]. Is there a kbuild-approved way > to determine if reproducible builds are in operation? > > Alan > > [1] > https://lore.kernel.org/dwarves/20240412211604.789632-12-acme@kernel.org/ Kbuild already provides a standard mechanism. config DEBUG_INFO_BTF_REPRODUCIBLE bool "Generate reproducible BTF" depends on DEBUG_INFO_BTF depends on PAHOLE_VERSION >= 126 pahole-flags-$(CONFIG_DEBUG_INFO_BTF_REPRODUCIBLE) = --btf_features=reproducible_build Presumably, it is better to add a new section in Documentation/kbuild/reproducible-builds.rst -- Best Regards Masahiro Yamada
Masahiro Yamada wrote on Mon, Jul 15, 2024 at 08:19:53PM +0900: > Kbuild already provides a standard mechanism. > > > > > > config DEBUG_INFO_BTF_REPRODUCIBLE Ah, I thought this was already an existing option but it's a suggestion for a new one. With my distro hat I have to say I have no interest in chasing more reproducible knobs for the kernel -- the current trend in 99% of the projects is that just setting SOURCE_DATE_EPOCH "just works" and the every non-default option that is required is work for people to track, enable in various packaging frameworks, backport to old kernels, update if it ever changes... I could understand a new option "toggle everything reproducible" that would select this, even if it doesn't do anything else right now (perhaps also complain if KBUILD_BUILD_TIMESTAMP isn't set?), but I can't say I'm thrilled about a new option pahole-specific that would have to be individually enabled. (I guess that means I've just volunteered to add both... perhaps next year when I find time, for now the distro pahole patch toggling reproducible if SOURCE_DATE_EPOCH is set will do)
diff --git a/scripts/Makefile.btf b/scripts/Makefile.btf index b75f09f3f424..40bb72662967 100644 --- a/scripts/Makefile.btf +++ b/scripts/Makefile.btf @@ -21,6 +21,10 @@ else # Switch to using --btf_features for v1.26 and later. pahole-flags-$(call test-ge, $(pahole-ver), 126) = -j --btf_features=encode_force,var,float,enum64,decl_tag,type_tag,optimized_func,consistent_func,decl_tag_kfuncs +ifneq ($(KBUILD_BUILD_TIMESTAMP),) +pahole-flags-$(call test-ge, $(pahole-ver), 126) += --btf_features=reproducible_build +endif + ifneq ($(KBUILD_EXTMOD),) module-pahole-flags-$(call test-ge, $(pahole-ver), 126) += --btf_features=distilled_base endif
Reproducible builds [1] require that the same source code with the same set of tools can build identical objects each time, but pahole in parallel mode was non-deterministic in BTF generation prior to dba7b5e ("pahole: Encode BTF serially in a reproducible build") This was a problem since said BTF is baked into kernels and modules in .BTF sections, so parallel pahole was causing non-reproducible binary generation. Now with the above commit we have support for parallel reproducible BTF generation in pahole. KBUILD_BUILD_TIMESTAMP is set for reproducible builds, so if it is set, add reproducible_build to --btf_features. [1] Documentation/kbuild/reproducible-builds.rst Signed-off-by: Alan Maguire <alan.maguire@oracle.com> --- scripts/Makefile.btf | 4 ++++ 1 file changed, 4 insertions(+)