| Message ID | 20240701173133.3283312-1-alan.maguire@oracle.com (mailing list archive) |
|---|---|
| State | New |
| Headers | show |
| Series | [bpf-next] kbuild, bpf: reproducible BTF from pahole when KBUILD_BUILD_TIMESTAMP set | expand |
On Mon, Jul 01, 2024 at 06:31:33PM +0100, Alan Maguire wrote: > Reproducible builds [1] require that the same source code with > the same set of tools can build identical objects each time, > but pahole in parallel mode was non-deterministic in > BTF generation prior to > > dba7b5e ("pahole: Encode BTF serially in a reproducible build") > > This was a problem since said BTF is baked into kernels and modules in > .BTF sections, so parallel pahole was causing non-reproducible binary > generation. Now with the above commit we have support for parallel > reproducible BTF generation in pahole. > > KBUILD_BUILD_TIMESTAMP is set for reproducible builds, so if it > is set, add reproducible_build to --btf_features. > > [1] Documentation/kbuild/reproducible-builds.rst > > Signed-off-by: Alan Maguire <alan.maguire@oracle.com> makes sense Acked-by: Jiri Olsa <jolsa@kernel.org> jirka > --- > scripts/Makefile.btf | 4 ++++ > 1 file changed, 4 insertions(+) > > diff --git a/scripts/Makefile.btf b/scripts/Makefile.btf > index b75f09f3f424..40bb72662967 100644 > --- a/scripts/Makefile.btf > +++ b/scripts/Makefile.btf > @@ -21,6 +21,10 @@ else > # Switch to using --btf_features for v1.26 and later. > pahole-flags-$(call test-ge, $(pahole-ver), 126) = -j --btf_features=encode_force,var,float,enum64,decl_tag,type_tag,optimized_func,consistent_func,decl_tag_kfuncs > > +ifneq ($(KBUILD_BUILD_TIMESTAMP),) > +pahole-flags-$(call test-ge, $(pahole-ver), 126) += --btf_features=reproducible_build > +endif > + > ifneq ($(KBUILD_EXTMOD),) > module-pahole-flags-$(call test-ge, $(pahole-ver), 126) += --btf_features=distilled_base > endif > -- > 2.31.1 >
On Tue, Jul 2, 2024 at 2:32 AM Alan Maguire <alan.maguire@oracle.com> wrote: > > Reproducible builds [1] require that the same source code with > the same set of tools can build identical objects each time, > but pahole in parallel mode was non-deterministic in > BTF generation prior to > > dba7b5e ("pahole: Encode BTF serially in a reproducible build") > > This was a problem since said BTF is baked into kernels and modules in > .BTF sections, so parallel pahole was causing non-reproducible binary > generation. Now with the above commit we have support for parallel > reproducible BTF generation in pahole. > > KBUILD_BUILD_TIMESTAMP is set for reproducible builds, so if it > is set, add reproducible_build to --btf_features. > > [1] Documentation/kbuild/reproducible-builds.rst > > Signed-off-by: Alan Maguire <alan.maguire@oracle.com> Does not make sense. KBUILD_BUILD_TIMESTAMP is not a switch for "please enable the reproducible build". KBUILD_BUILD_TIMESTAMP requires the build code to use the given time in the output where timestamps are used. Your patch does not use the timestamp at all. If --btf_features=reproducible_build has no downside, please add it whenever supported. > --- > scripts/Makefile.btf | 4 ++++ > 1 file changed, 4 insertions(+) > > diff --git a/scripts/Makefile.btf b/scripts/Makefile.btf > index b75f09f3f424..40bb72662967 100644 > --- a/scripts/Makefile.btf > +++ b/scripts/Makefile.btf > @@ -21,6 +21,10 @@ else > # Switch to using --btf_features for v1.26 and later. > pahole-flags-$(call test-ge, $(pahole-ver), 126) = -j --btf_features=encode_force,var,float,enum64,decl_tag,type_tag,optimized_func,consistent_func,decl_tag_kfuncs > > +ifneq ($(KBUILD_BUILD_TIMESTAMP),) > +pahole-flags-$(call test-ge, $(pahole-ver), 126) += --btf_features=reproducible_build > +endif > + > ifneq ($(KBUILD_EXTMOD),) > module-pahole-flags-$(call test-ge, $(pahole-ver), 126) += --btf_features=distilled_base > endif > -- > 2.31.1 >
Masahiro Yamada wrote on Tue, Jul 02, 2024 at 04:58:50PM +0900: > If --btf_features=reproducible_build has no downside, > please add it whenever supported. It makes the build slightly slower (from [1], 3.858 -> 3.991 (+3%) on my crippled machine for the vmlinux BTF phase -- the modules also get similar treatment, I'm not sure how big the total time exactly is -- for large kernels with tons of modules it definitely adds up but for small kernels it's probably "short enough") [1] https://lkml.kernel.org/r/20240626032253.3406460-1-asmadeus@codewreck.org I don't particularly mind either way, so this is mostly out of curiosity: do we have any other setting that would be closer better than this KBUILD_BUILD_TIMESTAMP to say "make this reproducible", or is the kernel build supposed to be reproducible except for the date by default? Thank you,
On 02/07/2024 08:58, Masahiro Yamada wrote: > On Tue, Jul 2, 2024 at 2:32 AM Alan Maguire <alan.maguire@oracle.com> wrote: >> >> Reproducible builds [1] require that the same source code with >> the same set of tools can build identical objects each time, >> but pahole in parallel mode was non-deterministic in >> BTF generation prior to >> >> dba7b5e ("pahole: Encode BTF serially in a reproducible build") >> >> This was a problem since said BTF is baked into kernels and modules in >> .BTF sections, so parallel pahole was causing non-reproducible binary >> generation. Now with the above commit we have support for parallel >> reproducible BTF generation in pahole. >> >> KBUILD_BUILD_TIMESTAMP is set for reproducible builds, so if it >> is set, add reproducible_build to --btf_features. >> >> [1] Documentation/kbuild/reproducible-builds.rst >> >> Signed-off-by: Alan Maguire <alan.maguire@oracle.com> > > > > > Does not make sense. > > > > KBUILD_BUILD_TIMESTAMP is not a switch for > "please enable the reproducible build". > > > KBUILD_BUILD_TIMESTAMP requires the build code > to use the given time in the output where timestamps are used. > > Your patch does not use the timestamp at all. > No, and that's not the intention. It is used as a signal to pahole to enable reproducibility in parallel build. There is a cost to this so it's not advisable in all scenarios [1]. Is there a kbuild-approved way to determine if reproducible builds are in operation? Alan [1] https://lore.kernel.org/dwarves/20240412211604.789632-12-acme@kernel.org/ > > If --btf_features=reproducible_build has no downside, > please add it whenever supported. > > > > > > > >> --- >> scripts/Makefile.btf | 4 ++++ >> 1 file changed, 4 insertions(+) >> >> diff --git a/scripts/Makefile.btf b/scripts/Makefile.btf >> index b75f09f3f424..40bb72662967 100644 >> --- a/scripts/Makefile.btf >> +++ b/scripts/Makefile.btf >> @@ -21,6 +21,10 @@ else >> # Switch to using --btf_features for v1.26 and later. >> pahole-flags-$(call test-ge, $(pahole-ver), 126) = -j --btf_features=encode_force,var,float,enum64,decl_tag,type_tag,optimized_func,consistent_func,decl_tag_kfuncs >> >> +ifneq ($(KBUILD_BUILD_TIMESTAMP),) >> +pahole-flags-$(call test-ge, $(pahole-ver), 126) += --btf_features=reproducible_build >> +endif >> + >> ifneq ($(KBUILD_EXTMOD),) >> module-pahole-flags-$(call test-ge, $(pahole-ver), 126) += --btf_features=distilled_base >> endif >> -- >> 2.31.1 >> > >
diff --git a/scripts/Makefile.btf b/scripts/Makefile.btf index b75f09f3f424..40bb72662967 100644 --- a/scripts/Makefile.btf +++ b/scripts/Makefile.btf @@ -21,6 +21,10 @@ else # Switch to using --btf_features for v1.26 and later. pahole-flags-$(call test-ge, $(pahole-ver), 126) = -j --btf_features=encode_force,var,float,enum64,decl_tag,type_tag,optimized_func,consistent_func,decl_tag_kfuncs +ifneq ($(KBUILD_BUILD_TIMESTAMP),) +pahole-flags-$(call test-ge, $(pahole-ver), 126) += --btf_features=reproducible_build +endif + ifneq ($(KBUILD_EXTMOD),) module-pahole-flags-$(call test-ge, $(pahole-ver), 126) += --btf_features=distilled_base endif
Reproducible builds [1] require that the same source code with the same set of tools can build identical objects each time, but pahole in parallel mode was non-deterministic in BTF generation prior to dba7b5e ("pahole: Encode BTF serially in a reproducible build") This was a problem since said BTF is baked into kernels and modules in .BTF sections, so parallel pahole was causing non-reproducible binary generation. Now with the above commit we have support for parallel reproducible BTF generation in pahole. KBUILD_BUILD_TIMESTAMP is set for reproducible builds, so if it is set, add reproducible_build to --btf_features. [1] Documentation/kbuild/reproducible-builds.rst Signed-off-by: Alan Maguire <alan.maguire@oracle.com> --- scripts/Makefile.btf | 4 ++++ 1 file changed, 4 insertions(+)