diff mbox series

[dwarves,2/3] pahole: add reproducible_build to --btf_features

Message ID 20240416143718.2857981-3-alan.maguire@oracle.com (mailing list archive)
State Not Applicable
Delegated to: BPF
Headers show
Series pahole: support nonstandard btf_features | expand

Checks

Context Check Description
netdev/tree_selection success Not a local patch

Commit Message

Alan Maguire April 16, 2024, 2:37 p.m. UTC
...as a non-standard feature, so it will not be enabled for
"--btf_features=all"

Signed-off-by: Alan Maguire <alan.maguire@oracle.com>
---
 man-pages/pahole.1 | 8 ++++++++
 pahole.c           | 1 +
 2 files changed, 9 insertions(+)

Comments

Arnaldo Carvalho de Melo April 16, 2024, 3:24 p.m. UTC | #1
On Tue, Apr 16, 2024 at 03:37:17PM +0100, Alan Maguire wrote:
> ...as a non-standard feature, so it will not be enabled for
> "--btf_features=all"

How did you test this?

⬢[acme@toolbox pahole]$ pahole --btf_features_strict=bgasd
Feature 'bgasd' in 'bgasd' is not supported.  Supported BTF features are:
encode_force,var,float,decl_tag,type_tag,enum64,optimized_func,consistent_func,reproducible_build
⬢[acme@toolbox pahole]$ pahole -j --btf_features=all,reproducible_build --btf_encode_detached=vmlinux.btf.parallel.reproducible_build-via-btf_features vmlinux
⬢[acme@toolbox pahole]$ bpftool btf dump file vmlinux.btf.parallel.reproducible_build-via-btf_features > output.vmlinux.btf.parallel.reproducible_build-via-btf_features
⬢[acme@toolbox pahole]$ diff -u output.vmlinux.btf.serial output.vmlinux.btf.parallel.reproducible
⬢[acme@toolbox pahole]$ diff -u output.vmlinux.btf.parallel.reproducible_build output.vmlinux.btf.parallel.reproducible_build-via-btf_features | head
--- output.vmlinux.btf.parallel.reproducible_build	2024-04-16 12:20:28.513462223 -0300
+++ output.vmlinux.btf.parallel.reproducible_build-via-btf_features	2024-04-16 12:23:37.792962930 -0300
@@ -265,7 +265,7 @@
 	'target' type_id=33 bits_offset=32
 	'key' type_id=43 bits_offset=64
 [164] PTR '(anon)' type_id=163
-[165] PTR '(anon)' type_id=35751
+[165] PTR '(anon)' type_id=14983
 [166] STRUCT 'static_key' size=16 vlen=2
 	'enabled' type_id=88 bits_offset=0
⬢[acme@toolbox pahole]$

I'm double checking things now...

- Arnaldo
 
> Signed-off-by: Alan Maguire <alan.maguire@oracle.com>
> ---
>  man-pages/pahole.1 | 8 ++++++++
>  pahole.c           | 1 +
>  2 files changed, 9 insertions(+)
> 
> diff --git a/man-pages/pahole.1 b/man-pages/pahole.1
> index 2c08e97..64de343 100644
> --- a/man-pages/pahole.1
> +++ b/man-pages/pahole.1
> @@ -310,6 +310,14 @@ Encode BTF using the specified feature list, or specify 'all' for all standard f
>  	                   in different CUs.
>  .fi
>  
> +Supported non-standard features (not enabled for 'all')
> +
> +.nf
> +	reproducible_build Ensure generated BTF is consistent every time;
> +	                   without this parallel BTF encoding can result in
> +	                   inconsistent BTF ids.
> +.fi
> +
>  So for example, specifying \-\-btf_encode=var,enum64 will result in a BTF encoding that (as well as encoding basic BTF information) will contain variables and enum64 values.
>  
>  .TP
> diff --git a/pahole.c b/pahole.c
> index 890ef81..38cc636 100644
> --- a/pahole.c
> +++ b/pahole.c
> @@ -1286,6 +1286,7 @@ struct btf_feature {
>  	BTF_FEATURE(enum64, skip_encoding_btf_enum64, true, true),
>  	BTF_FEATURE(optimized_func, btf_gen_optimized, false, true),
>  	BTF_FEATURE(consistent_func, skip_encoding_btf_inconsistent_proto, false, true),
> +	BTF_FEATURE(reproducible_build, reproducible_build, false, false),
>  };
>  
>  #define BTF_MAX_FEATURE_STR	1024
Alan Maguire April 16, 2024, 3:41 p.m. UTC | #2
On 16/04/2024 16:24, Arnaldo Carvalho de Melo wrote:
> On Tue, Apr 16, 2024 at 03:37:17PM +0100, Alan Maguire wrote:
>> ...as a non-standard feature, so it will not be enabled for
>> "--btf_features=all"
> 
> How did you test this?
> 
> ⬢[acme@toolbox pahole]$ pahole --btf_features_strict=bgasd
> Feature 'bgasd' in 'bgasd' is not supported.  Supported BTF features are:
> encode_force,var,float,decl_tag,type_tag,enum64,optimized_func,consistent_func,reproducible_build
> ⬢[acme@toolbox pahole]$ pahole -j --btf_features=all,reproducible_build --btf_encode_detached=vmlinux.btf.parallel.reproducible_build-via-btf_features vmlinux
> ⬢[acme@toolbox pahole]$ bpftool btf dump file vmlinux.btf.parallel.reproducible_build-via-btf_features > output.vmlinux.btf.parallel.reproducible_build-via-btf_features
> ⬢[acme@toolbox pahole]$ diff -u output.vmlinux.btf.serial output.vmlinux.btf.parallel.reproducible
> ⬢[acme@toolbox pahole]$ diff -u output.vmlinux.btf.parallel.reproducible_build output.vmlinux.btf.parallel.reproducible_build-via-btf_features | head
> --- output.vmlinux.btf.parallel.reproducible_build	2024-04-16 12:20:28.513462223 -0300
> +++ output.vmlinux.btf.parallel.reproducible_build-via-btf_features	2024-04-16 12:23:37.792962930 -0300
> @@ -265,7 +265,7 @@
>  	'target' type_id=33 bits_offset=32
>  	'key' type_id=43 bits_offset=64
>  [164] PTR '(anon)' type_id=163
> -[165] PTR '(anon)' type_id=35751
> +[165] PTR '(anon)' type_id=14983
>  [166] STRUCT 'static_key' size=16 vlen=2
>  	'enabled' type_id=88 bits_offset=0
> ⬢[acme@toolbox pahole]$
> 
> I'm double checking things now...
>

The test worked for me on x86_64/aarch64. Did you test with patch 3
applied? Because the test in its original state prior to patch 3 sets
--reproducible_build before setting --btf_features=all, you won't get a
reproducible build since the command line is saying "enable reproducible
builds but also enable standard features only"; the second action undoes
the first. switching to using --btf_features=all,reproducible_build
fixes things for me.
> - Arnaldo
>  
>> Signed-off-by: Alan Maguire <alan.maguire@oracle.com>
>> ---
>>  man-pages/pahole.1 | 8 ++++++++
>>  pahole.c           | 1 +
>>  2 files changed, 9 insertions(+)
>>
>> diff --git a/man-pages/pahole.1 b/man-pages/pahole.1
>> index 2c08e97..64de343 100644
>> --- a/man-pages/pahole.1
>> +++ b/man-pages/pahole.1
>> @@ -310,6 +310,14 @@ Encode BTF using the specified feature list, or specify 'all' for all standard f
>>  	                   in different CUs.
>>  .fi
>>  
>> +Supported non-standard features (not enabled for 'all')
>> +
>> +.nf
>> +	reproducible_build Ensure generated BTF is consistent every time;
>> +	                   without this parallel BTF encoding can result in
>> +	                   inconsistent BTF ids.
>> +.fi
>> +
>>  So for example, specifying \-\-btf_encode=var,enum64 will result in a BTF encoding that (as well as encoding basic BTF information) will contain variables and enum64 values.
>>  
>>  .TP
>> diff --git a/pahole.c b/pahole.c
>> index 890ef81..38cc636 100644
>> --- a/pahole.c
>> +++ b/pahole.c
>> @@ -1286,6 +1286,7 @@ struct btf_feature {
>>  	BTF_FEATURE(enum64, skip_encoding_btf_enum64, true, true),
>>  	BTF_FEATURE(optimized_func, btf_gen_optimized, false, true),
>>  	BTF_FEATURE(consistent_func, skip_encoding_btf_inconsistent_proto, false, true),
>> +	BTF_FEATURE(reproducible_build, reproducible_build, false, false),
>>  };
>>  
>>  #define BTF_MAX_FEATURE_STR	1024
> 
> 
>
Arnaldo Carvalho de Melo April 16, 2024, 8:44 p.m. UTC | #3
On Tue, Apr 16, 2024 at 04:41:20PM +0100, Alan Maguire wrote:
> On 16/04/2024 16:24, Arnaldo Carvalho de Melo wrote:
> > On Tue, Apr 16, 2024 at 03:37:17PM +0100, Alan Maguire wrote:
> >> ...as a non-standard feature, so it will not be enabled for
> >> "--btf_features=all"
> > 
> > How did you test this?
> > 
> > ⬢[acme@toolbox pahole]$ pahole --btf_features_strict=bgasd
> > Feature 'bgasd' in 'bgasd' is not supported.  Supported BTF features are:
> > encode_force,var,float,decl_tag,type_tag,enum64,optimized_func,consistent_func,reproducible_build
> > ⬢[acme@toolbox pahole]$ pahole -j --btf_features=all,reproducible_build --btf_encode_detached=vmlinux.btf.parallel.reproducible_build-via-btf_features vmlinux
> > ⬢[acme@toolbox pahole]$ bpftool btf dump file vmlinux.btf.parallel.reproducible_build-via-btf_features > output.vmlinux.btf.parallel.reproducible_build-via-btf_features
> > ⬢[acme@toolbox pahole]$ diff -u output.vmlinux.btf.serial output.vmlinux.btf.parallel.reproducible
> > ⬢[acme@toolbox pahole]$ diff -u output.vmlinux.btf.parallel.reproducible_build output.vmlinux.btf.parallel.reproducible_build-via-btf_features | head
> > --- output.vmlinux.btf.parallel.reproducible_build	2024-04-16 12:20:28.513462223 -0300
> > +++ output.vmlinux.btf.parallel.reproducible_build-via-btf_features	2024-04-16 12:23:37.792962930 -0300
> > @@ -265,7 +265,7 @@
> >  	'target' type_id=33 bits_offset=32
> >  	'key' type_id=43 bits_offset=64
> >  [164] PTR '(anon)' type_id=163
> > -[165] PTR '(anon)' type_id=35751
> > +[165] PTR '(anon)' type_id=14983
> >  [166] STRUCT 'static_key' size=16 vlen=2
> >  	'enabled' type_id=88 bits_offset=0
> > ⬢[acme@toolbox pahole]$
> > 
> > I'm double checking things now...
 
> The test worked for me on x86_64/aarch64. Did you test with patch 3
> applied? Because the test in its original state prior to patch 3 sets

With all patches applied:

⬢[acme@toolbox pahole]$ git log --oneline -5
ecd3b0852ab1f1ff (HEAD -> master) tests/reproducible_build: use --btf_features=all,reproducible_build
f9c8f5856b2aafea pahole: Add reproducible_build to --btf_features
0412978e8e6f8f76 pahole: Allow --btf_features to not participate in "all"
a9738ddc828d5ea0 tests/reproducible_build: Use --btf_features=all when encoding
d7edf9ae0388fb97 tests/reproducible_build: Validate the vmlinux file
⬢[acme@toolbox pahole]$ 

I made some mistake:

⬢[acme@toolbox pahole]$ pahole --btf_encode_detached=vmlinux.btf.serial --btf_features=all vmlinux
⬢[acme@toolbox pahole]$ pahole --btf_encode_detached=vmlinux.btf.parallel --btf_features=all -j vmlinux
⬢[acme@toolbox pahole]$ pahole --btf_encode_detached=vmlinux.btf.parallel--reproducible_build --btf_features=all --reproducible_build -j vmlinux
⬢[acme@toolbox pahole]$ pahole --btf_encode_detached=vmlinux.btf.parallel--btf_features=all,reproducible_build --btf_features=all,reproducible_build -j vmlinux
⬢[acme@toolbox pahole]$ bpftool btf dump file vmlinux.btf.serial > output.vmlinux.btf.serial
⬢[acme@toolbox pahole]$ bpftool btf dump file vmlinux.btf.parallel > output.vmlinux.btf.parallel
⬢[acme@toolbox pahole]$ bpftool btf dump file vmlinux.btf.parallel--reproducible_build > output.vmlinux.btf.parallel--reproducible_build
⬢[acme@toolbox pahole]$ bpftool btf dump file vmlinux.btf.parallel--btf_features=all,reproducible_build > output.vmlinux.btf.parallel--btf_features=all,reproducible_build
⬢[acme@toolbox pahole]$ diff -u output.vmlinux.btf.serial output.vmlinux.btf.parallel | wc -l
629165
⬢[acme@toolbox pahole]$ diff -u output.vmlinux.btf.serial output.vmlinux.btf.parallel--reproducible_build | wc -l
0
⬢[acme@toolbox pahole]$ diff -u output.vmlinux.btf.serial output.vmlinux.btf.parallel--btf_features=all,reproducible_build | wc -l
0
⬢[acme@toolbox pahole]$ diff -u output.vmlinux.btf.parallel--reproducible_build output.vmlinux.btf.parallel--btf_features=all,reproducible_build | wc -l
0
⬢[acme@toolbox pahole]$

And of course:

⬢[acme@toolbox pahole]$ time tests/reproducible_build.sh  vmlinux
Parallel reproducible DWARF Loading/Serial BTF encoding: Ok

real	1m25.241s
user	3m10.144s
sys	0m48.104s
⬢[acme@toolbox pahole]$

> --reproducible_build before setting --btf_features=all, you won't get a
> reproducible build since the command line is saying "enable reproducible
> builds but also enable standard features only"; the second action undoes

Makes sense.

> the first. switching to using --btf_features=all,reproducible_build
> fixes things for me.

Ok, all tested now, will push to the 'next' branch, waiting for Eduard's
review.

- Arnaldo

> > - Arnaldo
> >  
> >> Signed-off-by: Alan Maguire <alan.maguire@oracle.com>
> >> ---
> >>  man-pages/pahole.1 | 8 ++++++++
> >>  pahole.c           | 1 +
> >>  2 files changed, 9 insertions(+)
> >>
> >> diff --git a/man-pages/pahole.1 b/man-pages/pahole.1
> >> index 2c08e97..64de343 100644
> >> --- a/man-pages/pahole.1
> >> +++ b/man-pages/pahole.1
> >> @@ -310,6 +310,14 @@ Encode BTF using the specified feature list, or specify 'all' for all standard f
> >>  	                   in different CUs.
> >>  .fi
> >>  
> >> +Supported non-standard features (not enabled for 'all')
> >> +
> >> +.nf
> >> +	reproducible_build Ensure generated BTF is consistent every time;
> >> +	                   without this parallel BTF encoding can result in
> >> +	                   inconsistent BTF ids.
> >> +.fi
> >> +
> >>  So for example, specifying \-\-btf_encode=var,enum64 will result in a BTF encoding that (as well as encoding basic BTF information) will contain variables and enum64 values.
> >>  
> >>  .TP
> >> diff --git a/pahole.c b/pahole.c
> >> index 890ef81..38cc636 100644
> >> --- a/pahole.c
> >> +++ b/pahole.c
> >> @@ -1286,6 +1286,7 @@ struct btf_feature {
> >>  	BTF_FEATURE(enum64, skip_encoding_btf_enum64, true, true),
> >>  	BTF_FEATURE(optimized_func, btf_gen_optimized, false, true),
> >>  	BTF_FEATURE(consistent_func, skip_encoding_btf_inconsistent_proto, false, true),
> >> +	BTF_FEATURE(reproducible_build, reproducible_build, false, false),
> >>  };
> >>  
> >>  #define BTF_MAX_FEATURE_STR	1024
> > 
> > 
> >
diff mbox series

Patch

diff --git a/man-pages/pahole.1 b/man-pages/pahole.1
index 2c08e97..64de343 100644
--- a/man-pages/pahole.1
+++ b/man-pages/pahole.1
@@ -310,6 +310,14 @@  Encode BTF using the specified feature list, or specify 'all' for all standard f
 	                   in different CUs.
 .fi
 
+Supported non-standard features (not enabled for 'all')
+
+.nf
+	reproducible_build Ensure generated BTF is consistent every time;
+	                   without this parallel BTF encoding can result in
+	                   inconsistent BTF ids.
+.fi
+
 So for example, specifying \-\-btf_encode=var,enum64 will result in a BTF encoding that (as well as encoding basic BTF information) will contain variables and enum64 values.
 
 .TP
diff --git a/pahole.c b/pahole.c
index 890ef81..38cc636 100644
--- a/pahole.c
+++ b/pahole.c
@@ -1286,6 +1286,7 @@  struct btf_feature {
 	BTF_FEATURE(enum64, skip_encoding_btf_enum64, true, true),
 	BTF_FEATURE(optimized_func, btf_gen_optimized, false, true),
 	BTF_FEATURE(consistent_func, skip_encoding_btf_inconsistent_proto, false, true),
+	BTF_FEATURE(reproducible_build, reproducible_build, false, false),
 };
 
 #define BTF_MAX_FEATURE_STR	1024