| Message ID | 20250107190855.2312210-1-ihor.solodrai@pm.me (mailing list archive) |
|---|---|
| Headers | show |
| Series | pahole: faster reproducible BTF encoding | expand |
On Tuesday, January 7th, 2025 at 11:08 AM, Ihor Solodrai <ihor.solodrai@pm.me> wrote: > > > This is v4 of the series aiming to speed up parallel reproducible BTF > encoding. This version mostly addresses feedback from Jiri Olsa on v3. > > A notable adition is a patch 10/10, which changes func_states in > btf_encoder from a list to an array. > > [...] Alan, Arnaldo, I'd like to also remind about two small patches that I sent separately: * dwarves: set cu->obstack chunk size to 128Kb * https://lore.kernel.org/dwarves/20241221030445.33907-1-ihor.solodrai@pm.me/ * btf_encoder: fix memory access bugs * https://lore.kernel.org/dwarves/20241216183112.206072-1-ihor.solodrai@pm.me/ In particular, obstack chunk size change significantly improves the runtime of BTF encoding on top of this patch series, on the machines\ with lots of cores. Please take a look at them when you get a chance. They can be applied independently. Thanks.
On Tue, Jan 07, 2025 at 07:08:59PM +0000, Ihor Solodrai wrote: > This is v4 of the series aiming to speed up parallel reproducible BTF > encoding. This version mostly addresses feedback from Jiri Olsa on v3. > b4 is having trouble with this series, I'm trying to cherry pick things... ⬢ [acme@toolbox pahole]$ b4 am -ctsl --cc-trailers 20250107190855.2312210-1-ihor.solodrai@pm.me Grabbing thread from lore.kernel.org/all/20250107190855.2312210-1-ihor.solodrai@pm.me/t.mbox.gz Checking for newer revisions Grabbing search results from lore.kernel.org Analyzing 13 messages in the thread WARNING: duplicate messages found at index 1 Subject 1: dwarf_loader: multithreading with a job/worker model Subject 2: btf_encoder: simplify function encoding 2 is not a reply... assume additional patch Assuming new revision: v2 ([PATCH dwarves v4] btf_encoder: switch func_states from a list to an array) Will use the latest revision: v4 You can pick other revisions using the -vN flag Checking attestation on all messages, may take a moment... --- ✗ [PATCH v4] dwarf_loader: multithreading with a job/worker model ✗ BADSIG: DKIM/pm.me + Link: https://lore.kernel.org/r/20250107190855.2312210-9-ihor.solodrai@pm.me + Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com> ✗ [PATCH v4 1/10] btf_encoder: simplify function encoding ✗ BADSIG: DKIM/pm.me + Acked-by: Eduard Zingerman <eddyz87@gmail.com> (✗ DKIM/gmail.com) + Acked-by: Jiri Olsa <jolsa@kernel.org> (✗ DKIM/gmail.com) + Link: https://lore.kernel.org/r/20250107190855.2312210-2-ihor.solodrai@pm.me + Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com> ✗ [PATCH v4 3/10] btf_encoder: separate elf function, saved function representations ✗ BADSIG: DKIM/pm.me + Link: https://lore.kernel.org/r/20250107190855.2312210-4-ihor.solodrai@pm.me + Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com> ✗ [PATCH v4 4/10] btf_encoder: introduce elf_functions struct type ✗ BADSIG: DKIM/pm.me + Link: https://lore.kernel.org/r/20250107190855.2312210-5-ihor.solodrai@pm.me + Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com> ✗ [PATCH v4 5/10] btf_encoder: introduce elf_functions_list ✗ BADSIG: DKIM/pm.me + Link: https://lore.kernel.org/r/20250107190855.2312210-6-ihor.solodrai@pm.me + Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com> ✗ [PATCH v4 6/10] btf_encoder: remove skip_encoding_inconsistent_proto ✗ BADSIG: DKIM/pm.me + Link: https://lore.kernel.org/r/20250107190855.2312210-7-ihor.solodrai@pm.me + Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com> ✗ [PATCH v4 7/10] dwarf_loader: introduce cu->id ✗ BADSIG: DKIM/pm.me + Link: https://lore.kernel.org/r/20250107190855.2312210-8-ihor.solodrai@pm.me + Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com> ERROR: missing [8/10]! ✗ [PATCH v4 9/10] btf_encoder: clean up global encoders list ✗ BADSIG: DKIM/pm.me + Link: https://lore.kernel.org/r/20250107190855.2312210-10-ihor.solodrai@pm.me + Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com> ERROR: missing [10/10]! ERROR: missing [11/10]! --- Total patches: 8 --- WARNING: Thread incomplete! Cover: ./v4_20250107_ihor_solodrai_pahole_faster_reproducible_btf_encoding.cover Link: https://lore.kernel.org/r/20250107190855.2312210-1-ihor.solodrai@pm.me Base: not specified git am ./v4_20250107_ihor_solodrai_pahole_faster_reproducible_btf_encoding.mbx ⬢ [acme@toolbox pahole]$ > A notable adition is a patch 10/10, which changes func_states in > btf_encoder from a list to an array.
On Thursday, January 9th, 2025 at 10:32 AM, Arnaldo Carvalho de Melo <acme@kernel.org> wrote: > > > On Tue, Jan 07, 2025 at 07:08:59PM +0000, Ihor Solodrai wrote: > > > This is v4 of the series aiming to speed up parallel reproducible BTF > > encoding. This version mostly addresses feedback from Jiri Olsa on v3. > > > b4 is having trouble with this series, I'm trying to cherry pick > things... Sorry, I think I messed up some patches with manual changes before submitting. I'll resend a clean series in a bit. Thanks. > > ⬢ [acme@toolbox pahole]$ b4 am -ctsl --cc-trailers 20250107190855.2312210-1-ihor.solodrai@pm.me > Grabbing thread from lore.kernel.org/all/20250107190855.2312210-1-ihor.solodrai@pm.me/t.mbox.gz > Checking for newer revisions > Grabbing search results from lore.kernel.org > Analyzing 13 messages in the thread > WARNING: duplicate messages found at index 1 > Subject 1: dwarf_loader: multithreading with a job/worker model > Subject 2: btf_encoder: simplify function encoding > 2 is not a reply... assume additional patch > Assuming new revision: v2 ([PATCH dwarves v4] btf_encoder: switch func_states from a list to an array) > Will use the latest revision: v4 > You can pick other revisions using the -vN flag > Checking attestation on all messages, may take a moment... > --- > ✗ [PATCH v4] dwarf_loader: multithreading with a job/worker model > ✗ BADSIG: DKIM/pm.me > + Link: https://lore.kernel.org/r/20250107190855.2312210-9-ihor.solodrai@pm.me > + Signed-off-by: Arnaldo Carvalho de Melo acme@redhat.com > > ✗ [PATCH v4 1/10] btf_encoder: simplify function encoding > ✗ BADSIG: DKIM/pm.me > + Acked-by: Eduard Zingerman eddyz87@gmail.com (✗ DKIM/gmail.com) > > + Acked-by: Jiri Olsa jolsa@kernel.org (✗ DKIM/gmail.com) > > + Link: https://lore.kernel.org/r/20250107190855.2312210-2-ihor.solodrai@pm.me > + Signed-off-by: Arnaldo Carvalho de Melo acme@redhat.com > > ✗ [PATCH v4 3/10] btf_encoder: separate elf function, saved function representations > ✗ BADSIG: DKIM/pm.me > + Link: https://lore.kernel.org/r/20250107190855.2312210-4-ihor.solodrai@pm.me > + Signed-off-by: Arnaldo Carvalho de Melo acme@redhat.com > > ✗ [PATCH v4 4/10] btf_encoder: introduce elf_functions struct type > ✗ BADSIG: DKIM/pm.me > + Link: https://lore.kernel.org/r/20250107190855.2312210-5-ihor.solodrai@pm.me > + Signed-off-by: Arnaldo Carvalho de Melo acme@redhat.com > > ✗ [PATCH v4 5/10] btf_encoder: introduce elf_functions_list > ✗ BADSIG: DKIM/pm.me > + Link: https://lore.kernel.org/r/20250107190855.2312210-6-ihor.solodrai@pm.me > + Signed-off-by: Arnaldo Carvalho de Melo acme@redhat.com > > ✗ [PATCH v4 6/10] btf_encoder: remove skip_encoding_inconsistent_proto > ✗ BADSIG: DKIM/pm.me > + Link: https://lore.kernel.org/r/20250107190855.2312210-7-ihor.solodrai@pm.me > + Signed-off-by: Arnaldo Carvalho de Melo acme@redhat.com > > ✗ [PATCH v4 7/10] dwarf_loader: introduce cu->id > > ✗ BADSIG: DKIM/pm.me > + Link: https://lore.kernel.org/r/20250107190855.2312210-8-ihor.solodrai@pm.me > + Signed-off-by: Arnaldo Carvalho de Melo acme@redhat.com > > ERROR: missing [8/10]! > ✗ [PATCH v4 9/10] btf_encoder: clean up global encoders list > ✗ BADSIG: DKIM/pm.me > + Link: https://lore.kernel.org/r/20250107190855.2312210-10-ihor.solodrai@pm.me > + Signed-off-by: Arnaldo Carvalho de Melo acme@redhat.com > > ERROR: missing [10/10]! > ERROR: missing [11/10]! > --- > Total patches: 8 > --- > WARNING: Thread incomplete! > Cover: ./v4_20250107_ihor_solodrai_pahole_faster_reproducible_btf_encoding.cover > Link: https://lore.kernel.org/r/20250107190855.2312210-1-ihor.solodrai@pm.me > Base: not specified > git am ./v4_20250107_ihor_solodrai_pahole_faster_reproducible_btf_encoding.mbx > ⬢ [acme@toolbox pahole]$ > > > A notable adition is a patch 10/10, which changes func_states in > > btf_encoder from a list to an array. > > >
This is v4 of the series aiming to speed up parallel reproducible BTF encoding. This version mostly addresses feedback from Jiri Olsa on v3. A notable adition is a patch 10/10, which changes func_states in btf_encoder from a list to an array. Testing: vmlinux=/home/theihor/kernels/bpf-next/kbuild-output/.tmp_vmlinux1 PATH=$(realpath build):$PATH ./tests/tests 1: Validation of BTF encoding of functions; this may take some time: Ok 2: Default BTF on a system without BTF: Ok 3: Flexible arrays accounting: pahole: type 'nft_pipapo_elem' not found pahole: type 'tls_rec' not found pahole: type 'fuse_direntplus' not found pahole: type 'nft_rhash_elem' not found pahole: type 'nft_hash_elem' not found pahole: type 'nft_bitmap_elem' not found pahole: type 'ipt_standard' not found pahole: type 'nft_rule_dp_last' not found pahole: type 'ip6t_standard' not found pahole: type 'ipt_error' not found pahole: type 'ip6t_error' not found pahole: type 'nft_rbtree_elem' not found Ok 4: Check that pfunct can print btf_decl_tags read from BTF: Ok 5: Pretty printing of files using DWARF type information: Ok 6: Parallel reproducible DWARF Loading/Serial BTF encoding: Ok The warnings about not found types are also present at pahole/next, so not related to this patchset. Performance check. This patchset (always reproducible): Performance counter stats for '/home/theihor/dev/dwarves/build/pahole -J -j24 --btf_features=encode_force,var,float,enum64,decl_tag,type_tag,optimized_func,consistent_func,decl_tag_kfuncs,reproducible_build --btf_encode_detached=/dev/null --lang_exclude=rust /home/theihor/kernels/bpf-next/kbuild-output/.tmp_vmlinux1' (13 runs): 5,788.22 msec cpu-clock:u # 3.776 CPUs utilized ( +- 0.17% ) 1.53288 +- 0.00334 seconds time elapsed ( +- 0.22% ) pahole/next (d444eb6), parallel non-reproducible: Performance counter stats for '/home/theihor/dev/dwarves/build/pahole -J -j24 --btf_features=encode_force,var,float,enum64,decl_tag,type_tag,optimized_func,consistent_func,decl_tag_kfuncs --btf_encode_detached=/dev/null --lang_exclude=rust /home/theihor/kernels/bpf-next/kbuild-output/.tmp_vmlinux1' (13 runs): 10,462.38 msec cpu-clock:u # 6.678 CPUs utilized ( +- 0.15% ) 1.56670 +- 0.00548 seconds time elapsed ( +- 0.35% ) pahole/next (d444eb6), parallel reproducible: Performance counter stats for '/home/theihor/dev/dwarves/build/pahole -J -j24 --btf_features=encode_force,var,float,enum64,decl_tag,type_tag,optimized_func,consistent_func,decl_tag_kfuncs,reproducible_build --btf_encode_detached=/dev/null --lang_exclude=rust /home/theihor/kernels/bpf-next/kbuild-output/.tmp_vmlinux1' (13 runs): 6,399.88 msec cpu-clock:u # 3.164 CPUs utilized ( +- 0.22% ) 2.02269 +- 0.00359 seconds time elapsed ( +- 0.18% ) v3: https://lore.kernel.org/dwarves/20241221012245.243845-1-ihor.solodrai@pm.me/ v2: https://lore.kernel.org/dwarves/20241213223641.564002-1-ihor.solodrai@pm.me/ v1: https://lore.kernel.org/dwarves/20241128012341.4081072-1-ihor.solodrai@pm.me/ Alan Maguire (2): btf_encoder: simplify function encoding btf_encoder: separate elf function, saved function representations Ihor Solodrai (8): btf_encoder: free encoder->secinfo in btf_encoder__delete btf_encoder: introduce elf_functions struct type btf_encoder: introduce elf_functions_list btf_encoder: remove skip_encoding_inconsistent_proto dwarf_loader: introduce cu->id dwarf_loader: multithreading with a job/worker model btf_encoder: clean up global encoders list btf_encoder: switch func_states from a list to an array btf_encoder.c | 662 +++++++++++++++++++----------------- btf_encoder.h | 7 +- btf_loader.c | 2 +- ctf_loader.c | 2 +- dwarf_loader.c | 335 ++++++++++++------ dwarves.c | 44 --- dwarves.h | 21 +- pahole.c | 230 ++----------- pdwtags.c | 3 +- pfunct.c | 3 +- tests/reproducible_build.sh | 5 +- 11 files changed, 624 insertions(+), 690 deletions(-)