| Message ID | 20240228032142.396719-1-jhubbard@nvidia.com (mailing list archive) |
|---|---|
| State | Changes Requested |
| Delegated to: | BPF |
| Headers | show |
| Series | fix linux kernel BTF builds: increase max percpu variables by 10x | expand |
On Tue, Feb 27, 2024 at 07:21:42PM -0800, John Hubbard wrote: > When building the Linux kernel with a distro .config, most or even all > possible kernel modules are built. This adds up to 4500+ modules, and > based on my testing, this causes the pahole utility to run out of space, > which shows up like this (CONFIG_DEBUG_INFO_BTF=y is required in order > to reproduce this): > > LD .tmp_vmlinux.btf > BTF .btf.vmlinux.bin.o > Reached the limit of per-CPU variables: 4096 > ...repeated many times... > Reached the limit of per-CPU variables: 4096 > LD .tmp_vmlinux.kallsyms1 > NM .tmp_vmlinux.kallsyms1.syms > KSYMS .tmp_vmlinux.kallsyms1.S > AS .tmp_vmlinux.kallsyms1.S > LD .tmp_vmlinux.kallsyms2 > NM .tmp_vmlinux.kallsyms2.syms > KSYMS .tmp_vmlinux.kallsyms2.S > AS .tmp_vmlinux.kallsyms2.S > LD vmlinux > BTFIDS vmlinux > libbpf: failed to find '.BTF' ELF section in vmlinux > FAILED: load BTF from vmlinux: No data available > make[2]: *** [scripts/Makefile.vmlinux:37: vmlinux] Error 255 > make[2]: *** Deleting file 'vmlinux' > make[1]: *** [/kernel_work/linux-people/Makefile:1162: vmlinux] Error 2 > make: *** [Makefile:240: __sub-make] Error 2 > > Increasing MAX_PERCPU_VAR_CNT by 10x avoids running out of space, and > allows the build to succeed. do you have an actual count of percpu variables for your config? 10x seems a lot to me this might be a workaround, but we should make encoder->percpu.vars dynamically allocated like we do for functions jirka > > Signed-off-by: John Hubbard <jhubbard@nvidia.com> > --- > btf_encoder.c | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > > diff --git a/btf_encoder.c b/btf_encoder.c > index fd04008..d9f4e80 100644 > --- a/btf_encoder.c > +++ b/btf_encoder.c > @@ -50,7 +50,7 @@ struct elf_function { > struct btf_encoder_state state; > }; > > -#define MAX_PERCPU_VAR_CNT 4096 > +#define MAX_PERCPU_VAR_CNT 40960 > > struct var_info { > uint64_t addr; > -- > 2.44.0 > >
On 28/02/2024 09:20, Jiri Olsa wrote: > On Tue, Feb 27, 2024 at 07:21:42PM -0800, John Hubbard wrote: >> When building the Linux kernel with a distro .config, most or even all >> possible kernel modules are built. This adds up to 4500+ modules, and >> based on my testing, this causes the pahole utility to run out of space, >> which shows up like this (CONFIG_DEBUG_INFO_BTF=y is required in order >> to reproduce this): >> >> LD .tmp_vmlinux.btf >> BTF .btf.vmlinux.bin.o >> Reached the limit of per-CPU variables: 4096 >> ...repeated many times... >> Reached the limit of per-CPU variables: 4096 >> LD .tmp_vmlinux.kallsyms1 >> NM .tmp_vmlinux.kallsyms1.syms >> KSYMS .tmp_vmlinux.kallsyms1.S >> AS .tmp_vmlinux.kallsyms1.S >> LD .tmp_vmlinux.kallsyms2 >> NM .tmp_vmlinux.kallsyms2.syms >> KSYMS .tmp_vmlinux.kallsyms2.S >> AS .tmp_vmlinux.kallsyms2.S >> LD vmlinux >> BTFIDS vmlinux >> libbpf: failed to find '.BTF' ELF section in vmlinux >> FAILED: load BTF from vmlinux: No data available >> make[2]: *** [scripts/Makefile.vmlinux:37: vmlinux] Error 255 >> make[2]: *** Deleting file 'vmlinux' >> make[1]: *** [/kernel_work/linux-people/Makefile:1162: vmlinux] Error 2 >> make: *** [Makefile:240: __sub-make] Error 2 >> >> Increasing MAX_PERCPU_VAR_CNT by 10x avoids running out of space, and >> allows the build to succeed. > > do you have an actual count of percpu variables for your config? > 10x seems a lot to me > > this might be a workaround, but we should make encoder->percpu.vars > dynamically allocated like we do for functions > > jirka > Good idea Jiri; John would you mind trying the attached patch? Thanks! Alan
On 2/28/24 04:04, Alan Maguire wrote: > On 28/02/2024 09:20, Jiri Olsa wrote: >> On Tue, Feb 27, 2024 at 07:21:42PM -0800, John Hubbard wrote: ... >> do you have an actual count of percpu variables for your config? That's a very reasonable question... >> 10x seems a lot to me Me too. This was a "make the problem go away now please" type of "fix". :) >> >> this might be a workaround, but we should make encoder->percpu.vars >> dynamically allocated like we do for functions Yes, that's a much better design imho. >> >> jirka >> > > Good idea Jiri; John would you mind trying the attached patch? Thanks! It works perfectly for me. For that patch, please feel free to add: Tested-by: John Hubbard <jhubbard@nvidia.com> thanks,
On 28/02/2024 23:21, John Hubbard wrote: > On 2/28/24 04:04, Alan Maguire wrote: >> On 28/02/2024 09:20, Jiri Olsa wrote: >>> On Tue, Feb 27, 2024 at 07:21:42PM -0800, John Hubbard wrote: > ... >>> do you have an actual count of percpu variables for your config? > > That's a very reasonable question... > >>> 10x seems a lot to me > > Me too. This was a "make the problem go away now please" type of "fix". :) > Running bpftool btf dump file vmlinux |grep "] VAR" ...should give us a sense of what's going on. I only see 375 per-cpu variables when I do this so maybe there's something kernel-config-specific that might explain why you have so many more? >>> >>> this might be a workaround, but we should make encoder->percpu.vars >>> dynamically allocated like we do for functions > > Yes, that's a much better design imho. > >>> >>> jirka >>> >> >> Good idea Jiri; John would you mind trying the attached patch? Thanks! > > It works perfectly for me. For that patch, please feel free to add: > > Tested-by: John Hubbard <jhubbard@nvidia.com> > that's great, thanks for testing this John! Alan > > thanks,
> ... > Running > > bpftool btf dump file vmlinux |grep "] VAR" > $ bpftool btf dump file vmlinux |grep "] VAR" | wc -l 4852 $ bpftool btf dump file vmlinux |grep "] VAR" | tail -5 [136994] VAR '_alloc_tag_cntr.9' type_id=703, linkage=static [137003] VAR '_alloc_tag_cntr.5' type_id=703, linkage=static [137004] VAR '_alloc_tag_cntr.7' type_id=703, linkage=static [137005] VAR '_alloc_tag_cntr.17' type_id=703, linkage=static [137018] VAR '_alloc_tag_cntr.14' type_id=703, linkage=static > ...should give us a sense of what's going on. I only see 375 per-cpu > variables when I do this so maybe there's something > kernel-config-specific that might explain why you have so many more? Yes, as mentioned earlier, this is specifically due to the .config. The .config is a huge distro configuration that has a lot of modules enabled. thanks,
On Thu, Feb 29, 2024 at 10:15:23AM -0800, John Hubbard wrote: > > ... > > Running > > > > bpftool btf dump file vmlinux |grep "] VAR" > > > > $ bpftool btf dump file vmlinux |grep "] VAR" | wc -l > 4852 > > $ bpftool btf dump file vmlinux |grep "] VAR" | tail -5 > [136994] VAR '_alloc_tag_cntr.9' type_id=703, linkage=static > [137003] VAR '_alloc_tag_cntr.5' type_id=703, linkage=static > [137004] VAR '_alloc_tag_cntr.7' type_id=703, linkage=static > [137005] VAR '_alloc_tag_cntr.17' type_id=703, linkage=static > [137018] VAR '_alloc_tag_cntr.14' type_id=703, linkage=static > > > ...should give us a sense of what's going on. I only see 375 per-cpu > > variables when I do this so maybe there's something > > kernel-config-specific that might explain why you have so many more? > > Yes, as mentioned earlier, this is specifically due to the .config. > The .config is a huge distro configuration that has a lot of modules > enabled. could you share your .config? I tried with fedora .config and got 396 per cpu variables, I wonder where this is coming from thanks, jirka
On 3/1/24 06:14, Jiri Olsa wrote: > On Thu, Feb 29, 2024 at 10:15:23AM -0800, John Hubbard wrote: >>> ... >>> Running >>> >>> bpftool btf dump file vmlinux |grep "] VAR" >>> >> >> $ bpftool btf dump file vmlinux |grep "] VAR" | wc -l >> 4852 >> >> $ bpftool btf dump file vmlinux |grep "] VAR" | tail -5 >> [136994] VAR '_alloc_tag_cntr.9' type_id=703, linkage=static >> [137003] VAR '_alloc_tag_cntr.5' type_id=703, linkage=static >> [137004] VAR '_alloc_tag_cntr.7' type_id=703, linkage=static >> [137005] VAR '_alloc_tag_cntr.17' type_id=703, linkage=static >> [137018] VAR '_alloc_tag_cntr.14' type_id=703, linkage=static >> >>> ...should give us a sense of what's going on. I only see 375 per-cpu >>> variables when I do this so maybe there's something >>> kernel-config-specific that might explain why you have so many more? >> >> Yes, as mentioned earlier, this is specifically due to the .config. >> The .config is a huge distro configuration that has a lot of modules >> enabled. > > could you share your .config? I tried with fedora .config and got 396 > per cpu variables, I wonder where this is coming from > Attaching it. And based on your results, I think that Suren's Memory allocation profiling patchset v4 [1] may also be required, as that is what I was building. Cc: Suren and Kent. btw, I the whole reason I went down this path was that I recommended your patchset in order to zero in on a memory leak that a colleague is debugging. This patchset provides a new view of allocations and leaks and we have high hopes for it. :) [1] https://lore.kernel.org/20240221194052.927623-1-surenb@google.com thanks,
On Fri, Mar 1, 2024 at 6:33 PM John Hubbard <jhubbard@nvidia.com> wrote: > > On 3/1/24 06:14, Jiri Olsa wrote: > > On Thu, Feb 29, 2024 at 10:15:23AM -0800, John Hubbard wrote: > >>> ... > >>> Running > >>> > >>> bpftool btf dump file vmlinux |grep "] VAR" > >>> > >> > >> $ bpftool btf dump file vmlinux |grep "] VAR" | wc -l > >> 4852 > >> > >> $ bpftool btf dump file vmlinux |grep "] VAR" | tail -5 > >> [136994] VAR '_alloc_tag_cntr.9' type_id=703, linkage=static > >> [137003] VAR '_alloc_tag_cntr.5' type_id=703, linkage=static > >> [137004] VAR '_alloc_tag_cntr.7' type_id=703, linkage=static > >> [137005] VAR '_alloc_tag_cntr.17' type_id=703, linkage=static > >> [137018] VAR '_alloc_tag_cntr.14' type_id=703, linkage=static > >> > >>> ...should give us a sense of what's going on. I only see 375 per-cpu > >>> variables when I do this so maybe there's something > >>> kernel-config-specific that might explain why you have so many more? > >> > >> Yes, as mentioned earlier, this is specifically due to the .config. > >> The .config is a huge distro configuration that has a lot of modules > >> enabled. > > > > could you share your .config? I tried with fedora .config and got 396 > > per cpu variables, I wonder where this is coming from > > > > Attaching it. And based on your results, I think that Suren's Memory > allocation profiling patchset v4 [1] may also be required, as that is what > I was building. Yes, that will definitely increase the number of required per-cpu variables since it adds a per-cpu variable for each kernel allocation to track it. I vaguely remember now that Johannes also mentioned hitting this limit when he was using our patchset. Allocating encoder->percpu.vars dynamically seems to be a great way to fix this limitation. > > Cc: Suren and Kent. btw, I the whole reason I went down this path was that > I recommended your patchset in order to zero in on a memory leak that a > colleague is debugging. This patchset provides a new view of allocations > and leaks and we have high hopes for it. :) Thanks for trying it out John and CC'ing us on this thread! Suren. > > > [1] https://lore.kernel.org/20240221194052.927623-1-surenb@google.com > > > thanks, > -- > John Hubbard > NVIDIA
diff --git a/btf_encoder.c b/btf_encoder.c index fd04008..d9f4e80 100644 --- a/btf_encoder.c +++ b/btf_encoder.c @@ -50,7 +50,7 @@ struct elf_function { struct btf_encoder_state state; }; -#define MAX_PERCPU_VAR_CNT 4096 +#define MAX_PERCPU_VAR_CNT 40960 struct var_info { uint64_t addr;
When building the Linux kernel with a distro .config, most or even all possible kernel modules are built. This adds up to 4500+ modules, and based on my testing, this causes the pahole utility to run out of space, which shows up like this (CONFIG_DEBUG_INFO_BTF=y is required in order to reproduce this): LD .tmp_vmlinux.btf BTF .btf.vmlinux.bin.o Reached the limit of per-CPU variables: 4096 ...repeated many times... Reached the limit of per-CPU variables: 4096 LD .tmp_vmlinux.kallsyms1 NM .tmp_vmlinux.kallsyms1.syms KSYMS .tmp_vmlinux.kallsyms1.S AS .tmp_vmlinux.kallsyms1.S LD .tmp_vmlinux.kallsyms2 NM .tmp_vmlinux.kallsyms2.syms KSYMS .tmp_vmlinux.kallsyms2.S AS .tmp_vmlinux.kallsyms2.S LD vmlinux BTFIDS vmlinux libbpf: failed to find '.BTF' ELF section in vmlinux FAILED: load BTF from vmlinux: No data available make[2]: *** [scripts/Makefile.vmlinux:37: vmlinux] Error 255 make[2]: *** Deleting file 'vmlinux' make[1]: *** [/kernel_work/linux-people/Makefile:1162: vmlinux] Error 2 make: *** [Makefile:240: __sub-make] Error 2 Increasing MAX_PERCPU_VAR_CNT by 10x avoids running out of space, and allows the build to succeed. Signed-off-by: John Hubbard <jhubbard@nvidia.com> --- btf_encoder.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)