| Message ID | 20230728113415.21067-4-will@kernel.org (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
| Series | Fix 'faddr2line' for LLVM arm64 builds | expand |
On Fri, Jul 28, 2023 at 4:34 AM Will Deacon <will@kernel.org> wrote: > > Some symbols emitted in the readelf output but filtered from System.map > can confuse the 'faddr2line' symbol size calculation, resulting in the > erroneous rejection of valid offsets. This is especially prevalent when > building an arm64 kernel with CONFIG_CFI_CLANG=y, where most functions > are prefixed with a 32-bit data value in a '$d.n' section. For example: > > 447538: ffff800080014b80 548 FUNC GLOBAL DEFAULT 2 do_one_initcall > 104: ffff800080014c74 0 NOTYPE LOCAL DEFAULT 2 $x.73 > 106: ffff800080014d30 0 NOTYPE LOCAL DEFAULT 2 $x.75 > 111: ffff800080014da4 0 NOTYPE LOCAL DEFAULT 2 $d.78 > 112: ffff800080014da8 0 NOTYPE LOCAL DEFAULT 2 $x.79 > 36: ffff800080014de0 200 FUNC LOCAL DEFAULT 2 run_init_process Sami, Should we change the llvm-ir linkage type for these symbols from `internal` to `private`? https://llvm.org/docs/LangRef.html#linkage-types Then they would not appear in the symbol table. At first, I thought other modules might need to directly reference this data, but with the local binding, I don't think they can. > > Adding a warning to do_one_initcall() results in: > > | WARNING: CPU: 0 PID: 1 at init/main.c:1236 do_one_initcall+0xf4/0x260 > > Which 'faddr2line' refuses to accept: > > $ ./scripts/faddr2line vmlinux do_one_initcall+0xf4/0x260 > skipping do_one_initcall address at 0xffff800080014c74 due to size mismatch (0x260 != 0x224) > no match for do_one_initcall+0xf4/0x260 > > Filter out entries from readelf using the 'sysmap-ignored-syms.sed' > script used to construct System.map, so that the size of a symbol is > calculated as a delta to the next symbol present in ksymtab. > > Cc: Josh Poimboeuf <jpoimboe@kernel.org> > Cc: John Stultz <jstultz@google.com> > Signed-off-by: Will Deacon <will@kernel.org> > --- > scripts/faddr2line | 3 ++- > 1 file changed, 2 insertions(+), 1 deletion(-) > > diff --git a/scripts/faddr2line b/scripts/faddr2line > index 62a3fa6f6f59..da734af90036 100755 > --- a/scripts/faddr2line > +++ b/scripts/faddr2line > @@ -64,6 +64,7 @@ else > UTIL_PREFIX=${CROSS_COMPILE:-} > fi > > +IGNORED_SYMS=$(dirname $0)/sysmap-ignored-syms.sed > READELF="${UTIL_PREFIX}readelf" > ADDR2LINE="${UTIL_PREFIX}addr2line" > AWK="awk" > @@ -185,7 +186,7 @@ __faddr2line() { > found=2 > break > fi > - done < <(${READELF} --symbols --wide $objfile | sed 's/\[.*\]//' | ${AWK} -v sec=$sym_sec '$7 == sec' | sort --key=2) > + done < <(${READELF} --symbols --wide $objfile | sed -f ${IGNORED_SYMS} -e 's/\[.*\]//' | ${AWK} -v sec=$sym_sec '$7 == sec' | sort --key=2) > > if [[ $found = 0 ]]; then > warn "can't find symbol: sym_name: $sym_name sym_sec: $sym_sec sym_addr: $sym_addr sym_elf_size: $sym_elf_size" > -- > 2.41.0.487.g6d72f3e995-goog >
On Tue, Aug 1, 2023 at 9:42 AM Nick Desaulniers <ndesaulniers@google.com> wrote: > > On Fri, Jul 28, 2023 at 4:34 AM Will Deacon <will@kernel.org> wrote: > > > > Some symbols emitted in the readelf output but filtered from System.map > > can confuse the 'faddr2line' symbol size calculation, resulting in the > > erroneous rejection of valid offsets. This is especially prevalent when > > building an arm64 kernel with CONFIG_CFI_CLANG=y, where most functions > > are prefixed with a 32-bit data value in a '$d.n' section. For example: > > > > 447538: ffff800080014b80 548 FUNC GLOBAL DEFAULT 2 do_one_initcall > > 104: ffff800080014c74 0 NOTYPE LOCAL DEFAULT 2 $x.73 > > 106: ffff800080014d30 0 NOTYPE LOCAL DEFAULT 2 $x.75 > > 111: ffff800080014da4 0 NOTYPE LOCAL DEFAULT 2 $d.78 > > 112: ffff800080014da8 0 NOTYPE LOCAL DEFAULT 2 $x.79 > > 36: ffff800080014de0 200 FUNC LOCAL DEFAULT 2 run_init_process > > Sami, > Should we change the llvm-ir linkage type for these symbols from > `internal` to `private`? > https://llvm.org/docs/LangRef.html#linkage-types > > Then they would not appear in the symbol table. > > At first, I thought other modules might need to directly reference > this data, but with the local binding, I don't think they can. For arm64, we don't explicitly emit symbols for type prefixes (see AsmPrinter::emitKCFITypeId). These mapping symbols are emitted by AArch64ELFStreamer to mark data and code regions. According to the AArch64 ELF specification, "All mapping symbols have type STT_NOTYPE and binding STB_LOCAL," so I assume changing the linkage type isn't an option: https://github.com/ARM-software/abi-aa/blob/main/aaelf64/aaelf64.rst#mapping-symbols Sami
On Fri, Jul 28, 2023 at 8:34 PM Will Deacon <will@kernel.org> wrote: > > Some symbols emitted in the readelf output but filtered from System.map > can confuse the 'faddr2line' symbol size calculation, resulting in the > erroneous rejection of valid offsets. This is especially prevalent when > building an arm64 kernel with CONFIG_CFI_CLANG=y, where most functions > are prefixed with a 32-bit data value in a '$d.n' section. For example: > > 447538: ffff800080014b80 548 FUNC GLOBAL DEFAULT 2 do_one_initcall > 104: ffff800080014c74 0 NOTYPE LOCAL DEFAULT 2 $x.73 > 106: ffff800080014d30 0 NOTYPE LOCAL DEFAULT 2 $x.75 > 111: ffff800080014da4 0 NOTYPE LOCAL DEFAULT 2 $d.78 > 112: ffff800080014da8 0 NOTYPE LOCAL DEFAULT 2 $x.79 > 36: ffff800080014de0 200 FUNC LOCAL DEFAULT 2 run_init_process > > Adding a warning to do_one_initcall() results in: > > | WARNING: CPU: 0 PID: 1 at init/main.c:1236 do_one_initcall+0xf4/0x260 > > Which 'faddr2line' refuses to accept: > > $ ./scripts/faddr2line vmlinux do_one_initcall+0xf4/0x260 > skipping do_one_initcall address at 0xffff800080014c74 due to size mismatch (0x260 != 0x224) > no match for do_one_initcall+0xf4/0x260 > > Filter out entries from readelf using the 'sysmap-ignored-syms.sed' > script used to construct System.map, so that the size of a symbol is > calculated as a delta to the next symbol present in ksymtab. I do not think this patch set is the right approach. I assume faddr2line is meant to work with both vmlinux and modules. A problem is that we have different filtering policies wrt kallsyms. scripts/mksysmap filters symbols in vmlinux, while kernel/module/kallsyms.c filters ones in modules. This patch tries to get aligned with the stacktrace of vmlinux, but that does not seem optimal to the stacktrace of modules. I have not checked the details, but I guess the module kallsyms filters less symbols. https://github.com/torvalds/linux/blob/v6.5-rc4/kernel/module/kallsyms.c#L288 I prefer filtering symbols in the intersection of vmlinux and modules. is_mapping_symbol() filters symbols you are addressing. > Cc: Josh Poimboeuf <jpoimboe@kernel.org> > Cc: John Stultz <jstultz@google.com> > Signed-off-by: Will Deacon <will@kernel.org> > --- > scripts/faddr2line | 3 ++- > 1 file changed, 2 insertions(+), 1 deletion(-) > > diff --git a/scripts/faddr2line b/scripts/faddr2line > index 62a3fa6f6f59..da734af90036 100755 > --- a/scripts/faddr2line > +++ b/scripts/faddr2line > @@ -64,6 +64,7 @@ else > UTIL_PREFIX=${CROSS_COMPILE:-} > fi > > +IGNORED_SYMS=$(dirname $0)/sysmap-ignored-syms.sed > READELF="${UTIL_PREFIX}readelf" > ADDR2LINE="${UTIL_PREFIX}addr2line" > AWK="awk" > @@ -185,7 +186,7 @@ __faddr2line() { > found=2 > break > fi > - done < <(${READELF} --symbols --wide $objfile | sed 's/\[.*\]//' | ${AWK} -v sec=$sym_sec '$7 == sec' | sort --key=2) > + done < <(${READELF} --symbols --wide $objfile | sed -f ${IGNORED_SYMS} -e 's/\[.*\]//' | ${AWK} -v sec=$sym_sec '$7 == sec' | sort --key=2) > > if [[ $found = 0 ]]; then > warn "can't find symbol: sym_name: $sym_name sym_sec: $sym_sec sym_addr: $sym_addr sym_elf_size: $sym_elf_size" > -- > 2.41.0.487.g6d72f3e995-goog > -- Best Regards Masahiro Yamada
On Thu, Aug 03, 2023 at 04:54:37AM +0900, Masahiro Yamada wrote: > On Fri, Jul 28, 2023 at 8:34 PM Will Deacon <will@kernel.org> wrote: > > > > Some symbols emitted in the readelf output but filtered from System.map > > can confuse the 'faddr2line' symbol size calculation, resulting in the > > erroneous rejection of valid offsets. This is especially prevalent when > > building an arm64 kernel with CONFIG_CFI_CLANG=y, where most functions > > are prefixed with a 32-bit data value in a '$d.n' section. For example: > > > > 447538: ffff800080014b80 548 FUNC GLOBAL DEFAULT 2 do_one_initcall > > 104: ffff800080014c74 0 NOTYPE LOCAL DEFAULT 2 $x.73 > > 106: ffff800080014d30 0 NOTYPE LOCAL DEFAULT 2 $x.75 > > 111: ffff800080014da4 0 NOTYPE LOCAL DEFAULT 2 $d.78 > > 112: ffff800080014da8 0 NOTYPE LOCAL DEFAULT 2 $x.79 > > 36: ffff800080014de0 200 FUNC LOCAL DEFAULT 2 run_init_process > > > > Adding a warning to do_one_initcall() results in: > > > > | WARNING: CPU: 0 PID: 1 at init/main.c:1236 do_one_initcall+0xf4/0x260 > > > > Which 'faddr2line' refuses to accept: > > > > $ ./scripts/faddr2line vmlinux do_one_initcall+0xf4/0x260 > > skipping do_one_initcall address at 0xffff800080014c74 due to size mismatch (0x260 != 0x224) > > no match for do_one_initcall+0xf4/0x260 > > > > Filter out entries from readelf using the 'sysmap-ignored-syms.sed' > > script used to construct System.map, so that the size of a symbol is > > calculated as a delta to the next symbol present in ksymtab. > > > I do not think this patch set is the right approach. > > I assume faddr2line is meant to work with both vmlinux > and modules. Huh, it seems to be busted for modules :/ I get: | error: unknown argument '--section=.text' with llvm and: | addr2line: DWARF error: invalid or unhandled FORM value: 0x25 with binutils. I'll look into this, as I don't think it's related to symbol filtering. > A problem is that we have different filtering policies wrt kallsyms. > > scripts/mksysmap filters symbols in vmlinux, > while kernel/module/kallsyms.c filters ones in modules. I don't understand why we need two different ways of filtering out symbols, but it appears that the module case only filters out local labels and mapping symbols, both of which are filtered out of vmlinux as well. Is that right? > This patch tries to get aligned with the stacktrace of vmlinux, > but that does not seem optimal to the stacktrace of modules. > > > I have not checked the details, but I guess > the module kallsyms filters less symbols. > > https://github.com/torvalds/linux/blob/v6.5-rc4/kernel/module/kallsyms.c#L288 > > I prefer filtering symbols in the intersection of vmlinux and modules. I think mksysmap filters out a superset of the symbols which are filtered for modules, so why is the intersection the right thing to do? That will mean that faddr2line considers a whole bunch of symbols that aren't in the ksymtab of vmlinux. > is_mapping_symbol() filters symbols you are addressing. That's a C function and faddr2line is a shell script. What exactly do you want me to do? My first hack just matched on symbols starting with '$' but I ended up with this after other review feedback. Josh -- how do you want to proceed here? Will
On Fri, Aug 4, 2023 at 11:30 PM Will Deacon <will@kernel.org> wrote: > > On Thu, Aug 03, 2023 at 04:54:37AM +0900, Masahiro Yamada wrote: > > On Fri, Jul 28, 2023 at 8:34 PM Will Deacon <will@kernel.org> wrote: > > > > > > Some symbols emitted in the readelf output but filtered from System.map > > > can confuse the 'faddr2line' symbol size calculation, resulting in the > > > erroneous rejection of valid offsets. This is especially prevalent when > > > building an arm64 kernel with CONFIG_CFI_CLANG=y, where most functions > > > are prefixed with a 32-bit data value in a '$d.n' section. For example: > > > > > > 447538: ffff800080014b80 548 FUNC GLOBAL DEFAULT 2 do_one_initcall > > > 104: ffff800080014c74 0 NOTYPE LOCAL DEFAULT 2 $x.73 > > > 106: ffff800080014d30 0 NOTYPE LOCAL DEFAULT 2 $x.75 > > > 111: ffff800080014da4 0 NOTYPE LOCAL DEFAULT 2 $d.78 > > > 112: ffff800080014da8 0 NOTYPE LOCAL DEFAULT 2 $x.79 > > > 36: ffff800080014de0 200 FUNC LOCAL DEFAULT 2 run_init_process > > > > > > Adding a warning to do_one_initcall() results in: > > > > > > | WARNING: CPU: 0 PID: 1 at init/main.c:1236 do_one_initcall+0xf4/0x260 > > > > > > Which 'faddr2line' refuses to accept: > > > > > > $ ./scripts/faddr2line vmlinux do_one_initcall+0xf4/0x260 > > > skipping do_one_initcall address at 0xffff800080014c74 due to size mismatch (0x260 != 0x224) > > > no match for do_one_initcall+0xf4/0x260 > > > > > > Filter out entries from readelf using the 'sysmap-ignored-syms.sed' > > > script used to construct System.map, so that the size of a symbol is > > > calculated as a delta to the next symbol present in ksymtab. > > > > > > I do not think this patch set is the right approach. > > > > I assume faddr2line is meant to work with both vmlinux > > and modules. > > Huh, it seems to be busted for modules :/ I get: > > | error: unknown argument '--section=.text' > > with llvm and: > > | addr2line: DWARF error: invalid or unhandled FORM value: 0x25 > > with binutils. > > I'll look into this, as I don't think it's related to symbol filtering. > > > A problem is that we have different filtering policies wrt kallsyms. > > > > scripts/mksysmap filters symbols in vmlinux, > > while kernel/module/kallsyms.c filters ones in modules. > > I don't understand why we need two different ways of filtering out > symbols, but it appears that the module case only filters out local > labels and mapping symbols, both of which are filtered out of vmlinux > as well. Is that right? For vmlinux kallsyms, the reason is the annoyance in the build process. kallsyms requires linking vmlinux three times. (see scripts/link-vmlinux.sh if you are interested) [1a] link vmlinux without symbol table [1b] Generate symbol list from [1a] [2a] link vmlinux, embedding the temp kallsyms created by [1b] [2b] Generate symbol list from [2a] [3a] link vmlinux, embedding the final kallsyms created by [2b] Our assumption is that [1b] and [2b] have the same number of symbols, so that [2a] and [3a] can embed the same size kallsyms. However, the process [2a] inserts the kallsyms table in the middle of vmlinux. It may cause the compiler to emit additional symbols (e.g. ARM veneers) due to the inserted kallsyms. So, we filter out all compiler-generated symbols so that [1b] and [2b] have the same number of symbols. If we do not do it, we may have more and more kallsyms steps. Emitted veneers increase kallsyms size -> Increased kallsyms causes more veneers -> The new veneers increase kallsyms size -> Increased kallsyms causes yet more veneers -> The new veneers increase kallsyms size again -> (this cycle may continue again and again) > > This patch tries to get aligned with the stacktrace of vmlinux, > > but that does not seem optimal to the stacktrace of modules. > > > > > > I have not checked the details, but I guess > > the module kallsyms filters less symbols. > > > > https://github.com/torvalds/linux/blob/v6.5-rc4/kernel/module/kallsyms.c#L288 > > > > I prefer filtering symbols in the intersection of vmlinux and modules. > > I think mksysmap filters out a superset of the symbols which are filtered > for modules, so why is the intersection the right thing to do? That will > mean that faddr2line considers a whole bunch of symbols that aren't in > the ksymtab of vmlinux. If A is a subset of B, (A & B) == A. I meant "the intersection of the filtering rules of vmlinux and modules" == "the filtering rule of modules". > > > is_mapping_symbol() filters symbols you are addressing. > > That's a C function and faddr2line is a shell script. What exactly do > you want me to do? My first hack just matched on symbols starting with > '$' but I ended up with this after other review feedback. In linux-next, I see this: static inline int is_mapping_symbol(const char *str) { if (str[0] == '.' && str[1] == 'L') return true; if (str[0] == 'L' && str[1] == '0') return true; return str[0] == '$'; } It is pretty easy to write a sed script to do the same thing. Another possibility is to use different filtering rules between vmlinux and modules. (but I do not know if this complexity is worthwhile) [pseudo code] get_symbol_list () { if [ "${is_vmlinux}" = 1 ]; then {READELF} --symbols --wide $objfile | sed -f scripts/sysmap-ignored-syms.sed else {READELF} --symbols --wide $objfile | sed 'remove ".L", 'L0", "^$"' fi } > > Josh -- how do you want to proceed here? > > Will
diff --git a/scripts/faddr2line b/scripts/faddr2line index 62a3fa6f6f59..da734af90036 100755 --- a/scripts/faddr2line +++ b/scripts/faddr2line @@ -64,6 +64,7 @@ else UTIL_PREFIX=${CROSS_COMPILE:-} fi +IGNORED_SYMS=$(dirname $0)/sysmap-ignored-syms.sed READELF="${UTIL_PREFIX}readelf" ADDR2LINE="${UTIL_PREFIX}addr2line" AWK="awk" @@ -185,7 +186,7 @@ __faddr2line() { found=2 break fi - done < <(${READELF} --symbols --wide $objfile | sed 's/\[.*\]//' | ${AWK} -v sec=$sym_sec '$7 == sec' | sort --key=2) + done < <(${READELF} --symbols --wide $objfile | sed -f ${IGNORED_SYMS} -e 's/\[.*\]//' | ${AWK} -v sec=$sym_sec '$7 == sec' | sort --key=2) if [[ $found = 0 ]]; then warn "can't find symbol: sym_name: $sym_name sym_sec: $sym_sec sym_addr: $sym_addr sym_elf_size: $sym_elf_size"
Some symbols emitted in the readelf output but filtered from System.map can confuse the 'faddr2line' symbol size calculation, resulting in the erroneous rejection of valid offsets. This is especially prevalent when building an arm64 kernel with CONFIG_CFI_CLANG=y, where most functions are prefixed with a 32-bit data value in a '$d.n' section. For example: 447538: ffff800080014b80 548 FUNC GLOBAL DEFAULT 2 do_one_initcall 104: ffff800080014c74 0 NOTYPE LOCAL DEFAULT 2 $x.73 106: ffff800080014d30 0 NOTYPE LOCAL DEFAULT 2 $x.75 111: ffff800080014da4 0 NOTYPE LOCAL DEFAULT 2 $d.78 112: ffff800080014da8 0 NOTYPE LOCAL DEFAULT 2 $x.79 36: ffff800080014de0 200 FUNC LOCAL DEFAULT 2 run_init_process Adding a warning to do_one_initcall() results in: | WARNING: CPU: 0 PID: 1 at init/main.c:1236 do_one_initcall+0xf4/0x260 Which 'faddr2line' refuses to accept: $ ./scripts/faddr2line vmlinux do_one_initcall+0xf4/0x260 skipping do_one_initcall address at 0xffff800080014c74 due to size mismatch (0x260 != 0x224) no match for do_one_initcall+0xf4/0x260 Filter out entries from readelf using the 'sysmap-ignored-syms.sed' script used to construct System.map, so that the size of a symbol is calculated as a delta to the next symbol present in ksymtab. Cc: Josh Poimboeuf <jpoimboe@kernel.org> Cc: John Stultz <jstultz@google.com> Signed-off-by: Will Deacon <will@kernel.org> --- scripts/faddr2line | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)