diff mbox series

[1/4] kbuild: fix UNUSED_KSYMS_WHITELIST for Clang LTO

Message ID 20210225160247.2959903-2-masahiroy@kernel.org (mailing list archive)
State New, archived
Headers show
Series kbuild: build speed improvment of CONFIG_TRIM_UNUSED_KSYMS | expand

Commit Message

Masahiro Yamada Feb. 25, 2021, 4:02 p.m. UTC 
Commit fbe078d397b4 ("kbuild: lto: add a default list of used symbols")
does not work as expected if the .config file has already specified
CONFIG_UNUSED_KSYMS_WHITELIST="my/own/white/list" before enabling
CONFIG_LTO_CLANG.

So, the user-supplied whitelist and LTO-specific white list must be
independent of each other.

I refactored the shell script so CONFIG_MODVERSIONS and CONFIG_CLANG_LTO
handle whitelists in the same way.

Fixes: fbe078d397b4 ("kbuild: lto: add a default list of used symbols")
Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
---

 init/Kconfig                    |  1 -
 scripts/gen_autoksyms.sh        | 33 ++++++++++++++++++++++++---------
 scripts/lto-used-symbollist.txt |  5 -----
 3 files changed, 24 insertions(+), 15 deletions(-)
 delete mode 100644 scripts/lto-used-symbollist.txt

Comments

Sami Tolvanen Feb. 25, 2021, 5:45 p.m. UTC | #1 
Hi Masahiro,

On Thu, Feb 25, 2021 at 8:03 AM Masahiro Yamada <masahiroy@kernel.org> wrote:
>
> Commit fbe078d397b4 ("kbuild: lto: add a default list of used symbols")
> does not work as expected if the .config file has already specified
> CONFIG_UNUSED_KSYMS_WHITELIST="my/own/white/list" before enabling
> CONFIG_LTO_CLANG.
>
> So, the user-supplied whitelist and LTO-specific white list must be
> independent of each other.
>
> I refactored the shell script so CONFIG_MODVERSIONS and CONFIG_CLANG_LTO
> handle whitelists in the same way.
>
> Fixes: fbe078d397b4 ("kbuild: lto: add a default list of used symbols")
> Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
> ---
>
>  init/Kconfig                    |  1 -
>  scripts/gen_autoksyms.sh        | 33 ++++++++++++++++++++++++---------
>  scripts/lto-used-symbollist.txt |  5 -----
>  3 files changed, 24 insertions(+), 15 deletions(-)
>  delete mode 100644 scripts/lto-used-symbollist.txt

> +
> +ksym_wl=
>  if [ -n "$CONFIG_UNUSED_KSYMS_WHITELIST" ]; then
>         # Use 'eval' to expand the whitelist path and check if it is relative
>         eval ksym_wl="$CONFIG_UNUSED_KSYMS_WHITELIST"
> @@ -40,16 +57,14 @@ cat > "$output_file" << EOT
>  EOT
>
>  [ -f modules.order ] && modlist=modules.order || modlist=/dev/null
> -sed 's/ko$/mod/' $modlist |
> -xargs -n1 sed -n -e '2{s/ /\n/g;/^$/!p;}' -- |
> -cat - "$ksym_wl" |
> +
> +{
> +       sed 's/ko$/mod/' $modlist | xargs -n1 sed -n -e '2p'
> +       echo "$needed_symbols"
> +       [ -n "$ksym_wl" ] && cat "$ksym_wl"
> +} | sed -e 's/ /\n/g' | sed -n -e '/^$/!p' |
>  # Remove the dot prefix for ppc64; symbol names with a dot (.) hold entry
>  # point addresses.
>  sed -e 's/^\.//' |
>  sort -u |
>  sed -e 's/\(.*\)/#define __KSYM_\1 1/' >> "$output_file"
> -
> -# Special case for modversions (see modpost.c)
> -if [ -n "$CONFIG_MODVERSIONS" ]; then
> -       echo "#define __KSYM_module_layout 1" >> "$output_file"
> -fi
> diff --git a/scripts/lto-used-symbollist.txt b/scripts/lto-used-symbollist.txt
> deleted file mode 100644
> index 38e7bb9ebaae..000000000000
> --- a/scripts/lto-used-symbollist.txt
> +++ /dev/null
> @@ -1,5 +0,0 @@
> -memcpy
> -memmove
> -memset
> -__stack_chk_fail
> -__stack_chk_guard
> --
> 2.27.0
>
>
> diff --git a/init/Kconfig b/init/Kconfig
> index 0bf5b340b80e..351161326e3c 100644
> --- a/init/Kconfig
> +++ b/init/Kconfig
> @@ -2277,7 +2277,6 @@ config TRIM_UNUSED_KSYMS
>  config UNUSED_KSYMS_WHITELIST
>         string "Whitelist of symbols to keep in ksymtab"
>         depends on TRIM_UNUSED_KSYMS
> -       default "scripts/lto-used-symbollist.txt" if LTO_CLANG
>         help
>           By default, all unused exported symbols will be un-exported from the
>           build when TRIM_UNUSED_KSYMS is selected.
> diff --git a/scripts/gen_autoksyms.sh b/scripts/gen_autoksyms.sh
> index d54dfba15bf2..b74d5949fea6 100755
> --- a/scripts/gen_autoksyms.sh
> +++ b/scripts/gen_autoksyms.sh
> @@ -19,7 +19,24 @@ esac
>  # We need access to CONFIG_ symbols
>  . include/config/auto.conf
>
> -ksym_wl=/dev/null
> +needed_symbols=
> +
> +# Special case for modversions (see modpost.c)
> +if [ -n "$CONFIG_MODVERSIONS" ]; then
> +       needed_symbols="$needed_symbols module_layout"
> +fi
> +
> +# With CONFIG_LTO_CLANG, LLVM bitcode has not yet been compiled into a binary
> +# when the .mod files are generated, which means they don't yet contain
> +# references to certain symbols that will be present in the final binaries.
> +if [ -n "$CONFIG_LTO_CLANG" ]; then
> +       # intrinsic functions
> +       needed_symbols="$needed_symbols memcpy memmove memset"
> +       # stack protector symbols
> +       needed_symbols="$needed_symbols __stack_chk_fail __stack_chk_guard"
> +fi

Thank you for the patch!

Arnd just reported that _mcount is also needed with some
configurations. Would you mind including that in the next version?

https://lore.kernel.org/r/20210225143456.3829513-1-arnd@kernel.org/

Sami
Masahiro Yamada Feb. 25, 2021, 7:08 p.m. UTC | #2 
On Fri, Feb 26, 2021 at 2:46 AM Sami Tolvanen <samitolvanen@google.com> wrote:
>
> Hi Masahiro,
>
> On Thu, Feb 25, 2021 at 8:03 AM Masahiro Yamada <masahiroy@kernel.org> wrote:
> >
> > Commit fbe078d397b4 ("kbuild: lto: add a default list of used symbols")
> > does not work as expected if the .config file has already specified
> > CONFIG_UNUSED_KSYMS_WHITELIST="my/own/white/list" before enabling
> > CONFIG_LTO_CLANG.
> >
> > So, the user-supplied whitelist and LTO-specific white list must be
> > independent of each other.
> >
> > I refactored the shell script so CONFIG_MODVERSIONS and CONFIG_CLANG_LTO
> > handle whitelists in the same way.
> >
> > Fixes: fbe078d397b4 ("kbuild: lto: add a default list of used symbols")
> > Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
> > ---
> >
> >  init/Kconfig                    |  1 -
> >  scripts/gen_autoksyms.sh        | 33 ++++++++++++++++++++++++---------
> >  scripts/lto-used-symbollist.txt |  5 -----
> >  3 files changed, 24 insertions(+), 15 deletions(-)
> >  delete mode 100644 scripts/lto-used-symbollist.txt
>
> > +
> > +ksym_wl=
> >  if [ -n "$CONFIG_UNUSED_KSYMS_WHITELIST" ]; then
> >         # Use 'eval' to expand the whitelist path and check if it is relative
> >         eval ksym_wl="$CONFIG_UNUSED_KSYMS_WHITELIST"
> > @@ -40,16 +57,14 @@ cat > "$output_file" << EOT
> >  EOT
> >
> >  [ -f modules.order ] && modlist=modules.order || modlist=/dev/null
> > -sed 's/ko$/mod/' $modlist |
> > -xargs -n1 sed -n -e '2{s/ /\n/g;/^$/!p;}' -- |
> > -cat - "$ksym_wl" |
> > +
> > +{
> > +       sed 's/ko$/mod/' $modlist | xargs -n1 sed -n -e '2p'
> > +       echo "$needed_symbols"
> > +       [ -n "$ksym_wl" ] && cat "$ksym_wl"
> > +} | sed -e 's/ /\n/g' | sed -n -e '/^$/!p' |
> >  # Remove the dot prefix for ppc64; symbol names with a dot (.) hold entry
> >  # point addresses.
> >  sed -e 's/^\.//' |
> >  sort -u |
> >  sed -e 's/\(.*\)/#define __KSYM_\1 1/' >> "$output_file"
> > -
> > -# Special case for modversions (see modpost.c)
> > -if [ -n "$CONFIG_MODVERSIONS" ]; then
> > -       echo "#define __KSYM_module_layout 1" >> "$output_file"
> > -fi
> > diff --git a/scripts/lto-used-symbollist.txt b/scripts/lto-used-symbollist.txt
> > deleted file mode 100644
> > index 38e7bb9ebaae..000000000000
> > --- a/scripts/lto-used-symbollist.txt
> > +++ /dev/null
> > @@ -1,5 +0,0 @@
> > -memcpy
> > -memmove
> > -memset
> > -__stack_chk_fail
> > -__stack_chk_guard
> > --
> > 2.27.0
> >
> >
> > diff --git a/init/Kconfig b/init/Kconfig
> > index 0bf5b340b80e..351161326e3c 100644
> > --- a/init/Kconfig
> > +++ b/init/Kconfig
> > @@ -2277,7 +2277,6 @@ config TRIM_UNUSED_KSYMS
> >  config UNUSED_KSYMS_WHITELIST
> >         string "Whitelist of symbols to keep in ksymtab"
> >         depends on TRIM_UNUSED_KSYMS
> > -       default "scripts/lto-used-symbollist.txt" if LTO_CLANG
> >         help
> >           By default, all unused exported symbols will be un-exported from the
> >           build when TRIM_UNUSED_KSYMS is selected.
> > diff --git a/scripts/gen_autoksyms.sh b/scripts/gen_autoksyms.sh
> > index d54dfba15bf2..b74d5949fea6 100755
> > --- a/scripts/gen_autoksyms.sh
> > +++ b/scripts/gen_autoksyms.sh
> > @@ -19,7 +19,24 @@ esac
> >  # We need access to CONFIG_ symbols
> >  . include/config/auto.conf
> >
> > -ksym_wl=/dev/null
> > +needed_symbols=
> > +
> > +# Special case for modversions (see modpost.c)
> > +if [ -n "$CONFIG_MODVERSIONS" ]; then
> > +       needed_symbols="$needed_symbols module_layout"
> > +fi
> > +
> > +# With CONFIG_LTO_CLANG, LLVM bitcode has not yet been compiled into a binary
> > +# when the .mod files are generated, which means they don't yet contain
> > +# references to certain symbols that will be present in the final binaries.
> > +if [ -n "$CONFIG_LTO_CLANG" ]; then
> > +       # intrinsic functions
> > +       needed_symbols="$needed_symbols memcpy memmove memset"
> > +       # stack protector symbols
> > +       needed_symbols="$needed_symbols __stack_chk_fail __stack_chk_guard"
> > +fi
>
> Thank you for the patch!
>
> Arnd just reported that _mcount is also needed with some
> configurations. Would you mind including that in the next version?
>
> https://lore.kernel.org/r/20210225143456.3829513-1-arnd@kernel.org/

Sure, I can even pick it up
although that patch was not addressed to me or kbuild ML.
diff mbox series

Patch

diff --git a/init/Kconfig b/init/Kconfig
index 0bf5b340b80e..351161326e3c 100644
--- a/init/Kconfig
+++ b/init/Kconfig
@@ -2277,7 +2277,6 @@  config TRIM_UNUSED_KSYMS
 config UNUSED_KSYMS_WHITELIST
 	string "Whitelist of symbols to keep in ksymtab"
 	depends on TRIM_UNUSED_KSYMS
-	default "scripts/lto-used-symbollist.txt" if LTO_CLANG
 	help
 	  By default, all unused exported symbols will be un-exported from the
 	  build when TRIM_UNUSED_KSYMS is selected.
diff --git a/scripts/gen_autoksyms.sh b/scripts/gen_autoksyms.sh
index d54dfba15bf2..b74d5949fea6 100755
--- a/scripts/gen_autoksyms.sh
+++ b/scripts/gen_autoksyms.sh
@@ -19,7 +19,24 @@  esac
 # We need access to CONFIG_ symbols
 . include/config/auto.conf
 
-ksym_wl=/dev/null
+needed_symbols=
+
+# Special case for modversions (see modpost.c)
+if [ -n "$CONFIG_MODVERSIONS" ]; then
+	needed_symbols="$needed_symbols module_layout"
+fi
+
+# With CONFIG_LTO_CLANG, LLVM bitcode has not yet been compiled into a binary
+# when the .mod files are generated, which means they don't yet contain
+# references to certain symbols that will be present in the final binaries.
+if [ -n "$CONFIG_LTO_CLANG" ]; then
+	# intrinsic functions
+	needed_symbols="$needed_symbols memcpy memmove memset"
+	# stack protector symbols
+	needed_symbols="$needed_symbols __stack_chk_fail __stack_chk_guard"
+fi
+
+ksym_wl=
 if [ -n "$CONFIG_UNUSED_KSYMS_WHITELIST" ]; then
 	# Use 'eval' to expand the whitelist path and check if it is relative
 	eval ksym_wl="$CONFIG_UNUSED_KSYMS_WHITELIST"
@@ -40,16 +57,14 @@  cat > "$output_file" << EOT
 EOT
 
 [ -f modules.order ] && modlist=modules.order || modlist=/dev/null
-sed 's/ko$/mod/' $modlist |
-xargs -n1 sed -n -e '2{s/ /\n/g;/^$/!p;}' -- |
-cat - "$ksym_wl" |
+
+{
+	sed 's/ko$/mod/' $modlist | xargs -n1 sed -n -e '2p'
+	echo "$needed_symbols"
+	[ -n "$ksym_wl" ] && cat "$ksym_wl"
+} | sed -e 's/ /\n/g' | sed -n -e '/^$/!p' |
 # Remove the dot prefix for ppc64; symbol names with a dot (.) hold entry
 # point addresses.
 sed -e 's/^\.//' |
 sort -u |
 sed -e 's/\(.*\)/#define __KSYM_\1 1/' >> "$output_file"
-
-# Special case for modversions (see modpost.c)
-if [ -n "$CONFIG_MODVERSIONS" ]; then
-	echo "#define __KSYM_module_layout 1" >> "$output_file"
-fi
diff --git a/scripts/lto-used-symbollist.txt b/scripts/lto-used-symbollist.txt
deleted file mode 100644
index 38e7bb9ebaae..000000000000
--- a/scripts/lto-used-symbollist.txt
+++ /dev/null
@@ -1,5 +0,0 @@ 
-memcpy
-memmove
-memset
-__stack_chk_fail
-__stack_chk_guard