| Message ID | 20240727074526.1771247-5-masahiroy@kernel.org (mailing list archive) |
|---|---|
| State | New |
| Headers | show |
| Series | kbuild: cross-compile linux-headers package | expand |
On Sat, Jul 27, 2024 at 04:42:04PM +0900, Masahiro Yamada wrote: > There are known limitations: > > - GCC plugins > > It would possible to rebuild GCC plugins for the target architecture > by passing HOSTCXX=${CROSS_COMPILE}g++ with necessary packages > installed, but gcc on the installed system emits > "cc1: error: incompatible gcc/plugin versions". I did not find a > solution for this because 'gcc' on a foreign architecture is a > different compiler after all. Do you mean having a plugins as part of a distro package? Does anyone do this?
On Sat, Jul 27, 2024 at 04:42:04PM +0900, Masahiro Yamada wrote: > A long standing issue in the upstream kernel packaging is that the > linux-headers package is not cross-compiled. > > For example, you can cross-build Debian packages for arm64 by running > the following command: > > $ make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- bindeb-pkg > > However, the generated linux-headers-*_arm64.deb is useless because the > host programs in it were built for your build machine architecture > (likely x86), not arm64. > > The Debian kernel maintains its own Makefiles to cross-compile host > tools without relying on Kbuild. [1] > > Instead of adding such full custom Makefiles, this commit adds a small > piece of code to cross-compile host programs located under the scripts/ > directory. > > A straightforward solution is to pass HOSTCC=${CROSS_COMPILE}gcc, but it > would also cross-compile scripts/basic/fixdep, which needs to be native > to process the if_changed_dep macro. (This approach may work under some > circumstances; you can execute foreign architecture programs with the > help of binfmt_misc because Debian systems enable CONFIG_BINFMT_MISC, > but it would require installing QEMU and libc for that architecture.) > > A trick is to use the external module build (KBUILD_EXTMOD=), which > does not rebuild scripts/basic/fixdep. ${CC} needs to be able to link > userspace programs (CONFIG_CC_CAN_LINK=y). > > There are known limitations: > > - GCC plugins > > It would possible to rebuild GCC plugins for the target architecture > by passing HOSTCXX=${CROSS_COMPILE}g++ with necessary packages > installed, but gcc on the installed system emits > "cc1: error: incompatible gcc/plugin versions". I did not find a > solution for this because 'gcc' on a foreign architecture is a > different compiler after all. > > - objtool and resolve_btfids > > These are built by the tools build system. They are not covered by > the current solution. > > I only tested this with Debian, but it should work for other package > systems as well. > > [1]: https://salsa.debian.org/kernel-team/linux/-/blob/debian/6.9.9-1/debian/rules.real#L586 > > Signed-off-by: Masahiro Yamada <masahiroy@kernel.org> > --- > > scripts/package/install-extmod-build | 34 ++++++++++++++++++++++++++++ > 1 file changed, 34 insertions(+) > > diff --git a/scripts/package/install-extmod-build b/scripts/package/install-extmod-build > index cc335945dfbc..0b56d3d7b48f 100755 > --- a/scripts/package/install-extmod-build > +++ b/scripts/package/install-extmod-build > @@ -43,4 +43,38 @@ mkdir -p "${destdir}" > fi > } | tar -c -f - -T - | tar -xf - -C "${destdir}" > > +# When ${CC} and ${HOSTCC} differ, we are likely cross-compiling. Rebuild host > +# programs using ${CC}. This assumes CC=${CROSS_COMPILE}gcc, which is usually > +# the case for package building. It does not cross-compile when CC=clang. > +# > +# This caters to host programs that participate in Kbuild. objtool and > +# resolve_btfids are out of scope. Just for clarification: Why do you call both "out of scope" here? Because they're not being built by kbuild, or because they will never be needed for building oot kmods? > +if [ "${CC}" != "${HOSTCC}" ] && is_enabled CONFIG_CC_CAN_LINK; then > + echo "Rebuilding host programs with ${CC}..." > + > + cat <<-'EOF' > "${destdir}/Kbuild" > + subdir-y := scripts > + EOF > + > + # HOSTCXX is not overridden. The C++ compiler is used to build: > + # - scripts/kconfig/qconf, which is unneeded for external module builds > + # - GCC plugins, which will not work on the installed system even with > + # being rebuilt. > + # > + # Use the single-target build to avoid the modpost invocation, which > + # would overwrite Module.symvers. > + "${MAKE}" HOSTCC="${CC}" KBUILD_EXTMOD="${destdir}" scripts/ > + > + cat <<-'EOF' > "${destdir}/scripts/Kbuild" > + subdir-y := basic > + hostprogs-always-y := mod/modpost > + mod/modpost-objs := $(addprefix mod/, modpost.o file2alias.o sumversion.o symsearch.o) > + EOF > + > + # Run once again to rebuild scripts/basic/ and scripts/mod/modpost. > + "${MAKE}" HOSTCC="${CC}" KBUILD_EXTMOD="${destdir}" scripts/ > + > + rm -f "${destdir}/Kbuild" "${destdir}/scripts/Kbuild" > +fi > + > find "${destdir}" \( -name '.*.cmd' -o -name '*.o' \) -delete > -- > 2.43.0 > > Thanks for fixing this; looks good to me. Reviewed-by: Nicolas Schier <nicolas@fjasle.eu>
On Tue, Jul 30, 2024 at 10:03 AM Kees Cook <kees@kernel.org> wrote: > > On Sat, Jul 27, 2024 at 04:42:04PM +0900, Masahiro Yamada wrote: > > There are known limitations: > > > > - GCC plugins > > > > It would possible to rebuild GCC plugins for the target architecture > > by passing HOSTCXX=${CROSS_COMPILE}g++ with necessary packages > > installed, but gcc on the installed system emits > > "cc1: error: incompatible gcc/plugin versions". I did not find a > > solution for this because 'gcc' on a foreign architecture is a > > different compiler after all. > > Do you mean having a plugins as part of a distro package? Does anyone do > this? I think the use of GCC plugins is not so common in distributions, presumably due to its strong limitation. In my quick research, Debian, Ubuntu, Fedora disable CONFIG_GCC_PLUGINS. Arch Linux enables CONFIG_GCC_PLUGINS. > -- > Kees Cook >
On Thu, Aug 1, 2024 at 6:10 AM Nicolas Schier <nicolas@fjasle.eu> wrote: > > On Sat, Jul 27, 2024 at 04:42:04PM +0900, Masahiro Yamada wrote: > > A long standing issue in the upstream kernel packaging is that the > > linux-headers package is not cross-compiled. > > > > For example, you can cross-build Debian packages for arm64 by running > > the following command: > > > > $ make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- bindeb-pkg > > > > However, the generated linux-headers-*_arm64.deb is useless because the > > host programs in it were built for your build machine architecture > > (likely x86), not arm64. > > > > The Debian kernel maintains its own Makefiles to cross-compile host > > tools without relying on Kbuild. [1] > > > > Instead of adding such full custom Makefiles, this commit adds a small > > piece of code to cross-compile host programs located under the scripts/ > > directory. > > > > A straightforward solution is to pass HOSTCC=${CROSS_COMPILE}gcc, but it > > would also cross-compile scripts/basic/fixdep, which needs to be native > > to process the if_changed_dep macro. (This approach may work under some > > circumstances; you can execute foreign architecture programs with the > > help of binfmt_misc because Debian systems enable CONFIG_BINFMT_MISC, > > but it would require installing QEMU and libc for that architecture.) > > > > A trick is to use the external module build (KBUILD_EXTMOD=), which > > does not rebuild scripts/basic/fixdep. ${CC} needs to be able to link > > userspace programs (CONFIG_CC_CAN_LINK=y). > > > > There are known limitations: > > > > - GCC plugins > > > > It would possible to rebuild GCC plugins for the target architecture > > by passing HOSTCXX=${CROSS_COMPILE}g++ with necessary packages > > installed, but gcc on the installed system emits > > "cc1: error: incompatible gcc/plugin versions". I did not find a > > solution for this because 'gcc' on a foreign architecture is a > > different compiler after all. > > > > - objtool and resolve_btfids > > > > These are built by the tools build system. They are not covered by > > the current solution. > > > > I only tested this with Debian, but it should work for other package > > systems as well. > > > > [1]: https://salsa.debian.org/kernel-team/linux/-/blob/debian/6.9.9-1/debian/rules.real#L586 > > > > Signed-off-by: Masahiro Yamada <masahiroy@kernel.org> > > --- > > > > scripts/package/install-extmod-build | 34 ++++++++++++++++++++++++++++ > > 1 file changed, 34 insertions(+) > > > > diff --git a/scripts/package/install-extmod-build b/scripts/package/install-extmod-build > > index cc335945dfbc..0b56d3d7b48f 100755 > > --- a/scripts/package/install-extmod-build > > +++ b/scripts/package/install-extmod-build > > @@ -43,4 +43,38 @@ mkdir -p "${destdir}" > > fi > > } | tar -c -f - -T - | tar -xf - -C "${destdir}" > > > > +# When ${CC} and ${HOSTCC} differ, we are likely cross-compiling. Rebuild host > > +# programs using ${CC}. This assumes CC=${CROSS_COMPILE}gcc, which is usually > > +# the case for package building. It does not cross-compile when CC=clang. > > +# > > +# This caters to host programs that participate in Kbuild. objtool and > > +# resolve_btfids are out of scope. > > Just for clarification: Why do you call both "out of scope" here? > Because they're not being built by kbuild, or because they will never be > needed for building oot kmods? I meant the former. Debian applies a tricky patch to the tools build system in order to cross-compile objtool: https://salsa.debian.org/kernel-team/linux/-/blob/debian/6.9.9-1/debian/patches/debian/fixdep-allow-overriding-hostcc-and-hostld.patch It is not an elegant solution, though. I still believe the right thing to do is converting Makefiles for objtool and resolve_btfids into Kbuild style. objtool and resolve_btfids are necessary for building external modules, when CONFIG_OBJTOOL=y and CONFIG_DEBUG_INFO_BTF=y, respectively. If these comments are confusing, I can delete them. > > > +if [ "${CC}" != "${HOSTCC}" ] && is_enabled CONFIG_CC_CAN_LINK; then > > + echo "Rebuilding host programs with ${CC}..." > > + > > + cat <<-'EOF' > "${destdir}/Kbuild" > > + subdir-y := scripts > > + EOF > > + > > + # HOSTCXX is not overridden. The C++ compiler is used to build: > > + # - scripts/kconfig/qconf, which is unneeded for external module builds > > + # - GCC plugins, which will not work on the installed system even with > > + # being rebuilt. > > + # > > + # Use the single-target build to avoid the modpost invocation, which > > + # would overwrite Module.symvers. > > + "${MAKE}" HOSTCC="${CC}" KBUILD_EXTMOD="${destdir}" scripts/ > > + > > + cat <<-'EOF' > "${destdir}/scripts/Kbuild" > > + subdir-y := basic > > + hostprogs-always-y := mod/modpost > > + mod/modpost-objs := $(addprefix mod/, modpost.o file2alias.o sumversion.o symsearch.o) > > + EOF > > + > > + # Run once again to rebuild scripts/basic/ and scripts/mod/modpost. > > + "${MAKE}" HOSTCC="${CC}" KBUILD_EXTMOD="${destdir}" scripts/ > > + > > + rm -f "${destdir}/Kbuild" "${destdir}/scripts/Kbuild" > > +fi > > + > > find "${destdir}" \( -name '.*.cmd' -o -name '*.o' \) -delete > > -- > > 2.43.0 > > > > > > Thanks for fixing this; looks good to me. > > Reviewed-by: Nicolas Schier <nicolas@fjasle.eu> >
On Thu, Aug 01, 2024 at 11:37:30AM +0900, Masahiro Yamada wrote: > On Thu, Aug 1, 2024 at 6:10 AM Nicolas Schier <nicolas@fjasle.eu> wrote: > > > > On Sat, Jul 27, 2024 at 04:42:04PM +0900, Masahiro Yamada wrote: > > > A long standing issue in the upstream kernel packaging is that the > > > linux-headers package is not cross-compiled. > > > > > > For example, you can cross-build Debian packages for arm64 by running > > > the following command: > > > > > > $ make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- bindeb-pkg > > > > > > However, the generated linux-headers-*_arm64.deb is useless because the > > > host programs in it were built for your build machine architecture > > > (likely x86), not arm64. > > > > > > The Debian kernel maintains its own Makefiles to cross-compile host > > > tools without relying on Kbuild. [1] > > > > > > Instead of adding such full custom Makefiles, this commit adds a small > > > piece of code to cross-compile host programs located under the scripts/ > > > directory. > > > > > > A straightforward solution is to pass HOSTCC=${CROSS_COMPILE}gcc, but it > > > would also cross-compile scripts/basic/fixdep, which needs to be native > > > to process the if_changed_dep macro. (This approach may work under some > > > circumstances; you can execute foreign architecture programs with the > > > help of binfmt_misc because Debian systems enable CONFIG_BINFMT_MISC, > > > but it would require installing QEMU and libc for that architecture.) > > > > > > A trick is to use the external module build (KBUILD_EXTMOD=), which > > > does not rebuild scripts/basic/fixdep. ${CC} needs to be able to link > > > userspace programs (CONFIG_CC_CAN_LINK=y). > > > > > > There are known limitations: > > > > > > - GCC plugins > > > > > > It would possible to rebuild GCC plugins for the target architecture > > > by passing HOSTCXX=${CROSS_COMPILE}g++ with necessary packages > > > installed, but gcc on the installed system emits > > > "cc1: error: incompatible gcc/plugin versions". I did not find a > > > solution for this because 'gcc' on a foreign architecture is a > > > different compiler after all. > > > > > > - objtool and resolve_btfids > > > > > > These are built by the tools build system. They are not covered by > > > the current solution. > > > > > > I only tested this with Debian, but it should work for other package > > > systems as well. > > > > > > [1]: https://salsa.debian.org/kernel-team/linux/-/blob/debian/6.9.9-1/debian/rules.real#L586 > > > > > > Signed-off-by: Masahiro Yamada <masahiroy@kernel.org> > > > --- > > > > > > scripts/package/install-extmod-build | 34 ++++++++++++++++++++++++++++ > > > 1 file changed, 34 insertions(+) > > > > > > diff --git a/scripts/package/install-extmod-build b/scripts/package/install-extmod-build > > > index cc335945dfbc..0b56d3d7b48f 100755 > > > --- a/scripts/package/install-extmod-build > > > +++ b/scripts/package/install-extmod-build > > > @@ -43,4 +43,38 @@ mkdir -p "${destdir}" > > > fi > > > } | tar -c -f - -T - | tar -xf - -C "${destdir}" > > > > > > +# When ${CC} and ${HOSTCC} differ, we are likely cross-compiling. Rebuild host > > > +# programs using ${CC}. This assumes CC=${CROSS_COMPILE}gcc, which is usually > > > +# the case for package building. It does not cross-compile when CC=clang. > > > +# > > > +# This caters to host programs that participate in Kbuild. objtool and > > > +# resolve_btfids are out of scope. > > > > Just for clarification: Why do you call both "out of scope" here? > > Because they're not being built by kbuild, or because they will never be > > needed for building oot kmods? > > > I meant the former. > > > Debian applies a tricky patch to the tools build system > in order to cross-compile objtool: > > https://salsa.debian.org/kernel-team/linux/-/blob/debian/6.9.9-1/debian/patches/debian/fixdep-allow-overriding-hostcc-and-hostld.patch > > It is not an elegant solution, though. > > > I still believe the right thing to do is > converting Makefiles for objtool and resolve_btfids into Kbuild style. > > > objtool and resolve_btfids are necessary for building external modules, > when CONFIG_OBJTOOL=y and CONFIG_DEBUG_INFO_BTF=y, respectively. > If these comments are confusing, I can delete them. I think it's good to mention that cross-built linux-headers package is still broken for CONFIG_OBJTOOL=y and CONFIG_DEBUG_INFO_BTF=y. I think I'd add a sentence to the commit message and keep the comment here as it is. Kind regards, Nicolas
diff --git a/scripts/package/install-extmod-build b/scripts/package/install-extmod-build index cc335945dfbc..0b56d3d7b48f 100755 --- a/scripts/package/install-extmod-build +++ b/scripts/package/install-extmod-build @@ -43,4 +43,38 @@ mkdir -p "${destdir}" fi } | tar -c -f - -T - | tar -xf - -C "${destdir}" +# When ${CC} and ${HOSTCC} differ, we are likely cross-compiling. Rebuild host +# programs using ${CC}. This assumes CC=${CROSS_COMPILE}gcc, which is usually +# the case for package building. It does not cross-compile when CC=clang. +# +# This caters to host programs that participate in Kbuild. objtool and +# resolve_btfids are out of scope. +if [ "${CC}" != "${HOSTCC}" ] && is_enabled CONFIG_CC_CAN_LINK; then + echo "Rebuilding host programs with ${CC}..." + + cat <<-'EOF' > "${destdir}/Kbuild" + subdir-y := scripts + EOF + + # HOSTCXX is not overridden. The C++ compiler is used to build: + # - scripts/kconfig/qconf, which is unneeded for external module builds + # - GCC plugins, which will not work on the installed system even with + # being rebuilt. + # + # Use the single-target build to avoid the modpost invocation, which + # would overwrite Module.symvers. + "${MAKE}" HOSTCC="${CC}" KBUILD_EXTMOD="${destdir}" scripts/ + + cat <<-'EOF' > "${destdir}/scripts/Kbuild" + subdir-y := basic + hostprogs-always-y := mod/modpost + mod/modpost-objs := $(addprefix mod/, modpost.o file2alias.o sumversion.o symsearch.o) + EOF + + # Run once again to rebuild scripts/basic/ and scripts/mod/modpost. + "${MAKE}" HOSTCC="${CC}" KBUILD_EXTMOD="${destdir}" scripts/ + + rm -f "${destdir}/Kbuild" "${destdir}/scripts/Kbuild" +fi + find "${destdir}" \( -name '.*.cmd' -o -name '*.o' \) -delete
A long standing issue in the upstream kernel packaging is that the linux-headers package is not cross-compiled. For example, you can cross-build Debian packages for arm64 by running the following command: $ make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- bindeb-pkg However, the generated linux-headers-*_arm64.deb is useless because the host programs in it were built for your build machine architecture (likely x86), not arm64. The Debian kernel maintains its own Makefiles to cross-compile host tools without relying on Kbuild. [1] Instead of adding such full custom Makefiles, this commit adds a small piece of code to cross-compile host programs located under the scripts/ directory. A straightforward solution is to pass HOSTCC=${CROSS_COMPILE}gcc, but it would also cross-compile scripts/basic/fixdep, which needs to be native to process the if_changed_dep macro. (This approach may work under some circumstances; you can execute foreign architecture programs with the help of binfmt_misc because Debian systems enable CONFIG_BINFMT_MISC, but it would require installing QEMU and libc for that architecture.) A trick is to use the external module build (KBUILD_EXTMOD=), which does not rebuild scripts/basic/fixdep. ${CC} needs to be able to link userspace programs (CONFIG_CC_CAN_LINK=y). There are known limitations: - GCC plugins It would possible to rebuild GCC plugins for the target architecture by passing HOSTCXX=${CROSS_COMPILE}g++ with necessary packages installed, but gcc on the installed system emits "cc1: error: incompatible gcc/plugin versions". I did not find a solution for this because 'gcc' on a foreign architecture is a different compiler after all. - objtool and resolve_btfids These are built by the tools build system. They are not covered by the current solution. I only tested this with Debian, but it should work for other package systems as well. [1]: https://salsa.debian.org/kernel-team/linux/-/blob/debian/6.9.9-1/debian/rules.real#L586 Signed-off-by: Masahiro Yamada <masahiroy@kernel.org> --- scripts/package/install-extmod-build | 34 ++++++++++++++++++++++++++++ 1 file changed, 34 insertions(+)