| Message ID | 20210707224310.1403944-1-ndesaulniers@google.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | infer CROSS_COMPILE from ARCH for LLVM=1 LLVM_IAS=1 | expand |
On Wed, Jul 07, 2021 at 03:43:08PM -0700, Nick Desaulniers wrote: > We get constant feedback that the command line invocation of make is too > long. CROSS_COMPILE is helpful when a toolchain has a prefix of the > target triple, or is an absolute path outside of $PATH, but it's mostly > redundant for a given ARCH. > > Instead, let's infer it from ARCH, and move some flag handling into a > new file included from the top level Makefile. Why only for LLVM? I really hate the mess we currently have with ARCH and CROSS_COMPILE. Being able to set both in .config (and maybe even inferring CROSS_COMPILE where possible) would make my life so much easier.
On Thu, Jul 8, 2021 at 7:49 AM Christoph Hellwig <hch@infradead.org> wrote: > > On Wed, Jul 07, 2021 at 03:43:08PM -0700, Nick Desaulniers wrote: > > We get constant feedback that the command line invocation of make is too > > long. CROSS_COMPILE is helpful when a toolchain has a prefix of the > > target triple, or is an absolute path outside of $PATH, but it's mostly > > redundant for a given ARCH. > > > > Instead, let's infer it from ARCH, and move some flag handling into a > > new file included from the top level Makefile. > > Why only for LLVM? I really hate the mess we currently have with > ARCH and CROSS_COMPILE. Being able to set both in .config (and maybe > even inferring CROSS_COMPILE where possible) would make my life so > much easier. I agree this would be best, but solving the problem for llvm is trivial with a 1:1 mapping between ARCH= and --target= strings. Doing the same for gcc requires enumerating all possible target triples, and possibly deciding between different versions, e.g. when your path contains /usr/bin/powerpc64-linux-gnu-gcc-5.2.0 /usr/bin/powerpc64-linux-gnu-gcc -> powerpc64-linux-gnu-gcc-5.2.0 /usr/local/bin/ppc64le-linux-gcc-9 ~/bin/powerpc/powerpc-linux-unknown-gcc-12.0.20210708.experimental all of these should be able to cross-build any powerpc kernel, but there is no obvious first choice (highest version, first in path, ordered list of target triples, ...). I tried coming up with a heuristic to pick a reasonable toolchain, but at some point gave up because I failed to express that in a readable bash or Makefile syntax. Arnd
On Thu, Jul 8, 2021 at 12:27 AM Arnd Bergmann <arnd@kernel.org> wrote: > > On Thu, Jul 8, 2021 at 7:49 AM Christoph Hellwig <hch@infradead.org> wrote: > > > > On Wed, Jul 07, 2021 at 03:43:08PM -0700, Nick Desaulniers wrote: > > > We get constant feedback that the command line invocation of make is too > > > long. CROSS_COMPILE is helpful when a toolchain has a prefix of the > > > target triple, or is an absolute path outside of $PATH, but it's mostly > > > redundant for a given ARCH. > > > > > > Instead, let's infer it from ARCH, and move some flag handling into a > > > new file included from the top level Makefile. > > > > Why only for LLVM? I really hate the mess we currently have with > > ARCH and CROSS_COMPILE. I agree. > > Being able to set both in .config (and maybe > > even inferring CROSS_COMPILE where possible) would make my life so > > much easier. > > I agree this would be best, but solving the problem for llvm is > trivial with a 1:1 > mapping between ARCH= and --target= strings. Doing the same for gcc > requires enumerating all possible target triples, and possibly deciding between > different versions, e.g. when your path contains Yes. There is a seldom used Kbuild macro for this: cc-cross-prefix. It's hard coded to check for prefixed versions of gcc, but it will handle the enumeration. Perhaps it could be used more widely, once we know whether we're building with gcc or clang. But as you note, enumeration is kind of a waste of time for clang as these target triples are known ahead of time. I guess we could check that your build of clang was configured with support for that target ARCH (since you can disable backends when building clang), but I consider disabling backends an antipattern, and I think the user will figure out pretty quickly when they're trying to build a target that LLVM doesn't support (whether due to configured-off or unimplemented target). > /usr/bin/powerpc64-linux-gnu-gcc-5.2.0 > /usr/bin/powerpc64-linux-gnu-gcc -> powerpc64-linux-gnu-gcc-5.2.0 > /usr/local/bin/ppc64le-linux-gcc-9 > ~/bin/powerpc/powerpc-linux-unknown-gcc-12.0.20210708.experimental > > all of these should be able to cross-build any powerpc kernel, but > there is no obvious first choice (highest version, first in path, > ordered list of target triples, ...). I tried coming up with a heuristic > to pick a reasonable toolchain, but at some point gave up because > I failed to express that in a readable bash or Makefile syntax. Right; foremost in my mind was arm-linux-gnueabi-gcc vs arm-linux-gnueabihf-gcc. That's not even to mention the versioned suffixes. In terms of multiversion support; this series doesn't regress doing things the hard/verbose way. But I think for most users we can have a simpler common case; folks can play with their $PATH or focus on more hermetic builds if they want this new feature (CROSS_COMPILE inference) AND support for multiple versions of the same toolchain.
On Thu, Jul 8, 2021 at 8:04 PM 'Nick Desaulniers' via Clang Built Linux <clang-built-linux@googlegroups.com> wrote: > > /usr/bin/powerpc64-linux-gnu-gcc-5.2.0 > > /usr/bin/powerpc64-linux-gnu-gcc -> powerpc64-linux-gnu-gcc-5.2.0 > > /usr/local/bin/ppc64le-linux-gcc-9 > > ~/bin/powerpc/powerpc-linux-unknown-gcc-12.0.20210708.experimental > > > > all of these should be able to cross-build any powerpc kernel, but > > there is no obvious first choice (highest version, first in path, > > ordered list of target triples, ...). I tried coming up with a heuristic > > to pick a reasonable toolchain, but at some point gave up because > > I failed to express that in a readable bash or Makefile syntax. > > Right; foremost in my mind was arm-linux-gnueabi-gcc vs > arm-linux-gnueabihf-gcc. That's not even to mention the versioned > suffixes. > > In terms of multiversion support; this series doesn't regress doing > things the hard/verbose way. But I think for most users we can have a > simpler common case; folks can play with their $PATH or focus on more > hermetic builds if they want this new feature (CROSS_COMPILE > inference) AND support for multiple versions of the same toolchain. Fair enough. So how something like this: powerpc-targets := powerpc32 powerpc64 powerpc32le \ powerpc32be powerpc64le powerpc64be ppc64le ppc64be arm-targets := arm-linux-gnueabi arm-linux-gnueabihf x86-targets := x86_64 i386 i686 x86_64-targets := x86 i386-targets := i686 x86 x86_64 parisc-targets := hppa64 hppa ... CROSS_COMPILE ?= `find-toolchain $(ARCH) $($(ARCH)-targets)` where find-toolchain just finds the first working toolchain based, looking for $(target)-linux-gcc $(target)-gcc $(target)-unknown-linux-gcc etc in $(PATH) but ignoring the versions? What I had actually planned was a set of helpers that allow you to do this in multiple steps: - if $(objtree)/scripts/cross/bin/gcc (or something else we pick) exists and CROSS_COMPILE is not set, set CROSS_COMPILE to $(objtree)/scripts/cross/bin/ in the Makefile - add script to enumerate the installed toolchains - add a second script to symlink one of those toolchains to $(objtree)/scripts/cross/bin - add a third script to download a cross-toolchain from kernel.org for $(ARCH) and install it to one of the locations that the first script looks for (/opt/cross/, $(HOME)/cross/, $(objtree)scripts/cross/) Arnd
On Fri, Jul 9, 2021 at 1:07 AM Arnd Bergmann <arnd@kernel.org> wrote: > > On Thu, Jul 8, 2021 at 8:04 PM 'Nick Desaulniers' via Clang Built > Linux <clang-built-linux@googlegroups.com> wrote: > > > > /usr/bin/powerpc64-linux-gnu-gcc-5.2.0 > > > /usr/bin/powerpc64-linux-gnu-gcc -> powerpc64-linux-gnu-gcc-5.2.0 > > > /usr/local/bin/ppc64le-linux-gcc-9 > > > ~/bin/powerpc/powerpc-linux-unknown-gcc-12.0.20210708.experimental > > > > > > all of these should be able to cross-build any powerpc kernel, but > > > there is no obvious first choice (highest version, first in path, > > > ordered list of target triples, ...). I tried coming up with a heuristic > > > to pick a reasonable toolchain, but at some point gave up because > > > I failed to express that in a readable bash or Makefile syntax. > > > > Right; foremost in my mind was arm-linux-gnueabi-gcc vs > > arm-linux-gnueabihf-gcc. That's not even to mention the versioned > > suffixes. > > > > In terms of multiversion support; this series doesn't regress doing > > things the hard/verbose way. But I think for most users we can have a > > simpler common case; folks can play with their $PATH or focus on more > > hermetic builds if they want this new feature (CROSS_COMPILE > > inference) AND support for multiple versions of the same toolchain. > > Fair enough. So how something like this: > > powerpc-targets := powerpc32 powerpc64 powerpc32le \ > powerpc32be powerpc64le powerpc64be ppc64le ppc64be > arm-targets := arm-linux-gnueabi arm-linux-gnueabihf > x86-targets := x86_64 i386 i686 > x86_64-targets := x86 > i386-targets := i686 x86 x86_64 > parisc-targets := hppa64 hppa > ... > > CROSS_COMPILE ?= `find-toolchain $(ARCH) $($(ARCH)-targets)` > > where find-toolchain just finds the first working toolchain based, looking > for $(target)-linux-gcc $(target)-gcc $(target)-unknown-linux-gcc etc > in $(PATH) but ignoring the versions? Sure, debian doesn't even package different versions of the cross GCC packages AFAIK; no idea about other distros. Though the user may have built from source, or have multiple versions fetched from tarballs. I think we can simplify the common case of "I just wan't to cross compile, I don't necessarily care about an older compiler version co-installed with a newer one." ("and if I did, I could still use CROSS_COMPILE the verbose way"). > What I had actually planned was a set of helpers that allow you to > do this in multiple steps: > > - if $(objtree)/scripts/cross/bin/gcc (or something else we pick) > exists and CROSS_COMPILE is not set, set CROSS_COMPILE > to $(objtree)/scripts/cross/bin/ in the Makefile > - add script to enumerate the installed toolchains > - add a second script to symlink one of those toolchains to > $(objtree)/scripts/cross/bin (and check the symlink isn't broken should the user uninstall a toolchain, or have their distro update their toolchain version) > - add a third script to download a cross-toolchain from kernel.org > for $(ARCH) and install it to one of the locations that the first > script looks for (/opt/cross/, $(HOME)/cross/, $(objtree)scripts/cross/) Would the user be prompted for the download? So during `defconfig`/configuration we could prompt and say "it looks like you're cross compiling without setting CROSS_COMPILE, would you like me to fetch a cross compiler for you?" Seems reasonable, when cross compiling with GCC.
On Wed, Jul 14, 2021 at 8:09 PM 'Nick Desaulniers' via Clang Built Linux <clang-built-linux@googlegroups.com> wrote: > > On Fri, Jul 9, 2021 at 1:07 AM Arnd Bergmann <arnd@kernel.org> wrote: > > > > On Thu, Jul 8, 2021 at 8:04 PM 'Nick Desaulniers' via Clang Built > > Linux <clang-built-linux@googlegroups.com> wrote: > > > > > > /usr/bin/powerpc64-linux-gnu-gcc-5.2.0 > > > > /usr/bin/powerpc64-linux-gnu-gcc -> powerpc64-linux-gnu-gcc-5.2.0 > > > > /usr/local/bin/ppc64le-linux-gcc-9 > > > > ~/bin/powerpc/powerpc-linux-unknown-gcc-12.0.20210708.experimental > > > > > > > > all of these should be able to cross-build any powerpc kernel, but > > > > there is no obvious first choice (highest version, first in path, > > > > ordered list of target triples, ...). I tried coming up with a heuristic > > > > to pick a reasonable toolchain, but at some point gave up because > > > > I failed to express that in a readable bash or Makefile syntax. > > > > > > Right; foremost in my mind was arm-linux-gnueabi-gcc vs > > > arm-linux-gnueabihf-gcc. That's not even to mention the versioned > > > suffixes. > > > > > > In terms of multiversion support; this series doesn't regress doing > > > things the hard/verbose way. But I think for most users we can have a > > > simpler common case; folks can play with their $PATH or focus on more > > > hermetic builds if they want this new feature (CROSS_COMPILE > > > inference) AND support for multiple versions of the same toolchain. > > > > Fair enough. So how something like this: > > > > powerpc-targets := powerpc32 powerpc64 powerpc32le \ > > powerpc32be powerpc64le powerpc64be ppc64le ppc64be > > arm-targets := arm-linux-gnueabi arm-linux-gnueabihf > > x86-targets := x86_64 i386 i686 > > x86_64-targets := x86 > > i386-targets := i686 x86 x86_64 > > parisc-targets := hppa64 hppa > > ... > > > > CROSS_COMPILE ?= `find-toolchain $(ARCH) $($(ARCH)-targets)` > > > > where find-toolchain just finds the first working toolchain based, looking > > for $(target)-linux-gcc $(target)-gcc $(target)-unknown-linux-gcc etc > > in $(PATH) but ignoring the versions? > > Sure, debian doesn't even package different versions of the cross GCC > packages AFAIK; no idea about other distros. Though the user may have > built from source, or have multiple versions fetched from tarballs. I have an Ubuntu installation with gcc-9, gcc-10 and gcc-11 cross toolchains installed from through apt, but none that is just 'gcc' without a version as the ones I built myself are. > I think we can simplify the common case of "I just want to cross > compile, I don't necessarily care about an older compiler version > co-installed with a newer one." ("and if I did, I could still use > CROSS_COMPILE the verbose way"). Right, in my example above one would still have to set CC= since the detected target triple has no $(CROSS_COMPILE)gcc, only the versioned ones. Setting up the symlinks however should get you there. > > What I had actually planned was a set of helpers that allow you to > > do this in multiple steps: > > > > - if $(objtree)/scripts/cross/bin/gcc (or something else we pick) > > exists and CROSS_COMPILE is not set, set CROSS_COMPILE > > to $(objtree)/scripts/cross/bin/ in the Makefile > > - add script to enumerate the installed toolchains > > - add a second script to symlink one of those toolchains to > > $(objtree)/scripts/cross/bin > > (and check the symlink isn't broken should the user uninstall a > toolchain, or have their distro update their toolchain version) There are lots of options for the policy here. My preference would be to just pick the one from the symlink before any other, and then have a step 'make config' family of commands that checks that the selected toolchain actually produces object code for the selected architecture. > > - add a third script to download a cross-toolchain from kernel.org > > for $(ARCH) and install it to one of the locations that the first > > script looks for (/opt/cross/, $(HOME)/cross/, $(objtree)scripts/cross/) > > Would the user be prompted for the download? So during > `defconfig`/configuration we could prompt and say "it looks like > you're cross compiling without setting CROSS_COMPILE, would you like > me to fetch a cross compiler for you?" > > Seems reasonable, when cross compiling with GCC. I don't think we want interactive commands in the build system other than 'make config', but it would be reasonable to print something like "no compiler found", "selected compiler ${CC} does not match architecture ${ARCH}", or "selected compiler ${CC} does not work", followed by a list of suggested commands to configure or download a different toolchain. Arnd
Masahiro, Do you have further thoughts on this series? On Wed, Jul 7, 2021 at 3:43 PM Nick Desaulniers <ndesaulniers@google.com> wrote: > > We get constant feedback that the command line invocation of make is too > long. CROSS_COMPILE is helpful when a toolchain has a prefix of the > target triple, or is an absolute path outside of $PATH, but it's mostly > redundant for a given ARCH. > > Instead, let's infer it from ARCH, and move some flag handling into a > new file included from the top level Makefile. > > Nick Desaulniers (2): > Makefile: move initial clang flag handling into scripts/Makefile.clang > Makefile: drop CROSS_COMPILE for LLVM=1 LLVM_IAS=1 > > Documentation/kbuild/llvm.rst | 5 ++++ > MAINTAINERS | 1 + > Makefile | 15 +---------- > scripts/Makefile.clang | 48 +++++++++++++++++++++++++++++++++++ > 4 files changed, 55 insertions(+), 14 deletions(-) > create mode 100644 scripts/Makefile.clang > > > base-commit: a0e781a2a35a8dd4e6a38571998d59c6b0e32cd8 > -- > 2.32.0.93.g670b81a890-goog > -- Thanks, ~Nick Desaulniers