| Message ID | 508e3783d284fd2d3bd4840907ed0bdc20bc1b23.1727881164.git.ps@pks.im (mailing list archive) |
|---|---|
| State | Superseded |
| Headers | show |
| Series | Modernize the build system | expand |
Patrick Steinhardt <ps@pks.im> writes: > We're contemplating whether to eventually replace our build systems with > a build system that is easier to use. Add a comparison of build systems > to our technical documentation as a baseline for discussion. > > Signed-off-by: Patrick Steinhardt <ps@pks.im> > --- Thanks for starting this. > diff --git a/Documentation/technical/build-systems.txt b/Documentation/technical/build-systems.txt > new file mode 100644 > index 0000000000..8fac36ce1d > --- /dev/null > +++ b/Documentation/technical/build-systems.txt > @@ -0,0 +1,164 @@ > += Build Systems > + > +The build system is the primary way for both developers and system integrators > +to interact with the Git project. As such, we consider it important to pick a > +build system that fulfills our requirements while being easy to use and extend. Maybe it is me, but this sounds as if _we_ do not need "easy to use and extend", but "easy to use and extend" are extra afterthought consideration we are making for others, which is a bit disturbing. Let me try paraphrasing what I think you wanted to say As such, being easy to use and extend for those who are not directly developing Git itself is just as important as other requirements we have on any potential build system. which hopefully would sound more like we are not treating those who only build and tweak the product of our project as second-class citizens. > +This document outlines the different requirements that we have for the build > +system and then compares available build systems using these criteria. > + > +== Requirements > + > +The following subsections prevent a list of requirements that we have for any I think you meant "present". > +potential build system. Sections are sorted by decreasing priority, even though > +these priorities will naturally differ between users. > + > +=== Platform support > + > +The most important criterium is whether the build system supports all of the > +platforms that Git supports. The most important platforms include: Maybe it is just me, but when I hear "The most", I expect an exhaustive list (hence "include" that came later somewhat surprised me), and I consider these "the more important systems" (implying "than others"). > + - Linux > + - macOS > + - Windows > + - FreeBSD > + - OpenBSD > + - NetBSD > + > +The platforms which must be supported by the tool should be aligned with our > +[platform support policy](platform-support.txt). > +Auto-detection of the following items is considered to be important: > + > + - Check for the existence of headers. > + - Check for the existence of libraries. > + - Check for the existence of exectuables. > + - Check for the runtime behavior of specific functions. - Check for specific link order requirements when multiple libraries are involved. > +=== Ease of use > + > +The build system SHOULD be both easy to use and easy to extend. While this is > +naturally a subjective metric it is likely not controversial to say that some > +build systems are considerably harder to use than others. If we want to pull RFC-2119/BCP-14 with shouting "SHOULD", we probably should use something similar in the previous "Platform support" section. > +=== IDE support > +=== Out-of-tree builds Cross platform builds (e.g., building for arm on x86-64 host)? > +=== Rust support > + > +Many long-time Git contributors are nowadays in favor of adopting Rust as a > +second language next to C. The build system SHOULD thus support Rust such that > +we do not have to reopen the discussion once we decide to pick up Rust. What reasons do you have in mind that, without spelling this out, Rust will be left behind while C is fully supported? If the build system can keep track of dependencies by knowing foo.o is made from foo.c and turning foo.c into foo.o takes running cc, it can do the same for rustc. Do you mean include file dependencies and the like? If the reason why we say "Language X needs to be supported" is because it is not enough to be able to run a compiler on source files written in the language, but it is also necessary to know when to run the compiler (i.e., by dependency tracking), it is better to spell it out. In any case, instead of singling out Rust in the title, name the section "languate support", and give an enumeration of compilers languages, processors that we care about, just like we did for platforms. For exaple, we may not necessarily want to treat "C support, done as an afterthought of supporting C++" and "C support, done natively" as equivalents (of course, the latter is better). Shouldn't it also "support" asciidoc/asciidoctor/makeinfo for the documentation toolchain? Are there other things we use Makefile for in our current system that we are forgetting in this requirements section, like "running lint" or "running tests"? Thanks.
On Wed, Oct 2, 2024 at 11:16 AM Patrick Steinhardt <ps@pks.im> wrote: > We're contemplating whether to eventually replace our build systems with > a build system that is easier to use. Add a comparison of build systems > to our technical documentation as a baseline for discussion. > > Signed-off-by: Patrick Steinhardt <ps@pks.im> > --- > +=== Rust support > + > +Many long-time Git contributors are nowadays in favor of adopting Rust as a > +second language next to C. The build system SHOULD thus support Rust such that > +we do not have to reopen the discussion once we decide to pick up Rust. I've been watching the Rust discussion mostly from the sidelines, so it's possible that I've missed something, but I was more than a little surprised to see the claim that "Many long-time Git contributors [are] in favor of adopting Rust" since my impression of the discussions does not reflect that claim, nor could I find sufficient references to support such a strong statement. My understanding of the situation is rather different; I've seen both proponents and opponents. With regards to project regulars... * The most vocal proponent has been brian who gives the impression of being quite a fan of Rust, but hasn't provided particularly compelling reasons to adopt it, and some of his arguments are highly subjective (for instance, "Rust excels at multithreading"[1]; while it's true that Rust may make multithreading safer, it is highly subjective to say that it "excels" at it -- especially, for instance, for anyone familiar with Go's approach to the same subject). * The Google team is interested in having a Rust wrapper for core Git library[2], but that doesn't imply or require adopting Rust into the Git project itself. * Randall has severe misgivings[3] about Rust being introduced into the project due to its exclusionary nature with regards to the platform(s) he supports. * At least one well-established contributor has stated[4] that "Rust scares" him. (Possibly this was said somewhat in jest, but I suspect the feeling may be shared by more than a few long-time Git contributors who see Rust as an overly complex, complicated, and convoluted language but who haven't spoken up because they understand that the programming world in general is moving toward more modern languages such as Rust and Go.) * The only properly compelling arguments I've seen in favor of adopting Rust came from Elijah[5,6] who is always well-spoken, well-reasoned, thorough, and backs up his statements with plenty of factual evidence. Those aside, I don't recall seeing any other long-time Git contributors speak up in favor of adopting Rust. [1]: https://lore.kernel.org/git/ZrqJM-vmPaJbdHP2@tapette.crustytoothpaste.net/ [2]: https://lore.kernel.org/git/cover.1723054623.git.steadmon@google.com/ [3]: https://lore.kernel.org/git/053f01db0b79$0d885b30$28991190$@nexbridge.com/ [4]: https://lore.kernel.org/git/Zu2D%2Fb1ZJbTlC1ml@nand.local/ [5]: https://lore.kernel.org/git/CABPp-BFWsWCGogqQ=haMsS4OhOdSwc3frcAxa6soQR5ORTceOA@mail.gmail.com/ [6]: https://lore.kernel.org/git/CABPp-BFOmwV-xBtjvtenb6RFz9wx2VWVpTeho0k=D8wsCCVwqQ@mail.gmail.com/
On 10/2/24 2:14 PM, Junio C Hamano wrote: >> +=== Rust support >> + >> +Many long-time Git contributors are nowadays in favor of adopting Rust as a >> +second language next to C. The build system SHOULD thus support Rust such that >> +we do not have to reopen the discussion once we decide to pick up Rust. > > What reasons do you have in mind that, without spelling this out, > Rust will be left behind while C is fully supported? If the build > system can keep track of dependencies by knowing foo.o is made from > foo.c and turning foo.c into foo.o takes running cc, it can do the > same for rustc. Do you mean include file dependencies and the like? It's actually tremendously more complicated, speaking as someone who as reviewed and dabbled with support for rustc in meson. :) Rust can trivially compile a single source file into a single executable fairly easily, and it looks much like running a C compiler. Correctly invoking rustc in any more complicated project, such as but not limited to: - building an internal rust static library ("rlib", typically in the form of a crate) - linking to a C library - exporting a C library - cross compilation - proc-macros= quickly turns into a maze of options to ensure correct linkage, and you also have to pass special options to do name-mangling right down the entire dependency tree. Note also that the rust compiler driver has decided for its own reasons to *partially* not rely on the C compiler as a linker driver (it passes -nodefaultlibs for "ideological reasons"), so you have to detect the correct libraries that need to be referenced by a staticlib using rustc --print native-static-libs and then order matters and sanitizers are broken anyway. In short, you really need a build system for Rust, you cannot just run rustc, which means you need to make the decision about a build system if you ever end up using Rust. > If the reason why we say "Language X needs to be supported" is > because it is not enough to be able to run a compiler on source > files written in the language, but it is also necessary to know when > to run the compiler (i.e., by dependency tracking), it is better to > spell it out. Dependency tracking is tame stuff and just table stakes for build systems. :) > In any case, instead of singling out Rust in the title, name the > section "languate support", and give an enumeration of compilers > languages, processors that we care about, just like we did for > platforms. For exaple, we may not necessarily want to treat "C > support, done as an afterthought of supporting C++" and "C support, > done natively" as equivalents (of course, the latter is better). > > Shouldn't it also "support" asciidoc/asciidoctor/makeinfo for > the documentation toolchain? Running asciidoc is pretty simple as it doesn't involve fiddly domain-specific compiler internals. You just run a command -- the same command everywhere -- with an "infile" and "outfile" parameter and the asciidoc tool is essentially well behaved and does what you want it to do. makeinfo isn't really any different. > Are there other things we use Makefile for in our current system > that we are forgetting in this requirements section, like "running > lint" or "running tests"? > > Thanks.
On Wed, Oct 02, 2024 at 04:24:10PM -0400, Eric Sunshine wrote: > On Wed, Oct 2, 2024 at 11:16 AM Patrick Steinhardt <ps@pks.im> wrote: > > We're contemplating whether to eventually replace our build systems with > > a build system that is easier to use. Add a comparison of build systems > > to our technical documentation as a baseline for discussion. > > > > Signed-off-by: Patrick Steinhardt <ps@pks.im> > > --- > > +=== Rust support > > + > > +Many long-time Git contributors are nowadays in favor of adopting Rust as a > > +second language next to C. The build system SHOULD thus support Rust such that > > +we do not have to reopen the discussion once we decide to pick up Rust. > > I've been watching the Rust discussion mostly from the sidelines, so > it's possible that I've missed something, but I was more than a little > surprised to see the claim that "Many long-time Git contributors [are] > in favor of adopting Rust" since my impression of the discussions does > not reflect that claim, nor could I find sufficient references to > support such a strong statement. My understanding of the situation is > rather different; I've seen both proponents and opponents. With > regards to project regulars... I felt that during the Git Contributor's summit the room was in favor of adopting Rust in general. The discussion seemed to rather revolve around _how_ we start to adopt it, not _if_ we do so. I also cannot really remember anybody pushing back on adopting Rust. In any case, I'm happy to tone this down a bit, as the document is not here to make the case for Rust. I think that it is very likely that we will eventually start to adopt Rust, and because of that I tend to think that we should be prepared for such a future instead of having to reevaluate our build system because it cannot handle such a potential future. [snip] > * At least one well-established contributor has stated[4] that "Rust > scares" him. (Possibly this was said somewhat in jest, but I suspect > the feeling may be shared by more than a few long-time Git > contributors who see Rust as an overly complex, complicated, and > convoluted language but who haven't spoken up because they understand > that the programming world in general is moving toward more modern > languages such as Rust and Go.) Well, that was me :) I _am_ scared of the learning curve of Rust, but I also think that it puts enough on the table to be a worthwhile addition. So it was said half-jokingly. Patrick
On Wed, Oct 02, 2024 at 11:14:52AM -0700, Junio C Hamano wrote: > Patrick Steinhardt <ps@pks.im> writes: > > diff --git a/Documentation/technical/build-systems.txt b/Documentation/technical/build-systems.txt > > new file mode 100644 > > index 0000000000..8fac36ce1d > > --- /dev/null > > +++ b/Documentation/technical/build-systems.txt > > @@ -0,0 +1,164 @@ > > += Build Systems > > + > > +The build system is the primary way for both developers and system integrators > > +to interact with the Git project. As such, we consider it important to pick a > > +build system that fulfills our requirements while being easy to use and extend. > > Maybe it is me, but this sounds as if _we_ do not need "easy to use > and extend", but "easy to use and extend" are extra afterthought > consideration we are making for others, which is a bit disturbing. > > Let me try paraphrasing what I think you wanted to say > > As such, being easy to use and extend for those who are not > directly developing Git itself is just as important as other > requirements we have on any potential build system. > > which hopefully would sound more like we are not treating those who > only build and tweak the product of our project as second-class > citizens. Oh, that sounds way better, yes! > > +This document outlines the different requirements that we have for the build > > +system and then compares available build systems using these criteria. > > + > > +== Requirements > > + > > +The following subsections prevent a list of requirements that we have for any > > I think you meant "present". Yup. > > +potential build system. Sections are sorted by decreasing priority, even though > > +these priorities will naturally differ between users. > > + > > +=== Platform support > > + > > +The most important criterium is whether the build system supports all of the > > +platforms that Git supports. The most important platforms include: > > Maybe it is just me, but when I hear "The most", I expect an > exhaustive list (hence "include" that came later somewhat surprised > me), and I consider these "the more important systems" (implying > "than others"). Fair point. Ideally, I would be able to just point to our platform support policy, but that document stops short of defining what kind of guarantees we give for each platform. So what are our primary platforms? Based on the criteria we have it would be Windows, Ubuntu and macOS because we have CI runners for these, all of which are included. Providing support for the BSDs is nice, but these are likely rather "secondary" platforms. > > +=== Ease of use > > + > > +The build system SHOULD be both easy to use and easy to extend. While this is > > +naturally a subjective metric it is likely not controversial to say that some > > +build systems are considerably harder to use than others. > > If we want to pull RFC-2119/BCP-14 with shouting "SHOULD", we > probably should use something similar in the previous "Platform > support" section. I think I originally wanted to use RFC-2119, but then noticed that it's rather pointless. I'll stop shouting. > > +=== IDE support > > +=== Out-of-tree builds > > Cross platform builds (e.g., building for arm on x86-64 host)? Oh, yes, that one is indeed an ommission in the current doc. > > +=== Rust support > > + > > +Many long-time Git contributors are nowadays in favor of adopting Rust as a > > +second language next to C. The build system SHOULD thus support Rust such that > > +we do not have to reopen the discussion once we decide to pick up Rust. > > What reasons do you have in mind that, without spelling this out, > Rust will be left behind while C is fully supported? If the build > system can keep track of dependencies by knowing foo.o is made from > foo.c and turning foo.c into foo.o takes running cc, it can do the > same for rustc. Do you mean include file dependencies and the like? > If the reason why we say "Language X needs to be supported" is > because it is not enough to be able to run a compiler on source > files written in the language, but it is also necessary to know when > to run the compiler (i.e., by dependency tracking), it is better to > spell it out. Eli already did a good job of explaining this. While it should be possible to "retrofit" Rust support to basically all build systems out there, it certainly would be nice if we could hide away most of the intricate details of how Rust code needs to be compiled. It can get complex rather fast. > In any case, instead of singling out Rust in the title, name the > section "languate support", and give an enumeration of compilers > languages, processors that we care about, just like we did for > platforms. For exaple, we may not necessarily want to treat "C > support, done as an afterthought of supporting C++" and "C support, > done natively" as equivalents (of course, the latter is better). That makes sense to me, will do. > Shouldn't it also "support" asciidoc/asciidoctor/makeinfo for > the documentation toolchain? It certainly should be flexible enough to allow for this, yes. > Are there other things we use Makefile for in our current system > that we are forgetting in this requirements section, like "running > lint" or "running tests"? I was a bit torn on whether or not to add "running tests", mostly because it was supported by all of the proposed build systems. But that's probably not a good reason to not make this requirement explicit. Patrick
Patrick Steinhardt <ps@pks.im> writes: >> Are there other things we use Makefile for in our current system >> that we are forgetting in this requirements section, like "running >> lint" or "running tests"? > > I was a bit torn on whether or not to add "running tests", mostly > because it was supported by all of the proposed build systems. But > that's probably not a good reason to not make this requirement explicit. If a magic build system can omit tests that it can prove (by looking at dependency graph) that they will not be affected with the change I made to this single source file "builtin/cat-file.c", that would certainly be welcome. We can dream for build system that specifically cater to and know intricate details of a language. We should also be able to dream of such a thing ;-) Thanks.
diff --git a/Documentation/Makefile b/Documentation/Makefile index 0f55baa252..e23cffb5f9 100644 --- a/Documentation/Makefile +++ b/Documentation/Makefile @@ -111,6 +111,7 @@ TECH_DOCS += MyFirstObjectWalk TECH_DOCS += SubmittingPatches TECH_DOCS += ToolsForGit TECH_DOCS += technical/bitmap-format +TECH_DOCS += technical/build-systems TECH_DOCS += technical/bundle-uri TECH_DOCS += technical/hash-function-transition TECH_DOCS += technical/long-running-process-protocol diff --git a/Documentation/technical/build-systems.txt b/Documentation/technical/build-systems.txt new file mode 100644 index 0000000000..8fac36ce1d --- /dev/null +++ b/Documentation/technical/build-systems.txt @@ -0,0 +1,164 @@ += Build Systems + +The build system is the primary way for both developers and system integrators +to interact with the Git project. As such, we consider it important to pick a +build system that fulfills our requirements while being easy to use and extend. + +This document outlines the different requirements that we have for the build +system and then compares available build systems using these criteria. + +== Requirements + +The following subsections prevent a list of requirements that we have for any +potential build system. Sections are sorted by decreasing priority, even though +these priorities will naturally differ between users. + +=== Platform support + +The most important criterium is whether the build system supports all of the +platforms that Git supports. The most important platforms include: + + - Linux + - macOS + - Windows + - FreeBSD + - OpenBSD + - NetBSD + +The platforms which must be supported by the tool should be aligned with our +[platform support policy](platform-support.txt). + +=== Auto-detection of supported features + +The build system MUST support auto-detection of features which are or aren't +available on the current platform. Platform maintainers should not be required +to manually configure the complete build. + +Auto-detection of the following items is considered to be important: + + - Check for the existence of headers. + - Check for the existence of libraries. + - Check for the existence of exectuables. + - Check for the runtime behavior of specific functions. + +=== Ease of use + +The build system SHOULD be both easy to use and easy to extend. While this is +naturally a subjective metric it is likely not controversial to say that some +build systems are considerably harder to use than others. + +=== IDE support + +The build system SHOULD integrate with well-known IDEs. Well-known IDEs include: + + - Microsoft Visual Studio + - Visual Studio Code + - Xcode + +There are four levels of support: + + - Native integration into the IDE. + - Integration into the IDE via a plugin. + - Integration into the IDE via generating a project description with the build + system. + - No integration. + +Native integration is preferable, but integration via either a plugin or by +generating a project description via the build system are considered feasible +alternatives. + +Another important distinction is the level of integration. There are two +features that one generally wants to have: + + - Integration of build targets. + - Automatic setup of features like code completion with detected build + dependencies. + +The first bullet point is the bare minimum, but is not sufficient to be +considered proper integration. + +=== Out-of-tree builds + +The build system SHOULD support out-of-tree builds. Out-of-tree builds allow a +developer to configure multiple different build directories with different +configuration, e.g. one "debug" build and one "release" build. + +=== Rust support + +Many long-time Git contributors are nowadays in favor of adopting Rust as a +second language next to C. The build system SHOULD thus support Rust such that +we do not have to reopen the discussion once we decide to pick up Rust. + +== Comparison + +The following list of build systems are considered: + +- GNU Make +- autoconf +- CMake +- Meson + +=== GNU Make + +- Platform support: ubitquitous on all platforms, but not well-integrated into Windows. +- Auto-detection: no built-in support for auto-detection of features. +- Ease of use: easy to use, but discovering available options is hard. Makefile + rules can quickly get out of hand once reaching a certain scope. +- IDE support: execution of Makefile targets is supported by many IDEs +- Out-of-tree builds: supported in theory, not wired up in practice. +- Rust: can be added. + +=== autoconf + +- Platform support: ubiquitous on all platforms, but not well-integrated into Windows. +- Auto-detection: supported. +- Ease of use: easy to use, discovering available options is comparatively + easy. The autoconf syntax is prohibitively hard to extend though due to its + complex set of interacting files and the hard-to-understand M4 language. +- IDE support: no integration into IDEs at generation time. The generated + Makefiles have the same level of support as GNU Make. +- Out-of-tree builds: supported in theory, not wired up in practice. +- Rust: not supported. + +=== CMake + +- Platform support: not as extensive as GNU Make or autoconf, but all major + platforms are supported. + - AIX + - Cygwin + - FreeBSD + - Linux + - OpenBSD + - Solaris + - Windows + - macOS +- Ease of use: easy to use, discovering available options is not always + trivial. The scripting language used by CMake is somewhat cumbersome to use, + but extending CMake build instructions is doable. +- IDE support: natively integrated into Microsoft Visual Studio. Can generate + project descriptions for Xcode. An extension is available for Visual Studio + Code. Many other IDEs have plugins for CMake. +- Out-of-tree builds: supported. +- Rust: not supported. + +=== Meson + +- Platform: not as extensive as GNU Make or autoconf, but all major platforms + and some smaller ones are supported. + - AIX + - Cygwin + - DragonflyBSD + - FreeBSD + - Haiku + - Linux + - NetBSD + - OpenBSD + - Solaris + - Windows + - macOS +- Ease of use: easy to use, discovering available options is easy. The + scripting language is straight-forward to use. +- IDE support: Supports generating build instructions for Xcode and Microsoft + Visual Studio, a plugin exists for Visual Studio Code. +- Out-of-tree builds: supported. +- Rust: supported.
We're contemplating whether to eventually replace our build systems with a build system that is easier to use. Add a comparison of build systems to our technical documentation as a baseline for discussion. Signed-off-by: Patrick Steinhardt <ps@pks.im> --- Documentation/Makefile | 1 + Documentation/technical/build-systems.txt | 164 ++++++++++++++++++++++ 2 files changed, 165 insertions(+) create mode 100644 Documentation/technical/build-systems.txt