mbox series

[RFC,v6,00/19] Modernize the build system

Message ID 20241112-pks-meson-v6-0-648b30996827@pks.im (mailing list archive)
Headers show
Series Modernize the build system | expand

Message

Patrick Steinhardt Nov. 12, 2024, 5:02 p.m. UTC
Hi,

this patch series modernizes our build infrasturcture. It refactors
various parts of it to make it possible to perform out-of-tree builds in
theory.

The series then continues to wire up Meson as a proposed alternative as
a modern replacement for our current build systems. It provides better
integration into IDEs than our Makefiles, better usability than our
Makefiles and CMake, better configuration and discoverability thereof
than autoconf and CMake (which is of course my personal opinion).

The intent is to have Meson replace autoconf, which is mostly unused and
broken. Eventually, the plan would to also replace CMake and our
Makefiles in case we figure out that this replacement is indeed a net
benefit for the project, but all of this would happen over the course of
multiple releases.

Changes in v6:

  - Add documentation for how to use Meson.

  - Fix compilation in case libintl exists, but required tools like
    msgfmt and related don't. Instead of failing, we now automatically
    compile with the equivalent of NO_GETTEXT=YesPlease.

  - Update last patch that fixes semantic merge conflics with in-flight
    topics in "seen".

  - I've changed the series to be based on "master" at b31fb630c0 (Merge
    https://github.com/j6t/git-gui, 2024-11-11) as all prereqs of this
    series have since landed. This doesn't make any difference though
    as there aren't any merge conflicts or additional changes caused by
    this. I mostly mention this so that there is no confusion around the
    changed base commit.

As before, the tip of this patch series catches up with what is
happening in "seen". If you want to test it without topics in "seen" you
have to revert it.

Links to previous versions:

  - Link to v1: https://lore.kernel.org/r/cover.1727881164.git.ps@pks.im
  - Link to v2: https://lore.kernel.org/r/cover.1728485139.git.ps@pks.im
  - Link to v3: https://lore.kernel.org/r/cover.1729254070.git.ps@pks.im
  - Link to v4: https://lore.kernel.org/r/cover.1729771605.git.ps@pks.im
  - Link to v5: https://lore.kernel.org/r/cover.1731335938.git.ps@pks.im

Thanks!

Patrick

To: git@vger.kernel.org
Cc: Eli Schwartz <eschwartz@gentoo.org>
Cc: Eric Sunshine <sunshine@sunshineco.com>
Cc: Phillip Wood <phillip.wood123@gmail.com>
Cc: Junio C Hamano <gitster@pobox.com>
Cc: Ramsay Jones <ramsay@ramsayjones.plus.com>
Cc: Taylor Blau <me@ttaylorr.com>
Cc: David Aguilar <davvid@gmail.com>
Cc: Jeff King <peff@peff.net>

Patrick Steinhardt (19):
      Makefile: use common template for GIT-BUILD-OPTIONS
      Makefile: consistently use @PLACEHOLDER@ to substitute
      Makefile: consistently use PERL_PATH
      Makefile: extract script to massage Perl scripts
      Makefile: use "generate-perl.sh" to massage Perl library
      Makefile: extract script to massage Shell scripts
      Makefile: extract script to generate gitweb.cgi
      Makefile: refactor GIT-VERSION-GEN to be reusable
      Makefile: refactor generators to be PWD-independent
      Makefile: allow "bin-wrappers/" directory to exist
      Makefile: simplify building of templates
      Documentation: allow sourcing generated includes from separate dir
      Documentation: teach "cmd-list.perl" about out-of-tree builds
      Documentation: extract script to generate a list of mergetools
      t: better support for out-of-tree builds
      t: allow overriding build dir
      Documentation: add comparison of build systems
      Introduce support for the Meson build system
      meson: fix conflicts with in-flight topics

 .gitignore                                         |    1 -
 Documentation/CodingGuidelines                     |    2 +-
 Documentation/Makefile                             |   28 +-
 Documentation/build-docdep.perl                    |    2 +
 Documentation/cmd-list.perl                        |   23 +-
 Documentation/config/diff.txt                      |    2 +-
 Documentation/config/merge.txt                     |    2 +-
 Documentation/generate-mergetool-list.sh           |   17 +
 Documentation/git.txt                              |   24 +-
 Documentation/meson.build                          |  317 ++++
 Documentation/technical/build-systems.txt          |  224 +++
 GIT-BUILD-OPTIONS.in                               |   47 +
 GIT-VERSION-GEN                                    |   12 +-
 Makefile                                           |  209 +--
 bin-wrappers/.gitignore                            |    9 +
 bin-wrappers/meson.build                           |   28 +
 bin-wrappers/wrap-for-bin.sh                       |   37 +
 configure.ac                                       |    2 +-
 contrib/buildsystems/CMakeLists.txt                |  230 +--
 contrib/completion/meson.build                     |    8 +
 contrib/meson.build                                |    1 +
 generate-cmdlist.sh                                |   42 +-
 generate-configlist.sh                             |   20 +-
 generate-hooklist.sh                               |   15 +-
 generate-perl.sh                                   |   36 +
 generate-script.sh                                 |   34 +
 git-cvsserver.perl                                 |    2 +-
 git-instaweb.sh                                    |    8 +-
 git-request-pull.sh                                |    2 +-
 git-send-email.perl                                |    2 +-
 git-sh-i18n.sh                                     |    6 +-
 git-sh-setup.sh                                    |    6 +-
 git-svn.perl                                       |    2 +-
 gitk-git/po/vi.po                                  |    2 +-
 gitweb/GITWEB-BUILD-OPTIONS.in                     |   25 +
 gitweb/Makefile                                    |   58 +-
 gitweb/generate-gitweb.sh                          |   45 +
 gitweb/gitweb.perl                                 |   44 +-
 gitweb/meson.build                                 |   63 +
 meson.build                                        | 1780 ++++++++++++++++++++
 meson_options.txt                                  |   73 +
 perl/FromCPAN/Mail/meson.build                     |    7 +
 perl/FromCPAN/meson.build                          |    9 +
 perl/Git/I18N.pm                                   |    6 +-
 perl/Git/LoadCPAN.pm                               |    6 +-
 perl/Git/LoadCPAN/Mail/meson.build                 |    7 +
 perl/Git/LoadCPAN/meson.build                      |    9 +
 perl/Git/SVN/Memoize/meson.build                   |    7 +
 perl/Git/SVN/meson.build                           |   20 +
 perl/Git/meson.build                               |   18 +
 perl/header_templates/fixed_prefix.template.pl     |    2 +-
 perl/header_templates/runtime_prefix.template.pl   |    8 +-
 perl/meson.build                                   |   12 +
 po/meson.build                                     |   27 +
 subprojects/.gitignore                             |    1 +
 subprojects/curl.wrap                              |   13 +
 subprojects/expat.wrap                             |   13 +
 subprojects/openssl.wrap                           |   15 +
 subprojects/pcre2.wrap                             |   16 +
 subprojects/zlib.wrap                              |   13 +
 t/helper/meson.build                               |   93 +
 t/lib-gettext.sh                                   |    4 +-
 t/meson.build                                      | 1107 ++++++++++++
 t/t7609-mergetool--lib.sh                          |    2 +-
 t/test-lib.sh                                      |   22 +-
 templates/Makefile                                 |   39 +-
 templates/branches--                               |    1 -
 templates/{this--description => description}       |    0
 .../applypatch-msg.sample}                         |    0
 .../commit-msg.sample}                             |    0
 .../fsmonitor-watchman.sample}                     |    0
 templates/hooks/meson.build                        |   24 +
 .../post-update.sample}                            |    0
 .../pre-applypatch.sample}                         |    0
 .../pre-commit.sample}                             |    0
 .../pre-merge-commit.sample}                       |    0
 .../pre-push.sample}                               |    0
 .../pre-rebase.sample}                             |    0
 .../pre-receive.sample}                            |    0
 .../prepare-commit-msg.sample}                     |    0
 .../push-to-checkout.sample}                       |    0
 .../sendemail-validate.sample}                     |    0
 .../{hooks--update.sample => hooks/update.sample}  |    0
 templates/{info--exclude => info/exclude}          |    0
 templates/info/meson.build                         |    5 +
 templates/meson.build                              |   13 +
 unimplemented.sh                                   |    2 +-
 wrap-for-bin.sh                                    |   36 -
 88 files changed, 4600 insertions(+), 447 deletions(-)

Range-diff versus v5:

 1:  ed28e754e8 =  1:  687ae57464 Makefile: use common template for GIT-BUILD-OPTIONS
 2:  178ba0e5d6 =  2:  53d5f0103a Makefile: consistently use @PLACEHOLDER@ to substitute
 3:  66b82c152b =  3:  15a4864450 Makefile: consistently use PERL_PATH
 4:  8382a6b840 =  4:  ed09cc6027 Makefile: extract script to massage Perl scripts
 5:  51491b778d =  5:  56caa7a075 Makefile: use "generate-perl.sh" to massage Perl library
 6:  c3946a69db =  6:  ee2135d178 Makefile: extract script to massage Shell scripts
 7:  a58c901466 =  7:  13fbeee749 Makefile: extract script to generate gitweb.cgi
 8:  a9b886b27a =  8:  f0073e58c8 Makefile: refactor GIT-VERSION-GEN to be reusable
 9:  f2eccc7d65 =  9:  fd412ba3a0 Makefile: refactor generators to be PWD-independent
10:  c891e56e32 = 10:  2e3833a0a3 Makefile: allow "bin-wrappers/" directory to exist
11:  4e9525f64e = 11:  7d93c8afa0 Makefile: simplify building of templates
12:  b209ab75b5 = 12:  8dad1314b2 Documentation: allow sourcing generated includes from separate dir
13:  5e8be2473c = 13:  90d6e0b4eb Documentation: teach "cmd-list.perl" about out-of-tree builds
14:  621d18d281 = 14:  0911b47694 Documentation: extract script to generate a list of mergetools
15:  f56eb0cd86 = 15:  3861eab108 t: better support for out-of-tree builds
16:  9618c7745e = 16:  2226f932d3 t: allow overriding build dir
17:  568287b16a = 17:  a9686e2940 Documentation: add comparison of build systems
18:  42bc858172 ! 18:  a7a7f7efe3 Introduce support for the Meson build system
    @@ Commit message
             symbolic links pointing to the former. This rule is consistent in
             itself and thus easier to reason about.
     
    -      - We do not install dashed binaries into `$libexec/git-core` anymore.
    -        So there won't e.g. be a symlink for git-add(1). These are not
    +      - We do not install dashed binaries into `$libexec/git-core` anymore,
    +        so there won't e.g. be a symlink for git-add(1). These are not
             required by modern Git and there isn't really much of a use case for
             those anymore. By not installing those symlinks we thus start the
             deprecation of this layout.
     
    -      - Documentation does not yet exist. Same here, it will follow if the
    -        project can agree on Meson.
    -
           - We're targeting Meson 1.3.0, which has been released relatively
             recently November 2023. The only feature we use from that version is
             `fs.relative_to()`, which we could replace if necessary. If so, we
    @@ gitweb/meson.build (new)
     
      ## meson.build (new) ##
     @@
    ++# Meson build system
    ++# ==================
    ++#
    ++# The Meson build system is an alternative to our Makefile that you can use to
    ++# build, test and install Git. Using Meson results in a couple of benefits:
    ++#
    ++#  - Out-of-tree builds.
    ++#  - Better integration into IDEs.
    ++#  - Easy-to-use autoconfiguration of available features on your system.
    ++#
    ++# To use Meson from the command line you need to have both Meson and Ninja
    ++# installed. Alternatively, if you do not have Python available on your system,
    ++# you can also use Muon instead of Meson and Samurai instead of Ninja, both of
    ++# which are drop-ins replacement that only depend on C.
    ++#
    ++# Basic usage
    ++# ===========
    ++#
    ++# In the most trivial case, you can configure, build and install Git like this:
    ++#
    ++#  1. Set up the build directory. This only needs to happen once per build
    ++#     directory you want to have. You can also configure multiple different
    ++#     build directories with different configurations.
    ++#
    ++#      $ meson setup build/
    ++#
    ++#     The build directory gets ignored by Git automatically as Meson will write
    ++#     a ".gitignore" file into it. From hereon, we will assume that you execute
    ++#     commands inside this build directory.
    ++#
    ++# 2. Compile Git. You can either use Meson, Ninja or Samurai to do this, so all
    ++#    of the following invocations are equivalent:
    ++#
    ++#      $ meson compile
    ++#      $ ninja
    ++#      $ samu
    ++#
    ++#   The different invocations should ultimately not make much of a difference.
    ++#   Using Meson also works with other generators though, like when the build
    ++#   directory has been set up for use with Microsoft Visual Studio.
    ++#
    ++# 3. Execute tests. Again, you can either use Meson, Ninja or Samurai to do this:
    ++#
    ++#      $ meson test
    ++#      $ ninja test
    ++#      $ samu test
    ++#
    ++#   It is recommended to use Meson in this case though as it also provides you
    ++#   additional features that the other build systems don't have available.
    ++#   You can e.g. pass additional arguments to the test executables or run
    ++#   individual tests:
    ++#
    ++#      # Execute the t0000-basic integration test and t-reftable-stack unit test.
    ++#      $ meson test t0000-basic t-reftable-stack
    ++#
    ++#      # Execute all reftable unit tests.
    ++#      $ meson test t-reftable-*
    ++#
    ++#      # Execute all tests and stop with the first failure.
    ++#      $ meson test --maxfail 1
    ++#
    ++#      # Execute single test interactively such that features like `debug ()` work.
    ++#      $ meson test -i --test-args='-ix' t1400-update-ref
    ++#
    ++# 4. Install the Git distribution. Again, this can be done via Meson, Ninja or
    ++#    Samurai:
    ++#
    ++#      $ meson install
    ++#      $ ninja install
    ++#      $ samu install
    ++#
    ++#    The prefix into which Git shall be installed is defined when setting up
    ++#    the build directory. More on that in the "Configuration" section.
    ++#
    ++# Meson supports multiple backends. The default backend generates Ninja build
    ++# instructions, but it also supports the generation of Microsoft Visual
    ++# Studio solutions as well as Xcode projects. IDEs like Eclipse and Visual
    ++# Studio Code provide plugins to import Meson files directly.
    ++#
    ++# Configuration
    ++# =============
    ++#
    ++# The exact configuration of Git is determined when setting up the Git
    ++# directory. Unless told otherwise, Meson will automatically detect the
    ++# availability of various bits and pieces. There are two different kinds of
    ++# options that can be used to further tweak the build:
    ++#
    ++#   - Built-in options provided by Meson.
    ++#
    ++#   - Options defined by the project in the "meson_options.txt" file.
    ++#
    ++# Both kinds of options can be inspected by running `meson configure` in the
    ++# build directory, which will give you a list of the current value for all
    ++# options.
    ++#
    ++# Options can be configured either when setting up the build directory or can
    ++# be changed in preexisting build directories:
    ++#
    ++#      # Set up a build directory with optimized settings that will be
    ++#      # installed into an alternative prefix.
    ++#      $ meson setup --buildtype release --optimization 3 --strip --prefix=/home/$USER
    ++#
    ++#      # Set up a build directory with 'address' and 'undefined' sanitizers
    ++#      # using Clang.
    ++#      $ CC=clang meson setup -Db_sanitize=address,undefined
    ++#
    ++#      # Disable tests in a preexisting build directory.
    ++#      $ meson configure -Dtests=false
    ++#
    ++#      # Disable features based on Python
    ++#      $ meson configure -Dpython=disabled
    ++#
    ++#      # Disable features based on Python
    ++#      $ meson configure -Dpython=disabled
    ++#
    ++# Options have a type like booleans, choices, strings or features. Features are
    ++# somewhat special as they can have one of three values: enabled, disabled or
    ++# auto. While the first two values are self-explanatory, "auto" will enable or
    ++# disable the feature based on the availability of prerequisites to support it.
    ++# Python-based features for example will be enabled automatically when a Python
    ++# interpreter could be found. The default value of such features can be changed
    ++# globally via `meson setup --auto-features={enabled,disabled,auto}`, which
    ++# will set the value of all features with a value of "auto" to the provided one
    ++# by default.
    ++#
    ++# It is also possible to store a set of configuration options in machine files.
    ++# This can be useful in case you regularly want to reuse the same set of options:
    ++#
    ++#   [binaries]
    ++#   c = ['clang']
    ++#   ar = ['ar']
    ++#
    ++#   [project options]
    ++#   gettext = 'disabled'
    ++#   default_editor = 'vim'
    ++#
    ++#   [built-in options]
    ++#   b_lto = true
    ++#   b_sanitize = 'address,undefined'
    ++#
    ++# These machine files can be passed to Meson via `meson setup --native-file`.
    ++#
    ++# Subproject wrappers
    ++# ===================
    ++#
    ++# Subproject wrappers are a feature provided by Meson that allow the automatic
    ++# fallback to a "wrapped" dependency in case the dependency is not provided by
    ++# the system. For example if the system is lacking curl, then Meson will use
    ++# "subprojects/curl.wrap" to set up curl as a subproject and compile and link
    ++# the dependency into Git itself. This is especially helpful on systems like
    ++# Windows, where you typically don't have such dependencies installed.
    ++#
    ++# The use of subproject wrappers can be disabled by executing `meson setup
    ++# --wrap-mode nofallback`.
    ++
     +project('git', 'c',
     +  meson_version: '>=1.3.0',
     +  # MSVC does not support GNU99, and C99 does not define __STDC_VERSION__
    @@ meson.build (new)
     +  build_options_config.set('NO_PTHREADS', '1')
     +endif
     +
    -+if get_option('gettext').allowed() and host_machine.system() == 'darwin' and get_option('macos_use_homebrew_gettext')
    ++msgfmt = find_program('msgfmt', dirs: program_path, required: false)
    ++gettext_option = get_option('gettext').disable_auto_if(not msgfmt.found())
    ++if not msgfmt.found() and gettext_option.enabled()
    ++  error('Internationalization via libintl requires msgfmt')
    ++endif
    ++
    ++if gettext_option.allowed() and host_machine.system() == 'darwin' and get_option('macos_use_homebrew_gettext')
     +  if host_machine.cpu_family() == 'x86_64'
     +    libintl_prefix = '/usr/local'
     +  elif host_machine.cpu_family() == 'aarch64'
    @@ meson.build (new)
     +    error('Homebrew workaround not supported on current architecture')
     +  endif
     +
    -+  intl = compiler.find_library('intl', dirs: libintl_prefix / 'lib', required: get_option('gettext'))
    ++  intl = compiler.find_library('intl', dirs: libintl_prefix / 'lib', required: gettext_option)
     +  if intl.found()
     +    intl = declare_dependency(
     +      dependencies: intl,
    @@ meson.build (new)
     +    )
     +  endif
     +else
    -+  intl = dependency('intl', required: get_option('gettext'))
    ++  intl = dependency('intl', required: gettext_option)
     +endif
     +if intl.found()
     +  libgit_dependencies += intl
19:  f483916825 ! 19:  8315d432f1 meson: fix conflicts with in-flight topics
    @@ Commit message
         conflicts with topics in "seen". The following conflicts are being
         addressed in this commit:
     
    -      - ej/cat-file-remote-object-info adds t1017.
    +      - ej/cat-file-remote-object-info adds t1017 and "fetch-object-info.c".
     
           - cc/promisor-remote-capability adds t5710.
     
           - ds/path-walk adds t6601 as well as "path-walk.c" and
             "test-path-walk.c".
     
    -      - am/git-blame-ignore-revs-by-default adds t8015 and t8016.
    -
           - ps/reftable-detach adds "reftable/system.c".
     
           - js/libgit-rust adds "common-exit.c" and "common-init.c".
     
    +      - ps/clar-build-improvement adapts the awk(1) script that generates
    +        "clar.suites" to instead be a shell script.
    +
    +      - ds/full-name-hash adds "t/helper/test-name-hash.c".
    +
    +      - cw/worktree-extension deletes t2408.
    +
         This is somewhat painful in the current state where Meson is not yet
         part of the main tree, but we'll have to live with that for the time
         being.
    @@ meson.build: libgit_sources = [
        'compat/nonblock.c',
        'compat/obstack.c',
        'compat/terminal.c',
    +@@ meson.build: libgit_sources = [
    +   'ewah/ewah_rlw.c',
    +   'exec-cmd.c',
    +   'fetch-negotiator.c',
    ++  'fetch-object-info.c',
    +   'fetch-pack.c',
    +   'fmt-merge-msg.c',
    +   'fsck.c',
     @@ meson.build: libgit_sources = [
        'parse-options.c',
        'patch-delta.c',
    @@ meson.build: libgit_sources = [
     
      ## t/helper/meson.build ##
     @@ t/helper/meson.build: test_tool_sources = [
    +   'test-match-trees.c',
    +   'test-mergesort.c',
    +   'test-mktemp.c',
    ++  'test-name-hash.c',
    +   'test-online-cpus.c',
    +   'test-pack-mtimes.c',
    +   'test-parse-options.c',
        'test-parse-pathspec-file.c',
        'test-partial-clone.c',
        'test-path-utils.c',
    @@ t/helper/meson.build: test_tool_sources = [
        'test-proc-receive.c',
     
      ## t/meson.build ##
    +@@ t/meson.build: clar_sources += clar_decls_h
    + clar_sources += custom_target(
    +   input: clar_decls_h,
    +   output: 'clar.suite',
    +-  feed: true,
    +-  capture: true,
    +-  command : [awk, '-f', meson.current_source_dir() + '/unit-tests/clar-generate.awk'],
    ++  command : [shell, meson.current_source_dir() + '/unit-tests/generate-clar-suites.sh', '@INPUT@', '@OUTPUT@'],
    ++  env: script_environment,
    + )
    + 
    + clar_unit_tests = executable('unit-tests',
     @@ t/meson.build: integration_tests = [
        't1014-read-tree-confusing.sh',
        't1015-read-index-unmerged.sh',
    @@ t/meson.build: integration_tests = [
        't1020-subdirectory.sh',
        't1021-rerere-in-workdir.sh',
        't1022-read-tree-partial-clone.sh',
    +@@ t/meson.build: integration_tests = [
    +   't2405-worktree-submodule.sh',
    +   't2406-worktree-repair.sh',
    +   't2407-worktree-heads.sh',
    +-  't2408-worktree-relative.sh',
    +   't2500-untracked-overwriting.sh',
    +   't2501-cwd-empty.sh',
    +   't3000-ls-files-others.sh',
     @@ t/meson.build: integration_tests = [
        't5703-upload-pack-ref-in-want.sh',
        't5704-protocol-violations.sh',
    @@ t/meson.build: integration_tests = [
        't6700-tree-depth.sh',
        't7001-mv.sh',
        't7002-mv-sparse-checkout.sh',
    -@@ t/meson.build: integration_tests = [
    -   't8012-blame-colors.sh',
    -   't8013-blame-ignore-revs.sh',
    -   't8014-blame-ignore-fuzzy.sh',
    -+  't8015-blame-default-ignore-revs.sh',
    -+  't8016-blame-override-ignore-revs.sh',
    -   't9001-send-email.sh',
    -   't9002-column.sh',
    -   't9003-help-autocorrect.sh',

---
base-commit: b31fb630c0fc6869a33ed717163e8a1210460d94
change-id: 20241112-pks-meson-95b665c08f06

Comments

Ramsay Jones Nov. 13, 2024, 3:30 a.m. UTC | #1
On 12/11/2024 17:02, Patrick Steinhardt wrote:
> Hi,
> 
> this patch series modernizes our build infrasturcture. It refactors
> various parts of it to make it possible to perform out-of-tree builds in
> theory.
> 
> The series then continues to wire up Meson as a proposed alternative as
> a modern replacement for our current build systems. It provides better
> integration into IDEs than our Makefiles, better usability than our
> Makefiles and CMake, better configuration and discoverability thereof
> than autoconf and CMake (which is of course my personal opinion).
> 
> The intent is to have Meson replace autoconf, which is mostly unused and
> broken. Eventually, the plan would to also replace CMake and our
> Makefiles in case we figure out that this replacement is indeed a net
> benefit for the project, but all of this would happen over the course of
> multiple releases.
> 
> Changes in v6:
> 
>   - Add documentation for how to use Meson.
> 
>   - Fix compilation in case libintl exists, but required tools like
>     msgfmt and related don't. Instead of failing, we now automatically
>     compile with the equivalent of NO_GETTEXT=YesPlease.
> 
>   - Update last patch that fixes semantic merge conflics with in-flight
>     topics in "seen".
> 
>   - I've changed the series to be based on "master" at b31fb630c0 (Merge
>     https://github.com/j6t/git-gui, 2024-11-11) as all prereqs of this
>     series have since landed. This doesn't make any difference though
>     as there aren't any merge conflicts or additional changes caused by
>     this. I mostly mention this so that there is no confusion around the
>     changed base commit.
> 

Tested this version, without issue, on Linux and cygwin (build + test).

On cygwin, the unit test now no longer times out:

  $ cd git/build
  $ tail test-out
  1030/1030 t9902-completion                               OK             114.74s

  Ok:                 1030
  Expected Fail:      0   
  Fail:               0   
  Unexpected Pass:    0   
  Skipped:            0   
  Timeout:            0   

  Full log written to /home/ramsay/git/build/meson-logs/testlog.txt
  $ 

Having said that, I was expecting to see an 'Unexpected Pass', since
meson decides to not set NO_REGEX. (The 'make' build on cygwin will
set 'NO_REGEX = UnfortunatelyYes' in the config.mak.uname).

In the meson-logs/testlog.txt file, for the t7815-grep-binary.sh test
stdout, we find:

----------------------------------- stdout -----------------------------------
ok 1 - setup
ok 2 - git grep ina a
ok 3 - git grep -ah ina a
ok 4 - git grep -I ina a
ok 5 - git grep -c ina a
ok 6 - git grep -l ina a
ok 7 - git grep -L bar a
ok 8 - git grep -q ina a
ok 9 - git grep -F ile a
ok 10 - git grep -Fi iLE a
ok 11 - git grep ile a
ok 12 - git grep .fi a # TODO known breakage vanished
ok 13 - grep respects binary diff attribute
ok 14 - grep --cached respects binary diff attribute
ok 15 - grep --cached respects binary diff attribute (2)
ok 16 - grep revision respects binary diff attribute
ok 17 - grep respects not-binary diff attribute
ok 18 - setup textconv filters
ok 19 - grep does not honor textconv
ok 20 - grep --textconv honors textconv
ok 21 - grep --no-textconv does not honor textconv
ok 22 - grep --textconv blob honors textconv
# 1 known breakage(s) vanished; please update test(s)
# passed all remaining 21 test(s)
1..22
==============================================================================

Note the 'TODO known breakage vanished' on test 12.

Setting NO_REGEX on cygwin is for 'reasons' (not necessarily good) and we
need a way to override it's decision here. (In general, we need to be able
to use *any* of the many build variables to override meson's automatic
setting of these build variables).


Thanks!

ATB,
Ramsay Jones
Christian Couder Nov. 13, 2024, 10:45 a.m. UTC | #2
On Tue, Nov 12, 2024 at 9:42 PM Patrick Steinhardt <ps@pks.im> wrote:

>     ++#    The prefix into which Git shall be installed is defined when setting up
>     ++#    the build directory. More on that in the "Configuration" section.
>     ++#
>     ++# Meson supports multiple backends. The default backend generates Ninja build
>     ++# instructions, but it also supports the generation of Microsoft Visual
>     ++# Studio solutions as well as Xcode projects. IDEs like Eclipse and Visual
>     ++# Studio Code provide plugins to import Meson files directly.
>     ++#
>     ++# Configuration
>     ++# =============
>     ++#
>     ++# The exact configuration of Git is determined when setting up the Git
>     ++# directory.

Is it the "Git directory" or the "build directory"?

It might be helpful to repeat that this happens when `meson setup
build/` is run (as far as I understand it). Maybe something like:

"The exact configuration of Git is determined when setting up the
build directory, so when running `meson setup <build-dir>/`."

>Unless told otherwise, Meson will automatically detect the
>     ++# availability of various bits and pieces. There are two different kinds of
>     ++# options that can be used to further tweak the build:
>     ++#
>     ++#   - Built-in options provided by Meson.
>     ++#
>     ++#   - Options defined by the project in the "meson_options.txt" file.
>     ++#
>     ++# Both kinds of options can be inspected by running `meson configure` in the
>     ++# build directory, which will give you a list of the current value for all
>     ++# options.
>     ++#
>     ++# Options can be configured either when setting up the build directory or can
>     ++# be changed in preexisting build directories:
>     ++#
>     ++#      # Set up a build directory with optimized settings that will be
>     ++#      # installed into an alternative prefix.
>     ++#      $ meson setup --buildtype release --optimization 3 --strip --prefix=/home/$USER

It's not very clear if the above instruction should be run inside an
existing build directory to modify its configuration, or if it creates
a new build directory. If it creates one, it's also not clear what the
name of that directory would be. Maybe "build", but then what if an
existing build directory exists with that name?

>     ++#      # Set up a build directory with 'address' and 'undefined' sanitizers
>     ++#      # using Clang.
>     ++#      $ CC=clang meson setup -Db_sanitize=address,undefined

Same as above.

>     ++#      # Disable tests in a preexisting build directory.
>     ++#      $ meson configure -Dtests=false

As here the command is "configure" instead of "setup", I guess it
doesn't create a build directory and should be run inside a one.

>     ++#      # Disable features based on Python
>     ++#      $ meson configure -Dpython=disabled
>     ++#
>     ++#      # Disable features based on Python
>     ++#      $ meson configure -Dpython=disabled

It looks like there is some duplication above.

>     ++# Options have a type like booleans, choices, strings or features. Features are
>     ++# somewhat special as they can have one of three values: enabled, disabled or
>     ++# auto. While the first two values are self-explanatory, "auto" will enable or
>     ++# disable the feature based on the availability of prerequisites to support it.
>     ++# Python-based features for example will be enabled automatically when a Python
>     ++# interpreter could be found. The default value of such features can be changed
>     ++# globally via `meson setup --auto-features={enabled,disabled,auto}`, which
>     ++# will set the value of all features with a value of "auto" to the provided one
>     ++# by default.
>     ++#
>     ++# It is also possible to store a set of configuration options in machine files.

It's not very clear what a "machine file" is. How is it different from
a config file?

>     ++# This can be useful in case you regularly want to reuse the same set of options:
>     ++#
>     ++#   [binaries]
>     ++#   c = ['clang']
>     ++#   ar = ['ar']
>     ++#
>     ++#   [project options]
>     ++#   gettext = 'disabled'
>     ++#   default_editor = 'vim'
>     ++#
>     ++#   [built-in options]
>     ++#   b_lto = true
>     ++#   b_sanitize = 'address,undefined'
>     ++#
>     ++# These machine files can be passed to Meson via `meson setup --native-file`.
>     ++#
>     ++# Subproject wrappers
>     ++# ===================
>     ++#
>     ++# Subproject wrappers are a feature provided by Meson that allow the automatic
>     ++# fallback to a "wrapped" dependency in case the dependency is not provided by
>     ++# the system. For example if the system is lacking curl, then Meson will use
>     ++# "subprojects/curl.wrap" to set up curl as a subproject and compile and link
>     ++# the dependency into Git itself. This is especially helpful on systems like
>     ++# Windows, where you typically don't have such dependencies installed.
>     ++#
>     ++# The use of subproject wrappers can be disabled by executing `meson setup
>     ++# --wrap-mode nofallback`.

I guess this will make the build fail if the system doesn't provide curl then.
Patrick Steinhardt Nov. 13, 2024, 1:24 p.m. UTC | #3
On Wed, Nov 13, 2024 at 11:45:41AM +0100, Christian Couder wrote:
> On Tue, Nov 12, 2024 at 9:42 PM Patrick Steinhardt <ps@pks.im> wrote:
> 
> >     ++#    The prefix into which Git shall be installed is defined when setting up
> >     ++#    the build directory. More on that in the "Configuration" section.
> >     ++#
> >     ++# Meson supports multiple backends. The default backend generates Ninja build
> >     ++# instructions, but it also supports the generation of Microsoft Visual
> >     ++# Studio solutions as well as Xcode projects. IDEs like Eclipse and Visual
> >     ++# Studio Code provide plugins to import Meson files directly.
> >     ++#
> >     ++# Configuration
> >     ++# =============
> >     ++#
> >     ++# The exact configuration of Git is determined when setting up the Git
> >     ++# directory.
> 
> Is it the "Git directory" or the "build directory"?
> 
> It might be helpful to repeat that this happens when `meson setup
> build/` is run (as far as I understand it). Maybe something like:
> 
> "The exact configuration of Git is determined when setting up the
> build directory, so when running `meson setup <build-dir>/`."

Good catch, this is of course the build directory. I've also adapted
in the spirit of your proposed text.

> >Unless told otherwise, Meson will automatically detect the
> >     ++# availability of various bits and pieces. There are two different kinds of
> >     ++# options that can be used to further tweak the build:
> >     ++#
> >     ++#   - Built-in options provided by Meson.
> >     ++#
> >     ++#   - Options defined by the project in the "meson_options.txt" file.
> >     ++#
> >     ++# Both kinds of options can be inspected by running `meson configure` in the
> >     ++# build directory, which will give you a list of the current value for all
> >     ++# options.
> >     ++#
> >     ++# Options can be configured either when setting up the build directory or can
> >     ++# be changed in preexisting build directories:
> >     ++#
> >     ++#      # Set up a build directory with optimized settings that will be
> >     ++#      # installed into an alternative prefix.
> >     ++#      $ meson setup --buildtype release --optimization 3 --strip --prefix=/home/$USER
> 
> It's not very clear if the above instruction should be run inside an
> existing build directory to modify its configuration, or if it creates
> a new build directory. If it creates one, it's also not clear what the
> name of that directory would be. Maybe "build", but then what if an
> existing build directory exists with that name?

Oh, that's because I forgot to add the name of the build directory.

[snip]
> >     ++#      # Disable features based on Python
> >     ++#      $ meson configure -Dpython=disabled
> >     ++#
> >     ++#      # Disable features based on Python
> >     ++#      $ meson configure -Dpython=disabled
> 
> It looks like there is some duplication above.

Ah, indeed.

> >     ++# Options have a type like booleans, choices, strings or features. Features are
> >     ++# somewhat special as they can have one of three values: enabled, disabled or
> >     ++# auto. While the first two values are self-explanatory, "auto" will enable or
> >     ++# disable the feature based on the availability of prerequisites to support it.
> >     ++# Python-based features for example will be enabled automatically when a Python
> >     ++# interpreter could be found. The default value of such features can be changed
> >     ++# globally via `meson setup --auto-features={enabled,disabled,auto}`, which
> >     ++# will set the value of all features with a value of "auto" to the provided one
> >     ++# by default.
> >     ++#
> >     ++# It is also possible to store a set of configuration options in machine files.
> 
> It's not very clear what a "machine file" is. How is it different from
> a config file?

It's not, the machine file is a configuration file. It's just what Meson
calls it.

> >     ++# This can be useful in case you regularly want to reuse the same set of options:
> >     ++#
> >     ++#   [binaries]
> >     ++#   c = ['clang']
> >     ++#   ar = ['ar']
> >     ++#
> >     ++#   [project options]
> >     ++#   gettext = 'disabled'
> >     ++#   default_editor = 'vim'
> >     ++#
> >     ++#   [built-in options]
> >     ++#   b_lto = true
> >     ++#   b_sanitize = 'address,undefined'
> >     ++#
> >     ++# These machine files can be passed to Meson via `meson setup --native-file`.
> >     ++#
> >     ++# Subproject wrappers
> >     ++# ===================
> >     ++#
> >     ++# Subproject wrappers are a feature provided by Meson that allow the automatic
> >     ++# fallback to a "wrapped" dependency in case the dependency is not provided by
> >     ++# the system. For example if the system is lacking curl, then Meson will use
> >     ++# "subprojects/curl.wrap" to set up curl as a subproject and compile and link
> >     ++# the dependency into Git itself. This is especially helpful on systems like
> >     ++# Windows, where you typically don't have such dependencies installed.
> >     ++#
> >     ++# The use of subproject wrappers can be disabled by executing `meson setup
> >     ++# --wrap-mode nofallback`.
> 
> I guess this will make the build fail if the system doesn't provide curl then.

It depends on the value of the feature. If it's set to 'enabled' then
yes, Meson would fail. If it's set to 'auto' then it would simply
disable the feature and continue.

Right now it's set to 'enabled', so yes, we'd fail.

Patrick
Patrick Steinhardt Nov. 13, 2024, 1:24 p.m. UTC | #4
On Wed, Nov 13, 2024 at 03:30:20AM +0000, Ramsay Jones wrote:
> On 12/11/2024 17:02, Patrick Steinhardt wrote:
> > Hi,
> > 
> > this patch series modernizes our build infrasturcture. It refactors
> > various parts of it to make it possible to perform out-of-tree builds in
> > theory.
> > 
> > The series then continues to wire up Meson as a proposed alternative as
> > a modern replacement for our current build systems. It provides better
> > integration into IDEs than our Makefiles, better usability than our
> > Makefiles and CMake, better configuration and discoverability thereof
> > than autoconf and CMake (which is of course my personal opinion).
> > 
> > The intent is to have Meson replace autoconf, which is mostly unused and
> > broken. Eventually, the plan would to also replace CMake and our
> > Makefiles in case we figure out that this replacement is indeed a net
> > benefit for the project, but all of this would happen over the course of
> > multiple releases.
> > 
> > Changes in v6:
> > 
> >   - Add documentation for how to use Meson.
> > 
> >   - Fix compilation in case libintl exists, but required tools like
> >     msgfmt and related don't. Instead of failing, we now automatically
> >     compile with the equivalent of NO_GETTEXT=YesPlease.
> > 
> >   - Update last patch that fixes semantic merge conflics with in-flight
> >     topics in "seen".
> > 
> >   - I've changed the series to be based on "master" at b31fb630c0 (Merge
> >     https://github.com/j6t/git-gui, 2024-11-11) as all prereqs of this
> >     series have since landed. This doesn't make any difference though
> >     as there aren't any merge conflicts or additional changes caused by
> >     this. I mostly mention this so that there is no confusion around the
> >     changed base commit.
> > 
> 
> Tested this version, without issue, on Linux and cygwin (build + test).

Thanks for verifying!

> On cygwin, the unit test now no longer times out:
> 
>   $ cd git/build
>   $ tail test-out
>   1030/1030 t9902-completion                               OK             114.74s
> 
>   Ok:                 1030
>   Expected Fail:      0   
>   Fail:               0   
>   Unexpected Pass:    0   
>   Skipped:            0   
>   Timeout:            0   
> 
>   Full log written to /home/ramsay/git/build/meson-logs/testlog.txt
>   $ 

Great.

> Having said that, I was expecting to see an 'Unexpected Pass', since
> meson decides to not set NO_REGEX. (The 'make' build on cygwin will
> set 'NO_REGEX = UnfortunatelyYes' in the config.mak.uname).
> 
> In the meson-logs/testlog.txt file, for the t7815-grep-binary.sh test
> stdout, we find:
> 
> ----------------------------------- stdout -----------------------------------
> ok 1 - setup
> ok 2 - git grep ina a
> ok 3 - git grep -ah ina a
> ok 4 - git grep -I ina a
> ok 5 - git grep -c ina a
> ok 6 - git grep -l ina a
> ok 7 - git grep -L bar a
> ok 8 - git grep -q ina a
> ok 9 - git grep -F ile a
> ok 10 - git grep -Fi iLE a
> ok 11 - git grep ile a
> ok 12 - git grep .fi a # TODO known breakage vanished
> ok 13 - grep respects binary diff attribute
> ok 14 - grep --cached respects binary diff attribute
> ok 15 - grep --cached respects binary diff attribute (2)
> ok 16 - grep revision respects binary diff attribute
> ok 17 - grep respects not-binary diff attribute
> ok 18 - setup textconv filters
> ok 19 - grep does not honor textconv
> ok 20 - grep --textconv honors textconv
> ok 21 - grep --no-textconv does not honor textconv
> ok 22 - grep --textconv blob honors textconv
> # 1 known breakage(s) vanished; please update test(s)
> # passed all remaining 21 test(s)
> 1..22
> ==============================================================================
> 
> Note the 'TODO known breakage vanished' on test 12.

Interesting. Right now we don't parse the output of the test executions
themselves, so we only operate on the level of whether or not the whole
test suite passed. So it's expected that this does not get reported as
an unexpected pass.

While Meson can itself generate TAP output, I don't think that it has
support for _reading_ the TAP format.

> Setting NO_REGEX on cygwin is for 'reasons' (not necessarily good) and we
> need a way to override it's decision here. (In general, we need to be able
> to use *any* of the many build variables to override meson's automatic
> setting of these build variables).

I can add an option for this. Out of curiosity, what is the reason? Is
this anything that we can autodetect?

Patrick