ci: only run win+VS build & tests in Git for Windows' fork

Commit Message

Johannes Schindelin Dec. 19, 2022, 2:50 p.m. UTC

From: Johannes Schindelin <[e-mail]>

It has been a frequent matter of contention that the win+VS jobs not
only take a long time to run, but are also more easily broken than the
other jobs (because they do not use the same `Makefile`-based builds as
all other jobs), and to make matters worse, these breakages are also
much harder to diagnose and fix than other jobs', especially for
contributors who are happy to stay away from Windows.

The purpose of these win+VS jobs is to maintain the CMake-based build
of Git, with the target audience being Visual Studio users on Windows
who are typically quite unfamiliar with `make` and POSIX shell
scripting, but the benefit of whose expertise we want for the Git
project nevertheless.

The CMake support was introduced for that specific purpose, and already
early on concerns were raised that it would put an undue burden on
contributors to ensure that these jobs pass in CI, when they do not have
access to Windows machines (nor want to have that).

This developer's initial hope was that it would be enough to fix win+VS
failures and provide the changes to be squashed into contributors'
patches, and that it would be worth the benefit of attracting
Windows-based developers' contributions.

Neither of these hopes have panned out.

To lower the frustration, and incidentally benefit from using way less
build minutes, let's just not run the win+VS jobs by default, which
appears to be the consensus of the mail thread leading up to
https://lore.kernel.org/git/xmqqk0311blt.fsf@gitster.g/

Since the Git for Windows project still needs to at least try to attract
more of said Windows-based developers, let's keep the jobs, but disable
them everywhere except in Git for Windows' fork. This will help because
Git for Windows' branch thicket is "continuously rebased" via automation
to the `shears/maint`, `shears/main`, `shears/next` and `shears/seen`
branches at https://github.com/git-for-windows/git. That way, the Git
for Windows project will still be notified early on about potential
breakages, but the Git project won't be burdened with fixing them
anymore, which seems to be the best compromise we can get on this issue.

Signed-off-by: Johannes Schindelin <[e-mail]>
---
    Explicitly make CMake-based CI failures the responsibility of "Windows
    folks"
    
    Ævar and I have brain-stormed off-list what would be the best way to
    resolve the mounting frustration with CI failures that are caused by
    needing to mirror Makefile changes into the CMake-based build, a burden
    that the Git project never wanted to bear.
    
    While he still wants to improve the CMake support (which will benefit
    Git for Windows), the main driver of trying to extend CMake support to
    Linux (which does not need it because make works very well there,
    indeed) was said frustration with CI failures.
    
    A much quicker method to reduce that friction to close to nil is to
    simply disable the win+VS jobs, which is what this proposal is about.
    (Almost, at least, we still want to keep those job definitions and run
    them in Git for Windows' fork to ensure that CMake-based builds still
    work.)
    
    A very welcome side effect is to reduce the CI build time again, which
    became alarmingly long as of recent, causing friction on its own.

Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-1445%2Fdscho%2Fonly-run-win%2BVS-in-the-git-for-windows-fork-v1
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1445/dscho/only-run-win+VS-in-the-git-for-windows-fork-v1
Pull-Request: https://github.com/gitgitgadget/git/pull/1445

 .github/workflows/main.yml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)


base-commit: c48035d29b4e524aed3a32f0403676f0d9128863

Comments

Phillip Wood Dec. 19, 2022, 3:04 p.m. UTC | #1

Hi Dscho

On 19/12/2022 14:50, Johannes Schindelin via GitGitGadget wrote:
> From: Johannes Schindelin <[e-mail]>
> [...] 
> This developer's initial hope was that it would be enough to fix win+VS
> failures and provide the changes to be squashed into contributors'
> patches, and that it would be worth the benefit of attracting
> Windows-based developers' contributions.
> 
> Neither of these hopes have panned out.

I'm sad that those hopes have not panned out, but I think this patch is 
probably the best way forward. It is certainly the lowest effort for 
non-windows developers and the improvement it CI minutes is very welcome.

Best Wishes

Phillip

> To lower the frustration, and incidentally benefit from using way less
> build minutes, let's just not run the win+VS jobs by default, which
> appears to be the consensus of the mail thread leading up to
> https://lore.kernel.org/git/xmqqk0311blt.fsf@gitster.g/
> 
> Since the Git for Windows project still needs to at least try to attract
> more of said Windows-based developers, let's keep the jobs, but disable
> them everywhere except in Git for Windows' fork. This will help because
> Git for Windows' branch thicket is "continuously rebased" via automation
> to the `shears/maint`, `shears/main`, `shears/next` and `shears/seen`
> branches at https://github.com/git-for-windows/git. That way, the Git
> for Windows project will still be notified early on about potential
> breakages, but the Git project won't be burdened with fixing them
> anymore, which seems to be the best compromise we can get on this issue.
> 
> Signed-off-by: Johannes Schindelin <[e-mail]>
> ---
>      Explicitly make CMake-based CI failures the responsibility of "Windows
>      folks"
>      
>      Ævar and I have brain-stormed off-list what would be the best way to
>      resolve the mounting frustration with CI failures that are caused by
>      needing to mirror Makefile changes into the CMake-based build, a burden
>      that the Git project never wanted to bear.
>      
>      While he still wants to improve the CMake support (which will benefit
>      Git for Windows), the main driver of trying to extend CMake support to
>      Linux (which does not need it because make works very well there,
>      indeed) was said frustration with CI failures.
>      
>      A much quicker method to reduce that friction to close to nil is to
>      simply disable the win+VS jobs, which is what this proposal is about.
>      (Almost, at least, we still want to keep those job definitions and run
>      them in Git for Windows' fork to ensure that CMake-based builds still
>      work.)
>      
>      A very welcome side effect is to reduce the CI build time again, which
>      became alarmingly long as of recent, causing friction on its own.
> 
> Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-1445%2Fdscho%2Fonly-run-win%2BVS-in-the-git-for-windows-fork-v1
> Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1445/dscho/only-run-win+VS-in-the-git-for-windows-fork-v1
> Pull-Request: https://github.com/gitgitgadget/git/pull/1445
> 
>   .github/workflows/main.yml | 2 +-
>   1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml
> index e67847a682c..8af3c67f605 100644
> --- a/.github/workflows/main.yml
> +++ b/.github/workflows/main.yml
> @@ -132,7 +132,7 @@ jobs:
>     vs-build:
>       name: win+VS build
>       needs: ci-config
> -    if: needs.ci-config.outputs.enabled == 'yes'
> +    if: github.event.repository.owner.login == 'git-for-windows' && needs.ci-config.outputs.enabled == 'yes'
>       env:
>         NO_PERL: 1
>         GIT_CONFIG_PARAMETERS: "'user.name=CI' 'user.email=ci@git'"
> 
> base-commit: c48035d29b4e524aed3a32f0403676f0d9128863

Ævar Arnfjörð Bjarmason Dec. 19, 2022, 5:49 p.m. UTC | #2

On Mon, Dec 19 2022, Johannes Schindelin via GitGitGadget wrote:

> From: Johannes Schindelin <[e-mail]>
> [...]
>     Explicitly make CMake-based CI failures the responsibility of "Windows
>     folks"
>     
>     Ævar and I have brain-stormed off-list what would be the best way to
>     resolve the mounting frustration with CI failures that are caused by
>     needing to mirror Makefile changes into the CMake-based build, a burden
>     that the Git project never wanted to bear.
>     
>     While he still wants to improve the CMake support (which will benefit
>     Git for Windows), the main driver of trying to extend CMake support to
>     Linux (which does not need it because make works very well there,
>     indeed) was said frustration with CI failures.
>     
>     A much quicker method to reduce that friction to close to nil is to
>     simply disable the win+VS jobs, which is what this proposal is about.
>     (Almost, at least, we still want to keep those job definitions and run
>     them in Git for Windows' fork to ensure that CMake-based builds still
>     work.)
>     
>     A very welcome side effect is to reduce the CI build time again, which
>     became alarmingly long as of recent, causing friction on its own.
>
> Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-1445%2Fdscho%2Fonly-run-win%2BVS-in-the-git-for-windows-fork-v1
> Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1445/dscho/only-run-win+VS-in-the-git-for-windows-fork-v1
> Pull-Request: https://github.com/gitgitgadget/git/pull/1445
>
>  .github/workflows/main.yml | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml
> index e67847a682c..8af3c67f605 100644
> --- a/.github/workflows/main.yml
> +++ b/.github/workflows/main.yml
> @@ -132,7 +132,7 @@ jobs:
>    vs-build:
>      name: win+VS build
>      needs: ci-config
> -    if: needs.ci-config.outputs.enabled == 'yes'
> +    if: github.event.repository.owner.login == 'git-for-windows' && needs.ci-config.outputs.enabled == 'yes'
>      env:
>        NO_PERL: 1
>        GIT_CONFIG_PARAMETERS: "'user.name=CI' 'user.email=ci@git'"

In terms of implementation: I would like this to use ci-config.

Yes, it will take us some CI time just to spin up the image to decide on
that (only a few seconds), but it's preferrable to hardcoding
github.com/git-for-windows, as you'll be able to opt-in to this to test
changes.

The outstanding "tb/ci-concurrency" topic shows how to do that (and
tweaks an earlier submission of yours where it was similarly hardcoded).

> The purpose of these win+VS jobs is to maintain the CMake-based build
> of Git, with the target audience being Visual Studio users on Windows
> who are typically quite unfamiliar with `make` and POSIX shell

I thought the initial purpose of it was to test compiling & testing with
MSVC rather than GCC?

It's only after 4c2c38e800f (ci: modification of main.yml to use cmake
for vs-build job, 2020-06-26) that it got co-opted to test cmake too.

Whatever we do about the job at this point not discussing 4c2c38e800f in
the commit seems like a big omission, why not revert & adjust it if the
cmake part of it is a perceived hassle?

Now, I don't think that it's worth spending time on running two sets of
Windows tests all the time just to tease out GCC v.s. MSVC differences,
but I think we should still run the build on both. That takes around
~5m, and seems worth it to avoid and flag any portability issues.

Before this we test on gcc/clang/msvc, after this we'll have dropped
that last one. Whatever we do about the tests and/or cmake I think we
really should avoid compiler monoculture, and fix portability issues
early.

> A very welcome side effect is to reduce the CI build time again, which
> became alarmingly long as of recent, causing friction on its own.

I think this is a good goal, but what does the "as of recent" refer to
here? When the ASAN job was introduced?

Junio C Hamano Dec. 20, 2022, 12:52 a.m. UTC | #3

"Johannes Schindelin via GitGitGadget" <[e-mail]>
writes:

>     Ævar and I have brain-stormed off-list what would be the best way to
>     resolve the mounting frustration with CI failures that are caused by
>     needing to mirror Makefile changes into the CMake-based build, a burden
>     that the Git project never wanted to bear.

Thanks, both.

>     A very welcome side effect is to reduce the CI build time again, which
>     became alarmingly long as of recent, causing friction on its own.

;-)

Johannes Schindelin Dec. 20, 2022, 9:35 a.m. UTC | #4

Hi Ævar,

On Mon, 19 Dec 2022, Ævar Arnfjörð Bjarmason wrote:

> On Mon, Dec 19 2022, Johannes Schindelin via GitGitGadget wrote:
>
> > diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml
> > index e67847a682c..8af3c67f605 100644
> > --- a/.github/workflows/main.yml
> > +++ b/.github/workflows/main.yml
> > @@ -132,7 +132,7 @@ jobs:
> >    vs-build:
> >      name: win+VS build
> >      needs: ci-config
> > -    if: needs.ci-config.outputs.enabled == 'yes'
> > +    if: github.event.repository.owner.login == 'git-for-windows' && needs.ci-config.outputs.enabled == 'yes'
> >      env:
> >        NO_PERL: 1
> >        GIT_CONFIG_PARAMETERS: "'user.name=CI' 'user.email=ci@git'"
>
> In terms of implementation: I would like this to use ci-config.

I really do not see any value in that.

If you _really_ want to run those tests, you can easily add a commit on
top of your branch.

It's not like you always want to run the win+VS tests on a certain subset
of your branches.

But. A big part of the reason for this patch is to codify that it
is _not_ the responsibility of core Git contributors to take care of these
tests. Trying to do this via the utterly convoluted and under-documented
`ci-config` would only water down this quite clear statement.

> The outstanding "tb/ci-concurrency" topic shows how to do that (and
> tweaks an earlier submission of yours where it was similarly hardcoded).

I fear that the `tb/ci-concurrency` topic is a good example of "death by
committee". Let's put in a collective effort to avoid that here.

> > The purpose of these win+VS jobs is to maintain the CMake-based build
> > of Git, with the target audience being Visual Studio users on Windows
> > who are typically quite unfamiliar with `make` and POSIX shell
>
> I thought the initial purpose of it was to test compiling & testing with
> MSVC rather than GCC?

That might have been the motivation, but there have been preciously few
patches coming in from that side.

So few, in fact, that the question "is it worth spending all of that
energy to run these builds and tests all the time" most likely has to be
answered with a resounding "No".

I can think of just two bugs that were identified in the MSVC builds, one
where we had to change some code to explicitly use a stable sort (which
MSVC would otherwise not have used), and one where we now avoid an
unsigned/signed comparison where MSVC cast the signed value to a
now-insanely-large unsigned, i.e. in the wrong direction.

That's two bugs identified in how many years?

We still can catch those bugs in git-for-windows/git. And avoid some of
the long build times.

> > A very welcome side effect is to reduce the CI build time again, which
> > became alarmingly long as of recent, causing friction on its own.
>
> I think this is a good goal, but what does the "as of recent" refer to
> here? When the ASAN job was introduced?

It's not only `linux-asan`. The overall build time of every single job has
gone up. I picked randomly two runs that were roughly 6 months apart,
https://github.com/git/git/actions/runs/2537915353 and
https://github.com/git/git/actions/runs/3728047162. As an indicator, the
`osx-gcc` job took 35m half a year ago, now it takes 40m. The overall time
went up from 7h40 (which is already _quite_ a cost!) to 8h01.

I don't think that we, collectively, are judicious enough about the
balance between cost and benefit here.

Having said that, there is a silver lining: while the linux-asan job is
new and takes a whopping 44m to complete, the difference between the total
run time 6 months ago and today is less than that, so we must have saved
on _something_, _somewhere_. Or GitHub bought faster hardware.

Ciao,
Johannes

Patch

diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml
index e67847a682c..8af3c67f605 100644
--- a/.github/workflows/main.yml
+++ b/.github/workflows/main.yml
@@ -132,7 +132,7 @@  jobs:
   vs-build:
     name: win+VS build
     needs: ci-config
-    if: needs.ci-config.outputs.enabled == 'yes'
+    if: github.event.repository.owner.login == 'git-for-windows' && needs.ci-config.outputs.enabled == 'yes'
     env:
       NO_PERL: 1
       GIT_CONFIG_PARAMETERS: "'user.name=CI' 'user.email=ci@git'"