| Message ID | pull.632.git.1589316430595.gitgitgadget@gmail.com (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
| Series | ci: avoid pounding on the poor ci-artifacts container | expand |
"Johannes Schindelin via GitGitGadget" <gitgitgadget@gmail.com> writes: > Let's switch back to using the Build Artifacts of our trusty Azure > Pipeline for the time being. > > To avoid unnecessary hammering of the Azure Pipeline artifacts, we use > the GitHub Action `actions/upload-artifact` in the `windows-build` job > and the GitHub Action `actions/download-artifact` in the `windows-test` > and `vs-test` jobs (the latter now depends on `windows-build` for that > reason, too). I guess this answers a question I sent earlier to the list (our mails almost crossed, I guess, as two of us were looking at the same problem at around the same time?). Hopefully when cmake-for-windows-build topic lands, this can go away altogether, but that is probably at least 8 weeks away (3 weeks remaining before the next cycle opens, plus a half of 10 week per cycle for a typical major release). Today's final integration (these days I'm pushing out twice or three times a day) contains this one, and it seems to have passed ;-) Thanks.
On 2020-05-12 20:47:10+0000, Johannes Schindelin via GitGitGadget <gitgitgadget@gmail.com> wrote: > - run: a=git-sdk-64-minimal && mkdir -p $a && curl -# https://wingit.blob.core.windows.net/ci-artifacts/$a.tar.xz | tar -C $a -xJf - > + run: | > + ## Add `json_pp` to the search path > + PATH=$PATH:/usr/bin/core_perl > + > + ## Get artifact > + urlbase=https://dev.azure.com/git-for-windows/git/_apis/build/builds > + id=$(curl "$urlbase?definitions=22&statusFilter=completed&resultFilter=succeeded&\$top=1" | > + json_pp | > + sed -n 's/^ "id" : \([1-9][0-9]*\).*/\1/p') > + download_url="$(curl "$urlbase/$id/artifacts" | > + json_pp | > + sed -n '/^ {/{:1;N;/\n }/b2;b1;:2;/"name" : "git-sdk-64-minimal"/{s/.*"downloadUrl" : "\([^\"]*\).*/\1/p}}')" Hi Dscho, I wonder if it's acceptable to introduce jq (already installed in GitHub Actions and Travis) into our codebase (only in GitHub Actions). If yes, I think this will be easier to follow than depending on static number of space and sed branching. ---------------------8<----------------- diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml index e2e1611aa2..482df46651 100644 --- a/.github/workflows/main.yml +++ b/.github/workflows/main.yml @@ -13,17 +13,12 @@ jobs: - name: download git-sdk-64-minimal shell: bash run: | - ## Add `json_pp` to the search path - PATH=$PATH:/usr/bin/core_perl - ## Get artifact urlbase=https://dev.azure.com/git-for-windows/git/_apis/build/builds id=$(curl "$urlbase?definitions=22&statusFilter=completed&resultFilter=succeeded&\$top=1" | - json_pp | - sed -n 's/^ "id" : \([1-9][0-9]*\).*/\1/p') + jq -r ".value[] | .id") download_url="$(curl "$urlbase/$id/artifacts" | - json_pp | - sed -n '/^ {/{:1;N;/\n }/b2;b1;:2;/"name" : "git-sdk-64-minimal"/{s/.*"downloadUrl" : "\([^\"]*\).*/\1/p}}')" + jq -r '.value[] | select(.name == "git-sdk-64-minimal").resource.downloadUrl')" curl --connect-timeout 10 --retry 5 --retry-delay 0 --retry-max-time 240 \ -o artifacts.zip "$download_url" @@ -104,17 +99,12 @@ jobs: - name: download git-sdk-64-minimal shell: bash run: | - ## Add `json_pp` to the search path - PATH=$PATH:/usr/bin/core_perl - ## Get artifact urlbase=https://dev.azure.com/git-for-windows/git/_apis/build/builds id=$(curl "$urlbase?definitions=22&statusFilter=completed&resultFilter=succeeded&\$top=1" | - json_pp | - sed -n 's/^ "id" : \([1-9][0-9]*\).*/\1/p') + jq -r ".value[] | .id") download_url="$(curl "$urlbase/$id/artifacts" | - json_pp | - sed -n '/^ {/{:1;N;/\n }/b2;b1;:2;/"name" : "git-sdk-64-minimal"/{s/.*"downloadUrl" : "\([^\"]*\).*/\1/p}}')" + jq -r '.value[] | select(.name == "git-sdk-64-minimal").resource.downloadUrl')" curl --connect-timeout 10 --retry 5 --retry-delay 0 --retry-max-time 240 \ -o artifacts.zip "$download_url" ----------->8------------
Hi Junio, On Tue, 12 May 2020, Junio C Hamano wrote: > "Johannes Schindelin via GitGitGadget" <gitgitgadget@gmail.com> > writes: > > > Let's switch back to using the Build Artifacts of our trusty Azure > > Pipeline for the time being. > > > > To avoid unnecessary hammering of the Azure Pipeline artifacts, we use > > the GitHub Action `actions/upload-artifact` in the `windows-build` job > > and the GitHub Action `actions/download-artifact` in the `windows-test` > > and `vs-test` jobs (the latter now depends on `windows-build` for that > > reason, too). > > I guess this answers a question I sent earlier to the list (our > mails almost crossed, I guess, as two of us were looking at the same > problem at around the same time?). I am terribly sorry, but I did not get to read the Git mailing list at all this week (or for that matter, my private mail). So I would not even have seen your message... :-( > Hopefully when cmake-for-windows-build topic lands, this can go away > altogether, but that is probably at least 8 weeks away (3 weeks > remaining before the next cycle opens, plus a half of 10 week per > cycle for a typical major release). The `cmake-for-windows-build` would address only the build part for Visual Studio. The regular Windows build, as well as the parallelized tests _still_ need the `git-sdk-64-minimal` artifact. With or without CMake. That's because neither CMake nor Visual Studio can accommodate the fact that our test suite is implemented in shell script _and_ requires a working Perl interpreter. > Today's final integration (these days I'm pushing out twice or three > times a day) contains this one, and it seems to have passed ;-) Excellent! Thanks, Dscho
Hi Danh, On Wed, 13 May 2020, Đoàn Trần Công Danh wrote: > On 2020-05-12 20:47:10+0000, Johannes Schindelin via GitGitGadget <gitgitgadget@gmail.com> wrote: > > - run: a=git-sdk-64-minimal && mkdir -p $a && curl -# https://wingit.blob.core.windows.net/ci-artifacts/$a.tar.xz | tar -C $a -xJf - > > + run: | > > + ## Add `json_pp` to the search path > > + PATH=$PATH:/usr/bin/core_perl > > + > > + ## Get artifact > > + urlbase=https://dev.azure.com/git-for-windows/git/_apis/build/builds > > + id=$(curl "$urlbase?definitions=22&statusFilter=completed&resultFilter=succeeded&\$top=1" | > > + json_pp | > > + sed -n 's/^ "id" : \([1-9][0-9]*\).*/\1/p') > > + download_url="$(curl "$urlbase/$id/artifacts" | > > + json_pp | > > + sed -n '/^ {/{:1;N;/\n }/b2;b1;:2;/"name" : "git-sdk-64-minimal"/{s/.*"downloadUrl" : "\([^\"]*\).*/\1/p}}')" > > Hi Dscho, > > I wonder if it's acceptable to introduce jq (already installed in > GitHub Actions and Travis) into our codebase (only in GitHub Actions). > > If yes, I think this will be easier to follow than depending on static > number of space and sed branching. > ---------------------8<----------------- > diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml > index e2e1611aa2..482df46651 100644 > --- a/.github/workflows/main.yml > +++ b/.github/workflows/main.yml > @@ -13,17 +13,12 @@ jobs: > - name: download git-sdk-64-minimal > shell: bash > run: | > - ## Add `json_pp` to the search path > - PATH=$PATH:/usr/bin/core_perl > - > ## Get artifact > urlbase=https://dev.azure.com/git-for-windows/git/_apis/build/builds > id=$(curl "$urlbase?definitions=22&statusFilter=completed&resultFilter=succeeded&\$top=1" | > - json_pp | > - sed -n 's/^ "id" : \([1-9][0-9]*\).*/\1/p') > + jq -r ".value[] | .id") > download_url="$(curl "$urlbase/$id/artifacts" | > - json_pp | > - sed -n '/^ {/{:1;N;/\n }/b2;b1;:2;/"name" : "git-sdk-64-minimal"/{s/.*"downloadUrl" : "\([^\"]*\).*/\1/p}}')" > + jq -r '.value[] | select(.name == "git-sdk-64-minimal").resource.downloadUrl')" > curl --connect-timeout 10 --retry 5 --retry-delay 0 --retry-max-time 240 \ > -o artifacts.zip "$download_url" > > @@ -104,17 +99,12 @@ jobs: > - name: download git-sdk-64-minimal > shell: bash > run: | > - ## Add `json_pp` to the search path > - PATH=$PATH:/usr/bin/core_perl > - > ## Get artifact > urlbase=https://dev.azure.com/git-for-windows/git/_apis/build/builds > id=$(curl "$urlbase?definitions=22&statusFilter=completed&resultFilter=succeeded&\$top=1" | > - json_pp | > - sed -n 's/^ "id" : \([1-9][0-9]*\).*/\1/p') > + jq -r ".value[] | .id") > download_url="$(curl "$urlbase/$id/artifacts" | > - json_pp | > - sed -n '/^ {/{:1;N;/\n }/b2;b1;:2;/"name" : "git-sdk-64-minimal"/{s/.*"downloadUrl" : "\([^\"]*\).*/\1/p}}')" > + jq -r '.value[] | select(.name == "git-sdk-64-minimal").resource.downloadUrl')" > curl --connect-timeout 10 --retry 5 --retry-delay 0 --retry-max-time 240 \ > -o artifacts.zip "$download_url" > > ----------->8------------ Thank you for that. Indeed, I should have checked whether `jq` is available on the Windows build agents (and then I should have learned how to hold that tool right). I sent out a v2 with this change, and I already merged that version into Git for Windows' `master` branch in preparation for v2.27.0-rc0. Thanks, Dscho
diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml index fd4df939b50..e2e1611aa22 100644 --- a/.github/workflows/main.yml +++ b/.github/workflows/main.yml @@ -12,7 +12,24 @@ jobs: - uses: actions/checkout@v1 - name: download git-sdk-64-minimal shell: bash - run: a=git-sdk-64-minimal && mkdir -p $a && curl -# https://wingit.blob.core.windows.net/ci-artifacts/$a.tar.xz | tar -C $a -xJf - + run: | + ## Add `json_pp` to the search path + PATH=$PATH:/usr/bin/core_perl + + ## Get artifact + urlbase=https://dev.azure.com/git-for-windows/git/_apis/build/builds + id=$(curl "$urlbase?definitions=22&statusFilter=completed&resultFilter=succeeded&\$top=1" | + json_pp | + sed -n 's/^ "id" : \([1-9][0-9]*\).*/\1/p') + download_url="$(curl "$urlbase/$id/artifacts" | + json_pp | + sed -n '/^ {/{:1;N;/\n }/b2;b1;:2;/"name" : "git-sdk-64-minimal"/{s/.*"downloadUrl" : "\([^\"]*\).*/\1/p}}')" + curl --connect-timeout 10 --retry 5 --retry-delay 0 --retry-max-time 240 \ + -o artifacts.zip "$download_url" + + ## Unzip and remove the artifact + unzip artifacts.zip + rm artifacts.zip - name: build shell: powershell env: @@ -30,6 +47,11 @@ jobs: with: name: windows-artifacts path: artifacts + - name: upload git-sdk-64-minimal + uses: actions/upload-artifact@v1 + with: + name: git-sdk-64-minimal + path: git-sdk-64-minimal windows-test: runs-on: windows-latest needs: [windows-build] @@ -38,9 +60,6 @@ jobs: nr: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] steps: - uses: actions/checkout@v1 - - name: download git-sdk-64-minimal - shell: bash - run: a=git-sdk-64-minimal && mkdir -p $a && curl -# https://wingit.blob.core.windows.net/ci-artifacts/$a.tar.xz | tar -C $a -xJf - - name: download build artifacts uses: actions/download-artifact@v1 with: @@ -49,6 +68,11 @@ jobs: - name: extract build artifacts shell: bash run: tar xf artifacts.tar.gz + - name: download git-sdk-64-minimal + uses: actions/download-artifact@v1 + with: + name: git-sdk-64-minimal + path: ${{github.workspace}}/git-sdk-64-minimal/ - name: test shell: powershell run: | @@ -79,7 +103,24 @@ jobs: - uses: actions/checkout@v1 - name: download git-sdk-64-minimal shell: bash - run: a=git-sdk-64-minimal && mkdir -p $a && curl -# https://wingit.blob.core.windows.net/ci-artifacts/$a.tar.xz | tar -C $a -xJf - + run: | + ## Add `json_pp` to the search path + PATH=$PATH:/usr/bin/core_perl + + ## Get artifact + urlbase=https://dev.azure.com/git-for-windows/git/_apis/build/builds + id=$(curl "$urlbase?definitions=22&statusFilter=completed&resultFilter=succeeded&\$top=1" | + json_pp | + sed -n 's/^ "id" : \([1-9][0-9]*\).*/\1/p') + download_url="$(curl "$urlbase/$id/artifacts" | + json_pp | + sed -n '/^ {/{:1;N;/\n }/b2;b1;:2;/"name" : "git-sdk-64-minimal"/{s/.*"downloadUrl" : "\([^\"]*\).*/\1/p}}')" + curl --connect-timeout 10 --retry 5 --retry-delay 0 --retry-max-time 240 \ + -o artifacts.zip "$download_url" + + ## Unzip and remove the artifact + unzip artifacts.zip + rm artifacts.zip - name: generate Visual Studio solution shell: powershell run: | @@ -119,15 +160,17 @@ jobs: path: artifacts vs-test: runs-on: windows-latest - needs: [vs-build] + needs: [vs-build, windows-build] strategy: matrix: nr: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] steps: - uses: actions/checkout@v1 - - name: download git-64-portable - shell: bash - run: a=git-64-portable && mkdir -p $a && curl -# https://wingit.blob.core.windows.net/ci-artifacts/$a.tar.xz | tar -C $a -xJf - + - name: download git-sdk-64-minimal + uses: actions/download-artifact@v1 + with: + name: git-sdk-64-minimal + path: ${{github.workspace}}/git-sdk-64-minimal/ - name: download build artifacts uses: actions/download-artifact@v1 with: @@ -143,9 +186,9 @@ jobs: NO_SVN_TESTS: 1 GIT_TEST_SKIP_REBASE_P: 1 run: | - & git-64-portable\git-cmd.exe --command=usr\bin\bash.exe -lc @" + & .\git-sdk-64-minimal\usr\bin\bash.exe -lc @" # Let Git ignore the SDK and the test-cache - printf '%s\n' /git-64-portable/ /test-cache/ >>.git/info/exclude + printf '%s\n' /git-sdk-64-minimal/ /test-cache/ >>.git/info/exclude cd t && PATH=\"`$PWD/helper:`$PATH\" &&