diff mbox series

[v3,2/3] cirrus: Fixing and speedup the msys2/mingw CI

Message ID 20201007035154.109-3-luoyonggang@gmail.com (mailing list archive)
State New, archived
Headers show
Series Improve cirrus msys2 | expand

Commit Message

Yonggang Luo Oct. 7, 2020, 3:51 a.m. UTC
Using a prepared https://github.com/lygstate/qemu/releases/download/v5.1.0/msys2-x86_64.tar.xz
for speed up the msys2/mingw prepare speed on the cirrus CI.
This file are generated by the msys2.docker

Signed-off-by: Yonggang Luo <luoyonggang@gmail.com>
---
 .cirrus.yml | 60 ++++++++++++++---------------------------------------
 1 file changed, 15 insertions(+), 45 deletions(-)

Comments

Paolo Bonzini Oct. 7, 2020, 8:18 a.m. UTC | #1
On 07/10/20 05:51, Yonggang Luo wrote:
> Using a prepared https://github.com/lygstate/qemu/releases/download/v5.1.0/msys2-x86_64.tar.xz
> for speed up the msys2/mingw prepare speed on the cirrus CI.
> This file are generated by the msys2.docker

The speedup is nice, but we need to be able to regenerate the tarball
without access to a Windows machine.  Otherwise whoever adds a new
dependency cannot test it himself with CI.

Ideally, the tarball would be generated by the containers job in the
Gitlab CI, but I'm not sure if that's possible.

On top of this, whoever commits this needs to replace the URL with
something under download.qemu.org, but that's the easy part.

Paolo
Daniel P. Berrangé Oct. 7, 2020, 8:32 a.m. UTC | #2
On Wed, Oct 07, 2020 at 10:18:48AM +0200, Paolo Bonzini wrote:
> On 07/10/20 05:51, Yonggang Luo wrote:
> > Using a prepared https://github.com/lygstate/qemu/releases/download/v5.1.0/msys2-x86_64.tar.xz
> > for speed up the msys2/mingw prepare speed on the cirrus CI.
> > This file are generated by the msys2.docker
> 
> The speedup is nice, but we need to be able to regenerate the tarball
> without access to a Windows machine.  Otherwise whoever adds a new
> dependency cannot test it himself with CI.
> 
> Ideally, the tarball would be generated by the containers job in the
> Gitlab CI, but I'm not sure if that's possible.

IMHO any caching should be integrated into the Cirrus CI job itself.
Cirrus has builtin support for caching the contents of a folder, so
the first time the job runs it should populate the cache, and that
can be used thereafter.

Regards,
Daniel
Thomas Huth Oct. 7, 2020, 8:41 a.m. UTC | #3
On 07/10/2020 10.18, Paolo Bonzini wrote:
[...]
> 
> On top of this, whoever commits this needs to replace the URL with
> something under download.qemu.org, but that's the easy part.

I agree with Daniel - please let's not get go this way again. We've had this
with the various Free-/Net-/OpenBSD images in the past, and then we were
suddenly stuck with a fixed image where most people were not able to change
it anymore. That was very frustrating.

Apart from that, you also have to respect the GPL if you provide binaries
... so you'd need to gather all the sources for that blob, too, and be ready
to provide those in case somebody asks...

Yonggang, could you please try to use the caching mechanism of Cirrus-CI
instead? See https://cirrus-ci.org/guide/writing-tasks/#cache-instruction
... then we're a little bit more independent from downloading the msys2 for
each CI run.

 Thomas
Yonggang Luo Oct. 7, 2020, 8:54 a.m. UTC | #4
On Wed, Oct 7, 2020 at 4:41 PM Thomas Huth <thuth@redhat.com> wrote:
>
> On 07/10/2020 10.18, Paolo Bonzini wrote:
> [...]
> >
> > On top of this, whoever commits this needs to replace the URL with
> > something under download.qemu.org, but that's the easy part.
>
> I agree with Daniel - please let's not get go this way again. We've had
this
> with the various Free-/Net-/OpenBSD images in the past, and then we were
> suddenly stuck with a fixed image where most people were not able to
change
> it anymore. That was very frustrating.
>
> Apart from that, you also have to respect the GPL if you provide binaries
> ... so you'd need to gather all the sources for that blob, too, and be
ready
> to provide those in case somebody asks...
>
> Yonggang, could you please try to use the caching mechanism of Cirrus-CI
> instead? See https://cirrus-ci.org/guide/writing-tasks/#cache-instruction
> ... then we're a little bit more independent from downloading the msys2
for
> each CI run.
Looks good for me, let's me investigate
>
>  Thomas
>


--
         此致
礼
罗勇刚
Yours
    sincerely,
Yonggang Luo
diff mbox series

Patch

diff --git a/.cirrus.yml b/.cirrus.yml
index d58782ce67..bf5a0990ec 100644
--- a/.cirrus.yml
+++ b/.cirrus.yml
@@ -47,63 +47,33 @@  macos_xcode_task:
 
 windows_msys2_task:
   windows_container:
-    image: cirrusci/windowsservercore:cmake
+    image: cirrusci/windowsservercore:2019
     os_version: 2019
     cpu: 8
     memory: 8G
   env:
+    CIRRUS_SHELL: powershell
     MSYS: winsymlinks:nativestrict
     MSYSTEM: MINGW64
     CHERE_INVOKING: 1
-  printenv_script:
-    - C:\tools\msys64\usr\bin\bash.exe -lc 'printenv'
   install_script:
-    - C:\tools\msys64\usr\bin\bash.exe -lc "cd /c/tools &&
-        curl -O http://repo.msys2.org/msys/x86_64/msys2-keyring-r21.b39fb11-1-any.pkg.tar.xz"
-    - C:\tools\msys64\usr\bin\bash.exe -lc "cd /c/tools &&
-        curl -O http://repo.msys2.org/msys/x86_64/msys2-keyring-r21.b39fb11-1-any.pkg.tar.xz.sig"
-    - C:\tools\msys64\usr\bin\bash.exe -lc "cd /c/tools &&
-        pacman -U --noconfirm msys2-keyring-r21.b39fb11-1-any.pkg.tar.xz"
-    - C:\tools\msys64\usr\bin\bash.exe -lc "pacman --noconfirm -Sy"
-    - C:\tools\msys64\usr\bin\bash.exe -lc "pacman --noconfirm -S --needed
-        bash pacman pacman-mirrors msys2-runtime"
-    - taskkill /F /IM gpg-agent.exe
-    - C:\tools\msys64\usr\bin\bash.exe -lc "pacman --noconfirm -Su"
-    - C:\tools\msys64\usr\bin\bash.exe -lc "pacman --noconfirm -S --needed
-        base-devel
-        git
-        mingw-w64-x86_64-python
-        mingw-w64-x86_64-python-setuptools
-        mingw-w64-x86_64-toolchain
-        mingw-w64-x86_64-capstone
-        mingw-w64-x86_64-SDL2
-        mingw-w64-x86_64-SDL2_image
-        mingw-w64-x86_64-gtk3
-        mingw-w64-x86_64-glib2
-        mingw-w64-x86_64-ninja
-        mingw-w64-x86_64-make
-        mingw-w64-x86_64-lzo2
-        mingw-w64-x86_64-zstd
-        mingw-w64-x86_64-libjpeg-turbo
-        mingw-w64-x86_64-pixman
-        mingw-w64-x86_64-libgcrypt
-        mingw-w64-x86_64-libpng
-        mingw-w64-x86_64-libssh
-        mingw-w64-x86_64-libxml2
-        mingw-w64-x86_64-snappy
-        mingw-w64-x86_64-libusb
-        mingw-w64-x86_64-usbredir
-        mingw-w64-x86_64-libtasn1
-        mingw-w64-x86_64-nettle
-        mingw-w64-x86_64-cyrus-sasl
-        mingw-w64-x86_64-curl
-        mingw-w64-x86_64-gnutls
-        mingw-w64-x86_64-zstd"
+    - |
+      Write-Output $env:PATH
+      md C:\tools
+      $start_time = Get-Date
+      bitsadmin /transfer msys_download /dynamic /download /priority FOREGROUND https://github.com/lygstate/qemu/releases/download/v5.1.0/msys2-x86_64.tar.xz C:\tools\msys2-x86_64.tar.xz
+      Write-Output "Download time taken: $((Get-Date).Subtract($start_time).Seconds) second(s)"
+      $start_time = Get-Date
+      cd C:\tools
+      choco install -y --no-progress 7zip
+      cmd /C "7z x msys2-x86_64.tar.xz -so | 7z x -aoa -simsys2-x86_64.tar -ttar -omsys64"
+      Write-Output "Extract time taken: $((Get-Date).Subtract($start_time).Seconds) second(s)"
+
   script:
     - C:\tools\msys64\usr\bin\bash.exe -lc "mkdir build"
     - C:\tools\msys64\usr\bin\bash.exe -lc "cd build && ../configure
       --python=python3 --ninja=ninja
       --target-list-exclude=i386-softmmu,arm-softmmu,ppc-softmmu,mips-softmmu"
-    - C:\tools\msys64\usr\bin\bash.exe -lc "cd build && make -j$NUMBER_OF_PROCESSORS"
+    - C:\tools\msys64\usr\bin\bash.exe -lc "cd build && make -j8"
   test_script:
     - C:\tools\msys64\usr\bin\bash.exe -lc "cd build && make V=1 check"