| Message ID | c7bd8aa95e8bd85f3e42ed4f07d69444f5435e25.1744403499.git-series.marmarek@invisiblethingslab.com (mailing list archive) |
|---|---|
| State | New |
| Headers | show |
| Series | Several CI cleanups and improvements around initrd/rootfs | expand |
On 11/04/2025 9:32 pm, Marek Marczykowski-Górecki wrote: > The prerequisite for this is building rootfs.cpio.gz, not only > initrd.tar.gz. That's a change in the test-artifacts repository in the > "Rootfs generation" series from Andrew. > > Having that, do not repack the whole initrd, but only pack modified > files and rely on Linux handling of concatenated archives. > This allows packing just test-related files (which includes the whole > toolstack), instead of the whole initrd. > > DomU initrd handling is a bit more complicated thing. It's sent to the > target host as part of the dom0 initrd. For now include prepared domU > initrd as a whole in dom0's rootfs "overlay", which means compressing it > (again) each job (this takes 3s). This can be later improved by > splitting domU initrd into two parts (base + overlay) and concatenate > them in target dom0 already. For this to be useful, test-artifacts would > need to provide pre-made double-cpio version too (rootfs.cpio.gz wrapper > in yet another cpio.gz as boot/domU-initrd). > > Since full initrd is not unpacked now when preparing domU (and dom0) > rootfs, a couple of minor changes are needed to how they are prepared. > This includes creating whole etc/issue file, instead of modifying > existing one, and a need to create a couple directories. > > Finally, move adding "rc_verbose=yes" to /etc/rc.conf to initrd > building. > > Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com> > --- > Changes in v3: > - adjust for Andrew's version of cpio rootfs artifacts > - don't require double-cpio archive for domU initrd, instead add it via > dom0 initrd as plain cpio to avoid re-compressing the same thing > New in v2 > > Jobs xilinx-* are untested, as I don't have necessary access. I can test these. > --- > automation/gitlab-ci/test.yaml | 6 +++-- > automation/scripts/qemu-alpine-x86_64.sh | 9 +++----- > automation/scripts/qemu-smoke-dom0-arm64.sh | 7 ++---- > automation/scripts/qemu-smoke-dom0less-arm64.sh | 7 ++---- > automation/scripts/qubes-x86-64.sh | 18 +++++++--------- > automation/scripts/xilinx-smoke-dom0-x86_64.sh | 15 ++++++------- > automation/scripts/xilinx-smoke-dom0less-arm64.sh | 14 +++++------- > 7 files changed, 35 insertions(+), 41 deletions(-) > > diff --git a/automation/gitlab-ci/test.yaml b/automation/gitlab-ci/test.yaml > index 59a2de28c864..3e02bf0e4d21 100644 > --- a/automation/gitlab-ci/test.yaml > +++ b/automation/gitlab-ci/test.yaml > @@ -11,7 +11,9 @@ > - project: xen-project/hardware/test-artifacts > job: linux-6.6.86-arm64 > ref: master > - - alpine-3.18-arm64-rootfs-export > + - project: xen-project/hardware/test-artifacts > + job: alpine-3.18-arm64-rootfs > + ref: master > - qemu-system-aarch64-6.0.0-arm64-export > > .arm32-test-needs: &arm32-test-needs > @@ -22,7 +24,7 @@ > job: linux-6.6.56-x86_64 > ref: master > - project: xen-project/hardware/test-artifacts > - job: x86_64-rootfs-alpine-3.18 > + job: alpine-3.18-x86_64-rootfs > ref: master > > .qemu-arm64: > diff --git a/automation/scripts/qemu-alpine-x86_64.sh b/automation/scripts/qemu-alpine-x86_64.sh > index 89bdb4df7dac..6e0a67b4018c 100755 > --- a/automation/scripts/qemu-alpine-x86_64.sh > +++ b/automation/scripts/qemu-alpine-x86_64.sh There are some very poor choices of name in this script. I'll do a prep patch to make them consistent. > @@ -28,16 +28,15 @@ cd initrd > find . | cpio --create --format='newc' | gzip > ../initrd.cpio.gz > cd .. > > -# initrd.tar.gz is Dom0 rootfs > +# Dom0 rootfs - overlay on top of rootfs.cpio.gz > mkdir -p rootfs > cd rootfs > -tar xvzf ../initrd.tar.gz > mkdir proc > mkdir run > mkdir srv > mkdir sys > -rm var/run I'm not sure why we need to make proc, run, srv, sys in the overlay image, but this is repeated everywhere. srv is just a plain empty dir. It's missing from $PATHS in the rootfs generation, so that's easy. proc, run and sys are more complicated, because we don't want to simply copy them from the rootfs generation environment. dev is also in the same boat, and what we do now seems to cause openrc not to seed it normally, which seems unwise. I'll see what I can do to clean this all up. > diff --git a/automation/scripts/xilinx-smoke-dom0-x86_64.sh b/automation/scripts/xilinx-smoke-dom0-x86_64.sh > index 4db249530823..52a0e05e3f1b 100755 > --- a/automation/scripts/xilinx-smoke-dom0-x86_64.sh > +++ b/automation/scripts/xilinx-smoke-dom0-x86_64.sh > @@ -80,6 +80,7 @@ argo-exec -l -p 28333 -- /bin/echo > " > copy_dom0_files () > { > + mkdir -p root usr/local/lib > cp "${WORKDIR}/binaries/xen-argo.ko" "root/" > cp -ar "${WORKDIR}/binaries/lib/"* "usr/local/lib/" > cp "${WORKDIR}/binaries/argo-exec" "usr/local/bin/" This would be better based on top of my patch to switch to the new argo, at which point it will simply be to append argo.cpio.gz. ~Andrew
On Sat, Apr 12, 2025 at 03:05:13PM +0100, Andrew Cooper wrote: > On 11/04/2025 9:32 pm, Marek Marczykowski-Górecki wrote: > > @@ -28,16 +28,15 @@ cd initrd > > find . | cpio --create --format='newc' | gzip > ../initrd.cpio.gz > > cd .. > > > > -# initrd.tar.gz is Dom0 rootfs > > +# Dom0 rootfs - overlay on top of rootfs.cpio.gz > > mkdir -p rootfs > > cd rootfs > > -tar xvzf ../initrd.tar.gz > > mkdir proc > > mkdir run > > mkdir srv > > mkdir sys > > -rm var/run > > I'm not sure why we need to make proc, run, srv, sys in the overlay > image, but this is repeated everywhere. > > srv is just a plain empty dir. It's missing from $PATHS in the rootfs > generation, so that's easy. srv is indeed likely not needed, but there is enough changes here already. > proc, run and sys are more complicated, because we don't want to simply > copy them from the rootfs generation environment. As for those mountpoints, they looks to be necessary - in a job that failed to unpack this overlay part, openrc failed to mount those filesystems and crashed rather badly (especially without /run...). I'm not sure if that's something that is missing in the base archive, or is a problem for another reason, but it seems to be necessary to have it here, at least without changes elsewhere. Looking at PATHS in alpine-rootfs.sh they may be simply skipped there. > dev is also in the same boat, and what we do now seems to cause openrc > not to seed it normally, which seems unwise. It mounts devtmpfs there anyway, no? > I'll see what I can do to clean this all up. > > > diff --git a/automation/scripts/xilinx-smoke-dom0-x86_64.sh b/automation/scripts/xilinx-smoke-dom0-x86_64.sh > > index 4db249530823..52a0e05e3f1b 100755 > > --- a/automation/scripts/xilinx-smoke-dom0-x86_64.sh > > +++ b/automation/scripts/xilinx-smoke-dom0-x86_64.sh > > @@ -80,6 +80,7 @@ argo-exec -l -p 28333 -- /bin/echo > > " > > copy_dom0_files () > > { > > + mkdir -p root usr/local/lib > > cp "${WORKDIR}/binaries/xen-argo.ko" "root/" > > cp -ar "${WORKDIR}/binaries/lib/"* "usr/local/lib/" > > cp "${WORKDIR}/binaries/argo-exec" "usr/local/bin/" > > This would be better based on top of my patch to switch to the new argo, > at which point it will simply be to append argo.cpio.gz. > > ~Andrew
On 11/04/2025 9:32 pm, Marek Marczykowski-Górecki wrote: > The prerequisite for this is building rootfs.cpio.gz, not only > initrd.tar.gz. That's a change in the test-artifacts repository in the > "Rootfs generation" series from Andrew. > > Having that, do not repack the whole initrd, but only pack modified > files and rely on Linux handling of concatenated archives. > This allows packing just test-related files (which includes the whole > toolstack), instead of the whole initrd. It's probably worth saying ", fixed subsequently)" for the toolstack, as it's the single largest aspect. > DomU initrd handling is a bit more complicated thing. It's sent to the > target host as part of the dom0 initrd. For now include prepared domU > initrd as a whole in dom0's rootfs "overlay", which means compressing it > (again) each job (this takes 3s). This can be later improved by > splitting domU initrd into two parts (base + overlay) and concatenate > them in target dom0 already. For this to be useful, test-artifacts would > need to provide pre-made double-cpio version too (rootfs.cpio.gz wrapper > in yet another cpio.gz as boot/domU-initrd). > > Since full initrd is not unpacked now when preparing domU (and dom0) > rootfs, a couple of minor changes are needed to how they are prepared. > This includes creating whole etc/issue file, instead of modifying > existing one, and a need to create a couple directories. > > Finally, move adding "rc_verbose=yes" to /etc/rc.conf to initrd > building. These two paragraphs are a little stale now. etc/issue has already been committed, and there's no initd building now I've moved arm into test artefacts. However, they're going to need tweaking again if I can get the rootfs generation working more nicely, so lets hold off for now. > > Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com> > --- > Changes in v3: > - adjust for Andrew's version of cpio rootfs artifacts > - don't require double-cpio archive for domU initrd, instead add it via > dom0 initrd as plain cpio to avoid re-compressing the same thing > New in v2 > > Jobs xilinx-* are untested, as I don't have necessary access. > --- > automation/gitlab-ci/test.yaml | 6 +++-- > automation/scripts/qemu-alpine-x86_64.sh | 9 +++----- > automation/scripts/qemu-smoke-dom0-arm64.sh | 7 ++---- > automation/scripts/qemu-smoke-dom0less-arm64.sh | 7 ++---- > automation/scripts/qubes-x86-64.sh | 18 +++++++--------- > automation/scripts/xilinx-smoke-dom0-x86_64.sh | 15 ++++++------- > automation/scripts/xilinx-smoke-dom0less-arm64.sh | 14 +++++------- > 7 files changed, 35 insertions(+), 41 deletions(-) One thing that does matter now is the order of appending archives. I've reworked this locally to be of the form: # DomU rootfs cp ../binaries/rootfs.cpio.gz ../binaries/domU-rootfs.cpio.gz # test-local configuration mkdir rootfs ... find . | cpio -H newc -o | gzip >> ../binaries/domU-rootfs.cpio.gz This then ends up looking rather more clean in followon patches, where we end up with: # Dom0 rootfs cp ../binaries/ucode.cpio ../binaries/dom0-rootfs.cpio.gz cat ../binaries/rootfs.cpio.gz >> ../binaries/dom0-rootfs.cpio.gz cat ../binaries/toolstack.cpio.gz >> ../binaries/dom0-rootfs.cpio.gz if [[ "${TEST}" == argo ]]; then cat ../binaries/argo.cpio.gz >> ../binaries/dom0-rootfs.cpio.gz fi # test-local configuration ... find . | cpio -H newc -o | gzip >> ../binaries/dom0-rootfs.cpio.gz > diff --git a/automation/scripts/xilinx-smoke-dom0-x86_64.sh b/automation/scripts/xilinx-smoke-dom0-x86_64.sh > index 4db249530823..52a0e05e3f1b 100755 > --- a/automation/scripts/xilinx-smoke-dom0-x86_64.sh > +++ b/automation/scripts/xilinx-smoke-dom0-x86_64.sh > @@ -80,6 +80,7 @@ argo-exec -l -p 28333 -- /bin/echo > " > copy_dom0_files () > { > + mkdir -p root usr/local/lib > cp "${WORKDIR}/binaries/xen-argo.ko" "root/" > cp -ar "${WORKDIR}/binaries/lib/"* "usr/local/lib/" > cp "${WORKDIR}/binaries/argo-exec" "usr/local/bin/" > @@ -92,9 +93,8 @@ fi > # Set up domU rootfs. > mkdir -p rootfs > cd rootfs > -tar xzf ../binaries/initrd.tar.gz > mkdir proc run srv sys > -rm var/run > +mkdir -p etc/local.d > echo "#!/bin/sh > set -x > export LD_LIBRARY_PATH=/usr/local/lib > @@ -102,23 +102,22 @@ PATH=/usr/local/bin:/usr/local/sbin:\$PATH > ${DOMU_CMD} > " > etc/local.d/xen.start > chmod +x etc/local.d/xen.start > -echo "rc_verbose=yes" >> etc/rc.conf > echo "domU Welcome to Alpine Linux 3.18 > Kernel \r on an \m (\l) > > " > etc/issue > copy_domU_files > -find . | cpio -H newc -o | gzip > ../binaries/domU-rootfs.cpio.gz > +cp ../binaries/initrd.cpio.gz ../binaries/domU-rootfs.cpio.gz Stray initrd. Found while reworking. I'll give the whole lot commit-by-commit full runs, when I'm done tinkering. ~Andrew
diff --git a/automation/gitlab-ci/test.yaml b/automation/gitlab-ci/test.yaml index 59a2de28c864..3e02bf0e4d21 100644 --- a/automation/gitlab-ci/test.yaml +++ b/automation/gitlab-ci/test.yaml @@ -11,7 +11,9 @@ - project: xen-project/hardware/test-artifacts job: linux-6.6.86-arm64 ref: master - - alpine-3.18-arm64-rootfs-export + - project: xen-project/hardware/test-artifacts + job: alpine-3.18-arm64-rootfs + ref: master - qemu-system-aarch64-6.0.0-arm64-export .arm32-test-needs: &arm32-test-needs @@ -22,7 +24,7 @@ job: linux-6.6.56-x86_64 ref: master - project: xen-project/hardware/test-artifacts - job: x86_64-rootfs-alpine-3.18 + job: alpine-3.18-x86_64-rootfs ref: master .qemu-arm64: diff --git a/automation/scripts/qemu-alpine-x86_64.sh b/automation/scripts/qemu-alpine-x86_64.sh index 89bdb4df7dac..6e0a67b4018c 100755 --- a/automation/scripts/qemu-alpine-x86_64.sh +++ b/automation/scripts/qemu-alpine-x86_64.sh @@ -28,16 +28,15 @@ cd initrd find . | cpio --create --format='newc' | gzip > ../initrd.cpio.gz cd .. -# initrd.tar.gz is Dom0 rootfs +# Dom0 rootfs - overlay on top of rootfs.cpio.gz mkdir -p rootfs cd rootfs -tar xvzf ../initrd.tar.gz mkdir proc mkdir run mkdir srv mkdir sys -rm var/run cp -ar ../dist/install/* . +mkdir -p root etc/local.d mv ../initrd.cpio.gz ./root cp ../bzImage ./root echo "name=\"test\" @@ -60,9 +59,9 @@ xl -vvv create -c /root/test.cfg " > etc/local.d/xen.start chmod +x etc/local.d/xen.start -echo "rc_verbose=yes" >> etc/rc.conf # rebuild Dom0 rootfs -find . |cpio -H newc -o|gzip > ../xen-rootfs.cpio.gz +cp ../rootfs.cpio.gz ../xen-rootfs.cpio.gz +find . |cpio -H newc -o|gzip >> ../xen-rootfs.cpio.gz cd ../.. cat >> binaries/pxelinux.0 << EOF diff --git a/automation/scripts/qemu-smoke-dom0-arm64.sh b/automation/scripts/qemu-smoke-dom0-arm64.sh index 51d037b0003e..86047ccd9e7b 100755 --- a/automation/scripts/qemu-smoke-dom0-arm64.sh +++ b/automation/scripts/qemu-smoke-dom0-arm64.sh @@ -29,13 +29,12 @@ cd .. mkdir -p rootfs cd rootfs -tar xvzf ../initrd.tar.gz mkdir proc mkdir run mkdir srv mkdir sys -rm var/run cp -ar ../dist/install/* . +mkdir -p etc/local.d root mv ../initrd.cpio.gz ./root cp ../Image ./root echo "name=\"test\" @@ -56,8 +55,8 @@ xl -vvv create -c /root/test.cfg " > etc/local.d/xen.start chmod +x etc/local.d/xen.start -echo "rc_verbose=yes" >> etc/rc.conf -find . |cpio -H newc -o|gzip > ../xen-rootfs.cpio.gz +cp ../rootfs.cpio.gz ../xen-rootfs.cpio.gz +find . |cpio -H newc -o|gzip >> ../xen-rootfs.cpio.gz cd ../.. # XXX QEMU looks for "efi-virtio.rom" even if it is unneeded diff --git a/automation/scripts/qemu-smoke-dom0less-arm64.sh b/automation/scripts/qemu-smoke-dom0less-arm64.sh index f72d20936181..b095a5008e30 100755 --- a/automation/scripts/qemu-smoke-dom0less-arm64.sh +++ b/automation/scripts/qemu-smoke-dom0less-arm64.sh @@ -117,13 +117,12 @@ cd .. # DOM0 rootfs mkdir -p rootfs cd rootfs -tar xzf ../binaries/initrd.tar.gz mkdir proc mkdir run mkdir srv mkdir sys -rm var/run cp -ar ../binaries/dist/install/* . +mkdir -p etc/local.d echo "#!/bin/bash @@ -142,8 +141,8 @@ xl network-attach 1 type=vif ${dom0_check} " > etc/local.d/xen.start chmod +x etc/local.d/xen.start -echo "rc_verbose=yes" >> etc/rc.conf -find . | cpio -H newc -o | gzip > ../binaries/dom0-rootfs.cpio.gz +cp ../binaries/rootfs.cpio.gz ../binaries/dom0-rootfs.cpio.gz +find . | cpio -H newc -o | gzip >> ../binaries/dom0-rootfs.cpio.gz cd .. # ImageBuilder diff --git a/automation/scripts/qubes-x86-64.sh b/automation/scripts/qubes-x86-64.sh index dd88a1398f2b..99c2ece304f0 100755 --- a/automation/scripts/qubes-x86-64.sh +++ b/automation/scripts/qubes-x86-64.sh @@ -167,13 +167,11 @@ if [ -n "$domU_check" ]; then # DomU mkdir -p rootfs cd rootfs - # fakeroot is needed to preserve device nodes in rootless podman container - fakeroot -s ../fakeroot-save tar xzf ../binaries/initrd.tar.gz mkdir proc mkdir run mkdir srv mkdir sys - rm var/run + mkdir -p etc/local.d echo "#!/bin/sh echo 8 > /proc/sys/kernel/printk @@ -181,26 +179,25 @@ echo 8 > /proc/sys/kernel/printk ${domU_check} " > etc/local.d/xen.start chmod +x etc/local.d/xen.start - echo "rc_verbose=yes" >> etc/rc.conf echo "domU Welcome to Alpine Linux 3.18 Kernel \r on an \m (\l) " > etc/issue - find . | fakeroot -i ../fakeroot-save cpio -H newc -o | gzip > ../binaries/domU-rootfs.cpio.gz + cp ../binaries/rootfs.cpio.gz ../binaries/domU-rootfs.cpio.gz + find . | cpio -H newc -o | gzip >> ../binaries/domU-rootfs.cpio.gz cd .. rm -rf rootfs fi -# DOM0 rootfs +# DOM0 rootfs - this will be an overlay over alpine's initrd mkdir -p rootfs cd rootfs -fakeroot -s ../fakeroot-save tar xzf ../binaries/initrd.tar.gz mkdir boot mkdir proc mkdir run mkdir srv mkdir sys -rm var/run +mkdir -p etc/local.d cp -ar ../binaries/dist/install/* . cp -ar ../binaries/tests . cp -a ../automation/scripts/run-tools-tests tests/ @@ -237,7 +234,6 @@ fi chmod +x etc/local.d/xen.start echo "$domU_config" > etc/xen/domU.cfg -echo "rc_verbose=yes" >> etc/rc.conf echo "XENCONSOLED_TRACE=all" >> etc/default/xencommons echo "QEMU_XEN=/bin/false" >> etc/default/xencommons mkdir -p var/log/xen/console @@ -245,7 +241,9 @@ cp ../binaries/bzImage boot/vmlinuz if [ -n "$domU_check" ]; then cp ../binaries/domU-rootfs.cpio.gz boot/initrd-domU fi -find . | fakeroot -i ../fakeroot-save cpio -H newc -o | gzip > ../binaries/dom0-rootfs.cpio.gz +# take base initrd and append test-specific files +cp ../binaries/rootfs.cpio.gz ../binaries/dom0-rootfs.cpio.gz +find . | cpio -H newc -o | gzip >> ../binaries/dom0-rootfs.cpio.gz cd .. diff --git a/automation/scripts/xilinx-smoke-dom0-x86_64.sh b/automation/scripts/xilinx-smoke-dom0-x86_64.sh index 4db249530823..52a0e05e3f1b 100755 --- a/automation/scripts/xilinx-smoke-dom0-x86_64.sh +++ b/automation/scripts/xilinx-smoke-dom0-x86_64.sh @@ -80,6 +80,7 @@ argo-exec -l -p 28333 -- /bin/echo " copy_dom0_files () { + mkdir -p root usr/local/lib cp "${WORKDIR}/binaries/xen-argo.ko" "root/" cp -ar "${WORKDIR}/binaries/lib/"* "usr/local/lib/" cp "${WORKDIR}/binaries/argo-exec" "usr/local/bin/" @@ -92,9 +93,8 @@ fi # Set up domU rootfs. mkdir -p rootfs cd rootfs -tar xzf ../binaries/initrd.tar.gz mkdir proc run srv sys -rm var/run +mkdir -p etc/local.d echo "#!/bin/sh set -x export LD_LIBRARY_PATH=/usr/local/lib @@ -102,23 +102,22 @@ PATH=/usr/local/bin:/usr/local/sbin:\$PATH ${DOMU_CMD} " > etc/local.d/xen.start chmod +x etc/local.d/xen.start -echo "rc_verbose=yes" >> etc/rc.conf echo "domU Welcome to Alpine Linux 3.18 Kernel \r on an \m (\l) " > etc/issue copy_domU_files -find . | cpio -H newc -o | gzip > ../binaries/domU-rootfs.cpio.gz +cp ../binaries/initrd.cpio.gz ../binaries/domU-rootfs.cpio.gz +find . | cpio -H newc -o | gzip >> ../binaries/domU-rootfs.cpio.gz cd .. rm -rf rootfs # Set up dom0 rootfs. mkdir -p rootfs cd rootfs -tar xzf ../binaries/initrd.tar.gz mkdir boot proc run srv sys -rm var/run cp -ar ../binaries/dist/install/* . +mkdir -p etc/local.d echo "#!/bin/bash set -x export LD_LIBRARY_PATH=/usr/local/lib @@ -127,14 +126,14 @@ ${DOM0_CMD} " > etc/local.d/xen.start chmod +x etc/local.d/xen.start echo "${DOMU_CFG}${DOMU_CFG_EXTRA}" > etc/xen/domU.cfg -echo "rc_verbose=yes" >> etc/rc.conf echo "XENCONSOLED_TRACE=all" >> etc/default/xencommons echo "QEMU_XEN=/bin/false" >> etc/default/xencommons mkdir -p var/log/xen/console cp ../binaries/bzImage boot/vmlinuz cp ../binaries/domU-rootfs.cpio.gz boot/initrd-domU copy_dom0_files -find . | cpio -H newc -o | gzip > ../binaries/dom0-rootfs.cpio.gz +cp ../binaries/rootfs.cpio.gz ../binaries/dom0-rootfs.cpio.gz +find . | cpio -H newc -o | gzip >> ../binaries/dom0-rootfs.cpio.gz cd .. # Load software into TFTP server directory. diff --git a/automation/scripts/xilinx-smoke-dom0less-arm64.sh b/automation/scripts/xilinx-smoke-dom0less-arm64.sh index 3e1fcf6bf93c..97190d0cbb9c 100755 --- a/automation/scripts/xilinx-smoke-dom0less-arm64.sh +++ b/automation/scripts/xilinx-smoke-dom0less-arm64.sh @@ -41,33 +41,31 @@ fi # DomU mkdir -p rootfs cd rootfs -tar xzf ../binaries/initrd.tar.gz mkdir proc mkdir run mkdir srv mkdir sys -rm var/run +mkdir -p etc/local.d echo "#!/bin/sh ${domU_check} /bin/sh" > etc/local.d/xen.start chmod +x etc/local.d/xen.start -echo "rc_verbose=yes" >> etc/rc.conf -find . | cpio -H newc -o | gzip > ../binaries/domU-rootfs.cpio.gz +cp ../binaries/rootfs.cpio.gz ../binaries/domU-rootfs.cpio.gz +find . | cpio -H newc -o | gzip >> ../binaries/domU-rootfs.cpio.gz cd .. rm -rf rootfs # DOM0 rootfs mkdir -p rootfs cd rootfs -tar xzf ../binaries/initrd.tar.gz mkdir proc mkdir run mkdir srv mkdir sys -rm var/run cp -ar ../binaries/dist/install/* . +mkdir -p etc/local.d echo "#!/bin/bash export LD_LIBRARY_PATH=/usr/local/lib @@ -78,8 +76,8 @@ bash /etc/init.d/xencommons start ${dom0_check} " > etc/local.d/xen.start chmod +x etc/local.d/xen.start -echo "rc_verbose=yes" >> etc/rc.conf -find . | cpio -H newc -o | gzip > ../binaries/dom0-rootfs.cpio.gz +cp ../binaries/rootfs.cpio.gz ../binaries/dom0-rootfs.cpio.gz +find . | cpio -H newc -o | gzip >> ../binaries/dom0-rootfs.cpio.gz cd ..
The prerequisite for this is building rootfs.cpio.gz, not only initrd.tar.gz. That's a change in the test-artifacts repository in the "Rootfs generation" series from Andrew. Having that, do not repack the whole initrd, but only pack modified files and rely on Linux handling of concatenated archives. This allows packing just test-related files (which includes the whole toolstack), instead of the whole initrd. DomU initrd handling is a bit more complicated thing. It's sent to the target host as part of the dom0 initrd. For now include prepared domU initrd as a whole in dom0's rootfs "overlay", which means compressing it (again) each job (this takes 3s). This can be later improved by splitting domU initrd into two parts (base + overlay) and concatenate them in target dom0 already. For this to be useful, test-artifacts would need to provide pre-made double-cpio version too (rootfs.cpio.gz wrapper in yet another cpio.gz as boot/domU-initrd). Since full initrd is not unpacked now when preparing domU (and dom0) rootfs, a couple of minor changes are needed to how they are prepared. This includes creating whole etc/issue file, instead of modifying existing one, and a need to create a couple directories. Finally, move adding "rc_verbose=yes" to /etc/rc.conf to initrd building. Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com> --- Changes in v3: - adjust for Andrew's version of cpio rootfs artifacts - don't require double-cpio archive for domU initrd, instead add it via dom0 initrd as plain cpio to avoid re-compressing the same thing New in v2 Jobs xilinx-* are untested, as I don't have necessary access. --- automation/gitlab-ci/test.yaml | 6 +++-- automation/scripts/qemu-alpine-x86_64.sh | 9 +++----- automation/scripts/qemu-smoke-dom0-arm64.sh | 7 ++---- automation/scripts/qemu-smoke-dom0less-arm64.sh | 7 ++---- automation/scripts/qubes-x86-64.sh | 18 +++++++--------- automation/scripts/xilinx-smoke-dom0-x86_64.sh | 15 ++++++------- automation/scripts/xilinx-smoke-dom0less-arm64.sh | 14 +++++------- 7 files changed, 35 insertions(+), 41 deletions(-)