| Message ID | xmqq34e9kmef.fsf@gitster.g (mailing list archive) |
|---|---|
| State | New |
| Headers | show |
| Series | [RFH] fedora: pedantic fails complaining that awk is missing | expand |
Hi, Junio C Hamano wrote: > GitHub Actions CI started failing the pedantic (fedora) job at the > tip of 'master' few days ago, and the log claims the failure is due > to missing "awk". Even though we have seen a few topics to rewrite > Perl scriptlet, and I think at least one of them uses awk, but they > haven't hit 'master' yet, so it is puzzling why this started failing > all of a sudden. > > So here is a band-aid. I wouldn't be surprised if the base image > was updated without telling us. We have seen Ubuntu 20.04 base > image retired to cause CI failures as well. > > Almalinux does not even seem to have awk available as a package > (or it may be called differently, but we are not suffering from > the lack of awk on that platform anyway), so make sure we ask for > awk only on fedora. > > Signed-off-by: Junio C Hamano <gitster@pobox.com> > --- > > * As the scheduled retirement of U20.04 image broke CI jobs running > for 'master', I started looking at making it pass again with the > minimum change, i.e. by merging dd/sparse-glibc-workaround and > js/ci-github-update-ubuntu topics. > > Then I somehow found that another job is broken. And this patch > seems to make it work. Not knowing how it got broken is > unsatisfactory, though. I can hazard a guess (or several)... It is likely that the image pointed to by the fedora:latest tag has moved from fedora 41 to 42, which was released today. The fedora 41 container images have awk installed while the fedora 42 images do not. That change is, I suspect, just part of reducing the size of the base container images. In both AlmaLinux and Fedora (as well as other RHEL derivatives/relatives), awk is provided by the gawk package. On Fedora, `dnf install awk` uses the package filelist data to determine that /usr/bin/awk is provided by gawk and installs gawk as a result. On AlmaLinux (8 & 9, by my quick testing), that is not the case and you'd need to use `dnf install gawk` or `dnf install '*bin/awk'` to get it installed. Though awk is included in the current AlmaLinux 8 and 9 images, so it isn't strictly needed. But it's probably better to be explicit that we need it installed, as a defense against some future change to the AlmaLinux container removing awk. Using gawk (or even '*bin/awk') would likely be clearer. That avoids relying on 1) a case statement to set an awk variable; and 2) the different behavior of older and newer releases of dnf with respect to the install argument matching on the names of binaries provided by the package. But that's certainly subjective. ;) > ci/install-dependencies.sh | 4 +++- > 1 file changed, 3 insertions(+), 1 deletion(-) > > diff --git a/ci/install-dependencies.sh b/ci/install-dependencies.sh > index 8700c0f292..a7c613ce4c 100755 > --- a/ci/install-dependencies.sh > +++ b/ci/install-dependencies.sh > @@ -30,8 +30,10 @@ alpine-*) > bash cvs gnupg perl-cgi perl-dbd-sqlite perl-io-tty >/dev/null > ;; > fedora-*|almalinux-*) > + awk= > + case "$distro" in fedora-*) awk=awk ;; esac > dnf -yq update >/dev/null && > - dnf -yq install shadow-utils sudo make gcc findutils diffutils perl python3 gettext zlib-devel expat-devel openssl-devel curl-devel pcre2-devel >/dev/null > + dnf -yq install shadow-utils sudo make gcc findutils diffutils perl python3 gettext zlib-devel expat-devel openssl-devel curl-devel pcre2-devel $awk >/dev/null > ;; > ubuntu-*|i386/ubuntu-*|debian-*) > # Required so that apt doesn't wait for user input on certain packages.
Hi Todd, On Wed, 16 Apr 2025, Todd Zullinger wrote: > Junio C Hamano wrote: > > GitHub Actions CI started failing the pedantic (fedora) job at the > > tip of 'master' few days ago, and the log claims the failure is due > > to missing "awk". Even though we have seen a few topics to rewrite > > Perl scriptlet, and I think at least one of them uses awk, but they > > haven't hit 'master' yet, so it is puzzling why this started failing > > all of a sudden. > > > > So here is a band-aid. I wouldn't be surprised if the base image > > was updated without telling us. We have seen Ubuntu 20.04 base > > image retired to cause CI failures as well. > > > > Almalinux does not even seem to have awk available as a package > > (or it may be called differently, but we are not suffering from > > the lack of awk on that platform anyway), so make sure we ask for > > awk only on fedora. > > > > Signed-off-by: Junio C Hamano <gitster@pobox.com> > > --- > > > > * As the scheduled retirement of U20.04 image broke CI jobs running > > for 'master', I started looking at making it pass again with the > > minimum change, i.e. by merging dd/sparse-glibc-workaround and > > js/ci-github-update-ubuntu topics. > > > > Then I somehow found that another job is broken. And this patch > > seems to make it work. Not knowing how it got broken is > > unsatisfactory, though. > > I can hazard a guess (or several)... > > It is likely that the image pointed to by the fedora:latest > tag has moved from fedora 41 to 42, which was released > today. The fedora 41 container images have awk installed > while the fedora 42 images do not. That change is, I > suspect, just part of reducing the size of the base > container images. Thank you for that excellent and thorough insight. This is indeed the case in the instances I looked at (for example, Git for Windows' `shears/main` branch was using F42 most recently, and failing: https://github.com/git-for-windows/git/actions/runs/14485163916/job/40629342280#step:2:38 whereas the preceding build used F41 and succeeded: https://github.com/git-for-windows/git/actions/runs/14473893559/job/40594656287#step:2:37). > In both AlmaLinux and Fedora (as well as other RHEL > derivatives/relatives), awk is provided by the gawk package. > > On Fedora, `dnf install awk` uses the package filelist data > to determine that /usr/bin/awk is provided by gawk and > installs gawk as a result. > > On AlmaLinux (8 & 9, by my quick testing), that is not the > case and you'd need to use `dnf install gawk` or `dnf > install '*bin/awk'` to get it installed. Though awk is > included in the current AlmaLinux 8 and 9 images, so it > isn't strictly needed. But it's probably better to be > explicit that we need it installed, as a defense against > some future change to the AlmaLinux container removing awk. > > Using gawk (or even '*bin/awk') would likely be clearer. > That avoids relying on 1) a case statement to set an awk > variable; and 2) the different behavior of older and newer > releases of dnf with respect to the install argument > matching on the names of binaries provided by the package. > > But that's certainly subjective. ;) This reasoning is very convincing, so I allowed myself to turn that into the following commit (which fixed the build: https://github.com/git-for-windows/git/actions/runs/14485317015/job/40629741228): -- snipsnap -- From 4890df8bf1c1311ef0c3b5a92412ccc4f833f944 Mon Sep 17 00:00:00 2001 From: Johannes Schindelin <johannes.schindelin@gmx.de> Date: Wed, 16 Apr 2025 07:17:24 +0200 Subject: [PATCH] ci(pedantic): ensure that awk is installed The image pointed to by the fedora:latest tag has moved from fedora 41 to 42. The fedora 41 container images have awk installed while the fedora 42 images do not. That change is most likely just part of reducing the size of the base container images. In both AlmaLinux and Fedora (as well as other RHEL derivatives/relatives), awk is provided by the gawk package. On Fedora, `dnf install awk` would work, but for unintended reasons! It uses the package filelist data to determine that /usr/bin/awk is provided by gawk and installs gawk as a result. On AlmaLinux (8 & 9, by my quick testing), that is not the case and you'd need to use `dnf install gawk` or `dnf install '*bin/awk'` to get it installed. Having said that, awk _is_ included in the current AlmaLinux 8 and 9 images, so it isn't strictly needed. But it's probably better to be explicit that we need it installed, as a defense against some future change to the AlmaLinux container removing awk. Using the package name "gawk" is the right thing to do. Note that even '*bin/awk' would have worked, but it is less specific. And who knows, maybe in the far future a BSD variant of awk is offered, too, and would then cause ambiguities. Best to avoid that. Suggested-by: Todd Zullinger <tmz@pobox.com> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> --- ci/install-dependencies.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/ci/install-dependencies.sh b/ci/install-dependencies.sh index 8700c0f2924d..be9ba5e30a47 100755 --- a/ci/install-dependencies.sh +++ b/ci/install-dependencies.sh @@ -31,7 +31,7 @@ alpine-*) ;; fedora-*|almalinux-*) dnf -yq update >/dev/null && - dnf -yq install shadow-utils sudo make gcc findutils diffutils perl python3 gettext zlib-devel expat-devel openssl-devel curl-devel pcre2-devel >/dev/null + dnf -yq install shadow-utils sudo make gcc findutils diffutils perl python3 gawk gettext zlib-devel expat-devel openssl-devel curl-devel pcre2-devel >/dev/null ;; ubuntu-*|i386/ubuntu-*|debian-*) # Required so that apt doesn't wait for user input on certain packages. --
diff --git a/ci/install-dependencies.sh b/ci/install-dependencies.sh index 8700c0f292..a7c613ce4c 100755 --- a/ci/install-dependencies.sh +++ b/ci/install-dependencies.sh @@ -30,8 +30,10 @@ alpine-*) bash cvs gnupg perl-cgi perl-dbd-sqlite perl-io-tty >/dev/null ;; fedora-*|almalinux-*) + awk= + case "$distro" in fedora-*) awk=awk ;; esac dnf -yq update >/dev/null && - dnf -yq install shadow-utils sudo make gcc findutils diffutils perl python3 gettext zlib-devel expat-devel openssl-devel curl-devel pcre2-devel >/dev/null + dnf -yq install shadow-utils sudo make gcc findutils diffutils perl python3 gettext zlib-devel expat-devel openssl-devel curl-devel pcre2-devel $awk >/dev/null ;; ubuntu-*|i386/ubuntu-*|debian-*) # Required so that apt doesn't wait for user input on certain packages.
GitHub Actions CI started failing the pedantic (fedora) job at the tip of 'master' few days ago, and the log claims the failure is due to missing "awk". Even though we have seen a few topics to rewrite Perl scriptlet, and I think at least one of them uses awk, but they haven't hit 'master' yet, so it is puzzling why this started failing all of a sudden. So here is a band-aid. I wouldn't be surprised if the base image was updated without telling us. We have seen Ubuntu 20.04 base image retired to cause CI failures as well. Almalinux does not even seem to have awk available as a package (or it may be called differently, but we are not suffering from the lack of awk on that platform anyway), so make sure we ask for awk only on fedora. Signed-off-by: Junio C Hamano <gitster@pobox.com> --- * As the scheduled retirement of U20.04 image broke CI jobs running for 'master', I started looking at making it pass again with the minimum change, i.e. by merging dd/sparse-glibc-workaround and js/ci-github-update-ubuntu topics. Then I somehow found that another job is broken. And this patch seems to make it work. Not knowing how it got broken is unsatisfactory, though. ci/install-dependencies.sh | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)