From patchwork Mon Oct 30 12:14:43 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13440529 Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6A4EB156C0 for ; Mon, 30 Oct 2023 12:14:49 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b="pYsR0IRS"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="Rl4z4WRZ" Received: from wout4-smtp.messagingengine.com (wout4-smtp.messagingengine.com [64.147.123.20]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 354CAC6 for ; Mon, 30 Oct 2023 05:14:48 -0700 (PDT) Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailout.west.internal (Postfix) with ESMTP id 67C313200944; Mon, 30 Oct 2023 08:14:47 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute3.internal (MEProxy); Mon, 30 Oct 2023 08:14:47 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:sender :subject:subject:to:to; s=fm3; t=1698668086; x=1698754486; bh=jg 8MgYs/3AmAdC73f9i6i0Y61AHYtzfu4EpnaSOHYPE=; b=pYsR0IRSEciDAr5Cnh TQVyaIAWbQWfuSCuOVaSj2A9OTaHOe9qG+KvLVhZ6R2SQnUhmKtiOgKJcLYWPy8l yxU/JRj36wP+htDDABLkFiw3ICtgld31CUpMox6GefQI8hKgA+Y734oW1ZBA7/0u S6LEMKGT78yzRXeD8WrztJqMpwmCsSkLlF7sy+GhC8IK9Eay0sNQzLjCX8r4giek G5LULIGl6qvGwlIvAia4KqF1M5HWc/GT7+dgpyd6KS3xwlKayuNhytUv1LmpUQgC xpQ4fxnxKrlKkBfgubxpU6qbsyupq6dJQjglNvHw0WpHFe7Mlhdk7aC8o3NB/CrP MEWw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:sender:subject :subject:to:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm3; t=1698668086; x=1698754486; bh=jg8MgYs/3AmAd C73f9i6i0Y61AHYtzfu4EpnaSOHYPE=; b=Rl4z4WRZlFUxTqAjBYCR3V74dslgt QsampYd2q3f0JvBuziL+fAOcbACtmcSntYx+QoJ/S52e3j3564ebY/Sc2KCVIfWV Qj2Krvh4ws6yGb4ojc4WDLKgR5C0mlosBwJaQjmqH7ZXamYtb3eTq6w3bQIwmiYn p0g4QA6/h/SK0N6zeXvMriQbZZiB+weQ6/FJ/QctnbMZ0F1130CSvDS56QRAC4z4 3bUhMcYleJ0/rQnTpPMCU3AauhFuDgOdl1ZEEkO3ozbxPvmaBzAL9C43A13O1Q5G sXm2n3tfIymzOUczK43Otws0F7kkjzp7sQ+PqFKb3pRpoY9lnY9ldhmmg== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvkedruddttddgfeehucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhepfffhvfevuffkfhggtggujgesghdtreertddtvdenucfhrhhomheprfgrthhr ihgtkhcuufhtvghinhhhrghrughtuceophhssehpkhhsrdhimheqnecuggftrfgrthhtvg hrnhepueektdevtdffveeljeetgfehheeigeekleduvdeffeeghefgledttdehjeelffet necuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepphhsse hpkhhsrdhimh X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 30 Oct 2023 08:14:45 -0400 (EDT) Received: by vm-mail (OpenSMTPD) with ESMTPSA id bc932a16 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Mon, 30 Oct 2023 12:14:40 +0000 (UTC) Date: Mon, 30 Oct 2023 13:14:43 +0100 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Junio C Hamano , Phillip Wood , Oswald Buddenhagen Subject: [PATCH v3 1/8] ci: reorder definitions for grouping functions Message-ID: References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: We define a set of grouping functions that are used to group together output in our CI, where these groups then end up as collapsible sections in the respective pipeline platform. The way these functions are defined is not easily extensible though as we have an up front check for the CI _not_ being GitHub Actions, where we define the non-stub logic in the else branch. Reorder the conditional branches such that we explicitly handle GitHub Actions. Signed-off-by: Patrick Steinhardt --- ci/lib.sh | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/ci/lib.sh b/ci/lib.sh index 6fbb5bade12..eb384f4e952 100755 --- a/ci/lib.sh +++ b/ci/lib.sh @@ -1,16 +1,7 @@ # Library of functions shared by all CI scripts -if test true != "$GITHUB_ACTIONS" +if test true = "$GITHUB_ACTIONS" then - begin_group () { :; } - end_group () { :; } - - group () { - shift - "$@" - } - set -x -else begin_group () { need_to_end_group=t echo "::group::$1" >&2 @@ -42,6 +33,15 @@ else } begin_group "CI setup" +else + begin_group () { :; } + end_group () { :; } + + group () { + shift + "$@" + } + set -x fi # Set 'exit on error' for all CI scripts to let the caller know that From patchwork Mon Oct 30 12:14:48 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13440530 Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CBD3D15E96 for ; Mon, 30 Oct 2023 12:14:53 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b="VHjuVAqY"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="ONF1CAGT" Received: from wout4-smtp.messagingengine.com (wout4-smtp.messagingengine.com [64.147.123.20]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 964FAA2 for ; Mon, 30 Oct 2023 05:14:52 -0700 (PDT) Received: from compute2.internal (compute2.nyi.internal [10.202.2.46]) by mailout.west.internal (Postfix) with ESMTP id C504832009BA; Mon, 30 Oct 2023 08:14:51 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute2.internal (MEProxy); Mon, 30 Oct 2023 08:14:52 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:sender :subject:subject:to:to; s=fm3; t=1698668091; x=1698754491; bh=aP dr+FDQJqUSn8SOprKEUrOnjm41kutOxCvKrwPcWrk=; b=VHjuVAqYgkK2G+Vbb+ nBlKramT+Uw7QI7Ha48MVLuPj+jxrid+0D2BDz9cFgVd9YlgRjld9anaqEBAif9j F2znrwNk2BbnhYtoxl++Gatea2dxXWGk30KLEMEM4aSALGIbzQnaWJGx94eTovhn 7Gv49B38a3sL98raCbpvplS6U7EdR3dz9HgdKuk+HC710LLaKkJ0lJcJ5JjPmi4E 07MRLTW5oK9mavMgJ4BwNa8G9jrgXCodAw53eGxUCPwC49h7sHX2u7XgB+nnODJN D7+9m+fBpNjYUUr45MTcziBhWbhwz2NjOJxsGmciV8v9+7QuYDYLjzLgR2phzk7Q RpNw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:sender:subject :subject:to:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm3; t=1698668091; x=1698754491; bh=aPdr+FDQJqUSn 8SOprKEUrOnjm41kutOxCvKrwPcWrk=; b=ONF1CAGTfpw3lz/5p3UbhZvL4FuIV OG99ZpujEw6ta76xX3JKVo4Y3Cjl4+R0+tug1zLJxClFMWYTecq751czUYmwg+s0 P96fdvM304MEBC3/ic/2/ga83cHHS2CQYJOmLUDqfPkDbyUsy/RNSmBmLrxpT/X1 3RF8kzZzuzIsSDgFnyGzt83fUPa6QudGwOCAXgSjFFJw9otUGOg0uP7unRfId7dV CgSOfeIIAaZ5j4BUgIWnrf55SBElH3aLcbGYZ4Pz8FglpGkBc/kzS0bDb57ZdsX7 4+n/zWvXdKaGz2u93Fc1f8Ih+hujBD3mXJOh9p5klFSfRBHN8r8hGWbTQ== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvkedruddttddgfeehucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhepfffhvfevuffkfhggtggujgesghdtreertddtvdenucfhrhhomheprfgrthhr ihgtkhcuufhtvghinhhhrghrughtuceophhssehpkhhsrdhimheqnecuggftrfgrthhtvg hrnhepueektdevtdffveeljeetgfehheeigeekleduvdeffeeghefgledttdehjeelffet necuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepphhsse hpkhhsrdhimh X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 30 Oct 2023 08:14:50 -0400 (EDT) Received: by vm-mail (OpenSMTPD) with ESMTPSA id ecd14b32 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Mon, 30 Oct 2023 12:14:44 +0000 (UTC) Date: Mon, 30 Oct 2023 13:14:48 +0100 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Junio C Hamano , Phillip Wood , Oswald Buddenhagen Subject: [PATCH v3 2/8] ci: make grouping setup more generic Message-ID: <77798fa7a7aa084c1bbd9fc9118346c47770125a.1698667545.git.ps@pks.im> References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Make the grouping setup more generic by always calling `begin_group ()` and `end_group ()` regardless of whether we have stubbed those functions or not. This ensures we can more readily add support for additional CI platforms. Furthermore, the `group ()` function is made generic so that it is the same for both GitHub Actions and for other platforms. There is a semantic conflict here though: GitHub Actions used to call `set +x` in `group ()` whereas the non-GitHub case unconditionally uses `set -x`. The latter would get overriden if we kept the `set +x` in the generic version of `group ()`. To resolve this conflict, we simply drop the `set +x` in the generic variant of this function. As `begin_group ()` calls `set -x` anyway this is not much of a change though, as the only commands that aren't printed anymore now are the ones between the beginning of `group ()` and the end of `begin_group ()`. Last, this commit changes `end_group ()` to also accept a parameter that indicates _which_ group should end. This will be required by a later commit that introduces support for GitLab CI. Signed-off-by: Patrick Steinhardt --- ci/lib.sh | 46 ++++++++++++++++++++++------------------------ 1 file changed, 22 insertions(+), 24 deletions(-) diff --git a/ci/lib.sh b/ci/lib.sh index eb384f4e952..b3411afae8e 100755 --- a/ci/lib.sh +++ b/ci/lib.sh @@ -14,36 +14,34 @@ then need_to_end_group= echo '::endgroup::' >&2 } - trap end_group EXIT - - group () { - set +x - begin_group "$1" - shift - # work around `dash` not supporting `set -o pipefail` - ( - "$@" 2>&1 - echo $? >exit.status - ) | - sed 's/^\(\([^ ]*\):\([0-9]*\):\([0-9]*:\) \)\(error\|warning\): /::\5 file=\2,line=\3::\1/' - res=$(cat exit.status) - rm exit.status - end_group - return $res - } - - begin_group "CI setup" else begin_group () { :; } end_group () { :; } - group () { - shift - "$@" - } set -x fi +group () { + group="$1" + shift + begin_group "$group" + + # work around `dash` not supporting `set -o pipefail` + ( + "$@" 2>&1 + echo $? >exit.status + ) | + sed 's/^\(\([^ ]*\):\([0-9]*\):\([0-9]*:\) \)\(error\|warning\): /::\5 file=\2,line=\3::\1/' + res=$(cat exit.status) + rm exit.status + + end_group "$group" + return $res +} + +begin_group "CI setup" +trap "end_group 'CI setup'" EXIT + # Set 'exit on error' for all CI scripts to let the caller know that # something went wrong. # @@ -287,5 +285,5 @@ esac MAKEFLAGS="$MAKEFLAGS CC=${CC:-cc}" -end_group +end_group "CI setup" set -x From patchwork Mon Oct 30 12:14:52 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13440531 Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D8B17156DB for ; Mon, 30 Oct 2023 12:14:57 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b="Jim0uzko"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="ZZAe4T5w" Received: from wout4-smtp.messagingengine.com (wout4-smtp.messagingengine.com [64.147.123.20]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DED45A2 for ; Mon, 30 Oct 2023 05:14:56 -0700 (PDT) Received: from compute7.internal (compute7.nyi.internal [10.202.2.48]) by mailout.west.internal (Postfix) with ESMTP id 1DB1F320093A; Mon, 30 Oct 2023 08:14:56 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute7.internal (MEProxy); Mon, 30 Oct 2023 08:14:56 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:sender :subject:subject:to:to; s=fm3; t=1698668095; x=1698754495; bh=Lk QAu2zM6yIi8eCNUjGipyHY/7oos4+Ssxi0pY25KFs=; b=Jim0uzkofBOcYtX6mE gofHEDkwOzGt371Mvlts8yKCjJrlrzVQM5D+ty6F1Vu4tqdbfB4w7SugTu9SNL28 5OCqObk692nX27HuskOyAyVytkr3l5Y7h9ucl+VjzYpuNLw8vBe+9LuE9wiwTEEQ S23PU6KJCSdhaLTbBsURM+FUU/nZqJE1f1Jw3zCw5z0AJoSa6TECaWg2B90qlB7X ln4j1lTjAH0IK33u9KeuQ6lGULC2DaTuwWNS2YgfLv81pwS39FXqy67Z6hqA7rfJ tpxufRwvJC7Mz0YP2Oa/5Kpn7g3G91/87pZ4CvM2Y0tnGYsK36IetW0IpeX6vwUN Z8jQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:sender:subject :subject:to:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm3; t=1698668095; x=1698754495; bh=LkQAu2zM6yIi8 eCNUjGipyHY/7oos4+Ssxi0pY25KFs=; b=ZZAe4T5wkpkG4IlcLYHo9gwKoco9p lks3r1Hr4v088WZcwoYqCwhyJ5nyiL2o0WGn/bT2lRUBLgwNjb4Vd07v6PYFP+H4 6IzKmmWYk0qIR3jhf+5CqgpyE6TVZiTeBJfFSFdGYTDegcbgyAqkBGl6nP4RoEKT asqgeRXPJmVGp0Y1lTEp0Scv3Lmxtyve/8aWNo4nYlEyZg5A/bCJmLN3N5ez5oZC jK2iJw9EfLr6zJsUpCsJ4eZjO4R+TVGLygXAZlUD1cwF1v/f9xgsMgYPxorpq+HC tFz88lp1websiZhqaHAMpJhW6UvZ4Lwfz/k1tEfNKgnn54RUqufLkyX9Q== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvkedruddttddgfeehucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhepfffhvfevuffkfhggtggujgesghdtreertddtvdenucfhrhhomheprfgrthhr ihgtkhcuufhtvghinhhhrghrughtuceophhssehpkhhsrdhimheqnecuggftrfgrthhtvg hrnhepueektdevtdffveeljeetgfehheeigeekleduvdeffeeghefgledttdehjeelffet necuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepphhsse hpkhhsrdhimh X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 30 Oct 2023 08:14:54 -0400 (EDT) Received: by vm-mail (OpenSMTPD) with ESMTPSA id 555056a3 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Mon, 30 Oct 2023 12:14:48 +0000 (UTC) Date: Mon, 30 Oct 2023 13:14:52 +0100 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Junio C Hamano , Phillip Wood , Oswald Buddenhagen Subject: [PATCH v3 3/8] ci: group installation of Docker dependencies Message-ID: <4542bd38dc270735f5a2674cf058f008746adb53.1698667545.git.ps@pks.im> References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: The output of CI jobs tends to be quite long-winded and hard to digest. To help with this, many CI systems provide the ability to group output into collapsible sections, and we're also doing this in some of our scripts. One notable omission is the script to install Docker dependencies. Address it to bring more structure to the output for Docker-based jobs. Signed-off-by: Patrick Steinhardt --- ci/install-docker-dependencies.sh | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/ci/install-docker-dependencies.sh b/ci/install-docker-dependencies.sh index 78b7e326da6..d0bc19d3bb3 100755 --- a/ci/install-docker-dependencies.sh +++ b/ci/install-docker-dependencies.sh @@ -3,6 +3,10 @@ # Install dependencies required to build and test Git inside container # +. ${0%/*}/lib.sh + +begin_group "Install dependencies" + case "$jobname" in linux32) linux32 --32bit i386 sh -c ' @@ -20,3 +24,5 @@ pedantic) dnf -yq install make gcc findutils diffutils perl python3 gettext zlib-devel expat-devel openssl-devel curl-devel pcre2-devel >/dev/null ;; esac + +end_group "Install dependencies" From patchwork Mon Oct 30 12:14:57 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13440532 Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9171415EAA for ; Mon, 30 Oct 2023 12:15:02 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b="ZNl+Uooq"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="NkEBlrmC" Received: from wout4-smtp.messagingengine.com (wout4-smtp.messagingengine.com [64.147.123.20]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 27A82C9 for ; Mon, 30 Oct 2023 05:15:01 -0700 (PDT) Received: from compute7.internal (compute7.nyi.internal [10.202.2.48]) by mailout.west.internal (Postfix) with ESMTP id 58AA03200944; Mon, 30 Oct 2023 08:15:00 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute7.internal (MEProxy); Mon, 30 Oct 2023 08:15:00 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:sender :subject:subject:to:to; s=fm3; t=1698668099; x=1698754499; bh=Ed zvlZ7v+/E6saB3Tsm6WxxNGDfx0V1FcydChnHKS3s=; b=ZNl+UooqyqqKvp50xs jgbPPGjgnQdIsYWDKGWpYScETZ7KXSHtB+cRQvaW1ppuKHT+rSVzjfUB+4b6wNxj wMPF0OBZWkVSXVxofnjs4mUYeeg46Bns18Wp1qSH/5cVAcHMK5P+h6m7rNoqVuac LKs4aU1B5nNccNSaaQFeepeSyEpvETxNUrHW+VK4Me5NuS25PUsX2nGCYihrsT3u Xh9h+HAZxWzxLxMpQAm5XAdjDpT04qhYS8bxueHtkLP+vGmz4TmJQE8kwgbuEZo8 FIe39DSptZUUIDwOZWozUxjrdUoQH7AozHs4QbYPprbjzP44vqU/frzTZr8R03UE nfng== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:sender:subject :subject:to:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm3; t=1698668099; x=1698754499; bh=EdzvlZ7v+/E6s aB3Tsm6WxxNGDfx0V1FcydChnHKS3s=; b=NkEBlrmCdTUCIKF0cXJWnqnCqn54/ fW3OGAFCuboisrC/4kLGCbEGgGTXgPv/txtib3fFcHF9eXF72V/j4h6SR8qk2CiO uJHFAQek7ny6/RWva0sp47bxs4JNieIYDJMUTHf/O3vx87HS/VyxQlys5hXRwyO8 LIHEd5IoKoRbPwg2sst1M/Ur3xkqKbUweR1E8SBBtfTmhxmL1G0sKY2tw5mfa5Kb HdQuKOE0fvT2p1mAbA7Ebh5TKDuWHHAsxMFtdmLZfdI94nUxqCpUmNKYqsnhrZag VanfvxojFzt0XI/q4xvzggF9gV36lzS0EaeL8WoxGkOggdSsnBiy/hlvg== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvkedruddttddgfeehucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhepfffhvfevuffkfhggtggujgesghdtreertddtvdenucfhrhhomheprfgrthhr ihgtkhcuufhtvghinhhhrghrughtuceophhssehpkhhsrdhimheqnecuggftrfgrthhtvg hrnhepueektdevtdffveeljeetgfehheeigeekleduvdeffeeghefgledttdehjeelffet necuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepphhsse hpkhhsrdhimh X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 30 Oct 2023 08:14:58 -0400 (EDT) Received: by vm-mail (OpenSMTPD) with ESMTPSA id 137b1bda (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Mon, 30 Oct 2023 12:14:53 +0000 (UTC) Date: Mon, 30 Oct 2023 13:14:57 +0100 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Junio C Hamano , Phillip Wood , Oswald Buddenhagen Subject: [PATCH v3 4/8] ci: split out logic to set up failed test artifacts Message-ID: <5fdda7fd83f21422fb47ee86ae5372357cc4d846.1698667545.git.ps@pks.im> References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: We have some logic in place to create a directory with the output from failed tests, which will then subsequently be uploaded as CI artifacts. We're about to add support for GitLab CI, which will want to reuse the logic. Split the logic into a separate function so that it is reusable. Signed-off-by: Patrick Steinhardt --- ci/lib.sh | 40 ++++++++++++++++++++++------------------ 1 file changed, 22 insertions(+), 18 deletions(-) diff --git a/ci/lib.sh b/ci/lib.sh index b3411afae8e..9ffdf743903 100755 --- a/ci/lib.sh +++ b/ci/lib.sh @@ -131,6 +131,27 @@ handle_failed_tests () { return 1 } +create_failed_test_artifacts () { + mkdir -p t/failed-test-artifacts + + for test_exit in t/test-results/*.exit + do + test 0 != "$(cat "$test_exit")" || continue + + test_name="${test_exit%.exit}" + test_name="${test_name##*/}" + printf "\\e[33m\\e[1m=== Failed test: ${test_name} ===\\e[m\\n" + echo "The full logs are in the 'print test failures' step below." + echo "See also the 'failed-tests-*' artifacts attached to this run." + cat "t/test-results/$test_name.markup" + + trash_dir="t/trash directory.$test_name" + cp "t/test-results/$test_name.out" t/failed-test-artifacts/ + tar czf t/failed-test-artifacts/"$test_name".trash.tar.gz "$trash_dir" + done + return 1 +} + # GitHub Action doesn't set TERM, which is required by tput export TERM=${TERM:-dumb} @@ -171,25 +192,8 @@ then CC="${CC_PACKAGE:-${CC:-gcc}}" DONT_SKIP_TAGS=t handle_failed_tests () { - mkdir -p t/failed-test-artifacts echo "FAILED_TEST_ARTIFACTS=t/failed-test-artifacts" >>$GITHUB_ENV - - for test_exit in t/test-results/*.exit - do - test 0 != "$(cat "$test_exit")" || continue - - test_name="${test_exit%.exit}" - test_name="${test_name##*/}" - printf "\\e[33m\\e[1m=== Failed test: ${test_name} ===\\e[m\\n" - echo "The full logs are in the 'print test failures' step below." - echo "See also the 'failed-tests-*' artifacts attached to this run." - cat "t/test-results/$test_name.markup" - - trash_dir="t/trash directory.$test_name" - cp "t/test-results/$test_name.out" t/failed-test-artifacts/ - tar czf t/failed-test-artifacts/"$test_name".trash.tar.gz "$trash_dir" - done - return 1 + create_failed_test_artifacts } cache_dir="$HOME/none" From patchwork Mon Oct 30 12:15:01 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13440533 Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A8A3A15EA8 for ; Mon, 30 Oct 2023 12:15:06 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b="Z4JG31gz"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="ebZf67fJ" Received: from wout4-smtp.messagingengine.com (wout4-smtp.messagingengine.com [64.147.123.20]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 88F12C9 for ; Mon, 30 Oct 2023 05:15:05 -0700 (PDT) Received: from compute6.internal (compute6.nyi.internal [10.202.2.47]) by mailout.west.internal (Postfix) with ESMTP id BADCE32009BA; Mon, 30 Oct 2023 08:15:04 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute6.internal (MEProxy); Mon, 30 Oct 2023 08:15:05 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:sender :subject:subject:to:to; s=fm3; t=1698668104; x=1698754504; bh=jK Qrnie4M5x5j1pDKTv4+KP5ezHU70EmoEkBjqL1qUo=; b=Z4JG31gzM0PMqmjMJ4 R3spjewP5Aod6wEd853gy7hh8gfJK/7Y0qMrH8bloLAoSSp3C5OvtiibFuel2/6C uuWia8r8GeOwJpHw9Ii6s5ecjwbYEaLf7NAPy4bKbxbqcFH4b1LSOVtSNtaW97IG 62/CxG6tGEohAr4rkq/2yVnwi2cXMc9482ma4bXVgpJHy8uZxs9iK2o1REnoylBY GJeeQJG+cDa0sr8ZT3sJna6DqwuuMVbhM5WPzLXMsrweMby8dMpxyGE08GFmSRKq OKdKtVUHonNCuxtauaHlWxLE49Px3ySBLBjUmHJjSvgIbgYuJI+tgMsMyqqDE4Vq FmXQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:sender:subject :subject:to:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm3; t=1698668104; x=1698754504; bh=jKQrnie4M5x5j 1pDKTv4+KP5ezHU70EmoEkBjqL1qUo=; b=ebZf67fJDKk0YDXIXcx4VD0Hyp9TR 1764oOv6XFrrIjKp/JQpmj3HwbtPrTe+ywTfqeTwsk4wwMKXgVGt4TBXYulgcng3 IMIXk/1tSJH7vp1pX2lW2d8EEFan6SnB0Ge6Lh9Xlku0722SwOcdYW5wwgjh4Ozd KMzk7luvGuehmoPLj4dpUYNbj3P3nJn+HHeT5BS+jGRcQNL6s3c5KUU3eqnM8xJw ad5bMGGV6F0JzN4rwXcCLq3XhBTSaZuPZfaithP/cwLAId5x2n8eNyJHTsYzsK7Q GoqigXJIP2H84zml/g1hwEWLXKRTf6+lyrOouSBmcm19/fkQJCHRnCPuA== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvkedruddttddgfeehucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhepfffhvfevuffkfhggtggujgesghdtreertddtvdenucfhrhhomheprfgrthhr ihgtkhcuufhtvghinhhhrghrughtuceophhssehpkhhsrdhimheqnecuggftrfgrthhtvg hrnhepueektdevtdffveeljeetgfehheeigeekleduvdeffeeghefgledttdehjeelffet necuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepphhsse hpkhhsrdhimh X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 30 Oct 2023 08:15:03 -0400 (EDT) Received: by vm-mail (OpenSMTPD) with ESMTPSA id e0edf0dc (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Mon, 30 Oct 2023 12:14:57 +0000 (UTC) Date: Mon, 30 Oct 2023 13:15:01 +0100 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Junio C Hamano , Phillip Wood , Oswald Buddenhagen Subject: [PATCH v3 5/8] ci: unify setup of some environment variables Message-ID: <6af0075fd875f176e7fdf6c219e7117dac5cd71c.1698667545.git.ps@pks.im> References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Both GitHub Actions and Azue Pipelines set up the environment variables GIT_TEST_OPTS, GIT_PROVE_OPTS and MAKEFLAGS. And while most values are actually the same, the setup is completely duplicate. With the upcoming support for GitLab CI this duplication would only extend even further. Unify the setup of those environment variables so that only the uncommon parts are separated. While at it, we also perform some additional small improvements: - We use nproc instead of a hardcoded count of jobs for make and prove. This ensures that the number of concurrent processes adapts to the host automatically. - We now always pass `--state=failed,slow,save` via GIT_PROVE_OPTS. It doesn't hurt on platforms where we don't persist the state, so this further reduces boilerplate. - When running on Windows systems we set `--no-chain-lint` and `--no-bin-wrappers`. Interestingly though, we did so _after_ already having exported the respective environment variables. - We stop using `export VAR=value` syntax, which is a Bashism. It's not quite worth it as we still use this syntax all over the place, but it doesn't hurt readability either. Signed-off-by: Patrick Steinhardt --- ci/lib.sh | 24 ++++++++++++++---------- 1 file changed, 14 insertions(+), 10 deletions(-) diff --git a/ci/lib.sh b/ci/lib.sh index 9ffdf743903..c7a716a6e3f 100755 --- a/ci/lib.sh +++ b/ci/lib.sh @@ -175,11 +175,7 @@ then # among *all* phases) cache_dir="$HOME/test-cache/$SYSTEM_PHASENAME" - export GIT_PROVE_OPTS="--timer --jobs 10 --state=failed,slow,save" - export GIT_TEST_OPTS="--verbose-log -x --write-junit-xml" - MAKEFLAGS="$MAKEFLAGS --jobs=10" - test windows_nt != "$CI_OS_NAME" || - GIT_TEST_OPTS="--no-chain-lint --no-bin-wrappers $GIT_TEST_OPTS" + GIT_TEST_OPTS="--write-junit-xml" elif test true = "$GITHUB_ACTIONS" then CI_TYPE=github-actions @@ -198,17 +194,25 @@ then cache_dir="$HOME/none" - export GIT_PROVE_OPTS="--timer --jobs 10" - export GIT_TEST_OPTS="--verbose-log -x --github-workflow-markup" - MAKEFLAGS="$MAKEFLAGS --jobs=10" - test windows != "$CI_OS_NAME" || - GIT_TEST_OPTS="--no-chain-lint --no-bin-wrappers $GIT_TEST_OPTS" + GIT_TEST_OPTS="--github-workflow-markup" else echo "Could not identify CI type" >&2 env >&2 exit 1 fi +MAKEFLAGS="$MAKEFLAGS --jobs=$(nproc)" +GIT_PROVE_OPTS="--timer --jobs $(nproc) --state=failed,slow,save" + +GIT_TEST_OPTS="$GIT_TEST_OPTS --verbose-log -x" +if test windows = "$CI_OS_NAME" +then + GIT_TEST_OPTS="$GIT_TEST_OPTS --no-chain-lint --no-bin-wrappers" +fi + +export GIT_TEST_OPTS +export GIT_PROVE_OPTS + good_trees_file="$cache_dir/good-trees" mkdir -p "$cache_dir" From patchwork Mon Oct 30 12:15:05 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13440534 Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 38F4315EA8 for ; Mon, 30 Oct 2023 12:15:10 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b="BVMoqZ53"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="p/yLXOLW" Received: from wout4-smtp.messagingengine.com (wout4-smtp.messagingengine.com [64.147.123.20]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CF8BDC9 for ; Mon, 30 Oct 2023 05:15:09 -0700 (PDT) Received: from compute7.internal (compute7.nyi.internal [10.202.2.48]) by mailout.west.internal (Postfix) with ESMTP id 0C33A32009D3; Mon, 30 Oct 2023 08:15:08 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute7.internal (MEProxy); Mon, 30 Oct 2023 08:15:09 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:sender :subject:subject:to:to; s=fm3; t=1698668108; x=1698754508; bh=2F Xgdpt8/DxxQZM69ttlWZbcwXlSCeJ+FRpw58fMcxg=; b=BVMoqZ53jFb/DVUst0 xESC33BC0Qspdlmgxv9s3QzTHlQrvlx5jyov3+THhkJMy1XmdN2RHIcWcdWmUwlQ nrnxBDuBokW177ZrJxT/SBSCFl64f0c4jH65DGC8VI61GwEQlhbnKQlBKHCCrjdY A8XWgvrzktuDCRbbNTzUAe2Cb4t+pSkcRx3AnZ1GKr3Rnq+uC+OWA9C6NjFPhv25 JWZK9KXqRMCqatQNDpgeyT4+dEnvMunCUWbTaicJDNjOq0WSA/jdaeVf2iDpn01i 0PeSSkjmtRx9Q+w66E1huj7OQOX0jU4jK3nIdoyFOIw/wy2v2eMhqNMpWzepy62v Mt5A== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:sender:subject :subject:to:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm3; t=1698668108; x=1698754508; bh=2FXgdpt8/DxxQ ZM69ttlWZbcwXlSCeJ+FRpw58fMcxg=; b=p/yLXOLW/CAUn3B9O+OItWWtEhhZ1 vcf2TRS+mw6ZEGGpLZI136P8A3DfreFnutcQiRwHxwDQf8f9lRsH6UE50LZ+NJiL yH8qRcXyNG2pK3DXrhfsxbhTFHA8iK3xIoB0fS31KnBotRdPH7TLHX+WNO717YmZ LexJQBI99EEWZTzeP1CTdanC6iuQsMPMpYfVV3ZyPyL/xFB5LX8zwIJDpxr+V7Wm FCHiCUbDFwLOy1wKGY6XvcSi5CLQiHV+xqezX2rQWwRQYGXCx6mdaaSCjrddcihi cMtRZBEwZmRErJrDsGR7/l2IO6rJjrApDGZw+HkUcMjHv6ks0d70Xue/g== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvkedruddttddgfeehucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhepfffhvfevuffkfhggtggujgesghdtreertddtvdenucfhrhhomheprfgrthhr ihgtkhcuufhtvghinhhhrghrughtuceophhssehpkhhsrdhimheqnecuggftrfgrthhtvg hrnhepueektdevtdffveeljeetgfehheeigeekleduvdeffeeghefgledttdehjeelffet necuvehluhhsthgvrhfuihiivgepvdenucfrrghrrghmpehmrghilhhfrhhomhepphhsse hpkhhsrdhimh X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 30 Oct 2023 08:15:07 -0400 (EDT) Received: by vm-mail (OpenSMTPD) with ESMTPSA id bf1630b6 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Mon, 30 Oct 2023 12:15:01 +0000 (UTC) Date: Mon, 30 Oct 2023 13:15:05 +0100 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Junio C Hamano , Phillip Wood , Oswald Buddenhagen Subject: [PATCH v3 6/8] ci: squelch warnings when testing with unusable Git repo Message-ID: <78d863bf24ea3190c87240fa0c2f6b42fdcde3c3.1698667545.git.ps@pks.im> References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Our CI jobs that run on Docker also use mostly the same architecture to build and test Git via the "ci/run-build-and-tests.sh" script. These scripts also provide some functionality to massage the Git repository we're supposedly operating in. In our Docker-based infrastructure we may not even have a Git repository available though, which leads to warnings when those functions execute. Make the helpers exit gracefully in case either there is no Git in our PATH, or when not running in a Git repository. Signed-off-by: Patrick Steinhardt --- ci/lib.sh | 32 ++++++++++++++++++++++++++++++++ 1 file changed, 32 insertions(+) diff --git a/ci/lib.sh b/ci/lib.sh index c7a716a6e3f..c13b6527cac 100755 --- a/ci/lib.sh +++ b/ci/lib.sh @@ -69,10 +69,32 @@ skip_branch_tip_with_tag () { fi } +# Check whether we can use the path passed via the first argument as Git +# repository. +is_usable_git_repository () { + # We require Git in our PATH, otherwise we cannot access repositories + # at all. + if ! command -v git >/dev/null + then + return 1 + fi + + # And the target directory needs to be a proper Git repository. + if ! git -C "$1" rev-parse 2>/dev/null + then + return 1 + fi +} + # Save some info about the current commit's tree, so we can skip the build # job if we encounter the same tree again and can provide a useful info # message. save_good_tree () { + if ! is_usable_git_repository . + then + return + fi + echo "$(git rev-parse $CI_COMMIT^{tree}) $CI_COMMIT $CI_JOB_NUMBER $CI_JOB_ID" >>"$good_trees_file" # limit the file size tail -1000 "$good_trees_file" >"$good_trees_file".tmp @@ -88,6 +110,11 @@ skip_good_tree () { return fi + if ! is_usable_git_repository . + then + return + fi + if ! good_tree_info="$(grep "^$(git rev-parse $CI_COMMIT^{tree}) " "$good_trees_file")" then # Haven't seen this tree yet, or no cached good trees file yet. @@ -119,6 +146,11 @@ skip_good_tree () { } check_unignored_build_artifacts () { + if ! is_usable_git_repository . + then + return + fi + ! git ls-files --other --exclude-standard --error-unmatch \ -- ':/*' 2>/dev/null || { From patchwork Mon Oct 30 12:15:10 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13440535 Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6268916418 for ; Mon, 30 Oct 2023 12:15:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b="BjU9NW+d"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="vkRklL3P" Received: from wout4-smtp.messagingengine.com (wout4-smtp.messagingengine.com [64.147.123.20]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 23877CC for ; Mon, 30 Oct 2023 05:15:14 -0700 (PDT) Received: from compute5.internal (compute5.nyi.internal [10.202.2.45]) by mailout.west.internal (Postfix) with ESMTP id 540A632009CB; Mon, 30 Oct 2023 08:15:13 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute5.internal (MEProxy); Mon, 30 Oct 2023 08:15:13 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:sender :subject:subject:to:to; s=fm3; t=1698668112; x=1698754512; bh=VX nvvvveis5W2s+nZGJGYW822Jn/Yd7Qf9yJtrJgWAM=; b=BjU9NW+dAM7iZ4KH+8 QJrQDQvpGyIgZsv5nA95OjrYXm0GzBQiVjwagZSajPWLvXvRXve4fAzp07e7i3e0 SFDaGLN9oPw2g2GK3KzzwKBeI11nEnInpGmc0UCT8zVDEQjKjv70ry52uzV8N8aa pFAwmOLgMBc3kw5nvkh2/85VYDsFmo/H+9vosznKVGrn/2ZtWpR6n7X/208Qum7n MLMlCctlup0rvt9wPIjGx6hcyt4g40an6h5IvtHuspXOUNMbX5EeRvWY+lUd97c5 YPbfuMCYwz6Z+WyQE7J285vzUOGt+0FjnVLf/i932Sg3OySxGhIHYXthc93t1O+U ix1g== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:sender:subject :subject:to:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm3; t=1698668112; x=1698754512; bh=VXnvvvveis5W2 s+nZGJGYW822Jn/Yd7Qf9yJtrJgWAM=; b=vkRklL3P+DlH2iHM0AAi7S6HqOPZR hl11xa8D8ffjqajtvg9Puoeg6HTUPQLDvOgZmRqqim/ZZuDKWISLU0RmP/HRBzdx AaX5Mjjl0ea1L3qtnYmZzDWT2Paq6aPXWdcnd51bmUHKUAqsiWMZvUz3Z6of5Aky nKbvtb3yQre+X8btrfPFpUP5IUsa2a7KTM6+pvRCmsV6ibUBDEUERrA4CuUeHzIi goKB5AUGFfawxKPfX+DbGu6NAvndBtN7pNDtPVSqI5DDcLl14ScP54TM9Ftb77SJ Ig0xseIdaLUytiqKtDHcEGkmiCvQVaZ0uEBoYL7Ey5oEeRm4R6poA0bTg== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvkedruddttddgfeehucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhepfffhvfevuffkfhggtggujgesghdtreertddtvdenucfhrhhomheprfgrthhr ihgtkhcuufhtvghinhhhrghrughtuceophhssehpkhhsrdhimheqnecuggftrfgrthhtvg hrnhepvdehteeggfevueevhedtleelveeigfefieduiefgvdfffeegvdeitefgteevveeu necuffhomhgrihhnpehhthhtphgurdhshhenucevlhhushhtvghrufhiiigvpedtnecurf grrhgrmhepmhgrihhlfhhrohhmpehpshesphhkshdrihhm X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 30 Oct 2023 08:15:11 -0400 (EDT) Received: by vm-mail (OpenSMTPD) with ESMTPSA id 4c0cca15 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Mon, 30 Oct 2023 12:15:06 +0000 (UTC) Date: Mon, 30 Oct 2023 13:15:10 +0100 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Junio C Hamano , Phillip Wood , Oswald Buddenhagen Subject: [PATCH v3 7/8] ci: install test dependencies for linux-musl Message-ID: References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: The linux-musl CI job executes tests on Alpine Linux, which is based on musl libc instead of glibc. We're missing some test dependencies though, which causes us to skip a subset of tests. Install these test dependencies to increase our test coverage on this platform. There are still some missing test dependecies, but these do not have a corresponding package in the Alpine repositories: - p4 and p4d, both parts of the Perforce version control system. - cvsps, which generates patch sets for CVS. - Subversion and the SVN::Core Perl library, the latter of which is not available in the Alpine repositories. While the tool itself is available, all Subversion-related tests are skipped without the SVN::Core Perl library anyway. Furthermore, in order to make the Apache-based tests work, this commit also adds the Alpine-specific modules path of it to the list of known paths. Signed-off-by: Patrick Steinhardt --- ci/install-docker-dependencies.sh | 3 ++- t/lib-httpd.sh | 3 ++- 2 files changed, 4 insertions(+), 2 deletions(-) diff --git a/ci/install-docker-dependencies.sh b/ci/install-docker-dependencies.sh index d0bc19d3bb3..05dde5c5d40 100755 --- a/ci/install-docker-dependencies.sh +++ b/ci/install-docker-dependencies.sh @@ -17,7 +17,8 @@ linux32) ;; linux-musl) apk add --update build-base curl-dev openssl-dev expat-dev gettext \ - pcre2-dev python3 musl-libintl perl-utils ncurses >/dev/null + pcre2-dev python3 musl-libintl perl-utils ncurses \ + apache2 bash cvs gnupg perl-cgi perl-dbd-sqlite >/dev/null ;; pedantic) dnf -yq update >/dev/null && diff --git a/t/lib-httpd.sh b/t/lib-httpd.sh index 2fb1b2ae561..9791f94b16f 100644 --- a/t/lib-httpd.sh +++ b/t/lib-httpd.sh @@ -67,7 +67,8 @@ for DEFAULT_HTTPD_MODULE_PATH in '/usr/libexec/apache2' \ '/usr/lib/apache2/modules' \ '/usr/lib64/httpd/modules' \ '/usr/lib/httpd/modules' \ - '/usr/libexec/httpd' + '/usr/libexec/httpd' \ + '/usr/lib/apache2' do if test -d "$DEFAULT_HTTPD_MODULE_PATH" then From patchwork Mon Oct 30 12:15:14 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13440536 Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5D91C15E8C for ; Mon, 30 Oct 2023 12:15:22 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b="HwKZeQXf"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="LcEjOIv6" Received: from wout4-smtp.messagingengine.com (wout4-smtp.messagingengine.com [64.147.123.20]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5CBCDD6 for ; Mon, 30 Oct 2023 05:15:19 -0700 (PDT) Received: from compute5.internal (compute5.nyi.internal [10.202.2.45]) by mailout.west.internal (Postfix) with ESMTP id 8E86C320093A; Mon, 30 Oct 2023 08:15:18 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute5.internal (MEProxy); Mon, 30 Oct 2023 08:15:18 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:sender :subject:subject:to:to; s=fm3; t=1698668118; x=1698754518; bh=ju aHphGLcO+NQUb6XYg+mtkW09uSuNdCY1Kf0c0RZUk=; b=HwKZeQXfZj1tSeQxbE 3zjakhqRcnBs1XwM2FLxzFL8iKPyHE4TCWtOuQcZakjAsX+lIq/s4PCt1+4RlOEY GTzzYSbm7teeYNTFkwUxVpf8n5KmDfoXSFGCOZo1x93ozTn8HvHP5DXDYity8XC+ pFNeA1zwjHTlXjXGhAbH5TNlvyrvVgnSG1Bu9aud9Y0+5Hnvb+7opXfPNHK73PjL Tb17x+rrlSlkoDYs2bIaYN2LPgmRLBs/GT1+37N5GqCPHi+WKU3OBRVXzXw3ltvH NS/tBjrMlIdoayQU2uEEwTPrB/xqgfT2oHYN3JlgDev5HpeGFKClf83q7FZ3XBp8 pexQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:sender:subject :subject:to:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm3; t=1698668118; x=1698754518; bh=juaHphGLcO+NQ Ub6XYg+mtkW09uSuNdCY1Kf0c0RZUk=; b=LcEjOIv63eP43kVdlQQR650TZdCuN rB7lR/Dh8on9Tg1QHn6peNH6EOO6ZfkuOs25VsHrQQmPZIYfRQquZYOUaX5TiROv 5SFB8GpUhgGgQE1LUk0wZW+kloMoXHr16mhya3uuSE+TulqthJL+gKFvQbAV2f5r xITqt/3p1sAn09zvAfgF0HBr5f8rGNy9inpp9zYKyGvQOq1szz0R/sHXdenklgM7 jU056dH5v/eP+xV7CZsRQ9iqFYGiJz8P0nEqDBBCcmatgZbI5nj9pPwsHy/IUEBb ptFE5L0/vIT4g72QVN/poqXcri7kLOVMIkCk6isqFvqbqtbZD1r+iFIeQ== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvkedruddttddgfeehucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhepfffhvfevuffkfhggtggujgesghdtreertddtvdenucfhrhhomheprfgrthhr ihgtkhcuufhtvghinhhhrghrughtuceophhssehpkhhsrdhimheqnecuggftrfgrthhtvg hrnheplefgiedtffekvdetffegudeuvdfftdffheetfeelffevvdffhedvffegleeggeei necuffhomhgrihhnpehgihhtlhgrsgdrtghomhenucevlhhushhtvghrufhiiigvpedtne curfgrrhgrmhepmhgrihhlfhhrohhmpehpshesphhkshdrihhm X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 30 Oct 2023 08:15:17 -0400 (EDT) Received: by vm-mail (OpenSMTPD) with ESMTPSA id fdd663ce (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Mon, 30 Oct 2023 12:15:10 +0000 (UTC) Date: Mon, 30 Oct 2023 13:15:14 +0100 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Junio C Hamano , Phillip Wood , Oswald Buddenhagen Subject: [PATCH v3 8/8] ci: add support for GitLab CI Message-ID: <5272d66d9f138987def342ff7661515b845318bb.1698667545.git.ps@pks.im> References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: We already support Azure Pipelines and GitHub Workflows in the Git project, but until now we do not have support for GitLab CI. While it is arguably not in the interest of the Git project to maintain a ton of different CI platforms, GitLab has recently ramped up its efforts and tries to contribute to the Git project more regularly. Part of a problem we hit at GitLab rather frequently is that our own, custom CI setup we have is so different to the setup that the Git project has. More esoteric jobs like "linux-TEST-vars" that also set a couple of environment variables do not exist in GitLab's custom CI setup, and maintaining them to keep up with what Git does feels like wasted time. The result is that we regularly send patch series upstream that fail to compile or pass tests in GitHub Workflows. We would thus like to integrate the GitLab CI configuration into the Git project to help us send better patch series upstream and thus reduce overhead for the maintainer. The integration does not necessarily have to be a first-class citizen, which would in practice only add to the fallout that pipeline failures have for the maintainer. That being said, we are happy to maintain this alternative CI setup for the Git project and will make test results available as part of our own mirror of the Git project at [1]. This commit introduces the integration into our regular CI scripts so that most of the setup continues to be shared across all of the CI solutions. Note that as the builds on GitLab CI run as unprivileged user, we need to pull in both sudo and shadow packages to our Alpine based job to set this up. [1]: https://gitlab.com/gitlab-org/git Signed-off-by: Patrick Steinhardt --- .gitlab-ci.yml | 53 +++++++++++++++++++++++++++++++ ci/install-docker-dependencies.sh | 13 +++++++- ci/lib.sh | 43 +++++++++++++++++++++++++ ci/print-test-failures.sh | 6 ++++ 4 files changed, 114 insertions(+), 1 deletion(-) create mode 100644 .gitlab-ci.yml diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml new file mode 100644 index 00000000000..cd98bcb18aa --- /dev/null +++ b/.gitlab-ci.yml @@ -0,0 +1,53 @@ +default: + timeout: 2h + +workflow: + rules: + - if: $CI_PIPELINE_SOURCE == "merge_request_event" + - if: $CI_COMMIT_TAG + - if: $CI_COMMIT_REF_PROTECTED == "true" + +test: + image: $image + before_script: + - ./ci/install-docker-dependencies.sh + script: + - useradd builder --create-home + - chown -R builder "${CI_PROJECT_DIR}" + - sudo --preserve-env --set-home --user=builder ./ci/run-build-and-tests.sh + after_script: + - | + if test "$CI_JOB_STATUS" != 'success' + then + sudo --preserve-env --set-home --user=builder ./ci/print-test-failures.sh + fi + parallel: + matrix: + - jobname: linux-sha256 + image: ubuntu:latest + CC: clang + - jobname: linux-gcc + image: ubuntu:20.04 + CC: gcc + CC_PACKAGE: gcc-8 + - jobname: linux-TEST-vars + image: ubuntu:20.04 + CC: gcc + CC_PACKAGE: gcc-8 + - jobname: linux-gcc-default + image: ubuntu:latest + CC: gcc + - jobname: linux-leaks + image: ubuntu:latest + CC: gcc + - jobname: linux-asan-ubsan + image: ubuntu:latest + CC: clang + - jobname: pedantic + image: fedora:latest + - jobname: linux-musl + image: alpine:latest + artifacts: + paths: + - t/failed-test-artifacts + when: on_failure diff --git a/ci/install-docker-dependencies.sh b/ci/install-docker-dependencies.sh index 05dde5c5d40..5e28adf55b6 100755 --- a/ci/install-docker-dependencies.sh +++ b/ci/install-docker-dependencies.sh @@ -7,6 +7,9 @@ begin_group "Install dependencies" +# Required so that apt doesn't wait for user input on certain packages. +export DEBIAN_FRONTEND=noninteractive + case "$jobname" in linux32) linux32 --32bit i386 sh -c ' @@ -16,10 +19,18 @@ linux32) ' ;; linux-musl) - apk add --update build-base curl-dev openssl-dev expat-dev gettext \ + apk add --update shadow sudo build-base curl-dev openssl-dev expat-dev gettext \ pcre2-dev python3 musl-libintl perl-utils ncurses \ apache2 bash cvs gnupg perl-cgi perl-dbd-sqlite >/dev/null ;; +linux-*) + apt update -q && + apt install -q -y sudo git make language-pack-is libsvn-perl apache2 libssl-dev \ + libcurl4-openssl-dev libexpat-dev tcl tk gettext zlib1g-dev \ + perl-modules liberror-perl libauthen-sasl-perl libemail-valid-perl \ + libdbd-sqlite3-perl libio-socket-ssl-perl libnet-smtp-ssl-perl ${CC_PACKAGE:-${CC:-gcc}} \ + apache2 cvs cvsps gnupg libcgi-pm-perl subversion + ;; pedantic) dnf -yq update >/dev/null && dnf -yq install make gcc findutils diffutils perl python3 gettext zlib-devel expat-devel openssl-devel curl-devel pcre2-devel >/dev/null diff --git a/ci/lib.sh b/ci/lib.sh index c13b6527cac..d836fad7dd6 100755 --- a/ci/lib.sh +++ b/ci/lib.sh @@ -14,6 +14,22 @@ then need_to_end_group= echo '::endgroup::' >&2 } +elif test true = "$GITLAB_CI" +then + begin_group () { + need_to_end_group=t + printf "\e[0Ksection_start:$(date +%s):$(echo "$1" | tr ' ' _)\r\e[0K$1\n" + trap "end_group '$1'" EXIT + set -x + } + + end_group () { + test -n "$need_to_end_group" || return 0 + set +x + need_to_end_group= + printf "\e[0Ksection_end:$(date +%s):$(echo "$1" | tr ' ' _)\r\e[0K\n" + trap - EXIT + } else begin_group () { :; } end_group () { :; } @@ -227,6 +243,33 @@ then cache_dir="$HOME/none" GIT_TEST_OPTS="--github-workflow-markup" +elif test true = "$GITLAB_CI" +then + CI_TYPE=gitlab-ci + CI_BRANCH="$CI_COMMIT_REF_NAME" + CI_COMMIT="$CI_COMMIT_SHA" + case "$CI_JOB_IMAGE" in + macos-*) + CI_OS_NAME=osx;; + alpine:*|fedora:*|ubuntu:*) + CI_OS_NAME=linux;; + *) + echo "Could not identify OS image" >&2 + env >&2 + exit 1 + ;; + esac + CI_REPO_SLUG="$CI_PROJECT_PATH" + CI_JOB_ID="$CI_JOB_ID" + CC="${CC_PACKAGE:-${CC:-gcc}}" + DONT_SKIP_TAGS=t + handle_failed_tests () { + create_failed_test_artifacts + } + + cache_dir="$HOME/none" + + runs_on_pool=$(echo "$CI_JOB_IMAGE" | tr : -) else echo "Could not identify CI type" >&2 env >&2 diff --git a/ci/print-test-failures.sh b/ci/print-test-failures.sh index 57277eefcd0..c33ad4e3a22 100755 --- a/ci/print-test-failures.sh +++ b/ci/print-test-failures.sh @@ -51,6 +51,12 @@ do tar czf failed-test-artifacts/"$test_name".trash.tar.gz "$trash_dir" continue ;; + gitlab-ci) + mkdir -p failed-test-artifacts + cp "${TEST_EXIT%.exit}.out" failed-test-artifacts/ + tar czf failed-test-artifacts/"$test_name".trash.tar.gz "$trash_dir" + continue + ;; *) echo "Unhandled CI type: $CI_TYPE" >&2 exit 1