From patchwork Tue Oct 31 09:04:41 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13441300 Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4ED8A12B79 for ; Tue, 31 Oct 2023 09:04:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b="B6o708lU"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="dHB3EIv7" Received: from wout1-smtp.messagingengine.com (wout1-smtp.messagingengine.com [64.147.123.24]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2FE3CB3 for ; Tue, 31 Oct 2023 02:04:46 -0700 (PDT) Received: from compute2.internal (compute2.nyi.internal [10.202.2.46]) by mailout.west.internal (Postfix) with ESMTP id 2D3A13200A06; Tue, 31 Oct 2023 05:04:45 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute2.internal (MEProxy); Tue, 31 Oct 2023 05:04:45 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:sender :subject:subject:to:to; s=fm3; t=1698743084; x=1698829484; bh=6v T9Pny6vRasIsYaESvuM4GMnh4WqCUwuyRpLuXp6tU=; b=B6o708lU6Ca34fu8v9 DZt7UI2BErPwFqcobqoIiyu3+iS7OxMGKgpmxcOSrMCqEQ5UJ32g4cxIACgHQXag Hy15jrmpks8l2zj9SkVAR8/dTuLoLgv0NSV9G4UxPEpGPLAKvRx3QAdsxd8uXv2O RDVSYQ7gS8JeO27j7EV8y6lHksFdueSSA68+ZJXaMTAZIZRXBkEDwkzR1Ryrk2tr vvZ6JQoIGmGLpbGbFZACzfVrtZKhZ05ebcwum9ILtM0QHY0tXDOjG++X7jxkh1YM qxVMg0PyJft8neuzFXI9sstG/ZhQXHoQ8UYNsfXxlJ+Usb6N0boExuJy1ZuIId6F 5QVA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:sender:subject :subject:to:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm3; t=1698743084; x=1698829484; bh=6vT9Pny6vRasI sYaESvuM4GMnh4WqCUwuyRpLuXp6tU=; b=dHB3EIv7CxcSTGjwv37nbNKMqUgcO vgem7BvHexg/ZqdX3/ULgg6DyjhlkZZPo4DIj+bU6HF2vZg6Mg1lnO2x2p156v+A NwmPeSR5s4nGpJJ9x810NadhM40ajgJ9gMlTs7DLccTLFTQgBCePrIIz2cufsmjv OB1cIGqYDVKvornB2dKYiI7WqBHaFbAhp1hds9QQk3rH9kmDvDe2CbqoDdXyu8wJ UWILKXYv5ISQfOZ5hZGJj3B9708luiU0DGfDg5/eJBuVc0pr2RYXIwpKETvuE5fm IJN0OdAOyJABJZNxJIj1E3J6BTXKyM18XuaZLQhqEtyzPbB+Yym0QQztw== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvkedruddtuddguddvgecutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfgh necuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmd enucfjughrpeffhffvvefukfhfgggtuggjsehgtderredttddvnecuhfhrohhmpefrrght rhhitghkucfuthgvihhnhhgrrhguthcuoehpshesphhkshdrihhmqeenucggtffrrghtth gvrhhnpeeukedtvedtffevleejtefgheehieegkeeluddvfeefgeehgfeltddtheejleff teenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpehpsh esphhkshdrihhm X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Tue, 31 Oct 2023 05:04:43 -0400 (EDT) Received: by vm-mail (OpenSMTPD) with ESMTPSA id a6678cc8 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Tue, 31 Oct 2023 09:04:36 +0000 (UTC) Date: Tue, 31 Oct 2023 10:04:41 +0100 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Taylor Blau , Junio C Hamano , Phillip Wood , Oswald Buddenhagen Subject: [PATCH v4 1/8] ci: reorder definitions for grouping functions Message-ID: <8595fe5016ab99ac3818e132c787303e88bcef84.1698742590.git.ps@pks.im> References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: We define a set of grouping functions that are used to group together output in our CI, where these groups then end up as collapsible sections in the respective pipeline platform. The way these functions are defined is not easily extensible though as we have an up front check for the CI _not_ being GitHub Actions, where we define the non-stub logic in the else branch. Reorder the conditional branches such that we explicitly handle GitHub Actions. Signed-off-by: Patrick Steinhardt --- ci/lib.sh | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/ci/lib.sh b/ci/lib.sh index 6fbb5bade12..eb384f4e952 100755 --- a/ci/lib.sh +++ b/ci/lib.sh @@ -1,16 +1,7 @@ # Library of functions shared by all CI scripts -if test true != "$GITHUB_ACTIONS" +if test true = "$GITHUB_ACTIONS" then - begin_group () { :; } - end_group () { :; } - - group () { - shift - "$@" - } - set -x -else begin_group () { need_to_end_group=t echo "::group::$1" >&2 @@ -42,6 +33,15 @@ else } begin_group "CI setup" +else + begin_group () { :; } + end_group () { :; } + + group () { + shift + "$@" + } + set -x fi # Set 'exit on error' for all CI scripts to let the caller know that From patchwork Tue Oct 31 09:04:46 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13441301 Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 39D4013AC6 for ; Tue, 31 Oct 2023 09:04:52 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b="o/XCSOQl"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="gV+kWKzl" Received: from wout1-smtp.messagingengine.com (wout1-smtp.messagingengine.com [64.147.123.24]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 86451A2 for ; Tue, 31 Oct 2023 02:04:50 -0700 (PDT) Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailout.west.internal (Postfix) with ESMTP id 9C7563200077; Tue, 31 Oct 2023 05:04:49 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute1.internal (MEProxy); Tue, 31 Oct 2023 05:04:50 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:sender :subject:subject:to:to; s=fm3; t=1698743089; x=1698829489; bh=o+ /fVWUHURDneiMahar9ykRw0+Xn8DG6uPcEBqOXIjg=; b=o/XCSOQlTeCBDUhAPX VBsLTen+FsbU0ES6YvrcnDBOI3BRcKs7G/QPo9LDRO6MlpUKQBXW5vemqrp6aRnY jXSTJel71YzXbs4tyZcWGFKbS3uTQHDXxceGX+s8S01eu2cMuRneDL/U0GBbHvpf bTB059tJcmsOgjkHEbe7zU8wNRZds4whsl28c3uBZRt402iZdfa4JKZWcAh2eTbv nj50OvKaJoUHufkPal1hvyhVywKaqerx4Ict2NLu4z2VpfAjsLHmzBFhPmbhE1vl fdVKpxnfpcnO+t+Eq7SRd0PaVs9EzE/apGl4S2GmQFHyydxeT902AeZbZHAt8ZNf 4VqA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:sender:subject :subject:to:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm3; t=1698743089; x=1698829489; bh=o+/fVWUHURDne iMahar9ykRw0+Xn8DG6uPcEBqOXIjg=; b=gV+kWKzlgzf202lBYS0h1qANi4aoS S5487x8/MTVbXWgFhSqn1fkaY30Zv/9/PH4YRaATIORMJl8SMApAaD1ViI2o/TnS 0f7yRJDGl7JuvofhPI3oB/5QgNraE2vP8Xjh6Qh2iCEWzU0WAEJ7ueECMAWYB3sA 9IWn+KGLvCM9Fs0IFarA2ctcf3rM1ZeCO+iq1ZFo4Yd4aTasMNhuwjzwkTO0IIJB CAqDsXRpRDQWaGr9sXX3SB8C4lnV7DpDOsClQaRAHhYb0ckAgOy3Z85wZKEUftRY qxwVgABeaIMjRqQ97G9seuaDoMtZtxnrf9TOUNc1JyqcfZBWDd5PGdEew== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvkedruddtuddguddvhecutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfgh necuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmd enucfjughrpeffhffvvefukfhfgggtuggjsehgtderredttddvnecuhfhrohhmpefrrght rhhitghkucfuthgvihhnhhgrrhguthcuoehpshesphhkshdrihhmqeenucggtffrrghtth gvrhhnpeeukedtvedtffevleejtefgheehieegkeeluddvfeefgeehgfeltddtheejleff teenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpehpsh esphhkshdrihhm X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Tue, 31 Oct 2023 05:04:47 -0400 (EDT) Received: by vm-mail (OpenSMTPD) with ESMTPSA id c894318e (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Tue, 31 Oct 2023 09:04:40 +0000 (UTC) Date: Tue, 31 Oct 2023 10:04:46 +0100 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Taylor Blau , Junio C Hamano , Phillip Wood , Oswald Buddenhagen Subject: [PATCH v4 2/8] ci: make grouping setup more generic Message-ID: <7358a9433927aeb61cd9dac1b8ae651f13e11f5d.1698742590.git.ps@pks.im> References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Make the grouping setup more generic by always calling `begin_group ()` and `end_group ()` regardless of whether we have stubbed those functions or not. This ensures we can more readily add support for additional CI platforms. Furthermore, the `group ()` function is made generic so that it is the same for both GitHub Actions and for other platforms. There is a semantic conflict here though: GitHub Actions used to call `set +x` in `group ()` whereas the non-GitHub case unconditionally uses `set -x`. The latter would get overriden if we kept the `set +x` in the generic version of `group ()`. To resolve this conflict, we simply drop the `set +x` in the generic variant of this function. As `begin_group ()` calls `set -x` anyway this is not much of a change though, as the only commands that aren't printed anymore now are the ones between the beginning of `group ()` and the end of `begin_group ()`. Last, this commit changes `end_group ()` to also accept a parameter that indicates _which_ group should end. This will be required by a later commit that introduces support for GitLab CI. Signed-off-by: Patrick Steinhardt --- ci/lib.sh | 46 ++++++++++++++++++++++------------------------ 1 file changed, 22 insertions(+), 24 deletions(-) diff --git a/ci/lib.sh b/ci/lib.sh index eb384f4e952..b3411afae8e 100755 --- a/ci/lib.sh +++ b/ci/lib.sh @@ -14,36 +14,34 @@ then need_to_end_group= echo '::endgroup::' >&2 } - trap end_group EXIT - - group () { - set +x - begin_group "$1" - shift - # work around `dash` not supporting `set -o pipefail` - ( - "$@" 2>&1 - echo $? >exit.status - ) | - sed 's/^\(\([^ ]*\):\([0-9]*\):\([0-9]*:\) \)\(error\|warning\): /::\5 file=\2,line=\3::\1/' - res=$(cat exit.status) - rm exit.status - end_group - return $res - } - - begin_group "CI setup" else begin_group () { :; } end_group () { :; } - group () { - shift - "$@" - } set -x fi +group () { + group="$1" + shift + begin_group "$group" + + # work around `dash` not supporting `set -o pipefail` + ( + "$@" 2>&1 + echo $? >exit.status + ) | + sed 's/^\(\([^ ]*\):\([0-9]*\):\([0-9]*:\) \)\(error\|warning\): /::\5 file=\2,line=\3::\1/' + res=$(cat exit.status) + rm exit.status + + end_group "$group" + return $res +} + +begin_group "CI setup" +trap "end_group 'CI setup'" EXIT + # Set 'exit on error' for all CI scripts to let the caller know that # something went wrong. # @@ -287,5 +285,5 @@ esac MAKEFLAGS="$MAKEFLAGS CC=${CC:-cc}" -end_group +end_group "CI setup" set -x From patchwork Tue Oct 31 09:04:50 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13441302 Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2123513AC6 for ; Tue, 31 Oct 2023 09:04:56 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b="fRpOV336"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="FD2EUEaJ" Received: from wout1-smtp.messagingengine.com (wout1-smtp.messagingengine.com [64.147.123.24]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id EC54DA2 for ; Tue, 31 Oct 2023 02:04:54 -0700 (PDT) Received: from compute2.internal (compute2.nyi.internal [10.202.2.46]) by mailout.west.internal (Postfix) with ESMTP id 15A8432001FC; Tue, 31 Oct 2023 05:04:54 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute2.internal (MEProxy); Tue, 31 Oct 2023 05:04:54 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:sender :subject:subject:to:to; s=fm3; t=1698743093; x=1698829493; bh=qa 4CdCuRnvenfmxGO+tLqy6c3DBmf/3+B1n98/Uf7WE=; b=fRpOV336Ez+4IMUJ1K zsm5unaQ//s0YoGJ/MdY7VIILkox0oLQK69mpo66ffteuMXDw3chTYKmhMG5aCeW HkeOve3L3tlZl6PwVNOAOdcJkqF7EziCMtg4G7zgU4IjUJrATtzEtqPejoTgyOTP XF0mg7x5Sm4jByuZJu3lG012JzCqUVYoOg69lNHCO2PCxnUX0hPqJnRsM6Id5gsX Nwkuzr8X+9D5Vw+idX0pRaNouD22lWY2tAEBxMEiIz9PXYoe1nCTREMKY+i6iZGc GzUMXe9j7x3mZOsJdb1YBz8oOCyKsJK8Q+Zf547POL1vb4E/c2qUeIpd1dk+K6fl tUNg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:sender:subject :subject:to:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm3; t=1698743093; x=1698829493; bh=qa4CdCuRnvenf mxGO+tLqy6c3DBmf/3+B1n98/Uf7WE=; b=FD2EUEaJKW6+aahDi9PjmVJxOH963 VoAoJNr4uG8cxehhOBNQVxXx+0JEZABOOvA5nrgdyUWpaSswacIWQd/fczOellqI yqwQKZVBHyZdxjkcbD0BViQ7l6eXec+WRrI5W6RNJYCnkJQHnbUdjyAw40SKkZZ/ WRk9p/aGnZOVecU9RXii8hKC52/437EzhpBafYHSjWykRRSrJcx9wsaJbnqcf/CY 4mHRhDRVRTThRN8QQA3sMc2VkJ9naAmonrnZZU+ZZ8mSbingEDq10QYUK1tYPxOa TPh3YlFnecCAFM9d9+5I39ECGNu8MrK0XHgMiFTUQnV+YP0IayFxoZSZQ== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvkedruddtuddguddvgecutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfgh necuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmd enucfjughrpeffhffvvefukfhfgggtuggjsehgtderredttddvnecuhfhrohhmpefrrght rhhitghkucfuthgvihhnhhgrrhguthcuoehpshesphhkshdrihhmqeenucggtffrrghtth gvrhhnpeeukedtvedtffevleejtefgheehieegkeeluddvfeefgeehgfeltddtheejleff teenucevlhhushhtvghrufhiiigvpedunecurfgrrhgrmhepmhgrihhlfhhrohhmpehpsh esphhkshdrihhm X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Tue, 31 Oct 2023 05:04:52 -0400 (EDT) Received: by vm-mail (OpenSMTPD) with ESMTPSA id c163f161 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Tue, 31 Oct 2023 09:04:44 +0000 (UTC) Date: Tue, 31 Oct 2023 10:04:50 +0100 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Taylor Blau , Junio C Hamano , Phillip Wood , Oswald Buddenhagen Subject: [PATCH v4 3/8] ci: group installation of Docker dependencies Message-ID: <6d842592c6fb085b40997466d117637742de44d6.1698742590.git.ps@pks.im> References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: The output of CI jobs tends to be quite long-winded and hard to digest. To help with this, many CI systems provide the ability to group output into collapsible sections, and we're also doing this in some of our scripts. One notable omission is the script to install Docker dependencies. Address it to bring more structure to the output for Docker-based jobs. Signed-off-by: Patrick Steinhardt --- ci/install-docker-dependencies.sh | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/ci/install-docker-dependencies.sh b/ci/install-docker-dependencies.sh index 78b7e326da6..d0bc19d3bb3 100755 --- a/ci/install-docker-dependencies.sh +++ b/ci/install-docker-dependencies.sh @@ -3,6 +3,10 @@ # Install dependencies required to build and test Git inside container # +. ${0%/*}/lib.sh + +begin_group "Install dependencies" + case "$jobname" in linux32) linux32 --32bit i386 sh -c ' @@ -20,3 +24,5 @@ pedantic) dnf -yq install make gcc findutils diffutils perl python3 gettext zlib-devel expat-devel openssl-devel curl-devel pcre2-devel >/dev/null ;; esac + +end_group "Install dependencies" From patchwork Tue Oct 31 09:04:54 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13441303 Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A982B168B7 for ; Tue, 31 Oct 2023 09:04:59 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b="IuXKLp3A"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="k1BtKeuI" Received: from wout1-smtp.messagingengine.com (wout1-smtp.messagingengine.com [64.147.123.24]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 661B591 for ; Tue, 31 Oct 2023 02:04:58 -0700 (PDT) Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailout.west.internal (Postfix) with ESMTP id 80AD432001FC; Tue, 31 Oct 2023 05:04:57 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute3.internal (MEProxy); Tue, 31 Oct 2023 05:04:57 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:sender :subject:subject:to:to; s=fm3; t=1698743097; x=1698829497; bh=CB TC27/gk3SfAJlkjrofJ1X981YGe0tiPYnh8gyQi0U=; b=IuXKLp3APBeF7Cnmh4 9iUjq0IciS8sEJj8Dm1ACgC5iZjUJOG3LucnKCLPxyPh8akoWRS+C8yeJHnhdgS2 QlGcLLSoVNw1NhpsQau2n3v+3bEn//v+pl7Thz8YeR3/tkDixYUDj8Fk8e3fR2uC Qg+5XFN0seL61fFIjE2/ZM/Q5S2BYLSeubNeX8oEY38uwsiRtYo5qQ8rVl44hP+u IXqePzyCh0HdxdBZA90Jueg/3NvA7+grgEyBDiaVf1pL77lZ7fT0O4L6NQLTLWFv 3YqoTv3s0QuAJHhgEa8hAHmlcIGo8ID49WVUJbF77kYeV5A4ixpHsPyQnaW2L4F0 hDrg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:sender:subject :subject:to:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm3; t=1698743097; x=1698829497; bh=CBTC27/gk3SfA JlkjrofJ1X981YGe0tiPYnh8gyQi0U=; b=k1BtKeuI9Cqxqs3dbCFVco5BqfKv3 tUGDdWU3SZ31xRLuRvWyzvW2tmz3YDikz1b8yB9Xvs9Ll7APQyRaMQ35L8WRQqKk vNQHIYuODmgVRRogAe/EzQQUNAYuZfVv+EVXJA9awpqm0SouuN3GBfCLnE71j4wM noPOqaalm8+/UebSWVc4Ug9gNDqnR2rpyefIhJUoAiiT2deiYlveNlPN54pD8xYH O1YEFzCaOp215BekETd/AhCMmz8LC+uiigvLksItjNsTMjmaPjBpcGUn3GPdtJsc Sus4e1kgJwKn3qsL6nW9dzKhe0CNkxCFWNP8e1Bh1tx4pGgXRaoIok2eg== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvkedruddtuddguddvgecutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfgh necuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmd enucfjughrpeffhffvvefukfhfgggtuggjsehgtderredttddvnecuhfhrohhmpefrrght rhhitghkucfuthgvihhnhhgrrhguthcuoehpshesphhkshdrihhmqeenucggtffrrghtth gvrhhnpeeukedtvedtffevleejtefgheehieegkeeluddvfeefgeehgfeltddtheejleff teenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpehpsh esphhkshdrihhm X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Tue, 31 Oct 2023 05:04:55 -0400 (EDT) Received: by vm-mail (OpenSMTPD) with ESMTPSA id bd5ebe73 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Tue, 31 Oct 2023 09:04:48 +0000 (UTC) Date: Tue, 31 Oct 2023 10:04:54 +0100 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Taylor Blau , Junio C Hamano , Phillip Wood , Oswald Buddenhagen Subject: [PATCH v4 4/8] ci: split out logic to set up failed test artifacts Message-ID: References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: We have some logic in place to create a directory with the output from failed tests, which will then subsequently be uploaded as CI artifacts. We're about to add support for GitLab CI, which will want to reuse the logic. Split the logic into a separate function so that it is reusable. Signed-off-by: Patrick Steinhardt --- ci/lib.sh | 40 ++++++++++++++++++++++------------------ 1 file changed, 22 insertions(+), 18 deletions(-) diff --git a/ci/lib.sh b/ci/lib.sh index b3411afae8e..9ffdf743903 100755 --- a/ci/lib.sh +++ b/ci/lib.sh @@ -131,6 +131,27 @@ handle_failed_tests () { return 1 } +create_failed_test_artifacts () { + mkdir -p t/failed-test-artifacts + + for test_exit in t/test-results/*.exit + do + test 0 != "$(cat "$test_exit")" || continue + + test_name="${test_exit%.exit}" + test_name="${test_name##*/}" + printf "\\e[33m\\e[1m=== Failed test: ${test_name} ===\\e[m\\n" + echo "The full logs are in the 'print test failures' step below." + echo "See also the 'failed-tests-*' artifacts attached to this run." + cat "t/test-results/$test_name.markup" + + trash_dir="t/trash directory.$test_name" + cp "t/test-results/$test_name.out" t/failed-test-artifacts/ + tar czf t/failed-test-artifacts/"$test_name".trash.tar.gz "$trash_dir" + done + return 1 +} + # GitHub Action doesn't set TERM, which is required by tput export TERM=${TERM:-dumb} @@ -171,25 +192,8 @@ then CC="${CC_PACKAGE:-${CC:-gcc}}" DONT_SKIP_TAGS=t handle_failed_tests () { - mkdir -p t/failed-test-artifacts echo "FAILED_TEST_ARTIFACTS=t/failed-test-artifacts" >>$GITHUB_ENV - - for test_exit in t/test-results/*.exit - do - test 0 != "$(cat "$test_exit")" || continue - - test_name="${test_exit%.exit}" - test_name="${test_name##*/}" - printf "\\e[33m\\e[1m=== Failed test: ${test_name} ===\\e[m\\n" - echo "The full logs are in the 'print test failures' step below." - echo "See also the 'failed-tests-*' artifacts attached to this run." - cat "t/test-results/$test_name.markup" - - trash_dir="t/trash directory.$test_name" - cp "t/test-results/$test_name.out" t/failed-test-artifacts/ - tar czf t/failed-test-artifacts/"$test_name".trash.tar.gz "$trash_dir" - done - return 1 + create_failed_test_artifacts } cache_dir="$HOME/none" From patchwork Tue Oct 31 09:04:58 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13441304 Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5879C182A4 for ; Tue, 31 Oct 2023 09:05:04 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b="KaVf4Yaf"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="R7I19XNl" Received: from wout1-smtp.messagingengine.com (wout1-smtp.messagingengine.com [64.147.123.24]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E10A291 for ; Tue, 31 Oct 2023 02:05:02 -0700 (PDT) Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailout.west.internal (Postfix) with ESMTP id 06C473200979; Tue, 31 Oct 2023 05:05:01 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute1.internal (MEProxy); Tue, 31 Oct 2023 05:05:02 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:sender :subject:subject:to:to; s=fm3; t=1698743101; x=1698829501; bh=+z 8YuG/jUwTGAh1ZCpIH0dHYmgGp07sh9tJvJ0CAWIo=; b=KaVf4YafHqZwAggX2+ Bqp2msnR5XbaTMVGeqylJeJxrw8DAfwZJhgTay0t+GBZGkP0jdb2u+4+CbfAWX8k wEcrzLv+1d1lEeUkqrpZ2UjZjvZMoY0doC86wJ6529CbojL2rwne2ZftdWuaOE6A NYMuQchXdxVLiZAwage8V4SdNRVLwyPine8+zr6uwJF1Jb75Kzlr6fk+1XApMIs+ mEFLrLNOU9zLbmEyT+rlA26KHtGGvkRYzWGhpQxaIyY5lcSukE29Dwh2GlbwDb0x LqVDz8WdrfGoFTDjxgKxfxoR05Zw/th2LZ7U10elH5qqMnwauAx39FS50ScwkaE5 8lTg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:sender:subject :subject:to:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm3; t=1698743101; x=1698829501; bh=+z8YuG/jUwTGA h1ZCpIH0dHYmgGp07sh9tJvJ0CAWIo=; b=R7I19XNlIlB5/sBUcPe9benh7Qs4C GowCa7quuIizg0Asrt9LSgg6bxYssKoT347jGM7Nq/fc9fYXJ/n81Gf1l3edRWBh M0DfeFAhBZpsfNTdPis110yqRgfRO0A6vtRnHWdd7vzEXDmk3084sx8a7rbeCFzm DJEczT0xrX9LrMOY6N+pjVNZnDlyhnOeGBnTwIjXqmAAlvMhhX59aEXfrwgHOSIJ +g+wL6BKTrN9wCByrsAAj35JbJCMxcCWB4jrUKZLSXeaqkG9NzpNAtg3OYUEeuPF VXUr5kh7+z65mewQLoQ5SKqHrZY7toeiNPg+9IYqp3IGq04l0voRF3I7w== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvkedruddtuddguddvhecutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfgh necuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmd enucfjughrpeffhffvvefukfhfgggtuggjsehgtderredttddvnecuhfhrohhmpefrrght rhhitghkucfuthgvihhnhhgrrhguthcuoehpshesphhkshdrihhmqeenucggtffrrghtth gvrhhnpeeukedtvedtffevleejtefgheehieegkeeluddvfeefgeehgfeltddtheejleff teenucevlhhushhtvghrufhiiigvpedunecurfgrrhgrmhepmhgrihhlfhhrohhmpehpsh esphhkshdrihhm X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Tue, 31 Oct 2023 05:05:00 -0400 (EDT) Received: by vm-mail (OpenSMTPD) with ESMTPSA id e492443f (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Tue, 31 Oct 2023 09:04:52 +0000 (UTC) Date: Tue, 31 Oct 2023 10:04:58 +0100 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Taylor Blau , Junio C Hamano , Phillip Wood , Oswald Buddenhagen Subject: [PATCH v4 5/8] ci: unify setup of some environment variables Message-ID: References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Both GitHub Actions and Azue Pipelines set up the environment variables GIT_TEST_OPTS, GIT_PROVE_OPTS and MAKEFLAGS. And while most values are actually the same, the setup is completely duplicate. With the upcoming support for GitLab CI this duplication would only extend even further. Unify the setup of those environment variables so that only the uncommon parts are separated. While at it, we also perform some additional small improvements: - We now always pass `--state=failed,slow,save` via GIT_PROVE_OPTS. It doesn't hurt on platforms where we don't persist the state, so this further reduces boilerplate. - When running on Windows systems we set `--no-chain-lint` and `--no-bin-wrappers`. Interestingly though, we did so _after_ already having exported the respective environment variables. - We stop using `export VAR=value` syntax, which is a Bashism. It's not quite worth it as we still use this syntax all over the place, but it doesn't hurt readability either. Signed-off-by: Patrick Steinhardt --- ci/lib.sh | 24 ++++++++++++++---------- 1 file changed, 14 insertions(+), 10 deletions(-) diff --git a/ci/lib.sh b/ci/lib.sh index 9ffdf743903..9a9b92c05b3 100755 --- a/ci/lib.sh +++ b/ci/lib.sh @@ -175,11 +175,7 @@ then # among *all* phases) cache_dir="$HOME/test-cache/$SYSTEM_PHASENAME" - export GIT_PROVE_OPTS="--timer --jobs 10 --state=failed,slow,save" - export GIT_TEST_OPTS="--verbose-log -x --write-junit-xml" - MAKEFLAGS="$MAKEFLAGS --jobs=10" - test windows_nt != "$CI_OS_NAME" || - GIT_TEST_OPTS="--no-chain-lint --no-bin-wrappers $GIT_TEST_OPTS" + GIT_TEST_OPTS="--write-junit-xml" elif test true = "$GITHUB_ACTIONS" then CI_TYPE=github-actions @@ -198,17 +194,25 @@ then cache_dir="$HOME/none" - export GIT_PROVE_OPTS="--timer --jobs 10" - export GIT_TEST_OPTS="--verbose-log -x --github-workflow-markup" - MAKEFLAGS="$MAKEFLAGS --jobs=10" - test windows != "$CI_OS_NAME" || - GIT_TEST_OPTS="--no-chain-lint --no-bin-wrappers $GIT_TEST_OPTS" + GIT_TEST_OPTS="--github-workflow-markup" else echo "Could not identify CI type" >&2 env >&2 exit 1 fi +MAKEFLAGS="$MAKEFLAGS --jobs=10" +GIT_PROVE_OPTS="--timer --jobs 10 --state=failed,slow,save" + +GIT_TEST_OPTS="$GIT_TEST_OPTS --verbose-log -x" +if test windows = "$CI_OS_NAME" +then + GIT_TEST_OPTS="$GIT_TEST_OPTS --no-chain-lint --no-bin-wrappers" +fi + +export GIT_TEST_OPTS +export GIT_PROVE_OPTS + good_trees_file="$cache_dir/good-trees" mkdir -p "$cache_dir" From patchwork Tue Oct 31 09:05:02 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13441305 Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 97FF118C17 for ; Tue, 31 Oct 2023 09:05:08 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b="lvTQZIch"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="Wh9ve5JD" Received: from wout1-smtp.messagingengine.com (wout1-smtp.messagingengine.com [64.147.123.24]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4D35E91 for ; Tue, 31 Oct 2023 02:05:07 -0700 (PDT) Received: from compute6.internal (compute6.nyi.internal [10.202.2.47]) by mailout.west.internal (Postfix) with ESMTP id 6E59932009AB; Tue, 31 Oct 2023 05:05:06 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute6.internal (MEProxy); Tue, 31 Oct 2023 05:05:06 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:sender :subject:subject:to:to; s=fm3; t=1698743105; x=1698829505; bh=KQ 0aEE34Aj/6/S03u89yypI1E3Y+iieXBmnRz8N2H3M=; b=lvTQZIchDXSRVKjkav I/667USh0SBSlSSAGQFB88ao776uZdwLpI6d1EhflBC5jWIy1VMSNrGmIebDPgUF 4szy9Mw80nTB5QUAbK9mIiq70QJO07IYETEKOeMbzC9UsZ7f+CunujPJS/d55LuD qjzXPBD9b1lUXD13JgVyvxA9l4QXyGDYv6araBXOUkzE/Rlfzp6dyfHW1auEfWPu A7ZFd51Y+IHDvMF6VPj462so9hsGLafFxDXYjbEAf/hMiMXF4jBFbkq5kcar4UDb C0e8wAS+/sRn+/+t5F9wkmV8Sef2le+QGpQzbrlBzSyO89Mq/mNicaHYz55kDTyb GD7w== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:sender:subject :subject:to:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm3; t=1698743105; x=1698829505; bh=KQ0aEE34Aj/6/ S03u89yypI1E3Y+iieXBmnRz8N2H3M=; b=Wh9ve5JDOC/pGwUmqQ6EvZ4XcApHr hs8AEz8z/K3Z8JsKb1sgd8thDyzoLh2J4SI4OU1q3xt7FBtpFVX6JiFWogjNbYD9 RyTMPptulI4LgKEsd0IE58B+MiZBk1Fi5b4BjQZ0qXitNxL1+zOOqitMWmxZ59oA Uc2ZQQezmJFHTjRoehVU21XDvyGhr2y8kTUVSQ/53jowhZzONZECaH8znKsFkIgi bPZubosigr06fdlVtceCmJJXkGf1PrOscAGQdyBZ8b3PyZY7jbeqV0bDuRZNFD53 V648aRmAwXt+MqIHyU4Ttv2mlF+f2+NmQcCoADq2XOVIApRuQDdaxiAXg== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvkedruddtuddguddvgecutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfgh necuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmd enucfjughrpeffhffvvefukfhfgggtuggjsehgtderredttddvnecuhfhrohhmpefrrght rhhitghkucfuthgvihhnhhgrrhguthcuoehpshesphhkshdrihhmqeenucggtffrrghtth gvrhhnpeeukedtvedtffevleejtefgheehieegkeeluddvfeefgeehgfeltddtheejleff teenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpehpsh esphhkshdrihhm X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Tue, 31 Oct 2023 05:05:04 -0400 (EDT) Received: by vm-mail (OpenSMTPD) with ESMTPSA id 89d1653d (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Tue, 31 Oct 2023 09:04:57 +0000 (UTC) Date: Tue, 31 Oct 2023 10:05:02 +0100 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Taylor Blau , Junio C Hamano , Phillip Wood , Oswald Buddenhagen Subject: [PATCH v4 6/8] ci: squelch warnings when testing with unusable Git repo Message-ID: References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Our CI jobs that run on Docker also use mostly the same architecture to build and test Git via the "ci/run-build-and-tests.sh" script. These scripts also provide some functionality to massage the Git repository we're supposedly operating in. In our Docker-based infrastructure we may not even have a Git repository available though, which leads to warnings when those functions execute. Make the helpers exit gracefully in case either there is no Git in our PATH, or when not running in a Git repository. Signed-off-by: Patrick Steinhardt --- ci/lib.sh | 32 ++++++++++++++++++++++++++++++++ 1 file changed, 32 insertions(+) diff --git a/ci/lib.sh b/ci/lib.sh index 9a9b92c05b3..e14b1029fad 100755 --- a/ci/lib.sh +++ b/ci/lib.sh @@ -69,10 +69,32 @@ skip_branch_tip_with_tag () { fi } +# Check whether we can use the path passed via the first argument as Git +# repository. +is_usable_git_repository () { + # We require Git in our PATH, otherwise we cannot access repositories + # at all. + if ! command -v git >/dev/null + then + return 1 + fi + + # And the target directory needs to be a proper Git repository. + if ! git -C "$1" rev-parse 2>/dev/null + then + return 1 + fi +} + # Save some info about the current commit's tree, so we can skip the build # job if we encounter the same tree again and can provide a useful info # message. save_good_tree () { + if ! is_usable_git_repository . + then + return + fi + echo "$(git rev-parse $CI_COMMIT^{tree}) $CI_COMMIT $CI_JOB_NUMBER $CI_JOB_ID" >>"$good_trees_file" # limit the file size tail -1000 "$good_trees_file" >"$good_trees_file".tmp @@ -88,6 +110,11 @@ skip_good_tree () { return fi + if ! is_usable_git_repository . + then + return + fi + if ! good_tree_info="$(grep "^$(git rev-parse $CI_COMMIT^{tree}) " "$good_trees_file")" then # Haven't seen this tree yet, or no cached good trees file yet. @@ -119,6 +146,11 @@ skip_good_tree () { } check_unignored_build_artifacts () { + if ! is_usable_git_repository . + then + return + fi + ! git ls-files --other --exclude-standard --error-unmatch \ -- ':/*' 2>/dev/null || { From patchwork Tue Oct 31 09:05:06 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13441306 Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3958F1944E for ; Tue, 31 Oct 2023 09:05:14 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b="NtQ+KbIl"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="T3FEv4Hz" Received: from wout1-smtp.messagingengine.com (wout1-smtp.messagingengine.com [64.147.123.24]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C0BE0E8 for ; Tue, 31 Oct 2023 02:05:11 -0700 (PDT) Received: from compute6.internal (compute6.nyi.internal [10.202.2.47]) by mailout.west.internal (Postfix) with ESMTP id D171032009AB; Tue, 31 Oct 2023 05:05:10 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute6.internal (MEProxy); Tue, 31 Oct 2023 05:05:11 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:sender :subject:subject:to:to; s=fm3; t=1698743110; x=1698829510; bh=Be wxoY8nghTnmOg2GZFL8ChRX032VIIVJiBpyAt6cgU=; b=NtQ+KbIlRKSo+7QYPk h3uRiH7V3T3wlkEOuJMEz8YPUK5DU0NtX6DQDPL0cl2z6eSXxmtePc33UwYHBHiC liyPK6e7Ax4B0KOzzxgVZ7qE5+zXtLsDAvQPINo4F2DC+yiuCxFIx4nd31agy85r maHuz4xijFzRBN6cF9TgNQOpIFZ3B+4gel7t1jtIInPZC+0VdeBpi/759R/MboR7 eMFSxlLN9nL1Cjic7xpzUQ1e1YNPzzAB1rBx5VJEZTTTqDZQeFynHo1U8AiN/c9B pZbOhrDUu2jTr8NCtKUeXYcJfUMQfFBmJXuAF8fJBJW7yGy3zizFbj94xdLKM+G3 0Y7A== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:sender:subject :subject:to:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm3; t=1698743110; x=1698829510; bh=BewxoY8nghTnm Og2GZFL8ChRX032VIIVJiBpyAt6cgU=; b=T3FEv4HzoXqMKty78jY0qVtXFS39E UFASMruHxjOoO0n1sklzOrzACC3ncDwYaP98Xweyez+ykLPq6J3mMiYNwtRfRoJJ Edzvmn931QilM7MrfCOr6CGlUayH+kB1lNxQNX1ogO/DwTQfmYN2vp+4p3lABjjx Wzs/oRvAm/mXmm4CDj9PibBr0IH5kh9Hs6M4pR2CcF2i8pxoWvbajBsFRY6jqFcb kLenLjK5xTrdh8Cm6WBQZ2Tt4sZ2Yeo/bs0Q012zQPASxhLE5M/NgrMbLua+nEa6 ctwD0e60cxGiM/lRccDi/h8Wk5LJ0njl5Miu4fRaEoVLv7LVxb7bmfunA== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvkedruddtuddguddvgecutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfgh necuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmd enucfjughrpeffhffvvefukfhfgggtuggjsehgtderredttddvnecuhfhrohhmpefrrght rhhitghkucfuthgvihhnhhgrrhguthcuoehpshesphhkshdrihhmqeenucggtffrrghtth gvrhhnpeejteefleeitedtleefueduhfeghffhveduveegudeuuddvtedvteeghfdthfeg veenucffohhmrghinheprghlphhinhgvlhhinhhugidrohhrghdphhhtthhpugdrshhhne cuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepphhssehp khhsrdhimh X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Tue, 31 Oct 2023 05:05:09 -0400 (EDT) Received: by vm-mail (OpenSMTPD) with ESMTPSA id ab49968d (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Tue, 31 Oct 2023 09:05:01 +0000 (UTC) Date: Tue, 31 Oct 2023 10:05:06 +0100 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Taylor Blau , Junio C Hamano , Phillip Wood , Oswald Buddenhagen Subject: [PATCH v4 7/8] ci: install test dependencies for linux-musl Message-ID: <9b43b0d90e30c3ec49fac29060dc19d21274139f.1698742590.git.ps@pks.im> References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: The linux-musl CI job executes tests on Alpine Linux, which is based on musl libc instead of glibc. We're missing some test dependencies though, which causes us to skip a subset of tests. Install these test dependencies to increase our test coverage on this platform. There are still some missing test dependecies, but these do not have a corresponding package in the Alpine repositories: - p4 and p4d, both parts of the Perforce version control system. - cvsps, which generates patch sets for CVS. - Subversion and the SVN::Core Perl library, the latter of which is not available in the Alpine repositories. While the tool itself is available, all Subversion-related tests are skipped without the SVN::Core Perl library anyway. The Apache2-based tests require a bit more care though. For one, the module path is different on Alpine Linux, which requires us to add it to the list of known module paths to detect it. But second, the WebDAV module on Alpine Linux is broken because it does not bundle the default database backend [1]. We thus need to skip the WebDAV-based tests on Alpine Linux for now. [1]: https://gitlab.alpinelinux.org/alpine/aports/-/issues/13112 Signed-off-by: Patrick Steinhardt --- ci/install-docker-dependencies.sh | 4 +++- t/lib-httpd.sh | 17 ++++++++++++++++- 2 files changed, 19 insertions(+), 2 deletions(-) diff --git a/ci/install-docker-dependencies.sh b/ci/install-docker-dependencies.sh index d0bc19d3bb3..6e845283680 100755 --- a/ci/install-docker-dependencies.sh +++ b/ci/install-docker-dependencies.sh @@ -17,7 +17,9 @@ linux32) ;; linux-musl) apk add --update build-base curl-dev openssl-dev expat-dev gettext \ - pcre2-dev python3 musl-libintl perl-utils ncurses >/dev/null + pcre2-dev python3 musl-libintl perl-utils ncurses \ + apache2 apache2-http2 apache2-proxy apache2-ssl apache2-webdav apr-util-dbd_sqlite3 \ + bash cvs gnupg perl-cgi perl-dbd-sqlite >/dev/null ;; pedantic) dnf -yq update >/dev/null && diff --git a/t/lib-httpd.sh b/t/lib-httpd.sh index 2fb1b2ae561..9ea74927c40 100644 --- a/t/lib-httpd.sh +++ b/t/lib-httpd.sh @@ -67,7 +67,8 @@ for DEFAULT_HTTPD_MODULE_PATH in '/usr/libexec/apache2' \ '/usr/lib/apache2/modules' \ '/usr/lib64/httpd/modules' \ '/usr/lib/httpd/modules' \ - '/usr/libexec/httpd' + '/usr/libexec/httpd' \ + '/usr/lib/apache2' do if test -d "$DEFAULT_HTTPD_MODULE_PATH" then @@ -127,6 +128,20 @@ else "Could not identify web server at '$LIB_HTTPD_PATH'" fi +if test -n "$LIB_HTTPD_DAV" && test -f /etc/os-release +then + case "$(grep "^ID=" /etc/os-release | cut -d= -f2-)" in + alpine) + # The WebDAV module in Alpine Linux is broken at least up to + # Alpine v3.16 as the default DBM driver is missing. + # + # https://gitlab.alpinelinux.org/alpine/aports/-/issues/13112 + test_skip_or_die GIT_TEST_HTTPD \ + "Apache WebDAV module does not have default DBM backend driver" + ;; + esac +fi + install_script () { write_script "$HTTPD_ROOT_PATH/$1" <"$TEST_PATH/$1" } From patchwork Tue Oct 31 09:05:11 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13441307 Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5E055199CA for ; Tue, 31 Oct 2023 09:05:18 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b="DZCIpWl1"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="PqbR5jE0" Received: from wout1-smtp.messagingengine.com (wout1-smtp.messagingengine.com [64.147.123.24]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 26BFBFA for ; Tue, 31 Oct 2023 02:05:15 -0700 (PDT) Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailout.west.internal (Postfix) with ESMTP id 43CB4320093C; Tue, 31 Oct 2023 05:05:14 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute3.internal (MEProxy); Tue, 31 Oct 2023 05:05:14 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:sender :subject:subject:to:to; s=fm3; t=1698743113; x=1698829513; bh=Tk GQR18xeEb0kF3Ko4vb8pA/m9+aGFQXdCvGYUNyR78=; b=DZCIpWl1wgBQhXUELS Gjc3xE+OoMkfAYi50INgxEfdLBB3cRa32FKem2OGZa13r4gtxamciFlL+sf4Z8AP GaybNU0tCt8JgJFDtBZ8OScx9mFx4k8erpVltq16N2ra0ZyVWcvdaz9IvJ+vIxoj RD8GOXSLptOjiEVP8v2cb+QWsFQaUJOApfuajvdit0aMVZq/MbBbrWcENixwJOJH 93wIOolniXXCl86ucqOM0bCWipsompdDGrr5TOgjAZpuf+HJqZLwzW2w8Zntc+hB YGdRRaGFR3NFIAbts2tQYkeqcG41hRF1SH2fbM2rFA+36xGOuqjIGHMOtzINtNH0 e4Gg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:sender:subject :subject:to:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm3; t=1698743113; x=1698829513; bh=TkGQR18xeEb0k F3Ko4vb8pA/m9+aGFQXdCvGYUNyR78=; b=PqbR5jE0UaYy5dS5ec1NyU1aD2sg0 cuAjXPbPq+ODV0v1Fli3rsEH0ftneiokuyjIHzUXKNAB30it+cB5gFOP0gNJmdqK H1OpMOlg6RwmV47poPw9PYOrg7IM17a86bjza1DoaFwhANvippjwMTESJFVQBMQ/ 1I2e7UM+CRUqtZSHR1CpZ+1aLuph2MSuQHlEcqAVRtiYaxz0Qp97wf5kaNtINd6D qJKHRRrO8HfjcqQ1+YgfwpnKczLgwHEyL3b+byKw11JHRSRNfbhoq5R3W7no3n/E 30X4rCJpAu1Zucc9f5soBhs2E4mm65UE5XHEl17JqEgl7saMX1ExFbf7A== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvkedruddtuddguddvgecutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfgh necuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmd enucfjughrpeffhffvvefukfhfgggtuggjsehgtderredttddvnecuhfhrohhmpefrrght rhhitghkucfuthgvihhnhhgrrhguthcuoehpshesphhkshdrihhmqeenucggtffrrghtth gvrhhnpeelgfeitdffkedvteffgeduuedvffdtffehteefleffvedvffehvdffgeelgeeg ieenucffohhmrghinhepghhithhlrggsrdgtohhmnecuvehluhhsthgvrhfuihiivgeptd enucfrrghrrghmpehmrghilhhfrhhomhepphhssehpkhhsrdhimh X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Tue, 31 Oct 2023 05:05:12 -0400 (EDT) Received: by vm-mail (OpenSMTPD) with ESMTPSA id 5c5b8331 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Tue, 31 Oct 2023 09:05:05 +0000 (UTC) Date: Tue, 31 Oct 2023 10:05:11 +0100 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Taylor Blau , Junio C Hamano , Phillip Wood , Oswald Buddenhagen Subject: [PATCH v4 8/8] ci: add support for GitLab CI Message-ID: References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: We already support Azure Pipelines and GitHub Workflows in the Git project, but until now we do not have support for GitLab CI. While it is arguably not in the interest of the Git project to maintain a ton of different CI platforms, GitLab has recently ramped up its efforts and tries to contribute to the Git project more regularly. Part of a problem we hit at GitLab rather frequently is that our own, custom CI setup we have is so different to the setup that the Git project has. More esoteric jobs like "linux-TEST-vars" that also set a couple of environment variables do not exist in GitLab's custom CI setup, and maintaining them to keep up with what Git does feels like wasted time. The result is that we regularly send patch series upstream that fail to compile or pass tests in GitHub Workflows. We would thus like to integrate the GitLab CI configuration into the Git project to help us send better patch series upstream and thus reduce overhead for the maintainer. The integration does not necessarily have to be a first-class citizen, which would in practice only add to the fallout that pipeline failures have for the maintainer. That being said, we are happy to maintain this alternative CI setup for the Git project and will make test results available as part of our own mirror of the Git project at [1]. This commit introduces the integration into our regular CI scripts so that most of the setup continues to be shared across all of the CI solutions. Note that as the builds on GitLab CI run as unprivileged user, we need to pull in both sudo and shadow packages to our Alpine based job to set this up. [1]: https://gitlab.com/gitlab-org/git Signed-off-by: Patrick Steinhardt --- .gitlab-ci.yml | 53 +++++++++++++++++++++++++++++++ ci/install-docker-dependencies.sh | 13 +++++++- ci/lib.sh | 50 +++++++++++++++++++++++++++-- ci/print-test-failures.sh | 6 ++++ 4 files changed, 119 insertions(+), 3 deletions(-) create mode 100644 .gitlab-ci.yml diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml new file mode 100644 index 00000000000..cd98bcb18aa --- /dev/null +++ b/.gitlab-ci.yml @@ -0,0 +1,53 @@ +default: + timeout: 2h + +workflow: + rules: + - if: $CI_PIPELINE_SOURCE == "merge_request_event" + - if: $CI_COMMIT_TAG + - if: $CI_COMMIT_REF_PROTECTED == "true" + +test: + image: $image + before_script: + - ./ci/install-docker-dependencies.sh + script: + - useradd builder --create-home + - chown -R builder "${CI_PROJECT_DIR}" + - sudo --preserve-env --set-home --user=builder ./ci/run-build-and-tests.sh + after_script: + - | + if test "$CI_JOB_STATUS" != 'success' + then + sudo --preserve-env --set-home --user=builder ./ci/print-test-failures.sh + fi + parallel: + matrix: + - jobname: linux-sha256 + image: ubuntu:latest + CC: clang + - jobname: linux-gcc + image: ubuntu:20.04 + CC: gcc + CC_PACKAGE: gcc-8 + - jobname: linux-TEST-vars + image: ubuntu:20.04 + CC: gcc + CC_PACKAGE: gcc-8 + - jobname: linux-gcc-default + image: ubuntu:latest + CC: gcc + - jobname: linux-leaks + image: ubuntu:latest + CC: gcc + - jobname: linux-asan-ubsan + image: ubuntu:latest + CC: clang + - jobname: pedantic + image: fedora:latest + - jobname: linux-musl + image: alpine:latest + artifacts: + paths: + - t/failed-test-artifacts + when: on_failure diff --git a/ci/install-docker-dependencies.sh b/ci/install-docker-dependencies.sh index 6e845283680..48cb2e735b5 100755 --- a/ci/install-docker-dependencies.sh +++ b/ci/install-docker-dependencies.sh @@ -7,6 +7,9 @@ begin_group "Install dependencies" +# Required so that apt doesn't wait for user input on certain packages. +export DEBIAN_FRONTEND=noninteractive + case "$jobname" in linux32) linux32 --32bit i386 sh -c ' @@ -16,11 +19,19 @@ linux32) ' ;; linux-musl) - apk add --update build-base curl-dev openssl-dev expat-dev gettext \ + apk add --update shadow sudo build-base curl-dev openssl-dev expat-dev gettext \ pcre2-dev python3 musl-libintl perl-utils ncurses \ apache2 apache2-http2 apache2-proxy apache2-ssl apache2-webdav apr-util-dbd_sqlite3 \ bash cvs gnupg perl-cgi perl-dbd-sqlite >/dev/null ;; +linux-*) + apt update -q && + apt install -q -y sudo git make language-pack-is libsvn-perl apache2 libssl-dev \ + libcurl4-openssl-dev libexpat-dev tcl tk gettext zlib1g-dev \ + perl-modules liberror-perl libauthen-sasl-perl libemail-valid-perl \ + libdbd-sqlite3-perl libio-socket-ssl-perl libnet-smtp-ssl-perl ${CC_PACKAGE:-${CC:-gcc}} \ + apache2 cvs cvsps gnupg libcgi-pm-perl subversion + ;; pedantic) dnf -yq update >/dev/null && dnf -yq install make gcc findutils diffutils perl python3 gettext zlib-devel expat-devel openssl-devel curl-devel pcre2-devel >/dev/null diff --git a/ci/lib.sh b/ci/lib.sh index e14b1029fad..6e3d64004ec 100755 --- a/ci/lib.sh +++ b/ci/lib.sh @@ -14,6 +14,22 @@ then need_to_end_group= echo '::endgroup::' >&2 } +elif test true = "$GITLAB_CI" +then + begin_group () { + need_to_end_group=t + printf "\e[0Ksection_start:$(date +%s):$(echo "$1" | tr ' ' _)\r\e[0K$1\n" + trap "end_group '$1'" EXIT + set -x + } + + end_group () { + test -n "$need_to_end_group" || return 0 + set +x + need_to_end_group= + printf "\e[0Ksection_end:$(date +%s):$(echo "$1" | tr ' ' _)\r\e[0K\n" + trap - EXIT + } else begin_group () { :; } end_group () { :; } @@ -208,6 +224,7 @@ then cache_dir="$HOME/test-cache/$SYSTEM_PHASENAME" GIT_TEST_OPTS="--write-junit-xml" + JOBS=10 elif test true = "$GITHUB_ACTIONS" then CI_TYPE=github-actions @@ -227,14 +244,43 @@ then cache_dir="$HOME/none" GIT_TEST_OPTS="--github-workflow-markup" + JOBS=10 +elif test true = "$GITLAB_CI" +then + CI_TYPE=gitlab-ci + CI_BRANCH="$CI_COMMIT_REF_NAME" + CI_COMMIT="$CI_COMMIT_SHA" + case "$CI_JOB_IMAGE" in + macos-*) + CI_OS_NAME=osx;; + alpine:*|fedora:*|ubuntu:*) + CI_OS_NAME=linux;; + *) + echo "Could not identify OS image" >&2 + env >&2 + exit 1 + ;; + esac + CI_REPO_SLUG="$CI_PROJECT_PATH" + CI_JOB_ID="$CI_JOB_ID" + CC="${CC_PACKAGE:-${CC:-gcc}}" + DONT_SKIP_TAGS=t + handle_failed_tests () { + create_failed_test_artifacts + } + + cache_dir="$HOME/none" + + runs_on_pool=$(echo "$CI_JOB_IMAGE" | tr : -) + JOBS=$(nproc) else echo "Could not identify CI type" >&2 env >&2 exit 1 fi -MAKEFLAGS="$MAKEFLAGS --jobs=10" -GIT_PROVE_OPTS="--timer --jobs 10 --state=failed,slow,save" +MAKEFLAGS="$MAKEFLAGS --jobs=${JOBS}" +GIT_PROVE_OPTS="--timer --jobs ${JOBS} --state=failed,slow,save" GIT_TEST_OPTS="$GIT_TEST_OPTS --verbose-log -x" if test windows = "$CI_OS_NAME" diff --git a/ci/print-test-failures.sh b/ci/print-test-failures.sh index 57277eefcd0..c33ad4e3a22 100755 --- a/ci/print-test-failures.sh +++ b/ci/print-test-failures.sh @@ -51,6 +51,12 @@ do tar czf failed-test-artifacts/"$test_name".trash.tar.gz "$trash_dir" continue ;; + gitlab-ci) + mkdir -p failed-test-artifacts + cp "${TEST_EXIT%.exit}.out" failed-test-artifacts/ + tar czf failed-test-artifacts/"$test_name".trash.tar.gz "$trash_dir" + continue + ;; *) echo "Unhandled CI type: $CI_TYPE" >&2 exit 1