From patchwork Thu Oct 26 08:00:03 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13437329 Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A024F18026 for ; Thu, 26 Oct 2023 08:00:12 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b="PvVL8kIB"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="I8teNE5o" Received: from wout5-smtp.messagingengine.com (wout5-smtp.messagingengine.com [64.147.123.21]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 83108CE for ; Thu, 26 Oct 2023 01:00:07 -0700 (PDT) Received: from compute2.internal (compute2.nyi.internal [10.202.2.46]) by mailout.west.internal (Postfix) with ESMTP id B0D6132009DE for ; Thu, 26 Oct 2023 04:00:06 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute2.internal (MEProxy); Thu, 26 Oct 2023 04:00:06 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:sender :subject:subject:to:to; s=fm3; t=1698307206; x=1698393606; bh=uh H5iKgsPcwLOOGQOT2mmZFbMOOYKVxy+XwteZhupG0=; b=PvVL8kIBKNRJzrQG0Z vKhEk/9dxa3Ie2ql+RRYiHpsW4H2vDKRtkRCMtJyesu1Beu3mjU+at7b+xH4qB7u Hp4u9KSgbaXne/hUfC/ZDXQAHR6adrG24TGVE66Uirgkp+5IVAK1LKTJQg5dPeAJ eJkoog6AXt6CWtjZavkmYnIm++Fujh9tnBT/V03hAyzoTJO8XqNeldmqnBqhc7eq GFZyGD+olpIRKY2PBn6LIiJ+s+rDAmVWIsN16og9lZ+r6PDDyHGXd+/gp+rLLY58 TpVsJyN6nJGujEI7hmtZxtnV2dANLazXnk5gi/8vRNdcv/L6cFfDfIl0E1u6loZl rf1A== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:sender:subject :subject:to:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm3; t=1698307206; x=1698393606; bh=uhH5iKgsPcwLO OGQOT2mmZFbMOOYKVxy+XwteZhupG0=; b=I8teNE5oStMC8apP6O58Cd8CZdhcJ HE/kWgQoqCJRcDcYsTUtIGXedCMujEfPyPpwT26CJva8mFgFNT5Hi+ANV5aHxjgl xXPzrkGM5yptUAkgPjREN1jqxLVD3oRV0C/jCCG2Ky3SdVpu8JFm9GFucZo6rd+g NK9xO+CAKtKI7r/1X0UvaA7187dH7lGWR1jTFkYiICG+7SUDpWuIZSMeCW1jvN2p qvF5m1aMfiy2ezGxUv38FXN4/e2+adlBWPqbv675J7d/rmqhmRc54EdwXou+Mpqe 1M8LPgM/GVnFndZdWgWetmdsKHnVZjfQf9VKVqeKK4G2wAsKzXrDJ9HUg== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvkedrledugdduvdehucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucenucfjughrpeffhffvuffkfhggtggujgesghdtre ertddtvdenucfhrhhomheprfgrthhrihgtkhcuufhtvghinhhhrghrughtuceophhssehp khhsrdhimheqnecuggftrfgrthhtvghrnhepheeghfdtfeeuffehkefgffduleffjedthf dvjeektdfhhedvlefgtefgvdettdfhnecuvehluhhsthgvrhfuihiivgeptdenucfrrghr rghmpehmrghilhhfrhhomhepphhssehpkhhsrdhimh X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA for ; Thu, 26 Oct 2023 04:00:05 -0400 (EDT) Received: by vm-mail (OpenSMTPD) with ESMTPSA id a75e2e27 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO) for ; Thu, 26 Oct 2023 07:59:57 +0000 (UTC) Date: Thu, 26 Oct 2023 10:00:03 +0200 From: Patrick Steinhardt To: git@vger.kernel.org Subject: [PATCH 1/5] ci: reorder definitions for grouping functions Message-ID: <586a8d1003b6559177d238ceda2c6ef2f16cfb8d.1698305961.git.ps@pks.im> References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: We define a set of grouping functions that are used to group together output in our CI, where these groups then end up as collapsible sections in the respective pipeline platform. The way these functions are defined is not easily extensible though as we have an up front check for the CI _not_ being GitLab Actions, where we define the non-stub logic in the else branch. Reorder the definitions such that we explicitly handle GitHub Actions. Signed-off-by: Patrick Steinhardt --- ci/lib.sh | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/ci/lib.sh b/ci/lib.sh index 6fbb5bade12..eb384f4e952 100755 --- a/ci/lib.sh +++ b/ci/lib.sh @@ -1,16 +1,7 @@ # Library of functions shared by all CI scripts -if test true != "$GITHUB_ACTIONS" +if test true = "$GITHUB_ACTIONS" then - begin_group () { :; } - end_group () { :; } - - group () { - shift - "$@" - } - set -x -else begin_group () { need_to_end_group=t echo "::group::$1" >&2 @@ -42,6 +33,15 @@ else } begin_group "CI setup" +else + begin_group () { :; } + end_group () { :; } + + group () { + shift + "$@" + } + set -x fi # Set 'exit on error' for all CI scripts to let the caller know that From patchwork Thu Oct 26 08:00:08 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13437331 Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2131618644 for ; Thu, 26 Oct 2023 08:00:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b="bZwwa/DY"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="QoI26bHQ" Received: from wout5-smtp.messagingengine.com (wout5-smtp.messagingengine.com [64.147.123.21]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 991C918D for ; Thu, 26 Oct 2023 01:00:11 -0700 (PDT) Received: from compute6.internal (compute6.nyi.internal [10.202.2.47]) by mailout.west.internal (Postfix) with ESMTP id 206F43200A05 for ; Thu, 26 Oct 2023 04:00:11 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute6.internal (MEProxy); Thu, 26 Oct 2023 04:00:11 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:sender :subject:subject:to:to; s=fm3; t=1698307210; x=1698393610; bh=Ba cV0TdBwgF9XfrN//Zw53Br5b9V+qEXQ6hAxsurEDQ=; b=bZwwa/DYSkbsoAIUsa THWrmwh9X7HvwtBB2LdqYZiOXWiQ+fLKv6H2WKLXb08IbhoHZlBlpzzPwQK2+Pb+ 2NviSFR3ZttX5JqZ+Ue5WzBZZbrNFgh3X450MKbBJqwELFN2H6Qx0LNUiQUzFrdU amwxbrq7MXUXIZG4v9yifr+oPYvMnuTi4m1Boo6QTk/cY0AntGGKUgwwJcOSQXc3 yJcFwAM/LA/pyLJRHkuY4rhjG3ni5qEy3iW9ZzrcJC/yvxbvho0xwADPb0RLjGDO 8yQOOc5cJQYklrDSS2NyEEsDmGJeof2UFxMXfICtkYQH3OS4LFdCV7msTGD+aGRW ZaOg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:sender:subject :subject:to:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm3; t=1698307210; x=1698393610; bh=BacV0TdBwgF9X frN//Zw53Br5b9V+qEXQ6hAxsurEDQ=; b=QoI26bHQdUuC4E6JrL/80nv4+cEiv DN1S5cJje2DZte1Qo+9Ql89mC+6Z522NUCirak+Hdej9F+ZG41ezhlA78BNyKfC8 abK35IZZNkxunXud67OyXrSskMDr+HYxGFiPBH5AwOwzmBctRSr2dLHbvzRwRS2O pxURfDTaBpBnLcukmEjCMX/4ykhGsO2dLUvhe+OjAZUskp+ubSyIEkVq5GkMUFbM sMq+FQ05LMLm4JvOwWj8mtgjEZS8Q7uckKru1yjQz1VsTtmGBcbSSczkdN4nQjUz xhrbC11KWsirPZ176rqVoOZRAiLQ7pcTP19SGPJhD4HKFN3tmK4GAYrvw== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvkedrledugdduvdehucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucenucfjughrpeffhffvuffkfhggtggujgesghdtre ertddtvdenucfhrhhomheprfgrthhrihgtkhcuufhtvghinhhhrghrughtuceophhssehp khhsrdhimheqnecuggftrfgrthhtvghrnhepheeghfdtfeeuffehkefgffduleffjedthf dvjeektdfhhedvlefgtefgvdettdfhnecuvehluhhsthgvrhfuihiivgeptdenucfrrghr rghmpehmrghilhhfrhhomhepphhssehpkhhsrdhimh X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA for ; Thu, 26 Oct 2023 04:00:10 -0400 (EDT) Received: by vm-mail (OpenSMTPD) with ESMTPSA id 341a2c29 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO) for ; Thu, 26 Oct 2023 08:00:01 +0000 (UTC) Date: Thu, 26 Oct 2023 10:00:08 +0200 From: Patrick Steinhardt To: git@vger.kernel.org Subject: [PATCH 2/5] ci: make grouping setup more generic Message-ID: References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Make the grouping setup more generic by always calling `begin_group ()` and `end_group ()` regardless of whether we have stubbed those functions or not. This ensures we can more readily add support for additional CI platforms. Second, this commit changes `end_group ()` to also accept a parameter that indicates _which_ group should end. This will be required by a later commit that introduces support for GitLab CI. Signed-off-by: Patrick Steinhardt --- ci/lib.sh | 16 ++++++++++------ 1 file changed, 10 insertions(+), 6 deletions(-) diff --git a/ci/lib.sh b/ci/lib.sh index eb384f4e952..957fd152d9c 100755 --- a/ci/lib.sh +++ b/ci/lib.sh @@ -14,12 +14,14 @@ then need_to_end_group= echo '::endgroup::' >&2 } - trap end_group EXIT group () { set +x - begin_group "$1" + + group="$1" shift + begin_group "$group" + # work around `dash` not supporting `set -o pipefail` ( "$@" 2>&1 @@ -28,11 +30,10 @@ then sed 's/^\(\([^ ]*\):\([0-9]*\):\([0-9]*:\) \)\(error\|warning\): /::\5 file=\2,line=\3::\1/' res=$(cat exit.status) rm exit.status - end_group + + end_group "$group" return $res } - - begin_group "CI setup" else begin_group () { :; } end_group () { :; } @@ -44,6 +45,9 @@ else set -x fi +begin_group "CI setup" +trap "end_group 'CI setup'" EXIT + # Set 'exit on error' for all CI scripts to let the caller know that # something went wrong. # @@ -287,5 +291,5 @@ esac MAKEFLAGS="$MAKEFLAGS CC=${CC:-cc}" -end_group +end_group "CI setup" set -x From patchwork Thu Oct 26 08:00:12 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13437332 Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 252D21945F for ; Thu, 26 Oct 2023 08:00:16 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b="irlTzin2"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="Bf77x1vw" Received: from wout5-smtp.messagingengine.com (wout5-smtp.messagingengine.com [64.147.123.21]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CEDCACE for ; Thu, 26 Oct 2023 01:00:15 -0700 (PDT) Received: from compute5.internal (compute5.nyi.internal [10.202.2.45]) by mailout.west.internal (Postfix) with ESMTP id 5C7743200A0C for ; Thu, 26 Oct 2023 04:00:15 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute5.internal (MEProxy); Thu, 26 Oct 2023 04:00:15 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:sender :subject:subject:to:to; s=fm3; t=1698307215; x=1698393615; bh=ho pNqA1aCKeGo4sKc3LlhiLkwLBxItuLsm1UGuPVoBk=; b=irlTzin2Lw97son+Fr Kl2BWwpBj5trB8qy/UI7xd+TlFSMBAbclmQTmL9Wzdv6IU+wUKmiqp6omJBZd38n tYfgUUQZomjNBtqezPSUXfFxzp0fi04/UxJhEmaPNbcJV6NRBmunJ5n8ln0J1OrG tsr0U6CbgBoBGWuUk87dnkkn7bknnZwkgjo898Lf77JTon0+2xLU71ObuUHAQ+G6 jXRcQn1YZkjc8U3tXH4fJC9P1s3FfLl8CXQP1sMOSvO2APn0oKcJgy7U+8rzMm06 PDVXxOhHT7xkhzp7rsNDu3nclGC2p0NpPNNLWpScsqkzkey7NyWWxEENbg1QBdiN FQxw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:sender:subject :subject:to:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm3; t=1698307215; x=1698393615; bh=hopNqA1aCKeGo 4sKc3LlhiLkwLBxItuLsm1UGuPVoBk=; b=Bf77x1vw3YAp4CTdOmzgV5bYJSvwr elZcS/MtYcCfcaZz9ChD9h5pYLKicizSyOUIHe1O3Tt7Fj6DHEOgVJJsMl/X++sG LuvbeC4WJKQCzJrWLqusBAICG9lMLkAIlnQQ9Rgmta+rhESRdzpoOa2RpMmhoFJl ATW0N0Or1IVrd/+nG+a9ozzFplY7criQMqGcES8hvH0S5xw/TUTvCKrxfuf4U4py CIxkbqwjx7+wvnVN8e4Qhv9uZbeulpXsW3Sr4S1YN/JdcLmkAHtGmUOwU3fJl6NF 6R7iKpq1H/yWDtbCLmM5+wBcuPj6NVrUGXJR+B0nh2eHR8F9OE4aNYATw== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvkedrledugdduvdehucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucenucfjughrpeffhffvuffkfhggtggujgesghdtre ertddtvdenucfhrhhomheprfgrthhrihgtkhcuufhtvghinhhhrghrughtuceophhssehp khhsrdhimheqnecuggftrfgrthhtvghrnhepheeghfdtfeeuffehkefgffduleffjedthf dvjeektdfhhedvlefgtefgvdettdfhnecuvehluhhsthgvrhfuihiivgeptdenucfrrghr rghmpehmrghilhhfrhhomhepphhssehpkhhsrdhimh X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA for ; Thu, 26 Oct 2023 04:00:14 -0400 (EDT) Received: by vm-mail (OpenSMTPD) with ESMTPSA id c789c6ff (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO) for ; Thu, 26 Oct 2023 08:00:05 +0000 (UTC) Date: Thu, 26 Oct 2023 10:00:12 +0200 From: Patrick Steinhardt To: git@vger.kernel.org Subject: [PATCH 3/5] ci: group installation of Docker dependencies Message-ID: References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Pull in "lib.sh" into "install-docker-dependencies.sh" such that we can set up proper groups for those dependencise. This allows the reader to collapse sections in the CI output on GitHub Actions (and later on on GitLab CI). Signed-off-by: Patrick Steinhardt --- ci/install-docker-dependencies.sh | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/ci/install-docker-dependencies.sh b/ci/install-docker-dependencies.sh index 78b7e326da6..d0bc19d3bb3 100755 --- a/ci/install-docker-dependencies.sh +++ b/ci/install-docker-dependencies.sh @@ -3,6 +3,10 @@ # Install dependencies required to build and test Git inside container # +. ${0%/*}/lib.sh + +begin_group "Install dependencies" + case "$jobname" in linux32) linux32 --32bit i386 sh -c ' @@ -20,3 +24,5 @@ pedantic) dnf -yq install make gcc findutils diffutils perl python3 gettext zlib-devel expat-devel openssl-devel curl-devel pcre2-devel >/dev/null ;; esac + +end_group "Install dependencies" From patchwork Thu Oct 26 08:00:16 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13437333 Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 752151C6B0 for ; Thu, 26 Oct 2023 08:00:22 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b="KR1hJwdL"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="qMMa5zhB" Received: from wout5-smtp.messagingengine.com (wout5-smtp.messagingengine.com [64.147.123.21]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3CAE4B8 for ; Thu, 26 Oct 2023 01:00:20 -0700 (PDT) Received: from compute6.internal (compute6.nyi.internal [10.202.2.47]) by mailout.west.internal (Postfix) with ESMTP id BBB913200A13 for ; Thu, 26 Oct 2023 04:00:19 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute6.internal (MEProxy); Thu, 26 Oct 2023 04:00:19 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:sender :subject:subject:to:to; s=fm3; t=1698307219; x=1698393619; bh=9Y RxLSlcTgN+nhEkpBVuC+J5zbIdi7doC2MHWgQ5A+U=; b=KR1hJwdLEwx09aWJ3i km7VxhseP20D+RuxNGdjAimR+MLcbHGAyhCxoW6IB+Lkly4Te3knsx2N8X76AnAV RSS9t88bwrV/YGALhWNzfztEfnPXXF/4jAIgQqE1YECMtaHDKmg0gMihAW8gHXAC M0XePEm9PmGKqfKzCHmUvD289R+3INUizalT4B+SZvUeHN5/QwEGuPYYyWDpSBLl vPb6fk3JCFU0Qgv3AF/w8hPpE0HTXLG0E5m9bFjxF4BuBYleyHgtK/bm87gyllzj 8FFYbMSVbeQNUQ/60aWKPRz1j1mAIYwIzIWN3DcYAAp4oJ9/RR2fCgEMF26hX2JP NOxA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:sender:subject :subject:to:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm3; t=1698307219; x=1698393619; bh=9YRxLSlcTgN+n hEkpBVuC+J5zbIdi7doC2MHWgQ5A+U=; b=qMMa5zhB+ZZmgsVdCmtf4fDeS0yzX yLeq2w+KiRBz2m4ndFbPZ9bpLS1S3lsuIXi7kM2uTpggk/F3JOcS19tp+d/mn0PM XbWGsbsWcYNkWzX9S+PvF1mrJOIAml86ENF4poD2kc+a1C5bvakhgCAE4DaYTudf YOe6d05f5SszOnKVsqHxVJz+Q/SBA+0tpMJPHbmoHarhGV3QO2FF17W/OMuw+J1x SPeWtFjdyLRBiMGLonnNwQIzwaT/Z4LYbZHPdgLDaRTgLIpr3vLPn7DcdtYguGeB SmoFrB719SGY19WcOFi++vAypv0rRi6lYcZLRwIbYZcWnwlI3FA3TvSQw== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvkedrledugdduvdehucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucenucfjughrpeffhffvuffkfhggtggujgesghdtre ertddtvdenucfhrhhomheprfgrthhrihgtkhcuufhtvghinhhhrghrughtuceophhssehp khhsrdhimheqnecuggftrfgrthhtvghrnhepheeghfdtfeeuffehkefgffduleffjedthf dvjeektdfhhedvlefgtefgvdettdfhnecuvehluhhsthgvrhfuihiivgepudenucfrrghr rghmpehmrghilhhfrhhomhepphhssehpkhhsrdhimh X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA for ; Thu, 26 Oct 2023 04:00:18 -0400 (EDT) Received: by vm-mail (OpenSMTPD) with ESMTPSA id 0755e1b2 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO) for ; Thu, 26 Oct 2023 08:00:09 +0000 (UTC) Date: Thu, 26 Oct 2023 10:00:16 +0200 From: Patrick Steinhardt To: git@vger.kernel.org Subject: [PATCH 4/5] ci: split out logic to set up failed test artifacts Message-ID: <4a864a1d174f7d4d36202a375302d450fbe9f2a3.1698305961.git.ps@pks.im> References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: We have some logic in place to create a directory with the output from failed tests, which will then subsequently be uploaded as CI artifact. We're about to add support for GitLab CI, which will want to reuse the logic. Split the logic into a separate function so that it is reusable. Signed-off-by: Patrick Steinhardt --- ci/lib.sh | 40 ++++++++++++++++++++++------------------ 1 file changed, 22 insertions(+), 18 deletions(-) diff --git a/ci/lib.sh b/ci/lib.sh index 957fd152d9c..33005854520 100755 --- a/ci/lib.sh +++ b/ci/lib.sh @@ -137,6 +137,27 @@ handle_failed_tests () { return 1 } +create_failed_test_artifacts () { + mkdir -p t/failed-test-artifacts + + for test_exit in t/test-results/*.exit + do + test 0 != "$(cat "$test_exit")" || continue + + test_name="${test_exit%.exit}" + test_name="${test_name##*/}" + printf "\\e[33m\\e[1m=== Failed test: ${test_name} ===\\e[m\\n" + echo "The full logs are in the 'print test failures' step below." + echo "See also the 'failed-tests-*' artifacts attached to this run." + cat "t/test-results/$test_name.markup" + + trash_dir="t/trash directory.$test_name" + cp "t/test-results/$test_name.out" t/failed-test-artifacts/ + tar czf t/failed-test-artifacts/"$test_name".trash.tar.gz "$trash_dir" + done + return 1 +} + # GitHub Action doesn't set TERM, which is required by tput export TERM=${TERM:-dumb} @@ -177,25 +198,8 @@ then CC="${CC_PACKAGE:-${CC:-gcc}}" DONT_SKIP_TAGS=t handle_failed_tests () { - mkdir -p t/failed-test-artifacts echo "FAILED_TEST_ARTIFACTS=t/failed-test-artifacts" >>$GITHUB_ENV - - for test_exit in t/test-results/*.exit - do - test 0 != "$(cat "$test_exit")" || continue - - test_name="${test_exit%.exit}" - test_name="${test_name##*/}" - printf "\\e[33m\\e[1m=== Failed test: ${test_name} ===\\e[m\\n" - echo "The full logs are in the 'print test failures' step below." - echo "See also the 'failed-tests-*' artifacts attached to this run." - cat "t/test-results/$test_name.markup" - - trash_dir="t/trash directory.$test_name" - cp "t/test-results/$test_name.out" t/failed-test-artifacts/ - tar czf t/failed-test-artifacts/"$test_name".trash.tar.gz "$trash_dir" - done - return 1 + create_failed_test_artifacts } cache_dir="$HOME/none" From patchwork Thu Oct 26 08:00:20 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13437334 Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7D5D71CFB7 for ; Thu, 26 Oct 2023 08:00:25 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b="DacffOrQ"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="ffPL6p9m" Received: from wout5-smtp.messagingengine.com (wout5-smtp.messagingengine.com [64.147.123.21]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9B5D6CE for ; Thu, 26 Oct 2023 01:00:23 -0700 (PDT) Received: from compute7.internal (compute7.nyi.internal [10.202.2.48]) by mailout.west.internal (Postfix) with ESMTP id 256913200A05 for ; Thu, 26 Oct 2023 04:00:23 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute7.internal (MEProxy); Thu, 26 Oct 2023 04:00:23 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:sender :subject:subject:to:to; s=fm3; t=1698307222; x=1698393622; bh=hh qsCkmkssUaVxhnbdy7UhmtShdrOMYWJ8lXQdQH9Uc=; b=DacffOrQtLOcUxCXAv nZu+KJBNbTA+aUkitY/DpJVv54HhTdbAGJbs4u26kAUzEYAot2EFlgq8EJIzkHLr UtouDjlFsc+0ftpadePr66DYHVcJUwICoaNCRz6ZZjRZ5jaIaFr4sbFkDJ0rlPnX WH8PrINV26fG1+W74o25L/zyVVJpM2kKc5pb6TiQwi/bALR4XiEx72WQ+RJ8fikg 2o1dRCddU92yKqv7+kz9bOKU2rgvlCvExDCHaSg4RzMuIXMOfNIfq/amwy957oXi rPpGT4AGXw7dQ3Jsoe1wbHQBP+HXJwnOxO3JMw/2BVvdX+VqPuyiISSUQgVavmbd UpYQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:sender:subject :subject:to:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm3; t=1698307222; x=1698393622; bh=hhqsCkmkssUaV xhnbdy7UhmtShdrOMYWJ8lXQdQH9Uc=; b=ffPL6p9mjCdPOI4yDsi14g7R9Xu1E NcxzQi5Lqy9e+bCneFxrnhyrE4vTJ9xCXTMMWA4P/Y2lAxjU9UZzYHCNwO82I0eQ mskusImrLRMPRvPZhMCGOaNaaZeZIOUyWF0A2zGCCpf+JuFKEatKI5jzpJgQhdFd KZ3o0L6p4pepKfo9VmSaajR1yLV7nu3kiUnECa0NZIG6wVYyPDgs8d8Ew1qMX7Eg 5h5eMWz/9hCpnpngU6gWPVDejdLOhc1TE9wcIPzwLVjmleBpNVWZ9AOsqkmlHLSz qAWk9V73vekAEGviSGgxL49H8HmS6myyMKA7IsFbj8VHxda63Njz5FzlQ== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvkedrledugdduvdeiucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucenucfjughrpeffhffvuffkfhggtggujgesghdtre ertddtvdenucfhrhhomheprfgrthhrihgtkhcuufhtvghinhhhrghrughtuceophhssehp khhsrdhimheqnecuggftrfgrthhtvghrnhepjeejtdeigfegkefgjeehveevjeejveeuvd dtieekffevleeglefhgffgjeejfeefnecuffhomhgrihhnpehgihhtlhgrsgdrtghomhen ucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpehpshesph hkshdrihhm X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA for ; Thu, 26 Oct 2023 04:00:22 -0400 (EDT) Received: by vm-mail (OpenSMTPD) with ESMTPSA id d856c048 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO) for ; Thu, 26 Oct 2023 08:00:13 +0000 (UTC) Date: Thu, 26 Oct 2023 10:00:20 +0200 From: Patrick Steinhardt To: git@vger.kernel.org Subject: [PATCH 5/5] ci: add support for GitLab CI Message-ID: <35b07e5378d960b93ae8990a3abb525e1762d97d.1698305961.git.ps@pks.im> References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: We already support Azure Pipelines and GitHub Workflows in the Git project, but until now we do not have support for GitLab CI. While it is arguably not in the interest of the Git project to maintain a ton of different CI platforms, GitLab has recently ramped up its efforts and tries to contribute to the Git project more regularly. Part of a problem we hit at GitLab rather frequently is that our own, custom CI setup we have is so different to the setup that the Git project has. More esoteric jobs like "linux-TEST-vars" that also sets a couple of environment variables do not exist in GitLab's custom CI setup, and maintaining them to keep up with what Git does feels like wasted time. The result is that we regularly send patch series upstream that would otherwise fail to compile or pass tests in GitHub Workflows. We would thus like to integrate the GitLab CI configuration into the Git project to help us ensure to send better patch series upstream and thus reduce overhead for the maintainer. The integration does not necessarily have to be a first-class citizen, which would in practice only add to the fallout that pipeline failures have for the maintainer. That being said, we are happy to maintain this alternative CI setup for the Git project and will make test results available as part of our own mirror of the Git project at [1]. This commit introduces the integration into our regular CI scripts so that most of the setup continues to be shared across all of the CI solutions. [1]: https://gitlab.com/gitlab-org/git Signed-off-by: Patrick Steinhardt --- .gitlab-ci.yml | 51 +++++++++++++++++++++++ ci/install-docker-dependencies.sh | 9 +++- ci/lib.sh | 69 +++++++++++++++++++++++++++++++ ci/print-test-failures.sh | 6 +++ 4 files changed, 134 insertions(+), 1 deletion(-) create mode 100644 .gitlab-ci.yml diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml new file mode 100644 index 00000000000..43d3a961fa0 --- /dev/null +++ b/.gitlab-ci.yml @@ -0,0 +1,51 @@ +default: + timeout: 2h + +workflow: + rules: + - if: $CI_PIPELINE_SOURCE == "merge_request_event" + - if: $CI_COMMIT_TAG + - if: $CI_COMMIT_REF_PROTECTED == "true" + +test: + image: $image + before_script: + - ./ci/install-docker-dependencies.sh + script: + - useradd builder --home-dir "${CI_PROJECT_DIR}" + - chown -R builder "${CI_PROJECT_DIR}" + - sudo --preserve-env --set-home --user=builder ./ci/run-build-and-tests.sh + after_script: + - | + if test "$CI_JOB_STATUS" != 'success' + then + sudo --preserve-env --set-home --user=builder ./ci/print-test-failures.sh + fi + parallel: + matrix: + - jobname: linux-sha256 + image: ubuntu:latest + CC: clang + - jobname: linux-gcc + image: ubuntu:20.04 + CC: gcc + CC_PACKAGE: gcc-8 + - jobname: linux-TEST-vars + image: ubuntu:20.04 + CC: gcc + CC_PACKAGE: gcc-8 + - jobname: linux-gcc-default + image: ubuntu:latest + CC: gcc + - jobname: linux-leaks + image: ubuntu:latest + CC: gcc + - jobname: linux-asan-ubsan + image: ubuntu:latest + CC: clang + - jobname: linux-musl + image: alpine:latest + artifacts: + paths: + - t/failed-test-artifacts + when: on_failure diff --git a/ci/install-docker-dependencies.sh b/ci/install-docker-dependencies.sh index d0bc19d3bb3..1cd92db1876 100755 --- a/ci/install-docker-dependencies.sh +++ b/ci/install-docker-dependencies.sh @@ -7,6 +7,9 @@ begin_group "Install dependencies" +# Required so that apt doesn't wait for user input on certain packages. +export DEBIAN_FRONTEND=noninteractive + case "$jobname" in linux32) linux32 --32bit i386 sh -c ' @@ -16,9 +19,13 @@ linux32) ' ;; linux-musl) - apk add --update build-base curl-dev openssl-dev expat-dev gettext \ + apk add --update git shadow sudo build-base curl-dev openssl-dev expat-dev gettext \ pcre2-dev python3 musl-libintl perl-utils ncurses >/dev/null ;; +linux-*) + apt update -q && + apt install -q -y sudo git make language-pack-is libsvn-perl apache2 libssl-dev libcurl4-openssl-dev libexpat-dev tcl tk gettext zlib1g-dev perl-modules liberror-perl libauthen-sasl-perl libemail-valid-perl libio-socket-ssl-perl libnet-smtp-ssl-perl ${CC_PACKAGE:-${CC:-gcc}} + ;; pedantic) dnf -yq update >/dev/null && dnf -yq install make gcc findutils diffutils perl python3 gettext zlib-devel expat-devel openssl-devel curl-devel pcre2-devel >/dev/null diff --git a/ci/lib.sh b/ci/lib.sh index 33005854520..102e9d04a1f 100755 --- a/ci/lib.sh +++ b/ci/lib.sh @@ -15,6 +15,42 @@ then echo '::endgroup::' >&2 } + group () { + set +x + + group="$1" + shift + begin_group "$group" + + # work around `dash` not supporting `set -o pipefail` + ( + "$@" 2>&1 + echo $? >exit.status + ) | + sed 's/^\(\([^ ]*\):\([0-9]*\):\([0-9]*:\) \)\(error\|warning\): /::\5 file=\2,line=\3::\1/' + res=$(cat exit.status) + rm exit.status + + end_group "$group" + return $res + } +elif test true = "$GITLAB_CI" +then + begin_group () { + need_to_end_group=t + echo -e "\e[0Ksection_start:$(date +%s):$(echo "$1" | tr ' ' _)\r\e[0K$1" + trap "end_group '$1'" EXIT + set -x + } + + end_group () { + test -n "$need_to_end_group" || return 0 + set +x + need_to_end_group= + echo -e "\e[0Ksection_end:$(date +%s):$(echo "$1" | tr ' ' _)\r\e[0K" + trap - EXIT + } + group () { set +x @@ -209,6 +245,39 @@ then MAKEFLAGS="$MAKEFLAGS --jobs=10" test windows != "$CI_OS_NAME" || GIT_TEST_OPTS="--no-chain-lint --no-bin-wrappers $GIT_TEST_OPTS" +elif test true = "$GITLAB_CI" +then + CI_TYPE=gitlab-ci + CI_BRANCH="$CI_COMMIT_REF_NAME" + CI_COMMIT="$CI_COMMIT_SHA" + case "$CI_JOB_IMAGE" in + macos-*) + CI_OS_NAME=osx;; + alpine:*|ubuntu:*) + CI_OS_NAME=linux;; + *) + echo "Could not identify OS image" >&2 + env >&2 + exit 1 + ;; + esac + CI_REPO_SLUG="$CI_PROJECT_PATH" + CI_JOB_ID="$CI_JOB_ID" + CC="${CC_PACKAGE:-${CC:-gcc}}" + DONT_SKIP_TAGS=t + handle_failed_tests () { + create_failed_test_artifacts + } + + cache_dir="$HOME/none" + + runs_on_pool=$(echo "$CI_JOB_IMAGE" | tr : -) + + export GIT_PROVE_OPTS="--timer --jobs $(nproc)" + export GIT_TEST_OPTS="--verbose-log -x" + MAKEFLAGS="$MAKEFLAGS --jobs=$(nproc)" + test windows != "$CI_OS_NAME" || + GIT_TEST_OPTS="--no-chain-lint --no-bin-wrappers $GIT_TEST_OPTS" else echo "Could not identify CI type" >&2 env >&2 diff --git a/ci/print-test-failures.sh b/ci/print-test-failures.sh index 57277eefcd0..c33ad4e3a22 100755 --- a/ci/print-test-failures.sh +++ b/ci/print-test-failures.sh @@ -51,6 +51,12 @@ do tar czf failed-test-artifacts/"$test_name".trash.tar.gz "$trash_dir" continue ;; + gitlab-ci) + mkdir -p failed-test-artifacts + cp "${TEST_EXIT%.exit}.out" failed-test-artifacts/ + tar czf failed-test-artifacts/"$test_name".trash.tar.gz "$trash_dir" + continue + ;; *) echo "Unhandled CI type: $CI_TYPE" >&2 exit 1