From patchwork Wed May 25 10:03:40 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= X-Patchwork-Id: 12860912 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id CDFB2C433F5 for ; Wed, 25 May 2022 10:04:46 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240991AbiEYKEq (ORCPT ); Wed, 25 May 2022 06:04:46 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57352 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230033AbiEYKEp (ORCPT ); Wed, 25 May 2022 06:04:45 -0400 Received: from mail-wm1-x330.google.com (mail-wm1-x330.google.com [IPv6:2a00:1450:4864:20::330]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 557196D94A for ; Wed, 25 May 2022 03:04:44 -0700 (PDT) Received: by mail-wm1-x330.google.com with SMTP id 67-20020a1c1946000000b00397382b44f4so3023493wmz.2 for ; Wed, 25 May 2022 03:04:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=pyO8MT9SvBP/RX9AuLl9N3flkPYsp1/wEZuSabJg8eU=; b=QDVx+wfSy+9pNQC3tl0E4IsJln8NtvoUSyCqI0LIz014VBRx2eDkJEgCuiOSkzOmws OcvU6kfX4Gs9pw0Jfs4x3fKtPBQlpOXE30OfB4nEsM/73UDCO8+EeOhC5ErjL82mbjm3 F+sktc8iOVoTj3wpxScuVO7jj1N7kj+b287LIcwZxd9LN7oJB569eDPl375pj/NPCdKp Hpjwz817+5xjlf33IMcTwZCkw/4pVEhJPAzts4sbdIBv0lkhCubrLyk51xCVTHMUSj0r 9YzsOeVCwRfIIQb+ofbrngT+stt8RolQfvckcIm5PkiM0b7g0maipRRROc2WYq+b+FKh eAXw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=pyO8MT9SvBP/RX9AuLl9N3flkPYsp1/wEZuSabJg8eU=; b=DQShAn0acGCNGIgHpfYijOanitcGREyB7wARjOQJBKONg4Q2/HybxiD7BkQp9dP+nr kasuW7S/Tbso2dX2pjit2U56XDmpASjEP4mj8Mc3z38rZnzoL5SVYUI2veUPfs/opdRF tzPE2n5dngJE6xMjpwWWSTDfXWSPdzdnvjhJqwPfVLsWSrt2J2+o4wxyHKuMvR3bBkpq 9vPnFueEdsNVaE2zLB+YW7ibRunKMJyglAymUsSSK45J5sRI3bDRaN9AlCYfjmGmvE6T /knyytLCF2P5Z62HFOjVWzJC7P2zZOFkaueiXx1yXLvjNnDD06VcP46dxWU484CbdqU/ mUMA== X-Gm-Message-State: AOAM532VA/IZTtV0u0btUYVT87rVKnEtMFVRmYXrEiUw6JmFDJN4ENSg Qz4w0OAnDcww1ML7aRd6ATXaylM+3iNQ8g== X-Google-Smtp-Source: ABdhPJy+AH6RDnyi3oFElnRLOHwJCsmOg2Jc0kxc8C4KWpPYk9GyON/eUyEcZRGb03EgXplGoNiWBw== X-Received: by 2002:a05:600c:1986:b0:394:867f:984c with SMTP id t6-20020a05600c198600b00394867f984cmr7543236wmq.20.1653473052726; Wed, 25 May 2022 03:04:12 -0700 (PDT) Received: from vm.nix.is (vm.nix.is. [2a01:4f8:120:2468::2]) by smtp.gmail.com with ESMTPSA id o4-20020a05600002c400b0020d0cdbf7eesm1649452wry.111.2022.05.25.03.04.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 25 May 2022 03:04:12 -0700 (PDT) From: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= To: git@vger.kernel.org Cc: Junio C Hamano , Taylor Blau , =?utf-8?q?SZEDER_G=C3=A1bor?= , =?utf-8?b?xJBvw6Bu?= =?utf-8?b?IFRy4bqnbiBDw7RuZyBEYW5o?= , =?utf-8?q?Car?= =?utf-8?q?lo_Marcelo_Arenas_Bel=C3=B3n?= , Johannes Schindelin , Victoria Dye , Matheus Tavares , Lars Schneider , Eric Sunshine , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBC?= =?utf-8?b?amFybWFzb24=?= Subject: [PATCH v6 01/29] CI: run "set -ex" early in ci/lib.sh Date: Wed, 25 May 2022 12:03:40 +0200 Message-Id: X-Mailer: git-send-email 2.36.1.1045.gf356b5617dd In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Change ci/lib.sh to run "set -ex" before it does anything else, before this we were omitting a few "export" commands in our trace output that happened before we enabled tracing. Signed-off-by: Ævar Arnfjörð Bjarmason --- ci/lib.sh | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/ci/lib.sh b/ci/lib.sh index 86e37da9bc5..57121dd664a 100755 --- a/ci/lib.sh +++ b/ci/lib.sh @@ -1,5 +1,11 @@ # Library of functions shared by all CI scripts +# Set 'exit on error' for all CI scripts to let the caller know that +# something went wrong. +# Set tracing executed commands, primarily setting environment variables +# and installing dependencies. +set -ex + skip_branch_tip_with_tag () { # Sometimes, a branch is pushed at the same time the tag that points # at the same commit as the tip of the branch is pushed, and building @@ -85,12 +91,6 @@ export TERM=${TERM:-dumb} # Clear MAKEFLAGS that may come from the outside world. export MAKEFLAGS= -# Set 'exit on error' for all CI scripts to let the caller know that -# something went wrong. -# Set tracing executed commands, primarily setting environment variables -# and installing dependencies. -set -ex - if test -n "$SYSTEM_COLLECTIONURI" || test -n "$SYSTEM_TASKDEFINITIONSURI" then CI_TYPE=azure-pipelines From patchwork Wed May 25 10:03:41 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= X-Patchwork-Id: 12860913 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id CB0EAC433F5 for ; Wed, 25 May 2022 10:04:48 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S241259AbiEYKEs (ORCPT ); Wed, 25 May 2022 06:04:48 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57468 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230033AbiEYKEq (ORCPT ); Wed, 25 May 2022 06:04:46 -0400 Received: from mail-wr1-x433.google.com (mail-wr1-x433.google.com [IPv6:2a00:1450:4864:20::433]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A8E216D4EB for ; Wed, 25 May 2022 03:04:45 -0700 (PDT) Received: by mail-wr1-x433.google.com with SMTP id z15so5563734wrg.11 for ; Wed, 25 May 2022 03:04:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=mGq2G6LCbUodDDDwZ4xSVV0PavCX1b3/qP5KUC2Weu4=; b=RpdrqbBPnX0zYisRHTJi/uH6COpOZV3KQu1wBGfqL0In80IkQ8C6+dLVjLgh9hTEKm 1pncBXG/s7+rwiIvxBSVq8Fkz5z0JMwGvn/iqJ7Xjut/ajT32tmOfxQbISyQRxLBdwpT 11xiDInGX5vIyjAn5r7densGrIWDxcQ0cjxd5aCbxbbevrZiiFz2qIzzbQ3hxskiCHht iOc7Vq95IabBHkeECdfzaLV7AamglrtC5t49YVsnikiAo6DgwL5LqzFYMBmEOVXuXSQE vZ2YMCmAGPJj1Jpa2kWVZaQYzXNZQCLzCHEEXeovjs0FD/GI7VtRn18mV07HMEzoGkv+ kHhw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=mGq2G6LCbUodDDDwZ4xSVV0PavCX1b3/qP5KUC2Weu4=; b=aeQ5XETs4iToCpVQtEP6nD/bK1uAjPWQ7DGn1t2w4xOg4a/+YXHbZzTfAAou85KDRG 0xiz0hBOVNpsY1TQ0XyQUdUvGKem6WnM0roMzgu4UwdSbNxbeUMefwyuHxY3hBDQufW6 JHfOPeN7xBJBzy9vgzt7wKVZRrPOAzRN7YBaCTXgaSbW7YWwmU8oWzDlqHX7HVfferJw pcuugs7yrCidOJLL+HwT6+BLJWpBNpPU+RJOqCDPdm7zZvUyyIAXWvi5MT/084v2W66E JByFbP/cE7O5sAwOETxs09G9rKxpyT+tDyJ9mP0AXUn4TzvTuq1AKl1hi5hHXomJIGnJ uz+A== X-Gm-Message-State: AOAM5337SiX8FWxfqogS2hCqxPT4FJEvA1payRtViZeIfRAHECpwpxRx HkBZPRCfBvRW673vopGcwb3kBrUdnKHSDA== X-Google-Smtp-Source: ABdhPJz9fTNqjQgOLh86xBvf83gbOLfieVCzIjlf5A4dVlbZJPXsE/ATLBEBTT4r4FDmTeyI266Ggg== X-Received: by 2002:adf:d1e9:0:b0:20f:d6b4:56a9 with SMTP id g9-20020adfd1e9000000b0020fd6b456a9mr13004488wrd.169.1653473083803; Wed, 25 May 2022 03:04:43 -0700 (PDT) Received: from vm.nix.is (vm.nix.is. [2a01:4f8:120:2468::2]) by smtp.gmail.com with ESMTPSA id o4-20020a05600002c400b0020d0cdbf7eesm1649452wry.111.2022.05.25.03.04.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 25 May 2022 03:04:43 -0700 (PDT) From: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= To: git@vger.kernel.org Cc: Junio C Hamano , Taylor Blau , =?utf-8?q?SZEDER_G=C3=A1bor?= , =?utf-8?b?xJBvw6Bu?= =?utf-8?b?IFRy4bqnbiBDw7RuZyBEYW5o?= , =?utf-8?q?Car?= =?utf-8?q?lo_Marcelo_Arenas_Bel=C3=B3n?= , Johannes Schindelin , Victoria Dye , Matheus Tavares , Lars Schneider , Eric Sunshine , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBC?= =?utf-8?b?amFybWFzb24=?= Subject: [PATCH v6 02/29] CI: make "$jobname" explicit, remove fallback Date: Wed, 25 May 2022 12:03:41 +0200 Message-Id: X-Mailer: git-send-email 2.36.1.1045.gf356b5617dd In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Remove the $jobname fallback last adjusted in b011fabd6e0 (ci/lib.sh: encapsulate Travis-specific things, 2019-01-27). This makes all of our CI jobs have an explicit $jobname, and removes the special-case of the Windows jobs implicitly setting one (that they don't currently use). In subsequent commits we'll use this new $jobname explicitly. This is a follow-up to my 25715419bf4 (CI: don't run "make test" twice in one job, 2021-11-23), since then all the jobs except the Windows jobs have had an explicit $jobname. In the case of the Windows jobs we were using a jobname of windows-gcc for both, due to picking "gcc" as a default fallback for "$CC" in ci/lib.sh. Since we'll need this to run ci/lib.sh it makes sense to move the check to the top of the file. Subsequent commits will add more such assertions to it. Signed-off-by: Ævar Arnfjörð Bjarmason --- .github/workflows/main.yml | 8 ++++++++ ci/lib.sh | 12 +++++++----- 2 files changed, 15 insertions(+), 5 deletions(-) diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml index c35200defb9..309b9141249 100644 --- a/.github/workflows/main.yml +++ b/.github/workflows/main.yml @@ -79,6 +79,8 @@ jobs: windows-build: name: win build + env: + jobname: windows-build needs: ci-config if: needs.ci-config.outputs.enabled == 'yes' runs-on: windows-latest @@ -100,6 +102,8 @@ jobs: path: artifacts windows-test: name: win test + env: + jobname: windows-test runs-on: windows-latest needs: [windows-build] strategy: @@ -131,6 +135,8 @@ jobs: path: ${{env.FAILED_TEST_ARTIFACTS}} vs-build: name: win+VS build + env: + jobname: vs-build needs: ci-config if: needs.ci-config.outputs.enabled == 'yes' env: @@ -183,6 +189,8 @@ jobs: path: artifacts vs-test: name: win+VS test + env: + jobname: vs-test runs-on: windows-latest needs: vs-build strategy: diff --git a/ci/lib.sh b/ci/lib.sh index 57121dd664a..3061de96d8f 100755 --- a/ci/lib.sh +++ b/ci/lib.sh @@ -6,6 +6,13 @@ # and installing dependencies. set -ex +# Starting assertions +if test -z "$jobname" +then + echo "error: must set a CI jobname in the environment" >&2 + exit 1 +fi + skip_branch_tip_with_tag () { # Sometimes, a branch is pushed at the same time the tag that points # at the same commit as the tip of the branch is pushed, and building @@ -146,11 +153,6 @@ test -n "${DONT_SKIP_TAGS-}" || skip_branch_tip_with_tag skip_good_tree -if test -z "$jobname" -then - jobname="$CI_OS_NAME-$CC" -fi - export DEVELOPER=1 export DEFAULT_TEST_TARGET=prove export GIT_TEST_CLONE_2GB=true From patchwork Wed May 25 10:03:42 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= X-Patchwork-Id: 12860915 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C44B3C433EF for ; Wed, 25 May 2022 10:04:58 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S241336AbiEYKE5 (ORCPT ); Wed, 25 May 2022 06:04:57 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57734 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230033AbiEYKEt (ORCPT ); Wed, 25 May 2022 06:04:49 -0400 Received: from mail-wr1-x433.google.com (mail-wr1-x433.google.com [IPv6:2a00:1450:4864:20::433]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DD95C6D4EB for ; Wed, 25 May 2022 03:04:46 -0700 (PDT) Received: by mail-wr1-x433.google.com with SMTP id j25so4963101wrb.6 for ; Wed, 25 May 2022 03:04:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=iTbXPYe3qbqrW7WI2UBxMMiaKkciTrVHG3FWOwYYZZk=; b=Lby6w/nNw2djxpTcoa9JSosY2ghgxPyjBjckH4Kxej1+uHjjXAbqyyvI9JB6UKbB7/ fIJESl6icw5U4npWaJYLsG4RBt7NOudheYj8Fph3oK46LZ9bmiaRiLJSqU629a3YRdrp 2AgdTsPXGfqBm8owKl3RDQyVPO25vmljMEldfHu0dg9/VCH/UFQcx7DiqdbAxCOtNh2Z ejRUvVWaU5wJFPatwGpZfUwFScfQftUYSxp+mk2tBjog2Dsc3DAUKIG+57A/QaYUpdco dZJbIIZn/jPiJn0FCDvgUp7KjwBE/Tntu0APfwW67LkTET/ReciOWUZM7qHTZvufFoAc JOrw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=iTbXPYe3qbqrW7WI2UBxMMiaKkciTrVHG3FWOwYYZZk=; b=piVYN/cs2sp1ciQvy+y5vks9YM2Ne3N1Ssttzb0I8HaWt5GLb8PpKA1nLokBibPLGV Dl4JF4IPesEtzEdUXc2aqz7KL6frZokkHXD0T+Z7Zzmxosr/zugTsFq1ZnmwpUP3/KQh 4yQzkjdaGGHbCbUd+ymWIZ5+arnzSp65nQp+yRQILuLCM7nK2/dLnTLcgYbUPq3oXhcW u5SmdWPw+APzWg6wU3rhVMpnRDLk7BMC3/eVcd8nmcQyWxjcwLdjfvV/12SAIP/wGoP7 qPhtN5viP0lLqtktmWlTuoa4M9aEKWhsOTELlmXxsCIiwn+Zw2zG3FcSP4eO7HRmzUI0 32sw== X-Gm-Message-State: AOAM533NXGOrjnzEnXBE0DVjgoBxCkewtVrCeUu+sn+OrpgdbrQKkcBC K9BogcgZpE+UVzyUPv88CtdKkCQSGJB16A== X-Google-Smtp-Source: ABdhPJz9ke2eJLLX3/I192PDcAHm5EajCfm7gt40JjzK8hZBVwUdwmZXpmxjSMvBw/sUKCA4xkV8Ug== X-Received: by 2002:a5d:4e08:0:b0:210:297:7d4b with SMTP id p8-20020a5d4e08000000b0021002977d4bmr892723wrt.112.1653473084852; Wed, 25 May 2022 03:04:44 -0700 (PDT) Received: from vm.nix.is (vm.nix.is. [2a01:4f8:120:2468::2]) by smtp.gmail.com with ESMTPSA id o4-20020a05600002c400b0020d0cdbf7eesm1649452wry.111.2022.05.25.03.04.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 25 May 2022 03:04:44 -0700 (PDT) From: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= To: git@vger.kernel.org Cc: Junio C Hamano , Taylor Blau , =?utf-8?q?SZEDER_G=C3=A1bor?= , =?utf-8?b?xJBvw6Bu?= =?utf-8?b?IFRy4bqnbiBDw7RuZyBEYW5o?= , =?utf-8?q?Car?= =?utf-8?q?lo_Marcelo_Arenas_Bel=C3=B3n?= , Johannes Schindelin , Victoria Dye , Matheus Tavares , Lars Schneider , Eric Sunshine , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBC?= =?utf-8?b?amFybWFzb24=?= Subject: [PATCH v6 03/29] CI: remove more dead Travis CI support Date: Wed, 25 May 2022 12:03:42 +0200 Message-Id: X-Mailer: git-send-email 2.36.1.1045.gf356b5617dd In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Remove code that I missed in 4a6e4b96026 (CI: remove Travis CI support, 2021-11-23). This code was only called from or used by the now-removed .travis.yml, or needed by the Travis CI environment. For the symlinking in ci/run-build-and-tests.sh: Back when 3c93b829205 (travis-ci: build Git during the 'script' phase, 2018-01-08) and 4b060a4d973 (ci: use a junction on Windows instead of a symlink, 2019-01-27) adjusted this "ln" command, the Windows build would use ci/run-build-and-tests.sh. As seen in 889cacb6897 (ci: configure GitHub Actions for CI/PR, 2020-04-11) the current windows build uses a different entry point under the GitHub CI, which doesn't use this .prove caching. Namely "ci/run-test-slice.sh". We can be certain that it's never used in "ci/run-test-slice.sh" because to have a ".prove" file we'd need to use "--state=save", which we only do in the dead Azure codepath in ci/lib.sh. If it were used it would do the wrong thing, because the different test slices would each try to clobber the same "t/.prove" file. If a subsequent run then used the "--state=failed,slow,save" it would defeat the purpose of "ci/run-test-slice.sh", since all slices would then run all tests. I.e. behavior of prove's "--state" options is to select tests to run from the provided "--state" file, in addition to those specified on the command-line. For ci/run-docker{,-build}.sh: It was likewise last referenced in the .travis.yml removed in my 4a6e4b96026. The current "dockerized" run in ".github/workflows/main.yml" calls the same entry points as the main "regular" job. For "ci/print-test-failures.sh" the creation of the tarball added in aea8879a6ac (travis-ci: include the trash directories of failed tests in the trace log, 2018-08-01) has been dead since my 4a6e4b96026, which removed the fall-through case from the "case/esac" statement. Signed-off-by: Ævar Arnfjörð Bjarmason --- ci/lib.sh | 10 ++---- ci/print-test-failures.sh | 39 --------------------- ci/run-build-and-tests.sh | 5 --- ci/run-docker-build.sh | 66 ----------------------------------- ci/run-docker.sh | 47 ------------------------- ci/run-test-slice.sh | 5 --- ci/util/extract-trash-dirs.sh | 50 -------------------------- 7 files changed, 2 insertions(+), 220 deletions(-) delete mode 100755 ci/run-docker-build.sh delete mode 100755 ci/run-docker.sh delete mode 100755 ci/util/extract-trash-dirs.sh diff --git a/ci/lib.sh b/ci/lib.sh index 3061de96d8f..2e60c0a71b5 100755 --- a/ci/lib.sh +++ b/ci/lib.sh @@ -106,9 +106,6 @@ then CI_COMMIT="$BUILD_SOURCEVERSION" CI_JOB_ID="$BUILD_BUILDID" CI_JOB_NUMBER="$BUILD_BUILDNUMBER" - CI_OS_NAME="$(echo "$AGENT_OS" | tr A-Z a-z)" - test darwin != "$CI_OS_NAME" || CI_OS_NAME=osx - CI_REPO_SLUG="$(expr "$BUILD_REPOSITORY_URI" : '.*/\([^/]*/[^/]*\)$')" CC="${CC:-gcc}" # use a subdirectory of the cache dir (because the file share is shared @@ -118,16 +115,13 @@ then export GIT_PROVE_OPTS="--timer --jobs 10 --state=failed,slow,save" export GIT_TEST_OPTS="--verbose-log -x --write-junit-xml" MAKEFLAGS="$MAKEFLAGS --jobs=10" - test windows_nt != "$CI_OS_NAME" || + test Windows_NT != "$AGENT_OS" || GIT_TEST_OPTS="--no-chain-lint --no-bin-wrappers $GIT_TEST_OPTS" elif test true = "$GITHUB_ACTIONS" then CI_TYPE=github-actions CI_BRANCH="$GITHUB_REF" CI_COMMIT="$GITHUB_SHA" - CI_OS_NAME="$(echo "$RUNNER_OS" | tr A-Z a-z)" - test macos != "$CI_OS_NAME" || CI_OS_NAME=osx - CI_REPO_SLUG="$GITHUB_REPOSITORY" CI_JOB_ID="$GITHUB_RUN_ID" CC="${CC_PACKAGE:-${CC:-gcc}}" DONT_SKIP_TAGS=t @@ -137,7 +131,7 @@ then export GIT_PROVE_OPTS="--timer --jobs 10" export GIT_TEST_OPTS="--verbose-log -x" MAKEFLAGS="$MAKEFLAGS --jobs=10" - test windows != "$CI_OS_NAME" || + test Windows != "$RUNNER_OS" || GIT_TEST_OPTS="--no-chain-lint --no-bin-wrappers $GIT_TEST_OPTS" else echo "Could not identify CI type" >&2 diff --git a/ci/print-test-failures.sh b/ci/print-test-failures.sh index 57277eefcd0..740ebb7c529 100755 --- a/ci/print-test-failures.sh +++ b/ci/print-test-failures.sh @@ -16,15 +16,6 @@ then exit fi -case "$jobname" in -osx-clang|osx-gcc) - # base64 in OSX doesn't wrap its output at 76 columns by - # default, but prints a single, very long line. - base64_opts="-b 76" - ;; -esac - -combined_trash_size=0 for TEST_EXIT in test-results/*.exit do if [ "$(cat "$TEST_EXIT")" != "0" ] @@ -42,47 +33,17 @@ do azure-pipelines) mkdir -p failed-test-artifacts mv "$trash_dir" failed-test-artifacts - continue ;; github-actions) mkdir -p failed-test-artifacts echo "FAILED_TEST_ARTIFACTS=t/failed-test-artifacts" >>$GITHUB_ENV cp "${TEST_EXIT%.exit}.out" failed-test-artifacts/ tar czf failed-test-artifacts/"$test_name".trash.tar.gz "$trash_dir" - continue ;; *) echo "Unhandled CI type: $CI_TYPE" >&2 exit 1 ;; esac - trash_tgz_b64="trash.$test_name.base64" - if [ -d "$trash_dir" ] - then - tar czp "$trash_dir" |base64 $base64_opts >"$trash_tgz_b64" - - trash_size=$(wc -c <"$trash_tgz_b64") - if [ $trash_size -gt 1048576 ] - then - # larger than 1MB - echo "$(tput setaf 1)Didn't include the trash directory of '$test_name' in the trace log, it's too big$(tput sgr0)" - continue - fi - - new_combined_trash_size=$(($combined_trash_size + $trash_size)) - if [ $new_combined_trash_size -gt 1048576 ] - then - echo "$(tput setaf 1)Didn't include the trash directory of '$test_name' in the trace log, there is plenty of trash in there already.$(tput sgr0)" - continue - fi - combined_trash_size=$new_combined_trash_size - - # DO NOT modify these two 'echo'-ed strings below - # without updating 'ci/util/extract-trash-dirs.sh' - # as well. - echo "$(tput setaf 1)Start of trash directory of '$test_name':$(tput sgr0)" - cat "$trash_tgz_b64" - echo "$(tput setaf 1)End of trash directory of '$test_name'$(tput sgr0)" - fi fi done diff --git a/ci/run-build-and-tests.sh b/ci/run-build-and-tests.sh index 280dda7d285..0a1ec8c2bae 100755 --- a/ci/run-build-and-tests.sh +++ b/ci/run-build-and-tests.sh @@ -5,11 +5,6 @@ . ${0%/*}/lib.sh -case "$CI_OS_NAME" in -windows*) cmd //c mklink //j t\\.prove "$(cygpath -aw "$cache_dir/.prove")";; -*) ln -s "$cache_dir/.prove" t/.prove;; -esac - export MAKE_TARGETS="all test" case "$jobname" in diff --git a/ci/run-docker-build.sh b/ci/run-docker-build.sh deleted file mode 100755 index 6cd832efb9c..00000000000 --- a/ci/run-docker-build.sh +++ /dev/null @@ -1,66 +0,0 @@ -#!/bin/sh -# -# Build and test Git inside container -# -# Usage: -# run-docker-build.sh -# - -set -ex - -if test $# -ne 1 || test -z "$1" -then - echo >&2 "usage: run-docker-build.sh " - exit 1 -fi - -case "$jobname" in -linux32) - switch_cmd="linux32 --32bit i386" - ;; -linux-musl) - switch_cmd= - useradd () { adduser -D "$@"; } - ;; -*) - exit 1 - ;; -esac - -"${0%/*}/install-docker-dependencies.sh" - -# If this script runs inside a docker container, then all commands are -# usually executed as root. Consequently, the host user might not be -# able to access the test output files. -# If a non 0 host user id is given, then create a user "ci" with that -# user id to make everything accessible to the host user. -HOST_UID=$1 -if test $HOST_UID -eq 0 -then - # Just in case someone does want to run the test suite as root. - CI_USER=root -else - CI_USER=ci - if test "$(id -u $CI_USER 2>/dev/null)" = $HOST_UID - then - echo "user '$CI_USER' already exists with the requested ID $HOST_UID" - else - useradd -u $HOST_UID $CI_USER - fi -fi - -# Build and test -command $switch_cmd su -m -l $CI_USER -c " - set -ex - export DEVELOPER='$DEVELOPER' - export DEFAULT_TEST_TARGET='$DEFAULT_TEST_TARGET' - export GIT_PROVE_OPTS='$GIT_PROVE_OPTS' - export GIT_TEST_OPTS='$GIT_TEST_OPTS' - export GIT_TEST_CLONE_2GB='$GIT_TEST_CLONE_2GB' - export MAKEFLAGS='$MAKEFLAGS' - export cache_dir='$cache_dir' - cd /usr/src/git - test -n '$cache_dir' && ln -s '$cache_dir/.prove' t/.prove - make - make test -" diff --git a/ci/run-docker.sh b/ci/run-docker.sh deleted file mode 100755 index af89d1624a4..00000000000 --- a/ci/run-docker.sh +++ /dev/null @@ -1,47 +0,0 @@ -#!/bin/sh -# -# Download and run Docker image to build and test Git -# - -. ${0%/*}/lib.sh - -case "$jobname" in -linux32) - CI_CONTAINER="daald/ubuntu32:xenial" - ;; -linux-musl) - CI_CONTAINER=alpine - ;; -*) - exit 1 - ;; -esac - -docker pull "$CI_CONTAINER" - -# Use the following command to debug the docker build locally: -# must be 0 if podman is used as drop-in replacement for docker -# $ docker run -itv "${PWD}:/usr/src/git" --entrypoint /bin/sh "$CI_CONTAINER" -# root@container:/# export jobname= -# root@container:/# /usr/src/git/ci/run-docker-build.sh - -container_cache_dir=/tmp/container-cache - -docker run \ - --interactive \ - --env DEVELOPER \ - --env DEFAULT_TEST_TARGET \ - --env GIT_PROVE_OPTS \ - --env GIT_TEST_OPTS \ - --env GIT_TEST_CLONE_2GB \ - --env MAKEFLAGS \ - --env jobname \ - --env cache_dir="$container_cache_dir" \ - --volume "${PWD}:/usr/src/git" \ - --volume "$cache_dir:$container_cache_dir" \ - "$CI_CONTAINER" \ - /usr/src/git/ci/run-docker-build.sh $(id -u $USER) - -check_unignored_build_artifacts - -save_good_tree diff --git a/ci/run-test-slice.sh b/ci/run-test-slice.sh index f8c2c3106a2..b9a682b4bcd 100755 --- a/ci/run-test-slice.sh +++ b/ci/run-test-slice.sh @@ -5,11 +5,6 @@ . ${0%/*}/lib.sh -case "$CI_OS_NAME" in -windows*) cmd //c mklink //j t\\.prove "$(cygpath -aw "$cache_dir/.prove")";; -*) ln -s "$cache_dir/.prove" t/.prove;; -esac - make --quiet -C t T="$(cd t && ./helper/test-tool path-utils slice-tests "$1" "$2" t[0-9]*.sh | tr '\n' ' ')" diff --git a/ci/util/extract-trash-dirs.sh b/ci/util/extract-trash-dirs.sh deleted file mode 100755 index 8e67bec21a2..00000000000 --- a/ci/util/extract-trash-dirs.sh +++ /dev/null @@ -1,50 +0,0 @@ -#!/bin/sh - -error () { - echo >&2 "error: $@" - exit 1 -} - -find_embedded_trash () { - while read -r line - do - case "$line" in - *Start\ of\ trash\ directory\ of\ \'t[0-9][0-9][0-9][0-9]-*\':*) - test_name="${line#*\'}" - test_name="${test_name%\'*}" - - return 0 - esac - done - - return 1 -} - -extract_embedded_trash () { - while read -r line - do - case "$line" in - *End\ of\ trash\ directory\ of\ \'$test_name\'*) - return - ;; - *) - printf '%s\n' "$line" - ;; - esac - done - - error "unexpected end of input" -} - -# Raw logs from Linux build jobs have CRLF line endings, while OSX -# build jobs mostly have CRCRLF, except an odd line every now and -# then that has CRCRCRLF. 'base64 -d' from 'coreutils' doesn't like -# CRs and complains about "invalid input", so remove all CRs at the -# end of lines. -sed -e 's/\r*$//' | \ -while find_embedded_trash -do - echo "Extracting trash directory of '$test_name'" - - extract_embedded_trash |base64 -d |tar xzp -done From patchwork Wed May 25 10:03:43 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= X-Patchwork-Id: 12860917 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6FBC9C433F5 for ; Wed, 25 May 2022 10:05:15 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S241617AbiEYKE7 (ORCPT ); Wed, 25 May 2022 06:04:59 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57782 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S241261AbiEYKEt (ORCPT ); Wed, 25 May 2022 06:04:49 -0400 Received: from mail-wr1-x431.google.com (mail-wr1-x431.google.com [IPv6:2a00:1450:4864:20::431]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0288A6D961 for ; Wed, 25 May 2022 03:04:48 -0700 (PDT) Received: by mail-wr1-x431.google.com with SMTP id x12so5141390wrg.2 for ; Wed, 25 May 2022 03:04:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=AZ3/tKEcQLu4mh5k5aLUKcrTGIGBhgkdKtbJX55YB3o=; b=EB0PnfXK3si5vIF+0eV5ZYOB4r0PmNKpCXxJNOQWlli+bGlCEqfdETfKBdrtuortS/ PRF1wD2EvdOAbGG9ZrB5mmMaI1C1VyaevviOwRQPh31olgzSnE0dUAIatVQpnc9v1j6B vVIyJNamJ8fFlIHF14vHnT+unhuJuOY6DASj/+D+ieAsjxbkPN5q0qQ/GaNBuFlm5sBc zuc4Wt7QPdxGDCm6zsVV/HuZLYdX7/KRZLC13Chdut94wCIbumKtI49I4qhCidYTmVpA 09lOzP8HGphakcH6slJznjiIvbAffRhLLB5EJAT5vYvlovrENAvE8s8L8Ejr/tbzea43 fmBw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=AZ3/tKEcQLu4mh5k5aLUKcrTGIGBhgkdKtbJX55YB3o=; b=w9d2/L5/k6p+uVTVziLii81Y+FR/59UkhgjLVPsM0Y+D9gElNQhzX3/V8fuAkpDYZ6 KzPoo9TyvQ6IxBSCtCUFcfUrOj4G6+nPRb+4yYhilnkggwfszqx7ih7y9sqEk8cKjJYC NmaKGiszE9s7P1qtOGdoXUqBHsLZXRzwuQ80y4j7Hszz33s6bZOHYS6A+9G/XDr00DiS uyfRy5L3X0qY18E5+A7k24nGDm+GTwu0cox7umcp4i7XSRKn0FHn7SYhMNYVjWWaGaM/ 1vzU+eCqyC08aSR6BUiX0MGbevLHfX/kKOW20QbQFusy2h+B8sPLksyLjKAZBE7SfCXz eIvg== X-Gm-Message-State: AOAM532XCUiRJbQ0pcj7RguW47ymhoK3pqX9ZWuUYKILP7eBPqzc+ENs IgX+zu1/yKGny6YQTavg0WoczyO1l6TSkA== X-Google-Smtp-Source: ABdhPJytJRIp4FKYluluP1XV72A3PvMOLNu7xh4IFr6yY6NklD2TRfvLxutswXyII0BpSaXN5whqRg== X-Received: by 2002:adf:f90d:0:b0:20c:de32:4d35 with SMTP id b13-20020adff90d000000b0020cde324d35mr25914605wrr.583.1653473086228; Wed, 25 May 2022 03:04:46 -0700 (PDT) Received: from vm.nix.is (vm.nix.is. [2a01:4f8:120:2468::2]) by smtp.gmail.com with ESMTPSA id o4-20020a05600002c400b0020d0cdbf7eesm1649452wry.111.2022.05.25.03.04.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 25 May 2022 03:04:45 -0700 (PDT) From: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= To: git@vger.kernel.org Cc: Junio C Hamano , Taylor Blau , =?utf-8?q?SZEDER_G=C3=A1bor?= , =?utf-8?b?xJBvw6Bu?= =?utf-8?b?IFRy4bqnbiBDw7RuZyBEYW5o?= , =?utf-8?q?Car?= =?utf-8?q?lo_Marcelo_Arenas_Bel=C3=B3n?= , Johannes Schindelin , Victoria Dye , Matheus Tavares , Lars Schneider , Eric Sunshine , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBC?= =?utf-8?b?amFybWFzb24=?= Subject: [PATCH v6 04/29] CI: remove dead "tree skipping" code Date: Wed, 25 May 2022 12:03:43 +0200 Message-Id: X-Mailer: git-send-email 2.36.1.1045.gf356b5617dd In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Remove the code related to skipping CI runs if the tree was successfully tested before. With the GitHub CI this is done with the "ci-config" step, see e76eec35540 (ci: allow per-branch config for GitHub Actions, 2020-05-07) and 7d78d5fc1a9 (ci: skip GitHub workflow runs for already-tested commits/trees, 2020-10-08). This code hasn't been used since 4a6e4b96026 (CI: remove Travis CI support, 2021-11-23), and before that for the retired Azure pipeline support removed in 6081d3898fe (ci: retire the Azure Pipelines definition, 2020-04-11). This change is needed because this and subsequent commits are turning "ci/lib.sh" into a dumber library that'll only be tasked with setting variables for CI jobs, or for specific steps within those jobs. Now we don't need to worry about it potentially skipping the run on its own. This change also removes a subtle potential logic error introduced in 0e7696c64db (ci: disallow directional formatting, 2021-11-04). The "ci/check-directional-formatting.bash" script would have been made to run after the "save_good_tree" in invoked in "ci/run-static-analysis.sh". So if e.g. the "make coccicheck" failed we'd still mark the tree as good. I.e. the addition of "ci/check-directional-formatting.bash" didn't take into account that the various users of the "save_good_tree" function in ci/*.sh made a hard assumption that they're the only ci/*.sh script that's being run (and they use "set -e"). This is entirely academic since we weren't actually running this code, but is something to be careful of if anyone ever needs to resurrect parts of this. Finally, the cache_dir="$HOME/none" added in a3f2eec862b (ci/lib: allow running in GitHub Actions, 2020-04-08) for the GitHub CI can be removed. It appears to have been added to appease the subsequent 'mkdir -p "$cache_dir"', which wasn't needed in GitHub CI. 1. https://lore.kernel.org/git/patch-1.1-eec0a8c3164-20211217T000418Z-avarab@gmail.com/ Signed-off-by: Ævar Arnfjörð Bjarmason --- ci/lib.sh | 91 --------------------------------------- ci/run-build-and-tests.sh | 2 - ci/run-static-analysis.sh | 2 - ci/test-documentation.sh | 2 - 4 files changed, 97 deletions(-) diff --git a/ci/lib.sh b/ci/lib.sh index 2e60c0a71b5..e04b53b428d 100755 --- a/ci/lib.sh +++ b/ci/lib.sh @@ -13,75 +13,6 @@ then exit 1 fi -skip_branch_tip_with_tag () { - # Sometimes, a branch is pushed at the same time the tag that points - # at the same commit as the tip of the branch is pushed, and building - # both at the same time is a waste. - # - # When the build is triggered by a push to a tag, $CI_BRANCH will - # have that tagname, e.g. v2.14.0. Let's see if $CI_BRANCH is - # exactly at a tag, and if so, if it is different from $CI_BRANCH. - # That way, we can tell if we are building the tip of a branch that - # is tagged and we can skip the build because we won't be skipping a - # build of a tag. - - if TAG=$(git describe --exact-match "$CI_BRANCH" 2>/dev/null) && - test "$TAG" != "$CI_BRANCH" - then - echo "$(tput setaf 2)Tip of $CI_BRANCH is exactly at $TAG$(tput sgr0)" - exit 0 - fi -} - -# Save some info about the current commit's tree, so we can skip the build -# job if we encounter the same tree again and can provide a useful info -# message. -save_good_tree () { - echo "$(git rev-parse $CI_COMMIT^{tree}) $CI_COMMIT $CI_JOB_NUMBER $CI_JOB_ID" >>"$good_trees_file" - # limit the file size - tail -1000 "$good_trees_file" >"$good_trees_file".tmp - mv "$good_trees_file".tmp "$good_trees_file" -} - -# Skip the build job if the same tree has already been built and tested -# successfully before (e.g. because the branch got rebased, changing only -# the commit messages). -skip_good_tree () { - if test true = "$GITHUB_ACTIONS" - then - return - fi - - if ! good_tree_info="$(grep "^$(git rev-parse $CI_COMMIT^{tree}) " "$good_trees_file")" - then - # Haven't seen this tree yet, or no cached good trees file yet. - # Continue the build job. - return - fi - - echo "$good_tree_info" | { - read tree prev_good_commit prev_good_job_number prev_good_job_id - - if test "$CI_JOB_ID" = "$prev_good_job_id" - then - cat <<-EOF - $(tput setaf 2)Skipping build job for commit $CI_COMMIT.$(tput sgr0) - This commit has already been built and tested successfully by this build job. - To force a re-build delete the branch's cache and then hit 'Restart job'. - EOF - else - cat <<-EOF - $(tput setaf 2)Skipping build job for commit $CI_COMMIT.$(tput sgr0) - This commit's tree has already been built and tested successfully in build job $prev_good_job_number for commit $prev_good_commit. - The log of that build job is available at $SYSTEM_TASKDEFINITIONSURI$SYSTEM_TEAMPROJECT/_build/results?buildId=$prev_good_job_id - To force a re-build delete the branch's cache and then hit 'Restart job'. - EOF - fi - } - - exit 0 -} - check_unignored_build_artifacts () { ! git ls-files --other --exclude-standard --error-unmatch \ @@ -102,16 +33,8 @@ if test -n "$SYSTEM_COLLECTIONURI" || test -n "$SYSTEM_TASKDEFINITIONSURI" then CI_TYPE=azure-pipelines # We are running in Azure Pipelines - CI_BRANCH="$BUILD_SOURCEBRANCH" - CI_COMMIT="$BUILD_SOURCEVERSION" - CI_JOB_ID="$BUILD_BUILDID" - CI_JOB_NUMBER="$BUILD_BUILDNUMBER" CC="${CC:-gcc}" - # use a subdirectory of the cache dir (because the file share is shared - # among *all* phases) - cache_dir="$HOME/test-cache/$SYSTEM_PHASENAME" - export GIT_PROVE_OPTS="--timer --jobs 10 --state=failed,slow,save" export GIT_TEST_OPTS="--verbose-log -x --write-junit-xml" MAKEFLAGS="$MAKEFLAGS --jobs=10" @@ -120,13 +43,7 @@ then elif test true = "$GITHUB_ACTIONS" then CI_TYPE=github-actions - CI_BRANCH="$GITHUB_REF" - CI_COMMIT="$GITHUB_SHA" - CI_JOB_ID="$GITHUB_RUN_ID" CC="${CC_PACKAGE:-${CC:-gcc}}" - DONT_SKIP_TAGS=t - - cache_dir="$HOME/none" export GIT_PROVE_OPTS="--timer --jobs 10" export GIT_TEST_OPTS="--verbose-log -x" @@ -139,14 +56,6 @@ else exit 1 fi -good_trees_file="$cache_dir/good-trees" - -mkdir -p "$cache_dir" - -test -n "${DONT_SKIP_TAGS-}" || -skip_branch_tip_with_tag -skip_good_tree - export DEVELOPER=1 export DEFAULT_TEST_TARGET=prove export GIT_TEST_CLONE_2GB=true diff --git a/ci/run-build-and-tests.sh b/ci/run-build-and-tests.sh index 0a1ec8c2bae..e5a21985b44 100755 --- a/ci/run-build-and-tests.sh +++ b/ci/run-build-and-tests.sh @@ -45,5 +45,3 @@ esac # start running tests. make $MAKE_TARGETS check_unignored_build_artifacts - -save_good_tree diff --git a/ci/run-static-analysis.sh b/ci/run-static-analysis.sh index 65bcebda41a..5a87b1cac96 100755 --- a/ci/run-static-analysis.sh +++ b/ci/run-static-analysis.sh @@ -28,5 +28,3 @@ fi make hdr-check || exit 1 - -save_good_tree diff --git a/ci/test-documentation.sh b/ci/test-documentation.sh index de41888430a..9e0652c30dd 100755 --- a/ci/test-documentation.sh +++ b/ci/test-documentation.sh @@ -41,5 +41,3 @@ grep ' X-Patchwork-Id: 12860914 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 706BCC433F5 for ; Wed, 25 May 2022 10:04:57 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S241477AbiEYKEy (ORCPT ); Wed, 25 May 2022 06:04:54 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58018 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S241336AbiEYKEw (ORCPT ); Wed, 25 May 2022 06:04:52 -0400 Received: from mail-wr1-x42f.google.com (mail-wr1-x42f.google.com [IPv6:2a00:1450:4864:20::42f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 40FBA6E8C3 for ; Wed, 25 May 2022 03:04:49 -0700 (PDT) Received: by mail-wr1-x42f.google.com with SMTP id e2so17781523wrc.1 for ; Wed, 25 May 2022 03:04:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=JmiK0IZ8ftEe9QrndmCy4xy1TM6BpU5EDxYQfhy4NiI=; b=DWjiocseQ8qImwcTGRr+5epZucMQ/MqTq23xOwR5oE5uj6T88mnIAO6SWN/8mEuCnF 4YC2WCt3XdYowD6NztQ8iBOIURezgvj/E8fcvH6ionSCRwwtXOg9ZimeULkTa6mbsmB2 3xfS36QHvGXupqvoAHblZ6xOssEElyOU+xVODXrUM3yuqwMmeYh3m21+CPCZvqc2QM6f dakOZy92FTz72CYSDzHD7dNbE8lzCuUr1kCkb18zxZv4bK8D+bumL6XWnaKuYpSW84/J 7xCjGTBcTmMSmBS+ZegaPGhBVd3jRadGlQSCHmfFcjDrMjpX5U8BOTDYt60vE6v/6DnN 3ICg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=JmiK0IZ8ftEe9QrndmCy4xy1TM6BpU5EDxYQfhy4NiI=; b=hGKzpPvEiOUTL7VM/Njxe5X2EOB04HxbDU70vkm8txJ1Gofwo0kW+oMjDPm3/t1LLw rQa5zuwPWOLrhnk9pydLLIlRGkiwZ+KTfCItoYRsyVbSFHAN9kHFcDfa/Eu/PRXewWec Kz73jYM2QM27ekzPDbaPllT6amjItdKp8g9DGqgyFM9r4dq9Lvea+W+2DZTeKi8RF3BA 3lGgL55vyZVsv8IBIt9Mzan6WDqBUMg9OWrwr2rIF1Ld9oooaJsqvCdwNmXExq40zuce rpsFkd7XcSvDo4ULI3I6hdGltRMEPvDFUR2ukM81mIgj2aDyS7kNUG4mfbqp7Cgyg1du NyKA== X-Gm-Message-State: AOAM532Sv0Hha5/i89yefM2KBlpDhBcvpUjXcncjMgJ66vJCOhjhFDiR B2l7UiJ1fGz0rAYQP8+5fb/kxlVMA82EkA== X-Google-Smtp-Source: ABdhPJx/c6GUuqjusbN1pEwyEeEzRiqMvVt2svXzhXqCy/597Hoqpn7AWD5jflkhQjPfIKW0cLHtmQ== X-Received: by 2002:adf:fc01:0:b0:20c:ff9a:2c53 with SMTP id i1-20020adffc01000000b0020cff9a2c53mr25987204wrr.142.1653473087259; Wed, 25 May 2022 03:04:47 -0700 (PDT) Received: from vm.nix.is (vm.nix.is. [2a01:4f8:120:2468::2]) by smtp.gmail.com with ESMTPSA id o4-20020a05600002c400b0020d0cdbf7eesm1649452wry.111.2022.05.25.03.04.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 25 May 2022 03:04:46 -0700 (PDT) From: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= To: git@vger.kernel.org Cc: Junio C Hamano , Taylor Blau , =?utf-8?q?SZEDER_G=C3=A1bor?= , =?utf-8?b?xJBvw6Bu?= =?utf-8?b?IFRy4bqnbiBDw7RuZyBEYW5o?= , =?utf-8?q?Car?= =?utf-8?q?lo_Marcelo_Arenas_Bel=C3=B3n?= , Johannes Schindelin , Victoria Dye , Matheus Tavares , Lars Schneider , Eric Sunshine , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBC?= =?utf-8?b?amFybWFzb24=?= Subject: [PATCH v6 05/29] CI: remove unused Azure ci/* code Date: Wed, 25 May 2022 12:03:44 +0200 Message-Id: X-Mailer: git-send-email 2.36.1.1045.gf356b5617dd In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Remove Azure-specific code that's been unused since 6081d3898fe (ci: retire the Azure Pipelines definition, 2020-04-11). As noted in a larger removal of all of the Azure-supporting code in [1] (although that missed some of this) there is more of it in-tree, but let's focus on only the ci/* code for now. This is needed because in subsequent commits this unused code would either need to be changed to accommodate changes in ci/*, or be removed. As we'll see in those subsequent commits the end-state we're heading towards will actually make it easier to add new CI types in the future, even though the only one we're left with now is the GitHub CI. I.e. the "ci/*" framework will be made to do less, not more. We'll be offloading more of what it does to our generic build and test system. While I'm at it (since the line needs to be touched anyway) change an odd 'if test true == "$GITHUB_ACTIONS"' to the more usual style used in other places of 'if test "$GITHUB_ACTIONS" = "true"'. 1. https://lore.kernel.org/git/patch-1.1-eec0a8c3164-20211217T000418Z-avarab@gmail.com/ Signed-off-by: Ævar Arnfjörð Bjarmason --- ci/install-dependencies.sh | 3 --- ci/lib.sh | 13 +------------ ci/mount-fileshare.sh | 25 ------------------------- ci/print-test-failures.sh | 4 ---- 4 files changed, 1 insertion(+), 44 deletions(-) delete mode 100755 ci/mount-fileshare.sh diff --git a/ci/install-dependencies.sh b/ci/install-dependencies.sh index 107757a1fea..4fa3e282dda 100755 --- a/ci/install-dependencies.sh +++ b/ci/install-dependencies.sh @@ -34,8 +34,6 @@ macos-latest) export HOMEBREW_NO_AUTO_UPDATE=1 HOMEBREW_NO_INSTALL_CLEANUP=1 # Uncomment this if you want to run perf tests: # brew install gnu-time - test -z "$BREW_INSTALL_PACKAGES" || - brew install $BREW_INSTALL_PACKAGES brew link --force gettext mkdir -p $HOME/bin ( @@ -71,7 +69,6 @@ Documentation) sudo apt-get -q update sudo apt-get -q -y install asciidoc xmlto docbook-xsl-ns make - test -n "$ALREADY_HAVE_ASCIIDOCTOR" || sudo gem install --version 1.5.8 asciidoctor ;; linux-gcc-default) diff --git a/ci/lib.sh b/ci/lib.sh index e04b53b428d..ebb9461ff42 100755 --- a/ci/lib.sh +++ b/ci/lib.sh @@ -29,18 +29,7 @@ export TERM=${TERM:-dumb} # Clear MAKEFLAGS that may come from the outside world. export MAKEFLAGS= -if test -n "$SYSTEM_COLLECTIONURI" || test -n "$SYSTEM_TASKDEFINITIONSURI" -then - CI_TYPE=azure-pipelines - # We are running in Azure Pipelines - CC="${CC:-gcc}" - - export GIT_PROVE_OPTS="--timer --jobs 10 --state=failed,slow,save" - export GIT_TEST_OPTS="--verbose-log -x --write-junit-xml" - MAKEFLAGS="$MAKEFLAGS --jobs=10" - test Windows_NT != "$AGENT_OS" || - GIT_TEST_OPTS="--no-chain-lint --no-bin-wrappers $GIT_TEST_OPTS" -elif test true = "$GITHUB_ACTIONS" +if test "$GITHUB_ACTIONS" = "true" then CI_TYPE=github-actions CC="${CC_PACKAGE:-${CC:-gcc}}" diff --git a/ci/mount-fileshare.sh b/ci/mount-fileshare.sh deleted file mode 100755 index 26b58a80960..00000000000 --- a/ci/mount-fileshare.sh +++ /dev/null @@ -1,25 +0,0 @@ -#!/bin/sh - -die () { - echo "$*" >&2 - exit 1 -} - -test $# = 4 || -die "Usage: $0 " - -mkdir -p "$4" || die "Could not create $4" - -case "$(uname -s)" in -Linux) - sudo mount -t cifs -o vers=3.0,username="$2",password="$3",dir_mode=0777,file_mode=0777,serverino "$1" "$4" - ;; -Darwin) - pass="$(echo "$3" | sed -e 's/\//%2F/g' -e 's/+/%2B/g')" && - mount -t smbfs,soft "smb://$2:$pass@${1#//}" "$4" - ;; -*) - die "No support for $(uname -s)" - ;; -esac || -die "Could not mount $4" diff --git a/ci/print-test-failures.sh b/ci/print-test-failures.sh index 740ebb7c529..47e2ea1ad10 100755 --- a/ci/print-test-failures.sh +++ b/ci/print-test-failures.sh @@ -30,10 +30,6 @@ do test_name="${test_name##*/}" trash_dir="trash directory.$test_name" case "$CI_TYPE" in - azure-pipelines) - mkdir -p failed-test-artifacts - mv "$trash_dir" failed-test-artifacts - ;; github-actions) mkdir -p failed-test-artifacts echo "FAILED_TEST_ARTIFACTS=t/failed-test-artifacts" >>$GITHUB_ENV From patchwork Wed May 25 10:03:45 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= X-Patchwork-Id: 12860927 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7188AC433EF for ; Wed, 25 May 2022 10:06:56 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S241932AbiEYKFE (ORCPT ); Wed, 25 May 2022 06:05:04 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58020 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S241463AbiEYKEw (ORCPT ); Wed, 25 May 2022 06:04:52 -0400 Received: from mail-wr1-x431.google.com (mail-wr1-x431.google.com [IPv6:2a00:1450:4864:20::431]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4616D6D94A for ; Wed, 25 May 2022 03:04:50 -0700 (PDT) Received: by mail-wr1-x431.google.com with SMTP id p10so10760695wrg.12 for ; Wed, 25 May 2022 03:04:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=BSj37EzJPLuDRfP7HEYqYWfn15WSak6aOwmxzbPZScs=; b=GNKhmju+fsuA9uS/tOk4E4oEuxPHVPBQ4s43KGDiqgI0AX7Dhs9cOs5fvf8JOAr3AG OV8sjyKevLfZo1542iTRZYVS5W1Z7vu4fPwyw7bX07kY3FxyOgY3jK3613xcv0C5+ePH 8t54SgtiPBQosFJ6QB9ltULz7f50Q/CmUiMgpo70k5enRs5CIRqTq2E/OCPWSCdWA93i /QA2NKm1bftFNn6ihDHe3KD4nUwgYVPqeplgbQep1pvb8TKPWzb07gtpGycttgijyOUo I0JcR4rNrPqrp89i16kMhNvlHR6IKx2PA4y/GII6KGIRFt3QeEozy8sRUJIJbCPHEEmY JtHg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=BSj37EzJPLuDRfP7HEYqYWfn15WSak6aOwmxzbPZScs=; b=41AuVZTk4cMemEgFSA4LZ2iRrCcvN1W1OfJ8BHnfQm07YJ94mG98dArzD/SJkRPLrT MsDT88NfaFJ1dSzukQVBH3yoIoNsZEo7sC0yuDvqP3Rud5wvLl4s2WhiM/8/DZZXpYFl /zwkI9fTU3PrY/iUuqVUWrPiQRfCI9ZBQlmgYZftE2NucgPJgkp6r+i+3Oj83gmO1CVe 2oQWS4a9bmsxTaGBKSbUOAzgLv7yTLYfIQCw/+Ex21Y6IGO8+b7HD/FgutoAmpV8T0F/ QZDEJ2fr33jkql6JpKhdgNHW/Vz7Khy2SmHFKE4Wt5uhG2EQ8Sg1jTImwviJQ/C7BRuN egyw== X-Gm-Message-State: AOAM532XfcRELtZwYLe3oFvDq9Av83YAE851bmESpsML+e+ZDIfIYoQf dG6W9iEiU9UAE2wZPntKoaDMQ3/o0Y+Hfg== X-Google-Smtp-Source: ABdhPJzCD8J2rkS86payY+olU1jL5WM8V3Dcp7Gw53PCA3cOGCZDAfJD2FRSSrYQCalDKcv4BRF+3Q== X-Received: by 2002:a5d:5641:0:b0:20d:c15:1b33 with SMTP id j1-20020a5d5641000000b0020d0c151b33mr25646195wrw.66.1653473088305; Wed, 25 May 2022 03:04:48 -0700 (PDT) Received: from vm.nix.is (vm.nix.is. [2a01:4f8:120:2468::2]) by smtp.gmail.com with ESMTPSA id o4-20020a05600002c400b0020d0cdbf7eesm1649452wry.111.2022.05.25.03.04.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 25 May 2022 03:04:47 -0700 (PDT) From: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= To: git@vger.kernel.org Cc: Junio C Hamano , Taylor Blau , =?utf-8?q?SZEDER_G=C3=A1bor?= , =?utf-8?b?xJBvw6Bu?= =?utf-8?b?IFRy4bqnbiBDw7RuZyBEYW5o?= , =?utf-8?q?Car?= =?utf-8?q?lo_Marcelo_Arenas_Bel=C3=B3n?= , Johannes Schindelin , Victoria Dye , Matheus Tavares , Lars Schneider , Eric Sunshine , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBC?= =?utf-8?b?amFybWFzb24=?= Subject: [PATCH v6 06/29] CI/lib.sh: stop adding leading whitespace to $MAKEFLAGS Date: Wed, 25 May 2022 12:03:45 +0200 Message-Id: X-Mailer: git-send-email 2.36.1.1045.gf356b5617dd In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org In a8c51f77d16 (ci: clear and mark MAKEFLAGS exported just once, 2019-02-07) the assignment of MAKEFLAGS was done up-front to future-proof this code for the bug fixed in 406f93ae488 (ci: make sure we build Git parallel, 2019-02-07). But since 4a6e4b96026 (CI: remove Travis CI support, 2021-11-23) the --jobs=10 has been an invariant here, so let's assign --jobs=$NPROC to the MAKEFLAGS right away, this gets rid of the leading whitespace from the $MAKEFLAGS variable, as it would previously be empty on the first assignment. Signed-off-by: Ævar Arnfjörð Bjarmason --- ci/lib.sh | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/ci/lib.sh b/ci/lib.sh index ebb9461ff42..5f95f55a147 100755 --- a/ci/lib.sh +++ b/ci/lib.sh @@ -26,17 +26,19 @@ check_unignored_build_artifacts () # GitHub Action doesn't set TERM, which is required by tput export TERM=${TERM:-dumb} +# How many jobs to run in parallel? +NPROC=10 + # Clear MAKEFLAGS that may come from the outside world. -export MAKEFLAGS= +export MAKEFLAGS=--jobs=$NPROC if test "$GITHUB_ACTIONS" = "true" then CI_TYPE=github-actions CC="${CC_PACKAGE:-${CC:-gcc}}" - export GIT_PROVE_OPTS="--timer --jobs 10" + export GIT_PROVE_OPTS="--timer --jobs $NPROC" export GIT_TEST_OPTS="--verbose-log -x" - MAKEFLAGS="$MAKEFLAGS --jobs=10" test Windows != "$RUNNER_OS" || GIT_TEST_OPTS="--no-chain-lint --no-bin-wrappers $GIT_TEST_OPTS" else From patchwork Wed May 25 10:03:46 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= X-Patchwork-Id: 12860916 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8F6EBC433FE for ; Wed, 25 May 2022 10:05:15 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S241721AbiEYKFC (ORCPT ); Wed, 25 May 2022 06:05:02 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58108 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S241492AbiEYKEw (ORCPT ); Wed, 25 May 2022 06:04:52 -0400 Received: from mail-wr1-x42a.google.com (mail-wr1-x42a.google.com [IPv6:2a00:1450:4864:20::42a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2456B6E8ED for ; Wed, 25 May 2022 03:04:51 -0700 (PDT) Received: by mail-wr1-x42a.google.com with SMTP id e2so17781637wrc.1 for ; Wed, 25 May 2022 03:04:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=xwcDN2nNxoVnG4m23bb8Qfy2kECmBICM1NIuw/Hu06w=; b=aY4BQAAzHYE174OgOuwWfbJfbWqTp5Oun+gHjyfXWmDTCX+KlRhKfQ7rOVsVouUwhk ey1HKhf974Eine3kUHMq0+muxz+QZuyog2LPyer7jmYx6lu4F8wEWtB7F04GlmIXe14O E9JWXR75wi1OAUPoW/kViHqSVWzFhUcTku0F9bSWbL8JrI6l1IDUkyiw3RVQiywB3O9j fztL4ijIUeT0TCPItOTjDc3LTggW82jYbMwXm4WRF3YgYc5pZdkM9RvR3aQY1uZRzm52 ZiE21hmBMCdoL6iWoszSLBqqjra1bdInZPMJLJU1TDqlshTUQFfIngOHpCaqHFibLnzQ bKbw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=xwcDN2nNxoVnG4m23bb8Qfy2kECmBICM1NIuw/Hu06w=; b=0d7OMpeNVus18F6RWhQpvQsMO2daegjsb3q8sv7cgQvL01p1G8u9Qd6cf96KMhAw7R cu2yRz8NzaejeRWu+63/IBXiW4gTCGp6BYLUQ35/9gRUm6Dy7BoRUJ/Kw2D/n6h4pfiS 0Odwp9PU5uZWEsuRoZCuw3hcUG/G+IXnPfWs33Dmiu/3VhTBuMif3S8y1ActLIKUsznN CrNPnrdMgLREjvZkLbuO0sYPwqLb7OtoAH3GD1OX7DIBpqnKI3aWSAdXa5JRx5lF3ThA EBooqWBi/njt9yojqIHDGzR4Mmlr7LOWxeP/lfFDTJaQrOfJxocIpmRl7I/p6GcEXFW/ 2XQg== X-Gm-Message-State: AOAM531j9ZaYpHbALhWhG4WJNhHwsXy2hQrOZ8T7QNqcpZEI0KxT/Sqe 2Hl6tWTpHDEcdY1d+hMiKLjpE3D8QMqaoQ== X-Google-Smtp-Source: ABdhPJzx+HyhYIWUUxSu/XXVXp5HBSpeUil53gwyaWjJRHSMRdHFpbf12JL+9nLutFOwG1HlQQDpDQ== X-Received: by 2002:a5d:678c:0:b0:20e:699d:1877 with SMTP id v12-20020a5d678c000000b0020e699d1877mr25549754wru.490.1653473089316; Wed, 25 May 2022 03:04:49 -0700 (PDT) Received: from vm.nix.is (vm.nix.is. [2a01:4f8:120:2468::2]) by smtp.gmail.com with ESMTPSA id o4-20020a05600002c400b0020d0cdbf7eesm1649452wry.111.2022.05.25.03.04.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 25 May 2022 03:04:48 -0700 (PDT) From: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= To: git@vger.kernel.org Cc: Junio C Hamano , Taylor Blau , =?utf-8?q?SZEDER_G=C3=A1bor?= , =?utf-8?b?xJBvw6Bu?= =?utf-8?b?IFRy4bqnbiBDw7RuZyBEYW5o?= , =?utf-8?q?Car?= =?utf-8?q?lo_Marcelo_Arenas_Bel=C3=B3n?= , Johannes Schindelin , Victoria Dye , Matheus Tavares , Lars Schneider , Eric Sunshine , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBC?= =?utf-8?b?amFybWFzb24=?= Subject: [PATCH v6 07/29] CI: don't have "git grep" invoke a pager in tree content check Date: Wed, 25 May 2022 12:03:46 +0200 Message-Id: X-Mailer: git-send-email 2.36.1.1045.gf356b5617dd In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Fix the omission of not adding "-P" (a.k.a. "--no-pager") to the "git" invocation that spaws "git grep", added in 0e7696c64db (ci: disallow directional formatting, 2021-11-04). This didn't matter for the GitHub CI, but does when running the check locally. A subsequent commit will start invoking it via a new check from the top-level Makefile, and without this we'd visibly spawn a pager when doing so. Signed-off-by: Ævar Arnfjörð Bjarmason --- ci/check-directional-formatting.bash | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/ci/check-directional-formatting.bash b/ci/check-directional-formatting.bash index e6211b141aa..de8e6e486f0 100755 --- a/ci/check-directional-formatting.bash +++ b/ci/check-directional-formatting.bash @@ -23,5 +23,5 @@ # U+2066..U+2069: LRI, RLI, FSI and PDI regex='(\u202a|\u202b|\u202c|\u202d|\u202e|\u2066|\u2067|\u2068|\u2069)' -! LC_CTYPE=C git grep -El "$(LC_CTYPE=C.UTF-8 printf "$regex")" \ +! LC_CTYPE=C git -P grep -El "$(LC_CTYPE=C.UTF-8 printf "$regex")" \ -- ':(exclude,attr:binary)' ':(exclude)*.po' From patchwork Wed May 25 10:03:47 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= X-Patchwork-Id: 12860928 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 915E2C433F5 for ; Wed, 25 May 2022 10:06:56 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S242062AbiEYKFH (ORCPT ); Wed, 25 May 2022 06:05:07 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58184 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S241322AbiEYKEx (ORCPT ); Wed, 25 May 2022 06:04:53 -0400 Received: from mail-wr1-x436.google.com (mail-wr1-x436.google.com [IPv6:2a00:1450:4864:20::436]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1CD3C68FB0 for ; Wed, 25 May 2022 03:04:52 -0700 (PDT) Received: by mail-wr1-x436.google.com with SMTP id k30so29386402wrd.5 for ; Wed, 25 May 2022 03:04:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=ZxdyJpHHnZSDeVz0Pez10G6kzVb7IN2rA/U/3nkx5kM=; b=m5/zcUAudDjO3OLMRfLw4Tyy+keEWy1BBOfgaW5B4mnPbBkgUKHOA/tvEEn2H4aWV2 whAA86Jivxf/AoyHbH2QWF2GvibdTbk14fhYxkBW3G8n7Fzm5goQJvBvgVgN+Oc4KHY+ WUUR/3h6MvDtPgRYVFtWrL9hGJN4eAgf6L+vstr6bKXIAivLd2jFwsCIMAUUJCJcPvz5 FcwCqgW/a0zxRgijWyFBaXdfSlqq/fob2ds5GoHKJky129tJCp4QEOQu3mNAX+eknuQM 6a1UseYPEuz2XsB99e1q8+Vr8WcWR3h04NLFeuSA0jyfOWBeom3LFmKA1glNrm7mVbJB iDgQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=ZxdyJpHHnZSDeVz0Pez10G6kzVb7IN2rA/U/3nkx5kM=; b=oGv9xaVK/lTAasxTWVlni2j+Ch1/PhgLlTd5CiLKi30oaXa1gTfKsJG1o/FDZiR3KR cK4azVhwm7TAqjUTvpn1LmZ32ogJmAe7xTZzNHgs9bQG5DAapFGyp8oM2ZoMj1lJO0N0 olpqj+V5RRhZp5BWrxLoFjwiijDaTrm7q4YXR/T+QKVC9tGGKR5FcEee70eJnz7UsZrR ot6OuEuPQjw2kPP8ETYrzryA6J38a6jQIw+64fYgmAvomXUwYB3yPJ37tPeU2pJDm8hE VsJRRFT5MLC/eLNMyOy+1K4h/vMN9gbwG8HdpOQyvN4jU6yyd3A//CiugZTxi8kh3hGc dc6A== X-Gm-Message-State: AOAM530djNbW1s+gHbxa2tSiVpTibm/3lG3ODGe33/Rolkvbo61fXtvS tB8mohPWlZbssaQjgrJC1skzEemYARckgg== X-Google-Smtp-Source: ABdhPJwBuplKS8xyB/xBnFtotofXbZFYRkt3j5x88/j0nawDSSFaHkwC0zfmA58e1APpT8UMgbgQwg== X-Received: by 2002:a5d:6e88:0:b0:20e:5f83:46fa with SMTP id k8-20020a5d6e88000000b0020e5f8346famr26413818wrz.475.1653473090328; Wed, 25 May 2022 03:04:50 -0700 (PDT) Received: from vm.nix.is (vm.nix.is. [2a01:4f8:120:2468::2]) by smtp.gmail.com with ESMTPSA id o4-20020a05600002c400b0020d0cdbf7eesm1649452wry.111.2022.05.25.03.04.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 25 May 2022 03:04:49 -0700 (PDT) From: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= To: git@vger.kernel.org Cc: Junio C Hamano , Taylor Blau , =?utf-8?q?SZEDER_G=C3=A1bor?= , =?utf-8?b?xJBvw6Bu?= =?utf-8?b?IFRy4bqnbiBDw7RuZyBEYW5o?= , =?utf-8?q?Car?= =?utf-8?q?lo_Marcelo_Arenas_Bel=C3=B3n?= , Johannes Schindelin , Victoria Dye , Matheus Tavares , Lars Schneider , Eric Sunshine , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBC?= =?utf-8?b?amFybWFzb24=?= Subject: [PATCH v6 08/29] CI: have "static-analysis" run a "make ci-static-analysis" target Date: Wed, 25 May 2022 12:03:47 +0200 Message-Id: X-Mailer: git-send-email 2.36.1.1045.gf356b5617dd In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Make it easier to run the CI logic locally by creating "ci-static-analysis" target. Now the "check-coccicheck" logic added in 0860a7641ba (travis-ci: fail if Coccinelle static analysis found something to transform, 2018-07-23) can be easily tested outside of CI, and likewise the sanity check added in 0e7696c64db (ci: disallow directional formatting, 2021-11-04). These targets could be improved, the "hdr-check" target will re-do all of its work every time it's run, and could with a change similar to my c234e8a0ecf (Makefile: make the "sparse" target non-.PHONY, 2021-09-23). The same goes for the new "ci-check-directional-formatting" target. But without those improvements being able to easily run a 1=1 equivalent of the checks we do in CI during a local build is already an improvement. Subsequent commits will expand on this ability. This change will also make the CI check faster, since we can take advantage of parallelism. The "make coccicheck" takes a very long time, and at the tail end we might only have one job pegging 100% of one CPU core for a long time. Now any extra cores will be free to run the rest of the targets under "ci-static-analysis". Signed-off-by: Ævar Arnfjörð Bjarmason --- .github/workflows/main.yml | 3 +-- Makefile | 28 ++++++++++++++++++++++++++++ ci/run-static-analysis.sh | 30 ------------------------------ shared.mak | 1 + 4 files changed, 30 insertions(+), 32 deletions(-) delete mode 100755 ci/run-static-analysis.sh diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml index 309b9141249..ad752010102 100644 --- a/.github/workflows/main.yml +++ b/.github/workflows/main.yml @@ -317,8 +317,7 @@ jobs: steps: - uses: actions/checkout@v2 - run: ci/install-dependencies.sh - - run: ci/run-static-analysis.sh - - run: ci/check-directional-formatting.bash + - run: make ci-static-analysis sparse: needs: ci-config if: needs.ci-config.outputs.enabled == 'yes' diff --git a/Makefile b/Makefile index 61aadf3ce88..9ae7e1a373e 100644 --- a/Makefile +++ b/Makefile @@ -3034,6 +3034,20 @@ coccicheck: $(addsuffix .patch,$(filter-out %.pending.cocci,$(wildcard contrib/c # See contrib/coccinelle/README coccicheck-pending: $(addsuffix .patch,$(wildcard contrib/coccinelle/*.pending.cocci)) +.PHONY: check-coccicheck +check-coccicheck: coccicheck + $(QUIET_CHECK)for cocci_patch in contrib/coccinelle/*.patch; do \ + if test -s "$$cocci_patch"; then \ + echo "Coccinelle suggests the following changes in '$$cocci_patch':"; \ + cat "$$cocci_patch"; \ + fail=UnfortunatelyYes; \ + fi \ + done; \ + if test -n "$$fail"; then \ + echo "error: Coccinelle suggested some changes"; \ + exit 1; \ + fi + .PHONY: coccicheck coccicheck-pending ### Installation rules @@ -3469,3 +3483,17 @@ $(FUZZ_PROGRAMS): all $(XDIFF_OBJS) $(EXTLIBS) git.o $@.o $(LIB_FUZZING_ENGINE) -o $@ fuzz-all: $(FUZZ_PROGRAMS) + +### CI "check" targets +# +# These targets are run from the CI, see .github/workflows/main.yml, +# but can also be run manually to run the same assertions locally. + +.PHONY: ci-check-directional-formatting +ci-check-directional-formatting: + $(QUIET_CHECK)ci/check-directional-formatting.bash + +.PHONY: ci-static-analysis +ci-static-analysis: ci-check-directional-formatting +ci-static-analysis: check-coccicheck +ci-static-analysis: hdr-check diff --git a/ci/run-static-analysis.sh b/ci/run-static-analysis.sh deleted file mode 100755 index 5a87b1cac96..00000000000 --- a/ci/run-static-analysis.sh +++ /dev/null @@ -1,30 +0,0 @@ -#!/bin/sh -# -# Perform various static code analysis checks -# - -. ${0%/*}/lib.sh - -make coccicheck - -set +x - -fail= -for cocci_patch in contrib/coccinelle/*.patch -do - if test -s "$cocci_patch" - then - echo "$(tput setaf 1)Coccinelle suggests the following changes in '$cocci_patch':$(tput sgr0)" - cat "$cocci_patch" - fail=UnfortunatelyYes - fi -done - -if test -n "$fail" -then - echo "$(tput setaf 1)error: Coccinelle suggested some changes$(tput sgr0)" - exit 1 -fi - -make hdr-check || -exit 1 diff --git a/shared.mak b/shared.mak index 50d4596f0d6..83e0aa16e7d 100644 --- a/shared.mak +++ b/shared.mak @@ -68,6 +68,7 @@ ifndef V QUIET_HDR = @echo ' ' HDR $(<:hcc=h); QUIET_RC = @echo ' ' RC $@; QUIET_SPATCH = @echo ' ' SPATCH $<; + QUIET_CHECK = @echo ' ' CHECK $@; ## Used in "Documentation/Makefile" QUIET_ASCIIDOC = @echo ' ' ASCIIDOC $@; From patchwork Wed May 25 10:03:48 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= X-Patchwork-Id: 12860918 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C6290C433EF for ; Wed, 25 May 2022 10:05:19 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S241463AbiEYKFQ (ORCPT ); Wed, 25 May 2022 06:05:16 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58238 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S241450AbiEYKEy (ORCPT ); Wed, 25 May 2022 06:04:54 -0400 Received: from mail-wr1-x433.google.com (mail-wr1-x433.google.com [IPv6:2a00:1450:4864:20::433]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id EECE46E8C3 for ; Wed, 25 May 2022 03:04:52 -0700 (PDT) Received: by mail-wr1-x433.google.com with SMTP id f2so29435607wrc.0 for ; Wed, 25 May 2022 03:04:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=+G96LJNZc5E4Mv+6QJTtQVuzM5ldVC5qI9hZ7L2MoZA=; b=lCDNbUsBD5v54KC0x7oFE8UlaPayAlVKL9xMCdv9duA4htzl6gbS7037bi6AJpfcSv TbkEKiCNIU2dSfYgxd656Tdewxv290IPKbXNiGVSNIt7Bb307XK0xxSrG6NQ+FEzGq02 scE5UX0gQMgoA8dpPI0EsGOc8RFexGjewv3qMpwQ9X3ULDxbDuxkirFtDWmhW2DKYlum PQF6yhFjOVJpNE6o+dhLyEh0hv8YiAp13q7JRRhrkBOyGjfMZZROaMVGsOsN4cHzo+G1 6PFdhHAo3b2ZPHlT0L53Uc+l5bcmTXwtePASfo2crFVNJ+lAmYXGFOn+GEKKSU6YAvn3 6dKg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=+G96LJNZc5E4Mv+6QJTtQVuzM5ldVC5qI9hZ7L2MoZA=; b=AORt9PRk3pdqRtoq3pCVSqb0sI8N4FPntuD8Rl7ZmASPWHOT++h0WWfy6wtmU3TUGl hUIvZUD+R39MJWLjIgD9y80Ipp1yICANrKCOmR1+fV1YIZ/rccpPUppH2hwRadX1rsbj Gz2dN6XPPCi0nSHA7w3ri7CXniK0fjwCXqcCfEz9Zqf0r/eAHmPLGhfdVtbK0OV4T2y2 AFJBhWlCoFNQ0COsoQkR+t7IZGSlr6AfUZEcy0XzUrDCeNNXKYuiot8tQHgHVN3NR5gn F/BAwlY+/GrHCHuYMao+szOBpiz2EbJwR+2+SP8lU+JKaCIfgbfc/MPdA3SN/OAdqOZY fPYg== X-Gm-Message-State: AOAM531ZLF+fLZ6SqtU+vc6x2+hgGMZvf7yoqkR9dKDoDtvS1SyZIyAs nPhTbToHqU/qy36Z9vlUht7PHZOcvta16w== X-Google-Smtp-Source: ABdhPJw0gWcAaThwxiK8+H4mVRLMu+d6l0oFrzFSd/7Gz92lvHYu0vOpmZcANsdjGi+QeRCc3WOILw== X-Received: by 2002:a05:6000:354:b0:20f:fe61:88fd with SMTP id e20-20020a056000035400b0020ffe6188fdmr1968415wre.339.1653473091287; Wed, 25 May 2022 03:04:51 -0700 (PDT) Received: from vm.nix.is (vm.nix.is. [2a01:4f8:120:2468::2]) by smtp.gmail.com with ESMTPSA id o4-20020a05600002c400b0020d0cdbf7eesm1649452wry.111.2022.05.25.03.04.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 25 May 2022 03:04:50 -0700 (PDT) From: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= To: git@vger.kernel.org Cc: Junio C Hamano , Taylor Blau , =?utf-8?q?SZEDER_G=C3=A1bor?= , =?utf-8?b?xJBvw6Bu?= =?utf-8?b?IFRy4bqnbiBDw7RuZyBEYW5o?= , =?utf-8?q?Car?= =?utf-8?q?lo_Marcelo_Arenas_Bel=C3=B3n?= , Johannes Schindelin , Victoria Dye , Matheus Tavares , Lars Schneider , Eric Sunshine , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBC?= =?utf-8?b?amFybWFzb24=?= Subject: [PATCH v6 09/29] CI: have "static-analysis" run "check-builtins", not "documentation" Date: Wed, 25 May 2022 12:03:48 +0200 Message-Id: X-Mailer: git-send-email 2.36.1.1045.gf356b5617dd In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Move the "make check-builtins" check from the "documentation" job to the "static-analysis" job. The "check-builtins" target added in c74390e4a1d (cherry is built-in, do not ship git-cherry.sh, 2006-11-05) is unrelated to the documentation, so it's odd that b98712b9aa9 (travis-ci: build documentation, 2016-05-04) added it to the "documentation" job. Let's just move it to the "ci-static-analysis" target, and while we're at it improve the output with $(QUIET_CHECK). Signed-off-by: Ævar Arnfjörð Bjarmason --- Makefile | 3 ++- ci/test-documentation.sh | 1 - 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/Makefile b/Makefile index 9ae7e1a373e..3398d3f2e56 100644 --- a/Makefile +++ b/Makefile @@ -3405,7 +3405,7 @@ check-docs:: ### Make sure built-ins do not have dups and listed in git.c # check-builtins:: - ./check-builtins.sh + $(QUIET_CHECK)./check-builtins.sh ### Test suite coverage testing # @@ -3495,5 +3495,6 @@ ci-check-directional-formatting: .PHONY: ci-static-analysis ci-static-analysis: ci-check-directional-formatting +ci-static-analysis: check-builtins ci-static-analysis: check-coccicheck ci-static-analysis: hdr-check diff --git a/ci/test-documentation.sh b/ci/test-documentation.sh index 9e0652c30dd..41e2b126311 100755 --- a/ci/test-documentation.sh +++ b/ci/test-documentation.sh @@ -14,7 +14,6 @@ filter_log () { "$1" } -make check-builtins make check-docs # Build docs with AsciiDoc From patchwork Wed May 25 10:03:49 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= X-Patchwork-Id: 12860919 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3C620C433F5 for ; Wed, 25 May 2022 10:05:21 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S242226AbiEYKFU (ORCPT ); Wed, 25 May 2022 06:05:20 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58410 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S241512AbiEYKEz (ORCPT ); Wed, 25 May 2022 06:04:55 -0400 Received: from mail-wr1-x436.google.com (mail-wr1-x436.google.com [IPv6:2a00:1450:4864:20::436]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0BCBE6D970 for ; Wed, 25 May 2022 03:04:54 -0700 (PDT) Received: by mail-wr1-x436.google.com with SMTP id t13so10213608wrg.9 for ; Wed, 25 May 2022 03:04:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=pv23aLRm7M7M9zoZ6G4oiLO8NHwTny1h9XdPo91/9/o=; b=gXXrnOyMSGqIbtCgYyPmBErucLEa2eFKLTDLoYoD0A8JdOE20tHaRIXGXWys7b2Ubu I3whN4ajanbFQtBwDW7Elp+rXPuJYwTSvZljIpCT4305oLvPXtvLQqu5JapfwCwmmw3a T/PtR+9KsgfnYhQzYyJkh7NRub4h0633J3bGhkS74kWSrWmo5OSRlrQD7PGDzpnBUhR7 FiO+loZleAUP2/UoEXc2n2TZ9GU7U+HvBFBn2w2LqPIsfI75YibDDW63dOVRI/YPtgk3 lq/4xGJJ0LDQ3SxwMmbkr7D3YR3jMyraRqeuSRdbd4yLiXTGggLkhyLQGd4A+hfdEJ/K kZxQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=pv23aLRm7M7M9zoZ6G4oiLO8NHwTny1h9XdPo91/9/o=; b=T3/fRvlhlr5WC5ANM+DVKNdVLBPiCJRr7FtPz4UJi9q1x6FjatNXZtnonD6kMxBlgq PmB/BjXkLDKb8jaokEwLSSmK0dllG4UdteP/nZ20f6H23AY/r+v5GeZZHGwgNS6Iyd2k r+WqOKOBIhlBew92cP8bwGhmme1yqdAqhiV9JQPpa0qYD8yEb1opL6uWjoYdti+aNpps IFj3Gj/pDD1TjBnPf6AeqCv97/NKdWhF16SKlJJf3wQuVpKoNiw8bBO9ahFRHstuFbAd Hzo3CRoUeBNf2xOwq3h/25oafW1jCZz5vyL1YnhkMEXCAaG11glp7SZVkiiQBFhC++P+ gGxw== X-Gm-Message-State: AOAM532e7JmlND48zlEHv9vVYbxx9XyjvaUvYsZiSrl3UY75JVzuzx5U 30JBw3AM+7EoIH+H2aZf/OjOnCkRdPKcjA== X-Google-Smtp-Source: ABdhPJxjfiiPM951xOjOpWQZAbvo1dTixVC9yMNS5aJAVtVyutavwjitvF4XbPTPLodgEAVg6JSCgw== X-Received: by 2002:a05:6000:791:b0:20e:615c:aae4 with SMTP id bu17-20020a056000079100b0020e615caae4mr26535754wrb.206.1653473092293; Wed, 25 May 2022 03:04:52 -0700 (PDT) Received: from vm.nix.is (vm.nix.is. [2a01:4f8:120:2468::2]) by smtp.gmail.com with ESMTPSA id o4-20020a05600002c400b0020d0cdbf7eesm1649452wry.111.2022.05.25.03.04.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 25 May 2022 03:04:51 -0700 (PDT) From: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= To: git@vger.kernel.org Cc: Junio C Hamano , Taylor Blau , =?utf-8?q?SZEDER_G=C3=A1bor?= , =?utf-8?b?xJBvw6Bu?= =?utf-8?b?IFRy4bqnbiBDw7RuZyBEYW5o?= , =?utf-8?q?Car?= =?utf-8?q?lo_Marcelo_Arenas_Bel=C3=B3n?= , Johannes Schindelin , Victoria Dye , Matheus Tavares , Lars Schneider , Eric Sunshine , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBC?= =?utf-8?b?amFybWFzb24=?= Subject: [PATCH v6 10/29] CI: move p4 and git-lfs variables to ci/install-dependencies.sh Date: Wed, 25 May 2022 12:03:49 +0200 Message-Id: X-Mailer: git-send-email 2.36.1.1045.gf356b5617dd In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Move the declaration of variables that are only used by the "ubuntu-latest" block in "ci/install-dependencies.sh" there from "ci/lib.sh". This makes the code easier to follow, and changes "ci/lib.sh" to a library that only exports CI variables for general use, we didn't need to export these $P4_PATH and $GIT_LFS_PATH variables. Signed-off-by: Ævar Arnfjörð Bjarmason --- ci/install-dependencies.sh | 17 +++++++++++++++-- ci/lib.sh | 12 ------------ 2 files changed, 15 insertions(+), 14 deletions(-) diff --git a/ci/install-dependencies.sh b/ci/install-dependencies.sh index 4fa3e282dda..41a9185bb1e 100755 --- a/ci/install-dependencies.sh +++ b/ci/install-dependencies.sh @@ -5,14 +5,27 @@ . ${0%/*}/lib.sh -P4WHENCE=https://cdist2.perforce.com/perforce/r$LINUX_P4_VERSION -LFSWHENCE=https://github.com/github/git-lfs/releases/download/v$LINUX_GIT_LFS_VERSION UBUNTU_COMMON_PKGS="make libssl-dev libcurl4-openssl-dev libexpat-dev tcl tk gettext zlib1g-dev perl-modules liberror-perl libauthen-sasl-perl libemail-valid-perl libio-socket-ssl-perl libnet-smtp-ssl-perl" case "$runs_on_pool" in ubuntu-latest) + # The Linux build installs the defined dependency versions below. + # The OS X build installs much more recent versions, whichever + # were recorded in the Homebrew database upon creating the OS X + # image. + # Keep that in mind when you encounter a broken OS X build! + LINUX_P4_VERSION="16.2" + LINUX_GIT_LFS_VERSION="1.5.2" + + P4_PATH="$HOME/custom/p4" + GIT_LFS_PATH="$HOME/custom/git-lfs" + export PATH="$GIT_LFS_PATH:$P4_PATH:$PATH" + + P4WHENCE=https://cdist2.perforce.com/perforce/r$LINUX_P4_VERSION + LFSWHENCE=https://github.com/github/git-lfs/releases/download/v$LINUX_GIT_LFS_VERSION + sudo apt-get -q update sudo apt-get -q -y install language-pack-is libsvn-perl apache2 \ $UBUNTU_COMMON_PKGS $CC_PACKAGE diff --git a/ci/lib.sh b/ci/lib.sh index 5f95f55a147..e789cd2a1cd 100755 --- a/ci/lib.sh +++ b/ci/lib.sh @@ -67,18 +67,6 @@ ubuntu-latest) fi export GIT_TEST_HTTPD=true - - # The Linux build installs the defined dependency versions below. - # The OS X build installs much more recent versions, whichever - # were recorded in the Homebrew database upon creating the OS X - # image. - # Keep that in mind when you encounter a broken OS X build! - export LINUX_P4_VERSION="16.2" - export LINUX_GIT_LFS_VERSION="1.5.2" - - P4_PATH="$HOME/custom/p4" - GIT_LFS_PATH="$HOME/custom/git-lfs" - export PATH="$GIT_LFS_PATH:$P4_PATH:$PATH" ;; macos-latest) if [ "$jobname" = osx-gcc ] From patchwork Wed May 25 10:03:50 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= X-Patchwork-Id: 12860920 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id CA7ADC433F5 for ; Wed, 25 May 2022 10:05:24 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S242252AbiEYKFW (ORCPT ); Wed, 25 May 2022 06:05:22 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58500 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S241540AbiEYKE4 (ORCPT ); Wed, 25 May 2022 06:04:56 -0400 Received: from mail-wm1-x32c.google.com (mail-wm1-x32c.google.com [IPv6:2a00:1450:4864:20::32c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4DF1C6D961 for ; Wed, 25 May 2022 03:04:55 -0700 (PDT) Received: by mail-wm1-x32c.google.com with SMTP id z17so5306459wmf.1 for ; Wed, 25 May 2022 03:04:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=hsfcG5Is4/F/Q6NsqyPjH2sSp8Hxv3gJNN7z4CWwErQ=; b=j1WU1R3XYdefsn6eg4kM+SReWFir2HGnTJ0+2tJi6+XLahWrQrwifN/7KRVs6VRlfc SUCVd5oljZ6FUJicCbuJXSVof/TgD/CFj1rpcWNJqP6X7XpBmoRoS9qC7Lf2mSH+aRZO dSEEXcFNzUvw3Y4VtD3Bb5tlzub5pBoQ3Cc0Kybx02w8pOUEAmufioKhMNK8Zl2u1N2x fNkGUmEXkHPRLhmUlnnJnJ4IQw3XVzfEysupfQYvxCpGTnQ77oqEEhrjVoOgxLSWpMGz FNTCmMJ5WTtjhObOfDhw5eYAaqyjTUVx8GdkKqGRtGZDGHb5SDxFFPK642IB1k1Bxx/0 70FQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=hsfcG5Is4/F/Q6NsqyPjH2sSp8Hxv3gJNN7z4CWwErQ=; b=7ojF9vsPNjEovuC352Nof486gocQfn4xmPqMYVVvTiY1aeplXR4wJVBgQqYVv8SzCu tOG5s8qgLtQRzIbP9gB6k2ZVZRtwFORhVK3jQtA0oNjbcqFdZhn+DNfgAGRbVla+2ywd ThB+LtH8DS0El/jAMvM6MD2gL1SB7YXBPcmjFdfkfSilVocn6VPtUluA/sBNximwPJ2g 9O5UBqaVdad9SgnAyCp6KPxF616kXj7cMcmNi1SYRmsPOA4ObIQdhs45yx5LCsWualQK Rklkpr1QevWaSo6spUt2T9jTixKyaxuRmKgM0VTlKzmSlGdH933tUlAlTfZDkVcdPK9W 25tA== X-Gm-Message-State: AOAM533j9gRp2lUBYc/C20MqFLszekYHA08Z9thMFj/nmRsfeAQvbbIv 8OlZqTrH/rQPMO0G2z3Kf5lwui5fz8eXkA== X-Google-Smtp-Source: ABdhPJykavSUkSWiKt0LfgAIljJvCI51e1uugf6RYou2F9QpZJWfEMm0tHT5COKtxlQ9z8zzgVZEuA== X-Received: by 2002:a1c:7912:0:b0:397:41e9:872d with SMTP id l18-20020a1c7912000000b0039741e9872dmr7390861wme.184.1653473093313; Wed, 25 May 2022 03:04:53 -0700 (PDT) Received: from vm.nix.is (vm.nix.is. [2a01:4f8:120:2468::2]) by smtp.gmail.com with ESMTPSA id o4-20020a05600002c400b0020d0cdbf7eesm1649452wry.111.2022.05.25.03.04.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 25 May 2022 03:04:52 -0700 (PDT) From: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= To: git@vger.kernel.org Cc: Junio C Hamano , Taylor Blau , =?utf-8?q?SZEDER_G=C3=A1bor?= , =?utf-8?b?xJBvw6Bu?= =?utf-8?b?IFRy4bqnbiBDw7RuZyBEYW5o?= , =?utf-8?q?Car?= =?utf-8?q?lo_Marcelo_Arenas_Bel=C3=B3n?= , Johannes Schindelin , Victoria Dye , Matheus Tavares , Lars Schneider , Eric Sunshine , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBC?= =?utf-8?b?amFybWFzb24=?= Subject: [PATCH v6 11/29] CI: consistently use "export" in ci/lib.sh Date: Wed, 25 May 2022 12:03:50 +0200 Message-Id: X-Mailer: git-send-email 2.36.1.1045.gf356b5617dd In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Change the "ci/lib.sh" script co consistently use "export", for e.g. MAKEFLAGS we were exporting it, and then assigning to it, let's do it the other way around. Right now this doesn't matter, since we in e.g. "ci/install-dependencies.sh" source this file, and don't use something like "env(1)" to retrieve these variables. But in a subsequent commit we'll "export" these variables through a wrapper (to additionally write them to a GitHub CI-specific $GITHUB_ENV file). This change makes that subsequent change easier to read, as it won't need to do any control flow refactoring. Signed-off-by: Ævar Arnfjörð Bjarmason --- ci/lib.sh | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/ci/lib.sh b/ci/lib.sh index e789cd2a1cd..1b5b805c5ca 100755 --- a/ci/lib.sh +++ b/ci/lib.sh @@ -30,7 +30,7 @@ export TERM=${TERM:-dumb} NPROC=10 # Clear MAKEFLAGS that may come from the outside world. -export MAKEFLAGS=--jobs=$NPROC +MAKEFLAGS=--jobs=$NPROC if test "$GITHUB_ACTIONS" = "true" then @@ -38,9 +38,10 @@ then CC="${CC_PACKAGE:-${CC:-gcc}}" export GIT_PROVE_OPTS="--timer --jobs $NPROC" - export GIT_TEST_OPTS="--verbose-log -x" + GIT_TEST_OPTS="--verbose-log -x" test Windows != "$RUNNER_OS" || GIT_TEST_OPTS="--no-chain-lint --no-bin-wrappers $GIT_TEST_OPTS" + export GIT_TEST_OPTS else echo "Could not identify CI type" >&2 env >&2 @@ -94,4 +95,4 @@ linux-leaks) ;; esac -MAKEFLAGS="$MAKEFLAGS CC=${CC:-cc}" +export MAKEFLAGS="$MAKEFLAGS CC=${CC:-cc}" From patchwork Wed May 25 10:03:51 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= X-Patchwork-Id: 12860929 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id CE6F8C433EF for ; Wed, 25 May 2022 10:06:59 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237674AbiEYKG6 (ORCPT ); Wed, 25 May 2022 06:06:58 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58622 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S241582AbiEYKE5 (ORCPT ); Wed, 25 May 2022 06:04:57 -0400 Received: from mail-wm1-x332.google.com (mail-wm1-x332.google.com [IPv6:2a00:1450:4864:20::332]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2C12E6E8F5 for ; Wed, 25 May 2022 03:04:56 -0700 (PDT) Received: by mail-wm1-x332.google.com with SMTP id n124-20020a1c2782000000b003972dfca96cso3019844wmn.4 for ; Wed, 25 May 2022 03:04:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=IhzSjn0WuXr2jsAs+JpNIYgmHEjB7UXzOKchE/FG7x0=; b=lswXSmRy6pj8mHiIXukX1yVB1G1QOQCTX15f4M0Sjz3zxKCGNyCQap18xQOUnJVF3P OD6Fw9JU8paNeDI2VUsllifBkXvJjKNQU+BqZxrkmNQMvWUjywldS/DoJr4vyXLhwCMR VMbgTIa5pxeF7Ljo8D1nNng61L9TZER/bHFWkj75drvM0Ik5RoMaGlibHzmAgZ9L3iVl jXtrRmdyHq2d37QAJUSnSoykiNw1nEyeN3eZoIiTF+h/k0USJrE62eIoX0JvLL0vRjfg iW4A/a7iBxsuw3D5q49iUztFkVhjntvxys22Wek3QwdQNSY4DOoCF+lI+l81lI6iKH7k 8+nQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=IhzSjn0WuXr2jsAs+JpNIYgmHEjB7UXzOKchE/FG7x0=; b=3vtDOX0S6M6gO5AaZ96T+TFRXxK7TcB7SkFAaTIqfJyNxfgs2bu1gNS7T285Y30vvJ zBE15UTRYgLw2ZuQSJyiCocVDH44NrO6ETw5eSZDJMVQ+EcW9xLE8gypWSI1i61WYHhG fzas3w+Zg9ac60Fpmpm4y0WCHZSkeBNKuD2lNUtvg0TFM395DLG9nqWz7gaiMKjhSNoR oZpnGR+3CAHVUJ3r8kYSTirTF74RpPlpLzrev3VI4gqnkRnRwMD5P2qp8MZyoaAs77LV 7Nych2LY2xYJnGM27uUQcn1IoIdEFJnccS46BA5byvJ+IMmMppPitW34dvsoHvZ/wuIk ZB4Q== X-Gm-Message-State: AOAM530K6oyZYN6hrNgflteBjsyZTZL6r2ysYZyOtNaBuBbncD6eCWll t8dqllxLoUQlLs7kU0K1n6nUVa3y3YfeKg== X-Google-Smtp-Source: ABdhPJyMWug7+8zoBdx3LkEsArAjhHC+Z2WEPuuJGhGW0QXv3L2/nvvfcU87ILxZ9QfMuL4+TGLFzQ== X-Received: by 2002:a05:600c:35cc:b0:394:7b56:ef38 with SMTP id r12-20020a05600c35cc00b003947b56ef38mr7619915wmq.34.1653473094404; Wed, 25 May 2022 03:04:54 -0700 (PDT) Received: from vm.nix.is (vm.nix.is. [2a01:4f8:120:2468::2]) by smtp.gmail.com with ESMTPSA id o4-20020a05600002c400b0020d0cdbf7eesm1649452wry.111.2022.05.25.03.04.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 25 May 2022 03:04:53 -0700 (PDT) From: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= To: git@vger.kernel.org Cc: Junio C Hamano , Taylor Blau , =?utf-8?q?SZEDER_G=C3=A1bor?= , =?utf-8?b?xJBvw6Bu?= =?utf-8?b?IFRy4bqnbiBDw7RuZyBEYW5o?= , =?utf-8?q?Car?= =?utf-8?q?lo_Marcelo_Arenas_Bel=C3=B3n?= , Johannes Schindelin , Victoria Dye , Matheus Tavares , Lars Schneider , Eric Sunshine , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBC?= =?utf-8?b?amFybWFzb24=?= Subject: [PATCH v6 12/29] CI: export variables via a wrapper Date: Wed, 25 May 2022 12:03:51 +0200 Message-Id: X-Mailer: git-send-email 2.36.1.1045.gf356b5617dd In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Add and use a "setenv" wrapper to export the variables set in ci/lib.sh. For now this merely adds debugging information about what variables are set, but in a subsequent commit we'll expand this to set variables in $GITHUB_ENV instead of the environment, and thus carry variable settings between steps in a job. We'll also eventually be interested in what type of variable we're setting, and when it needs to be used, because we'll prominently expose what variables we set in what CI "step", so having only the ones we actually use will improve readability. To that end call the "setenv" function with optional "--build", "--test" and "--all" parameters to indicate whether they're needed for building, testing or both. We currently ignore the "varname" parameter, but will make use of it in a subsequent commit. Adding it here reduces the needless churn, as we won't need to change all these lines again for that subsequent change. Signed-off-by: Ævar Arnfjörð Bjarmason --- ci/lib.sh | 51 ++++++++++++++++++++++++++++++--------- ci/run-build-and-tests.sh | 34 +++++++++++++------------- 2 files changed, 57 insertions(+), 28 deletions(-) diff --git a/ci/lib.sh b/ci/lib.sh index 1b5b805c5ca..71e6d6e08e7 100755 --- a/ci/lib.sh +++ b/ci/lib.sh @@ -13,6 +13,35 @@ then exit 1 fi +# Helper functions +setenv () { + while test $# != 0 + do + case "$1" in + --build) + ;; + --test) + ;; + --all) + ;; + -*) + echo "BUG: bad setenv() option '$1'" >&2 + exit 1 + ;; + *) + break + ;; + esac + shift + done + + key=$1 + val=$2 + shift 2 + + eval "export $key=\"$val\"" +} + check_unignored_build_artifacts () { ! git ls-files --other --exclude-standard --error-unmatch \ @@ -24,7 +53,7 @@ check_unignored_build_artifacts () } # GitHub Action doesn't set TERM, which is required by tput -export TERM=${TERM:-dumb} +setenv TERM ${TERM:-dumb} # How many jobs to run in parallel? NPROC=10 @@ -37,21 +66,21 @@ then CI_TYPE=github-actions CC="${CC_PACKAGE:-${CC:-gcc}}" - export GIT_PROVE_OPTS="--timer --jobs $NPROC" + setenv --test GIT_PROVE_OPTS "--timer --jobs $NPROC" GIT_TEST_OPTS="--verbose-log -x" test Windows != "$RUNNER_OS" || GIT_TEST_OPTS="--no-chain-lint --no-bin-wrappers $GIT_TEST_OPTS" - export GIT_TEST_OPTS + setenv --test GIT_TEST_OPTS "$GIT_TEST_OPTS" else echo "Could not identify CI type" >&2 env >&2 exit 1 fi -export DEVELOPER=1 -export DEFAULT_TEST_TARGET=prove -export GIT_TEST_CLONE_2GB=true -export SKIP_DASHED_BUILT_INS=YesPlease +setenv --build DEVELOPER 1 +setenv --test DEFAULT_TEST_TARGET prove +setenv --test GIT_TEST_CLONE_2GB true +setenv --build SKIP_DASHED_BUILT_INS YesPlease case "$runs_on_pool" in ubuntu-latest) @@ -67,7 +96,7 @@ ubuntu-latest) MAKEFLAGS="$MAKEFLAGS PYTHON_PATH=/usr/bin/python2" fi - export GIT_TEST_HTTPD=true + setenv --test GIT_TEST_HTTPD true ;; macos-latest) if [ "$jobname" = osx-gcc ] @@ -90,9 +119,9 @@ linux-musl) MAKEFLAGS="$MAKEFLAGS GIT_TEST_UTF8_LOCALE=C.UTF-8" ;; linux-leaks) - export SANITIZE=leak - export GIT_TEST_PASSING_SANITIZE_LEAK=true + setenv --build SANITIZE leak + setenv --test GIT_TEST_PASSING_SANITIZE_LEAK true ;; esac -export MAKEFLAGS="$MAKEFLAGS CC=${CC:-cc}" +setenv --all MAKEFLAGS "$MAKEFLAGS CC=${CC:-cc}" diff --git a/ci/run-build-and-tests.sh b/ci/run-build-and-tests.sh index e5a21985b44..35d45a9373e 100755 --- a/ci/run-build-and-tests.sh +++ b/ci/run-build-and-tests.sh @@ -9,33 +9,33 @@ export MAKE_TARGETS="all test" case "$jobname" in linux-gcc) - export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main + setenv --test GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME main ;; linux-TEST-vars) - export GIT_TEST_SPLIT_INDEX=yes - export GIT_TEST_MERGE_ALGORITHM=recursive - export GIT_TEST_FULL_IN_PACK_ARRAY=true - export GIT_TEST_OE_SIZE=10 - export GIT_TEST_OE_DELTA_SIZE=5 - export GIT_TEST_COMMIT_GRAPH=1 - export GIT_TEST_COMMIT_GRAPH_CHANGED_PATHS=1 - export GIT_TEST_MULTI_PACK_INDEX=1 - export GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=1 - export GIT_TEST_ADD_I_USE_BUILTIN=1 - export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=master - export GIT_TEST_WRITE_REV_INDEX=1 - export GIT_TEST_CHECKOUT_WORKERS=2 + setenv --test GIT_TEST_SPLIT_INDEX yes + setenv --test GIT_TEST_MERGE_ALGORITHM recursive + setenv --test GIT_TEST_FULL_IN_PACK_ARRAY true + setenv --test GIT_TEST_OE_SIZE 10 + setenv --test GIT_TEST_OE_DELTA_SIZE 5 + setenv --test GIT_TEST_COMMIT_GRAPH 1 + setenv --test GIT_TEST_COMMIT_GRAPH_CHANGED_PATHS 1 + setenv --test GIT_TEST_MULTI_PACK_INDEX 1 + setenv --test GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP 1 + setenv --test GIT_TEST_ADD_I_USE_BUILTIN 1 + setenv --test GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME master + setenv --test GIT_TEST_WRITE_REV_INDEX 1 + setenv --test GIT_TEST_CHECKOUT_WORKERS 2 ;; linux-clang) - export GIT_TEST_DEFAULT_HASH=sha1 + setenv --test GIT_TEST_DEFAULT_HASH sha1 ;; linux-sha256) - export GIT_TEST_DEFAULT_HASH=sha256 + setenv --test GIT_TEST_DEFAULT_HASH sha256 ;; pedantic) # Don't run the tests; we only care about whether Git can be # built. - export DEVOPTS=pedantic + setenv --build DEVOPTS pedantic export MAKE_TARGETS=all ;; esac From patchwork Wed May 25 10:03:52 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= X-Patchwork-Id: 12860936 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 28963C433F5 for ; Wed, 25 May 2022 10:07:17 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240227AbiEYKHQ (ORCPT ); Wed, 25 May 2022 06:07:16 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58836 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S241590AbiEYKE7 (ORCPT ); Wed, 25 May 2022 06:04:59 -0400 Received: from mail-wm1-x336.google.com (mail-wm1-x336.google.com [IPv6:2a00:1450:4864:20::336]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 61A3E6D970 for ; Wed, 25 May 2022 03:04:57 -0700 (PDT) Received: by mail-wm1-x336.google.com with SMTP id r82-20020a1c4455000000b003975cedb52bso221385wma.1 for ; Wed, 25 May 2022 03:04:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=CmjQBAJLV1bES/9ZKdj/+IQ8wDI4oIVtY7eQOBqQbuI=; b=eNo0E5UKhd/kHcmxsyvLt7fiHtMeApvVt1RQOfKfLzBSdHBd0V/k1O5IMrUfcZ3rde NCDlXkiV7X/mM2HazW97A/Q33bGn8H9BQdVGDq90/0l2DQEu+cdaDvFdx8+wJIfKzoRl 9agP/rswxSsRnlIVD3aDWgSdj222vG4mYJxw2ukXBZCIY4zCntDpigYm2wXVomShVDcc YN8PPxVrzTPlpEq7Myi9YhSt3/zVi3wxTAVZvOVnkOHFuR5cpZVytSx4vyukyzZ6JUMu UH5DYlpBGp0gORt3j02VWMUcGO9FPSUJrJ7BBF5U9x75ZVip23GH9nkuk9O+5/FauEmO /FwQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=CmjQBAJLV1bES/9ZKdj/+IQ8wDI4oIVtY7eQOBqQbuI=; b=VapGSOjw2JmyNwTTDoMSROxPq9sNpr3nlfMqMRFnHmnCTgRAHZcfM1aQQ6kYiVEuje xlx5ndb+3SJ1G2UnsDcZ1xqyuq27HBUJE5irLzUJqqMERSk87BfknUTgx9Mqfh1v18TD WWzotOtOUGS5yVL8ts2NYJuvyvYvKIDWtTNJ8aADpVZ3sCTL/S5hZEEOTcW9r/nk4rsw mRvQvqCmtd48bmNZ/BTEcG4J0s9rkRaJzDNHYC8EW9D94CM0SHuxl34J3yOU4LmGRiIE uK7vdmrauFNvqtr4qzdF1ifcFkIKtkGt5o6jIqmLvjExez5Hj//l5DPtHI/9zpbcxJwo ERuA== X-Gm-Message-State: AOAM531J7sQAziDy8gsILjFecwbBP0+HP+tSANAbNHHnAaXlRNDakq9u x5FRqZR7LNxXoYq09kj1VIj+n0nRAmLFeQ== X-Google-Smtp-Source: ABdhPJwjd91w4pY5CMriHO10mtADCqoVcqDszKQq54rJAZBQEFZL6+ePgqCH3eTFXsp8h6P3uSoNog== X-Received: by 2002:a7b:cb48:0:b0:394:b64:8b88 with SMTP id v8-20020a7bcb48000000b003940b648b88mr7378384wmj.183.1653473095502; Wed, 25 May 2022 03:04:55 -0700 (PDT) Received: from vm.nix.is (vm.nix.is. [2a01:4f8:120:2468::2]) by smtp.gmail.com with ESMTPSA id o4-20020a05600002c400b0020d0cdbf7eesm1649452wry.111.2022.05.25.03.04.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 25 May 2022 03:04:54 -0700 (PDT) From: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= To: git@vger.kernel.org Cc: Junio C Hamano , Taylor Blau , =?utf-8?q?SZEDER_G=C3=A1bor?= , =?utf-8?b?xJBvw6Bu?= =?utf-8?b?IFRy4bqnbiBDw7RuZyBEYW5o?= , =?utf-8?q?Car?= =?utf-8?q?lo_Marcelo_Arenas_Bel=C3=B3n?= , Johannes Schindelin , Victoria Dye , Matheus Tavares , Lars Schneider , Eric Sunshine , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBC?= =?utf-8?b?amFybWFzb24=?= Subject: [PATCH v6 13/29] CI: remove "run-build-and-tests.sh", run "make [test]" directly Date: Wed, 25 May 2022 12:03:52 +0200 Message-Id: X-Mailer: git-send-email 2.36.1.1045.gf356b5617dd In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Remove the already thin "ci/run-build-and-tests.sh" wrapper and instead make the CI run "make" or "make test" directly. By doing this we'll be able to easily see at a glance whether our failure was in the compilation or testing, whether that's via human eyes or improve machine readability. We also need to run our new "ci/check-unignored-build-artifacts.sh" on success() in the CI now, just like we already had a step conditional on failure() running ci/print-test-failures.sh. The reason we used a "ci/run-build-and-tests.sh" wrapper in the first place had to do with Travis CI-specific constraints that no longer apply to us, as the Travis CI support has been removed. Instead we can configure the CI in an earlier step by running "ci/lib.sh", which under GitHub CI will write the environment variables we need to the "$GITHUB_ENV" file. We'll then have access to them in subsequent steps, and crucially those variables will be prominently visible at the start of each step via an expandable drop-down in the UI.drop-do. I.e. this changes the CI run from a top-down flow like (pseudocode): - job: - step1: - use ci/lib.sh to set env vars - run a script like ci/run-build-and-tests.sh - step2: - if: failure() - use ci/lib.sh to set env vars - run ci/print-test-failures.sh To: - job: - step1: - set variables in $GITHUB_ENV using ci/lib.sh - step2: - make - step3: - make test - step4: - if: failure() - run ci/print-test-failures.sh - step5: - if: success() - run ci/check-unignored-build-artifacts.sh There is a proposal[2] to get some of the benefits of this approach by not re-arranging our variable setup in this way, but to instead use the GitHub CI grouping syntax to focus on the relevant parts of "make" or "make test" when we have failures. Doing it this way makes for better looking GitHub CI UI, and lays much better ground work for our CI going forward. Because: * The CI logic will be more portable to a future CI system, since a common feature of them is to run various commands in sequence, but a future system won't necessarily support the GitHub-specifics syntax of "grouping" output within a "step". Even if those systems don't support a "$GITHUB_ENV" emulating will be much easier than to deal with some CI-specific grouping syntax. * At the start of every step the GitHub CI presents an expandable list of environment variables from "$GITHUB_ENV". We'll now see exactly what variables affected that step (although we currently overshoot that a bit, and always define all variables). * CI failures will be easier to reproduce locally, as this makes the relevant ci/* scripts something that sets up our environment, but leaves "make" and "make test" working as they do locally. To reproduce a run the user only needs to set the variables discussed in the drop-down above, either manually or by running "ci/lib.sh". * The output will be less verbose. The "ci/lib.sh" script uses "set -x", and before this e.g. "ci/static-analysis.sh" would start with 40 lines of trace output, culminating in using "export" to export the relevant environment variables. Now that verbosity is in the earlier "ci/lib.sh" step, and not in any subsequent one. The "make" targets then start out with the relevant non-trace output right away. * If we do want to use the grouping syntax within a "step" it'll now be easier to do so. It doesn't support nesting, so we'd have to make a choice between using it for e.g. "make" v.s. "make test", or individual test failures. See "sadly" in [3]. 1. https://lore.kernel.org/git/211120.86k0h30zuw.gmgdl@evledraar.gmail.com/ 2. https://lore.kernel.org/git/pull.1117.git.1643050574.gitgitgadget@gmail.com/ 3. https://lore.kernel.org/git/9333ba781b8240f704e739b00d274f8c3d887e39.1643050574.git.gitgitgadget@gmail.com/ Signed-off-by: Ævar Arnfjörð Bjarmason --- .github/workflows/main.yml | 14 ++++++-- ci/check-unignored-build-artifacts.sh | 8 +++++ ci/install-dependencies.sh | 4 +++ ci/lib.sh | 36 +++++++++++++++++++- ci/make-test-artifacts.sh | 2 -- ci/run-build-and-tests.sh | 47 --------------------------- ci/run-test-slice.sh | 2 -- ci/test-documentation.sh | 1 - 8 files changed, 58 insertions(+), 56 deletions(-) create mode 100755 ci/check-unignored-build-artifacts.sh delete mode 100755 ci/run-build-and-tests.sh diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml index ad752010102..57b2db06923 100644 --- a/.github/workflows/main.yml +++ b/.github/workflows/main.yml @@ -268,7 +268,10 @@ jobs: steps: - uses: actions/checkout@v2 - run: ci/install-dependencies.sh - - run: ci/run-build-and-tests.sh + - run: ci/lib.sh + - run: make + - run: make test + if: success() - run: ci/print-test-failures.sh if: failure() - name: Upload failed tests' directories @@ -292,6 +295,7 @@ jobs: image: daald/ubuntu32:xenial - jobname: pedantic image: fedora + skip-tests: yes env: jobname: ${{matrix.vector.jobname}} runs-on: ubuntu-latest @@ -299,9 +303,12 @@ jobs: steps: - uses: actions/checkout@v1 - run: ci/install-docker-dependencies.sh - - run: ci/run-build-and-tests.sh + - run: ci/lib.sh + - run: make + - run: make test + if: success() && matrix.vector.skip-tests != 'yes' - run: ci/print-test-failures.sh - if: failure() + if: failure() && matrix.vector.skip-tests != 'yes' - name: Upload failed tests' directories if: failure() && env.FAILED_TEST_ARTIFACTS != '' uses: actions/upload-artifact@v1 @@ -317,6 +324,7 @@ jobs: steps: - uses: actions/checkout@v2 - run: ci/install-dependencies.sh + - run: ci/lib.sh - run: make ci-static-analysis sparse: needs: ci-config diff --git a/ci/check-unignored-build-artifacts.sh b/ci/check-unignored-build-artifacts.sh new file mode 100755 index 00000000000..56d04b0db9a --- /dev/null +++ b/ci/check-unignored-build-artifacts.sh @@ -0,0 +1,8 @@ +#!/bin/sh +# +# Check whether the build created anything not in our .gitignore +# + +. ${0%/*}/lib.sh + +check_unignored_build_artifacts diff --git a/ci/install-dependencies.sh b/ci/install-dependencies.sh index 41a9185bb1e..5d50604e724 100755 --- a/ci/install-dependencies.sh +++ b/ci/install-dependencies.sh @@ -22,6 +22,10 @@ ubuntu-latest) P4_PATH="$HOME/custom/p4" GIT_LFS_PATH="$HOME/custom/git-lfs" export PATH="$GIT_LFS_PATH:$P4_PATH:$PATH" + if test -n "$GITHUB_PATH" + then + echo "$PATH" >>"$GITHUB_PATH" + fi P4WHENCE=https://cdist2.perforce.com/perforce/r$LINUX_P4_VERSION LFSWHENCE=https://github.com/github/git-lfs/releases/download/v$LINUX_GIT_LFS_VERSION diff --git a/ci/lib.sh b/ci/lib.sh index 71e6d6e08e7..d18ee1a91e7 100755 --- a/ci/lib.sh +++ b/ci/lib.sh @@ -1,3 +1,5 @@ +#!/bin/sh + # Library of functions shared by all CI scripts # Set 'exit on error' for all CI scripts to let the caller know that @@ -39,7 +41,10 @@ setenv () { val=$2 shift 2 - eval "export $key=\"$val\"" + if test -n "$GITHUB_ENV" + then + echo "$key=$val" >>"$GITHUB_ENV" + fi } check_unignored_build_artifacts () @@ -109,6 +114,35 @@ macos-latest) esac case "$jobname" in +linux-gcc) + setenv --test GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME main + ;; +linux-TEST-vars) + setenv --test GIT_TEST_SPLIT_INDEX yes + setenv --test GIT_TEST_MERGE_ALGORITHM recursive + setenv --test GIT_TEST_FULL_IN_PACK_ARRAY true + setenv --test GIT_TEST_OE_SIZE 10 + setenv --test GIT_TEST_OE_DELTA_SIZE 5 + setenv --test GIT_TEST_COMMIT_GRAPH 1 + setenv --test GIT_TEST_COMMIT_GRAPH_CHANGED_PATHS 1 + setenv --test GIT_TEST_MULTI_PACK_INDEX 1 + setenv --test GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP 1 + setenv --test GIT_TEST_ADD_I_USE_BUILTIN 1 + setenv --test GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME master + setenv --test GIT_TEST_WRITE_REV_INDEX 1 + setenv --test GIT_TEST_CHECKOUT_WORKERS 2 + ;; +linux-clang) + setenv --test GIT_TEST_DEFAULT_HASH sha1 + ;; +linux-sha256) + setenv --test GIT_TEST_DEFAULT_HASH sha256 + ;; +pedantic) + # Don't run the tests; we only care about whether Git can be + # built. + setenv --build DEVOPTS pedantic + ;; linux32) CC=gcc ;; diff --git a/ci/make-test-artifacts.sh b/ci/make-test-artifacts.sh index 646967481f6..45298562982 100755 --- a/ci/make-test-artifacts.sh +++ b/ci/make-test-artifacts.sh @@ -8,5 +8,3 @@ mkdir -p "$1" # in case ci/lib.sh decides to quit early . ${0%/*}/lib.sh make artifacts-tar ARTIFACTS_DIRECTORY="$1" - -check_unignored_build_artifacts diff --git a/ci/run-build-and-tests.sh b/ci/run-build-and-tests.sh deleted file mode 100755 index 35d45a9373e..00000000000 --- a/ci/run-build-and-tests.sh +++ /dev/null @@ -1,47 +0,0 @@ -#!/bin/sh -# -# Build and test Git -# - -. ${0%/*}/lib.sh - -export MAKE_TARGETS="all test" - -case "$jobname" in -linux-gcc) - setenv --test GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME main - ;; -linux-TEST-vars) - setenv --test GIT_TEST_SPLIT_INDEX yes - setenv --test GIT_TEST_MERGE_ALGORITHM recursive - setenv --test GIT_TEST_FULL_IN_PACK_ARRAY true - setenv --test GIT_TEST_OE_SIZE 10 - setenv --test GIT_TEST_OE_DELTA_SIZE 5 - setenv --test GIT_TEST_COMMIT_GRAPH 1 - setenv --test GIT_TEST_COMMIT_GRAPH_CHANGED_PATHS 1 - setenv --test GIT_TEST_MULTI_PACK_INDEX 1 - setenv --test GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP 1 - setenv --test GIT_TEST_ADD_I_USE_BUILTIN 1 - setenv --test GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME master - setenv --test GIT_TEST_WRITE_REV_INDEX 1 - setenv --test GIT_TEST_CHECKOUT_WORKERS 2 - ;; -linux-clang) - setenv --test GIT_TEST_DEFAULT_HASH sha1 - ;; -linux-sha256) - setenv --test GIT_TEST_DEFAULT_HASH sha256 - ;; -pedantic) - # Don't run the tests; we only care about whether Git can be - # built. - setenv --build DEVOPTS pedantic - export MAKE_TARGETS=all - ;; -esac - -# Any new "test" targets should not go after this "make", but should -# adjust $MAKE_TARGETS. Otherwise compilation-only targets above will -# start running tests. -make $MAKE_TARGETS -check_unignored_build_artifacts diff --git a/ci/run-test-slice.sh b/ci/run-test-slice.sh index b9a682b4bcd..70326961454 100755 --- a/ci/run-test-slice.sh +++ b/ci/run-test-slice.sh @@ -8,5 +8,3 @@ make --quiet -C t T="$(cd t && ./helper/test-tool path-utils slice-tests "$1" "$2" t[0-9]*.sh | tr '\n' ' ')" - -check_unignored_build_artifacts diff --git a/ci/test-documentation.sh b/ci/test-documentation.sh index 41e2b126311..6b5cce03bd7 100755 --- a/ci/test-documentation.sh +++ b/ci/test-documentation.sh @@ -39,4 +39,3 @@ test -s Documentation/git.html grep ' X-Patchwork-Id: 12860934 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 442FFC433EF for ; Wed, 25 May 2022 10:07:16 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238998AbiEYKHN (ORCPT ); Wed, 25 May 2022 06:07:13 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58972 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S241630AbiEYKFA (ORCPT ); Wed, 25 May 2022 06:05:00 -0400 Received: from mail-wm1-x32b.google.com (mail-wm1-x32b.google.com [IPv6:2a00:1450:4864:20::32b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 859746D94A for ; Wed, 25 May 2022 03:04:58 -0700 (PDT) Received: by mail-wm1-x32b.google.com with SMTP id r82-20020a1c4455000000b003975cedb52bso221422wma.1 for ; Wed, 25 May 2022 03:04:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=UtFDoWsa11wQreI4soCiOzKYIe9CS7KqDO77PyDrAMQ=; b=MU8clROUv1vgfnENzEkZ2AxUQqdbppaYxLy2/0lywcdXY8krAR5qYlMIrOElM+DlT1 6QzeKp6wNFQom1UJIAw05gDhpybVQzcA2HXcQBm4mYJmeKsrPvRcWalxylyEABY3NN4p QoSSjVWV60A9FauLHsKFUjH//uuma7WH/EjAlUj66WS5KKxkRn1NALMpPxFxhu+ao5wr LZTrgy6W+gpkF0g5kdod+jGmz9GzWTdIXjK5BTsI/X9Gp57Z6LhPeWrhSKgr4w69EJaI 8VIkGIetA0ozzbQsWuzrLie5Ny6hg7xSeeCSBgz37A+1iAhv0JAsl7R801gHGKi/b6Iy 15oQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=UtFDoWsa11wQreI4soCiOzKYIe9CS7KqDO77PyDrAMQ=; b=WEskXEO0xtr3vz5FEBNlV2GXNg9SvaZZZ/BKoYMC70QNdM7xs4jI9pSIULrNGRvAoU nQAEeCNunneJHW8welmr1T67BT+vFEOrlV3fyJhU7h5n4HqECdeR3vnw9e31EO5gQn+I Xz5hOIRDmHdMh7fAopSD8J6CzoMtOPStjNnTgtxhWASNg8riZ2MIAv/SEQwwUDDpjXQa pnRhzSywSRtllhrok97/DWut14m2yWLRgiPfmVFKQldan+jaqRxVbYu5Q5uuKsZm+QPm wpLlx+DLMtkgb56G10bgUDJ8wJ8v2BY3RNfsoV3uiBzgVVo3vRWjXVYuGtZyI2Z+C65s YXJg== X-Gm-Message-State: AOAM5330TFn0OirmizFuBdA/JST2zXHF1Zg1pO6QZCyu/RUHjnwbHKlJ jvo+vuEWD+nbQNb4g1WNnCjRUVECkHu/Hg== X-Google-Smtp-Source: ABdhPJxFmWa76TMJfNeYt3pJmMB0IfyJ4bBj5BFhO/+dI0pTi77qW8X+T9CyfkypGoPH+v7EyimI1g== X-Received: by 2002:a05:600c:348d:b0:397:55ca:78e3 with SMTP id a13-20020a05600c348d00b0039755ca78e3mr7354572wmq.167.1653473096755; Wed, 25 May 2022 03:04:56 -0700 (PDT) Received: from vm.nix.is (vm.nix.is. [2a01:4f8:120:2468::2]) by smtp.gmail.com with ESMTPSA id o4-20020a05600002c400b0020d0cdbf7eesm1649452wry.111.2022.05.25.03.04.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 25 May 2022 03:04:55 -0700 (PDT) From: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= To: git@vger.kernel.org Cc: Junio C Hamano , Taylor Blau , =?utf-8?q?SZEDER_G=C3=A1bor?= , =?utf-8?b?xJBvw6Bu?= =?utf-8?b?IFRy4bqnbiBDw7RuZyBEYW5o?= , =?utf-8?q?Car?= =?utf-8?q?lo_Marcelo_Arenas_Bel=C3=B3n?= , Johannes Schindelin , Victoria Dye , Matheus Tavares , Lars Schneider , Eric Sunshine , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBC?= =?utf-8?b?amFybWFzb24=?= Subject: [PATCH v6 14/29] CI: make ci/{lib,install-dependencies}.sh POSIX-compatible Date: Wed, 25 May 2022 12:03:53 +0200 Message-Id: X-Mailer: git-send-email 2.36.1.1045.gf356b5617dd In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Change the unportable constructs in the "ci/install-dependencies.sh" scripts to use POSIX replacements, and have it run under "/bin/sh" instead of "/usr/bin/env bash". We needed to: * Replace pushd/popd with a sub-shell. * Change "mkdir --parents" to "mkdir -p". * Use POSIX "export" syntax. This is in preparation for combining this script and "ci/install-docker-dependencies.sh", which doesn't have access to "bash". While we're at it change "if [" to "if test" in ci/lib.sh. This isn't strictly speaking a case of POSIX-incompatibility, but that syntax was being used there because this code was using idiomatic bash constructs. Let's have it use our usual style of "if test" instead. Signed-off-by: Ævar Arnfjörð Bjarmason --- ci/install-dependencies.sh | 27 +++++++++++++++++---------- ci/lib.sh | 4 ++-- 2 files changed, 19 insertions(+), 12 deletions(-) diff --git a/ci/install-dependencies.sh b/ci/install-dependencies.sh index 5d50604e724..3e05d38d4ab 100755 --- a/ci/install-dependencies.sh +++ b/ci/install-dependencies.sh @@ -1,4 +1,4 @@ -#!/usr/bin/env bash +#!/bin/sh # # Install dependencies required to build and test Git on Linux and macOS # @@ -21,7 +21,8 @@ ubuntu-latest) P4_PATH="$HOME/custom/p4" GIT_LFS_PATH="$HOME/custom/git-lfs" - export PATH="$GIT_LFS_PATH:$P4_PATH:$PATH" + PATH="$GIT_LFS_PATH:$P4_PATH:$PATH" + export PATH if test -n "$GITHUB_PATH" then echo "$PATH" >>"$GITHUB_PATH" @@ -33,22 +34,28 @@ ubuntu-latest) sudo apt-get -q update sudo apt-get -q -y install language-pack-is libsvn-perl apache2 \ $UBUNTU_COMMON_PKGS $CC_PACKAGE - mkdir --parents "$P4_PATH" - pushd "$P4_PATH" + mkdir -p "$P4_PATH" + ( + cd "$P4_PATH" wget --quiet "$P4WHENCE/bin.linux26x86_64/p4d" wget --quiet "$P4WHENCE/bin.linux26x86_64/p4" chmod u+x p4d chmod u+x p4 - popd - mkdir --parents "$GIT_LFS_PATH" - pushd "$GIT_LFS_PATH" + ) + mkdir -p "$GIT_LFS_PATH" + ( + cd "$GIT_LFS_PATH" wget --quiet "$LFSWHENCE/git-lfs-linux-amd64-$LINUX_GIT_LFS_VERSION.tar.gz" tar --extract --gunzip --file "git-lfs-linux-amd64-$LINUX_GIT_LFS_VERSION.tar.gz" cp git-lfs-$LINUX_GIT_LFS_VERSION/git-lfs . - popd + ) ;; macos-latest) - export HOMEBREW_NO_AUTO_UPDATE=1 HOMEBREW_NO_INSTALL_CLEANUP=1 + HOMEBREW_NO_AUTO_UPDATE=1 + export HOMEBREW_NO_AUTO_UPDATE + HOMEBREW_NO_INSTALL_CLEANUP=1 + export HOMEBREW_NO_INSTALL_CLEANUP + # Uncomment this if you want to run perf tests: # brew install gnu-time brew link --force gettext @@ -64,7 +71,7 @@ macos-latest) if test -n "$CC_PACKAGE" then - BREW_PACKAGE=${CC_PACKAGE/-/@} + BREW_PACKAGE=$(echo $CC_PACKAGE | tr '-' '@') brew install "$BREW_PACKAGE" brew link "$BREW_PACKAGE" fi diff --git a/ci/lib.sh b/ci/lib.sh index d18ee1a91e7..8df133b840b 100755 --- a/ci/lib.sh +++ b/ci/lib.sh @@ -94,7 +94,7 @@ ubuntu-latest) break fi - if [ "$jobname" = linux-gcc ] + if test "$jobname" = linux-gcc then MAKEFLAGS="$MAKEFLAGS PYTHON_PATH=/usr/bin/python3" else @@ -104,7 +104,7 @@ ubuntu-latest) setenv --test GIT_TEST_HTTPD true ;; macos-latest) - if [ "$jobname" = osx-gcc ] + if test "$jobname" = osx-gcc then MAKEFLAGS="$MAKEFLAGS PYTHON_PATH=$(which python3)" else From patchwork Wed May 25 10:03:54 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= X-Patchwork-Id: 12860931 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4B1F4C433EF for ; Wed, 25 May 2022 10:07:08 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S241582AbiEYKHG (ORCPT ); Wed, 25 May 2022 06:07:06 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58618 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S241675AbiEYKFC (ORCPT ); Wed, 25 May 2022 06:05:02 -0400 Received: from mail-wm1-x332.google.com (mail-wm1-x332.google.com [IPv6:2a00:1450:4864:20::332]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 860916E8FB for ; Wed, 25 May 2022 03:04:58 -0700 (PDT) Received: by mail-wm1-x332.google.com with SMTP id n124-20020a1c2782000000b003972dfca96cso3019844wmn.4 for ; Wed, 25 May 2022 03:04:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=0/F7LWIF1Lp7TZ18EzMuiXw18x5YsnrFO3fUgX76E8Q=; b=MTG/KvrOWdTM3WXSACGQp93W/hclhxSVmPvnkpWvUVUmMczzRm5smA1hREQhd7eA5D irrNur0z+DfKNWDx8pze8B48Ri6gjSmL7YbAiXbuhZxKw00sEEFjCEzCWLU3ewur+Ntd VbktV0/iZ6An+pKtjduI/lDPkUe3CxaSAM9+BEcPfynlE7I6e/a8nn/Di81dFfPhMgOy D1M8SeZOP+BU88lP+MmCuiCyq9pYeYD+06wAaVNhbcpctkdJZhtZ2hCSU0H1I/aMTEWT RtBCL7epnjMB8ewpT5LbJ5JKdWo7oEmkFTY79jg/qWfK6gkByYflsPdUiivsZSmpVa32 b4dA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=0/F7LWIF1Lp7TZ18EzMuiXw18x5YsnrFO3fUgX76E8Q=; b=EJj1kblSGttVa8ObhFUWMpmF+qw1ZzA9Of8pfG18T2d1oGc7CwgpIoTGYuAyjvTTAU DV6CGKLjErMTr7F1ipVEXmZhffkjJU5EvdNFIZS6xgJqb7TyWcPEkW1cezDl96GEZoFn V3nHTFfdJllD9OfuvLPvqKUQNZPSp5tXT7Tu/w3uJSI+D/iLIy0A3ZyhR2ujDGJ20K9h ib0lJNsItBRBk7zx9rDqygkMLDknztC5yCEA0ieBmvfZsr0O0yy8oldFhcwcvHx5UI/I siLUsVRYSKbGf4H9uK9nJEV33mVunmxTeLTHuykVW5gIdc/ZiBBx4rcQqPh1szEBiHQP cBYw== X-Gm-Message-State: AOAM5308HRai6RWj0rnpUes7VlpLlPBr5p31h2R8hw9XDcqk95ZO7/Gu jOWUZCoKGn1BxsnWD8Tjcj8VYAgDODu98Q== X-Google-Smtp-Source: ABdhPJzdoJuVANgGbFk8cRiStrPNrDaTrbomuVZgBW1E5fI4/iLlrWZdKteMK5rbi9igZ/WogGKA8g== X-Received: by 2002:a05:600c:3b02:b0:397:5cfb:b849 with SMTP id m2-20020a05600c3b0200b003975cfbb849mr6142877wms.183.1653473097884; Wed, 25 May 2022 03:04:57 -0700 (PDT) Received: from vm.nix.is (vm.nix.is. [2a01:4f8:120:2468::2]) by smtp.gmail.com with ESMTPSA id o4-20020a05600002c400b0020d0cdbf7eesm1649452wry.111.2022.05.25.03.04.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 25 May 2022 03:04:57 -0700 (PDT) From: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= To: git@vger.kernel.org Cc: Junio C Hamano , Taylor Blau , =?utf-8?q?SZEDER_G=C3=A1bor?= , =?utf-8?b?xJBvw6Bu?= =?utf-8?b?IFRy4bqnbiBDw7RuZyBEYW5o?= , =?utf-8?q?Car?= =?utf-8?q?lo_Marcelo_Arenas_Bel=C3=B3n?= , Johannes Schindelin , Victoria Dye , Matheus Tavares , Lars Schneider , Eric Sunshine , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBC?= =?utf-8?b?amFybWFzb24=?= Subject: [PATCH v6 15/29] CI: check ignored unignored build artifacts in "win[+VS] build" too Date: Wed, 25 May 2022 12:03:54 +0200 Message-Id: X-Mailer: git-send-email 2.36.1.1045.gf356b5617dd In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Use the newly created "ci/check-unignored-build-artifacts.sh" script in the "win build" and "win+VS build" steps too. These steps weren't doing this sanity check of untracked files against .gitignore before, but as the check passes for them let's keep doing that extra sanity check. Signed-off-by: Ævar Arnfjörð Bjarmason --- .github/workflows/main.yml | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml index 57b2db06923..c97fbd0f0b3 100644 --- a/.github/workflows/main.yml +++ b/.github/workflows/main.yml @@ -93,6 +93,9 @@ jobs: HOME: ${{runner.workspace}} NO_PERL: 1 run: . /etc/profile && ci/make-test-artifacts.sh artifacts + - run: ci/check-unignored-build-artifacts.sh + if: success() + shell: bash - name: zip up tracked files run: git archive -o artifacts/tracked.tar.gz HEAD - name: upload tracked files and build artifacts @@ -180,6 +183,9 @@ jobs: run: | mkdir -p artifacts && eval "$(make -n artifacts-tar INCLUDE_DLLS_IN_ARTIFACTS=YesPlease ARTIFACTS_DIRECTORY=artifacts NO_GETTEXT=YesPlease 2>&1 | grep ^tar)" + - run: ci/check-unignored-build-artifacts.sh + if: success() + shell: bash - name: zip up tracked files run: git archive -o artifacts/tracked.tar.gz HEAD - name: upload tracked files and build artifacts From patchwork Wed May 25 10:03:55 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= X-Patchwork-Id: 12860932 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0DEABC433FE for ; Wed, 25 May 2022 10:07:09 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S241657AbiEYKHI (ORCPT ); Wed, 25 May 2022 06:07:08 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58624 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S241800AbiEYKFD (ORCPT ); Wed, 25 May 2022 06:05:03 -0400 Received: from mail-wm1-x32d.google.com (mail-wm1-x32d.google.com [IPv6:2a00:1450:4864:20::32d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BC97E703C1 for ; Wed, 25 May 2022 03:05:00 -0700 (PDT) Received: by mail-wm1-x32d.google.com with SMTP id 206-20020a1c02d7000000b00397345f2c6fso365559wmc.4 for ; Wed, 25 May 2022 03:05:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=E+RORJLNVcruhzmElFkl0dR0vyiJrvvxEsEmgxzfHyc=; b=GzGehHhPyF5dqR5Wu7+Z22NyFxzgLZNmodDKhpiDgQebngwzaQO7zEDuI+PyvhXZLd LGZoygjNtEROaVX3dth42jLvMKQJx4zfmz6n5VjXd7A+l0rb/Ah9vT6jS3BFRHYoraeH Wrdl7ckFQHpXrp28OgT9h/80WrrD8ywDFAqzGkjoZB3sBDpFvQRKNmQvoeHB+5eDPXfC TKKnNdTQbLpmr6Aaw4t1CGBkntf61vaH9IvXGhtYMjB5ne64OXGB9cnnx9/Uv/BCzntV OEIUNtId7sjiqvvLQPRpp8x7jL7oZxWIGMUrtpkW+EzR6kW56RuGtXOLVaSKKttRGxxc 5m1Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=E+RORJLNVcruhzmElFkl0dR0vyiJrvvxEsEmgxzfHyc=; b=HENTKFGAUA27oRPmTfglt95Z/B1QMh4oB3MatmBpY7Nv4CmMElPkKCabpT/XjklONH UE944Ttu681+gwI3LzybcEFjMQbT357D/aG999zTcJZigT5V8Y1mxVd8IcoxsXlfhR5L 6Zj3+3V7/uMe8w3wXauWkLsgkB/AY0SiDuP4Dbu9JUyS+70x3WidNDy6Eyhbz6DMATCw 6hHR0DpcxQYdj7rarGcigi/7SwshnMGly0U9GWk1nADSnODAiMyq3YVR2PkER1fajaYt LHLTStVIxFoT4thDfKJoL72RYeLjxi+T6ZrbGsgfiZZeYMCU92jtqCYUpG1Gu6ZfwOQv It5Q== X-Gm-Message-State: AOAM532uvSERXA6szKwi1wUWEDrcA/XIZKjwo1FyQzPsQQp63iZ7eS7r s9MU0A0Hk0WqHm+ZEAZHsu9VhDHo40MF0w== X-Google-Smtp-Source: ABdhPJy30RbYhu7AyK1y13hfBLA+zwL91o7nBCcSJ8E3g//DUjqKYPib1dtrljlQkC12kc6qwa01Hw== X-Received: by 2002:a05:600c:4e41:b0:397:104:b1d3 with SMTP id e1-20020a05600c4e4100b003970104b1d3mr7339156wmq.101.1653473098820; Wed, 25 May 2022 03:04:58 -0700 (PDT) Received: from vm.nix.is (vm.nix.is. [2a01:4f8:120:2468::2]) by smtp.gmail.com with ESMTPSA id o4-20020a05600002c400b0020d0cdbf7eesm1649452wry.111.2022.05.25.03.04.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 25 May 2022 03:04:58 -0700 (PDT) From: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= To: git@vger.kernel.org Cc: Junio C Hamano , Taylor Blau , =?utf-8?q?SZEDER_G=C3=A1bor?= , =?utf-8?b?xJBvw6Bu?= =?utf-8?b?IFRy4bqnbiBDw7RuZyBEYW5o?= , =?utf-8?q?Car?= =?utf-8?q?lo_Marcelo_Arenas_Bel=C3=B3n?= , Johannes Schindelin , Victoria Dye , Matheus Tavares , Lars Schneider , Eric Sunshine , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBC?= =?utf-8?b?amFybWFzb24=?= Subject: [PATCH v6 16/29] CI: invoke "make artifacts-tar" directly in windows-build Date: Wed, 25 May 2022 12:03:55 +0200 Message-Id: X-Mailer: git-send-email 2.36.1.1045.gf356b5617dd In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Change the windows-build job to invoke the "make artifacts-tar" step directly, instead of calling a "ci/make-test-artifacts.sh" script. The script was needed because "ci/lib.sh" would set up various environment variables for us, but now we can instead use the "ci/lib.sh" in its script mode. The "mkdir -p" added in b819f1d2cec (ci: parallelize testing on Windows, 2019-01-29) isn't needed, the same commit added that "mkdir -p" to the "artifacts-tar" rule itself, so we can have "make" create the directory for us. This also has the benefit of making the "build" step less chatty, since it won't start with the verbose "set -x" output, that's now contained in the "ci/lib.sh" step. Signed-off-by: Ævar Arnfjörð Bjarmason --- .github/workflows/main.yml | 5 ++++- ci/make-test-artifacts.sh | 10 ---------- 2 files changed, 4 insertions(+), 11 deletions(-) delete mode 100755 ci/make-test-artifacts.sh diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml index c97fbd0f0b3..ca993afd098 100644 --- a/.github/workflows/main.yml +++ b/.github/workflows/main.yml @@ -87,12 +87,15 @@ jobs: steps: - uses: actions/checkout@v2 - uses: git-for-windows/setup-git-for-windows-sdk@v1 + - run: ci/lib.sh + shell: bash - name: build shell: bash env: HOME: ${{runner.workspace}} NO_PERL: 1 - run: . /etc/profile && ci/make-test-artifacts.sh artifacts + run: . /etc/profile && make artifacts-tar ARTIFACTS_DIRECTORY=artifacts + if: success() - run: ci/check-unignored-build-artifacts.sh if: success() shell: bash diff --git a/ci/make-test-artifacts.sh b/ci/make-test-artifacts.sh deleted file mode 100755 index 45298562982..00000000000 --- a/ci/make-test-artifacts.sh +++ /dev/null @@ -1,10 +0,0 @@ -#!/bin/sh -# -# Build Git and store artifacts for testing -# - -mkdir -p "$1" # in case ci/lib.sh decides to quit early - -. ${0%/*}/lib.sh - -make artifacts-tar ARTIFACTS_DIRECTORY="$1" From patchwork Wed May 25 10:03:56 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= X-Patchwork-Id: 12860930 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4264FC433F5 for ; Wed, 25 May 2022 10:07:06 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S241572AbiEYKHD (ORCPT ); Wed, 25 May 2022 06:07:03 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58622 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S241795AbiEYKFD (ORCPT ); Wed, 25 May 2022 06:05:03 -0400 Received: from mail-wr1-x42f.google.com (mail-wr1-x42f.google.com [IPv6:2a00:1450:4864:20::42f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BC49E6F4B8 for ; Wed, 25 May 2022 03:05:00 -0700 (PDT) Received: by mail-wr1-x42f.google.com with SMTP id e2so17781523wrc.1 for ; Wed, 25 May 2022 03:05:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=eXRS9d3MQoEDIcDCZxS4Wfp9X6wzHmVtwPn7KbEj5Lw=; b=cp8YPBZ8REb6GScGQchdrMp2w1XNtt0KRHvQAMADc/kLsfPv5kUY77MmUNWkyg4DQE wtiprjdtcf6Q1jJCEfkGCJO0sXQygLAQjq48dI96zX7/707uZ2ItzU2f34M10oR6oYIb xcfE49jiTJZIVSjGKI5xHwXnu9EHhOozdIikZOhS8nZXs6On6apz7CmgjnZay2gDg46A TZGNhbAtDzavVO3Q2efhWZfAozwg0SXty6fTWF/Xdpa42rUDOzP2GsiyfCpMWZNyjviB 8+vCCHlZ3sAHED8XH9gVJEqf75jsG845iWNyhyNM557ypJiErhXUj05mfqsK45EQndZb eiBg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=eXRS9d3MQoEDIcDCZxS4Wfp9X6wzHmVtwPn7KbEj5Lw=; b=oEoK3hC42xZ4blHCJqACUXB1jgH3hPTQDnqmWlc5Pvn0HlVEfHkwELP4qcxWi42pRL FgpMvMt0PpFEh01RwEQKCy5+47e9nsOsvIMQDf4uZMcI1rGsRo5XUtzhUPfqKXrsxtWI OeZne2dpmM7x09trS+m+lXzXrhNswbRs7Tjk8jRU1XG/sGH+ZroHZu4YucNTCLjt4NIZ iy/9q0iFYGTxkVIYpZmxB1ObToDj8vr4kBvB/K1gwamTmw5EEZXBYBtuZPBJwXNFwhzf OyOdINTgpKCYqGGlZhhRuGlRQNa6yPTnaGGtqGrCU8FQMBdsGy5xieNdJHDspzcOkeYP jCiw== X-Gm-Message-State: AOAM532gYEAo2Y8lhYw15ywQBkC/3iKGPd1bmofq3yYhG/MfeiwHiBPJ ypy7VHUt9kNe5gcIKAgvPXQn4RejeZDlnw== X-Google-Smtp-Source: ABdhPJynHlH7QgS7ZipkJ2BuzgJVXkPiA9XnmNZ59UUYilqZXSaHltTDKV/ZcJ+BDbNlZO4zbcrJiA== X-Received: by 2002:a5d:588d:0:b0:20f:f290:ed83 with SMTP id n13-20020a5d588d000000b0020ff290ed83mr5681073wrf.588.1653473099860; Wed, 25 May 2022 03:04:59 -0700 (PDT) Received: from vm.nix.is (vm.nix.is. [2a01:4f8:120:2468::2]) by smtp.gmail.com with ESMTPSA id o4-20020a05600002c400b0020d0cdbf7eesm1649452wry.111.2022.05.25.03.04.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 25 May 2022 03:04:59 -0700 (PDT) From: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= To: git@vger.kernel.org Cc: Junio C Hamano , Taylor Blau , =?utf-8?q?SZEDER_G=C3=A1bor?= , =?utf-8?b?xJBvw6Bu?= =?utf-8?b?IFRy4bqnbiBDw7RuZyBEYW5o?= , =?utf-8?q?Car?= =?utf-8?q?lo_Marcelo_Arenas_Bel=C3=B3n?= , Johannes Schindelin , Victoria Dye , Matheus Tavares , Lars Schneider , Eric Sunshine , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBC?= =?utf-8?b?amFybWFzb24=?= Subject: [PATCH v6 17/29] CI: split up and reduce "ci/test-documentation.sh" Date: Wed, 25 May 2022 12:03:56 +0200 Message-Id: X-Mailer: git-send-email 2.36.1.1045.gf356b5617dd In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Change the "ci/test-documentation.sh" script to run the bash-specific parts in as one command in the CI job itself, and to run the two "make doc" commands at the top-level. It'll now be obvious from the title of the step if we failed in the asciidoc or asciidoctor step. Since the "check_unignored_build_artifacts()" function is now only used in "ci/check-unignored-build-artifacts.sh" move that function there. The recipe for the job in ".github/workflows/main.yml" is now a bit verbose because it's effectively the same job twice, with a "make clean" in-between. It would be better for the verbosity to run it via a matrix as done in the alternate approach in [1] does, but then we'd sacrifice overall CPU time for the brevity. It might still be worth doing, but let's go for this simpler approach for now. 1. https://lore.kernel.org/git/patch-v2-6.6-7c423c8283d-20211120T030848Z-avarab@gmail.com/ Signed-off-by: Ævar Arnfjörð Bjarmason --- .github/workflows/main.yml | 16 +++++++++++++- ci/check-unignored-build-artifacts.sh | 10 +++++++++ ci/lib.sh | 10 --------- ci/test-documentation.sh | 31 +++++++++------------------ 4 files changed, 35 insertions(+), 32 deletions(-) diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml index ca993afd098..88eb2ef2dad 100644 --- a/.github/workflows/main.yml +++ b/.github/workflows/main.yml @@ -365,4 +365,18 @@ jobs: steps: - uses: actions/checkout@v2 - run: ci/install-dependencies.sh - - run: ci/test-documentation.sh + - run: ci/lib.sh + - run: make check-docs + - run: "make doc > >(tee stdout.log) 2> >(tee stderr.raw >&2)" + shell: bash + - run: ci/test-documentation.sh AsciiDoc + if: success() + - run: ci/check-unignored-build-artifacts.sh + if: success() + - run: make clean + - run: "make USE_ASCIIDOCTOR=1 doc > >(tee stdout.log) 2> >(tee stderr.raw >&2)" + shell: bash + - run: ci/test-documentation.sh Asciidoctor + if: success() + - run: ci/check-unignored-build-artifacts.sh + if: success() diff --git a/ci/check-unignored-build-artifacts.sh b/ci/check-unignored-build-artifacts.sh index 56d04b0db9a..0bc04f32804 100755 --- a/ci/check-unignored-build-artifacts.sh +++ b/ci/check-unignored-build-artifacts.sh @@ -5,4 +5,14 @@ . ${0%/*}/lib.sh +check_unignored_build_artifacts () +{ + ! git ls-files --other --exclude-standard --error-unmatch \ + -- ':/*' 2>/dev/null || + { + echo "$(tput setaf 1)error: found unignored build artifacts$(tput sgr0)" + false + } +} + check_unignored_build_artifacts diff --git a/ci/lib.sh b/ci/lib.sh index 8df133b840b..5141de85024 100755 --- a/ci/lib.sh +++ b/ci/lib.sh @@ -47,16 +47,6 @@ setenv () { fi } -check_unignored_build_artifacts () -{ - ! git ls-files --other --exclude-standard --error-unmatch \ - -- ':/*' 2>/dev/null || - { - echo "$(tput setaf 1)error: found unignored build artifacts$(tput sgr0)" - false - } -} - # GitHub Action doesn't set TERM, which is required by tput setenv TERM ${TERM:-dumb} diff --git a/ci/test-documentation.sh b/ci/test-documentation.sh index 6b5cce03bd7..b8a6a6f664e 100755 --- a/ci/test-documentation.sh +++ b/ci/test-documentation.sh @@ -1,10 +1,12 @@ -#!/usr/bin/env bash +#!/bin/sh # -# Perform sanity checks on documentation and build it. +# Perform sanity checks on "make doc" output and built documentation # . ${0%/*}/lib.sh +generator=$1 + filter_log () { sed -e '/^GIT_VERSION = /d' \ -e "/constant Gem::ConfigMap is deprecated/d" \ @@ -14,28 +16,15 @@ filter_log () { "$1" } -make check-docs - -# Build docs with AsciiDoc -make doc > >(tee stdout.log) 2> >(tee stderr.raw >&2) -cat stderr.raw -filter_log stderr.raw >stderr.log -test ! -s stderr.log -test -s Documentation/git.html -test -s Documentation/git.xml -test -s Documentation/git.1 -grep ' X-Patchwork-Id: 12860933 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 482BDC433F5 for ; Wed, 25 May 2022 10:07:13 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230132AbiEYKHL (ORCPT ); Wed, 25 May 2022 06:07:11 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58618 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S241902AbiEYKFE (ORCPT ); Wed, 25 May 2022 06:05:04 -0400 Received: from mail-wr1-x42e.google.com (mail-wr1-x42e.google.com [IPv6:2a00:1450:4864:20::42e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8B26D6E8FE for ; Wed, 25 May 2022 03:05:02 -0700 (PDT) Received: by mail-wr1-x42e.google.com with SMTP id s28so29254562wrb.7 for ; Wed, 25 May 2022 03:05:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=JpJIpnNMNCEnGQxKxI7826SyALKsnBXTM1S8FbHNPUM=; b=C1LqAZ0ospXLppVVshNdqwGCrFF1+ZqDJEERWwZy6IhcVpivB2WyLMURcVKPtfEfcu fDQhzd504sq1EWuNGyO8Y5GN5FigDlG/JGj7KeCFMmlTw2WvnPuP7aM5FsIZ49uL8It8 8AQ8R4URvyvPUkjX/UfvKPqNG/JOSE9d0l5S3cIOMV04TN8LyEUm9N3G7rDL6Y8bJyRC wcglaf6QF2/Bb/v9EnrS2piMcFRGIiCPubx9O5IokkYxegaPh/hlVKuBqz+Q/JWBKjj0 FFX2R9pQhRV3/0KwPlkUqPi8VIwUggHpvSXZEkYTuQYCRp1Vx+6vaHdooyZrB32BNrqP jRww== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=JpJIpnNMNCEnGQxKxI7826SyALKsnBXTM1S8FbHNPUM=; b=KK3aL1RP5kkCT6J9RCyc09DPmJi8uiISbNFFc6W8i5m331ovKHXT0TKh7HE2qwfKa/ TstCE4n04zUkqSDr+Ch/G2lbUHgRdA0lkfr0ngb+dPHQlrrLE2nkEStnQoQFXpUCwSPH HGX8BvAIDfKL6PNTamjy+r1uMg4/HEmYqoMvXt+s3Y76qHE9wFflcvQocIBjJJ/J52o3 X48agDKAauiXGdfbjiJIJYNxgF/Mf5G8oUyriPS2E1d7G+8CqgGctEePLteNofbH1Wsl x2Lmo8jZvJV0qudrTJD806lKcTDOAAuiAe5AC+qHMp6k6lD49wcI7OBrHWTeVIbMVm1o MDEQ== X-Gm-Message-State: AOAM530v8x+hGfb5SCVz/tqUFQ8sLP3zR9kSEtChJchYDp6w0RZzJSeg W1oZch0yscJ82xUxK/P2wyBsUv4vNvP5Ew== X-Google-Smtp-Source: ABdhPJz+3eTi/i0bpzcWb4A38bKLNiOEQ/IpZB0tMF84Z8VA+cBXSW4jfY+6yBSu6uiJ8fOx9soYtw== X-Received: by 2002:a5d:52ce:0:b0:20c:e030:b05a with SMTP id r14-20020a5d52ce000000b0020ce030b05amr25872386wrv.174.1653473100818; Wed, 25 May 2022 03:05:00 -0700 (PDT) Received: from vm.nix.is (vm.nix.is. [2a01:4f8:120:2468::2]) by smtp.gmail.com with ESMTPSA id o4-20020a05600002c400b0020d0cdbf7eesm1649452wry.111.2022.05.25.03.04.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 25 May 2022 03:05:00 -0700 (PDT) From: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= To: git@vger.kernel.org Cc: Junio C Hamano , Taylor Blau , =?utf-8?q?SZEDER_G=C3=A1bor?= , =?utf-8?b?xJBvw6Bu?= =?utf-8?b?IFRy4bqnbiBDw7RuZyBEYW5o?= , =?utf-8?q?Car?= =?utf-8?q?lo_Marcelo_Arenas_Bel=C3=B3n?= , Johannes Schindelin , Victoria Dye , Matheus Tavares , Lars Schneider , Eric Sunshine , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBC?= =?utf-8?b?amFybWFzb24=?= Subject: [PATCH v6 18/29] CI: combine ci/install{,-docker}-dependencies.sh Date: Wed, 25 May 2022 12:03:57 +0200 Message-Id: X-Mailer: git-send-email 2.36.1.1045.gf356b5617dd In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Combine the "ci/install-dependencies.sh" and "ci/install-docker-dependencies.sh" into one script. The "case" statement in the latter only cared about "$jobname", and can be folded into the same "case" statement in the former. The reason they split up is historical, until a preceding commit "ci/lib.sh" required "bash", which might not have been available in "docker". This also fixes issue in "ci/install-docker-dependencies.sh" where we'd hide errors due to not using "set -e". Now that we include "ci/lib.sh" we won't have that potential issue. Signed-off-by: Ævar Arnfjörð Bjarmason --- .github/workflows/main.yml | 2 +- ci/install-dependencies.sh | 15 +++++++++++++++ ci/install-docker-dependencies.sh | 22 ---------------------- 3 files changed, 16 insertions(+), 23 deletions(-) delete mode 100755 ci/install-docker-dependencies.sh diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml index 88eb2ef2dad..42944020436 100644 --- a/.github/workflows/main.yml +++ b/.github/workflows/main.yml @@ -311,7 +311,7 @@ jobs: container: ${{matrix.vector.image}} steps: - uses: actions/checkout@v1 - - run: ci/install-docker-dependencies.sh + - run: ci/install-dependencies.sh - run: ci/lib.sh - run: make - run: make test diff --git a/ci/install-dependencies.sh b/ci/install-dependencies.sh index 3e05d38d4ab..cc7fd3bd7f8 100755 --- a/ci/install-dependencies.sh +++ b/ci/install-dependencies.sh @@ -99,6 +99,21 @@ linux-gcc-default) sudo apt-get -q update sudo apt-get -q -y install $UBUNTU_COMMON_PKGS ;; +linux32) + linux32 --32bit i386 sh -c ' + apt update >/dev/null && + apt install -y build-essential libcurl4-openssl-dev \ + libssl-dev libexpat-dev gettext python >/dev/null + ' + ;; +linux-musl) + apk add --update build-base curl-dev openssl-dev expat-dev gettext \ + pcre2-dev python3 musl-libintl perl-utils ncurses >/dev/null + ;; +pedantic) + dnf -yq update >/dev/null && + dnf -yq install make gcc findutils diffutils perl python3 gettext zlib-devel expat-devel openssl-devel curl-devel pcre2-devel >/dev/null + ;; esac if type p4d >/dev/null 2>&1 && type p4 >/dev/null 2>&1 diff --git a/ci/install-docker-dependencies.sh b/ci/install-docker-dependencies.sh deleted file mode 100755 index 78b7e326da6..00000000000 --- a/ci/install-docker-dependencies.sh +++ /dev/null @@ -1,22 +0,0 @@ -#!/bin/sh -# -# Install dependencies required to build and test Git inside container -# - -case "$jobname" in -linux32) - linux32 --32bit i386 sh -c ' - apt update >/dev/null && - apt install -y build-essential libcurl4-openssl-dev \ - libssl-dev libexpat-dev gettext python >/dev/null - ' - ;; -linux-musl) - apk add --update build-base curl-dev openssl-dev expat-dev gettext \ - pcre2-dev python3 musl-libintl perl-utils ncurses >/dev/null - ;; -pedantic) - dnf -yq update >/dev/null && - dnf -yq install make gcc findutils diffutils perl python3 gettext zlib-devel expat-devel openssl-devel curl-devel pcre2-devel >/dev/null - ;; -esac From patchwork Wed May 25 10:03:58 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= X-Patchwork-Id: 12860937 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 91C68C433F5 for ; Wed, 25 May 2022 10:07:22 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237783AbiEYKHT (ORCPT ); Wed, 25 May 2022 06:07:19 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58500 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S241970AbiEYKFG (ORCPT ); Wed, 25 May 2022 06:05:06 -0400 Received: from mail-wr1-x42d.google.com (mail-wr1-x42d.google.com [IPv6:2a00:1450:4864:20::42d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8E5856E8ED for ; Wed, 25 May 2022 03:05:03 -0700 (PDT) Received: by mail-wr1-x42d.google.com with SMTP id t6so29405000wra.4 for ; Wed, 25 May 2022 03:05:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=j22Ozmc3VKTxAUICVuk4RP+1HJC4qIBn0TUM1cHZmAo=; b=EbdB4WYhgZ6X48KMRKXMfQnyIFNvcnr+2r07iWtqfSt9Ns/n6HF40m/axnvht+MKrB FT6GS5Fx/6cvlIl5V6LQ7+xrc4fpdDOnxuReDTOacLeZFPYJU7G7VkpccwvmUuUUj8m6 9zfpEocFOqoB6AhDnvji4Mb8/4rmF/YNdwBhxFQkFNsoZJzr/4ONoh1HV0wELkqR7CQV w75fc4RG00KIV+691aK8rJVrpSa4mg0Lj4XOWnOTPFfM9I8naiWJK9UXag2hVxIs+Pc6 vb7MGuMTj2jAT4oIwrGwWKRPuH3+zUuVSHTer/VNTAhrvl1v8d8vmQEqUcDpefH+wD79 6uVg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=j22Ozmc3VKTxAUICVuk4RP+1HJC4qIBn0TUM1cHZmAo=; b=nMpdWUHav7TK1OT65cPSmQSfP2/z78H6nM5SDmcddyjRMF57mDBKmTvmk1lMiMpZWI O6TpqucsF3gDMNjdO+RMeVAUTez254LbN3bEpbPE+zfpgHiLOjzRB6ZzD0V4yMB7t6ik 4ACaJbIp+XrhO/sbZipy33GdDzxUkpW4XCwS/qgVPcjPUXgX7v9XRMjc/q+r6X6ResnD Cg00ENa57NkBYBk1IEPB3CwtP4jjmPDl1L62BAHyMBmdYyf7Evq2OVXP6T+j1KLI/9sY Ka8JNMPl7hAvwkibC8/pykdFpny6AVpdN2nhCK8R30tstNnBa8o1Rh/fuh9fQnUaGjzv 3kVg== X-Gm-Message-State: AOAM530b+sItsFKfxKTdWE6FCM/WKyy0bWnyU1pxYh8bybTWVh2eIGOu GQLAR3u8OZ5dWYo+Qv/anoo/BKaxANFXAg== X-Google-Smtp-Source: ABdhPJzu8o0dZ2UWDO+0pxsVxEmIni4Sih9HeGVSc5ebJVut6VW6b849kFn4s/lOgWKPNieor2PHHQ== X-Received: by 2002:a5d:644e:0:b0:20e:7267:9ef7 with SMTP id d14-20020a5d644e000000b0020e72679ef7mr22853544wrw.520.1653473101901; Wed, 25 May 2022 03:05:01 -0700 (PDT) Received: from vm.nix.is (vm.nix.is. [2a01:4f8:120:2468::2]) by smtp.gmail.com with ESMTPSA id o4-20020a05600002c400b0020d0cdbf7eesm1649452wry.111.2022.05.25.03.05.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 25 May 2022 03:05:01 -0700 (PDT) From: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= To: git@vger.kernel.org Cc: Junio C Hamano , Taylor Blau , =?utf-8?q?SZEDER_G=C3=A1bor?= , =?utf-8?b?xJBvw6Bu?= =?utf-8?b?IFRy4bqnbiBDw7RuZyBEYW5o?= , =?utf-8?q?Car?= =?utf-8?q?lo_Marcelo_Arenas_Bel=C3=B3n?= , Johannes Schindelin , Victoria Dye , Matheus Tavares , Lars Schneider , Eric Sunshine , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBC?= =?utf-8?b?amFybWFzb24=?= Subject: [PATCH v6 19/29] CI: move "env" definitions into ci/lib.sh Date: Wed, 25 May 2022 12:03:58 +0200 Message-Id: X-Mailer: git-send-email 2.36.1.1045.gf356b5617dd In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Have the GitHub CI jobs use the "ci/lib".sh as a source of truth for environment variables that affect their builds and tests. This brings them in line with the rest of the jobs, and make it easier to see the entire CI configuration per jobname at a glance. To do this we need to add a ci/lib.sh "step" to the remaining jobs that didn't have it added in preceding commits. The Makefile parameters are luckily all accepted via the environment, so we can export these instead of adding them to MAKEFLAGS. Let's also use the documented GIT_CONIFG_* mechanism for setting config instead of the internal GIT_CONFIG_PARAMETERS variable. This adjusts code added in 889cacb6897 (ci: configure GitHub Actions for CI/PR, 2020-04-11), we could probably use the GIT_AUTHOR_NAME etc, but let's keep setting this via config, just with the documented mechanism. By setting "ARTIFACTS_DIRECTORY=artifacts" once we don't need to repeat it in various places, let's get it from the environment instead. Finally, the "DEVELOPER: 1" was only needed by the "sparse" job, which wasn't using "ci/lib.sh" (and which sets DEVELOPER=1). Let's instead have the "sparse" job use the "ci/lib.sh" and remove DEVELOPER=1 from ".github/workflows/main.yml". This substantially speeds up the "sparse" job, since it'll now pick up the "--jobs" setting in MAKEFLAGS that we use everywhere else. Before it ran in around 4m30s, now in around 2m30s. Signed-off-by: Ævar Arnfjörð Bjarmason --- .github/workflows/main.yml | 29 +++++++++++------------------ ci/lib.sh | 21 +++++++++++++++++++++ 2 files changed, 32 insertions(+), 18 deletions(-) diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml index 42944020436..5f167ac5be6 100644 --- a/.github/workflows/main.yml +++ b/.github/workflows/main.yml @@ -2,9 +2,6 @@ name: CI on: [push, pull_request] -env: - DEVELOPER: 1 - jobs: ci-config: name: config @@ -93,8 +90,7 @@ jobs: shell: bash env: HOME: ${{runner.workspace}} - NO_PERL: 1 - run: . /etc/profile && make artifacts-tar ARTIFACTS_DIRECTORY=artifacts + run: . /etc/profile && make artifacts-tar if: success() - run: ci/check-unignored-build-artifacts.sh if: success() @@ -105,7 +101,7 @@ jobs: uses: actions/upload-artifact@v2 with: name: windows-artifacts - path: artifacts + path: ${{env.ARTIFACTS_DIRECTORY}} windows-test: name: win test env: @@ -126,6 +122,8 @@ jobs: shell: bash run: tar xf artifacts.tar.gz && tar xf tracked.tar.gz - uses: git-for-windows/setup-git-for-windows-sdk@v1 + - run: ci/lib.sh + shell: bash - name: test shell: bash run: . /etc/profile && ci/run-test-slice.sh ${{matrix.nr}} 10 @@ -145,9 +143,6 @@ jobs: jobname: vs-build needs: ci-config if: needs.ci-config.outputs.enabled == 'yes' - env: - NO_PERL: 1 - GIT_CONFIG_PARAMETERS: "'user.name=CI' 'user.email=ci@git'" runs-on: windows-latest steps: - uses: actions/checkout@v2 @@ -171,6 +166,8 @@ jobs: - name: copy dlls to root shell: cmd run: compat\vcbuild\vcpkg_copy_dlls.bat release + - run: ci/lib.sh + shell: bash - name: generate Visual Studio solution shell: bash run: | @@ -181,21 +178,16 @@ jobs: - name: bundle artifact tar shell: bash env: - MSVC: 1 VCPKG_ROOT: ${{github.workspace}}\compat\vcbuild\vcpkg run: | - mkdir -p artifacts && - eval "$(make -n artifacts-tar INCLUDE_DLLS_IN_ARTIFACTS=YesPlease ARTIFACTS_DIRECTORY=artifacts NO_GETTEXT=YesPlease 2>&1 | grep ^tar)" - - run: ci/check-unignored-build-artifacts.sh - if: success() - shell: bash + eval "$(make -n artifacts-tar 2>&1 | grep -e ^mkdir -e ^tar)" - name: zip up tracked files run: git archive -o artifacts/tracked.tar.gz HEAD - name: upload tracked files and build artifacts uses: actions/upload-artifact@v2 with: name: vs-artifacts - path: artifacts + path: ${{env.ARTIFACTS_DIRECTORY}} vs-test: name: win+VS test env: @@ -216,10 +208,10 @@ jobs: - name: extract tracked files and build artifacts shell: bash run: tar xf artifacts.tar.gz && tar xf tracked.tar.gz + - run: ci/lib.sh + shell: bash - name: test shell: bash - env: - NO_SVN_TESTS: 1 run: . /etc/profile && ci/run-test-slice.sh ${{matrix.nr}} 10 - name: ci/print-test-failures.sh if: failure() @@ -354,6 +346,7 @@ jobs: - uses: actions/checkout@v2 - name: Install other dependencies run: ci/install-dependencies.sh + - run: ci/lib.sh - run: make sparse documentation: name: documentation diff --git a/ci/lib.sh b/ci/lib.sh index 5141de85024..de61914134e 100755 --- a/ci/lib.sh +++ b/ci/lib.sh @@ -104,6 +104,27 @@ macos-latest) esac case "$jobname" in +windows-build) + setenv --build NO_PERL NoThanks + setenv --build ARTIFACTS_DIRECTORY artifacts + ;; +vs-build) + setenv --build NO_PERL NoThanks + setenv --build NO_GETTEXT NoThanks + setenv --build ARTIFACTS_DIRECTORY artifacts + setenv --build INCLUDE_DLLS_IN_ARTIFACTS YesPlease + setenv --build MSVC YesPlease + + setenv --build GIT_CONFIG_COUNT 2 + setenv --build GIT_CONFIG_KEY_0 user.name + setenv --build GIT_CONFIG_VALUE_0 CI + setenv --build GIT_CONFIG_KEY_1 user.emailname + setenv --build GIT_CONFIG_VALUE_1 ci@git + setenv --build GIT_CONFIG_VALUE_1 ci@git + ;; +vs-test) + setenv --test NO_SVN_TESTS YesPlease + ;; linux-gcc) setenv --test GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME main ;; From patchwork Wed May 25 10:03:59 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= X-Patchwork-Id: 12860935 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id AA071C4332F for ; Wed, 25 May 2022 10:07:17 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S241776AbiEYKHR (ORCPT ); Wed, 25 May 2022 06:07:17 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58618 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S241977AbiEYKFG (ORCPT ); Wed, 25 May 2022 06:05:06 -0400 Received: from mail-wr1-x433.google.com (mail-wr1-x433.google.com [IPv6:2a00:1450:4864:20::433]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8E3C868FB0 for ; Wed, 25 May 2022 03:05:03 -0700 (PDT) Received: by mail-wr1-x433.google.com with SMTP id f2so29435607wrc.0 for ; Wed, 25 May 2022 03:05:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=lFNU64t9l9NKVz5+L5HcRhgpnKRYfDA/2HV6zPPBjIo=; b=QEFopy13SJS9ihY0iLVFRquVGC45WL62AMQ/KWkNopkpO2tRvKiNVnQcOdToYSToa5 ccbyNwg/ujUeTmgfFnXClvzkyyiWhuKKm2B/mmp10XHRoGQBoSZOHj/vpPi+c6jCrRg6 QER5AEw4SjJsYqAs+eR886tMvwzXUKKONKyDzwNi74jwKq61jgs2CCz3lM6iBU7Q4mMB FouCa7etscxInegnRAlWns5H34qioVihl83ojSdzcJ77xTnRct8jbzT9z3i8ftfwl6IG 36pHTg/XgC1V1qR0QefEWnPLLDauxSkAyYZga8ba+ECcq7L2qtmt1pvBCX2+xjQ2TA0e 1udA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=lFNU64t9l9NKVz5+L5HcRhgpnKRYfDA/2HV6zPPBjIo=; b=ujDadNTKx3sfcAm8NDelsqBL1VqN6B6JQpH5nXY3FwD8ukls/QfLzOYzA03ZtR2JKU JeW3wwG+7UHqyICB+8/31E4oLfziVlXmiNHWSmwhJVLM5o5lPDBCXZSaMWr/5Xw0I1ZY 9HKU2prRb/FNHleVE/tefGXibtyqk+bViMxmkuF4IS+pgibidPe+gn7nizD6aQwzZ2ZR uDQ9NGaQMjKxQgmSWBUZAuKB530pcWTQS0HKGUVMlH2k6OzMM3tsdvCGWBfHbyc/4ayR tUL9TIpdRxWe58BP2UOgqFFwFt7UyUmgfKAeF/iwJ0oCKIxqYSm3TRInSNWBBEkJ9fZi 9tPw== X-Gm-Message-State: AOAM531tlrAhbD7ZxefzS2sI41Yab1EAFXhizn+KqqHyZABUXAXdUr1I hK3/fsQhINWVX8TYOd4MfnkCtEsHtED03Q== X-Google-Smtp-Source: ABdhPJyt56IKskrKa4sIjoQgPojgO3/4LqojkC9eYbtNYA/vloO703cF4BGAAKHWbn6P833yK7RKeg== X-Received: by 2002:a05:6000:791:b0:20e:615c:aae4 with SMTP id bu17-20020a056000079100b0020e615caae4mr26536539wrb.206.1653473103000; Wed, 25 May 2022 03:05:03 -0700 (PDT) Received: from vm.nix.is (vm.nix.is. [2a01:4f8:120:2468::2]) by smtp.gmail.com with ESMTPSA id o4-20020a05600002c400b0020d0cdbf7eesm1649452wry.111.2022.05.25.03.05.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 25 May 2022 03:05:02 -0700 (PDT) From: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= To: git@vger.kernel.org Cc: Junio C Hamano , Taylor Blau , =?utf-8?q?SZEDER_G=C3=A1bor?= , =?utf-8?b?xJBvw6Bu?= =?utf-8?b?IFRy4bqnbiBDw7RuZyBEYW5o?= , =?utf-8?q?Car?= =?utf-8?q?lo_Marcelo_Arenas_Bel=C3=B3n?= , Johannes Schindelin , Victoria Dye , Matheus Tavares , Lars Schneider , Eric Sunshine , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBC?= =?utf-8?b?amFybWFzb24=?= Subject: [PATCH v6 20/29] ci/run-test-slice.sh: replace shelling out with "echo" Date: Wed, 25 May 2022 12:03:59 +0200 Message-Id: X-Mailer: git-send-email 2.36.1.1045.gf356b5617dd In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Amend this code added in b819f1d2cec (ci: parallelize testing on Windows, 2019-01-29) to stop shelling out to "tr", all we're trying to do here is to turn '\n'-delimited output into a ' '-delimited list. To do that we can just use "echo". None of these files will contain the odd characters using "echo" might choke on in the general case (i.e. '\'-escapes and the like). This has the added benefit of not adding a redundant ' ' to the end of the T-parameter, which didn't matter in practice, but as we'll see in a subsequent commit makes for slightly nicer output, as we'll show this variable using a GitHub CI-native feature. Signed-off-by: Ævar Arnfjörð Bjarmason --- ci/run-test-slice.sh | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/ci/run-test-slice.sh b/ci/run-test-slice.sh index 70326961454..1de6a18ca47 100755 --- a/ci/run-test-slice.sh +++ b/ci/run-test-slice.sh @@ -5,6 +5,6 @@ . ${0%/*}/lib.sh -make --quiet -C t T="$(cd t && - ./helper/test-tool path-utils slice-tests "$1" "$2" t[0-9]*.sh | - tr '\n' ' ')" +tests=$(echo $(cd t && ./helper/test-tool path-utils slice-tests "$1" "$2" \ + t[0-9]*.sh)) +make --quiet -C t T="$tests" From patchwork Wed May 25 10:04:00 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= X-Patchwork-Id: 12860938 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E917DC433F5 for ; Wed, 25 May 2022 10:07:25 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S241840AbiEYKHY (ORCPT ); Wed, 25 May 2022 06:07:24 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59272 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S242014AbiEYKFH (ORCPT ); Wed, 25 May 2022 06:05:07 -0400 Received: from mail-wr1-x42b.google.com (mail-wr1-x42b.google.com [IPv6:2a00:1450:4864:20::42b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C9B926E8FB for ; Wed, 25 May 2022 03:05:05 -0700 (PDT) Received: by mail-wr1-x42b.google.com with SMTP id e28so28782712wra.10 for ; Wed, 25 May 2022 03:05:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Co9lpyAu/F+uSXVbhPWToK3DntcWRS98c1sNH7eopDQ=; b=hgtKJFnTIYp7hRv6Yh7/sNpobx08kJWbpzBzH4YFmpK9Hzz/pSrJaXqED/pB72JgCp 27OppSfUyEyw/O+GB6aFmMeeFmr/ZRbyqAKH0SP1V5JIYNJg1JQ2zpX5uEdq79tKEuE5 GkJCMR8gdH0pdWFbavI0eRdfMsTVzyEP9oDA8K8+J3VB6KiUQw3KojCHsXwGuCLBbDGE KseOjTkUfMFNWT5cRCyJ4xyC70NNiV3ufuIaZRjtBPl2YIw2Q03t0kbr+sQ0y+nocVSd qS4rYhiDgkwccoGrp+rAPFgREk/6OK7tnwF5erIhCEEQlVoi7XLTF8K+ymvLhorUxxHN VcVQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Co9lpyAu/F+uSXVbhPWToK3DntcWRS98c1sNH7eopDQ=; b=dNSyYqg/XCmbgm0VQ52G+dwiWDSl5rGkL/1yRLuZEqby6tw4TRTGP8ph5/RcS7M+Bi IJdroNmDGD6y3N85CGwg6JnadG0nJKULMrdairlgNbXQa5BvSCkNPneG2ub/EAIJxxvA G7PWFPhGpr0+HbmtJxhYMaSCUS9uq1HzDEm5OmEpL3IOJ8MlD75LB2TjkFHiXZXmZu0D 6A5RFtFHRl6l4nLcpESQjBxlB6uyH+HtubiMe84/Dw1YtYn8GhKzBb0/1g7jWiZVFJ2Y yKsc/n7bpQMTB/YI9Z4PW3rN6b5VJXkoFo/pMj7EvCrkjBn8+rMotmvZpxwoHdzGVesN yFUg== X-Gm-Message-State: AOAM530jEPLByI7VDm8J4WKNn00C3O3uLCTKAJMPDAMEcyxwvFYaEUYO 9Ud24pGsaLGRUfxgJzY0Wvk0W/cW0l+y+Q== X-Google-Smtp-Source: ABdhPJxw63FiSHGOMNq6muszkUNtJ9lEp5daVWANYueBfoKoj15UhGSfVg5MxjdxOgTMRE5E5HI/UQ== X-Received: by 2002:a05:6000:1887:b0:20f:e0c4:1eca with SMTP id a7-20020a056000188700b0020fe0c41ecamr11181618wri.465.1653473103997; Wed, 25 May 2022 03:05:03 -0700 (PDT) Received: from vm.nix.is (vm.nix.is. [2a01:4f8:120:2468::2]) by smtp.gmail.com with ESMTPSA id o4-20020a05600002c400b0020d0cdbf7eesm1649452wry.111.2022.05.25.03.05.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 25 May 2022 03:05:03 -0700 (PDT) From: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= To: git@vger.kernel.org Cc: Junio C Hamano , Taylor Blau , =?utf-8?q?SZEDER_G=C3=A1bor?= , =?utf-8?b?xJBvw6Bu?= =?utf-8?b?IFRy4bqnbiBDw7RuZyBEYW5o?= , =?utf-8?q?Car?= =?utf-8?q?lo_Marcelo_Arenas_Bel=C3=B3n?= , Johannes Schindelin , Victoria Dye , Matheus Tavares , Lars Schneider , Eric Sunshine , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBC?= =?utf-8?b?amFybWFzb24=?= Subject: [PATCH v6 21/29] CI: pre-select test slice in Windows & VS tests Date: Wed, 25 May 2022 12:04:00 +0200 Message-Id: X-Mailer: git-send-email 2.36.1.1045.gf356b5617dd In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org In preceding commits the tests have been changed to do their setup via $GITHUB_ENV in one step, and to then have subsequent steps that re-use that environment. Let's change the "test slice" tests added in b819f1d2cec (ci: parallelize testing on Windows, 2019-01-29) to do the same. These tests select 10% of the tests to run in 10 "test slices". Now we'll select those in a step that immediately precedes the testing step, and then simply invoke "make -C t -e". This has the advantage that the tests to be run are now listed in the standard "Run" drop-down at the start of the "test" step. Since the "T" variable in "t/Makefile" doesn't normally accept overrides from the environment we need to invoke "make" with the "-e" option (a.k.a. "--environment-overrides"). We could also make $(T) in t/Makefile be a "?=" assigned variable, but this way works, and is arguably clearer as it's more obvious that we're injecting a special list of tests that override the normal behavior of that Makefile. Note that we cannot run the top-level "make test" here, because of how the Windows CI builds git, i.e. either via CMake or some option that would cause "make test" to recompile git itself. Instead we run "make -C t [...]". Signed-off-by: Ævar Arnfjörð Bjarmason --- .github/workflows/main.yml | 10 ++++++++-- ci/{run-test-slice.sh => select-test-slice.sh} | 7 +++++-- 2 files changed, 13 insertions(+), 4 deletions(-) rename ci/{run-test-slice.sh => select-test-slice.sh} (50%) diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml index 5f167ac5be6..83e0aa1f469 100644 --- a/.github/workflows/main.yml +++ b/.github/workflows/main.yml @@ -124,9 +124,12 @@ jobs: - uses: git-for-windows/setup-git-for-windows-sdk@v1 - run: ci/lib.sh shell: bash + - name: select tests + run: . /etc/profile && ci/select-test-slice.sh ${{matrix.nr}} 10 + shell: bash - name: test shell: bash - run: . /etc/profile && ci/run-test-slice.sh ${{matrix.nr}} 10 + run: . /etc/profile && make -C t -e - name: ci/print-test-failures.sh if: failure() shell: bash @@ -210,9 +213,12 @@ jobs: run: tar xf artifacts.tar.gz && tar xf tracked.tar.gz - run: ci/lib.sh shell: bash + - name: select tests + run: . /etc/profile && ci/select-test-slice.sh ${{matrix.nr}} 10 + shell: bash - name: test shell: bash - run: . /etc/profile && ci/run-test-slice.sh ${{matrix.nr}} 10 + run: . /etc/profile && make -C t -e - name: ci/print-test-failures.sh if: failure() shell: bash diff --git a/ci/run-test-slice.sh b/ci/select-test-slice.sh similarity index 50% rename from ci/run-test-slice.sh rename to ci/select-test-slice.sh index 1de6a18ca47..f59d8cadda5 100755 --- a/ci/run-test-slice.sh +++ b/ci/select-test-slice.sh @@ -1,10 +1,13 @@ #!/bin/sh # -# Test Git in parallel +# Select a portion of the tests for testing Git in parallel # . ${0%/*}/lib.sh tests=$(echo $(cd t && ./helper/test-tool path-utils slice-tests "$1" "$2" \ t[0-9]*.sh)) -make --quiet -C t T="$tests" +if test -n "$GITHUB_ENV" +then + echo T="$tests" >>$GITHUB_ENV +fi From patchwork Wed May 25 10:04:01 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= X-Patchwork-Id: 12860939 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id BEF12C433F5 for ; Wed, 25 May 2022 10:07:28 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S241902AbiEYKH0 (ORCPT ); Wed, 25 May 2022 06:07:26 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59308 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S242110AbiEYKFI (ORCPT ); Wed, 25 May 2022 06:05:08 -0400 Received: from mail-wm1-x332.google.com (mail-wm1-x332.google.com [IPv6:2a00:1450:4864:20::332]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 099026EB29 for ; Wed, 25 May 2022 03:05:06 -0700 (PDT) Received: by mail-wm1-x332.google.com with SMTP id p19so3707549wmg.2 for ; Wed, 25 May 2022 03:05:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=BWrM+ZV2xX4dLLmYkFDwdkaZo8iwEXs/IquC+0fKYdw=; b=pVFS0KE3fjjV7gXfthBghzm+CFTPI9kkAULwz3DPrvv34x6SfKvUQOfz2Oo7kkdNXR GWpUvmPJK+kg5285dTxajB4b1GVgzaxFLSprKz3EWbkJQdbIKOCzxBzCs4m2GTaaf/K0 G0Lrqe/DQFjpZ+GAhDCcRidNabwpOyj0vk6IsLZRIDc6gcX//Qhw59sBypipWpkpVCXd 5kvNZ0wavC5gfDLyaWai0qCeDLVEMrWCvZCHFlpsf3u+kLNYNPymR3EHYCrra3kGuVLS tVekmOeej+Vmt84Px3IBaDxD/ltIoorWQqy3qhZBTQqZdZHcQpe+9Ws943V9N3huvKM4 BclQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=BWrM+ZV2xX4dLLmYkFDwdkaZo8iwEXs/IquC+0fKYdw=; b=7hrcNiqplZd9CLxNG8TQOr8oyunyhJFn/mVRMiGcirms8Kzi0H0wuVvIW178jE8FRQ JB9lklFx67ElWPw3ax4VcqAM9XLXLjuCgzVayWMPJE2WhsrcddXj1qUzPLrUWPYJdH2u BsOhiv6SGrg726+y/5VNmiFfKf4sc/rBKFUBExSD2ECI/h9EmsXLi+s37fR0yJAfYbjq M7dRvckNyNplqDrjQzteVFC8Wph7JgYcWYkT9PMAoXT+nyd4f2x6cwQ+FS59tL95G/Zn kmF7WRd+tK75WEZU5H/rHqqjTjVyMdvZZNuCTwu+EeHI7jcOow2IPJOo7Ngs3GBXrmJq SLYg== X-Gm-Message-State: AOAM533EuzddAnKQvng070xRIfzq7YD5oAzyq3T6R7CI1Wqy511HzOhi myHXHBf/Sf1BAHwEQdFvNjnaLgDFPANokw== X-Google-Smtp-Source: ABdhPJxUXMzduhSwvX+A2B7EbmPfBSWsuV+0nuVNMv3Zz6FwSMfgyDIjcTEQtoVtw373BJCmoStweA== X-Received: by 2002:a7b:c928:0:b0:397:65f3:67b6 with SMTP id h8-20020a7bc928000000b0039765f367b6mr3925569wml.45.1653473105070; Wed, 25 May 2022 03:05:05 -0700 (PDT) Received: from vm.nix.is (vm.nix.is. [2a01:4f8:120:2468::2]) by smtp.gmail.com with ESMTPSA id o4-20020a05600002c400b0020d0cdbf7eesm1649452wry.111.2022.05.25.03.05.04 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 25 May 2022 03:05:04 -0700 (PDT) From: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= To: git@vger.kernel.org Cc: Junio C Hamano , Taylor Blau , =?utf-8?q?SZEDER_G=C3=A1bor?= , =?utf-8?b?xJBvw6Bu?= =?utf-8?b?IFRy4bqnbiBDw7RuZyBEYW5o?= , =?utf-8?q?Car?= =?utf-8?q?lo_Marcelo_Arenas_Bel=C3=B3n?= , Johannes Schindelin , Victoria Dye , Matheus Tavares , Lars Schneider , Eric Sunshine , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBC?= =?utf-8?b?amFybWFzb24=?= Subject: [PATCH v6 22/29] CI: only invoke ci/lib.sh as "steps" in main.yml Date: Wed, 25 May 2022 12:04:01 +0200 Message-Id: X-Mailer: git-send-email 2.36.1.1045.gf356b5617dd In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Change the scripts in ci/ to stop using ci/lib.sh as a library, now that the only thing it did for them was to "set -ex" and possibly set TERM=dumb. Let's create a ci/lib-tput.sh for those that need to use "tput" instead, and have these scripts invoke "set -ex" themselves. This makes their invocation a lot less verbose, since they'll be relying on an earlier step in the CI job to have set the variables in $GITHUB_ENV, and won't be spewing their own trace output to set those variables again. Let's also create a ci/lib-ci-type.sh, and have ci/lib.sh and ci/print-test-failures.sh share the logic to discover the CI type. We could have set the CI_TYPE in the environment with "setenv", but let's avoid that verbosity for this purely internal variable. The "ci/lib.sh" is now no longer a "Library of functions shared by all CI scripts", so let's remove that commentary, and the misleading comment about "set -ex" being for "installing dependencies", we're now no longer using it in "ci/install-dependencies.sh" (but it does its own "set -ex"). Signed-off-by: Ævar Arnfjörð Bjarmason --- ci/check-unignored-build-artifacts.sh | 4 +++- ci/install-dependencies.sh | 2 +- ci/lib-ci-type.sh | 9 +++++++++ ci/lib-tput.sh | 5 +++++ ci/lib.sh | 27 ++++++++++----------------- ci/print-test-failures.sh | 6 +++--- ci/select-test-slice.sh | 2 +- ci/test-documentation.sh | 2 +- 8 files changed, 33 insertions(+), 24 deletions(-) create mode 100644 ci/lib-ci-type.sh create mode 100644 ci/lib-tput.sh diff --git a/ci/check-unignored-build-artifacts.sh b/ci/check-unignored-build-artifacts.sh index 0bc04f32804..c27d6a97f45 100755 --- a/ci/check-unignored-build-artifacts.sh +++ b/ci/check-unignored-build-artifacts.sh @@ -3,7 +3,9 @@ # Check whether the build created anything not in our .gitignore # -. ${0%/*}/lib.sh +set -ex + +. ${0%/*}/lib-tput.sh check_unignored_build_artifacts () { diff --git a/ci/install-dependencies.sh b/ci/install-dependencies.sh index cc7fd3bd7f8..9ee52769cbe 100755 --- a/ci/install-dependencies.sh +++ b/ci/install-dependencies.sh @@ -3,7 +3,7 @@ # Install dependencies required to build and test Git on Linux and macOS # -. ${0%/*}/lib.sh +set -ex UBUNTU_COMMON_PKGS="make libssl-dev libcurl4-openssl-dev libexpat-dev tcl tk gettext zlib1g-dev perl-modules liberror-perl libauthen-sasl-perl diff --git a/ci/lib-ci-type.sh b/ci/lib-ci-type.sh new file mode 100644 index 00000000000..6f01fd9e5d9 --- /dev/null +++ b/ci/lib-ci-type.sh @@ -0,0 +1,9 @@ +#!/bin/sh + +if test "$GITHUB_ACTIONS" = "true" +then + CI_TYPE=github-actions +else + echo "Could not identify CI type" >&2 + exit 1 +fi diff --git a/ci/lib-tput.sh b/ci/lib-tput.sh new file mode 100644 index 00000000000..b62ef13c44c --- /dev/null +++ b/ci/lib-tput.sh @@ -0,0 +1,5 @@ +#!/bin/sh + +# GitHub Action doesn't set TERM, which is required by tput +TERM=${TERM:-dumb} +export TERM diff --git a/ci/lib.sh b/ci/lib.sh index de61914134e..15a0011f6aa 100755 --- a/ci/lib.sh +++ b/ci/lib.sh @@ -1,13 +1,9 @@ #!/bin/sh - -# Library of functions shared by all CI scripts - -# Set 'exit on error' for all CI scripts to let the caller know that -# something went wrong. -# Set tracing executed commands, primarily setting environment variables -# and installing dependencies. set -ex +# Helper libraries +. ${0%/*}/lib-ci-type.sh + # Starting assertions if test -z "$jobname" then @@ -47,18 +43,14 @@ setenv () { fi } -# GitHub Action doesn't set TERM, which is required by tput -setenv TERM ${TERM:-dumb} - # How many jobs to run in parallel? NPROC=10 # Clear MAKEFLAGS that may come from the outside world. MAKEFLAGS=--jobs=$NPROC -if test "$GITHUB_ACTIONS" = "true" -then - CI_TYPE=github-actions +case "$CI_TYPE" in +github-actions) CC="${CC_PACKAGE:-${CC:-gcc}}" setenv --test GIT_PROVE_OPTS "--timer --jobs $NPROC" @@ -66,11 +58,12 @@ then test Windows != "$RUNNER_OS" || GIT_TEST_OPTS="--no-chain-lint --no-bin-wrappers $GIT_TEST_OPTS" setenv --test GIT_TEST_OPTS "$GIT_TEST_OPTS" -else - echo "Could not identify CI type" >&2 - env >&2 + ;; +*) + echo "Unhandled CI type: $CI_TYPE" >&2 exit 1 -fi + ;; +esac setenv --build DEVELOPER 1 setenv --test DEFAULT_TEST_TARGET prove diff --git a/ci/print-test-failures.sh b/ci/print-test-failures.sh index 47e2ea1ad10..29f8c332eca 100755 --- a/ci/print-test-failures.sh +++ b/ci/print-test-failures.sh @@ -3,10 +3,10 @@ # Print output of failing tests # -. ${0%/*}/lib.sh +set -e -# Tracing executed commands would produce too much noise in the loop below. -set +x +. ${0%/*}/lib-ci-type.sh +. ${0%/*}/lib-tput.sh cd t/ diff --git a/ci/select-test-slice.sh b/ci/select-test-slice.sh index f59d8cadda5..eccf9a48104 100755 --- a/ci/select-test-slice.sh +++ b/ci/select-test-slice.sh @@ -3,7 +3,7 @@ # Select a portion of the tests for testing Git in parallel # -. ${0%/*}/lib.sh +set -ex tests=$(echo $(cd t && ./helper/test-tool path-utils slice-tests "$1" "$2" \ t[0-9]*.sh)) diff --git a/ci/test-documentation.sh b/ci/test-documentation.sh index b8a6a6f664e..64ff212cdaa 100755 --- a/ci/test-documentation.sh +++ b/ci/test-documentation.sh @@ -3,7 +3,7 @@ # Perform sanity checks on "make doc" output and built documentation # -. ${0%/*}/lib.sh +set -ex generator=$1 From patchwork Wed May 25 10:04:02 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= X-Patchwork-Id: 12860940 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 802C8C433EF for ; Wed, 25 May 2022 10:07:33 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240832AbiEYKHb (ORCPT ); Wed, 25 May 2022 06:07:31 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59328 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S242116AbiEYKFJ (ORCPT ); Wed, 25 May 2022 06:05:09 -0400 Received: from mail-wr1-x431.google.com (mail-wr1-x431.google.com [IPv6:2a00:1450:4864:20::431]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DD0A868FB0 for ; Wed, 25 May 2022 03:05:07 -0700 (PDT) Received: by mail-wr1-x431.google.com with SMTP id u3so29410403wrg.3 for ; Wed, 25 May 2022 03:05:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=DBaSE9xs/DJloVTK+2kFOfNhxCA9fEHIoPLLlR9YFp0=; b=UjlXXOIYOp3IJRqjaY9KvQq/MTkciQWDLMuWsfRExlyiZkrN25WH0s0e+cedx0JIFU 2UAL7c92CMpKZzLtZX+dlOcVX5FatAKRhTq56ENBARagGeroD5Xse9PwglvsLDHUjkCW 0Jl/ioUlszYxgR5SscIiq+KbLQpGfdy2CJ2wiyjYIl4Vx84+2K/zokZU6ikPTcRgqZB7 ZTF18DqZ4luGNCes3md5MZHtdkWF0L5TX4A7iX/dhnGyTLpPlR064TS1b67m+QgkwXms HHpiMiuavPFQYV2Y6/qXBCew3cN0NH2V/FV+gLWUpMetC/dnD1A66EHdUzJTnkrvqtmM gnrg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=DBaSE9xs/DJloVTK+2kFOfNhxCA9fEHIoPLLlR9YFp0=; b=cdHUDZQOkzkU743OiilwV/v5q6ASDa4cyCQhKgnZKlcjejQDZ8edrRIN+JHpk1D+4W ALzUZdA5aZLlt7YJEY80UEgEF3b/irrHr0tuZtLUJLDxF4uI7LrSbaMVMnhNcM+VstKm RnxBNu29nouk+xvTZZVFMA9010V3SYsUfSNVN7bOU8RE4B2iKIvSIpgIaAQZbKZBpMEO Cslz1uQRG+Khf0Ilkt3bui1gCtgxqhA5txmndZKRjNmvd5Kn0gSjl4oWiy1SpVdeYoBE KXNqd5DL7uLXKR3FWma8HjJJOHkLjJ5tHxNj7iBJSe2d9fWdfOwZld9LxGVznxxa6Y0i H2pw== X-Gm-Message-State: AOAM533tOcczLKp21EYHN3aaLA62KtUi4YD6AkKQCmcxiQu4aCcrNvkU 0Ddr/7BFFJslZ+wQmVI8bTnGcdg4M/SnfQ== X-Google-Smtp-Source: ABdhPJwKGMN3dpjSlYU2Icy95SLekJkB2DuACoPwndfmaREUaiscEEzBnmnRIc0T/SDZT3mCKSvQvg== X-Received: by 2002:a5d:4e08:0:b0:210:297:7d4b with SMTP id p8-20020a5d4e08000000b0021002977d4bmr894290wrt.112.1653473106061; Wed, 25 May 2022 03:05:06 -0700 (PDT) Received: from vm.nix.is (vm.nix.is. [2a01:4f8:120:2468::2]) by smtp.gmail.com with ESMTPSA id o4-20020a05600002c400b0020d0cdbf7eesm1649452wry.111.2022.05.25.03.05.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 25 May 2022 03:05:05 -0700 (PDT) From: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= To: git@vger.kernel.org Cc: Junio C Hamano , Taylor Blau , =?utf-8?q?SZEDER_G=C3=A1bor?= , =?utf-8?b?xJBvw6Bu?= =?utf-8?b?IFRy4bqnbiBDw7RuZyBEYW5o?= , =?utf-8?q?Car?= =?utf-8?q?lo_Marcelo_Arenas_Bel=C3=B3n?= , Johannes Schindelin , Victoria Dye , Matheus Tavares , Lars Schneider , Eric Sunshine , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBC?= =?utf-8?b?amFybWFzb24=?= Subject: [PATCH v6 23/29] CI: narrow down variable definitions in --build and --test Date: Wed, 25 May 2022 12:04:02 +0200 Message-Id: X-Mailer: git-send-email 2.36.1.1045.gf356b5617dd In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org In a preceding step the "setvar" function was made to take a "--build", "--test" or "--all" argument to indicate where the variables it sets were used. Let's make use of that by having the relevant parts of ".github/workflows/main.yml" invoke "ci/lib.sh" with those options. By doing this the set of variables shown in build-only steps will be fewer, which makes diagnosing anything going on there easier, as we won't have to look at a deluge of e.g. GIT_TEST_* variables. For the "pedantic" job (which has no test phase) we won't run the "ci/lib.sh --test" step, which will be clearly visible as a skipped step in the UX. Since we'll now always run "--build" for "make" and "--test" for "make test" we can stop setting the "fat" MAKEFLAGS entirely on the "test" steps, i.e. the one with PYTHON_PATH, CC etc. These will all be carried over from the earlier --build step. That won't be true in the case of the "windows-test" and "vs-test" jobs, since they run separately from the corresponding "build" step. So we'll need to make sure that we have the --jobs=N argument for those. This doesn't matter that much, as we'll still have --jobs=N in GIT_PROVE_OPTS. So the only thing we'll use it for is parallelism in the t/Makefile before we get to running "prove". Still, it's good to be consistent for good measure, and to run the t/Makefile itself in parallel. Signed-off-by: Ævar Arnfjörð Bjarmason --- .github/workflows/main.yml | 21 ++++++++------- ci/lib.sh | 52 +++++++++++++++++++++++++++++++++++--- 2 files changed, 60 insertions(+), 13 deletions(-) diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml index 83e0aa1f469..bb62b4ff725 100644 --- a/.github/workflows/main.yml +++ b/.github/workflows/main.yml @@ -84,7 +84,7 @@ jobs: steps: - uses: actions/checkout@v2 - uses: git-for-windows/setup-git-for-windows-sdk@v1 - - run: ci/lib.sh + - run: ci/lib.sh --build shell: bash - name: build shell: bash @@ -122,7 +122,7 @@ jobs: shell: bash run: tar xf artifacts.tar.gz && tar xf tracked.tar.gz - uses: git-for-windows/setup-git-for-windows-sdk@v1 - - run: ci/lib.sh + - run: ci/lib.sh --test shell: bash - name: select tests run: . /etc/profile && ci/select-test-slice.sh ${{matrix.nr}} 10 @@ -169,7 +169,7 @@ jobs: - name: copy dlls to root shell: cmd run: compat\vcbuild\vcpkg_copy_dlls.bat release - - run: ci/lib.sh + - run: ci/lib.sh --build shell: bash - name: generate Visual Studio solution shell: bash @@ -211,7 +211,7 @@ jobs: - name: extract tracked files and build artifacts shell: bash run: tar xf artifacts.tar.gz && tar xf tracked.tar.gz - - run: ci/lib.sh + - run: ci/lib.sh --test shell: bash - name: select tests run: . /etc/profile && ci/select-test-slice.sh ${{matrix.nr}} 10 @@ -275,8 +275,9 @@ jobs: steps: - uses: actions/checkout@v2 - run: ci/install-dependencies.sh - - run: ci/lib.sh + - run: ci/lib.sh --build - run: make + - run: ci/lib.sh --test - run: make test if: success() - run: ci/print-test-failures.sh @@ -310,8 +311,10 @@ jobs: steps: - uses: actions/checkout@v1 - run: ci/install-dependencies.sh - - run: ci/lib.sh + - run: ci/lib.sh --build - run: make + - run: ci/lib.sh --test + if: success() && matrix.vector.skip-tests != 'yes' - run: make test if: success() && matrix.vector.skip-tests != 'yes' - run: ci/print-test-failures.sh @@ -331,7 +334,7 @@ jobs: steps: - uses: actions/checkout@v2 - run: ci/install-dependencies.sh - - run: ci/lib.sh + - run: ci/lib.sh --build - run: make ci-static-analysis sparse: needs: ci-config @@ -352,7 +355,7 @@ jobs: - uses: actions/checkout@v2 - name: Install other dependencies run: ci/install-dependencies.sh - - run: ci/lib.sh + - run: ci/lib.sh --build - run: make sparse documentation: name: documentation @@ -364,7 +367,7 @@ jobs: steps: - uses: actions/checkout@v2 - run: ci/install-dependencies.sh - - run: ci/lib.sh + - run: ci/lib.sh --build - run: make check-docs - run: "make doc > >(tee stdout.log) 2> >(tee stderr.raw >&2)" shell: bash diff --git a/ci/lib.sh b/ci/lib.sh index 15a0011f6aa..96919882c11 100755 --- a/ci/lib.sh +++ b/ci/lib.sh @@ -4,6 +4,30 @@ set -ex # Helper libraries . ${0%/*}/lib-ci-type.sh +# Parse options +mode_build= +mode_test= +while test $# != 0 +do + case "$1" in + --build) + mode_build=t + ;; + --test) + mode_test=t + ;; + -*) + echo "error: invalid option: $1" >&2 + exit 1 + ;; + *) + echo "error: invalid argument: $1" >&2 + exit 1 + ;; + esac + shift +done + # Starting assertions if test -z "$jobname" then @@ -11,16 +35,28 @@ then exit 1 fi +if test "$mode_test$mode_build" != "t" +then + echo "error: need one mode, e.g. --build or --test" >&2 + exit 1 +fi + # Helper functions setenv () { while test $# != 0 do case "$1" in --build) + if test -z "$mode_build" + then + return 0 + fi ;; --test) - ;; - --all) + if test -z "$mode_test" + then + return 0 + fi ;; -*) echo "BUG: bad setenv() option '$1'" >&2 @@ -46,8 +82,12 @@ setenv () { # How many jobs to run in parallel? NPROC=10 +# For "--test" we carry the MAKEFLAGS over from earlier steps, except +# in stand-alone jobs which will use $COMMON_MAKEFLAGS. +COMMON_MAKEFLAGS=--jobs=$NPROC + # Clear MAKEFLAGS that may come from the outside world. -MAKEFLAGS=--jobs=$NPROC +MAKEFLAGS=$COMMON_MAKEFLAGS case "$CI_TYPE" in github-actions) @@ -101,6 +141,9 @@ windows-build) setenv --build NO_PERL NoThanks setenv --build ARTIFACTS_DIRECTORY artifacts ;; +windows-test) + setenv --test MAKEFLAGS "$COMMON_MAKEFLAGS" + ;; vs-build) setenv --build NO_PERL NoThanks setenv --build NO_GETTEXT NoThanks @@ -117,6 +160,7 @@ vs-build) ;; vs-test) setenv --test NO_SVN_TESTS YesPlease + setenv --test MAKEFLAGS "$COMMON_MAKEFLAGS" ;; linux-gcc) setenv --test GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME main @@ -162,4 +206,4 @@ linux-leaks) ;; esac -setenv --all MAKEFLAGS "$MAKEFLAGS CC=${CC:-cc}" +setenv --build MAKEFLAGS "$MAKEFLAGS CC=${CC:-cc}" From patchwork Wed May 25 10:04:03 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= X-Patchwork-Id: 12860922 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B10CAC433EF for ; Wed, 25 May 2022 10:05:30 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S242315AbiEYKF2 (ORCPT ); Wed, 25 May 2022 06:05:28 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59342 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S242153AbiEYKFK (ORCPT ); Wed, 25 May 2022 06:05:10 -0400 Received: from mail-wm1-x330.google.com (mail-wm1-x330.google.com [IPv6:2a00:1450:4864:20::330]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C476B6D94A for ; Wed, 25 May 2022 03:05:08 -0700 (PDT) Received: by mail-wm1-x330.google.com with SMTP id i20-20020a05600c355400b0039456976dcaso2270460wmq.1 for ; Wed, 25 May 2022 03:05:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=DBl2S8ZK0TYIowZNmFyVVner8SL3mzEtlRBS7yrc8L0=; b=XnfKJ236G1bP6iymelT+eHiAEymL8hZlt1fZfrHBWe8dvPgSLAMzZCd3w8hPfdE9ZR H4uyAeL1ganLQ/aOedmqEZvnnZtL4KQCgnzF7vduKgqMQwA+GEPYUj0nLALyFjH6p1Q0 P9Vu6vnIM9U77+dsiwguXjFpBe1eiC4d3TcV3ZaqtnngXOVWrlKjvG4KHC0SevzNF9dU yUD7l9ELtY1OIDMFhcA0nlFBYEsm822w2zEhahYA62E52tBerK5d18T4NCQl11KxWS6o 49SiXiRItzlzw6pAvSlXiaXwffsfmXC2AcpnmQFyj5+omLTdBFx7aJJH5ZmHQBazjgUM pJVQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=DBl2S8ZK0TYIowZNmFyVVner8SL3mzEtlRBS7yrc8L0=; b=HuN2SVlz7KeckZGo7HVXpBPt+YShBr1D864fWVpXV9w5z7Dm3WY8eY+XrEqEiSs1Rh LRiNIUn5F935PDLvpnjqq9ErGRmlol+Aktgzu87XvSqf8MIVUFNuZ0LtSH0GH/REmk8A TeuALpO5a39EAPazC/DoIb/Ar9gvmTgjtWntLm03TjyW0QujTa0nEQ3xkl1Yyrnp1K+d jKGcNqt54VfUnD3Gem/4MAglbE52lXdBr98LxM7uDf6ut+V5TDZEBz9m6UF4XH0kYQod eWQco1aOli/ztD2mS4Ryb7RtoSglO/NRiEQteo713MIcbTHu2q4Xq4yIcPLKChysdQEr uqtA== X-Gm-Message-State: AOAM532O9FNP40i4EcoO1vYwmhDkO3GLsT1ceiHAMhY70pLE7uneN7Ll /ZRj5FXml4k9hNpaTXWrHN8k+5rXPVlyzg== X-Google-Smtp-Source: ABdhPJyjV2e2wrWU058Db/5b68BfszOZpBBVF5hsZWA0F6EcMAFt9tKX2S9JHAf01tLMwBs1C1h4Cw== X-Received: by 2002:a7b:cf11:0:b0:397:33e3:87b2 with SMTP id l17-20020a7bcf11000000b0039733e387b2mr7411768wmg.152.1653473107148; Wed, 25 May 2022 03:05:07 -0700 (PDT) Received: from vm.nix.is (vm.nix.is. [2a01:4f8:120:2468::2]) by smtp.gmail.com with ESMTPSA id o4-20020a05600002c400b0020d0cdbf7eesm1649452wry.111.2022.05.25.03.05.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 25 May 2022 03:05:06 -0700 (PDT) From: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= To: git@vger.kernel.org Cc: Junio C Hamano , Taylor Blau , =?utf-8?q?SZEDER_G=C3=A1bor?= , =?utf-8?b?xJBvw6Bu?= =?utf-8?b?IFRy4bqnbiBDw7RuZyBEYW5o?= , =?utf-8?q?Car?= =?utf-8?q?lo_Marcelo_Arenas_Bel=C3=B3n?= , Johannes Schindelin , Victoria Dye , Matheus Tavares , Lars Schneider , Eric Sunshine , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBC?= =?utf-8?b?amFybWFzb24=?= Subject: [PATCH v6 24/29] CI: add more variables to MAKEFLAGS, except under vs-build Date: Wed, 25 May 2022 12:04:03 +0200 Message-Id: X-Mailer: git-send-email 2.36.1.1045.gf356b5617dd In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org It's clearer that "DEVELOPER" is a flag that affects the Makefile itself in particular if it's put into "MAKEFLAGS than" if it generically sits in the environment. Let's move both it and "SKIP_DASHED_BUILT_INS" to "MAKEFLAGS". We can't do this under vs-build, since that invokes cmake. Let's have only that job set these in the environment. Signed-off-by: Ævar Arnfjörð Bjarmason --- ci/lib.sh | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/ci/lib.sh b/ci/lib.sh index 96919882c11..16bac966d76 100755 --- a/ci/lib.sh +++ b/ci/lib.sh @@ -89,6 +89,13 @@ COMMON_MAKEFLAGS=--jobs=$NPROC # Clear MAKEFLAGS that may come from the outside world. MAKEFLAGS=$COMMON_MAKEFLAGS +# Use common options for "make" (cmake in "vs-build" below uses the +# intermediate variables directly) +DEVELOPER=1 +MAKEFLAGS="$MAKEFLAGS DEVELOPER=$DEVELOPER" +SKIP_DASHED_BUILT_INS=YesPlease +MAKEFLAGS="$MAKEFLAGS SKIP_DASHED_BUILT_INS=$SKIP_DASHED_BUILT_INS" + case "$CI_TYPE" in github-actions) CC="${CC_PACKAGE:-${CC:-gcc}}" @@ -105,10 +112,8 @@ github-actions) ;; esac -setenv --build DEVELOPER 1 setenv --test DEFAULT_TEST_TARGET prove setenv --test GIT_TEST_CLONE_2GB true -setenv --build SKIP_DASHED_BUILT_INS YesPlease case "$runs_on_pool" in ubuntu-latest) @@ -145,6 +150,9 @@ windows-test) setenv --test MAKEFLAGS "$COMMON_MAKEFLAGS" ;; vs-build) + setenv --build DEVELOPER $DEVELOPER + setenv --build SKIP_DASHED_BUILT_INS $SKIP_DASHED_BUILT_INS + setenv --build NO_PERL NoThanks setenv --build NO_GETTEXT NoThanks setenv --build ARTIFACTS_DIRECTORY artifacts From patchwork Wed May 25 10:04:04 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= X-Patchwork-Id: 12860923 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C6471C433EF for ; Wed, 25 May 2022 10:05:33 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S241560AbiEYKFb (ORCPT ); Wed, 25 May 2022 06:05:31 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59344 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S242146AbiEYKFK (ORCPT ); Wed, 25 May 2022 06:05:10 -0400 Received: from mail-wr1-x42d.google.com (mail-wr1-x42d.google.com [IPv6:2a00:1450:4864:20::42d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id F0F276D965 for ; Wed, 25 May 2022 03:05:08 -0700 (PDT) Received: by mail-wr1-x42d.google.com with SMTP id t6so29405000wra.4 for ; Wed, 25 May 2022 03:05:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=geuTPMRLF/iDT2MlysZcLrWTlwVSkSOnoLSL9MP8SkQ=; b=l8g4L9rfiZNeeuOiir+tmJYnW+OGdC7NwaHWQ8XF2x86ZjUyxsNP8F1tTXs1hxiBOC O+jv4D5JNgGPsX1lfvqNZwLdnaHd5FLW59qnfnDDPuzO7jSgWu3GBkkNKHgTd5sxOJBW vqWvCMInpZZUEL4G/Yd+GCRY8NVNXeqYtMxeJZBc286gQmuQrvQAvIZ+64d2nfF2iO/z Hx5twKllKQIeSKaTXuIZWr390Aq46v9C9eU6XkkXP6P3eJPs/+xflGfMDXB44u1/e7Pm kFIq2X3GXoM/O3nhYW0Hkvx2+aaCQT6b/E3Gg5rDoVHbhvSDAj+omHivG5Uec1hsVCPJ /0xg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=geuTPMRLF/iDT2MlysZcLrWTlwVSkSOnoLSL9MP8SkQ=; b=3i+Kd1uUiF0fuYVAp1zRmAcTn9VOXIUvIf5DmHDBfxnP3l0tHu5O5gIERtKoqdWwb6 Q5ki+Uw5zuyZKz+Ychmly7j7PGzWloZwZRPRhl9d91zjlJwlLT7yUvpeU3L4Q8Vc0OTT y6wUNAbihdDQ3Vay1xgH0jsGyVplcg74T544JvkRzfUNG/3LJ2d1KumJ+F0X4UWuFv5X qb8QfjZgF29xyeGVDHg5nCqBe6JOcIJNR+WL9oDQJCQCIO0YfYv1eofrUKOt1enwLxlt qxkyIticOGxMwOmA8ynzqskK8qgWLMHXSwIc7D8wE/vHsTzuwCvGSApC2uYbrds7f62B RXvQ== X-Gm-Message-State: AOAM532FHvvl9pI2KLM2iaUCl9I+4ziuee5FIwaBwvvNbLRrko68C9TJ sTuYZIAwnmjzzglYJldoCHfMienR4TRgEg== X-Google-Smtp-Source: ABdhPJz9AF/fa6+hpBZT4Q6c5qF3roNy/8Mk0FTMiXS/2oXE3LF5/T7gJEUbMJmeFf9BG/5UJ8sUOQ== X-Received: by 2002:a05:6000:1621:b0:20f:f723:ec49 with SMTP id v1-20020a056000162100b0020ff723ec49mr5210872wrb.284.1653473108201; Wed, 25 May 2022 03:05:08 -0700 (PDT) Received: from vm.nix.is (vm.nix.is. [2a01:4f8:120:2468::2]) by smtp.gmail.com with ESMTPSA id o4-20020a05600002c400b0020d0cdbf7eesm1649452wry.111.2022.05.25.03.05.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 25 May 2022 03:05:07 -0700 (PDT) From: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= To: git@vger.kernel.org Cc: Junio C Hamano , Taylor Blau , =?utf-8?q?SZEDER_G=C3=A1bor?= , =?utf-8?b?xJBvw6Bu?= =?utf-8?b?IFRy4bqnbiBDw7RuZyBEYW5o?= , =?utf-8?q?Car?= =?utf-8?q?lo_Marcelo_Arenas_Bel=C3=B3n?= , Johannes Schindelin , Victoria Dye , Matheus Tavares , Lars Schneider , Eric Sunshine , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBC?= =?utf-8?b?amFybWFzb24=?= Subject: [PATCH v6 25/29] CI: set CC in MAKEFLAGS directly, don't add it to the environment Date: Wed, 25 May 2022 12:04:04 +0200 Message-Id: X-Mailer: git-send-email 2.36.1.1045.gf356b5617dd In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Rather than pass a "$CC" in the environment to be picked up in ci/lib.sh let's instead have ci/lib.sh itself add it directly to MAKEFLAGS. For "$CC_PACKAGE" its setting and use can stay within ci/install-dependencies.sh. Setting CC=gcc by default made for confusing trace output, and since a preceding change to carry it and others over across "steps" in the GitHub CI it's been even more misleading. E.g. the "win+VS build" job confusingly has CC=gcc set, even though it builds with MSVC. Let's instead rely on the Makefile default of CC=cc, and only override it for those jobs where it's needed. This does mean that we'll need to set it for the "pedantic" job, which previously relied on the default CC=gcc in case "clang" become the default on that platform. This partially reverts my 707d2f2fe86 (CI: use "$runs_on_pool", not "$jobname" to select packages & config, 2021-11-23), i.e. we're now aiming to only set those variables specific jobs need. Since we don't need to make this generic enough to handle "gcc-9" turning into "gcc@9" for "brew install" let's remove what was originally a bash-specific replacement in 707d2f2fe86 (CI: use "$runs_on_pool", not "$jobname" to select packages & config, 2021-11-23), and which a preceding commit changed to a "tr" invocation. Signed-off-by: Ævar Arnfjörð Bjarmason --- .github/workflows/main.yml | 13 ------------- ci/install-dependencies.sh | 18 ++++++++++++++---- ci/lib.sh | 23 ++++++++++++++++++++--- 3 files changed, 34 insertions(+), 20 deletions(-) diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml index bb62b4ff725..6835e942280 100644 --- a/.github/workflows/main.yml +++ b/.github/workflows/main.yml @@ -238,37 +238,24 @@ jobs: matrix: vector: - jobname: linux-clang - cc: clang pool: ubuntu-latest - jobname: linux-sha256 - cc: clang os: ubuntu pool: ubuntu-latest - jobname: linux-gcc - cc: gcc - cc_package: gcc-8 pool: ubuntu-latest - jobname: linux-TEST-vars - cc: gcc os: ubuntu - cc_package: gcc-8 pool: ubuntu-latest - jobname: osx-clang - cc: clang pool: macos-latest - jobname: osx-gcc - cc: gcc - cc_package: gcc-9 pool: macos-latest - jobname: linux-gcc-default - cc: gcc pool: ubuntu-latest - jobname: linux-leaks - cc: gcc pool: ubuntu-latest env: - CC: ${{matrix.vector.cc}} - CC_PACKAGE: ${{matrix.vector.cc_package}} jobname: ${{matrix.vector.jobname}} runs_on_pool: ${{matrix.vector.pool}} runs-on: ${{matrix.vector.pool}} diff --git a/ci/install-dependencies.sh b/ci/install-dependencies.sh index 9ee52769cbe..b2be92f4370 100755 --- a/ci/install-dependencies.sh +++ b/ci/install-dependencies.sh @@ -9,6 +9,17 @@ UBUNTU_COMMON_PKGS="make libssl-dev libcurl4-openssl-dev libexpat-dev tcl tk gettext zlib1g-dev perl-modules liberror-perl libauthen-sasl-perl libemail-valid-perl libio-socket-ssl-perl libnet-smtp-ssl-perl" +CC_PACKAGE= +BREW_CC_PACKAGE= +case "$jobname" in +linux-gcc | linux-TEST-vars) + CC_PACKAGE=gcc-8 + ;; +osx-gcc) + BREW_CC_PACKAGE=gcc@9 + ;; +esac + case "$runs_on_pool" in ubuntu-latest) # The Linux build installs the defined dependency versions below. @@ -69,11 +80,10 @@ macos-latest) PATH="$PATH:${HOME}/bin" export PATH - if test -n "$CC_PACKAGE" + if test -n "$BREW_CC_PACKAGE" then - BREW_PACKAGE=$(echo $CC_PACKAGE | tr '-' '@') - brew install "$BREW_PACKAGE" - brew link "$BREW_PACKAGE" + brew install "$BREW_CC_PACKAGE" + brew link "$BREW_CC_PACKAGE" fi ;; esac diff --git a/ci/lib.sh b/ci/lib.sh index 16bac966d76..8d19ca5ced8 100755 --- a/ci/lib.sh +++ b/ci/lib.sh @@ -79,6 +79,9 @@ setenv () { fi } +# Clear variables that may come from the outside world. +CC= + # How many jobs to run in parallel? NPROC=10 @@ -98,8 +101,6 @@ MAKEFLAGS="$MAKEFLAGS SKIP_DASHED_BUILT_INS=$SKIP_DASHED_BUILT_INS" case "$CI_TYPE" in github-actions) - CC="${CC_PACKAGE:-${CC:-gcc}}" - setenv --test GIT_PROVE_OPTS "--timer --jobs $NPROC" GIT_TEST_OPTS="--verbose-log -x" test Windows != "$RUNNER_OS" || @@ -171,9 +172,14 @@ vs-test) setenv --test MAKEFLAGS "$COMMON_MAKEFLAGS" ;; linux-gcc) + CC=gcc setenv --test GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME main ;; +linux-gcc-default) + CC=gcc + ;; linux-TEST-vars) + CC=gcc setenv --test GIT_TEST_SPLIT_INDEX yes setenv --test GIT_TEST_MERGE_ALGORITHM recursive setenv --test GIT_TEST_FULL_IN_PACK_ARRAY true @@ -188,13 +194,22 @@ linux-TEST-vars) setenv --test GIT_TEST_WRITE_REV_INDEX 1 setenv --test GIT_TEST_CHECKOUT_WORKERS 2 ;; +osx-gcc) + CC=gcc + ;; +osx-clang) + CC=clang + ;; linux-clang) + CC=clang setenv --test GIT_TEST_DEFAULT_HASH sha1 ;; linux-sha256) + CC=clang setenv --test GIT_TEST_DEFAULT_HASH sha256 ;; pedantic) + CC=gcc # Don't run the tests; we only care about whether Git can be # built. setenv --build DEVOPTS pedantic @@ -209,9 +224,11 @@ linux-musl) MAKEFLAGS="$MAKEFLAGS GIT_TEST_UTF8_LOCALE=C.UTF-8" ;; linux-leaks) + CC=gcc setenv --build SANITIZE leak setenv --test GIT_TEST_PASSING_SANITIZE_LEAK true ;; esac -setenv --build MAKEFLAGS "$MAKEFLAGS CC=${CC:-cc}" +MAKEFLAGS="$MAKEFLAGS${CC:+ CC=$CC}" +setenv --build MAKEFLAGS "$MAKEFLAGS" From patchwork Wed May 25 10:04:05 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= X-Patchwork-Id: 12860921 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 33EF6C433EF for ; Wed, 25 May 2022 10:05:28 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S241540AbiEYKF0 (ORCPT ); Wed, 25 May 2022 06:05:26 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59354 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S242154AbiEYKFK (ORCPT ); Wed, 25 May 2022 06:05:10 -0400 Received: from mail-wr1-x433.google.com (mail-wr1-x433.google.com [IPv6:2a00:1450:4864:20::433]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B61D36D970 for ; Wed, 25 May 2022 03:05:09 -0700 (PDT) Received: by mail-wr1-x433.google.com with SMTP id f2so29435607wrc.0 for ; Wed, 25 May 2022 03:05:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=4srXT5BHtJZZxKgsAR5TATfiegFgKI2HbOM+OwQefac=; b=Zad+jgWLmXpPRI866JeZiVNuCh/d+uinqHHOo1WX8nKFj44wxeh+9ClLpqhiki65L/ uAGurOG+Ei7mnvSRI2QKBMEwi1rTqD8Lde2uMxXvfriASzXXTL4eHdeS/Frzn0t11ff6 1Dly12FjvoZsswwHzCCgCAYKS9Rj09zmG7Q12AUF39TQd+XMalLJkrjaDfvOKSW/m5NM 52x0sGK+hByEy7DqXxS/MgJsTcQ4exc55GhvcR/2e5658csJTQ3QhqtDR5dGaXT/OXrn RtNzmQOUT/ZkB72U8S8KPSainAMTd7bZJmSRdOU7fDLLrhI+4bq5hWJtF2DKB2OtYCxU xuQQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=4srXT5BHtJZZxKgsAR5TATfiegFgKI2HbOM+OwQefac=; b=i6EIFQl4HYVVG1Tu1wkGa5p1nQMcBA5PLYExg/wooxwwaTbDOT65qFs4F9rrRhCdl+ fbvKcs8YaaEOrnSGKudqy3kLBWy9Pg+iokdQf0MbkCK5iwv07w+toVpCUhlZFzFTQ634 jENJeAUVizs9rpFjqJu/avfxmOwKgxAFZ6WYlH5oKisZTdQdFGG7SdULeKme601cqKZO zA6yit2+KtPgxs1PvPSP52RZLN0NxFs0UpD48H2VEnkimPB19i7BQe0JkcrpIPIlcojJ 1oX9Zqzs+62Tt13Ry3GLhJniqHk5GZSuAe1mBK7sfr3E/3/OC3jjb359N6KG0NHBtFU3 GWPA== X-Gm-Message-State: AOAM533x1EhYRcLhf41qlIhOASGduw0fqS5D/qkIfkSK5vmfMzWbOkSI o64OOMpHcc0GnKYa4L4CaN7PERx31ufr7Q== X-Google-Smtp-Source: ABdhPJxoyEbXzfhc7U6Rr149CP2Kbvb1Emdewq7KSNdCJaj37bit7wCAm2UUzMfnaN2YulLkiVAgPg== X-Received: by 2002:adf:f90d:0:b0:20c:de32:4d35 with SMTP id b13-20020adff90d000000b0020cde324d35mr25916294wrr.583.1653473109042; Wed, 25 May 2022 03:05:09 -0700 (PDT) Received: from vm.nix.is (vm.nix.is. [2a01:4f8:120:2468::2]) by smtp.gmail.com with ESMTPSA id o4-20020a05600002c400b0020d0cdbf7eesm1649452wry.111.2022.05.25.03.05.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 25 May 2022 03:05:08 -0700 (PDT) From: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= To: git@vger.kernel.org Cc: Junio C Hamano , Taylor Blau , =?utf-8?q?SZEDER_G=C3=A1bor?= , =?utf-8?b?xJBvw6Bu?= =?utf-8?b?IFRy4bqnbiBDw7RuZyBEYW5o?= , =?utf-8?q?Car?= =?utf-8?q?lo_Marcelo_Arenas_Bel=C3=B3n?= , Johannes Schindelin , Victoria Dye , Matheus Tavares , Lars Schneider , Eric Sunshine , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBC?= =?utf-8?b?amFybWFzb24=?= Subject: [PATCH v6 26/29] CI: set SANITIZE=leak in MAKEFLAGS directly Date: Wed, 25 May 2022 12:04:05 +0200 Message-Id: X-Mailer: git-send-email 2.36.1.1045.gf356b5617dd In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org As in the preceding commit's change to do likewise for CC, let's not set SANITIZE in the environment. When I added the "linux-leaks" job in in 956d2e4639b (tests: add a test mode for SANITIZE=leak, run it in CI, 2021-09-23) I missed that I could do that. Signed-off-by: Ævar Arnfjörð Bjarmason --- ci/lib.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/ci/lib.sh b/ci/lib.sh index 8d19ca5ced8..b6e7a6647bb 100755 --- a/ci/lib.sh +++ b/ci/lib.sh @@ -225,7 +225,7 @@ linux-musl) ;; linux-leaks) CC=gcc - setenv --build SANITIZE leak + MAKEFLAGS="$MAKEFLAGS SANITIZE=leak" setenv --test GIT_TEST_PASSING_SANITIZE_LEAK true ;; esac From patchwork Wed May 25 10:04:06 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= X-Patchwork-Id: 12860924 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4DCC0C433EF for ; Wed, 25 May 2022 10:05:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S242273AbiEYKFe (ORCPT ); Wed, 25 May 2022 06:05:34 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59366 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S242178AbiEYKFL (ORCPT ); Wed, 25 May 2022 06:05:11 -0400 Received: from mail-wr1-x431.google.com (mail-wr1-x431.google.com [IPv6:2a00:1450:4864:20::431]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E26476D966 for ; Wed, 25 May 2022 03:05:10 -0700 (PDT) Received: by mail-wr1-x431.google.com with SMTP id u3so29410403wrg.3 for ; Wed, 25 May 2022 03:05:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=mStoIdNkwfQz/HeoGy0QNWkorleVJjQCBXdp7Pi1pTg=; b=OekiTj+GusXiO/gXyZmoxgxF22L+CuIbifeUTz5FKSW59lthxbXF/Yf+zPugtJN269 fsqL/+Mbq6UqojTPGgagC9gqNTNqCnIS+R4MBD2gKrX36Pow6Co0W8QCLd1syctMqquL 3S6Ti2BUppm0xnKenmW441ha9rrL9/GAgBvTNcADHum7ou8slXhKfMQSbXcSl/9d9+Zf cCuXnrIQrEkN/sSc4FcjZqpQjjdN1ObbPSGLK8EM86w3vfCOMH9xRFxWKXWhYMPfT/Tt kmN9dJae4f6RJukY28W+B6PDrhB9N/Avgzd/vEqauYBVeP14diCmkdOvfKn6honC4lTo u8Ig== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=mStoIdNkwfQz/HeoGy0QNWkorleVJjQCBXdp7Pi1pTg=; b=drk0GuiLFDfxxA0occc2IZlwuV03b+FDjZLYMLkgIm+NyfH7H9pRvUdCYevOdiBbp8 wlMIqFmHzsCpC/kWJ18iWhNVqKf/ndx/NGo4Jdtga47KucTN7qP1GE+Hs8LAIVvaAkKv eWwt+BWE0+rw1QeVBcbdTF3pj4ak7up2w3MKQeeDfPrrcftYBuui2QlW/41A92ddeVJ7 569NGNNQdkU7j5HSK7VRXVylNXqmVf7MP/09uFZPde9ez1Q0XlTQ4OcoUayXWfSRS27l uF8ikZfr+RJQzanAdwlKLoKp6x4YM9PNaxslvIVSjfnXsvw052AMSN3JW2G8E59bTcfA 5IUQ== X-Gm-Message-State: AOAM530Kn6avS3WFBJGoaN5JThvQkGzVvMBAse7cWYklejXQXQCeD44i uJu2lii61ssg06L+k81DXjb1y1fm+an9iA== X-Google-Smtp-Source: ABdhPJwdp9MpDdywr3VdaFjf/Mdteh5RD5AAjfTulCKAt2pq1hi70IS/msDssRq/LCzidBeLdjEJxw== X-Received: by 2002:a5d:588d:0:b0:20f:f290:ed83 with SMTP id n13-20020a5d588d000000b0020ff290ed83mr5681821wrf.588.1653473110209; Wed, 25 May 2022 03:05:10 -0700 (PDT) Received: from vm.nix.is (vm.nix.is. [2a01:4f8:120:2468::2]) by smtp.gmail.com with ESMTPSA id o4-20020a05600002c400b0020d0cdbf7eesm1649452wry.111.2022.05.25.03.05.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 25 May 2022 03:05:09 -0700 (PDT) From: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= To: git@vger.kernel.org Cc: Junio C Hamano , Taylor Blau , =?utf-8?q?SZEDER_G=C3=A1bor?= , =?utf-8?b?xJBvw6Bu?= =?utf-8?b?IFRy4bqnbiBDw7RuZyBEYW5o?= , =?utf-8?q?Car?= =?utf-8?q?lo_Marcelo_Arenas_Bel=C3=B3n?= , Johannes Schindelin , Victoria Dye , Matheus Tavares , Lars Schneider , Eric Sunshine , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBC?= =?utf-8?b?amFybWFzb24=?= Subject: [PATCH v6 27/29] CI: set PYTHON_PATH setting for osx-{clang,gcc} into "$jobname" case Date: Wed, 25 May 2022 12:04:06 +0200 Message-Id: X-Mailer: git-send-email 2.36.1.1045.gf356b5617dd In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Move the setting of "PYTHON_PATH=[...]" to the "$jobname" case statement. This partially backs out of my 707d2f2fe86 (CI: use "$runs_on_pool", not "$jobname" to select packages & config, 2021-11-23), now that we have a "osx-{clang,gcc}" anyway for setting "$CC" we might as well do away with this part of the "$runs_on_pool" case. Signed-off-by: Ævar Arnfjörð Bjarmason --- ci/lib.sh | 10 ++-------- 1 file changed, 2 insertions(+), 8 deletions(-) diff --git a/ci/lib.sh b/ci/lib.sh index b6e7a6647bb..f2f75901a9c 100755 --- a/ci/lib.sh +++ b/ci/lib.sh @@ -132,14 +132,6 @@ ubuntu-latest) setenv --test GIT_TEST_HTTPD true ;; -macos-latest) - if test "$jobname" = osx-gcc - then - MAKEFLAGS="$MAKEFLAGS PYTHON_PATH=$(which python3)" - else - MAKEFLAGS="$MAKEFLAGS PYTHON_PATH=$(which python2)" - fi - ;; esac case "$jobname" in @@ -195,9 +187,11 @@ linux-TEST-vars) setenv --test GIT_TEST_CHECKOUT_WORKERS 2 ;; osx-gcc) + MAKEFLAGS="$MAKEFLAGS PYTHON_PATH=$(which python3)" CC=gcc ;; osx-clang) + MAKEFLAGS="$MAKEFLAGS PYTHON_PATH=$(which python2)" CC=clang ;; linux-clang) From patchwork Wed May 25 10:04:07 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= X-Patchwork-Id: 12860925 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id BF6E0C433F5 for ; Wed, 25 May 2022 10:05:39 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S242294AbiEYKFh (ORCPT ); Wed, 25 May 2022 06:05:37 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59390 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S242234AbiEYKFO (ORCPT ); Wed, 25 May 2022 06:05:14 -0400 Received: from mail-wr1-x42b.google.com (mail-wr1-x42b.google.com [IPv6:2a00:1450:4864:20::42b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 373526D965 for ; Wed, 25 May 2022 03:05:13 -0700 (PDT) Received: by mail-wr1-x42b.google.com with SMTP id u3so29410747wrg.3 for ; Wed, 25 May 2022 03:05:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=dmXO/rolLNsURpmIJrB/NpDYwTwLGCY+Fd7CvzKlmRg=; b=YrNczV9x+sLFKHwWfmkL22NHOoIYnGpRYOln0jLdroi3IY58DZYPwImPrTln0lDbs5 Ib5+E/izoguEI5Uf989b4pr5eCtzoo690tkCXl3kLiZvvX5qUVsaD5NBmcpWLtggYpvh UOIBvRoLCJm5tB2KfTHd9yRMr6G6hQciRTcpt/ZBt61mKOu34frQUY5cxyaMG+HgMose O9DMyV8AI4VPsl4aX6H4n0d5JxjFJ7p1Wu04+l0gAr1/AC2FEZbabChz1gLonoDQ1Hwg ArZT+9EFD9y+EVsxYfIaDsaCJKdi1qrhUklZtkEJ1HLOuQoRfat7zrKtFewuaYTOMvYb 36rw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=dmXO/rolLNsURpmIJrB/NpDYwTwLGCY+Fd7CvzKlmRg=; b=x7wnl887gcoZNm7/GNHTfDy+y2O1ncMLFluzFweEXH2rjJ3/XhTCEOWPeDjgWx6kJ+ 3hhCVQg205LKTIy8YDdFfuIrgs00Hf5lpBmCdKtg0oZNR/N7eN2U2GaXmyeaRFTo2Bxv VYWPuPZ07Y+gYOQrzkw7Zfwxq/30RShoJHG0MmITkmdrQCCVANfffVje0cTy5yJvlG/A ZNXJr9UQKKBDSxxk8jax1AGIHTz1LgcDhtrVJCa7ZQyGCc6KFd+jPbCd1wK+kU3NG4sN ru6xQEcKz2Pj0v0rT5Yubz24HR3IJP5zevc7O458h7MyfcCXrADyXD68/wJPK2L/yh+O V0rQ== X-Gm-Message-State: AOAM5316cEU+8Cq2Uz8UmXoau5BMDvCGgqwvI2yetqBSz3WLAIn24A24 +VFx5dp2492Aq4FxZi5YRzfTMh4KTKgOEw== X-Google-Smtp-Source: ABdhPJyVR60QlE450CMMX0PZ7ORO1C/tZHbqAxjmhMTzICiJTLYY0psQZDlb3OdjmLJkF/7dpEpviw== X-Received: by 2002:adf:fc01:0:b0:20c:ff9a:2c53 with SMTP id i1-20020adffc01000000b0020cff9a2c53mr25988875wrr.142.1653473111403; Wed, 25 May 2022 03:05:11 -0700 (PDT) Received: from vm.nix.is (vm.nix.is. [2a01:4f8:120:2468::2]) by smtp.gmail.com with ESMTPSA id o4-20020a05600002c400b0020d0cdbf7eesm1649452wry.111.2022.05.25.03.05.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 25 May 2022 03:05:10 -0700 (PDT) From: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= To: git@vger.kernel.org Cc: Junio C Hamano , Taylor Blau , =?utf-8?q?SZEDER_G=C3=A1bor?= , =?utf-8?b?xJBvw6Bu?= =?utf-8?b?IFRy4bqnbiBDw7RuZyBEYW5o?= , =?utf-8?q?Car?= =?utf-8?q?lo_Marcelo_Arenas_Bel=C3=B3n?= , Johannes Schindelin , Victoria Dye , Matheus Tavares , Lars Schneider , Eric Sunshine , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBC?= =?utf-8?b?amFybWFzb24=?= Subject: [PATCH v6 28/29] CI: don't use "set -x" in "ci/lib.sh" output Date: Wed, 25 May 2022 12:04:07 +0200 Message-Id: X-Mailer: git-send-email 2.36.1.1045.gf356b5617dd In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Remove the "set -x" from the "ci/lib.sh" output. Before preceding commits the logic in that file was much more complex, and likely to fail in some scenarios. Now we only task "ci/lib.sh" with setting various variables for subsequent steps in our jobs, so we can start emitting more tailored debugging output, which makes what it's doing easier to read. This change also changes the output of the "ci/print-test-failures.sh" script, since it's the only other user of "ci/lib-ci-type.sh". In that case it's also helpful to know what "$CI_TYPE" we're using, as that script doesn't "set -x" and will act differently depending on the $CI_TYPE. It's useful when developing this script to see what variables it *doesn't* set, but such output isn't useful to users, so let's hide the "SKIP" output behind a --debug option. Signed-off-by: Ævar Arnfjörð Bjarmason --- ci/lib.sh | 35 ++++++++++++++++++++++++++++++++--- 1 file changed, 32 insertions(+), 3 deletions(-) diff --git a/ci/lib.sh b/ci/lib.sh index f2f75901a9c..9e38c893890 100755 --- a/ci/lib.sh +++ b/ci/lib.sh @@ -1,5 +1,5 @@ #!/bin/sh -set -ex +set -e # Helper libraries . ${0%/*}/lib-ci-type.sh @@ -7,15 +7,22 @@ set -ex # Parse options mode_build= mode_test= +mode_debug= while test $# != 0 do case "$1" in --build) + echo "MODE: $1" >&2 mode_build=t ;; --test) + echo "MODE: $1" >&2 mode_test=t ;; + --debug) + echo "DEBUG: $1" >&2 + mode_debug=t + ;; -*) echo "error: invalid option: $1" >&2 exit 1 @@ -41,21 +48,32 @@ then exit 1 fi +# Show our configuration +echo "CONFIG: CI_TYPE=$CI_TYPE" >&2 +echo "CONFIG: jobname=$jobname" >&2 +echo "CONFIG: runs_on_pool=$runs_on_pool" >&2 +if test -n "$GITHUB_ENV" +then + echo "CONFIG: GITHUB_ENV=$GITHUB_ENV" >&2 +fi +echo >&2 + # Helper functions setenv () { + skip= while test $# != 0 do case "$1" in --build) if test -z "$mode_build" then - return 0 + skip=t fi ;; --test) if test -z "$mode_test" then - return 0 + skip=t fi ;; -*) @@ -73,10 +91,21 @@ setenv () { val=$2 shift 2 + if test -n "$skip" + then + if test -n "$mode_debug" + then + echo "SKIP: '$key=$val'" >&2 + fi + return 0 + fi + if test -n "$GITHUB_ENV" then echo "$key=$val" >>"$GITHUB_ENV" fi + + echo "SET: '$key=$val'" >&2 } # Clear variables that may come from the outside world. From patchwork Wed May 25 10:04:08 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= X-Patchwork-Id: 12860926 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D99FAC433EF for ; Wed, 25 May 2022 10:05:41 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S242299AbiEYKFk (ORCPT ); Wed, 25 May 2022 06:05:40 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59400 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S242247AbiEYKFO (ORCPT ); Wed, 25 May 2022 06:05:14 -0400 Received: from mail-wr1-x431.google.com (mail-wr1-x431.google.com [IPv6:2a00:1450:4864:20::431]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4253C6D966 for ; Wed, 25 May 2022 03:05:13 -0700 (PDT) Received: by mail-wr1-x431.google.com with SMTP id u3so29410403wrg.3 for ; Wed, 25 May 2022 03:05:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=1rbfG0lxqBh3TKDLdOki7ZzMwBV4WGgdDQKe9E14CKI=; b=O6cZ/kzzg0aAuHFFeb8UYsP0UfCkQcLcI7SuGjSeImk8DSysXTe3bTqJYzhPPKCGmv 551GG0WRgkszSTB9XuWqXSzMZ8NfmvlVruBsLal0ZJjwdRu20ZUeOQMlw3h5zwGQC5HK MDyTrPY9QBtAr5kPbgz7Wka9OxAQ1JWtloOHfxcKa8h8rb64hKD7ccIsCUXdLaR7Nz0b 3A1vH0+IKLL5GsOnKHCfK1yJ9o82NfV26k/NS9P+EdScv3qlx4m2E8cCq7as9Pb5ADtv rZJfXF5l+XD8w5LnUz4vxOWCNbFBuInetPipfjvqTWQvPxXGo9N0Hn44BQG+yEiL71OT bQfA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=1rbfG0lxqBh3TKDLdOki7ZzMwBV4WGgdDQKe9E14CKI=; b=LgsJ7WtLybAaXwqMxG53rTbsJqvy7Bgjkg/wvbrZAiHRVKT6t2i3CkjTh/L25Nu8JX FjG7pcq5oSzVFvJmTek3+XLyoy5SLqMiZY3YZYbu025R6nIwWhY62unxQ1Z5LT43l/3L MLVZ068Wtj3t6vYafqLP06NH/dLfJ1xPQ7saaz8cP+gFSfIiUZIp8pQ14/NY8NtUPwsC 9uvP9tkkoo1KPRNIokO1P0En/GcwNbHJRxfszQOEpB+yeGhhqKfyUmg8VtiXmGNZ+e4m Je1TMrdvi4zD4KsxRzxsor6PfdzuyQ9ZH//P1PqUAtT4odN+ixAXgTqSvdWb+Ol8E6gs 8xcA== X-Gm-Message-State: AOAM532THrH2/QkhzHUvrd/UtiMtsg+BXurLITpNAeWJGPoveQ0N3U8l kJ0VHYq9EteKDadzGJ6vcA4gBIUtycbG5A== X-Google-Smtp-Source: ABdhPJwNwfonorVw9mowhbAB0TvZbsiP70X2DjAttaCQygpyH0h2/ZDK2Sn+hRToiOQk+uJSJR6CQA== X-Received: by 2002:a5d:5641:0:b0:20d:c15:1b33 with SMTP id j1-20020a5d5641000000b0020d0c151b33mr25647953wrw.66.1653473112485; Wed, 25 May 2022 03:05:12 -0700 (PDT) Received: from vm.nix.is (vm.nix.is. [2a01:4f8:120:2468::2]) by smtp.gmail.com with ESMTPSA id o4-20020a05600002c400b0020d0cdbf7eesm1649452wry.111.2022.05.25.03.05.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 25 May 2022 03:05:11 -0700 (PDT) From: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= To: git@vger.kernel.org Cc: Junio C Hamano , Taylor Blau , =?utf-8?q?SZEDER_G=C3=A1bor?= , =?utf-8?b?xJBvw6Bu?= =?utf-8?b?IFRy4bqnbiBDw7RuZyBEYW5o?= , =?utf-8?q?Car?= =?utf-8?q?lo_Marcelo_Arenas_Bel=C3=B3n?= , Johannes Schindelin , Victoria Dye , Matheus Tavares , Lars Schneider , Eric Sunshine , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBC?= =?utf-8?b?amFybWFzb24=?= Subject: [PATCH v6 29/29] CI: make it easy to use ci/*.sh outside of CI Date: Wed, 25 May 2022 12:04:08 +0200 Message-Id: X-Mailer: git-send-email 2.36.1.1045.gf356b5617dd In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org In preceding commits the ci/.sh scripts have lost most of their CI-specific assumptions. Let's go even further and explicitly support running ci/lib.sh outside of CI. This was possible before by faking up enough CI-specific variables, but as shown in the new "help" output being added here using the ci/lib.sh to provide "CI-like" has now become trivial. The ci/print-test-failures.sh scripts can now be used outside of CI as well, the only GitHub CI-specific part is now guarded by a check that we'll pass if outside of GitHub CI. There's also a special-case here to not clobber $MAKEFLAGS in the environment if we're outside of CI, in case the user has e.g. "jN" or other flags to "make" that they'd prefer configured already. Using "ci/lib.sh" as a stand-alone script is much more useful if it doesn't hardcode NPROC=10, let's provide a poor shellscript replacement for the online_cpus() we have in thread-utils.c to cover the most common OS's. Helped-by: Eric Sunshine Signed-off-by: Ævar Arnfjörð Bjarmason --- ci/lib-ci-type.sh | 3 -- ci/lib-online_cpus.sh | 26 +++++++++++++ ci/lib.sh | 80 ++++++++++++++++++++++++++++++++++----- ci/print-test-failures.sh | 4 -- 4 files changed, 96 insertions(+), 17 deletions(-) create mode 100644 ci/lib-online_cpus.sh diff --git a/ci/lib-ci-type.sh b/ci/lib-ci-type.sh index 6f01fd9e5d9..09acab7aaec 100644 --- a/ci/lib-ci-type.sh +++ b/ci/lib-ci-type.sh @@ -3,7 +3,4 @@ if test "$GITHUB_ACTIONS" = "true" then CI_TYPE=github-actions -else - echo "Could not identify CI type" >&2 - exit 1 fi diff --git a/ci/lib-online_cpus.sh b/ci/lib-online_cpus.sh new file mode 100644 index 00000000000..7cf0cd042fa --- /dev/null +++ b/ci/lib-online_cpus.sh @@ -0,0 +1,26 @@ +#!/bin/sh + +# TODO: Ideally we'd compile t/helper/test-online-cpus.c early, but +# that currently presents a chicken & egg problem. We need this before +# we build (much of) anything. +online_cpus() { + NPROC= + + if command -v nproc >/dev/null + then + # GNU coreutils + NPROC=$(nproc) + elif command -v sysctl >/dev/null + then + # BSD & Mac OS X + NPROC=$(sysctl -n hw.ncpu) + elif test -n "$NUMBER_OF_PROCESSORS" + then + # Windows + NPROC="$NUMBER_OF_PROCESSORS" + else + NPROC=1 + fi + + echo $NPROC +} diff --git a/ci/lib.sh b/ci/lib.sh index 9e38c893890..80e89f89b7f 100755 --- a/ci/lib.sh +++ b/ci/lib.sh @@ -1,16 +1,55 @@ #!/bin/sh set -e +# Usage +CI_TYPE_HELP_COMMANDS=' + # run "make all test" like the "linux-leaks" job + (eval $(jobname=linux-leaks ci/lib.sh --all) && make test) + + # run "make all test" like the "linux-musl" job + (eval $(jobname=linux-musl ci/lib.sh --all) && make test) + + # run "make test" like the "linux-TEST-vars" job (uses various GIT_TEST_* modes) + make && (eval $(jobname=linux-TEST-vars ci/lib.sh --test) && make test) + + # run "make test" like the "linux-sha256" job + make && (eval $(jobname=linux-sha256 ci/lib.sh --test) && make test) +' + +CI_TYPE_HELP=" +Running $0 outside of CI? You can use ci/lib.sh to set up your +environment like a given CI job. E.g.: +$CI_TYPE_HELP_COMMANDS +Note that some of these (e.g. the linux-musl one) may not work as +expected due to the CI job configuring a platform that may not match +yours." + +usage() { + echo "usage: jobname= [runs_on_pool=] $0 (--build | --test)" >&2 + if test -z "$CI_TYPE" + then + echo "$CI_TYPE_HELP" >&2 + fi + exit 1 +} + # Helper libraries . ${0%/*}/lib-ci-type.sh # Parse options +mode_all= mode_build= mode_test= mode_debug= while test $# != 0 do case "$1" in + --all) + echo "MODE: $1" >&2 + mode_all=t + mode_build=t + mode_test=t + ;; --build) echo "MODE: $1" >&2 mode_build=t @@ -25,11 +64,11 @@ do ;; -*) echo "error: invalid option: $1" >&2 - exit 1 + usage ;; *) echo "error: invalid argument: $1" >&2 - exit 1 + usage ;; esac shift @@ -39,13 +78,13 @@ done if test -z "$jobname" then echo "error: must set a CI jobname in the environment" >&2 - exit 1 + usage fi -if test "$mode_test$mode_build" != "t" +if test "$mode_test$mode_build" != "t" && test -z "$mode_all" then echo "error: need one mode, e.g. --build or --test" >&2 - exit 1 + usage fi # Show our configuration @@ -65,13 +104,13 @@ setenv () { do case "$1" in --build) - if test -z "$mode_build" + if test -z "$mode_build$mode_all" then skip=t fi ;; --test) - if test -z "$mode_test" + if test -z "$mode_test$mode_all" then skip=t fi @@ -103,6 +142,10 @@ setenv () { if test -n "$GITHUB_ENV" then echo "$key=$val" >>"$GITHUB_ENV" + elif test -z "$CI_TYPE" + then + echo "$key=\"$val\"" + echo "export $key" fi echo "SET: '$key=$val'" >&2 @@ -113,10 +156,25 @@ CC= # How many jobs to run in parallel? NPROC=10 +case "$CI_TYPE" in +'') + . ${0%/*}/lib-online_cpus.sh + NPROC=$(online_cpus) + + if test -n "$MAKEFLAGS" + then + COMMON_MAKEFLAGS="$MAKEFLAGS" + else + COMMON_MAKEFLAGS=--jobs=$NPROC + fi + ;; +*) + # For "--test" we carry the MAKEFLAGS over from earlier steps, except + # in stand-alone jobs which will use $COMMON_MAKEFLAGS. + COMMON_MAKEFLAGS=--jobs=$NPROC + ;; +esac -# For "--test" we carry the MAKEFLAGS over from earlier steps, except -# in stand-alone jobs which will use $COMMON_MAKEFLAGS. -COMMON_MAKEFLAGS=--jobs=$NPROC # Clear MAKEFLAGS that may come from the outside world. MAKEFLAGS=$COMMON_MAKEFLAGS @@ -136,6 +194,8 @@ github-actions) GIT_TEST_OPTS="--no-chain-lint --no-bin-wrappers $GIT_TEST_OPTS" setenv --test GIT_TEST_OPTS "$GIT_TEST_OPTS" ;; +'') + ;; *) echo "Unhandled CI type: $CI_TYPE" >&2 exit 1 diff --git a/ci/print-test-failures.sh b/ci/print-test-failures.sh index 29f8c332eca..ec395c79ccd 100755 --- a/ci/print-test-failures.sh +++ b/ci/print-test-failures.sh @@ -36,10 +36,6 @@ do cp "${TEST_EXIT%.exit}.out" failed-test-artifacts/ tar czf failed-test-artifacts/"$test_name".trash.tar.gz "$trash_dir" ;; - *) - echo "Unhandled CI type: $CI_TYPE" >&2 - exit 1 - ;; esac fi done