From patchwork Tue Mar 1 10:24:41 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Schindelin X-Patchwork-Id: 12764505 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 37A5FC433F5 for ; Tue, 1 Mar 2022 10:25:01 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234238AbiCAKZj (ORCPT ); Tue, 1 Mar 2022 05:25:39 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42682 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233676AbiCAKZf (ORCPT ); Tue, 1 Mar 2022 05:25:35 -0500 Received: from mail-wr1-x42d.google.com (mail-wr1-x42d.google.com [IPv6:2a00:1450:4864:20::42d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8FD8C8EB56 for ; Tue, 1 Mar 2022 02:24:52 -0800 (PST) Received: by mail-wr1-x42d.google.com with SMTP id p9so19663399wra.12 for ; Tue, 01 Mar 2022 02:24:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=0+6ZofhIChFrhi0R37fNMY11dRkTRc5ZEYlAU1BlzXA=; b=HKOoA1qsVB0Pmmd2nakVusGY0CJEb2vWn1exkcMgS2adBFjN5r3rLfXfzqUlefL55l U96Pm3X9Ye/vfYnw5H3Wq6iMug91710MQFHTOBpaSm0ASPJwL9UUU4zcJT6KEFdXMmvj nwMWVu900wnIqK5gM19rKEqdUZ2TlV6tTxllVXC+wY3GggAeeNlK6NqKqQXuUanPrdqq GbC26KVrKNosvUzE+94IjHQj1e6MnVbzF8xGU8zOrFyRKgSVjUM3MnaLfLVUEA/68p/O el/Z46ak8MoaWZvxoQyvqwSFCzXfyVkJlh7qDi5XT2BYIJvE4b+vFekfIzlpn3/WVUVD rbaA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=0+6ZofhIChFrhi0R37fNMY11dRkTRc5ZEYlAU1BlzXA=; b=QkFQEDs4NM5OboI7Y4b1yHRIAJCPEobJapvsQ3DzFDkln2baQLcIkDc34wIUm1la1d +IMQ55VmkRc+ccH6KrxWNYvjLkX1tC611E6bFXRjSV3MqAC1Sgb4x2FSfebG0ykLBHck 7B6OUNfUzVhvFjPnBmyqYaaAmgGeuhv2MLhsTlbQchMrxepoaBuTnLMSopS3u+GWcHcA e3V4+RPrObXcCPL+wx6znYMa5l7UQw7zdaSXnrB0iWrNXTysqZWp4RFuwumOifkDArwz ZqcNnAPkzAcg5NvzCELJhT5Rb0ExycPoTCWbDBIIaC4+O6EGdNMYyDZdAi0y8p0R3H4d L5GA== X-Gm-Message-State: AOAM532WPGfkz+K+/wLajWrREnhGj+68ck/Eu7Xr+93gDXvDz/hoV+1t 1XkLY9fa44t3PyU6ss7G/j0sYSd1iww= X-Google-Smtp-Source: ABdhPJyL0MaY3DapzkoR/h9diW6xKXQLbZAfJtkR7TtW4VpjXNDWW/F4pWiPVG4jqCaKeT6UL3vkDQ== X-Received: by 2002:adf:f70a:0:b0:1ee:33bf:3864 with SMTP id r10-20020adff70a000000b001ee33bf3864mr17295416wrp.4.1646130291038; Tue, 01 Mar 2022 02:24:51 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id z5-20020a05600c0a0500b0037fa93193a8sm2755358wmp.44.2022.03.01.02.24.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 01 Mar 2022 02:24:50 -0800 (PST) Message-Id: In-Reply-To: References: Date: Tue, 01 Mar 2022 10:24:41 +0000 Subject: [PATCH v2 1/9] ci: fix code style Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Eric Sunshine , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7A=?= Bjarmason , Johannes Schindelin , Johannes Schindelin Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Johannes Schindelin From: Johannes Schindelin In b92cb86ea14 (travis-ci: check that all build artifacts are .gitignore-d, 2017-12-31), a function was introduced with a code style that is different from the surrounding code: it added the opening curly brace on its own line, when all the existing functions in the same file cuddle that brace on the same line as the function name. Let's make the code style consistent again. Signed-off-by: Johannes Schindelin --- ci/lib.sh | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/ci/lib.sh b/ci/lib.sh index 9d28ab50fb4..ebb502640fa 100755 --- a/ci/lib.sh +++ b/ci/lib.sh @@ -69,8 +69,7 @@ skip_good_tree () { exit 0 } -check_unignored_build_artifacts () -{ +check_unignored_build_artifacts () { ! git ls-files --other --exclude-standard --error-unmatch \ -- ':/*' 2>/dev/null || { From patchwork Tue Mar 1 10:24:42 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Schindelin X-Patchwork-Id: 12764506 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A530CC433EF for ; Tue, 1 Mar 2022 10:25:02 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232809AbiCAKZl (ORCPT ); Tue, 1 Mar 2022 05:25:41 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42730 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231495AbiCAKZg (ORCPT ); Tue, 1 Mar 2022 05:25:36 -0500 Received: from mail-wr1-x42b.google.com (mail-wr1-x42b.google.com [IPv6:2a00:1450:4864:20::42b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5E6FB8EB6C for ; Tue, 1 Mar 2022 02:24:53 -0800 (PST) Received: by mail-wr1-x42b.google.com with SMTP id b5so19750248wrr.2 for ; Tue, 01 Mar 2022 02:24:53 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=IALL0QMC0v8ysobvDm4KtZMwa6z5A/re01nuKPjfC2M=; b=lspBwPY6bzkDX7k1wvY3KMadqrcO60U3T58RSGi5tIcIqkVUG44n2VUBep2eWxD2vN 3tFaXnRyI3rSIL4YGpsW0tFvC9Tp6JADljWBOYGPfnEZJw6dO3QmANw/9quReC0kOq96 fKEBKasC6Du0EzGbQKvD+ZDbn/bIkj8y54GD/ImvX+1qREpXdmZQRZjrXNiRjo9LXVd0 YMvgRwHdhJ4mBriuUqkHeJZ1EUYRVLkHD5We+QoA+8l15HiN5kI7BoKBXMm31uChPj/I Ts/POmRywaY3zalwz8av0gmIcN5wMPQsqe2ahOcVWdHqwgcWIfYDzonr7lMvAjWVWq/M 2bzQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=IALL0QMC0v8ysobvDm4KtZMwa6z5A/re01nuKPjfC2M=; b=PDy6ZRscPVZm7oD8XeXqLBd2+OBEpaE2yBYqu1IKUD92eXubjA82KSTLIFVnHJx0cR dmbVPif2DVRBigW8ONjgIXaUhOhE/4SqKNpHweXuqNyn8Bk/PkWV7VK/1VPl6vtGu4be 8HfHg4RQ0KlFl+M2sMRfmzps6lYTSXM2cbznw04PxQx/XsH6nD6zBMPZ+rEDFYn7sA4y xzX8fVagojPA503rQy1QRc2uY2rTCwKbPuwSDndZ5uwa4ldIsjqZ6q0tmIOpXL4/4hI3 3B0S24ePO6XoTJCZqNldG2u6STR/ZFpEqbhy+csFaySR7KgvnSla4HrJxnlJ+6peim07 5ufg== X-Gm-Message-State: AOAM532G5RQ09wuklcOOmlF+5eRKHXeFwVRjQ5HVx6msmtTAdniMvNpE 1tjn1+5mgak+A3beOp1SpsCgJvP2OxM= X-Google-Smtp-Source: ABdhPJyHNzIQBVMJvQGEjuz60B+os7AEi4U2GfqZAnHVMM9DKuzy0GfV3qFkm4sNCWauk3Ros1jDCQ== X-Received: by 2002:adf:f48c:0:b0:1ea:9cb7:6c8b with SMTP id l12-20020adff48c000000b001ea9cb76c8bmr19291322wro.60.1646130291727; Tue, 01 Mar 2022 02:24:51 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id n4-20020a05600c4f8400b00380e45cd564sm2443667wmq.8.2022.03.01.02.24.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 01 Mar 2022 02:24:51 -0800 (PST) Message-Id: <42ff3e170bff42f9b9ca6adb433b3d7c2a39c729.1646130289.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Tue, 01 Mar 2022 10:24:42 +0000 Subject: [PATCH v2 2/9] ci/run-build-and-tests: take a more high-level view Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Eric Sunshine , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7A=?= Bjarmason , Johannes Schindelin , Johannes Schindelin Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Johannes Schindelin From: Johannes Schindelin In the web UI of GitHub workflows, failed runs are presented with the job step that failed auto-expanded. In the current setup, this is not helpful at all because that shows only the output of `prove`, which says which test failed, but not in what way. What would help understand the reader what went wrong is the verbose test output of the failed test. The logs of the failed runs do contain that verbose test output, but it is shown in the _next_ step (which is marked as succeeding, and is therefore _not_ auto-expanded). Anyone not intimately familiar with this would completely miss the verbose test output, being left mostly puzzled with the test failures. We are about to show the failed test cases' output in the _same_ step, so that the user has a much easier time to figure out what was going wrong. But first, we must partially revert the change that tried to improve the CI runs by combining the `Makefile` targets to build into a single `make` invocation. That might have sounded like a good idea at the time, but it does make it rather impossible for the CI script to determine whether the _build_ failed, or the _tests_. If the tests were run at all, that is. So let's go back to calling `make` for the build, and call `make test` separately so that we can easily detect that _that_ invocation failed, and react appropriately. Signed-off-by: Johannes Schindelin --- ci/run-build-and-tests.sh | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/ci/run-build-and-tests.sh b/ci/run-build-and-tests.sh index 280dda7d285..2818b3046ae 100755 --- a/ci/run-build-and-tests.sh +++ b/ci/run-build-and-tests.sh @@ -10,7 +10,7 @@ windows*) cmd //c mklink //j t\\.prove "$(cygpath -aw "$cache_dir/.prove")";; *) ln -s "$cache_dir/.prove" t/.prove;; esac -export MAKE_TARGETS="all test" +run_tests=t case "$jobname" in linux-gcc) @@ -41,14 +41,15 @@ pedantic) # Don't run the tests; we only care about whether Git can be # built. export DEVOPTS=pedantic - export MAKE_TARGETS=all + run_tests= ;; esac -# Any new "test" targets should not go after this "make", but should -# adjust $MAKE_TARGETS. Otherwise compilation-only targets above will -# start running tests. -make $MAKE_TARGETS +make +if test -n "$run_tests" +then + make test +fi check_unignored_build_artifacts save_good_tree From patchwork Tue Mar 1 10:24:43 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Schindelin X-Patchwork-Id: 12764507 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 48703C433F5 for ; Tue, 1 Mar 2022 10:25:04 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231495AbiCAKZm (ORCPT ); Tue, 1 Mar 2022 05:25:42 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42740 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234223AbiCAKZg (ORCPT ); Tue, 1 Mar 2022 05:25:36 -0500 Received: from mail-wr1-x42b.google.com (mail-wr1-x42b.google.com [IPv6:2a00:1450:4864:20::42b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5213C8EB50 for ; Tue, 1 Mar 2022 02:24:54 -0800 (PST) Received: by mail-wr1-x42b.google.com with SMTP id bk29so2396136wrb.4 for ; Tue, 01 Mar 2022 02:24:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=19KxTyVBW5pzW3ZGrYAIS/BhLki2fSSQ1gcZyZ5LL1U=; b=miKKfe2SN+gXxsco+nrNBW6CCk1OcUQZhPbfeVyhn3meLLSajx2vdsZNurHk87FP8F kRUmzTwUuXlVtEi5AD3n5lnxfCvxEeueplV1BkY+aom/YugPh5KRSNG7Xcg5h2YA4w8S E/kQemzLZx/eYL6uhRJ2DCy+OFxnuzNhpaFBgAqUuC5vVcaVCGn5YO/TDcerzBG6hIPP WNV8Iv2h2JRYuR2TUe3PuCXiTHlW1TfsR0izebpT+cJ6vqJEDjokVr2ZZ5Aj+FgXEKeT FcS3kVhe5Wp0rhc6cwAKD+odrojlBxXUhP15KEmW+0E592w+Xgmgbip5MhDHDMfnXHm3 Oogg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=19KxTyVBW5pzW3ZGrYAIS/BhLki2fSSQ1gcZyZ5LL1U=; b=NdE2slMYwVslQp7XFCeYLj4PW3uy+Mm12LEkGr450e9n6jBm3dZPzutpcpwfHcONX+ zROURyOMPHvrU0O1ewuToTjeuExH1F654XLZYg2/hBzuCNLXKIvF5lR/sKn6mAOx4qyJ 0NjTj6d5h3RVgNhw+c/Mxg26vJOu07Ur8CyISW1GsEBHUumBuW7fs3WnSa6vivWJHBg+ ezfwXltCSIVaNV71gjH1dcSZQ89bEHwEUIgNzfqF8d9pGWrwU8iOoXImJ6JSsU1z4sU9 thZyIIYMc+qhNxJLdx+UwhFNULiXOLTNZsrNV/4qchI+ELa2vJaKWlrIUSVycgEFEAU2 Y9zA== X-Gm-Message-State: AOAM532dP5CeANE5hNK4tzY/UfecujLnO0D8T7u2Cxco6GD7oMHKBoNp JoFjqsuLmxS8Fkdo8eTtRHh6gjC75Pg= X-Google-Smtp-Source: ABdhPJz8UvUmRPnMDIgMfZZnF7LTyBqE03OF9Iw0BJCuKqoZNmvD+C54fUrodm9EyaWTDnYO1QYerQ== X-Received: by 2002:adf:f5cb:0:b0:1ef:9483:9d4e with SMTP id k11-20020adff5cb000000b001ef94839d4emr9695199wrp.483.1646130292564; Tue, 01 Mar 2022 02:24:52 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id e15-20020a5d6d0f000000b001ef7dca67fasm12387601wrq.114.2022.03.01.02.24.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 01 Mar 2022 02:24:52 -0800 (PST) Message-Id: In-Reply-To: References: Date: Tue, 01 Mar 2022 10:24:43 +0000 Subject: [PATCH v2 3/9] ci: make it easier to find failed tests' logs in the GitHub workflow Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Eric Sunshine , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7A=?= Bjarmason , Johannes Schindelin , Johannes Schindelin Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Johannes Schindelin From: Johannes Schindelin When investigating a test failure, the time that matters most is the time it takes from getting aware of the failure to displaying the output of the failing test case. You currently have to know a lot of implementation details when investigating test failures in the CI runs. The first step is easy: the failed job is marked quite clearly, but when opening it, the failed step is expanded, which in our case is the one running `ci/run-build-and-tests.sh`. This step, most notably, only offers a high-level view of what went wrong: it prints the output of `prove` which merely tells the reader which test script failed. The actually interesting part is in the detailed log of said failed test script. But that log is shown in the CI run's step that runs `ci/print-test-failures.sh`. And that step is _not_ expanded in the web UI by default. It is even marked as "successful", which makes it very easy to miss that there is useful information hidden in there. Let's help the reader by showing the failed tests' detailed logs in the step that is expanded automatically, i.e. directly after the test suite failed. This also helps the situation where the _build_ failed and the `print-test-failures` step was executed under the assumption that the _test suite_ failed, and consequently failed to find any failed tests. An alternative way to implement this patch would be to source `ci/print-test-failures.sh` in the `handle_test_failures` function to show these logs. However, over the course of the next few commits, we want to introduce some grouping which would be harder to achieve that way (for example, we do want a leaner, and colored, preamble for each failed test script, and it would be trickier to accommodate the lack of nested groupings in GitHub workflows' output). Signed-off-by: Johannes Schindelin --- .github/workflows/main.yml | 12 ------------ ci/lib.sh | 23 +++++++++++++++++++++++ ci/run-build-and-tests.sh | 3 ++- ci/run-test-slice.sh | 3 ++- 4 files changed, 27 insertions(+), 14 deletions(-) diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml index c35200defb9..3fa88b78b6d 100644 --- a/.github/workflows/main.yml +++ b/.github/workflows/main.yml @@ -119,10 +119,6 @@ jobs: - name: test shell: bash run: . /etc/profile && ci/run-test-slice.sh ${{matrix.nr}} 10 - - name: ci/print-test-failures.sh - if: failure() - shell: bash - run: ci/print-test-failures.sh - name: Upload failed tests' directories if: failure() && env.FAILED_TEST_ARTIFACTS != '' uses: actions/upload-artifact@v2 @@ -204,10 +200,6 @@ jobs: env: NO_SVN_TESTS: 1 run: . /etc/profile && ci/run-test-slice.sh ${{matrix.nr}} 10 - - name: ci/print-test-failures.sh - if: failure() - shell: bash - run: ci/print-test-failures.sh - name: Upload failed tests' directories if: failure() && env.FAILED_TEST_ARTIFACTS != '' uses: actions/upload-artifact@v2 @@ -261,8 +253,6 @@ jobs: - uses: actions/checkout@v2 - run: ci/install-dependencies.sh - run: ci/run-build-and-tests.sh - - run: ci/print-test-failures.sh - if: failure() - name: Upload failed tests' directories if: failure() && env.FAILED_TEST_ARTIFACTS != '' uses: actions/upload-artifact@v2 @@ -292,8 +282,6 @@ jobs: - uses: actions/checkout@v1 - run: ci/install-docker-dependencies.sh - run: ci/run-build-and-tests.sh - - run: ci/print-test-failures.sh - if: failure() - name: Upload failed tests' directories if: failure() && env.FAILED_TEST_ARTIFACTS != '' uses: actions/upload-artifact@v1 diff --git a/ci/lib.sh b/ci/lib.sh index ebb502640fa..2b2c0932320 100755 --- a/ci/lib.sh +++ b/ci/lib.sh @@ -78,6 +78,10 @@ check_unignored_build_artifacts () { } } +handle_failed_tests () { + return 1 +} + # GitHub Action doesn't set TERM, which is required by tput export TERM=${TERM:-dumb} @@ -123,6 +127,25 @@ then CI_JOB_ID="$GITHUB_RUN_ID" CC="${CC:-gcc}" DONT_SKIP_TAGS=t + handle_failed_tests () { + mkdir -p t/failed-test-artifacts + echo "FAILED_TEST_ARTIFACTS=t/failed-test-artifacts" >>$GITHUB_ENV + + for test_exit in t/test-results/*.exit + do + test 0 != "$(cat "$test_exit")" || continue + + test_name="${test_exit%.exit}" + test_name="${test_name##*/}" + printf "\\e[33m\\e[1m=== Failed test: ${test_name} ===\\e[m\\n" + cat "t/test-results/$test_name.out" + + trash_dir="t/trash directory.$test_name" + cp "t/test-results/$test_name.out" t/failed-test-artifacts/ + tar czf t/failed-test-artifacts/"$test_name".trash.tar.gz "$trash_dir" + done + return 1 + } cache_dir="$HOME/none" diff --git a/ci/run-build-and-tests.sh b/ci/run-build-and-tests.sh index 2818b3046ae..1ede75e5556 100755 --- a/ci/run-build-and-tests.sh +++ b/ci/run-build-and-tests.sh @@ -48,7 +48,8 @@ esac make if test -n "$run_tests" then - make test + make test || + handle_failed_tests fi check_unignored_build_artifacts diff --git a/ci/run-test-slice.sh b/ci/run-test-slice.sh index f8c2c3106a2..63358c23e11 100755 --- a/ci/run-test-slice.sh +++ b/ci/run-test-slice.sh @@ -12,6 +12,7 @@ esac make --quiet -C t T="$(cd t && ./helper/test-tool path-utils slice-tests "$1" "$2" t[0-9]*.sh | - tr '\n' ' ')" + tr '\n' ' ')" || +handle_failed_tests check_unignored_build_artifacts From patchwork Tue Mar 1 10:24:44 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Schindelin X-Patchwork-Id: 12764508 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 214CDC433F5 for ; Tue, 1 Mar 2022 10:25:08 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234246AbiCAKZq (ORCPT ); Tue, 1 Mar 2022 05:25:46 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42764 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234227AbiCAKZg (ORCPT ); Tue, 1 Mar 2022 05:25:36 -0500 Received: from mail-wr1-x42d.google.com (mail-wr1-x42d.google.com [IPv6:2a00:1450:4864:20::42d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 01A325A082 for ; Tue, 1 Mar 2022 02:24:55 -0800 (PST) Received: by mail-wr1-x42d.google.com with SMTP id d3so19732809wrf.1 for ; Tue, 01 Mar 2022 02:24:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=nJre/5YVUl/6Rb6oFRIeWPosN+8SFbb9ia2xvqQ32t4=; b=H/jjCseVDS7CNF3UyswaXUh1jTozYO7ZV5D2MXoHvrfMNIWHYvgZM1HbkcFwJn0xSt Wrl4bX1Z2pmHIy/IbP+cyyJvsG82ASNLxDMXtZ5QrbwkY8waVKLwdRhOB+SHki/4B8hO PIPeMA9gASplh7AxRvxvl9zIKYksP67Be+O5/JhzrVQIJamOW8I+7k9lM3DgOhaQTqgE usRJp1XZNrbYpbd18NjOrEyC/kKlcuvLPKzXKFKZqHBJDgP8vjwzH0WW7xoR8yhUIQuI f8QMKfEHcKtI1jb67h8wCD7JztpWgYuxgDi9gTaQwGcPrjqUgmiOpK58EbfFWFRrmbQM YgZQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=nJre/5YVUl/6Rb6oFRIeWPosN+8SFbb9ia2xvqQ32t4=; b=4+rX4zwc8J4g2bxv7zYvBI/f0XTXzO/AqOWNnvYbdJhm5NfyJI+2dEwhgOUc7YwXNj bnpJxvIjeZhtVTLTU7Ba2L7CLkfD9BgTDJ9IDcwCWM62v+ikXAzYnLWreoStIPj/pWmS jjVk8DHje2ZIUJfV1502FjN+a7XKl09rCTYY4Z05F7xWvR9jijfz4zz9WH/EsHEEZxVR 0mnEDM5tdJ8SMGF1jRZc/412HdJmwwxyEoxuOg74iyagKvxA0FwmP/6UXwngBZcykuP0 JDbcD0RvPtyhROCFVPATx+mdP6RwFW92FaD8mqgVOR2af+Rq9qwyq7jsHN2vaizdaJtZ M9fw== X-Gm-Message-State: AOAM530uAbY2FDfuFt5SPe+YdhuYizLqZCjzHqVKCCmDvSCpv2pG4au7 kwCTzDKgve5U4Tyf7GuW6HZFaZxUUvI= X-Google-Smtp-Source: ABdhPJwatdej7wexb4CWUNBaqhUQi+f7emwqOYE7CXIXp322Y34eNo5Rsao+pdfpwM8xEbFyVoAYiQ== X-Received: by 2002:a05:6000:1242:b0:1ef:846b:290a with SMTP id j2-20020a056000124200b001ef846b290amr11320120wrx.235.1646130293412; Tue, 01 Mar 2022 02:24:53 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id r1-20020a05600c2f0100b00381821b4dbdsm1985920wmn.6.2022.03.01.02.24.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 01 Mar 2022 02:24:53 -0800 (PST) Message-Id: In-Reply-To: References: Date: Tue, 01 Mar 2022 10:24:44 +0000 Subject: [PATCH v2 4/9] ci/run-build-and-tests: add some structure to the GitHub workflow output Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Eric Sunshine , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7A=?= Bjarmason , Johannes Schindelin , Johannes Schindelin Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Johannes Schindelin From: Johannes Schindelin The current output of Git's GitHub workflow can be quite confusing, especially for contributors new to the project. To make it more helpful, let's introduce some collapsible grouping. Initially, readers will see the high-level view of what actually happened (did the build fail, or the test suite?). To drill down, the respective group can be expanded. Note: sadly, workflow output currently cannot contain any nested groups (see https://github.com/actions/runner/issues/802 for details), therefore we take pains to ensure to end any previous group before starting a new one. Signed-off-by: Johannes Schindelin --- ci/lib.sh | 56 ++++++++++++++++++++++++++++++++++----- ci/run-build-and-tests.sh | 4 +-- ci/run-test-slice.sh | 2 +- 3 files changed, 52 insertions(+), 10 deletions(-) diff --git a/ci/lib.sh b/ci/lib.sh index 2b2c0932320..2a1b22db12a 100755 --- a/ci/lib.sh +++ b/ci/lib.sh @@ -1,5 +1,50 @@ # Library of functions shared by all CI scripts +if test true != "$GITHUB_ACTIONS" +then + begin_group () { :; } + end_group () { :; } + + group () { + shift + "$@" + } + set -x +else + begin_group () { + need_to_end_group=t + echo "::group::$1" >&2 + set -x + } + + end_group () { + test -n "$need_to_end_group" || return 0 + set +x + need_to_end_group= + echo '::endgroup::' >&2 + } + trap end_group EXIT + + group () { + set +x + begin_group "$1" + shift + "$@" + res=$? + end_group + return $res + } + + begin_group "CI setup" +fi + +# Set 'exit on error' for all CI scripts to let the caller know that +# something went wrong. +# +# We already enabled tracing executed commands earlier. This helps by showing +# how # environment variables are set and and dependencies are installed. +set -e + skip_branch_tip_with_tag () { # Sometimes, a branch is pushed at the same time the tag that points # at the same commit as the tip of the branch is pushed, and building @@ -88,12 +133,6 @@ export TERM=${TERM:-dumb} # Clear MAKEFLAGS that may come from the outside world. export MAKEFLAGS= -# Set 'exit on error' for all CI scripts to let the caller know that -# something went wrong. -# Set tracing executed commands, primarily setting environment variables -# and installing dependencies. -set -ex - if test -n "$SYSTEM_COLLECTIONURI" || test -n "$SYSTEM_TASKDEFINITIONSURI" then CI_TYPE=azure-pipelines @@ -138,7 +177,7 @@ then test_name="${test_exit%.exit}" test_name="${test_name##*/}" printf "\\e[33m\\e[1m=== Failed test: ${test_name} ===\\e[m\\n" - cat "t/test-results/$test_name.out" + group "Failed test: $test_name" cat "t/test-results/$test_name.out" trash_dir="t/trash directory.$test_name" cp "t/test-results/$test_name.out" t/failed-test-artifacts/ @@ -234,3 +273,6 @@ linux-leaks) esac MAKEFLAGS="$MAKEFLAGS CC=${CC:-cc}" + +end_group +set -x diff --git a/ci/run-build-and-tests.sh b/ci/run-build-and-tests.sh index 1ede75e5556..7abfa00adc0 100755 --- a/ci/run-build-and-tests.sh +++ b/ci/run-build-and-tests.sh @@ -45,10 +45,10 @@ pedantic) ;; esac -make +group Build make if test -n "$run_tests" then - make test || + group "Run tests" make test || handle_failed_tests fi check_unignored_build_artifacts diff --git a/ci/run-test-slice.sh b/ci/run-test-slice.sh index 63358c23e11..a3c67956a8d 100755 --- a/ci/run-test-slice.sh +++ b/ci/run-test-slice.sh @@ -10,7 +10,7 @@ windows*) cmd //c mklink //j t\\.prove "$(cygpath -aw "$cache_dir/.prove")";; *) ln -s "$cache_dir/.prove" t/.prove;; esac -make --quiet -C t T="$(cd t && +group "Run tests" make --quiet -C t T="$(cd t && ./helper/test-tool path-utils slice-tests "$1" "$2" t[0-9]*.sh | tr '\n' ' ')" || handle_failed_tests From patchwork Tue Mar 1 10:24:45 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Schindelin X-Patchwork-Id: 12764509 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1B2DDC433EF for ; Tue, 1 Mar 2022 10:25:09 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231517AbiCAKZr (ORCPT ); Tue, 1 Mar 2022 05:25:47 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42862 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234220AbiCAKZi (ORCPT ); Tue, 1 Mar 2022 05:25:38 -0500 Received: from mail-wr1-x434.google.com (mail-wr1-x434.google.com [IPv6:2a00:1450:4864:20::434]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0539A8EB67 for ; Tue, 1 Mar 2022 02:24:56 -0800 (PST) Received: by mail-wr1-x434.google.com with SMTP id b5so19750475wrr.2 for ; Tue, 01 Mar 2022 02:24:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=x3+QTZjSRvoWaKsczsulWNBMqebhwXtC5ksMNH2RWTM=; b=c12kGT89EJFaqCoVR4OaFKHMdjaoRfyDNWm/Jg57Ej31USV1/DvC9HqxeVpU8HSCJQ MLMRuPGQYMPStxOt33Etzztwu/wNTzzylZ506so2szAWRFySHmHST4rsZGFVCk1x8e03 /wwOqPu/X+Bf96xyWfvyjvDXxfAVwimiUMD3/9WihR2vcMfBEv/krbNXhS8YFfjBy3qy +2MQwhIM/vwfWS+8npVeYhGn+2N2QUn2hzLOYV0+JLcKUAXy9TOqffHG0Ndez+z/WSuY 5glKSNpnev0r7ks7SjUFU9hMFTEsMMU/kh0U7TboLldPe8OZcbRlfi66mAHHd/wyaFIG BI9g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=x3+QTZjSRvoWaKsczsulWNBMqebhwXtC5ksMNH2RWTM=; b=Wx2yT3prflBmEqLoQnItqkxV6Y20STUSMw4YbWvUxhoWSDdSjiIBjfV/DCqyPcZMre +Arpd1a1LlDPAVynRfgzrO3kLmT/wZsAx8vq2jBMoYCgjsQkUclHqed9DcgMo5tVZeyD MW7Cd245F1pOZOrtgocJpxWJ3d/sTb8OBBMknTV/Za/4bc1Bpn4sL6IfgbnBLdGd5Ryf EKVtyhyKgLm2Vr3vvwtrZOhtbG3uVytkxph464bqubrD1khFu0UFhzVCoa7HGeZbesiB b7qCRTlgHQ2JpgiL9R5f5FS83OH2oLvZReMIsQUmDk2SiiZzUSsDiirjtecuIdLCXdka 67BQ== X-Gm-Message-State: AOAM531/wI+XRr5ls2ALyQZJ4shqBjyCW2Khl6aMN7Hq+ZyAF7z5IJPG 1cQdBMqAIeXtivYdbV1Svns5is8pQAI= X-Google-Smtp-Source: ABdhPJxWHgH7uTuvvi4Nb2qhr1GXFyt0zkxv7SJf7/bc7O/OBD8vGkb2Bck5OTcDZCIDtjbCsgtVvQ== X-Received: by 2002:a05:6000:11c9:b0:1ed:b92d:8c68 with SMTP id i9-20020a05600011c900b001edb92d8c68mr18862482wrx.692.1646130294269; Tue, 01 Mar 2022 02:24:54 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id f11-20020a7bcc0b000000b0037e0c362b6dsm1999563wmh.31.2022.03.01.02.24.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 01 Mar 2022 02:24:53 -0800 (PST) Message-Id: <9eda6574313c7316d7f72e06d642bfbe6edd4492.1646130289.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Tue, 01 Mar 2022 10:24:45 +0000 Subject: [PATCH v2 5/9] tests: refactor --write-junit-xml code Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Eric Sunshine , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7A=?= Bjarmason , Johannes Schindelin , Johannes Schindelin Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Johannes Schindelin From: Johannes Schindelin The code writing JUnit XML is interspersed directly with all the code in `t/test-lib.sh`, and it is therefore not only ill-separated, but introducing yet another output format would make the situation even worse. Let's introduce an abstraction layer by hiding the JUnit XML code behind four new functions that are supposed to be called before and after each test and test case. This is not just an academic exercise, refactoring for refactoring's sake. We _actually_ want to introduce such a new output format, to make it substantially easier to diagnose test failures in our GitHub workflow, therefore we do need this refactoring. This commit is best viewed with `git show --color-moved --color-moved-ws=allow-indentation-change `. Signed-off-by: Johannes Schindelin --- t/test-lib-junit.sh | 126 ++++++++++++++++++++++++++++++++++++++++++++ t/test-lib.sh | 124 ++++++------------------------------------- 2 files changed, 142 insertions(+), 108 deletions(-) create mode 100644 t/test-lib-junit.sh diff --git a/t/test-lib-junit.sh b/t/test-lib-junit.sh new file mode 100644 index 00000000000..9d55d74d764 --- /dev/null +++ b/t/test-lib-junit.sh @@ -0,0 +1,126 @@ +# Library of functions to format test scripts' output in JUnit XML +# format, to support Git's test suite result to be presented in an +# easily digestible way on Azure Pipelines. +# +# Copyright (c) 2022 Johannes Schindelin +# +# This program is free software: you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation, either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see http://www.gnu.org/licenses/ . +# +# The idea is for `test-lib.sh` to source this file when the user asks +# for JUnit XML; these functions will then override (empty) functions +# that are are called at the appropriate times during the test runs. + +start_test_output () { + junit_xml_dir="$TEST_OUTPUT_DIRECTORY/out" + mkdir -p "$junit_xml_dir" + junit_xml_base=${1##*/} + junit_xml_path="$junit_xml_dir/TEST-${junit_xml_base%.sh}.xml" + junit_attrs="name=\"${junit_xml_base%.sh}\"" + junit_attrs="$junit_attrs timestamp=\"$(TZ=UTC \ + date +%Y-%m-%dT%H:%M:%S)\"" + write_junit_xml --truncate "" " " + junit_suite_start=$(test-tool date getnanos) + if test -n "$GIT_TEST_TEE_OUTPUT_FILE" + then + GIT_TEST_TEE_OFFSET=0 + fi +} + +start_test_case_output () { + junit_start=$(test-tool date getnanos) +} + +finalize_test_case_output () { + test_case_result=$1 + shift + case "$test_case_result" in + ok) + set "$*" + ;; + failure) + junit_insert="" + junit_insert="$junit_insert $(xml_attr_encode \ + "$(if test -n "$GIT_TEST_TEE_OUTPUT_FILE" + then + test-tool path-utils skip-n-bytes \ + "$GIT_TEST_TEE_OUTPUT_FILE" $GIT_TEST_TEE_OFFSET + else + printf '%s\n' "$@" | sed 1d + fi)")" + junit_insert="$junit_insert" + if test -n "$GIT_TEST_TEE_OUTPUT_FILE" + then + junit_insert="$junit_insert$(xml_attr_encode \ + "$(cat "$GIT_TEST_TEE_OUTPUT_FILE")")" + fi + set "$1" " $junit_insert" + ;; + fixed) + set "$* (breakage fixed)" + ;; + broken) + set "$* (known breakage)" + ;; + skip) + message="$(xml_attr_encode "$skipped_reason")" + set "$1" " " + ;; + esac + + junit_attrs="name=\"$(xml_attr_encode "$this_test.$test_count $1")\"" + shift + junit_attrs="$junit_attrs classname=\"$this_test\"" + junit_attrs="$junit_attrs time=\"$(test-tool \ + date getnanos $junit_start)\"" + write_junit_xml "$(printf '%s\n' \ + " " "$@" " ")" + junit_have_testcase=t +} + +finalize_test_output () { + if test -n "$junit_xml_path" + then + test -n "$junit_have_testcase" || { + junit_start=$(test-tool date getnanos) + write_junit_xml_testcase "all tests skipped" + } + + # adjust the overall time + junit_time=$(test-tool date getnanos $junit_suite_start) + sed -e "s/\(]*/& time=\"$junit_time\"/" \ + -e '/^ *<\/testsuite/d' \ + <"$junit_xml_path" >"$junit_xml_path.new" + mv "$junit_xml_path.new" "$junit_xml_path" + + write_junit_xml " " "" + write_junit_xml= + fi +} + +write_junit_xml () { + case "$1" in + --truncate) + >"$junit_xml_path" + junit_have_testcase= + shift + ;; + esac + printf '%s\n' "$@" >>"$junit_xml_path" +} + +xml_attr_encode () { + printf '%s\n' "$@" | test-tool xml-encode +} diff --git a/t/test-lib.sh b/t/test-lib.sh index 0f7a137c7d8..e13e1cb9124 100644 --- a/t/test-lib.sh +++ b/t/test-lib.sh @@ -107,6 +107,12 @@ mark_option_requires_arg () { store_arg_to=$2 } +# These functions can be overridden e.g. to output JUnit XML +start_test_output () { :; } +start_test_case_output () { :; } +finalize_test_case_output () { :; } +finalize_test_output () { :; } + parse_option () { local opt="$1" @@ -166,7 +172,7 @@ parse_option () { tee=t ;; --write-junit-xml) - write_junit_xml=t + . "$TEST_DIRECTORY/test-lib-junit.sh" ;; --stress) stress=t ;; @@ -613,7 +619,7 @@ exec 6<&0 exec 7>&2 _error_exit () { - finalize_junit_xml + finalize_test_output GIT_EXIT_OK=t exit 1 } @@ -723,35 +729,13 @@ trap '{ code=$?; set +x; } 2>/dev/null; exit $code' INT TERM HUP # the test_expect_* functions instead. test_ok_ () { - if test -n "$write_junit_xml" - then - write_junit_xml_testcase "$*" - fi + finalize_test_case_output ok "$@" test_success=$(($test_success + 1)) say_color "" "ok $test_count - $@" } test_failure_ () { - if test -n "$write_junit_xml" - then - junit_insert="" - junit_insert="$junit_insert $(xml_attr_encode \ - "$(if test -n "$GIT_TEST_TEE_OUTPUT_FILE" - then - test-tool path-utils skip-n-bytes \ - "$GIT_TEST_TEE_OUTPUT_FILE" $GIT_TEST_TEE_OFFSET - else - printf '%s\n' "$@" | sed 1d - fi)")" - junit_insert="$junit_insert" - if test -n "$GIT_TEST_TEE_OUTPUT_FILE" - then - junit_insert="$junit_insert$(xml_attr_encode \ - "$(cat "$GIT_TEST_TEE_OUTPUT_FILE")")" - fi - write_junit_xml_testcase "$1" " $junit_insert" - fi + finalize_test_case_output failure "$@" test_failure=$(($test_failure + 1)) say_color error "not ok $test_count - $1" shift @@ -760,19 +744,13 @@ test_failure_ () { } test_known_broken_ok_ () { - if test -n "$write_junit_xml" - then - write_junit_xml_testcase "$* (breakage fixed)" - fi + finalize_test_case_output fixed "$@" test_fixed=$(($test_fixed+1)) say_color error "ok $test_count - $@ # TODO known breakage vanished" } test_known_broken_failure_ () { - if test -n "$write_junit_xml" - then - write_junit_xml_testcase "$* (known breakage)" - fi + finalize_test_case_output broken "$@" test_broken=$(($test_broken+1)) say_color warn "not ok $test_count - $@ # TODO known breakage" } @@ -1049,10 +1027,7 @@ test_start_ () { test_count=$(($test_count+1)) maybe_setup_verbose maybe_setup_valgrind - if test -n "$write_junit_xml" - then - junit_start=$(test-tool date getnanos) - fi + start_test_case_output } test_finish_ () { @@ -1103,12 +1078,7 @@ test_skip () { case "$to_skip" in t) - if test -n "$write_junit_xml" - then - message="$(xml_attr_encode "$skipped_reason")" - write_junit_xml_testcase "$1" \ - " " - fi + finalize_test_case_output skip "$@" say_color skip "ok $test_count # skip $1 ($skipped_reason)" : true @@ -1124,53 +1094,6 @@ test_at_end_hook_ () { : } -write_junit_xml () { - case "$1" in - --truncate) - >"$junit_xml_path" - junit_have_testcase= - shift - ;; - esac - printf '%s\n' "$@" >>"$junit_xml_path" -} - -xml_attr_encode () { - printf '%s\n' "$@" | test-tool xml-encode -} - -write_junit_xml_testcase () { - junit_attrs="name=\"$(xml_attr_encode "$this_test.$test_count $1")\"" - shift - junit_attrs="$junit_attrs classname=\"$this_test\"" - junit_attrs="$junit_attrs time=\"$(test-tool \ - date getnanos $junit_start)\"" - write_junit_xml "$(printf '%s\n' \ - " " "$@" " ")" - junit_have_testcase=t -} - -finalize_junit_xml () { - if test -n "$write_junit_xml" && test -n "$junit_xml_path" - then - test -n "$junit_have_testcase" || { - junit_start=$(test-tool date getnanos) - write_junit_xml_testcase "all tests skipped" - } - - # adjust the overall time - junit_time=$(test-tool date getnanos $junit_suite_start) - sed -e "s/\(]*/& time=\"$junit_time\"/" \ - -e '/^ *<\/testsuite/d' \ - <"$junit_xml_path" >"$junit_xml_path.new" - mv "$junit_xml_path.new" "$junit_xml_path" - - write_junit_xml " " "" - write_junit_xml= - fi -} - test_atexit_cleanup=: test_atexit_handler () { # In a succeeding test script 'test_atexit_handler' is invoked @@ -1193,7 +1116,7 @@ test_done () { # removed, so the commands can access pidfiles and socket files. test_atexit_handler - finalize_junit_xml + finalize_test_output if test -z "$HARNESS_ACTIVE" then @@ -1484,22 +1407,7 @@ fi # in subprocesses like git equals our $PWD (for pathname comparisons). cd -P "$TRASH_DIRECTORY" || exit 1 -if test -n "$write_junit_xml" -then - junit_xml_dir="$TEST_OUTPUT_DIRECTORY/out" - mkdir -p "$junit_xml_dir" - junit_xml_base=${0##*/} - junit_xml_path="$junit_xml_dir/TEST-${junit_xml_base%.sh}.xml" - junit_attrs="name=\"${junit_xml_base%.sh}\"" - junit_attrs="$junit_attrs timestamp=\"$(TZ=UTC \ - date +%Y-%m-%dT%H:%M:%S)\"" - write_junit_xml --truncate "" " " - junit_suite_start=$(test-tool date getnanos) - if test -n "$GIT_TEST_TEE_OUTPUT_FILE" - then - GIT_TEST_TEE_OFFSET=0 - fi -fi +start_test_output "$0" # Convenience # A regexp to match 5 and 35 hexdigits From patchwork Tue Mar 1 10:24:46 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Schindelin X-Patchwork-Id: 12764510 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 67026C433F5 for ; Tue, 1 Mar 2022 10:25:10 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234249AbiCAKZt (ORCPT ); Tue, 1 Mar 2022 05:25:49 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42894 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234221AbiCAKZi (ORCPT ); Tue, 1 Mar 2022 05:25:38 -0500 Received: from mail-wm1-x32a.google.com (mail-wm1-x32a.google.com [IPv6:2a00:1450:4864:20::32a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8E82575E53 for ; Tue, 1 Mar 2022 02:24:56 -0800 (PST) Received: by mail-wm1-x32a.google.com with SMTP id c192so7236729wma.4 for ; Tue, 01 Mar 2022 02:24:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=iGAE3SZGJU7eojvkhG0qydiDez1X+uPg8UYp38uGbF0=; b=Y8uIFiaLmQSFZnLFqP/Iu7ayQaHg3tegF1DHJ4CmHXW+36x+w+KU0f+uSFCecTk8zW hhNWLsk9meW26TIZaA/13PCtPZKRmMtP1o88LDbIFW3EMqRM8Lwd/tlchBrPwWuAzxrl J5cObSccvkHJaQfSlY+LgyRngSGO1tfHp1L87/w5XrRxgzB6CulO9r4SL7AoVRY92ZGy uMfPDGmRIagectVDFOISLE3LaRyxSfuIpj1+KD9qDSeAwTDHVkiDJt6uTi1vS4UwVyNR 2V4hcEwECs9pn0Lh4/Oi5+wJz0qFH6LHWu/m2PrYznHPRgWFg2GWjCD88EGAYYuu+scF dEyA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=iGAE3SZGJU7eojvkhG0qydiDez1X+uPg8UYp38uGbF0=; b=M2nlbvuOBxq82H7NNjVmfWAM4EDqyR16vJo/RU2vfwtXj/oNibXIyKLl04InVB1rSk 4d6dP6j144aIlqiRCKoWZE3fwW+dQxH/kKDebx+txM2bI2Mfggq2kY3KVcp4SF7wpL0Y lw/8pDZs/dqJrUTSYAiHJWZmKr/NT1CvZhB3ltAAyAVNGB30Upe61DGvEBdoXIlQjXA4 /F2BhFach9tUPkb3Z6RMq3gU9s4BKPhnu9kfrp/6p9AoNOCFNNPhKsYcNT6QA55saJPG 76DbLza+B2Ewgv+Ht/h3WYVBB6CXS5SXZGali1bgN64wBMMYPk5tEXemM3G/+lEJ6jOD zNRQ== X-Gm-Message-State: AOAM530Qym+b4MZtr7xJrocZdhMCHsPm2tRFHwFQCE6t4t0V8vofDlP0 MfRgkFjbRcjICiz4bVMrDbIPhaAJ6CY= X-Google-Smtp-Source: ABdhPJzBFE2BgeVLK2ZLEwVPfMZOHGrvUNb2I5IR/etGQ7R/nFwA2sfVgQ9sbNrF2nxGv6WfDCkYww== X-Received: by 2002:a05:600c:190d:b0:381:6ea1:8cc with SMTP id j13-20020a05600c190d00b003816ea108ccmr5838630wmq.118.1646130295004; Tue, 01 Mar 2022 02:24:55 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id u23-20020a7bcb17000000b0037bdfa1665asm2764747wmj.18.2022.03.01.02.24.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 01 Mar 2022 02:24:54 -0800 (PST) Message-Id: In-Reply-To: References: Date: Tue, 01 Mar 2022 10:24:46 +0000 Subject: [PATCH v2 6/9] test(junit): avoid line feeds in XML attributes Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Eric Sunshine , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7A=?= Bjarmason , Johannes Schindelin , Johannes Schindelin Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Johannes Schindelin From: Johannes Schindelin In the test case's output, we do want newline characters, but in the XML attributes we do not want them. However, the `xml_attr_encode` function always adds a Line Feed at the end (which are then encoded as ` `, even for XML attributes. This seems not to faze Azure Pipelines' XML parser, but it still is incorrect, so let's fix it. Signed-off-by: Johannes Schindelin --- t/test-lib-junit.sh | 14 ++++++++++---- 1 file changed, 10 insertions(+), 4 deletions(-) diff --git a/t/test-lib-junit.sh b/t/test-lib-junit.sh index 9d55d74d764..c959183c7e2 100644 --- a/t/test-lib-junit.sh +++ b/t/test-lib-junit.sh @@ -50,7 +50,7 @@ finalize_test_case_output () { ;; failure) junit_insert="" + junit_insert="$junit_insert $(xml_attr_encode --no-lf "$1")\">" junit_insert="$junit_insert $(xml_attr_encode \ "$(if test -n "$GIT_TEST_TEE_OUTPUT_FILE" then @@ -74,12 +74,12 @@ finalize_test_case_output () { set "$* (known breakage)" ;; skip) - message="$(xml_attr_encode "$skipped_reason")" + message="$(xml_attr_encode --no-lf "$skipped_reason")" set "$1" " " ;; esac - junit_attrs="name=\"$(xml_attr_encode "$this_test.$test_count $1")\"" + junit_attrs="name=\"$(xml_attr_encode --no-lf "$this_test.$test_count $1")\"" shift junit_attrs="$junit_attrs classname=\"$this_test\"" junit_attrs="$junit_attrs time=\"$(test-tool \ @@ -122,5 +122,11 @@ write_junit_xml () { } xml_attr_encode () { - printf '%s\n' "$@" | test-tool xml-encode + if test "x$1" = "x--no-lf" + then + shift + printf '%s' "$*" | test-tool xml-encode + else + printf '%s\n' "$@" | test-tool xml-encode + fi } From patchwork Tue Mar 1 10:24:47 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Schindelin X-Patchwork-Id: 12764511 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 09974C433EF for ; Tue, 1 Mar 2022 10:25:12 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234229AbiCAKZu (ORCPT ); Tue, 1 Mar 2022 05:25:50 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43066 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234240AbiCAKZj (ORCPT ); Tue, 1 Mar 2022 05:25:39 -0500 Received: from mail-wr1-x435.google.com (mail-wr1-x435.google.com [IPv6:2a00:1450:4864:20::435]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9F2898EB4D for ; Tue, 1 Mar 2022 02:24:57 -0800 (PST) Received: by mail-wr1-x435.google.com with SMTP id n14so19673614wrq.7 for ; Tue, 01 Mar 2022 02:24:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=0cTVlvy3koO8kIFX46QMstD+ERk03HMw6aT8GNBxMTM=; b=Um4TPbXWSzH3YXre4rKc6ly2/HI4nHUe0J9GgeYudGyZI3awy8HZL/TCS87y3qjgPf jDImtZVQArsFNXFCqRq7Q/Ej6dQ8rsgrOEBDiSBj5fJQitnDnZ6MF8DGDcs22hy8ybNz p23D4Z0GmO5owBB8ipaZCOjxMVGQ59a5JGGlhH2L6dk37scfX1ZQz3RbEIbrKxt3EhEM 1AP6T43lB+mfLrggKbiA3JD70YaqqaY0HTvdFUyHbELHdJjRPpx6fOSG5lJlTGVw+/hx 9VoIDzxBOE7vfdlr52OzmuD/PZ0VuLyb/sD2qGtPKp3m6nvm36dkiX2eZpAG7Gcon0VE HvjQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=0cTVlvy3koO8kIFX46QMstD+ERk03HMw6aT8GNBxMTM=; b=AHfpdZc/cPZX8n8thldt06u5SC0ergaeY/u6MFLu+NIP6qcwcoWepDlnqVX5Jz/qZq 9rZn/VrrSi0FUqoJM+pBEnavWRcK4ySp3Sne/OIPVJaCZNPNwvWnvCq4szZHhJOh4f57 isAgZOjpS9kf5zfGg3UsYAt69bHoYuXyddYCjc1PYPtDcuM0B1QOib1OedHh8cl+iza0 gyxe+FzKVYYwBBNG68Rt7/yZ66nq91kww77D7td9ojQliCHtMSC2qmOw78hlY0xbx1pK XpToM3EoEHE4QNGLhjGRFLBRY6eq+bQihU+DBRFlWoe0Ov7XISlWDQqZY18eAJzdTK4S Ctvw== X-Gm-Message-State: AOAM533Abc78qUtqeVFOA7yVPvnJZ/bpEnhbfYU15BBnXKv0r+ocSDo0 /IIc4m28mIRAQLzNWheoVDSgVISQl38= X-Google-Smtp-Source: ABdhPJwzLw10xEc8uruzPwhrIZ8efYeos9wSXF3v7igwCGs6zy5Jc2NNJt+HCx99WFwSrXtF0/16cA== X-Received: by 2002:a5d:588a:0:b0:1e8:b478:e74f with SMTP id n10-20020a5d588a000000b001e8b478e74fmr19349870wrf.210.1646130295765; Tue, 01 Mar 2022 02:24:55 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id 17-20020a05600c241100b0037c01ad7152sm2915196wmp.14.2022.03.01.02.24.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 01 Mar 2022 02:24:55 -0800 (PST) Message-Id: <15f199e810e08701b0c3bb84f1e9737672c100d9.1646130289.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Tue, 01 Mar 2022 10:24:47 +0000 Subject: [PATCH v2 7/9] ci: optionally mark up output in the GitHub workflow Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Eric Sunshine , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7A=?= Bjarmason , Johannes Schindelin , Johannes Schindelin Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Johannes Schindelin From: Johannes Schindelin A couple of commands exist to spruce up the output in GitHub workflows: https://docs.github.com/en/actions/learn-github-actions/workflow-commands-for-github-actions In addition to the `::group::