From patchwork Wed Jan 23 14:40:14 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Schindelin via GitGitGadget X-Patchwork-Id: 10777325 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id A1A7E746 for ; Wed, 23 Jan 2019 14:40:51 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 8E1892C9DA for ; Wed, 23 Jan 2019 14:40:51 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 824442C9E2; Wed, 23 Jan 2019 14:40:51 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 1FF802C9DC for ; Wed, 23 Jan 2019 14:40:51 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727023AbfAWOkR (ORCPT ); Wed, 23 Jan 2019 09:40:17 -0500 Received: from mail-ed1-f66.google.com ([209.85.208.66]:43375 "EHLO mail-ed1-f66.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726354AbfAWOkR (ORCPT ); Wed, 23 Jan 2019 09:40:17 -0500 Received: by mail-ed1-f66.google.com with SMTP id f9so1862650eds.10 for ; Wed, 23 Jan 2019 06:40:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=date:message-id:in-reply-to:references:from:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=JRloxJWyjmBVigttLX7WyMhlPzKiV+c4cwHlZMGyXTE=; b=PMRaZaPtGd2gs7CMvCNu/1z6NBdi9CL6fit/pmi584pBqkuj5DtpdvG7G6R/5fE7l5 517cPd4IudM1Z7F6zpBCL1K+JTKLvbhDz3jBpi8aDfDomlw4RJ1Fffe5eHBAjiinQSh3 kBUJCgYGxbZoA350J7NcOK1D78HUaA+SrSKW7PhOFZO4+W3eGc/nTyBE2j16RcgUd0ft oOME5VyZGQW9bjtOfhshPpKzWNeVzcCuBsjYVTYUTizjoxAx6967VxtLwUMFGbt88x/k /2t8S5+4aKK1aDxdLejF5t8ByaPnLF22LylkJjzfjKLryPuHh35uZoImpGAjSNDlbDH/ 2/Xg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:message-id:in-reply-to:references:from :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=JRloxJWyjmBVigttLX7WyMhlPzKiV+c4cwHlZMGyXTE=; b=UlP24ehqmuzyNEtrKDI9kpEc3atj89+ubus78dLWTQAu352FUGs+SsOLpTNWRLg4Kn z41ApZWTMCbsAptHPa8Qk1LhNf3m9nOv2CywIKQUztOPH6q1lQvjHaxfCMEE4QBR/Dzm tqHnvXBXkTg+iEtne+cJAhpBN420xZ6kTfsjlg9z7w0qxun1yMQ5S9vZuygHXWzJjYuS mDwaZcXy75yeu2a3/yC/ftX1AzGSP3u4Tr5TsewmL30E2oTtadRvG3jtfhCWusqXi4pW Kz14gyswtUsAZ87OdJ+G2SQY7CCZHZQy3Fu0m7T4yRSYhKluX2RgLA0DJhgWEr/2HYww BzCg== X-Gm-Message-State: AJcUukfaqjTK2hlMCCvaRlPfoN+9MKloPBVShC1vKXKdEqOomdgtoHkq VF2OR8Eiw4HMno0hRNf75ooHJ9AM X-Google-Smtp-Source: ALg8bN6M7rUV2dDC9v0Gf0JL8fa7LfWO/ZmYzDlTMqATNzWbbSqaRngYyEbKaz11jJ0OX2A+zGNhWQ== X-Received: by 2002:a17:906:8249:: with SMTP id f9mr2790200ejx.134.1548254414905; Wed, 23 Jan 2019 06:40:14 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id c29sm10373289eda.75.2019.01.23.06.40.14 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 23 Jan 2019 06:40:14 -0800 (PST) Date: Wed, 23 Jan 2019 06:40:14 -0800 (PST) X-Google-Original-Date: Wed, 23 Jan 2019 14:39:52 GMT Message-Id: <6ed2f2a35c394987be2bebd2364915c4daed888d.1548254412.git.gitgitgadget@gmail.com> In-Reply-To: References: From: "Johannes Schindelin via GitGitGadget" Subject: [PATCH v4 01/21] travis: fix skipping tagged releases Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Junio C Hamano , Johannes Schindelin Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Johannes Schindelin When building a PR, TRAVIS_BRANCH refers to the *target branch*. Therefore, if a PR targets `master`, and `master` happened to be tagged, we skipped the build by mistake. Fix this by using TRAVIS_PULL_REQUEST_BRANCH (i.e. the *source branch*) when available, falling back to TRAVIS_BRANCH (i.e. for CI builds, also known as "push builds"). Let's give it a new variable name, too: CI_BRANCH (as it is different from TRAVIS_BRANCH). This also prepares for the upcoming patches which will make our ci/* code a bit more independent from Travis and open it to other CI systems (in particular to Azure Pipelines). Signed-off-by: Johannes Schindelin --- ci/lib-travisci.sh | 25 ++++++++++++++----------- 1 file changed, 14 insertions(+), 11 deletions(-) diff --git a/ci/lib-travisci.sh b/ci/lib-travisci.sh index 69dff4d1ec..c26bb6a274 100755 --- a/ci/lib-travisci.sh +++ b/ci/lib-travisci.sh @@ -1,22 +1,25 @@ # Library of functions shared by all CI scripts +# When building a PR, TRAVIS_BRANCH refers to the *target* branch. Not what we +# want here. We want the source branch instead. +CI_BRANCH="${TRAVIS_PULL_REQUEST_BRANCH:-$TRAVIS_BRANCH}" + skip_branch_tip_with_tag () { # Sometimes, a branch is pushed at the same time the tag that points # at the same commit as the tip of the branch is pushed, and building # both at the same time is a waste. # - # Travis gives a tagname e.g. v2.14.0 in $TRAVIS_BRANCH when - # the build is triggered by a push to a tag. Let's see if - # $TRAVIS_BRANCH is exactly at a tag, and if so, if it is - # different from $TRAVIS_BRANCH. That way, we can tell if - # we are building the tip of a branch that is tagged and - # we can skip the build because we won't be skipping a build - # of a tag. - - if TAG=$(git describe --exact-match "$TRAVIS_BRANCH" 2>/dev/null) && - test "$TAG" != "$TRAVIS_BRANCH" + # When the build is triggered by a push to a tag, $CI_BRANCH will + # have that tagname, e.g. v2.14.0. Let's see if $CI_BRANCH is + # exactly at a tag, and if so, if it is different from $CI_BRANCH. + # That way, we can tell if we are building the tip of a branch that + # is tagged and we can skip the build because we won't be skipping a + # build of a tag. + + if TAG=$(git describe --exact-match "$CI_BRANCH" 2>/dev/null) && + test "$TAG" != "$CI_BRANCH" then - echo "$(tput setaf 2)Tip of $TRAVIS_BRANCH is exactly at $TAG$(tput sgr0)" + echo "$(tput setaf 2)Tip of $CI_BRANCH is exactly at $TAG$(tput sgr0)" exit 0 fi } From patchwork Wed Jan 23 14:40:15 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Schindelin via GitGitGadget X-Patchwork-Id: 10777283 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 2FE67746 for ; Wed, 23 Jan 2019 14:40:21 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 1E13F2C8F2 for ; Wed, 23 Jan 2019 14:40:21 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 11F0A2C9D9; Wed, 23 Jan 2019 14:40:21 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 79ADD2C8F2 for ; Wed, 23 Jan 2019 14:40:20 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727100AbfAWOkT (ORCPT ); Wed, 23 Jan 2019 09:40:19 -0500 Received: from mail-ed1-f66.google.com ([209.85.208.66]:35958 "EHLO mail-ed1-f66.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726934AbfAWOkR (ORCPT ); Wed, 23 Jan 2019 09:40:17 -0500 Received: by mail-ed1-f66.google.com with SMTP id f23so1888826edb.3 for ; Wed, 23 Jan 2019 06:40:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=date:message-id:in-reply-to:references:from:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=96M+SI3VKYDk3uCf1BpYpa5//cuUAVpfWWWcGr5kIog=; b=iNJe4+xp4RKgQdWCTl14IihAnHK5OzQg/aFRjfIqdakCOlz/taDDir2qdYVE8V7aQ9 wekLATr4p7mCgo5HYQz/l8LPlqII6GS4/H2n+MPpRykCxZ7PCx9hb21XiygUkqt6EF// LiUq8g7XiTBm2UTdZhhwOq8mJ2t8v8S1saaknpNwVRbvctolWo1CQwlIoyCWz1swSEoA Y4948qzx+amIVV8G8qI6myTHBYXGj6m2+NFesShEjMHNYzhNxrikLZfIBKEKsL53E7pa IW2LhOOt5nnto6attMJGE1zfP8cedxeVxRgfq9dwtoyb2eaU8wPPY9eua8qM34KUeYSD A9MQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:message-id:in-reply-to:references:from :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=96M+SI3VKYDk3uCf1BpYpa5//cuUAVpfWWWcGr5kIog=; b=KcOc9GzrVHfdgA7sgYdodQR4S4fyb5Pe3iprliMSr0Z8w4jx2yAvJGlC+tVPDPiFKj X0Zpv+vgxLiSx/hFhypEF9lrwjNoBSJ38lHEMOf8vWQ/DCNG8WzVsXVjCTuj4Qh/BFkP Nxhnwbvbhy1hxAwTMTJoodE1MpKWcxzSCr4bwjV8SlJo3PhDMR/1zxeJ80iN012dmO1k YzFreenfhpLLAsdufgGdtdG5JqJaEkROgCjvQVBKvnk0yO/hejSQ70cg8LOeGPw6t8zV qWv964RRw/zVIS88y3f/b5tNvdopbFYW4ZBjmgphaDO8BUy4dnyByt13ON30xsoFeirz J0Vw== X-Gm-Message-State: AJcUukcmi350kvyNEWN9wfI9LC4F4T3hpNJYGOXfxDHDowLBmmKKp3oh ph4RcKtUuCfUzIWn+fY33ZuxzJFq X-Google-Smtp-Source: ALg8bN6Ux/JHF/PS2Pf8JWUbSr5P/mv1RGgJKNilDE88W26PLLJcj+4zFabrtBtzJ0BQQeB6Qg9OQQ== X-Received: by 2002:a50:8689:: with SMTP id r9mr2837547eda.227.1548254415706; Wed, 23 Jan 2019 06:40:15 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id g37sm10768793edb.85.2019.01.23.06.40.15 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 23 Jan 2019 06:40:15 -0800 (PST) Date: Wed, 23 Jan 2019 06:40:15 -0800 (PST) X-Google-Original-Date: Wed, 23 Jan 2019 14:39:53 GMT Message-Id: In-Reply-To: References: From: "Johannes Schindelin via GitGitGadget" Subject: [PATCH v4 02/21] ci: rename the library of common functions Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Junio C Hamano , Johannes Schindelin Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Johannes Schindelin The name is hard-coded to reflect that we use Travis CI for continuous testing. In the next commits, we will extend this to be able use Azure DevOps, too. So let's adjust the name to make it more generic. Signed-off-by: Johannes Schindelin --- ci/install-dependencies.sh | 2 +- ci/{lib-travisci.sh => lib.sh} | 0 ci/print-test-failures.sh | 2 +- ci/run-build-and-tests.sh | 2 +- ci/run-linux32-docker.sh | 2 +- ci/run-static-analysis.sh | 2 +- ci/run-windows-build.sh | 2 +- ci/test-documentation.sh | 2 +- 8 files changed, 7 insertions(+), 7 deletions(-) rename ci/{lib-travisci.sh => lib.sh} (100%) diff --git a/ci/install-dependencies.sh b/ci/install-dependencies.sh index 06c3546e1e..fe65144152 100755 --- a/ci/install-dependencies.sh +++ b/ci/install-dependencies.sh @@ -3,7 +3,7 @@ # Install dependencies required to build and test Git on Linux and macOS # -. ${0%/*}/lib-travisci.sh +. ${0%/*}/lib.sh P4WHENCE=http://filehost.perforce.com/perforce/r$LINUX_P4_VERSION LFSWHENCE=https://github.com/github/git-lfs/releases/download/v$LINUX_GIT_LFS_VERSION diff --git a/ci/lib-travisci.sh b/ci/lib.sh similarity index 100% rename from ci/lib-travisci.sh rename to ci/lib.sh diff --git a/ci/print-test-failures.sh b/ci/print-test-failures.sh index d55460a212..7aef39a2fd 100755 --- a/ci/print-test-failures.sh +++ b/ci/print-test-failures.sh @@ -3,7 +3,7 @@ # Print output of failing tests # -. ${0%/*}/lib-travisci.sh +. ${0%/*}/lib.sh # Tracing executed commands would produce too much noise in the loop below. set +x diff --git a/ci/run-build-and-tests.sh b/ci/run-build-and-tests.sh index cda170d5c2..db342bb6a8 100755 --- a/ci/run-build-and-tests.sh +++ b/ci/run-build-and-tests.sh @@ -3,7 +3,7 @@ # Build and test Git # -. ${0%/*}/lib-travisci.sh +. ${0%/*}/lib.sh ln -s "$cache_dir/.prove" t/.prove diff --git a/ci/run-linux32-docker.sh b/ci/run-linux32-docker.sh index 21637903ce..751acfcf8a 100755 --- a/ci/run-linux32-docker.sh +++ b/ci/run-linux32-docker.sh @@ -3,7 +3,7 @@ # Download and run Docker image to build and test 32-bit Git # -. ${0%/*}/lib-travisci.sh +. ${0%/*}/lib.sh docker pull daald/ubuntu32:xenial diff --git a/ci/run-static-analysis.sh b/ci/run-static-analysis.sh index 5688f261d0..dc189c7456 100755 --- a/ci/run-static-analysis.sh +++ b/ci/run-static-analysis.sh @@ -3,7 +3,7 @@ # Perform various static code analysis checks # -. ${0%/*}/lib-travisci.sh +. ${0%/*}/lib.sh make --jobs=2 coccicheck diff --git a/ci/run-windows-build.sh b/ci/run-windows-build.sh index d99a180e52..a73a4eca0a 100755 --- a/ci/run-windows-build.sh +++ b/ci/run-windows-build.sh @@ -6,7 +6,7 @@ # supported) and a commit hash. # -. ${0%/*}/lib-travisci.sh +. ${0%/*}/lib.sh test $# -ne 2 && echo "Unexpected number of parameters" && exit 1 test -z "$GFW_CI_TOKEN" && echo "GFW_CI_TOKEN not defined" && exit diff --git a/ci/test-documentation.sh b/ci/test-documentation.sh index a20de9ca12..d3cdbac73f 100755 --- a/ci/test-documentation.sh +++ b/ci/test-documentation.sh @@ -3,7 +3,7 @@ # Perform sanity checks on documentation and build it. # -. ${0%/*}/lib-travisci.sh +. ${0%/*}/lib.sh gem install asciidoctor From patchwork Wed Jan 23 14:40:16 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Schindelin via GitGitGadget X-Patchwork-Id: 10777285 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 05839746 for ; Wed, 23 Jan 2019 14:40:23 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E50F72C8F2 for ; Wed, 23 Jan 2019 14:40:22 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id D8E772C9D9; Wed, 23 Jan 2019 14:40:22 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 423512C8F2 for ; Wed, 23 Jan 2019 14:40:22 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727114AbfAWOkU (ORCPT ); Wed, 23 Jan 2019 09:40:20 -0500 Received: from mail-ed1-f67.google.com ([209.85.208.67]:43378 "EHLO mail-ed1-f67.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727070AbfAWOkT (ORCPT ); Wed, 23 Jan 2019 09:40:19 -0500 Received: by mail-ed1-f67.google.com with SMTP id f9so1862730eds.10 for ; Wed, 23 Jan 2019 06:40:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=date:message-id:in-reply-to:references:from:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=pgp8E1BaMlU1zdLpdXCMie6m/c9J3XLHle9qf0mBpU4=; b=XpwuMREA6yXoq6xGszCgUgV040M6EaLQ67mbe0/AP0xAWbJUSaJcZpOLJxxZWJYvpg qSBQUCxg74uBR0pshRG21N5RbWR/BB1PeINp4cPv03RrR7Dn5ayUdJUfmJP23ZxdsG3r FcLuRaOov55+MVD3Ek5ZNCqtRHaPCLTMtQRHqSYiqIhS+pSBxBZRC0SRB6NHBFhJk+MB D5FF7O1j+7H5ZHKvKMsmFwYjHu6ZsWCbAZQIPskJL0oyZA3lfY6KgGu2itkEuHVm/VRP XEA35uK5G2qWgjT6umEFQiZMPpD259iKNwwLTQcT07pM4+RSzfsJTWRV72FpzEwgFD6f jJJw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:message-id:in-reply-to:references:from :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=pgp8E1BaMlU1zdLpdXCMie6m/c9J3XLHle9qf0mBpU4=; b=cPrduHjC8t2sUXUbnGmkqAAMMAgTcEu6GUgUzv2s3AQH3UPbpNwdxu2wB5HMlR8KQG X4wahq4tA57M2p9YDeM545MSJCvxuX0MaQFwLh3HvLYiJILjF3lxDGm3nfZ3eTj2Sw+u MIjxX0buYlBGiDp+QCapuop0xVB4X1aYqKa5fE1X/A/+yNiRGZ7jjXBRaoT7lAzYCUsv DuimEgrERUb8xxLEHiMkwBEHyhUFkdOL/iYt7Ax/LIU4dDnUMO2kFuh8hhm+91oWeEH5 bnS8+PhYMKGmT4r7iG3edXfJJCChVp3+aislxbTConPV6Kp5IZDNx02XBxzqiEdj3LsQ LuKw== X-Gm-Message-State: AJcUukdfVm7n2epMsM6/X8cfGjuBzyoifRXJInXAKHEvrJ99LJbWNNUz CwTMu8kOVU8qXnz+rzZjvazcH3ZF X-Google-Smtp-Source: ALg8bN4QykDD8/80Uq5s4O9UN1DDjhLX6ESf4oQbKKcLI6OF/8b2Cnlgpn3cWZDeNoqU5iGP5oBlqA== X-Received: by 2002:aa7:d9d6:: with SMTP id v22mr2898303eds.265.1548254416669; Wed, 23 Jan 2019 06:40:16 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id gy12-v6sm5800653ejb.44.2019.01.23.06.40.15 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 23 Jan 2019 06:40:16 -0800 (PST) Date: Wed, 23 Jan 2019 06:40:16 -0800 (PST) X-Google-Original-Date: Wed, 23 Jan 2019 14:39:54 GMT Message-Id: <2d0b62f1862d5fce3ac0d7b813399f560600917d.1548254412.git.gitgitgadget@gmail.com> In-Reply-To: References: From: "Johannes Schindelin via GitGitGadget" Subject: [PATCH v4 03/21] ci/lib.sh: encapsulate Travis-specific things Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Junio C Hamano , Johannes Schindelin Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Johannes Schindelin The upcoming patches will allow building git.git via Azure Pipelines (i.e. Azure DevOps' Continuous Integration), where variable names and URLs look a bit different than in Travis CI. Signed-off-by: Johannes Schindelin --- ci/install-dependencies.sh | 3 ++- ci/lib.sh | 41 ++++++++++++++++++++++++++------------ ci/print-test-failures.sh | 2 +- ci/test-documentation.sh | 1 + 4 files changed, 32 insertions(+), 15 deletions(-) diff --git a/ci/install-dependencies.sh b/ci/install-dependencies.sh index fe65144152..bcdcc71592 100755 --- a/ci/install-dependencies.sh +++ b/ci/install-dependencies.sh @@ -37,7 +37,8 @@ osx-clang|osx-gcc) brew update --quiet # Uncomment this if you want to run perf tests: # brew install gnu-time - brew install git-lfs gettext + test -z "$BREW_INSTALL_PACKAGES" || + brew install $BREW_INSTALL_PACKAGES brew link --force gettext brew install caskroom/cask/perforce ;; diff --git a/ci/lib.sh b/ci/lib.sh index c26bb6a274..4456dbbcb0 100755 --- a/ci/lib.sh +++ b/ci/lib.sh @@ -1,8 +1,26 @@ # Library of functions shared by all CI scripts -# When building a PR, TRAVIS_BRANCH refers to the *target* branch. Not what we -# want here. We want the source branch instead. -CI_BRANCH="${TRAVIS_PULL_REQUEST_BRANCH:-$TRAVIS_BRANCH}" +if test true = "$TRAVIS" +then + # When building a PR, TRAVIS_BRANCH refers to the *target* branch. Not + # what we want here. We want the source branch instead. + CI_BRANCH="${TRAVIS_PULL_REQUEST_BRANCH:-$TRAVIS_BRANCH}" + CI_COMMIT="$TRAVIS_COMMIT" + CI_JOB_ID="$TRAVIS_JOB_ID" + CI_JOB_NUMBER="$TRAVIS_JOB_NUMBER" + CI_OS_NAME="$TRAVIS_OS_NAME" + CI_REPO_SLUG="$TRAVIS_REPO_SLUG" + + cache_dir="$HOME/travis-cache" + + url_for_job_id () { + echo "https://travis-ci.org/$CI_REPO_SLUG/jobs/$1" + } + + BREW_INSTALL_PACKAGES="git-lfs gettext" + export GIT_PROVE_OPTS="--timer --jobs 3 --state=failed,slow,save" + export GIT_TEST_OPTS="--verbose-log -x --immediate" +fi skip_branch_tip_with_tag () { # Sometimes, a branch is pushed at the same time the tag that points @@ -28,7 +46,7 @@ skip_branch_tip_with_tag () { # job if we encounter the same tree again and can provide a useful info # message. save_good_tree () { - echo "$(git rev-parse $TRAVIS_COMMIT^{tree}) $TRAVIS_COMMIT $TRAVIS_JOB_NUMBER $TRAVIS_JOB_ID" >>"$good_trees_file" + echo "$(git rev-parse $CI_COMMIT^{tree}) $CI_COMMIT $CI_JOB_NUMBER $CI_JOB_ID" >>"$good_trees_file" # limit the file size tail -1000 "$good_trees_file" >"$good_trees_file".tmp mv "$good_trees_file".tmp "$good_trees_file" @@ -38,7 +56,7 @@ save_good_tree () { # successfully before (e.g. because the branch got rebased, changing only # the commit messages). skip_good_tree () { - if ! good_tree_info="$(grep "^$(git rev-parse $TRAVIS_COMMIT^{tree}) " "$good_trees_file")" + if ! good_tree_info="$(grep "^$(git rev-parse $CI_COMMIT^{tree}) " "$good_trees_file")" then # Haven't seen this tree yet, or no cached good trees file yet. # Continue the build job. @@ -48,18 +66,18 @@ skip_good_tree () { echo "$good_tree_info" | { read tree prev_good_commit prev_good_job_number prev_good_job_id - if test "$TRAVIS_JOB_ID" = "$prev_good_job_id" + if test "$CI_JOB_ID" = "$prev_good_job_id" then cat <<-EOF - $(tput setaf 2)Skipping build job for commit $TRAVIS_COMMIT.$(tput sgr0) + $(tput setaf 2)Skipping build job for commit $CI_COMMIT.$(tput sgr0) This commit has already been built and tested successfully by this build job. To force a re-build delete the branch's cache and then hit 'Restart job'. EOF else cat <<-EOF - $(tput setaf 2)Skipping build job for commit $TRAVIS_COMMIT.$(tput sgr0) + $(tput setaf 2)Skipping build job for commit $CI_COMMIT.$(tput sgr0) This commit's tree has already been built and tested successfully in build job $prev_good_job_number for commit $prev_good_commit. - The log of that build job is available at https://travis-ci.org/$TRAVIS_REPO_SLUG/jobs/$prev_good_job_id + The log of that build job is available at $(url_for_job_id $prev_good_job_id) To force a re-build delete the branch's cache and then hit 'Restart job'. EOF fi @@ -84,7 +102,6 @@ check_unignored_build_artifacts () # and installing dependencies. set -ex -cache_dir="$HOME/travis-cache" good_trees_file="$cache_dir/good-trees" mkdir -p "$cache_dir" @@ -94,13 +111,11 @@ skip_good_tree if test -z "$jobname" then - jobname="$TRAVIS_OS_NAME-$CC" + jobname="$CI_OS_NAME-$CC" fi export DEVELOPER=1 export DEFAULT_TEST_TARGET=prove -export GIT_PROVE_OPTS="--timer --jobs 3 --state=failed,slow,save" -export GIT_TEST_OPTS="--verbose-log -x --immediate" export GIT_TEST_CLONE_2GB=YesPlease if [ "$jobname" = linux-gcc ]; then export CC=gcc-8 diff --git a/ci/print-test-failures.sh b/ci/print-test-failures.sh index 7aef39a2fd..d2045b63a6 100755 --- a/ci/print-test-failures.sh +++ b/ci/print-test-failures.sh @@ -69,7 +69,7 @@ do fi done -if [ $combined_trash_size -gt 0 ] +if [ -n "$TRAVIS_JOB_ID" -a $combined_trash_size -gt 0 ] then echo "------------------------------------------------------------------------" echo "Trash directories embedded in this log can be extracted by running:" diff --git a/ci/test-documentation.sh b/ci/test-documentation.sh index d3cdbac73f..7d0beb2832 100755 --- a/ci/test-documentation.sh +++ b/ci/test-documentation.sh @@ -5,6 +5,7 @@ . ${0%/*}/lib.sh +test -n "$ALREADY_HAVE_ASCIIDOCTOR" || gem install asciidoctor make check-builtins From patchwork Wed Jan 23 14:40:16 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Schindelin via GitGitGadget X-Patchwork-Id: 10777289 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 96DE9746 for ; Wed, 23 Jan 2019 14:40:24 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 855F02C8F2 for ; Wed, 23 Jan 2019 14:40:24 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 79BB52C9D9; Wed, 23 Jan 2019 14:40:24 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 31E5A2C8F2 for ; Wed, 23 Jan 2019 14:40:24 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727146AbfAWOkW (ORCPT ); Wed, 23 Jan 2019 09:40:22 -0500 Received: from mail-ed1-f67.google.com ([209.85.208.67]:41343 "EHLO mail-ed1-f67.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727090AbfAWOkT (ORCPT ); Wed, 23 Jan 2019 09:40:19 -0500 Received: by mail-ed1-f67.google.com with SMTP id a20so1862126edc.8 for ; Wed, 23 Jan 2019 06:40:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=date:message-id:in-reply-to:references:from:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=yjmVd1mMTh5O9pgiFT5EpWKGV+TAK5GJK5Kpo9RboIc=; b=jnLssD+aKSTCVyuhVYoeUmcFD5v1+zIpK3EiucJzk9KcwLArNyzga2336+/XFvbgNU nks4zYXQgtWmGn5bxNa1FKZi4o5ADEukmQvz/sWqp1/b7v7pJTFh9uy3GYU7oK2kJ0+o 5JHBtc8CMiDeTH891zbQ+WyyOI5DYxcZUB4z6aJ+NECcKoiyT+/04thchqVxfmFRQy5h fV3UHofKSN0aPSh46eNndIMbJdaOy3Hg564nkb0Nb6mW601mVc9QvIOvNSqQKPTcBq8s 6XUNanSfhzDsP5YlFQdc+CY53edwn8K22SGmz6QtaEmcrgUFJy64IwhgQT847PyQNGNq LHVw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:message-id:in-reply-to:references:from :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=yjmVd1mMTh5O9pgiFT5EpWKGV+TAK5GJK5Kpo9RboIc=; b=NLm2nir+j95ndhmMNrPH+k4MnSn9b1WV06JMSDXfcM2hMy68wSmBIkhDBBN0N/X2tm cHwHPQyKjdhNX7Gpwo51thRW4A2AJ6OpHOyzCFMIj0hPLXuolZRyXLZI4yrXzuf0Dy5K 3gouDFaLb1sz6j4lmsn+x9+cEUC17dOww+fdwIlYWmzcaxgJei6YMhocCm4AiEr4eQTg IONb54VC243k86qhHTWDu9FfnZ9IO7jcwG7VQzogWoi1HDL+pFBVKkZn6+E5jf3OSHz+ 570NFP9uW/OjQc23RaC8SWF6Wob8aNg/4vA1FZRzqWjbNr5SS6QzL0EkVD1tc3OuV2r4 /rmw== X-Gm-Message-State: AJcUukcyJ1cB+uIAKyl4puD3nEIe08IVy7Dv3g9Wi3Ul5v6RST2ore+X hD4kPi65pVcfKRNTayeJCPcqCzwx X-Google-Smtp-Source: ALg8bN7bXhy0SD+I5B42IE7jUFMbhEiDgxXdxBuvioCe2ShKkCYp5E4myQygxVooafZeyFV4WzgS9w== X-Received: by 2002:a50:a517:: with SMTP id y23mr3017202edb.219.1548254417538; Wed, 23 Jan 2019 06:40:17 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id p36sm10528133edc.78.2019.01.23.06.40.16 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 23 Jan 2019 06:40:16 -0800 (PST) Date: Wed, 23 Jan 2019 06:40:16 -0800 (PST) X-Google-Original-Date: Wed, 23 Jan 2019 14:39:55 GMT Message-Id: <83b92a87e7698cee1e2c44252b934ad436d75a90.1548254412.git.gitgitgadget@gmail.com> In-Reply-To: References: From: "Johannes Schindelin via GitGitGadget" Subject: [PATCH v4 04/21] ci: inherit --jobs via MAKEFLAGS in run-build-and-tests Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Junio C Hamano , Johannes Schindelin Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Johannes Schindelin Let's not decide in the generic ci/ script how many jobs to run in parallel; it is easy enough to hand that information down via the `MAKEFLAGS`. Signed-off-by: Johannes Schindelin --- ci/run-build-and-tests.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/ci/run-build-and-tests.sh b/ci/run-build-and-tests.sh index db342bb6a8..80d72d120f 100755 --- a/ci/run-build-and-tests.sh +++ b/ci/run-build-and-tests.sh @@ -7,7 +7,7 @@ ln -s "$cache_dir/.prove" t/.prove -make --jobs=2 +make make --quiet test if test "$jobname" = "linux-gcc" then From patchwork Wed Jan 23 14:40:17 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Schindelin via GitGitGadget X-Patchwork-Id: 10777287 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 6E2701515 for ; Wed, 23 Jan 2019 14:40:23 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 5ADA82C8F2 for ; Wed, 23 Jan 2019 14:40:23 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 4F2B42C9D6; Wed, 23 Jan 2019 14:40:23 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id EDCEE2C9DA for ; Wed, 23 Jan 2019 14:40:22 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727132AbfAWOkV (ORCPT ); Wed, 23 Jan 2019 09:40:21 -0500 Received: from mail-ed1-f68.google.com ([209.85.208.68]:39873 "EHLO mail-ed1-f68.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726895AbfAWOkT (ORCPT ); Wed, 23 Jan 2019 09:40:19 -0500 Received: by mail-ed1-f68.google.com with SMTP id b14so1876512edt.6 for ; Wed, 23 Jan 2019 06:40:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=date:message-id:in-reply-to:references:from:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=lCqeYJ5kfLqoi2oIKKtYTdsgfR0s8LiDnVTc/mVzstc=; b=bsztkRWHkRvvTjxoZ8Zzi6cRMnnXJdcmGDRzSiTE/PLbbrjWlqTxcKYzojFcd7ND9a 8MTpxMfEuqIUlWXXYIqCek7odjTSABmpm6Ex7yOHIuVzhOe3EfyAkTsPDw7PtSSBE0/L jXUclDtFLKQ2mRmVOOibCG/OwbRWLII9yiNLsB9b3AUnb3xkl2Oa2SJJeDP4WeNxTnhu nz3w6otkawtfLwbq2fYXOFoVyt0tbI3+V6AwZEmAT02RkvHzlnfAVIf/lMlKg2tP3dQ0 GvYMJ5HDFZhqohkOB0mYjZE/khKTg5Kds4M/huYwsehnNRkFpMFu64KYMLKTxii7Anbu HwZg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:message-id:in-reply-to:references:from :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=lCqeYJ5kfLqoi2oIKKtYTdsgfR0s8LiDnVTc/mVzstc=; b=ofoSbmRCgGDerzx6BIB4/oe5jQ4zbR/pCp0Ol6/jsYd0qBBiZECI2m7XcHTqF8sw6g u1FM8E9DjYb22SvDI3ceHGRZBaMPD7Vtq1DvmJZbY7kcTR++lLaho4cycta6MHJhLZSA 8/1gnOk/IuWTcM1uFejbpWoGfT9i4ULBkzMQVtdLKjvj42znvTydWFwbDC4hYKrrlyHQ jL2MjjIrFbYFpDWbkJzI25szXAEygRPO90pZsb7axScb26Xr1IB8H3YVyl+m0VPY08xB ulKT493vfAGO7BRLYAYjHqi2v4t1OIhTk46BHwtMQ9cvUgkTfNi+9V32+okpShWnKVC3 3W9Q== X-Gm-Message-State: AJcUukdqV2LSwGKJb4DcnihzVMQwSFjVfvvCYTyBHWqW3999WiwBVbfh TXBfWB/4lyRBNJnMj12ndLdhkBq/ X-Google-Smtp-Source: ALg8bN5trHad4fVZH5iCmwyXn+koWPW8HlftxFgfPthTZM4p24R1ohVixoOmTiwi4jEG2WZei3jQuQ== X-Received: by 2002:a50:b623:: with SMTP id b32mr2910520ede.55.1548254418371; Wed, 23 Jan 2019 06:40:18 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id k11sm9866974edq.51.2019.01.23.06.40.17 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 23 Jan 2019 06:40:17 -0800 (PST) Date: Wed, 23 Jan 2019 06:40:17 -0800 (PST) X-Google-Original-Date: Wed, 23 Jan 2019 14:39:56 GMT Message-Id: <6c9eb4e33e1e98210ed5efe7b52fe1a827b2c3ac.1548254412.git.gitgitgadget@gmail.com> In-Reply-To: References: From: "Johannes Schindelin via GitGitGadget" Subject: [PATCH v4 05/21] ci: use a junction on Windows instead of a symlink Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Junio C Hamano , Johannes Schindelin Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Johannes Schindelin Symbolic links are still not quite as easy to use on Windows as on Linux (for example, on versions older than Windows 10, only administrators can create symlinks, and on Windows 10 you still need to be in developer mode for regular users to have permission), but NTFS junctions can give us a way out. Signed-off-by: Johannes Schindelin --- ci/run-build-and-tests.sh | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/ci/run-build-and-tests.sh b/ci/run-build-and-tests.sh index 80d72d120f..74d838ea01 100755 --- a/ci/run-build-and-tests.sh +++ b/ci/run-build-and-tests.sh @@ -5,7 +5,10 @@ . ${0%/*}/lib.sh -ln -s "$cache_dir/.prove" t/.prove +case "$CI_OS_NAME" in +windows*) cmd //c mklink //j t\\.prove "$(cygpath -aw "$cache_dir/.prove")";; +*) ln -s "$cache_dir/.prove" t/.prove;; +esac make make --quiet test From patchwork Wed Jan 23 14:40:18 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Schindelin via GitGitGadget X-Patchwork-Id: 10777291 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 8871D746 for ; Wed, 23 Jan 2019 14:40:25 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 762E82C8F2 for ; Wed, 23 Jan 2019 14:40:25 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 6A5D52C9D9; Wed, 23 Jan 2019 14:40:25 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0D05D2C8F2 for ; Wed, 23 Jan 2019 14:40:25 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727160AbfAWOkY (ORCPT ); Wed, 23 Jan 2019 09:40:24 -0500 Received: from mail-ed1-f51.google.com ([209.85.208.51]:41408 "EHLO mail-ed1-f51.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727101AbfAWOkV (ORCPT ); Wed, 23 Jan 2019 09:40:21 -0500 Received: by mail-ed1-f51.google.com with SMTP id a20so1862200edc.8 for ; Wed, 23 Jan 2019 06:40:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=date:message-id:in-reply-to:references:from:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=bnpLIlyyLtIXH9ZdKeZynSDp+QyusEe60+Hv8ehTygc=; b=Xvu7gkv+gWscJ2kBRpmfqIgNUvAz47PnZ0Dl7CQJCWUPK5P2fNTq3ggFPDJfG30X+U bCF29/P5adfS9VAgd/8xXFkGUFUiCFAnSj18AtD60Py5d/SrpBUxiWkL62UkHFMAWvLO Ej4KLRZ/bNRAYvQVtBJX2NFsOi89sN4NnIaHBCGCLKv7IGHdiBYV6E5/Hj8J+i/9mkXt f33t/JnuSHr0eXyEQoYQ6cCw2e+z1+/KN4jPgfr3vX1ohrbie+5V549k+4yElIPYphNN 7CoKjX++k19UwuQxmq69KHQQLLwDMio7iqWmqwH9ho1WdubbnsAk4wcGSGQHaTU/3Irt ZQ6A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:message-id:in-reply-to:references:from :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=bnpLIlyyLtIXH9ZdKeZynSDp+QyusEe60+Hv8ehTygc=; b=LcUconUacRdqghKrlrhLSo7ahrtFWLm7LGqceUZ2eI/061O9vFtwO0Pvv9lch8yad7 O6wSC0Lkmrc8HRyfPUPBzeCUianHHUvT9XSU3yx47UxMTWd0NdSbtO/isntGLFKkTnbI Xcddf7qbq4phE9HkHo/brmNpuMmw9x14QiZK+OQW4RS72EwmewWtWMgfWuD+AqN+1A51 1aYmZeMGLPN96AXsq1MnrUp8Q16IrAhdtzAczVs64QS3qIc3NKL30npPCv3E5XuWpt6p 0/OWGJ0dfiQgnBlAJoeiTMB6eUVSdX5e32hQhAC+Tgzmj7K00UOjfQuOU+mcwFXiV/6f 7IaA== X-Gm-Message-State: AJcUukcf+NDGNlF+imN60VMf6HmPwGfhesQFxHvFXWpiu4ssoFzj4hRX CKMCXH6ryDyFjaRqjxF/E+Ki9A1G X-Google-Smtp-Source: ALg8bN4N9Z5PrZcO4w/A2s4bKc0iuDmkCBfRSOh8PtH7Nu0RHYkN7qlFqZyWiKjANz8Q4EV+aWB4ZQ== X-Received: by 2002:a50:b36f:: with SMTP id r44mr2958838edd.284.1548254419203; Wed, 23 Jan 2019 06:40:19 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id i8-v6sm3946308eja.45.2019.01.23.06.40.18 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 23 Jan 2019 06:40:18 -0800 (PST) Date: Wed, 23 Jan 2019 06:40:18 -0800 (PST) X-Google-Original-Date: Wed, 23 Jan 2019 14:39:57 GMT Message-Id: In-Reply-To: References: From: "Johannes Schindelin via GitGitGadget" Subject: [PATCH v4 06/21] test-date: add a subcommand to measure times in shell scripts Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Junio C Hamano , Johannes Schindelin Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Johannes Schindelin In the next commit, we want to teach Git's test suite to optionally output test results in JUnit-style .xml files. These files contain information about the time spent. So we need a way to measure time. While we could use `date +%s` for that, this will give us only seconds, i.e. very coarse-grained timings. GNU `date` supports `date +%s.%N` (i.e. nanosecond-precision output), but there is no equivalent in BSD `date` (read: on macOS, we would not be able to obtain precise timings). So let's introduce `test-tool date getnanos`, with an optional start time, that outputs preciser values. Granted, it is a bit pointless to try measuring times accurately in shell scripts, certainly to nanosecond precision. But it is better than second-granularity. Signed-off-by: Johannes Schindelin --- t/helper/test-date.c | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/t/helper/test-date.c b/t/helper/test-date.c index a0837371ab..792a805374 100644 --- a/t/helper/test-date.c +++ b/t/helper/test-date.c @@ -7,6 +7,7 @@ static const char *usage_msg = "\n" " test-tool date parse [date]...\n" " test-tool date approxidate [date]...\n" " test-tool date timestamp [date]...\n" +" test-tool date getnanos [start-nanos]\n" " test-tool date is64bit\n" " test-tool date time_t-is64bit\n"; @@ -82,6 +83,15 @@ static void parse_approx_timestamp(const char **argv, struct timeval *now) } } +static void getnanos(const char **argv, struct timeval *now) +{ + double seconds = getnanotime() / 1.0e9; + + if (*argv) + seconds -= strtod(*argv, NULL); + printf("%lf\n", seconds); +} + int cmd__date(int argc, const char **argv) { struct timeval now; @@ -108,6 +118,8 @@ int cmd__date(int argc, const char **argv) parse_approxidate(argv+1, &now); else if (!strcmp(*argv, "timestamp")) parse_approx_timestamp(argv+1, &now); + else if (!strcmp(*argv, "getnanos")) + getnanos(argv+1, &now); else if (!strcmp(*argv, "is64bit")) return sizeof(timestamp_t) == 8 ? 0 : 1; else if (!strcmp(*argv, "time_t-is64bit")) From patchwork Wed Jan 23 14:40:19 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Schindelin via GitGitGadget X-Patchwork-Id: 10777313 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 2FD381390 for ; Wed, 23 Jan 2019 14:40:43 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 1E49B2C8F2 for ; Wed, 23 Jan 2019 14:40:43 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 12D9C2C9DC; Wed, 23 Jan 2019 14:40:43 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 571B92C8F2 for ; Wed, 23 Jan 2019 14:40:42 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727175AbfAWOkZ (ORCPT ); Wed, 23 Jan 2019 09:40:25 -0500 Received: from mail-ed1-f48.google.com ([209.85.208.48]:44524 "EHLO mail-ed1-f48.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727129AbfAWOkW (ORCPT ); Wed, 23 Jan 2019 09:40:22 -0500 Received: by mail-ed1-f48.google.com with SMTP id y56so1854966edd.11 for ; Wed, 23 Jan 2019 06:40:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=date:message-id:in-reply-to:references:from:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=+n7i2tQJa0DlH1QtOlg+5zTg3n07wnZ+0QGC/00TeMc=; b=GFJi7AhAuT8Ej/1ZGa3xecH0EMr1Lq4NL5iheqCk8nWqwJKkRLwBq6XLfmamLnMQdx 8Sc5SJKslc7RUdlPNxhVO4l6/Lm+gAeTbTnRfiOQLrqYdbcdCQa8EXyk9GchyF5Uzjsc c5vhoku2nqGASXfP4Fg7Cvihfxsgcsun9Uduc1FE7omf28/cOZm8R1STHMV55NSsePWb aRY7xTxyqj5WOaUEp/C3CVjmX+9H+unXMYwNP2jgRnHY+ZlfPbd9dH/jN1OkZizmOrp/ yaCMMCH2pwuUNwHFDYQjBNZaWqe47bkXWhQRiJYUfyUF3B570R84SGXQNHR8PDXXymg4 FU1A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:message-id:in-reply-to:references:from :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=+n7i2tQJa0DlH1QtOlg+5zTg3n07wnZ+0QGC/00TeMc=; b=CXmulyuexpJW2d0qFe2uJZVXIkzn+QREJ7hLQjqZTaPuHp3LPTtGbffj4z4WeEN3U0 On+InabGcRLS1h1wHHAvwSLlVfSVBmjxIWoZPtdYE9lRL9Tlm2BuN7HdxzdBNp+X/e7r bx3tpTm+/W7a2wGywPGiWKhO38MthB9m5SkhkSjiwLqjygUOBSrFlMwwfTByTLrp5mrS kHpKXa+yPAASdy80Fth5nDwPLaY8HdfeTc6ygHVd/aArZNqCyrP0xl68uGsymj337+Y5 m9dP146fBO+bWiAkywVlAmEAL+UNJt5/yvW6VWp53uAV3Wd9hFfHkLjwSFg17k4qCewT 7pWA== X-Gm-Message-State: AJcUukdF+IBTASSE85Ikyv5uil9SeW5BZUvQMMwRA3Buv5Xd1GbKI6DU AVNVucfb8Hq5JliFJ6kqNhpF9zQ8 X-Google-Smtp-Source: ALg8bN6oYqdHfV57IinxoqHEI1r1Q7NYSqOf0CgNNocdqUEHzxygL00xjjhI3PdzflRrz5eIZyaOqg== X-Received: by 2002:a50:c299:: with SMTP id o25mr3011797edf.281.1548254420027; Wed, 23 Jan 2019 06:40:20 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id y17-v6sm5817130ejj.43.2019.01.23.06.40.19 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 23 Jan 2019 06:40:19 -0800 (PST) Date: Wed, 23 Jan 2019 06:40:19 -0800 (PST) X-Google-Original-Date: Wed, 23 Jan 2019 14:39:58 GMT Message-Id: <07fb37e0a6b7382b990739c491a6499e95b25b06.1548254412.git.gitgitgadget@gmail.com> In-Reply-To: References: From: "Johannes Schindelin via GitGitGadget" Subject: [PATCH v4 07/21] tests: optionally write results as JUnit-style .xml Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Junio C Hamano , Johannes Schindelin Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Johannes Schindelin This will come in handy when publishing the results of Git's test suite during an automated Azure DevOps run. Note: we need to make extra sure that invalid UTF-8 encoding is turned into valid UTF-8 (using the Replacement Character, \uFFFD) because t9902's trace contains such invalid byte sequences, and the task in the Azure Pipeline that uploads the test results would refuse to do anything if it was asked to parse an .xml file with invalid UTF-8 in it. Signed-off-by: Johannes Schindelin --- Makefile | 1 + t/.gitignore | 1 + t/helper/test-tool.c | 1 + t/helper/test-tool.h | 1 + t/helper/test-xml-encode.c | 80 +++++++++++++++++++++++++++++++++ t/test-lib.sh | 91 ++++++++++++++++++++++++++++++++++++++ 6 files changed, 175 insertions(+) create mode 100644 t/helper/test-xml-encode.c diff --git a/Makefile b/Makefile index 1a44c811aa..044b4f77bd 100644 --- a/Makefile +++ b/Makefile @@ -754,6 +754,7 @@ TEST_BUILTINS_OBJS += test-submodule-config.o TEST_BUILTINS_OBJS += test-submodule-nested-repo-config.o TEST_BUILTINS_OBJS += test-subprocess.o TEST_BUILTINS_OBJS += test-urlmatch-normalization.o +TEST_BUILTINS_OBJS += test-xml-encode.o TEST_BUILTINS_OBJS += test-wildmatch.o TEST_BUILTINS_OBJS += test-windows-named-pipe.o TEST_BUILTINS_OBJS += test-write-cache.o diff --git a/t/.gitignore b/t/.gitignore index 348715f0e4..91cf5772fe 100644 --- a/t/.gitignore +++ b/t/.gitignore @@ -2,3 +2,4 @@ /test-results /.prove /chainlinttmp +/out/ diff --git a/t/helper/test-tool.c b/t/helper/test-tool.c index bfb195b1a8..4b4b397d93 100644 --- a/t/helper/test-tool.c +++ b/t/helper/test-tool.c @@ -49,6 +49,7 @@ static struct test_cmd cmds[] = { { "submodule-nested-repo-config", cmd__submodule_nested_repo_config }, { "subprocess", cmd__subprocess }, { "urlmatch-normalization", cmd__urlmatch_normalization }, + { "xml-encode", cmd__xml_encode }, { "wildmatch", cmd__wildmatch }, #ifdef GIT_WINDOWS_NATIVE { "windows-named-pipe", cmd__windows_named_pipe }, diff --git a/t/helper/test-tool.h b/t/helper/test-tool.h index 042f12464b..c0ab65e370 100644 --- a/t/helper/test-tool.h +++ b/t/helper/test-tool.h @@ -45,6 +45,7 @@ int cmd__submodule_config(int argc, const char **argv); int cmd__submodule_nested_repo_config(int argc, const char **argv); int cmd__subprocess(int argc, const char **argv); int cmd__urlmatch_normalization(int argc, const char **argv); +int cmd__xml_encode(int argc, const char **argv); int cmd__wildmatch(int argc, const char **argv); #ifdef GIT_WINDOWS_NATIVE int cmd__windows_named_pipe(int argc, const char **argv); diff --git a/t/helper/test-xml-encode.c b/t/helper/test-xml-encode.c new file mode 100644 index 0000000000..367c4875e6 --- /dev/null +++ b/t/helper/test-xml-encode.c @@ -0,0 +1,80 @@ +#include "test-tool.h" + +static const char *utf8_replace_character = "�"; + +/* + * Encodes (possibly incorrect) UTF-8 on to , to be embedded + * in an XML file. + */ +int cmd__xml_encode(int argc, const char **argv) +{ + unsigned char buf[1024], tmp[4], *tmp2 = NULL; + ssize_t cur = 0, len = 1, remaining = 0; + unsigned char ch; + + for (;;) { + if (++cur == len) { + len = xread(0, buf, sizeof(buf)); + if (!len) + return 0; + if (len < 0) + die_errno("Could not read "); + cur = 0; + } + ch = buf[cur]; + + if (tmp2) { + if ((ch & 0xc0) != 0x80) { + fputs(utf8_replace_character, stdout); + tmp2 = 0; + cur--; + continue; + } + *tmp2 = ch; + tmp2++; + if (--remaining == 0) { + fwrite(tmp, tmp2 - tmp, 1, stdout); + tmp2 = 0; + } + continue; + } + + if (!(ch & 0x80)) { + /* 0xxxxxxx */ + if (ch == '&') + fputs("&", stdout); + else if (ch == '\'') + fputs("'", stdout); + else if (ch == '"') + fputs(""", stdout); + else if (ch == '<') + fputs("<", stdout); + else if (ch == '>') + fputs(">", stdout); + else if (ch >= 0x20) + fputc(ch, stdout); + else if (ch == 0x09 || ch == 0x0a || ch == 0x0d) + fprintf(stdout, "&#x%02x;", ch); + else + fputs(utf8_replace_character, stdout); + } else if ((ch & 0xe0) == 0xc0) { + /* 110XXXXx 10xxxxxx */ + tmp[0] = ch; + remaining = 1; + tmp2 = tmp + 1; + } else if ((ch & 0xf0) == 0xe0) { + /* 1110XXXX 10Xxxxxx 10xxxxxx */ + tmp[0] = ch; + remaining = 2; + tmp2 = tmp + 1; + } else if ((ch & 0xf8) == 0xf0) { + /* 11110XXX 10XXxxxx 10xxxxxx 10xxxxxx */ + tmp[0] = ch; + remaining = 3; + tmp2 = tmp + 1; + } else + fputs(utf8_replace_character, stdout); + } + + return 0; +} diff --git a/t/test-lib.sh b/t/test-lib.sh index a1abb1177a..a3b2166cb5 100644 --- a/t/test-lib.sh +++ b/t/test-lib.sh @@ -139,6 +139,9 @@ do verbose_log=t tee=t ;; + --write-junit-xml) + write_junit_xml=t + ;; --stress) stress=t ;; --stress=*) @@ -622,11 +625,24 @@ trap 'exit $?' INT TERM HUP # the test_expect_* functions instead. test_ok_ () { + if test -n "$write_junit_xml" + then + write_junit_xml_testcase "$*" + fi test_success=$(($test_success + 1)) say_color "" "ok $test_count - $@" } test_failure_ () { + if test -n "$write_junit_xml" + then + junit_insert="" + junit_insert="$junit_insert $(xml_attr_encode \ + "$(printf '%s\n' "$@" | sed 1d)")" + junit_insert="$junit_insert" + write_junit_xml_testcase "$1" " $junit_insert" + fi test_failure=$(($test_failure + 1)) say_color error "not ok $test_count - $1" shift @@ -635,11 +651,19 @@ test_failure_ () { } test_known_broken_ok_ () { + if test -n "$write_junit_xml" + then + write_junit_xml_testcase "$* (breakage fixed)" + fi test_fixed=$(($test_fixed+1)) say_color error "ok $test_count - $@ # TODO known breakage vanished" } test_known_broken_failure_ () { + if test -n "$write_junit_xml" + then + write_junit_xml_testcase "$* (known breakage)" + fi test_broken=$(($test_broken+1)) say_color warn "not ok $test_count - $@ # TODO known breakage" } @@ -897,6 +921,10 @@ test_start_ () { test_count=$(($test_count+1)) maybe_setup_verbose maybe_setup_valgrind + if test -n "$write_junit_xml" + then + junit_start=$(test-tool date getnanos) + fi } test_finish_ () { @@ -934,6 +962,13 @@ test_skip () { case "$to_skip" in t) + if test -n "$write_junit_xml" + then + message="$(xml_attr_encode "$skipped_reason")" + write_junit_xml_testcase "$1" \ + " " + fi + say_color skip >&3 "skipping test: $@" say_color skip "ok $test_count # skip $1 ($skipped_reason)" : true @@ -949,9 +984,51 @@ test_at_end_hook_ () { : } +write_junit_xml () { + case "$1" in + --truncate) + >"$junit_xml_path" + junit_have_testcase= + shift + ;; + esac + printf '%s\n' "$@" >>"$junit_xml_path" +} + +xml_attr_encode () { + printf '%s\n' "$@" | test-tool xml-encode +} + +write_junit_xml_testcase () { + junit_attrs="name=\"$(xml_attr_encode "$this_test.$test_count $1")\"" + shift + junit_attrs="$junit_attrs classname=\"$this_test\"" + junit_attrs="$junit_attrs time=\"$(test-tool \ + date getnanos $junit_start)\"" + write_junit_xml "$(printf '%s\n' \ + " " "$@" " ")" + junit_have_testcase=t +} + test_done () { GIT_EXIT_OK=t + if test -n "$write_junit_xml" && test -n "$junit_xml_path" + then + test -n "$junit_have_testcase" || { + junit_start=$(test-tool date getnanos) + write_junit_xml_testcase "all tests skipped" + } + + # adjust the overall time + junit_time=$(test-tool date getnanos $junit_suite_start) + sed "s/]*/& time=\"$junit_time\"/" \ + <"$junit_xml_path" >"$junit_xml_path.new" + mv "$junit_xml_path.new" "$junit_xml_path" + + write_junit_xml " " "" + fi + if test -z "$HARNESS_ACTIVE" then mkdir -p "$TEST_RESULTS_DIR" @@ -1178,6 +1255,7 @@ then else mkdir -p "$TRASH_DIRECTORY" fi + # Use -P to resolve symlinks in our working directory so that the cwd # in subprocesses like git equals our $PWD (for pathname comparisons). cd -P "$TRASH_DIRECTORY" || exit 1 @@ -1191,6 +1269,19 @@ then test_done fi +if test -n "$write_junit_xml" +then + junit_xml_dir="$TEST_OUTPUT_DIRECTORY/out" + mkdir -p "$junit_xml_dir" + junit_xml_base=${0##*/} + junit_xml_path="$junit_xml_dir/TEST-${junit_xml_base%.sh}.xml" + junit_attrs="name=\"${junit_xml_base%.sh}\"" + junit_attrs="$junit_attrs timestamp=\"$(TZ=UTC \ + date +%Y-%m-%dT%H:%M:%S)\"" + write_junit_xml --truncate "" " " + junit_suite_start=$(test-tool date getnanos) +fi + # Provide an implementation of the 'yes' utility yes () { if test $# = 0 From patchwork Wed Jan 23 14:40:20 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Schindelin via GitGitGadget X-Patchwork-Id: 10777321 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 331F11390 for ; Wed, 23 Jan 2019 14:40:46 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 21B112C8F2 for ; Wed, 23 Jan 2019 14:40:46 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 1643D2C9D9; Wed, 23 Jan 2019 14:40:46 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B3EEC2C8F2 for ; Wed, 23 Jan 2019 14:40:45 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726997AbfAWOkn (ORCPT ); Wed, 23 Jan 2019 09:40:43 -0500 Received: from mail-ed1-f44.google.com ([209.85.208.44]:44521 "EHLO mail-ed1-f44.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727136AbfAWOkW (ORCPT ); Wed, 23 Jan 2019 09:40:22 -0500 Received: by mail-ed1-f44.google.com with SMTP id y56so1854989edd.11 for ; Wed, 23 Jan 2019 06:40:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=date:message-id:in-reply-to:references:from:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=Euzt0m1wqeilLrDM6dfdK5D57Sibiv935ElLJjIpVQo=; b=r53iiX9w/EWT7Gz8N4xymlU28IJ6lxy8X0wgSH+LxqNNKGbo61S7CJl4qjkDLOCZRU CnzFGr5KAnBZUcAC2hXmpZm56H8aT2/2eWMzhu7BrivGu7G4/7eo5JYgn/OGEKouiKni H68E+2MyXGPjXoiI9R+nyaGNHBS4lrBAhaguED0C7ob30LYJFAp3YQwLlKZYq4UUP0at jVgk3RupXbROcm2i5o+uwjgwjTjADLC4o5yhgh47v2RDix8Rd3AAQWFFv50qN8i+3KHL iyH7+B3DlUyOkbCCpkHVN+CKC9qFVq+hREScuTJXCFm7yh5kAjrmaqWtLVJXjuSAIeUL mC6w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:message-id:in-reply-to:references:from :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=Euzt0m1wqeilLrDM6dfdK5D57Sibiv935ElLJjIpVQo=; b=lwheCTP46AbxUzNIykgAmyD0rq1oV2b8LoaeCbpTzPhym+jq9MK7e6P0f3dLHPgfgj 62/u2ORHZesbR7bPh9a/bwQSb76dAMg7Z6y15dxFKOyrHkmZ3mK8lVx2C27x+Z0qa1cf ozJFAWlyVEOnlQVo7o1RFM+iP1/EbXz4RMmsRDKzrGogmROYLlwlrPGiGrMLY49GeqxW GxQpJn2y568+Xu9A17BKv4Rv9IyGOjHiI3KrToX38VK0KAKmzT+rLVT1OXkd32C5qXVo k4T3N8An5MX+VxQyy5C4cCPXygjZyZ2q9QFcrfm6VxDiJcjLEcWH/OTjq6vYCvJ6boLV 4buQ== X-Gm-Message-State: AJcUukfhEr1Ek/XXa5qOXgfpPV44K4CbLW1bHTlNeLfBxt8PTTgLfPLt jFT5Eg9E60fSY/ZDpGKLBH07uqvV X-Google-Smtp-Source: ALg8bN6WkkOFLWo/A17o2I0A/W5NuVH3N7qk8ZcE3kcQY92XXRbNCAWT15ibhDzLUvmhRcXtIpmz1w== X-Received: by 2002:a17:906:14cc:: with SMTP id y12mr2751440ejc.144.1548254420671; Wed, 23 Jan 2019 06:40:20 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id l18sm10013642edq.87.2019.01.23.06.40.20 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 23 Jan 2019 06:40:20 -0800 (PST) Date: Wed, 23 Jan 2019 06:40:20 -0800 (PST) X-Google-Original-Date: Wed, 23 Jan 2019 14:39:59 GMT Message-Id: <178dc9b789caef8e2402ab518bb5ca36da2c32ad.1548254412.git.gitgitgadget@gmail.com> In-Reply-To: References: From: "Johannes Schindelin via GitGitGadget" Subject: [PATCH v4 08/21] ci/lib.sh: add support for Azure Pipelines Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Junio C Hamano , Johannes Schindelin Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Johannes Schindelin This patch introduces a conditional arm that defines some environment variables and a function that displays the URL given the job id (to identify previous runs for known-good trees). For example, we do not have to install the git-lfs and gettext packages on Azure Pipelines' macOS agents: they are already installed, and trying to install them again would result in an error. Signed-off-by: Johannes Schindelin --- ci/lib.sh | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) diff --git a/ci/lib.sh b/ci/lib.sh index 4456dbbcb0..a9eb4f4eae 100755 --- a/ci/lib.sh +++ b/ci/lib.sh @@ -20,6 +20,29 @@ then BREW_INSTALL_PACKAGES="git-lfs gettext" export GIT_PROVE_OPTS="--timer --jobs 3 --state=failed,slow,save" export GIT_TEST_OPTS="--verbose-log -x --immediate" +elif test -n "$SYSTEM_COLLECTIONURI" || test -n "$SYSTEM_TASKDEFINITIONSURI" +then + # We are running in Azure Pipelines + CI_BRANCH="$BUILD_SOURCEBRANCH" + CI_COMMIT="$BUILD_SOURCEVERSION" + CI_JOB_ID="$BUILD_BUILDID" + CI_JOB_NUMBER="$BUILD_BUILDNUMBER" + CI_OS_NAME="$(echo "$AGENT_OS" | tr A-Z a-z)" + test darwin != "$CI_OS_NAME" || CI_OS_NAME=osx + CI_REPO_SLUG="$(expr "$BUILD_REPOSITORY_URI" : '.*/\([^/]*/[^/]*\)$')" + CC="${CC:-gcc}" + + # use a subdirectory of the cache dir (because the file share is shared + # among *all* phases) + cache_dir="$HOME/test-cache/$SYSTEM_PHASENAME" + + url_for_job_id () { + echo "$SYSTEM_TASKDEFINITIONSURI$SYSTEM_TEAMPROJECT/_build/results?buildId=$1" + } + + BREW_INSTALL_PACKAGES= + export GIT_PROVE_OPTS="--timer --jobs 10 --state=failed,slow,save" + export GIT_TEST_OPTS="--quiet --write-junit-xml" fi skip_branch_tip_with_tag () { From patchwork Wed Jan 23 14:40:20 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Schindelin via GitGitGadget X-Patchwork-Id: 10777293 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id EDE4D1390 for ; Wed, 23 Jan 2019 14:40:28 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id DAC122C8F2 for ; Wed, 23 Jan 2019 14:40:28 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id CEFA82C9D9; Wed, 23 Jan 2019 14:40:28 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 68BFC2C8F2 for ; Wed, 23 Jan 2019 14:40:27 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727181AbfAWOk0 (ORCPT ); Wed, 23 Jan 2019 09:40:26 -0500 Received: from mail-ed1-f47.google.com ([209.85.208.47]:36230 "EHLO mail-ed1-f47.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727151AbfAWOkY (ORCPT ); Wed, 23 Jan 2019 09:40:24 -0500 Received: by mail-ed1-f47.google.com with SMTP id f23so1889083edb.3 for ; Wed, 23 Jan 2019 06:40:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=date:message-id:in-reply-to:references:from:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=xTVONC8LZDYxfgUme8ZkdxX0vuhf8/Re4Um+ykWIxds=; b=k3chW0GSzYozBN0ZVF/WBRD7WTi5UHLEMYzzM3aJ1Lgd6OPnUMA/0NSnXAFtJsH/l3 chaxnUt6y7DgOiDDQ/kP2WxR4/O86R9nVilVdIFvIUR6MiLhdZP6+/rhIzbgcJSOK3Er QLVKenm2jZqHiSVDTuaCjlE0sfoixlNSKeqml0J2ifuctW/fhSaGx1pw3f3qWNtk3l4d 8UH+XVYPHAERZdU7dBZaL35JkYCKE4MkegAEyVFsdH2go46m9kfkdElzhEX8yh5CnQ23 KDPhS6etNCDjuoTDOVzSA3HKu/XEIF4dMhlipDHo4nWAnHcFjN70eem0LrK9+al24vZt lIPA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:message-id:in-reply-to:references:from :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=xTVONC8LZDYxfgUme8ZkdxX0vuhf8/Re4Um+ykWIxds=; b=ZFVJd/m3YIuvLNmnwIWf05qZFPjTuuBTLNH28y08cnCRF1nGkwoSW152fkipP+bdWK ZMoILDD4YdweZm4iR76eQNpPaCME0TWM+FyeZikZ4e4lhswg2DCfJY7apRC1d/smLzgL H7oO2rZYRTAfSahgnM3qT4QQewiCuTnhjEfh2AWl6moxWX4p7tgg1xyogBtfBhnXHxIo wK1tESXAQLOTDFxjwM1IylPZDNWQ2FWiVBF2r67ELHm1o1sRz1lCqlt7+QJLjQCSV+bz VzZAQ8PhtlU86t5Q1Cgqa//JuPqQYAHnpSdACAyrYdHgJL6RluRVPxLLcxTDULCFbivp PJwQ== X-Gm-Message-State: AJcUukfj3QHJlFhJcpj/v/f3Ge/9VNlZ31FZQrDGdR/TPO3ov5WtrxSN QijcO/jS3JbsSAlk2bdPDYGDzfSC X-Google-Smtp-Source: ALg8bN7x1HLAp4WX0N0WSep75+CyIu2j5chxugEyS8WccES6WJST3nQyds4wveZqgeSU2TKhPesnyg== X-Received: by 2002:a50:b6d2:: with SMTP id f18mr2949859ede.263.1548254421474; Wed, 23 Jan 2019 06:40:21 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id r8sm9876238edo.11.2019.01.23.06.40.20 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 23 Jan 2019 06:40:20 -0800 (PST) Date: Wed, 23 Jan 2019 06:40:20 -0800 (PST) X-Google-Original-Date: Wed, 23 Jan 2019 14:40:00 GMT Message-Id: In-Reply-To: References: From: "Johannes Schindelin via GitGitGadget" Subject: [PATCH v4 09/21] Add a build definition for Azure DevOps Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Junio C Hamano , Johannes Schindelin Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Johannes Schindelin This commit adds an azure-pipelines.yml file which is Azure DevOps' equivalent to Travis CI's .travis.yml. To make things a bit easier to understand, we refrain from using the `matrix` feature here because (while it is powerful) it can be a bit confusing to users who are not familiar with CI setups. Therefore, we use a separate phase even for similar configurations (such as GCC vs Clang on Linux, GCC vs Clang on macOS). Also, we make use of the shiny new feature we just introduced where the test suite can output JUnit-style .xml files. This information is made available in a nice UI that allows the viewer to filter by phase and/or test number, and to see trends such as: number of (failing) tests, time spent running the test suite, etc. Signed-off-by: Johannes Schindelin --- azure-pipelines.yml | 306 ++++++++++++++++++++++++++++++++++++++++++ ci/lib.sh | 2 +- ci/mount-fileshare.sh | 25 ++++ 3 files changed, 332 insertions(+), 1 deletion(-) create mode 100644 azure-pipelines.yml create mode 100755 ci/mount-fileshare.sh diff --git a/azure-pipelines.yml b/azure-pipelines.yml new file mode 100644 index 0000000000..f3cabb0dd0 --- /dev/null +++ b/azure-pipelines.yml @@ -0,0 +1,306 @@ +resources: +- repo: self + fetchDepth: 1 + +jobs: +- job: linux_clang + displayName: linux-clang + condition: succeeded() + pool: Hosted Ubuntu 1604 + steps: + - bash: | + test "$GITFILESHAREPWD" = '$(gitfileshare.pwd)' || ci/mount-fileshare.sh //gitfileshare.file.core.windows.net/test-cache gitfileshare "$GITFILESHAREPWD" "$HOME/test-cache" || exit 1 + + sudo apt-get update && + sudo apt-get -y install git gcc make libssl-dev libcurl4-openssl-dev libexpat-dev tcl tk gettext git-email zlib1g-dev apache2-bin && + + export CC=clang || exit 1 + + ci/install-dependencies.sh || exit 1 + ci/run-build-and-tests.sh || { + ci/print-test-failures.sh + exit 1 + } + + test "$GITFILESHAREPWD" = '$(gitfileshare.pwd)' || sudo umount "$HOME/test-cache" || exit 1 + displayName: 'ci/run-build-and-tests.sh' + env: + GITFILESHAREPWD: $(gitfileshare.pwd) + - task: PublishTestResults@2 + displayName: 'Publish Test Results **/TEST-*.xml' + inputs: + mergeTestResults: true + testRunTitle: 'linux-clang' + platform: Linux + publishRunAttachments: false + condition: succeededOrFailed() + +- job: linux_gcc + displayName: linux-gcc + condition: succeeded() + pool: Hosted Ubuntu 1604 + steps: + - bash: | + test "$GITFILESHAREPWD" = '$(gitfileshare.pwd)' || ci/mount-fileshare.sh //gitfileshare.file.core.windows.net/test-cache gitfileshare "$GITFILESHAREPWD" "$HOME/test-cache" || exit 1 + + sudo add-apt-repository ppa:ubuntu-toolchain-r/test && + sudo apt-get update && + sudo apt-get -y install git gcc make libssl-dev libcurl4-openssl-dev libexpat-dev tcl tk gettext git-email zlib1g-dev apache2 language-pack-is git-svn gcc-8 || exit 1 + + ci/install-dependencies.sh || exit 1 + ci/run-build-and-tests.sh || { + ci/print-test-failures.sh + exit 1 + } + + test "$GITFILESHAREPWD" = '$(gitfileshare.pwd)' || sudo umount "$HOME/test-cache" || exit 1 + displayName: 'ci/run-build-and-tests.sh' + env: + GITFILESHAREPWD: $(gitfileshare.pwd) + - task: PublishTestResults@2 + displayName: 'Publish Test Results **/TEST-*.xml' + inputs: + mergeTestResults: true + testRunTitle: 'linux-gcc' + platform: Linux + publishRunAttachments: false + condition: succeededOrFailed() + +- job: osx_clang + displayName: osx-clang + condition: succeeded() + pool: Hosted macOS + steps: + - bash: | + test "$GITFILESHAREPWD" = '$(gitfileshare.pwd)' || ci/mount-fileshare.sh //gitfileshare.file.core.windows.net/test-cache gitfileshare "$GITFILESHAREPWD" "$HOME/test-cache" || exit 1 + + export CC=clang + + ci/install-dependencies.sh || exit 1 + ci/run-build-and-tests.sh || { + ci/print-test-failures.sh + exit 1 + } + + test "$GITFILESHAREPWD" = '$(gitfileshare.pwd)' || umount "$HOME/test-cache" || exit 1 + displayName: 'ci/run-build-and-tests.sh' + env: + GITFILESHAREPWD: $(gitfileshare.pwd) + - task: PublishTestResults@2 + displayName: 'Publish Test Results **/TEST-*.xml' + inputs: + mergeTestResults: true + testRunTitle: 'osx-clang' + platform: macOS + publishRunAttachments: false + condition: succeededOrFailed() + +- job: osx_gcc + displayName: osx-gcc + condition: succeeded() + pool: Hosted macOS + steps: + - bash: | + test "$GITFILESHAREPWD" = '$(gitfileshare.pwd)' || ci/mount-fileshare.sh //gitfileshare.file.core.windows.net/test-cache gitfileshare "$GITFILESHAREPWD" "$HOME/test-cache" || exit 1 + + ci/install-dependencies.sh || exit 1 + ci/run-build-and-tests.sh || { + ci/print-test-failures.sh + exit 1 + } + + test "$GITFILESHAREPWD" = '$(gitfileshare.pwd)' || umount "$HOME/test-cache" || exit 1 + displayName: 'ci/run-build-and-tests.sh' + env: + GITFILESHAREPWD: $(gitfileshare.pwd) + - task: PublishTestResults@2 + displayName: 'Publish Test Results **/TEST-*.xml' + inputs: + mergeTestResults: true + testRunTitle: 'osx-gcc' + platform: macOS + publishRunAttachments: false + condition: succeededOrFailed() + +- job: gettext_poison + displayName: GETTEXT_POISON + condition: succeeded() + pool: Hosted Ubuntu 1604 + steps: + - bash: | + test "$GITFILESHAREPWD" = '$(gitfileshare.pwd)' || ci/mount-fileshare.sh //gitfileshare.file.core.windows.net/test-cache gitfileshare "$GITFILESHAREPWD" "$HOME/test-cache" || exit 1 + + sudo apt-get update && + sudo apt-get -y install git gcc make libssl-dev libcurl4-openssl-dev libexpat-dev tcl tk gettext git-email zlib1g-dev && + + export jobname=GETTEXT_POISON || exit 1 + + ci/run-build-and-tests.sh || { + ci/print-test-failures.sh + exit 1 + } + + test "$GITFILESHAREPWD" = '$(gitfileshare.pwd)' || sudo umount "$HOME/test-cache" || exit 1 + displayName: 'ci/run-build-and-tests.sh' + env: + GITFILESHAREPWD: $(gitfileshare.pwd) + - task: PublishTestResults@2 + displayName: 'Publish Test Results **/TEST-*.xml' + inputs: + mergeTestResults: true + testRunTitle: 'gettext-poison' + platform: Linux + publishRunAttachments: false + condition: succeededOrFailed() + +- job: windows + displayName: Windows + condition: succeeded() + pool: Hosted + timeoutInMinutes: 240 + steps: + - powershell: | + if ("$GITFILESHAREPWD" -ne "" -and "$GITFILESHAREPWD" -ne "`$`(gitfileshare.pwd)") { + net use s: \\gitfileshare.file.core.windows.net\test-cache "$GITFILESHAREPWD" /user:AZURE\gitfileshare /persistent:no + cmd /c mklink /d "$(Build.SourcesDirectory)\test-cache" S:\ + } + displayName: 'Mount test-cache' + env: + GITFILESHAREPWD: $(gitfileshare.pwd) + - powershell: | + # Helper to check the error level of the latest command (exit with error when appropriate) + function c() { if (!$?) { exit(1) } } + + # Add build agent's MinGit to PATH + $env:PATH = $env:AGENT_HOMEDIRECTORY +"\externals\\git\cmd;" +$env:PATH + + # Helper to initialize (or update) a Git worktree + function init ($path, $url, $set_origin) { + if (Test-Path $path) { + cd $path; c + if (Test-Path .git) { + & git init; c + } else { + & git status + } + } else { + & git init $path; c + cd $path; c + } + & git config core.autocrlf false; c + & git config core.untrackedCache true; c + if (($set_origin -ne 0) -and !(git config remote.origin.url)) { + & git remote add origin $url; c + } + & git fetch --depth=1 $url master; c + & git reset --hard FETCH_HEAD; c + & git clean -df; c + } + + # Initialize Git for Windows' SDK + $sdk_path = "$(Build.SourcesDirectory)\git-sdk-64" + init "$sdk_path" "https://dev.azure.com/git-for-windows/git-sdk-64/_git/git-sdk-64" 0 + + # Let Git ignore the SDK and the test-cache + "/git-sdk-64/`n/test-cache/`n" | Out-File -NoNewLine -Encoding ascii -Append "$(Build.SourcesDirectory)\.git\info\exclude" + displayName: 'Initialize the Git for Windows SDK' + - powershell: | + & "git-sdk-64\git-cmd.exe" --command=usr\\bin\\bash.exe -lc @" + export MAKEFLAGS=-j10 + export DEVELOPER=1 + export NO_PERL=1 + export NO_SVN_TESTS=1 + export GIT_TEST_SKIP_REBASE_P=1 + + ci/run-build-and-tests.sh || { + ci/print-test-failures.sh + exit 1 + } + "@ + if (!$?) { exit(1) } + displayName: 'Build & Test' + env: + HOME: $(Build.SourcesDirectory) + MSYSTEM: MINGW64 + - powershell: | + if ("$GITFILESHAREPWD" -ne "" -and "$GITFILESHAREPWD" -ne "`$`(gitfileshare.pwd)") { + cmd /c rmdir "$(Build.SourcesDirectory)\test-cache" + } + displayName: 'Unmount test-cache' + condition: true + env: + GITFILESHAREPWD: $(gitfileshare.pwd) + - task: PublishTestResults@2 + displayName: 'Publish Test Results **/TEST-*.xml' + inputs: + mergeTestResults: true + testRunTitle: 'windows' + platform: Windows + publishRunAttachments: false + condition: succeededOrFailed() + +- job: linux32 + displayName: Linux32 + condition: succeeded() + pool: Hosted Ubuntu 1604 + steps: + - bash: | + test "$GITFILESHAREPWD" = '$(gitfileshare.pwd)' || ci/mount-fileshare.sh //gitfileshare.file.core.windows.net/test-cache gitfileshare "$GITFILESHAREPWD" "$HOME/test-cache" || exit 1 + + sudo AGENT_OS="$AGENT_OS" BUILD_BUILDNUMBER="$BUILD_BUILDNUMBER" BUILD_REPOSITORY_URI="$BUILD_REPOSITORY_URI" BUILD_SOURCEBRANCH="$BUILD_SOURCEBRANCH" BUILD_SOURCEVERSION="$BUILD_SOURCEVERSION" SYSTEM_PHASENAME="$SYSTEM_PHASENAME" SYSTEM_TASKDEFINITIONSURI="$SYSTEM_TASKDEFINITIONSURI" SYSTEM_TEAMPROJECT="$SYSTEM_TEAMPROJECT" CC=$CC MAKEFLAGS=-j3 bash -lxc ci/run-linux32-docker.sh || exit 1 + + sudo chmod a+r t/out/TEST-*.xml + + test "$GITFILESHAREPWD" = '$(gitfileshare.pwd)' || sudo umount "$HOME/test-cache" || exit 1 + displayName: 'ci/run-linux32-docker.sh' + env: + GITFILESHAREPWD: $(gitfileshare.pwd) + - task: PublishTestResults@2 + displayName: 'Publish Test Results **/TEST-*.xml' + inputs: + mergeTestResults: true + testRunTitle: 'linux32' + platform: Linux + publishRunAttachments: false + condition: succeededOrFailed() + +- job: static_analysis + displayName: StaticAnalysis + condition: succeeded() + pool: Hosted Ubuntu 1604 + steps: + - bash: | + test "$GITFILESHAREPWD" = '$(gitfileshare.pwd)' || ci/mount-fileshare.sh //gitfileshare.file.core.windows.net/test-cache gitfileshare "$GITFILESHAREPWD" "$HOME/test-cache" || exit 1 + + sudo apt-get update && + sudo apt-get install -y coccinelle && + + export jobname=StaticAnalysis && + + ci/run-static-analysis.sh || exit 1 + + test "$GITFILESHAREPWD" = '$(gitfileshare.pwd)' || sudo umount "$HOME/test-cache" || exit 1 + displayName: 'ci/run-static-analysis.sh' + env: + GITFILESHAREPWD: $(gitfileshare.pwd) + +- job: documentation + displayName: Documentation + condition: succeeded() + pool: Hosted Ubuntu 1604 + steps: + - bash: | + test "$GITFILESHAREPWD" = '$(gitfileshare.pwd)' || ci/mount-fileshare.sh //gitfileshare.file.core.windows.net/test-cache gitfileshare "$GITFILESHAREPWD" "$HOME/test-cache" || exit 1 + + sudo apt-get update && + sudo apt-get install -y asciidoc xmlto asciidoctor && + + export ALREADY_HAVE_ASCIIDOCTOR=yes. && + export jobname=Documentation && + + ci/test-documentation.sh || exit 1 + + test "$GITFILESHAREPWD" = '$(gitfileshare.pwd)' || sudo umount "$HOME/test-cache" || exit 1 + displayName: 'ci/test-documentation.sh' + env: + GITFILESHAREPWD: $(gitfileshare.pwd) diff --git a/ci/lib.sh b/ci/lib.sh index a9eb4f4eae..91cf1402bf 100755 --- a/ci/lib.sh +++ b/ci/lib.sh @@ -42,7 +42,7 @@ then BREW_INSTALL_PACKAGES= export GIT_PROVE_OPTS="--timer --jobs 10 --state=failed,slow,save" - export GIT_TEST_OPTS="--quiet --write-junit-xml" + export GIT_TEST_OPTS="--verbose-log -x --write-junit-xml" fi skip_branch_tip_with_tag () { diff --git a/ci/mount-fileshare.sh b/ci/mount-fileshare.sh new file mode 100755 index 0000000000..26b58a8096 --- /dev/null +++ b/ci/mount-fileshare.sh @@ -0,0 +1,25 @@ +#!/bin/sh + +die () { + echo "$*" >&2 + exit 1 +} + +test $# = 4 || +die "Usage: $0 " + +mkdir -p "$4" || die "Could not create $4" + +case "$(uname -s)" in +Linux) + sudo mount -t cifs -o vers=3.0,username="$2",password="$3",dir_mode=0777,file_mode=0777,serverino "$1" "$4" + ;; +Darwin) + pass="$(echo "$3" | sed -e 's/\//%2F/g' -e 's/+/%2B/g')" && + mount -t smbfs,soft "smb://$2:$pass@${1#//}" "$4" + ;; +*) + die "No support for $(uname -s)" + ;; +esac || +die "Could not mount $4" From patchwork Wed Jan 23 14:40:21 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Schindelin via GitGitGadget X-Patchwork-Id: 10777319 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 4DA3A746 for ; Wed, 23 Jan 2019 14:40:45 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 38A652C8F2 for ; Wed, 23 Jan 2019 14:40:45 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 2D3BF2C9D9; Wed, 23 Jan 2019 14:40:45 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 86D772C8F2 for ; Wed, 23 Jan 2019 14:40:44 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726798AbfAWOkn (ORCPT ); Wed, 23 Jan 2019 09:40:43 -0500 Received: from mail-ed1-f50.google.com ([209.85.208.50]:38561 "EHLO mail-ed1-f50.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727156AbfAWOkZ (ORCPT ); Wed, 23 Jan 2019 09:40:25 -0500 Received: by mail-ed1-f50.google.com with SMTP id h50so1881298ede.5 for ; Wed, 23 Jan 2019 06:40:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=date:message-id:in-reply-to:references:from:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=sWRntq7x9k3MAHX6XyhqaSg3cYHmaN4xC2+w1hNZ+fo=; b=VcgIedq+IHAobfjmRgreWKCENGEYNiN1OLpd4ZVrVGwOcI4jtdqHUBlqrPx3QryxLA O1wFio0qYn2g+leSjJGQfcMN6mnPEUv5NQVV5uX0wpVwBXvhDHzzQyT7sZjNfl6CJ0EX kNDHiK9YYgbpZvYL888y22BX3IHzEOx4U5i5/X+mdxGjuwiuzTpvMBLmEQKSsQzCxxr7 zO1c2SbpB3YH2MbiU3v9X1pRuRFa8SyCafnxx6XQULexAo6VLh9ilAm/h6noeGrEwQd4 MCcHZ4LQJqOvJ9BXrchqU3oAXUjn4ksHijcYb7Uwgey+MUw6h4s4cIOK5Dve+t5AvjWe DPlw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:message-id:in-reply-to:references:from :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=sWRntq7x9k3MAHX6XyhqaSg3cYHmaN4xC2+w1hNZ+fo=; b=QoMTGQie986RDZKOsMUbB/0QdfHRxw6zDFOZ7H35XRBk5CW8LmQ4382TmVZj+NrMs1 uhq+xI2tAcGJ5LiBikHFy3XM/eeImrdPzt86r2Si7d1Kmc4YpuEvccme2Tmakuyiw9x2 ZbqSrXF9AuwCbAbcTO9mAJf9JWoSpr0QDJENOwi0lPBowGdQnHRMtzUztjwxxioKG0Wi 6mLQxEzt7i03Ku84t7l/T2whKphrASMcuHZ8ZGZZmkbhK/W1Gh98qEG6pP35ECUB3qEY 8ESRenyTu4Q88MrAegTD/C8qh5i91UevzTHVeCr+at/T6InwlPWvvb9Fgf2w/x0Hs0jq jSbw== X-Gm-Message-State: AJcUukfbgMbgqggknb/gg4YukKcJDER/wzo99ATYKPsVAyUgVBjtgW2L WGN3ck2oHqgxI3jiUyw59+boYlF4 X-Google-Smtp-Source: ALg8bN7zWtb5U54y3XoetT8bNIBI/ZL0dvGq7TS1W6hDB5Csrd0XkvBEh6H4tgB6G4zCQOGpzuRzUw== X-Received: by 2002:a17:906:55a:: with SMTP id k26-v6mr2733542eja.218.1548254422441; Wed, 23 Jan 2019 06:40:22 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id k12sm47203ejz.4.2019.01.23.06.40.21 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 23 Jan 2019 06:40:21 -0800 (PST) Date: Wed, 23 Jan 2019 06:40:21 -0800 (PST) X-Google-Original-Date: Wed, 23 Jan 2019 14:40:01 GMT Message-Id: <5bdc6a08a8b8040de3082b1690f16538fcc08682.1548254412.git.gitgitgadget@gmail.com> In-Reply-To: References: From: "Johannes Schindelin via GitGitGadget" Subject: [PATCH v4 10/21] ci: move the Windows job to the top Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Junio C Hamano , Johannes Schindelin Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Johannes Schindelin The Windows job currently takes a whopping ~1h20m to complete. Which is *far* longer than the next-longest job takes (linux-gcc, ~35m). As such, it makes sense to start the Windows job first, to minimize the overall run time (which is now pretty safely the run time of the Windows job). This affects only the Azure Pipelines configuration, not the Travis one, of course, as Travis cannot run our Windows job: 1h20m is distinctly longer than the 50 minute timeout of Travis' free tier. This commit is best viewed with `--color-moved`. Signed-off-by: Johannes Schindelin --- azure-pipelines.yml | 172 ++++++++++++++++++++++---------------------- 1 file changed, 86 insertions(+), 86 deletions(-) diff --git a/azure-pipelines.yml b/azure-pipelines.yml index f3cabb0dd0..e44d2733a4 100644 --- a/azure-pipelines.yml +++ b/azure-pipelines.yml @@ -3,6 +3,92 @@ resources: fetchDepth: 1 jobs: +- job: windows + displayName: Windows + condition: succeeded() + pool: Hosted + timeoutInMinutes: 240 + steps: + - powershell: | + if ("$GITFILESHAREPWD" -ne "" -and "$GITFILESHAREPWD" -ne "`$`(gitfileshare.pwd)") { + net use s: \\gitfileshare.file.core.windows.net\test-cache "$GITFILESHAREPWD" /user:AZURE\gitfileshare /persistent:no + cmd /c mklink /d "$(Build.SourcesDirectory)\test-cache" S:\ + } + displayName: 'Mount test-cache' + env: + GITFILESHAREPWD: $(gitfileshare.pwd) + - powershell: | + # Helper to check the error level of the latest command (exit with error when appropriate) + function c() { if (!$?) { exit(1) } } + + # Add build agent's MinGit to PATH + $env:PATH = $env:AGENT_HOMEDIRECTORY +"\externals\\git\cmd;" +$env:PATH + + # Helper to initialize (or update) a Git worktree + function init ($path, $url, $set_origin) { + if (Test-Path $path) { + cd $path; c + if (Test-Path .git) { + & git init; c + } else { + & git status + } + } else { + & git init $path; c + cd $path; c + } + & git config core.autocrlf false; c + & git config core.untrackedCache true; c + if (($set_origin -ne 0) -and !(git config remote.origin.url)) { + & git remote add origin $url; c + } + & git fetch --depth=1 $url master; c + & git reset --hard FETCH_HEAD; c + & git clean -df; c + } + + # Initialize Git for Windows' SDK + $sdk_path = "$(Build.SourcesDirectory)\git-sdk-64" + init "$sdk_path" "https://dev.azure.com/git-for-windows/git-sdk-64/_git/git-sdk-64" 0 + + # Let Git ignore the SDK and the test-cache + "/git-sdk-64/`n/test-cache/`n" | Out-File -NoNewLine -Encoding ascii -Append "$(Build.SourcesDirectory)\.git\info\exclude" + displayName: 'Initialize the Git for Windows SDK' + - powershell: | + & "git-sdk-64\git-cmd.exe" --command=usr\\bin\\bash.exe -lc @" + export MAKEFLAGS=-j10 + export DEVELOPER=1 + export NO_PERL=1 + export NO_SVN_TESTS=1 + export GIT_TEST_SKIP_REBASE_P=1 + + ci/run-build-and-tests.sh || { + ci/print-test-failures.sh + exit 1 + } + "@ + if (!$?) { exit(1) } + displayName: 'Build & Test' + env: + HOME: $(Build.SourcesDirectory) + MSYSTEM: MINGW64 + - powershell: | + if ("$GITFILESHAREPWD" -ne "" -and "$GITFILESHAREPWD" -ne "`$`(gitfileshare.pwd)") { + cmd /c rmdir "$(Build.SourcesDirectory)\test-cache" + } + displayName: 'Unmount test-cache' + condition: true + env: + GITFILESHAREPWD: $(gitfileshare.pwd) + - task: PublishTestResults@2 + displayName: 'Publish Test Results **/TEST-*.xml' + inputs: + mergeTestResults: true + testRunTitle: 'windows' + platform: Windows + publishRunAttachments: false + condition: succeededOrFailed() + - job: linux_clang displayName: linux-clang condition: succeeded() @@ -153,92 +239,6 @@ jobs: publishRunAttachments: false condition: succeededOrFailed() -- job: windows - displayName: Windows - condition: succeeded() - pool: Hosted - timeoutInMinutes: 240 - steps: - - powershell: | - if ("$GITFILESHAREPWD" -ne "" -and "$GITFILESHAREPWD" -ne "`$`(gitfileshare.pwd)") { - net use s: \\gitfileshare.file.core.windows.net\test-cache "$GITFILESHAREPWD" /user:AZURE\gitfileshare /persistent:no - cmd /c mklink /d "$(Build.SourcesDirectory)\test-cache" S:\ - } - displayName: 'Mount test-cache' - env: - GITFILESHAREPWD: $(gitfileshare.pwd) - - powershell: | - # Helper to check the error level of the latest command (exit with error when appropriate) - function c() { if (!$?) { exit(1) } } - - # Add build agent's MinGit to PATH - $env:PATH = $env:AGENT_HOMEDIRECTORY +"\externals\\git\cmd;" +$env:PATH - - # Helper to initialize (or update) a Git worktree - function init ($path, $url, $set_origin) { - if (Test-Path $path) { - cd $path; c - if (Test-Path .git) { - & git init; c - } else { - & git status - } - } else { - & git init $path; c - cd $path; c - } - & git config core.autocrlf false; c - & git config core.untrackedCache true; c - if (($set_origin -ne 0) -and !(git config remote.origin.url)) { - & git remote add origin $url; c - } - & git fetch --depth=1 $url master; c - & git reset --hard FETCH_HEAD; c - & git clean -df; c - } - - # Initialize Git for Windows' SDK - $sdk_path = "$(Build.SourcesDirectory)\git-sdk-64" - init "$sdk_path" "https://dev.azure.com/git-for-windows/git-sdk-64/_git/git-sdk-64" 0 - - # Let Git ignore the SDK and the test-cache - "/git-sdk-64/`n/test-cache/`n" | Out-File -NoNewLine -Encoding ascii -Append "$(Build.SourcesDirectory)\.git\info\exclude" - displayName: 'Initialize the Git for Windows SDK' - - powershell: | - & "git-sdk-64\git-cmd.exe" --command=usr\\bin\\bash.exe -lc @" - export MAKEFLAGS=-j10 - export DEVELOPER=1 - export NO_PERL=1 - export NO_SVN_TESTS=1 - export GIT_TEST_SKIP_REBASE_P=1 - - ci/run-build-and-tests.sh || { - ci/print-test-failures.sh - exit 1 - } - "@ - if (!$?) { exit(1) } - displayName: 'Build & Test' - env: - HOME: $(Build.SourcesDirectory) - MSYSTEM: MINGW64 - - powershell: | - if ("$GITFILESHAREPWD" -ne "" -and "$GITFILESHAREPWD" -ne "`$`(gitfileshare.pwd)") { - cmd /c rmdir "$(Build.SourcesDirectory)\test-cache" - } - displayName: 'Unmount test-cache' - condition: true - env: - GITFILESHAREPWD: $(gitfileshare.pwd) - - task: PublishTestResults@2 - displayName: 'Publish Test Results **/TEST-*.xml' - inputs: - mergeTestResults: true - testRunTitle: 'windows' - platform: Windows - publishRunAttachments: false - condition: succeededOrFailed() - - job: linux32 displayName: Linux32 condition: succeeded() From patchwork Wed Jan 23 14:40:22 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Schindelin via GitGitGadget X-Patchwork-Id: 10777295 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id BC73D1390 for ; Wed, 23 Jan 2019 14:40:29 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A9F092C8F2 for ; Wed, 23 Jan 2019 14:40:29 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 9E4B02C9D9; Wed, 23 Jan 2019 14:40:29 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 2E60B2C8F2 for ; Wed, 23 Jan 2019 14:40:29 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727198AbfAWOk2 (ORCPT ); Wed, 23 Jan 2019 09:40:28 -0500 Received: from mail-ed1-f42.google.com ([209.85.208.42]:42648 "EHLO mail-ed1-f42.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727159AbfAWOkZ (ORCPT ); Wed, 23 Jan 2019 09:40:25 -0500 Received: by mail-ed1-f42.google.com with SMTP id y20so1864448edw.9 for ; Wed, 23 Jan 2019 06:40:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=date:message-id:in-reply-to:references:from:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=sI0J8sK8qLVY5A7TwFi50tscdkZxAfA3T+dfrtp7SBQ=; b=ZJYdmtqX0TqfEcbIzGHWSycEGvC7AmFFPkqqOGCeZUb4l6gwx1BHPc/8B0DFWRUTeR IrqTsLZYmVvY84y4E6fSqx4FIsRd3RwPcuQFhEJEEJilRnRDYRgfTTn1p9U8JMpaXEdc +WlckplvgR+dyZ5MBI1b4o3/52jxGP4GYqj4GDLgjCyhJislqEuGZxotacKK88uoq1ad Qu5T6POM/8FojuOhRFowC36ZJDQjbBYhpaHJioGEpJjJptMFt4dHmuJj32/rkFKUNAsT 36o6p5e8ByCgyZvZP+8YpyrReJqu3/cp601bYtozuOCJ8BFvdsC61+l2yfmBcP8wRtOU NC3w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:message-id:in-reply-to:references:from :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=sI0J8sK8qLVY5A7TwFi50tscdkZxAfA3T+dfrtp7SBQ=; b=jKgdqYyOGvm3mRlRAq/0p1sRmcbKkq+RVwarezwcU8lwHnu44kKyq9dq9m2xED5vrX BBeK+tMvmTrCEC8D5FJWf+BUx3XjMAy+Vh8TJkCe4Mch+CSVgh7sQp3C8xdl1YSl2ADI q630qjzVZG+FJ+lpP9/R4z6ZrtfVGdCs7kWZhmEid+ghh4SWVIVNybbpD7oCmSENHukg wsqXN8OUXCyc7e1cqF0PguYM9+yedZIPx14crXsPv2tbjzxwY9mGQqbJf36LmhLwNfbt ORHhZn3gpMx4u5oADgmXINagWBg4QKKNhKAd1DjOF+7eh6CtT9/78e9Yd0EO5DBECiaw cZjg== X-Gm-Message-State: AJcUukfNn7lOUSEMmkLwNPrJxNq8f9sfP7uf0+VexsLF4om929gbwmkn 0qClIMDhSUzWYx8IKDtl+guz0Eao X-Google-Smtp-Source: ALg8bN7fpZJEgskHXIMxQvadsRh4SAF+09gTeUjyIx3Edm0vM2bXGIzB32hY+TBaMpwwpSCW2U6k5Q== X-Received: by 2002:a17:906:4584:: with SMTP id t4mr751235ejq.135.1548254423168; Wed, 23 Jan 2019 06:40:23 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id r51sm10559209eda.64.2019.01.23.06.40.22 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 23 Jan 2019 06:40:22 -0800 (PST) Date: Wed, 23 Jan 2019 06:40:22 -0800 (PST) X-Google-Original-Date: Wed, 23 Jan 2019 14:40:02 GMT Message-Id: <93eebb74ce1e5a2dcacee40c5e1750979bd83b10.1548254412.git.gitgitgadget@gmail.com> In-Reply-To: References: From: "Johannes Schindelin via GitGitGadget" Subject: [PATCH v4 11/21] ci: use git-sdk-64-minimal build artifact Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Junio C Hamano , Johannes Schindelin Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Johannes Schindelin Instead of a shallow fetch followed by a sparse checkout, we are better off by using a separate, dedicated Pipeline that bundles the SDK as a build artifact, and then consuming that build artifact here. In fact, since this artifact will be used a lot, we spent substantial time on figuring out a minimal subset of the Git for Windows SDK, just enough to build and test Git. The result is a size reduction from around 1GB (compressed) to around 55MB (compressed). This also comes with the change where we now call `usr\bin\bash.exe` directly, as `git-cmd.exe` is not included in the minimal SDK. That reduces the time to initialize Git for Windows' SDK from anywhere between 2m30s-7m to a little over 1m. Note: in theory, we could also use the DownloadBuildArtifacts@0 task here. However, restricted permissions that are in effect when building from forks would let this fail for PR builds, defeating the whole purpose of the Azure Pipelines support for git.git. Signed-off-by: Johannes Schindelin --- azure-pipelines.yml | 44 +++++++++----------------------------------- 1 file changed, 9 insertions(+), 35 deletions(-) diff --git a/azure-pipelines.yml b/azure-pipelines.yml index e44d2733a4..3085ff6ace 100644 --- a/azure-pipelines.yml +++ b/azure-pipelines.yml @@ -18,44 +18,18 @@ jobs: env: GITFILESHAREPWD: $(gitfileshare.pwd) - powershell: | - # Helper to check the error level of the latest command (exit with error when appropriate) - function c() { if (!$?) { exit(1) } } - - # Add build agent's MinGit to PATH - $env:PATH = $env:AGENT_HOMEDIRECTORY +"\externals\\git\cmd;" +$env:PATH - - # Helper to initialize (or update) a Git worktree - function init ($path, $url, $set_origin) { - if (Test-Path $path) { - cd $path; c - if (Test-Path .git) { - & git init; c - } else { - & git status - } - } else { - & git init $path; c - cd $path; c - } - & git config core.autocrlf false; c - & git config core.untrackedCache true; c - if (($set_origin -ne 0) -and !(git config remote.origin.url)) { - & git remote add origin $url; c - } - & git fetch --depth=1 $url master; c - & git reset --hard FETCH_HEAD; c - & git clean -df; c - } - - # Initialize Git for Windows' SDK - $sdk_path = "$(Build.SourcesDirectory)\git-sdk-64" - init "$sdk_path" "https://dev.azure.com/git-for-windows/git-sdk-64/_git/git-sdk-64" 0 + $urlbase = "https://dev.azure.com/git-for-windows/git/_apis/build/builds" + $id = ((Invoke-WebRequest -UseBasicParsing "${urlbase}?definitions=22&statusFilter=completed&resultFilter=succeeded&`$top=1").content | ConvertFrom-JSON).value[0].id + $downloadUrl = ((Invoke-WebRequest -UseBasicParsing "${urlbase}/$id/artifacts").content | ConvertFrom-JSON).value[1].resource.downloadUrl + (New-Object Net.WebClient).DownloadFile($downloadUrl,"git-sdk-64-minimal.zip") + Expand-Archive git-sdk-64-minimal.zip -DestinationPath . -Force + Remove-Item git-sdk-64-minimal.zip # Let Git ignore the SDK and the test-cache - "/git-sdk-64/`n/test-cache/`n" | Out-File -NoNewLine -Encoding ascii -Append "$(Build.SourcesDirectory)\.git\info\exclude" - displayName: 'Initialize the Git for Windows SDK' + "/git-sdk-64-minimal/`n/test-cache/`n" | Out-File -NoNewLine -Encoding ascii -Append "$(Build.SourcesDirectory)\.git\info\exclude" + displayName: 'Download git-sdk-64-minimal' - powershell: | - & "git-sdk-64\git-cmd.exe" --command=usr\\bin\\bash.exe -lc @" + & git-sdk-64-minimal\usr\bin\bash.exe -lc @" export MAKEFLAGS=-j10 export DEVELOPER=1 export NO_PERL=1 From patchwork Wed Jan 23 14:40:23 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Schindelin via GitGitGadget X-Patchwork-Id: 10777315 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 72D511515 for ; Wed, 23 Jan 2019 14:40:43 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 62A252C8F2 for ; Wed, 23 Jan 2019 14:40:43 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 56D9F2C9D9; Wed, 23 Jan 2019 14:40:43 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A2C542C9D6 for ; Wed, 23 Jan 2019 14:40:41 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726880AbfAWOk1 (ORCPT ); Wed, 23 Jan 2019 09:40:27 -0500 Received: from mail-ed1-f52.google.com ([209.85.208.52]:37279 "EHLO mail-ed1-f52.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727172AbfAWOk0 (ORCPT ); Wed, 23 Jan 2019 09:40:26 -0500 Received: by mail-ed1-f52.google.com with SMTP id h15so1880463edb.4 for ; Wed, 23 Jan 2019 06:40:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=date:message-id:in-reply-to:references:from:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=Nx9l6UGI/Mo+cXJbFVbhoaNDqQugjJei4UPiehgiQbs=; b=S93LwYKCKmAa+Tq8i9U6NGSu+dBuhLRKQzXl13nNNYoonrhHixyF4BHx4gjayDCxM0 YJHOyKS6yb+ygB6CddRkB5b75tBP+ej4bfAXg8oEf+Q09S8TbPZhmapkSU3l3f/JeZie jFfuEzWVOXy3h5X4iAX5T6PGqGIlPtsmp6Ekvgp4ktA841isUrjkcKSF526f66PkwSR7 I3ja5SQClidDN+1RaeTGNAj/f+3RiOF3o9RTd9h6EU7//dQFboi4dBXp2cLwyqfc1BU4 6dhAOt+PywAT/8r/8UE3Kfrg7uge/hTNab8pn8NvNvRV9PgwpNjkqDLaDD1kuj3BqN5o ruNg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:message-id:in-reply-to:references:from :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=Nx9l6UGI/Mo+cXJbFVbhoaNDqQugjJei4UPiehgiQbs=; b=UF1LxxX0PLoLvdsIwez3sfrNBbBXBW8oOrfX5YtAYgesWjd215+ew8TNuGbdpmK9Hq 1aFin+LlC0eWiXpsEBU/x3xxvBxok4AGCY3XV5ZeNz0vOSn3sRiKQVlT5GQCmeQYGSMF 3ihvexi8/ByvhTwmv64w8ktSmBQtw6JCtNUmCQLXG/7t6kK9JgP3q8VOL5IaxhIBdJk8 H8lEtK8W8zFdD8zUEN/+Uc63vjIzVQgc7+X0r9jI2+oiVdCMba4CG9MyGdaTm7wg3OqA /OMDsnYdkM77QVcNQ2umEsGG3Awh8Z0UGvcOy+63ES9XytStGFczqm5mGBaeekT8OgjA 0sLg== X-Gm-Message-State: AJcUukfcaTqQS59XxeF8utCamDuK/oqxjzHLbYKfxZhEC4FHCPlhQtsG IONyEqwRydkSE6vFY+kBLPvXvdre X-Google-Smtp-Source: ALg8bN4J5a/mNXSQzx1pcnS8LhbiaxmmHfMLnD3WeV7IkRuxcMS06iCjL7J+4p2ILK4LjKdjIiWkgA== X-Received: by 2002:a50:9784:: with SMTP id e4mr2956798edb.165.1548254424037; Wed, 23 Jan 2019 06:40:24 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id g5-v6sm5587967ejm.15.2019.01.23.06.40.23 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 23 Jan 2019 06:40:23 -0800 (PST) Date: Wed, 23 Jan 2019 06:40:23 -0800 (PST) X-Google-Original-Date: Wed, 23 Jan 2019 14:40:03 GMT Message-Id: <18d76823e56547e47be39ad657a1795603659593.1548254412.git.gitgitgadget@gmail.com> In-Reply-To: References: From: "Johannes Schindelin via GitGitGadget" Subject: [PATCH v4 12/21] mingw: be more generous when wrapping up the setitimer() emulation Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Junio C Hamano , Johannes Schindelin Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Johannes Schindelin Every once in a while, the Azure Pipeline fails with some semi-random error: timer thread did not terminate timely This error message means that the thread that is used to emulate the setitimer() function did not terminate within 1,000 milliseconds. The most likely explanation (and therefore the one we should assume to be true, according to Occam's Razor) is that the timeout of one second is simply not enough because we try to run so many tasks in parallel. So let's give it ten seconds instead of only one. That should be enough. Signed-off-by: Johannes Schindelin --- compat/mingw.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/compat/mingw.c b/compat/mingw.c index b459e1a291..e0dfe8844d 100644 --- a/compat/mingw.c +++ b/compat/mingw.c @@ -2100,7 +2100,7 @@ static void stop_timer_thread(void) if (timer_event) SetEvent(timer_event); /* tell thread to terminate */ if (timer_thread) { - int rc = WaitForSingleObject(timer_thread, 1000); + int rc = WaitForSingleObject(timer_thread, 10000); if (rc == WAIT_TIMEOUT) error("timer thread did not terminate timely"); else if (rc != WAIT_OBJECT_0) From patchwork Wed Jan 23 14:40:24 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Schindelin via GitGitGadget X-Patchwork-Id: 10777317 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 57262746 for ; Wed, 23 Jan 2019 14:40:44 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 45D332C9D6 for ; Wed, 23 Jan 2019 14:40:44 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 3A1BF2C8F2; Wed, 23 Jan 2019 14:40:44 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C87D02C8F2 for ; Wed, 23 Jan 2019 14:40:43 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726632AbfAWOkl (ORCPT ); Wed, 23 Jan 2019 09:40:41 -0500 Received: from mail-ed1-f54.google.com ([209.85.208.54]:41420 "EHLO mail-ed1-f54.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727161AbfAWOk0 (ORCPT ); Wed, 23 Jan 2019 09:40:26 -0500 Received: by mail-ed1-f54.google.com with SMTP id a20so1862478edc.8 for ; Wed, 23 Jan 2019 06:40:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=date:message-id:in-reply-to:references:from:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=yIFmvH39JP873kVuTr99FvXDnugqBCZqKax9pCZMHbE=; b=KkvXPAOaUzCUowYT6x1+JBShs+ZWnq2OUOD5ukrQYhijAtrpCnyI/4VCSGULBZGQki LH+KcUt+0+PdB2hwFTSuEIw0HCHASr9stTSwE7qNA+RdPnXDzeVVwdCTa7G0kR6EmEWn wc0jSvR7GAErodYotlypTSV0USI633eUOVMwSxJjoubGBEDI4gCpjTJ3SwhwpMxjGjLY SiPawTi+S8+Lz7Gx+5XbBQPAO0Xim4ksOOVacmrylHGoVGUKbtI2oPHc7urgO6EPdy9x FjiAJ7ZarcYBGigx1VIBGMfA3ukTKGE4AwU587HnjqGdCL6QyXoHaMpDqyqdxrqeR+1L kE6w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:message-id:in-reply-to:references:from :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=yIFmvH39JP873kVuTr99FvXDnugqBCZqKax9pCZMHbE=; b=m9uaFQlwkFyzfNIUxVxDN6GcOnMbEHNoz3gSWxV6fPfLBybkof02sdLm8DeHAfSUNK zWFdogS+G1+UfGqLYZ77+rbuMZJz67lodwKEwwffTHKacOVkxgvcz/YmkjdOSb9KWgps cubiMI9eih3lQPwk300LZ/L5Hh2e1LTPysZyIomBxgD9sv+tOlvjd0skOwhk6rbRIZRc uIo3L6FQ2zqOcChOw+Ys/5A6sAdd5s+IOG/WWo/J7AgU7u13w4YXQS09r58rWzgvph+g 22uN8obJ2TLEr+d5255hWB57FyUYlp4I8YjUJZKfwOhoYXcewMHGvHeXNB/nnhcUzrEW YWtQ== X-Gm-Message-State: AJcUukfgL2ke6NbWCpVD7fUneqEWiSBp91hBtys1RrAYP+tFSRicXzcr vPVIS4ryg3JaVAj9kvcRo3a3j/J8 X-Google-Smtp-Source: ALg8bN7aikDSyUtNpiRRrgeKGicuAU/28VFawNP0WbnaaNxYZrnSGJnDkpJT7j7Rd8H28qY7bHkGqw== X-Received: by 2002:a50:d6c5:: with SMTP id l5mr2972368edj.145.1548254425013; Wed, 23 Jan 2019 06:40:25 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id n16-v6sm5515410eja.6.2019.01.23.06.40.24 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 23 Jan 2019 06:40:24 -0800 (PST) Date: Wed, 23 Jan 2019 06:40:24 -0800 (PST) X-Google-Original-Date: Wed, 23 Jan 2019 14:40:04 GMT Message-Id: <5d593acee5742a82c74f24f8e3f04ba3a8bc6d42.1548254412.git.gitgitgadget@gmail.com> In-Reply-To: References: From: "Johannes Schindelin via GitGitGadget" Subject: [PATCH v4 13/21] README: add a build badge (status of the Azure Pipelines build) Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Junio C Hamano , Johannes Schindelin Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Johannes Schindelin Just like so many other OSS projects, we now also have a build badge. Signed-off-by: Johannes Schindelin --- README.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/README.md b/README.md index f920a42fad..764c480c66 100644 --- a/README.md +++ b/README.md @@ -1,3 +1,5 @@ +[![Build Status](https://dev.azure.com/git/git/_apis/build/status/test-git.git)](https://dev.azure.com/git/git/_build/latest?definitionId=2) + Git - fast, scalable, distributed revision control system ========================================================= From patchwork Wed Jan 23 14:40:25 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Schindelin via GitGitGadget X-Patchwork-Id: 10777297 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 6946E746 for ; Wed, 23 Jan 2019 14:40:31 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 58A0F2C8F2 for ; Wed, 23 Jan 2019 14:40:31 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 4CC792C9D9; Wed, 23 Jan 2019 14:40:31 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D3F792C8F2 for ; Wed, 23 Jan 2019 14:40:30 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726960AbfAWOk3 (ORCPT ); Wed, 23 Jan 2019 09:40:29 -0500 Received: from mail-ed1-f50.google.com ([209.85.208.50]:37280 "EHLO mail-ed1-f50.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727182AbfAWOk1 (ORCPT ); Wed, 23 Jan 2019 09:40:27 -0500 Received: by mail-ed1-f50.google.com with SMTP id h15so1880525edb.4 for ; Wed, 23 Jan 2019 06:40:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=date:message-id:in-reply-to:references:from:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=qFU9SlpnDsxWdLk7t1RD4T2tVhhrbl/R3qfgIkC/7lI=; b=LS2nl8XhZuSt04amVh11fNE6ktVe4idpSWO/2WSZgy1bYVmnmnHV2Tv5LAuQdUWTjk sZITMeMUX6fdgzNG3/bESGht0hofhBuvmQZyRZq4yWbhMQeETFT1KfghiWTkBMj+sNef LvUoBCYc8uiBzxCySfUwExuz3snrB9KGFzN1GJyg+Rjt9K+g3RO8bDvUsWg7Fhi3qfMC /RfLB95N8848Ws/0NU8Wo7Myf87aS3/vPSsaRWMq9UgbgA4d5hTkYW1xbxzck8XuOTCh 2TcRHQpZB8JywU5ksnQA+PY5UFxBY5O5M1OcMAV7pLOufXmHCKUNmOeDQOlHDX/ZWxnz j2wg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:message-id:in-reply-to:references:from :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=qFU9SlpnDsxWdLk7t1RD4T2tVhhrbl/R3qfgIkC/7lI=; b=JEU614/rzZq49HZ6k6asP6DUbA5v/kC8gXVlrcU5g8AGbPS+0WRCrGMGZz4tQ2HlHW Ui3uecaE3V2KKu662bGimomvdKvjwL9m4N+Wgu3tO7fj5JJZOKhXLU3WmIBA8iZF1l85 1WrXdpA3p/5m26lEXmqqwvRNnjBd4jCGmg+fCnzKjZwIz0msVFqcRGpDbbqDSG1XD7+P feJ+7ka2SZgCBMTt29yd+Ql9C6POqcOglrsscl4vCp6DCkXeR+FPjA7pk5MHhdJ4/xZH zLJMGbabAbDO4OlQIIY38EC9Bcm2jhzTe7R96VsFDU89iEMGXA9Z3lPEd4KRAUsuTTvS IkjQ== X-Gm-Message-State: AJcUukf1gPtsvSYMtJRWSmsar29SdIn9rzNmbALDxYyns/ObUm/y11H2 jPCcCxq/OtTGm1rk6iE1cOcn6SxJ X-Google-Smtp-Source: ALg8bN5zeOQtx3ghlEXzx64BcVecsRsP5NAQoyCGasyrW1SJlBfPxnIbvfPQD67W6LArQhrLOrJdlA== X-Received: by 2002:a50:9063:: with SMTP id z32mr2984330edz.133.1548254425909; Wed, 23 Jan 2019 06:40:25 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id n23-v6sm5527651ejx.57.2019.01.23.06.40.25 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 23 Jan 2019 06:40:25 -0800 (PST) Date: Wed, 23 Jan 2019 06:40:25 -0800 (PST) X-Google-Original-Date: Wed, 23 Jan 2019 14:40:05 GMT Message-Id: <3083041e33a3b0c63af621e9b6bc704e288c401f.1548254412.git.gitgitgadget@gmail.com> In-Reply-To: References: From: "Johannes Schindelin via GitGitGadget" Subject: [PATCH v4 14/21] tests: avoid calling Perl just to determine file sizes Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Junio C Hamano , Johannes Schindelin Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Johannes Schindelin It is a bit ridiculous to spin up a full-blown Perl instance (especially on Windows, where that means spinning up a full POSIX emulation layer, AKA the MSYS2 runtime) just to tell how large a given file is. So let's just use the test-tool to do that job instead. This command will also be used over the next commits, to allow for cutting out individual test cases' verbose log from the file generated via --verbose-log. Signed-off-by: Johannes Schindelin --- t/helper/test-path-utils.c | 12 ++++++++++++ t/t0021-conversion.sh | 2 +- t/t1050-large.sh | 2 +- t/t5315-pack-objects-compression.sh | 2 +- t/t9303-fast-import-compression.sh | 2 +- 5 files changed, 16 insertions(+), 4 deletions(-) diff --git a/t/helper/test-path-utils.c b/t/helper/test-path-utils.c index ae091d9b3e..30211d6d64 100644 --- a/t/helper/test-path-utils.c +++ b/t/helper/test-path-utils.c @@ -291,6 +291,18 @@ int cmd__path_utils(int argc, const char **argv) return !!res; } + if (argc > 2 && !strcmp(argv[1], "file-size")) { + int res = 0, i; + struct stat st; + + for (i = 2; i < argc; i++) + if (stat(argv[i], &st)) + res = error_errno("Cannot stat '%s'", argv[i]); + else + printf("%"PRIuMAX"\n", (uintmax_t)st.st_size); + return !!res; + } + fprintf(stderr, "%s: unknown function name: %s\n", argv[0], argv[1] ? argv[1] : "(there was none)"); return 1; diff --git a/t/t0021-conversion.sh b/t/t0021-conversion.sh index fd5f1ac649..e10f5f787f 100755 --- a/t/t0021-conversion.sh +++ b/t/t0021-conversion.sh @@ -24,7 +24,7 @@ generate_random_characters () { } file_size () { - perl -e 'print -s $ARGV[0]' "$1" + test-tool path-utils file-size "$1" } filter_git () { diff --git a/t/t1050-large.sh b/t/t1050-large.sh index 1a9b21b293..dcb4dbba67 100755 --- a/t/t1050-large.sh +++ b/t/t1050-large.sh @@ -8,7 +8,7 @@ test_description='adding and checking out large blobs' # This should be moved to test-lib.sh together with the # copy in t0021 after both topics have graduated to 'master'. file_size () { - perl -e 'print -s $ARGV[0]' "$1" + test-tool path-utils file-size "$1" } test_expect_success setup ' diff --git a/t/t5315-pack-objects-compression.sh b/t/t5315-pack-objects-compression.sh index 34c47dae09..df970d7584 100755 --- a/t/t5315-pack-objects-compression.sh +++ b/t/t5315-pack-objects-compression.sh @@ -7,7 +7,7 @@ test_description='pack-object compression configuration' # This should be moved to test-lib.sh together with the # copy in t0021 after both topics have graduated to 'master'. file_size () { - perl -e 'print -s $ARGV[0]' "$1" + test-tool path-utils file-size "$1" } test_expect_success setup ' diff --git a/t/t9303-fast-import-compression.sh b/t/t9303-fast-import-compression.sh index 856219f46a..5045f02a53 100755 --- a/t/t9303-fast-import-compression.sh +++ b/t/t9303-fast-import-compression.sh @@ -6,7 +6,7 @@ test_description='compression setting of fast-import utility' # This should be moved to test-lib.sh together with the # copy in t0021 after both topics have graduated to 'master'. file_size () { - perl -e 'print -s $ARGV[0]' "$1" + test-tool path-utils file-size "$1" } import_large () { From patchwork Wed Jan 23 14:40:26 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Schindelin via GitGitGadget X-Patchwork-Id: 10777299 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B55A0746 for ; Wed, 23 Jan 2019 14:40:33 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A29D92C8F2 for ; Wed, 23 Jan 2019 14:40:33 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 96B322C9D9; Wed, 23 Jan 2019 14:40:33 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D53DB2C8F2 for ; Wed, 23 Jan 2019 14:40:32 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726166AbfAWOkb (ORCPT ); Wed, 23 Jan 2019 09:40:31 -0500 Received: from mail-ed1-f49.google.com ([209.85.208.49]:42662 "EHLO mail-ed1-f49.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727189AbfAWOk3 (ORCPT ); Wed, 23 Jan 2019 09:40:29 -0500 Received: by mail-ed1-f49.google.com with SMTP id y20so1864620edw.9 for ; Wed, 23 Jan 2019 06:40:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=date:message-id:in-reply-to:references:from:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=mErgkvF3RwT1uEOg4N3negTT4Om6xVcCHQyQL+1J7Sw=; b=gEmFnHqAiaM/wISAkqOZOnN7BOG6FcU3lWhJJ94KDJdjeULQb5bXCTxRdsr4W7YKX0 IWutGywfnMWjjAYFgZ22dzusFo7cEM1Vkj7ghCM5sx8rixwO+/ElGZD5Pfh9FJ0qI+QJ cJn9J3N1UT7puVDWWF3bIUDkq54y1eAIdCfT0HXnYSIGRvBX/1dNTC7QvGeE7ApZYFkt 0GnZYxVz58WOgcrqzcDf11mnSAhWrhfCXk/Kj2U1bPK5vcauat4KHr4rxyj2DlVQS5ud eIREMWA1Tk1XHfRtoJNqaK8CpzC2thytpE2886WJDquHR8vtUYs9V7jVfb4rCrEccrOZ DOKw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:message-id:in-reply-to:references:from :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=mErgkvF3RwT1uEOg4N3negTT4Om6xVcCHQyQL+1J7Sw=; b=AxIwxgdV0Jko9cjkzWvXePwkiWmZTDQo0Ua/xbgmgFcmEEz6bgkzkBYxoTizAEeatE gPrs4XRNzZCsgoL81jVBDSwbq9MBvu1Fv1yy02pjt8ieAPLv31nKZaJZqmOPr5GOfer1 dBc4sp3epcrD0QUL7WUmSyw+CiOLp3ZqshR7cn7A6pF6nlHkWsVqnsrTaTJGFDwlRaa4 Qd3LSiqxj5PdjmQvheHpxVR8lm0+Abnr/ijI9qadnqKFE0QhYeD/4aZGAXLtWWo3IJ++ g3VKfGS1L6oC55NX/CpWt0BH7/9+ajlnmcbGlnh3/fdR18GothhQwaBLh5ra+ACZKv/8 TRyQ== X-Gm-Message-State: AJcUukcKd50g98DNBWTloUhgf0/31t5ZUAgiNZeJC8KDNC+bSOkiWwmt luusKZs4/ukycNTwWz/N7S8fan6G X-Google-Smtp-Source: ALg8bN6fMPXfkBjwFBELyPz8+jSWwWPqLtNV/g6LDTaCUJb+o6E53osG8LyLw17wKyWhDauE8/ImgQ== X-Received: by 2002:aa7:dd0f:: with SMTP id i15mr2854475edv.29.1548254426786; Wed, 23 Jan 2019 06:40:26 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id w24-v6sm5751144eja.71.2019.01.23.06.40.26 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 23 Jan 2019 06:40:26 -0800 (PST) Date: Wed, 23 Jan 2019 06:40:26 -0800 (PST) X-Google-Original-Date: Wed, 23 Jan 2019 14:40:06 GMT Message-Id: <1c44d3b8b81946c66839b7e750da2336a7ec4ad3.1548254412.git.gitgitgadget@gmail.com> In-Reply-To: References: From: "Johannes Schindelin via GitGitGadget" Subject: [PATCH v4 15/21] tests: include detailed trace logs with --write-junit-xml upon failure Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Junio C Hamano , Johannes Schindelin Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Johannes Schindelin The JUnit XML format lends itself to be presented in a powerful UI, where you can drill down to the information you are interested in very quickly. For test failures, this usually means that you want to see the detailed trace of the failing tests. With Travis CI, we passed the `--verbose-log` option to get those traces. However, that seems excessive, as we do not need/use the logs in almost all of those cases: only when a test fails do we have a way to include the trace. So let's do something different when using Azure DevOps: let's run all the tests with `--quiet` first, and only if a failure is encountered, try to trace the commands as they are executed. Of course, we cannot turn on `--verbose-log` after the fact. So let's just re-run the test with all the same options, adding `--verbose-log`. And then munging the output file into the JUnit XML on the fly. Note: there is an off chance that re-running the test in verbose mode "fixes" the failures (and this does happen from time to time!). That is a possibility we should be able to live with. Ideally, we would label this as "Passed upon rerun", and Azure Pipelines even know about that outcome, but it is not available when using the JUnit XML format for now: https://github.com/Microsoft/azure-pipelines-agent/blob/master/src/Agent.Worker/TestResults/JunitResultReader.cs Signed-off-by: Johannes Schindelin --- t/helper/test-path-utils.c | 21 +++++++++++++++++++++ t/test-lib.sh | 22 +++++++++++++++++++++- 2 files changed, 42 insertions(+), 1 deletion(-) diff --git a/t/helper/test-path-utils.c b/t/helper/test-path-utils.c index 30211d6d64..6efde6f5ba 100644 --- a/t/helper/test-path-utils.c +++ b/t/helper/test-path-utils.c @@ -303,6 +303,27 @@ int cmd__path_utils(int argc, const char **argv) return !!res; } + if (argc == 4 && !strcmp(argv[1], "skip-n-bytes")) { + int fd = open(argv[2], O_RDONLY), offset = atoi(argv[3]); + char buffer[65536]; + + if (fd < 0) + die_errno("could not open '%s'", argv[2]); + if (lseek(fd, offset, SEEK_SET) < 0) + die_errno("could not skip %d bytes", offset); + for (;;) { + ssize_t count = read(fd, buffer, sizeof(buffer)); + if (count < 0) + die_errno("could not read '%s'", argv[2]); + if (!count) + break; + if (write(1, buffer, count) < 0) + die_errno("could not write to stdout"); + } + close(fd); + return 0; + } + fprintf(stderr, "%s: unknown function name: %s\n", argv[0], argv[1] ? argv[1] : "(there was none)"); return 1; diff --git a/t/test-lib.sh b/t/test-lib.sh index a3b2166cb5..f31a1c8f79 100644 --- a/t/test-lib.sh +++ b/t/test-lib.sh @@ -639,8 +639,19 @@ test_failure_ () { junit_insert="" junit_insert="$junit_insert $(xml_attr_encode \ - "$(printf '%s\n' "$@" | sed 1d)")" + "$(if test -n "$GIT_TEST_TEE_OUTPUT_FILE" + then + test-tool path-utils skip-n-bytes \ + "$GIT_TEST_TEE_OUTPUT_FILE" $GIT_TEST_TEE_OFFSET + else + printf '%s\n' "$@" | sed 1d + fi)")" junit_insert="$junit_insert" + if test -n "$GIT_TEST_TEE_OUTPUT_FILE" + then + junit_insert="$junit_insert$(xml_attr_encode \ + "$(cat "$GIT_TEST_TEE_OUTPUT_FILE")")" + fi write_junit_xml_testcase "$1" " $junit_insert" fi test_failure=$(($test_failure + 1)) @@ -931,6 +942,11 @@ test_finish_ () { echo >&3 "" maybe_teardown_valgrind maybe_teardown_verbose + if test -n "$GIT_TEST_TEE_OFFSET" + then + GIT_TEST_TEE_OFFSET=$(test-tool path-utils file-size \ + "$GIT_TEST_TEE_OUTPUT_FILE") + fi } test_skip () { @@ -1280,6 +1296,10 @@ then date +%Y-%m-%dT%H:%M:%S)\"" write_junit_xml --truncate "" " " junit_suite_start=$(test-tool date getnanos) + if test -n "$GIT_TEST_TEE_OUTPUT_FILE" + then + GIT_TEST_TEE_OFFSET=0 + fi fi # Provide an implementation of the 'yes' utility From patchwork Wed Jan 23 14:40:27 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Schindelin via GitGitGadget X-Patchwork-Id: 10777305 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 369E5746 for ; Wed, 23 Jan 2019 14:40:37 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 26D262C8F2 for ; Wed, 23 Jan 2019 14:40:37 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 1B7822C9D9; Wed, 23 Jan 2019 14:40:37 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B91D12C8F2 for ; Wed, 23 Jan 2019 14:40:36 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726088AbfAWOka (ORCPT ); Wed, 23 Jan 2019 09:40:30 -0500 Received: from mail-ed1-f68.google.com ([209.85.208.68]:40411 "EHLO mail-ed1-f68.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726183AbfAWOk3 (ORCPT ); Wed, 23 Jan 2019 09:40:29 -0500 Received: by mail-ed1-f68.google.com with SMTP id g22so1874541edr.7 for ; Wed, 23 Jan 2019 06:40:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=date:message-id:in-reply-to:references:from:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=tz7VD9jZtEnv8yooHJRr2Q/rjZIeYRglb+VYrbixkYA=; b=BiVf4F3KEKetkdGJhp8Wk3rzjjOBtEDkGyQjct2p7jyHjCnj30iceIGbhYKQrCMXn8 AR9nXI4Vvf/lS8yXtkACZa5dIlRKrpqqnu5P5xFfwbspBb0ZZxB/L1niwVeo+kK9t0F9 VNwBRBVY7eN5YDmS5EYat823lghmNPGum20du2ZExOYA5NoHPiFXKpnqMtmm2QSAmAgu nMZtK/xvlVS+gpQVJPEQGQLSqEWKcK1puNDpPo0EBZnwvVjPi1qDkdx8GX4VTQt5HF1z j/80L4mpNC/loCNUsyTuydEVgI6waizGjvY9XLSRyMW6Lw4jZmPSQbqa/DyvSAf5RKtS xzcA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:message-id:in-reply-to:references:from :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=tz7VD9jZtEnv8yooHJRr2Q/rjZIeYRglb+VYrbixkYA=; b=cUO6RRhHXP9lozxW6WvuvLDHfDG0o/j40QKy5rNefZzBWFx16SvH+yYAoEddbZwG0Q t878SqmrAHokdKpj9G8pk3KIbWDNdrqIpnkLSbnYSjd+Y8yxdihsuwz1RB7+BDNWBTLi m7DAqp1igWhIgSq2tECKMZCCr/6cZPS+d64BOmNQITpEsH2DRAYqOCa8VWOi1d9hheMz cU8I/88KdDKRjDKYV3CAA0Wf/JTPy/aN4gdrQEHOL8oIc0+iLNJVrNAxORzh97weJQV1 lwpYOcBz8aFmhWUGaN0IXB/rHf0doQemL4C0fYBf5gdIai49HKYD5Qv5GrLQrcFDpuRG Prcw== X-Gm-Message-State: AJcUukfTkb1bJ6oidTzsniZO6E2VjU/VwbLCzmG1Vkgzzu4KfMJDHygs AIwE+Okpdxc1CfmsGbikwdEeZvlB X-Google-Smtp-Source: ALg8bN7P9UIfJDZ0/h3tb1UFm5M0HXCwTLIUSbFQnwZ/dFVbjwBY8duP14bBX/ZCP/bQo8lRoKxlww== X-Received: by 2002:a50:ac81:: with SMTP id x1mr3017050edc.71.1548254427820; Wed, 23 Jan 2019 06:40:27 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id b14sm9915338edt.6.2019.01.23.06.40.27 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 23 Jan 2019 06:40:27 -0800 (PST) Date: Wed, 23 Jan 2019 06:40:27 -0800 (PST) X-Google-Original-Date: Wed, 23 Jan 2019 14:40:07 GMT Message-Id: <59c1194ae2da0c4986f595ab9600c9dc518ef1b1.1548254412.git.gitgitgadget@gmail.com> In-Reply-To: References: From: "Johannes Schindelin via GitGitGadget" Subject: [PATCH v4 16/21] mingw: try to work around issues with the test cleanup Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Junio C Hamano , Johannes Schindelin Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Johannes Schindelin It seems that every once in a while in the Git for Windows SDK, there are some transient file locking issues preventing the test clean up to delete the trash directory. Let's be gentle and try again five seconds later, and only error out if it still fails the second time. This change helps Windows, and does not hurt any other platform (normally, it is highly unlikely that said deletion fails, and if it does, normally it will fail again even 5 seconds later). Signed-off-by: Johannes Schindelin --- t/test-lib.sh | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/t/test-lib.sh b/t/test-lib.sh index f31a1c8f79..9c0ca5effb 100644 --- a/t/test-lib.sh +++ b/t/test-lib.sh @@ -1104,7 +1104,11 @@ test_done () { error "Tests passed but trash directory already removed before test cleanup; aborting" cd "$TRASH_DIRECTORY/.." && - rm -fr "$TRASH_DIRECTORY" || + rm -fr "$TRASH_DIRECTORY" || { + # try again in a bit + sleep 5; + rm -fr "$TRASH_DIRECTORY" + } || error "Tests passed but test cleanup failed; aborting" fi test_at_end_hook_ From patchwork Wed Jan 23 14:40:28 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Schindelin via GitGitGadget X-Patchwork-Id: 10777307 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id BF21D1390 for ; Wed, 23 Jan 2019 14:40:39 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id AF6D82C8F2 for ; Wed, 23 Jan 2019 14:40:39 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id A45B92C9D9; Wed, 23 Jan 2019 14:40:39 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 48CB42C8F2 for ; Wed, 23 Jan 2019 14:40:39 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726778AbfAWOki (ORCPT ); Wed, 23 Jan 2019 09:40:38 -0500 Received: from mail-ed1-f67.google.com ([209.85.208.67]:35986 "EHLO mail-ed1-f67.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725976AbfAWOka (ORCPT ); Wed, 23 Jan 2019 09:40:30 -0500 Received: by mail-ed1-f67.google.com with SMTP id f23so1889432edb.3 for ; Wed, 23 Jan 2019 06:40:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=date:message-id:in-reply-to:references:from:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=6wZF5XyFnvK4tOVLEGObR7qCxdrcyoiAaAwrz8JEDdo=; b=pDdaIsYqmzvZ1SroOUM3GHgpkhHQ7OUscNEETIJveNoV09827tPOmEC4+BY+u3nbCu U60UjW9Zl/SnkMi9Q+hgWwJCv9uD8TClAE40XQIPRtyJtN76i+y9nVG846GgqmhdBCHq 8ZOTTliAA9S7344mTn+eb8zzEZDrZy8UhZP1YfAwD62OFD59miro9FBlVB9CKWNk1Kyy QbNn/SGqmp7D2DumFg7WCRp/J1Cdi7sDFNwNS8h5JnxlvKJwqlf3X5tSAaHGEOY5QCZH 6fehXErBhq5EBc0hlN5A+pWjkHDLFqCVAfDUlQChuj1V94pbP8A//ZffGOChstAsUDRC xXlw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:message-id:in-reply-to:references:from :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=6wZF5XyFnvK4tOVLEGObR7qCxdrcyoiAaAwrz8JEDdo=; b=QpPa9iysB4aJMZMu14vWq+3wZDDDMM8LPdm6PJcjCCg4cFybfQmSrRG/7AV/7xa9CM expPqHXNc0oJL1fIwFtL3iDiIrMvVrHmKBaa3pMSidKQka4WaKYrZ/BpzCpkZG8wh6Gg Wu5d6c/C7AOBPNiW9t18PCwdIIxwpd2n2wGCINlkcvPfbncDjazBrMEYDpLEXl4Y4Nrw cFZkwr/2eBJP9CHmm/yO81ND5SyoUNJpg89XNyb9OWfDXPW14m+JRMGU+PJVAAzuIrRr eciiIaISpjGr1KdZef0aB3JQLYohsrZu4Ve0VvE+GwN+YTE0mXlrRbo7I7AJXseNeLy3 RKaw== X-Gm-Message-State: AJcUukemN6zMFwfi0HZ+BedA2Su9h6ZbP9R9mslEub/nA51d3neQrfj4 /AOY5Vn702VPAa5Ua2O3AzoRnz/H X-Google-Smtp-Source: ALg8bN7km3sqLZ3PkhqLFo6G5kVBzOAh8YGFbRmvnNI/b3WbqH3S6ZYNAPOIs2bLnFesiIQe8tYYMA== X-Received: by 2002:a17:906:828b:: with SMTP id h11-v6mr2718876ejx.115.1548254428752; Wed, 23 Jan 2019 06:40:28 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id v20sm10446591edm.29.2019.01.23.06.40.28 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 23 Jan 2019 06:40:28 -0800 (PST) Date: Wed, 23 Jan 2019 06:40:28 -0800 (PST) X-Google-Original-Date: Wed, 23 Jan 2019 14:40:08 GMT Message-Id: <966c412f03013902826ffd188ee9af973be60833.1548254412.git.gitgitgadget@gmail.com> In-Reply-To: References: From: "Johannes Schindelin via GitGitGadget" Subject: [PATCH v4 17/21] tests: add t/helper/ to the PATH with --with-dashes Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Junio C Hamano , Johannes Schindelin Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Johannes Schindelin We really need to be able to find the test helpers... Really. This change was forgotten when we moved the test helpers into t/helper/ Signed-off-by: Johannes Schindelin --- t/test-lib.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/t/test-lib.sh b/t/test-lib.sh index 9c0ca5effb..c790e98fd2 100644 --- a/t/test-lib.sh +++ b/t/test-lib.sh @@ -1227,7 +1227,7 @@ else # normal case, use ../bin-wrappers only unless $with_dashes: GIT_EXEC_PATH=$GIT_BUILD_DIR if test -n "$with_dashes" then - PATH="$GIT_BUILD_DIR:$PATH" + PATH="$GIT_BUILD_DIR:$GIT_BUILD_DIR/t/helper:$PATH" fi fi GIT_TEMPLATE_DIR="$GIT_BUILD_DIR"/templates/blt From patchwork Wed Jan 23 14:40:29 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Schindelin via GitGitGadget X-Patchwork-Id: 10777303 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id CF29C1390 for ; Wed, 23 Jan 2019 14:40:35 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id BE32D2C8F2 for ; Wed, 23 Jan 2019 14:40:35 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id B2BE02C9D9; Wed, 23 Jan 2019 14:40:35 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 57F502C8F2 for ; Wed, 23 Jan 2019 14:40:35 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726870AbfAWOke (ORCPT ); Wed, 23 Jan 2019 09:40:34 -0500 Received: from mail-ed1-f65.google.com ([209.85.208.65]:46582 "EHLO mail-ed1-f65.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726030AbfAWOkb (ORCPT ); Wed, 23 Jan 2019 09:40:31 -0500 Received: by mail-ed1-f65.google.com with SMTP id o10so1845788edt.13 for ; Wed, 23 Jan 2019 06:40:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=date:message-id:in-reply-to:references:from:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=dxzqfZAxZ+TPwgPiHydUMW7hZp+IB66oCyS61ZHth2s=; b=FbCZTrsyI/9en0xEAZ0TorS3FlyO2ts4f1YhvpaaaMqUqjXLoYKCWLX8kXc3CPo9nD CX1yp0cqMUHeGpyC4PdsjjIFNCxuOE/zp9i9Xja1GPMxytqyNqx9UBL6UwtoAXT7UD8w 2rzan2kOJNg1IDJHYuKi0fSNVJKgQTR8bNUSUIw7oLqeCpA/10jZr2OfxUZYGSH32vD8 Jg22GYSsoHF14f7s7CbmhE8hgEzavtGsqRzJwyP7D6dWipCaXnKpEt/XldebjHLv772u aiah2QpqXctesl0poveO/hsYpQNj4OUVe9iJwIWBJm1JDJCi6D3DvEwd6n39pabo3N7J jYlQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:message-id:in-reply-to:references:from :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=dxzqfZAxZ+TPwgPiHydUMW7hZp+IB66oCyS61ZHth2s=; b=kGSTjOIVkVe2ASXhP39kabwNMlGUPEzi57dxtSipwx5vwN2OxE6ERy+M5pTFaEMr7y KYEgVrVkCiFJGS6IxxR0iBZAme+MIL9ulIvtqV3WTtlQ1OeZo4DW9432i7euJ6V193nr yg0oETayEV4JT1fKy2PXxoLuQNpsDkFflvpmBHHNv+haRsBv339rWvY3g//AsxloI+1/ yO0k3SLwRIXX+fdYVWVF3ZFQsLA9rHck2s9BFLha0wrDuc4XqDbKiVBEllz2NYccSf8f F9rZxHML7m4iL2sLEdDnu3cw6wTFaUK1/ybjyQbDfiRhHyUR0dSVPsDY4OCChR1KbXW8 +Dvw== X-Gm-Message-State: AJcUukeXbq0PVm5JnlZaf0fo4xXApGGaxwcF7JC+A/725Tum2z6yVpou opxEGFEGCuKBw3IvRgWt66vUvgTm X-Google-Smtp-Source: ALg8bN64YfxEQNxfhLz0uWUV0qHFfk2etMnD5Q49Ml0yT9yajYM4HdFM/SzV+sCyTnt2H9B+VEIz7w== X-Received: by 2002:a50:ca86:: with SMTP id x6mr2876146edh.287.1548254429499; Wed, 23 Jan 2019 06:40:29 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id s36sm10754165edb.43.2019.01.23.06.40.28 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 23 Jan 2019 06:40:29 -0800 (PST) Date: Wed, 23 Jan 2019 06:40:29 -0800 (PST) X-Google-Original-Date: Wed, 23 Jan 2019 14:40:09 GMT Message-Id: In-Reply-To: References: From: "Johannes Schindelin via GitGitGadget" Subject: [PATCH v4 18/21] t0061: fix with --with-dashes and RUNTIME_PREFIX Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Junio C Hamano , Johannes Schindelin Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Johannes Schindelin When building Git with RUNTIME_PREFIX and starting a test helper from t/helper/, it fails to detect the system prefix correctly. This is the reason that the warning RUNTIME_PREFIX requested, but prefix computation failed. [...] to be printed. In t0061, we did not expect that to happen, and it actually did not happen in the normal case, because bin-wrappers/test-tool specifically sets GIT_TEXTDOMAINDIR (and as a consequence, nothing in test-tool wants to know about the runtime prefix). However, with --with-dashes, bin-wrappers/test-tool is no longer called, but t/helper/test-tool is called directly. So let's just ignore the RUNTIME_PREFIX warning. Signed-off-by: Johannes Schindelin --- t/t0061-run-command.sh | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/t/t0061-run-command.sh b/t/t0061-run-command.sh index 99a614bc7c..5a2d087bf0 100755 --- a/t/t0061-run-command.sh +++ b/t/t0061-run-command.sh @@ -166,7 +166,8 @@ test_trace () { expect="$1" shift GIT_TRACE=1 test-tool run-command "$@" run-command true 2>&1 >/dev/null | \ - sed -e 's/.* run_command: //' -e '/trace: .*/d' >actual && + sed -e 's/.* run_command: //' -e '/trace: .*/d' \ + -e '/RUNTIME_PREFIX requested/d' >actual && echo "$expect true" >expect && test_cmp expect actual } From patchwork Wed Jan 23 14:40:29 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Schindelin via GitGitGadget X-Patchwork-Id: 10777301 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id CA7DA1390 for ; Wed, 23 Jan 2019 14:40:34 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B95162C8F2 for ; Wed, 23 Jan 2019 14:40:34 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id AE1CA2C9D9; Wed, 23 Jan 2019 14:40:34 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 51A2E2C8F2 for ; Wed, 23 Jan 2019 14:40:34 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726790AbfAWOkd (ORCPT ); Wed, 23 Jan 2019 09:40:33 -0500 Received: from mail-ed1-f67.google.com ([209.85.208.67]:41374 "EHLO mail-ed1-f67.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726183AbfAWOkc (ORCPT ); Wed, 23 Jan 2019 09:40:32 -0500 Received: by mail-ed1-f67.google.com with SMTP id a20so1862728edc.8 for ; Wed, 23 Jan 2019 06:40:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=date:message-id:in-reply-to:references:from:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=7gtFKAG898gIOKd93T1TNRDKFMtd3jqrEAjEXx25Pgw=; b=B+cetceaZfHJjzANQVVEavDr7BTNiXh5dsprzjT4lP+wcdLRFssjya1jFx30440jbH KSAk7zeifamtrVOvfDl7FtRWIC0+e5FUzUR+NdLmlKolQvBbbYY4kL7FWdNMbW2kGzw0 bFlPQMuUoRh//cH0o/a3sDMt9MjgFVmlXYUJHZXv6kh8GH3ACCksuv+w9/KsrVMTcrqc sn7c6TWJg0oTcnI73NPpkycWiMT8p6lgFWLMbvnckyRJ6HL+GiULL4pn/2dWltjQE2Uq Uw6GldpOhytLhr0u3EVQKAIQscc6inPoUMX5XPWhnfs5slgN5Cj1ac7Mvuvm0ivKv5AD Htjg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:message-id:in-reply-to:references:from :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=7gtFKAG898gIOKd93T1TNRDKFMtd3jqrEAjEXx25Pgw=; b=EuXLadJhTYVfeDKTFzrE8JRMtMVsARx1XX+qSOulHMMwCkT5u5J6SHcBUYlXWJmAzm DmbaDfpXdEUvjNztrKWNqPACG/JNUOD+2NjVog0ANH5IiTc7gUgel+he5hykXr3nEOTK hEAj8nmKq2Y7zpyVZK8enXOSvHxHdCdHLIQyPs6w4H2glZXq0dOYP+i1Sri9gJTnLTQy pOMFB20zqA2ZxfXGFVY+4CSvKuM4T/GuD4yUvRFVz4jKO8lmxpXr05PeoVDFs+ia8qGS zIFEz6+9a5EN0dXH4+fp7gPQ1stZKEaJU6WHq8uUG7gh24H+zMlIfYFQzY/CH3ArhIwB +8Bw== X-Gm-Message-State: AJcUukcqQFj4DkFTkhxjv/F/uab6iKrPuzs5rtrwKOtp+EH3q9UnF1bl 3Ynr9f/6FPPHRwpeHYDcmtj22Z+B X-Google-Smtp-Source: ALg8bN47ov1CzuyXPuFkL5rFlV2OsjxainBqxhz8ooEZEg9TLB6dxgi8iEFTJteOHsFpnCoNg5hTKg== X-Received: by 2002:a50:e0cb:: with SMTP id j11mr2962416edl.195.1548254430333; Wed, 23 Jan 2019 06:40:30 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id k11sm9867050edq.51.2019.01.23.06.40.29 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 23 Jan 2019 06:40:29 -0800 (PST) Date: Wed, 23 Jan 2019 06:40:29 -0800 (PST) X-Google-Original-Date: Wed, 23 Jan 2019 14:40:10 GMT Message-Id: In-Reply-To: References: From: "Johannes Schindelin via GitGitGadget" Subject: [PATCH v4 19/21] tests: optionally skip bin-wrappers/ Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Junio C Hamano , Johannes Schindelin Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Johannes Schindelin This speeds up the tests by a bit on Windows, where running Unix shell scripts (and spawning processes) is not exactly a cheap operation. Signed-off-by: Johannes Schindelin --- t/README | 9 +++++++++ t/test-lib.sh | 19 +++++++++++++------ 2 files changed, 22 insertions(+), 6 deletions(-) diff --git a/t/README b/t/README index 11ce7675e3..063530234f 100644 --- a/t/README +++ b/t/README @@ -170,6 +170,15 @@ appropriately before running "make". implied by other options like --valgrind and GIT_TEST_INSTALLED. +--no-bin-wrappers:: + By default, the test suite uses the wrappers in + `../bin-wrappers/` to execute `git` and friends. With this option, + `../git` and friends are run directly. This is not recommended + in general, as the wrappers contain safeguards to ensure that no + files from an installed Git are used, but can speed up test runs + especially on platforms where running shell scripts is expensive + (most notably, Windows). + --root=:: Create "trash" directories used to store all temporary data during testing under , instead of the t/ directory. diff --git a/t/test-lib.sh b/t/test-lib.sh index c790e98fd2..25e649c997 100644 --- a/t/test-lib.sh +++ b/t/test-lib.sh @@ -111,6 +111,8 @@ do test -z "$HARNESS_ACTIVE" && quiet=t ;; --with-dashes) with_dashes=t ;; + --no-bin-wrappers) + no_bin_wrappers=t ;; --no-color) color= ;; --va|--val|--valg|--valgr|--valgri|--valgrin|--valgrind) @@ -1214,16 +1216,21 @@ then PATH=$GIT_TEST_INSTALLED:$GIT_BUILD_DIR/t/helper:$PATH GIT_EXEC_PATH=${GIT_TEST_EXEC_PATH:-$GIT_EXEC_PATH} else # normal case, use ../bin-wrappers only unless $with_dashes: - git_bin_dir="$GIT_BUILD_DIR/bin-wrappers" - if ! test -x "$git_bin_dir/git" + if test -n "$no_bin_wrappers" then - if test -z "$with_dashes" + with_dashes=t + else + git_bin_dir="$GIT_BUILD_DIR/bin-wrappers" + if ! test -x "$git_bin_dir/git" then - say "$git_bin_dir/git is not executable; using GIT_EXEC_PATH" + if test -z "$with_dashes" + then + say "$git_bin_dir/git is not executable; using GIT_EXEC_PATH" + fi + with_dashes=t fi - with_dashes=t + PATH="$git_bin_dir:$PATH" fi - PATH="$git_bin_dir:$PATH" GIT_EXEC_PATH=$GIT_BUILD_DIR if test -n "$with_dashes" then From patchwork Wed Jan 23 14:40:30 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Schindelin via GitGitGadget X-Patchwork-Id: 10777309 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id A62B51390 for ; Wed, 23 Jan 2019 14:40:40 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 957412C8F2 for ; Wed, 23 Jan 2019 14:40:40 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 8A0592C9D9; Wed, 23 Jan 2019 14:40:40 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 2D8522C8F2 for ; Wed, 23 Jan 2019 14:40:40 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726763AbfAWOkh (ORCPT ); Wed, 23 Jan 2019 09:40:37 -0500 Received: from mail-ed1-f44.google.com ([209.85.208.44]:41423 "EHLO mail-ed1-f44.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725871AbfAWOkd (ORCPT ); Wed, 23 Jan 2019 09:40:33 -0500 Received: by mail-ed1-f44.google.com with SMTP id a20so1862780edc.8 for ; Wed, 23 Jan 2019 06:40:32 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=date:message-id:in-reply-to:references:from:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=D/xMBG3VYCTIdwTucGhD4/lsdRH5mFPt1pjIs1nYDZQ=; b=AT9t7IRph917FUm/AHuNx34TO5/hx6nV9MmZXGsnKhfORxpaHv9qypAeVBtEExK0Ww h0H0tXPekBSbLpDx4/60OJjSvaOSgMSKuf2g9/Pn+E8HHeg5Rt3x9IxdkKBZllBeC7cy NhW+Oj/V5rcpcnV5j6wrHZ326S2kxC6s5jSxaUgZXOYUBOOPf5kq1+Pj0lemUJsAC5CC 5Otsunv6Gn7h/cyUS1R0dUY1BwrkAmE+BKTfgp8D+/OEg9l+G7p09dwKaT1fXmFAa5S6 In69qPUVE1SUQrM/16SVZBQE6trRjWm83G74HGtvA7vV4DraiwrLwpWPv7OoCUwrl40Q pn+g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:message-id:in-reply-to:references:from :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=D/xMBG3VYCTIdwTucGhD4/lsdRH5mFPt1pjIs1nYDZQ=; b=nH28Vv2fTytl9hkM5jjGsFM1PkWepNVdn9Oiqi9f5D1UwozbrDRkBdmRMWwjy0uIj+ 5hzOYC/C3zM3d1r0q13naXYbazwmF7j8pDtnDSfzuMFhYLcttLtZdXg9fuS3UyzT1D5J nFiNJjc3WJS+u4IyInJtamAvOv1w7LXm8qsMK9ksIi2YsXt5RYNWdXRiUHoAjiBPGMG8 fZp8+HdSO9sJ1956VQOdFte13Cd/saSwzcigL166IkTyRiDDGB9cRdi57h7j7HxY4kQR XMCTbdvTK3Cwn2b125jTX25npM9/NMvEfXaE9ivoD+b+X8MX0m14p4tLNj5pvZIdOYUh 1Bdg== X-Gm-Message-State: AJcUukf4RzZk9eEfk4hWIWJfGt8Vs9TqgxeIYaxLPQUZZ2sTD7gqJsdt C2xuLjbAhnnSkHgCXbq5T3QGPRgF X-Google-Smtp-Source: ALg8bN4udsCc2Wt9xGQgPGmDs4FLAa5HzcODwSbM0oUwITSavqlvO9ALoJM8TNiA9Sb+pwLewK3Wtg== X-Received: by 2002:a17:906:7057:: with SMTP id r23-v6mr2656988ejj.113.1548254431281; Wed, 23 Jan 2019 06:40:31 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id q16sm9875619eds.60.2019.01.23.06.40.30 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 23 Jan 2019 06:40:30 -0800 (PST) Date: Wed, 23 Jan 2019 06:40:30 -0800 (PST) X-Google-Original-Date: Wed, 23 Jan 2019 14:40:11 GMT Message-Id: <6520f4603c78d38f5168164218dc108833a5583d.1548254412.git.gitgitgadget@gmail.com> In-Reply-To: References: From: "Johannes Schindelin via GitGitGadget" Subject: [PATCH v4 20/21] ci: speed up Windows phase Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Junio C Hamano , Johannes Schindelin Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Johannes Schindelin As Unix shell scripting comes at a hefty price on Windows, we have to see where we can save some time to run the test suite. Let's skip the chain linting and the bin-wrappers/ redirection on Windows; this seems to shave of anywhere between 10-30% from the overall runtime. Signed-off-by: Johannes Schindelin --- ci/lib.sh | 2 ++ 1 file changed, 2 insertions(+) diff --git a/ci/lib.sh b/ci/lib.sh index 91cf1402bf..e1d5222ceb 100755 --- a/ci/lib.sh +++ b/ci/lib.sh @@ -43,6 +43,8 @@ then BREW_INSTALL_PACKAGES= export GIT_PROVE_OPTS="--timer --jobs 10 --state=failed,slow,save" export GIT_TEST_OPTS="--verbose-log -x --write-junit-xml" + test windows_nt != "$CI_OS_NAME" || + GIT_TEST_OPTS="--no-chain-lint --no-bin-wrappers $GIT_TEST_OPTS" fi skip_branch_tip_with_tag () { From patchwork Wed Jan 23 14:40:31 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Schindelin via GitGitGadget X-Patchwork-Id: 10777311 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 18706746 for ; Wed, 23 Jan 2019 14:40:42 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 061C72C8F2 for ; Wed, 23 Jan 2019 14:40:42 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id EE6F52C9DC; Wed, 23 Jan 2019 14:40:41 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 2EA7D2C8F2 for ; Wed, 23 Jan 2019 14:40:41 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726256AbfAWOkg (ORCPT ); Wed, 23 Jan 2019 09:40:36 -0500 Received: from mail-ed1-f66.google.com ([209.85.208.66]:37298 "EHLO mail-ed1-f66.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726763AbfAWOke (ORCPT ); Wed, 23 Jan 2019 09:40:34 -0500 Received: by mail-ed1-f66.google.com with SMTP id h15so1880817edb.4 for ; Wed, 23 Jan 2019 06:40:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=date:message-id:in-reply-to:references:from:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=fSYfsCglrTsPvdE/rLHEBuZDG7QQbq1p8mnMzRUaygE=; b=i3hL+JQWGnbwP32XuI/sTydpZpm9xy0/vQgtxZP4zQyB1ULbOzLHXEmCtCcv4y6uFF W1kmjKc21bjRJzix+RuMqqGahjyTXU9WNS5ucP6tP6RrMHUkcamFg7oWwdatWiYMiqHW Tym2z5sc1WXCz9YrtxWN13gH337a4AxULCLFdMLRcyG+6xEBor/7TDm/Ti9oG62i85OE Xf8zx63lBmIfPEc83hMucam1lCva7RBWXF7JGc7c5y2diJYKqSYBi6Ms767VqqW69vim NNhhrCJTZF+iZ27vHwUNU1pWI5AqFZhzevYa3dM2fA212wmbMzdee32r6ZGP4kMddOsS Yl9A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:message-id:in-reply-to:references:from :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=fSYfsCglrTsPvdE/rLHEBuZDG7QQbq1p8mnMzRUaygE=; b=WQIeez5e15+Gjt48/eXsxPlfQjBnJoo+OVbo0S9IR1+OsonB81uYUjmXAzf3AYJCAa eCb4RIHJdJevldTCqvqnQNpZZBHPxgbFduS8kWI9C3WyaAF/DARwwwu7beA05gYSAFIk cOj44pl/GXnH/dCbyyZK0kndMtXER3Vf2pSivqgcBiGUxoOAsPQlAOJ2H3LVNF9MgrBe VARyGItGG6HZ3+kmgWN19dgwmpQRpaijeIEyqet/mc28fqICjxcV8jUCbnl2COzS+eKr BHI2GYNY9ZYJpMrtIeWMHYnoGMmmgszo0dCKA8wFlOLRc7t5/OLkJ5XKyg4IsgCZybB3 lU7Q== X-Gm-Message-State: AJcUukerPcLES2DPV4HiDQaPTTIngPLqi+vlHJBYE5U+S8P+epCmjXH1 weUeQXGKFUQpQsEPIpYLij0dh2qt X-Google-Smtp-Source: ALg8bN5/D1kYG1hbDpPKIrtxU4B8Dv8kCKaGqWKcc7e/T32J0er8HTveskvp/jnGS5uL527ZmoKGJg== X-Received: by 2002:a50:ac81:: with SMTP id x1mr3017302edc.71.1548254432157; Wed, 23 Jan 2019 06:40:32 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id b45sm10405974eda.34.2019.01.23.06.40.31 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 23 Jan 2019 06:40:31 -0800 (PST) Date: Wed, 23 Jan 2019 06:40:31 -0800 (PST) X-Google-Original-Date: Wed, 23 Jan 2019 14:40:12 GMT Message-Id: <8bdd9804a1a7751c9646c585022ab4d043eb2168.1548254412.git.gitgitgadget@gmail.com> In-Reply-To: References: From: "Johannes Schindelin via GitGitGadget" Subject: [PATCH v4 21/21] ci: parallelize testing on Windows Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Junio C Hamano , Johannes Schindelin Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Johannes Schindelin The fact that Git's test suite is implemented in Unix shell script that is as portable as we can muster, combined with the fact that Unix shell scripting is foreign to Windows (and therefore has to be emulated), results in pretty abysmal speed of the test suite on that platform, for pretty much no other reason than that language choice. For comparison: while the Linux build & test is typically done within about 8 minutes, the Windows build & test typically lasts about 80 minutes in Azure Pipelines. To help with that, let's use the Azure Pipeline feature where you can parallelize jobs, make jobs depend on each other, and pass artifacts between them. The tests are distributed using the following heuristic: listing all test scripts ordered by size in descending order (as a cheap way to estimate the overall run time), every Nth script is run (where N is the total number of parallel jobs), starting at the index corresponding to the parallel job. This slicing is performed by a new function that is added to the `test-tool`. To optimize the overall runtime of the entire Pipeline, we need to move the Windows jobs to the beginning (otherwise there would be a very decent chance for the Pipeline to be run only the Windows build, while all the parallel Windows test jobs wait for this single one). We use Azure Pipelines Artifacts for both the minimal Git for Windows SDK as well as the built executables, as deduplication and caching close to the agents makes that really fast. For comparison: while downloading and unpacking the minimal Git for Windows SDK via PowerShell takes only one minute (down from anywhere between 2.5 to 7 when using a shallow clone), uploading it as Pipeline Artifact takes less than 30s and downloading and unpacking less than 20s (sometimes even as little as only twelve seconds). Signed-off-by: Johannes Schindelin --- Makefile | 10 +++++ azure-pipelines.yml | 82 +++++++++++++++++++++++++++++++++----- ci/make-test-artifacts.sh | 12 ++++++ ci/run-test-slice.sh | 17 ++++++++ t/helper/test-path-utils.c | 31 ++++++++++++++ 5 files changed, 143 insertions(+), 9 deletions(-) create mode 100755 ci/make-test-artifacts.sh create mode 100755 ci/run-test-slice.sh diff --git a/Makefile b/Makefile index 044b4f77bd..daa318fe17 100644 --- a/Makefile +++ b/Makefile @@ -2927,6 +2927,16 @@ rpm:: @false .PHONY: rpm +artifacts-tar:: $(ALL_PROGRAMS) $(SCRIPT_LIB) $(BUILT_INS) $(OTHER_PROGRAMS) \ + GIT-BUILD-OPTIONS $(TEST_PROGRAMS) $(test_bindir_programs) \ + $(NO_INSTALL) $(MOFILES) + $(QUIET_SUBDIR0)templates $(QUIET_SUBDIR1) \ + SHELL_PATH='$(SHELL_PATH_SQ)' PERL_PATH='$(PERL_PATH_SQ)' + test -n "$(ARTIFACTS_DIRECTORY)" + mkdir -p "$(ARTIFACTS_DIRECTORY)" + $(TAR) czf "$(ARTIFACTS_DIRECTORY)/artifacts.tar.gz" $^ templates/blt/ +.PHONY: artifacts-tar + htmldocs = git-htmldocs-$(GIT_VERSION) manpages = git-manpages-$(GIT_VERSION) .PHONY: dist-doc distclean diff --git a/azure-pipelines.yml b/azure-pipelines.yml index 3085ff6ace..6cd27b3483 100644 --- a/azure-pipelines.yml +++ b/azure-pipelines.yml @@ -3,8 +3,8 @@ resources: fetchDepth: 1 jobs: -- job: windows - displayName: Windows +- job: windows_build + displayName: Windows Build condition: succeeded() pool: Hosted timeoutInMinutes: 240 @@ -30,22 +30,86 @@ jobs: displayName: 'Download git-sdk-64-minimal' - powershell: | & git-sdk-64-minimal\usr\bin\bash.exe -lc @" - export MAKEFLAGS=-j10 - export DEVELOPER=1 - export NO_PERL=1 - export NO_SVN_TESTS=1 - export GIT_TEST_SKIP_REBASE_P=1 + ci/make-test-artifacts.sh artifacts + "@ + if (!$?) { exit(1) } + displayName: Build + env: + HOME: $(Build.SourcesDirectory) + MSYSTEM: MINGW64 + MAKEFLAGS: -j10 + DEVELOPER: 1 + NO_PERL: 1 + - task: PublishPipelineArtifact@0 + displayName: 'Publish Pipeline Artifact: test artifacts' + inputs: + artifactName: 'windows-artifacts' + targetPath: '$(Build.SourcesDirectory)\artifacts' + - task: PublishPipelineArtifact@0 + displayName: 'Publish Pipeline Artifact: git-sdk-64-minimal' + inputs: + artifactName: 'git-sdk-64-minimal' + targetPath: '$(Build.SourcesDirectory)\git-sdk-64-minimal' + - powershell: | + if ("$GITFILESHAREPWD" -ne "" -and "$GITFILESHAREPWD" -ne "`$`(gitfileshare.pwd)") { + cmd /c rmdir "$(Build.SourcesDirectory)\test-cache" + } + displayName: 'Unmount test-cache' + condition: true + env: + GITFILESHAREPWD: $(gitfileshare.pwd) + +- job: windows_test + displayName: Windows Test + dependsOn: windows_build + condition: succeeded() + pool: Hosted + timeoutInMinutes: 240 + strategy: + parallel: 10 + steps: + - powershell: | + if ("$GITFILESHAREPWD" -ne "" -and "$GITFILESHAREPWD" -ne "`$`(gitfileshare.pwd)") { + net use s: \\gitfileshare.file.core.windows.net\test-cache "$GITFILESHAREPWD" /user:AZURE\gitfileshare /persistent:no + cmd /c mklink /d "$(Build.SourcesDirectory)\test-cache" S:\ + } + displayName: 'Mount test-cache' + env: + GITFILESHAREPWD: $(gitfileshare.pwd) + - task: DownloadPipelineArtifact@0 + displayName: 'Download Pipeline Artifact: test artifacts' + inputs: + artifactName: 'windows-artifacts' + targetPath: '$(Build.SourcesDirectory)' + - task: DownloadPipelineArtifact@0 + displayName: 'Download Pipeline Artifact: git-sdk-64-minimal' + inputs: + artifactName: 'git-sdk-64-minimal' + targetPath: '$(Build.SourcesDirectory)\git-sdk-64-minimal' + - powershell: | + & git-sdk-64-minimal\usr\bin\bash.exe -lc @" + test -f artifacts.tar.gz || { + echo No test artifacts found\; skipping >&2 + exit 0 + } + tar xf artifacts.tar.gz || exit 1 + + # Let Git ignore the SDK and the test-cache + printf '%s\n' /git-sdk-64-minimal/ /test-cache/ >>.git/info/exclude - ci/run-build-and-tests.sh || { + ci/run-test-slice.sh `$SYSTEM_JOBPOSITIONINPHASE `$SYSTEM_TOTALJOBSINPHASE || { ci/print-test-failures.sh exit 1 } "@ if (!$?) { exit(1) } - displayName: 'Build & Test' + displayName: 'Test (parallel)' env: HOME: $(Build.SourcesDirectory) MSYSTEM: MINGW64 + MAKEFLAGS: -j10 + NO_SVN_TESTS: 1 + GIT_TEST_SKIP_REBASE_P: 1 - powershell: | if ("$GITFILESHAREPWD" -ne "" -and "$GITFILESHAREPWD" -ne "`$`(gitfileshare.pwd)") { cmd /c rmdir "$(Build.SourcesDirectory)\test-cache" diff --git a/ci/make-test-artifacts.sh b/ci/make-test-artifacts.sh new file mode 100755 index 0000000000..646967481f --- /dev/null +++ b/ci/make-test-artifacts.sh @@ -0,0 +1,12 @@ +#!/bin/sh +# +# Build Git and store artifacts for testing +# + +mkdir -p "$1" # in case ci/lib.sh decides to quit early + +. ${0%/*}/lib.sh + +make artifacts-tar ARTIFACTS_DIRECTORY="$1" + +check_unignored_build_artifacts diff --git a/ci/run-test-slice.sh b/ci/run-test-slice.sh new file mode 100755 index 0000000000..f8c2c3106a --- /dev/null +++ b/ci/run-test-slice.sh @@ -0,0 +1,17 @@ +#!/bin/sh +# +# Test Git in parallel +# + +. ${0%/*}/lib.sh + +case "$CI_OS_NAME" in +windows*) cmd //c mklink //j t\\.prove "$(cygpath -aw "$cache_dir/.prove")";; +*) ln -s "$cache_dir/.prove" t/.prove;; +esac + +make --quiet -C t T="$(cd t && + ./helper/test-tool path-utils slice-tests "$1" "$2" t[0-9]*.sh | + tr '\n' ' ')" + +check_unignored_build_artifacts diff --git a/t/helper/test-path-utils.c b/t/helper/test-path-utils.c index 6efde6f5ba..5d543ad21f 100644 --- a/t/helper/test-path-utils.c +++ b/t/helper/test-path-utils.c @@ -177,6 +177,14 @@ static int is_dotgitmodules(const char *path) return is_hfs_dotgitmodules(path) || is_ntfs_dotgitmodules(path); } +static int cmp_by_st_size(const void *a, const void *b) +{ + intptr_t x = (intptr_t)((struct string_list_item *)a)->util; + intptr_t y = (intptr_t)((struct string_list_item *)b)->util; + + return x > y ? -1 : (x < y ? +1 : 0); +} + int cmd__path_utils(int argc, const char **argv) { if (argc == 3 && !strcmp(argv[1], "normalize_path_copy")) { @@ -324,6 +332,29 @@ int cmd__path_utils(int argc, const char **argv) return 0; } + if (argc > 5 && !strcmp(argv[1], "slice-tests")) { + int res = 0; + long offset, stride, i; + struct string_list list = STRING_LIST_INIT_NODUP; + struct stat st; + + offset = strtol(argv[2], NULL, 10); + stride = strtol(argv[3], NULL, 10); + if (stride < 1) + stride = 1; + for (i = 4; i < argc; i++) + if (stat(argv[i], &st)) + res = error_errno("Cannot stat '%s'", argv[i]); + else + string_list_append(&list, argv[i])->util = + (void *)(intptr_t)st.st_size; + QSORT(list.items, list.nr, cmp_by_st_size); + for (i = offset; i < list.nr; i+= stride) + printf("%s\n", list.items[i].string); + + return !!res; + } + fprintf(stderr, "%s: unknown function name: %s\n", argv[0], argv[1] ? argv[1] : "(there was none)"); return 1;