[RFC] travis-ci: remove bogus 'pyenv' in the Linux jobs
diff mbox series

Message ID 20200721161225.6769-1-szeder.dev@gmail.com
State New
Headers show
Series
  • [RFC] travis-ci: remove bogus 'pyenv' in the Linux jobs
Related show

Commit Message

SZEDER Gábor July 21, 2020, 4:12 p.m. UTC
In our test suite, when 'git p4' invokes a Git command as a
subprocesses, then it should run the 'git' binary we are testing.
Unfortunately, this is not the case in the 'linux-clang' and
'linux-gcc' jobs on Travis CI, where 'git p4' runs the system
'/usr/bin/git' instead.

Travis CI's default Linux image includes 'pyenv', and all Python
invocations that involve PATH lookup go through 'pyenv', e.g. our
'PYTHON_PATH=$(which python3)' sets '/opt/pyenv/shims/python3' as
PYTHON_PATH, which in turn will invoke '/usr/bin/python3'.  Alas, the
'pyenv' version included in this image is buggy, and prepends the
directory containing the Python binary to PATH even if that is a
system directory already in PATH near the end.  Consequently, 'git p4'
in those jobs ends up with its PATH starting with '/usr/bin', and then
runs '/usr/bin/git'.

So remove 'pyenv' in Travis CI's Linux jobs to prevet it from
interfering with our 'git p4' tests.

Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com>
---

Notes:
    This issue triggered test failures in all 'git p4' test scripts in the
    recent GIT_TEST_DEFAULT_HASH=sha256 test runs, because the system Git
    doesn't understand the 'objectformat' extension, e.g.:
    
      https://travis-ci.org/github/git/git/jobs/710159470#L3267
    
    This is not an issue in Travis CI's macOS jobs or on Azure Pipelines,
    because they don't use 'pyenv'.
    
    However, perhaps removing 'pyenv' is not the best solution here.
    
    We set PYTHON_PATH in 'ci/lib.sh', which is sourced at the beginning
    of (almost) all of our CI scripts.  Consequently, in these jobs we
    first run 'ci/install-dependencies.sh', which sources 'ci/lib.sh',
    assigns PYTHON_PATH=/opt/pyenv/shims/python3 (which is never used),
    and removes 'pyenv', and then run 'ci/run-build-and-test.sh', which
    sources 'ci/lib.sh' and assigns PYTHON_PATH=/usr/bin/python3 (which is
    then used in the build process to set 'pit p4's shebang line).  Not
    really nice, is it.
    
    Alternatively, we could avoid the PATH lookup and thus the bogus
    'pyenv' by explicitly using '/usr/bin/python{2,3}'.  The Linux images
    in both Travis CI and Azure Pipelines are standard Ubuntu images, so I
    think we can safely rely on these Python paths.

 ci/install-dependencies.sh | 8 ++++++++
 1 file changed, 8 insertions(+)

Comments

Eric Sunshine July 21, 2020, 4:23 p.m. UTC | #1
On Tue, Jul 21, 2020 at 12:12 PM SZEDER Gábor <szeder.dev@gmail.com> wrote:
> In our test suite, when 'git p4' invokes a Git command as a
> subprocesses, then it should run the 'git' binary we are testing.
> Unfortunately, this is not the case in the 'linux-clang' and
> 'linux-gcc' jobs on Travis CI, where 'git p4' runs the system
> '/usr/bin/git' instead.
>
> Travis CI's default Linux image includes 'pyenv', and all Python
> invocations that involve PATH lookup go through 'pyenv', e.g. our
> 'PYTHON_PATH=$(which python3)' sets '/opt/pyenv/shims/python3' as
> PYTHON_PATH, which in turn will invoke '/usr/bin/python3'.  Alas, the
> 'pyenv' version included in this image is buggy, and prepends the
> directory containing the Python binary to PATH even if that is a
> system directory already in PATH near the end.  Consequently, 'git p4'
> in those jobs ends up with its PATH starting with '/usr/bin', and then
> runs '/usr/bin/git'.
>
> So remove 'pyenv' in Travis CI's Linux jobs to prevet it from

s/prevet/prevent/

> interfering with our 'git p4' tests.
>
> diff --git a/ci/install-dependencies.sh b/ci/install-dependencies.sh
> @@ -36,6 +36,14 @@ linux-clang|linux-gcc)
> +
> +       if test true = "$TRAVIS" &&
> +          pyenv_root=$(pyenv root)
> +       then
> +               # pyenv in Travis CI's current default (xenial) Linux
> +               # image messes up PATH for 'git p4'.

I wonder if this comment should give more precise detail about exactly
how it "messes up" PATH.

> +               sudo rm -rf "$pyenv_root"
> +       fi
Junio C Hamano July 21, 2020, 8:34 p.m. UTC | #2
SZEDER Gábor <szeder.dev@gmail.com> writes:

>     Alternatively, we could avoid the PATH lookup and thus the bogus
>     'pyenv' by explicitly using '/usr/bin/python{2,3}'.  The Linux images
>     in both Travis CI and Azure Pipelines are standard Ubuntu images, so I
>     think we can safely rely on these Python paths.

That sounds quite sensible, actually.  The CI environment being
"special" in the sense that we have certain control over it makes
this rather straight-forward.
brian m. carlson July 21, 2020, 10:20 p.m. UTC | #3
On 2020-07-21 at 16:12:25, SZEDER Gábor wrote:
> In our test suite, when 'git p4' invokes a Git command as a
> subprocesses, then it should run the 'git' binary we are testing.
> Unfortunately, this is not the case in the 'linux-clang' and
> 'linux-gcc' jobs on Travis CI, where 'git p4' runs the system
> '/usr/bin/git' instead.
> 
> Travis CI's default Linux image includes 'pyenv', and all Python
> invocations that involve PATH lookup go through 'pyenv', e.g. our
> 'PYTHON_PATH=$(which python3)' sets '/opt/pyenv/shims/python3' as
> PYTHON_PATH, which in turn will invoke '/usr/bin/python3'.  Alas, the
> 'pyenv' version included in this image is buggy, and prepends the
> directory containing the Python binary to PATH even if that is a
> system directory already in PATH near the end.  Consequently, 'git p4'
> in those jobs ends up with its PATH starting with '/usr/bin', and then
> runs '/usr/bin/git'.

Ouch.  So we're testing git-p4, but not the git binary underlying it.
Fixing this definitely seems like a good idea.

Patch
diff mbox series

diff --git a/ci/install-dependencies.sh b/ci/install-dependencies.sh
index 0229a77f7d..b4bdcbcba2 100755
--- a/ci/install-dependencies.sh
+++ b/ci/install-dependencies.sh
@@ -36,6 +36,14 @@  linux-clang|linux-gcc)
 		tar --extract --gunzip --file "git-lfs-linux-amd64-$LINUX_GIT_LFS_VERSION.tar.gz"
 		cp git-lfs-$LINUX_GIT_LFS_VERSION/git-lfs .
 	popd
+
+	if test true = "$TRAVIS" &&
+	   pyenv_root=$(pyenv root)
+	then
+		# pyenv in Travis CI's current default (xenial) Linux
+		# image messes up PATH for 'git p4'.
+		sudo rm -rf "$pyenv_root"
+	fi
 	;;
 osx-clang|osx-gcc)
 	export HOMEBREW_NO_AUTO_UPDATE=1 HOMEBREW_NO_INSTALL_CLEANUP=1