| Message ID | 372ab24acffbc956407cd93ed34135f83156e10d.1571316454.git.gitgitgadget@gmail.com (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
| Series | ci: update caskroom/cask/perforce to new location | expand |
On Thu, Oct 17, 2019 at 12:47:33PM +0000, Johannes Schindelin via GitGitGadget wrote: > From: Johannes Schindelin <johannes.schindelin@gmx.de> > > The CI builds are failing for Mac OS X due to a change in the s/CI/Azure Pipelines/ Our Travis CI builds are fine. > location of the perforce cask. The command outputs the following > error: > > + brew install caskroom/cask/perforce > Error: caskroom/cask was moved. Tap homebrew/cask-cask instead. > > So let's try to call `brew cask install perforce` first (which is what > that error message suggests, in a most round-about way). > > The "caskroom" way was added in 672f51cb (travis-ci: > fix Perforce install on macOS, 2017-01-22) and the justification > is that the call "brew cask install perforce" can fail due to a checksum > mismatch: the recipe simply downloads the official Perforce distro, and > whenever that is updated, the recipe needs to be updated, too. This paragraph is wrong, it mixes up things too much. Prior to 672f51cb we used to install the 'perforce' _package_ with 'brew install perforce' (note: no 'cask' in there). The justification for 672f51cb was that the command 'brew install perforce' simply stopped working, after Homebrew folks decided that it's better to move the 'perforce' package to a "cask". It was _their_ justification for this move that 'brew install perforce' "can fail due to a checksum mismatch ...", and casks can be installed without checksum verification. And indeed, both 'brew cask install perforce' and 'brew install caskroom/cask/perforce' printed something along the lines of: ==> No checksum defined for Cask perforce, skipping verification It's unclear to me why 672f51cb used 'brew install caskroom/cask/perforce' instead of 'brew cask install perforce'. It appears (by running both commands on old Travis CI macOS images) that both commands worked all the same already back then. Anyway, as the error message at the top of the log message shows, 'brew install caskroom/cask/perforce' has stopped working recently, but 'brew cask install perforce' still does, so let's use that. Note that on Travis CI we explicitly specify which macOS image to use, and nowadays we don't run 'brew update' during the build process [1], so both commands work in our builds there. [1] f2f4715033 (ci: don't update Homebrew, 2019-07-03) > CI servers are typically fresh virtual machines, but not always. To > accommodate for that, let's try harder if `brew cask install perforce` > fails, by specifically pulling the latest `master` of the > `homebrew-cask` repository. Homebrew didn't record a checksum for Perforce versions r17.1, r17.2 and r18.1, so installing those still works fine. Our Travis CI images install r18.1. However, when Homebrew updated to Perforce r19.1, they included the checksum again for some reason (intentional or accidental; I didn't look why). This worked fine for a while, until a couple of days ago Perforce updated the r19.1 binaries in place, breaking those checksums. If we were to still run 'brew update', then it would shortly fix the checksum mismatch. But we don't run it, and we do not want to run it because it takes ages. Falling back to pull from the 'homebrew-cask' repository could be a reasonable and quick workaround. > This will still fail, of course, when `homebrew-cask` falls behind > Perforce's release schedule. But once it is updated, we can now simply > re-run the failed jobs and they will pick up that update. In our CI builds we don't at all care what the checksums of the Perforce binaries are, so I would really like to tell 'brew' to ignore any checksum mismatch when installing 'perforce'. Alas, it appears that 'brew' has no public options to turn of or to ignore checksum verification. Now, let's take a step back. All 'brew cask install perforce' really does is run 'curl' to download a tar.gz from the Perforce servers, verify its checksum, unpack it, and put the executables somewhere on $PATH. That's not rocket science, we could easily do that ourselves; we don't even have to deal with a tar.gz, the 'p4' and 'p4d' binaries for mac are readily available for download at: http://filehost.perforce.com/perforce/r19.1/bin.macosx1010x86_64/ And, in fact, that's what we have been doing in some of our Linux jobs since the very beginning, so basically only the download URL has to be adjusted. > As for updating `homebrew-cask`: the beginnings of automating this in > https://dev.azure.com/gitgitgadget/git/_build?definitionId=11&_a=summary > will be finished once the next Perforce upgrade comes around. > > Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> > Signed-off-by: Derrick Stolee <dstolee@microsoft.com> > --- > ci/install-dependencies.sh | 5 +++++ > 1 file changed, 5 insertions(+) > > diff --git a/ci/install-dependencies.sh b/ci/install-dependencies.sh > index 85a9d6b15c..ce149ed39c 100755 > --- a/ci/install-dependencies.sh > +++ b/ci/install-dependencies.sh > @@ -40,6 +40,11 @@ osx-clang|osx-gcc) > test -z "$BREW_INSTALL_PACKAGES" || > brew install $BREW_INSTALL_PACKAGES > brew link --force gettext > + brew cask install perforce || { > + # Update the definitions and try again > + git -C "$(brew --repository)"/Library/Taps/homebrew/homebrew-cask pull && > + brew cask install perforce > + } || > brew install caskroom/cask/perforce > case "$jobname" in > osx-gcc) > -- > gitgitgadget
SZEDER Gábor <szeder.dev@gmail.com> writes: > On Thu, Oct 17, 2019 at 12:47:33PM +0000, Johannes Schindelin via GitGitGadget wrote: >> From: Johannes Schindelin <johannes.schindelin@gmx.de> >> >> The CI builds are failing for Mac OS X due to a change in the > > s/CI/Azure Pipelines/ > > Our Travis CI builds are fine. > >> location of the perforce cask. The command outputs the following >> error: >> >> + brew install caskroom/cask/perforce >> Error: caskroom/cask was moved. Tap homebrew/cask-cask instead. >> >> So let's try to call `brew cask install perforce` first (which is what >> that error message suggests, in a most round-about way). >> >> The "caskroom" way was added in 672f51cb (travis-ci: >> fix Perforce install on macOS, 2017-01-22) and the justification >> is that the call "brew cask install perforce" can fail due to a checksum >> mismatch: the recipe simply downloads the official Perforce distro, and >> whenever that is updated, the recipe needs to be updated, too. > > This paragraph is wrong, it mixes up things too much. > > Prior to 672f51cb we used to install the 'perforce' _package_ with > 'brew install perforce' (note: no 'cask' in there). The justification > for 672f51cb was that the command 'brew install perforce' simply > stopped working, after Homebrew folks decided that it's better to move > the 'perforce' package to a "cask". It was _their_ justification for > this move that 'brew install perforce' "can fail due to a checksum > mismatch ...", and casks can be installed without checksum > verification. And indeed, both 'brew cask install perforce' and 'brew > install caskroom/cask/perforce' printed something along the lines of: > > ==> No checksum defined for Cask perforce, skipping verification > > It's unclear to me why 672f51cb used 'brew install > caskroom/cask/perforce' instead of 'brew cask install perforce'. It > appears (by running both commands on old Travis CI macOS images) that > both commands worked all the same already back then. > > Anyway, as the error message at the top of the log message shows, > 'brew install caskroom/cask/perforce' has stopped working recently, > but 'brew cask install perforce' still does, so let's use that. > > Note that on Travis CI we explicitly specify which macOS image to use, > and nowadays we don't run 'brew update' during the build process [1], > so both commands work in our builds there. > ... > Now, let's take a step back. > > All 'brew cask install perforce' really does is ... > ... in fact, that's what we have been doing in some of our Linux jobs > since the very beginning, so basically only the download URL has to be > adjusted. This is already in 'next' X-<; reverting a merge is cheap but I prefer to do so when we already have a replacement. Thanks for taking a closer look.
Hi Gábor, On Fri, 18 Oct 2019, SZEDER Gábor wrote: > On Thu, Oct 17, 2019 at 12:47:33PM +0000, Johannes Schindelin via GitGitGadget wrote: > > From: Johannes Schindelin <johannes.schindelin@gmx.de> > > > > The CI builds are failing for Mac OS X due to a change in the > > s/CI/Azure Pipelines/ > > Our Travis CI builds are fine. For the moment ;-) > > location of the perforce cask. The command outputs the following > > error: > > > > + brew install caskroom/cask/perforce > > Error: caskroom/cask was moved. Tap homebrew/cask-cask instead. > > > > So let's try to call `brew cask install perforce` first (which is what > > that error message suggests, in a most round-about way). > > > > The "caskroom" way was added in 672f51cb (travis-ci: > > fix Perforce install on macOS, 2017-01-22) and the justification > > is that the call "brew cask install perforce" can fail due to a checksum > > mismatch: the recipe simply downloads the official Perforce distro, and > > whenever that is updated, the recipe needs to be updated, too. > > This paragraph is wrong, it mixes up things too much. > > Prior to 672f51cb we used to install the 'perforce' _package_ with > 'brew install perforce' (note: no 'cask' in there). The justification > for 672f51cb was that the command 'brew install perforce' simply > stopped working, after Homebrew folks decided that it's better to move > the 'perforce' package to a "cask". It was _their_ justification for > this move that 'brew install perforce' "can fail due to a checksum > mismatch ...", and casks can be installed without checksum > verification. And indeed, both 'brew cask install perforce' and 'brew > install caskroom/cask/perforce' printed something along the lines of: > > ==> No checksum defined for Cask perforce, skipping verification > > It's unclear to me why 672f51cb used 'brew install > caskroom/cask/perforce' instead of 'brew cask install perforce'. It > appears (by running both commands on old Travis CI macOS images) that > both commands worked all the same already back then. > > Anyway, as the error message at the top of the log message shows, > 'brew install caskroom/cask/perforce' has stopped working recently, > but 'brew cask install perforce' still does, so let's use that. If you don't mind, I am going to copy/edit these three paragraphs into the commit message, > Note that on Travis CI we explicitly specify which macOS image to use, > and nowadays we don't run 'brew update' during the build process [1], > so both commands work in our builds there. > > [1] f2f4715033 (ci: don't update Homebrew, 2019-07-03) > > > CI servers are typically fresh virtual machines, but not always. To > > accommodate for that, let's try harder if `brew cask install perforce` > > fails, by specifically pulling the latest `master` of the > > `homebrew-cask` repository. > > Homebrew didn't record a checksum for Perforce versions r17.1, r17.2 > and r18.1, so installing those still works fine. Our Travis CI images > install r18.1. > > However, when Homebrew updated to Perforce r19.1, they included the > checksum again for some reason (intentional or accidental; I didn't > look why). This worked fine for a while, until a couple of days ago > Perforce updated the r19.1 binaries in place, breaking those > checksums. > > If we were to still run 'brew update', then it would shortly fix the > checksum mismatch. But we don't run it, and we do not want to run it > because it takes ages. Falling back to pull from the 'homebrew-cask' > repository could be a reasonable and quick workaround. Okay, good. > > This will still fail, of course, when `homebrew-cask` falls behind > > Perforce's release schedule. But once it is updated, we can now simply > > re-run the failed jobs and they will pick up that update. > > In our CI builds we don't at all care what the checksums of the > Perforce binaries are, so I would really like to tell 'brew' to ignore > any checksum mismatch when installing 'perforce'. Alas, it appears > that 'brew' has no public options to turn of or to ignore checksum > verification. Sad, yet true, that we indeed have no command-line option to say "you know what, your checksum possibly mismatches, but we really don't care". > Now, let's take a step back. > > All 'brew cask install perforce' really does is run 'curl' to download > a tar.gz from the Perforce servers, verify its checksum, unpack it, > and put the executables somewhere on $PATH. That's not rocket > science, we could easily do that ourselves; we don't even have to deal > with a tar.gz, the 'p4' and 'p4d' binaries for mac are readily > available for download at: > > http://filehost.perforce.com/perforce/r19.1/bin.macosx1010x86_64/ > > And, in fact, that's what we have been doing in some of our Linux jobs > since the very beginning, so basically only the download URL has to be > adjusted. I'd rather not. Just because there is no better way on Linux, and just because the current `perforce` cask recipe happens to just download and unpack that file does not mean that this won't change. And if it changes, we will be a lot better off by using the provided package. As I wrote here: > > As for updating `homebrew-cask`: the beginnings of automating this in > > https://dev.azure.com/gitgitgadget/git/_build?definitionId=11&_a=summary > > will be finished once the next Perforce upgrade comes around. I am quite willing to do my share to keep the Homebrew recipe for `perforce` up to date. We'll all be better off that way. Ciao, Dscho > > > > Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> > > Signed-off-by: Derrick Stolee <dstolee@microsoft.com> > > --- > > ci/install-dependencies.sh | 5 +++++ > > 1 file changed, 5 insertions(+) > > > > diff --git a/ci/install-dependencies.sh b/ci/install-dependencies.sh > > index 85a9d6b15c..ce149ed39c 100755 > > --- a/ci/install-dependencies.sh > > +++ b/ci/install-dependencies.sh > > @@ -40,6 +40,11 @@ osx-clang|osx-gcc) > > test -z "$BREW_INSTALL_PACKAGES" || > > brew install $BREW_INSTALL_PACKAGES > > brew link --force gettext > > + brew cask install perforce || { > > + # Update the definitions and try again > > + git -C "$(brew --repository)"/Library/Taps/homebrew/homebrew-cask pull && > > + brew cask install perforce > > + } || > > brew install caskroom/cask/perforce > > case "$jobname" in > > osx-gcc) > > -- > > gitgitgadget >
Hi Junio, On Mon, 21 Oct 2019, Junio C Hamano wrote: > SZEDER Gábor <szeder.dev@gmail.com> writes: > > > On Thu, Oct 17, 2019 at 12:47:33PM +0000, Johannes Schindelin via GitGitGadget wrote: > >> From: Johannes Schindelin <johannes.schindelin@gmx.de> > >> > >> The CI builds are failing for Mac OS X due to a change in the > > > > s/CI/Azure Pipelines/ > > > > Our Travis CI builds are fine. > > > >> location of the perforce cask. The command outputs the following > >> error: > >> > >> + brew install caskroom/cask/perforce > >> Error: caskroom/cask was moved. Tap homebrew/cask-cask instead. > >> > >> So let's try to call `brew cask install perforce` first (which is what > >> that error message suggests, in a most round-about way). > >> > >> The "caskroom" way was added in 672f51cb (travis-ci: > >> fix Perforce install on macOS, 2017-01-22) and the justification > >> is that the call "brew cask install perforce" can fail due to a checksum > >> mismatch: the recipe simply downloads the official Perforce distro, and > >> whenever that is updated, the recipe needs to be updated, too. > > > > This paragraph is wrong, it mixes up things too much. > > > > Prior to 672f51cb we used to install the 'perforce' _package_ with > > 'brew install perforce' (note: no 'cask' in there). The justification > > for 672f51cb was that the command 'brew install perforce' simply > > stopped working, after Homebrew folks decided that it's better to move > > the 'perforce' package to a "cask". It was _their_ justification for > > this move that 'brew install perforce' "can fail due to a checksum > > mismatch ...", and casks can be installed without checksum > > verification. And indeed, both 'brew cask install perforce' and 'brew > > install caskroom/cask/perforce' printed something along the lines of: > > > > ==> No checksum defined for Cask perforce, skipping verification > > > > It's unclear to me why 672f51cb used 'brew install > > caskroom/cask/perforce' instead of 'brew cask install perforce'. It > > appears (by running both commands on old Travis CI macOS images) that > > both commands worked all the same already back then. > > > > Anyway, as the error message at the top of the log message shows, > > 'brew install caskroom/cask/perforce' has stopped working recently, > > but 'brew cask install perforce' still does, so let's use that. > > > > Note that on Travis CI we explicitly specify which macOS image to use, > > and nowadays we don't run 'brew update' during the build process [1], > > so both commands work in our builds there. > > ... > > Now, let's take a step back. > > > > All 'brew cask install perforce' really does is ... > > ... in fact, that's what we have been doing in some of our Linux jobs > > since the very beginning, so basically only the download URL has to be > > adjusted. > > This is already in 'next' X-<; reverting a merge is cheap but I > prefer to do so when we already have a replacement. I force-pushed (see https://github.com/gitgitgadget/git/pull/400), and once Stolee approves, he will submit v3. This will only change the commit message, though, as I disagree that hard-coding the URL would be an improvement: the nice thing about a package management system is that the user does not need to know the details (or need to know if the details change, like, ever). Ciao, Dscho > > Thanks for taking a closer look. >
Johannes Schindelin <Johannes.Schindelin@gmx.de> writes: >> This is already in 'next' X-<; reverting a merge is cheap but I >> prefer to do so when we already have a replacement. > > I force-pushed (see https://github.com/gitgitgadget/git/pull/400), and > once Stolee approves, he will submit v3. This will only change the > commit message, though, as I disagree that hard-coding the URL would be > an improvement: the nice thing about a package management system is that > the user does not need to know the details (or need to know if the > details change, like, ever). If this were meant for the upcoming release, I would rather see us copy a butt-ugly-but-known-working procedure if we have one this close to -rc1. If the hard-coded URL ever changes, the procedure we would be copying from would be broken anyway. But I agree 100% that we should take a conceptually cleaner approach for the longer term. Let's replace the original one with this and cook in 'next'---it would be ideal if the ugly-but-know-working one be updated to match in the meantime, but if it is bypassing package management for a reason (the upstream just publishes the URL to download from without packaging it properly, for example?), that would not be possible, and it is OK if that is the case. Thanks.
On Wed, Oct 23, 2019 at 01:23:25AM +0200, Johannes Schindelin wrote: > On Fri, 18 Oct 2019, SZEDER Gábor wrote: > > > On Thu, Oct 17, 2019 at 12:47:33PM +0000, Johannes Schindelin via GitGitGadget wrote: > > > From: Johannes Schindelin <johannes.schindelin@gmx.de> > > > > > > The CI builds are failing for Mac OS X due to a change in the > > > > s/CI/Azure Pipelines/ > > > > Our Travis CI builds are fine. > > For the moment ;-) Touché. Believe it or not, I did wrote "at least for now" at the end of that sentence, but then deleted it. Serves me right, now there is some new breakage with gcc@8... :) > If you don't mind, I am going to copy/edit these three paragraphs into > the commit message, Sure, go ahead. > > In our CI builds we don't at all care what the checksums of the > > Perforce binaries are, so I would really like to tell 'brew' to ignore > > any checksum mismatch when installing 'perforce'. Alas, it appears > > that 'brew' has no public options to turn of or to ignore checksum > > verification. > > Sad, yet true, that we indeed have no command-line option to say "you > know what, your checksum possibly mismatches, but we really don't care". Actually, 'brew' does have some undocumented options, but I didn't even bothered to check, because it's not really sensible to rely on an undocumented option (especially when even the documented options break every once in a while...). > > Now, let's take a step back. > > > > All 'brew cask install perforce' really does is run 'curl' to download > > a tar.gz from the Perforce servers, verify its checksum, unpack it, > > and put the executables somewhere on $PATH. That's not rocket > > science, we could easily do that ourselves; we don't even have to deal > > with a tar.gz, the 'p4' and 'p4d' binaries for mac are readily > > available for download at: > > > > http://filehost.perforce.com/perforce/r19.1/bin.macosx1010x86_64/ > > > > And, in fact, that's what we have been doing in some of our Linux jobs > > since the very beginning, so basically only the download URL has to be > > adjusted. > > I'd rather not. > > Just because there is no better way on Linux, and just because the > current `perforce` cask recipe happens to just download and unpack that > file does not mean that this won't change. Yeah, I'm fairly sure that the way Homebrew installs Perforce will change, but if we download Perforce ourselves, then it won't matter at all. Downloading the Perforce binaries with 'wget' worked fairly well for almost four years, except from that server glitch a couple of weeks ago; I think downloading the macOS binaries from the same server would work just as well. OTOH, this is the fourth time that we have to tweak how we install Perforce via Homebrew. FWIW, it looks like this: https://github.com/szeder/git/blob/ci-osx-wget-perforce/ci/install-dependencies.sh#L11
diff --git a/ci/install-dependencies.sh b/ci/install-dependencies.sh index 85a9d6b15c..ce149ed39c 100755 --- a/ci/install-dependencies.sh +++ b/ci/install-dependencies.sh @@ -40,6 +40,11 @@ osx-clang|osx-gcc) test -z "$BREW_INSTALL_PACKAGES" || brew install $BREW_INSTALL_PACKAGES brew link --force gettext + brew cask install perforce || { + # Update the definitions and try again + git -C "$(brew --repository)"/Library/Taps/homebrew/homebrew-cask pull && + brew cask install perforce + } || brew install caskroom/cask/perforce case "$jobname" in osx-gcc)