Message ID | cover-v2-0.8-00000000000-20210910T105523Z-avarab@gmail.com (mailing list archive) |
---|---|
Headers | show |
Series | post-v2.33 "drop support for ancient curl" follow-up | expand |
On Fri, Sep 10, 2021 at 01:04:25PM +0200, Ævar Arnfjörð Bjarmason wrote: > Version 1 of this had a really bad bug where we'd effectively make all > supported curl versions act like 7.19.4, i.e. the oldest supported > version except for a couple of our supported features. This is because > most of the things checked with the "ifdef" checks are enum fields, > not macros. So basically the "devil's advocate" Jeff King pointed out > in [2] was already the case. Oops! Oops. :) Well, I'm glad I mentioned it, then (I really was just playing devil's advocate). It does seem a little scary that we'd compile without all of these features and not even notice it. I guess it's hard to test for those features when we don't know for sure that our libcurl supports them. We could perhaps make the tests optional, but how do we set up the prereqs? Version checks based on "curl --version" or even "pkg-config --modversion libcurl" seems error-prone. The curl binary might not match the library we used, or the user might have given us a specific libcurl to use rather than the pkg-config version. We could teach Git to tell us which features it thinks curl supports, but that's depending on the very thing we're trying to test. We could have Git output the LIBCURL_VERSION_NUM it saw at build time, but then we're just implementing the same "if version > X, we have this feature" logic now in the test suite. So I dunno. I do not have any clever solution that would have caught this automatically without creating an even bigger maintenance headache. (Though for other reasons, it might be nice to report the curl version from "git version --build-options". This is a bit tricky because we avoid linking at libcurl at all in the main binary. Definitely orthogonal to your series, anyway). > This means that anyone on an older distro with backported features > will need to -DGIT_CURL_HAVE_* if their version of curl supports some > of these via a backport, not ideal, but an acceptable trade-off. If we > cared about this we could have some "detect-curl" script similar to my > proposed "detect-compiler"[3] (or another homegrown autoconf > replacement). I think that's acceptable. This should be rare, and they can always set CFLAGS appropriately. -Peff
On Fri, Sep 10 2021, Jeff King wrote: > On Fri, Sep 10, 2021 at 01:04:25PM +0200, Ævar Arnfjörð Bjarmason wrote: > >> Version 1 of this had a really bad bug where we'd effectively make all >> supported curl versions act like 7.19.4, i.e. the oldest supported >> version except for a couple of our supported features. This is because >> most of the things checked with the "ifdef" checks are enum fields, >> not macros. So basically the "devil's advocate" Jeff King pointed out >> in [2] was already the case. Oops! > > Oops. :) > > Well, I'm glad I mentioned it, then (I really was just playing devil's > advocate). It does seem a little scary that we'd compile without all of > these features and not even notice it. > > I guess it's hard to test for those features when we don't know for sure > that our libcurl supports them. We could perhaps make the tests > optional, but how do we set up the prereqs? > > Version checks based on "curl --version" or even "pkg-config > --modversion libcurl" seems error-prone. The curl binary might not match > the library we used, or the user might have given us a specific libcurl > to use rather than the pkg-config version. We could teach Git to tell us > which features it thinks curl supports, but that's depending on the very > thing we're trying to test. We could have Git output the > LIBCURL_VERSION_NUM it saw at build time, but then we're just > implementing the same "if version > X, we have this feature" logic now > in the test suite. > > So I dunno. I do not have any clever solution that would have caught > this automatically without creating an even bigger maintenance headache. Yeah, ideally we'd have tests for these optional features, and we'd then just add them to GIT-BUILD-OPTIONS and skip dedicated tests appropriately, then it would be more visible to see those tests skipped. Presumably someone testing this would run "make test" against a glob that includes *curl*. But that's not easy to do in practice since the flags are either not visible from the outside in terms of behavior, or if they are it's all something that requires proxies, SSL etc, which we don't test currently. We can and should test that, but requires e.g. extending lib-httpd.sh to start apache with ssl support, or maybe we could do it more easily with an optional stunnel or something... > (Though for other reasons, it might be nice to report the curl version > from "git version --build-options". This is a bit tricky because we > avoid linking at libcurl at all in the main binary. Definitely > orthogonal to your series, anyway). We could always ship a git-version-curl and shell out to it, or embed the version curl-config --vernum gave us at compile time via Makefile -> -DGIT_CURL_VERSION. That version could be changed at runtime. We could call that a feature. It's --build-options, not --runtime-libraries :)
On Fri, Sep 10, 2021 at 05:08:32PM +0200, Ævar Arnfjörð Bjarmason wrote: > > So I dunno. I do not have any clever solution that would have caught > > this automatically without creating an even bigger maintenance headache. > > Yeah, ideally we'd have tests for these optional features, and we'd then > just add them to GIT-BUILD-OPTIONS and skip dedicated tests > appropriately, then it would be more visible to see those tests > skipped. Presumably someone testing this would run "make test" against a > glob that includes *curl*. > > But that's not easy to do in practice since the flags are either not > visible from the outside in terms of behavior, or if they are it's all > something that requires proxies, SSL etc, which we don't test currently. > > We can and should test that, but requires e.g. extending lib-httpd.sh to > start apache with ssl support, or maybe we could do it more easily with > an optional stunnel or something... Yeah, even after we get the right version/feature flags into the test suite, most of these are pretty obscure and hard to test. I definitely think that's not something we should worry about for this series. > > (Though for other reasons, it might be nice to report the curl version > > from "git version --build-options". This is a bit tricky because we > > avoid linking at libcurl at all in the main binary. Definitely > > orthogonal to your series, anyway). > > We could always ship a git-version-curl and shell out to it, or embed > the version curl-config --vernum gave us at compile time via Makefile -> > -DGIT_CURL_VERSION. Yeah, I was thinking "git remote-curl --curl-version" or something. Let's punt on it for now, though. I think this series is getting close to ready, and this is all only semi-related. -Peff
Ævar Arnfjörð Bjarmason <avarab@gmail.com> writes: > Version 1 of this had a really bad bug where we'd effectively make all > supported curl versions act like 7.19.4, i.e. the oldest supported > version except for a couple of our supported features. This is because > most of the things checked with the "ifdef" checks are enum fields, > not macros. So basically the "devil's advocate" Jeff King pointed out > in [2] was already the case. Oops! Wow. Thanks for bothering to actually test ;-) Because we learned that "#ifdef CURL_SOME_FEATURE_WE_WANT" is not a generally applicable way to conditionally build for various features and we'd need to switch on the version numbers, it is now clear (at least to me) that the central registry approach would be a direction to go. > In this v2 we're instead checking LIBCURL_VERSION_NUM consistently, > even in those cases where we are checking things that are defined via > macros. Nice. > ++ * For each X symbol we need from curl we define our own > ++ * GIT_CURL_HAVE_X. If multiple similar symbols with the same prefix > ++ * were defined in the same version we pick one and check for that name. > + * > + * Keep any symbols in date order of when their support was > + * introduced, oldest first, in the official version of cURL library. > @@ git-curl-compat.h (new) > +/** > + * CURLOPT_TCP_KEEPALIVE was added in 7.25.0, released in March 2012. > + */ > -+#ifdef CURLOPT_TCP_KEEPALIVE > ++#if LIBCURL_VERSION_NUM >= 0x071900 > +#define GITCURL_HAVE_CURLOPT_TCP_KEEPALIVE 1 > +#endif What we have in the posted patch is perfectly OK and serviceable, but this organization somewhat surprised me. Instead of one #if...#endif block per a feature macro, I expected to see a sequence of /* 7.34.0 (0x072200) - Dec 2013 */ #if 0x072200 < VERSION #define HAVE_FOO 1 #define HAVE_BAR 1 ... #endif /* 7.25.0 (0x071900) - March 2012 */ #if 0x071900 < VERSION #define HAVE_BAZ 1 #define HAVE_QUX 1 ... #endif because it would make it more clear which features can now be unconditionally used when we raise the cut-off point for the oldest supported version if we group these entries along the versions. Thanks.
On September 10, 2021 12:53 PM, Junio C Hamano wrote: >To: Ævar Arnfjörð Bjarmason <avarab@gmail.com> >Cc: git@vger.kernel.org; Jeff King <peff@peff.net>; brian m . carlson <sandals@crustytoothpaste.net>; Bagas Sanjaya ><bagasdotme@gmail.com> >Subject: Re: [PATCH v2 0/8] post-v2.33 "drop support for ancient curl" follow-up > >Ævar Arnfjörð Bjarmason <avarab@gmail.com> writes: > >> Version 1 of this had a really bad bug where we'd effectively make all >> supported curl versions act like 7.19.4, i.e. the oldest supported >> version except for a couple of our supported features. This is because >> most of the things checked with the "ifdef" checks are enum fields, >> not macros. So basically the "devil's advocate" Jeff King pointed out >> in [2] was already the case. Oops! > >Wow. Thanks for bothering to actually test ;-) > >Because we learned that "#ifdef CURL_SOME_FEATURE_WE_WANT" is not a generally applicable way to conditionally build for various >features and we'd need to switch on the version numbers, it is now clear (at least to me) that the central registry approach would be a >direction to go. > >> In this v2 we're instead checking LIBCURL_VERSION_NUM consistently, >> even in those cases where we are checking things that are defined via >> macros. > >Nice. > >> ++ * For each X symbol we need from curl we define our own >> ++ * GIT_CURL_HAVE_X. If multiple similar symbols with the same prefix >> ++ * were defined in the same version we pick one and check for that name. >> + * >> + * Keep any symbols in date order of when their support was >> + * introduced, oldest first, in the official version of cURL library. >> @@ git-curl-compat.h (new) >> +/** >> + * CURLOPT_TCP_KEEPALIVE was added in 7.25.0, released in March 2012. >> + */ >> -+#ifdef CURLOPT_TCP_KEEPALIVE >> ++#if LIBCURL_VERSION_NUM >= 0x071900 >> +#define GITCURL_HAVE_CURLOPT_TCP_KEEPALIVE 1 >> +#endif > >What we have in the posted patch is perfectly OK and serviceable, but this organization somewhat surprised me. > >Instead of one #if...#endif block per a feature macro, I expected to see a sequence of > > /* 7.34.0 (0x072200) - Dec 2013 */ > #if 0x072200 < VERSION > #define HAVE_FOO 1 > #define HAVE_BAR 1 > ... > #endif > > /* 7.25.0 (0x071900) - March 2012 */ > #if 0x071900 < VERSION > #define HAVE_BAZ 1 > #define HAVE_QUX 1 > ... > #endif > > >because it would make it more clear which features can now be unconditionally used when we raise the cut-off point for the oldest >supported version if we group these entries along the versions. > >Thanks. Not that it might matter much, but both NonStop platforms are now on curl 7.78.0 and we are deploying git under OpenSSL 3 this week or next. So any objections I might previously have had are now fortunately really moot. Cheers, -Randall
Ævar Arnfjörð Bjarmason <avarab@gmail.com> writes: > This is a follow-up to the already-integrated topic for dropping > support for older curl versions submitted before the v2.33 release[1]. To which commit are these expected to apply? I am having trouble wiggling 5 and 7 in. Thanks.
Junio C Hamano <gitster@pobox.com> writes: > Ævar Arnfjörð Bjarmason <avarab@gmail.com> writes: > >> This is a follow-up to the already-integrated topic for dropping >> support for older curl versions submitted before the v2.33 release[1]. > > To which commit are these expected to apply? I am having trouble > wiggling 5 and 7 in. Nevermind. I seem to be getting garbage from "b4 am", possibly an operator error. I've seen "b4 am" getting confused on a thread with multiple versions (like the config-based-hook thing where the mixture of Emily's v9 and your v5 for base in the same huge thread seem to confuse it).
On Fri, Sep 10 2021, Junio C Hamano wrote: > Ævar Arnfjörð Bjarmason <avarab@gmail.com> writes: > >> Version 1 of this had a really bad bug where we'd effectively make all >> supported curl versions act like 7.19.4, i.e. the oldest supported >> version except for a couple of our supported features. This is because >> most of the things checked with the "ifdef" checks are enum fields, >> not macros. So basically the "devil's advocate" Jeff King pointed out >> in [2] was already the case. Oops! > > Wow. Thanks for bothering to actually test ;-) FWIW I'd tested that this worked for CURL_SOCKOPT_OK for v1, but there I happened to pick one that *is* a macro. Then I also tested GIT_CURL_HAVE_CURLOPT_PINNEDPUBLICKEY, which isn't, but the way I wrote the code would have thrown a warning/error about the unused variable, except in v1 I managed to scew that up in the last patch, so we unconditionally used the relevant variable either way...
On Fri, Sep 10 2021, Junio C Hamano wrote: > Ævar Arnfjörð Bjarmason <avarab@gmail.com> writes: > >> This is a follow-up to the already-integrated topic for dropping >> support for older curl versions submitted before the v2.33 release[1]. > > To which commit are these expected to apply? I am having trouble > wiggling 5 and 7 in. I based this on top of the current master (8463beaeb69) and these apply cleanly to me on top of that: vm git (master $>) 0 $ git reset --hard @{u} HEAD is now at 8463beaeb69 The fourth batch vm git (master $=) 0 $ git am 000*patch Applying: INSTALL: don't mention the "curl" executable at all Applying: INSTALL: mention that we need libcurl 7.19.4 or newer to build Applying: Makefile: drop support for curl < 7.9.8 (again) Applying: http: drop support for curl < 7.18.0 (again) Applying: http: correct version check for CURL_HTTP_VERSION_2 Applying: http: correct curl version check for CURLOPT_PINNEDPUBLICKEY Applying: http: centralize the accounting of libcurl dependencies Applying: http: don't hardcode the value of CURL_SOCKOPT_OK So, weird, I can't imagine why they wouldn't apply...
On Fri, Sep 10, 2021 at 10:32:52AM -0700, Junio C Hamano wrote: > >> This is a follow-up to the already-integrated topic for dropping > >> support for older curl versions submitted before the v2.33 release[1]. > > > > To which commit are these expected to apply? I am having trouble > > wiggling 5 and 7 in. > > Nevermind. I seem to be getting garbage from "b4 am", possibly an > operator error. I've seen "b4 am" getting confused on a thread with > multiple versions (like the config-based-hook thing where the > mixture of Emily's v9 and your v5 for base in the same huge thread > seem to confuse it). Yeah, sorry, the thread is not great for figuring out which patch goes where (there are two v2 versions, one from July and one from today, and a v4 from July that b4 considers the latest). The only solution I can offer is doing "b4 mbox" first, then deleting everything you don't want, and then doing "b4 am -m that.mbox". I could add extra heuristics that also looks at series age, but I'm worried that the "what you get is what you (probably) meant" logic will just get more and more tortured. -K
Konstantin Ryabitsev <konstantin@linuxfoundation.org> writes: > The only solution I can offer is doing "b4 mbox" first, then deleting > everything you don't want, and then doing "b4 am -m that.mbox". Thanks. Marking the articles I want in GNUS (I read via nntp://lore) and downloading works just fine ;-)