[1/1] remote-curl: unbreak http.extraHeader with custom allocators

Commit Message

John Passaro via GitGitGadget Nov. 5, 2019, 9:59 p.m. UTC

From: Johannes Schindelin <[e-mail]>

In 93b980e58f5 (http: use xmalloc with cURL, 2019-08-15), we started to
ask cURL to use `xmalloc()`, and if compiled with nedmalloc, that means
implicitly a different allocator than the system one.

Which means that all of cURL's allocations and releases now _need_ to
use that allocator.

However, the `http_options()` function used `slist_append()` to add any
configured extra HTTP header(s) _before_ asking cURL to use `xmalloc()`,
and `http_cleanup()` would release them _afterwards_, i.e. in the
presence of custom allocators, cURL would attempt to use the wrong
allocator to release the memory.

Let's fix this by moving the initialization _before_ the
`http_options()` function is called.

Signed-off-by: Johannes Schindelin <[e-mail]>
---
 http.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

Comments

Jeff King Nov. 6, 2019, 4:16 a.m. UTC | #1

On Tue, Nov 05, 2019 at 09:59:18PM +0000, Johannes Schindelin via GitGitGadget wrote:

> From: Johannes Schindelin <[e-mail]>
> 
> In 93b980e58f5 (http: use xmalloc with cURL, 2019-08-15), we started to
> ask cURL to use `xmalloc()`, and if compiled with nedmalloc, that means
> implicitly a different allocator than the system one.
> 
> Which means that all of cURL's allocations and releases now _need_ to
> use that allocator.
> 
> However, the `http_options()` function used `slist_append()` to add any
> configured extra HTTP header(s) _before_ asking cURL to use `xmalloc()`,
> and `http_cleanup()` would release them _afterwards_, i.e. in the
> presence of custom allocators, cURL would attempt to use the wrong
> allocator to release the memory.
> 
> Let's fix this by moving the initialization _before_ the
> `http_options()` function is called.

Nicely explained.

Another option would be to separate our config mechanism from curl
entirely by putting the list of headers into a string_list, and then
transforming it later into a curl_slist. I don't think that really buys
us much, though. This is all inside http.c, so it's fairly contained.
It's not like other random parts of Git are using curl's slist before
calling http_init().

I did briefly grep around for other slist users, but they're all what
you'd expect: code in http-push.c and remote-curl.c creating header
lists while working with an active http request (so well after
http_init() has been called).

> ---
>  http.c | 6 +++---
>  1 file changed, 3 insertions(+), 3 deletions(-)

The patch itself looks good.

-Peff

Johannes Schindelin Nov. 6, 2019, 9:14 a.m. UTC | #2

Hi Peff,

On Tue, 5 Nov 2019, Jeff King wrote:

> On Tue, Nov 05, 2019 at 09:59:18PM +0000, Johannes Schindelin via GitGitGadget wrote:
>
> > From: Johannes Schindelin <[e-mail]>
> >
> > In 93b980e58f5 (http: use xmalloc with cURL, 2019-08-15), we started to
> > ask cURL to use `xmalloc()`, and if compiled with nedmalloc, that means
> > implicitly a different allocator than the system one.
> >
> > Which means that all of cURL's allocations and releases now _need_ to
> > use that allocator.
> >
> > However, the `http_options()` function used `slist_append()` to add any
> > configured extra HTTP header(s) _before_ asking cURL to use `xmalloc()`,
> > and `http_cleanup()` would release them _afterwards_, i.e. in the
> > presence of custom allocators, cURL would attempt to use the wrong
> > allocator to release the memory.
> >
> > Let's fix this by moving the initialization _before_ the
> > `http_options()` function is called.
>
> Nicely explained.
>
> Another option would be to separate our config mechanism from curl
> entirely by putting the list of headers into a string_list, and then
> transforming it later into a curl_slist. I don't think that really buys
> us much, though.

Alas, it _does_ buy us a lot, as I *just* found out (can you imagine how
glad I am not to have rushed out another Git for Windows release?). It
buys us one less bug: my patch introduces the bug where
`http.sslbackend` can no longer be used, because `curl_global_sslset()`
needs to be called _before_ `curl_global_init()`, but my patch breaks
that because we _need_ to parse the config before we can ask cURL for a
specific backend.

> This is all inside http.c, so it's fairly contained.  It's not like
> other random parts of Git are using curl's slist before calling
> http_init().

Indeed. We cannot use cURL's slist anywhere outside of the
cURL-dependent code because we want to keep `NO_CURL=Yep` working.

> I did briefly grep around for other slist users, but they're all what
> you'd expect: code in http-push.c and remote-curl.c creating header
> lists while working with an active http request (so well after
> http_init() has been called).

Indeed, I came to the same conclusion that Carlo's patch only broke
support for `http.extraheaders` (and only with custom allocators),
nothing else.

I will change the patch to avoid using `slist` early and send another
iteration.

Thanks,
Dscho

> > ---
> >  http.c | 6 +++---
> >  1 file changed, 3 insertions(+), 3 deletions(-)
>
> The patch itself looks good.
>
> -Peff
>

Junio C Hamano Nov. 6, 2019, 9:49 a.m. UTC | #3

Johannes Schindelin <[e-mail]> writes:

> On Tue, 5 Nov 2019, Jeff King wrote:
>
>> ... transforming it later into a curl_slist. I don't think that really buys
>> us much, though.
>
> Alas, it _does_ buy us a lot, as I *just* found out (can you imagine how
> glad I am not to have rushed out another Git for Windows release?).

Thanks both, and especially thanks Dscho for both for a fix and for
restraining yourself from the urge to pull trigger too soon ;-)

> I will change the patch to avoid using `slist` early and send another
> iteration.

Will look forward to seeing it.

Thanks.

Johannes Schindelin Nov. 6, 2019, 7:38 p.m. UTC | #4

Hi Junio,

On Wed, 6 Nov 2019, Junio C Hamano wrote:

> Johannes Schindelin <[e-mail]> writes:
>
> > On Tue, 5 Nov 2019, Jeff King wrote:
> >
> >> ... transforming it later into a curl_slist. I don't think that really buys
> >> us much, though.
> >
> > Alas, it _does_ buy us a lot, as I *just* found out (can you imagine how
> > glad I am not to have rushed out another Git for Windows release?).
>
> Thanks both, and especially thanks Dscho for both for a fix and for
> restraining yourself from the urge to pull trigger too soon ;-)

Well, I _will_ trigger the pipeline publishing Git for Windows
v2.24.0(2) in a moment, with this fix, but more importantly with a fix
for the installer that tries to install some Cygwin-style symlinks,
which was broken in non-US locales (most notably in a Japanese setup;
apparently there _still_ are developers who did not yet move away from a
Japanese locale... at least on Windows).

Ciao,
Dscho

>
> > I will change the patch to avoid using `slist` early and send another
> > iteration.
>
> Will look forward to seeing it.
>
> Thanks.
>

Junio C Hamano Nov. 8, 2019, 8:34 a.m. UTC | #5

Johannes Schindelin <[e-mail]> writes:

> ... which was broken in non-US locales (most notably in a Japanese setup;
> apparently there _still_ are developers who did not yet move away from a
> Japanese locale... at least on Windows).

I think the above was a reference to a tangential comment I made in
a response to Peff.

Oh, I do not expect Japanese Windows users would "move away from"
Japanese locale, ever.

I do however know that an app that supports only legacy encoding is
frowned upon these days.  They still use and will keep using
Japanese menus and messages, and they still write their documents in
Japanese and not in US English.

But they store their Japenese documents encoded in UTF-8 on their
system that is in Japanese locale.

There are two models of gadgets I am interested in getting, between
which one of them that is slightly older supports UTF-8 and MS-Kanji
(aka Shift-JIS) while the latest model only supports MS-Kanji.  The
list price of them are comparable (actually, the latest one lists a
bit more), but the latest model is deeply discounted while the other
one with UTF-8 is not as much.  At shopping sites, user reviews
often mention "I've migrated my text to UTF-8 already and going back
to Shift JIS in this year is too cumbersome, so I'll skip this
latest model".

I am hoping a software update might happen, and will pull the
trigger once the latest one starts supporting UTF-8 ;-)

Johannes Schindelin Nov. 8, 2019, 1:44 p.m. UTC | #6

Hi Junio,

On Fri, 8 Nov 2019, Junio C Hamano wrote:

> Johannes Schindelin <[e-mail]> writes:
>
> > ... which was broken in non-US locales (most notably in a Japanese setup;
> > apparently there _still_ are developers who did not yet move away from a
> > Japanese locale... at least on Windows).
>
> I think the above was a reference to a tangential comment I made in
> a response to Peff.
>
> Oh, I do not expect Japanese Windows users would "move away from"
> Japanese locale, ever.
>
> I do however know that an app that supports only legacy encoding is
> frowned upon these days.  They still use and will keep using
> Japanese menus and messages, and they still write their documents in
> Japanese and not in US English.
>
> But they store their Japenese documents encoded in UTF-8 on their
> system that is in Japanese locale.
>
> There are two models of gadgets I am interested in getting, between
> which one of them that is slightly older supports UTF-8 and MS-Kanji
> (aka Shift-JIS) while the latest model only supports MS-Kanji.  The
> list price of them are comparable (actually, the latest one lists a
> bit more), but the latest model is deeply discounted while the other
> one with UTF-8 is not as much.  At shopping sites, user reviews
> often mention "I've migrated my text to UTF-8 already and going back
> to Shift JIS in this year is too cumbersome, so I'll skip this
> latest model".
>
> I am hoping a software update might happen, and will pull the
> trigger once the latest one starts supporting UTF-8 ;-)

Thank you for that interesting note.

Ciao,
Dscho

Patch

diff --git a/http.c b/http.c
index 27aa0a3192..13f50ba158 100644
--- a/http.c
+++ b/http.c
@@ -1062,6 +1062,9 @@  void http_init(struct remote *remote, const char *url, int proactive_auth)
 	char *normalized_url;
 	struct urlmatch_config config = { STRING_LIST_INIT_DUP };
 
+	if (curl_global_init(CURL_GLOBAL_ALL) != CURLE_OK)
+		die("curl_global_init failed");
+
 	config.section = "http";
 	config.key = NULL;
 	config.collect_fn = http_options;
@@ -1101,9 +1104,6 @@  void http_init(struct remote *remote, const char *url, int proactive_auth)
 	}
 #endif
 
-	if (curl_global_init(CURL_GLOBAL_ALL) != CURLE_OK)
-		die("curl_global_init failed");
-
 	http_proactive_auth = proactive_auth;
 
 	if (remote && remote->http_proxy)