From patchwork Fri Oct 21 17:07:58 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew John Cheetham X-Patchwork-Id: 13015173 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4C551FA373E for ; Fri, 21 Oct 2022 17:08:15 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230164AbiJURIN (ORCPT ); Fri, 21 Oct 2022 13:08:13 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49150 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230078AbiJURIL (ORCPT ); Fri, 21 Oct 2022 13:08:11 -0400 Received: from mail-wr1-x42c.google.com (mail-wr1-x42c.google.com [IPv6:2a00:1450:4864:20::42c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0338F280800 for ; Fri, 21 Oct 2022 10:08:10 -0700 (PDT) Received: by mail-wr1-x42c.google.com with SMTP id bv10so5992261wrb.4 for ; Fri, 21 Oct 2022 10:08:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:from:to:cc:subject:date :message-id:reply-to; bh=ZS+vJncVOAy2CYmx4D6/h8tcRsGlPjDDfC+97kQiBFE=; b=q0A9sLNu+c8MdyDCPMtTHNhMp7KNCgxQDu9tLdv14BnlfH43M/FVh5UZ8dkCGlDKt4 0YsiuSG9n1hZmwTimYg0OmELNyL2kRfj8/4l2sHvpVJ/kCRpxUMrEebx09jIB/9keD/D MR56TPS4ouhaYmSjUGBlHWQB/mjYeEqjTJhtuI4iBVCxSbpzaAh6NoEl9vz8/QFhoauZ iMzTZ6SymuFZgrcDWGwjQckKAAVDGzJgsLCEDpD6aMZLQas0BGEnz+A12y4nb4GXje0Y HzXzCSPHCIfnP/eEK0LTCj5w8raQMpyyPHFhLrYzpHh1vnJ7ohD4UB6lIbQ1z6KHjMz9 qd0w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ZS+vJncVOAy2CYmx4D6/h8tcRsGlPjDDfC+97kQiBFE=; b=RqsNYGGQR1qYonud4PMDhekRXepuexqylzmCKUunzre9Kvu/FR5GTuHNoVBoj/s5Np aapBNAgkV0gQzCS9QsU7NJyLxHhmuerBlQVG0BdThr4GJBPaR0TAJHM62bJKdsWRu/Jy 2U9RyEW2507ywgF2o/3GOR7BQbFM1wdrMEsadyUx40h0BMfdPTisILJB2civ9I+7OXeB u1hrfc6ZI6rBpOFUhtfehdEBwGGo+KBeD1PxDpdEOCp+02iCz7NXLfVu9upbtrJ32jao olTykYqoEoHIl5+dbBJnLLIusPbepNNYYBpkwOd92JiA0UqBltn5r91U2+MBdfsHmL50 Xchw== X-Gm-Message-State: ACrzQf31Q2VItuMQ2vX73MpQPkLw4G6jYOnLcW0G6Zub/+tmqfvXdNIc rEX+UGrewxpFwn/bmnULNIX+ewmgVVc= X-Google-Smtp-Source: AMsMyM4X7umMFKK15t5sZfEjNhetMLrRtVw+QtOSk+qy5dvhmIrL4MlLzuQuROdH3Bex+qzgG8tHaQ== X-Received: by 2002:a05:6000:1e18:b0:235:e8f7:564f with SMTP id bj24-20020a0560001e1800b00235e8f7564fmr6453983wrb.153.1666372088188; Fri, 21 Oct 2022 10:08:08 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id k16-20020a5d6290000000b0022ae4f8395dsm18773752wru.96.2022.10.21.10.08.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 21 Oct 2022 10:08:07 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Fri, 21 Oct 2022 17:07:58 +0000 Subject: [PATCH v2 1/6] http: read HTTP WWW-Authenticate response headers Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Derrick Stolee , Lessley Dennington , Matthew John Cheetham , Matthew John Cheetham , Matthew John Cheetham Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Matthew John Cheetham From: Matthew John Cheetham Read and store the HTTP WWW-Authenticate response headers made for a particular request. This will allow us to pass important authentication challenge information to credential helpers or others that would otherwise have been lost. According to RFC2616 Section 4.2 [1], header field names are not case-sensitive meaning when collecting multiple values for the same field name, we can just use the case of the first observed instance of each field name and no normalisation is required. libcurl only provides us with the ability to read all headers recieved for a particular request, including any intermediate redirect requests or proxies. The lines returned by libcurl include HTTP status lines delinating any intermediate requests such as "HTTP/1.1 200". We use these lines to reset the strvec of WWW-Authenticate header values as we encounter them in order to only capture the final response headers. The collection of all header values matching the WWW-Authenticate header is complicated by the fact that it is legal for header fields to be continued over multiple lines, but libcurl only gives us one line at a time. In the future [2] we may be able to leverage functions to read headers from libcurl itself, but as of today we must do this ourselves. [1] https://datatracker.ietf.org/doc/html/rfc2616#section-4.2 [2] https://daniel.haxx.se/blog/2022/03/22/a-headers-api-for-libcurl/ Signed-off-by: Matthew John Cheetham --- credential.c | 1 + credential.h | 15 ++++++++++ http.c | 78 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 94 insertions(+) diff --git a/credential.c b/credential.c index f6389a50684..897b4679333 100644 --- a/credential.c +++ b/credential.c @@ -22,6 +22,7 @@ void credential_clear(struct credential *c) free(c->username); free(c->password); string_list_clear(&c->helpers, 0); + strvec_clear(&c->wwwauth_headers); credential_init(c); } diff --git a/credential.h b/credential.h index f430e77fea4..6f2e5bc610b 100644 --- a/credential.h +++ b/credential.h @@ -2,6 +2,7 @@ #define CREDENTIAL_H #include "string-list.h" +#include "strvec.h" /** * The credentials API provides an abstracted way of gathering username and @@ -115,6 +116,19 @@ struct credential { */ struct string_list helpers; + /** + * A `strvec` of WWW-Authenticate header values. Each string + * is the value of a WWW-Authenticate header in an HTTP response, + * in the order they were received in the response. + */ + struct strvec wwwauth_headers; + + /** + * Internal use only. Used to keep track of split header fields + * in order to fold multiple lines into one value. + */ + unsigned header_is_last_match:1; + unsigned approved:1, configured:1, quit:1, @@ -130,6 +144,7 @@ struct credential { #define CREDENTIAL_INIT { \ .helpers = STRING_LIST_INIT_DUP, \ + .wwwauth_headers = STRVEC_INIT, \ } /* Initialize a credential structure, setting all fields to empty. */ diff --git a/http.c b/http.c index 5d0502f51fd..03d43d352e7 100644 --- a/http.c +++ b/http.c @@ -183,6 +183,82 @@ size_t fwrite_buffer(char *ptr, size_t eltsize, size_t nmemb, void *buffer_) return nmemb; } +static size_t fwrite_wwwauth(char *ptr, size_t eltsize, size_t nmemb, void *p) +{ + size_t size = eltsize * nmemb; + struct strvec *values = &http_auth.wwwauth_headers; + struct strbuf buf = STRBUF_INIT; + const char *val; + const char *z = NULL; + + /* + * Header lines may not come NULL-terminated from libcurl so we must + * limit all scans to the maximum length of the header line, or leverage + * strbufs for all operations. + * + * In addition, it is possible that header values can be split over + * multiple lines as per RFC 2616 (even though this has since been + * deprecated in RFC 7230). A continuation header field value is + * identified as starting with a space or horizontal tab. + * + * The formal definition of a header field as given in RFC 2616 is: + * + * message-header = field-name ":" [ field-value ] + * field-name = token + * field-value = *( field-content | LWS ) + * field-content = + */ + + strbuf_add(&buf, ptr, size); + + /* Strip the CRLF that should be present at the end of each field */ + strbuf_trim_trailing_newline(&buf); + + /* Start of a new WWW-Authenticate header */ + if (skip_iprefix(buf.buf, "www-authenticate:", &val)) { + while (isspace(*val)) + val++; + + strvec_push(values, val); + http_auth.header_is_last_match = 1; + goto exit; + } + + /* + * This line could be a continuation of the previously matched header + * field. If this is the case then we should append this value to the + * end of the previously consumed value. + */ + if (http_auth.header_is_last_match && isspace(*buf.buf)) { + const char **v = values->v + values->nr - 1; + char *append = xstrfmt("%s%.*s", *v, (int)(size - 1), ptr + 1); + + free((void*)*v); + *v = append; + + goto exit; + } + + /* This is the start of a new header we don't care about */ + http_auth.header_is_last_match = 0; + + /* + * If this is a HTTP status line and not a header field, this signals + * a different HTTP response. libcurl writes all the output of all + * response headers of all responses, including redirects. + * We only care about the last HTTP request response's headers so clear + * the existing array. + */ + if (skip_iprefix(buf.buf, "http/", &z)) + strvec_clear(values); + +exit: + strbuf_release(&buf); + return size; +} + size_t fwrite_null(char *ptr, size_t eltsize, size_t nmemb, void *strbuf) { return nmemb; @@ -1829,6 +1905,8 @@ static int http_request(const char *url, fwrite_buffer); } + curl_easy_setopt(slot->curl, CURLOPT_HEADERFUNCTION, fwrite_wwwauth); + accept_language = http_get_accept_language_header(); if (accept_language) From patchwork Fri Oct 21 17:07:59 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew John Cheetham X-Patchwork-Id: 13015174 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C64E4FA373D for ; Fri, 21 Oct 2022 17:08:17 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230198AbiJURIQ (ORCPT ); Fri, 21 Oct 2022 13:08:16 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49160 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230142AbiJURIN (ORCPT ); Fri, 21 Oct 2022 13:08:13 -0400 Received: from mail-wr1-x435.google.com (mail-wr1-x435.google.com [IPv6:2a00:1450:4864:20::435]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4A93027D4C6 for ; Fri, 21 Oct 2022 10:08:12 -0700 (PDT) Received: by mail-wr1-x435.google.com with SMTP id bk15so5898365wrb.13 for ; Fri, 21 Oct 2022 10:08:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:from:to:cc:subject:date :message-id:reply-to; bh=AkxtgP/J3nLS9pfMEEWqqmQ9TtoIhKkytBuFPBUtbV4=; b=oSQUUmZuOm/zxi9UZBaUkKxP+T7NXFka+uNlPWz8o/EKaGVbZIO0k63Kumcyplgir3 GZLU83AXk2uJnbEdNT+koFD/FC7Toq9by7ChjCLiwRc7uX1Ub2fVaDu8ldaC5vWCIwEf kfw7Xv2fYp3ikXwf//vqwBEEAT9vstbOsqGLVpGicR5WP6zOf9lIONmOj/xRYg/IaTqx YwlwWFjKtlts590G/YcwGjqoSjIWpnd/R5jjFziRioQfk8tEZhrselNA4Zzw2Rw/Zpgs 9FrcBKI20TTnRf7llnbcTS6Wjvh4FnLm7yBlD4MNFe+XpWB03CXYyfLeZzX1aOu+BVkE fQNA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=AkxtgP/J3nLS9pfMEEWqqmQ9TtoIhKkytBuFPBUtbV4=; b=vjOf4pBvIPHv+uXbjguHEYMLk5cNxlnIDhEXiy3mVyKsPcS9ApklYeeYhaBRWQqfPf HryvGgCK6E5wEZhvUpvubUOwlCbHjydRKIqQr3CwULs7WCTnJrILinBWc2TfnX/V9VBM oH3eEEu+HUd0YXWeG9N/s4TZ2U0jub1IGuIft2FA+XBWTApPzS3N1Gvn0tZHjc3TW5w/ GyWs4by17s3DqDgdjihV08fMAdxO0Aas3k72RWUZFZNgy41WCmjmdkg0Xipo5c9Il3r3 cQQhnWU0V0H+nfGZ3LGBCMOvBhD+qoGSzNqQCweNrjE056cgnXcTuKchLNMmlhsYiCa8 4vxA== X-Gm-Message-State: ACrzQf0IXBCRT81MT5I18ttXKCBZmDk4/7ALoEKiBsHsazSz/uKoiSit CgjH0VvncnBjSkazgD4uj0M6Oh9TAtw= X-Google-Smtp-Source: AMsMyM7ebDDJDTbx25ZOI3TGB2NSIBSpapnBzRqI4C16S4xGpCfQubeL4qK0RZuUUfnioYzUOwpP7w== X-Received: by 2002:adf:e7c9:0:b0:22e:3524:9b4b with SMTP id e9-20020adfe7c9000000b0022e35249b4bmr12586678wrn.520.1666372090490; Fri, 21 Oct 2022 10:08:10 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id n9-20020a05600c3b8900b003b4ff30e566sm10134392wms.3.2022.10.21.10.08.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 21 Oct 2022 10:08:09 -0700 (PDT) Message-Id: <0838d992744a4b06523be6df0edb046ebba033ee.1666372083.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Fri, 21 Oct 2022 17:07:59 +0000 Subject: [PATCH v2 2/6] credential: add WWW-Authenticate header to cred requests Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Derrick Stolee , Lessley Dennington , Matthew John Cheetham , Matthew John Cheetham , Matthew John Cheetham Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Matthew John Cheetham From: Matthew John Cheetham Add the value of the WWW-Authenticate response header to credential requests. Credential helpers that understand and support HTTP authentication and authorization can use this standard header (RFC 2616 Section 14.47 [1]) to generate valid credentials. WWW-Authenticate headers can contain information pertaining to the authority, authentication mechanism, or extra parameters/scopes that are required. The current I/O format for credential helpers only allows for unique names for properties/attributes, so in order to transmit multiple header values (with a specific order) we introduce a new convention whereby a C-style array syntax is used in the property name to denote multiple ordered values for the same property. In this case we send multiple `wwwauth[n]` properties where `n` is a zero-indexed number, reflecting the order the WWW-Authenticate headers appeared in the HTTP response. [1] https://datatracker.ietf.org/doc/html/rfc2616#section-14.47 Signed-off-by: Matthew John Cheetham --- Documentation/git-credential.txt | 9 +++++++++ credential.c | 12 ++++++++++++ 2 files changed, 21 insertions(+) diff --git a/Documentation/git-credential.txt b/Documentation/git-credential.txt index f18673017f5..0ff3cbc25b9 100644 --- a/Documentation/git-credential.txt +++ b/Documentation/git-credential.txt @@ -160,6 +160,15 @@ empty string. Components which are missing from the URL (e.g., there is no username in the example above) will be left unset. +`wwwauth[]`:: + + When an HTTP response is received that includes one or more + 'WWW-Authenticate' authentication headers, these can be passed to Git + (and subsequent credential helpers) with these attributes. + Each 'WWW-Authenticate' header value should be passed as a separate + attribute 'wwwauth[]' where the order of the attributes is the same + as they appear in the HTTP response. + GIT --- Part of the linkgit:git[1] suite diff --git a/credential.c b/credential.c index 897b4679333..8a3ad6c0ae2 100644 --- a/credential.c +++ b/credential.c @@ -263,6 +263,17 @@ static void credential_write_item(FILE *fp, const char *key, const char *value, fprintf(fp, "%s=%s\n", key, value); } +static void credential_write_strvec(FILE *fp, const char *key, + const struct strvec *vec) +{ + int i = 0; + const char *full_key = xstrfmt("%s[]", key); + for (; i < vec->nr; i++) { + credential_write_item(fp, full_key, vec->v[i], 0); + } + free((void*)full_key); +} + void credential_write(const struct credential *c, FILE *fp) { credential_write_item(fp, "protocol", c->protocol, 1); @@ -270,6 +281,7 @@ void credential_write(const struct credential *c, FILE *fp) credential_write_item(fp, "path", c->path, 0); credential_write_item(fp, "username", c->username, 0); credential_write_item(fp, "password", c->password, 0); + credential_write_strvec(fp, "wwwauth", &c->wwwauth_headers); } static int run_credential_helper(struct credential *c, From patchwork Fri Oct 21 17:08:00 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew John Cheetham X-Patchwork-Id: 13015175 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 505BFC433FE for ; Fri, 21 Oct 2022 17:08:29 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230272AbiJURI1 (ORCPT ); Fri, 21 Oct 2022 13:08:27 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49190 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230176AbiJURIO (ORCPT ); Fri, 21 Oct 2022 13:08:14 -0400 Received: from mail-wr1-x42c.google.com (mail-wr1-x42c.google.com [IPv6:2a00:1450:4864:20::42c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AD1C027FEB6 for ; Fri, 21 Oct 2022 10:08:12 -0700 (PDT) Received: by mail-wr1-x42c.google.com with SMTP id n12so5903206wrp.10 for ; Fri, 21 Oct 2022 10:08:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:from:to:cc:subject:date :message-id:reply-to; bh=R/YUvbaJBrsIHAs71DEDewu6Xn5aAU6IxEv4wPX9Dws=; b=ll7E3EcyAjVAarA+Q9ELvnFJgjAR6/WBpWS1tTps8BuCy2alXbaGkp2eNr/OQg+kRZ 3uPZrRQ/yjoPg6vW6AJ35e9Q9H0+0QdwWsmFlXEE2htcJQVLww8kCN28T38LV+QdLKrV 1ewA+vFOj8oLBkopNHDXioErz0Mix2fK+6z+D79lpCkBwvN7bkX0qNcMEDtA8AbcW8ll 33CO+PGXPmqTuUVqXf/Fj8vIxE4SICOpqKunCP7p51cJAlD9O7cpYcciuoRWFKdAPOww WcLyBFAkOKDdSEKS4NLz4D/xOioIcgQGrAP/MMqxCkMgx6FlEtlQQoR/LSaTfEuDQ3lf Rgew== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=R/YUvbaJBrsIHAs71DEDewu6Xn5aAU6IxEv4wPX9Dws=; b=3dwlOIuU6+0XJwepYb46tX+1FomE5Ign1DYhiV3nPQsNOHmtpC38ch8NKpD1XTSKeg wtMiWzBKic174gbinsP2DkFIiIUkkpPPHGlu2WBuzxVylOy7dCl9eY/tjn/25USeWVKF NvaykLlaLpDjG+HSJHMs4DLSI3jcJldvft8lsLhl4+VIHpOmNxLcPxkMUg35RdEXboNw Wn8072bNbxSNV0ZYJ+DLF2vJtWuA2r6qM3k24F5U4E3x6We0rk+yHzMPQy1eUHHIdweQ 33jumf9VMVfBF4VnRP8vDcNlJZNsJMzVBRrK5o9nWBJnVfUm9bbvLL4QbKtHLM0xDx94 Y/wQ== X-Gm-Message-State: ACrzQf24WHbPkN9OvHfowGhItuylXmicH4sMTASJQOU4z5gAYFj9w4T8 E5ApSa/78Rz20sciaVj4dZsJUOxPHtg= X-Google-Smtp-Source: AMsMyM5RM8EglBsTayOGcLSp5k4wxHuX/Ohw92Obi7ZsOb573MNrEU9Mr5KzCRlm1/Z7zsq7HfSOOg== X-Received: by 2002:a05:6000:1051:b0:22e:5d8a:d3d5 with SMTP id c17-20020a056000105100b0022e5d8ad3d5mr12945055wrx.476.1666372091735; Fri, 21 Oct 2022 10:08:11 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id t11-20020a05600c41cb00b003b47ff307e1sm2790856wmh.31.2022.10.21.10.08.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 21 Oct 2022 10:08:11 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Fri, 21 Oct 2022 17:08:00 +0000 Subject: [PATCH v2 3/6] http: store all request headers on active_request_slot Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Derrick Stolee , Lessley Dennington , Matthew John Cheetham , Matthew John Cheetham , Matthew John Cheetham Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Matthew John Cheetham From: Matthew John Cheetham Once a list of headers has been set on the curl handle, it is not possible to recover that `struct curl_slist` instance to add or modify headers. In future commits we will want to modify the set of request headers in response to an authentication challenge/401 response from the server, with information provided by a credential helper. There are a number of different places where curl is used for an HTTP request, and they do not have a common handling of request headers. However, given that they all do call the `start_active_slot()` function, either directly or indirectly via `run_slot()` or `run_one_slot()`, we use this as the point to set the `CURLOPT_HTTPHEADER` option just before the request is made. We collect all request headers in a `struct curl_slist` on the `struct active_request_slot` that is obtained from a call to `get_active_slot(int)`. This function now takes a single argument to define if the initial set of headers on the slot should include the "Pragma: no-cache" header, along with all extra headers specified via `http.extraHeader` config values. The active request slot obtained from `get_active_slot(int)` will always contain a fresh set of default headers and any headers set in previous usages of this slot will be freed. Signed-off-by: Matthew John Cheetham --- http-push.c | 103 ++++++++++++++++++++++---------------------------- http-walker.c | 2 +- http.c | 82 ++++++++++++++++++---------------------- http.h | 4 +- remote-curl.c | 36 +++++++++--------- 5 files changed, 101 insertions(+), 126 deletions(-) diff --git a/http-push.c b/http-push.c index 5f4340a36e6..2b40959b376 100644 --- a/http-push.c +++ b/http-push.c @@ -211,29 +211,29 @@ static void curl_setup_http(CURL *curl, const char *url, curl_easy_setopt(curl, CURLOPT_UPLOAD, 1); } -static struct curl_slist *get_dav_token_headers(struct remote_lock *lock, enum dav_header_flag options) +static struct curl_slist *append_dav_token_headers(struct curl_slist *headers, + struct remote_lock *lock, enum dav_header_flag options) { struct strbuf buf = STRBUF_INIT; - struct curl_slist *dav_headers = http_copy_default_headers(); if (options & DAV_HEADER_IF) { strbuf_addf(&buf, "If: (<%s>)", lock->token); - dav_headers = curl_slist_append(dav_headers, buf.buf); + headers = curl_slist_append(headers, buf.buf); strbuf_reset(&buf); } if (options & DAV_HEADER_LOCK) { strbuf_addf(&buf, "Lock-Token: <%s>", lock->token); - dav_headers = curl_slist_append(dav_headers, buf.buf); + headers = curl_slist_append(headers, buf.buf); strbuf_reset(&buf); } if (options & DAV_HEADER_TIMEOUT) { strbuf_addf(&buf, "Timeout: Second-%ld", lock->timeout); - dav_headers = curl_slist_append(dav_headers, buf.buf); + headers = curl_slist_append(headers, buf.buf); strbuf_reset(&buf); } strbuf_release(&buf); - return dav_headers; + return headers; } static void finish_request(struct transfer_request *request); @@ -281,7 +281,7 @@ static void start_mkcol(struct transfer_request *request) request->url = get_remote_object_url(repo->url, hex, 1); - slot = get_active_slot(); + slot = get_active_slot(0); slot->callback_func = process_response; slot->callback_data = request; curl_setup_http_get(slot->curl, request->url, DAV_MKCOL); @@ -399,7 +399,7 @@ static void start_put(struct transfer_request *request) strbuf_add(&buf, request->lock->tmpfile_suffix, the_hash_algo->hexsz + 1); request->url = strbuf_detach(&buf, NULL); - slot = get_active_slot(); + slot = get_active_slot(0); slot->callback_func = process_response; slot->callback_data = request; curl_setup_http(slot->curl, request->url, DAV_PUT, @@ -417,15 +417,13 @@ static void start_put(struct transfer_request *request) static void start_move(struct transfer_request *request) { struct active_request_slot *slot; - struct curl_slist *dav_headers = http_copy_default_headers(); - slot = get_active_slot(); + slot = get_active_slot(0); slot->callback_func = process_response; slot->callback_data = request; curl_setup_http_get(slot->curl, request->url, DAV_MOVE); - dav_headers = curl_slist_append(dav_headers, request->dest); - dav_headers = curl_slist_append(dav_headers, "Overwrite: T"); - curl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, dav_headers); + slot->headers = curl_slist_append(slot->headers, request->dest); + slot->headers = curl_slist_append(slot->headers, "Overwrite: T"); if (start_active_slot(slot)) { request->slot = slot; @@ -440,17 +438,16 @@ static int refresh_lock(struct remote_lock *lock) { struct active_request_slot *slot; struct slot_results results; - struct curl_slist *dav_headers; int rc = 0; lock->refreshing = 1; - dav_headers = get_dav_token_headers(lock, DAV_HEADER_IF | DAV_HEADER_TIMEOUT); - - slot = get_active_slot(); + slot = get_active_slot(0); slot->results = &results; + slot->headers = append_dav_token_headers(slot->headers, lock, + DAV_HEADER_IF | DAV_HEADER_TIMEOUT); + curl_setup_http_get(slot->curl, lock->url, DAV_LOCK); - curl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, dav_headers); if (start_active_slot(slot)) { run_active_slot(slot); @@ -464,7 +461,6 @@ static int refresh_lock(struct remote_lock *lock) } lock->refreshing = 0; - curl_slist_free_all(dav_headers); return rc; } @@ -838,7 +834,6 @@ static struct remote_lock *lock_remote(const char *path, long timeout) char *ep; char timeout_header[25]; struct remote_lock *lock = NULL; - struct curl_slist *dav_headers = http_copy_default_headers(); struct xml_ctx ctx; char *escaped; @@ -849,7 +844,7 @@ static struct remote_lock *lock_remote(const char *path, long timeout) while (ep) { char saved_character = ep[1]; ep[1] = '\0'; - slot = get_active_slot(); + slot = get_active_slot(0); slot->results = &results; curl_setup_http_get(slot->curl, url, DAV_MKCOL); if (start_active_slot(slot)) { @@ -875,14 +870,15 @@ static struct remote_lock *lock_remote(const char *path, long timeout) strbuf_addf(&out_buffer.buf, LOCK_REQUEST, escaped); free(escaped); + slot = get_active_slot(0); + slot->results = &results; + xsnprintf(timeout_header, sizeof(timeout_header), "Timeout: Second-%ld", timeout); - dav_headers = curl_slist_append(dav_headers, timeout_header); - dav_headers = curl_slist_append(dav_headers, "Content-Type: text/xml"); + slot->headers = curl_slist_append(slot->headers, timeout_header); + slot->headers = curl_slist_append(slot->headers, + "Content-Type: text/xml"); - slot = get_active_slot(); - slot->results = &results; curl_setup_http(slot->curl, url, DAV_LOCK, &out_buffer, fwrite_buffer); - curl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, dav_headers); curl_easy_setopt(slot->curl, CURLOPT_WRITEDATA, &in_buffer); CALLOC_ARRAY(lock, 1); @@ -921,7 +917,6 @@ static struct remote_lock *lock_remote(const char *path, long timeout) fprintf(stderr, "Unable to start LOCK request\n"); } - curl_slist_free_all(dav_headers); strbuf_release(&out_buffer.buf); strbuf_release(&in_buffer); @@ -945,15 +940,14 @@ static int unlock_remote(struct remote_lock *lock) struct active_request_slot *slot; struct slot_results results; struct remote_lock *prev = repo->locks; - struct curl_slist *dav_headers; int rc = 0; - dav_headers = get_dav_token_headers(lock, DAV_HEADER_LOCK); - - slot = get_active_slot(); + slot = get_active_slot(0); slot->results = &results; + slot->headers = append_dav_token_headers(slot->headers, lock, + DAV_HEADER_LOCK); + curl_setup_http_get(slot->curl, lock->url, DAV_UNLOCK); - curl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, dav_headers); if (start_active_slot(slot)) { run_active_slot(slot); @@ -966,8 +960,6 @@ static int unlock_remote(struct remote_lock *lock) fprintf(stderr, "Unable to start UNLOCK request\n"); } - curl_slist_free_all(dav_headers); - if (repo->locks == lock) { repo->locks = lock->next; } else { @@ -1121,7 +1113,6 @@ static void remote_ls(const char *path, int flags, struct slot_results results; struct strbuf in_buffer = STRBUF_INIT; struct buffer out_buffer = { STRBUF_INIT, 0 }; - struct curl_slist *dav_headers = http_copy_default_headers(); struct xml_ctx ctx; struct remote_ls_ctx ls; @@ -1134,14 +1125,14 @@ static void remote_ls(const char *path, int flags, strbuf_addstr(&out_buffer.buf, PROPFIND_ALL_REQUEST); - dav_headers = curl_slist_append(dav_headers, "Depth: 1"); - dav_headers = curl_slist_append(dav_headers, "Content-Type: text/xml"); - - slot = get_active_slot(); + slot = get_active_slot(0); slot->results = &results; + slot->headers = curl_slist_append(slot->headers, "Depth: 1"); + slot->headers = curl_slist_append(slot->headers, + "Content-Type: text/xml"); + curl_setup_http(slot->curl, url, DAV_PROPFIND, &out_buffer, fwrite_buffer); - curl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, dav_headers); curl_easy_setopt(slot->curl, CURLOPT_WRITEDATA, &in_buffer); if (start_active_slot(slot)) { @@ -1177,7 +1168,6 @@ static void remote_ls(const char *path, int flags, free(url); strbuf_release(&out_buffer.buf); strbuf_release(&in_buffer); - curl_slist_free_all(dav_headers); } static void get_remote_object_list(unsigned char parent) @@ -1199,7 +1189,6 @@ static int locking_available(void) struct slot_results results; struct strbuf in_buffer = STRBUF_INIT; struct buffer out_buffer = { STRBUF_INIT, 0 }; - struct curl_slist *dav_headers = http_copy_default_headers(); struct xml_ctx ctx; int lock_flags = 0; char *escaped; @@ -1208,14 +1197,14 @@ static int locking_available(void) strbuf_addf(&out_buffer.buf, PROPFIND_SUPPORTEDLOCK_REQUEST, escaped); free(escaped); - dav_headers = curl_slist_append(dav_headers, "Depth: 0"); - dav_headers = curl_slist_append(dav_headers, "Content-Type: text/xml"); - - slot = get_active_slot(); + slot = get_active_slot(0); slot->results = &results; + slot->headers = curl_slist_append(slot->headers, "Depth: 0"); + slot->headers = curl_slist_append(slot->headers, + "Content-Type: text/xml"); + curl_setup_http(slot->curl, repo->url, DAV_PROPFIND, &out_buffer, fwrite_buffer); - curl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, dav_headers); curl_easy_setopt(slot->curl, CURLOPT_WRITEDATA, &in_buffer); if (start_active_slot(slot)) { @@ -1257,7 +1246,6 @@ static int locking_available(void) strbuf_release(&out_buffer.buf); strbuf_release(&in_buffer); - curl_slist_free_all(dav_headers); return lock_flags; } @@ -1374,17 +1362,16 @@ static int update_remote(const struct object_id *oid, struct remote_lock *lock) struct active_request_slot *slot; struct slot_results results; struct buffer out_buffer = { STRBUF_INIT, 0 }; - struct curl_slist *dav_headers; - - dav_headers = get_dav_token_headers(lock, DAV_HEADER_IF); strbuf_addf(&out_buffer.buf, "%s\n", oid_to_hex(oid)); - slot = get_active_slot(); + slot = get_active_slot(0); slot->results = &results; + slot->headers = append_dav_token_headers(slot->headers, lock, + DAV_HEADER_IF); + curl_setup_http(slot->curl, lock->url, DAV_PUT, &out_buffer, fwrite_null); - curl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, dav_headers); if (start_active_slot(slot)) { run_active_slot(slot); @@ -1486,18 +1473,18 @@ static void update_remote_info_refs(struct remote_lock *lock) struct buffer buffer = { STRBUF_INIT, 0 }; struct active_request_slot *slot; struct slot_results results; - struct curl_slist *dav_headers; remote_ls("refs/", (PROCESS_FILES | RECURSIVE), add_remote_info_ref, &buffer.buf); if (!aborted) { - dav_headers = get_dav_token_headers(lock, DAV_HEADER_IF); - slot = get_active_slot(); + slot = get_active_slot(0); slot->results = &results; + slot->headers = append_dav_token_headers(slot->headers, lock, + DAV_HEADER_IF); + curl_setup_http(slot->curl, lock->url, DAV_PUT, &buffer, fwrite_null); - curl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, dav_headers); if (start_active_slot(slot)) { run_active_slot(slot); @@ -1652,7 +1639,7 @@ static int delete_remote_branch(const char *pattern, int force) if (dry_run) return 0; url = xstrfmt("%s%s", repo->url, remote_ref->name); - slot = get_active_slot(); + slot = get_active_slot(0); slot->results = &results; curl_setup_http_get(slot->curl, url, DAV_DELETE); if (start_active_slot(slot)) { diff --git a/http-walker.c b/http-walker.c index b8f0f98ae14..8747de2fcdb 100644 --- a/http-walker.c +++ b/http-walker.c @@ -373,7 +373,7 @@ static void fetch_alternates(struct walker *walker, const char *base) * Use a callback to process the result, since another request * may fail and need to have alternates loaded before continuing */ - slot = get_active_slot(); + slot = get_active_slot(0); slot->callback_func = process_alternates_response; alt_req.walker = walker; slot->callback_data = &alt_req; diff --git a/http.c b/http.c index 03d43d352e7..f2ebb17c8c4 100644 --- a/http.c +++ b/http.c @@ -124,8 +124,6 @@ static unsigned long empty_auth_useless = | CURLAUTH_DIGEST_IE | CURLAUTH_DIGEST; -static struct curl_slist *pragma_header; -static struct curl_slist *no_pragma_header; static struct string_list extra_http_headers = STRING_LIST_INIT_DUP; static struct curl_slist *host_resolutions; @@ -1133,11 +1131,6 @@ void http_init(struct remote *remote, const char *url, int proactive_auth) if (remote) var_override(&http_proxy_authmethod, remote->http_proxy_authmethod); - pragma_header = curl_slist_append(http_copy_default_headers(), - "Pragma: no-cache"); - no_pragma_header = curl_slist_append(http_copy_default_headers(), - "Pragma:"); - { char *http_max_requests = getenv("GIT_HTTP_MAX_REQUESTS"); if (http_max_requests) @@ -1199,6 +1192,8 @@ void http_cleanup(void) while (slot != NULL) { struct active_request_slot *next = slot->next; + if (slot->headers) + curl_slist_free_all(slot->headers); if (slot->curl) { xmulti_remove_handle(slot); curl_easy_cleanup(slot->curl); @@ -1215,12 +1210,6 @@ void http_cleanup(void) string_list_clear(&extra_http_headers, 0); - curl_slist_free_all(pragma_header); - pragma_header = NULL; - - curl_slist_free_all(no_pragma_header); - no_pragma_header = NULL; - curl_slist_free_all(host_resolutions); host_resolutions = NULL; @@ -1255,7 +1244,18 @@ void http_cleanup(void) FREE_AND_NULL(cached_accept_language); } -struct active_request_slot *get_active_slot(void) +static struct curl_slist *http_copy_default_headers(void) +{ + struct curl_slist *headers = NULL; + const struct string_list_item *item; + + for_each_string_list_item(item, &extra_http_headers) + headers = curl_slist_append(headers, item->string); + + return headers; +} + +struct active_request_slot *get_active_slot(int no_pragma_header) { struct active_request_slot *slot = active_queue_head; struct active_request_slot *newslot; @@ -1277,6 +1277,7 @@ struct active_request_slot *get_active_slot(void) newslot->curl = NULL; newslot->in_use = 0; newslot->next = NULL; + newslot->headers = NULL; slot = active_queue_head; if (!slot) { @@ -1294,6 +1295,15 @@ struct active_request_slot *get_active_slot(void) curl_session_count++; } + if (slot->headers) + curl_slist_free_all(slot->headers); + + slot->headers = http_copy_default_headers(); + + if (!no_pragma_header) + slot->headers = curl_slist_append(slot->headers, + "Pragma: no-cache"); + active_requests++; slot->in_use = 1; slot->results = NULL; @@ -1303,7 +1313,6 @@ struct active_request_slot *get_active_slot(void) curl_easy_setopt(slot->curl, CURLOPT_COOKIEFILE, curl_cookie_file); if (curl_save_cookies) curl_easy_setopt(slot->curl, CURLOPT_COOKIEJAR, curl_cookie_file); - curl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, pragma_header); curl_easy_setopt(slot->curl, CURLOPT_RESOLVE, host_resolutions); curl_easy_setopt(slot->curl, CURLOPT_ERRORBUFFER, curl_errorstr); curl_easy_setopt(slot->curl, CURLOPT_CUSTOMREQUEST, NULL); @@ -1335,9 +1344,12 @@ struct active_request_slot *get_active_slot(void) int start_active_slot(struct active_request_slot *slot) { - CURLMcode curlm_result = curl_multi_add_handle(curlm, slot->curl); + CURLMcode curlm_result; int num_transfers; + curl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, slot->headers); + curlm_result = curl_multi_add_handle(curlm, slot->curl); + if (curlm_result != CURLM_OK && curlm_result != CURLM_CALL_MULTI_PERFORM) { warning("curl_multi_add_handle failed: %s", @@ -1652,17 +1664,6 @@ int run_one_slot(struct active_request_slot *slot, return handle_curl_result(results); } -struct curl_slist *http_copy_default_headers(void) -{ - struct curl_slist *headers = NULL; - const struct string_list_item *item; - - for_each_string_list_item(item, &extra_http_headers) - headers = curl_slist_append(headers, item->string); - - return headers; -} - static CURLcode curlinfo_strbuf(CURL *curl, CURLINFO info, struct strbuf *buf) { char *ptr; @@ -1880,12 +1881,11 @@ static int http_request(const char *url, { struct active_request_slot *slot; struct slot_results results; - struct curl_slist *headers = http_copy_default_headers(); - struct strbuf buf = STRBUF_INIT; + int no_cache = options && options->no_cache; const char *accept_language; int ret; - slot = get_active_slot(); + slot = get_active_slot(!no_cache); curl_easy_setopt(slot->curl, CURLOPT_HTTPGET, 1); if (!result) { @@ -1910,27 +1910,23 @@ static int http_request(const char *url, accept_language = http_get_accept_language_header(); if (accept_language) - headers = curl_slist_append(headers, accept_language); + slot->headers = curl_slist_append(slot->headers, + accept_language); - strbuf_addstr(&buf, "Pragma:"); - if (options && options->no_cache) - strbuf_addstr(&buf, " no-cache"); if (options && options->initial_request && http_follow_config == HTTP_FOLLOW_INITIAL) curl_easy_setopt(slot->curl, CURLOPT_FOLLOWLOCATION, 1); - headers = curl_slist_append(headers, buf.buf); - /* Add additional headers here */ if (options && options->extra_headers) { const struct string_list_item *item; for_each_string_list_item(item, options->extra_headers) { - headers = curl_slist_append(headers, item->string); + slot->headers = curl_slist_append(slot->headers, + item->string); } } curl_easy_setopt(slot->curl, CURLOPT_URL, url); - curl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, headers); curl_easy_setopt(slot->curl, CURLOPT_ENCODING, ""); curl_easy_setopt(slot->curl, CURLOPT_FAILONERROR, 0); @@ -1948,9 +1944,6 @@ static int http_request(const char *url, curlinfo_strbuf(slot->curl, CURLINFO_EFFECTIVE_URL, options->effective_url); - curl_slist_free_all(headers); - strbuf_release(&buf); - return ret; } @@ -2311,12 +2304,10 @@ struct http_pack_request *new_direct_http_pack_request( goto abort; } - preq->slot = get_active_slot(); + preq->slot = get_active_slot(1); curl_easy_setopt(preq->slot->curl, CURLOPT_WRITEDATA, preq->packfile); curl_easy_setopt(preq->slot->curl, CURLOPT_WRITEFUNCTION, fwrite); curl_easy_setopt(preq->slot->curl, CURLOPT_URL, preq->url); - curl_easy_setopt(preq->slot->curl, CURLOPT_HTTPHEADER, - no_pragma_header); /* * If there is data present from a previous transfer attempt, @@ -2481,14 +2472,13 @@ struct http_object_request *new_http_object_request(const char *base_url, } } - freq->slot = get_active_slot(); + freq->slot = get_active_slot(1); curl_easy_setopt(freq->slot->curl, CURLOPT_WRITEDATA, freq); curl_easy_setopt(freq->slot->curl, CURLOPT_FAILONERROR, 0); curl_easy_setopt(freq->slot->curl, CURLOPT_WRITEFUNCTION, fwrite_sha1_file); curl_easy_setopt(freq->slot->curl, CURLOPT_ERRORBUFFER, freq->errorstr); curl_easy_setopt(freq->slot->curl, CURLOPT_URL, freq->url); - curl_easy_setopt(freq->slot->curl, CURLOPT_HTTPHEADER, no_pragma_header); /* * If we have successfully processed data from a previous fetch diff --git a/http.h b/http.h index 3c94c479100..a304cc408b2 100644 --- a/http.h +++ b/http.h @@ -22,6 +22,7 @@ struct slot_results { struct active_request_slot { CURL *curl; int in_use; + struct curl_slist *headers; CURLcode curl_result; long http_code; int *finished; @@ -43,7 +44,7 @@ size_t fwrite_null(char *ptr, size_t eltsize, size_t nmemb, void *strbuf); curlioerr ioctl_buffer(CURL *handle, int cmd, void *clientp); /* Slot lifecycle functions */ -struct active_request_slot *get_active_slot(void); +struct active_request_slot *get_active_slot(int no_pragma_header); int start_active_slot(struct active_request_slot *slot); void run_active_slot(struct active_request_slot *slot); void finish_all_active_slots(void); @@ -64,7 +65,6 @@ void step_active_slots(void); void http_init(struct remote *remote, const char *url, int proactive_auth); void http_cleanup(void); -struct curl_slist *http_copy_default_headers(void); extern long int git_curl_ipresolve; extern int active_requests; diff --git a/remote-curl.c b/remote-curl.c index 72dfb8fb86a..edbd4504beb 100644 --- a/remote-curl.c +++ b/remote-curl.c @@ -847,14 +847,13 @@ static int run_slot(struct active_request_slot *slot, static int probe_rpc(struct rpc_state *rpc, struct slot_results *results) { struct active_request_slot *slot; - struct curl_slist *headers = http_copy_default_headers(); struct strbuf buf = STRBUF_INIT; int err; - slot = get_active_slot(); + slot = get_active_slot(0); - headers = curl_slist_append(headers, rpc->hdr_content_type); - headers = curl_slist_append(headers, rpc->hdr_accept); + slot->headers = curl_slist_append(slot->headers, rpc->hdr_content_type); + slot->headers = curl_slist_append(slot->headers, rpc->hdr_accept); curl_easy_setopt(slot->curl, CURLOPT_NOBODY, 0); curl_easy_setopt(slot->curl, CURLOPT_POST, 1); @@ -862,13 +861,11 @@ static int probe_rpc(struct rpc_state *rpc, struct slot_results *results) curl_easy_setopt(slot->curl, CURLOPT_ENCODING, NULL); curl_easy_setopt(slot->curl, CURLOPT_POSTFIELDS, "0000"); curl_easy_setopt(slot->curl, CURLOPT_POSTFIELDSIZE, 4); - curl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, headers); curl_easy_setopt(slot->curl, CURLOPT_WRITEFUNCTION, fwrite_buffer); curl_easy_setopt(slot->curl, CURLOPT_WRITEDATA, &buf); err = run_slot(slot, results); - curl_slist_free_all(headers); strbuf_release(&buf); return err; } @@ -888,7 +885,6 @@ static curl_off_t xcurl_off_t(size_t len) static int post_rpc(struct rpc_state *rpc, int stateless_connect, int flush_received) { struct active_request_slot *slot; - struct curl_slist *headers = http_copy_default_headers(); int use_gzip = rpc->gzip_request; char *gzip_body = NULL; size_t gzip_size = 0; @@ -930,21 +926,23 @@ static int post_rpc(struct rpc_state *rpc, int stateless_connect, int flush_rece needs_100_continue = 1; } - headers = curl_slist_append(headers, rpc->hdr_content_type); - headers = curl_slist_append(headers, rpc->hdr_accept); - headers = curl_slist_append(headers, needs_100_continue ? +retry: + slot = get_active_slot(0); + + slot->headers = curl_slist_append(slot->headers, rpc->hdr_content_type); + slot->headers = curl_slist_append(slot->headers, rpc->hdr_accept); + slot->headers = curl_slist_append(slot->headers, needs_100_continue ? "Expect: 100-continue" : "Expect:"); /* Add Accept-Language header */ if (rpc->hdr_accept_language) - headers = curl_slist_append(headers, rpc->hdr_accept_language); + slot->headers = curl_slist_append(slot->headers, + rpc->hdr_accept_language); /* Add the extra Git-Protocol header */ if (rpc->protocol_header) - headers = curl_slist_append(headers, rpc->protocol_header); - -retry: - slot = get_active_slot(); + slot->headers = curl_slist_append(slot->headers, + rpc->protocol_header); curl_easy_setopt(slot->curl, CURLOPT_NOBODY, 0); curl_easy_setopt(slot->curl, CURLOPT_POST, 1); @@ -955,7 +953,8 @@ retry: /* The request body is large and the size cannot be predicted. * We must use chunked encoding to send it. */ - headers = curl_slist_append(headers, "Transfer-Encoding: chunked"); + slot->headers = curl_slist_append(slot->headers, + "Transfer-Encoding: chunked"); rpc->initial_buffer = 1; curl_easy_setopt(slot->curl, CURLOPT_READFUNCTION, rpc_out); curl_easy_setopt(slot->curl, CURLOPT_INFILE, rpc); @@ -1002,7 +1001,8 @@ retry: gzip_size = stream.total_out; - headers = curl_slist_append(headers, "Content-Encoding: gzip"); + slot->headers = curl_slist_append(slot->headers, + "Content-Encoding: gzip"); curl_easy_setopt(slot->curl, CURLOPT_POSTFIELDS, gzip_body); curl_easy_setopt(slot->curl, CURLOPT_POSTFIELDSIZE_LARGE, xcurl_off_t(gzip_size)); @@ -1025,7 +1025,6 @@ retry: } } - curl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, headers); curl_easy_setopt(slot->curl, CURLOPT_WRITEFUNCTION, rpc_in); rpc_in_data.rpc = rpc; rpc_in_data.slot = slot; @@ -1055,7 +1054,6 @@ retry: if (stateless_connect) packet_response_end(rpc->in); - curl_slist_free_all(headers); free(gzip_body); return err; } From patchwork Fri Oct 21 17:08:01 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew John Cheetham X-Patchwork-Id: 13015176 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9E716C433FE for ; Fri, 21 Oct 2022 17:08:36 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230239AbiJURIe (ORCPT ); Fri, 21 Oct 2022 13:08:34 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49298 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230218AbiJURIS (ORCPT ); Fri, 21 Oct 2022 13:08:18 -0400 Received: from mail-wr1-x436.google.com (mail-wr1-x436.google.com [IPv6:2a00:1450:4864:20::436]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BCC1F285102 for ; Fri, 21 Oct 2022 10:08:14 -0700 (PDT) Received: by mail-wr1-x436.google.com with SMTP id v1so5918930wrt.11 for ; Fri, 21 Oct 2022 10:08:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:from:to:cc:subject:date :message-id:reply-to; bh=wTgFRH8NvEuDL6qrOKVv8nBrf31Xj7wIMu+UBji2a3E=; b=TXl84XvHBUhNXeXInNFq2P6Tqc5C1Mkjlym5CFcYFSNLCK+z3urrq3xPHfDb61osDG vsnTvw9C1nUltMKmjkNO9LgMRxHAPTVvKoTZPBUy209T34NUeFtusn2jyk7vpJ+qWXQI NALzdKNex06pV+ADSGyIhu27D1Tjs6ITnPgs/aX+QsCZ8z6XPWTIrHKbCaJBqwO3zODY +CFJyZijSYMScnwrZHHStLaRa3kJd2Q3NzfnxQmLhhWtOqD4wnLPji7pomT0XFBJI+5R 6ChmAS9mQFrVim8WCc/vU7WUJi+aDp/YfMlvlIzC4Xlu3Zfm6ZohIHaUeht/13mwMdRc jgOQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=wTgFRH8NvEuDL6qrOKVv8nBrf31Xj7wIMu+UBji2a3E=; b=yW6DKGeAH1tcfF/OwbHHMJEeSwnSG64XPqa6DOYlRIumGVp3qHvqcMjWsDp7Gsv9RN HsYia4UJgBYDDcIQNqn6hhZKbJd1g2gclJa+cV/5duBk9ofiJcW7wte9hrjGx/IJsiCb mnZ02NE5Q0WL/4r8ndouVZjZ5GT23mk71W5o6N4msDgeAptVekbUIasxPhUiNGCYEtxo 7UQt20pFleGPA5n3XKXzMXGKZqE5G8gLXnXXpJhOE+4SqoYRCdSIcLWPtQgSRSebeX9H CmOBsdhMdg6PK6fsErETJDQ3S0huM6YdcyEEF6vWavXj4mvncQkOjy0BOIE8ARwdaExE vt2g== X-Gm-Message-State: ACrzQf3mVmX6EyYEkhFSXaD1pOfM/dJBhiKhVuh1gpRGdf69jN3RvrX0 rPKobpGEZPCS6G+xHcq7izO8f/WZ5wI= X-Google-Smtp-Source: AMsMyM4vDhw+m5BphaEkUnPg0S68i2P70M0gnmcRitZUeO97PlBc8D2U/KsroAWUtqNe44E+ncfqcQ== X-Received: by 2002:a5d:6da9:0:b0:22e:53bd:31c1 with SMTP id u9-20020a5d6da9000000b0022e53bd31c1mr13999568wrs.358.1666372092918; Fri, 21 Oct 2022 10:08:12 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id b21-20020a05600c4e1500b003a83ca67f73sm3302882wmq.3.2022.10.21.10.08.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 21 Oct 2022 10:08:12 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Fri, 21 Oct 2022 17:08:01 +0000 Subject: [PATCH v2 4/6] http: move proactive auth to first slot creation Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Derrick Stolee , Lessley Dennington , Matthew John Cheetham , Matthew John Cheetham , Matthew John Cheetham Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Matthew John Cheetham From: Matthew John Cheetham Rather than proactively seek credentials to authenticate a request at `http_init()` time, do it when the first `active_request_slot` is created. Because credential helpers may modify the headers used for a request we can only auth when a slot is created (when we can first start to gather request headers). Signed-off-by: Matthew John Cheetham --- http.c | 20 +++++++++++--------- 1 file changed, 11 insertions(+), 9 deletions(-) diff --git a/http.c b/http.c index f2ebb17c8c4..17b47195d22 100644 --- a/http.c +++ b/http.c @@ -515,18 +515,18 @@ static int curl_empty_auth_enabled(void) return 0; } -static void init_curl_http_auth(CURL *result) +static void init_curl_http_auth(struct active_request_slot *slot) { if (!http_auth.username || !*http_auth.username) { if (curl_empty_auth_enabled()) - curl_easy_setopt(result, CURLOPT_USERPWD, ":"); + curl_easy_setopt(slot->curl, CURLOPT_USERPWD, ":"); return; } credential_fill(&http_auth); - curl_easy_setopt(result, CURLOPT_USERNAME, http_auth.username); - curl_easy_setopt(result, CURLOPT_PASSWORD, http_auth.password); + curl_easy_setopt(slot->curl, CURLOPT_USERNAME, http_auth.username); + curl_easy_setopt(slot->curl, CURLOPT_PASSWORD, http_auth.password); } /* *var must be free-able */ @@ -901,9 +901,6 @@ static CURL *get_curl_handle(void) #endif } - if (http_proactive_auth) - init_curl_http_auth(result); - if (getenv("GIT_SSL_VERSION")) ssl_version = getenv("GIT_SSL_VERSION"); if (ssl_version && *ssl_version) { @@ -1260,6 +1257,7 @@ struct active_request_slot *get_active_slot(int no_pragma_header) struct active_request_slot *slot = active_queue_head; struct active_request_slot *newslot; + int proactive_auth = 0; int num_transfers; /* Wait for a slot to open up if the queue is full */ @@ -1282,6 +1280,9 @@ struct active_request_slot *get_active_slot(int no_pragma_header) slot = active_queue_head; if (!slot) { active_queue_head = newslot; + + /* Auth first slot if asked for proactive auth */ + proactive_auth = http_proactive_auth; } else { while (slot->next != NULL) slot = slot->next; @@ -1336,8 +1337,9 @@ struct active_request_slot *get_active_slot(int no_pragma_header) curl_easy_setopt(slot->curl, CURLOPT_IPRESOLVE, git_curl_ipresolve); curl_easy_setopt(slot->curl, CURLOPT_HTTPAUTH, http_auth_methods); - if (http_auth.password || curl_empty_auth_enabled()) - init_curl_http_auth(slot->curl); + + if (http_auth.password || curl_empty_auth_enabled() || proactive_auth) + init_curl_http_auth(slot); return slot; } From patchwork Fri Oct 21 17:08:02 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew John Cheetham X-Patchwork-Id: 13015177 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 05B16C433FE for ; Fri, 21 Oct 2022 17:08:48 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230333AbiJURIr (ORCPT ); Fri, 21 Oct 2022 13:08:47 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49602 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230298AbiJURIc (ORCPT ); Fri, 21 Oct 2022 13:08:32 -0400 Received: from mail-wr1-x433.google.com (mail-wr1-x433.google.com [IPv6:2a00:1450:4864:20::433]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5DA8528E062 for ; Fri, 21 Oct 2022 10:08:16 -0700 (PDT) Received: by mail-wr1-x433.google.com with SMTP id b4so6028277wrs.1 for ; Fri, 21 Oct 2022 10:08:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:from:to:cc:subject:date :message-id:reply-to; bh=QCpc+ltT3AIh4jG8dPe0QF3HCFsDuta84IrLFJ6YXPU=; b=YTRdFIOW3Pbc714R22LUbfeX63b7S+GXOmKBa/4dUR/pplFmNqCHVdgIkAP4R5wXBY D5321Z8rOUS3BpVDCAwmYX00Aj06StUgBLke7FhYGg0WeFeuYrTkpWb6OvriVD6EcKI/ 6OozyBP/xU1PryQQVxij9n/nzKf4afwMRCVH2cmOWLQTRlpl/oYHwc7ogpXrmZuIa/Dn kx4TulOEb74dID3IKDtRzRVi52wWRSs9KTANlYiEI461vmYfs9m8JvJfeEOZ8i3bRpLY VuW8Cy0gMYw1Fmd448gey9qIBWIjRG6YTbMW+iIyAwnPzNa8SxEfKZ/bk3BKynyzseoA +kvg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=QCpc+ltT3AIh4jG8dPe0QF3HCFsDuta84IrLFJ6YXPU=; b=7VivNpPxcRGBCsEINbG+spk0VFqMFHeX4DsSl5LDnLfCQaBbqmh3SSIInZQO70WuGD o5THIn+PRqTrqXOM0A8X+DYN4+FgyCRtsiY+PHqvaAS3tHfybevMMfkTQgQW3Wp9/tct UKkWefTbDo0CWRmX+Ui4qqSI1ui8sYtXElZc4CYQRv6k64vIA+fFFP+sz3igxkLj+dt4 bkd+KbMnn02wDY6L8uMcGjfziZrkH/qmecTCwuRr7eWLW9p1RQV4Xn9ed4cE0qWxnGqY ba2pCAB3vS9D0slN4Qkm1oV1CR4EeykNAZ04k9Wh1noPDUmRi6h+/Y9bkt8RQHHdbfel HSew== X-Gm-Message-State: ACrzQf0+UUCh74cx4O1u2Pi/8hgQLYUEJME5VxbqCvmN11L96pVAVf/I JljssrNRaXNuN6NEqHH43vfsy1GVpt8= X-Google-Smtp-Source: AMsMyM6xBcGy3ipS8/SvQLokzrfuzI56ReDGnzbJTtBirgKud4chWByrKJvtq6nNFiO0UyjLYjN4TA== X-Received: by 2002:adf:edc3:0:b0:22c:dbe9:e3b6 with SMTP id v3-20020adfedc3000000b0022cdbe9e3b6mr12853865wro.282.1666372094855; Fri, 21 Oct 2022 10:08:14 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id f18-20020a05600c155200b003b3365b38f9sm3716517wmg.10.2022.10.21.10.08.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 21 Oct 2022 10:08:13 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Fri, 21 Oct 2022 17:08:02 +0000 Subject: [PATCH v2 5/6] http: set specific auth scheme depending on credential Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Derrick Stolee , Lessley Dennington , Matthew John Cheetham , Matthew John Cheetham , Matthew John Cheetham Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Matthew John Cheetham From: Matthew John Cheetham Introduce a new credential field `authtype` that can be used by credential helpers to indicate the type of the credential or authentication mechanism to use for a request. Modify http.c to now specify the correct authentication scheme or credential type when authenticating the curl handle. If the new `authtype` field in the credential structure is `NULL` or "Basic" then use the existing username/password options. If the field is "Bearer" then use the OAuth bearer token curl option. Otherwise, the `authtype` field is the authentication scheme and the `password` field is the raw, unencoded value. Signed-off-by: Matthew John Cheetham --- Documentation/git-credential.txt | 9 +++++++++ credential.c | 5 +++++ credential.h | 1 + git-curl-compat.h | 10 ++++++++++ http.c | 24 +++++++++++++++++++++--- 5 files changed, 46 insertions(+), 3 deletions(-) diff --git a/Documentation/git-credential.txt b/Documentation/git-credential.txt index 0ff3cbc25b9..82ade09b5e9 100644 --- a/Documentation/git-credential.txt +++ b/Documentation/git-credential.txt @@ -169,6 +169,15 @@ username in the example above) will be left unset. attribute 'wwwauth[]' where the order of the attributes is the same as they appear in the HTTP response. +`authtype`:: + + Indicates the type of authentication scheme used. If this is not + present the default is "Basic". + Known values include "Basic", "Digest", and "Bearer". + If an unknown value is provided, this is taken as the authentication + scheme for the `Authorization` header, and the `password` field is + used as the raw unencoded authorization parameters of the same header. + GIT --- Part of the linkgit:git[1] suite diff --git a/credential.c b/credential.c index 8a3ad6c0ae2..a556f9f375a 100644 --- a/credential.c +++ b/credential.c @@ -21,6 +21,7 @@ void credential_clear(struct credential *c) free(c->path); free(c->username); free(c->password); + free(c->authtype); string_list_clear(&c->helpers, 0); strvec_clear(&c->wwwauth_headers); @@ -235,6 +236,9 @@ int credential_read(struct credential *c, FILE *fp) } else if (!strcmp(key, "path")) { free(c->path); c->path = xstrdup(value); + } else if (!strcmp(key, "authtype")) { + free(c->authtype); + c->authtype = xstrdup(value); } else if (!strcmp(key, "url")) { credential_from_url(c, value); } else if (!strcmp(key, "quit")) { @@ -281,6 +285,7 @@ void credential_write(const struct credential *c, FILE *fp) credential_write_item(fp, "path", c->path, 0); credential_write_item(fp, "username", c->username, 0); credential_write_item(fp, "password", c->password, 0); + credential_write_item(fp, "authtype", c->authtype, 0); credential_write_strvec(fp, "wwwauth", &c->wwwauth_headers); } diff --git a/credential.h b/credential.h index 6f2e5bc610b..8d580b054d0 100644 --- a/credential.h +++ b/credential.h @@ -140,6 +140,7 @@ struct credential { char *protocol; char *host; char *path; + char *authtype; }; #define CREDENTIAL_INIT { \ diff --git a/git-curl-compat.h b/git-curl-compat.h index 56a83b6bbd8..839049f6dfe 100644 --- a/git-curl-compat.h +++ b/git-curl-compat.h @@ -126,4 +126,14 @@ #define GIT_CURL_HAVE_CURLSSLSET_NO_BACKENDS #endif +/** + * CURLAUTH_BEARER was added in 7.61.0, released in July 2018. + * However, only 7.69.0 fixes a bug where Bearer headers were not + * actually sent with reused connections on subsequent transfers + * (curl/curl@dea17b519dc1). + */ +#if LIBCURL_VERSION_NUM >= 0x074500 +#define GIT_CURL_HAVE_CURLAUTH_BEARER +#endif + #endif diff --git a/http.c b/http.c index 17b47195d22..ac620bcbf0c 100644 --- a/http.c +++ b/http.c @@ -517,7 +517,8 @@ static int curl_empty_auth_enabled(void) static void init_curl_http_auth(struct active_request_slot *slot) { - if (!http_auth.username || !*http_auth.username) { + if (!http_auth.authtype && + (!http_auth.username || !*http_auth.username)) { if (curl_empty_auth_enabled()) curl_easy_setopt(slot->curl, CURLOPT_USERPWD, ":"); return; @@ -525,8 +526,25 @@ static void init_curl_http_auth(struct active_request_slot *slot) credential_fill(&http_auth); - curl_easy_setopt(slot->curl, CURLOPT_USERNAME, http_auth.username); - curl_easy_setopt(slot->curl, CURLOPT_PASSWORD, http_auth.password); + if (!http_auth.authtype || !strcasecmp(http_auth.authtype, "basic") + || !strcasecmp(http_auth.authtype, "digest")) { + curl_easy_setopt(slot->curl, CURLOPT_USERNAME, + http_auth.username); + curl_easy_setopt(slot->curl, CURLOPT_PASSWORD, + http_auth.password); +#ifdef GIT_CURL_HAVE_CURLAUTH_BEARER + } else if (!strcasecmp(http_auth.authtype, "bearer")) { + curl_easy_setopt(slot->curl, CURLOPT_HTTPAUTH, CURLAUTH_BEARER); + curl_easy_setopt(slot->curl, CURLOPT_XOAUTH2_BEARER, + http_auth.password); +#endif + } else { + struct strbuf auth = STRBUF_INIT; + strbuf_addf(&auth, "Authorization: %s %s", + http_auth.authtype, http_auth.password); + slot->headers = curl_slist_append(slot->headers, auth.buf); + strbuf_release(&auth); + } } /* *var must be free-able */ From patchwork Fri Oct 21 17:08:03 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew John Cheetham X-Patchwork-Id: 13015178 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0B95BFA373E for ; Fri, 21 Oct 2022 17:08:50 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230367AbiJURIs (ORCPT ); Fri, 21 Oct 2022 13:08:48 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49528 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230337AbiJURIo (ORCPT ); Fri, 21 Oct 2022 13:08:44 -0400 Received: from mail-wm1-x334.google.com (mail-wm1-x334.google.com [IPv6:2a00:1450:4864:20::334]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BEF4D28E07C for ; Fri, 21 Oct 2022 10:08:19 -0700 (PDT) Received: by mail-wm1-x334.google.com with SMTP id t4so2652843wmj.5 for ; Fri, 21 Oct 2022 10:08:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:from:to:cc:subject:date :message-id:reply-to; bh=7r9Sbql5TP5U2KeqRbixAhzyb4722wcFjGJfMXFTIu0=; b=dCrtkaOejlrf/hV0HrisvWRW6n8yr+CkAWh1OQaOjOgG7ouQSGtJIJluBn23t+vtHl /StjkKfq1b0IHv4iWMuncgBfMQeKmS3oMSeCM4lSFLXKdwlRu1CDn6lb+hn3hxYTao4z pkKQf/+u1vLbycqLp59z9ltyLMqb/y0pJwB2W/nLghnlr3VWsugy5np1aW/M7IatTtLS rDgbfShIARQDDQcu5z33B256cGTGg8z99saKfDE7PKGfvSOor/cD0jtQ36M5SP51XBox uawP95daIwCCab5vWsog0JtYkVYw54fozmuUXhFqxMd7PW0VPfybPxgw3kL56sdNj/h8 BDSA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=7r9Sbql5TP5U2KeqRbixAhzyb4722wcFjGJfMXFTIu0=; b=KgzBUbUiWNch+LUr1Dmioi0xHmOSE2tWFT7sbzOUbIpWjgGRRU78Df0m1x/IoAi3KE bGG8faSvcZIoI1Nc5li6KBT6TY14WhEslKcCnV0PCmfIvlMVmQy7uDuRbQK+k7hf7fxk CfirOOXBtTJK/goEi00F+uxih+JIvfLZM5B+/tlcF3RYfxD/kSxuZ6BDu5tE2PJUSQlA YGL1F+6V6x+Nc34/zbi1jXFGp6c+wi6Xi8cPN+RkYM03C9yBzWlsCQWLcUwYTiWap+/Y +RmOaTq8W4dKtrICo72zLbkd/qg7CCLfp+x0G+aew3ww26setoVrPZxVBOAVE0Qwjz2w 0Yqw== X-Gm-Message-State: ACrzQf3yBsSPLwwZk1J41kYROVPU5Iv8fod/5oPWOVBZpIWP4xG4Tfrp GtDY1o0j/9mrSNnIUYs/Kjq4/BnDOuo= X-Google-Smtp-Source: AMsMyM7X4ZXyIjN0AXlg0NqFtx97UZrwONrKOMD8q89SkccekwuJLQs+f9bTjhpjos4zQi3fvsqCeA== X-Received: by 2002:a05:600c:3509:b0:3c6:fd36:ef19 with SMTP id h9-20020a05600c350900b003c6fd36ef19mr14209849wmq.191.1666372096867; Fri, 21 Oct 2022 10:08:16 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id a18-20020a056000101200b0023655e51c33sm1695002wrx.4.2022.10.21.10.08.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 21 Oct 2022 10:08:16 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Fri, 21 Oct 2022 17:08:03 +0000 Subject: [PATCH v2 6/6] t5556-http-auth: add test for HTTP auth hdr logic Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Derrick Stolee , Lessley Dennington , Matthew John Cheetham , Matthew John Cheetham , Matthew John Cheetham Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Matthew John Cheetham From: Matthew John Cheetham Add a series of tests to exercise the HTTP authentication header parsing and the interop with credential helpers. Credential helpers can respond to requests that contain WWW-Authenticate information with the ability to select the response Authenticate header scheme. Introduce a mini HTTP server helper that provides a frontend for the git-http-backend, with support for arbitrary authentication schemes. The test-http-server is based heavily on the git-daemon, and forwards all successfully authenticated requests to the http-backend. Signed-off-by: Matthew John Cheetham --- Makefile | 2 + contrib/buildsystems/CMakeLists.txt | 13 + t/helper/.gitignore | 1 + t/helper/test-credential-helper-replay.sh | 14 + t/helper/test-http-server.c | 1134 +++++++++++++++++++++ t/t5556-http-auth.sh | 260 +++++ 6 files changed, 1424 insertions(+) create mode 100755 t/helper/test-credential-helper-replay.sh create mode 100644 t/helper/test-http-server.c create mode 100755 t/t5556-http-auth.sh diff --git a/Makefile b/Makefile index d93ad956e58..39b130f711d 100644 --- a/Makefile +++ b/Makefile @@ -1500,6 +1500,8 @@ else endif BASIC_CFLAGS += $(CURL_CFLAGS) + TEST_PROGRAMS_NEED_X += test-http-server + REMOTE_CURL_PRIMARY = git-remote-http$X REMOTE_CURL_ALIASES = git-remote-https$X git-remote-ftp$X git-remote-ftps$X REMOTE_CURL_NAMES = $(REMOTE_CURL_PRIMARY) $(REMOTE_CURL_ALIASES) diff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt index 787738e6fa3..45251695ce0 100644 --- a/contrib/buildsystems/CMakeLists.txt +++ b/contrib/buildsystems/CMakeLists.txt @@ -989,6 +989,19 @@ set(wrapper_scripts set(wrapper_test_scripts test-fake-ssh test-tool) +if(CURL_FOUND) + list(APPEND wrapper_test_scripts test-http-server) + + add_executable(test-http-server ${CMAKE_SOURCE_DIR}/t/helper/test-http-server.c) + target_link_libraries(test-http-server common-main) + + if(MSVC) + set_target_properties(test-http-server + PROPERTIES RUNTIME_OUTPUT_DIRECTORY_DEBUG ${CMAKE_BINARY_DIR}/t/helper) + set_target_properties(test-http-server + PROPERTIES RUNTIME_OUTPUT_DIRECTORY_RELEASE ${CMAKE_BINARY_DIR}/t/helper) + endif() +endif() foreach(script ${wrapper_scripts}) file(STRINGS ${CMAKE_SOURCE_DIR}/wrap-for-bin.sh content NEWLINE_CONSUME) diff --git a/t/helper/.gitignore b/t/helper/.gitignore index 8c2ddcce95f..1a94ab6eed5 100644 --- a/t/helper/.gitignore +++ b/t/helper/.gitignore @@ -1,2 +1,3 @@ /test-tool /test-fake-ssh +test-http-server diff --git a/t/helper/test-credential-helper-replay.sh b/t/helper/test-credential-helper-replay.sh new file mode 100755 index 00000000000..03e5e63dad6 --- /dev/null +++ b/t/helper/test-credential-helper-replay.sh @@ -0,0 +1,14 @@ +cmd=$1 +teefile=$cmd-actual.cred +catfile=$cmd-response.cred +rm -f $teefile +while read line; +do + if test -z "$line"; then + break; + fi + echo "$line" >> $teefile +done +if test "$cmd" = "get"; then + cat $catfile +fi diff --git a/t/helper/test-http-server.c b/t/helper/test-http-server.c new file mode 100644 index 00000000000..92139c04c90 --- /dev/null +++ b/t/helper/test-http-server.c @@ -0,0 +1,1134 @@ +#include "config.h" +#include "run-command.h" +#include "strbuf.h" +#include "string-list.h" +#include "trace2.h" +#include "version.h" +#include "dir.h" +#include "date.h" + +#define TR2_CAT "test-http-server" + +static const char *pid_file; +static int verbose; +static int reuseaddr; + +static const char test_http_auth_usage[] = +"http-server [--verbose]\n" +" [--timeout=] [--init-timeout=] [--max-connections=]\n" +" [--reuseaddr] [--pid-file=]\n" +" [--listen=]* [--port=]\n" +" [--anonymous-allowed]\n" +" [--auth=[:] [--auth-token=:]]*\n" +; + +/* Timeout, and initial timeout */ +static unsigned int timeout; +static unsigned int init_timeout; + +static void logreport(const char *label, const char *err, va_list params) +{ + struct strbuf msg = STRBUF_INIT; + + strbuf_addf(&msg, "[%"PRIuMAX"] %s: ", (uintmax_t)getpid(), label); + strbuf_vaddf(&msg, err, params); + strbuf_addch(&msg, '\n'); + + fwrite(msg.buf, sizeof(char), msg.len, stderr); + fflush(stderr); + + strbuf_release(&msg); +} + +__attribute__((format (printf, 1, 2))) +static void logerror(const char *err, ...) +{ + va_list params; + va_start(params, err); + logreport("error", err, params); + va_end(params); +} + +__attribute__((format (printf, 1, 2))) +static void loginfo(const char *err, ...) +{ + va_list params; + if (!verbose) + return; + va_start(params, err); + logreport("info", err, params); + va_end(params); +} + +static void set_keep_alive(int sockfd) +{ + int ka = 1; + + if (setsockopt(sockfd, SOL_SOCKET, SO_KEEPALIVE, &ka, sizeof(ka)) < 0) { + if (errno != ENOTSOCK) + logerror("unable to set SO_KEEPALIVE on socket: %s", + strerror(errno)); + } +} + +////////////////////////////////////////////////////////////////// +// The code in this section is used by "worker" instances to service +// a single connection from a client. The worker talks to the client +// on 0 and 1. +////////////////////////////////////////////////////////////////// + +enum worker_result { + /* + * Operation successful. + * Caller *might* keep the socket open and allow keep-alive. + */ + WR_OK = 0, + /* + * Various errors while processing the request and/or the response. + * Close the socket and clean up. + * Exit child-process with non-zero status. + */ + WR_IO_ERROR = 1<<0, + /* + * Close the socket and clean up. Does not imply an error. + */ + WR_HANGUP = 1<<1, + + WR_STOP_THE_MUSIC = (WR_IO_ERROR | WR_HANGUP), +}; + +/* + * Fields from a parsed HTTP request. + */ +struct req { + struct strbuf start_line; + + const char *method; + const char *http_version; + + struct strbuf uri_path; + struct strbuf query_args; + + struct string_list header_list; + const char *content_type; + ssize_t content_length; +}; + +#define REQ__INIT { \ + .start_line = STRBUF_INIT, \ + .uri_path = STRBUF_INIT, \ + .query_args = STRBUF_INIT, \ + .header_list = STRING_LIST_INIT_NODUP, \ + .content_type = NULL, \ + .content_length = -1 \ + } + +static void req__release(struct req *req) +{ + strbuf_release(&req->start_line); + + strbuf_release(&req->uri_path); + strbuf_release(&req->query_args); + + string_list_clear(&req->header_list, 0); +} + +static enum worker_result send_http_error( + int fd, + int http_code, const char *http_code_name, + int retry_after_seconds, struct string_list *response_headers, + enum worker_result wr_in) +{ + struct strbuf response_header = STRBUF_INIT; + struct strbuf response_content = STRBUF_INIT; + struct string_list_item *h; + enum worker_result wr; + + strbuf_addf(&response_content, "Error: %d %s\r\n", + http_code, http_code_name); + if (retry_after_seconds > 0) + strbuf_addf(&response_content, "Retry-After: %d\r\n", + retry_after_seconds); + + strbuf_addf (&response_header, "HTTP/1.1 %d %s\r\n", http_code, http_code_name); + strbuf_addstr(&response_header, "Cache-Control: private\r\n"); + strbuf_addstr(&response_header, "Content-Type: text/plain\r\n"); + strbuf_addf (&response_header, "Content-Length: %d\r\n", (int)response_content.len); + if (retry_after_seconds > 0) + strbuf_addf (&response_header, "Retry-After: %d\r\n", retry_after_seconds); + strbuf_addf( &response_header, "Server: test-http-server/%s\r\n", git_version_string); + strbuf_addf( &response_header, "Date: %s\r\n", show_date(time(NULL), 0, DATE_MODE(RFC2822))); + if (response_headers) + for_each_string_list_item(h, response_headers) + strbuf_addf(&response_header, "%s\r\n", h->string); + strbuf_addstr(&response_header, "\r\n"); + + if (write_in_full(fd, response_header.buf, response_header.len) < 0) { + logerror("unable to write response header"); + wr = WR_IO_ERROR; + goto done; + } + + if (write_in_full(fd, response_content.buf, response_content.len) < 0) { + logerror("unable to write response content body"); + wr = WR_IO_ERROR; + goto done; + } + + wr = wr_in; + +done: + strbuf_release(&response_header); + strbuf_release(&response_content); + + return wr; +} + +/* + * Read the HTTP request up to the start of the optional message-body. + * We do this byte-by-byte because we have keep-alive turned on and + * cannot rely on an EOF. + * + * https://tools.ietf.org/html/rfc7230 + * + * We cannot call die() here because our caller needs to properly + * respond to the client and/or close the socket before this + * child exits so that the client doesn't get a connection reset + * by peer error. + */ +static enum worker_result req__read(struct req *req, int fd) +{ + struct strbuf h = STRBUF_INIT; + struct string_list start_line_fields = STRING_LIST_INIT_DUP; + int nr_start_line_fields; + const char *uri_target; + const char *query; + char *hp; + const char *hv; + + enum worker_result result = WR_OK; + + /* + * Read line 0 of the request and split it into component parts: + * + * SP SP CRLF + * + */ + if (strbuf_getwholeline_fd(&req->start_line, fd, '\n') == EOF) { + result = WR_OK | WR_HANGUP; + goto done; + } + + strbuf_trim_trailing_newline(&req->start_line); + + nr_start_line_fields = string_list_split(&start_line_fields, + req->start_line.buf, + ' ', -1); + if (nr_start_line_fields != 3) { + logerror("could not parse request start-line '%s'", + req->start_line.buf); + result = WR_IO_ERROR; + goto done; + } + + req->method = xstrdup(start_line_fields.items[0].string); + req->http_version = xstrdup(start_line_fields.items[2].string); + + uri_target = start_line_fields.items[1].string; + + if (strcmp(req->http_version, "HTTP/1.1")) { + logerror("unsupported version '%s' (expecting HTTP/1.1)", + req->http_version); + result = WR_IO_ERROR; + goto done; + } + + query = strchr(uri_target, '?'); + + if (query) { + strbuf_add(&req->uri_path, uri_target, (query - uri_target)); + strbuf_trim_trailing_dir_sep(&req->uri_path); + strbuf_addstr(&req->query_args, query + 1); + } else { + strbuf_addstr(&req->uri_path, uri_target); + strbuf_trim_trailing_dir_sep(&req->uri_path); + } + + /* + * Read the set of HTTP headers into a string-list. + */ + while (1) { + if (strbuf_getwholeline_fd(&h, fd, '\n') == EOF) + goto done; + strbuf_trim_trailing_newline(&h); + + if (!h.len) + goto done; /* a blank line ends the header */ + + hp = strbuf_detach(&h, NULL); + string_list_append(&req->header_list, hp); + + /* store common request headers separately */ + if (skip_prefix(hp, "Content-Type: ", &hv)) { + req->content_type = hv; + } else if (skip_prefix(hp, "Content-Length: ", &hv)) { + req->content_length = strtol(hv, &hp, 10); + } + } + + /* + * We do not attempt to read the , if it exists. + * We let our caller read/chunk it in as appropriate. + */ + +done: + string_list_clear(&start_line_fields, 0); + + /* + * This is useful for debugging the request, but very noisy. + */ + if (trace2_is_enabled()) { + struct string_list_item *item; + trace2_printf("%s: %s", TR2_CAT, req->start_line.buf); + trace2_printf("%s: hver: %s", TR2_CAT, req->http_version); + trace2_printf("%s: hmth: %s", TR2_CAT, req->method); + trace2_printf("%s: path: %s", TR2_CAT, req->uri_path.buf); + trace2_printf("%s: qury: %s", TR2_CAT, req->query_args.buf); + if (req->content_length >= 0) + trace2_printf("%s: clen: %d", TR2_CAT, req->content_length); + if (req->content_type) + trace2_printf("%s: ctyp: %s", TR2_CAT, req->content_type); + for_each_string_list_item(item, &req->header_list) + trace2_printf("%s: hdrs: %s", TR2_CAT, item->string); + } + + return result; +} + +static int is_git_request(struct req *req) +{ + static regex_t *smart_http_regex; + static int initialized; + + if (!initialized) { + smart_http_regex = xmalloc(sizeof(*smart_http_regex)); + if (regcomp(smart_http_regex, "^/(HEAD|info/refs|" + "objects/info/[^/]+|git-(upload|receive)-pack)$", + REG_EXTENDED)) { + warning("could not compile smart HTTP regex"); + smart_http_regex = NULL; + } + initialized = 1; + } + + return smart_http_regex && + !regexec(smart_http_regex, req->uri_path.buf, 0, NULL, 0); +} + +static enum worker_result do__git(struct req *req, const char *user) +{ + const char *ok = "HTTP/1.1 200 OK\r\n"; + struct child_process cp = CHILD_PROCESS_INIT; + int res; + + if (write(1, ok, strlen(ok)) < 0) + return error(_("could not send '%s'"), ok); + + if (user) + strvec_pushf(&cp.env, "REMOTE_USER=%s", user); + + strvec_pushf(&cp.env, "REQUEST_METHOD=%s", req->method); + strvec_pushf(&cp.env, "PATH_TRANSLATED=%s", + req->uri_path.buf); + strvec_push(&cp.env, "SERVER_PROTOCOL=HTTP/1.1"); + if (req->query_args.len) + strvec_pushf(&cp.env, "QUERY_STRING=%s", + req->query_args.buf); + if (req->content_type) + strvec_pushf(&cp.env, "CONTENT_TYPE=%s", + req->content_type); + if (req->content_length >= 0) + strvec_pushf(&cp.env, "CONTENT_LENGTH=%" PRIdMAX, + (intmax_t)req->content_length); + cp.git_cmd = 1; + strvec_push(&cp.args, "http-backend"); + res = run_command(&cp); + close(1); + close(0); + return !!res; +} + +enum auth_result { + AUTH_UNKNOWN = 0, + AUTH_DENY = 1, + AUTH_ALLOW = 2, +}; + +struct auth_module { + const char *scheme; + const char *challenge_params; + struct string_list *tokens; +}; + +static int allow_anonymous; +static struct auth_module **auth_modules = NULL; +static size_t auth_modules_nr = 0; +static size_t auth_modules_alloc = 0; + +static struct auth_module *get_auth_module(struct strbuf *scheme) +{ + int i; + struct auth_module *mod; + for (i = 0; i < auth_modules_nr; i++) { + mod = auth_modules[i]; + if (!strcasecmp(mod->scheme, scheme->buf)) + return mod; + } + + return NULL; +} + +static void add_auth_module(struct auth_module *mod) +{ + ALLOC_GROW(auth_modules, auth_modules_nr + 1, auth_modules_alloc); + auth_modules[auth_modules_nr++] = mod; +} + +static int is_authed(struct req *req, const char **user, enum worker_result *wr) +{ + enum auth_result result = AUTH_UNKNOWN; + struct string_list hdrs = STRING_LIST_INIT_NODUP; + struct auth_module *mod; + + struct string_list_item *hdr; + struct string_list_item *token; + const char *v; + struct strbuf **split = NULL; + int i; + char *challenge; + + /* ask all auth modules to validate the request */ + for_each_string_list_item(hdr, &req->header_list) { + if (skip_iprefix(hdr->string, "Authorization: ", &v)) { + split = strbuf_split_str(v, ' ', 2); + if (!split[0] || !split[1]) continue; + + // trim trailing space ' ' + strbuf_setlen(split[0], split[0]->len - 1); + + mod = get_auth_module(split[0]); + if (mod) { + + for_each_string_list_item(token, mod->tokens) { + if (!strcmp(split[1]->buf, token->string)) { + result = AUTH_ALLOW; + goto done; + } + } + + if (result != AUTH_UNKNOWN) + goto done; + } + } + } + +done: + switch (result) { + case AUTH_ALLOW: + trace2_printf("%s: auth '%s' ALLOW", TR2_CAT, mod->scheme); + *user = "VALID_TEST_USER"; + *wr = WR_OK; + break; + + case AUTH_DENY: + trace2_printf("%s: auth '%s' DENY", TR2_CAT, mod->scheme); + /* fall-through */ + + case AUTH_UNKNOWN: + if (allow_anonymous) + break; + for (i = 0; i < auth_modules_nr; i++) { + mod = auth_modules[i]; + if (mod->challenge_params) + challenge = xstrfmt("WWW-Authenticate: %s %s", + mod->scheme, + mod->challenge_params); + else + challenge = xstrfmt("WWW-Authenticate: %s", + mod->scheme); + string_list_append(&hdrs, challenge); + } + *wr = send_http_error(1, 401, "Unauthorized", -1, &hdrs, *wr); + } + + strbuf_list_free(split); + string_list_clear(&hdrs, 0); + + return result == AUTH_ALLOW || + (result == AUTH_UNKNOWN && allow_anonymous); +} + +static enum worker_result dispatch(struct req *req) +{ + enum worker_result wr = WR_OK; + const char *user = NULL; + + if (!is_authed(req, &user, &wr)) + return wr; + + if (is_git_request(req)) + return do__git(req, user); + + return send_http_error(1, 501, "Not Implemented", -1, NULL, + WR_OK | WR_HANGUP); +} + +static enum worker_result worker(void) +{ + struct req req = REQ__INIT; + char *client_addr = getenv("REMOTE_ADDR"); + char *client_port = getenv("REMOTE_PORT"); + enum worker_result wr = WR_OK; + + if (client_addr) + loginfo("Connection from %s:%s", client_addr, client_port); + + set_keep_alive(0); + + while (1) { + req__release(&req); + + alarm(init_timeout ? init_timeout : timeout); + wr = req__read(&req, 0); + alarm(0); + + if (wr & WR_STOP_THE_MUSIC) + break; + + wr = dispatch(&req); + if (wr & WR_STOP_THE_MUSIC) + break; + } + + close(0); + close(1); + + return !!(wr & WR_IO_ERROR); +} + +////////////////////////////////////////////////////////////////// +// This section contains the listener and child-process management +// code used by the primary instance to accept incoming connections +// and dispatch them to async child process "worker" instances. +////////////////////////////////////////////////////////////////// + +static int addrcmp(const struct sockaddr_storage *s1, + const struct sockaddr_storage *s2) +{ + const struct sockaddr *sa1 = (const struct sockaddr*) s1; + const struct sockaddr *sa2 = (const struct sockaddr*) s2; + + if (sa1->sa_family != sa2->sa_family) + return sa1->sa_family - sa2->sa_family; + if (sa1->sa_family == AF_INET) + return memcmp(&((struct sockaddr_in *)s1)->sin_addr, + &((struct sockaddr_in *)s2)->sin_addr, + sizeof(struct in_addr)); +#ifndef NO_IPV6 + if (sa1->sa_family == AF_INET6) + return memcmp(&((struct sockaddr_in6 *)s1)->sin6_addr, + &((struct sockaddr_in6 *)s2)->sin6_addr, + sizeof(struct in6_addr)); +#endif + return 0; +} + +static int max_connections = 32; + +static unsigned int live_children; + +static struct child { + struct child *next; + struct child_process cld; + struct sockaddr_storage address; +} *firstborn; + +static void add_child(struct child_process *cld, struct sockaddr *addr, socklen_t addrlen) +{ + struct child *newborn, **cradle; + + newborn = xcalloc(1, sizeof(*newborn)); + live_children++; + memcpy(&newborn->cld, cld, sizeof(*cld)); + memcpy(&newborn->address, addr, addrlen); + for (cradle = &firstborn; *cradle; cradle = &(*cradle)->next) + if (!addrcmp(&(*cradle)->address, &newborn->address)) + break; + newborn->next = *cradle; + *cradle = newborn; +} + +/* + * This gets called if the number of connections grows + * past "max_connections". + * + * We kill the newest connection from a duplicate IP. + */ +static void kill_some_child(void) +{ + const struct child *blanket, *next; + + if (!(blanket = firstborn)) + return; + + for (; (next = blanket->next); blanket = next) + if (!addrcmp(&blanket->address, &next->address)) { + kill(blanket->cld.pid, SIGTERM); + break; + } +} + +static void check_dead_children(void) +{ + int status; + pid_t pid; + + struct child **cradle, *blanket; + for (cradle = &firstborn; (blanket = *cradle);) + if ((pid = waitpid(blanket->cld.pid, &status, WNOHANG)) > 1) { + const char *dead = ""; + if (status) + dead = " (with error)"; + loginfo("[%"PRIuMAX"] Disconnected%s", (uintmax_t)pid, dead); + + /* remove the child */ + *cradle = blanket->next; + live_children--; + child_process_clear(&blanket->cld); + free(blanket); + } else + cradle = &blanket->next; +} + +static struct strvec cld_argv = STRVEC_INIT; +static void handle(int incoming, struct sockaddr *addr, socklen_t addrlen) +{ + struct child_process cld = CHILD_PROCESS_INIT; + + if (max_connections && live_children >= max_connections) { + kill_some_child(); + sleep(1); /* give it some time to die */ + check_dead_children(); + if (live_children >= max_connections) { + close(incoming); + logerror("Too many children, dropping connection"); + return; + } + } + + if (addr->sa_family == AF_INET) { + char buf[128] = ""; + struct sockaddr_in *sin_addr = (void *) addr; + inet_ntop(addr->sa_family, &sin_addr->sin_addr, buf, sizeof(buf)); + strvec_pushf(&cld.env, "REMOTE_ADDR=%s", buf); + strvec_pushf(&cld.env, "REMOTE_PORT=%d", + ntohs(sin_addr->sin_port)); +#ifndef NO_IPV6 + } else if (addr->sa_family == AF_INET6) { + char buf[128] = ""; + struct sockaddr_in6 *sin6_addr = (void *) addr; + inet_ntop(AF_INET6, &sin6_addr->sin6_addr, buf, sizeof(buf)); + strvec_pushf(&cld.env, "REMOTE_ADDR=[%s]", buf); + strvec_pushf(&cld.env, "REMOTE_PORT=%d", + ntohs(sin6_addr->sin6_port)); +#endif + } + + strvec_pushv(&cld.args, cld_argv.v); + cld.in = incoming; + cld.out = dup(incoming); + + if (cld.out < 0) + logerror("could not dup() `incoming`"); + else if (start_command(&cld)) + logerror("unable to fork"); + else + add_child(&cld, addr, addrlen); +} + +static void child_handler(int signo) +{ + /* + * Otherwise empty handler because systemcalls will get interrupted + * upon signal receipt + * SysV needs the handler to be rearmed + */ + signal(SIGCHLD, child_handler); +} + +static int set_reuse_addr(int sockfd) +{ + int on = 1; + + if (!reuseaddr) + return 0; + return setsockopt(sockfd, SOL_SOCKET, SO_REUSEADDR, + &on, sizeof(on)); +} + +struct socketlist { + int *list; + size_t nr; + size_t alloc; +}; + +static const char *ip2str(int family, struct sockaddr *sin, socklen_t len) +{ +#ifdef NO_IPV6 + static char ip[INET_ADDRSTRLEN]; +#else + static char ip[INET6_ADDRSTRLEN]; +#endif + + switch (family) { +#ifndef NO_IPV6 + case AF_INET6: + inet_ntop(family, &((struct sockaddr_in6*)sin)->sin6_addr, ip, len); + break; +#endif + case AF_INET: + inet_ntop(family, &((struct sockaddr_in*)sin)->sin_addr, ip, len); + break; + default: + xsnprintf(ip, sizeof(ip), ""); + } + return ip; +} + +#ifndef NO_IPV6 + +static int setup_named_sock(char *listen_addr, int listen_port, struct socketlist *socklist) +{ + int socknum = 0; + char pbuf[NI_MAXSERV]; + struct addrinfo hints, *ai0, *ai; + int gai; + long flags; + + xsnprintf(pbuf, sizeof(pbuf), "%d", listen_port); + memset(&hints, 0, sizeof(hints)); + hints.ai_family = AF_UNSPEC; + hints.ai_socktype = SOCK_STREAM; + hints.ai_protocol = IPPROTO_TCP; + hints.ai_flags = AI_PASSIVE; + + gai = getaddrinfo(listen_addr, pbuf, &hints, &ai0); + if (gai) { + logerror("getaddrinfo() for %s failed: %s", listen_addr, gai_strerror(gai)); + return 0; + } + + for (ai = ai0; ai; ai = ai->ai_next) { + int sockfd; + + sockfd = socket(ai->ai_family, ai->ai_socktype, ai->ai_protocol); + if (sockfd < 0) + continue; + if (sockfd >= FD_SETSIZE) { + logerror("Socket descriptor too large"); + close(sockfd); + continue; + } + +#ifdef IPV6_V6ONLY + if (ai->ai_family == AF_INET6) { + int on = 1; + setsockopt(sockfd, IPPROTO_IPV6, IPV6_V6ONLY, + &on, sizeof(on)); + /* Note: error is not fatal */ + } +#endif + + if (set_reuse_addr(sockfd)) { + logerror("Could not set SO_REUSEADDR: %s", strerror(errno)); + close(sockfd); + continue; + } + + set_keep_alive(sockfd); + + if (bind(sockfd, ai->ai_addr, ai->ai_addrlen) < 0) { + logerror("Could not bind to %s: %s", + ip2str(ai->ai_family, ai->ai_addr, ai->ai_addrlen), + strerror(errno)); + close(sockfd); + continue; /* not fatal */ + } + if (listen(sockfd, 5) < 0) { + logerror("Could not listen to %s: %s", + ip2str(ai->ai_family, ai->ai_addr, ai->ai_addrlen), + strerror(errno)); + close(sockfd); + continue; /* not fatal */ + } + + flags = fcntl(sockfd, F_GETFD, 0); + if (flags >= 0) + fcntl(sockfd, F_SETFD, flags | FD_CLOEXEC); + + ALLOC_GROW(socklist->list, socklist->nr + 1, socklist->alloc); + socklist->list[socklist->nr++] = sockfd; + socknum++; + } + + freeaddrinfo(ai0); + + return socknum; +} + +#else /* NO_IPV6 */ + +static int setup_named_sock(char *listen_addr, int listen_port, struct socketlist *socklist) +{ + struct sockaddr_in sin; + int sockfd; + long flags; + + memset(&sin, 0, sizeof sin); + sin.sin_family = AF_INET; + sin.sin_port = htons(listen_port); + + if (listen_addr) { + /* Well, host better be an IP address here. */ + if (inet_pton(AF_INET, listen_addr, &sin.sin_addr.s_addr) <= 0) + return 0; + } else { + sin.sin_addr.s_addr = htonl(INADDR_ANY); + } + + sockfd = socket(AF_INET, SOCK_STREAM, 0); + if (sockfd < 0) + return 0; + + if (set_reuse_addr(sockfd)) { + logerror("Could not set SO_REUSEADDR: %s", strerror(errno)); + close(sockfd); + return 0; + } + + set_keep_alive(sockfd); + + if ( bind(sockfd, (struct sockaddr *)&sin, sizeof sin) < 0 ) { + logerror("Could not bind to %s: %s", + ip2str(AF_INET, (struct sockaddr *)&sin, sizeof(sin)), + strerror(errno)); + close(sockfd); + return 0; + } + + if (listen(sockfd, 5) < 0) { + logerror("Could not listen to %s: %s", + ip2str(AF_INET, (struct sockaddr *)&sin, sizeof(sin)), + strerror(errno)); + close(sockfd); + return 0; + } + + flags = fcntl(sockfd, F_GETFD, 0); + if (flags >= 0) + fcntl(sockfd, F_SETFD, flags | FD_CLOEXEC); + + ALLOC_GROW(socklist->list, socklist->nr + 1, socklist->alloc); + socklist->list[socklist->nr++] = sockfd; + return 1; +} + +#endif + +static void socksetup(struct string_list *listen_addr, int listen_port, struct socketlist *socklist) +{ + if (!listen_addr->nr) + setup_named_sock("127.0.0.1", listen_port, socklist); + else { + int i, socknum; + for (i = 0; i < listen_addr->nr; i++) { + socknum = setup_named_sock(listen_addr->items[i].string, + listen_port, socklist); + + if (socknum == 0) + logerror("unable to allocate any listen sockets for host %s on port %u", + listen_addr->items[i].string, listen_port); + } + } +} + +static int service_loop(struct socketlist *socklist) +{ + struct pollfd *pfd; + int i; + + CALLOC_ARRAY(pfd, socklist->nr); + + for (i = 0; i < socklist->nr; i++) { + pfd[i].fd = socklist->list[i]; + pfd[i].events = POLLIN; + } + + signal(SIGCHLD, child_handler); + + for (;;) { + int i; + int nr_ready; + int timeout = (pid_file ? 100 : -1); + + check_dead_children(); + + nr_ready = poll(pfd, socklist->nr, timeout); + if (nr_ready < 0) { + if (errno != EINTR) { + logerror("Poll failed, resuming: %s", + strerror(errno)); + sleep(1); + } + continue; + } + else if (nr_ready == 0) { + /* + * If we have a pid_file, then we watch it. + * If someone deletes it, we shutdown the service. + * The shell scripts in the test suite will use this. + */ + if (!pid_file || file_exists(pid_file)) + continue; + goto shutdown; + } + + for (i = 0; i < socklist->nr; i++) { + if (pfd[i].revents & POLLIN) { + union { + struct sockaddr sa; + struct sockaddr_in sai; +#ifndef NO_IPV6 + struct sockaddr_in6 sai6; +#endif + } ss; + socklen_t sslen = sizeof(ss); + int incoming = accept(pfd[i].fd, &ss.sa, &sslen); + if (incoming < 0) { + switch (errno) { + case EAGAIN: + case EINTR: + case ECONNABORTED: + continue; + default: + die_errno("accept returned"); + } + } + handle(incoming, &ss.sa, sslen); + } + } + } + +shutdown: + loginfo("Starting graceful shutdown (pid-file gone)"); + for (i = 0; i < socklist->nr; i++) + close(socklist->list[i]); + + return 0; +} + +static int serve(struct string_list *listen_addr, int listen_port) +{ + struct socketlist socklist = { NULL, 0, 0 }; + + socksetup(listen_addr, listen_port, &socklist); + if (socklist.nr == 0) + die("unable to allocate any listen sockets on port %u", + listen_port); + + loginfo("Ready to rumble"); + + /* + * Wait to create the pid-file until we've setup the sockets + * and are open for business. + */ + if (pid_file) + write_file(pid_file, "%"PRIuMAX, (uintmax_t) getpid()); + + return service_loop(&socklist); +} + +////////////////////////////////////////////////////////////////// +// This section is executed by both the primary instance and all +// worker instances. So, yes, each child-process re-parses the +// command line argument and re-discovers how it should behave. +////////////////////////////////////////////////////////////////// + +int cmd_main(int argc, const char **argv) +{ + int listen_port = 0; + struct string_list listen_addr = STRING_LIST_INIT_NODUP; + int worker_mode = 0; + int i; + struct auth_module *mod = NULL; + + trace2_cmd_name("test-http-server"); + setup_git_directory_gently(NULL); + + for (i = 1; i < argc; i++) { + const char *arg = argv[i]; + const char *v; + + if (skip_prefix(arg, "--listen=", &v)) { + string_list_append(&listen_addr, xstrdup_tolower(v)); + continue; + } + if (skip_prefix(arg, "--port=", &v)) { + char *end; + unsigned long n; + n = strtoul(v, &end, 0); + if (*v && !*end) { + listen_port = n; + continue; + } + } + if (!strcmp(arg, "--worker")) { + worker_mode = 1; + trace2_cmd_mode("worker"); + continue; + } + if (!strcmp(arg, "--verbose")) { + verbose = 1; + continue; + } + if (skip_prefix(arg, "--timeout=", &v)) { + timeout = atoi(v); + continue; + } + if (skip_prefix(arg, "--init-timeout=", &v)) { + init_timeout = atoi(v); + continue; + } + if (skip_prefix(arg, "--max-connections=", &v)) { + max_connections = atoi(v); + if (max_connections < 0) + max_connections = 0; /* unlimited */ + continue; + } + if (!strcmp(arg, "--reuseaddr")) { + reuseaddr = 1; + continue; + } + if (skip_prefix(arg, "--pid-file=", &v)) { + pid_file = v; + continue; + } + if (skip_prefix(arg, "--allow-anonymous", &v)) { + allow_anonymous = 1; + continue; + } + if (skip_prefix(arg, "--auth=", &v)) { + struct strbuf **p = strbuf_split_str(v, ':', 2); + + if (!p[0]) { + error("invalid argument '%s'", v); + usage(test_http_auth_usage); + } + + // trim trailing ':' + if (p[1]) + strbuf_setlen(p[0], p[0]->len - 1); + + if (get_auth_module(p[0])) { + error("duplicate auth scheme '%s'\n", p[0]->buf); + usage(test_http_auth_usage); + } + + mod = xmalloc(sizeof(struct auth_module)); + mod->scheme = xstrdup(p[0]->buf); + mod->challenge_params = p[1] ? xstrdup(p[1]->buf) : NULL; + mod->tokens = xmalloc(sizeof(struct string_list)); + string_list_init_dup(mod->tokens); + + add_auth_module(mod); + + strbuf_list_free(p); + continue; + } + if (skip_prefix(arg, "--auth-token=", &v)) { + struct strbuf **p = strbuf_split_str(v, ':', 2); + if (!p[0]) { + error("invalid argument '%s'", v); + usage(test_http_auth_usage); + } + + if (!p[1]) { + error("missing token value '%s'\n", v); + usage(test_http_auth_usage); + } + + // trim trailing ':' + strbuf_setlen(p[0], p[0]->len - 1); + + mod = get_auth_module(p[0]); + if (!mod) { + error("auth scheme not defined '%s'\n", p[0]->buf); + usage(test_http_auth_usage); + } + + string_list_append(mod->tokens, p[1]->buf); + strbuf_list_free(p); + continue; + } + + fprintf(stderr, "error: unknown argument '%s'\n", arg); + usage(test_http_auth_usage); + } + + /* avoid splitting a message in the middle */ + setvbuf(stderr, NULL, _IOFBF, 4096); + + if (listen_port == 0) + listen_port = DEFAULT_GIT_PORT; + + /* + * If no --listen= args are given, the setup_named_sock() + * code will use receive a NULL address and set INADDR_ANY. + * This exposes both internal and external interfaces on the + * port. + * + * Disallow that and default to the internal-use-only loopback + * address. + */ + if (!listen_addr.nr) + string_list_append(&listen_addr, "127.0.0.1"); + + /* + * worker_mode is set in our own child process instances + * (that are bound to a connected socket from a client). + */ + if (worker_mode) + return worker(); + + /* + * `cld_argv` is a bit of a clever hack. The top-level instance + * of test-http-server does the normal bind/listen/accept stuff. + * For each incoming socket, the top-level process spawns + * a child instance of test-http-server *WITH* the additional + * `--worker` argument. This causes the child to set `worker_mode` + * and immediately call `worker()` using the connected socket (and + * without the usual need for fork() or threads). + * + * The magic here is made possible because `cld_argv` is static + * and handle() (called by service_loop()) knows about it. + */ + strvec_push(&cld_argv, argv[0]); + strvec_push(&cld_argv, "--worker"); + for (i = 1; i < argc; ++i) + strvec_push(&cld_argv, argv[i]); + + /* + * Setup primary instance to listen for connections. + */ + return serve(&listen_addr, listen_port); +} diff --git a/t/t5556-http-auth.sh b/t/t5556-http-auth.sh new file mode 100755 index 00000000000..43f1791a0fe --- /dev/null +++ b/t/t5556-http-auth.sh @@ -0,0 +1,260 @@ +#!/bin/sh + +test_description='test http auth header and credential helper interop' + +. ./test-lib.sh + +test_set_port GIT_TEST_HTTP_PROTOCOL_PORT + +# Setup a repository +# +REPO_DIR="$(pwd)"/repo + +# Setup some lookback URLs where test-http-server will be listening. +# We will spawn it directly inside the repo directory, so we avoid +# any need to configure directory mappings etc - we only serve this +# repository from the root '/' of the server. +# +HOST_PORT=127.0.0.1:$GIT_TEST_HTTP_PROTOCOL_PORT +ORIGIN_URL=http://$HOST_PORT/ + +# The pid-file is created by test-http-server when it starts. +# The server will shutdown if/when we delete it (this is easier than +# killing it by PID). +# +PID_FILE="$(pwd)"/pid-file.pid +SERVER_LOG="$(pwd)"/OUT.server.log + +PATH="$GIT_BUILD_DIR/t/helper/:$PATH" && export PATH +CREDENTIAL_HELPER="$GIT_BUILD_DIR/t/helper/test-credential-helper-replay.sh" \ + && export CREDENTIAL_HELPER + +test_expect_success 'setup repos' ' + test_create_repo "$REPO_DIR" && + git -C "$REPO_DIR" branch -M main +' + +stop_http_server () { + if ! test -f "$PID_FILE" + then + return 0 + fi + # + # The server will shutdown automatically when we delete the pid-file. + # + rm -f "$PID_FILE" + # + # Give it a few seconds to shutdown (mainly to completely release the + # port before the next test start another instance and it attempts to + # bind to it). + # + for k in 0 1 2 3 4 + do + if grep -q "Starting graceful shutdown" "$SERVER_LOG" + then + return 0 + fi + sleep 1 + done + + echo "stop_http_server: timeout waiting for server shutdown" + return 1 +} + +start_http_server () { + # + # Launch our server into the background in repo_dir. + # + ( + cd "$REPO_DIR" + test-http-server --verbose \ + --listen=127.0.0.1 \ + --port=$GIT_TEST_HTTP_PROTOCOL_PORT \ + --reuseaddr \ + --pid-file="$PID_FILE" \ + "$@" \ + 2>"$SERVER_LOG" & + ) + # + # Give it a few seconds to get started. + # + for k in 0 1 2 3 4 + do + if test -f "$PID_FILE" + then + return 0 + fi + sleep 1 + done + + echo "start_http_server: timeout waiting for server startup" + return 1 +} + +per_test_cleanup () { + stop_http_server && + rm -f OUT.* && + rm -f *.cred +} + +test_expect_success 'http auth anonymous no challenge' ' + test_when_finished "per_test_cleanup" && + start_http_server --allow-anonymous && + + # Attempt to read from a protected repository + git ls-remote $ORIGIN_URL +' + +test_expect_success 'http auth www-auth headers to credential helper bearer valid' ' + test_when_finished "per_test_cleanup" && + start_http_server \ + --auth=bearer:authority=\"id.example.com\"\ q=1\ p=0 \ + --auth=basic:realm=\"example.com\" \ + --auth-token=bearer:secret-token && + + cat >get-expected.cred <<-EOF && + protocol=http + host=$HOST_PORT + wwwauth[]=bearer authority="id.example.com" q=1 p=0 + wwwauth[]=basic realm="example.com" + EOF + + cat >store-expected.cred <<-EOF && + protocol=http + host=$HOST_PORT + username=alice + password=secret-token + authtype=bearer + EOF + + cat >get-response.cred <<-EOF && + protocol=http + host=$HOST_PORT + username=alice + password=secret-token + authtype=bearer + EOF + + git -c credential.helper="$CREDENTIAL_HELPER" ls-remote $ORIGIN_URL && + + test_cmp get-expected.cred get-actual.cred && + test_cmp store-expected.cred store-actual.cred +' + +test_expect_success 'http auth www-auth headers to credential helper basic valid' ' + test_when_finished "per_test_cleanup" && + # base64("alice:secret-passwd") + USERPASS64=YWxpY2U6c2VjcmV0LXBhc3N3ZA== && + export USERPASS64 && + + start_http_server \ + --auth=bearer:authority=\"id.example.com\"\ q=1\ p=0 \ + --auth=basic:realm=\"example.com\" \ + --auth-token=basic:$USERPASS64 && + + cat >get-expected.cred <<-EOF && + protocol=http + host=$HOST_PORT + wwwauth[]=bearer authority="id.example.com" q=1 p=0 + wwwauth[]=basic realm="example.com" + EOF + + cat >store-expected.cred <<-EOF && + protocol=http + host=$HOST_PORT + username=alice + password=secret-passwd + authtype=basic + EOF + + cat >get-response.cred <<-EOF && + protocol=http + host=$HOST_PORT + username=alice + password=secret-passwd + authtype=basic + EOF + + git -c credential.helper="$CREDENTIAL_HELPER" ls-remote $ORIGIN_URL && + + test_cmp get-expected.cred get-actual.cred && + test_cmp store-expected.cred store-actual.cred +' + +test_expect_success 'http auth www-auth headers to credential helper custom scheme' ' + test_when_finished "per_test_cleanup" && + start_http_server \ + --auth=foobar:alg=test\ widget=1 \ + --auth=bearer:authority=\"id.example.com\"\ q=1\ p=0 \ + --auth=basic:realm=\"example.com\" \ + --auth-token=foobar:SECRET-FOOBAR-VALUE && + + cat >get-expected.cred <<-EOF && + protocol=http + host=$HOST_PORT + wwwauth[]=foobar alg=test widget=1 + wwwauth[]=bearer authority="id.example.com" q=1 p=0 + wwwauth[]=basic realm="example.com" + EOF + + cat >store-expected.cred <<-EOF && + protocol=http + host=$HOST_PORT + username=alice + password=SECRET-FOOBAR-VALUE + authtype=foobar + EOF + + cat >get-response.cred <<-EOF && + protocol=http + host=$HOST_PORT + username=alice + password=SECRET-FOOBAR-VALUE + authtype=foobar + EOF + + git -c credential.helper="$CREDENTIAL_HELPER" ls-remote $ORIGIN_URL && + + test_cmp get-expected.cred get-actual.cred && + test_cmp store-expected.cred store-actual.cred +' + +test_expect_success 'http auth www-auth headers to credential helper invalid' ' + test_when_finished "per_test_cleanup" && + start_http_server \ + --auth=bearer:authority=\"id.example.com\"\ q=1\ p=0 \ + --auth=basic:realm=\"example.com\" \ + --auth-token=bearer:secret-token && + + cat >get-expected.cred <<-EOF && + protocol=http + host=$HOST_PORT + wwwauth[]=bearer authority="id.example.com" q=1 p=0 + wwwauth[]=basic realm="example.com" + EOF + + cat >erase-expected.cred <<-EOF && + protocol=http + host=$HOST_PORT + username=alice + password=invalid-token + authtype=bearer + wwwauth[]=bearer authority="id.example.com" q=1 p=0 + wwwauth[]=basic realm="example.com" + EOF + + cat >get-response.cred <<-EOF && + protocol=http + host=$HOST_PORT + username=alice + password=invalid-token + authtype=bearer + EOF + + test_must_fail git -c credential.helper="$CREDENTIAL_HELPER" ls-remote $ORIGIN_URL && + + test_cmp get-expected.cred get-actual.cred && + test_cmp erase-expected.cred erase-actual.cred +' + +test_done