From patchwork Wed Jan 18 03:30:06 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew John Cheetham X-Patchwork-Id: 13105483 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 07406C00A5A for ; Wed, 18 Jan 2023 03:30:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229766AbjARDae (ORCPT ); Tue, 17 Jan 2023 22:30:34 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51638 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229728AbjARDaY (ORCPT ); Tue, 17 Jan 2023 22:30:24 -0500 Received: from mail-wr1-x432.google.com (mail-wr1-x432.google.com [IPv6:2a00:1450:4864:20::432]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8EA8D47EEE for ; Tue, 17 Jan 2023 19:30:22 -0800 (PST) Received: by mail-wr1-x432.google.com with SMTP id d2so12483669wrp.8 for ; Tue, 17 Jan 2023 19:30:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:from:to:cc:subject:date :message-id:reply-to; bh=qZfvZ8RTcDBAofAevq0+3MPqq1v9AIkhChP91QQElkQ=; b=K+lwu0QXnjKJIgsu5T4dzfdw8kPk0pamsTZRVA4ML69OfUwXNYPQ4B0FjiNZuE3XXO 5LH6W0BFGOch82yzZa+INnO02B6RBB9nE2kezr7kEAdTNKHZ9lFOrrdE7/jGJA1ajKTk Xjj5/inR9R43E3UvOs/Gc655p4sC8MZ2NYOLIEcaPnrUqcP9RdgC9r0PL1qru783tYha vLnGkAqdb9XdU7V92MVUvx1szhGr+1QMpPn13lBM68CEKgjZSrpRQRFKqASNWNtzxwvX kT23H0J7DCt0pP1FidPFUDIHfRkmCV1GBTWUG2P7WamETc6UEOXMEr2cfs33HzE8YLtW +cUg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=qZfvZ8RTcDBAofAevq0+3MPqq1v9AIkhChP91QQElkQ=; b=JUSgNy50HshSqKHJUUZJujq+sMnTqu0wIBjnTXyuoO+lDThLWjtFTV+fUJVzlsbMbk o0zC2cqP3xWgVh9R6Gbm3olecS5+3Kk5q3JWmn/d4Sn2dJ3tBtdyffs49M9DpC4ZuDtz GnBo9LHdM5op71SVDKjJOXOsM4uXLX66Jlv9sp+eFX/aQdV2mtVyd6vjyJexd+e0JCX0 NsnuhZPWREYwFUH7dtlP59RB6INSSYA+H0Fi3ClpS0EYH18flYoaqHHuyYbI9p/IE1o5 V0PbYAcovjhaQClIv0QsrYzYuh5HfzxwYQew4NLExwseMOLz7dwOGOruOmngDg8nQOBV McMg== X-Gm-Message-State: AFqh2kobIRsxViShrKPrqgIlWA2DCwlReOMn62gA5HY/ht0V3uwiPo4q d9gFnwfDIe0oAwvLaCsLQmYdNoTzuIs= X-Google-Smtp-Source: AMrXdXsZuVYjs/idQLSS19MEWs211vO/yPWK2octYpCVr0a3NeBStl2d/SicJ3mfuLYtpEaTkPt+mA== X-Received: by 2002:adf:f18e:0:b0:2bd:e8bd:79ce with SMTP id h14-20020adff18e000000b002bde8bd79cemr4521850wro.20.1674012620813; Tue, 17 Jan 2023 19:30:20 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id m15-20020adffe4f000000b002bdd155ca4dsm15868774wrs.48.2023.01.17.19.30.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 17 Jan 2023 19:30:20 -0800 (PST) Message-Id: <74b0de14185120c9d53d7470e59f57fa20a1927f.1674012618.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Wed, 18 Jan 2023 03:30:06 +0000 Subject: [PATCH v6 01/12] daemon: libify socket setup and option functions Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Derrick Stolee , Lessley Dennington , Matthew John Cheetham , M Hickford , Jeff Hostetler , Glen Choo , Victoria Dye , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsA==?= Bjarmason , Matthew John Cheetham , Matthew John Cheetham Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Matthew John Cheetham From: Matthew John Cheetham Extract functions for setting up listening sockets and keep-alive options from `daemon.c` to new `daemon-utils.{c,h}` files. Remove direct dependencies on global state by inlining the behaviour at the callsites for all libified functions. Signed-off-by: Matthew John Cheetham --- Makefile | 1 + daemon-utils.c | 209 +++++++++++++++++++++++++++++++++++++++++++++++ daemon-utils.h | 23 ++++++ daemon.c | 214 +------------------------------------------------ 4 files changed, 237 insertions(+), 210 deletions(-) create mode 100644 daemon-utils.c create mode 100644 daemon-utils.h diff --git a/Makefile b/Makefile index b258fdbed86..2654094dbb5 100644 --- a/Makefile +++ b/Makefile @@ -1003,6 +1003,7 @@ LIB_OBJS += credential.o LIB_OBJS += csum-file.o LIB_OBJS += ctype.o LIB_OBJS += date.o +LIB_OBJS += daemon-utils.o LIB_OBJS += decorate.o LIB_OBJS += delta-islands.o LIB_OBJS += diagnose.o diff --git a/daemon-utils.c b/daemon-utils.c new file mode 100644 index 00000000000..b96b55962db --- /dev/null +++ b/daemon-utils.c @@ -0,0 +1,209 @@ +#include "cache.h" +#include "daemon-utils.h" + +void set_keep_alive(int sockfd, log_fn logerror) +{ + int ka = 1; + + if (setsockopt(sockfd, SOL_SOCKET, SO_KEEPALIVE, &ka, sizeof(ka)) < 0) { + if (errno != ENOTSOCK) + logerror("unable to set SO_KEEPALIVE on socket: %s", + strerror(errno)); + } +} + +static int set_reuse_addr(int sockfd) +{ + int on = 1; + + return setsockopt(sockfd, SOL_SOCKET, SO_REUSEADDR, + &on, sizeof(on)); +} + +static const char *ip2str(int family, struct sockaddr *sin, socklen_t len) +{ +#ifdef NO_IPV6 + static char ip[INET_ADDRSTRLEN]; +#else + static char ip[INET6_ADDRSTRLEN]; +#endif + + switch (family) { +#ifndef NO_IPV6 + case AF_INET6: + inet_ntop(family, &((struct sockaddr_in6*)sin)->sin6_addr, ip, len); + break; +#endif + case AF_INET: + inet_ntop(family, &((struct sockaddr_in*)sin)->sin_addr, ip, len); + break; + default: + xsnprintf(ip, sizeof(ip), ""); + } + return ip; +} + +#ifndef NO_IPV6 + +static int setup_named_sock(char *listen_addr, int listen_port, + struct socketlist *socklist, int reuseaddr, + log_fn logerror) +{ + int socknum = 0; + char pbuf[NI_MAXSERV]; + struct addrinfo hints, *ai0, *ai; + int gai; + long flags; + + xsnprintf(pbuf, sizeof(pbuf), "%d", listen_port); + memset(&hints, 0, sizeof(hints)); + hints.ai_family = AF_UNSPEC; + hints.ai_socktype = SOCK_STREAM; + hints.ai_protocol = IPPROTO_TCP; + hints.ai_flags = AI_PASSIVE; + + gai = getaddrinfo(listen_addr, pbuf, &hints, &ai0); + if (gai) { + logerror("getaddrinfo() for %s failed: %s", listen_addr, gai_strerror(gai)); + return 0; + } + + for (ai = ai0; ai; ai = ai->ai_next) { + int sockfd; + + sockfd = socket(ai->ai_family, ai->ai_socktype, ai->ai_protocol); + if (sockfd < 0) + continue; + if (sockfd >= FD_SETSIZE) { + logerror("Socket descriptor too large"); + close(sockfd); + continue; + } + +#ifdef IPV6_V6ONLY + if (ai->ai_family == AF_INET6) { + int on = 1; + setsockopt(sockfd, IPPROTO_IPV6, IPV6_V6ONLY, + &on, sizeof(on)); + /* Note: error is not fatal */ + } +#endif + + if (reuseaddr && set_reuse_addr(sockfd)) { + logerror("Could not set SO_REUSEADDR: %s", strerror(errno)); + close(sockfd); + continue; + } + + set_keep_alive(sockfd, logerror); + + if (bind(sockfd, ai->ai_addr, ai->ai_addrlen) < 0) { + logerror("Could not bind to %s: %s", + ip2str(ai->ai_family, ai->ai_addr, ai->ai_addrlen), + strerror(errno)); + close(sockfd); + continue; /* not fatal */ + } + if (listen(sockfd, 5) < 0) { + logerror("Could not listen to %s: %s", + ip2str(ai->ai_family, ai->ai_addr, ai->ai_addrlen), + strerror(errno)); + close(sockfd); + continue; /* not fatal */ + } + + flags = fcntl(sockfd, F_GETFD, 0); + if (flags >= 0) + fcntl(sockfd, F_SETFD, flags | FD_CLOEXEC); + + ALLOC_GROW(socklist->list, socklist->nr + 1, socklist->alloc); + socklist->list[socklist->nr++] = sockfd; + socknum++; + } + + freeaddrinfo(ai0); + + return socknum; +} + +#else /* NO_IPV6 */ + +static int setup_named_sock(char *listen_addr, int listen_port, + struct socketlist *socklist, int reuseaddr, + log_fn logerror) +{ + struct sockaddr_in sin; + int sockfd; + long flags; + + memset(&sin, 0, sizeof sin); + sin.sin_family = AF_INET; + sin.sin_port = htons(listen_port); + + if (listen_addr) { + /* Well, host better be an IP address here. */ + if (inet_pton(AF_INET, listen_addr, &sin.sin_addr.s_addr) <= 0) + return 0; + } else { + sin.sin_addr.s_addr = htonl(INADDR_ANY); + } + + sockfd = socket(AF_INET, SOCK_STREAM, 0); + if (sockfd < 0) + return 0; + + if (reuseaddr && set_reuse_addr(sockfd)) { + logerror("Could not set SO_REUSEADDR: %s", strerror(errno)); + close(sockfd); + return 0; + } + + set_keep_alive(sockfd, logerror); + + if ( bind(sockfd, (struct sockaddr *)&sin, sizeof sin) < 0 ) { + logerror("Could not bind to %s: %s", + ip2str(AF_INET, (struct sockaddr *)&sin, sizeof(sin)), + strerror(errno)); + close(sockfd); + return 0; + } + + if (listen(sockfd, 5) < 0) { + logerror("Could not listen to %s: %s", + ip2str(AF_INET, (struct sockaddr *)&sin, sizeof(sin)), + strerror(errno)); + close(sockfd); + return 0; + } + + flags = fcntl(sockfd, F_GETFD, 0); + if (flags >= 0) + fcntl(sockfd, F_SETFD, flags | FD_CLOEXEC); + + ALLOC_GROW(socklist->list, socklist->nr + 1, socklist->alloc); + socklist->list[socklist->nr++] = sockfd; + return 1; +} + +#endif + +void socksetup(struct string_list *listen_addr, int listen_port, + struct socketlist *socklist, int reuseaddr, + log_fn logerror) +{ + if (!listen_addr->nr) + setup_named_sock(NULL, listen_port, socklist, reuseaddr, + logerror); + else { + int i, socknum; + for (i = 0; i < listen_addr->nr; i++) { + socknum = setup_named_sock(listen_addr->items[i].string, + listen_port, socklist, reuseaddr, + logerror); + + if (socknum == 0) + logerror("unable to allocate any listen sockets for host %s on port %u", + listen_addr->items[i].string, listen_port); + } + } +} diff --git a/daemon-utils.h b/daemon-utils.h new file mode 100644 index 00000000000..6710a2a6dc0 --- /dev/null +++ b/daemon-utils.h @@ -0,0 +1,23 @@ +#ifndef DAEMON_UTILS_H +#define DAEMON_UTILS_H + +#include "git-compat-util.h" +#include "string-list.h" + +typedef void (*log_fn)(const char *msg, ...); + +struct socketlist { + int *list; + size_t nr; + size_t alloc; +}; + +/* Enable sending of keep-alive messages on the socket. */ +void set_keep_alive(int sockfd, log_fn logerror); + +/* Setup a number of sockets to listen on the provided addresses. */ +void socksetup(struct string_list *listen_addr, int listen_port, + struct socketlist *socklist, int reuseaddr, + log_fn logerror); + +#endif diff --git a/daemon.c b/daemon.c index 0ae7d12b5c1..1ed4e705680 100644 --- a/daemon.c +++ b/daemon.c @@ -1,9 +1,9 @@ #include "cache.h" #include "config.h" +#include "daemon-utils.h" #include "pkt-line.h" #include "run-command.h" #include "strbuf.h" -#include "string-list.h" #ifdef NO_INITGROUPS #define initgroups(x, y) (0) /* nothing */ @@ -737,17 +737,6 @@ static void hostinfo_clear(struct hostinfo *hi) strbuf_release(&hi->tcp_port); } -static void set_keep_alive(int sockfd) -{ - int ka = 1; - - if (setsockopt(sockfd, SOL_SOCKET, SO_KEEPALIVE, &ka, sizeof(ka)) < 0) { - if (errno != ENOTSOCK) - logerror("unable to set SO_KEEPALIVE on socket: %s", - strerror(errno)); - } -} - static int execute(void) { char *line = packet_buffer; @@ -759,7 +748,7 @@ static int execute(void) if (addr) loginfo("Connection from %s:%s", addr, port); - set_keep_alive(0); + set_keep_alive(0, logerror); alarm(init_timeout ? init_timeout : timeout); pktlen = packet_read(0, packet_buffer, sizeof(packet_buffer), 0); alarm(0); @@ -938,202 +927,6 @@ static void child_handler(int signo) signal(SIGCHLD, child_handler); } -static int set_reuse_addr(int sockfd) -{ - int on = 1; - - if (!reuseaddr) - return 0; - return setsockopt(sockfd, SOL_SOCKET, SO_REUSEADDR, - &on, sizeof(on)); -} - -struct socketlist { - int *list; - size_t nr; - size_t alloc; -}; - -static const char *ip2str(int family, struct sockaddr *sin, socklen_t len) -{ -#ifdef NO_IPV6 - static char ip[INET_ADDRSTRLEN]; -#else - static char ip[INET6_ADDRSTRLEN]; -#endif - - switch (family) { -#ifndef NO_IPV6 - case AF_INET6: - inet_ntop(family, &((struct sockaddr_in6*)sin)->sin6_addr, ip, len); - break; -#endif - case AF_INET: - inet_ntop(family, &((struct sockaddr_in*)sin)->sin_addr, ip, len); - break; - default: - xsnprintf(ip, sizeof(ip), ""); - } - return ip; -} - -#ifndef NO_IPV6 - -static int setup_named_sock(char *listen_addr, int listen_port, struct socketlist *socklist) -{ - int socknum = 0; - char pbuf[NI_MAXSERV]; - struct addrinfo hints, *ai0, *ai; - int gai; - long flags; - - xsnprintf(pbuf, sizeof(pbuf), "%d", listen_port); - memset(&hints, 0, sizeof(hints)); - hints.ai_family = AF_UNSPEC; - hints.ai_socktype = SOCK_STREAM; - hints.ai_protocol = IPPROTO_TCP; - hints.ai_flags = AI_PASSIVE; - - gai = getaddrinfo(listen_addr, pbuf, &hints, &ai0); - if (gai) { - logerror("getaddrinfo() for %s failed: %s", listen_addr, gai_strerror(gai)); - return 0; - } - - for (ai = ai0; ai; ai = ai->ai_next) { - int sockfd; - - sockfd = socket(ai->ai_family, ai->ai_socktype, ai->ai_protocol); - if (sockfd < 0) - continue; - if (sockfd >= FD_SETSIZE) { - logerror("Socket descriptor too large"); - close(sockfd); - continue; - } - -#ifdef IPV6_V6ONLY - if (ai->ai_family == AF_INET6) { - int on = 1; - setsockopt(sockfd, IPPROTO_IPV6, IPV6_V6ONLY, - &on, sizeof(on)); - /* Note: error is not fatal */ - } -#endif - - if (set_reuse_addr(sockfd)) { - logerror("Could not set SO_REUSEADDR: %s", strerror(errno)); - close(sockfd); - continue; - } - - set_keep_alive(sockfd); - - if (bind(sockfd, ai->ai_addr, ai->ai_addrlen) < 0) { - logerror("Could not bind to %s: %s", - ip2str(ai->ai_family, ai->ai_addr, ai->ai_addrlen), - strerror(errno)); - close(sockfd); - continue; /* not fatal */ - } - if (listen(sockfd, 5) < 0) { - logerror("Could not listen to %s: %s", - ip2str(ai->ai_family, ai->ai_addr, ai->ai_addrlen), - strerror(errno)); - close(sockfd); - continue; /* not fatal */ - } - - flags = fcntl(sockfd, F_GETFD, 0); - if (flags >= 0) - fcntl(sockfd, F_SETFD, flags | FD_CLOEXEC); - - ALLOC_GROW(socklist->list, socklist->nr + 1, socklist->alloc); - socklist->list[socklist->nr++] = sockfd; - socknum++; - } - - freeaddrinfo(ai0); - - return socknum; -} - -#else /* NO_IPV6 */ - -static int setup_named_sock(char *listen_addr, int listen_port, struct socketlist *socklist) -{ - struct sockaddr_in sin; - int sockfd; - long flags; - - memset(&sin, 0, sizeof sin); - sin.sin_family = AF_INET; - sin.sin_port = htons(listen_port); - - if (listen_addr) { - /* Well, host better be an IP address here. */ - if (inet_pton(AF_INET, listen_addr, &sin.sin_addr.s_addr) <= 0) - return 0; - } else { - sin.sin_addr.s_addr = htonl(INADDR_ANY); - } - - sockfd = socket(AF_INET, SOCK_STREAM, 0); - if (sockfd < 0) - return 0; - - if (set_reuse_addr(sockfd)) { - logerror("Could not set SO_REUSEADDR: %s", strerror(errno)); - close(sockfd); - return 0; - } - - set_keep_alive(sockfd); - - if ( bind(sockfd, (struct sockaddr *)&sin, sizeof sin) < 0 ) { - logerror("Could not bind to %s: %s", - ip2str(AF_INET, (struct sockaddr *)&sin, sizeof(sin)), - strerror(errno)); - close(sockfd); - return 0; - } - - if (listen(sockfd, 5) < 0) { - logerror("Could not listen to %s: %s", - ip2str(AF_INET, (struct sockaddr *)&sin, sizeof(sin)), - strerror(errno)); - close(sockfd); - return 0; - } - - flags = fcntl(sockfd, F_GETFD, 0); - if (flags >= 0) - fcntl(sockfd, F_SETFD, flags | FD_CLOEXEC); - - ALLOC_GROW(socklist->list, socklist->nr + 1, socklist->alloc); - socklist->list[socklist->nr++] = sockfd; - return 1; -} - -#endif - -static void socksetup(struct string_list *listen_addr, int listen_port, struct socketlist *socklist) -{ - if (!listen_addr->nr) - setup_named_sock(NULL, listen_port, socklist); - else { - int i, socknum; - for (i = 0; i < listen_addr->nr; i++) { - socknum = setup_named_sock(listen_addr->items[i].string, - listen_port, socklist); - - if (socknum == 0) - logerror("unable to allocate any listen sockets for host %s on port %u", - listen_addr->items[i].string, listen_port); - } - } -} - static int service_loop(struct socketlist *socklist) { struct pollfd *pfd; @@ -1246,7 +1039,8 @@ static int serve(struct string_list *listen_addr, int listen_port, { struct socketlist socklist = { NULL, 0, 0 }; - socksetup(listen_addr, listen_port, &socklist); + socksetup(listen_addr, listen_port, &socklist, reuseaddr, + logerror); if (socklist.nr == 0) die("unable to allocate any listen sockets on port %u", listen_port); From patchwork Wed Jan 18 03:30:07 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew John Cheetham X-Patchwork-Id: 13105484 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 300F7C00A5A for ; Wed, 18 Jan 2023 03:30:43 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229804AbjARDaj (ORCPT ); Tue, 17 Jan 2023 22:30:39 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51656 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229729AbjARDaZ (ORCPT ); Tue, 17 Jan 2023 22:30:25 -0500 Received: from mail-wr1-x42f.google.com (mail-wr1-x42f.google.com [IPv6:2a00:1450:4864:20::42f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A4E015087C for ; Tue, 17 Jan 2023 19:30:23 -0800 (PST) Received: by mail-wr1-x42f.google.com with SMTP id e3so23384631wru.13 for ; Tue, 17 Jan 2023 19:30:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:from:to:cc:subject:date :message-id:reply-to; bh=8QfNUqa5/m+ZSldR0jAxuOmtcZQE1zGK1yQCte9bfm4=; b=IbB11nuBw+q+ahHnH7MgLfBVxWymxbbd2DSh4GxRbU8lsFDq6YUzeiRQUolSwc9Mv8 o+a6fZPXPQ1ly+Y/Mu2ZkQSbcl5oFrR6F48nsHhud83HNobdmdqlz0Xi4CJG7BYEf0K4 3pYsnhexgB3aKsVbIBKR8WPp3DOEpqf8YvfyM5EO7pFQ03rCwyjlCgKtuDXn2gbFOtt9 p7vW+Tfa0MNIg3cKDHY3BZvUrcLUEVSq2/cWKx5Av3oAFT8SPvbJp0ZZSpvB1CXwsgqi Qa/dKQl4JSinG52JKX7C3GDgGf7XrzufkrvwgowtFJH6bE9ls43o8VhNTrPiW62vLViP UIFw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=8QfNUqa5/m+ZSldR0jAxuOmtcZQE1zGK1yQCte9bfm4=; b=1WVDnsCVKshSu/MAPt2wFAnPcNSW7uSukkFrn4Oz77Q4NVqrl9PdjTfbwuunaa6/vq rrabBDPyO2FyaP2ADNeDob18HBTIf6d3TRwNapFUN1+dtWTDtK0/rsX0kqLG3Euh4eT6 UbueXv7Lka5o5cawQv0ic6zwqKbLJT/LuZ0ypspsDHgOE6JCzUQI1SthtyILeW1Dt949 iNfud0RIKvbXKroETW4KuDZqSBKvhwWTrrcmPc3jXNvBk8mvKuuJl8q8yxWVJh0ST/9I 5vb97J6DY2Mj1bpOABRf/o1RDUiJ3U1VmgxH9TgvdCe1ewrp3dXgUC3mzHugxOwSyrGR qr1g== X-Gm-Message-State: AFqh2koksPf/uTjdwVpTnk2nEZE2pO1S6NpGXfwFqAylZ61bz3VZIUXq T8kG7ZgVG/7RMZ1iJtaXZC/5titxlCw= X-Google-Smtp-Source: AMrXdXvGCi7qeuEgmOCGqAS4oM+MG7YqFOs716gQ3WWZMkARdbmY1bI4jI8xU6otpK+nkPl9S0UOfw== X-Received: by 2002:a5d:5c07:0:b0:2bd:fa1d:5291 with SMTP id cc7-20020a5d5c07000000b002bdfa1d5291mr5214541wrb.67.1674012621925; Tue, 17 Jan 2023 19:30:21 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id j15-20020adfd20f000000b002be2a4b521fsm1694808wrh.45.2023.01.17.19.30.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 17 Jan 2023 19:30:21 -0800 (PST) Message-Id: In-Reply-To: References: Date: Wed, 18 Jan 2023 03:30:07 +0000 Subject: [PATCH v6 02/12] daemon: libify child process handling functions Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Derrick Stolee , Lessley Dennington , Matthew John Cheetham , M Hickford , Jeff Hostetler , Glen Choo , Victoria Dye , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsA==?= Bjarmason , Matthew John Cheetham , Matthew John Cheetham Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Matthew John Cheetham From: Matthew John Cheetham Extract functions and structures for managing child processes started from the parent daemon-like process from `daemon.c` to the new shared `daemon-utils.{c,h}` files. One minor functional change is introduced to `check_dead_children()` where the logging of a dead/disconnected child is now optional. With the 'libification' of these functions we extract the call to `loginfo` to a call to a function pointer, and guard the log message creation and logging behind a `NULL` check. Callers can now skip logging by passing `NULL` as the `log_fn loginfo` argument. The behaviour of callers in `daemon.c` remains the same (save one extra NULL check) however as a pointer to `loginfo` is always passed. Signed-off-by: Matthew John Cheetham --- daemon-utils.c | 77 ++++++++++++++++++++++++++++++++++++++++++ daemon-utils.h | 32 ++++++++++++++++++ daemon.c | 92 +++----------------------------------------------- 3 files changed, 114 insertions(+), 87 deletions(-) diff --git a/daemon-utils.c b/daemon-utils.c index b96b55962db..8506664b440 100644 --- a/daemon-utils.c +++ b/daemon-utils.c @@ -207,3 +207,80 @@ void socksetup(struct string_list *listen_addr, int listen_port, } } } + +static int addrcmp(const struct sockaddr_storage *s1, + const struct sockaddr_storage *s2) +{ + const struct sockaddr *sa1 = (const struct sockaddr*) s1; + const struct sockaddr *sa2 = (const struct sockaddr*) s2; + + if (sa1->sa_family != sa2->sa_family) + return sa1->sa_family - sa2->sa_family; + if (sa1->sa_family == AF_INET) + return memcmp(&((struct sockaddr_in *)s1)->sin_addr, + &((struct sockaddr_in *)s2)->sin_addr, + sizeof(struct in_addr)); +#ifndef NO_IPV6 + if (sa1->sa_family == AF_INET6) + return memcmp(&((struct sockaddr_in6 *)s1)->sin6_addr, + &((struct sockaddr_in6 *)s2)->sin6_addr, + sizeof(struct in6_addr)); +#endif + return 0; +} + +void add_child(struct child_process *cld, struct sockaddr *addr, socklen_t addrlen, + struct child *firstborn , unsigned int *live_children) +{ + struct child *newborn, **cradle; + + CALLOC_ARRAY(newborn, 1); + (*live_children)++; + memcpy(&newborn->cld, cld, sizeof(*cld)); + memcpy(&newborn->address, addr, addrlen); + for (cradle = &firstborn; *cradle; cradle = &(*cradle)->next) + if (!addrcmp(&(*cradle)->address, &newborn->address)) + break; + newborn->next = *cradle; + *cradle = newborn; +} + +void kill_some_child(struct child *firstborn) +{ + const struct child *blanket, *next; + + if (!(blanket = firstborn)) + return; + + for (; (next = blanket->next); blanket = next) + if (!addrcmp(&blanket->address, &next->address)) { + kill(blanket->cld.pid, SIGTERM); + break; + } +} + +void check_dead_children(struct child *firstborn, unsigned int *live_children, + log_fn loginfo) +{ + int status; + pid_t pid; + + struct child **cradle, *blanket; + for (cradle = &firstborn; (blanket = *cradle);) + if ((pid = waitpid(blanket->cld.pid, &status, WNOHANG)) > 1) { + if (loginfo) { + const char *dead = ""; + if (status) + dead = " (with error)"; + loginfo("[%"PRIuMAX"] Disconnected%s", + (uintmax_t)pid, dead); + } + + /* remove the child */ + *cradle = blanket->next; + (*live_children)--; + child_process_clear(&blanket->cld); + free(blanket); + } else + cradle = &blanket->next; +} diff --git a/daemon-utils.h b/daemon-utils.h index 6710a2a6dc0..97e5cae20b8 100644 --- a/daemon-utils.h +++ b/daemon-utils.h @@ -2,6 +2,7 @@ #define DAEMON_UTILS_H #include "git-compat-util.h" +#include "run-command.h" #include "string-list.h" typedef void (*log_fn)(const char *msg, ...); @@ -20,4 +21,35 @@ void socksetup(struct string_list *listen_addr, int listen_port, struct socketlist *socklist, int reuseaddr, log_fn logerror); +struct child { + struct child *next; + struct child_process cld; + struct sockaddr_storage address; +}; + +/* + * Add the child_process to the set of children and increment the number of + * live children. + */ +void add_child(struct child_process *cld, struct sockaddr *addr, socklen_t addrlen, + struct child *firstborn, unsigned int *live_children); + +/* + * Kill the newest connection from a duplicate IP. + * + * This function should be called if the number of connections grows + * past the maximum number of allowed connections. + */ +void kill_some_child(struct child *firstborn); + +/* + * Check for children that have disconnected and remove them from the + * active set, decrementing the number of live children. + * + * Optionally log the child PID that disconnected by passing a loginfo + * function. + */ +void check_dead_children(struct child *firstborn, unsigned int *live_children, + log_fn loginfo); + #endif diff --git a/daemon.c b/daemon.c index 1ed4e705680..ec3b407ecbc 100644 --- a/daemon.c +++ b/daemon.c @@ -785,93 +785,11 @@ static int execute(void) return -1; } -static int addrcmp(const struct sockaddr_storage *s1, - const struct sockaddr_storage *s2) -{ - const struct sockaddr *sa1 = (const struct sockaddr*) s1; - const struct sockaddr *sa2 = (const struct sockaddr*) s2; - - if (sa1->sa_family != sa2->sa_family) - return sa1->sa_family - sa2->sa_family; - if (sa1->sa_family == AF_INET) - return memcmp(&((struct sockaddr_in *)s1)->sin_addr, - &((struct sockaddr_in *)s2)->sin_addr, - sizeof(struct in_addr)); -#ifndef NO_IPV6 - if (sa1->sa_family == AF_INET6) - return memcmp(&((struct sockaddr_in6 *)s1)->sin6_addr, - &((struct sockaddr_in6 *)s2)->sin6_addr, - sizeof(struct in6_addr)); -#endif - return 0; -} - static int max_connections = 32; static unsigned int live_children; -static struct child { - struct child *next; - struct child_process cld; - struct sockaddr_storage address; -} *firstborn; - -static void add_child(struct child_process *cld, struct sockaddr *addr, socklen_t addrlen) -{ - struct child *newborn, **cradle; - - CALLOC_ARRAY(newborn, 1); - live_children++; - memcpy(&newborn->cld, cld, sizeof(*cld)); - memcpy(&newborn->address, addr, addrlen); - for (cradle = &firstborn; *cradle; cradle = &(*cradle)->next) - if (!addrcmp(&(*cradle)->address, &newborn->address)) - break; - newborn->next = *cradle; - *cradle = newborn; -} - -/* - * This gets called if the number of connections grows - * past "max_connections". - * - * We kill the newest connection from a duplicate IP. - */ -static void kill_some_child(void) -{ - const struct child *blanket, *next; - - if (!(blanket = firstborn)) - return; - - for (; (next = blanket->next); blanket = next) - if (!addrcmp(&blanket->address, &next->address)) { - kill(blanket->cld.pid, SIGTERM); - break; - } -} - -static void check_dead_children(void) -{ - int status; - pid_t pid; - - struct child **cradle, *blanket; - for (cradle = &firstborn; (blanket = *cradle);) - if ((pid = waitpid(blanket->cld.pid, &status, WNOHANG)) > 1) { - const char *dead = ""; - if (status) - dead = " (with error)"; - loginfo("[%"PRIuMAX"] Disconnected%s", (uintmax_t)pid, dead); - - /* remove the child */ - *cradle = blanket->next; - live_children--; - child_process_clear(&blanket->cld); - free(blanket); - } else - cradle = &blanket->next; -} +static struct child *firstborn; static struct strvec cld_argv = STRVEC_INIT; static void handle(int incoming, struct sockaddr *addr, socklen_t addrlen) @@ -879,9 +797,9 @@ static void handle(int incoming, struct sockaddr *addr, socklen_t addrlen) struct child_process cld = CHILD_PROCESS_INIT; if (max_connections && live_children >= max_connections) { - kill_some_child(); + kill_some_child(firstborn); sleep(1); /* give it some time to die */ - check_dead_children(); + check_dead_children(firstborn, &live_children, loginfo); if (live_children >= max_connections) { close(incoming); logerror("Too many children, dropping connection"); @@ -914,7 +832,7 @@ static void handle(int incoming, struct sockaddr *addr, socklen_t addrlen) if (start_command(&cld)) logerror("unable to fork"); else - add_child(&cld, addr, addrlen); + add_child(&cld, addr, addrlen, firstborn, &live_children); } static void child_handler(int signo) @@ -944,7 +862,7 @@ static int service_loop(struct socketlist *socklist) for (;;) { int i; - check_dead_children(); + check_dead_children(firstborn, &live_children, loginfo); if (poll(pfd, socklist->nr, -1) < 0) { if (errno != EINTR) { From patchwork Wed Jan 18 03:30:08 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew John Cheetham X-Patchwork-Id: 13105485 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9F1ACC00A5A for ; Wed, 18 Jan 2023 03:30:48 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229729AbjARDao (ORCPT ); Tue, 17 Jan 2023 22:30:44 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51684 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229732AbjARDaZ (ORCPT ); Tue, 17 Jan 2023 22:30:25 -0500 Received: from mail-wr1-x42a.google.com (mail-wr1-x42a.google.com [IPv6:2a00:1450:4864:20::42a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 89B5353558 for ; Tue, 17 Jan 2023 19:30:24 -0800 (PST) Received: by mail-wr1-x42a.google.com with SMTP id d2so12483713wrp.8 for ; Tue, 17 Jan 2023 19:30:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:from:to:cc:subject:date :message-id:reply-to; bh=KsvXxRc41yRRXoCeuMp+jDk6Yf1HTqCPl2KPj4S0kA8=; b=RJ2xkSF4elsKtjriaTR+wRhtc64jpk2bMXyh7Mj40T/xJ1qaozhdvCqjPbUNcIGWVv emZt8dmVwUnu0XFip5i+FSC6pK0qTyjGKUf38vMr7HQ7hJ5mFPKgGGeOpaOufq9ea1ql BXoFPeccP2xJ2rZ4sZaKxHBMrhP3jMLpjDdmKRlWv1/szqDkxF888J4TZicnIquknvgJ 3RFP0U7CaGfZo0T415QsyUqMFg1KGEywfBlcZ/Fei8w1lEu2A3MStQSfk/93DbY86+f8 +ccKOiR9jDNmtgv8U7qofwiS2BOHijJSW9rsBc1AWOjYMtMt9hfOPFbrvUvkRBRgjSaL CZbQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=KsvXxRc41yRRXoCeuMp+jDk6Yf1HTqCPl2KPj4S0kA8=; b=6/lXSTxWkWgrz8EU0OZ68GKVclscux9gGJF/REpAsd+PLcV5o+kpOjQLeftE/ZOzQH xxsgRIOnyhXxgQoZHK+t2BWToSl74kQ8hlxfcol4YqUrMJvGauzjpTmSHPBRtCSqQqTm v9/5WsNVSHT7nihaS/JuBB75E/I/bVJLN33fZXo80M0IoTI0l4ET4AUcfGHMK2GyCmYj uzZuGwSxs4dpKq70WBR0K3wAyYGCHju1hYFOVKGSwQDE2e2eARPhBRtZAPdsjEwNXNlK 1qNFCqNibcfCHtI7+6uAY9nH8O9mHU4rk9FlX4Pqbf9Ob/cBmxjW8XQ1L/woKfmjdtub BH/g== X-Gm-Message-State: AFqh2kp7Q+nFefU+7czhI2Iza0fP/8CsydEqhS3Mz016rGskDkmTBq3i YGei7X7KzdnK8Va4Ro7WVO9yhy21fAg= X-Google-Smtp-Source: AMrXdXuMexIxbiXdqKqUW4zfRfSbPCMzRkD3XwLy91gAt8lGjnvVuwr0+xgKVCwEXmkWsI7RiITPaw== X-Received: by 2002:a5d:5642:0:b0:2bb:d412:9866 with SMTP id j2-20020a5d5642000000b002bbd4129866mr4571577wrw.25.1674012622833; Tue, 17 Jan 2023 19:30:22 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id g2-20020a5d4882000000b00286ad197346sm30114632wrq.70.2023.01.17.19.30.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 17 Jan 2023 19:30:22 -0800 (PST) Message-Id: <9967401c972cab547d7619a208c3a0e6a3923cd4.1674012618.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Wed, 18 Jan 2023 03:30:08 +0000 Subject: [PATCH v6 03/12] daemon: rename some esoteric/laboured terminology Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Derrick Stolee , Lessley Dennington , Matthew John Cheetham , M Hickford , Jeff Hostetler , Glen Choo , Victoria Dye , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsA==?= Bjarmason , Matthew John Cheetham , Matthew John Cheetham Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Matthew John Cheetham From: Matthew John Cheetham Rename some of the variables and function arguments used to manage child processes. The existing names are esoteric; stretching an analogy too far to the point of being confusing to understand. Rename "firstborn" to "first_child", "newborn" to "new_cld", "blanket" to "current" and "cradle" to "ptr". Signed-off-by: Matthew John Cheetham --- daemon-utils.c | 46 +++++++++++++++++++++++----------------------- daemon-utils.h | 6 +++--- daemon.c | 10 +++++----- 3 files changed, 31 insertions(+), 31 deletions(-) diff --git a/daemon-utils.c b/daemon-utils.c index 8506664b440..f23ea35ed7b 100644 --- a/daemon-utils.c +++ b/daemon-utils.c @@ -230,44 +230,44 @@ static int addrcmp(const struct sockaddr_storage *s1, } void add_child(struct child_process *cld, struct sockaddr *addr, socklen_t addrlen, - struct child *firstborn , unsigned int *live_children) + struct child *first_child, unsigned int *live_children) { - struct child *newborn, **cradle; + struct child *new_cld, **current; - CALLOC_ARRAY(newborn, 1); + CALLOC_ARRAY(new_cld, 1); (*live_children)++; - memcpy(&newborn->cld, cld, sizeof(*cld)); - memcpy(&newborn->address, addr, addrlen); - for (cradle = &firstborn; *cradle; cradle = &(*cradle)->next) - if (!addrcmp(&(*cradle)->address, &newborn->address)) + memcpy(&new_cld->cld, cld, sizeof(*cld)); + memcpy(&new_cld->address, addr, addrlen); + for (current = &first_child; *current; current = &(*current)->next) + if (!addrcmp(&(*current)->address, &new_cld->address)) break; - newborn->next = *cradle; - *cradle = newborn; + new_cld->next = *current; + *current = new_cld; } -void kill_some_child(struct child *firstborn) +void kill_some_child(struct child *first_child) { - const struct child *blanket, *next; + const struct child *current, *next; - if (!(blanket = firstborn)) + if (!(current = first_child)) return; - for (; (next = blanket->next); blanket = next) - if (!addrcmp(&blanket->address, &next->address)) { - kill(blanket->cld.pid, SIGTERM); + for (; (next = current->next); current = next) + if (!addrcmp(¤t->address, &next->address)) { + kill(current->cld.pid, SIGTERM); break; } } -void check_dead_children(struct child *firstborn, unsigned int *live_children, +void check_dead_children(struct child *first_child, unsigned int *live_children, log_fn loginfo) { int status; pid_t pid; - struct child **cradle, *blanket; - for (cradle = &firstborn; (blanket = *cradle);) - if ((pid = waitpid(blanket->cld.pid, &status, WNOHANG)) > 1) { + struct child **ptr, *current; + for (ptr = &first_child; (current = *ptr);) + if ((pid = waitpid(current->cld.pid, &status, WNOHANG)) > 1) { if (loginfo) { const char *dead = ""; if (status) @@ -277,10 +277,10 @@ void check_dead_children(struct child *firstborn, unsigned int *live_children, } /* remove the child */ - *cradle = blanket->next; + *ptr = current->next; (*live_children)--; - child_process_clear(&blanket->cld); - free(blanket); + child_process_clear(¤t->cld); + free(current); } else - cradle = &blanket->next; + ptr = ¤t->next; } diff --git a/daemon-utils.h b/daemon-utils.h index 97e5cae20b8..c866e9c9a4e 100644 --- a/daemon-utils.h +++ b/daemon-utils.h @@ -32,7 +32,7 @@ struct child { * live children. */ void add_child(struct child_process *cld, struct sockaddr *addr, socklen_t addrlen, - struct child *firstborn, unsigned int *live_children); + struct child *first_child, unsigned int *live_children); /* * Kill the newest connection from a duplicate IP. @@ -40,7 +40,7 @@ void add_child(struct child_process *cld, struct sockaddr *addr, socklen_t addrl * This function should be called if the number of connections grows * past the maximum number of allowed connections. */ -void kill_some_child(struct child *firstborn); +void kill_some_child(struct child *first_child); /* * Check for children that have disconnected and remove them from the @@ -49,7 +49,7 @@ void kill_some_child(struct child *firstborn); * Optionally log the child PID that disconnected by passing a loginfo * function. */ -void check_dead_children(struct child *firstborn, unsigned int *live_children, +void check_dead_children(struct child *first_child, unsigned int *live_children, log_fn loginfo); #endif diff --git a/daemon.c b/daemon.c index ec3b407ecbc..d3e7d81de18 100644 --- a/daemon.c +++ b/daemon.c @@ -789,7 +789,7 @@ static int max_connections = 32; static unsigned int live_children; -static struct child *firstborn; +static struct child *first_child; static struct strvec cld_argv = STRVEC_INIT; static void handle(int incoming, struct sockaddr *addr, socklen_t addrlen) @@ -797,9 +797,9 @@ static void handle(int incoming, struct sockaddr *addr, socklen_t addrlen) struct child_process cld = CHILD_PROCESS_INIT; if (max_connections && live_children >= max_connections) { - kill_some_child(firstborn); + kill_some_child(first_child); sleep(1); /* give it some time to die */ - check_dead_children(firstborn, &live_children, loginfo); + check_dead_children(first_child, &live_children, loginfo); if (live_children >= max_connections) { close(incoming); logerror("Too many children, dropping connection"); @@ -832,7 +832,7 @@ static void handle(int incoming, struct sockaddr *addr, socklen_t addrlen) if (start_command(&cld)) logerror("unable to fork"); else - add_child(&cld, addr, addrlen, firstborn, &live_children); + add_child(&cld, addr, addrlen, first_child, &live_children); } static void child_handler(int signo) @@ -862,7 +862,7 @@ static int service_loop(struct socketlist *socklist) for (;;) { int i; - check_dead_children(firstborn, &live_children, loginfo); + check_dead_children(first_child, &live_children, loginfo); if (poll(pfd, socklist->nr, -1) < 0) { if (errno != EINTR) { From patchwork Wed Jan 18 03:30:09 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew John Cheetham X-Patchwork-Id: 13105488 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 61569C38142 for ; Wed, 18 Jan 2023 03:30:54 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229812AbjARDaw (ORCPT ); Tue, 17 Jan 2023 22:30:52 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51910 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229738AbjARDad (ORCPT ); Tue, 17 Jan 2023 22:30:33 -0500 Received: from mail-wr1-x431.google.com (mail-wr1-x431.google.com [IPv6:2a00:1450:4864:20::431]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5BB865399B for ; Tue, 17 Jan 2023 19:30:25 -0800 (PST) Received: by mail-wr1-x431.google.com with SMTP id b5so11439925wrn.0 for ; Tue, 17 Jan 2023 19:30:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:from:to:cc:subject:date :message-id:reply-to; bh=cKNlkhSdZ7ACwMk9ugShnF1r2+k5RVjpgvl5Ij8APwY=; b=c+BkEzgbk6mL3RgXyZZvVz745ezYY+t49OK1THzyqA0VKNTHK20qb6nPAlvNBQyogS 72YG+ZMd46uFmxjCt9SiA0qiiUtNuLwJjMAPim2V+Lm9C3qY0VOTh6MwD2xi6pFhzkNB taHXZVXHoROtljctEGZoaN7FTMj3oOterbxjZFt7urL9fNwOBuk8+z5zoaXRy3qEjt8q aIp60xDkZgWFnvSocDPknmGBkfme/BDP2D6kZHRx5dwiFxfsVkvKnA6xDkcPflckXDOR BFhOz5kWqBrQXSF1WrB5cVYsSleTM8aDBb7QpM5CwoT4RD7anYgLDV4LIEdl0ILAgO/T r5OQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=cKNlkhSdZ7ACwMk9ugShnF1r2+k5RVjpgvl5Ij8APwY=; b=6srZtlTiMdntRormHoHuwqIN6ydWVYmyKHGlr/cKUZ3PkcfMm+KCH8dhks1PvILa4Y F5hnOkS2zYXSJEqMjlTHsOT1kPGwL7AlP69tplYlUwl6Iv0GkyXAXFA0QbvYPx/hzkxb pu0HOkznQcryOKKNid1rE8qQCx2ns3qbmngEmdgvEL/IoEwXSSe2rSyLUL8uB8YKRgjf qxTQ36uRnw2jVk68fRZl7UCRu1O6fGqZrBnGA6tqcX5FZWXreiyxAOF3JRQIZKmoV4FN gPM1O4AzOFScqAaKGd0kDEGBKfmU7Bp6O7Q6WMxg3bM2W8abPJhPf17kHSzZ9nU5edAr KU4w== X-Gm-Message-State: AFqh2kpcMZh1bYSEyqP5NoA6dMg6ih2oz1V+wGlIUqsd/KGWclQWPs+V FN8CtPcOIiB1nOcUFaov606nayf+ggg= X-Google-Smtp-Source: AMrXdXuJu0GWAP2ULIBHV2zeoyy54EDkMOrGzo8W1jvnZ326Ep9F/OcogPaHLT4glgmomBIRE21ANA== X-Received: by 2002:a05:6000:1290:b0:2b6:8a41:a949 with SMTP id f16-20020a056000129000b002b68a41a949mr4790671wrx.46.1674012623585; Tue, 17 Jan 2023 19:30:23 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id m9-20020a056000024900b002bdec340a1csm10304113wrz.110.2023.01.17.19.30.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 17 Jan 2023 19:30:23 -0800 (PST) Message-Id: In-Reply-To: References: Date: Wed, 18 Jan 2023 03:30:09 +0000 Subject: [PATCH v6 04/12] test-http-server: add stub HTTP server test helper Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Derrick Stolee , Lessley Dennington , Matthew John Cheetham , M Hickford , Jeff Hostetler , Glen Choo , Victoria Dye , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsA==?= Bjarmason , Matthew John Cheetham , Matthew John Cheetham Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Matthew John Cheetham From: Matthew John Cheetham Introduce a mini HTTP server helper that in the future will be enhanced to provide a frontend for the git-http-backend, with support for arbitrary authentication schemes. Right now, test-http-server is a pared-down copy of the git-daemon that always returns a 501 Not Implemented response to all callers. Signed-off-by: Matthew John Cheetham --- Makefile | 1 + contrib/buildsystems/CMakeLists.txt | 11 +- t/helper/.gitignore | 1 + t/helper/test-http-server.c | 385 ++++++++++++++++++++++++++++ 4 files changed, 396 insertions(+), 2 deletions(-) create mode 100644 t/helper/test-http-server.c diff --git a/Makefile b/Makefile index 2654094dbb5..3cd61c792ac 100644 --- a/Makefile +++ b/Makefile @@ -865,6 +865,7 @@ TEST_BUILTINS_OBJS += test-xml-encode.o # Do not add more tests here unless they have extra dependencies. Add # them in TEST_BUILTINS_OBJS above. TEST_PROGRAMS_NEED_X += test-fake-ssh +TEST_PROGRAMS_NEED_X += test-http-server TEST_PROGRAMS_NEED_X += test-tool TEST_PROGRAMS = $(patsubst %,t/helper/%$X,$(TEST_PROGRAMS_NEED_X)) diff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt index 2f6e0197ffa..5d949dcb16c 100644 --- a/contrib/buildsystems/CMakeLists.txt +++ b/contrib/buildsystems/CMakeLists.txt @@ -961,6 +961,9 @@ if(BUILD_TESTING) add_executable(test-fake-ssh ${CMAKE_SOURCE_DIR}/t/helper/test-fake-ssh.c) target_link_libraries(test-fake-ssh common-main) +add_executable(test-http-server ${CMAKE_SOURCE_DIR}/t/helper/test-http-server.c) +target_link_libraries(test-http-server common-main) + #reftable-tests parse_makefile_for_sources(test-reftable_SOURCES "REFTABLE_TEST_OBJS") list(TRANSFORM test-reftable_SOURCES PREPEND "${CMAKE_SOURCE_DIR}/") @@ -980,6 +983,11 @@ if(MSVC) PROPERTIES RUNTIME_OUTPUT_DIRECTORY_DEBUG ${CMAKE_BINARY_DIR}/t/helper) set_target_properties(test-fake-ssh test-tool PROPERTIES RUNTIME_OUTPUT_DIRECTORY_RELEASE ${CMAKE_BINARY_DIR}/t/helper) + + set_target_properties(test-http-server + PROPERTIES RUNTIME_OUTPUT_DIRECTORY_DEBUG ${CMAKE_BINARY_DIR}/t/helper) + set_target_properties(test-http-server + PROPERTIES RUNTIME_OUTPUT_DIRECTORY_RELEASE ${CMAKE_BINARY_DIR}/t/helper) endif() #wrapper scripts @@ -987,8 +995,7 @@ set(wrapper_scripts git git-upload-pack git-receive-pack git-upload-archive git-shell git-remote-ext scalar) set(wrapper_test_scripts - test-fake-ssh test-tool) - + test-http-server test-fake-ssh test-tool) foreach(script ${wrapper_scripts}) file(STRINGS ${CMAKE_SOURCE_DIR}/wrap-for-bin.sh content NEWLINE_CONSUME) diff --git a/t/helper/.gitignore b/t/helper/.gitignore index 8c2ddcce95f..9aa9c752997 100644 --- a/t/helper/.gitignore +++ b/t/helper/.gitignore @@ -1,2 +1,3 @@ /test-tool /test-fake-ssh +/test-http-server diff --git a/t/helper/test-http-server.c b/t/helper/test-http-server.c new file mode 100644 index 00000000000..11071b1dd89 --- /dev/null +++ b/t/helper/test-http-server.c @@ -0,0 +1,385 @@ +#include "daemon-utils.h" +#include "config.h" +#include "run-command.h" +#include "strbuf.h" +#include "string-list.h" +#include "trace2.h" +#include "version.h" +#include "dir.h" +#include "date.h" + +#define TR2_CAT "test-http-server" + +static const char *pid_file; +static int verbose; +static int reuseaddr; + +static const char test_http_auth_usage[] = +"http-server [--verbose]\n" +" [--timeout=] [--max-connections=]\n" +" [--reuseaddr] [--pid-file=]\n" +" [--listen=]* [--port=]\n" +; + +static unsigned int timeout; + +static void logreport(const char *label, const char *err, va_list params) +{ + struct strbuf msg = STRBUF_INIT; + + strbuf_addf(&msg, "[%"PRIuMAX"] %s: ", (uintmax_t)getpid(), label); + strbuf_vaddf(&msg, err, params); + strbuf_addch(&msg, '\n'); + + fwrite(msg.buf, sizeof(char), msg.len, stderr); + fflush(stderr); + + strbuf_release(&msg); +} + +__attribute__((format (printf, 1, 2))) +static void logerror(const char *err, ...) +{ + va_list params; + va_start(params, err); + logreport("error", err, params); + va_end(params); +} + +__attribute__((format (printf, 1, 2))) +static void loginfo(const char *err, ...) +{ + va_list params; + if (!verbose) + return; + va_start(params, err); + logreport("info", err, params); + va_end(params); +} + +/* + * The code in this section is used by "worker" instances to service + * a single connection from a client. The worker talks to the client + * on 0 and 1. + */ + +enum worker_result { + /* + * Operation successful. + * Caller *might* keep the socket open and allow keep-alive. + */ + WR_OK = 0, + + /* + * Various errors while processing the request and/or the response. + * Close the socket and clean up. + * Exit child-process with non-zero status. + */ + WR_IO_ERROR = 1<<0, + + /* + * Close the socket and clean up. Does not imply an error. + */ + WR_HANGUP = 1<<1, +}; + +static enum worker_result worker(void) +{ + const char *response = "HTTP/1.1 501 Not Implemented\r\n"; + char *client_addr = getenv("REMOTE_ADDR"); + char *client_port = getenv("REMOTE_PORT"); + enum worker_result wr = WR_OK; + + if (client_addr) + loginfo("Connection from %s:%s", client_addr, client_port); + + set_keep_alive(0, logerror); + + while (1) { + if (write_in_full(STDOUT_FILENO, response, strlen(response)) < 0) { + logerror("unable to write response"); + wr = WR_IO_ERROR; + } + + if (wr != WR_OK) + break; + } + + close(STDIN_FILENO); + close(STDOUT_FILENO); + + return !!(wr & WR_IO_ERROR); +} + +static int max_connections = 32; + +static unsigned int live_children; + +static struct child *first_child; + +static struct strvec cld_argv = STRVEC_INIT; +static void handle(int incoming, struct sockaddr *addr, socklen_t addrlen) +{ + struct child_process cld = CHILD_PROCESS_INIT; + + if (max_connections && live_children >= max_connections) { + kill_some_child(first_child); + sleep(1); /* give it some time to die */ + check_dead_children(first_child, &live_children, loginfo); + if (live_children >= max_connections) { + close(incoming); + logerror("Too many children, dropping connection"); + return; + } + } + + if (addr->sa_family == AF_INET) { + char buf[128] = ""; + struct sockaddr_in *sin_addr = (void *) addr; + inet_ntop(addr->sa_family, &sin_addr->sin_addr, buf, sizeof(buf)); + strvec_pushf(&cld.env, "REMOTE_ADDR=%s", buf); + strvec_pushf(&cld.env, "REMOTE_PORT=%d", + ntohs(sin_addr->sin_port)); +#ifndef NO_IPV6 + } else if (addr->sa_family == AF_INET6) { + char buf[128] = ""; + struct sockaddr_in6 *sin6_addr = (void *) addr; + inet_ntop(AF_INET6, &sin6_addr->sin6_addr, buf, sizeof(buf)); + strvec_pushf(&cld.env, "REMOTE_ADDR=[%s]", buf); + strvec_pushf(&cld.env, "REMOTE_PORT=%d", + ntohs(sin6_addr->sin6_port)); +#endif + } + + strvec_pushv(&cld.args, cld_argv.v); + cld.in = incoming; + cld.out = dup(incoming); + + if (cld.out < 0) + logerror("could not dup() `incoming`"); + else if (start_command(&cld)) + logerror("unable to fork"); + else + add_child(&cld, addr, addrlen, first_child, &live_children); +} + +static void child_handler(int signo) +{ + /* + * Otherwise empty handler because systemcalls will get interrupted + * upon signal receipt + * SysV needs the handler to be rearmed + */ + signal(SIGCHLD, child_handler); +} + +static int service_loop(struct socketlist *socklist) +{ + struct pollfd *pfd; + int i; + + CALLOC_ARRAY(pfd, socklist->nr); + + for (i = 0; i < socklist->nr; i++) { + pfd[i].fd = socklist->list[i]; + pfd[i].events = POLLIN; + } + + signal(SIGCHLD, child_handler); + + for (;;) { + int i; + int nr_ready; + int timeout = (pid_file ? 100 : -1); + + check_dead_children(first_child, &live_children, loginfo); + + nr_ready = poll(pfd, socklist->nr, timeout); + if (nr_ready < 0) { + if (errno != EINTR) { + logerror("Poll failed, resuming: %s", + strerror(errno)); + sleep(1); + } + continue; + } + else if (nr_ready == 0) { + /* + * If we have a pid_file, then we watch it. + * If someone deletes it, we shutdown the service. + * The shell scripts in the test suite will use this. + */ + if (!pid_file || file_exists(pid_file)) + continue; + goto shutdown; + } + + for (i = 0; i < socklist->nr; i++) { + if (pfd[i].revents & POLLIN) { + union { + struct sockaddr sa; + struct sockaddr_in sai; +#ifndef NO_IPV6 + struct sockaddr_in6 sai6; +#endif + } ss; + socklen_t sslen = sizeof(ss); + int incoming = accept(pfd[i].fd, &ss.sa, &sslen); + if (incoming < 0) { + switch (errno) { + case EAGAIN: + case EINTR: + case ECONNABORTED: + continue; + default: + die_errno("accept returned"); + } + } + handle(incoming, &ss.sa, sslen); + } + } + } + +shutdown: + loginfo("Starting graceful shutdown (pid-file gone)"); + for (i = 0; i < socklist->nr; i++) + close(socklist->list[i]); + + return 0; +} + +static int serve(struct string_list *listen_addr, int listen_port) +{ + struct socketlist socklist = { NULL, 0, 0 }; + + socksetup(listen_addr, listen_port, &socklist, reuseaddr, logerror); + if (socklist.nr == 0) + die("unable to allocate any listen sockets on port %u", + listen_port); + + loginfo("Ready to rumble"); + + /* + * Wait to create the pid-file until we've setup the sockets + * and are open for business. + */ + if (pid_file) + write_file(pid_file, "%"PRIuMAX, (uintmax_t) getpid()); + + return service_loop(&socklist); +} + +/* + * This section is executed by both the primary instance and all + * worker instances. So, yes, each child-process re-parses the + * command line argument and re-discovers how it should behave. + */ + +int cmd_main(int argc, const char **argv) +{ + int listen_port = 0; + struct string_list listen_addr = STRING_LIST_INIT_NODUP; + int worker_mode = 0; + int i; + + trace2_cmd_name("test-http-server"); + trace2_cmd_list_config(); + trace2_cmd_list_env_vars(); + setup_git_directory_gently(NULL); + + for (i = 1; i < argc; i++) { + const char *arg = argv[i]; + const char *v; + + if (skip_prefix(arg, "--listen=", &v)) { + string_list_append(&listen_addr, xstrdup_tolower(v)); + continue; + } + if (skip_prefix(arg, "--port=", &v)) { + char *end; + unsigned long n; + n = strtoul(v, &end, 0); + if (*v && !*end) { + listen_port = n; + continue; + } + } + if (!strcmp(arg, "--worker")) { + worker_mode = 1; + trace2_cmd_mode("worker"); + continue; + } + if (!strcmp(arg, "--verbose")) { + verbose = 1; + continue; + } + if (skip_prefix(arg, "--timeout=", &v)) { + timeout = atoi(v); + continue; + } + if (skip_prefix(arg, "--max-connections=", &v)) { + max_connections = atoi(v); + if (max_connections < 0) + max_connections = 0; /* unlimited */ + continue; + } + if (!strcmp(arg, "--reuseaddr")) { + reuseaddr = 1; + continue; + } + if (skip_prefix(arg, "--pid-file=", &v)) { + pid_file = v; + continue; + } + + fprintf(stderr, "error: unknown argument '%s'\n", arg); + usage(test_http_auth_usage); + } + + /* avoid splitting a message in the middle */ + setvbuf(stderr, NULL, _IOFBF, 4096); + + if (listen_port == 0) + listen_port = DEFAULT_GIT_PORT; + + /* + * If no --listen= args are given, the setup_named_sock() + * code will use receive a NULL address and set INADDR_ANY. + * This exposes both internal and external interfaces on the + * port. + * + * Disallow that and default to the internal-use-only loopback + * address. + */ + if (!listen_addr.nr) + string_list_append(&listen_addr, "127.0.0.1"); + + /* + * worker_mode is set in our own child process instances + * (that are bound to a connected socket from a client). + */ + if (worker_mode) + return worker(); + + /* + * `cld_argv` is a bit of a clever hack. The top-level instance + * of test-http-server does the normal bind/listen/accept stuff. + * For each incoming socket, the top-level process spawns + * a child instance of test-http-server *WITH* the additional + * `--worker` argument. This causes the child to set `worker_mode` + * and immediately call `worker()` using the connected socket (and + * without the usual need for fork() or threads). + * + * The magic here is made possible because `cld_argv` is static + * and handle() (called by service_loop()) knows about it. + */ + strvec_push(&cld_argv, argv[0]); + strvec_push(&cld_argv, "--worker"); + for (i = 1; i < argc; ++i) + strvec_push(&cld_argv, argv[i]); + + /* + * Setup primary instance to listen for connections. + */ + return serve(&listen_addr, listen_port); +} From patchwork Wed Jan 18 03:30:10 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew John Cheetham X-Patchwork-Id: 13105486 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1046CC38142 for ; Wed, 18 Jan 2023 03:30:50 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229734AbjARDas (ORCPT ); Tue, 17 Jan 2023 22:30:48 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51686 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229455AbjARDaZ (ORCPT ); Tue, 17 Jan 2023 22:30:25 -0500 Received: from mail-wr1-x432.google.com (mail-wr1-x432.google.com [IPv6:2a00:1450:4864:20::432]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0A27B5357B for ; Tue, 17 Jan 2023 19:30:25 -0800 (PST) Received: by mail-wr1-x432.google.com with SMTP id d2so12483749wrp.8 for ; Tue, 17 Jan 2023 19:30:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:from:to:cc:subject:date :message-id:reply-to; bh=tkCyGdRuFvoxQvnRn4u0Bc0mNZYPHa0MqDbP3iMoKyk=; b=QN+DJVk/8TJk85yUtEprV8bi7ZNlLN4AzbeI4JkLRGT/NkGK2d4kHtkJ3gJVmu+IPM QDEIKzgSk1n3E4oTw+YBpxLazUFXGUoVLK1n331wPtYvpnFpofV3k3h4m3DsmX4XYaLh BjAzkbcVeU+UKW+ijvezJPleR1dOfSZl5X1LKEwsGRyE9cebIDioGwLmpCPV/wQ1LiPH kV49TcUbSinrOnO2oVXiEUzPYR/VQspGDRK9HMzrTbV2aob2DiN3FM9cyeuRX5pIHNM5 uJd58s2eTroeJwFnUXFLbIzyfQLsq5AA74A/uthm4lUatPg1YmLV/wktnYTdhzERAxvD 1xAg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=tkCyGdRuFvoxQvnRn4u0Bc0mNZYPHa0MqDbP3iMoKyk=; b=l2qp6XatjlGjAfF83espcn/2hHCbhJvx6pJV2siPilf61m6jN7fbU9qi76bbuQOokB cFgrkHOO8ZG/a0Q+YPxKf+i1+2JMfE8vIt4jYdEI6hlu46fCdL1piJ/XfZGOXRjTO21h rBvsRMztyOuJmSbYkseoVglYlEjnSUvxKo2GdWcvmvhvr+nbg1UoNOBLEorxV9eArueb Ic6MnNerZkSXTH6UsEpAVkSpNz43LlHxEKcRO8WW/bn5TUdQdUN243Qp81UHoIsSaKSF LeWD63edJrP3kzTqvRhXZGJ/3VB6rJlI/LG5SyBIQHL+k7w6flmvQMG86CVSd8Wt7RRX SyIg== X-Gm-Message-State: AFqh2kqizXeUyEoRHcHmiQx0KrdxnJ1w3cz7wwR//XPDWEmEY+iT/i36 dAxtMArngm44tR7UpNrdxhqjZKsXrl8= X-Google-Smtp-Source: AMrXdXtSDU01uh8/oVT2R8OXy9aXbCq3eZh4iSjACQq6VLGCO6rVWaTFLsjtzrF6bbMBJ2jvBiPcGw== X-Received: by 2002:adf:f10d:0:b0:2bc:7e97:3d5f with SMTP id r13-20020adff10d000000b002bc7e973d5fmr4742236wro.0.1674012624476; Tue, 17 Jan 2023 19:30:24 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id f8-20020a0560001b0800b002423edd7e50sm29913366wrz.32.2023.01.17.19.30.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 17 Jan 2023 19:30:24 -0800 (PST) Message-Id: <79805f042b984bb8ca7c9aaf6a15f8101037c375.1674012618.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Wed, 18 Jan 2023 03:30:10 +0000 Subject: [PATCH v6 05/12] test-http-server: add HTTP error response function Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Derrick Stolee , Lessley Dennington , Matthew John Cheetham , M Hickford , Jeff Hostetler , Glen Choo , Victoria Dye , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsA==?= Bjarmason , Matthew John Cheetham , Matthew John Cheetham Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Matthew John Cheetham From: Matthew John Cheetham Introduce a function to the test-http-server test helper to write more full and valid HTTP error responses, including all the standard response headers like `Server` and `Date`. Signed-off-by: Matthew John Cheetham --- t/helper/test-http-server.c | 58 +++++++++++++++++++++++++++++++++---- 1 file changed, 53 insertions(+), 5 deletions(-) diff --git a/t/helper/test-http-server.c b/t/helper/test-http-server.c index 11071b1dd89..6cdac223a55 100644 --- a/t/helper/test-http-server.c +++ b/t/helper/test-http-server.c @@ -83,9 +83,59 @@ enum worker_result { WR_HANGUP = 1<<1, }; +static enum worker_result send_http_error( + int fd, + int http_code, const char *http_code_name, + int retry_after_seconds, struct string_list *response_headers, + enum worker_result wr_in) +{ + struct strbuf response_header = STRBUF_INIT; + struct strbuf response_content = STRBUF_INIT; + struct string_list_item *h; + enum worker_result wr; + + strbuf_addf(&response_content, "Error: %d %s\r\n", + http_code, http_code_name); + if (retry_after_seconds > 0) + strbuf_addf(&response_content, "Retry-After: %d\r\n", + retry_after_seconds); + + strbuf_addf (&response_header, "HTTP/1.1 %d %s\r\n", http_code, http_code_name); + strbuf_addstr(&response_header, "Cache-Control: private\r\n"); + strbuf_addstr(&response_header, "Content-Type: text/plain\r\n"); + strbuf_addf (&response_header, "Content-Length: %d\r\n", (int)response_content.len); + if (retry_after_seconds > 0) + strbuf_addf(&response_header, "Retry-After: %d\r\n", retry_after_seconds); + strbuf_addf( &response_header, "Server: test-http-server/%s\r\n", git_version_string); + strbuf_addf( &response_header, "Date: %s\r\n", show_date(time(NULL), 0, DATE_MODE(RFC2822))); + if (response_headers) + for_each_string_list_item(h, response_headers) + strbuf_addf(&response_header, "%s\r\n", h->string); + strbuf_addstr(&response_header, "\r\n"); + + if (write_in_full(fd, response_header.buf, response_header.len) < 0) { + logerror("unable to write response header"); + wr = WR_IO_ERROR; + goto done; + } + + if (write_in_full(fd, response_content.buf, response_content.len) < 0) { + logerror("unable to write response content body"); + wr = WR_IO_ERROR; + goto done; + } + + wr = wr_in; + +done: + strbuf_release(&response_header); + strbuf_release(&response_content); + + return wr; +} + static enum worker_result worker(void) { - const char *response = "HTTP/1.1 501 Not Implemented\r\n"; char *client_addr = getenv("REMOTE_ADDR"); char *client_port = getenv("REMOTE_PORT"); enum worker_result wr = WR_OK; @@ -96,10 +146,8 @@ static enum worker_result worker(void) set_keep_alive(0, logerror); while (1) { - if (write_in_full(STDOUT_FILENO, response, strlen(response)) < 0) { - logerror("unable to write response"); - wr = WR_IO_ERROR; - } + wr = send_http_error(STDOUT_FILENO, 501, "Not Implemented", -1, + NULL, WR_OK | WR_HANGUP); if (wr != WR_OK) break; From patchwork Wed Jan 18 03:30:11 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew John Cheetham X-Patchwork-Id: 13105489 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 36111C38159 for ; Wed, 18 Jan 2023 03:30:56 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229818AbjARDay (ORCPT ); Tue, 17 Jan 2023 22:30:54 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51922 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229751AbjARDad (ORCPT ); Tue, 17 Jan 2023 22:30:33 -0500 Received: from mail-wr1-x42c.google.com (mail-wr1-x42c.google.com [IPv6:2a00:1450:4864:20::42c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1F2E9539B0 for ; Tue, 17 Jan 2023 19:30:26 -0800 (PST) Received: by mail-wr1-x42c.google.com with SMTP id z5so31554884wrt.6 for ; Tue, 17 Jan 2023 19:30:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:from:to:cc:subject:date :message-id:reply-to; bh=mza3bVH56jY8ViF4P8N6fixUf8LkOvX8UaGasQaTaZ8=; b=Y7YAnksMudhhcTLeGDSMwDRfPXnOxD94CeDY8roa4IZ4Nj1e0tCi+UZDidFPEE7lw3 vE+Cl18CDwjrWbJ2NpEBN4OVLh9ibmWW80mZwzgnPmEJ6f7jHWJFQz6TjLEAUahzBiY7 PXmXEN5GO3kFt3+eIV/fQJ+pTVK73fKP1XFCHLkAvCcyNI8Jo2kwa1ko69b5EW4568qT pop7Rvbg/NJc2y9LT5W6XpiBfgajmaoRrOor5Wt/b85wY5K1XNPfKVyhays0MCIUOEZt 3BkLiMg/SBaXEoYfWZ9cgg5K5e89kJ4qjT4+fZlB3dI6w6NJjt8CUqkG9MshhsiWdZ65 ShHQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=mza3bVH56jY8ViF4P8N6fixUf8LkOvX8UaGasQaTaZ8=; b=vYTZB+hwcokMS5jdxzi7KTBb0PfgR1NeBaE7nAVX/6gJ0kZr6p/chCp33ku6+BpDbh DbuekkKTvPJNG+6SR/pa1mNSax4+5b3zhbavKWu2HZ/WrkYe5vLqfYkuOb4iSZMw/+fS jf0+Rw3QT+PHCp6kdS63IZY86Qr1ddH1TQK2uI+DhW1ZJLXBD/UJzz3Q6RDAgRLioCWm GmiCTuYwJb6ahqKiWANCxEl3wKFYfsh3p2SfnQHQ+OjBx5kCNqxPGJR2/JEbCBNhgJCt MyswdY8jgkHELm8QV81xvjKQ9NAa+Zgij384Oz0uSi4NFf+bYZhWu7mpkVE4PS37dTKe SE+Q== X-Gm-Message-State: AFqh2kqPMiutR5Gldgan8Hrdxqsb/PrX9lCVHPNsTfnIEfaj8ev2wwqo nBiCc9m9RuMoT8tDgbesQpYD1WiIklg= X-Google-Smtp-Source: AMrXdXt/I1qkrLICSGqYgiZDHhFdWwUiByMDn0ACS/crB39OHR339BB/qNsfe0dR0St/dqvYyDGSpA== X-Received: by 2002:a05:6000:104a:b0:2bb:4b40:2d18 with SMTP id c10-20020a056000104a00b002bb4b402d18mr4303237wrx.62.1674012625294; Tue, 17 Jan 2023 19:30:25 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id c18-20020adffb52000000b0025e86026866sm35630015wrs.0.2023.01.17.19.30.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 17 Jan 2023 19:30:24 -0800 (PST) Message-Id: <252098db219574527c587bc601565eab81b40c2c.1674012618.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Wed, 18 Jan 2023 03:30:11 +0000 Subject: [PATCH v6 06/12] test-http-server: add HTTP request parsing Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Derrick Stolee , Lessley Dennington , Matthew John Cheetham , M Hickford , Jeff Hostetler , Glen Choo , Victoria Dye , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsA==?= Bjarmason , Matthew John Cheetham , Matthew John Cheetham Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Matthew John Cheetham From: Matthew John Cheetham Add ability to parse HTTP requests to the test-http-server test helper. Signed-off-by: Matthew John Cheetham --- t/helper/test-http-server.c | 175 +++++++++++++++++++++++++++++++++++- 1 file changed, 173 insertions(+), 2 deletions(-) diff --git a/t/helper/test-http-server.c b/t/helper/test-http-server.c index 6cdac223a55..36f4a54fe6d 100644 --- a/t/helper/test-http-server.c +++ b/t/helper/test-http-server.c @@ -83,6 +83,42 @@ enum worker_result { WR_HANGUP = 1<<1, }; +/* + * Fields from a parsed HTTP request. + */ +struct req { + struct strbuf start_line; + + const char *method; + const char *http_version; + + struct strbuf uri_path; + struct strbuf query_args; + + struct string_list header_list; + const char *content_type; + ssize_t content_length; +}; + +#define REQ__INIT { \ + .start_line = STRBUF_INIT, \ + .uri_path = STRBUF_INIT, \ + .query_args = STRBUF_INIT, \ + .header_list = STRING_LIST_INIT_NODUP, \ + .content_type = NULL, \ + .content_length = -1 \ + } + +static void req__release(struct req *req) +{ + strbuf_release(&req->start_line); + + strbuf_release(&req->uri_path); + strbuf_release(&req->query_args); + + string_list_clear(&req->header_list, 0); +} + static enum worker_result send_http_error( int fd, int http_code, const char *http_code_name, @@ -134,8 +170,136 @@ done: return wr; } +/* + * Read the HTTP request up to the start of the optional message-body. + * We do this byte-by-byte because we have keep-alive turned on and + * cannot rely on an EOF. + * + * https://tools.ietf.org/html/rfc7230 + * + * We cannot call die() here because our caller needs to properly + * respond to the client and/or close the socket before this + * child exits so that the client doesn't get a connection reset + * by peer error. + */ +static enum worker_result req__read(struct req *req, int fd) +{ + struct strbuf h = STRBUF_INIT; + struct string_list start_line_fields = STRING_LIST_INIT_DUP; + int nr_start_line_fields; + const char *uri_target; + const char *query; + char *hp; + const char *hv; + + enum worker_result result = WR_OK; + + /* + * Read line 0 of the request and split it into component parts: + * + * SP SP CRLF + * + */ + if (strbuf_getwholeline_fd(&req->start_line, fd, '\n') == EOF) { + result = WR_OK | WR_HANGUP; + goto done; + } + + strbuf_trim_trailing_newline(&req->start_line); + + nr_start_line_fields = string_list_split(&start_line_fields, + req->start_line.buf, + ' ', -1); + if (nr_start_line_fields != 3) { + logerror("could not parse request start-line '%s'", + req->start_line.buf); + result = WR_IO_ERROR; + goto done; + } + + req->method = xstrdup(start_line_fields.items[0].string); + req->http_version = xstrdup(start_line_fields.items[2].string); + + uri_target = start_line_fields.items[1].string; + + if (strcmp(req->http_version, "HTTP/1.1")) { + logerror("unsupported version '%s' (expecting HTTP/1.1)", + req->http_version); + result = WR_IO_ERROR; + goto done; + } + + query = strchr(uri_target, '?'); + + if (query) { + strbuf_add(&req->uri_path, uri_target, (query - uri_target)); + strbuf_trim_trailing_dir_sep(&req->uri_path); + strbuf_addstr(&req->query_args, query + 1); + } else { + strbuf_addstr(&req->uri_path, uri_target); + strbuf_trim_trailing_dir_sep(&req->uri_path); + } + + /* + * Read the set of HTTP headers into a string-list. + */ + while (1) { + if (strbuf_getwholeline_fd(&h, fd, '\n') == EOF) + goto done; + strbuf_trim_trailing_newline(&h); + + if (!h.len) + goto done; /* a blank line ends the header */ + + hp = strbuf_detach(&h, NULL); + string_list_append(&req->header_list, hp); + + /* also store common request headers as struct req members */ + if (skip_prefix(hp, "Content-Type: ", &hv)) { + req->content_type = hv; + } else if (skip_prefix(hp, "Content-Length: ", &hv)) { + req->content_length = strtol(hv, &hp, 10); + } + } + + /* + * We do not attempt to read the , if it exists. + * We let our caller read/chunk it in as appropriate. + */ + +done: + string_list_clear(&start_line_fields, 0); + + /* + * This is useful for debugging the request, but very noisy. + */ + if (trace2_is_enabled()) { + struct string_list_item *item; + trace2_printf("%s: %s", TR2_CAT, req->start_line.buf); + trace2_printf("%s: hver: %s", TR2_CAT, req->http_version); + trace2_printf("%s: hmth: %s", TR2_CAT, req->method); + trace2_printf("%s: path: %s", TR2_CAT, req->uri_path.buf); + trace2_printf("%s: qury: %s", TR2_CAT, req->query_args.buf); + if (req->content_length >= 0) + trace2_printf("%s: clen: %d", TR2_CAT, req->content_length); + if (req->content_type) + trace2_printf("%s: ctyp: %s", TR2_CAT, req->content_type); + for_each_string_list_item(item, &req->header_list) + trace2_printf("%s: hdrs: %s", TR2_CAT, item->string); + } + + return result; +} + +static enum worker_result dispatch(struct req *req) +{ + return send_http_error(STDOUT_FILENO, 501, "Not Implemented", -1, NULL, + WR_OK | WR_HANGUP); +} + static enum worker_result worker(void) { + struct req req = REQ__INIT; char *client_addr = getenv("REMOTE_ADDR"); char *client_port = getenv("REMOTE_PORT"); enum worker_result wr = WR_OK; @@ -146,9 +310,16 @@ static enum worker_result worker(void) set_keep_alive(0, logerror); while (1) { - wr = send_http_error(STDOUT_FILENO, 501, "Not Implemented", -1, - NULL, WR_OK | WR_HANGUP); + req__release(&req); + + alarm(timeout); + wr = req__read(&req, 0); + alarm(0); + + if (wr != WR_OK) + break; + wr = dispatch(&req); if (wr != WR_OK) break; } From patchwork Wed Jan 18 03:30:12 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew John Cheetham X-Patchwork-Id: 13105493 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6D78AC00A5A for ; Wed, 18 Jan 2023 03:31:05 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229827AbjARDbB (ORCPT ); Tue, 17 Jan 2023 22:31:01 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51924 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229750AbjARDad (ORCPT ); Tue, 17 Jan 2023 22:30:33 -0500 Received: from mail-wm1-x336.google.com (mail-wm1-x336.google.com [IPv6:2a00:1450:4864:20::336]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B5016521D3 for ; Tue, 17 Jan 2023 19:30:27 -0800 (PST) Received: by mail-wm1-x336.google.com with SMTP id q8so11937879wmo.5 for ; Tue, 17 Jan 2023 19:30:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:from:to:cc:subject:date :message-id:reply-to; bh=AJsZr9uc7gd0E/0o2TzFzv3kDRvmk40ohT91L6SsIrA=; b=MyOZr+MXj1xV26+gRC5QgmPRrAl/VZmm3Yl5b66iHcPUC4RjQgS4ml6IRyR5a3PVCW kRLAuBD9YT2yH5dR7WDeM5z6f2SOfwNs7Cq6O1Xu2Ix5YquhBXmAhrQJKUdgOju9LjHB kLmKwOFoYHy4RZoWaOJzdOG9AIvH/iiUh8/ORw5y6mcYUOONJo6DMtpsrd4Y5o6Ry0j/ nNHX5Cdd/8Kk1xF5ygwjue4DgF2OdJLs0GU7AOqK2ZKpBNbb9AicUlbl7HDqu4Jlxh3A zBChwThKidLxD4bNUwKcpVX9AsTz878emiMpvptALCTj6SP4XIGRndxqVs/DCnSp0MZo l1rQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=AJsZr9uc7gd0E/0o2TzFzv3kDRvmk40ohT91L6SsIrA=; b=HkkzETAMTHIAkDudaK5f2CfkgnVDfD4SrVEFnSO6AjAgchKarUmbFpRFKM5Q1ZiBjD V/1gS8MdD01DHkil42yaMhiw/8vjcw42pa/StQLXBtvvNzhnIaTbAV2otDqZADPRbyGY zs5S81dtvKYcnDUu2ePImJHupuBTCg69U21OFVIoJLheI7cboqoeJGoZgBEF8IpCwKjY k1GfaIaxHZPO5oniz96kVffeFu+pKzrQPzKHEaf1yGHHaR8vZIbVnvT304bbEG0RwGYc 3N7JHHJVF3lFMqaIAqmHWEEH+XAHcuT5mblaGgQrX8LcTRg6IrfwsFZ2rMXWzp2cl6l4 kYQw== X-Gm-Message-State: AFqh2kqZ3X2DBmikj7h3LWnVG3cs+MOjMo202JPVPwAu0eXH4obeqDWD 28TNqnccENP4qRTIMrkcJix9BHLtyKo= X-Google-Smtp-Source: AMrXdXu86aio7uUZCzCG7UC1vdSOccbHELsl2822bThqFjqrvF/H4T1IdBh6Ao4KahsNuxkLemXH4g== X-Received: by 2002:a05:600c:202:b0:3da:f80d:7230 with SMTP id 2-20020a05600c020200b003daf80d7230mr5036914wmi.8.1674012626083; Tue, 17 Jan 2023 19:30:26 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id i17-20020a05600c355100b003dafcd9b750sm578697wmq.43.2023.01.17.19.30.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 17 Jan 2023 19:30:25 -0800 (PST) Message-Id: In-Reply-To: References: Date: Wed, 18 Jan 2023 03:30:12 +0000 Subject: [PATCH v6 07/12] test-http-server: pass Git requests to http-backend Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Derrick Stolee , Lessley Dennington , Matthew John Cheetham , M Hickford , Jeff Hostetler , Glen Choo , Victoria Dye , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsA==?= Bjarmason , Matthew John Cheetham , Matthew John Cheetham Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Matthew John Cheetham From: Matthew John Cheetham Teach the test-http-sever test helper to forward Git requests to the `git-http-backend`. Introduce a new test script t5556-http-auth.sh that spins up the test HTTP server and attempts an `ls-remote` on the served repository, without any authentication. Signed-off-by: Matthew John Cheetham --- t/helper/test-http-server.c | 71 ++++++++++++++++++++++++ t/t5556-http-auth.sh | 107 ++++++++++++++++++++++++++++++++++++ 2 files changed, 178 insertions(+) create mode 100755 t/t5556-http-auth.sh diff --git a/t/helper/test-http-server.c b/t/helper/test-http-server.c index 36f4a54fe6d..ae17c738259 100644 --- a/t/helper/test-http-server.c +++ b/t/helper/test-http-server.c @@ -291,8 +291,79 @@ done: return result; } +static int is_git_request(struct req *req) +{ + static regex_t *smart_http_regex; + static int initialized; + + if (!initialized) { + smart_http_regex = xmalloc(sizeof(*smart_http_regex)); + /* + * This regular expression matches all dumb and smart HTTP + * requests that are currently in use, and defined in + * Documentation/gitprotocol-http.txt. + * + */ + if (regcomp(smart_http_regex, "^/(HEAD|info/refs|" + "objects/info/[^/]+|git-(upload|receive)-pack)$", + REG_EXTENDED)) { + warning("could not compile smart HTTP regex"); + smart_http_regex = NULL; + } + initialized = 1; + } + + return smart_http_regex && + !regexec(smart_http_regex, req->uri_path.buf, 0, NULL, 0); +} + +static enum worker_result do__git(struct req *req) +{ + const char *ok = "HTTP/1.1 200 OK\r\n"; + struct child_process cp = CHILD_PROCESS_INIT; + int res; + + /* + * Note that we always respond with a 200 OK response even if the + * http-backend process exits with an error. This helper is intended + * only to be used to exercise the HTTP auth handling in the Git client, + * and specifically around authentication (not handled by http-backend). + * + * If we wanted to respond with a more 'valid' HTTP response status then + * we'd need to buffer the output of http-backend, wait for and grok the + * exit status of the process, then write the HTTP status line followed + * by the http-backend output. This is outside of the scope of this test + * helper's use at time of writing. + */ + if (write(STDOUT_FILENO, ok, strlen(ok)) < 0) + return error(_("could not send '%s'"), ok); + + strvec_pushf(&cp.env, "REQUEST_METHOD=%s", req->method); + strvec_pushf(&cp.env, "PATH_TRANSLATED=%s", + req->uri_path.buf); + strvec_push(&cp.env, "SERVER_PROTOCOL=HTTP/1.1"); + if (req->query_args.len) + strvec_pushf(&cp.env, "QUERY_STRING=%s", + req->query_args.buf); + if (req->content_type) + strvec_pushf(&cp.env, "CONTENT_TYPE=%s", + req->content_type); + if (req->content_length >= 0) + strvec_pushf(&cp.env, "CONTENT_LENGTH=%" PRIdMAX, + (intmax_t)req->content_length); + cp.git_cmd = 1; + strvec_push(&cp.args, "http-backend"); + res = run_command(&cp); + close(STDOUT_FILENO); + close(STDIN_FILENO); + return !!res; +} + static enum worker_result dispatch(struct req *req) { + if (is_git_request(req)) + return do__git(req); + return send_http_error(STDOUT_FILENO, 501, "Not Implemented", -1, NULL, WR_OK | WR_HANGUP); } diff --git a/t/t5556-http-auth.sh b/t/t5556-http-auth.sh new file mode 100755 index 00000000000..ce1abffa6aa --- /dev/null +++ b/t/t5556-http-auth.sh @@ -0,0 +1,107 @@ +#!/bin/sh + +test_description='test http auth header and credential helper interop' + +TEST_NO_CREATE_REPO=1 +. ./test-lib.sh + +test_set_port GIT_TEST_HTTP_PROTOCOL_PORT + +# Setup a repository +# +REPO_DIR="$TRASH_DIRECTORY"/repo + +# Setup some lookback URLs where test-http-server will be listening. +# We will spawn it directly inside the repo directory, so we avoid +# any need to configure directory mappings etc - we only serve this +# repository from the root '/' of the server. +# +HOST_PORT=127.0.0.1:$GIT_TEST_HTTP_PROTOCOL_PORT +ORIGIN_URL=http://$HOST_PORT/ + +# The pid-file is created by test-http-server when it starts. +# The server will shutdown if/when we delete it (this is easier than +# killing it by PID). +# +PID_FILE="$TRASH_DIRECTORY"/pid-file.pid +SERVER_LOG="$TRASH_DIRECTORY"/OUT.server.log + +PATH="$GIT_BUILD_DIR/t/helper/:$PATH" && export PATH + +test_expect_success 'setup repos' ' + test_create_repo "$REPO_DIR" && + git -C "$REPO_DIR" branch -M main +' + +stop_http_server () { + if ! test -f "$PID_FILE" + then + return 0 + fi + # + # The server will shutdown automatically when we delete the pid-file. + # + rm -f "$PID_FILE" + # + # Give it a few seconds to shutdown (mainly to completely release the + # port before the next test start another instance and it attempts to + # bind to it). + # + for k in 0 1 2 3 4 + do + if grep -q "Starting graceful shutdown" "$SERVER_LOG" + then + return 0 + fi + sleep 1 + done + + echo "stop_http_server: timeout waiting for server shutdown" + return 1 +} + +start_http_server () { + # + # Launch our server into the background in repo_dir. + # + ( + cd "$REPO_DIR" + test-http-server --verbose \ + --listen=127.0.0.1 \ + --port=$GIT_TEST_HTTP_PROTOCOL_PORT \ + --reuseaddr \ + --pid-file="$PID_FILE" \ + "$@" \ + 2>"$SERVER_LOG" & + ) + # + # Give it a few seconds to get started. + # + for k in 0 1 2 3 4 + do + if test -f "$PID_FILE" + then + return 0 + fi + sleep 1 + done + + echo "start_http_server: timeout waiting for server startup" + return 1 +} + +per_test_cleanup () { + stop_http_server && + rm -f OUT.* +} + +test_expect_success 'http auth anonymous no challenge' ' + test_when_finished "per_test_cleanup" && + + start_http_server && + + # Attempt to read from a protected repository + git ls-remote $ORIGIN_URL +' + +test_done From patchwork Wed Jan 18 03:30:13 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew John Cheetham X-Patchwork-Id: 13105490 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7AE61C00A5A for ; Wed, 18 Jan 2023 03:30:57 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229820AbjARDa4 (ORCPT ); Tue, 17 Jan 2023 22:30:56 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51940 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229760AbjARDae (ORCPT ); Tue, 17 Jan 2023 22:30:34 -0500 Received: from mail-wr1-x42c.google.com (mail-wr1-x42c.google.com [IPv6:2a00:1450:4864:20::42c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AA79753B14 for ; Tue, 17 Jan 2023 19:30:28 -0800 (PST) Received: by mail-wr1-x42c.google.com with SMTP id h16so32596417wrz.12 for ; Tue, 17 Jan 2023 19:30:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:from:to:cc:subject:date :message-id:reply-to; bh=RiuOwIpPh651aZ2hK+ip9NyOtHBGpuZ3/Qaj6I00OOk=; b=J3U1ohbLK7XAlLR6/pTd49QgiTqhs39b6/jRhLf23tjHkA+uezbqxEJivvCPCIVrkk e1YoH5HF6W5TnFE6B4/buu5l30LenqByfNWj/CEcgm+dl4MPl/DESrQuykgsbQ02mSk1 kc/YrrIh4i8Rjt4mkdZrZTzm0aAE5jWGuO14id/LwjBzsmh7HwCDbnkqD6/HKU0mmMn8 HsnrwfcOTsyfvxrMjhrr1pnCiKt7zzWt+wE97kxCwBBEOh4Hh0SHOBLC62nMCEvu9ecF LKe+YSP3J2gmDlXkQC1oOzXO7EFlmihVySgwaUv7dXFyB9+k9sprZpilR12n4zyR52r2 M2PA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=RiuOwIpPh651aZ2hK+ip9NyOtHBGpuZ3/Qaj6I00OOk=; b=6Wxd5bKtOu4iE6DWBX+Xi0MMPPD4byfOUihfxvU7/GAOWBUHjdoG1rwTnsfZgNqQo/ JqdiUE2xQR36R91ab/a/4FVuJne9rjXYX/0pifZpOCP8xtHO6nipEVozlDpuY1U2SLgv OfX2251KqAxfPWcYzbTGd4vPSKKfo5Rium3luiAAaS1dd/jH5WkgExieMBCIMoMorHQ2 9MuMzqFfiNRppMsosY0gXuDUuiThuXAVo1A5S9wJdt/4GHYsPkoQUpl9UI+tPBIUp1tQ azzczgzdgLj5yr1xwAUDnpgemlHHX2FdDzpFcfCWkGUJGqTFNsJ/gTDBBYgnJxEg7rns w2pg== X-Gm-Message-State: AFqh2kqaQKLDN4jqt55k1qvepjExP1Rp7lEU4YgIkHwlJwuv4bzPnM5C 5Y2K+qUcVvZEvD4GKFvFR/UcCA7iI1c= X-Google-Smtp-Source: AMrXdXveg1Kh9ZZYAfgIjrYh7UXOJObS89s4bzBCiHl58pmtPmayRHNC2u8DYnPJZvXdOLFcduzD0A== X-Received: by 2002:a5d:554f:0:b0:2be:14e:15f3 with SMTP id g15-20020a5d554f000000b002be014e15f3mr4438254wrw.53.1674012626893; Tue, 17 Jan 2023 19:30:26 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id g2-20020a5d64e2000000b002be063f6820sm8895726wri.81.2023.01.17.19.30.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 17 Jan 2023 19:30:26 -0800 (PST) Message-Id: In-Reply-To: References: Date: Wed, 18 Jan 2023 03:30:13 +0000 Subject: [PATCH v6 08/12] test-http-server: add simple authentication Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Derrick Stolee , Lessley Dennington , Matthew John Cheetham , M Hickford , Jeff Hostetler , Glen Choo , Victoria Dye , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsA==?= Bjarmason , Matthew John Cheetham , Matthew John Cheetham Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Matthew John Cheetham From: Matthew John Cheetham Add simple authentication to the test-http-server test helper. Authentication schemes and sets of valid tokens can be specified via a configuration file (in the normal gitconfig file format). Incoming requests are compared against the set of valid schemes and tokens and only approved if a matching token is found, or if no auth was provided and anonymous auth is enabled. Configuration for auth includes a simple set of three options: [auth] challenge = [:] token = :[]* allowAnonymous = `auth.challenge` allows you define what authentication schemes, and optional challenge parameters the server should use. Scheme names are unique and subsequently specified challenge parameters in the config file will replace previously specified ones. `auth.token` allows you to define the set of value token values for an authentication scheme. This is a multi-var and each entry in the config file will append to the set of valid tokens for that scheme. Specifying an empty token value will clear the list of tokens so far for that scheme, i.e. `token = :`. `auth.allowAnonymous` controls whether or not unauthenticated requests (those without any `Authorization` headers) should succeed or not, and trigger a 401 Unauthorized response. Signed-off-by: Matthew John Cheetham --- t/helper/test-http-server.c | 233 +++++++++++++++++++++++++++++++++++- t/t5556-http-auth.sh | 43 ++++++- 2 files changed, 273 insertions(+), 3 deletions(-) diff --git a/t/helper/test-http-server.c b/t/helper/test-http-server.c index ae17c738259..691fbfb51d6 100644 --- a/t/helper/test-http-server.c +++ b/t/helper/test-http-server.c @@ -7,6 +7,7 @@ #include "version.h" #include "dir.h" #include "date.h" +#include "config.h" #define TR2_CAT "test-http-server" @@ -19,6 +20,7 @@ static const char test_http_auth_usage[] = " [--timeout=] [--max-connections=]\n" " [--reuseaddr] [--pid-file=]\n" " [--listen=]* [--port=]\n" +" [--auth-config=]\n" ; static unsigned int timeout; @@ -317,7 +319,7 @@ static int is_git_request(struct req *req) !regexec(smart_http_regex, req->uri_path.buf, 0, NULL, 0); } -static enum worker_result do__git(struct req *req) +static enum worker_result do__git(struct req *req, const char *user) { const char *ok = "HTTP/1.1 200 OK\r\n"; struct child_process cp = CHILD_PROCESS_INIT; @@ -334,10 +336,16 @@ static enum worker_result do__git(struct req *req) * exit status of the process, then write the HTTP status line followed * by the http-backend output. This is outside of the scope of this test * helper's use at time of writing. + * + * The important auth responses (401) we are handling prior to getting + * to this point. */ if (write(STDOUT_FILENO, ok, strlen(ok)) < 0) return error(_("could not send '%s'"), ok); + if (user) + strvec_pushf(&cp.env, "REMOTE_USER=%s", user); + strvec_pushf(&cp.env, "REQUEST_METHOD=%s", req->method); strvec_pushf(&cp.env, "PATH_TRANSLATED=%s", req->uri_path.buf); @@ -359,10 +367,218 @@ static enum worker_result do__git(struct req *req) return !!res; } +enum auth_result { + /* No auth module matches the request. */ + AUTH_UNKNOWN = 0, + + /* Auth module denied the request. */ + AUTH_DENY = 1, + + /* Auth module successfully validated the request. */ + AUTH_ALLOW = 2, +}; + +struct auth_module { + char *scheme; + char *challenge_params; + struct string_list *tokens; +}; + +static int allow_anonymous; +static struct auth_module **auth_modules = NULL; +static size_t auth_modules_nr = 0; +static size_t auth_modules_alloc = 0; + +static struct auth_module *get_auth_module(const char *scheme, int create) +{ + int i; + struct auth_module *mod; + for (i = 0; i < auth_modules_nr; i++) { + mod = auth_modules[i]; + if (!strcasecmp(mod->scheme, scheme)) + return mod; + } + + if (create) { + struct auth_module *mod = xmalloc(sizeof(struct auth_module)); + mod->scheme = xstrdup(scheme); + mod->challenge_params = NULL; + CALLOC_ARRAY(mod->tokens, 1); + string_list_init_dup(mod->tokens); + + ALLOC_GROW(auth_modules, auth_modules_nr + 1, auth_modules_alloc); + auth_modules[auth_modules_nr++] = mod; + + return mod; + } + + return NULL; +} + +static int is_authed(struct req *req, const char **user, enum worker_result *wr) +{ + enum auth_result result = AUTH_UNKNOWN; + struct string_list hdrs = STRING_LIST_INIT_NODUP; + struct auth_module *mod; + + struct string_list_item *hdr; + struct string_list_item *token; + const char *v; + struct strbuf **split = NULL; + int i; + char *challenge; + + /* + * Check all auth modules and try to validate the request. + * The first Authorization header that matches a known auth module + * scheme will be consulted to either approve or deny the request. + * If no module is found, or if there is no valid token, then 401 error. + * Otherwise, only permit the request if anonymous auth is enabled. + * It's atypical for user agents/clients to send multiple Authorization + * headers, but not explicitly forbidden or defined. + */ + for_each_string_list_item(hdr, &req->header_list) { + if (skip_iprefix(hdr->string, "Authorization: ", &v)) { + split = strbuf_split_str(v, ' ', 2); + if (!split[0] || !split[1]) continue; + + /* trim trailing space ' ' */ + strbuf_setlen(split[0], split[0]->len - 1); + + mod = get_auth_module(split[0]->buf, 0); + if (mod) { + result = AUTH_DENY; + + for_each_string_list_item(token, mod->tokens) { + if (!strcmp(split[1]->buf, token->string)) { + result = AUTH_ALLOW; + break; + } + } + + goto done; + } + } + } + +done: + switch (result) { + case AUTH_ALLOW: + trace2_printf("%s: auth '%s' ALLOW", TR2_CAT, mod->scheme); + *user = "VALID_TEST_USER"; + *wr = WR_OK; + break; + + case AUTH_DENY: + trace2_printf("%s: auth '%s' DENY", TR2_CAT, mod->scheme); + /* fall-through */ + + case AUTH_UNKNOWN: + if (result != AUTH_DENY && allow_anonymous) + break; + + for (i = 0; i < auth_modules_nr; i++) { + mod = auth_modules[i]; + if (mod->challenge_params) + challenge = xstrfmt("WWW-Authenticate: %s %s", + mod->scheme, + mod->challenge_params); + else + challenge = xstrfmt("WWW-Authenticate: %s", + mod->scheme); + string_list_append(&hdrs, challenge); + } + + *wr = send_http_error(STDOUT_FILENO, 401, "Unauthorized", -1, + &hdrs, *wr); + } + + strbuf_list_free(split); + string_list_clear(&hdrs, 0); + + return result == AUTH_ALLOW || + (result == AUTH_UNKNOWN && allow_anonymous); +} + +static int split_auth_param(const char *str, char **scheme, char **val) +{ + struct strbuf **p = strbuf_split_str(str, ':', 2); + + if (!p[0]) + return -1; + + /* trim trailing ':' */ + if (p[0]->len > 0 && p[0]->buf[p[0]->len - 1] == ':') + strbuf_setlen(p[0], p[0]->len - 1); + + *scheme = strbuf_detach(p[0], NULL); + + if (p[1]) + *val = strbuf_detach(p[1], NULL); + + strbuf_list_free(p); + return 0; +} + +static int read_auth_config(const char *name, const char *val, void *data) +{ + int ret = 0; + char *scheme = NULL; + char *token = NULL; + char *challenge = NULL; + struct auth_module *mod = NULL; + + if (!strcmp(name, "auth.challenge")) { + if (split_auth_param(val, &scheme, &challenge)) { + ret = error("invalid auth challenge '%s'", val); + goto cleanup; + } + + mod = get_auth_module(scheme, 1); + + /* Replace any existing challenge parameters */ + free(mod->challenge_params); + mod->challenge_params = challenge ? xstrdup(challenge) : NULL; + } else if (!strcmp(name, "auth.token")) { + if (split_auth_param(val, &scheme, &token)) { + ret = error("invalid auth token '%s'", val); + goto cleanup; + } + + mod = get_auth_module(scheme, 1); + + /* + * Append to set of valid tokens unless an empty token value + * is provided, then clear the existing list. + */ + if (token) + string_list_append(mod->tokens, token); + else + string_list_clear(mod->tokens, 1); + } else if (!strcmp(name, "auth.allowanonymous")) { + allow_anonymous = git_config_bool(name, val); + } else { + warning("unknown auth config '%s'", name); + } + +cleanup: + free(scheme); + free(token); + free(challenge); + + return ret; +} + static enum worker_result dispatch(struct req *req) { + enum worker_result wr = WR_OK; + const char *user = NULL; + + if (!is_authed(req, &user, &wr)) + return wr; + if (is_git_request(req)) - return do__git(req); + return do__git(req, user); return send_http_error(STDOUT_FILENO, 501, "Not Implemented", -1, NULL, WR_OK | WR_HANGUP); @@ -621,6 +837,19 @@ int cmd_main(int argc, const char **argv) pid_file = v; continue; } + if (skip_prefix(arg, "--auth-config=", &v)) { + if (!strlen(v)) { + error("invalid argument - missing file path"); + usage(test_http_auth_usage); + } + + if (git_config_from_file(read_auth_config, v, NULL)) { + error("failed to read auth config file '%s'", v); + usage(test_http_auth_usage); + } + + continue; + } fprintf(stderr, "error: unknown argument '%s'\n", arg); usage(test_http_auth_usage); diff --git a/t/t5556-http-auth.sh b/t/t5556-http-auth.sh index ce1abffa6aa..cb5562a41bf 100755 --- a/t/t5556-http-auth.sh +++ b/t/t5556-http-auth.sh @@ -95,10 +95,51 @@ per_test_cleanup () { rm -f OUT.* } +test_expect_success CURL 'http auth server auth config' ' + #test_when_finished "per_test_cleanup" && + + cat >auth.config <<-EOF && + [auth] + challenge = no-params + challenge = with-params:foo=\"bar\" p=1 + challenge = with-params:foo=\"replaced\" q=1 + + token = no-explicit-challenge:valid-token + token = no-explicit-challenge:also-valid + token = reset-tokens:these-tokens + token = reset-tokens:will-be-reset + token = reset-tokens: + token = reset-tokens:the-only-valid-one + + allowAnonymous = false + EOF + + cat >OUT.expected <<-EOF && + WWW-Authenticate: no-params + WWW-Authenticate: with-params foo="replaced" q=1 + WWW-Authenticate: no-explicit-challenge + WWW-Authenticate: reset-tokens + + Error: 401 Unauthorized + EOF + + start_http_server --auth-config="$TRASH_DIRECTORY/auth.config" && + + curl --include $ORIGIN_URL >OUT.curl && + tr -d "\r" OUT.actual && + + test_cmp OUT.expected OUT.actual +' + test_expect_success 'http auth anonymous no challenge' ' test_when_finished "per_test_cleanup" && - start_http_server && + cat >auth.config <<-EOF && + [auth] + allowAnonymous = true + EOF + + start_http_server --auth-config="$TRASH_DIRECTORY/auth.config" && # Attempt to read from a protected repository git ls-remote $ORIGIN_URL From patchwork Wed Jan 18 03:30:14 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew John Cheetham X-Patchwork-Id: 13105491 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5067CC6379F for ; Wed, 18 Jan 2023 03:30:58 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229822AbjARDa5 (ORCPT ); Tue, 17 Jan 2023 22:30:57 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51890 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229765AbjARDae (ORCPT ); Tue, 17 Jan 2023 22:30:34 -0500 Received: from mail-wm1-x333.google.com (mail-wm1-x333.google.com [IPv6:2a00:1450:4864:20::333]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 37A4F53B35 for ; Tue, 17 Jan 2023 19:30:29 -0800 (PST) Received: by mail-wm1-x333.google.com with SMTP id m5-20020a05600c4f4500b003db03b2559eso557517wmq.5 for ; Tue, 17 Jan 2023 19:30:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:from:to:cc:subject:date :message-id:reply-to; bh=tELCOU0LU2YgYDUa6yeAM5xwoAzYpT9GhRqvk/i/2MU=; b=Rk2nBMJymouKrs2Bw+X1OQgPwOWB4/se8pWTtAblKcWEzGMkchDFJddTpvA3d6iOsJ gGpQ+c9CMxc47bkxlIz5TfJ5ak/AiPWgKHfSDBT8Wkz6b5jimWVR1WjUa78ilX8iAe3u xAZWLPhoCMugjxd0VVjCVOk/uLOB79Htenf2rmPHkFyCT7LB6q0Rg9U8qj45YpwcRgdb id+UPKkAT+eZ51YqXDVcZAafe0yIZfV2+QdPyAxzuUNwStQWv1IK6vCNkgrRqqghPJww yiVO1mJb5vIqJ3uIfGKfx6nDZxo7T98aqzlDZDlLOkqhCBkhw2AXEsTvpR+buYFfUmRQ QHMA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=tELCOU0LU2YgYDUa6yeAM5xwoAzYpT9GhRqvk/i/2MU=; b=OxRB5vz+QiMV2iBx6HseDsbhG/vX0XlxMlyLB0EqoIvFaBdBWbtVaYm6d1KfPffF2Q EIjkjO1jFT+GUCmzUOduqVgn9QYzi6eMax8dYHUsrRS/U3yuSdslQh+m/K4qm8ZIqzh7 2FANCdtNBu4BNIAKvHVOFVwAbjlH+2D9zY0avbJW9YGWpBQnbrjux1rB9e3oEDk6eevR YlYE9v21nNMmpYOnlInJwkx34oLFl2JvxXljfO1XuAiyWFVbq2c68WlhoZPIINKAYIPa mUkvIPGCiAJKdjjj/EU8+xf466ey5nACdxM95tpPTlU7XldhglwXW4zCm7mprScIMzGH EtGw== X-Gm-Message-State: AFqh2ko31SkmTsPfx8Z4FxhFZ8w/Z5w1iWI+/64HSaifL2rOenmXQAca sjbvfSU3a4dyfizyHVFDCMsiE6zdfwY= X-Google-Smtp-Source: AMrXdXs/ej6BRR4QqVKGD19nT5lKfaR2a5WtzcQZq4sZ07FMYwVjkzeuB10wa8c60AEybLqAzr4TCw== X-Received: by 2002:a05:600c:1c1b:b0:3d9:ebf9:7004 with SMTP id j27-20020a05600c1c1b00b003d9ebf97004mr4942621wms.29.1674012627699; Tue, 17 Jan 2023 19:30:27 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id g22-20020a7bc4d6000000b003daffc2ecdesm616811wmk.13.2023.01.17.19.30.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 17 Jan 2023 19:30:27 -0800 (PST) Message-Id: <76125cdf239df7bebc63a27099e68e71a8216798.1674012618.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Wed, 18 Jan 2023 03:30:14 +0000 Subject: [PATCH v6 09/12] test-http-server: add sending of arbitrary headers Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Derrick Stolee , Lessley Dennington , Matthew John Cheetham , M Hickford , Jeff Hostetler , Glen Choo , Victoria Dye , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsA==?= Bjarmason , Matthew John Cheetham , Matthew John Cheetham Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Matthew John Cheetham From: Matthew John Cheetham Add the ability to send arbitrary headers in HTTP responses from the test-http-server. This is useful when we want to test 'malformed' response message handling. Add the following option to the server auth config file: [auth] extraHeader = []* Each `auth.extraHeader` value will be appended to the response headers verbatim. Signed-off-by: Matthew John Cheetham --- t/helper/test-http-server.c | 6 ++++++ t/t5556-http-auth.sh | 7 +++++++ 2 files changed, 13 insertions(+) diff --git a/t/helper/test-http-server.c b/t/helper/test-http-server.c index 691fbfb51d6..cbaee4fc0f4 100644 --- a/t/helper/test-http-server.c +++ b/t/helper/test-http-server.c @@ -388,6 +388,7 @@ static int allow_anonymous; static struct auth_module **auth_modules = NULL; static size_t auth_modules_nr = 0; static size_t auth_modules_alloc = 0; +static struct strvec extra_headers = STRVEC_INIT; static struct auth_module *get_auth_module(const char *scheme, int create) { @@ -489,6 +490,9 @@ done: string_list_append(&hdrs, challenge); } + for (i = 0; i < extra_headers.nr; i++) + string_list_append(&hdrs, extra_headers.v[i]); + *wr = send_http_error(STDOUT_FILENO, 401, "Unauthorized", -1, &hdrs, *wr); } @@ -557,6 +561,8 @@ static int read_auth_config(const char *name, const char *val, void *data) string_list_clear(mod->tokens, 1); } else if (!strcmp(name, "auth.allowanonymous")) { allow_anonymous = git_config_bool(name, val); + } else if (!strcmp(name, "auth.extraheader")) { + strvec_push(&extra_headers, val); } else { warning("unknown auth config '%s'", name); } diff --git a/t/t5556-http-auth.sh b/t/t5556-http-auth.sh index cb5562a41bf..e36107ea95d 100755 --- a/t/t5556-http-auth.sh +++ b/t/t5556-http-auth.sh @@ -112,6 +112,10 @@ test_expect_success CURL 'http auth server auth config' ' token = reset-tokens:the-only-valid-one allowAnonymous = false + + extraHeader = X-Extra-Header: abc + extraHeader = X-Extra-Header: 123 + extraHeader = X-Another: header\twith\twhitespace! EOF cat >OUT.expected <<-EOF && @@ -119,6 +123,9 @@ test_expect_success CURL 'http auth server auth config' ' WWW-Authenticate: with-params foo="replaced" q=1 WWW-Authenticate: no-explicit-challenge WWW-Authenticate: reset-tokens + X-Extra-Header: abc + X-Extra-Header: 123 + X-Another: header with whitespace! Error: 401 Unauthorized EOF From patchwork Wed Jan 18 03:30:15 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew John Cheetham X-Patchwork-Id: 13105492 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D123BC38142 for ; Wed, 18 Jan 2023 03:31:00 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229823AbjARDa6 (ORCPT ); Tue, 17 Jan 2023 22:30:58 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51946 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229762AbjARDae (ORCPT ); Tue, 17 Jan 2023 22:30:34 -0500 Received: from mail-wm1-x330.google.com (mail-wm1-x330.google.com [IPv6:2a00:1450:4864:20::330]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E070C530F2 for ; Tue, 17 Jan 2023 19:30:29 -0800 (PST) Received: by mail-wm1-x330.google.com with SMTP id f25-20020a1c6a19000000b003da221fbf48so567125wmc.1 for ; Tue, 17 Jan 2023 19:30:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:from:to:cc:subject:date :message-id:reply-to; bh=qCY5CaeVsGFH4YCKVAhrjSkUdtoSk2eEGMnVuvUbaAA=; b=KArevTKHxNA9sahTX4G3fPdoLd2FzIM7U3Fu6cL7g9vT85XOZ3FO6AHgqRFFYvrLNF zTSTOvbN83unrcMQHCpBev6HGuT5HdyPlIvGwT/WGU0EyF37zbbEOQ9E11Me+T+vX2F4 xH6w84JVLDHfJ/0CYkN62FMf4lttAt84G7LUy2WNIrTFRfufqKj60+Apt20Yak7gWy+t m3Y7oWLshHjz8mN3N1Xr17p11e+AuyTuofteUnh5S9IceiqQ7BGZ0DwPpgCy35Jr4Tc9 Hhk0R012romINAZiFlbqBQjMKa1n1cNyx/SbmHgNoY1s7PJBVG1M50ZGfdrgoik4x/bC rD9w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=qCY5CaeVsGFH4YCKVAhrjSkUdtoSk2eEGMnVuvUbaAA=; b=vfq9fvmUvzqeHHlfw7ulvNKeteDo/lp+FQMsULngXmX6Jp6CVrSctc0pDqom6UffL5 X4/VnwwnG73/kGTdXHvMo7W3D3/zMpsDdVP7LliyoLEbzQ7INPadSdtl1L8hbLMQm87W BQ9HPC9iWCwZ2V2rJBlBu12uoMfV1y1hS0Vy82Rm9d2gOE7+R008MPmDyujrXgIJklC0 exfEFHQCIvfZ8lBHyKUcWEBXw4F378fAuDusLv2WdvELpOdZsfNacsqRgwzdyAw5fEJ9 uvX/37uHB8FFPRjTZMbSJgT3OPsRy/a1HpFpwLUHbdB2mSjKStCO/lwd76IYCVRXc+Dl WcSg== X-Gm-Message-State: AFqh2krcdoBCz8DAKhgwkE2vVRRvMYXdTSNvGEe1sEzAFNV1whRtvOsG 5uynVmcLF/u2mCopD+F0p6uCMRoGhvU= X-Google-Smtp-Source: AMrXdXuKWWrAz29vN+f0Qke8FmKDwCautmDsMOTDtsKL2+sw8liOpmzlt0LCi4+aXItFcCwdAmV0OA== X-Received: by 2002:a05:600c:54eb:b0:3da:2090:d404 with SMTP id jb11-20020a05600c54eb00b003da2090d404mr5079997wmb.18.1674012628370; Tue, 17 Jan 2023 19:30:28 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id f17-20020a1c6a11000000b003d9fba3c7a4sm605900wmc.16.2023.01.17.19.30.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 17 Jan 2023 19:30:28 -0800 (PST) Message-Id: In-Reply-To: References: Date: Wed, 18 Jan 2023 03:30:15 +0000 Subject: [PATCH v6 10/12] http: replace unsafe size_t multiplication with st_mult Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Derrick Stolee , Lessley Dennington , Matthew John Cheetham , M Hickford , Jeff Hostetler , Glen Choo , Victoria Dye , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsA==?= Bjarmason , Matthew John Cheetham , Matthew John Cheetham Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Matthew John Cheetham From: Matthew John Cheetham Replace direct multiplication of two size_t parameters in curl response stream handling callback functions with `st_mult` to guard against overflows. Signed-off-by: Matthew John Cheetham --- http.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/http.c b/http.c index 8a5ba3f4776..a2a80318bb2 100644 --- a/http.c +++ b/http.c @@ -146,7 +146,7 @@ static int http_schannel_use_ssl_cainfo; size_t fread_buffer(char *ptr, size_t eltsize, size_t nmemb, void *buffer_) { - size_t size = eltsize * nmemb; + size_t size = st_mult(eltsize, nmemb); struct buffer *buffer = buffer_; if (size > buffer->buf.len - buffer->posn) @@ -176,7 +176,7 @@ curlioerr ioctl_buffer(CURL *handle, int cmd, void *clientp) size_t fwrite_buffer(char *ptr, size_t eltsize, size_t nmemb, void *buffer_) { - size_t size = eltsize * nmemb; + size_t size = st_mult(eltsize, nmemb); struct strbuf *buffer = buffer_; strbuf_add(buffer, ptr, size); From patchwork Wed Jan 18 03:30:16 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew John Cheetham X-Patchwork-Id: 13105494 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 23371C00A5A for ; Wed, 18 Jan 2023 03:31:30 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229832AbjARDb2 (ORCPT ); Tue, 17 Jan 2023 22:31:28 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51920 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229777AbjARDaf (ORCPT ); Tue, 17 Jan 2023 22:30:35 -0500 Received: from mail-wm1-x32a.google.com (mail-wm1-x32a.google.com [IPv6:2a00:1450:4864:20::32a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DDF9E53E45 for ; Tue, 17 Jan 2023 19:30:30 -0800 (PST) Received: by mail-wm1-x32a.google.com with SMTP id f12-20020a7bc8cc000000b003daf6b2f9b9so519170wml.3 for ; Tue, 17 Jan 2023 19:30:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:from:to:cc:subject:date :message-id:reply-to; bh=8E4ibgh1OESySHvLIe11OZBSXNLpINkcmnXPIcQ6rbo=; b=AWx+tmqSVGkbPhwKvwlqdY/RFCEBEERHfkfDM4oqAXKMeAN4dy3t1G1q2+os/WLq1p o9+Y6hMyaL7jEYx66iFC3qK/sswi8dj1ehd2/+wDPg1vKI0UWEm3/YQZHyBPsC5sXJEj 36MWGm14BVWfUnJnP8ojvSfK5qlyoO1pMsikuKoC3+S3WUq+4Gw66QpcdcMmxDy9VfGQ kH8s23EpBExln7xdv8jm6vVgo3H646vvf9AJZuCiA029A5TppfLgY9Ezw/WDPE2eZgsP dXsvWKpGztHePHlC1/4FrleaAYk8yAdduUsuixUxgeIiDfXcX7VxF57BDvQmQOYsfl9q FJzQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=8E4ibgh1OESySHvLIe11OZBSXNLpINkcmnXPIcQ6rbo=; b=IDhPAnMjaApOYuJlr3+2ilpefpHtXE9DpGHB+7kRchD5tfd4L+nYRVLP//Mrg3AeDE oZ8g2PKIvZMZXKM86pZl2bhqAo8Rxvf6Wz176TmpvdmMNNB2mFidZGvb27oCqPpaF3Qu 3Uf/UVjwoMHTIcJH+nyzzCWVK3okGHJY68JKetKJrRnK5D0mlm9gZWx8c/lTZh3v0Acm ZJDClBYzdLje0f650oJrNWnnmZGoZU+TM8Qi5qExOiZR7ZjNe1KJCIJgIjuPw7drdLf+ te284IuOoBKC/d0wKQLVN8HJo1CEv8I+Pwnla5Lix6v9uMdG8zKil38yT8E/fZd5AHtA Tb3w== X-Gm-Message-State: AFqh2kp7ZFqtuLUWlmT+JCrpqKcaYhroZFSb9RJyao0hUXpcINOfiTs5 w+XU07qCVxmc/e0tdmlCWSkZT8RjpbE= X-Google-Smtp-Source: AMrXdXvnfwDyqjFC67jQZt5Nr5wDdW17PXgFwNMsVoHHqdjH6roqUY5NJldFTMt0YrROSyy/55X/1w== X-Received: by 2002:a05:600c:511f:b0:3d0:bd9:edd4 with SMTP id o31-20020a05600c511f00b003d00bd9edd4mr5075688wms.0.1674012629137; Tue, 17 Jan 2023 19:30:29 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id l22-20020a05600c4f1600b003db0b0cc2afsm619782wmq.30.2023.01.17.19.30.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 17 Jan 2023 19:30:28 -0800 (PST) Message-Id: In-Reply-To: References: Date: Wed, 18 Jan 2023 03:30:16 +0000 Subject: [PATCH v6 11/12] http: read HTTP WWW-Authenticate response headers Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Derrick Stolee , Lessley Dennington , Matthew John Cheetham , M Hickford , Jeff Hostetler , Glen Choo , Victoria Dye , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsA==?= Bjarmason , Matthew John Cheetham , Matthew John Cheetham Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Matthew John Cheetham From: Matthew John Cheetham Read and store the HTTP WWW-Authenticate response headers made for a particular request. This will allow us to pass important authentication challenge information to credential helpers or others that would otherwise have been lost. According to RFC2616 Section 4.2 [1], header field names are not case-sensitive meaning when collecting multiple values for the same field name, we can just use the case of the first observed instance of each field name and no normalisation is required. libcurl only provides us with the ability to read all headers recieved for a particular request, including any intermediate redirect requests or proxies. The lines returned by libcurl include HTTP status lines delinating any intermediate requests such as "HTTP/1.1 200". We use these lines to reset the strvec of WWW-Authenticate header values as we encounter them in order to only capture the final response headers. The collection of all header values matching the WWW-Authenticate header is complicated by the fact that it is legal for header fields to be continued over multiple lines, but libcurl only gives us one line at a time. In the future [2] we may be able to leverage functions to read headers from libcurl itself, but as of today we must do this ourselves. [1] https://datatracker.ietf.org/doc/html/rfc2616#section-4.2 [2] https://daniel.haxx.se/blog/2022/03/22/a-headers-api-for-libcurl/ Signed-off-by: Matthew John Cheetham --- credential.c | 1 + credential.h | 15 +++++++++ http.c | 94 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 110 insertions(+) diff --git a/credential.c b/credential.c index f6389a50684..897b4679333 100644 --- a/credential.c +++ b/credential.c @@ -22,6 +22,7 @@ void credential_clear(struct credential *c) free(c->username); free(c->password); string_list_clear(&c->helpers, 0); + strvec_clear(&c->wwwauth_headers); credential_init(c); } diff --git a/credential.h b/credential.h index f430e77fea4..6f2e5bc610b 100644 --- a/credential.h +++ b/credential.h @@ -2,6 +2,7 @@ #define CREDENTIAL_H #include "string-list.h" +#include "strvec.h" /** * The credentials API provides an abstracted way of gathering username and @@ -115,6 +116,19 @@ struct credential { */ struct string_list helpers; + /** + * A `strvec` of WWW-Authenticate header values. Each string + * is the value of a WWW-Authenticate header in an HTTP response, + * in the order they were received in the response. + */ + struct strvec wwwauth_headers; + + /** + * Internal use only. Used to keep track of split header fields + * in order to fold multiple lines into one value. + */ + unsigned header_is_last_match:1; + unsigned approved:1, configured:1, quit:1, @@ -130,6 +144,7 @@ struct credential { #define CREDENTIAL_INIT { \ .helpers = STRING_LIST_INIT_DUP, \ + .wwwauth_headers = STRVEC_INIT, \ } /* Initialize a credential structure, setting all fields to empty. */ diff --git a/http.c b/http.c index a2a80318bb2..595c93bc7a3 100644 --- a/http.c +++ b/http.c @@ -183,6 +183,98 @@ size_t fwrite_buffer(char *ptr, size_t eltsize, size_t nmemb, void *buffer_) return nmemb; } +static size_t fwrite_wwwauth(char *ptr, size_t eltsize, size_t nmemb, void *p) +{ + size_t size = st_mult(eltsize, nmemb); + struct strvec *values = &http_auth.wwwauth_headers; + struct strbuf buf = STRBUF_INIT; + const char *val; + + /* + * Header lines may not come NULL-terminated from libcurl so we must + * limit all scans to the maximum length of the header line, or leverage + * strbufs for all operations. + * + * In addition, it is possible that header values can be split over + * multiple lines as per RFC 2616 (even though this has since been + * deprecated in RFC 7230). A continuation header field value is + * identified as starting with a space or horizontal tab. + * + * The formal definition of a header field as given in RFC 2616 is: + * + * message-header = field-name ":" [ field-value ] + * field-name = token + * field-value = *( field-content | LWS ) + * field-content = + */ + + strbuf_add(&buf, ptr, size); + + /* Strip the CRLF that should be present at the end of each field */ + strbuf_trim_trailing_newline(&buf); + + /* Start of a new WWW-Authenticate header */ + if (skip_iprefix(buf.buf, "www-authenticate:", &val)) { + while (isspace(*val)) + val++; + + strvec_push(values, val); + http_auth.header_is_last_match = 1; + goto exit; + } + + /* + * This line could be a continuation of the previously matched header + * field. If this is the case then we should append this value to the + * end of the previously consumed value. + * Continuation lines start with at least one whitespace, maybe more, + * so we should collapse these down to a single SP (valid per the spec). + */ + if (http_auth.header_is_last_match && isspace(*buf.buf)) { + /* Trim leading whitespace from this continuation hdr line. */ + strbuf_ltrim(&buf); + + /* + * At this point we should always have at least one existing + * value, even if it is empty. Do not bother appending the new + * value if this continuation header is itself empty. + */ + if (!values->nr) { + BUG("should have at least one existing header value"); + } else if (buf.len) { + char *prev = xstrdup(values->v[values->nr - 1]); + + /* Join two non-empty values with a single space. */ + const char *const sp = *prev ? " " : ""; + + strvec_pop(values); + strvec_pushf(values, "%s%s%s", prev, sp, buf.buf); + free(prev); + } + + goto exit; + } + + /* This is the start of a new header we don't care about */ + http_auth.header_is_last_match = 0; + + /* + * If this is a HTTP status line and not a header field, this signals + * a different HTTP response. libcurl writes all the output of all + * response headers of all responses, including redirects. + * We only care about the last HTTP request response's headers so clear + * the existing array. + */ + if (istarts_with(buf.buf, "http/")) + strvec_clear(values); + +exit: + strbuf_release(&buf); + return size; +} + size_t fwrite_null(char *ptr, size_t eltsize, size_t nmemb, void *strbuf) { return nmemb; @@ -1864,6 +1956,8 @@ static int http_request(const char *url, fwrite_buffer); } + curl_easy_setopt(slot->curl, CURLOPT_HEADERFUNCTION, fwrite_wwwauth); + accept_language = http_get_accept_language_header(); if (accept_language) From patchwork Wed Jan 18 03:30:17 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew John Cheetham X-Patchwork-Id: 13105495 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 32EFBC00A5A for ; Wed, 18 Jan 2023 03:31:38 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229842AbjARDbh (ORCPT ); Tue, 17 Jan 2023 22:31:37 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51986 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229784AbjARDag (ORCPT ); Tue, 17 Jan 2023 22:30:36 -0500 Received: from mail-wr1-x42b.google.com (mail-wr1-x42b.google.com [IPv6:2a00:1450:4864:20::42b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B9B7D51C4D for ; Tue, 17 Jan 2023 19:30:31 -0800 (PST) Received: by mail-wr1-x42b.google.com with SMTP id q10so13290580wrs.2 for ; Tue, 17 Jan 2023 19:30:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:from:to:cc:subject:date :message-id:reply-to; bh=oqov+LakXJJTebf7RM2+rAZA+HKTzRzGbXvTUFu8gW4=; b=egK81/GCigZo9dwWi1sMnXmcTMobT8qZKcVRX7Qf7LSkfcD1zY+50kf43pW4yaJkJZ T6m/9pswLNsOprwjOGj3FVp1N9gg8ZtF0mCWXvYP3lyWBxgL793IlD1WWRdXOrTYGbSc 80aQY/oR4c53oIsT0C7IhrAbt4PZo8Qh5hNgG07tLPZZWrm3HTSZaEObr0jG4toABfiQ egHmbcCPL5xp/gasK02g8k11yO3xF7EhYuNtPma96j91Tr07FymSGJHoC4HV4Z1C0zO9 90wu5rXW7tNPfIGHrCVcdbX18IHefAGsBuydguBneTqnQaBLj+u//fc/W70Eww0cGw8C pj4g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=oqov+LakXJJTebf7RM2+rAZA+HKTzRzGbXvTUFu8gW4=; b=AB13kS3VeGjPfNOvgt+s7fVUc1Ktd5OEdvSSIhDuB6cdriT43Z8Hnswsy9TIcgnHv0 qgyzFsdHynp38GVuJZGOru7WV5J2z/3arzhbg+Tl5Jdk3Lprjg+mhP7Qz97FlpYAKp2b 1PO1GSQUZU3678+tZwJAseZ8YNkE6WEyBdY1HnwqMqvQ4hX62uPKq/Iy9SRDYUOQZcW0 v9+4nemhpDYEiNljLGYCZMby3Jssz8poqmvwze0cve2m+pnVYk6FrIGGELZP757ltvp6 QuuYcTxQpyRGUTr7P3XugEziNWSWeceZe2VSIdMgi03vRDGKWSKEiRLouy9SlpY/bVN0 oOlg== X-Gm-Message-State: AFqh2kq14U1VfZAPEaGxgE5Ik/qNWmjfbJv557CSkwTicjeu4o6HzrXg 8PB8RmvPZHWknAApjctbNl09RuY7210= X-Google-Smtp-Source: AMrXdXvKe7GJpuoHjsUuV7rVA4Hw6iUSa4fm/HtyKWqTH1CEAu6K47UK4GLfMfvE10dRizsmAYhTcg== X-Received: by 2002:a05:6000:11c6:b0:2bd:bf72:76f2 with SMTP id i6-20020a05600011c600b002bdbf7276f2mr4301750wrx.14.1674012629939; Tue, 17 Jan 2023 19:30:29 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id t18-20020adff612000000b002be2f18938csm406245wrp.41.2023.01.17.19.30.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 17 Jan 2023 19:30:29 -0800 (PST) Message-Id: <7c8229f0b11693310ae47551fcc5e58f0bb64a0a.1674012618.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Wed, 18 Jan 2023 03:30:17 +0000 Subject: [PATCH v6 12/12] credential: add WWW-Authenticate header to cred requests Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Derrick Stolee , Lessley Dennington , Matthew John Cheetham , M Hickford , Jeff Hostetler , Glen Choo , Victoria Dye , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsA==?= Bjarmason , Matthew John Cheetham , Matthew John Cheetham Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Matthew John Cheetham From: Matthew John Cheetham Add the value of the WWW-Authenticate response header to credential requests. Credential helpers that understand and support HTTP authentication and authorization can use this standard header (RFC 2616 Section 14.47 [1]) to generate valid credentials. WWW-Authenticate headers can contain information pertaining to the authority, authentication mechanism, or extra parameters/scopes that are required. The current I/O format for credential helpers only allows for unique names for properties/attributes, so in order to transmit multiple header values (with a specific order) we introduce a new convention whereby a C-style array syntax is used in the property name to denote multiple ordered values for the same property. In this case we send multiple `wwwauth[]` properties where the order that the repeated attributes appear in the conversation reflects the order that the WWW-Authenticate headers appeared in the HTTP response. Add a set of tests to exercise the HTTP authentication header parsing and the interop with credential helpers. Credential helpers will receive WWW-Authenticate information in credential requests. [1] https://datatracker.ietf.org/doc/html/rfc2616#section-14.47 Signed-off-by: Matthew John Cheetham --- Documentation/git-credential.txt | 19 ++- credential.c | 11 ++ t/lib-credential-helper.sh | 27 ++++ t/t5556-http-auth.sh | 245 ++++++++++++++++++++++++++++++- 4 files changed, 300 insertions(+), 2 deletions(-) create mode 100644 t/lib-credential-helper.sh diff --git a/Documentation/git-credential.txt b/Documentation/git-credential.txt index ac2818b9f66..50759153ef1 100644 --- a/Documentation/git-credential.txt +++ b/Documentation/git-credential.txt @@ -113,7 +113,13 @@ separated by an `=` (equals) sign, followed by a newline. The key may contain any bytes except `=`, newline, or NUL. The value may contain any bytes except newline or NUL. -In both cases, all bytes are treated as-is (i.e., there is no quoting, +Attributes with keys that end with C-style array brackets `[]` can have +multiple values. Each instance of a multi-valued attribute forms an +ordered list of values - the order of the repeated attributes defines +the order of the values. An empty multi-valued attribute (`key[]=\n`) +acts to clear any previous entries and reset the list. + +In all cases, all bytes are treated as-is (i.e., there is no quoting, and one cannot transmit a value with newline or NUL in it). The list of attributes is terminated by a blank line or end-of-file. @@ -160,6 +166,17 @@ empty string. Components which are missing from the URL (e.g., there is no username in the example above) will be left unset. +`wwwauth[]`:: + + When an HTTP response is received by Git that includes one or more + 'WWW-Authenticate' authentication headers, these will be passed by Git + to credential helpers. ++ +Each 'WWW-Authenticate' header value is passed as a multi-valued +attribute 'wwwauth[]', where the order of the attributes is the same as +they appear in the HTTP response. This attribute is 'one-way' from Git +to pass additional information to credential helpers. + Unrecognised attributes are silently discarded. GIT diff --git a/credential.c b/credential.c index 897b4679333..9f39ebc3c7e 100644 --- a/credential.c +++ b/credential.c @@ -263,6 +263,16 @@ static void credential_write_item(FILE *fp, const char *key, const char *value, fprintf(fp, "%s=%s\n", key, value); } +static void credential_write_strvec(FILE *fp, const char *key, + const struct strvec *vec) +{ + char *full_key = xstrfmt("%s[]", key); + for (size_t i = 0; i < vec->nr; i++) { + credential_write_item(fp, full_key, vec->v[i], 0); + } + free(full_key); +} + void credential_write(const struct credential *c, FILE *fp) { credential_write_item(fp, "protocol", c->protocol, 1); @@ -270,6 +280,7 @@ void credential_write(const struct credential *c, FILE *fp) credential_write_item(fp, "path", c->path, 0); credential_write_item(fp, "username", c->username, 0); credential_write_item(fp, "password", c->password, 0); + credential_write_strvec(fp, "wwwauth", &c->wwwauth_headers); } static int run_credential_helper(struct credential *c, diff --git a/t/lib-credential-helper.sh b/t/lib-credential-helper.sh new file mode 100644 index 00000000000..8b0e4414234 --- /dev/null +++ b/t/lib-credential-helper.sh @@ -0,0 +1,27 @@ +setup_credential_helper() { + test_expect_success 'setup credential helper' ' + CREDENTIAL_HELPER="$TRASH_DIRECTORY/credential-helper.sh" && + export CREDENTIAL_HELPER && + echo $CREDENTIAL_HELPER && + + write_script "$CREDENTIAL_HELPER" <<-\EOF + cmd=$1 + teefile=$cmd-query.cred + catfile=$cmd-reply.cred + sed -n -e "/^$/q" -e "p" >> $teefile + if test "$cmd" = "get"; then + cat $catfile + fi + EOF + ' +} + +set_credential_reply() { + cat >"$TRASH_DIRECTORY/$1-reply.cred" +} + +expect_credential_query() { + cat >"$TRASH_DIRECTORY/$1-expect.cred" && + test_cmp "$TRASH_DIRECTORY/$1-expect.cred" \ + "$TRASH_DIRECTORY/$1-query.cred" +} diff --git a/t/t5556-http-auth.sh b/t/t5556-http-auth.sh index e36107ea95d..79122c611a1 100755 --- a/t/t5556-http-auth.sh +++ b/t/t5556-http-auth.sh @@ -4,6 +4,7 @@ test_description='test http auth header and credential helper interop' TEST_NO_CREATE_REPO=1 . ./test-lib.sh +. "$TEST_DIRECTORY"/lib-credential-helper.sh test_set_port GIT_TEST_HTTP_PROTOCOL_PORT @@ -33,6 +34,8 @@ test_expect_success 'setup repos' ' git -C "$REPO_DIR" branch -M main ' +setup_credential_helper + stop_http_server () { if ! test -f "$PID_FILE" then @@ -92,7 +95,9 @@ start_http_server () { per_test_cleanup () { stop_http_server && - rm -f OUT.* + rm -f OUT.* && + rm -f *.cred && + rm -f auth.config } test_expect_success CURL 'http auth server auth config' ' @@ -152,4 +157,242 @@ test_expect_success 'http auth anonymous no challenge' ' git ls-remote $ORIGIN_URL ' +test_expect_success 'http auth www-auth headers to credential helper basic valid' ' + test_when_finished "per_test_cleanup" && + # base64("alice:secret-passwd") + USERPASS64=YWxpY2U6c2VjcmV0LXBhc3N3ZA== && + export USERPASS64 && + + cat >auth.config <<-EOF && + [auth] + challenge = basic:realm=\"example.com\" + token = basic:$USERPASS64 + EOF + + start_http_server --auth-config="$TRASH_DIRECTORY/auth.config" && + + set_credential_reply get <<-EOF && + protocol=http + host=$HOST_PORT + username=alice + password=secret-passwd + EOF + + git -c "credential.helper=!\"$CREDENTIAL_HELPER\"" ls-remote $ORIGIN_URL && + + expect_credential_query get <<-EOF && + protocol=http + host=$HOST_PORT + wwwauth[]=basic realm="example.com" + EOF + + expect_credential_query store <<-EOF + protocol=http + host=$HOST_PORT + username=alice + password=secret-passwd + EOF +' + +test_expect_success 'http auth www-auth headers to credential helper ignore case valid' ' + test_when_finished "per_test_cleanup" && + # base64("alice:secret-passwd") + USERPASS64=YWxpY2U6c2VjcmV0LXBhc3N3ZA== && + export USERPASS64 && + + cat >auth.config <<-EOF && + [auth] + challenge = basic:realm=\"example.com\" + token = basic:$USERPASS64 + extraHeader = wWw-aUtHeNtIcAtE: bEaRer auThoRiTy=\"id.example.com\" + EOF + + start_http_server --auth-config="$TRASH_DIRECTORY/auth.config" && + + set_credential_reply get <<-EOF && + protocol=http + host=$HOST_PORT + username=alice + password=secret-passwd + EOF + + git -c "credential.helper=!\"$CREDENTIAL_HELPER\"" ls-remote $ORIGIN_URL && + + expect_credential_query get <<-EOF && + protocol=http + host=$HOST_PORT + wwwauth[]=basic realm="example.com" + wwwauth[]=bEaRer auThoRiTy="id.example.com" + EOF + + expect_credential_query store <<-EOF + protocol=http + host=$HOST_PORT + username=alice + password=secret-passwd + EOF +' + +test_expect_success 'http auth www-auth headers to credential helper continuation hdr' ' + test_when_finished "per_test_cleanup" && + # base64("alice:secret-passwd") + USERPASS64=YWxpY2U6c2VjcmV0LXBhc3N3ZA== && + export USERPASS64 && + + cat >auth.config <<-EOF && + [auth] + challenge = "bearer:authority=\"id.example.com\"\\n q=1\\n \\t p=0" + challenge = basic:realm=\"example.com\" + token = basic:$USERPASS64 + EOF + + start_http_server --auth-config="$TRASH_DIRECTORY/auth.config" && + + set_credential_reply get <<-EOF && + protocol=http + host=$HOST_PORT + username=alice + password=secret-passwd + EOF + + git -c "credential.helper=!\"$CREDENTIAL_HELPER\"" ls-remote $ORIGIN_URL && + + expect_credential_query get <<-EOF && + protocol=http + host=$HOST_PORT + wwwauth[]=bearer authority="id.example.com" q=1 p=0 + wwwauth[]=basic realm="example.com" + EOF + + expect_credential_query store <<-EOF + protocol=http + host=$HOST_PORT + username=alice + password=secret-passwd + EOF +' + +test_expect_success 'http auth www-auth headers to credential helper empty continuation hdrs' ' + test_when_finished "per_test_cleanup" && + # base64("alice:secret-passwd") + USERPASS64=YWxpY2U6c2VjcmV0LXBhc3N3ZA== && + export USERPASS64 && + + cat >auth.config <<-EOF && + [auth] + challenge = basic:realm=\"example.com\" + token = basic:$USERPASS64 + extraheader = "WWW-Authenticate:" + extraheader = " " + extraheader = " bearer authority=\"id.example.com\"" + EOF + + start_http_server --auth-config="$TRASH_DIRECTORY/auth.config" && + + set_credential_reply get <<-EOF && + protocol=http + host=$HOST_PORT + username=alice + password=secret-passwd + EOF + + git -c "credential.helper=!\"$CREDENTIAL_HELPER\"" ls-remote $ORIGIN_URL && + + expect_credential_query get <<-EOF && + protocol=http + host=$HOST_PORT + wwwauth[]=basic realm="example.com" + wwwauth[]=bearer authority="id.example.com" + EOF + + expect_credential_query store <<-EOF + protocol=http + host=$HOST_PORT + username=alice + password=secret-passwd + EOF +' + +test_expect_success 'http auth www-auth headers to credential helper custom schemes' ' + test_when_finished "per_test_cleanup" && + # base64("alice:secret-passwd") + USERPASS64=YWxpY2U6c2VjcmV0LXBhc3N3ZA== && + export USERPASS64 && + + cat >auth.config <<-EOF && + [auth] + challenge = "foobar:alg=test widget=1" + challenge = "bearer:authority=\"id.example.com\" q=1 p=0" + challenge = basic:realm=\"example.com\" + token = basic:$USERPASS64 + EOF + + start_http_server --auth-config="$TRASH_DIRECTORY/auth.config" && + + set_credential_reply get <<-EOF && + protocol=http + host=$HOST_PORT + username=alice + password=secret-passwd + EOF + + git -c "credential.helper=!\"$CREDENTIAL_HELPER\"" ls-remote $ORIGIN_URL && + + expect_credential_query get <<-EOF && + protocol=http + host=$HOST_PORT + wwwauth[]=foobar alg=test widget=1 + wwwauth[]=bearer authority="id.example.com" q=1 p=0 + wwwauth[]=basic realm="example.com" + EOF + + expect_credential_query store <<-EOF + protocol=http + host=$HOST_PORT + username=alice + password=secret-passwd + EOF +' + +test_expect_success 'http auth www-auth headers to credential helper invalid' ' + test_when_finished "per_test_cleanup" && + # base64("alice:secret-passwd") + USERPASS64=YWxpY2U6c2VjcmV0LXBhc3N3ZA== && + export USERPASS64 && + + cat >auth.config <<-EOF && + [auth] + challenge = "bearer:authority=\"id.example.com\" q=1 p=0" + challenge = basic:realm=\"example.com\" + token = basic:$USERPASS64 + EOF + + start_http_server --auth-config="$TRASH_DIRECTORY/auth.config" && + + set_credential_reply get <<-EOF && + protocol=http + host=$HOST_PORT + username=alice + password=invalid-passwd + EOF + + test_must_fail git -c "credential.helper=!\"$CREDENTIAL_HELPER\"" ls-remote $ORIGIN_URL && + + expect_credential_query get <<-EOF && + protocol=http + host=$HOST_PORT + wwwauth[]=bearer authority="id.example.com" q=1 p=0 + wwwauth[]=basic realm="example.com" + EOF + + expect_credential_query erase <<-EOF + protocol=http + host=$HOST_PORT + username=alice + password=invalid-passwd + wwwauth[]=bearer authority="id.example.com" q=1 p=0 + wwwauth[]=basic realm="example.com" + EOF +' + test_done