From patchwork Fri Jan 20 22:08:39 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew John Cheetham X-Patchwork-Id: 13110574 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 58101C27C76 for ; Fri, 20 Jan 2023 22:09:14 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229949AbjATWJN (ORCPT ); Fri, 20 Jan 2023 17:09:13 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58190 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229716AbjATWJA (ORCPT ); Fri, 20 Jan 2023 17:09:00 -0500 Received: from mail-wm1-x335.google.com (mail-wm1-x335.google.com [IPv6:2a00:1450:4864:20::335]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 35BECA25AF for ; Fri, 20 Jan 2023 14:08:55 -0800 (PST) Received: by mail-wm1-x335.google.com with SMTP id e19-20020a05600c439300b003db1cac0c1fso5279010wmn.5 for ; Fri, 20 Jan 2023 14:08:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:from:to:cc:subject:date :message-id:reply-to; bh=qZfvZ8RTcDBAofAevq0+3MPqq1v9AIkhChP91QQElkQ=; b=VvPdci32aZJRSYVBnp8rl+QHwGscqKNISgq1bM0hX12REB7xc/UnXVit0fagHSBZLf 5qn29hvIY8WqEoX51FRFcV5fn9HXplL5tATuQyPwsVk9X8heiG9x7IKz44cYIa/A36QQ NkCOKmZYyaeXYp/4+1D8cvwPT5VazIyzdj1k399kWvb7lW5gF/uy0HMveGUIdy82BxV7 zK246f8xMMZ3Jf4BsBKN5EPJ+n+kEx5IAM3MGN1lq+l4wyg2mehLJXQDmGhhuojciff/ tVbyuTZB0f0hzxLd99E27mNmuTweYLjfCB0vC/OJoBajNZuYfveu8Iiz5j4flMlmS13i nl3g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=qZfvZ8RTcDBAofAevq0+3MPqq1v9AIkhChP91QQElkQ=; b=iJyDAHL240wU7zSfkwLdYUeBLrRJ+xXaJKS4xVjBahjqB5w3aSZQHcgLD1JLEAXmYe HDgJ4HqrNZWh9NRawp8M3kdq/o3tk5h1aw+XdLJHW9YbUnDuWSS+QYqqJ6XXNTiouFAF 0/Rz96gWr7Cv8IWeCToVaTQTalNsmoGN1Gsucgo0EgI3vx07fkNdUe3mcwgixMcla9ua crAFV+Aw9p9A7uXWadlQu5nobmlLWd9YVauvRXsIq8tUShMSdtqAvWh9kk+8WQvtPETJ 4ZTwjJ7QQKQFb7rAVpHLP67323S9gFrRHwuCQtdF4mkuwvfUjfYIEMoF0LlVMDGlE5k7 ExbQ== X-Gm-Message-State: AFqh2krZ6qaIXgb5QqhD1A47C81634jpXlfYYpsQTOcJOU8RaIHizlr8 pxs9B3jPRUXtnTBv8yY5qtb2WRof4YY= X-Google-Smtp-Source: AMrXdXsDoUGSVSdtYjK+3sIyY+R0fjBwqmpdjGBNQTiB/AlR6w5auV/r8oYUfRWyu4mNX1vFa5YD0A== X-Received: by 2002:a05:600c:3488:b0:3d2:370b:97f4 with SMTP id a8-20020a05600c348800b003d2370b97f4mr24157409wmq.16.1674252533491; Fri, 20 Jan 2023 14:08:53 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id he7-20020a05600c540700b003d9fb04f658sm3373388wmb.4.2023.01.20.14.08.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 20 Jan 2023 14:08:53 -0800 (PST) Message-Id: <74b0de14185120c9d53d7470e59f57fa20a1927f.1674252531.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Fri, 20 Jan 2023 22:08:39 +0000 Subject: [PATCH v7 01/12] daemon: libify socket setup and option functions Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Derrick Stolee , Lessley Dennington , Matthew John Cheetham , M Hickford , Jeff Hostetler , Glen Choo , Victoria Dye , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsA==?= Bjarmason , Matthew John Cheetham , Matthew John Cheetham Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Matthew John Cheetham From: Matthew John Cheetham Extract functions for setting up listening sockets and keep-alive options from `daemon.c` to new `daemon-utils.{c,h}` files. Remove direct dependencies on global state by inlining the behaviour at the callsites for all libified functions. Signed-off-by: Matthew John Cheetham --- Makefile | 1 + daemon-utils.c | 209 +++++++++++++++++++++++++++++++++++++++++++++++ daemon-utils.h | 23 ++++++ daemon.c | 214 +------------------------------------------------ 4 files changed, 237 insertions(+), 210 deletions(-) create mode 100644 daemon-utils.c create mode 100644 daemon-utils.h diff --git a/Makefile b/Makefile index b258fdbed86..2654094dbb5 100644 --- a/Makefile +++ b/Makefile @@ -1003,6 +1003,7 @@ LIB_OBJS += credential.o LIB_OBJS += csum-file.o LIB_OBJS += ctype.o LIB_OBJS += date.o +LIB_OBJS += daemon-utils.o LIB_OBJS += decorate.o LIB_OBJS += delta-islands.o LIB_OBJS += diagnose.o diff --git a/daemon-utils.c b/daemon-utils.c new file mode 100644 index 00000000000..b96b55962db --- /dev/null +++ b/daemon-utils.c @@ -0,0 +1,209 @@ +#include "cache.h" +#include "daemon-utils.h" + +void set_keep_alive(int sockfd, log_fn logerror) +{ + int ka = 1; + + if (setsockopt(sockfd, SOL_SOCKET, SO_KEEPALIVE, &ka, sizeof(ka)) < 0) { + if (errno != ENOTSOCK) + logerror("unable to set SO_KEEPALIVE on socket: %s", + strerror(errno)); + } +} + +static int set_reuse_addr(int sockfd) +{ + int on = 1; + + return setsockopt(sockfd, SOL_SOCKET, SO_REUSEADDR, + &on, sizeof(on)); +} + +static const char *ip2str(int family, struct sockaddr *sin, socklen_t len) +{ +#ifdef NO_IPV6 + static char ip[INET_ADDRSTRLEN]; +#else + static char ip[INET6_ADDRSTRLEN]; +#endif + + switch (family) { +#ifndef NO_IPV6 + case AF_INET6: + inet_ntop(family, &((struct sockaddr_in6*)sin)->sin6_addr, ip, len); + break; +#endif + case AF_INET: + inet_ntop(family, &((struct sockaddr_in*)sin)->sin_addr, ip, len); + break; + default: + xsnprintf(ip, sizeof(ip), ""); + } + return ip; +} + +#ifndef NO_IPV6 + +static int setup_named_sock(char *listen_addr, int listen_port, + struct socketlist *socklist, int reuseaddr, + log_fn logerror) +{ + int socknum = 0; + char pbuf[NI_MAXSERV]; + struct addrinfo hints, *ai0, *ai; + int gai; + long flags; + + xsnprintf(pbuf, sizeof(pbuf), "%d", listen_port); + memset(&hints, 0, sizeof(hints)); + hints.ai_family = AF_UNSPEC; + hints.ai_socktype = SOCK_STREAM; + hints.ai_protocol = IPPROTO_TCP; + hints.ai_flags = AI_PASSIVE; + + gai = getaddrinfo(listen_addr, pbuf, &hints, &ai0); + if (gai) { + logerror("getaddrinfo() for %s failed: %s", listen_addr, gai_strerror(gai)); + return 0; + } + + for (ai = ai0; ai; ai = ai->ai_next) { + int sockfd; + + sockfd = socket(ai->ai_family, ai->ai_socktype, ai->ai_protocol); + if (sockfd < 0) + continue; + if (sockfd >= FD_SETSIZE) { + logerror("Socket descriptor too large"); + close(sockfd); + continue; + } + +#ifdef IPV6_V6ONLY + if (ai->ai_family == AF_INET6) { + int on = 1; + setsockopt(sockfd, IPPROTO_IPV6, IPV6_V6ONLY, + &on, sizeof(on)); + /* Note: error is not fatal */ + } +#endif + + if (reuseaddr && set_reuse_addr(sockfd)) { + logerror("Could not set SO_REUSEADDR: %s", strerror(errno)); + close(sockfd); + continue; + } + + set_keep_alive(sockfd, logerror); + + if (bind(sockfd, ai->ai_addr, ai->ai_addrlen) < 0) { + logerror("Could not bind to %s: %s", + ip2str(ai->ai_family, ai->ai_addr, ai->ai_addrlen), + strerror(errno)); + close(sockfd); + continue; /* not fatal */ + } + if (listen(sockfd, 5) < 0) { + logerror("Could not listen to %s: %s", + ip2str(ai->ai_family, ai->ai_addr, ai->ai_addrlen), + strerror(errno)); + close(sockfd); + continue; /* not fatal */ + } + + flags = fcntl(sockfd, F_GETFD, 0); + if (flags >= 0) + fcntl(sockfd, F_SETFD, flags | FD_CLOEXEC); + + ALLOC_GROW(socklist->list, socklist->nr + 1, socklist->alloc); + socklist->list[socklist->nr++] = sockfd; + socknum++; + } + + freeaddrinfo(ai0); + + return socknum; +} + +#else /* NO_IPV6 */ + +static int setup_named_sock(char *listen_addr, int listen_port, + struct socketlist *socklist, int reuseaddr, + log_fn logerror) +{ + struct sockaddr_in sin; + int sockfd; + long flags; + + memset(&sin, 0, sizeof sin); + sin.sin_family = AF_INET; + sin.sin_port = htons(listen_port); + + if (listen_addr) { + /* Well, host better be an IP address here. */ + if (inet_pton(AF_INET, listen_addr, &sin.sin_addr.s_addr) <= 0) + return 0; + } else { + sin.sin_addr.s_addr = htonl(INADDR_ANY); + } + + sockfd = socket(AF_INET, SOCK_STREAM, 0); + if (sockfd < 0) + return 0; + + if (reuseaddr && set_reuse_addr(sockfd)) { + logerror("Could not set SO_REUSEADDR: %s", strerror(errno)); + close(sockfd); + return 0; + } + + set_keep_alive(sockfd, logerror); + + if ( bind(sockfd, (struct sockaddr *)&sin, sizeof sin) < 0 ) { + logerror("Could not bind to %s: %s", + ip2str(AF_INET, (struct sockaddr *)&sin, sizeof(sin)), + strerror(errno)); + close(sockfd); + return 0; + } + + if (listen(sockfd, 5) < 0) { + logerror("Could not listen to %s: %s", + ip2str(AF_INET, (struct sockaddr *)&sin, sizeof(sin)), + strerror(errno)); + close(sockfd); + return 0; + } + + flags = fcntl(sockfd, F_GETFD, 0); + if (flags >= 0) + fcntl(sockfd, F_SETFD, flags | FD_CLOEXEC); + + ALLOC_GROW(socklist->list, socklist->nr + 1, socklist->alloc); + socklist->list[socklist->nr++] = sockfd; + return 1; +} + +#endif + +void socksetup(struct string_list *listen_addr, int listen_port, + struct socketlist *socklist, int reuseaddr, + log_fn logerror) +{ + if (!listen_addr->nr) + setup_named_sock(NULL, listen_port, socklist, reuseaddr, + logerror); + else { + int i, socknum; + for (i = 0; i < listen_addr->nr; i++) { + socknum = setup_named_sock(listen_addr->items[i].string, + listen_port, socklist, reuseaddr, + logerror); + + if (socknum == 0) + logerror("unable to allocate any listen sockets for host %s on port %u", + listen_addr->items[i].string, listen_port); + } + } +} diff --git a/daemon-utils.h b/daemon-utils.h new file mode 100644 index 00000000000..6710a2a6dc0 --- /dev/null +++ b/daemon-utils.h @@ -0,0 +1,23 @@ +#ifndef DAEMON_UTILS_H +#define DAEMON_UTILS_H + +#include "git-compat-util.h" +#include "string-list.h" + +typedef void (*log_fn)(const char *msg, ...); + +struct socketlist { + int *list; + size_t nr; + size_t alloc; +}; + +/* Enable sending of keep-alive messages on the socket. */ +void set_keep_alive(int sockfd, log_fn logerror); + +/* Setup a number of sockets to listen on the provided addresses. */ +void socksetup(struct string_list *listen_addr, int listen_port, + struct socketlist *socklist, int reuseaddr, + log_fn logerror); + +#endif diff --git a/daemon.c b/daemon.c index 0ae7d12b5c1..1ed4e705680 100644 --- a/daemon.c +++ b/daemon.c @@ -1,9 +1,9 @@ #include "cache.h" #include "config.h" +#include "daemon-utils.h" #include "pkt-line.h" #include "run-command.h" #include "strbuf.h" -#include "string-list.h" #ifdef NO_INITGROUPS #define initgroups(x, y) (0) /* nothing */ @@ -737,17 +737,6 @@ static void hostinfo_clear(struct hostinfo *hi) strbuf_release(&hi->tcp_port); } -static void set_keep_alive(int sockfd) -{ - int ka = 1; - - if (setsockopt(sockfd, SOL_SOCKET, SO_KEEPALIVE, &ka, sizeof(ka)) < 0) { - if (errno != ENOTSOCK) - logerror("unable to set SO_KEEPALIVE on socket: %s", - strerror(errno)); - } -} - static int execute(void) { char *line = packet_buffer; @@ -759,7 +748,7 @@ static int execute(void) if (addr) loginfo("Connection from %s:%s", addr, port); - set_keep_alive(0); + set_keep_alive(0, logerror); alarm(init_timeout ? init_timeout : timeout); pktlen = packet_read(0, packet_buffer, sizeof(packet_buffer), 0); alarm(0); @@ -938,202 +927,6 @@ static void child_handler(int signo) signal(SIGCHLD, child_handler); } -static int set_reuse_addr(int sockfd) -{ - int on = 1; - - if (!reuseaddr) - return 0; - return setsockopt(sockfd, SOL_SOCKET, SO_REUSEADDR, - &on, sizeof(on)); -} - -struct socketlist { - int *list; - size_t nr; - size_t alloc; -}; - -static const char *ip2str(int family, struct sockaddr *sin, socklen_t len) -{ -#ifdef NO_IPV6 - static char ip[INET_ADDRSTRLEN]; -#else - static char ip[INET6_ADDRSTRLEN]; -#endif - - switch (family) { -#ifndef NO_IPV6 - case AF_INET6: - inet_ntop(family, &((struct sockaddr_in6*)sin)->sin6_addr, ip, len); - break; -#endif - case AF_INET: - inet_ntop(family, &((struct sockaddr_in*)sin)->sin_addr, ip, len); - break; - default: - xsnprintf(ip, sizeof(ip), ""); - } - return ip; -} - -#ifndef NO_IPV6 - -static int setup_named_sock(char *listen_addr, int listen_port, struct socketlist *socklist) -{ - int socknum = 0; - char pbuf[NI_MAXSERV]; - struct addrinfo hints, *ai0, *ai; - int gai; - long flags; - - xsnprintf(pbuf, sizeof(pbuf), "%d", listen_port); - memset(&hints, 0, sizeof(hints)); - hints.ai_family = AF_UNSPEC; - hints.ai_socktype = SOCK_STREAM; - hints.ai_protocol = IPPROTO_TCP; - hints.ai_flags = AI_PASSIVE; - - gai = getaddrinfo(listen_addr, pbuf, &hints, &ai0); - if (gai) { - logerror("getaddrinfo() for %s failed: %s", listen_addr, gai_strerror(gai)); - return 0; - } - - for (ai = ai0; ai; ai = ai->ai_next) { - int sockfd; - - sockfd = socket(ai->ai_family, ai->ai_socktype, ai->ai_protocol); - if (sockfd < 0) - continue; - if (sockfd >= FD_SETSIZE) { - logerror("Socket descriptor too large"); - close(sockfd); - continue; - } - -#ifdef IPV6_V6ONLY - if (ai->ai_family == AF_INET6) { - int on = 1; - setsockopt(sockfd, IPPROTO_IPV6, IPV6_V6ONLY, - &on, sizeof(on)); - /* Note: error is not fatal */ - } -#endif - - if (set_reuse_addr(sockfd)) { - logerror("Could not set SO_REUSEADDR: %s", strerror(errno)); - close(sockfd); - continue; - } - - set_keep_alive(sockfd); - - if (bind(sockfd, ai->ai_addr, ai->ai_addrlen) < 0) { - logerror("Could not bind to %s: %s", - ip2str(ai->ai_family, ai->ai_addr, ai->ai_addrlen), - strerror(errno)); - close(sockfd); - continue; /* not fatal */ - } - if (listen(sockfd, 5) < 0) { - logerror("Could not listen to %s: %s", - ip2str(ai->ai_family, ai->ai_addr, ai->ai_addrlen), - strerror(errno)); - close(sockfd); - continue; /* not fatal */ - } - - flags = fcntl(sockfd, F_GETFD, 0); - if (flags >= 0) - fcntl(sockfd, F_SETFD, flags | FD_CLOEXEC); - - ALLOC_GROW(socklist->list, socklist->nr + 1, socklist->alloc); - socklist->list[socklist->nr++] = sockfd; - socknum++; - } - - freeaddrinfo(ai0); - - return socknum; -} - -#else /* NO_IPV6 */ - -static int setup_named_sock(char *listen_addr, int listen_port, struct socketlist *socklist) -{ - struct sockaddr_in sin; - int sockfd; - long flags; - - memset(&sin, 0, sizeof sin); - sin.sin_family = AF_INET; - sin.sin_port = htons(listen_port); - - if (listen_addr) { - /* Well, host better be an IP address here. */ - if (inet_pton(AF_INET, listen_addr, &sin.sin_addr.s_addr) <= 0) - return 0; - } else { - sin.sin_addr.s_addr = htonl(INADDR_ANY); - } - - sockfd = socket(AF_INET, SOCK_STREAM, 0); - if (sockfd < 0) - return 0; - - if (set_reuse_addr(sockfd)) { - logerror("Could not set SO_REUSEADDR: %s", strerror(errno)); - close(sockfd); - return 0; - } - - set_keep_alive(sockfd); - - if ( bind(sockfd, (struct sockaddr *)&sin, sizeof sin) < 0 ) { - logerror("Could not bind to %s: %s", - ip2str(AF_INET, (struct sockaddr *)&sin, sizeof(sin)), - strerror(errno)); - close(sockfd); - return 0; - } - - if (listen(sockfd, 5) < 0) { - logerror("Could not listen to %s: %s", - ip2str(AF_INET, (struct sockaddr *)&sin, sizeof(sin)), - strerror(errno)); - close(sockfd); - return 0; - } - - flags = fcntl(sockfd, F_GETFD, 0); - if (flags >= 0) - fcntl(sockfd, F_SETFD, flags | FD_CLOEXEC); - - ALLOC_GROW(socklist->list, socklist->nr + 1, socklist->alloc); - socklist->list[socklist->nr++] = sockfd; - return 1; -} - -#endif - -static void socksetup(struct string_list *listen_addr, int listen_port, struct socketlist *socklist) -{ - if (!listen_addr->nr) - setup_named_sock(NULL, listen_port, socklist); - else { - int i, socknum; - for (i = 0; i < listen_addr->nr; i++) { - socknum = setup_named_sock(listen_addr->items[i].string, - listen_port, socklist); - - if (socknum == 0) - logerror("unable to allocate any listen sockets for host %s on port %u", - listen_addr->items[i].string, listen_port); - } - } -} - static int service_loop(struct socketlist *socklist) { struct pollfd *pfd; @@ -1246,7 +1039,8 @@ static int serve(struct string_list *listen_addr, int listen_port, { struct socketlist socklist = { NULL, 0, 0 }; - socksetup(listen_addr, listen_port, &socklist); + socksetup(listen_addr, listen_port, &socklist, reuseaddr, + logerror); if (socklist.nr == 0) die("unable to allocate any listen sockets on port %u", listen_port); From patchwork Fri Jan 20 22:08:40 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew John Cheetham X-Patchwork-Id: 13110570 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B7987C25B50 for ; Fri, 20 Jan 2023 22:09:06 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229924AbjATWJG (ORCPT ); Fri, 20 Jan 2023 17:09:06 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58180 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229587AbjATWJA (ORCPT ); Fri, 20 Jan 2023 17:09:00 -0500 Received: from mail-wm1-x329.google.com (mail-wm1-x329.google.com [IPv6:2a00:1450:4864:20::329]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 36EADA314C for ; Fri, 20 Jan 2023 14:08:56 -0800 (PST) Received: by mail-wm1-x329.google.com with SMTP id l41-20020a05600c1d2900b003daf986faaeso4662020wms.3 for ; Fri, 20 Jan 2023 14:08:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:from:to:cc:subject:date :message-id:reply-to; bh=8QfNUqa5/m+ZSldR0jAxuOmtcZQE1zGK1yQCte9bfm4=; b=nZQSbD0FWZnQGAa/ezbE8WXiWiqn75fuXFvDbJKaaVPhFTNXA593uCTH5iGQsFtqg2 dFl/HEedhhB4bslm3b4LKLXC+d3RAHtVpHlMf4LFn7XO+I/7oDpMsLv+9dpgk6EHBGiC 4rFLnKoYNbOExiSQNQBF15A8k4zI59wFLOBCPoCaRoNmTSpF9W2d/nAINAU2w0HEHYLw M6eBxVjd9CS4FHrDyjn9MCvF0hpUBZPI9q3F26KZkk/8QnZfJIQXZbpEa6OtJ89B8y1z OwG0feuMxQg3JDkV7Fol6fiTFhKssHhssPjTi11fB1PJ3L9+HGP8AZAnDM92XicjeCGo lzNA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=8QfNUqa5/m+ZSldR0jAxuOmtcZQE1zGK1yQCte9bfm4=; b=eozYWs8G6MkIHzBJGJmXzjJfybUgGH47J2NvtoeCVTA2uL5bbgxCx+DRBT5G/yFbOF ZPe2ZWecu9o9at5UPwCDmcMX6jHarNhzwc7NhBcN237Ougd7t7ctc8FU2HRrSW8IVifp Q8Kz3L0LBjCUcs9EhoPcoPocqDF6voAI4i7gfAFwYsK2LmNbIqXV5ctjIPuB4r7NcCX2 Hru1fGLcEuaQyKv5AHuwRSz13XvMcSK2/aJymAHvD784hpondnkGTr8Bnvh6oUwk14Q+ WCWRWT/cUaTfj72lWIkh/k+9W80y3FsMKfVZqgneIUoh964gycoRAN5iPHiKgkgdPsWF HzPg== X-Gm-Message-State: AFqh2kqn5zHh0Gv/MWGpQL6cLM99FACHfJ0hhL3YDkd/tgqH3OQomV+B 7MMSAwtQUPe/9xAJiNYVXB8Fd3yxzqI= X-Google-Smtp-Source: AMrXdXtyeWR/XZdqUSFwgv+0KN5sNLYv7u7jTjFsUj0RoO5z0fYOVcgTZ6fhigTAXLJK8zbSkOThYA== X-Received: by 2002:a05:600c:34d1:b0:3db:1434:c51a with SMTP id d17-20020a05600c34d100b003db1434c51amr10538590wmq.40.1674252534381; Fri, 20 Jan 2023 14:08:54 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id x10-20020a1c7c0a000000b003da119d7251sm3308460wmc.21.2023.01.20.14.08.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 20 Jan 2023 14:08:54 -0800 (PST) Message-Id: In-Reply-To: References: Date: Fri, 20 Jan 2023 22:08:40 +0000 Subject: [PATCH v7 02/12] daemon: libify child process handling functions Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Derrick Stolee , Lessley Dennington , Matthew John Cheetham , M Hickford , Jeff Hostetler , Glen Choo , Victoria Dye , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsA==?= Bjarmason , Matthew John Cheetham , Matthew John Cheetham Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Matthew John Cheetham From: Matthew John Cheetham Extract functions and structures for managing child processes started from the parent daemon-like process from `daemon.c` to the new shared `daemon-utils.{c,h}` files. One minor functional change is introduced to `check_dead_children()` where the logging of a dead/disconnected child is now optional. With the 'libification' of these functions we extract the call to `loginfo` to a call to a function pointer, and guard the log message creation and logging behind a `NULL` check. Callers can now skip logging by passing `NULL` as the `log_fn loginfo` argument. The behaviour of callers in `daemon.c` remains the same (save one extra NULL check) however as a pointer to `loginfo` is always passed. Signed-off-by: Matthew John Cheetham --- daemon-utils.c | 77 ++++++++++++++++++++++++++++++++++++++++++ daemon-utils.h | 32 ++++++++++++++++++ daemon.c | 92 +++----------------------------------------------- 3 files changed, 114 insertions(+), 87 deletions(-) diff --git a/daemon-utils.c b/daemon-utils.c index b96b55962db..8506664b440 100644 --- a/daemon-utils.c +++ b/daemon-utils.c @@ -207,3 +207,80 @@ void socksetup(struct string_list *listen_addr, int listen_port, } } } + +static int addrcmp(const struct sockaddr_storage *s1, + const struct sockaddr_storage *s2) +{ + const struct sockaddr *sa1 = (const struct sockaddr*) s1; + const struct sockaddr *sa2 = (const struct sockaddr*) s2; + + if (sa1->sa_family != sa2->sa_family) + return sa1->sa_family - sa2->sa_family; + if (sa1->sa_family == AF_INET) + return memcmp(&((struct sockaddr_in *)s1)->sin_addr, + &((struct sockaddr_in *)s2)->sin_addr, + sizeof(struct in_addr)); +#ifndef NO_IPV6 + if (sa1->sa_family == AF_INET6) + return memcmp(&((struct sockaddr_in6 *)s1)->sin6_addr, + &((struct sockaddr_in6 *)s2)->sin6_addr, + sizeof(struct in6_addr)); +#endif + return 0; +} + +void add_child(struct child_process *cld, struct sockaddr *addr, socklen_t addrlen, + struct child *firstborn , unsigned int *live_children) +{ + struct child *newborn, **cradle; + + CALLOC_ARRAY(newborn, 1); + (*live_children)++; + memcpy(&newborn->cld, cld, sizeof(*cld)); + memcpy(&newborn->address, addr, addrlen); + for (cradle = &firstborn; *cradle; cradle = &(*cradle)->next) + if (!addrcmp(&(*cradle)->address, &newborn->address)) + break; + newborn->next = *cradle; + *cradle = newborn; +} + +void kill_some_child(struct child *firstborn) +{ + const struct child *blanket, *next; + + if (!(blanket = firstborn)) + return; + + for (; (next = blanket->next); blanket = next) + if (!addrcmp(&blanket->address, &next->address)) { + kill(blanket->cld.pid, SIGTERM); + break; + } +} + +void check_dead_children(struct child *firstborn, unsigned int *live_children, + log_fn loginfo) +{ + int status; + pid_t pid; + + struct child **cradle, *blanket; + for (cradle = &firstborn; (blanket = *cradle);) + if ((pid = waitpid(blanket->cld.pid, &status, WNOHANG)) > 1) { + if (loginfo) { + const char *dead = ""; + if (status) + dead = " (with error)"; + loginfo("[%"PRIuMAX"] Disconnected%s", + (uintmax_t)pid, dead); + } + + /* remove the child */ + *cradle = blanket->next; + (*live_children)--; + child_process_clear(&blanket->cld); + free(blanket); + } else + cradle = &blanket->next; +} diff --git a/daemon-utils.h b/daemon-utils.h index 6710a2a6dc0..97e5cae20b8 100644 --- a/daemon-utils.h +++ b/daemon-utils.h @@ -2,6 +2,7 @@ #define DAEMON_UTILS_H #include "git-compat-util.h" +#include "run-command.h" #include "string-list.h" typedef void (*log_fn)(const char *msg, ...); @@ -20,4 +21,35 @@ void socksetup(struct string_list *listen_addr, int listen_port, struct socketlist *socklist, int reuseaddr, log_fn logerror); +struct child { + struct child *next; + struct child_process cld; + struct sockaddr_storage address; +}; + +/* + * Add the child_process to the set of children and increment the number of + * live children. + */ +void add_child(struct child_process *cld, struct sockaddr *addr, socklen_t addrlen, + struct child *firstborn, unsigned int *live_children); + +/* + * Kill the newest connection from a duplicate IP. + * + * This function should be called if the number of connections grows + * past the maximum number of allowed connections. + */ +void kill_some_child(struct child *firstborn); + +/* + * Check for children that have disconnected and remove them from the + * active set, decrementing the number of live children. + * + * Optionally log the child PID that disconnected by passing a loginfo + * function. + */ +void check_dead_children(struct child *firstborn, unsigned int *live_children, + log_fn loginfo); + #endif diff --git a/daemon.c b/daemon.c index 1ed4e705680..ec3b407ecbc 100644 --- a/daemon.c +++ b/daemon.c @@ -785,93 +785,11 @@ static int execute(void) return -1; } -static int addrcmp(const struct sockaddr_storage *s1, - const struct sockaddr_storage *s2) -{ - const struct sockaddr *sa1 = (const struct sockaddr*) s1; - const struct sockaddr *sa2 = (const struct sockaddr*) s2; - - if (sa1->sa_family != sa2->sa_family) - return sa1->sa_family - sa2->sa_family; - if (sa1->sa_family == AF_INET) - return memcmp(&((struct sockaddr_in *)s1)->sin_addr, - &((struct sockaddr_in *)s2)->sin_addr, - sizeof(struct in_addr)); -#ifndef NO_IPV6 - if (sa1->sa_family == AF_INET6) - return memcmp(&((struct sockaddr_in6 *)s1)->sin6_addr, - &((struct sockaddr_in6 *)s2)->sin6_addr, - sizeof(struct in6_addr)); -#endif - return 0; -} - static int max_connections = 32; static unsigned int live_children; -static struct child { - struct child *next; - struct child_process cld; - struct sockaddr_storage address; -} *firstborn; - -static void add_child(struct child_process *cld, struct sockaddr *addr, socklen_t addrlen) -{ - struct child *newborn, **cradle; - - CALLOC_ARRAY(newborn, 1); - live_children++; - memcpy(&newborn->cld, cld, sizeof(*cld)); - memcpy(&newborn->address, addr, addrlen); - for (cradle = &firstborn; *cradle; cradle = &(*cradle)->next) - if (!addrcmp(&(*cradle)->address, &newborn->address)) - break; - newborn->next = *cradle; - *cradle = newborn; -} - -/* - * This gets called if the number of connections grows - * past "max_connections". - * - * We kill the newest connection from a duplicate IP. - */ -static void kill_some_child(void) -{ - const struct child *blanket, *next; - - if (!(blanket = firstborn)) - return; - - for (; (next = blanket->next); blanket = next) - if (!addrcmp(&blanket->address, &next->address)) { - kill(blanket->cld.pid, SIGTERM); - break; - } -} - -static void check_dead_children(void) -{ - int status; - pid_t pid; - - struct child **cradle, *blanket; - for (cradle = &firstborn; (blanket = *cradle);) - if ((pid = waitpid(blanket->cld.pid, &status, WNOHANG)) > 1) { - const char *dead = ""; - if (status) - dead = " (with error)"; - loginfo("[%"PRIuMAX"] Disconnected%s", (uintmax_t)pid, dead); - - /* remove the child */ - *cradle = blanket->next; - live_children--; - child_process_clear(&blanket->cld); - free(blanket); - } else - cradle = &blanket->next; -} +static struct child *firstborn; static struct strvec cld_argv = STRVEC_INIT; static void handle(int incoming, struct sockaddr *addr, socklen_t addrlen) @@ -879,9 +797,9 @@ static void handle(int incoming, struct sockaddr *addr, socklen_t addrlen) struct child_process cld = CHILD_PROCESS_INIT; if (max_connections && live_children >= max_connections) { - kill_some_child(); + kill_some_child(firstborn); sleep(1); /* give it some time to die */ - check_dead_children(); + check_dead_children(firstborn, &live_children, loginfo); if (live_children >= max_connections) { close(incoming); logerror("Too many children, dropping connection"); @@ -914,7 +832,7 @@ static void handle(int incoming, struct sockaddr *addr, socklen_t addrlen) if (start_command(&cld)) logerror("unable to fork"); else - add_child(&cld, addr, addrlen); + add_child(&cld, addr, addrlen, firstborn, &live_children); } static void child_handler(int signo) @@ -944,7 +862,7 @@ static int service_loop(struct socketlist *socklist) for (;;) { int i; - check_dead_children(); + check_dead_children(firstborn, &live_children, loginfo); if (poll(pfd, socklist->nr, -1) < 0) { if (errno != EINTR) { From patchwork Fri Jan 20 22:08:41 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew John Cheetham X-Patchwork-Id: 13110571 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C309DC27C76 for ; Fri, 20 Jan 2023 22:09:08 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229929AbjATWJH (ORCPT ); Fri, 20 Jan 2023 17:09:07 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58182 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229687AbjATWJA (ORCPT ); Fri, 20 Jan 2023 17:09:00 -0500 Received: from mail-wm1-x329.google.com (mail-wm1-x329.google.com [IPv6:2a00:1450:4864:20::329]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0D3B4A3179 for ; Fri, 20 Jan 2023 14:08:57 -0800 (PST) Received: by mail-wm1-x329.google.com with SMTP id c10-20020a05600c0a4a00b003db0636ff84so4681525wmq.0 for ; Fri, 20 Jan 2023 14:08:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:from:to:cc:subject:date :message-id:reply-to; bh=KsvXxRc41yRRXoCeuMp+jDk6Yf1HTqCPl2KPj4S0kA8=; b=gK+kKG7cs53nPdtDD8sejY0IQk6Th75xNhlmI1vWriXHn4P4yT5yC91On/Ds1Vt8/g sroc31LjFKb2A6z3HE0lVrg+X3vRqGWEY9TBVukCojhQvFWGVwo19XUWF9VCVEnB2UUy OfeiprStPUXYvUtPbVeJA+d+cgy7nFHImW7gpLu8dr5/pTYrsZfd/tB/8zvAGVI/40fZ TS5PvVHceA7Jh2H/EWK5XV/tHj4tvCfkMK6kcEO2WpsD8tqoc7Wv8KFimosCKuwYX9qQ Ew+crB8OBXk8Is7rhyklqQACVhGIHQwmtsTvvfCuHGAwUKIdf27lq1RzhVTh/XjNYvmy RWPA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=KsvXxRc41yRRXoCeuMp+jDk6Yf1HTqCPl2KPj4S0kA8=; b=435HZ0rv5f7RQ0BuauzX+/I/zPdcc7Ix2aUEnDp2nKP/ElQrhGmhpMFQhX4v77sNSA uhfrmNf6Dd6yhXHN894htEVMyiJ+U1Cjnqomo1H8kt6aFH95aW9NNftaJuxHxvcWu6Fk 9I4RP0hwOXnVcvremqKrMPtMKMNtbL76wZtOb+lSR20GIgnRfB9qV/f7gpIcoYTs5uiW ItF8LF3uIBkvvSnuSGna3w1yEJPeVSZV3ZnL+/Ak/Sk9Jh2+dlg2qGUoK6ZUaXHPH42n tJILWcezSZ1xU9sUCYDZrLZCReNTgzkgpFm6fF1YbiVVKD2/sZ9pdIvL01UuO6mOcdlS sPNQ== X-Gm-Message-State: AFqh2kqQvJQVfDJWNMfBp5WtUqJcBaXMveXhViYNC5EdhqRegs2FuT80 NOPWOzjZAlX0c8gqHq9dOablkeL0bho= X-Google-Smtp-Source: AMrXdXuXM+m1NpjdQXkJH47nAcZNzg9CgVxyR8yqFXJe/V3F/GKw/+93EXklJDj/cZAm7ENw4hWXbg== X-Received: by 2002:a05:600c:3b29:b0:3da:f678:1322 with SMTP id m41-20020a05600c3b2900b003daf6781322mr16481977wms.38.1674252535335; Fri, 20 Jan 2023 14:08:55 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id m27-20020a05600c3b1b00b003db012d49b7sm14069655wms.2.2023.01.20.14.08.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 20 Jan 2023 14:08:55 -0800 (PST) Message-Id: <9967401c972cab547d7619a208c3a0e6a3923cd4.1674252531.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Fri, 20 Jan 2023 22:08:41 +0000 Subject: [PATCH v7 03/12] daemon: rename some esoteric/laboured terminology Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Derrick Stolee , Lessley Dennington , Matthew John Cheetham , M Hickford , Jeff Hostetler , Glen Choo , Victoria Dye , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsA==?= Bjarmason , Matthew John Cheetham , Matthew John Cheetham Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Matthew John Cheetham From: Matthew John Cheetham Rename some of the variables and function arguments used to manage child processes. The existing names are esoteric; stretching an analogy too far to the point of being confusing to understand. Rename "firstborn" to "first_child", "newborn" to "new_cld", "blanket" to "current" and "cradle" to "ptr". Signed-off-by: Matthew John Cheetham --- daemon-utils.c | 46 +++++++++++++++++++++++----------------------- daemon-utils.h | 6 +++--- daemon.c | 10 +++++----- 3 files changed, 31 insertions(+), 31 deletions(-) diff --git a/daemon-utils.c b/daemon-utils.c index 8506664b440..f23ea35ed7b 100644 --- a/daemon-utils.c +++ b/daemon-utils.c @@ -230,44 +230,44 @@ static int addrcmp(const struct sockaddr_storage *s1, } void add_child(struct child_process *cld, struct sockaddr *addr, socklen_t addrlen, - struct child *firstborn , unsigned int *live_children) + struct child *first_child, unsigned int *live_children) { - struct child *newborn, **cradle; + struct child *new_cld, **current; - CALLOC_ARRAY(newborn, 1); + CALLOC_ARRAY(new_cld, 1); (*live_children)++; - memcpy(&newborn->cld, cld, sizeof(*cld)); - memcpy(&newborn->address, addr, addrlen); - for (cradle = &firstborn; *cradle; cradle = &(*cradle)->next) - if (!addrcmp(&(*cradle)->address, &newborn->address)) + memcpy(&new_cld->cld, cld, sizeof(*cld)); + memcpy(&new_cld->address, addr, addrlen); + for (current = &first_child; *current; current = &(*current)->next) + if (!addrcmp(&(*current)->address, &new_cld->address)) break; - newborn->next = *cradle; - *cradle = newborn; + new_cld->next = *current; + *current = new_cld; } -void kill_some_child(struct child *firstborn) +void kill_some_child(struct child *first_child) { - const struct child *blanket, *next; + const struct child *current, *next; - if (!(blanket = firstborn)) + if (!(current = first_child)) return; - for (; (next = blanket->next); blanket = next) - if (!addrcmp(&blanket->address, &next->address)) { - kill(blanket->cld.pid, SIGTERM); + for (; (next = current->next); current = next) + if (!addrcmp(¤t->address, &next->address)) { + kill(current->cld.pid, SIGTERM); break; } } -void check_dead_children(struct child *firstborn, unsigned int *live_children, +void check_dead_children(struct child *first_child, unsigned int *live_children, log_fn loginfo) { int status; pid_t pid; - struct child **cradle, *blanket; - for (cradle = &firstborn; (blanket = *cradle);) - if ((pid = waitpid(blanket->cld.pid, &status, WNOHANG)) > 1) { + struct child **ptr, *current; + for (ptr = &first_child; (current = *ptr);) + if ((pid = waitpid(current->cld.pid, &status, WNOHANG)) > 1) { if (loginfo) { const char *dead = ""; if (status) @@ -277,10 +277,10 @@ void check_dead_children(struct child *firstborn, unsigned int *live_children, } /* remove the child */ - *cradle = blanket->next; + *ptr = current->next; (*live_children)--; - child_process_clear(&blanket->cld); - free(blanket); + child_process_clear(¤t->cld); + free(current); } else - cradle = &blanket->next; + ptr = ¤t->next; } diff --git a/daemon-utils.h b/daemon-utils.h index 97e5cae20b8..c866e9c9a4e 100644 --- a/daemon-utils.h +++ b/daemon-utils.h @@ -32,7 +32,7 @@ struct child { * live children. */ void add_child(struct child_process *cld, struct sockaddr *addr, socklen_t addrlen, - struct child *firstborn, unsigned int *live_children); + struct child *first_child, unsigned int *live_children); /* * Kill the newest connection from a duplicate IP. @@ -40,7 +40,7 @@ void add_child(struct child_process *cld, struct sockaddr *addr, socklen_t addrl * This function should be called if the number of connections grows * past the maximum number of allowed connections. */ -void kill_some_child(struct child *firstborn); +void kill_some_child(struct child *first_child); /* * Check for children that have disconnected and remove them from the @@ -49,7 +49,7 @@ void kill_some_child(struct child *firstborn); * Optionally log the child PID that disconnected by passing a loginfo * function. */ -void check_dead_children(struct child *firstborn, unsigned int *live_children, +void check_dead_children(struct child *first_child, unsigned int *live_children, log_fn loginfo); #endif diff --git a/daemon.c b/daemon.c index ec3b407ecbc..d3e7d81de18 100644 --- a/daemon.c +++ b/daemon.c @@ -789,7 +789,7 @@ static int max_connections = 32; static unsigned int live_children; -static struct child *firstborn; +static struct child *first_child; static struct strvec cld_argv = STRVEC_INIT; static void handle(int incoming, struct sockaddr *addr, socklen_t addrlen) @@ -797,9 +797,9 @@ static void handle(int incoming, struct sockaddr *addr, socklen_t addrlen) struct child_process cld = CHILD_PROCESS_INIT; if (max_connections && live_children >= max_connections) { - kill_some_child(firstborn); + kill_some_child(first_child); sleep(1); /* give it some time to die */ - check_dead_children(firstborn, &live_children, loginfo); + check_dead_children(first_child, &live_children, loginfo); if (live_children >= max_connections) { close(incoming); logerror("Too many children, dropping connection"); @@ -832,7 +832,7 @@ static void handle(int incoming, struct sockaddr *addr, socklen_t addrlen) if (start_command(&cld)) logerror("unable to fork"); else - add_child(&cld, addr, addrlen, firstborn, &live_children); + add_child(&cld, addr, addrlen, first_child, &live_children); } static void child_handler(int signo) @@ -862,7 +862,7 @@ static int service_loop(struct socketlist *socklist) for (;;) { int i; - check_dead_children(firstborn, &live_children, loginfo); + check_dead_children(first_child, &live_children, loginfo); if (poll(pfd, socklist->nr, -1) < 0) { if (errno != EINTR) { From patchwork Fri Jan 20 22:08:42 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew John Cheetham X-Patchwork-Id: 13110573 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 45050C25B50 for ; Fri, 20 Jan 2023 22:09:13 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229947AbjATWJM (ORCPT ); Fri, 20 Jan 2023 17:09:12 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58192 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229864AbjATWJB (ORCPT ); Fri, 20 Jan 2023 17:09:01 -0500 Received: from mail-wm1-x32d.google.com (mail-wm1-x32d.google.com [IPv6:2a00:1450:4864:20::32d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E426FA500B for ; Fri, 20 Jan 2023 14:08:57 -0800 (PST) Received: by mail-wm1-x32d.google.com with SMTP id q8so5063054wmo.5 for ; Fri, 20 Jan 2023 14:08:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:from:to:cc:subject:date :message-id:reply-to; bh=i/fKd4Jtqfa/YFl4PxGgOhCP4o4uTXtc+mEJ6ZoI0h8=; b=dyJ2xPpi0surGE1bB1lV3D4cERsM2zNoLmAQrC/vyPGlbvgHPFbV+skxEuI9Wc0YD0 hLEBsxgFTIhsDY2UaUsVW3YEGL7gmH0+K628X3PcGxo6FtNWQch8bPCrWkOJHTjH3keO xUBGQSfyRyFaCUhwm9kH/kb3glQI35NDS5R+o0s/TJHCBQ1d17MfowsWC0/P9L6uP+1R gr+L+JqD8PmZSHS7t4wYsRvmXlDwJpX8/9FesMWqCqnxkBmyLgmAuKf8I6U6pOfobgq6 Ivh0QY6zzxgZnkkat+79R+six+FhmjJFJC6dH+uuKElY075P3oXy56z9oAIzXFXVaD7/ XZMQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=i/fKd4Jtqfa/YFl4PxGgOhCP4o4uTXtc+mEJ6ZoI0h8=; b=cztCgGhPJ6gGJZPkS/G7PaL9UuvdiR8Ew2A5Ixw1xyufYiBvRYNPy6mMXdzpSU/9Ud hGMIytT3taBrwvsuNsR4EZqOEYv+9k8S6xNmWCFE33mpv9Wl5MRskwQ4TLIh5KlvVl7T M82I+y4/zLzAgsNtp9lPoRn04bLFbIhGNGNFAnPGXR1hTvMytd8Tgzb7qnxkvHG8XDn7 2VHhziWod6e9dOJ4Cs1bzxRueJf6Oy2hfqDxyNc9IGBwwE4y6oULlkRoghuEYVoWDC0R enZRo0d6KEw9t8kumyw0OdSbAst/2KpxZuKmlhNgWC7408Ks8f+TFsIIRAFi02jNyxjf U4ig== X-Gm-Message-State: AFqh2kqt9sONAaDslqLY+AQttebOL/TRqnuo3qmF4Ql09iG+K3t+kxmv J+hFQVogZtJO4RsSPQUk73cwk6Y0yEQ= X-Google-Smtp-Source: AMrXdXuvp1qSxuQav59HbhRlsy9MoYR4vULJwEPoS0bs3n4wsZ38vvtOAMMb3C5IkWC41tEW75rSQQ== X-Received: by 2002:a05:600c:3c86:b0:3da:2a59:8a4f with SMTP id bg6-20020a05600c3c8600b003da2a598a4fmr15861835wmb.38.1674252536159; Fri, 20 Jan 2023 14:08:56 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id m27-20020a05600c3b1b00b003db012d49b7sm14069692wms.2.2023.01.20.14.08.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 20 Jan 2023 14:08:55 -0800 (PST) Message-Id: <17c890ee1080abc81267e44a1eaff4609ee41690.1674252531.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Fri, 20 Jan 2023 22:08:42 +0000 Subject: [PATCH v7 04/12] test-http-server: add stub HTTP server test helper Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Derrick Stolee , Lessley Dennington , Matthew John Cheetham , M Hickford , Jeff Hostetler , Glen Choo , Victoria Dye , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsA==?= Bjarmason , Matthew John Cheetham , Matthew John Cheetham Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Matthew John Cheetham From: Matthew John Cheetham Introduce a mini HTTP server helper that in the future will be enhanced to provide a frontend for the git-http-backend, with support for arbitrary authentication schemes. Right now, test-http-server is a pared-down copy of the git-daemon that always returns a 501 Not Implemented response to all callers. Signed-off-by: Matthew John Cheetham --- Makefile | 1 + contrib/buildsystems/CMakeLists.txt | 11 +- t/helper/.gitignore | 1 + t/helper/test-http-server.c | 381 ++++++++++++++++++++++++++++ 4 files changed, 392 insertions(+), 2 deletions(-) create mode 100644 t/helper/test-http-server.c diff --git a/Makefile b/Makefile index 2654094dbb5..3cd61c792ac 100644 --- a/Makefile +++ b/Makefile @@ -865,6 +865,7 @@ TEST_BUILTINS_OBJS += test-xml-encode.o # Do not add more tests here unless they have extra dependencies. Add # them in TEST_BUILTINS_OBJS above. TEST_PROGRAMS_NEED_X += test-fake-ssh +TEST_PROGRAMS_NEED_X += test-http-server TEST_PROGRAMS_NEED_X += test-tool TEST_PROGRAMS = $(patsubst %,t/helper/%$X,$(TEST_PROGRAMS_NEED_X)) diff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt index 2f6e0197ffa..5d949dcb16c 100644 --- a/contrib/buildsystems/CMakeLists.txt +++ b/contrib/buildsystems/CMakeLists.txt @@ -961,6 +961,9 @@ if(BUILD_TESTING) add_executable(test-fake-ssh ${CMAKE_SOURCE_DIR}/t/helper/test-fake-ssh.c) target_link_libraries(test-fake-ssh common-main) +add_executable(test-http-server ${CMAKE_SOURCE_DIR}/t/helper/test-http-server.c) +target_link_libraries(test-http-server common-main) + #reftable-tests parse_makefile_for_sources(test-reftable_SOURCES "REFTABLE_TEST_OBJS") list(TRANSFORM test-reftable_SOURCES PREPEND "${CMAKE_SOURCE_DIR}/") @@ -980,6 +983,11 @@ if(MSVC) PROPERTIES RUNTIME_OUTPUT_DIRECTORY_DEBUG ${CMAKE_BINARY_DIR}/t/helper) set_target_properties(test-fake-ssh test-tool PROPERTIES RUNTIME_OUTPUT_DIRECTORY_RELEASE ${CMAKE_BINARY_DIR}/t/helper) + + set_target_properties(test-http-server + PROPERTIES RUNTIME_OUTPUT_DIRECTORY_DEBUG ${CMAKE_BINARY_DIR}/t/helper) + set_target_properties(test-http-server + PROPERTIES RUNTIME_OUTPUT_DIRECTORY_RELEASE ${CMAKE_BINARY_DIR}/t/helper) endif() #wrapper scripts @@ -987,8 +995,7 @@ set(wrapper_scripts git git-upload-pack git-receive-pack git-upload-archive git-shell git-remote-ext scalar) set(wrapper_test_scripts - test-fake-ssh test-tool) - + test-http-server test-fake-ssh test-tool) foreach(script ${wrapper_scripts}) file(STRINGS ${CMAKE_SOURCE_DIR}/wrap-for-bin.sh content NEWLINE_CONSUME) diff --git a/t/helper/.gitignore b/t/helper/.gitignore index 8c2ddcce95f..9aa9c752997 100644 --- a/t/helper/.gitignore +++ b/t/helper/.gitignore @@ -1,2 +1,3 @@ /test-tool /test-fake-ssh +/test-http-server diff --git a/t/helper/test-http-server.c b/t/helper/test-http-server.c new file mode 100644 index 00000000000..6e9a1c479ce --- /dev/null +++ b/t/helper/test-http-server.c @@ -0,0 +1,381 @@ +#include "daemon-utils.h" +#include "config.h" +#include "run-command.h" +#include "strbuf.h" +#include "string-list.h" +#include "trace2.h" +#include "version.h" +#include "dir.h" +#include "date.h" + +#define TR2_CAT "test-http-server" + +static const char *pid_file; +static int verbose; +static int reuseaddr; + +static const char test_http_auth_usage[] = +"http-server [--verbose]\n" +" [--timeout=] [--max-connections=]\n" +" [--reuseaddr] [--pid-file=]\n" +" [--listen=]* [--port=]\n" +; + +static unsigned int timeout; + +static void logreport(const char *label, const char *err, va_list params) +{ + struct strbuf msg = STRBUF_INIT; + + strbuf_addf(&msg, "[%"PRIuMAX"] %s: ", (uintmax_t)getpid(), label); + strbuf_vaddf(&msg, err, params); + strbuf_addch(&msg, '\n'); + + fwrite(msg.buf, sizeof(char), msg.len, stderr); + fflush(stderr); + + strbuf_release(&msg); +} + +__attribute__((format (printf, 1, 2))) +static void logerror(const char *err, ...) +{ + va_list params; + va_start(params, err); + logreport("error", err, params); + va_end(params); +} + +__attribute__((format (printf, 1, 2))) +static void loginfo(const char *err, ...) +{ + va_list params; + if (!verbose) + return; + va_start(params, err); + logreport("info", err, params); + va_end(params); +} + +/* + * The code in this section is used by "worker" instances to service + * a single connection from a client. The worker talks to the client + * on stdin and stdout. + */ + +enum worker_result { + /* + * Operation successful. + * Caller *might* keep the socket open and allow keep-alive. + */ + WR_OK = 0, + + /* + * Fatal error that is not recoverable. + * Close the socket and clean up. + * Exit child-process with non-zero status. + */ + WR_FATAL_ERROR = 1, +}; + +static enum worker_result worker(void) +{ + const char *response = "HTTP/1.1 501 Not Implemented\r\n"; + char *client_addr = getenv("REMOTE_ADDR"); + char *client_port = getenv("REMOTE_PORT"); + enum worker_result wr = WR_OK; + + if (client_addr) + loginfo("Connection from %s:%s", client_addr, client_port); + + set_keep_alive(0, logerror); + + while (1) { + if (write_in_full(STDOUT_FILENO, response, strlen(response)) < 0) { + logerror("unable to write response"); + wr = WR_FATAL_ERROR; + } + + if (wr != WR_OK) + break; + } + + close(STDIN_FILENO); + close(STDOUT_FILENO); + + /* Only WR_OK should result in a non-zero exit code */ + return wr != WR_OK; +} + +static int max_connections = 32; + +static unsigned int live_children; + +static struct child *first_child; + +static struct strvec cld_argv = STRVEC_INIT; +static void handle(int incoming, struct sockaddr *addr, socklen_t addrlen) +{ + struct child_process cld = CHILD_PROCESS_INIT; + + if (max_connections && live_children >= max_connections) { + kill_some_child(first_child); + sleep(1); /* give it some time to die */ + check_dead_children(first_child, &live_children, loginfo); + if (live_children >= max_connections) { + close(incoming); + logerror("Too many children, dropping connection"); + return; + } + } + + if (addr->sa_family == AF_INET) { + char buf[128] = ""; + struct sockaddr_in *sin_addr = (void *) addr; + inet_ntop(addr->sa_family, &sin_addr->sin_addr, buf, sizeof(buf)); + strvec_pushf(&cld.env, "REMOTE_ADDR=%s", buf); + strvec_pushf(&cld.env, "REMOTE_PORT=%d", + ntohs(sin_addr->sin_port)); +#ifndef NO_IPV6 + } else if (addr->sa_family == AF_INET6) { + char buf[128] = ""; + struct sockaddr_in6 *sin6_addr = (void *) addr; + inet_ntop(AF_INET6, &sin6_addr->sin6_addr, buf, sizeof(buf)); + strvec_pushf(&cld.env, "REMOTE_ADDR=[%s]", buf); + strvec_pushf(&cld.env, "REMOTE_PORT=%d", + ntohs(sin6_addr->sin6_port)); +#endif + } + + strvec_pushv(&cld.args, cld_argv.v); + cld.in = incoming; + cld.out = dup(incoming); + + if (cld.out < 0) + logerror("could not dup() `incoming`"); + else if (start_command(&cld)) + logerror("unable to fork"); + else + add_child(&cld, addr, addrlen, first_child, &live_children); +} + +static void child_handler(int signo) +{ + /* + * Otherwise empty handler because systemcalls will get interrupted + * upon signal receipt + * SysV needs the handler to be rearmed + */ + signal(SIGCHLD, child_handler); +} + +static int service_loop(struct socketlist *socklist) +{ + struct pollfd *pfd; + int i; + + CALLOC_ARRAY(pfd, socklist->nr); + + for (i = 0; i < socklist->nr; i++) { + pfd[i].fd = socklist->list[i]; + pfd[i].events = POLLIN; + } + + signal(SIGCHLD, child_handler); + + for (;;) { + int i; + int nr_ready; + int timeout = (pid_file ? 100 : -1); + + check_dead_children(first_child, &live_children, loginfo); + + nr_ready = poll(pfd, socklist->nr, timeout); + if (nr_ready < 0) { + if (errno != EINTR) { + logerror("Poll failed, resuming: %s", + strerror(errno)); + sleep(1); + } + continue; + } + else if (nr_ready == 0) { + /* + * If we have a pid_file, then we watch it. + * If someone deletes it, we shutdown the service. + * The shell scripts in the test suite will use this. + */ + if (!pid_file || file_exists(pid_file)) + continue; + goto shutdown; + } + + for (i = 0; i < socklist->nr; i++) { + if (pfd[i].revents & POLLIN) { + union { + struct sockaddr sa; + struct sockaddr_in sai; +#ifndef NO_IPV6 + struct sockaddr_in6 sai6; +#endif + } ss; + socklen_t sslen = sizeof(ss); + int incoming = accept(pfd[i].fd, &ss.sa, &sslen); + if (incoming < 0) { + switch (errno) { + case EAGAIN: + case EINTR: + case ECONNABORTED: + continue; + default: + die_errno("accept returned"); + } + } + handle(incoming, &ss.sa, sslen); + } + } + } + +shutdown: + loginfo("Starting graceful shutdown (pid-file gone)"); + for (i = 0; i < socklist->nr; i++) + close(socklist->list[i]); + + return 0; +} + +static int serve(struct string_list *listen_addr, int listen_port) +{ + struct socketlist socklist = { NULL, 0, 0 }; + + socksetup(listen_addr, listen_port, &socklist, reuseaddr, logerror); + if (socklist.nr == 0) + die("unable to allocate any listen sockets on port %u", + listen_port); + + loginfo("Ready to rumble"); + + /* + * Wait to create the pid-file until we've setup the sockets + * and are open for business. + */ + if (pid_file) + write_file(pid_file, "%"PRIuMAX, (uintmax_t) getpid()); + + return service_loop(&socklist); +} + +/* + * This section is executed by both the primary instance and all + * worker instances. So, yes, each child-process re-parses the + * command line argument and re-discovers how it should behave. + */ + +int cmd_main(int argc, const char **argv) +{ + int listen_port = 0; + struct string_list listen_addr = STRING_LIST_INIT_NODUP; + int worker_mode = 0; + int i; + + trace2_cmd_name("test-http-server"); + trace2_cmd_list_config(); + trace2_cmd_list_env_vars(); + setup_git_directory_gently(NULL); + + for (i = 1; i < argc; i++) { + const char *arg = argv[i]; + const char *v; + + if (skip_prefix(arg, "--listen=", &v)) { + string_list_append(&listen_addr, xstrdup_tolower(v)); + continue; + } + if (skip_prefix(arg, "--port=", &v)) { + char *end; + unsigned long n; + n = strtoul(v, &end, 0); + if (*v && !*end) { + listen_port = n; + continue; + } + } + if (!strcmp(arg, "--worker")) { + worker_mode = 1; + trace2_cmd_mode("worker"); + continue; + } + if (!strcmp(arg, "--verbose")) { + verbose = 1; + continue; + } + if (skip_prefix(arg, "--timeout=", &v)) { + timeout = atoi(v); + continue; + } + if (skip_prefix(arg, "--max-connections=", &v)) { + max_connections = atoi(v); + if (max_connections < 0) + max_connections = 0; /* unlimited */ + continue; + } + if (!strcmp(arg, "--reuseaddr")) { + reuseaddr = 1; + continue; + } + if (skip_prefix(arg, "--pid-file=", &v)) { + pid_file = v; + continue; + } + + fprintf(stderr, "error: unknown argument '%s'\n", arg); + usage(test_http_auth_usage); + } + + /* avoid splitting a message in the middle */ + setvbuf(stderr, NULL, _IOFBF, 4096); + + if (listen_port == 0) + listen_port = DEFAULT_GIT_PORT; + + /* + * If no --listen= args are given, the setup_named_sock() + * code will use receive a NULL address and set INADDR_ANY. + * This exposes both internal and external interfaces on the + * port. + * + * Disallow that and default to the internal-use-only loopback + * address. + */ + if (!listen_addr.nr) + string_list_append(&listen_addr, "127.0.0.1"); + + /* + * worker_mode is set in our own child process instances + * (that are bound to a connected socket from a client). + */ + if (worker_mode) + return worker(); + + /* + * `cld_argv` is a bit of a clever hack. The top-level instance + * of test-http-server does the normal bind/listen/accept stuff. + * For each incoming socket, the top-level process spawns + * a child instance of test-http-server *WITH* the additional + * `--worker` argument. This causes the child to set `worker_mode` + * and immediately call `worker()` using the connected socket (and + * without the usual need for fork() or threads). + * + * The magic here is made possible because `cld_argv` is static + * and handle() (called by service_loop()) knows about it. + */ + strvec_push(&cld_argv, argv[0]); + strvec_push(&cld_argv, "--worker"); + for (i = 1; i < argc; ++i) + strvec_push(&cld_argv, argv[i]); + + /* + * Setup primary instance to listen for connections. + */ + return serve(&listen_addr, listen_port); +} From patchwork Fri Jan 20 22:08:43 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew John Cheetham X-Patchwork-Id: 13110568 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 12312C27C76 for ; Fri, 20 Jan 2023 22:09:06 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229922AbjATWJF (ORCPT ); Fri, 20 Jan 2023 17:09:05 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58184 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229656AbjATWJA (ORCPT ); Fri, 20 Jan 2023 17:09:00 -0500 Received: from mail-wm1-x333.google.com (mail-wm1-x333.google.com [IPv6:2a00:1450:4864:20::333]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8319EA578D for ; Fri, 20 Jan 2023 14:08:58 -0800 (PST) Received: by mail-wm1-x333.google.com with SMTP id f19-20020a1c6a13000000b003db0ef4dedcso6759596wmc.4 for ; Fri, 20 Jan 2023 14:08:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:from:to:cc:subject:date :message-id:reply-to; bh=svxIuOTb7q/EEc6/1OQt4zD+rbN5gnQd/ELeNNOs2Ec=; b=E4p2uGiPCY9aVmI4zXMihMLgR5KUbYHBXHIrmTMDc/EZYb8HGh+BDUcAPdGZ76mV0i cG5ccBobHTDk3g2Yfti/jrbAVaPP6BbCKL3sTaI+qQpjyCi+PvNovvPKtiy2WgowLJwF pq9RXlMORlDK8txoCqLeBDEZAufuu0u0uJFNdadnGqLGHS+8r0t8BXzmbQDZF+qFedWN 5fOGdhAvxqJ0rZpWEykZbM3Kuopkd4kpraHnmj2+0Shzr/b1OL0/XMn3lAHaSIYZ7g36 qJz1cJ8lxQXE+2YfX+RMbTod+FvH2jsx+Cc30HwWD9MFvADwvOwSZ+1yhn7u5UQt2XMN LilQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=svxIuOTb7q/EEc6/1OQt4zD+rbN5gnQd/ELeNNOs2Ec=; b=cV0P7w8FyOmos3qTDvWagEYMlPPErzRkn8WvlrQeCxUlmez6JO+ajTaLdO5VIPTCZJ rbs9mDO9PuH98viAaLY9tD0YSa8qXRDuZe4zOMHKnFxeRo95zY+BVZADq3yi0p13+3Ai mZjF1o/HwfakvpNn0T2YS9io91GG2eRoxHQ/2lfT30u7sjTWWWbYbr1Qnf6cX+RB6xP7 0rbcYfqGMrfTnFXL4Cda47x91M3Or2UUb1uk/akaPV5r9wC2zxr6TzsYxcOQY3A6IaCF 3yXa/TFFXXpIiV4vb3v9bVu18X+F3I+eHFxP9mjSSc+goru26baut0AXtJ9/ZX9Z+QHP z++g== X-Gm-Message-State: AFqh2krmhZfYxwYa2OvtZsFHeMG35It1JxSwlYYRE2+QC7HO2a49qZR6 Y0cyNvP1pivU8Z+JR8sKdSBK2erQsWI= X-Google-Smtp-Source: AMrXdXtiyA8dPtQxRcUx8Nj9sWD84zUfrXs213gxe1yhO4MQMxs/oC9r9JARd5n935dEigDrg0zF1g== X-Received: by 2002:a05:600c:540c:b0:3cf:7704:50ce with SMTP id he12-20020a05600c540c00b003cf770450cemr15090193wmb.38.1674252536965; Fri, 20 Jan 2023 14:08:56 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id az41-20020a05600c602900b003dab77aa911sm3540428wmb.23.2023.01.20.14.08.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 20 Jan 2023 14:08:56 -0800 (PST) Message-Id: <6e70e304cfe6372444a8070d27c7bcdc40ade046.1674252531.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Fri, 20 Jan 2023 22:08:43 +0000 Subject: [PATCH v7 05/12] test-http-server: add HTTP error response function Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Derrick Stolee , Lessley Dennington , Matthew John Cheetham , M Hickford , Jeff Hostetler , Glen Choo , Victoria Dye , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsA==?= Bjarmason , Matthew John Cheetham , Matthew John Cheetham Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Matthew John Cheetham From: Matthew John Cheetham Introduce a function to the test-http-server test helper to write more full and valid HTTP error responses, including all the standard response headers like `Server` and `Date`. Signed-off-by: Matthew John Cheetham --- t/helper/test-http-server.c | 76 +++++++++++++++++++++++++++++++++---- 1 file changed, 69 insertions(+), 7 deletions(-) diff --git a/t/helper/test-http-server.c b/t/helper/test-http-server.c index 6e9a1c479ce..7ca4ddc7999 100644 --- a/t/helper/test-http-server.c +++ b/t/helper/test-http-server.c @@ -76,11 +76,75 @@ enum worker_result { * Exit child-process with non-zero status. */ WR_FATAL_ERROR = 1, + + /* + * Close the socket and clean up. Does not imply an error. + */ + WR_HANGUP = 2, }; +static enum worker_result send_http_error(int fd, int http_code, + const char *http_code_name, + int retry_after_seconds, + struct string_list *response_headers, + enum worker_result wr_in) +{ + struct strbuf response_header = STRBUF_INIT; + struct strbuf response_content = STRBUF_INIT; + struct string_list_item *h; + enum worker_result wr; + + strbuf_addf(&response_content, "Error: %d %s\r\n", http_code, + http_code_name); + + if (retry_after_seconds > 0) + strbuf_addf(&response_content, "Retry-After: %d\r\n", + retry_after_seconds); + + strbuf_addf(&response_header, "HTTP/1.1 %d %s\r\n", http_code, + http_code_name); + strbuf_addstr(&response_header, "Cache-Control: private\r\n"); + strbuf_addstr(&response_header, "Content-Type: text/plain\r\n"); + strbuf_addf(&response_header, "Content-Length: %"PRIuMAX"\r\n", + (uintmax_t)response_content.len); + + if (retry_after_seconds > 0) + strbuf_addf(&response_header, "Retry-After: %d\r\n", + retry_after_seconds); + + strbuf_addf(&response_header, "Server: test-http-server/%s\r\n", + git_version_string); + strbuf_addf(&response_header, "Date: %s\r\n", show_date(time(NULL), 0, + DATE_MODE(RFC2822))); + + if (response_headers) + for_each_string_list_item(h, response_headers) + strbuf_addf(&response_header, "%s\r\n", h->string); + strbuf_addstr(&response_header, "\r\n"); + + if (write_in_full(fd, response_header.buf, response_header.len) < 0) { + logerror("unable to write response header"); + wr = WR_FATAL_ERROR; + goto done; + } + + if (write_in_full(fd, response_content.buf, response_content.len) < 0) { + logerror("unable to write response content body"); + wr = WR_FATAL_ERROR; + goto done; + } + + wr = wr_in; + +done: + strbuf_release(&response_header); + strbuf_release(&response_content); + + return wr; +} + static enum worker_result worker(void) { - const char *response = "HTTP/1.1 501 Not Implemented\r\n"; char *client_addr = getenv("REMOTE_ADDR"); char *client_port = getenv("REMOTE_PORT"); enum worker_result wr = WR_OK; @@ -91,10 +155,8 @@ static enum worker_result worker(void) set_keep_alive(0, logerror); while (1) { - if (write_in_full(STDOUT_FILENO, response, strlen(response)) < 0) { - logerror("unable to write response"); - wr = WR_FATAL_ERROR; - } + wr = send_http_error(STDOUT_FILENO, 501, "Not Implemented", -1, + NULL, WR_HANGUP); if (wr != WR_OK) break; @@ -103,8 +165,8 @@ static enum worker_result worker(void) close(STDIN_FILENO); close(STDOUT_FILENO); - /* Only WR_OK should result in a non-zero exit code */ - return wr != WR_OK; + /* Only WR_OK and WR_HANGUP should result in a non-zero exit code */ + return wr != WR_OK && wr != WR_HANGUP; } static int max_connections = 32; From patchwork Fri Jan 20 22:08:44 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Matthew John Cheetham X-Patchwork-Id: 13110575 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D59B5C05027 for ; Fri, 20 Jan 2023 22:09:15 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229954AbjATWJO (ORCPT ); Fri, 20 Jan 2023 17:09:14 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58200 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229910AbjATWJB (ORCPT ); Fri, 20 Jan 2023 17:09:01 -0500 Received: from mail-wm1-x331.google.com (mail-wm1-x331.google.com [IPv6:2a00:1450:4864:20::331]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6363BA5CCB for ; Fri, 20 Jan 2023 14:08:59 -0800 (PST) Received: by mail-wm1-x331.google.com with SMTP id q10-20020a1cf30a000000b003db0edfdb74so1546895wmq.1 for ; Fri, 20 Jan 2023 14:08:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:from:to:cc:subject:date :message-id:reply-to; bh=aJnt4vqCoq8Tp+oulxwG6ozKog0Qc4/RGRUDUNJ3DtY=; b=NZTY2S+UfOS4Iwt2PulyaP3JI3ZezZB1FPK/xoric6u6W97dgcc6YyXUWa7Hlxf+/U B35NYnvP6Cta5ZxuanngmFaz3SZFqQhVoOYSJau0ZfZCPBWOCFT6fbChlY1K+q4sVG9D Z+/449PvKKqFF8HldBLiaPvCuMncwU94AVwdKnjmgxS+D1hxCUmRtGRfrXHKpMbza6SY aFXpb11qSpsQWHiezH/0zqyK4CG6jts/NF1p8KJzuiX8mWCqWGl7P1BfeDqjDBXvqIe5 7ilvbH3z7GXPdt7P1qd7TWDSaSBmJEwKfSsfpp/FOVte+6jUW0E6TOGJhxoykC268aQ0 UQTA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=aJnt4vqCoq8Tp+oulxwG6ozKog0Qc4/RGRUDUNJ3DtY=; b=o1OJJXKN7Uidh3EmHsJU1WknkVDCsi/UhTiUtDJnM0ZDP/+wNfJyVbzo21EXO0jSK7 NurTErW206GN8a6L07itu0EDa94tsCskgQqINtizGMVDFQMui0sk9ZA6JQ+7oWMecETe jGgnws4yAqP+0LftKigDS6qUU8I3a/KtLGYdeSPgrqUTL7SXegO/bzeSsHwmgBPuFpb7 lwZHGDcFh+RqNDgm7xoz1zWyd797+yruRoCC3c2U1zDmlJyox9Bthac1aqZYpyjAZUpT 9fTtjU8OJFG7ZAwo+U1Gbt9qNkw9EGTiepqZ2Bprr0h5ag0Vvi+qjfPVtgngnxOZCGbd 1mjg== X-Gm-Message-State: AFqh2kohD0DZZWgByQOEI8rshAFbL7CY+0JDoDE40hOvEhZbjkrIaNcL Z5+C3P1MEuEP7s3PKsnanftBtdgwLTQ= X-Google-Smtp-Source: AMrXdXsjA+UEC0EnWGjwGP6hmkQQ7O6qBRO2mLwG7wgFrGhCNCHdYbCcL11Duzx2DlNJLNuH0Tp/+A== X-Received: by 2002:a05:600c:b85:b0:3cf:ae53:9193 with SMTP id fl5-20020a05600c0b8500b003cfae539193mr15591170wmb.39.1674252537703; Fri, 20 Jan 2023 14:08:57 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id n35-20020a05600c3ba300b003db12112fcfsm4136487wms.4.2023.01.20.14.08.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 20 Jan 2023 14:08:57 -0800 (PST) Message-Id: <43f1cdcbb82022521558dc649213eb4538364870.1674252531.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Fri, 20 Jan 2023 22:08:44 +0000 Subject: [PATCH v7 06/12] test-http-server: add HTTP request parsing Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Derrick Stolee , Lessley Dennington , Matthew John Cheetham , M Hickford , Jeff Hostetler , Glen Choo , Victoria Dye , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsA==?= Bjarmason , Matthew John Cheetham , Matthew John Cheetham Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Matthew John Cheetham From: Matthew John Cheetham Add ability to parse HTTP requests to the test-http-server test helper. Introduce `struct req` to store request information including: * HTTP method & version * Request path and query parameters * Headers * Content type and length (from `Content-Type` and `-Length` headers) Failure to parse the request results in a 400 Bad Request response to the client. Note that we're not trying to support all possible requests here, but just enough to exercise all code under test. Signed-off-by: Matthew John Cheetham --- t/helper/test-http-server.c | 202 +++++++++++++++++++++++++++++++++++- t/t5556-http-auth.sh | 90 ++++++++++++++++ 2 files changed, 290 insertions(+), 2 deletions(-) create mode 100755 t/t5556-http-auth.sh diff --git a/t/helper/test-http-server.c b/t/helper/test-http-server.c index 7ca4ddc7999..900f5733cc1 100644 --- a/t/helper/test-http-server.c +++ b/t/helper/test-http-server.c @@ -81,8 +81,53 @@ enum worker_result { * Close the socket and clean up. Does not imply an error. */ WR_HANGUP = 2, + + /* + * Unexpected request message or error in request parsing. + * Respond with an 400 error. Close the socket and cleanup. + * Exit child-process with a non-zero status. + */ + WR_CLIENT_ERROR = 3, +}; + +/* + * Fields from a parsed HTTP request. + */ +struct req { + struct strbuf start_line; + + const char *method; + const char *http_version; + + struct strbuf uri_path; + struct strbuf query_args; + + struct string_list header_list; + const char *content_type; + uintmax_t content_length; + unsigned has_content_length:1; }; +#define REQ__INIT { \ + .start_line = STRBUF_INIT, \ + .uri_path = STRBUF_INIT, \ + .query_args = STRBUF_INIT, \ + .header_list = STRING_LIST_INIT_NODUP, \ + .content_type = NULL, \ + .content_length = 0, \ + .has_content_length = 0, \ +} + +static void req__release(struct req *req) +{ + strbuf_release(&req->start_line); + + strbuf_release(&req->uri_path); + strbuf_release(&req->query_args); + + string_list_clear(&req->header_list, 0); +} + static enum worker_result send_http_error(int fd, int http_code, const char *http_code_name, int retry_after_seconds, @@ -143,8 +188,150 @@ done: return wr; } +/* + * Read the HTTP request up to the start of the optional message-body. + * We do this byte-by-byte because we have keep-alive turned on and + * cannot rely on an EOF. + * + * https://tools.ietf.org/html/rfc7230 + * + * We cannot call die() here because our caller needs to properly + * respond to the client and/or close the socket before this + * child exits so that the client doesn't get a connection reset + * by peer error. + */ +static enum worker_result req__read(struct req *req, int fd) +{ + struct strbuf h = STRBUF_INIT; + struct string_list start_line_fields = STRING_LIST_INIT_DUP; + int nr_start_line_fields; + const char *uri_target; + const char *query; + char *hp; + const char *hv; + + enum worker_result result = WR_OK; + + /* + * Read line 0 of the request and split it into component parts: + * + * SP SP CRLF + * + */ + if (strbuf_getwholeline_fd(&req->start_line, fd, '\n') == EOF) { + result = WR_HANGUP; + goto done; + } + + strbuf_trim_trailing_newline(&req->start_line); + + nr_start_line_fields = string_list_split(&start_line_fields, + req->start_line.buf, + ' ', -1); + if (nr_start_line_fields != 3) { + logerror("could not parse request start-line '%s'", + req->start_line.buf); + result = WR_CLIENT_ERROR; + goto done; + } + + req->method = xstrdup(start_line_fields.items[0].string); + req->http_version = xstrdup(start_line_fields.items[2].string); + + uri_target = start_line_fields.items[1].string; + + if (strcmp(req->http_version, "HTTP/1.1")) { + logerror("unsupported version '%s' (expecting HTTP/1.1)", + req->http_version); + result = WR_CLIENT_ERROR; + goto done; + } + + query = strchr(uri_target, '?'); + + if (query) { + strbuf_add(&req->uri_path, uri_target, (query - uri_target)); + strbuf_trim_trailing_dir_sep(&req->uri_path); + strbuf_addstr(&req->query_args, query + 1); + } else { + strbuf_addstr(&req->uri_path, uri_target); + strbuf_trim_trailing_dir_sep(&req->uri_path); + } + + /* + * Read the set of HTTP headers into a string-list. + */ + while (1) { + if (strbuf_getwholeline_fd(&h, fd, '\n') == EOF) + goto done; + strbuf_trim_trailing_newline(&h); + + if (!h.len) + goto done; /* a blank line ends the header */ + + hp = strbuf_detach(&h, NULL); + string_list_append(&req->header_list, hp); + + /* also store common request headers as struct req members */ + if (skip_iprefix(hp, "Content-Type: ", &hv)) { + req->content_type = hv; + } else if (skip_iprefix(hp, "Content-Length: ", &hv)) { + /* + * Content-Length is always non-negative, but has no + * upper bound according to RFC 7230 (§3.3.2). + */ + intmax_t len = 0; + if (sscanf(hv, "%"PRIdMAX, &len) != 1 || len < 0 || + len == INTMAX_MAX) { + logerror("invalid content-length: '%s'", hv); + result = WR_CLIENT_ERROR; + goto done; + } + + req->content_length = (uintmax_t)len; + req->has_content_length = 1; + } + } + + /* + * We do not attempt to read the , if it exists. + * We let our caller read/chunk it in as appropriate. + */ + +done: + string_list_clear(&start_line_fields, 0); + + /* + * This is useful for debugging the request, but very noisy. + */ + if (trace2_is_enabled()) { + struct string_list_item *item; + trace2_printf("%s: %s", TR2_CAT, req->start_line.buf); + trace2_printf("%s: hver: %s", TR2_CAT, req->http_version); + trace2_printf("%s: hmth: %s", TR2_CAT, req->method); + trace2_printf("%s: path: %s", TR2_CAT, req->uri_path.buf); + trace2_printf("%s: qury: %s", TR2_CAT, req->query_args.buf); + if (req->has_content_length) + trace2_printf("%s: clen: %"PRIuMAX, TR2_CAT, + req->content_length); + if (req->content_type) + trace2_printf("%s: ctyp: %s", TR2_CAT, req->content_type); + for_each_string_list_item(item, &req->header_list) + trace2_printf("%s: hdrs: %s", TR2_CAT, item->string); + } + + return result; +} + +static enum worker_result dispatch(struct req *req) +{ + return send_http_error(STDOUT_FILENO, 501, "Not Implemented", -1, NULL, + WR_HANGUP); +} + static enum worker_result worker(void) { + struct req req = REQ__INIT; char *client_addr = getenv("REMOTE_ADDR"); char *client_port = getenv("REMOTE_PORT"); enum worker_result wr = WR_OK; @@ -155,9 +342,20 @@ static enum worker_result worker(void) set_keep_alive(0, logerror); while (1) { - wr = send_http_error(STDOUT_FILENO, 501, "Not Implemented", -1, - NULL, WR_HANGUP); + req__release(&req); + + alarm(timeout); + wr = req__read(&req, 0); + alarm(0); + + if (wr == WR_CLIENT_ERROR) + wr = send_http_error(STDOUT_FILENO, 400, "Bad Request", + -1, NULL, wr); + + if (wr != WR_OK) + break; + wr = dispatch(&req); if (wr != WR_OK) break; } diff --git a/t/t5556-http-auth.sh b/t/t5556-http-auth.sh new file mode 100755 index 00000000000..06efc85ca53 --- /dev/null +++ b/t/t5556-http-auth.sh @@ -0,0 +1,90 @@ +#!/bin/sh + +test_description='test http auth header and credential helper interop' + +TEST_NO_CREATE_REPO=1 +. ./test-lib.sh + +# Setup a repository +# +REPO_DIR="$TRASH_DIRECTORY"/repo + +SERVER_LOG="$TRASH_DIRECTORY"/OUT.server.log + +PATH="$GIT_BUILD_DIR/t/helper/:$PATH" && export PATH + +test_expect_success 'setup repos' ' + test_create_repo "$REPO_DIR" && + git -C "$REPO_DIR" branch -M main +' + +run_http_server_worker() { + ( + cd "$REPO_DIR" + test-http-server --worker "$@" 2>"$SERVER_LOG" | tr -d "\r" + ) +} + +per_test_cleanup () { + rm -f OUT.* && + rm -f IN.* && +} + +test_expect_success 'http auth server request parsing' ' + test_when_finished "per_test_cleanup" && + + cat >auth.config <<-EOF && + [auth] + allowAnonymous = true + EOF + + echo "HTTP/1.1 400 Bad Request" >OUT.http400 && + echo "HTTP/1.1 200 OK" >OUT.http200 && + + cat >IN.http.valid <<-EOF && + GET /info/refs HTTP/1.1 + Content-Length: 0 + EOF + + cat >IN.http.badfirstline <<-EOF && + /info/refs GET HTTP + EOF + + cat >IN.http.badhttpver <<-EOF && + GET /info/refs HTTP/999.9 + EOF + + cat >IN.http.ltzlen <<-EOF && + GET /info/refs HTTP/1.1 + Content-Length: -1 + EOF + + cat >IN.http.badlen <<-EOF && + GET /info/refs HTTP/1.1 + Content-Length: not-a-number + EOF + + cat >IN.http.overlen <<-EOF && + GET /info/refs HTTP/1.1 + Content-Length: 9223372036854775807 + EOF + + run_http_server_worker \ + --auth-config="$TRASH_DIRECTORY/auth.config" OUT.actual && + test_cmp OUT.http200 OUT.actual && + + run_http_server_worker OUT.actual && + test_cmp OUT.http400 OUT.actual && + + run_http_server_worker OUT.actual && + test_cmp OUT.http400 OUT.actual && + + run_http_server_worker OUT.actual && + test_cmp OUT.http400 OUT.actual && + + run_http_server_worker OUT.actual && + test_cmp OUT.http400 OUT.actual +' + +test_done From patchwork Fri Jan 20 22:08:45 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew John Cheetham X-Patchwork-Id: 13110572 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id AB000C05027 for ; Fri, 20 Jan 2023 22:09:11 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229943AbjATWJK (ORCPT ); Fri, 20 Jan 2023 17:09:10 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58198 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229907AbjATWJB (ORCPT ); Fri, 20 Jan 2023 17:09:01 -0500 Received: from mail-wr1-x432.google.com (mail-wr1-x432.google.com [IPv6:2a00:1450:4864:20::432]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1CE2AA5CD3 for ; Fri, 20 Jan 2023 14:09:00 -0800 (PST) Received: by mail-wr1-x432.google.com with SMTP id r9so6012658wrw.4 for ; Fri, 20 Jan 2023 14:09:00 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:from:to:cc:subject:date :message-id:reply-to; bh=vt6qt5xy2LTOWOA/ba75VrNCLenL8Di1Ghv67LM82hE=; b=eVZhcTQYCm6Xvc8wIm0POK+m3SuItk1x2qHBk+VRyvEl3AFpNT6dY1q0t1B12IR1S4 AHZESoiSx4OdipOg20klPd84CYzh2fjoyrPckAU+kOgyT1i1fCjDEnN+w2iwEtbPUFPl d76OH8p697XKm6DW48rLKnJteb2usWZyyp0vwf/8ISYkz/JjSqSx0ZuGRw/hYjmaUEhV Ym0q/oj5nGf8i1jKJrPAELV3EZDM53PcxRBKWOqPGtu8RqhM3NmxAeZ6XY5ha8eKZIaL MmttbdAzYVIfrSeO7vTpZXcg8IPn54vn0M3RO8HJiOgbmS7e0rOFHnS1DoZUTkP3xdoB rFAQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=vt6qt5xy2LTOWOA/ba75VrNCLenL8Di1Ghv67LM82hE=; b=SgToTNYVwMG9nAZqohvZUID1xQlFzezsEVA2NRp+2svvZuxefR1kL65crK+vSUbJgh atC+cHt0q/lWMMutyzmjXmM3UNE3/rQPC6SQ166zpqY7j+k94Ds70Fy23yh9qgp9QdZE MGUJnLUHRSHhEyZ5NUwZaJiNk4uwsv+ygHJu16MBsSS5s5L8GBYSRRkYeA1q2MO38Apb T//H8FHQrFf6bgik4eJdLUDIos7aNjnTenqfS1qN+XYrj7lm/cz8IC/x+6pDXieET1aJ 91VDEwO7vFs/NAljsv5shTWUQcwg9RgoFzF+4uiIb/ZU17wDjSUB8yEbR1ulVT5tgeQb dwLw== X-Gm-Message-State: AFqh2kq0yB9DHBuk8Wvaf9P5A7Dn9KxEeeU3w6un5uIMkkhTGDY+KRaP wKVCyJzgF2RSpcXb6bspIQdj278PgBc= X-Google-Smtp-Source: AMrXdXtqRI0rxknm7NcPlPcwN6z59xqDITZ4UyLGy+cBY5d1vtdb9TZw232tgz8K5EsvKZ1juXXhUw== X-Received: by 2002:a5d:65cc:0:b0:2be:493f:3b2d with SMTP id e12-20020a5d65cc000000b002be493f3b2dmr5611529wrw.10.1674252538469; Fri, 20 Jan 2023 14:08:58 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id p8-20020a5d4588000000b002bdfb97e029sm14810164wrq.19.2023.01.20.14.08.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 20 Jan 2023 14:08:58 -0800 (PST) Message-Id: In-Reply-To: References: Date: Fri, 20 Jan 2023 22:08:45 +0000 Subject: [PATCH v7 07/12] test-http-server: pass Git requests to http-backend Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Derrick Stolee , Lessley Dennington , Matthew John Cheetham , M Hickford , Jeff Hostetler , Glen Choo , Victoria Dye , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsA==?= Bjarmason , Matthew John Cheetham , Matthew John Cheetham Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Matthew John Cheetham From: Matthew John Cheetham Teach the test-http-sever test helper to forward Git requests to the `git-http-backend`. Introduce a new test script t5556-http-auth.sh that spins up the test HTTP server and attempts an `ls-remote` on the served repository, without any authentication. Signed-off-by: Matthew John Cheetham --- t/helper/test-http-server.c | 68 ++++++++++++++++++++++++++++++ t/t5556-http-auth.sh | 83 +++++++++++++++++++++++++++++++++++++ 2 files changed, 151 insertions(+) diff --git a/t/helper/test-http-server.c b/t/helper/test-http-server.c index 900f5733cc1..4191daf3c64 100644 --- a/t/helper/test-http-server.c +++ b/t/helper/test-http-server.c @@ -323,8 +323,76 @@ done: return result; } +static int is_git_request(struct req *req) +{ + static regex_t *smart_http_regex; + static int initialized; + + if (!initialized) { + smart_http_regex = xmalloc(sizeof(*smart_http_regex)); + /* + * This regular expression matches all dumb and smart HTTP + * requests that are currently in use, and defined in + * Documentation/gitprotocol-http.txt. + * + */ + if (regcomp(smart_http_regex, "^/(HEAD|info/refs|" + "objects/info/[^/]+|git-(upload|receive)-pack)$", + REG_EXTENDED)) { + warning("could not compile smart HTTP regex"); + smart_http_regex = NULL; + } + initialized = 1; + } + + return smart_http_regex && + !regexec(smart_http_regex, req->uri_path.buf, 0, NULL, 0); +} + +static enum worker_result do__git(struct req *req) +{ + const char *ok = "HTTP/1.1 200 OK\r\n"; + struct child_process cp = CHILD_PROCESS_INIT; + int res; + + /* + * Note that we always respond with a 200 OK response even if the + * http-backend process exits with an error. This helper is intended + * only to be used to exercise the HTTP auth handling in the Git client, + * and specifically around authentication (not handled by http-backend). + * + * If we wanted to respond with a more 'valid' HTTP response status then + * we'd need to buffer the output of http-backend, wait for and grok the + * exit status of the process, then write the HTTP status line followed + * by the http-backend output. This is outside of the scope of this test + * helper's use at time of writing. + */ + if (write(STDOUT_FILENO, ok, strlen(ok)) < 0) + return error(_("could not send '%s'"), ok); + + strvec_pushf(&cp.env, "REQUEST_METHOD=%s", req->method); + strvec_pushf(&cp.env, "PATH_TRANSLATED=%s", req->uri_path.buf); + strvec_push(&cp.env, "SERVER_PROTOCOL=HTTP/1.1"); + if (req->query_args.len) + strvec_pushf(&cp.env, "QUERY_STRING=%s", req->query_args.buf); + if (req->content_type) + strvec_pushf(&cp.env, "CONTENT_TYPE=%s", req->content_type); + if (req->has_content_length) + strvec_pushf(&cp.env, "CONTENT_LENGTH=%" PRIuMAX, + (uintmax_t)req->content_length); + cp.git_cmd = 1; + strvec_push(&cp.args, "http-backend"); + res = run_command(&cp); + close(STDOUT_FILENO); + close(STDIN_FILENO); + return !!res; +} + static enum worker_result dispatch(struct req *req) { + if (is_git_request(req)) + return do__git(req); + return send_http_error(STDOUT_FILENO, 501, "Not Implemented", -1, NULL, WR_HANGUP); } diff --git a/t/t5556-http-auth.sh b/t/t5556-http-auth.sh index 06efc85ca53..c0a47ce342b 100755 --- a/t/t5556-http-auth.sh +++ b/t/t5556-http-auth.sh @@ -5,10 +5,25 @@ test_description='test http auth header and credential helper interop' TEST_NO_CREATE_REPO=1 . ./test-lib.sh +test_set_port GIT_TEST_HTTP_PROTOCOL_PORT + # Setup a repository # REPO_DIR="$TRASH_DIRECTORY"/repo +# Setup some lookback URLs where test-http-server will be listening. +# We will spawn it directly inside the repo directory, so we avoid +# any need to configure directory mappings etc - we only serve this +# repository from the root '/' of the server. +# +HOST_PORT=127.0.0.1:$GIT_TEST_HTTP_PROTOCOL_PORT +ORIGIN_URL=http://$HOST_PORT/ + +# The pid-file is created by test-http-server when it starts. +# The server will shutdown if/when we delete it (this is easier than +# killing it by PID). +# +PID_FILE="$TRASH_DIRECTORY"/pid-file.pid SERVER_LOG="$TRASH_DIRECTORY"/OUT.server.log PATH="$GIT_BUILD_DIR/t/helper/:$PATH" && export PATH @@ -25,7 +40,65 @@ run_http_server_worker() { ) } +stop_http_server () { + if ! test -f "$PID_FILE" + then + return 0 + fi + # + # The server will shutdown automatically when we delete the pid-file. + # + rm -f "$PID_FILE" + # + # Give it a few seconds to shutdown (mainly to completely release the + # port before the next test start another instance and it attempts to + # bind to it). + # + for k in 0 1 2 3 4 + do + if grep -q "Starting graceful shutdown" "$SERVER_LOG" + then + return 0 + fi + sleep 1 + done + + echo "stop_http_server: timeout waiting for server shutdown" + return 1 +} + +start_http_server () { + # + # Launch our server into the background in repo_dir. + # + ( + cd "$REPO_DIR" + test-http-server --verbose \ + --listen=127.0.0.1 \ + --port=$GIT_TEST_HTTP_PROTOCOL_PORT \ + --reuseaddr \ + --pid-file="$PID_FILE" \ + "$@" \ + 2>"$SERVER_LOG" & + ) + # + # Give it a few seconds to get started. + # + for k in 0 1 2 3 4 + do + if test -f "$PID_FILE" + then + return 0 + fi + sleep 1 + done + + echo "start_http_server: timeout waiting for server startup" + return 1 +} + per_test_cleanup () { + stop_http_server && rm -f OUT.* && rm -f IN.* && } @@ -87,4 +160,14 @@ test_expect_success 'http auth server request parsing' ' test_cmp OUT.http400 OUT.actual ' + +test_expect_success 'http auth anonymous no challenge' ' + test_when_finished "per_test_cleanup" && + + start_http_server && + + # Attempt to read from a protected repository + git ls-remote $ORIGIN_URL +' + test_done From patchwork Fri Jan 20 22:08:46 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew John Cheetham X-Patchwork-Id: 13110576 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3ADB2C25B50 for ; Fri, 20 Jan 2023 22:09:17 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229963AbjATWJQ (ORCPT ); Fri, 20 Jan 2023 17:09:16 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58216 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229915AbjATWJD (ORCPT ); Fri, 20 Jan 2023 17:09:03 -0500 Received: from mail-wm1-x32b.google.com (mail-wm1-x32b.google.com [IPv6:2a00:1450:4864:20::32b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0FC8DA19AC for ; Fri, 20 Jan 2023 14:09:01 -0800 (PST) Received: by mail-wm1-x32b.google.com with SMTP id l8so5076196wms.3 for ; Fri, 20 Jan 2023 14:09:00 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:from:to:cc:subject:date :message-id:reply-to; bh=D8JoO8W1wcgDBqTd5Hya39NGP1gdBhM31YoNnivPdCY=; b=Pg9sybeub1pD0EqbnswBVRv60nYxvsu8EJtVQWuyeOI1gSR7jSTcryRLo3gl0fptwg Ypf9Jp8EMg/6Cd1HH11b/JNDMdVHxXCHAMQKs2rD3Wuq0Mhxn+FQIzR5Su1ABQDcNwXc UJ/vu+tZAPBHKsOsVCEm8cDQ4wSL2PVDvkyTCv5oy7k7zxYMzLNNckWVAIkHzCJ2ChZX 0kix/RkwOfJa2rOHLFETQ+ezdzobVz+GiMQC9oBwOAIV1Qag0jsqNq7sfzzMCThsflJ/ EDa/AjZ80+lFOXgIDnoZvIaTJ4weNX0wI2UTByrYBdiz71tu8umNcXRvQXI+3Tzbgvin fU1g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=D8JoO8W1wcgDBqTd5Hya39NGP1gdBhM31YoNnivPdCY=; b=z3ZnDdpipHtGdZ3mXtcdEvh2TyNMDK5YeLIzyH5gnKRg8lxnJOXE8CILuHwtHjD/vR ipfb1AR8qKGVO52DnlD1jISvK7dmQ5l7/ECTPW6lBLAigUIhLos0Gu7BXyp/Pz8RdbD2 Sdv2UVZ1B5p4QvuPXhb019efkriJ/W20Hc3dY+ZfEZDD7ipD88Y3RZPY0k0WAxYn7so9 iBPPc8hTXhd/X1q42LIrKBhBrWNqWXU01oClL3QXiTtp4ce33x6m5p5b206MJPhKUb3Z PRkd/JflgyVKqdLgfHhaMTJamFTACZMeof8ZxLwa/yFhNrXx7qwlKYrHHqfHjY5mgJKj OJkQ== X-Gm-Message-State: AFqh2krhvFMpASCrHjxzxYToUXZShteHGBwKleUHJExD0L1mVgFucRVu 2xkG6y2eDnxlOnOtyp4NuoXiUf9ZOfE= X-Google-Smtp-Source: AMrXdXvHZk8vc9FbL7TQAkVnnZ5aHk7Mnw3OEvxvPfiFMh5RVRRPfwCbKDxUlpKBNZcz8NCCcGuUAg== X-Received: by 2002:a05:600c:810:b0:3da:f0e:fe34 with SMTP id k16-20020a05600c081000b003da0f0efe34mr15096312wmp.32.1674252539339; Fri, 20 Jan 2023 14:08:59 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id l7-20020a7bc447000000b003dafa04ecc4sm3381999wmi.6.2023.01.20.14.08.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 20 Jan 2023 14:08:58 -0800 (PST) Message-Id: In-Reply-To: References: Date: Fri, 20 Jan 2023 22:08:46 +0000 Subject: [PATCH v7 08/12] test-http-server: add simple authentication Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Derrick Stolee , Lessley Dennington , Matthew John Cheetham , M Hickford , Jeff Hostetler , Glen Choo , Victoria Dye , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsA==?= Bjarmason , Matthew John Cheetham , Matthew John Cheetham Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Matthew John Cheetham From: Matthew John Cheetham Add simple authentication to the test-http-server test helper. Authentication schemes and sets of valid tokens can be specified via a configuration file (in the normal gitconfig file format). Incoming requests are compared against the set of valid schemes and tokens and only approved if a matching token is found, or if no auth was provided and anonymous auth is enabled. Configuration for auth includes a simple set of three options: [auth] challenge = [:] token = :[]* allowAnonymous = `auth.challenge` allows you define what authentication schemes, and optional challenge parameters the server should use. Scheme names are unique and subsequently specified challenge parameters in the config file will replace previously specified ones. `auth.token` allows you to define the set of value token values for an authentication scheme. This is a multi-var and each entry in the config file will append to the set of valid tokens for that scheme. Specifying an empty token value will clear the list of tokens so far for that scheme, i.e. `token = :`. `auth.allowAnonymous` controls whether or not unauthenticated requests (those without any `Authorization` headers) should succeed or not, and trigger a 401 Unauthorized response. Signed-off-by: Matthew John Cheetham --- t/helper/test-http-server.c | 232 +++++++++++++++++++++++++++++++++++- t/t5556-http-auth.sh | 43 ++++++- 2 files changed, 272 insertions(+), 3 deletions(-) diff --git a/t/helper/test-http-server.c b/t/helper/test-http-server.c index 4191daf3c64..72c6cca7e5c 100644 --- a/t/helper/test-http-server.c +++ b/t/helper/test-http-server.c @@ -7,6 +7,7 @@ #include "version.h" #include "dir.h" #include "date.h" +#include "config.h" #define TR2_CAT "test-http-server" @@ -19,6 +20,7 @@ static const char test_http_auth_usage[] = " [--timeout=] [--max-connections=]\n" " [--reuseaddr] [--pid-file=]\n" " [--listen=]* [--port=]\n" +" [--auth-config=]\n" ; static unsigned int timeout; @@ -349,7 +351,7 @@ static int is_git_request(struct req *req) !regexec(smart_http_regex, req->uri_path.buf, 0, NULL, 0); } -static enum worker_result do__git(struct req *req) +static enum worker_result do__git(struct req *req, const char *user) { const char *ok = "HTTP/1.1 200 OK\r\n"; struct child_process cp = CHILD_PROCESS_INIT; @@ -366,10 +368,16 @@ static enum worker_result do__git(struct req *req) * exit status of the process, then write the HTTP status line followed * by the http-backend output. This is outside of the scope of this test * helper's use at time of writing. + * + * The important auth responses (401) we are handling prior to getting + * to this point. */ if (write(STDOUT_FILENO, ok, strlen(ok)) < 0) return error(_("could not send '%s'"), ok); + if (user) + strvec_pushf(&cp.env, "REMOTE_USER=%s", user); + strvec_pushf(&cp.env, "REQUEST_METHOD=%s", req->method); strvec_pushf(&cp.env, "PATH_TRANSLATED=%s", req->uri_path.buf); strvec_push(&cp.env, "SERVER_PROTOCOL=HTTP/1.1"); @@ -388,10 +396,217 @@ static enum worker_result do__git(struct req *req) return !!res; } +enum auth_result { + /* No auth module matches the request. */ + AUTH_UNKNOWN = 0, + + /* Auth module denied the request. */ + AUTH_DENY = 1, + + /* Auth module successfully validated the request. */ + AUTH_ALLOW = 2, +}; + +struct auth_module { + char *scheme; + char *challenge_params; + struct string_list *tokens; +}; + +static int allow_anonymous; +static struct auth_module **auth_modules = NULL; +static size_t auth_modules_nr = 0; +static size_t auth_modules_alloc = 0; + +static struct auth_module *get_auth_module(const char *scheme, int create) +{ + struct auth_module *mod; + for (size_t i = 0; i < auth_modules_nr; i++) { + mod = auth_modules[i]; + if (!strcasecmp(mod->scheme, scheme)) + return mod; + } + + if (create) { + struct auth_module *mod = xmalloc(sizeof(struct auth_module)); + mod->scheme = xstrdup(scheme); + mod->challenge_params = NULL; + ALLOC_ARRAY(mod->tokens, 1); + string_list_init_dup(mod->tokens); + + ALLOC_GROW(auth_modules, auth_modules_nr + 1, auth_modules_alloc); + auth_modules[auth_modules_nr++] = mod; + + return mod; + } + + return NULL; +} + +static int is_authed(struct req *req, const char **user, enum worker_result *wr) +{ + enum auth_result result = AUTH_UNKNOWN; + struct string_list hdrs = STRING_LIST_INIT_NODUP; + struct auth_module *mod; + + struct string_list_item *hdr; + struct string_list_item *token; + const char *v; + struct strbuf **split = NULL; + int i; + char *challenge; + + /* + * Check all auth modules and try to validate the request. + * The first Authorization header that matches a known auth module + * scheme will be consulted to either approve or deny the request. + * If no module is found, or if there is no valid token, then 401 error. + * Otherwise, only permit the request if anonymous auth is enabled. + * It's atypical for user agents/clients to send multiple Authorization + * headers, but not explicitly forbidden or defined. + */ + for_each_string_list_item(hdr, &req->header_list) { + if (skip_iprefix(hdr->string, "Authorization: ", &v)) { + split = strbuf_split_str(v, ' ', 2); + if (split[0] && split[1]) { + /* trim trailing space ' ' */ + strbuf_rtrim(split[0]); + + mod = get_auth_module(split[0]->buf, 0); + if (mod) { + result = AUTH_DENY; + + for_each_string_list_item(token, mod->tokens) { + if (!strcmp(split[1]->buf, token->string)) { + result = AUTH_ALLOW; + break; + } + } + + strbuf_list_free(split); + goto done; + } + } + + strbuf_list_free(split); + } + } + +done: + switch (result) { + case AUTH_ALLOW: + trace2_printf("%s: auth '%s' ALLOW", TR2_CAT, mod->scheme); + *user = "VALID_TEST_USER"; + *wr = WR_OK; + break; + + case AUTH_DENY: + trace2_printf("%s: auth '%s' DENY", TR2_CAT, mod->scheme); + /* fall-through */ + + case AUTH_UNKNOWN: + if (result != AUTH_DENY && allow_anonymous) + break; + + for (i = 0; i < auth_modules_nr; i++) { + mod = auth_modules[i]; + if (mod->challenge_params) + challenge = xstrfmt("WWW-Authenticate: %s %s", + mod->scheme, + mod->challenge_params); + else + challenge = xstrfmt("WWW-Authenticate: %s", + mod->scheme); + string_list_append(&hdrs, challenge); + } + + *wr = send_http_error(STDOUT_FILENO, 401, "Unauthorized", -1, + &hdrs, *wr); + } + + string_list_clear(&hdrs, 0); + + return result == AUTH_ALLOW || + (result == AUTH_UNKNOWN && allow_anonymous); +} + +static int split_auth_param(const char *str, char **scheme, char **val) +{ + struct strbuf **p = strbuf_split_str(str, ':', 2); + + if (!p[0]) + return -1; + + /* trim trailing ':' */ + if (p[0]->len && p[0]->buf[p[0]->len - 1] == ':') + strbuf_setlen(p[0], p[0]->len - 1); + + *scheme = strbuf_detach(p[0], NULL); + *val = p[1] ? strbuf_detach(p[1], NULL) : NULL; + + strbuf_list_free(p); + return 0; +} + +static int read_auth_config(const char *name, const char *val, void *data) +{ + int ret = 0; + char *scheme = NULL; + char *token = NULL; + char *challenge = NULL; + struct auth_module *mod; + + if (!strcmp(name, "auth.challenge")) { + if (split_auth_param(val, &scheme, &challenge)) { + ret = error("invalid auth challenge '%s'", val); + goto cleanup; + } + + mod = get_auth_module(scheme, 1); + + /* Replace any existing challenge parameters */ + free(mod->challenge_params); + mod->challenge_params = challenge ? xstrdup(challenge) : NULL; + } else if (!strcmp(name, "auth.token")) { + if (split_auth_param(val, &scheme, &token)) { + ret = error("invalid auth token '%s'", val); + goto cleanup; + } + + mod = get_auth_module(scheme, 1); + + /* + * Append to set of valid tokens unless an empty token value + * is provided, then clear the existing list. + */ + if (token) + string_list_append(mod->tokens, token); + else + string_list_clear(mod->tokens, 1); + } else if (!strcmp(name, "auth.allowanonymous")) { + allow_anonymous = git_config_bool(name, val); + } else { + warning("unknown auth config '%s'", name); + } + +cleanup: + free(scheme); + free(token); + free(challenge); + + return ret; +} + static enum worker_result dispatch(struct req *req) { + enum worker_result wr = WR_OK; + const char *user = NULL; + + if (!is_authed(req, &user, &wr)) + return wr; + if (is_git_request(req)) - return do__git(req); + return do__git(req, user); return send_http_error(STDOUT_FILENO, 501, "Not Implemented", -1, NULL, WR_HANGUP); @@ -655,6 +870,19 @@ int cmd_main(int argc, const char **argv) pid_file = v; continue; } + if (skip_prefix(arg, "--auth-config=", &v)) { + if (!strlen(v)) { + error("invalid argument - missing file path"); + usage(test_http_auth_usage); + } + + if (git_config_from_file(read_auth_config, v, NULL)) { + error("failed to read auth config file '%s'", v); + usage(test_http_auth_usage); + } + + continue; + } fprintf(stderr, "error: unknown argument '%s'\n", arg); usage(test_http_auth_usage); diff --git a/t/t5556-http-auth.sh b/t/t5556-http-auth.sh index c0a47ce342b..20fd9b09aef 100755 --- a/t/t5556-http-auth.sh +++ b/t/t5556-http-auth.sh @@ -101,6 +101,7 @@ per_test_cleanup () { stop_http_server && rm -f OUT.* && rm -f IN.* && + rm -f auth.config } test_expect_success 'http auth server request parsing' ' @@ -160,11 +161,51 @@ test_expect_success 'http auth server request parsing' ' test_cmp OUT.http400 OUT.actual ' +test_expect_success CURL 'http auth server auth config' ' + test_when_finished "per_test_cleanup" && + + cat >auth.config <<-EOF && + [auth] + challenge = no-params + challenge = with-params:foo=\"bar\" p=1 + challenge = with-params:foo=\"replaced\" q=1 + + token = no-explicit-challenge:valid-token + token = no-explicit-challenge:also-valid + token = reset-tokens:these-tokens + token = reset-tokens:will-be-reset + token = reset-tokens: + token = reset-tokens:the-only-valid-one + + allowAnonymous = false + EOF + + cat >OUT.expected <<-EOF && + WWW-Authenticate: no-params + WWW-Authenticate: with-params foo="replaced" q=1 + WWW-Authenticate: no-explicit-challenge + WWW-Authenticate: reset-tokens + + Error: 401 Unauthorized + EOF + + start_http_server --auth-config="$TRASH_DIRECTORY/auth.config" && + + curl --include $ORIGIN_URL >OUT.curl && + tr -d "\r" OUT.actual && + + test_cmp OUT.expected OUT.actual +' test_expect_success 'http auth anonymous no challenge' ' test_when_finished "per_test_cleanup" && - start_http_server && + cat >auth.config <<-EOF && + [auth] + allowAnonymous = true + EOF + + start_http_server --auth-config="$TRASH_DIRECTORY/auth.config" && # Attempt to read from a protected repository git ls-remote $ORIGIN_URL From patchwork Fri Jan 20 22:08:47 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew John Cheetham X-Patchwork-Id: 13110577 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D137BC27C76 for ; Fri, 20 Jan 2023 22:09:18 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229966AbjATWJS (ORCPT ); Fri, 20 Jan 2023 17:09:18 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58214 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229912AbjATWJD (ORCPT ); Fri, 20 Jan 2023 17:09:03 -0500 Received: from mail-wr1-x42d.google.com (mail-wr1-x42d.google.com [IPv6:2a00:1450:4864:20::42d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BA168A5CDA for ; Fri, 20 Jan 2023 14:09:01 -0800 (PST) Received: by mail-wr1-x42d.google.com with SMTP id b7so6019124wrt.3 for ; Fri, 20 Jan 2023 14:09:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:from:to:cc:subject:date :message-id:reply-to; bh=qTWpx0WyAIrrtseRIMjsPnNsN9rcivimHrEZq2BXS9w=; b=a8Db2/tCoOQQmy3GOI2Q7m6PWFjHUxweBarHo57ld+r1va6w5aQRZlp4hcejaHILLH 7jeCTzH7IybQt7Xa1+0W04Av3ubtGA54do8YL2G/oRAUn5nLlLqhgWrRhJ6q/l34Zn1I GEyYCERVmZ1ZBAqlTM7jq3v3A9K1/jSalvo0mqrrZuQ6Kn0OqtLmn4JBiCp9dvkU1TK7 s3Og7RulZu4i3ozzwi8Ns5Qb2BtC43R8n6VueibkMIzBfVawRu/bnT/kjyPzfcpsRE3V BSdXqOyutnMGv/YGf1DI3TFdQG9ICTssSi50S5A4XSnVGgVcxb6vOviMoVjvpmtsOZ75 aE8A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=qTWpx0WyAIrrtseRIMjsPnNsN9rcivimHrEZq2BXS9w=; b=ewppDQqoNTT+V0gQMdQ4K/tELNrpp9xBYKaFuGE97LFvCgn6Eyga1iGZGVvqDkE8No WPRArcH3RdAZW3RzR/IXF0Tr2FtV933HA/HAoiyXeWBiCaEw0uRuF6Xk4OdkeNC3UPn0 Giv6slbFGY5lJ5md4EDfOodzqA+CHrqXFRo0w15AwBSCYYGbGabiw7O0jqXXvt+nfhOG bxYZI+1JUrHNF3563b6Lhb7OdL78DH8Znk27ZxMbnUIZai6+1k1wh+fqd+tuAYehmWj8 bU8a/X35FxvZ2G01vGJeMWJ8FToy49W4vSGek2T8KoGkpxWZ5Hpd3xK1Wiv9XBB4al04 euZA== X-Gm-Message-State: AFqh2kp1xlU8d0VIa7+4HywtN6NoLypfniIImSommldlqFoXG4jQzEj5 3mjLa+7yC7sM1PJaLWIPUJciyrSibUU= X-Google-Smtp-Source: AMrXdXtTx2w/hu6jS4FUj7ZM+hRqIfirpFAR/eGvz3uB6qNH2m6VmW3ZTHfexLwbASd4z+RSYRgNig== X-Received: by 2002:a5d:4f90:0:b0:2bd:d542:e01e with SMTP id d16-20020a5d4f90000000b002bdd542e01emr14604620wru.10.1674252540184; Fri, 20 Jan 2023 14:09:00 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id f2-20020adfdb42000000b0024274a5db0asm36752818wrj.2.2023.01.20.14.08.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 20 Jan 2023 14:08:59 -0800 (PST) Message-Id: <2f97c94f67981dcfbbed00a9800cec4cd26c594d.1674252531.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Fri, 20 Jan 2023 22:08:47 +0000 Subject: [PATCH v7 09/12] test-http-server: add sending of arbitrary headers Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Derrick Stolee , Lessley Dennington , Matthew John Cheetham , M Hickford , Jeff Hostetler , Glen Choo , Victoria Dye , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsA==?= Bjarmason , Matthew John Cheetham , Matthew John Cheetham Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Matthew John Cheetham From: Matthew John Cheetham Add the ability to send arbitrary headers in HTTP responses from the test-http-server. This is useful when we want to test 'malformed' response message handling. Add the following option to the server auth config file: [auth] extraHeader = []* Each `auth.extraHeader` value will be appended to the response headers verbatim. Signed-off-by: Matthew John Cheetham --- t/helper/test-http-server.c | 6 ++++++ t/t5556-http-auth.sh | 7 +++++++ 2 files changed, 13 insertions(+) diff --git a/t/helper/test-http-server.c b/t/helper/test-http-server.c index 72c6cca7e5c..70bf15c3fa1 100644 --- a/t/helper/test-http-server.c +++ b/t/helper/test-http-server.c @@ -417,6 +417,7 @@ static int allow_anonymous; static struct auth_module **auth_modules = NULL; static size_t auth_modules_nr = 0; static size_t auth_modules_alloc = 0; +static struct strvec extra_headers = STRVEC_INIT; static struct auth_module *get_auth_module(const char *scheme, int create) { @@ -520,6 +521,9 @@ done: string_list_append(&hdrs, challenge); } + for (i = 0; i < extra_headers.nr; i++) + string_list_append(&hdrs, extra_headers.v[i]); + *wr = send_http_error(STDOUT_FILENO, 401, "Unauthorized", -1, &hdrs, *wr); } @@ -585,6 +589,8 @@ static int read_auth_config(const char *name, const char *val, void *data) string_list_clear(mod->tokens, 1); } else if (!strcmp(name, "auth.allowanonymous")) { allow_anonymous = git_config_bool(name, val); + } else if (!strcmp(name, "auth.extraheader")) { + strvec_push(&extra_headers, val); } else { warning("unknown auth config '%s'", name); } diff --git a/t/t5556-http-auth.sh b/t/t5556-http-auth.sh index 20fd9b09aef..2c16c8f72a5 100755 --- a/t/t5556-http-auth.sh +++ b/t/t5556-http-auth.sh @@ -178,6 +178,10 @@ test_expect_success CURL 'http auth server auth config' ' token = reset-tokens:the-only-valid-one allowAnonymous = false + + extraHeader = X-Extra-Header: abc + extraHeader = X-Extra-Header: 123 + extraHeader = X-Another: header\twith\twhitespace! EOF cat >OUT.expected <<-EOF && @@ -185,6 +189,9 @@ test_expect_success CURL 'http auth server auth config' ' WWW-Authenticate: with-params foo="replaced" q=1 WWW-Authenticate: no-explicit-challenge WWW-Authenticate: reset-tokens + X-Extra-Header: abc + X-Extra-Header: 123 + X-Another: header with whitespace! Error: 401 Unauthorized EOF From patchwork Fri Jan 20 22:08:48 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew John Cheetham X-Patchwork-Id: 13110579 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id DEB8DC27C76 for ; Fri, 20 Jan 2023 22:09:21 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229971AbjATWJU (ORCPT ); Fri, 20 Jan 2023 17:09:20 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58222 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229917AbjATWJE (ORCPT ); Fri, 20 Jan 2023 17:09:04 -0500 Received: from mail-wr1-x435.google.com (mail-wr1-x435.google.com [IPv6:2a00:1450:4864:20::435]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 97A4AA7301 for ; Fri, 20 Jan 2023 14:09:02 -0800 (PST) Received: by mail-wr1-x435.google.com with SMTP id r9so6012718wrw.4 for ; Fri, 20 Jan 2023 14:09:02 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:from:to:cc:subject:date :message-id:reply-to; bh=qCY5CaeVsGFH4YCKVAhrjSkUdtoSk2eEGMnVuvUbaAA=; b=B4UrI/jMnX84i7g4eryKywwkDLkJMrPZYak1NfGTP3hfkYd34pdAgsPAy896MEqFfw 8KtsKyip9fHSwTSR7P+sOwxFZVIw3yK2wlGH/2DF7vpRn8ZaUkZmCT5HV6ckSbBzUE3N fbtFU3qfSP5HWooolw6LbAIOrtI6piUJBInCY8b2Td9gNqsACdJ1KY9Bpuw2iN3pDl+U HWF3+fYQ73QvjE8RUoxP9dpbuebwFr/ZO5OlcI3Hu2hMD0p/RNUVWYNMZXFcTExUTEQH GaeysZ2hQLr6mfp9mYqsCATpAr2i0zQovSEOjj7H30XVjXhzjlNwsmLrEQJLMl3XAwrH 3cMw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=qCY5CaeVsGFH4YCKVAhrjSkUdtoSk2eEGMnVuvUbaAA=; b=sHkpQT4TXzvxQkzE39uD7IyqiIo+KMqmIznZSoh8xNet7BzZEOJVIXjoBXJeJhxVSN d2VzhuGXV+J2+1sTUpgr5XpwIqeXdn/plNz9KEKhl7Ll/Lq6N5DLpPaUWNdyW+Wk87xx k6r6b9/HMuyFQprGUzks4zdhYt0pW8lcwZGALF5NmKJV6b9UtNjbXkqEY1W+P8z3t5lL MwFoNwr4bdlordrlfdQgHcHMlFV2mzNAImW0udefVv06gUwV7CxgMY4/Al+KzHi5zaqq VlvOfPlXS1UM4vEY2JTKcjQv9iqLu+z9M4l5RucfVVTw48GGV/AjelB8QY++XMV1Bbvm g4Yw== X-Gm-Message-State: AFqh2krE+DPTNqQJ7/HnttdPGPf1jqAU6O8hV5Ye+K6ecXOmCPuXTx7Z ApN9VN+mVbKJARBgZcQusG2yUpJQ39s= X-Google-Smtp-Source: AMrXdXs2yC5jIyathdyeNSnJXScAXAxHZamk1hVs0EFb68Nu5SJFSr7amaNdyvr5q5BV/zCyqzueOw== X-Received: by 2002:a05:6000:384:b0:2bd:c83a:6e3 with SMTP id u4-20020a056000038400b002bdc83a06e3mr10087528wrf.7.1674252540907; Fri, 20 Jan 2023 14:09:00 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id w5-20020adfcd05000000b002bdc914a139sm28332160wrm.108.2023.01.20.14.09.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 20 Jan 2023 14:09:00 -0800 (PST) Message-Id: <4b1635b3f6968f8d755bdf6bc4ec7af77aefd315.1674252531.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Fri, 20 Jan 2023 22:08:48 +0000 Subject: [PATCH v7 10/12] http: replace unsafe size_t multiplication with st_mult Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Derrick Stolee , Lessley Dennington , Matthew John Cheetham , M Hickford , Jeff Hostetler , Glen Choo , Victoria Dye , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsA==?= Bjarmason , Matthew John Cheetham , Matthew John Cheetham Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Matthew John Cheetham From: Matthew John Cheetham Replace direct multiplication of two size_t parameters in curl response stream handling callback functions with `st_mult` to guard against overflows. Signed-off-by: Matthew John Cheetham --- http.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/http.c b/http.c index 8a5ba3f4776..a2a80318bb2 100644 --- a/http.c +++ b/http.c @@ -146,7 +146,7 @@ static int http_schannel_use_ssl_cainfo; size_t fread_buffer(char *ptr, size_t eltsize, size_t nmemb, void *buffer_) { - size_t size = eltsize * nmemb; + size_t size = st_mult(eltsize, nmemb); struct buffer *buffer = buffer_; if (size > buffer->buf.len - buffer->posn) @@ -176,7 +176,7 @@ curlioerr ioctl_buffer(CURL *handle, int cmd, void *clientp) size_t fwrite_buffer(char *ptr, size_t eltsize, size_t nmemb, void *buffer_) { - size_t size = eltsize * nmemb; + size_t size = st_mult(eltsize, nmemb); struct strbuf *buffer = buffer_; strbuf_add(buffer, ptr, size); From patchwork Fri Jan 20 22:08:49 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew John Cheetham X-Patchwork-Id: 13110578 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B9448C05027 for ; Fri, 20 Jan 2023 22:09:20 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229969AbjATWJT (ORCPT ); Fri, 20 Jan 2023 17:09:19 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58224 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229919AbjATWJE (ORCPT ); Fri, 20 Jan 2023 17:09:04 -0500 Received: from mail-wm1-x32d.google.com (mail-wm1-x32d.google.com [IPv6:2a00:1450:4864:20::32d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6D898A19A9 for ; Fri, 20 Jan 2023 14:09:02 -0800 (PST) Received: by mail-wm1-x32d.google.com with SMTP id q8so5063216wmo.5 for ; Fri, 20 Jan 2023 14:09:02 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:from:to:cc:subject:date :message-id:reply-to; bh=8E4ibgh1OESySHvLIe11OZBSXNLpINkcmnXPIcQ6rbo=; b=d7ZoTiUHYf1SL2gt299OQ/3JgrJryfp+sbevKEt83hjZUe8p1DUVNjXrdW98HOX1Bc 6w8DKIxYFfeLYGmIvBBfcmyfh6ytMLBFbhNsvyYFNKXPBrh71n4ieHWsBeU2lXk8mpPy gTPsbiP2Mmq2qqoJPvvA9dmqswop4UDWMxewuzDmTqbVYR2jG4VPm2azgRkeSJFJl0M7 gsq1lC99yFszTo4twMYc5DKbyi6pl0XYl74oaarlyNHHYyyspigOHXXyAYuKI/lxc4Wl ZHUoyDOD0l+wVVCCIeirhOJgaL/RIRWD+4r7WCINE4IDqjGfF8BIqX1sTUf9OHlLNQlv Xe7w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=8E4ibgh1OESySHvLIe11OZBSXNLpINkcmnXPIcQ6rbo=; b=DXP3PImGmCSvwY32+XyKoHzq4vJaBVmzV2i28KhdVsD+mf7MdfFRK9j/Oganwqo0Vw /8LFApxSqp2knXUvffa87n7OH5d5NYRXEN89EnZsyfBK/Y0jHoPPpenoi2pQIOLC/rp1 8O5AJ3A87dvSxTrNIe0Bj7QRjGsi1oSP3fIx+MGybrdpxQDvxcF7v5MHSiA8QUUgIrVI t7MSjYAXj9Vprna6oNTG7hRkrIAEEZY2I7gJ3aHRTkZeh9f/mzRKoVFEPfdMvlJ/9U/S cCJf/s+Lu4RW2vTUlwyKeuR0Hfg+v/HKpE6dAuQsUPI1KvOTP1Pb55Chc+dJID/lM9LR gllw== X-Gm-Message-State: AFqh2ko2o18M0LyM6xk3v59Mrc0oqpxkHnIqsgvn1nMN0VvHtnu8/erU 1r4Qs6mgVvpXDBQJ4gpG4O9roj+f3QM= X-Google-Smtp-Source: AMrXdXsdZtHDCGCq4/XG8nGXevCbhsPLEk2KZBaSP4OOHKqOz6rES6Pzzhi6Gg60S/dqf0t+rfNS6Q== X-Received: by 2002:a05:600c:920:b0:3da:22a6:7b6b with SMTP id m32-20020a05600c092000b003da22a67b6bmr15764296wmp.13.1674252541815; Fri, 20 Jan 2023 14:09:01 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id t15-20020a05600c198f00b003db32ccf4bfsm3108864wmq.41.2023.01.20.14.09.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 20 Jan 2023 14:09:01 -0800 (PST) Message-Id: <5f5e46038cf526714f3c5b89ffef2b895b503242.1674252531.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Fri, 20 Jan 2023 22:08:49 +0000 Subject: [PATCH v7 11/12] http: read HTTP WWW-Authenticate response headers Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Derrick Stolee , Lessley Dennington , Matthew John Cheetham , M Hickford , Jeff Hostetler , Glen Choo , Victoria Dye , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsA==?= Bjarmason , Matthew John Cheetham , Matthew John Cheetham Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Matthew John Cheetham From: Matthew John Cheetham Read and store the HTTP WWW-Authenticate response headers made for a particular request. This will allow us to pass important authentication challenge information to credential helpers or others that would otherwise have been lost. According to RFC2616 Section 4.2 [1], header field names are not case-sensitive meaning when collecting multiple values for the same field name, we can just use the case of the first observed instance of each field name and no normalisation is required. libcurl only provides us with the ability to read all headers recieved for a particular request, including any intermediate redirect requests or proxies. The lines returned by libcurl include HTTP status lines delinating any intermediate requests such as "HTTP/1.1 200". We use these lines to reset the strvec of WWW-Authenticate header values as we encounter them in order to only capture the final response headers. The collection of all header values matching the WWW-Authenticate header is complicated by the fact that it is legal for header fields to be continued over multiple lines, but libcurl only gives us one line at a time. In the future [2] we may be able to leverage functions to read headers from libcurl itself, but as of today we must do this ourselves. [1] https://datatracker.ietf.org/doc/html/rfc2616#section-4.2 [2] https://daniel.haxx.se/blog/2022/03/22/a-headers-api-for-libcurl/ Signed-off-by: Matthew John Cheetham --- credential.c | 1 + credential.h | 15 +++++++++ http.c | 94 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 110 insertions(+) diff --git a/credential.c b/credential.c index f6389a50684..897b4679333 100644 --- a/credential.c +++ b/credential.c @@ -22,6 +22,7 @@ void credential_clear(struct credential *c) free(c->username); free(c->password); string_list_clear(&c->helpers, 0); + strvec_clear(&c->wwwauth_headers); credential_init(c); } diff --git a/credential.h b/credential.h index f430e77fea4..6f2e5bc610b 100644 --- a/credential.h +++ b/credential.h @@ -2,6 +2,7 @@ #define CREDENTIAL_H #include "string-list.h" +#include "strvec.h" /** * The credentials API provides an abstracted way of gathering username and @@ -115,6 +116,19 @@ struct credential { */ struct string_list helpers; + /** + * A `strvec` of WWW-Authenticate header values. Each string + * is the value of a WWW-Authenticate header in an HTTP response, + * in the order they were received in the response. + */ + struct strvec wwwauth_headers; + + /** + * Internal use only. Used to keep track of split header fields + * in order to fold multiple lines into one value. + */ + unsigned header_is_last_match:1; + unsigned approved:1, configured:1, quit:1, @@ -130,6 +144,7 @@ struct credential { #define CREDENTIAL_INIT { \ .helpers = STRING_LIST_INIT_DUP, \ + .wwwauth_headers = STRVEC_INIT, \ } /* Initialize a credential structure, setting all fields to empty. */ diff --git a/http.c b/http.c index a2a80318bb2..595c93bc7a3 100644 --- a/http.c +++ b/http.c @@ -183,6 +183,98 @@ size_t fwrite_buffer(char *ptr, size_t eltsize, size_t nmemb, void *buffer_) return nmemb; } +static size_t fwrite_wwwauth(char *ptr, size_t eltsize, size_t nmemb, void *p) +{ + size_t size = st_mult(eltsize, nmemb); + struct strvec *values = &http_auth.wwwauth_headers; + struct strbuf buf = STRBUF_INIT; + const char *val; + + /* + * Header lines may not come NULL-terminated from libcurl so we must + * limit all scans to the maximum length of the header line, or leverage + * strbufs for all operations. + * + * In addition, it is possible that header values can be split over + * multiple lines as per RFC 2616 (even though this has since been + * deprecated in RFC 7230). A continuation header field value is + * identified as starting with a space or horizontal tab. + * + * The formal definition of a header field as given in RFC 2616 is: + * + * message-header = field-name ":" [ field-value ] + * field-name = token + * field-value = *( field-content | LWS ) + * field-content = + */ + + strbuf_add(&buf, ptr, size); + + /* Strip the CRLF that should be present at the end of each field */ + strbuf_trim_trailing_newline(&buf); + + /* Start of a new WWW-Authenticate header */ + if (skip_iprefix(buf.buf, "www-authenticate:", &val)) { + while (isspace(*val)) + val++; + + strvec_push(values, val); + http_auth.header_is_last_match = 1; + goto exit; + } + + /* + * This line could be a continuation of the previously matched header + * field. If this is the case then we should append this value to the + * end of the previously consumed value. + * Continuation lines start with at least one whitespace, maybe more, + * so we should collapse these down to a single SP (valid per the spec). + */ + if (http_auth.header_is_last_match && isspace(*buf.buf)) { + /* Trim leading whitespace from this continuation hdr line. */ + strbuf_ltrim(&buf); + + /* + * At this point we should always have at least one existing + * value, even if it is empty. Do not bother appending the new + * value if this continuation header is itself empty. + */ + if (!values->nr) { + BUG("should have at least one existing header value"); + } else if (buf.len) { + char *prev = xstrdup(values->v[values->nr - 1]); + + /* Join two non-empty values with a single space. */ + const char *const sp = *prev ? " " : ""; + + strvec_pop(values); + strvec_pushf(values, "%s%s%s", prev, sp, buf.buf); + free(prev); + } + + goto exit; + } + + /* This is the start of a new header we don't care about */ + http_auth.header_is_last_match = 0; + + /* + * If this is a HTTP status line and not a header field, this signals + * a different HTTP response. libcurl writes all the output of all + * response headers of all responses, including redirects. + * We only care about the last HTTP request response's headers so clear + * the existing array. + */ + if (istarts_with(buf.buf, "http/")) + strvec_clear(values); + +exit: + strbuf_release(&buf); + return size; +} + size_t fwrite_null(char *ptr, size_t eltsize, size_t nmemb, void *strbuf) { return nmemb; @@ -1864,6 +1956,8 @@ static int http_request(const char *url, fwrite_buffer); } + curl_easy_setopt(slot->curl, CURLOPT_HEADERFUNCTION, fwrite_wwwauth); + accept_language = http_get_accept_language_header(); if (accept_language) From patchwork Fri Jan 20 22:08:50 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew John Cheetham X-Patchwork-Id: 13110580 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A71B6C25B50 for ; Fri, 20 Jan 2023 22:09:24 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229973AbjATWJW (ORCPT ); Fri, 20 Jan 2023 17:09:22 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58288 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229587AbjATWJG (ORCPT ); Fri, 20 Jan 2023 17:09:06 -0500 Received: from mail-wm1-x329.google.com (mail-wm1-x329.google.com [IPv6:2a00:1450:4864:20::329]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6A131A314C for ; Fri, 20 Jan 2023 14:09:04 -0800 (PST) Received: by mail-wm1-x329.google.com with SMTP id d4-20020a05600c3ac400b003db1de2aef0so4662134wms.2 for ; Fri, 20 Jan 2023 14:09:04 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:from:to:cc:subject:date :message-id:reply-to; bh=FIBSatUxCSvGNYiN6zTAUA65IC5YxBv/B6yrODjpvhQ=; b=jcX+i3hFP/tc3XiSLszpmiH479YqmjHTaYPAEwrpFhKrRVylqMhWe+gReFzek8Tq/a UmOaeAdjyJHGvhiIZp804x7zZ+fhbj9VmALBb4NeSAOyqWynriy+d1I8KNOOxLmPrbMQ oif5j6GrdFKe03RX4SZkoluA5i1A2pAhkN/gUYpENUqddd8qmGechrpAiF0zYV64i9y+ xQGNEa+bMAD/si7T7AK4j+L6ufqD8LCxL1J/fQfODrhDuNQcGP0kVaxkCvr3LfY6wV/s jf0QbOHSf09WUR4dgYQRLFxAqhlfhVQQuusLm4RDrCU4IIv5A8pqjJK3IhBtHhyNmdOc 46bg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=FIBSatUxCSvGNYiN6zTAUA65IC5YxBv/B6yrODjpvhQ=; b=gwbSU7CP1xxmf8Yo7MmyPs7QsqljdphfiFfnf4gmd+7PJ+gNU/k6N74DVDBff54Q2Z tQ8QoEwvjDCQN0JmgkJNdBlPU1KAeG5kkjEtvpVkFvCqgUAG417zyKalNvtchaFXFst1 hrcmJChRFuE9/vaCLncIyDWyTQ7Gmse73pg/re57+qNqtK0qektIlPo1wIv+2sdCwob2 xvqOvlt2OJm67HT1EZ1yx92k5rJRRz31b1JZd2bHpCpb6Blq1dhouV8NelC8ZwHSOPSp f8jci9PaX90nLUlvAmcU2X3IVOlBbZWfSLUn2Cn/iod63LWBd2d4RCdLrUpN6+vVnfYq CYQg== X-Gm-Message-State: AFqh2kpRLbK2kqpoUjrdaEKgtMiooIRN0uvIt+swTyxwAeGutgL4IzAP Zmj/nlue+ubQ2680hf9uneqXiXD5udg= X-Google-Smtp-Source: AMrXdXv1ZDPoOUurlVuu+Z4Y2ZlpQMaaYPD0WoQi1IDjvQd4oS7tYW1Inlk49O1K3sgQd+BoR+T7JA== X-Received: by 2002:a05:600c:540c:b0:3cf:7704:50ce with SMTP id he12-20020a05600c540c00b003cf770450cemr15090369wmb.38.1674252542666; Fri, 20 Jan 2023 14:09:02 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id z7-20020a1c4c07000000b003d9a86a13bfsm3367726wmf.28.2023.01.20.14.09.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 20 Jan 2023 14:09:02 -0800 (PST) Message-Id: <09164f77d56e8efd1450091cf1b12af2bc6cf2f5.1674252531.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Fri, 20 Jan 2023 22:08:50 +0000 Subject: [PATCH v7 12/12] credential: add WWW-Authenticate header to cred requests Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Derrick Stolee , Lessley Dennington , Matthew John Cheetham , M Hickford , Jeff Hostetler , Glen Choo , Victoria Dye , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsA==?= Bjarmason , Matthew John Cheetham , Matthew John Cheetham Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Matthew John Cheetham From: Matthew John Cheetham Add the value of the WWW-Authenticate response header to credential requests. Credential helpers that understand and support HTTP authentication and authorization can use this standard header (RFC 2616 Section 14.47 [1]) to generate valid credentials. WWW-Authenticate headers can contain information pertaining to the authority, authentication mechanism, or extra parameters/scopes that are required. The current I/O format for credential helpers only allows for unique names for properties/attributes, so in order to transmit multiple header values (with a specific order) we introduce a new convention whereby a C-style array syntax is used in the property name to denote multiple ordered values for the same property. In this case we send multiple `wwwauth[]` properties where the order that the repeated attributes appear in the conversation reflects the order that the WWW-Authenticate headers appeared in the HTTP response. Add a set of tests to exercise the HTTP authentication header parsing and the interop with credential helpers. Credential helpers will receive WWW-Authenticate information in credential requests. [1] https://datatracker.ietf.org/doc/html/rfc2616#section-14.47 Signed-off-by: Matthew John Cheetham --- Documentation/git-credential.txt | 19 ++- credential.c | 11 ++ t/lib-credential-helper.sh | 27 ++++ t/t5556-http-auth.sh | 242 +++++++++++++++++++++++++++++++ 4 files changed, 298 insertions(+), 1 deletion(-) create mode 100644 t/lib-credential-helper.sh diff --git a/Documentation/git-credential.txt b/Documentation/git-credential.txt index ac2818b9f66..50759153ef1 100644 --- a/Documentation/git-credential.txt +++ b/Documentation/git-credential.txt @@ -113,7 +113,13 @@ separated by an `=` (equals) sign, followed by a newline. The key may contain any bytes except `=`, newline, or NUL. The value may contain any bytes except newline or NUL. -In both cases, all bytes are treated as-is (i.e., there is no quoting, +Attributes with keys that end with C-style array brackets `[]` can have +multiple values. Each instance of a multi-valued attribute forms an +ordered list of values - the order of the repeated attributes defines +the order of the values. An empty multi-valued attribute (`key[]=\n`) +acts to clear any previous entries and reset the list. + +In all cases, all bytes are treated as-is (i.e., there is no quoting, and one cannot transmit a value with newline or NUL in it). The list of attributes is terminated by a blank line or end-of-file. @@ -160,6 +166,17 @@ empty string. Components which are missing from the URL (e.g., there is no username in the example above) will be left unset. +`wwwauth[]`:: + + When an HTTP response is received by Git that includes one or more + 'WWW-Authenticate' authentication headers, these will be passed by Git + to credential helpers. ++ +Each 'WWW-Authenticate' header value is passed as a multi-valued +attribute 'wwwauth[]', where the order of the attributes is the same as +they appear in the HTTP response. This attribute is 'one-way' from Git +to pass additional information to credential helpers. + Unrecognised attributes are silently discarded. GIT diff --git a/credential.c b/credential.c index 897b4679333..9f39ebc3c7e 100644 --- a/credential.c +++ b/credential.c @@ -263,6 +263,16 @@ static void credential_write_item(FILE *fp, const char *key, const char *value, fprintf(fp, "%s=%s\n", key, value); } +static void credential_write_strvec(FILE *fp, const char *key, + const struct strvec *vec) +{ + char *full_key = xstrfmt("%s[]", key); + for (size_t i = 0; i < vec->nr; i++) { + credential_write_item(fp, full_key, vec->v[i], 0); + } + free(full_key); +} + void credential_write(const struct credential *c, FILE *fp) { credential_write_item(fp, "protocol", c->protocol, 1); @@ -270,6 +280,7 @@ void credential_write(const struct credential *c, FILE *fp) credential_write_item(fp, "path", c->path, 0); credential_write_item(fp, "username", c->username, 0); credential_write_item(fp, "password", c->password, 0); + credential_write_strvec(fp, "wwwauth", &c->wwwauth_headers); } static int run_credential_helper(struct credential *c, diff --git a/t/lib-credential-helper.sh b/t/lib-credential-helper.sh new file mode 100644 index 00000000000..8b0e4414234 --- /dev/null +++ b/t/lib-credential-helper.sh @@ -0,0 +1,27 @@ +setup_credential_helper() { + test_expect_success 'setup credential helper' ' + CREDENTIAL_HELPER="$TRASH_DIRECTORY/credential-helper.sh" && + export CREDENTIAL_HELPER && + echo $CREDENTIAL_HELPER && + + write_script "$CREDENTIAL_HELPER" <<-\EOF + cmd=$1 + teefile=$cmd-query.cred + catfile=$cmd-reply.cred + sed -n -e "/^$/q" -e "p" >> $teefile + if test "$cmd" = "get"; then + cat $catfile + fi + EOF + ' +} + +set_credential_reply() { + cat >"$TRASH_DIRECTORY/$1-reply.cred" +} + +expect_credential_query() { + cat >"$TRASH_DIRECTORY/$1-expect.cred" && + test_cmp "$TRASH_DIRECTORY/$1-expect.cred" \ + "$TRASH_DIRECTORY/$1-query.cred" +} diff --git a/t/t5556-http-auth.sh b/t/t5556-http-auth.sh index 2c16c8f72a5..93b7c178da6 100755 --- a/t/t5556-http-auth.sh +++ b/t/t5556-http-auth.sh @@ -4,6 +4,7 @@ test_description='test http auth header and credential helper interop' TEST_NO_CREATE_REPO=1 . ./test-lib.sh +. "$TEST_DIRECTORY"/lib-credential-helper.sh test_set_port GIT_TEST_HTTP_PROTOCOL_PORT @@ -33,6 +34,8 @@ test_expect_success 'setup repos' ' git -C "$REPO_DIR" branch -M main ' +setup_credential_helper + run_http_server_worker() { ( cd "$REPO_DIR" @@ -101,6 +104,7 @@ per_test_cleanup () { stop_http_server && rm -f OUT.* && rm -f IN.* && + rm -f *.cred && rm -f auth.config } @@ -218,4 +222,242 @@ test_expect_success 'http auth anonymous no challenge' ' git ls-remote $ORIGIN_URL ' +test_expect_success 'http auth www-auth headers to credential helper basic valid' ' + test_when_finished "per_test_cleanup" && + # base64("alice:secret-passwd") + USERPASS64=YWxpY2U6c2VjcmV0LXBhc3N3ZA== && + export USERPASS64 && + + cat >auth.config <<-EOF && + [auth] + challenge = basic:realm=\"example.com\" + token = basic:$USERPASS64 + EOF + + start_http_server --auth-config="$TRASH_DIRECTORY/auth.config" && + + set_credential_reply get <<-EOF && + protocol=http + host=$HOST_PORT + username=alice + password=secret-passwd + EOF + + git -c "credential.helper=!\"$CREDENTIAL_HELPER\"" ls-remote $ORIGIN_URL && + + expect_credential_query get <<-EOF && + protocol=http + host=$HOST_PORT + wwwauth[]=basic realm="example.com" + EOF + + expect_credential_query store <<-EOF + protocol=http + host=$HOST_PORT + username=alice + password=secret-passwd + EOF +' + +test_expect_success 'http auth www-auth headers to credential helper ignore case valid' ' + test_when_finished "per_test_cleanup" && + # base64("alice:secret-passwd") + USERPASS64=YWxpY2U6c2VjcmV0LXBhc3N3ZA== && + export USERPASS64 && + + cat >auth.config <<-EOF && + [auth] + challenge = basic:realm=\"example.com\" + token = basic:$USERPASS64 + extraHeader = wWw-aUtHeNtIcAtE: bEaRer auThoRiTy=\"id.example.com\" + EOF + + start_http_server --auth-config="$TRASH_DIRECTORY/auth.config" && + + set_credential_reply get <<-EOF && + protocol=http + host=$HOST_PORT + username=alice + password=secret-passwd + EOF + + git -c "credential.helper=!\"$CREDENTIAL_HELPER\"" ls-remote $ORIGIN_URL && + + expect_credential_query get <<-EOF && + protocol=http + host=$HOST_PORT + wwwauth[]=basic realm="example.com" + wwwauth[]=bEaRer auThoRiTy="id.example.com" + EOF + + expect_credential_query store <<-EOF + protocol=http + host=$HOST_PORT + username=alice + password=secret-passwd + EOF +' + +test_expect_success 'http auth www-auth headers to credential helper continuation hdr' ' + test_when_finished "per_test_cleanup" && + # base64("alice:secret-passwd") + USERPASS64=YWxpY2U6c2VjcmV0LXBhc3N3ZA== && + export USERPASS64 && + + cat >auth.config <<-EOF && + [auth] + challenge = "bearer:authority=\"id.example.com\"\\n q=1\\n \\t p=0" + challenge = basic:realm=\"example.com\" + token = basic:$USERPASS64 + EOF + + start_http_server --auth-config="$TRASH_DIRECTORY/auth.config" && + + set_credential_reply get <<-EOF && + protocol=http + host=$HOST_PORT + username=alice + password=secret-passwd + EOF + + git -c "credential.helper=!\"$CREDENTIAL_HELPER\"" ls-remote $ORIGIN_URL && + + expect_credential_query get <<-EOF && + protocol=http + host=$HOST_PORT + wwwauth[]=bearer authority="id.example.com" q=1 p=0 + wwwauth[]=basic realm="example.com" + EOF + + expect_credential_query store <<-EOF + protocol=http + host=$HOST_PORT + username=alice + password=secret-passwd + EOF +' + +test_expect_success 'http auth www-auth headers to credential helper empty continuation hdrs' ' + test_when_finished "per_test_cleanup" && + # base64("alice:secret-passwd") + USERPASS64=YWxpY2U6c2VjcmV0LXBhc3N3ZA== && + export USERPASS64 && + + cat >auth.config <<-EOF && + [auth] + challenge = basic:realm=\"example.com\" + token = basic:$USERPASS64 + extraheader = "WWW-Authenticate:" + extraheader = " " + extraheader = " bearer authority=\"id.example.com\"" + EOF + + start_http_server --auth-config="$TRASH_DIRECTORY/auth.config" && + + set_credential_reply get <<-EOF && + protocol=http + host=$HOST_PORT + username=alice + password=secret-passwd + EOF + + git -c "credential.helper=!\"$CREDENTIAL_HELPER\"" ls-remote $ORIGIN_URL && + + expect_credential_query get <<-EOF && + protocol=http + host=$HOST_PORT + wwwauth[]=basic realm="example.com" + wwwauth[]=bearer authority="id.example.com" + EOF + + expect_credential_query store <<-EOF + protocol=http + host=$HOST_PORT + username=alice + password=secret-passwd + EOF +' + +test_expect_success 'http auth www-auth headers to credential helper custom schemes' ' + test_when_finished "per_test_cleanup" && + # base64("alice:secret-passwd") + USERPASS64=YWxpY2U6c2VjcmV0LXBhc3N3ZA== && + export USERPASS64 && + + cat >auth.config <<-EOF && + [auth] + challenge = "foobar:alg=test widget=1" + challenge = "bearer:authority=\"id.example.com\" q=1 p=0" + challenge = basic:realm=\"example.com\" + token = basic:$USERPASS64 + EOF + + start_http_server --auth-config="$TRASH_DIRECTORY/auth.config" && + + set_credential_reply get <<-EOF && + protocol=http + host=$HOST_PORT + username=alice + password=secret-passwd + EOF + + git -c "credential.helper=!\"$CREDENTIAL_HELPER\"" ls-remote $ORIGIN_URL && + + expect_credential_query get <<-EOF && + protocol=http + host=$HOST_PORT + wwwauth[]=foobar alg=test widget=1 + wwwauth[]=bearer authority="id.example.com" q=1 p=0 + wwwauth[]=basic realm="example.com" + EOF + + expect_credential_query store <<-EOF + protocol=http + host=$HOST_PORT + username=alice + password=secret-passwd + EOF +' + +test_expect_success 'http auth www-auth headers to credential helper invalid' ' + test_when_finished "per_test_cleanup" && + # base64("alice:secret-passwd") + USERPASS64=YWxpY2U6c2VjcmV0LXBhc3N3ZA== && + export USERPASS64 && + + cat >auth.config <<-EOF && + [auth] + challenge = "bearer:authority=\"id.example.com\" q=1 p=0" + challenge = basic:realm=\"example.com\" + token = basic:$USERPASS64 + EOF + + start_http_server --auth-config="$TRASH_DIRECTORY/auth.config" && + + set_credential_reply get <<-EOF && + protocol=http + host=$HOST_PORT + username=alice + password=invalid-passwd + EOF + + test_must_fail git -c "credential.helper=!\"$CREDENTIAL_HELPER\"" ls-remote $ORIGIN_URL && + + expect_credential_query get <<-EOF && + protocol=http + host=$HOST_PORT + wwwauth[]=bearer authority="id.example.com" q=1 p=0 + wwwauth[]=basic realm="example.com" + EOF + + expect_credential_query erase <<-EOF + protocol=http + host=$HOST_PORT + username=alice + password=invalid-passwd + wwwauth[]=bearer authority="id.example.com" q=1 p=0 + wwwauth[]=basic realm="example.com" + EOF +' + test_done