From patchwork Fri Sep 9 15:06:21 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sun Chao X-Patchwork-Id: 12971893 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 841F0ECAAA1 for ; Fri, 9 Sep 2022 15:06:35 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231811AbiIIPGe (ORCPT ); Fri, 9 Sep 2022 11:06:34 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39418 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231660AbiIIPGc (ORCPT ); Fri, 9 Sep 2022 11:06:32 -0400 Received: from mail-wm1-x32b.google.com (mail-wm1-x32b.google.com [IPv6:2a00:1450:4864:20::32b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 163201398AF for ; Fri, 9 Sep 2022 08:06:30 -0700 (PDT) Received: by mail-wm1-x32b.google.com with SMTP id az6so1606867wmb.4 for ; Fri, 09 Sep 2022 08:06:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:from:to:cc:subject:date; bh=pIG82mq/Y3cFYjwIHAD0Rz74+6bfAtDlCiMLrgOVCn8=; b=gEbhnVNA6DYZBvKKOnIINftoumiuedsvB6WNWKcGhEdVLt8kbT2g/ZE9Y0V3YWwpNl kavR2SFfVoLhOUY4fZRySXNGJdIaqzdxPV3kHuwdplpC/714Eis7Oc5M5JEc/gZkLJf3 2gVXXQ8QdTNGqBHuAGxuUyKZeyYLcqStPjkpheutncpztnHdXG488oT3qSaf36C3eJvN Z+z72zDLEiGYS7qjcddtAm3nQMWvZEY2n1Tq7zP947Mjx6PHKjlW50EB+Xot8D1fgCGQ RhNOqts9JDG3JNAosNAbCJ475jvMUvad/VGvyY6uHENO5St6YB4t5M7lyc9iKHfWQ/cH 85lg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:x-gm-message-state:from:to:cc :subject:date; bh=pIG82mq/Y3cFYjwIHAD0Rz74+6bfAtDlCiMLrgOVCn8=; b=reeKhH8UiXPjRYHZVYLGMGVztw+WHTuux1BCpdzzPsS60MsNq9oaI4zdQUAHK8XfKX G4DD0HEbBpmq9uhmRfZnyu8pIUHEn26A4sieip15qMZWpOhuyc9gRMmY7UqNTU/ffrmi zkVCdHzfnIzlQYQSfte1H7o5R3ykpu/TUpBTxugCkRbRhLnTWGVwKSLsNLcBTueBPMng Uzz2Cvxns5HyvnQWtU0xb03IyTKnxV4ny8MlI3dtJxkD+C/JNqBdhK8vvea4zNHhgw7F RgQuyxPlwycqmbpVh7P7n/pxKksSj3ZseNV4agewJoYtr6165ltSHruz+PT8iDpL970A M9sA== X-Gm-Message-State: ACgBeo1VEuosPXJJ0TzMz8WJWfHM6VGbODNzacQYwyI/xrdjQ83UB44F 8dkFfLiX5XGvJDIB4Qkc+oCspwVZOHE= X-Google-Smtp-Source: AA6agR7rFUM75DvhtesytSQPD93TT7ZOgn5oY8FIFsExT14j5H5sLobalQGnpmHMUkETsD9bNycd0Q== X-Received: by 2002:a1c:770f:0:b0:3a5:ef7f:2973 with SMTP id t15-20020a1c770f000000b003a5ef7f2973mr5793371wmi.111.1662735988301; Fri, 09 Sep 2022 08:06:28 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id l10-20020a5d560a000000b00226d1711276sm761183wrv.1.2022.09.09.08.06.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 09 Sep 2022 08:06:27 -0700 (PDT) Message-Id: <278bd185aec26285f8c00aca838f89e5f3877748.1662735985.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Fri, 09 Sep 2022 15:06:21 +0000 Subject: [PATCH v5 1/5] hiderefs: add hide-refs hook to hide refs dynamically Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Sun Chao <16657101987@163.com>, Sun Chao Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Sun Chao From: Sun Chao Gerrit is implemented by JGit and is known as a centralized workflow system which supports reference-level access control for repository. If we choose to work in centralized workflow like what Gerrit provided, reference-level access control is needed and we might add a reference filter hook `hide-refs` to hide the private data. This hook would be invoked by 'git-receive-pack' and 'git-upload-pack' during the reference discovery phase, each reference will be filtered with this hook. The hook executes once with no arguments for each 'git-upload-pack' and 'git-receive-pack' process. Once the hook is invoked, a version number and server process name ('uploadpack' or 'receive') will send to it in pkt-line format, followed by a flush-pkt. The hook should respond with its version number. During reference discovery phase, each reference will be filtered by this hook. In the following example, the letter 'G' stands for 'git-receive-pack' or 'git-upload-pack' and the letter 'H' stands for this hook. The hook decides if the reference will be hidden or not, it sends result back in pkt-line format protocol, a response "hide" means the references will be hidden to the client. # Version negotiation G: PKT-LINE(version=1\0uploadpack) G: flush-pkt H: PKT-LINE(version=1) H: flush-pkt # Send reference filter request to hook G: PKT-LINE(ref :) G: flush-pkt # Receive result from the hook. # Case 1: this reference is hidden H: PKT-LINE(hide) H: flush-pkt # Case 2: this reference can be advertised H: flush-pkt To enable the `hide-refs` hook, we should config hiderefs with `hook:` option, eg: git config --add transfer.hiderefs hook:refs/prefix1/ git config --add uploadpack.hiderefs hook:!refs/prefix2/ Signed-off-by: Sun Chao --- refs.c | 229 ++++++++++++++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 219 insertions(+), 10 deletions(-) diff --git a/refs.c b/refs.c index 92819732ab7..a99734fedcd 100644 --- a/refs.c +++ b/refs.c @@ -8,6 +8,7 @@ #include "lockfile.h" #include "iterator.h" #include "refs.h" +#include "pkt-line.h" #include "refs/refs-internal.h" #include "run-command.h" #include "hook.h" @@ -1384,10 +1385,14 @@ char *shorten_unambiguous_ref(const char *refname, int strict) } static struct string_list *hide_refs; +static struct string_list *hook_hide_refs; +static struct strbuf hide_refs_section = STRBUF_INIT; int parse_hide_refs_config(const char *var, const char *value, const char *section) { const char *key; + int hook = 0; + if (!strcmp("transfer.hiderefs", var) || (!parse_config_key(var, section, NULL, NULL, &key) && !strcmp(key, "hiderefs"))) { @@ -1396,27 +1401,218 @@ int parse_hide_refs_config(const char *var, const char *value, const char *secti if (!value) return config_error_nonbool(var); + + /* + * the prefix 'hook:' means that the matched refs will be + * checked by the hide-refs hook dynamically, we need to put + * the 'ref' string to the hook_hide_refs list + */ + if (skip_prefix(value, "hook:", &value)) { + if (!strlen(value)) + return error(_("missing value for '%s' after hook option"), var); + hook = 1; + } + ref = xstrdup(value); len = strlen(ref); while (len && ref[len - 1] == '/') ref[--len] = '\0'; - if (!hide_refs) { - CALLOC_ARRAY(hide_refs, 1); - hide_refs->strdup_strings = 1; + + if (hook) { + if (!hook_hide_refs) { + CALLOC_ARRAY(hook_hide_refs, 1); + hook_hide_refs->strdup_strings = 1; + } + string_list_append(hook_hide_refs, ref); + } else { + if (!hide_refs) { + CALLOC_ARRAY(hide_refs, 1); + hide_refs->strdup_strings = 1; + } + string_list_append(hide_refs, ref); } - string_list_append(hide_refs, ref); } + + /* + * Once hide-refs hook is invoked, Git need to do version negotiation, + * with it, version number and process name ('uploadpack' or 'receive') + * will send to it in pkt-line format, the proccess name is recorded + * by hide_refs_section + */ + if (hook && hide_refs_section.len == 0) + strbuf_addstr(&hide_refs_section, section); + return 0; } -int ref_is_hidden(const char *refname, const char *refname_full) +static struct child_process *hide_refs_proc; +static struct packet_reader *hide_refs_reader; + +/* + * Create the hide-refs hook child process and complete version negotiation, + * return non-zero upon success, otherwise 0 + */ +static int create_hide_refs_process(void) +{ + struct child_process *proc; + struct packet_reader *reader; + const char *hook_path; + int version = 0; + int err; + + hook_path = find_hook("hide-refs"); + if (!hook_path) + return 0; + + proc = (struct child_process *)xcalloc(1, sizeof (struct child_process)); + reader = (struct packet_reader *)xcalloc(1, sizeof(struct packet_reader)); + + child_process_init(proc); + strvec_push(&proc->args, hook_path); + proc->in = -1; + proc->out = -1; + proc->trace2_hook_name = "hide-refs"; + proc->err = 0; + + err = start_command(proc); + if (err) + goto cleanup; + + sigchain_push(SIGPIPE, SIG_IGN); + + /* Version negotiaton */ + packet_reader_init(reader, proc->out, NULL, 0, + PACKET_READ_CHOMP_NEWLINE | PACKET_READ_GENTLE_ON_EOF); + err = packet_write_fmt_gently(proc->in, "version=1%c%s", '\0', hide_refs_section.buf); + if (!err) + err = packet_flush_gently(proc->in); + + if (!err) + for (;;) { + enum packet_read_status status; + + status = packet_reader_read(reader); + if (status != PACKET_READ_NORMAL) { + /* Check whether hide-refs exited abnormally */ + if (status == PACKET_READ_EOF) + goto failure; + break; + } + + if (reader->pktlen > 8 && starts_with(reader->line, "version=")) { + version = atoi(reader->line + 8); + } + } + + if (err) + goto failure; + + switch (version) { + case 0: + /* fallthrough */ + case 1: + break; + default: + trace_printf(_("hook hide-refs version '%d' is not supported"), version); + goto failure; + } + + sigchain_pop(SIGPIPE); + + hide_refs_proc = proc; + hide_refs_reader = reader; + return 1; + +failure: + close(proc->in); + close(proc->out); + kill(proc->pid, SIGTERM); + finish_command_in_signal(proc); + +cleanup: + free(proc); + free(reader); + sigchain_pop(SIGPIPE); + return 0; +} + +/* If hide-refs child process start failed, set skip_hide_refs_proc to true */ +static int skip_hide_refs_proc; + +/* + * Return non-zero if hide-refs hook want to hide the ref and 0 otherwise, + * and return 0 if hide-refs child proccess start failed or exit abnormally + */ +static int ref_hidden_check_by_hook(const char *refname, const char *refname_full) +{ + struct strbuf buf = STRBUF_INIT; + int err; + int ret = 0; + + if (skip_hide_refs_proc) + return 0; + + if (!hide_refs_proc) + if (!create_hide_refs_process()) { + skip_hide_refs_proc = 1; + return 0; + } + + sigchain_push(SIGPIPE, SIG_IGN); + err = packet_write_fmt_gently(hide_refs_proc->in, "ref %s:%s", refname, refname_full); + if (err) + goto cleanup; + + err = packet_flush_gently(hide_refs_proc->in); + if (err) + goto cleanup; + + for (;;) { + enum packet_read_status status; + + status = packet_reader_read(hide_refs_reader); + if (status != PACKET_READ_NORMAL) { + /* Check whether hide-refs exited abnormally */ + if (status == PACKET_READ_EOF) + goto cleanup; + break; + } + + strbuf_addstr(&buf, hide_refs_reader->line); + } + + if (!strncmp("hide", buf.buf, 4)) + ret = 1; + + sigchain_pop(SIGPIPE); + return ret; + +cleanup: + close(hide_refs_proc->in); + close(hide_refs_proc->out); + kill(hide_refs_proc->pid, SIGTERM); + finish_command_in_signal(hide_refs_proc); + + free(hide_refs_proc); + free(hide_refs_reader); + sigchain_pop(SIGPIPE); + + skip_hide_refs_proc = 1; + return 0; +} + +static int ref_hidden_check(const char *refname, const char *refname_full, int hook) { + struct string_list *hide_refs_list = hide_refs; int i; - if (!hide_refs) + if (hook) + hide_refs_list = hook_hide_refs; + + if (!hide_refs_list) return 0; - for (i = hide_refs->nr - 1; i >= 0; i--) { - const char *match = hide_refs->items[i].string; + for (i = hide_refs_list->nr - 1; i >= 0; i--) { + const char *match = hide_refs_list->items[i].string; const char *subject; int neg = 0; const char *p; @@ -1436,12 +1632,25 @@ int ref_is_hidden(const char *refname, const char *refname_full) /* refname can be NULL when namespaces are used. */ if (subject && skip_prefix(subject, match, &p) && - (!*p || *p == '/')) - return !neg; + (!*p || *p == '/')) { + if (neg) + return 0; + if (!hook) + return 1; + return ref_hidden_check_by_hook(refname, refname_full); + } } return 0; } +int ref_is_hidden(const char *refname, const char *refname_full) +{ + if (ref_hidden_check(refname, refname_full, 0) || + ref_hidden_check(refname, refname_full, 1)) + return 1; + return 0; +} + const char *find_descendant_ref(const char *dirname, const struct string_list *extras, const struct string_list *skip) From patchwork Fri Sep 9 15:06:22 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sun Chao X-Patchwork-Id: 12971895 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 01D4FECAAA1 for ; Fri, 9 Sep 2022 15:06:41 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231995AbiIIPGj (ORCPT ); Fri, 9 Sep 2022 11:06:39 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39538 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231826AbiIIPGe (ORCPT ); Fri, 9 Sep 2022 11:06:34 -0400 Received: from mail-wr1-x42a.google.com (mail-wr1-x42a.google.com [IPv6:2a00:1450:4864:20::42a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BAF8D13BC60 for ; Fri, 9 Sep 2022 08:06:31 -0700 (PDT) Received: by mail-wr1-x42a.google.com with SMTP id d2so3276983wrn.1 for ; Fri, 09 Sep 2022 08:06:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:from:to:cc:subject:date; bh=gKH+uDBlVmFKJIO50p1B6TcRcnRBM1A9OiZ+h0u7f2U=; b=NasHWLuHOMEZMpTnGPkwKkEpCdLEA3mLXWfrYoHxiBYyE6qHCRQCTwlcxQt9xcQI4f cdmxDwPTCFY+zHHkpUv03xohnrJ7GjSJyUo/zTOL+BblGAo2x5dxJHpGFwnJZG3zagg0 bMnUOjXPFde6jax22v3ShjdAYv/dAWa89PIjW1lRP7nJepGrncR5mIZ6disWcnbDsKCg CaKfU9ayzSNF3uYXK06JCRxpftNXLPLNKkOhI7e+H8JObKRs6HE8e08lzGc4Aw+Zw/4z G1DwafhA1yCCQkmt1YiYYOJz+L56aBVuOR8TsGrIpq69UITu/X4TiJrCRC2ePu+AKDKL dzlg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:x-gm-message-state:from:to:cc :subject:date; bh=gKH+uDBlVmFKJIO50p1B6TcRcnRBM1A9OiZ+h0u7f2U=; b=WuLqXZ6A5EfGnhTovmN27FsKU0MOJ6sslN1vGMKVn/WxrNhpQs04k0KuI9MQReP7ES LQi9e7LJtOG25hnwuNgcuh0uoX4r9SpxaU9yz+3T0qMn+ikIxYJLBbxFrdsDnsmVcG1o Imc3rMwQMYYCH8Yyw35pEepibhT4jPVBuQ5YthMS6i2nBIroMI5E0n4TSuEAaiKtAiCU M4wKotGe0HtBBz6F1u72R9wOsj2vIBEpwReLRHsXeKRKCP4QcTBubPtsYkfQGj3AusZT ddMM+xeHxg8elPK4ClnASf5dBdqUDUy8iAqijWFXfVRPvZEh4zEQ8bsVQRmGxxoyK5Qy Uk5Q== X-Gm-Message-State: ACgBeo2nwD+0Rs/S5DK2sNLyy/FAA73ZGmM2yUqpJXpsysvaLAqVt2Cu aGy4+gPLSY6HRINKy4xK+zZJ5u2yzt8= X-Google-Smtp-Source: AA6agR4ybIRV3tJGwpGOW/G2nCdED7EW+v1A63fb3DrTRB03uHO4DFHw5SnwoizQkX6Of2ZM7DyMuA== X-Received: by 2002:a05:6000:144d:b0:21f:a4a0:dbfa with SMTP id v13-20020a056000144d00b0021fa4a0dbfamr8309874wrx.701.1662735989886; Fri, 09 Sep 2022 08:06:29 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id c17-20020a05600c4a1100b003a61306d79dsm819215wmp.41.2022.09.09.08.06.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 09 Sep 2022 08:06:29 -0700 (PDT) Message-Id: <0df5ecc216d86e9308473a8eb207d33b37fa2aa4.1662735985.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Fri, 09 Sep 2022 15:06:22 +0000 Subject: [PATCH v5 2/5] hiderefs: use new flag to mark force hidden refs Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Sun Chao <16657101987@163.com>, Sun Chao Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Sun Chao From: Sun Chao If uploadpack.allowTipSHA1InWant or uploadpack.allowReachableSHA1InWant are set to true, the private commits of hide refs can be fetched by client. The new hide-refs hook are used to hide our refs and we wish to hide the private commits either. Now we have hide-refs hook to hide refs dynamically, a new flag `HIDDEN_REF_FORCE` is used to mark a ref if hide-refs hook decide to hide it, and we make sure the wire protocol V1 will reject to send the private commits of these refs even if uploadpack.allowTipSHA1InWant or uploadpack.allowReachableSHA1InWant are set to true. Signed-off-by: Sun Chao --- refs.c | 39 +++++++++++++++++++++++++++++++++++++++ refs.h | 3 +++ upload-pack.c | 14 -------------- 3 files changed, 42 insertions(+), 14 deletions(-) diff --git a/refs.c b/refs.c index a99734fedcd..3fbdf967bc6 100644 --- a/refs.c +++ b/refs.c @@ -1651,6 +1651,45 @@ int ref_is_hidden(const char *refname, const char *refname_full) return 0; } +#define OUR_REF (1u << 12) +#define HIDDEN_REF (1u << 19) +/* + * Use this flag to mark a ref that is hided by hide-refs hook, its private + * commits (tip or non-tip commits, not reachable by the refs not hided by + * hide-refs hook) will be force hidden to the client, which means client can + * not fetch such kind of commit even uploadpack.allowTipSHA1InWant or + * uploadpack.allowReachableSHA1InWant are set to true + */ +#define HIDDEN_REF_FORCE (1u << 20) + +static unsigned int ref_hidden_flag(const char *refname, const char *refname_full) +{ + if (ref_hidden_check(refname, refname_full, 1)) + return HIDDEN_REF_FORCE; + else if (ref_hidden_check(refname, refname_full, 0)) + return HIDDEN_REF; + return OUR_REF; +} + +int mark_our_ref(const char *refname, const char *refname_full, + const struct object_id *oid) +{ + struct object *o; + unsigned int flag; + + if (!oid || is_null_oid(oid)) { + return 0; + } + + o = lookup_unknown_object(the_repository, oid); + flag = ref_hidden_flag(refname, refname_full); + o->flags |= flag; + + if (flag & OUR_REF) + return 0; + return 1; +} + const char *find_descendant_ref(const char *dirname, const struct string_list *extras, const struct string_list *skip) diff --git a/refs.h b/refs.h index d6575b8c2bd..2feabfe35c4 100644 --- a/refs.h +++ b/refs.h @@ -819,6 +819,9 @@ int parse_hide_refs_config(const char *var, const char *value, const char *); * parameter always points to the full ref name. */ int ref_is_hidden(const char *, const char *); +/* return non-zero if the ref is hidden, otherwise 0 */ +int mark_our_ref(const char *refname, const char *refname_full, + const struct object_id *oid); enum ref_type { REF_TYPE_PER_WORKTREE, /* refs inside refs/ but not shared */ diff --git a/upload-pack.c b/upload-pack.c index b217a1f469e..a8ca5d1c26e 100644 --- a/upload-pack.c +++ b/upload-pack.c @@ -1155,20 +1155,6 @@ static void receive_needs(struct upload_pack_data *data, packet_flush(1); } -/* return non-zero if the ref is hidden, otherwise 0 */ -static int mark_our_ref(const char *refname, const char *refname_full, - const struct object_id *oid) -{ - struct object *o = lookup_unknown_object(the_repository, oid); - - if (ref_is_hidden(refname, refname_full)) { - o->flags |= HIDDEN_REF; - return 1; - } - o->flags |= OUR_REF; - return 0; -} - static int check_ref(const char *refname_full, const struct object_id *oid, int flag, void *cb_data) { From patchwork Fri Sep 9 15:06:23 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sun Chao X-Patchwork-Id: 12971896 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5555AECAAD3 for ; Fri, 9 Sep 2022 15:06:43 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232000AbiIIPGm (ORCPT ); Fri, 9 Sep 2022 11:06:42 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39542 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231833AbiIIPGe (ORCPT ); Fri, 9 Sep 2022 11:06:34 -0400 Received: from mail-wm1-x332.google.com (mail-wm1-x332.google.com [IPv6:2a00:1450:4864:20::332]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 56B501398AF for ; Fri, 9 Sep 2022 08:06:33 -0700 (PDT) Received: by mail-wm1-x332.google.com with SMTP id az6so1606999wmb.4 for ; Fri, 09 Sep 2022 08:06:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:from:to:cc:subject:date; bh=pYCt+GIxlPZabbIqw3S4SCnTIXZpAJVLZCV4Z03hD3Q=; b=ToklzmSvJDUwxywLPIwcwyvaXBwscZJwklHc9V17qbWWWIuoSohXGTHC5PdhddsTJ9 vL6E3PChql2bV7qSU2WuB4u7VTUK5DDAYI696DNUcJ3rfwGcJxUhk42a8rzHKJgD7TrH QVSLgp4UuZpF99o7Q0Ar0nlUXtlGtGsnvb74DMnQWNgCYBN0NS9RCUai2EOFK9fuGpyx W8j43QoBFwHHTJkTDGyYwXaLboRzHyVhx+v73zIUZhKrP/HHM5BbIbsFfqnXmXOVbeUp zNX3lZ/Uq9QIFUh6yKE9zz6Hu564FEpdekZ9G1wOO83dlDZRZoiUVElFBh/P6QgZTSjc dRJg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:x-gm-message-state:from:to:cc :subject:date; bh=pYCt+GIxlPZabbIqw3S4SCnTIXZpAJVLZCV4Z03hD3Q=; b=I82FvAD0cog4dj19TDJtF0vkTpPS+zUjJpqQ3cw6H7a0TImNvThbe1UiilPvYjVtk5 H22180VVluauDvL0IiqYAUsrK0jYo7MhKA1EbKlfLPt8NDcR8H3SA5fpppeD31UhEiaH 8CI8idPxGXAlFVaynyMM8PCfH70FSlTKNTnl+JXiVBFInlCZyfNGqm9ffHh0ezWLm7DP 8c+YV5p31lM1XVX8VNyPr9WC/fdSXqnWDTblzmP9NK3EF/tHQSH9iw+n3e8UruT4Utu2 Gl7ACgpVLeZMP2/PhWaYpCzNLtdjuTbmfIr9WRBAlm+br07JfsdPHCmqURGh6m5kU7qz wlLQ== X-Gm-Message-State: ACgBeo3Wau95s1oR3NdlHcrpKew4z1cYdFmfAFzh8MiGofDUsN9ByBTi obXYapQu3q4SLa3e7pEdLdhDwUZObic= X-Google-Smtp-Source: AA6agR7lLfrdwEE4dV+Y+eT+VQNfMNNQEVgjlNs9TLtOYsCrni2DgDZm6c6d3s8LFU+X79+SVZtWOQ== X-Received: by 2002:a05:600c:198f:b0:3a6:2482:b2be with SMTP id t15-20020a05600c198f00b003a62482b2bemr5882102wmq.110.1662735991000; Fri, 09 Sep 2022 08:06:31 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id c7-20020a05600c0ac700b003a331c6bffdsm908630wmr.47.2022.09.09.08.06.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 09 Sep 2022 08:06:30 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Fri, 09 Sep 2022 15:06:23 +0000 Subject: [PATCH v5 3/5] hiderefs: hornor hide flags in wire protocol V2 Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Sun Chao <16657101987@163.com>, Sun Chao Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Sun Chao From: Sun Chao Previously hide refs can not protect private data of hide refs in wire protocol V2, for example a `ALL_FLAGS` will be used to clear all the objects before handling the fetch requests. Hornor the hide flags by removing `HIDDEN_REFS` flag from `ALL_FLAGS` and make sure all the refs will check its hidden flags before sending pack to client, especially during stateless rpc. And if there are refs with `HIDDEN_REF_FORCE` flag, use `check_non_tip` to protect the private date of force hidden refs. Signed-off-by: Sun Chao --- ls-refs.c | 2 +- refs.c | 20 ++++++++++++++++++++ refs.h | 4 ++++ upload-pack.c | 11 +++++++++-- 4 files changed, 34 insertions(+), 3 deletions(-) diff --git a/ls-refs.c b/ls-refs.c index 98e69373c84..b5cb1316d38 100644 --- a/ls-refs.c +++ b/ls-refs.c @@ -84,7 +84,7 @@ static int send_ref(const char *refname, const struct object_id *oid, strbuf_reset(&data->buf); - if (ref_is_hidden(refname_nons, refname)) + if (mark_our_ref(refname_nons, refname, oid)) return 0; if (!ref_match(&data->prefixes, refname_nons)) diff --git a/refs.c b/refs.c index 3fbdf967bc6..1424de0048e 100644 --- a/refs.c +++ b/refs.c @@ -1662,6 +1662,25 @@ int ref_is_hidden(const char *refname, const char *refname_full) */ #define HIDDEN_REF_FORCE (1u << 20) +/* Use this variable to record existing object hidden flags */ +static unsigned int objects_hidden_flags; + +/* Return non-zero if need to batch check hidden refs, otherwise 0 */ +int need_check_hidden_refs(void) +{ + if (!objects_hidden_flags) + return 1; + return 0; +} + +/* Return non-zero if some ref is force hidden, otherwise 0 */ +int has_force_hidden_refs(void) +{ + if (objects_hidden_flags & HIDDEN_REF_FORCE) + return 1; + return 0; +} + static unsigned int ref_hidden_flag(const char *refname, const char *refname_full) { if (ref_hidden_check(refname, refname_full, 1)) @@ -1684,6 +1703,7 @@ int mark_our_ref(const char *refname, const char *refname_full, o = lookup_unknown_object(the_repository, oid); flag = ref_hidden_flag(refname, refname_full); o->flags |= flag; + objects_hidden_flags |= flag; if (flag & OUR_REF) return 0; diff --git a/refs.h b/refs.h index 2feabfe35c4..8deb36a95cc 100644 --- a/refs.h +++ b/refs.h @@ -822,6 +822,10 @@ int ref_is_hidden(const char *, const char *); /* return non-zero if the ref is hidden, otherwise 0 */ int mark_our_ref(const char *refname, const char *refname_full, const struct object_id *oid); +/* return non-zero if need to batch check hidden refs, otherwise 0 */ +int need_check_hidden_refs(void); +/* return non-zero if some ref is force hidden, otherwise 0 */ +int has_force_hidden_refs(void); enum ref_type { REF_TYPE_PER_WORKTREE, /* refs inside refs/ but not shared */ diff --git a/upload-pack.c b/upload-pack.c index a8ca5d1c26e..a9a24399d8e 100644 --- a/upload-pack.c +++ b/upload-pack.c @@ -39,8 +39,8 @@ #define CLIENT_SHALLOW (1u << 18) #define HIDDEN_REF (1u << 19) -#define ALL_FLAGS (THEY_HAVE | OUR_REF | WANTED | COMMON_KNOWN | SHALLOW | \ - NOT_SHALLOW | CLIENT_SHALLOW | HIDDEN_REF) +#define ALL_FLAGS (THEY_HAVE |WANTED | COMMON_KNOWN | SHALLOW | \ + NOT_SHALLOW | CLIENT_SHALLOW) /* Enum for allowed unadvertised object request (UOR) */ enum allow_uor { @@ -1726,6 +1726,13 @@ int upload_pack_v2(struct repository *r, struct packet_reader *request) state = FETCH_DONE; break; case FETCH_SEND_PACK: + if (need_check_hidden_refs()) { + head_ref_namespaced(check_ref, NULL); + for_each_namespaced_ref(check_ref, NULL); + } + if (has_force_hidden_refs()) + check_non_tip(&data); + send_wanted_ref_info(&data); send_shallow_info(&data); From patchwork Fri Sep 9 15:06:24 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sun Chao X-Patchwork-Id: 12971897 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B8A66ECAAA1 for ; Fri, 9 Sep 2022 15:07:02 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232005AbiIIPHB (ORCPT ); Fri, 9 Sep 2022 11:07:01 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39624 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231912AbiIIPGh (ORCPT ); Fri, 9 Sep 2022 11:06:37 -0400 Received: from mail-wr1-x430.google.com (mail-wr1-x430.google.com [IPv6:2a00:1450:4864:20::430]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 71BA7139AC1 for ; Fri, 9 Sep 2022 08:06:34 -0700 (PDT) Received: by mail-wr1-x430.google.com with SMTP id e20so3215658wri.13 for ; Fri, 09 Sep 2022 08:06:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:from:to:cc:subject:date; bh=lcOSRYa78Rrx4p3Uz2exv6+8IceCq5u0zo3bx6EaMI8=; b=lht4Xs86BJe1EBxeYpkrDby7lFA8L55vZ5tiEQOgJPmF5b/qDu+Xb5DM7phO/r50k+ U04wekDsvtuB55bwSz0B7kmGyoO3KCrDsfAeQf43DUiBRH4lFIrB6w+2ImqAAoA4oakq vhS8gqpSIuR8S0q0wcQOGvTQZvzqUTCzb6gcOMWZXMH2r7aftlNWKBqxAhoqLpAIq7Se i7LJhmEtvRI6+RUA34afneIWn+MFeE2azDA9GmoWyy0jOzZiObfVyzSU870IH91syVi1 2FJWutz0Sy73AcBZ4N7AmEnpw8s+pMWxDCUf6YKJRhsnZb6VfQBhTPvcQ99pG+ro2/nZ Jxsw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:x-gm-message-state:from:to:cc :subject:date; bh=lcOSRYa78Rrx4p3Uz2exv6+8IceCq5u0zo3bx6EaMI8=; b=RFB1TqkwejuTev8Y0XbUGyA9Un27uMPUFxUZ7CuN9bZ5HX89eXIlsyaIaz6kSrOupQ +asxyfV3uI0kfs4ghX3Q0fiDosxkDkK5g8v9061HGSVI9z7Okl5HC5oy5943g8t8mTGk v89XxJOBZkTqbI93ETmjsp8q9tCH+FYALtkdoOVSKnQX8/zMTDLAXjgOnrxwxtcJNtAb S6fg+1DF3OAqNR/nGv849hqwe8pxHO0M3UcTPNaenmQopGTbd/kc8pA0hgsqzup7zOD8 UgraQfMIGam3MEQq5mQebVE6rEV67j5S5OO6i7TModipgLUk8exrWxtMk6WLLk8Mymcg hkIg== X-Gm-Message-State: ACgBeo3tslWqkxpiQNFCDhIuktNqHS/xPTkGbnsJuj5tb4fgNEtitlVr ebKsfrfnhniUgt4eY9iLQli0b3rYdQA= X-Google-Smtp-Source: AA6agR5I3DyXvRbjuqGxiQxyrafnlVsnm5LD8YS4BdYJyKZh8OlXG5W83B7FcMTY+komJSpJC8ZCvg== X-Received: by 2002:a5d:6050:0:b0:228:6128:b0fb with SMTP id j16-20020a5d6050000000b002286128b0fbmr7926027wrt.424.1662735992291; Fri, 09 Sep 2022 08:06:32 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id k13-20020adfe3cd000000b00228655a5c8fsm695861wrm.28.2022.09.09.08.06.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 09 Sep 2022 08:06:31 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Fri, 09 Sep 2022 15:06:24 +0000 Subject: [PATCH v5 4/5] test: add test cases for hide-refs hook Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Sun Chao <16657101987@163.com>, Sun Chao Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Sun Chao From: Sun Chao Add test cases for the new 'hide-refs' hook which is used to filter the references during reference discovery phase. Signed-off-by: Sun Chao --- Makefile | 1 + t/helper/test-hide-refs.c | 107 +++++++++++++++++ t/helper/test-tool.c | 1 + t/helper/test-tool.h | 1 + t/t1419-hide-refs-hook.sh | 112 ++++++++++++++++++ t/t1419/abnormal-hide-refs-hook.sh | 82 +++++++++++++ t/t1419/common-functions.sh | 74 ++++++++++++ ...test-0001-ls-remote-with-hide-refs-hook.sh | 45 +++++++ ...st-0002-upload-pack-with-hide-refs-hook.sh | 47 ++++++++ ...t-0003-receive-pack-with-hide-refs-hook.sh | 42 +++++++ 10 files changed, 512 insertions(+) create mode 100644 t/helper/test-hide-refs.c create mode 100755 t/t1419-hide-refs-hook.sh create mode 100644 t/t1419/abnormal-hide-refs-hook.sh create mode 100644 t/t1419/common-functions.sh create mode 100644 t/t1419/test-0001-ls-remote-with-hide-refs-hook.sh create mode 100644 t/t1419/test-0002-upload-pack-with-hide-refs-hook.sh create mode 100644 t/t1419/test-0003-receive-pack-with-hide-refs-hook.sh diff --git a/Makefile b/Makefile index 924b864ae83..c6793681b41 100644 --- a/Makefile +++ b/Makefile @@ -794,6 +794,7 @@ TEST_BUILTINS_OBJS += test-wildmatch.o TEST_BUILTINS_OBJS += test-windows-named-pipe.o TEST_BUILTINS_OBJS += test-write-cache.o TEST_BUILTINS_OBJS += test-xml-encode.o +TEST_BUILTINS_OBJS += test-hide-refs.o # Do not add more tests here unless they have extra dependencies. Add # them in TEST_BUILTINS_OBJS above. diff --git a/t/helper/test-hide-refs.c b/t/helper/test-hide-refs.c new file mode 100644 index 00000000000..751fc6213f3 --- /dev/null +++ b/t/helper/test-hide-refs.c @@ -0,0 +1,107 @@ +#include "cache.h" +#include "hash.h" +#include "config.h" +#include "connect.h" +#include "parse-options.h" +#include "pkt-line.h" +#include "sigchain.h" +#include "test-tool.h" + +static const char *hide_refs_usage[] = { + "test-tool hide-refs [...]", + NULL +}; + +static int die_before_read_ref; +static int die_after_proc_ref; +static int version = 1; +static int hash_size = GIT_SHA1_HEXSZ; +static struct string_list hidelist = STRING_LIST_INIT_NODUP; + +static void hide_refs_verison(struct packet_reader *reader) { + int server_version = 0; + + for (;;) { + if (packet_reader_read(reader) != PACKET_READ_NORMAL) + break; + + if (reader->pktlen > 8 && starts_with(reader->line, "version=")) { + server_version = atoi(reader->line+8); + if (server_version != 1) + die("bad protocol version: %d", server_version); + } + } + + packet_write_fmt(1, "version=%d\n", version); + packet_flush(1); +} + +static void hide_refs_proc(struct packet_reader *reader) +{ + const char *p; + struct strbuf buf = STRBUF_INIT; + enum packet_read_status status; + + if (die_before_read_ref) + die("die with the --die-before-read-ref option"); + + for (;;) { + status = packet_reader_read(reader); + if (status == PACKET_READ_EOF) + exit(0); + + if (status != PACKET_READ_NORMAL) + break; + + p = reader->line; + strbuf_reset(&buf); + strbuf_addstr(&buf, reader->line); + } + + p = strchr(buf.buf, ':'); + if (unsorted_string_list_has_string(&hidelist, p + 1)) { + packet_write_fmt(1, "hide"); + } + + if (die_after_proc_ref) + die("die with the --die-after-proc-refs option"); + + packet_flush(1); +} + +int cmd__hide_refs(int argc, const char **argv) { + int nongit_ok = 0; + struct packet_reader reader; + const char *value = NULL; + struct option options[] = { + OPT_BOOL(0, "die-before-read-ref", &die_before_read_ref, + "die when reading first reference"), + OPT_BOOL(0, "die-after-proc-refs", &die_after_proc_ref, + "die after proc ref"), + OPT_STRING_LIST('H', "hide", &hidelist, "refs-to-force-hidden", + "refs that will be force hidden"), + OPT_INTEGER('V', "version", &version, + "use this protocol version number"), + OPT_END() + }; + + setup_git_directory_gently(&nongit_ok); + + argc = parse_options(argc, argv, "test-tools", options, hide_refs_usage, 0); + if (argc > 0) + usage_msg_opt("Too many arguments.", hide_refs_usage, options); + + packet_reader_init(&reader, 0, NULL, 0, PACKET_READ_CHOMP_NEWLINE | PACKET_READ_GENTLE_ON_EOF); + + if (!git_config_get_value("extensions.objectformat", &value)) { + if (!strcmp(value, "sha256")) + hash_size = GIT_SHA256_HEXSZ; + } + + hide_refs_verison(&reader); + for (;;) { + hide_refs_proc(&reader); + } + + return 0; +} diff --git a/t/helper/test-tool.c b/t/helper/test-tool.c index 80055886798..c5bd7f1f806 100644 --- a/t/helper/test-tool.c +++ b/t/helper/test-tool.c @@ -72,6 +72,7 @@ static struct test_cmd cmds[] = { { "regex", cmd__regex }, { "repository", cmd__repository }, { "revision-walking", cmd__revision_walking }, + { "hide-refs", cmd__hide_refs }, { "run-command", cmd__run_command }, { "scrap-cache-tree", cmd__scrap_cache_tree }, { "serve-v2", cmd__serve_v2 }, diff --git a/t/helper/test-tool.h b/t/helper/test-tool.h index a432cc77d92..cba8d3b093d 100644 --- a/t/helper/test-tool.h +++ b/t/helper/test-tool.h @@ -61,6 +61,7 @@ int cmd__reftable(int argc, const char **argv); int cmd__regex(int argc, const char **argv); int cmd__repository(int argc, const char **argv); int cmd__revision_walking(int argc, const char **argv); +int cmd__hide_refs(int argc, const char **argv); int cmd__run_command(int argc, const char **argv); int cmd__scrap_cache_tree(int argc, const char **argv); int cmd__serve_v2(int argc, const char **argv); diff --git a/t/t1419-hide-refs-hook.sh b/t/t1419-hide-refs-hook.sh new file mode 100755 index 00000000000..3b505f0fa9d --- /dev/null +++ b/t/t1419-hide-refs-hook.sh @@ -0,0 +1,112 @@ +#!/bin/sh +# +# Copyright (c) 2022 Sun Chao +# + +test_description='Test hide-refs hook' + +. ./test-lib.sh +. "$TEST_DIRECTORY"/t1419/common-functions.sh + +GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main +export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME + +setup_test_repos () { + test_expect_success "setup bare_repo and work_repo" ' + rm -rf bare_repo.git && + rm -rf work_repo && + git init --bare bare_repo.git && + git init work_repo && + + # create new commits and references + create_commits_in work_repo A B C D && + ( + cd work_repo && + git config --local core.abbrev 7 && + git update-ref refs/heads/main $A && + git update-ref refs/heads/dev $B && + git update-ref refs/pull-requests/1/head $C && + git tag -m "v123" v123 $D && + git push ../bare_repo.git +refs/heads/*:refs/heads/* && + git push ../bare_repo.git +refs/tags/*:refs/tags/* && + git push ../bare_repo.git +refs/pull-requests/*:refs/pull-requests/* + ) && + TAG=$(git -C work_repo rev-parse v123) && + + # config transfer.hiderefs values with "hook:" prefix + ( + git -C bare_repo.git config --local http.receivepack true && + git -C bare_repo.git config --add transfer.hiderefs hook:HEAD && + git -C bare_repo.git config --add transfer.hiderefs hook:refs + ) + ' +} + +setup_httpd() { + ROOT_PATH="$PWD" + . "$TEST_DIRECTORY"/lib-gpg.sh + . "$TEST_DIRECTORY"/lib-httpd.sh + . "$TEST_DIRECTORY"/lib-terminal.sh + + start_httpd + set_askpass user@host pass@host + setup_askpass_helper +} + +# Run test cases when hide-refs hook exit abnormally +run_tests_for_abnormal_hook() { + GIT_TEST_PROTOCOL_VERSION=$1 + BAREREPO_GIT_DIR="$(pwd)/bare_repo.git" + + for t in "$TEST_DIRECTORY"/t1419/abnormal-*.sh + do + setup_test_repos + + . "$t" + done +} + +# Run test cases under local/HTTP protocol +run_tests_for_normal_hook() { + for t in "$TEST_DIRECTORY"/t1419/test-*.sh + do + setup_test_repos + case $1 in + http) + PROTOCOL="HTTP protocol" + + # bare_repo.git need move to httpd sever root path + BAREREPO_GIT_DIR="$HTTPD_DOCUMENT_ROOT_PATH/bare_repo.git" + rm -rf "$BAREREPO_GIT_DIR" + mv bare_repo.git "$BAREREPO_GIT_DIR" + + # setup the repository service URL address of http protocol + BAREREPO_PREFIX="$HTTPD_URL"/smart + BAREREPO_URL="$BAREREPO_PREFIX/bare_repo.git" + ;; + local) + PROTOCOL="builtin protocol" + BAREREPO_GIT_DIR="$(pwd)/bare_repo.git" + + # setup the repository service address of builtin protocol + BAREREPO_PREFIX="$(pwd)" + BAREREPO_URL="$BAREREPO_PREFIX/bare_repo.git" + ;; + esac + + GIT_TEST_PROTOCOL_VERSION=$2 + git -C work_repo remote add origin "$BAREREPO_URL" + + . "$t" + done +} + +setup_httpd +for protocol in 1 2 +do + run_tests_for_abnormal_hook $protocol + run_tests_for_normal_hook local $protocol + run_tests_for_normal_hook http $protocol +done + +test_done diff --git a/t/t1419/abnormal-hide-refs-hook.sh b/t/t1419/abnormal-hide-refs-hook.sh new file mode 100644 index 00000000000..c615a74f3a0 --- /dev/null +++ b/t/t1419/abnormal-hide-refs-hook.sh @@ -0,0 +1,82 @@ +#!/bin/sh + +# Upstream repository (bare_repo.git) contains the configurations: +# +# [transfer] +# hiderefs = hook:HEAD +# hiderefs = hook:refs +# +# During refs advertise phase the hide-refs hook will be invoked and each ref will be checked by it, +# we should make sure Git works correctly in some speicail cases + +# If hide-refs not exists, Git should not invoke it and continue advertise all the refs +test_expect_success "protocol $GIT_TEST_PROTOCOL_VERSION: advertise-refs while hide-refs hook not exists" ' + rm -f "$BAREREPO_GIT_DIR/hooks/hide-refs" && + git -c protocol.version=$GIT_TEST_PROTOCOL_VERSION upload-pack --advertise-refs "$BAREREPO_GIT_DIR" >out 2>&1 && + cat out | make_user_friendly_and_stable_output >actual && + format_and_save_expect <<-EOF && + HEAD + refs/heads/dev + refs/heads/main + refs/pull-requests/1/head + refs/tags/v123 + refs/tags/v123^{} + EOF + test_cmp expect actual +' + +# If hide-refs hook run with incompatible version, Git should not invoke it and continue to advertise all the refs +test_expect_success "protocol $GIT_TEST_PROTOCOL_VERSION: advertise-refs while hide-refs hook run with incompatible version" ' + write_script "$BAREREPO_GIT_DIR/hooks/hide-refs" <<-EOF && + test-tool hide-refs --version=2 + EOF + git -c protocol.version=$GIT_TEST_PROTOCOL_VERSION upload-pack --advertise-refs "$BAREREPO_GIT_DIR" >out 2>&1 && + cat out | make_user_friendly_and_stable_output >actual && + format_and_save_expect <<-EOF && + HEAD + refs/heads/dev + refs/heads/main + refs/pull-requests/1/head + refs/tags/v123 + refs/tags/v123^{} + EOF + test_cmp expect actual +' + +# If hide-refs hook exit before processing any refs, Git should not die and continue to advertise all the refs +test_expect_success "protocol $GIT_TEST_PROTOCOL_VERSION: advertise-refs while hide-refs hook die before read ref" ' + write_script "$BAREREPO_GIT_DIR/hooks/hide-refs" <<-EOF && + test-tool hide-refs --die-before-read-ref + EOF + git -c protocol.version=$GIT_TEST_PROTOCOL_VERSION upload-pack --advertise-refs "$BAREREPO_GIT_DIR" >out 2>&1 && + cat out | make_user_friendly_and_stable_output | grep -v "^error:" >actual && + format_and_save_expect <<-EOF && + fatal: die with the --die-before-read-ref option + HEAD + refs/heads/dev + refs/heads/main + refs/pull-requests/1/head + refs/tags/v123 + refs/tags/v123^{} + EOF + test_cmp expect actual +' + +# If hide-refs hook exit abnormally, Git should not die and continue to advertise left refs +test_expect_success "protocol $GIT_TEST_PROTOCOL_VERSION: advertise-refs while hide-refs hook die after proc ref" ' + write_script "$BAREREPO_GIT_DIR/hooks/hide-refs" <<-EOF && + test-tool hide-refs --die-after-proc-refs + EOF + git -c protocol.version=$GIT_TEST_PROTOCOL_VERSION upload-pack --advertise-refs "$BAREREPO_GIT_DIR" >out 2>&1 && + cat out | make_user_friendly_and_stable_output | grep -v "^error:" >actual && + format_and_save_expect <<-EOF && + fatal: die with the --die-after-proc-refs option + HEAD + refs/heads/dev + refs/heads/main + refs/pull-requests/1/head + refs/tags/v123 + refs/tags/v123^{} + EOF + test_cmp expect actual +' diff --git a/t/t1419/common-functions.sh b/t/t1419/common-functions.sh new file mode 100644 index 00000000000..7841241b038 --- /dev/null +++ b/t/t1419/common-functions.sh @@ -0,0 +1,74 @@ +# Create commits in and assign each commit's oid to shell variables +# given in the arguments (A, B, and C). E.g.: +# +# create_commits_in A B C +# +# NOTE: Never calling this function from a subshell since variable +# assignments will disappear when subshell exits. +create_commits_in () { + repo="$1" && test -d "$repo" || + error "Repository $repo does not exist." + shift && + while test $# -gt 0 + do + name=$1 && + shift && + test_commit -C "$repo" --no-tag "$name" && + eval $name=$(git -C "$repo" rev-parse HEAD) + done +} + +get_abbrev_oid () { + oid=$1 && + suffix=${oid#???????} && + oid=${oid%$suffix} && + if test -n "$oid" + then + echo "$oid" + else + echo "undefined-oid" + fi +} + +# Format the output of git-fetch, git-ls-remote and other commands to make a +# user-friendly and stable text. We can easily prepare the expect text +# without having to worry about changes of the commit ID (full or abbrev.) +# of the output. Single quotes are replaced with double quotes, because +# it is boring to prepare unquoted single quotes in expect text. +make_user_friendly_and_stable_output () { + tr '\0' '@' | sed \ + -e "s/'/\"/g" \ + -e "s/@.*//g" \ + -e "s/$(get_abbrev_oid $A)[0-9a-f]*//g" \ + -e "s/$(get_abbrev_oid $B)[0-9a-f]*//g" \ + -e "s/$(get_abbrev_oid $C)[0-9a-f]*//g" \ + -e "s/$(get_abbrev_oid $D)[0-9a-f]*//g" \ + -e "s/$(get_abbrev_oid $TAG)[0-9a-f]*//g" \ + -e "s/$ZERO_OID//g" \ + -e "s#$BAREREPO_PREFIX/bare_repo.git##" \ + -e 's/^[0-9a-f]\{4\}//g' + +} + +filter_out_hide_refs_output() { + make_user_friendly_and_stable_output | sed 's/^[0-9a-f]\{4\}//g' +} + +format_and_save_expect () { + sed -e 's/^> //' -e 's/Z$//' >expect +} + +test_cmp_refs () { + indir= + if test "$1" = "-C" + then + shift + indir="$1" + shift + fi + indir=${indir:+"$indir"/} + cat >show-ref.expect && + git ${indir:+ -C "$indir"} show-ref >show-ref.pristine && + make_user_friendly_and_stable_output show-ref.filtered && + test_cmp show-ref.expect show-ref.filtered +} diff --git a/t/t1419/test-0001-ls-remote-with-hide-refs-hook.sh b/t/t1419/test-0001-ls-remote-with-hide-refs-hook.sh new file mode 100644 index 00000000000..9803994b6a8 --- /dev/null +++ b/t/t1419/test-0001-ls-remote-with-hide-refs-hook.sh @@ -0,0 +1,45 @@ +#!/bin/sh + +# Upstream repository (bare_repo.git) contains the configurations: +# +# [transfer] +# hiderefs = hook:HEAD +# hiderefs = hook:refs +# +# During refs advertise phase the hide-refs hook will be invoked and each ref will be checked by it + +# Git will not advertise the refs that are hided by hide-refs hook +test_expect_success "$PROTOCOL (protocol: $GIT_TEST_PROTOCOL_VERSION): ls-remote while hide-refs hook hide part of refs" ' + write_script "$BAREREPO_GIT_DIR/hooks/hide-refs" <<-EOF && + test-tool hide-refs \ + -H "refs/pull-requests/1/head" \ + -H "refs/tags/v123" + EOF + git -c protocol.version=$GIT_TEST_PROTOCOL_VERSION ls-remote "$BAREREPO_URL" >out 2>&1 && + make_user_friendly_and_stable_output actual && + format_and_save_expect <<-EOF && + HEAD + refs/heads/dev + refs/heads/main + EOF + test_cmp expect actual +' + +# The hide-ref hook should not change the default effects of [transfer|uploadpack|receive].hiderefs configurations, +# if it hide no refs, the original hiderefs rules should works +test_expect_success "$PROTOCOL (protocol: $GIT_TEST_PROTOCOL_VERSION): ls-remote while hide-refs hook hide no refs" ' + write_script "$BAREREPO_GIT_DIR/hooks/hide-refs" <<-EOF && + test-tool hide-refs + EOF + git -C "$BAREREPO_GIT_DIR" config --add transfer.hiderefs refs/heads/dev && + git -c protocol.version=$GIT_TEST_PROTOCOL_VERSION ls-remote "$BAREREPO_URL" >out 2>&1 && + make_user_friendly_and_stable_output actual && + format_and_save_expect <<-EOF && + HEAD + refs/heads/main + refs/pull-requests/1/head + refs/tags/v123 + refs/tags/v123^{} + EOF + test_cmp expect actual +' diff --git a/t/t1419/test-0002-upload-pack-with-hide-refs-hook.sh b/t/t1419/test-0002-upload-pack-with-hide-refs-hook.sh new file mode 100644 index 00000000000..d1174266fd3 --- /dev/null +++ b/t/t1419/test-0002-upload-pack-with-hide-refs-hook.sh @@ -0,0 +1,47 @@ +#!/bin/sh + +# Upstream repository (bare_repo.git) contains the configurations: +# +# [transfer] +# hiderefs = hook:HEAD +# hiderefs = hook:refs +# +# During refs advertise phase the hide-refs hook will be invoked and each ref will be checked by it + +# Git client can not fetch the refs that are hided by hide-refs hook +test_expect_success "$PROTOCOL (protocol: $GIT_TEST_PROTOCOL_VERSION): mirror clone while hide-refs hide part of refs" ' + rm -rf local.git && + write_script "$BAREREPO_GIT_DIR/hooks/hide-refs" <<-EOF && + test-tool hide-refs \ + -H "HEAD" \ + -H "refs/heads/dev" \ + -H "refs/heads/main" + EOF + git -c protocol.version=$GIT_TEST_PROTOCOL_VERSION clone --mirror "$BAREREPO_URL" local.git && + git -C local.git show-ref -d >out 2>&1 && + make_user_friendly_and_stable_output actual && + format_and_save_expect <<-EOF && + refs/pull-requests/1/head + refs/tags/v123 + refs/tags/v123^{} + EOF + test_cmp expect actual +' + +# If a ref is hided by hide-refs hook, its private commits (tip or non-tip) will be force hidden +# to the client, and the client can not fetch such kind of commit even the server set allowTipSHA1InWant +# or allowReachableSHA1InWant to true +test_expect_success "$PROTOCOL (protocol: $GIT_TEST_PROTOCOL_VERSION): fetch a commit which is hided by hide-refs hook" ' + rm -rf local.git && + write_script "$BAREREPO_GIT_DIR/hooks/hide-refs" <<-EOF && + test-tool hide-refs \ + -H "refs/heads/dev" \ + -H "refs/pull-requests/1/head" \ + -H "refs/tags/v123" + EOF + git -C "$BAREREPO_GIT_DIR" config uploadpack.allowTipSHA1InWant true && + git -C "$BAREREPO_GIT_DIR" config uploadpack.allowReachableSHA1InWant true && + git init local.git && + git -C local.git remote add origin "$BAREREPO_URL" && + test_must_fail git -C local.git -c protocol.version=$GIT_TEST_PROTOCOL_VERSION fetch "$BAREREPO_URL" $B +' diff --git a/t/t1419/test-0003-receive-pack-with-hide-refs-hook.sh b/t/t1419/test-0003-receive-pack-with-hide-refs-hook.sh new file mode 100644 index 00000000000..0fe3c2e1389 --- /dev/null +++ b/t/t1419/test-0003-receive-pack-with-hide-refs-hook.sh @@ -0,0 +1,42 @@ +#!/bin/sh + +# Upstream repository (bare_repo.git) contains the configurations: +# +# [transfer] +# hiderefs = hook:HEAD +# hiderefs = hook:refs +# +# During refs advertise phase the hide-refs hook will be invoked and each ref will be checked by it + +test_expect_success "$PROTOCOL (protocol: $GIT_TEST_PROTOCOL_VERSION): push to main while hide-refs hook does not hide it" ' + write_script "$BAREREPO_GIT_DIR/hooks/hide-refs" <<-EOF && + test-tool hide-refs + EOF + create_commits_in work_repo E && + git -c protocol.version=$GIT_TEST_PROTOCOL_VERSION -C work_repo push origin HEAD:main >out 2>&1 && + make_user_friendly_and_stable_output out.tmp && + sed "s/$(get_abbrev_oid $E)[0-9a-f]*//g" actual && + format_and_save_expect <<-EOF && + To + .. HEAD -> main + EOF + test_cmp expect actual +' + +# If hide-refs hook hide some ref, git push will be rejected +test_expect_success "$PROTOCOL (protocol: $GIT_TEST_PROTOCOL_VERSION): push to main while hide-refs hook hide it" ' + write_script "$BAREREPO_GIT_DIR/hooks/hide-refs" <<-EOF && + test-tool hide-refs \ + -H "refs/heads/main" + EOF + create_commits_in work_repo F && + test_must_fail git -c protocol.version=$GIT_TEST_PROTOCOL_VERSION -C work_repo push origin HEAD:main >out 2>&1 && + make_user_friendly_and_stable_output out.tmp && + sed "s/$(get_abbrev_oid $E)[0-9a-f]*//g" actual && + format_and_save_expect <<-EOF && + To + ! [remote rejected] HEAD -> main (deny updating a hidden ref) + error: failed to push some refs to "" + EOF + test_cmp expect actual +' From patchwork Fri Sep 9 15:06:25 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sun Chao X-Patchwork-Id: 12971898 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 827BCECAAD3 for ; Fri, 9 Sep 2022 15:07:04 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232227AbiIIPHD (ORCPT ); Fri, 9 Sep 2022 11:07:03 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39700 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231931AbiIIPGi (ORCPT ); Fri, 9 Sep 2022 11:06:38 -0400 Received: from mail-wr1-x433.google.com (mail-wr1-x433.google.com [IPv6:2a00:1450:4864:20::433]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6A5A913BC6B for ; Fri, 9 Sep 2022 08:06:35 -0700 (PDT) Received: by mail-wr1-x433.google.com with SMTP id t7so3222401wrm.10 for ; Fri, 09 Sep 2022 08:06:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:from:to:cc:subject:date; bh=iutZqULIGZxPg0oNpzRUy6xgLT5AIrsk9K+6CPN49AU=; b=c61+qHkfWor4awthp2xKQcQ5ZXzQ0JheuJDDCdMu6Ihq0AE51ApvzMtT/VoxTPyJ3F uIWwnPvmAlqoODpCHxfILvoRnhLFbTujRRoxrtNK2hmN9LSTQV+kiAzfcsmZ8Vbp72Jm u4TI+8FJIR3j9wFK1SQcJgS8fqvKMb0Ju/JWWSR5GM0R/Ke9G1bHlgS1YYCSkCgfWuBo PCfYSw9e9P8uKP18CgHTUfyT655vmc0L3gcD2ldtaWRBA0qS8Ac59QVc9gtei2+6TX9+ nWI2gjlkj1y89uL8EOPwwbhUq6lfYwR1y2sCaKNDdUkmBjv8G30au64tIJmvSU6q9lmp tXVQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:x-gm-message-state:from:to:cc :subject:date; bh=iutZqULIGZxPg0oNpzRUy6xgLT5AIrsk9K+6CPN49AU=; b=zkADBOcidAa75zkeP8LMeCAdSA9SEW0ChBDmzdyRIhDp/7h91WAa0+3dyZnH+nzMbK ONK9+b1WzDnLB7kIdxIjEyJCy3yHWZaGILt+Z+HETQ3lh9OlYgjPRNktBWFcLPetscH1 74Q9IiQZbqEhW+T+74S7b0pv6wDoXRpoWMoNZlhghjcykSYW+GSKG9KR4QUJywazKryL 9Jk5wuUS8bdLzFBicIBJUajnRVURGL5+p4Nl4UV/Mbt0Yo8JR9s51VlLDv0Og6FdGAIo VA3WRmTo0NRna78eYMg5b826902GyDxAYWxWFfVUeXErx+HUPpKz7+4X8jN9Fo60azmh Nr2g== X-Gm-Message-State: ACgBeo1n0ELbqd1zmWKNoFNvT/dPZZMkmZlx26xmP0DExlqGXy6t8fUz ewGovKGA+Q8CaVPiwvBz9JK9wSIs8Tw= X-Google-Smtp-Source: AA6agR4lI+F4IfHxz1boGKDvYGO4zBuyAEBnkKmIF0i7mr2451iVpLIuzn1kZ5sEV8HpNF5cjwTQQg== X-Received: by 2002:a05:6000:168c:b0:226:f4c2:d6db with SMTP id y12-20020a056000168c00b00226f4c2d6dbmr8148449wrd.659.1662735993666; Fri, 09 Sep 2022 08:06:33 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id f13-20020adfc98d000000b00228d7078c4esm932602wrh.4.2022.09.09.08.06.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 09 Sep 2022 08:06:33 -0700 (PDT) Message-Id: <8a5f7762c2783fb133514f5ee2cda6c541be312b.1662735985.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Fri, 09 Sep 2022 15:06:25 +0000 Subject: [PATCH v5 5/5] doc: add documentation for the hide-refs hook Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Sun Chao <16657101987@163.com>, Sun Chao Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Sun Chao From: Sun Chao If uploadpack.allowTipSHA1InWant or uploadpack.allowReachableSHA1InWant are set to true, the private commits of hide refs can be fetched by client. The new "hide-refs" hook are used to hide our refs and we wish to hide the private commits either. "git upload-pack" or "git receive-pack" can use "hide-refs" hook to filter the references during reference discovery phase. If a ref is hided by "hide-refs" hook, its private data cannot be fetched by client even if uploadpack.allowTipSHA1InWant or uploadpack.allowReachableSHA1InWant are set to true. Signed-off-by: Sun Chao --- Documentation/githooks.txt | 48 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 48 insertions(+) diff --git a/Documentation/githooks.txt b/Documentation/githooks.txt index a16e62bc8c8..314bddedc1f 100644 --- a/Documentation/githooks.txt +++ b/Documentation/githooks.txt @@ -249,6 +249,54 @@ If this hook exits with a non-zero status, `git push` will abort without pushing anything. Information about why the push is rejected may be sent to the user by writing to standard error. +[[hide-refs]] +hide-refs +~~~~~~~~~ + +This hook would be invoked by 'git-receive-pack' and 'git-upload-pack' +during the reference discovery phase, each reference and will be filtered +by this hook. The hook executes once with no arguments for each +'git-upload-pack' and 'git-receive-pack' process. Once the hook is invoked, +a version number and server process name ('uploadpack' or 'receive') will +send to it in pkt-line format, followed by a flush-pkt. The hook should +respond with its version number. + +During reference discovery phase, each reference will be filtered by this +hook. In the following example, the letter 'G' stands for 'git-receive-pack' +or 'git-upload-pack' and the letter 'H' stands for this hook. The hook +decides if the reference will be hidden or not, it sends result back in +pkt-line format protocol, a response "hide" the references will hidden +to the client. + + # Version negotiation + G: PKT-LINE(version=1\0uploadpack) + G: flush-pkt + H: PKT-LINE(version=1) + H: flush-pkt + + # Send reference filter request to hook + G: PKT-LINE(ref :) + G: flush-pkt + + # Receive result from the hook. + # Case 1: this reference is hidden + H: PKT-LINE(hide) + H: flush-pkt + + # Case 2: this reference can be advertised + H: flush-pkt + +To enable the `hide-refs` hook, we should config hiderefs with `hook:` +option, eg: + + git config --add transfer.hiderefs hook:refs/prefix1/ + git config --add uploadpack.hiderefs hook:!refs/prefix2/ + +the `hide-refs` will be called during reference discovery phase and +check each matched reference, a 'hide' response means the reference will +be hidden for its private data even if `allowTipSHA1InWant` and +`allowReachableSHA1InWant` are set to true. + [[pre-receive]] pre-receive ~~~~~~~~~~~