From patchwork Tue Sep 20 08:22:43 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sun Chao X-Patchwork-Id: 12981615 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 99B79ECAAD8 for ; Tue, 20 Sep 2022 08:25:55 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231358AbiITIZx (ORCPT ); Tue, 20 Sep 2022 04:25:53 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45588 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231472AbiITIZG (ORCPT ); Tue, 20 Sep 2022 04:25:06 -0400 Received: from mail-wm1-x332.google.com (mail-wm1-x332.google.com [IPv6:2a00:1450:4864:20::332]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 807125B7B1 for ; Tue, 20 Sep 2022 01:22:53 -0700 (PDT) Received: by mail-wm1-x332.google.com with SMTP id c2-20020a1c3502000000b003b2973dafb7so6368067wma.2 for ; Tue, 20 Sep 2022 01:22:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:from:to:cc:subject:date; bh=DwTWJ6N47SM/rdtyan7U+NESnFMlwr8QaBDguqQpB/U=; b=oqL50cxpT9jmhkvII4XiVVmNB88wTSJjjShzjn3G5z8Eka2A5omf3IvIJVGnH2IHIr 4uBnVpCQgTsUC1JJ6av7B2k27QcmTwYiQPWSu8r9vAnlv2ztY76SXBsAN1s94lM4sNJg +nLNcItOfKQWWBi71AjoW3l0HSYLHqEmJoez4lbWSeL0igaMPaTwJADN4o8tzUpWQD0X wTnQnedLhwLfqoNUnes51TT1ETDo2D/2LdbuL58JEJ2ByuRjJUyJ0ntJNhc/QbSSpa4j E3rBihOH4mAftq9K7mpBqFM9im6zn7mA39AZlQ28vJ8qhFwnFFnIlc7MTEogOBvk0lTg MOIg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:x-gm-message-state:from:to:cc :subject:date; bh=DwTWJ6N47SM/rdtyan7U+NESnFMlwr8QaBDguqQpB/U=; b=VAjdS5FafExMNZ4eHlAucBjSuu6yAoR4Zohiw6RER/8g+/c26mjJGmcvoxH1VkMY+H tccKcFU7LeUoJANWFa5snvp2JtJ3Hbus9O03o1GSUheoJesoOSENyFYRMxb2+RGqZ1gV q+nM1cKFBEE+tlsjR1dLlmuqdSLRASvTeYGfm9JAAEQZsL2iSw1tH7HdFk0ZupxVPeJL eYcACh/2utiBk7kvvAN83aHozeogiKVl/W6u1ymVIogV+azuk+pvA7GXo6ZsUpl5xdWl OXsHUuciep5KEBlU0MVrPzItxMeJZW3b8+JcQQoUTvpgLeie8Jc03AYG1sUkC6Ve7uZf rJzA== X-Gm-Message-State: ACrzQf18kwVEypAgeNSi3uLsgiWMWuXSEwj2E1HRmQRpDJSRvzxwH42Y L803FhyMkjfzNT6dv7YQjM7X4XY/UJQ= X-Google-Smtp-Source: AMsMyM6RBUJ0Gqk6usgETwS06YzIoTKc+yS1eEgs+q59kNs4w5vpP2WuRjMVr3nHeRejE04I1dcCXA== X-Received: by 2002:a05:600c:3d93:b0:3b4:c28f:6a2b with SMTP id bi19-20020a05600c3d9300b003b4c28f6a2bmr1448675wmb.121.1663662169539; Tue, 20 Sep 2022 01:22:49 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id o17-20020a05600c511100b003b4935f04a4sm2085393wms.5.2022.09.20.01.22.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 20 Sep 2022 01:22:49 -0700 (PDT) Message-Id: <99402c1b89fd6dcaf23cd43e5df91d3ac850ebca.1663662167.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Tue, 20 Sep 2022 08:22:43 +0000 Subject: [PATCH v6 1/5] hiderefs: add hide-refs hook to hide refs dynamically Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Sun Chao <16657101987@163.com>, Sun Chao Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Sun Chao From: Sun Chao Muti-branch workflows are used in some development scenarios, especially for large teams, where different small teams are assigned to implement different features on different branches or to develop secret features on special branches. If we can control the visible reference list based on developer permissions, we can reduce the interference between reference lists of different teams, and can achieve the protection of critical core code (only certain team members can see it before making it public). This kind of reference management makes sense, and on some platforms, sunch as Gerrit implement it through server-side reference access control. We can use '{transfer,uploadpack,receive}.hiderefs' config items to control which references need to hide from clients, but the config items are static and cannot satisfy the above requirements. We need the Git server to hide references according to the user's permissions, we can try to implement this mechanism by introducing a server-side hook 'hide-refs' to dynamically hide references during reference discovery phase. This hook would be invoked by 'git-receive-pack' and 'git-upload-pack' during the reference discovery phase, each reference will be filtered with this hook. The hook executes once with no arguments for each 'git-upload-pack' and 'git-receive-pack' process. Once the hook is invoked, a version number and server process name ('uploadpack' or 'receive') will send to it in pkt-line format, followed by a flush-pkt. The hook should respond with its version number. During reference discovery phase, each reference will be filtered by this hook. In the following example, the letter 'G' stands for 'git-receive-pack' or 'git-upload-pack', and the letter 'H' stands for this hook. The hook decides if the reference will be hidden or not, and it sends the result back in pkt-line format protocol, a response "hide" means the references will be hidden to the client. # Version negotiation G: PKT-LINE(version=1\0uploadpack) G: flush-pkt H: PKT-LINE(version=1) H: flush-pkt # Send reference filter request to hook G: PKT-LINE(ref :) G: flush-pkt # Receive result from the hook. # Case 1: this reference is hidden H: PKT-LINE(hide) H: flush-pkt # Case 2: this reference can be advertised H: flush-pkt To enable the `hide-refs` hook, we should config hiderefs with `hook:` option, e.g. if we want to pass all the refs to the new hook except for the tags: git config --add transfer.hiderefs hook: git config --add transfer.hiderefs hook:!refs/tags/ Signed-off-by: Sun Chao --- refs.c | 250 ++++++++++++++++++++++++++++++++++++++++++++++++++++----- 1 file changed, 228 insertions(+), 22 deletions(-) diff --git a/refs.c b/refs.c index 92819732ab7..68368055946 100644 --- a/refs.c +++ b/refs.c @@ -8,6 +8,7 @@ #include "lockfile.h" #include "iterator.h" #include "refs.h" +#include "pkt-line.h" #include "refs/refs-internal.h" #include "run-command.h" #include "hook.h" @@ -1384,39 +1385,221 @@ char *shorten_unambiguous_ref(const char *refname, int strict) } static struct string_list *hide_refs; +static struct string_list *hook_hide_refs; +static struct strbuf hide_refs_section = STRBUF_INIT; int parse_hide_refs_config(const char *var, const char *value, const char *section) { + struct string_list **refs_list= &hide_refs; const char *key; - if (!strcmp("transfer.hiderefs", var) || - (!parse_config_key(var, section, NULL, NULL, &key) && - !strcmp(key, "hiderefs"))) { - char *ref; - int len; - - if (!value) - return config_error_nonbool(var); - ref = xstrdup(value); - len = strlen(ref); - while (len && ref[len - 1] == '/') - ref[--len] = '\0'; - if (!hide_refs) { - CALLOC_ARRAY(hide_refs, 1); - hide_refs->strdup_strings = 1; + char *ref; + int len; + + if (strcmp("transfer.hiderefs", var) && + !(!parse_config_key(var, section, NULL, NULL, &key) && + !strcmp(key, "hiderefs"))) + return 0; + + if (!value) + return config_error_nonbool(var); + + if (skip_prefix(value, "hook:", &value)) { + refs_list = &hook_hide_refs; + + /* + * Once the 'hide-refs' hook is invoked, Git needs to do + * version negotiation with it, the version number and the + * process name ('uploadpack' or 'receive') will send to + * it in pkt-line format, and the process name is recorded + * by hide_refs_section + */ + if (hide_refs_section.len == 0) + strbuf_addstr(&hide_refs_section, section); + } + + ref = xstrdup(value); + len = strlen(ref); + while (len && ref[len - 1] == '/') + ref[--len] = '\0'; + + if (!*refs_list) { + CALLOC_ARRAY(*refs_list, 1); + (*refs_list)->strdup_strings = 1; + } + string_list_append(*refs_list, ref); + + return 0; +} + +static struct child_process *hide_refs_proc; +static struct packet_reader *hide_refs_reader; + +/* + * Create the hide-refs hook child process and complete version negotiation, + * return non-zero upon success, otherwise 0 + */ +static int create_hide_refs_process(void) +{ + struct child_process *proc; + struct packet_reader *reader; + const char *hook_path; + int version = 0; + int err; + + hook_path = find_hook("hide-refs"); + if (!hook_path) + return 0; + + proc = (struct child_process *)xcalloc(1, sizeof (struct child_process)); + reader = (struct packet_reader *)xcalloc(1, sizeof(struct packet_reader)); + + child_process_init(proc); + strvec_push(&proc->args, hook_path); + proc->in = -1; + proc->out = -1; + proc->trace2_hook_name = "hide-refs"; + proc->err = 0; + + err = start_command(proc); + if (err) + goto cleanup; + + sigchain_push(SIGPIPE, SIG_IGN); + + /* Version negotiaton */ + packet_reader_init(reader, proc->out, NULL, 0, + PACKET_READ_CHOMP_NEWLINE | PACKET_READ_GENTLE_ON_EOF); + err = packet_write_fmt_gently(proc->in, "version=1%c%s", '\0', hide_refs_section.buf); + if (!err) + err = packet_flush_gently(proc->in); + + if (!err) + for (;;) { + enum packet_read_status status; + + status = packet_reader_read(reader); + if (status != PACKET_READ_NORMAL) { + /* Check whether hide-refs exited abnormally */ + if (status == PACKET_READ_EOF) + goto failure; + break; + } + + if (reader->pktlen > 8 && starts_with(reader->line, "version=")) { + version = atoi(reader->line + 8); + } } - string_list_append(hide_refs, ref); + + if (err) + goto failure; + + switch (version) { + case 0: + /* fallthrough */ + case 1: + break; + default: + trace_printf(_("hook hide-refs version '%d' is not supported"), version); + goto failure; } + + sigchain_pop(SIGPIPE); + + hide_refs_proc = proc; + hide_refs_reader = reader; + return 1; + +failure: + close(proc->in); + close(proc->out); + kill(proc->pid, SIGTERM); + finish_command_in_signal(proc); + +cleanup: + free(proc); + free(reader); + sigchain_pop(SIGPIPE); return 0; } -int ref_is_hidden(const char *refname, const char *refname_full) +/* If hide-refs child process start failed, set skip_hide_refs_proc to true */ +static int skip_hide_refs_proc; + +/* + * Return non-zero if hide-refs hook want to hide the ref and 0 otherwise, + * and return 0 if hide-refs child proccess start failed or exit abnormally + */ +static int ref_hidden_check_by_hook(const char *refname, const char *refname_full) +{ + struct strbuf buf = STRBUF_INIT; + int err; + int ret = 0; + + if (skip_hide_refs_proc) + return 0; + + if (!hide_refs_proc) + if (!create_hide_refs_process()) { + skip_hide_refs_proc = 1; + return 0; + } + + sigchain_push(SIGPIPE, SIG_IGN); + err = packet_write_fmt_gently(hide_refs_proc->in, "ref %s:%s", refname, refname_full); + if (err) + goto cleanup; + + err = packet_flush_gently(hide_refs_proc->in); + if (err) + goto cleanup; + + for (;;) { + enum packet_read_status status; + + status = packet_reader_read(hide_refs_reader); + if (status != PACKET_READ_NORMAL) { + /* Check whether hide-refs exited abnormally */ + if (status == PACKET_READ_EOF) + goto cleanup; + break; + } + + strbuf_addstr(&buf, hide_refs_reader->line); + } + + if (!strncmp("hide", buf.buf, 4)) + ret = 1; + + sigchain_pop(SIGPIPE); + return ret; + +cleanup: + close(hide_refs_proc->in); + close(hide_refs_proc->out); + kill(hide_refs_proc->pid, SIGTERM); + finish_command_in_signal(hide_refs_proc); + + free(hide_refs_proc); + free(hide_refs_reader); + sigchain_pop(SIGPIPE); + + skip_hide_refs_proc = 1; + return 0; +} + +static int ref_hidden_check(const char *refname, const char *refname_full, int hook) { + struct string_list *hide_refs_list = hide_refs; int i; + int match_all = 0; - if (!hide_refs) + if (hook) + hide_refs_list = hook_hide_refs; + if (!hide_refs_list) return 0; - for (i = hide_refs->nr - 1; i >= 0; i--) { - const char *match = hide_refs->items[i].string; + + for (i = hide_refs_list->nr - 1; i >= 0; i--) { + const char *match = hide_refs_list->items[i].string; const char *subject; int neg = 0; const char *p; @@ -1426,6 +1609,12 @@ int ref_is_hidden(const char *refname, const char *refname_full) match++; } + /* empty string with the 'hook:' option matches all the refs */ + if (hook && !*match) { + match_all = !neg; + continue; + } + if (*match == '^') { subject = refname_full; match++; @@ -1436,9 +1625,26 @@ int ref_is_hidden(const char *refname, const char *refname_full) /* refname can be NULL when namespaces are used. */ if (subject && skip_prefix(subject, match, &p) && - (!*p || *p == '/')) - return !neg; + (!*p || *p == '/')) { + if (neg) + return 0; + if (!hook) + return 1; + return ref_hidden_check_by_hook(refname, refname_full); + } } + + if (hook && match_all) + return ref_hidden_check_by_hook(refname, refname_full); + + return 0; +} + +int ref_is_hidden(const char *refname, const char *refname_full) +{ + if (ref_hidden_check(refname, refname_full, 0) || + ref_hidden_check(refname, refname_full, 1)) + return 1; return 0; } From patchwork Tue Sep 20 08:22:44 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sun Chao X-Patchwork-Id: 12981618 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 81E40C6FA8E for ; Tue, 20 Sep 2022 08:26:01 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229689AbiITI0A (ORCPT ); Tue, 20 Sep 2022 04:26:00 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41180 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229973AbiITIZ0 (ORCPT ); Tue, 20 Sep 2022 04:25:26 -0400 Received: from mail-wr1-x432.google.com (mail-wr1-x432.google.com [IPv6:2a00:1450:4864:20::432]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C14E963F2B for ; Tue, 20 Sep 2022 01:23:03 -0700 (PDT) Received: by mail-wr1-x432.google.com with SMTP id g3so2922371wrq.13 for ; Tue, 20 Sep 2022 01:23:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:from:to:cc:subject:date; bh=S3wPBgJTO6dkmn/hje0kzTD5udOG5soGoSKMPEL/fwo=; b=ZUPRlQZ8jls4QW7e2DJJZh/c8BB/607GNStf+AVaLTVr0NLns8f8aPE9xB6/i4rZOK 0g5xX6fLp3CMU+gPg4E8wjRvvbVgCXy23jyw0VNytru4Iw1UnyVCW8+AezHaiUTS//JP +k6BaizZ+IVm7pbm85kT8YAM3dqUFyeNZ+sBmpi9rkAgrL4fawIEKULz8TXUw47OiT2V KBZcMnT2Qok46EL5xpNNzytk7ePsGaRX/eoO2Y3VIG2wBrdfhB90pAUQVyuTrzwHT8z2 Yp8u8DyutoinRDprfUqp1IWOgcZ+nqnUHJvXG/jvkJ2OncIaUCY+SfaAP8lnl2kXDhfV JAVg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:x-gm-message-state:from:to:cc :subject:date; bh=S3wPBgJTO6dkmn/hje0kzTD5udOG5soGoSKMPEL/fwo=; b=sRe4dWGe7TzanpdQ9HWTlDQxtrNhNuLjvSQEOmUdiWLirIxGAhn9IppZ22+dJVjIBO yOK/X8STO0i5qy1SNYeODawHq+iJEG+eVEWjT2IB4CK6D5zbuwIBdaPMVRSYPzH62pWV qM7/YyEwV9p983gi/65ObPwrm1K6mjpBohiMAWfo0CKxGPFArWIjYY6yAlXDTPuZKOvl AgWbkJZYwKs1DGVI5pwIC9n+d+wJxFDJ8oB3uPxyBckwkXCe+LfMUX+Bn33X6Hb+dYxn qW76hLFtR3lbT/YwMHdQf7FhqNv1gHTNbilQvhdos1kM4FlCQ5AvwMvCxLm1gmKe8mvX Y1zA== X-Gm-Message-State: ACrzQf19DMqtvBr4Ux1P90KvpvZENVyiHVGLoGqGAbSn4AUeikdR+V8c NAvwHrzwJCKBUQmy10q/jXXYrBoLkAc= X-Google-Smtp-Source: AMsMyM5P6WqqyYNmomQ8tXINCH/ey0ZO7yheHYkx/KWOqnlivzFUQAqRUunDn159fQvgIW4ih+oG1g== X-Received: by 2002:adf:c713:0:b0:22a:3670:b08d with SMTP id k19-20020adfc713000000b0022a3670b08dmr13346866wrg.175.1663662170432; Tue, 20 Sep 2022 01:22:50 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id j32-20020a05600c1c2000b003b4a699ce8esm19632036wms.6.2022.09.20.01.22.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 20 Sep 2022 01:22:49 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Tue, 20 Sep 2022 08:22:44 +0000 Subject: [PATCH v6 2/5] hiderefs: use a new flag to mark force hidden refs Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Sun Chao <16657101987@163.com>, Sun Chao Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Sun Chao From: Sun Chao If uploadpack.allowTipSHA1InWant or uploadpack.allowReachableSHA1InWant is set to true, the private commits of hiding refs can be fetched by the client. The new hide-refs hook is used to hide our refs and we wish to hide the private commits either. A new flag `HIDDEN_REF_FORCE` is used to mark a ref if hide-refs hook decides to hide it, and we make sure the wire protocol V1 will reject to send the private commits of this kind of refs even if uploadpack.allowTipSHA1InWant or uploadpack.allowReachableSHA1InWant is set to true. Signed-off-by: Sun Chao --- refs.c | 39 +++++++++++++++++++++++++++++++++++++++ refs.h | 3 +++ upload-pack.c | 14 -------------- 3 files changed, 42 insertions(+), 14 deletions(-) diff --git a/refs.c b/refs.c index 68368055946..5a9079fd4c4 100644 --- a/refs.c +++ b/refs.c @@ -1648,6 +1648,45 @@ int ref_is_hidden(const char *refname, const char *refname_full) return 0; } +#define OUR_REF (1u << 12) +#define HIDDEN_REF (1u << 19) +/* + * Use this flag to mark a ref that is hidden by the hide-refs hook, its private + * commits (tip or non-tip commits, not reachable by the refs not hidden by the + * hide-refs hook) will be forced hidden to the client, which means a client can + * not fetch such kind of commits even uploadpack.allowTipSHA1InWant or + * uploadpack.allowReachableSHA1InWant are set to true + */ +#define HIDDEN_REF_FORCE (1u << 20) + +static unsigned int ref_hidden_flag(const char *refname, const char *refname_full) +{ + if (ref_hidden_check(refname, refname_full, 1)) + return HIDDEN_REF_FORCE; + else if (ref_hidden_check(refname, refname_full, 0)) + return HIDDEN_REF; + return OUR_REF; +} + +int mark_our_ref(const char *refname, const char *refname_full, + const struct object_id *oid) +{ + struct object *o; + unsigned int flag; + + if (!oid || is_null_oid(oid)) { + return 0; + } + + o = lookup_unknown_object(the_repository, oid); + flag = ref_hidden_flag(refname, refname_full); + o->flags |= flag; + + if (flag & OUR_REF) + return 0; + return 1; +} + const char *find_descendant_ref(const char *dirname, const struct string_list *extras, const struct string_list *skip) diff --git a/refs.h b/refs.h index d6575b8c2bd..2feabfe35c4 100644 --- a/refs.h +++ b/refs.h @@ -819,6 +819,9 @@ int parse_hide_refs_config(const char *var, const char *value, const char *); * parameter always points to the full ref name. */ int ref_is_hidden(const char *, const char *); +/* return non-zero if the ref is hidden, otherwise 0 */ +int mark_our_ref(const char *refname, const char *refname_full, + const struct object_id *oid); enum ref_type { REF_TYPE_PER_WORKTREE, /* refs inside refs/ but not shared */ diff --git a/upload-pack.c b/upload-pack.c index b217a1f469e..a8ca5d1c26e 100644 --- a/upload-pack.c +++ b/upload-pack.c @@ -1155,20 +1155,6 @@ static void receive_needs(struct upload_pack_data *data, packet_flush(1); } -/* return non-zero if the ref is hidden, otherwise 0 */ -static int mark_our_ref(const char *refname, const char *refname_full, - const struct object_id *oid) -{ - struct object *o = lookup_unknown_object(the_repository, oid); - - if (ref_is_hidden(refname, refname_full)) { - o->flags |= HIDDEN_REF; - return 1; - } - o->flags |= OUR_REF; - return 0; -} - static int check_ref(const char *refname_full, const struct object_id *oid, int flag, void *cb_data) { From patchwork Tue Sep 20 08:22:45 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sun Chao X-Patchwork-Id: 12981617 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4D763C6FA82 for ; Tue, 20 Sep 2022 08:26:00 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231394AbiITIZ6 (ORCPT ); Tue, 20 Sep 2022 04:25:58 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40788 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231511AbiITIZS (ORCPT ); Tue, 20 Sep 2022 04:25:18 -0400 Received: from mail-wm1-x32a.google.com (mail-wm1-x32a.google.com [IPv6:2a00:1450:4864:20::32a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D3B5965557 for ; Tue, 20 Sep 2022 01:23:03 -0700 (PDT) Received: by mail-wm1-x32a.google.com with SMTP id n17-20020a05600c501100b003a84bf9b68bso1115172wmr.3 for ; Tue, 20 Sep 2022 01:23:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:from:to:cc:subject:date; bh=w98ayTfiOelGTSb248qoEl8t0T22TZDz6ldsyPa8uf0=; b=W1VflUwi8Xjf7ADkdS0UDUXdZW951Our4u4n7LP/BKbOqMenRzJf9gxE6O0BZ9dIFo /UfSoNjqd6kEiFgQfPOu77cDUsTVZOhfV3XkfyW4VqaTHrpoNsENmAazCCbe5IZJjiL/ 8HKGSnvnU5hvbAVeG/5zDq8c1LpJGLmUiRy/VfDQFDBew82Z94x5le2qZ9Az1czJm6Bf ee/3G9pa5jyxGV5DwlgQr+quUb6Cjm8MIbtvg/lOT2Y3XcdRsODM9yKd+XBJuHpa1A2a eMzAm2+gF/sQcYx48+Siz5EtIgWA5LkF0Nn7mwtVOUwTx/HTRUdtrAXuZ89sTwXJGVBh iDkQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:x-gm-message-state:from:to:cc :subject:date; bh=w98ayTfiOelGTSb248qoEl8t0T22TZDz6ldsyPa8uf0=; b=EzPPLbekhDbTD8m3BGPwbFUb44x5TySJC9EZZwoS9r7nLAaiNY9Z4O1PLW9LMVOUUR 6hFv8elOohDduUjc2PygtzM8mIyA+NBdYBAEwPN53z7kEVVrAnTD7WC0yIITEf8rNawu bM/UWjdFNUXXAYpbxhu5bT+63BOp7suygybgnPcU0HxOSvX6/9PrMwfDqVhCANq1WWNi kzfM+I+u04UWpLUBPNWfbJju+bQEjKCOdU3qz3gVZQVOtpXIHCKtfBytUWxdlw2eI+Cb BWjS273TAytCR9JnrRMKEUKYFI4CBY9ibq9Egc2MWR8+CvrFH/KVjjgrUYKKoJM88Ltl dcdQ== X-Gm-Message-State: ACrzQf1pTkUpdzD6S/T1a7Yiogr1qzaPOaqSjcEKiaIUiM9nXguURF7V MQcL2hkXkD7n7J1WYAJuu4tyzOVQ/QQ= X-Google-Smtp-Source: AMsMyM7i+9xRhbhblH7bYiQSyI6WjVa+1yn4PJLERJW8A79lV91L/oCo/sX3SbC6zuKRDLIADrMf6A== X-Received: by 2002:a05:600c:198d:b0:3b4:b6b6:6fa with SMTP id t13-20020a05600c198d00b003b4b6b606famr1466683wmq.110.1663662171303; Tue, 20 Sep 2022 01:22:51 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id o13-20020a05600c4fcd00b003b332a7bf15sm17482312wmq.7.2022.09.20.01.22.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 20 Sep 2022 01:22:50 -0700 (PDT) Message-Id: <0013476266e05aebd41e455b3d6305e8ec347a9e.1663662167.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Tue, 20 Sep 2022 08:22:45 +0000 Subject: [PATCH v6 3/5] hiderefs: hornor hide flags in wire protocol V2 Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Sun Chao <16657101987@163.com>, Sun Chao Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Sun Chao From: Sun Chao Previously hiderefs configurations can not protect the private data of hiding refs in wire protocol V2, for example, an `ALL_FLAGS` flag will be used to clear all the objects before handling the fetch requests. Hornor the hide flags by removing the `HIDDEN_REFS` flag from the `ALL_FLAGS` and make sure all the refs will check its hidden flags before sending the pack to client, especially during stateless RPC. And if there are refs with `HIDDEN_REF_FORCE` flag, use `check_non_tip` to protect the private data of force-hidden refs. Signed-off-by: Sun Chao --- ls-refs.c | 2 +- refs.c | 20 ++++++++++++++++++++ refs.h | 4 ++++ upload-pack.c | 11 +++++++++-- 4 files changed, 34 insertions(+), 3 deletions(-) diff --git a/ls-refs.c b/ls-refs.c index 98e69373c84..b5cb1316d38 100644 --- a/ls-refs.c +++ b/ls-refs.c @@ -84,7 +84,7 @@ static int send_ref(const char *refname, const struct object_id *oid, strbuf_reset(&data->buf); - if (ref_is_hidden(refname_nons, refname)) + if (mark_our_ref(refname_nons, refname, oid)) return 0; if (!ref_match(&data->prefixes, refname_nons)) diff --git a/refs.c b/refs.c index 5a9079fd4c4..847f7c003e6 100644 --- a/refs.c +++ b/refs.c @@ -1659,6 +1659,25 @@ int ref_is_hidden(const char *refname, const char *refname_full) */ #define HIDDEN_REF_FORCE (1u << 20) +/* Use this variable to record existing object hidden flags */ +static unsigned int objects_hidden_flags; + +/* Return non-zero if need to batch check hidden refs, otherwise 0 */ +int need_check_hidden_refs(void) +{ + if (!objects_hidden_flags) + return 1; + return 0; +} + +/* Return non-zero if some ref is force hidden, otherwise 0 */ +int has_force_hidden_refs(void) +{ + if (objects_hidden_flags & HIDDEN_REF_FORCE) + return 1; + return 0; +} + static unsigned int ref_hidden_flag(const char *refname, const char *refname_full) { if (ref_hidden_check(refname, refname_full, 1)) @@ -1681,6 +1700,7 @@ int mark_our_ref(const char *refname, const char *refname_full, o = lookup_unknown_object(the_repository, oid); flag = ref_hidden_flag(refname, refname_full); o->flags |= flag; + objects_hidden_flags |= flag; if (flag & OUR_REF) return 0; diff --git a/refs.h b/refs.h index 2feabfe35c4..8deb36a95cc 100644 --- a/refs.h +++ b/refs.h @@ -822,6 +822,10 @@ int ref_is_hidden(const char *, const char *); /* return non-zero if the ref is hidden, otherwise 0 */ int mark_our_ref(const char *refname, const char *refname_full, const struct object_id *oid); +/* return non-zero if need to batch check hidden refs, otherwise 0 */ +int need_check_hidden_refs(void); +/* return non-zero if some ref is force hidden, otherwise 0 */ +int has_force_hidden_refs(void); enum ref_type { REF_TYPE_PER_WORKTREE, /* refs inside refs/ but not shared */ diff --git a/upload-pack.c b/upload-pack.c index a8ca5d1c26e..a9a24399d8e 100644 --- a/upload-pack.c +++ b/upload-pack.c @@ -39,8 +39,8 @@ #define CLIENT_SHALLOW (1u << 18) #define HIDDEN_REF (1u << 19) -#define ALL_FLAGS (THEY_HAVE | OUR_REF | WANTED | COMMON_KNOWN | SHALLOW | \ - NOT_SHALLOW | CLIENT_SHALLOW | HIDDEN_REF) +#define ALL_FLAGS (THEY_HAVE |WANTED | COMMON_KNOWN | SHALLOW | \ + NOT_SHALLOW | CLIENT_SHALLOW) /* Enum for allowed unadvertised object request (UOR) */ enum allow_uor { @@ -1726,6 +1726,13 @@ int upload_pack_v2(struct repository *r, struct packet_reader *request) state = FETCH_DONE; break; case FETCH_SEND_PACK: + if (need_check_hidden_refs()) { + head_ref_namespaced(check_ref, NULL); + for_each_namespaced_ref(check_ref, NULL); + } + if (has_force_hidden_refs()) + check_non_tip(&data); + send_wanted_ref_info(&data); send_shallow_info(&data); From patchwork Tue Sep 20 08:22:46 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sun Chao X-Patchwork-Id: 12981620 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9DBF6C54EE9 for ; Tue, 20 Sep 2022 08:26:24 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231435AbiITI0D (ORCPT ); Tue, 20 Sep 2022 04:26:03 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47506 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230225AbiITIZa (ORCPT ); Tue, 20 Sep 2022 04:25:30 -0400 Received: from mail-wr1-x430.google.com (mail-wr1-x430.google.com [IPv6:2a00:1450:4864:20::430]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C19C965552 for ; Tue, 20 Sep 2022 01:23:04 -0700 (PDT) Received: by mail-wr1-x430.google.com with SMTP id g3so2922510wrq.13 for ; Tue, 20 Sep 2022 01:23:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:from:to:cc:subject:date; bh=RAsRaME0rS6blRDqVRLawMM3/n2QxQ5ecSc+Gp7peDE=; b=MC5VAJQV1XdStFkzJzAYnOgFTed2yio7Z+uFQ0yYtpE5Za2thiXoDHHwwM5X56rCEU 1Vua8zAF9HbGItiw/R4chlSDguCYU3RUI+L+W3K16D64tm/dES9sCB8WVGYFf4G0H6DQ IpK2h/vShIVMW5JJNNgdpavDzsmrT3+v1R6civ6MKNzwpzHDGkJzSsXE3YpGKbtHteUf tCiYQJsngMxefV8n4qeVBPPL2UmaajcOXKgB7QTAVCfm6Sa8WQGfggJ/gk9grGMkgiGA jcLaUBNk5S8YGaVOpmo5mkWBBOi9mX7ue+UsJ80aLkWc8YaXUv8w3lBg0y/j9BJQyWEO uGRA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:x-gm-message-state:from:to:cc :subject:date; bh=RAsRaME0rS6blRDqVRLawMM3/n2QxQ5ecSc+Gp7peDE=; b=nqCYp5EpXE01XJNk4ibd8s1avt1710FPN+ZRjlgPiktGIuJjF0Aw1ujEbwK7ELnIks sVfOvuTpE/c9K4yrenmUX64e5B+QCk/U2dYauy6LzJj3fIuj5/VTrmfSXYbAjhz7Zg6U PAU3PyX0xW54vMF7SstZrhH5j0aTtciQz0MHF0jXVp/s3WJAbkJqdIhTYC6Y1kTHQiRT RaSDSFPZmeaA3cjxOm+s8pbcvnx+ebcCZSdk+ZGTDkTdWt+q2BPLV2WhbbPbyBYPbdum aK4yzOvMn2uSMLwXkMDPvSQXatAIZIRTFJcJKMgPrxhiDAJsp3fn53ugGdZTNsKo1Hix wKXA== X-Gm-Message-State: ACrzQf3cGdz76Si5Y1OepsFImSJ0UDEgCS8IcOxov33uuc/gUNyOyH+t saATIz1JYmyFWyOjD1m412/1oDQsHg0= X-Google-Smtp-Source: AMsMyM5hvZOCs9mB164/VG5BzneDJh8UYbC/NHBpDGq7aW3Qa19IczcbIU/Na24ybzNp8xuHrpPLUA== X-Received: by 2002:a5d:444c:0:b0:22a:e6a2:c498 with SMTP id x12-20020a5d444c000000b0022ae6a2c498mr10086310wrr.531.1663662172096; Tue, 20 Sep 2022 01:22:52 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id a13-20020a05600c348d00b003a5ffec0b91sm1437984wmq.30.2022.09.20.01.22.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 20 Sep 2022 01:22:51 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Tue, 20 Sep 2022 08:22:46 +0000 Subject: [PATCH v6 4/5] test: add test cases for hide-refs hook Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Sun Chao <16657101987@163.com>, Sun Chao Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Sun Chao From: Sun Chao Add test cases for the new 'hide-refs' hook which is used to filter the references during the reference discovery phase. Signed-off-by: Sun Chao --- Makefile | 1 + t/helper/test-hide-refs.c | 107 +++++++++++++++++ t/helper/test-tool.c | 1 + t/helper/test-tool.h | 1 + t/t1419-hide-refs-hook.sh | 111 ++++++++++++++++++ t/t1419/abnormal-hide-refs-hook.sh | 80 +++++++++++++ t/t1419/common-functions.sh | 74 ++++++++++++ ...test-0001-ls-remote-with-hide-refs-hook.sh | 43 +++++++ ...st-0002-upload-pack-with-hide-refs-hook.sh | 45 +++++++ ...t-0003-receive-pack-with-hide-refs-hook.sh | 40 +++++++ 10 files changed, 503 insertions(+) create mode 100644 t/helper/test-hide-refs.c create mode 100755 t/t1419-hide-refs-hook.sh create mode 100644 t/t1419/abnormal-hide-refs-hook.sh create mode 100644 t/t1419/common-functions.sh create mode 100644 t/t1419/test-0001-ls-remote-with-hide-refs-hook.sh create mode 100644 t/t1419/test-0002-upload-pack-with-hide-refs-hook.sh create mode 100644 t/t1419/test-0003-receive-pack-with-hide-refs-hook.sh diff --git a/Makefile b/Makefile index 924b864ae83..c6793681b41 100644 --- a/Makefile +++ b/Makefile @@ -794,6 +794,7 @@ TEST_BUILTINS_OBJS += test-wildmatch.o TEST_BUILTINS_OBJS += test-windows-named-pipe.o TEST_BUILTINS_OBJS += test-write-cache.o TEST_BUILTINS_OBJS += test-xml-encode.o +TEST_BUILTINS_OBJS += test-hide-refs.o # Do not add more tests here unless they have extra dependencies. Add # them in TEST_BUILTINS_OBJS above. diff --git a/t/helper/test-hide-refs.c b/t/helper/test-hide-refs.c new file mode 100644 index 00000000000..751fc6213f3 --- /dev/null +++ b/t/helper/test-hide-refs.c @@ -0,0 +1,107 @@ +#include "cache.h" +#include "hash.h" +#include "config.h" +#include "connect.h" +#include "parse-options.h" +#include "pkt-line.h" +#include "sigchain.h" +#include "test-tool.h" + +static const char *hide_refs_usage[] = { + "test-tool hide-refs [...]", + NULL +}; + +static int die_before_read_ref; +static int die_after_proc_ref; +static int version = 1; +static int hash_size = GIT_SHA1_HEXSZ; +static struct string_list hidelist = STRING_LIST_INIT_NODUP; + +static void hide_refs_verison(struct packet_reader *reader) { + int server_version = 0; + + for (;;) { + if (packet_reader_read(reader) != PACKET_READ_NORMAL) + break; + + if (reader->pktlen > 8 && starts_with(reader->line, "version=")) { + server_version = atoi(reader->line+8); + if (server_version != 1) + die("bad protocol version: %d", server_version); + } + } + + packet_write_fmt(1, "version=%d\n", version); + packet_flush(1); +} + +static void hide_refs_proc(struct packet_reader *reader) +{ + const char *p; + struct strbuf buf = STRBUF_INIT; + enum packet_read_status status; + + if (die_before_read_ref) + die("die with the --die-before-read-ref option"); + + for (;;) { + status = packet_reader_read(reader); + if (status == PACKET_READ_EOF) + exit(0); + + if (status != PACKET_READ_NORMAL) + break; + + p = reader->line; + strbuf_reset(&buf); + strbuf_addstr(&buf, reader->line); + } + + p = strchr(buf.buf, ':'); + if (unsorted_string_list_has_string(&hidelist, p + 1)) { + packet_write_fmt(1, "hide"); + } + + if (die_after_proc_ref) + die("die with the --die-after-proc-refs option"); + + packet_flush(1); +} + +int cmd__hide_refs(int argc, const char **argv) { + int nongit_ok = 0; + struct packet_reader reader; + const char *value = NULL; + struct option options[] = { + OPT_BOOL(0, "die-before-read-ref", &die_before_read_ref, + "die when reading first reference"), + OPT_BOOL(0, "die-after-proc-refs", &die_after_proc_ref, + "die after proc ref"), + OPT_STRING_LIST('H', "hide", &hidelist, "refs-to-force-hidden", + "refs that will be force hidden"), + OPT_INTEGER('V', "version", &version, + "use this protocol version number"), + OPT_END() + }; + + setup_git_directory_gently(&nongit_ok); + + argc = parse_options(argc, argv, "test-tools", options, hide_refs_usage, 0); + if (argc > 0) + usage_msg_opt("Too many arguments.", hide_refs_usage, options); + + packet_reader_init(&reader, 0, NULL, 0, PACKET_READ_CHOMP_NEWLINE | PACKET_READ_GENTLE_ON_EOF); + + if (!git_config_get_value("extensions.objectformat", &value)) { + if (!strcmp(value, "sha256")) + hash_size = GIT_SHA256_HEXSZ; + } + + hide_refs_verison(&reader); + for (;;) { + hide_refs_proc(&reader); + } + + return 0; +} diff --git a/t/helper/test-tool.c b/t/helper/test-tool.c index 80055886798..c5bd7f1f806 100644 --- a/t/helper/test-tool.c +++ b/t/helper/test-tool.c @@ -72,6 +72,7 @@ static struct test_cmd cmds[] = { { "regex", cmd__regex }, { "repository", cmd__repository }, { "revision-walking", cmd__revision_walking }, + { "hide-refs", cmd__hide_refs }, { "run-command", cmd__run_command }, { "scrap-cache-tree", cmd__scrap_cache_tree }, { "serve-v2", cmd__serve_v2 }, diff --git a/t/helper/test-tool.h b/t/helper/test-tool.h index a432cc77d92..cba8d3b093d 100644 --- a/t/helper/test-tool.h +++ b/t/helper/test-tool.h @@ -61,6 +61,7 @@ int cmd__reftable(int argc, const char **argv); int cmd__regex(int argc, const char **argv); int cmd__repository(int argc, const char **argv); int cmd__revision_walking(int argc, const char **argv); +int cmd__hide_refs(int argc, const char **argv); int cmd__run_command(int argc, const char **argv); int cmd__scrap_cache_tree(int argc, const char **argv); int cmd__serve_v2(int argc, const char **argv); diff --git a/t/t1419-hide-refs-hook.sh b/t/t1419-hide-refs-hook.sh new file mode 100755 index 00000000000..81a8c36190b --- /dev/null +++ b/t/t1419-hide-refs-hook.sh @@ -0,0 +1,111 @@ +#!/bin/sh +# +# Copyright (c) 2022 Sun Chao +# + +test_description='Test hide-refs hook' + +. ./test-lib.sh +. "$TEST_DIRECTORY"/t1419/common-functions.sh + +GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main +export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME + +setup_test_repos () { + test_expect_success "setup bare_repo and work_repo" ' + rm -rf bare_repo.git && + rm -rf work_repo && + git init --bare bare_repo.git && + git init work_repo && + + # create new commits and references + create_commits_in work_repo A B C D && + ( + cd work_repo && + git config --local core.abbrev 7 && + git update-ref refs/heads/main $A && + git update-ref refs/heads/dev $B && + git update-ref refs/pull-requests/1/head $C && + git tag -m "v123" v123 $D && + git push ../bare_repo.git +refs/heads/*:refs/heads/* && + git push ../bare_repo.git +refs/tags/*:refs/tags/* && + git push ../bare_repo.git +refs/pull-requests/*:refs/pull-requests/* + ) && + TAG=$(git -C work_repo rev-parse v123) && + + # config transfer.hiderefs values with "hook:" prefix + ( + git -C bare_repo.git config --local http.receivepack true && + git -C bare_repo.git config --add transfer.hiderefs hook: + ) + ' +} + +setup_httpd() { + ROOT_PATH="$PWD" + . "$TEST_DIRECTORY"/lib-gpg.sh + . "$TEST_DIRECTORY"/lib-httpd.sh + . "$TEST_DIRECTORY"/lib-terminal.sh + + start_httpd + set_askpass user@host pass@host + setup_askpass_helper +} + +# Run test cases when hide-refs hook exit abnormally +run_tests_for_abnormal_hook() { + GIT_TEST_PROTOCOL_VERSION=$1 + BAREREPO_GIT_DIR="$(pwd)/bare_repo.git" + + for t in "$TEST_DIRECTORY"/t1419/abnormal-*.sh + do + setup_test_repos + + . "$t" + done +} + +# Run test cases under local/HTTP protocol +run_tests_for_normal_hook() { + for t in "$TEST_DIRECTORY"/t1419/test-*.sh + do + setup_test_repos + case $1 in + http) + PROTOCOL="HTTP protocol" + + # bare_repo.git need move to httpd sever root path + BAREREPO_GIT_DIR="$HTTPD_DOCUMENT_ROOT_PATH/bare_repo.git" + rm -rf "$BAREREPO_GIT_DIR" + mv bare_repo.git "$BAREREPO_GIT_DIR" + + # setup the repository service URL address of http protocol + BAREREPO_PREFIX="$HTTPD_URL"/smart + BAREREPO_URL="$BAREREPO_PREFIX/bare_repo.git" + ;; + local) + PROTOCOL="builtin protocol" + BAREREPO_GIT_DIR="$(pwd)/bare_repo.git" + + # setup the repository service address of builtin protocol + BAREREPO_PREFIX="$(pwd)" + BAREREPO_URL="$BAREREPO_PREFIX/bare_repo.git" + ;; + esac + + GIT_TEST_PROTOCOL_VERSION=$2 + git -C work_repo remote add origin "$BAREREPO_URL" + + . "$t" + done +} + +setup_httpd +for protocol in 1 2 +do + run_tests_for_abnormal_hook $protocol + run_tests_for_normal_hook local $protocol + run_tests_for_normal_hook http $protocol +done + +test_done diff --git a/t/t1419/abnormal-hide-refs-hook.sh b/t/t1419/abnormal-hide-refs-hook.sh new file mode 100644 index 00000000000..1ce768191c8 --- /dev/null +++ b/t/t1419/abnormal-hide-refs-hook.sh @@ -0,0 +1,80 @@ +#!/bin/sh + +# The upstream repository (bare_repo.git) contains the configurations: +# +# [transfer] hiderefs = hook: +# +# During the reference advertise phase the hide-refs hook will be invoked and all the refs will be checked by it, +# we should make sure Git works correctly in some special cases + +# If the hide-refs does not exist, Git should not invoke it and continue to advertise all the refs +test_expect_success "protocol $GIT_TEST_PROTOCOL_VERSION: advertise-refs while hide-refs hook not exists" ' + rm -f "$BAREREPO_GIT_DIR/hooks/hide-refs" && + git -c protocol.version=$GIT_TEST_PROTOCOL_VERSION upload-pack --advertise-refs "$BAREREPO_GIT_DIR" >out 2>&1 && + cat out | make_user_friendly_and_stable_output >actual && + format_and_save_expect <<-EOF && + HEAD + refs/heads/dev + refs/heads/main + refs/pull-requests/1/head + refs/tags/v123 + refs/tags/v123^{} + EOF + test_cmp expect actual +' + +# If the hide-refs hook run with incompatible version, Git should not invoke it and continue to advertise all the refs +test_expect_success "protocol $GIT_TEST_PROTOCOL_VERSION: advertise-refs while hide-refs hook run with incompatible version" ' + write_script "$BAREREPO_GIT_DIR/hooks/hide-refs" <<-EOF && + test-tool hide-refs --version=2 + EOF + git -c protocol.version=$GIT_TEST_PROTOCOL_VERSION upload-pack --advertise-refs "$BAREREPO_GIT_DIR" >out 2>&1 && + cat out | make_user_friendly_and_stable_output >actual && + format_and_save_expect <<-EOF && + HEAD + refs/heads/dev + refs/heads/main + refs/pull-requests/1/head + refs/tags/v123 + refs/tags/v123^{} + EOF + test_cmp expect actual +' + +# If the hide-refs hook exit before processing any refs, Git should not die and continue to advertise all the refs +test_expect_success "protocol $GIT_TEST_PROTOCOL_VERSION: advertise-refs while hide-refs hook die before read ref" ' + write_script "$BAREREPO_GIT_DIR/hooks/hide-refs" <<-EOF && + test-tool hide-refs --die-before-read-ref + EOF + git -c protocol.version=$GIT_TEST_PROTOCOL_VERSION upload-pack --advertise-refs "$BAREREPO_GIT_DIR" >out 2>&1 && + cat out | make_user_friendly_and_stable_output | grep -v "^error:" >actual && + format_and_save_expect <<-EOF && + fatal: die with the --die-before-read-ref option + HEAD + refs/heads/dev + refs/heads/main + refs/pull-requests/1/head + refs/tags/v123 + refs/tags/v123^{} + EOF + test_cmp expect actual +' + +# If the hide-refs hook exit abnormally, Git should not die and continue to advertise left refs +test_expect_success "protocol $GIT_TEST_PROTOCOL_VERSION: advertise-refs while hide-refs hook die after proc ref" ' + write_script "$BAREREPO_GIT_DIR/hooks/hide-refs" <<-EOF && + test-tool hide-refs --die-after-proc-refs + EOF + git -c protocol.version=$GIT_TEST_PROTOCOL_VERSION upload-pack --advertise-refs "$BAREREPO_GIT_DIR" >out 2>&1 && + cat out | make_user_friendly_and_stable_output | grep -v "^error:" >actual && + format_and_save_expect <<-EOF && + fatal: die with the --die-after-proc-refs option + HEAD + refs/heads/dev + refs/heads/main + refs/pull-requests/1/head + refs/tags/v123 + refs/tags/v123^{} + EOF + test_cmp expect actual +' diff --git a/t/t1419/common-functions.sh b/t/t1419/common-functions.sh new file mode 100644 index 00000000000..7841241b038 --- /dev/null +++ b/t/t1419/common-functions.sh @@ -0,0 +1,74 @@ +# Create commits in and assign each commit's oid to shell variables +# given in the arguments (A, B, and C). E.g.: +# +# create_commits_in A B C +# +# NOTE: Never calling this function from a subshell since variable +# assignments will disappear when subshell exits. +create_commits_in () { + repo="$1" && test -d "$repo" || + error "Repository $repo does not exist." + shift && + while test $# -gt 0 + do + name=$1 && + shift && + test_commit -C "$repo" --no-tag "$name" && + eval $name=$(git -C "$repo" rev-parse HEAD) + done +} + +get_abbrev_oid () { + oid=$1 && + suffix=${oid#???????} && + oid=${oid%$suffix} && + if test -n "$oid" + then + echo "$oid" + else + echo "undefined-oid" + fi +} + +# Format the output of git-fetch, git-ls-remote and other commands to make a +# user-friendly and stable text. We can easily prepare the expect text +# without having to worry about changes of the commit ID (full or abbrev.) +# of the output. Single quotes are replaced with double quotes, because +# it is boring to prepare unquoted single quotes in expect text. +make_user_friendly_and_stable_output () { + tr '\0' '@' | sed \ + -e "s/'/\"/g" \ + -e "s/@.*//g" \ + -e "s/$(get_abbrev_oid $A)[0-9a-f]*//g" \ + -e "s/$(get_abbrev_oid $B)[0-9a-f]*//g" \ + -e "s/$(get_abbrev_oid $C)[0-9a-f]*//g" \ + -e "s/$(get_abbrev_oid $D)[0-9a-f]*//g" \ + -e "s/$(get_abbrev_oid $TAG)[0-9a-f]*//g" \ + -e "s/$ZERO_OID//g" \ + -e "s#$BAREREPO_PREFIX/bare_repo.git##" \ + -e 's/^[0-9a-f]\{4\}//g' + +} + +filter_out_hide_refs_output() { + make_user_friendly_and_stable_output | sed 's/^[0-9a-f]\{4\}//g' +} + +format_and_save_expect () { + sed -e 's/^> //' -e 's/Z$//' >expect +} + +test_cmp_refs () { + indir= + if test "$1" = "-C" + then + shift + indir="$1" + shift + fi + indir=${indir:+"$indir"/} + cat >show-ref.expect && + git ${indir:+ -C "$indir"} show-ref >show-ref.pristine && + make_user_friendly_and_stable_output show-ref.filtered && + test_cmp show-ref.expect show-ref.filtered +} diff --git a/t/t1419/test-0001-ls-remote-with-hide-refs-hook.sh b/t/t1419/test-0001-ls-remote-with-hide-refs-hook.sh new file mode 100644 index 00000000000..3831521d345 --- /dev/null +++ b/t/t1419/test-0001-ls-remote-with-hide-refs-hook.sh @@ -0,0 +1,43 @@ +#!/bin/sh + +# The upstream repository (bare_repo.git) contains the configurations: +# +# [transfer] hiderefs = hook: +# +# During the reference advertise phase the hide-refs hook will be invoked and all the refs will be checked by it + +# Git will not advertise the refs that are hidden by the hide-refs hook +test_expect_success "$PROTOCOL (protocol: $GIT_TEST_PROTOCOL_VERSION): ls-remote while hide-refs hook hide part of refs" ' + write_script "$BAREREPO_GIT_DIR/hooks/hide-refs" <<-EOF && + test-tool hide-refs \ + -H "refs/pull-requests/1/head" \ + -H "refs/tags/v123" + EOF + git -c protocol.version=$GIT_TEST_PROTOCOL_VERSION ls-remote "$BAREREPO_URL" >out 2>&1 && + make_user_friendly_and_stable_output actual && + format_and_save_expect <<-EOF && + HEAD + refs/heads/dev + refs/heads/main + EOF + test_cmp expect actual +' + +# The hide-ref hook should not change the default effects of '{transfer,uploadpack,receive}.hiderefs' +# configurations, if it hides no refs, the original hiderefs configurations should work +test_expect_success "$PROTOCOL (protocol: $GIT_TEST_PROTOCOL_VERSION): ls-remote while hide-refs hook hide no refs" ' + write_script "$BAREREPO_GIT_DIR/hooks/hide-refs" <<-EOF && + test-tool hide-refs + EOF + git -C "$BAREREPO_GIT_DIR" config --add transfer.hiderefs refs/heads/dev && + git -c protocol.version=$GIT_TEST_PROTOCOL_VERSION ls-remote "$BAREREPO_URL" >out 2>&1 && + make_user_friendly_and_stable_output actual && + format_and_save_expect <<-EOF && + HEAD + refs/heads/main + refs/pull-requests/1/head + refs/tags/v123 + refs/tags/v123^{} + EOF + test_cmp expect actual +' diff --git a/t/t1419/test-0002-upload-pack-with-hide-refs-hook.sh b/t/t1419/test-0002-upload-pack-with-hide-refs-hook.sh new file mode 100644 index 00000000000..618a54e16a1 --- /dev/null +++ b/t/t1419/test-0002-upload-pack-with-hide-refs-hook.sh @@ -0,0 +1,45 @@ +#!/bin/sh + +# The upstream repository (bare_repo.git) contains the configurations: +# +# [transfer] hiderefs = hook: +# +# During the reference advertise phase the hide-refs hook will be invoked and all the refs will be checked by it + +# Git client can not fetch the refs that are hidden by the hide-refs hook +test_expect_success "$PROTOCOL (protocol: $GIT_TEST_PROTOCOL_VERSION): mirror clone while hide-refs hide part of refs" ' + rm -rf local.git && + write_script "$BAREREPO_GIT_DIR/hooks/hide-refs" <<-EOF && + test-tool hide-refs \ + -H "HEAD" \ + -H "refs/heads/dev" \ + -H "refs/heads/main" + EOF + git -c protocol.version=$GIT_TEST_PROTOCOL_VERSION clone --mirror "$BAREREPO_URL" local.git && + git -C local.git show-ref -d >out 2>&1 && + make_user_friendly_and_stable_output actual && + format_and_save_expect <<-EOF && + refs/pull-requests/1/head + refs/tags/v123 + refs/tags/v123^{} + EOF + test_cmp expect actual +' + +# If a ref is hidden by the hide-refs hook, its private commits (tip or non-tip) will be forced hidden +# to the client, and the client can not fetch such kind of commits even if the server set allowTipSHA1InWant +# or allowReachableSHA1InWant to true +test_expect_success "$PROTOCOL (protocol: $GIT_TEST_PROTOCOL_VERSION): fetch a commit which is hided by hide-refs hook" ' + rm -rf local.git && + write_script "$BAREREPO_GIT_DIR/hooks/hide-refs" <<-EOF && + test-tool hide-refs \ + -H "refs/heads/dev" \ + -H "refs/pull-requests/1/head" \ + -H "refs/tags/v123" + EOF + git -C "$BAREREPO_GIT_DIR" config uploadpack.allowTipSHA1InWant true && + git -C "$BAREREPO_GIT_DIR" config uploadpack.allowReachableSHA1InWant true && + git init local.git && + git -C local.git remote add origin "$BAREREPO_URL" && + test_must_fail git -C local.git -c protocol.version=$GIT_TEST_PROTOCOL_VERSION fetch "$BAREREPO_URL" $B +' diff --git a/t/t1419/test-0003-receive-pack-with-hide-refs-hook.sh b/t/t1419/test-0003-receive-pack-with-hide-refs-hook.sh new file mode 100644 index 00000000000..11f0f255b31 --- /dev/null +++ b/t/t1419/test-0003-receive-pack-with-hide-refs-hook.sh @@ -0,0 +1,40 @@ +#!/bin/sh + +# The upstream repository (bare_repo.git) contains the configurations: +# +# [transfer] hiderefs = hook: +# +# During the reference advertise phase the hide-refs hook will be invoked and all the refs will be checked by it + +test_expect_success "$PROTOCOL (protocol: $GIT_TEST_PROTOCOL_VERSION): push to main while hide-refs hook does not hide it" ' + write_script "$BAREREPO_GIT_DIR/hooks/hide-refs" <<-EOF && + test-tool hide-refs + EOF + create_commits_in work_repo E && + git -c protocol.version=$GIT_TEST_PROTOCOL_VERSION -C work_repo push origin HEAD:main >out 2>&1 && + make_user_friendly_and_stable_output out.tmp && + sed "s/$(get_abbrev_oid $E)[0-9a-f]*//g" actual && + format_and_save_expect <<-EOF && + To + .. HEAD -> main + EOF + test_cmp expect actual +' + +# If hide-refs hook hide some ref, git push will be rejected +test_expect_success "$PROTOCOL (protocol: $GIT_TEST_PROTOCOL_VERSION): push to main while hide-refs hook hide it" ' + write_script "$BAREREPO_GIT_DIR/hooks/hide-refs" <<-EOF && + test-tool hide-refs \ + -H "refs/heads/main" + EOF + create_commits_in work_repo F && + test_must_fail git -c protocol.version=$GIT_TEST_PROTOCOL_VERSION -C work_repo push origin HEAD:main >out 2>&1 && + make_user_friendly_and_stable_output out.tmp && + sed "s/$(get_abbrev_oid $E)[0-9a-f]*//g" actual && + format_and_save_expect <<-EOF && + To + ! [remote rejected] HEAD -> main (deny updating a hidden ref) + error: failed to push some refs to "" + EOF + test_cmp expect actual +' From patchwork Tue Sep 20 08:22:47 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sun Chao X-Patchwork-Id: 12981619 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 63300ECAAD8 for ; Tue, 20 Sep 2022 08:26:24 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230158AbiITI0C (ORCPT ); Tue, 20 Sep 2022 04:26:02 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47490 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230150AbiITIZ3 (ORCPT ); Tue, 20 Sep 2022 04:25:29 -0400 Received: from mail-wr1-x433.google.com (mail-wr1-x433.google.com [IPv6:2a00:1450:4864:20::433]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D401265564 for ; Tue, 20 Sep 2022 01:23:04 -0700 (PDT) Received: by mail-wr1-x433.google.com with SMTP id e16so2984666wrx.7 for ; Tue, 20 Sep 2022 01:23:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:from:to:cc:subject:date; bh=icPj4+n9T0s0E0oT3Ca0H/SF1i7z0HjmcAHYFYrnHyY=; b=S82NTOfqZWdCCY3Z3EYuVh3v5pJD9QBl8NTM1vWpqn3siUoChfTg+Uy4pD7w3d0PLZ 2/YOTHuyrjfyixzhuSBizbmo1ixuwqWQ+u73ngqlNhXuNi69QrUJ0ZG77x16qdXUpgFc DDlIQ2UvGD1e4MKiPWd4brkXfgKfEVSgsRxq/nqqItN6DRYrfiG+n8djzun/6rNxdwqC D+Y8rBiJSUdmTn40mDHa5k7enmWq7jEKnYKrGIRBA2Zy/KdYl9sFiaGGoOYNtpfabALw aMhbwdryoGb5VrhHKaC8OnJwrDLU5Zi7QWnXpB4Y3ON314LK4CPHtcy/MolDr+iVpj+i ffgg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:x-gm-message-state:from:to:cc :subject:date; bh=icPj4+n9T0s0E0oT3Ca0H/SF1i7z0HjmcAHYFYrnHyY=; b=nb+oe0Xrv8sOxmuSPDuLLbUN0x7pvYPFe3OsmRj+X9rIb3k6J7ooCI0pUDyNgIY24i Wv6pea+9gr3cEQWLH+4XGYjLKoMqngPdxj+nHoTfrkR+iqe5SWf2VqeO6283rAvRDmtD 12SibUr8VIL0bYCfTJp/gBrlu13KLfTQVpFBtx8IFbTguiNeEQxhZLtkytdRTiX7VdNv ZFtC4g/HuxLd1V9LpJBP8krX9m2YYhrPCU+TKT2LCsbWalva/VspJ+Z8ihs489LuL+9s b9UfM7LNvjCvoyBLHVBFwu6nWWsGhNGk/GXhqL0A9cy+LvomQlzX1Qsyghn8ZTYIBS1r pOFg== X-Gm-Message-State: ACrzQf1U3fHAqYdkmjgaCUOquuOb+ClWF1ZcHH441TiIbjBbXZuXn3Hg 29dGvAu/n7weamRyPgFpC64aTVOfd20= X-Google-Smtp-Source: AMsMyM7ivXSo69/XMWtlmZ5tyKQCWeXZTUzOyTPvxOvemsNwaURBbHsMj109NsqaRrLd9dys0VoSsQ== X-Received: by 2002:a05:6000:1541:b0:22a:3b77:6ef4 with SMTP id 1-20020a056000154100b0022a3b776ef4mr13112351wry.303.1663662173162; Tue, 20 Sep 2022 01:22:53 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id l36-20020a05600c1d2400b003a62400724bsm1690738wms.0.2022.09.20.01.22.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 20 Sep 2022 01:22:52 -0700 (PDT) Message-Id: <0cb019b28da7d9089ca122f8bf4a1a1312090712.1663662167.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Tue, 20 Sep 2022 08:22:47 +0000 Subject: [PATCH v6 5/5] doc: add documentation for the hide-refs hook Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Sun Chao <16657101987@163.com>, Sun Chao Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Sun Chao From: Sun Chao If uploadpack.allowTipSHA1InWant or uploadpack.allowReachableSHA1InWant is set to true, the private commits of hiding refs can be fetched by the client. The new "hide-refs" hook is used to hide our refs and we wish to hide the private commits either. "git-upload-pack" or "git-receive-pack" can use "hide-refs" hook to filter the references during the reference discovery phase. If a ref is hidden by the "hide-refs" hook, its private data cannot be fetched by the client even if uploadpack.allowTipSHA1InWant or uploadpack.allowReachableSHA1InWant is set to true. Signed-off-by: Sun Chao --- Documentation/githooks.txt | 49 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 49 insertions(+) diff --git a/Documentation/githooks.txt b/Documentation/githooks.txt index a16e62bc8c8..b26e50a4ea7 100644 --- a/Documentation/githooks.txt +++ b/Documentation/githooks.txt @@ -249,6 +249,55 @@ If this hook exits with a non-zero status, `git push` will abort without pushing anything. Information about why the push is rejected may be sent to the user by writing to standard error. +[[hide-refs]] +hide-refs +~~~~~~~~~ + +This hook would be invoked by 'git-receive-pack' and 'git-upload-pack' +during the reference discovery phase, each reference will be filtered +by this hook. The hook executes once with no arguments for each +'git-upload-pack' and 'git-receive-pack' process. Once the hook is invoked, +a version number and server process name ('uploadpack' or 'receive') will +send to it in pkt-line format, followed by a flush-pkt. The hook should +respond with its version number. + +During the reference discovery phase, each reference will be filtered by +this hook. In the following example, the letter 'G' stands for +'git-receive-pack' or 'git-upload-pack', and the letter 'H' stands for +this hook. The hook decides if the reference will be hidden or not, it +sends the result back in pkt-line format protocol, and a response 'hide' +means the references will be hidden to the client. + + # Version negotiation + G: PKT-LINE(version=1\0uploadpack) + G: flush-pkt + H: PKT-LINE(version=1) + H: flush-pkt + + # Send reference filter request to hook + G: PKT-LINE(ref :) + G: flush-pkt + + # Receive the result from the hook. + # Case 1: this reference is hidden + H: PKT-LINE(hide) + H: flush-pkt + + # Case 2: this reference can be advertised + H: flush-pkt + +To enable the `hide-refs` hook, we should config hiderefs with a `hook:` +option, e.g. if we want to pass all the refs to the new hook except for +the tags: + + git config --add transfer.hiderefs hook: + git config --add transfer.hiderefs hook:!refs/tags/ + +the `hide-refs` will be called during the reference discovery phase and +check each matched reference, a 'hide' response means the reference will +be hidden for its private data even if `allowTipSHA1InWant` or +`allowReachableSHA1InWant` is set to true. + [[pre-receive]] pre-receive ~~~~~~~~~~~