From patchwork Fri Nov 11 06:49:54 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13039748 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E39DDC4332F for ; Fri, 11 Nov 2022 06:50:05 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233019AbiKKGuF (ORCPT ); Fri, 11 Nov 2022 01:50:05 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39328 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230075AbiKKGuC (ORCPT ); Fri, 11 Nov 2022 01:50:02 -0500 Received: from wout1-smtp.messagingengine.com (wout1-smtp.messagingengine.com [64.147.123.24]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6A9366C73A for ; Thu, 10 Nov 2022 22:50:01 -0800 (PST) Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailout.west.internal (Postfix) with ESMTP id 7A9BB3200935; Fri, 11 Nov 2022 01:49:59 -0500 (EST) Received: from mailfrontend1 ([10.202.2.162]) by compute3.internal (MEProxy); Fri, 11 Nov 2022 01:49:59 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:date:date:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:sender:subject :subject:to:to; s=fm1; t=1668149399; x=1668235799; bh=QNI8Rbb5Tm XGQu3/3gYTspNT24g8aC846zBIrCsfWLc=; b=F/C6uZ3G/bdDkL1oYi1OsbFk0u tPTxOhRN9lzfWao3MQYXITuzsP1W3xRz7P0kjoXUlzVo58KU3Uzn5V1QjlmUOmdp D7cngox2ueOtK2zfpI15Xklj/Kbdj8KNdBIPSAGiM5O3glI0v3Z18jn1zu3urSzm ncbLCrV+8cW7ZWEGdZejiD0in6FXFY3mWnKjcidoTEI6cGsaQYfDdLtmoIwU8bkF 7xcZVSF/NsHI1c4NXxxP8gLP1kUXR1hGw11Fw8kr4ZrTX2e4XPjytggmKQlZtLSu lJHaC+ZE78mnAYTaNB20TIyl2qszypCkNIrDp2t5LHPWshx6vTIr4OdgYlAw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:date:date:feedback-id :feedback-id:from:from:in-reply-to:in-reply-to:message-id :mime-version:references:reply-to:sender:subject:subject:to:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm1; t=1668149399; x=1668235799; bh=QNI8Rbb5TmXGQu3/3gYTspNT24g8 aC846zBIrCsfWLc=; b=Mx/5y+8n0UtmcW1lO7xgJo/eR/Zi7oLA1XwMrrDGq2Z+ Vu3XNua+TKcGlQj82ygrqzlzny7fqixqOt9Suf4vyGeCfyd4ifA1aMPSG5sW2HMn m7/Q1L82iVJ3t8hXTDdl5dgIVWTdNlZKfONv55LkA4FuOES0ABHKIuQ8HFekS2Ko FWZFyOZBUm2Vvz5i7Ysh6QWlD4WuyjGeToC8L5/b6BPqpP5Juvsp917m+gTolCTE mYZ1YF8+VtECG+mTD4Xv3Knrh1m0VcU5t+EdCTdBoVpWjNX6wtJOxHPUcfpgFK0d qCZN3Ly7N/tqN2SNBB2K5jMMTddLcwfLQY+pLUzgXA== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvgedrfeehgddutddvucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucenucfjughrpeffhffvvefukfhfgggtuggjsehgtd erredttddvnecuhfhrohhmpefrrghtrhhitghkucfuthgvihhnhhgrrhguthcuoehpshes phhkshdrihhmqeenucggtffrrghtthgvrhhnpeeukedtvedtffevleejtefgheehieegke eluddvfeefgeehgfeltddtheejleffteenucevlhhushhtvghrufhiiigvpedtnecurfgr rhgrmhepmhgrihhlfhhrohhmpehpshesphhkshdrihhm X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Fri, 11 Nov 2022 01:49:57 -0500 (EST) Received: by pks.im (OpenSMTPD) with ESMTPSA id a092501c (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Fri, 11 Nov 2022 06:49:53 +0000 (UTC) Date: Fri, 11 Nov 2022 07:49:54 +0100 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Junio C Hamano , =?iso-8859-1?q?=C6var_Arnfj=F6r=F0?= Bjarmason , Taylor Blau , Jeff King Subject: [PATCH v5 1/7] refs: fix memory leak when parsing hideRefs config Message-ID: References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org When parsing the hideRefs configuration, we first duplicate the config value so that we can modify it. We then subsequently append it to the `hide_refs` string list, which is initialized with `strdup_strings` enabled. As a consequence we again reallocate the string, but never free the first duplicate and thus have a memory leak. While we never clean up the static `hide_refs` variable anyway, this is no excuse to make the leak worse by leaking every value twice. We are also about to change the way this variable will be handled so that we do indeed start to clean it up. So let's fix the memory leak by using the `string_list_append_nodup()` so that we pass ownership of the allocated string to `hide_refs`. Signed-off-by: Patrick Steinhardt --- refs.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/refs.c b/refs.c index 1491ae937e..a4ab264d74 100644 --- a/refs.c +++ b/refs.c @@ -1435,7 +1435,7 @@ int parse_hide_refs_config(const char *var, const char *value, const char *secti CALLOC_ARRAY(hide_refs, 1); hide_refs->strdup_strings = 1; } - string_list_append(hide_refs, ref); + string_list_append_nodup(hide_refs, ref); } return 0; } From patchwork Fri Nov 11 06:49:58 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13039750 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 34566C4332F for ; Fri, 11 Nov 2022 06:50:09 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233101AbiKKGuH (ORCPT ); Fri, 11 Nov 2022 01:50:07 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39348 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233098AbiKKGuE (ORCPT ); Fri, 11 Nov 2022 01:50:04 -0500 Received: from wout1-smtp.messagingengine.com (wout1-smtp.messagingengine.com [64.147.123.24]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DCE7472982 for ; Thu, 10 Nov 2022 22:50:02 -0800 (PST) Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailout.west.internal (Postfix) with ESMTP id 1CA2432008C0; Fri, 11 Nov 2022 01:50:02 -0500 (EST) Received: from mailfrontend1 ([10.202.2.162]) by compute1.internal (MEProxy); Fri, 11 Nov 2022 01:50:02 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:date:date:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:sender:subject :subject:to:to; s=fm1; t=1668149401; x=1668235801; bh=Jj/ydwjw7k QnY1ggaO7wOSbplvDEsgF7N3C4AoYwBH8=; b=qUIzj+mPyoPFHD6d60+IUEXZ59 NIpeAzLgXh+RoVaXgOGRqlyvRZm0Nw17cYlOu7GRhOn3FiJ5ULb+mc1xoRnnmi/j 0Ry5STWnE9kQkND9rNksxSsTSB97FDk8Op5hfgmTlh8YdiDmzJ4Qw7hAz4iZpvxH NWtPZ5YJtZpna4ULhZoPdG2uyq1qyC/hAHZr0VYGat3lxRzzwwDOLd2Fhf1Uudzd XmIWSCCBlqbBFSPpqxmYF/YclIZVRgr/nkuUmZJdgNvocmHij/yXtWZVcpASTDNs YQ4HNjIx6srirGwk6IzD4wd4oVWdlfV334anhfT6EwD+fLKWKWSuomBZ4gOQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:date:date:feedback-id :feedback-id:from:from:in-reply-to:in-reply-to:message-id :mime-version:references:reply-to:sender:subject:subject:to:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm1; t=1668149401; x=1668235801; bh=Jj/ydwjw7kQnY1ggaO7wOSbplvDE sgF7N3C4AoYwBH8=; b=FWIZ8qtx3kLXmkvKY4euLBig3ySnh+6zyNN2JpRDaRPC HlTy77y+Ict8YHBz4fzY1b3vaqsG6wTn1bOg9ApY9mUjso4RHBK1sKuSoRUSpBAM FriudloBBJ+ILikcBqSmeFGj0OyaF+delwialK6RLxaz/Blm7bQLOIFGYPAjhZMK MLOAolPvmDEYSX6DXP4Uu6nOEQJaj/9upOx/gFDfg5s48S0PfkPpiZp7NFamhC2O gGqQ0/u0tuTaO1sp5Gkvj6aAa9k7ZogMf/cqC7buGY+oAuGDzFDWGd994DIMDiwJ ZfzAZklr7f+fVQ/GcbKfjSZAHmEVZrd1RFBFMsEk4Q== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvgedrfeehgddutddvucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucenucfjughrpeffhffvvefukfhfgggtuggjsehgtd erredttddvnecuhfhrohhmpefrrghtrhhitghkucfuthgvihhnhhgrrhguthcuoehpshes phhkshdrihhmqeenucggtffrrghtthgvrhhnpeeukedtvedtffevleejtefgheehieegke eluddvfeefgeehgfeltddtheejleffteenucevlhhushhtvghrufhiiigvpedtnecurfgr rhgrmhepmhgrihhlfhhrohhmpehpshesphhkshdrihhm X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Fri, 11 Nov 2022 01:50:00 -0500 (EST) Received: by pks.im (OpenSMTPD) with ESMTPSA id 84b8696b (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Fri, 11 Nov 2022 06:49:58 +0000 (UTC) Date: Fri, 11 Nov 2022 07:49:58 +0100 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Junio C Hamano , =?iso-8859-1?q?=C6var_Arnfj=F6r=F0?= Bjarmason , Taylor Blau , Jeff King Subject: [PATCH v5 2/7] refs: get rid of global list of hidden refs Message-ID: References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org We're about to add a new argument to git-rev-list(1) that allows it to add all references that are visible when taking `transfer.hideRefs` et al into account. This will require us to potentially parse multiple sets of hidden refs, which is not easily possible right now as there is only a single, global instance of the list of parsed hidden refs. Refactor `parse_hide_refs_config()` and `ref_is_hidden()` so that both take the list of hidden references as input and adjust callers to keep a local list, instead. This allows us to easily use multiple hidden-ref lists. Furthermore, it allows us to properly free this list before we exit. Signed-off-by: Patrick Steinhardt --- builtin/receive-pack.c | 8 +++++--- ls-refs.c | 13 +++++++++---- refs.c | 14 ++++---------- refs.h | 5 +++-- upload-pack.c | 30 ++++++++++++++++++------------ 5 files changed, 39 insertions(+), 31 deletions(-) diff --git a/builtin/receive-pack.c b/builtin/receive-pack.c index 44bcea3a5b..1e24b31a0a 100644 --- a/builtin/receive-pack.c +++ b/builtin/receive-pack.c @@ -80,6 +80,7 @@ static struct object_id push_cert_oid; static struct signature_check sigcheck; static const char *push_cert_nonce; static const char *cert_nonce_seed; +static struct string_list hidden_refs = STRING_LIST_INIT_DUP; static const char *NONCE_UNSOLICITED = "UNSOLICITED"; static const char *NONCE_BAD = "BAD"; @@ -130,7 +131,7 @@ static enum deny_action parse_deny_action(const char *var, const char *value) static int receive_pack_config(const char *var, const char *value, void *cb) { - int status = parse_hide_refs_config(var, value, "receive"); + int status = parse_hide_refs_config(var, value, "receive", &hidden_refs); if (status) return status; @@ -296,7 +297,7 @@ static int show_ref_cb(const char *path_full, const struct object_id *oid, struct oidset *seen = data; const char *path = strip_namespace(path_full); - if (ref_is_hidden(path, path_full)) + if (ref_is_hidden(path, path_full, &hidden_refs)) return 0; /* @@ -1794,7 +1795,7 @@ static void reject_updates_to_hidden(struct command *commands) strbuf_setlen(&refname_full, prefix_len); strbuf_addstr(&refname_full, cmd->ref_name); - if (!ref_is_hidden(cmd->ref_name, refname_full.buf)) + if (!ref_is_hidden(cmd->ref_name, refname_full.buf, &hidden_refs)) continue; if (is_null_oid(&cmd->new_oid)) cmd->error_string = "deny deleting a hidden ref"; @@ -2591,6 +2592,7 @@ int cmd_receive_pack(int argc, const char **argv, const char *prefix) packet_flush(1); oid_array_clear(&shallow); oid_array_clear(&ref); + string_list_clear(&hidden_refs, 0); free((void *)push_cert_nonce); return 0; } diff --git a/ls-refs.c b/ls-refs.c index fa0d01b47c..fb6769742c 100644 --- a/ls-refs.c +++ b/ls-refs.c @@ -6,6 +6,7 @@ #include "ls-refs.h" #include "pkt-line.h" #include "config.h" +#include "string-list.h" static int config_read; static int advertise_unborn; @@ -73,6 +74,7 @@ struct ls_refs_data { unsigned symrefs; struct strvec prefixes; struct strbuf buf; + struct string_list hidden_refs; unsigned unborn : 1; }; @@ -84,7 +86,7 @@ static int send_ref(const char *refname, const struct object_id *oid, strbuf_reset(&data->buf); - if (ref_is_hidden(refname_nons, refname)) + if (ref_is_hidden(refname_nons, refname, &data->hidden_refs)) return 0; if (!ref_match(&data->prefixes, refname_nons)) @@ -137,14 +139,15 @@ static void send_possibly_unborn_head(struct ls_refs_data *data) } static int ls_refs_config(const char *var, const char *value, - void *data UNUSED) + void *cb_data) { + struct ls_refs_data *data = cb_data; /* * We only serve fetches over v2 for now, so respect only "uploadpack" * config. This may need to eventually be expanded to "receive", but we * don't yet know how that information will be passed to ls-refs. */ - return parse_hide_refs_config(var, value, "uploadpack"); + return parse_hide_refs_config(var, value, "uploadpack", &data->hidden_refs); } int ls_refs(struct repository *r, struct packet_reader *request) @@ -154,9 +157,10 @@ int ls_refs(struct repository *r, struct packet_reader *request) memset(&data, 0, sizeof(data)); strvec_init(&data.prefixes); strbuf_init(&data.buf, 0); + string_list_init_dup(&data.hidden_refs); ensure_config_read(); - git_config(ls_refs_config, NULL); + git_config(ls_refs_config, &data); while (packet_reader_read(request) == PACKET_READ_NORMAL) { const char *arg = request->line; @@ -195,6 +199,7 @@ int ls_refs(struct repository *r, struct packet_reader *request) packet_fflush(stdout); strvec_clear(&data.prefixes); strbuf_release(&data.buf); + string_list_clear(&data.hidden_refs, 0); return 0; } diff --git a/refs.c b/refs.c index a4ab264d74..2c7e88b190 100644 --- a/refs.c +++ b/refs.c @@ -1414,9 +1414,8 @@ char *shorten_unambiguous_ref(const char *refname, int strict) refname, strict); } -static struct string_list *hide_refs; - -int parse_hide_refs_config(const char *var, const char *value, const char *section) +int parse_hide_refs_config(const char *var, const char *value, const char *section, + struct string_list *hide_refs) { const char *key; if (!strcmp("transfer.hiderefs", var) || @@ -1431,21 +1430,16 @@ int parse_hide_refs_config(const char *var, const char *value, const char *secti len = strlen(ref); while (len && ref[len - 1] == '/') ref[--len] = '\0'; - if (!hide_refs) { - CALLOC_ARRAY(hide_refs, 1); - hide_refs->strdup_strings = 1; - } string_list_append_nodup(hide_refs, ref); } return 0; } -int ref_is_hidden(const char *refname, const char *refname_full) +int ref_is_hidden(const char *refname, const char *refname_full, + const struct string_list *hide_refs) { int i; - if (!hide_refs) - return 0; for (i = hide_refs->nr - 1; i >= 0; i--) { const char *match = hide_refs->items[i].string; const char *subject; diff --git a/refs.h b/refs.h index 8958717a17..3266fd8f57 100644 --- a/refs.h +++ b/refs.h @@ -808,7 +808,8 @@ int update_ref(const char *msg, const char *refname, const struct object_id *new_oid, const struct object_id *old_oid, unsigned int flags, enum action_on_err onerr); -int parse_hide_refs_config(const char *var, const char *value, const char *); +int parse_hide_refs_config(const char *var, const char *value, const char *, + struct string_list *); /* * Check whether a ref is hidden. If no namespace is set, both the first and @@ -818,7 +819,7 @@ int parse_hide_refs_config(const char *var, const char *value, const char *); * the ref is outside that namespace, the first parameter is NULL. The second * parameter always points to the full ref name. */ -int ref_is_hidden(const char *, const char *); +int ref_is_hidden(const char *, const char *, const struct string_list *); /* Is this a per-worktree ref living in the refs/ namespace? */ int is_per_worktree_ref(const char *refname); diff --git a/upload-pack.c b/upload-pack.c index 0b8311bd68..551f22ffa5 100644 --- a/upload-pack.c +++ b/upload-pack.c @@ -62,6 +62,7 @@ struct upload_pack_data { struct object_array have_obj; struct oid_array haves; /* v2 only */ struct string_list wanted_refs; /* v2 only */ + struct string_list hidden_refs; struct object_array shallows; struct string_list deepen_not; @@ -118,6 +119,7 @@ static void upload_pack_data_init(struct upload_pack_data *data) { struct string_list symref = STRING_LIST_INIT_DUP; struct string_list wanted_refs = STRING_LIST_INIT_DUP; + struct string_list hidden_refs = STRING_LIST_INIT_DUP; struct object_array want_obj = OBJECT_ARRAY_INIT; struct object_array have_obj = OBJECT_ARRAY_INIT; struct oid_array haves = OID_ARRAY_INIT; @@ -130,6 +132,7 @@ static void upload_pack_data_init(struct upload_pack_data *data) memset(data, 0, sizeof(*data)); data->symref = symref; data->wanted_refs = wanted_refs; + data->hidden_refs = hidden_refs; data->want_obj = want_obj; data->have_obj = have_obj; data->haves = haves; @@ -151,6 +154,7 @@ static void upload_pack_data_clear(struct upload_pack_data *data) { string_list_clear(&data->symref, 1); string_list_clear(&data->wanted_refs, 1); + string_list_clear(&data->hidden_refs, 0); object_array_clear(&data->want_obj); object_array_clear(&data->have_obj); oid_array_clear(&data->haves); @@ -842,8 +846,8 @@ static void deepen(struct upload_pack_data *data, int depth) * Checking for reachable shallows requires that our refs be * marked with OUR_REF. */ - head_ref_namespaced(check_ref, NULL); - for_each_namespaced_ref(check_ref, NULL); + head_ref_namespaced(check_ref, data); + for_each_namespaced_ref(check_ref, data); get_reachable_list(data, &reachable_shallows); result = get_shallow_commits(&reachable_shallows, @@ -1158,11 +1162,11 @@ static void receive_needs(struct upload_pack_data *data, /* return non-zero if the ref is hidden, otherwise 0 */ static int mark_our_ref(const char *refname, const char *refname_full, - const struct object_id *oid) + const struct object_id *oid, const struct string_list *hidden_refs) { struct object *o = lookup_unknown_object(the_repository, oid); - if (ref_is_hidden(refname, refname_full)) { + if (ref_is_hidden(refname, refname_full, hidden_refs)) { o->flags |= HIDDEN_REF; return 1; } @@ -1171,11 +1175,12 @@ static int mark_our_ref(const char *refname, const char *refname_full, } static int check_ref(const char *refname_full, const struct object_id *oid, - int flag UNUSED, void *cb_data UNUSED) + int flag UNUSED, void *cb_data) { const char *refname = strip_namespace(refname_full); + struct upload_pack_data *data = cb_data; - mark_our_ref(refname, refname_full, oid); + mark_our_ref(refname, refname_full, oid, &data->hidden_refs); return 0; } @@ -1204,7 +1209,7 @@ static int send_ref(const char *refname, const struct object_id *oid, struct object_id peeled; struct upload_pack_data *data = cb_data; - if (mark_our_ref(refname_nons, refname, oid)) + if (mark_our_ref(refname_nons, refname, oid, &data->hidden_refs)) return 0; if (capabilities) { @@ -1327,7 +1332,7 @@ static int upload_pack_config(const char *var, const char *value, void *cb_data) if (parse_object_filter_config(var, value, data) < 0) return -1; - return parse_hide_refs_config(var, value, "uploadpack"); + return parse_hide_refs_config(var, value, "uploadpack", &data->hidden_refs); } static int upload_pack_protected_config(const char *var, const char *value, void *cb_data) @@ -1375,8 +1380,8 @@ void upload_pack(const int advertise_refs, const int stateless_rpc, advertise_shallow_grafts(1); packet_flush(1); } else { - head_ref_namespaced(check_ref, NULL); - for_each_namespaced_ref(check_ref, NULL); + head_ref_namespaced(check_ref, &data); + for_each_namespaced_ref(check_ref, &data); } if (!advertise_refs) { @@ -1441,6 +1446,7 @@ static int parse_want(struct packet_writer *writer, const char *line, static int parse_want_ref(struct packet_writer *writer, const char *line, struct string_list *wanted_refs, + struct string_list *hidden_refs, struct object_array *want_obj) { const char *refname_nons; @@ -1451,7 +1457,7 @@ static int parse_want_ref(struct packet_writer *writer, const char *line, struct strbuf refname = STRBUF_INIT; strbuf_addf(&refname, "%s%s", get_git_namespace(), refname_nons); - if (ref_is_hidden(refname_nons, refname.buf) || + if (ref_is_hidden(refname_nons, refname.buf, hidden_refs) || read_ref(refname.buf, &oid)) { packet_writer_error(writer, "unknown ref %s", refname_nons); die("unknown ref %s", refname_nons); @@ -1508,7 +1514,7 @@ static void process_args(struct packet_reader *request, continue; if (data->allow_ref_in_want && parse_want_ref(&data->writer, arg, &data->wanted_refs, - &data->want_obj)) + &data->hidden_refs, &data->want_obj)) continue; /* process have line */ if (parse_have(arg, &data->haves)) From patchwork Fri Nov 11 06:50:02 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13039751 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 386C9C4332F for ; Fri, 11 Nov 2022 06:50:13 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233111AbiKKGuL (ORCPT ); Fri, 11 Nov 2022 01:50:11 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39380 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232784AbiKKGuH (ORCPT ); Fri, 11 Nov 2022 01:50:07 -0500 Received: from wout1-smtp.messagingengine.com (wout1-smtp.messagingengine.com [64.147.123.24]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 784F272989 for ; Thu, 10 Nov 2022 22:50:06 -0800 (PST) Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailout.west.internal (Postfix) with ESMTP id ADEBD3200938; Fri, 11 Nov 2022 01:50:05 -0500 (EST) Received: from mailfrontend1 ([10.202.2.162]) by compute3.internal (MEProxy); Fri, 11 Nov 2022 01:50:06 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:date:date:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:sender:subject :subject:to:to; s=fm1; t=1668149405; x=1668235805; bh=4PqcJpHMSp wNyuIi+2ZF/4YiCLW6D8Sgw1/clclBzfs=; b=a33P3WbJfXopiuQvhbyRLCprC0 0n5wsqnT3L6hlpF2t92I55rWNnQOfqvOESWCD6iyapZlNnmXp+Mzw2UMAqv16oUW Znx0VT6h24CSDMNKiC0/tAcN6SkGdx8qjxKMiZGwkicBO+LZODr71IHSaHUdZIYm ClzXTcVjbwVOk7UP8qXcPZcxEJcbDFuoYAyhbCe7OJIGDrrTvz2A1/lNEWGva9SI I3JlHkMjxKMbpsJ1uxpbTraBm3tq4JKn+g3p2IcYpcdZBfHbz3P2rGMLecBL1JSl JeEiky2BSjdqYaW9Mpu5xRa2VdleMCNPPUwxw0KdT6DN9HWhU0fI1vNwNFRA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:date:date:feedback-id :feedback-id:from:from:in-reply-to:in-reply-to:message-id :mime-version:references:reply-to:sender:subject:subject:to:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm1; t=1668149405; x=1668235805; bh=4PqcJpHMSpwNyuIi+2ZF/4YiCLW6 D8Sgw1/clclBzfs=; b=tHJqAURhoPyVoh4CdtdQIV3J5yD+NG69e2+nAtR8vY3Z AuDZZoqWjimRwnNPk2/fBOPd6BVM/JIfMp7FxI49m2Dkzb+wSEOO4YDphMLYm/6p rO6V6I/K6DNFARbtwWUzkdKGm1rLoghK5a6VvA/11/2uuhigWusig6S+qiSIc3Mb yr2QHGTW5lEO6x99NPjj+FzBVxeJN9dSeN9GSOe9ZInndhZzuP1p7YlPN5aDmr0Z 6IvNtL+ryGf9D1uumfw+tcsotB4C+6+W7pQbMFs3aTBkS1XJ0JSS/IAkjanYuZli VUMSVdDeD9i2mCeP+UQgaYI1D6dhOKmNO5gSSh/+Dw== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvgedrfeehgddutddvucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucenucfjughrpeffhffvvefukfhfgggtuggjsehgtd erredttddvnecuhfhrohhmpefrrghtrhhitghkucfuthgvihhnhhgrrhguthcuoehpshes phhkshdrihhmqeenucggtffrrghtthgvrhhnpeeukedtvedtffevleejtefgheehieegke eluddvfeefgeehgfeltddtheejleffteenucevlhhushhtvghrufhiiigvpedunecurfgr rhgrmhepmhgrihhlfhhrohhmpehpshesphhkshdrihhm X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Fri, 11 Nov 2022 01:50:03 -0500 (EST) Received: by pks.im (OpenSMTPD) with ESMTPSA id f196d6d8 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Fri, 11 Nov 2022 06:50:02 +0000 (UTC) Date: Fri, 11 Nov 2022 07:50:02 +0100 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Junio C Hamano , =?iso-8859-1?q?=C6var_Arnfj=F6r=F0?= Bjarmason , Taylor Blau , Jeff King Subject: [PATCH v5 3/7] revision: move together exclusion-related functions Message-ID: <93a627fb7f79610f67a7eefb29aac6191ea201b0.1668149149.git.ps@pks.im> References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Move together the definitions of functions that handle exclusions of refs so that related functionality sits in a single place, only. Signed-off-by: Patrick Steinhardt --- revision.c | 52 ++++++++++++++++++++++++++-------------------------- 1 file changed, 26 insertions(+), 26 deletions(-) diff --git a/revision.c b/revision.c index 0760e78936..be755670e2 100644 --- a/revision.c +++ b/revision.c @@ -1517,14 +1517,6 @@ static void add_rev_cmdline_list(struct rev_info *revs, } } -struct all_refs_cb { - int all_flags; - int warned_bad_reflog; - struct rev_info *all_revs; - const char *name_for_errormsg; - struct worktree *wt; -}; - int ref_excluded(struct string_list *ref_excludes, const char *path) { struct string_list_item *item; @@ -1538,6 +1530,32 @@ int ref_excluded(struct string_list *ref_excludes, const char *path) return 0; } +void clear_ref_exclusion(struct string_list **ref_excludes_p) +{ + if (*ref_excludes_p) { + string_list_clear(*ref_excludes_p, 0); + free(*ref_excludes_p); + } + *ref_excludes_p = NULL; +} + +void add_ref_exclusion(struct string_list **ref_excludes_p, const char *exclude) +{ + if (!*ref_excludes_p) { + CALLOC_ARRAY(*ref_excludes_p, 1); + (*ref_excludes_p)->strdup_strings = 1; + } + string_list_append(*ref_excludes_p, exclude); +} + +struct all_refs_cb { + int all_flags; + int warned_bad_reflog; + struct rev_info *all_revs; + const char *name_for_errormsg; + struct worktree *wt; +}; + static int handle_one_ref(const char *path, const struct object_id *oid, int flag UNUSED, void *cb_data) @@ -1563,24 +1581,6 @@ static void init_all_refs_cb(struct all_refs_cb *cb, struct rev_info *revs, cb->wt = NULL; } -void clear_ref_exclusion(struct string_list **ref_excludes_p) -{ - if (*ref_excludes_p) { - string_list_clear(*ref_excludes_p, 0); - free(*ref_excludes_p); - } - *ref_excludes_p = NULL; -} - -void add_ref_exclusion(struct string_list **ref_excludes_p, const char *exclude) -{ - if (!*ref_excludes_p) { - CALLOC_ARRAY(*ref_excludes_p, 1); - (*ref_excludes_p)->strdup_strings = 1; - } - string_list_append(*ref_excludes_p, exclude); -} - static void handle_refs(struct ref_store *refs, struct rev_info *revs, unsigned flags, int (*for_each)(struct ref_store *, each_ref_fn, void *)) From patchwork Fri Nov 11 06:50:06 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13039752 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8576FC43219 for ; Fri, 11 Nov 2022 06:50:27 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233170AbiKKGu0 (ORCPT ); Fri, 11 Nov 2022 01:50:26 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39686 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233098AbiKKGuW (ORCPT ); Fri, 11 Nov 2022 01:50:22 -0500 Received: from wout1-smtp.messagingengine.com (wout1-smtp.messagingengine.com [64.147.123.24]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0A8E4742C7 for ; Thu, 10 Nov 2022 22:50:11 -0800 (PST) Received: from compute5.internal (compute5.nyi.internal [10.202.2.45]) by mailout.west.internal (Postfix) with ESMTP id 41CDA32008C3; Fri, 11 Nov 2022 01:50:10 -0500 (EST) Received: from mailfrontend1 ([10.202.2.162]) by compute5.internal (MEProxy); Fri, 11 Nov 2022 01:50:10 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:date:date:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:sender:subject :subject:to:to; s=fm1; t=1668149409; x=1668235809; bh=LocJ8dfo9C auVuLbVRV26U2s6/04fz2AQstx0lmlE+M=; b=vvMhj/5EwYsCQ5x8DGdfQp2LYo EkTVfy2IWAA0KlBJ94XwbrA4uwKESCqPpDeHF+k0m3X7AveaDBVRzeu0sBiAYMx7 OA9HJsOYwHGK83YJmp4aXCkyzY6KJyJ8VU4kqgUAKKYQ+dFtjQrEOBAhL/R9x0em DEDg3FNny61SteX5aYRPPQhJ+8Y+bB474i7IQdEg7LvCpw7BpP3OouCnuJitU6bh OVrUlxyzduQeRZOiDGYK1Uv83eAI/kgEYEdsDWSELJaQlr2qFKnOHCGvzsn8pglB j/SB5nkz36Es+7gKCHZBM37QrzW8mIpqwXwn6yX8UMyGzXxCq+UxaclSGVEA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:date:date:feedback-id :feedback-id:from:from:in-reply-to:in-reply-to:message-id :mime-version:references:reply-to:sender:subject:subject:to:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm1; t=1668149409; x=1668235809; bh=LocJ8dfo9CauVuLbVRV26U2s6/04 fz2AQstx0lmlE+M=; b=RgsY3jOfgClJJCkMByq5ZvToQ7TdeDBboDl8mMQRSQGk QNYc2H2Kq5YawKwDfUh6bNlVpL5Hxrug8t2MjnoAYU4eLyLfBR5YrdoJxbue7xyy ibSkfQrMJN7pOx96644jUd94kwJNIAfu6ZqvYJc2hf/r6hg1LsNxZ3PhUrL6ql+G a/tIKP2RS3X1pt+IwhOVJuP3DcYUa8poa05l7F6Ap9hJF1auvn8JOKLDCKIfHbbo iaua2zCOryxVFqMBYYr4XUjaTZmcEPNzvhyFGlafm04U4S6mB/QxSpkuivshKm/u NcsNmxSC2Ywofh9Ip+7oXF75m0ElwCt1HvAhF70r0Q== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvgedrfeehgddutddvucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucenucfjughrpeffhffvvefukfhfgggtuggjsehgtd erredttddvnecuhfhrohhmpefrrghtrhhitghkucfuthgvihhnhhgrrhguthcuoehpshes phhkshdrihhmqeenucggtffrrghtthgvrhhnpeeukedtvedtffevleejtefgheehieegke eluddvfeefgeehgfeltddtheejleffteenucevlhhushhtvghrufhiiigvpedtnecurfgr rhgrmhepmhgrihhlfhhrohhmpehpshesphhkshdrihhm X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Fri, 11 Nov 2022 01:50:08 -0500 (EST) Received: by pks.im (OpenSMTPD) with ESMTPSA id bc112d80 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Fri, 11 Nov 2022 06:50:06 +0000 (UTC) Date: Fri, 11 Nov 2022 07:50:06 +0100 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Junio C Hamano , =?iso-8859-1?q?=C6var_Arnfj=F6r=F0?= Bjarmason , Taylor Blau , Jeff King Subject: [PATCH v5 4/7] revision: introduce struct to handle exclusions Message-ID: References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org The functions that handle exclusion of refs work on a single string list. We're about to add a second mechanism for excluding refs though, and it makes sense to reuse much of the same architecture for both kinds of exclusion. Introduce a new `struct ref_exclusions` that encapsulates all the logic related to excluding refs and move the `struct string_list` that holds all wildmatch patterns of excluded refs into it. Rename functions that operate on this struct to match its name. Signed-off-by: Patrick Steinhardt --- builtin/rev-parse.c | 8 ++++---- revision.c | 48 +++++++++++++++++++++------------------------ revision.h | 27 +++++++++++++++++++------ 3 files changed, 47 insertions(+), 36 deletions(-) diff --git a/builtin/rev-parse.c b/builtin/rev-parse.c index 8f61050bde..7fa5b6991b 100644 --- a/builtin/rev-parse.c +++ b/builtin/rev-parse.c @@ -39,7 +39,7 @@ static int abbrev_ref_strict; static int output_sq; static int stuck_long; -static struct string_list *ref_excludes; +static struct ref_exclusions ref_excludes = REF_EXCLUSIONS_INIT; /* * Some arguments are relevant "revision" arguments, @@ -198,7 +198,7 @@ static int show_default(void) static int show_reference(const char *refname, const struct object_id *oid, int flag UNUSED, void *cb_data UNUSED) { - if (ref_excluded(ref_excludes, refname)) + if (ref_excluded(&ref_excludes, refname)) return 0; show_rev(NORMAL, oid, refname); return 0; @@ -585,7 +585,7 @@ static void handle_ref_opt(const char *pattern, const char *prefix) for_each_glob_ref_in(show_reference, pattern, prefix, NULL); else for_each_ref_in(prefix, show_reference, NULL); - clear_ref_exclusion(&ref_excludes); + clear_ref_exclusions(&ref_excludes); } enum format_type { @@ -863,7 +863,7 @@ int cmd_rev_parse(int argc, const char **argv, const char *prefix) } if (!strcmp(arg, "--all")) { for_each_ref(show_reference, NULL); - clear_ref_exclusion(&ref_excludes); + clear_ref_exclusions(&ref_excludes); continue; } if (skip_prefix(arg, "--disambiguate=", &arg)) { diff --git a/revision.c b/revision.c index be755670e2..fe3ec98f46 100644 --- a/revision.c +++ b/revision.c @@ -1517,35 +1517,30 @@ static void add_rev_cmdline_list(struct rev_info *revs, } } -int ref_excluded(struct string_list *ref_excludes, const char *path) +int ref_excluded(const struct ref_exclusions *exclusions, const char *path) { struct string_list_item *item; - - if (!ref_excludes) - return 0; - for_each_string_list_item(item, ref_excludes) { + for_each_string_list_item(item, &exclusions->excluded_refs) { if (!wildmatch(item->string, path, 0)) return 1; } return 0; } -void clear_ref_exclusion(struct string_list **ref_excludes_p) +void init_ref_exclusions(struct ref_exclusions *exclusions) { - if (*ref_excludes_p) { - string_list_clear(*ref_excludes_p, 0); - free(*ref_excludes_p); - } - *ref_excludes_p = NULL; + struct ref_exclusions blank = REF_EXCLUSIONS_INIT; + memcpy(exclusions, &blank, sizeof(*exclusions)); } -void add_ref_exclusion(struct string_list **ref_excludes_p, const char *exclude) +void clear_ref_exclusions(struct ref_exclusions *exclusions) { - if (!*ref_excludes_p) { - CALLOC_ARRAY(*ref_excludes_p, 1); - (*ref_excludes_p)->strdup_strings = 1; - } - string_list_append(*ref_excludes_p, exclude); + string_list_clear(&exclusions->excluded_refs, 0); +} + +void add_ref_exclusion(struct ref_exclusions *exclusions, const char *exclude) +{ + string_list_append(&exclusions->excluded_refs, exclude); } struct all_refs_cb { @@ -1563,7 +1558,7 @@ static int handle_one_ref(const char *path, const struct object_id *oid, struct all_refs_cb *cb = cb_data; struct object *object; - if (ref_excluded(cb->all_revs->ref_excludes, path)) + if (ref_excluded(&cb->all_revs->ref_excludes, path)) return 0; object = get_reference(cb->all_revs, path, oid, cb->all_flags); @@ -1901,6 +1896,7 @@ void repo_init_revisions(struct repository *r, init_display_notes(&revs->notes_opt); list_objects_filter_init(&revs->filter); + init_ref_exclusions(&revs->ref_excludes); } static void add_pending_commit_list(struct rev_info *revs, @@ -2689,10 +2685,10 @@ static int handle_revision_pseudo_opt(struct rev_info *revs, init_all_refs_cb(&cb, revs, *flags); other_head_refs(handle_one_ref, &cb); } - clear_ref_exclusion(&revs->ref_excludes); + clear_ref_exclusions(&revs->ref_excludes); } else if (!strcmp(arg, "--branches")) { handle_refs(refs, revs, *flags, refs_for_each_branch_ref); - clear_ref_exclusion(&revs->ref_excludes); + clear_ref_exclusions(&revs->ref_excludes); } else if (!strcmp(arg, "--bisect")) { read_bisect_terms(&term_bad, &term_good); handle_refs(refs, revs, *flags, for_each_bad_bisect_ref); @@ -2701,15 +2697,15 @@ static int handle_revision_pseudo_opt(struct rev_info *revs, revs->bisect = 1; } else if (!strcmp(arg, "--tags")) { handle_refs(refs, revs, *flags, refs_for_each_tag_ref); - clear_ref_exclusion(&revs->ref_excludes); + clear_ref_exclusions(&revs->ref_excludes); } else if (!strcmp(arg, "--remotes")) { handle_refs(refs, revs, *flags, refs_for_each_remote_ref); - clear_ref_exclusion(&revs->ref_excludes); + clear_ref_exclusions(&revs->ref_excludes); } else if ((argcount = parse_long_opt("glob", argv, &optarg))) { struct all_refs_cb cb; init_all_refs_cb(&cb, revs, *flags); for_each_glob_ref(handle_one_ref, optarg, &cb); - clear_ref_exclusion(&revs->ref_excludes); + clear_ref_exclusions(&revs->ref_excludes); return argcount; } else if ((argcount = parse_long_opt("exclude", argv, &optarg))) { add_ref_exclusion(&revs->ref_excludes, optarg); @@ -2718,17 +2714,17 @@ static int handle_revision_pseudo_opt(struct rev_info *revs, struct all_refs_cb cb; init_all_refs_cb(&cb, revs, *flags); for_each_glob_ref_in(handle_one_ref, optarg, "refs/heads/", &cb); - clear_ref_exclusion(&revs->ref_excludes); + clear_ref_exclusions(&revs->ref_excludes); } else if (skip_prefix(arg, "--tags=", &optarg)) { struct all_refs_cb cb; init_all_refs_cb(&cb, revs, *flags); for_each_glob_ref_in(handle_one_ref, optarg, "refs/tags/", &cb); - clear_ref_exclusion(&revs->ref_excludes); + clear_ref_exclusions(&revs->ref_excludes); } else if (skip_prefix(arg, "--remotes=", &optarg)) { struct all_refs_cb cb; init_all_refs_cb(&cb, revs, *flags); for_each_glob_ref_in(handle_one_ref, optarg, "refs/remotes/", &cb); - clear_ref_exclusion(&revs->ref_excludes); + clear_ref_exclusions(&revs->ref_excludes); } else if (!strcmp(arg, "--reflog")) { add_reflogs_to_pending(revs, *flags); } else if (!strcmp(arg, "--indexed-objects")) { diff --git a/revision.h b/revision.h index afe1b77985..5c8ab16047 100644 --- a/revision.h +++ b/revision.h @@ -81,6 +81,21 @@ struct rev_cmdline_info { } *rev; }; +struct ref_exclusions { + /* + * Excluded refs is a list of wildmatch patterns. If any of the + * patterns matches, the reference will be excluded. + */ + struct string_list excluded_refs; +}; + +/** + * Initialize a `struct ref_exclusions` with a macro. + */ +#define REF_EXCLUSIONS_INIT { \ + .excluded_refs = STRING_LIST_INIT_DUP, \ +} + struct oidset; struct topo_walk_info; @@ -103,7 +118,7 @@ struct rev_info { struct list_objects_filter_options filter; /* excluding from --branches, --refs, etc. expansion */ - struct string_list *ref_excludes; + struct ref_exclusions ref_excludes; /* Basic information */ const char *prefix; @@ -439,12 +454,12 @@ void mark_trees_uninteresting_sparse(struct repository *r, struct oidset *trees) void show_object_with_name(FILE *, struct object *, const char *); /** - * Helpers to check if a "struct string_list" item matches with - * wildmatch(). + * Helpers to check if a reference should be excluded. */ -int ref_excluded(struct string_list *, const char *path); -void clear_ref_exclusion(struct string_list **); -void add_ref_exclusion(struct string_list **, const char *exclude); +int ref_excluded(const struct ref_exclusions *exclusions, const char *path); +void init_ref_exclusions(struct ref_exclusions *); +void clear_ref_exclusions(struct ref_exclusions *); +void add_ref_exclusion(struct ref_exclusions *, const char *exclude); /** * This function can be used if you want to add commit objects as revision From patchwork Fri Nov 11 06:50:10 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13039753 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B1623C433FE for ; Fri, 11 Nov 2022 06:50:34 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233117AbiKKGud (ORCPT ); Fri, 11 Nov 2022 01:50:33 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39742 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233116AbiKKGuZ (ORCPT ); Fri, 11 Nov 2022 01:50:25 -0500 Received: from wout1-smtp.messagingengine.com (wout1-smtp.messagingengine.com [64.147.123.24]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BD1E077238 for ; Thu, 10 Nov 2022 22:50:15 -0800 (PST) Received: from compute4.internal (compute4.nyi.internal [10.202.2.44]) by mailout.west.internal (Postfix) with ESMTP id D8B6E3200934; Fri, 11 Nov 2022 01:50:14 -0500 (EST) Received: from mailfrontend1 ([10.202.2.162]) by compute4.internal (MEProxy); Fri, 11 Nov 2022 01:50:15 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:date:date:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:sender:subject :subject:to:to; s=fm1; t=1668149414; x=1668235814; bh=xX+Js5IEQI +g0MkdCSI9uMtBeidvZ5NgjkT47DgB5l0=; b=KCEG4bMu6Ly34XsfYiieIuw5Rv IVOcskoO86KVNXG1F9bqW6q/3ysUhmDzwV8+1lOkUkjd9sL7XU6gV9OvIpAvOkvd Fez3IPG9/PSxPwncOI4OrVnpAm58zmkIx2GzqeJCvEoqT01BjOzH7RM+vTCet2Q+ PbharqMNQhOstqW9gVWmEC5CP1Swi3GgrmuPLM4+JXG2uncFcnqhgqf2TEIMrW/U mHA1B4etUV+JAPPqn+PoYfp5aOJxwYD119bsvT+qNmXhJ3ZOpU7/sNJlP98QlS9x LEH9TJOqG3Nba0aJi3sWxJcFsNwSTk44mplqY6+S8jMeYbFoXlqJ56fafZhQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:date:date:feedback-id :feedback-id:from:from:in-reply-to:in-reply-to:message-id :mime-version:references:reply-to:sender:subject:subject:to:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm1; t=1668149414; x=1668235814; bh=xX+Js5IEQI+g0MkdCSI9uMtBeidv Z5NgjkT47DgB5l0=; b=CCIMqNYiIEahiuF0QkDWvs7fZiSgcAV+KlIM5urKtDtw 7+kdrp7evA8Buu3OXWI2+8LtxpQ9cHC19R5cceVVh/S1gL2NzUR8jUfxyBHf1f8V s+MqWgNIEw2NgLUbjdWUC9oBu3UeNQeLdZttZupuseNExiLPk/nYDZJWZIrlAE7g SGwKiEc0sJlTDcAbfHM804OEHO4srSNb9qaePKTUH+r/3/6bln+FVzpMTpTk6DIg z8wL5J2VsLbwYHJpNjbhoPMsizmol0ZM3TX5tqHkpFcohCC3emMd+BpIKiOrXP5C magKfRhkO8YQLdu45tFNBEHMUJH/+HEgvrG8Sm720g== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvgedrfeehgddutddvucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucenucfjughrpeffhffvvefukfhfgggtuggjsehgtd erredttddvnecuhfhrohhmpefrrghtrhhitghkucfuthgvihhnhhgrrhguthcuoehpshes phhkshdrihhmqeenucggtffrrghtthgvrhhnpeeukedtvedtffevleejtefgheehieegke eluddvfeefgeehgfeltddtheejleffteenucevlhhushhtvghrufhiiigvpedtnecurfgr rhgrmhepmhgrihhlfhhrohhmpehpshesphhkshdrihhm X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Fri, 11 Nov 2022 01:50:12 -0500 (EST) Received: by pks.im (OpenSMTPD) with ESMTPSA id 62daf3c5 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Fri, 11 Nov 2022 06:50:10 +0000 (UTC) Date: Fri, 11 Nov 2022 07:50:10 +0100 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Junio C Hamano , =?iso-8859-1?q?=C6var_Arnfj=F6r=F0?= Bjarmason , Taylor Blau , Jeff King Subject: [PATCH v5 5/7] revision: add new parameter to exclude hidden refs Message-ID: References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Users can optionally hide refs from remote users in git-upload-pack(1), git-receive-pack(1) and others via the `transfer.hideRefs`, but there is not an easy way to obtain the list of all visible or hidden refs right now. We'll require just that though for a performance improvement in our connectivity check. Add a new option `--exclude-hidden=` that excludes any hidden refs from the next pseudo-ref like `--all` or `--branches`. Signed-off-by: Patrick Steinhardt --- Documentation/rev-list-options.txt | 7 ++ builtin/rev-list.c | 1 + revision.c | 55 +++++++++- revision.h | 16 +++ t/t6021-rev-list-exclude-hidden.sh | 163 +++++++++++++++++++++++++++++ 5 files changed, 241 insertions(+), 1 deletion(-) create mode 100755 t/t6021-rev-list-exclude-hidden.sh diff --git a/Documentation/rev-list-options.txt b/Documentation/rev-list-options.txt index 1837509566..ff68e48406 100644 --- a/Documentation/rev-list-options.txt +++ b/Documentation/rev-list-options.txt @@ -195,6 +195,13 @@ respectively, and they must begin with `refs/` when applied to `--glob` or `--all`. If a trailing '/{asterisk}' is intended, it must be given explicitly. +--exclude-hidden=[receive|uploadpack]:: + Do not include refs that would be hidden by `git-receive-pack` or + `git-upload-pack` by consulting the appropriate `receive.hideRefs` or + `uploadpack.hideRefs` configuration along with `transfer.hideRefs` (see + linkgit:git-config[1]). This option affects the next pseudo-ref option + `--all` or `--glob` and is cleared after processing them. + --reflog:: Pretend as if all objects mentioned by reflogs are listed on the command line as ``. diff --git a/builtin/rev-list.c b/builtin/rev-list.c index 3acd93f71e..d42db0b0cc 100644 --- a/builtin/rev-list.c +++ b/builtin/rev-list.c @@ -38,6 +38,7 @@ static const char rev_list_usage[] = " --tags\n" " --remotes\n" " --stdin\n" +" --exclude-hidden=[receive|uploadpack]\n" " --quiet\n" " ordering output:\n" " --topo-order\n" diff --git a/revision.c b/revision.c index fe3ec98f46..bc32fb819a 100644 --- a/revision.c +++ b/revision.c @@ -1,4 +1,5 @@ #include "cache.h" +#include "config.h" #include "object-store.h" #include "tag.h" #include "blob.h" @@ -1519,11 +1520,17 @@ static void add_rev_cmdline_list(struct rev_info *revs, int ref_excluded(const struct ref_exclusions *exclusions, const char *path) { + const char *stripped_path = strip_namespace(path); struct string_list_item *item; + for_each_string_list_item(item, &exclusions->excluded_refs) { if (!wildmatch(item->string, path, 0)) return 1; } + + if (ref_is_hidden(stripped_path, path, &exclusions->hidden_refs)) + return 1; + return 0; } @@ -1536,6 +1543,8 @@ void init_ref_exclusions(struct ref_exclusions *exclusions) void clear_ref_exclusions(struct ref_exclusions *exclusions) { string_list_clear(&exclusions->excluded_refs, 0); + string_list_clear(&exclusions->hidden_refs, 0); + exclusions->hidden_refs_configured = 0; } void add_ref_exclusion(struct ref_exclusions *exclusions, const char *exclude) @@ -1543,6 +1552,35 @@ void add_ref_exclusion(struct ref_exclusions *exclusions, const char *exclude) string_list_append(&exclusions->excluded_refs, exclude); } +struct exclude_hidden_refs_cb { + struct ref_exclusions *exclusions; + const char *section; +}; + +static int hide_refs_config(const char *var, const char *value, void *cb_data) +{ + struct exclude_hidden_refs_cb *cb = cb_data; + cb->exclusions->hidden_refs_configured = 1; + return parse_hide_refs_config(var, value, cb->section, + &cb->exclusions->hidden_refs); +} + +void exclude_hidden_refs(struct ref_exclusions *exclusions, const char *section) +{ + struct exclude_hidden_refs_cb cb; + + if (strcmp(section, "receive") && strcmp(section, "uploadpack")) + die(_("unsupported section for hidden refs: %s"), section); + + if (exclusions->hidden_refs_configured) + die(_("--exclude-hidden= passed more than once")); + + cb.exclusions = exclusions; + cb.section = section; + + git_config(hide_refs_config, &cb); +} + struct all_refs_cb { int all_flags; int warned_bad_reflog; @@ -2221,7 +2259,7 @@ static int handle_revision_opt(struct rev_info *revs, int argc, const char **arg !strcmp(arg, "--bisect") || starts_with(arg, "--glob=") || !strcmp(arg, "--indexed-objects") || !strcmp(arg, "--alternate-refs") || - starts_with(arg, "--exclude=") || + starts_with(arg, "--exclude=") || starts_with(arg, "--exclude-hidden=") || starts_with(arg, "--branches=") || starts_with(arg, "--tags=") || starts_with(arg, "--remotes=") || starts_with(arg, "--no-walk=")) { @@ -2687,6 +2725,8 @@ static int handle_revision_pseudo_opt(struct rev_info *revs, } clear_ref_exclusions(&revs->ref_excludes); } else if (!strcmp(arg, "--branches")) { + if (revs->ref_excludes.hidden_refs_configured) + return error(_("--exclude-hidden cannot be used together with --branches")); handle_refs(refs, revs, *flags, refs_for_each_branch_ref); clear_ref_exclusions(&revs->ref_excludes); } else if (!strcmp(arg, "--bisect")) { @@ -2696,9 +2736,13 @@ static int handle_revision_pseudo_opt(struct rev_info *revs, for_each_good_bisect_ref); revs->bisect = 1; } else if (!strcmp(arg, "--tags")) { + if (revs->ref_excludes.hidden_refs_configured) + return error(_("--exclude-hidden cannot be used together with --tags")); handle_refs(refs, revs, *flags, refs_for_each_tag_ref); clear_ref_exclusions(&revs->ref_excludes); } else if (!strcmp(arg, "--remotes")) { + if (revs->ref_excludes.hidden_refs_configured) + return error(_("--exclude-hidden cannot be used together with --remotes")); handle_refs(refs, revs, *flags, refs_for_each_remote_ref); clear_ref_exclusions(&revs->ref_excludes); } else if ((argcount = parse_long_opt("glob", argv, &optarg))) { @@ -2710,18 +2754,27 @@ static int handle_revision_pseudo_opt(struct rev_info *revs, } else if ((argcount = parse_long_opt("exclude", argv, &optarg))) { add_ref_exclusion(&revs->ref_excludes, optarg); return argcount; + } else if ((argcount = parse_long_opt("exclude-hidden", argv, &optarg))) { + exclude_hidden_refs(&revs->ref_excludes, optarg); + return argcount; } else if (skip_prefix(arg, "--branches=", &optarg)) { struct all_refs_cb cb; + if (revs->ref_excludes.hidden_refs_configured) + return error(_("--exclude-hidden cannot be used together with --branches")); init_all_refs_cb(&cb, revs, *flags); for_each_glob_ref_in(handle_one_ref, optarg, "refs/heads/", &cb); clear_ref_exclusions(&revs->ref_excludes); } else if (skip_prefix(arg, "--tags=", &optarg)) { struct all_refs_cb cb; + if (revs->ref_excludes.hidden_refs_configured) + return error(_("--exclude-hidden cannot be used together with --tags")); init_all_refs_cb(&cb, revs, *flags); for_each_glob_ref_in(handle_one_ref, optarg, "refs/tags/", &cb); clear_ref_exclusions(&revs->ref_excludes); } else if (skip_prefix(arg, "--remotes=", &optarg)) { struct all_refs_cb cb; + if (revs->ref_excludes.hidden_refs_configured) + return error(_("--exclude-hidden cannot be used together with --remotes")); init_all_refs_cb(&cb, revs, *flags); for_each_glob_ref_in(handle_one_ref, optarg, "refs/remotes/", &cb); clear_ref_exclusions(&revs->ref_excludes); diff --git a/revision.h b/revision.h index 5c8ab16047..adb810c2f8 100644 --- a/revision.h +++ b/revision.h @@ -87,6 +87,19 @@ struct ref_exclusions { * patterns matches, the reference will be excluded. */ struct string_list excluded_refs; + + /* + * Hidden refs is a list of patterns that is to be hidden via + * `ref_is_hidden()`. + */ + struct string_list hidden_refs; + + /* + * Indicates whether hidden refs have been configured. This is to + * distinguish between no hidden refs existing and hidden refs not + * being parsed. + */ + char hidden_refs_configured; }; /** @@ -94,6 +107,7 @@ struct ref_exclusions { */ #define REF_EXCLUSIONS_INIT { \ .excluded_refs = STRING_LIST_INIT_DUP, \ + .hidden_refs = STRING_LIST_INIT_DUP, \ } struct oidset; @@ -456,10 +470,12 @@ void show_object_with_name(FILE *, struct object *, const char *); /** * Helpers to check if a reference should be excluded. */ + int ref_excluded(const struct ref_exclusions *exclusions, const char *path); void init_ref_exclusions(struct ref_exclusions *); void clear_ref_exclusions(struct ref_exclusions *); void add_ref_exclusion(struct ref_exclusions *, const char *exclude); +void exclude_hidden_refs(struct ref_exclusions *, const char *section); /** * This function can be used if you want to add commit objects as revision diff --git a/t/t6021-rev-list-exclude-hidden.sh b/t/t6021-rev-list-exclude-hidden.sh new file mode 100755 index 0000000000..018796d41c --- /dev/null +++ b/t/t6021-rev-list-exclude-hidden.sh @@ -0,0 +1,163 @@ +#!/bin/sh + +test_description='git rev-list --exclude-hidden test' + +. ./test-lib.sh + +test_expect_success 'setup' ' + test_commit_bulk --id=commit --ref=refs/heads/main 1 && + COMMIT=$(git rev-parse refs/heads/main) && + test_commit_bulk --id=tag --ref=refs/tags/lightweight 1 && + TAG=$(git rev-parse refs/tags/lightweight) && + test_commit_bulk --id=hidden --ref=refs/hidden/commit 1 && + HIDDEN=$(git rev-parse refs/hidden/commit) && + test_commit_bulk --id=namespace --ref=refs/namespaces/namespace/refs/namespaced/commit 1 && + NAMESPACE=$(git rev-parse refs/namespaces/namespace/refs/namespaced/commit) +' + +test_expect_success 'invalid section' ' + echo "fatal: unsupported section for hidden refs: unsupported" >expected && + test_must_fail git rev-list --exclude-hidden=unsupported 2>err && + test_cmp expected err +' + +for section in receive uploadpack +do + test_expect_success "$section: passed multiple times" ' + echo "fatal: --exclude-hidden= passed more than once" >expected && + test_must_fail git rev-list --exclude-hidden=$section --exclude-hidden=$section 2>err && + test_cmp expected err + ' + + test_expect_success "$section: without hiddenRefs" ' + git rev-list --exclude-hidden=$section --all >out && + cat >expected <<-EOF && + $NAMESPACE + $HIDDEN + $TAG + $COMMIT + EOF + test_cmp expected out + ' + + test_expect_success "$section: hidden via transfer.hideRefs" ' + git -c transfer.hideRefs=refs/hidden/ rev-list --exclude-hidden=$section --all >out && + cat >expected <<-EOF && + $NAMESPACE + $TAG + $COMMIT + EOF + test_cmp expected out + ' + + test_expect_success "$section: hidden via $section.hideRefs" ' + git -c $section.hideRefs=refs/hidden/ rev-list --exclude-hidden=$section --all >out && + cat >expected <<-EOF && + $NAMESPACE + $TAG + $COMMIT + EOF + test_cmp expected out + ' + + test_expect_success "$section: respects both transfer.hideRefs and $section.hideRefs" ' + git -c transfer.hideRefs=refs/tags/ -c $section.hideRefs=refs/hidden/ rev-list --exclude-hidden=$section --all >out && + cat >expected <<-EOF && + $NAMESPACE + $COMMIT + EOF + test_cmp expected out + ' + + test_expect_success "$section: negation without hidden refs marks everything as uninteresting" ' + git rev-list --all --exclude-hidden=$section --not --all >out && + test_must_be_empty out + ' + + test_expect_success "$section: negation with hidden refs marks them as interesting" ' + git -c transfer.hideRefs=refs/hidden/ rev-list --all --exclude-hidden=$section --not --all >out && + cat >expected <<-EOF && + $HIDDEN + EOF + test_cmp expected out + ' + + test_expect_success "$section: hidden refs and excludes work together" ' + git -c transfer.hideRefs=refs/hidden/ rev-list --exclude=refs/tags/* --exclude-hidden=$section --all >out && + cat >expected <<-EOF && + $NAMESPACE + $COMMIT + EOF + test_cmp expected out + ' + + test_expect_success "$section: excluded hidden refs get reset" ' + git -c transfer.hideRefs=refs/ rev-list --exclude-hidden=$section --all --all >out && + cat >expected <<-EOF && + $NAMESPACE + $HIDDEN + $TAG + $COMMIT + EOF + test_cmp expected out + ' + + test_expect_success "$section: excluded hidden refs can be used with multiple pseudo-refs" ' + git -c transfer.hideRefs=refs/ rev-list --exclude-hidden=$section --all --exclude-hidden=$section --all >out && + test_must_be_empty out + ' + + test_expect_success "$section: works with --glob" ' + git -c transfer.hideRefs=refs/hidden/ rev-list --exclude-hidden=$section --glob=refs/h* >out && + cat >expected <<-EOF && + $COMMIT + EOF + test_cmp expected out + ' + + test_expect_success "$section: operates on stripped refs by default" ' + GIT_NAMESPACE=namespace git -c transfer.hideRefs=refs/namespaced/ rev-list --exclude-hidden=$section --all >out && + cat >expected <<-EOF && + $HIDDEN + $TAG + $COMMIT + EOF + test_cmp expected out + ' + + test_expect_success "$section: does not hide namespace by default" ' + GIT_NAMESPACE=namespace git -c transfer.hideRefs=refs/namespaces/namespace/ rev-list --exclude-hidden=$section --all >out && + cat >expected <<-EOF && + $NAMESPACE + $HIDDEN + $TAG + $COMMIT + EOF + test_cmp expected out + ' + + test_expect_success "$section: can operate on unstripped refs" ' + GIT_NAMESPACE=namespace git -c transfer.hideRefs=^refs/namespaces/namespace/ rev-list --exclude-hidden=$section --all >out && + cat >expected <<-EOF && + $HIDDEN + $TAG + $COMMIT + EOF + test_cmp expected out + ' + + for pseudoopt in remotes branches tags + do + test_expect_success "$section: fails with --$pseudoopt" ' + test_must_fail git rev-list --exclude-hidden=$section --$pseudoopt 2>err && + test_i18ngrep "error: --exclude-hidden cannot be used together with --$pseudoopt" err + ' + + test_expect_success "$section: fails with --$pseudoopt=pattern" ' + test_must_fail git rev-list --exclude-hidden=$section --$pseudoopt=pattern 2>err && + test_i18ngrep "error: --exclude-hidden cannot be used together with --$pseudoopt" err + ' + done +done + +test_done From patchwork Fri Nov 11 06:50:14 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13039754 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 70B31C43217 for ; Fri, 11 Nov 2022 06:50:35 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233109AbiKKGue (ORCPT ); Fri, 11 Nov 2022 01:50:34 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39752 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233162AbiKKGuZ (ORCPT ); Fri, 11 Nov 2022 01:50:25 -0500 Received: from wout1-smtp.messagingengine.com (wout1-smtp.messagingengine.com [64.147.123.24]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3D4CC7830F for ; Thu, 10 Nov 2022 22:50:19 -0800 (PST) Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailout.west.internal (Postfix) with ESMTP id 738493200937; Fri, 11 Nov 2022 01:50:18 -0500 (EST) Received: from mailfrontend1 ([10.202.2.162]) by compute3.internal (MEProxy); Fri, 11 Nov 2022 01:50:18 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:date:date:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:sender:subject :subject:to:to; s=fm1; t=1668149418; x=1668235818; bh=PvB38zTEMO H/6CBIhegdDch3qeD4WkVZF1MV4VZh0Q8=; b=Vz3kHiWcFgDA1YJT2QRQt2tdEE yTK2knUAUFEyKnrUKkWkPGkXRyoUeVMyuPR6otytwOWW0+Vj3xbu0ZVy8gzmq1ub rA9UpfA0z2HMirTvVGa9MfM4WAkmwRdpWyGlO01pvd1Pk1a2x9u3LhX7yj/dqUhF s0PeUzKPTFzmnJ0vypnYc+nVySCcp0a9i1SWMrIKuXILdT5+hnAxQp+1tEvzb3rX O9uAsWTSjeQjCUc8Erz8lpm8qgDdobtDOASEEpx2dFRbHuOKBPaUsbhS/Y7P8/Hn +TKPewX1A/WJpbd34MCG2V3DjNY3vre1NSsCZYH48GDpk8yWGBxpkJvRqkFA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:date:date:feedback-id :feedback-id:from:from:in-reply-to:in-reply-to:message-id :mime-version:references:reply-to:sender:subject:subject:to:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm1; t=1668149418; x=1668235818; bh=PvB38zTEMOH/6CBIhegdDch3qeD4 WkVZF1MV4VZh0Q8=; b=o+rgEL1ztIlPw/OeqLo8Rhg/9HmxxeThxoo1HdIKK9yl lVIl0sjRA7hrKQbVuerJ/zN/j3IhqpJ2ddDztjbqY4yugRDfTA63Gy4CewPxvNec cTpPIzLG0epifO7bkhd2GA6l8gS8b3eJzzRGX4bTXGUAzmNonTdPQTrkh0axToIB Qr6m71TMruCc/6x8k7HA5HGKE7q0o4neyGwUVmQSxyLc8GK2gJvKFaVHxyCJaTrL GsGmS99NXGG2NK/bACQFQeJRaAbmp0uw3MjH5Tfp5EcyA66GfZNrV+/mFecKpXWA pc7nvqAAxw4H83aLAYd1QFheLrM2AywwKv5g7AkDxA== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvgedrfeehgddutdefucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucenucfjughrpeffhffvvefukfhfgggtuggjsehgtd erredttddvnecuhfhrohhmpefrrghtrhhitghkucfuthgvihhnhhgrrhguthcuoehpshes phhkshdrihhmqeenucggtffrrghtthgvrhhnpeeukedtvedtffevleejtefgheehieegke eluddvfeefgeehgfeltddtheejleffteenucevlhhushhtvghrufhiiigvpedtnecurfgr rhgrmhepmhgrihhlfhhrohhmpehpshesphhkshdrihhm X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Fri, 11 Nov 2022 01:50:16 -0500 (EST) Received: by pks.im (OpenSMTPD) with ESMTPSA id 3f2958ed (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Fri, 11 Nov 2022 06:50:14 +0000 (UTC) Date: Fri, 11 Nov 2022 07:50:14 +0100 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Junio C Hamano , =?iso-8859-1?q?=C6var_Arnfj=F6r=F0?= Bjarmason , Taylor Blau , Jeff King Subject: [PATCH v5 6/7] rev-parse: add `--exclude-hidden=` option Message-ID: <2eeb25eef0cdf6c5f9f01058326d8b0a9992c461.1668149149.git.ps@pks.im> References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Add a new `--exclude-hidden=` option that is similar to the one we just added to git-rev-list(1). Given a seciton name `uploadpack` or `receive` as argument, it causes us to exclude all references that would be hidden by the respective `$section.hideRefs` configuration. Signed-off-by: Patrick Steinhardt --- Documentation/git-rev-parse.txt | 7 ++++++ builtin/rev-parse.c | 10 +++++++++ t/t6018-rev-list-glob.sh | 40 +++++++++++++++++++++++++++++++++ 3 files changed, 57 insertions(+) diff --git a/Documentation/git-rev-parse.txt b/Documentation/git-rev-parse.txt index 6b8ca085aa..bcd8069287 100644 --- a/Documentation/git-rev-parse.txt +++ b/Documentation/git-rev-parse.txt @@ -197,6 +197,13 @@ respectively, and they must begin with `refs/` when applied to `--glob` or `--all`. If a trailing '/{asterisk}' is intended, it must be given explicitly. +--exclude-hidden=[receive|uploadpack]:: + Do not include refs that would be hidden by `git-receive-pack` or + `git-upload-pack` by consulting the appropriate `receive.hideRefs` or + `uploadpack.hideRefs` configuration along with `transfer.hideRefs` (see + linkgit:git-config[1]). This option affects the next pseudo-ref option + `--all` or `--glob` and is cleared after processing them. + --disambiguate=:: Show every object whose name begins with the given prefix. The must be at least 4 hexadecimal digits long to diff --git a/builtin/rev-parse.c b/builtin/rev-parse.c index 7fa5b6991b..b5666a03bd 100644 --- a/builtin/rev-parse.c +++ b/builtin/rev-parse.c @@ -876,10 +876,14 @@ int cmd_rev_parse(int argc, const char **argv, const char *prefix) continue; } if (opt_with_value(arg, "--branches", &arg)) { + if (ref_excludes.hidden_refs_configured) + return error(_("--exclude-hidden cannot be used together with --branches")); handle_ref_opt(arg, "refs/heads/"); continue; } if (opt_with_value(arg, "--tags", &arg)) { + if (ref_excludes.hidden_refs_configured) + return error(_("--exclude-hidden cannot be used together with --tags")); handle_ref_opt(arg, "refs/tags/"); continue; } @@ -888,6 +892,8 @@ int cmd_rev_parse(int argc, const char **argv, const char *prefix) continue; } if (opt_with_value(arg, "--remotes", &arg)) { + if (ref_excludes.hidden_refs_configured) + return error(_("--exclude-hidden cannot be used together with --remotes")); handle_ref_opt(arg, "refs/remotes/"); continue; } @@ -895,6 +901,10 @@ int cmd_rev_parse(int argc, const char **argv, const char *prefix) add_ref_exclusion(&ref_excludes, arg); continue; } + if (skip_prefix(arg, "--exclude-hidden=", &arg)) { + exclude_hidden_refs(&ref_excludes, arg); + continue; + } if (!strcmp(arg, "--show-toplevel")) { const char *work_tree = get_git_work_tree(); if (work_tree) diff --git a/t/t6018-rev-list-glob.sh b/t/t6018-rev-list-glob.sh index e1abc5c2b3..aabf590dda 100755 --- a/t/t6018-rev-list-glob.sh +++ b/t/t6018-rev-list-glob.sh @@ -187,6 +187,46 @@ test_expect_success 'rev-parse --exclude=ref with --remotes=glob' ' compare rev-parse "--exclude=upstream/x --remotes=upstream/*" "upstream/one upstream/two" ' +for section in receive uploadpack +do + test_expect_success "rev-parse --exclude-hidden=$section with --all" ' + compare "-c transfer.hideRefs=refs/remotes/ rev-parse" "--branches --tags" "--exclude-hidden=$section --all" + ' + + test_expect_success "rev-parse --exclude-hidden=$section with --all" ' + compare "-c transfer.hideRefs=refs/heads/subspace/ rev-parse" "--exclude=refs/heads/subspace/* --all" "--exclude-hidden=$section --all" + ' + + test_expect_success "rev-parse --exclude-hidden=$section with --glob" ' + compare "-c transfer.hideRefs=refs/heads/subspace/ rev-parse" "--exclude=refs/heads/subspace/* --glob=refs/heads/*" "--exclude-hidden=$section --glob=refs/heads/*" + ' + + test_expect_success "rev-parse --exclude-hidden=$section can be passed once per pseudo-ref" ' + compare "-c transfer.hideRefs=refs/remotes/ rev-parse" "--branches --tags --branches --tags" "--exclude-hidden=$section --all --exclude-hidden=$section --all" + ' + + test_expect_success "rev-parse --exclude-hidden=$section can only be passed once per pseudo-ref" ' + echo "fatal: --exclude-hidden= passed more than once" >expected && + test_must_fail git rev-parse --exclude-hidden=$section --exclude-hidden=$section 2>err && + test_cmp expected err + ' + + for pseudoopt in branches tags remotes + do + test_expect_success "rev-parse --exclude-hidden=$section fails with --$pseudoopt" ' + echo "error: --exclude-hidden cannot be used together with --$pseudoopt" >expected && + test_must_fail git rev-parse --exclude-hidden=$section --$pseudoopt 2>err && + test_cmp expected err + ' + + test_expect_success "rev-parse --exclude-hidden=$section fails with --$pseudoopt=pattern" ' + echo "error: --exclude-hidden cannot be used together with --$pseudoopt" >expected && + test_must_fail git rev-parse --exclude-hidden=$section --$pseudoopt=pattern 2>err && + test_cmp expected err + ' + done +done + test_expect_success 'rev-list --exclude=glob with --branches=glob' ' compare rev-list "--exclude=subspace-* --branches=sub*" "subspace/one subspace/two" ' From patchwork Fri Nov 11 06:50:18 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13039755 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2231AC4332F for ; Fri, 11 Nov 2022 06:50:38 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233152AbiKKGuf (ORCPT ); Fri, 11 Nov 2022 01:50:35 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39756 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232182AbiKKGu0 (ORCPT ); Fri, 11 Nov 2022 01:50:26 -0500 Received: from wout1-smtp.messagingengine.com (wout1-smtp.messagingengine.com [64.147.123.24]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CE1F8729A7 for ; Thu, 10 Nov 2022 22:50:23 -0800 (PST) Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailout.west.internal (Postfix) with ESMTP id 130313200938; Fri, 11 Nov 2022 01:50:23 -0500 (EST) Received: from mailfrontend1 ([10.202.2.162]) by compute1.internal (MEProxy); Fri, 11 Nov 2022 01:50:23 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:date:date:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:sender:subject :subject:to:to; s=fm1; t=1668149422; x=1668235822; bh=JlPedTWuIA YaW+eX9ZtJH414ZCoT6LhOujm+XZlR+CQ=; b=PLdI/oYJolp5AgUCYYaTeGduJ8 HfkAK+wLI2CBGO0XUhK1BJRCCOTNeLRWVwNS53KEtCpdXD7wlmu+D9+sT6vc4hHR aKRQzRNYUxp+imIvTr0+FTxHVxUO43ct7FQprUZiKci7+VEWrYxfktSMRwRAKbo8 LDHFWYSQgvfZH9vOy5rdjtnKXWI5obdo87vYVACf4/JJh8eMmUkrKPZPs4yNmcoq i9hasLbclv7+49bQhhE5ePYsOgyEznKf4BBh9uDUeA12Zl7bQJh7EgkEAvndWrmI jxCaT0E6Ln4auZ+DdFOztcNca02sIfYPvUBPeWEuSBNBBvvNQZM/eGU/KHmg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:date:date:feedback-id :feedback-id:from:from:in-reply-to:in-reply-to:message-id :mime-version:references:reply-to:sender:subject:subject:to:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm1; t=1668149422; x=1668235822; bh=JlPedTWuIAYaW+eX9ZtJH414ZCoT 6LhOujm+XZlR+CQ=; b=KPaxP17h0FKQfTrEpwWrDq99c+dKbXWgBz+AObz3svg/ AcXxrUoyECaFaraoQKoLWCpADgjl8sv/aPrtR7q7UPgb1W08L6X4whaVxB57iS20 k399TFRJrrta1d8RNy1cEBSR5/mwADIvDCg5zpm+8lhp5y6Byqx+pWwsNHg3U7sq 50fI4WNbOexd3rweXjoBaKYHxZWxw0XGmgVyIqze+pkYJ8xp6sBsj0H+T3xRXKFW mIejlMH9ls3UZfHWV67LwSUM3P1/6656u/xr5RuNJM6ZQU3jD0fwrmQOZXb7SYT+ Z1f/RbMBSaXUbwqaKnMARgPtqikvOJTf+V//CMmCrw== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvgedrfeehgddutddvucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucenucfjughrpeffhffvvefukfhfgggtuggjsehgtd erredttdejnecuhfhrohhmpefrrghtrhhitghkucfuthgvihhnhhgrrhguthcuoehpshes phhkshdrihhmqeenucggtffrrghtthgvrhhnpeetueevhffhudefvdegieeuieelgedthf egfedtueevjeejtdfgjeehudejuedtudenucevlhhushhtvghrufhiiigvpedtnecurfgr rhgrmhepmhgrihhlfhhrohhmpehpshesphhkshdrihhm X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Fri, 11 Nov 2022 01:50:21 -0500 (EST) Received: by pks.im (OpenSMTPD) with ESMTPSA id 3935c296 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Fri, 11 Nov 2022 06:50:18 +0000 (UTC) Date: Fri, 11 Nov 2022 07:50:18 +0100 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Junio C Hamano , =?iso-8859-1?q?=C6var_Arnfj=F6r=F0?= Bjarmason , Taylor Blau , Jeff King Subject: [PATCH v5 7/7] receive-pack: only use visible refs for connectivity check Message-ID: References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org When serving a push, git-receive-pack(1) needs to verify that the packfile sent by the client contains all objects that are required by the updated references. This connectivity check works by marking all preexisting references as uninteresting and using the new reference tips as starting point for a graph walk. Marking all preexisting references as uninteresting can be a problem when it comes to performance. Git forges tend to do internal bookkeeping to keep alive sets of objects for internal use or make them easy to find via certain references. These references are typically hidden away from the user so that they are neither advertised nor writeable. At GitLab, we have one particular repository that contains a total of 7 million references, of which 6.8 million are indeed internal references. With the current connectivity check we are forced to load all these references in order to mark them as uninteresting, and this alone takes around 15 seconds to compute. We can optimize this by only taking into account the set of visible refs when marking objects as uninteresting. This means that we may now walk more objects until we hit any object that is marked as uninteresting. But it is rather unlikely that clients send objects that make large parts of objects reachable that have previously only ever been hidden, whereas the common case is to push incremental changes that build on top of the visible object graph. This provides a huge boost to performance in the mentioned repository, where the vast majority of its refs hidden. Pushing a new commit into this repo with `transfer.hideRefs` set up to hide 6.8 million of 7 refs as it is configured in Gitaly leads to a 4.5-fold speedup: Benchmark 1: main Time (mean ± σ): 30.977 s ± 0.157 s [User: 30.226 s, System: 1.083 s] Range (min … max): 30.796 s … 31.071 s 3 runs Benchmark 2: pks-connectivity-check-hide-refs Time (mean ± σ): 6.799 s ± 0.063 s [User: 6.803 s, System: 0.354 s] Range (min … max): 6.729 s … 6.850 s 3 runs Summary 'pks-connectivity-check-hide-refs' ran 4.56 ± 0.05 times faster than 'main' As we mostly go through the same codepaths even in the case where there are no hidden refs at all compared to the code before there is no change in performance when no refs are hidden: Benchmark 1: main Time (mean ± σ): 48.188 s ± 0.432 s [User: 49.326 s, System: 5.009 s] Range (min … max): 47.706 s … 48.539 s 3 runs Benchmark 2: pks-connectivity-check-hide-refs Time (mean ± σ): 48.027 s ± 0.500 s [User: 48.934 s, System: 5.025 s] Range (min … max): 47.504 s … 48.500 s 3 runs Summary 'pks-connectivity-check-hide-refs' ran 1.00 ± 0.01 times faster than 'main' Signed-off-by: Patrick Steinhardt --- builtin/receive-pack.c | 2 ++ connected.c | 3 +++ connected.h | 7 +++++++ 3 files changed, 12 insertions(+) diff --git a/builtin/receive-pack.c b/builtin/receive-pack.c index 1e24b31a0a..a90af30363 100644 --- a/builtin/receive-pack.c +++ b/builtin/receive-pack.c @@ -1929,6 +1929,8 @@ static void execute_commands(struct command *commands, opt.err_fd = err_fd; opt.progress = err_fd && !quiet; opt.env = tmp_objdir_env(tmp_objdir); + opt.exclude_hidden_refs_section = "receive"; + if (check_connected(iterate_receive_command_list, &data, &opt)) set_connectivity_errors(commands, si); diff --git a/connected.c b/connected.c index 74a20cb32e..4f6388eed7 100644 --- a/connected.c +++ b/connected.c @@ -100,6 +100,9 @@ int check_connected(oid_iterate_fn fn, void *cb_data, strvec_push(&rev_list.args, "--exclude-promisor-objects"); if (!opt->is_deepening_fetch) { strvec_push(&rev_list.args, "--not"); + if (opt->exclude_hidden_refs_section) + strvec_pushf(&rev_list.args, "--exclude-hidden=%s", + opt->exclude_hidden_refs_section); strvec_push(&rev_list.args, "--all"); } strvec_push(&rev_list.args, "--quiet"); diff --git a/connected.h b/connected.h index 6e59c92aa3..16b2c84f2e 100644 --- a/connected.h +++ b/connected.h @@ -46,6 +46,13 @@ struct check_connected_options { * during a fetch. */ unsigned is_deepening_fetch : 1; + + /* + * If not NULL, use `--exclude-hidden=$section` to exclude all refs + * hidden via the `$section.hideRefs` config from the set of + * already-reachable refs. + */ + const char *exclude_hidden_refs_section; }; #define CHECK_CONNECTED_INIT { 0 }