From patchwork Wed Aug 7 12:43:48 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13756207 Received: from fout8-smtp.messagingengine.com (fout8-smtp.messagingengine.com [103.168.172.151]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A81481E4F13 for ; Wed, 7 Aug 2024 12:43:52 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=103.168.172.151 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723034635; cv=none; b=uQEFOdPoaai0Lyvayjep9U6gmtYzYD15tWQcjVf8gmHkfYwHaxKMwgFusRowbiWNwgsRiMw8L5a0Z7+IBwNtM7IPnB1J1ReMKRIjfnCmzIh76PtkjWTuudjGXxmM8AbOHbvY7ENyzOUGpDoTjxTRoSWWe+mHRZjSFinMeo+taJo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723034635; c=relaxed/simple; bh=J9ju/WYAcmZTrZGXDNQiSighlsoCJw57xcG9bFFdyXo=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=e6JbHpz/JhqvmfTIXTwsZhNo9eTJbWZ6R463LwfBECzuMvHHqcCQwsEWM8v7ziTcHXbM7XX3U1mOl0ofprb6+9PL9GpbUiBhzxbEzSJMrSlcDUt/AUeVp6/VPJyYiK9V5iZxuaMPDEtpNZ0R/9TdAMJLvebL6fLDMwHEi83/ZtA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=pks.im; spf=pass smtp.mailfrom=pks.im; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b=Z5AEdgkH; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=qSFEhdsY; arc=none smtp.client-ip=103.168.172.151 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=pks.im Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=pks.im Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b="Z5AEdgkH"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="qSFEhdsY" Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailfout.nyi.internal (Postfix) with ESMTP id B9CDE138FC27; Wed, 7 Aug 2024 08:43:51 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute3.internal (MEProxy); Wed, 07 Aug 2024 08:43:51 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:subject :subject:to:to; s=fm3; t=1723034631; x=1723121031; bh=sLcDYWcLxB WewFijaQACv6JRKMtVXOiM2lYqyBr5Qsk=; b=Z5AEdgkHUowklnXVdTotZGqk+W XnLvCRDqlnb7yknL43sps3fIg8oJiSubNcQlo4dojPlvJrzKRkjzgk13u7hcgdfj haho+kv1CUc4OtqrRDHfnfUSESzlXJa6dtg0nYdxv91EgxrZvN1RqtYtfbQWiyNS 3gS7pFOndha9S6tC8o5ozSiAds7qoO4LoXG8bPMLb0sfd/4F/5NySvcrsAdn24hF 6rS9TmXw1XjISb8p+mvdhFdg7r1AjfUiMRSq56H5PaR7k0wTxX764scE5CLQMBY2 vDxjiK14UzZW2wzWcgearavLfAl77AH6K3mCuqKzGWVVY+P62Ik5tsKYcW+w== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm3; t=1723034631; x=1723121031; bh=sLcDYWcLxBWewFijaQACv6JRKMtV XOiM2lYqyBr5Qsk=; b=qSFEhdsYA+rHekr1WZOh3wlh9uiZmnR0p0gnCSBfAKpI RW+QYYB7oqSIKVrZI6FnMqDOJn/xNS6ZiZTtSzrmyBmAHno9FkqplGuEy9nf6MD2 fiVfG+M+CFm6T7Pk8DBFbxbo+zUpFa3lKFCmSHkSUxaeTorIWjWTH3yRhI1h1qXY RuteZg2A/jInmDXbsStF4pI6P9gi+PwWYBeeijq+ZQedjXoxr8ZvBGs+gjz8qIIX Lv9yJ+IJn7b1RryWShNvVkQB70HPUxmbphVgMggpnZ5dQreJSSeZ8zJSJOWMaOSE af1ic1dNKmaLn1VJVEhIULAZ2Z8YJlDYtvq5O6MZvg== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeeftddrledtgdehhecutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc fjughrpeffhffvvefukfhfgggtuggjsehgtderredttddvnecuhfhrohhmpefrrghtrhhi tghkucfuthgvihhnhhgrrhguthcuoehpshesphhkshdrihhmqeenucggtffrrghtthgvrh hnpeeukedtvedtffevleejtefgheehieegkeeluddvfeefgeehgfeltddtheejleffteen ucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpehpshesph hkshdrihhmpdhnsggprhgtphhtthhopedt X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Wed, 7 Aug 2024 08:43:50 -0400 (EDT) Received: by vm-mail (OpenSMTPD) with ESMTPSA id d4dbe104 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Wed, 7 Aug 2024 12:43:46 +0000 (UTC) Date: Wed, 7 Aug 2024 14:43:48 +0200 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Eric Sunshine , Jeppe =?utf-8?b?w5hsYW5k?= Subject: [PATCH 1/6] builtin/submodule: allow cloning with different ref storage format Message-ID: References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: As submodules are proper self-contained repositories, it is perfectly valid for them to have a different ref storage format than their parent repository. There is no obvious way for users to ask for the ref storage format when initializing submodules though. Whether the setup of such mixed-ref-storage-format constellations is all that useful remains to be seen. But there is no good reason to not expose such an option, and we will require it in a subsequent patch. Introduce a new `--ref-format=` option for git-submodule(1) that allows the user to pick the ref storage format. This option will also be used in a subsequent commit, where we start to propagate the same flag from git-clone(1) to cloning submodules with the `--recursive` switch. Signed-off-by: Patrick Steinhardt --- Documentation/git-submodule.txt | 5 +++- builtin/submodule--helper.c | 30 +++++++++++++++++++ git-submodule.sh | 9 ++++++ t/t7424-submodule-mixed-ref-formats.sh | 41 ++++++++++++++++++++++++++ 4 files changed, 84 insertions(+), 1 deletion(-) create mode 100755 t/t7424-submodule-mixed-ref-formats.sh diff --git a/Documentation/git-submodule.txt b/Documentation/git-submodule.txt index ca0347a37b..73ef8b9696 100644 --- a/Documentation/git-submodule.txt +++ b/Documentation/git-submodule.txt @@ -136,7 +136,7 @@ If you really want to remove a submodule from the repository and commit that use linkgit:git-rm[1] instead. See linkgit:gitsubmodules[7] for removal options. -update [--init] [--remote] [-N|--no-fetch] [--[no-]recommend-shallow] [-f|--force] [--checkout|--rebase|--merge] [--reference ] [--depth ] [--recursive] [--jobs ] [--[no-]single-branch] [--filter ] [--] [...]:: +update [--init] [--remote] [-N|--no-fetch] [--[no-]recommend-shallow] [-f|--force] [--checkout|--rebase|--merge] [--reference ] [--ref-format ] [--depth ] [--recursive] [--jobs ] [--[no-]single-branch] [--filter ] [--] [...]:: + -- Update the registered submodules to match what the superproject @@ -185,6 +185,9 @@ submodule with the `--init` option. If `--recursive` is specified, this command will recurse into the registered submodules, and update any nested submodules within. +If `--ref-format ` is specified, the ref storage format of newly +cloned submodules will be set accordingly. + If `--filter ` is specified, the given partial clone filter will be applied to the submodule. See linkgit:git-rev-list[1] for details on filter specifications. diff --git a/builtin/submodule--helper.c b/builtin/submodule--helper.c index f1218a1995..42a36bc2f7 100644 --- a/builtin/submodule--helper.c +++ b/builtin/submodule--helper.c @@ -1532,6 +1532,7 @@ struct module_clone_data { const char *url; const char *depth; struct list_objects_filter_options *filter_options; + enum ref_storage_format ref_storage_format; unsigned int quiet: 1; unsigned int progress: 1; unsigned int dissociate: 1; @@ -1540,6 +1541,7 @@ struct module_clone_data { }; #define MODULE_CLONE_DATA_INIT { \ .single_branch = -1, \ + .ref_storage_format = REF_STORAGE_FORMAT_UNKNOWN, \ } struct submodule_alternate_setup { @@ -1738,6 +1740,9 @@ static int clone_submodule(const struct module_clone_data *clone_data, strvec_pushl(&cp.args, "--reference", item->string, NULL); } + if (clone_data->ref_storage_format != REF_STORAGE_FORMAT_UNKNOWN) + strvec_pushf(&cp.args, "--ref-format=%s", + ref_storage_format_to_name(clone_data->ref_storage_format)); if (clone_data->dissociate) strvec_push(&cp.args, "--dissociate"); if (sm_gitdir && *sm_gitdir) @@ -1832,6 +1837,7 @@ static int module_clone(int argc, const char **argv, const char *prefix) struct string_list reference = STRING_LIST_INIT_NODUP; struct list_objects_filter_options filter_options = LIST_OBJECTS_FILTER_INIT; + const char *ref_storage_format = NULL; struct option module_clone_options[] = { OPT_STRING(0, "prefix", &clone_data.prefix, @@ -1849,6 +1855,8 @@ static int module_clone(int argc, const char **argv, const char *prefix) OPT_STRING_LIST(0, "reference", &reference, N_("repo"), N_("reference repository")), + OPT_STRING(0, "ref-format", &ref_storage_format, N_("format"), + N_("specify the reference format to use")), OPT_BOOL(0, "dissociate", &dissociate, N_("use --reference only while cloning")), OPT_STRING(0, "depth", &clone_data.depth, @@ -1875,6 +1883,11 @@ static int module_clone(int argc, const char **argv, const char *prefix) argc = parse_options(argc, argv, prefix, module_clone_options, git_submodule_helper_usage, 0); + if (ref_storage_format) { + clone_data.ref_storage_format = ref_storage_format_by_name(ref_storage_format); + if (clone_data.ref_storage_format == REF_STORAGE_FORMAT_UNKNOWN) + die(_("unknown ref storage format '%s'"), ref_storage_format); + } clone_data.dissociate = !!dissociate; clone_data.quiet = !!quiet; clone_data.progress = !!progress; @@ -1974,6 +1987,7 @@ struct update_data { struct submodule_update_strategy update_strategy; struct list_objects_filter_options *filter_options; struct module_list list; + enum ref_storage_format ref_storage_format; int depth; int max_jobs; int single_branch; @@ -1997,6 +2011,7 @@ struct update_data { #define UPDATE_DATA_INIT { \ .update_strategy = SUBMODULE_UPDATE_STRATEGY_INIT, \ .list = MODULE_LIST_INIT, \ + .ref_storage_format = REF_STORAGE_FORMAT_UNKNOWN, \ .recommend_shallow = -1, \ .references = STRING_LIST_INIT_DUP, \ .single_branch = -1, \ @@ -2132,6 +2147,9 @@ static int prepare_to_clone_next_submodule(const struct cache_entry *ce, expand_list_objects_filter_spec(suc->update_data->filter_options)); if (suc->update_data->require_init) strvec_push(&child->args, "--require-init"); + if (suc->update_data->ref_storage_format != REF_STORAGE_FORMAT_UNKNOWN) + strvec_pushf(&child->args, "--ref-format=%s", + ref_storage_format_to_name(suc->update_data->ref_storage_format)); strvec_pushl(&child->args, "--path", sub->path, NULL); strvec_pushl(&child->args, "--name", sub->name, NULL); strvec_pushl(&child->args, "--url", url, NULL); @@ -2562,6 +2580,9 @@ static void update_data_to_args(const struct update_data *update_data, for_each_string_list_item(item, &update_data->references) strvec_pushl(args, "--reference", item->string, NULL); } + if (update_data->ref_storage_format != REF_STORAGE_FORMAT_UNKNOWN) + strvec_pushf(args, "--ref-format=%s", + ref_storage_format_to_name(update_data->ref_storage_format)); if (update_data->filter_options && update_data->filter_options->choice) strvec_pushf(args, "--filter=%s", expand_list_objects_filter_spec( @@ -2737,6 +2758,7 @@ static int module_update(int argc, const char **argv, const char *prefix) struct update_data opt = UPDATE_DATA_INIT; struct list_objects_filter_options filter_options = LIST_OBJECTS_FILTER_INIT; + const char *ref_storage_format = NULL; int ret; struct option module_update_options[] = { OPT__SUPER_PREFIX(&opt.super_prefix), @@ -2760,6 +2782,8 @@ static int module_update(int argc, const char **argv, const char *prefix) SM_UPDATE_REBASE), OPT_STRING_LIST(0, "reference", &opt.references, N_("repo"), N_("reference repository")), + OPT_STRING(0, "ref-format", &ref_storage_format, N_("format"), + N_("specify the reference format to use")), OPT_BOOL(0, "dissociate", &opt.dissociate, N_("use --reference only while cloning")), OPT_INTEGER(0, "depth", &opt.depth, @@ -2803,6 +2827,12 @@ static int module_update(int argc, const char **argv, const char *prefix) module_update_options); } + if (ref_storage_format) { + opt.ref_storage_format = ref_storage_format_by_name(ref_storage_format); + if (opt.ref_storage_format == REF_STORAGE_FORMAT_UNKNOWN) + die(_("unknown ref storage format '%s'"), ref_storage_format); + } + opt.filter_options = &filter_options; opt.prefix = prefix; diff --git a/git-submodule.sh b/git-submodule.sh index 7f9582d923..dcbe943071 100755 --- a/git-submodule.sh +++ b/git-submodule.sh @@ -268,6 +268,14 @@ cmd_update() -r|--rebase) rebase=1 ;; + --ref-format) + case "$2" in '') usage ;; esac + ref_format="--ref-format=$2" + shift + ;; + --ref-format=*) + ref_format="$1" + ;; --reference) case "$2" in '') usage ;; esac reference="--reference=$2" @@ -349,6 +357,7 @@ cmd_update() ${rebase:+--rebase} \ ${merge:+--merge} \ ${checkout:+--checkout} \ + ${ref_format:+"$ref_format"} \ ${reference:+"$reference"} \ ${dissociate:+"--dissociate"} \ ${depth:+"$depth"} \ diff --git a/t/t7424-submodule-mixed-ref-formats.sh b/t/t7424-submodule-mixed-ref-formats.sh new file mode 100755 index 0000000000..de84007554 --- /dev/null +++ b/t/t7424-submodule-mixed-ref-formats.sh @@ -0,0 +1,41 @@ +#!/bin/sh + +test_description='submodules handle mixed ref storage formats' + +. ./test-lib.sh + +test_ref_format () { + echo "$2" >expect && + git -C "$1" rev-parse --show-ref-format >actual && + test_cmp expect actual +} + +for OTHER_FORMAT in files reftable +do + if test "$OTHER_FORMAT" = "$GIT_DEFAULT_REF_FORMAT" + then + continue + fi + +test_expect_success 'setup' ' + git config set --global protocol.file.allow always +' + +test_expect_success 'clone submodules with different ref storage format' ' + test_when_finished "rm -rf submodule upstream downstream" && + + git init submodule && + test_commit -C submodule submodule-initial && + git init upstream && + git -C upstream submodule add "file://$(pwd)/submodule" && + git -C upstream commit -m "upstream submodule" && + + git clone --no-recurse-submodules "file://$(pwd)/upstream" downstream && + test_ref_format downstream "$GIT_DEFAULT_REF_FORMAT" && + git -C downstream submodule update --init --ref-format=$OTHER_FORMAT && + test_ref_format downstream/submodule "$OTHER_FORMAT" +' + +done + +test_done From patchwork Wed Aug 7 12:43:53 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13756208 Received: from fout8-smtp.messagingengine.com (fout8-smtp.messagingengine.com [103.168.172.151]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8E2121E6724 for ; Wed, 7 Aug 2024 12:43:57 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=103.168.172.151 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723034639; cv=none; b=IAdD3N6owiO/I4u/RGAbVKC+qUbZhdsnHFO554KP6tXuULTghOhx+m3x++1tpeecH7evi4coo9ITxH2jkrY9HxtrSAWiZvsnoPXHHSuV8R1If0T3UmiLqQKp11sELWjx5PYD0DbWCh/jYGJOcLMf+18bGClwBEhRgCMaR+jRU7g= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723034639; c=relaxed/simple; bh=IWHfI1sszDbDg4jcRh0APQ4qluqJJEL0Twxp3KhmWN0=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=FVFIFLWsx4T+CGeZWUSj1YJLuzNsN4+3cGYReIQSQ3n+/+qf+/31gMIy9gd1zxP2K7BcUrkgqBVp+rKm6IOIhty2ZP93IOSJtQJR3UOCDZ8fSTyoMK8UIjye63RHZDpB3Na1y9gJ38P5psC4aO7n99ZZLm5q4wRhKdLtoJC6ofc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=pks.im; spf=pass smtp.mailfrom=pks.im; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b=de5xAMu4; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=d5pjUBpC; arc=none smtp.client-ip=103.168.172.151 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=pks.im Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=pks.im Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b="de5xAMu4"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="d5pjUBpC" Received: from compute2.internal (compute2.nyi.internal [10.202.2.46]) by mailfout.nyi.internal (Postfix) with ESMTP id D69D7138EC5F; Wed, 7 Aug 2024 08:43:56 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute2.internal (MEProxy); Wed, 07 Aug 2024 08:43:56 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:subject :subject:to:to; s=fm3; t=1723034636; x=1723121036; bh=oBkxJsmiRp u5AJfgQXyjPmNcfBoUgXyKE9W30yFu63k=; b=de5xAMu4n8GBzFttaM9wEQOMva YIe4iuHC+I3Za6qaJx+bwoqw9E1p1DB9iD+ApVirbo54UKqu0yHGGM5JlBeTzdMn KfyTCiq1l7pzGiAMogRVPcGQdpfROjuEkCWjoyi5t2SfaPyy3u2tlwEsPmfhRFOO BTyS34wqoAJw5ejhAZV+0onixpg2jkQwHD0CaOyTkGLZh8W1I21ytzISu6mohbBu NnajmAVgA7k4xEub2ukSnFKHUaxPmOOvd5YVkAlymbrAMX7Ike/I9L/5/lJx+Z9C /y6G9N6IUIH6HaGffMHagoWX+LktpNupXdSrwn0ss20sH5eZkUkfJGPDVHMw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm3; t=1723034636; x=1723121036; bh=oBkxJsmiRpu5AJfgQXyjPmNcfBoU gXyKE9W30yFu63k=; b=d5pjUBpC+5aj5uN7XIInkCMkwnY5pqL6ouJCoC5a4JYN Bu8jkdQmF3GBzZKzvUh9sLSD22UDigBN76VpRGUO2PwTw/2kzDsKcfSQ/rfM+Sig r9gP+UWGMWK2R46Nz30Z1Rif8hSWAto91pv04giqrnSAlPliCGOZkuIOcTQH3quD IRhAdW0gSKYqr6yxDDys54+TREKaDkKGRn68OR59Ji/JV3sN2dCyaOdxW0gxFbGD XKJm6Ngp5HtbiV4MM3KTzku4jFhcWTd1GEI5VLOwNmdf3p4owRM8ZTFkwhmEtC7Q Us7WrvpF6Y7B1lss38rMcWjhJ9PM+rUe2RTZdzHwXw== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeeftddrledtgdehhecutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc fjughrpeffhffvvefukfhfgggtuggjsehgtderredttddvnecuhfhrohhmpefrrghtrhhi tghkucfuthgvihhnhhgrrhguthcuoehpshesphhkshdrihhmqeenucggtffrrghtthgvrh hnpeeukedtvedtffevleejtefgheehieegkeeluddvfeefgeehgfeltddtheejleffteen ucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpehpshesph hkshdrihhmpdhnsggprhgtphhtthhopedt X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Wed, 7 Aug 2024 08:43:55 -0400 (EDT) Received: by vm-mail (OpenSMTPD) with ESMTPSA id 5ef33583 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Wed, 7 Aug 2024 12:43:50 +0000 (UTC) Date: Wed, 7 Aug 2024 14:43:53 +0200 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Eric Sunshine , Jeppe =?utf-8?b?w5hsYW5k?= Subject: [PATCH 2/6] builtin/clone: propagate ref storage format to submodules Message-ID: References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: When recursively cloning a repository with a non-default ref storage format, e.g. by passing the `--ref-format=` option, then only the top-level repository will end up will end up using that ref storage format. All recursively cloned submodules will instead use the default format. While mixed-format constellations are expected to work alright, the outcome still is somewhat surprising as we have essentially ignored the user's request. Fix this by propagating the requested ref format to cloned submodules. Signed-off-by: Patrick Steinhardt --- builtin/clone.c | 10 ++++++++-- t/t7424-submodule-mixed-ref-formats.sh | 23 +++++++++++++++++++++++ 2 files changed, 31 insertions(+), 2 deletions(-) diff --git a/builtin/clone.c b/builtin/clone.c index af6017d41a..75b15b5773 100644 --- a/builtin/clone.c +++ b/builtin/clone.c @@ -729,7 +729,8 @@ static int git_sparse_checkout_init(const char *repo) return result; } -static int checkout(int submodule_progress, int filter_submodules) +static int checkout(int submodule_progress, int filter_submodules, + enum ref_storage_format ref_storage_format) { struct object_id oid; char *head; @@ -813,6 +814,10 @@ static int checkout(int submodule_progress, int filter_submodules) strvec_push(&cmd.args, "--no-fetch"); } + if (ref_storage_format != REF_STORAGE_FORMAT_UNKNOWN) + strvec_pushf(&cmd.args, "--ref-format=%s", + ref_storage_format_to_name(ref_storage_format)); + if (filter_submodules && filter_options.choice) strvec_pushf(&cmd.args, "--filter=%s", expand_list_objects_filter_spec(&filter_options)); @@ -1536,7 +1541,8 @@ int cmd_clone(int argc, const char **argv, const char *prefix) return 1; junk_mode = JUNK_LEAVE_REPO; - err = checkout(submodule_progress, filter_submodules); + err = checkout(submodule_progress, filter_submodules, + ref_storage_format); free(remote_name); strbuf_release(&reflog_msg); diff --git a/t/t7424-submodule-mixed-ref-formats.sh b/t/t7424-submodule-mixed-ref-formats.sh index de84007554..4e4991d04c 100755 --- a/t/t7424-submodule-mixed-ref-formats.sh +++ b/t/t7424-submodule-mixed-ref-formats.sh @@ -21,6 +21,29 @@ test_expect_success 'setup' ' git config set --global protocol.file.allow always ' +test_expect_success 'recursive clone propagates ref storage format' ' + test_when_finished "rm -rf submodule upstream downstream" && + + git init submodule && + test_commit -C submodule submodule-initial && + git init upstream && + git -C upstream submodule add "file://$(pwd)/submodule" && + git -C upstream commit -am "add submodule" && + + # The upstream repository and its submodule should be using the default + # ref format. + test_ref_format upstream "$GIT_DEFAULT_REF_FORMAT" && + test_ref_format upstream/submodule "$GIT_DEFAULT_REF_FORMAT" && + + # The cloned repositories should use the other ref format that we have + # specified via `--ref-format`. The option should propagate to cloned + # submodules. + git clone --ref-format=$OTHER_FORMAT --recurse-submodules \ + upstream downstream && + test_ref_format downstream "$OTHER_FORMAT" && + test_ref_format downstream/submodule "$OTHER_FORMAT" +' + test_expect_success 'clone submodules with different ref storage format' ' test_when_finished "rm -rf submodule upstream downstream" && From patchwork Wed Aug 7 12:43:58 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13756209 Received: from fhigh5-smtp.messagingengine.com (fhigh5-smtp.messagingengine.com [103.168.172.156]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D5D211E6733 for ; Wed, 7 Aug 2024 12:44:02 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=103.168.172.156 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723034644; cv=none; b=OhE3NXrxiOV0Z0ILeBzwTOcTFS7PY0u8CsLW9aKgfSUn9LyG7UonkOm9TdHZ+ko/mgaeApREVzuVzzXsn6TZJuEQmsWHkE3Rc3JK61PtLgse2HdlqyPD5kNg+FZEO8KspgjBCLIuXOwHc1wgaChJdAudVFSZCCKuosUc08NxfFI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723034644; c=relaxed/simple; bh=quIH67V2kLMxx/cRqh/5P7CROVHKf4oYvsqDtc0iVqE=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=ePwdAzI/KhtYCt3Hf3gjBmZbkP323RDi7Q7Aiz3FtFZi++omAs2UKVejvAbLM7H7wVtBcSdm4eJULiYUoCb7GoBSsioKwJW3yEUvXrvW1mGc/3ogUtrnQoSTLHqIex/fgGokiqIcbKLOzBqnuYAHH2zS1oNEKNpgKlnRBl9lqcI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=pks.im; spf=pass smtp.mailfrom=pks.im; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b=bpvzBM4v; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=Kzmkh/rC; arc=none smtp.client-ip=103.168.172.156 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=pks.im Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=pks.im Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b="bpvzBM4v"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="Kzmkh/rC" Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailfhigh.nyi.internal (Postfix) with ESMTP id 07ED011509B2; Wed, 7 Aug 2024 08:44:02 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute3.internal (MEProxy); Wed, 07 Aug 2024 08:44:02 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:subject :subject:to:to; s=fm3; t=1723034642; x=1723121042; bh=vaMQfGSUuy 2czQjfk6+UAcxU12brGGmWkqv1dWJ9IVw=; b=bpvzBM4vvoG0Bn7NlUQHOIyS13 uv9FZcdr/nriYscmcOFWeQ9kYKa5C96tP0Pghlxa7a0dQKr22/+LSGSFfg0oPSwq 9OLy6xBrX4v76GBj8w/ZzvR6SOrsNXKA+WneP5kdOUrkTr0C3EusDSWrds/2Mrt0 EYtG8j4RQjz6UVaGO6kMtSWLR5679bx2ztkiu5DiwzwNqRziuc4fOpQtEbexkKtQ kys2DwZm1tBql18Gvq0dS44c8xu5Vm3We0AA+VbjuI+B2EAdfJFu9HvFXsp+KaVJ MqElmjOp2rEgbAysOC7EtxMU+bTEd6IhUVnfgouK6+SuN5heE9kVQBUWgi2Q== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm3; t=1723034642; x=1723121042; bh=vaMQfGSUuy2czQjfk6+UAcxU12br GGmWkqv1dWJ9IVw=; b=Kzmkh/rCv9dIURD0QGJblv6afdV/ULZHeHTHEtA5Ghx8 gCYHFillxoP1EkB2mzN02QF68SpdQJwnYXmIXGNqdxUx3KAluVyFqNHotuVwIt46 SRqvSMKZw8UDbzXNq4pMMGGBbQsr8iiCxQPANJcuRK+w10PINQ6gxpqM0owDUFtF XcDEZO4eGiB3xWFD7eKKUuih3NDWxj8vH1zJIKGUABlX9dgkLBwiiX7zJxiPFIS4 ayR521+0FtfI+BfdBHnyj7o1uWcO062sCAhFYNXNUEGk9WhRo058a1VKYqKLgskh N2tzW3WHwLvB8Zx04A/9RSOGjV/qdn1LoChOx0ioLQ== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeeftddrledtgdehhecutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc fjughrpeffhffvvefukfhfgggtuggjsehgtderredttdejnecuhfhrohhmpefrrghtrhhi tghkucfuthgvihhnhhgrrhguthcuoehpshesphhkshdrihhmqeenucggtffrrghtthgvrh hnpeetueevhffhudefvdegieeuieelgedthfegfedtueevjeejtdfgjeehudejuedtuden ucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpehpshesph hkshdrihhmpdhnsggprhgtphhtthhopedt X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Wed, 7 Aug 2024 08:44:01 -0400 (EDT) Received: by vm-mail (OpenSMTPD) with ESMTPSA id 6746a51d (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Wed, 7 Aug 2024 12:43:55 +0000 (UTC) Date: Wed, 7 Aug 2024 14:43:58 +0200 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Eric Sunshine , Jeppe =?utf-8?b?w5hsYW5k?= Subject: [PATCH 3/6] refs: fix ref storage format for submodule ref stores Message-ID: References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: When opening a submodule ref storage we accidentally use the ref storage format of the owning repository, not of the submodule repository. As submodules may have a different storage format than their parent repo this can lead to bugs when trying to access the submodule ref storage from the parent repository. One such bug was reported when performing a recursive pull with mixed ref stores, which fails with: $ git pull --recursive fatal: Unable to find current revision in submodule path 'path/to/sub' Fix the bug by using the submodule repository's ref storage format instead. Note that only the second added test fails without this fix. The other one is included regardless as it exercises other parts where we might end up accessing submodule ref stores. Reported-by: Jeppe Øland Signed-off-by: Patrick Steinhardt --- refs.c | 2 +- t/t7424-submodule-mixed-ref-formats.sh | 57 +++++++++++++++++++++++++- 2 files changed, 57 insertions(+), 2 deletions(-) diff --git a/refs.c b/refs.c index 915aeb4d1d..e4b1f4f8b1 100644 --- a/refs.c +++ b/refs.c @@ -2011,7 +2011,7 @@ struct ref_store *repo_get_submodule_ref_store(struct repository *repo, free(subrepo); goto done; } - refs = ref_store_init(subrepo, the_repository->ref_storage_format, + refs = ref_store_init(subrepo, subrepo->ref_storage_format, submodule_sb.buf, REF_STORE_READ | REF_STORE_ODB); register_ref_store_map(&repo->submodule_ref_stores, "submodule", diff --git a/t/t7424-submodule-mixed-ref-formats.sh b/t/t7424-submodule-mixed-ref-formats.sh index 4e4991d04c..998a5d82e9 100755 --- a/t/t7424-submodule-mixed-ref-formats.sh +++ b/t/t7424-submodule-mixed-ref-formats.sh @@ -18,7 +18,10 @@ do fi test_expect_success 'setup' ' - git config set --global protocol.file.allow always + git config set --global protocol.file.allow always && + # Some tests migrate the ref storage format, which does not work with + # reflogs at the time of writing these tests. + git config set --global core.logAllRefUpdates false ' test_expect_success 'recursive clone propagates ref storage format' ' @@ -59,6 +62,58 @@ test_expect_success 'clone submodules with different ref storage format' ' test_ref_format downstream/submodule "$OTHER_FORMAT" ' +test_expect_success 'status with mixed submodule ref storages' ' + test_when_finished "rm -rf submodule main" && + + git init submodule && + test_commit -C submodule submodule-initial && + git init main && + git -C main submodule add "file://$(pwd)/submodule" && + git -C main commit -m "add submodule" && + git -C main/submodule refs migrate --ref-format=$OTHER_FORMAT && + + # The main repository should use the default ref format now, whereas + # the submodule should use the other format. + test_ref_format main "$GIT_DEFAULT_REF_FORMAT" && + test_ref_format main/submodule "$OTHER_FORMAT" && + + cat >expect <<-EOF && + $(git -C main/submodule rev-parse HEAD) submodule (submodule-initial) + EOF + git -C main submodule status >actual && + test_cmp expect actual +' + +test_expect_success 'recursive pull with mixed formats' ' + test_when_finished "rm -rf submodule upstream downstream" && + + # Set up the initial structure with an upstream repository that has a + # submodule, as well as a downstream clone of the upstream repository. + git init submodule && + test_commit -C submodule submodule-initial && + git init upstream && + git -C upstream submodule add "file://$(pwd)/submodule" && + git -C upstream commit -m "upstream submodule" && + + # Clone the upstream repository such that the main repo and its + # submodules have different formats. + git clone --no-recurse-submodules "file://$(pwd)/upstream" downstream && + git -C downstream submodule update --init --ref-format=$OTHER_FORMAT && + test_ref_format downstream "$GIT_DEFAULT_REF_FORMAT" && + test_ref_format downstream/submodule "$OTHER_FORMAT" && + + # Update the upstream submodule as well as the owning repository such + # that we can do a recursive pull. + test_commit -C submodule submodule-update && + git -C upstream/submodule pull && + git -C upstream commit -am "update the submodule" && + + git -C downstream pull --recurse-submodules && + git -C upstream/submodule rev-parse HEAD >expect && + git -C downstream/submodule rev-parse HEAD >actual && + test_cmp expect actual +' + done test_done From patchwork Wed Aug 7 12:44:03 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13756210 Received: from fhigh5-smtp.messagingengine.com (fhigh5-smtp.messagingengine.com [103.168.172.156]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3C0481E6744 for ; Wed, 7 Aug 2024 12:44:06 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=103.168.172.156 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723034649; cv=none; b=Uewt2QRlPxr/xOMyChsC75+8NorWOWAcKMsWCxcN6AzoX3mpIdmFu1cNDUQLh77YUhWG8GLszYB9ZS7vwQBt2Kpmr3PYKdPLUVuwXKrrBCaCFqPkiud9+eR9lTGUNf/d4k+RuaOfbTsYrCa985R7ysu0tlMlK3SjYldvhGAbnRE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723034649; c=relaxed/simple; bh=cSjsvq54kHdKX98EhYGQyRpXvjtcUEM+9WFMuWBdZeA=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=aL7f4lvTun96bolFe0Og/cWC5zD+k0fNJGbAhMaCgq19u5mOvADvEvKpDC8lp5kBLxW2KkZjcXH0Ugez16GlsVgptMlYSvnPKZib3ho+yUsTxjE1GumVChFWEpvLr/aEZ+keyg+de2TmeL25wkydwOAMp273Ff3ElyvS6VGcr+k= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=pks.im; spf=pass smtp.mailfrom=pks.im; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b=qoCpkYwq; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=kixGpZcB; arc=none smtp.client-ip=103.168.172.156 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=pks.im Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=pks.im Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b="qoCpkYwq"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="kixGpZcB" Received: from compute5.internal (compute5.nyi.internal [10.202.2.45]) by mailfhigh.nyi.internal (Postfix) with ESMTP id 2CFAA11508E7; Wed, 7 Aug 2024 08:44:06 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute5.internal (MEProxy); Wed, 07 Aug 2024 08:44:06 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:subject :subject:to:to; s=fm3; t=1723034646; x=1723121046; bh=1NS8aK/DNG ieuabCVBfZL4neZzfyFwW3rGOmHUtmYLA=; b=qoCpkYwquUrH7G+FM2Ixff5EQE jcbjO3x0h7Lc5ELz1qfuM8srVZUSdfrsP7p5kZUAZIwjFq4XC85t/D1hOyBX+4NM WXWSQqI9eSfk44Zso5pSkItQFxgj/oBbb3kOgr+0GXXj3Tgw5YYWemwwlAX8NKM3 ZklhZnICPXhUol8G8zoP4pxCCa88pSk1VcHJy2rwVn1iPlbC219h82fLA0d2uc3q swOx0wJzh7VACl0tZIx5ysdTfwXOdFvy2bA1UZq7PH3f1wFS6XJBjhqSIm71/ADg BDEijA6D7N7YeVo4CzDDFpwGYlR7VM5cxzVXTdeGVCe9awRgb4xhCTsQ6D4g== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm3; t=1723034646; x=1723121046; bh=1NS8aK/DNGieuabCVBfZL4neZzfy FwW3rGOmHUtmYLA=; b=kixGpZcBjWJnxG30ki0TD/W99ZJ3jPyNWJFSas9V5u4B RxE1wBbQU92prToajUMFWrF76n5yeI/yEOt0Jw9sqP+RCRZJnh33Qeq7F7j3PWJJ 90v92H9Cq762Zhk5+4eB6uLC5EqKtXeseGUNNwMw76+trAR3TghCb5N9fEJ5FBpO hZtYDk1GLAWxYZ9D8D17RnghkYxMMBx/lCIbSINzaoIQdQ2RIgwj9pOrCQ1xHMKS 6LZ+EuXjqAquLVj8fxKNIHIX1PbdJPG7f7W4IFJTv7ZrNWBYEniHYKQ/NsslDYme p2XF3i0m4WsL19dyfgKENQlCeBpThUQYvJPaeCz9hg== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeeftddrledtgdehhecutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc fjughrpeffhffvvefukfhfgggtuggjsehgtderredttddvnecuhfhrohhmpefrrghtrhhi tghkucfuthgvihhnhhgrrhguthcuoehpshesphhkshdrihhmqeenucggtffrrghtthgvrh hnpeeukedtvedtffevleejtefgheehieegkeeluddvfeefgeehgfeltddtheejleffteen ucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpehpshesph hkshdrihhmpdhnsggprhgtphhtthhopedt X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Wed, 7 Aug 2024 08:44:05 -0400 (EDT) Received: by vm-mail (OpenSMTPD) with ESMTPSA id 2cd8c1c1 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Wed, 7 Aug 2024 12:44:00 +0000 (UTC) Date: Wed, 7 Aug 2024 14:44:03 +0200 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Eric Sunshine , Jeppe =?utf-8?b?w5hsYW5k?= Subject: [PATCH 4/6] submodule: fix leaking fetch tasks Message-ID: <8f8371c18a5f0a0f9b1412605199a21db334e90d.1723032100.git.ps@pks.im> References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: When done with a fetch task used for parallel fetches of submodules, we need to both call `fetch_task_release()` to release the task's contents and `free()` to release the task itself. Most sites do this already, but some only call `fetch_task_release()` and thus leak memory. While we could trivially fix this by adding the two missing calls to free(3P), the result would be that we always call both functions. Let's thus refactor the code such that `fetch_task_release()` also frees the structure itself. Rename it to `fetch_task_free()` accordingly. Signed-off-by: Patrick Steinhardt --- submodule.c | 17 +++++++---------- 1 file changed, 7 insertions(+), 10 deletions(-) diff --git a/submodule.c b/submodule.c index ab99a30253..f027a6455e 100644 --- a/submodule.c +++ b/submodule.c @@ -1496,7 +1496,7 @@ static const struct submodule *get_non_gitmodules_submodule(const char *path) return (const struct submodule *) ret; } -static void fetch_task_release(struct fetch_task *p) +static void fetch_task_free(struct fetch_task *p) { if (p->free_sub) free((void*)p->sub); @@ -1508,6 +1508,7 @@ static void fetch_task_release(struct fetch_task *p) FREE_AND_NULL(p->repo); strvec_clear(&p->git_args); + free(p); } static struct repository *get_submodule_repo_for(struct repository *r, @@ -1576,8 +1577,7 @@ static struct fetch_task *fetch_task_create(struct submodule_parallel_fetch *spf return task; cleanup: - fetch_task_release(task); - free(task); + fetch_task_free(task); return NULL; } @@ -1607,8 +1607,7 @@ get_fetch_task_from_index(struct submodule_parallel_fetch *spf, } else { struct strbuf empty_submodule_path = STRBUF_INIT; - fetch_task_release(task); - free(task); + fetch_task_free(task); /* * An empty directory is normal, @@ -1654,8 +1653,7 @@ get_fetch_task_from_changed(struct submodule_parallel_fetch *spf, cs_data->path, repo_find_unique_abbrev(the_repository, cs_data->super_oid, DEFAULT_ABBREV)); - fetch_task_release(task); - free(task); + fetch_task_free(task); continue; } @@ -1763,7 +1761,7 @@ static int fetch_start_failure(struct strbuf *err UNUSED, spf->result = 1; - fetch_task_release(task); + fetch_task_free(task); return 0; } @@ -1828,8 +1826,7 @@ static int fetch_finish(int retvalue, struct strbuf *err UNUSED, } out: - fetch_task_release(task); - + fetch_task_free(task); return 0; } From patchwork Wed Aug 7 12:44:07 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13756211 Received: from fhigh5-smtp.messagingengine.com (fhigh5-smtp.messagingengine.com [103.168.172.156]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 21BE11E6750 for ; Wed, 7 Aug 2024 12:44:11 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=103.168.172.156 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723034653; cv=none; b=aIy9pWWOcnP9+IObruLxQ+XO5qgad4pKgZod7ltOO9/e4t335FJAya0ckZ/qCHnY/3vmjN7JSnHzbjSJF9ZF+r2SBlNX/jOGUklppbYscLZ2tJ/Ir45gz69nLGNp3pZRAbzT1y9ADvzuexkx3/F7Gwcy1qD+J0ok8HC7+punvyA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723034653; c=relaxed/simple; bh=KxW4ti2nPZcEaNsu6/c2RUARrE+YVdoWJcdCiTpg5D4=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=SXQu0cOzHQpfRRaIOoqZxDWQSwfnxIYS4rneZ7jrC6v9ZPBO/oC/LSE5mVfRZmWGqI39ELAsk62rhtLgUcYEIO7W8vOrVKkOmiJYOWXBEuLr6xYoo0nGY4LZesglsezcjMle0rZ+uYjzWFXhqHi2zd9AjJII4/hSZf3gT+Jtzas= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=pks.im; spf=pass smtp.mailfrom=pks.im; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b=lHT/+AO7; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=AdD2IDVZ; arc=none smtp.client-ip=103.168.172.156 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=pks.im Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=pks.im Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b="lHT/+AO7"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="AdD2IDVZ" Received: from compute2.internal (compute2.nyi.internal [10.202.2.46]) by mailfhigh.nyi.internal (Postfix) with ESMTP id 482FD1150011; Wed, 7 Aug 2024 08:44:11 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute2.internal (MEProxy); Wed, 07 Aug 2024 08:44:11 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:subject :subject:to:to; s=fm3; t=1723034651; x=1723121051; bh=sYU492Gyhi OO4cnF+eKfH2mqIsWPdr3imyaNaMMgyHk=; b=lHT/+AO7zywg22taF21HGVZrr3 5Z/xFSkgAQO3lTgGp6FcMxKdQG0n/IVyGd02RM85Gm6VDzQlxdvKB8fS/BVfq4es TxOmDB8lR+Du1aTD39lWEDbYnFAuH9Y0ux6h3/4CI3k/SpjhWgFRrkNrkhiryP/A CoqaZ3dnBUz6di4++x9wHniqTU1h6WMttAMTGJQ5UD+KVQJFiJ8m4nC5laT1NEPv 9cuEJZwbmBN/1m58Pk6ehfCuW3ZnqOr348p90CJlyeB+QwRck+EB6iy7bHgmXSCm nbxG6bmEhAnfYPtisZ2ZrHiiCSev1rvHzdrJZDetvMbo7uMuv+HBzY7fx7+g== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm3; t=1723034651; x=1723121051; bh=sYU492GyhiOO4cnF+eKfH2mqIsWP dr3imyaNaMMgyHk=; b=AdD2IDVZjw6sA7zoLqyxzCuzpcsyUdqIs63Z2X/Ywuds F4z4q6LR9YoZqO9q1DdIw4bHHmeZI051k2rdCGHfWHXyJhuNC8ucyP8jlCskumX/ rw4AW/+ahS4ylhp+RKGOIvlZrxAbDap3PSsjJmfQJh5VW9l8R6GRt2fSVq+9Zhz3 D+9v2Iz1Br8VndPJKrKnvG91ald7IXEWhif7ChSwfoHrfFPDiyDzn0dq3Bfxb5ET uIK6dZT5/lix565FVP0NDbyIsmSpytfy1EWFBz1d5mxxiYHoDlPDonw4RM76evTF NugfQ1f4AqpXtlnNqZdkXL0EPgODhr2ZjsbgpOtHnw== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeeftddrledtgdehiecutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc fjughrpeffhffvvefukfhfgggtuggjsehgtderredttddvnecuhfhrohhmpefrrghtrhhi tghkucfuthgvihhnhhgrrhguthcuoehpshesphhkshdrihhmqeenucggtffrrghtthgvrh hnpeeukedtvedtffevleejtefgheehieegkeeluddvfeefgeehgfeltddtheejleffteen ucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpehpshesph hkshdrihhmpdhnsggprhgtphhtthhopedt X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Wed, 7 Aug 2024 08:44:10 -0400 (EDT) Received: by vm-mail (OpenSMTPD) with ESMTPSA id c97814c8 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Wed, 7 Aug 2024 12:44:04 +0000 (UTC) Date: Wed, 7 Aug 2024 14:44:07 +0200 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Eric Sunshine , Jeppe =?utf-8?b?w5hsYW5k?= Subject: [PATCH 5/6] submodule: fix leaking seen submodule names Message-ID: <732142aaa6da70f7840eb0c4ae82ebba16f05c55.1723032100.git.ps@pks.im> References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: We keep track of submodules we have already seen via a string map such that we don't process the same submodule twice. We never free that map though, causing a memory leak. Fix this leak by clearing the map. Signed-off-by: Patrick Steinhardt --- submodule.c | 1 + t/t5572-pull-submodule.sh | 1 + t/t7418-submodule-sparse-gitmodules.sh | 1 + 3 files changed, 3 insertions(+) diff --git a/submodule.c b/submodule.c index f027a6455e..13b8f8c19c 100644 --- a/submodule.c +++ b/submodule.c @@ -1880,6 +1880,7 @@ int fetch_submodules(struct repository *r, strvec_clear(&spf.args); out: free_submodules_data(&spf.changed_submodule_names); + string_list_clear(&spf.seen_submodule_names, 0); return spf.result; } diff --git a/t/t5572-pull-submodule.sh b/t/t5572-pull-submodule.sh index 51744521f7..916e58c166 100755 --- a/t/t5572-pull-submodule.sh +++ b/t/t5572-pull-submodule.sh @@ -5,6 +5,7 @@ test_description='pull can handle submodules' GIT_TEST_FATAL_REGISTER_SUBMODULE_ODB=1 export GIT_TEST_FATAL_REGISTER_SUBMODULE_ODB +TEST_PASSES_SANITIZE_LEAK=true . ./test-lib.sh . "$TEST_DIRECTORY"/lib-submodule-update.sh diff --git a/t/t7418-submodule-sparse-gitmodules.sh b/t/t7418-submodule-sparse-gitmodules.sh index dde11ecce8..e1d9bf2ee3 100755 --- a/t/t7418-submodule-sparse-gitmodules.sh +++ b/t/t7418-submodule-sparse-gitmodules.sh @@ -15,6 +15,7 @@ also by committing .gitmodules and then just removing it from the filesystem. GIT_TEST_FATAL_REGISTER_SUBMODULE_ODB=1 export GIT_TEST_FATAL_REGISTER_SUBMODULE_ODB +TEST_PASSES_SANITIZE_LEAK=true . ./test-lib.sh test_expect_success 'setup' ' From patchwork Wed Aug 7 12:44:12 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13756212 Received: from fhigh5-smtp.messagingengine.com (fhigh5-smtp.messagingengine.com [103.168.172.156]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 48BA61E6750 for ; Wed, 7 Aug 2024 12:44:16 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=103.168.172.156 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723034657; cv=none; b=Ps2sbBfy2XVrigF5/QgJez5jB9E/NtfPqC+7V9nqMiVIyEyjyhZmP2eTvafjrT5F+R4Y/h/dGo4rblrRve9+1PI70Pk4qX+xOth5zOfGuuDwgiOPR5C4WctdnoWNywtBK9ij8Lpgta6Es+R6N3wfpaKjUZnGMM3OTMdKWI6wRh8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723034657; c=relaxed/simple; bh=W5DOTHjijoz5McRGzmwS+BeSsMFV0eKFNNvmU/w+CIw=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=uiRSIdd7DLC7A0fnhK/++4naD1lxmSEkcddx+NnuKkcFra7/30xryALFFPJ3GSTHL41mbw5WFiEUb1k0fENBusyipNqq4ZC/BLNGd4iulyrUA83q90gtPM8E0+NUK1i9pI5ZCK+QVcCDJ07lswEP8R3sUU+UKYwEce6dBailhgY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=pks.im; spf=pass smtp.mailfrom=pks.im; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b=nx/hvRC2; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=sEYZZgmq; arc=none smtp.client-ip=103.168.172.156 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=pks.im Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=pks.im Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b="nx/hvRC2"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="sEYZZgmq" Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailfhigh.nyi.internal (Postfix) with ESMTP id 6D9D01150DE1; Wed, 7 Aug 2024 08:44:15 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute1.internal (MEProxy); Wed, 07 Aug 2024 08:44:15 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:subject :subject:to:to; s=fm3; t=1723034655; x=1723121055; bh=pCgjG/TpmS jm214B3qSKmejk1cq2wG8Mqr0xeHMNk6k=; b=nx/hvRC2RCk2jfOfaszBiibvvC UQlwEcvH28f5F75NvwtHc73M2wiCXMZsbNRF06eoJre3ctmvIjH30ZmLmt3xfWIW 8bB2E82LLyvUjYrGk4diz5xJ23zsX3/E2NBLLeENPq5pet/SRZ4yVNhjoxbuA14D 2pt13MI8QFclhzpucyg8IxsMLPRJuV9iK67CjY2Z1j3iIJroUspY+xpbIVV4vE6Z LqPKKEUnxOoDs3QSJgjmAyMrYMEdBXMZBiF2/ZKXscWwx9oSWtLiRdZP2Mlo6rdn lhmY7aBPmh/gsq/SH8kt9LJu7zwyYpYGYMTwcpmtOROBiVeg1d+90BiH+zGQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm3; t=1723034655; x=1723121055; bh=pCgjG/TpmSjm214B3qSKmejk1cq2 wG8Mqr0xeHMNk6k=; b=sEYZZgmq3sOjIbPMhZ/PkWmEb8zB3u/9+G+tRaiWgzV2 mlZuyG+j59ZjhCdwCufuHPJYAUU5Dni4z/JSdoBjysYXOh+eWbPoiD/ox6Ur0yrT BUI2olfEKSJYGC/dPqmb7GXoNd6ou5+Q7odLtgyZy8+XsSt0RPaJ9fZJKQDZeRdn 1xxD0jhZb2jdF2DVo6UV3Qy3A5bR/BRYMndQt69NQWvPDu0744+V5yeZhXqPL1rr 7ItKygn2AjADvW9vMmdiW1+Ty4iL9UTvy7kWeVuZPg28qm0XH+CStnMug2lFdn/o 7vDTb7YvyivHx7BClTALpLiib3R/fVsEvURqmc9P/A== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeeftddrledtgdehiecutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc fjughrpeffhffvvefukfhfgggtuggjsehgtderredttddvnecuhfhrohhmpefrrghtrhhi tghkucfuthgvihhnhhgrrhguthcuoehpshesphhkshdrihhmqeenucggtffrrghtthgvrh hnpeeukedtvedtffevleejtefgheehieegkeeluddvfeefgeehgfeltddtheejleffteen ucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpehpshesph hkshdrihhmpdhnsggprhgtphhtthhopedt X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Wed, 7 Aug 2024 08:44:14 -0400 (EDT) Received: by vm-mail (OpenSMTPD) with ESMTPSA id 945217ad (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Wed, 7 Aug 2024 12:44:09 +0000 (UTC) Date: Wed, 7 Aug 2024 14:44:12 +0200 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Eric Sunshine , Jeppe =?utf-8?b?w5hsYW5k?= Subject: [PATCH 6/6] object: fix leaking packfiles when closing object store Message-ID: <8dc7cc76d588ffe143d1d35e78e6eb86ed6d0b20.1723032100.git.ps@pks.im> References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: When calling `raw_object_store_clear()`, we close and free several resources associated with the object store. Part of that is to close and free all the packfiles, which is handled by `close_object_store()`. That function really only ends up closing the packfiles though, but it doesn't free them. And in fact it can't, as that function is being called via `run_command()` when `close_object_store = 1`, which is done e.g. when we execute git-maintenance(1). At that point, other structures may still have references on those packfiles, and thus we cannot free them here. So while it is in fact intentional that we really only close them, the result is a memory leak because `raw_object_store_clear()` does not free them, either. Fix the leak by freeing the packfiles in `raw_object_store_clear()`. Signed-off-by: Patrick Steinhardt --- object.c | 9 +++++++++ t/t7424-submodule-mixed-ref-formats.sh | 1 + 2 files changed, 10 insertions(+) diff --git a/object.c b/object.c index 0c0fcb76c0..d756c7f2ea 100644 --- a/object.c +++ b/object.c @@ -614,6 +614,15 @@ void raw_object_store_clear(struct raw_object_store *o) INIT_LIST_HEAD(&o->packed_git_mru); close_object_store(o); + + /* + * `close_object_store()` only closes the packfiles, but doesn't free + * them. We thus have to do this manually. + */ + for (struct packed_git *p = o->packed_git, *next; p; p = next) { + next = p->next; + free(p); + } o->packed_git = NULL; hashmap_clear(&o->pack_map); diff --git a/t/t7424-submodule-mixed-ref-formats.sh b/t/t7424-submodule-mixed-ref-formats.sh index 998a5d82e9..812496dc3b 100755 --- a/t/t7424-submodule-mixed-ref-formats.sh +++ b/t/t7424-submodule-mixed-ref-formats.sh @@ -2,6 +2,7 @@ test_description='submodules handle mixed ref storage formats' +TEST_PASSES_SANITIZE_LEAK=true . ./test-lib.sh test_ref_format () {