From patchwork Wed Feb 2 12:51:48 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 12732871 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 31901C433EF for ; Wed, 2 Feb 2022 12:51:52 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S245063AbiBBMvv (ORCPT ); Wed, 2 Feb 2022 07:51:51 -0500 Received: from out2-smtp.messagingengine.com ([66.111.4.26]:46931 "EHLO out2-smtp.messagingengine.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230258AbiBBMvu (ORCPT ); Wed, 2 Feb 2022 07:51:50 -0500 Received: from compute5.internal (compute5.nyi.internal [10.202.2.45]) by mailout.nyi.internal (Postfix) with ESMTP id 7D54D5C013A; Wed, 2 Feb 2022 07:51:50 -0500 (EST) Received: from mailfrontend1 ([10.202.2.162]) by compute5.internal (MEProxy); Wed, 02 Feb 2022 07:51:50 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:date:date:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:sender:subject :subject:to:to; s=fm3; bh=F7iFXmbgAK1JaXvx26yhzNB1lHVT+oAEYqezXJ daO0c=; b=CAD23dyzbe4HVLrljdLnQX5ujdka4hWzhRLqxdyUkC65wq/7ZDmO15 4MtKmsXWWMv8J1kt2jUStp5eeuaNEYo4MFrThsuQ8r+yhksjFGK3+Az45K93iNyo U4J/E+pdAPnkDrr0JjNMEGjfsuu3rkcf5QD4auwEddIAe7VULhCQ+rd4sY1+mh+K /x5vsuEWVl3rNu26bGyYtu3bsSLiWBF/MqtLpbo2yXWAA0+rdc4kqdb/JgA6+rZS AFy/G72Kqk9UTIR9aBTM6o1PnqeMG7ZTaXijpGC9f9U005mtl7dv3HCG/cAsqX4Z qUeTNjkFYT9xI78Yn3ukNLgEnWuK6lPw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:date:date:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:sender:subject:subject:to:to:x-me-proxy:x-me-proxy :x-me-sender:x-me-sender:x-sasl-enc; s=fm2; bh=F7iFXmbgAK1JaXvx2 6yhzNB1lHVT+oAEYqezXJdaO0c=; b=ldFfItrL2f+rzy4GzjHGeesp3tz2E6v9I 4KakZ+ITecRDiGlL6EaYcunxiwicl7Sjs+A3p8GSo3ZhKE5mrlZVrvh1KrxEKkyL hb8dD2mgESMKEEKlGT7tQOm52VxQ4Tr1Gss675XmQsRpComUAxgmkhBN9QWfUU+n xjBIn+VbX+HBJR36FiLbe4c0icAgkCnrjMV6KpSGMqDjx8YcI8mt4+7k5k8JlsZl ryzAjAQEfqO3NnX/wkDmyp4+nJHKLdwGXbCXgm+eWoKF4zO7NuwCcrJOI2Vw9EcO 2TLy1sZXdqqZVm0nFzGUGwAmKx9kEO6X1ORwG+vW/Q0MYcsTR7WMQ== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvvddrgeehgdeghecutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc fjughrpeffhffvuffkfhggtggujgesghdtreertddtjeenucfhrhhomheprfgrthhrihgt khcuufhtvghinhhhrghrughtuceophhssehpkhhsrdhimheqnecuggftrfgrthhtvghrnh ephefgjeeuveejteduhefgffefffdvjeefjeeivdekfffgkeeugfehveetueefleeknecu vehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepphhssehpkh hsrdhimh X-ME-Proxy: Received: by mail.messagingengine.com (Postfix) with ESMTPA; Wed, 2 Feb 2022 07:51:49 -0500 (EST) Received: from localhost (ncase [10.192.0.11]) by vm-mail.pks.im (OpenSMTPD) with ESMTPSA id 9e8b8c8d (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Wed, 2 Feb 2022 12:51:49 +0000 (UTC) Date: Wed, 2 Feb 2022 13:51:48 +0100 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Taylor Blau , Derrick Stolee Subject: [PATCH v2 1/2] fetch-pack: use commit-graph when computing cutoff Message-ID: <6fac914f0fe77df4c3058340642bea2a45a850cd.1643806143.git.ps@pks.im> References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org During packfile negotiation we iterate over all refs announced by the remote side to check whether their IDs refer to commits already known to us. If a commit is known to us already, then its date is a potential cutoff point for commits we have in common with the remote side. There is potentially a lot of commits announced by the remote depending on how many refs there are in the remote repository, and for every one of them we need to search for it in our object database and, if found, parse the corresponding object to find out whether it is a candidate for the cutoff date. This can be sped up by trying to look up commits via the commit-graph first, which is a lot more efficient. Benchmarks in a repository with about 2,1 million refs and an up-to-date commit-graph show a 20% speedup when mirror-fetching: Benchmark 1: git fetch --atomic +refs/*:refs/* (v2.35.0) Time (mean ± σ): 75.264 s ± 1.115 s [User: 68.199 s, System: 10.094 s] Range (min … max): 74.145 s … 76.862 s 5 runs Benchmark 2: git fetch --atomic +refs/*:refs/* (HEAD) Time (mean ± σ): 62.350 s ± 0.854 s [User: 55.412 s, System: 9.976 s] Range (min … max): 61.224 s … 63.216 s 5 runs Summary 'git fetch --atomic +refs/*:refs/* (HEAD)' ran 1.21 ± 0.02 times faster than 'git fetch --atomic +refs/*:refs/* (v2.35.0)' Signed-off-by: Patrick Steinhardt --- fetch-pack.c | 28 ++++++++++++++++------------ 1 file changed, 16 insertions(+), 12 deletions(-) diff --git a/fetch-pack.c b/fetch-pack.c index dd6ec449f2..c5967e228e 100644 --- a/fetch-pack.c +++ b/fetch-pack.c @@ -696,26 +696,30 @@ static void mark_complete_and_common_ref(struct fetch_negotiator *negotiator, trace2_region_enter("fetch-pack", "parse_remote_refs_and_find_cutoff", NULL); for (ref = *refs; ref; ref = ref->next) { - struct object *o; + struct commit *commit; - if (!has_object_file_with_flags(&ref->old_oid, + commit = lookup_commit_in_graph(the_repository, &ref->old_oid); + if (!commit) { + struct object *o; + + if (!has_object_file_with_flags(&ref->old_oid, OBJECT_INFO_QUICK | - OBJECT_INFO_SKIP_FETCH_OBJECT)) - continue; - o = parse_object(the_repository, &ref->old_oid); - if (!o) - continue; + OBJECT_INFO_SKIP_FETCH_OBJECT)) + continue; + o = parse_object(the_repository, &ref->old_oid); + if (!o || o->type != OBJ_COMMIT) + continue; + + commit = (struct commit *)o; + } /* * We already have it -- which may mean that we were * in sync with the other side at some time after * that (it is OK if we guess wrong here). */ - if (o->type == OBJ_COMMIT) { - struct commit *commit = (struct commit *)o; - if (!cutoff || cutoff < commit->date) - cutoff = commit->date; - } + if (!cutoff || cutoff < commit->date) + cutoff = commit->date; } trace2_region_leave("fetch-pack", "parse_remote_refs_and_find_cutoff", NULL); From patchwork Wed Feb 2 12:51:52 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 12732872 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6BCB9C433F5 for ; Wed, 2 Feb 2022 12:51:59 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1344227AbiBBMv6 (ORCPT ); Wed, 2 Feb 2022 07:51:58 -0500 Received: from out2-smtp.messagingengine.com ([66.111.4.26]:44295 "EHLO out2-smtp.messagingengine.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1344230AbiBBMvz (ORCPT ); Wed, 2 Feb 2022 07:51:55 -0500 Received: from compute4.internal (compute4.nyi.internal [10.202.2.44]) by mailout.nyi.internal (Postfix) with ESMTP id 903545C013A; Wed, 2 Feb 2022 07:51:55 -0500 (EST) Received: from mailfrontend1 ([10.202.2.162]) by compute4.internal (MEProxy); Wed, 02 Feb 2022 07:51:55 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:date:date:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:sender:subject :subject:to:to; s=fm3; bh=ydSS7V3sA45Kb1BVH1BLS58jRWvrBsDMLxRgfP MapPk=; b=qFpZtYd/nrkLWVD+ojvIpY19MJa92BMNlRk16pqcp7D1s5b/ERkYQi hG07rsdgO6LJtBE6gIChQrj6CghqDwBcxK4t/EbhYEOmNkgs4upjijwxWRoXiLQX ew52lKx5evR1iNJA2Mkgr00I8r6Yq6HMaUJOvLCEch2A5DlXMd6KXinjAWB90e/F A+ib8otADjX2NuUZajtys7bMFNh+XAOp+pqaJlRkfSkZ8E7xdH+qg9POZk4LrZ1I RP9so8sRohfdh32YCoyccFVOg5+QGHa1qDxqH6xBKaxchxXJCWk1PsbQ40Ix3U8l XSNehuXFj+uUs4T9ga7Sojai9eWFKY0A== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:date:date:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:sender:subject:subject:to:to:x-me-proxy:x-me-proxy :x-me-sender:x-me-sender:x-sasl-enc; s=fm2; bh=ydSS7V3sA45Kb1BVH 1BLS58jRWvrBsDMLxRgfPMapPk=; b=iYwI6/q2tGSFbI5tkN9wtEKLpH4XEqP7B oxG31MGXGsV6IUSE2gmqK91PW8u1GiC/DPSpIY7t/ufLr88kJd6lzeKe7Nv4CvK5 F0/Ijh75F5KiFIPr5xzSdvSrIachzQIbp7nn0itB/Fa60aUVHgsafbN7hQ2HIJZM r1dmFTcOrrhrUmhzH65lVsZrBpEmVAxJlPxFZA1fql+b4XyHlfdpk/myjdI3slld YBDawIpy93ZP1n5Xu4zKY4mvzmIkz6niYLGAQY3jF8UmlrhyGwIA75Tr3jmv4uPN K5qFBxJVoL4EimMgXE34FNxtfMjRyVUV83voUIDaKZnMloQYn90Cw== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvvddrgeehgdeggecutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc fjughrpeffhffvuffkfhggtggujgesghdtreertddtjeenucfhrhhomheprfgrthhrihgt khcuufhtvghinhhhrghrughtuceophhssehpkhhsrdhimheqnecuggftrfgrthhtvghrnh ephefgjeeuveejteduhefgffefffdvjeefjeeivdekfffgkeeugfehveetueefleeknecu vehluhhsthgvrhfuihiivgepudenucfrrghrrghmpehmrghilhhfrhhomhepphhssehpkh hsrdhimh X-ME-Proxy: Received: by mail.messagingengine.com (Postfix) with ESMTPA; Wed, 2 Feb 2022 07:51:54 -0500 (EST) Received: from localhost (ncase [10.192.0.11]) by vm-mail.pks.im (OpenSMTPD) with ESMTPSA id 3fe5e561 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Wed, 2 Feb 2022 12:51:53 +0000 (UTC) Date: Wed, 2 Feb 2022 13:51:52 +0100 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Taylor Blau , Derrick Stolee Subject: [PATCH v2 2/2] fetch: skip computing output width when not printing anything Message-ID: <4b9bbcf7958da6db2be771f910974a6bffb2d94a.1643806143.git.ps@pks.im> References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org When updating references via git-fetch(1), then by default we report to the user which references have been changed. This output is formatted in a nice table such that the different columns are aligned. Because the first column contains abbreviated object IDs we thus need to iterate over all refs which have changed and compute the minimum length for their respective abbreviated hashes. While this effort makes sense in most cases, it is wasteful when the user passes the `--quiet` flag: we don't print the summary, but still compute the length. Skip computing the summary width when the user asked for us to be quiet. This gives us a small speedup of nearly 10% when doing a dry-run mirror-fetch in a repository with thousands of references being updated: Benchmark 1: git fetch --prune --dry-run +refs/*:refs/* (HEAD~) Time (mean ± σ): 34.048 s ± 0.233 s [User: 30.739 s, System: 4.640 s] Range (min … max): 33.785 s … 34.296 s 5 runs Benchmark 2: git fetch --prune --dry-run +refs/*:refs/* (HEAD) Time (mean ± σ): 30.768 s ± 0.287 s [User: 27.534 s, System: 4.565 s] Range (min … max): 30.432 s … 31.181 s 5 runs Summary 'git fetch --prune --dry-run +refs/*:refs/* (HEAD)' ran 1.11 ± 0.01 times faster than 'git fetch --prune --dry-run +refs/*:refs/* (HEAD~)' Signed-off-by: Patrick Steinhardt --- builtin/fetch.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/builtin/fetch.c b/builtin/fetch.c index 5f06b21f8e..ebbde5d56d 100644 --- a/builtin/fetch.c +++ b/builtin/fetch.c @@ -1093,12 +1093,15 @@ static int store_updated_refs(const char *raw_url, const char *remote_name, struct ref *rm; char *url; int want_status; - int summary_width = transport_summary_width(ref_map); + int summary_width = 0; rc = open_fetch_head(&fetch_head); if (rc) return -1; + if (verbosity >= 0) + summary_width = transport_summary_width(ref_map); + if (raw_url) url = transport_anonymize_url(raw_url); else @@ -1344,7 +1347,6 @@ static int prune_refs(struct refspec *rs, struct ref *ref_map, int url_len, i, result = 0; struct ref *ref, *stale_refs = get_stale_heads(rs, ref_map); char *url; - int summary_width = transport_summary_width(stale_refs); const char *dangling_msg = dry_run ? _(" (%s will become dangling)") : _(" (%s has become dangling)"); @@ -1373,6 +1375,8 @@ static int prune_refs(struct refspec *rs, struct ref *ref_map, } if (verbosity >= 0) { + int summary_width = transport_summary_width(stale_refs); + for (ref = stale_refs; ref; ref = ref->next) { struct strbuf sb = STRBUF_INIT; if (!shown_url) {