From patchwork Wed Feb 23 12:35:23 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 12756869 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 40999C433EF for ; Wed, 23 Feb 2022 12:35:43 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240463AbiBWMgI (ORCPT ); Wed, 23 Feb 2022 07:36:08 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42334 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232550AbiBWMgE (ORCPT ); Wed, 23 Feb 2022 07:36:04 -0500 Received: from wout2-smtp.messagingengine.com (wout2-smtp.messagingengine.com [64.147.123.25]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5AC6D9D0F1 for ; Wed, 23 Feb 2022 04:35:33 -0800 (PST) Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailout.west.internal (Postfix) with ESMTP id 581393201805 for ; Wed, 23 Feb 2022 07:35:30 -0500 (EST) Received: from mailfrontend1 ([10.202.2.162]) by compute1.internal (MEProxy); Wed, 23 Feb 2022 07:35:30 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc :content-type:date:date:from:from:in-reply-to:message-id :mime-version:reply-to:sender:subject:subject:to:to; s=fm3; bh=V lmi5hZBRdTxxLlwnr45dTNITrDRVXd18OVLPZqNV4s=; b=KCKnf26p25uJ+TnUS gG3hENGuNurfXFWjj2IZ0Qi3LU449vYhR/sXlzKjDWA27knE+5d2Ib+3kMfSpVKb q1W7aY73dqvHAmmSkptCUUjKhSy6TG0UeBDwGx7YEMuNQUJ1ySyB7ycYvp+PcHKJ cyT6fUMcyPlp8+vo/dN60zCgHzgPtlSEcIPE5mTKWNrPQWRMDLZKrgfUBdj9tvSC PGuuWmn60ulKl00h0ljstO5yQJmTrSATMguI3mQLgPawwj+qchASwiWibmhFkjEN PpmfixMBy/JbJ8XEXStdi/B604njpe453/dKgHkuok5RlyWu1eCI/THFiOM3/iXT kwspQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:date:date:from:from :in-reply-to:message-id:mime-version:reply-to:sender:subject :subject:to:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm2; bh=Vlmi5hZBRdTxxLlwnr45dTNITrDRVXd18OVLPZqNV 4s=; b=QnlGtCipqm717Blj2j480yeVr2YnkoJj3ooU5dqic5GINAekrtwXwhdJD lSHZpx1l7VNvLfj7Tf+aOxI8WvLtzd580rH7DpkdmnCjZoeQOXlW/T0fFWWqu6Z8 FwR89CNl2Np41tcQZWughiugW0BDhdsl/o/xEuvxgnd0fwMgbhgu1iwjfW4fJNNj Mxd0ZOYszIjrbAhhDSeh2mT4LleXKmG/oyCLLLJzS+cci+inMkK8lToXN0pdg74o f6kQkplM20mCMI02gghCnrhuVWsV68mxSH/iO7JgCf2uxL9CwNu8d8Vabz+fy9Kn cGtkVzPwHy41gL0d6gWoWJvF8CwdQ== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvvddrledtgdegudcutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecunecujfgurhepfffhvffukfggtggusehgtderredttd dvnecuhfhrohhmpefrrghtrhhitghkucfuthgvihhnhhgrrhguthcuoehpshesphhkshdr ihhmqeenucggtffrrghtthgvrhhnpeejieefvdeuleffgfejudffvdeghfeigfejgfdvvd efudevffefveffhffgkeeiffenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhep mhgrihhlfhhrohhmpehpshesphhkshdrihhm X-ME-Proxy: Received: by mail.messagingengine.com (Postfix) with ESMTPA for ; Wed, 23 Feb 2022 07:35:29 -0500 (EST) Received: from localhost (ncase [10.192.0.11]) by vm-mail.pks.im (OpenSMTPD) with ESMTPSA id e7193f91 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO) for ; Wed, 23 Feb 2022 12:35:24 +0000 (UTC) Date: Wed, 23 Feb 2022 13:35:23 +0100 From: Patrick Steinhardt To: git@vger.kernel.org Subject: [PATCH 0/5] fetch: more optimizations for mirror fetches Message-ID: MIME-Version: 1.0 Content-Disposition: inline Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Hi, this is another patch series with the aim to speed up mirror fetches. It applies on top of e6ebfd0e8c (The sixth batch, 2022-02-18) with 3824153b23 (Merge branch 'ps/fetch-atomic' into next, 2022-02-18) merged into it to fix a conflict. The patch series is structured as follows: - Patch 1: Another conversion to use the commit-graph, this time to look up "want" and "want-ref" lines in git-upload-pack(1). - Patch 2: This skips useless commit lookups when we don't write FETCH_HEAD. These were only used to compute the correct order in which we append to FETCH_HEAD. - Patch 3-5: This optimizes the way we look up symbolic refs in the files backend. Currently we always search the packed-refs file, which cannot ever contain symbolic references. In total, this patch series brings down the time it takes to replicate one of our biggest repos in numbers of references from 75 seconds to 60 seconds. Patrick Patrick Steinhardt (5): upload-pack: look up "want" lines via commit-graph fetch: avoid lookup of commits when not appending to FETCH_HEAD refs: add ability for backends to special-case reading of symbolic refs remote: read symbolic refs via `refs_read_symbolic_ref()` refs/files-backend: optimize reading of symbolic refs builtin/fetch.c | 42 +++++++++++++++++++++++++++--------------- builtin/remote.c | 8 +++++--- refs.c | 17 +++++++++++++++++ refs.h | 3 +++ refs/debug.c | 1 + refs/files-backend.c | 33 ++++++++++++++++++++++++++++----- refs/packed-backend.c | 1 + refs/refs-internal.h | 16 ++++++++++++++++ remote.c | 14 +++++++------- upload-pack.c | 20 +++++++++++++++++--- 10 files changed, 122 insertions(+), 33 deletions(-)