From patchwork Mon Jun 6 19:55:20 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12870805 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2DD12C433EF for ; Mon, 6 Jun 2022 19:55:52 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233000AbiFFTzv (ORCPT ); Mon, 6 Jun 2022 15:55:51 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37528 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232850AbiFFTzc (ORCPT ); Mon, 6 Jun 2022 15:55:32 -0400 Received: from mail-wr1-x433.google.com (mail-wr1-x433.google.com [IPv6:2a00:1450:4864:20::433]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 50CA729815 for ; Mon, 6 Jun 2022 12:55:30 -0700 (PDT) Received: by mail-wr1-x433.google.com with SMTP id h5so21283254wrb.0 for ; Mon, 06 Jun 2022 12:55:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=8NIPRl4i6HIsx2lAWw8AbAOtbptqiahyMLImaHqkIII=; b=p2vmKGb9mlvBtDGz5WzAwBLULt9gvcraZ4xUzVXWeqb82o2frWpt6GBTE4x1ozR3vx 3VgOU/TySgms39NkAPMlX2lwpDOzvyGnK3wUGxg4fbIXGKkaj5eoFxPy1SEn8LDusX6n WB6J6eAWrklnQO1yp/+xm6stddbuhYoidw4OwLWcbxN74IJTmbCuNZikZNap3Whk8DS5 MzxLSiR+flPOLHexlj8YoMyqldCtVHbnoef8A+UgKgPsifOf+OjK2ZmUHOOtrjVIFJhD wEpcqNtlVBySlvjDHF0yj9z/bok6lQtffEJCNqkI3uPjojDRzgXT75URW+c/vfbiA42L 5uBQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=8NIPRl4i6HIsx2lAWw8AbAOtbptqiahyMLImaHqkIII=; b=5eO9VZ4uz9PtEwYkxwHJ6AH1KG6DTL6EqIzfyWmPxLJAqh2TdbmQC42obgeiMay3Zy R3xexB5yAgpDj4zaDf9xrjlPGO0mIaMlAYTtlLZaClhN+n9ATE2atUpj75etFcD8A4XE JNks7UHQAov23El6QOwgJ9p2XMEzn6AQNtgmKTb8TLR4fZtZu8/HBslhfwODwpLSHcKL j5Se0b4RtB7Nj06U4oIJ46cmXvLw1cskjmFI2PPst2DsJ9pzoC23DlWhNiLY/PP8KVsu DBd2s42kxPMY9FlHdqYardQZTOB4DRtnL5i7iMZjMo5ViRuNH/yLjVT+87ZXQhzpHLiy c9dQ== X-Gm-Message-State: AOAM531RYNBDWWtCGTUe5vYwVIpWgRVvYsk6g2jq9d8V76si0/7S21d/ wHh7WX7G+TFZG9ETPxADjOVzaNGcc4UeuV+b X-Google-Smtp-Source: ABdhPJycBTNc9CXyXucbVeTobE9gz1l0AFFdY3W5vsPYZmdbOo7/DjBsFXtj/Fe6bM+Ml8zAfDY21Q== X-Received: by 2002:a5d:59ad:0:b0:20f:e3bc:c719 with SMTP id p13-20020a5d59ad000000b0020fe3bcc719mr24185454wrr.562.1654545327785; Mon, 06 Jun 2022 12:55:27 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id m2-20020adfe942000000b0020fcaba73bcsm16274243wrn.104.2022.06.06.12.55.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 06 Jun 2022 12:55:27 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Mon, 06 Jun 2022 19:55:20 +0000 Subject: [PATCH 1/6] docs: document bundle URI standard Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, me@ttaylorr.com, newren@gmail.com, avarab@gmail.com, dyroneteng@gmail.com, Johannes.Schindelin@gmx.de, Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee Introduce the idea of bundle URIs to the Git codebase through an aspirational design document. This document includes the full design intended to include the feature in its fully-implemented form. This will take several steps as detailed in the Implementation Plan section. By committing this document now, it can be used to motivate changes necessary to reach these final goals. The design can still be altered as new information is discovered. Signed-off-by: Derrick Stolee --- Documentation/technical/bundle-uri.txt | 475 +++++++++++++++++++++++++ 1 file changed, 475 insertions(+) create mode 100644 Documentation/technical/bundle-uri.txt diff --git a/Documentation/technical/bundle-uri.txt b/Documentation/technical/bundle-uri.txt new file mode 100644 index 00000000000..6657ba079ab --- /dev/null +++ b/Documentation/technical/bundle-uri.txt @@ -0,0 +1,475 @@ +Bundle URIs +=========== + +Bundle URIs are locations where Git can download one or more bundles in +order to bootstrap the object database in advance of fetching the remaining +objects from a remote. + +One goal is to speed up clones and fetches for users with poor network +connectivity to the origin server. Another benefit is to allow heavy users, +such as CI build farms, to use local resources for the majority of Git data +and thereby reducing the load on the origin server. + +To enable the bundle URI feature, users can specify a bundle URI using +command-line options or the origin server can advertise one or more URIs +via a protocol v2 capability. + +Design Goals +------------ + +The bundle URI standard aims to be flexible enough to satisfy multiple +workloads. The bundle provider and the Git client have several choices in +how they create and consume bundle URIs. + +* Bundles can have whatever name the server desires. This name could refer + to immutable data by using a hash of the bundle contents. However, this + means that a new URI will be needed after every update of the content. + This might be acceptable if the server is advertising the URI (and the + server is aware of new bundles being generated) but would not be + ergonomic for users using the command line option. + +* The bundles could be organized specifically for bootstrapping full + clones, but could also be organized with the intention of bootstrapping + incremental fetches. The bundle provider must decide on one of several + organization schemes to minimize client downloads during incremental + fetches, but the Git client can also choose whether to use bundles for + either of these operations. + +* The bundle provider can choose to support full clones, partial clones, + or both. The client can detect which bundles are appropriate for the + repository's partial clone filter, if any. + +* The bundle provider can use a single bundle (for clones only), or a + list of bundles. When using a list of bundles, the provider can specify + whether or not the client needs _all_ of the bundle URIs for a full + clone, or if _any_ one of the bundle URIs is sufficient. This allows the + bundle provider to use different URIs for different geographies. + +* The bundle provider can organize the bundles using heuristics, such as + timestamps or tokens, to help the client prevent downloading bundles it + does not need. When the bundle provider does not provide these + heuristics, the client can use optimizations to minimize how much of the + data is downloaded. + +* The bundle provider does not need to be associated with the Git server. + The client can choose to use the bundle provider without it being + advertised by the Git server. + +* The client can choose to discover bundle providers that are advertised + by the Git server. This could happen during `git clone`, during + `git fetch`, both, or neither. The user can choose which combination + works best for them. + +* The client can choose to configure a bundle provider manually at any + time. The client can also choose to specify a bundle provider manually + as a command-line option to `git clone`. + +Each repository is different and every Git server has different needs. +Hopefully the bundle URI feature is flexible enough to satisfy all needs. +If not, then the feature can be extended through its versioning mechanism. + +Server requirements +------------------- + +To provide a server-side implementation of bundle servers, no other parts +of the Git protocol are required. This allows server maintainers to use +static content solutions such as CDNs in order to serve the bundle files. + +At the current scope of the bundle URI feature, all URIs are expected to +be HTTP(S) URLs where content is downloaded to a local file using a `GET` +request to that URL. The server could include authentication requirements +to those requests with the aim of triggering the configured credential +helper for secure access. (Future extensions could use "file://" URIs or +SSH URIs.) + +Assuming a `200 OK` response from the server, the content at the URL is +expected to be of one of two forms: + +1. Bundle: A Git bundle file of version 2 or higher. + +2. Bundle List: A plain-text file that is parsable using Git's + config file parser. This file describes one or more bundles that are + accessible from other URIs. + +Any other data provided by the server is considered erroneous. + +Bundle Lists +------------ + +The Git server can advertise bundle URIs using a set of `key=value` pairs. +A bundle URI can also serve a plain-text file in the Git config format +containing these same `key=value` pairs. In both cases, we consider this +to be a _bundle list_. The pairs specify information about the bundles +that the client can use to make decisions for which bundles to download +and which to ignore. + +A few keys focus on properties of the list itself. + +bundle.list.version:: + (Required) This value provides a version number for the table of + contents. If a future Git change enables a feature that needs the Git + client to react to a new key in the bundle list file, then this version + will increment. The only current version number is 1, and if any other + value is specified then Git will fail to use this file. + +bundle.list.mode:: + (Required) This value has one of two values: `all` and `any`. When `all` + is specified, then the client should expect to need all of the listed + bundle URIs that match their repository's requirements. When `any` is + specified, then the client should expect that any one of the bundle URIs + that match their repository's requirements will suffice. Typically, the + `any` option is used to list a number of different bundle servers + located in different geographies. + +bundle.list.heuristic:: + If this string-valued key exists, then the bundle list is designed to + work well with incremental `git fetch` commands. The heuristic signals + that there are additional keys available for each bundle that help + determine which subset of bundles the client should download. + +The remaining keys include an `` segment which is a server-designated +name for each available bundle. + +bundle..uri:: + (Required) This string value is the URI for downloading bundle ``. + If the URI begins with a protocol (`http://` or `https://`) then the URI + is absolute. Otherwise, the URI is interpreted as relative to the URI + used for the bundle list. If the URI begins with `/`, then that relative + path is relative to the domain name used for the bundle list. (This use + of relative paths is intended to make it easier to distribute a set of + bundles across a large number of servers or CDNs with different domain + names.) + +bundle..list:: + This boolean value indicates whether the client should expect the + content from this URI to be a list (if `true`) or a bundle (if `false`). + This is typically used when `bundle.list.mode` is `any`. + +bundle..filter:: + This string value represents an object filter that should also appear in + the header of this bundle. The server uses this value to differentiate + different kinds of bundles from which the client can choose those that + match their object filters. + +bundle..timestamp:: + This value is the number of seconds since Unix epoch (UTC) that this + bundle was created. This is used as an approximation of a point in time + that the bundle matches the data available at the origin server. This is + used when `bundle.list.heuristic=timestamp`. + +bundle..requires:: + This string value represents the ID of another bundle. When present, the + server is indicating that this bundle contains a thin packfile. If the + client does not have all necessary objects to unbundle this packfile, + then the client can download the bundle with the `requires` ID and try + again. (Note: it may be beneficial to allow the server to specify + multiple `requires` bundles.) This is used when + `bundle.list.heuristic=timestamp`. + +bundle..location:: + This string value advertises a real-world location from where the bundle + URI is served. This can be used to present the user with an option for + which bundle URI to use. This is only valuable when `bundle.list.mode` + is `any`. + +Here is an example bundle list using the Git config format: + +``` +[bundle "list"] + version = 1 + mode = all + heuristic = timestamp + +[bundle "2022-02-09-1644442601-daily"] + uri = https://bundles.fake.com/git/git/2022-02-09-1644442601-daily.bundle + timestamp = 1644442601 + requires = 2022-02-02-1643842562 + +[bundle "2022-02-02-1643842562"] + uri = https://bundles.fake.com/git/git/2022-02-02-1643842562.bundle + timestamp = 1643842562 + +[bundle "2022-02-09-1644442631-daily-blobless"] + uri = 2022-02-09-1644442631-daily-blobless.bundle + timestamp = 1644442631 + requires = 2022-02-02-1643842568-blobless + filter = blob:none + +[bundle "2022-02-02-1643842568-blobless"] + uri = /git/git/2022-02-02-1643842568-blobless.bundle + timestamp = 1643842568 + filter = blob:none +``` + +This example uses `bundle.list.mode=all` as well as the +`bundle..timestamp` heuristic. It also uses the `bundle..filter` +options to present two parallel sets of bundles: one for full clones and +another for blobless partial clones. + +Suppose that this bundle list was found at the URI +`https://bundles.fake.com/git/git/` and so the two blobless bundles have +the following fully-expanded URIs: + +* `https://bundles.fake.com/git/git/2022-02-09-1644442631-daily-blobless.bundle` +* `https://bundles.fake.com/git/git/2022-02-02-1643842568-blobless.bundle` + +Advertising Bundle URIs +----------------------- + +If a user knows a bundle URI for the repository they are cloning, then +they can specify that URI manually through a command-line option. However, +a Git host may want to advertise bundle URIs during the clone operation, +helping users unaware of the feature. + +The only thing required for this feature is that the server can advertise +one or more bundle URIs. This advertisement takes the form of a new +protocol v2 capability specifically for discovering bundle URIs. + +The client could choose an arbitrary bundle URI as an option _or_ select +the URI with lowest latency by some exploratory checks. It is up to the +bundle provider to decide if having multiple URIs is preferable to a +single URI that is geodistributed through server-side infrastructure. + +Cloning with Bundle URIs +------------------------ + +The primary need for bundle URIs is to speed up clones. The Git client +will interact with bundle URIs according to the following flow: + +1. The user specifies a bundle URI with the `--bundle-uri` command-line + option _or_ the client discovers a bundle list advertised by the + Git server. + +2. If the downloaded data from a bundle URI is a bundle, then the client + inspects the bundle headers to check that the negative commit OIDs are + present in the client repository. If some are missing, then the client + delays unbundling until other bundles have been unbundled, making those + OIDs present. When all required OIDs are present, the client unbundles + that data using a refspec. The default refspec is + `+refs/heads/*:refs/bundles/*`, but this can be configured. + +3. If the file is instead a bundle list, then the client inspects the + `bundle.list.mode` to see if the list is of the `all` or `any` form. + + a. If `bundle.list.mode=all`, then the client considers all bundle + URIs. The list is reduced based on the `bundle..filter` options + matching the client repository's partial clone filter. Then, all + bundle URIs are requested. If the `bundle..timestamp` heuristic + is provided, then the bundles are downloaded in reverse- + chronological order, stopping when a bundle has all required OIDs. + The bundles can then be unbundled in chronological order. The client + stores the latest timestamp as a heuristic for avoiding future + downloads if the bundle list does not advertise newer bundles. + + b. If `bundle.list.mode=any`, then the client can choose any one of the + bundle URIs to inspect. The client can use a variety of ways to + choose among these URIs. The client can also fallback to another URI + if the initial choice fails to return a result. + +Note that during a clone we expect that all bundles will be required, and +heuristics such as `bundle..timestamp` can be used to download bundles +in chronological order or in parallel. + +If a given bundle URI is a bundle list with a `bundle.list.heuristic` +value, then the client can choose to store that URI as its chosen bundle +URI. The client can then navigate directly to that URI during later `git +fetch` calls. + +When downloading bundle URIs, the client can choose to inspect the initial +content before committing to downloading the entire content. This may +provide enough information to determine if the URI is a bundle list or +a bundle. In the case of a bundle, the client may inspect the bundle +header to determine that all advertised tips are already in the client +repository and cancel the remaining download. + +Fetching with Bundle URIs +------------------------- + +When the client fetches new data, it can decide to fetch from bundle +servers before fetching from the origin remote. This could be done via a +command-line option, but it is more likely useful to use a config value +such as the one specified during the clone. + +The fetch operation follows the same procedure to download bundles from a +bundle list (although we do _not_ want to use parallel downloads here). We +expect that the process will end when all negative commit OIDs in a thin +bundle are already in the object database. + +When using the `timestamp` heuristic, the client can avoid downloading any +bundles if their timestamps are not larger than the stored timestamp. +After fetching new bundles, this local timestamp value is updated. + +If the bundle provider does not provide a heuristic, then the client +should attempt to inspect the bundle headers before downloading the full +bundle data in case the bundle tips already exist in the client +repository. + +Error Conditions +---------------- + +If the Git client discovers something unexpected while downloading +information according to a bundle URI or the bundle list found at that +location, then Git can ignore that data and continue as if it was not +given a bundle URI. The remote Git server is the ultimate source of truth, +not the bundle URI. + +Here are a few example error conditions: + +* The client fails to connect with a server at the given URI or a connection + is lost without any chance to recover. + +* The client receives a response other than `200 OK` (such as `404 Not Found`, + `401 Not Authorized`, or `500 Internal Server Error`). The client should + use the `credential.helper` to attempt authentication after the first + `401 Not Authorized` response, but a second such response is a failure. + +* The client receives data that is not parsable as a bundle or table of + contents. + +* The bundle list describes a directed cycle in the + `bundle..requires` links. + +* A bundle includes a filter that does not match expectations. + +* The client cannot unbundle the bundles because the negative commit OIDs + are not in the object database and there are no more + `bundle..requires` links to follow. + +There are also situations that could be seen as wasteful, but are not +error conditions: + +* The downloaded bundles contain more information than is requested by + the clone or fetch request. A primary example is if the user requests + a clone with `--single-branch` but downloads bundles that store every + reachable commit from all `refs/heads/*` references. This might be + initially wasteful, but perhaps these objects will become reachable by + a later ref update that the client cares about. + +* A bundle download during a `git fetch` contains objects already in the + object database. This is probably unavoidable if we are using bundles + for fetches, since the client will almost always be slightly ahead of + the bundle servers after performing its "catch-up" fetch to the remote + server. This extra work is most wasteful when the client is fetching + much more frequently than the server is computing bundles, such as if + the client is using hourly prefetches with background maintenance, but + the server is computing bundles weekly. For this reason, the client + should not use bundle URIs for fetch unless the server has explicitly + recommended it through the `bundle.list.forFetch = true` value. + +Implementation Plan +------------------- + +This design document is being submitted on its own as an aspirational +document, with the goal of implementing all of the mentioned client +features over the course of several patch series. Here is a potential +outline for submitting these features: + +1. Integrate bundle URIs into `git clone` with a `--bundle-uri` option. + This will include a new `git fetch --bundle-uri` mode for use as the + implementation underneath `git clone`. The initial version here will + expect a single bundle at the given URI. + +2. Implement the ability to parse a bundle list from a bundle URI and + update the `git fetch --bundle-uri` logic to properly distinguish + between `bundle.list.mode` options. Specifically design the feature so + that the config format parsing feeds a list of key-value pairs into the + bundle list logic. + +3. Create the `bundle-uri` protocol v2 verb so Git servers can advertise + bundle URIs using the key-value pairs. Plug into the existing key-value + input to the bundle list logic. Allow `git clone` to discover these + bundle URIs and bootstrap the client repository from the bundle data. + (This choice is an opt-in via a config option and a command-line + option.) + +4. Allow the client to understand the `bundle.list.forFetch` configuration + and the `bundle..timestamp` heuristic. When `git clone` discovers a + bundle URI with `bundle.list.forFetch=true`, it configures the client + repository to check that bundle URI during later `git fetch ` + commands. + +5. Allow clients to discover bundle URIs during `git fetch` and configure + a bundle URI for later fetches if `bundle.list.forFetch=true`. + +6. Implement the "inspect headers" heuristic to reduce data downloads when + the `bundle..timestamp` heuristic is not available. + +As these features are reviewed, this plan might be updated. We also expect +that new designs will be discovered and implemented as this feature +matures and becomes used in real-world scenarios. + +Related Work: Packfile URIs +--------------------------- + +The Git protocol already has a capability where the Git server can list +a set of URLs along with the packfile response when serving a client +request. The client is then expected to download the packfiles at those +locations in order to have a complete understanding of the response. + +This mechanism is used by the Gerrit server (implemented with JGit) and +has been effective at reducing CPU load and improving user performance for +clones. + +A major downside to this mechanism is that the origin server needs to know +_exactly_ what is in those packfiles, and the packfiles need to be available +to the user for some time after the server has responded. This coupling +between the origin and the packfile data is difficult to manage. + +Further, this implementation is extremely hard to make work with fetches. + +Related Work: GVFS Cache Servers +-------------------------------- + +The GVFS Protocol [2] is a set of HTTP endpoints designed independently of +the Git project before Git's partial clone was created. One feature of this +protocol is the idea of a "cache server" which can be colocated with build +machines or developer offices to transfer Git data without overloading the +central server. + +The endpoint that VFS for Git is famous for is the `GET /gvfs/objects/{oid}` +endpoint, which allows downloading an object on-demand. This is a critical +piece of the filesystem virtualization of that product. + +However, a more subtle need is the `GET /gvfs/prefetch?lastPackTimestamp=` +endpoint. Given an optional timestamp, the cache server responds with a list +of precomputed packfiles containing the commits and trees that were introduced +in those time intervals. + +The cache server computes these "prefetch" packfiles using the following +strategy: + +1. Every hour, an "hourly" pack is generated with a given timestamp. +2. Nightly, the previous 24 hourly packs are rolled up into a "daily" pack. +3. Nightly, all prefetch packs more than 30 days old are rolled up into + one pack. + +When a user runs `gvfs clone` or `scalar clone` against a repo with cache +servers, the client requests all prefetch packfiles, which is at most +`24 + 30 + 1` packfiles downloading only commits and trees. The client +then follows with a request to the origin server for the references, and +attempts to checkout that tip reference. (There is an extra endpoint that +helps get all reachable trees from a given commit, in case that commit +was not already in a prefetch packfile.) + +During a `git fetch`, a hook requests the prefetch endpoint using the +most-recent timestamp from a previously-downloaded prefetch packfile. +Only the list of packfiles with later timestamps are downloaded. Most +users fetch hourly, so they get at most one hourly prefetch pack. Users +whose machines have been off or otherwise have not fetched in over 30 days +might redownload all prefetch packfiles. This is rare. + +It is important to note that the clients always contact the origin server +for the refs advertisement, so the refs are frequently "ahead" of the +prefetched pack data. The missing objects are downloaded on-demand using +the `GET gvfs/objects/{oid}` requests, when needed by a command such as +`git checkout` or `git log`. Some Git optimizations disable checks that +would cause these on-demand downloads to be too aggressive. + +See Also +-------- + +[1] https://lore.kernel.org/git/RFC-cover-00.13-0000000000-20210805T150534Z-avarab@gmail.com/ + An earlier RFC for a bundle URI feature. + +[2] https://github.com/microsoft/VFSForGit/blob/master/Protocol.md + The GVFS Protocol From patchwork Mon Jun 6 19:55:21 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12870806 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1CD2BC433EF for ; Mon, 6 Jun 2022 19:55:56 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233028AbiFFTzx (ORCPT ); Mon, 6 Jun 2022 15:55:53 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40172 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232849AbiFFTzb (ORCPT ); Mon, 6 Jun 2022 15:55:31 -0400 Received: from mail-wr1-x42b.google.com (mail-wr1-x42b.google.com [IPv6:2a00:1450:4864:20::42b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9B6775C663 for ; Mon, 6 Jun 2022 12:55:30 -0700 (PDT) Received: by mail-wr1-x42b.google.com with SMTP id t13so21230374wrg.9 for ; Mon, 06 Jun 2022 12:55:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=roYviVZ8WQy8VHN136fWD/I4C5Ve+n/NlfvpptSSn5w=; b=jKXy3R8Q3jeAN2/7oOjC2BNashYX1PspUIG4pNdPiph91eSF2tZZc6kW7h/5T8e24U djlqeyYyjtrhlwWSq2iLP1kL2ANQAy3QNxMLsXd1Zm0bEGbZaM+eq0srIFT/20bRGYMf YB3gbYbNTfXbYH6rkahuDf2Q0e1lBYvcjt5lnqtXmagPBx6rADl41/2ESfgLj437o6W7 uIFZZmN+cwLJUouXLKM8llpzkOrTq8wzjI47EiysKBuuT4zosYGMwH7EjoLCMPbPefJS dBc/8sFKNVvzn8RuZC5UnWfcN7qLJts+JWhoDaxK+iKVnVetcb7SgXqLQkNSFE3HVH01 28dg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=roYviVZ8WQy8VHN136fWD/I4C5Ve+n/NlfvpptSSn5w=; b=ZTqtnygIZx89Ls9ukqibihAPkG9sM6a/yLEvsPjAllhvUGmvSL6NaINSqjyj4/ZLny +24D6utrDwjBEv8dvhJzA/mmbmnAkF+Gxcp7QpLATwq0JmXxVPyLnfxcwFJTTAvv2p9J RgVS4XMgOZkuLel98DakxSI7/DDrrhi5h6PrHSg1Lyr/NVO/paC9pwgP5rBiIk21e1Gp M1EovvfDBpvUHzKySKCe07cj4hom4eBSO2TtJTZCYgoFpf93nW3BZgNScx1DHtcEi6gP ouwggI0K+goWG4FUAt33REGiK+Y2i5vXKZxcH6LLljZVkKb9caf9RfTbWDOj4ETGy6dL QwJg== X-Gm-Message-State: AOAM5322Y07JPBrI7K/iJaogtO3j828uSwRBoTbGV/D6Jb+tnSbzVGUK ZC2MgP8GonoHwtFh8HRvlZM4w/d44TqeJ5s7 X-Google-Smtp-Source: ABdhPJwRFv1m/Y7xjMCYT4rYTQuabRsBYXR8wT8peMwSMBGn0vI5Xno/22Kpj7KprL4neyiUoozHRg== X-Received: by 2002:adf:e0c3:0:b0:20c:5672:9577 with SMTP id m3-20020adfe0c3000000b0020c56729577mr23545275wri.466.1654545328800; Mon, 06 Jun 2022 12:55:28 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id h11-20020a5d688b000000b0020e63ab5d78sm15999527wru.26.2022.06.06.12.55.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 06 Jun 2022 12:55:28 -0700 (PDT) Message-Id: <977f0af40fc5cf3f0a7f167e2d2a47099168f47f.1654545325.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Mon, 06 Jun 2022 19:55:21 +0000 Subject: [PATCH 2/6] remote-curl: add 'get' capability Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, me@ttaylorr.com, newren@gmail.com, avarab@gmail.com, dyroneteng@gmail.com, Johannes.Schindelin@gmx.de, Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee A future change will want a way to download a file over HTTP(S) using the simplest of download mechanisms. We do not want to assume that the server on the other side understands anything about the Git protocol but could be a simple static web server. Create the new 'get' capability for the remote helpers which advertises that the 'get' command is avalable. A caller can send a line containing 'get ' to download the file at into the file at . Signed-off-by: Derrick Stolee --- Documentation/gitremote-helpers.txt | 9 +++++++ remote-curl.c | 33 +++++++++++++++++++++++++ t/t5557-http-get.sh | 37 +++++++++++++++++++++++++++++ transport-helper.c | 5 +++- 4 files changed, 83 insertions(+), 1 deletion(-) create mode 100755 t/t5557-http-get.sh diff --git a/Documentation/gitremote-helpers.txt b/Documentation/gitremote-helpers.txt index 6f1e269ae43..ed8da428c98 100644 --- a/Documentation/gitremote-helpers.txt +++ b/Documentation/gitremote-helpers.txt @@ -168,6 +168,9 @@ Supported commands: 'list', 'import'. Can guarantee that when a clone is requested, the received pack is self contained and is connected. +'get':: + Can use the 'get' command to download a file from a given URI. + If a helper advertises 'connect', Git will use it if possible and fall back to another capability if the helper requests so when connecting (see the 'connect' command under COMMANDS). @@ -418,6 +421,12 @@ Supported if the helper has the "connect" capability. + Supported if the helper has the "stateless-connect" capability. +'get' :: + Downloads the file from the given `` to the given ``. If + `.temp` exists, then Git assumes that the `.temp` file is a + partial download from a previous attempt and will resume the + download from that position. + If a fatal error occurs, the program writes the error message to stderr and exits. The caller should expect that a suitable error message has been printed if the child closes the connection without diff --git a/remote-curl.c b/remote-curl.c index 67f178b1120..f005419f872 100644 --- a/remote-curl.c +++ b/remote-curl.c @@ -1276,6 +1276,34 @@ static void parse_fetch(struct strbuf *buf) strbuf_reset(buf); } +static void parse_get(struct strbuf *buf) +{ + struct http_get_options opts = { 0 }; + struct strbuf url = STRBUF_INIT; + struct strbuf path = STRBUF_INIT; + const char *p, *space; + + if (!skip_prefix(buf->buf, "get ", &p)) + die(_("http transport does not support %s"), buf->buf); + + space = strchr(p, ' '); + + if (!space) + die(_("protocol error: expected ' ', missing space")); + + strbuf_add(&url, p, space - p); + strbuf_addstr(&path, space + 1); + + if (http_get_file(url.buf, path.buf, &opts)) + die(_("failed to download file at URL '%s'"), url.buf); + + strbuf_release(&url); + strbuf_release(&path); + printf("\n"); + fflush(stdout); + strbuf_reset(buf); +} + static int push_dav(int nr_spec, const char **specs) { struct child_process child = CHILD_PROCESS_INIT; @@ -1549,9 +1577,14 @@ int cmd_main(int argc, const char **argv) printf("unsupported\n"); fflush(stdout); + } else if (skip_prefix(buf.buf, "get ", &arg)) { + parse_get(&buf); + fflush(stdout); + } else if (!strcmp(buf.buf, "capabilities")) { printf("stateless-connect\n"); printf("fetch\n"); + printf("get\n"); printf("option\n"); printf("push\n"); printf("check-connectivity\n"); diff --git a/t/t5557-http-get.sh b/t/t5557-http-get.sh new file mode 100755 index 00000000000..1fd4ded3eb1 --- /dev/null +++ b/t/t5557-http-get.sh @@ -0,0 +1,37 @@ +#!/bin/sh + +test_description='test downloading a file by URL' + +. ./test-lib.sh + +. "$TEST_DIRECTORY"/lib-httpd.sh +start_httpd + +test_expect_success 'get by URL: 404' ' + url="$HTTPD_URL/none.txt" && + cat >input <<-EOF && + capabilities + get $url file1 + EOF + + test_must_fail git remote-http $url $url err && + test_path_is_missing file1 && + grep "failed to download file at URL" err && + rm file1.temp +' + +test_expect_success 'get by URL: 200' ' + echo data >"$HTTPD_DOCUMENT_ROOT_PATH/exists.txt" && + + url="$HTTPD_URL/exists.txt" && + cat >input <<-EOF && + capabilities + get $url file2 + + EOF + + GIT_TRACE2_PERF=1 git remote-http $url $url no_private_update = 1; } else if (starts_with(capname, "object-format")) { data->object_format = 1; + } else if (!strcmp(capname, "get")) { + data->get = 1; } else if (mandatory) { die(_("unknown mandatory capability %s; this remote " "helper probably needs newer version of Git"), From patchwork Mon Jun 6 19:55:22 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12870804 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2832BC43334 for ; Mon, 6 Jun 2022 19:55:51 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232972AbiFFTzt (ORCPT ); Mon, 6 Jun 2022 15:55:49 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40208 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232852AbiFFTzc (ORCPT ); Mon, 6 Jun 2022 15:55:32 -0400 Received: from mail-wm1-x330.google.com (mail-wm1-x330.google.com [IPv6:2a00:1450:4864:20::330]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D013764D11 for ; Mon, 6 Jun 2022 12:55:31 -0700 (PDT) Received: by mail-wm1-x330.google.com with SMTP id r123-20020a1c2b81000000b0039c1439c33cso8423004wmr.5 for ; Mon, 06 Jun 2022 12:55:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=JJkBD4+BoKpJTXlEBLl6bQZVX+aCJpdJfA0oOWbRDZM=; b=PLGOMOHyXJNnAXxnwFHtKQeaZvp+tK4M5r+8nTskqhIPh9OFE/ayKy2nMtmkcY92d2 4kVi+hi4imzMKJce5B0aQ7XRylD7QSeTR0f+jF6hIKMSPTwNUqbVlXNEiPzLwuzn/cRe jY1AKdWg9AcT4XTw6jYg/Qc3GwlIgDZnrObYMuEEMN1JWKEsin/rldKEGSA6BQ3qvL8u PoJDXdziyLPsaoaymsbyVGAB7VM8O4/qJ+nK488Llz1r6HQ31fODipL1GK+Pd/ZjjD2s FrVmxaKiTiwJqmsHBehgKlJtzoUXNZZnQddgRhookz5NipZcouNelHqWnKqxZdAbGsY9 aDSg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=JJkBD4+BoKpJTXlEBLl6bQZVX+aCJpdJfA0oOWbRDZM=; b=brPaPq8Wu1qtNDbj0U8698wG0ZgtxqfMHH0qoZg4yLm4qzq6YEDrDKh1gf+0GjUirs kLxAFy7VGnnx1Rkepl+1J0X+bi4BHr8wPzK6LMYHZD4zTpfJ1ZsP9qJKrYPRv0WtmLsM +ej0bULrLiEyYXi3/tILc3LGUJCWp2EAuojApq6CiFAWuqQi9KC/21K3fwMosoNSRbWr txY5cRp9mM78C/1rA3tUBWPwRRnMXJmRpJowUjRaHzmvRy+9lXa4qcInHQzWeoDW5RxR V8wSBWwkZZYGeZEBwX9JfjuMh0qR2C11s0e37v7GBDfl/aAOVhAbkuk/wJvC4fxD7lbS nGFQ== X-Gm-Message-State: AOAM531hTzRTqZERERWp4mx64DwElc9UgWeTHWU/QmS7vuKgHqKkBrRK W9q+TzcqWMh9laZtE1wFTUQY4Cupcag9Ign3 X-Google-Smtp-Source: ABdhPJxVFf/o2rzBMdJG6OT+x3lbkJdRS/QpT3/nDJ77v157gGFUvJrIwcJ5laubmhFDf4wVFPt+pw== X-Received: by 2002:a1c:a301:0:b0:392:9bc5:203c with SMTP id m1-20020a1ca301000000b003929bc5203cmr24600011wme.67.1654545329901; Mon, 06 Jun 2022 12:55:29 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id l19-20020a1ced13000000b0039c1396b495sm18231343wmh.9.2022.06.06.12.55.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 06 Jun 2022 12:55:29 -0700 (PDT) Message-Id: <8a18acfbf05aca39c36302d0f5b7d1f7516f2113.1654545325.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Mon, 06 Jun 2022 19:55:22 +0000 Subject: [PATCH 3/6] bundle-uri: create basic file-copy logic Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, me@ttaylorr.com, newren@gmail.com, avarab@gmail.com, dyroneteng@gmail.com, Johannes.Schindelin@gmx.de, Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee Before implementing a way to fetch bundles into a repository, create the basic logic. Assume that the URI is actually a file path. Future logic will make this more careful to other protocols. For now, we also only succeed if the content at the URI is a bundle file, not a bundle list. Bundle lists will be implemented in a future change. Signed-off-by: Derrick Stolee --- Makefile | 1 + bundle-uri.c | 92 ++++++++++++++++++++++++++++++++++++++++++++++++++++ bundle-uri.h | 14 ++++++++ 3 files changed, 107 insertions(+) create mode 100644 bundle-uri.c create mode 100644 bundle-uri.h diff --git a/Makefile b/Makefile index f8bccfab5e9..8f27310836d 100644 --- a/Makefile +++ b/Makefile @@ -887,6 +887,7 @@ LIB_OBJS += blob.o LIB_OBJS += bloom.o LIB_OBJS += branch.o LIB_OBJS += bulk-checkin.o +LIB_OBJS += bundle-uri.o LIB_OBJS += bundle.o LIB_OBJS += cache-tree.o LIB_OBJS += cbtree.o diff --git a/bundle-uri.c b/bundle-uri.c new file mode 100644 index 00000000000..07b4bfe4e11 --- /dev/null +++ b/bundle-uri.c @@ -0,0 +1,92 @@ +#include "cache.h" +#include "bundle-uri.h" +#include "bundle.h" +#include "object-store.h" +#include "refs.h" +#include "run-command.h" + +static void find_temp_filename(struct strbuf *name) +{ + int fd; + /* + * Find a temporray filename that is available. This is briefly + * racy, but unlikely to collide. + */ + fd = odb_mkstemp(name, "bundles/tmp_uri_XXXXXX"); + if (fd < 0) + die(_("failed to create temporary file")); + close(fd); + unlink(name->buf); +} + +static int copy_uri_to_file(const char *uri, const char *file) +{ + /* Copy as a file */ + return copy_file(file, uri, 0444); +} + +static int unbundle_from_file(struct repository *r, const char *file) +{ + int result = 0; + int bundle_fd; + struct bundle_header header = BUNDLE_HEADER_INIT; + struct strvec extra_index_pack_args = STRVEC_INIT; + struct string_list_item *refname; + struct strbuf bundle_ref = STRBUF_INIT; + size_t bundle_prefix_len; + + if ((bundle_fd = read_bundle_header(file, &header)) < 0) + return 1; + + result = unbundle(r, &header, bundle_fd, &extra_index_pack_args); + + /* + * Convert all refs/heads/ from the bundle into refs/bundles/ + * in the local repository. + */ + strbuf_addstr(&bundle_ref, "refs/bundles/"); + bundle_prefix_len = bundle_ref.len; + + for_each_string_list_item(refname, &header.references) { + struct object_id *oid = refname->util; + struct object_id old_oid; + const char *branch_name; + int has_old; + + if (!skip_prefix(refname->string, "refs/heads/", &branch_name)) + continue; + + strbuf_setlen(&bundle_ref, bundle_prefix_len); + strbuf_addstr(&bundle_ref, branch_name); + + has_old = !read_ref(bundle_ref.buf, &old_oid); + update_ref("fetched bundle", bundle_ref.buf, oid, + has_old ? &old_oid : NULL, + REF_SKIP_OID_VERIFICATION, + UPDATE_REFS_MSG_ON_ERR); + } + + bundle_header_release(&header); + return result; +} + +int fetch_bundle_uri(struct repository *r, const char *uri) +{ + int result = 0; + struct strbuf filename = STRBUF_INIT; + + find_temp_filename(&filename); + if ((result = copy_uri_to_file(uri, filename.buf))) + goto cleanup; + + if ((result = !is_bundle(filename.buf, 0))) + goto cleanup; + + if ((result = unbundle_from_file(r, filename.buf))) + goto cleanup; + +cleanup: + unlink(filename.buf); + strbuf_release(&filename); + return result; +} diff --git a/bundle-uri.h b/bundle-uri.h new file mode 100644 index 00000000000..8a152f1ef14 --- /dev/null +++ b/bundle-uri.h @@ -0,0 +1,14 @@ +#ifndef BUNDLE_URI_H +#define BUNDLE_URI_H + +struct repository; + +/** + * Fetch data from the given 'uri' and unbundle the bundle data found + * based on that information. + * + * Returns non-zero if no bundle information is found at the given 'uri'. + */ +int fetch_bundle_uri(struct repository *r, const char *uri); + +#endif From patchwork Mon Jun 6 19:55:23 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12870802 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id AF62BC43334 for ; Mon, 6 Jun 2022 19:55:46 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232873AbiFFTzp (ORCPT ); Mon, 6 Jun 2022 15:55:45 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37384 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232788AbiFFTzg (ORCPT ); Mon, 6 Jun 2022 15:55:36 -0400 Received: from mail-wr1-x435.google.com (mail-wr1-x435.google.com [IPv6:2a00:1450:4864:20::435]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0A0586D96D for ; Mon, 6 Jun 2022 12:55:33 -0700 (PDT) Received: by mail-wr1-x435.google.com with SMTP id x17so21233785wrg.6 for ; Mon, 06 Jun 2022 12:55:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=DvUWcVg6QiFElQLPBSGxt1zjmUCpH8UE/q7c40PX5PA=; b=HDHuizG8HSLkRANR6AN9omdb8DLRxYhZgkB63qx/221+kin3sCK2t9tW04CICpihTH bfk1In9O+HlvI9S8BG1YGVLRTfEx0PlVpPjn2oDfPpvyt+vYfIWMj6r0Qr6wt7Z4Xf2Z 3cFl2Tx2cwIsF0tHQuk3vOaVkF5GOiNMqwlCu/QBezqGvUVyBW3YIfB/Kz973nZWmBrz LaCl4jFdFR3+Jtbh3N864skembpR1UGlh2hQw5aVop2HTfzMZMjPB1UirG1qtmoWmVTH FA1mmxIquGDkOoULkqLiV86Gklr8DUd+rZbVyaCDszBlDX4/jj3WHMppa3azNj3T+WAT /69Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=DvUWcVg6QiFElQLPBSGxt1zjmUCpH8UE/q7c40PX5PA=; b=x2FBD1FURSr2TfcxOJ7VDFDDvsUGIZk5PXJXhRrIZEObT5QxBVkTW2tVQh4msmUNWs 4hLHMNE51c7NdMOkqTAXEaBmWHEDHEQKKd+wv0RxFOK9mDEwSJSmC8pf+kmXddSh2aOo 1vSGyKgJcMJLa/kML0dr2It9oIx8SkiTWXbnigvmlLVCB+gMqImii5e4A2SkRtSZ/fho AettM0V7uM9IXvuJ3tIdWA534C5kVgcYaAagC0esWFKpOfOFN2uYG07iV5WJj7NENlfE MayiaH7d4FAIhXsv223vHzUtAkqk7Gz10lRiseuCaCZ3dHJmfzb5OldP7CTMLn7sV47m b4qQ== X-Gm-Message-State: AOAM530tzau+EnITYLnJd4vfVqgEDTVnVY0QapxyfYemJnnZMwsGrIwL YOUUD15ffXqIjlHgz3/heQoOp7CeXQwvvCIP X-Google-Smtp-Source: ABdhPJw/vr6LZDsdfrF8feNIkPKWZs7714sggxCKHmG7MlP/XFgvLDU9/VP6eeZrp9dN3qluhEagLg== X-Received: by 2002:a5d:6d0f:0:b0:213:1c53:6c85 with SMTP id e15-20020a5d6d0f000000b002131c536c85mr21465334wrq.18.1654545331125; Mon, 06 Jun 2022 12:55:31 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id z13-20020adfe54d000000b002103cfd2fbasm16990807wrm.65.2022.06.06.12.55.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 06 Jun 2022 12:55:30 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Mon, 06 Jun 2022 19:55:23 +0000 Subject: [PATCH 4/6] fetch: add --bundle-uri option Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, me@ttaylorr.com, newren@gmail.com, avarab@gmail.com, dyroneteng@gmail.com, Johannes.Schindelin@gmx.de, Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee Teach 'git fetch' a new --bundle-uri= option which changes the mode from fetching from a remote using the Git protocol to fetching a bundle from the given . See Documentation/technical/bundle-uri.txt for more information on the design of this feature. This implementation is limited to the most basic version of the feature. We expect the content at that URI to be a bundle file, not a bundle list. Bundle lists will be implemented later. This implementation is sufficient for a bundle provider to create a single bootstrap bundle for a large repository. The user would bootstrap a repository using a sequence of Git commands, such as: 1. git init && cd 2. git fetch --bundle-uri= 3. git remote add origin 4. git fetch origin 5. git checkout FETCH_HEAD Later changes will make this seamless within a 'git clone' command, but this implementation is large enough to delay that integration. Currently, this option supports local filenames. Other protocols will be added in the future. Signed-off-by: Derrick Stolee --- Documentation/fetch-options.txt | 5 +++++ Documentation/git-fetch.txt | 1 + builtin/fetch.c | 10 ++++++++++ bundle-uri.c | 8 ++++++-- t/t5558-fetch-bundle-uri.sh | 34 +++++++++++++++++++++++++++++++++ 5 files changed, 56 insertions(+), 2 deletions(-) create mode 100755 t/t5558-fetch-bundle-uri.sh diff --git a/Documentation/fetch-options.txt b/Documentation/fetch-options.txt index 622bd84768b..09bd1feeed8 100644 --- a/Documentation/fetch-options.txt +++ b/Documentation/fetch-options.txt @@ -317,3 +317,8 @@ endif::git-pull[] -6:: --ipv6:: Use IPv6 addresses only, ignoring IPv4 addresses. + +--bundle-uri=:: + Instead of fetching from a remote, fetch a bundle from the given + `` and unbundle the data into the local repository. The refs + in the bundle will be stored under the `refs/bundle/*` namespace. diff --git a/Documentation/git-fetch.txt b/Documentation/git-fetch.txt index e9d364669af..4fd8911b336 100644 --- a/Documentation/git-fetch.txt +++ b/Documentation/git-fetch.txt @@ -13,6 +13,7 @@ SYNOPSIS 'git fetch' [] 'git fetch' --multiple [] [( | )...] 'git fetch' --all [] +'git fetch' --bundle-uri= [] DESCRIPTION diff --git a/builtin/fetch.c b/builtin/fetch.c index e3791f09ed5..cb0d2fbe82c 100644 --- a/builtin/fetch.c +++ b/builtin/fetch.c @@ -29,6 +29,7 @@ #include "commit-graph.h" #include "shallow.h" #include "worktree.h" +#include "bundle-uri.h" #define FORCED_UPDATES_DELAY_WARNING_IN_MS (10 * 1000) @@ -37,6 +38,7 @@ static const char * const builtin_fetch_usage[] = { N_("git fetch [] "), N_("git fetch --multiple [] [( | )...]"), N_("git fetch --all []"), + N_("git fetch --bundle-uri= []"), NULL }; @@ -86,6 +88,7 @@ static struct string_list negotiation_tip = STRING_LIST_INIT_NODUP; static int fetch_write_commit_graph = -1; static int stdin_refspecs = 0; static int negotiate_only; +static const char *bundle_uri; static int git_fetch_config(const char *k, const char *v, void *cb) { @@ -224,6 +227,8 @@ static struct option builtin_fetch_options[] = { N_("write the commit-graph after fetching")), OPT_BOOL(0, "stdin", &stdin_refspecs, N_("accept refspecs from stdin")), + OPT_STRING(0, "bundle-uri", &bundle_uri, N_("uri"), + N_("download bundle data from the given URI instead of from a remote")), OPT_END() }; @@ -2181,6 +2186,11 @@ int cmd_fetch(int argc, const char **argv, const char *prefix) if (dry_run) write_fetch_head = 0; + if (bundle_uri) { + result = fetch_bundle_uri(the_repository, bundle_uri); + goto cleanup; + } + if (all) { if (argc == 1) die(_("fetch --all does not take a repository argument")); diff --git a/bundle-uri.c b/bundle-uri.c index 07b4bfe4e11..095779352e7 100644 --- a/bundle-uri.c +++ b/bundle-uri.c @@ -76,11 +76,15 @@ int fetch_bundle_uri(struct repository *r, const char *uri) struct strbuf filename = STRBUF_INIT; find_temp_filename(&filename); - if ((result = copy_uri_to_file(uri, filename.buf))) + if ((result = copy_uri_to_file(uri, filename.buf))) { + error(_("failed to download bundle from URI '%s'"), uri); goto cleanup; + } - if ((result = !is_bundle(filename.buf, 0))) + if ((result = !is_bundle(filename.buf, 0))) { + error(_("file at URI '%s' is not a bundle"), uri); goto cleanup; + } if ((result = unbundle_from_file(r, filename.buf))) goto cleanup; diff --git a/t/t5558-fetch-bundle-uri.sh b/t/t5558-fetch-bundle-uri.sh new file mode 100755 index 00000000000..381e56cac20 --- /dev/null +++ b/t/t5558-fetch-bundle-uri.sh @@ -0,0 +1,34 @@ +#!/bin/sh + +test_description='test fetching bundles with --bundle-uri' + +. ./test-lib.sh + +test_expect_success 'fail to fetch from non-existent file' ' + test_must_fail git fetch --bundle-uri="$(pwd)/does-not-exist" 2>err && + grep "failed to download bundle from URI" err +' + +test_expect_success 'fail to fetch from non-bundle file' ' + echo bogus >bogus && + test_must_fail git fetch --bundle-uri="$(pwd)/bogus" 2>err && + grep "is not a bundle" err +' + +test_expect_success 'create bundle' ' + git init fetch-from && + git -C fetch-from checkout -b topic && + test_commit -C fetch-from A && + test_commit -C fetch-from B && + git -C fetch-from bundle create B.bundle topic +' + +test_expect_success 'fetch file bundle' ' + git init fetch-to && + git -C fetch-to fetch --bundle-uri="$(pwd)/fetch-from/B.bundle" && + git -C fetch-to rev-parse refs/bundles/topic >actual && + git -C fetch-from rev-parse topic >expect && + test_cmp expect actual +' + +test_done From patchwork Mon Jun 6 19:55:24 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12870803 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D3380CCA473 for ; Mon, 6 Jun 2022 19:55:48 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232938AbiFFTzs (ORCPT ); Mon, 6 Jun 2022 15:55:48 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37550 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232864AbiFFTzf (ORCPT ); Mon, 6 Jun 2022 15:55:35 -0400 Received: from mail-wm1-x336.google.com (mail-wm1-x336.google.com [IPv6:2a00:1450:4864:20::336]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 09E5C6898C for ; Mon, 6 Jun 2022 12:55:33 -0700 (PDT) Received: by mail-wm1-x336.google.com with SMTP id o37-20020a05600c512500b0039c4ba4c64dso2573869wms.2 for ; Mon, 06 Jun 2022 12:55:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=hIYGdJt4f8CNfq862K1JDKESeUt40SEBwGrg50HJfWc=; b=iM5oFo6qDxaz7zuGDw9T9TMWZdJF/PWZOjUyOCk24yPwcUdfjT0zP4i7xs38Y9axdk JvvtZ/cm6vOdl1Qs3IHUHYrUv0x5SXfwZir0HoxFLoMzhv5Kg4rQnZckqiARy97tis+A USBye8mMbbbEzMMKIrplJSYwiPOiN8TqTKwih8dhEVlCH2NdWEr7+x2rKOrLHk9nHIK1 qiV0nZWh+jZakP5zq1IARY2c5gsDY4t12dyl/KISUmX6NZz7xqFWY7W+KYS0a5wE15R6 5vIHuR29MvYwRkm+cjEnf2wsdQA1Ocv1dSu/b7w1r0PktpQK/ttzxgg2Ke8jy7A3BbXa U6Gg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=hIYGdJt4f8CNfq862K1JDKESeUt40SEBwGrg50HJfWc=; b=IDyuxZasXqtD5wD1NNi03Im9/6lHHTiHlRqCjsabIADEgqVFl/f1yCqPOWZ0WUY9zO 37FUUjuBQ8fm2ZrpPxmnTV5XtleWI+11zgXZbDsFCEGQPWHmvLHe6TdtN8lH6f8S4Xlj gQklv9FBGJO0r716XIpoau7I0A7Rz9WlJXmgnjBaVwVoie1tr5J7PuvrZV7gkgrKoU43 eCqZ67QvELvjuneWGY6C6p1h6ELsovmSvKwiHrdYP30EawaGzQNLyIxGzGMnXabdWVZA OLF1RLvggeOKHp22/G2Wry/oSLdp2LYyGXHvtquTsLwYHxiBFOB2CK+TRPSP26DzPzwe oCZw== X-Gm-Message-State: AOAM533I0DC9h3zfJkkKhRpXS3htxVAgo+mgOmmF+RcBqwhrA7GD/sJF nasIQYbLgLLGsVslAS/gXSmDIBZF9seLBnfa X-Google-Smtp-Source: ABdhPJwY4lA7ai7mRRQu7yot5d733zfT0kYY4nza4vMmOs/w3CAbCK5X0OyUwroQMP85tGM+mU161w== X-Received: by 2002:a05:600c:34d2:b0:397:7209:c1f0 with SMTP id d18-20020a05600c34d200b003977209c1f0mr25106128wmq.132.1654545332224; Mon, 06 Jun 2022 12:55:32 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id k22-20020a05600c1c9600b0039c25ee5dbcsm16652103wms.14.2022.06.06.12.55.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 06 Jun 2022 12:55:31 -0700 (PDT) Message-Id: <923feef84e4aac03a5d451e4b7759cfa291ac747.1654545325.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Mon, 06 Jun 2022 19:55:24 +0000 Subject: [PATCH 5/6] bundle-uri: add support for http(s):// and file:// Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, me@ttaylorr.com, newren@gmail.com, avarab@gmail.com, dyroneteng@gmail.com, Johannes.Schindelin@gmx.de, Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee The previous change created the logic for copying a file by name. Now, first inspect the URI for an HTTP(S) prefix and use git-remote-https as the way to download the data at that URI. Otherwise, check to see if file:// is present and modify the prefix accordingly. Signed-off-by: Derrick Stolee --- bundle-uri.c | 65 ++++++++++++++++++++++++++++++++++++- t/t5558-fetch-bundle-uri.sh | 37 +++++++++++++++++++++ 2 files changed, 101 insertions(+), 1 deletion(-) diff --git a/bundle-uri.c b/bundle-uri.c index 095779352e7..576ced271cb 100644 --- a/bundle-uri.c +++ b/bundle-uri.c @@ -19,10 +19,73 @@ static void find_temp_filename(struct strbuf *name) unlink(name->buf); } +static int download_https_uri_to_file(const char *uri, const char *file) +{ + int result = 0; + struct child_process cp = CHILD_PROCESS_INIT; + FILE *child_in = NULL, *child_out = NULL; + struct strbuf line = STRBUF_INIT; + int found_get = 0; + + strvec_pushl(&cp.args, "git-remote-https", "origin", uri, NULL); + cp.in = -1; + cp.out = -1; + + if (start_command(&cp)) + return 1; + + child_in = fdopen(cp.in, "w"); + if (!child_in) { + result = 1; + goto cleanup; + } + + child_out = fdopen(cp.out, "r"); + if (!child_out) { + result = 1; + goto cleanup; + } + + fprintf(child_in, "capabilities\n"); + fflush(child_in); + + while (!strbuf_getline(&line, child_out)) { + if (!line.len) + break; + if (!strcmp(line.buf, "get")) + found_get = 1; + } + strbuf_release(&line); + + if (!found_get) { + result = error(_("insufficient capabilities")); + goto cleanup; + } + + fprintf(child_in, "get %s %s\n\n", uri, file); + +cleanup: + if (child_in) + fclose(child_in); + if (finish_command(&cp)) + return 1; + if (child_out) + fclose(child_out); + return result; +} + static int copy_uri_to_file(const char *uri, const char *file) { + const char *out; + if (skip_prefix(uri, "https:", &out) || + skip_prefix(uri, "http:", &out)) + return download_https_uri_to_file(uri, file); + + if (!skip_prefix(uri, "file://", &out)) + out = uri; + /* Copy as a file */ - return copy_file(file, uri, 0444); + return !!copy_file(file, out, 0); } static int unbundle_from_file(struct repository *r, const char *file) diff --git a/t/t5558-fetch-bundle-uri.sh b/t/t5558-fetch-bundle-uri.sh index 381e56cac20..919db6f4551 100755 --- a/t/t5558-fetch-bundle-uri.sh +++ b/t/t5558-fetch-bundle-uri.sh @@ -31,4 +31,41 @@ test_expect_success 'fetch file bundle' ' test_cmp expect actual ' +test_expect_success 'fetch file:// bundle' ' + git init fetch-file && + git -C fetch-file fetch --bundle-uri="file://$(pwd)/fetch-from/B.bundle" && + git -C fetch-file rev-parse refs/bundles/topic >actual && + git -C fetch-from rev-parse topic >expect && + test_cmp expect actual +' + +######################################################################### +# HTTP tests begin here + +. "$TEST_DIRECTORY"/lib-httpd.sh +start_httpd + +test_expect_success 'fail to fetch from non-existent HTTP URL' ' + test_must_fail git fetch --bundle-uri="$HTTPD_URL/does-not-exist" 2>err && + grep "failed to download bundle from URI" err +' + +test_expect_success 'fail to fetch from non-bundle HTTP URL' ' + echo bogus >"$HTTPD_DOCUMENT_ROOT_PATH/bogus" && + test_must_fail git fetch --bundle-uri="$HTTPD_URL/bogus" 2>err && + grep "is not a bundle" err +' + +test_expect_success 'fetch HTTP bundle' ' + cp fetch-from/B.bundle "$HTTPD_DOCUMENT_ROOT_PATH/B.bundle" && + git init fetch-http && + git -C fetch-http fetch --bundle-uri="$HTTPD_URL/B.bundle" && + git -C fetch-http rev-parse refs/bundles/topic >actual && + git -C fetch-from rev-parse topic >expect && + test_cmp expect actual +' + +# Do not add tests here unless they use the HTTP server, as they will +# not run unless the HTTP dependencies exist. + test_done From patchwork Mon Jun 6 19:55:25 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12870801 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E63D5C433EF for ; Mon, 6 Jun 2022 19:55:45 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232847AbiFFTzn (ORCPT ); Mon, 6 Jun 2022 15:55:43 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37332 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232792AbiFFTzg (ORCPT ); Mon, 6 Jun 2022 15:55:36 -0400 Received: from mail-wr1-x436.google.com (mail-wr1-x436.google.com [IPv6:2a00:1450:4864:20::436]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 12BE07983F for ; Mon, 6 Jun 2022 12:55:35 -0700 (PDT) Received: by mail-wr1-x436.google.com with SMTP id q26so10857814wra.1 for ; Mon, 06 Jun 2022 12:55:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=tJAYv2N4IRoqgWVKReeL+zXTG1mmfKgd5arVI+g3Idg=; b=GnhEQ61SHKX1xWCfNHHRyynBGE2dh/aDoaTo6Na3Ty3LhzIEnI5hZmDX7krBOGEW6p +9DabT+CWfw9+E+zasNbrilYgG2dDSjfqy1EzANb6IsVzthwUozkfsxOB+xPldN2/aak q+J4BFT7g5idSkRDLLqiHDn4UwjQPTFaajQJD4hwgYCNM0Hx+Lj5TOiusO5OEGs88cy9 DKForcGuj7S61LZgOyPBEtr6HNMBeZJkn3tY0zZqUiTLsKXr2x/hDG/vXajJCu9nwGh4 lvURRP+IbbHOfHNz7lB40N5ifwG0GEcidP3Y/tZAmgdkLXBOFEGqvWBx0DbMCrEvITFa YsZg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=tJAYv2N4IRoqgWVKReeL+zXTG1mmfKgd5arVI+g3Idg=; b=lThvtSckNvSyYpzYM+vfIU5TMUk6nzj/HfhYdFjp2gYvg5U7sApTkD93aZCXclU1Yp HOUe/OlUFGpfaBeXIfFAjWlWognMqgWsieVQTCjx/YhKHQt5z3J8Q7AquyZww2LfQ3Jy THyQmIwyqUuA4UZcJdjVClWLgQkpPksX2+IjjTiMnMq9tsfpSXwGlCzfDdSYP6Ha9MJ1 CzEuzMDOte/ZmpI3C5DKyeK+YVH09heiF/LaNWV8OcoB7XDLstiLvtiklJoRMBazhUrP J0j3W7rU2Fd0cbk/VNzewuaTWfqnHj/We5KmWtI5uQjn0rlSA4fhc0ync8cMOR9i+RgD P3KQ== X-Gm-Message-State: AOAM532wU4sgfyD+IO5aS/gghPwCDBEEHEY8gSUgZzFa0mSsObjk6PsH 3tkCOifZU4hU8W+sBiQMMA73AiugzQPSwKDV X-Google-Smtp-Source: ABdhPJzIr1PDCe0s4aBmRoxuY/c2I8v1u41AIgkW8qESZ1WxMrHhmEvXrGdc91Vf9VC4+VemGDHWmg== X-Received: by 2002:a5d:4fc8:0:b0:210:3520:7479 with SMTP id h8-20020a5d4fc8000000b0021035207479mr23853719wrw.610.1654545333323; Mon, 06 Jun 2022 12:55:33 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id s1-20020adfea81000000b00210320d9fbfsm19373924wrm.18.2022.06.06.12.55.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 06 Jun 2022 12:55:32 -0700 (PDT) Message-Id: <5ca02b841dbdb6392370797f3c5dace46d1e70ef.1654545325.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Mon, 06 Jun 2022 19:55:25 +0000 Subject: [PATCH 6/6] fetch: add 'refs/bundle/' to log.excludeDecoration Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, me@ttaylorr.com, newren@gmail.com, avarab@gmail.com, dyroneteng@gmail.com, Johannes.Schindelin@gmx.de, Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee When fetching from a bundle URI, the branches of that bundle are stored in a different ref namespace: refs/bundles/. This namespace is intended to assist with later 'git fetch' negotiations with a Git server, allowing the client to advertise which data it already has from a bundle URI. These references can be confusing for a user when they appear as a decoration in 'git log' output. Add "refs/bundles/" to the multi-valued log.excludeDecoration config value. This is similar to the way "refs/prefetch/" is hidden by background prefetch operations in 'git maintenance' as added by 96eaffebb (maintenance: set log.excludeDecoration durin prefetch, 2021-01-19). Signed-off-by: Derrick Stolee --- Documentation/fetch-options.txt | 3 ++- bundle-uri.c | 7 +++++++ t/t5558-fetch-bundle-uri.sh | 12 +++++++++--- 3 files changed, 18 insertions(+), 4 deletions(-) diff --git a/Documentation/fetch-options.txt b/Documentation/fetch-options.txt index 09bd1feeed8..8b801bcc2f3 100644 --- a/Documentation/fetch-options.txt +++ b/Documentation/fetch-options.txt @@ -321,4 +321,5 @@ endif::git-pull[] --bundle-uri=:: Instead of fetching from a remote, fetch a bundle from the given `` and unbundle the data into the local repository. The refs - in the bundle will be stored under the `refs/bundle/*` namespace. + in the bundle will be stored under the hidden `refs/bundle/*` + namespace. diff --git a/bundle-uri.c b/bundle-uri.c index 576ced271cb..a3ffe0d129e 100644 --- a/bundle-uri.c +++ b/bundle-uri.c @@ -1,6 +1,7 @@ #include "cache.h" #include "bundle-uri.h" #include "bundle.h" +#include "config.h" #include "object-store.h" #include "refs.h" #include "run-command.h" @@ -152,6 +153,12 @@ int fetch_bundle_uri(struct repository *r, const char *uri) if ((result = unbundle_from_file(r, filename.buf))) goto cleanup; + git_config_set_multivar_gently("log.excludedecoration", + "refs/bundle/", + "refs/bundle/", + CONFIG_FLAGS_FIXED_VALUE | + CONFIG_FLAGS_MULTI_REPLACE); + cleanup: unlink(filename.buf); strbuf_release(&filename); diff --git a/t/t5558-fetch-bundle-uri.sh b/t/t5558-fetch-bundle-uri.sh index 919db6f4551..563df6de5e3 100755 --- a/t/t5558-fetch-bundle-uri.sh +++ b/t/t5558-fetch-bundle-uri.sh @@ -28,7 +28,9 @@ test_expect_success 'fetch file bundle' ' git -C fetch-to fetch --bundle-uri="$(pwd)/fetch-from/B.bundle" && git -C fetch-to rev-parse refs/bundles/topic >actual && git -C fetch-from rev-parse topic >expect && - test_cmp expect actual + test_cmp expect actual && + + test_config log.excludedecoration refs/bundle/ ' test_expect_success 'fetch file:// bundle' ' @@ -36,7 +38,9 @@ test_expect_success 'fetch file:// bundle' ' git -C fetch-file fetch --bundle-uri="file://$(pwd)/fetch-from/B.bundle" && git -C fetch-file rev-parse refs/bundles/topic >actual && git -C fetch-from rev-parse topic >expect && - test_cmp expect actual + test_cmp expect actual && + + test_config log.excludedecoration refs/bundle/ ' ######################################################################### @@ -62,7 +66,9 @@ test_expect_success 'fetch HTTP bundle' ' git -C fetch-http fetch --bundle-uri="$HTTPD_URL/B.bundle" && git -C fetch-http rev-parse refs/bundles/topic >actual && git -C fetch-from rev-parse topic >expect && - test_cmp expect actual + test_cmp expect actual && + + test_config log.excludedecoration refs/bundle/ ' # Do not add tests here unless they use the HTTP server, as they will