From patchwork Fri Sep 9 14:33:27 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12971873 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3D476ECAAD3 for ; Fri, 9 Sep 2022 14:33:47 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231208AbiIIOdp (ORCPT ); Fri, 9 Sep 2022 10:33:45 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47844 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229876AbiIIOdo (ORCPT ); Fri, 9 Sep 2022 10:33:44 -0400 Received: from mail-wr1-x42b.google.com (mail-wr1-x42b.google.com [IPv6:2a00:1450:4864:20::42b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1C983F16CA for ; Fri, 9 Sep 2022 07:33:41 -0700 (PDT) Received: by mail-wr1-x42b.google.com with SMTP id bz13so3099236wrb.2 for ; Fri, 09 Sep 2022 07:33:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:from:to:cc:subject:date; bh=ewWdgINeDogjbAxyhyHuxTCApDJunU7409v6MqwQtl4=; b=cf3i4XXSRFR6TjpopzvRoMT1cavOKG4YLqipW3RgDBt0mAvgq6QSHIoV9xrIVNYW2L 8L9STi9JFJ1F+zZt8IwSoKBDZjWiZxUXlIhKy39ShkYclF66OJl0N1Cj9vvYw/GflHLj WdctLmQfTUPGHdbruxnInE2qVE6iiXgNZR0jmIVwlEHU9y8NV/uR0SiRImni8pLG6tzx jUxytQMIjRuizcAEOb7GYNLGbGcm0wSrNBFXacefyu91Yz8mVBQyvk7EvMNWlfQw7eG4 DCY6gb2pWnMP8n1gvgzMqVYaeVTNN6KEa5YkBN6SkxEWkGVmX7o2xr2AQVmx5l00u1WH y9FA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:x-gm-message-state:from:to:cc :subject:date; bh=ewWdgINeDogjbAxyhyHuxTCApDJunU7409v6MqwQtl4=; b=Y2sE69iT3WOduacC7StRYeHKRiHBjM6+lf9l5oPtCBIW5zxwtefQW/Gg2sm3sbRcwA ZcWy4GbDKia2m61IIqXDPRmqc4CqBx/8FEJsqiYti1LCkRXLguwtSGDT8Uyj87nQqfM3 mcGmKtI69xVIVD9NK34qZPvdiRIP+NyuBECxIuGFgki7wIT78EeRwyxFrEWs1lrPN4IB qn1EUfEol0UtrgFru/8g1tINUR0durIcWXgrhLwk4W0V6drXyFpBuA5UN2mR6Hw7hE9d NYfjiOrDNgXyRAkcAqGliqQbCceu6am/QhB46nwn7gYzHS89oSM0b/wSrhfyoJrhpSSo FCRw== X-Gm-Message-State: ACgBeo3Dm+VOZOMhdbvXpaji64wjwygC/oT1JSrEN30NEMTfe33EHtFe a+xnZFO8ReWuyaAF4tF8DYbW7l2VIv8= X-Google-Smtp-Source: AA6agR68CyNVpRLHSBlnpUfiBmJcUIwCEK6gjrfDIY0tf99tXjQ2T0GT1YLes/L4MAc64eO7TvO8Sg== X-Received: by 2002:a5d:4c4c:0:b0:22a:35bd:84a9 with SMTP id n12-20020a5d4c4c000000b0022a35bd84a9mr3896044wrt.103.1662734019457; Fri, 09 Sep 2022 07:33:39 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id z13-20020adfd0cd000000b0022587413219sm706303wrh.16.2022.09.09.07.33.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 09 Sep 2022 07:33:38 -0700 (PDT) Message-Id: <2ca431e6c377f42d70ea9ce0ac74058fc983ea2c.1662734015.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Fri, 09 Sep 2022 14:33:27 +0000 Subject: [PATCH v2 1/9] bundle-uri: short-circuit capability parsing Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, me@ttaylorr.com, newren@gmail.com, avarab@gmail.com, mjcheetham@outlook.com, steadmon@google.com, Glen Choo , Jonathan Tan , Teng Long , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee When parsing the capability lines from the 'git remote-https' process, we can stop reading the lines once we notice the 'get' capability. Reported-by: Teng Long Signed-off-by: Derrick Stolee --- bundle-uri.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/bundle-uri.c b/bundle-uri.c index 4a8cc74ed05..7173ed065e9 100644 --- a/bundle-uri.c +++ b/bundle-uri.c @@ -56,8 +56,10 @@ static int download_https_uri_to_file(const char *file, const char *uri) while (!strbuf_getline(&line, child_out)) { if (!line.len) break; - if (!strcmp(line.buf, "get")) + if (!strcmp(line.buf, "get")) { found_get = 1; + break; + } } strbuf_release(&line); From patchwork Fri Sep 9 14:33:28 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12971875 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0E0F8ECAAD3 for ; Fri, 9 Sep 2022 14:33:51 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231553AbiIIOdt (ORCPT ); Fri, 9 Sep 2022 10:33:49 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47874 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231509AbiIIOdq (ORCPT ); Fri, 9 Sep 2022 10:33:46 -0400 Received: from mail-wm1-x335.google.com (mail-wm1-x335.google.com [IPv6:2a00:1450:4864:20::335]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E2CFBF2D65 for ; Fri, 9 Sep 2022 07:33:42 -0700 (PDT) Received: by mail-wm1-x335.google.com with SMTP id m3-20020a05600c3b0300b003a5e0557150so3651845wms.0 for ; Fri, 09 Sep 2022 07:33:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:from:to:cc:subject:date; bh=7F28BWtCZ59vu+NnXKYZCwpYTfT3548e0gBYFM+oqdo=; b=JSIbCzlM+n9LlL1LULEC9fCGZj7Gk6RQs28ZXSIA1BDwwbq0gk0IFM9xI7B7KmcQLB N7wGLsmBssOYE8CsMfBMwzfUiGIKO/qIANh20ueNE+OXHtXMcZihrpZiTm8HA4a/Oghp ogbDiuuD8Wgl8ebtJM93k99ESkZEe0c89AUoqN97TvkREi2NuAqfW6/Mop4f6IuQRx/z llFpzKubQsZBiN1+FSne2CIziBkXiYpp254Xmo+ARWcRJbGD/jG6pkqA42g2aYvTdI2v zIWD9d4gWuTB3xfgiUl6AuHaS+Nl7hUDck2yFl3mIZ48hkQlgtTTbxCTOAA0jx8Pd5Xz 85kw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:x-gm-message-state:from:to:cc :subject:date; bh=7F28BWtCZ59vu+NnXKYZCwpYTfT3548e0gBYFM+oqdo=; b=YuN8xH2GFVIw/Vkz1lR2p/9ZzKnA7Qz8UQEqe1DffuQfAW0AuO3GlszlP4R7KJaoY6 UuJyDdXveyPyxNQWo5GNVPNSrmt27nCwUYNqSBeK1he4s1i4o8XdPZotGfTzXuAyhl/R FZ6LgKmB2uVfnOB3M2j43GPodLmPU2FHcApOXEPBbg/G3JNKW8iG9GBRWVaHWt2DlqlO /h0wrvsG1Q1PCB6VKnu/N5Qxr7tbs6NPCwLuGNS20yj05Y4pVIxQuxZoV3N4PUv5w7ty BykVI9zwpF9wte2N4EJNmPKFvE4+uPcvgPqzqW6EUsO7IxG84tMmXv/tYILXvPZ+IOFJ wrDg== X-Gm-Message-State: ACgBeo3OmevACzHT8vkB0vGcWYccSlSxo/XCIp0qMGpW/sTyjsAJrsSa ZSVvNcS5ZyBiXKDCIZvDbjajSoeX+r0= X-Google-Smtp-Source: AA6agR4b28D5A5DlXzpBDCwKb9YkGnrVebJw52OAtaNPOovp4F4sQExvEe8FVr4Oof4r8bE5/E6+cA== X-Received: by 2002:a05:600c:4982:b0:3b4:f9b:8fa1 with SMTP id h2-20020a05600c498200b003b40f9b8fa1mr2262527wmp.168.1662734020957; Fri, 09 Sep 2022 07:33:40 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id o14-20020a5d684e000000b002287d99b455sm665832wrw.15.2022.09.09.07.33.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 09 Sep 2022 07:33:40 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Fri, 09 Sep 2022 14:33:28 +0000 Subject: [PATCH v2 2/9] bundle-uri: use plain string in find_temp_filename() Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, me@ttaylorr.com, newren@gmail.com, avarab@gmail.com, mjcheetham@outlook.com, steadmon@google.com, Glen Choo , Jonathan Tan , Teng Long , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee The find_temp_filename() method was created in 53a50892be2 (bundle-uri: create basic file-copy logic, 2022-08-09) and uses odb_mkstemp() to create a temporary filename. The odb_mkstemp() method uses a strbuf in its interface, but we do not need to continue carrying a strbuf throughout the bundle URI code. Convert the find_temp_filename() method to use a 'char *' and modify its only caller. This makes sense that we don't actually need to modify this filename directly later, so using a strbuf is overkill. This change will simplify the data structure for tracking a bundle list to use plain strings instead of strbufs. Signed-off-by: Derrick Stolee --- bundle-uri.c | 28 ++++++++++++++++------------ 1 file changed, 16 insertions(+), 12 deletions(-) diff --git a/bundle-uri.c b/bundle-uri.c index 7173ed065e9..c52b2a2a64a 100644 --- a/bundle-uri.c +++ b/bundle-uri.c @@ -5,22 +5,23 @@ #include "refs.h" #include "run-command.h" -static int find_temp_filename(struct strbuf *name) +static char *find_temp_filename(void) { int fd; + struct strbuf name = STRBUF_INIT; /* * Find a temporary filename that is available. This is briefly * racy, but unlikely to collide. */ - fd = odb_mkstemp(name, "bundles/tmp_uri_XXXXXX"); + fd = odb_mkstemp(&name, "bundles/tmp_uri_XXXXXX"); if (fd < 0) { warning(_("failed to create temporary file")); - return -1; + return NULL; } close(fd); - unlink(name->buf); - return 0; + unlink(name.buf); + return strbuf_detach(&name, NULL); } static int download_https_uri_to_file(const char *file, const char *uri) @@ -143,28 +144,31 @@ static int unbundle_from_file(struct repository *r, const char *file) int fetch_bundle_uri(struct repository *r, const char *uri) { int result = 0; - struct strbuf filename = STRBUF_INIT; + char *filename; - if ((result = find_temp_filename(&filename))) + if (!(filename = find_temp_filename())) { + result = -1; goto cleanup; + } - if ((result = copy_uri_to_file(filename.buf, uri))) { + if ((result = copy_uri_to_file(filename, uri))) { warning(_("failed to download bundle from URI '%s'"), uri); goto cleanup; } - if ((result = !is_bundle(filename.buf, 0))) { + if ((result = !is_bundle(filename, 0))) { warning(_("file at URI '%s' is not a bundle"), uri); goto cleanup; } - if ((result = unbundle_from_file(r, filename.buf))) { + if ((result = unbundle_from_file(r, filename))) { warning(_("failed to unbundle bundle from URI '%s'"), uri); goto cleanup; } cleanup: - unlink(filename.buf); - strbuf_release(&filename); + if (filename) + unlink(filename); + free(filename); return result; } From patchwork Fri Sep 9 14:33:29 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12971877 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id F02B8ECAAD3 for ; Fri, 9 Sep 2022 14:33:59 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231590AbiIIOdz (ORCPT ); Fri, 9 Sep 2022 10:33:55 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47844 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231534AbiIIOdq (ORCPT ); Fri, 9 Sep 2022 10:33:46 -0400 Received: from mail-wm1-x332.google.com (mail-wm1-x332.google.com [IPv6:2a00:1450:4864:20::332]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D24A9F2D78 for ; Fri, 9 Sep 2022 07:33:43 -0700 (PDT) Received: by mail-wm1-x332.google.com with SMTP id k6-20020a05600c1c8600b003a54ecc62f6so1461860wms.5 for ; Fri, 09 Sep 2022 07:33:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:from:to:cc:subject:date; bh=L9r7t9bcS3ByqCXgq8jwoUw8iGjCzuqWojuoM/pMy+k=; b=BGxMVX8aUMj55ZSQ1lhDstjqWdKrdwSMWRq4m8mBvYPOWYAi6MLCur+U+zf2d3zeRG 0PNKZ4houXy+aSrM1E0HgbROEbsLHYrxFUETqmN/vn/8bvsIUhqmBFhpod42T/yuJa2n 4A5GsdLOQmnd6X9Z5YJbnZlNpPUhuORs3F9c1Z4PuoW004djN4l0yh5wl1CaIGI9A+ww wx+40yVu8jyxgElgfbwr6qH6IhCS3+DtgGK0kkUsPHEFbDKx6709XsmXuMZrFRUfjFwH WkMcHC6V6CE0x7qdhZzP19AhBeORvwkUAsi2+QyJQh2+dPGMR+gsl/72hpt3FwkZ7+0A VzSg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:x-gm-message-state:from:to:cc :subject:date; bh=L9r7t9bcS3ByqCXgq8jwoUw8iGjCzuqWojuoM/pMy+k=; b=3X8QJj9U/Am/9BhaBzw/BuCRG2g76HaZsCUEGPtbu5BxmgyfJSNVaLxLgYgZg+rTkp tJbr8e7vEw/NPOyvZRbpaftWkdQ3bxVA9bC6pwdLRPMjMAOZL3kV8S5hbUMf1ccUWP8U 5e2cJJ93pwB1GVGBQrDlrOzws7bZOrSE9Wn3QaQdRPgVzdcxW9MUi+Vd3RmHyPtzEw7X ZiMwBhwsBe9AlDWdLw2I1ZNinb/6M7SD1IWerq/REWpkZ+zAwNoaIYiAFz4j/5k56GxO B+D+CIck+CDtPw0ZduQKXE5sqCZVXsBDJ/VrOWbq5zElXQD/HMdySdgdtJWqPizgmOti Z/hg== X-Gm-Message-State: ACgBeo2NoKDQ+x3OrWI1bzZcESRp0WeuyOjmqWC1SEKQtetj09jwxS36 4wO4O7eaDUNTi9gOb5G/KFyjygHbF3s= X-Google-Smtp-Source: AA6agR7btAfXMf/ia0DCu1/njCC+G98mJNxo7p6U0k1DSS5hc4ORC6ReeAYvAv0z4XNcGebL4BLdRA== X-Received: by 2002:a7b:c846:0:b0:3a8:5390:c3fe with SMTP id c6-20020a7bc846000000b003a85390c3femr5818444wml.7.1662734022023; Fri, 09 Sep 2022 07:33:42 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id q6-20020a05600c46c600b003a342933727sm1051832wmo.3.2022.09.09.07.33.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 09 Sep 2022 07:33:41 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Fri, 09 Sep 2022 14:33:29 +0000 Subject: [PATCH v2 3/9] bundle-uri: create bundle_list struct and helpers Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, me@ttaylorr.com, newren@gmail.com, avarab@gmail.com, mjcheetham@outlook.com, steadmon@google.com, Glen Choo , Jonathan Tan , Teng Long , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee It will likely be rare where a user uses a single bundle URI and expects that URI to point to a bundle. Instead, that URI will likely be a list of bundles provided in some format. Alternatively, the Git server could advertise a list of bundles. In anticipation of these two ways of advertising multiple bundles, create a data structure that represents such a list. This will be populated using a common API, but for now focus on what data can be represented. Each list contains a number of remote_bundle_info structs. These contain an 'id' that is used to uniquely identify them in the list, and also a 'uri' that contains the location of its data. Finally, there is a strbuf containing the filename used when Git downloads the contents to disk. The list itself stores these remote_bundle_info structs in a hashtable using 'id' as the key. The order of the structs in the input is considered unimportant, but future modifications to the format and these data structures will place ordering possibilities on the set. The list also has a few "global" properties, including the version (used when parsing the list) and the mode. The mode is one of these two options: 1. BUNDLE_MODE_ALL: all listed URIs are intended to be combined together. The client should download all of the advertised data to have a complete copy of the data. 2. BUNDLE_MODE_ANY: any one listed item is sufficient to have a complete copy of the data. The client can choose arbitrarily from these options. In the future, the client may use pings to find the closest URI among geodistributed replicas, or use some other heuristic information added to the format. This API is currently unused, but will soon be expanded with parsing logic and then be consumed by the bundle URI download logic. Signed-off-by: Derrick Stolee --- bundle-uri.c | 60 ++++++++++++++++++++++++++++++++++++++++++++++++++++ bundle-uri.h | 56 ++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 116 insertions(+) diff --git a/bundle-uri.c b/bundle-uri.c index c52b2a2a64a..7a0bada6eda 100644 --- a/bundle-uri.c +++ b/bundle-uri.c @@ -4,6 +4,66 @@ #include "object-store.h" #include "refs.h" #include "run-command.h" +#include "hashmap.h" +#include "pkt-line.h" + +static int compare_bundles(const void *hashmap_cmp_fn_data, + const struct hashmap_entry *he1, + const struct hashmap_entry *he2, + const void *id) +{ + const struct remote_bundle_info *e1 = + container_of(he1, const struct remote_bundle_info, ent); + const struct remote_bundle_info *e2 = + container_of(he2, const struct remote_bundle_info, ent); + + return strcmp(e1->id, id ? (const char *)id : e2->id); +} + +void init_bundle_list(struct bundle_list *list) +{ + memset(list, 0, sizeof(*list)); + + /* Implied defaults. */ + list->mode = BUNDLE_MODE_ALL; + list->version = 1; + + hashmap_init(&list->bundles, compare_bundles, NULL, 0); +} + +static int clear_remote_bundle_info(struct remote_bundle_info *bundle, + void *data) +{ + FREE_AND_NULL(bundle->id); + FREE_AND_NULL(bundle->uri); + return 0; +} + +void clear_bundle_list(struct bundle_list *list) +{ + if (!list) + return; + + for_all_bundles_in_list(list, clear_remote_bundle_info, NULL); + hashmap_clear_and_free(&list->bundles, struct remote_bundle_info, ent); +} + +int for_all_bundles_in_list(struct bundle_list *list, + bundle_iterator iter, + void *data) +{ + struct remote_bundle_info *info; + struct hashmap_iter i; + + hashmap_for_each_entry(&list->bundles, &i, info, ent) { + int result = iter(info, data); + + if (result) + return result; + } + + return 0; +} static char *find_temp_filename(void) { diff --git a/bundle-uri.h b/bundle-uri.h index 8a152f1ef14..ff7e3fd3fb2 100644 --- a/bundle-uri.h +++ b/bundle-uri.h @@ -1,7 +1,63 @@ #ifndef BUNDLE_URI_H #define BUNDLE_URI_H +#include "hashmap.h" +#include "strbuf.h" + struct repository; +struct string_list; + +/** + * The remote_bundle_info struct contains information for a single bundle + * URI. This may be initialized simply by a given URI or might have + * additional metadata associated with it if the bundle was advertised by + * a bundle list. + */ +struct remote_bundle_info { + struct hashmap_entry ent; + + /** + * The 'id' is a name given to the bundle for reference + * by other bundle infos. + */ + char *id; + + /** + * The 'uri' is the location of the remote bundle so + * it can be downloaded on-demand. This will be NULL + * if there was no table of contents. + */ + char *uri; +}; + +#define REMOTE_BUNDLE_INFO_INIT { 0 } + +enum bundle_list_mode { + BUNDLE_MODE_NONE = 0, + BUNDLE_MODE_ALL, + BUNDLE_MODE_ANY +}; + +/** + * A bundle_list contains an unordered set of remote_bundle_info structs, + * as well as information about the bundle listing, such as version and + * mode. + */ +struct bundle_list { + int version; + enum bundle_list_mode mode; + struct hashmap bundles; +}; + +void init_bundle_list(struct bundle_list *list); +void clear_bundle_list(struct bundle_list *list); + +typedef int (*bundle_iterator)(struct remote_bundle_info *bundle, + void *data); + +int for_all_bundles_in_list(struct bundle_list *list, + bundle_iterator iter, + void *data); /** * Fetch data from the given 'uri' and unbundle the bundle data found From patchwork Fri Sep 9 14:33:30 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12971876 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3C504ECAAD3 for ; Fri, 9 Sep 2022 14:33:53 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231574AbiIIOdv (ORCPT ); Fri, 9 Sep 2022 10:33:51 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47896 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231548AbiIIOdq (ORCPT ); Fri, 9 Sep 2022 10:33:46 -0400 Received: from mail-wr1-x433.google.com (mail-wr1-x433.google.com [IPv6:2a00:1450:4864:20::433]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D1DE870E4B for ; Fri, 9 Sep 2022 07:33:44 -0700 (PDT) Received: by mail-wr1-x433.google.com with SMTP id b17so3084693wrq.3 for ; Fri, 09 Sep 2022 07:33:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:from:to:cc:subject:date; bh=XPliiiedpipNoHfTWL4Hk4yzxWT7G5AlKCKO0ykXu+w=; b=S2HU3fCf7nikLGK7RU6JqovgDWNksAaMjw4AyHgrQgqHcoJ8b6caM77Z1u04JQI57o TFy9fNAtJY8sGkYMotu4DlPglH3hzpSRbp0zEX6mmgdQQgDhGOU+Yn4VrelSnk4DdYE3 MWwOV8DSzn9you27+Rvhl1OmZ16bhilC2K+NERpUrtmg+F5QSS6m1Zbc0SYB1FNFpdNs +/iAtS3RnWHPtQ3JjJuMVZJHWMj+tsMgmjF/a0aSTO/za/3PRAi4+ZayY++bwOkmYAee gJuvD2xOq6ewi4nATl735A2u/bhyQM1Wy6Ctv9vjuTULbxjkJnOD5v3BhJwIwKuD9nEJ EPnQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:x-gm-message-state:from:to:cc :subject:date; bh=XPliiiedpipNoHfTWL4Hk4yzxWT7G5AlKCKO0ykXu+w=; b=6KvnwyIVikEkYN0/2yL4qnB02wRJdA2X47R953s/Phk7yI5nHprBu3v1WtMrM+Yc+f GMvJHhuG4aQV0g0Y+Fk63M/owp8QpeXlhBhq9aTGL+QdUka2B5l2nKHplia4IOl5kfY8 Yd9sCx5cQ71EPfazGa3ZEoROY+N+p6gsxFXolkOyKvGc3g75eRJbborxhta/d1BB3j/F JiakfGknXuJilRY6te3zHH1t6dVZP9TeBEwKJSxGsxbMrGVH/HqPy5AQauqhIu1pxGL+ HC4CVF/6z4GCRTLvt1egBlp+Ha3ivaMXi7lir9on2EOaJGMmQISnIbr3g8zrvsE+eEkW Eabw== X-Gm-Message-State: ACgBeo2FrMpQ+F8CBtdgUA56Xzfn9Todxu8CbNDHoWhM6hLrZ67U9jtt 86N7DlF3X96tKDj18tke3kQX1KDPXzw= X-Google-Smtp-Source: AA6agR49ZyDHq2SnnlbsFuQzZq2Do/ZdE2Gmg/5PdM46zvR7CpbbTVMudrhRoKuSiqQ1GsJAbqXPhQ== X-Received: by 2002:adf:e192:0:b0:228:d066:a844 with SMTP id az18-20020adfe192000000b00228d066a844mr8551948wrb.54.1662734023080; Fri, 09 Sep 2022 07:33:43 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id bk12-20020a0560001d8c00b0022762b0e2a2sm711000wrb.6.2022.09.09.07.33.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 09 Sep 2022 07:33:42 -0700 (PDT) Message-Id: <70daef66833744fb8b870f431af8201222c7a2ba.1662734015.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Fri, 09 Sep 2022 14:33:30 +0000 Subject: [PATCH v2 4/9] bundle-uri: create base key-value pair parsing Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, me@ttaylorr.com, newren@gmail.com, avarab@gmail.com, mjcheetham@outlook.com, steadmon@google.com, Glen Choo , Jonathan Tan , Teng Long , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee There will be two primary ways to advertise a bundle list: as a list of packet lines in Git's protocol v2 and as a config file served from a bundle URI. Both of these fundamentally use a list of key-value pairs. We will use the same set of key-value pairs across these formats. Create a new bundle_list_update() method that is currently unusued, but will be used in the next change. It inspects each key to see if it is understood and then applies it to the given bundle_list. Here are the keys that we teach Git to understand: * bundle.version: This value should be an integer. Git currently understands only version 1 and will ignore the list if the version is any other value. This version can be increased in the future if we need to add new keys that Git should not ignore. We can add new "heuristic" keys without incrementing the version. * bundle.mode: This value should be one of "all" or "any". If this mode is not understood, then Git will ignore the list. This mode indicates whether Git needs all of the bundle list items to make a complete view of the content or if any single item is sufficient. The rest of the keys use a bundle identifier "" as part of the key name. Keys using the same "" describe a single bundle list item. * bundle..uri: This stores the URI of the bundle item. This currently is expected to be an absolute URI, but will be relaxed to be a relative URI in the future. While parsing, return an error if a URI key is repeated, since we can make that restriction with bundle lists. Make the git_parse_int() method global so we can parse the integer version value carefully. Signed-off-by: Derrick Stolee --- Documentation/config.txt | 2 + Documentation/config/bundle.txt | 24 +++++++++++ bundle-uri.c | 76 +++++++++++++++++++++++++++++++++ config.c | 2 +- config.h | 1 + 5 files changed, 104 insertions(+), 1 deletion(-) create mode 100644 Documentation/config/bundle.txt diff --git a/Documentation/config.txt b/Documentation/config.txt index e376d547ce0..4280af6992e 100644 --- a/Documentation/config.txt +++ b/Documentation/config.txt @@ -387,6 +387,8 @@ include::config/branch.txt[] include::config/browser.txt[] +include::config/bundle.txt[] + include::config/checkout.txt[] include::config/clean.txt[] diff --git a/Documentation/config/bundle.txt b/Documentation/config/bundle.txt new file mode 100644 index 00000000000..daa21eb674a --- /dev/null +++ b/Documentation/config/bundle.txt @@ -0,0 +1,24 @@ +bundle.*:: + The `bundle.*` keys may appear in a bundle list file found via the + `git clone --bundle-uri` option. These keys currently have no effect + if placed in a repository config file, though this will change in the + future. See link:technical/bundle-uri.html[the bundle URI design + document] for more details. + +bundle.version:: + This integer value advertises the version of the bundle list format + used by the bundle list. Currently, the only accepted value is `1`. + +bundle.mode:: + This string value should be either `all` or `any`. This value describes + whether all of the advertised bundles are required to unbundle a + complete understanding of the bundled information (`all`) or if any one + of the listed bundle URIs is sufficient (`any`). + +bundle..*:: + The `bundle..*` keys are used to describe a single item in the + bundle list, grouped under `` for identification purposes. + +bundle..uri:: + This string value defines the URI by which Git can reach the contents + of this ``. This URI may be a bundle file or another bundle list. diff --git a/bundle-uri.c b/bundle-uri.c index 7a0bada6eda..4ccd14c8936 100644 --- a/bundle-uri.c +++ b/bundle-uri.c @@ -6,6 +6,7 @@ #include "run-command.h" #include "hashmap.h" #include "pkt-line.h" +#include "config.h" static int compare_bundles(const void *hashmap_cmp_fn_data, const struct hashmap_entry *he1, @@ -65,6 +66,81 @@ int for_all_bundles_in_list(struct bundle_list *list, return 0; } +/** + * Given a key-value pair, update the state of the given bundle list. + * Returns 0 if the key-value pair is understood. Returns 1 if the key + * is not understood or the value is malformed. + */ +MAYBE_UNUSED +static int bundle_list_update(const char *key, const char *value, + struct bundle_list *list) +{ + struct strbuf id = STRBUF_INIT; + struct remote_bundle_info lookup = REMOTE_BUNDLE_INFO_INIT; + struct remote_bundle_info *bundle; + const char *subsection, *subkey; + size_t subsection_len; + + if (parse_config_key(key, "bundle", &subsection, &subsection_len, &subkey)) + return -1; + + if (!subsection_len) { + if (!strcmp(subkey, "version")) { + int version; + if (!git_parse_int(value, &version)) + return -1; + if (version != 1) + return -1; + + list->version = version; + return 0; + } + + if (!strcmp(subkey, "mode")) { + if (!strcmp(value, "all")) + list->mode = BUNDLE_MODE_ALL; + else if (!strcmp(value, "any")) + list->mode = BUNDLE_MODE_ANY; + else + return -1; + return 0; + } + + /* Ignore other unknown global keys. */ + return 0; + } + + strbuf_add(&id, subsection, subsection_len); + + /* + * Check for an existing bundle with this , or create one + * if necessary. + */ + lookup.id = id.buf; + hashmap_entry_init(&lookup.ent, strhash(lookup.id)); + if (!(bundle = hashmap_get_entry(&list->bundles, &lookup, ent, NULL))) { + CALLOC_ARRAY(bundle, 1); + bundle->id = strbuf_detach(&id, NULL); + hashmap_entry_init(&bundle->ent, strhash(bundle->id)); + hashmap_add(&list->bundles, &bundle->ent); + } + strbuf_release(&id); + + if (!strcmp(subkey, "uri")) { + if (bundle->uri) + return -1; + bundle->uri = xstrdup(value); + return 0; + } + + /* + * At this point, we ignore any information that we don't + * understand, assuming it to be hints for a heuristic the client + * does not currently understand. + */ + return 0; +} + static char *find_temp_filename(void) { int fd; diff --git a/config.c b/config.c index 015bec360f5..e93101249f6 100644 --- a/config.c +++ b/config.c @@ -1214,7 +1214,7 @@ static int git_parse_unsigned(const char *value, uintmax_t *ret, uintmax_t max) return 0; } -static int git_parse_int(const char *value, int *ret) +int git_parse_int(const char *value, int *ret) { intmax_t tmp; if (!git_parse_signed(value, &tmp, maximum_signed_value_of_type(int))) diff --git a/config.h b/config.h index ca994d77147..ef9eade6414 100644 --- a/config.h +++ b/config.h @@ -206,6 +206,7 @@ int config_with_options(config_fn_t fn, void *, int git_parse_ssize_t(const char *, ssize_t *); int git_parse_ulong(const char *, unsigned long *); +int git_parse_int(const char *value, int *ret); /** * Same as `git_config_bool`, except that it returns -1 on error rather From patchwork Fri Sep 9 14:33:31 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= X-Patchwork-Id: 12971878 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id AC80BECAAD3 for ; Fri, 9 Sep 2022 14:34:02 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231609AbiIIOeB (ORCPT ); Fri, 9 Sep 2022 10:34:01 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47898 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231549AbiIIOdq (ORCPT ); Fri, 9 Sep 2022 10:33:46 -0400 Received: from mail-wm1-x330.google.com (mail-wm1-x330.google.com [IPv6:2a00:1450:4864:20::330]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B3886F10D8 for ; Fri, 9 Sep 2022 07:33:45 -0700 (PDT) Received: by mail-wm1-x330.google.com with SMTP id n17-20020a05600c501100b003a84bf9b68bso1464188wmr.3 for ; Fri, 09 Sep 2022 07:33:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:fcc:content-transfer-encoding:mime-version:subject:date:from :references:in-reply-to:message-id:from:to:cc:subject:date; bh=gEIVtXhI5F+5BWXtEe8SSzTdoZuIerEF4hKNmUJrZs0=; b=ma8wzQ5RQXrs030C//kd9KQ9zn36dWIfDXzWJStJiMQ1kY46B8dlHo6iTXGdeUMJrg apgWnkvq6tTyEl1Axyd+g8Or+bMlba8GQgSFvIKCI8CUHMXbbhGcmLFRrrAA0HCcuvUD /SkcQvYkbjFEcUS8muQPSmLBJeYeTlruS/h/GSPwVXxHZ9KXcqhCGo0g+XBoPJghTFuL iL7S4cHYWZnA7ZOJpPsLR2J2cCKuq4tQuCaGtwWlPagFjkR1OFw+moLOZyYxGdTY2QMk FtrA1L6TCogT7cI4EufwHYOxFSPxwgFBfYd9Jx2i8V4tS6HtXL+Eo4Wr5NR7vrV3oYoA VPAw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:fcc:content-transfer-encoding:mime-version:subject:date:from :references:in-reply-to:message-id:x-gm-message-state:from:to:cc :subject:date; bh=gEIVtXhI5F+5BWXtEe8SSzTdoZuIerEF4hKNmUJrZs0=; b=KrvGxLAX5EdYraaicNZFYhDO7E+69vWjvDqSA7naEwgKyyFYkGO7Cf7ekfSRAuXd8t d18I80KU8vTx4scWvvUOQbErwJM77+e0v4igSmUR75QPAY4rt2UaSWbi8FK+lUZkT73v L6++sihcZUm47wMr7lQGFoKkHbvriYMvahUcox3S0i/EvfBm0yPC43Bq9dR4IFm/SyGy K1wF/q+F1gVBG7L9D72lwp+UUukyU1eERKz++66J76iqSmKLnplaCbEHbj9HEiEpS0Vt kOvMgybIKIdDhC0afyUFyUlM+JNSUOoj/W1pvG1nPOB9MuJa6l/4mOCovRb6LA/VK7CC xLXA== X-Gm-Message-State: ACgBeo0k1Ys3jO937I1fFw+oLXrIUZd1VacqHimdh0hmRJyWx1gk3DuU 6d2+pwZGJzwM09TgHZozXstXTuoaE3M= X-Google-Smtp-Source: AA6agR7/0kDVIHdEHFM/BcFQFkJAu8hudnSncR2AoaK83ae1Po/KYznRLlHllP8p25kYEs62D2aQKw== X-Received: by 2002:a1c:4c11:0:b0:3a5:4d01:54be with SMTP id z17-20020a1c4c11000000b003a54d0154bemr5930221wmf.32.1662734024069; Fri, 09 Sep 2022 07:33:44 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id o12-20020a5d4a8c000000b002285f73f11dsm818450wrq.81.2022.09.09.07.33.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 09 Sep 2022 07:33:43 -0700 (PDT) Message-Id: <4df3f83402908d88ef0bf491c1cc2a745f6660e5.1662734015.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Fri, 09 Sep 2022 14:33:31 +0000 Subject: [PATCH v2 5/9] bundle-uri: create "key=value" line parsing MIME-Version: 1.0 Fcc: Sent To: git@vger.kernel.org Cc: gitster@pobox.com, me@ttaylorr.com, newren@gmail.com, avarab@gmail.com, mjcheetham@outlook.com, steadmon@google.com, Glen Choo , Jonathan Tan , Teng Long , Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7AgQmphcm1hc29u?= Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= From: =?UTF-8?q?=C3=86var=20Arnfj=C3=B6r=C3=B0=20Bjarmason?= When advertising a bundle list over Git's protocol v2, we will use packet lines. Each line will be of the form "key=value" representing a bundle list. Connect the API necessary for Git's transport to the key-value pair parsing created in the previous change. We are not currently implementing this protocol v2 functionality, but instead preparing to expose this parsing to be unit-testable. Co-authored-by: Derrick Stolee Signed-off-by: Ævar Arnfjörð Bjarmason Signed-off-by: Derrick Stolee --- bundle-uri.c | 27 ++++++++++++++++++++++++++- bundle-uri.h | 12 ++++++++++++ 2 files changed, 38 insertions(+), 1 deletion(-) diff --git a/bundle-uri.c b/bundle-uri.c index 4ccd14c8936..d4eb1ec7d4d 100644 --- a/bundle-uri.c +++ b/bundle-uri.c @@ -71,7 +71,6 @@ int for_all_bundles_in_list(struct bundle_list *list, * Returns 0 if the key-value pair is understood. Returns 1 if the key * is not understood or the value is malformed. */ -MAYBE_UNUSED static int bundle_list_update(const char *key, const char *value, struct bundle_list *list) { @@ -308,3 +307,29 @@ cleanup: free(filename); return result; } + +/** + * General API for {transport,connect}.c etc. + */ +int bundle_uri_parse_line(struct bundle_list *list, const char *line) +{ + int result; + const char *equals; + struct strbuf key = STRBUF_INIT; + + if (!strlen(line)) + return error(_("bundle-uri: got an empty line")); + + equals = strchr(line, '='); + + if (!equals) + return error(_("bundle-uri: line is not of the form 'key=value'")); + if (line == equals || !*(equals + 1)) + return error(_("bundle-uri: line has empty key or value")); + + strbuf_add(&key, line, equals - line); + result = bundle_list_update(key.buf, equals + 1, list); + strbuf_release(&key); + + return result; +} diff --git a/bundle-uri.h b/bundle-uri.h index ff7e3fd3fb2..90583461929 100644 --- a/bundle-uri.h +++ b/bundle-uri.h @@ -67,4 +67,16 @@ int for_all_bundles_in_list(struct bundle_list *list, */ int fetch_bundle_uri(struct repository *r, const char *uri); +/** + * General API for {transport,connect}.c etc. + */ + +/** + * Parse a "key=value" packet line from the bundle-uri verb. + * + * Returns 0 on success and non-zero on error. + */ +int bundle_uri_parse_line(struct bundle_list *list, + const char *line); + #endif From patchwork Fri Sep 9 14:33:32 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= X-Patchwork-Id: 12971879 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E587AECAAD3 for ; Fri, 9 Sep 2022 14:34:15 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231719AbiIIOeO (ORCPT ); Fri, 9 Sep 2022 10:34:14 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47952 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231513AbiIIOds (ORCPT ); Fri, 9 Sep 2022 10:33:48 -0400 Received: from mail-wr1-x42f.google.com (mail-wr1-x42f.google.com [IPv6:2a00:1450:4864:20::42f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E129EF2954 for ; Fri, 9 Sep 2022 07:33:46 -0700 (PDT) Received: by mail-wr1-x42f.google.com with SMTP id c11so3047208wrp.11 for ; Fri, 09 Sep 2022 07:33:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:fcc:content-transfer-encoding:mime-version:subject:date:from :references:in-reply-to:message-id:from:to:cc:subject:date; bh=H5bS4TQm1mwv1bke+KKYSXNRXXVVOxARR0H18fuSdeU=; b=Qb6lEoXqCYFRtl9FvsFjV+sc88srpWHKKhteyFfjK8VB3qWiQUiGDuH8WiI2V5aH6e BOfyJOIgl0oerT5GHcOjoKtrLl0jxkiP8IvuN03DDjHDY7o+h+6gnUgU545TalZVH5X8 7LCL3koEmV3I+qW5hSyc+AXXeg8Td513MV+/gcC9E6/r+t8l4adRcxn3oydNvsLIcEre 4mw5BYDDnnMaf6n3IEqXJd3W+AeKusrgFlreSxsdATWJnOUZ0qXBVDVzPbDwO1vzEMg3 +fcti9gbX3+/0rLrNM5/qqQfl35vVR+5yfpIPSu/SZ+z4IuChSUgFzbkeVlQMd+8QicV 3Dlg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:fcc:content-transfer-encoding:mime-version:subject:date:from :references:in-reply-to:message-id:x-gm-message-state:from:to:cc :subject:date; bh=H5bS4TQm1mwv1bke+KKYSXNRXXVVOxARR0H18fuSdeU=; b=yVvSLMqfBrP/FgBRvN9oLEAYzwAX+bkaQ65fY54iV7E3v1NZyxzlHjx2UvcxB/7H8m B69g+9uevMV4W04acwC4uUdAsoeHmeMCXbJKXtH+FwfFx4LCnpHq7J8IgT2WSHLHEKq7 vfWbbtj5Y+mnE/z9gIYJGwBMMTH/P3u1VX12zCdVy6Knq3GbbwVD1li9mrf25PbF6b6+ 6O9p4tQOG7tdizbm8U5p/tlOkTgXLIWsEYkeLm3dvFdnq/FY9wnIbU2FawPsS+92Jm20 vuys462AT6jjboZBR8z1NPu3jdSzjrb/v/ZCkrsSCw727pNqvcynfF9tnJ/pEoV48lNT c+SA== X-Gm-Message-State: ACgBeo1B3wmZ+bxl1TqTxayE92KqD9SKaKORtQQ53DrMH0qTPg6C6Lo5 snCW5ztbFvNAFgMg3SfdEwAAG8sqbvA= X-Google-Smtp-Source: AA6agR73N+ib7gkV5uKgg3zp/cC9aJQly8rg/E5RHzIDPmlfEviqYsnuyIlSHMBOdYFilfqG2322Xw== X-Received: by 2002:a05:6000:2ce:b0:226:d420:db7a with SMTP id o14-20020a05600002ce00b00226d420db7amr7757150wry.489.1662734025091; Fri, 09 Sep 2022 07:33:45 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id m3-20020a7bca43000000b003b341a2cfadsm859500wml.17.2022.09.09.07.33.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 09 Sep 2022 07:33:44 -0700 (PDT) Message-Id: <91c5b58f011ad711c301eddc2fecf79ab1ba726a.1662734015.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Fri, 09 Sep 2022 14:33:32 +0000 Subject: [PATCH v2 6/9] bundle-uri: unit test "key=value" parsing MIME-Version: 1.0 Fcc: Sent To: git@vger.kernel.org Cc: gitster@pobox.com, me@ttaylorr.com, newren@gmail.com, avarab@gmail.com, mjcheetham@outlook.com, steadmon@google.com, Glen Choo , Jonathan Tan , Teng Long , Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7AgQmphcm1hc29u?= Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= From: =?UTF-8?q?=C3=86var=20Arnfj=C3=B6r=C3=B0=20Bjarmason?= Create a new 'test-tool bundle-uri' test helper. This helper will assist in testing logic deep in the bundle URI feature. This change introduces the 'parse-key-values' subcommand, which parses an input file as a list of lines. These are fed into bundle_uri_parse_line() to test how we construct a 'struct bundle_list' from that data. The list is then output to stdout as if the key-value pairs were a Git config file. We use an input file instead of stdin because of a future change to parse in config-file format that works better as an input file. Co-authored-by: Derrick Stolee Signed-off-by: Ævar Arnfjörð Bjarmason Signed-off-by: Derrick Stolee --- Makefile | 1 + bundle-uri.c | 33 ++++++++++ bundle-uri.h | 3 + t/helper/test-bundle-uri.c | 70 +++++++++++++++++++++ t/helper/test-tool.c | 1 + t/helper/test-tool.h | 1 + t/t5750-bundle-uri-parse.sh | 121 ++++++++++++++++++++++++++++++++++++ t/test-lib-functions.sh | 11 ++++ 8 files changed, 241 insertions(+) create mode 100644 t/helper/test-bundle-uri.c create mode 100755 t/t5750-bundle-uri-parse.sh diff --git a/Makefile b/Makefile index 7d5f48069ea..7dee0329c49 100644 --- a/Makefile +++ b/Makefile @@ -722,6 +722,7 @@ PROGRAMS += $(patsubst %.o,git-%$X,$(PROGRAM_OBJS)) TEST_BUILTINS_OBJS += test-advise.o TEST_BUILTINS_OBJS += test-bitmap.o TEST_BUILTINS_OBJS += test-bloom.o +TEST_BUILTINS_OBJS += test-bundle-uri.o TEST_BUILTINS_OBJS += test-chmtime.o TEST_BUILTINS_OBJS += test-config.o TEST_BUILTINS_OBJS += test-crontab.o diff --git a/bundle-uri.c b/bundle-uri.c index d4eb1ec7d4d..74d5695e99e 100644 --- a/bundle-uri.c +++ b/bundle-uri.c @@ -66,6 +66,39 @@ int for_all_bundles_in_list(struct bundle_list *list, return 0; } +static int summarize_bundle(struct remote_bundle_info *info, void *data) +{ + FILE *fp = data; + fprintf(fp, "[bundle \"%s\"]\n", info->id); + fprintf(fp, "\turi = %s\n", info->uri); + return 0; +} + +void print_bundle_list(FILE *fp, struct bundle_list *list) +{ + const char *mode; + + switch (list->mode) { + case BUNDLE_MODE_ALL: + mode = "all"; + break; + + case BUNDLE_MODE_ANY: + mode = "any"; + break; + + case BUNDLE_MODE_NONE: + default: + mode = ""; + } + + fprintf(fp, "[bundle]\n"); + fprintf(fp, "\tversion = %d\n", list->version); + fprintf(fp, "\tmode = %s\n", mode); + + for_all_bundles_in_list(list, summarize_bundle, fp); +} + /** * Given a key-value pair, update the state of the given bundle list. * Returns 0 if the key-value pair is understood. Returns 1 if the key diff --git a/bundle-uri.h b/bundle-uri.h index 90583461929..0e56ab2ae5a 100644 --- a/bundle-uri.h +++ b/bundle-uri.h @@ -59,6 +59,9 @@ int for_all_bundles_in_list(struct bundle_list *list, bundle_iterator iter, void *data); +struct FILE; +void print_bundle_list(FILE *fp, struct bundle_list *list); + /** * Fetch data from the given 'uri' and unbundle the bundle data found * based on that information. diff --git a/t/helper/test-bundle-uri.c b/t/helper/test-bundle-uri.c new file mode 100644 index 00000000000..0329c56544f --- /dev/null +++ b/t/helper/test-bundle-uri.c @@ -0,0 +1,70 @@ +#include "test-tool.h" +#include "parse-options.h" +#include "bundle-uri.h" +#include "strbuf.h" +#include "string-list.h" + +static int cmd__bundle_uri_parse(int argc, const char **argv) +{ + const char *key_value_usage[] = { + "test-tool bundle-uri parse-key-values ", + NULL + }; + const char **usage = key_value_usage; + struct option options[] = { + OPT_END(), + }; + struct strbuf sb = STRBUF_INIT; + struct bundle_list list; + int err = 0; + FILE *fp; + + argc = parse_options(argc, argv, NULL, options, usage, 0); + if (argc != 1) + goto usage; + + init_bundle_list(&list); + fp = fopen(argv[0], "r"); + if (!fp) + die("failed to open '%s'", argv[0]); + + while (strbuf_getline(&sb, fp) != EOF) { + if (bundle_uri_parse_line(&list, sb.buf)) + err = error("bad line: '%s'", sb.buf); + } + strbuf_release(&sb); + fclose(fp); + + print_bundle_list(stdout, &list); + + clear_bundle_list(&list); + + return !!err; + +usage: + usage_with_options(usage, options); +} + +int cmd__bundle_uri(int argc, const char **argv) +{ + const char *usage[] = { + "test-tool bundle-uri []", + NULL + }; + struct option options[] = { + OPT_END(), + }; + + argc = parse_options(argc, argv, NULL, options, usage, + PARSE_OPT_STOP_AT_NON_OPTION | + PARSE_OPT_KEEP_ARGV0); + if (argc == 1) + goto usage; + + if (!strcmp(argv[1], "parse-key-values")) + return cmd__bundle_uri_parse(argc - 1, argv + 1); + error("there is no test-tool bundle-uri tool '%s'", argv[1]); + +usage: + usage_with_options(usage, options); +} diff --git a/t/helper/test-tool.c b/t/helper/test-tool.c index 318fdbab0c3..fbe2d9d8108 100644 --- a/t/helper/test-tool.c +++ b/t/helper/test-tool.c @@ -17,6 +17,7 @@ static struct test_cmd cmds[] = { { "advise", cmd__advise_if_enabled }, { "bitmap", cmd__bitmap }, { "bloom", cmd__bloom }, + { "bundle-uri", cmd__bundle_uri }, { "chmtime", cmd__chmtime }, { "config", cmd__config }, { "crontab", cmd__crontab }, diff --git a/t/helper/test-tool.h b/t/helper/test-tool.h index bb799271631..b2aa1f39a8f 100644 --- a/t/helper/test-tool.h +++ b/t/helper/test-tool.h @@ -7,6 +7,7 @@ int cmd__advise_if_enabled(int argc, const char **argv); int cmd__bitmap(int argc, const char **argv); int cmd__bloom(int argc, const char **argv); +int cmd__bundle_uri(int argc, const char **argv); int cmd__chmtime(int argc, const char **argv); int cmd__config(int argc, const char **argv); int cmd__crontab(int argc, const char **argv); diff --git a/t/t5750-bundle-uri-parse.sh b/t/t5750-bundle-uri-parse.sh new file mode 100755 index 00000000000..fd142a66ad5 --- /dev/null +++ b/t/t5750-bundle-uri-parse.sh @@ -0,0 +1,121 @@ +#!/bin/sh + +test_description="Test bundle-uri bundle_uri_parse_line()" + +TEST_NO_CREATE_REPO=1 +TEST_PASSES_SANITIZE_LEAK=true +. ./test-lib.sh + +test_expect_success 'bundle_uri_parse_line() just URIs' ' + cat >in <<-\EOF && + bundle.one.uri=http://example.com/bundle.bdl + bundle.two.uri=https://example.com/bundle.bdl + bundle.three.uri=file:///usr/share/git/bundle.bdl + EOF + + cat >expect <<-\EOF && + [bundle] + version = 1 + mode = all + [bundle "one"] + uri = http://example.com/bundle.bdl + [bundle "two"] + uri = https://example.com/bundle.bdl + [bundle "three"] + uri = file:///usr/share/git/bundle.bdl + EOF + + test-tool bundle-uri parse-key-values in >actual 2>err && + test_must_be_empty err && + test_cmp_config_output expect actual +' + +test_expect_success 'bundle_uri_parse_line() parsing edge cases: empty key or value' ' + cat >in <<-\EOF && + =bogus-value + bogus-key= + EOF + + cat >err.expect <<-EOF && + error: bundle-uri: line has empty key or value + error: bad line: '\''=bogus-value'\'' + error: bundle-uri: line has empty key or value + error: bad line: '\''bogus-key='\'' + EOF + + cat >expect <<-\EOF && + [bundle] + version = 1 + mode = all + EOF + + test_must_fail test-tool bundle-uri parse-key-values in >actual 2>err && + test_cmp err.expect err && + test_cmp_config_output expect actual +' + +test_expect_success 'bundle_uri_parse_line() parsing edge cases: empty lines' ' + cat >in <<-\EOF && + bundle.one.uri=http://example.com/bundle.bdl + + bundle.two.uri=https://example.com/bundle.bdl + + bundle.three.uri=file:///usr/share/git/bundle.bdl + EOF + + cat >err.expect <<-\EOF && + error: bundle-uri: got an empty line + error: bad line: '\'''\'' + error: bundle-uri: got an empty line + error: bad line: '\'''\'' + EOF + + # We fail, but try to continue parsing regardless + cat >expect <<-\EOF && + [bundle] + version = 1 + mode = all + [bundle "one"] + uri = http://example.com/bundle.bdl + [bundle "two"] + uri = https://example.com/bundle.bdl + [bundle "three"] + uri = file:///usr/share/git/bundle.bdl + EOF + + test_must_fail test-tool bundle-uri parse-key-values in >actual 2>err && + test_cmp err.expect err && + test_cmp_config_output expect actual +' + +test_expect_success 'bundle_uri_parse_line() parsing edge cases: duplicate lines' ' + cat >in <<-\EOF && + bundle.one.uri=http://example.com/bundle.bdl + bundle.two.uri=https://example.com/bundle.bdl + bundle.one.uri=https://example.com/bundle-2.bdl + bundle.three.uri=file:///usr/share/git/bundle.bdl + EOF + + cat >err.expect <<-\EOF && + error: bad line: '\''bundle.one.uri=https://example.com/bundle-2.bdl'\'' + EOF + + # We fail, but try to continue parsing regardless + cat >expect <<-\EOF && + [bundle] + version = 1 + mode = all + [bundle "one"] + uri = http://example.com/bundle.bdl + [bundle "two"] + uri = https://example.com/bundle.bdl + [bundle "three"] + uri = file:///usr/share/git/bundle.bdl + EOF + + test_must_fail test-tool bundle-uri parse-key-values in >actual 2>err && + test_cmp err.expect err && + test_cmp_config_output expect actual +' + +test_done diff --git a/t/test-lib-functions.sh b/t/test-lib-functions.sh index 6da7273f1d5..3175d665add 100644 --- a/t/test-lib-functions.sh +++ b/t/test-lib-functions.sh @@ -1956,3 +1956,14 @@ test_is_magic_mtime () { rm -f .git/test-mtime-actual return $ret } + +# Given two filenames, parse both using 'git config --list --file' +# and compare the sorted output of those commands. Useful when +# wanting to ignore whitespace differences and sorting concerns. +test_cmp_config_output () { + git config --list --file="$1" >config-expect && + git config --list --file="$2" >config-actual && + sort config-expect >sorted-expect && + sort config-actual >sorted-actual && + test_cmp sorted-expect sorted-actual +} From patchwork Fri Sep 9 14:33:33 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12971880 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 63E0BECAAD3 for ; Fri, 9 Sep 2022 14:34:19 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231674AbiIIOeR (ORCPT ); Fri, 9 Sep 2022 10:34:17 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47998 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231555AbiIIOdt (ORCPT ); Fri, 9 Sep 2022 10:33:49 -0400 Received: from mail-wr1-x432.google.com (mail-wr1-x432.google.com [IPv6:2a00:1450:4864:20::432]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 274C070E4B for ; Fri, 9 Sep 2022 07:33:48 -0700 (PDT) Received: by mail-wr1-x432.google.com with SMTP id n12so3068764wru.6 for ; Fri, 09 Sep 2022 07:33:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:from:to:cc:subject:date; bh=QCqqG8TLVSaUa3UDxn5Wa/bnuk48jJ5S4hqAcItvZu4=; b=Y9rNY+1asRuxt/NebC1spKjPqzpWlmgYj1Q2vFhFunv2hpD4OmjgcbSB9JsMlj0tpw R1gjq7shb6BNPvs+OKQFaMBS9wcsB/deXDTZpRnmhqEm6Buuav4fut3s1o/wbzbW6gGl HsbR/2d8xh7B/syaN9n63TTI2J15ogGH5xyQOvpHOuIsF9N7dfB13x6+6uCl4ILoOaxm rb0SVnZAombskoH7IxQ0+hh58sItwInpstMqLFdnRjOfFnju9u9yP6lJ5QCP1W1hfJ85 KyLVcsEPIUElDUrD+grflBdlQvXNIs1bKYOeH43T/IUYymN8Nd8ysrZWnmy1QbN9Frov 0iXA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:x-gm-message-state:from:to:cc :subject:date; bh=QCqqG8TLVSaUa3UDxn5Wa/bnuk48jJ5S4hqAcItvZu4=; b=tKlZ2PFgrKjOTpWlNeh05X3jVK3qHdHcOjCovQhWE+9kNuW8GbLenylo+VQaBafx6U tGUGhh0Cr+ZpuD8ZEogiDjvHYF4Yo1HwbhxSrNGT7J1zjI51OJhIxx8nCGIwxBJ+KLZm 9E49PbKeohe5npGb3AVZH+ZU7m2nUEG6Sn0THydWwNTVsk5cFsERjaOGY/s1nIdq7ocK tJy45BvPc35pzuhIrny7yBRL+MBEMrF+oTF2qBTe+oja4tj6vHDu2WC2MV5ulI+uKaU7 6TuXvxox7Pg9vJy1QOWGdNf03rHEF4vQQyxmpP8Q9p7uwQXjqyrWRv9zfvo8td5Jx9jE GckQ== X-Gm-Message-State: ACgBeo35b4usL5bhPh43DVG7k1fNTEi07LHq3amMGB1gL97jLo5QV5Xn XRgGFW6SaugpH9LK0GudtTO62Nl2H4I= X-Google-Smtp-Source: AA6agR4NGaDiX9zO0LCfpjxg/D5pnUJFQAF2eG9CDlwWKUUMi3RExljP0Cgk3Styn5WQobvcd9c76g== X-Received: by 2002:adf:db90:0:b0:226:cecd:d10b with SMTP id u16-20020adfdb90000000b00226cecdd10bmr8275617wri.531.1662734026274; Fri, 09 Sep 2022 07:33:46 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id l20-20020a05600c4f1400b003a607e395ebsm1376207wmq.9.2022.09.09.07.33.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 09 Sep 2022 07:33:45 -0700 (PDT) Message-Id: <1492b8f5ef031f30374a626b4f6e772427531e23.1662734015.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Fri, 09 Sep 2022 14:33:33 +0000 Subject: [PATCH v2 7/9] bundle-uri: parse bundle list in config format Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, me@ttaylorr.com, newren@gmail.com, avarab@gmail.com, mjcheetham@outlook.com, steadmon@google.com, Glen Choo , Jonathan Tan , Teng Long , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee When a bundle provider wants to operate independently from a Git remote, they want to provide a single, consistent URI that users can use in their 'git clone --bundle-uri' commands. At this point, the Git client expects that URI to be a single bundle that can be unbundled and used to bootstrap the rest of the clone from the Git server. This single bundle cannot be re-used to assist with future incremental fetches. To allow for the incremental fetch case, teach Git to understand a bundle list that could be advertised at an independent bundle URI. Such a bundle list is likely to be inspected by human readers, even if only by the bundle provider creating the list. For this reason, we can take our expected "key=value" pairs and instead format them using Git config format. Create bundle_uri_parse_config_format() to parse a file in config format and convert that into a 'struct bundle_list' filled with its understanding of the contents. Be careful to use error_action CONFIG_ERROR_ERROR when calling git_config_from_file_with_options() because the default action for git_config_from_file() is to die() on a parsing error. The current warning isn't particularly helpful if it arises to a user, but it will be made more verbose at a higher layer later. Update 'test-tool bundle-uri' to take this config file format as input. It uses a filename instead of stdin because there is no existing way to parse a FILE pointer in the config machinery. Using git_config_from_mem() is overly complicated and more likely to introduce bugs than this simpler version. Signed-off-by: Derrick Stolee --- bundle-uri.c | 27 ++++++++++++++++++++ bundle-uri.h | 9 +++++++ t/helper/test-bundle-uri.c | 49 +++++++++++++++++++++++++++--------- t/t5750-bundle-uri-parse.sh | 50 +++++++++++++++++++++++++++++++++++++ 4 files changed, 123 insertions(+), 12 deletions(-) diff --git a/bundle-uri.c b/bundle-uri.c index 74d5695e99e..92354aa3bbd 100644 --- a/bundle-uri.c +++ b/bundle-uri.c @@ -173,6 +173,33 @@ static int bundle_list_update(const char *key, const char *value, return 0; } +static int config_to_bundle_list(const char *key, const char *value, void *data) +{ + struct bundle_list *list = data; + return bundle_list_update(key, value, list); +} + +int bundle_uri_parse_config_format(const char *uri, + const char *filename, + struct bundle_list *list) +{ + int result; + struct config_options opts = { + .error_action = CONFIG_ERROR_ERROR, + }; + + result = git_config_from_file_with_options(config_to_bundle_list, + filename, list, + &opts); + + if (!result && list->mode == BUNDLE_MODE_NONE) { + warning(_("bundle list at '%s' has no mode"), uri); + result = 1; + } + + return result; +} + static char *find_temp_filename(void) { int fd; diff --git a/bundle-uri.h b/bundle-uri.h index 0e56ab2ae5a..bc13d4c9929 100644 --- a/bundle-uri.h +++ b/bundle-uri.h @@ -62,6 +62,15 @@ int for_all_bundles_in_list(struct bundle_list *list, struct FILE; void print_bundle_list(FILE *fp, struct bundle_list *list); +/** + * A bundle URI may point to a bundle list where the key=value + * pairs are provided in config file format. This method is + * exposed publicly for testing purposes. + */ +int bundle_uri_parse_config_format(const char *uri, + const char *filename, + struct bundle_list *list); + /** * Fetch data from the given 'uri' and unbundle the bundle data found * based on that information. diff --git a/t/helper/test-bundle-uri.c b/t/helper/test-bundle-uri.c index 0329c56544f..25afd393428 100644 --- a/t/helper/test-bundle-uri.c +++ b/t/helper/test-bundle-uri.c @@ -4,12 +4,21 @@ #include "strbuf.h" #include "string-list.h" -static int cmd__bundle_uri_parse(int argc, const char **argv) +enum input_mode { + KEY_VALUE_PAIRS, + CONFIG_FILE, +}; + +static int cmd__bundle_uri_parse(int argc, const char **argv, enum input_mode mode) { const char *key_value_usage[] = { "test-tool bundle-uri parse-key-values ", NULL }; + const char *config_usage[] = { + "test-tool bundle-uri parse-config ", + NULL + }; const char **usage = key_value_usage; struct option options[] = { OPT_END(), @@ -19,21 +28,35 @@ static int cmd__bundle_uri_parse(int argc, const char **argv) int err = 0; FILE *fp; - argc = parse_options(argc, argv, NULL, options, usage, 0); - if (argc != 1) - goto usage; + if (mode == CONFIG_FILE) + usage = config_usage; + + argc = parse_options(argc, argv, NULL, options, usage, + PARSE_OPT_STOP_AT_NON_OPTION); init_bundle_list(&list); - fp = fopen(argv[0], "r"); - if (!fp) - die("failed to open '%s'", argv[0]); - while (strbuf_getline(&sb, fp) != EOF) { - if (bundle_uri_parse_line(&list, sb.buf)) - err = error("bad line: '%s'", sb.buf); + switch (mode) { + case KEY_VALUE_PAIRS: + if (argc != 1) + goto usage; + fp = fopen(argv[0], "r"); + if (!fp) + die("failed to open '%s'", argv[0]); + while (strbuf_getline(&sb, fp) != EOF) { + if (bundle_uri_parse_line(&list, sb.buf)) + err = error("bad line: '%s'", sb.buf); + } + fclose(fp); + break; + + case CONFIG_FILE: + if (argc != 1) + goto usage; + err = bundle_uri_parse_config_format("", argv[0], &list); + break; } strbuf_release(&sb); - fclose(fp); print_bundle_list(stdout, &list); @@ -62,7 +85,9 @@ int cmd__bundle_uri(int argc, const char **argv) goto usage; if (!strcmp(argv[1], "parse-key-values")) - return cmd__bundle_uri_parse(argc - 1, argv + 1); + return cmd__bundle_uri_parse(argc - 1, argv + 1, KEY_VALUE_PAIRS); + if (!strcmp(argv[1], "parse-config")) + return cmd__bundle_uri_parse(argc - 1, argv + 1, CONFIG_FILE); error("there is no test-tool bundle-uri tool '%s'", argv[1]); usage: diff --git a/t/t5750-bundle-uri-parse.sh b/t/t5750-bundle-uri-parse.sh index fd142a66ad5..c2fe3f9c5a5 100755 --- a/t/t5750-bundle-uri-parse.sh +++ b/t/t5750-bundle-uri-parse.sh @@ -118,4 +118,54 @@ test_expect_success 'bundle_uri_parse_line() parsing edge cases: duplicate lines test_cmp_config_output expect actual ' +test_expect_success 'parse config format: just URIs' ' + cat >expect <<-\EOF && + [bundle] + version = 1 + mode = all + [bundle "one"] + uri = http://example.com/bundle.bdl + [bundle "two"] + uri = https://example.com/bundle.bdl + [bundle "three"] + uri = file:///usr/share/git/bundle.bdl + EOF + + test-tool bundle-uri parse-config expect >actual 2>err && + test_must_be_empty err && + test_cmp_config_output expect actual +' + +test_expect_success 'parse config format edge cases: empty key or value' ' + cat >in1 <<-\EOF && + = bogus-value + EOF + + cat >err1 <<-EOF && + error: bad config line 1 in file in1 + EOF + + cat >expect <<-\EOF && + [bundle] + version = 1 + mode = all + EOF + + test_must_fail test-tool bundle-uri parse-config in1 >actual 2>err && + test_cmp err1 err && + test_cmp_config_output expect actual && + + cat >in2 <<-\EOF && + bogus-key = + EOF + + cat >err2 <<-EOF && + error: bad config line 1 in file in2 + EOF + + test_must_fail test-tool bundle-uri parse-config in2 >actual 2>err && + test_cmp err2 err && + test_cmp_config_output expect actual +' + test_done From patchwork Fri Sep 9 14:33:34 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12971881 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B33F6ECAAA1 for ; Fri, 9 Sep 2022 14:34:21 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231555AbiIIOeU (ORCPT ); Fri, 9 Sep 2022 10:34:20 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48108 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231585AbiIIOdx (ORCPT ); Fri, 9 Sep 2022 10:33:53 -0400 Received: from mail-wm1-x32a.google.com (mail-wm1-x32a.google.com [IPv6:2a00:1450:4864:20::32a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 37F08F16CA for ; Fri, 9 Sep 2022 07:33:49 -0700 (PDT) Received: by mail-wm1-x32a.google.com with SMTP id h1so1533273wmd.3 for ; Fri, 09 Sep 2022 07:33:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:from:to:cc:subject:date; bh=JnsLUcB0ykBlbNUWKol07sfqMcsXd7medZlS/hujmw0=; b=GxzxCyIr+nqmojyMg40sgK9vN8QzEPkJh3kR37/81Qc7IzeA2w1O1kRbpISTgOb2dB 5lYXlx62A8/kBT7uz6ivGaDF3ycFWG3XP7o54I7274Gs9PMzBvJnUSOKeJQ0wUmPjAjN n6DGb61ZmFYso0RCmF3WAJ81xhJKb4jswQqATr2SHhK/qMtWQhM/dqhZe6os7TpVkeRD y2gt/W+0EdwgHWbM4xTfbuzSV6RSeeKX7oFAOsCeyzS7KH4lJi+cbir4+EZn4NDaYBTU wSI/sjywpOL/xtlJWlyD9q0Vmlum1Og6zWGwzqIGyGjQ8Q63abW4ywP1Ccn4KrJcBrgd HCfA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:x-gm-message-state:from:to:cc :subject:date; bh=JnsLUcB0ykBlbNUWKol07sfqMcsXd7medZlS/hujmw0=; b=Ft2UJmw52nCJe6EYzZsi3x9Yz1OJTk9noAl8H7szQ3QO5vKN+PmJi7GNEUuqf6oJoq jjKEOFFti4Ph+eauCT6zfPgmA8vDC63bcPatK6L6kZJsP07ckhFPPbsHMPINsqrDb/lw XrB4C+//gfOmOTUxrXSa7Q8hEyLWP5tOVqJhrkYKR+zd8gGdEKeW2L9SRkQ2yn9Pko5Y erTO8kYo0vaTdvPmNG+LjRWFTTEiKTWBuGyLXej97rCTnJrfRgHk5jzDpjafAfYdP2QI O6rctcvhbeBR2s83hzKSfOifsHH8/cdMwN7t8tkhh4Lh4T+zCziEkDT4zqba95FbBScM dSlA== X-Gm-Message-State: ACgBeo04ebevvTjYgGrKeLDP+dCsdOSkCmIQPty1RzPGuG3ZeBsmiD01 FFfUSHMuBWw1Tsh03YXWat/TA8KfxNU= X-Google-Smtp-Source: AA6agR5aU4ARONbBSuWfG4k7hcunU2dsOCXPhVA8mm/7c3cbmUWzJJm8lWzTk1L6ROlrZLYzXMTweQ== X-Received: by 2002:a05:600c:3d93:b0:3a5:eb9b:b495 with SMTP id bi19-20020a05600c3d9300b003a5eb9bb495mr5612278wmb.91.1662734027486; Fri, 09 Sep 2022 07:33:47 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id 17-20020a05600c241100b003a5de95b105sm756368wmp.41.2022.09.09.07.33.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 09 Sep 2022 07:33:46 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Fri, 09 Sep 2022 14:33:34 +0000 Subject: [PATCH v2 8/9] bundle-uri: limit recursion depth for bundle lists Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, me@ttaylorr.com, newren@gmail.com, avarab@gmail.com, mjcheetham@outlook.com, steadmon@google.com, Glen Choo , Jonathan Tan , Teng Long , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee The next change will start allowing us to parse bundle lists that are downloaded from a provided bundle URI. Those lists might point to other lists, which could proceed to an arbitrary depth (and even create cycles). Restructure fetch_bundle_uri() to have an internal version that has a recursion depth. Compare that to a new max_bundle_uri_depth constant that is twice as high as we expect this depth to be for any legitimate use of bundle list linking. We can consider making max_bundle_uri_depth a configurable value if there is demonstrated value in the future. Signed-off-by: Derrick Stolee --- bundle-uri.c | 21 ++++++++++++++++++++- 1 file changed, 20 insertions(+), 1 deletion(-) diff --git a/bundle-uri.c b/bundle-uri.c index 92354aa3bbd..b8ca6cd9493 100644 --- a/bundle-uri.c +++ b/bundle-uri.c @@ -336,11 +336,25 @@ static int unbundle_from_file(struct repository *r, const char *file) return result; } -int fetch_bundle_uri(struct repository *r, const char *uri) +/** + * This limits the recursion on fetch_bundle_uri_internal() when following + * bundle lists. + */ +static int max_bundle_uri_depth = 4; + +static int fetch_bundle_uri_internal(struct repository *r, + const char *uri, + int depth) { int result = 0; char *filename; + if (depth >= max_bundle_uri_depth) { + warning(_("exceeded bundle URI recursion limit (%d)"), + max_bundle_uri_depth); + return -1; + } + if (!(filename = find_temp_filename())) { result = -1; goto cleanup; @@ -368,6 +382,11 @@ cleanup: return result; } +int fetch_bundle_uri(struct repository *r, const char *uri) +{ + return fetch_bundle_uri_internal(r, uri, 0); +} + /** * General API for {transport,connect}.c etc. */ From patchwork Fri Sep 9 14:33:35 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12971882 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 09A43ECAAD3 for ; Fri, 9 Sep 2022 14:34:24 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231679AbiIIOeV (ORCPT ); Fri, 9 Sep 2022 10:34:21 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48644 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231626AbiIIOeM (ORCPT ); Fri, 9 Sep 2022 10:34:12 -0400 Received: from mail-wr1-x42f.google.com (mail-wr1-x42f.google.com [IPv6:2a00:1450:4864:20::42f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8D162F410E for ; Fri, 9 Sep 2022 07:33:49 -0700 (PDT) Received: by mail-wr1-x42f.google.com with SMTP id c11so3047503wrp.11 for ; Fri, 09 Sep 2022 07:33:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:from:to:cc:subject:date; bh=bCy8XBdlw0xOpTCBlQZPBdOInI5PeVg+J8kMZfqUPD4=; b=Nxi48QxeyVen/6NqmMS7KkMjyclvnHSdWahCrX58YTZn2DGx5D9jnNDzdba3EMKaJY e7zJ4PzXjvsW4APHi40rC+3uR3CQiomaswRt3UQlqqOCh+LfZjGZWtPjjejGThzL1oGW dcH0H7rmNwuRPfGyQwFKW5T8ct7P8jXxXlrnc23mZpIORf4/WeJ1DOdDpCKYD6mGf9xJ XD1PVdmhsLgCfJqzGEErfsUZrnx9qvK9vPa04RtfOqQndr0edo2c5mKf9We0GMkQfYbn B8482bURALOxpc/GDepDRcoSv0IBJPrHVP3UzRvvHCh7TKOP8fa3mTEstCLclS3t2l4o hE5A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:x-gm-message-state:from:to:cc :subject:date; bh=bCy8XBdlw0xOpTCBlQZPBdOInI5PeVg+J8kMZfqUPD4=; b=AVOtH8CM7xfMuLxHvvK/UoU43uBNY4UVU7Moqq9iB29sASqpJ4fXhdNgOonr0ugzx9 35YZfvqCck4Nrz+K4N4GQlJzNuleqEQgSSlq2OJNytU5H1JS591rpU+vz1ovLuPMm6A3 OqXIW2xH+Nvnj9vHHsaol4/USO1qe6Ei+j7ZLPwvpYifKn7h3FKhC6MnrGl2yW+7OGUw t6Nz2a4N1dXwZXvdyhksHEyykivUWP/mmvlEJB2GhAB0NiZONMbvqw5RQzBdCoJiAGA5 kFvXS3DpO+MSeQVLNVadpQRlgS5/jq21MxRB8dPOWzhvvmTHPgFkWjGxYjC5roEPoalJ C3Fw== X-Gm-Message-State: ACgBeo3I/lciXodAdWIAzFzdHH3DYP5ErGhcuxpEkxDr9NaQ5LbQAYYG 5q5aSq3Mvra02AzD7VS8I2rTjHyWtC8= X-Google-Smtp-Source: AA6agR4TW99Fm+YJ7N3BzJG7EN7hNdKp7HgV52Ntf4xD37nB34+4qQBZjRGR588vxXjApnNRPW0Hlw== X-Received: by 2002:a5d:6050:0:b0:228:6128:b0fb with SMTP id j16-20020a5d6050000000b002286128b0fbmr7836144wrt.424.1662734028771; Fri, 09 Sep 2022 07:33:48 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id q3-20020a05600000c300b00228d6edade0sm658280wrx.46.2022.09.09.07.33.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 09 Sep 2022 07:33:48 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Fri, 09 Sep 2022 14:33:35 +0000 Subject: [PATCH v2 9/9] bundle-uri: fetch a list of bundles Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, me@ttaylorr.com, newren@gmail.com, avarab@gmail.com, mjcheetham@outlook.com, steadmon@google.com, Glen Choo , Jonathan Tan , Teng Long , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee When the content at a given bundle URI is not understood as a bundle (based on inspecting the initial content), then Git currently gives up and ignores that content. Independent bundle providers may want to split up the bundle content into multiple bundles, but still make them available from a single URI. Teach Git to attempt parsing the bundle URI content as a Git config file providing the key=value pairs for a bundle list. Git then looks at the mode of the list to see if ANY single bundle is sufficient or if ALL bundles are required. The content at the selected URIs are downloaded and the content is inspected again, creating a recursive process. To guard the recursion against malformed or malicious content, limit the recursion depth to a reasonable four for now. This can be converted to a configured value in the future if necessary. The value of four is twice as high as expected to be useful (a bundle list is unlikely to point to more bundle lists). To test this scenario, create an interesting bundle topology where three incremental bundles are built on top of a single full bundle. By using a merge commit, the two middle bundles are "independent" in that they do not require each other in order to unbundle themselves. They each only need the base bundle. The bundle containing the merge commit requires both of the middle bundles, though. This leads to some interesting decisions when unbundling, especially when we later implement heuristics that promote downloading bundles until the prerequisite commits are satisfied. Signed-off-by: Derrick Stolee --- bundle-uri.c | 216 +++++++++++++++++++++++++++++++++--- bundle-uri.h | 13 +++ t/t5558-clone-bundle-uri.sh | 93 ++++++++++++++++ 3 files changed, 306 insertions(+), 16 deletions(-) diff --git a/bundle-uri.c b/bundle-uri.c index b8ca6cd9493..6a2fea26a94 100644 --- a/bundle-uri.c +++ b/bundle-uri.c @@ -37,6 +37,8 @@ static int clear_remote_bundle_info(struct remote_bundle_info *bundle, { FREE_AND_NULL(bundle->id); FREE_AND_NULL(bundle->uri); + FREE_AND_NULL(bundle->file); + bundle->unbundled = 0; return 0; } @@ -336,18 +338,111 @@ static int unbundle_from_file(struct repository *r, const char *file) return result; } +struct bundle_list_context { + struct repository *r; + struct bundle_list *list; + enum bundle_list_mode mode; + int count; + int depth; +}; + +/* + * This early definition is necessary because we use indirect recursion: + * + * While iterating through a bundle list that was downloaded as part + * of fetch_bundle_uri_internal(), iterator methods eventually call it + * again, but with depth + 1. + */ +static int fetch_bundle_uri_internal(struct repository *r, + struct remote_bundle_info *bundle, + int depth, + struct bundle_list *list); + +static int download_bundle_to_file(struct remote_bundle_info *bundle, void *data) +{ + int res; + struct bundle_list_context *ctx = data; + + if (ctx->mode == BUNDLE_MODE_ANY && ctx->count) + return 0; + + res = fetch_bundle_uri_internal(ctx->r, bundle, ctx->depth + 1, ctx->list); + + /* + * Only increment count if the download succeeded. If our mode is + * BUNDLE_MODE_ANY, then we will want to try other URIs in the + * list in case they work instead. + */ + if (!res) + ctx->count++; + return res; +} + +static int download_bundle_list(struct repository *r, + struct bundle_list *local_list, + struct bundle_list *global_list, + int depth) +{ + struct bundle_list_context ctx = { + .r = r, + .list = global_list, + .depth = depth + 1, + .mode = local_list->mode, + }; + + return for_all_bundles_in_list(local_list, download_bundle_to_file, &ctx); +} + +static int fetch_bundle_list_in_config_format(struct repository *r, + struct bundle_list *global_list, + struct remote_bundle_info *bundle, + int depth) +{ + int result; + struct bundle_list list_from_bundle; + + init_bundle_list(&list_from_bundle); + + if ((result = bundle_uri_parse_config_format(bundle->uri, + bundle->file, + &list_from_bundle))) + goto cleanup; + + if (list_from_bundle.mode == BUNDLE_MODE_NONE) { + warning(_("unrecognized bundle mode from URI '%s'"), + bundle->uri); + result = -1; + goto cleanup; + } + + if ((result = download_bundle_list(r, &list_from_bundle, + global_list, depth))) + goto cleanup; + +cleanup: + clear_bundle_list(&list_from_bundle); + return result; +} + /** * This limits the recursion on fetch_bundle_uri_internal() when following * bundle lists. */ static int max_bundle_uri_depth = 4; +/** + * Recursively download all bundles advertised at the given URI + * to files. If the file is a bundle, then add it to the given + * 'list'. Otherwise, expect a bundle list and recurse on the + * URIs in that list according to the list mode (ANY or ALL). + */ static int fetch_bundle_uri_internal(struct repository *r, - const char *uri, - int depth) + struct remote_bundle_info *bundle, + int depth, + struct bundle_list *list) { int result = 0; - char *filename; + struct remote_bundle_info *bcopy; if (depth >= max_bundle_uri_depth) { warning(_("exceeded bundle URI recursion limit (%d)"), @@ -355,36 +450,125 @@ static int fetch_bundle_uri_internal(struct repository *r, return -1; } - if (!(filename = find_temp_filename())) { + if (!bundle->file && + !(bundle->file = find_temp_filename())) { result = -1; goto cleanup; } - if ((result = copy_uri_to_file(filename, uri))) { - warning(_("failed to download bundle from URI '%s'"), uri); + if ((result = copy_uri_to_file(bundle->file, bundle->uri))) { + warning(_("failed to download bundle from URI '%s'"), bundle->uri); goto cleanup; } - if ((result = !is_bundle(filename, 0))) { - warning(_("file at URI '%s' is not a bundle"), uri); + if ((result = !is_bundle(bundle->file, 1))) { + result = fetch_bundle_list_in_config_format( + r, list, bundle, depth); + if (result) + warning(_("file at URI '%s' is not a bundle or bundle list"), + bundle->uri); goto cleanup; } - if ((result = unbundle_from_file(r, filename))) { - warning(_("failed to unbundle bundle from URI '%s'"), uri); - goto cleanup; - } + /* Copy the bundle and insert it into the global list. */ + CALLOC_ARRAY(bcopy, 1); + bcopy->id = xstrdup(bundle->id); + bcopy->file = xstrdup(bundle->file); + hashmap_entry_init(&bcopy->ent, strhash(bcopy->id)); + hashmap_add(&list->bundles, &bcopy->ent); cleanup: - if (filename) - unlink(filename); - free(filename); + if (result && bundle->file) + unlink(bundle->file); return result; } +struct attempt_unbundle_context { + struct repository *r; + int success_count; + int failure_count; +}; + +static int attempt_unbundle(struct remote_bundle_info *info, void *data) +{ + struct attempt_unbundle_context *ctx = data; + + if (info->unbundled || !unbundle_from_file(ctx->r, info->file)) { + ctx->success_count++; + info->unbundled = 1; + } else { + ctx->failure_count++; + } + + return 0; +} + +static int unbundle_all_bundles(struct repository *r, + struct bundle_list *list) +{ + int last_success_count = -1; + struct attempt_unbundle_context ctx = { + .r = r, + }; + + /* + * Iterate through all bundles looking for ones that can + * successfully unbundle. If any succeed, then perhaps another + * will succeed in the next attempt. + */ + while (last_success_count < ctx.success_count) { + last_success_count = ctx.success_count; + + ctx.success_count = 0; + ctx.failure_count = 0; + for_all_bundles_in_list(list, attempt_unbundle, &ctx); + } + + if (ctx.success_count) + git_config_set_multivar_gently("log.excludedecoration", + "refs/bundle/", + "refs/bundle/", + CONFIG_FLAGS_FIXED_VALUE | + CONFIG_FLAGS_MULTI_REPLACE); + + if (ctx.failure_count) + warning(_("failed to unbundle %d bundles"), + ctx.failure_count); + + return 0; +} + +static int unlink_bundle(struct remote_bundle_info *info, void *data) +{ + if (info->file) + unlink_or_warn(info->file); + return 0; +} + int fetch_bundle_uri(struct repository *r, const char *uri) { - return fetch_bundle_uri_internal(r, uri, 0); + int result; + struct bundle_list list; + struct remote_bundle_info bundle = { + .uri = xstrdup(uri), + .id = xstrdup(""), + }; + + init_bundle_list(&list); + + /* If a bundle is added to this global list, then it is required. */ + list.mode = BUNDLE_MODE_ALL; + + if ((result = fetch_bundle_uri_internal(r, &bundle, 0, &list))) + goto cleanup; + + result = unbundle_all_bundles(r, &list); + +cleanup: + for_all_bundles_in_list(&list, unlink_bundle, NULL); + clear_bundle_list(&list); + clear_remote_bundle_info(&bundle, NULL); + return result; } /** diff --git a/bundle-uri.h b/bundle-uri.h index bc13d4c9929..4dbc269823c 100644 --- a/bundle-uri.h +++ b/bundle-uri.h @@ -28,6 +28,19 @@ struct remote_bundle_info { * if there was no table of contents. */ char *uri; + + /** + * If the bundle has been downloaded, then 'file' is a + * filename storing its contents. Otherwise, 'file' is + * NULL. + */ + char *file; + + /** + * If the bundle has been unbundled successfully, then + * this boolean is true. + */ + unsigned unbundled:1; }; #define REMOTE_BUNDLE_INFO_INIT { 0 } diff --git a/t/t5558-clone-bundle-uri.sh b/t/t5558-clone-bundle-uri.sh index ad666a2d28a..592790b49f0 100755 --- a/t/t5558-clone-bundle-uri.sh +++ b/t/t5558-clone-bundle-uri.sh @@ -41,6 +41,72 @@ test_expect_success 'clone with file:// bundle' ' test_cmp expect actual ' +# To get interesting tests for bundle lists, we need to construct a +# somewhat-interesting commit history. +# +# ---------------- bundle-4 +# +# 4 +# / \ +# ----|---|------- bundle-3 +# | | +# | 3 +# | | +# ----|---|------- bundle-2 +# | | +# 2 | +# | | +# ----|---|------- bundle-1 +# \ / +# 1 +# | +# (previous commits) +test_expect_success 'construct incremental bundle list' ' + ( + cd clone-from && + git checkout -b base && + test_commit 1 && + git checkout -b left && + test_commit 2 && + git checkout -b right base && + test_commit 3 && + git checkout -b merge left && + git merge right -m "4" && + + git bundle create bundle-1.bundle base && + git bundle create bundle-2.bundle base..left && + git bundle create bundle-3.bundle base..right && + git bundle create bundle-4.bundle merge --not left right + ) +' + +test_expect_success 'clone bundle list (file, no heuristic)' ' + cat >bundle-list <<-EOF && + [bundle] + version = 1 + mode = all + + [bundle "bundle-1"] + uri = file://$(pwd)/clone-from/bundle-1.bundle + + [bundle "bundle-2"] + uri = file://$(pwd)/clone-from/bundle-2.bundle + + [bundle "bundle-3"] + uri = file://$(pwd)/clone-from/bundle-3.bundle + + [bundle "bundle-4"] + uri = file://$(pwd)/clone-from/bundle-4.bundle + EOF + + git clone --bundle-uri="file://$(pwd)/bundle-list" . clone-list-file && + for oid in $(git -C clone-from for-each-ref --format="%(objectname)") + do + git -C clone-list-file rev-parse $oid || return 1 + done +' + + ######################################################################### # HTTP tests begin here @@ -75,6 +141,33 @@ test_expect_success 'clone HTTP bundle' ' test_config -C clone-http log.excludedecoration refs/bundle/ ' +test_expect_success 'clone bundle list (HTTP, no heuristic)' ' + cp clone-from/bundle-*.bundle "$HTTPD_DOCUMENT_ROOT_PATH/" && + cat >"$HTTPD_DOCUMENT_ROOT_PATH/bundle-list" <<-EOF && + [bundle] + version = 1 + mode = all + + [bundle "bundle-1"] + uri = $HTTPD_URL/bundle-1.bundle + + [bundle "bundle-2"] + uri = $HTTPD_URL/bundle-2.bundle + + [bundle "bundle-3"] + uri = $HTTPD_URL/bundle-3.bundle + + [bundle "bundle-4"] + uri = $HTTPD_URL/bundle-4.bundle + EOF + + git clone --bundle-uri="$HTTPD_URL/bundle-list" . clone-list-http && + for oid in $(git -C clone-from for-each-ref --format="%(objectname)") + do + git -C clone-list-http rev-parse $oid || return 1 + done +' + # Do not add tests here unless they use the HTTP server, as they will # not run unless the HTTP dependencies exist.