From patchwork Wed Oct 12 12:52:28 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 13005045 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4C942C433FE for ; Wed, 12 Oct 2022 12:52:50 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229513AbiJLMws (ORCPT ); Wed, 12 Oct 2022 08:52:48 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37704 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229527AbiJLMwq (ORCPT ); Wed, 12 Oct 2022 08:52:46 -0400 Received: from mail-wm1-x32e.google.com (mail-wm1-x32e.google.com [IPv6:2a00:1450:4864:20::32e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 82810B40DF for ; Wed, 12 Oct 2022 05:52:45 -0700 (PDT) Received: by mail-wm1-x32e.google.com with SMTP id l32so2732584wms.2 for ; Wed, 12 Oct 2022 05:52:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:from:to:cc:subject:date :message-id:reply-to; bh=C0RWbkNTtXe3sf2lfCXZhtBtwN32JnfhLRZybJP7Qo8=; b=BmiRWCB35F0DIO1tA34bIm4hldA0Tk2ex76YA2SXH3zvcCqlIYzmdz+nVabGiHSyqm JCaO0p7Jd8Xyj/g/JALYgqdYyO8Ox4j95iXoVZBQtkC7KYPyDI6ZZEBHwWwgO1DXa3Jy HmWpItmT9az/z2FMxERRFXYv74GX79ZtbUmP8jGQa6JpWPkJ4udY8f9Ah1m3n5GRrjhz 2Yr6+WN6Q06ykVxx/+WxujIs/14hKX3k33D3BN45jJKG9vgAm0/p0ojObGMkAR6TUJph LJjv4fqEp8DvlO2prrYfBD2UNiKYLioKMZMmofCDkjpa48SUm09ehnRZXBC7S3ZKPQuC bsdQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=C0RWbkNTtXe3sf2lfCXZhtBtwN32JnfhLRZybJP7Qo8=; b=hDui7UBWO0v0gPREhDCIbROr4Ia1mpvoWL0w28xwDJh5GCFet87kiLlpkZOZ2glDdN AIAbX6Ri++0ZLv/nBV18YsMFRbpLOBPmE3I2GC+RsrFuapym/+jAqIOBCZQroHvh4UTp slOyTnB29QxQwXqMY+Bo3Lju5pyvVLpTDm4KjCbtVHevMEg7ze2s2FFj/lxaSPq7OBk+ iSmQd0CanMCKwtN1KVWgPCPhcBpLEdHYdp//+CjjElRMpsBQjcYUSQEL8w29ian5pEAL /PXDfz/NbrYHdsE+dxytsa6qmdo8YmFT51G5bbG3QGlw5oDq+NrX7/aoc8A5T85wOyPt b/mA== X-Gm-Message-State: ACrzQf1HWglYC+PXNfsIPQHQIf8m60Bf0MbdOkiJcm4atXcdsOZdB9yc UO7qqB6q+zPGhbTR7g6L3jzjZec0whI= X-Google-Smtp-Source: AMsMyM6q4ux5gruz/NoANUVABGzwDK5MIDPqiMAqcqP6M1LebFOOhx1uAlxdl51iZ+uZGSW38Fi57w== X-Received: by 2002:a05:600c:288:b0:3c6:c44a:1d30 with SMTP id 8-20020a05600c028800b003c6c44a1d30mr2752467wmk.46.1665579163622; Wed, 12 Oct 2022 05:52:43 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id u6-20020adfed46000000b0022e04bfa661sm13711849wro.59.2022.10.12.05.52.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 12 Oct 2022 05:52:42 -0700 (PDT) Message-Id: <48beccb0f5efe6f9247968cd0d4c455c23a24c53.1665579160.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Wed, 12 Oct 2022 12:52:28 +0000 Subject: [PATCH v5 01/12] bundle-uri: use plain string in find_temp_filename() Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, me@ttaylorr.com, newren@gmail.com, avarab@gmail.com, mjcheetham@outlook.com, steadmon@google.com, Glen Choo , Jonathan Tan , Teng Long , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee The find_temp_filename() method was created in 53a50892be2 (bundle-uri: create basic file-copy logic, 2022-08-09) and uses odb_mkstemp() to create a temporary filename. The odb_mkstemp() method uses a strbuf in its interface, but we do not need to continue carrying a strbuf throughout the bundle URI code. Convert the find_temp_filename() method to use a 'char *' and modify its only caller. This makes sense that we don't actually need to modify this filename directly later, so using a strbuf is overkill. This change will simplify the data structure for tracking a bundle list to use plain strings instead of strbufs. Signed-off-by: Derrick Stolee --- bundle-uri.c | 28 ++++++++++++++++------------ 1 file changed, 16 insertions(+), 12 deletions(-) diff --git a/bundle-uri.c b/bundle-uri.c index 4a8cc74ed05..8b2f4e08c9c 100644 --- a/bundle-uri.c +++ b/bundle-uri.c @@ -5,22 +5,23 @@ #include "refs.h" #include "run-command.h" -static int find_temp_filename(struct strbuf *name) +static char *find_temp_filename(void) { int fd; + struct strbuf name = STRBUF_INIT; /* * Find a temporary filename that is available. This is briefly * racy, but unlikely to collide. */ - fd = odb_mkstemp(name, "bundles/tmp_uri_XXXXXX"); + fd = odb_mkstemp(&name, "bundles/tmp_uri_XXXXXX"); if (fd < 0) { warning(_("failed to create temporary file")); - return -1; + return NULL; } close(fd); - unlink(name->buf); - return 0; + unlink(name.buf); + return strbuf_detach(&name, NULL); } static int download_https_uri_to_file(const char *file, const char *uri) @@ -141,28 +142,31 @@ static int unbundle_from_file(struct repository *r, const char *file) int fetch_bundle_uri(struct repository *r, const char *uri) { int result = 0; - struct strbuf filename = STRBUF_INIT; + char *filename; - if ((result = find_temp_filename(&filename))) + if (!(filename = find_temp_filename())) { + result = -1; goto cleanup; + } - if ((result = copy_uri_to_file(filename.buf, uri))) { + if ((result = copy_uri_to_file(filename, uri))) { warning(_("failed to download bundle from URI '%s'"), uri); goto cleanup; } - if ((result = !is_bundle(filename.buf, 0))) { + if ((result = !is_bundle(filename, 0))) { warning(_("file at URI '%s' is not a bundle"), uri); goto cleanup; } - if ((result = unbundle_from_file(r, filename.buf))) { + if ((result = unbundle_from_file(r, filename))) { warning(_("failed to unbundle bundle from URI '%s'"), uri); goto cleanup; } cleanup: - unlink(filename.buf); - strbuf_release(&filename); + if (filename) + unlink(filename); + free(filename); return result; } From patchwork Wed Oct 12 12:52:29 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 13005046 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6980BC4332F for ; Wed, 12 Oct 2022 12:52:52 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229724AbiJLMwu (ORCPT ); Wed, 12 Oct 2022 08:52:50 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37712 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229702AbiJLMws (ORCPT ); Wed, 12 Oct 2022 08:52:48 -0400 Received: from mail-wr1-x436.google.com (mail-wr1-x436.google.com [IPv6:2a00:1450:4864:20::436]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DC225B2DAF for ; Wed, 12 Oct 2022 05:52:46 -0700 (PDT) Received: by mail-wr1-x436.google.com with SMTP id bv10so22543018wrb.4 for ; Wed, 12 Oct 2022 05:52:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:from:to:cc:subject:date :message-id:reply-to; bh=0yrgCoeY0AokBCk/7CF7non/FPEu/1rHXlEpUZhWD/s=; b=DpO26lbv2hm/QUW/1L73qG8QTE8uU3HlS2TUv9SNafckj4hjhrkpHAmyN4UfEx0Lnj pGSYiTC4r7XYfM1kEinwmPhUB8Dbx0lniqK/zHmD3YAhxvn4ra64GBf/mYJDNfBLQ+Sr rSjdMFa3llLusgdCrjvgG0OJcQebBvPk9B8rnZ7ccFeT/6l+JGUskNYwYTbwab9r1w5I FsW2aG4XjPigIf7xptK5ZSCs0CH9zaCqUpSZKF6VR8YToo/mswK4KT4bxEBaMb8pNRsP K7M1BqX9EWFpmSkwWqBFwBcUg7pcLXrSkpfW7Xci/LM6Byyo7FnXrM0bC5g3927XFdX2 vVeQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=0yrgCoeY0AokBCk/7CF7non/FPEu/1rHXlEpUZhWD/s=; b=RzVriXfzn45IYmwsbAYR3bo1hcK7f+MBaM+jP5qP73I8mdird8LFrQ+toW09fOkFSg S5zLJtFpfymDj7ljSSrG/7S1yfgNtdOmk7AWWMyknmf6iPuMTPXwN5ykSk2hwSux80Yx QYTQfAamNxdtEfR74odaSna0W8UFzWkTUj4Sb4KpEdIm+qCmgjDEcPGM2/SWqp0ulxlQ qhqu/REZOmdx+jB9uc1zD2ed7uTsda8d3hRoMsSZzjAaQjlE1yhhpjhs3A2mcj2STtc3 Ba6GaQ2f0G/cQI9Q5fXavsniuAmyBhjnyJ+9e666Nl1sk4NghFfHFGgdcfKp/496VyoD dUVg== X-Gm-Message-State: ACrzQf2iSI0d9Huxax0MyCUu9/fwIfwVndlZNCXSAng92KTA/mH++mFL t/YuJCCmMJFDaflLA1sfZUIexKU8qTo= X-Google-Smtp-Source: AMsMyM4oVI5iJn6Vgm34MHR1ZsAgZ2KJHF8UjbKqG7yQ0aCUBkAjJV/6RBdwlL548/aWGfC5aLUhNQ== X-Received: by 2002:a5d:4fcb:0:b0:230:3f05:c18e with SMTP id h11-20020a5d4fcb000000b002303f05c18emr11315560wrw.530.1665579165159; Wed, 12 Oct 2022 05:52:45 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id n10-20020a05600c3b8a00b003c6b7f55673sm1846456wms.2.2022.10.12.05.52.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 12 Oct 2022 05:52:44 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Wed, 12 Oct 2022 12:52:29 +0000 Subject: [PATCH v5 02/12] bundle-uri: create bundle_list struct and helpers Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, me@ttaylorr.com, newren@gmail.com, avarab@gmail.com, mjcheetham@outlook.com, steadmon@google.com, Glen Choo , Jonathan Tan , Teng Long , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee It will likely be rare where a user uses a single bundle URI and expects that URI to point to a bundle. Instead, that URI will likely be a list of bundles provided in some format. Alternatively, the Git server could advertise a list of bundles. In anticipation of these two ways of advertising multiple bundles, create a data structure that represents such a list. This will be populated using a common API, but for now focus on what data can be represented. Each list contains a number of remote_bundle_info structs. These contain an 'id' that is used to uniquely identify them in the list, and also a 'uri' that contains the location of its data. Finally, there is a strbuf containing the filename used when Git downloads the contents to disk. The list itself stores these remote_bundle_info structs in a hashtable using 'id' as the key. The order of the structs in the input is considered unimportant, but future modifications to the format and these data structures will place ordering possibilities on the set. The list also has a few "global" properties, including the version (used when parsing the list) and the mode. The mode is one of these two options: 1. BUNDLE_MODE_ALL: all listed URIs are intended to be combined together. The client should download all of the advertised data to have a complete copy of the data. 2. BUNDLE_MODE_ANY: any one listed item is sufficient to have a complete copy of the data. The client can choose arbitrarily from these options. In the future, the client may use pings to find the closest URI among geodistributed replicas, or use some other heuristic information added to the format. This API is currently unused, but will soon be expanded with parsing logic and then be consumed by the bundle URI download logic. Signed-off-by: Derrick Stolee --- bundle-uri.c | 60 ++++++++++++++++++++++++++++++++++++++++++++++++++++ bundle-uri.h | 56 ++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 116 insertions(+) diff --git a/bundle-uri.c b/bundle-uri.c index 8b2f4e08c9c..f9a8db221bc 100644 --- a/bundle-uri.c +++ b/bundle-uri.c @@ -4,6 +4,66 @@ #include "object-store.h" #include "refs.h" #include "run-command.h" +#include "hashmap.h" +#include "pkt-line.h" + +static int compare_bundles(const void *hashmap_cmp_fn_data, + const struct hashmap_entry *he1, + const struct hashmap_entry *he2, + const void *id) +{ + const struct remote_bundle_info *e1 = + container_of(he1, const struct remote_bundle_info, ent); + const struct remote_bundle_info *e2 = + container_of(he2, const struct remote_bundle_info, ent); + + return strcmp(e1->id, id ? (const char *)id : e2->id); +} + +void init_bundle_list(struct bundle_list *list) +{ + memset(list, 0, sizeof(*list)); + + /* Implied defaults. */ + list->mode = BUNDLE_MODE_ALL; + list->version = 1; + + hashmap_init(&list->bundles, compare_bundles, NULL, 0); +} + +static int clear_remote_bundle_info(struct remote_bundle_info *bundle, + void *data) +{ + FREE_AND_NULL(bundle->id); + FREE_AND_NULL(bundle->uri); + return 0; +} + +void clear_bundle_list(struct bundle_list *list) +{ + if (!list) + return; + + for_all_bundles_in_list(list, clear_remote_bundle_info, NULL); + hashmap_clear_and_free(&list->bundles, struct remote_bundle_info, ent); +} + +int for_all_bundles_in_list(struct bundle_list *list, + bundle_iterator iter, + void *data) +{ + struct remote_bundle_info *info; + struct hashmap_iter i; + + hashmap_for_each_entry(&list->bundles, &i, info, ent) { + int result = iter(info, data); + + if (result) + return result; + } + + return 0; +} static char *find_temp_filename(void) { diff --git a/bundle-uri.h b/bundle-uri.h index 8a152f1ef14..ff7e3fd3fb2 100644 --- a/bundle-uri.h +++ b/bundle-uri.h @@ -1,7 +1,63 @@ #ifndef BUNDLE_URI_H #define BUNDLE_URI_H +#include "hashmap.h" +#include "strbuf.h" + struct repository; +struct string_list; + +/** + * The remote_bundle_info struct contains information for a single bundle + * URI. This may be initialized simply by a given URI or might have + * additional metadata associated with it if the bundle was advertised by + * a bundle list. + */ +struct remote_bundle_info { + struct hashmap_entry ent; + + /** + * The 'id' is a name given to the bundle for reference + * by other bundle infos. + */ + char *id; + + /** + * The 'uri' is the location of the remote bundle so + * it can be downloaded on-demand. This will be NULL + * if there was no table of contents. + */ + char *uri; +}; + +#define REMOTE_BUNDLE_INFO_INIT { 0 } + +enum bundle_list_mode { + BUNDLE_MODE_NONE = 0, + BUNDLE_MODE_ALL, + BUNDLE_MODE_ANY +}; + +/** + * A bundle_list contains an unordered set of remote_bundle_info structs, + * as well as information about the bundle listing, such as version and + * mode. + */ +struct bundle_list { + int version; + enum bundle_list_mode mode; + struct hashmap bundles; +}; + +void init_bundle_list(struct bundle_list *list); +void clear_bundle_list(struct bundle_list *list); + +typedef int (*bundle_iterator)(struct remote_bundle_info *bundle, + void *data); + +int for_all_bundles_in_list(struct bundle_list *list, + bundle_iterator iter, + void *data); /** * Fetch data from the given 'uri' and unbundle the bundle data found From patchwork Wed Oct 12 12:52:30 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 13005047 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 851E5C433FE for ; Wed, 12 Oct 2022 12:53:02 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229788AbiJLMxB (ORCPT ); Wed, 12 Oct 2022 08:53:01 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37730 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229615AbiJLMwt (ORCPT ); Wed, 12 Oct 2022 08:52:49 -0400 Received: from mail-wr1-x42e.google.com (mail-wr1-x42e.google.com [IPv6:2a00:1450:4864:20::42e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D55ABB40DF for ; Wed, 12 Oct 2022 05:52:47 -0700 (PDT) Received: by mail-wr1-x42e.google.com with SMTP id u10so26112137wrq.2 for ; Wed, 12 Oct 2022 05:52:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:from:to:cc:subject:date :message-id:reply-to; bh=0hj8uYxEbb7O8YWYr7EtPuF6YNvQZhm/4XCGYE1+v8Y=; b=PGjtDnHhVqK8h6fJyGVeLY5AACTk6XrOTKg+azbLZgCg2bY0ZFMsLgPe4lu86mwfR5 McjGCBAFwD/qOLdnywCz+RzS4m4lyUN1aw6tvSM//eYRKr0RxeIdvVYLlNBQbWEgXSS6 cCg7Xp0bk00RfzFNoiJsYfnEy9ldfXiq3GzamuMhYuHa9O6OX0M78mSpCMzgoDzRsAHA JTdgyO5E+XBP9DyVhxeatprddFUfWFBZcpoHAqGm/yXITX00V8NWf9A+GBb5wTWJXfNN rKR6sRILo8D9319R3WlQVtM1nEIJ6loXjLTUXL0nuoI2bLEOHwupxayw1M/EQZzLvkXH VZlA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=0hj8uYxEbb7O8YWYr7EtPuF6YNvQZhm/4XCGYE1+v8Y=; b=zOEPtMr8OShgQBl1e5rVtPlOx4KGESdx+5zOFNLTMCEc67DkHDeMpbkv/QjfG9NvdA Rq5yBpjzFmtb/eVarIR7QonrWvd69zoSb+MJZ9t7OsUvqsNVV8EWCRIzCtdM7LYT3puf m19fQzaN4YhxZp8vznewNSZ0uC4fyYnzNmH8fXf09V1ydiMtlGWyplaRGRO7szEbn/6W ciY9hfo69WDLuVEFdIIpiKaduRlEVImWmfzhlIQGWu259BcG+i8bcxhzdSAiIbqNnEzS BBhpF2csVerYUEWwx/QVspvz7G7rFTgna6QE3h6wSAmb5ARhTyS9iAqvCv4sX23WmpwM D44A== X-Gm-Message-State: ACrzQf2OU26I7YsFGLEq7tkvulAynBbmCb3Z+dNE7X4NDdFlbadqFN3C cGXvuVyHWiBVV3iE0Met+/r91hoxvW8= X-Google-Smtp-Source: AMsMyM57h/2qz5kzUXyUi8kWx2CjHssRfFaWMZTDeTGc4vyqm8th/i8vUI2wJ02vpbakDReMTJruQw== X-Received: by 2002:adf:df8b:0:b0:230:5212:d364 with SMTP id z11-20020adfdf8b000000b002305212d364mr10386614wrl.124.1665579166116; Wed, 12 Oct 2022 05:52:46 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id q65-20020a1c4344000000b003a8434530bbsm1768939wma.13.2022.10.12.05.52.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 12 Oct 2022 05:52:45 -0700 (PDT) Message-Id: <430e01cd2a4127a17c4bfc1b0db7209964736de8.1665579160.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Wed, 12 Oct 2022 12:52:30 +0000 Subject: [PATCH v5 03/12] bundle-uri: create base key-value pair parsing Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, me@ttaylorr.com, newren@gmail.com, avarab@gmail.com, mjcheetham@outlook.com, steadmon@google.com, Glen Choo , Jonathan Tan , Teng Long , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee There will be two primary ways to advertise a bundle list: as a list of packet lines in Git's protocol v2 and as a config file served from a bundle URI. Both of these fundamentally use a list of key-value pairs. We will use the same set of key-value pairs across these formats. Create a new bundle_list_update() method that is currently unusued, but will be used in the next change. It inspects each key to see if it is understood and then applies it to the given bundle_list. Here are the keys that we teach Git to understand: * bundle.version: This value should be an integer. Git currently understands only version 1 and will ignore the list if the version is any other value. This version can be increased in the future if we need to add new keys that Git should not ignore. We can add new "heuristic" keys without incrementing the version. * bundle.mode: This value should be one of "all" or "any". If this mode is not understood, then Git will ignore the list. This mode indicates whether Git needs all of the bundle list items to make a complete view of the content or if any single item is sufficient. The rest of the keys use a bundle identifier "" as part of the key name. Keys using the same "" describe a single bundle list item. * bundle..uri: This stores the URI of the bundle item. This currently is expected to be an absolute URI, but will be relaxed to be a relative URI in the future. While parsing, return an error if a URI key is repeated, since we can make that restriction with bundle lists. Make the git_parse_int() method global so we can parse the integer version value carefully. Signed-off-by: Derrick Stolee --- Documentation/config.txt | 2 + Documentation/config/bundle.txt | 24 +++++++++++ bundle-uri.c | 76 +++++++++++++++++++++++++++++++++ config.c | 2 +- config.h | 1 + 5 files changed, 104 insertions(+), 1 deletion(-) create mode 100644 Documentation/config/bundle.txt diff --git a/Documentation/config.txt b/Documentation/config.txt index e376d547ce0..4280af6992e 100644 --- a/Documentation/config.txt +++ b/Documentation/config.txt @@ -387,6 +387,8 @@ include::config/branch.txt[] include::config/browser.txt[] +include::config/bundle.txt[] + include::config/checkout.txt[] include::config/clean.txt[] diff --git a/Documentation/config/bundle.txt b/Documentation/config/bundle.txt new file mode 100644 index 00000000000..daa21eb674a --- /dev/null +++ b/Documentation/config/bundle.txt @@ -0,0 +1,24 @@ +bundle.*:: + The `bundle.*` keys may appear in a bundle list file found via the + `git clone --bundle-uri` option. These keys currently have no effect + if placed in a repository config file, though this will change in the + future. See link:technical/bundle-uri.html[the bundle URI design + document] for more details. + +bundle.version:: + This integer value advertises the version of the bundle list format + used by the bundle list. Currently, the only accepted value is `1`. + +bundle.mode:: + This string value should be either `all` or `any`. This value describes + whether all of the advertised bundles are required to unbundle a + complete understanding of the bundled information (`all`) or if any one + of the listed bundle URIs is sufficient (`any`). + +bundle..*:: + The `bundle..*` keys are used to describe a single item in the + bundle list, grouped under `` for identification purposes. + +bundle..uri:: + This string value defines the URI by which Git can reach the contents + of this ``. This URI may be a bundle file or another bundle list. diff --git a/bundle-uri.c b/bundle-uri.c index f9a8db221bc..0bc59dd9c34 100644 --- a/bundle-uri.c +++ b/bundle-uri.c @@ -6,6 +6,7 @@ #include "run-command.h" #include "hashmap.h" #include "pkt-line.h" +#include "config.h" static int compare_bundles(const void *hashmap_cmp_fn_data, const struct hashmap_entry *he1, @@ -65,6 +66,81 @@ int for_all_bundles_in_list(struct bundle_list *list, return 0; } +/** + * Given a key-value pair, update the state of the given bundle list. + * Returns 0 if the key-value pair is understood. Returns -1 if the key + * is not understood or the value is malformed. + */ +MAYBE_UNUSED +static int bundle_list_update(const char *key, const char *value, + struct bundle_list *list) +{ + struct strbuf id = STRBUF_INIT; + struct remote_bundle_info lookup = REMOTE_BUNDLE_INFO_INIT; + struct remote_bundle_info *bundle; + const char *subsection, *subkey; + size_t subsection_len; + + if (parse_config_key(key, "bundle", &subsection, &subsection_len, &subkey)) + return -1; + + if (!subsection_len) { + if (!strcmp(subkey, "version")) { + int version; + if (!git_parse_int(value, &version)) + return -1; + if (version != 1) + return -1; + + list->version = version; + return 0; + } + + if (!strcmp(subkey, "mode")) { + if (!strcmp(value, "all")) + list->mode = BUNDLE_MODE_ALL; + else if (!strcmp(value, "any")) + list->mode = BUNDLE_MODE_ANY; + else + return -1; + return 0; + } + + /* Ignore other unknown global keys. */ + return 0; + } + + strbuf_add(&id, subsection, subsection_len); + + /* + * Check for an existing bundle with this , or create one + * if necessary. + */ + lookup.id = id.buf; + hashmap_entry_init(&lookup.ent, strhash(lookup.id)); + if (!(bundle = hashmap_get_entry(&list->bundles, &lookup, ent, NULL))) { + CALLOC_ARRAY(bundle, 1); + bundle->id = strbuf_detach(&id, NULL); + hashmap_entry_init(&bundle->ent, strhash(bundle->id)); + hashmap_add(&list->bundles, &bundle->ent); + } + strbuf_release(&id); + + if (!strcmp(subkey, "uri")) { + if (bundle->uri) + return -1; + bundle->uri = xstrdup(value); + return 0; + } + + /* + * At this point, we ignore any information that we don't + * understand, assuming it to be hints for a heuristic the client + * does not currently understand. + */ + return 0; +} + static char *find_temp_filename(void) { int fd; diff --git a/config.c b/config.c index 015bec360f5..e93101249f6 100644 --- a/config.c +++ b/config.c @@ -1214,7 +1214,7 @@ static int git_parse_unsigned(const char *value, uintmax_t *ret, uintmax_t max) return 0; } -static int git_parse_int(const char *value, int *ret) +int git_parse_int(const char *value, int *ret) { intmax_t tmp; if (!git_parse_signed(value, &tmp, maximum_signed_value_of_type(int))) diff --git a/config.h b/config.h index ca994d77147..ef9eade6414 100644 --- a/config.h +++ b/config.h @@ -206,6 +206,7 @@ int config_with_options(config_fn_t fn, void *, int git_parse_ssize_t(const char *, ssize_t *); int git_parse_ulong(const char *, unsigned long *); +int git_parse_int(const char *value, int *ret); /** * Same as `git_config_bool`, except that it returns -1 on error rather From patchwork Wed Oct 12 12:52:31 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= X-Patchwork-Id: 13005049 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 949C2C433FE for ; Wed, 12 Oct 2022 12:53:06 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229682AbiJLMxF (ORCPT ); Wed, 12 Oct 2022 08:53:05 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37828 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229727AbiJLMww (ORCPT ); Wed, 12 Oct 2022 08:52:52 -0400 Received: from mail-wm1-x32f.google.com (mail-wm1-x32f.google.com [IPv6:2a00:1450:4864:20::32f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4DAFBC45A6 for ; Wed, 12 Oct 2022 05:52:49 -0700 (PDT) Received: by mail-wm1-x32f.google.com with SMTP id l16-20020a05600c4f1000b003c6c0d2a445so1148502wmq.4 for ; Wed, 12 Oct 2022 05:52:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:fcc:content-transfer-encoding:mime-version:subject:date:from :references:in-reply-to:message-id:from:to:cc:subject:date :message-id:reply-to; bh=fnuvRU/s4CwFnk6H9aTDmG6Z/zW4bmJU9S4El1WdLLY=; b=pd/SrKg+4mfOrC38oQqQp+0AfPY30nu4ANHJ+55mAxwjT0rJzxlULSANKslo2zwcKg 4t9ngpESbyquHmSVbi8bRh42iT1+q4QZBXwKcSI+Ad7WwFbKpu+MMO5vaZRMH5PfL+TZ PSLnE4779iPJFBNtAy6LDECsbeuvtkAjBkAembAq7YXhR7ipmf0Y8Tc16WPUYkqoGTtE FX5n7DuwWWGStKrt8v/Ut86rcaQWzpWyq3/0ib5wyS/KPRsRd6fgk6bLdp8zHK8TGbf5 p4l1cjVlG+Lk5ikqR7wd49W3M5lhXshxlsPeYrk4ofM3bOPJ9mdTIYO7m4c227WyDv4W lyxQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:fcc:content-transfer-encoding:mime-version:subject:date:from :references:in-reply-to:message-id:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=fnuvRU/s4CwFnk6H9aTDmG6Z/zW4bmJU9S4El1WdLLY=; b=gGsGnGPQWE4uf4vav87zG+BTazIAfoUcHzrP05UZGP+Fr7EeqZ/l8h4ktMzRhnHRFM wMQ4Q6KsDJmQ4J9AsBkvQRRwUz2HjGcRNubW7gH1I+d7RU4sVWs2ZFbLTg1K54u2qVUV vbgi4DlaOekK0PANNpeNO/cVZJ7X8rAI09tCbQ4CjplLSyqdJLK67J93oJO8eJuhoClq kAfWzxVjF8ge7SEiuvIrSeuX0Ib8Ol9oqlejdSx9WMYepBq4DMSlzVXtmUaOaynr3IXO 3SScRgrYue2CwfEneoQOQHUIoM14pWOjpmTYe4FIYGunAN66JFOnYu9+u/kCsvAG58nj OP2A== X-Gm-Message-State: ACrzQf3K4whvYbvca9WKRpGIm0vMRgzzRVmTjm4K54lxPivToZSB/joA dqFMz97YNQp3FTwA5bw5/hZiOeeWq0o= X-Google-Smtp-Source: AMsMyM5eAbZ2J5BqQATVwnTgZjVrmBU5vzjOy4TXYMpOLe14BqV39AsKOCFq4pJ4LbCcfTTjGAgnBg== X-Received: by 2002:a05:600c:474c:b0:3c6:c790:1f95 with SMTP id w12-20020a05600c474c00b003c6c7901f95mr2716200wmo.18.1665579167329; Wed, 12 Oct 2022 05:52:47 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id p2-20020adfce02000000b0022a3a887ceasm13633120wrn.49.2022.10.12.05.52.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 12 Oct 2022 05:52:46 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Wed, 12 Oct 2022 12:52:31 +0000 Subject: [PATCH v5 04/12] bundle-uri: create "key=value" line parsing MIME-Version: 1.0 Fcc: Sent To: git@vger.kernel.org Cc: gitster@pobox.com, me@ttaylorr.com, newren@gmail.com, avarab@gmail.com, mjcheetham@outlook.com, steadmon@google.com, Glen Choo , Jonathan Tan , Teng Long , Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7AgQmphcm1hc29u?= Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= From: =?UTF-8?q?=C3=86var=20Arnfj=C3=B6r=C3=B0=20Bjarmason?= When advertising a bundle list over Git's protocol v2, we will use packet lines. Each line will be of the form "key=value" representing a bundle list. Connect the API necessary for Git's transport to the key-value pair parsing created in the previous change. We are not currently implementing this protocol v2 functionality, but instead preparing to expose this parsing to be unit-testable. Co-authored-by: Derrick Stolee Signed-off-by: Ævar Arnfjörð Bjarmason Signed-off-by: Derrick Stolee --- bundle-uri.c | 27 ++++++++++++++++++++++++++- bundle-uri.h | 12 ++++++++++++ 2 files changed, 38 insertions(+), 1 deletion(-) diff --git a/bundle-uri.c b/bundle-uri.c index 0bc59dd9c34..372e6fac5cf 100644 --- a/bundle-uri.c +++ b/bundle-uri.c @@ -71,7 +71,6 @@ int for_all_bundles_in_list(struct bundle_list *list, * Returns 0 if the key-value pair is understood. Returns -1 if the key * is not understood or the value is malformed. */ -MAYBE_UNUSED static int bundle_list_update(const char *key, const char *value, struct bundle_list *list) { @@ -306,3 +305,29 @@ cleanup: free(filename); return result; } + +/** + * General API for {transport,connect}.c etc. + */ +int bundle_uri_parse_line(struct bundle_list *list, const char *line) +{ + int result; + const char *equals; + struct strbuf key = STRBUF_INIT; + + if (!strlen(line)) + return error(_("bundle-uri: got an empty line")); + + equals = strchr(line, '='); + + if (!equals) + return error(_("bundle-uri: line is not of the form 'key=value'")); + if (line == equals || !*(equals + 1)) + return error(_("bundle-uri: line has empty key or value")); + + strbuf_add(&key, line, equals - line); + result = bundle_list_update(key.buf, equals + 1, list); + strbuf_release(&key); + + return result; +} diff --git a/bundle-uri.h b/bundle-uri.h index ff7e3fd3fb2..90583461929 100644 --- a/bundle-uri.h +++ b/bundle-uri.h @@ -67,4 +67,16 @@ int for_all_bundles_in_list(struct bundle_list *list, */ int fetch_bundle_uri(struct repository *r, const char *uri); +/** + * General API for {transport,connect}.c etc. + */ + +/** + * Parse a "key=value" packet line from the bundle-uri verb. + * + * Returns 0 on success and non-zero on error. + */ +int bundle_uri_parse_line(struct bundle_list *list, + const char *line); + #endif From patchwork Wed Oct 12 12:52:32 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= X-Patchwork-Id: 13005048 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id CBFD4C4332F for ; Wed, 12 Oct 2022 12:53:04 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229745AbiJLMxC (ORCPT ); Wed, 12 Oct 2022 08:53:02 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37812 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229702AbiJLMww (ORCPT ); Wed, 12 Oct 2022 08:52:52 -0400 Received: from mail-wr1-x436.google.com (mail-wr1-x436.google.com [IPv6:2a00:1450:4864:20::436]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6A4F1C45A9 for ; Wed, 12 Oct 2022 05:52:49 -0700 (PDT) Received: by mail-wr1-x436.google.com with SMTP id bv10so22543207wrb.4 for ; Wed, 12 Oct 2022 05:52:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:fcc:content-transfer-encoding:mime-version:subject:date:from :references:in-reply-to:message-id:from:to:cc:subject:date :message-id:reply-to; bh=iaFbGpA3MUI0PWvxsx9+QfUfh+g+qL2/Zoy6IB3I3j0=; b=f2gzN4elbFq+H13pGzaQtmIkWIzxp54BLBo1D125mLs2M1rbd92Bhc3s3aw5Z4Jk64 HaHF7J8nFgcJF8AApT6MfGS07VGyEmbSSKR67K/xuhDtOkS8C1gabd8FKFQ9I4ZdF+IU HHmnnG3wq54UCvIRQpNDwlDmcgWT3VNYjd9dGoPYAfndP1ByatvGWVxV8P5XqL1QTXxS enlcLpOOus24sJtVSwwtsbwhnNua/SRvdY8GBeM7WqpABQbBL6+Ux2saO4ZzE1XzFQMC 8jY1sqAuj6qnJPK90SZqJZ3P/PLfqVkZlb+AkDsVR1cHpkf1Lyha6iG42ked+naSzB+i pKWQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:fcc:content-transfer-encoding:mime-version:subject:date:from :references:in-reply-to:message-id:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=iaFbGpA3MUI0PWvxsx9+QfUfh+g+qL2/Zoy6IB3I3j0=; b=vSjDIgcvvWPLcPy040GfSupVw8MqVeiKXVoI+/1wVg8JfZ8BPPDC4ZYWHcm0amlG7G NyYHoe7UTU9LJy619OwsoOUB6FbSMZdd7l5IgVQqUygHLYKhkdEDpxmgpb0k7CEmvYQO 3w7XXB4Sp/j9Zy2yUKSRvJ1bIG2wHFvIgFQQWSXhA2XSZzIV9Btxh05+C5j5pcVw6sxv 9TJ4QQcbnIXSk/jK31kDVmyvBaD8kIacqnUu1uL+3enVRZoRdWXZAYB9iJTHjFlG5sFx 7QETOwtsAIc2fWZ8497vrgChsktLJ5zUgEq9BTCfrUXfpl5VTd0+3ewQ0DRnTXxe9Yfp nm6w== X-Gm-Message-State: ACrzQf3M/QVqsxQIS0vvUi8gAdPcNr2IoOS82lzuYZQ2JSflSSBq2xuN LVvyleyAVaVN3q0J9h+mZUKj+tK1eTw= X-Google-Smtp-Source: AMsMyM7t/0UaQaquax+y/zDAXczkGESKHW7iXPhyxBTFat04id2PbN6DAAGlyuHibDPnWTuOvEZK6A== X-Received: by 2002:a5d:614a:0:b0:230:a4f8:4da2 with SMTP id y10-20020a5d614a000000b00230a4f84da2mr9129550wrt.277.1665579168371; Wed, 12 Oct 2022 05:52:48 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id co11-20020a0560000a0b00b00228d52b935asm14163402wrb.71.2022.10.12.05.52.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 12 Oct 2022 05:52:47 -0700 (PDT) Message-Id: <4d8cac67f66fc9c6477efabde795da95fafc51ec.1665579160.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Wed, 12 Oct 2022 12:52:32 +0000 Subject: [PATCH v5 05/12] bundle-uri: unit test "key=value" parsing MIME-Version: 1.0 Fcc: Sent To: git@vger.kernel.org Cc: gitster@pobox.com, me@ttaylorr.com, newren@gmail.com, avarab@gmail.com, mjcheetham@outlook.com, steadmon@google.com, Glen Choo , Jonathan Tan , Teng Long , Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7AgQmphcm1hc29u?= Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= From: =?UTF-8?q?=C3=86var=20Arnfj=C3=B6r=C3=B0=20Bjarmason?= Create a new 'test-tool bundle-uri' test helper. This helper will assist in testing logic deep in the bundle URI feature. This change introduces the 'parse-key-values' subcommand, which parses an input file as a list of lines. These are fed into bundle_uri_parse_line() to test how we construct a 'struct bundle_list' from that data. The list is then output to stdout as if the key-value pairs were a Git config file. We use an input file instead of stdin because of a future change to parse in config-file format that works better as an input file. Co-authored-by: Derrick Stolee Signed-off-by: Ævar Arnfjörð Bjarmason Signed-off-by: Derrick Stolee --- Makefile | 1 + bundle-uri.c | 33 ++++++++++ bundle-uri.h | 3 + t/helper/test-bundle-uri.c | 70 +++++++++++++++++++++ t/helper/test-tool.c | 1 + t/helper/test-tool.h | 1 + t/t5750-bundle-uri-parse.sh | 121 ++++++++++++++++++++++++++++++++++++ t/test-lib-functions.sh | 11 ++++ 8 files changed, 241 insertions(+) create mode 100644 t/helper/test-bundle-uri.c create mode 100755 t/t5750-bundle-uri-parse.sh diff --git a/Makefile b/Makefile index 7d5f48069ea..7dee0329c49 100644 --- a/Makefile +++ b/Makefile @@ -722,6 +722,7 @@ PROGRAMS += $(patsubst %.o,git-%$X,$(PROGRAM_OBJS)) TEST_BUILTINS_OBJS += test-advise.o TEST_BUILTINS_OBJS += test-bitmap.o TEST_BUILTINS_OBJS += test-bloom.o +TEST_BUILTINS_OBJS += test-bundle-uri.o TEST_BUILTINS_OBJS += test-chmtime.o TEST_BUILTINS_OBJS += test-config.o TEST_BUILTINS_OBJS += test-crontab.o diff --git a/bundle-uri.c b/bundle-uri.c index 372e6fac5cf..c02e7f62eb1 100644 --- a/bundle-uri.c +++ b/bundle-uri.c @@ -66,6 +66,39 @@ int for_all_bundles_in_list(struct bundle_list *list, return 0; } +static int summarize_bundle(struct remote_bundle_info *info, void *data) +{ + FILE *fp = data; + fprintf(fp, "[bundle \"%s\"]\n", info->id); + fprintf(fp, "\turi = %s\n", info->uri); + return 0; +} + +void print_bundle_list(FILE *fp, struct bundle_list *list) +{ + const char *mode; + + switch (list->mode) { + case BUNDLE_MODE_ALL: + mode = "all"; + break; + + case BUNDLE_MODE_ANY: + mode = "any"; + break; + + case BUNDLE_MODE_NONE: + default: + mode = ""; + } + + fprintf(fp, "[bundle]\n"); + fprintf(fp, "\tversion = %d\n", list->version); + fprintf(fp, "\tmode = %s\n", mode); + + for_all_bundles_in_list(list, summarize_bundle, fp); +} + /** * Given a key-value pair, update the state of the given bundle list. * Returns 0 if the key-value pair is understood. Returns -1 if the key diff --git a/bundle-uri.h b/bundle-uri.h index 90583461929..0e56ab2ae5a 100644 --- a/bundle-uri.h +++ b/bundle-uri.h @@ -59,6 +59,9 @@ int for_all_bundles_in_list(struct bundle_list *list, bundle_iterator iter, void *data); +struct FILE; +void print_bundle_list(FILE *fp, struct bundle_list *list); + /** * Fetch data from the given 'uri' and unbundle the bundle data found * based on that information. diff --git a/t/helper/test-bundle-uri.c b/t/helper/test-bundle-uri.c new file mode 100644 index 00000000000..0329c56544f --- /dev/null +++ b/t/helper/test-bundle-uri.c @@ -0,0 +1,70 @@ +#include "test-tool.h" +#include "parse-options.h" +#include "bundle-uri.h" +#include "strbuf.h" +#include "string-list.h" + +static int cmd__bundle_uri_parse(int argc, const char **argv) +{ + const char *key_value_usage[] = { + "test-tool bundle-uri parse-key-values ", + NULL + }; + const char **usage = key_value_usage; + struct option options[] = { + OPT_END(), + }; + struct strbuf sb = STRBUF_INIT; + struct bundle_list list; + int err = 0; + FILE *fp; + + argc = parse_options(argc, argv, NULL, options, usage, 0); + if (argc != 1) + goto usage; + + init_bundle_list(&list); + fp = fopen(argv[0], "r"); + if (!fp) + die("failed to open '%s'", argv[0]); + + while (strbuf_getline(&sb, fp) != EOF) { + if (bundle_uri_parse_line(&list, sb.buf)) + err = error("bad line: '%s'", sb.buf); + } + strbuf_release(&sb); + fclose(fp); + + print_bundle_list(stdout, &list); + + clear_bundle_list(&list); + + return !!err; + +usage: + usage_with_options(usage, options); +} + +int cmd__bundle_uri(int argc, const char **argv) +{ + const char *usage[] = { + "test-tool bundle-uri []", + NULL + }; + struct option options[] = { + OPT_END(), + }; + + argc = parse_options(argc, argv, NULL, options, usage, + PARSE_OPT_STOP_AT_NON_OPTION | + PARSE_OPT_KEEP_ARGV0); + if (argc == 1) + goto usage; + + if (!strcmp(argv[1], "parse-key-values")) + return cmd__bundle_uri_parse(argc - 1, argv + 1); + error("there is no test-tool bundle-uri tool '%s'", argv[1]); + +usage: + usage_with_options(usage, options); +} diff --git a/t/helper/test-tool.c b/t/helper/test-tool.c index 318fdbab0c3..fbe2d9d8108 100644 --- a/t/helper/test-tool.c +++ b/t/helper/test-tool.c @@ -17,6 +17,7 @@ static struct test_cmd cmds[] = { { "advise", cmd__advise_if_enabled }, { "bitmap", cmd__bitmap }, { "bloom", cmd__bloom }, + { "bundle-uri", cmd__bundle_uri }, { "chmtime", cmd__chmtime }, { "config", cmd__config }, { "crontab", cmd__crontab }, diff --git a/t/helper/test-tool.h b/t/helper/test-tool.h index bb799271631..b2aa1f39a8f 100644 --- a/t/helper/test-tool.h +++ b/t/helper/test-tool.h @@ -7,6 +7,7 @@ int cmd__advise_if_enabled(int argc, const char **argv); int cmd__bitmap(int argc, const char **argv); int cmd__bloom(int argc, const char **argv); +int cmd__bundle_uri(int argc, const char **argv); int cmd__chmtime(int argc, const char **argv); int cmd__config(int argc, const char **argv); int cmd__crontab(int argc, const char **argv); diff --git a/t/t5750-bundle-uri-parse.sh b/t/t5750-bundle-uri-parse.sh new file mode 100755 index 00000000000..fd142a66ad5 --- /dev/null +++ b/t/t5750-bundle-uri-parse.sh @@ -0,0 +1,121 @@ +#!/bin/sh + +test_description="Test bundle-uri bundle_uri_parse_line()" + +TEST_NO_CREATE_REPO=1 +TEST_PASSES_SANITIZE_LEAK=true +. ./test-lib.sh + +test_expect_success 'bundle_uri_parse_line() just URIs' ' + cat >in <<-\EOF && + bundle.one.uri=http://example.com/bundle.bdl + bundle.two.uri=https://example.com/bundle.bdl + bundle.three.uri=file:///usr/share/git/bundle.bdl + EOF + + cat >expect <<-\EOF && + [bundle] + version = 1 + mode = all + [bundle "one"] + uri = http://example.com/bundle.bdl + [bundle "two"] + uri = https://example.com/bundle.bdl + [bundle "three"] + uri = file:///usr/share/git/bundle.bdl + EOF + + test-tool bundle-uri parse-key-values in >actual 2>err && + test_must_be_empty err && + test_cmp_config_output expect actual +' + +test_expect_success 'bundle_uri_parse_line() parsing edge cases: empty key or value' ' + cat >in <<-\EOF && + =bogus-value + bogus-key= + EOF + + cat >err.expect <<-EOF && + error: bundle-uri: line has empty key or value + error: bad line: '\''=bogus-value'\'' + error: bundle-uri: line has empty key or value + error: bad line: '\''bogus-key='\'' + EOF + + cat >expect <<-\EOF && + [bundle] + version = 1 + mode = all + EOF + + test_must_fail test-tool bundle-uri parse-key-values in >actual 2>err && + test_cmp err.expect err && + test_cmp_config_output expect actual +' + +test_expect_success 'bundle_uri_parse_line() parsing edge cases: empty lines' ' + cat >in <<-\EOF && + bundle.one.uri=http://example.com/bundle.bdl + + bundle.two.uri=https://example.com/bundle.bdl + + bundle.three.uri=file:///usr/share/git/bundle.bdl + EOF + + cat >err.expect <<-\EOF && + error: bundle-uri: got an empty line + error: bad line: '\'''\'' + error: bundle-uri: got an empty line + error: bad line: '\'''\'' + EOF + + # We fail, but try to continue parsing regardless + cat >expect <<-\EOF && + [bundle] + version = 1 + mode = all + [bundle "one"] + uri = http://example.com/bundle.bdl + [bundle "two"] + uri = https://example.com/bundle.bdl + [bundle "three"] + uri = file:///usr/share/git/bundle.bdl + EOF + + test_must_fail test-tool bundle-uri parse-key-values in >actual 2>err && + test_cmp err.expect err && + test_cmp_config_output expect actual +' + +test_expect_success 'bundle_uri_parse_line() parsing edge cases: duplicate lines' ' + cat >in <<-\EOF && + bundle.one.uri=http://example.com/bundle.bdl + bundle.two.uri=https://example.com/bundle.bdl + bundle.one.uri=https://example.com/bundle-2.bdl + bundle.three.uri=file:///usr/share/git/bundle.bdl + EOF + + cat >err.expect <<-\EOF && + error: bad line: '\''bundle.one.uri=https://example.com/bundle-2.bdl'\'' + EOF + + # We fail, but try to continue parsing regardless + cat >expect <<-\EOF && + [bundle] + version = 1 + mode = all + [bundle "one"] + uri = http://example.com/bundle.bdl + [bundle "two"] + uri = https://example.com/bundle.bdl + [bundle "three"] + uri = file:///usr/share/git/bundle.bdl + EOF + + test_must_fail test-tool bundle-uri parse-key-values in >actual 2>err && + test_cmp err.expect err && + test_cmp_config_output expect actual +' + +test_done diff --git a/t/test-lib-functions.sh b/t/test-lib-functions.sh index 6da7273f1d5..3175d665add 100644 --- a/t/test-lib-functions.sh +++ b/t/test-lib-functions.sh @@ -1956,3 +1956,14 @@ test_is_magic_mtime () { rm -f .git/test-mtime-actual return $ret } + +# Given two filenames, parse both using 'git config --list --file' +# and compare the sorted output of those commands. Useful when +# wanting to ignore whitespace differences and sorting concerns. +test_cmp_config_output () { + git config --list --file="$1" >config-expect && + git config --list --file="$2" >config-actual && + sort config-expect >sorted-expect && + sort config-actual >sorted-actual && + test_cmp sorted-expect sorted-actual +} From patchwork Wed Oct 12 12:52:33 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 13005050 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8F728C4332F for ; Wed, 12 Oct 2022 12:53:15 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229880AbiJLMxO (ORCPT ); Wed, 12 Oct 2022 08:53:14 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38088 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229798AbiJLMxB (ORCPT ); Wed, 12 Oct 2022 08:53:01 -0400 Received: from mail-wm1-x32e.google.com (mail-wm1-x32e.google.com [IPv6:2a00:1450:4864:20::32e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C8BB7C90CA for ; Wed, 12 Oct 2022 05:52:51 -0700 (PDT) Received: by mail-wm1-x32e.google.com with SMTP id e18so10362339wmq.3 for ; Wed, 12 Oct 2022 05:52:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:from:to:cc:subject:date :message-id:reply-to; bh=23xdSrTs1L8KeO1BbMKQpiLa02wLnB63jG2LCHd+drg=; b=Mh1upTGl8hKWsUGZaGg18wabFmu94bEM50gED2oG8Dcb1IyHeQmME5PloPUe5qJLnc zHsiQ1vDZTZ+PYU7c1oZ6jR/FoycG5LMAcWPRoRyc3+rtHcln6qrlhEhCgEkIOJndcwA OY4LVsdU+Tj8xf00zaelZmpm5XUbHlrSMCf+hi90FwV/3dN0FVm+IfndihseMI3Uloma Ld18CoChnkNGxBz0K6f+NAm8l19rzty1d34xFudjgDIJ3Mt+N0VuEZsIjEGSR7zj59eH +ymKXVOXxWhOthT33+yyCyjnDtfDkv+KeEnnULMlPlDtQYyBprNIziv5Bb1esi3sVkqN ALWQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=23xdSrTs1L8KeO1BbMKQpiLa02wLnB63jG2LCHd+drg=; b=yQsTi/jXukKdZumiSln/A7yuvhDHxK6oifhCMmZHaL8C5X9YFtTPkyYw6smUxZakMV LSXmsGgWySsMVPudIpZ2+aIh3gXuU37pFZX8jrKOyof5TgqN9UU2Ut3GUZl8L1S118C9 JxBoyB+LBiAyxJRA2v9+0MvkkMliPsaUuh2bXc6iS7+MVKeF9rkUCZ0gsyqCfYHUZNKF CZqzAUUmlkWu8naavLm19C8PN/Kss8I0wuGhH5WUN0J+b85l7OMAwkKaQ8Xl2adrlpHu ClKDZYRUvw6kMG1G9CqIxj/phGGaMUZLqPzKU38FTo1MNtTA9bHZ+Svu4Q68WP2T4h9K Glrg== X-Gm-Message-State: ACrzQf2RQm212JQowrkZ94lNEHJUdcydR+HZNCx4IsQDFALe7S7GunJs RVg5NBK+B/uvHb5ZKBvC1E9CSn2vSfY= X-Google-Smtp-Source: AMsMyM4J7WuZw+SH+CuCiD3hizlmRPAipTx/7SIyWeUIvYTdW76nGf211ULlkopHHd9YqiIvqVrIbQ== X-Received: by 2002:a05:600c:4f13:b0:3b4:9a07:efdb with SMTP id l19-20020a05600c4f1300b003b49a07efdbmr2671532wmq.94.1665579169545; Wed, 12 Oct 2022 05:52:49 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id ba6-20020a0560001c0600b0022e57e66824sm17347643wrb.99.2022.10.12.05.52.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 12 Oct 2022 05:52:48 -0700 (PDT) Message-Id: <0ecae3a44b360fbb03a52b73536f83c457a6668d.1665579160.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Wed, 12 Oct 2022 12:52:33 +0000 Subject: [PATCH v5 06/12] bundle-uri: parse bundle list in config format Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, me@ttaylorr.com, newren@gmail.com, avarab@gmail.com, mjcheetham@outlook.com, steadmon@google.com, Glen Choo , Jonathan Tan , Teng Long , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee When a bundle provider wants to operate independently from a Git remote, they want to provide a single, consistent URI that users can use in their 'git clone --bundle-uri' commands. At this point, the Git client expects that URI to be a single bundle that can be unbundled and used to bootstrap the rest of the clone from the Git server. This single bundle cannot be re-used to assist with future incremental fetches. To allow for the incremental fetch case, teach Git to understand a bundle list that could be advertised at an independent bundle URI. Such a bundle list is likely to be inspected by human readers, even if only by the bundle provider creating the list. For this reason, we can take our expected "key=value" pairs and instead format them using Git config format. Create bundle_uri_parse_config_format() to parse a file in config format and convert that into a 'struct bundle_list' filled with its understanding of the contents. Be careful to use error_action CONFIG_ERROR_ERROR when calling git_config_from_file_with_options() because the default action for git_config_from_file() is to die() on a parsing error. The current warning isn't particularly helpful if it arises to a user, but it will be made more verbose at a higher layer later. Update 'test-tool bundle-uri' to take this config file format as input. It uses a filename instead of stdin because there is no existing way to parse a FILE pointer in the config machinery. Using git_config_from_mem() is overly complicated and more likely to introduce bugs than this simpler version. Signed-off-by: Derrick Stolee --- bundle-uri.c | 27 ++++++++++++++++++++ bundle-uri.h | 9 +++++++ t/helper/test-bundle-uri.c | 49 +++++++++++++++++++++++++++--------- t/t5750-bundle-uri-parse.sh | 50 +++++++++++++++++++++++++++++++++++++ 4 files changed, 123 insertions(+), 12 deletions(-) diff --git a/bundle-uri.c b/bundle-uri.c index c02e7f62eb1..3d44ec2b1e6 100644 --- a/bundle-uri.c +++ b/bundle-uri.c @@ -173,6 +173,33 @@ static int bundle_list_update(const char *key, const char *value, return 0; } +static int config_to_bundle_list(const char *key, const char *value, void *data) +{ + struct bundle_list *list = data; + return bundle_list_update(key, value, list); +} + +int bundle_uri_parse_config_format(const char *uri, + const char *filename, + struct bundle_list *list) +{ + int result; + struct config_options opts = { + .error_action = CONFIG_ERROR_ERROR, + }; + + result = git_config_from_file_with_options(config_to_bundle_list, + filename, list, + &opts); + + if (!result && list->mode == BUNDLE_MODE_NONE) { + warning(_("bundle list at '%s' has no mode"), uri); + result = 1; + } + + return result; +} + static char *find_temp_filename(void) { int fd; diff --git a/bundle-uri.h b/bundle-uri.h index 0e56ab2ae5a..bc13d4c9929 100644 --- a/bundle-uri.h +++ b/bundle-uri.h @@ -62,6 +62,15 @@ int for_all_bundles_in_list(struct bundle_list *list, struct FILE; void print_bundle_list(FILE *fp, struct bundle_list *list); +/** + * A bundle URI may point to a bundle list where the key=value + * pairs are provided in config file format. This method is + * exposed publicly for testing purposes. + */ +int bundle_uri_parse_config_format(const char *uri, + const char *filename, + struct bundle_list *list); + /** * Fetch data from the given 'uri' and unbundle the bundle data found * based on that information. diff --git a/t/helper/test-bundle-uri.c b/t/helper/test-bundle-uri.c index 0329c56544f..25afd393428 100644 --- a/t/helper/test-bundle-uri.c +++ b/t/helper/test-bundle-uri.c @@ -4,12 +4,21 @@ #include "strbuf.h" #include "string-list.h" -static int cmd__bundle_uri_parse(int argc, const char **argv) +enum input_mode { + KEY_VALUE_PAIRS, + CONFIG_FILE, +}; + +static int cmd__bundle_uri_parse(int argc, const char **argv, enum input_mode mode) { const char *key_value_usage[] = { "test-tool bundle-uri parse-key-values ", NULL }; + const char *config_usage[] = { + "test-tool bundle-uri parse-config ", + NULL + }; const char **usage = key_value_usage; struct option options[] = { OPT_END(), @@ -19,21 +28,35 @@ static int cmd__bundle_uri_parse(int argc, const char **argv) int err = 0; FILE *fp; - argc = parse_options(argc, argv, NULL, options, usage, 0); - if (argc != 1) - goto usage; + if (mode == CONFIG_FILE) + usage = config_usage; + + argc = parse_options(argc, argv, NULL, options, usage, + PARSE_OPT_STOP_AT_NON_OPTION); init_bundle_list(&list); - fp = fopen(argv[0], "r"); - if (!fp) - die("failed to open '%s'", argv[0]); - while (strbuf_getline(&sb, fp) != EOF) { - if (bundle_uri_parse_line(&list, sb.buf)) - err = error("bad line: '%s'", sb.buf); + switch (mode) { + case KEY_VALUE_PAIRS: + if (argc != 1) + goto usage; + fp = fopen(argv[0], "r"); + if (!fp) + die("failed to open '%s'", argv[0]); + while (strbuf_getline(&sb, fp) != EOF) { + if (bundle_uri_parse_line(&list, sb.buf)) + err = error("bad line: '%s'", sb.buf); + } + fclose(fp); + break; + + case CONFIG_FILE: + if (argc != 1) + goto usage; + err = bundle_uri_parse_config_format("", argv[0], &list); + break; } strbuf_release(&sb); - fclose(fp); print_bundle_list(stdout, &list); @@ -62,7 +85,9 @@ int cmd__bundle_uri(int argc, const char **argv) goto usage; if (!strcmp(argv[1], "parse-key-values")) - return cmd__bundle_uri_parse(argc - 1, argv + 1); + return cmd__bundle_uri_parse(argc - 1, argv + 1, KEY_VALUE_PAIRS); + if (!strcmp(argv[1], "parse-config")) + return cmd__bundle_uri_parse(argc - 1, argv + 1, CONFIG_FILE); error("there is no test-tool bundle-uri tool '%s'", argv[1]); usage: diff --git a/t/t5750-bundle-uri-parse.sh b/t/t5750-bundle-uri-parse.sh index fd142a66ad5..c2fe3f9c5a5 100755 --- a/t/t5750-bundle-uri-parse.sh +++ b/t/t5750-bundle-uri-parse.sh @@ -118,4 +118,54 @@ test_expect_success 'bundle_uri_parse_line() parsing edge cases: duplicate lines test_cmp_config_output expect actual ' +test_expect_success 'parse config format: just URIs' ' + cat >expect <<-\EOF && + [bundle] + version = 1 + mode = all + [bundle "one"] + uri = http://example.com/bundle.bdl + [bundle "two"] + uri = https://example.com/bundle.bdl + [bundle "three"] + uri = file:///usr/share/git/bundle.bdl + EOF + + test-tool bundle-uri parse-config expect >actual 2>err && + test_must_be_empty err && + test_cmp_config_output expect actual +' + +test_expect_success 'parse config format edge cases: empty key or value' ' + cat >in1 <<-\EOF && + = bogus-value + EOF + + cat >err1 <<-EOF && + error: bad config line 1 in file in1 + EOF + + cat >expect <<-\EOF && + [bundle] + version = 1 + mode = all + EOF + + test_must_fail test-tool bundle-uri parse-config in1 >actual 2>err && + test_cmp err1 err && + test_cmp_config_output expect actual && + + cat >in2 <<-\EOF && + bogus-key = + EOF + + cat >err2 <<-EOF && + error: bad config line 1 in file in2 + EOF + + test_must_fail test-tool bundle-uri parse-config in2 >actual 2>err && + test_cmp err2 err && + test_cmp_config_output expect actual +' + test_done From patchwork Wed Oct 12 12:52:34 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 13005051 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 168DEC43217 for ; Wed, 12 Oct 2022 12:53:17 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229780AbiJLMxP (ORCPT ); Wed, 12 Oct 2022 08:53:15 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38052 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229811AbiJLMxC (ORCPT ); Wed, 12 Oct 2022 08:53:02 -0400 Received: from mail-wr1-x434.google.com (mail-wr1-x434.google.com [IPv6:2a00:1450:4864:20::434]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7A31CC90E9 for ; Wed, 12 Oct 2022 05:52:53 -0700 (PDT) Received: by mail-wr1-x434.google.com with SMTP id j16so26090220wrh.5 for ; Wed, 12 Oct 2022 05:52:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:from:to:cc:subject:date :message-id:reply-to; bh=fMNeldL6yVM+YKrj7w8lZD7yCcA2s/qR6KN4/yWumMc=; b=QwZQTfszRDPUbIs7iIb1XmhJe5Qz5zcW6LY7FtmmLC80U2OZIjLTKh64Doju6hcj7d 5tRUDsi8LwSTzE1Ol8wQTAay3czxr6U+AyItr85Qz75WfMia05hnO9/I4ay4M1xyCNut 8Xzhl8C/RmhsLkVQpqE1lh2t6hRYHG97ndv0kZGzsYHufjZqgxs78pJrtqvC4vVSEtLs I1Vc50JI+s5EDkO01m21to0zcpviKSy/8EZXbvrksoMv9cqMdf1Zfupm4mwtnFAbVMUl nDwO8RVtD8ndThOld86XrmoJWId3XlnOW00llrBlQlMoImL+2DQs3TdN7BijdeMtbKj2 SwDA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=fMNeldL6yVM+YKrj7w8lZD7yCcA2s/qR6KN4/yWumMc=; b=MIwC+KXcMrz2yRsdXQaI69LJptn/i8AWfbHiO0efBAlq4oqMCq2c1YFndY4jKDiDer AlGNRJ4kPKKQzGMOWClYp28Z+ze7UUYycbZuOneXJ09jLlIWRIpkY96AptkTIhJJDvEE /XJ2YUymFmPlbdRVvirRkW1w20O9cpipSaQuEl6YKoQj8N/M52/QaJIsEfdtr3KxW4kl d8pJLNFNrBYNyz3g7WB+iJHmVEbj3hDyjvcM01eZMBdWc1O+sjzdsA9dzQ0BzAZnFiho 3F4BQ6VomkrJsQyTQhQWHx3N4HwSCYEWBncQOhK/x0O50P8isZiuUoICv9dnM3bejlaJ nPPg== X-Gm-Message-State: ACrzQf0lgU985zlqRPcPUe0i2KUHjg3UEM64exhjdjV/Flo7xrzyWT39 lrjFjVnA/+4bvpr0Vqxubpcil43D60M= X-Google-Smtp-Source: AMsMyM5xiLAcEMuMobC/wf4+eLHUR2VBLUU+lrQflUh1+whlMujcQoNOBZrMRxAPS6Hivc9kHxwHdw== X-Received: by 2002:adf:dd8d:0:b0:22e:4675:86c9 with SMTP id x13-20020adfdd8d000000b0022e467586c9mr17394136wrl.643.1665579170982; Wed, 12 Oct 2022 05:52:50 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id h11-20020a05600c314b00b003c6d896a17dsm599087wmo.32.2022.10.12.05.52.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 12 Oct 2022 05:52:50 -0700 (PDT) Message-Id: <7e6b32313b0f1922c0d0bfd104a288b8606306ec.1665579160.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Wed, 12 Oct 2022 12:52:34 +0000 Subject: [PATCH v5 07/12] bundle-uri: limit recursion depth for bundle lists Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, me@ttaylorr.com, newren@gmail.com, avarab@gmail.com, mjcheetham@outlook.com, steadmon@google.com, Glen Choo , Jonathan Tan , Teng Long , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee The next change will start allowing us to parse bundle lists that are downloaded from a provided bundle URI. Those lists might point to other lists, which could proceed to an arbitrary depth (and even create cycles). Restructure fetch_bundle_uri() to have an internal version that has a recursion depth. Compare that to a new max_bundle_uri_depth constant that is twice as high as we expect this depth to be for any legitimate use of bundle list linking. We can consider making max_bundle_uri_depth a configurable value if there is demonstrated value in the future. Signed-off-by: Derrick Stolee --- bundle-uri.c | 21 ++++++++++++++++++++- 1 file changed, 20 insertions(+), 1 deletion(-) diff --git a/bundle-uri.c b/bundle-uri.c index 3d44ec2b1e6..8a7c11c6393 100644 --- a/bundle-uri.c +++ b/bundle-uri.c @@ -334,11 +334,25 @@ static int unbundle_from_file(struct repository *r, const char *file) return result; } -int fetch_bundle_uri(struct repository *r, const char *uri) +/** + * This limits the recursion on fetch_bundle_uri_internal() when following + * bundle lists. + */ +static int max_bundle_uri_depth = 4; + +static int fetch_bundle_uri_internal(struct repository *r, + const char *uri, + int depth) { int result = 0; char *filename; + if (depth >= max_bundle_uri_depth) { + warning(_("exceeded bundle URI recursion limit (%d)"), + max_bundle_uri_depth); + return -1; + } + if (!(filename = find_temp_filename())) { result = -1; goto cleanup; @@ -366,6 +380,11 @@ cleanup: return result; } +int fetch_bundle_uri(struct repository *r, const char *uri) +{ + return fetch_bundle_uri_internal(r, uri, 0); +} + /** * General API for {transport,connect}.c etc. */ From patchwork Wed Oct 12 12:52:35 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 13005053 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A3219C4332F for ; Wed, 12 Oct 2022 12:53:23 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229867AbiJLMxU (ORCPT ); Wed, 12 Oct 2022 08:53:20 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38100 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229846AbiJLMxC (ORCPT ); Wed, 12 Oct 2022 08:53:02 -0400 Received: from mail-wr1-x42e.google.com (mail-wr1-x42e.google.com [IPv6:2a00:1450:4864:20::42e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 80717C90F1 for ; Wed, 12 Oct 2022 05:52:54 -0700 (PDT) Received: by mail-wr1-x42e.google.com with SMTP id w18so26098462wro.7 for ; Wed, 12 Oct 2022 05:52:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:from:to:cc:subject:date :message-id:reply-to; bh=vWpVJInZugH1z+IBM1n7b7iFlEZKKmBfgM3uDXDEkNI=; b=TrlwwyUCa2JLbP4+eZDfqRRw+KItBCX4KFgPW/QR1U226Qu4UK28dJ0orWFHYnOBXk M6WvqNld0SYJv6UR4Bgk2c//VHMxbjZWUc7taT81s9wbuL0ip0gO/4z2PlJr1h9P5KBB IKt0WNJBwGlEK5AASYghn0IvOv53igeWSen8fk/GKWwFLSKSqA1A0nIUKqcs8iBJxwZv deApwJ/kJFTXucQCRqEZFjsT5Ok2zeeL2u7gUyp51Lt/vQxunvI2EPx40LC4vXvx7ckJ 5nPTpffptWhHszLW0UKadxxEkeZvsCDRT/QuC5/3k8Dz6EN/5wW7Ti/nHpOi0Zx9VhlN IAgQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=vWpVJInZugH1z+IBM1n7b7iFlEZKKmBfgM3uDXDEkNI=; b=q4x1ZqsGKhBV1scogjGuWioI3hY4oXHeSb60wrjbQX6v6M3Z9vRF0U2Zda1qKFRGJq pB7IKUEr7g4bCPsoAiQJ3SAnBXMWws+vJxK9at5OH1G6xkP2CuP15gbURKlHwPFWrXLN pjM5j9wVcywWHMxB7p2bFpUUz7Guiuf4syR/DX6bw2XYZru2/gQcy5UKhyVrtAqPCT7P PcQiVU8VDpNbBr0rqprI8Z97hHNkU8NSV2wnxU9XzT9Jmn7cWaGgECjUlPvRG2m7Fw25 gXBAH/0IgVy+w9zMn2vcVlqFRQQjy5Ow3Fff1g50mzGtAMriKHVVSIpcPW4AUWWe8FnL ZMjg== X-Gm-Message-State: ACrzQf0SSabBL7dWQyTWojkklrk1JtdT3syP/9c6hiJd+WOBiqLMOKAS rgo3j1HzUkI97ZOCwo4lmCNsfwlBDqY= X-Google-Smtp-Source: AMsMyM5ww9cZ/jIwBWkFFZh/NyA2WUtfLmRUG14yDnFjppBXvezp77+bTIg9x+Fe79v3sh52HGJtyg== X-Received: by 2002:adf:ef05:0:b0:230:32dc:a7ce with SMTP id e5-20020adfef05000000b0023032dca7cemr10709926wro.319.1665579172236; Wed, 12 Oct 2022 05:52:52 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id 8-20020a05600c228800b003c6c76b43a1sm1711192wmf.13.2022.10.12.05.52.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 12 Oct 2022 05:52:51 -0700 (PDT) Message-Id: <8dc5a8e4e63dc98642176e5b78be739ef721d2d8.1665579160.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Wed, 12 Oct 2022 12:52:35 +0000 Subject: [PATCH v5 08/12] bundle: properly clear all revision flags Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, me@ttaylorr.com, newren@gmail.com, avarab@gmail.com, mjcheetham@outlook.com, steadmon@google.com, Glen Choo , Jonathan Tan , Teng Long , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee The verify_bundle() method checks two things for a bundle's prerequisites: 1. Are these objects in the object store? 2. Are these objects reachable from our references? In this second question, multiple uses of verify_bundle() in the same process can report an invalid bundle even though it is correct. The reason is due to not clearing all of the commit marks on the commits previously walked. The revision walk machinery was first introduced in-process by fb9a54150d3 (git-bundle: avoid fork() in verify_bundle(), 2007-02-22). This implementation used "-1" as the set of flags to clear. The next meaningful change came in 2b064697a5b (revision traversal: retire BOUNDARY_SHOW, 2007-03-05), which introduced the PREREQ_MARK flag instead of a flag normally controlled by the revision-walk machinery. In 86a0a408b90 (commit: factor out clear_commit_marks_for_object_array, 2011-10-01), the loop over the array of commits was replaced with a new clear_commit_marks_for_object_array(), but simultaneously the "-1" value was replaced with "ALL_REV_FLAGS", which stopped un-setting the PREREQ_MARK flag. This means that if multiple commits were marked by the PREREQ_MARK in a previous run of verify_bundle(), then this loop could terminate early due to 'i' going to zero: while (i && (commit = get_revision(&revs))) if (commit->object.flags & PREREQ_MARK) i--; The flag clearing work was changed again in 63647391e6c (bundle: avoid using the rev_info flag leak_pending, 2017-12-25), but that was only cosmetic and did not change the behavior. It may seem that it would be sufficient to add the PREREQ_MARK flag to the clear_commit_marks() call in its current location. However, we actually need to do it in the "cleanup:" step, since the first loop checking "Are these objects in the object store?" might add the PREREQ_MARK flag to some objects and then terminate without performing a walk due to one missing object. By clearing the flags in all cases, we avoid this issue when running verify_bundle() multiple times in the same process. Moving this loop to the cleanup step alone would cause a segfault when running 'git bundle verify' outside of a repository, but this is because of that error condition using "goto cleanup" when returning is perfectly safe. Nothing has been initialized at that point, so we can return immediately without causing any leaks. This behavior is verified carefully by a test that will be added soon when Git learns to download bundle lists in a 'git clone --bundle-uri' command. Signed-off-by: Derrick Stolee --- bundle.c | 23 ++++++++++------------- 1 file changed, 10 insertions(+), 13 deletions(-) diff --git a/bundle.c b/bundle.c index 0208e6d90d3..c277f3b9360 100644 --- a/bundle.c +++ b/bundle.c @@ -202,10 +202,8 @@ int verify_bundle(struct repository *r, int i, ret = 0, req_nr; const char *message = _("Repository lacks these prerequisite commits:"); - if (!r || !r->objects || !r->objects->odb) { - ret = error(_("need a repository to verify a bundle")); - goto cleanup; - } + if (!r || !r->objects || !r->objects->odb) + return error(_("need a repository to verify a bundle")); repo_init_revisions(r, &revs, NULL); for (i = 0; i < p->nr; i++) { @@ -250,15 +248,6 @@ int verify_bundle(struct repository *r, error("%s %s", oid_to_hex(oid), name); } - /* Clean up objects used, as they will be reused. */ - for (i = 0; i < p->nr; i++) { - struct string_list_item *e = p->items + i; - struct object_id *oid = e->util; - commit = lookup_commit_reference_gently(r, oid, 1); - if (commit) - clear_commit_marks(commit, ALL_REV_FLAGS); - } - if (verbose) { struct string_list *r; @@ -287,6 +276,14 @@ int verify_bundle(struct repository *r, list_objects_filter_spec(&header->filter)); } cleanup: + /* Clean up objects used, as they will be reused. */ + for (i = 0; i < p->nr; i++) { + struct string_list_item *e = p->items + i; + struct object_id *oid = e->util; + commit = lookup_commit_reference_gently(r, oid, 1); + if (commit) + clear_commit_marks(commit, ALL_REV_FLAGS | PREREQ_MARK); + } release_revisions(&revs); return ret; } From patchwork Wed Oct 12 12:52:36 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 13005054 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A9891C433FE for ; Wed, 12 Oct 2022 12:53:26 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229799AbiJLMxY (ORCPT ); Wed, 12 Oct 2022 08:53:24 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38106 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229868AbiJLMxC (ORCPT ); Wed, 12 Oct 2022 08:53:02 -0400 Received: from mail-wr1-x432.google.com (mail-wr1-x432.google.com [IPv6:2a00:1450:4864:20::432]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 37B6FC90D1 for ; Wed, 12 Oct 2022 05:52:56 -0700 (PDT) Received: by mail-wr1-x432.google.com with SMTP id r13so26069443wrj.11 for ; Wed, 12 Oct 2022 05:52:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:from:to:cc:subject:date :message-id:reply-to; bh=vS6iU4oT3LmEfgLgwhFPp66xE2BF3msulqVY5UlQxL0=; b=YyB1pl/PASGnW3+g6URgfQBj+yzCbS+STyOOvt/maWhkqILK6UA0uelrbwopXcm4kz lz1WDiO0/uTLgWAtnsISF6XfR/H6ak8EO5FtlCkpqRKmC2Qe6oMI3HKbFd0SVpnFfwih r41d4b1Q3L0azGx2ul/K80j9+RC5+vufqCin8ZOgCcY5cFjynVwNb2XpVYO4heGbrA6j JeWFc0VtsIO0cpaISwymUIU8s3nuyqSJObVcPA99assaYtx1pL+JXxjs9AJClmr5rE9V 2cw/dZolLBg/Bv2YN71/0k389MkPyL0cvzZNmegY5wsHUmqUzPYJFOPi6fis5il5tMIh AHig== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=vS6iU4oT3LmEfgLgwhFPp66xE2BF3msulqVY5UlQxL0=; b=0Ka2/2mfQXFX0Q8IfQOjIe3y9QqDCLzPpvfLkPO2wp8S2aKVk4W0JviWLH+9AkwHix LW1qyMNN0gO/iwW11ZsfJXxV2bmJQeWXunjfKjrwGDZ4gEcAWnATBD9P/j6JFqY133g9 +l3hvJYsSqW5kBrzT3ItO5nXXtNTLpPMv0JSyAa2KISEACRHgPgd28DvnZZ9F5YRObXG Cs8okGQiGFjnKjKnnzXsa+/nX/kBW0R6T6fzuNr1VQoeCljWT68HYfmiZ8OZK/co8wc2 mGY6kZf1TAVSRSKEDejsdmRKCkRR/RcRv+Utm1R5Msl5DKfTWYR01+SYPX5QMZ3gecr2 7+Gg== X-Gm-Message-State: ACrzQf0jq65vc8RqQTfTsKucP59ngIQNcnTRLRAPU9ePXVY0n+EO4bKx V+QDkoeDyaeep4GodDJq3FnxsrrlCTw= X-Google-Smtp-Source: AMsMyM6z8Y8IgGQ3iWJCSfLahDMQNRF1ltTpblNz3zHGEk5INGtATTLeDovILhuO/hyn1Gjx8mtjbA== X-Received: by 2002:a5d:64c2:0:b0:22e:41b0:42ca with SMTP id f2-20020a5d64c2000000b0022e41b042camr18463707wri.411.1665579173436; Wed, 12 Oct 2022 05:52:53 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id h4-20020a05600c350400b003a83ca67f73sm1933452wmq.3.2022.10.12.05.52.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 12 Oct 2022 05:52:52 -0700 (PDT) Message-Id: <51e9b8474fb040c4277b96d4b13a8da58bdb3682.1665579160.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Wed, 12 Oct 2022 12:52:36 +0000 Subject: [PATCH v5 09/12] bundle-uri: fetch a list of bundles Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, me@ttaylorr.com, newren@gmail.com, avarab@gmail.com, mjcheetham@outlook.com, steadmon@google.com, Glen Choo , Jonathan Tan , Teng Long , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee When the content at a given bundle URI is not understood as a bundle (based on inspecting the initial content), then Git currently gives up and ignores that content. Independent bundle providers may want to split up the bundle content into multiple bundles, but still make them available from a single URI. Teach Git to attempt parsing the bundle URI content as a Git config file providing the key=value pairs for a bundle list. Git then looks at the mode of the list to see if ANY single bundle is sufficient or if ALL bundles are required. The content at the selected URIs are downloaded and the content is inspected again, creating a recursive process. To guard the recursion against malformed or malicious content, limit the recursion depth to a reasonable four for now. This can be converted to a configured value in the future if necessary. The value of four is twice as high as expected to be useful (a bundle list is unlikely to point to more bundle lists). To test this scenario, create an interesting bundle topology where three incremental bundles are built on top of a single full bundle. By using a merge commit, the two middle bundles are "independent" in that they do not require each other in order to unbundle themselves. They each only need the base bundle. The bundle containing the merge commit requires both of the middle bundles, though. This leads to some interesting decisions when unbundling, especially when we later implement heuristics that promote downloading bundles until the prerequisite commits are satisfied. Signed-off-by: Derrick Stolee --- bundle-uri.c | 203 ++++++++++++++++++++++++++--- bundle-uri.h | 13 ++ t/t5558-clone-bundle-uri.sh | 248 ++++++++++++++++++++++++++++++++++++ 3 files changed, 448 insertions(+), 16 deletions(-) diff --git a/bundle-uri.c b/bundle-uri.c index 8a7c11c6393..70bfd2defee 100644 --- a/bundle-uri.c +++ b/bundle-uri.c @@ -37,6 +37,8 @@ static int clear_remote_bundle_info(struct remote_bundle_info *bundle, { FREE_AND_NULL(bundle->id); FREE_AND_NULL(bundle->uri); + FREE_AND_NULL(bundle->file); + bundle->unbundled = 0; return 0; } @@ -334,18 +336,117 @@ static int unbundle_from_file(struct repository *r, const char *file) return result; } +struct bundle_list_context { + struct repository *r; + struct bundle_list *list; + enum bundle_list_mode mode; + int count; + int depth; +}; + +/* + * This early definition is necessary because we use indirect recursion: + * + * While iterating through a bundle list that was downloaded as part + * of fetch_bundle_uri_internal(), iterator methods eventually call it + * again, but with depth + 1. + */ +static int fetch_bundle_uri_internal(struct repository *r, + struct remote_bundle_info *bundle, + int depth, + struct bundle_list *list); + +static int download_bundle_to_file(struct remote_bundle_info *bundle, void *data) +{ + int res; + struct bundle_list_context *ctx = data; + + if (ctx->mode == BUNDLE_MODE_ANY && ctx->count) + return 0; + + res = fetch_bundle_uri_internal(ctx->r, bundle, ctx->depth + 1, ctx->list); + + /* + * Only increment count if the download succeeded. If our mode is + * BUNDLE_MODE_ANY, then we will want to try other URIs in the + * list in case they work instead. + */ + if (!res) + ctx->count++; + + /* + * To be opportunistic as possible, we continue iterating and + * download as many bundles as we can, so we can apply the ones + * that work, even in BUNDLE_MODE_ALL mode. + */ + return 0; +} + +static int download_bundle_list(struct repository *r, + struct bundle_list *local_list, + struct bundle_list *global_list, + int depth) +{ + struct bundle_list_context ctx = { + .r = r, + .list = global_list, + .depth = depth + 1, + .mode = local_list->mode, + }; + + return for_all_bundles_in_list(local_list, download_bundle_to_file, &ctx); +} + +static int fetch_bundle_list_in_config_format(struct repository *r, + struct bundle_list *global_list, + struct remote_bundle_info *bundle, + int depth) +{ + int result; + struct bundle_list list_from_bundle; + + init_bundle_list(&list_from_bundle); + + if ((result = bundle_uri_parse_config_format(bundle->uri, + bundle->file, + &list_from_bundle))) + goto cleanup; + + if (list_from_bundle.mode == BUNDLE_MODE_NONE) { + warning(_("unrecognized bundle mode from URI '%s'"), + bundle->uri); + result = -1; + goto cleanup; + } + + if ((result = download_bundle_list(r, &list_from_bundle, + global_list, depth))) + goto cleanup; + +cleanup: + clear_bundle_list(&list_from_bundle); + return result; +} + /** * This limits the recursion on fetch_bundle_uri_internal() when following * bundle lists. */ static int max_bundle_uri_depth = 4; +/** + * Recursively download all bundles advertised at the given URI + * to files. If the file is a bundle, then add it to the given + * 'list'. Otherwise, expect a bundle list and recurse on the + * URIs in that list according to the list mode (ANY or ALL). + */ static int fetch_bundle_uri_internal(struct repository *r, - const char *uri, - int depth) + struct remote_bundle_info *bundle, + int depth, + struct bundle_list *list) { int result = 0; - char *filename; + struct remote_bundle_info *bcopy; if (depth >= max_bundle_uri_depth) { warning(_("exceeded bundle URI recursion limit (%d)"), @@ -353,36 +454,106 @@ static int fetch_bundle_uri_internal(struct repository *r, return -1; } - if (!(filename = find_temp_filename())) { + if (!bundle->file && + !(bundle->file = find_temp_filename())) { result = -1; goto cleanup; } - if ((result = copy_uri_to_file(filename, uri))) { - warning(_("failed to download bundle from URI '%s'"), uri); + if ((result = copy_uri_to_file(bundle->file, bundle->uri))) { + warning(_("failed to download bundle from URI '%s'"), bundle->uri); goto cleanup; } - if ((result = !is_bundle(filename, 0))) { - warning(_("file at URI '%s' is not a bundle"), uri); + if ((result = !is_bundle(bundle->file, 1))) { + result = fetch_bundle_list_in_config_format( + r, list, bundle, depth); + if (result) + warning(_("file at URI '%s' is not a bundle or bundle list"), + bundle->uri); goto cleanup; } - if ((result = unbundle_from_file(r, filename))) { - warning(_("failed to unbundle bundle from URI '%s'"), uri); - goto cleanup; - } + /* Copy the bundle and insert it into the global list. */ + CALLOC_ARRAY(bcopy, 1); + bcopy->id = xstrdup(bundle->id); + bcopy->file = xstrdup(bundle->file); + hashmap_entry_init(&bcopy->ent, strhash(bcopy->id)); + hashmap_add(&list->bundles, &bcopy->ent); cleanup: - if (filename) - unlink(filename); - free(filename); + if (result && bundle->file) + unlink(bundle->file); return result; } +/** + * This loop iterator breaks the loop with nonzero return code on the + * first successful unbundling of a bundle. + */ +static int attempt_unbundle(struct remote_bundle_info *info, void *data) +{ + struct repository *r = data; + + if (!info->file || info->unbundled) + return 0; + + if (!unbundle_from_file(r, info->file)) { + info->unbundled = 1; + return 1; + } + + return 0; +} + +static int unbundle_all_bundles(struct repository *r, + struct bundle_list *list) +{ + /* + * Iterate through all bundles looking for ones that can + * successfully unbundle. If any succeed, then perhaps another + * will succeed in the next attempt. + * + * Keep in mind that a non-zero result for the loop here means + * the loop terminated early on a successful unbundling, which + * signals that we can try again. + */ + while (for_all_bundles_in_list(list, attempt_unbundle, r)) ; + + return 0; +} + +static int unlink_bundle(struct remote_bundle_info *info, void *data) +{ + if (info->file) + unlink_or_warn(info->file); + return 0; +} + int fetch_bundle_uri(struct repository *r, const char *uri) { - return fetch_bundle_uri_internal(r, uri, 0); + int result; + struct bundle_list list; + struct remote_bundle_info bundle = { + .uri = xstrdup(uri), + .id = xstrdup(""), + }; + + init_bundle_list(&list); + + /* If a bundle is added to this global list, then it is required. */ + list.mode = BUNDLE_MODE_ALL; + + if ((result = fetch_bundle_uri_internal(r, &bundle, 0, &list))) + goto cleanup; + + result = unbundle_all_bundles(r, &list); + +cleanup: + for_all_bundles_in_list(&list, unlink_bundle, NULL); + clear_bundle_list(&list); + clear_remote_bundle_info(&bundle, NULL); + return result; } /** diff --git a/bundle-uri.h b/bundle-uri.h index bc13d4c9929..4dbc269823c 100644 --- a/bundle-uri.h +++ b/bundle-uri.h @@ -28,6 +28,19 @@ struct remote_bundle_info { * if there was no table of contents. */ char *uri; + + /** + * If the bundle has been downloaded, then 'file' is a + * filename storing its contents. Otherwise, 'file' is + * NULL. + */ + char *file; + + /** + * If the bundle has been unbundled successfully, then + * this boolean is true. + */ + unsigned unbundled:1; }; #define REMOTE_BUNDLE_INFO_INIT { 0 } diff --git a/t/t5558-clone-bundle-uri.sh b/t/t5558-clone-bundle-uri.sh index ad666a2d28a..a86dc04f528 100755 --- a/t/t5558-clone-bundle-uri.sh +++ b/t/t5558-clone-bundle-uri.sh @@ -41,6 +41,195 @@ test_expect_success 'clone with file:// bundle' ' test_cmp expect actual ' +# To get interesting tests for bundle lists, we need to construct a +# somewhat-interesting commit history. +# +# ---------------- bundle-4 +# +# 4 +# / \ +# ----|---|------- bundle-3 +# | | +# | 3 +# | | +# ----|---|------- bundle-2 +# | | +# 2 | +# | | +# ----|---|------- bundle-1 +# \ / +# 1 +# | +# (previous commits) +test_expect_success 'construct incremental bundle list' ' + ( + cd clone-from && + git checkout -b base && + test_commit 1 && + git checkout -b left && + test_commit 2 && + git checkout -b right base && + test_commit 3 && + git checkout -b merge left && + git merge right -m "4" && + + git bundle create bundle-1.bundle base && + git bundle create bundle-2.bundle base..left && + git bundle create bundle-3.bundle base..right && + git bundle create bundle-4.bundle merge --not left right + ) +' + +test_expect_success 'clone bundle list (file, no heuristic)' ' + cat >bundle-list <<-EOF && + [bundle] + version = 1 + mode = all + + [bundle "bundle-1"] + uri = file://$(pwd)/clone-from/bundle-1.bundle + + [bundle "bundle-2"] + uri = file://$(pwd)/clone-from/bundle-2.bundle + + [bundle "bundle-3"] + uri = file://$(pwd)/clone-from/bundle-3.bundle + + [bundle "bundle-4"] + uri = file://$(pwd)/clone-from/bundle-4.bundle + EOF + + git clone --bundle-uri="file://$(pwd)/bundle-list" clone-from clone-list-file && + git -C clone-from for-each-ref --format="%(objectname)" >oids && + git -C clone-list-file cat-file --batch-check refs && + grep "refs/bundles/" refs >actual && + cat >expect <<-\EOF && + refs/bundles/base + refs/bundles/left + refs/bundles/merge + refs/bundles/right + EOF + test_cmp expect actual +' + +test_expect_success 'clone bundle list (file, all mode, some failures)' ' + cat >bundle-list <<-EOF && + [bundle] + version = 1 + mode = all + + # Does not exist. Should be skipped. + [bundle "bundle-0"] + uri = file://$(pwd)/clone-from/bundle-0.bundle + + [bundle "bundle-1"] + uri = file://$(pwd)/clone-from/bundle-1.bundle + + [bundle "bundle-2"] + uri = file://$(pwd)/clone-from/bundle-2.bundle + + # No bundle-3 means bundle-4 will not apply. + + [bundle "bundle-4"] + uri = file://$(pwd)/clone-from/bundle-4.bundle + + # Does not exist. Should be skipped. + [bundle "bundle-5"] + uri = file://$(pwd)/clone-from/bundle-5.bundle + EOF + + GIT_TRACE2_PERF=1 \ + git clone --bundle-uri="file://$(pwd)/bundle-list" clone-from clone-all-some && + git -C clone-from for-each-ref --format="%(objectname)" >oids && + git -C clone-all-some cat-file --batch-check refs && + grep "refs/bundles/" refs >actual && + cat >expect <<-\EOF && + refs/bundles/base + refs/bundles/left + EOF + test_cmp expect actual +' + +test_expect_success 'clone bundle list (file, all mode, all failures)' ' + cat >bundle-list <<-EOF && + [bundle] + version = 1 + mode = all + + # Does not exist. Should be skipped. + [bundle "bundle-0"] + uri = file://$(pwd)/clone-from/bundle-0.bundle + + # Does not exist. Should be skipped. + [bundle "bundle-5"] + uri = file://$(pwd)/clone-from/bundle-5.bundle + EOF + + git clone --bundle-uri="file://$(pwd)/bundle-list" clone-from clone-all-fail && + git -C clone-from for-each-ref --format="%(objectname)" >oids && + git -C clone-all-fail cat-file --batch-check refs && + ! grep "refs/bundles/" refs +' + +test_expect_success 'clone bundle list (file, any mode)' ' + cat >bundle-list <<-EOF && + [bundle] + version = 1 + mode = any + + # Does not exist. Should be skipped. + [bundle "bundle-0"] + uri = file://$(pwd)/clone-from/bundle-0.bundle + + [bundle "bundle-1"] + uri = file://$(pwd)/clone-from/bundle-1.bundle + + # Does not exist. Should be skipped. + [bundle "bundle-5"] + uri = file://$(pwd)/clone-from/bundle-5.bundle + EOF + + git clone --bundle-uri="file://$(pwd)/bundle-list" clone-from clone-any-file && + git -C clone-from for-each-ref --format="%(objectname)" >oids && + git -C clone-any-file cat-file --batch-check refs && + grep "refs/bundles/" refs >actual && + cat >expect <<-\EOF && + refs/bundles/base + EOF + test_cmp expect actual +' + +test_expect_success 'clone bundle list (file, any mode, all failures)' ' + cat >bundle-list <<-EOF && + [bundle] + version = 1 + mode = any + + # Does not exist. Should be skipped. + [bundle "bundle-0"] + uri = $HTTPD_URL/bundle-0.bundle + + # Does not exist. Should be skipped. + [bundle "bundle-5"] + uri = $HTTPD_URL/bundle-5.bundle + EOF + + git clone --bundle-uri="file://$(pwd)/bundle-list" clone-from clone-any-fail && + git -C clone-from for-each-ref --format="%(objectname)" >oids && + git -C clone-any-fail cat-file --batch-check refs && + ! grep "refs/bundles/" refs +' + ######################################################################### # HTTP tests begin here @@ -75,6 +264,65 @@ test_expect_success 'clone HTTP bundle' ' test_config -C clone-http log.excludedecoration refs/bundle/ ' +test_expect_success 'clone bundle list (HTTP, no heuristic)' ' + cp clone-from/bundle-*.bundle "$HTTPD_DOCUMENT_ROOT_PATH/" && + cat >"$HTTPD_DOCUMENT_ROOT_PATH/bundle-list" <<-EOF && + [bundle] + version = 1 + mode = all + + [bundle "bundle-1"] + uri = $HTTPD_URL/bundle-1.bundle + + [bundle "bundle-2"] + uri = $HTTPD_URL/bundle-2.bundle + + [bundle "bundle-3"] + uri = $HTTPD_URL/bundle-3.bundle + + [bundle "bundle-4"] + uri = $HTTPD_URL/bundle-4.bundle + EOF + + git clone --bundle-uri="$HTTPD_URL/bundle-list" clone-from clone-list-http && + git -C clone-from for-each-ref --format="%(objectname)" >oids && + git -C clone-list-http cat-file --batch-check "$HTTPD_DOCUMENT_ROOT_PATH/bundle-list" <<-EOF && + [bundle] + version = 1 + mode = any + + # Does not exist. Should be skipped. + [bundle "bundle-0"] + uri = $HTTPD_URL/bundle-0.bundle + + [bundle "bundle-1"] + uri = $HTTPD_URL/bundle-1.bundle + + # Does not exist. Should be skipped. + [bundle "bundle-5"] + uri = $HTTPD_URL/bundle-5.bundle + EOF + + git clone --bundle-uri="$HTTPD_URL/bundle-list" clone-from clone-any-http && + git -C clone-from for-each-ref --format="%(objectname)" >oids && + git -C clone-any-http cat-file --batch-check refs && + grep "refs/bundles/" refs >actual && + cat >expect <<-\EOF && + refs/bundles/base + refs/bundles/left + refs/bundles/merge + refs/bundles/right + EOF + test_cmp expect actual +' + # Do not add tests here unless they use the HTTP server, as they will # not run unless the HTTP dependencies exist. From patchwork Wed Oct 12 12:52:37 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 13005052 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 36D96C4332F for ; Wed, 12 Oct 2022 12:53:19 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229471AbiJLMxS (ORCPT ); Wed, 12 Oct 2022 08:53:18 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38096 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229821AbiJLMxC (ORCPT ); Wed, 12 Oct 2022 08:53:02 -0400 Received: from mail-wm1-x32e.google.com (mail-wm1-x32e.google.com [IPv6:2a00:1450:4864:20::32e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8A21EC90F7 for ; Wed, 12 Oct 2022 05:52:55 -0700 (PDT) Received: by mail-wm1-x32e.google.com with SMTP id e18so10362449wmq.3 for ; Wed, 12 Oct 2022 05:52:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:from:to:cc:subject:date :message-id:reply-to; bh=Ye6lJQwKN7ZqHsRHUGxLWmS3tm9YC/ga6sYyrz6AN5c=; b=R2MysE85M/QhyQNcH58w1bHmEyvUxSr7VxOBCWAfuQ2PsIL28bjzQXYwM4APSwf0WW 3l3w0+2DBrGJm0h6YKdJn9CkJ/saJxlbnyODqX9n9lIqTmuciGcLYlBSK3A1jgKdFx3v KB+HI0Zu/rbRaQn/uMt5FA5ghaayXUWFSUbTfBEslSF+P9LITgppfOf3IBGhDHF7sRt2 olQWpd6b3P3S/VG9yObi2hQlq9gIFaCRl2UMM1ZolIvCWBfZT6PYGd//8WlT6B2ZmTRq 32HNYcjcEqLg51IwzAyTgZqC+uIqO0EyI3p9wIWX0qA8Hgai6l4W95fmiB5X/wdGK66P xPfQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Ye6lJQwKN7ZqHsRHUGxLWmS3tm9YC/ga6sYyrz6AN5c=; b=bYOmwB+jlaKa8OevPFqz71Qo0Y1Srj2xY+beZDVbBzvEKcBhNO+K/cxD/4sO8r/Xnd n9hxTO5yR01V18AHJVl5FXvTZk0hlF0CDz9F/kPdYRlmnWqRQPedv8xCAo3nEdkCC2qB oWBiuJePXbAEZs8i6J8EJ84SUXFjUrHV4dvb5+yFT+DDAYMr9ZsKZllpIHKmfxat0IRX 9nF6I3vUUwb1zEmJ44UVdX6sb8fxzDR2FDZxjYlofPEa8Ag1NnMs9MkwSvude6H7XlmX dRsEY4hW/zCyc9YvtosJ8ZdKn59vtvAhALOb5ofDcu2EdNEQd+LNDqtUWLJOOzgHURcq H6CQ== X-Gm-Message-State: ACrzQf1nGuxSkCvekxfFA2UDwluxsaLYadrePcWSv+0mJLJt0KOxIi7u AdJgSdhSpCwDK7RJP92vBtnBPrryVIg= X-Google-Smtp-Source: AMsMyM4/PLzUgEeXarwG0wv0fEkTe67qlna7cJzD850syYUNGM0RLasDNZIokV/peXTxsEf8AE5xqQ== X-Received: by 2002:a05:600c:3554:b0:3c3:fdfd:f393 with SMTP id i20-20020a05600c355400b003c3fdfdf393mr2750456wmq.188.1665579174564; Wed, 12 Oct 2022 05:52:54 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id bv10-20020a0560001f0a00b00228fa832b7asm13637688wrb.52.2022.10.12.05.52.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 12 Oct 2022 05:52:53 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Wed, 12 Oct 2022 12:52:37 +0000 Subject: [PATCH v5 10/12] bundle: add flags to verify_bundle() Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, me@ttaylorr.com, newren@gmail.com, avarab@gmail.com, mjcheetham@outlook.com, steadmon@google.com, Glen Choo , Jonathan Tan , Teng Long , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee The verify_bundle() method has a 'verbose' option, but we will want to extend this method to have more granular control over its output. First, replace this 'verbose' option with a new 'flags' option with a single possible value: VERIFY_BUNDLE_VERBOSE. Signed-off-by: Derrick Stolee --- builtin/bundle.c | 5 +++-- bundle-uri.c | 7 ++++++- bundle.c | 9 +++++---- bundle.h | 14 ++++++++++++-- transport.c | 2 +- 5 files changed, 27 insertions(+), 10 deletions(-) diff --git a/builtin/bundle.c b/builtin/bundle.c index 2adad545a2e..7d983a238f0 100644 --- a/builtin/bundle.c +++ b/builtin/bundle.c @@ -119,7 +119,8 @@ static int cmd_bundle_verify(int argc, const char **argv, const char *prefix) { goto cleanup; } close(bundle_fd); - if (verify_bundle(the_repository, &header, !quiet)) { + if (verify_bundle(the_repository, &header, + quiet ? 0 : VERIFY_BUNDLE_VERBOSE)) { ret = 1; goto cleanup; } @@ -185,7 +186,7 @@ static int cmd_bundle_unbundle(int argc, const char **argv, const char *prefix) strvec_pushl(&extra_index_pack_args, "-v", "--progress-title", _("Unbundling objects"), NULL); ret = !!unbundle(the_repository, &header, bundle_fd, - &extra_index_pack_args) || + &extra_index_pack_args, 0) || list_bundle_refs(&header, argc, argv); bundle_header_release(&header); cleanup: diff --git a/bundle-uri.c b/bundle-uri.c index 70bfd2defee..d9060be707e 100644 --- a/bundle-uri.c +++ b/bundle-uri.c @@ -303,7 +303,12 @@ static int unbundle_from_file(struct repository *r, const char *file) if ((bundle_fd = read_bundle_header(file, &header)) < 0) return 1; - if ((result = unbundle(r, &header, bundle_fd, NULL))) + /* + * Skip the reachability walk here, since we will be adding + * a reachable ref pointing to the new tips, which will reach + * the prerequisite commits. + */ + if ((result = unbundle(r, &header, bundle_fd, NULL, 0))) return 1; /* diff --git a/bundle.c b/bundle.c index c277f3b9360..1f6a7f782e1 100644 --- a/bundle.c +++ b/bundle.c @@ -189,7 +189,7 @@ static int list_refs(struct string_list *r, int argc, const char **argv) int verify_bundle(struct repository *r, struct bundle_header *header, - int verbose) + enum verify_bundle_flags flags) { /* * Do fast check, then if any prereqs are missing then go line by line @@ -248,7 +248,7 @@ int verify_bundle(struct repository *r, error("%s %s", oid_to_hex(oid), name); } - if (verbose) { + if (flags & VERIFY_BUNDLE_VERBOSE) { struct string_list *r; r = &header->references; @@ -617,7 +617,8 @@ err: } int unbundle(struct repository *r, struct bundle_header *header, - int bundle_fd, struct strvec *extra_index_pack_args) + int bundle_fd, struct strvec *extra_index_pack_args, + enum verify_bundle_flags flags) { struct child_process ip = CHILD_PROCESS_INIT; strvec_pushl(&ip.args, "index-pack", "--fix-thin", "--stdin", NULL); @@ -631,7 +632,7 @@ int unbundle(struct repository *r, struct bundle_header *header, strvec_clear(extra_index_pack_args); } - if (verify_bundle(r, header, 0)) + if (verify_bundle(r, header, flags)) return -1; ip.in = bundle_fd; ip.no_stdout = 1; diff --git a/bundle.h b/bundle.h index 0c052f54964..6652e819981 100644 --- a/bundle.h +++ b/bundle.h @@ -29,7 +29,13 @@ int read_bundle_header_fd(int fd, struct bundle_header *header, int create_bundle(struct repository *r, const char *path, int argc, const char **argv, struct strvec *pack_options, int version); -int verify_bundle(struct repository *r, struct bundle_header *header, int verbose); + +enum verify_bundle_flags { + VERIFY_BUNDLE_VERBOSE = (1 << 0), +}; + +int verify_bundle(struct repository *r, struct bundle_header *header, + enum verify_bundle_flags flags); /** * Unbundle after reading the header with read_bundle_header(). @@ -40,9 +46,13 @@ int verify_bundle(struct repository *r, struct bundle_header *header, int verbos * Provide "extra_index_pack_args" to pass any extra arguments * (e.g. "-v" for verbose/progress), NULL otherwise. The provided * "extra_index_pack_args" (if any) will be strvec_clear()'d for you. + * + * Before unbundling, this method will call verify_bundle() with the + * given 'flags'. */ int unbundle(struct repository *r, struct bundle_header *header, - int bundle_fd, struct strvec *extra_index_pack_args); + int bundle_fd, struct strvec *extra_index_pack_args, + enum verify_bundle_flags flags); int list_bundle_refs(struct bundle_header *header, int argc, const char **argv); diff --git a/transport.c b/transport.c index 52db7a3cb09..c5d3042731a 100644 --- a/transport.c +++ b/transport.c @@ -178,7 +178,7 @@ static int fetch_refs_from_bundle(struct transport *transport, if (!data->get_refs_from_bundle_called) get_refs_from_bundle_inner(transport); ret = unbundle(the_repository, &data->header, data->fd, - &extra_index_pack_args); + &extra_index_pack_args, 0); transport->hash_algo = data->header.hash_algo; return ret; } From patchwork Wed Oct 12 12:52:38 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 13005056 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 539E8C4332F for ; Wed, 12 Oct 2022 12:53:39 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229806AbiJLMxb (ORCPT ); Wed, 12 Oct 2022 08:53:31 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38050 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229754AbiJLMxM (ORCPT ); Wed, 12 Oct 2022 08:53:12 -0400 Received: from mail-wm1-x32a.google.com (mail-wm1-x32a.google.com [IPv6:2a00:1450:4864:20::32a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 87C28C823F for ; Wed, 12 Oct 2022 05:52:57 -0700 (PDT) Received: by mail-wm1-x32a.google.com with SMTP id bg9-20020a05600c3c8900b003bf249616b0so1150860wmb.3 for ; Wed, 12 Oct 2022 05:52:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:from:to:cc:subject:date :message-id:reply-to; bh=GP2j5/Hfsewkj2oC+Yr4l+5lEuk6HhreKDodb/i3CSg=; b=A5A6xZv0ermae83JAo4JUUy0/p8V2qXlXOCHMUT7SGffTj0iurd19ajaPDvaYG4Q9o 0ahPmywvUyWi5zQT9Uz4URHuSlDNHi+C6qWJmM0mY8m+TVxI57XOnjVk8siB+VBZQ4zW iZ0EnRDtD8cpNs2Q2YYZfHB3QRlKfGT3Qvn5C/zUrSqoZHaAH6Vf8aw3gg9loo03sGmO MUmf0jsej9vXtiO48KhYV8UZ4o3N6QR+fU1q91/jlsssQKdiH2Ib4MSjAIBF48lgmRSS 8Vu5gq0dUWfcAr79xn+7gguVz8NERrhx6LDcJsAs/U4MJwJaUFQA7kniCRS7qt/29Dex 2lww== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=GP2j5/Hfsewkj2oC+Yr4l+5lEuk6HhreKDodb/i3CSg=; b=QdyYZHFA94DfNJaBEfOE341aJMNsoQGsHP/it9WhzzuwT1itHYGStgGxSqNodut98h PaXWIpYYaE0jas3KlRBIz9F/n1yNSiiL3YFJ5Jwt9P+urh9N8fz40PLF0D8TqmaQPx3M y/i3PhpF081rKJts+PnDeCKrwrUxbfOwS4KakFFAEFW3LYu3MlTcIEWfmHAz+7RTFyjU FeYGBwMxpCVmHiiyRQGMNY9vrvrgg2Xb9wutPUnzAzrZL4iaVXiJ+cqbVK3ToAn93L1Z a1QUWAXaW+cJKjzd498dpmB+rOKKWdNwyoC8zWPH0xX0wUXM+mMym9/I3cjcANeWyGjF tLyw== X-Gm-Message-State: ACrzQf2wKoMgOb8zmd2/RWBlAwslDvkfgzTDKl8FcL1cvbVMi+Ri7+0C RVjJlMHQd2wlTa4Py7gZ6uI6i/B6vL4= X-Google-Smtp-Source: AMsMyM5P5UX7+d7UXi9Gskbv4CQOiVGwau4+saJXHmBo6CXLpIDwYf953ttWDWTd36fO9Qp+zVlq7A== X-Received: by 2002:a05:600c:214f:b0:3c6:ce02:8a68 with SMTP id v15-20020a05600c214f00b003c6ce028a68mr2739907wml.187.1665579175807; Wed, 12 Oct 2022 05:52:55 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id p22-20020a05600c065600b003c6c2ff7f25sm1641298wmm.15.2022.10.12.05.52.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 12 Oct 2022 05:52:55 -0700 (PDT) Message-Id: <2e0bfa834f14e1f4f66adf36d4326e5f1c2c9c20.1665579160.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Wed, 12 Oct 2022 12:52:38 +0000 Subject: [PATCH v5 11/12] bundle-uri: quiet failed unbundlings Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, me@ttaylorr.com, newren@gmail.com, avarab@gmail.com, mjcheetham@outlook.com, steadmon@google.com, Glen Choo , Jonathan Tan , Teng Long , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee When downloading a list of bundles in "all" mode, Git has no understanding of the dependencies between the bundles. Git attempts to unbundle the bundles in some order, but some may not pass the verify_bundle() step because of missing prerequisites. This is passed as error messages to the user, even when they eventually succeed in later attempts after their dependent bundles are unbundled. Add a new VERIFY_BUNDLE_QUIET flag to verify_bundle() that avoids the error messages from the missing prerequisite commits. The method still returns the number of missing prerequisit commits, allowing callers to unbundle() to notice that the bundle failed to apply. Use this flag in bundle-uri.c and test that the messages go away for 'git clone --bundle-uri' commands. Signed-off-by: Derrick Stolee --- builtin/bundle.c | 2 +- bundle-uri.c | 3 ++- bundle.c | 10 ++++++++-- bundle.h | 1 + t/t5558-clone-bundle-uri.sh | 25 ++++++++++++++++++++----- 5 files changed, 32 insertions(+), 9 deletions(-) diff --git a/builtin/bundle.c b/builtin/bundle.c index 7d983a238f0..fd4586b09e0 100644 --- a/builtin/bundle.c +++ b/builtin/bundle.c @@ -120,7 +120,7 @@ static int cmd_bundle_verify(int argc, const char **argv, const char *prefix) { } close(bundle_fd); if (verify_bundle(the_repository, &header, - quiet ? 0 : VERIFY_BUNDLE_VERBOSE)) { + quiet ? VERIFY_BUNDLE_QUIET : VERIFY_BUNDLE_VERBOSE)) { ret = 1; goto cleanup; } diff --git a/bundle-uri.c b/bundle-uri.c index d9060be707e..d872acf5ab0 100644 --- a/bundle-uri.c +++ b/bundle-uri.c @@ -308,7 +308,8 @@ static int unbundle_from_file(struct repository *r, const char *file) * a reachable ref pointing to the new tips, which will reach * the prerequisite commits. */ - if ((result = unbundle(r, &header, bundle_fd, NULL, 0))) + if ((result = unbundle(r, &header, bundle_fd, NULL, + VERIFY_BUNDLE_QUIET))) return 1; /* diff --git a/bundle.c b/bundle.c index 1f6a7f782e1..4ef7256aa11 100644 --- a/bundle.c +++ b/bundle.c @@ -216,7 +216,10 @@ int verify_bundle(struct repository *r, add_pending_object(&revs, o, name); continue; } - if (++ret == 1) + ret++; + if (flags & VERIFY_BUNDLE_QUIET) + continue; + if (ret == 1) error("%s", message); error("%s %s", oid_to_hex(oid), name); } @@ -243,7 +246,10 @@ int verify_bundle(struct repository *r, assert(o); /* otherwise we'd have returned early */ if (o->flags & SHOWN) continue; - if (++ret == 1) + ret++; + if (flags & VERIFY_BUNDLE_QUIET) + continue; + if (ret == 1) error("%s", message); error("%s %s", oid_to_hex(oid), name); } diff --git a/bundle.h b/bundle.h index 6652e819981..575c34245d1 100644 --- a/bundle.h +++ b/bundle.h @@ -32,6 +32,7 @@ int create_bundle(struct repository *r, const char *path, enum verify_bundle_flags { VERIFY_BUNDLE_VERBOSE = (1 << 0), + VERIFY_BUNDLE_QUIET = (1 << 1), }; int verify_bundle(struct repository *r, struct bundle_header *header, diff --git a/t/t5558-clone-bundle-uri.sh b/t/t5558-clone-bundle-uri.sh index a86dc04f528..9b159078386 100755 --- a/t/t5558-clone-bundle-uri.sh +++ b/t/t5558-clone-bundle-uri.sh @@ -99,7 +99,10 @@ test_expect_success 'clone bundle list (file, no heuristic)' ' uri = file://$(pwd)/clone-from/bundle-4.bundle EOF - git clone --bundle-uri="file://$(pwd)/bundle-list" clone-from clone-list-file && + git clone --bundle-uri="file://$(pwd)/bundle-list" \ + clone-from clone-list-file 2>err && + ! grep "Repository lacks these prerequisite commits" err && + git -C clone-from for-each-ref --format="%(objectname)" >oids && git -C clone-list-file cat-file --batch-check err && + ! grep "Repository lacks these prerequisite commits" err && + git -C clone-from for-each-ref --format="%(objectname)" >oids && git -C clone-all-some cat-file --batch-check err && + ! grep "Repository lacks these prerequisite commits" err && + git -C clone-from for-each-ref --format="%(objectname)" >oids && git -C clone-all-fail cat-file --batch-check err && + ! grep "Repository lacks these prerequisite commits" err && + git -C clone-from for-each-ref --format="%(objectname)" >oids && git -C clone-any-file cat-file --batch-check err && + ! grep "Repository lacks these prerequisite commits" err && + git -C clone-from for-each-ref --format="%(objectname)" >oids && git -C clone-list-http cat-file --batch-check X-Patchwork-Id: 13005055 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8FA5AC4332F for ; Wed, 12 Oct 2022 12:53:28 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229573AbiJLMx1 (ORCPT ); Wed, 12 Oct 2022 08:53:27 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38390 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229755AbiJLMxN (ORCPT ); Wed, 12 Oct 2022 08:53:13 -0400 Received: from mail-wr1-x436.google.com (mail-wr1-x436.google.com [IPv6:2a00:1450:4864:20::436]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 972F1C97C4 for ; Wed, 12 Oct 2022 05:52:57 -0700 (PDT) Received: by mail-wr1-x436.google.com with SMTP id bv10so22543698wrb.4 for ; Wed, 12 Oct 2022 05:52:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:from:to:cc:subject:date :message-id:reply-to; bh=2n1oOEOH54xL4jFO2BgRG4PT6/KCIkQNTgOeWChEVmg=; b=qH4TBTwYjWep+19ldgpjTTDaQohHOGAWVpCnvNAAe1lfhR8yHzILM+E+KsvvEBeW+1 Xf0ttCtHWqn7j210xBywiogxXgKdBVLmmjUXL6ZrMIrsGrnBOqMPqUWz3EfdeAqAOQNq BXIBf4NUCLhN5q+Cxv65JsJtMBe5kyDKWUEWJsZX9UTAjmBeuJo0iF+aH0PIIB8vHsB4 IMGlhmrdK1OxrAdfPzanyyerI9nDkAJhdVwm9r5x/h/tfal14mesX5EP2myYzP9kJsYe ynPP4rPcaIVat2x6prNYNymUi2IQilXd/Fpp6DAvGbRog+HR74Aq4t9xSl5GY59PdbdH UHWg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=2n1oOEOH54xL4jFO2BgRG4PT6/KCIkQNTgOeWChEVmg=; b=pf8gvEx3m8zhoELTclKQH+wdzk1IkiqayoAaKpX0yZ+c0FN00OPGkFcAsF9/N0Ithk kc0lzu6nxhJgKs4v7WAWEVmhNvg7LQar/bpRdMdL/mgYIws0g60BXzpijJFJckTPq1UN xNfG09qjOoqGekalQNkT8Yp6bCXV3lllSGp3fAfSnHCdRQb/MuNnG/lx0E3xSqE5UbfI 9WPLEup/i0rI2EBcbNUDbt7i2l9xStzQ6JjmNOITnAlsVQRD44a1JYtVqsDb3pXxyiiN Wbjna1lEbZhyOPvypU5hXKJWkh4+Dfa8bgW+QPSjluoD/JWlrlqUxHhbmj8Gq8Ehc3mQ laMw== X-Gm-Message-State: ACrzQf1xF3JPn5DdsCvImxxMQ9iOlUP3MLi58gyumOetrO9nIMuuy58a ecwjd1FjTa4/fPTH+mP+iJR7vSab9Ss= X-Google-Smtp-Source: AMsMyM5+Wp9s8yJn/3Ri2eS5L1xfwmBGChLWZpZYKCReuquVB0F4yzg4wJjCG2jst3Xa3DBIwmNUlg== X-Received: by 2002:a05:6000:2c1:b0:22e:7507:a182 with SMTP id o1-20020a05600002c100b0022e7507a182mr18032972wry.550.1665579176909; Wed, 12 Oct 2022 05:52:56 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id p17-20020a5d4591000000b0022e32f4c06asm13388473wrq.11.2022.10.12.05.52.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 12 Oct 2022 05:52:56 -0700 (PDT) Message-Id: <5729ff2af4bb56a68624b7942b8afa67601adb43.1665579160.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Wed, 12 Oct 2022 12:52:39 +0000 Subject: [PATCH v5 12/12] bundle-uri: suppress stderr from remote-https Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, me@ttaylorr.com, newren@gmail.com, avarab@gmail.com, mjcheetham@outlook.com, steadmon@google.com, Glen Choo , Jonathan Tan , Teng Long , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee When downloading bundles from a git-remote-https subprocess, the bundle URI logic wants to be opportunistic and download as much as possible and work with what did succeed. This is particularly important in the "any" mode, where any single bundle success will work. If the URI is not available, the git-remote-https process will die() with a "fatal:" error message, even though that error is not actually fatal to the super process. Since stderr is passed through, it looks like a fatal error to the user. Suppress stderr to avoid these errors from bubbling to the surface. The bundle URI API adds its own warning() messages on these failures. Signed-off-by: Derrick Stolee --- bundle-uri.c | 1 + t/t5558-clone-bundle-uri.sh | 16 ++++++++++++++-- 2 files changed, 15 insertions(+), 2 deletions(-) diff --git a/bundle-uri.c b/bundle-uri.c index d872acf5ab0..79a914f961b 100644 --- a/bundle-uri.c +++ b/bundle-uri.c @@ -230,6 +230,7 @@ static int download_https_uri_to_file(const char *file, const char *uri) int found_get = 0; strvec_pushl(&cp.args, "git-remote-https", uri, NULL); + cp.err = -1; cp.in = -1; cp.out = -1; diff --git a/t/t5558-clone-bundle-uri.sh b/t/t5558-clone-bundle-uri.sh index 9b159078386..9155f31fa2c 100755 --- a/t/t5558-clone-bundle-uri.sh +++ b/t/t5558-clone-bundle-uri.sh @@ -147,6 +147,8 @@ test_expect_success 'clone bundle list (file, all mode, some failures)' ' git clone --bundle-uri="file://$(pwd)/bundle-list" \ clone-from clone-all-some 2>err && ! grep "Repository lacks these prerequisite commits" err && + ! grep "fatal" err && + grep "warning: failed to download bundle from URI" err && git -C clone-from for-each-ref --format="%(objectname)" >oids && git -C clone-all-some cat-file --batch-check err && ! grep "Repository lacks these prerequisite commits" err && + ! grep "fatal" err && + grep "warning: failed to download bundle from URI" err && git -C clone-from for-each-ref --format="%(objectname)" >oids && git -C clone-all-fail cat-file --batch-check err && + ! grep "fatal" err && + grep "warning: failed to download bundle from URI" err && + git -C clone-from for-each-ref --format="%(objectname)" >oids && git -C clone-any-fail cat-file --batch-check err && + ! grep "fatal" err && + grep "warning: failed to download bundle from URI" err && + git -C clone-from for-each-ref --format="%(objectname)" >oids && git -C clone-any-http cat-file --batch-check