From patchwork Wed May 15 13:25:41 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christian Couder X-Patchwork-Id: 13665273 Received: from mail-wm1-f41.google.com (mail-wm1-f41.google.com [209.85.128.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A91EE824A4 for ; Wed, 15 May 2024 13:25:59 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.41 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715779561; cv=none; b=ddOi0jiQ0KaYQ03sK7tPDd38kHBMm3dTbvi62cQ+rOahrfte5WLHRDj4XBiasRQ4zwHvFNEQexUsv8p80QGVlvKEgj42Ut10UNoN443KagvtuQMfv9+TK2vUvsNJqYa3mlfoq33viUQ5RrM92pMxcmER1PXWeRXDVo9Ts7SBoG0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715779561; c=relaxed/simple; bh=KG0/oIlW41ni4yfiBN4yXwA7Xg7ZK1cOEIoxF5ejiFk=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=NGrucA55HDYjZ5mxsGPs8ep6rYfxk0BwRTonyE7kJv/jNEU2ZWF5CJkyIp8DChwKRkqjWRq48RJ5E2gp6OUHqJ/wGcVsMA0BQIisgzH62lKoEEaQ1yRDiDJDNOKMw3+kXD6bJu9JIbehizXoy3+17K/RVs9sYfMoZ1ZM1JFrLKk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=Tk4tzBQl; arc=none smtp.client-ip=209.85.128.41 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Tk4tzBQl" Received: by mail-wm1-f41.google.com with SMTP id 5b1f17b1804b1-420180b5897so23243035e9.3 for ; Wed, 15 May 2024 06:25:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1715779557; x=1716384357; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=XFtFCVfMlXMkA3jBFB2kvJWS+iqT35YBFqX/f36bGVw=; b=Tk4tzBQliEMSw/2byJ6/BGEuEbFPH4/KZzCNlFwjTNIiyQmmZWtWgL1BL9RYHc4z9A sSvkSfxXO0yaSvKvNpe5FSzDVcL+8JUwzDlHZ+yut/30F8KhlGgoJNevPP7A8Obyb3mi +n7Dp9wQeDqP6q5cdzGMnHercaYplI2iXB6jYEhBVWu0oyoR1eaj17o6KU8D/haVxr3i vsUNSCYEa37mvc5Hnb1DnZLsceLJZihK9reKvT03b7ug4Cmlf7opobd6MGCAyMMrAP24 cMruhWrqPrHo22WdQXvQDPcnQi//6eeNDVYG2Zc2vqyhg0yn22xnDflHzrP0xxNlAd9/ iwZA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1715779557; x=1716384357; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=XFtFCVfMlXMkA3jBFB2kvJWS+iqT35YBFqX/f36bGVw=; b=ny/X8ebn2fWpdJ3OqYJ7OVg2eYwi+LZcnULwlA44HKqnBYlcQSoS7IYzhoxw06jpVZ V2KIUhj4hYJJLK3NcA+qBlGamSz90i0krnK4LUdQkxpSUJV6y+qU64+Sox6Yb+gSVvzs jc6F87mMiEm1lRcWyL7OAKllUmwH+p+nuR/x9NUA07fP4cSwYP1kZ8X11TGYiellWE3N PB2OAPUN1zFfdW0m1fsJPo0LzuA7z5REyRBYexIU2Vntfd/raHQMuV9GGiZGDb04Z+Sf kWtoNXS+0e2EpJMpZbTUBtXWaxd1ehrMlnpNXtepwLmRyCzoFt+IpWioEKhCVxG0B/bq eZGQ== X-Gm-Message-State: AOJu0YyKameE7AYMw9XoyGh+ydJqW8VBdIUUGomJAy42aDpUnfKJdhM5 QoIeizb/ygSAwYqVOXsLfJ/Kso+tzB/oKTdaCr9yQH6YBBneWqIPhO636g== X-Google-Smtp-Source: AGHT+IGgSx+1jZx7C+wcgj8GnjsTaKPtj3EEDvejM945sdsPgRjM6jbpJ4296UNoWUX0kHgojirsIQ== X-Received: by 2002:a05:600c:4744:b0:418:f991:8ad4 with SMTP id 5b1f17b1804b1-41fea93213dmr127015615e9.6.1715779557457; Wed, 15 May 2024 06:25:57 -0700 (PDT) Received: from christian-Precision-5550.. (176-138-135-207.abo.bbox.fr. [176.138.135.207]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-4201592e5f3sm117171695e9.43.2024.05.15.06.25.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 May 2024 06:25:57 -0700 (PDT) From: Christian Couder To: git@vger.kernel.org Cc: Junio C Hamano , John Cai , Patrick Steinhardt , Christian Couder , Christian Couder Subject: [PATCH v2 1/3] rev-list: refactor --missing= Date: Wed, 15 May 2024 15:25:41 +0200 Message-ID: <20240515132543.851987-2-christian.couder@gmail.com> X-Mailer: git-send-email 2.45.1.148.g0f5efb064b In-Reply-To: <20240515132543.851987-1-christian.couder@gmail.com> References: <20221012135114.294680-1-christian.couder@gmail.com> <20240515132543.851987-1-christian.couder@gmail.com> Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Both `git rev-list` and `git pack-objects` support a `--missing=` feature, but they currently don't share any code for that. Refactor the support for `--missing=` in "builtin/rev-list.c" into new "missing.{c,h}" files. In a following commit, that refactored code will be used in "builtin/pack-objects.c" too. In yet a following commit, we are going to add support for a similar 'missing-action' feature to another command, and we are also going to reuse code from the new "missing.{c,h}" files. As `enum missing_action` and parse_missing_action_value() are moved to "missing.{c,h}", we need to modify the latter a bit, so that it stops updating any global variable, but instead returns the parsed value or -1 on error. Signed-off-by: Christian Couder --- Makefile | 1 + builtin/rev-list.c | 43 ++++++++----------------------------------- missing.c | 20 ++++++++++++++++++++ missing.h | 17 +++++++++++++++++ 4 files changed, 46 insertions(+), 35 deletions(-) create mode 100644 missing.c create mode 100644 missing.h diff --git a/Makefile b/Makefile index 0285db5630..e0ddcc2cbd 100644 --- a/Makefile +++ b/Makefile @@ -1062,6 +1062,7 @@ LIB_OBJS += merge-recursive.o LIB_OBJS += merge.o LIB_OBJS += midx.o LIB_OBJS += midx-write.o +LIB_OBJS += missing.o LIB_OBJS += name-hash.o LIB_OBJS += negotiator/default.o LIB_OBJS += negotiator/noop.o diff --git a/builtin/rev-list.c b/builtin/rev-list.c index 77803727e0..40aa770c47 100644 --- a/builtin/rev-list.c +++ b/builtin/rev-list.c @@ -20,6 +20,7 @@ #include "reflog-walk.h" #include "oidset.h" #include "packfile.h" +#include "missing.h" static const char rev_list_usage[] = "git rev-list [] ... [--] [...]\n" @@ -71,12 +72,6 @@ static struct oidset omitted_objects; static int arg_print_omitted; /* print objects omitted by filter */ static struct oidset missing_objects; -enum missing_action { - MA_ERROR = 0, /* fail if any missing objects are encountered */ - MA_ALLOW_ANY, /* silently allow ALL missing objects */ - MA_PRINT, /* print ALL missing objects in special section */ - MA_ALLOW_PROMISOR, /* silently allow all missing PROMISOR objects */ -}; static enum missing_action arg_missing_action; /* display only the oid of each object encountered */ @@ -392,34 +387,6 @@ static void print_disk_usage(off_t size) strbuf_release(&sb); } -static inline int parse_missing_action_value(const char *value) -{ - if (!strcmp(value, "error")) { - arg_missing_action = MA_ERROR; - return 1; - } - - if (!strcmp(value, "allow-any")) { - arg_missing_action = MA_ALLOW_ANY; - fetch_if_missing = 0; - return 1; - } - - if (!strcmp(value, "print")) { - arg_missing_action = MA_PRINT; - fetch_if_missing = 0; - return 1; - } - - if (!strcmp(value, "allow-promisor")) { - arg_missing_action = MA_ALLOW_PROMISOR; - fetch_if_missing = 0; - return 1; - } - - return 0; -} - static int try_bitmap_count(struct rev_info *revs, int filter_provided_objects) { @@ -569,10 +536,16 @@ int cmd_rev_list(int argc, const char **argv, const char *prefix) for (i = 1; i < argc; i++) { const char *arg = argv[i]; if (skip_prefix(arg, "--missing=", &arg)) { + int res; if (revs.exclude_promisor_objects) die(_("options '%s' and '%s' cannot be used together"), "--exclude-promisor-objects", "--missing"); - if (parse_missing_action_value(arg)) + res = parse_missing_action_value(arg); + if (res >= 0) { + if (res != MA_ERROR) + fetch_if_missing = 0; + arg_missing_action = res; break; + } } } diff --git a/missing.c b/missing.c new file mode 100644 index 0000000000..ce3cf734a8 --- /dev/null +++ b/missing.c @@ -0,0 +1,20 @@ +#include "git-compat-util.h" +#include "missing.h" +#include "object-file.h" + +int parse_missing_action_value(const char *value) +{ + if (!strcmp(value, "error")) + return MA_ERROR; + + if (!strcmp(value, "allow-any")) + return MA_ALLOW_ANY; + + if (!strcmp(value, "print")) + return MA_PRINT; + + if (!strcmp(value, "allow-promisor")) + return MA_ALLOW_PROMISOR; + + return -1; +} diff --git a/missing.h b/missing.h new file mode 100644 index 0000000000..1e378d6215 --- /dev/null +++ b/missing.h @@ -0,0 +1,17 @@ +#ifndef MISSING_H +#define MISSING_H + +enum missing_action { + MA_ERROR = 0, /* fail if any missing objects are encountered */ + MA_ALLOW_ANY, /* silently allow ALL missing objects */ + MA_PRINT, /* print ALL missing objects in special section */ + MA_ALLOW_PROMISOR, /* silently allow all missing PROMISOR objects */ +}; + +/* + Return an `enum missing_action` in case parsing is successful or -1 + if parsing failed. +*/ +int parse_missing_action_value(const char *value); + +#endif /* MISSING_H */ From patchwork Wed May 15 13:25:42 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christian Couder X-Patchwork-Id: 13665274 Received: from mail-wm1-f41.google.com (mail-wm1-f41.google.com [209.85.128.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D8F0585959 for ; Wed, 15 May 2024 13:26:01 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.41 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715779563; cv=none; b=bG9Z9JSoslNSjC6ymkRDcbhxNWdm+cmIpBgTL/s1UJkaeguPwnNGCkrecFatUwR/eabjtfBhc2d91r++EPZygUKxZ8Dy5ROW5HeGJdw3mOOSvK5n99KPlaJW+lUIcWbRROVrjz4jBWZ6PqkCG3mN4u7YM/6IOulFwRvN9JaIS8A= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715779563; c=relaxed/simple; bh=8nN/fE2FtKpcrq9oD8RiDbzkezPbsmM/6LDkMaESHMc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=PxXeerN+tCICt5n1qpS30LAYXqRaQ7etc2IS08eWygHet+faXBXYH8dm10UYQgGABzpyD/LCBZp/YzBwlq4O6hgMMMpdh5vvwL+MH/gpBCZByv2C+DoIsKrBnDorrz3yqGp131nKVbdAl4hjz3HOfVHaTb5ooTsdXXEV/6ARKv0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=eFcUFXQb; arc=none smtp.client-ip=209.85.128.41 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="eFcUFXQb" Received: by mail-wm1-f41.google.com with SMTP id 5b1f17b1804b1-41ffad2426eso51965475e9.3 for ; Wed, 15 May 2024 06:26:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1715779559; x=1716384359; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=T0hmo/p0VAGo1p4QnVSIM0GabAIbA3aaLJ5QyA1Viqg=; b=eFcUFXQbh8YJLEO/WKzniyG2CaW1sLwR4AEHAeG6FN01OaKbgrV0v3GpzvDT9aQrTQ vs4SZbTDfFnibjbimvUgxiuCl10XIHYBp0JbY6YpoK6PbWHqWC1NQsssp9INHiKSxnhD /xeX0vD2V7quHXxaDfvpLnl3/6abm6FN4fntLJePCe8nm1dDAPAMGu7QSpn0Ig95cCDE H5ZBmx0TV6d4MBz5b6VQrFBuEDny9D45yNLW51XBhvUYpv2n+zjgeymg28GwK96y64/A jDAEnnog+oWPjnJrqvLLXJ4T2fdZU73FTjux80FcFHHvbKO/do9FxuRleLH0rcnKgie9 Lgfg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1715779559; x=1716384359; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=T0hmo/p0VAGo1p4QnVSIM0GabAIbA3aaLJ5QyA1Viqg=; b=kaMeQq7GKjMfa6ASIHiNmf6xd5oW4pbhNpab7jvN7LBpwlRvzlsCeZNxJVn6MRtBN5 vIscHkKeuVW20pYnFwIQ7xOjXuNGJI1S6MQ0mbhsIH0K8CVXs4pDVrFHsOs7l9XwZqKv jmEabg+arzQysjahzb/u+cDH9Sqv/jpPlup5iH4EoWXHF20jigpdosh0B+EnGATxQEzP skspRAPsSTyIHypAiuZDCjdcNWutaQnJBCht31QwWH4tP8hzO/Ahoc5lZAT/m1BKM/JE WCLXnn8CYdYCpuSSfrVt0TQo4s+2VTelMmsqr4a11T8ndGtfqGH2x5WQs5vGsq/bcbq5 KMQQ== X-Gm-Message-State: AOJu0YxpT/zw0U/tpaahEKoZI5mdCs8X4bqeQKiSfK27lPTkCoE7HZFI QCDTyJ8HU4DIkYCf6njAlz2TDaS1oAZYy1XcecoX1Z7kc2UkMaQu9p0uJA== X-Google-Smtp-Source: AGHT+IGOfrlEN91KJHnEBhS6tTzF8I8NYSSScTpEtJWt326yxXAQFCBAfH0tgFaWAu4hWeYKfSuvFA== X-Received: by 2002:a05:600c:1c0e:b0:41c:503:9ae4 with SMTP id 5b1f17b1804b1-41feac55e4dmr150265315e9.25.1715779559281; Wed, 15 May 2024 06:25:59 -0700 (PDT) Received: from christian-Precision-5550.. (176-138-135-207.abo.bbox.fr. [176.138.135.207]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-4201592e5f3sm117171695e9.43.2024.05.15.06.25.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 May 2024 06:25:57 -0700 (PDT) From: Christian Couder To: git@vger.kernel.org Cc: Junio C Hamano , John Cai , Patrick Steinhardt , Christian Couder , Christian Couder Subject: [PATCH v2 2/3] pack-objects: use the missing action API Date: Wed, 15 May 2024 15:25:42 +0200 Message-ID: <20240515132543.851987-3-christian.couder@gmail.com> X-Mailer: git-send-email 2.45.1.148.g0f5efb064b In-Reply-To: <20240515132543.851987-1-christian.couder@gmail.com> References: <20221012135114.294680-1-christian.couder@gmail.com> <20240515132543.851987-1-christian.couder@gmail.com> Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Both `git rev-list` and `git pack-objects` support a `--missing=` option. Previous commits created an API in "missing.{c,h}" to help supporting that option, but only `git rev-list` has been using that API so far. Let's make `git pack-objects` use it too. This involves creating a new show_object_fn_from_action() function to set the `fn_show_object` variable independently from parsing the missing action, which is now performed by the parse_missing_action_value() API function. Signed-off-by: Christian Couder --- builtin/pack-objects.c | 48 ++++++++++++++++++++++-------------------- 1 file changed, 25 insertions(+), 23 deletions(-) diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c index baf0090fc8..55d08c686d 100644 --- a/builtin/pack-objects.c +++ b/builtin/pack-objects.c @@ -39,6 +39,7 @@ #include "promisor-remote.h" #include "pack-mtimes.h" #include "parse-options.h" +#include "missing.h" /* * Objects we are going to pack are collected in the `to_pack` structure. @@ -250,11 +251,6 @@ static unsigned long window_memory_limit = 0; static struct string_list uri_protocols = STRING_LIST_INIT_NODUP; -enum missing_action { - MA_ERROR = 0, /* fail if any missing objects are encountered */ - MA_ALLOW_ANY, /* silently allow ALL missing objects */ - MA_ALLOW_PROMISOR, /* silently allow all missing PROMISOR objects */ -}; static enum missing_action arg_missing_action; static show_object_fn fn_show_object; @@ -3826,33 +3822,39 @@ static void show_object__ma_allow_promisor(struct object *obj, const char *name, show_object(obj, name, data); } +static show_object_fn show_object_fn_from_action(enum missing_action action) +{ + switch (action) { + case MA_ERROR: + return show_object; + case MA_ALLOW_ANY: + return show_object__ma_allow_any; + case MA_ALLOW_PROMISOR: + return show_object__ma_allow_promisor; + default: + BUG("invalid missing action %d", action); + } +} + static int option_parse_missing_action(const struct option *opt UNUSED, const char *arg, int unset) { + int res; + assert(arg); assert(!unset); - if (!strcmp(arg, "error")) { - arg_missing_action = MA_ERROR; - fn_show_object = show_object; - return 0; - } + res = parse_missing_action_value(arg); + if (res < 0 || (res != MA_ERROR && + res != MA_ALLOW_ANY && + res != MA_ALLOW_PROMISOR)) + die(_("invalid value for '%s': '%s'"), "--missing", arg); - if (!strcmp(arg, "allow-any")) { - arg_missing_action = MA_ALLOW_ANY; + if (res != MA_ERROR) fetch_if_missing = 0; - fn_show_object = show_object__ma_allow_any; - return 0; - } - - if (!strcmp(arg, "allow-promisor")) { - arg_missing_action = MA_ALLOW_PROMISOR; - fetch_if_missing = 0; - fn_show_object = show_object__ma_allow_promisor; - return 0; - } + arg_missing_action = res; + fn_show_object = show_object_fn_from_action(arg_missing_action); - die(_("invalid value for '%s': '%s'"), "--missing", arg); return 0; } From patchwork Wed May 15 13:25:43 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christian Couder X-Patchwork-Id: 13665275 Received: from mail-wm1-f47.google.com (mail-wm1-f47.google.com [209.85.128.47]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4943D86130 for ; Wed, 15 May 2024 13:26:02 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.47 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715779564; cv=none; b=HVEJjtozo8rKWML7ZtvRo3lz9nfCA9GQuVw3OyYGed+1X4gKbc2QGdLRCBNdu1Khw2iVCpixVHwinlMuQQ82O49OOcKad9kBUUcEmvOtAA/RGywrW+Ofh6tn0K4VTDgctqJSy3vQl1CcIk//LankS74oIjN/k3FHXYEjH2I9hUE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715779564; c=relaxed/simple; bh=REoIPoe24jdwxjxkeG2CYZAOTqz3du3DXTf4krpNuZc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=DfNtTIaUjm6x2zLXZHQET9o0yj2pRKKKoGKtM6+MWcjmKfov/gN7fw2/0uAtVIIdKUKStU10nnmpIksataSASXrLD2R4KtJfKnODVHFghemJLTpOFNGBdafkgLfHThWHfs4xEfpgwgR+7jFav1vk+4/S9kCxVG1V6SwQd77rznk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=AWj92axW; arc=none smtp.client-ip=209.85.128.47 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="AWj92axW" Received: by mail-wm1-f47.google.com with SMTP id 5b1f17b1804b1-420180b5838so22215535e9.2 for ; Wed, 15 May 2024 06:26:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1715779560; x=1716384360; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=K++TS67hHUmVSqDEzgjYWpQZqD4sAfrDB03HxcdpZiY=; b=AWj92axWXaNbC4i5S+qi4CjULLpP6xlzlZG4VbzoEU1SGdhl11BlnHQynV3LK/FTMk e9U347zI40meOVFFwu2f/sDAiZv46i6IrNvQ2W8RuwRKxYOFeBi6PNJQ8sj+vMs+P5f2 YUzY6GnmGQiG2XEZTmKZGPjKMaZTFXogVAOtjbjjymlxBU2fUYylTWqHV6t11KWFORqZ P2G5aHPEg24Om5+mjxXgf2wwCByHAF513Nh3o5UCykNDanf70QuQROBnVJMATKG96uYG FU5Lb4rYSFiq8cSolhItD4nMHgrNZ+uCfDKxtwfzuH9l17ftxTdDlqGC7LZpCdi4LDRB Djkw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1715779560; x=1716384360; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=K++TS67hHUmVSqDEzgjYWpQZqD4sAfrDB03HxcdpZiY=; b=P/RvtEXRAPd17udoWhluGuDbU/t5T57SGToAoy2+eIJLG7Mab9mv/uURFX4gnmX8Y/ wGNGGq8PPVncTj8DtA9/FgcFXRK776bG1miPOsyrJ5XaC3Ti0M4aozImpf08hiEry8kq jx+PlHwfXlDDfh4misT5oV1yYh/5Svv4tcNjHaAjzP3YhoO0OvFO8XiJofD+hQyjXSgj SNuHSGAlzaDLfHYZGVs6b9ZK3Mul16O9JD3CumywuU+wSr4WoZ8Ps0ZIflORy4L+HnUN yjjLtk/YCOIv7nOs5Q6xTTb61ygn4zM9Xu5wiz7tcEzYy6miVa/FVJWiVp/tH5SwxcZO VI1A== X-Gm-Message-State: AOJu0YwJr60nugArwRroMEnojrKS+VFzrWfGBGGZU0AA08SHt4ti4S6i 16LNDi18xLChYRQT3+fP4uzraCTW9URxQnMVtz0+LO3xNVrVRY+rLiM75A== X-Google-Smtp-Source: AGHT+IHjPP8+Tc2BbatE5Mwvm31E60267vdBcaFGZrEg0bgOLaHHx9b14bvL0gxFZQYvV0U/IKrhGg== X-Received: by 2002:a05:600c:45c5:b0:41b:935:2492 with SMTP id 5b1f17b1804b1-41fead6443fmr185200685e9.36.1715779560162; Wed, 15 May 2024 06:26:00 -0700 (PDT) Received: from christian-Precision-5550.. (176-138-135-207.abo.bbox.fr. [176.138.135.207]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-4201592e5f3sm117171695e9.43.2024.05.15.06.25.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 May 2024 06:25:59 -0700 (PDT) From: Christian Couder To: git@vger.kernel.org Cc: Junio C Hamano , John Cai , Patrick Steinhardt , Christian Couder Subject: [PATCH v2 3/3] upload-pack: allow configuring a missing-action Date: Wed, 15 May 2024 15:25:43 +0200 Message-ID: <20240515132543.851987-4-christian.couder@gmail.com> X-Mailer: git-send-email 2.45.1.148.g0f5efb064b In-Reply-To: <20240515132543.851987-1-christian.couder@gmail.com> References: <20221012135114.294680-1-christian.couder@gmail.com> <20240515132543.851987-1-christian.couder@gmail.com> Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Christian Couder In case some objects are missing from a server, it might still be useful to be able to fetch or clone from it if the client already has the missing objects or can get them in some way. For example, in case both the server and the client are using a separate promisor remote that contain some objects, it can be better if the server doesn't try to send such objects back to the client, but instead let the client get those objects separately from the promisor remote. (The client needs to have the separate promisor remote configured, for that to work.) Another example could be a server where some objects have been corrupted or deleted. It could still be useful for clients who could get those objects from another source, like perhaps a different client, to be able to fetch or clone from the server. To configure what a server should do if there are such missing objects, let's teach `git upload-pack` a new `uploadpack.missingAction` configuration variable. This new missing-action works in a similar way as what `git rev-list` and `git pack-objects` already support with their `--missing=` command line options. In fact when `uploadpack.missingAction` is set to something different than "error", `git upload-pack` will just pass the corresponding `--missing=` to `git pack-objects`. So this new missing-action has the same limitations as `git pack-objects --missing=`. Especially when not using promisor remotes, 'allow-any' works only for blobs. Another limitation comes from `git upload-pack` itself which requires setting `GIT_NO_LAZY_FETCH` to 0 since 7b70e9efb1 (upload-pack: disable lazy-fetching by default, 2024-04-16) for lazy fetching from a promisor remote to work on the server side. Signed-off-by: Christian Couder --- Documentation/config/uploadpack.txt | 9 ++ missing.c | 16 ++++ missing.h | 2 + t/t5706-upload-pack-missing.sh | 125 ++++++++++++++++++++++++++++ upload-pack.c | 19 +++++ 5 files changed, 171 insertions(+) create mode 100755 t/t5706-upload-pack-missing.sh diff --git a/Documentation/config/uploadpack.txt b/Documentation/config/uploadpack.txt index 16264d82a7..cd70e853b3 100644 --- a/Documentation/config/uploadpack.txt +++ b/Documentation/config/uploadpack.txt @@ -82,3 +82,12 @@ uploadpack.allowRefInWant:: is intended for the benefit of load-balanced servers which may not have the same view of what OIDs their refs point to due to replication delay. + +uploadpack.missingAction:: + If this option is set, `upload-pack` will call + linkgit:git-pack-objects[1] passing it the corresponding + `--missing=` command line option. See the + documentation for that option, to see the valid values and + their meaning. This could be useful if some objects are + missing on a server, but a client already has them or could + still get them from somewhere else. diff --git a/missing.c b/missing.c index ce3cf734a8..75036d2089 100644 --- a/missing.c +++ b/missing.c @@ -18,3 +18,19 @@ int parse_missing_action_value(const char *value) return -1; } + +const char *missing_action_to_string(enum missing_action action) +{ + switch (action) { + case MA_ERROR: + return "error"; + case MA_ALLOW_ANY: + return "allow-any"; + case MA_PRINT: + return "print"; + case MA_ALLOW_PROMISOR: + return "allow-promisor"; + default: + BUG("invalid missing action %d", action); + } +} diff --git a/missing.h b/missing.h index 1e378d6215..74753d7887 100644 --- a/missing.h +++ b/missing.h @@ -14,4 +14,6 @@ enum missing_action { */ int parse_missing_action_value(const char *value); +const char *missing_action_to_string(enum missing_action action); + #endif /* MISSING_H */ diff --git a/t/t5706-upload-pack-missing.sh b/t/t5706-upload-pack-missing.sh new file mode 100755 index 0000000000..0f39c0ddd0 --- /dev/null +++ b/t/t5706-upload-pack-missing.sh @@ -0,0 +1,125 @@ +#!/bin/sh + +test_description='handling of missing objects in upload-pack' + +TEST_PASSES_SANITIZE_LEAK=true +. ./test-lib.sh + +# Setup the repository with three commits, this way HEAD is always +# available and we can hide commit 1 or 2. +test_expect_success 'setup: create "template" repository' ' + git init template && + test_commit -C template 1 && + test_commit -C template 2 && + test_commit -C template 3 && + test-tool genrandom foo 10240 >template/foo && + git -C template add foo && + git -C template commit -m foo +' + +# A bare repo will act as a server repo with unpacked objects. +test_expect_success 'setup: create bare "server" repository' ' + git clone --bare --no-local template server && + mv server/objects/pack/pack-* . && + packfile=$(ls pack-*.pack) && + git -C server unpack-objects --strict <"$packfile" +' + +# Fetching with 'uploadpack.missingAction=allow-any' only works with +# blobs, as `git pack-objects --missing=allow-any` fails if a missing +# object is not a blob. +test_expect_success "fetch with uploadpack.missingAction=allow-any" ' + oid="$(git -C server rev-parse HEAD:1.t)" && + oid_path="$(test_oid_to_path $oid)" && + path="server/objects/$oid_path" && + + mv "$path" "$path.hidden" && + test_when_finished "mv $path.hidden $path" && + + git init client && + test_when_finished "rm -rf client" && + + # Client needs the missing objects to be available somehow + client_path="client/.git/objects/$oid_path" && + mkdir -p $(dirname "$client_path") && + cp "$path.hidden" "$client_path" && + + test_must_fail git -C client fetch ../server && + git -C server config uploadpack.missingAction error && + test_must_fail git -C client fetch ../server && + git -C server config uploadpack.missingAction allow-any && + git -C client fetch ../server && + git -C server config --unset uploadpack.missingAction +' + +check_missing_objects () { + git -C "$1" rev-list --objects --all --missing=print > all.txt && + sed -n "s/^\?\(.*\)/\1/p" missing.txt && + test_line_count = "$2" missing.txt && + test "$3" = "$(cat missing.txt)" +} + +test_expect_success "setup for testing uploadpack.missingAction=allow-promisor" ' + # Create another bare repo called "server2" + git init --bare server2 && + + # Copy the largest object from server to server2 + obj="HEAD:foo" && + oid="$(git -C server rev-parse $obj)" && + oid_path="$(test_oid_to_path $oid)" && + path="server/objects/$oid_path" && + path2="server2/objects/$oid_path" && + mkdir -p $(dirname "$path2") && + cp "$path" "$path2" && + + # Repack everything first + git -C server -c repack.writebitmaps=false repack -a -d && + + # Repack without the largest object and create a promisor pack on server + git -C server -c repack.writebitmaps=false repack -a -d \ + --filter=blob:limit=5k --filter-to="$(pwd)" && + promisor_file=$(ls server/objects/pack/*.pack | sed "s/\.pack/.promisor/") && + > "$promisor_file" && + + # Check that only one object is missing on the server + check_missing_objects server 1 "$oid" && + + # Configure server2 as promisor remote for server + git -C server remote add server2 "file://$(pwd)/server2" && + git -C server config remote.server2.promisor true && + + git -C server2 config uploadpack.allowFilter true && + git -C server2 config uploadpack.allowAnySHA1InWant true && + git -C server config uploadpack.allowFilter true && + git -C server config uploadpack.allowAnySHA1InWant true +' + +test_expect_success "fetch with uploadpack.missingAction=allow-promisor" ' + git -C server config uploadpack.missingAction allow-promisor && + + # Clone from server to create a client + GIT_NO_LAZY_FETCH=0 git clone -c remote.server2.promisor=true \ + -c remote.server2.fetch="+refs/heads/*:refs/remotes/server2/*" \ + -c remote.server2.url="file://$(pwd)/server2" \ + --no-local --filter="blob:limit=5k" server client && + test_when_finished "rm -rf client" && + + # Check that the largest object is still missing on the server + check_missing_objects server 1 "$oid" +' + +test_expect_success "fetch without uploadpack.missingAction=allow-promisor" ' + git -C server config --unset uploadpack.missingAction && + + # Clone from server to create a client + GIT_NO_LAZY_FETCH=0 git clone -c remote.server2.promisor=true \ + -c remote.server2.fetch="+refs/heads/*:refs/remotes/server2/*" \ + -c remote.server2.url="file://$(pwd)/server2" \ + --no-local --filter="blob:limit=5k" server client && + test_when_finished "rm -rf client" && + + # Check that the largest object is not missing on the server anymore + check_missing_objects server 0 "" +' + +test_done diff --git a/upload-pack.c b/upload-pack.c index 902144b9d3..f114f92197 100644 --- a/upload-pack.c +++ b/upload-pack.c @@ -29,6 +29,8 @@ #include "write-or-die.h" #include "json-writer.h" #include "strmap.h" +#include "missing.h" +#include "object-file.h" /* Remember to update object flag allocation in object.h */ #define THEY_HAVE (1u << 11) @@ -96,6 +98,8 @@ struct upload_pack_data { const char *pack_objects_hook; + enum missing_action missing_action; + unsigned stateless_rpc : 1; /* v0 only */ unsigned no_done : 1; /* v0 only */ unsigned daemon_mode : 1; /* v0 only */ @@ -315,6 +319,9 @@ static void create_pack_file(struct upload_pack_data *pack_data, strvec_push(&pack_objects.args, "--delta-base-offset"); if (pack_data->use_include_tag) strvec_push(&pack_objects.args, "--include-tag"); + if (pack_data->missing_action) + strvec_pushf(&pack_objects.args, "--missing=%s", + missing_action_to_string(pack_data->missing_action)); if (pack_data->filter_options.choice) { const char *spec = expand_list_objects_filter_spec(&pack_data->filter_options); @@ -1371,6 +1378,18 @@ static int upload_pack_config(const char *var, const char *value, precomposed_unicode = git_config_bool(var, value); } else if (!strcmp("transfer.advertisesid", var)) { data->advertise_sid = git_config_bool(var, value); + } else if (!strcmp("uploadpack.missingaction", var)) { + int res = parse_missing_action_value(value); + if (res < 0 || (res != MA_ERROR && + res != MA_ALLOW_ANY && + res != MA_ALLOW_PROMISOR)) + die(_("invalid value for '%s': '%s'"), var, value); + /* Allow fetching only from promisor remotes */ + if (res == MA_ALLOW_PROMISOR) + fetch_if_missing = 1; + if (res == MA_ALLOW_ANY) + fetch_if_missing = 0; + data->missing_action = res; } if (parse_object_filter_config(var, value, ctx->kvi, data) < 0)