From patchwork Mon Apr 1 16:40:35 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christian Couder X-Patchwork-Id: 10880269 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 5095C139A for ; Mon, 1 Apr 2019 16:41:13 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 3AAB6287C5 for ; Mon, 1 Apr 2019 16:41:13 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 38E9628899; Mon, 1 Apr 2019 16:41:13 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A8B2A287C5 for ; Mon, 1 Apr 2019 16:41:12 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728555AbfDAQlL (ORCPT ); Mon, 1 Apr 2019 12:41:11 -0400 Received: from mail-wm1-f65.google.com ([209.85.128.65]:39339 "EHLO mail-wm1-f65.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728255AbfDAQlL (ORCPT ); Mon, 1 Apr 2019 12:41:11 -0400 Received: by mail-wm1-f65.google.com with SMTP id n25so95843wmk.4 for ; Mon, 01 Apr 2019 09:41:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=g9F06A1ISgB4uAKOxyw6RV1cQApHEBPv50zgn8Xdy5U=; b=Ucg5bBJaC7AON+n5q9Ba6GaAFnaNMqAdn2vtAmMqhNItGAEHCiktDhT5+KL99InLz1 X2E3sdktBLkgMhBNLiRdyAjGk30l3glV/N5Iya0rQ1GM/T3VKFw7ytWBLSTb9l1Z9OBR 6drj52f1Jp46L7mYdpIxt9b17Vl9gOi7+KQp7IKdnDPtDsCp3h/D9VRFaqTgtJyc+mrW ZgpRdkd5BVm7cy+mfEtIDcukWYy6RC0BRcInX+Jww/jMPDFr+k3Ai6QUt3Q/xzmWZUjd yButR9ox5F2bMoSVD1PwXszp+lsiGqxXtzRxVBheQ2EyOA1WxDN5YfAvgWdFqHBCC9Kl pKfg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=g9F06A1ISgB4uAKOxyw6RV1cQApHEBPv50zgn8Xdy5U=; b=EIAYW2B6sN5AWSWEzhdWTYG3UekHd7BlXVkwwdoTt6B8GSj8TS/XO/KU30ORfvTsQM c+WzmHeDbHvuVIEoUh1S5nWyRCL4SNvNcYcSahwoWUwTZLCOn3H3BtgZtHDT67982NdJ qCkIDfPEPa9W96IUSS2QMT6rLSMdzvwqLQfL98uCkzIEqEjU4BBz4+kDCRO7VNJLJNPU PwDFLdqBSPDXw8mcFq+brfmr47tC9d0HDkEdODuqFyoadfWBEEcdZR1noX1tsP24QL7z 5yufrr5IA5BQx+vTLO+v1tadWZjUWyC0/3jbRCJqYHo5za/VIBsGJ3uZ/zbVgJSW3uEe 5VwA== X-Gm-Message-State: APjAAAVeASGgX4euJhO0CgdGsl/2tdupTWPQRCdDS2uope25JsLwsfPI w1yVmYfauqfUyZkRIQxOHu4PCcyB X-Google-Smtp-Source: APXvYqwZ9P4ITx5IqoP7pp7PXvindFWhVs4x3BazRSlJsXmNu5SxXTecV7sDyKHbQ4WhDcR+s6hl2w== X-Received: by 2002:a1c:41d6:: with SMTP id o205mr384593wma.2.1554136868857; Mon, 01 Apr 2019 09:41:08 -0700 (PDT) Received: from localhost.localdomain ([2a04:cec0:101d:3e80:8542:d1d4:667f:a0da]) by smtp.gmail.com with ESMTPSA id s10sm11749070wmh.0.2019.04.01.09.41.06 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 01 Apr 2019 09:41:08 -0700 (PDT) From: Christian Couder X-Google-Original-From: Christian Couder To: git@vger.kernel.org Cc: Junio C Hamano , Jeff King , Ben Peart , Jonathan Tan , Jonathan Nieder , Nguyen Thai Ngoc Duy , Mike Hommey , Lars Schneider , Eric Wong , Christian Couder , Jeff Hostetler , Eric Sunshine , Beat Bolli , Christian Couder Subject: [PATCH v4 01/11] fetch-object: make functions return an error code Date: Mon, 1 Apr 2019 18:40:35 +0200 Message-Id: <20190401164045.17328-2-chriscool@tuxfamily.org> X-Mailer: git-send-email 2.21.0.203.gd44fa53258 In-Reply-To: <20190401164045.17328-1-chriscool@tuxfamily.org> References: <20190401164045.17328-1-chriscool@tuxfamily.org> MIME-Version: 1.0 Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Christian Couder The callers of the fetch_object() and fetch_objects() might be interested in knowing if these functions succeeded or not. Signed-off-by: Christian Couder Signed-off-by: Junio C Hamano --- fetch-object.c | 13 ++++++++----- fetch-object.h | 4 ++-- sha1-file.c | 4 ++-- 3 files changed, 12 insertions(+), 9 deletions(-) diff --git a/fetch-object.c b/fetch-object.c index 4266548800..eac4d448ef 100644 --- a/fetch-object.c +++ b/fetch-object.c @@ -5,11 +5,12 @@ #include "transport.h" #include "fetch-object.h" -static void fetch_refs(const char *remote_name, struct ref *ref) +static int fetch_refs(const char *remote_name, struct ref *ref) { struct remote *remote; struct transport *transport; int original_fetch_if_missing = fetch_if_missing; + int res; fetch_if_missing = 0; remote = remote_get(remote_name); @@ -19,12 +20,14 @@ static void fetch_refs(const char *remote_name, struct ref *ref) transport_set_option(transport, TRANS_OPT_FROM_PROMISOR, "1"); transport_set_option(transport, TRANS_OPT_NO_DEPENDENTS, "1"); - transport_fetch_refs(transport, ref); + res = transport_fetch_refs(transport, ref); fetch_if_missing = original_fetch_if_missing; + + return res; } -void fetch_objects(const char *remote_name, const struct object_id *oids, - int oid_nr) +int fetch_objects(const char *remote_name, const struct object_id *oids, + int oid_nr) { struct ref *ref = NULL; int i; @@ -36,5 +39,5 @@ void fetch_objects(const char *remote_name, const struct object_id *oids, new_ref->next = ref; ref = new_ref; } - fetch_refs(remote_name, ref); + return fetch_refs(remote_name, ref); } diff --git a/fetch-object.h b/fetch-object.h index d6444caa5a..7bcc7cadb0 100644 --- a/fetch-object.h +++ b/fetch-object.h @@ -3,7 +3,7 @@ struct object_id; -void fetch_objects(const char *remote_name, const struct object_id *oids, - int oid_nr); +int fetch_objects(const char *remote_name, const struct object_id *oids, + int oid_nr); #endif diff --git a/sha1-file.c b/sha1-file.c index 494606f771..01cc0590f4 100644 --- a/sha1-file.c +++ b/sha1-file.c @@ -1372,8 +1372,8 @@ int oid_object_info_extended(struct repository *r, const struct object_id *oid, if (fetch_if_missing && repository_format_partial_clone && !already_retried && r == the_repository) { /* - * TODO Investigate having fetch_object() return - * TODO error/success and stopping the music here. + * TODO Investigate checking fetch_object() return + * TODO value and stopping on error here. * TODO Pass a repository struct through fetch_object, * such that arbitrary repositories work. */ From patchwork Mon Apr 1 16:40:36 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christian Couder X-Patchwork-Id: 10880271 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 3573514DE for ; Mon, 1 Apr 2019 16:41:16 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 1E09D28680 for ; Mon, 1 Apr 2019 16:41:16 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 1B77E2877E; Mon, 1 Apr 2019 16:41:16 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A293128749 for ; Mon, 1 Apr 2019 16:41:15 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728629AbfDAQlO (ORCPT ); Mon, 1 Apr 2019 12:41:14 -0400 Received: from mail-wm1-f65.google.com ([209.85.128.65]:36916 "EHLO mail-wm1-f65.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728255AbfDAQlO (ORCPT ); Mon, 1 Apr 2019 12:41:14 -0400 Received: by mail-wm1-f65.google.com with SMTP id v14so113371wmf.2 for ; Mon, 01 Apr 2019 09:41:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=+uJKNeyVs7eJOOFHMl19MBknHdS/OUfniwpw59xGsIs=; b=hLzJbZ4Pun9VrnTWKuxIzhxKlvb9NzSB3f0XyjcGzFQL7f1p81GWuH4gjGp4+T11QD VmSmnjLfdvvLoXskvjnXlbi0Tzxj0u+yzy1TXgc7eCwLheY7L6nIMgWfQy58VM7pwV0p 82NJjjmvRlJiDkbWZgD95mMRUGJHaP1oOAXB+uvP4Bxt7V8u7HG10eg5b9G2ZqVHoXxe VGuiLz9mjNHcdNpiy8qRNyk43u3xx4qPHlK+9XkHuR3Ar0ZAs421S6mYJ5tlDAEGPUsM 2c67sSUpowqVqUmJfrpTwuIzdjkNK+Olx29Uty2qsifbyLy+nnPsWRXwP+1jVKkQjTSw Lj5g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=+uJKNeyVs7eJOOFHMl19MBknHdS/OUfniwpw59xGsIs=; b=Bvlvj2d70NpWkWmfWeizFJOCSwTgtYyJ9NJUjP2DqEvBa3tqH5RfJDXasX5N3PscoW ozVUcc9HdIwG72RD8Bp+86V6LhGnRSMaPxNJaIB4xSvPzRe794YjXywKisn2LXgMe7bi b2B5IDpewwbXFfz1dt5T5WHefG4YZvoHmdO8IkXdIWyJrFHTcP8XyQ7CmDZd9L6DmVL7 6d08GYD0qMsxzMUQV9ucvSWzPfFUWXaMpx6zcn+tfCYquvw1ghkBzkvaFfYYIOz+af5k 7GTba15apIiiUhyuN+SCGNehlkuMnA7qMIDFUXTTPlltDp0felFY2XGxl1zZlEThKbmo XBeg== X-Gm-Message-State: APjAAAXirV4qgW2RtskRmp3sN6mNkzSThm3w4Y5aIzmfnC0ZBleAv474 WBXSazOcCiN2A8dwqK/6RFoQgn6N X-Google-Smtp-Source: APXvYqwFW6yuDIAzXlfFFlPpHPfUXZgWJp9leinLe0/r2OZFQ1abBXz8ZQnEt0Dc2hTZBJuZ3frbUw== X-Received: by 2002:a7b:c00e:: with SMTP id c14mr352754wmb.110.1554136871264; Mon, 01 Apr 2019 09:41:11 -0700 (PDT) Received: from localhost.localdomain ([2a04:cec0:101d:3e80:8542:d1d4:667f:a0da]) by smtp.gmail.com with ESMTPSA id s10sm11749070wmh.0.2019.04.01.09.41.08 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 01 Apr 2019 09:41:10 -0700 (PDT) From: Christian Couder X-Google-Original-From: Christian Couder To: git@vger.kernel.org Cc: Junio C Hamano , Jeff King , Ben Peart , Jonathan Tan , Jonathan Nieder , Nguyen Thai Ngoc Duy , Mike Hommey , Lars Schneider , Eric Wong , Christian Couder , Jeff Hostetler , Eric Sunshine , Beat Bolli , Christian Couder Subject: [PATCH v4 02/11] Add initial support for many promisor remotes Date: Mon, 1 Apr 2019 18:40:36 +0200 Message-Id: <20190401164045.17328-3-chriscool@tuxfamily.org> X-Mailer: git-send-email 2.21.0.203.gd44fa53258 In-Reply-To: <20190401164045.17328-1-chriscool@tuxfamily.org> References: <20190401164045.17328-1-chriscool@tuxfamily.org> MIME-Version: 1.0 Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Christian Couder The promisor-remote.{c,h} files will contain functions to manage many promisor remotes. We expect that there will not be a lot of promisor remotes, so it is ok to use a simple linked list to manage them. Helped-by: Jeff King Signed-off-by: Christian Couder --- Makefile | 1 + promisor-remote.c | 92 +++++++++++++++++++++++++++++++++++++++++++++++ promisor-remote.h | 16 +++++++++ 3 files changed, 109 insertions(+) create mode 100644 promisor-remote.c create mode 100644 promisor-remote.h diff --git a/Makefile b/Makefile index 3e03290d8f..79b45060a1 100644 --- a/Makefile +++ b/Makefile @@ -953,6 +953,7 @@ LIB_OBJS += preload-index.o LIB_OBJS += pretty.o LIB_OBJS += prio-queue.o LIB_OBJS += progress.o +LIB_OBJS += promisor-remote.o LIB_OBJS += prompt.o LIB_OBJS += protocol.o LIB_OBJS += quote.o diff --git a/promisor-remote.c b/promisor-remote.c new file mode 100644 index 0000000000..0c768210ee --- /dev/null +++ b/promisor-remote.c @@ -0,0 +1,92 @@ +#include "cache.h" +#include "promisor-remote.h" +#include "config.h" + +static struct promisor_remote *promisors; +static struct promisor_remote **promisors_tail = &promisors; + +static struct promisor_remote *promisor_remote_new(const char *remote_name) +{ + struct promisor_remote *r; + + if (*remote_name == '/') { + warning(_("promisor remote name cannot begin with '/': %s"), + remote_name); + return NULL; + } + + FLEX_ALLOC_STR(r, name, remote_name); + + *promisors_tail = r; + promisors_tail = &r->next; + + return r; +} + +static struct promisor_remote *promisor_remote_lookup(const char *remote_name, + struct promisor_remote **previous) +{ + struct promisor_remote *r, *p; + + for (p = NULL, r = promisors; r; p = r, r = r->next) + if (r->name && !strcmp(r->name, remote_name)) { + if (previous) + *previous = p; + return r; + } + + return NULL; +} + +static int promisor_remote_config(const char *var, const char *value, void *data) +{ + const char *name; + int namelen; + const char *subkey; + + if (parse_config_key(var, "remote", &name, &namelen, &subkey) < 0) + return 0; + + if (!strcmp(subkey, "promisor")) { + char *remote_name; + + if (!git_config_bool(var, value)) + return 0; + + remote_name = xmemdupz(name, namelen); + + if (!promisor_remote_lookup(remote_name, NULL)) + promisor_remote_new(remote_name); + + free(remote_name); + return 0; + } + + return 0; +} + +static void promisor_remote_init(void) +{ + static int initialized; + + if (initialized) + return; + initialized = 1; + + git_config(promisor_remote_config, NULL); +} + +struct promisor_remote *promisor_remote_find(const char *remote_name) +{ + promisor_remote_init(); + + if (!remote_name) + return promisors; + + return promisor_remote_lookup(remote_name, NULL); +} + +int has_promisor_remote(void) +{ + return !!promisor_remote_find(NULL); +} diff --git a/promisor-remote.h b/promisor-remote.h new file mode 100644 index 0000000000..01dcdf4dc7 --- /dev/null +++ b/promisor-remote.h @@ -0,0 +1,16 @@ +#ifndef PROMISOR_REMOTE_H +#define PROMISOR_REMOTE_H + +/* + * Promisor remote linked list + * Its information come from remote.XXX config entries. + */ +struct promisor_remote { + struct promisor_remote *next; + const char name[FLEX_ARRAY]; +}; + +extern struct promisor_remote *promisor_remote_find(const char *remote_name); +extern int has_promisor_remote(void); + +#endif /* PROMISOR_REMOTE_H */ From patchwork Mon Apr 1 16:40:37 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christian Couder X-Patchwork-Id: 10880273 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 18D5F14DE for ; Mon, 1 Apr 2019 16:41:18 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 01E7528892 for ; Mon, 1 Apr 2019 16:41:18 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id E9E3228630; Mon, 1 Apr 2019 16:41:17 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 86AF828630 for ; Mon, 1 Apr 2019 16:41:17 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728645AbfDAQlQ (ORCPT ); Mon, 1 Apr 2019 12:41:16 -0400 Received: from mail-wm1-f65.google.com ([209.85.128.65]:51640 "EHLO mail-wm1-f65.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728589AbfDAQlP (ORCPT ); Mon, 1 Apr 2019 12:41:15 -0400 Received: by mail-wm1-f65.google.com with SMTP id 4so104932wmf.1 for ; Mon, 01 Apr 2019 09:41:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=XntUJc4k1Zp4N1xIrmXLeuG+ksVkNtIoVn5atpCh3yI=; b=YszJLOfJo2OeLtTtRQdr30EyU1z5l2k8dbTDsUvklnboVXCrQvg2gtSyxQno738YDz Bo1JnxVHA2SvyavYIXIzv5DoqvJdX249FKzY+6GHDvZyiG10pEhWb/mtNWqa5ilOwiy5 cQltrRPj1UWA+0pxuiGlhFxInB/nzxcK1XHxe1iHEM4GTS/P6LVMVMMago0sZhqcwt2C OTXBzsI5NZprG4E9STTd3KuSm21jznOh94BpBBDoE2kCyEAcnkAPgJUibg+7dDYy/pCu E40ZvNEAeoaXD5V3F1y8ppyVJ7Yw4APlhBNEcl1NSzDmxT839Y9f7o3k3AwuzPeVsGDc gYsQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=XntUJc4k1Zp4N1xIrmXLeuG+ksVkNtIoVn5atpCh3yI=; b=g57tjDaas7Rz4Fq/AnuxJ1vPKvXL0sBd/Ffywxp7bEed/YR4+ojo598OwzVkKM4YXz QNo+Bs12xn2ttKKsVRVBGkchtjbhC9hHiiZbIr0tZC2gFc0+a8m7mZiLt88GQkuIX988 PZdgiaVSsb9I2j2OZnvy3VFZLmjb2URZ8ghEw+kJCwmThTVF9Dlv59UVh2D/nTtGLD7B cmIy9nIgNsVBmrDI++N58XK3BUA5EVPv0CpKQ7BmnsDlSlId1mchfm1E2Um6QrwTfepx jWbZcCMJomGhFuhjXaDJAV6bh5k0bFIUEbc9YnIPKV2Nikm13Djx6mCXxc5RqH9Qoij1 vvzA== X-Gm-Message-State: APjAAAXe5u07yUGdC6iLgckh9fwDVcsF0VEKYne8wz+AqFaGsbliBOc8 9pHQZ/XDdmFfV96O+8F5sFOIIGWw X-Google-Smtp-Source: APXvYqzABbPqN3eRUoiYOBXSsNrWUfC3TiizAUmsL843zwreb7DiXDFebih4WZjsHBdDIRnV8M+rXw== X-Received: by 2002:a1c:ac07:: with SMTP id v7mr351077wme.49.1554136873474; Mon, 01 Apr 2019 09:41:13 -0700 (PDT) Received: from localhost.localdomain ([2a04:cec0:101d:3e80:8542:d1d4:667f:a0da]) by smtp.gmail.com with ESMTPSA id s10sm11749070wmh.0.2019.04.01.09.41.11 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 01 Apr 2019 09:41:12 -0700 (PDT) From: Christian Couder X-Google-Original-From: Christian Couder To: git@vger.kernel.org Cc: Junio C Hamano , Jeff King , Ben Peart , Jonathan Tan , Jonathan Nieder , Nguyen Thai Ngoc Duy , Mike Hommey , Lars Schneider , Eric Wong , Christian Couder , Jeff Hostetler , Eric Sunshine , Beat Bolli , Christian Couder Subject: [PATCH v4 03/11] promisor-remote: implement promisor_remote_get_direct() Date: Mon, 1 Apr 2019 18:40:37 +0200 Message-Id: <20190401164045.17328-4-chriscool@tuxfamily.org> X-Mailer: git-send-email 2.21.0.203.gd44fa53258 In-Reply-To: <20190401164045.17328-1-chriscool@tuxfamily.org> References: <20190401164045.17328-1-chriscool@tuxfamily.org> MIME-Version: 1.0 Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Christian Couder This is implemented for now by calling fetch_objects(). It fetches from all the promisor remotes. Signed-off-by: Christian Couder --- promisor-remote.c | 64 +++++++++++++++++++++++++++++++++++++++++++++++ promisor-remote.h | 1 + 2 files changed, 65 insertions(+) diff --git a/promisor-remote.c b/promisor-remote.c index 0c768210ee..a3d601e45a 100644 --- a/promisor-remote.c +++ b/promisor-remote.c @@ -1,6 +1,8 @@ #include "cache.h" +#include "object-store.h" #include "promisor-remote.h" #include "config.h" +#include "fetch-object.h" static struct promisor_remote *promisors; static struct promisor_remote **promisors_tail = &promisors; @@ -90,3 +92,65 @@ int has_promisor_remote(void) { return !!promisor_remote_find(NULL); } + +static int remove_fetched_oids(struct object_id **oids, int oid_nr, int to_free) +{ + int i, missing_nr = 0; + int *missing = xcalloc(oid_nr, sizeof(*missing)); + struct object_id *old_oids = *oids; + struct object_id *new_oids; + int old_fetch_if_missing = fetch_if_missing; + + fetch_if_missing = 0; + + for (i = 0; i < oid_nr; i++) + if (oid_object_info_extended(the_repository, &old_oids[i], NULL, 0)) { + missing[i] = 1; + missing_nr++; + } + + fetch_if_missing = old_fetch_if_missing; + + if (missing_nr) { + int j = 0; + new_oids = xcalloc(missing_nr, sizeof(*new_oids)); + for (i = 0; i < oid_nr; i++) + if (missing[i]) + oidcpy(&new_oids[j++], &old_oids[i]); + *oids = new_oids; + if (to_free) + free(old_oids); + } + + return missing_nr; +} + +int promisor_remote_get_direct(const struct object_id *oids, int oid_nr) +{ + struct promisor_remote *r; + struct object_id *missing_oids = (struct object_id *)oids; + int missing_nr = oid_nr; + int to_free = 0; + int res = -1; + + promisor_remote_init(); + + for (r = promisors; r; r = r->next) { + if (fetch_objects(r->name, missing_oids, missing_nr) < 0) { + if (missing_nr == 1) + continue; + missing_nr = remove_fetched_oids(&missing_oids, missing_nr, to_free); + if (missing_nr) { + to_free = 1; + continue; + } + } + res = 0; + break; + } + + if (to_free) + free(missing_oids); + + return res; +} diff --git a/promisor-remote.h b/promisor-remote.h index 01dcdf4dc7..30c0b3dcb6 100644 --- a/promisor-remote.h +++ b/promisor-remote.h @@ -12,5 +12,6 @@ struct promisor_remote { extern struct promisor_remote *promisor_remote_find(const char *remote_name); extern int has_promisor_remote(void); +extern int promisor_remote_get_direct(const struct object_id *oids, int oid_nr); #endif /* PROMISOR_REMOTE_H */ From patchwork Mon Apr 1 16:40:38 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christian Couder X-Patchwork-Id: 10880275 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 11762139A for ; Mon, 1 Apr 2019 16:41:20 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id F109028614 for ; Mon, 1 Apr 2019 16:41:19 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id EF0B5287C2; Mon, 1 Apr 2019 16:41:19 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9929F2888D for ; Mon, 1 Apr 2019 16:41:19 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728653AbfDAQlS (ORCPT ); Mon, 1 Apr 2019 12:41:18 -0400 Received: from mail-wm1-f66.google.com ([209.85.128.66]:50292 "EHLO mail-wm1-f66.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728589AbfDAQlS (ORCPT ); Mon, 1 Apr 2019 12:41:18 -0400 Received: by mail-wm1-f66.google.com with SMTP id z11so113618wmi.0 for ; Mon, 01 Apr 2019 09:41:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=lNgQHRwwRhSPGUvpTQ6SpDXPfZtJY7PSnNgcTAQb6Uk=; b=u1HGragltT2G5dfZVVtTFsAWO79WLzUun9VWxTEx38GPukaPGkYDpfqdqCZnXrKY4I Dg+AtxzbjUnkg/4OKgkMHggCn/u45NDkHvhWD7xr5zL16KYDkBs/QB4RSjTxdxA+vERL g1cjymZd12JIjL2byvfGnJoTLCrIjm2FHYvsv5WACLC8TRkfBNqalnPyWoZKBWv9py41 4ggLKGAFDXimWXrvQyVwFAaH/5GeaQf2Mx4Gvp1PTKlH37h4jTjDUb1KC0FTFYZQlcxP iVzAXj9u9lMpji7gJ4D+ejlM2ae5X2eg9TZjdE93pvejK+ilZluVG5imqIQQb2tlMAdv dgEA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=lNgQHRwwRhSPGUvpTQ6SpDXPfZtJY7PSnNgcTAQb6Uk=; b=AAS5Lb4XsRU/0hoiz+OGXn1Y4KAzzHWDJIEAOOceuGGiIL/pNwTxBMT6A3K511sclQ Adn7BmXX+mzddnwyEuhXWQaAKTcivD3OJKJnciVVWZiCszzyH2FYZNFaD7gypeRL04q3 EPFLRXZMq5xOXyy3NmTpjCVH+IU5xz5/Dxb/H9wY21Yl9Lf/gxPR8w6rMEMApoMWcWsT UawTPoJxmQi9kiAh5NP8H+two6L+h66iUVJIzAQd73XPaLfP19kcXqQfWZD4lXnybsXX 4rrWdAlD/gyO4I4wtuaXuNYxfPqAiR0z3+wuQ+4rM/lGQRMTZSjuXqMH/EQ0f0ayLgjv 3VZw== X-Gm-Message-State: APjAAAVJPgH71tgi6aQuntax7dB4QQUmKcu+J4XDkYT7/PA6IQ9jtS3O KVLrNJ0AjHa4UVx6RBeu8gCxMkm5 X-Google-Smtp-Source: APXvYqw1VeY5rvak8G7Lv5SinTYNwmg0GZIHV2HoPV6XtoT+Vo4OqhqFPTWVDWh7VQamPCb33PxyPg== X-Received: by 2002:a1c:988d:: with SMTP id a135mr378095wme.24.1554136876223; Mon, 01 Apr 2019 09:41:16 -0700 (PDT) Received: from localhost.localdomain ([2a04:cec0:101d:3e80:8542:d1d4:667f:a0da]) by smtp.gmail.com with ESMTPSA id s10sm11749070wmh.0.2019.04.01.09.41.13 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 01 Apr 2019 09:41:15 -0700 (PDT) From: Christian Couder X-Google-Original-From: Christian Couder To: git@vger.kernel.org Cc: Junio C Hamano , Jeff King , Ben Peart , Jonathan Tan , Jonathan Nieder , Nguyen Thai Ngoc Duy , Mike Hommey , Lars Schneider , Eric Wong , Christian Couder , Jeff Hostetler , Eric Sunshine , Beat Bolli , Christian Couder Subject: [PATCH v4 04/11] promisor-remote: add promisor_remote_reinit() Date: Mon, 1 Apr 2019 18:40:38 +0200 Message-Id: <20190401164045.17328-5-chriscool@tuxfamily.org> X-Mailer: git-send-email 2.21.0.203.gd44fa53258 In-Reply-To: <20190401164045.17328-1-chriscool@tuxfamily.org> References: <20190401164045.17328-1-chriscool@tuxfamily.org> MIME-Version: 1.0 Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Christian Couder We will need to reinitialize the promisor remote configuration as we will make some changes to it in a later commit. Signed-off-by: Christian Couder --- promisor-remote.c | 22 ++++++++++++++++++++-- promisor-remote.h | 1 + 2 files changed, 21 insertions(+), 2 deletions(-) diff --git a/promisor-remote.c b/promisor-remote.c index a3d601e45a..d1fcd1a8bf 100644 --- a/promisor-remote.c +++ b/promisor-remote.c @@ -67,10 +67,10 @@ static int promisor_remote_config(const char *var, const char *value, void *data return 0; } +static int initialized; + static void promisor_remote_init(void) { - static int initialized; - if (initialized) return; initialized = 1; @@ -78,6 +78,24 @@ static void promisor_remote_init(void) git_config(promisor_remote_config, NULL); } +static void promisor_remote_clear(void) +{ + while (promisors) { + struct promisor_remote *r = promisors; + promisors = promisors->next; + free(r); + } + + promisors_tail = &promisors; +} + +void promisor_remote_reinit(void) +{ + initialized = 0; + promisor_remote_clear(); + promisor_remote_init(); +} + struct promisor_remote *promisor_remote_find(const char *remote_name) { promisor_remote_init(); diff --git a/promisor-remote.h b/promisor-remote.h index 30c0b3dcb6..071dd6bbca 100644 --- a/promisor-remote.h +++ b/promisor-remote.h @@ -10,6 +10,7 @@ struct promisor_remote { const char name[FLEX_ARRAY]; }; +extern void promisor_remote_reinit(void); extern struct promisor_remote *promisor_remote_find(const char *remote_name); extern int has_promisor_remote(void); extern int promisor_remote_get_direct(const struct object_id *oids, int oid_nr); From patchwork Mon Apr 1 16:40:39 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christian Couder X-Patchwork-Id: 10880277 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 3132214DE for ; Mon, 1 Apr 2019 16:41:22 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 1C35D28680 for ; Mon, 1 Apr 2019 16:41:22 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 1A6C928689; Mon, 1 Apr 2019 16:41:22 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A4D3B28749 for ; Mon, 1 Apr 2019 16:41:21 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728665AbfDAQlU (ORCPT ); Mon, 1 Apr 2019 12:41:20 -0400 Received: from mail-wm1-f65.google.com ([209.85.128.65]:36389 "EHLO mail-wm1-f65.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728589AbfDAQlU (ORCPT ); Mon, 1 Apr 2019 12:41:20 -0400 Received: by mail-wm1-f65.google.com with SMTP id h18so122041wml.1 for ; Mon, 01 Apr 2019 09:41:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=xvVqrtz4KuRtT7e+5l9Em2GkoUPl0NAf+wMQJ1Lrje8=; b=D9AN4eE0SE5FAdewRJJ7nlOrNF9DJJ8SbXWc+1X0U+sMiWUjOU+N0qtoDjH5vSqj3z GgO624fwgwmd2nd4p+Pzc9DGTpy4+ztT6jdHoTnDTgSDTISCmTqdfkiPTXYvevbMdSpp fE40ekllZoDpPVT7pW6nhat+wyBFJA5qCBuK2RLjFIvHPDJdUX7/VMW5DV3ieaheVJW2 zdtZxgkQqKwdPH4Jx3JKCZIHCytnJjZFix0+YYxipF0oyvcCTWfpu1v/Xux+glMBM/dA fNR74Rh3fSdCIr31753zwNdbbCy6dIvaPVrZAuLMixVc9gMnCtTXCuWUf4ywSv71xTiK 5mNQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=xvVqrtz4KuRtT7e+5l9Em2GkoUPl0NAf+wMQJ1Lrje8=; b=YmaYuzkEom1tiaZGZnzsSrzenkZCSzp6o3aZLoVQJi/7ZTJQ6jGYv5ZjEp99/oHb9y w0zlx5F/Q4MeNbCDPZAhx8xlGUe1exbYKgu86xNZm62xTIv7sr3mZy3Mt3/RHnwLKseh AJVt5yAwBhMi745qEsPKKn00Avc0x+LsAFTHn5aSXpG7IH1LlsRUz2t0RSn0vIZOmRN4 YHNXXVHpmH+TZpUs4nLAuyiwYyDzpNrpe1IZezYEPMca7DbIRoSovWT1kl717x7C5UaD nE9o4djf3llNuC8C62fjOLRfconrHEI4v0mY1cQmKBez00daKNBRHdqGqwfdcdb2fkhc 6fyA== X-Gm-Message-State: APjAAAXCdD3WP/WLm6awK/t92ry3skXdo3TG+4zYHAL+7qJ8/RXhZxnY Yz/xMg/PJNd9YxsPc5ivm7BUlGUL X-Google-Smtp-Source: APXvYqxNN3+p1zuPH71iQezkx1nTn6rtYS29lTBx+VTgmszk1y/iKXKu5+Z+V80PARa/AsOuovvUow== X-Received: by 2002:a1c:c504:: with SMTP id v4mr331082wmf.45.1554136877949; Mon, 01 Apr 2019 09:41:17 -0700 (PDT) Received: from localhost.localdomain ([2a04:cec0:101d:3e80:8542:d1d4:667f:a0da]) by smtp.gmail.com with ESMTPSA id s10sm11749070wmh.0.2019.04.01.09.41.16 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 01 Apr 2019 09:41:17 -0700 (PDT) From: Christian Couder X-Google-Original-From: Christian Couder To: git@vger.kernel.org Cc: Junio C Hamano , Jeff King , Ben Peart , Jonathan Tan , Jonathan Nieder , Nguyen Thai Ngoc Duy , Mike Hommey , Lars Schneider , Eric Wong , Christian Couder , Jeff Hostetler , Eric Sunshine , Beat Bolli Subject: [PATCH v4 05/11] promisor-remote: use repository_format_partial_clone Date: Mon, 1 Apr 2019 18:40:39 +0200 Message-Id: <20190401164045.17328-6-chriscool@tuxfamily.org> X-Mailer: git-send-email 2.21.0.203.gd44fa53258 In-Reply-To: <20190401164045.17328-1-chriscool@tuxfamily.org> References: <20190401164045.17328-1-chriscool@tuxfamily.org> MIME-Version: 1.0 Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP A remote specified using the extensions.partialClone config option should be considered a promisor remote too. This remote should be at the end of the promisor remote list, so that it is used only if objects have not been found in other remotes. If we are using promisor remotes other than the origin, it is because these other promisor remotes are likely to be better for some reason (maybe they are closer or faster for some kind of objects) than the origin, so it's logical that we try to use them first. Signed-off-by: Christian Couder --- promisor-remote.c | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) diff --git a/promisor-remote.c b/promisor-remote.c index d1fcd1a8bf..6a689737f0 100644 --- a/promisor-remote.c +++ b/promisor-remote.c @@ -40,6 +40,18 @@ static struct promisor_remote *promisor_remote_lookup(const char *remote_name, return NULL; } +static void promisor_remote_move_to_tail(struct promisor_remote *r, + struct promisor_remote *previous) +{ + if (previous) + previous->next = r->next; + else + promisors = r->next ? r->next : r; + r->next = NULL; + *promisors_tail = r; + promisors_tail = &r->next; +} + static int promisor_remote_config(const char *var, const char *value, void *data) { const char *name; @@ -76,6 +88,17 @@ static void promisor_remote_init(void) initialized = 1; git_config(promisor_remote_config, NULL); + + if (repository_format_partial_clone) { + struct promisor_remote *o, *previous; + + o = promisor_remote_lookup(repository_format_partial_clone, + &previous); + if (o) + promisor_remote_move_to_tail(o, previous); + else + promisor_remote_new(repository_format_partial_clone); + } } static void promisor_remote_clear(void) From patchwork Mon Apr 1 16:40:40 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christian Couder X-Patchwork-Id: 10880279 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id C1CCF14DE for ; Mon, 1 Apr 2019 16:41:25 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id AB49D2863D for ; Mon, 1 Apr 2019 16:41:25 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id A5CBD2889B; Mon, 1 Apr 2019 16:41:25 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A1EAA28643 for ; Mon, 1 Apr 2019 16:41:24 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728682AbfDAQlX (ORCPT ); Mon, 1 Apr 2019 12:41:23 -0400 Received: from mail-wm1-f67.google.com ([209.85.128.67]:51655 "EHLO mail-wm1-f67.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728589AbfDAQlX (ORCPT ); Mon, 1 Apr 2019 12:41:23 -0400 Received: by mail-wm1-f67.google.com with SMTP id 4so105333wmf.1 for ; Mon, 01 Apr 2019 09:41:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Bt4kbFQbcfjz3iRMojlDI1GEww8sY5SEeR95ZQwXnQ8=; b=NZLFPnepF1GrUgFOXu6VH2OohZTtgsXKDM820SHnM9TwAaFpATREZ2uXI9yrq5ecZp sHyexWh5+sD8NTw3ftva0ZZlB7mYlNMvHUs0KepLCREyBrkIwtfFhgs5V7g86a/HAAdN iTPyN21xGSgfR6iZ87qzTy6wjZ5yMhnoHMiWGZxut+L921A19QxHCISPWwcsYM0BGa3j oaQxVXdz69pHDH1KHH/obzU0TrxrZ55nL9fftXWTelN80eNVoekbzVbiy97PuripU4BO sx9+hgMBmcDJ8uCuLVDkzLWbKNWtU5U3r1//KVgv+yJm8+oLT81FF049P1k7KowKEm9x iEOQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Bt4kbFQbcfjz3iRMojlDI1GEww8sY5SEeR95ZQwXnQ8=; b=AHMQZVHy9X5Q7W6XD+pOnuPnywcacQrGKIJXrI4YrgW4PjmHjGsOl/Y4TWxE3ZVQn+ aIGx0azMu1NcO2eJZt5jL53GA8ENb+nem4TcMlkQDLmENyQZH7ryu8XElGtMgFZJcIrO o6erkcgdyX3fnwQwNLKQFFk56b0RyJbccfAv2OptMRvIxASqc2x+rPPiWV/Ot8M5PdEg gbiz+hNZi3RX36TT8q0/tjqvMVUDMhGv5tOw63eHSohGF1QVRXWqeagCuQ3uYV8/9BGt q2R/94yPMFFnmgwoIJQF+WA/rpKLHKSrujQ5UIRzh2+yP6avNJMe1XohrYhNjcEeoZun Wh5Q== X-Gm-Message-State: APjAAAWA54Fhn7KxAe9RSaZaPh7YJhU+cOCfEtsxwfuuvHWvJsX5gaZL 7Pi4D69lqZUfzkDLx887moq5v38o X-Google-Smtp-Source: APXvYqxe12sBjPQOJUcagePOgP1QR6GP0Y44u4039jhBbxi8of8HT0+Amw7Y4NkVVQVJWq5B1rlvvw== X-Received: by 2002:a1c:ac07:: with SMTP id v7mr351468wme.49.1554136880353; Mon, 01 Apr 2019 09:41:20 -0700 (PDT) Received: from localhost.localdomain ([2a04:cec0:101d:3e80:8542:d1d4:667f:a0da]) by smtp.gmail.com with ESMTPSA id s10sm11749070wmh.0.2019.04.01.09.41.18 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 01 Apr 2019 09:41:19 -0700 (PDT) From: Christian Couder X-Google-Original-From: Christian Couder To: git@vger.kernel.org Cc: Junio C Hamano , Jeff King , Ben Peart , Jonathan Tan , Jonathan Nieder , Nguyen Thai Ngoc Duy , Mike Hommey , Lars Schneider , Eric Wong , Christian Couder , Jeff Hostetler , Eric Sunshine , Beat Bolli Subject: [PATCH v4 06/11] Use promisor_remote_get_direct() and has_promisor_remote() Date: Mon, 1 Apr 2019 18:40:40 +0200 Message-Id: <20190401164045.17328-7-chriscool@tuxfamily.org> X-Mailer: git-send-email 2.21.0.203.gd44fa53258 In-Reply-To: <20190401164045.17328-1-chriscool@tuxfamily.org> References: <20190401164045.17328-1-chriscool@tuxfamily.org> MIME-Version: 1.0 Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Instead of using the repository_format_partial_clone global and fetch_objects() directly, let's use has_promisor_remote() and promisor_remote_get_direct(). This way all the configured promisor remotes will be taken into account, not only the one specified by extensions.partialClone. Also when cloning or fetching using a partial clone filter, remote.origin.promisor will be set to "true" instead of setting extensions.partialClone to "origin". This makes it possible to use many promisor remote just by fetching from them. Signed-off-by: Christian Couder --- builtin/cat-file.c | 5 +++-- builtin/fetch.c | 11 ++++++----- builtin/gc.c | 3 ++- builtin/repack.c | 3 ++- cache-tree.c | 3 ++- connected.c | 3 ++- list-objects-filter-options.c | 28 +++++++++++++++------------- packfile.c | 3 ++- sha1-file.c | 14 ++++++++------ t/t5601-clone.sh | 2 +- t/t5616-partial-clone.sh | 2 +- unpack-trees.c | 6 +++--- 12 files changed, 47 insertions(+), 36 deletions(-) diff --git a/builtin/cat-file.c b/builtin/cat-file.c index 0f092382e1..85ae10bf0b 100644 --- a/builtin/cat-file.c +++ b/builtin/cat-file.c @@ -15,6 +15,7 @@ #include "sha1-array.h" #include "packfile.h" #include "object-store.h" +#include "promisor-remote.h" struct batch_options { int enabled; @@ -523,8 +524,8 @@ static int batch_objects(struct batch_options *opt) if (opt->all_objects) { struct object_cb_data cb; - if (repository_format_partial_clone) - warning("This repository has extensions.partialClone set. Some objects may not be loaded."); + if (has_promisor_remote()) + warning("This repository uses promisor remotes. Some objects may not be loaded."); cb.opt = opt; cb.expand = &data; diff --git a/builtin/fetch.c b/builtin/fetch.c index 4ba63d5ac6..f74bd78144 100644 --- a/builtin/fetch.c +++ b/builtin/fetch.c @@ -23,6 +23,7 @@ #include "packfile.h" #include "list-objects-filter-options.h" #include "commit-reach.h" +#include "promisor-remote.h" static const char * const builtin_fetch_usage[] = { N_("git fetch [] [ [...]]"), @@ -1460,7 +1461,7 @@ static inline void fetch_one_setup_partial(struct remote *remote) * If no prior partial clone/fetch and the current fetch DID NOT * request a partial-fetch, do a normal fetch. */ - if (!repository_format_partial_clone && !filter_options.choice) + if (!has_promisor_remote() && !filter_options.choice) return; /* @@ -1468,7 +1469,7 @@ static inline void fetch_one_setup_partial(struct remote *remote) * on this repo and remember the given filter-spec as the default * for subsequent fetches to this remote. */ - if (!repository_format_partial_clone && filter_options.choice) { + if (!has_promisor_remote() && filter_options.choice) { partial_clone_register(remote->name, &filter_options); return; } @@ -1477,7 +1478,7 @@ static inline void fetch_one_setup_partial(struct remote *remote) * We are currently limited to only ONE promisor remote and only * allow partial-fetches from the promisor remote. */ - if (strcmp(remote->name, repository_format_partial_clone)) { + if (!promisor_remote_find(remote->name)) { if (filter_options.choice) die(_("--filter can only be used with the remote " "configured in extensions.partialClone")); @@ -1611,7 +1612,7 @@ int cmd_fetch(int argc, const char **argv, const char *prefix) if (depth || deepen_since || deepen_not.nr) deepen = 1; - if (filter_options.choice && !repository_format_partial_clone) + if (filter_options.choice && !has_promisor_remote()) die("--filter can only be used when extensions.partialClone is set"); if (all) { @@ -1645,7 +1646,7 @@ int cmd_fetch(int argc, const char **argv, const char *prefix) } if (remote) { - if (filter_options.choice || repository_format_partial_clone) + if (filter_options.choice || has_promisor_remote()) fetch_one_setup_partial(remote); result = fetch_one(remote, argc, argv, prune_tags_ok); } else { diff --git a/builtin/gc.c b/builtin/gc.c index 020f725acc..0bec41b25f 100644 --- a/builtin/gc.c +++ b/builtin/gc.c @@ -27,6 +27,7 @@ #include "pack-objects.h" #include "blob.h" #include "tree.h" +#include "promisor-remote.h" #define FAILED_RUN "failed to run %s" @@ -640,7 +641,7 @@ int cmd_gc(int argc, const char **argv, const char *prefix) argv_array_push(&prune, prune_expire); if (quiet) argv_array_push(&prune, "--no-progress"); - if (repository_format_partial_clone) + if (has_promisor_remote()) argv_array_push(&prune, "--exclude-promisor-objects"); if (run_command_v_opt(prune.argv, RUN_GIT_CMD)) diff --git a/builtin/repack.c b/builtin/repack.c index 67f8978043..3b935690c8 100644 --- a/builtin/repack.c +++ b/builtin/repack.c @@ -11,6 +11,7 @@ #include "midx.h" #include "packfile.h" #include "object-store.h" +#include "promisor-remote.h" static int delta_base_offset = 1; static int pack_kept_objects = -1; @@ -366,7 +367,7 @@ int cmd_repack(int argc, const char **argv, const char *prefix) argv_array_push(&cmd.args, "--all"); argv_array_push(&cmd.args, "--reflog"); argv_array_push(&cmd.args, "--indexed-objects"); - if (repository_format_partial_clone) + if (has_promisor_remote()) argv_array_push(&cmd.args, "--exclude-promisor-objects"); if (write_bitmaps) argv_array_push(&cmd.args, "--write-bitmap-index"); diff --git a/cache-tree.c b/cache-tree.c index b13bfaf71e..64c285a746 100644 --- a/cache-tree.c +++ b/cache-tree.c @@ -5,6 +5,7 @@ #include "cache-tree.h" #include "object-store.h" #include "replace-object.h" +#include "promisor-remote.h" #ifndef DEBUG #define DEBUG 0 @@ -357,7 +358,7 @@ static int update_one(struct cache_tree *it, } ce_missing_ok = mode == S_IFGITLINK || missing_ok || - (repository_format_partial_clone && + (has_promisor_remote() && ce_skip_worktree(ce)); if (is_null_oid(oid) || (!ce_missing_ok && !has_object_file(oid))) { diff --git a/connected.c b/connected.c index 1bba888eff..0eaaedee6a 100644 --- a/connected.c +++ b/connected.c @@ -4,6 +4,7 @@ #include "connected.h" #include "transport.h" #include "packfile.h" +#include "promisor-remote.h" /* * If we feed all the commits we want to verify to this command @@ -56,7 +57,7 @@ int check_connected(oid_iterate_fn fn, void *cb_data, argv_array_push(&rev_list.args,"rev-list"); argv_array_push(&rev_list.args, "--objects"); argv_array_push(&rev_list.args, "--stdin"); - if (repository_format_partial_clone) + if (has_promisor_remote()) argv_array_push(&rev_list.args, "--exclude-promisor-objects"); if (!opt->is_deepening_fetch) { argv_array_push(&rev_list.args, "--not"); diff --git a/list-objects-filter-options.c b/list-objects-filter-options.c index c0036f7378..f41a831fce 100644 --- a/list-objects-filter-options.c +++ b/list-objects-filter-options.c @@ -6,6 +6,7 @@ #include "list-objects.h" #include "list-objects-filter.h" #include "list-objects-filter-options.h" +#include "promisor-remote.h" /* * Parse value of the argument to the "filter" keyword. @@ -144,30 +145,31 @@ void partial_clone_register( const char *remote, const struct list_objects_filter_options *filter_options) { - /* - * Record the name of the partial clone remote in the - * config and in the global variable -- the latter is - * used throughout to indicate that partial clone is - * enabled and to expect missing objects. - */ - if (repository_format_partial_clone && - *repository_format_partial_clone && - strcmp(remote, repository_format_partial_clone)) - die(_("cannot change partial clone promisor remote")); + char *cfg_name; - git_config_set("core.repositoryformatversion", "1"); - git_config_set("extensions.partialclone", remote); + /* Check if it is already registered */ + if (!promisor_remote_find(remote)) { + git_config_set("core.repositoryformatversion", "1"); - repository_format_partial_clone = xstrdup(remote); + /* Add promisor config for the remote */ + cfg_name = xstrfmt("remote.%s.promisor", remote); + git_config_set(cfg_name, "true"); + free(cfg_name); + } /* * Record the initial filter-spec in the config as * the default for subsequent fetches from this remote. + * + * TODO: record it into remote..partialclonefilter */ core_partial_clone_filter_default = xstrdup(filter_options->filter_spec); git_config_set("core.partialclonefilter", core_partial_clone_filter_default); + + /* Make sure the config info are reset */ + promisor_remote_reinit(); } void partial_clone_get_default_filter_spec( diff --git a/packfile.c b/packfile.c index 16bcb75262..0f95cdc1c9 100644 --- a/packfile.c +++ b/packfile.c @@ -16,6 +16,7 @@ #include "tree.h" #include "object-store.h" #include "midx.h" +#include "promisor-remote.h" char *odb_pack_name(struct strbuf *buf, const unsigned char *sha1, @@ -2119,7 +2120,7 @@ int is_promisor_object(const struct object_id *oid) static int promisor_objects_prepared; if (!promisor_objects_prepared) { - if (repository_format_partial_clone) { + if (has_promisor_remote()) { for_each_packed_object(add_promisor_object, &promisor_objects, FOR_EACH_OBJECT_PROMISOR_ONLY); diff --git a/sha1-file.c b/sha1-file.c index 01cc0590f4..715a2b882a 100644 --- a/sha1-file.c +++ b/sha1-file.c @@ -32,6 +32,7 @@ #include "packfile.h" #include "fetch-object.h" #include "object-store.h" +#include "promisor-remote.h" /* The maximum size for an object header. */ #define MAX_HEADER_LEN 32 @@ -1369,15 +1370,16 @@ int oid_object_info_extended(struct repository *r, const struct object_id *oid, } /* Check if it is a missing object */ - if (fetch_if_missing && repository_format_partial_clone && + if (fetch_if_missing && has_promisor_remote() && !already_retried && r == the_repository) { /* - * TODO Investigate checking fetch_object() return - * TODO value and stopping on error here. - * TODO Pass a repository struct through fetch_object, - * such that arbitrary repositories work. + * TODO Investigate checking promisor_remote_get_direct() + * TODO return value and stopping on error here. + * TODO Pass a repository struct through + * promisor_remote_get_direct(), such that arbitrary + * repositories work. */ - fetch_objects(repository_format_partial_clone, real, 1); + promisor_remote_get_direct(real, 1); already_retried = 1; continue; } diff --git a/t/t5601-clone.sh b/t/t5601-clone.sh index d6948cbdab..dd658f8b32 100755 --- a/t/t5601-clone.sh +++ b/t/t5601-clone.sh @@ -653,7 +653,7 @@ partial_clone () { git -C client fsck && # Ensure that unneeded blobs are not inadvertently fetched. - test_config -C client extensions.partialclone "not a remote" && + test_config -C client remote.origin.promisor "false" && test_must_fail git -C client cat-file -e "$HASH1" && # But this blob was fetched, because clone performs an initial checkout diff --git a/t/t5616-partial-clone.sh b/t/t5616-partial-clone.sh index 9643acb161..e8ca825ab7 100755 --- a/t/t5616-partial-clone.sh +++ b/t/t5616-partial-clone.sh @@ -42,7 +42,7 @@ test_expect_success 'do partial clone 1' ' test_cmp expect_1.oids observed.oids && test "$(git -C pc1 config --local core.repositoryformatversion)" = "1" && - test "$(git -C pc1 config --local extensions.partialclone)" = "origin" && + test "$(git -C pc1 config --local remote.origin.promisor)" = "true" && test "$(git -C pc1 config --local core.partialclonefilter)" = "blob:none" ' diff --git a/unpack-trees.c b/unpack-trees.c index 22c41a3ba8..47353d85c3 100644 --- a/unpack-trees.c +++ b/unpack-trees.c @@ -17,6 +17,7 @@ #include "fsmonitor.h" #include "object-store.h" #include "fetch-object.h" +#include "promisor-remote.h" /* * Error messages expected by scripts out of plumbing commands such as @@ -398,7 +399,7 @@ static int check_updates(struct unpack_trees_options *o) load_gitmodules_file(index, &state); enable_delayed_checkout(&state); - if (repository_format_partial_clone && o->update && !o->dry_run) { + if (has_promisor_remote() && o->update && !o->dry_run) { /* * Prefetch the objects that are to be checked out in the loop * below. @@ -415,8 +416,7 @@ static int check_updates(struct unpack_trees_options *o) } } if (to_fetch.nr) - fetch_objects(repository_format_partial_clone, - to_fetch.oid, to_fetch.nr); + promisor_remote_get_direct(to_fetch.oid, to_fetch.nr); fetch_if_missing = fetch_if_missing_store; oid_array_clear(&to_fetch); } From patchwork Mon Apr 1 16:40:41 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christian Couder X-Patchwork-Id: 10880281 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id AF91B139A for ; Mon, 1 Apr 2019 16:41:27 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9B7E42861D for ; Mon, 1 Apr 2019 16:41:27 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 99C5328680; Mon, 1 Apr 2019 16:41:27 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id F2232286BF for ; Mon, 1 Apr 2019 16:41:26 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728697AbfDAQlZ (ORCPT ); Mon, 1 Apr 2019 12:41:25 -0400 Received: from mail-wr1-f66.google.com ([209.85.221.66]:41910 "EHLO mail-wr1-f66.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728589AbfDAQlZ (ORCPT ); Mon, 1 Apr 2019 12:41:25 -0400 Received: by mail-wr1-f66.google.com with SMTP id r4so12925427wrq.8 for ; Mon, 01 Apr 2019 09:41:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=cRvaOHZrhq0jAA/oA15gHJuClvo2KX/XJJUNX8gMdB8=; b=LnFnt93jo/tVGlfa+YoNG+Or3x3jZyFkrocPLr0jrC1UXAFOpn/Kz8vDEc/Z4zjaXR XXY7a+wJVIKM29I6kf3JqoIP1xzz5bNA15VOqtjWEaQXowLAlOpG0mgXmLp7b6QQT5QK zNpqpmheAEDEGw/DhHZFQLKwOFYNTrjy0qoSfRnvbs+74nsl4PlsR0tY/z8+Vzf+DU2G x8UIpGPI/+fq73yyN64D4ZfDvXl6lU8cRg2fGUoB82IyJ42ZsbEVelH//ots582NE1iW K+0l6DzKYbDSFbAFgHNBgWm4TlJ3KcgZOfQ7WE0hvUzhxAEWr8TUzVOOUZe29KRJeknC d6tw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=cRvaOHZrhq0jAA/oA15gHJuClvo2KX/XJJUNX8gMdB8=; b=hs+uiHQ+Iq/dUCfcx5G3xXmXeqIBX/sLSxO63e2gpO6qJiItUiRVP3DAEMtfVI+NGb zELRS0F5+Bg9ZV1BgD46OF9mOk5hft8/RQGOWAVoc8UA6bSdFgnEzq9pJYrwltRwKRE8 d1Ms9UT5qza2xskNFk4Hd65uW6beL94bFT3qTvNGDSwcVxbT8cfGvG+k2LaJcj2slors eishuUvSMES/BnAXnOYo/C10asqYccViRv7PZiMc+jYMWTSY/patgpUw9MqUlqmbDSgv nLDnGth+mq8D1ZhtccDKuKNTi7VcTVnrUnJNMCSlUmOe/JSIUS5XX6itKnruq9m1RNL3 27kA== X-Gm-Message-State: APjAAAUQ6O1NMzOUq0lqVq+0qB8sBJjWzFrbDBp23Ru6mx+hXXaVUthe LQUK6/NLhZ7iz8MIQnspYXBPI3uV X-Google-Smtp-Source: APXvYqzRcAZLUt4D+WNlqKDJqjHyeaShxjrPUGFQnPF/5WdzFUuwn60vj9yI7wFCi4W6fK+ALzgwfw== X-Received: by 2002:a5d:6a89:: with SMTP id s9mr43852178wru.58.1554136882850; Mon, 01 Apr 2019 09:41:22 -0700 (PDT) Received: from localhost.localdomain ([2a04:cec0:101d:3e80:8542:d1d4:667f:a0da]) by smtp.gmail.com with ESMTPSA id s10sm11749070wmh.0.2019.04.01.09.41.20 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 01 Apr 2019 09:41:22 -0700 (PDT) From: Christian Couder X-Google-Original-From: Christian Couder To: git@vger.kernel.org Cc: Junio C Hamano , Jeff King , Ben Peart , Jonathan Tan , Jonathan Nieder , Nguyen Thai Ngoc Duy , Mike Hommey , Lars Schneider , Eric Wong , Christian Couder , Jeff Hostetler , Eric Sunshine , Beat Bolli Subject: [PATCH v4 07/11] promisor-remote: parse remote.*.partialclonefilter Date: Mon, 1 Apr 2019 18:40:41 +0200 Message-Id: <20190401164045.17328-8-chriscool@tuxfamily.org> X-Mailer: git-send-email 2.21.0.203.gd44fa53258 In-Reply-To: <20190401164045.17328-1-chriscool@tuxfamily.org> References: <20190401164045.17328-1-chriscool@tuxfamily.org> MIME-Version: 1.0 Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP This makes it possible to specify a different partial clone filter for each promisor remote. Signed-off-by: Christian Couder --- builtin/fetch.c | 2 +- list-objects-filter-options.c | 27 +++++++++++++++------------ list-objects-filter-options.h | 3 ++- promisor-remote.c | 15 +++++++++++++++ promisor-remote.h | 5 ++++- t/t0410-partial-clone.sh | 2 +- t/t5601-clone.sh | 1 + t/t5616-partial-clone.sh | 2 +- 8 files changed, 40 insertions(+), 17 deletions(-) diff --git a/builtin/fetch.c b/builtin/fetch.c index f74bd78144..13d8133130 100644 --- a/builtin/fetch.c +++ b/builtin/fetch.c @@ -1491,7 +1491,7 @@ static inline void fetch_one_setup_partial(struct remote *remote) * the config. */ if (!filter_options.choice) - partial_clone_get_default_filter_spec(&filter_options); + partial_clone_get_default_filter_spec(&filter_options, remote->name); return; } diff --git a/list-objects-filter-options.c b/list-objects-filter-options.c index f41a831fce..02f48b7c40 100644 --- a/list-objects-filter-options.c +++ b/list-objects-filter-options.c @@ -30,6 +30,9 @@ static int gently_parse_list_objects_filter( { const char *v0; + if (!arg) + return 0; + if (filter_options->choice) { if (errbuf) { strbuf_addstr( @@ -146,6 +149,7 @@ void partial_clone_register( const struct list_objects_filter_options *filter_options) { char *cfg_name; + char *filter_name; /* Check if it is already registered */ if (!promisor_remote_find(remote)) { @@ -160,27 +164,26 @@ void partial_clone_register( /* * Record the initial filter-spec in the config as * the default for subsequent fetches from this remote. - * - * TODO: record it into remote..partialclonefilter */ - core_partial_clone_filter_default = - xstrdup(filter_options->filter_spec); - git_config_set("core.partialclonefilter", - core_partial_clone_filter_default); + filter_name = xstrfmt("remote.%s.partialclonefilter", remote); + git_config_set(filter_name, filter_options->filter_spec); + free(filter_name); /* Make sure the config info are reset */ promisor_remote_reinit(); } void partial_clone_get_default_filter_spec( - struct list_objects_filter_options *filter_options) + struct list_objects_filter_options *filter_options, + const char *remote) { + struct promisor_remote *promisor = promisor_remote_find(remote); + /* * Parse default value, but silently ignore it if it is invalid. */ - if (!core_partial_clone_filter_default) - return; - gently_parse_list_objects_filter(filter_options, - core_partial_clone_filter_default, - NULL); + if (promisor) + gently_parse_list_objects_filter(filter_options, + promisor->partial_clone_filter, + NULL); } diff --git a/list-objects-filter-options.h b/list-objects-filter-options.h index e3adc78ebf..70d27f44ef 100644 --- a/list-objects-filter-options.h +++ b/list-objects-filter-options.h @@ -89,6 +89,7 @@ void partial_clone_register( const char *remote, const struct list_objects_filter_options *filter_options); void partial_clone_get_default_filter_spec( - struct list_objects_filter_options *filter_options); + struct list_objects_filter_options *filter_options, + const char *remote); #endif /* LIST_OBJECTS_FILTER_OPTIONS_H */ diff --git a/promisor-remote.c b/promisor-remote.c index 6a689737f0..dbbd1853ee 100644 --- a/promisor-remote.c +++ b/promisor-remote.c @@ -75,6 +75,21 @@ static int promisor_remote_config(const char *var, const char *value, void *data free(remote_name); return 0; } + if (!strcmp(subkey, "partialclonefilter")) { + struct promisor_remote *r; + char *remote_name = xmemdupz(name, namelen); + + r = promisor_remote_lookup(remote_name, NULL); + if (!r) + r = promisor_remote_new(remote_name); + + free(remote_name); + + if (!r) + return 0; + + return git_config_string(&r->partial_clone_filter, var, value); + } return 0; } diff --git a/promisor-remote.h b/promisor-remote.h index 071dd6bbca..e29cf507ab 100644 --- a/promisor-remote.h +++ b/promisor-remote.h @@ -3,10 +3,13 @@ /* * Promisor remote linked list - * Its information come from remote.XXX config entries. + * + * Information in its fields come from remote.XXX config entries or + * from extensions.partialclone or core.partialclonefilter. */ struct promisor_remote { struct promisor_remote *next; + const char *partial_clone_filter; const char name[FLEX_ARRAY]; }; diff --git a/t/t0410-partial-clone.sh b/t/t0410-partial-clone.sh index bce02788e6..9266037714 100755 --- a/t/t0410-partial-clone.sh +++ b/t/t0410-partial-clone.sh @@ -26,7 +26,7 @@ promise_and_delete () { test_expect_success 'extensions.partialclone without filter' ' test_create_repo server && git clone --filter="blob:none" "file://$(pwd)/server" client && - git -C client config --unset core.partialclonefilter && + git -C client config --unset remote.origin.partialclonefilter && git -C client fetch origin ' diff --git a/t/t5601-clone.sh b/t/t5601-clone.sh index dd658f8b32..2e82c70f33 100755 --- a/t/t5601-clone.sh +++ b/t/t5601-clone.sh @@ -654,6 +654,7 @@ partial_clone () { # Ensure that unneeded blobs are not inadvertently fetched. test_config -C client remote.origin.promisor "false" && + git -C client config --unset remote.origin.partialclonefilter && test_must_fail git -C client cat-file -e "$HASH1" && # But this blob was fetched, because clone performs an initial checkout diff --git a/t/t5616-partial-clone.sh b/t/t5616-partial-clone.sh index e8ca825ab7..ba0489b41a 100755 --- a/t/t5616-partial-clone.sh +++ b/t/t5616-partial-clone.sh @@ -43,7 +43,7 @@ test_expect_success 'do partial clone 1' ' test_cmp expect_1.oids observed.oids && test "$(git -C pc1 config --local core.repositoryformatversion)" = "1" && test "$(git -C pc1 config --local remote.origin.promisor)" = "true" && - test "$(git -C pc1 config --local core.partialclonefilter)" = "blob:none" + test "$(git -C pc1 config --local remote.origin.partialclonefilter)" = "blob:none" ' # checkout master to force dynamic object fetch of blobs at HEAD. From patchwork Mon Apr 1 16:40:42 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christian Couder X-Patchwork-Id: 10880283 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 74CA414DE for ; Mon, 1 Apr 2019 16:41:28 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 60EF728643 for ; Mon, 1 Apr 2019 16:41:28 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 5F33B28680; Mon, 1 Apr 2019 16:41:28 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0A9AC286BF for ; Mon, 1 Apr 2019 16:41:28 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728701AbfDAQl0 (ORCPT ); Mon, 1 Apr 2019 12:41:26 -0400 Received: from mail-wr1-f66.google.com ([209.85.221.66]:33163 "EHLO mail-wr1-f66.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728684AbfDAQl0 (ORCPT ); Mon, 1 Apr 2019 12:41:26 -0400 Received: by mail-wr1-f66.google.com with SMTP id q1so12949493wrp.0 for ; Mon, 01 Apr 2019 09:41:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=1bZvtaGY+8FOPSRVBeNMEfbpzbaMtxjtD9btOaHkWdc=; b=mmVDlpAcZKubxm4KWFqXTo9YnBEEALlVcrgGlCRL5qgC+py8oB2GgXGlcXSh/6IiAC ZzlvBsrjGGc4elbBvP0P0a8ZT0kV0L1EGKSsyeBncZ+Btv2K9S+HE1xxBke0YFR5MEzM TUTy/gSUFkAM7z0zW1ZtxrxIZq8p5F+nxFW8UjCQ5a1ij3pJASGnbgQ29I8Ftp3nTJhb 3apMOThvQRlfxmDF2ExdZO6RVcdADgy/H1Y6ODki2uKfDGJQ0aT3q+uc0J/nBXpjRYY5 7LWajhYuytT4MiinlTd+sTxMmu4gwq0Y74n7F5JV2eqaRre6VA7XYSNIITV6bpWxxRyW /geQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=1bZvtaGY+8FOPSRVBeNMEfbpzbaMtxjtD9btOaHkWdc=; b=ZhzRd0BOj5UOpVaHlXoGrQitenWdWhlteZ+SorYK2oWpOzq+SdInT0Bv2pMF1UCDF6 Yx/uL9isl0X8ZeHbYP5nSiEcfzz0BqmB/C1D6erLCyxeg0LHq81+wim52AeBDLR+VD7g 9bP/XhoBvy31n/ogCozG/dGd6kE9VHQs0jN0VqRWvWrAlfP1uWMJBIar0x8rDMGHInPt GTwfRUe+6jEOfBxSbhrMMiXlwfggXTibxV0NDV+y/886mDkclhMSQ0H5gRQymGc1ncao kCiAEuQNhz31uEbeUnvZ+yXZWb4jc0/EiowYyp/eDrjy7M/z5zwahSsZHNM29CfLM/xQ Tn+A== X-Gm-Message-State: APjAAAWFG9DI840K4DrKarR8aA+tJXtFGoPHbRMOsjUA5q4iwjMDVwX7 wXcRjygbycfaghTTu0DUYp98D87i X-Google-Smtp-Source: APXvYqxW8Z+w78oumIGbvAWpqE1dGKnePa4sXDHtFAhscWX30CIQ0L5CaZij4rjygCIzQj5874sF6Q== X-Received: by 2002:adf:f344:: with SMTP id e4mr40074433wrp.77.1554136884548; Mon, 01 Apr 2019 09:41:24 -0700 (PDT) Received: from localhost.localdomain ([2a04:cec0:101d:3e80:8542:d1d4:667f:a0da]) by smtp.gmail.com with ESMTPSA id s10sm11749070wmh.0.2019.04.01.09.41.22 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 01 Apr 2019 09:41:23 -0700 (PDT) From: Christian Couder X-Google-Original-From: Christian Couder To: git@vger.kernel.org Cc: Junio C Hamano , Jeff King , Ben Peart , Jonathan Tan , Jonathan Nieder , Nguyen Thai Ngoc Duy , Mike Hommey , Lars Schneider , Eric Wong , Christian Couder , Jeff Hostetler , Eric Sunshine , Beat Bolli Subject: [PATCH v4 08/11] builtin/fetch: remove unique promisor remote limitation Date: Mon, 1 Apr 2019 18:40:42 +0200 Message-Id: <20190401164045.17328-9-chriscool@tuxfamily.org> X-Mailer: git-send-email 2.21.0.203.gd44fa53258 In-Reply-To: <20190401164045.17328-1-chriscool@tuxfamily.org> References: <20190401164045.17328-1-chriscool@tuxfamily.org> MIME-Version: 1.0 Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP As the infrastructure for more than one promisor remote has been introduced in previous patches, we can remove code that forbids the registration of more than one promisor remote. Signed-off-by: Christian Couder --- builtin/fetch.c | 20 +++++--------------- 1 file changed, 5 insertions(+), 15 deletions(-) diff --git a/builtin/fetch.c b/builtin/fetch.c index 13d8133130..5657d054ec 100644 --- a/builtin/fetch.c +++ b/builtin/fetch.c @@ -1465,26 +1465,16 @@ static inline void fetch_one_setup_partial(struct remote *remote) return; /* - * If this is the FIRST partial-fetch request, we enable partial - * on this repo and remember the given filter-spec as the default - * for subsequent fetches to this remote. + * If this is a partial-fetch request, we enable partial on + * this repo if not already enabled and remember the given + * filter-spec as the default for subsequent fetches to this + * remote. */ - if (!has_promisor_remote() && filter_options.choice) { + if (filter_options.choice) { partial_clone_register(remote->name, &filter_options); return; } - /* - * We are currently limited to only ONE promisor remote and only - * allow partial-fetches from the promisor remote. - */ - if (!promisor_remote_find(remote->name)) { - if (filter_options.choice) - die(_("--filter can only be used with the remote " - "configured in extensions.partialClone")); - return; - } - /* * Do a partial-fetch from the promisor remote using either the * explicitly given filter-spec or inherit the filter-spec from From patchwork Mon Apr 1 16:40:43 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christian Couder X-Patchwork-Id: 10880285 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 17EB714DE for ; Mon, 1 Apr 2019 16:41:32 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 021D52863D for ; Mon, 1 Apr 2019 16:41:32 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id E9D9428769; Mon, 1 Apr 2019 16:41:31 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7F57E28681 for ; Mon, 1 Apr 2019 16:41:31 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728710AbfDAQla (ORCPT ); Mon, 1 Apr 2019 12:41:30 -0400 Received: from mail-wm1-f66.google.com ([209.85.128.66]:55498 "EHLO mail-wm1-f66.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728589AbfDAQla (ORCPT ); Mon, 1 Apr 2019 12:41:30 -0400 Received: by mail-wm1-f66.google.com with SMTP id o25so76610wmf.5 for ; Mon, 01 Apr 2019 09:41:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=SwY4YQ9CmNMFl8SEZH8ZEdg5yuh6M1j0CxuCRDDi3SQ=; b=XHYdlNOrKMI1jqbV8CKiv7kY+dUdBi0uPSGC3C+VIp4k4toA8h7k/pJu2wkXu9w8O0 rpSyMDan2O22JAvXcRhFLDn4mdkYcoxdX+eRX/mA66UaNSQVwm7cyvIHCB4Ox5NhsYIF mjW2uAzBQwco5AD0xXYMMLQMT+aa4ub0huFrRg0fnMDnEplEfEoHdjijf5lfWvP/qa4m upDixf4gLUaj3iHz/a5nshuHE1Qp4wW3TSXePEMQyvvVxYXrntHEh3yK0Xqj4RUh5IuH UL7kWGaPXskP4ghHCFS1YsVx1z3+bkF3mwEUDPvmGIutjc3Ck778KfvLvrnEPMoDfJwC F4nA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=SwY4YQ9CmNMFl8SEZH8ZEdg5yuh6M1j0CxuCRDDi3SQ=; b=JDDoDZmZuZmbzhUt9z6Qpk8O9tjXJJIq1B36q+/iD80qgTUNR54Li5EfpbOKmzHC0C pJzs/qZmQTf1FEEJiPiYUWb52Y3b6MmwwW8Tg0c/IkKXwGQNHmBAJSvlkjou51fXi08L VAuTFF9pPsgTJaMVfbvw8HAzxtvUtx9Xbr3t2kByEdk/rNs8QsZdPmagPGoUjbtD3oDc omX/n2fYnbGfTCwVbjDwBZEodJ0jyqdJuu5ClLzAa9GTYJsLL5DM6eB3/6ZJgx9qovZ2 GPolyB530dmAGqxYW/gHZtvufhZv46vRNzWhQgO0deRrmvKlkF3IP2UFXoyP4EjjoERq USbQ== X-Gm-Message-State: APjAAAWf7YsseokEcbsidai2EE0hHNT4PffUi6XPMzqIRwdXOy2yuOTf YnS93GnbocEtAlZihkfbAdyE41kp X-Google-Smtp-Source: APXvYqyINGwJEjAbn7rR/jmfJrqV5j2NlYqVfYMF60A4lArU4r3o8RydLwd1FU6yqYGHIj/RRuKzLw== X-Received: by 2002:a7b:c923:: with SMTP id h3mr354130wml.34.1554136887640; Mon, 01 Apr 2019 09:41:27 -0700 (PDT) Received: from localhost.localdomain ([2a04:cec0:101d:3e80:8542:d1d4:667f:a0da]) by smtp.gmail.com with ESMTPSA id s10sm11749070wmh.0.2019.04.01.09.41.24 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 01 Apr 2019 09:41:26 -0700 (PDT) From: Christian Couder X-Google-Original-From: Christian Couder To: git@vger.kernel.org Cc: Junio C Hamano , Jeff King , Ben Peart , Jonathan Tan , Jonathan Nieder , Nguyen Thai Ngoc Duy , Mike Hommey , Lars Schneider , Eric Wong , Christian Couder , Jeff Hostetler , Eric Sunshine , Beat Bolli , Christian Couder Subject: [PATCH v4 09/11] t0410: test fetching from many promisor remotes Date: Mon, 1 Apr 2019 18:40:43 +0200 Message-Id: <20190401164045.17328-10-chriscool@tuxfamily.org> X-Mailer: git-send-email 2.21.0.203.gd44fa53258 In-Reply-To: <20190401164045.17328-1-chriscool@tuxfamily.org> References: <20190401164045.17328-1-chriscool@tuxfamily.org> MIME-Version: 1.0 Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Christian Couder This shows that it is now possible to fetch objects from more than one promisor remote, and that fetching from a new promisor remote can configure it as one. Signed-off-by: Christian Couder --- t/t0410-partial-clone.sh | 47 +++++++++++++++++++++++++++++++++++++++- 1 file changed, 46 insertions(+), 1 deletion(-) diff --git a/t/t0410-partial-clone.sh b/t/t0410-partial-clone.sh index 9266037714..146b0a1e03 100755 --- a/t/t0410-partial-clone.sh +++ b/t/t0410-partial-clone.sh @@ -182,8 +182,53 @@ test_expect_success 'fetching of missing objects works with ref-in-want enabled' grep "git< fetch=.*ref-in-want" trace ' +test_expect_success 'fetching of missing objects from another promisor remote' ' + git clone "file://$(pwd)/server" server2 && + test_commit -C server2 bar && + git -C server2 repack -a -d --write-bitmap-index && + HASH2=$(git -C server2 rev-parse bar) && + + git -C repo remote add server2 "file://$(pwd)/server2" && + git -C repo config remote.server2.promisor true && + git -C repo cat-file -p "$HASH2" && + + git -C repo fetch server2 && + rm -rf repo/.git/objects/* && + git -C repo cat-file -p "$HASH2" && + + # Ensure that the .promisor file is written, and check that its + # associated packfile contains the object + ls repo/.git/objects/pack/pack-*.promisor >promisorlist && + test_line_count = 1 promisorlist && + IDX=$(cat promisorlist | sed "s/promisor$/idx/") && + git verify-pack --verbose "$IDX" | grep "$HASH2" +' + +test_expect_success 'fetching of missing objects configures a promisor remote' ' + git clone "file://$(pwd)/server" server3 && + test_commit -C server3 baz && + git -C server3 repack -a -d --write-bitmap-index && + HASH3=$(git -C server3 rev-parse baz) && + git -C server3 config uploadpack.allowfilter 1 && + + rm repo/.git/objects/pack/pack-*.promisor && + + git -C repo remote add server3 "file://$(pwd)/server3" && + git -C repo fetch --filter="blob:none" server3 $HASH3 && + + test "$(git -C repo config remote.server3.promisor)" = "true" && + + # Ensure that the .promisor file is written, and check that its + # associated packfile contains the object + ls repo/.git/objects/pack/pack-*.promisor >promisorlist && + test_line_count = 1 promisorlist && + IDX=$(cat promisorlist | sed "s/promisor$/idx/") && + git verify-pack --verbose "$IDX" | grep "$HASH3" +' + test_expect_success 'fetching of missing blobs works' ' - rm -rf server repo && + rm -rf server server2 repo && + rm -rf server server3 repo && test_create_repo server && test_commit -C server foo && git -C server repack -a -d --write-bitmap-index && From patchwork Mon Apr 1 16:40:44 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christian Couder X-Patchwork-Id: 10880287 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id DD772139A for ; Mon, 1 Apr 2019 16:41:34 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C7E8F2856A for ; Mon, 1 Apr 2019 16:41:34 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id BBD1028680; Mon, 1 Apr 2019 16:41:34 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 1FF89286CF for ; Mon, 1 Apr 2019 16:41:34 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728718AbfDAQld (ORCPT ); Mon, 1 Apr 2019 12:41:33 -0400 Received: from mail-wm1-f65.google.com ([209.85.128.65]:36422 "EHLO mail-wm1-f65.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728589AbfDAQlc (ORCPT ); Mon, 1 Apr 2019 12:41:32 -0400 Received: by mail-wm1-f65.google.com with SMTP id h18so122826wml.1 for ; Mon, 01 Apr 2019 09:41:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=+7wMX1FZ8lsyFPdUzNMI0GN84DRQK86a+6V7g/ygoP0=; b=cqOCWlOPfnZI1E+/gv29Te0AQAkm1ydslavkU/rdT/b1CdljLlV95+xMD/ULMA+hF5 UbODLETYQF/teitXAnuNCzvSQMACe9Ecj6eElXhYsTR1Lz7koEnfzztQC/0LRxDW3/bq ZUq6VWrGqwSNuAese8PTY+eoHLATyH6QYbeBRl8FKUxN3sIblurC4LHXfLv+LcFB/t2s FqIZvuDqaQaqTo4/YEuj8V3Z6ynJq11TUdFHiGY39R1WutdHh2rl+tFMrKElHZIQZqt+ UEyDiVSmXYG9hQ7lc96wxYliOb0VARe6BDoOXPl2CWWv2pEWhnzg9RR0/d6KmviLXBfl sPUw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=+7wMX1FZ8lsyFPdUzNMI0GN84DRQK86a+6V7g/ygoP0=; b=NGrEJoWNucyqdVdoGg253BY2NvIZZ3ivNOfH8CuFFbxiojlYb9ukxnGGexpehGCzGs 8iuOugbdoOb3VxgYt+A+JE3S1KWRKC4ObzwxiViAQLDsCSMNDRCh+bvpDCUDJPQnYVHt vewbx4nXsMlSAuwftDXeffrL3OVb1BB9A2dRYiNnuo6XnH5M8zDnmvgGsg3VhHu/fC6J 5+5pTvrVP3A2/Ra0NZ8G5mdMUpr3JHJaXoSlroYFAvTl5WCaVNnync/a83hsXbrLvq/G JJWIsOG5mx3OPwMAFkLh3vy/pNpDb0/+oT4rsJdyf8jwP36MGr/VMdNCSz7+KbkT/r/x oMfA== X-Gm-Message-State: APjAAAXWhwo6RSJP+Vfi1s5z1GrzPqoBlpUFwTu7RQMcPdxTHRlvTWoB +/DMfLsgSArZASBH+4mng8MruH/v X-Google-Smtp-Source: APXvYqylfSTOxTLTbZ7T3cXWnzHZbXiGNSXt55axyiT9+48TCNzgyLrzsc4PZOBgh/WkHxT70oA3tw== X-Received: by 2002:a1c:c504:: with SMTP id v4mr331739wmf.45.1554136889904; Mon, 01 Apr 2019 09:41:29 -0700 (PDT) Received: from localhost.localdomain ([2a04:cec0:101d:3e80:8542:d1d4:667f:a0da]) by smtp.gmail.com with ESMTPSA id s10sm11749070wmh.0.2019.04.01.09.41.27 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 01 Apr 2019 09:41:29 -0700 (PDT) From: Christian Couder X-Google-Original-From: Christian Couder To: git@vger.kernel.org Cc: Junio C Hamano , Jeff King , Ben Peart , Jonathan Tan , Jonathan Nieder , Nguyen Thai Ngoc Duy , Mike Hommey , Lars Schneider , Eric Wong , Christian Couder , Jeff Hostetler , Eric Sunshine , Beat Bolli Subject: [PATCH v4 10/11] partial-clone: add multiple remotes in the doc Date: Mon, 1 Apr 2019 18:40:44 +0200 Message-Id: <20190401164045.17328-11-chriscool@tuxfamily.org> X-Mailer: git-send-email 2.21.0.203.gd44fa53258 In-Reply-To: <20190401164045.17328-1-chriscool@tuxfamily.org> References: <20190401164045.17328-1-chriscool@tuxfamily.org> MIME-Version: 1.0 Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP While at it, let's remove a reference to ODB effort as the ODB effort has been replaced by directly enhancing partial clone and promisor remote features. Signed-off-by: Christian Couder --- Documentation/technical/partial-clone.txt | 83 ++++++++++++++++------- 1 file changed, 58 insertions(+), 25 deletions(-) diff --git a/Documentation/technical/partial-clone.txt b/Documentation/technical/partial-clone.txt index 896c7b3878..58adcc5ce1 100644 --- a/Documentation/technical/partial-clone.txt +++ b/Documentation/technical/partial-clone.txt @@ -100,18 +100,18 @@ or commits that reference missing trees. Handling Missing Objects ------------------------ -- An object may be missing due to a partial clone or fetch, or missing due - to repository corruption. To differentiate these cases, the local - repository specially indicates such filtered packfiles obtained from the - promisor remote as "promisor packfiles". +- An object may be missing due to a partial clone or fetch, or missing + due to repository corruption. To differentiate these cases, the + local repository specially indicates such filtered packfiles + obtained from promisor remotes as "promisor packfiles". + These promisor packfiles consist of a ".promisor" file with arbitrary contents (like the ".keep" files), in addition to their ".pack" and ".idx" files. - The local repository considers a "promisor object" to be an object that - it knows (to the best of its ability) that the promisor remote has promised - that it has, either because the local repository has that object in one of + it knows (to the best of its ability) that promisor remotes have promised + that they have, either because the local repository has that object in one of its promisor packfiles, or because another promisor object refers to it. + When Git encounters a missing object, Git can see if it is a promisor object @@ -123,7 +123,7 @@ expensive-to-modify list of missing objects.[a] - Since almost all Git code currently expects any referenced object to be present locally and because we do not want to force every command to do a dry-run first, a fallback mechanism is added to allow Git to attempt - to dynamically fetch missing objects from the promisor remote. + to dynamically fetch missing objects from promisor remotes. + When the normal object lookup fails to find an object, Git invokes fetch-object to try to get the object from the server and then retry @@ -157,8 +157,7 @@ and prefetch those objects in bulk. + We are not happy with this global variable and would like to remove it, but that requires significant refactoring of the object code to pass an -additional flag. We hope that concurrent efforts to add an ODB API can -encompass this. +additional flag. Fetching Missing Objects @@ -182,21 +181,53 @@ has been updated to not use any object flags when the corresponding argument though they are not necessary. +Using many promisor remotes +--------------------------- + +Many promisor remotes can be configured and used. + +This allows for example a user to have multiple geographically-close +cache servers for fetching missing blobs while continuing to do +filtered `git-fetch` commands from the central server. + +When fetching objects, promisor remotes are tried one after the other +until all the objects have been fetched. + +Remotes that are considered "promisor" remotes are those specified by +the following configuration variables: + +- `extensions.partialClone = ` + +- `remote..promisor = true` + +- `remote..partialCloneFilter = ...` + +Only one promisor remote can be configured using the +`extensions.partialClone` config variable. This promisor remote will +be the last one tried when fetching objects. + +The other promisor remotes will be tried in the order they appear in +the config file. + Current Limitations ------------------- -- The remote used for a partial clone (or the first partial fetch - following a regular clone) is marked as the "promisor remote". +- It is not possible to specify the order in which the promisor + remotes are tried in other ways than the order in which they appear + in the config file. + -We are currently limited to a single promisor remote and only that -remote may be used for subsequent partial fetches. +It is also not possible to specify an order to be used when fetching +from one remote and a different order when fetching from another +remote. + +- It is not possible to push only specific objects to a promisor + remote. + -We accept this limitation because we believe initial users of this -feature will be using it on repositories with a strong single central -server. +It is not possible to push at the same time to multiple promisor +remote in a specific order. -- Dynamic object fetching will only ask the promisor remote for missing - objects. We assume that the promisor remote has a complete view of the +- Dynamic object fetching will only ask promisor remotes for missing + objects. We assume that promisor remotes have a complete view of the repository and can satisfy all such requests. - Repack essentially treats promisor and non-promisor packfiles as 2 @@ -218,15 +249,17 @@ server. Future Work ----------- -- Allow more than one promisor remote and define a strategy for fetching - missing objects from specific promisor remotes or of iterating over the - set of promisor remotes until a missing object is found. +- Improve the way to specify the order in which promisor remotes are + tried. + -A user might want to have multiple geographically-close cache servers -for fetching missing blobs while continuing to do filtered `git-fetch` -commands from the central server, for example. +For example this could allow to specify explicitly something like: +"When fetching from this remote, I want to use these promisor remotes +in this order, though, when pushing or fetching to that remote, I want +to use those promisor remotes in that order." + +- Allow pushing to promisor remotes. + -Or the user might want to work in a triangular work flow with multiple +The user might want to work in a triangular work flow with multiple promisor remotes that each have an incomplete view of the repository. - Allow repack to work on promisor packfiles (while keeping them distinct From patchwork Mon Apr 1 16:40:45 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christian Couder X-Patchwork-Id: 10880289 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B9BEC139A for ; Mon, 1 Apr 2019 16:41:37 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A584728680 for ; Mon, 1 Apr 2019 16:41:37 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 9997F2866C; Mon, 1 Apr 2019 16:41:37 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 3C8D42866C for ; Mon, 1 Apr 2019 16:41:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728723AbfDAQlg (ORCPT ); Mon, 1 Apr 2019 12:41:36 -0400 Received: from mail-wr1-f65.google.com ([209.85.221.65]:34318 "EHLO mail-wr1-f65.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728589AbfDAQlf (ORCPT ); Mon, 1 Apr 2019 12:41:35 -0400 Received: by mail-wr1-f65.google.com with SMTP id p10so12953432wrq.1 for ; Mon, 01 Apr 2019 09:41:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=1x8uNGx+JGD7TkXaud87aiR6haI6KNGdh9Zlyne/c0I=; b=WEMdw9Bp2g3sb+Ez+ZsRbwgun5cBw3OR5bmAzHDDmER/qv2CjNuGktWPmw2ih3zJW1 /E5s1WDML9HLW3SG3pf7GZExk5Xu29tyloGjthTUTSGDj0LGVfRRbrXHQnGOK/XKeLft Tk9MyiJ/rYL+U3UCdTcx7AzxhaDe0MW/Lp6GjTWTuIUACrcGQwHFxqZpnxjAJ+aydn2o WuI1iGiDj618tJcFR/YJcG3hq4KJwRtRN2e1t03tk/pSZ7tVt70u6rXhkeDW/vFVNyCb YSxJGfkg3vkrSYXOweTgu4dWJCMSuAEiPXZfGvWW8j0nlM9kQx4sJbCCcU0NjPQAHGRL nu0w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=1x8uNGx+JGD7TkXaud87aiR6haI6KNGdh9Zlyne/c0I=; b=XTdeNZCIEaHu5j1PORoop9STEPi386o6nK+UPJJzk2bnBs6PTFWXX3yo2YeSSBPQFP lHmQGaOo8k1tDqCpc2EJX1njBBdxwSqaW7V+c3wbvf/8PnAHcybGbKLCgROiIYJHUm3y rMirwJSkQBiX9Cyf6ObejPfll1nA7dGLrotmY3b7yEABgEQSEPx1H0+lJxcZr+4g/Kcx UVDG9blUouNdQPC4ApdYQprKCPkHKuv2WnHNzHUizQIqWJ6+FbCzzY+/O1hFJv0PVaNF ZA/iurebS8SwXwloFZq0QMtr+CPDSE52NGJfurT1ZiGdE66qJp92O1rkiPanR4PvfHcP KyJg== X-Gm-Message-State: APjAAAVkFmHAyzp7P5NoVnzqlTy/Lpbt1efkJMCC6iQ26YR5zkcbliN0 2ZVi0ReuuOgqfg8SN1lJAuQKfXlJ X-Google-Smtp-Source: APXvYqyFCvqGHW0CeMXntsz9Zus3BYppMdljZ1+7UCvxTlNEj8KsrQ50nE5/AXMSYpfsXS2WUi1Ikg== X-Received: by 2002:a5d:684e:: with SMTP id o14mr43792155wrw.138.1554136893845; Mon, 01 Apr 2019 09:41:33 -0700 (PDT) Received: from localhost.localdomain ([2a04:cec0:101d:3e80:8542:d1d4:667f:a0da]) by smtp.gmail.com with ESMTPSA id s10sm11749070wmh.0.2019.04.01.09.41.31 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 01 Apr 2019 09:41:33 -0700 (PDT) From: Christian Couder X-Google-Original-From: Christian Couder To: git@vger.kernel.org Cc: Junio C Hamano , Jeff King , Ben Peart , Jonathan Tan , Jonathan Nieder , Nguyen Thai Ngoc Duy , Mike Hommey , Lars Schneider , Eric Wong , Christian Couder , Jeff Hostetler , Eric Sunshine , Beat Bolli Subject: [PATCH v4 11/11] remote: add promisor and partial clone config to the doc Date: Mon, 1 Apr 2019 18:40:45 +0200 Message-Id: <20190401164045.17328-12-chriscool@tuxfamily.org> X-Mailer: git-send-email 2.21.0.203.gd44fa53258 In-Reply-To: <20190401164045.17328-1-chriscool@tuxfamily.org> References: <20190401164045.17328-1-chriscool@tuxfamily.org> MIME-Version: 1.0 Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Signed-off-by: Christian Couder --- Documentation/config/remote.txt | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/Documentation/config/remote.txt b/Documentation/config/remote.txt index 6c4cad83a2..a8e6437a90 100644 --- a/Documentation/config/remote.txt +++ b/Documentation/config/remote.txt @@ -76,3 +76,11 @@ remote..pruneTags:: + See also `remote..prune` and the PRUNING section of linkgit:git-fetch[1]. + +remote..promisor:: + When set to true, this remote will be used to fetch promisor + objects. + +remote..partialclonefilter:: + The filter that will be applied when fetching from this + promisor remote.