From patchwork Tue Mar 12 13:29:49 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christian Couder X-Patchwork-Id: 10849215 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 7D744186E for ; Tue, 12 Mar 2019 13:30:19 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6053129535 for ; Tue, 12 Mar 2019 13:30:19 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 5E4012964A; Tue, 12 Mar 2019 13:30:19 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id CC881295FF for ; Tue, 12 Mar 2019 13:30:18 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726556AbfCLNaQ (ORCPT ); Tue, 12 Mar 2019 09:30:16 -0400 Received: from mail-ed1-f67.google.com ([209.85.208.67]:34971 "EHLO mail-ed1-f67.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726193AbfCLNaQ (ORCPT ); Tue, 12 Mar 2019 09:30:16 -0400 Received: by mail-ed1-f67.google.com with SMTP id g19so2311164edp.2 for ; Tue, 12 Mar 2019 06:30:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=ehKvBAHxrFYqsUiHITsavJlab95T8SUH+U73Eze0XRs=; b=RlM8ber1fRw9hrgezbY0WiPenFIeTM3utxs9WkD9IpYH2QiWFnvl/BQsdOjpDFNXJB vxIk4b9ozUBwhy8d4w+Q7xw3KiE9iuImomm0Oz0ghDY7I/wisSl71P3l5ePU5SJZtfAW uw2GquNBrmzDvxK1+HZEcstxXdKdr/wQRVeo3a/etAAx3qpwbQTeodBjvC9nMWBgKusN cWafOVqYzvl83EM6D8XsXIptYRf1wvD/6xNf0ztkxIlm3qVoz1xWwIrqQAyNeTKiJksR PGNqoXRO3VhJnszUCDcNIvnKz3ILIlik/EdEMGjfTkdo5llgfiQ2UkD0P5O9vJp1JXI6 b8Hw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=ehKvBAHxrFYqsUiHITsavJlab95T8SUH+U73Eze0XRs=; b=cFLDv1SYNHfBI1dX+Lnr+dFwt5LL2X9n56kJzb4AkItgiArdIOL8kqfu5Irku98R/2 JkMS6JE+rPTeI2qiR21EobGyers3Xh+AVWKBoAAoRKhL+i4erNJMDlhfLVBvdnOeVRuB EOYykMaMBCgMk1XBFvxIbUY8X8FHeKxChcTKpR7qJaAidlajDAavpop+zbx3UoK1JaNu Z0sezajJdIr540NJ/GK0hNXdsSSOjXVx/lxhQPvxwDzWTqjucqoByT3P67bcpyaBLvMd Y57IphHihMlDqxjkwolTfRBRb2KW9Bc/n2kix6Ra562ynJd2GCuvjvVNlkHari/+dDg/ yvnQ== X-Gm-Message-State: APjAAAVCwi4PXLYC0wDoCM3bJjFvByCOQF1XkJFB7me3gUPExYs1jkSC IlumP0D+CDPIW8kKtZOU1TRdowJL X-Google-Smtp-Source: APXvYqzRdhLw1YWvAu5kMDfL/qUDyijLQbm3Qu6XUEz2BftW+flosQ8o+Ag4SWeF8iIwkccuQ/dsKw== X-Received: by 2002:a17:906:28c4:: with SMTP id p4mr26061387ejd.63.1552397414289; Tue, 12 Mar 2019 06:30:14 -0700 (PDT) Received: from ubuntu-N750HU.booking.pcln.com (access-114.38.rev.fr.colt.net. [213.41.38.114]) by smtp.gmail.com with ESMTPSA id h8sm5875213edk.21.2019.03.12.06.30.13 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 12 Mar 2019 06:30:13 -0700 (PDT) From: Christian Couder X-Google-Original-From: Christian Couder To: git@vger.kernel.org Cc: Junio C Hamano , Jeff King , Ben Peart , Jonathan Tan , Jonathan Nieder , Stefan Beller , Nguyen Thai Ngoc Duy , Mike Hommey , Lars Schneider , Eric Wong , Christian Couder , Jeff Hostetler , Eric Sunshine , Beat Bolli , Christian Couder Subject: [PATCH v3 01/11] fetch-object: make functions return an error code Date: Tue, 12 Mar 2019 14:29:49 +0100 Message-Id: <20190312132959.11764-2-chriscool@tuxfamily.org> X-Mailer: git-send-email 2.21.0.166.gb5e4dbcfd3 In-Reply-To: <20190312132959.11764-1-chriscool@tuxfamily.org> References: <20190312132959.11764-1-chriscool@tuxfamily.org> MIME-Version: 1.0 Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Christian Couder The callers of the fetch_object() and fetch_objects() might be interested in knowing if these functions succeeded or not. Signed-off-by: Christian Couder Signed-off-by: Junio C Hamano --- fetch-object.c | 13 ++++++++----- fetch-object.h | 4 ++-- sha1-file.c | 4 ++-- 3 files changed, 12 insertions(+), 9 deletions(-) diff --git a/fetch-object.c b/fetch-object.c index 4266548800..eac4d448ef 100644 --- a/fetch-object.c +++ b/fetch-object.c @@ -5,11 +5,12 @@ #include "transport.h" #include "fetch-object.h" -static void fetch_refs(const char *remote_name, struct ref *ref) +static int fetch_refs(const char *remote_name, struct ref *ref) { struct remote *remote; struct transport *transport; int original_fetch_if_missing = fetch_if_missing; + int res; fetch_if_missing = 0; remote = remote_get(remote_name); @@ -19,12 +20,14 @@ static void fetch_refs(const char *remote_name, struct ref *ref) transport_set_option(transport, TRANS_OPT_FROM_PROMISOR, "1"); transport_set_option(transport, TRANS_OPT_NO_DEPENDENTS, "1"); - transport_fetch_refs(transport, ref); + res = transport_fetch_refs(transport, ref); fetch_if_missing = original_fetch_if_missing; + + return res; } -void fetch_objects(const char *remote_name, const struct object_id *oids, - int oid_nr) +int fetch_objects(const char *remote_name, const struct object_id *oids, + int oid_nr) { struct ref *ref = NULL; int i; @@ -36,5 +39,5 @@ void fetch_objects(const char *remote_name, const struct object_id *oids, new_ref->next = ref; ref = new_ref; } - fetch_refs(remote_name, ref); + return fetch_refs(remote_name, ref); } diff --git a/fetch-object.h b/fetch-object.h index d6444caa5a..7bcc7cadb0 100644 --- a/fetch-object.h +++ b/fetch-object.h @@ -3,7 +3,7 @@ struct object_id; -void fetch_objects(const char *remote_name, const struct object_id *oids, - int oid_nr); +int fetch_objects(const char *remote_name, const struct object_id *oids, + int oid_nr); #endif diff --git a/sha1-file.c b/sha1-file.c index 494606f771..01cc0590f4 100644 --- a/sha1-file.c +++ b/sha1-file.c @@ -1372,8 +1372,8 @@ int oid_object_info_extended(struct repository *r, const struct object_id *oid, if (fetch_if_missing && repository_format_partial_clone && !already_retried && r == the_repository) { /* - * TODO Investigate having fetch_object() return - * TODO error/success and stopping the music here. + * TODO Investigate checking fetch_object() return + * TODO value and stopping on error here. * TODO Pass a repository struct through fetch_object, * such that arbitrary repositories work. */ From patchwork Tue Mar 12 13:29:50 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christian Couder X-Patchwork-Id: 10849219 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id E2F48186E for ; Tue, 12 Mar 2019 13:30:22 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id CD7E52964C for ; Tue, 12 Mar 2019 13:30:22 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id CB50329535; Tue, 12 Mar 2019 13:30:22 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 5206929572 for ; Tue, 12 Mar 2019 13:30:22 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726657AbfCLNaU (ORCPT ); Tue, 12 Mar 2019 09:30:20 -0400 Received: from mail-ed1-f65.google.com ([209.85.208.65]:43092 "EHLO mail-ed1-f65.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726280AbfCLNaS (ORCPT ); Tue, 12 Mar 2019 09:30:18 -0400 Received: by mail-ed1-f65.google.com with SMTP id m35so2264910ede.10 for ; Tue, 12 Mar 2019 06:30:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Hw5FkOkjpdxozIvis1yT3kMadFYOiBoUwesRa2Fd4lE=; b=RblRkCXnOrSX/7necRtOgSecCnngBuEE6QeU2lDPAzxXsUeYrUub9eg1AQIhghF+to NTPXBPH+DSNNJxkRbP1BW1dcU5dZUygxTaftZXq0y0Lq7UE93dJXe6vETDOOlXVF2QaI xsj5qdynnTisELzGk9ttxg5AUUmgdFBfQohmbpqwuzMHVIyDG0yuysDSKI6cwBTfIivC Y0yQtEGvmryWM2n9wK63yQqag7JySP3eTWIPnyZEV8bwoaSCwSRasi26bHDG02nEgE1l v9DcuWozhwHTmZlqsr56/bPc7eHhSO/Nct0CafV0UBlFADINK4dFndKn+/OLqTUPnfp8 nJSw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Hw5FkOkjpdxozIvis1yT3kMadFYOiBoUwesRa2Fd4lE=; b=aEeOQWNUvayfuN4y53/QlZmByvcSJOUcP4X9bl21EOhSrQAXCAe0BIRtBsYEjvFB6d aeO2tVgZQ87WLPSMX70REs3Hm17jt6g7Ms8agfn8DKzMISCJ0F+6JhRSVvuD98S7AUXt 0yeEPGckJjvEUq+7m6MNfUFvcLAC7HkVphB0qpQJh3TvwrhXqznUdQVbZHoFI3eM9pQl 0L4pMckepw5r6FjgyVbI/diYRpdEZsYPQwAk58ZO/ORn7dSJERFSs271kT2nZ0eVwii+ PCLyEgbrOP156sv0PWeSSLk14WhsSQ5LaDyhuejpX3yGRi+VQlA19ASpXVyY9ewTq3Gj O4XQ== X-Gm-Message-State: APjAAAV2TnvVAHW3VfTQUgDccR651kAEs5uhh4nEEmupgs4MR4+7fZUb m0d8eNxefET2IOxEmRZtyTJTzybK X-Google-Smtp-Source: APXvYqxlumMJhKRkaDban5GLnDdAGGWz0k1pK+kyryMlAcRxGoE5N3GsFHzs77D/XIRVvgoUOwu9cg== X-Received: by 2002:aa7:dd8b:: with SMTP id g11mr3462187edv.75.1552397415641; Tue, 12 Mar 2019 06:30:15 -0700 (PDT) Received: from ubuntu-N750HU.booking.pcln.com (access-114.38.rev.fr.colt.net. [213.41.38.114]) by smtp.gmail.com with ESMTPSA id h8sm5875213edk.21.2019.03.12.06.30.14 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 12 Mar 2019 06:30:15 -0700 (PDT) From: Christian Couder X-Google-Original-From: Christian Couder To: git@vger.kernel.org Cc: Junio C Hamano , Jeff King , Ben Peart , Jonathan Tan , Jonathan Nieder , Stefan Beller , Nguyen Thai Ngoc Duy , Mike Hommey , Lars Schneider , Eric Wong , Christian Couder , Jeff Hostetler , Eric Sunshine , Beat Bolli , Christian Couder Subject: [PATCH v3 02/11] Add initial support for many promisor remotes Date: Tue, 12 Mar 2019 14:29:50 +0100 Message-Id: <20190312132959.11764-3-chriscool@tuxfamily.org> X-Mailer: git-send-email 2.21.0.166.gb5e4dbcfd3 In-Reply-To: <20190312132959.11764-1-chriscool@tuxfamily.org> References: <20190312132959.11764-1-chriscool@tuxfamily.org> MIME-Version: 1.0 Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Christian Couder The promisor-remote.{c,h} files will contain functions to manage many promisor remotes. We expect that there will not be a lot of promisor remotes, so it is ok to use a simple linked list to manage them. Helped-by: Jeff King Signed-off-by: Christian Couder --- Makefile | 1 + promisor-remote.c | 100 ++++++++++++++++++++++++++++++++++++++++++++++ promisor-remote.h | 17 ++++++++ 3 files changed, 118 insertions(+) create mode 100644 promisor-remote.c create mode 100644 promisor-remote.h diff --git a/Makefile b/Makefile index 537493822b..4f24ccb3dc 100644 --- a/Makefile +++ b/Makefile @@ -972,6 +972,7 @@ LIB_OBJS += preload-index.o LIB_OBJS += pretty.o LIB_OBJS += prio-queue.o LIB_OBJS += progress.o +LIB_OBJS += promisor-remote.o LIB_OBJS += prompt.o LIB_OBJS += protocol.o LIB_OBJS += quote.o diff --git a/promisor-remote.c b/promisor-remote.c new file mode 100644 index 0000000000..d2f574651e --- /dev/null +++ b/promisor-remote.c @@ -0,0 +1,100 @@ +#include "cache.h" +#include "promisor-remote.h" +#include "config.h" + +static struct promisor_remote *promisors; +static struct promisor_remote **promisors_tail = &promisors; + +struct promisor_remote *promisor_remote_new(const char *remote_name) +{ + struct promisor_remote *o; + + o = xcalloc(1, sizeof(*o)); + o->remote_name = xstrdup(remote_name); + + *promisors_tail = o; + promisors_tail = &o->next; + + return o; +} + +static struct promisor_remote *promisor_remote_look_up(const char *remote_name, + struct promisor_remote **previous) +{ + struct promisor_remote *o, *p; + + for (p = NULL, o = promisors; o; p = o, o = o->next) + if (o->remote_name && !strcmp(o->remote_name, remote_name)) { + if (previous) + *previous = p; + return o; + } + + return NULL; +} + +static void promisor_remote_move_to_tail(struct promisor_remote *o, + struct promisor_remote *previous) +{ + if (previous) + previous->next = o->next; + else + promisors = o->next ? o->next : o; + o->next = NULL; + *promisors_tail = o; + promisors_tail = &o->next; +} + +static int promisor_remote_config(const char *var, const char *value, void *data) +{ + struct promisor_remote *o; + const char *name; + int namelen; + const char *subkey; + + if (parse_config_key(var, "remote", &name, &namelen, &subkey) < 0) + return 0; + + if (!strcmp(subkey, "promisor")) { + char *remote_name; + + if (!git_config_bool(var, value)) + return 0; + + remote_name = xmemdupz(name, namelen); + + if (!promisor_remote_look_up(remote_name, NULL)) + promisor_remote_new(remote_name); + + free(remote_name); + return 0; + } + + return 0; +} + +static void promisor_remote_init(void) +{ + static int initialized; + + if (initialized) + return; + initialized = 1; + + git_config(promisor_remote_config, NULL); +} + +struct promisor_remote *promisor_remote_find(const char *remote_name) +{ + promisor_remote_init(); + + if (!remote_name) + return promisors; + + return promisor_remote_look_up(remote_name, NULL); +} + +int has_promisor_remote(void) +{ + return !!promisor_remote_find(NULL); +} diff --git a/promisor-remote.h b/promisor-remote.h new file mode 100644 index 0000000000..bfbf7c0f21 --- /dev/null +++ b/promisor-remote.h @@ -0,0 +1,17 @@ +#ifndef PROMISOR_REMOTE_H +#define PROMISOR_REMOTE_H + +/* + * Promisor remote linked list + * Its information come from remote.XXX config entries. + */ +struct promisor_remote { + const char *remote_name; + struct promisor_remote *next; +}; + +extern struct promisor_remote *promisor_remote_new(const char *remote_name); +extern struct promisor_remote *promisor_remote_find(const char *remote_name); +extern int has_promisor_remote(void); + +#endif /* PROMISOR_REMOTE_H */ From patchwork Tue Mar 12 13:29:51 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christian Couder X-Patchwork-Id: 10849217 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 4906B6C2 for ; Tue, 12 Mar 2019 13:30:22 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 34D082964D for ; Tue, 12 Mar 2019 13:30:22 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 32C0B29652; Tue, 12 Mar 2019 13:30:22 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B1DA129655 for ; Tue, 12 Mar 2019 13:30:21 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726671AbfCLNaU (ORCPT ); Tue, 12 Mar 2019 09:30:20 -0400 Received: from mail-ed1-f65.google.com ([209.85.208.65]:32862 "EHLO mail-ed1-f65.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725774AbfCLNaT (ORCPT ); Tue, 12 Mar 2019 09:30:19 -0400 Received: by mail-ed1-f65.google.com with SMTP id d12so2320124edp.0 for ; Tue, 12 Mar 2019 06:30:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=UTtyVVmechrkSs9dBFE04OBtxYGIF4H9ahHN5WolIZE=; b=KbwD06oV4/VZ0i6e00lbACxYFoVkXAnTr//qCN4e4ziU6LKZ0RExD5Ezofav2jX0pE VuNOw44ukM+wX+IBfw90IBUlhfxlqS0Ip3P8dWAKlHgCq0ZPMOecW5Wgk2YXYUD4igVx 3jrquHwUWW+rzWhk6AIAb0tXmxehn4uliJL9Ir5f+CkdZaLNFwr86qQYQLKmEGRYPC3/ oy+3R4i81hsY23kekJxYBERk6I8N3B87mzS7gTc/xo7sOd1Sa8e7+XwSLUV253Vkyy5S tL6sq2qL0XsUPqyRCXQv8Eux5yHgAxgfyPQ2enI7eH6XBpmeDZxmygoG1L0yshAPE7Z9 vVUA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=UTtyVVmechrkSs9dBFE04OBtxYGIF4H9ahHN5WolIZE=; b=adco32IaRVHEScVSKs0OJe4IU3FDFeJ7/WFiitevH0EW7TNDaUvHZemcCtDPaDoycB CCsR7lIlDzMwB3BZw0VPyu8pK++wFxGnh0GsCcMvd+QIUzR9GicTxVHvJpL7gdK3I5Vu kLWENH3iRHNhTPIswKiHRp7Zn9Of11iBhgSkqXA3y/oK4/xG2Im7+ag9umW5QpY35hWf /7cYLa+Ih8l8BsH3f7ICdZPPSm04Yfu32kGqHdn48TO57ItvFFDvwp5kIMqU5J+U29o3 VHftwY1HimCcEMWqXnicVhqaeGppvYMj0DUgyFx1AWfOJ6DrDM//A4mhMYqLxVWwgn5h ixLA== X-Gm-Message-State: APjAAAX81YKcMfqkW3EOSIiM6vzMVtdeqPh0ktieampSJFIwxcg+nVV5 /yrlcyu11oHpRbdMQ0C0AQ4UqeCa X-Google-Smtp-Source: APXvYqwU3z76EXwai7orW+D09QNGyk6hVByqeNYWDin3iMxKgiOzVsrCvweDdA2wwF2CJ9N2tuE0ZA== X-Received: by 2002:a17:906:1942:: with SMTP id b2mr26132376eje.5.1552397416649; Tue, 12 Mar 2019 06:30:16 -0700 (PDT) Received: from ubuntu-N750HU.booking.pcln.com (access-114.38.rev.fr.colt.net. [213.41.38.114]) by smtp.gmail.com with ESMTPSA id h8sm5875213edk.21.2019.03.12.06.30.15 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 12 Mar 2019 06:30:15 -0700 (PDT) From: Christian Couder X-Google-Original-From: Christian Couder To: git@vger.kernel.org Cc: Junio C Hamano , Jeff King , Ben Peart , Jonathan Tan , Jonathan Nieder , Stefan Beller , Nguyen Thai Ngoc Duy , Mike Hommey , Lars Schneider , Eric Wong , Christian Couder , Jeff Hostetler , Eric Sunshine , Beat Bolli , Christian Couder Subject: [PATCH v3 03/11] promisor-remote: implement promisor_remote_get_direct() Date: Tue, 12 Mar 2019 14:29:51 +0100 Message-Id: <20190312132959.11764-4-chriscool@tuxfamily.org> X-Mailer: git-send-email 2.21.0.166.gb5e4dbcfd3 In-Reply-To: <20190312132959.11764-1-chriscool@tuxfamily.org> References: <20190312132959.11764-1-chriscool@tuxfamily.org> MIME-Version: 1.0 Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Christian Couder This is implemented for now by calling fetch_objects(). It fetches from all the promisor remotes. Signed-off-by: Christian Couder --- promisor-remote.c | 17 +++++++++++++++++ promisor-remote.h | 1 + 2 files changed, 18 insertions(+) diff --git a/promisor-remote.c b/promisor-remote.c index d2f574651e..f86f9d0b84 100644 --- a/promisor-remote.c +++ b/promisor-remote.c @@ -1,6 +1,7 @@ #include "cache.h" #include "promisor-remote.h" #include "config.h" +#include "fetch-object.h" static struct promisor_remote *promisors; static struct promisor_remote **promisors_tail = &promisors; @@ -98,3 +99,19 @@ int has_promisor_remote(void) { return !!promisor_remote_find(NULL); } + +int promisor_remote_get_direct(const struct object_id *oids, int oid_nr) +{ + struct promisor_remote *o; + + promisor_remote_init(); + + for (o = promisors; o; o = o->next) { + if (fetch_objects(o->remote_name, oids, oid_nr) < 0) + continue; + return 0; + } + + return -1; +} + diff --git a/promisor-remote.h b/promisor-remote.h index bfbf7c0f21..f9f5825417 100644 --- a/promisor-remote.h +++ b/promisor-remote.h @@ -13,5 +13,6 @@ struct promisor_remote { extern struct promisor_remote *promisor_remote_new(const char *remote_name); extern struct promisor_remote *promisor_remote_find(const char *remote_name); extern int has_promisor_remote(void); +extern int promisor_remote_get_direct(const struct object_id *oids, int oid_nr); #endif /* PROMISOR_REMOTE_H */ From patchwork Tue Mar 12 13:29:52 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christian Couder X-Patchwork-Id: 10849221 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 153186C2 for ; Tue, 12 Mar 2019 13:30:24 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0242F29655 for ; Tue, 12 Mar 2019 13:30:24 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 009532964F; Tue, 12 Mar 2019 13:30:23 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A108B29652 for ; Tue, 12 Mar 2019 13:30:23 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1725874AbfCLNaW (ORCPT ); Tue, 12 Mar 2019 09:30:22 -0400 Received: from mail-ed1-f66.google.com ([209.85.208.66]:43097 "EHLO mail-ed1-f66.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726255AbfCLNaU (ORCPT ); Tue, 12 Mar 2019 09:30:20 -0400 Received: by mail-ed1-f66.google.com with SMTP id m35so2265030ede.10 for ; Tue, 12 Mar 2019 06:30:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=L0Cen3rMdL3+841sDs2ZwexsAwpHx7Iy/yGgBR/7zno=; b=ACo+qUwFNlBh/Ng7E1vD7e8A24Hk9aBlGDx1zF5X1fO4Hmj3YEhEQO702cQzqpsapm 7ofplC0OgtoSzP10Q1wfdbyBqkUVzHIhQVRAGjenBT7bM+UDkcv1CrcFt9RwEVf5jATZ FW+DD635giAh/GsB1AWyVOqc3xG5vqsOi6shENUeGG3ByxQbyMbrQXCOTKdLHfZxGYpM jIq2CKyWbFZLv1Hkw5SmZgtZn7BJoM/EUrGlpgvo4zrbjn+K9E6acgBZN0NJZ552Ha5u AE1HcWQLM0i+fPXUb1p4Wheu8xdoUZuF3jHepdtfOfXT9lXKUQHYSy3Ba/Tqmqc3mru8 qnqg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=L0Cen3rMdL3+841sDs2ZwexsAwpHx7Iy/yGgBR/7zno=; b=Ei11xbh1GxYQS8z9YytmpMYzJ5DjlZc0wmFgyaEEEKEg/aJDh9b0TxDADmuLbckMsW fFVr4Y7yjjNUbFiRv66bVd387eyIYj6in0y354RYkKPdi3atxqyK4yT4Vrsit27xAh0h 00W2+k9yEolCrrFWEm9sg3wb1Yi4a2M/nLaGqUJAqxnxlghvIQ7qZQ7JnCzbVdoXdXRY x5/vTYln6QZv97wF0i2Rv93X1t3YnZFYF9jxBGBnjpyQVa1ajV7IJx6jmbbMfB1qiF4+ pQ5wZ+nsa4EB6OrfeGgv21UImDq7wH6k90KmPzhOm1uRex9zbO8QKEznV5KAHftlrSxZ D6/w== X-Gm-Message-State: APjAAAV2DUnUj6caQjX6z7qc0hq0B0ukKoNB1PSj9iqqN/K26WorIkPM tW9OsRCjx+ugJ4T8jejmULep5f8Q X-Google-Smtp-Source: APXvYqxsQ0UBtblAJvYeNpUC6v9II2hmbEqACzzbW4hFid+wgjHELkpFZ5XfbmwmtXY4ismO4iSWlA== X-Received: by 2002:a17:906:7711:: with SMTP id q17mr25828653ejm.174.1552397417828; Tue, 12 Mar 2019 06:30:17 -0700 (PDT) Received: from ubuntu-N750HU.booking.pcln.com (access-114.38.rev.fr.colt.net. [213.41.38.114]) by smtp.gmail.com with ESMTPSA id h8sm5875213edk.21.2019.03.12.06.30.16 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 12 Mar 2019 06:30:16 -0700 (PDT) From: Christian Couder X-Google-Original-From: Christian Couder To: git@vger.kernel.org Cc: Junio C Hamano , Jeff King , Ben Peart , Jonathan Tan , Jonathan Nieder , Stefan Beller , Nguyen Thai Ngoc Duy , Mike Hommey , Lars Schneider , Eric Wong , Christian Couder , Jeff Hostetler , Eric Sunshine , Beat Bolli , Christian Couder Subject: [PATCH v3 04/11] promisor-remote: add promisor_remote_reinit() Date: Tue, 12 Mar 2019 14:29:52 +0100 Message-Id: <20190312132959.11764-5-chriscool@tuxfamily.org> X-Mailer: git-send-email 2.21.0.166.gb5e4dbcfd3 In-Reply-To: <20190312132959.11764-1-chriscool@tuxfamily.org> References: <20190312132959.11764-1-chriscool@tuxfamily.org> MIME-Version: 1.0 Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Christian Couder We will need to reinitialize the promisor remote configuration as we will make some changes to it in a later commit. Signed-off-by: Christian Couder --- promisor-remote.c | 14 ++++++++++++-- promisor-remote.h | 1 + 2 files changed, 13 insertions(+), 2 deletions(-) diff --git a/promisor-remote.c b/promisor-remote.c index f86f9d0b84..ea74f6d8a8 100644 --- a/promisor-remote.c +++ b/promisor-remote.c @@ -74,17 +74,27 @@ static int promisor_remote_config(const char *var, const char *value, void *data return 0; } -static void promisor_remote_init(void) +static void promisor_remote_do_init(int force) { static int initialized; - if (initialized) + if (!force && initialized) return; initialized = 1; git_config(promisor_remote_config, NULL); } +static inline void promisor_remote_init(void) +{ + promisor_remote_do_init(0); +} + +void promisor_remote_reinit(void) +{ + promisor_remote_do_init(1); +} + struct promisor_remote *promisor_remote_find(const char *remote_name) { promisor_remote_init(); diff --git a/promisor-remote.h b/promisor-remote.h index f9f5825417..f96722bc66 100644 --- a/promisor-remote.h +++ b/promisor-remote.h @@ -10,6 +10,7 @@ struct promisor_remote { struct promisor_remote *next; }; +extern void promisor_remote_reinit(void); extern struct promisor_remote *promisor_remote_new(const char *remote_name); extern struct promisor_remote *promisor_remote_find(const char *remote_name); extern int has_promisor_remote(void); From patchwork Tue Mar 12 13:29:53 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christian Couder X-Patchwork-Id: 10849223 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 841E36C2 for ; Tue, 12 Mar 2019 13:30:25 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 71C5529574 for ; Tue, 12 Mar 2019 13:30:25 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 6F84529656; Tue, 12 Mar 2019 13:30:25 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0E48E2962F for ; Tue, 12 Mar 2019 13:30:25 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726703AbfCLNaY (ORCPT ); Tue, 12 Mar 2019 09:30:24 -0400 Received: from mail-ed1-f67.google.com ([209.85.208.67]:32868 "EHLO mail-ed1-f67.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725774AbfCLNaV (ORCPT ); Tue, 12 Mar 2019 09:30:21 -0400 Received: by mail-ed1-f67.google.com with SMTP id d12so2320288edp.0 for ; Tue, 12 Mar 2019 06:30:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=U4N+VL7xrP7XWFLQZvSKADPDiuJ6H3btFTJWpW9kTe0=; b=Qyo0LrukHUR4W4+u56Fsrw36KMed4T1w8ksJQhhzPTaeata7sZ9ORzbnCIIO8Fm9Db 0eE1NVq0DM/LFHgJ6tVNpeFjRy/hFV346s3+OKdwKX/kkhhh9ZBn9WYLuiq6IaSl+BvH 7ZtjXVw4GyQcwUhetE/CAeWg2xP3/Zpnt0FtESKBH1hRBvOA5IgRPV6jlcxAMiDSl5V5 WQ2E70pdN778cGJXNQ+XZ2z15AOHu+IDYTyVg/RTLX3JetoeFP6n9hA5cdj/FnRyibcn XA5jOjVlQPVQlqHMEJ46qAjkzBjPvmlE09rWvVh1m4jTTDRMHaRoASOdZGwdT1N6yDTl Nbrg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=U4N+VL7xrP7XWFLQZvSKADPDiuJ6H3btFTJWpW9kTe0=; b=N/z8jVitYtEBsWEQ86PRcE0WimYOc1cOXRbphSe33x7mYtkz5yRUCeoLzOGwd8Z20O y57pSuRxiGhG6RtgW5LhogFS4+aXq4Hsfk9LeBIHy1GyfqltlW3K0pvYPdy+9SK39hyG /JStRmEzmZcJluHF1K/a/56XLG+Hu+xleu2y27EDkWB7E+HuZKtt4yYMggWqIOwirDNZ KpbKP1xWN71NKxhyRR4hlSha/M31IxbF2qjrlluvAvC7UZsDAbqEYB/3y28AQ/cVJfZZ my0pMAjGIukHhiCI5UZJJdoxd7TCdGf/HL9WRgyybj2r1KYvngJFavnkfl/ecpGo0Gsj +f/w== X-Gm-Message-State: APjAAAU3exqyvyj2DP52jH/1yoGq9gbUXsQCmg/tj4uEQY7jN6vjrfZ3 dFQBzHWTlmgrQjdd+PJGGluqf15y X-Google-Smtp-Source: APXvYqxXHtpGqpksXK20XIeWMUy2IzsHAVNaGpm5C6My20boHHSNO1XbuvuW3WE5gOX7Z57uEVK6sQ== X-Received: by 2002:a17:906:3c3:: with SMTP id c3mr25006240eja.181.1552397419758; Tue, 12 Mar 2019 06:30:19 -0700 (PDT) Received: from ubuntu-N750HU.booking.pcln.com (access-114.38.rev.fr.colt.net. [213.41.38.114]) by smtp.gmail.com with ESMTPSA id h8sm5875213edk.21.2019.03.12.06.30.17 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 12 Mar 2019 06:30:18 -0700 (PDT) From: Christian Couder X-Google-Original-From: Christian Couder To: git@vger.kernel.org Cc: Junio C Hamano , Jeff King , Ben Peart , Jonathan Tan , Jonathan Nieder , Stefan Beller , Nguyen Thai Ngoc Duy , Mike Hommey , Lars Schneider , Eric Wong , Christian Couder , Jeff Hostetler , Eric Sunshine , Beat Bolli Subject: [PATCH v3 05/11] promisor-remote: use repository_format_partial_clone Date: Tue, 12 Mar 2019 14:29:53 +0100 Message-Id: <20190312132959.11764-6-chriscool@tuxfamily.org> X-Mailer: git-send-email 2.21.0.166.gb5e4dbcfd3 In-Reply-To: <20190312132959.11764-1-chriscool@tuxfamily.org> References: <20190312132959.11764-1-chriscool@tuxfamily.org> MIME-Version: 1.0 Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP A remote specified using the extensions.partialClone config option should be considered a promisor remote too. This remote should be at the end of the promisor remote list, so that it is used only if objects have not been found in other remotes. Signed-off-by: Christian Couder --- promisor-remote.c | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/promisor-remote.c b/promisor-remote.c index ea74f6d8a8..dcf6ef6521 100644 --- a/promisor-remote.c +++ b/promisor-remote.c @@ -83,6 +83,17 @@ static void promisor_remote_do_init(int force) initialized = 1; git_config(promisor_remote_config, NULL); + + if (repository_format_partial_clone) { + struct promisor_remote *o, *previous; + + o = promisor_remote_look_up(repository_format_partial_clone, + &previous); + if (o) + promisor_remote_move_to_tail(o, previous); + else + promisor_remote_new(repository_format_partial_clone); + } } static inline void promisor_remote_init(void) From patchwork Tue Mar 12 13:29:54 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christian Couder X-Patchwork-Id: 10849229 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id EA84D6C2 for ; Tue, 12 Mar 2019 13:30:30 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D6AA6296AD for ; Tue, 12 Mar 2019 13:30:30 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id D42C929699; Tue, 12 Mar 2019 13:30:30 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B64332969D for ; Tue, 12 Mar 2019 13:30:29 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726731AbfCLNaZ (ORCPT ); Tue, 12 Mar 2019 09:30:25 -0400 Received: from mail-ed1-f44.google.com ([209.85.208.44]:36857 "EHLO mail-ed1-f44.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726697AbfCLNaX (ORCPT ); Tue, 12 Mar 2019 09:30:23 -0400 Received: by mail-ed1-f44.google.com with SMTP id e4so2307874edi.3 for ; Tue, 12 Mar 2019 06:30:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=TBPEpDuRPZ2/YjbyPaJ1s8okeJOTok1vnDs1HCxXPHc=; b=QgfxgVmftuDHuQyvXDvCg05ynkxKyuUVNoySYVX80JqYb1JA8O7rp17OXAZf+zp6wb emNPHNq3V/HEJwbSZtFC1fRddOrnFnB8zT3nQoWyQ0iQ9xmNDaJpBKPilR+tHGKtoTgM fPydZaovjeKWdXzQRmV5h5PKj/Mm5+rOQZljWdHvjOTRIWxF+TWBAkglZYVGTmJ4PVqZ TteLWw0afmjrK6XpQBG1VVim5EC0nvdTpfLvYV2AAdTtuMrjy3a97YkmO2mPBK86Mm3m SqseOFFv7hsKzRzo5/jDii/dmPOTf8EO/+pxC2oxVxMrWaP6GMG7uzSovXRjGBXn/aCX BLmA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=TBPEpDuRPZ2/YjbyPaJ1s8okeJOTok1vnDs1HCxXPHc=; b=Z6+cTw0ZqfK0Z8hm37rJIfKAI2xkNmD8tDagi4bpbbocvNmcCAypsHzIwIE76ubmK2 JvFhRO7dkYVr06+SQJEgKTeXuftmIVmNTMqjOFlEkKzF+tLdhPa/dNaQi9VpWFzOpfjg LhCDJshByqtb52ofpBv2HlyGpNHhoxE+RJqJss6n9tZqycLHulyjrj3GFKHjQ4j52L0Y pXdSzjSGSsoXpuxOFD2i+zDJJhcAKYdBUUbDFRSrlAw+SPY8lNlI6BMBTpl3wEuTip/4 PHkm3xNJSBIdebSoPa2nCQJUOT5ueO3MoCnvornKPY1b3UUypgCiDZHqnZpl7aRC6zwU MfNw== X-Gm-Message-State: APjAAAWfwv6IL07bpteXfXbynTiWCRmJFQxACVE1H0dArbQ+CIS5J0Md 9xPiZ3sQ/nTEBioRv/viMbKn4Wwx X-Google-Smtp-Source: APXvYqxjUC6Ng32RpUX4xk8DVRrbwfWMyUcP8/wwr0YdcQvlhOG1EmVCx+LWhSI3Un1o4HUT0p+06w== X-Received: by 2002:a17:906:2797:: with SMTP id j23mr9845448ejc.128.1552397420943; Tue, 12 Mar 2019 06:30:20 -0700 (PDT) Received: from ubuntu-N750HU.booking.pcln.com (access-114.38.rev.fr.colt.net. [213.41.38.114]) by smtp.gmail.com with ESMTPSA id h8sm5875213edk.21.2019.03.12.06.30.19 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 12 Mar 2019 06:30:20 -0700 (PDT) From: Christian Couder X-Google-Original-From: Christian Couder To: git@vger.kernel.org Cc: Junio C Hamano , Jeff King , Ben Peart , Jonathan Tan , Jonathan Nieder , Stefan Beller , Nguyen Thai Ngoc Duy , Mike Hommey , Lars Schneider , Eric Wong , Christian Couder , Jeff Hostetler , Eric Sunshine , Beat Bolli Subject: [PATCH v3 06/11] Use promisor_remote_get_direct() and has_promisor_remote() Date: Tue, 12 Mar 2019 14:29:54 +0100 Message-Id: <20190312132959.11764-7-chriscool@tuxfamily.org> X-Mailer: git-send-email 2.21.0.166.gb5e4dbcfd3 In-Reply-To: <20190312132959.11764-1-chriscool@tuxfamily.org> References: <20190312132959.11764-1-chriscool@tuxfamily.org> MIME-Version: 1.0 Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Instead of using the repository_format_partial_clone global and fetch_objects() directly, let's use has_promisor_remote() and promisor_remote_get_direct(). This way all the configured promisor remotes will be taken into account, not only the one specified by extensions.partialClone. Also when cloning or fetching using a partial clone filter, remote.origin.promisor will be set to "true" instead of setting extensions.partialClone to "origin". This makes it possible to use many promisor remote just by fetching from them. Signed-off-by: Christian Couder --- builtin/cat-file.c | 5 +++-- builtin/fetch.c | 11 ++++++----- builtin/gc.c | 3 ++- builtin/repack.c | 3 ++- cache-tree.c | 3 ++- connected.c | 3 ++- list-objects-filter-options.c | 28 +++++++++++++++------------- packfile.c | 3 ++- sha1-file.c | 14 ++++++++------ t/t5601-clone.sh | 2 +- t/t5616-partial-clone.sh | 2 +- unpack-trees.c | 6 +++--- 12 files changed, 47 insertions(+), 36 deletions(-) diff --git a/builtin/cat-file.c b/builtin/cat-file.c index 0f092382e1..85ae10bf0b 100644 --- a/builtin/cat-file.c +++ b/builtin/cat-file.c @@ -15,6 +15,7 @@ #include "sha1-array.h" #include "packfile.h" #include "object-store.h" +#include "promisor-remote.h" struct batch_options { int enabled; @@ -523,8 +524,8 @@ static int batch_objects(struct batch_options *opt) if (opt->all_objects) { struct object_cb_data cb; - if (repository_format_partial_clone) - warning("This repository has extensions.partialClone set. Some objects may not be loaded."); + if (has_promisor_remote()) + warning("This repository uses promisor remotes. Some objects may not be loaded."); cb.opt = opt; cb.expand = &data; diff --git a/builtin/fetch.c b/builtin/fetch.c index b620fd54b4..7edf70ae6c 100644 --- a/builtin/fetch.c +++ b/builtin/fetch.c @@ -23,6 +23,7 @@ #include "packfile.h" #include "list-objects-filter-options.h" #include "commit-reach.h" +#include "promisor-remote.h" static const char * const builtin_fetch_usage[] = { N_("git fetch [] [ [...]]"), @@ -1460,7 +1461,7 @@ static inline void fetch_one_setup_partial(struct remote *remote) * If no prior partial clone/fetch and the current fetch DID NOT * request a partial-fetch, do a normal fetch. */ - if (!repository_format_partial_clone && !filter_options.choice) + if (!has_promisor_remote() && !filter_options.choice) return; /* @@ -1468,7 +1469,7 @@ static inline void fetch_one_setup_partial(struct remote *remote) * on this repo and remember the given filter-spec as the default * for subsequent fetches to this remote. */ - if (!repository_format_partial_clone && filter_options.choice) { + if (!has_promisor_remote() && filter_options.choice) { partial_clone_register(remote->name, &filter_options); return; } @@ -1477,7 +1478,7 @@ static inline void fetch_one_setup_partial(struct remote *remote) * We are currently limited to only ONE promisor remote and only * allow partial-fetches from the promisor remote. */ - if (strcmp(remote->name, repository_format_partial_clone)) { + if (!promisor_remote_find(remote->name)) { if (filter_options.choice) die(_("--filter can only be used with the remote " "configured in extensions.partialClone")); @@ -1609,7 +1610,7 @@ int cmd_fetch(int argc, const char **argv, const char *prefix) if (depth || deepen_since || deepen_not.nr) deepen = 1; - if (filter_options.choice && !repository_format_partial_clone) + if (filter_options.choice && !has_promisor_remote()) die("--filter can only be used when extensions.partialClone is set"); if (all) { @@ -1643,7 +1644,7 @@ int cmd_fetch(int argc, const char **argv, const char *prefix) } if (remote) { - if (filter_options.choice || repository_format_partial_clone) + if (filter_options.choice || has_promisor_remote()) fetch_one_setup_partial(remote); result = fetch_one(remote, argc, argv, prune_tags_ok); } else { diff --git a/builtin/gc.c b/builtin/gc.c index 020f725acc..0bec41b25f 100644 --- a/builtin/gc.c +++ b/builtin/gc.c @@ -27,6 +27,7 @@ #include "pack-objects.h" #include "blob.h" #include "tree.h" +#include "promisor-remote.h" #define FAILED_RUN "failed to run %s" @@ -640,7 +641,7 @@ int cmd_gc(int argc, const char **argv, const char *prefix) argv_array_push(&prune, prune_expire); if (quiet) argv_array_push(&prune, "--no-progress"); - if (repository_format_partial_clone) + if (has_promisor_remote()) argv_array_push(&prune, "--exclude-promisor-objects"); if (run_command_v_opt(prune.argv, RUN_GIT_CMD)) diff --git a/builtin/repack.c b/builtin/repack.c index 67f8978043..3b935690c8 100644 --- a/builtin/repack.c +++ b/builtin/repack.c @@ -11,6 +11,7 @@ #include "midx.h" #include "packfile.h" #include "object-store.h" +#include "promisor-remote.h" static int delta_base_offset = 1; static int pack_kept_objects = -1; @@ -366,7 +367,7 @@ int cmd_repack(int argc, const char **argv, const char *prefix) argv_array_push(&cmd.args, "--all"); argv_array_push(&cmd.args, "--reflog"); argv_array_push(&cmd.args, "--indexed-objects"); - if (repository_format_partial_clone) + if (has_promisor_remote()) argv_array_push(&cmd.args, "--exclude-promisor-objects"); if (write_bitmaps) argv_array_push(&cmd.args, "--write-bitmap-index"); diff --git a/cache-tree.c b/cache-tree.c index b13bfaf71e..64c285a746 100644 --- a/cache-tree.c +++ b/cache-tree.c @@ -5,6 +5,7 @@ #include "cache-tree.h" #include "object-store.h" #include "replace-object.h" +#include "promisor-remote.h" #ifndef DEBUG #define DEBUG 0 @@ -357,7 +358,7 @@ static int update_one(struct cache_tree *it, } ce_missing_ok = mode == S_IFGITLINK || missing_ok || - (repository_format_partial_clone && + (has_promisor_remote() && ce_skip_worktree(ce)); if (is_null_oid(oid) || (!ce_missing_ok && !has_object_file(oid))) { diff --git a/connected.c b/connected.c index 1bba888eff..0eaaedee6a 100644 --- a/connected.c +++ b/connected.c @@ -4,6 +4,7 @@ #include "connected.h" #include "transport.h" #include "packfile.h" +#include "promisor-remote.h" /* * If we feed all the commits we want to verify to this command @@ -56,7 +57,7 @@ int check_connected(oid_iterate_fn fn, void *cb_data, argv_array_push(&rev_list.args,"rev-list"); argv_array_push(&rev_list.args, "--objects"); argv_array_push(&rev_list.args, "--stdin"); - if (repository_format_partial_clone) + if (has_promisor_remote()) argv_array_push(&rev_list.args, "--exclude-promisor-objects"); if (!opt->is_deepening_fetch) { argv_array_push(&rev_list.args, "--not"); diff --git a/list-objects-filter-options.c b/list-objects-filter-options.c index c0036f7378..f41a831fce 100644 --- a/list-objects-filter-options.c +++ b/list-objects-filter-options.c @@ -6,6 +6,7 @@ #include "list-objects.h" #include "list-objects-filter.h" #include "list-objects-filter-options.h" +#include "promisor-remote.h" /* * Parse value of the argument to the "filter" keyword. @@ -144,30 +145,31 @@ void partial_clone_register( const char *remote, const struct list_objects_filter_options *filter_options) { - /* - * Record the name of the partial clone remote in the - * config and in the global variable -- the latter is - * used throughout to indicate that partial clone is - * enabled and to expect missing objects. - */ - if (repository_format_partial_clone && - *repository_format_partial_clone && - strcmp(remote, repository_format_partial_clone)) - die(_("cannot change partial clone promisor remote")); + char *cfg_name; - git_config_set("core.repositoryformatversion", "1"); - git_config_set("extensions.partialclone", remote); + /* Check if it is already registered */ + if (!promisor_remote_find(remote)) { + git_config_set("core.repositoryformatversion", "1"); - repository_format_partial_clone = xstrdup(remote); + /* Add promisor config for the remote */ + cfg_name = xstrfmt("remote.%s.promisor", remote); + git_config_set(cfg_name, "true"); + free(cfg_name); + } /* * Record the initial filter-spec in the config as * the default for subsequent fetches from this remote. + * + * TODO: record it into remote..partialclonefilter */ core_partial_clone_filter_default = xstrdup(filter_options->filter_spec); git_config_set("core.partialclonefilter", core_partial_clone_filter_default); + + /* Make sure the config info are reset */ + promisor_remote_reinit(); } void partial_clone_get_default_filter_spec( diff --git a/packfile.c b/packfile.c index 16bcb75262..0f95cdc1c9 100644 --- a/packfile.c +++ b/packfile.c @@ -16,6 +16,7 @@ #include "tree.h" #include "object-store.h" #include "midx.h" +#include "promisor-remote.h" char *odb_pack_name(struct strbuf *buf, const unsigned char *sha1, @@ -2119,7 +2120,7 @@ int is_promisor_object(const struct object_id *oid) static int promisor_objects_prepared; if (!promisor_objects_prepared) { - if (repository_format_partial_clone) { + if (has_promisor_remote()) { for_each_packed_object(add_promisor_object, &promisor_objects, FOR_EACH_OBJECT_PROMISOR_ONLY); diff --git a/sha1-file.c b/sha1-file.c index 01cc0590f4..715a2b882a 100644 --- a/sha1-file.c +++ b/sha1-file.c @@ -32,6 +32,7 @@ #include "packfile.h" #include "fetch-object.h" #include "object-store.h" +#include "promisor-remote.h" /* The maximum size for an object header. */ #define MAX_HEADER_LEN 32 @@ -1369,15 +1370,16 @@ int oid_object_info_extended(struct repository *r, const struct object_id *oid, } /* Check if it is a missing object */ - if (fetch_if_missing && repository_format_partial_clone && + if (fetch_if_missing && has_promisor_remote() && !already_retried && r == the_repository) { /* - * TODO Investigate checking fetch_object() return - * TODO value and stopping on error here. - * TODO Pass a repository struct through fetch_object, - * such that arbitrary repositories work. + * TODO Investigate checking promisor_remote_get_direct() + * TODO return value and stopping on error here. + * TODO Pass a repository struct through + * promisor_remote_get_direct(), such that arbitrary + * repositories work. */ - fetch_objects(repository_format_partial_clone, real, 1); + promisor_remote_get_direct(real, 1); already_retried = 1; continue; } diff --git a/t/t5601-clone.sh b/t/t5601-clone.sh index d6948cbdab..dd658f8b32 100755 --- a/t/t5601-clone.sh +++ b/t/t5601-clone.sh @@ -653,7 +653,7 @@ partial_clone () { git -C client fsck && # Ensure that unneeded blobs are not inadvertently fetched. - test_config -C client extensions.partialclone "not a remote" && + test_config -C client remote.origin.promisor "false" && test_must_fail git -C client cat-file -e "$HASH1" && # But this blob was fetched, because clone performs an initial checkout diff --git a/t/t5616-partial-clone.sh b/t/t5616-partial-clone.sh index 9643acb161..e8ca825ab7 100755 --- a/t/t5616-partial-clone.sh +++ b/t/t5616-partial-clone.sh @@ -42,7 +42,7 @@ test_expect_success 'do partial clone 1' ' test_cmp expect_1.oids observed.oids && test "$(git -C pc1 config --local core.repositoryformatversion)" = "1" && - test "$(git -C pc1 config --local extensions.partialclone)" = "origin" && + test "$(git -C pc1 config --local remote.origin.promisor)" = "true" && test "$(git -C pc1 config --local core.partialclonefilter)" = "blob:none" ' diff --git a/unpack-trees.c b/unpack-trees.c index 22c41a3ba8..47353d85c3 100644 --- a/unpack-trees.c +++ b/unpack-trees.c @@ -17,6 +17,7 @@ #include "fsmonitor.h" #include "object-store.h" #include "fetch-object.h" +#include "promisor-remote.h" /* * Error messages expected by scripts out of plumbing commands such as @@ -398,7 +399,7 @@ static int check_updates(struct unpack_trees_options *o) load_gitmodules_file(index, &state); enable_delayed_checkout(&state); - if (repository_format_partial_clone && o->update && !o->dry_run) { + if (has_promisor_remote() && o->update && !o->dry_run) { /* * Prefetch the objects that are to be checked out in the loop * below. @@ -415,8 +416,7 @@ static int check_updates(struct unpack_trees_options *o) } } if (to_fetch.nr) - fetch_objects(repository_format_partial_clone, - to_fetch.oid, to_fetch.nr); + promisor_remote_get_direct(to_fetch.oid, to_fetch.nr); fetch_if_missing = fetch_if_missing_store; oid_array_clear(&to_fetch); } From patchwork Tue Mar 12 13:29:55 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christian Couder X-Patchwork-Id: 10849225 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id F18C617E6 for ; Tue, 12 Mar 2019 13:30:27 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id DBCF12966D for ; Tue, 12 Mar 2019 13:30:27 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id CF09A296A0; Tue, 12 Mar 2019 13:30:27 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 283DA2968A for ; Tue, 12 Mar 2019 13:30:27 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726734AbfCLNa0 (ORCPT ); Tue, 12 Mar 2019 09:30:26 -0400 Received: from mail-ed1-f68.google.com ([209.85.208.68]:38243 "EHLO mail-ed1-f68.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726711AbfCLNaZ (ORCPT ); Tue, 12 Mar 2019 09:30:25 -0400 Received: by mail-ed1-f68.google.com with SMTP id h58so2304111edb.5 for ; Tue, 12 Mar 2019 06:30:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=2vgE8u3JELRG3wC0Dr88hM793h48CEHKwFV0X/e7YFs=; b=F7nIfxL/kC+ZiSbawXOLdAnb7g5nVw40sBC1j7MI7ZOovjV6rO4S26fwiCDQaYViQs YEKNz8Rt9K7p4HEYtdMly3kliy4fOmUPKotiqHZ+1Wp2iVzk75/phPp2tE2ecq0bKNDB njrlJkmnGQpOIlnGNrT3t0xCoTX5JRV1TBc3eD2zs9Nd4mHEhNy0prN9fCbT963n/o8h Q945NldIwEr4ubyw1RUtL+WgqALOaVEz3ER8wUW2u/BtZrFGpp+6+wo9EuEuo98ipbM4 K9F1rdtqhJMvxEqqd5R32yAyRQT58Y0ptipla9n+lPh8ktPccFvLrHhswhRZo50ILVHp lUJQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=2vgE8u3JELRG3wC0Dr88hM793h48CEHKwFV0X/e7YFs=; b=U/Dj1InQR/O1G0CUfqMy1lR6Y8Gqf3R3t5sG4porK0NXezUOIas+BW7thNEJc/3q9d FnxOSdsR2ozLfLd/WLnLedwkvG2+zkNwESQS0XkA4qde1xN+h6q1Cr0ems7myv8CiS7K TKJ7L5Kyzq2mY45mN4qGzzA04H3nhF+kv8Z6XTC4pG9YEk7M2RfdKgAJGrRGpwMGk7fF lUIrTCBSN8E3V804wmpLhFPuIaNtpRA0s8zVmzDaCuxQWBORUo6E7SpSErXBOTxpuK5W uEPyOA4UxUqNbw6yovTcv23v5vT/9fmtNAUAziRDE1hImE93uzn5D3CKq8PC7WOSJQn7 5f6g== X-Gm-Message-State: APjAAAUhSUYtH9tT9m6cFtrIINq5hVFfPeQiPx0eYycg4//+bTj5NqAv w5vQSgPVEpC8pDTMbz7Pp/HTFVyD X-Google-Smtp-Source: APXvYqwvEVJlIOOOosy15pKp/QQlqKp2LC7It29+v9ow2Ut3LDJ/jQZS1UZDGHRKDZD4IYGEzm+7DA== X-Received: by 2002:a17:906:d71:: with SMTP id s17mr24379683ejh.233.1552397422565; Tue, 12 Mar 2019 06:30:22 -0700 (PDT) Received: from ubuntu-N750HU.booking.pcln.com (access-114.38.rev.fr.colt.net. [213.41.38.114]) by smtp.gmail.com with ESMTPSA id h8sm5875213edk.21.2019.03.12.06.30.20 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 12 Mar 2019 06:30:21 -0700 (PDT) From: Christian Couder X-Google-Original-From: Christian Couder To: git@vger.kernel.org Cc: Junio C Hamano , Jeff King , Ben Peart , Jonathan Tan , Jonathan Nieder , Stefan Beller , Nguyen Thai Ngoc Duy , Mike Hommey , Lars Schneider , Eric Wong , Christian Couder , Jeff Hostetler , Eric Sunshine , Beat Bolli Subject: [PATCH v3 07/11] promisor-remote: parse remote.*.partialclonefilter Date: Tue, 12 Mar 2019 14:29:55 +0100 Message-Id: <20190312132959.11764-8-chriscool@tuxfamily.org> X-Mailer: git-send-email 2.21.0.166.gb5e4dbcfd3 In-Reply-To: <20190312132959.11764-1-chriscool@tuxfamily.org> References: <20190312132959.11764-1-chriscool@tuxfamily.org> MIME-Version: 1.0 Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP This makes it possible to specify a different partial clone filter for each promisor remote. Signed-off-by: Christian Couder --- builtin/fetch.c | 2 +- list-objects-filter-options.c | 27 +++++++++++++++------------ list-objects-filter-options.h | 3 ++- promisor-remote.c | 10 ++++++++++ promisor-remote.h | 5 ++++- t/t0410-partial-clone.sh | 2 +- t/t5601-clone.sh | 1 + t/t5616-partial-clone.sh | 2 +- 8 files changed, 35 insertions(+), 17 deletions(-) diff --git a/builtin/fetch.c b/builtin/fetch.c index 7edf70ae6c..9d436bd727 100644 --- a/builtin/fetch.c +++ b/builtin/fetch.c @@ -1491,7 +1491,7 @@ static inline void fetch_one_setup_partial(struct remote *remote) * the config. */ if (!filter_options.choice) - partial_clone_get_default_filter_spec(&filter_options); + partial_clone_get_default_filter_spec(&filter_options, remote->name); return; } diff --git a/list-objects-filter-options.c b/list-objects-filter-options.c index f41a831fce..02f48b7c40 100644 --- a/list-objects-filter-options.c +++ b/list-objects-filter-options.c @@ -30,6 +30,9 @@ static int gently_parse_list_objects_filter( { const char *v0; + if (!arg) + return 0; + if (filter_options->choice) { if (errbuf) { strbuf_addstr( @@ -146,6 +149,7 @@ void partial_clone_register( const struct list_objects_filter_options *filter_options) { char *cfg_name; + char *filter_name; /* Check if it is already registered */ if (!promisor_remote_find(remote)) { @@ -160,27 +164,26 @@ void partial_clone_register( /* * Record the initial filter-spec in the config as * the default for subsequent fetches from this remote. - * - * TODO: record it into remote..partialclonefilter */ - core_partial_clone_filter_default = - xstrdup(filter_options->filter_spec); - git_config_set("core.partialclonefilter", - core_partial_clone_filter_default); + filter_name = xstrfmt("remote.%s.partialclonefilter", remote); + git_config_set(filter_name, filter_options->filter_spec); + free(filter_name); /* Make sure the config info are reset */ promisor_remote_reinit(); } void partial_clone_get_default_filter_spec( - struct list_objects_filter_options *filter_options) + struct list_objects_filter_options *filter_options, + const char *remote) { + struct promisor_remote *promisor = promisor_remote_find(remote); + /* * Parse default value, but silently ignore it if it is invalid. */ - if (!core_partial_clone_filter_default) - return; - gently_parse_list_objects_filter(filter_options, - core_partial_clone_filter_default, - NULL); + if (promisor) + gently_parse_list_objects_filter(filter_options, + promisor->partial_clone_filter, + NULL); } diff --git a/list-objects-filter-options.h b/list-objects-filter-options.h index e3adc78ebf..70d27f44ef 100644 --- a/list-objects-filter-options.h +++ b/list-objects-filter-options.h @@ -89,6 +89,7 @@ void partial_clone_register( const char *remote, const struct list_objects_filter_options *filter_options); void partial_clone_get_default_filter_spec( - struct list_objects_filter_options *filter_options); + struct list_objects_filter_options *filter_options, + const char *remote); #endif /* LIST_OBJECTS_FILTER_OPTIONS_H */ diff --git a/promisor-remote.c b/promisor-remote.c index dcf6ef6521..ed33dfa86c 100644 --- a/promisor-remote.c +++ b/promisor-remote.c @@ -70,6 +70,16 @@ static int promisor_remote_config(const char *var, const char *value, void *data free(remote_name); return 0; } + if (!strcmp(subkey, "partialclonefilter")) { + char *remote_name = xmemdupz(name, namelen); + + o = promisor_remote_look_up(remote_name, NULL); + if (!o) + o = promisor_remote_new(remote_name); + + free(remote_name); + return git_config_string(&o->partial_clone_filter, var, value); + } return 0; } diff --git a/promisor-remote.h b/promisor-remote.h index f96722bc66..f0d609a3f5 100644 --- a/promisor-remote.h +++ b/promisor-remote.h @@ -3,10 +3,13 @@ /* * Promisor remote linked list - * Its information come from remote.XXX config entries. + * + * Information in its fields come from remote.XXX config entries or + * from extensions.partialclone or core.partialclonefilter. */ struct promisor_remote { const char *remote_name; + const char *partial_clone_filter; struct promisor_remote *next; }; diff --git a/t/t0410-partial-clone.sh b/t/t0410-partial-clone.sh index bce02788e6..9266037714 100755 --- a/t/t0410-partial-clone.sh +++ b/t/t0410-partial-clone.sh @@ -26,7 +26,7 @@ promise_and_delete () { test_expect_success 'extensions.partialclone without filter' ' test_create_repo server && git clone --filter="blob:none" "file://$(pwd)/server" client && - git -C client config --unset core.partialclonefilter && + git -C client config --unset remote.origin.partialclonefilter && git -C client fetch origin ' diff --git a/t/t5601-clone.sh b/t/t5601-clone.sh index dd658f8b32..2e82c70f33 100755 --- a/t/t5601-clone.sh +++ b/t/t5601-clone.sh @@ -654,6 +654,7 @@ partial_clone () { # Ensure that unneeded blobs are not inadvertently fetched. test_config -C client remote.origin.promisor "false" && + git -C client config --unset remote.origin.partialclonefilter && test_must_fail git -C client cat-file -e "$HASH1" && # But this blob was fetched, because clone performs an initial checkout diff --git a/t/t5616-partial-clone.sh b/t/t5616-partial-clone.sh index e8ca825ab7..ba0489b41a 100755 --- a/t/t5616-partial-clone.sh +++ b/t/t5616-partial-clone.sh @@ -43,7 +43,7 @@ test_expect_success 'do partial clone 1' ' test_cmp expect_1.oids observed.oids && test "$(git -C pc1 config --local core.repositoryformatversion)" = "1" && test "$(git -C pc1 config --local remote.origin.promisor)" = "true" && - test "$(git -C pc1 config --local core.partialclonefilter)" = "blob:none" + test "$(git -C pc1 config --local remote.origin.partialclonefilter)" = "blob:none" ' # checkout master to force dynamic object fetch of blobs at HEAD. From patchwork Tue Mar 12 13:29:56 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christian Couder X-Patchwork-Id: 10849227 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 9BB7F186E for ; Tue, 12 Mar 2019 13:30:28 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 88BA2295F2 for ; Tue, 12 Mar 2019 13:30:28 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 862EC296AA; Tue, 12 Mar 2019 13:30:28 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 2A1A129652 for ; Tue, 12 Mar 2019 13:30:28 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726738AbfCLNa1 (ORCPT ); Tue, 12 Mar 2019 09:30:27 -0400 Received: from mail-ed1-f65.google.com ([209.85.208.65]:42659 "EHLO mail-ed1-f65.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725774AbfCLNa0 (ORCPT ); Tue, 12 Mar 2019 09:30:26 -0400 Received: by mail-ed1-f65.google.com with SMTP id j89so2272684edb.9 for ; Tue, 12 Mar 2019 06:30:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=AhApy9Y6AQjlxzwFR3k6fXwn9HT58J9/Qy4FTI7TZp8=; b=p5JC5019QqYr4S0IX3rw/TdaTPFgEHnDq7eZPm0NVPG5Bn+s+5smMP7zPrWwQ3UEUg xZzK3yt0Lt729JCpIBPnyB0B4gGREW67Dj4UHsh680LILdToIWCLTqJJesk5cpPciucP FPoNX1Lpg23oO4/WM9HxRdlseIuaUFqH0F3RCoJNJdVWiADuz6UbEAEcLOkUyzEDw3nv Reat+kEAkHpIztPaeS1dprtwbuoMJeskMXJNl5kkYrqk96oObGO7tqfTfhiLJbYGrmQC NNqNr1Y6V2cxQCicNGRZjss2Eenm34qY+lR2BzGE6J5JKwREa8iUrR39IJhFlMTDLMzs h6Fg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=AhApy9Y6AQjlxzwFR3k6fXwn9HT58J9/Qy4FTI7TZp8=; b=cc7N6zBJpHrTlJRQ8K4rvjzWNx4IxQoYMvuP/xveCkr2ybZ3YCTa2T10mBT/VPCOev 0QhAudDxveCcLq0wwG3/7Z09XmI/sohK/1/FTUqT6ICLR98aEFCc05bxIba/FhLaPUhd QVSeUTBBVZZsWWd/fuhnPyo3s3IxqY3BXTj/hUHg+ZfAJ8mqP1cQjV951bdAOubrcJDL JD6Vej8wlvAL5UQAq5YwfbIKws1KBgHZuQLxadXl5dnjYD3PLrpVl7RsorBuDZpku/Yi DKoJN03UArWv6IkDloG5jgZ+TJ/F5PNXCJEgGZRkr1VE30uCRGP5mph/ocZ1PJqesVlB IrpA== X-Gm-Message-State: APjAAAXQazi1FoD/5kf1dmboIk6vl2sqcU5o0tCT9YDMzeJuvjy4R/U+ CC23l32kQgIoFrJjtr/rZzXHK8QP X-Google-Smtp-Source: APXvYqzCpgQM5ADkaeaw9PRMdL71XzwXkxA23vnvWOFJDwVuSziPzE29n1IlseipL0tk/CwIgnh8Ag== X-Received: by 2002:a17:906:4595:: with SMTP id t21mr24688180ejq.120.1552397424039; Tue, 12 Mar 2019 06:30:24 -0700 (PDT) Received: from ubuntu-N750HU.booking.pcln.com (access-114.38.rev.fr.colt.net. [213.41.38.114]) by smtp.gmail.com with ESMTPSA id h8sm5875213edk.21.2019.03.12.06.30.22 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 12 Mar 2019 06:30:23 -0700 (PDT) From: Christian Couder X-Google-Original-From: Christian Couder To: git@vger.kernel.org Cc: Junio C Hamano , Jeff King , Ben Peart , Jonathan Tan , Jonathan Nieder , Stefan Beller , Nguyen Thai Ngoc Duy , Mike Hommey , Lars Schneider , Eric Wong , Christian Couder , Jeff Hostetler , Eric Sunshine , Beat Bolli Subject: [PATCH v3 08/11] builtin/fetch: remove unique promisor remote limitation Date: Tue, 12 Mar 2019 14:29:56 +0100 Message-Id: <20190312132959.11764-9-chriscool@tuxfamily.org> X-Mailer: git-send-email 2.21.0.166.gb5e4dbcfd3 In-Reply-To: <20190312132959.11764-1-chriscool@tuxfamily.org> References: <20190312132959.11764-1-chriscool@tuxfamily.org> MIME-Version: 1.0 Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP As the infrastructure for more than one promisor remote has been introduced in previous patches, we can remove code that forbids the registration of more than one promisor remote. Signed-off-by: Christian Couder --- builtin/fetch.c | 20 +++++--------------- 1 file changed, 5 insertions(+), 15 deletions(-) diff --git a/builtin/fetch.c b/builtin/fetch.c index 9d436bd727..56c6827165 100644 --- a/builtin/fetch.c +++ b/builtin/fetch.c @@ -1465,26 +1465,16 @@ static inline void fetch_one_setup_partial(struct remote *remote) return; /* - * If this is the FIRST partial-fetch request, we enable partial - * on this repo and remember the given filter-spec as the default - * for subsequent fetches to this remote. + * If this is a partial-fetch request, we enable partial on + * this repo if not already enabled and remember the given + * filter-spec as the default for subsequent fetches to this + * remote. */ - if (!has_promisor_remote() && filter_options.choice) { + if (filter_options.choice) { partial_clone_register(remote->name, &filter_options); return; } - /* - * We are currently limited to only ONE promisor remote and only - * allow partial-fetches from the promisor remote. - */ - if (!promisor_remote_find(remote->name)) { - if (filter_options.choice) - die(_("--filter can only be used with the remote " - "configured in extensions.partialClone")); - return; - } - /* * Do a partial-fetch from the promisor remote using either the * explicitly given filter-spec or inherit the filter-spec from From patchwork Tue Mar 12 13:29:57 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christian Couder X-Patchwork-Id: 10849231 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 413CE186E for ; Tue, 12 Mar 2019 13:30:31 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 2DF702962E for ; Tue, 12 Mar 2019 13:30:31 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 2C1B229634; Tue, 12 Mar 2019 13:30:31 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B66AE296E9 for ; Tue, 12 Mar 2019 13:30:30 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726750AbfCLNa3 (ORCPT ); Tue, 12 Mar 2019 09:30:29 -0400 Received: from mail-ed1-f66.google.com ([209.85.208.66]:40238 "EHLO mail-ed1-f66.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726733AbfCLNa1 (ORCPT ); Tue, 12 Mar 2019 09:30:27 -0400 Received: by mail-ed1-f66.google.com with SMTP id r23so2285151edm.7 for ; Tue, 12 Mar 2019 06:30:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=UMxtXHPYeEAsRXbsaHEyWhDNJtC66wNT+1EoAbigC1s=; b=G9na3Fl5QLsD8F1kgHr/qvkzacQeT+rPOMduvxOIp2s6VZua8U3MkXOy8XGM0jnt7B iKsqhxVh76DrKSfEuMQ9K4Uh1a+ijxSw3oW1STgDxyBz5ZE5l3cNOSl7hSinTy5F473A HloZkWy8LmRrIEE8PNFFQc5hj3w3emydcUy/pmXLXYiPtNyl9wRD/1HhmM1uZogKRHQB uS9hSAiSF8NL744QZBE0njNQRSJEXtpOu/8OAoSauo0r4yCcQjDY4tAxYvcsKcYg0i4C 0vIfGYW8Pnz4ajqr+mokYBputVLJRIYfXLoDr0oOw4vKggAVORDxygAfmHdJVtH7X2iq iaxQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=UMxtXHPYeEAsRXbsaHEyWhDNJtC66wNT+1EoAbigC1s=; b=g8dUQJ4VweJl+MABMHy3BV6IaQGHT0hLNM+Fjh5tfcv9XwM7zG2lqlo6jDs+c7StRC /RQRz2y1OsImw4iz45UZp2PYJH0vX7ryzGFQVlImXMHvzqj5gI8yg2UdVewKFTjJ2R81 2gmJZFT0iRt2QtJ21TOo/FkL7c3bUPDHdotph64U+PnryQpYQSUjq1HSau6U1XmdrxAQ 8E/aY40K9o9M9LwUVVa4dn+lmamy49vEdXrwjNjwAV/n/XBIPB676yIE7Ptz/2dT4EZM kP0hbNkbupSWXqfttxzZhA4Hdrte/aemhYJidnuKNuwJINO8p3Y56RLr3Akna/oKJGbq CqGQ== X-Gm-Message-State: APjAAAWIkgWDr4MJlrKFyXlXzJQ89oJPOPweVA9hKoCgXISWMtNFEKTK 3o7+sxdgLJrgIhe1M5VH0uxiidqv X-Google-Smtp-Source: APXvYqyuYk7lbP+6MzM6B/vBcQw5Oidi2HO9vyy+F9mfmAoB5B8Iok/ZuV3i6o5xwAyfQWzIW+5XrA== X-Received: by 2002:a17:906:1156:: with SMTP id i22mr25719654eja.13.1552397425182; Tue, 12 Mar 2019 06:30:25 -0700 (PDT) Received: from ubuntu-N750HU.booking.pcln.com (access-114.38.rev.fr.colt.net. [213.41.38.114]) by smtp.gmail.com with ESMTPSA id h8sm5875213edk.21.2019.03.12.06.30.24 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 12 Mar 2019 06:30:24 -0700 (PDT) From: Christian Couder X-Google-Original-From: Christian Couder To: git@vger.kernel.org Cc: Junio C Hamano , Jeff King , Ben Peart , Jonathan Tan , Jonathan Nieder , Stefan Beller , Nguyen Thai Ngoc Duy , Mike Hommey , Lars Schneider , Eric Wong , Christian Couder , Jeff Hostetler , Eric Sunshine , Beat Bolli , Christian Couder Subject: [PATCH v3 09/11] t0410: test fetching from many promisor remotes Date: Tue, 12 Mar 2019 14:29:57 +0100 Message-Id: <20190312132959.11764-10-chriscool@tuxfamily.org> X-Mailer: git-send-email 2.21.0.166.gb5e4dbcfd3 In-Reply-To: <20190312132959.11764-1-chriscool@tuxfamily.org> References: <20190312132959.11764-1-chriscool@tuxfamily.org> MIME-Version: 1.0 Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Christian Couder This shows that it is now possible to fetch objects from more than one promisor remote, and that fetching from a new promisor remote can configure it as one. Signed-off-by: Christian Couder --- t/t0410-partial-clone.sh | 47 +++++++++++++++++++++++++++++++++++++++- 1 file changed, 46 insertions(+), 1 deletion(-) diff --git a/t/t0410-partial-clone.sh b/t/t0410-partial-clone.sh index 9266037714..146b0a1e03 100755 --- a/t/t0410-partial-clone.sh +++ b/t/t0410-partial-clone.sh @@ -182,8 +182,53 @@ test_expect_success 'fetching of missing objects works with ref-in-want enabled' grep "git< fetch=.*ref-in-want" trace ' +test_expect_success 'fetching of missing objects from another promisor remote' ' + git clone "file://$(pwd)/server" server2 && + test_commit -C server2 bar && + git -C server2 repack -a -d --write-bitmap-index && + HASH2=$(git -C server2 rev-parse bar) && + + git -C repo remote add server2 "file://$(pwd)/server2" && + git -C repo config remote.server2.promisor true && + git -C repo cat-file -p "$HASH2" && + + git -C repo fetch server2 && + rm -rf repo/.git/objects/* && + git -C repo cat-file -p "$HASH2" && + + # Ensure that the .promisor file is written, and check that its + # associated packfile contains the object + ls repo/.git/objects/pack/pack-*.promisor >promisorlist && + test_line_count = 1 promisorlist && + IDX=$(cat promisorlist | sed "s/promisor$/idx/") && + git verify-pack --verbose "$IDX" | grep "$HASH2" +' + +test_expect_success 'fetching of missing objects configures a promisor remote' ' + git clone "file://$(pwd)/server" server3 && + test_commit -C server3 baz && + git -C server3 repack -a -d --write-bitmap-index && + HASH3=$(git -C server3 rev-parse baz) && + git -C server3 config uploadpack.allowfilter 1 && + + rm repo/.git/objects/pack/pack-*.promisor && + + git -C repo remote add server3 "file://$(pwd)/server3" && + git -C repo fetch --filter="blob:none" server3 $HASH3 && + + test "$(git -C repo config remote.server3.promisor)" = "true" && + + # Ensure that the .promisor file is written, and check that its + # associated packfile contains the object + ls repo/.git/objects/pack/pack-*.promisor >promisorlist && + test_line_count = 1 promisorlist && + IDX=$(cat promisorlist | sed "s/promisor$/idx/") && + git verify-pack --verbose "$IDX" | grep "$HASH3" +' + test_expect_success 'fetching of missing blobs works' ' - rm -rf server repo && + rm -rf server server2 repo && + rm -rf server server3 repo && test_create_repo server && test_commit -C server foo && git -C server repack -a -d --write-bitmap-index && From patchwork Tue Mar 12 13:29:58 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christian Couder X-Patchwork-Id: 10849233 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id EEA9317E6 for ; Tue, 12 Mar 2019 13:30:32 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D8CF62964D for ; Tue, 12 Mar 2019 13:30:32 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id D687A2963C; Tue, 12 Mar 2019 13:30:32 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 2FCC929699 for ; Tue, 12 Mar 2019 13:30:32 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726754AbfCLNab (ORCPT ); Tue, 12 Mar 2019 09:30:31 -0400 Received: from mail-ed1-f65.google.com ([209.85.208.65]:34989 "EHLO mail-ed1-f65.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725774AbfCLNa3 (ORCPT ); Tue, 12 Mar 2019 09:30:29 -0400 Received: by mail-ed1-f65.google.com with SMTP id g19so2311744edp.2 for ; Tue, 12 Mar 2019 06:30:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Z8inGzJVVs3vWJ2vLGz2G3C7Zs7OmL5oh2wy/bfdAG8=; b=l0bhlv/9KEDWwXfO1UdoghsWejmhL2xRMBLceVbDl6O9T3eYrzEXL3em2HbvT9GqpG uNr+Y3R+cQdKEcXWaHl3KKrBfeIjxeC3N7Gj+F2zf7KtyCZ21y5sEyWJDxDIfLFQVzS9 isC/Wdi+CcFwfrwg2gEoAPVUNFh4A5CGKbJRncUrwdaAFrrrkIF5kivqWRvrC0uCduwW 6kPktRAl2v2wT2H7pJRzdhpT9VxJiY6Z53YLJficT8XAoAzsV7dBGkBuunkhRW3fC5Yf TODg3rZzC2O/UKav8fsBpzWKoGpM1T+EuaRjn8er+nNeemLmqs6RxU71f6yVH5qAShtT HSjQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Z8inGzJVVs3vWJ2vLGz2G3C7Zs7OmL5oh2wy/bfdAG8=; b=kYLoyJWu/Fo6L/PZyAENxG2V7XA1mmKLgLB1fqYY3EoThU6/5D7hUJEmlZSbI8J40w s+Is0q6OEQUEwR/eZ+wpl5c6D/yuS8M9nsucxOuKA8MkYV5o+66D0/4otTQZxEvZxuxT BLcH0SoYyK6+CGe4aOg1OPxsyLu2wMvKzUP3Kxm4/N2Usk7S7O8Hbmze9l/hQsXdEz/F 8DCh3VxWoOo4mBM5MhR6J/HNtNJn7/BrqXizuhnuq1VcXthvpCuUQLC8DAH6b1pvbI4y H0ca8UI6/SssaG0m9W8rEn0ubKvXRSOqXJmXBclBT2oVKvwwxw1pf1g+BJkVfc7eZQXf ptgw== X-Gm-Message-State: APjAAAU1byayreVwGoQgHdBhZLRPZavZ/Kvke8zF02mbUne1H8MOMdLh NoktgufB9cCNJHXDKLNezLjOGc9+ X-Google-Smtp-Source: APXvYqzb20Qcngab1LQRrvjecTVjLKPf6noX+GmHFhDWtxYEnjDZwFzSzqkrec/bZNg97vGIQz4BKA== X-Received: by 2002:a50:b527:: with SMTP id y36mr3529958edd.83.1552397426174; Tue, 12 Mar 2019 06:30:26 -0700 (PDT) Received: from ubuntu-N750HU.booking.pcln.com (access-114.38.rev.fr.colt.net. [213.41.38.114]) by smtp.gmail.com with ESMTPSA id h8sm5875213edk.21.2019.03.12.06.30.25 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 12 Mar 2019 06:30:25 -0700 (PDT) From: Christian Couder X-Google-Original-From: Christian Couder To: git@vger.kernel.org Cc: Junio C Hamano , Jeff King , Ben Peart , Jonathan Tan , Jonathan Nieder , Stefan Beller , Nguyen Thai Ngoc Duy , Mike Hommey , Lars Schneider , Eric Wong , Christian Couder , Jeff Hostetler , Eric Sunshine , Beat Bolli Subject: [PATCH v3 10/11] partial-clone: add multiple remotes in the doc Date: Tue, 12 Mar 2019 14:29:58 +0100 Message-Id: <20190312132959.11764-11-chriscool@tuxfamily.org> X-Mailer: git-send-email 2.21.0.166.gb5e4dbcfd3 In-Reply-To: <20190312132959.11764-1-chriscool@tuxfamily.org> References: <20190312132959.11764-1-chriscool@tuxfamily.org> MIME-Version: 1.0 Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP While at it, let's remove a reference to ODB effort as the ODB effort has been replaced by directly enhancing partial clone and promisor remote features. Signed-off-by: Christian Couder --- Documentation/technical/partial-clone.txt | 83 ++++++++++++++++------- 1 file changed, 58 insertions(+), 25 deletions(-) diff --git a/Documentation/technical/partial-clone.txt b/Documentation/technical/partial-clone.txt index 896c7b3878..58adcc5ce1 100644 --- a/Documentation/technical/partial-clone.txt +++ b/Documentation/technical/partial-clone.txt @@ -100,18 +100,18 @@ or commits that reference missing trees. Handling Missing Objects ------------------------ -- An object may be missing due to a partial clone or fetch, or missing due - to repository corruption. To differentiate these cases, the local - repository specially indicates such filtered packfiles obtained from the - promisor remote as "promisor packfiles". +- An object may be missing due to a partial clone or fetch, or missing + due to repository corruption. To differentiate these cases, the + local repository specially indicates such filtered packfiles + obtained from promisor remotes as "promisor packfiles". + These promisor packfiles consist of a ".promisor" file with arbitrary contents (like the ".keep" files), in addition to their ".pack" and ".idx" files. - The local repository considers a "promisor object" to be an object that - it knows (to the best of its ability) that the promisor remote has promised - that it has, either because the local repository has that object in one of + it knows (to the best of its ability) that promisor remotes have promised + that they have, either because the local repository has that object in one of its promisor packfiles, or because another promisor object refers to it. + When Git encounters a missing object, Git can see if it is a promisor object @@ -123,7 +123,7 @@ expensive-to-modify list of missing objects.[a] - Since almost all Git code currently expects any referenced object to be present locally and because we do not want to force every command to do a dry-run first, a fallback mechanism is added to allow Git to attempt - to dynamically fetch missing objects from the promisor remote. + to dynamically fetch missing objects from promisor remotes. + When the normal object lookup fails to find an object, Git invokes fetch-object to try to get the object from the server and then retry @@ -157,8 +157,7 @@ and prefetch those objects in bulk. + We are not happy with this global variable and would like to remove it, but that requires significant refactoring of the object code to pass an -additional flag. We hope that concurrent efforts to add an ODB API can -encompass this. +additional flag. Fetching Missing Objects @@ -182,21 +181,53 @@ has been updated to not use any object flags when the corresponding argument though they are not necessary. +Using many promisor remotes +--------------------------- + +Many promisor remotes can be configured and used. + +This allows for example a user to have multiple geographically-close +cache servers for fetching missing blobs while continuing to do +filtered `git-fetch` commands from the central server. + +When fetching objects, promisor remotes are tried one after the other +until all the objects have been fetched. + +Remotes that are considered "promisor" remotes are those specified by +the following configuration variables: + +- `extensions.partialClone = ` + +- `remote..promisor = true` + +- `remote..partialCloneFilter = ...` + +Only one promisor remote can be configured using the +`extensions.partialClone` config variable. This promisor remote will +be the last one tried when fetching objects. + +The other promisor remotes will be tried in the order they appear in +the config file. + Current Limitations ------------------- -- The remote used for a partial clone (or the first partial fetch - following a regular clone) is marked as the "promisor remote". +- It is not possible to specify the order in which the promisor + remotes are tried in other ways than the order in which they appear + in the config file. + -We are currently limited to a single promisor remote and only that -remote may be used for subsequent partial fetches. +It is also not possible to specify an order to be used when fetching +from one remote and a different order when fetching from another +remote. + +- It is not possible to push only specific objects to a promisor + remote. + -We accept this limitation because we believe initial users of this -feature will be using it on repositories with a strong single central -server. +It is not possible to push at the same time to multiple promisor +remote in a specific order. -- Dynamic object fetching will only ask the promisor remote for missing - objects. We assume that the promisor remote has a complete view of the +- Dynamic object fetching will only ask promisor remotes for missing + objects. We assume that promisor remotes have a complete view of the repository and can satisfy all such requests. - Repack essentially treats promisor and non-promisor packfiles as 2 @@ -218,15 +249,17 @@ server. Future Work ----------- -- Allow more than one promisor remote and define a strategy for fetching - missing objects from specific promisor remotes or of iterating over the - set of promisor remotes until a missing object is found. +- Improve the way to specify the order in which promisor remotes are + tried. + -A user might want to have multiple geographically-close cache servers -for fetching missing blobs while continuing to do filtered `git-fetch` -commands from the central server, for example. +For example this could allow to specify explicitly something like: +"When fetching from this remote, I want to use these promisor remotes +in this order, though, when pushing or fetching to that remote, I want +to use those promisor remotes in that order." + +- Allow pushing to promisor remotes. + -Or the user might want to work in a triangular work flow with multiple +The user might want to work in a triangular work flow with multiple promisor remotes that each have an incomplete view of the repository. - Allow repack to work on promisor packfiles (while keeping them distinct From patchwork Tue Mar 12 13:29:59 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christian Couder X-Patchwork-Id: 10849235 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id E1BAC17E6 for ; Tue, 12 Mar 2019 13:30:34 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id CECA42964E for ; Tue, 12 Mar 2019 13:30:34 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id C305929574; Tue, 12 Mar 2019 13:30:34 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6789B29574 for ; Tue, 12 Mar 2019 13:30:33 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726758AbfCLNac (ORCPT ); Tue, 12 Mar 2019 09:30:32 -0400 Received: from mail-ed1-f68.google.com ([209.85.208.68]:46380 "EHLO mail-ed1-f68.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726284AbfCLNa3 (ORCPT ); Tue, 12 Mar 2019 09:30:29 -0400 Received: by mail-ed1-f68.google.com with SMTP id n17so2255824edt.13 for ; Tue, 12 Mar 2019 06:30:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=ZlFfQvdTesOwcJ+FpZBsUDW33v8Ne5TN7SitZEfxQgw=; b=sQNLhifI8nnsxZC7S2yXlzM9B3CIuT9zyoZKW3v8VcGoAv6k5o+L0V3OCu6QF0Oq5a YIixbvxKP9fSVVH+r/SfrbGf+iVE8PyF/zZEz4iPkOzgZF1hgZEU5bOvXs5YlbDY+3l5 IvTpijslk2X+SX5Pqzhf9VLurzNfWTH3qvt39f3UBcrINxN+77bSiHlWWwP+1UFTCKgs 590Bs7bIbS1qEDA0CbMKJmTIJNx1F8ejM5P+nlUKHDOhLG2NUFHuvDy8rH5ZUSpN9ISD da0gl5XDqC/9X7n1FtM+aBvaumOCB8wFa0ple6kRXKWMvdHv3hk5GSQlT2WyUPDnQulu TIrA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=ZlFfQvdTesOwcJ+FpZBsUDW33v8Ne5TN7SitZEfxQgw=; b=X8DjlCw93esUPRv0kMFpWBps1y3fNDs70S0qF/uD/fADf11MWyreH3zXOcJfSbo2Ah 9Ez3uZbX4ZUPaMJzKTKnkuZ+JUNuM+QFtoeQ6cm9GQDkRdjw8s3lmsfyF+isH1b+KRUd SpRD2uSS6JWe56zsR5NUNGACAEZRLcd7QuMsb0mSYBVtTImVgfsdQHIljBW6ncqTbHnM oZ2iLk9RAm97oLlnvqFab4qZafUkVL6kaFMVbT2AOjycoDRpHe/plJPcFopXT2CxHFtY 9+U5WmdJHUzoHJU+4Tol6HpKMaOcotvHdyalO7fMGQ+1eRZV4MkZWV/Az/f+IGyjT1NG tdpg== X-Gm-Message-State: APjAAAWpOr4tPAqPHvIRNI0uzbFLcUg1lsyw5WSJdISCW1ADNUBMsTBj m7pcqZkBwq+OBQPZFu+lt+alWfV8 X-Google-Smtp-Source: APXvYqy45ZL/hwDmIbtzUdF2rCzL+jXuMapKws/oQlYbKYyA9u/XNJcDA68CBGaOgEsdf/ghoKQOEg== X-Received: by 2002:a17:906:49c7:: with SMTP id w7mr26071396ejv.226.1552397427900; Tue, 12 Mar 2019 06:30:27 -0700 (PDT) Received: from ubuntu-N750HU.booking.pcln.com (access-114.38.rev.fr.colt.net. [213.41.38.114]) by smtp.gmail.com with ESMTPSA id h8sm5875213edk.21.2019.03.12.06.30.26 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 12 Mar 2019 06:30:26 -0700 (PDT) From: Christian Couder X-Google-Original-From: Christian Couder To: git@vger.kernel.org Cc: Junio C Hamano , Jeff King , Ben Peart , Jonathan Tan , Jonathan Nieder , Stefan Beller , Nguyen Thai Ngoc Duy , Mike Hommey , Lars Schneider , Eric Wong , Christian Couder , Jeff Hostetler , Eric Sunshine , Beat Bolli Subject: [PATCH v3 11/11] remote: add promisor and partial clone config to the doc Date: Tue, 12 Mar 2019 14:29:59 +0100 Message-Id: <20190312132959.11764-12-chriscool@tuxfamily.org> X-Mailer: git-send-email 2.21.0.166.gb5e4dbcfd3 In-Reply-To: <20190312132959.11764-1-chriscool@tuxfamily.org> References: <20190312132959.11764-1-chriscool@tuxfamily.org> MIME-Version: 1.0 Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Signed-off-by: Christian Couder --- Documentation/config/remote.txt | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/Documentation/config/remote.txt b/Documentation/config/remote.txt index 6c4cad83a2..a8e6437a90 100644 --- a/Documentation/config/remote.txt +++ b/Documentation/config/remote.txt @@ -76,3 +76,11 @@ remote..pruneTags:: + See also `remote..prune` and the PRUNING section of linkgit:git-fetch[1]. + +remote..promisor:: + When set to true, this remote will be used to fetch promisor + objects. + +remote..partialclonefilter:: + The filter that will be applied when fetching from this + promisor remote.