From patchwork Tue Sep 25 11:53:34 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christian Couder X-Patchwork-Id: 10613873 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B13DE14DA for ; Tue, 25 Sep 2018 11:54:23 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A208D298B3 for ; Tue, 25 Sep 2018 11:54:23 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 95D6629A16; Tue, 25 Sep 2018 11:54:23 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 853A829A02 for ; Tue, 25 Sep 2018 11:54:18 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728584AbeIYSBP (ORCPT ); Tue, 25 Sep 2018 14:01:15 -0400 Received: from mail-wr1-f65.google.com ([209.85.221.65]:32850 "EHLO mail-wr1-f65.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727507AbeIYSBO (ORCPT ); Tue, 25 Sep 2018 14:01:14 -0400 Received: by mail-wr1-f65.google.com with SMTP id f10-v6so2019453wrs.0 for ; Tue, 25 Sep 2018 04:54:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=l8Za3NKR6YbHuyVKGcB9opNA75jNvx1ISXPuaGiDZUI=; b=IuEfQv0Ee7SAc++CIA4W1l83zbzfp8X8SO0GlGj87old60gkSzCpeZGME+RYxZfPC4 9dVA0/zf9kPB2dA9m11biBnlXU/1JQ5Ac97/toFxEy8MQILqFYALPcWv5Yuo3C78fsam H9HJl1tmlzNSkK86qEX6yzJa11FofHEB/AIncfLlp7qk10pujqEpV4mq6SEPaBma2S0C COGuqDT63dFrmF8UlDZAsHARRGNpnVHKmpDRTuojIdieorFw0b98yZ1Td58XqcQxN6NL 5rAlGk7sIWXsSX7Er3JSXDxZuPtbE7lrJx5Igq7Sk1Ano6UXEXGu0VFIHAFG0PGQQWvy RidA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=l8Za3NKR6YbHuyVKGcB9opNA75jNvx1ISXPuaGiDZUI=; b=pR7XIcSJJwq0GIHe3xyOCjqndCCs1mGVmrKSI46AAhVrSaUTdq3lCfW+i2qXdNGnaR 799mA0GsARl6dZ3e25hknyJ+3/v6ApCv7kigcePR1hy2ygmjoAsTn/e2Sl9wC7nENbAA rXJO1comfS85zlZQRERCdSXufo77HulvG58rRvl7O4kdnZhstzrH996q9Vg3pWThTSUb 4HDJL/wuAKQtb0IGXKJDRvywVBvzKA6/wXNfBgmNn3cF7FPErUS2iF/yQSVkieDxpk6g wSy2LGjJoDxcZOAp2Yy7iwwEsDeODvY8bmhocxtzd9om8dhu/3uE20zJvFyO18unT6sT rvgg== X-Gm-Message-State: ABuFfogMxDn3ImzzW4TfPdwD62c4wST53/iui/PLtTD9xT32AtuYBQ1Z 95iXx1NbRNdTxRCfhxNjc1jFbSyV X-Google-Smtp-Source: ACcGV63w/CRCue7ucxYiJkZjjZCuu1bVFuQ2uB8dksJz6s7vqudRN59YJ4IBXkSgsgnF99twCSuZTg== X-Received: by 2002:adf:df09:: with SMTP id y9-v6mr633088wrl.26.1537876442444; Tue, 25 Sep 2018 04:54:02 -0700 (PDT) Received: from localhost.localdomain (89-95-107-230.abo.bbox.fr. [89.95.107.230]) by smtp.gmail.com with ESMTPSA id z14-v6sm1998628wrr.91.2018.09.25.04.54.00 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 25 Sep 2018 04:54:01 -0700 (PDT) From: Christian Couder X-Google-Original-From: Christian Couder To: git@vger.kernel.org Cc: Junio C Hamano , Jeff King , Ben Peart , Jonathan Tan , Nguyen Thai Ngoc Duy , Mike Hommey , Lars Schneider , Eric Wong , Christian Couder , Jeff Hostetler , Eric Sunshine , Beat Bolli , Christian Couder Subject: [PATCH v5 1/8] fetch-object: make functions return an error code Date: Tue, 25 Sep 2018 13:53:34 +0200 Message-Id: <20180925115341.19248-2-chriscool@tuxfamily.org> X-Mailer: git-send-email 2.19.0.278.gca5b891cac In-Reply-To: <20180925115341.19248-1-chriscool@tuxfamily.org> References: <20180925115341.19248-1-chriscool@tuxfamily.org> MIME-Version: 1.0 Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Christian Couder The callers of the fetch_object() and fetch_objects() might be interested in knowing if these functions succeeded or not. Signed-off-by: Christian Couder Signed-off-by: Junio C Hamano --- fetch-object.c | 13 ++++++++----- fetch-object.h | 4 ++-- sha1-file.c | 4 ++-- 3 files changed, 12 insertions(+), 9 deletions(-) diff --git a/fetch-object.c b/fetch-object.c index 4266548800..eac4d448ef 100644 --- a/fetch-object.c +++ b/fetch-object.c @@ -5,11 +5,12 @@ #include "transport.h" #include "fetch-object.h" -static void fetch_refs(const char *remote_name, struct ref *ref) +static int fetch_refs(const char *remote_name, struct ref *ref) { struct remote *remote; struct transport *transport; int original_fetch_if_missing = fetch_if_missing; + int res; fetch_if_missing = 0; remote = remote_get(remote_name); @@ -19,12 +20,14 @@ static void fetch_refs(const char *remote_name, struct ref *ref) transport_set_option(transport, TRANS_OPT_FROM_PROMISOR, "1"); transport_set_option(transport, TRANS_OPT_NO_DEPENDENTS, "1"); - transport_fetch_refs(transport, ref); + res = transport_fetch_refs(transport, ref); fetch_if_missing = original_fetch_if_missing; + + return res; } -void fetch_objects(const char *remote_name, const struct object_id *oids, - int oid_nr) +int fetch_objects(const char *remote_name, const struct object_id *oids, + int oid_nr) { struct ref *ref = NULL; int i; @@ -36,5 +39,5 @@ void fetch_objects(const char *remote_name, const struct object_id *oids, new_ref->next = ref; ref = new_ref; } - fetch_refs(remote_name, ref); + return fetch_refs(remote_name, ref); } diff --git a/fetch-object.h b/fetch-object.h index d2f996d4e8..8cc8c14b9d 100644 --- a/fetch-object.h +++ b/fetch-object.h @@ -1,7 +1,7 @@ #ifndef FETCH_OBJECT_H #define FETCH_OBJECT_H -void fetch_objects(const char *remote_name, const struct object_id *oids, - int oid_nr); +int fetch_objects(const char *remote_name, const struct object_id *oids, + int oid_nr); #endif diff --git a/sha1-file.c b/sha1-file.c index a4367b8f04..5c44873d37 100644 --- a/sha1-file.c +++ b/sha1-file.c @@ -1312,8 +1312,8 @@ int oid_object_info_extended(struct repository *r, const struct object_id *oid, if (fetch_if_missing && repository_format_partial_clone && !already_retried && r == the_repository) { /* - * TODO Investigate having fetch_object() return - * TODO error/success and stopping the music here. + * TODO Investigate checking fetch_object() return + * TODO value and stopping on error here. * TODO Pass a repository struct through fetch_object, * such that arbitrary repositories work. */ From patchwork Tue Sep 25 11:53:35 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christian Couder X-Patchwork-Id: 10613869 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 004A214DA for ; Tue, 25 Sep 2018 11:54:20 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E46FC29A16 for ; Tue, 25 Sep 2018 11:54:19 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id D7A9129DD0; Tue, 25 Sep 2018 11:54:19 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A883229A16 for ; Tue, 25 Sep 2018 11:54:18 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728659AbeIYSBR (ORCPT ); Tue, 25 Sep 2018 14:01:17 -0400 Received: from mail-wr1-f65.google.com ([209.85.221.65]:43132 "EHLO mail-wr1-f65.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727507AbeIYSBR (ORCPT ); Tue, 25 Sep 2018 14:01:17 -0400 Received: by mail-wr1-f65.google.com with SMTP id z14-v6so5539872wrs.10 for ; Tue, 25 Sep 2018 04:54:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=VPeP8i6bKrpA+3592AD3e1ZeNEnOBey+uQBJ2lCdL6g=; b=pR9LpnxS8bui2VWDgLA3krUI6V4m6ECe5/z+iCN5TZCp4Ga/Nj8w/l56tHDCBj9LIa FSHhcVlAyyRduMbdNPwD2GcQTCZDmxCvO+lTta3hdck3TbREejssda+mmz6PSXhyKfB0 q24FF+eoRBJ2FpuWZLyWKBdzONvMXDGKfiAJ5oHVhsRyrW3HYJZEzB6bjIhVCx8I2uRm GUiRZQRnCUGdBXgznQhx3Y2k5/ebdfRoPBNRKt9JCPcZTeeAODiC5dVbCJXFSZdwZs/a GozHMweAaYj8P6mbn9VjN8Xc03r9HyF+TJn1LThNS8H2crOvE1Kxy7CyTN18fSmXqYrg 6ovA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=VPeP8i6bKrpA+3592AD3e1ZeNEnOBey+uQBJ2lCdL6g=; b=JoX4zFfHzduRdGJ8Qb4DuyBo1u7cPYds5DfPjvmRTMnu0ZF3dyXg5C3ICGlml1BukK cyc2oNtSfgF7PY8Qfjh4RSEpyq/ygzH/ZmCK+m3DPXDwQs0p3zk/hhtAYJGKcxH3gRKU 9EFD094rXxaHq+T430xyFhIk8GS8isyo+hHVOwJxRto5DoOCErzYfoMjJO0rcCDq/oy+ bENQIDSeorJ2n2MUOUkn9sMJamYQwRN2hJ+wUSTGfI9vzDrYbueu2/9GXjZMxiNI2RLd 3QiBTz9MYPgk1x1slUKbeczdTldrirrQPHdlKxgZ5T52mIfB6Lq3OHbVJ5CheJTO0rZ1 aeaw== X-Gm-Message-State: ABuFfohe8PpwPv9oU4yweBE0ii3EXE7Tx9rA2ZGc/qHMDGWvNgFqluQi aK9llPFtWY35gtbpCg5fmVWtb1pn X-Google-Smtp-Source: ACcGV61tE3gA7OE70MoSzDKPoQhEie+Tl222Goxds7NJbzEvMPxSrNVmzl9WRNGC4XY8ipncnzGAIw== X-Received: by 2002:a5d:4e84:: with SMTP id e4-v6mr728799wru.32.1537876444031; Tue, 25 Sep 2018 04:54:04 -0700 (PDT) Received: from localhost.localdomain (89-95-107-230.abo.bbox.fr. [89.95.107.230]) by smtp.gmail.com with ESMTPSA id z14-v6sm1998628wrr.91.2018.09.25.04.54.02 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 25 Sep 2018 04:54:03 -0700 (PDT) From: Christian Couder X-Google-Original-From: Christian Couder To: git@vger.kernel.org Cc: Junio C Hamano , Jeff King , Ben Peart , Jonathan Tan , Nguyen Thai Ngoc Duy , Mike Hommey , Lars Schneider , Eric Wong , Christian Couder , Jeff Hostetler , Eric Sunshine , Beat Bolli , Christian Couder Subject: [PATCH v5 2/8] Add initial remote odb support Date: Tue, 25 Sep 2018 13:53:35 +0200 Message-Id: <20180925115341.19248-3-chriscool@tuxfamily.org> X-Mailer: git-send-email 2.19.0.278.gca5b891cac In-Reply-To: <20180925115341.19248-1-chriscool@tuxfamily.org> References: <20180925115341.19248-1-chriscool@tuxfamily.org> MIME-Version: 1.0 Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Christian Couder The remote-odb.{c,h} files will contain the functions that are called by the rest of Git mostly from "sha1-file.c" to access the objects managed by the remote odbs. The odb-helper.{c,h} files will contain the functions to actually implement communication with either the internal functions or the external scripts or processes that will manage and provide remote git objects. For now only infrastructure to create helpers from the config and to check if there are remote odbs and odb helpers is implemented. We expect that there will not be a lot of helpers, so it is ok to use a simple linked list to manage them. Helped-by: Jeff King Signed-off-by: Christian Couder Signed-off-by: Junio C Hamano --- Makefile | 2 ++ odb-helper.c | 16 +++++++++ odb-helper.h | 19 +++++++++++ remote-odb.c | 91 ++++++++++++++++++++++++++++++++++++++++++++++++++++ remote-odb.h | 7 ++++ 5 files changed, 135 insertions(+) create mode 100644 odb-helper.c create mode 100644 odb-helper.h create mode 100644 remote-odb.c create mode 100644 remote-odb.h diff --git a/Makefile b/Makefile index 13e1c52478..bed0b64ff7 100644 --- a/Makefile +++ b/Makefile @@ -915,6 +915,8 @@ LIB_OBJS += notes-cache.o LIB_OBJS += notes-merge.o LIB_OBJS += notes-utils.o LIB_OBJS += object.o +LIB_OBJS += odb-helper.o +LIB_OBJS += remote-odb.o LIB_OBJS += oidmap.o LIB_OBJS += oidset.o LIB_OBJS += packfile.o diff --git a/odb-helper.c b/odb-helper.c new file mode 100644 index 0000000000..b4d403ffa9 --- /dev/null +++ b/odb-helper.c @@ -0,0 +1,16 @@ +#include "cache.h" +#include "object.h" +#include "argv-array.h" +#include "odb-helper.h" +#include "run-command.h" +#include "sha1-lookup.h" + +struct odb_helper *odb_helper_new(const char *name, int namelen) +{ + struct odb_helper *o; + + o = xcalloc(1, sizeof(*o)); + o->name = xmemdupz(name, namelen); + + return o; +} diff --git a/odb-helper.h b/odb-helper.h new file mode 100644 index 0000000000..4b792a3896 --- /dev/null +++ b/odb-helper.h @@ -0,0 +1,19 @@ +#ifndef ODB_HELPER_H +#define ODB_HELPER_H + +/* + * An odb helper is a way to access a remote odb. + * + * Information in its fields comes from the config file [odb "NAME"] + * entries. + */ +struct odb_helper { + const char *name; /* odb..* */ + const char *remote; /* odb..promisorRemote */ + + struct odb_helper *next; +}; + +extern struct odb_helper *odb_helper_new(const char *name, int namelen); + +#endif /* ODB_HELPER_H */ diff --git a/remote-odb.c b/remote-odb.c new file mode 100644 index 0000000000..03be54ba2e --- /dev/null +++ b/remote-odb.c @@ -0,0 +1,91 @@ +#include "cache.h" +#include "remote-odb.h" +#include "odb-helper.h" +#include "config.h" + +static struct odb_helper *helpers; +static struct odb_helper **helpers_tail = &helpers; + +static struct odb_helper *find_or_create_helper(const char *name, int len) +{ + struct odb_helper *o; + + for (o = helpers; o; o = o->next) + if (!strncmp(o->name, name, len) && !o->name[len]) + return o; + + o = odb_helper_new(name, len); + *helpers_tail = o; + helpers_tail = &o->next; + + return o; +} + +static struct odb_helper *do_find_odb_helper(const char *remote) +{ + struct odb_helper *o; + + for (o = helpers; o; o = o->next) + if (o->remote && !strcmp(o->remote, remote)) + return o; + + return NULL; +} + +static int remote_odb_config(const char *var, const char *value, void *data) +{ + struct odb_helper *o; + const char *name; + int namelen; + const char *subkey; + + if (parse_config_key(var, "odb", &name, &namelen, &subkey) < 0) + return 0; + + o = find_or_create_helper(name, namelen); + + if (!strcmp(subkey, "promisorremote")) { + const char *remote; + int res = git_config_string(&remote, var, value); + + if (res) + return res; + + if (do_find_odb_helper(remote)) + return error(_("when parsing config key '%s' " + "helper for remote '%s' already exists"), + var, remote); + + o->remote = remote; + + return 0; + } + + return 0; +} + +static void remote_odb_init(void) +{ + static int initialized; + + if (initialized) + return; + initialized = 1; + + git_config(remote_odb_config, NULL); +} + +struct odb_helper *find_odb_helper(const char *remote) +{ + remote_odb_init(); + + if (!remote) + return helpers; + + return do_find_odb_helper(remote); +} + +int has_remote_odb(void) +{ + return !!find_odb_helper(NULL); +} diff --git a/remote-odb.h b/remote-odb.h new file mode 100644 index 0000000000..4c7b13695f --- /dev/null +++ b/remote-odb.h @@ -0,0 +1,7 @@ +#ifndef REMOTE_ODB_H +#define REMOTE_ODB_H + +extern struct odb_helper *find_odb_helper(const char *remote); +extern int has_remote_odb(void); + +#endif /* REMOTE_ODB_H */ From patchwork Tue Sep 25 11:53:36 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christian Couder X-Patchwork-Id: 10613871 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 133A86CB for ; Tue, 25 Sep 2018 11:54:21 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 059D229A16 for ; Tue, 25 Sep 2018 11:54:21 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id ED87929B6E; Tue, 25 Sep 2018 11:54:20 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C9D0A29A1A for ; Tue, 25 Sep 2018 11:54:18 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728676AbeIYSBS (ORCPT ); Tue, 25 Sep 2018 14:01:18 -0400 Received: from mail-wr1-f67.google.com ([209.85.221.67]:45351 "EHLO mail-wr1-f67.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728585AbeIYSBS (ORCPT ); Tue, 25 Sep 2018 14:01:18 -0400 Received: by mail-wr1-f67.google.com with SMTP id m16so6558477wrx.12 for ; Tue, 25 Sep 2018 04:54:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=BFA9rS8p4c61LWvQAensQ0gE4ApKPIjwa/4qy1dhlns=; b=BJD8A/hlcOSu0hTjaAV8RPNvqJyY0pRyg9gBr9eZVnpdP9lHibqY1lQcXgoN0QWNkc fxtX2oe3TlYsBWFxMecdw5evsG6CJfW/sJjQKvMOxbBaed7zvm3E/Tq4TYsE0k+l9Fd+ UiBE3uQKJRNh+5LGYZnCbCQJ8DYld8vhgTgCcueuRrN9UQefjD1YLml3qRsmHfUGwuYi bwR42tJhq0toBV4E7kMjdi0tCJTYIW4yJkhP0vGiVudh+yHK145lyhBVj4VgRDWtYEgU mvffYZDE4E3lqD+G+YbzLAyx2fYO8BAMfhliBxHtfvY+ONLoD3t4C3iT/Oh1UBHgB3RE IBCQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=BFA9rS8p4c61LWvQAensQ0gE4ApKPIjwa/4qy1dhlns=; b=TtTfbEKFyEwGt+gfWawFuBERX2V51jdL58Ar/qgRdzW17fSapHHx19RgNJJTUiKUhn Qhry4N+w2ucyV9T5cMWwNf42NF7AVgkQHzm6OY2PF5Z5232SdSGFHJPcIzhC7E79XU71 GzNP0HKu/wxPLR6xTw18clxZvNzgZQBJMU1xzZy8MKhh/E9OdqIDLumM8pU/Rsc9lRmd 0k4jFHaDygRjE0vw5tGziLwLMqtZVT659MKLjAjNu8mAtucbGVObN3WVugchn2nLkf7G UNnWKJOdMJ2NRmU8fyTF2l2Z2ZwheZ2ok1NZLwgvGzxbcbmKVhJi+5oazIqwQHzJVFKU kgYg== X-Gm-Message-State: ABuFfogu/VNPUNdCHhneYBt7LRmv4DHnTEhCI6Qm3u7Tyi5v/8E9Qtv0 beHzUkX9WSX0UoM4U/nQBSKrWEy4 X-Google-Smtp-Source: ACcGV60gaK+9Iql9O+eSQzvfx5Z1xVZsLA1e7s/vkpdQc44Y46FeustbhkuxkK5KG89h1wPVoKgytQ== X-Received: by 2002:adf:ecc4:: with SMTP id s4-v6mr728765wro.142.1537876445227; Tue, 25 Sep 2018 04:54:05 -0700 (PDT) Received: from localhost.localdomain (89-95-107-230.abo.bbox.fr. [89.95.107.230]) by smtp.gmail.com with ESMTPSA id z14-v6sm1998628wrr.91.2018.09.25.04.54.04 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 25 Sep 2018 04:54:04 -0700 (PDT) From: Christian Couder X-Google-Original-From: Christian Couder To: git@vger.kernel.org Cc: Junio C Hamano , Jeff King , Ben Peart , Jonathan Tan , Nguyen Thai Ngoc Duy , Mike Hommey , Lars Schneider , Eric Wong , Christian Couder , Jeff Hostetler , Eric Sunshine , Beat Bolli , Christian Couder Subject: [PATCH v5 3/8] remote-odb: implement remote_odb_get_direct() Date: Tue, 25 Sep 2018 13:53:36 +0200 Message-Id: <20180925115341.19248-4-chriscool@tuxfamily.org> X-Mailer: git-send-email 2.19.0.278.gca5b891cac In-Reply-To: <20180925115341.19248-1-chriscool@tuxfamily.org> References: <20180925115341.19248-1-chriscool@tuxfamily.org> MIME-Version: 1.0 Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Christian Couder This is implemented only in the promisor remote mode for now by calling fetch_objects(). Signed-off-by: Christian Couder Signed-off-by: Junio C Hamano --- odb-helper.c | 15 +++++++++++++++ odb-helper.h | 3 +++ remote-odb.c | 17 +++++++++++++++++ remote-odb.h | 1 + 4 files changed, 36 insertions(+) diff --git a/odb-helper.c b/odb-helper.c index b4d403ffa9..6458fe2e32 100644 --- a/odb-helper.c +++ b/odb-helper.c @@ -4,6 +4,7 @@ #include "odb-helper.h" #include "run-command.h" #include "sha1-lookup.h" +#include "fetch-object.h" struct odb_helper *odb_helper_new(const char *name, int namelen) { @@ -14,3 +15,17 @@ struct odb_helper *odb_helper_new(const char *name, int namelen) return o; } + +int odb_helper_get_direct(struct odb_helper *o, + const struct object_id *oids, + int oid_nr) +{ + int res; + uint64_t start = getnanotime(); + + res = fetch_objects(o->remote, oids, oid_nr); + + trace_performance_since(start, "odb_helper_get_direct"); + + return res; +} diff --git a/odb-helper.h b/odb-helper.h index 4b792a3896..950f9f9cd1 100644 --- a/odb-helper.h +++ b/odb-helper.h @@ -15,5 +15,8 @@ struct odb_helper { }; extern struct odb_helper *odb_helper_new(const char *name, int namelen); +extern int odb_helper_get_direct(struct odb_helper *o, + const struct object_id *oids, + int oid_nr); #endif /* ODB_HELPER_H */ diff --git a/remote-odb.c b/remote-odb.c index 03be54ba2e..53900203ae 100644 --- a/remote-odb.c +++ b/remote-odb.c @@ -89,3 +89,20 @@ int has_remote_odb(void) { return !!find_odb_helper(NULL); } + +int remote_odb_get_direct(const struct object_id *oids, int oid_nr) +{ + struct odb_helper *o; + + trace_printf("trace: remote_odb_get_direct: nr: %d", oid_nr); + + remote_odb_init(); + + for (o = helpers; o; o = o->next) { + if (odb_helper_get_direct(o, oids, oid_nr) < 0) + continue; + return 0; + } + + return -1; +} diff --git a/remote-odb.h b/remote-odb.h index 4c7b13695f..221305d1b6 100644 --- a/remote-odb.h +++ b/remote-odb.h @@ -3,5 +3,6 @@ extern struct odb_helper *find_odb_helper(const char *remote); extern int has_remote_odb(void); +extern int remote_odb_get_direct(const struct object_id *oids, int oid_nr); #endif /* REMOTE_ODB_H */ From patchwork Tue Sep 25 11:53:37 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christian Couder X-Patchwork-Id: 10613875 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 7012D14DA for ; Tue, 25 Sep 2018 11:54:24 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 62BF729A1A for ; Tue, 25 Sep 2018 11:54:24 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 55290298B3; Tue, 25 Sep 2018 11:54:24 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id EF499298B3 for ; Tue, 25 Sep 2018 11:54:18 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728691AbeIYSBT (ORCPT ); Tue, 25 Sep 2018 14:01:19 -0400 Received: from mail-wr1-f65.google.com ([209.85.221.65]:34766 "EHLO mail-wr1-f65.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727507AbeIYSBT (ORCPT ); Tue, 25 Sep 2018 14:01:19 -0400 Received: by mail-wr1-f65.google.com with SMTP id t15so15133797wrx.1 for ; Tue, 25 Sep 2018 04:54:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=x1TPPzAYx8eTf+3tateUOwAke8GGbgZ4/amSbQjcKUE=; b=N2iG4u3eVudnYNOCa3lyRXA94/7ndlMWk1uEKfCbWC7lae/AjmbhzyuHaBhi7iVqwZ UdrRvohFLiMX1mDoje6bQt6Ihh73KZsa0GwjXo2Yoh6PQnoayKPx0ZHg15cXigSrodAc PDsy/PxI02Sp5MbzcfEW7eyHMPS0OPozVyN1aT1RfuLu2RsE5K4wrZmSXXAwCFlM+JxH ZXEVzm9ThXesgFDP0D9deifZqlEF9vpZZ3WbfK60FIBAB+KUnrT3HLeEI6KhX25F1DLr KY8DVpZ270HZFSgHc1R4w04peo/80phz/Dqu/9MHnaU7ZFlZjLLaG8kZhaNeiJlSZM8G PtnA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=x1TPPzAYx8eTf+3tateUOwAke8GGbgZ4/amSbQjcKUE=; b=iZTa4jCZoqbaa8YwkpP62ZjNBCTUnTpiTR3VkWRVlKNvir9g3Fcb/+jolvueGYJjCL iGLpP2ePQ4QJujoHT8rvCjapYLKQLHVvLEiNAiTZ8Elm4tnmYquTRmqmJg3ZGb5TLVNf 7KJTJNR9wzH6ublE1clov4MtgfZLM8dxbzU1uSzyAcNuC7oD4vM0MvekaNideLFpuXcS imDcB9DkSFcruRvRp53sHbh4pc5hky5EwZvuUWBEKAwFBt7rn5lpA7NRMbecoyglOh6L BQ7f2TAK8ZBrYan1xpkYsE45Y840Qxwsgu2E1YuO6G0B2rYwl/HmM3rM9ZvQsRIU727w SdCQ== X-Gm-Message-State: ABuFfoizr2ZbNwyLB/KToicNtSipQS8xFZ0aKg6MoR3mXv377o8VcGQV jm2oXzMuyKnfdfxuFFq+ArT2G4ZN X-Google-Smtp-Source: ACcGV62pCNzBKkuWWrkyXhuTimwvanxFoUM8MpvCJTqpjqz491+AE6+v9d7VKkvCTcJXnwGFC/uxMQ== X-Received: by 2002:a05:6000:82:: with SMTP id m2mr676124wrx.28.1537876446606; Tue, 25 Sep 2018 04:54:06 -0700 (PDT) Received: from localhost.localdomain (89-95-107-230.abo.bbox.fr. [89.95.107.230]) by smtp.gmail.com with ESMTPSA id z14-v6sm1998628wrr.91.2018.09.25.04.54.05 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 25 Sep 2018 04:54:05 -0700 (PDT) From: Christian Couder X-Google-Original-From: Christian Couder To: git@vger.kernel.org Cc: Junio C Hamano , Jeff King , Ben Peart , Jonathan Tan , Nguyen Thai Ngoc Duy , Mike Hommey , Lars Schneider , Eric Wong , Christian Couder , Jeff Hostetler , Eric Sunshine , Beat Bolli , Christian Couder Subject: [PATCH v5 4/8] remote-odb: add remote_odb_reinit() Date: Tue, 25 Sep 2018 13:53:37 +0200 Message-Id: <20180925115341.19248-5-chriscool@tuxfamily.org> X-Mailer: git-send-email 2.19.0.278.gca5b891cac In-Reply-To: <20180925115341.19248-1-chriscool@tuxfamily.org> References: <20180925115341.19248-1-chriscool@tuxfamily.org> MIME-Version: 1.0 Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Christian Couder We will need to reinitialize the remote odb configuration as we will make some changes to it in a later commit when we will detect that a remote is also a remote odb. Signed-off-by: Christian Couder Signed-off-by: Junio C Hamano --- remote-odb.c | 14 ++++++++++++-- remote-odb.h | 1 + 2 files changed, 13 insertions(+), 2 deletions(-) diff --git a/remote-odb.c b/remote-odb.c index 53900203ae..cd1b393b79 100644 --- a/remote-odb.c +++ b/remote-odb.c @@ -64,17 +64,27 @@ static int remote_odb_config(const char *var, const char *value, void *data) return 0; } -static void remote_odb_init(void) +static void remote_odb_do_init(int force) { static int initialized; - if (initialized) + if (!force && initialized) return; initialized = 1; git_config(remote_odb_config, NULL); } +static inline void remote_odb_init(void) +{ + remote_odb_do_init(0); +} + +void remote_odb_reinit(void) +{ + remote_odb_do_init(1); +} + struct odb_helper *find_odb_helper(const char *remote) { remote_odb_init(); diff --git a/remote-odb.h b/remote-odb.h index 221305d1b6..0d1170cf41 100644 --- a/remote-odb.h +++ b/remote-odb.h @@ -1,6 +1,7 @@ #ifndef REMOTE_ODB_H #define REMOTE_ODB_H +extern void remote_odb_reinit(void); extern struct odb_helper *find_odb_helper(const char *remote); extern int has_remote_odb(void); extern int remote_odb_get_direct(const struct object_id *oids, int oid_nr); From patchwork Tue Sep 25 11:53:38 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christian Couder X-Patchwork-Id: 10613883 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B93EC161F for ; Tue, 25 Sep 2018 11:54:25 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A8EFF298B3 for ; Tue, 25 Sep 2018 11:54:25 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 9D15229A02; Tue, 25 Sep 2018 11:54:25 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 3C63629A16 for ; Tue, 25 Sep 2018 11:54:24 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728762AbeIYSBZ (ORCPT ); Tue, 25 Sep 2018 14:01:25 -0400 Received: from mail-wm1-f66.google.com ([209.85.128.66]:38465 "EHLO mail-wm1-f66.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727114AbeIYSBY (ORCPT ); Tue, 25 Sep 2018 14:01:24 -0400 Received: by mail-wm1-f66.google.com with SMTP id z16-v6so6887596wmi.3 for ; Tue, 25 Sep 2018 04:54:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=E+iCXS7jPu6phr0WIDO4E4yvYclZrJsSWEcjwmPKfeE=; b=PPAk7WgLsRS2vTE96IIIpJ0E11iK1tZW4LDKrrS4NuWjIM97Np7w+MltUarBEnMoIk o4S75Ts2uT9YFGCKLBDSLXzdLMSMJxUbJKSn5UO1SJ+9cTbXl8Ah07Ej0sm468Sl+8pQ E+Ix4ftf8FDzbR7WqJliJsmx1/TuMQav/+Bsaam8nB7Ge/NvYDf4J7wwsbsg/3qcnh5H FTxzfBfYGB7posQ0MEvfhuz+GrTI0KjBho7ARG1K9sAbQVGMPzR9UMBJyPDXEnwypbDC jrV1rJa2OYbDSJu1nNsBMywrQ8IY3pUoPlGBKM1fVuMMSIlSQo+4mDda7WWSLytqmMOa 7AJg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=E+iCXS7jPu6phr0WIDO4E4yvYclZrJsSWEcjwmPKfeE=; b=DUNnBXp6CAi7B6kx0b3PvGhFacTf40VjN33e02jWFyuJ2VhGKZL9zcHmEEhelrYRYb bVjVeQcyPZueLVcY35Mf+newILShedkMlac1UnML8KeYLWrjrnartA4cUySkVPqXPw9r WsxAACASFTIzegXGUjOge9kPrieOq+07F/MKTRJjASEPzj3K3qO4CT2EYKiJlCd1Y4fC anG/xAuoZFFxmxqBrRkfUm3R7oL2qexGiba1jV3BtoIYe+NpgZ8wg0bBrWz9+VIp6+kP C5Z1FPjaLsrIgdwrE6QXeWwdM6XEWPR7VDbxcZZlSG8gCiNwLW351dQIVjIP0qifYvFb 2mRQ== X-Gm-Message-State: ABuFfohLBfiwmrcADCT1oLQp496Gcg/L3B9+HnGnCpUMFM8U+S50xFQL P4RkjxupQp6hlDPOn1ESB35f2O8x X-Google-Smtp-Source: ACcGV635ZcoiAyxdh4ZJzmF7rP0wdyfgks4+bJDdcXYBWQkWDx8ze92rf3VrtTgWyw4fG/1RSaGJcA== X-Received: by 2002:a1c:55c2:: with SMTP id j185-v6mr396354wmb.104.1537876448418; Tue, 25 Sep 2018 04:54:08 -0700 (PDT) Received: from localhost.localdomain (89-95-107-230.abo.bbox.fr. [89.95.107.230]) by smtp.gmail.com with ESMTPSA id z14-v6sm1998628wrr.91.2018.09.25.04.54.06 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 25 Sep 2018 04:54:07 -0700 (PDT) From: Christian Couder X-Google-Original-From: Christian Couder To: git@vger.kernel.org Cc: Junio C Hamano , Jeff King , Ben Peart , Jonathan Tan , Nguyen Thai Ngoc Duy , Mike Hommey , Lars Schneider , Eric Wong , Christian Couder , Jeff Hostetler , Eric Sunshine , Beat Bolli Subject: [PATCH v5 5/8] Use remote_odb_get_direct() and has_remote_odb() Date: Tue, 25 Sep 2018 13:53:38 +0200 Message-Id: <20180925115341.19248-6-chriscool@tuxfamily.org> X-Mailer: git-send-email 2.19.0.278.gca5b891cac In-Reply-To: <20180925115341.19248-1-chriscool@tuxfamily.org> References: <20180925115341.19248-1-chriscool@tuxfamily.org> MIME-Version: 1.0 Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Instead of using the repository_format_partial_clone global and fetch_objects() directly, let's use has_remote_odb() and remote_odb_get_direct(). Signed-off-by: Christian Couder Signed-off-by: Junio C Hamano --- builtin/cat-file.c | 5 +++-- builtin/fetch.c | 11 ++++++----- builtin/gc.c | 3 ++- builtin/repack.c | 3 ++- cache.h | 2 -- connected.c | 3 ++- environment.c | 1 - list-objects-filter-options.c | 27 ++++++++++++++------------ packfile.c | 3 ++- setup.c | 7 +------ sha1-file.c | 14 ++++++++------ t/t0410-partial-clone.sh | 36 +++++++++++++++++------------------ t/t5500-fetch-pack.sh | 4 ++-- t/t5601-clone.sh | 2 +- t/t5616-partial-clone.sh | 2 +- t/t5702-protocol-v2.sh | 2 +- unpack-trees.c | 6 +++--- 17 files changed, 67 insertions(+), 64 deletions(-) diff --git a/builtin/cat-file.c b/builtin/cat-file.c index 64ec1745ab..7a2986d938 100644 --- a/builtin/cat-file.c +++ b/builtin/cat-file.c @@ -14,6 +14,7 @@ #include "sha1-array.h" #include "packfile.h" #include "object-store.h" +#include "remote-odb.h" struct batch_options { int enabled; @@ -507,8 +508,8 @@ static int batch_objects(struct batch_options *opt) if (opt->all_objects) { struct object_cb_data cb; - if (repository_format_partial_clone) - warning("This repository has extensions.partialClone set. Some objects may not be loaded."); + if (has_remote_odb()) + warning("This repository uses an odb. Some objects may not be loaded."); cb.opt = opt; cb.expand = &data; diff --git a/builtin/fetch.c b/builtin/fetch.c index 0696abfc2a..e01b65950c 100644 --- a/builtin/fetch.c +++ b/builtin/fetch.c @@ -23,6 +23,7 @@ #include "packfile.h" #include "list-objects-filter-options.h" #include "commit-reach.h" +#include "remote-odb.h" static const char * const builtin_fetch_usage[] = { N_("git fetch [] [ [...]]"), @@ -1372,7 +1373,7 @@ static inline void fetch_one_setup_partial(struct remote *remote) * If no prior partial clone/fetch and the current fetch DID NOT * request a partial-fetch, do a normal fetch. */ - if (!repository_format_partial_clone && !filter_options.choice) + if (!has_remote_odb() && !filter_options.choice) return; /* @@ -1380,7 +1381,7 @@ static inline void fetch_one_setup_partial(struct remote *remote) * on this repo and remember the given filter-spec as the default * for subsequent fetches to this remote. */ - if (!repository_format_partial_clone && filter_options.choice) { + if (!has_remote_odb() && filter_options.choice) { partial_clone_register(remote->name, &filter_options); return; } @@ -1389,7 +1390,7 @@ static inline void fetch_one_setup_partial(struct remote *remote) * We are currently limited to only ONE promisor remote and only * allow partial-fetches from the promisor remote. */ - if (strcmp(remote->name, repository_format_partial_clone)) { + if (!find_odb_helper(remote->name)) { if (filter_options.choice) die(_("--filter can only be used with the remote configured in core.partialClone")); return; @@ -1520,7 +1521,7 @@ int cmd_fetch(int argc, const char **argv, const char *prefix) if (depth || deepen_since || deepen_not.nr) deepen = 1; - if (filter_options.choice && !repository_format_partial_clone) + if (filter_options.choice && !has_remote_odb()) die("--filter can only be used when extensions.partialClone is set"); if (all) { @@ -1554,7 +1555,7 @@ int cmd_fetch(int argc, const char **argv, const char *prefix) } if (remote) { - if (filter_options.choice || repository_format_partial_clone) + if (filter_options.choice || has_remote_odb()) fetch_one_setup_partial(remote); result = fetch_one(remote, argc, argv, prune_tags_ok); } else { diff --git a/builtin/gc.c b/builtin/gc.c index 2b592260e9..356db8ab70 100644 --- a/builtin/gc.c +++ b/builtin/gc.c @@ -27,6 +27,7 @@ #include "pack-objects.h" #include "blob.h" #include "tree.h" +#include "remote-odb.h" #define FAILED_RUN "failed to run %s" @@ -623,7 +624,7 @@ int cmd_gc(int argc, const char **argv, const char *prefix) argv_array_push(&prune, prune_expire); if (quiet) argv_array_push(&prune, "--no-progress"); - if (repository_format_partial_clone) + if (has_remote_odb()) argv_array_push(&prune, "--exclude-promisor-objects"); if (run_command_v_opt(prune.argv, RUN_GIT_CMD)) diff --git a/builtin/repack.c b/builtin/repack.c index c6a7943d5c..16f9a952a0 100644 --- a/builtin/repack.c +++ b/builtin/repack.c @@ -11,6 +11,7 @@ #include "midx.h" #include "packfile.h" #include "object-store.h" +#include "remote-odb.h" static int delta_base_offset = 1; static int pack_kept_objects = -1; @@ -366,7 +367,7 @@ int cmd_repack(int argc, const char **argv, const char *prefix) argv_array_push(&cmd.args, "--all"); argv_array_push(&cmd.args, "--reflog"); argv_array_push(&cmd.args, "--indexed-objects"); - if (repository_format_partial_clone) + if (has_remote_odb()) argv_array_push(&cmd.args, "--exclude-promisor-objects"); if (write_bitmaps) argv_array_push(&cmd.args, "--write-bitmap-index"); diff --git a/cache.h b/cache.h index d508f3d4f8..195de98c2e 100644 --- a/cache.h +++ b/cache.h @@ -955,13 +955,11 @@ extern int grafts_replace_parents; #define GIT_REPO_VERSION 0 #define GIT_REPO_VERSION_READ 1 extern int repository_format_precious_objects; -extern char *repository_format_partial_clone; extern const char *core_partial_clone_filter_default; struct repository_format { int version; int precious_objects; - char *partial_clone; /* value of extensions.partialclone */ int is_bare; int hash_algo; char *work_tree; diff --git a/connected.c b/connected.c index 1bba888eff..7743b78290 100644 --- a/connected.c +++ b/connected.c @@ -4,6 +4,7 @@ #include "connected.h" #include "transport.h" #include "packfile.h" +#include "remote-odb.h" /* * If we feed all the commits we want to verify to this command @@ -56,7 +57,7 @@ int check_connected(oid_iterate_fn fn, void *cb_data, argv_array_push(&rev_list.args,"rev-list"); argv_array_push(&rev_list.args, "--objects"); argv_array_push(&rev_list.args, "--stdin"); - if (repository_format_partial_clone) + if (has_remote_odb()) argv_array_push(&rev_list.args, "--exclude-promisor-objects"); if (!opt->is_deepening_fetch) { argv_array_push(&rev_list.args, "--not"); diff --git a/environment.c b/environment.c index 3f3c8746c2..87c871e5f1 100644 --- a/environment.c +++ b/environment.c @@ -31,7 +31,6 @@ int warn_ambiguous_refs = 1; int warn_on_object_refname_ambiguity = 1; int ref_paranoia = -1; int repository_format_precious_objects; -char *repository_format_partial_clone; const char *core_partial_clone_filter_default; const char *git_commit_encoding; const char *git_log_output_encoding; diff --git a/list-objects-filter-options.c b/list-objects-filter-options.c index c0e2bd6a06..60452c8f36 100644 --- a/list-objects-filter-options.c +++ b/list-objects-filter-options.c @@ -6,6 +6,7 @@ #include "list-objects.h" #include "list-objects-filter.h" #include "list-objects-filter-options.h" +#include "remote-odb.h" /* * Parse value of the argument to the "filter" keyword. @@ -114,30 +115,32 @@ void partial_clone_register( const char *remote, const struct list_objects_filter_options *filter_options) { - /* - * Record the name of the partial clone remote in the - * config and in the global variable -- the latter is - * used throughout to indicate that partial clone is - * enabled and to expect missing objects. - */ - if (repository_format_partial_clone && - *repository_format_partial_clone && - strcmp(remote, repository_format_partial_clone)) - die(_("cannot change partial clone promisor remote")); + char *cfg_name; + + /* Check if it is already registered */ + if (find_odb_helper(remote)) + return; git_config_set("core.repositoryformatversion", "1"); - git_config_set("extensions.partialclone", remote); - repository_format_partial_clone = xstrdup(remote); + /* Add odb config for the remote */ + cfg_name = xstrfmt("odb.%s.promisorRemote", remote); + git_config_set(cfg_name, remote); + free(cfg_name); /* * Record the initial filter-spec in the config as * the default for subsequent fetches from this remote. + * + * TODO: move core.partialclonefilter into odb. */ core_partial_clone_filter_default = xstrdup(filter_options->filter_spec); git_config_set("core.partialclonefilter", core_partial_clone_filter_default); + + /* Make sure the config info are reset */ + remote_odb_reinit(); } void partial_clone_get_default_filter_spec( diff --git a/packfile.c b/packfile.c index 841b36182f..16d12c8e86 100644 --- a/packfile.c +++ b/packfile.c @@ -16,6 +16,7 @@ #include "tree.h" #include "object-store.h" #include "midx.h" +#include "remote-odb.h" char *odb_pack_name(struct strbuf *buf, const unsigned char *sha1, @@ -2101,7 +2102,7 @@ int is_promisor_object(const struct object_id *oid) static int promisor_objects_prepared; if (!promisor_objects_prepared) { - if (repository_format_partial_clone) { + if (has_remote_odb()) { for_each_packed_object(add_promisor_object, &promisor_objects, FOR_EACH_OBJECT_PROMISOR_ONLY); diff --git a/setup.c b/setup.c index b24c811c1c..4eeb7fe073 100644 --- a/setup.c +++ b/setup.c @@ -419,11 +419,7 @@ static int check_repo_format(const char *var, const char *value, void *vdata) ; else if (!strcmp(ext, "preciousobjects")) data->precious_objects = git_config_bool(var, value); - else if (!strcmp(ext, "partialclone")) { - if (!value) - return config_error_nonbool(var); - data->partial_clone = xstrdup(value); - } else + else string_list_append(&data->unknown_extensions, ext); } else if (strcmp(var, "core.bare") == 0) { data->is_bare = git_config_bool(var, value); @@ -465,7 +461,6 @@ static int check_repository_format_gently(const char *gitdir, struct repository_ } repository_format_precious_objects = candidate->precious_objects; - repository_format_partial_clone = candidate->partial_clone; string_list_clear(&candidate->unknown_extensions, 0); if (!has_common) { if (candidate->is_bare != -1) { diff --git a/sha1-file.c b/sha1-file.c index 5c44873d37..c9ed42e513 100644 --- a/sha1-file.c +++ b/sha1-file.c @@ -32,6 +32,7 @@ #include "packfile.h" #include "fetch-object.h" #include "object-store.h" +#include "remote-odb.h" /* The maximum size for an object header. */ #define MAX_HEADER_LEN 32 @@ -1309,15 +1310,16 @@ int oid_object_info_extended(struct repository *r, const struct object_id *oid, } /* Check if it is a missing object */ - if (fetch_if_missing && repository_format_partial_clone && + if (fetch_if_missing && has_remote_odb() && !already_retried && r == the_repository) { /* - * TODO Investigate checking fetch_object() return - * TODO value and stopping on error here. - * TODO Pass a repository struct through fetch_object, - * such that arbitrary repositories work. + * TODO Investigate checking remote_odb_get_direct() + * TODO return value and stopping on error here. + * TODO Pass a repository struct through + * remote_odb_get_direct(), such that arbitrary + * repositories work. */ - fetch_objects(repository_format_partial_clone, real, 1); + remote_odb_get_direct(real, 1); already_retried = 1; continue; } diff --git a/t/t0410-partial-clone.sh b/t/t0410-partial-clone.sh index cfd0655ea1..9ead9860f5 100755 --- a/t/t0410-partial-clone.sh +++ b/t/t0410-partial-clone.sh @@ -23,7 +23,7 @@ promise_and_delete () { delete_object repo "$HASH" } -test_expect_success 'extensions.partialclone without filter' ' +test_expect_success 'promisor remote without filter' ' test_create_repo server && git clone --filter="blob:none" "file://$(pwd)/server" client && git -C client config --unset core.partialclonefilter && @@ -51,7 +51,7 @@ test_expect_success 'missing reflog object, but promised by a commit, passes fsc # But with the extension, it succeeds git -C repo config core.repositoryformatversion 1 && - git -C repo config extensions.partialclone "arbitrary string" && + git -C repo config odb.magic.promisorRemote "arbitrary string" && git -C repo fsck ' @@ -74,7 +74,7 @@ test_expect_success 'missing reflog object, but promised by a tag, passes fsck' printf "$T\n" | pack_as_from_promisor && git -C repo config core.repositoryformatversion 1 && - git -C repo config extensions.partialclone "arbitrary string" && + git -C repo config odb.magic.promisorRemote "arbitrary string" && git -C repo fsck ' @@ -92,7 +92,7 @@ test_expect_success 'missing reflog object alone fails fsck, even with extension delete_object repo "$A" && git -C repo config core.repositoryformatversion 1 && - git -C repo config extensions.partialclone "arbitrary string" && + git -C repo config odb.magic.promisorRemote "arbitrary string" && test_must_fail git -C repo fsck ' @@ -108,7 +108,7 @@ test_expect_success 'missing ref object, but promised, passes fsck' ' promise_and_delete "$A" && git -C repo config core.repositoryformatversion 1 && - git -C repo config extensions.partialclone "arbitrary string" && + git -C repo config odb.magic.promisorRemote "arbitrary string" && git -C repo fsck ' @@ -131,7 +131,7 @@ test_expect_success 'missing object, but promised, passes fsck' ' promise_and_delete "$AT" && git -C repo config core.repositoryformatversion 1 && - git -C repo config extensions.partialclone "arbitrary string" && + git -C repo config odb.magic.promisorRemote "arbitrary string" && git -C repo fsck ' @@ -144,7 +144,7 @@ test_expect_success 'missing CLI object, but promised, passes fsck' ' promise_and_delete "$A" && git -C repo config core.repositoryformatversion 1 && - git -C repo config extensions.partialclone "arbitrary string" && + git -C repo config odb.magic.promisorRemote "arbitrary string" && git -C repo fsck "$A" ' @@ -159,7 +159,7 @@ test_expect_success 'fetching of missing objects' ' rm -rf repo/.git/objects/* && git -C repo config core.repositoryformatversion 1 && - git -C repo config extensions.partialclone "origin" && + git -C repo config odb.magic.promisorRemote "origin" && git -C repo cat-file -p "$HASH" && # Ensure that the .promisor file is written, and check that its @@ -192,7 +192,7 @@ test_expect_success 'rev-list stops traversal at missing and promised commit' ' promise_and_delete "$FOO" && git -C repo config core.repositoryformatversion 1 && - git -C repo config extensions.partialclone "arbitrary string" && + git -C repo config odb.magic.promisorRemote "arbitrary string" && GIT_TEST_COMMIT_GRAPH=0 git -C repo rev-list --exclude-promisor-objects --objects bar >out && grep $(git -C repo rev-parse bar) out && ! grep $FOO out @@ -217,7 +217,7 @@ test_expect_success 'rev-list stops traversal at missing and promised tree' ' promise_and_delete "$TREE2" && git -C repo config core.repositoryformatversion 1 && - git -C repo config extensions.partialclone "arbitrary string" && + git -C repo config odb.magic.promisorRemote "arbitrary string" && git -C repo rev-list --exclude-promisor-objects --objects HEAD >out && grep $(git -C repo rev-parse foo) out && ! grep $TREE out && @@ -236,7 +236,7 @@ test_expect_success 'rev-list stops traversal at missing and promised blob' ' promise_and_delete "$BLOB" && git -C repo config core.repositoryformatversion 1 && - git -C repo config extensions.partialclone "arbitrary string" && + git -C repo config odb.magic.promisorRemote "arbitrary string" && git -C repo rev-list --exclude-promisor-objects --objects HEAD >out && grep $(git -C repo rev-parse HEAD) out && ! grep $BLOB out @@ -255,7 +255,7 @@ test_expect_success 'rev-list stops traversal at promisor commit, tree, and blob printf "%s\n%s\n%s\n" $COMMIT $TREE $BLOB | pack_as_from_promisor && git -C repo config core.repositoryformatversion 1 && - git -C repo config extensions.partialclone "arbitrary string" && + git -C repo config odb.magic.promisorRemote "arbitrary string" && git -C repo rev-list --exclude-promisor-objects --objects HEAD >out && ! grep $COMMIT out && ! grep $TREE out && @@ -279,7 +279,7 @@ test_expect_success 'rev-list accepts missing and promised objects on command li promise_and_delete $BLOB && git -C repo config core.repositoryformatversion 1 && - git -C repo config extensions.partialclone "arbitrary string" && + git -C repo config odb.magic.promisorRemote "arbitrary string" && git -C repo rev-list --exclude-promisor-objects --objects "$COMMIT" "$TREE" "$BLOB" ' @@ -295,7 +295,7 @@ test_expect_success 'gc repacks promisor objects separately from non-promisor ob printf "$TREE_TWO\n" | pack_as_from_promisor && git -C repo config core.repositoryformatversion 1 && - git -C repo config extensions.partialclone "arbitrary string" && + git -C repo config odb.magic.promisorRemote "arbitrary string" && git -C repo gc && # Ensure that exactly one promisor packfile exists, and that it @@ -329,7 +329,7 @@ test_expect_success 'gc does not repack promisor objects if there are none' ' test_commit -C repo one && git -C repo config core.repositoryformatversion 1 && - git -C repo config extensions.partialclone "arbitrary string" && + git -C repo config odb.magic.promisorRemote "arbitrary string" && git -C repo gc && # Ensure that only one pack exists @@ -351,7 +351,7 @@ test_expect_success 'repack -d does not irreversibly delete promisor objects' ' rm -rf repo && test_create_repo repo && git -C repo config core.repositoryformatversion 1 && - git -C repo config extensions.partialclone "arbitrary string" && + git -C repo config odb.magic.promisorRemote "arbitrary string" && git -C repo commit --allow-empty -m one && git -C repo commit --allow-empty -m two && @@ -379,7 +379,7 @@ test_expect_success 'gc stops traversal when a missing but promised object is re HASH=$(promise_and_delete $TREE_HASH) && git -C repo config core.repositoryformatversion 1 && - git -C repo config extensions.partialclone "arbitrary string" && + git -C repo config odb.magic.promisorRemote "arbitrary string" && git -C repo gc && # Ensure that the promisor packfile still exists, and remove it @@ -410,7 +410,7 @@ test_expect_success 'fetching of missing objects from an HTTP server' ' rm -rf repo/.git/objects/* && git -C repo config core.repositoryformatversion 1 && - git -C repo config extensions.partialclone "origin" && + git -C repo config odb.magic.promisorRemote "origin" && git -C repo cat-file -p "$HASH" && # Ensure that the .promisor file is written, and check that its diff --git a/t/t5500-fetch-pack.sh b/t/t5500-fetch-pack.sh index 1b5a4a6d38..7be1dd9a4c 100755 --- a/t/t5500-fetch-pack.sh +++ b/t/t5500-fetch-pack.sh @@ -884,14 +884,14 @@ fetch_filter_blob_limit_zero () { test_config -C "$SERVER" uploadpack.allowfilter 1 && git clone "$URL" client && - test_config -C client extensions.partialclone origin && + test_config -C client odb.magic.promisorRemote origin && test_commit -C "$SERVER" two && git -C client fetch --filter=blob:limit=0 origin HEAD:somewhere && # Ensure that commit is fetched, but blob is not - test_config -C client extensions.partialclone "arbitrary string" && + test_config -C client odb.magic.promisorRemote "arbitrary string" && git -C client cat-file -e $(git -C "$SERVER" rev-parse two) && test_must_fail git -C client cat-file -e $(git hash-object "$SERVER/two.t") } diff --git a/t/t5601-clone.sh b/t/t5601-clone.sh index f1a49e94f5..d0cd8b6233 100755 --- a/t/t5601-clone.sh +++ b/t/t5601-clone.sh @@ -653,7 +653,7 @@ partial_clone () { git -C client fsck && # Ensure that unneeded blobs are not inadvertently fetched. - test_config -C client extensions.partialclone "not a remote" && + test_config -C client odb.origin.promisorRemote "not a remote" && test_must_fail git -C client cat-file -e "$HASH1" && # But this blob was fetched, because clone performs an initial checkout diff --git a/t/t5616-partial-clone.sh b/t/t5616-partial-clone.sh index bbbe7537df..9897b8db12 100755 --- a/t/t5616-partial-clone.sh +++ b/t/t5616-partial-clone.sh @@ -40,7 +40,7 @@ test_expect_success 'do partial clone 1' ' | sort >observed.oids && test_cmp expect_1.oids observed.oids && test "$(git -C pc1 config --local core.repositoryformatversion)" = "1" && - test "$(git -C pc1 config --local extensions.partialclone)" = "origin" && + test "$(git -C pc1 config --local odb.origin.promisorRemote)" = "origin" && test "$(git -C pc1 config --local core.partialclonefilter)" = "blob:none" ' diff --git a/t/t5702-protocol-v2.sh b/t/t5702-protocol-v2.sh index 3beeed4546..e683dd4ca1 100755 --- a/t/t5702-protocol-v2.sh +++ b/t/t5702-protocol-v2.sh @@ -290,7 +290,7 @@ test_expect_success 'partial fetch' ' rm -rf client "$(pwd)/trace" && git init client && SERVER="file://$(pwd)/server" && - test_config -C client extensions.partialClone "$SERVER" && + test_config -C client odb.magic.promisorRemote "$SERVER" && GIT_TRACE_PACKET="$(pwd)/trace" git -C client -c protocol.version=2 \ fetch --filter=blob:none "$SERVER" master:refs/heads/other && diff --git a/unpack-trees.c b/unpack-trees.c index 51bfac6aa0..a5ce98a450 100644 --- a/unpack-trees.c +++ b/unpack-trees.c @@ -18,6 +18,7 @@ #include "fsmonitor.h" #include "object-store.h" #include "fetch-object.h" +#include "remote-odb.h" /* * Error messages expected by scripts out of plumbing commands such as @@ -418,7 +419,7 @@ static int check_updates(struct unpack_trees_options *o) load_gitmodules_file(index, &state); enable_delayed_checkout(&state); - if (repository_format_partial_clone && o->update && !o->dry_run) { + if (has_remote_odb() && o->update && !o->dry_run) { /* * Prefetch the objects that are to be checked out in the loop * below. @@ -435,8 +436,7 @@ static int check_updates(struct unpack_trees_options *o) } } if (to_fetch.nr) - fetch_objects(repository_format_partial_clone, - to_fetch.oid, to_fetch.nr); + remote_odb_get_direct(to_fetch.oid, to_fetch.nr); fetch_if_missing = fetch_if_missing_store; oid_array_clear(&to_fetch); } From patchwork Tue Sep 25 11:53:39 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christian Couder X-Patchwork-Id: 10613877 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B7A7D6CB for ; Tue, 25 Sep 2018 11:54:24 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id AAEFC29A02 for ; Tue, 25 Sep 2018 11:54:24 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 9F74F29B6E; Tue, 25 Sep 2018 11:54:24 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 1C0BE29A02 for ; Tue, 25 Sep 2018 11:54:24 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728743AbeIYSBY (ORCPT ); Tue, 25 Sep 2018 14:01:24 -0400 Received: from mail-wm1-f67.google.com ([209.85.128.67]:35112 "EHLO mail-wm1-f67.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727507AbeIYSBY (ORCPT ); Tue, 25 Sep 2018 14:01:24 -0400 Received: by mail-wm1-f67.google.com with SMTP id o18-v6so13350504wmc.0 for ; Tue, 25 Sep 2018 04:54:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=82/YC43NZuh9XZK3VNrWiW2adioe3Bp9GacznZijBkw=; b=UyiRSbzj3bNOrONdOw/bZ0KfNNUW8Yc/SFmLVhZyCIjK36RY2km3DSaRbcDm3Br6zB 53Q/RSTPTPKqVyYTZ4RcVPp/tajoGmy85q7VyiwN52g1jg5lH+EA66X1qSZvlyXmaeF5 i0xAUTZ+/nVIdR5AI9woe0BZMGEKDgawmfj9tZtToQ1+FVPOIoucBEyyQKAXB+T3fhIT RzIyXyGjSIIsrLYI6/xeu1twtBcdgMMg2mVeGq9UuH6srbrLYa8ncUxp6WS/V/jN/eXr 2dV8JGWUZGUFDiw1CqRiUOKz7W9Om76Q6y/udo36arwblqts8tF4cAamLi+McKaYqZhq UL5A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=82/YC43NZuh9XZK3VNrWiW2adioe3Bp9GacznZijBkw=; b=ZKI9LCOFxeu4vzQ/fE+TdoRPpyIrBVpHajJR8GVY9uHTc+bB1PbdM1mxMZXH/lvJ27 xliJGsxhqEV16hwgzMgeM/zM9FJZzyY8vNphKeGEwYHDuFyLHFnLSQBugJYme63u92ly +bV3UAk24WA2bzJ43wseeCQFZMIGFGQ4WZZcd/U++z1/ZqPSUBAi4BGstnYJtwmliLpC oWHth/bI6TgYQKjikg6z2O58Uq9Wr+GIAQ1MdOxamy6M+fTs62tFm9PUpfdCQhJCll/2 O/gGhKU/CaP1u33S4uBh8SIbmUedERN5iUgLICvCWlrbnxhggNk9bUJnrFdfCIVO/TlY mE/A== X-Gm-Message-State: ABuFfoioI/0GIq07jWeDAgiHZYw+B3m5pOy8cty4gvSJoTwonTs3RYYy ei8IB6neNISa7w3GNfFaoG/WSMOw X-Google-Smtp-Source: ACcGV60g7reiWmNdSp4ZVI+bKAiTpLyAglz/r2sKeXrlCPtN0OuykpzKSVEMhRnWq2DB19hB7b7C8g== X-Received: by 2002:a1c:cc1a:: with SMTP id h26-v6mr393520wmb.112.1537876449745; Tue, 25 Sep 2018 04:54:09 -0700 (PDT) Received: from localhost.localdomain (89-95-107-230.abo.bbox.fr. [89.95.107.230]) by smtp.gmail.com with ESMTPSA id z14-v6sm1998628wrr.91.2018.09.25.04.54.08 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 25 Sep 2018 04:54:09 -0700 (PDT) From: Christian Couder X-Google-Original-From: Christian Couder To: git@vger.kernel.org Cc: Junio C Hamano , Jeff King , Ben Peart , Jonathan Tan , Nguyen Thai Ngoc Duy , Mike Hommey , Lars Schneider , Eric Wong , Christian Couder , Jeff Hostetler , Eric Sunshine , Beat Bolli , Christian Couder Subject: [PATCH v5 6/8] Use odb.origin.partialclonefilter instead of core.partialclonefilter Date: Tue, 25 Sep 2018 13:53:39 +0200 Message-Id: <20180925115341.19248-7-chriscool@tuxfamily.org> X-Mailer: git-send-email 2.19.0.278.gca5b891cac In-Reply-To: <20180925115341.19248-1-chriscool@tuxfamily.org> References: <20180925115341.19248-1-chriscool@tuxfamily.org> MIME-Version: 1.0 Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Christian Couder Let's make the partial clone filter specific to one odb instead of general to all the odbs. This makes it possible to have different partial clone filters for different odbs. Signed-off-by: Christian Couder Signed-off-by: Junio C Hamano --- builtin/fetch.c | 2 +- list-objects-filter-options.c | 28 ++++++++++++++++------------ list-objects-filter-options.h | 3 ++- odb-helper.h | 1 + remote-odb.c | 2 ++ t/t0410-partial-clone.sh | 2 +- t/t5616-partial-clone.sh | 2 +- 7 files changed, 24 insertions(+), 16 deletions(-) diff --git a/builtin/fetch.c b/builtin/fetch.c index e01b65950c..b8701ef7bc 100644 --- a/builtin/fetch.c +++ b/builtin/fetch.c @@ -1402,7 +1402,7 @@ static inline void fetch_one_setup_partial(struct remote *remote) * the config. */ if (!filter_options.choice) - partial_clone_get_default_filter_spec(&filter_options); + partial_clone_get_default_filter_spec(&filter_options, remote->name); return; } diff --git a/list-objects-filter-options.c b/list-objects-filter-options.c index 60452c8f36..755a793664 100644 --- a/list-objects-filter-options.c +++ b/list-objects-filter-options.c @@ -7,6 +7,7 @@ #include "list-objects-filter.h" #include "list-objects-filter-options.h" #include "remote-odb.h" +#include "odb-helper.h" /* * Parse value of the argument to the "filter" keyword. @@ -29,6 +30,9 @@ static int gently_parse_list_objects_filter( { const char *v0; + if (!arg) + return 0; + if (filter_options->choice) { if (errbuf) { strbuf_init(errbuf, 0); @@ -116,6 +120,7 @@ void partial_clone_register( const struct list_objects_filter_options *filter_options) { char *cfg_name; + char *filter_name; /* Check if it is already registered */ if (find_odb_helper(remote)) @@ -131,27 +136,26 @@ void partial_clone_register( /* * Record the initial filter-spec in the config as * the default for subsequent fetches from this remote. - * - * TODO: move core.partialclonefilter into odb. */ - core_partial_clone_filter_default = - xstrdup(filter_options->filter_spec); - git_config_set("core.partialclonefilter", - core_partial_clone_filter_default); + filter_name = xstrfmt("odb.%s.partialclonefilter", remote); + git_config_set(filter_name, filter_options->filter_spec); + free(filter_name); /* Make sure the config info are reset */ remote_odb_reinit(); } void partial_clone_get_default_filter_spec( - struct list_objects_filter_options *filter_options) + struct list_objects_filter_options *filter_options, + const char *remote) { + struct odb_helper *helper = find_odb_helper(remote); + /* * Parse default value, but silently ignore it if it is invalid. */ - if (!core_partial_clone_filter_default) - return; - gently_parse_list_objects_filter(filter_options, - core_partial_clone_filter_default, - NULL); + if (helper) + gently_parse_list_objects_filter(filter_options, + helper->partial_clone_filter, + NULL); } diff --git a/list-objects-filter-options.h b/list-objects-filter-options.h index 0000a61f82..12ceef3230 100644 --- a/list-objects-filter-options.h +++ b/list-objects-filter-options.h @@ -74,6 +74,7 @@ void partial_clone_register( const char *remote, const struct list_objects_filter_options *filter_options); void partial_clone_get_default_filter_spec( - struct list_objects_filter_options *filter_options); + struct list_objects_filter_options *filter_options, + const char *remote); #endif /* LIST_OBJECTS_FILTER_OPTIONS_H */ diff --git a/odb-helper.h b/odb-helper.h index 950f9f9cd1..9ec3ff5141 100644 --- a/odb-helper.h +++ b/odb-helper.h @@ -10,6 +10,7 @@ struct odb_helper { const char *name; /* odb..* */ const char *remote; /* odb..promisorRemote */ + const char *partial_clone_filter; /* odb..partialCloneFilter */ struct odb_helper *next; }; diff --git a/remote-odb.c b/remote-odb.c index cd1b393b79..57d0215aaa 100644 --- a/remote-odb.c +++ b/remote-odb.c @@ -60,6 +60,8 @@ static int remote_odb_config(const char *var, const char *value, void *data) return 0; } + if (!strcmp(subkey, "partialclonefilter")) + return git_config_string(&o->partial_clone_filter, var, value); return 0; } diff --git a/t/t0410-partial-clone.sh b/t/t0410-partial-clone.sh index 9ead9860f5..8b32be6417 100755 --- a/t/t0410-partial-clone.sh +++ b/t/t0410-partial-clone.sh @@ -26,7 +26,7 @@ promise_and_delete () { test_expect_success 'promisor remote without filter' ' test_create_repo server && git clone --filter="blob:none" "file://$(pwd)/server" client && - git -C client config --unset core.partialclonefilter && + git -C client config --unset odb.origin.partialclonefilter && git -C client fetch origin ' diff --git a/t/t5616-partial-clone.sh b/t/t5616-partial-clone.sh index 9897b8db12..12e964c1bc 100755 --- a/t/t5616-partial-clone.sh +++ b/t/t5616-partial-clone.sh @@ -41,7 +41,7 @@ test_expect_success 'do partial clone 1' ' test_cmp expect_1.oids observed.oids && test "$(git -C pc1 config --local core.repositoryformatversion)" = "1" && test "$(git -C pc1 config --local odb.origin.promisorRemote)" = "origin" && - test "$(git -C pc1 config --local core.partialclonefilter)" = "blob:none" + test "$(git -C pc1 config --local odb.origin.partialclonefilter)" = "blob:none" ' # checkout master to force dynamic object fetch of blobs at HEAD. From patchwork Tue Sep 25 11:53:40 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christian Couder X-Patchwork-Id: 10613881 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 28E9214DA for ; Tue, 25 Sep 2018 11:54:25 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 1A9EF298B3 for ; Tue, 25 Sep 2018 11:54:25 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 0F36229A02; Tue, 25 Sep 2018 11:54:25 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 929B729AD5 for ; Tue, 25 Sep 2018 11:54:24 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728733AbeIYSBY (ORCPT ); Tue, 25 Sep 2018 14:01:24 -0400 Received: from mail-wm1-f68.google.com ([209.85.128.68]:50855 "EHLO mail-wm1-f68.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728706AbeIYSBY (ORCPT ); Tue, 25 Sep 2018 14:01:24 -0400 Received: by mail-wm1-f68.google.com with SMTP id s12-v6so13101013wmc.0 for ; Tue, 25 Sep 2018 04:54:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=utcEQkLTMiEtKiW/tm+AQXYhpj5a+7iQ7z+RB055oGA=; b=bMhuVY3dHPUUDZ6n6+qF0oCkfFOUiQ8/sxkfjJ7mra0VA8pfbBt5xSBf57sLBCV1/k abWBTHGcqNrdqcgoSlDNer620yG9zkvaQbeFXHw2Sn0R5LDNayW1qbYR+7tWC2ynBV6r txzfyfd9EP93F9c2U9BxE7C/6knJ5KHynb6VqrReR7IFB1qQtJ9VUAYkZQr/oI9wJA2F q3HBWEqjVV5OxqndBZy6meH+6wjy2TChWE7/oV7Xw9cUe88KviSEneVbXCIW//bQj1tN E8u485qMDrKvG7jNTglomZZgU+5NK+0OT+e/ICzEMo38ueG3ce0fO9Fbm41T+Mu+hKS9 WiTA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=utcEQkLTMiEtKiW/tm+AQXYhpj5a+7iQ7z+RB055oGA=; b=rLpRZjCmpP7bl7LowXvByd143a8MCsI0rbC8cpB3uRzsWhWXRYnQt7olfSsT1+FXYa +JjuDy0bfc8zTwhMNqnbpVXO43XWb/7FxA1bWuWUehKHUQbI9zVqhVx/nKgC3QWDfh7K Cswathj5CjvYZcOrF0moBL3tIQrnAK/3P+cVwIdQpn44xZtrvXQcc6nlA3r0v6ir3gqz jL3plukqXzn+lHeJIK183kuLPcnZ0SIixJC0QWJav8eoH9OoOQ0y3CBRH5HHh+0gx4+x JQ4nOZyPmK8UGMTGLnCFVBCeuFVJ1+irWVXNytNJmllPUBewPwjMhoKIPSIHqH3V3lxT En4g== X-Gm-Message-State: ABuFfogWQWHZY+MVR6XPHyqC5ndk1ZitNmyHIyicAlKc6fCYQsVgHlmO uBC3lBhjH8xH8vjUD9RZAGqhhI5g X-Google-Smtp-Source: ACcGV63s09atQAAFjEVplt6w+wtVHxH/oPjG0rvGdJtdG2bZb1mzI3iKUpTUDsLxsemtP7yvYmfPug== X-Received: by 2002:a1c:e289:: with SMTP id z131-v6mr426986wmg.32.1537876451024; Tue, 25 Sep 2018 04:54:11 -0700 (PDT) Received: from localhost.localdomain (89-95-107-230.abo.bbox.fr. [89.95.107.230]) by smtp.gmail.com with ESMTPSA id z14-v6sm1998628wrr.91.2018.09.25.04.54.09 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 25 Sep 2018 04:54:10 -0700 (PDT) From: Christian Couder X-Google-Original-From: Christian Couder To: git@vger.kernel.org Cc: Junio C Hamano , Jeff King , Ben Peart , Jonathan Tan , Nguyen Thai Ngoc Duy , Mike Hommey , Lars Schneider , Eric Wong , Christian Couder , Jeff Hostetler , Eric Sunshine , Beat Bolli , Christian Couder Subject: [PATCH v5 7/8] t0410: test fetching from many promisor remotes Date: Tue, 25 Sep 2018 13:53:40 +0200 Message-Id: <20180925115341.19248-8-chriscool@tuxfamily.org> X-Mailer: git-send-email 2.19.0.278.gca5b891cac In-Reply-To: <20180925115341.19248-1-chriscool@tuxfamily.org> References: <20180925115341.19248-1-chriscool@tuxfamily.org> MIME-Version: 1.0 Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Christian Couder Signed-off-by: Christian Couder Signed-off-by: Junio C Hamano --- t/t0410-partial-clone.sh | 24 +++++++++++++++++++++++- 1 file changed, 23 insertions(+), 1 deletion(-) diff --git a/t/t0410-partial-clone.sh b/t/t0410-partial-clone.sh index 8b32be6417..3fbd8d919e 100755 --- a/t/t0410-partial-clone.sh +++ b/t/t0410-partial-clone.sh @@ -170,6 +170,28 @@ test_expect_success 'fetching of missing objects' ' git verify-pack --verbose "$IDX" | grep "$HASH" ' +test_expect_success 'fetching of missing objects from another odb remote' ' + git clone "file://$(pwd)/server" server2 && + test_commit -C server2 bar && + git -C server2 repack -a -d --write-bitmap-index && + HASH2=$(git -C server2 rev-parse bar) && + + git -C repo remote add server2 "file://$(pwd)/server2" && + git -C repo config odb.magic2.promisorRemote server2 && + git -C repo cat-file -p "$HASH2" && + + git -C repo fetch server2 && + rm -rf repo/.git/objects/* && + git -C repo cat-file -p "$HASH2" && + + # Ensure that the .promisor file is written, and check that its + # associated packfile contains the object + ls repo/.git/objects/pack/pack-*.promisor >promisorlist && + test_line_count = 1 promisorlist && + IDX=$(cat promisorlist | sed "s/promisor$/idx/") && + git verify-pack --verbose "$IDX" | grep "$HASH2" +' + test_expect_success 'fetching of missing objects works with ref-in-want enabled' ' # ref-in-want requires protocol version 2 git -C server config protocol.version 2 && @@ -183,7 +205,7 @@ test_expect_success 'fetching of missing objects works with ref-in-want enabled' ' test_expect_success 'rev-list stops traversal at missing and promised commit' ' - rm -rf repo && + rm -rf repo server server2 && test_create_repo repo && test_commit -C repo foo && test_commit -C repo bar && From patchwork Tue Sep 25 11:53:41 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christian Couder X-Patchwork-Id: 10613879 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id EF92E1803 for ; Tue, 25 Sep 2018 11:54:24 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E0858298B3 for ; Tue, 25 Sep 2018 11:54:24 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id D501829A1A; Tue, 25 Sep 2018 11:54:24 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 71D19298B3 for ; Tue, 25 Sep 2018 11:54:24 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728771AbeIYSB0 (ORCPT ); Tue, 25 Sep 2018 14:01:26 -0400 Received: from mail-wm1-f65.google.com ([209.85.128.65]:34065 "EHLO mail-wm1-f65.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728720AbeIYSBZ (ORCPT ); Tue, 25 Sep 2018 14:01:25 -0400 Received: by mail-wm1-f65.google.com with SMTP id j25-v6so12427107wmc.1 for ; Tue, 25 Sep 2018 04:54:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=fV41yInvCdhsvUv2wVmgXInhjEVSuqiAazcH8xp6V44=; b=vgaiU4eb7nv67DmxkB5Y7EsgqAkfrOLWpiRzIJQ/42lUYExBwlDIP3tHXXGvK10Tj4 85dKe5P0Rtdlg+2pNuUdbYtLDTeQ852mOkeHGT7uwq4rql63Hs7wkDGfK6ve2ZYJejDe rzzQb3cQ4rorOCoU1Q3jKZ1fK88syHdkxuwmsp86pm7oyCoOGMfpHMDwF3NA6AnU4Eyq aD/ivYlcgBC61WfVkIa5qIMCiXcroR8j7O+URXGOH9Tgyj6qPKXsvNsTDhbQJ1fS5nru jBajfjEZ/Gut1QNqOvOknEOrwSvvzwCj9NfJ+sRtOGM7q03lmWahLBvtDDBdySNA/ym6 Nk8Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=fV41yInvCdhsvUv2wVmgXInhjEVSuqiAazcH8xp6V44=; b=sxG8l8Ms2KGiYYjXFfU2K5Ub1ejtpREYh0JYPaArS4rihkLuj9UdQe/7vHdqK25KVX j6lXCcnnbbSXfA6fvw/U2dR/6cA20Q1NyYN1kWY6r/JXu45WX+xi+l03yWlWfSRNPyPO zJ8CfJCvOGwmfuqROkRXjjDdiNu5TSEXUGqm88uefpbZ4YWfcLesyizNGST4aZiOk8pQ ORikZ5reFOzdcgfJrl9w3sVf8DRUvwcmVQe+zHmjL+CVPoO5RaOwPr6TiQsrPVBH5ZQe 5z3uASERobxcK3XPbn/+gGkviKAOqU6XncZSO1B2UQOrnFFRq2r/bbNxlTkfpJ+SdCT+ 9qQg== X-Gm-Message-State: ABuFfog7rBLQjFB+gix47nhxhGXFWN5PiGBEuSFeqb99iFSWSOnfjki3 npvB+pJGpE37EQen3yJVHP/Eo7Ku X-Google-Smtp-Source: ACcGV621955V2cDm7zV67UIOaZCNwHX7v2bKqrQ+J3DmLphN7UCQCvZtxb4fNP/VZjW11qJXcf7sfg== X-Received: by 2002:a1c:c8d:: with SMTP id 135-v6mr394897wmm.116.1537876452628; Tue, 25 Sep 2018 04:54:12 -0700 (PDT) Received: from localhost.localdomain (89-95-107-230.abo.bbox.fr. [89.95.107.230]) by smtp.gmail.com with ESMTPSA id z14-v6sm1998628wrr.91.2018.09.25.04.54.11 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 25 Sep 2018 04:54:11 -0700 (PDT) From: Christian Couder X-Google-Original-From: Christian Couder To: git@vger.kernel.org Cc: Junio C Hamano , Jeff King , Ben Peart , Jonathan Tan , Nguyen Thai Ngoc Duy , Mike Hommey , Lars Schneider , Eric Wong , Christian Couder , Jeff Hostetler , Eric Sunshine , Beat Bolli , Christian Couder Subject: [PATCH v5 8/8] Documentation/config: add odb..promisorRemote Date: Tue, 25 Sep 2018 13:53:41 +0200 Message-Id: <20180925115341.19248-9-chriscool@tuxfamily.org> X-Mailer: git-send-email 2.19.0.278.gca5b891cac In-Reply-To: <20180925115341.19248-1-chriscool@tuxfamily.org> References: <20180925115341.19248-1-chriscool@tuxfamily.org> MIME-Version: 1.0 Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Christian Couder Signed-off-by: Junio C Hamano --- Documentation/config.txt | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/Documentation/config.txt b/Documentation/config.txt index ad0f4510c3..9df988adb9 100644 --- a/Documentation/config.txt +++ b/Documentation/config.txt @@ -2655,6 +2655,11 @@ This setting can be overridden with the `GIT_NOTES_REWRITE_REF` environment variable, which must be a colon separated list of refs or globs. +odb..promisorRemote:: + The name of a promisor remote. For now promisor remotes are + the only kind of remote object database (odb) that is + supported. + pack.window:: The size of the window used by linkgit:git-pack-objects[1] when no window size is given on the command line. Defaults to 10.